Pandoc: A Tool I Use and Like

Today I want to talk to you about one of my favorite command-line tools, Pandoc. From the project website:

If you need to convert files from one markup format into another, pandoc is your swiss-army knife.

I spend a lot of time writing, and I love Vim, Markdown, and the command line (and avoid browser-based WYSIWYG editors when I can), so that’s where a lot of my Pandoc use comes in, but it has a ton of utility outside of that – really, anywhere you need to move between different text-based formats, Pandoc can probably help. A few examples from recent memory:

Markdown ➞ Craft Blog Post

This website you’re reading presently uses Craft CMS, a flexible and powerful content management system that doesn’t perfectly match my writing process¹. Rather than composing directly in Craft, I prefer to write locally, pipe the output through Pandoc, and put the resulting HTML into a text block in the CMS. This gets me a few things I really like:

• Curly quotes in place of straight ones and en-dashes in place of -- (from the smart extension)
• Daring Fireball-style footnotes with return links

By default, Pandoc uses Pandoc Markdown when converting Markdown docs to other formats, an “extended and slightly revised version” of the original syntax, which is how footnotes and a bunch of other things work.

Markdown ➞ Rich Text (Basecamp)

I also sometimes find myself writing decently long Basecamp posts. Basecamp 3 has a fine WYSIWYG editor (🪦 Textile), but again, I’d rather be in Vim. Pasting HTML into Basecamp doesn’t work (just shows the code verbatim), but I’ve found that if I convert my Markdown notes to HTML and open the HTML in a browser, I can copy and paste that directly into Basecamp with good results. Leveraging MacOS’ open command, this one-liner does the trick²:

cat [filename.md] \
  | pandoc -t html \
  > /tmp/output.html \
  && open /tmp/output.html \
  && read -n 1 \
  && rm /tmp/output.html

This will convert the contents to HTML, save that to a file, open the file in a browser, wait for the user to hit enter, and the remove the file. Without that read -n 1, it’ll remove the file before the browser has a chance to open it.

HTML ➞ Text

We built an app for one of our clients that takes in news articles (in HTML) via an API and sends them as emails to their clients (think big brands) if certain criteria are met. Recently, we were making improvements to the plain text version of the emails, and we noticed that some of the articles were coming in without any linebreaks in the content. When we removed the HTML (via Rails’ strip_tags helper), the resulting content was all on one line, which wasn’t very readable. So imagine an article like this:

<h1>Headline</h1> <p>A paragraph.</p> <ul><li>List item #1</li> <li>List item #2</li></ul>

Our initial approach (with strip_tags) gives us this:

Headline A paragraph. List item #1 List item #2

Not great! But fortunately, some bright fellow had the idea to pull in Pandoc, and some even brighter person packaged up some Ruby bindings for it. Taking that same content and running it through PandocRuby.html(content).to_plain gives us:

Headline

A paragraph.

-   List item #1
-   List item #2

Much better, and though you can’t tell from this basic example, Pandoc does a great job with spacing and wrapping to generate really nice-looking plain text from HTML.

HTML Element ➞ Text

A few months ago, we were doing Pointless Weekend and needed a domain for our Thrillr app. A few of us were looking through lists of fun top-level domains, but we realized that AWS Route 53 only supports a limited set of them. In order to get everyone the actual list, I needed a way to get all the content out of an HTML <select> element, and you’ll never guess what I did (unless you guessed “use Pandoc”). In Firefox:

• Right click the select element, then click “Inspect”
• Find the <select> in the DOM view that pops up
• Right click it, then go to “Copy”, then “Inner HTML”
• You’ll now have all of the <option> elements on your clipboard
• In your terminal, run pbpaste | pandoc -t plain

The result is something like this:

.ac - $76.00
.academy - $12.00
.accountants - $94.00
.agency - $19.00
.apartments - $47.00
.associates - $29.00
.au - $15.00
.auction - $29.00
...

Preview Mermaid/Markdown (--standalone)

A different client recently asked for an architecture diagram of a complex system that Andrew and I were working on, and we opted to use Mermaid (which is rad BTW) to create sequence diagrams to illustrate all of the interactions. Both GitHub and GitLab support Mermaid natively, which is really neat, but we wanted a way to quickly iterate on our diagrams without having to push changes to the remote repo.

We devised a simple build chain (demo version available here) that watches for changes to a Markdown file, converts the Mermaid blocks to SVG, and then uses Pandoc to take the resulting document and convert it to a styled HTML page using the --standalone option (here’s the key line). Then we could simply make our changes and refresh the page to see our progress.

Generate a PDF

Finally, and this is not something I need to do very often, but Pandoc also includes several ways to create PDF documents. The simplest (IMO) is to install wkhtmltopdf, then instruct Pandoc to convert its input to HTML but use .pdf in the output filename, so something like:

echo "# Hello\n\nIs it me you're looking for?" | pandoc -t html -o hello.pdf

The result is quite nice.

I think that’s about all I have to say about Pandoc for today. A couple final thoughts:

• Pandoc is incredibly powerful – I’ve really only scratched the surface here. Look at the man page to get a sense of everything it can do.
• Pandoc is written in Haskell, and the source is pretty fun to look through if you’re a certain kind of person.

So install Pandoc with your package manager of choice and give it a shot. I think you’ll find it unexpectedly useful.

Swiss army knife icons created by smalllikeart - Flaticon