Migrating our Documentation to GitHub Pages

Synchro Labs recently migrated our documentation from Zendesk to a GitHub Pages / Jekyll solution. See the new docs at http://docs.synchro.io. Details on why we did this, and how we overcame some of the technical challenges of the new solution, are provided below...

First Attempt: Zendesk Docs

We use Zendesk for customer support incident / ticket management. When we set up our Zendesk system over a year ago, we were excited to see that Zendesk offered a documentation system that was integrated with their support system. We were especially interested in features like the ability to see what documentation page the user was looking at when they opened a support ticket, and to collect customer feedback on our documentation.

We proceeded to move all of our documentation to Zendesk, and create new documentation in that system. We spent a significant amount of time tuning our Zendesk docs site to get the look and feel we wanted (consistent with our main website). I don't want to go into too much detail or to beat Zendesk up too much here, but the process involved a lot of metal-on-metal grinding sounds.

We had invested a significant amount of effort into the Zendesk templates for our documentation system, but there was no mechanism to backup or restore any of that code (you type it into a web UX they provide, and that's it). So before making any changes, which was itself reasonably cumbersome, you had to remember to copy and paste the text of every template you changed and go save them somewhere offline in case you needed to roll back.

The documentation editing environment was also kind of a nightmare. They don't support Markdown, or anything comparable. You enter documentation using a terrible WYSIWIG control through their UX, and that control actually adds HTML tags to the text that you enter. Once you get to a certain point of complexity, that starts to fail (if you edit raw code, which you have to do often for things like code samples, or anything remotely "custom", and then accidentally use the WYSIWIG mode to make a tweak, it can corrupt the raw code changes you made, as just one example of the kind of fun it is). Again, there's no versioning or roll back, no way to track changes, or see who made a change.

OK, That Didn't Work, What Now?

It became apparent that the Zendesk Docs system was not going to work going forward. It was so much of a hassle that we were actively discouraged from making docs changes because we didn't want to deal with the docs system, and that meant we had to make a change. Creating and updating docs should be as easy and inviting as possible, and anyone on the team should feel comfortable working on docs when needed, with a minimum of friction.

We needed a docs system where:

  • docs are written in a system that is format free and easy to understand
  • code samples are handled well (we have a lot of code samples in our docs)
  • versioning and audit are supported (the ability to see change history, including who made a given change, and to roll back as needed)

And in a perfect world:

  • docs are maintained alongside the code they document

GitHub Pages to the Rescue

GitHub has a system called GitHub Pages that can be used to publish any static website from a GitHub repo. We looked at GitHub Pages quite a while back and we didn't really like it. The main issue was that the content of the pages was required to be in a specific branch (gh-pages). So the docs were not really right next to the code, and it was a pain to have to maintain multiple workspaces or switch branches just to work on docs. That was pretty much a deal breaker.

But luckily, GitHub changed this a while back, and you can now put your "Pages" (docs, in our case) in a /docs directory in your project. This makes it easier to find and manage your docs, and it allows you to make changes to code and docs in a single commit (or pull request).

GitHub Pages uses Jekyll under the covers, and Jekyll supports a number of text engines, including ones that support Markdown formatting. Markdown is our text formatting system of choice (we use it for this blog as well), and it supports code blocks well. And since the docs are in GitHub, in a folder in our project, we have full versioning and audit support, and the docs are maintained alongside the code they document. So that checks all the boxes.

Woo Hoo, This Should Take Like 5 Minutes!

I'm not going to cover the nuts and bolts of getting your GitHub Pages / Jekyll environment up and running. Both GitHub and Jekyll have guides to help with that. See them at:

The one suggestion that I will make is that you should make a local copy of the template you choose, because you will pretty much definitely need to edit it in multiple places, pretty much as soon as you start trying to make your docs site work.

When you have everything configured and installed correctly, you can run your Jekyll site locally by doing:

bundle exec jekyll serve

Wait, My Docs Site isn't a Blog!

Jekyll is, out of the box, very "bloggy". The small amount of pain we had with Jekyll mostly had to do with trying to make it overcome its bias for "posts" organized by date (as expressed in the document filename of the "post").

You can look at the source code for the complete system we discuss below in the Synchro GitHub repo docs directory.

Collections

In our case, our documentation was grouped into collections of documents (General, Controls, Tutorial, Samples, and FAQ). So we decided to use Jekyll Collections to organize them. To make a collection, you create a directory with the appropriate collection name, prefixed with an underscore. Then you add a collections entry to your _config.yml file. Here is what our collections entry looks like:

collections:
  general:
    permalink: /:collection/:slug
    output: true
    name: 'General'
  controls:
    permalink: /:collection/:slug
    output: true
    name: 'Controls'
  tutorial:
    permalink: /:collection/:slug
    output: true
    name: 'Tutorial'
  samples:
    permalink: /:collection/:slug
    output: true
    name: 'Samples'
  faq:
    permalink: /:collection/:slug
    output: true
    name: 'Frequently Asked Questions'

The permalink values above override the default permalink, which would be :/collection/:path. This is important if you will be overriding any of the document names at any point (such as we will discuss below). By using /:slug, the document links will be similar to the default links using /:path by default, but can be easily overridden on a per-document basis by setting a slug value in the front matter of a given document.

The output: true part tells Jekyll to create a document/page for each file it finds in the corresponding directory (_general, _controls, etc).

Contents and Ordering using Weights

We needed a "Contents" or directory page for each section that listed the documents in the section. To do this, we created a top level markdown file for each section that was named with the section name. Here is the general.md file (for the "General" section):

---
layout: page
title: General
description: Synchro Architecture, Design, Usage, and API Documentation
permalink: /general/
collection: general
weight: 1
---

## {{ page.description }}

{% assign items = site.general | sort: 'weight' %}
{% for item in items %}
  <a href="{{ site.baseurl }}{{ item.url }}">{{ item.title }}</a>
{% endfor %}

This page shows the section description (from the front matter of the document), then it iterates the members of the collection and lists them. One important thing to point out is that our documents needed to be shown in a defined order. Since Jekyll shows the documents in order of filename, we needed a mechanism to override that and list the documents in our preferred order. There are a number of techniques for doing this, but we chose to assign each document in the collection a weight, and have the contents page use that value to order the documents. That is what is happening in the document above. For example, the first document in the General collection has front matter that looks like this:

---
title: Application Model
weight: 1
---

One of the advantages of the weight approach is that you can change the order of the documents without having to change the filename (though you may have to edit the weights of several documents). This is in contrast to the other main solution we have seen to this problem, which is to prefix each document filename with something that sorts it in the proper order (like 000_App_Model.md).

Of course you could start with higher granularity weight values, like 100, 200, 300, etc, to leave room in between for inserting future documents.

Contents and Ordering using Filenames (mostly)

A couple of our collections had a document ordering that was almost the same as the document title / filename. For example, our Controls collection lists the controls alphabetically, with a couple of exceptions. We have a page with common properties that apply to all controls that needs to come first, and we have three pages of platform-specific controls that come last. Since we intend to always have the control pages in the middle be listed in alphanumeric order, even when we add new ones, we decided on an approach that wouldn't require us to edit or maintain a weight in each document.

We named the "Common Control Attributes" page 0_common.md and we named the platform-specific pages in the form z_controlname.md. This caused the pages to sort in the correct order by filename. In order to correct the permalinks to these "special" pages (so our hack would not be visible to users), we overrode the slug value in the front matter of just those specific pages. This is one reason that using the slug in the permalink (as discussed above) is important. For example, the front matter for 0_common.md looks like this:

---
title: Common Control Attributes
slug: common
---

Using collections, and using these two techniques to maintain document order, allowed us to easily show the contents of these document collections on a "Contents" page, and to show them in the correct order.

Not So Fast with the Built-In Code Formatting

The recommended markdown engine in Jekyll is kramdown, and kramdown supports code highlighting using the Rouge engine. Jekyll, kramdown, Rouge, and all their friends, are implemented in Ruby. My guess is that the Rouge formatting for Ruby itself is probably top notch. But in our many Javascript snippets, we ran into a whole host of issues. In looking at the Rouge GitHub repo, we did not get a good feeling. I don't want to necessarily accuse it of being abandoned, but several of our problems were documented in issues that have been open for a long time. There are lots of open issues and pull requests. The primary maintainer made some noises about not having a lot of time to work on it due to a change in employment. Anyway, our issues weren't going to get fixed anytime soon, and I had no desire to get waist-deep in Ruby to fix them myself.

In our previous system, we used Highlight.js to do syntax highlighting client-side. That library is very solid, and very well supported. We found that we could disable Rouge formatting in kramdown such that it would still produce <code> blocks from markdown, but not apply any highlighting markup/formatting (so that we could do that with Highlight.js instead). To do this, we had to do the following in _config.yml:

kramdown:
  input: GFM
  hard_wrap: false
  syntax_highlighter: rouge
  syntax_highlighter_opts:
    disable: true

Then we dropped the Hightlight.js CSS and Javascript into docs/_includes/head.html, as below:

<link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/8.5/styles/github.min.css">
<script src="//cdnjs.cloudflare.com/ajax/libs/highlight.js/8.5/highlight.min.js"></script>
<script>hljs.initHighlightingOnLoad();</script>

That's all you need to do to swap out Rouge and swap in Highlight.js.

Search Would be Nice

There is no search facility provided by GitHub Pages or Jekyll. These are just static sites we're generating/serving after all. After doing some research, we came up with two search solutions that we felt were viable:

The lunr.js solution does client-side indexing and searching. It seems reasonably widely-used and well supported. It's a little more work to integrate than DocSearch, and it might not scale well with large documents or document sets. But it's self-contained and easy to customize and extend.

DocSearch is basically "search-as-a-service" (provided by Algolia, as they will remind you and your users in your search UX). It's pretty easy to integrate, and they give you a nice autocomplete-style search box (which actually uses autocomplete.js).

We chose DocSearch.

Once you sign up for DocSearch, you will get an email with a set of integration instructions that looks like this:

- Copy the following CSS/JS snippets and add them to your page

<!-- at the end of the HEAD -->
<link rel="stylesheet" href="https://cdn.jsdelivr.net/docsearch.js/2/docsearch.min.css" />
<!-- at the end of the BODY -->
<script type="text/javascript" src="https://cdn.jsdelivr.net/docsearch.js/2/docsearch.min.js"></script>
<script type="text/javascript"> docsearch({
  apiKey: '### OUR_API_KEY_WAS_HERE ###',
  indexName: 'synchro',
  inputSelector: '### REPLACE ME ####',
  debug: false // Set debug to true if you want to inspect the dropdown
});
</script>

- Add a search input if you don't have any yet, and update the inputSelector value in the code snippet to a CSS selector that targets your input field.

- Optionally customize the look and feel by following the DocSearch documentation (https://community.algolia.com/docsearch/documentation/)

- You can also check your configuration in the github repo (https://github.com/algolia/docsearch-configs/blob/master/configs/synchro.json). Feel free to open pull requests!

We accomplished this as follows:

We are pretty happy with the resulting search functionality, especially given how easy it was to implement.

Conclusion

Our docs site hosted by GitHub is available at docs.synchro.io. Feel free to explore. Try out the search. If you'd like to see the full source for that site, look no further than the docs directory of the Synchro Server repo.

We are much, much happier with our new GitHub Pages / Jekyll docs solution, and we're confident that it's going to contribute to us keeping our docs thorough and up-to-date in the future.