CI/CD Documentation for people who hate writing docs

I like solving problems. I hate writing up the documentation that comes along with it. Since I took a new position within Red Hat, I have found an increasing amount of my time taken up with writing docs. I decided the time had come for some innovation and workflow improvement.

Problem Statement

  • Our current document store of record is Google Drive. So any solution has to keep the final product in there. It can keep them in other locations, but this one is a requirement.
  • I don’t want to have to transcribe notes from calls and meetings. It’s annoying enough to take notes. It’s doubly-annoying to have to then transcribe them into another format for consumption by other people. A little clean-up is OK, but nothing far beyond that.
  • Copy/Paste into multiple platforms isn’t something I want to do. I want to take my notes, perform an action and have them published.
  • I need a universal format. PDF, HTML. Something.

My Solution

Available Tools

Internally, Red Hat uses the following self-service tools that I am utilizing for this time-saver.

  • DDNS (Dynamic DNS)
  • OpenStack
  • GitLab for version control
  • Jenkins for CI/CD
  • Google Drive for docs store


I looked around quite a bit before setting on using asciidoctor to process asciidoc files for me. It will take extremely light markdown and use it to render really pretty HTML. I’m a huge fan of it. I won’t be providing a primer on it here, but the one from my bookmarks that I use the most is found on the asciidoctor website.

The biggest benefit is that I can generate it almost as fast as someone can talk. So after a meeting, it’s just a minute or two of clean up and clarification and BANG, I have a consumable record of the event.

The Workflow

Having settled on asciidoc as my format, the workflow cleared up a lot.

  1. Generate asciidoc files during meetings / events / whatever.
  2. Manage them per-project / customer in git repos on GitLab.
  3. When a push is made to a given repo, have GitLab trigger a Jenkins build job.
  4. The Jenkins build job will take the updated repo, render the finished HTML, and upload it to Google Drive as well as a secondary web server that I will maintain (my choice, not a hard requirement).
  5. Profit and World Domination


I ran into a few obstacles when I started bringing this to life

  • I had never really used the more advanced features in GitLab.
  • I hadn’t used Jenkins in years
  • I am not familiar with the Google Drive API.

Dynamic DNS

I used our internal DynamicDNS for my web server and my Jenkins server. I don’t control any DNS zones inside Red Hat, so this was a quick and easy solution.

We have an internal registration page, as well as an RPM that configures a system. I just edit a file with the host, domain and hash and POOF, I have DDNS wherever I want it.

Setting up GitLab

The GitLab instance I’m using is 7.2.2, and is maintained by our internal IT team. So I won’t be covering how to set it up. I have done this in the past and it was dead simple, however. I followed their walkthrough and it worked like a champ.

Installing and Configuring Jenkins

We do have multiple internal Jenkins servers for our Engineers. However, I decided to go with my own so I could play around with plugins and break it without incurring the wrath of some project manager or delaying a major product release. The process was very straight-forward. I followed their wiki to get it  up and running in approximately 20 minutes.

Of course, adding a job to an existing Jenkins environment is possible, too. You just need the correct plugins installed.

Jenkins Plugins

I am utilizing a handful of Jenkins plugins to produce this workflow. Note: A few of these me come installed in a default install. I simply don’t remember. It comes with quite a few plugins to enable the default configuration.

  • Git plugin (this may be pulled in with the GitLab plugin, but I installed it first while experimenting)
  • Gitlab plugin – for integration with our GitLab instance
  • Publish Over SSH plugin – for publishing to a simple web server

Helpful Tip I forgot about Jenkins

Make sure your Jenkins server has any needed build software installed. In my case, git and asciidoctor are very important. That is 20 minutes I’ll never get back.

Integrating with Google Drive

This turned out to be the biggest challenge. There is no good glue out there for this already. The biggest obstacle is OAuth. It’s just designed for user interaction. I didn’t want to enable less secure passwords, so I decided to try and tackle this.

This is the only place I had to write any new code. I ended up using PyDrive to access the Google Drive API more easily because I’m not very familiar with the API itself. It worked well. Since GDrive is really an object store more than anything else, updating a document instead of just adding another copy of it. This is a first attempt to deal with that cleanly. I worked on it for about an hour, so there is no concept of polish there as of yet. Think of it more as a POC that it’s doable.

The code is in a public Github repo. Ideas and code-heckling are welcome.

Gluing it all together

I now have my asciidoc code, GitLab, Jenkins, GDrive, and a web server. Now I need to glue them together to make my life easier.

The local git repo itself doesn’t get changed at all. No post_commit hooks, although that method would work as well I’m sure.

  1. Create a GitLab repo
    1. This is outside of this blog’s scope, and more importantly it’s dead easy.
  2. Getting Jenkins Connected
    1. Define a server to SSH your finished HTML to
      1. Manage Jenkins > Configure System
        1. Publish Over SSH
          1. Key
            1. a private key that will work on your web server. Since I’m using a VM from our internal OpenStack instance, I am using the same key I use for ‘cloud-user’ on those VM’s
          2. SSH Server
            1. Name – anything you like. I used ‘Web Docs Server’
            2. Hostname – I used the DDNS name I set up for this system
            3. Username – cloud-user (the key is already there)
            4. Remote Directory – /var/www/html
              1. Since this is a standard RHEL 7.1 install, that is where the default DocRoot is for apache. Since I am the only one using this system and it has no external visibility I chown’d /var/www/html to I know that’s a total hack, but this is also just a POC. I promise, I do know a little bit about web security.
  3. Setting up the google drive updated code to work on your Jenkins server
    1. This was put together from several PyDrive tutorials, especially this one.
    2. Since this code uses OAuth2 to handle authentication, you have to set that up for your Google Account.
    3. Go into the Google API Control Panel and create a new Application. Their instructions are pretty solid.
      1. You will need the ‘Client ID’ and ‘Client Secret’ for that application.
      2. Inside the App Details, click the ‘Download JSON’. Save this file as ‘client_secrets.json’ (what PyDrive looks for by default)
      3. Create a file called ‘settings.yaml’ and populate it like the sample file in the links above. All you need to change are the values for your Client ID and Client Secret.
      4. At this point I used their demo code at to generate an additional file named ‘credentials.json’. This is the active token that is referenced during the login session. It is refreshed by the OAuth code in PyDrive. Take these 4 files and upload them somewhere easily read on your Jenkins server. I placed them all in /usr/local/bin/gdriveupdate. Be sure to make sure the gdriveupdate file itself is executable. It is what will be called during the Jenkins Build
        1. I’m not sure how long this token will be refreshed. I guess I will ultimately know that once the build fails because of it. Hopefully I’ll have conquered that little challenge by then. Feel free to file an issue on GitHub.
  4. Create a Jenkins job
    1. Fill in the GitLab Repository Name (user/project)
    2. Source Control Management
      1. Git
      2. Repository URL – the ssh compact version for your project
      3. Credentials
        1. I’m using an ssh key for this one. It’s associated with my Jenkins user credentials
      4. Repository Broswer – gitlab
        1. URL – URL for your project
        2. Version – this auto-populated for me
    3. Build Triggers
      1. Build when a change is pushed to GitLab
        1. make note of the CI Service URL
      2. I took the default values
    4. Build Environment
      1. Select the Server you created previously
      2. Source Files – index.html (or more if you’re generating other stuff)
      3. Remote Directory – this will be auto-created in the remote server’s root directory auto-magically. You can name it anything that makes sense for your project
    5. Build – Execute Shell
      1. /usr/bin/asciidoctor -dbook index.adoc
        cp -r /usr/local/bin/gdriveupdate/* $WORKSPACE
        /usr/local/bin/gdriveupdate/gdriveupdate -f $WORKSPACE/index.html -g CSA_Philips_Home_Monitoring
      2. index.adoc is just the convention I’ve adopted. You can redirect the output name via the command line and call it whatever you like.
      3. copying everything for the GDrive into the build workspace is a total hack. I know that. PyDrive can’t find the json and secrets files unless they’re in the current working directory for some reason. Some weird pathing issue that I don’t yet feel like debugging in the project.
  5. Configure GitLab to trigger a Jenkins build
    1. This is based on the GitLab plugin for Jenkins documentation
      1. It is very version specific, but I found that the simple instructions for version 8.0 and higher worked just fine for me. You just have to create a web hook for push and merge events.
      2. The Jenkins URL for the webhook is in the Project Config page in the Build Trigger section where you select the GitLab option.
      3. Go to your GitLab project > Settings > Web Hooks
        1. Select Merge Request and Push events
        2. Paste in the URL from your Jenkins project.


And that’s it. You write about 100 lines of Python to incorporate Google Drive, create a GitLab repo and Jenkins build job. You then link GitLab to Jenkins with a web hook. The Jenkins build then creates your HTML (or your desired format) from your asciidoc and uploads it to Google Drive and your web server.

Now, when I make a push to my GitLab repo after taking notes or writing docs for a given project or customer, the workflow kicks off and publishes my docs in both locations. The build takes < 10 seconds on average. And since it’s a push and not a poll-driven event, they are available almost instantly.


Technical Knowledge Needed – 8/10. You’re not writing kernel modules but it is gluing together several large tools).
Time Requirement – 4/10. This is less than a full day’s work once you have the answers in front of you. To generate the workflow took me about 2 days, all told.

git2changelog – converting your git logs into a spec file changelog automatically

Especially when it comes to application development, I’m a bit lazy and a general malcontent. I love solving the problem, but I hate dealing with the packaging and versioning and all of the stuff that makes something usable.  One of the things I always have trouble with is keeping track of my spec file changelog when I am rolling something into an RPM.

To help ease that I put together a small script that will take a git repository’s log between any two tags and output it in a format that is acceptable in an RPM spec file.

To do this I started with the Fedora Packaging Guidelines for Changelogs. This gave me the proper formatting to adhere to for my script.

Next I used the changelog in the sosreport package for inspiration. It’s available in its spec file.

The script I wrote is designed to run inside of a git repository.  If you can come up with a better way to collage this data from the .git directory then please feel free to share. So I’ve stuck it in a git repo so you can grab it if you want.

The output of my soscleaner app looks like this:

$ ./git2changelog -b 0.1-8

* Sat Jun 07 2014 Jamie Duncan <> - HEAD:UNRELEASED
- 2f78c26 =review - added comment in _skip_file to likely remove a now useless if clause
- Merge pull request #11 from bmr-cymru/bmr-libmagic-fixes : Commit 4427b06
- Convert python to use native libmagic python-magic bindings : Commit 7db6a99
- Rename __init__ options argument for clarity : Commit f1353ea
- cleaning up the magic import - fixes #10 : Commit 554af49
- removing shebang from py module : Commit f70b856
- more cleanup : Commit 6ee1339

* Wed Jun 04 2014 Jamie Duncan <> - 0.1-12
- 47b3a47 =adding dist flag to spec file
- clean up : Commit 400a74a
- getting in with the guidelines : Commit ea1ad7c
- more spec refinements : Commit 64eb638
- more spec refinements : Commit 34eadd4
- making source ref in spec file a URL : Commit aefdd9b
- bringing spec file inline with Fedora standards : Commit 767e1d2
- updating macros in spec file for koji : Commit e4b46ea
- adding spec file : Commit e6738c0

* Tue Jun 03 2014 Jamie Duncan <> - 0.1-11
- ab0050b =updating changelog
- packaging cleanup : Commit d4a3428
- tweaking for rhn-support-tool : Commit 06a151f
- minor cleanup of an unused module and a repetitive line or 2 : Commit 7eb1726
- Update : Commit 97f047f
- cleaning up tarball paths. fixes #9 : Commit caa6536
- cleaning up tarball paths. fixes #9 : Commit e4e1cef
- updated : Commit f6c064a
- updated : Commit a070ebd
- fixing issue where checking compression type could error out because of capital letters where i thought it would always be capitalized : Commit cbf0e7d
- removing some cruft : Commit fdf82eb
- removed some old chmod bits that became un-needed when it became required to run as root : Commit cb15d97
- removed xsos option that never existed in the first place. we should just use xsos for that. : Commit ed5992e
- removed xsos option that never existed in the first place. we should just use xsos for that. : Commit 5c2dc32
- removed xsos option that never existed in the first place. we should just use xsos for that. : Commit 1df7e0d
- Merge branch 'master' of : Commit e462b7b
- minor cleanup : Commit 5fc8ce2
- Updating with better File Creation explanations : Commit bac8368
- Updating with better usage examples : Commit 6589b4c
- Updating : Commit 7dce4a9
- disallowing octects > 3 digits - to help cut down on false positives : Commit 319591f
- no longer matches IP addys starting with a 0 : Commit 0223d82

If you specify and ‘end tag’, then you won’t see the untagged commits in HEAD.