This was probably the biggest advantage compared to our previous setups. All of the configuration is now in code. In our previous Jenkins instance, the configuration was mostly ad-hoc, manual, often depending on specific configuration our slaves. Changes to configuration was done with direct ssh to the slave through a shared account. Untangling the configuration was fun.

Now we use

• credentials-binding plugin to share our centrally stored secrets across our jobs
• config-file-provider to share the non-secret configuration
• plain-credentials and ssh-credentials to store the secrets centrally on the master Jenkins

The interesting thing here is automating the update of the credentials. Currently, we rely on combination of groovy scripts and sh scripts running the jenkins-cli to run the groovy on Jenkins and so far it worked very well.

Figuring out the groovy script can be tricky at times, but the upside is undeniable, and currently if we wanted to, we could configure a new instance of Jenkins under 30 minutes, including provisioning.

The ability to use Jenkinsfiles and groovy pipelines was one of the reasons we wanted to create new Jenkins infrastructure. The old Jenkins wouldn't be able to run these and when we were deciding whether to try to update the old or build a new one, we have decided to build a new one.

And pipelines did deliver. Groovy is much more flexible language than bash that we used previously for most of our automation. We managed to automate most of our release process and large parts of our deployment process. We don't think we would have managed that without groovy, as during the release process we need to process, build and tag over 60 repositories.

Another nice thing that emerged over the year is better support for groovy itself. Running pipelines should be resilient to things like shutdowns and restarts of Jenkins. Unfortunately this meant that for long time, you couldn't use standard groovy methods like collect or each, because the resulting code wouldn't be serializable, and you would need to use workaround with @NonCPS annotation.

Fortunately, since May 2017, most of these work, making for a much nicer environment to program in!

Even before we rebuilt our CI infrastructure, we have already been building most of our projects on Jenkins. To make the configuration more robust, we started experimenting with tools such as Jenkins Job Builder, because maintaining 60+ repositories in Github by hand was becoming unwieldy. Unfortunately Jenkins Job Builder scripts are quite complicated and cumbersome to use.

After we switched to new Jenkins, we could take advantage of the GitHub Organization Plugin, It automatically scans through all of our repositories in GitHub and creates a job for every repository that has a Jenkinsfile defining the CI pipeline for that repo. This lets us have our own infrastructure to provide something like Travis or CircleCI to our developers, while having much better control over the test environment.

I have mentioned that we have previously used Jenkins Job Builder for configuring build jobs for each of our repositories and that it was cumbersome. I could identify several reasons why it wasn't working well:

• We needed all of our developers to be able to understand most of the job definitions
• We needed to utilize templating, to map our job definitions to 60+ repositories
• Definitions themselves were mostly bash inside of templated yaml configuration, which lead to problems with escaping special characters
• All the jobs were updated at once

After we moved GitHub Organization Plugin:

• Developers can be interested just in the Pipelines that are useful for themselves
• Pipeline libraries are much better way for code-sharing than yaml templating
• Groovy code is more readable than bash inside of yaml
• If you need to update a job, you just change the Jenkinsfile in a single repo

So why are we still using Jenkins Job Builder? It turns out, that you can use Jenkinsfiles with Jenkins Job Builder, and this solves most of the problems we encountered with the weird combination of templated yaml and bash. We still needed a way to define some of our long-running pipeline jobs, that weren't tied to any repository in particular, and for this, because most of the heavy lifting is done by the groovy code in separate file, Job Builder shines.

Job definition is

• a simple, non-templated yaml file
• defines name
• defines triggers
• points to the relevant Jenkinsfile

This means we can have all of the pipeline definitions in a single repository, and there is relatively few of them.

The last thing I should probably mention, is that I have learned to enjoy working with Jenkins. Despite sometimes being arcane, fiddly and hard to debug, it has over time become my favorite among the various CI systems, and where other people might build a static blog like this with travis, I run my personal Jenkins instance now!