Jenkins Ops

In you account settings, found at https://$JENKINS_URL/user/$USERNAME/configure you should set your ssh public key.

If you are not familiar with setting up ssh keys, or you would like to generate new one, but keep forgetting how did you do that last time (as happens to me roughly once a month), you can consult a great guide on Github that explains the ssh generation step-by-step.

The preferred option for connecting for CLI is JNLP. In theory, if this isn't enabled, the CLI binary should fall back to http. In practice, for certain commands, such as the groovy command we wanted to be using, you'd receive error about JNLP not being available.

To enable it, go to Manage Jenkins > Configure Global Security and choose "Fixed" or "Random" under TCP port for JNLP agents.

I would advise to set a specific port, so that you can enable the port in your firewall.

Before you can use groovy scripts you need to enable remoting over CLI by checking the "Enable CLI over Remoting" in Manage Jenkins > Configure Global Security. As I mentioned, Jenkins will advise you against this, unfortunately, to take advantage of the groovy scripting through Jenkins CLI, this really needs to be enabled.

You can find the download link directly in your Jenkins instance, through Manage Jenkins > Jenkins CLI, you should even be able to download it directly from https://$JENKINS_URL/jnlpJars/jenkins-cli.jar. To execute it, you will need java runtime present.

To test out the setup, lets use the example you'd find if you wanted to try groovy-scripting from the web-console. The example there could be translated to a file plugin_list.groovy like this:

You should now be able to execute the file by running:

java -jar  jenkins-cli.jar -remoting -s https://$JENKINS_URL groovy plugin_list.groovy

and see something along the lines:

[Plugin:ssh-credentials, Plugin:workflow-job, Plugin:pipeline-github-lib, Plugin:workflow-basic-steps, Plugin:ws-cleanup, Plugin:pipeline-rest-api, Plugin:pipeline-model-api, Plugin:handlebars, Plugin:ace-editor]

Just to make this upfront, I have tested this on Jenkins 2.89.3, with Credentials Binding plugin 1.14. I think I should make myself a CI/CD to test these snippets one day, because nobody likes a an out of date how-to, especially if that person will be me, in a year, trying to quickly solve a Jenkins configuration problem.

I assume that you have the account setup for cli usage, with alias for jenkins-cli in your shell. Second, there might be a better way. For example, if you are using Jenkins just to run your ansible-scripts, you could use ansible-valut, if you are relying on Kubernetes, you could configure secrets through kubectl.

In my experience, storing secrets in Jenkins is completely acceptable. The secrets are encrypted on rest. When you log them to console, Jenkins will transparently display ** instead of them. There is support for various types of them:

• Plain text
• Username/Password
• SSH private key
• AWS credentials

and more. Of course you could configure these through the management UI in the credentials section, but we didn't want to do that. We want to be able to have our credentials configured from code.

Second disclaimer, you might want to investigate if you want to have your credential scopes in Jenkins. We wave all of our credentials in the global scope, available of all jobs to use. We don't mind, because with our setup we would often have several distinct Jenkins instances, housing various secrets. For example Jenkins that is publicly accessible only has AWS credentials to push build artifacts to S3 bucket, but other instance, behind VPN, might have AWS credentials for EC2 as well, and we might run pipelines that spawn/update our staging infrastructure from there.

For example with this groovy file, I can add credentials containing username and password:

There are only two lines really specific to plain text credentials. You can replace i.e.:

secret = new UsernamePasswordCredentialsImpl( //...

with other credential definitions. For example for plain-text credential (that has no username):

import org.jenkinsci.plugins.plaincredentials.impl.*
import hudson.util.Secret
def secret = new StringCredentialsImpl(CredentialsScope.GLOBAL,credId,description, Secret.fromString(password))

For ssh credentials:

import com.cloudbees.jenkins.plugins.sshcredentials.impl.*
import hudson.plugins.sshslaves.*;
def username = "HARDCODED"
def privateKeyStr = "-----BEGIN RSA PRIVATE KEY-----\nMII..."
def privateKeySource = new BasicSSHUserPrivateKey.DirectEntryPrivateKeySource(privateKeyStr)
def secret = new BasicSSHUserPrivateKey(
	CredentialsScope.GLOBAL,
	credId,
	username,
	privateKeySource,
	"",
	description
)

In case of ssh credentials, we usually hardcode the private key string in the groovy-file itself, because we encountered some problems where escaping mechanisms of shell seem to have collided with escaping mechanisms of groovy script transported over http. This would get even trickier if you'd needed to create a credential that is binary.

For AWS credentials, if we supply the access key as username and secret key as password:

import com.cloudbees.jenkins.plugins.awscredentials.*
def secret = new AWSCredentialsImpl(
	CredentialsScope.GLOBAL,
	credId,
	username,
	password,
	description
)

To create the credentials (in this case, I am assuming username password), just run the jenkins-cli groovy command. I have alias jenkins-cli=java -jar /opt/jenkins-cli.jar, and ssh key configured on my Jenkins master which means that I would run:

When I use these in my scripts, pipelines and Jenkinsfiles, I have to admit, that I usually generate the actual code with snippet-generator. This makes my work much easier especially if I need to work with credential I haven't worked with, or with several credentials at once. You should be able to find the generator on https://$JENKINSURL/pipeline-syntax/

If you'd then tried out a pipeline script such as this, you should be able to use the password injected in the environment variable.

node {
  withCredentials([string(credentialsId: 'example_secret, variable: 'TOKEN')]) {
    print "The secret is ${env.TOKEN}"
  }
}

Beware, you shouldn't see the password itself if you print it, and this should give you some level of protection, i.e if you'd want to have your job-logs public (and we did want that for our opensource projects).

If you liked this, and want to ask anything, (or didn't and want to tell me how to do something better), feel free to write me an email to adam at asaleh.net!

This was probably the biggest advantage compared to our previous setups. All of the configuration is now in code. In our previous Jenkins instance, the configuration was mostly ad-hoc, manual, often depending on specific configuration our slaves. Changes to configuration was done with direct ssh to the slave through a shared account. Untangling the configuration was fun.

Now we use

• credentials-binding plugin to share our centrally stored secrets across our jobs
• config-file-provider to share the non-secret configuration
• plain-credentials and ssh-credentials to store the secrets centrally on the master Jenkins

The interesting thing here is automating the update of the credentials. Currently, we rely on combination of groovy scripts and sh scripts running the jenkins-cli to run the groovy on Jenkins and so far it worked very well.

Figuring out the groovy script can be tricky at times, but the upside is undeniable, and currently if we wanted to, we could configure a new instance of Jenkins under 30 minutes, including provisioning.

The ability to use Jenkinsfiles and groovy pipelines was one of the reasons we wanted to create new Jenkins infrastructure. The old Jenkins wouldn't be able to run these and when we were deciding whether to try to update the old or build a new one, we have decided to build a new one.

And pipelines did deliver. Groovy is much more flexible language than bash that we used previously for most of our automation. We managed to automate most of our release process and large parts of our deployment process. We don't think we would have managed that without groovy, as during the release process we need to process, build and tag over 60 repositories.

Another nice thing that emerged over the year is better support for groovy itself. Running pipelines should be resilient to things like shutdowns and restarts of Jenkins. Unfortunately this meant that for long time, you couldn't use standard groovy methods like collect or each, because the resulting code wouldn't be serializable, and you would need to use workaround with @NonCPS annotation.

Fortunately, since May 2017, most of these work, making for a much nicer environment to program in!

Even before we rebuilt our CI infrastructure, we have already been building most of our projects on Jenkins. To make the configuration more robust, we started experimenting with tools such as Jenkins Job Builder, because maintaining 60+ repositories in Github by hand was becoming unwieldy. Unfortunately Jenkins Job Builder scripts are quite complicated and cumbersome to use.

After we switched to new Jenkins, we could take advantage of the GitHub Organization Plugin, It automatically scans through all of our repositories in GitHub and creates a job for every repository that has a Jenkinsfile defining the CI pipeline for that repo. This lets us have our own infrastructure to provide something like Travis or CircleCI to our developers, while having much better control over the test environment.

I have mentioned that we have previously used Jenkins Job Builder for configuring build jobs for each of our repositories and that it was cumbersome. I could identify several reasons why it wasn't working well:

• We needed all of our developers to be able to understand most of the job definitions
• We needed to utilize templating, to map our job definitions to 60+ repositories
• Definitions themselves were mostly bash inside of templated yaml configuration, which lead to problems with escaping special characters
• All the jobs were updated at once

After we moved GitHub Organization Plugin:

• Developers can be interested just in the Pipelines that are useful for themselves
• Pipeline libraries are much better way for code-sharing than yaml templating
• Groovy code is more readable than bash inside of yaml
• If you need to update a job, you just change the Jenkinsfile in a single repo

So why are we still using Jenkins Job Builder? It turns out, that you can use Jenkinsfiles with Jenkins Job Builder, and this solves most of the problems we encountered with the weird combination of templated yaml and bash. We still needed a way to define some of our long-running pipeline jobs, that weren't tied to any repository in particular, and for this, because most of the heavy lifting is done by the groovy code in separate file, Job Builder shines.

Job definition is

• a simple, non-templated yaml file
• defines name
• defines triggers
• points to the relevant Jenkinsfile

This means we can have all of the pipeline definitions in a single repository, and there is relatively few of them.

The last thing I should probably mention, is that I have learned to enjoy working with Jenkins. Despite sometimes being arcane, fiddly and hard to debug, it has over time become my favorite among the various CI systems, and where other people might build a static blog like this with travis, I run my personal Jenkins instance now!