Jenkinsfile -- To collocate or not to collocate

To collocate or not to collocate Jenkinsfile

Problem

While building Pipeline-As-Code recently for one of our projects, we were faced with a conundrum; whether to co-locate our Jenkinsfiles with application code, or not. Or, does it even matter?

Default Solution

Our default opinion was to co-locate Jenkinsfile with application code, as that's the whole point - from the same code base we build and deploy code, such as below:

This idea had some advantages. With just a default checkout, Jenkins will be able to find code as well as pipeline to build and deploy it. We use Bitbucket for our development, so this approach comes with the added advantage that we could use multibranch pipelines without any additional effort.

Challenges

However, pretty soon after we started doing this, we ran into some challenge. While DevOps Engineer was modifying Jenkinsfile (remember we're the first ones to build it), and the application developers were simultaneously modifying code base, it resulted into multiple deployments, aka server restarts, while the developers were checking if their code was working in development. At times, this also resulted in broken builds, while DevOps Engineer was trying to fix the pipeline, such as, by adding Sonar scan. We knew, as first people to start using Jenkins Pipeline in enterprise there would be challenges and we chose to live with these challenges.

The application development continued rapidly, and then stabilized, things looked good, deployments were happening to dev and test as expected. However, we felt we were not doing the right thing. But why? We couldn't really put it in words. Until, we wanted to deploy to Acceptance environment, which we thought would be un-eventful. Except that, whenever we modify our pipeline, such as, to build deployment stage for ACPT, we were modifying the code base. And that's when we confirmed our problem, we were violating principle of keeping code and configuration separate, ref https://12factor.net/config. This meant that whenever we have changes to our pipeline, we would have to build the code again, not what we wanted. The code smell was obvious.

Final Approach

By now, we had realized that Jenkinsfile should not really be co-located, but we still wanted developers to be able to build code, run various tests on it, check code quality, and potentially deploy to a dev-like environment themselves. It was a choice between giving more powers to developers versus following sane conventions and keeping production deployments in the hands of people more experienced with doing that.

We eventually decided to have two kinds of Jenkinsfiles:

A usual Jenkinsfile, called just that, that does a build and runs tests on it (and potentially deploys to dev in a future state), used on feature branches

This was configured on Jenkins to run multibranch as well, ensuring that we are able to run those tests for each feature branch (which is created per story),

This sends emails to developers and culprits upon failure

Developers have full control over it and they can change it as needed, eg when our developer was working on a story to fix code Qualityissues, she was running Sonar and Nexus IQ Scans on this, which we generally don't run on feature branches.

Another set of Jenkinsfile, that is kept separate from code, in a different repository, and is used to build AND deploy code, from master This is really our deployment pipeline, that builds, deploys, and performs the whole nine yards of activities needed for taking code to production

This ensures that our pipeline, which is a configuration, remains separate from our code, and can be built and modified, without impacting code base

This sees more changes, especially now, where we are doing this for first time, although it will eventually stabilize too This is a little more controlled - and modified usually by DevOps Engineer only. However, developers have permissions to modify it

Failures to this pipeline should trigger emails to entire team

We did consider having a single Jenkinsfile that builds off of master and feature branches, with different workflows for feature vs master branch. However, we chose not to go this route, given our inexperience with Jenksfile, this would probably make our Jenkinsfile more complex than what we want. We want our developers to be able to understand and modify Jenkinsfile, but we dont want to burden them with too much information, that they usually don't need to dig in.

Looking forward

I believe eventually, we will move to a single Jenkinsfile, which is kept separate than code-base and has different workflows for master, feature branches and release branches. This may happen after we, including developers and DevOps engineers, become more proficient with Jenkinsfile usage.

We don't have any workflows for Pull Requests and neither are we using shared libraries at the moment, but both of these are on our bucket list. We don't think either of them would impact where we keep our Jenkinsfiles.

DEVOPS: Automatically updating job parameter in Jenkins

This post would ideally follow up after a discussion on configuring Jenkins jobs with Sprint number as a parameter to build jobs. At the start of every sprint, that Sprint number needs to be updated to new sprint. While I'll save the philosophy on that for a later discussion, here's whats here

Problem Statement:
A Jenkins job parameter needs to be updated every 'x' days. This needs to be done for 'n' jobs. A default way is to do it manually, an obviously error-prone way of doing it, where we run the risk of making a wrong update, as well as miss one of the 'n' jobs. In the following config XML of a Jenkins job, the Sprint number needs to be incremented as part of this job

<hudson.model.StringParameterDefinition>
    <name>SPRINT</name>
    <description></description>
    <defaultValue>10</defaultValue>
</hudson.model.StringParameterDefinition>

Resolution:
In true spirit of automation, this task can be automated by creating a Jenkins job that looks at these jobs and updates each of them. Here is a sample shell script (hey, Jenkins is running on Unix, right ;), and can be easily customized for powershell script.

#!/bin/bash
# This job uses Jenkins REST API to fetch job config, invokes python to increment SPRINT
#   and again uses Jenkins REST API to publish updated job config
# It defines an array of jobs that needs to be updated
# It defines a python script that will update job configurations
# It defines a shell function that will compile python function and pass jenkins config, and execute it
# The updated config is POST'ed back to Jenkins


JOB_NAMES=(first-build-job second-build-job third-build-job)
# Step 1: assign python code to increment Sprint value to a shell variable
_increment_sprint_script=$(cat <<'EOF'
import sys, xml.etree.ElementTree as ET
doc = ET.fromstring(sys.stdin.read())
for node in doc.findall('.//hudson.model.StringParameterDefinition'):
    name_el = node.find('./name')
    if name_el is not None and name_el.text == 'SPRINT':
        default_el = node.find('./defaultValue')
        if default_el is None: continue
        default_el.text = str(int(default_el.text) + 1)
print ET.tostring(doc)
EOF
)


# Step 2: define a function that calls the interpreter with that code
increment_sprint() { python -c "$_increment_sprint_script" "$@"; }


# Step 3: Iterate over all jobs and update them
for JOB_NAME in "${JOB_NAMES[@]}"
do
  echo "Updating $JOB_NAME ..."
  updated_config=$(curl "$JENKINS_URL/job/$JOB_NAME/config.xml" | increment_sprint)
  curl -v -X POST --data-binary "$updated_config" -H 'Content-Type: application/xml' -u "$USERID:$TOKEN" "$JENKINS_URL/job/$JOB_NAME/config.xml";
done

The finer details:
1. Notice the quotations in curl commands - pretty important, miss these, and the Jenkins configuration of x-build-job will turn into a single line xml, blotching its script.
2. Its easy to get the configuration wrong by a simpler script that just looks for value of "defaultValue" of current Sprint. While this may usually work, it can fail if there are multiple parameters with same defaultValue as current Sprint