Problem
While building Pipeline-As-Code
recently for one of our projects, we were faced with a conundrum; whether to
co-locate our Jenkinsfiles with application code, or not. Or, does it even
matter?
Default Solution
Our default opinion was to
co-locate Jenkinsfile with application code, as that's the whole point - from
the same code base we build and deploy code, such as below:
This idea had some advantages.
With just a default checkout, Jenkins will be able to find code as well as
pipeline to build and deploy it. We use Bitbucket for our development, so this
approach comes with the added advantage that we could use multibranch pipelines
without any additional effort.
Challenges
However, pretty soon after we
started doing this, we ran into some challenge. While DevOps Engineer was
modifying Jenkinsfile (remember we're the first ones to build it), and the
application developers were simultaneously modifying code base, it resulted
into multiple deployments, aka server restarts, while the developers were
checking if their code was working in development. At times, this also resulted
in broken builds, while DevOps Engineer was trying to fix the pipeline, such as, by adding Sonar scan. We knew, as first
people to start using Jenkins Pipeline in enterprise there would be challenges
and we chose to live with these challenges.
The application development
continued rapidly, and then stabilized, things looked good, deployments were
happening to dev and test as expected. However, we felt we were not doing the
right thing. But why? We couldn't really put it in words. Until, we wanted to
deploy to Acceptance environment, which we thought would be un-eventful. Except
that, whenever we modify our pipeline, such as, to build deployment stage for
ACPT, we were modifying the code base. And that's when we confirmed our
problem, we were violating principle of keeping code and configuration
separate, ref https://12factor.net/config.
This meant that whenever we have changes to our pipeline, we would have to
build the code again, not what we wanted. The code smell was obvious.
Final Approach
By now, we had
realized that Jenkinsfile should not really be co-located, but we still wanted
developers to be able to build code, run various tests on it, check code
quality, and potentially deploy to a dev-like environment themselves. It was a
choice between giving more powers to developers
versus following sane conventions and keeping production deployments in the
hands of people more experienced with doing
that.
We eventually decided to have two kinds of Jenkinsfiles:
A usual Jenkinsfile, called just that,
that does a build and runs tests on it (and potentially deploys to dev in a
future state), used on feature branchesThis was configured on Jenkins to run
multibranch as well, ensuring that we are able to run those tests for each
feature branch (which is created per story),This sends emails to developers and culprits upon
failure
Developers have full control over it and
they can change it as needed, eg when our developer was working on a story to
fix code Qualityissues, she was running Sonar and Nexus IQ Scans on this, which
we generally don't run on feature branches.
Another set
of Jenkinsfile, that is kept separate from code, in a different repository, and
is used to build AND deploy code, from master
This is really our deployment pipeline, that builds, deploys, and performs
the whole nine yards of activities needed for taking code to production
This ensures that our pipeline, which
is a configuration, remains separate from our code, and can be built and
modified, without impacting code base
This sees more changes, especially
now, where we are doing this for first time, although it will eventually
stabilize too This is a little more controlled - and modified usually by DevOps
Engineer only. However, developers have permissions to modify itFailures to this pipeline should
trigger emails to entire team
We did consider having a single
Jenkinsfile that builds off of master and feature branches, with different
workflows for feature vs master branch. However, we chose not to go this route,
given our inexperience with Jenksfile, this would probably make our Jenkinsfile
more complex than what we want. We want our developers to be able to understand
and modify Jenkinsfile, but we dont want to burden them with too much
information, that they usually don't need to dig in.
Looking forward
I believe eventually, we will move
to a single Jenkinsfile, which is kept separate than code-base and has
different workflows for master, feature branches and release branches. This may
happen after we, including developers and DevOps engineers, become more
proficient with Jenkinsfile usage.
We don't have any workflows for
Pull Requests and neither are we using shared libraries at the moment, but both
of these are on our bucket list. We don't think either of them would impact
where we keep our Jenkinsfiles.
