Jenkinsfile -- To collocate or not to collocate

To collocate or not to collocate Jenkinsfile

Problem

While building Pipeline-As-Code recently for one of our projects, we were faced with a conundrum; whether to co-locate our Jenkinsfiles with application code, or not. Or, does it even matter?

Default Solution

Our default opinion was to co-locate Jenkinsfile with application code, as that's the whole point - from the same code base we build and deploy code, such as below:

This idea had some advantages. With just a default checkout, Jenkins will be able to find code as well as pipeline to build and deploy it. We use Bitbucket for our development, so this approach comes with the added advantage that we could use multibranch pipelines without any additional effort.

Challenges

However, pretty soon after we started doing this, we ran into some challenge. While DevOps Engineer was modifying Jenkinsfile (remember we're the first ones to build it), and the application developers were simultaneously modifying code base, it resulted into multiple deployments, aka server restarts, while the developers were checking if their code was working in development. At times, this also resulted in broken builds, while DevOps Engineer was trying to fix the pipeline, such as, by adding Sonar scan. We knew, as first people to start using Jenkins Pipeline in enterprise there would be challenges and we chose to live with these challenges.

The application development continued rapidly, and then stabilized, things looked good, deployments were happening to dev and test as expected. However, we felt we were not doing the right thing. But why? We couldn't really put it in words. Until, we wanted to deploy to Acceptance environment, which we thought would be un-eventful. Except that, whenever we modify our pipeline, such as, to build deployment stage for ACPT, we were modifying the code base. And that's when we confirmed our problem, we were violating principle of keeping code and configuration separate, ref https://12factor.net/config. This meant that whenever we have changes to our pipeline, we would have to build the code again, not what we wanted. The code smell was obvious.

Final Approach

By now, we had realized that Jenkinsfile should not really be co-located, but we still wanted developers to be able to build code, run various tests on it, check code quality, and potentially deploy to a dev-like environment themselves. It was a choice between giving more powers to developers versus following sane conventions and keeping production deployments in the hands of people more experienced with doing that.

We eventually decided to have two kinds of Jenkinsfiles:

A usual Jenkinsfile, called just that, that does a build and runs tests on it (and potentially deploys to dev in a future state), used on feature branches

This was configured on Jenkins to run multibranch as well, ensuring that we are able to run those tests for each feature branch (which is created per story),

This sends emails to developers and culprits upon failure

Developers have full control over it and they can change it as needed, eg when our developer was working on a story to fix code Qualityissues, she was running Sonar and Nexus IQ Scans on this, which we generally don't run on feature branches.

Another set of Jenkinsfile, that is kept separate from code, in a different repository, and is used to build AND deploy code, from master This is really our deployment pipeline, that builds, deploys, and performs the whole nine yards of activities needed for taking code to production

This ensures that our pipeline, which is a configuration, remains separate from our code, and can be built and modified, without impacting code base

This sees more changes, especially now, where we are doing this for first time, although it will eventually stabilize too This is a little more controlled - and modified usually by DevOps Engineer only. However, developers have permissions to modify it

Failures to this pipeline should trigger emails to entire team

We did consider having a single Jenkinsfile that builds off of master and feature branches, with different workflows for feature vs master branch. However, we chose not to go this route, given our inexperience with Jenksfile, this would probably make our Jenkinsfile more complex than what we want. We want our developers to be able to understand and modify Jenkinsfile, but we dont want to burden them with too much information, that they usually don't need to dig in.

Looking forward

I believe eventually, we will move to a single Jenkinsfile, which is kept separate than code-base and has different workflows for master, feature branches and release branches. This may happen after we, including developers and DevOps engineers, become more proficient with Jenkinsfile usage.

We don't have any workflows for Pull Requests and neither are we using shared libraries at the moment, but both of these are on our bucket list. We don't think either of them would impact where we keep our Jenkinsfiles.

Managing Jar Hell in Tomcat 8

Classloading in Tomcat 8

Problem

We recently began development on a new microservice, that connects to an existing Sybase database and is deployed on tcServer 4.0.1, which has bundled Tomcat 8. For reference, we had a similar microservice that connects to Oracle database, and a legacy monolith that connects to Sybase database, but deployed on JBoss container. We were not doing anything fundamentally new, and we expected this development to be quite straightforward.

However, when we deployed this application, we started running into weird issues with Sybase jar (JConn4), a Cybe-Ark provider jar, that masks connection to database, using its own driver, to fetch connection details from vault.
We spent quite some time trying to analyze this with various teams to figure out what is going wrong. We also have a third party jar integrated as a handler through logging.properties (in tcserver/conf) that send alerts when it finds errors in logs, and that just complicated things more. 
I guess it’s that lucky time in my career, where I finally run into Jar Hell!

Analysis

Our default setup consists of loading some jars from a given file path, instead of using them from within the war / tcServer lib, as these are expected to be consistent across multiple applications we deploy on tcServer.
We tried experimenting with modifying where these jars are declared, versus where they are kept (external file / tcServer/lib / inside war) and kind of got sense that the issues seem to be due classes not being loaded when they are getting invoked.
This led us to analyze and understand classloading in Tomcat 8. Here are details on how Tomcat 8 loads classes and what tools we could use to debug.

Classloaders in Tomcat 8

When Tomcat is started, it creates a set of class loaders that are organized into the following parent-child relationships, where the parent class loader is above the child class loader. The default classloaders are:

• Bootstrap : This class loader contains the basic runtime classes provided by the Java Virtual Machine, plus any classes from JAR files present in the System Extensions directory ($JAVA_HOME/jre/lib/ext). Generally speaking, you wouldn’t setup anything here.
• System : This class loader is normally initialized from the contents of the CLASSPATH environment variable. In our context, this one is actually important, as this loads up all jars by putting them in classpath. Also, this classloader is responsible for loading Tomcat's logging implementation, implying this loads up logging.properties, which was also important for us.
• Common : This classloader by default loads classes / resources/ jars from tcServer/lib directory. Per Tomcat documentation, normally, application classes should NOT be placed here. However, our enterprise bundle adds tcapp/lib to this. Jars loaded by this can be configured through common.loader property in tcserver/conf/catalina.properties
• WebappX : This loads classes from WEB-INF/classes and WEB-INF/lib

Classloading Hierarchy :

The default loading hierarchy for loading of these classes are :

1. Bootstrap
2. Webapp
3. System
4. Common

That does seem a little weird to me, but I guess, there must be a good reason on why that is the default loading mechanism. Also, the order in which jars are loaded by a given classloader is not defined (see bug in references below). Coming from a Spring background, which holds bean initialization / dependency resolution as late as possible, to load other beans, this was a surprising fact. (And yes, I am aware that loading /wiring beans, is an entirely different thing than loading the classes themselves).

Customization 1: Specifying Loader Delegate as True

This hierarchy can be configured by specifying, in context.xml,then the order becomes:

1. Bootstrap
2. System
3. Common
4. Webapp

We did some funny combinations of the three jars, with combinations on where they are placed, with different loader delegate conditions and almost hacked ourselves to death, trying to figure out what is going on with classloading. Sometimes, the classes would load up, but on other times, with what seemed reasonable approach, they would not. We tried loading all three under same classloader, but they would fail, which seemed weird, but remember, the order in which jars are loaded by a given classloader is not defined 

Customization 2 : Explicitly loading classes before/after in Webapp classloader

Tomcat believes that depending upon a class to be loaded before should be done by putting it in a way that it is loaded by a different classloader which is loaded first, and ordering of jars within a given classloader is a smell. However, if we necessarily need this, this can be achieved by modifying the context.xml, to include Pre/Post Resources. So, in order to load files from a given path first, we could use the following block to order classloading:

                   base="/Users/theuser/mypictures" webAppMount="/pictures" />

Note here, that we don't need a custom class for this. There are couple of classes available from Tomcat that can be used to look at directory/ file /jar (DirResourceSet/ FileResourceSet / JarResourceSet). The resources such loaded can be made available to one or all contexts, using the webAppMount element.

Debug Tools

To our rescue, we added, -verbose:class to JAVA_OPTS in ApplicationEnv. With this we could see the actual order in which classes were getting loaded and that really helped with understanding what is going on. Although, the logs were interleaved between System and Webapp classloaders, to some extent, overall it was a big help.

The second thing we did (although it makes logs very very confusing) was plain and simple, to enable logging for Tomcat, by adding following to logging.properties in tcserver/conf:

With the help of these, we were eventually able to resolve our classloading issues. We also identified there is no one single way to achieve similar results. We could have used PreResources using default classloading hierarchy, but we ended up using Delegate=true, and customizing the load order.

References

• https://tomcat.apache.org/tomcat-8.0-doc/class-loader-howto.html
• https://issues.apache.org/bugzilla/show_bug.cgi?id=57129
• http://mikusa.blogspot.com/2014/07/tips-on-migrating-to-tomat-8-resources.html
• https://tomcat.apache.org/tomcat-8.0-doc/config/resources.html
• https://dzone.com/articles/what-is-jar-hell

Introduction to Git

Published another talk I gave at https://www.slideshare.net/Ram0603/introduction-to-git-108820284

Introduction to DevOps

Published an older presentation I provided on

https://www.slideshare.net/Ram0603/devops-a-peek-into-high-performing-organizations-108817405

Install pip on AWS EC2

While stock EC2 AMI comes with python installed, pip version is pretty old. Updating pip results in pip going out of path - it gets installed on a different location.

Here is sample script to update pip and still keep pip running:

#!/bin/bash
yum update -y
pip install -U pip
export PATH=$PATH:/usr/local/bin

Please make sure that it is run as root user only, otherwise it will error out with permission error

SVN: Error running context: An error occurred during authentication

Problem Statement: My corporate laptop suddenly started giving "Error: Error running context: An error occurred during authentication" on all Tortoise SVN requests. I switched to subclipse, and I got same. I switched to command line, and I got same!

Root Cause: My corporate password had changed and somehow all SVN clients were using the old password.

Fix:
I see multiple fixes mentioned online, eg http://stackoverflow.com/questions/914895/how-to-change-credentials-for-svn-repository-in-eclipse and http://www.wandisco.com/svnforum/forum/opensource-subversion-forums/general-setup-and-troubleshooting/tortoisesvn-community-support/12445-basic-authentication-not-working and others, but 

DEVOPS: Automatically updating job parameter in Jenkins

This post would ideally follow up after a discussion on configuring Jenkins jobs with Sprint number as a parameter to build jobs. At the start of every sprint, that Sprint number needs to be updated to new sprint. While I'll save the philosophy on that for a later discussion, here's whats here

Problem Statement:
A Jenkins job parameter needs to be updated every 'x' days. This needs to be done for 'n' jobs. A default way is to do it manually, an obviously error-prone way of doing it, where we run the risk of making a wrong update, as well as miss one of the 'n' jobs. In the following config XML of a Jenkins job, the Sprint number needs to be incremented as part of this job

<hudson.model.StringParameterDefinition>
    <name>SPRINT</name>
    <description></description>
    <defaultValue>10</defaultValue>
</hudson.model.StringParameterDefinition>

Resolution:
In true spirit of automation, this task can be automated by creating a Jenkins job that looks at these jobs and updates each of them. Here is a sample shell script (hey, Jenkins is running on Unix, right ;), and can be easily customized for powershell script.

#!/bin/bash
# This job uses Jenkins REST API to fetch job config, invokes python to increment SPRINT
#   and again uses Jenkins REST API to publish updated job config
# It defines an array of jobs that needs to be updated
# It defines a python script that will update job configurations
# It defines a shell function that will compile python function and pass jenkins config, and execute it
# The updated config is POST'ed back to Jenkins


JOB_NAMES=(first-build-job second-build-job third-build-job)
# Step 1: assign python code to increment Sprint value to a shell variable
_increment_sprint_script=$(cat <<'EOF'
import sys, xml.etree.ElementTree as ET
doc = ET.fromstring(sys.stdin.read())
for node in doc.findall('.//hudson.model.StringParameterDefinition'):
    name_el = node.find('./name')
    if name_el is not None and name_el.text == 'SPRINT':
        default_el = node.find('./defaultValue')
        if default_el is None: continue
        default_el.text = str(int(default_el.text) + 1)
print ET.tostring(doc)
EOF
)


# Step 2: define a function that calls the interpreter with that code
increment_sprint() { python -c "$_increment_sprint_script" "$@"; }


# Step 3: Iterate over all jobs and update them
for JOB_NAME in "${JOB_NAMES[@]}"
do
  echo "Updating $JOB_NAME ..."
  updated_config=$(curl "$JENKINS_URL/job/$JOB_NAME/config.xml" | increment_sprint)
  curl -v -X POST --data-binary "$updated_config" -H 'Content-Type: application/xml' -u "$USERID:$TOKEN" "$JENKINS_URL/job/$JOB_NAME/config.xml";
done

The finer details:
1. Notice the quotations in curl commands - pretty important, miss these, and the Jenkins configuration of x-build-job will turn into a single line xml, blotching its script.
2. Its easy to get the configuration wrong by a simpler script that just looks for value of "defaultValue" of current Sprint. While this may usually work, it can fail if there are multiple parameters with same defaultValue as current Sprint