Friday, January 4, 2019

Managing Jar Hell in Tomcat 8



Classloading in Tomcat 8

Problem

We recently began development on a new microservice, that connects to an existing Sybase database and is deployed on tcServer 4.0.1, which has bundled Tomcat 8. For reference, we had a similar microservice that connects to Oracle database, and a legacy monolith that connects to Sybase database, but deployed on JBoss container. We were not doing anything fundamentally new, and we expected this development to be quite straightforward.

However, when we deployed this application, we started running into weird issues with Sybase jar (JConn4), a Cybe-Ark provider jar, that masks connection to database, using its own driver, to fetch connection details from vault.
We spent quite some time trying to analyze this with various teams to figure out what is going wrong. We also have a third party jar integrated as a handler through logging.properties (in tcserver/conf) that send alerts when it finds errors in logs, and that just complicated things more. 
I guess it’s that lucky time in my career, where I finally run into Jar Hell!

Analysis

Our default setup consists of loading some jars from a given file path, instead of using them from within the war / tcServer lib, as these are expected to be consistent across multiple applications we deploy on tcServer.
We tried experimenting with modifying where these jars are declared, versus where they are kept (external file / tcServer/lib / inside war) and kind of got sense that the issues seem to be due classes not being loaded when they are getting invoked.
This led us to analyze and understand classloading in Tomcat 8. Here are details on how Tomcat 8 loads classes and what tools we could use to debug.

Classloaders in Tomcat 8

When Tomcat is started, it creates a set of class loaders that are organized into the following parent-child relationships, where the parent class loader is above the child class loader. The default classloaders are:



  • Bootstrap : This class loader contains the basic runtime classes provided by the Java Virtual Machine, plus any classes from JAR files present in the System Extensions directory ($JAVA_HOME/jre/lib/ext). Generally speaking, you wouldn’t setup anything here.
  • System : This class loader is normally initialized from the contents of the CLASSPATH environment variable. In our context, this one is actually important, as this loads up all jars by putting them in classpath. Also, this classloader is responsible for loading Tomcat's logging implementation, implying this loads up logging.properties, which was also important for us.
  • Common : This classloader by default loads classes / resources/ jars from tcServer/lib directory. Per Tomcat documentation, normally, application classes should NOT be placed here. However, our enterprise bundle adds tcapp/lib to this. Jars loaded by this can be configured through common.loader property in tcserver/conf/catalina.properties
  • WebappX : This loads classes from WEB-INF/classes and WEB-INF/lib

Classloading Hierarchy :

The default loading hierarchy for loading of these classes are :
  1. Bootstrap
  2. Webapp
  3. System
  4. Common
That does seem a little weird to me, but I guess, there must be a good reason on why that is the default loading mechanism. Also, the order in which jars are loaded by a given classloader is not defined (see bug in references below). Coming from a Spring background, which holds bean initialization / dependency resolution as late as possible, to load other beans, this was a surprising fact. (And yes, I am aware that loading /wiring beans, is an entirely different thing than loading the classes themselves).

Customization 1: Specifying Loader Delegate as True

This hierarchy can be configured by specifying, in context.xml,then the order becomes:
  1. Bootstrap
  2. System
  3. Common
  4. Webapp
We did some funny combinations of the three jars, with combinations on where they are placed, with different loader delegate conditions and almost hacked ourselves to death, trying to figure out what is going on with classloading. Sometimes, the classes would load up, but on other times, with what seemed reasonable approach, they would not. We tried loading all three under same classloader, but they would fail, which seemed weird, but remember, the order in which jars are loaded by a given classloader is not defined 

Customization 2 : Explicitly loading classes before/after in Webapp classloader

Tomcat believes that depending upon a class to be loaded before should be done by putting it in a way that it is loaded by a different classloader which is loaded first, and ordering of jars within a given classloader is a smell. However, if we necessarily need this, this can be achieved by modifying the context.xml, to include Pre/Post Resources. So, in order to load files from a given path first, we could use the following block to order classloading:
                   base="/Users/theuser/mypictures" webAppMount="/pictures" />

Note here, that we don't need a custom class for this. There are couple of classes available from Tomcat that can be used to look at directory/ file /jar (DirResourceSet/ FileResourceSet / JarResourceSet). The resources such loaded can be made available to one or all contexts, using the webAppMount element.

Debug Tools

To our rescue, we added, -verbose:class to JAVA_OPTS in ApplicationEnv. With this we could see the actual order in which classes were getting loaded and that really helped with understanding what is going on. Although, the logs were interleaved between System and Webapp classloaders, to some extent, overall it was a big help.
The second thing we did (although it makes logs very very confusing) was plain and simple, to enable logging for Tomcat, by adding following to logging.properties in tcserver/conf:



With the help of these, we were eventually able to resolve our classloading issues. We also identified there is no one single way to achieve similar results. We could have used PreResources using default classloading hierarchy, but we ended up using Delegate=true, and customizing the load order.

References


Monday, August 6, 2018

Introduction to Git

Published another talk I gave at https://www.slideshare.net/Ram0603/introduction-to-git-108820284

Introduction to DevOps

Published an older presentation I provided on

https://www.slideshare.net/Ram0603/devops-a-peek-into-high-performing-organizations-108817405

Thursday, February 16, 2017

Install pip on AWS EC2

While stock EC2 AMI comes with python installed, pip version is pretty old. Updating pip results in pip going out of path - it gets installed on a different location.

Here is sample script to update pip and still keep pip running:

#!/bin/bash
yum update -y
pip install -U pip
export PATH=$PATH:/usr/local/bin

Please make sure that it is run as root user only, otherwise it will error out with permission error

Wednesday, February 1, 2017

SVN: Error running context: An error occurred during authentication

Problem Statement: My corporate laptop suddenly started giving "Error: Error running context: An error occurred during authentication" on all Tortoise SVN requests. I switched to subclipse, and I got same. I switched to command line, and I got same!

Root Cause: My corporate password had changed and somehow all SVN clients were using the old password.

Fix:
I see multiple fixes mentioned online, eg http://stackoverflow.com/questions/914895/how-to-change-credentials-for-svn-repository-in-eclipse and http://www.wandisco.com/svnforum/forum/opensource-subversion-forums/general-setup-and-troubleshooting/tortoisesvn-community-support/12445-basic-authentication-not-working and others, but 

Tuesday, May 31, 2016

DEVOPS: Automatically updating job parameter in Jenkins

This post would ideally follow up after a discussion on configuring Jenkins jobs with Sprint number as a parameter to build jobs. At the start of every sprint, that Sprint number needs to be updated to new sprint. While I'll save the philosophy on that for a later discussion, here's whats here

Problem Statement:
A Jenkins job parameter needs to be updated every 'x' days. This needs to be done for 'n' jobs. A default way is to do it manually, an obviously error-prone way of doing it, where we run the risk of making a wrong update, as well as miss one of the 'n' jobs. In the following config XML of a Jenkins job, the Sprint number needs to be incremented as part of this job

<hudson.model.StringParameterDefinition>
    <name>SPRINT</name>
    <description></description>
    <defaultValue>10</defaultValue>
</hudson.model.StringParameterDefinition>


Resolution:
In true spirit of automation, this task can be automated by creating a Jenkins job that looks at these jobs and updates each of them. Here is a sample shell script (hey, Jenkins is running on Unix, right ;), and can be easily customized for powershell script.

#!/bin/bash
# This job uses Jenkins REST API to fetch job config, invokes python to increment SPRINT
#   and again uses Jenkins REST API to publish updated job config
# It defines an array of jobs that needs to be updated
# It defines a python script that will update job configurations
# It defines a shell function that will compile python function and pass jenkins config, and execute it
# The updated config is POST'ed back to Jenkins


JOB_NAMES=(first-build-job second-build-job third-build-job)
# Step 1: assign python code to increment Sprint value to a shell variable
_increment_sprint_script=$(cat <<'EOF'
import sys, xml.etree.ElementTree as ET
doc = ET.fromstring(sys.stdin.read())
for node in doc.findall('.//hudson.model.StringParameterDefinition'):
    name_el = node.find('./name')
    if name_el is not None and name_el.text == 'SPRINT':
        default_el = node.find('./defaultValue')
        if default_el is None: continue
        default_el.text = str(int(default_el.text) + 1)
print ET.tostring(doc)
EOF
)


# Step 2: define a function that calls the interpreter with that code
increment_sprint() { python -c "$_increment_sprint_script" "$@"; }


# Step 3: Iterate over all jobs and update them
for JOB_NAME in "${JOB_NAMES[@]}"
do
  echo "Updating $JOB_NAME ..."
  updated_config=$(curl "$JENKINS_URL/job/$JOB_NAME/config.xml" | increment_sprint)
  curl -v -X POST --data-binary "$updated_config" -H 'Content-Type: application/xml' -u "$USERID:$TOKEN" "$JENKINS_URL/job/$JOB_NAME/config.xml";
done

The finer details:
1. Notice the quotations in curl commands - pretty important, miss these, and the Jenkins configuration of x-build-job will turn into a single line xml, blotching its script.
2. Its easy to get the configuration wrong by a simpler script that just looks for value of "defaultValue" of current Sprint. While this may usually work, it can fail if there are multiple parameters with same defaultValue as current Sprint



Tuesday, August 4, 2015

Externalizing system properties or truststore

The need, aka Why: 

Set system properties based on environment. While there are many ways to externalize properties in java world (JNDI), spring profile, or as Stack Overflow search reveals, JBoss System Properties; there aren't many ways to externalize system properties - other than setting them up in .profile or similar file. However, if you need a consistent and more clear way of setting them up, there aren't as many options (or I am not aware of).

A solution, aka What:

A clear, clean and intuitive ( and I'd say smart) solution is to let Spring handle it for you. Yep, it can be done, and pretty easily too!

The following code sets the system property javax.net.ssl.truststore to specify location of truststore, defined in a standard property file that is easily configured per-environment. It is searchable in editor and consistent across operating systems, so is kinda neat.


Of course, if you are setting the truststore, you might need to set the trust store password as well - which would not be plain text property, and hence not such a simple task, but thats for another day ;)

How:

Well, the code is pretty self-explanatory, Spring uses Map / property bean titled "systemProperties" to store the system properties. The above code uses Spring's MethodInvokingFactoryBean to modify this bean to add additional properties to System properties. At least that's what I believe is happening.

Challenges:

The above solution worked for me when the truststore was in classpath, otherwise on Windows 8, it gave me a weird InvalidAlgorithmParameterException. I looked at SO but that doesnt help me, though it looks like a generally correct response. Well, again I'll leave that for another day!