Managing Jar Hell in Tomcat 8

Classloading in Tomcat 8

Problem

We recently began development on a new microservice, that connects to an existing Sybase database and is deployed on tcServer 4.0.1, which has bundled Tomcat 8. For reference, we had a similar microservice that connects to Oracle database, and a legacy monolith that connects to Sybase database, but deployed on JBoss container. We were not doing anything fundamentally new, and we expected this development to be quite straightforward.

However, when we deployed this application, we started running into weird issues with Sybase jar (JConn4), a Cybe-Ark provider jar, that masks connection to database, using its own driver, to fetch connection details from vault.
We spent quite some time trying to analyze this with various teams to figure out what is going wrong. We also have a third party jar integrated as a handler through logging.properties (in tcserver/conf) that send alerts when it finds errors in logs, and that just complicated things more. 
I guess it’s that lucky time in my career, where I finally run into Jar Hell!

Analysis

Our default setup consists of loading some jars from a given file path, instead of using them from within the war / tcServer lib, as these are expected to be consistent across multiple applications we deploy on tcServer.
We tried experimenting with modifying where these jars are declared, versus where they are kept (external file / tcServer/lib / inside war) and kind of got sense that the issues seem to be due classes not being loaded when they are getting invoked.
This led us to analyze and understand classloading in Tomcat 8. Here are details on how Tomcat 8 loads classes and what tools we could use to debug.

Classloaders in Tomcat 8

When Tomcat is started, it creates a set of class loaders that are organized into the following parent-child relationships, where the parent class loader is above the child class loader. The default classloaders are:

• Bootstrap : This class loader contains the basic runtime classes provided by the Java Virtual Machine, plus any classes from JAR files present in the System Extensions directory ($JAVA_HOME/jre/lib/ext). Generally speaking, you wouldn’t setup anything here.
• System : This class loader is normally initialized from the contents of the CLASSPATH environment variable. In our context, this one is actually important, as this loads up all jars by putting them in classpath. Also, this classloader is responsible for loading Tomcat's logging implementation, implying this loads up logging.properties, which was also important for us.
• Common : This classloader by default loads classes / resources/ jars from tcServer/lib directory. Per Tomcat documentation, normally, application classes should NOT be placed here. However, our enterprise bundle adds tcapp/lib to this. Jars loaded by this can be configured through common.loader property in tcserver/conf/catalina.properties
• WebappX : This loads classes from WEB-INF/classes and WEB-INF/lib

Classloading Hierarchy :

The default loading hierarchy for loading of these classes are :

1. Bootstrap
2. Webapp
3. System
4. Common

That does seem a little weird to me, but I guess, there must be a good reason on why that is the default loading mechanism. Also, the order in which jars are loaded by a given classloader is not defined (see bug in references below). Coming from a Spring background, which holds bean initialization / dependency resolution as late as possible, to load other beans, this was a surprising fact. (And yes, I am aware that loading /wiring beans, is an entirely different thing than loading the classes themselves).

Customization 1: Specifying Loader Delegate as True

This hierarchy can be configured by specifying, in context.xml,then the order becomes:

1. Bootstrap
2. System
3. Common
4. Webapp

We did some funny combinations of the three jars, with combinations on where they are placed, with different loader delegate conditions and almost hacked ourselves to death, trying to figure out what is going on with classloading. Sometimes, the classes would load up, but on other times, with what seemed reasonable approach, they would not. We tried loading all three under same classloader, but they would fail, which seemed weird, but remember, the order in which jars are loaded by a given classloader is not defined 

Customization 2 : Explicitly loading classes before/after in Webapp classloader

Tomcat believes that depending upon a class to be loaded before should be done by putting it in a way that it is loaded by a different classloader which is loaded first, and ordering of jars within a given classloader is a smell. However, if we necessarily need this, this can be achieved by modifying the context.xml, to include Pre/Post Resources. So, in order to load files from a given path first, we could use the following block to order classloading:

                   base="/Users/theuser/mypictures" webAppMount="/pictures" />

Note here, that we don't need a custom class for this. There are couple of classes available from Tomcat that can be used to look at directory/ file /jar (DirResourceSet/ FileResourceSet / JarResourceSet). The resources such loaded can be made available to one or all contexts, using the webAppMount element.

Debug Tools

To our rescue, we added, -verbose:class to JAVA_OPTS in ApplicationEnv. With this we could see the actual order in which classes were getting loaded and that really helped with understanding what is going on. Although, the logs were interleaved between System and Webapp classloaders, to some extent, overall it was a big help.

The second thing we did (although it makes logs very very confusing) was plain and simple, to enable logging for Tomcat, by adding following to logging.properties in tcserver/conf:

With the help of these, we were eventually able to resolve our classloading issues. We also identified there is no one single way to achieve similar results. We could have used PreResources using default classloading hierarchy, but we ended up using Delegate=true, and customizing the load order.

References

• https://tomcat.apache.org/tomcat-8.0-doc/class-loader-howto.html
• https://issues.apache.org/bugzilla/show_bug.cgi?id=57129
• http://mikusa.blogspot.com/2014/07/tips-on-migrating-to-tomat-8-resources.html
• https://tomcat.apache.org/tomcat-8.0-doc/config/resources.html
• https://dzone.com/articles/what-is-jar-hell