Classloading in Tomcat 8

Problem

We recently began development on a new microservice, that connects to an existing Sybase database and is deployed on tcServer 4.0.1, which has bundled Tomcat 8. For reference, we had a similar microservice that connects to Oracle database, and a legacy monolith that connects to Sybase database, but deployed on JBoss container. We were not doing anything fundamentally new, and we expected this development to be quite straightforward.

However, when we deployed this application, we started running into weird issues with Sybase jar (JConn4), a Cybe-Ark provider jar, that masks connection to database, using its own driver, to fetch connection details from vault.
We spent quite some time trying to analyze this with various teams to figure out what is going wrong. We also have a third party jar integrated as a handler through logging.properties (in tcserver/conf) that send alerts when it finds errors in logs, and that just complicated things more.
I guess it’s that lucky time in my career, where I finally run into Jar Hell!

Analysis

Our default setup consists of loading some jars from a given file path, instead of using them from within the war / tcServer lib, as these are expected to be consistent across multiple applications we deploy on tcServer.
We tried experimenting with modifying where these jars are declared, versus where they are kept (external file / tcServer/lib / inside war) and kind of got sense that the issues seem to be due classes not being loaded when they are getting invoked.
This led us to analyze and understand classloading in Tomcat 8. Here are details on how Tomcat 8 loads classes and what tools we could use to debug.

Classloaders in Tomcat 8

When Tomcat is started, it creates a set of class loaders that are organized into the following parent-child relationships, where the parent class loader is above the child class loader. The default classloaders are:

Bootstrap : This class loader contains the basic runtime classes provided by the Java Virtual Machine, plus any classes from JAR files present in the System Extensions directory ($JAVA_HOME/jre/lib/ext). Generally speaking, you wouldn’t setup anything here.
System : This class loader is normally initialized from the contents of the CLASSPATH environment variable. In our context, this one is actually important, as this loads up all jars by putting them in classpath. Also, this classloader is responsible for loading Tomcat's logging implementation, implying this loads up logging.properties, which was also important for us.
Common : This classloader by default loads classes / resources/ jars from tcServer/lib directory. Per Tomcat documentation, normally, application classes should NOT be placed here. However, our enterprise bundle adds tcapp/lib to this. Jars loaded by this can be configured through common.loader property in tcserver/conf/catalina.properties
WebappX : This loads classes from WEB-INF/classes and WEB-INF/lib

Classloading Hierarchy :

The default loading hierarchy for loading of these classes are :

Bootstrap
Webapp
System
Common

That does seem a little weird to me, but I guess, there must be a good reason on why that is the default loading mechanism. Also, the order in which jars are loaded by a given classloader is not defined (see bug in references below). Coming from a Spring background, which holds bean initialization / dependency resolution as late as possible, to load other beans, this was a surprising fact. (And yes, I am aware that loading /wiring beans, is an entirely different thing than loading the classes themselves).

Customization 1: Specifying Loader Delegate as True

This hierarchy can be configured by specifying, in context.xml,then the order becomes:

Bootstrap
System
Common
Webapp

We did some funny combinations of the three jars, with combinations on where they are placed, with different loader delegate conditions and almost hacked ourselves to death, trying to figure out what is going on with classloading. Sometimes, the classes would load up, but on other times, with what seemed reasonable approach, they would not. We tried loading all three under same classloader, but they would fail, which seemed weird, but remember, the order in which jars are loaded by a given classloader is not defined

Customization 2 : Explicitly loading classes before/after in Webapp classloader

Tomcat believes that depending upon a class to be loaded before should be done by putting it in a way that it is loaded by a different classloader which is loaded first, and ordering of jars within a given classloader is a smell. However, if we necessarily need this, this can be achieved by modifying the context.xml, to include Pre/Post Resources. So, in order to load files from a given path first, we could use the following block to order classloading:

                   base="/Users/theuser/mypictures" webAppMount="/pictures" />

Note here, that we don't need a custom class for this. There are couple of classes available from Tomcat that can be used to look at directory/ file /jar (

DirResourceSet/ FileResourceSet /
JarResourceSet

). The resources such loaded can be made available to one or all contexts, using the webAppMount element.

Debug Tools

To our rescue, we added, -verbose:class to JAVA_OPTS in ApplicationEnv. With this we could see the actual order in which classes were getting loaded and that really helped with understanding what is going on. Although, the logs were interleaved between System and Webapp classloaders, to some extent, overall it was a big help.

The second thing we did (although it makes logs very very confusing) was plain and simple, to enable logging for Tomcat, by adding following to logging.properties in tcserver/conf:

With the help of these, we were eventually able to resolve our classloading issues. We also identified there is no one single way to achieve similar results. We could have used PreResources using default classloading hierarchy, but we ended up using Delegate=true, and customizing the load order.

References

Serialization in Java 201

This blog / cheat sheet is meant as a ready reckon-er for intermediate / advanced Java serialization users. It provides quick look at features provided in Serialization and standard FAQs.

Absolute Basics

Serializable - marker interface

Default mechanism works fine, including circular references, eg spouse of a person has spouse as this person. The critical things that the Java Object Serialization specification can manage automatically are:
Adding new fields to a class
Changing the fields from static to nonstatic
Changing the fields from transient to nontransient

Q: Why is Object not serializable?
A:
1. If you do not make the effort to design a custom serialized form, but merely accept the default, the serialized form will forever be tied to the class’s original internal representation. In other words, if you accept the default serialized form, the class’s private and package-private instance fields become part of its exported API, and the practice of minimizing access to fields loses its effectiveness as a tool for information hiding. Ref: Effective java
2. A second cost of implementing Serializable is that it increases the likelihood of bugs and security holes. Relying on the default deserialization mechanism can easily leave objects open to invariant corruption and illegal access. eg datasource with db credentials may get serialized, object initialisation may not occur and an invalid state object may be created.
3. A third cost of implementing Serializable is that it increases the testing burden associated with releasing a new version of a class.
4. Host of classes that do not need serialization - Thread, OutputStream, Socket, ServletRequest, etc
5. Issues with inner classes

serialVersionUID

If not defined, is computed using complex algorithm, consisting of class name, access modifier, various fields, constructor, initializer, etc.
Behind the scenes
Q: If there are multiple instances of an object in a given object chain, will it be written multiple times?
A: No, the first time an object is written, JVM assigns it a handle. On all subsequent writes, it writes only this handle and not entire object.

Q: If Objects are written using Object streams, how are primitives written?
A: DataOutputStream has methods to write primitives, eg writeInt, writeChar, writeUTF, etc

serialPersistentFields

Q: Is there an alternate way than to mark fields as transient to have only some fields Serialized?
A: Yes, define serialPersistentFields as below to specify fields that will be serialised
private static final ObjectStreamField[] serialPersistentFields = {new ObjectStreamField("next", List.class)};
If the field's value is null or is otherwise not an instance of ObjectStreamField[], or if the field does not have the required modifiers, then the behavior is as if the field were not declared at all.

Q: How to prevent a subclass of Serializable implementation to not be serializable?
A: Throw NotSerializableException in writeObject

Q: Is constructor called in deserialization? What about constructor of first non-serializable parent?
A: No, constructor of serialized class and parent classes that are serializable are not invoked. Constructor of all non-serializable parents are invoked.
Q: If I have ComputerTable extends Table implements Serializable, where Table has single constructor Table(int seatingCapacity), what will be seating capacity on deserialization if I set seatingCapacity as 1 on ComputerTable?
A: It will be 0, since Table is not serializable and its property seatingCapacity will not be serialized.

Modifying what is written / read

Default mechanism is public information and can be read and understood by anyone and hence is not secure.
private void writeObject(ObjectOutputStream) and private void readObject(ObjectInputStream) can be used to provide custom hooks. eg to obscure a field by maybe applying a function on it. Notice that these MUST be private, proving that neither method is inherited and overridden or overloaded
Q: What if writeObject is provided but no readObject is provided?
A:
If writeObject contains only defaultWriteObject(), it will work fine.
If writeObject does any processing beyond defaultWriteObject, that will be lost and only defaultObject will be read
If writeObject does not use defaultWriteObject and provides only custom processing, java.io.StreamCorruptedException is thrown while deserializing

Q: What does "this" refer to in readObject?
A: It refers to the new object that is under construction.

If another custom object should be written/read instead of actual object, private Object writeReplace and private Object readResolve methods may be provided. These may be used for serialization across different (otherwise mismatched) versions, or to serialize/ deserialize custom data. This custom object may be a proxy of original object.
Q: If I have a singleton object in a JVM, I serialize it and then deserialize it in same JVM, how many instances will I have?
A: Two instances, breaking singleton. Alternative is to use readResolve to fetch the singleton.

Q: What if object is modified after it has been written to stream?
A: The changes will be lost. Alternative is to reset the ObjectOutputStream and write again.

Q: What would be the best way to provide custom implmentation on how an enum is serialized?
A: enums are maintained by JVM and their serialization cannot be customised.

Q: Can a subclass override super class’s serialization mechanism?
A: Yes, by providing protected writeObjectOverride and protected readObjectOverride methods.

Sensitive information

The easiest technique is to mark fields that contain sensitive data as private transient. If they must be serialized, following may be used.

Serialization can be made secure easily by wrapping it in SealedObject (for confidentiality) or SignedObject (for integrity) or both (rather than by putting custom algorithms in read/writeObject methods)

Object Validation

To validate an object on construction, ObjectInputValidation maybe implemented. The method validateObject() needs to be overridden.

Externalizable

public void writeExternal(ObjectOutput out) throws IOException;
public void readExternal(ObjectInput in) throws IOException, ClassNotFoundException

Notice that these methods are "public"

ref:
http://www.ibm.com/developerworks/java/library/j-5things1/index.html
http://java.sun.com/developer/technicalArticles/Programming/serialization/
http://docs.oracle.com/javase/7/docs/platform/serialization/spec/serialTOC.html
Effective Java by Joshua Bloch, 2nd Ed

Dabbling In Software | Tricks | Cheat sheet | Musings

Friday, January 4, 2019

Managing Jar Hell in Tomcat 8