Tuesday, 2 February 2010

Replacing the XML stack used in OSGi with Xerces

I've recently been doing a lot of work on Android which uses the dalvikvm as its Java virtual machine. The class library that dalvik uses is based on Apache harmony and which, like all things free, is slightly different to the class library provided by Sun's implementation of Java. Not only is the implementation different, on Android it is fairly substantially crippled - missing large swathes of core Java functionality that we have all come to rely on. In particular Android only provides part of the javax.xml package space and the part it provides appears to be a 1.4 implementation which confuses things like Spring.

So, to cut a long story short, I needed a new XML stack, but the wrinkle is that I am running in an OSGi container. How should I do this?

The lazy option is to cram the new classes into the bootclasspath - but if you are going to do that why use OSGi in the first place?

So the first thing I tried was installing the relevant XML pieces as OSGi bundles. These are available in SpringSource's Enterprise Bundle Repository as xmlcommons, xerces and xml.resolver. The first is important as it provides the pieces of the javax.xml namespace that I was missing. The second is equally important since it provides the default implementations for this namespace. SpringSource have helpfully added OSGi manifests to the packages so I could immediately try them out in my OSGi container. But I wasn't going to try this directly on Android - that's far too much like hard work! (ok so hindsight is a beautiful thing and I did try this originally on Android). Since these packages are in the javax space I can configure a profile for my osgi container that only exports the packages I know are supported by the underlying VM:

org.osgi.framework.system.packages = \
javax.xml.parsers,\
...
org.osgi.framework.bootdelegation = \
javax.xml.parsers,\
...

(Note that I have not shown many other packages that are exported by the VM.) This profile lived in a file that I then referenced from my equinox command-line:

-Dosgi.java.profile=file:[file]

It's worth discussing these properties a little as it took me quite a long time - and pouring over numerous references - to find out the exact behaviour that they imply. Both of these are standard OSGi options which Costin describes quite well. org.osgi.framework.system.packages is actually an export declaration like Export-Packages and so must contain specific package declarations together with any required directives such as version. It declares the packages actually exported by the system bundle, so in my case its just exporting the javax.xml.parsers package. Any bundle importing this package would then get wired to the system bundle.

So far so good, but this is not the same as the classloader search performed by any code doing the equivalent of Class.forName(). Just because a package hasn't been wired doesn't mean it can't be loaded. This is where the second property comes in, it specifies the ClassLoader parent delegation model that will be employed. In other words if the current bundle can't load a class we are interested in, where will the system look next? In our case it will delegate to the ClassLoader of the system bundle for classes in javax.xml.parsers.

Now you might ask why these aren't the same thing. Well the first is the proper OSGi way of doing things - if you need javax.xml.parsers you specify that package as an import in your bundle and you will then get wired to the system bundle for that package. But there is a lot of code that is not well behaved like this. It doesn't know about OSGi and therefore may not import the package. In this case requests for classes in javax.xml.parsers will still succeed because of the boot delegation.

So that's what I tried initially and it failed dismally! The problem, I believe, is the boot delegation, although I'm not totally sure since I couldn't totally figure out why. The problem, as usual, manifests itself as ClassNotFoundException or NoClassDefFoundError and seems to be caused by the built-in parser using different versions of Xerces classes or vice versa. In theory it should work, but I could not make it so.

So the next thing I tried was to replace the entire stack. This involved removing javax.xml.* from the profile entirely. In theory this means that if a bundle needs something from javax.xml it will always look at its imports to find it.

This gave more predictable results in that javax.xml.parsers.DocumentBuilderFactory complained about not being able to find org.apache.xerces.jaxp.DocumentBuilderFactoryImpl. The latter (which lives in the xerces distribution) is the default apache implementation for the former (which lives in the xmlcommons distribution). What was missing were the dynamic imports in xmlcommons for the apache implementations. If you remember I got both from SpringSource's EBR - so what's going on here? These are supposed to be compliant OSGi bundles surely? Well yes, but not quite - the EBR bundles are automatically generated and it would be tricky, although not impossible, for a tool to figure out that it needed to dynamically import these classes.

So I added the xerces implementation classes as dynamic imports and this worked somewhat better, in particular my attempts to use javax.xml.parsers.DocumentBuilderFactory succeeded, but somewhat perplexingly, others failed - in particular javax.xml.validation.SchemaFactoryFinder and javax.xml.datatype.FactoryFinder. It turns out that these two use a slightly different classloading model for finding implementations. DocumentBuilderFactory first tries loading using the ContextClassLoader. If that fails then it tries Class.forName() which uses the classloader of the current class, which in our case is the bundle loader for xmlcommons. The bundle loader looks at the imports - static and dynamic - and, Bob's your uncle, the class is loaded.

The other FactoryFinders miss out the last step, using first the CCL and then failing. Well that's a bit daft! Why on earth would they choose two different loading strategies in the same implementation stack? Ours is not to reason why, but to hack and die. So I modifed the implementations to mirror the loading strategy of DocumentBuilderFactory, dynamically imported the appropriate implementations classes and, presto! Success!

I raised two issues for these problems with XML commons: 48698 & 48700

So to summarise:
  • Use profiles to prevent XML classes from being loaded from the system bundle
  • Hack XML commons to support a reasonable classloading strategy
  • Create a Bundle from XML commons that dynamically imports the Xerces implementation classes

1 comment:

Lindsay said...

I have had to do something similar. We removed all javax.xml packages from the system imports, and modified the commons-xml jar to ignore system properties. Etc etc - the whole xml loading strategy that javax uses needs a rewrite in my opinion.