Tuesday, 5 January 2016

Android Enlightenment

My New Year's resolution is to blog more.

It wouldn't be hard - my last post was in 2010. I should clarify, my last personal post was in 2010. I do blog on and off as my alter-ego CTO, Push Technology at www.pushtechnology.com. But company blogging doesn't allow you to establish a narrative or a following or any of those ego-enhancing things that the technorati cherish, but more about that another time.

I like blogs to be useful, informative, pithy, witty and above all correct. I shall aim to cover all of these in this inagural posting (pithy might be a stretch, but for the right reasons).

So Android.

I've had a few run-ins with Android in an earlier life. It's Java innit, but not as you know it Jim. It's got this generated dalvik stuff that's supposed to make it more compact and other good things, but in fact seems to make it more inflexible. It's got a so-sue-me-Oracle version of Java that's subtely different and broken in different ways to the way that the Oracle JDK is broken. It's got some standard libraries and others not-so-much. Then there's the SDK. I used to whine about the fruit company and it's walled garden, but at least the iOS SDK works in a relatively predicatable way. The Android SDK appears to work in a somewhat unpredictable and undocumented way. At some level that's ok - it's sort of open source and you can go and look at the code if you are really stumped, but it does add to the complexity and general drag involved in development.

And then there's the emulator.

Excuse me while I open the door and scream into the darkness.

On the good side the emulator works on a large number of platforms and generally seems to reasonably emulate what happens on a real device. It's just that it's so unbelievably slow. Ok so it has this Intel HAXM technology which is suppose to accelerate performance but it's still unbelievably slow. Unbearably slow.

So we have a platform that most of the connected world is using, but a platform that has any number of development pitfalls. So the absolute wrong way to develop for this platform would be to write-once run everywhere. Automate the hell out of it I say!

But how can you do this when the emulator is so slow? It get's worse - the only technology - HAXM - that has any hope of making automation usable is a virtualization technology and therefore will not work on top of other virtualization technologies - you know things like VMWare and Xen that you might possibly be using in your continuous integration environment.

But this is a success story. I have a solution, but in the spirit of Fermat you will have to wait for subsequent posts for the details.

Happy New Year!

Tuesday, 2 February 2010

Replacing the XML stack used in OSGi with Xerces

I've recently been doing a lot of work on Android which uses the dalvikvm as its Java virtual machine. The class library that dalvik uses is based on Apache harmony and which, like all things free, is slightly different to the class library provided by Sun's implementation of Java. Not only is the implementation different, on Android it is fairly substantially crippled - missing large swathes of core Java functionality that we have all come to rely on. In particular Android only provides part of the javax.xml package space and the part it provides appears to be a 1.4 implementation which confuses things like Spring.

So, to cut a long story short, I needed a new XML stack, but the wrinkle is that I am running in an OSGi container. How should I do this?

The lazy option is to cram the new classes into the bootclasspath - but if you are going to do that why use OSGi in the first place?

So the first thing I tried was installing the relevant XML pieces as OSGi bundles. These are available in SpringSource's Enterprise Bundle Repository as xmlcommons, xerces and xml.resolver. The first is important as it provides the pieces of the javax.xml namespace that I was missing. The second is equally important since it provides the default implementations for this namespace. SpringSource have helpfully added OSGi manifests to the packages so I could immediately try them out in my OSGi container. But I wasn't going to try this directly on Android - that's far too much like hard work! (ok so hindsight is a beautiful thing and I did try this originally on Android). Since these packages are in the javax space I can configure a profile for my osgi container that only exports the packages I know are supported by the underlying VM:

org.osgi.framework.system.packages = \
org.osgi.framework.bootdelegation = \

(Note that I have not shown many other packages that are exported by the VM.) This profile lived in a file that I then referenced from my equinox command-line:


It's worth discussing these properties a little as it took me quite a long time - and pouring over numerous references - to find out the exact behaviour that they imply. Both of these are standard OSGi options which Costin describes quite well. org.osgi.framework.system.packages is actually an export declaration like Export-Packages and so must contain specific package declarations together with any required directives such as version. It declares the packages actually exported by the system bundle, so in my case its just exporting the javax.xml.parsers package. Any bundle importing this package would then get wired to the system bundle.

So far so good, but this is not the same as the classloader search performed by any code doing the equivalent of Class.forName(). Just because a package hasn't been wired doesn't mean it can't be loaded. This is where the second property comes in, it specifies the ClassLoader parent delegation model that will be employed. In other words if the current bundle can't load a class we are interested in, where will the system look next? In our case it will delegate to the ClassLoader of the system bundle for classes in javax.xml.parsers.

Now you might ask why these aren't the same thing. Well the first is the proper OSGi way of doing things - if you need javax.xml.parsers you specify that package as an import in your bundle and you will then get wired to the system bundle for that package. But there is a lot of code that is not well behaved like this. It doesn't know about OSGi and therefore may not import the package. In this case requests for classes in javax.xml.parsers will still succeed because of the boot delegation.

So that's what I tried initially and it failed dismally! The problem, I believe, is the boot delegation, although I'm not totally sure since I couldn't totally figure out why. The problem, as usual, manifests itself as ClassNotFoundException or NoClassDefFoundError and seems to be caused by the built-in parser using different versions of Xerces classes or vice versa. In theory it should work, but I could not make it so.

So the next thing I tried was to replace the entire stack. This involved removing javax.xml.* from the profile entirely. In theory this means that if a bundle needs something from javax.xml it will always look at its imports to find it.

This gave more predictable results in that javax.xml.parsers.DocumentBuilderFactory complained about not being able to find org.apache.xerces.jaxp.DocumentBuilderFactoryImpl. The latter (which lives in the xerces distribution) is the default apache implementation for the former (which lives in the xmlcommons distribution). What was missing were the dynamic imports in xmlcommons for the apache implementations. If you remember I got both from SpringSource's EBR - so what's going on here? These are supposed to be compliant OSGi bundles surely? Well yes, but not quite - the EBR bundles are automatically generated and it would be tricky, although not impossible, for a tool to figure out that it needed to dynamically import these classes.

So I added the xerces implementation classes as dynamic imports and this worked somewhat better, in particular my attempts to use javax.xml.parsers.DocumentBuilderFactory succeeded, but somewhat perplexingly, others failed - in particular javax.xml.validation.SchemaFactoryFinder and javax.xml.datatype.FactoryFinder. It turns out that these two use a slightly different classloading model for finding implementations. DocumentBuilderFactory first tries loading using the ContextClassLoader. If that fails then it tries Class.forName() which uses the classloader of the current class, which in our case is the bundle loader for xmlcommons. The bundle loader looks at the imports - static and dynamic - and, Bob's your uncle, the class is loaded.

The other FactoryFinders miss out the last step, using first the CCL and then failing. Well that's a bit daft! Why on earth would they choose two different loading strategies in the same implementation stack? Ours is not to reason why, but to hack and die. So I modifed the implementations to mirror the loading strategy of DocumentBuilderFactory, dynamically imported the appropriate implementations classes and, presto! Success!

I raised two issues for these problems with XML commons: 48698 & 48700

So to summarise:
  • Use profiles to prevent XML classes from being loaded from the system bundle
  • Hack XML commons to support a reasonable classloading strategy
  • Create a Bundle from XML commons that dynamically imports the Xerces implementation classes

Tuesday, 15 July 2008

ClassLoader hell the OSGi way

Recently I have been trying to write a proxy service in OSGi. The reasons for wanting to do this are varied, but it is mostly about imports. In OSGi (and Spring DM) you have to import packages that you use. This is all well and good until you start needing to import packages that you didn't know you used - proxies in particular. Spring uses proxies to augment bean functionality usually using AOP and Spring DM uses proxies to represent client services. All well and good - I don't want to describe here the benefits of proxies. The issue is that in my code, or Spring context file, I specify that I am creating a FooBean so clearly I need to import FooBean's package. However when it gets proxied I need to also start importing interfaces exposed by the proxy. In Spring's case this is org.springframework.aop.SpringProxy and org.springframework.aop.framework.Advised.

Fine, I will import those two packages. FooBean is an impl so I am using cglib to proxy it. cglib adds net.cglib.proxy.Factory to the proxied interface set so I need to add that import as well. I can just about live with those, although explaining them to users will be a pain. Next, I want to hide all of the proxy implementation classes inside another bundle, so I write a service that provides approximately the same set of functionality as org.springframework.aop.framework.ProxyFactory. Easy as pie.

You would think.

The use of Factory is the first problem. When I run my little service I start getting NoClassDefFoundErrors for net.sf.cglib.core.Signature. This package is imported by my proxy service provider bundle, and the provider bundle's classloader is the one I am passing to cglib's Enhancer for proxy generation. So why can't the bundle classloader for Enhancer see Signature?
Well the bundle classloader for Enhancer turns out to be the client bundle because Enhancer and Factory are in the same package. So the fact that the client needs to see Factory also means that it imports Enhancer (rather than my provider bundle) and the whole thing unravels from there.

Spring has the same problem. The cglib entry point is defined by org.springframework.aop.framework.Cglib2AopProxy. Does that package look familiar somehow? Right, its the same package as Advised - needed by the proxy. Doh!

So now my options are:
  1. Change Spring and cglib - bit of a backwards compatibility problem there.
  2. Start massaging classloaders for individual classes - not exactly the OSGi way
  3. Chain the required packages into the client classloader - again subverting the OSGi class space protection model
  4. Create some entirely new classloader for the proxy - not clear to me that this will actually work
  5. Try using Enhancer.setUseFactory()
  6. Import the packages into my client bundle, *sigh*
The moral of the story: keep interfaces in separate packages when you are targetting OSGi.


I have actually now used a modified version of (3) above, and it works quite nicely. The theory goes like this - the client bundle knows about classes that need augmenting, and contains the appropriate imports for these. The provider bundle knows about proxying and contains the appropriate imports for the mechanics of this. So what we actually need is a classloader that synthesizes these two. Constructing a simple delegating classloader which first tries the client classloader and then the provider ends up working really nicely. Even better I would argue that this does not subvert the OSGi model, the reasoning is that this delegate classloader is only used for proxying - no client classes are magically loaded by it - and the class space represented by the union of the two classloaders involved is still bounded by their respective imports. No need to use dynamic imports or other ugly hacks. Furthermore the class space boundary is neatly delineated by the service definition. Only classes on the client-side need to be imported and no details of the service implementation are exposed by the required imports.