Assorted facts about JBoss. Fact 2: classloading. Broken by default.

The classloading situation in JBoss is a mess. You know, evolutionary kind of mess. They began with something messy, then started to add more things, configuration parameters, bells and whistles ... Some old classloading problems went away, just to be replaced by new problems.

I had my share of ClassNotFoundExceptions, NoClassDefFoundErrors, and LinkageErrors before. Not all of them were caused by JBoss, but those that were caused by JBoss were the toughest to resolve.

Recently JBoss (JBoss Application Server 6) appeared again on my professional horizon. And one of the first problems I ran into was a LinkageError. Even that problem with StAX API jar happened later.

The deployment of the application fails. Sometimes. And sometimes it is OK. If it fails, the error is

java.lang.LinkageError:

    loader constraint violation:

    loader (instance of org/jboss/classloader/spi/base/BaseClassLoader)

    previously initiated loading for a different type

    with name "org/xml/sax/Attributes"

The class name is not always org.xml.sax.Attributes, but it is always a class from org.xml.sax. And if the deployment is OK, the same error happens later, at runtime.

Nothing new. This error screams "Duplicate class". So I looked around and found jtidy-4aug2000r7-dev.jar packaged in EAR/lib (and only there) which has its own copy of org.xml.sax and org.w3c.dom classes. JMX classloader bean for the application confirmed that the classes are coming from the application's EAR.

The reason why the classes are packaged into jtidy-4aug2000r7-dev.jar is not really important here. But I was really surprised (well, initially) that JBoss uses these classes instead of JDK classes. I deployed the application on the stock version of JBoss AS 6 without any modifications of classloading configuration. There was even no jboss-app.xml (I was planning to add it, just in case) let alone other classloading specific files. So I expect JBoss to have JEE compatible classloading behavior.

Initially I did not have time to investigate the problem. After notifying the project owner of the problem the decision is made: remove JTidy jar from EAR/lib and continue. JTidy is used in some really obscure piece of code that is called not that often. We can deal with this later. There is even a chance that this functionality will be rewritten to get rid of JTidy. But now we need a version of the application running in JBoss.

I did just that and went ahead. But I kept wondering. I could not understand why the presence of jtidy-4aug2000r7-dev.jar in EAR/lib causes such an error. It goes against all my knowledge, understanding of and experience with java classloading. Except that I am dealing with JBoss. But even JBoss would not do that, would it?

More importantly that problem might have a much broader effect on the project but I could not even imagine what kind of effect. More likely negative, that I was sure of.

Finally I have got some spare time. I have created a very simple application with a single EJB module with one SessionBean and jtidy-4aug2000r7-dev.jar as a dependency, packaged as an EAR application. After deploying it into JBoss I went into JMX classloader bean and verified that org.xml.sax classes are coming from the test application. The SessionBean has a business method that does Class.forName() and then returns getProtectionDomain().getCodeSource() of the loaded class. I created a simple EJB client application that calls the business method with different class names. All the org.xml.sax and org.w3c.dom classes present in jtidy-4aug2000r7-dev.jar were indeed coming from this jar file.

Next I have created another jar file manually packaging some classes from various places and packages like javax.xml.bind, org.dom4j, org.hibernate, java.text, javax.management, etc. I replaced jtidy jar with this new jar file and repeated the test. Only with java.* classes the business method was returning null which means that only those classes were coming from the primordial class loader. All other classes present in the jar were loaded from it. Well, well, well, JBoss at it again.

Trying to find anything specific about the problem on the net did not help. Classloading problems in JBoss is a really hot topic after all!

If nothing else helps ... Use the Source Luke!

Actually even before going deep I noticed one interesting thing in JBoss JMX Console. It appears that my demo application has a classloader domain JMX even if I did not have JBoss specific deployment descriptors. I clicked on it and again one thing stood out just screaming "Look at me":

ParentPolicyName, MBean Attribute, AFTER_BUT_JAVA_BEFORE.

"After but java before". Sounds familiar. pseudoTransactionEnlistment anyone? Or maybe BOZOSLIVEHERE?

The rest was easy. The biggest problem was to get the right source files quickly. You can't nowadays just download a fat zip or gz file with all the sources in it. After that was done, a bit of grepping and the like reveals the truth:

Class org.jboss.classloader.spi.ParentPolicy with some predefined instances like BEFORE, AFTER, BEFORE_BUT_JAVA_ONLY, etc. The comments around these predefined instances explained the meaning. In my case it was AFTER_BUT_JAVA_BEFORE. Except that the source file claims that AFTER_BUT_JAVA_BEFORE means "Java and Javax classes before, everything else after" and my tests show that javax.* classes also come from the jar file in EAR/lib. A bit more looking around led me to this piece of code in class org.jboss.classloading.spi.dependency.Module:

public ParentPolicy getDeterminedParentPolicy()

{

    if (isJ2seClassLoadingCompliance())

        return ParentPolicy.BEFORE;

    else

        return ParentPolicy.AFTER_BUT_ONLY_JAVA_BEFORE;

}

Since I do not have any JBoss specific deployment descriptors isJ2seClassLoadingCompliance() returns false resulting in ParentPolicy.AFTER_BUT_ONLY_JAVA_BEFORE being used. Not AFTER_BUT_JAVA_BEFORE. The comments next to AFTER_BUT_ONLY_JAVA_BEFORE in ParentPolicy.java clearly match the observed behavior: "Java classes before, everything else after". What am I missing?

Turns out there is one more small thing: a copy-paste error in the definition of AFTER_BUT_ONLY_JAVA_BEFORE:

/** Java and Javax classes before, everything else after */

public static final ParentPolicy AFTER_BUT_JAVA_BEFORE =

     new ParentPolicy(ClassFilterUtils.JAVA_ONLY, 

     ClassFilterUtils.EVERYTHING,

    "AFTER_BUT_JAVA_BEFORE");



/** Java classes before, everything else after */

public static final ParentPolicy AFTER_BUT_ONLY_JAVA_BEFORE =

     new ParentPolicy(ClassFilterUtils.NOTHING_BUT_JAVA,

     ClassFilterUtils.EVERYTHING,

     "AFTER_BUT_JAVA_BEFORE");

Mystery solved. I have added jboss-app.xml to the EAR with <loader-repository-config>java2ParentDelegation=true</loader-repository-config> (might just as well have added jboss-classloading.xml), redeployed the application and sure enough I have got BEFORE as ParentPolicyName in JMX Console, but more importantly I have now the expected classloading behavior. For each class present the test jar in EAR/lib both JMX Console and my SessionBean load the class not from the test jar but form some other place like JDK or jars from <jboss>/common/lib.

I have mentioned above that I was planning to add jboss-app.xml to the application anyway because I do not trust JBoss. Boy I was right. The end result for me would have been the same but I would have missed all that fun.

But ... Who in their right mind comes up with such interesting classloading logic?? What they were trying to achieve? Why the hell I have to explicitly "opt in" to get the most sensible classloading configuration? *


* Note: ideally. The current state of classloading affairs in JEE containers makes it much harder than necessary. Internal container classes and classes from various third-party jars that container is using leak into an application. This is a big deal even if the application does not have conflicts with those third-party jars. In case of conflicts all bets are off. Granted containers provide some mechanisms to fine tune classloading, but these mechanisms do not always work. Yes, JBoss, it is about you. But I do think that this "delegate to the parent first except when in WAR" is the most sensible classloading configuration and definitely the one to start with and to try to stick to as much as possible.

Assorted facts about JBoss. Fact 1: StAX (Streaming API for XML) and the meaning of -711357515002332258.

Every time I have to do some serious work with JBoss I come across a situation that requires patching JBoss. HelloWorld kinds of applications tend to work, but as soon as things get complicated there is always something...

This time it is JBoss6 and StAX API. You see, there is <jboss>/lib/endorsed directory with some files in it. Normally if you start jboss with <jboss>/bin/run[.bat] the JVM is started with -Djava.endorsed.dirs=<jboss>/lib/endorsed

No problem, it is the desired and documented behavior if one wants to have newer versions of some APIs available in JDK. But <jboss>/lib/endorsed/stax-api.jar is a bit different. It is there for the sake of JDK 1.5. As of JDK 1.6 StAX is part of JDK itself. And it is not like JBoss packages a better or newer version of StAX. So if you run JBoss on JDK 1.6, do yourself a favor: delete <jboss>/lib/endorsed/stax-api.jar right now.

The solution for JDK 1.5 is not so simple because JDK 1.5 does not provide StAX. But first, what is the problem? This is it:

java.io.InvalidClassException: javax.xml.namespace.QName;
local class incompatible: stream classdesc serialVersionUID = -9120448754896609940, 
local class serialVersionUID = -711357515002332258

<jboss>/lib/endorsed/stax-api.jar contains more than just StAX classes. It contains some old versions of classes that long ago present in JDK. And because the classes are in an endorsed jar, they override standard JDK classes.

If you look into JDK source code you will see that class javax.xml.namespace.QName goes to some lengths to initialize private static final long serialVersionUID with some known good value. The version packaged in <jboss>/lib/endorsed/stax-api.jar does not define field serialVersionUID leaving you at mercy of the JVM algorithm to calculate serial version UID. Which produces -711357515002332258 in this particular case.

Bad luck if you have a serialized instance of a class which has a non-transient field of type javax.xml.namespace.QName. Or if you have QName as a parameter in one of your remote interfaces.

So I fixed the problem by removing stax-api.jar since I am running under JDK 1.6 and went ahead.

The simplest solution for JDK 1.5 is probably deletion of everything that is not under javax.xml.stream from <jboss>/lib/endorsed/stax-api.jar. There are also stax-api jars around that include only javax.xml.stream.* classes.

But still ... One thing bothered me. This is quite an easy mistake to make especially if this file was added to the endorsed dir some time ago. I can see that it is present in JBoss5; I did not check earlier versions. But come on is it that difficult to review these things for every major release?

I can't be the first one to hit this problem. A bit of googling brought me here (JBPAPP-4223). OK, a bug is reported but I guess nobody is going to do a thing about it. After all it was reported on the 5th of May 2010, JBoss 6.0.0.Final was released half a year later, still with the problem.

And then I found this little gem. The beauty here is the recommendation of the JBoss EJB3 Lead Developer. Just read it. He seriously proposes to add the broken stax-api.jar to the client endorsed jar set. WTF?! JBoss EJB3 Lead Developer? No kidding?

Am I really surprised? Not at all.

Mule, HTTP and transaction management

Mule has support for transactions, see here. So if the inbound and outbound endpoints are transactional, like JDBC or JMS, it is easy to make sure the messages are handled transactional.

It is not so easy if an endpoint is not transactional. For example we have a configuration with a jms:inbound-endpoint and an http:outbound-endpoint. A message is retrieved from the queue and sent via HTTP to some receiver. Of course the message must be removed from the queue only if it is successfully received (or handled) by the receiver.

The inbound-endpoint configuration is easy:

    <inbound>

        <jms:inbound-endpoint queue="${queue_name}" connector-ref="jmsConnector">

            <jms:transaction action="ALWAYS_BEGIN"/>

        </jms:inbound-endpoint>

    </inbound> 

ALWAYS_BEGIN ensures that a new transaction is started and a message is received in this transaction.

This leaves the outbound-endpoint. Just saying

        <http:outbound-endpoint address="${http_address}"/>

is not enough because it automatically means action="NONE". Mule throws an exception complaining that the outbound endpoint cannot join the active transaction because it is configured with action="NONE". Fair enough, let's change this into

        <http:outbound-endpoint address="${http_address}">

            <http:transaction action="JOIN_IF_POSSIBLE"/>

        </http:outbound-endpoint>

But this does not work because "http:transaction" is not recognized as a valid element. Ooops. This is logical, HTTP is not transactional per definition. But we really need JMS to be transactional.

The solution?

        <http:outbound-endpoint address="${http_address}">

            <jms:transaction action="JOIN_IF_POSSIBLE"/>

        </http:outbound-endpoint>

Mule is happy with this and it does the right thing. If there is a problem connecting to the target or sending the message to it HTTP endpoint makes sure that the message gets "exception payload" set. This triggers Mule transaction support to rollback the active transaction.

This is not a generic solution, but it suits us: no messages are lost; the message is back in the queue and is redelivered later. The only problem with this approach is that the transaction can be rolled back after the message was successfully received by the HTTP receiver. This results in redelivery of the same message to the HTTP receiver which must be able to handle this.

But this is not a big deal in our case. It is so happens that most of the time the message ends up in a dispatcher that looks in its registry for subscribers. The first successful delivery of the message caused subscribers to unsubscribe so the dispatcher just silently drops the message.

In some other cases the message is just notification of some kind so nobody really cares if the same notification appears twice.

The only case when this might cause some trouble in our system is when such a message results in a creation of a BPEL process instance. Most of the time the newly created instance fails with "conflicting receive" error because the instance created after the first message delivery is still running and the process has some <onMessage> with a correlation set. But these cases are easily recognized by the administrators. And I must say if one is using BPEL then "conflicting receive" is the least of one's worry.

Fairy tale of developer's heaven

Ah maven! Promises of repeatable and portable builds... Fairy tales of developer's heaven...

Shall I tell one as well?

From time to time I do some things on a JEE project being developed by some other people. The resulting EAR file is quite dependent on some JEE container. Recently a decision was made to migrate the project to another JEE container. It is not a one-off migration effort: we need to be able to build the project for the original container and for the new one. I was asked to look into that.

One possibility to handle this is to have a separate branch for each container. Just thinking about all coming cross merges makes me cry.

Another possibility is to have a single branch and just package additional container specific deployment descriptors here and there. Nice and simple, perfectly JEE compliant ...

Except for one seemingly small detail. The project is using say version 1 of a particular 3rd party dependency. Unfortunately the project with this version can't be deployed into the newly targeted container. Fortunately there is already version 2 of the dependency, and the project built against this version can be deployed in the new container. Unfortunately it is precisely the other way around with version 2: the old container can't handle it.

And it is not that we can build the project against one of the versions and then just package differently: the versions are not binary compatible. There are also some minor source-code compatibility issues, but they can be relatively simply solved. Anyway, bottom line: if we build our project against one of the versions of the library it will not work with the other. We really need to compile our software against the correct version of the library and then package accordingly.

So far so good. The situation is probably quite common, and not very difficult to handle. Normally.

But the project is using maven, more specifically, maven 2.2.1. So far it worked pretty well for the project except for some of maven's WAR/EAR packaging "features". But for this new deployment target I hit a wall.

The end result I wanted to achieve: one checks out the source code, sets up the container specific environment, runs 'mvn install' and gets a set of properly versioned container specific artifacts, say, application-X.Y-container1.ear or application-X.Y-container2.ear. The "properly versioned" part is very important. This way we can refer to the correct versions of artifacts in our poms, we can properly release container specific versions of the project, etc.

The very first question was: how do we achieve that versioning scheme in maven? The 3rd party library is used in all ejb and almost all jar modules making them depend on the library. Web modules also depend on it (indirectly, via ejb/jar modules). The same is true for the ear module.

Why, it is easy I thought. I define a property, say, 'target.container', then put it in <artifactid> or <version> tag so the project's poms have <artifactid>usermanager-ejb-${target.container}</artifactid> or <version>0.1-${target.container}-SNAPSHOT</version> in their maven coordinates. Then I start maven with -Dtarget.container=container1 (or container2). This results in <artifactid>usermanager-ejb-container1</artifactid> or <artifactid>usermanager-ejb-container2</artifactid> (or version <version>0.1-container1-SNAPSHOT</version> or <version>0.1-container2-SNAPSHOT</version>) at build time. Problem solved.

Funny thing: it worked. Damn, I should have been more suspicious. I ended up trying both variants, and both worked when maven was executed from command line. That was actually the last thing that worked. Following hours made me really unhappy.

First I noticed that Eclipse (m2eclipse plugin) does not really like my new poms. It kept complaining that it could not find the project's poms and their dependencies. Executing 'maven install' from Eclipse produced a lot of warnings like "'artifactId' contains an expression but should be a constant." That prompted me to move ${target.container} from <artifactid> to <version>. And again, running maven from command line worked. Eclipse kept complaining.

Googling the message I came across a lot of posts related to the same issue. The message from these posts was clear: maven does not support it.

Strange, because maven documentation does not clearly spells it. For example, POM Reference does not say that artifactId or version must be constant. In fact, the very same POM Reference says here:

Maven properties are value placeholder, like properties in Ant. Their values are accessible anywhere within a POM by using the notation ${X}, where X is the property.

Yet people claim that it should not work, and if it works then it is a bug. For example, here or here.

And it looks like maven finally began enforcing this stupidity in version 3. Which is a pity. The funniest thing here is the reason why. Take for example MNG-4297:

Maven currently allows properties in the groupId, artifactId and version of a pom. This causes artifacts to be produced that require full inheritance and interpolation before they can be uniquely identified. It also poses potential problems if the properties are defined in settings, env or profiles where the consumer can't exactly identify the artifact after the fact.

I am just speechless. Between things like downloaded newer versions of plugins (wow, they fixed this one actually), disappeared artifacts from public repositories, rearranged and/or moved public repositories they finally nailed the real problem preventing maven users to have reproducible builds: property-based project coordinates. Strange, last time I looked maven still supports profiles and properties in general...

Anyway, it looks like the way to do what I want is classifiers. They are mentioned here:

The Maven provided solution for your situation is 'classifiers'.

and are really awesome described here (5.5.3. Platform Classifiers). These guys work for Sonatype, so they should know a thing or two about maven you would think, right? Just replace <classifier>win</classifier> with <classifier>container1</classifier> and <classifier>linux</classifier> with <classifier>container2</classifier>, and we are back in business.

Damn you, people who like to misinform others.

I went ahead and modified poms. Build the project from the command line produced some strange results, for example maven built modules 1-6 successfully and then failed building module 7 because of my mistake in the module's pom. I fixed the problem, ran 'mvn install' for module 7, it completed without failure. Then I ran 'mvn clean install' for the parent project, and it failed building module 5 because some classes from one of the dependencies were not found. Huh?

Finally after some more pom changes the build managed to produce an EAR which was successfully deployed in the old container. Then I cleaned up some poms, added some things to <dependencyManagement> here and there, and executed 'mvn clean install' again. I was not able to deploy the resulting EAR in the old container because of some missing classes. It turned out about 1/3 of jars were missing from EAR/lib this time. WTF?!

Running mvn dependency:tree and analyzing its result explained why: there were no dependencies under the dependencies with classifiers! Time to ask google again.

Apparently maven cannot handle transitive dependencies of a dependency with a classifier, see for example MNG-2759. The story is a bit more complicated because this works sometimes for some people. This worked at least once for me. But most of the time it does not work. And maven is not planning to fix it: MNG-2759 has status "Won't Fix".

Yeah, use classifiers if you need some quality headache. Thanks for advice, guys!

The only solution is to spell all dependencies explicitly everywhere I use a module with a classifier. Thanks, but no, thanks. I already have to do it too many times. <dependencyManagement>, <dependency>, WAR packaging exclusion, EAR packaging to make sure that what is excluded from WAR is packaged in EAR/lib.....

What can I say? Indeed, maven is really a "project ... comprehension tool".

And what I am going to do with all this mess? Nothing, really. I have dropped the requirement of having properly versioned artifacts. I have just removed all the classifiers. The profiles stay. So any time I build a project I get a version 0.1-SNAPSHOT which happens to be for one of the containers. Which one? It depends on the chosen profile. This all means more work during release, but frankly I do not care at the moment. Do you want to refer to the released version of one of the submodules in your pom file? You'd better be absolutely sure you know which version you use. Do you want to refer to a SNAPSHOT version?

Reproducible builds and maven? Do not make me laugh.

Mule startup quirks

Mule saga continues. Next problem I wanted to tackle was "we had to restart the mule server and the problems went away … or not ... and we had to restart it again ...".

A small detour: Mule can run as a standalone application where it handles everything itself. It can also be embedded into some application or packaged as a JEE web application and then deployed in a web container. Originally the project's Mule was running as a standalone server. This was changed at the client's request some time before going into production and now Mule is an EAR file with a single web application that runs in a separate OC4J instance. Mule provides the necessary code to make it work: a servlet context listener ( MuleXmlBuilderContextListener) to initialize Mule and a servlet ( MuleReceiverServlet) to receive HTTP requests. As a side effect of this embedded mode Mule can't really handle its lifecycle. Everything is managed through OC4J and OPMN. This was also given as the reason to use embedded mode: "look, this way we get Mule into our management infrastructure!"

Back to the topic: needless to say stopping/starting the mule server multiple times locally or in a development or test environment did not cause any problems. Time to look at the production log files. No surprises, it is full with exception stack traces, which makes it much harder to find something if one does not know what to look for. It took me some time to find this lonely line:

ERROR [OC4J Launcher] org.mule.retry.notifiers.ConnectNotifier: Failed to connect/reconnect: jms://QUEUE_NAME. Root Exception was: Io exception: Connection reset. Type: class java.sql.SQLException

After looking at the log entries before and after the found one I realized what has happened and why it was necessary to restart Mule multiple times. It goes like this:

1. The Oracle database that provides AQ JMS support experiences some problems. As a result Mule also has problems, connections get dropped, etc.


2. Administrators restart Mule server without (fully) solving database problems.


3. Mule JMS support code fails to initialize JMS connection at Mule startup. This stops Mule startup sequence but not OC4J instance or OC4J Mule application. Mule can't stop itself because it does not control its lifecycle when it runs embedded.


4. As a result Mule is started in a half broken state with at least some services uninitialized but is seen by OPMN as successfully started. So much for the management infrastructure.


5. Subsequent requests to Mule fail because even though Mule servlet is up and running most of Mule services are not.

Okay, the problem is clear. Admins are notified to watch for that "Failed to connect/reconnect: jms://QUEUE_NAME" message in the logs. Except that it does not help much because OC4J (actually OPMN) can choose to bring OC4J instances down and up any time OPMN finds necessary. Given the software runs in OC4J cluster this can lead to situations when Mule runs on one node in the cluster and fails on another. Oops.

Now the question is: can anything be done to improve the situation?
I looked at the Mule initialization logic (class MuleXmlBuilderContextListener) and found that it does something ... interesting.

Method MuleXmlBuilderContextListener.initialize(ServletContext context) is responsible for initialization and here it is (from Mule 2.2.1), with some minor details omitted:

    public void initialize(ServletContext context)
    {
        ...
        try
        {
            muleContext = createMuleContext(config, context);
            muleContext.start();
        }
        catch (MuleException ex)
        {
            context.log(ex.getMessage(), ex);
            // Logging is not configured OOTB for Tomcat, so we'd better make a
            // start-up failure plain to see.
            ex.printStackTrace();
        }
        catch (Error error)
        {
            ...
            throw error;
        }
    }

Basically almost any failure during initialization, be it configuration file errors, connection problems, whatever, are ignored. Sure they are logged, at best, but that is it. RuntimeExceptions are not caught, but most of Mule failures during startup result in MuleException, which is caught. For example, errors in the configuration file lead to an exception thrown somewhere down the chain from createMuleContext(). Some connection failures happen somewhere in muleContext.start() and do not throw any exception but do stop the initialization. What were they thinking?!

Is it possible to do better than that? I ended up with this, but a copy-paste version of MuleXmlBuilderContextListener with some modification would work as well:

public class CustomMuleXmlBuilderContextListener extends MuleXmlBuilderContextListener {
 
    public void contextInitialized(ServletContextEvent event) {
        initialize(event.getServletContext());
        if (muleContextEx != null) {
            String errMsg = "Mule context is not initialized";
            logger.error(errMsg, muleContextEx);
            throw new IllegalStateException(errMsg, muleContextEx);
        }
        if (muleContext != null && !muleContext.isStarted()) {
            String errMsg = "Mule context is initialized but some services failed to start";
            IllegalStateException ex = new IllegalStateException(errMsg);
            logger.error(errMsg, ex);
            throw ex;
        }
    }

    protected MuleContext createMuleContext(String configResource, ServletContext context)
        throws ConfigurationException, InitialisationException
    {
        try {
            return muleContext = super.createMuleContext(configResource, context);
        } catch (ConfigurationException ex) {
            muleContextEx = ex;
            throw ex;
        } catch (InitialisationException ex) {
            muleContextEx = ex;
            throw ex;
        }
    }

    //
    private MuleContext muleContext;
    private MuleException muleContextEx;
}

I had to override createMuleContext to get the created context instance because muleContext.isStarted() is the only way to find out if Mule has actually started successfully. This new listener makes sure that:

1. If Mule is not started but reports no exception the listener makes sure there is one, and it is logged. (Yeah, even more noise in the log.) But this makes a failure like the one caused this investigation "Failed to connect/reconnect: jms://QUEUE_NAME" be more visible in the log.


2. A failure during Mule initialization and startup is reported back to the web container. This hits another grey area of servlet specification which does not say what MUST or SHOULD happen in this case. There is a note, for example, in v. 2.4 of Java Servlet Specification:
   

   SRV.10.6 Listener Exceptions
   
   Some exceptions do not occur under the call stack of another component in the application. An example of this is ... a ServletContextListener that throws an unhandled exception during a notification of servlet context initialization ... In this case, the Developer has no opportunity to handle the exception. The container may respond to all subsequent requests to the Web application with an HTTP status code 500 to indicate an application error....

   
   OC4J has here fortunately a sensible behavior: the web application is marked internally as not started. Any HTTP request to it triggers an attempt to start it. So the listener gets another chance to start Mule and if the problem went away Mule starts and ready to serve requests. This does not work of course for configuration file failures but does its job nicely for connection failures.

Note: I know about RetryPolicy and its friends. It is a subject of one of my upcoming posts.

What should I think of mule?

"Every SOA project must use an ESB". I do not remember who said that to me but some people do follow this advice. The project mentioned in the previous post is not an exception. Being a nice example of the vendor driven architecture the project had used Oracle ESB for some time. Apparently that thing was so ridden with bugs the team decided to use something else instead, namely, mule community edition. By the time I have joined the project the switch was completed so I was not really involved with ESB. Until recently.

The production environment has being behaving quite flaky for quite some time. Some of the problems were traced back to the Oracle BPEL software. But there were also problems related to the mule server. The problem descriptions were not really helpful. It went like that: "oh, we have messages remaining in the queue", or "the problems begin after one of other services goes down; we suspect that mule holds HTTP connections open ...", or "we had to restart the mule server and the problems went away … or not ... and we had to restart it again ...".

Time to investigate.

The mule configuration used by the project is not that complex. There are some proxy services that were added under "URL transparency" motto. Couple of those proxy services perform some minor request/response transformations. And then there are some services that logically look like one way WS endpoints but internally use Oracle AQ messaging and a simple content-based routing to decouple sending and receiving components and to add message recovery.

The first thing I wanted to check was that "we suspect that mule holds HTTP connections open ..." and see if it is possible to force mule to close an HTTP connection after each request. After all the task sounded easy.

As a seasoned developer who knows next to nothing about a piece of software I google for "mule http connection keepalive". The first link HTTP Transport - Mule 2.x User Guide - mulesoft.org looks promising.

Wait a minute... to read documentation I have to be registered? Rrrrright. Given I need to investigate several possible problems it might be better to register, but this can wait. Google cache first.

I should say the documentation looks nice. Except of course you need to know all the mule story and architecture before you can get sense of the documentation. And it still does not explain why they need to have both 'keepAlive' and 'keep-alive' properties. Anyway after some reading I thought my problem is solved. After all this is what the documentation says:

Attributes of <outbound-endpoint...>
...

keep-alive

boolean

Controls if the socket connection is kept alive. If set to true, a keep-alive header with the connection timeout specified in the connector will be returned. If set to false, a "Connection: close" header will be returned.

So let's try it out. First I verified that the current configuration does not close the HTTP connection after each request. It does not: it uses HTTP 1.1 and sends no "Connection: close" header. Hitting one of the proxy services multiple times in a row results in a single HTTP connection open.

Next I have added keep-alive="false", redeployed mule and executed the same test. There is nice "keepAlive: false" HTTP header and no "Connection" header. Who said it is going to work the first time?

I hate this. I mean I do everything as documented and it does not work. Given I just started poking around mule I would not immediately suspect the documentation is wrong. So I check and double check, and run more tests. And check again and make sure I looked at the correct documentation since the project is using mule 2.2.1 and the latest version is 3.something. Nope, everything looks correct, it just does not work.

Reading the doc page further I come across section Including Custom Header Properties. The same way of specifying Connection: close is also discussed for example here: Disable HTTP Connection keep-alive in CXF proxy.

Guess what? It does not work either. Just for kicks I have added another custom property "ILikeTheMuleDocumentation: false". This one works. Wow!

Next I tried something else that was mentioned in the same thread Disable HTTP Connection keep-alive in CXF proxy, namely, adding "Connection: close" with a deprecated "http.custom.headers" property. This one finally worked. And indeed produced a deprecation warning in logs.

By this time I became quite curious. I mean one must be quite … hmm ... talented developer to achieve such an interesting result of working and non-working features. So I looked in the sources and, after some digging, I found this:

package org.mule.transport.http.transformers;

...

public class ObjectToHttpClientMethodRequest
...

  protected void setHeaders(HttpMethod httpMethod, MuleMessage msg)
  {
      // Standard requestHeaders
      String headerValue;
      String headerName;

      for (Iterator iterator = msg.getPropertyNames().iterator(); iterator.hasNext();)
      {
          headerName = (String) iterator.next();

          if (headerName.equalsIgnoreCase(HttpConnector.HTTP_CUSTOM_HEADERS_MAP_PROPERTY))
          {
              if (logger.isInfoEnabled())
              {
                  logger.warn("Deprecation warning:  There is not need to set custom headers using: " + HttpConnector.HTTP_CUSTOM_HEADERS_MAP_PROPERTY
                      + " you can now add the properties directly to the outbound endpoint or use the OUTBOUND property scope on the message.");
              }
            
              Map customHeaders = (Map) msg.getProperty(HttpConnector.HTTP_CUSTOM_HEADERS_MAP_PROPERTY);
              if (customHeaders != null)
              { 
                  for (Iterator headerItr = customHeaders.entrySet().iterator(); headerItr.hasNext();)
                  {
                      Map.Entry entry = (Map.Entry) headerItr.next();
                      if (entry.getValue() != null)
                      {
                          httpMethod.addRequestHeader(entry.getKey().toString(), entry.getValue().toString());
                      }
                  }
              }
          }
          else if (HttpConstants.REQUEST_HEADER_NAMES.get(headerName) == null
                 && !HttpConnector.HTTP_INBOUND_PROPERTIES.contains(headerName))
          {
              headerValue = msg.getStringProperty(headerName, null);
              if (headerName.startsWith(MuleProperties.PROPERTY_PREFIX))
              {
                  headerName = new StringBuffer(30).append("X-").append(headerName).toString();
              }

              httpMethod.addRequestHeader(headerName, headerValue);
          }
      }

      Set attNams = msg.getAttachmentNames();
      if (msg.getPayload() instanceof InputStream
              && attNams != null && attNams.size() > 0)
      {
          // must set this for receiver to properly parse attachments
          httpMethod.addRequestHeader(HttpConstants.HEADER_CONTENT_TYPE, "multipart/related");
      }
  }
...

Brilliant, is it not? And, by the way, I did not find any place in the outbound path where 'keep-alive' property is checked. So much for a nice documentation. So I ended up using "http.custom.headers" to set "Connection: close" header.

Well, by that time I was well prepared to investigate the remaining issues with our mule configuration. I was sure I would come across even more pearls.

Oracle BPEL process: how to ask an administrator for a little attention

Well, I have never thought I would be writing anything longer than couple of words about BPEL. I mean I know it, I have to use it (more like: I am forced to use it), but this is a technology so cumbersome to use that I can't imagine anyone besides BPEL software vendors would promote it.

Anyway here we go. Recently I needed to do something uncommon in one of processes I have to maintain. The systems runs under Oracle BPEL Process Manager v. 10.1.3.5 and this software has a module called BPEL Console where an administrator can keep an eye on the running processes. One of the things the BPEL Console allows administrators to perform is recovery of process instances. Depending on a lot of configuration (fault policies) and a process definition the server can mark a running process instance as pending manual recovery and then let administrators inspect the instance's variables, modify them if needed and "recover" the instance, for example, retry the failed operation or just skip it and continue the execution.

Under particular circumstances I wanted to report an error in the process execution that an administrator can fix by changing a variable and let the execution continue. While the Oracle BPEL software has also support for human interaction (and the maintained application is using this as well) I did not want to use it. I really wanted that the problem is reported to BPEL administrators and not business users. There is already a nice place for that in the BPEL Console, I just needed the BPEL process instance to do something so that the server marks the process instance for manual recovery.

Sounds easy. After all, BPEL has that nice <throw> activity to report an error. Except that this does not work the way I need it to work. The only thing I can do with a fault thrown explicitly by a process (using <throw> activity) or generated by the BPEL server due to an error in the process execution is catch it with <catch> or <catchAll>.

Well, I can also do nothing but then the process instance dies with "unhandled fault" error. The fault is then rethrown by the BPEL Server in the parent process instance. Or not: this is actually the place where Oracle BPEL software looks at the configuration and decides what to do with the fault: rethrow it or may be let administrators do something about it.

It looks like I have to create a new process B just to throw a fault and invoke process B from process A so that I can configure Oracle BPEL software to pause process A when process B throws a fault?! Nice. The system has already 70 processes which is actually 70 more than needed.

After reading some Oracle documentation and a bit of experimenting I found a better way. The fault policies mechanism works on <invoke> boundaries. It does not matter what triggers an <invoke> failure: a fault message from the invoked service or just failure to find that service in the first place. The solution I implemented ended up being really simple: I defined a WSDL file with some request/response messages, a port type and a <service> pointing to a non-existing URL. I fill in a request message with error details and try to invoke the service. The BPEL server gets 404 HTTP response and happily fails with {http://schemas.oracle.com/bpel/extension}remoteFault. The only thing left is configure the fault policies so that the process is marked for manual recovery.

This solution has an additional advantage over "2 processes" implementation: it does not produce faulted process instances in the BPEL Console. One thing less for administrators to worry about.