Convention over configuration, maven style

Maven is regarded as a very nice example of convention over configuration. I do not dispute that, it is true ... to some extent, until you try something just a bit out of maven's notion of "convention". And then you have a problem.

I mentioned this in one of the earlier posts:

But maven makes creation of such an EAR hmmm ... interesting. I just hate to think what amount of work I would need to do if I needed some dependencies in EAR/lib and some in WEB-INF/lib and most importantly what amount of work would it be to maintain this structure.

The time has come. I need some jars in WEB-INF/lib (tag libraries) and the rest in <EAR>/lib. So let's look at "convention over configuration" a little bit closer. This is what I have to do:

1. WAR's pom.xml: declare dependencies on external jars and on some other subprojects (couple of EJBs), nothing unusual, should be done anyway. I mention it here for completeness.


2. EAR's pom.xml: again, nothing unusual, declare dependency on EJB en WAR projects.

That is all. Maven does its magic producing correct WAR and EAR files. Convention over configuration does wonders, does it not?

Sorry, got carried away. Ah, dreams, dreams...

Reality is quite different. Sure, if I do what I just mentioned, the resulting WAR file is correct. But EAR is completely screwed up because the WAR is included "as is". Some dependency jars are present in <EAR>/lib and in <WAR>/WEB-INF/lib. All EJB jars are also present twice, in <WAR>/WEB-INF/lib and in the root of EAR.

Maven offers some ... hmmm ... solution for the problem, documented here: "Solving the Skinny Wars problem". They honestly warn you saying:

The Maven WAR and EAR Plugins do not directly support this mode of operation but we can fake it through some POM and configuration magic.

Let's look at their "configuration magic". It is consists of 2 pieces of configuration that has to be applied to WAR and EAR pom files:

1. You have to configure WAR pom.xml to remove jars from WEB-INF/lib and to add their names in MANIFEST.MF. Note that these options are orthogonal and you have to specify both.
   
   And if you need to leave some of the jars in WEB-INF/lib (I need to), then the configuration becomes really crazy. Watch the "POM and configuration magic":
   
   
   
   • option 1:
     

     
         <plugin>
             <artifactId>maven-war-plugin</artifactId>
             <configuration>
                 <packagingExcludes>
                     WEB-INF/lib/C*.jar,
                     WEB-INF/lib/R*.jar,
                     WEB-INF/lib/a*.jar,
                     WEB-INF/lib/b*.jar,
                     WEB-INF/lib/c*.jar,
                     WEB-INF/lib/d*.jar,
                     WEB-INF/lib/e*.jar,
                     WEB-INF/lib/h*.jar,
                     WEB-INF/lib/jb*.jar,
                     WEB-INF/lib/js*.jar,
                     WEB-INF/lib/joda-time-1*.jar,
                     WEB-INF/lib/joda-time-h*.jar,
                     WEB-INF/lib/sl*.jar
                 </packagingExcludes>
                 <archive>
                     <manifest>
                         <addClasspath>true</addClasspath>
                         <classpathPrefix>lib/</classpathPrefix>
                     </manifest>
                 </archive>
             </configuration>
         </plugin>

   • option 2:
     

     
         <plugin>
             <artifactId>maven-war-plugin</artifactId>
             <configuration>
                 <packagingIncludes>
                     WEB-INF/lib/standard-1.1.2.jar,
                     WEB-INF/lib/joda-time-jsptags-1.0.2.jar,
                     **/*.xml,
                     public/**/*.*,
                     images/**/*.*,
                     css/**/*.*
                     static/**/*.*
                 </packagingIncludes>
                 <archive>
                     <manifest>
                         <addClasspath>true</addClasspath>
                         <classpathPrefix>lib/</classpathPrefix>
                     </manifest>
                 </archive>
             </configuration>
         </plugin>

   At one time I also thought of extracting TLD files from tag libraries, placing them under WEB-INF, and still removing all jar files from WEB-INF/lib. But I decided against it.
   
   No matter how you look at the problem it is not really "convention". Hell, it is "configuration, configuration, configuration, and even more configuration". And, worst of all, the resulting WAR file cannot be deployed separately, only as part of an EAR. Oops, here go all your nice integration tests. Fortunately it is not a problem for our current project because we do not deploy WAR files separately. But I can imagine some other people would like to do it.
   
   This "solution" has a minor problem that the resulting MANIFEST.MF is also incorrect: it has all jars mentioned in Class-Path: attribute, even those that stay in WAR/WEB-INF/lib.
   
   Oh yeah, and some annoyance: I have to specify exclusion by file name and not by dependency. I have only 4 dependencies in WAR's pom.xml and look at that exclusion list!


2. But it is not all! More "configuration magic": you also have to change EAR's pom.xml and add all WAR-only dependencies that are not included in <WAR>/WEB-INF/lib. Otherwise they are not packaged into EAR/lib. Lucky you, it is WAR-only dependencies, and not dependencies of EJBs.

Wow, finally you get the packaging you want. Now you have to make sure everybody on the team knows that updating projects' dependencies (not only WAR project but also all EJB projects) or adding some directories under src/main/webapp is not that easy as it should be. Because it will sure break WAR packaging.

This is of course not new. The related bug MWAR-9 has just turned 5 years. Congratulations, maven! Happy anniversary, MWAR-9!

This post was brought to you by maven notion of convention over configuration.

Multiple Hibernate configuration files

I must be living in a stone age, at least, with regard to Java persistency. Recently I had to add another EJB module to an application that is using Hibernate as its persistency. The application uses a much recommended HibernateUtil helper class to start up and access the org.hibernate.SessionFactory. It is almost a copy-paste from the Hibernate tutorial.

In the newly added module I want to add some HQL queries and possibly some more mapped classes. I do not want to change the existing configuration file because the new module is optional. Besides, the existing configuration lives in a different maven module. So adding new queries is possible, albeit not "clean", but adding new mapped classes is out of the question. I have to have a second configuration file but I want it to be available via the same HibernateUtil class. I do not want to copy-paste it into my module.

Quick googling shows that I am not alone but proposed solutions (at least linked from the first couple of result pages) are not really satisfying. Duplicating HibernateUtil was there as well as using org.hibernate.cfg.Configuration API. Or using a system property that defines which file to load. Yeah, this would work, except that I did not like it either: I would need to define the property in multiple places (unit tests or appserver deployment) depending on which modules are packaged. Of course, at the moment the application has max. 2 modules with a configuration file, so I can rely on a "default property value" when the new EJB module is not deployed and set the property explicitly only when the new EJB module is deployed. Will it work? Yes. But for the rest: no thanks.

Instead I have changed HibernateUtil to have more or less the following:

   private static SessionFactory buildSessionFactory() {
       Configuration cfg = new Configuration();

       try {
           ClassLoader ldr = Thread.currentThread().getContextClassLoader();
           Enumeration en = ldr.getResources("hibernate.cfg.xml");
           Set processed = new HashSet();

           while (en.hasMoreElements()) {
               URL next = en.nextElement();

               if (processed.add(next)) {
                   cfg.configure(next);
               }
           }
           if (processed.isEmpty()) {
               FAIL
           }

           return cfg.buildSessionFactory();
        }
        catch (Exception ex) {
           FAIL
        }
    }

I do not need to use any system properties. The code just "discovers" the configuration files and works correctly in unit tests and in the real container, with the new EJB module or without it.

Custom Hibernate configuration for the JBPM console

That Hibernate configuration exception still bothered me, so I looked into it as well.

It basically says that the JBPM console can find the application's Hibernate configuration file but fails to create the Hibernate Configuration object out of the file. The reason: the application is using Hibernate Annotations with corresponding entries in the config file, but the JBPM console tries to read the file as a plain Hibernate configuration (with xml mapping files). No surprise this fails.

I looked at the application code to see how the configuration file is read and how the Configuration object is used. The application has some utility class to load the configuration as Hibernate Annotation configuration. Each time after the application asks JBPM for a new JbpmContext it does jbpmContext.setSessionFactory(Util.getHibernateSessionFactory()). The JBPM console uses the Configuration the same way (jbpmContext.setSessionFactory), it just loads it differently.

No, I do not want to change the code of the JBPM console. I need to find a different way to make sure the JBPM console can find the correct SessionFactory object. No, actually I do not care about the console: it just passes the SessionFactory to JBPM. I need to make sure JBPM sees the correct SessionFactory object. What choices do I have? Option 1: register the Hibernate SessionFactory in JNDI, built-in Hibernate functionality. Option 2: there is a possibility to replace the way the JBPM configuration file is loaded so it is possible to create a custom JbpmContext, or, in my case, just a standard JbpmContext, but with the correct Hibernate SessionFactory.

Some experiments proved that it is not that easy. Registering a SessionFactory in JNDI does not work out of the box because both OpenEJB and OC4J treat the JNDI name as a local name, and the SessionFactory registered in one place is not visible in the console. And the reading of the JBPM configuration is "all or nothing" solution: there is just no way to get the created JbpmContext right after it is created without duplicating a lot of the existing JBPM code.

I think I would eventually make either of the solution work but I got distracted. Not for the first time a much better solution just came to me while I was busy with some other things.

JbpmContext itself does not do anything with SessionFactory: the only thing JbpmContext.setSessionFactory() does is passing the SessionFactory down to PersistencyService. And PersistencyService is configured in jbpm.cfg.xml (org.jbpm.persistence.db.DbPersistenceServiceFactory in our case). The solution: create the subclass of DbPersistenceServiceFactory that does just one thing: passes the application's SessionFactory to the superclass, the rest is up to the superclass. With the subclass registered in jbpm.cfg.xml the console does not give the Hibernate configuration errors any more; instead it nicely displays our processes.

The solution also makes all those jbpmContext.setSessionFactory(Util.getHibernateSessionFactory()) in the application code redundant, so I deleted them: less code less potential problems.

Deploying JBPM Console in OC4J

I actually enjoy that kind of work: usually it is short enough for me not to get bored, and I get a chance to try different things and do something I have not yet done. Sure I have to deal quite often with some outdated or obscure technology, but I do not mind. So I took up another challenge related to the same technologies: OC4J, JBPM and (this time) JSF: integrating JBPM JSF console into the application and deploying all that as a single EAR.

The team has managed to get the JBPM console WAR up and running in OC4J 10.1.3 as a standalone application but they wanted to integrate the JBPM console jar (and not the WAR) in their application. They were planning to have a somewhat different web interface than the JBPM console WAR provided and to reuse the functionality of classes in the jar. But they failed citing all kinds of incompatibilities between OC4J, JBMP and JSF. That sounded strange because they managed to overcome the only "incompatibility" I was aware of: the JBPM JSF console uses JSF 1.2 RI, and OC4J 10.1.3 does not support JSF 1.2, only JSF 1.1. (This is not actually an incompatibility from my point of view: the real incompatibility is between the required JSP version for JSF 1.2 and the version provided by the OC4J. But the JBPM console uses facelets and not JSPs.)

First I repeated the exercise of deploying JBPM console WAR in OC4J: I needed a reference point to say, OK, this works. The WAR file could not be deployed as is (it relies on some jars that are not in its WEB-INF/lib), so I had to add those jar. I also had to change some configuration files: hibernate to point to the project's database, and web.xml to get rid of all security configuration (security was not crucial at that moment, and it can be configured correctly later). In addition I had to add couple of project jars (EJBs) to WEB-INF/lib: the existing processes refer to some project classes, and the JBPM console gave some ClassNotFoundExceptions. Within 30 minutes I got the console deployed and running.

Next step: adding JBPM console jar to the project's pom.xml files. This turns to be interesting, mostly because of what the team had achieved thus far. For example, one of the WARs had more than 90 jars in WEB-INF/lib, some of them even twice, with different versions. And ant.jar? Or junit.jar? Who are those people? Er..., I mean, why do you need them at runtime? And myfaces 1.1? No wonder they could not get a JSF 1.2 stuff working. Time to do some cleanup... Why oh why some projects just can do it right? Why do I need all that apache jackrabbit stuff if JBPM has (but does not use) couple of classes that depend on it? Or just because JBPM has couple of ant tasks I need ant and all its dependencies in my WEB-INF/lib?

Right, 23 jars left, time to add JBPM console jar; copy all those facelets pages from the original JBPM console WAR and change web.xml. Package, deploy, ... wow, no errors? Now, where did I see that before? The application itself works and does exactly what it did before adding the JBPM console, but each page of the console I tried shows a nice JSF exception saying more or less "JSF is not initialized, can't find JSF factories".

OK, the JBPM console WAR deployed as a separate application works. The same code, just packaged differently, does not. Find 5 differences? Well, there are more, but two important are: WAR vs. EAR deployment, and "required jars in WEB-INF/lib of the WAR" vs. "skinny WARs in EAR".

Being lazy I did not want to make an EAR out of WAR. I also knew that OC4J makes an EAR out of WAR during the deployment. So I took that EAR and deployed it separately, and it worked. The first difference is ruled out.

On to the "skinny" WAR. All the EJB jars and web WARs that ended up in EAR have a lot of common dependencies, so it is only logical to have all of those dependencies in EAR/lib. It is also more "spec compliant". But maven makes creation of such an EAR hmmm ... interesting. I just hate to think what amount of work I would need to do if I needed some dependencies in EAR/lib and some in WEB-INF/lib and most importantly what amount of work would it be to maintain this structure. Thanks, maven.

Back to the JSF problem: I manually added all the jars from EAR/lib into the WAR with the JBPM console under WEB-INF/lib and then repackaged the EAR. Deploy, no errors, page access ... wow, I can see it. The page displays some error message about Hibernate configuration, but at least the menu and some other elements of the console are also displayed. OK, so the "skinny" WAR is the problem. But why?

I should say debugging the problem was quite an interesting process. Staring with just being able to start an OC4J server with the application under debugger from Eclipse WST on to all that package, undeploy, redeploy, deploy the JBPM console WAR, deploy the application EAR, scratch the head, "this just can't be", and so on. The end result is:

1. FacesServlet uses some factory implementations and complains if they are not found. The factory discovery process looks at several places and in case of JSF RI it finds only those factories that were already registered before the lookup.

2. The JSF implementation (JSF RI) registers the factories at startup with the help of some javax.servlet.ServletContextListener. This listener is invoked if JSF RI jars are placed under WEB-INF/lib and not invoked in case of "skinny" WAR.

3. The listener is not configured in any web.xml, and that was the reason for "this just can't be": the listener is invoked but it is not registered!

4. After some search I have found the listener in JSF RI.jar/META-INF/jsf_core.tld. OK, so it is also possible to add listeners to TLD descriptors (as of JSP 1.2)?! I did not know that. And it looks like OC4J looks for TLD only in jars from WEB-INF/lib, and not in classpath. And as usual with specifications the JSP 1.2 spec is not really clear here:

Tag libraries that have been delivered in the standard JAR format can be dropped directly into WEB-INF/lib.

J2EE 1.4 tutorial says the following:

Tag files can be packaged in the /META-INF/tags/ directory in a JAR file installed in the /WEB-INF/lib/ directory of the web application.

A lot of "cans" and no "must"/"must not" or "should"/"should not".

5. Anyway, the application must be deployed under OC4J and the rest is just theorization. The solution I chose is simple: all the listeners mentioned in JSF RI.jar/META-INF/jsf_core.tld are added to web.xml. Copy-paste, package ("skinny" WAR!), deploy, browser, open the page, and bingo! The menu and the frame of the console are there. And that other error message from Hibernate as well.

The solution has some pros and cons, as usual.
1. Pro: it works.
2. Pro: OC4J does not read unnecessary JSF JSP TLD files: the application does not use JSF JSP.
3. Pro: OC4J stopped complaining about an unsupported JSP TLD version.
4. Con: this is something that should be remembered and documented. If for any reason (migration to a different server, changes to packaging, etc.) the same javax.servlet.ServletContextListener is found registered twice the application startup would fail.

So that is it: I am done with the boring stuff and all the exiting things are left as exercise to the team.

Deploying JBPM Enterprise in OC4J, part 2: OC4J all other the place

To continue where I left:

1. Before deploying in OC4J I decided to get rid of semi-manual repackaging: I suspected I would need to do it a lot because of OC4J specific deployment descriptor. I succeeded. It is not an interesting subject, it is not the topic of this post, and I probably would not make many friends in maven or m2eclipse community if I say aloud what I think, so I say little: it is definitely not the primary task for maven.

2. Honestly, I do not know how many more times I had to change ejb-jar.xml before OC4J stopped complaining about JMS and started complaining about other things. Even more: some of those changes broke unit tests so I had to do even more changes to compensate for that. Eventually OC4J gave up: no more JMS errors during deployment.

3. Now I have TopLink errors (TopLink classes are not on the classpath). Remember I said the project was using Hibernate native API? The team had some problems making it work in OC4J with TopLink enabled, so they switched TopLink off for their application. I do not know how real the problems were, but this was quite a reasonable thing to do: your application is not using JPA, so you do not need support libraries for that, so you instruct OC4J to remove them from classpath.

But it turns out that OC4J is using TopLink for their entity bean support. And jbpm-enterprise has an entity bean. WTF? You are using Hibernate for all your persistency, "the best ORM in the world" (YMMV), and all of the sudden you need an entity bean?! OK, I have seen that bean, the code, and the deployment descriptor entries while I was busy with JMS configuration, so I knew it is there, and it did not give any troubles in openejb, so I did not worry about it...

Payback time. And more things to learn: I have never used the EJB timer service before, and the sole purpose of this entity bean is to work with the EJB timer service. It took me some reading to understand why jbpm-enterprise has two beans (an SLSB and this entity bean) to work with the EJB timer service, why the SLSB is deprecated, and what the reason for that deprecation is. I should say it is a clever trick, completely spec compliant and that I would probably never think of something like this. To introduce an entity bean, the only entity bean in the whole project, as a solution to a performance problem... Of course, if it is the only solution I would not say a word.

But within 5 minutes after I understood the reason why the entity bean is there I came up with another solution to the same problem that did not require any entity beans. Theoretically that should work, I did not have time to try it, but my guess I will get a chance in the near future. Anyway, at that moment the application did not use jbpm timers, so I went back to the deployment descriptor and now I had one bean less to deploy. Full build cycle, no unit test failures, deploy, no errors, success.

4. Almost all the time while working on this jbpm-enterprise I was thinking: drop it, it will not work. Do you ever get a feeling "one last step, and you are there"? Probably because of this feeling I carried on. And the feeling paid off, except I did not believe it. Successful deployment does not mean no errors at run time.

So I fired manual test scenario and got a big hello from OC4J JMS Database Persistence again. Very big. Stacktraces dozens of lines long. The exception was coming from JMS session.close() in org.jbpm.msg.jms.JmsMessageService. It said basically "rollback is not allowed in this context". Huh? Sequel "WTF comes back and strikes again". It is the same WTF I have mentioned before: javax.jms.Connection.createSession() parameters, nicely commented in class org.jbpm.msg.jms.JmsMessageService:

/*
* If the connection supports XA, the session will always take part in the global transaction.
* Otherwise the first parameter specifies whether message productions and consumptions
* are part of a single transaction (TRUE) or performed immediately (FALSE).
* Messages are never meant to be received before the database transaction commits,
* hence the transacted is preferable.
*/
session = connection.createSession(true,
Session.SESSION_TRANSACTED);

When I was running my test application with JMS Database Persistence implementation, I ignored all the exceptions from xyz.close() calls. Well, it was a testing application, so why bother. Big mistake. I revisited my test application and put some logging in catch, and sure, closing a session created with createSession(true, ...) would throw the same exception while closing a session created with createSession(false, ...) would not.

The strange thing: even if the exception was thrown, everything would work without problems if the exception was just ignored. Messages were sent and successfully received, or not, depending on what was going on in the sending method (nothing/EJBContext.setRollbackOnly()/throw (un-)checked exception).

5. I went as far as looked at Oracle classes under class disassembler. And sure enough closing a transacted session would trigger a rollback. This was also the explanation why having a transacted session with In-Memory JMS resulted in messages being lost: rollback attempt. Except that with In-Memory JMS session.close() would actually rollback the JMS transaction and succeed, and with JMS Database Persistence rollback would fail, and if that exception is caught, the transaction would be committed by the container. But the exception is reported and rethrown in jbpm-enterprise.

6. This was the last straw. I have actually built a modified jbpm-enterprise with changed parameters to createSession(), and satisfied my feeling of 'being there'. Everything worked. And then I sent an email to the team members saying "forget it, it is just not worth the trouble".

Deploying JBPM Enterprise in OC4J, part 1: not yet OC4J specific

I just can't get away from this OC4J and JBPM. After I have solved the problem with JMS transactions the team asked me to do another "small" thing: include jbpm-enterprise.jar in the project instead of copy-pasted code. The idea is nice and sound of course: after all, the only reason for that copy-pasted code was desire to get the project going in a short amount of time and to avoid that JMS transaction problem... Well, let's say I believed them.

After spending quite some time on it, first with another colleague and then on my own, I came to the conclusion: forget it, it is just not worth the trouble. (This is applicable to JBPM 3.3.1 and OC4J 10.1.3; the situation might be better with newer versions.)

This post is a bit more practical than the previous one because there might be somebody else out there trying to get it working in some other app server. Other than JBoss that is.

Here is the list of things I came across trying to make that work. Some of them are generic and some are OC4J specific, and the advice "forget it" is applicable to OC4J. Other app servers might be a bit easier as targets.

1. Of course, the first thing was to get rid of copy-paste code, add jbpm-enterprise.jar as a maven dependency, change jbpm.cfg.xml to point to relevant classes from jbpm-enterprise, and fire 'mvn clean install'. Well, that was easy. It compiles? Wow, we are almost there... Or not, because unit tests fail. Openejb does not like jbpm-enterprise, namely, its MDBs (there are two of them, JobListenerBean that extends CommandListenerBean). The error message is pretty clear:

When annotating a bean class as @MessageDriven without declaring messageListenerInterface, the bean must implement exactly one interface, no more and no less. beanClass=org.jbpm.ejb.impl. JobListenerBean

Yeah, sure, 'CommandListenerBean implements MessageListener' and 'JobListenerBean extends CommandListenerBean'. Except that the beans do not use any annotations. jbpm-enterprise uses a deployment descriptor instead. Annoying... the error message could have been better. Apparently, openejb does not distinguish (at least here) between annotations and deployment descriptor because changing the descriptor was enough to fix the problem (adding javax.jms.MessageListener to both beans). But this means that jbpm-enterprise can't be used as is and has to be repackaged. Not a big problem: we would need to repackage it anyway because it needs an OC4J specific deployment descriptor. The "right" solution is ... would be ... found later, we are just not in the mood for all this maven stuff now, let alone licensing. For now: repackage it manually, and place it back into the local maven repository. At least we can continue.

2. Tests still fail, except this time with javax.naming.NamingException. All the joys of server independent code, configuration in deployment descriptors, etc, etc, and so on were destroyed with one line:

String connectionFactoryJndiName = "java:JmsXA";

Nice, is it not? JBoss specific JNDI names, hardcoded in java code. Those guys know a thing or two about portable applications, don't they? OK, changing a deployment descriptor is one thing, but changing the code is too far. Fortunately those factories in JBPM are pluggable, so we just have to implement our own. Done in no time, subclass JmsMessageServiceFactory, override public ConnectionFactory getConnectionFactory(), except that... what JNDI name should we use? No so difficult to answer thought we, the deployment descriptor uses jms/JbpmConnectionFactory, so we use it as well.

3. I spare you... res-ref-names, openejb specific names, attempts to define openejb connection factories, ... nothing worked. I think that we would have been able to debug the problem eventually, but pure by chance we found it. All that fuss just because of a one letter! Meet the hero: ejb-jar.xml, line 79, javax.jms.ConnnectionFactory

4. Repackage, compile, test. Well, I should have guessed. java:JmsXA was not the only one. Queue names in JmsMessageServiceFactory.java are also JBoss specific. Let's call it a day for today.

Anyway, as of that moment I continued to work on the problem alone.

Sime time later:

5. Using res-ref-names and message-destination-ref-name from the deployment descriptor in my subclass of JmsMessageServiceFactory did not work. I had to use openejb specific JNDI names, and I did not like it at all. Well, at least most of unit tests are green. Only one is still failing.

6. I decided to get rid of the server specific JNDI names first. There were several possibilities, but most of them would require specifying some additional things in the deployment descriptor for each enterprise bean that ends up invoking JmsMessageServiceFactory, including any bean that would be added later. I did not like that, so I just created a single SLSB with the correct @Resource annotations and then used it from my JmsMessageServiceFactory implementation. At least I have to configure one bean correctly and never worry about any other beans. Test again, and the same test as before is failing, with the same message. It is a good sign. Really.

7. So what is interesting about that test? It is the only test so far that not only uses JBPM functionality to send JMS messages but also depends on it. There are some tests that have the necessary jbpm woo-doo to send JMS messages, but those tests never check results of JMS operations. There are other tests relying on JMS, but they are in another, dependent, maven project, and I am not yet that far. Some more logging-debugging, and the result: openejb uses different queues in SLSB and in MDBs (I should have realize it sooner). OK, some more changes to the deployment descriptor of jbpm-enterprise, package (aren't that tiring?), copy, compile, test... Success, everything is green.

Conclusion: if you need to deploy jbpm-enterprise in openejb (-derived) server, it can be done if you are prepared to edit the deployment descriptor and add couple of java classes.

But I need to deploy it in OC4J. To be continued...

Interesting JMS behavior in OC4J

Recently I was asked to help a project developed by colleagues with couple of problems they were having with JMS and transactions.

The project is basically a J2EE 1.5 (as much as possible) application that has to run in Oracle Application Server 10g Release 3 (OC4J 10.1.3). The app server version was a hard requirement so the team could not just ditch it and use something better or modern. The application uses JBoss jBPM 3.3.1.GA and because of this has to use Hibernate native API instead of JPA. They also wanted to use jbpm-enterprise.jar but did not manage to do it on time for some demos. Instead they reimplemented part of the functionality from jbpm-enterprise.jar.

One of the things they needed from jbpm-enterprise was usage of JMS for asynchronous messaging and transaction demarcation. The setup is quite simple: the implementation of org.jbpm.msg.MessageService uses stateless session bean to save data (org.jbpm.job.Job) in the database and to send a JMS message with jobId in it. There is also an MDB that receives the message, reads the Job and data executes it.

The problem was: sometimes, when running the application under OC4J, the MDB would receive a message and try to find a job based on the jobId from the message but it looked like the job was not in the database. But of course the job was there when you look in the database in Toad or SQL*Plus. The problem manifested itself randomly and only under OC4J, and never in their unit tests that were successfully running in openejb (the tests covered the failing code path as well).

The team realized they have a transaction problem and even went as far as creating unit tests to test transactional behavior with JMS. This I found funny: testing something not in the environment where it fails seemed useless to me.

They also came up with a solution: they have added a call to HibernateSession.flush() to SLSB right after HibernateSession.saveOrUpdate(job). The team members claimed it fixed their problem but felt it was a dirty hack, and because of that they needed to find a better solution. So they asked me to look at the problem.

Well, first of all, I would not call HibernateSession.flush() a dirty hack: if it really fixes the problem so be it. The thing is: I was surprised to hear it helped. It should not have worked unless they have done something very special with Hibernate sessions and transaction management. Well, they have not, so the very first thing I did was a small demonstration: adding Thread.sleep(100) right after JMS send resulted in the same error in the MDB. No (reasonable) delay with Thread.sleep() right after flush() solved the problem. OK, I was right, their solution was not a solution at all. The only possible explanation of why this worked I could come up with was a race condition: flush() does all the heavy lifting with SQL so it could just happen that the database transaction is committed just on time before MDB onMessage() is invoked. Without flush() Hibernate must perform all the hard work on session close, and it takes much more time than just commit.

Back to the problem. I did not feel like investigating the problem in the context of their project: it is a multi-module (2 ejb jars, 2 wars, packaged as an ear) maven project with some "features" on maven side, which resulted in a lot of wasted time to build, package and deploy the application. I have created a couple of simple projects to replicate the problem. Actually, I did not include any persistency at all; it was pure 'JMS send' in SLSB and an MDB. I was able to reproduce the problem immediately. Debugging and logging proved that everything worked without a hitch under openejb, but under OC4J the MDB's onMessage() is invoked before the SLSB's sending method completes. I also got quite a WTF feeling along the way when I experimented with javax.jms.Connection.createSession() parameters, but the real meaning of the WTF hit me much later. Another interesting thing: EJBContext.setRollbackOnly() in SLSB did not work either.

Googling did not really help: there were posts describing what could be similar or related problems, but no proposed solutions.

On top of that OC4J has 2 JMS implementation and the fact that Oracle renamed them along the way did not help either. I should probably have mentioned that the project used In-Memory JMS implementation as it was the easiest to configure. Switching to File-Based JMS implementation did not change a thing. There is also JMS Database Persistence implementation, but my initial attempts to configure it failed, so I stopped trying.

It took me quite some debugging and googling to find the solution. Actually, Oracle Containers for J2EE Services Guide, chapter 4, Using Oracle Enterprise Messaging Service (http://download.oracle.com/docs/cd/B32110_01/web.1013/b28958/jms.htm#g1088175), mentions it: setting oc4j.jms.pseudoTransactionEnlistment property in j2ee/home/config/jms.xml "solves" the problem. But who reads the documentation?! (I did, I just skipped most of it.)

Pseudo ... transaction ... enlistment ... sounds scary, but the solution worked for both In-Memory and File-Based JMS implementations and I did not care about the warnings about future releases: OC4J is being replaced with WebLogic anyway.

There are several things I still do not understand about all this, and this makes me feel I am joining the ranks of cargo cult programmers:

1. This should "just work". All the info I was able to find, including specs, tutorials and examples have similar code. And this works in openejb.

2. Apparently this was supported in earlier versions of OC4J but somebody decided that what appears to be a standard behavior was not correct?

3. During my debugging, before I found the solution, I tried to use XA variants of JMS objects, but got errors from OC4J either during application deployment (MDB configuration) or during runtime (SLSB JMS send code) saying that I can't use XA objects there. OK, that might just have been errors in JMS configurations I set up.

4. I also tried to use JMS local transactions in SLSB, with and without explicit rollback and commit. The local transactions were actually a reason for the WTF I mentioned above: if a JMS session is transacted, no messages are delivered to the MDB without explicit commit, but explicit commit/rollback caused OC4J to complain saying that commit/rollback are not allowed in that context. This was also my understanding of using JMS inside an enterprise bean. The parameters to createSession did not matter in openejb. The Java EE 5 Tutorial (and 1.4 as well), chapter "Using the JMS API in a Java EE Application"(http://download.oracle.com/docs/cd/E17477_01/javaee/5/tutorial/doc/bncgl.html) says (emphasis mine):

When you create a session in an enterprise bean, the container ignores the arguments you specify, because it manages all transactional properties for enterprise beans. It is still a good idea to specify arguments of true and 0 to the createSession method to make this situation clear:

session = connection.createSession(true, 0);

Apparently it matters to OC4J. Well, I would not cry when this thing is finally dead.

After I have found a solution for the problem, I decided to look again at configuring the application to use JMS Database Persistence. This would be a different, even better, solution: the project relied on Oracle database anyway, so why not?

I should say this thing is not that easy to configure the first time you try it. Every problem can be solved with another level of indirection? Ha, just imagine how many problems you can get with another level of indirection. Oracle's JMS support is a good example of this: there are "administered objects" and real objects and they all (or not - it depends) are visible in JNDI. And you are not allowed to mix them; you are supposed to use only "administered objects", which is probably a right idea. Except that in case of In-Memory and File-Based JMS you are using real objects most of the time; you can use administered objects as well but real objects are much easier to configure.

And then making sure your application runs successfully under OC4J with JMS Database Persistence and under openejb with ActiveMQ requires some work on deployment descriptors; you can't get away with annotations only. But in the end it works, and most of the time it is the only thing (well, the main one) that matters, right? A nice side effect: WTF is gone, it does not matter what parameters I pass into javax.jms.Connection.createSession().