Standalone HornetQ and JBoss, part 2: making it work without embedded server

Rant mode off.

Actually it is not that bad. It is just so happened that HornetQ standalone ended up being last on the list of things I have tried. And it became the last straw.

In reality the biggest single WTF was not HornetQ but ActiveMQ acknowledging received messages despite timed out transaction. The biggest collection of WTFs in a single place was of course (and is, and probably will be forever) OracleAQ.

But back to configuring JBoss AS 6 to work with the standalone HornetQ.

1. The most important thing is to use JBoss AS 6.1.0.Final and not 6.0. Details are here. This is something I already knew and it is not specific to the standalone configuration of HornetQ, but it is worth mentioning here anyway.


2. JBoss will be configured to talk to the external standalone HornetQ. The embedded HornetQ will be removed. This is what I want to achieve. Yes, it is possible to go for the standalone HornetQ environment and still run the embedded HornetQ just in case. But I find it pointless. Note also that no new resource adapter is added to JBoss. Instead the configuration of the existing one is changed.


3. Move JBOSS_HOME/server/<servername>/deploy/hornetq/jms-ds.xml to JBOSS_HOME/server/<servername>/deploy. This is where outbound connection factories are defined. The file will have to be changed, see below.


4. Delete JBOSS_HOME/server/<servername>/deploy/hornetq directory. This is where the configuration of the embedded HornetQ lives.


5. Delete JBOSS_HOME/server/<servername>/deployers/hornetq-deployers-jboss-beans.xml. This file specifies how HornetQ specific configuration files should be read to initialize embedded HornetQ server and its destination objects. No embedded HornetQ means no need to read these configuration files.


6. Change the resource adapter configuration file: JBOSS_HOME/server/<servername>/deploy/jms-ra.rar/META-INF/ra.xml. This is actually documented here: 32.4.4.1.1. Configuring the Incoming Adaptor or here: HORNETQ_HOME/examples/javaee/jca-remote/readme.html, section "configuring the incoming Adapter". Basically you need to change two RA properties:

   <config-property>
     <description>
       The transport type. Multiple connectors can be configured by using a comma separated list,
       i.e. org.hornetq.core.remoting.impl.invm.InVMConnectorFactory,org.hornetq.core.remoting.impl.invm.InVMConnectorFactory.
     </description>
     <config-property-name>ConnectorClassName</config-property-name>
     <config-property-type>java.lang.String</config-property-type>
   <!--  <config-property-value>org.hornetq.core.remoting.impl.invm.InVMConnectorFactory</config-property-value> -->
     <config-property-value>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</config-property-value>
   </config-property>
   <config-property>
     <description>The transport configuration. These values must be in the form of key=val;key=val;,
       if multiple connectors are used then each set must be separated by a comma i.e. host=host1;port=5445,host=host2;port=5446.
       Each set of params maps to the connector classname specified.
     </description>
     <config-property-name>ConnectionParameters</config-property-name>
     <config-property-type>java.lang.String</config-property-type>
   <!--  <config-property-value>server-id=0</config-property-value> -->
     <config-property-value>host=127.0.0.1;port=5445</config-property-value>
   </config-property>

   The incoming messages will be received from the standalone HornetQ server available at localhost:5445.


7. Change the connection factory configuration file (jms-ds.xml copied to deploy directory at step 3 above). It is also documented 32.4.4.1.2. Configuring the outgoing adaptor or here: HORNETQ_HOME/examples/javaee/jca-remote/readme.html, section "configuring the outgoing Adapter". This is what the resulting file looks like:

   <connection-factories>
     <tx-connection-factory>
       <jndi-name>jms/HornetQ/XACF</jndi-name>
       <xa-transaction/>
       <rar-name>jms-ra.rar</rar-name>
       <connection-definition>org.hornetq.ra.HornetQRAConnectionFactory</connection-definition>
       <config-property name="SessionDefaultType" type="java.lang.String">javax.jms.Topic</config-property>
       <config-property name="ConnectorClassName" type="java.lang.String">org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</config-property>
       <config-property name="ConnectionParameters" type="java.lang.String">host=127.0.0.1;port=5445</config-property>
       <max-pool-size>20</max-pool-size>
     </tx-connection-factory>
   </connection-factories>

The situation now is still not complete because no destination objects are registered in JBoss JNDI. JBoss can be started but deploying an application that uses @Resource will still fail. The good thing: the embedded HornetQ is out of picture.

The remaining piece of configuration depends actually on how the application uses JMS. If the application contains some MDBs to receive messages from destinations, then it is possible to skip JNDI registration completely. Whether it is a good idea is another question. Just add something like this to the MDB code or the equivalent configuration to its deployment descriptor:

@MessageDriven(activationConfig = {
    @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),
    @ActivationConfigProperty(propertyName = "destination", propertyValue = "TestQueue"),
    @ActivationConfigProperty(propertyName = "UseJNDI", propertyValue = "false"),
})

The key here is the property named "UseJNDI". The default value is true and the destination has to be registered in the JNDI. Setting the property to false skips JNDI lookup. After all, JNDI registration is just a bit more than a mapping from say "jms/hq/AQueue" JNDI name to the queue name ("TestQueue") known to the server. The HornetQ resource adapter will use "TestQueue" as the queue name in its communication with the server.

If the application needs that kind of indirection or if it sends messages and relies on @Resource injection line that:

@Resource(mappedName = "jms/hq/AQueue")
private Queue queue;

then some more configuration is needed. This configuration can be done at the server level so the destinations are available to all applications as soon as the server is started or at the application level. Note: I did not actually test the application level configuration. There might be initialization order problems: JNDI name must be available for injection but it is not yet there.

This is what is missing in HornetQ documentation. This and also the way to make it easier and less verbose:

Create a file named jboss-beans.xml in META-INF (or WEB-INF) directory of your application if you want to deploy the configuration with your application. Or create a file named <something meaningful>- jboss-beans.xml in JBOSS_HOME/server/<servername>/deploy. Copy the following to the file:

<?xml version="1.0" encoding="UTF-8"?>
<deployment xmlns="urn:jboss:bean-deployer:2.0">
  <bean name="HQ_JNDIBindingService" class="org.jboss.naming.JNDIBindingService">
    <property name="bindings">
      <bean class="org.jboss.naming.JNDIBindings">
        <property name="bindings">
          <array elementClass="org.jboss.naming.JNDIBinding">
            <bean class="org.jboss.naming.JNDIBinding">
              <property name="name">jms/hq/AQueue</property>
              <property name="value">
                <!-- Use org.hornetq.jms.client.HornetQTopic for topic -->                                
                <bean class="org.hornetq.jms.client.HornetQQueue">
                  <constructor>
                    <parameter>TestQueue</parameter>
                  </constructor>
                </bean>
              </property>
            </bean>

            <!-- You can add here more <bean class="org.jboss.naming.JNDIBinding"> definitions for other destinations -->

          </array>
        </property>
      </bean>
    </property>
  </bean>
</deployment>

This configuration creates a JNDI entry under name "jms/hq/AQueue" and makes sure that is it the correct class (org.hornetq.jms.client.HornetQQueue which implements javax.jms.Queue). This entry represents a queue named "TestQueue" on a server. Note: "on a server". As I have mentioned in the previous post this object is just a mapping to the destination name and it comes alive only with the correct ConnectionFactory/ConnectorFactory.

That is it. Hey, HornetQ, was it difficult to document?

Standalone HornetQ and JBoss

For the last couple of weeks I have been busy playing with JBoss and various JMS implementations, like Apache ActiveMQ, HornetQ and even OracleAQ. As usual, with different degree of success. This post is about trying to make HornetQ standalone work with JBoss.

HornetQ comes bundled with JBoss AS 6 as its default JMS provider. As such HornetQ runs embedded in JBoss JVM process. HornetQ home page mentions a lot of nice feature HornetQ has, for example "Solid high availability" or "Flexible clustering". These feature are available only when HornetQ runs as standalone server.

Shall we try it? No problem, HornetQ is downloaded and started. The minor inconvenience is the lack of any admin user interface. All the configuration must be done via config files or via JMX. Fortunately I need just a single queue for testing purposes. TestQueue is created, HornetQ server is restarted. So far so good.

Now it is time to configure JBoss side. I must say this is ... interesting. Very interesting.

Rant mode on.

It is unbelievable: HornetQ comes from the same company as JBoss AS so it must be easy, right? It is even documented, there are samples, and yet...

First of all there is no documentation on how to configure JBoss AS 6 to completely switch from the embedded HornetQ configuration to standalone. The available documentation describes only how to connect to the standalone server. As if it were enough. The documentation says nothing about the rest of HornetQ configuration files in JBoss.

Guess what, it is not enough. I tried to deploy my tests application and immediately got complaints from JBoss saying that my nice TestQueue is not there.

Nowhere could I find any description on how to make a queue configured in the standalone HornetQ server be visible in the JBoss JNDI tree so that @Resource injection worked.

Next thing I tried was adding my queue in JBOSS_HOME/server/<servername>/deploy/hornetq/hornetq-jms.xml. This looked like a very strange thing to do, but this actually worked. Wow, HornetQ is smart! The corresponding JNDI entry is created, the application is deployed and I can send messages and receive them with an MDB.

There are even couple of JMX mbeans corresponding to my test queue available in JBoss JMX console. Except the attributes of those JMX mbeans do not really match the expected values and operations do not do the expected things. It looks like no messages ever went through the queue. What gives?

Nothing new actually: documentation that is incomplete; adding new functionality is much more important than making it work in the useful configurations, you name it. Works in JBoss as embedded? Good enough, let's move on. Need to do something different? Good luck with that.

So why it works and how?

All the configuration files under deploy/hornetq are related to the embedded HornetQ. Adding a queue/topic definition to hornetq-jms.xml adds the destination to the embedded server. The JMX mbeans are created to manage this local destination. And do not get me started on the story of two JMX beans per queue/topic and the difference between "core" HornetQ and "jms" HornetQ. Layered architecture in practice.

As part of a queue/topic creation HornetQ registers it in JBoss JNDI. This registered object is more or less the destination's name on the server. Yes, the object is an implementation of javax.jms.Queue/Topic interface but this does not really matter. It is useless by itself. The "magic" happens only in combination with the configured ConnectionFactory (for the outbound communication) or ConnectorFactory (for the inbound communication). Is it local "InVM" connection? Then the embedded server is doing the work. Is it "Netty" connection pointing to the standalone HornetQ? Then messages go to/from that server.

The net result: since there is no documented way to register such "destination names" in JBoss JNDI directly one must keep the embedded HornetQ running. And then for each remote destination one must create the corresponding local destination to get the correct JNDI objects in place as a side effect. Really a way to go, HornetQ! Clear and understandable architecture. And you get (for free!) a pair of useless JMX mbeans per destination so that you can waste your time trying to understand what the hell is going on with the messages.

It is even more interesting for people who want to use HornetQ from other application servers. There is no <adminobject> in ra.xml (<adminobject>s are those "destination names" in JNDI). Naturally, since embedded HornetQ creates them programmatically. It works in JBoss as embedded? Good enough then.

javax.namespace.QName and serialization

I have already mentioned a particular quirk of javax.namespace.QName under JBoss AS 6 and how to avoid it. Unfortunately it is not the end of the story.

You see long time ago somebody made a mistake. Different JDK versions were shipped with serialization-incompatible versions of javax.namespace.QName. Ooops.

I did not try to find out when it was made and what exactly happened, like just forgotten serialVersionUID or something else. Can happen, was fixed. You can see the fix in the JDK source code. Of course the fix is not really complete if we take into account some versions of stax-api jar floating around.

Anyway, stax-api.jar in JBoss AS6 is taken care of. But our application started to fail after it was deployed in the test environment. The application is just migrated to JBoss and is running against an existing test database. And here is the problem: the application stores some JAXB unmarshalled objects in the database. JAXB really likes to use javax.xml.bind.JAXBElement class which has a field of type QName. And the previous version of the application was running under a different version of JDK. You've got the picture.

The "fix" provided by JDK does not help here. It just allows me to select the current serialVersionUID from a limited set of supported values. Deserializing an instance of javax.namespace.QName serialized with a different serialVersionUID fails.

It is possible to go to the database, identify the records that have a wrong serialVersionUID and try to change them. Hmm, understanding the format of serialized instances of java classes and Oracle BLOB functions is not what I really wanted to do. On top of that there are other situations where one might need to be able to support multiple values of serialVersionUID like for example remote invocations.

Fortunately these other situations were not applicable in my case. And I was also lucky because the framework that the application is using allows configuring the way the data is read from and written into a database. Sort of. Several configuration files and several classes after I actually could concentrate on the solution itself:

public class CustomObjectInputStream extends ObjectInputStream {

...
    //
    protected ObjectStreamClass readClassDescriptor() throws IOException, ClassNotFoundException {
        // This is the fix: if the description of QName class is read, check what value of
        // serialVersionUID it has. And if it is not the "current good" value, but matches
        // one of the "known good" values, just return the "current good" descriptor.
        ObjectStreamClass  desc = super.readClassDescriptor();

        if (desc != null && platformQNameDesc != null && platformQNameDesc.getName().equals(desc.getName())) {
            // OK, this is QName descriptor
            long  readVerUID = desc.getSerialVersionUID();

            if (readVerUID != platformQNameDesc.getSerialVersionUID()) {
                // SerialVersionUID of the serialised class DOES NOT match
                // the value used by this JDK process.
                if (readVerUID == JDK_defaultSerialVersionUID ||
                    readVerUID == JDK_compatibleSerialVersionUID ||
                    readVerUID == STAXAPI_SerialVersionUID)
                {
                    // OK, one of the "known good values". Pretend we read the correct one.
                    //
                    // This is not a very nice way, but the thing is: ObjectStreamClass
                    // is a class, not interface, and has a lot of (package) private fields
                    // that are directly read by other JDK classes. So it is either
                    // reflection and changing the private field or this one:
                    desc = platformQNameDesc;
                }
            }
        }

        return desc;
    }

    //
    private ObjectStreamClass  platformQNameDesc = ObjectStreamClass.lookup(QName.class);

    //
    private static final long  JDK_defaultSerialVersionUID = -9120448754896609940L;
    private static final long  JDK_compatibleSerialVersionUID = 4418622981026545151L;
    private static final long  STAXAPI_SerialVersionUID = -711357515002332258L;
}

I really think that something like this must be implemented by JDK, and not only for QName. Allowing to override serialVersionUID is not enough. And maybe class java.io.ObjectInputStream should be changed as well to be able to serialize objects with non-default values of serialVersionUID.