What should I think of mule?

"Every SOA project must use an ESB". I do not remember who said that to me but some people do follow this advice. The project mentioned in the previous post is not an exception. Being a nice example of the vendor driven architecture the project had used Oracle ESB for some time. Apparently that thing was so ridden with bugs the team decided to use something else instead, namely, mule community edition. By the time I have joined the project the switch was completed so I was not really involved with ESB. Until recently.

The production environment has being behaving quite flaky for quite some time. Some of the problems were traced back to the Oracle BPEL software. But there were also problems related to the mule server. The problem descriptions were not really helpful. It went like that: "oh, we have messages remaining in the queue", or "the problems begin after one of other services goes down; we suspect that mule holds HTTP connections open ...", or "we had to restart the mule server and the problems went away … or not ... and we had to restart it again ...".

Time to investigate.

The mule configuration used by the project is not that complex. There are some proxy services that were added under "URL transparency" motto. Couple of those proxy services perform some minor request/response transformations. And then there are some services that logically look like one way WS endpoints but internally use Oracle AQ messaging and a simple content-based routing to decouple sending and receiving components and to add message recovery.

The first thing I wanted to check was that "we suspect that mule holds HTTP connections open ..." and see if it is possible to force mule to close an HTTP connection after each request. After all the task sounded easy.

As a seasoned developer who knows next to nothing about a piece of software I google for "mule http connection keepalive". The first link HTTP Transport - Mule 2.x User Guide - mulesoft.org looks promising.

Wait a minute... to read documentation I have to be registered? Rrrrright. Given I need to investigate several possible problems it might be better to register, but this can wait. Google cache first.

I should say the documentation looks nice. Except of course you need to know all the mule story and architecture before you can get sense of the documentation. And it still does not explain why they need to have both 'keepAlive' and 'keep-alive' properties. Anyway after some reading I thought my problem is solved. After all this is what the documentation says:

Attributes of <outbound-endpoint...>
...

keep-alive

boolean

Controls if the socket connection is kept alive. If set to true, a keep-alive header with the connection timeout specified in the connector will be returned. If set to false, a "Connection: close" header will be returned.

So let's try it out. First I verified that the current configuration does not close the HTTP connection after each request. It does not: it uses HTTP 1.1 and sends no "Connection: close" header. Hitting one of the proxy services multiple times in a row results in a single HTTP connection open.

Next I have added keep-alive="false", redeployed mule and executed the same test. There is nice "keepAlive: false" HTTP header and no "Connection" header. Who said it is going to work the first time?

I hate this. I mean I do everything as documented and it does not work. Given I just started poking around mule I would not immediately suspect the documentation is wrong. So I check and double check, and run more tests. And check again and make sure I looked at the correct documentation since the project is using mule 2.2.1 and the latest version is 3.something. Nope, everything looks correct, it just does not work.

Reading the doc page further I come across section Including Custom Header Properties. The same way of specifying Connection: close is also discussed for example here: Disable HTTP Connection keep-alive in CXF proxy.

Guess what? It does not work either. Just for kicks I have added another custom property "ILikeTheMuleDocumentation: false". This one works. Wow!

Next I tried something else that was mentioned in the same thread Disable HTTP Connection keep-alive in CXF proxy, namely, adding "Connection: close" with a deprecated "http.custom.headers" property. This one finally worked. And indeed produced a deprecation warning in logs.

By this time I became quite curious. I mean one must be quite … hmm ... talented developer to achieve such an interesting result of working and non-working features. So I looked in the sources and, after some digging, I found this:

package org.mule.transport.http.transformers;

...

public class ObjectToHttpClientMethodRequest
...

  protected void setHeaders(HttpMethod httpMethod, MuleMessage msg)
  {
      // Standard requestHeaders
      String headerValue;
      String headerName;

      for (Iterator iterator = msg.getPropertyNames().iterator(); iterator.hasNext();)
      {
          headerName = (String) iterator.next();

          if (headerName.equalsIgnoreCase(HttpConnector.HTTP_CUSTOM_HEADERS_MAP_PROPERTY))
          {
              if (logger.isInfoEnabled())
              {
                  logger.warn("Deprecation warning:  There is not need to set custom headers using: " + HttpConnector.HTTP_CUSTOM_HEADERS_MAP_PROPERTY
                      + " you can now add the properties directly to the outbound endpoint or use the OUTBOUND property scope on the message.");
              }
            
              Map customHeaders = (Map) msg.getProperty(HttpConnector.HTTP_CUSTOM_HEADERS_MAP_PROPERTY);
              if (customHeaders != null)
              { 
                  for (Iterator headerItr = customHeaders.entrySet().iterator(); headerItr.hasNext();)
                  {
                      Map.Entry entry = (Map.Entry) headerItr.next();
                      if (entry.getValue() != null)
                      {
                          httpMethod.addRequestHeader(entry.getKey().toString(), entry.getValue().toString());
                      }
                  }
              }
          }
          else if (HttpConstants.REQUEST_HEADER_NAMES.get(headerName) == null
                 && !HttpConnector.HTTP_INBOUND_PROPERTIES.contains(headerName))
          {
              headerValue = msg.getStringProperty(headerName, null);
              if (headerName.startsWith(MuleProperties.PROPERTY_PREFIX))
              {
                  headerName = new StringBuffer(30).append("X-").append(headerName).toString();
              }

              httpMethod.addRequestHeader(headerName, headerValue);
          }
      }

      Set attNams = msg.getAttachmentNames();
      if (msg.getPayload() instanceof InputStream
              && attNams != null && attNams.size() > 0)
      {
          // must set this for receiver to properly parse attachments
          httpMethod.addRequestHeader(HttpConstants.HEADER_CONTENT_TYPE, "multipart/related");
      }
  }
...

Brilliant, is it not? And, by the way, I did not find any place in the outbound path where 'keep-alive' property is checked. So much for a nice documentation. So I ended up using "http.custom.headers" to set "Connection: close" header.

Well, by that time I was well prepared to investigate the remaining issues with our mule configuration. I was sure I would come across even more pearls.