Showing posts with label Maven. Show all posts
Showing posts with label Maven. Show all posts

Tuesday, July 5, 2011

Fairy tale of developer's heaven

Ah maven! Promises of repeatable and portable builds... Fairy tales of developer's heaven...

Shall I tell one as well?

From time to time I do some things on a JEE project being developed by some other people. The resulting EAR file is quite dependent on some JEE container. Recently a decision was made to migrate the project to another JEE container. It is not a one-off migration effort: we need to be able to build the project for the original container and for the new one. I was asked to look into that.

One possibility to handle this is to have a separate branch for each container. Just thinking about all coming cross merges makes me cry.

Another possibility is to have a single branch and just package additional container specific deployment descriptors here and there. Nice and simple, perfectly JEE compliant ...

Except for one seemingly small detail. The project is using say version 1 of a particular 3rd party dependency. Unfortunately the project with this version can't be deployed into the newly targeted container. Fortunately there is already version 2 of the dependency, and the project built against this version can be deployed in the new container. Unfortunately it is precisely the other way around with version 2: the old container can't handle it.

And it is not that we can build the project against one of the versions and then just package differently: the versions are not binary compatible. There are also some minor source-code compatibility issues, but they can be relatively simply solved. Anyway, bottom line: if we build our project against one of the versions of the library it will not work with the other. We really need to compile our software against the correct version of the library and then package accordingly.

So far so good. The situation is probably quite common, and not very difficult to handle. Normally.

But the project is using maven, more specifically, maven 2.2.1. So far it worked pretty well for the project except for some of maven's WAR/EAR packaging "features". But for this new deployment target I hit a wall.

The end result I wanted to achieve: one checks out the source code, sets up the container specific environment, runs 'mvn install' and gets a set of properly versioned container specific artifacts, say, application-X.Y-container1.ear or application-X.Y-container2.ear. The "properly versioned" part is very important. This way we can refer to the correct versions of artifacts in our poms, we can properly release container specific versions of the project, etc.

The very first question was: how do we achieve that versioning scheme in maven? The 3rd party library is used in all ejb and almost all jar modules making them depend on the library. Web modules also depend on it (indirectly, via ejb/jar modules). The same is true for the ear module.

Why, it is easy I thought. I define a property, say, 'target.container', then put it in <artifactid> or <version> tag so the project's poms have <artifactid>usermanager-ejb-${target.container}</artifactid> or <version>0.1-${target.container}-SNAPSHOT</version> in their maven coordinates. Then I start maven with -Dtarget.container=container1 (or container2). This results in <artifactid>usermanager-ejb-container1</artifactid> or <artifactid>usermanager-ejb-container2</artifactid> (or version <version>0.1-container1-SNAPSHOT</version> or <version>0.1-container2-SNAPSHOT</version>) at build time. Problem solved.

Funny thing: it worked. Damn, I should have been more suspicious. I ended up trying both variants, and both worked when maven was executed from command line. That was actually the last thing that worked. Following hours made me really unhappy.

First I noticed that Eclipse (m2eclipse plugin) does not really like my new poms. It kept complaining that it could not find the project's poms and their dependencies. Executing 'maven install' from Eclipse produced a lot of warnings like "'artifactId' contains an expression but should be a constant." That prompted me to move ${target.container} from <artifactid> to <version>. And again, running maven from command line worked. Eclipse kept complaining.

Googling the message I came across a lot of posts related to the same issue. The message from these posts was clear: maven does not support it.

Strange, because maven documentation does not clearly spells it. For example, POM Reference does not say that artifactId or version must be constant. In fact, the very same POM Reference says here:

Maven properties are value placeholder, like properties in Ant. Their values are accessible anywhere within a POM by using the notation ${X}, where X is the property.


Yet people claim that it should not work, and if it works then it is a bug. For example, here or here.

And it looks like maven finally began enforcing this stupidity in version 3. Which is a pity. The funniest thing here is the reason why. Take for example MNG-4297:
Maven currently allows properties in the groupId, artifactId and version of a pom. This causes artifacts to be produced that require full inheritance and interpolation before they can be uniquely identified. It also poses potential problems if the properties are defined in settings, env or profiles where the consumer can't exactly identify the artifact after the fact.


I am just speechless. Between things like downloaded newer versions of plugins (wow, they fixed this one actually), disappeared artifacts from public repositories, rearranged and/or moved public repositories they finally nailed the real problem preventing maven users to have reproducible builds: property-based project coordinates. Strange, last time I looked maven still supports profiles and properties in general...

Anyway, it looks like the way to do what I want is classifiers. They are mentioned here:

The Maven provided solution for your situation is 'classifiers'.


and are really awesome described here (5.5.3. Platform Classifiers). These guys work for Sonatype, so they should know a thing or two about maven you would think, right? Just replace <classifier>win</classifier> with <classifier>container1</classifier> and <classifier>linux</classifier> with <classifier>container2</classifier>, and we are back in business.

Damn you, people who like to misinform others.

I went ahead and modified poms. Build the project from the command line produced some strange results, for example maven built modules 1-6 successfully and then failed building module 7 because of my mistake in the module's pom. I fixed the problem, ran 'mvn install' for module 7, it completed without failure. Then I ran 'mvn clean install' for the parent project, and it failed building module 5 because some classes from one of the dependencies were not found. Huh?

Finally after some more pom changes the build managed to produce an EAR which was successfully deployed in the old container. Then I cleaned up some poms, added some things to <dependencyManagement> here and there, and executed 'mvn clean install' again. I was not able to deploy the resulting EAR in the old container because of some missing classes. It turned out about 1/3 of jars were missing from EAR/lib this time. WTF?!

Running mvn dependency:tree and analyzing its result explained why: there were no dependencies under the dependencies with classifiers! Time to ask google again.

Apparently maven cannot handle transitive dependencies of a dependency with a classifier, see for example MNG-2759. The story is a bit more complicated because this works sometimes for some people. This worked at least once for me. But most of the time it does not work. And maven is not planning to fix it: MNG-2759 has status "Won't Fix".

Yeah, use classifiers if you need some quality headache. Thanks for advice, guys!

The only solution is to spell all dependencies explicitly everywhere I use a module with a classifier. Thanks, but no, thanks. I already have to do it too many times. <dependencyManagement>, <dependency>, WAR packaging exclusion, EAR packaging to make sure that what is excluded from WAR is packaged in EAR/lib.....

What can I say? Indeed, maven is really a "project ... comprehension tool".

And what I am going to do with all this mess? Nothing, really. I have dropped the requirement of having properly versioned artifacts. I have just removed all the classifiers. The profiles stay. So any time I build a project I get a version 0.1-SNAPSHOT which happens to be for one of the containers. Which one? It depends on the chosen profile. This all means more work during release, but frankly I do not care at the moment. Do you want to refer to the released version of one of the submodules in your pom file? You'd better be absolutely sure you know which version you use. Do you want to refer to a SNAPSHOT version?

Reproducible builds and maven? Do not make me laugh.

Thursday, September 30, 2010

Convention over configuration, maven style

Maven is regarded as a very nice example of convention over configuration. I do not dispute that, it is true ... to some extent, until you try something just a bit out of maven's notion of "convention". And then you have a problem.

I mentioned this in one of the earlier posts:

But maven makes creation of such an EAR hmmm ... interesting. I just hate to think what amount of work I would need to do if I needed some dependencies in EAR/lib and some in WEB-INF/lib and most importantly what amount of work would it be to maintain this structure.

The time has come. I need some jars in WEB-INF/lib (tag libraries) and the rest in <EAR>/lib. So let's look at "convention over configuration" a little bit closer. This is what I have to do:

  1. WAR's pom.xml: declare dependencies on external jars and on some other subprojects (couple of EJBs), nothing unusual, should be done anyway. I mention it here for completeness.

  2. EAR's pom.xml: again, nothing unusual, declare dependency on EJB en WAR projects.

That is all. Maven does its magic producing correct WAR and EAR files. Convention over configuration does wonders, does it not?

Sorry, got carried away. Ah, dreams, dreams...

Reality is quite different. Sure, if I do what I just mentioned, the resulting WAR file is correct. But EAR is completely screwed up because the WAR is included "as is". Some dependency jars are present in <EAR>/lib and in <WAR>/WEB-INF/lib. All EJB jars are also present twice, in <WAR>/WEB-INF/lib and in the root of EAR.

Maven offers some ... hmmm ... solution for the problem, documented here: "Solving the Skinny Wars problem". They honestly warn you saying:
The Maven WAR and EAR Plugins do not directly support this mode of operation but we can fake it through some POM and configuration magic.

Let's look at their "configuration magic". It is consists of 2 pieces of configuration that has to be applied to WAR and EAR pom files:

  1. You have to configure WAR pom.xml to remove jars from WEB-INF/lib and to add their names in MANIFEST.MF. Note that these options are orthogonal and you have to specify both.

    And if you need to leave some of the jars in WEB-INF/lib (I need to), then the configuration becomes really crazy. Watch the "POM and configuration magic":

    • option 1:

      <plugin>
      <artifactId>maven-war-plugin</artifactId>
      <configuration>
      <packagingExcludes>
      WEB-INF/lib/C*.jar,
      WEB-INF/lib/R*.jar,
      WEB-INF/lib/a*.jar,
      WEB-INF/lib/b*.jar,
      WEB-INF/lib/c*.jar,
      WEB-INF/lib/d*.jar,
      WEB-INF/lib/e*.jar,
      WEB-INF/lib/h*.jar,
      WEB-INF/lib/jb*.jar,
      WEB-INF/lib/js*.jar,
      WEB-INF/lib/joda-time-1*.jar,
      WEB-INF/lib/joda-time-h*.jar,
      WEB-INF/lib/sl*.jar
      </packagingExcludes>
      <archive>
      <manifest>
      <addClasspath>true</addClasspath>
      <classpathPrefix>lib/</classpathPrefix>
      </manifest>
      </archive>
      </configuration>
      </plugin>
    • option 2:

      <plugin>
      <artifactId>maven-war-plugin</artifactId>
      <configuration>
      <packagingIncludes>
      WEB-INF/lib/standard-1.1.2.jar,
      WEB-INF/lib/joda-time-jsptags-1.0.2.jar,
      **/*.xml,
      public/**/*.*,
      images/**/*.*,
      css/**/*.*
      static/**/*.*
      </packagingIncludes>
      <archive>
      <manifest>
      <addClasspath>true</addClasspath>
      <classpathPrefix>lib/</classpathPrefix>
      </manifest>
      </archive>
      </configuration>
      </plugin>
    At one time I also thought of extracting TLD files from tag libraries, placing them under WEB-INF, and still removing all jar files from WEB-INF/lib. But I decided against it.

    No matter how you look at the problem it is not really "convention". Hell, it is "configuration, configuration, configuration, and even more configuration". And, worst of all, the resulting WAR file cannot be deployed separately, only as part of an EAR. Oops, here go all your nice integration tests. Fortunately it is not a problem for our current project because we do not deploy WAR files separately. But I can imagine some other people would like to do it.

    This "solution" has a minor problem that the resulting MANIFEST.MF is also incorrect: it has all jars mentioned in Class-Path: attribute, even those that stay in WAR/WEB-INF/lib.

    Oh yeah, and some annoyance: I have to specify exclusion by file name and not by dependency. I have only 4 dependencies in WAR's pom.xml and look at that exclusion list!

  2. But it is not all! More "configuration magic": you also have to change EAR's pom.xml and add all WAR-only dependencies that are not included in <WAR>/WEB-INF/lib. Otherwise they are not packaged into EAR/lib. Lucky you, it is WAR-only dependencies, and not dependencies of EJBs.
Wow, finally you get the packaging you want. Now you have to make sure everybody on the team knows that updating projects' dependencies (not only WAR project but also all EJB projects) or adding some directories under src/main/webapp is not that easy as it should be. Because it will sure break WAR packaging.

This is of course not new. The related bug MWAR-9 has just turned 5 years. Congratulations, maven! Happy anniversary, MWAR-9!

This post was brought to you by maven notion of convention over configuration.