My 11 day quest to connecting JMS client to remote GlassFish from Tomcat

To prevent anyone from being in this tormenting situation ever again, this article describes the result of my 11 day quest to get a remote JMS client (running in Tomcat 6) to send JMS messages to a clustered GlassFish v2.1.1.

The contents:

1. Step 1. Configure GlassFish JMS
2. Step 2. Configure GlassFish ports
3. Step 3. Configure the firewall
4. Step 4. Make the GlassFish jars available
5. Step 5. Cleanup the classpath
6. Step 6. Getting the JMS Queue or Topic with JNDI
7. Step 7. Making MQ client shutdown properly
8. Ideal world
9. Conclusions

Step 1. Configure GlassFish JMS

You’ll need to configure a JMS connection pool, and probably also the destination queue or topic. With the GlassFish administration console this should be rather straightforward so I’ll leave this up to you or your administrator.

Step 2. Configure GlassFish ports

In order to configure the firewall you need to fix which ports the JMS implementation is going to use. You do this by configuring a command line argument in the GlassFish administration console:

1. Open the GlassFish administration console
2. Select bottom option in menu (‘Configurations’)
3. Select your configuration
4. Select Java Message Service
5. In field ‘Start Arguments’ enter the text: -startRmiRegistry -rmiRegistryPort 34000 -Dimq.jms.tcp.port=43320
6. Click ‘Save’

(Based on information in Taming the beast: Binding imqbrokerd / OpenMQ to fixed ports.)

Step 3. Configure the firewall

First we need to find out on which ports GlassFish listens:

1. Open the GlassFish administration console
2. Select bottom option in menu (‘Configurations’)
3. Select your configuration
4. Click ‘System properties’ at the bottom
5. Note the port numbers for variables ‘IIOP_LISTENER_PORT’ and ‘JMS_PROVIDER_PORT’

For us these ports are 33700 and 37676. Open the firewall on the GlassFish host for all Tomcat hosts on these two ports, plus the two ports that were configured in the previous step (34000 and 43320).

Step 4. Make the GlassFish jars available.

If you put the GlassFish jars directly in your war, you’ll get strange exceptions such as java.lang.ClassNotFoundException: com.sun.enterprise. naming. SerialInitContextFactory and ClassNotFoundException: com.sun.corba.ee. impl. orbutil. CacheTable.Entry.

Carl Roberts suggests how to completely replace the Tomcat classloader to fix this. I choose to follow the article Tomcat 6.x … doing it the right way! from Niclas Meier as it is just less work.

So, we’ll make the GlassFish jars available in Tomcat through the shared class loader. In Tomcat 6 the shared/lib directory doesn’t exist yet (it does in Tomcat 5). Create it and in ${tomcat}/conf/catalina.properties set the following line:

shared.loader=${catalina.home}/shared/lib, \
  ${catalina.home}/shared/lib/*.jar

Place the following jars from the GlassFish distribution (yes, that adds up to 20 mega bytes of compiled java code) in that directory:

• appserv-admin.jar
• appserv-deployment-client.jar
• appserv-ext.jar
• appserv-launch.jar
• appserv-rt.jar
• appserv-ws.jar
• imqjmsra.jar
• javaee.jar

I have tried to minimize the list, but as you need the full 15MB appserv-rt.jar anyway (which includes the majority of GlassFish) I left it at this.

Step 5. Cleanup the classpath

Make sure you do not have any jars like java-naming.jar, naming.jar, jms.jar, javax-jms.jar in your war. The classes in these jars will take precedence over those in the shared class loader. If you have strange ClassCastExceptions, this is the cause.

For example, as we are using Maven and Spring, our parent pom needs an exclusion as follows:

  
    
      org.springframework
      spring-jms
      ${spring.version}
      
        
          javax.jms
          jms
        
      
    
  

You might need the JMS classes at compile time. In that case add the following dependency (notice the provided scope):

  javax.jms
  jms
  1.1
  provided

Manually verify the WEB-INF/lib directory of your war. To track the cause of other suspect jars with Maven, the command mvn dependency:tree is essential.

Step 6. Getting the JMS Queue or Topic with JNDI

In a plain Java application you can get the JMS destination from GlassFish as follows:

  String endpoints = "hostname:port,hostname:port";
  Properties p = new Properties();
  p.put("com.sun.appserv.iiop.endpoints", endpoints);
  p.put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.enterprise.naming.SerialInitContextFactory");
  p.put(Context.URL_PKG_PREFIXES, "com.sun.enterprise.naming");
  p.put(Context.STATE_FACTORIES, "com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl");
  p.put("com.sun.corba.ee.transport.ORBTCPTimeouts", "500:30000:30:999999");

  Context context = new InitialContext(p);
  QueueConnectionFactory qcf = (QueueConnectionFactory) context.lookup(factoryName);
  Queue queue = (Queue) context.lookup(queueName);

I have found many other ways to address the GlassFish host, but only com.sun.appserv.iiop.endpoints seems to work with a clustered GlassFish instance.

Another gotcha is that the GlassFish ORB insists that the endpoints are properly resolvable through DNS on both the GlassFish host as the Tomcat host. If you use an IP address, GlassFish will even do a reverse lookup in DNS and then passes that back to the client which will use it for further connections. (Quite weird as we configured a completely valid IP address.) Anyway, the best solution is to use full hostnames as they are available through DNS or in a /etc/hosts file on both hosts.

(Valuable hints were found in Glassfish mysteries #4: IIOP.)

Now you could send a message with (add try/finallies as appropriate):

  QueueConnection conn = qcf.createQueueConnection();
  QueueSession session = conn.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
  TextMessage msg = session.createTextMessage();
  msg.setText("Hello GlassFish!");
  MessageProducer sender = session.createProducer(queue);
  sender.send(msg);
  sender.close();
  session.close();
  conn.close();

Put this code in a simple standalone example application to test the connection through the firewall. Don’t forget to put the GlassFish jars mentioned above on the classpath.

Another approach, with Spring, is to declare a JmsTemplate. In a regular Java application (not running under Tomcat) this is declared as follows:

    
        
        
        <!-- more properties to declare marshaller, transactions, etc. -->
    

    
        
        
            
                java.naming.factory.initial=com.sun.enterprise.naming.SerialInitContextFactory
                java.naming.factory.url.pkgs=com.sun.enterprise.naming
                java.naming.factory.state=com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl
                com.sun.appserv.iiop.endpoints=someHost:somePort,someOtherHost:someOtherPort
                com.sun.corba.ee.transport.ORBTCPTimeouts=500:30000:30:999999
            
        
    

Unfortunately both approaches fail. Debugging showed that at several points during the lookup, a new InitialContext is created without any arguments. I am not sure why, but to me this just sounds like the big bug JNDI already is. Anyway, the effect is that passing in our environment properties is useless; they are simply not used except by the initial IntialContext you create yourself. Thereafter, only the system properties that were set by Tomcat’s JNDI implementation are used. Symptoms: “Failed to look up ConnectorDescriptor from JNDI” exception.

You might also get exceptions like:

Nov 30, 2010 11:37:45 AM com.sun.enterprise.connectors.ConnectorConnectionPoolAdminServiceImpl obtainManagedConnectionFactory
SEVERE: mcf_add_toregistry_failed
Nov 30, 2010 11:37:45 AM com.sun.enterprise.naming.SerialContext lookup
SEVERE: NAM0004: Exception during name lookup : {0}
com.sun.enterprise.connectors.ConnectorRuntimeException: Failed to register MCF in registry : JMSConnectionPool
 at com.sun.enterprise.connectors.ConnectorConnectionPoolAdminServiceImpl.obtainManagedConnectionFactory(ConnectorConnectionPoolAdminServiceImpl.java:1110)
 at com.sun.enterprise.connectors.ConnectorRuntime.obtainManagedConnectionFactory(ConnectorRuntime.java:275)
 at com.sun.enterprise.naming.factory.ConnectorObjectFactory.getObjectInstance(ConnectorObjectFactory.java:113)
 at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:304)
 at com.sun.enterprise.naming.SerialContext.lookup(SerialContext.java:414)
 at javax.naming.InitialContext.lookup(InitialContext.java:392)
 ...
Caused by: java.lang.NullPointerException
 at com.sun.enterprise.naming.SerialContext.lookup(SerialContext.java:385)
 at javax.naming.InitialContext.lookup(InitialContext.java:392)
 at com.sun.enterprise.resource.AbstractResourcePool.setPoolConfiguration(AbstractResourcePool.java:187)
 at com.sun.enterprise.resource.AbstractResourcePool.(AbstractResourcePool.java:170)
 ...

This is because the Spring implementation closes the InitialContext immediately after the lookup.

Use the following code to workaround these GlassFish JNDI bugs. Instead of Spring’s JndiObjectFactoryBean, we’ll use the following code which temporarily changes the system properties during the lookup. In addition it only closes the context when the entire application shuts down.

package nl.jteam.jeeworkarounds;

import org.springframework.beans.factory.DisposableBean;
import org.springframework.beans.factory.FactoryBean;
import org.springframework.beans.factory.FactoryBeanNotInitializedException;
import org.springframework.beans.factory.InitializingBean;
import org.springframework.beans.factory.annotation.Required;
import org.springframework.util.StringUtils;

import java.util.Properties;

import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;

/**
 * {@link org.springframework.beans.factory.FactoryBean} that looks up a
 * JNDI object from an external GlassFish host. Exposes the object found
 * in JNDI for bean references, e.g. for data access object's "dataSource"
 * property in case of a {@link javax.sql.DataSource}.
 *
 * <p>The typical usage will be to register this as singleton factory
 * (e.g. for a certain JNDI-bound DataSource) in an application context,
 * and give bean references to application services that need it.
 *
 * <p>The JNDI object is looked up on startup and cached. The InitialContext that
 * is used to do the lookup is closed when the application context terminates.
 *
 * <p>WARNING:
 * THIS CLASS DOES NOT SUPPORT CONCURRENCY DURING INITIALIZATION OF ANY KIND.
 * During initialization and before the lookup the system properties are changed,
 * they are restored after the lookup.
 *
 * <p>Might not work properly when a security manager is activated.
 *
 * @author Erik van Oosten
 */
public class GlassFishJndiObjectFactoryBean implements FactoryBean, InitializingBean, DisposableBean {

    private String jndiEndPoints;
    private String jndiName;

    private InitialContext context;
    private Object jndiObject;

    /**
     * @param jndiEndPoints a comma separated list of host:port pairs, each identifying
     *   a remote GlassFish installation, e.g. "trident:3600,exodus:3700" (not null)
     *
     * The ports can be found in the GlassFish administration console under the property
     * IIOP_LISTENER_PORT, for a cluster instance the default is 33700.
     */
    @Required
    public void setJndiEndPoints(String jndiEndPoints) {
        this.jndiEndPoints = jndiEndPoints;
    }

    /**
     * @param jndiName the name of the JNDI object (not null)
     */
    @Required
    public void setJndiName(String jndiName) {
        this.jndiName = jndiName;
    }

    /**
     * Look up the JNDI object and store it.
     */
    public void afterPropertiesSet() throws NamingException {
        if (!StringUtils.hasText(jndiEndPoints) || !StringUtils.hasText(jndiName)) {
            throw new FactoryBeanNotInitializedException("not all properties are set");
        }

        this.jndiObject = lookup();
    }

    /** {@inheritDoc} */
    private Object lookup() throws NamingException {
        Properties pre = System.getProperties();

        try {
            System.setProperty(Context.INITIAL_CONTEXT_FACTORY, "com.sun.enterprise.naming.SerialInitContextFactory");
            System.setProperty(Context.URL_PKG_PREFIXES, "com.sun.enterprise.naming");
            System.setProperty(Context.STATE_FACTORIES, "com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl");
            System.setProperty("com.sun.appserv.iiop.endpoints", jndiEndPoints);
            System.setProperty("com.sun.corba.ee.transport.ORBTCPTimeouts", "500:30000:30:999999");

            context = new InitialContext();
            return context.lookup(jndiName);

        } finally {
            // Restore system properties
            System.setProperties(pre);
        }
    }

    /** {@inheritDoc} */
    public Object getObject() throws Exception {
        return jndiObject;
    }

    @Override
    public Class getObjectType() {
        return null;
    }

    @Override
    public boolean isSingleton() {
        return false;
    }

    @Override
    public void destroy() throws Exception {
        if (context != null) {
            try {
                context.close();
            } catch (Exception e) {
                // Don't care, we've done our best.
            } finally {
                context = null;
            }
        }
    }
}

This is configured as follows:

    
        
        
        <!-- more properties to declare marshaller, transactions, etc. -->
    

    
        <!-- Only hostnames, no IP addresses! -->
        
        
    

    
        
        
    

    
        <!-- Only hostnames, no IP addresses! -->
        
        
    

I am not sure whether the ‘smartConnectionFactory’ is needed, but can’t hurt also.

Step 7. Making MQ client shutdown properly.

The connection pool from GlassFish has a bug (marked won’t fix) that will prevent your web application from terminating normally (not all threads exit). I tried to interrupt and then even stop those threads, but they simply ignore them and reconnect (to all framework developers: please don’t catch java.lang.Error‘s and certainly do not ignore ThreadDeath).

To make Tomcat stop anyway, you can either add a kill statement to your Tomcat script, or just call System.exit(0) by adding the following bean to your application:

package nl.jteam.jeeworkarounds;
import org.springframework.beans.factory.DisposableBean;

/**
 * Call System.exit(0) to kill all daemon threads from MQ as
 * MQ does not close properly (http://java.net/jira/browse/GLASSFISH-1429).
 *
 * @author Erik van Oosten
 */
public class SystemKillerBean implements DisposableBean {

    @Override
    public void destroy() throws Exception {
        // Delay execution to let all other shutdown code continue first.
        Thread thread = new Thread(new Runnable() {
            @Override
            public void run() {
                try {
                    // Give other threads chance to clean up
                    // TODO: consider making timeout configurable with a setter
                    Thread.sleep(3000L);

                    System.exit(0);

                } catch (Exception e) {
                    // Log4j is gone, use syserr
                    System.err.println("Failed to completely shutdown web-ui. Please kill tomcat manually.");
                }
            }
        });
        thread.setName("I-am-going-to-call-system.exit(0)");
        thread.setPriority(Thread.NORM_PRIORITY - 1);
        thread.setDaemon(true);
        thread.start();
    }
}

In your main web application context include the following line:

Ideal world

If you think the above is not so bad, lets contrast this to my picture of the ‘ideal world’; a practice that is available from many libraries. It goes something like this:

• Select a library.
• Copy the maven dependency from their website and paste it in a pom.xml (I am using IntelliJ, so the library is retrieved automatically).
• Only a single port needs to be opened in the firewall, an IP address will just work.
• In a Spring context, or programmatically construct a ‘connection’ with some simple properties such as host and port.
• Programmatically construct a message object and send it with the ‘connection’ object.

Conclusions

If you value your saneness, connecting a remote JMS client to a clustered GlassFish v2.1 is not something you should try to do. However, might you be in the situation anyway, follow this article and it might work.