Monitoring Hippo Connection pool using JMX and Groovy
For a project we are using Hippo to manage our content. We have a few components that interact with the repository using the connection pool as provided by hippo. I have modified the connection pool to increase the amount of logs and I have added statistics to the pool which can be exposed using JMX. Our custom components use this altered connection pool, but I the site did not. Our site makes use of the Hippo Site Toolkit, and I want to have this monitoring available as well.
In this blogpost I will explain the changes I made to the connection pool. After that I’ll show what to do to make this changed connection pool available to a site created with the HST and I’ll show a groovy script that reads the data from the remote servers using the JMX connection.
I started writing about JMX, groovy and spring on my personal blog. I wrote about exposing jmx through spring. Adding groovy spring to the spring context and a bit more about JMX. Check these two posts:
?http://www.gridshore.nl/2010/06/02/using-jmx-within-a-spring-application/
?http://www.gridshore.nl/2010/06/20/exposing-jmx-through-jmxmp-and-reading-the-jmx-data-with-groovy/
In this blog post I’ll give more details to use this in a Hippo environment.
Changes to the connection pool
Hippo provides the hst-session-pool and the ??hippo-ecm-repository-connector. The names a re descriptive enough to understand the goals for the two components. In this blog post I am particularly interested in the hst-session-pool. If you are creating an HST website, the spring configuration is configured in the SpringComponentManager-jcr.xml. This is not handled the same as a normal spring web application. But Hippo provides a mechanism to create your own instance of this file and overload the one that is provided by Hippo. More on this later.
For now let us have a look at the class we have created to replace the ?org.hippoecm.hst.core.jcr.pool.BasicPoolingRepository as provided by Hippo. I am not going over all the methods. I will give you an idea of the changes I made. First an image with the components that are involved.
The blue block is a changed Hippo class, the yellow ones are extensions to support monitoring. PoolingStatistics is used to track statistics like the amount of sessions that are obtained and the amount of sessions that are returned. The BasicPoolingRepositoryWihtLogging uses this statistics object to store this data and exposes it so other can read the data. The PoolingRepositoryMonitor reads the required data from the other two components and exposes this using JMX.
To get an idea of the type of changes I made, have a look at the following code block. The first method is the one from hippo, the second the one that I changed. Look at the use of the statistics object and the logging.
public Session login() throws LoginException, RepositoryException {
Session session = null;
try {
session = (Session) this.sessionPool.borrowObject();
} catch (NoSuchElementException e) {
throw new NoAvailableSessionException("No session is available now. Probably the session pool was exhasuted.");
} catch (Exception e) {
throw new LoginException("Failed to borrow session from the pool.", e);
}
return session;
}
public Session login() throws LoginException, RepositoryException {
log.debug("Request session by doing a login.");
try {
Session session = (Session) this.sessionPool.borrowObject();
log.debug("Return a session with object id {} from the pool after a login", session.hashCode());
statistics.sessionObtained();
return session;
} catch (NoSuchElementException e) {
log.debug("Problem obtaining session from the pool", e);
throw new NoAvailableSessionException("No session is available now. Probably the session pool was exhausted.");
} catch (Exception e) {
throw new LoginException("Failed to borrow session from the pool.", e);
}
}
Now we have the altered connection pool class, we have the class that can be exposed through JMX using Spring. Next step is to tell the HST to use this class. Look for the file SpringComponentManager-jcr.xml, make a copy and put this copy in the folder (taken that you use maven to build your project) src/main/resources/META-INF/hst-assembly/overrides. Look for all usages of org.hippoecm.hst.core.jcr.pool.BasicPoolingRepository and replace these with the name of your own implementation. At the time of writing there are 5 different repositories in use by the hst. Next up is to initialize the PoolingReposityMonitor beans. We create a bean for each repository bean, so we have 5 monitors. The final part in the spring configuration is to expose the jmx beans using the mBeanServer. Spring provides some beans for that as well. The following code block show the most important beans.
<context:mbean-server/>
<context:mbean-export registration="replaceExisting" server="mbeanServer"/>
<bean id="jmxServerConnector" class="org.springframework.jmx.support.ConnectorServerFactoryBean">
<property name="threaded" value="true"/>
<property name="daemon" value="true"/>
<property name="server" ref="mbeanServer"/>
<property name="serviceUrl" value="${jmx.serviceurl}"/>
</bean>
<bean id="defaultJmxRepository" class="package.name.of.monitor.PoolingRepositoryMonitor">
<constructor-arg ref="defaultPoolingRepository" index="0"/>
<constructor-arg value="readonly" index="1"/>
</bean>
<bean id="previewJmxRepository" class="package.name.of.monitor.PoolingRepositoryMonitor">
<constructor-arg ref="previewPoolingController" index="0"/>
<constructor-arg value="preview" index="1"/>
</bean>
<bean id="hstConfigJmxRepository" class="package.name.of.monitor.PoolingRepositoryMonitor">
<constructor-arg ref="hstConfigReaderPoolingRepository" index="0"/>
<constructor-arg value="hstConfig" index="1"/>
</bean>
<bean id="writeableJmxRepository" class="package.name.of.monitor.PoolingRepositoryMonitor">
<constructor-arg ref="writeablePoolingRepository" index="0"/>
<constructor-arg value="writeable" index="1"/>
</bean>
<bean id="binariesJmxRepository" class="package.name.of.monitor.PoolingRepositoryMonitor">
<constructor-arg ref="binaryPoolingRepository" index="0"/>
<constructor-arg value="binaries" index="1"/>
</bean>
It is now possible to use a jmx client like jconsole to have a look at the beans. But that is not what I want. I want to connect remote using groovy and I want a nice page that show the basic information. The hst has the possibility to using spring mvc within the hst. I am not going into details on how to do this. Woonsan has written an excellent blog post that explains how to do this.
?http://blogs.onehippo.org/woonsan/2009/06/spring_web_mvc_framework_suppo_1.html
I have configured a controller that has a dependency on a groovy bean. The groovy bean created a few instanced of a class MonitorGroup that just contains a title and a map of parameters. The first code block shows the spring configuration. The second code block shows the groovy part of reading the jmx beans for the hippo session pools.
<bean class="nl.rijksoverheid.monitoring.JmxController">
<constructor-arg ref="jmxMonitor"/>
</bean>
<lang:groovy script-source="classpath:groovy/JmxMonitor.groovy" id="jmxMonitor">
<lang:property name="mbeanServer" ref="mbeanServer"/>
</lang:groovy>
private MonitorGroup createHippoConnectionPoolMonitorGroup() {
def hippoPoolingQuery = new ObjectName('package.name.of.monitor:*')
MonitorGroup monitorGroup = new MonitorGroup() {
List<String> getColumns() {
return ["Name", "Type","Active", "Idle", "Obtained", "Returned", "Initial size", "Max active", "Max Idle", "Min Idle"];
}
List<Object> getValues() {
String[] connectionPools = mbeanServer.queryNames(hippoPoolingQuery, null)
def connectionModules = connectionPools.findAll { name ->
name.contains('type=PoolingRepositoryMonitor')
}.collect { new GroovyMBean(mbeanServer, it) }
return connectionModules.collect {[
it.name().getKeyProperty("name"),
it.PoolType,
it.NumSessionsActive,
it.NumSessionsIdle,
it.NumSessionsObtained,
it.NumSessionsReturned,
it.InitialSize,
it.MaxActive,
it.MaxIdle,
it.MinIdle
]}
}
String getName() {
return "Hippo connection pools";
}
}
return monitorGroup
}
Now we need to handle an incoming request, so we need a controller. And in the end we want to show the data on the screen. That is done using a jsp. The following two code blocks show the controller and the important part of the jsp.
@Controller
public class JmxController {
private GenericMonitor monitor;
@Autowired
public JmxController(GenericMonitor monitor) {
this.monitor = monitor;
}
@RequestMapping(value = "/spring/jmx.do", method = RequestMethod.GET)
public String adminView(ModelMap modelMap) {
List<MonitorGroup> list = monitor.monitorGroups();
modelMap.addAttribute("groups", list);
return "admin/jmx";
}
}
<c:forEach var="group" items="${groups}">
<div class="block-error">
<h2><c:out value="${group.name}"/></h2>
<table>
<tr>
<c:forEach var="columnHeader" items="${group.columns}">
<th><c:out value="${columnHeader}"/></th>
</c:forEach>
</tr>
<c:forEach var="valueRow" items="${group.values}">
<tr>
<c:forEach var="valueItem" items="${valueRow}">
<td><c:out value="${valueItem}"/></td>
</c:forEach>
</tr>
</c:forEach>
</table>
</div>
</c:forEach>
Of course you are now very curious what we have, well have a look at the following screendump.
Still here? Want more? Ok than we will have a look at the last part of this blog post. The groovy script to read the jmx data from a remote machine. The server side is fine, we can use that now. We have used spring to expose jmx beans over jmxmp. The implementation looks a lot like the JmxMonitor bean that has been discussed before. It should be easy to understand. The code block shows the implementation and after that the output.
def hippoPoolingQuery = new ObjectName('nl.rijksoverheid.importer.repository.pooling:*')
String[] connectionPools = server.queryNames(hippoPoolingQuery, null)
def connectionModules = connectionPools.findAll { name ->
name.contains('type=PoolingRepositoryMonitor')
}.collect { new GroovyMBean(server, it) }
println "**Found $connectionModules.size hippo connection pools"
if (connectionModules.size) println "Name Type Active Idle Obtained Returned Max Active Max Idle Min Idle Initial Created Destroyed Activated Passivated"
connectionModules.each {m->
def nameOfEqualLength = m.name().getKeyProperty("name")
def nameLength = nameOfEqualLength.size()
if (nameLength < 50) {
nameOfEqualLength = nameOfEqualLength.padRight (25)
}
print m.name().getKeyProperty("name").padRight(25)
print m.PoolType.padRight(12)
print "$m.NumSessionsActive".padRight(7)
print "$m.NumSessionsIdle".padRight(7)
print "$m.NumSessionsObtained".padRight(7)
print "$m.NumSessionsReturned".padRight(10)
print "".padRight(37)
print "$m.MaxActive".padRight(10)
print "$m.MaxIdle".padRight(10)
print "$m.MinIdle".padRight(10)
print "$m.InitialSize".padRight(12)
print "$m.NumItemsCreated".padRight(10)
print "$m.NumItemsDestroyed".padRight(10)
print "$m.NumItemsActivated".padRight(10)
print "$m.NumItemsPassivated".padRight(12)
print "\n"
}
**Found 5 hippo connection pools Name Type Active Idle Obtained Returned Max Active Max Idle Min Idle Initial Created Destroyed Activated Passivated writeableJmxRepository writeable 0 0 0 0 100 5 0 0 0 0 0 0 binariesJmxRepository binaries 0 0 0 0 100 10 0 0 0 0 0 0 hstConfigJmxRepository hstConfig 1 1 3 2 25 5 0 0 2 0 3 2 defaultJmxRepository readonly 0 0 0 0 100 25 0 0 0 0 0 0 previewJmxRepository preview 0 0 0 0 100 5 0 0 0 0 0 0
