Too many open files
Today I was asked to review a piece of code that a coworker of mine suspected was causing “Too many open files“ exceptions in a production environment The code looked innocuous enough. He was using ProcessBuilder to create and run an external application. Regardless I booted into Linux and ran some tests. Now on Linux its fairly easy to find out if a program is leaking file handles. Once a program is started you can look up the process in the /proc file system. The /proc filesystem is a virtual filesystem that resides in the kernels memory it provides you with an easy way to change the behavior of the linux kernel and allows you to view information about currently running processes.
To find out what filehandles a java process is holding onto you simply navigate to /proc/[PROCESSID]/fd There’s a file there for every open file handle.
To count all open file handles you can execute ls /proc/[PROCESSID]/fd | wc -w
Using these simple tools I quickly found out that every time the suspect code was run. 3 filehandles where created that were not being released
As it turns out you need to explicitly close the file handles allocated by Process (eg. Close inputstream, outputstream and errorstream) Process does not release them after the invoked program terminates!
This seems to be a little known fact about Process. Its not mentioned in the javadoc for the process class and just about all the authorative resources get this wrong. Eg.
http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html
http://www.ibm.com/developerworks/java/library/j-tiger09304.html
Etc
At least sun accepted a bug report for improving the documentation. However its been open since 2006..
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6462165
Now I must admit I still am kind of curious about the application my colleague is developing because i believe the max number of filehandles is 65536 on most modern linux systems 🙂
Update: you can also use lsof -p PID for this. The added benefit is that for sockets it will also cross reference the from – > to for socket connections
