Securing connections with TLS

by Erik van Oosten, November 10, 2009

In this article I’ll explore some ways to secure socket communication with TLS from a java application. To make it more concrete I’ll show you SubEtha SMTP (an excellent Java based bare bones SMTP server) and the recent TLS extensions I added to it.

What you’ll get from this article:

How to mix secure with insecure communication
How to start TLS, server side
How to make the connection really safe
How to add client authentication
How to apply this with SubEtha SMTP

I’ll assume you know Java, understand the concept of a socket and the purpose of TLS/SSL.

How to mix TLS with insecure communication

Often protocols need to operate both securely and insecurely. Be it for simplicity or for backward compatibility. The latter reason is important for SMTP as it is traditionally insecure.

There are two options to secure SMTP with TLS. Let discuss them shortly:

Option 1: use a separate server port for TLS. The client will connect to a separate port for secure mode and the TLS handshake will begin immediately after the socket is open. This mode is suitable for new protocols that need to be completely secure. This option is not suitable for protocols that also need to operate insecurely like SMTP. (See RFC2595 chapter 7 for more information.)
Option 2: use a single server port for both insecure as secure communication. The trick is to start the TLS handshake simultaneously on the client and the server. For telnet based protocols like SMTP, POP3 and IMAP, this is done with the STARTTLS command (RFC2595 and RFC3207).

This article will focus on the server side of option 2, but most of the code is identical to the client side and for option 1.

How to open a TLS connection – Server

TLS is implemented by JSSE which is part of the JDK since J2SDK 1.4. The JSSE Reference Guide is a good source of information, both general as very specific.

To listen on a port (without TLS), the server first needs to create a ServerSocket and then wait for new connections. This can be as simple as:

<br>
  int bindPort = 25;<br>
  InetSocketAddress bindAddress = new InetSocketAddress(bindPort);<br>
  ServerSocket serverSocket = new ServerSocket();<br>
  serverSocket.setReuseAddress(true);<br>
  serverSocket.bind(bindAddress);<br>
  while (true) {<br>
    Socket socket = serverSocket.accept();<br>
    // ...start a new thread that will communicate on the socket...<br>
  }<br>

Later, when both parties agree that TLS should be started, a SSLSocket is needed. The SSLSocket will wrap the normal socket that was created by the ServerSocket.

Here is the code. It is very similar to what SubEtha SMTP does (find it in SMTPServer#createSSLSocket).

<br>
  // Get the default SSLSocketFactory<br>
  SSLSocketFactory sf = ((SSLSocketFactory) SSLSocketFactory.getDefault());</p>
<p>  // Wrap 'socket' from above in a SSL socket<br>
  InetSocketAddress remoteAddress =<br>
          (InetSocketAddress) socket.getRemoteSocketAddress();<br>
  SSLSocket s = (SSLSocket) (sf.createSocket(<br>
          socket, remoteAddress.getHostName(), socket.getPort(), true));</p>
<p>  // we are a server<br>
  s.setUseClientMode(false);</p>
<p>  // allow all supported protocols and cipher suites<br>
  s.setEnabledProtocols(s.getSupportedProtocols());<br>
  s.setEnabledCipherSuites(s.getSupportedCipherSuites());</p>
<p>  // and go!<br>
  s.startHandshake();</p>
<p>  // continue communication on 'socket'<br>
  socket = s;<br>

When this code has executed, the application can continue communicating on socket as before, only now all data is encrypted prior to being sent over the big bad internet.

You are not done!

The code above mostly works but has some problems:

The default SSLSocketFactory is used. Configuring which private key is used is done through the JVM (info). If there is none, an anonymous cipher suite will be selected leading to vulnerability of man-in-the-middle-attacks. You have been warned!
In addition, you now depend on the list of trusted certificate authorities known to the JVM.
Allowing all supported protocols and cipher suites is insecure. SSLv1 and SSLv2 have problems, but worse, some older cipher suites are so badly broken that they hardly offer any protection.
It has no exception handling. You will need to catch IOException, but your are better of catching SSLHandshakeException separately as it is common to get the “no cipher suites in common” error (there is no need to log a whole stacktrace for that).

Lets fix the first 2 problems.

Adding key management

To get back some control over the used certificates, you’ll need to create your own SSLSocketFactory in the form of a SSLContext. Here are the steps needed to create a SSLContext:

First you’ll create the key store. The key store contains your own private key and all the certificates that signed that key. The key store is backed by a file and protected by a passphrase.

<br>
  // Key store for your own private key and signing certificates<br>
  InputStream keyStoreResource = new FileInputStream("/path/to/keystore.jks");<br>
  char[] keyStorePassphrase = "secret".toCharArray();<br>
  KeyStore ksKeys = KeyStore.getInstance("JKS");<br>
  ksKeys.load(keyStoreResource, keyStorePassphrase);<br>

The file /path/to/keystore.jks is in the default format as created by keytool which is part of the JDK.

A more interesting option is to use a key store in the standardized PKCS#12 format (which is also supported by openssl):

<br>
  // Key store for your own private key and signing certificates<br>
  InputStream keyStoreResource = new FileInputStream("/path/to/keystore.p12");<br>
  char[] keyStorePassphrase = "secret".toCharArray();<br>
  KeyStore ksKeys = KeyStore.getInstance("PKCS12");<br>
  ksKeys.load(keyStoreResource, keyStorePassphrase);<br>

As a key store potentially contains many keys, you need a KeyManager to determine which key to use. Under the assumption that there is only 1 key in the key store, the default KeyManager is fine. Of course, there is an extra indirection through a KeyManagerFactory:

<br>
  // KeyManager decides which key material to use.<br>
  KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");<br>
  kmf.init(ksKeys, keyStorePassphrase);<br>

The second component you need is the trust store. A trust store contains certificates of trusted certificate authorities. These are needed to validate client certificates. You can use the JDK trust store (located at either $JAVA_HOME/lib/security/jssecacerts or $JAVA_HOME/lib/security/cacerts), or you can create one from scratch with keytool.

<br>
  // Trust store contains certificates of trusted certificate authorities.<br>
  // Needed for client certificate validation.<br>
  InputStream trustStoreIS = new FileInputStream("/path/to/truststore.certs");<br>
  char[] trustStorePassphrase = "secret".toCharArray();<br>
  KeyStore ksTrust = KeyStore.getInstance("JKS");<br>
  ksTrust.load(trustStoreIS, trustStorePassphrase);<br>

And again a TrustManager (the default one accepts all certificate authorities in the trust store):

<br>
  // TrustManager decides which certificate authorities to use.<br>
  TrustManagerFactory tmf = TrustManagerFactory.getInstance("SunX509");<br>
  tmf.init(ksTrust);<br>

Finally you can create the SSLContext:

<br>
  SSLContext sslContext = SSLContext.getInstance("TLS");<br>
  sslContext.init(kmf.getKeyManagers(), tmf.getTrustManagers(), null);<br>

Luckily the code above only needs to run once. Thereafter you can get the SSLSocketFactory as follows.

<br>
  // Get your own custum SSLSocketFactory<br>
  SSLSocketFactory sf = sslContext.getSocketFactory();</p>
<p>  // continu as above<br>

Removing dangerous protocols and cipher suites

The second problem the code has is that it allows for broken protocols and cipher suites. To fix this you’ll need to indicate exactly which protocols and cipher suites you want to enable.

Look at the code above and replace the arguments to setEnabledProtocols and setEnabledCipherSuites:

<br>
  // select supported protocols and cipher suites<br>
  s.setEnabledProtocols(SslConstants.intersection(<br>
        s.getSupportedProtocols(), StrongSsl.ENABLED_PROTOCOLS));<br>
  s.setEnabledCipherSuites(SslConstants.intersection(<br>
        s.getSupportedCipherSuites(), StrongSsl.ENABLED_CIPHER_SUITES));<br>

StrongTls.java is a class that lists strong protocols and ciphersuites. You can use it under the Apache 2 license. The list was compiled by a security expert at the Dutch government.

Adding client authentication

Most internet applications do not require strict identification of the client. Therefore by default TLS only lets the client check the identify of the server (hence the need for a signed private key on the server). However, for some applications client authentication is just as important.

To indicate this, add the following code just before the start of the handshake:

<br>
  // Client must authenticate<br>
  s.setNeedClientAuth(true);<br>

When the client certificate is not signed by a trusted certificate authority, the handshake will fail. After a successful handshake the client certificates can be accessed with:

<br>
  // Get client certificates<br>
  Certificate[] peerCertificates = s.getSession().getPeerCertificates();<br>

Securing SubEtha SMTP

Now lets apply all this to SubEtha SMTP. First of all you’ll need to make sure that you have SubEtha SMTP version 3.1.2 or higher. This is the version I extended to optionally require TLS and to allow custom SSLSocket creation.

First create the SSLContext as described above:

<br>
  // Key store for your own private key and signing certificates.<br>
  InputStream keyStoreIS = new FileInputStream("/path/to/keystore.p12");<br>
  char[] keyStorePassphrase = "secret".toCharArray();<br>
  KeyStore ksKeys = KeyStore.getInstance("PKCS12");<br>
  ksKeys.load(keyStoreIS, keyStorePassphrase);</p>
<p>  // KeyManager decides which key material to use.<br>
  KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");<br>
  kmf.init(ksKeys, keyStorePassphrase);</p>
<p>  // Trust store contains certificates of trusted certificate authorities.<br>
  // We'll need this to do client authentication.<br>
  InputStream trustStoreIS = new FileInputStream("/path/to/truststore.certs");<br>
  char[] trustStorePassphrase = "secret".toCharArray();<br>
  KeyStore ksTrust = KeyStore.getInstance("JKS");<br>
  ksTrust.load(trustStoreIS, trustStorePassphrase);</p>
<p>  // TrustManager decides which certificate authorities to use.<br>
  TrustManagerFactory tmf = TrustManagerFactory.getInstance("SunX509");<br>
  tmf.init(ksTrust);</p>
<p>  SSLContext sslContext = SSLContext.getInstance("TLS");<br>
  sslContext.init(kmf.getKeyManagers(), tmf.getTrustManagers(), null);<br>

Then create a subclass of SMTPServer and start it:

<br>
  // Your message handler factory.<br>
  MessageHandlerFactory mhf = new YourMessageHandlerFactory();</p>
<p>  SMTPServer smtpServer = new SMTPServer(mhf) {<br>
    @Override<br>
    public SSLSocket createSSLSocket(Socket socket) throws IOException {<br>
      InetSocketAddress remoteAddress =<br>
              (InetSocketAddress) socket.getRemoteSocketAddress();</p>
<p>      SSLSocketFactory sf = sslContext.getSocketFactory();<br>
      SSLSocket s = (SSLSocket) (sf.createSocket(<br>
              socket, remoteAddress.getHostName(), socket.getPort(), true));</p>
<p>      // we are a server<br>
      s.setUseClientMode(false);</p>
<p>      // select strong protocols and cipher suites<br>
      s.setEnabledProtocols(StrongSsl.intersection(<br>
            s.getSupportedProtocols(), StrongSsl.ENABLED_PROTOCOLS));<br>
      s.setEnabledCipherSuites(StrongSsl.intersection(<br>
            s.getSupportedCipherSuites(), StrongSsl.ENABLED_CIPHER_SUITES));</p>
<p>      //// Client must authenticate<br>
      // s.setNeedClientAuth(true);</p>
<p>      return s;<br>
    }<br>
  };</p>
<p>  smtpServer.setHostName(host);<br>
  smtpServer.setPort(port);<br>
  smtpServer.setBindAddress(bindAddress);<br>
  smtpServer.setRequireTLS(true);<br>
  smtpServer.start();<br>

That’s it, SubEtha SMTP is now accessible only with TLS. If you also required client authentication, you can get the client certificates from the context parameter in the message handler.

Some further considerations

To prevent “no cipher suites in common” errors, it is wise to install the unlimited security library. Download “Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 6” from the Sun Java download site (last item in the list) and install it as described in the readme. This allows for longer keys which can improve security and increases TLS compatibility.

Conclusions

With some minor adjustments to the excellent SubEtha SMTP library, it is possible to have very secure e-mail transmission.