Upgrading OSB's SSL/TLS Implementation

Default SSL implementation is deprecated. Upgrading to JSSE however can make your oldest services fail.

Update: eventually we installed nginx as a SSL/TLS proxy between OSB and the outdated backends. We could control all properties of the TLS connection from nginx downstream, including what SSL/TLS protocol to use, what certificate to present, and what ciphers are available. Removing the direct dependency this way, we were able to upgrade OSB and backend systems separately, each on its own schedule.

Update 2: Here’s an example of a nginx configuration entry we use to arbitrage the TLS properties for one connection. Not all lines here are important. Main thing is that ESB connects to port 18401, and then nginx re-opens the connection to target backend server with SSLv3. Bingo!

server {
    listen       18401 ssl;
    keepalive_timeout 600;

    ssl_certificate      /srvrs/bsideproxy/nginx/ssl/nginx.crt;
    ssl_certificate_key  /srvrs/bsideproxy/nginx/ssl/nginx.key;

    ssl_session_cache shared:SSL:10m;
    ssl_session_tickets on;
    ssl_buffer_size 8k;
    ssl_session_timeout  10m;
    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers   on;

    location / {
        proxy_pass https://dc5ce9.qc.bzz.ca:4444;
        proxy_ssl_protocols SSLv3;
        proxy_connect_timeout 5s;
        proxy_read_timeout 30s;
    }
}

The default SSL implementation in Weblogic (and OSB) is Certicom.

However, support for the Certicom SSL implementation is deprecated and will eventually be removed.

We need a plan for switching to JSSE, the default JRE implementation.

And it is not as easy as clicking a checkbox.

A Short History of HTTPS, SSL and TLS

Every time OSB establishes a HTTPS connection, or when a client connects to OSB over HTTPS, a secure socket protocol is used.

There are two main protocol families: SSL and TLS.

SSL v1

SSL v1 was the original Netscape protocol. It is as old as a mammoth, and one big security flaw. It wasn’t even released to the public.

SSL v2

SSL v2 was an attempt to fix the obvious issues with SSL v1.

By modern standards, it is still insecure and is not supported as is.

An important trace it left is the so-called SSLv2Hello message. This is still widely used as a first message in the SSL handshake to negotiate the protocol version.

SSL v3

SSL v3 is yet another attempt to fix SSL, but a more successful one.

In 2014, however, an attack called POODLE was developed, which prompted some platforms to declare the end of support for SSLv3.

TLS v1.0

In 1999, TLS was published as an enhancement for the SSL v3 protocol. The changes were not very big, but still they made the protocols incompatible.

Over the wire, TLS v1.0 identifies itself as “3.1”, i.e. it declares itself a better version of SSL v3 (“3.0”).

TLS v1.1

TLS v1.1 closed a few gaps with TLS v1.0. Consider it a bug fix release.

TLS v1.2

TLS v1.2 made significant changes to make the data exchange more secure. New encryption algorithms were added, as well as better negotiation options.

Oracle recommends the use of TLS v1.1 or later in a production environment.

Weblogic Implementations: Certicom and JSSE

Certicom

In Weblogic, the default SSL/TLS implementation is provided by Certicom.

Certicom supports SSL v3 and TLS v1.0 (but not higher).

The version of Certicom supplied with Weblogic also doesn’t support newer ciphers and longer keys.

Oracle declared that the Certicom implementation is considered deprecated and will be removed in some future release.

JSSE

Java Secure Socket Extension (JSSE) is coming with JRE.

JSSE supports more modern protocols, including TLS v1.1 and TLS v1.2.

At the same time, JSSE deprecates and drops the support for older protocols. For instance, the JSSE released with JRE 1.7 has dropped support for SSLv2Hello message.

It is reasonable to expect SSL v3, too, to be removed in some future version of JSSE.

So… What Does it All Mean in Practice?

It means we, as OSB architects and developers, are not in a good place right now. We’re at a migration point, and a migration means pain.

Let’s look at an example.

How an Upgrade to JSSE Failed (a True Story)

Trouble Starts: Adding a Service with a Cutting-Edge Certificate

A few months ago, I received a request to assess adding yet another service to my client’s OSB domain. With more than 200 services already, how hard could it be?

It turned out, it was quite hard.

The new service was using a modern certificate, with a modern signature algorithm that Certicom couldn’t handle. The connection couldn’t be established, and the log contained this entry:

Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11

This OID refers to “SHA256 with RSA Encryption”, which was too new for Certicom.

Knee-Jerk Reaction: Let’s Switch to JSSE!

When it became clear that the Certicom implementation was the culprit, the first and immediate idea was to re-configure the domain to use JSSE. This is very easy to do in Weblogic: just go to the Admin Console, and for each managed server click “Enable JSSE” under the SSL tab - and reboot.

And so we did, and Weblogic was able to connect to the new service, and everyone lived happily ever after… well, no. This is not a fairy tale. It is an IT story, which rarely have happy endings.

Weblogic, indeed, was now able to connect to the new service, that much is true.

But after a global change like this one we’ve got to do a regression testing, which pretty quickly demonstrated that two other services had ceased to work.

You Cannot Support Both Old and New Services

The two failed services were some of our oldest and most important ones.

Using built-in JSSE debug properties, OpenSSL executable and a custom plain Java JSSE application, we pretty quickly found that:

Service A simply does not support anything but SSL v3, and fails to recognize the TLS v1.0 handshake. Forcing the protocol to SSL v3 caused some other services to fail.

Service B supports TLS v1.0, but still requires the client (OSB) to start the handshake with the SSLv2Hello, the message that was removed from JSSE in JRE 1.7.

Dead end.

We could not switch to JSSE without breaking those two services, and we could not add the new service without switching to JSSE.

Another point to consider is that we could easily test the backend services connectivity, but we could not know how many OSB clients would not be able to connect. The required regression testing had to involve every OSB service consumer, and this was a huge effort to coordinate.

We had to back out from the project, and advise the consumer to connect to the service via a different channel.

It’s Going to Happen More Often

Backing out is not a solution, of course.

With time, more and more services with modern security layer implementations and modern certificates are going to pop up. We will have to accommodate them, or cease to exist (what is the point of a service bus that cannot bus the services?).

Hence, our plan is to educate and help the outdated services to upgrade their SSL configuration so at some point in future we can switch to JSSE.

There is a trick: we’re going to enlist the corporate security team as our ally. They should be able to force the cessation of the use of outdated and insecure SSL protocols, up to and including SSL v3, the use of insecure hashes (such as MD5 and SHA1) for signatures and the use of short keys and ciphers. It is much easier to request a change if it is for security concerns rather than due to vendor implementation shortcomings.

It will take time, probably a year or more. Things move slowly in the enterprise world.

But we will get there.