How to bind ssl cert with iodine + Rails?

[danielnc]

Hi

We are moving from puma to iodine because of performance and proper websocket creation with anycable/action_cable

I am wondering what would be the proper way to setup rails + iodine since I couldn't find a lot of documentation around it:

Is this enough?

bundle exec rails s -b 'ssl://127.0.0.1:3443?key=config/ssl/localhost.key&cert=config/ssl/localhost.crt'

Do I need to change config.ru from rails or do anything else? Again I couldn't find a lot of documentation on how to properly setup the server

Another thing is that I've created this initializer file:

# Iodine setup - use conditional setup to allow command-line arguments to override these:
if defined?(Iodine)
  Iodine.threads = ENV.fetch("RAILS_MAX_THREADS", 64).to_i if Iodine.threads.zero?
  Iodine.workers = ENV.fetch("WEB_CONCURRENCY", 1).to_i if Iodine.workers.zero?
  Iodine::DEFAULT_SETTINGS[:port] = ENV.fetch("PORT", 3000).to_i
end

But I can't find a way to setup ssl here
Any suggestions on how to do this, the rails way?

[boazsegev]

Hi @danielnc ,

Thank you for your question and for your interest in iodine.

In general, the iodine command line interface (CLI) doesn't necessarily match Puma. You will need to use the iodine CLI for the command line arguments.

To get all available information about the iodine CLI usage, run (in the terminal):

iodine -h

SSL/TLS Command Line Interface (CLI)

For SSL/TLS certificates, the best CLI approach would be:

bundle exec iodine -p 3443 -key config/ssl/localhost.key -cert config/ssl/localhost.crt

Or (with a specific IP binding):

bundle exec iodine -p 3443 -b 127.0.0.1 -key config/ssl/localhost.key -cert config/ssl/localhost.crt

SSL/TLS from Ruby

The easiest way would be to update the Iodine::DEFAULT_SETTINGS hash with the values you need. i.e.:

Iodine::DEFAULT_SETTINGS[:tls] = Iodine::TLS.new(
                                             private_key: "config/ssl/localhost.key",
                                             certificate: "config/ssl/localhost.crt")

The Iodine::DEFAULT_SETTINGS hash accepts the same values as the Iodine.listen class method.

Other stuff to do

There's really little to do in order to use iodine as a drop in replacement for Puma.

You would need to replace any calls to Puma's on_worker_boot / before_fork with a call to the Iodine.on_state method, i.e.:

Iodine.on_state(:before_fork) do
    # whatever
end

Of course, iodine will perform better when using the built-in pub/sub and WebSocket layer (rather than using ActionCable). Also, iodine provides timers and task deferral methods that could also prove useful.

You might consider running the Iodine.patch_rack class method in the initializer file. It should improve Rack's performance (if you don't already patch Rack in other ways).

---

Good luck with the transition 👍🏻

If you need anything else, let me know.

Kindly,
Bo.

[danielnc]

Bo, thanks for the info!!

I am in the works for trying to get this deployed to our staging env and I am facing some problems with SSL cert.

I don't see any alerts of failures but whenever I try to check if the connection is secure I am getting the following errors:

# curl -k  -v https://0.0.0.0:3443/monitoring.json
* Expire in 0 ms for 6 (transfer 0x7f58c749f000)
*   Trying 0.0.0.0...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x7f58c749f000)
* Connected to 0.0.0.0 (127.0.0.1) port 3443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, handshake failure (552):
* error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
* Closing connection 0
curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

This is the logs and we can't find any issue

Linking secrets
Executing: RAILS_ENV=staging MALLOC_CONF=narenas:2 SSL_KEY_PATH=/certs/cert.key SSL_CERT_PATH=/certs/cert.crt WEB_CONCURRENCY=1 RAILS_MAX_THREADS=64 PORT=3443 exec bundle exec rails server
=> Booting Rack
=> Rails 6.0.3 application starting in staging http://0.0.0.0:3443
=> Run `rails server --help` for more startup options
INFO: Listening on port 3443
INFO: Starting up Iodine:
 * Iodine 0.7.39
 * Ruby 2.6.5
 * facil.io 0.7.4 (epoll)
 * 1 Workers X 64 Threads per worker.
 * Maximum 131056 open files / sockets per worker.
 * Master (root) process: 11.
INFO: Server is running 1 worker X 64 threads with facil.io 0.7.4 (epoll)
* Detected capacity: 131056 open file limit
* Root pid: 11

Maybe it's because of binding to http:// and we are trying to curl https://?

If I try to curl http:// based on the binding the curl output is different but still nothing on the server

# curl -v http://0.0.0.0:3443/monitoring.json
* Expire in 0 ms for 6 (transfer 0x7fc09209f000)
*   Trying 0.0.0.0...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x7fc09209f000)
* Connected to 0.0.0.0 (127.0.0.1) port 3443 (#0)
> GET /admin/monitoring/database.json HTTP/1.1
> Host: 0.0.0.0:3443
> User-Agent: curl/7.64.0
> Accept: */*
> 
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

Help will be greatly appreciated :)

[boazsegev]

Hi @danielnc ,

I'm not sure what's going on specifically.

I will need to be able to replicate the issue in order to debug this. It would be great if you could post a small code example that has the same issue.

However, I noticed your curl is attempting to handshake on TLS 1.3. I believe that iodine and the OpenSSL gem currently run TLS 1.2. Though iodine sets 1.2 as a minimal version, so I'm not sure.

Also, are you using nginx or some other reverse proxy in front of iodine? - if you are (and I hope you are), it would make better sense to bind iodine to the reverse proxy using a Unix Socket or some other way.

For me, when I run iodine with a self signed certificate, it runs okay (iodine -tls -t1)... perhaps this is related to the certificate type...

Kindly,
Bo.

[boazsegev]

Hi @danielnc ,

Just a quick update: I tested iodine with TLS 1.3 and it works... which means that I'm not sure at all where the issue you're experiencing might lay. It might be in the way you initialize things. Maybe the certificate requires a password... I'm not sure.

Could you test and tell me:

• Does the it work with an anonymous (self-signed) certificate (iodine -tls)?

• Does the it work with a Command Line supplied certificate path (see above)?

Also - could you run iodine with -V5 to enable Debug level logging (or use Iodine.verbosity = 5)? This will add information to the logs.

Kindly,
Bo.

P.S.

Make suer to upgrade to Iodine 0.7.40, it has TLS specific patches.

[raivil]

Hey @boazsegev,
I'm helping Daniel with those tests.

I was able to have it running with self-signed certificates (expired and valid). It works as expected. I've tried using env vars and parameters to specify the certificates and other configurations, and with/without -tls. All that I tested seems to work as expected.
Repeated the test with a newly created self-signed certificate and it also worked.

However, when using a valid certificate it does not work.
Even in Chrome, I get the error ERR_SSL_VERSION_OR_CIPHER_MISMATCH for the valid certificate only.

Executed tests with OpenSSL and curl.
The results for OpenSSL are below.

Simple ssl connect test, executed from the same container, and from a load balancer. No Nginx proxy at this time, since the basic webserver is not working with a direct connection.

Ps: Updated to Iodine 0.7.40.

# openssl s_client -connect 0.0.0.0:3443
CONNECTED(00000003)
140490970189632:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1544:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 283 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

Another result from a dev machine:

openssl s_client -connect X.X.X.X:443 (this is going through a load balancer)
CONNECTED(00000003)
4542955116:error:14004410:SSL routines:CONNECT_CR_SRVR_HELLO:sslv3 alert handshake failure:/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-47.100.4/libressl-2.8/ssl/ssl_pkt.c:1200:SSL alert number 40
4542955116:error:140040E5:SSL routines:CONNECT_CR_SRVR_HELLO:ssl handshake failure:/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-47.100.4/libressl-2.8/ssl/ssl_pkt.c:585:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Start Time: 1590607692
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

Logs from Iodine (snippet);

iodine -p 3443 -V 5 -key /path_to_certs/my.key -cert /path_to_certs/my.crt -t 64 -w 1 -b 0.0.0.0
DEBUG (fio_tls_openssl.c:373): destroyed TLS context for OpenSSL 0x7fcf0e448f00
DEBUG (fio_tls_openssl.c:516): (re)built TLS context for OpenSSL 0x7fcf0e448f00
DEBUG (fio_tls_openssl.c:373): destroyed TLS context for OpenSSL 0x7fcf0e448f00
DEBUG (fio_tls_openssl.c:415): TLS read private key from PEM file.
DEBUG (fio_tls_openssl.c:433): TLS adding certificate from PEM file.
DEBUG (fio_tls_openssl.c:433): TLS adding certificate from PEM file.
DEBUG (fio_tls_openssl.c:516): (re)built TLS context for OpenSSL 0x7fcf0e448f00
DEBUG (iodine_store.c:56): Ruby <=> C Memory storage stats (pid: 12):

....delete a lot of repeated logs...

DEBUG (fio_tls_openssl.c:373): destroyed TLS context for OpenSSL 0x7fcf0e448f00
DEBUG (fio_tls_openssl.c:415): TLS read private key from PEM file.
DEBUG (fio_tls_openssl.c:433): TLS adding certificate from PEM file.
DEBUG (fio_tls_openssl.c:433): TLS adding certificate from PEM file.
DEBUG (fio_tls_openssl.c:516): (re)built TLS context for OpenSSL 0x7fcf0e448f00
DEBUG (fio.c:327): FD 5 re-initialized (state: 0x502-open).
INFO: Listening on port 3443
INFO: Starting up Iodine:
 * Iodine 0.7.40
 * Ruby 2.6.5
 * facil.io 0.7.4 (epoll)
 * 1 Workers X 64 Threads per worker.
 * Maximum 131056 open files / sockets per worker.
 * Master (root) process: 12.

DEBUG (fio.c:327): FD 13 re-initialized (state: 0xd02-open).
DEBUG (fio.c:6219): (12) Listening to cluster: /tmp/facil-io-sock-12
INFO: Server is running 1 worker X 64 threads with facil.io 0.7.4 (epoll)
* Detected capacity: 131056 open file limit
* Root pid: 12
* Press ^C to stop

DEBUG (iodine_defer.c:50): IO thread started.
DEBUG (fio.c:4499): (12) started listening on port 3443
DEBUG (iodine_store.c:56): Ruby <=> C Memory storage stats (pid: 12):

When executing openssl s_client -connect, this is the log.

DEBUG (fio.c:327): FD 15 re-initialized (state: 0xf04-open).
DEBUG (fio_tls_openssl.c:868): Attaching TLS read/write hook for 0xf04 (server mode).
DEBUG (fio_tls_openssl.c:703): SSL_accept/SSL_connect 0xf04 error: SSL_ERROR_SSL (non SSL attempt?)
DEBUG (fio_tls_openssl.c:647): TLS cleanup for 0xf04
DEBUG (fio.c:327): FD 15 re-initialized (state: 0xf05-closed).

So far I have no idea of what is the issue.
The same valid certificate works on Puma directly as it was tested with curl and openssl.
Any help here is appreciated.

Best,
Ronaldo.

[boazsegev]

Hi @raivil ,

Thank you for joining the conversation and for your help.

I'm sorry if my responses are somewhat delayed, I have a big deadline on June 3rd and then I'll be playing catch-up for a while (trying to catch up with everything that was placed on hold).

It appears from the logs that iodine is adding 2 certificates and a single key:

DEBUG (fio_tls_openssl.c:415): TLS read private key from PEM file.
DEBUG (fio_tls_openssl.c:433): TLS adding certificate from PEM file.
DEBUG (fio_tls_openssl.c:433): TLS adding certificate from PEM file.

It is possible that your certificate file contains more certificates than keys, i.e., a certificate chain, containing the public certificate for the authority as well as your own certificate.

However, iodine loads all these certificates as happens here:

iodine/ext/iodine/fio_tls_openssl.c

Lines 429 to 440 in 18d0aa1

	for (int i = 0; i < sk_X509_INFO_num(inf); ++i) {
	/* for each element in PEM */
	X509_INFO *tmp = sk_X509_INFO_value(inf, i);
	if (tmp->x509) {
	FIO_LOG_DEBUG("TLS adding certificate from PEM file.");
	SSL_CTX_use_certificate(tls->ctx, tmp->x509);
	}
	if (tmp->x_pkey) {
	FIO_LOG_DEBUG("TLS adding private key from PEM file.");
	SSL_CTX_use_PrivateKey(tls->ctx, tmp->x_pkey->dec_pkey);
	}
	}

This might be resolved by making the loop run only once... one option would be to comment out the loop instructions like this:

          // for (int i = 0; i < sk_X509_INFO_num(inf); ++i) { // <- !!!commented out!!!
            /* for each element in PEM */
            X509_INFO *tmp = sk_X509_INFO_value(inf, i);
            if (tmp->x509) {
              FIO_LOG_DEBUG("TLS adding certificate from PEM file.");
              SSL_CTX_use_certificate(tls->ctx, tmp->x509);
            }
            if (tmp->x_pkey) {
              FIO_LOG_DEBUG("TLS adding private key from PEM file.");
              SSL_CTX_use_PrivateKey(tls->ctx, tmp->x_pkey->dec_pkey);
            }
          // }   // <- !!!commented out!!!

However, I'm not sure if this will break multi-domain certificates.

I'll have to dig into this. For now, you could fork the repo and try commenting the lines out and see if that helps.

Once I have more time, I'll try a few things myself.

Kindly,
Bo.

[raivil]

Hey @boazsegev,

Thank you so much for the help so far. Right now I'm out of bandwidth to fork the repo and execute the suggested tests.
I'll stick to Puma until this issue is fixed and will be actively monitoring Iodine releases.

Best Regards,
Ronaldo