This is a comparison between the connection pooling options and strategies of the Blaze and Ember http4s client backends.

Blaze

This information is current as of http4s-blaze-client-0.23.16.

Properties

• maxTotalConnections: the maximum number of connections a Blaze client will open at one time overall. Requests to any host count against this limit. Default: 10

• maxConnectionsPerRequestKey: the maximum number of connections a Blaze client will open at one time to a particular request key. A request key is the scheme (i.e., http or https), the host, and the port. This setting is represented as a function, so different request keys may have different limits. Default: _ => 256

• maxWaitQueueLimit: the number of requests that are queued by the client when a connection can’t be borrowed or created. Default: 256

• requestTimeout: maximum duration between the submission of a request and reading its entire body. Default: 45.seconds

• responseHeaderTimeout: maximum duration between the submission of a request and the reception of the response prelude (status line and headers, but not body) from the server. Default: Duration.Inf

Connection logic

1. When a request is under the limit of both maxTotalConnections and maxConnectionsPerRequestKey, it is submitted immediately.

2. If no connection can be created or borrowed due to either maxTotalConnections or maxConnectionsPerRequestKey, and the length of the wait queue is less than maxWaitQueueLimit, the request is enqueued to the wait queue.

3. If no connection can be created or borrowed and the wait queue is full, a WaitQueueFullFailure error is raised as the client’s response.

4. When a recyclable connection is released to the pool, the first wait-queued request with the same request key as the recycled connection is considered.

   1. If the request is expired – that is, if either requestTimeout or responseHeaderTimeout has passed since the request was wait queued – a WaitQueueTimeoutException error is raised as the client’s response, and the request is removed from the queue. The process is repeated, considering the next wait-queued request with the same key.

   2. If the request is unexpired, it is removed from the wait queue and submitted.

   3. If no request of the same key is wait-queued, all wait-queued requests are checked for expiry, and removed and completed with a WaitQueueTimeoutException. If there exists an unexpired wait-queued request for which a connection can be created under the maxConnectionsPerRequestKey limit, the recycled connection is disposed and the first such request is submitted with a freshly created connection.

5. When a non-recyclable connection is disposed, all wait-queued requests are checked for expiry, and removed and completed with a WaitQueueTimeoutException. The first unexpired request for which a connection can be added under the maxConnectionsPerRequestKey limit is removed from the wait queue and submitted with a fresh connection.

Wait queue

Implications

• Fails fast when the wait queue is full, but may pessimistically fail requests that might otherwise have succeeded.

• Improves tail latency(high percentile timings of requests that have a response time longer than 99.xxx percent), but may service fewer requests before timeout.

• Increases connection reuse on “hot” request keys, at the expense of latency on others.

Help, I’m getting too many =WaitQueueFullFailure=s

• Your max connections (total or per key) are too small: you might be able to achieve more throughput by increasing parallelism, so each request is more likely to have an available connection instead of filling up the wait queue.

• Your max connections (total or per key) are too large: the system might be overtaxed, and adding parallelism would decrease efficiency to the point that overall throughput is reduced. Even though more connections are served at once, they’re each taking so much longer that more requests end up in the wait queue.

• Your wait queue is too small: you are in the middle of a burst that you might have been able to handle before timing out, but you didn’t even try.

• Your retry strategy is too aggressive: you’re already underwater, and failures are coming back to you too quickly.

Help, I’m getting too many =WaitQueueTimeoutException=s

• Your max connections (total or per key) are too small: you’d get better throughput, and spend less of your timeout in the queue, with more parallelism.

• Your max connections (total or per key) are too large: the system is oversaturated, and more parallelism will slow things down further.

• Your wait queue is too large: bad news is best delivered early, and you’re not doing anyone any favors pretending you can handle this size burst.

• Your timeouts are too short: if whoever called you is willing to wait longer, carve out more time for yourself.

Ember

This information is current as of http4s-ember-client-0.23.27.

Properties

• maxTotal: the maximum number of connections an Ember client will open at one time overall. Requests to any host count against this limit. Default: 100

• maxPerKey: the maximum number of connections an Ember client will pool at one time to a particular request key. A request key is the scheme (i.e., http or https), the host, and the port. This setting is represented as a function, so different request keys may have different limits. This is a soft limit, enforced only when a connection is returned to the pool. Default: _ => 100

• idleTimeInPool: how long an idle connection will be maintained in the Ember connection pool. Default: 30.seconds

Connection logic

Connection management largely delegates to Keypool.

1. When there are less than maxTotal requests open, the request is submitted immediately. Note that maxPerKey is irrelevant: a new connection is allocated as long as maxTotal is not exceeded.

2. If no connection can be created or borrowed due to either maxTotal, the request semantically blocks on a semaphore until a relevant connection is recycled or disposed. The semaphore is effectively an unbounded wait queue.

Those pools make me sad, can I build my own?

Sure. You have the superpowers of Cats Effect and can always make a client middleware. There are a lot of clever things that you might do, including:

• a stack (“LIFO” order)
• a priority queue

There are some neat interactive demos of these techniques to inspire you.

The caveat is that both the Blaze and Ember connection pools have access to the request key for each connection that is recycled or closed, and can perform optimizations that are unavailable to a generic client wrapper (e.g., Blaze prioritizing queued requests for the same host when a connection is recycled). Building more flexible strategies into Keypool is worth consideration. Pull requests welcome.

Mapping between clients