HTTP Client

HttpClient Introduction

The Jetty HTTP client module provides easy-to-use APIs and utility classes to perform HTTP (or HTTPS) requests.

Jetty’s HTTP client is non-blocking and asynchronous. It offers an asynchronous API that never blocks for I/O, making it very efficient in thread utilization and well suited for high performance scenarios such as load testing or parallel computation.

However, when all you need to do is to perform a GET request to a resource, Jetty’s HTTP client offers also a synchronous API; a programming interface where the thread that issued the request blocks until the request/response conversation is complete.

Jetty’s HTTP client supports different transports protocols: HTTP/1.1, HTTP/2, HTTP/3 and FastCGI. This means that the semantic of an HTTP request such as: " GET the resource /index.html " can be carried over the network in different formats. The most common and default format is HTTP/1.1. That said, Jetty’s HTTP client can carry the same request using the HTTP/2 format, the HTTP/3 format, or the FastCGI format.

Furthermore, every transport protocol can be sent either over the network or via Unix-Domain sockets. Supports for Unix-Domain sockets requires Java 16 or later, since Unix-Domain sockets support has been introduced in OpenJDK with JEP 380.

The FastCGI transport is heavily used in Jetty’s FastCGI support that allows Jetty to work as a reverse proxy to PHP (exactly like Apache or Nginx do) and therefore be able to serve, for example, WordPress websites, often in conjunction with Unix-Domain sockets (although it’s possible to use FastCGI via network too).

The HTTP/2 transport allows Jetty’s HTTP client to perform requests using HTTP/2 to HTTP/2 enabled web sites, see also Jetty’s HTTP/2 support.

The HTTP/3 transport allows Jetty’s HTTP client to perform requests using HTTP/3 to HTTP/3 enabled web sites, see also Jetty’s HTTP/3 support.

Out of the box features that you get with the Jetty HTTP client include:

Redirect support — redirect codes such as 302 or 303 are automatically followed.
Cookies support — cookies sent by servers are stored and sent back to servers in matching requests.
Authentication support — HTTP "Basic", "Digest" and "SPNEGO" authentications are supported, others are pluggable.
Forward proxy support — HTTP proxying and SOCKS4 proxying.

Starting HttpClient

The Jetty artifact that provides the main HTTP client implementation is jetty-client. The Maven artifact coordinates are the following:

<dependency>
  <groupId>org.eclipse.jetty</groupId>
  <artifactId>jetty-client</artifactId>
  <version>10.0.24-SNAPSHOT</version>
</dependency>

The main class is named org.eclipse.jetty.client.HttpClient.

You can think of a HttpClient instance as a browser instance. Like a browser it can make requests to different domains, it manages redirects, cookies and authentication, you can configure it with a proxy, and it provides you with the responses to the requests you make.

In order to use HttpClient, you must instantiate it, configure it, and then start it:

// Instantiate HttpClient.
HttpClient httpClient = new HttpClient();

// Configure HttpClient, for example:
httpClient.setFollowRedirects(false);

// Start HttpClient.
httpClient.start();

You may create multiple instances of HttpClient, but typically one instance is enough for an application. There are several reasons for having multiple HttpClient instances including, but not limited to:

You want to specify different configuration parameters (for example, one instance is configured with a forward proxy while another is not).
You want the two instances to behave like two different browsers and hence have different cookies, different authentication credentials, etc.
You want to use different transports.

Like browsers, HTTPS requests are supported out-of-the-box (see this section for the TLS configuration), as long as the server provides a valid certificate. In case the server does not provide a valid certificate (or in case it is self-signed) you want to customize HttpClient's TLS configuration as described in this section.

Stopping HttpClient

It is recommended that when your application stops, you also stop the HttpClient instance (or instances) that you are using.

// Stop HttpClient.
httpClient.stop();

Stopping HttpClient makes sure that the memory it holds (for example, authentication credentials, cookies, etc.) is released, and that the thread pool and scheduler are properly stopped allowing all threads used by HttpClient to exit.

You cannot call HttpClient.stop() from one of its own threads, as it would cause a deadlock. It is recommended that you stop HttpClient from an unrelated thread, or from a newly allocated thread, for example:

// Stop HttpClient from a new thread.
// Use LifeCycle.stop(...) to rethrow checked exceptions as unchecked.
new Thread(() -> LifeCycle.stop(httpClient)).start();

HttpClient Architecture

A HttpClient instance can be thought as a browser instance, and it manages the following components:

a CookieStore (see this section).
a AuthenticationStore (see this section).
a ProxyConfiguration (see this section).
a set of destinations.

A destination is the client-side component that represents an origin server, and manages a queue of requests for that origin, and a pool of TCP connections to that origin.

An origin may be simply thought as the tuple (scheme, host, port) and it is where the client connects to in order to communicate with the server. However, this is not enough.

If you use HttpClient to write a proxy you may have different clients that want to contact the same server. In this case, you may not want to use the same proxy-to-server connection to proxy requests for both clients, for example for authentication reasons: the server may associate the connection with authentication credentials and you do not want to use the same connection for two different users that have different credentials. Instead, you want to use different connections for different clients and this can be achieved by "tagging" a destination with a tag object that represents the remote client (for example, it could be the remote client IP address).

Two origins with the same (scheme, host, port) but different tag create two different destinations and therefore two different connection pools. However, also this is not enough.

It is possible for a server to speak different protocols on the same port. A connection may start by speaking one protocol, for example HTTP/1.1, but then be upgraded to speak a different protocol, for example HTTP/2. After a connection has been upgraded to a second protocol, it cannot speak the first protocol anymore, so it can only be used to communicate using the second protocol.

Two origins with the same (scheme, host, port) but different protocol create two different destinations and therefore two different connection pools.

Therefore an origin is identified by the tuple (scheme, host, port, tag, protocol).

HttpClient Connection Pooling

A destination manages a org.eclipse.jetty.client.ConnectionPool, where connections to a particular origin are pooled for performance reasons: opening a connection is a costly operation and it’s better to reuse them for multiple requests.

Remember that to select a specific destination you must select a specific origin, and that an origin is identified by the tuple (scheme, host, port, tag, protocol), so you can have multiple destinations for the same host and port.

You can access the ConnectionPool in this way:

HttpClient httpClient = new HttpClient();
httpClient.start();

ConnectionPool connectionPool = httpClient.getDestinations().stream()
    // Cast to HttpDestination.
    .map(HttpDestination.class::cast)
    // Find the destination by filtering on the Origin.
    .filter(destination -> destination.getOrigin().getAddress().getHost().equals("domain.com"))
    .findAny()
    // Get the ConnectionPool.
    .map(HttpDestination::getConnectionPool)
    .orElse(null);

Jetty’s client library provides the following ConnectionPool implementations:

DuplexConnectionPool, historically the first implementation, only used by the HTTP/1.1 transport.
MultiplexConnectionPool, the generic implementation valid for any transport where connections are reused with a MRU (most recently used) algorithm (that is, the connections most recently returned to the connection pool are the more likely to be used again).
RoundRobinConnectionPool, similar to MultiplexConnectionPool but where connections are reused with a round-robin algorithm.

The ConnectionPool implementation can be customized for each destination in by setting a ConnectionPool.Factory on the HttpClientTransport:

HttpClient httpClient = new HttpClient();
httpClient.start();

// The max number of connections in the pool.
int maxConnectionsPerDestination = httpClient.getMaxConnectionsPerDestination();

// The max number of requests per connection (multiplexing).
// Start with 1, since this value is dynamically set to larger values if
// the transport supports multiplexing requests on the same connection.
int maxRequestsPerConnection = 1;

HttpClientTransport transport = httpClient.getTransport();

// Set the ConnectionPool.Factory using a lambda.
transport.setConnectionPoolFactory(destination ->
    new RoundRobinConnectionPool(destination,
        maxConnectionsPerDestination,
        destination,
        maxRequestsPerConnection));

HttpClient Request Processing

When a request is sent, an origin is computed from the request; HttpClient uses that origin to find (or create if it does not exist) the correspondent destination. The request is then queued onto the destination, and this causes the destination to ask its connection pool for a free connection. If a connection is available, it is returned, otherwise a new connection is created. Once the destination has obtained the connection, it dequeues the request and sends it over the connection.

The first request to a destination triggers the opening of the first connection. A second request with the same origin sent after the first request/response cycle is completed may reuse the same connection, depending on the connection pool implementation. A second request with the same origin sent concurrently with the first request will likely cause the opening of a second connection, depending on the connection pool implementation. The configuration parameter HttpClient.maxConnectionsPerDestination (see also the configuration section) controls the max number of connections that can be opened for a destination.

If opening connections to a given origin takes a long time, then requests for that origin will queue up in the corresponding destination until the connections are established.

Each connection can handle a limited number of concurrent requests. For HTTP/1.1, this number is always 1: there can only be one outstanding request for each connection. For HTTP/2 this number is determined by the server max_concurrent_stream setting (typically around 100, i.e. there can be up to 100 outstanding requests for every connection).

When a destination has maxed out its number of connections, and all connections have maxed out their number of outstanding requests, more requests sent to that destination will be queued. When the request queue is full, the request will be failed. The configuration parameter HttpClient.maxRequestsQueuedPerDestination (see also the configuration section) controls the max number of requests that can be queued for a destination.

HttpClient API Usage

HttpClient provides two types of APIs: a blocking API and a non-blocking API.

HttpClient Blocking APIs

The simpler way to perform a HTTP request is the following:

HttpClient httpClient = new HttpClient();
httpClient.start();

// Perform a simple GET and wait for the response.
ContentResponse response = httpClient.GET("http://domain.com/path?query");

The method HttpClient.GET(…) performs a HTTP GET request to the given URI and returns a ContentResponse when the request/response conversation completes successfully.

The ContentResponse object contains the HTTP response information: status code, headers and possibly content. The content length is limited by default to 2 MiB; for larger content see the section on response content handling.

If you want to customize the request, for example by issuing a HEAD request instead of a GET, and simulating a browser user agent, you can do it in this way:

ContentResponse response = httpClient.newRequest("http://domain.com/path?query")
    .method(HttpMethod.HEAD)
    .agent("Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:17.0) Gecko/20100101 Firefox/17.0")
    .send();

This is a shorthand for:

Request request = httpClient.newRequest("http://domain.com/path?query");
request.method(HttpMethod.HEAD);
request.agent("Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:17.0) Gecko/20100101 Firefox/17.0");
ContentResponse response = request.send();

You first create a request object using httpClient.newRequest(…), and then you customize it using the fluent API style (that is, a chained invocation of methods on the request object). When the request object is customized, you call request.send() that produces the ContentResponse when the request/response conversation is complete.

The Request object, despite being mutable, cannot be reused for other requests. This is true also when trying to send two or more identical requests: you have to create two or more Request objects.

Simple POST requests also have a shortcut method:

ContentResponse response = httpClient.POST("http://domain.com/entity/1")
    .param("p", "value")
    .send();

The POST parameter values added via the param() method are automatically URL-encoded.

Jetty’s HttpClient automatically follows redirects, so it handles the typical web pattern POST/Redirect/GET, and the response object contains the content of the response of the GET request. Following redirects is a feature that you can enable/disable on a per-request basis or globally.

File uploads also require one line, and make use of java.nio.file classes:

ContentResponse response = httpClient.POST("http://domain.com/upload")
    .file(Paths.get("file_to_upload.txt"), "text/plain")
    .send();

It is possible to impose a total timeout for the request/response conversation using the Request.timeout(…) method as follows:

ContentResponse response = httpClient.newRequest("http://domain.com/path?query")
    .timeout(5, TimeUnit.SECONDS)
    .send();

In the example above, when the 5 seconds expire, the request/response cycle is aborted and a java.util.concurrent.TimeoutException is thrown.

HttpClient Non-Blocking APIs

So far we have shown how to use Jetty HTTP client in a blocking style — that is, the thread that issues the request blocks until the request/response conversation is complete.

This section will look at Jetty’s HttpClient non-blocking, asynchronous APIs that are perfectly suited for large content downloads, for parallel processing of requests/responses and in cases where performance and efficient thread and resource utilization is a key factor.

The asynchronous APIs rely heavily on listeners that are invoked at various stages of request and response processing. These listeners are implemented by applications and may perform any kind of logic. The implementation invokes these listeners in the same thread that is used to process the request or response. Therefore, if the application code in these listeners takes a long time to execute, the request or response processing is delayed until the listener returns.

If you need to execute application code that takes long time inside a listener, you must spawn your own thread.

Request and response processing are executed by two different threads and therefore may happen concurrently. A typical example of this concurrent processing is an echo server, where a large upload may be concurrent with the large download echoed back.

Remember that responses may be processed and completed before requests; a typical example is a large upload that triggers a quick response, for example an error, by the server: the response may arrive and be completed while the request content is still being uploaded.

The application thread that calls Request.send(Response.CompleteListener) performs the processing of the request until either the request is fully sent over the network or until it would block on I/O, then it returns (and therefore never blocks). If it would block on I/O, the thread asks the I/O system to emit an event when the I/O will be ready to continue, then returns. When such an event is fired, a thread taken from the HttpClient thread pool will resume the processing of the request.

Response are processed from the I/O thread taken from the HttpClient thread pool that processes the event that bytes are ready to be read. Response processing continues until either the response is fully processed or until it would block for I/O. If it would block for I/O, the thread asks the I/O system to emit an event when the I/O will be ready to continue, then returns. When such an event is fired, a (possibly different) thread taken from the HttpClient thread pool will resume the processing of the response.

When the request and the response are both fully processed, the thread that finished the last processing (usually the thread that processes the response, but may also be the thread that processes the request — if the request takes more time than the response to be processed) is used to dequeue the next request for the same destination and to process it.

A simple non-blocking GET request that discards the response content can be written in this way:

httpClient.newRequest("http://domain.com/path")
    .send(result ->
    {
        // Your logic here
    });

Method Request.send(Response.CompleteListener) returns void and does not block; the Response.CompleteListener lambda provided as a parameter is notified when the request/response conversation is complete, and the Result parameter allows you to access the request and response objects as well as failures, if any.

You can impose a total timeout for the request/response conversation in the same way used by the synchronous API:

httpClient.newRequest("http://domain.com/path")
    .timeout(3, TimeUnit.SECONDS)
    .send(result ->
    {
        /* Your logic here */
    });

The example above will impose a total timeout of 3 seconds on the request/response conversation.

The HTTP client APIs use listeners extensively to provide hooks for all possible request and response events:

httpClient.newRequest("http://domain.com/path")
    // Add request hooks.
    .onRequestQueued(request -> { /* ... */ })
    .onRequestBegin(request -> { /* ... */ })
    .onRequestHeaders(request -> { /* ... */ })
    .onRequestCommit(request -> { /* ... */ })
    .onRequestContent((request, content) -> { /* ... */ })
    .onRequestFailure((request, failure) -> { /* ... */ })
    .onRequestSuccess(request -> { /* ... */ })
    // Add response hooks.
    .onResponseBegin(response -> { /* ... */ })
    .onResponseHeader((response, field) -> true)
    .onResponseHeaders(response -> { /* ... */ })
    .onResponseContentAsync((response, buffer, callback) -> callback.succeeded())
    .onResponseFailure((response, failure) -> { /* ... */ })
    .onResponseSuccess(response -> { /* ... */ })
    // Result hook.
    .send(result -> { /* ... */ });

This makes Jetty HTTP client suitable for HTTP load testing because, for example, you can accurately time every step of the request/response conversation (thus knowing where the request/response time is really spent).

Have a look at the Request.Listener class to know about request events, and to the Response.Listener class to know about response events.

Request Content Handling

Jetty’s HttpClient provides a number of utility classes off the shelf to handle request content.

You can provide request content as String, byte[], ByteBuffer, java.nio.file.Path, InputStream, and provide your own implementation of org.eclipse.jetty.client.api.Request.Content. Here’s an example that provides the request content using java.nio.file.Paths:

ContentResponse response = httpClient.POST("http://domain.com/upload")
    .body(new PathRequestContent("text/plain", Paths.get("file_to_upload.txt")))
    .send();

Alternatively, you can use FileInputStream via the InputStreamRequestContent utility class:

ContentResponse response = httpClient.POST("http://domain.com/upload")
    .body(new InputStreamRequestContent("text/plain", new FileInputStream("file_to_upload.txt")))
    .send();

Since InputStream is blocking, then also the send of the request will block if the input stream blocks, even in case of usage of the non-blocking HttpClient APIs.

If you have already read the content in memory, you can pass it as a byte[] (or a String) using the BytesRequestContent (or StringRequestContent) utility class:

ContentResponse bytesResponse = httpClient.POST("http://domain.com/upload")
    .body(new BytesRequestContent("text/plain", bytes))
    .send();

ContentResponse stringResponse = httpClient.POST("http://domain.com/upload")
    .body(new StringRequestContent("text/plain", string))
    .send();

If the request content is not immediately available, but your application will be notified of the content to send, you can use AsyncRequestContent in this way:

AsyncRequestContent content = new AsyncRequestContent();
httpClient.POST("http://domain.com/upload")
    .body(content)
    .send(result ->
    {
        // Your logic here
    });

// Content not available yet here.

// An event happens in some other class, in some other thread.
class ContentPublisher
{
    void publish(ByteBufferPool bufferPool, byte[] bytes, boolean lastContent)
    {
        // Wrap the bytes into a new ByteBuffer.
        ByteBuffer buffer = ByteBuffer.wrap(bytes);

        // Offer the content, and release the ByteBuffer
        // to the pool when the Callback is completed.
        content.offer(buffer, Callback.from(() -> bufferPool.release(buffer)));

        // Close AsyncRequestContent when all the content is arrived.
        if (lastContent)
            content.close();
    }
}

While the request content is awaited and consequently uploaded by the client application, the server may be able to respond (at least with the response headers) completely asynchronously. In this case, Response.Listener callbacks will be invoked before the request is fully sent. This allows fine-grained control of the request/response conversation: for example the server may reject contents that are too big, send a response to the client, which in turn may stop the content upload.

Another way to provide request content is by using an OutputStreamRequestContent, which allows applications to write request content when it is available to the OutputStream provided by OutputStreamRequestContent:

OutputStreamRequestContent content = new OutputStreamRequestContent();

// Use try-with-resources to close the OutputStream when all content is written.
try (OutputStream output = content.getOutputStream())
{
    httpClient.POST("http://localhost:8080/")
        .body(content)
        .send(result ->
        {
            // Your logic here
        });

    // Content not available yet here.

    // Content is now available.
    byte[] bytes = new byte[]{'h', 'e', 'l', 'l', 'o'};
    output.write(bytes);
}
// End of try-with-resource, output.close() called automatically to signal end of content.

Response Content Handling

Jetty’s HttpClient allows applications to handle response content in different ways.

You can buffer the response content in memory; this is done when using the blocking APIs and the content is buffered within a ContentResponse up to 2 MiB.

If you want to control the length of the response content (for example limiting to values smaller than the default of 2 MiB), then you can use a org.eclipse.jetty.client.util.FutureResponseListener in this way:

Request request = httpClient.newRequest("http://domain.com/path");

// Limit response content buffer to 512 KiB.
FutureResponseListener listener = new FutureResponseListener(request, 512 * 1024);

request.send(listener);

// Wait at most 5 seconds for request+response to complete.
ContentResponse response = listener.get(5, TimeUnit.SECONDS);

If the response content length is exceeded, the response will be aborted, and an exception will be thrown by method get(…).

You can buffer the response content in memory also using the non-blocking APIs, via the BufferingResponseListener utility class:

httpClient.newRequest("http://domain.com/path")
    // Buffer response content up to 8 MiB
    .send(new BufferingResponseListener(8 * 1024 * 1024)
    {
        @Override
        public void onComplete(Result result)
        {
            if (!result.isFailed())
            {
                byte[] responseContent = getContent();
                // Your logic here
            }
        }
    });

If you want to avoid buffering, you can wait for the response and then stream the content using the InputStreamResponseListener utility class:

InputStreamResponseListener listener = new InputStreamResponseListener();
httpClient.newRequest("http://domain.com/path")
    .send(listener);

// Wait for the response headers to arrive.
Response response = listener.get(5, TimeUnit.SECONDS);

// Look at the response before streaming the content.
if (response.getStatus() == HttpStatus.OK_200)
{
    // Use try-with-resources to close input stream.
    try (InputStream responseContent = listener.getInputStream())
    {
        // Your logic here
    }
}
else
{
    response.abort(new IOException("Unexpected HTTP response"));
}

Finally, let’s look at the advanced usage of the response content handling.

The response content is provided by the HttpClient implementation to application listeners following a reactive model similar to that of java.util.concurrent.Flow.

The listener that follows this model is Response.DemandedContentListener.

After the response headers have been processed by the HttpClient implementation, Response.DemandedContentListener.onBeforeContent(response, demand) is invoked. This allows the application to control whether to demand the first content or not. The default implementation of this method calls demand.accept(1), which demands one chunk of content to the implementation. The implementation will deliver the chunk of content as soon as it is available.

The chunks of content are delivered to the application by invoking Response.DemandedContentListener.onContent(response, demand, buffer, callback). Applications implement this method to process the content bytes in the buffer. Succeeding the callback signals to the implementation that the application has consumed the buffer so that the implementation can dispose/recycle the buffer. Failing the callback signals to the implementation to fail the response (no more content will be delivered, and the response failed event will be emitted).

Succeeding the callback must be done only after the buffer bytes have been consumed. When the callback is succeeded, the HttpClient implementation may reuse the buffer and overwrite the bytes with different bytes; if the application looks at the buffer after having succeeded the callback is may see other, unrelated, bytes.

The application uses the demand object to demand more content chunks. Applications will typically demand for just one more content via demand.accept(1), but may decide to demand for more via demand.accept(2) or demand "infinitely" once via demand.accept(Long.MAX_VALUE). Applications that demand for more than 1 chunk of content must be prepared to receive all the content that they have demanded.

Demanding for content and consuming the content are orthogonal activities.

An application can demand "infinitely" and store aside the pairs (buffer, callback) to consume them later. If not done carefully, this may lead to excessive memory consumption, since the buffers are not consumed. Succeeding the callbacks will result in the buffers to be disposed/recycled and may be performed at any time.

An application can also demand one chunk of content, consume it (by succeeding the associated callback) and then not demand for more content until a later time.

Subclass Response.AsyncContentListener overrides the behavior of Response.DemandedContentListener; when an application implementing its onContent(response, buffer, callback) succeeds the callback, it will have both the effect of disposing/recycling the buffer and the effect of demanding one more chunk of content.

Subclass Response.ContentListener overrides the behavior of Response.AsyncContentListener; when an application implementing its onContent(response, buffer) returns from the method itself, it will both the effect of disposing/recycling the buffer and the effect of demanding one more chunk of content.

Previous examples of response content handling were inefficient because they involved copying the buffer bytes, either to accumulate them aside so that the application could use them when the request was completed, or because they were provided to an API such as InputStream that made use of byte[] (and therefore a copy from ByteBuffer to byte[] is necessary).

An application that implements a forwarder between two servers can be implemented efficiently by handling the response content without copying the buffer bytes as in the following example:

// Prepare a request to server1, the source.
Request request1 = httpClient.newRequest(host1, port1)
    .path("/source");

// Prepare a request to server2, the sink.
AsyncRequestContent content2 = new AsyncRequestContent();
Request request2 = httpClient.newRequest(host2, port2)
    .path("/sink")
    .body(content2);

request1.onResponseContentDemanded(new Response.DemandedContentListener()
{
    @Override
    public void onBeforeContent(Response response, LongConsumer demand)
    {
        request2.onRequestCommit(request ->
        {
            // Only when the request to server2 has been sent,
            // then demand response content from server1.
            demand.accept(1);
        });

        // Send the request to server2.
        request2.send(result -> System.getLogger("forwarder").log(INFO, "Forwarding to server2 complete"));
    }

    @Override
    public void onContent(Response response, LongConsumer demand, ByteBuffer content, Callback callback)
    {
        // When response content is received from server1, forward it to server2.
        content2.offer(content, Callback.from(() ->
        {
            // When the request content to server2 is sent,
            // succeed the callback to recycle the buffer.
            callback.succeeded();
            // Then demand more response content from server1.
            demand.accept(1);
        }, callback::failed));
    }
});

// When the response content from server1 is complete,
// complete also the request content to server2.
request1.onResponseSuccess(response -> content2.close());

// Send the request to server1.
request1.send(result -> System.getLogger("forwarder").log(INFO, "Sourcing from server1 complete"));

HttpClient Configuration

HttpClient has a quite large number of configuration parameters. Please refer to the HttpClient javadocs for the complete list of configurable parameters.

The most common parameters are:

HttpClient.idleTimeout: same as ClientConnector.idleTimeout described in this section.
HttpClient.connectBlocking: same as ClientConnector.connectBlocking described in this section.
HttpClient.connectTimeout: same as ClientConnector.connectTimeout described in this section.
HttpClient.maxConnectionsPerDestination: the max number of TCP connections that are opened for a particular destination (defaults to 64).
HttpClient.maxRequestsQueuedPerDestination: the max number of requests queued (defaults to 1024).

HttpClient TLS Configuration

HttpClient supports HTTPS requests out-of-the-box like a browser does.

The support for HTTPS request is provided by a SslContextFactory.Client instance, typically configured in the ClientConnector. If not explicitly configured, the ClientConnector will allocate a default one when started.

SslContextFactory.Client sslContextFactory = new SslContextFactory.Client();

ClientConnector clientConnector = new ClientConnector();
clientConnector.setSslContextFactory(sslContextFactory);

HttpClient httpClient = new HttpClient(new HttpClientTransportDynamic(clientConnector));
httpClient.start();

The default SslContextFactory.Client verifies the certificate sent by the server by verifying the validity of the certificate with respect to the certificate chain, the expiration date, the server host name, etc. This means that requests to public websites that have a valid certificate (such as https://google.com) will work out-of-the-box, without the need to specify a KeyStore or a TrustStore.

However, requests made to sites that return an invalid or a self-signed certificate will fail (like they will in a browser). An invalid certificate may be expired or have the wrong server host name; a self-signed certificate has a certificate chain that cannot be verified.

The validation of the server host name present in the certificate is important, to guarantee that the client is connected indeed with the intended server.

The validation of the server host name is performed at two levels: at the TLS level (in the JDK) and, optionally, at the application level.

By default, the validation of the server host name at the TLS level is enabled, while it is disabled at the application level.

You can configure the SslContextFactory.Client to skip the validation of the server host name at the TLS level:

SslContextFactory.Client sslContextFactory = new SslContextFactory.Client();
// Disable the validation of the server host name at the TLS level.
sslContextFactory.setEndpointIdentificationAlgorithm(null);

When you disable the validation of the server host name at the TLS level, you are strongly recommended to enable it at the application level, otherwise you may risk to connect to a server different from the one you intend to connect to:

SslContextFactory.Client sslContextFactory = new SslContextFactory.Client();
// Only allow to connect to subdomains of domain.com.
sslContextFactory.setHostnameVerifier((hostName, session) -> hostName.endsWith(".domain.com"));

You may have the validation of the server host name enabled at both the TLS level and application level, typically when you want to further restrict the client to connect only to a smaller set of server hosts than those allowed in the certificate sent by the server.

Please refer to the SslContextFactory.Client javadocs for the complete list of configurable parameters.

HttpClient TLS TrustStore Configuration

TODO

HttpClient TLS Client Certificates Configuration

TODO

Jetty’s HttpClient supports cookies out of the box.

The HttpClient instance receives cookies from HTTP responses and stores them in a java.net.CookieStore, a class that is part of the JDK. When new requests are made, the cookie store is consulted and if there are matching cookies (that is, cookies that are not expired and that match domain and path of the request) then they are added to the requests.

Applications can programmatically access the cookie store to find the cookies that have been set:

CookieStore cookieStore = httpClient.getCookieStore();
List<HttpCookie> cookies = cookieStore.get(URI.create("http://domain.com/path"));

Applications can also programmatically set cookies as if they were returned from a HTTP response:

CookieStore cookieStore = httpClient.getCookieStore();
HttpCookie cookie = new HttpCookie("foo", "bar");
cookie.setDomain("domain.com");
cookie.setPath("/");
cookie.setMaxAge(TimeUnit.DAYS.toSeconds(1));
cookieStore.add(URI.create("http://domain.com"), cookie);

Cookies may be added explicitly only for a particular request:

ContentResponse response = httpClient.newRequest("http://domain.com/path")
    .cookie(new HttpCookie("foo", "bar"))
    .send();

You can remove cookies that you do not want to be sent in future HTTP requests:

CookieStore cookieStore = httpClient.getCookieStore();
URI uri = URI.create("http://domain.com");
List<HttpCookie> cookies = cookieStore.get(uri);
for (HttpCookie cookie : cookies)
{
    cookieStore.remove(uri, cookie);
}

If you want to totally disable cookie handling, you can install a HttpCookieStore.Empty. This must be done when HttpClient is used in a proxy application, in this way:

httpClient.setCookieStore(new HttpCookieStore.Empty());

You can enable cookie filtering by installing a cookie store that performs the filtering logic in this way:

class GoogleOnlyCookieStore extends HttpCookieStore
{
    @Override
    public void add(URI uri, HttpCookie cookie)
    {
        if (uri.getHost().endsWith("google.com"))
            super.add(uri, cookie);
    }
}

httpClient.setCookieStore(new GoogleOnlyCookieStore());

The example above will retain only cookies that come from the google.com domain or sub-domains.

Special Characters in Cookies

Jetty is compliant with RFC6265, and as such care must be taken when setting a cookie value that includes special characters such as ;.

Previously, Version=1 cookies defined in RFC2109 (and continued in https://tools.ietf.org/html/rfc2965[RFC2965]) allowed for special/reserved characters to be enclosed within double quotes when declared in a Set-Cookie response header:

Set-Cookie: foo="bar;baz";Version=1;Path="/secur"

This was added to the HTTP Response as follows:

protected void service(HttpServletRequest request, HttpServletResponse response)
{
    javax.servlet.http.Cookie cookie = new Cookie("foo", "bar;baz");
    cookie.setPath("/secure");
    response.addCookie(cookie);
}

The introduction of RFC6265 has rendered this approach no longer possible; users are now required to encode cookie values that use these special characters. This can be done utilizing javax.servlet.http.Cookie as follows:

javax.servlet.http.Cookie cookie = new Cookie("foo", URLEncoder.encode("bar;baz", "UTF-8"));

Jetty validates all cookie names and values being added to the HttpServletResponse via the addCookie(Cookie) method. If an illegal value is discovered Jetty will throw an IllegalArgumentException with the details.

HttpClient Authentication Support

Jetty’s HttpClient supports the BASIC and DIGEST authentication mechanisms defined by RFC 7235, as well as the SPNEGO authentication mechanism defined in RFC 4559.

The HTTP conversation, the sequence of related HTTP requests, for a request that needs authentication is the following:

Upon receiving a HTTP 401 response code, HttpClient looks at the WWW-Authenticate response header (the server challenge) and then tries to match configured authentication credentials to produce an Authentication header that contains the authentication credentials to access the resource.

You can configure authentication credentials in the HttpClient instance as follows:

// Add authentication credentials.
AuthenticationStore auth = httpClient.getAuthenticationStore();

URI uri1 = new URI("http://mydomain.com/secure");
auth.addAuthentication(new BasicAuthentication(uri1, "MyRealm", "userName1", "password1"));

URI uri2 = new URI("http://otherdomain.com/admin");
auth.addAuthentication(new BasicAuthentication(uri1, "AdminRealm", "admin", "password"));

Authentications are matched against the server challenge first by mechanism (e.g. BASIC or DIGEST), then by realm and then by URI.

If an Authentication match is found, the application does not receive events related to the HTTP 401 response. These events are handled internally by HttpClient which produces another (internal) request similar to the original request but with an additional Authorization header.

If the authentication is successful, the server responds with a HTTP 200 and HttpClient caches the Authentication.Result so that subsequent requests for a matching URI will not incur in the additional rountrip caused by the HTTP 401 response.

It is possible to clear Authentication.Results in order to force authentication again:

httpClient.getAuthenticationStore().clearAuthenticationResults();

Authentication results may be preempted to avoid the additional roundtrip due to the server challenge in this way:

AuthenticationStore auth = httpClient.getAuthenticationStore();
URI uri = URI.create("http://domain.com/secure");
auth.addAuthenticationResult(new BasicAuthentication.BasicResult(uri, "username", "password"));

In this way, requests for the given URI are enriched immediately with the Authorization header, and the server should respond with HTTP 200 (and the resource content) rather than with the 401 and the challenge.

It is also possible to preempt the authentication for a single request only, in this way:

URI uri = URI.create("http://domain.com/secure");
Authentication.Result authn = new BasicAuthentication.BasicResult(uri, "username", "password");
Request request = httpClient.newRequest(uri);
authn.apply(request);
request.send();

See also the proxy authentication section for further information about how authentication works with HTTP proxies.

HttpClient SPNEGO Authentication Support

TODO

HttpClient Proxy Support

Jetty’s HttpClient can be configured to use proxies to connect to destinations.

These types of proxies are available out of the box:

HTTP proxy (provided by class org.eclipse.jetty.client.HttpProxy)
SOCKS 4 proxy (provided by class org.eclipse.jetty.client.Socks4Proxy)
SOCKS 5 proxy (provided by class org.eclipse.jetty.client.Socks5Proxy)

Other implementations may be written by subclassing ProxyConfiguration.Proxy.

The following is a typical configuration:

HttpProxy proxy = new HttpProxy("proxyHost", 8888);

// Do not proxy requests for localhost:8080.
proxy.getExcludedAddresses().add("localhost:8080");

// Add the new proxy to the list of proxies already registered.
ProxyConfiguration proxyConfig = httpClient.getProxyConfiguration();
proxyConfig.addProxy(proxy);

ContentResponse response = httpClient.GET("http://domain.com/path");

You specify the proxy host and proxy port, and optionally also the addresses that you do not want to be proxied, and then add the proxy configuration on the ProxyConfiguration instance.

Configured in this way, HttpClient makes requests to the HTTP proxy (for plain-text HTTP requests) or establishes a tunnel via HTTP CONNECT (for encrypted HTTPS requests).

Proxying is supported for any version of the HTTP protocol.

SOCKS5 Proxy Support

SOCKS 5 (defined in RFC 1928) offers choices for authentication methods and supports IPv6 (things that SOCKS 4 does not support).

A typical SOCKS 5 proxy configuration with the username/password authentication method is the following:

Socks5Proxy proxy = new Socks5Proxy("proxyHost", 8888);
String socks5User = "jetty";
String socks5Pass = "secret";
var socks5AuthenticationFactory = new Socks5.UsernamePasswordAuthenticationFactory(socks5User, socks5Pass);
// Add the authentication method to the proxy.
proxy.putAuthenticationFactory(socks5AuthenticationFactory);

// Do not proxy requests for localhost:8080.
proxy.getExcludedAddresses().add("localhost:8080");

// Add the new proxy to the list of proxies already registered.
ProxyConfiguration proxyConfig = httpClient.getProxyConfiguration();
proxyConfig.addProxy(proxy);

ContentResponse response = httpClient.GET("http://domain.com/path");

HTTP Proxy Authentication Support

Jetty’s HttpClient supports HTTP proxy authentication in the same way it supports server authentication.

In the example below, the HTTP proxy requires BASIC authentication, but the server requires DIGEST authentication, and therefore:

AuthenticationStore auth = httpClient.getAuthenticationStore();

// Proxy credentials.
URI proxyURI = new URI("http://proxy.net:8080");
auth.addAuthentication(new BasicAuthentication(proxyURI, "ProxyRealm", "proxyUser", "proxyPass"));

// Server credentials.
URI serverURI = new URI("http://domain.com/secure");
auth.addAuthentication(new DigestAuthentication(serverURI, "ServerRealm", "serverUser", "serverPass"));

// Proxy configuration.
ProxyConfiguration proxyConfig = httpClient.getProxyConfiguration();
HttpProxy proxy = new HttpProxy("proxy.net", 8080);
proxyConfig.addProxy(proxy);

ContentResponse response = httpClient.newRequest(serverURI).send();

The HTTP conversation for successful authentications on both the proxy and the server is the following:

The application does not receive events related to the responses with code 407 and 401 since they are handled internally by HttpClient.

Similarly to the authentication section, the proxy authentication result and the server authentication result can be preempted to avoid, respectively, the 407 and 401 roundtrips.

HttpClient Pluggable Transports

Jetty’s HttpClient can be configured to use different transport protocols to carry the semantic of HTTP requests and responses.

This means that the intention of a client to request resource /index.html using the GET method can be carried over the network in different formats.

An HttpClient transport is the component that is in charge of converting a high-level, semantic, HTTP requests such as " GET resource /index.html " into the specific format understood by the server (for example, HTTP/2 or HTTP/3), and to convert the server response from the specific format (HTTP/2 or HTTP/3) into high-level, semantic objects that can be used by applications.

The most common protocol format is HTTP/1.1, a textual protocol with lines separated by \r\n:

GET /index.html HTTP/1.1\r\n
Host: domain.com\r\n
...
\r\n

However, the same request can be made using FastCGI, a binary protocol:

x01 x01 x00 x01 x00 x08 x00 x00
x00 x01 x01 x00 x00 x00 x00 x00
x01 x04 x00 x01 xLL xLL x00 x00
x0C x0B  D   O   C   U   M   E
 N   T   _   U   R   I   /   i
 n   d   e   x   .   h   t   m
 l
...

Similarly, HTTP/2 is a binary protocol that transports the same information in a yet different format via TCP, while HTTP/3 is a binary protocol that transports the same information in yet another format via UDP.

A protocol may be negotiated between client and server. A request for a resource may be sent using one protocol (for example, HTTP/1.1), but the response may arrive in a different protocol (for example, HTTP/2).

HttpClient supports these static transports, each speaking only one protocol:

HTTP/1.1 (both clear-text and TLS encrypted)
HTTP/2 (both clear-text and TLS encrypted)
HTTP/3 (only encrypted via QUIC+TLS)
FastCGI (both clear-text and TLS encrypted)

HttpClient also supports one dynamic transport, that can speak different protocols and can select the right protocol by negotiating it with the server or by explicit indication from applications.

Applications are typically not aware of the actual protocol being used. This allows them to write their logic against a high-level API that hides the details of the specific protocol being used over the network.

HTTP/1.1 Transport

HTTP/1.1 is the default transport.

// No transport specified, using default.
HttpClient httpClient = new HttpClient();
httpClient.start();

If you want to customize the HTTP/1.1 transport, you can explicitly configure it in this way:

// Configure HTTP/1.1 transport.
HttpClientTransportOverHTTP transport = new HttpClientTransportOverHTTP();
transport.setHeaderCacheSize(16384);

HttpClient client = new HttpClient(transport);
client.start();

HTTP/2 Transport

The HTTP/2 transport can be configured in this way:

// The HTTP2Client powers the HTTP/2 transport.
HTTP2Client h2Client = new HTTP2Client();
h2Client.setInitialSessionRecvWindow(64 * 1024 * 1024);

// Create and configure the HTTP/2 transport.
HttpClientTransportOverHTTP2 transport = new HttpClientTransportOverHTTP2(h2Client);
transport.setUseALPN(true);

HttpClient client = new HttpClient(transport);
client.start();

HTTP2Client is the lower-level client that provides an API based on HTTP/2 concepts such as sessions, streams and frames that are specific to HTTP/2. See the HTTP/2 client section for more information.

HttpClientTransportOverHTTP2 uses HTTP2Client to format high-level semantic HTTP requests (like "GET resource /index.html") into the HTTP/2 specific format.

HTTP/3 Transport

The HTTP/3 transport can be configured in this way:

// The HTTP3Client powers the HTTP/3 transport.
HTTP3Client h3Client = new HTTP3Client();
h3Client.getQuicConfiguration().setSessionRecvWindow(64 * 1024 * 1024);

// Create and configure the HTTP/3 transport.
HttpClientTransportOverHTTP3 transport = new HttpClientTransportOverHTTP3(h3Client);

HttpClient client = new HttpClient(transport);
client.start();

HTTP3Client is the lower-level client that provides an API based on HTTP/3 concepts such as sessions, streams and frames that are specific to HTTP/3. See the HTTP/3 client section for more information.

HttpClientTransportOverHTTP3 uses HTTP3Client to format high-level semantic HTTP requests (like "GET resource /index.html") into the HTTP/3 specific format.

FastCGI Transport

The FastCGI transport can be configured in this way:

String scriptRoot = "/var/www/wordpress";
HttpClientTransportOverFCGI transport = new HttpClientTransportOverFCGI(scriptRoot);

HttpClient client = new HttpClient(transport);
client.start();

In order to make requests using the FastCGI transport, you need to have a FastCGI server such as PHP-FPM (see also link:http://php.net/manual/en/install.fpm.php).

The FastCGI transport is primarily used by Jetty’s FastCGI support to serve PHP pages (WordPress for example).

Dynamic Transport

The static transports work well if you know in advance the protocol you want to speak with the server, or if the server only supports one protocol (such as FastCGI).

With the advent of HTTP/2 and HTTP/3, however, servers are now able to support multiple protocols, at least both HTTP/1.1 and HTTP/2.

The HTTP/2 protocol is typically negotiated between client and server. This negotiation can happen via ALPN, a TLS extension that allows the client to tell the server the list of protocol that the client supports, so that the server can pick one of the client supported protocols that also the server supports; or via HTTP/1.1 upgrade by means of the Upgrade header.

Applications can configure the dynamic transport with one or more application protocols such as HTTP/1.1 or HTTP/2. The implementation will take care of using TLS for HTTPS URIs, using ALPN if necessary, negotiating protocols, upgrading from one protocol to another, etc.

By default, the dynamic transport only speaks HTTP/1.1:

// Dynamic transport speaks HTTP/1.1 by default.
HttpClientTransportDynamic transport = new HttpClientTransportDynamic();

HttpClient client = new HttpClient(transport);
client.start();

The dynamic transport can be configured with just one protocol, making it equivalent to the corresponding static transport:

ClientConnector connector = new ClientConnector();

// Equivalent to HttpClientTransportOverHTTP.
HttpClientTransportDynamic http11Transport = new HttpClientTransportDynamic(connector, HttpClientConnectionFactory.HTTP11);

// Equivalent to HttpClientTransportOverHTTP2.
HTTP2Client http2Client = new HTTP2Client(connector);
HttpClientTransportDynamic http2Transport = new HttpClientTransportDynamic(connector, new ClientConnectionFactoryOverHTTP2.HTTP2(http2Client));

The dynamic transport, however, has been implemented to support multiple transports, in particular both HTTP/1.1 and HTTP/2:

ClientConnector connector = new ClientConnector();

ClientConnectionFactory.Info http1 = HttpClientConnectionFactory.HTTP11;

HTTP2Client http2Client = new HTTP2Client(connector);
ClientConnectionFactoryOverHTTP2.HTTP2 http2 = new ClientConnectionFactoryOverHTTP2.HTTP2(http2Client);

HttpClientTransportDynamic transport = new HttpClientTransportDynamic(connector, http1, http2);

HttpClient client = new HttpClient(transport);
client.start();

The order in which the protocols are specified to HttpClientTransportDynamic indicates what is the client preference.

When using TLS (i.e. URIs with the https scheme), the application protocol is negotiated between client and server via ALPN, and it is the server that decides what is the application protocol to use for the communication, regardless of the client preference.

When clear-text communication is used (i.e. URIs with the http scheme) there is no application protocol negotiation, and therefore the application must know a priori whether the server supports the protocol or not. For example, if the server only supports clear-text HTTP/2, and HttpClientTransportDynamic is configured as in the example above, the client will send, by default, a clear-text HTTP/1.1 request to a clear-text HTTP/2 only server, which will result in a communication failure.

Provided that the server supports both HTTP/1.1 and HTTP/2 clear-text, client applications can explicitly hint the version they want to use:

ClientConnector connector = new ClientConnector();
ClientConnectionFactory.Info http1 = HttpClientConnectionFactory.HTTP11;
HTTP2Client http2Client = new HTTP2Client(connector);
ClientConnectionFactoryOverHTTP2.HTTP2 http2 = new ClientConnectionFactoryOverHTTP2.HTTP2(http2Client);
HttpClientTransportDynamic transport = new HttpClientTransportDynamic(connector, http1, http2);
HttpClient client = new HttpClient(transport);
client.start();

// The server supports both HTTP/1.1 and HTTP/2 clear-text on port 8080.

// Make a clear-text request without explicit version.
// The first protocol specified to HttpClientTransportDynamic
// is picked, in this example will be HTTP/1.1.
ContentResponse http1Response = client.newRequest("host", 8080).send();

// Make a clear-text request with explicit version.
// Clear-text HTTP/2 is used for this request.
ContentResponse http2Response = client.newRequest("host", 8080)
    // Specify the version explicitly.
    .version(HttpVersion.HTTP_2)
    .send();

// Make a clear-text upgrade request from HTTP/1.1 to HTTP/2.
// The request will start as HTTP/1.1, but the response will be HTTP/2.
ContentResponse upgradedResponse = client.newRequest("host", 8080)
    .headers(headers -> headers
        .put(HttpHeader.UPGRADE, "h2c")
        .put(HttpHeader.HTTP2_SETTINGS, "")
        .put(HttpHeader.CONNECTION, "Upgrade, HTTP2-Settings"))
    .send();

In case of TLS encrypted communication using the https scheme, things are a little more complicated.

If the client application explicitly specifies the HTTP version, then ALPN is not used by the client. By specifying the HTTP version explicitly, the client application has prior-knowledge of what HTTP version the server supports, and therefore ALPN is not needed. If the server does not support the HTTP version chosen by the client, then the communication will fail.

If the client application does not explicitly specify the HTTP version, then ALPN will be used by the client. If the server also supports ALPN, then the protocol will be negotiated via ALPN and the server will choose the protocol to use. If the server does not support ALPN, the client will try to use the first protocol configured in HttpClientTransportDynamic, and the communication may succeed or fail depending on whether the server supports the protocol chosen by the client.

Unix-Domain Configuration

All the transports can be configured with a ClientConnector, the component that is responsible for the transmission of the bytes generated by the transport to the server.

By default, ClientConnector uses TCP networking to send bytes to the server and receive bytes from the server.

When you are using Java 16 or later, ClientConnector also support Unix-Domain sockets, and every transport can be configured to use Unix-Domain sockets instead of TCP networking.

To configure Unix-Domain sockets, you can create a ClientConnector instance in the following way:

// This is the path where the server "listens" on.
Path unixDomainPath = Path.of("/path/to/server.sock");

// Creates a ClientConnector that uses Unix-Domain
// sockets, not the network, to connect to the server.
ClientConnector unixDomainClientConnector = ClientConnector.forUnixDomain(unixDomainPath);

// Use Unix-Domain for HTTP/1.1.
HttpClientTransportOverHTTP http1Transport = new HttpClientTransportOverHTTP(unixDomainClientConnector);

// You can use Unix-Domain also for HTTP/2.
HTTP2Client http2Client = new HTTP2Client(unixDomainClientConnector);
HttpClientTransportOverHTTP2 http2Transport = new HttpClientTransportOverHTTP2(http2Client);

// You can also use UnixDomain for the dynamic transport.
ClientConnectionFactory.Info http1 = HttpClientConnectionFactory.HTTP11;
ClientConnectionFactoryOverHTTP2.HTTP2 http2 = new ClientConnectionFactoryOverHTTP2.HTTP2(http2Client);
HttpClientTransportDynamic dynamicTransport = new HttpClientTransportDynamic(unixDomainClientConnector, http1, http2);

// Choose the transport you prefer for HttpClient, for example the dynamic transport.
HttpClient httpClient = new HttpClient(dynamicTransport);
httpClient.start();

You can use Unix-Domain sockets support only when you run your client application with Java 16 or later.

You can configure a Jetty server to use Unix-Domain sockets, as explained in this section.

HTTP Client

HttpClient Introduction

Starting HttpClient

Stopping HttpClient

HttpClient Architecture

HttpClient Connection Pooling

HttpClient Request Processing

HttpClient API Usage

HttpClient Blocking APIs

HttpClient Non-Blocking APIs

Request Content Handling

Response Content Handling

HttpClient Configuration

HttpClient TLS Configuration

HttpClient TLS TrustStore Configuration

HttpClient TLS Client Certificates Configuration

HttpClient Cookie Support

Special Characters in Cookies

HttpClient Authentication Support

HttpClient SPNEGO Authentication Support

HttpClient Proxy Support

SOCKS5 Proxy Support

HTTP Proxy Authentication Support

HttpClient Pluggable Transports

HTTP/1.1 Transport

HTTP/2 Transport

HTTP/3 Transport

FastCGI Transport

Dynamic Transport

Unix-Domain Configuration