Troubleshooting

To troubleshoot Jetty when used as a standalone server, there are two main tools: the Jetty Server Dump and enabling DEBUG level logging.

It is typically very helpful to enable JMX, as detailed in this section. Enabling JMX allows to access the Jetty server (possibly a production server) from a remote location, and obtain the information exported by Jetty via JMX.

Access to the Jetty server via JMX allows to either diagnose the problem, possibly reconfigure and restart the Jetty server.

Make sure you read about how to secure the access to Jetty when using remote JMX.

Jetty Server Dump

The most useful too to troubleshoot a Jetty server is the Jetty Component Dump, that reports the internal state of the Jetty server and all its components (the thread pool, the ByteBuffer pool, the currently established connections, and so on).

The Jetty server dump can easily be obtained via JMX using a JMX console such as Java Mission Control (JMC), as explained in this section.

Dump at Server Start/Stop

The Jetty server dump may be reported just after the Server starts. This is useful to verify that all the expected components have been correctly configured and started for the optimal functioning of the server.

Similarly, the Jetty server dump may be reported just before the Server stops. This is useful to log the state of a server that is not working properly.

The server Jetty module defines, in this section, the properties jetty.server.dumpAfterStart and jetty.server.dumpBeforeStop that you can configure to report the Jetty server dump.

These properties can be specified temporarily as command line arguments, as explained in this section:

$ java -jar $JETTY_HOME/start.jar jetty.server.dumpAfterStart=true

Alternatively, you can specify these properties in the $JETTY_BASE/start.d/server.ini file, as explained in this section, to make the values persistent across server restarts.

Enabling DEBUG Logging

Enabling DEBUG level logging for the org.eclipse.jetty logger name provides the maximum amount of information to troubleshoot Jetty issues.

Refer to the logging section for more information about how to configure logging in Jetty.

Enabling DEBUG level logging for org.eclipse.jetty is very, very expensive.

Your server could be slowed down to almost a halt, especially if it is under heavy load. Furthermore, the log file could quickly fill up the entire filesystem (unless configured to roll over), so you want to be really careful using DEBUG logging.

For production servers, consider using the Jetty Server Dump first, and enable DEBUG logging only as a last resort.

However, sometimes issues are such that only DEBUG logging can really tell what’s going on in the system, and enabling DEBUG logging is your best chance to figure the issue out. Below you can find few suggestions that can help you reduce the impact when you have to enable DEBUG logging.

Jetty Behind a Load Balancer

If Jetty instances are behind a load balancer, you may configure the load balancer to send less load to a particular Jetty instance, and enable DEBUG logging in that instance only.

Enabling DEBUG Logging for a Short Time

In certain cases the issue can be reproduced reliably, but only in the production environment.

You can use JMX to temporarily enable DEBUG logging, reproduce the issue, and then disable DEBUG logging.

Alternatively, if you cannot reliably reproduce the issue, but you know it is happening, you can temporarily enable DEBUG logging for a small period of time, let’s say 10-60 seconds, and then disable DEBUG logging.

Changing the log level at runtime is a feature of the logging implementation that you are using.

The Jetty SLF4J implementation, used by default, exposes via JMX method boolean JettyLoggerFactoryMBean.setLoggerLevel(String loggerName, String levelName) that you can invoke via a JMX console to change the level for the specified logger name. The method returns true if the logger level was successfully changed.

For example, you can pass the string org.eclipse.jetty as the first parameter, and the string DEBUG (upper case) as the second parameter. You can then use the string INFO or WARN (upper case) to restore the logging level to its previous value.

Enabling DEBUG Logging for SubPackages

Enabling DEBUG logging for the org.eclipse.jetty logger name implies that all children logger names, recursively, inherit the DEBUG level.

Processing a single HTTP request involves many Jetty components: the I/O subsystem (under org.eclipse.jetty.io), the thread pool (under org.eclipse.jetty.util), the HTTP/1.1 parsing (under org.eclipse.jetty.http), etc.

If you can cut the amount of DEBUG logging to just what you need to troubleshoot the issue, the impact of enabling DEBUG logging will be much less than enabling it for all Jetty components.

For example, if you need to troubleshoot a client that sends bad HTTP/1.1 requests, it may be enough to enable only the org.eclipse.jetty.http logger name, therefore saving the large amount of DEBUG logging produced by the I/O subsystem and by the thread pool.

In another case, you may need to troubleshoot only HTTP/2 requests, and therefore enabling only the org.eclipse.jetty.http2 logger name could be enough.

Remote Debugging

The Java Virtual Machines allows remote processes on different hosts to connect for debugging purposes, by using specific command line options.

While it is possible to enable remote debugging on a Jetty server, it is typically not recommended for security and performance reasons. Only enable remote debugging on a Jetty server as a last resort to troubleshoot issues that could not be troubleshot otherwise.

You can easily create a custom Jetty module (see this section) with the following content:

remote-debug.mod

[description]
Enables remote debugging

[exec]
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005

The [exec] directive (documented here) is necessary to pass the -agentlib:jdwp JVM option to the forked JVM that runs Jetty, so that you can attach with a debugger.

The address parameter of the -agentlib:jdwp command line option specifies the network address and port the Jetty JVM listens on for remote debugging.

Please refer to the Java Debug Wire Protocol documentation for additional information about the -agentlib:jdwp command line option and its parameters.

You can now enable the remote-debug Jetty module with the following command issued from the $JETTY_BASE directory:

$ java -jar $JETTY_HOME/start.jar --add-modules=server,remote-debug

The command above minimally adds a Jetty server without connectors (via the server Jetty module) and the remote-debug Jetty module, and produces the following $JETTY_BASE directory structure:

$JETTY_BASE
├── modules
│   └── remote-debug.mod
├── resources
│   └── jetty-logging.properties
└── start.d
    ├── remote-debug.ini
    └── server.ini

You can easily disable the remote-debug Jetty module as explained in this section.

Alternatively, you can enable the remote-debug module on the command line, as explained in this section.

Starting the Jetty server with the remote-debug module enabled yields:

WARN  : Forking second JVM due to forking module(s): [remote-debug]. Use --dry-run to generate the command line to avoid forking.
Listening for transport dt_socket at address: 5005
2025-04-03 21:59:40.458:INFO :oejs.Server:main: jetty-12.0.20-SNAPSHOT; built: 2025-04-03T15:46:22.817Z; git: 8ff58de3621e640642922bbd1371599c43fd0846; jvm 24+36
2025-04-03 21:59:40.519:INFO :oejs.Server:main: Started oejs.Server@6328d34a{STARTING}[12.0.20-SNAPSHOT,sto=0] @1114ms

Note how the JVM is listening on port 5005 to allow remote debuggers to connect.

If you want to avoid to fork a second JVM to pass the -agentlib:jdwp JVM option, please read this section.

Troubleshooting Handlers

`StateTrackingHandler`

Jetty’s StateTrackingHandler (described in this module) can be used to troubleshoot problems in web applications.

StateTrackingHandler tracks the usages of Handler/Request/Response asynchronous APIs by web applications, emitting events (logged at warning level) when an invalid usage of the APIs is detected.

In conjunction with dumping the Jetty component tree, it dumps the state of current requests, detailing whether they have reads or writes that are pending, whether callbacks have been completed, along with thread stack traces (including virtual threads) of operations that have been started but not completed, or are stuck in blocking code.

You need to enable the state-tracking Jetty module, and configure it to track what you are interested in tracking (for more details, see the javadocs).