JMX Monitoring & Management
Monitoring and management of a Jetty server is important because it allows you to monitor the status of the server ("Is the server processing requests?") and to manage — i.e. read and possibly change — its configuration.
The ability to read and change the Jetty configuration is very important for troubleshooting Jetty — please refer to the troubleshooting section for more information.
Jetty relies on the Java Management Extensions (JMX) APIs included in OpenJDK to provide monitoring and management.
The JMX APIs support a JVM-local MBeanServer
, accessible only from within the JVM itself (or by tools that can attach to a running JVM), and a way to expose the MBeanServer
to remote clients via Java’s RMI (Remote Method Invocation).
Enabling Local JMX Support
As with many other Jetty features, local JMX support is enabled with the jmx
Jetty module:
$ java -jar $JETTY_HOME/start.jar --add-modules=jmx
With the jmx
Jetty module enabled, Jetty components will be exported as JMX MBeans to the JVM platform MBeanServer
, so that they can be accessed by JMX compliant tools.
Each Jetty component will export to its correspondent MBean relevant configuration parameters, so that a JMX tool can read and possibly change the component configuration through the MBean.
Note that the Jetty MBeans are registered into the platform MBeanServer
, but are not available to remote clients: they are local to the JVM.
This configuration is useful when you develop and test your Jetty server locally.
JMX compliant tools such as Java Mission Control (JMC) can be started locally on your machine and can attach to other JVMs running on your machine, showing you the registered MBeans among which you will find the Jetty MBeans.
Enabling only the local JMX support is the most secure option for monitoring and management, but only users that have local access to the JVM will be able to browse the MBeans. If you need to access the MBeans from a remote machine, read this section. |
Enabling Remote JMX Support
There are two ways to configure a Jetty server so that it is possible to access the JVM platform MBeans from remote clients:
-
Use the
com.sun.management.jmxremote
and related system properties when starting Jetty. Unfortunately, this solution does not work well with firewalls, and will not be discussed further. -
Use the
jmx-remote
Jetty module.
Both ways use Java’s Remote Method Invocation (RMI) to communicate between the client and the server.
Refresher: How RMI Works
A server application that wants to make an object available to remote clients must export the object. Exporting an object creates an RMI stub that contains the host/port of the RMI server that accepts incoming invocations from clients and forwards them to the object.
During the creation of the RMI stub, the host stored in the RMI stub is retrieved from the local name resolution system (for example, in Linux, from The RMI stub is then sent, along with a name that uniquely identifies the object, to the RMI registry. The RMI registry is a service that maps names to RMI stubs; it may be external to both clients and server, although often it is part of the server JVM. When a client application wants to connect to the server object using RMI, it first connects to the RMI registry to download the RMI stub for the RMI server; recall that the RMI stub contains the host/port to connect to the RMI server. Then, the client uses the RMI stub to connect to the RMI server, typically to a host/port that may be different from the RMI registry host/port (in particular, by default the RMI server port will be different from the RMI registry port). |
Remote access to the platform MBeans, and therefore the Jetty MBeans, is enabled by the jmx-remote
Jetty module:
$ java -jar $JETTY_HOME/start.jar --add-modules=jmx-remote
This command creates the jmx-remote.ini
file:
JETTY_BASE
└── start.d
└── jmx-remote.ini
Enabling the jmx-remote
module transitively enables the jmx
module as well.
The configuration for the RMI registry and the RMI server is specified by a JMXServiceURL
.
The string format of an RMI JMXServiceURL
is the following:
service:jmx:rmi://<rmi_server_host>:<rmi_server_port>/jndi/rmi://<rmi_registry_host>:<rmi_registry_port>/jmxrmi
Below you can find examples of JMXServiceURL
s:
service:jmx:rmi:///jndi/rmi:///jmxrmi
where:
rmi_server_host = local host address
rmi_server_port = randomly chosen
rmi_registry_host = local host address
rmi_registry_port = 1099
service:jmx:rmi://0.0.0.0:1099/jndi/rmi://0.0.0.0:1099/jmxrmi
where:
rmi_server_host = any address
rmi_server_port = 1099
rmi_registry_host = any address
rmi_registry_port = 1099
service:jmx:rmi://localhost:1100/jndi/rmi://localhost:1099/jmxrmi
where:
rmi_server_host = loopback address
rmi_server_port = 1100
rmi_registry_host = loopback address
rmi_registry_port = 1099
The default JMXServiceURL
configured by the jmx-remote
module is the following:
service:jmx:rmi://localhost:1099/jndi/rmi://localhost:1099/jmxrmi
With the default configuration, only clients that are local to the server machine can connect to the RMI registry and RMI server - this is done for security reasons. However, even with this local-only configuration, it would still be possible to access the MBeans from remote using an SSH tunnel, as explained in this section.
By specifying an appropriate JMXServiceURL
, you can fine tune the network address the RMI registry and the RMI server bind to, and the ports that the RMI registry and the RMI server listen to.
The RMI server and RMI registry hosts and ports can be the same (as in the default configuration) because RMI is able to multiplex traffic arriving to one port to multiple RMI objects.
If you need to allow JMX remote access through a firewall, you must open both the RMI registry and the RMI server ports.
The default configuration simplifies the firewall configuration because you only need to open port 1099
.
When Jetty is started with the The RMI stub contains the host/port to connect to the RMI server, but the host is typically the machine host name, not the host specified in the To control the host stored in the RMI stub you need to set the system property |
If your client cannot connect to the server, the most common cause is a mismatch between the RMI server host of the JMXServiceURL and the RMI server host of the RMI stub.
|
You can customize the RMI server host/port, the RMI registry host/port and the system property java.rmi.server.hostname
by editing the jmx-remote.ini
configuration file.
Further information about the jmx-remote
module configuration can be found here.
Remote JMX Access with Port Forwarding via SSH Tunnel
You can access JMX MBeans on a remote machine when the RMI ports are not open, for example because of firewall policies, but you have SSH access to the machine, using local port forwarding via an SSH tunnel.
In this case you want to configure the JMXServiceURL
that binds the RMI server and the RMI registry to the loopback interface only and to the same port:
service:jmx:rmi://localhost:1099/jndi/rmi://localhost:1099/jmxrmi
You must set the system property -Djava.rmi.server.hostname=localhost
so that the RMI stub contains localhost
as the host name to connect to.
This is, incidentally, the default configuration of the jmx-remote
module.
Then you set up the local port forwarding with the SSH tunnel:
$ ssh -L 1099:localhost:1099 <user>@<machine_host>
Thanks to the local port forwarding of the SSH tunnel, when the client connects to localhost:1099
on your local computer, the traffic will be forwarded to machine_host
and when there, the SSH daemon will forward the traffic to localhost:1099
on machine_host
, which is exactly where the RMI server and the RMI registry listens to.
The client first contacts the RMI registry, so it connects to localhost:1099
on your local computer; the traffic is forwarded to machine_host
through the SSH tunnel, connects to the RMI registry and the RMI stub is downloaded to the client.
Then the client uses the RMI stub to connect to the RMI server. The RMI stub contains localhost
as the RMI server host because that is what you have configured with the system property java.rmi.server.hostname
.
The client will connect again to localhost:1099
on your local computer, this time to contact the RMI server; the traffic is forwarded to machine_host
through the SSH tunnel, arrives to machine_host
and connects to the RMI server.
Remote JMX Access Authentication & Authorization
The standard javax.management.remote.JMXConnectorServer
class, used by the jmx-remote
module to provide remote JMX access to Jetty MBeans, provides several options to authenticate and authorize users.
For a complete guide to controlling authentication and authorization in JMX, see the official JMX documentation.
The simplest way to control JMX authentication and authorization is to specify two files: one contains username and password pairs, and the other contains username and permission pairs.
This is achieved by enabling the jmx-remote-auth
Jetty module:
$ java -jar $JETTY_HOME/start.jar --add-modules=jmx-remote-auth
Enabling the jmx-remote-auth
Jetty module creates the following files:
$JETTY_BASE ├── etc │ ├── jmxremote.access │ ├── jmxremote.password │ └── jmx-remote-auth.xml └── start.d ├── jmx-remote-auth.ini └── jmx-remote.ini
Then you edit the $JETTY_BASE/etc/jmxremote.password
file, adding the username/password pairs that you need:
# The file format is: <username> <password> alice wonderland bob marley
You must also edit the $JETTY_BASE/etc/jmxremote.access
file to give permissions to your users:
# The file format is: <username> <readonly|readwrite> alice readwrite bob readonly
The above files define user alice
with password wonderland
to have readwrite
access, and user bob
with password marley
to have readonly
access.
Securing Remote JMX Access with TLS
The JMX communication via RMI happens by default in clear-text, but it is possible to secure the JMX communication via RMI with TLS.
If you want to reuse the configuration that you are using for the https
module, you can just enable the jmx-remote-ssl.xml
Jetty module:
$ java -jar $JETTY_HOME/start.jar --add-modules=jmx-remote-ssl
The |
The KeyStore must contain a valid certificate signed by a Certification Authority. Having certificates signed by a Certification Authority simplifies by a lot the configuration needed to get the RMI communication over TLS working properly.
The RMI mechanic is the usual one: the RMI client (typically a monitoring console) will connect first to the RMI registry (using TLS), download the RMI stub that contains the address and port of the RMI server to connect to, then connect to the RMI server (using TLS).
This also mean that if the RMI registry and the RMI server are on different hosts, the RMI client must have available the cryptographic material to validate the certificates from both hosts. This is where having certificates signed by a Certification Authority simplifies the configuration: if they are signed by a well known Certification Authority, the client does not need any extra configuration — everything will be handled by the Java runtime.
If the certificates are not signed by a Certification Authority (for example the certificate is self-signed), then you need to specify the TLS system properties that allow RMI (especially when acting as an RMI client) to retrieve the cryptographic material necessary to establish the TLS connection.
When the RMI server exports the |
You must edit the $JETTY_BASE/start.d/jmx-remote-ssl.ini
file and add the TrustStore path and password:
--module=jmx-remote-ssl # System properties necessary for non-trusted certificates. -Djavax.net.ssl.trustStore=/path/to/trustStore.p12 -Djavax.net.ssl.trustStorePassword=password
The TrustStore must contain the certificate you want to trust. If you are using self-signed certificates, the KeyStore already contains the self-signed certificate and therefore the KeyStore can be used as a TrustStore, and the system properties above can refer to the KeyStore path and password. |
JMX compliant tools that offer a graphical user interface also must be started specifying the TrustStore path and password.
For example, to launch Java Mission Control (JMC):
$ jmc -vmargs -Djavax.net.ssl.trustStore=/path/to/trustStore.p12 -Djavax.net.ssl.trustStorePassword=password