Jetty JMX Support

The Java Management Extensions (JMX) APIs are standard API for managing and monitoring resources such as applications, devices, services, and the Java Virtual Machine itself.

The JMX API includes remote access, so a remote management console such as Java Mission Control can interact with a running application for these purposes.

Jetty architecture is based on components organized in a tree. Every time a component is added to or removed from the component tree, an event is emitted, and Container.Listener implementations can listen to those events and perform additional actions.

org.eclipse.jetty.jmx.MBeanContainer listens to those events and registers/unregisters the Jetty components as MBeans into the platform MBeanServer.

The Jetty components are annotated with Jetty JMX annotations so that they can provide specific JMX metadata such as attributes and operations that should be exposed via JMX.

Therefore, when a component is added to the component tree, MBeanContainer is notified, it creates the MBean from the component POJO and registers it to the MBeanServer. Similarly, when a component is removed from the tree, MBeanContainer is notified, and unregisters the MBean from the MBeanServer.

The Maven coordinates for the Jetty JMX support are:

<dependency>
  <groupId>org.eclipse.jetty</groupId>
  <artifactId>jetty-jmx</artifactId>
  <version>12.0.15-SNAPSHOT</version>
</dependency>

Enabling JMX Support

Enabling JMX support is always recommended because it provides valuable information about the system, both for monitoring purposes and for troubleshooting purposes in case of problems.

To enable JMX support on the server:

Server server = new Server();

// Create an MBeanContainer with the platform MBeanServer.
MBeanContainer mbeanContainer = new MBeanContainer(ManagementFactory.getPlatformMBeanServer());

// Add MBeanContainer to the root component.
server.addBean(mbeanContainer);

Similarly on the client:

HttpClient httpClient = new HttpClient();

// Create an MBeanContainer with the platform MBeanServer.
MBeanContainer mbeanContainer = new MBeanContainer(ManagementFactory.getPlatformMBeanServer());

// Add MBeanContainer to the root component.
httpClient.addBean(mbeanContainer);

The MBeans exported to the platform MBeanServer can only be accessed locally (from the same machine), not from remote machines.

This means that this configuration is enough for development, where you have easy access (with graphical user interface) to the machine where Jetty runs, but it is typically not enough when the machine where Jetty runs is remote, or only accessible via SSH or otherwise without graphical user interface support. In these cases, you have to enable JMX Remote Access.

Enabling JMX Remote Access

There are two ways of enabling remote connectivity so that JMC can connect to the remote JVM to visualize MBeans.

Use the com.sun.management.jmxremote system property on the command line. Unfortunately, this solution does not work well with firewalls and is not flexible.
Use Jetty’s ConnectorServer class.

org.eclipse.jetty.jmx.ConnectorServer will use by default RMI to allow connection from remote clients, and it is a wrapper around the standard JDK class JMXConnectorServer, which is the class that provides remote access to JMX clients.

Connecting to the remote JVM is a two step process:

First, the client will connect to the RMI registry to download the RMI stub for the JMXConnectorServer; this RMI stub contains the IP address and port to connect to the RMI server, i.e. the remote JMXConnectorServer.
Second, the client uses the RMI stub to connect to the RMI server (i.e. the remote JMXConnectorServer) typically on an address and port that may be different from the RMI registry address and port.

The host and port configuration for the RMI registry and the RMI server is specified by a JMXServiceURL. The string format of an RMI JMXServiceURL is:

service:jmx:rmi://<rmi_server_host>:<rmi_server_port>/jndi/rmi://<rmi_registry_host>:<rmi_registry_port>/jmxrmi

Default values are:

rmi_server_host = localhost
rmi_server_port = 1099
rmi_registry_host = localhost
rmi_registry_port = 1099

With the default configuration, only clients that are local to the server machine can connect to the RMI registry and RMI server - this is done for security reasons. With this configuration it would still be possible to access the MBeans from remote using a SSH tunnel.

By specifying an appropriate JMXServiceURL, you can fine tune the network interfaces the RMI registry and the RMI server bind to, and the ports that the RMI registry and the RMI server listen to. The RMI server and RMI registry hosts and ports can be the same (as in the default configuration) because RMI is able to multiplex traffic arriving to a port to multiple RMI objects.

If you need to allow JMX remote access through a firewall, you must open both the RMI registry and the RMI server ports.

JMXServiceURL common examples:

service:jmx:rmi:///jndi/rmi:///jmxrmi
  rmi_server_host = local host address
  rmi_server_port = randomly chosen
  rmi_registry_host = local host address
  rmi_registry_port = 1099

service:jmx:rmi://0.0.0.0:1099/jndi/rmi://0.0.0.0:1099/jmxrmi
  rmi_server_host = any address
  rmi_server_port = 1099
  rmi_registry_host = any address
  rmi_registry_port = 1099

service:jmx:rmi://localhost:1100/jndi/rmi://localhost:1099/jmxrmi
  rmi_server_host = loopback address
  rmi_server_port = 1100
  rmi_registry_host = loopback address
  rmi_registry_port = 1099

When ConnectorServer is started, its RMI stub is exported to the RMI registry. The RMI stub contains the IP address and port to connect to the RMI object, but the IP address is typically the machine host name, not the host specified in the JMXServiceURL.

To control the IP address stored in the RMI stub you need to set the system property java.rmi.server.hostname with the desired value. This is especially important when binding the RMI server host to the loopback address for security reasons. See also JMX Remote Access via SSH Tunnel.

To allow JMX remote access, create and configure a ConnectorServer:

Server server = new Server();

// Setup Jetty JMX.
MBeanContainer mbeanContainer = new MBeanContainer(ManagementFactory.getPlatformMBeanServer());
server.addBean(mbeanContainer);

// Setup ConnectorServer.

// Bind the RMI server to the wildcard address and port 1999.
// Bind the RMI registry to the wildcard address and port 1099.
JMXServiceURL jmxURL = new JMXServiceURL("rmi", null, 1999, "/jndi/rmi:///jmxrmi");
ConnectorServer jmxServer = new ConnectorServer(jmxURL, "org.eclipse.jetty.jmx:name=rmiconnectorserver");

// Add ConnectorServer as a bean, so it is started
// with the Server and also exported as MBean.
server.addBean(jmxServer);

server.start();

JMX Remote Access Authorization

The standard JMXConnectorServer provides several options to authorize access, for example via JAAS or via configuration files. For a complete guide to controlling authentication and authorization in JMX, see the official JMX documentation.

In the sections below we detail one way to setup JMX authentication and authorization, using configuration files for users, passwords and roles:

Server server = new Server();

// Setup Jetty JMX.
MBeanContainer mbeanContainer = new MBeanContainer(ManagementFactory.getPlatformMBeanServer());
server.addBean(mbeanContainer);

// Setup ConnectorServer.
JMXServiceURL jmxURL = new JMXServiceURL("rmi", null, 1099, "/jndi/rmi:///jmxrmi");
Map<String, Object> env = new HashMap<>();
env.put("com.sun.management.jmxremote.access.file", "/path/to/users.access");
env.put("com.sun.management.jmxremote.password.file", "/path/to/users.password");
ConnectorServer jmxServer = new ConnectorServer(jmxURL, env, "org.eclipse.jetty.jmx:name=rmiconnectorserver");
server.addBean(jmxServer);

server.start();

The users.access file format is defined in the $JAVA_HOME/conf/management/jmxremote.access file. A simplified version is the following:

users.access

user1 readonly
user2 readwrite

The users.password file format is defined in the $JAVA_HOME/conf/management/jmxremote.password.template file. A simplified version is the following:

users.password

user1 password1
user2 password2

The users.access and users.password files are not standard *.properties files — the user must be separated from the role or password by a space character.

Securing JMX Remote Access with TLS

The JMX communication via RMI happens by default in clear-text.

It is possible to configure the ConnectorServer with a SslContextFactory so that the JMX communication via RMI is encrypted:

Server server = new Server();

// Setup Jetty JMX.
MBeanContainer mbeanContainer = new MBeanContainer(ManagementFactory.getPlatformMBeanServer());
server.addBean(mbeanContainer);

// Setup SslContextFactory.
SslContextFactory.Server sslContextFactory = new SslContextFactory.Server();
sslContextFactory.setKeyStorePath("/path/to/keystore");
sslContextFactory.setKeyStorePassword("secret");

// Setup ConnectorServer with SslContextFactory.
JMXServiceURL jmxURL = new JMXServiceURL("rmi", null, 1099, "/jndi/rmi:///jmxrmi");
ConnectorServer jmxServer = new ConnectorServer(jmxURL, null, "org.eclipse.jetty.jmx:name=rmiconnectorserver", sslContextFactory);
server.addBean(jmxServer);

server.start();

It is possible to use the same SslContextFactory.Server used to configure the Jetty ServerConnector that supports TLS also for the JMX communication via RMI.

The keystore must contain a valid certificate signed by a Certification Authority.

The RMI mechanic is the usual one: the RMI client (typically a monitoring console) will connect first to the RMI registry (using TLS), download the RMI server stub that contains the address and port of the RMI server to connect to, then connect to the RMI server (using TLS).

This also mean that if the RMI registry and the RMI server are on different hosts, the RMI client must have available the cryptographic material to validate both hosts.

Having certificates signed by a Certification Authority simplifies by a lot the configuration needed to get the JMX communication over TLS working properly.

If that is not the case (for example the certificate is self-signed), then you need to specify the required system properties that allow RMI (especially when acting as an RMI client) to retrieve the cryptographic material necessary to establish the TLS connection.

For example, trying to connect using the JDK standard JMXConnector with both the RMI server and the RMI registry via TLS to domain.com with a self-signed certificate:

// System properties necessary for an RMI client to trust a self-signed certificate.
System.setProperty("javax.net.ssl.trustStore", "/path/to/trustStore");
System.setProperty("javax.net.ssl.trustStorePassword", "secret");

JMXServiceURL jmxURL = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://domain.com:1100/jmxrmi");

Map<String, Object> clientEnv = new HashMap<>();
// Required to connect to the RMI registry via TLS.
clientEnv.put(ConnectorServer.RMI_REGISTRY_CLIENT_SOCKET_FACTORY_ATTRIBUTE, new SslRMIClientSocketFactory());

try (JMXConnector client = JMXConnectorFactory.connect(jmxURL, clientEnv))
{
    Set<ObjectName> names = client.getMBeanServerConnection().queryNames(null, null);
}

Similarly, to launch JMC:

$ jmc -vmargs -Djavax.net.ssl.trustStore=/path/to/trustStore -Djavax.net.ssl.trustStorePassword=secret

These system properties are required when launching the ConnectorServer too, on the server, because it acts as an RMI client with respect to the RMI registry.

JMX Remote Access with Port Forwarding via SSH Tunnel

You can access JMX MBeans on a remote machine when the RMI ports are not open, for example because of firewall policies, but you have SSH access to the machine using local port forwarding via an SSH tunnel.

In this case you want to configure the ConnectorServer with a JMXServiceURL that binds the RMI server and the RMI registry to the loopback interface only: service:jmx:rmi://localhost:1099/jndi/rmi://localhost:1099/jmxrmi.

Then you setup the local port forwarding with the SSH tunnel:

$ ssh -L 1099:localhost:1099 <user>@<machine_host>

Now you can use JConsole or JMC to connect to localhost:1099 on your local computer. The traffic will be forwarded to machine_host and when there, SSH will forward the traffic to localhost:1099, which is exactly where the ConnectorServer listens.

When you configure ConnectorServer in this way, you must set the system property -Djava.rmi.server.hostname=localhost, on the server. This is required because when the RMI server is exported, its address and port are stored in the RMI stub. You want the address in the RMI stub to be localhost so that when the RMI stub is downloaded to the remote client, the RMI communication will go through the SSH tunnel.

Jetty JMX Annotations

The Jetty JMX support, and in particular MBeanContainer, is notified every time a bean is added to the component tree.

The bean is scanned for Jetty JMX annotations to obtain JMX metadata: the JMX attributes and JMX operations.

// Annotate the class with @ManagedObject and provide a description.
@ManagedObject("Services that provide useful features")
class Services
{
    private final Map<String, Object> services = new ConcurrentHashMap<>();
    private boolean enabled = true;

    // A read-only attribute with description.
    @ManagedAttribute(value = "The number of services", readonly = true)
    public int getServiceCount()
    {
        return services.size();
    }

    // A read-write attribute with description.
    // Only the getter is annotated.
    @ManagedAttribute(value = "Whether the services are enabled")
    public boolean isEnabled()
    {
        return enabled;
    }

    // There is no need to annotate the setter.
    public void setEnabled(boolean enabled)
    {
        this.enabled = enabled;
    }

    // An operation with description and impact.
    // The @Name annotation is used to annotate parameters
    // for example to display meaningful parameter names.
    @ManagedOperation(value = "Retrieves the service with the given name", impact = "INFO")
    public Object getService(@Name(value = "serviceName") String n)
    {
        return services.get(n);
    }
}

The JMX metadata and the bean are wrapped by an instance of org.eclipse.jetty.jmx.ObjectMBean that exposes the JMX metadata and, upon request from JMX consoles, invokes methods on the bean to get/set attribute values and perform operations.

You can provide a custom subclass of ObjectMBean to further customize how the bean is exposed to JMX.

The custom ObjectMBean subclass must respect the following naming convention: <package>.jmx.<class>MBean. For example, class com.acme.Foo may have a custom ObjectMBean subclass named com.acme.jmx.FooMBean.

//package com.acme;
@ManagedObject
class Service
{
}

//package com.acme.jmx;
class ServiceMBean extends ObjectMBean
{
    ServiceMBean(Object service)
    {
        super(service);
    }
}

The custom ObjectMBean subclass is also scanned for Jetty JMX annotations and overrides the JMX metadata obtained by scanning the bean class. This allows to annotate only the custom ObjectMBean subclass and keep the bean class free of the Jetty JMX annotations.

//package com.acme;
// No Jetty JMX annotations.
class CountService
{
    private int count;

    public int getCount()
    {
        return count;
    }

    public void addCount(int value)
    {
        count += value;
    }
}

//package com.acme.jmx;
@ManagedObject("the count service")
class CountServiceMBean extends ObjectMBean
{
    public CountServiceMBean(Object service)
    {
        super(service);
    }

    private CountService getCountService()
    {
        return (CountService)super.getManagedObject();
    }

    @ManagedAttribute("the current service count")
    public int getCount()
    {
        return getCountService().getCount();
    }

    @ManagedOperation(value = "adds the given value to the service count", impact = "ACTION")
    public void addCount(@Name("count delta") int value)
    {
        getCountService().addCount(value);
    }
}

The scan for Jetty JMX annotations is performed on the bean class and all the interfaces implemented by the bean class, then on the super-class and all the interfaces implemented by the super-class and so on until java.lang.Object is reached. For each type — class or interface, the corresponding *.jmx.*MBean is looked up and scanned as well with the same algorithm. For each type, the scan looks for the class-level annotation @ManagedObject. If it is found, the scan looks for method-level @ManagedAttribute and @ManagedOperation annotations; otherwise it skips the current type and moves to the next type to scan.

@ManagedObject

The @ManagedObject annotation is used on a class at the top level to indicate that it should be exposed as an MBean. It has only one attribute to it which is used as the description of the MBean.

@ManagedAttribute

The @ManagedAttribute annotation is used to indicate that a given method is exposed as a JMX attribute. This annotation is placed always on the getter method of a given attribute. Unless the readonly attribute is set to true in the annotation, a corresponding setter is looked up following normal naming conventions. For example if this annotation is on a method called String getFoo() then a method called void setFoo(String) would be looked up, and if found wired as the setter for the JMX attribute.

@ManagedOperation

The @ManagedOperation annotation is used to indicate that a given method is exposed as a JMX operation. A JMX operation has an impact that can be INFO if the operation returns a value without modifying the object, ACTION if the operation does not return a value but modifies the object, and "ACTION_INFO" if the operation both returns a value and modifies the object. If the impact is not specified, it has the default value of UNKNOWN.

@Name

The @Name annotation is used to assign a name and description to parameters in method signatures so that when rendered by JMX consoles it is clearer what the parameter meaning is.