Introduction

In this blog post, we share our experience from a recent engagement where we identified and exploited an exposed API spring endpoint.

The post aims to inform developers about the risks of exposing Spring Boot actuator endpoints and how it can lead to dangerous outcomes whilst also serving as a guide for bug-bounty hunters and security researchers that will provide ideas on demonstrating the impact of this misconfiguration.

Background

Spring Boot is an open-source framework that simplifies developing, deploying, and maintaining Java-based applications.

It allows developers to focus on application development instead of configuring complex systems and provides specific endpoints known as actuator endpoints for several functions, such as server health, info, environment variables, and heap dumps.

As will be demonstrated further in this blog, not all endpoints are intended to be publicly exposed.

Official documentation outlining this can be found here: Spring Boot Actuator: Exposing Endpoints.

Below is an overview of exposable spring boot actuator endpoints with a short description of what functionality they provide.

• /restart: restarts the application

• /shutdown: shuts the application down

• /mappings: shows all the MVC controller mappings

• /env: provides access to the configuration environment

• /threadump: displays thread dumps, including stack trace

• /heapdump: displays heap dumps that may have sensitive data like credentials

• /logfile: outputs contents of application logs, which may contain debugging details.

• /trace: displays the last several HTTP messages, which could include session identifiers

An important note, in Spring 1. x, all actuator endpoints are accessible without any authentication.

Although multiple spring boot endpoints can introduce risk to an application when publicly exposed, this post will focus on the risks specific to the heapdump endpoint.

What is a heapdump?

A heap dump is a snapshot of an application's memory at a specific point in time. It contains data structures, objects, and other memory-related information used by the application; this provides valuable insight into a system's internal workings, which can be helpful for developers when diagnosing performance issues or investigating potential security vulnerabilities.

The Risks of Exposed Heap Dumps

While heap dumps are valuable for troubleshooting and analysis, they can pose significant security risks if not adequately protected.

Exposed heapdumps can reveal sensitive information, such as passwords, API keys and encryption keys, that can be extracted by malicious actors.

Additionally, heapdump’s can provide insight into the application's architecture, revealing versioning and other internal information, making it easier for attackers to locate relevant exploits and vulnerabilities.

Once an attacker gains access to sensitive credentials through an exposed heapdump, it can lead to additional compromise, including unauthorised access to customer data, system access, and privilege escalation.

Developers and security professionals often underestimate the security impact of exposing heap dumps to the public; even after implementing access restrictions to these endpoints, they may still be vulnerable, as the attacker could have previously obtained the full heapdump, retaining access to all keys and sensitive information in memory.

It is crucial to recognise and address this threat to protect the integrity of your organisation's data and systems.

Extracting secrets from a memory dump

Whether you’re defending against or exploiting this type of misconfiguration as part of a security assessment, it is important to understand how an attacker can use developer-intended tools to extract sensitive information from heapdump’s once downloaded and stored offline.

Java VisualVM is a tool that provides a visual interface for viewing detailed information about Java applications while they are running on a Java Virtual Machine (JVM).

Exploiting heap dumps in the real world

To provide the reader with an understanding of how this could be exploited in the real world, here is an example of a recent finding that would be identical to the attack flow performed by an external attacker.

Finding the heapdump

Our team recently identified an exposed heapdump endpoint, which allowed us to download the heap dump file without any authentication due to a misconfiguration covered later in this blog.

Typically the heapdump, if exposed can be accessed with a browser after requesting endpoints similar to the following:

<https://target.com/actuator/heapdump>
<https://target.com/heapdump>
<https://target.com/api/actuator/heapdump>

It’s important when attempting to identify this exposure to check all ports if not at-least common web & API ports such as but not limited to: 80, 8080, 8081, 8082, 443, 10433 and so on.

Once identified, this critical oversight provided us unrestricted access to sensitive information in the application's memory.

By exploiting this misconfiguration, we were able to replicate the attack flow that an external attacker might follow, emphasising the importance of securing such endpoints.

Extracting Secrets from heap dumps using OQL

Once downloaded, a heapdump can be loaded into a too such as VisualVM to parse and search the heapdump for sensitive information, such as API keys, using specific OQL queries as seen below.

1. Loading the heapdump into VisualVM.

After obtaining the heap dump, we loaded the file into VisualVM.

1. Extracting heapdump secrets using OQL queries

Once the heap dump was loaded into VisualVM, we used the OQL console to search for sensitive information within the application's memory. In this case, we focused on extracting AWS API keys, as they could grant further access to the client's AWS resources.

What is OQL?

VisualVM includes an Object Query Language (OQL) console that allows users to perform complex queries on the heap dump data to inspect, filter, and analyse objects in the Java heap.

Attackers can use this feature to discover application secrets, reveal system information, and extract credentials.

Because the AWS Access Key ID is a predicable string, we crafted an OQL query targeting the known AWS API Key prefix (AKIA):

SELECT s FROM java.lang.String s WHERE s.toString().startsWith("AKIA")

It’s important to note that many other types of data that might be stored in an applications memory such as but not limited to; API keys, Credit Card data, credentials, SQL queries and more.

As such the OQL query can be modified to search for other strings as seen below

SELECT s FROM java.lang.String s WHERE s.toString().startsWith("apikey") 
SELECT s FROM java.lang.String s WHERE s.toString().startsWith("secret") 
SELECT s FROM java.lang.String s WHERE s.toString().startsWith("api_key") 
SELECT s FROM java.lang.String s WHERE s.toString().startsWith("password") 
SELECT s FROM java.lang.String s WHERE s.toString().startsWith("jwt")

Although a number of AWS Access Keys were identified within the heapdump, without the corresponding AWS Secret Keys they would be unusable.

Because AWS Secret Keys do not contain any common prefix, it was not as simple as identifying the AWS Access Key IDs which left us with two two options:

• Using OQL based regex to identify high entropy strings - This option was not appealing as there is very limited documentation on VisualVM’s OQL Regex support.

• Looking at adjacent memory locations - This option turned out to be highly useful not only for AWS Secrets but for many other API secrets and passwords that cannot be found using basic string based searches.

When analysing heapdumps with VisualVM, you might notice that AWS Access Key IDs and their corresponding secrets often appear in adjacent positions, such as java.lang.String#72811: AKIAXXXXXXXXXX and java.lang.String#72812: SECRETKEY. This is because of how the application stores these credentials in memory.

In many applications, developers store AWS Access Key IDs and their corresponding secret access keys as variables, often within the same scope or in close proximity to each other in the code. When the Java Virtual Machine (JVM) processes this code, it stores these variables in memory, typically as instances of the java.lang.String class.

During the execution of the application, the JVM manages the memory by allocating and deallocating objects. Since the AWS Access Key ID and its corresponding secret are closely related and used together, the JVM is likely to allocate memory for these strings in adjacent memory locations.

Using this knowledge we focused on the reference region asssociated with the AWS Access Key ID which is String#74620

As seen above, the AWS access key and the subsequent secret were stored adjacently in memory.

At this point we proceeded to take complete control of the target AWS account eventually obtaining administrative AWS privileges via the AWS command line interface.

How can I check if my company has an exposed heapdump?

As demonstrated, an exposed heapdump can have serious consequences, making it an attractive target for both security researchers and malicious actors.

Given it’s prevalence, numerous vulnerability scanning tools contain plugins to identify this misconfiguration.

One of which is an open-source tool called Nuclei that can be used to scan for these types of endpoints.

The tool including installation and usage instructions can be found on ProjectDiscovery’s Github https://github.com/projectdiscovery/nuclei.

Below is an example of Nuclei identifying multiple exposed heapdump endpoints on random Amazon EC2 servers.

To run this scan yourself you can use the following command

nuclei -l domains.txt -t nuclei-templates/misconfiguration/springboot/springboot-heapdump.yaml

Where domains.txt contains a list of domains or IP addresses that you want to scan.

Alternatively you can run the scan on a single domain or IP as follows

nuclei -l https://api.target.com -t nuclei-templates/misconfiguration/springboot/springboot-heapdump.yaml

Remediation & Response

The following section covers steps we recommend that organisations should take if should you discover that your heapdump endpoint has been exposed, including checking logs, analysing the heap dump, rotating secrets, and reviewing logs.

The root cause for this type of exposure is generally due to a configuration containing the following definition.

management.endpoints.web.exposure.include=* 
management.endpoints.web.exposure.exclude=env,beans

The configuration above exposes all actuator endpoints by setting the value of management.endpoints.web.exposure.include to =* which represents all management endpoints.

In Spring 2.x, apart from the /health and /info, all endpoints are considered sensitive and are not exposed by default as such developers should avoid exposing endpoints other than /health and /info in a production environment.

Securing Endpoints

To disable all endpoints by default, update the application.properties file with the following configuration.

management.endpoints.enabled-by-default = false

After disabling all endpoints, you can selectively enable the required endpoints by updating the respective configuration settings.

For example, to enable the health endpoint, add the following line:

management.endpoint.health.enabled = true

Responding to an exposed heapdump

As detailed in this post, responding to an exposed heapdump endpoint is a high priority action as it can pose a significant risk to an organisation.

The following is a high level guide on why actions to perform in the event of a confirmed or suspected heapdump exposure regardless of how long the endpoint has been exposed.

Step 1: Review Logs for Heapdump Access

The first step is to review your server logs to determine if anyone has accessed or downloaded the heap dump. Look for any suspicious activity or IP addresses that may indicate unauthorised access. If you find evidence of unauthorised access, it is crucial to act quickly to mitigate potential risks.

Step 2: Analyse the Heapdump

If the heapdump was accessed, it's essential to analyse its contents to understand the extent of the potential data exposure. Load the heap dump into a tool like VisualVM and search for sensitive information, such as API keys, secrets, and configuration variables. By doing this, you can assess the potential impact and prioritise your response efforts.

Step 3: Identify and Rotate Secrets

Once you've determined which secrets may have been exposed, consult your development team to create a comprehensive list of secrets used in the affected server or application. Rotate all exposed secrets, including API keys, passwords, and tokens, to prevent unauthorised access to your systems.

Step 4: Review Logs for Related Services

After rotating the secrets, review logs for any services or systems associated with the exposed keys. Look for suspicious activity or unauthorised access attempts that could indicate the exploitation of the exposed secrets. If you find any signs of compromise, take immediate action to contain the potential breach and conduct a thorough investigation.