A Guide to Reversing and Evading EDRs: Part 3


Diverting EDR Telemetry to Private Infrastructure

As I mentioned in the introduction, one of the objectives in this series is to be able to get access to telemetry in a lab environment without needing to use the product console. To me, this is game changing because we can use the event data as a feedback loop to "rehearse" attacks and develop bypasses — all without tipping off defenders. It's like having a private VirusTotal but for EDR instead of AV.

This idea isn't entirely novel. It's already possible to have a similar lab setup where telemetry is collected from free tools like Sysmon and osquery, but being able to privately collect telemetry from the specific EDR products you're up against provides an extra level of precision that's hard to compare. For example, several products have built-in detections and it would be useful to know in advance if an action would have triggered an alert:

It's worth emphasizing that gaining access to sensor telemetry will not necessarily tell you what gets detected. Although some products build a degree of detection logic into sensors (as shown above), many detection rules will be applied on the server-side. For example, if PowerShell is spawned from Internet Explorer, the sensor may produce telemetry that documents the child-process creation, and the server/cloud may produce a detection if it considers the behaviour to be suspicious. With the approach in this post, you won't necessarily gain access to server-based detections, but you will gain access to every produced event that flows into detections.

General Approaches

When it comes to extracting EDR event data out of sensors, some coding is involved and there are a couple ways to approach this task. It depends on the degree to which you want your code to be compatible with the EDR, or conversely, the degree to which the EDR should be compatible with your code.

On one end, you can write code for a mock server that mimics functionality of the product server. With this, you can simply change the sensor configuration to point to your mock server instead of the original one and collect the events from there. On the other end, you can instrument/patch the sensor code to dump events in the format you prefer. Sometimes it involves a bit of both.

I'd recommend starting with the approach that's easiest to make a proof-of-concept of. The benefit is that further dependent tasks in the project pipeline are not blocked. After that, you can circle back to see where improvements in performance and stability can be made. My preference is to write a mock server because, while it usually takes more time upfront, it's easier to maintain in the long term and involves less patching/instrumentation which could be brittle on some products.

Capturing and Parsing Raw Telemetry

A common starting point is to find out where to access the raw events and how to parse them.

Some products store a cache of events on disk in database files or temporary archives, whereas others store those buffers in memory only. If the events are accessible on disk, then it's worth investigating whether you can fetch that data without having to take additional steps to write a mock server or modify the sensor. The event buffers could be stored encrypted or a tampering alert might be triggered if they're accessed from an unexpected process, so those are potential impedements to be mindful of — especially if you could be in the position of "getting caught" during the research phase.

Once you have access to event buffers, the next step is to deserialize them. Events can be serialized in human-readable formats (e.g. JSON) or binary (e.g. ProtoBuf), but the most common I've noticed is the latter. Schema-ful binary formats have efficiency gains, but they also come with an added layer of obscurity as field names are lost. If a common binary format is being used, the sensor executable will likely have clues as to which one is being used. It's also worthwhile reading up on how they're structured at a byte level.

This stage can significantly slow down efforts for insufficiently-resourced developers who lack access to event schemas. Logging the raw serialized buffers into your private SIEM won't be productive for analysis, so the schemas need to be recovered somehow. Depending on the product, you might get lucky and find the schema online, or be able to extract them from the sensor binary, or use documentation to manually reconstruct them. In that case, the OSSEM project has data dictionaries for several products.

Examples and Recap

Here are a couple of fictional scenarios to illustrate steps that can be taken to capture and parse EDR telemetry:

  • Scenario 1: The sensor is configured to periodically check-in every minute with an on-premise server over HTTPS. Every time it sends events, a file containing telemetry is dropped to disk and is deleted after a 200 OK response is received to the upload request. The structure of the file indicates a series of ProtoBuf objects but no schema is publicly available. Fortunately, the ProtoBuf file descriptors are still embedded in the binary and can be extracted out using the PBTK tool. With the event schemas recovered, the events themselves can be retrieved by reading the dropped files before they're deleted, or by pointing the sensor to a mock HTTP server written to accept the uploaded file.

  • Scenario 2: This sensor connects to a hardcoded cloud-based endpoint over a secure WebSocket connection and the events are continuously streamed to it. They are also cached in an encrypted SQLite database. Some documentation is available about the event schemas and a tedious manual process is required to reconstruct them. With the schemas mostly recovered, a similar decision point from the prior scenario can be made for receiving the event buffers. One option could be to determine how to decrypt the database, and then poll it for new events. Another option could be to write a mock WebSocket server to receive the events. Seeing as the default sensor connection is hardcoded, some modifications would be made to point it to your server and bypass certificate pinning.

To recap, you should ask yourself the following questions when capturing raw EDR telemetry. These can help you determine the next course of action or additional avenues for investigation:

  1. Is the telemetry being cached on disk or in memory?
  2. If on disk, could it be in a database file, a temporary cache file, or be "fileless"?
  3. If in memory, how feasible is it to extract from memory or write a mock server?
  4. Does the product offer a diagnostic mechanism to query the telemetry?
  5. Do the buffers need to be decrypted or deobfuscated before being deserialized?
  6. How are they serialized (e.g. ProtoBuf, JSON, Bond, custom)?
  7. Are schemas for the event types available? If not, can they be recovered?

Implementing a Mock Server

All EDR products will send events to a server, and writing a mock server can allow you to receive the events in the way they are intended to be received by the server-side component of the product. This approach may not be worthwhile for every product, but I'd like to discuss it as there are some situations where it makes more sense.

Receiving a Connection from the Sensor

The first step of implementing a mock server is to determine whether the EDR allows for connections to on-premise servers or if it's strictly cloud managed only. On-premise servers inherently allow for configuration changes where you can point it to your mock server instead (e.g. by modifying config files or by using diagnostic tools). A cloud managed server usually requires more effort. In the cloud scenario, modifying the hosts file can be an option — assuming the sensor doesn't fall back to a hardcoded list of IP addresses if it detects an anomaly.

Some sensors also enforce certificate pinning. If the server details are configurable, then there might be an option to replace the stored certificate and use your own instead. If no configuration option exists (e.g. cloud managed sensors), then it's worth taking a closer look in the binary as to how the certificates are validated. Some products might validate the CN only so you can adapt your self-signed certificate to make it match. Other products might validate the entire certificate, which could be resolved by patching the binary. For example, the function that performs the validation can be patched to always return true.

Protocol Identification and Analysis

Now that you're able to establish a connection, you can perform analysis of imported APIs or libraries to determine whether a common network protocol (e.g. HTTP, WebSocket) or a custom protocol is being used.

If HTTPS is used, you can write a small Python script to get an idea of what the requests look like:

import ssl
from http.server import HTTPServer, SimpleHTTPRequestHandler

httpd = HTTPServer(('0.0.0.0', 443), SimpleHTTPRequestHandler)
httpd.socket = ssl.wrap_socket(httpd.socket, server_side=True, certfile='cert.pem')
httpd.serve_forever()
127.0.0.1 - - [27/Feb/2020 10:45:12] "POST /register HTTP/1.1" 200 -
127.0.0.1 - - [27/Feb/2020 10:45:14] "POST /check-in HTTP/1.1" 200 -
127.0.0.1 - - [27/Feb/2020 10:45:59] "POST /submit-events HTTP/1.1" 200 -
127.0.0.1 - - [27/Feb/2020 10:46:14] "POST /check-in HTTP/1.1" 200 -
127.0.0.1 - - [27/Feb/2020 10:47:14] "POST /check-in HTTP/1.1" 200 -
127.0.0.1 - - [27/Feb/2020 10:47:29] "POST /submit-events HTTP/1.1" 200 -
127.0.0.1 - - [27/Feb/2020 10:48:14] "POST /check-in HTTP/1.1" 200 -

The content type of these requests should be determined next (e.g. JSON, ProtoBuf). If a binary serialization format is being used, you can analyze the sensor further to see if there are embedded descriptors which can be used to parse the buffers into a more human-readable format.

The example code below receives connections over WebSocket and parses incoming ProtoBuf messages without requiring a schema (using protobuf-inspector by @mild_sunrise):

from asyncio import get_event_loop
from datetime import datetime
from io import BytesIO
from websockets import serve

# Using https://github.com/mildsunrise/protobuf-inspector.
from lib.types import StandardParser

# Load ProtoBuf parser.
p = StandardParser()
p.types['Message'] = {}

async def handler(websocket, path):
    data = await websocket.recv()
    print(f'Received message at {datetime.now().isoformat()}.')
    print(p.safe_call(p.match_handler('message'), BytesIO(data), 'Message'))

server = serve(handler, 'localhost', 8765)
get_event_loop().run_until_complete(server)
get_event_loop().run_forever()
Received message at 2020-03-06T19:01:49.817219.
Message:
    1 <varint> = 1
    2 <varint> = 2
    3 <varint> = 3
    4 <chunk>  = bytes (738)
        0000  1F 8B 08 00 1D 75 A0 5F 89 40 89 93 93 5E A2 69  .....u._...T[o.0
        0010  7D 1D 5D 6C 81 B1 37 8A 14 A0 15 38 85 5E EF 59  ..+(.K.K.;}KoZ..
        0020  BB 8F 82 76 68 7F 3E 57 0B 24 F7 B6 A2 10 41 A3  Uk.h...u+.4.lg0.
        ...

Sensor—Server Flows

Now that you're able to receive and parse messages coming from the sensor, you'll want to mimic the responses from a normal server so that you can receive the telemetry.

A typical flow between a sensor and server could look like this:

Depending on the product, there is usually an initial request where the sensor will register with the server to let it know that it's online. After the server responds back with acknowledgment and/or tasking, the sensor will start sending events over on a periodic or continuous basis. The server will usually respond back with acknowledgement (e.g. a simple 204, or a matching correlation ID).

A proxying tool like Fiddler or tcpprox can be used to review the legitimate requests and responses so that the mock server can be adapted to emulate the responses more accurately. If using a proxy isn't feasible, a breakpoint in the sensor can be set after the response has been decrypted.

Examples and Recap

Here are a few fictional scenarios to illustrate the steps that can be taken to implement a mock server:

  • Scenario 1: The product makes connections to on-premise servers and their URLs can be configured using the registry. As the URL indicates HTTPS, a simple Python server can be instantiated to collect requests. The mock server's certificate can also be specified in the registry to satisfy certificate pinning stipulations. Events are periodically uploaded as Zip files in POST requests, and after extraction they can be parsed as ProtoBuf messages. They are acknowledged with a 204 response.

  • Scenario 2: By default, this product connects to a cloud managed server but a diagnostic tool can be used to point it to another server and allow use of a self-signed certificate. Some cursory analysis of configurations and imported libraries indicates use of the WebSocket protocol. With certificate pinning checks disabled, the decrypted WebSocket traffic can be viewed in Fiddler where it's determined that the messages are serialized with ProtoBuf. The server responds to the sensor's initial request with an optional array of taskings. Sent events are acknowledged with ProtoBuf messages returning correlation IDs.

  • Scenario 3: This product can only connect to a cloud managed server that is hardcoded in the binary and has no configuration options. Breakpoints set in the sensor indicate that a custom protocol is being used. Certificate pinning is bypassed by patching the binary. The server responds to the sensor's initial request with a token for the session. Sent events are also acknowledged with messages returning correlation IDs.

To recap, you should ask yourself the following questions when implementing a mock server. These can help you determine the next course of action or additional avenues for investigation:

  1. Are connections made to a configurable on-premise server or a hardcoded managed server?
  2. Can modifications to sensors be made through configuration changes or diagnostic tools?
  3. If binary patches are required, which approaches would be the least brittle?
  4. If patches are in kernel-mode, how should they be applied (e.g. abusing wormhole drivers)?
  5. Are connections being established from the service executable and/or the driver?
  6. Can communications be introspected using a proxy (e.g. Burp, Fiddler)?
  7. Which network protocol is used (e.g. HTTPS, WSS, custom w/ TLS)?
  8. How are TLS certificates being validated and is it possible to use self-signed certificates?
  9. Are there certificate pinning checks to bypass?
  10. What does the initial exchange with the server look like?
  11. How are received events acknowledged (e.g. 200 OK, returned correlation ID)?

Configuring Elastic Stack

Now that you have a mock server that can receive deserializable telemetry, you can ship the events over to your private SIEM. In this series, I'm using Elastic Stack (a.k.a. ELK) because it's a free and popular option. Fair warning though, as I hadn't used ELK prior to this project and I'm still a newbie.

Logstash

Logstash, the L in ELK, can be used to ingest logs. In my setup, I configured an HTTP listener that accepted JSON data. The mock server would receive events from the sensor, deserialize them, convert them JSON, and send POST requests to the Logstash listener. In the configuration below, a couple filters are added to map the event's timestamp and remove extraneous HTTP headers.

input {
  http {
    host => "0.0.0.0"
    port => 8080
  }
}

filter { 
  mutate {
    remove_field => "headers"
  }

  date {
    match => ["unix_timestamp", "UNIX"]
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "events"
  }
}

Kibana

With the events going through Logstash, you can create an index in Kibana and watch the events roll in. After a sizeable amount of events have been imported, you should consider refreshing the field list in the settings page for the index.

Within the Discover page, you can write queries such as the one below to view recent command lines and their parent processes. You can also write a query to view a list of events tied to a specific process and use this to get a granular view of what the EDR notices about your implant.

Every implant action can be tested to determine what events they generate and whether any of combination of them will lead to a detection. An important caveat about this is that you usually won't get access to any custom server-side detection queries at the sensor level, but you will be able to see the raw events that flow into them.

You can also create visualizations to add to a dashboard, such as this one that shows a breakdown of event types over time and flags tampering events.

There are many opportunities to mine this data and get a sense of what looks "normal" on a given system. This will be discussed further in the next article for Blending In techniques.

Summary

The sequence diagrams below summarize the approaches discussed in this article at a higher level. For some sensors, it could be reasonable to use an event polling mechanism that fetches new events from a database file or temporary cache. Fetched events would be shipped to Logstash so that the data can be queried with Kibana.

For other sensors, writing a mock server could be a more stable approach. The sensor would be reconfigured to send telemetry to the mock server which, in turn, will send them to Logstash.

In some cases where kernel-driver modifications are appropriate for rerouting telemetry to your mock server, then the diagram below could be more fitting.

This was a long read to get through, so congrats for making it to the end, and thank you for taking the time to read it! We went over the process of accessing sensor telemetry with the goal of being able to divert it to private infrastructure. Accomplishing this sets the stage for developing the feedback loop that allows us to discover techniques for blending in, abusing blind spots, and sensor tampering. Stay tuned for future posts that will focus on each of those topics!

Please ping me if you have any feedback, questions, or notice any errors.

@Jackson_T