Skip to content

rate_limiting.md

Latest commit

 

History

History
217 lines (167 loc) · 9.7 KB

rate_limiting.md

File metadata and controls

217 lines (167 loc) · 9.7 KB
Raw

Client IP based Rate Limiting

Rate limits in HAProxy are based on stick-tables. The concept of stick-tables is explained in this blog article. It covers all relevant parts and gives a general idea on how one could rate limit based on certain attributes.

HAProxy-boshrelease can be configured to enforce these rate limits based on requests and connections per second per IP.

Configuration Options

See also jobs/haproxy/spec:

There are two rate limit configuration groups:

  • connections_rate_limit for connection based rate limiting on OSI layer 4/TCP
  • requests_rate_limit for request based rate limiting on OSI layer 7/HTTP

Both groups contain roughly the same attributes:

  • requests (for requests_rate_limit) and connections (for connections_rate_limit): the number of requests/connections allowed within a time window (see window_size) before further incoming requests/connections are denied/blocked
  • window_size: Window size for counting connections
  • table_size: Size of the stick table in which the IPs and counters are stored.
  • block: Whether or not to block connections. If block is disabled (or not provided), incoming requests/connections will still be tracked in the respective stick-tables, but will not be denied.

Effects of Rate Limiting

Once a rate limit is reached, haproxy-boshrelease will no longer proxy incoming requests from the rate-limited client IP to a backend. Depending on the type of rate limiting, HAProxy will respond with one of the following:

Request based Rate Limiting

HAProxy responds to the client with HTTP Status Code: 429: Too Many Requests.

Connection based Rate Limiting

The TCP connection will be rejected. This would for example show up as Empty reply from server for a curl-client. This will not result in a log statement on the HAProxy side, which can make tracing issues more difficult.

Note: If both rate-limits are reached simultaneously (e.g. if they are configured identically and every incoming HTTP request uses a new TCP connection), connection based rate-limiting will come into effect first, resulting in a dropped TCP connection.

Excluding CIDRs from Connection Rate Limiting

Some sources should never be rate-limited on connections — for example internal/NAT ranges, health checkers, or trusted peers whose traffic all appears to originate from a small set of IPs. connections_rate_limit.exclude_cidrs takes a list of CIDRs that are exempt from connection based rate limiting.

Excluded sources are still tracked in the st_tcp_conn_rate stick-table (so they remain visible via show table st_tcp_conn_rate), but they are never rejected, regardless of the configured threshold or the runtime block setting.

The list is rendered to /var/vcap/jobs/haproxy/config/rate_limit_exclusion_cidrs.txt and referenced by an ACL that negates the reject rule on both the http-in and https-in frontends.

Configuration

config:
    # [...]
    connections_rate_limit:
        connections: 10
        window_size: 10s
        table_size: 1m
        block: true
        exclude_cidrs:
        - 10.0.0.0/8
        - 192.168.0.0/16
        - 2001:db8::/32

The value may also be provided as a single base64-encoded, gzipped string (useful for very long lists), matching the format accepted by cidr_whitelist and cidr_blocklist_tcp.

Resulting haproxy.config

frontend http-in
    # [...]
    acl rate_limit_exclude src -f /var/vcap/jobs/haproxy/config/rate_limit_exclusion_cidrs.txt
    tcp-request connection track-sc0 src table st_tcp_conn_rate
    tcp-request connection reject if { var(proc.connections_rate_limit_block) -m bool } { var(proc.connections_rate_limit_connections) -m int gt 0 } { sc_conn_rate(0),sub(proc.connections_rate_limit_connections) gt 0 } !rate_limit_exclude

Configuration Examples

Note: The following examples assume only an http-in frontend is configured; an https-in frontend would behave identically.

Count Incoming Requests Only (No blocking)

Configuration (deployments/haproxy/config.yml)

config:
    # [...]
    requests_rate_limit:
        window_size: 10s
        table_size: 1m

Resulting haproxy.config

backend st_http_req_rate
    stick-table type ipv6 size 1m expire 10s store http_req_rate(10s)
# [...]
frontend http-in
    http-request track-sc1 src table st_http_req_rate

Request based Rate Limiting Fully Active

Configuration

config:
    # [...]
    requests_rate_limit:
        requests: 10
        window_size: 10s
        table_size: 1m
        block: true

Resulting haproxy.config

backend st_http_req_rate
    stick-table type ipv6 size 1m expire 10s store http_req_rate(10s)
# [...]
frontend http-in
    http-request track-sc1 src table st_http_req_rate
    http-request deny status 429 if { sc_http_req_rate(1) gt 10 }

Both Types Active

Configuration

config:
    # [...]
    requests_rate_limit:
        requests: 10
        window_size: 10s
        table_size: 1m
        block: true
    connections_rate_limit:
        connections: 10
        window_size: 10s
        table_size: 1m
        block: true

Resulting haproxy.config

backend st_http_req_rate
    stick-table type ipv6 size 1m expire 10s store http_req_rate(10s)

backend st_tcp_conn_rate
    stick-table type ipv6 size 1m expire 10s store conn_rate(10s)
# [...]
frontend http-in
    # [...]
    http-request track-sc1 src table st_http_req_rate
    http-request deny status 429 if { sc_http_req_rate(1) gt 10 }

    tcp-request content track-sc0 src table st_tcp_conn_rate
    tcp-request connection reject if { sc_conn_rate(0) gt 10}

Querying Current Stick-Table Status

To get more insight into what is going on inside HAProxy regarding its rate limits, you can query the stats socket at /var/vcap/sys/run/haproxy/stats.sock to get the raw table data:

$ echo "show table st_http_req_rate" | socat /var/vcap/sys/run/haproxy/stats.sock -
# table: st_http_req_rate, type: ip, size:10485760, used:1
0x...: key=:ffff:172.18.0.1 use=0 exp=7618 http_req_rate(10000)=10

echo "show table st_tcp_conn_rate" | socat stdio /var/vcap/sys/run/haproxy/stats.sock
# => # table: st_tcp_conn_rate, type: ipv6, size:1048576, used:2
# => 0x...: key=::ffff:203.0.113.42 use=0 exp=8123 shard=0 conn_rate(10000)=5

To find the IP with the highest connection rate, use:

echo "show table st_tcp_conn_rate" | socat stdio /var/vcap/sys/run/haproxy/stats.sock | sort -t= -k2 -rn | head -1

Note: You will likely need sudo permission to run socat.

Control Connection Rate Limiting via HAProxy Runtime API

Normally, changing rate-limit settings requires updating the manifest and reloading HAProxy. Using the HAProxy Runtime API, blocking can be enabled or disabled, and the connection threshold can be tightened or loosened while HAProxy continues running and serving traffic. This is particularly useful during an active incident, when a rapid reaction is needed.

Prerequisites

  • ha_proxy.master_cli_enable: true or ha_proxy.stats_enable: true must be set in the manifest to enable the HAProxy Runtime API.
  • ha_proxy.connections_rate_limit.table_size and ha_proxy.connections_rate_limit.window_size must be defined in the manifest to create the stick table and enable connection tracking.
  • root permissions are required to write to the socket.

How Runtime Control Works

When HAProxy starts, it reads connections_rate_limit.block and connections_rate_limit.connections from the manifest and stores them as process-level variables inside the running HAProxy process. Updating a variable instantly changes the behavior for all subsequent connections, as every new TCP connection is evaluated against these variables in real time.

These variables are updated by sending plain-text commands to the HAProxy stats socket. The socket is available as long as HAProxy is running, and any change persists until the next redeploy, at which point the manifest values are restored.

Note: the connections threshold is applied per the defined window_size, which is also used for counting connections. For example, if window_size is set to 10s and connections is set to 100, then the threshold of 100 connections applies to every 10-second window.

Inspect Current Variable Values

echo "get var proc.connections_rate_limit_connections" | sudo socat stdio /var/vcap/sys/run/haproxy/stats.sock
# => proc.connections_rate_limit_connections: type=sint value=<600>

echo "get var proc.connections_rate_limit_block" | sudo socat stdio /var/vcap/sys/run/haproxy/stats.sock
# => proc.connections_rate_limit_block: type=bool value=<1>

Enable or Disable Blocking at Runtime

# Enable blocking (equivalent to setting block: true in the manifest)
echo "experimental-mode on; set var proc.connections_rate_limit_block bool(true)" | sudo socat stdio /var/vcap/sys/run/haproxy/stats.sock

# Disable blocking without reloading (equivalent to setting block: false in the manifest)
echo "experimental-mode on; set var proc.connections_rate_limit_block bool(false)" | sudo socat stdio /var/vcap/sys/run/haproxy/stats.sock

Adjust the Connections Threshold at Runtime

# Allow up to 100 connections per window (equivalent to setting connections: 100 in the manifest)
echo "experimental-mode on; set var proc.connections_rate_limit_connections int(100)" | sudo socat stdio /var/vcap/sys/run/haproxy/stats.sock

Enable Rate Limiting and Set Threshold in One Step

echo "experimental-mode on; set var proc.connections_rate_limit_connections int(100); set var proc.connections_rate_limit_block bool(true)" | sudo socat stdio /var/vcap/sys/run/haproxy/stats.sock