Post

AWS WAF Beyond the Basics: Rate-Based Rules, Custom Rules, WAF Logs, and Moving from Count to Block

A follow-up to the web ACL intro: how to set up AWS WAF rate-based rules, handle false positives with custom rules, choose between CloudWatch Logs and S3 for WAF logs, and what to check before switching rules from Count to Block.

AWS WAF Beyond the Basics: Rate-Based Rules, Custom Rules, WAF Logs, and Moving from Count to Block

In the previous article, we created a web ACL and set the AWS managed rule groups to Count. The next questions usually look like this: how do I stop brute-force attempts on the login page? What do I do when a rule blocks file uploads? And after running in Count for a while, when is it safe to switch to Block?

Here’s the short answer up front: rate-based rules handle “the same source is sending too much,” custom rules handle “exceptions or restrictions under specific conditions,” and WAF logs are what tell you whether a rule is ready to move from Count to Block. Switching to Block without logs is guesswork.

Rate-based rules: limiting requests to the login page

A rate-based rule counts requests from each source over a time window and applies an action once the count goes over a limit. The most common use is limiting how many times a single IP can hit a login API. AWS: Rate-based rule statement

A few settings are worth understanding first:

SettingWhat it does
Evaluation window60, 120, 300, or 600 seconds; the default is 300
Rate limitNumber of requests allowed within the window; the minimum is 10
AggregationBy source IP, by an IP in a header (such as X-Forwarded-For), by custom keys, or by counting all requests together
Scope-down statementOnly counts requests that match a condition, such as requests to /login
ActionBlock, Count, CAPTCHA, or Challenge

AWS: Rate-based rule high-level settings

I’d recommend always adding a scope-down statement. Without one, every request to the site counts toward the limit. A single page load can trigger dozens of requests for static assets, so even normal users could get blocked. If you only count paths that tend to be abused, like login, sign-up, and password reset, the limit is much easier to tune and less likely to hit legitimate users.

Also keep in mind that rate-based rules don’t count each request in real time. There’s a short delay between going over the limit and the action kicking in, and the number of requests blocked won’t match the limit exactly. They work well for stopping obvious floods, not as a precise request counter. AWS: Rate-based rule caveats

If CloudFront sits in front of your ALB

If your architecture is CloudFront → ALB and the web ACL is associated with the ALB, the source IP the ALB sees belongs to a CloudFront edge server, not the user. Aggregating by source IP in this setup lumps a large group of users together as a single source.

Aggregating by X-Forwarded-For instead seems to fix this, but WAF uses the first IP in that header, and clients can set it to whatever they want. An attacker can bypass the limit by sending a different fake IP with each request. The more reliable approach is to associate the web ACL with CloudFront, so WAF sees the real client IP directly. AWS: Forwarded IP address

Terraform example

Add the following to aws_wafv2_web_acl.app from the previous article. If a single IP sends more than 100 requests to /api/login within 5 minutes, WAF responds with 429 Too Many Requests. A default Block returns 403, which makes it hard for the frontend to tell rate limiting apart from a permissions problem, so I usually change it to 429.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
rule {
  name     = "rate-limit-login"
  priority = 10

  action {
    block {
      custom_response {
        response_code = 429
      }
    }
  }

  statement {
    rate_based_statement {
      limit                 = 100
      evaluation_window_sec = 300
      aggregate_key_type    = "IP"

      scope_down_statement {
        byte_match_statement {
          search_string         = "/api/login"
          positional_constraint = "STARTS_WITH"

          field_to_match {
            uri_path {}
          }

          text_transformation {
            priority = 0
            type     = "LOWERCASE"
          }
        }
      }
    }
  }

  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "app-rate-limit-login"
    sampled_requests_enabled   = true
  }
}

100 is only an example. The right number depends on how your real users behave. For an internal system where everyone shares the same egress IP, the limit needs to be much higher. If you’re not sure, change action to count {}, let it run for a while, and use the logs to see where normal peaks land before deciding. Terraform: rate_based_statement

Info: evaluation_window_sec is a relatively new argument. Older versions of the Terraform AWS Provider won’t recognize it, so upgrade the provider if you get an error.

Custom rules: covering what managed rules can’t

Managed rules are general-purpose, so sooner or later you’ll hit a case where they don’t fit your site. That’s when you write your own rules. Two cases come up often.

Example 1: allowing the admin area only from office IPs

First, store your office egress IPs in an IP set. Then write a rule that says: if the path starts with /admin and the source is not in the IP set, block it.

1
2
3
4
5
6
resource "aws_wafv2_ip_set" "office" {
  name               = "office-ips"
  scope              = "REGIONAL"
  ip_address_version = "IPV4"
  addresses          = ["203.0.113.0/24"]
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
rule {
  name     = "admin-office-only"
  priority = 5

  action {
    block {}
  }

  statement {
    and_statement {
      statement {
        byte_match_statement {
          search_string         = "/admin"
          positional_constraint = "STARTS_WITH"

          field_to_match {
            uri_path {}
          }

          text_transformation {
            priority = 0
            type     = "LOWERCASE"
          }
        }
      }

      statement {
        not_statement {
          statement {
            ip_set_reference_statement {
              arn = aws_wafv2_ip_set.office.arn
            }
          }
        }
      }
    }
  }

  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "app-admin-office-only"
    sampled_requests_enabled   = true
  }
}

This is deliberately written as “block if not an office IP” rather than “allow if an office IP.” The reason is evaluation order: WAF evaluates rules by priority from lowest to highest, and stops as soon as it hits Allow or Block. Nothing after that gets checked. A high-priority Allow rule would let office IP requests skip every managed rule, effectively turning off protection for those sources. AWS: Rule actions and evaluation

Example 2: CRS blocks file uploads

This is probably the most common false positive after adding the Core Rule Set (CRS). The SizeRestrictions_BODY rule in CRS checks whether the request body is larger than 8 KB, so APIs that accept file uploads or long forms trip it easily.

Turning off the whole CRS is a waste. A better approach is to use labels: when a managed rule matches, it adds a label to the request, even if that rule is set to Count. So you can:

  1. Use rule_action_override in CRS to set only SizeRestrictions_BODY to Count.
  2. Add a custom rule after CRS that blocks requests carrying this label, unless the path is the upload API.
1
2
3
4
5
6
7
8
# Add this inside the managed_rule_group_statement for AWSManagedRulesCommonRuleSet
rule_action_override {
  name = "SizeRestrictions_BODY"

  action_to_use {
    count {}
  }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
rule {
  name     = "block-large-body-except-upload"
  priority = 20

  action {
    block {}
  }

  statement {
    and_statement {
      statement {
        label_match_statement {
          scope = "LABEL"
          key   = "awswaf:managed:aws:core-rule-set:SizeRestrictions_Body"
        }
      }

      statement {
        not_statement {
          statement {
            byte_match_statement {
              search_string         = "/api/upload"
              positional_constraint = "STARTS_WITH"

              field_to_match {
                uri_path {}
              }

              text_transformation {
                priority = 0
                type     = "LOWERCASE"
              }
            }
          }
        }
      }
    }
  }

  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "app-block-large-body"
    sampled_requests_enabled   = true
  }
}

Every other path still has the body size limit; only the upload API gets an exception. The label rule’s priority must be higher than CRS’s (meaning it runs later). Otherwise the label hasn’t been added yet when the rule is evaluated, and it will never match. AWS: Label match rule statement, AWS: CRS rules and labels

WAF logs: CloudWatch Logs or S3?

Sampled requests in the console only cover the last 3 hours, which isn’t enough to judge false positives. If you’re serious about tuning rules, you need WAF logging. AWS: Viewing a sample of web requests

WAF logs can go to three kinds of destinations, and a web ACL can only use one at a time:

DestinationGood forHow to query
CloudWatch LogsActively tuning rules and needing answers quicklyLogs Insights
S3Long-term retention and audit recordsAthena
Data FirehoseForwarding to a SIEM or another analytics platformDepends on the destination

Whichever you choose, the name must start with aws-waf-logs-, for example a log group called aws-waf-logs-app. This is easy to forget, and the configuration fails if the name is wrong. AWS: Logging destinations

My habit is to send logs to CloudWatch Logs during the weeks I’m tuning rules, since it’s the fastest to query. Once the rules are stable and I need long-term retention for audits, I switch to S3, which is cheaper to store. On cost: WAF logs sent to CloudWatch Logs are billed as vended logs, which adds up over a month on a high-traffic site. Check CloudWatch pricing before turning it on in production.

Logging only what you need

On a busy site, a logging filter can keep only blocked and counted requests and drop everything that was allowed normally. That cuts log volume a lot:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
resource "aws_cloudwatch_log_group" "waf" {
  name              = "aws-waf-logs-app"
  retention_in_days = 30
}

resource "aws_wafv2_web_acl_logging_configuration" "app" {
  resource_arn            = aws_wafv2_web_acl.app.arn
  log_destination_configs = [aws_cloudwatch_log_group.waf.arn]

  redacted_fields {
    single_header {
      name = "authorization"
    }
  }

  logging_filter {
    default_behavior = "DROP"

    filter {
      behavior    = "KEEP"
      requirement = "MEETS_ANY"

      condition {
        action_condition {
          action = "BLOCK"
        }
      }

      condition {
        action_condition {
          action = "COUNT"
        }
      }
    }
  }
}

redacted_fields masks the Authorization header in the logs so tokens don’t end up there. If you keep session data in cookies, you can redact those too. Terraform: Logging configuration, AWS: Redacting and filtering logs

Warning: There’s a trap here. When a managed rule group is set to Count as a whole with override_action { count {} }, the request’s final action is the web ACL’s default action, ALLOW, not COUNT, so the filter above drops it. If your observation period is mainly about managed rule groups set to Count, skip the logging filter for now, or keep those requests with a label condition (label_name_condition) instead.

Finding false positives with Logs Insights

Once logs are coming in, open CloudWatch Logs Insights, select aws-waf-logs-app, and start by checking which rules block the most and on which paths:

1
2
3
4
5
fields @timestamp, terminatingRuleId, httpRequest.clientIp, httpRequest.uri
| filter action = "BLOCK"
| stats count(*) as requests by terminatingRuleId, httpRequest.uri
| sort requests desc
| limit 20

During the observation period, what you want to know is what CRS would block while it’s in Count mode. You can find that by label:

1
2
3
4
5
fields @timestamp, httpRequest.clientIp, httpRequest.uri
| filter @message like /awswaf:managed:aws:core-rule-set/
| stats count(*) as hits, count_distinct(httpRequest.clientIp) as ips by httpRequest.uri
| sort hits desc
| limit 20

This is how I read the results: many hits from widely spread IPs on a path that’s one of your own normal features is very likely a false positive. Hits concentrated on a handful of IPs, on paths like /wp-login.php or /.env that don’t even exist on your site, are scanners, and blocking them is fine. AWS: Log fields, AWS: Logs Insights query syntax

If you send logs to S3

Logs are stored under AWSLogs/{account ID}/WAFLogs/{Region}/{web ACL name}/YYYY/MM/dd/HH/mm and written roughly every 5 minutes. To query them, create a table in Athena and use SQL. AWS provides a ready-made table definition. I’d recommend using partition projection with it; otherwise, once the data grows, every query scans the whole bucket. AWS: Sending WAF logs to S3, Athena: Querying AWS WAF logs

Moving from Count to Block: my checklist

Everything above leads to this step. My process goes roughly like this:

1. Make the observation period cover at least one full business cycle. For a typical site, that’s about two weeks. But if you have features that only run at certain times, like month-end billing or quarterly reports, wait until they’ve run at least once. False positives tend to show up in features nobody uses day to day.

2. Reach a conclusion for every rule that matched. Use the queries from the previous section to list every rule and path that matched in Count mode, and decide for each whether it’s an attack or a false positive. There are three ways to handle false positives, from best to worst:

ApproachWhen to use it
Label + custom rule as an exceptionOnly specific paths are false positives; other paths still need the rule
Add a scope-down statement to the rule groupThe whole rule group doesn’t fit certain paths, such as a webhook receiver
Keep the rule in Count permanently with rule_action_overrideThe rule is meaningless for your site

3. Switch one rule group at a time. Start with the group that has the fewest false positives, such as Known Bad Inputs, and leave CRS, which has many rules and a higher chance of false positives, for last. If you switch everything at once and something breaks, it’s hard to tell which group caused it.

4. Pick an off-peak window and have a rollback ready. This is where managing WAF with Terraform pays off: rolling back means changing override_action from none {} back to count {} and running apply. Before switching, make sure that PR can be merged right away.

5. Watch the metrics after switching. WAF’s CloudWatch metrics are in the AWS/WAFV2 namespace. Look for sudden increases in BlockedRequests, and keep an eye on the ALB’s HTTPCode_ELB_4XX_Count as well as support tickets and user reports. You can also set an alarm on BlockedRequests to notify you when the numbers look unusual. AWS: WAF CloudWatch metrics

Tip: AWS updates managed rule groups from time to time. If you’re worried that an update could introduce new false positives, use version in managed_rule_group_statement to pin a static version, and upgrade after testing. The trade-off is that new protections also won’t take effect until you upgrade manually. AWS: Managed rule group versioning

FAQ

What should the rate limit be?

There’s no standard answer. Run it in Count for a while, use the logs to find the highest number of requests a normal user sends within the window, and leave some headroom above that. Be especially generous when many users share an egress IP, such as offices, schools, and mobile networks.

How long does it take for an IP blocked by a rate-based rule to be unblocked?

You don’t have to unblock it manually. WAF keeps counting, and once that source’s requests within the window drop below the limit, the action no longer applies.

Do requests blocked by WAF show up in the backend access logs?

No. WAF handles the request before it reaches your backend application. If the web ACL is associated with an ALB, the ALB access logs will have a record, but to see the full reason (which rule matched), you still need the WAF logs.

Does turning on WAF logging affect site performance?

No. WAF writes logs asynchronously, so it doesn’t add latency to requests. The main concern is the cost of storing and querying the logs.

Conclusion

Writing WAF rules isn’t the hard part. The hard part is making sure they don’t block your own users. That’s why most of this article is about logs and the switching process rather than rule syntax. Turn on logging, collect data in Count mode, and decide what to block based on that data. It’s a lot less painful than blocking everything from day one and fixing things as users complain.

References

AWS official documentation

Terraform AWS Provider official documentation

This post is licensed under CC BY 4.0 by the author.