By Lawrence Dudley •

AWS Application Load Balancer (ALB) Logstash Parsing/Pattern

Parsing ALB logs (OK, any logs really) with Logstash can be a pain.

ALB logs are a different format to the ones you get from ELB (yay!) so you need a different matching pattern.

To save you the pain, we’ve included ours here for you to copy/paste. We’re not parsing domain_name or chosen_cert_name (see docs) as these aren’t useful to us, but these would be easy to add with another %{NOTSPACE} at the end of the first grok.

For some completely unknown reason, the logs for internal ALBs seem to follow a slightly different format so be aware that the below Logstash config will only work for public load balancers.


filter {
  if [type] == "application-load-balancer" {
    grok {
        match => ["message", "%{NOTSPACE:request_type} %{TIMESTAMP_ISO8601:log_timestamp} %{NOTSPACE:alb-name} %{NOTSPACE:client} %{NOTSPACE:target} %{NOTSPACE:request_processing_time:float} %{NOTSPACE:target_processing_time:float} %{NOTSPACE:response_processing_time:float} %{NOTSPACE:elb_status_code} %{NOTSPACE:target_status_code} %{NOTSPACE:received_bytes:float} %{NOTSPACE:sent_bytes:float} %{QUOTEDSTRING:request} %{QUOTEDSTRING:user_agent} %{NOTSPACE:ssl_cipher} %{NOTSPACE:ssl_protocol} %{NOTSPACE:target_group_arn} %{QUOTEDSTRING:trace_id}"]
    }
    date {
      match  => [ "log_timestamp", ISO8601 ]
    }
    mutate {
      gsub => [
        "request", '"', "",
        "trace_id", '"', "",
        "user_agent", '"', ""
      ]
    }
    grok {
        match => ["request", "(%{NOTSPACE:http_method})? (%{NOTSPACE:http_uri})? (%{NOTSPACE:http_version})?"]
    }
    grok {
        match => ["http_uri", "(%{WORD:protocol})?(://)?(%{IPORHOST:domain})?(:)?(%{INT:http_port})?(%{GREEDYDATA:request_uri})?"]
    }
    grok {
        match => ["client", "(%{IPORHOST:c_ip})?"]
    }
    geoip {
        source => "c_ip"
    }
  }
}

I’m not sure how many people are actively using the ELK stack for monitoring on AWS – let us know in the comments if you’d like us to share some more of our patterns.