Key Takeaways

  • CloudWatch subscription filters push matching log events from a log group to a destination — Lambda, Kinesis Data Streams, or Firehose — in near real time, without polling.
  • A subscription filter is defined by a filter pattern (what to match) and a destination ARN (where to send it).
  • Lambda must grant CloudWatch Logs permission to invoke it, and the Lambda execution role needs no special CloudWatch Logs permission for this direction — the push comes from CloudWatch Logs to Lambda.
  • Payloads arrive gzip-compressed and base64-encoded; your function must decode and inflate before parsing.
  • Each log group supports up to 2 subscription filters by default (soft limit, raisable via Service Quotas).

Streaming logs into Lambda for alerting, enrichment, or forwarding to a SIEM is one of the most common AWS observability patterns — but the plumbing (permissions, payload decoding, filter syntax) trips up a lot of teams on the first attempt. This guide walks through exactly how it fits together.

What Is a CloudWatch Subscription Filter?

A subscription filter is attached to a CloudWatch Logs log group. Whenever a new log event is ingested into that group and it matches the filter's pattern, CloudWatch Logs pushes it to a configured destination. This is push-based — no cron job, no polling Lambda, and latency is typically sub-second to a few seconds.

Common destinations:

  • AWS Lambda — transform, alert, or forward log data
  • Kinesis Data Streams — fan-out to multiple consumers, replay capability
  • Kinesis Data Firehose — batch delivery to S3, OpenSearch, or a third-party HTTP endpoint

This post focuses on the Lambda destination, since it's the most common for custom alerting and log-shipping pipelines.

Architecture Overview

Application / Service Logs CloudWatch Log Group + Subscription Filter (filter pattern) AWS Lambda (decode, process, alert / forward) gzip + base64

Step 1: Create the Lambda Function

A minimal Node.js handler that decodes and logs the matched events:

const zlib = require("zlib");

exports.handler = async (event) => {
  const payload = Buffer.from(event.awslogs.data, "base64");
  const decompressed = zlib.gunzipSync(payload).toString("utf-8");
  const logData = JSON.parse(decompressed);

  console.log(`Log group: ${logData.logGroup}`);
  console.log(`Log stream: ${logData.logStream}`);

  for (const logEvent of logData.logEvents) {
    console.log(`[${new Date(logEvent.timestamp).toISOString()}] ${logEvent.message}`);
    // Add alerting, enrichment, or forwarding logic here
  }

  return { statusCode: 200 };
};

Step 2: Grant CloudWatch Logs Permission to Invoke Lambda

CloudWatch Logs needs explicit permission to invoke your function. This is a resource-based policy on the Lambda, not an IAM role change:

aws lambda add-permission \
  --function-name log-processor \
  --statement-id CloudWatchLogsInvoke \
  --action lambda:InvokeFunction \
  --principal logs.amazonaws.com \
  --source-arn "arn:aws:logs:us-east-1:111122223333:log-group:/aws/myapp/prod:*"

Scope --source-arn to the specific log group ARN — not a wildcard account-wide ARN — to avoid any other account's log group invoking your function.

Step 3: Create the Subscription Filter

aws logs put-subscription-filter \
  --log-group-name "/aws/myapp/prod" \
  --filter-name "ErrorsToLambda" \
  --filter-pattern "ERROR" \
  --destination-arn "arn:aws:lambda:us-east-1:111122223333:function:log-processor"

Filter Pattern Examples

"ERROR"                              # substring match, any case-sensitive line containing ERROR
?ERROR ?WARN ?CRITICAL               # match any of these terms
{ $.statusCode = 500 }               # JSON log field match
[ip, user, timestamp, request, status_code=5*, size]   # space-delimited log match on status code

JSON-based patterns ({ $.field = value }) require your log entries to be valid JSON lines — ideal if your application already emits structured logs.

Terraform Equivalent

For teams managing infrastructure as code, here's the same setup in Terraform:

resource "aws_lambda_permission" "cloudwatch_invoke" {
  statement_id  = "CloudWatchLogsInvoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.log_processor.function_name
  principal     = "logs.amazonaws.com"
  source_arn    = "${aws_cloudwatch_log_group.app_logs.arn}:*"
}

resource "aws_cloudwatch_log_subscription_filter" "errors_to_lambda" {
  name            = "ErrorsToLambda"
  log_group_name  = aws_cloudwatch_log_group.app_logs.name
  filter_pattern  = "ERROR"
  destination_arn = aws_lambda_function.log_processor.arn
  depends_on      = [aws_lambda_permission.cloudwatch_invoke]
}

Common Pitfalls

  • Forgetting add-permission — the subscription filter will create successfully, but Lambda invocations silently fail with an access-denied error buried in CloudWatch metrics, not your function logs.
  • Not decoding the payload — the event body is gzip + base64, not plain JSON. Skipping decompression throws a parse error on the first invocation.
  • Throttling at scale — a high-volume log group can invoke Lambda far more often than expected. Consider routing through Kinesis Data Streams first if volume exceeds a few hundred events/second, to buffer and batch.
  • Cross-account forwarding — requires a resource policy on the destination Lambda and a correctly scoped source-arn, plus the log group's account ID allowed if going cross-account.

FAQ

Q: How many subscription filters can one log group have? A: Two per log group by default. This is a soft limit — request an increase via Service Quotas if you need to fan out to multiple destinations from one group.

Q: Does this work with CloudWatch Logs Insights queries? A: No — subscription filters operate on ingestion in real time. Logs Insights is for ad-hoc querying of already-stored log data. They're complementary, not substitutes.

Q: Can I use a subscription filter with Kinesis Firehose instead of Lambda? A: Yes — set --destination-arn to a Firehose delivery stream ARN instead. This is the standard pattern for shipping logs to S3 or an external SIEM without writing custom Lambda code.

Q: Why is my Lambda not being invoked even though the subscription filter shows as active? A: Check three things in order: the filter pattern actually matches your log format, the lambda:InvokeFunction permission exists with the correct source-arn, and the log events are landing in the exact log group (not a differently named one) the filter is attached to.

Get Help Building Your Observability Pipeline

Setting up resilient, cost-aware log pipelines across dozens of log groups and accounts is exactly the kind of AWS plumbing that benefits from an experienced partner. F9infotech's AWS managed services team can design and implement your end-to-end logging, alerting, and SIEM-forwarding architecture — talk to our AWS consultants to get started.