# The Ultimate Guide to Monitoring and Logging in DevOps: Concepts, Types, and Best Practices

Key Takeaways

  • Monitoring in DevOps tracks system health in real time (metrics, uptime, latency); logging records discrete events for later analysis and debugging.
  • The "three pillars of observability" are metrics, logs, and traces — each answers a different question about system behavior.
  • AWS-native tooling (CloudWatch, X-Ray, OpenSearch) covers all three pillars without third-party licensing.
  • Alerting should be tied to actionable thresholds, not noise — alert fatigue is the most common monitoring failure mode.
  • Centralized, structured logging (JSON logs shipped to a single store) is non-negotiable once you run more than a handful of services.

Why Monitoring and Logging Matter in DevOps

devops system monitoring exists to answer one question continuously: "is the system behaving as expected?" Logging answers a related but distinct question after the fact: "what exactly happened, and in what order?" Together, they form the feedback loop that lets teams detect, diagnose, and fix issues before — or quickly after — they affect users.

Without proper monitoring in devops, teams operate blind: outages are discovered by customer complaints rather than alerts, and root-cause analysis becomes guesswork.

The Three Pillars of Observability

Metrics

Numeric measurements sampled over time — CPU utilization, request latency, error rate, queue depth. Metrics are cheap to store and great for dashboards and alerting thresholds.

Logs

Discrete, timestamped records of events — an HTTP request, an exception stack trace, an authentication attempt. Logs provide the detailed "what happened" context metrics can't.

Traces

A trace follows a single request across multiple services, showing exactly where time was spent — essential for debugging latency in microservice architectures.

Setting Up Monitoring on AWS

CloudWatch Metrics and Alarms

aws cloudwatch put-metric-alarm \
  --alarm-name high-error-rate \
  --namespace AWS/ApplicationELB \
  --metric-name HTTPCode_Target_5XX_Count \
  --dimensions Name=LoadBalancer,Value=app/f9-prod-alb/50dc6c495c0c9188 \
  --statistic Sum \
  --period 60 \
  --threshold 10 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 2 \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:oncall-alerts

Custom Application Metrics (Python)

import boto3
import time

cloudwatch = boto3.client("cloudwatch")

def record_latency(duration_ms, endpoint):
    cloudwatch.put_metric_data(
        Namespace="F9App/API",
        MetricData=[
            {
                "MetricName": "RequestLatency",
                "Dimensions": [{"Name": "Endpoint", "Value": endpoint}],
                "Value": duration_ms,
                "Unit": "Milliseconds",
                "Timestamp": time.time(),
            }
        ],
    )

Centralized Logging

Shipping Logs to CloudWatch Logs

aws logs create-log-group --log-group-name /f9/app/orders-service
aws logs put-retention-policy --log-group-name /f9/app/orders-service --retention-in-days 30

Structured JSON Logging Pattern

import json
import logging

logger = logging.getLogger("orders-service")

def log_event(level, message, **context):
    logger.log(
        level,
        json.dumps({
            "message": message,
            "service": "orders-service",
            **context,
        }),
    )

log_event(logging.INFO, "order_created", order_id="4821", amount=129.99)

Structured (JSON) logs allow downstream tools like CloudWatch Logs Insights, OpenSearch, or Athena to query fields directly instead of regex-parsing free-text lines.

Querying with CloudWatch Logs Insights

fields @timestamp, order_id, amount
| filter message = "order_created"
| stats sum(amount) as total_revenue by bin(1h)

Observability Architecture

<svg viewBox="0 0 760 280" xmlns="http://www.w3.org/2000/svg" font-family="Arial, sans-serif" font-size="13" fill="currentColor">
  <defs>
    <marker id="a5" markerWidth="10" markerHeight="7" refX="9" refY="3.5" orient="auto">
      <polygon points="0 0, 10 3.5, 0 7" fill="currentColor"/>
    </marker>
  </defs>
  <style>
    .box { fill: none; stroke: currentColor; stroke-width: 1.6; }
    .arrow { stroke: currentColor; stroke-width: 1.6; fill: none; marker-end: url(#a5); }
  </style>

  <rect class="box" x="30" y="30" width="140" height="50" rx="8"/>
  <text x="100" y="60" text-anchor="middle">Service A</text>
  <rect class="box" x="30" y="100" width="140" height="50" rx="8"/>
  <text x="100" y="130" text-anchor="middle">Service B</text>
  <rect class="box" x="30" y="170" width="140" height="50" rx="8"/>
  <text x="100" y="200" text-anchor="middle">Service C</text>

  <line class="arrow" x1="170" y1="55" x2="300" y2="110"/>
  <line class="arrow" x1="170" y1="125" x2="300" y2="125"/>
  <line class="arrow" x1="170" y1="195" x2="300" y2="140"/>

  <rect class="box" x="300" y="80" width="150" height="90" rx="8"/>
  <text x="375" y="115" text-anchor="middle">CloudWatch</text>
  <text x="375" y="135" text-anchor="middle" font-size="11">Metrics + Logs</text>
  <text x="375" y="153" text-anchor="middle" font-size="11">+ X-Ray Traces</text>

  <line class="arrow" x1="450" y1="125" x2="560" y2="125"/>

  <rect class="box" x="560" y="90" width="170" height="70" rx="8"/>
  <text x="645" y="118" text-anchor="middle">Dashboards +</text>
  <text x="645" y="136" text-anchor="middle">Alarms (SNS)</text>
</svg>

Distributed Tracing with X-Ray

aws xray put-trace-segments --trace-segment-documents '[{
  "trace_id": "1-5f5c1234-abcd1234ef5678901234abcd",
  "id": "1234567890abcdef",
  "name": "orders-service",
  "start_time": 1700000000.0,
  "end_time": 1700000000.5
}]'

Most teams instrument tracing with the AWS X-Ray SDK rather than calling the API directly — the SDK auto-instruments AWS SDK calls, HTTP clients, and database queries.

Best Practices for Monitoring in DevOps

  1. Alert on symptoms, not causes — alert when error rate or latency breaches a threshold, not on every individual exception.
  2. Set SLO-based alarms — tie alarms to service-level objectives (e.g., "99.9% of requests under 300ms") rather than arbitrary thresholds.
  3. Retain logs with a clear policy — 30 days hot, archived to S3 Glacier for compliance beyond that, to control cost.
  4. Dashboards for humans, alarms for pages — reserve PagerDuty/SNS-triggered alarms for things that need immediate action; dashboards are for investigation.
  5. Correlate metrics, logs, and traces with a shared request ID — pass a trace/correlation ID through every service hop so you can jump from a metric spike straight to the relevant logs.

FAQ

What's the difference between monitoring and observability? Monitoring tells you that something is wrong (via predefined dashboards and alarms); observability gives you the tools to ask new questions about system behavior you didn't anticipate — typically via rich, queryable metrics, logs, and traces.

Do I need a third-party tool like Datadog for devops system monitoring on AWS? Not necessarily. CloudWatch, X-Ray, and OpenSearch together cover all three observability pillars natively; third-party tools add convenience (unified UI, advanced APM features) at additional cost.

How long should I retain logs? It depends on compliance requirements, but a common pattern is 30-90 days in a queryable hot store, then archived to cold storage (S3 Glacier) for 1-7 years depending on industry regulations.

What's "alert fatigue" and how do I avoid it? Alert fatigue happens when teams receive so many low-value alerts that they start ignoring all of them, including critical ones. Avoid it by tuning thresholds to real incidents, using alarm severity tiers, and routing non-urgent alerts to dashboards instead of pages.

Need Help Building Observability Into Your AWS Stack?

f9infotech designs end-to-end monitoring and logging architectures on AWS — CloudWatch dashboards, X-Ray tracing, and centralized log pipelines tuned to reduce noise and surface real incidents. Talk to our DevOps team about building observability into your infrastructure.