AWS CloudWatch

Icon

CloudWatch Icon

About

• Provides metrics for every services in AWS.

• Metric is a variable to monitor (CPU Utilization, Networking etc).

• Dimension is an attribute of metric (instanceid, environment, etc).

• Upto 30 dimensions per metric.

• Metrics have timestamps.

• Can create CloudWatch dashboard of metrics.

EC2 Detailed Monitoring

• EC2 instance have metrics every 5 minutes.

• With detailed monitoring (for a cost) every 1 minute.

• Use detailed monitoring if you want to scale faster for your ASG.

• Note : EC2 Memory usage is by default not pushed (must be pushed from inside the instance as a custom metric).

Customer Metrics

• Allows to define your own custom metrics to CloudWatch.

• Ability to use dimensions to segment metrics.

  • instance.id

  • Environment.name

• Metric resolution (StorageResolution API) parameter.

  • Standard : 1 minute

  • High Resolution : 1/5/10/30 seconds - Higher cost

• Accepts metrics data points two weeks in the past and two hours in the future.

CloudWatch Logs

• The CloudWatch Log group consists of,

  • Log groups: Arbitrary name, usually represents an application.

  • Log stream: Instances within application/log-files/ containers.

  • Log expiration policy: Never expire, 1 day-10 years.

  • CloudWatch Logs can be send to,

    • Amazon S3 - Export, takes upto 12 hours to be completed.

    • Kinesis Data Streams

    • Kinesis Data Firehose

    • AWS Lambda

    • OpenSearch

• Logs are encrypted by default.

• Can setup KMS-based encryption with your own keys.

• CloudWatch logs can be send using

  • SDK

  • CloudWatch Logs Agent

  • CloudWatch Unified Agent

  • Elastic Beanstalk, collection of logs from application.

  • Containers, collection of logs from containers.

  • Lambda, collection from function logs.

  • VPC Flow logs

  • API Gateway

  • Route53 to Log DNS queries

CloudWatch Logs Insights

• Search and analyze log data stored in CloudWatch logs.

• Provides a purpose-built query language, which automatically discovers fields from AWS services and JSON log events.

• Can save queries and add them to CloudWatch Dashboards.

• Fetch desired event filters, filter based on conditions, calculate aggregate statistics, sort events, limit number of event.

• Can query multiple Log Groups in different AWS accounts.

• Its a query language, not a real-time engine.

CloudWatch Logs Subscriptions

• Get a real-time log events from CloudWatch Logs for processing and analysis.

• Send to Kinesis Data Streams, Kinesis Data Firehose or Lambda.

• Subscription Filter allows to filter which logs are delivered to your destination.

• Destination allows Cross Account Subscription, which allows to send log events to resources in a different AWS account, appropiate destination access policy and IAM Role for cross-account must be configured to make this work.

CloudWatch Logs Aggregation

• It is possible to aggregate cloud watch logs from different account and regions to one common destination such as Kinesis Data Streams and sent to Kinesis Data Firehose from where in near real time can send logs to S3.

CloudWatch Logs for EC2

• To push logs of EC2 to CloudWatch, run the CloudWatch agent at EC2 instance to push the logs.

• Make sure relevnat IAM permissions are correct.

• CloudWatch log agent can be set up on-premises too.

• CloudWatch Logs Agent, which is an older agent. It can only send logs to CloudWatch Logs.

• CloudWatch Unified Agent, is a newer version.

  • It will collection additional details and metrics like RAM, process details etc.

  • Collect logs to send to CloudWatch Logs. You can also configure SSM Parameter Store.

  • Collect

  • Collect CPU metrics, Disk metrics, RAM metrics, Netstat, Process deails in an EC2 instance, swap space etc.

CloudWatch Logs Metric Filter

• CloudWatch Logs can use Filter Metrics expressions.

• Filters do not retroactively filter data. It will only publish metric data points for events that happen after filter was created.

Log Encryption

• Logs can be encrypted at log group level using KMS.

• The encryption can be enabled while creating one or after it is created.

• Cannot associate CMK with a log group using CloudWatch console.

APIS

• Make sure the KMS policy is allowed to make association with log group to work.

      # Create a log group, with encryption enabled
  
      aws logs create-log-group --log-group-name <log-group-name> --kms-key-id <arn-of-kms-key> --region <aws-region>
  
      # Associate KMS id with already created log group
  
      aws logs associate-kms-key --log-group-name <log-group-name> --kms-key-id <arn-of-kms-key> --region <aws-region>