AWS CloudWatch
Last updated
Last updated
Provides metrics for every services in AWS.
Metric is a variable to monitor (CPU Utilization, Networking etc).
Dimension is an attribute of metric (instanceid, environment, etc).
Upto 30 dimensions per metric.
Metrics have timestamps.
Can create CloudWatch dashboard of metrics.
EC2 instance have metrics every 5 minutes.
With detailed monitoring (for a cost) every 1 minute.
Use detailed monitoring if you want to scale faster for your ASG.
Note : EC2 Memory usage is by default not pushed (must be pushed from inside the instance as a custom metric).
Allows to define your own custom metrics to CloudWatch.
Ability to use dimensions to segment metrics.
instance.id
Environment.name
Metric resolution (StorageResolution API) parameter.
Standard : 1 minute
High Resolution : 1/5/10/30 seconds - Higher cost
Accepts metrics data points two weeks in the past and two hours in the future.
The CloudWatch Log group consists of,
Log groups: Arbitrary name, usually represents an application.
Log stream: Instances within application/log-files/ containers.
Log expiration policy: Never expire, 1 day-10 years.
CloudWatch Logs can be send to,
Amazon S3 - Export, takes upto 12 hours to be completed.
Kinesis Data Streams
Kinesis Data Firehose
AWS Lambda
OpenSearch
Logs are encrypted by default.
Can setup KMS-based encryption with your own keys.
CloudWatch logs can be send using
SDK
CloudWatch Logs Agent
CloudWatch Unified Agent
Elastic Beanstalk, collection of logs from application.
Containers, collection of logs from containers.
Lambda, collection from function logs.
VPC Flow logs
API Gateway
Route53 to Log DNS queries
Search and analyze log data stored in CloudWatch logs.
Provides a purpose-built query language, which automatically discovers fields from AWS services and JSON log events.
Can save queries and add them to CloudWatch Dashboards.
Fetch desired event filters, filter based on conditions, calculate aggregate statistics, sort events, limit number of event.
Can query multiple Log Groups in different AWS accounts.
Its a query language, not a real-time engine.
Get a real-time log events from CloudWatch Logs for processing and analysis.
Send to Kinesis Data Streams, Kinesis Data Firehose or Lambda.
Subscription Filter allows to filter which logs are delivered to your destination.
Destination allows Cross Account Subscription, which allows to send log events to resources in a different AWS account, appropiate destination access policy and IAM Role for cross-account must be configured to make this work.
It is possible to aggregate cloud watch logs from different account and regions to one common destination such as Kinesis Data Streams and sent to Kinesis Data Firehose from where in near real time can send logs to S3.
To push logs of EC2 to CloudWatch, run the CloudWatch agent at EC2 instance to push the logs.
Make sure relevnat IAM permissions are correct.
CloudWatch log agent can be set up on-premises too.
CloudWatch Logs Agent, which is an older agent. It can only send logs to CloudWatch Logs.
CloudWatch Unified Agent, is a newer version.
It will collection additional details and metrics like RAM, process details etc.
Collect logs to send to CloudWatch Logs. You can also configure SSM Parameter Store.
Collect
Collect CPU metrics, Disk metrics, RAM metrics, Netstat, Process deails in an EC2 instance, swap space etc.
CloudWatch Logs can use Filter Metrics expressions.
Logs can be encrypted at log group level using KMS.
The encryption can be enabled while creating one or after it is created.
Cannot associate CMK with a log group using CloudWatch console.
Make sure the KMS policy is allowed to make association with log group to work.
Filters do not retroactively filter data. It will only publish metric data points for events that happen after filter was created.
