Monitoring of AWS Infrastructure is one of the core part of DevOps job, AWS by default provides metrics of EC2 Status Check, CPU Utilisation, Network I/O, etc.

But it doesn’t provide metrics and insights about your disk usage by default.

Nevertheless, AWS do have ways to look into disk usage of your EC2 instance, but for that you need to install additional AWS Cloudwatch package and send metrics to AWS for monitoring.

In this article, we will see how to install custom AWS Cloudwatch Agent on your Ubuntu EC2 instance, how to setup Disk usage monitoring, and how to receive notification when disk utilisation is beyond 80%

1. Download and Install Cloudwatch Agent

Run following commands on your ubuntu instance

sudo apt update
cd /tmp
wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
dpkg -i amazon-cloudwatch-agent.deb

2. Create Disk Utilization Monitoring Script

• Use below-mentioned script to send metrics about disk utilization to AWS Cloudwatch metrics
• Save file as /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

{
  "agent": {
    "metrics_collection_interval": 60,
    "run_as_user": "cwagent"
  },
  "metrics": {
    "append_dimensions": {
        "InstanceId": "${aws:InstanceId}"
    },
    "metrics_collected": {
      "disk": {
        "measurement": [
          "used_percent"
        ],
        "metrics_collection_interval": 60,
        "resources": [
          "/"
        ]
      }
    }
  }
}

Above script will collect disk usage data at 60 seconds interval and send it to AWS Cloudwatch. It will only monitor Root Disk usage as per above script, for other mounted volumes, please add it in script

3. Give AWS EC2 access to send monitoring data to AWS Cloudwatch

• We need to setup a role that has access to “CloudWatchAgentServerPolicy” to send data from Amazon EC2 instance to Cloudwatch
• You can either add AWS_ACCESS_KEY and AWS_SECRET_KEY to EC2 instance, but more recommended way is attaching an EC2 role that does this. Let’s see detailed steps of how to setup that

1. Login to AWS Console
2. Go to Services –> IAM
3. Select Roles –> Create Role
4. Select “Select type of trusted entity” –> “AWS service” –> “EC2”
5. Click “Next Permission”
6. Search for “CloudWatchAgentServerPolicy” and select it, click on “Next: Tags”
7. Add “Key, Value” for tags –> “Next: Review”
8. Now, Give your Role Name and Description
9. Create Role

Your role has been created, now let’s attach the role to our EC2 instance

1. Go to Services –> EC2
2. Go to instances –> Select particular instance
3. Click on “Actions” –> “Security” –> “Modify IAM Role”
4. Select “IAM Role” that we just created –> Click on “Save”

Now, our IAM role is attached to EC2 instance, let’s send metrics data from EC2 to Cloudwatch

4. Restart Cloudwatch agent

• Restart Cloudwatch agent now, so out new policy can get started

systemctl restart amazon-cloudwatch-agent
systemctl status amazon-cloudwatch-agent

If above commands are executed successfully, then Let’s wait for Cloudwatch metrics to appear

5. Look for Metrics

• Login to AWS Console
• Go to Services –> Cloudwatch
• Select Region same as EC2 instances’
• Inside Metrics, Select All Metrics
• Look for “CWAgent”, if it hasn’t appeared yet, wait for sometime and refresh page
• Select the option, Go inside and look for instance ID where you setup Monitoring
• Select the metrics and after sometime you will be able to see fetched data

Now, let’s setup alarm if Disk usage goes beyond 80%, we can be notified

6. Create SNS topic for Notification

We need to create SNS topic, which can notify us for the CloudWatch Alarm, let’s setup that

• Go to AWS Console
• Services –> SNS (Simple Notification Service)
• Go to Topics –> “Create Topic”
• Select “Standard”
• Give Name and display name for your topic
• Keep everything else as it is, and select “Create topic”
• Now, go to created topic, and select “Create Subscription”
• In protocol select, Email
  • For other type of protocol, select as per your need
• For endpoint, enter your email ID
• Keep everything as it is and click Create Subscription

For Email subscription, you will receive an email to confirm the subscription, click on link received in email to confirm subscription

7. Setup Alarm to send Notification when disk usage is beyond 80%

Let’s now setup monitoring on our newly setup metrics, so we can be automatically notified when something goes wrong

• Go to Services –> Cloudwatch
• Go to “All Alarms” –> “Create Alarm”
• Select Metric –> Go to CWAgent and select recently added metric
• Change Metric Name as per requirement
• In Statistic and period, select as per your need, or keep default “average” and “5 minutes”
• In condition, Select “Static” –> “Greater” –> “than” 80 or as per your need
• keep everything else as it is and click on “Next”
• In notification, Select “In Alarm”
• Select an SNS topic, select “Existing SNS topic”
• Send a notification to –> select SNS topic created in above step
• Keep everything else as it is and Click “Next”
• Enter “Alarm Name” and “Alarm Description”, and “Next”
• Review all the setup and click on “Create Alarm”

Your Disk utilization Alarm is all set now.

If you face issues in any of the steps, feel free to reach out to us in comment section below.

If this is useful to you, do share it in your network.

Drafted On,
22^nd January 2022
DevOps @identicalCloud.com

• Twitter
• Instagram
• YouTube
• Facebook

Read More: How to secure remote server using MFA

References: Collecting metrics and logs from Amazon EC2 instances and on-premises servers with the CloudWatch agent