Overview

This guide provides step-by-step instructions for installing and configuring AWS CloudWatch Agent on your Amazon Machine Image (AMI). The CloudWatch Agent enables you to collect system metrics and logs from your EC2 instances and send them to Amazon CloudWatch for monitoring and analysis. This setup is specifically designed for use with Packer as your AMI builder, allowing you to pre-configure CloudWatch during the image creation process. Following these steps will ensure your instances automatically stream logs to CloudWatch upon deployment.

Installation and Configuration Steps

Step 1: Install CloudWatch Agent on AMI

Start by making sure AWS CloudWatch is installed and configured on your AMI. This configuration is specific to Packer, but the underlying commands can be adapted for any software AMI builder. The installation process downloads the CloudWatch Agent package, installs it, and configures it to collect logs from /tmp/circleci.log.

"provisioners": [
  {
    "type": "shell",
    "inline": [
      "set -e",
      "export DEBIAN_FRONTEND=noninteractive",
      "echo \"Installing AWS CloudWatch Agent...\"",
      "wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb",
      "sudo dpkg -i amazon-cloudwatch-agent.deb",
      "sudo apt-get install -f -y",
      
      "echo \"Configuring CloudWatch Agent...\"",
      "sudo mkdir -p /opt/aws/amazon-cloudwatch-agent/etc/",
      "cat << 'EOT' | sudo tee /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json",
      "{",
      "  \"logs\": {",
      "    \"logs_collected\": {",
      "      \"files\": {",
      "        \"collect_list\": [",
      "          {",
      "            \"file_path\": \"/tmp/circleci.log\",",
      "            \"log_group_name\": \"{<log_place>}\",",
      "            \"log_stream_name\": \"{instance_id}\"",
      "          }",
      "        ]",
      "      }",
      "    }",
      "  }",
      "}",
      "EOT",
      
      "echo \"Starting CloudWatch Agent...\"",
      "sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json",
      "sudo systemctl enable amazon-cloudwatch-agent"
    ]
  }
]

Important: Replace <log_place> with your desired log group destination. Keep the name within the curly brackets {}.

Step 2: Create IAM Role

Create the necessary IAM role and instance profile using Terraform to provide CloudWatch permissions to your EC2 instances.

variable "role_name" {
    description = "The role name"
    default     = ""
    type        = string
}

variable "instance_profile_name" {
    description = "The instance profile name"
    default     = ""
    type        = string
}

resource "aws_iam_role" "cloudwatch_agent_role" {
    name               = var.role_name
    assume_role_policy = data.aws_iam_policy_document.assume_role_policy.json
    tags               = var.logan_tags
}

data "aws_iam_policy_document" "assume_role_policy" {
    statement {
        actions = ["sts:AssumeRole"]
        principals {
            type        = "Service"
            identifiers = ["ec2.amazonaws.com"]
        }
    }
}

resource "aws_iam_policy_attachment" "attach_cloudwatch_agent_policy" {
    name       = "CloudWatchAgentPolicyAttachment"
    roles      = [aws_iam_role.cloudwatch_agent_role.name]
    policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
}

resource "aws_iam_instance_profile" "cloudwatch_instance_profile" {
    name = var.instance_profile_name
    role = aws_iam_role.cloudwatch_agent_role.name
}

output "instance_profile_arn" {
    value = aws_iam_instance_profile.cloudwatch_instance_profile.arn
}

output "role_arn" {
    value = aws_iam_role.cloudwatch_agent_role.arn
}

Step 3: Configure Machine Provisioner

Connect the IAM Instance role with the correct permissions to your machine provisioner using the assumedRoleArn flag in your server's values.yaml. This configuration applies the role to every machine created by the machine provisioner and enables log streaming to CloudWatch.

machine_provisioner:
    ...
    providers:
        ec2:
            ...
            assumedRoleArn: ""

Step 4: Deploy Configuration

Deploy both the new AMI for your server installation and the changes made to the values.yaml. After deployment, you should automatically start seeing logs from the Machine Provisioner binary inside of CloudWatch.

Additional Notes

Ensure that your AWS account has the necessary permissions to create CloudWatch log groups and streams. The CloudWatch Agent will automatically create the specified log group if it doesn't exist. Monitor your CloudWatch usage as log ingestion and storage may incur additional AWS charges.

Additional Resources

• AWS CloudWatch Agent Installation Guide - Official AWS installation documentation
• CloudWatch Agent Configuration Reference - Detailed configuration options