Collectord

Docker Centralized Logging with AWS CloudWatch Logs

Installation

Pre-requirements

Collectord automatically forwards containers logs, host logs and events in the LogGroup and LogStreams with the following format:

  • Container logs:
    • LogGroup: /docker/{{cluster}}/container_logs/{{container_image::replace(:,/)}}/
    • LogStream: /{{host}}/{{container_name}}/{{container_id}}/{{stream}}
  • Application logs:
    • LogGroup: /docker/{{cluster}}/container_logs/{{container_image::replace(:,/)}}/
    • LogStream: /{{host}}/{{container_name}}/{{container_id}}/{{volume_name}}/{{file_path}}
  • Host Logs:
    • LogGroup: /docker/{{cluster}}/host_logs/
    • LogStream: /{{host}}/{{file_path}}
  • Events:
    • LogGroup: /docker/{{cluster}}/events/
    • LogStream: /{{host}}/

You need to create a User with Programmatic access that has permissions to create new LogGroups, LogStreams and PutEvents.

Create IAM User with Programmatic access

Collectord uses IAM user for authentication purposes to push data to CloudWatch Logs. You can create IAM user with CLI or Web Console by following the guidance https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html. In the example below we restrict Collectord to write only to the log groups, and allow to create new log groups.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:PutRetentionPolicy"
            ],
            "Resource": [
                "arn:aws:logs:*:*:log-group:/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "*"
        }
    ]
}

CLI Example

Create policy

aws iam create-policy --policy-name collectord-cloudwatch --policy-document "{\"Version\": \"2012-10-17\", \"Statement\": [{\"Effect\": \"Allow\", \"Action\": [\"logs:CreateLogStream\", \"logs:PutLogEvents\", \"logs:PutRetentionPolicy\"], \"Resource\": [\"arn:aws:logs:*:*:log-group:/*\" ] }, {\"Effect\": \"Allow\", \"Action\": \"logs:CreateLogGroup\", \"Resource\": \"*\"} ] }"

From the output note the Arn of the policy (like arn:aws:iam::999999999999:policy/collectord-cloudwatch)

Create user

aws iam create-user --user collectord-cloudwatch

Attach just created policy to the user

Replace the ARN with the Policy ARN from the 2 steps before

aws iam attach-user-policy --user collectord-cloudwatch --policy-arn arn:aws:iam::999999999999:policy/collectord-cloudwatch

Create Access Key and Secret for collectord-cloudwatch user

aws iam create-access-key --user collectord-cloudwatch

From the output note the AccessKeyId and SecretAccessKey

Docker Container Runtime

Collectord uses JSON-files generated by JSON logging driver as a source for container logs.

Some linux distributions, CentOS for example, by default enable journald logging driver instead of default JSON logging driver. You can verify which driver is used by default

$ docker info | grep "Logging Driver"
Logging Driver: json-file

If docker configuration file location is /etc/sysconfig/docker (common in CentOS/RHEL case with Docker 1.13), you can change it and restart docker daemon after that with following commands.

$ sed -i 's/--log-driver=journald/--log-driver=json-file --log-opt max-size=100M --log-opt max-file=3/' /etc/sysconfig/docker
$ systemctl restart docker

If you configure Docker daemon with daemon.json in /etc/docker/daemon.json (common in Debian/Ubuntu), you can change it and restart docker daemon.

{
  "log-driver": "json-file",
  "log-opts" : {
    "max-size" : "100m",
    "max-file" : "3"
  }
}
$ systemctl restart docker

Please follow the manual to learn how to configure default logging driver for containers:

Log Configuration for ECS tasks

When you configure tasks on ECS instances, make sure to always setup a logConfiguration as following (adjust max-size and max-file as appropriate for your service)

"logConfiguration": {
    "logDriver": "json-file",
    "options": {
        "max-size": "10m",
        "max-file": "3"
    }
}

You can find it when you configure tasks as JSON.

JSON logging driver configuration

With the default configuration, docker does not rotate JSON log files, with time they can become large and consume all disk space. That is why we specify max-size and max-file with the default configurations. See Configure and troubleshoot the Docker daemon for more details.

Install Collectord

Install on Docker

The following command will schedule Collectord on your Docker and start forwarding all the logs to the CloudWatch. Modify the highlighted lines. You need to specify AWS Region, access key id and secret access key (created in pre-requirements), review and accept license agreement and include the license key (request an evaluation license key with this automated form)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
docker run -d \
    --name collectord-cloudwatch \
    --volume /var/lib/docker/:/rootfs/var/lib/docker/:ro \
    --volume /var/run/docker.sock:/rootfs/var/run/docker.sock:ro \
    --volume /var/log:/rootfs/var/log:ro \
    --volume collectord-cloudwatch-data:/data/ \
    --cpus=1 \
    --cpu-shares=100 \
    --memory=512M \
    --restart=always \
    --env "COLLECTORDCONFIG_PATH=/config/cloudwatch/docker/" \
    --env "COLLECTORD__AWS_REGION=aws__region=..." \
    --env "COLLECTORD__AWS_ACCESS_KEY_ID=aws__accessKeyID=..." \
    --env "COLLECTORD__AWS_SECRET_ACCESS_KEY=aws__secretAccessKey=..." \
    --env "COLLECTORD__ACCEPT_LICENSE=general__acceptLicense=false" \
    --env "COLLECTORD__LICENSE_KEY=general__license=..." \
    --env "COLLECTORD__CLUSTER=general__fields.cluster=-" \ 
    --privileged \
    outcoldsolutions/collectord:6.0.301

For example (you can also name your cluster if you want to separate logs from different clusters)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
docker run -d \
    --name collectord-cloudwatch \
    --volume /var/lib/docker/:/rootfs/var/lib/docker/:ro \
    --volume /var/run/docker.sock:/rootfs/var/run/docker.sock:ro \
    --volume /var/log:/rootfs/var/log:ro \
    --volume collectord-cloudwatch-data:/data/ \
    --cpus=1 \
    --cpu-shares=100 \
    --memory=512M \
    --restart=always \
    --env "COLLECTORDCONFIG_PATH=/config/cloudwatch/docker/" \
    --env "COLLECTORD__AWS_REGION=aws__region=us-west-2" \
    --env "COLLECTORD__AWS_ACCESS_KEY_ID=aws__accessKeyID=AKIAI7SNNYYAV456WBVQ" \
    --env "COLLECTORD__AWS_SECRET_ACCESS_KEY=aws__secretAccessKey=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" \
    --env "COLLECTORD__ACCEPT_LICENSE=general__acceptLicense=true" \
    --env "COLLECTORD__LICENSE_KEY=general__license=Qkc1MTgzUTQ0SUUyTTowOjoz...." \
    --env "COLLECTORD__CLUSTER=general__fields.cluster=dev" \ 
    --privileged \
    outcoldsolutions/collectord:6.0.301

Docker-compose v2

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
version: "2.2"
services:

  collectord-cloudwatch:
    image: outcoldsolutions/collectord:6.0.301
    volumes:
      - /var/log:/rootfs/var/log:ro
      - /var/lib/docker/:/rootfs/var/lib/docker/:ro
      - /var/run/docker.sock:/rootfs/var/run/docker.sock:ro
      - collectord_cloudwatch_data:/data/
    environment:
      - COLLECTORDCONFIG_PATH=/config/cloudwatch/docker/
      - COLLECTORD__AWS_REGION=aws__region=...
      - COLLECTORD__AWS_ACCESS_KEY_ID=aws__accessKeyID=...
      - COLLECTORD__AWS_SECRET_ACCESS_KEY=aws__secretAccessKey=...
      - COLLECTORD__ACCEPT_LICENSE=general__acceptLicense=false
      - COLLECTORD__LICENSE_KEY=general__license=...
      - COLLECTORD__CLUSTER=general__fields.cluster=-
    restart: always
    cpus: 1
    cpu_shares: 200
    mem_limit: 512M
    mem_reservation: 64M
    privileged: true

volumes:
  collectord_cloudwatch_data:

Docker-compose v3

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
version: "3"
services:

  collectord-cloudwatch:
    image: outcoldsolutions/collectord:6.0.301
    volumes:
      - /var/log:/rootfs/var/log:ro
      - /var/lib/docker/:/rootfs/var/lib/docker/:ro
      - /var/run/docker.sock:/rootfs/var/run/docker.sock:ro
      - collectord_cloudwatch_data:/data/
    environment:
      - COLLECTORDCONFIG_PATH=/config/cloudwatch/docker/
      - COLLECTORD__AWS_REGION=aws__region=...
      - COLLECTORD__AWS_ACCESS_KEY_ID=aws__accessKeyID=...
      - COLLECTORD__AWS_SECRET_ACCESS_KEY=aws__secretAccessKey=...
      - COLLECTORD__ACCEPT_LICENSE=general__acceptLicense=false
      - COLLECTORD__LICENSE_KEY=general__license=...
      - COLLECTORD__CLUSTER=general__fields.cluster=-
    restart: always
    deploy:
      mode: global
      restart_policy:
        condition: any
      resources:
        limits:
          cpus: '1'
          memory: 512M
        reservations:
          cpus: '0.1'
          memory: 64M
    privileged: true

volumes:
  collectord_cloudwatch_data:

Install on ECS

Create a task definition as follow. Make sure to specify the region, access key id, secret access key, review and accept license agreement and include the license key (request an evaluation license key with this automated form)

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
{
    "containerDefinitions": [
        {
            "name": "collectord-cloudwatch",
            "image": "outcoldsolutions/collectord:6.0.301",
            "memory": "512",
            "cpu": "256",
            "essential": true,
            "portMappings": [],
            "environment": [
                {
                    "name": "COLLECTORDCONFIG_PATH",
                    "value": "/config/cloudwatch/docker/"
                },
                {
                    "name": "COLLECTORD__AWS_REGION",
                    "value": "aws__region=..."
                },
                {
                    "name": "COLLECTORD__AWS_ACCESS_KEY_ID",
                    "value": "aws__accessKeyID=..."
                },
                {
                    "name": "COLLECTORD__AWS_SECRET_ACCESS_KEY",
                    "value": "aws__secretAccessKey=..."
                },
                {
                    "name": "COLLECTORD__ACCEPT_LICENSE",
                    "value": "general__acceptLicense=false"
                },
                {
                    "name": "COLLECTORD__LICENSE_KEY",
                    "value": "general__license=..."
                },
                {
                    "name": "COLLECTORD__CLUSTER",
                    "value": "general__fields.cluster=-"
                }
            ],
            "mountPoints": [
                {
                    "sourceVolume": "var_log",
                    "containerPath": "/rootfs/var/log",
                    "readOnly": true
                },
                {
                    "sourceVolume": "var_lib_docker_containers",
                    "containerPath": "/rootfs/var/lib/docker/",
                    "readOnly": true
                },
                {
                    "sourceVolume": "docker_socket",
                    "containerPath": "/rootfs/var/run/docker.sock",
                    "readOnly": true
                },
                {
                    "sourceVolume": "collectord_cloudwatch_data",
                    "containerPath": "/data",
                    "readOnly": false
                }
            ],
            "volumesFrom": null,
            "hostname": null,
            "user": null,
            "workingDirectory": null,
            "privileged": true,
            "readonlyRootFilesystem": true,
            "extraHosts": null,
            "ulimits": null,
            "dockerLabels": null,
            "logConfiguration": {
                "logDriver": "json-file",
                "options": {
                    "max-size": "1m",
                    "max-file": "3"
                }
            }
        }
    ],
    "volumes": [
        {
            "name": "var_log",
            "host": {
                "sourcePath": "/var/log"
            }
        },
        {
            "name": "var_lib_docker_containers",
            "host": {
                "sourcePath": "/var/lib/docker/"
            }
        },
        {
            "name": "docker_socket",
            "host": {
                "sourcePath": "/var/run/docker.sock"
            }
        },
        {
            "name": "collectord_cloudwatch_data",
            "host": {
                "sourcePath": "/var/lib/collectord-cloudwatch/data/"
            }
        }
    ],
    "networkMode": null,
    "memory": "512",
    "cpu": "0.5 vcpu",
    "placementConstraints": [],
    "family": "collectord-cloudwatch",
    "taskRoleArn": ""
}

Schedule this task on your ECS cluster as a service.

  • Launch Type EC2
  • Task definition collectord-cloudwatch
  • Cluster - the name of your ECS cluster
  • Service name - collectord-cloudwatch
  • Service type - DAEMON
  • On second step choose the Load balancer type - None
  • On third step Service Auto Scaling - Do not adjust the service’s desired count

Install on Docker Swarm

To install on docker swarm you can use docker-compose.yaml file from above and command stack deploy

docker stack deploy --compose-file ./docker-compose.yaml collectord-cloudwatch

Verify containers are running

Verify that container is running.

After that you can find all the Log Groups with logs and events in the CloudWatch console.

AWS CloudWatch Logs

  • Installation
    • Setup centralized Logging in 5 minutes.
    • Automatically forward host, container and application logs.
    • Test our solution with the 30 days evaluation license.
  • Annotations
    • Forwarding application logs.
    • Multi-line container logs.
    • Fields extraction for application and container logs (including timestamp extractions).
    • Hiding sensitive data, stripping terminal escape codes and colors.
  • Configuration
    • Advanced configurations for collectord.
  • Troubleshooting
    • Troubleshooting steps.
    • Verify configuration.

About Outcold Solutions

Outcold Solutions provides solutions for building centralized logging infrastructure and monitoring Kubernetes, OpenShift and Docker clusters. We provide easy to setup centralized logging infrastructure with AWS services. We offer Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers.