Collectord

Docker Centralized Logging with AWS S3, Athena, Glue and QuickSight

Installation

Pre-requirements

You should create or choose the AWS S3 bucket, where you want to keep logs and events, forwarded by Collectord. Secondary you need to create a user with programmatic access with the policy that will allow Collectord to push the files to S3, create the database, tables and partitions in Glue catalog.

The data is automatically stored under prefix /docker in S3 bucket and partitioned with formats:

  • Container Logs: cluster={{cluster::query_escape()}}/host={{host::query_escape()}}/container_image={{container_image::query_escape()}}/container_name={{container_name::query_escape()}}/dt={{timestamp::format(20060102)}}/
  • Host logs: cluster={{cluster::query_escape()}}/host={{host::query_escape()}}/dt={{timestamp::format(20060102)}}/
  • Events: cluster={{cluster::query_escape()}}/host={{host::query_escape()}}/dt={{timestamp::format(20060102)}}/

Creating a bucket

With you AWS account create a bucket, where you want to store the logs. You can choose to use aws cli tool or create it with AWS Web Console.

Write down the name of the bucket and the region where you have created it. In our examples below we will use the name example.com.logs as a bucket name, and us-west-2 as a region.

For security reasons we recommend to choose Automatically encrypt objects when they are stored in S3 and use AES-256 or AWS-KMS (the latest can add additional cost for the usage of the KMS).

CLI example

Save in bash environment variables the name of the bucket and the region.

To create a bucket example.com.logs in region us-west-2

aws s3api create-bucket --bucket example.com.logs --region us-west-2 --create-bucket-configuration LocationConstraint=us-west-2

To enable default encryption (encryption at rest) for the bucket example.com.logs

aws s3api put-bucket-encryption --bucket example.com.logs --server-side-encryption-configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm": "AES256"}}]}'

Create IAM User with Programmatic access

Collectord uses IAM user for authentication purposes to push data to S3 and create databases, tables and partitions in Glue. You can create IAM user with CLI or Web Console by following the guidance https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html. In the example below we restrict Collectord to write only to bucket example.com.logs, and provide write only access to create database, tables and partitions in Glue catalog.

Replace the bucket name example.com.logs with the bucket name that you created on previous step.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::example.com.logs/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "glue:CreateTable",
                "glue:CreateDatabase",
                "glue:CreatePartition"
            ],
            "Resource": "*"
        }
    ]
}

CLI Example

Create policy

Replace the bucket name example.com.logs with the bucket name that you created on previous step.

aws iam create-policy --policy-name collectord-s3 --policy-document "{\"Version\":\"2012-10-17\",\"Statement\": [{\"Effect\": \"Allow\", \"Action\": \"s3:PutObject\", \"Resource\": \"arn:aws:s3:::example.com.logs/*\"}, {\"Effect\": \"Allow\", \"Action\": [\"glue:CreateTable\", \"glue:CreateDatabase\", \"glue:CreatePartition\"], \"Resource\": \"*\"} ] }"

From the output note the Arn of the policy (like arn:aws:iam::999999999999:policy/collectord-s3)

Create user

aws iam create-user --user collectord-s3

Attach just created policy to the user

Replace the ARN with the Policy ARN from the 2 steps before

aws iam attach-user-policy --user collectord-s3 --policy-arn arn:aws:iam::999999999999:policy/collectord-s3

Create Access Key and Secret for collectord-s3 user

aws iam create-access-key --user collectord-s3

From the output note the AccessKeyId and SecretAccessKey

Docker Container Runtime

Collectord uses JSON-files generated by JSON logging driver as a source for container logs.

Some linux distributions, CentOS for example, by default enable journald logging driver instead of default JSON logging driver. You can verify which driver is used by default

$ docker info | grep "Logging Driver"
Logging Driver: json-file

If docker configuration file location is /etc/sysconfig/docker (common in CentOS/RHEL case with Docker 1.13), you can change it and restart docker daemon after that with following commands.

$ sed -i 's/--log-driver=journald/--log-driver=json-file --log-opt max-size=100M --log-opt max-file=3/' /etc/sysconfig/docker
$ systemctl restart docker

If you configure Docker daemon with daemon.json in /etc/docker/daemon.json (common in Debian/Ubuntu), you can change it and restart docker daemon.

{
  "log-driver": "json-file",
  "log-opts" : {
    "max-size" : "100m",
    "max-file" : "3"
  }
}
$ systemctl restart docker

Please follow the manual to learn how to configure default logging driver for containers:

Log Configuration for ECS tasks

When you configure tasks on ECS instances, make sure to always setup a logConfiguration as following (adjust max-size and max-file as appropriate for your service)

"logConfiguration": {
    "logDriver": "json-file",
    "options": {
        "max-size": "10m",
        "max-file": "3"
    }
}

You can find it when you configure tasks as JSON.

JSON logging driver configuration

With the default configuration, docker does not rotate JSON log files, with time they can become large and consume all disk space. That is why we specify max-size and max-file with the default configurations. See Configure and troubleshoot the Docker daemon for more details.

Install on Docker

The following command will schedule Collectord on your Docker and start forwarding all the logs to the S3. Modify the highlighted lines. You need to specify AWS Region, bucket name, access key id and secret access key (created in pre-requirements), review and accept license agreement and include the license key (request an evaluation license key with this automated form)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
docker run -d \
    --name collectord-s3 \
    --volume /var/lib/docker/:/rootfs/var/lib/docker/:ro \
    --volume /var/run/docker.sock:/rootfs/var/run/docker.sock:ro \
    --volume /var/log:/rootfs/var/log:ro \
    --volume collectord-s3-data:/data/ \
    --volume collectord-s3-var:/var/ \
    --cpus=1 \
    --cpu-shares=100 \
    --memory=512M \
    --restart=always \
    --env "COLLECTORDCONFIG_PATH=/config/s3/docker/" \
    --env "COLLECTORD__AWS_REGION=aws__region=..." \
    --env "COLLECTORD__S3_BUCKET=output.s3__bucket=..."  \
    --env "COLLECTORD__AWS_ACCESS_KEY_ID=aws__accessKeyID=..." \
    --env "COLLECTORD__AWS_SECRET_ACCESS_KEY=aws__secretAccessKey=..." \
    --env "COLLECTORD__ACCEPT_LICENSE=general__acceptLicense=false" \
    --env "COLLECTORD__LICENSE_KEY=general__license=..." \
    --env "COLLECTORD__CLUSTER=general__fields.cluster=-" \ 
    --privileged \
    outcoldsolutions/collectord:6.0.301

For example (you can also name your cluster if you want to separate logs from different clusters)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
docker run -d \
    --name collectord-s3 \
    --volume /var/lib/docker/:/rootfs/var/lib/docker/:ro \
    --volume /var/run/docker.sock:/rootfs/var/run/docker.sock:ro \
    --volume /var/log:/rootfs/var/log:ro \
    --volume collectord-s3-data:/data/ \
    --volume collectord-s3-var:/var/ \
    --cpus=1 \
    --cpu-shares=100 \
    --memory=512M \
    --restart=always \
    --env "COLLECTORDCONFIG_PATH=/config/s3/docker/" \
    --env "COLLECTORD__AWS_REGION=aws__region=us-west-2" \
    --env "COLLECTORD__S3_BUCKET=output.s3__bucket=example.com.logs"  \
    --env "COLLECTORD__AWS_ACCESS_KEY_ID=aws__accessKeyID=AKIAI7SNNYYAV456WBVQ" \
    --env "COLLECTORD__AWS_SECRET_ACCESS_KEY=aws__secretAccessKey=CKnnNl8DqPrqeudQxV2vPgXga6BCR0y4RMrlirmC" \
    --env "COLLECTORD__ACCEPT_LICENSE=general__acceptLicense=true" \
    --env "COLLECTORD__LICENSE_KEY=general__license=Qkc1MTgzUTQ0SUUyTTowOjoz...." \
    --env "COLLECTORD__CLUSTER=general__fields.cluster=dev" \ 
    --privileged \
    outcoldsolutions/collectord:6.0.301

Docker-compose v2

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
version: "2.2"
services:

  collectord-s3:
    image: outcoldsolutions/collectord:6.0.301
    volumes:
      - /var/log:/rootfs/var/log:ro
      - /var/lib/docker/:/rootfs/var/lib/docker/:ro
      - /var/run/docker.sock:/rootfs/var/run/docker.sock:ro
      - collectord_s3_data:/data/
      - collectord_s3_var:/var/
    environment:
      - COLLECTORDCONFIG_PATH=/config/s3/docker/
      - COLLECTORD__AWS_REGION=aws__region=...
      - COLLECTORD__S3_BUCKET=output.s3__bucket=...
      - COLLECTORD__AWS_ACCESS_KEY_ID=aws__accessKeyID=...
      - COLLECTORD__AWS_SECRET_ACCESS_KEY=aws__secretAccessKey=...
      - COLLECTORD__ACCEPT_LICENSE=general__acceptLicense=false
      - COLLECTORD__LICENSE_KEY=general__license=...
      - COLLECTORD__CLUSTER=general__fields.cluster=-
    restart: always
    cpus: 1
    cpu_shares: 200
    mem_limit: 512M
    mem_reservation: 64M
    privileged: true

volumes:
  collectord_s3_data:
  collectord_s3_var:

Docker-compose v3

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
version: "3"
services:

  collectord-s3:
    image: outcoldsolutions/collectord:6.0.301
    volumes:
      - /var/log:/rootfs/var/log:ro
      - /var/lib/docker/:/rootfs/var/lib/docker/:ro
      - /var/run/docker.sock:/rootfs/var/run/docker.sock:ro
      - collectord_s3_data:/data/
      - collectord_s3_var:/var/
    environment:
      - COLLECTORDCONFIG_PATH=/config/s3/docker/
      - COLLECTORD__AWS_REGION=aws__region=...
      - COLLECTORD__S3_BUCKET=output.s3__bucket=...
      - COLLECTORD__AWS_ACCESS_KEY_ID=aws__accessKeyID=...
      - COLLECTORD__AWS_SECRET_ACCESS_KEY=aws__secretAccessKey=...
      - COLLECTORD__ACCEPT_LICENSE=general__acceptLicense=false
      - COLLECTORD__LICENSE_KEY=general__license=...
      - COLLECTORD__CLUSTER=general__fields.cluster=-
    restart: always
    deploy:
      mode: global
      restart_policy:
        condition: any
      resources:
        limits:
          cpus: '1'
          memory: 512M
        reservations:
          cpus: '0.1'
          memory: 64M
    privileged: true

volumes:
  collectord_s3_data:
  collectord_s3_var:

Install on ECS

Create a task definition as follow. Make sure to specify the region, access key id, secret access key, review and accept license agreement and include the license key (request an evaluation license key with this automated form)

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
{
    "containerDefinitions": [
        {
            "name": "collectord-s3",
            "image": "outcoldsolutions/collectord:6.0.301",
            "memory": "512",
            "cpu": "256",
            "essential": true,
            "portMappings": [],
            "environment": [
                {
                    "name": "COLLECTORDCONFIG_PATH",
                    "value": "/config/s3/docker/"
                },
                {
                    "name": "COLLECTORD__AWS_REGION",
                    "value": "aws__region=..."
                },
                {
                    "name": "COLLECTORD__S3_BUCKET",
                    "value": "output.s3__bucket=..."
                },
                {
                    "name": "COLLECTORD__AWS_ACCESS_KEY_ID",
                    "value": "aws__accessKeyID=..."
                },
                {
                    "name": "COLLECTORD__AWS_SECRET_ACCESS_KEY",
                    "value": "aws__secretAccessKey=..."
                },
                {
                    "name": "COLLECTORD__ACCEPT_LICENSE",
                    "value": "general__acceptLicense=false"
                },
                {
                    "name": "COLLECTORD__LICENSE_KEY",
                    "value": "general__license=..."
                },
                {
                    "name": "COLLECTORD__CLUSTER",
                    "value": "general__fields.cluster=-"
                }
            ],
            "mountPoints": [
                {
                    "sourceVolume": "var_log",
                    "containerPath": "/rootfs/var/log",
                    "readOnly": true
                },
                {
                    "sourceVolume": "var_lib_docker_containers",
                    "containerPath": "/rootfs/var/lib/docker/",
                    "readOnly": true
                },
                {
                    "sourceVolume": "docker_socket",
                    "containerPath": "/rootfs/var/run/docker.sock",
                    "readOnly": true
                },
                {
                    "sourceVolume": "collectord_s3_data",
                    "containerPath": "/data",
                    "readOnly": false
                },
                {
                    "sourceVolume": "collectord_s3_var",
                    "containerPath": "/var",
                    "readOnly": false
                }
            ],
            "volumesFrom": null,
            "hostname": null,
            "user": null,
            "workingDirectory": null,
            "privileged": true,
            "readonlyRootFilesystem": true,
            "extraHosts": null,
            "ulimits": null,
            "dockerLabels": null,
            "logConfiguration": {
                "logDriver": "json-file",
                "options": {
                    "max-size": "1m",
                    "max-file": "3"
                }
            }
        }
    ],
    "volumes": [
        {
            "name": "var_log",
            "host": {
                "sourcePath": "/var/log"
            }
        },
        {
            "name": "var_lib_docker_containers",
            "host": {
                "sourcePath": "/var/lib/docker/"
            }
        },
        {
            "name": "docker_socket",
            "host": {
                "sourcePath": "/var/run/docker.sock"
            }
        },
        {
            "name": "collectord_s3_data",
            "host": {
                "sourcePath": "/var/lib/collectord-s3/data/"
            }
        },
        {
            "name": "collectord_s3_var",
            "host": {
                "sourcePath": "/var/lib/collectord-s3/var/"
            }
        }
    ],
    "networkMode": null,
    "memory": "512",
    "cpu": "0.5 vcpu",
    "placementConstraints": [],
    "family": "collectord-s3",
    "taskRoleArn": ""
}

Schedule this task on your ECS cluster as a service.

  • Launch Type EC2
  • Task definition collectord-s3
  • Cluster - the name of your ECS cluster
  • Service name - collectord-s3
  • Service type - DAEMON
  • On second step choose the Load balancer type - None
  • On third step Service Auto Scaling - Do not adjust the service’s desired count

Install on Docker Swarm

To install on docker swarm you can use docker-compose.yaml file from above and command stack deploy

docker stack deploy --compose-file ./docker-compose.yaml collectord-s3

Verify containers are running

Verify that container is running.

By default collectord forwards data every 10 minutes, if this is a freshly created cluster, it can take up to 10 minutes to see the data on S3. You can query it now with Athena.

  • Installation
    • Setup centralized Logging in 5 minutes.
    • Automatically forward host, container and application logs.
    • Test our solution with the 30 days evaluation license.
  • AWS Glue Catalog
    • Table definitions in Glue Catalog.
  • Querying data with Athena
    • Query automatically partitioned data with AWS Athena.
    • Best practices to work with Athena.
    • Query examples for container_logs, events and host_logs.
  • QuickSight for Dashboards and Reports
    • Connecting AWS QuickSight with the Athena.
    • Building dashboards.
  • Access control
    • Limit access to the data with IAM Policy.
  • Annotations
    • Forwarding application logs.
    • Multi-line container logs.
    • Fields extraction for application and container logs (including timestamp extractions).
    • Hiding sensitive data, stripping terminal escape codes and colors.
  • Configuration
    • Advanced configurations for collectord.
  • Troubleshooting
    • Troubleshooting steps.
    • Verify configuration.

About Outcold Solutions

Outcold Solutions provides solutions for building centralized logging infrastructure and monitoring Kubernetes, OpenShift and Docker clusters. We provide easy to setup centralized logging infrastructure with AWS services. We offer Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers.