Collectord

Kubernetes Centralized Logging with AWS CloudWatch Logs

Annotations

You can define annotations for the namespaces, workloads and pods. Annotations allows you to change how collector forwards the data. Annotations also helps collector where to discover the application logs.

If you want to specify an annotation, that you want to apply to all deployments of collectord (for example collectord-s3 and collectord-cloudwatch) you can prefix them with collectord.collectord.io/ instead of cloudwatch.collectord.io/. For example, that can be useful if you want to apply event pattern for all collectord deployments.

Override LogGroup and LogStream

With the annotations cloudwatch.collectord.io/logs-loggroup and cloudwatch.collectord.io/logs-logstream you can override the targeting LogGroup and LogStream

apiVersion: v1
kind: Pod
metadata:
  name: nginx-loggroup-and-log-stream
  annotations:
    cloudwatch.collectord.io/logs-logstream: '/{{pod_name}}/{{container_name}}/{{container_id}}'
    cloudwatch.collectord.io/logs-loggroup: '/kubernetes/{{cluster}}/container_logs/{{namespace}}/nginx/'
spec:
  containers:
  - name: nginx
    image: nginx

Make sure that the LogGroup and LogStream are unique per container.

Overriding LogGroup and LogStream for specific events in the stream

By specifying the match and the override loggroup and logstream you can change the destination log group and log stream for the specific events in the stream.

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  annotations:
    cloudwatch.collectord.io/logs-override.1-match: ^(\d{1,3}\.){3}\d{1,3}
    cloudwatch.collectord.io/logs-override.1-loggroup: /kubernetes/nginx-logs
spec:
  containers:
  - name: nginx
    image: nginx

Reference

  • cloudwatch.collectord.io/logs-logstream - specify format for the LogStream
  • cloudwatch.collectord.io/logs-loggroup specify format for the LogGroup
  • cloudwatch.collectord.io/logs-override.{N}-match - match for override pattern
  • cloudwatch.collectord.io/logs-override.{N}-loggroup - override logroup for matched events
  • cloudwatch.collectord.io/logs-override.{N}-logstream - override logsource for matched events
  • cloudwatch.collectord.io/stdout-logs-logstream - specify format for the LogStream
  • cloudwatch.collectord.io/stdout-logs-loggroup specify format for the LogGroup
  • cloudwatch.collectord.io/stdout-logs-override.{N}-match - match for override pattern
  • cloudwatch.collectord.io/stdout-logs-override.{N}-loggroup - override logroup for matched events
  • cloudwatch.collectord.io/stdout-logs-override.{N}-logstream - override logsource for matched events
  • cloudwatch.collectord.io/stderr-logs-logstream - specify format for the LogStream
  • cloudwatch.collectord.io/stderr-logs-loggroup specify format for the LogGroup
  • cloudwatch.collectord.io/stderr-logs-override.{N}-match - match for override pattern
  • cloudwatch.collectord.io/stderr-logs-override.{N}-loggroup - override logroup for matched events
  • cloudwatch.collectord.io/stderr-logs-override.{N}-logstream - override logsource for matched events

Replace patterns in events

You can define replace patterns with the annotations. That allows you to hide sensitive information, or drop unimportant information from the messages.

Replace patterns for container logs are configured with pair of annotations grouped with the same number cloudwatch.collectord.io/logs-replace.1-search and cloudwatch.collectord.io/logs-replace.2-val, first specifies the search pattern as a regular expression, second a replace pattern. In replace patterns you can use placeholders for matches, like $1 or $name for named patterns.

We are using Go regular expression library for replace pipes. You can find more information about the syntax at Package regexp and re2 syntax. We recommend to use https://regex101.com for testing your patterns (set the Flavor to golang).

Using nginx as an example, our logs have a default pattern like

172.17.0.1 - - [31/Aug/2018:21:11:26 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
172.17.0.1 - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
172.17.0.1 - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Example 1. Replacing IPv4 addresses with X.X.X.X

If we want to hide an IP address from the logs by replacing all IPv4 addresses with X.X.X.X

apiVersion: v1
kind: Pod
metadata:
  name: nginx-replace-example-1
  annotations:
    cloudwatch.collectord.io/logs-replace.1-search: (\d{1,3}\.){3}\d{1,3}
    cloudwatch.collectord.io/logs-replace.1-val: X.X.X.X
spec:
  containers:
  - name: nginx
    image: nginx

The result of this replace pattern will be in CloudWatch

X.X.X.X - - [31/Aug/2018:21:11:26 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
X.X.X.X - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
X.X.X.X - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

You can also keep the first part of the IPv4 with

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  annotations:
    cloudwatch.collectord.io/logs-replace.1-search: (?P<IPv4p1>\d{1,3})(\.\d{1,3}){3}
    cloudwatch.collectord.io/logs-replace.1-val: ${IPv4p1}.X.X.X
spec:
  containers:
  - name: nginx
    image: nginx

That results in

172.X.X.X - - [31/Aug/2018:21:11:26 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
172.X.X.X - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
172.X.X.X - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Example 2. Dropping messages

With the replace patterns you can drop messages that you don't want to see. With the example below we drop all log messages resulted from GET requests with 200 response

apiVersion: v1
kind: Pod
metadata:
  name: nginx-replace-example-2
  annotations:
    cloudwatch.collectord.io/logs-replace.1-search: '^.+\"GET [^\s]+ HTTP/[^"]+" 200 .+$'
    cloudwatch.collectord.io/logs-replace.1-val: ''
    cloudwatch.collectord.io/logs-replace.2-search: '(\d{1,3}\.){3}\d{1,3}'
    cloudwatch.collectord.io/logs-replace.2-val: 'X.X.X.X'
spec:
  containers:
  - name: nginx
    image: nginx

In this example we have two replace pipes. The apply in the alphabetical order (replace.1 comes first, before the replace.2).

X.X.X.X - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
X.X.X.X - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Reference

  • cloudwatch.collectord.io/logs-replace.{N}-search - define the search pattern for the replace pipe
  • cloudwatch.collectord.io/logs-replace.{N}-val - define the replace pattern for the replace pipe
  • cloudwatch.collectord.io/stdout-logs-replace.{N}-search - define the search pattern for the replace pipe
  • cloudwatch.collectord.io/stdout-logs-replace.{N}-val - define the replace pattern for the replace pipe
  • cloudwatch.collectord.io/stderr-logs-replace.{N}-search - define the search pattern for the replace pipe
  • cloudwatch.collectord.io/stderr-logs-replace.{N}-val - define the replace pattern for the replace pipe

Hashing values in logs

To hide sensitive data, you can use replace patterns or hashing functions.

apiVersion: v1
kind: Pod
metadata:
  name: nginx-hasing-example
  annotations:
    cloudwatch.collectord.io/logs-hashing.1-match: '(\d{1,3}\.){3}\d{1,3}'
    cloudwatch.collectord.io/logs-hashing.1-function: 'fnv-1a-64'
spec:
  containers:
  - name: nginx
    image: nginx

This example will replace values that look like an IP address in the string

172.17.0.1 - - [16/Nov/2018:11:17:17 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"

With the hashed value, in our example using the algorithm fnv-1a-64

gqsxydjtZL4 - - [16/Nov/2018:11:17:17 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"

Collectord supports a variety of hashing functions including cryptographic hashing functions. The list of supported functions and their performance is listed below (performance in the nanoseconds per operation to hash two IP addresses in a string source: 127.0.0.1, destination: 10.10.1.99)

| Function          | ns / op |
-------------------------------
| adler-32          |    1713 |
| crc-32-ieee       |    1807 |
| crc-32-castagnoli |    1758 |
| crc-32-koopman    |    1753 |
| crc-64-iso        |    1739 |
| crc-64-ecma       |    1740 |
| fnv-1-64          |    1711 |
| fnv-1a-64         |    1711 |
| fnv-1-32          |    1744 |
| fnv-1a-32         |    1738 |
| fnv-1-128         |    1852 |
| fnv-1a-128        |    1836 |
| md5               |    2032 |
| sha1              |    2037 |
| sha256            |    2220 |
| sha384            |    2432 |
| sha512            |    2516 |

Reference

  • cloudwatch.collectord.io/logs-hashing.{N}-match - the regexp for a matched value
  • cloudwatch.collectord.io/logs-hashing.{N}-function - hash function (default is sha256, available adler-32,crc-32-ieee,crc-32-castagnoli,crc-32-koopman,crc-64-iso,crc-64-ecma,fnv-1-64,fnv-1a-64,fnv-1-32,fnv-1a-32,fnv-1-128,fnv-1a-128,md5,sha1,sha256,sha384,sha512)
  • cloudwatch.collectord.io/stdout-logs-hashing.{N}-match - the regexp for a matched value
  • cloudwatch.collectord.io/stdout-logs-hashing.{N}-function - hash function (default is sha256, available adler-32,crc-32-ieee,crc-32-castagnoli,crc-32-koopman,crc-64-iso,crc-64-ecma,fnv-1-64,fnv-1a-64,fnv-1-32,fnv-1a-32,fnv-1-128,fnv-1a-128,md5,sha1,sha256,sha384,sha512)
  • cloudwatch.collectord.io/stderr-logs-hashing.{N}-match - the regexp for a matched value
  • cloudwatch.collectord.io/stderr-logs-hashing.{N}-function - hash function (default is sha256, available adler-32,crc-32-ieee,crc-32-castagnoli,crc-32-koopman,crc-64-iso,crc-64-ecma,fnv-1-64,fnv-1a-64,fnv-1-32,fnv-1a-32,fnv-1-128,fnv-1a-128,md5,sha1,sha256,sha384,sha512)

Escaping terminal sequences, including terminal colors

Some containers does not turn off terminal colors automatically, when they run inside container. For example if you run container with attached tty and define that you want to see colors

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-shell
spec:
  containers:
  - name: ubuntu
    image: ubuntu
    tty: true
    command: [/bin/sh, -c,
             'while true; do ls --color=auto /; sleep 5; done;']

You can find messages similar to below in cloudwatch logs

[01;34mboot  etc  lib   media  opt  root  sbin  sys  usr
[0mbin   dev  home  lib64  mnt  proc  run   srv  tmp  var

You can easily escape them with the annotation cloudwatch.collectord.io/logs-escapeterminalsequences='true'

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-shell
  annotations:
    cloudwatch.collectord.io/logs-escapeterminalsequences: 'true'
spec:
  containers:
  - name: ubuntu
    image: ubuntu
    tty: true
    command: [/bin/sh, -c,
             'while true; do ls --color=auto /; sleep 5; done;']

That way you will see logs in CloudWatch as you would expect

bin   dev  home  lib64  mnt  proc  run   srv  tmp  var
boot  etc  lib   media  opt  root  sbin  sys  usr

In the collector configuration file you can find [input.files]/stripTerminalEscapeSequencesRegex and [input.files]/stripTerminalEscapeSequences that defines default regexp used for removing terminal escape sequences and default value if collector should strip terminal escape sequences (defaults to false).

Reference

  • cloudwatch.collectord.io/logs-escapeterminalsequences - escape terminal sequences (including colors)
  • cloudwatch.collectord.io/stdout-logs-escapeterminalsequences - escape terminal sequences (including colors)
  • cloudwatch.collectord.io/stderr-logs-escapeterminalsequences - escape terminal sequences (including colors)

Extracting fields from the container logs

You can use fields extraction, that allows you to extract timestamps from the messages.

Using the same example with nginx we can define fields extraction for some of the fields.

172.17.0.1 - - [31/Aug/2018:21:11:26 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "-"
172.17.0.1 - - [31/Aug/2018:21:11:32 +0000] "POST / HTTP/1.1" 405 173 "-" "curl/7.54.0" "-"
172.17.0.1 - - [31/Aug/2018:21:11:35 +0000] "GET /404 HTTP/1.1" 404 612 "-" "curl/7.54.0" "-"

Important note, that first unnamed pattern is used as the message for the event.

Example 1. Extracting the timestamp

Assuming we want to keep whole message as is, and extract just a timestamp. We can define the extraction pattern with the regexp. Specify that the timestampfield is timestamp and define the timestampformat.

We use Go time parsing library, that defines the format with the specific date Mon Jan 2 15:04:05 MST 2006. See Go documentation for details.

apiVersion: v1
kind: Pod
metadata:
  name: nginx-fields-example-timestamp
  annotations:
    cloudwatch.collectord.io/logs-extraction: '^(.*\[(?P<timestamp>[^\]]+)\].+)$'
    cloudwatch.collectord.io/logs-timestampfield: timestamp
    cloudwatch.collectord.io/logs-timestampformat: '02/Jan/2006:15:04:05 -0700'
spec:
  containers:
  - name: nginx
    image: nginx

In that way you will get messages in cloudwatch logs with the exact timestamp as specified in your container logs.

Reference

  • cloudwatch.collectord.io/logs-extraction - define the regexp for fields extraction
  • cloudwatch.collectord.io/logs-timestampfield - define the field for timestamp (after fields extraction)
  • cloudwatch.collectord.io/logs-timestampformat - define the timestamp format
  • cloudwatch.collectord.io/logs-timestampsetmonth - define if month should be set to current for timestamp
  • cloudwatch.collectord.io/logs-timestampsetday - define if day should be set to current for timestamp
  • cloudwatch.collectord.io/logs-timestamplocation - define timestamp location if not set by format
  • cloudwatch.collectord.io/stdout-logs-extraction - define the regexp for fields extraction for stdout logs
  • cloudwatch.collectord.io/stdout-logs-timestampfield - define the field for timestamp (after fields extraction)
  • cloudwatch.collectord.io/stdout-logs-timestampformat - define the timestamp format
  • cloudwatch.collectord.io/stdout-logs-timestampsetmonth - define if month should be set to current for timestamp
  • cloudwatch.collectord.io/stdout-logs-timestampsetday - define if day should be set to current for timestamp
  • cloudwatch.collectord.io/stdout-logs-timestamplocation - define timestamp location if not set by format
  • cloudwatch.collectord.io/stderr-logs-extraction - define the regexp for fields extraction for stdout logs
  • cloudwatch.collectord.io/stderr-logs-timestampfield - define the field for timestamp (after fields extraction)
  • cloudwatch.collectord.io/stderr-logs-timestampformat - define the timestamp format
  • cloudwatch.collectord.io/stderr-logs-timestampsetmonth - define if month should be set to current for timestamp
  • cloudwatch.collectord.io/stderr-logs-timestampsetday - define if day should be set to current for timestamp
  • cloudwatch.collectord.io/stderr-logs-timestamplocation - define timestamp location if not set by format

Defining Event pattern

With the annotation cloudwatch.collectord.io/logs-eventpattern you can define how collector should identify new events in the pipe. The default event pattern is defined by the collectord as ^[^\s] (anything that does not start from a space character).

The default pattern works in most of the cases, but does not work in some, like Java exceptions, where the call stack of the error starts on the next line, and it does not start with the space character.

In example below we intentionally made a mistake in a configuration for the ElasticSearch (s-node should be a single-node) to get the error message

apiVersion: v1
kind: Pod
metadata:
  name: elasticsearch-pod
spec:
  containers:
  - name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:6.4.0
    env:
    - name: discovery.type
      value: s-node

Results in

[2018-08-31T22:44:56,433][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/92] [Main.cc@109] controller (64 bit): Version 6.4.0 (Build cf8246175efff5) Copyright (c) 2018 Elasticsearch BV
[2018-08-31T22:44:56,886][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Unknown discovery type [s-node]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.4.0.jar:6.4.0]
    at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.4.0.jar:6.4.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.0.jar:6.4.0]
Caused by: java.lang.IllegalArgumentException: Unknown discovery type [s-node]
    at org.elasticsearch.discovery.DiscoveryModule.<init>(DiscoveryModule.java:129) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.node.Node.<init>(Node.java:477) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.0.jar:6.4.0]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.0.jar:6.4.0]
    ... 6 more
[2018-08-31T22:44:56,892][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started

And with the default pattern we will not have the warning line [2018-08-31T22:44:56,886][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main] with the whole callstack.

We can define that every log event in this container should start with the [ character with the regular expression as

apiVersion: v1
kind: Pod
metadata:
  name: elasticsearch-pod
  annotations:
    cloudwatch.collectord.io/logs-eventpattern: '^\['
spec:
  containers:
  - name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:6.4.0
    env:
    - name: discovery.type
      value: s-node

Reference

  • cloudwatch.collectord.io/logs-eventpattern - set the regex identifying the event start pattern for container logs
  • cloudwatch.collectord.io/logs-joinpartial - join partial events (when container runtime splits them), defaults to true.
  • cloudwatch.collectord.io/logs-joinmultiline - join multiline logs, defaults to true
  • cloudwatch.collectord.io/stdout-logs-eventpattern - set the regex identifying the event start pattern for container logs on stdout
  • cloudwatch.collectord.io/stdout-logs-joinpartial - join partial events on stdout (when container runtime splits them), defaults to true.
  • cloudwatch.collectord.io/stdout-logs-joinmultiline - join multiline logs on stdout, defaults to true
  • cloudwatch.collectord.io/stderr-logs-eventpattern - set the regex identifying the event start pattern for container logs on stderr
  • cloudwatch.collectord.io/stderr-logs-joinpartial - join partial events on stderr (when container runtime splits them), defaults to true.
  • cloudwatch.collectord.io/stderr-logs-joinmultiline - join multiline logs on stderr, defaults to true

Application Logs

Sometimes it is hard or just not practical to redirect all logs from the container to stdout and stderr of the container. In that cases you keep the logs in the container. We call them application logs. With collector you can easily pick up these logs and forward them. No additional sidecars or processes required inside your container.

Let's take a look on the example below. We have a postgresql container, that redirects most of the logs to the path inside the container /var/log/postgresql. We define for this container a volume (emptyDir driver) with the name psql_logs and mount it to /var/log/postgresql/. With the annotation cloudwatch.collectord.io/volume.1-logs-name=psql_logs we tell collector to pick up all the logs with the default glob pattern *.log* (default glob pattern is set int the collector configuration, and you can override it with annotation cloudwatch.collectord.io/volume.{N}-logs-glob) in the volume and forward them automatically to CloudWatch.

When you need to forward logs from multiple volumes of the same container you can group the settings with the same number, for example cloudwatch.collectord.io/volume.1-logs-name=psql_logs and cloudwatch.collectord.io/volume.2-logs-name=psql_logs

Example 1. Forwarding application logs

apiVersion: v1
kind: Pod
metadata:
  name: postgres-pod
  annotations:
    cloudwatch.collectord.io/volume.1-logs-name: 'logs'
spec:
  containers:
  - name: postgres
    image: postgres
    command:
      - docker-entrypoint.sh
    args:
      - postgres
      - -c
      - logging_collector=on
      - -c
      - log_min_duration_statement=0
      - -c
      - log_directory=/var/log/postgresql
      - -c
      - log_min_messages=INFO
      - -c
      - log_rotation_age=1d
      - -c
      - log_rotation_size=10MB
    volumeMounts:
      - name: data
        mountPath: /var/lib/postgresql/data
      - name: logs
        mountPath: /var/log/postgresql/
  volumes:
  - name: data
    emptyDir: {}
  - name: logs
    emptyDir: {}

In the example above the logs from the container will have a source, similar to psql_logs:postgresql-2018-08-31_232946.log.

2018-08-31 23:31:02.034 UTC [133] LOG:  duration: 0.908 ms  statement: SELECT n.nspname as "Schema",
      c.relname as "Name",
      CASE c.relkind WHEN 'r' THEN 'table' WHEN 'v' THEN 'view' WHEN 'm' THEN 'materialized view' WHEN 'i' THEN 'index' WHEN 'S' THEN 'sequence' WHEN 's' THEN 'special' WHEN 'f' THEN 'foreign table' WHEN 'p' THEN 'table' END as "Type",
      pg_catalog.pg_get_userbyid(c.relowner) as "Owner"
    FROM pg_catalog.pg_class c
         LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
    WHERE c.relkind IN ('r','p','')
          AND n.nspname <> 'pg_catalog'
          AND n.nspname <> 'information_schema'
          AND n.nspname !~ '^pg_toast'
      AND pg_catalog.pg_table_is_visible(c.oid)
    ORDER BY 1,2;
2018-08-31 23:30:53.490 UTC [124] FATAL:  role "postgresql" does not exist

Example 2. Forwarding application logs with fields extraction and time parsing

With the annotations for application logs you can define fields extraction, replace patterns, override the indexes, sources and hosts.

As an example, with the extraction pattern and timestamp parsing you can do

apiVersion: v1
kind: Pod
metadata:
  name: postgres-pod
  annotations:
    cloudwatch.collectord.io/volume.1-logs-name: 'logs'
    cloudwatch.collectord.io/volume.1-logs-extraction: '^(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3} [^\s]+) (.+)$'
    cloudwatch.collectord.io/volume.1-logs-timestampfield: 'timestamp'
    cloudwatch.collectord.io/volume.1-logs-timestampformat: '2006-01-02 15:04:05.000 MST'
spec:
  containers:
  - name: postgres
    image: postgres
    command:
      - docker-entrypoint.sh
    args:
      - postgres
      - -c
      - logging_collector=on
      - -c
      - log_min_duration_statement=0
      - -c
      - log_directory=/var/log/postgresql
      - -c
      - log_min_messages=INFO
      - -c
      - log_rotation_age=1d
      - -c
      - log_rotation_size=10MB
    volumeMounts:
      - name: data
        mountPath: /var/lib/postgresql/data
      - name: logs
        mountPath: /var/log/postgresql/
  volumes:
  - name: data
    emptyDir: {}
  - name: logs
    emptyDir: {}

That way you will extract the timestamps and remove them from the _raw message

_time               | _raw
2018-08-31 23:31:02 | [133] LOG:  duration: 0.908 ms  statement: SELECT n.nspname as "Schema",
                    |     c.relname as "Name",
                    |     CASE c.relkind WHEN 'r' THEN 'table' WHEN 'v' THEN 'view' WHEN 'm' THEN 'materialized view' WHEN 'i' THEN 'index' WHEN 'S' THEN 'sequence' WHEN 's' THEN 'special' WHEN 'f' THEN 'foreign table' WHEN 'p' THEN 'table' END as "Type",
                    |     pg_catalog.pg_get_userbyid(c.relowner) as "Owner"
                    |   FROM pg_catalog.pg_class c
                    |        LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
                    |   WHERE c.relkind IN ('r','p','')
                    |         AND n.nspname <> 'pg_catalog'
                    |         AND n.nspname <> 'information_schema'
                    |         AND n.nspname !~ '^pg_toast'
                    |     AND pg_catalog.pg_table_is_visible(c.oid)
                    |   ORDER BY 1,2;
2018-08-31 23:30:53 |  UTC [124] FATAL:  role "postgresql" does not exist

Volume types

Collector supports two volume types for application logs: emptyDir and hostPath. Collectord has two settings that helps collector to autodiscover application logs. First is the [general.kubernetes]/volumesRootDir for discovering volumes created with emptyDir, second is [input.app_logs]/root for discovering host mounts, considering that they will be mounted with different path to collector.

Reference

  • cloudwatch.collectord.io/volume.{N}-logs-name - name of the volume attached to Pod
  • cloudwatch.collectord.io/volume.{N}-logs-eventpattern - change the event pattern defining new event for logs forwarded from the volume
  • cloudwatch.collectord.io/volume.{N}-logs-replace.{N}-search - specify the regex search for replace pipe for the logs
  • cloudwatch.collectord.io/volume.{N}-logs-replace.{N}-val - specify the regex replace pattern for replace pipe for the logs
  • cloudwatch.collectord.io/volume.{N}-logs-hashing.{N}-match - the regexp for a matched value
  • cloudwatch.collectord.io/volume.{N}-logs-hashing.{N}-function - hash function (default is sha256, available adler-32,crc-32-ieee,crc-32-castagnoli,crc-32-koopman,crc-64-iso,crc-64-ecma,fnv-1-64,fnv-1a-64,fnv-1-32,fnv-1a-32,fnv-1-128,fnv-1a-128,md5,sha1,sha256,sha384,sha512)
  • cloudwatch.collectord.io/volume.{N}-logs-extraction - specify the fields extraction with the regex the logs
  • cloudwatch.collectord.io/volume.{N}-logs-timestampfield - specify the timestamp field
  • cloudwatch.collectord.io/volume.{N}-logs-timestampformat - specify the format for timestamp field
  • cloudwatch.collectord.io/volume.{N}-logs-timestampsetmonth - define if month should be set to current for timestamp
  • cloudwatch.collectord.io/volume.{N}-logs-timestampsetday - define if day should be set to current for timestamp
  • cloudwatch.collectord.io/volume.{N}-logs-timestamplocation - define timestamp location if not set by format
  • cloudwatch.collectord.io/volume.{N}-logs-glob - set the glob pattern for matching logs
  • cloudwatch.collectord.io/volume.{N}-logs-match - set the regexp pattern for matching logs
  • cloudwatch.collectord.io/volume.{N}-logs-recursive - set if walker should walk the directory recursive
  • cloudwatch.collectord.io/volume.{N}-logs-sampling-percent - specify the % value of logs that should be forwarded to CloudWatch
  • cloudwatch.collectord.io/volume.{N}-logs-sampling-key - regexp pattern to specify the key for the sampling based on hash values
  • cloudwatch.collectord.io/volume.{N}-logs-logstream - specify format for the LogStream
  • cloudwatch.collectord.io/volume.{N}-logs-loggroup specify format for the LogGroup

Change output destination

By default collector forwards all the data to cloudwatch logs. You can configure containers to redirect data to devnull instead with annotation cloudwatch.collectord.io/logs-output=devnull. This annotations changes output for logs from this Pod.

For example:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  labels:
    app: MyApp
  annotations:
    cloudwatch.collectord.io/logs-output: 'devnull'
spec:
  containers:
  - name: nginx
    image: nginx

Reference

  • s3.collectord.io/logs-output changing the output for stdout and stderr streams
  • s3.collectord.io/stdout-logs-output changing the output for stdout stream
  • s3.collectord.io/stderr-logs-output changing the output for stderr stream

Logs sampling

Example 1. Random based sampling

When the application produces a high amount of logs, in some cases it could be enough to just on the sampled amount of the logs to understand how many failed requests the application has, or how it behaves. You can add an annotation for the logs to specify the percent amount of the logs that should be forwarded to CloudWatch.

In the following example, this application produces 300,000 log lines. Only about 60,000 log lines are going to be forwarded to CloudWatch.

apiVersion: v1
kind: Pod
metadata:
  name: logtest
  annotations:
    cloudwatch.collectord.io/logs-sampling-percent: '20'
spec:
  containers:
  - name: logtest
    image: docker.io/mffiedler/ocp-logtest:latest
    args: [python, ocp_logtest.py,
           --line-length=1024, --num-lines=300000, --rate=60000, --fixed-line]

Example 2. Hash-based sampling

In the situations where you want to look at the pattern for a specific user, you can specify that you want to sample logs based on the hash value, to be sure if the same key presents in two different log lines, both of them will be forwarded to CloudWatch.

In the following example we define a key (should be a named submatch pattern) as an IP address.

apiVersion: v1
kind: Pod
metadata:
  name: nginx-sampling
  annotations:
    cloudwatch.collectord.io/logs-sampling-percent: '20'
    cloudwatch.collectord.io/logs-sampling-key: '^(?P<key>(\d+\.){3}\d+)'
spec:
  containers:
  - name: nginx-sampling
    image: nginx

Handling multiple containers

Pod can have multiple containers. You can define annotations for a specific container with the name prefixed the annotation. The format of the annotations is cloudwatch.collectord.io/{container_name}--{annotation}: {annotation-value}. As an example.

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  annotations:
    cloudwatch.collectord.io/web--logs-index: 'web'
    cloudwatch.collectord.io/web--logs-replace.2-search: '(?P<IPv4p1>\d{1,3})(\.\d{1,3}){3}'
    cloudwatch.collectord.io/web--logs-replace.2-val: '${IPv4p1}.X.X.X'
    cloudwatch.collectord.io/user--logs-disabled: 'true'
spec:
  containers:
  - name: web
    image: nginx
  - name: user
    image: busybox
    args: [/bin/sh, -c,
           'while true; do wget -qO- localhost:80 &> /dev/null; sleep 5; done']

Troubleshooting

Check the collector logs for warning messages about the annotations, you can find if you made a misprint in the annotations if you see warnings like

WARN 2018/08/31 21:05:33.122978 core/input/annotations.go:76: invalid annotation ...
  • Installation
    • Setup centralized Logging in 5 minutes.
    • Automatically forward host, container and application logs.
    • Test our solution with the 30 days evaluation license.
  • Annotations
    • Forwarding application logs.
    • Multi-line container logs.
    • Fields extraction for application and container logs (including timestamp extractions).
    • Hiding sensitive data, stripping terminal escape codes and colors.
  • Configuration
    • Advanced configurations for collectord.
  • Troubleshooting
    • Troubleshooting steps.
    • Verify configuration.

About Outcold Solutions

Outcold Solutions provides solutions for building centralized logging infrastructure and monitoring Kubernetes, OpenShift and Docker clusters. We provide easy to setup centralized logging infrastructure with AWS services. We offer Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers.