Collectord

Docker Centralized Logging with AWS CloudWatch Logs

Configuration

Collectord has default configuration embedded. Changing this configuration allows you to control how often data is forwarded to CloudWatch, which host logs should be forwarded, default sampling for the logs and more.

Review configuration

You can always review all the configuration that is applied to collectord by executing the command on one of the running collectord container.

docker exec -it collectord-cloudwatch /collectord show-config

Configuration

Override configuration

Overriding configuration with environment variables

You can override configurations with the environment variables in format

--env "COLLECTOR__<ANYUNIQUENAME>=<section>__<key>=<value>"

Overriding configuration by embedding configuration files

You can create your configuration files, which overrides the default values in 001-general.conf. Just place only the values that you want to replace inside this file, for example, create a file 002-conf.conf

Create a Dockerfile

FROM outcoldsolutions/collectord:6.0.301

COPY 002-conf.conf /config/cloudwatch/docker/002-conf.conf

Build the image

docker build -t example.com/collectord:6.0.301 .

Use this image to start the collector with the instructions how we deploy the collector.

Disable forwarding of host logs

To disable forwarding of the host logs, just set disabled to true for the input.files::logs, input.files::syslog and input.journald.

docker run 
    ... \
    --env "COLLECTORD__DISABLE_HOST_LOGS=input.files::logs__disabled=true" \
    --env "COLLECTORD__DISABLE_SYSLOG=input.files::syslog__disabled=true" \
    --env "COLLECTORD__DISABLE_JOURNALD=input.journald__disabled=true" \
    ... \
    outcoldsolutions/collectord:6.0.301

Use opt-out by default behavior for container logs

By default collectord forwards all container logs to CloudWatch. If you want to disable that, and be able to specify with the annotations from which containers you want to forward logs you can change the configuration for collectord

docker run 
    ... \
    --env "COLLECTORD__CONTAINER_LOGS_DEVNULL=input.files__output=devnull" \
    ... \
    outcoldsolutions/collectord:6.0.301

Sample all container logs by default

To sample all container logs by default, you can set the percent of the logs, that should be forwarded to CloudWatch

docker run 
    ... \
    --env "COLLECTORD__CONTAINER_LOGS_SAMPLE=input.files__samplingPercent=5" \
    ... \
    outcoldsolutions/collectord:6.0.301

Change default retention setting for CloudWatch LogGroup

When collectord creates a new LogGroup it also updates the retention configuration. The default is 90 days.

docker run 
    ... \
    --env "COLLECTORD__CONTAINER_LOGS_RETENTION=output.cloudwatch.logs__retentionInDays=90" \
    ... \
    outcoldsolutions/collectord:6.0.301

Disable telemetry

Collectord forwards very basic telemetry about the performance and enabled configurations. You can disable it

docker run 
    ... \
    --env "COLLECTORD__TELEMETRY=general__telemetryEndpoint=" \
    ... \
    outcoldsolutions/collectord:6.0.301
  • Installation
    • Setup centralized Logging in 5 minutes.
    • Automatically forward host, container and application logs.
    • Test our solution with the 30 days evaluation license.
  • Annotations
    • Forwarding application logs.
    • Multi-line container logs.
    • Fields extraction for application and container logs (including timestamp extractions).
    • Hiding sensitive data, stripping terminal escape codes and colors.
  • Configuration
    • Advanced configurations for collectord.
  • Troubleshooting
    • Troubleshooting steps.
    • Verify configuration.

About Outcold Solutions

Outcold Solutions provides solutions for building centralized logging infrastructure and monitoring Kubernetes, OpenShift and Docker clusters. We provide easy to setup centralized logging infrastructure with AWS services. We offer Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers.