Docker Centralized Logging with AWS S3, Athena, Glue and QuickSight


Verify configuration

Run the following command to verify the configuration

docker exec collectord-s3 /collectord verify

The output fill look similar to

Version = 6.0.300
Build date = 190311
Environment = docker

  + conf: OK
  + db: OK
  + db-meta: OK
  + instanceID: OK
    instanceID = 2M5FDLFDAH1A0V2N6IQ941G6KG
  + license load: OK
  + license expiration: OK
  + license connection: OK

  Docker configuration:
  + connect: OK
    containers = 6
  + path: OK
  + files: OK

  File Inputs:
  + input(logs): OK
    path /rootfs/var/log/
  x input(syslog): FAILED
    no matches
  x input(journald): FAILED
    err = stat /rootfs/var/log/journal/: no such file or directory

Errors: 2

With the number of the errors at the end. In our example we show output from Docker for mac, where we see some invalid configurations, like

  • input(syslog) - there is no syslog messages for Docker for Mac
  • input(journald) - there is no journald logs for Docker for Mac

Collect diagnostic information

If you need to open a support case you can collect diagnostic information, including performance, metrics and configuration.

1. Collect diagnostics information run following command

The following command takes several minutes.

docker exec collectord-s3 /collector diag --stream 1>diag.tar.gz

You can extract a tar archive to verify the information that we collect. We include information about performance, memory usage, basic telemetry metrics, information file with the information of the host Linux version and basic information about the license.

2. Collect logs

docker logs --details --timestamps collectord-s3  1>collectord-s3.log 2>&1

3. Run verify

docker exec collectord-s3 /collectord verify > verify.log

4. Prepare tar archive

tar -czvf collectord-s3-$(date +%s).tar.gz verify.log collectord-s3.log diag.tar.gz
  • Installation
    • Setup centralized Logging in 5 minutes.
    • Automatically forward host, container and application logs.
    • Test our solution with the 30 days evaluation license.
  • AWS Glue Catalog
    • Table definitions in Glue Catalog.
  • Querying data with Athena
    • Query automatically partitioned data with AWS Athena.
    • Best practices to work with Athena.
    • Query examples for container_logs, events and host_logs.
  • QuickSight for Dashboards and Reports
    • Connecting AWS QuickSight with the Athena.
    • Building dashboards.
  • Access control
    • Limit access to the data with IAM Policy.
  • Annotations
    • Forwarding application logs.
    • Multi-line container logs.
    • Fields extraction for application and container logs (including timestamp extractions).
    • Hiding sensitive data, stripping terminal escape codes and colors.
  • Configuration
    • Advanced configurations for collectord.
  • Troubleshooting
    • Troubleshooting steps.
    • Verify configuration.

About Outcold Solutions

Outcold Solutions provides solutions for building centralized logging infrastructure and monitoring Kubernetes, OpenShift and Docker clusters. We provide easy to setup centralized logging infrastructure with AWS services. We offer Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers.