Collectord

Kubernetes Centralized Logging with AWS S3, Athena, Glue and QuickSight

Installation

Pre-requirements

You should create or choose the AWS S3 bucket, where you want to keep logs and events, forwarded by Collectord. Secondary you need to create a user with programmatic access with the policy that will allow Collectord to push the files to S3, create the database, tables and partitions in Glue catalog.

The data is automatically stored under prefix /kubernetes in S3 bucket and partitioned with formats:

  • Container Logs: cluster={{cluster::query_escape()}}/namespace={{namespace::query_escape()}}/workload={{::coalesce(daemonset_name, deployment_name, statefulset_name, cronjob_name, job_name, replicaset_name, pod_name)::query_escape()}}/pod_name={{pod_name::query_escape()}}/container_name={{container_name::query_escape()}}/dt={{timestamp::format(20060102)}}/
  • Host logs: cluster={{cluster::query_escape()}}/host={{host::query_escape()}}/dt={{timestamp::format(20060102)}}/
  • Events: cluster={{cluster::query_escape()}}/namespace={{namespace::query_escape()}}/dt={{timestamp::format(20060102)}}/

Creating a bucket

With you AWS account create a bucket, where you want to store the logs. You can choose to use aws cli tool or create it with AWS Web Console.

Write down the name of the bucket and the region where you have created it. In our examples below we will use the name example.com.logs as a bucket name, and us-west-2 as a region.

For security reasons we recommend to choose Automatically encrypt objects when they are stored in S3 and use AES-256 or AWS-KMS (the latest can add additional cost for the usage of the KMS).

CLI example

Save in bash environment variables the name of the bucket and the region.

To create a bucket example.com.logs in region us-west-2

aws s3api create-bucket --bucket example.com.logs --region us-west-2 --create-bucket-configuration LocationConstraint=us-west-2

To enable default encryption (encryption at rest) for the bucket example.com.logs

aws s3api put-bucket-encryption --bucket example.com.logs --server-side-encryption-configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm": "AES256"}}]}'

Create IAM User with Programmatic access

Collectord uses IAM user for authentication purposes to push data to S3 and create databases, tables and partitions in Glue. You can create IAM user with CLI or Web Console by following the guidance https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html. In the example below we restrict Collectord to write only to bucket example.com.logs, and provide write only access to create database, tables and partitions in Glue catalog.

Replace the bucket name example.com.logs with the bucket name that you created on previous step.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::example.com.logs/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "glue:CreateTable",
                "glue:CreateDatabase",
                "glue:CreatePartition"
            ],
            "Resource": "*"
        }
    ]
}

CLI Example

Create policy

Replace the bucket name example.com.logs with the bucket name that you created on previous step.

aws iam create-policy --policy-name collectord-s3 --policy-document "{\"Version\":\"2012-10-17\",\"Statement\": [{\"Effect\": \"Allow\", \"Action\": \"s3:PutObject\", \"Resource\": \"arn:aws:s3:::example.com.logs/*\"}, {\"Effect\": \"Allow\", \"Action\": [\"glue:CreateTable\", \"glue:CreateDatabase\", \"glue:CreatePartition\"], \"Resource\": \"*\"} ] }"

From the output note the Arn of the policy (like arn:aws:iam::999999999999:policy/collectord-s3)

Create user

aws iam create-user --user collectord-s3

Attach just created policy to the user

Replace the ARN with the Policy ARN from the 2 steps before

aws iam attach-user-policy --user collectord-s3 --policy-arn arn:aws:iam::999999999999:policy/collectord-s3

Create Access Key and Secret for collectord-s3 user

aws iam create-access-key --user collectord-s3

From the output note the AccessKeyId and SecretAccessKey

Container Runtime

Collector works out of the box with CRI-O and Docker as runtime engines.

Our default configuration is optimized for the Kubernetes clusters deployed in Production environments, some data might now be available with minikube. For example minikube forwards host logs to journald without persistence on the disk and combines multiple control plane components into one process.

Docker Container Runtime

If you use Docker as a Container Runtime, the collector uses JSON-files generated by JSON logging driver as a source for container logs.

Some linux distributions, CentOS for example, by default enable journald logging driver instead of default JSON logging driver. You can verify which driver is used by default

$ docker info | grep "Logging Driver"
Logging Driver: json-file

If docker configuration file location is /etc/sysconfig/docker (common in CentOS/RHEL case with Docker 1.13), you can change it and restart docker daemon after that with following commands.

$ sed -i 's/--log-driver=journald/--log-driver=json-file --log-opt max-size=100M --log-opt max-file=3/' /etc/sysconfig/docker
$ systemctl restart docker

If you configure Docker daemon with daemon.json in /etc/docker/daemon.json (common in Debian/Ubuntu), you can change it and restart docker daemon.

{
  "log-driver": "json-file",
  "log-opts" : {
    "max-size" : "100m",
    "max-file" : "3"
  }
}
$ systemctl restart docker

Please follow the manual to learn how to configure default logging driver for containers:

JSON logging driver configuration

With the default configuration, docker does not rotate JSON log files, with time they can become large and consume all disk space. That is why we specify max-size and max-file with the default configurations. See Configure and troubleshoot the Docker daemon for more details.

Install Collectord

With the default configuration you will start getting your logs in S3, automatically partitioned by the cluster name, host, namespace, workload, pod name, container name and the date (daily partitions). With the YAML file below we deploy two workloads, one is the daemonset that runs on each node to collect container logs, host logs and application logs, and second one is a deployment, that forwards Kubernetes events.

Deploying Collectord on Kubernetes

Copy following YAML file and save it as collectord-s3.yaml

apiVersion: v1
kind: Namespace
metadata:
  labels:
    app: collectord-s3
  name: collectord-s3
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: collectord-s3
  name: collectord-s3
  namespace: collectord-s3
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app: collectord-s3
  name: collectord-s3
rules:
- apiGroups: ['extensions']
  resources: ['podsecuritypolicies']
  verbs:     ['use']
  resourceNames:
  - privileged
- apiGroups:
  - ""
  - apps
  - batch
  - extensions
  - monitoring.coreos.com
  - etcd.database.coreos.com
  - vault.security.coreos.com
  resources:
  - alertmanagers
  - cronjobs
  - daemonsets
  - deployments
  - endpoints
  - events
  - jobs
  - namespaces
  - nodes
  - nodes/proxy
  - pods
  - prometheuses
  - replicasets
  - replicationcontrollers
  - scheduledjobs
  - services
  - statefulsets
  - vaultservices
  - etcdclusters
  verbs:
  - get
  - list
  - watch
- nonResourceURLs:
  - /metrics
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app: collectord-s3
  name: collectord-s3
  namespace: collectord-s3
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: collectord-s3
subjects:
  - kind: ServiceAccount
    name: collectord-s3
    namespace: collectord-s3
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: collectord-s3
  namespace: collectord-s3
  labels:
    app: collectord-s3
data:
  101-general.conf: |
    [general]
    # Review SLA at https://www.outcoldsolutions.com/docs/license-agreement/ and accept the license
    acceptLicense = false
    # Request the trial license with automated form https://www.outcoldsolutions.com/trial/request/
    license = 
    # If you are planning to setup log aggregation for multiple cluster, name the cluster
    fields.cluster = -

    [aws]
    # Specify AWS Region
    region = 

    [output.s3]
    # Specify Bucket Name
    bucket = 
  102-daemonset.conf: |

  103-addon.conf: |

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: collectord-s3
  namespace: collectord-s3
  labels:
    app: collectord-s3
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      daemon: collectord-s3
  template:
    metadata:
      name: collectord-s3
      labels:
        daemon: collectord-s3
    spec:
      dnsPolicy: ClusterFirstWithHostNet
      hostNetwork: true
      serviceAccountName: collectord-s3
      tolerations:
      - operator: "Exists"
        effect: "NoSchedule"
      - operator: "Exists"
        effect: "NoExecute"
      containers:
      - name: collectord-s3
        # Collector version
        image: outcoldsolutions/collectord:6.0.301
        imagePullPolicy: Always
        securityContext:
          runAsUser: 0
        resources:
          limits:
            cpu: 2
            memory: 256Mi
          requests:
            cpu: 200m
            memory: 64Mi
        env:
        - name: COLLECTORDCONFIG_PATH
          value: /config/s3/kubernetes/daemonset/
        - name: KUBERNETES_NODENAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        volumeMounts:
        # Working folder, where collectord writes log files before uploading to S3
        - name: collectord-s3-var
          mountPath: /var/lib/collectord-s3/var/
        # We store state in /data folder (file positions)
        - name: collectord-s3-state
          mountPath: /var/lib/collectord-s3/data/
        # Configuration file deployed with ConfigMap
        - name: collectord-s3-config
          mountPath: /config/s3/kubernetes/daemonset/user
          readOnly: true
        - name: collectord-s3-secret
          mountPath: /config/s3/kubernetes/daemonset/secret
          readOnly: true
        # Location of docker root (for container logs and metadata)
        - name: docker-root
          mountPath: /rootfs/var/lib/docker/
          readOnly: true
        # Host logs location (including CRI-O logs)
        - name: logs
          mountPath: /rootfs/var/log/
          readOnly: true
        # Docker socket
        - name: docker-unix-socket
          mountPath: /rootfs/var/run/docker.sock
          readOnly: true
        # CRI-O socket (if using CRI-O runtime)
        - name: crio-unix-socket
          mountPath: /rootfs/var/run/crio/
          readOnly: true
        # Application logs
        - name: volumes-root
          mountPath: /rootfs/var/lib/kubelet/
          readOnly: true
        # correct timezone
        - name: localtime
          mountPath: /etc/localtime
          readOnly: true
      volumes:
      # We store state directly on host, change this location, if
      # your persistent volume is somewhere else
      - name: collectord-s3-state
        hostPath:
          path: /var/lib/collectord-s3/data/
      # Working folder, where collectord writes log files before uploading to S3
      - name: collectord-s3-var
        hostPath:
          path: /var/lib/collectord-s3/var/
      # Location of docker root (for container logs and metadata)
      - name: docker-root
        hostPath:
          path: /var/lib/docker/
      # Host logs location (including CRI-O logs)
      - name: logs
        hostPath:
          path: /var/log
      # Docker socket
      - name: docker-unix-socket
        hostPath:
          path: /var/run/docker.sock
      # CRI-O socket (if using CRI-O runtime)
      - name: crio-unix-socket
        hostPath:
          path: /var/run/crio/
      # Location for kubelet mounts, to autodiscover application logs
      - name: volumes-root
        hostPath:
          path: /var/lib/kubelet/
      # correct timezone
      - name: localtime
        hostPath:
          path: /etc/localtime
      - name: collectord-s3-secret
        secret:
          secretName: collectord-s3
      # configuration from ConfigMap
      - name: collectord-s3-config
        configMap:
          name: collectord-s3
          items:
          - key: 101-general.conf
            path: 101-general.conf
          - key: 102-daemonset.conf
            path: 102-daemonset.conf
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: collectord-s3-addon
  namespace: collectord-s3
  labels:
    app: collectord-s3
spec:
  replicas: 1

  selector:
    matchLabels:
      daemon: collectord-s3

  template:
    metadata:
      name: collectord-s3-addon
      labels:
        daemon: collectord-s3
    spec:
      serviceAccountName: collectord-s3
      containers:
      - name: collectord-s3
        image: outcoldsolutions/collectord:6.0.301
        imagePullPolicy: Always
        securityContext:
          runAsUser: 0
          privileged: true
        resources:
          limits:
            cpu: 500m
            memory: 256Mi
          requests:
            cpu: 50m
            memory: 64Mi
        env:
        - name: COLLECTORDCONFIG_PATH
          value: /config/s3/kubernetes/addon
        - name: KUBERNETES_NODENAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        volumeMounts:
        - name: collectord-s3-state
          mountPath: /var/lib/collectord-s3/data/
        - name: collectord-s3-var
          mountPath: /var/lib/collectord-s3/var/
        - name: collectord-s3-config
          mountPath: /config/s3/kubernetes/addon/user
          readOnly: true
        - name: collectord-s3-secret
          mountPath: /config/s3/kubernetes/addon/secret
          readOnly: true
      volumes:
      - name: collectord-s3-state
        hostPath:
          path: /var/lib/collectord-s3/data/
      - name: collectord-s3-var
        emptyDir: {}
      - name: collectord-s3-secret
        secret:
          secretName: collectord-s3
      - name: collectord-s3-config
        configMap:
          name: collectord-s3
          items:
          - key: 101-general.conf
            path: 101-general.conf
          - key: 103-addon.conf
            path: 103-addon.conf

In the ConfigMap replace the values

[general]
# Review SLA at https://www.outcoldsolutions.com/docs/license-agreement/ and accept the license
acceptLicense = false
# Request the trial license with automated form https://www.outcoldsolutions.com/trial/request/
license = 
# If you are planning to setup log aggregation for multiple cluster, name the cluster
fields.cluster = -

[aws]
# Specify AWS Region
region = 

[output.s3]
# Specify Bucket Name
bucket = 

For example (request a trial license at https://www.outcoldsolutions.com/trial/request/)

[general]
# Review SLA at https://www.outcoldsolutions.com/docs/license-agreement/ and accept the license
acceptLicense = true
# Request the trial license with automated form https://www.outcoldsolutions.com/trial/request/
license = Qkc1MTgzUTQ0SUUyTTowOjoz....
# If you are planning to setup log aggregation for multiple cluster, name the cluster
fields.cluster = test

[aws]
# Specify AWS Region
region = us-west-2

[output.s3]
# Specify Bucket Name
bucket = example.com.logs

Apply the deployment

kubectl apply -f collectord-s3.yaml

Create a secret with Access Key and Secret Key

Create a file 100-general.conf with the following content (replace the AccessKeyId and SecretAccessKey with the values from the previous step

[aws]
accessKeyID = AccessKeyId
secretAccessKey = SecretAccessKey

For example

[aws]
accessKeyID = AKIAI7SNNYYAV456WBVQ
secretAccessKey = CKnnNl8DqPrqeudQxV2vPgXga6BCR0y4RMrlirmC

Create the secret

kubectl create secret generic collectord-s3 --from-file=./100-general.conf --namespace collectord-s3

Verify that Pods are running

With the following command verify the pods are running

kubectl get pods -n collectord-s3

You should see an output of the pods successfully running inside collectord-s3 namespace.

By default collectord forwards data every 10 minutes, if this is a freshly created cluster, it can take up to 10 minutes to see the data on S3. You can query it now with Athena.

  • Installation
    • Setup centralized Logging in 5 minutes.
    • Automatically forward host, container and application logs.
    • Test our solution with the 30 days evaluation license.
  • AWS Glue Catalog
    • Table definitions in Glue Catalog.
  • Querying data with Athena
    • Query automatically partitioned data with AWS Athena.
    • Best practices to work with Athena.
    • Query examples for container_logs, events and host_logs.
  • QuickSight for Dashboards and Reports
    • Connecting AWS QuickSight with the Athena.
    • Building dashboards.
  • Access control
    • Limit access to the data with IAM Policy.
  • Annotations
    • Forwarding application logs.
    • Multi-line container logs.
    • Fields extraction for application and container logs (including timestamp extractions).
    • Hiding sensitive data, stripping terminal escape codes and colors.
  • Configuration
    • Advanced configurations for collectord.
  • Troubleshooting
    • Troubleshooting steps.
    • Verify configuration.

About Outcold Solutions

Outcold Solutions provides solutions for building centralized logging infrastructure and monitoring Kubernetes, OpenShift and Docker clusters. We provide easy to setup centralized logging infrastructure with AWS services. We offer Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers.