This repo is now archieved. Please redirect to https://github.com/awslabs/amazon-eks-ami/tree/master/log-collector-script

EKS Logs Collector

This project was created to collect Amazon EKS log files and OS logs for troubleshooting Amazon EKS customer support cases.

Usage

• Collect EKS logs using SSM agent, jump to below section (or)

• Run this project as the root user:

curl -O https://raw.githubusercontent.com/nithu0115/eks-logs-collector/master/eks-log-collector.sh
sudo bash eks-log-collector.sh

Confirm if the tarball file was successfully created (it can be .tgz or .tar.gz)

Retrieving the logs

Download the tarball using your favourite Secure Copy tool.

Example output

The project can be used in normal or enable_debug(Caution: enable_debug will prompt to confirm if we can restart Docker daemon which would kill running containers).

# sudo bash eks-log-collector.sh --help
USAGE: eks-log-collector --help [ --mode=collect|enable_debug --ignore_introspection=true|false --ignore_metrics=true|false ]

OPTIONS:
   --mode  Has two parameters  1) collect or 2) enable_debug,:
             collect        Gathers basic operating system, Docker daemon, and
                            Amazon EKS related config files and logs. This is the default mode.
             enable_debug   Enables debug mode for the Docker daemon(Not for production use)

   --ignore_introspection   To ignore introspection of IPAMD; Pass this flag if DISABLE_INTROSPECTION is enabled on CNI
   
   --ignore_metrics         To ignore prometheus metrics collection; Pass this flag if DISABLE_METRICS enabled on CNI

   --help  Show this help message.

Example to Ignore IPAMD introspection: 
sudo bash eks-log-collector.sh --ignore_introspection=true

Example to Ignore IPAMD Prometheus metrics collection:  
sudo bash eks-log-collector.sh --ignore_metrics=true

Example to Ignore IPAMD introspection and Prometheus metrics collection:
sudo bash eks-log-collector.sh --ignore_introspection=true --ignore_metrics=true   

Example output in normal mode

The following output shows this project running in normal mode.

sudo bash eks-log-collector.sh

	This is version 0.5.0. New versions can be found at https://github.com/awslabs/amazon-eks-ami

Trying to collect common operating system logs... 
Trying to collect kernel logs... 
Trying to collect mount points and volume information... 
Trying to collect SELinux status... 
Trying to collect iptables information... 
Trying to collect installed packages... 
Trying to collect active system services... 
Trying to collect Docker daemon information... 
Trying to collect kubelet information... 
Trying to collect L-IPAMD information... 
Trying to collect sysctls information... 
Trying to collect networking infomation... 
Trying to collect CNI configuration information... 
Trying to collect running Docker containers and gather container data... 
Trying to collect Docker daemon logs... 
Trying to archive gathered information... 

	Done... your bundled logs are located in /opt/log-collector/eks_i-0717c9d54b6cfaa19_2019-02-02_0103-UTC_0.0.4.tar.gz

Collect EKS logs using SSM agent

To run EKS log collector script on Worker Node(s) and upload the bundle(tar) to a S3 Bucket using SSM agent, please follow below steps

Prerequisites:

• Configure AWS CLI on the system where you will run the below commands. The IAM entity (User/Role) should have permissions to run/invoke aws ssm send-command and get-command-invocation commands.

• SSM agent should be installed and running on Worker Node(s). How to Install SSM Agent link

• Worker Node(s) should have required permissions to communicate with SSM service. IAM managed role AmazonEC2RoleforSSM will have all the required permission for SSM agent to run on EC2 instances. The IAM managed role AmazonEC2RoleforSSM has S3:PutObject permission to all S3 resources.

        Note: For more granular control of the IAM permission check AWS Systems Manager Permissions link

• A S3 bucket location is required which is taken as an input parameter to aws ssm send-command command, to which the logs should be pushed.

To invoke SSM agent to run EKS log collector script and push bundle to S3 from Worker Node(s):

1. Create the SSM document named "EKSLogCollector" using the following command:
   

aws ssm create-document --name "EKSLogCollector" --document-type "Command" --content https://raw.githubusercontent.com/nithu0115/eks-logs-collector/master/eks-ssm-content.json

2. To execute the bash script in the SSM document and to collect the logs from worker, run the following command:
   

aws ssm send-command --instance-ids <EC2 Instance ID> --document-name "EKSLogCollector" --parameters "bucketName=<S3 bucket name to push the logs>" --output json

3. To check the status of SSM command submitted in previous step use the command
   

aws ssm get-command-invocation --command-id "<SSM command ID>" --instance-id "<EC2 Instance ID>" --output text

    SSM command IDOne of the response parameters after running aws ssm send-command in step2
    EC2 Instance IDThe EC2 Instance ID provided in the aws ssm send-command in step2

4. Once the above command is executed successfully, the logs should be present in the S3 bucket specified in the previous step.