{"id":266,"date":"2023-03-20T13:00:09","date_gmt":"2023-03-20T13:00:09","guid":{"rendered":"https:\/\/keyshell.net\/blog\/?p=266"},"modified":"2023-03-20T13:00:09","modified_gmt":"2023-03-20T13:00:09","slug":"setup-efk-stack-on-amazon-eks-cluster","status":"publish","type":"post","link":"https:\/\/keyshell.net\/blog\/2023\/03\/20\/setup-efk-stack-on-amazon-eks-cluster\/","title":{"rendered":"Setup EFK Stack on Amazon EKS cluster"},"content":{"rendered":"

About EFK<\/strong><\/h1>\n

When it comes to Kubernetes in production environment, logging has its important role to play. It help to understand where the problem is and what went wrong.<\/p>\n

EFK is used for log streaming, log analysis, and log monitoring. It is a combination of 3 components.<\/p>\n

1. Elasticsearch \u2014 logging backend for storing, searching and analyzing log data.<\/p>\n

2. Fluentd \u2014 logging agent which continuously streams log data to the logging backend.<\/p>\n

3. Kibana \u2014 A tool to visualize log data in the form of dashboards.<\/p>\n

Here, we are using K8s cluster created using AWS EKS. It has 4 nodes with type \u2018t3.medium\u2019 and AMI \u2018amazon linux 2\u2019. Also, add-ons used while setting up the cluster were \u2018kube-proxy\u2019, \u2018vpc-cni\u2019, \u2018coredns\u2019 and \u2018aws-ebs-csi-driver\u2019.<\/p>\n

Also, we have to create persistent volume for Elasticsearch. In that case, \u2018Amazon EBS CSI driver IAM role\u2019 need to be created. For that , please refer the AWS official documentation page:<\/p>\n

https:\/\/docs.aws.amazon.com\/eks\/latest\/userguide\/csi-iam-role.html<\/a><\/p>\n

(You may skip step 11 if KMS Key Encryption is not required).<\/p>\n

After setting up the EKS cluster, ssh to the Bastion host and install aws cli version 2. Configure the AWS credentials and add the access key ID and secret key using,<\/p>\n

aws configure<\/em><\/strong><\/p>\n

Add the cluster as a new context using the following command.<\/p>\n

aws eks \u2014 region=<region> update-kubeconfig \u2014 name <cluster_name><\/em><\/strong><\/p>\n

eg:<\/p>\n

aws eks \u2014 region=us-east-1 update-kubeconfig \u2014 name eks-cluster<\/p>\n

Added new context arn:aws:eks:us-east-1:103423222380:cluster\/eks-cluster to \/root\/.kube\/config<\/p>\n

Install kubectl so that we can manage out cluster from the Bastion host.<\/p>\n

kubectl get<\/span> nodes\r\nNAME STATUS ROLES AGE VERSION\r\nip-10<\/span>\u20130<\/span>\u2013129<\/span>\u2013220.<\/span>ec2.internal<\/span> Ready <none> 27<\/span>m v1.24<\/span>.10<\/span>-eks-48e63<\/span>af\r\nip-10<\/span>\u20130<\/span>\u2013149<\/span>\u201355.<\/span>ec2.internal<\/span> Ready <none> 26<\/span>m v1.24<\/span>.10<\/span>-eks-48e63<\/span>af\r\nip-10<\/span>\u20130<\/span>\u2013190<\/span>\u2013100.<\/span>ec2.internal<\/span> Ready <none> 30<\/span>m v1.24<\/span>.10<\/span>-eks-48e63<\/span>af\r\nip-10<\/span>\u20130<\/span>\u2013226<\/span>\u2013108.<\/span>ec2.internal<\/span> Ready <none> 30<\/span>m v1.24<\/span>.10<\/span>-eks-48e63<\/span>af\r\n\r\n<\/pre>\n
Setup EFK Stack<\/strong><\/h1>\n
\"\"<\/p>\n
Elasticsearch as a Statefulset<\/strong><\/h1>\n
Elasticsearch is deployed as a\u00a0Statefulset<\/strong>\u00a0and the multiple replicas connect with each other using a headless service (svc). The headless svc helps in the DNS domain of the pods.<\/p>\n
The statefulset creates the\u00a0Persistent Volume Claim (PVC)<\/strong>\u00a0with the default available storage class. If you have a custom storage class for PVC, you can add it in the volumeClaimTemplates by uncommenting the\u00a0storageClassName<\/strong>\u00a0parameter.<\/p>\n
Following are the manifests for statefulset and service.<\/p>\n
apiVersion: apps\/v1<\/em>\r\n\r\nkind: StatefulSet<\/em>\r\n\r\nmetadata:<\/em>\r\n\r\n\u00a0 name: es-cluster<\/em>\r\n\r\nspec:<\/em>\r\n\r\n\u00a0 serviceName: elasticsearch<\/em>\r\n\r\n\u00a0 replicas: 3<\/em>\r\n\r\n\u00a0 selector:<\/em>\r\n\r\n\u00a0\u00a0\u00a0 matchLabels:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 app: elasticsearch<\/em>\r\n\r\n\u00a0 template:<\/em>\r\n\r\n\u00a0\u00a0\u00a0 metadata:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 labels:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 app: elasticsearch<\/em>\r\n\r\n\u00a0\u00a0\u00a0 spec:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 containers:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 - name: elasticsearch<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 image: docker.elastic.co\/elasticsearch\/elasticsearch:7.5.0<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 resources:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 limits:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 cpu: 1000m<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 requests:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 cpu: 100m<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ports:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - containerPort: 9200<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 name: rest<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 protocol: TCP<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - containerPort: 9300<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 name: inter-node<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 protocol: TCP<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 volumeMounts:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - name: data<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 mountPath: \/usr\/share\/elasticsearch\/data<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 env:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - name: cluster.name<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 value: k8s-logs<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - name: node.name<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 valueFrom:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 fieldRef:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 fieldPath: metadata.name<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - name: discovery.seed_hosts<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 value: \"es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch\"<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - name: cluster.initial_master_nodes<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 value: \"es-cluster-0,es-cluster-1,es-cluster-2\"<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - name: ES_JAVA_OPTS<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 value: \"-Xms512m -Xmx512m\"<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 initContainers:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 - name: fix-permissions<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 image: busybox<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 command: [\"sh\", \"-c\", \"chown -R 1000:1000 \/usr\/share\/elasticsearch\/data\"]<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 securityContext:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 privileged: true<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 volumeMounts:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - name: data<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 mountPath: \/usr\/share\/elasticsearch\/data<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 - name: increase-vm-max-map<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 image: busybox<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 command: [\"sysctl\", \"-w\", \"vm.max_map_count=262144\"]<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 securityContext:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 privileged: true<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 - name: increase-fd-ulimit<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 image: busybox<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 command: [\"sh\", \"-c\", \"ulimit -n 65536\"]<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 securityContext:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 privileged: true<\/em>\r\n\r\n\u00a0 volumeClaimTemplates:<\/em>\r\n\r\n\u00a0 - metadata:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 name: data<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 labels:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 app: elasticsearch<\/em>\r\n\r\n\u00a0\u00a0\u00a0 spec:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 accessModes: [ \"ReadWriteOnce\" ]<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 # storageClassName: \"\"<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 resources:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 requests:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 storage: 3Gi<\/em><\/pre>\n
kubectl create -f es-sts.yaml<\/em><\/strong><\/p>\n
 <\/p>\n
apiVersion: v1<\/em>\r\n\r\nkind: Service<\/em>\r\n\r\nmetadata:<\/em>\r\n\r\n\u00a0 name: elasticsearch<\/em>\r\n\r\n\u00a0 labels:<\/em>\r\n\r\n\u00a0\u00a0 \u00a0app: elasticsearch<\/em>\r\n\r\nspec:<\/em>\r\n\r\n\u00a0 selector:<\/em>\r\n\r\n\u00a0\u00a0\u00a0 app: elasticsearch<\/em>\r\n\r\n\u00a0 clusterIP: None<\/em>\r\n\r\n\u00a0 ports:<\/em>\r\n\r\n\u00a0\u00a0\u00a0 - port: 9200<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 name: rest<\/em>\r\n\r\n\u00a0\u00a0\u00a0 - port: 9300<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 name: inter-node<\/em><\/pre>\n
kubectl create -f es-svc.yaml<\/em><\/strong><\/p>\n
 <\/p>\n
You can check the PVC status using,<\/p>\n
kubectl get<\/span> pvc\r\n\r\nNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\r\ndata<\/span>-es-cluster-0<\/span> Bound pvc-fefd5503\u201372e9<\/span>\u201348ed-8ebb-053c45fe372f 3Gi RWO gp2 24h\r\ndata<\/span>-es-cluster-1<\/span> Bound pvc-a3c272a1\u20137135<\/span>\u201340dc-a188\u201387fdf1804550 3Gi RWO gp2 24h\r\ndata<\/span>-es-cluster-2<\/span> Bound pvc-837d2edb-159a-4de1\u20138d14\u20135b8fbdb67237 3Gi RWO gp2 24h\r\n\r\nOnce the Elasticsearch pods come into running status,<\/pre>\n
kubectl get<\/span> pods\r\nNAME READY STATUS RESTARTS AGE\r\nes-<\/span>cluster-0<\/span> 1<\/span>\/<\/span>1<\/span> Running<\/span> 0<\/span> 20<\/span>h\r\nes-<\/span>cluster-1<\/span> 1<\/span>\/<\/span>1<\/span> Running<\/span> 0<\/span> 20<\/span>h\r\nes-<\/span>cluster-2<\/span> 1<\/span>\/<\/span>1<\/span> Running<\/span> 0<\/span> 20<\/span>h\r\n\r\n<\/pre>\n
Port-forward it using,<\/p>\n
kubectl port-forward es-cluster-0 9200:9200<\/em><\/strong><\/p>\n
To check the health of the Elasticsearch cluster, run the following command in the terminal which will give an output as shown.<\/p>\n
curl\u00a0<\/em><\/strong>http:\/\/localhost:9200\/_cluster\/health\/?pretty<\/em><\/strong><\/a><\/p>\n
{<\/span>\r\n\"cluster_name\"<\/span> :<\/span> \"k8s-logs\"<\/span>,<\/span>\r\n\"status\"<\/span> :<\/span> \"green\"<\/span>,<\/span>\r\n\"timed_out\"<\/span> :<\/span> false<\/span><\/span>,<\/span>\r\n\"number_of_nodes\"<\/span> :<\/span> 3<\/span>,<\/span>\r\n\"number_of_data_nodes\"<\/span> :<\/span> 3<\/span>,<\/span>\r\n\"active_primary_shards\"<\/span> :<\/span> 0<\/span>,<\/span>\r\n\"active_shards\"<\/span> :<\/span> 0<\/span>,<\/span>\r\n\"relocating_shards\"<\/span> :<\/span> 0<\/span>,<\/span>\r\n\"initializing_shards\"<\/span> :<\/span> 0<\/span>,<\/span>\r\n\"unassigned_shards\"<\/span> :<\/span> 0<\/span>,<\/span>\r\n\"delayed_unassigned_shards\"<\/span> :<\/span> 0<\/span>,<\/span>\r\n\"number_of_pending_tasks\"<\/span> :<\/span> 0<\/span>,<\/span>\r\n\"number_of_in_flight_fetch\"<\/span> :<\/span> 0<\/span>,<\/span>\r\n\"task_max_waiting_in_queue_millis\"<\/span> :<\/span> 0<\/span>,<\/span>\r\n\"active_shards_percent_as_number\"<\/span> :<\/span> 100.0<\/span>\r\n}<\/span><\/pre>\n
Kibana Deployment & Service<\/strong><\/h1>\n
Create the Kibana deployment manifest as kibana-deployment.yaml.<\/p>\n
apiVersion: apps\/v1<\/em>\r\n\r\nkind: Deployment<\/em>\r\n\r\nmetadata:<\/em>\r\n\r\n\u00a0 name: kibana<\/em>\r\n\r\n\u00a0 labels:<\/em>\r\n\r\n\u00a0\u00a0\u00a0 app: kibana<\/em>\r\n\r\nspec:<\/em>\r\n\r\n\u00a0 replicas: 1<\/em>\r\n\r\n\u00a0 selector:<\/em>\r\n\r\n\u00a0\u00a0\u00a0 matchLabels:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 app: kibana<\/em>\r\n\r\n\u00a0 template:<\/em>\r\n\r\n\u00a0\u00a0\u00a0 metadata:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 labels:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 app: kibana<\/em>\r\n\r\n\u00a0\u00a0\u00a0 spec:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 containers:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 - name: kibana<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 image: docker.elastic.co\/kibana\/kibana:7.5.0<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 resources:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 limits:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 cpu: 1000m<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 requests:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 cpu: 100m<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 env:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - name: ELASTICSEARCH_URL<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 value: http:\/\/elasticsearch:9200<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ports:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - containerPort: 5601<\/em><\/pre>\n
Create the manifest using,<\/p>\n
kubectl create -f kibana-deployment.yaml<\/em><\/strong><\/p>\n
Here, we are using a service type \u2018LoadBalancer\u2019 to access the Kibana UI.<\/p>\n
apiVersion: v1<\/em>\r\n\r\nkind: Service<\/em>\r\n\r\nmetadata:<\/em>\r\n\r\n\u00a0 name: kibana-np<\/em>\r\n\r\nspec:<\/em>\r\n\r\n\u00a0 selector:<\/em>\r\n\r\n\u00a0\u00a0\u00a0 app: kibana<\/em>\r\n\r\n\u00a0 type: LoadBalancer<\/em>\r\n\r\n\u00a0 ports:<\/em>\r\n\r\n\u00a0\u00a0\u00a0 - port: 8080<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 targetPort: 5601<\/em><\/pre>\n
Create the kibana-svc now.<\/p>\n
kubectl create -f kibana-svc.yaml<\/em><\/strong><\/p>\n
Check if the kibana deployment and pod are running using,<\/p>\n
kubectl get<\/span> deployment\r\nNAME READY UP-TO<\/span>-DATE<\/span> AVAILABLE AGE\r\nkibana 1<\/span>\/1<\/span> 1<\/span> 1<\/span> 24<\/span>h\r\n\r\n\r\nkubectl get<\/span> pods\r\nNAME READY STATUS RESTARTS AGE\r\nes-<\/span>cluster-0<\/span> 1<\/span>\/<\/span>1<\/span> Running<\/span> 0<\/span> 24<\/span>h\r\nes-<\/span>cluster-1<\/span> 1<\/span>\/<\/span>1<\/span> Running<\/span> 0<\/span> 24<\/span>h\r\nes-<\/span>cluster-2<\/span> 1<\/span>\/<\/span>1<\/span> Running<\/span> 0<\/span> 24<\/span>h\r\nkibana-6<\/span>db5f8d7c8-<\/span>zxjtf 1<\/span>\/<\/span>1<\/span> Running<\/span> 0<\/span> 3<\/span>h30m\r\n\r\n<\/pre>\n
Create the kibana-svc now.<\/p>\n
kubectl create -f kibana-svc.yaml<\/em><\/strong><\/p>\n
Check if the kibana deployment and pod are running using,<\/p>\n
Check using the load balancer endpoint,<\/p>\n
Fluentd Daemon set<\/strong><\/h1>\n
For Fluentd, we require a\u00a0ClusterRole<\/strong>, to give permissions for pods and namespaces.<\/p>\n
Create a manifest fluentd-role.yaml<\/p>\n
apiVersion: rbac.authorization.k8s.io\/v1<\/em>\r\n\r\nkind: ClusterRole<\/em>\r\n\r\nmetadata:<\/em>\r\n\r\n\u00a0 name: fluentd<\/em>\r\n\r\n\u00a0 labels:<\/em>\r\n\r\n\u00a0\u00a0\u00a0 app: fluentd<\/em>\r\n\r\nrules:<\/em>\r\n\r\n- apiGroups:<\/em>\r\n\r\n\u00a0\u00a0- \"\"<\/em>\r\n\r\n\u00a0 resources:<\/em>\r\n\r\n\u00a0 - pods<\/em>\r\n\r\n\u00a0 - namespaces<\/em>\r\n\r\n\u00a0 verbs:<\/em>\r\n\r\n\u00a0 - get<\/em>\r\n\r\n\u00a0 - list<\/em>\r\n\r\n\u00a0 - watch<\/em><\/pre>\n
Apply the manifest<\/p>\n
kubectl create -f fluentd-role.yaml<\/em><\/strong><\/p>\n
Next, is the service account fluentd-sa.yaml.<\/p>\n
apiVersion: v1<\/em>\r\n\r\nkind: ServiceAccount<\/em>\r\n\r\nmetadata:<\/em>\r\n\r\n\u00a0 name: fluentd<\/em>\r\n\r\n\u00a0 labels:<\/em>\r\n\r\n\u00a0\u00a0\u00a0 app: fluentd<\/em>\r\n\r\n<\/pre>\n
Apply the manifest<\/p>\n
kubectl create -f fluentd-sa.yaml<\/em><\/strong><\/p>\n
Now, we have to bind the role with service account. Create a manifest fluentd-rb.yaml.<\/p>\n
kind: ClusterRoleBinding<\/em>\r\n\r\napiVersion: rbac.authorization.k8s.io\/v1<\/em>\r\n\r\nmetadata:<\/em>\r\n\r\n\u00a0 name: fluentd<\/em>\r\n\r\nroleRef:<\/em>\r\n\r\n\u00a0 kind: ClusterRole<\/em>\r\n\r\n\u00a0 name: fluentd<\/em>\r\n\r\n\u00a0 apiGroup: rbac.authorization.k8s.io<\/em>\r\n\r\nsubjects:<\/em>\r\n\r\n- kind: ServiceAccount<\/em>\r\n\r\n\u00a0 name: fluentd<\/em>\r\n\r\n\u00a0 namespace: default<\/em><\/pre>\n
 <\/p>\n
Apply the manifest<\/p>\n
kubectl create -f fluentd-rb.yaml<\/em><\/strong><\/p>\n
We are deploying fluentd as a daemon set so that it runs in all the nodes and collect the container logs. Create fluentd-ds.yaml.<\/p>\n
apiVersion: apps\/v1<\/em>\r\n\r\nkind: DaemonSet<\/em>\r\n\r\nmetadata:<\/em>\r\n\r\n\u00a0 name: fluentd<\/em>\r\n\r\n\u00a0 labels:<\/em>\r\n\r\n\u00a0\u00a0\u00a0 app: fluentd<\/em>\r\n\r\nspec:<\/em>\r\n\r\n\u00a0 selector:<\/em>\r\n\r\n\u00a0\u00a0\u00a0 matchLabels:<\/em>\r\n\r\n\u00a0 \u00a0\u00a0\u00a0\u00a0app: fluentd<\/em>\r\n\r\n\u00a0 template:<\/em>\r\n\r\n\u00a0\u00a0\u00a0 metadata:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 labels:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 app: fluentd<\/em>\r\n\r\n\u00a0\u00a0\u00a0 spec:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 serviceAccount: fluentd<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 serviceAccountName: fluentd<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 containers:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 - name: fluentd<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 image: fluent\/fluentd-kubernetes-daemonset:v1.4.2-debian-elasticsearch-1.1<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 env:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - name:\u00a0 FLUENT_ELASTICSEARCH_HOST<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 value: \"elasticsearch.default.svc.cluster.local\"<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - name:\u00a0 FLUENT_ELASTICSEARCH_PORT<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 value: \"9200\"<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - name: FLUENT_ELASTICSEARCH_SCHEME<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 value: \"http\"<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - name: FLUENTD_SYSTEMD_CONF<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 value: disable<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 resources:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 limits:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 memory: 512Mi<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 requests:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 cpu: 100m<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 memory: 200Mi<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 volumeMounts:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - name: varlog<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 mountPath: \/var\/log<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - name: varlibdockercontainers<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 mountPath: \/var\/lib\/docker\/containers<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 readOnly: true<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 terminationGracePeriodSeconds: 30<\/em>\r\n\r\n\u00a0\u00a0 \u00a0\u00a0\u00a0volumes:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 - name: varlog<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 hostPath:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 path: \/var\/log<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 - name: varlibdockercontainers<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 hostPath:<\/em>\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 path: \/var\/lib\/docker\/containers<\/em><\/pre>\n
Apply the fluentd manifest<\/p>\n
kubectl create -f fluentd-ds.yaml<\/em><\/strong><\/p>\n
kubectl get<\/span> pods\r\nNAME READY STATUS RESTARTS AGE\r\nes-<\/span>cluster-0<\/span> 1<\/span>\/<\/span>1<\/span> Running<\/span> 0<\/span> 24<\/span>h\r\nes-<\/span>cluster-1<\/span> 1<\/span>\/<\/span>1<\/span> Running<\/span> 0<\/span> 24<\/span>h\r\nes-<\/span>cluster-2<\/span> 1<\/span>\/<\/span>1<\/span> Running<\/span> 0<\/span> 24<\/span>h\r\nfluentd-<\/span>d49sw 1<\/span>\/<\/span>1<\/span> Running<\/span> 0<\/span> 3<\/span>h30m\r\nfluentd-<\/span>pkh2l 1<\/span>\/<\/span>1<\/span> Running<\/span> 0<\/span> 3<\/span>h30m\r\nfluentd-<\/span>qd6f6 1<\/span>\/<\/span>1<\/span> Running<\/span> 0<\/span> 3<\/span>h31m\r\nfluentd-<\/span>rvdvx 1<\/span>\/<\/span>1<\/span> Running<\/span> 0<\/span> 3<\/span>h30m\r\nkibana-6<\/span>db5f8d7c8-<\/span>zxjtf 1<\/span>\/<\/span>1<\/span> Running<\/span> 0<\/span> 3<\/span>h30m\r\n\r\n<\/pre>\n
Now, the EFK setup is completed.<\/p>\n
Test Pod<\/strong><\/h1>\n
Save the following as test-pod.yaml<\/p>\n
apiVersion: v1<\/em>\r\n\r\nkind: Pod<\/em>\r\n\r\nmetadata:<\/em>\r\n\r\n\u00a0 name: counter<\/em>\r\n\r\nspec:<\/em>\r\n\r\n\u00a0 containers:<\/em>\r\n\r\n\u00a0 - name: count<\/em>\r\n\r\n\u00a0\u00a0\u00a0 image: busybox<\/em>\r\n\r\n\u00a0\u00a0\u00a0 args: [\/bin\/sh, -c,'i=0; while true; do echo \"Thanks for visiting devopscube! $i\"; i=$((i+1)); sleep 1; done']<\/em>\r\n\r\n<\/pre>\n
Apply the manifest<\/p>\n
kubectl create -f test-pod.yaml<\/em><\/strong><\/p>\n
Kibana Dashboard<\/strong><\/h1>\n
Access the Kibana dashboard using the load balancer endpoint along with post 8080.<\/p>\n
\"\"<\/p>\n
Click on the\u00a0management<\/strong>\u00a0icon at the bottom left corner of the page. There, under Kibana, select the option \u2018Index Patterns<\/strong>\u2019.<\/p>\n
Create a new Index Patten using the pattern \u2014 \u201clogstash-*\u201d, click next and select \u201c@timestamp\u201d in the timestamps option. Create index pattern.<\/p>\n
Move to the console option at the top left of the page where you can see the logs from our test pod.<\/p>\n","protected":false},"excerpt":{"rendered":"
About EFK When it comes to Kubernetes in production environment, logging has its important role to play. It help to understand where the problem is and what went wrong. EFK is used for log streaming, log analysis, and log monitoring. It is a combination of 3 components. 1. Elasticsearch \u2014 logging backend for storing, searching […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"pagelayer_contact_templates":[],"_pagelayer_content":"","footnotes":""},"categories":[1],"tags":[15,14,16,17],"class_list":["post-266","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-efk","tag-eks","tag-k8s","tag-kibana"],"_links":{"self":[{"href":"https:\/\/keyshell.net\/blog\/wp-json\/wp\/v2\/posts\/266","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/keyshell.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/keyshell.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/keyshell.net\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/keyshell.net\/blog\/wp-json\/wp\/v2\/comments?post=266"}],"version-history":[{"count":9,"href":"https:\/\/keyshell.net\/blog\/wp-json\/wp\/v2\/posts\/266\/revisions"}],"predecessor-version":[{"id":280,"href":"https:\/\/keyshell.net\/blog\/wp-json\/wp\/v2\/posts\/266\/revisions\/280"}],"wp:attachment":[{"href":"https:\/\/keyshell.net\/blog\/wp-json\/wp\/v2\/media?parent=266"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/keyshell.net\/blog\/wp-json\/wp\/v2\/categories?post=266"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/keyshell.net\/blog\/wp-json\/wp\/v2\/tags?post=266"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
curl\u00a0<\/em><\/strong>http:\/\/a3f05882d30a740e5a08ce942d84321b-694215210.us-east-1.elb.amazonaws.com:8080\/app\/kibana<\/em><\/strong><\/a><\/p>\n