본문 바로가기
AWS/EKS

[AEWS2] 4-1. EKS Observability

by okms1017 2024. 3. 31.
728x90

✍ Posted by Immersive Builder  Seong
 

1. 실습 환경 배포

이전 실습에서 사용한 원클릭 배포 파일에 AWS LB IRSA, EC2 IAM Role, EBS Addon 및 precmd 설정을 추가하여 리소스를 배포합니다. 
▶ 원클릭 배포 구성 : https://okms1017.tistory.com/35

 

[AEWS2] 2-1. AWS VPC CNI 소개

✍ Posted by Immersive Builder Seong 1. 기본 인프라 배포 AWS CloudFormation 원클릭 배포 AWS CloudFormation을 이용하여 하기 아키텍처와 같이 VPC 1, Public Subnet 3, Private Subnet 3, EKS Cluster(Control Plane), Managed node group(E

okms1017.tistory.com

이번 실습은 배포하는 파드의 수량이 많으므로 Worker Node 인스턴스 타입을 t3.medium → t3.xlarge로 변경하여 진행합니다. 
 

Instance Type (t3.xlarge)

 

# EKS Addon 확인 
$ eksctl get addon --cluster $CLUSTER_NAME
NAME                    VERSION                 STATUS  ISSUES  IAMROLE                                                                        UPDATE AVAILABLE CONFIGURATION VALUES
aws-ebs-csi-driver      v1.29.1-eksbuild.1      ACTIVE  0       arn:aws:iam::732659419746:role/eksctl-myeks-addon-aws-ebs-csi-driver-Role1-6UGS5zEIS6yM
coredns                 v1.10.1-eksbuild.7      ACTIVE  0
kube-proxy              v1.28.6-eksbuild.2      ACTIVE  0
vpc-cni                 v1.17.1-eksbuild.1      ACTIVE  0       arn:aws:iam::732659419746:role/eksctl-myeks-addon-vpc-cni-Role1-N0Tl9d51dbNG   enableNetworkPolicy: "true"

# Storage Class 확인
$ kubectl get sc
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp3             ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   5s

# IRSA 확인
$ eksctl get iamserviceaccount --cluster $CLUSTER_NAME
NAMESPACE       NAME                            ROLE ARN
kube-system     aws-load-balancer-controller    arn:aws:iam::732659419746:role/eksctl-myeks-addon-iamserviceaccount-kube-sys-Role1-f7SomgND5UQe

 
AWS LB는 IRSA 인증을 사용하고, 그 외 나머지 리소스(EBS, CloudWatch, CertManager, ExternalDNS 등)는 간단하게 IAM Role을 사용할 예정입니다. 
 

Managed nodegroup IAM Role

 


2. EKS Observability

EKS Console

AWS Management Console을 사용하여 워크로드, 클러스터, 구성, 권한 부여, 정책, 서비스 등 모든 표준 쿠버네티스 API 리소스 유형을 확인하고 정보를 탐색할 수 있습니다. 
 

EKS Console

 

  • 워크로드 : PodTemplates, Pods, ReplicaSets, Deployments, StatefulSets, DaemonSets, Job, Cronjob, PriorityClasses, HPAs
  • 클러스터 : Nodes, Namespace, APIServices, RuntimeClasses, FlowSchemas, PriorityLevelConfigurations
  • 서비스 및 네트워킹 : Service, Endpoint, EndpointSlices, Ingress, IngressClasses
  • 구성 및 보안 암호 : ConfigMap, Secrets
  • 스토리지 : PersistentVolumeClaims, PersistentVolumes, StorageClasses, VolumeAttachment, CSIDrivers, CSINodes, CSIStorageCapacities
  • 인증 : ServiceAccounts 
  • 권한 부여 : ClusterRoles, ClusterRoleBindings, Roles, RoleBindings 
  • 정책 : LimitRanges, ResourceQuotas, NetworkPolicies, PodDisruptionBudgets 
  • 확장프로그램 : CustomResourceDefinitions, MutatingWebhookConfigurations, ValidatingWebhookConfigurations

 

$ kubectl get ClusterRole | grep eks
eks:addon-manager                                                      2024-03-30T09:58:12Z
eks:az-poller                                                          2024-03-30T09:58:07Z
eks:certificate-controller-approver                                    2024-03-30T09:58:07Z
eks:certificate-controller-signer                                      2024-03-30T09:58:07Z
eks:cloud-controller-manager                                           2024-03-30T09:58:07Z
eks:cloud-provider-extraction-migration                                2024-03-30T09:58:08Z
eks:cloudwatch-agent-role                                              2024-03-30T09:58:07Z
eks:cluster-event-watcher                                              2024-03-30T09:58:07Z
eks:fargate-manager                                                    2024-03-30T09:58:12Z
eks:fargate-scheduler                                                  2024-03-30T09:58:07Z
eks:k8s-metrics                                                        2024-03-30T09:58:07Z
eks:network-policy-controller                                          2024-03-30T09:58:16Z
eks:node-bootstrapper                                                  2024-03-30T09:58:12Z
eks:node-manager                                                       2024-03-30T09:58:12Z
eks:nodewatcher                                                        2024-03-30T09:58:07Z
eks:pod-identity-mutating-webhook                                      2024-03-30T09:58:07Z
eks:service-operations                                                 2024-03-30T09:58:07Z
eks:tagging-controller                                                 2024-03-30T09:58:08Z

 

Logging in EKS

Amazon EKS Control Plane에서 발생하는 로그를 EKS 로깅 옵션을 설정하여 수집할 수 있습니다. 
API Server, Audit, Authenticator, Controller Manager, Scheduler 로그들을 CloudWatch로 직접 전송할 수 있습니다. 
 

  • API Server : 클러스터에 대한 모든 API 요청과 관련된 로그입니다. 
  • Audit : 클러스터에 영향을 준 개별 사용자 및 관리자 또는 시스템 구성요소에 대한 로그입니다. 
  • Authenticator : 클러스터에 대한 인증 요청과 관련된 로그로 Amazon EKS가 IAM 자격증명을 사용하는 쿠버네티스 RBAC(Role-Based Access Control) 인증을 기록합니다. 
  • Controller Manager : 클러스터 컨트롤러 상태와 관련된 로그입니다. 
  • Scheduler : 클러스터에서 파드를 실행하는 시점, 그리고 파드가 생성될 노드의 위치와 관련된 로그입니다. 

 

Control Plane Logging On - 1

 

Control Plane Logging On - 2

AWS CLI 명령어와 추가 옵션을 조합하여 로그를 정제하거나 필터링하여 확인할 수 있습니다. 
 

# Log Group 조회
$ aws logs describe-log-groups
  "logGroups": [
    {
      "logGroupName": "/aws/eks/myeks/cluster",
      "creationTime": 1711798531282,
      "metricFilterCount": 0,
      "arn": "arn:aws:logs:ap-northeast-2:732659419746:log-group:/aws/eks/myeks/cluster:*",
      "storedBytes": 0,
      "logGroupClass": "STANDARD",
      "logGroupArn": "arn:aws:logs:ap-northeast-2:732659419746:log-group:/aws/eks/myeks/cluster"
    }
  ]

# Log Tail
$ aws logs tail /aws/eks/$CLUSTER_NAME/cluster | more

# Log 실시간 출력
$ aws logs tail /aws/eks/$CLUSTER_NAME/cluster --follow
```
2024-03-30T11:55:18.328000+00:00 kube-apiserver-audit-74bf1e85716eaafd8b64c1e251fbf2c8 {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"cce33b0b-81d6-4551-b4a6-c7e39e413e43","stage":"ResponseComplete","requestURI":"/livez?exclude=kms-provider-0","verb":"get","user":{"username":"system:anonymous","groups":["system:unauthenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"Go-http-client/1.1","responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2024-03-30T11:55:17.553555Z","stageTimestamp":"2024-03-30T11:55:17.555833Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:public-info-viewer\" of ClusterRole \"system:public-info-viewer\" to Group \"system:unauthenticated\""}}
2024-03-30T11:55:18.328000+00:00 kube-apiserver-audit-74bf1e85716eaafd8b64c1e251fbf2c8 {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"0b796834-612d-4f43-84e3-208742c0a0fc","stage":"ResponseComplete","requestURI":"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager?timeout=5s","verb":"get","user":{"username":"eks:cloud-controller-manager","groups":["system:authenticated"]},"sourceIPs":["10.0.34.253"],"userAgent":"aws-cloud-controller-manager/v0.0.0 (linux/amd64) kubernetes/$Format/leader-election","objectRef":{"resource":"leases","namespace":"kube-system","name":"cloud-controller-manager","apiGroup":"coordination.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2024-03-30T11:55:18.127288Z","stageTimestamp":"2024-03-30T11:55:18.129949Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"eks:cloud-controller-manager\" of ClusterRole \"eks:cloud-controller-manager\" to User \"eks:cloud-controller-manager\""}}
```

# Log Stream: Controller Manager
$ kubectl scale deployment -n kube-system coredns --replicas=2
$ aws logs tail /aws/eks/$CLUSTER_NAME/cluster --log-stream-name-prefix kube-controller-manager --llow
```
2024-03-30T11:56:54.000000+00:00 kube-controller-manager-0d165a84d3560a18d4be4c4f6932bb5b I0330 11:56:54.234599      10 replica_set.go:676] "Finished syncing" kind="ReplicaSet" key="kube-system/coredns-55474bf7b9" duration="17.81543ms"
2024-03-30T11:56:54.000000+00:00 kube-controller-manager-0d165a84d3560a18d4be4c4f6932bb5b I0330 11:56:54.234723      10 replica_set.go:676] "Finished syncing" kind="ReplicaSet" key="kube-system/coredns-55474bf7b9" duration="86.567µs"
```

 
그리고 CloudWatch 서비스의 로그 그룹(/aws/eks/myeks/cluster)에서 저장된 로그 스트림을 확인할 수 있습니다. 
 

CloudWatch - Log Group
CloudWatch - Log Event

 
* Logs Insights : CloudWatch 대시보드에서 로그 그룹을 선택한 후 쿼리를 실행하여 결과를 화면으로 제공하는 기능입니다. 
 
아래는 EKS 로그 그룹의 kube-apiserver-audit 로그 스트림에서 'userAgent', 'requestURI', '@timestamp', '@message' 4개 필드를 검색하는 쿼리를 실행한 결과 화면입니다.  
 

CloudWatch - Logs Insights

 
* 컨테이너 로깅 
컨테이너 어플리케이션의 로그를 쿠버네티스 명령어로 조회할 수 있는 방법입니다. 
 
하기와 같이 컨테이너 이미지를 빌드할 때 어플리케이션의 로그를 링크로 설정해두면 사용자가 파드 안으로 접속하지 않아도 외부에서 'kubectl logs' 명령어로 로그를 조회할 수 있습니다.  
단, 파드가 종료되면 'kubectl logs'로 더 이상 조회가 불가하며 해당 로그 파일은 '10Mi'를 초과할 수 없습니다. 
 

RUN ln -sf /dev/stdout /opt/bitnami/nginx/logs/access.log
RUN ln -sf /dev/stderr /opt/bitnami/nginx/logs/error.log

 
Nginx 웹 서버를 배포하여 로그를 모니터링해보겠습니다. 
웹 서버에 접속을 시도할 때 쿠버네티스 로그에 Access 로그가 출력됨을 확인할 수 있습니다. 
 

# 웹서버 배포 
$ cat nginx-values.yaml
service:
  type: NodePort
networkPolicy:
  enabled: false
ingress:
  enabled: true
  ingressClassName: alb
  hostname: nginx.okms1017.name
  pathType: Prefix
  path: /
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443},{"HTTP":80}]'
    alb.ingress.kubernetes.io/certificate-arn:arn:aws:acm*********************
    alb.ingress.kubernetes.io/success-codes: 200-399
    alb.ingress.kubernetes.io/load-balancer-name: myeks-ingress-alb
    alb.ingress.kubernetes.io/group.name: study
    alb.ingress.kubernetes.io/ssl-redirect: '443'
$ helm install nginx bitnami/nginx --version 15.14.0 -f nginx-values.yaml
NAME: nginx
LAST DEPLOYED: Sat Mar 30 23:11:51 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: nginx
CHART VERSION: 15.14.0
APP VERSION: 1.25.4

# 웹서버 접속 시도 
$ while true; do curl -s https://nginx.$MyDomain -I | head -n 1; date; sleep 1; done
HTTP/2 200
Sat Mar 30 22:51:54 KST 2024
HTTP/2 200
Sat Mar 30 22:51:55 KST 2024

# 로그 모니터링 
$ kubectl logs deploy/nginx -f
192.168.2.58 - - [30/Mar/2024:13:51:53 +0000] "GET / HTTP/1.1" 200 409 "-" "ELB-HealthChecker/2.0" "-"
192.168.2.58 - - [30/Mar/2024:13:51:53 +0000] "HEAD / HTTP/1.1" 200 0 "-" "curl/8.3.0" "43.201.83.174"
192.168.3.219 - - [30/Mar/2024:13:51:54 +0000] "HEAD / HTTP/1.1" 200 0 "-" "curl/8.3.0" "43.201.83.174"
192.168.1.148 - - [30/Mar/2024:13:51:55 +0000] "HEAD / HTTP/1.1" 200 0 "-" "curl/8.3.0" "43.201.83.174"
192.168.2.58 - - [30/Mar/2024:13:51:56 +0000] "HEAD / HTTP/1.1" 200 0 "-" "curl/8.3.0" "43.201.83.174"
192.168.3.219 - - [30/Mar/2024:13:51:57 +0000] "HEAD / HTTP/1.1" 200 0 "-" "curl/8.3.0" "43.201.83.174"

# 컨테이너 로그파일 위치 확인 
$ kubectl exec -it deploy/nginx -- ls -l /opt/bitnami/nginx/logs/
lrwxrwxrwx 1 1001 1001 11 Mar 30 13:43 access.log -> /dev/stdout
lrwxrwxrwx 1 1001 1001 11 Mar 30 13:43 error.log -> /dev/stderr

 

CloudWatch Container Observability & Fluent Bit 

Worker Node에 CloudWatch Agent 파드와 Fluent Bit 파드가 데몬셋으로 배치되어 각기 컨테이너의 메트릭과 로그를 수집합니다. 사용자는 CloudWatch를 통해 시각화된 정보를 제공받습니다. 
 

  • Application Logs : 컨테이너(파드) 로그   ex) /var/log/containers 
  • Host Logs : 노드(호스트) 로그   ex) /var/log/dmesg, /var/log/secure, /var/log/messages 
  • Data Plane Logs :  쿠버네티스 데이터플레인 로그   ex) /var/log/journal for kubelet.service, kubeproxy.service, docker.service 

CloudWatch Container Observability for EKS

 

# Application Logs
$ for node in $N1 $N2 $N3; do echo ">>>>> $node <<<<<"; ssh ec2-user@$node sudo tree /var/log/containers; echo; done
$ for node in $N1 $N2 $N3; do echo ">>>>> $node <<<<<"; ssh ec2-user@$node sudo ls -al /var/log/containers; echo; done

# Host Logs
$ for node in $N1 $N2 $N3; do echo ">>>>> $node <<<<<"; ssh ec2-user@$node sudo tree /var/log/ -L 1; echo; done
$ for node in $N1 $N2 $N3; do echo ">>>>> $node <<<<<"; ssh ec2-user@$node sudo ls -la /var/log/; echo; done
$ for log in dmesg secure messages; do echo ">>>>> Node1: /var/log/$log <<<<<"; ssh ec2-user@$N1 sudo tail /var/log/$log; echo; done
$ for log in dmesg secure messages; do echo ">>>>> Node2: /var/log/$log <<<<<"; ssh ec2-user@$N2 sudo tail /var/log/$log; echo; done
$ for log in dmesg secure messages; do echo ">>>>> Node3: /var/log/$log <<<<<"; ssh ec2-user@$N3 sudo tail /var/log/$log; echo; done

# Data Plane Logs
$ for node in $N1 $N2 $N3; do echo ">>>>> $node <<<<<"; ssh ec2-user@$node sudo tree /var/log/journal -L 1; echo; done
$ ssh ec2-user@$N3 sudo journalctl -x -n 200
$ ssh ec2-user@$N3 sudo journalctl -f

 
CloudWatch Container Observability를 설치합니다. 
 

# CloudWatch Container Observability Addon 
$ aws eks create-addon --cluster-name $CLUSTER_NAME --addon-name amazon-cloudwatch-observability
    "addon": {
        "addonName": "amazon-cloudwatch-observability",
        "clusterName": "myeks",
        "status": "CREATING",
        "addonVersion": "v1.4.0-eksbuild.1",
        "health": {
            "issues": []
        },
        "addonArn": "arn:aws:eks:ap-northeast-2:732659419746:addon/myeks/amazon-cloudwatch-observability/dec747ee-ffbd-4f3b-9866-6e4698398bb5",
        "createdAt": "2024-03-30T23:57:54.771000+09:00",
        "modifiedAt": "2024-03-30T23:57:54.789000+09:00",
        "tags": {}
    }
$ aws eks list-addons --cluster-name myeks --output table
---------------------------------------
|             ListAddons              |
+-------------------------------------+
||              addons               ||
|+-----------------------------------+|
||  amazon-cloudwatch-observability  ||
||  aws-ebs-csi-driver               ||
||  coredns                          ||
||  kube-proxy                       ||
||  vpc-cni                          ||
|+-----------------------------------+|

# CloudWatch Container Observability 설치 확인
$ kubectl get pod -A
NAMESPACE           NAME                                                              READY   STATUS    RESTARTS   AGE
amazon-cloudwatch   amazon-cloudwatch-observability-controller-manager-8544df7hsmpl   1/1     Running   0          29s
amazon-cloudwatch   cloudwatch-agent-6w6hf                                            1/1     Running   0          25s
amazon-cloudwatch   cloudwatch-agent-grbh8                                            1/1     Running   0          25s
amazon-cloudwatch   cloudwatch-agent-nsdxh                                            1/1     Running   0          25s
amazon-cloudwatch   fluent-bit-6lrmc                                                  1/1     Running   0          29s
amazon-cloudwatch   fluent-bit-bc6nb                                                  1/1     Running   0          29s
amazon-cloudwatch   fluent-bit-vsr4j                                                  1/1     Running   0          29s

 
CloudWatch Agent 파드는 HostPath에 지정된 경로들의 파일을 수집하여 CloudWatch에 전송합니다. 
그러나 파드가 노드의 루트 경로를 HostPath로 사용하고 있으므로 보안상 안전하지는 않습니다. 
 

HostPath - Fluent Bit &CloudWatch Agent

 

# Fluent Bit: INPUT/FILTER/OUTPUT 설정
# 설정 파일: application-log.conf, dataplane-log.conf, fluent-bit.conf, host-log.conf, parsers.conf
$ kubectl describe cm fluent-bit-config -n amazon-cloudwatch
```
application-log.conf:
----
[INPUT]
    Name                tail
    Tag                 application.*
    Exclude_Path        /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
    Path                /var/log/containers/*.log
    multiline.parser    docker, cri
    DB                  /var/fluent-bit/state/flb_container.db
    Mem_Buf_Limit       50MB
    Skip_Long_Lines     On
    Refresh_Interval    10
    Rotate_Wait         30
    storage.type        filesystem
    Read_from_Head      ${READ_FROM_HEAD}
```
[FILTER]
    Name                kubernetes
    Match               application.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_Tag_Prefix     application.var.log.containers.
    Merge_Log           On
    Merge_Log_Key       log_processed
    K8S-Logging.Parser  On
    K8S-Logging.Exclude Off
    Labels              Off
    Annotations         Off
    Use_Kubelet         On
    Kubelet_Port        10250
    Buffer_Size         0
```
[OUTPUT]
    Name                cloudwatch_logs
    Match               application.*
    region              ${AWS_REGION}
    log_group_name      /aws/containerinsights/${CLUSTER_NAME}/application
    log_stream_prefix   ${HOST_NAME}-
    auto_create_group   true
    extra_user_agent    container-insights
 ```

 

CloudWatch Container Observability - Log Group

 
Container Insights 대시보드에서 시각화된 정보를 볼 수 있습니다. 
 

Container Insights

 

Container Insights Pods

 

Metrics-server

쿠버네티스에 내장된 서버로 kubelet으로부터 리소스 사용량 및 성능 데이터를 수집하고 제공하는 컴포넌트입니다. 
기본적으로 15초 간격으로 파드의 메트릭을 취득해 이를 기반으로 필요한 파드 수를 계산하고, 레플리카셋 및 디플로이먼트의 파드 수를 변경합니다.  
※ cAdvisor : kubelet에 포함된 컨테이너 메트릭을 수집, 집계, 노출하는 데몬 
 

Metrics-server

# Metrics server 배포
$ kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

# Metrics server 배포 확인
$ kubectl get pod -n kube-system -l k8s-app=metrics-server
NAME                              READY   STATUS    RESTARTS   AGE
metrics-server-6d94bc8694-swrz4   0/1     Running   0          21s
$ kubectl api-resources | grep metrics
nodes                                                  metrics.k8s.io/v1beta1                 false        NodeMetrics
pods                                                   metrics.k8s.io/v1beta1                 true         PodMetrics
$ kubectl get apiservices |egrep '(AVAILABLE|metrics)'
NAME                                   SERVICE                      AVAILABLE   AGE
v1beta1.metrics.k8s.io                 kube-system/metrics-server   True        55s

# Worker Node 메트릭 확인
$ kubectl top node
NAME                                               CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
ip-192-168-1-85.ap-northeast-2.compute.internal    80m          2%     701Mi           4%
ip-192-168-2-66.ap-northeast-2.compute.internal    92m          2%     840Mi           5%
ip-192-168-3-165.ap-northeast-2.compute.internal   60m          1%     739Mi           4%

# 파드 메트릭 확인
$ kubectl top pod -A
$ kubectl top pod -n kube-system --sort-by='cpu'
NAME                                            CPU(cores)   MEMORY(bytes)
kube-ops-view-9cc4bf44c-fg85l                   14m          35Mi
ebs-csi-controller-7df5f479d4-6sg2p             5m           63Mi
aws-node-p2jg4                                  5m           61Mi
aws-node-pdvvc                                  5m           61Mi
metrics-server-6d94bc8694-swrz4                 4m           19Mi
aws-load-balancer-controller-5f7b66cdd5-c598n   4m           29Mi
aws-node-f9xhk                                  4m           60Mi
coredns-55474bf7b9-bc2z9                        3m           14Mi
ebs-csi-controller-7df5f479d4-46gms             3m           53Mi
coredns-55474bf7b9-kmx7n                        3m           15Mi
aws-load-balancer-controller-5f7b66cdd5-bfw8n   2m           22Mi
ebs-csi-node-qnmj9                              1m           22Mi
external-dns-7f567f7947-lcpmk                   1m           19Mi
ebs-csi-node-mrs9l                              1m           23Mi
kube-proxy-7k5bm                                1m           12Mi
kube-proxy-xbtr9                                1m           12Mi
kube-proxy-zj4dv                                1m           12Mi
ebs-csi-node-c62xk                              1m           22Mi

$ kubectl top pod -n kube-system --sort-by='memory'
NAME                                            CPU(cores)   MEMORY(bytes)
ebs-csi-controller-7df5f479d4-6sg2p             5m           63Mi
aws-node-pdvvc                                  4m           61Mi
aws-node-p2jg4                                  4m           61Mi
aws-node-f9xhk                                  3m           60Mi
ebs-csi-controller-7df5f479d4-46gms             2m           53Mi
kube-ops-view-9cc4bf44c-fg85l                   10m          35Mi
aws-load-balancer-controller-5f7b66cdd5-c598n   3m           29Mi
ebs-csi-node-mrs9l                              1m           23Mi
aws-load-balancer-controller-5f7b66cdd5-bfw8n   2m           22Mi
ebs-csi-node-qnmj9                              1m           22Mi
ebs-csi-node-c62xk                              1m           22Mi
metrics-server-6d94bc8694-swrz4                 5m           19Mi
external-dns-7f567f7947-lcpmk                   1m           19Mi
coredns-55474bf7b9-kmx7n                        2m           15Mi
coredns-55474bf7b9-bc2z9                        3m           14Mi
kube-proxy-7k5bm                                1m           12Mi
kube-proxy-xbtr9                                1m           12Mi
kube-proxy-zj4dv                                1m           12Mi

 

**kwatch

쿠버네티스 클러스터의 리소스 및 이벤트를 실시간으로 모니터링하는 CLI 도구입니다. 
슬랙과 연동하여 이벤트에 대한 알림을 사용자에게 보낼 수 있습니다. 
 

 

botkube

쿠버네티스 클러스터를 모니터링하고 관리하는 오픈소스 도구입니다. 
슬랙, 팀즈, 디스코드와 같은 채팅 플랫폼과 통합되어 있고, 클러스터에서 발생하는 이벤트 및 경고를 실시간 알림으로 전달합니다.
채팅플랫폼에 권한을 부여한 후 명령어를 입력하였을 때 수행 결과를 출력하도록 만들 수도 있습니다.  
 

botkube Service Architecture

 


[출처] 
1) CloudNet@, AEWS 실습 스터디 
2) https://www.eksworkshop.com/docs/observability/resource-view/

 

View resources in EKS console | EKS Workshop

Gain visibility of Kubernetes resources in the Amazon Elastic Kubernetes Service console.

www.eksworkshop.com

3) https://docs.aws.amazon.com/eks/latest/userguide/view-kubernetes-resources.html

 

View Kubernetes resources - Amazon EKS

The role ARN can't include a path such as role/my-team/developers/my-console-viewer-role. The format of the ARN must be arn:aws:iam::111122223333:role/my-console-viewer-role. In this example, my-team/developers/ needs to be removed.

docs.aws.amazon.com

4) https://docs.aws.amazon.com/ko_kr/eks/latest/userguide/eks-observe.html

 

Amazon EKS의 관찰 가능성 - Amazon EKS

이 페이지에 작업이 필요하다는 점을 알려 주셔서 감사합니다. 실망시켜 드려 죄송합니다. 잠깐 시간을 내어 설명서를 향상시킬 수 있는 방법에 대해 말씀해 주십시오.

docs.aws.amazon.com

5) https://docs.aws.amazon.com/ko_kr/eks/latest/userguide/control-plane-logs.html

 

Amazon EKS 컨트롤 플레인 로깅 - Amazon EKS

로그 스트림의 시작 부분에 API 서버 로그가 표시되지 않으면 서버에서 API 서버 로깅을 활성화하기 전에 서버에서 API 서버 로그 파일이 교체되었을 수 있습니다. API 서버 로깅이 활성화되기 전에

docs.aws.amazon.com

6) https://docs.docker.com/config/containers/logging/

 

View container logs

Learn how to write to, view, and configure a container's logs

docs.docker.com

7) https://github.com/nginxinc/docker-nginx/blob/8921999083def7ba43a06fabd5f80e4406651353/mainline/jessie/Dockerfile#L21-L23

 

docker-nginx/mainline/jessie/Dockerfile at 8921999083def7ba43a06fabd5f80e4406651353 · nginxinc/docker-nginx

Official NGINX Dockerfiles. Contribute to nginxinc/docker-nginx development by creating an account on GitHub.

github.com

8) https://aws.amazon.com/ko/blogs/containers/fluent-bit-integration-in-cloudwatch-container-insights-for-eks/

 

Fluent Bit Integration in CloudWatch Container Insights for EKS | Amazon Web Services

Ugur KIRA, Dejun Hu, TP Kohli CloudWatch Container Insights CloudWatch Container Insights enables you to explore, analyze, and visualize your container metrics, Prometheus metrics, application logs, and performance log events through automated dashboards i

aws.amazon.com

9) https://fluentbit.io/

 

fluentbit

Fluent Bit is a CNCF (Cloud Native Computing Foundation) graduated project under the umbrella of Fluentd. We are part of a large open source community. No vendor lock-in.

fluentbit.io

10) https://www.lesstif.com/system-admin/linux-journalctl-82215080.html

 

Linux journalctl 사용법

 

www.lesstif.com

11) https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html#ContainerInsights-fluentbit-multiline

 

Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs - Amazon CloudWatch

If you already have FluentD configured in Container Insights and the FluentD DaemonSet is not running as expected (this can happen if you use the containerd runtime), you must uninstall it before installing Fluent Bit to prevent Fluent Bit from processing

docs.aws.amazon.com

12) https://kubernetes.io/ko/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/

 

리소스 메트릭 파이프라인

쿠버네티스에서, 메트릭 API(Metrics API) 는 자동 스케일링 및 비슷한 사용 사례를 지원하기 위한 기본적인 메트릭 집합을 제공한다. 이 API는 노드와 파드의 리소스 사용량 정보를 제공하며, 여기에

kubernetes.io

13) https://docs.botkube.io/next/architecture/

 

Architecture | Botkube

This document describes high-level Botkube architecture, all components in the system and interactions between them.

docs.botkube.io

728x90

'AWS > EKS' 카테고리의 다른 글

[AEWS2] 5-1. EKS AutoScaling  (1) 2024.04.07
[AEWS2] 4-2. Prometheus & Grafana  (0) 2024.03.31
[AEWS2] 3-1. EKS Storage & Nodegroup  (1) 2024.03.24
[AEWS2] 2-5. Network Policies with VPC CNI  (0) 2024.03.17
[AEWS2] 2-4. Ingress & External DNS  (4) 2024.03.17