본문 바로가기
AWS/EKS

[AEWS2] 1-2. Amazon EKS 배포

by okms1017 2024. 3. 9.
728x90

✍ Posted by Immersive Builder  Seong
 

1. 기본 인프라 배포 

 
Amazon EKS 클러스터를 배포하기 위한 사전 환경을 구성해야 합니다. 

 

AWS CloudFormation 스택 생성 

먼저 AWS CloudFormation을 이용하여 상기 Architecture와 같이 기본 인프라를 배포합니다.   
 

AWS CloudFormation - 기본 인프라 배포

스택이 성공적으로 생성 완료되면 eksctlhost 값을 통해 작업용 EC2 인스턴스의 퍼블릭 IP 주소를 확인할 수 있습니다. 
 

기본 정보 확인

작업용 EC2 인스턴스에는 kubectl, eksctl, docker engine이 설치되어 있습니다. 
 

# Client version = ± K8s version 
$ kubectl version --client=true -o yaml | yh
clientVersion:
  buildDate: "2024-01-02T20:37:32Z"
  compiler: gc
  gitCommit: e78a4be9da4c375a87a109e0f4a5f4a8d2bc17c0
  gitTreeState: clean
  gitVersion: v1.28.5-eks-5e0fdde
  goVersion: go1.20.12
  major: "1"
  minor: 28+
  platform: linux/amd64
kustomizeVersion: v5.0.4-0.20230601165947-6ce0bf390ce3

$ eksctl version
0.173.0

$ eksctl create cluster -h | grep version
--version string   Kubernetes version (valid options: 1.23, 1.24, 1.25, 1.26, 1.27, 1.28, 1.29) (default "1.29")

$ aws --version
aws-cli/2.15.26 Python/3.11.8 Linux/4.14.336-256.559.amzn2.x86_64 exe/x86_64.amzn.2 prompt/off

$ docker info | grep "Server Version"
Server Version: 20.10.25

 

자격 증명

AWS CLI를 사용하기 위해 관리자 권한 정책을 부여한 IAM User(admin)의 Access Key ID와 Secret Access Key를 입력하여 자격을 증명합니다. 
 

$ aws configure 
AWS Access Key ID [None]: AK*************
AWS Secret Access Key [None]: im************
Default region name [None]: ap-northeast-2
Default output format [None]:

$ cat ~/.aws/credentials
[default]
aws_access_key_id = AK*************
aws_secret_access_key = im************

 

VPC 정보 확인 및 환경 변수 설정 

 

# EKS 배포할 VPC 정보 확인 
$ aws ec2 describe-vpcs --filters "Name=tag:Name,Values=$CLUSTER_NAME-VPC" | jq 
```
"CidrBlock": "192.168.0.0/16",
"VpcId": "vpc-05779d2c35f884754",
"Key": "Name",
"Value": "myeks-VPC"
```

# VPC 환경변수 설정
$ export VPCID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=$CLUSTER_NAME-VPC" | jq -r .Vpcs[].VpcId)
$ echo "export VPCID=$VPCID" >> /etc/profile
$ echo $VPCID
vpc-05779d2c35f884754

# EKS 배포할 Subnet 정보 확인 
$ aws ec2 describe-subnets --filters "Name=vpc-id,Values=$VPCID" --output json | jq 
```
 {  
   "AvailabilityZone": "ap-northeast-2a",
   "AvailabilityZoneId": "apne2-az1",
   "CidrBlock": "192.168.3.0/24",
   "SubnetId": "subnet-0a437664f3baa8b61",
   "VpcId": "vpc-05779d2c35f884754",
   "Tags": [
     {
       "Key": "aws:cloudformation:logical-id",
       "Value": "PrivateSubnet1"
     }
   ]
 },
 {  
   "AvailabilityZone": "ap-northeast-2c",
   "AvailabilityZoneId": "apne2-az3",
   "CidrBlock": "192.168.2.0/24",
   "SubnetId": "subnet-0139abbd2902cb0e1",
   "VpcId": "vpc-05779d2c35f884754",
   "Tags": [
     {
       "Key": "aws:cloudformation:logical-id",
       "Value": "PublicSubnet2"
     }
   ]
 },
 {  
   "AvailabilityZone": "ap-northeast-2c",
   "AvailabilityZoneId": "apne2-az3",
   "CidrBlock": "192.168.4.0/24",
   "SubnetId": "subnet-0687adca0dcb89565",
   "VpcId": "vpc-05779d2c35f884754",
   "Tags": [
     {
       "Key": "aws:cloudformation:logical-id",
       "Value": "PrivateSubnet2"
     }
   ]
 },
 {  
   "AvailabilityZone": "ap-northeast-2a",
   "AvailabilityZoneId": "apne2-az1",
   "CidrBlock": "192.168.1.0/24",
   "SubnetId": "subnet-0b2b90c9a36be462f",
   "VpcId": "vpc-05779d2c35f884754",
   "Tags": [
     {
       "Key": "aws:cloudformation:logical-id",
       "Value": "PublicSubnet1"
     }
   ]
 }
```

# Public Subnet 환경변수 설정
$ export PubSubnet1=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-PublicSubnet1" --query "Subnets[0].[SubnetId]" --output text)
$ export PubSubnet2=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-PublicSubnet2" --query "Subnets[0].[SubnetId]" --output text)
$ echo "export PubSubnet1=$PubSubnet1" >> /etc/profile
$ echo "export PubSubnet2=$PubSubnet2" >> /etc/profile
$ echo $Pubsubnet1
subnet-0b2b90c9a36be462f
$ echo $Pubsubnet2
subnet-0139abbd2902cb0e1

 

2. Amazon EKS 배포 

eksctl 이란? 

EKS에서 쿠버네티스 클러스터를 생성 및 관리하기 위한 오픈소스 명령줄 도구입니다. 
eksctl을 통해 배포 및 업데이트 시, AWS CloudFormation 서비스를 통해서 스택을 생성하거나 업데이트를 수행합니다. 
 
공식 홈페이지 URL : https://eksctl.io/

 

eksctl - The official CLI for Amazon EKS

The official CLI for Amazon EKS

eksctl.io

사용 가이드 URL : https://eksctl.io/getting-started/

 

Introduction - eksctl

The official CLI for Amazon EKS

eksctl.io

Amazon EKS 클러스터 배포 

eksctl CLI를 이용해 EKS 클러스터를 배포합니다. 
 

# EKS 클러스터 & 관리형 노드그룹 배포 
# --dry-run option: 배포 실행계획 출력 
$ eksctl create cluster --name $CLUSTER_NAME --region=$AWS_DEFAULT_REGION --nodegroup-name=$CLUSTER_NAME-nodegroup --node-type=t3.medium \ 
--node-volume-size=30 --vpc-public-subnets "$PubSubnet1,$PubSubnet2" --version 1.28 --ssh-access --external-dns-access --verbose 4
```
2024-03-08 15:19:50 [▶]  Setting credentials expiry window to 30 minutes
2024-03-08 15:19:50 [ℹ]  using region ap-northeast-2
2024-03-08 15:19:51 [ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial managed nodegroup
2024-03-08 15:19:51 [▶]  started task: create cluster control plane "myeks"
2024-03-08 15:19:51 [ℹ]  building cluster stack "eksctl-myeks-cluster"
2024-03-08 15:19:51 [ℹ]  deploying stack "eksctl-myeks-cluster"
2024-03-08 15:20:21 [ℹ]  waiting for CloudFormation stack "eksctl-myeks-cluster"
2024-03-08 15:28:52 [▶]  completed task: create cluster control plane "myeks"
```
2024-03-08 15:30:52 [▶]  started task: create managed nodegroup "myeks-nodegroup"
2024-03-08 15:30:52 [ℹ]  building managed nodegroup stack "eksctl-myeks-nodegroup-myeks-nodegroup"
2024-03-08 15:30:52 [ℹ]  deploying stack "eksctl-myeks-nodegroup-myeks-nodegroup"
2024-03-08 15:30:53 [ℹ]  waiting for CloudFormation stack "eksctl-myeks-nodegroup-myeks-nodegroup"
2024-03-08 15:33:14 [✔]  saved kubeconfig as "/root/.kube/config"
2024-03-08 15:33:14 [✔]  all EKS cluster resources for "myeks" have been created
2024-03-08 15:33:14 [ℹ]  nodegroup "myeks-nodegroup" has 2 node(s)
2024-03-08 15:33:14 [ℹ]  node "ip-192-168-1-204.ap-northeast-2.compute.internal" is ready
2024-03-08 15:33:14 [ℹ]  node "ip-192-168-2-78.ap-northeast-2.compute.internal" is ready
2024-03-08 15:33:14 [ℹ]  waiting for at least 2 node(s) to become ready in "myeks-nodegroup"
2024-03-08 15:33:15 [ℹ]  kubectl command should work with "/root/.kube/config", try 'kubectl get nodes'
2024-03-08 15:33:15 [✔]  EKS cluster "myeks" in "ap-northeast-2" region is ready
```

# 모니터링 
$ while true; do aws ec2 describe-instances --query "Reservations[*].Instances[*].{PublicIPAdd:PublicIpAddress,PrivateIPAdd:PrivateIpAddress,InstanceName:Tags[?Key=='Name']|[0].Value,Status:State.Name}" \
--filters Name=instance-state-name,Values=running --output text ; echo "------------------------------" ; sleep 1; done
myeks-myeks-nodegroup-Node      192.168.2.78    13.209.22.97    running
myeks-host      192.168.1.100   43.200.252.56   running
myeks-myeks-nodegroup-Node      192.168.1.204   3.35.174.75     running

 

EKS 정보 확인 

클러스터 배포 시간은 약 15분 가량 소요됩니다. 
AWS CloudFormation 스택에서 리소스 생성 로그를 확인할 수 있습니다. 
 

AWS CloudFormation - EKS Cluster 배포

Amazon EKS 콘솔에서 클러스터 정보를 확인합니다. 
 

Amazon EKS Cluster Info

AutoScaling 그룹에서 구성 정보를 확인합니다. 
 

AutoScaling 그룹 구성 정보
EC2 인스턴스 정보

 

# EKS 클러스터 정보 확인 
$ kubectl cluster-info
Kubernetes control plane is running at https://3E8B0E1674195840701D48C80BEF7491.gr7.ap-northeast-2.eks.amazonaws.com
CoreDNS is running at https://3E8B0E1674195840701D48C80BEF7491.gr7.ap-northeast-2.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

# API Server Endpoint 
$ aws eks describe-cluster --name $CLUSTER_NAME | jq -r .cluster.endpoint
https://3E8B0E1674195840701D48C80BEF7491.gr7.ap-northeast-2.eks.amazonaws.com

$ APIDNS=$(aws eks describe-cluster --name $CLUSTER_NAME | jq -r .cluster.endpoint | cut -d '/' -f 3)
$ echo $APIDNS
3E8B0E1674195840701D48C80BEF7491.gr7.ap-northeast-2.eks.amazonaws.com
$ dig +short $APIDNS
3.38.61.180
3.34.148.207

# EKS Public Endpoint 접속 시도 
$ curl -k -s $(aws eks describe-cluster --name $CLUSTER_NAME | jq -r .cluster.endpoint)/version | jq
{
  "major": "1",
  "minor": "28+",
  "gitVersion": "v1.28.6-eks-508b6b3",
  "gitCommit": "25a726351cee8ee6facce01af4214605e089d5da",
  "gitTreeState": "clean",
  "buildDate": "2024-01-29T20:58:56Z",
  "goVersion": "go1.20.13",
  "compiler": "gc",
  "platform": "linux/amd64"
}

$ curl -k -s https://3.38.61.180/version | jq -r
{
  "major": "1",
  "minor": "28+",
  "gitVersion": "v1.28.6-eks-508b6b3",
  "gitCommit": "25a726351cee8ee6facce01af4214605e089d5da",
  "gitTreeState": "clean",
  "buildDate": "2024-01-29T20:58:56Z",
  "goVersion": "go1.20.13",
  "compiler": "gc",
  "platform": "linux/amd64"
}
$ curl -k -s https://3.34.148.207/version | jq -r
{
  "major": "1",
  "minor": "28+",
  "gitVersion": "v1.28.6-eks-508b6b3",
  "gitCommit": "25a726351cee8ee6facce01af4214605e089d5da",
  "gitTreeState": "clean",
  "buildDate": "2024-01-29T20:58:56Z",
  "goVersion": "go1.20.13",
  "compiler": "gc",
  "platform": "linux/amd64"
}

# API Server에 명령어 조회
$ kubectl get node -v=6
I0308 17:38:10.451193   11738 loader.go:395] Config loaded from file:  /root/.kube/config
I0308 17:38:11.434543   11738 round_trippers.go:553] GET https://3E8B0E1674195840701D48C80BEF7491.gr7.ap-northeast-2.eks.amazonaws.com/api/v1/nodes?limit=500 200 OK in 977 milliseconds
NAME                                               STATUS   ROLES    AGE    VERSION
ip-192-168-1-204.ap-northeast-2.compute.internal   Ready    <none>   125m   v1.28.5-eks-5e0fdde
ip-192-168-2-78.ap-northeast-2.compute.internal    Ready    <none>   125m   v1.28.5-eks-5e0fdde

 

노드 상세 정보 조회 

# Node IP 조회
$ aws ec2 describe-instances --query "Reservations[*].Instances[*].{PublicIPAdd:PublicIpAddress,PrivateIPAdd:PrivateIpAddress,InstanceName:Tags[?Key=='Name']|[0].Value,Status:State.Name}" \ 
--filters Name=instance-state-name,Values=running --output table
-----------------------------------------------------------------------------
|                             DescribeInstances                             |
+-----------------------------+----------------+----------------+-----------+
|        InstanceName         | PrivateIPAdd   |  PublicIPAdd   |  Status   |
+-----------------------------+----------------+----------------+-----------+
|  myeks-myeks-nodegroup-Node |  192.168.2.78  |  13.209.22.97  |  running  |
|  myeks-host                 |  192.168.1.100 |  43.200.252.56 |  running  |
|  myeks-myeks-nodegroup-Node |  192.168.1.204 |  3.35.174.75   |  running  |
+-----------------------------+----------------+----------------+-----------+
$ for i in $N1 $N2; do echo ">> node $i <<"; ssh ec2-user@$i sudo ip -c addr; echo; done
```
>> node 192.168.1.204 <<
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    inet 192.168.1.204/24 brd 192.168.1.255 scope global dynamic eth0
```
>> node 192.168.2.78 <<
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    inet 192.168.2.78/24 brd 192.168.2.255 scope global dynamic eth0
```

# Node Private IP 환경변수 설정 
$ N1=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2a -o jsonpath={.items[0].status.addresses[0].address})
$ N2=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2c -o jsonpath={.items[0].status.addresses[0].address})
$ echo $N1,$N2
192.168.1.204,192.168.2.78
$ echo "export N1=$N1" >> /etc/profile
$ echo "export N2=$N2" >> /etc/profile

# Security Group Rule 추가
$ NGSGID=$(aws ec2 describe-security-groups --filters Name=group-name,Values=*nodegroup* --query "SecurityGroups[*].[GroupId]" --output text)
$ echo $NGSGID
sg-02f7a9e6b19c06f0d
$ echo "export NGSGID=$NGSGID" >> /etc/profile
# -1: all protocol 
$ aws ec2 authorize-security-group-ingress --group-id $NGSGID --protocol '-1' --cidr 192.168.1.100/32
{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-0fb0a52fcc74d3e73",
            "GroupId": "sg-02f7a9e6b19c06f0d",
            "GroupOwnerId": "732659419746",
            "IsEgress": false,
            "IpProtocol": "-1",
            "FromPort": -1,
            "ToPort": -1,
            "CidrIpv4": "192.168.1.100/32"
        }
    ]
}

# 작업용 EC2 -> Worker Nodes (ICMP)
$ ping -c 1 $N1
PING 192.168.1.204 (192.168.1.204) 56(84) bytes of data.
64 bytes from 192.168.1.204: icmp_seq=1 ttl=255 time=0.535 ms
$ ping -c 1 $N2
PING 192.168.2.78 (192.168.2.78) 56(84) bytes of data.
64 bytes from 192.168.2.78: icmp_seq=1 ttl=255 time=1.04 ms

# 작업용 EC2 -> Worker Nodes (SSH)
$ ssh -i ~/.ssh/id_rsa ec2-user@$N1 hostname
Are you sure you want to continue connecting (yes/no)? yes
ip-192-168-1-204.ap-northeast-2.compute.internal
$ ssh -i ~/.ssh/id_rsa ec2-user@$N2 hostname
Are you sure you want to continue connecting (yes/no)? yes
ip-192-168-2-78.ap-northeast-2.compute.internal
$ ssh ec2-user@$N1 
$ ssh ec2-user@$N2 

# Node AWS VPC CNI 조회 
$ kubectl -n kube-system get ds aws-node
NAME       DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
aws-node   2         2         2       2            2           <none>          4h40m
$ kubectl describe ds aws-node -n kube-system | grep Image | cut -d "/" -f 2
amazon-k8s-cni-init:v1.15.1-eksbuild.1
amazon-k8s-cni:v1.15.1-eksbuild.1
amazon

# Node Process 조회 
$ for i in $N1 $N2; do echo ">> node $i <<"; ssh ec2-user@$i sudo systemctl status kubelet; echo; done
```
>> node 192.168.1.204 <<
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
```
>> node 192.168.2.78 <<
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
```
$ for i in $N1 $N2; do echo ">> node $i <<"; ssh ec2-user@$i sudo pstree; echo; done
$ for i in $N1 $N2; do echo ">> node $i <<"; ssh ec2-user@$i ps axf |grep /usr/bin/containerd; echo; done
$ for i in $N1 $N2; do echo ">> node $i <<"; ssh ec2-user@$i ls /etc/kubernetes/manifests/; echo; done
$ for i in $N1 $N2; do echo ">> node $i <<"; ssh ec2-user@$i ls /etc/kubernetes/kubelet/; echo; done
$ for i in $N1 $N2; do echo ">> node $i <<"; ssh ec2-user@$i cat /etc/kubernetes/kubelet/kubelet-config.json; echo; done

# Node Process 상세 조회 
$ for i in $N1 $N2; do echo ">> node $i <<"; ssh ec2-user@$i ps afxuwww; echo; done

# Node Storage 조회 
$ for i in $N1 $N2; do echo ">> node $i <<"; ssh ec2-user@$i lsblk; echo; done
>> node 192.168.1.204 <<
NAME          MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1       259:0    0  30G  0 disk
├─nvme0n1p1   259:1    0  30G  0 part /
└─nvme0n1p128 259:2    0   1M  0 part
>> node 192.168.2.78 <<
NAME          MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1       259:0    0  30G  0 disk
├─nvme0n1p1   259:1    0  30G  0 part /
└─nvme0n1p128 259:2    0   1M  0 part
$ for i in $N1 $N2; do echo ">> node $i <<"; ssh ec2-user@$i df -hT /; echo; done
>> node 192.168.1.204 <<
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/nvme0n1p1 xfs    30G  3.8G   27G  13% /
>> node 192.168.2.78 <<
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/nvme0n1p1 xfs    30G  3.9G   27G  13% /

 

** 매니페스트(YAML) 작성하여 클러스터 배포하기

 

3. Amazon EKS 삭제 

실습을 완료하고 리소스를 삭제합니다. 
클러스터 삭제 시간은 약 10분 가량 소요됩니다. 
 

# EKS 클러스터 삭제
$ eksctl delete cluster --name $CLUSTER_NAME

# 기본 인프라 삭제
$ aws cloudformation delete-stack --stack-name myeks

 

[출처] 
1) https://catalog.us-east-1.prod.workshops.aws/workshops/9c0aa9ab-90a9-44a6-abe1-8dff360ae428/ko-KR/10-intro/200-eks

 

Amazon EKS web application workshop

Building simple web application using Amazon EKS. This workshop covers from creating eks cluster to application's life cycle.

catalog.us-east-1.prod.workshops.aws

2) https://docs.aws.amazon.com/ko_kr/eks/latest/userguide/getting-started-eksctl.html

 

Amazon EKS 시작하기 - eksctl - Amazon EKS

이 페이지에 작업이 필요하다는 점을 알려 주셔서 감사합니다. 실망시켜 드려 죄송합니다. 잠깐 시간을 내어 설명서를 향상시킬 수 있는 방법에 대해 말씀해 주십시오.

docs.aws.amazon.com

 

728x90

'AWS > EKS' 카테고리의 다른 글

[AEWS2] 2-3. Service & AWS LoadBalancer Controller  (2) 2024.03.16
[AEWS2] 2-2. 파드 생성 개수 제한하기  (1) 2024.03.16
[AEWS2] 2-1. AWS VPC CNI 소개  (0) 2024.03.16
[AEWS2] 1-1. Amazon EKS 소개  (0) 2024.03.09
[AEWS2] 0. 스터디 사전준비  (2) 2024.03.06

관련글

티스토리툴바