본문 바로가기
AWS/EKS

[AEWS2] 2-4. Ingress & External DNS

by okms1017 2024. 3. 17.
728x90

✍ Posted by Immersive Builder  Seong

 

1. Ingress

Ingress 란? 

클러스터 내부의 서비스(ClusterIP, NodePort, LoadBalancer)를 외부로 노출(HTTP/HTTPS)하는 역할을 하는 컴포넌트입니다. AWS 환경에서 Ingress를 구현한 구현체가 ALB가 되며, LoadBalancer Controller에 의해 파드의 IP로 바로 통신이 가능해집니다. 

 

Ingress (CloudNet@)

 

서비스/파드 배포 테스트 (ALB) 

ALB 타입의 서비스와 파드 2개를 배포합니다. 

 

$ cat ingress1.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: game-2048
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: game-2048
  name: deployment-2048
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: app-2048
  replicas: 2
  template:
    metadata:
      labels:
        app.kubernetes.io/name: app-2048
    spec:
      containers:
      - image: public.ecr.aws/l6m2t8p7/docker-2048:latest
        imagePullPolicy: Always
        name: app-2048
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  namespace: game-2048
  name: service-2048
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  type: NodePort
  selector:
    app.kubernetes.io/name: app-2048
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: game-2048
  name: ingress-2048
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: service-2048
              port:
                number: 80
$ kubectl apply -f ingress1.yaml

 

# 생성 확인 
$ kubectl get-all -n game-2048
NAME                                                               NAMESPACE  AGE
configmap/kube-root-ca.crt                                         game-2048  10m
endpoints/service-2048                                             game-2048  10m
pod/deployment-2048-75db5866dd-8hlck                               game-2048  10m
pod/deployment-2048-75db5866dd-ljbvw                               game-2048  10m
serviceaccount/default                                             game-2048  10m
service/service-2048                                               game-2048  10m
deployment.apps/deployment-2048                                    game-2048  10m
replicaset.apps/deployment-2048-75db5866dd                         game-2048  10m
endpointslice.discovery.k8s.io/service-2048-kfdjb                  game-2048  10m
targetgroupbinding.elbv2.k8s.aws/k8s-game2048-service2-527512209d  game-2048  10m
ingress.networking.k8s.io/ingress-2048                             game-2048  10m

$ kubectl get targetgroupbindings -n game-2048
NAME                               SERVICE-NAME   SERVICE-PORT   TARGET-TYPE   AGE
k8s-game2048-service2-527512209d   service-2048   80             ip            15m

# ALB 생성 확인 
$ aws elbv2 describe-load-balancers --query 'LoadBalancers[?contains(LoadBalancerName, `k8s-game2048`) == `true`]' | jq
```
    "LoadBalancerArn": "arn:aws:elasticloadbalancing:ap-northeast-2:732659419746:loadbalancer/app/k8s-game2048-ingress2-70d50ce3fd/6a8b580c7172e547",
    "DNSName": "k8s-game2048-ingress2-70d50ce3fd-2012380533.ap-northeast-2.elb.amazonaws.com",
    "Scheme": "internet-facing",
    "State": {
      "Code": "active"
    },
    "Type": "application",
```
$ ALB_ARN=$(aws elbv2 describe-load-balancers --query 'LoadBalancers[?contains(LoadBalancerName, `k8s-game2048`) == `true`].LoadBalancerArn' | jq -r '.[0]')
$ aws elbv2 describe-target-groups --load-balancer-arn $ALB_ARN
$ TARGET_GROUP_ARN=$(aws elbv2 describe-target-groups --load-balancer-arn $ALB_ARN | jq -r '.TargetGroups[0].TargetGroupArn')
$ aws elbv2 describe-target-health --target-group-arn $TARGET_GROUP_ARN | jq
```
      "Target": {
        "Id": "192.168.1.217",
        "Port": 80,
        "AvailabilityZone": "ap-northeast-2a"
      },
      "HealthCheckPort": "80",
      "TargetHealth": {
        "State": "healthy"
      },
```
      "Target": {
        "Id": "192.168.2.250",
        "Port": 80,
        "AvailabilityZone": "ap-northeast-2b"
      },
      "HealthCheckPort": "80",
      "TargetHealth": {
        "State": "healthy"
      }
```

# Ingress 확인
$ kubectl describe ingress -n game-2048 ingress-2048
Name:             ingress-2048
Labels:           <none>
Namespace:        game-2048
Address:          k8s-game2048-ingress2-70d50ce3fd-2012380533.ap-northeast-2.elb.amazonaws.com
Ingress Class:    alb
Default backend:  <default>
Rules:
  Host        Path  Backends
  ----        ----  --------
  *
              /   service-2048:80 (192.168.1.217:80,192.168.2.250:80)
Annotations:  alb.ingress.kubernetes.io/scheme: internet-facing
              alb.ingress.kubernetes.io/target-type: ip
Events:
  Type    Reason                  Age   From     Message
  ----    ------                  ----  ----     -------
  Normal  SuccessfullyReconciled  30m   ingress  Successfully reconciled

# ALB URL
$ kubectl get ingress -n game-2048 ingress-2048 -o jsonpath={.status.loadBalancer.ingress[0].hostname} | awk '{ print "Game URL = http://"$1 }'
Game URL = http://k8s-game2048-ingress2-70d50ce3fd-2012380533.ap-northeast-2.elb.amazonaws.com

# 파드 IP 확인
kubectl get pod -n game-2048 -owide
NAME                               READY   STATUS    RESTARTS   AGE   IP              NODE                                               NOMINATED NODE   READINESS GATES
deployment-2048-75db5866dd-8hlck   1/1     Running   0          32m   192.168.1.217   ip-192-168-1-213.ap-northeast-2.compute.internal   <none>           <none>
deployment-2048-75db5866dd-ljbvw   1/1     Running   0          32m   192.168.2.250   ip-192-168-2-49.ap-northeast-2.compute.internal    <none>           <none>

 

AWS 콘솔에서 80 포트로 리스닝 중인 어플리케이션 로드밸런서와 도메인 주소를 확인할 수 있습니다. 

Target Group의 파드 또한 80 포트로 리스닝 중이며 정상 상태(healthy)로 올라와 있습니다. 

네트워크 로드밸런서 IP모드의 서비스와 마찬가지로 파드의 IP로 직접 통신합니다. 

 

AWS Application LoadBalancer Info
AWS Application LoadBalancer - Listener & Rule
Target Group
AWS Application LoadBalancer - Resource Map

잠시 머리 식힐겸.

도메인 주소로 접속하여 게임 한판하겠습니다.

 

이번에는 파드 수를 조정하여 동작을 확인해봅니다. 

 

# 파드 수 2 -> 3
$ kubectl scale deployment -n game-2048 deployment-2048 --replicas 3
deployment.apps/deployment-2048 scaled

## 모니터링(Health-check)
$ while true; do aws elbv2 describe-target-health --target-group-arn $TARGET_GROUP_ARN --output text; echo; done
TARGETHEALTHDESCRIPTIONS        80
TARGET  ap-northeast-2c 192.168.3.83    80
TARGETHEALTH    healthy
TARGETHEALTHDESCRIPTIONS        80
TARGET  ap-northeast-2a 192.168.1.217   80
TARGETHEALTH    healthy
TARGETHEALTHDESCRIPTIONS        80
TARGET  ap-northeast-2b 192.168.2.250   80
TARGETHEALTH    healthy

# 파드 수 3 -> 1
$ kubectl scale deployment -n game-2048 deployment-2048 --replicas 1
deployment.apps/deployment-2048 scaled

## 모니터링(Health-check)
$ while true; do aws elbv2 describe-target-health --target-group-arn $TARGET_GROUP_ARN --output text; echo; done
```
TARGETHEALTHDESCRIPTIONS        80
TARGET  ap-northeast-2c 192.168.3.83    80
TARGETHEALTH    Target deregistration is in progress    Target.DeregistrationInProgress draining
TARGETHEALTHDESCRIPTIONS        80
TARGET  ap-northeast-2a 192.168.1.217   80
TARGETHEALTH    Target deregistration is in progress    Target.DeregistrationInProgress draining
TARGETHEALTHDESCRIPTIONS        80
TARGET  ap-northeast-2b 192.168.2.250   80
TARGETHEALTH    healthy
```
TARGETHEALTHDESCRIPTIONS        80
TARGET  ap-northeast-2b 192.168.2.250   80
TARGETHEALTH    healthy
```

 

2. External DNS

External DNS 란?

서비스 또는 인그레스의 도메인 주소를 실제 접속하는 주소로 매핑하는 역할을 합니다. 

클라우드 환경에서 AWS Route 53, Azure DNS, GCP Cloud DNS 등으로 구현할 수 있습니다. 

 

External DNS (CloudNet@)

 

구현 방법으로 Node IAM Role, Static credentials, IRSA 3가지가 있습니다. 

여기서는 Node IAM Role을 적용하였습니다. 

 

Node IAM Role - External DNS

 

AWS Route 53 세팅 

소유한 퍼블릭 도메인 주소를 활용합니다. 

 

퍼블릭 도메인 주소(okms1017.name)

# 도메인 환경변수 설정
$ MyDomain=okms1017.name
$ echo $MyDomain
okms1017.name
$ MyDnzHostedZoneId=`aws route53 list-hosted-zones-by-name --dns-name "${MyDomain}." \ 
--query "HostedZones[0].Id" --output text`
$ echo $MyDnzHostedZoneId
/hostedzone/Z09617933Q2PHRDLYPQJO
$ while true; do aws route53 list-resource-record-sets --hosted-zone-id "${MyDnzHostedZoneId}" \ 
--query "ResourceRecordSets[?Type == 'A']" | jq ; date ; echo ; sleep 1; done
[]
Sun Mar 17 03:24:31 KST 2024

 

External DNS 배포 

퍼블릭 도메인 주소(okms1017.name)를 변수 값으로 넣어 External DNS를 배포합니다. 

 

# External DNS 배포
$ cat externaldns.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
  namespace: kube-system
  labels:
    app.kubernetes.io/name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-dns
  labels:
    app.kubernetes.io/name: external-dns
rules:
  - apiGroups: [""]
    resources: ["services","endpoints","pods","nodes"]
    verbs: ["get","watch","list"]
  - apiGroups: ["extensions","networking.k8s.io"]
    resources: ["ingresses"]
    verbs: ["get","watch","list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
  labels:
    app.kubernetes.io/name: external-dns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
  - kind: ServiceAccount
    name: external-dns
    namespace: kube-system # change to desired namespace: externaldns, kube-addons
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
  namespace: kube-system
  labels:
    app.kubernetes.io/name: external-dns
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: external-dns
  template:
    metadata:
      labels:
        app.kubernetes.io/name: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
        - name: external-dns
          image: registry.k8s.io/external-dns/external-dns:v0.14.0
          args:
            - --source=service
            - --source=ingress
            - --domain-filter=${MyDomain} # will make ExternalDNS see only the hosted zones matching provided domain, omit to process all available hosted zones
            - --provider=aws
            #- --policy=upsert-only # would prevent ExternalDNS from deleting any records, omit to enable full synchronization
            - --aws-zone-type=public # only look at public hosted zones (valid values are public, private or no value for both)
            - --registry=txt
            - --txt-owner-id=${MyDnzHostedZoneId}
          env:
            - name: AWS_DEFAULT_REGION
              value: ap-northeast-2 # change to region where EKS is installed(admin@myeks:d

$ MyDomain=$MyDomain MyDnzHostedZoneId=$MyDnzHostedZoneId envsubst < externaldns.yaml | kubectl apply -f -
serviceaccount/external-dns created
clusterrole.rbac.authorization.k8s.io/external-dns created
clusterrolebinding.rbac.authorization.k8s.io/external-dns-viewer created
deployment.apps/external-dns created

 

* --policy=upsert-only 옵션 ?

- External DNS을 삭제하더라도 AWS Route 53의 A, TXT 레코드를 그대로 남기는 기능입니다. (권장) 

 

NLB & External DNS 도메인 연동 

이제 테스트 파드를 하나 배포하고 NLB와 External DNS 간 도메인을 연동합니다.

 

# 테트리스 디플로이먼트 배포
$ cat <<EOF | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tetris
  labels:
    app: tetris
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tetris
  template:
    metadata:
      labels:
        app: tetris
    spec:
      containers:
      - name: tetris
        image: bsord/tetris
---
apiVersion: v1
kind: Service
metadata:
  name: tetris
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
    #service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "80"
spec:
  selector:
    app: tetris
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  type: LoadBalancer
  loadBalancerClass: service.k8s.aws/nlb
EOF

# External DNS로 NLB 도메인 연결 
$ kubectl annotate service tetris "external-dns.alpha.kubernetes.io/hostname=tetris.$MyDomain"

# External DNS 로그 모니터링
$ kubectl logs deploy/external-dns -n kube-system -f
time="2024-03-16T18:50:03Z" level=info msg="Applying provider record filter for domains: [okms1017.name. .okms1017.name.]"
time="2024-03-16T18:50:03Z" level=info msg="Desired change: CREATE cname-tetris.okms1017.name TXT [Id: /hostedzone/Z09617933Q2PHRDLYPQJO]"
time="2024-03-16T18:50:03Z" level=info msg="Desired change: CREATE tetris.okms1017.name A [Id: /hostedzone/Z09617933Q2PHRDLYPQJO]"
time="2024-03-16T18:50:03Z" level=info msg="Desired change: CREATE tetris.okms1017.name TXT [Id: /hostedzone/Z09617933Q2PHRDLYPQJO]"
time="2024-03-16T18:50:03Z" level=info msg="3 record(s) in zone okms1017.name. [Id: /hostedzone/Z09617933Q2PHRDLYPQJO] were successfully updated"

# 모니터링
$ watch -d 'kubectl get deploy,svc,ep tetris'
Every 2.0s: kubectl get deploy,svc,ep tetris                        Sun Mar 17 03:53:48 2024
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/tetris   1/1     1            1           5m43s
NAME             TYPE           CLUSTER-IP      EXTERNAL-IP
                                      PORT(S)        AGE
service/tetris   LoadBalancer   10.100.38.223   k8s-default-tetris-5592b1097a-7d37de4b32a494
9e.elb.ap-northeast-2.amazonaws.com   80:31015/TCP   5m43s
NAME               ENDPOINTS         AGE
endpoints/tetris   192.168.2.29:80   5m43s

# 도메인 조회 
$ dig +short tetris.okms1017.name
13.209.108.173
3.39.109.92
43.203.71.198

# URL 
$ echo -e "Tetris Game URL = http://tetris.$MyDomain"
Tetris Game URL = http://tetris.okms1017.name

 

Route 53 호스팅 영역에 3개의 레코드(A, TXT)가 추가 생성되고,

NLB의 도메인 주소와 매핑되어 있음을 확인할 수 있습니다. 

 

Amazon Route 53 - NLB 도메인 연동
Network LoadBalancer 도메인 주소

 

하기 사이트에서 도메인 주소가 유효한지 확인할 수 있습니다. 

https://www.whatsmydns.net/

 

DNS Propagation Checker - Global DNS Checker Tool

Instant DNS Propagation Check. Global DNS Propagation Checker - Check DNS records around the world.

www.whatsmydns.net

마지막으로!

 

실제 도메인 주소(tetris.okms1017.name)로 접속하여 

테트리스 게임을 한 판 해주고 리소스를 삭제합니다. 

 

 

 

[출처] 

1) CloudNet@, AEWS 실습 스터디 

2) https://edgehog.blog/a-self-hosted-external-dns-resolver-for-kubernetes-111a27d6fc2c

 

A Self-hosted external DNS resolver for Kubernetes.

Understanding solutions for external DNS resolution in Kubernetes and introducing our k8s_gateway plugin.

edgehog.blog

3) https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/aws.md#deploy-externaldns

728x90

'AWS > EKS' 카테고리의 다른 글

[AEWS2] 3-1. EKS Storage & Nodegroup  (1) 2024.03.24
[AEWS2] 2-5. Network Policies with VPC CNI  (0) 2024.03.17
[AEWS2] 2-3. Service & AWS LoadBalancer Controller  (2) 2024.03.16
[AEWS2] 2-2. 파드 생성 개수 제한하기  (1) 2024.03.16
[AEWS2] 2-1. AWS VPC CNI 소개  (0) 2024.03.16

관련글

티스토리툴바