[AEWS2] 6-5. Kyverno

✍ Posted by Immersive Builder  Seong

 

Kyverno 란?

쿠버네티스에서 정책 기반으로 리소스 관리를 제공하는 CNCF 오픈소스 프로젝트입니다.  

 

• K8s Native Policy Management 
• Policy as Code 
• YAML Format
• 리소스 구성파일을 생성(generate), 변경(mutate), 검증(validate)하기 위해 사용합니다. 
• 정책에 매치되는 리소스 대상 : kind, name, label selector, etc
• kube-system 네임스페이스에는 정책이 적용되지 않도록 예외처리할 것을 권장합니다. 

 

Kyverno 동작 원리 

쿠버네티스 클러스터의 Dynamic Admission Controller에 의해 동작합니다. 

Kyverno는 API Server로부터 mutating admission webhook 또는 validating admission webhook을 통해 HTTP 요청을 전달받습니다. 

그리고 정의된 정책을 평가하여 Admission Controller를 통해 리소스를 허용 혹은 거부할지 결과를 반환합니다.

. 

Kyverno Architecture

 

Kyverno 설치  

# Kyverno 설치 
$ cat << EOF > kyverno-value.yaml
config:
  resourceFiltersExcludeNamespaces: [ kube-system ]

admissionController:
  serviceMonitor:
    enabled: true

backgroundController:
  serviceMonitor:
    enabled: true

cleanupController:
  serviceMonitor:
    enabled: true

reportsController:
  serviceMonitor:
    enabled: true
EOF
$ kubectl create ns kyverno
namespace/kyverno created

$ helm repo add kyverno https://kyverno.github.io/kyverno/
"kyverno" has been added to your repositories
$ helm install kyverno kyverno/kyverno --version 3.2.0-rc.3 -f kyverno-value.yaml -n kyverno
NAME: kyverno
LAST DEPLOYED: Sun Apr 14 07:24:17 2024
NAMESPACE: kyverno
STATUS: deployed
REVISION: 1
NOTES:
Chart version: 3.2.0-rc.3
Kyverno version: v1.12.0-rc.3

Thank you for installing kyverno! Your release is named kyverno.

The following components have been installed in your cluster:
- CRDs
- Admission controller
- Reports controller
- Cleanup controller
- Background controller

# 설치 확인
$ kubectl get all -n kyverno
NAME                                                 READY   STATUS    RESTARTS   AGE
pod/kyverno-admission-controller-69665dff5-vctbk     1/1     Running   0          50s
pod/kyverno-background-controller-56bc88f4dc-7fmzh   1/1     Running   0          50s
pod/kyverno-cleanup-controller-64448c5b4d-zfgkn      1/1     Running   0          50s
pod/kyverno-reports-controller-6bbd8f8d4-5kwqp       1/1     Running   0          50s

NAME                                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/kyverno-background-controller-metrics   ClusterIP   10.100.222.47    <none>        8000/TCP   51s
service/kyverno-cleanup-controller              ClusterIP   10.100.192.47    <none>        443/TCP    51s
service/kyverno-cleanup-controller-metrics      ClusterIP   10.100.244.149   <none>        8000/TCP   51s
service/kyverno-reports-controller-metrics      ClusterIP   10.100.105.27    <none>        8000/TCP   51s
service/kyverno-svc                             ClusterIP   10.100.170.237   <none>        443/TCP    51s
service/kyverno-svc-metrics                     ClusterIP   10.100.140.55    <none>        8000/TCP   51s

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kyverno-admission-controller    1/1     1            1           51s
deployment.apps/kyverno-background-controller   1/1     1            1           51s
deployment.apps/kyverno-cleanup-controller      1/1     1            1           51s
deployment.apps/kyverno-reports-controller      1/1     1            1           51s

NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/kyverno-admission-controller-69665dff5     1         1         1       51s
replicaset.apps/kyverno-background-controller-56bc88f4dc   1         1         1       51s
replicaset.apps/kyverno-cleanup-controller-64448c5b4d      1         1         1       51s
replicaset.apps/kyverno-reports-controller-6bbd8f8d4       1         1         1       51s

NAME                                                      SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/kyverno-cleanup-admission-reports           */10 * * * *   False     0        <none>          51s
cronjob.batch/kyverno-cleanup-cluster-admission-reports   */10 * * * *   False     0        <none>          51s

$ kubectl get crd | grep kyverno
admissionreports.kyverno.io                  2024-04-13T22:24:21Z
backgroundscanreports.kyverno.io             2024-04-13T22:24:21Z
cleanuppolicies.kyverno.io                   2024-04-13T22:24:21Z
clusteradmissionreports.kyverno.io           2024-04-13T22:24:21Z
clusterbackgroundscanreports.kyverno.io      2024-04-13T22:24:21Z
clustercleanuppolicies.kyverno.io            2024-04-13T22:24:21Z
clusterephemeralreports.reports.kyverno.io   2024-04-13T22:24:21Z
clusterpolicies.kyverno.io                   2024-04-13T22:24:22Z
ephemeralreports.reports.kyverno.io          2024-04-13T22:24:21Z
globalcontextentries.kyverno.io              2024-04-13T22:24:21Z
policies.kyverno.io                          2024-04-13T22:24:22Z
policyexceptions.kyverno.io                  2024-04-13T22:24:21Z
updaterequests.kyverno.io                    2024-04-13T22:24:21Z

$ kubectl get pod,svc -n kyverno
NAME                                                 READY   STATUS    RESTARTS   AGE
pod/kyverno-admission-controller-69665dff5-vctbk     1/1     Running   0          82s
pod/kyverno-background-controller-56bc88f4dc-7fmzh   1/1     Running   0          82s
pod/kyverno-cleanup-controller-64448c5b4d-zfgkn      1/1     Running   0          82s
pod/kyverno-reports-controller-6bbd8f8d4-5kwqp       1/1     Running   0          82s

NAME                                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/kyverno-background-controller-metrics   ClusterIP   10.100.222.47    <none>        8000/TCP   83s
service/kyverno-cleanup-controller              ClusterIP   10.100.192.47    <none>        443/TCP    83s
service/kyverno-cleanup-controller-metrics      ClusterIP   10.100.244.149   <none>        8000/TCP   83s
service/kyverno-reports-controller-metrics      ClusterIP   10.100.105.27    <none>        8000/TCP   83s
service/kyverno-svc                             ClusterIP   10.100.170.237   <none>        443/TCP    83s
service/kyverno-svc-metrics                     ClusterIP   10.100.140.55    <none>        8000/TCP   83s

# step-cli 설치: 기본 인증서 확인 
$ wget https://dl.smallstep.com/cli/docs-cli-install/latest/step-cli_amd64.rpm
$ sudo rpm -i step-cli_amd64.rpm

# 인증서 정보 조회 
$ kubectl -n kyverno get secret
NAME                                                      TYPE                 DATA   AGE
kyverno-cleanup-controller.kyverno.svc.kyverno-tls-ca     kubernetes.io/tls    2      3m32s
kyverno-cleanup-controller.kyverno.svc.kyverno-tls-pair   kubernetes.io/tls    2      3m30s
kyverno-svc.kyverno.svc.kyverno-tls-ca                    kubernetes.io/tls    2      3m24s
kyverno-svc.kyverno.svc.kyverno-tls-pair                  kubernetes.io/tls    2      3m21s
sh.helm.release.v1.kyverno.v1                             helm.sh/release.v1   1      3m46s

$ kubectl -n kyverno get secret kyverno-svc.kyverno.svc.kyverno-tls-ca -o jsonpath='{.data.tls\.crt}' | base64 -d
$ kubectl -n kyverno get secret kyverno-svc.kyverno.svc.kyverno-tls-ca -o jsonpath='{.data.tls\.crt}' | base64 -d | step certificate inspect --short
X.509v3 Root CA Certificate (RSA 2048) [Serial: 0]
  Subject:     *.kyverno.svc
  Issuer:      *.kyverno.svc
  Valid from:  2024-04-13T21:24:43Z
          to:  2025-04-13T22:24:43Z

$ kubectl get validatingwebhookconfiguration kyverno-policy-validating-webhook-cfg -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | step certificate inspect --short
X.509v3 Root CA Certificate (RSA 2048) [Serial: 0]
  Subject:     *.kyverno.svc
  Issuer:      *.kyverno.svc
  Valid from:  2024-04-13T21:24:43Z
          to:  2025-04-13T22:24:43Z

 

Kyverno - Prometheus Stack 연동 

Prometheus와 연동하여 Kyverno의 메트릭을 수집하도록 합니다. 

 

Prometheus (Kyverno) - Table

Prometheus (Kyverno) - Service Discovery

Prometheus (Kyverno) - Targets

그리고 Grafana 대시보드를 하나 띄워서 Keyverno의 변화를 모니터링합니다. 

 

Grafana (Kyverno) - 1

Grafana (Kyverno) - 2

Grafana (Kyverno) - 3

Grafana (Kyverno) - 4

Grafana (Kyverno) - 5

 

Policy & Rule

Policy & Rule

• 규칙(Rule)은 matah/exclude 선언 및 validate/mutate/generate/verify images 선언 중 하나로 구성됩니다. 
• 각 규칙에는 단일 validate, mutate, generate, verify images 하위 선언만 포함될 수 있습니다. 

 

Kyverno Validation 

label 'team'이 명시되어야 파드가 배포되는 정책입니다.

 

# ClusterPolicy 적용
$ kubectl create -f- << EOF
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-labels
spec:
  validationFailureAction: Enforce
  rules:
  - name: check-team
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "label 'team' is required"
      pattern:
        metadata:
          labels:
            team: "?*"
EOF
clusterpolicy.kyverno.io/require-labels created

# 확인
$ kubectl get validatingwebhookconfigurations
NAME                                            WEBHOOKS   AGE
aws-load-balancer-webhook                       3          11h
eks-aws-auth-configmap-validation-webhook       1          12h
kube-prometheus-stack-admission                 1          11h
kyverno-cleanup-validating-webhook-cfg          1          27m
kyverno-exception-validating-webhook-cfg        1          27m
kyverno-global-context-validating-webhook-cfg   1          27m
kyverno-policy-validating-webhook-cfg           1          27m
kyverno-resource-validating-webhook-cfg         1          27m
kyverno-ttl-validating-webhook-cfg              1          27m
vpc-resource-validating-webhook                 2          12h

$ kubectl get ClusterPolicy
NAME             ADMISSION   BACKGROUND   VALIDATE ACTION   READY   AGE   MESSAGE
require-labels   true        true         Enforce           True    84s   Ready

# 디플로이먼트 생성 시도
$ kubectl create deployment nginx --image=nginx
error: failed to create deployment: admission webhook "validate.kyverno.svc-fail" denied the request:

resource Deployment/default/nginx was blocked due to the following policies

require-labels:
  autogen-check-team: 'validation error: label ''team'' is required. rule autogen-check-team
    failed at path /spec/template/metadata/labels/team/'

# 디플로이먼트 생성 시도
$ kubectl run nginx --image nginx --labels team=backend
pod/nginx created

$ kubectl get pod -l team=backend
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          11s

# 확인
$ kubectl get policyreport -o wide
NAME                                   KIND         NAME                          PASS   FAIL   WARN   ERROR   SKIP   AGE
6763ef75-3178-49ff-a1a9-c79be32223fc   Pod          nginx                         1      0      0      0       0      8s

$ kubectl get policyreport 6763ef75-3178-49ff-a1a9-c79be32223fc -o yaml | kubectl neat | yh
apiVersion: wgpolicyk8s.io/v1alpha2
kind: PolicyReport
metadata:
  labels:
    app.kubernetes.io/managed-by: kyverno
  name: 6763ef75-3178-49ff-a1a9-c79be32223fc
  namespace: default
results:
- message: validation rule 'check-team' passed.
  policy: require-labels
  result: pass
  rule: check-team
  scored: true
  source: kyverno
  timestamp:
    nanos: 0
    seconds: 1713048785
scope:
  apiVersion: v1
  kind: Pod
  name: nginx
  namespace: default
  uid: 6763ef75-3178-49ff-a1a9-c79be32223fc
summary:
  error: 0
  fail: 0
  pass: 1
  skip: 0
  warn: 0

# 정책 삭제
$ kubectl delete clusterpolicy require-labels
clusterpolicy.kyverno.io "require-labels" deleted

 

Kyverno Mutation 

label 'team'이 없으면 'team=bravo'를 추가하고, 'team'이 있으면 해당 값이 우선하는 정책입니다. 

 

$ kubectl create -f- << EOF
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-labels
spec:
  rules:
  - name: add-team
    match:
      any:
      - resources:
          kinds:
          - Pod
    mutate:
      patchStrategicMerge:
        metadata:
          labels:
            +(team): bravo
EOF
clusterpolicy.kyverno.io/add-labels created

# 확인
$ kubectl get mutatingwebhookconfigurations
NAME                                    WEBHOOKS   AGE
aws-load-balancer-webhook               3          11h
kube-prometheus-stack-admission         1          11h
kyverno-policy-mutating-webhook-cfg     1          37m
kyverno-resource-mutating-webhook-cfg   1          37m
kyverno-verify-mutating-webhook-cfg     1          37m
pod-identity-webhook                    1          12h
vpc-resource-mutating-webhook           1          12h

$ kubectl get ClusterPolicy
NAME         ADMISSION   BACKGROUND   VALIDATE ACTION   READY   AGE   MESSAGE
add-labels   true        true         Audit             True    24s   Ready

# 파드 생성 후 label 확인
$ kubectl run redis --image redis
pod/redis created
$ kubectl get pod redis --show-labels
NAME    READY   STATUS    RESTARTS   AGE   LABELS
redis   1/1     Running   0          8s    run=redis,team=bravo

# 파드 생성 후 label 확인
$ kubectl run newredis --image redis -l team=alpha
pod/newredis created
$ kubectl get pod newredis --show-labels
NAME       READY   STATUS    RESTARTS   AGE   LABELS
newredis   1/1     Running   0          8s    team=alpha

# 삭제
$ kubectl delete clusterpolicy add-labels
clusterpolicy.kyverno.io "add-labels" deleted

 

Kyverno Generation 

신규 네임스페이스 생성 시 default 네임스페이스의 Secret을 Clone(복제)하는 정책입니다. 

 

# First, create this Kubernetes Secret in your cluster which will simulate a real image pull secret.
$ kubectl -n default create secret docker-registry regcred \
  --docker-server=myinternalreg.corp.com \
  --docker-username=john.doe \
  --docker-password=Passw0rd123! \
  --docker-email=john.doe@corp.com
secret/regcred created

$ kubectl get secret regcred
NAME      TYPE                             DATA   AGE
regcred   kubernetes.io/dockerconfigjson   1      18s

# Policy 생성 
$ kubectl create -f- << EOF
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: sync-secrets
spec:
  rules:
  - name: sync-image-pull-secret
    match:
      any:
      - resources:
          kinds:
          - Namespace
    generate:
      apiVersion: v1
      kind: Secret
      name: regcred
      namespace: "{{request.object.metadata.name}}"
      synchronize: true
      clone:
        namespace: default
        name: regcred
EOF
clusterpolicy.kyverno.io/sync-secrets created

# Policy 생성 확인 
kubectl get ClusterPolicy
NAME           ADMISSION   BACKGROUND   VALIDATE ACTION   READY   AGE   MESSAGE
sync-secrets   true        true         Audit             True    10s   Ready

# 신규 네임스페이스 생성 후 확인
$ kubectl create ns mytestns
namespace/mytestns created
$ kubectl -n mytestns get secret
NAME      TYPE                             DATA   AGE
regcred   kubernetes.io/dockerconfigjson   1      8s

# 삭제
$ kubectl delete clusterpolicy sync-secrets
clusterpolicy.kyverno.io "sync-secrets" deleted

 

 

---

[출처]

1) CloudNet@, AEWS 실습 스터디

2) https://kyverno.io/

Kyverno

3) https://kyverno.io/docs/introduction/

Introduction

4) https://www.eksworkshop.com/docs/security/kyverno/

Policy management with Kyverno | EKS Workshop

5) https://aws.amazon.com/ko/blogs/containers/managing-pod-security-on-amazon-eks-with-kyverno/

Managing Pod Security on Amazon EKS with Kyverno | Amazon Web Services

6) https://aws.amazon.com/ko/blogs/containers/implementing-pod-security-standards-in-amazon-eks/

Implementing Pod Security Standards in Amazon EKS | Amazon Web Services

7) https://artifacthub.io/packages/helm/kyverno/kyverno

kyverno 3.2.0-rc.4 · kyverno/kyverno

8) https://kyverno.io/docs/installation/platform-notes/#notes-for-eks-users

Platform Notes

9) https://kyverno.io/docs/kyverno-policies/

Policies and Rules