110.10 e

Description: This is values file and setup setting for install Loki. Especially when you using some custom CIDR, you will get memberlist private error. After looking at the GitHub issue, I realized that the error was caused by the Helm package, and I got a clue.

#100#Infra#110#DevOps_Engineer_Infra#110.10#Kubernetes#110.10 e#Loki_with_Helm

아래의 IAM Role과 Policy의 경우 Loki가 s3를 쓰기위한 역할관 권한입니다. 로그 수집을 로컬로 하는 경우 해당 로그를 찾을때 각 노드에 따로따로 자장되면 에러가 발생합니다.(노드1에는 있지만 노드2에는 없는 로그를 노드2가 로컬에서 찾을때 라던가)
그래서 로그를 적절한 공용 저장소인 오브젝트스토리지에 저장하는 것이 중요합니다.

OIDC를 이용해서 LOKI POD는 service account형태로 AWS IAM의 권한을 사용하게 됩니다.

#AWS

AWS IAM
CHANGE

trustrelationships ROLE

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::123456784321:oidc-provider/oidc.eks.ap-northeast-2.amazonaws.com/id/A1B2C3D4E5F6078901234567890ABCD"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.ap-northeast-2.amazonaws.com/id/A1B2C3D4E5F6078901234567890ABCD:sub": "system:serviceaccount:[NAMESPACE]:[SERVICEACCOUNT_NAME]"
                }
            }
        }
    ]
}

s3_loki Policy
Change

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME*/*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME"
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME",
                "arn:aws:s3:::YOUR_BUCKET_NAME/*"
            ]
        }
    ]
}

values.yaml
change

loki:
  auth_enabled: false

  serviceAccount:
    create: true
    name: loki-sa
    namespace: monitoring
    annotations:
      eks.amazonaws.com/role-arn: "arn:aws:iam::123456784321:role/[CUSTOM_ROLE_NAME]"

  extraArgs:
    config.expand-env: "true"

  extraEnv:
    - name: MY_POD_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP

  config:
    memberlist:
      bind_addr:
        - ${MY_POD_IP}

    schema_config:
      configs:
        - from: 2021-05-12
          store: boltdb-shipper
          object_store: s3
          schema: v11
          index:
            prefix: loki_index_
            period: 24h

    storage_config:
      aws:
        s3: s3://<BUCKET_NAME>
        s3forcepathstyle: true
        bucketnames: <BUCKET_NAME>
        region: <YOUR_REGION>
        insecure: false
        sse_encryption: false
      boltdb_shipper:
        shared_store: s3
        cache_ttl: 24h
        active_index_directory: /data/loki/index
        cache_location: /data/loki/cache
    table_manager:
      retention_deletes_enabled: true
      retention_period: 300h  # Make sure this is correctly a duration, e.g., "720h" for 30 days if needed




grafana:
    enabled: false
    sidecar:
      datasources:
        enabled: true
    image:
      tag:  7.5.17 # <--there is a bug in grafana https://github.com/grafana/loki/issues/8136
      users:
        default_theme: dark

Reference
https://community.grafana.com/t/gossip-ring-memberlist-no-private-ip-address-found/52209/7