110.10 e

Description: This is values file and setup setting for install Loki. Especially when you using some custom CIDR, you will get memberlist private error. After looking at the GitHub issue, I realized that the error was caused by the Helm package, and I got a clue.

#100 #Infra#110 #DevOps_Engineer_Infra#110.10 #Kubernetes#110.10 e #Loki_with_Helm

아래의 IAM Role과 Policy의 경우 Loki가 s3를 쓰기위한 역할관 권한입니다. 로그 수집을 로컬로 하는 경우 해당 로그를 찾을때 각 노드에 따로따로 자장되면 에러가 발생합니다.(노드1에는 있지만 노드2에는 없는 로그를 노드2가 로컬에서 찾을때 라던가)
그래서 로그를 적절한 공용 저장소인 오브젝트스토리지에 저장하는 것이 중요합니다.

OIDC를 이용해서 LOKI POD는 service account형태로 AWS IAM의 권한을 사용하게 됩니다.

#AWS

AWS IAM
CHANGE

123456784321-> Your AWS Account
ap-northeast-2 -> Your Region
A1B2C3D4E5F6078901234567890ABCD -> Your EKS OIDC(You can see in UI or command)
NAMESPACE -> Loki installed namespace
SERVICEACCOUNT_NAME -> Service account in values yaml or check from EKS cluster

trustrelationships ROLE

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::123456784321:oidc-provider/oidc.eks.ap-northeast-2.amazonaws.com/id/A1B2C3D4E5F6078901234567890ABCD"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.ap-northeast-2.amazonaws.com/id/A1B2C3D4E5F6078901234567890ABCD:sub": "system:serviceaccount:[NAMESPACE]:[SERVICEACCOUNT_NAME]"
                }
            }
        }
    ]
}

s3_loki Policy
Change

YOUR_BUCKET_NAME -> BUCKETNAME for saving files from loki

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME*/*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME"
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME",
                "arn:aws:s3:::YOUR_BUCKET_NAME/*"
            ]
        }
    ]
}

values.yaml
change

123456784321-> YOUR AWS account
CUSTOM_ROLE_NAME -> Name of IAM role just before you create
<BUCKET_NAME> -> The bucket name you allowed in IAM in the previous step
<YOUR_REGION> -> YOUR BUCKET. REGION

loki:
  auth_enabled: false

  serviceAccount:
    create: true
    name: loki-sa
    namespace: monitoring
    annotations:
      eks.amazonaws.com/role-arn: "arn:aws:iam::123456784321:role/[CUSTOM_ROLE_NAME]"

  extraArgs:
    config.expand-env: "true"

  extraEnv:
    - name: MY_POD_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP

  config:
    memberlist:
      bind_addr:
        - ${MY_POD_IP}

    schema_config:
      configs:
        - from: 2021-05-12
          store: boltdb-shipper
          object_store: s3
          schema: v11
          index:
            prefix: loki_index_
            period: 24h

    storage_config:
      aws:
        s3: s3://<BUCKET_NAME>
        s3forcepathstyle: true
        bucketnames: <BUCKET_NAME>
        region: <YOUR_REGION>
        insecure: false
        sse_encryption: false
      boltdb_shipper:
        shared_store: s3
        cache_ttl: 24h
        active_index_directory: /data/loki/index
        cache_location: /data/loki/cache
    table_manager:
      retention_deletes_enabled: true
      retention_period: 300h  # Make sure this is correctly a duration, e.g., "720h" for 30 days if needed




grafana:
    enabled: false
    sidecar:
      datasources:
        enabled: true
    image:
      tag:  7.5.17 # <--there is a bug in grafana https://github.com/grafana/loki/issues/8136
      users:
        default_theme: dark

Reference
https://community.grafana.com/t/gossip-ring-memberlist-no-private-ip-address-found/52209/7