110.10 e
Description: This is values file and setup setting for install Loki. Especially when you using some custom CIDR, you will get memberlist private error. After looking at the GitHub issue, I realized that the error was caused by the Helm package, and I got a clue.
#100#Infra#110#DevOps_Engineer_Infra#110.10#Kubernetes#110.10 e#Loki_with_Helm
아래의 IAM Role과 Policy의 경우 Loki가 s3를 쓰기위한 역할관 권한입니다. 로그 수집을 로컬로 하는 경우 해당 로그를 찾을때 각 노드에 따로따로 자장되면 에러가 발생합니다.(노드1에는 있지만 노드2에는 없는 로그를 노드2가 로컬에서 찾을때 라던가)
그래서 로그를 적절한 공용 저장소인 오브젝트스토리지에 저장하는 것이 중요합니다.
OIDC를 이용해서 LOKI POD는 service account형태로 AWS IAM의 권한을 사용하게 됩니다.
AWS IAM
CHANGE
- 123456784321-> Your AWS Account
- ap-northeast-2 -> Your Region
- A1B2C3D4E5F6078901234567890ABCD -> Your EKS OIDC(You can see in UI or command)
- NAMESPACE -> Loki installed namespace
- SERVICEACCOUNT_NAME -> Service account in values yaml or check from EKS cluster
trustrelationships ROLE
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::123456784321:oidc-provider/oidc.eks.ap-northeast-2.amazonaws.com/id/A1B2C3D4E5F6078901234567890ABCD"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.ap-northeast-2.amazonaws.com/id/A1B2C3D4E5F6078901234567890ABCD:sub": "system:serviceaccount:[NAMESPACE]:[SERVICEACCOUNT_NAME]"
}
}
}
]
}
s3_loki Policy
Change
- YOUR_BUCKET_NAME -> BUCKETNAME for saving files from loki
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": "arn:aws:s3:::YOUR_BUCKET_NAME*/*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::YOUR_BUCKET_NAME"
},
{
"Sid": "VisualEditor2",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:ListBucket",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::YOUR_BUCKET_NAME",
"arn:aws:s3:::YOUR_BUCKET_NAME/*"
]
}
]
}
values.yaml
change
- 123456784321-> YOUR AWS account
- CUSTOM_ROLE_NAME -> Name of IAM role just before you create
- <BUCKET_NAME> -> The bucket name you allowed in IAM in the previous step
- <YOUR_REGION> -> YOUR BUCKET. REGION
loki:
auth_enabled: false
serviceAccount:
create: true
name: loki-sa
namespace: monitoring
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::123456784321:role/[CUSTOM_ROLE_NAME]"
extraArgs:
config.expand-env: "true"
extraEnv:
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
config:
memberlist:
bind_addr:
- ${MY_POD_IP}
schema_config:
configs:
- from: 2021-05-12
store: boltdb-shipper
object_store: s3
schema: v11
index:
prefix: loki_index_
period: 24h
storage_config:
aws:
s3: s3://<BUCKET_NAME>
s3forcepathstyle: true
bucketnames: <BUCKET_NAME>
region: <YOUR_REGION>
insecure: false
sse_encryption: false
boltdb_shipper:
shared_store: s3
cache_ttl: 24h
active_index_directory: /data/loki/index
cache_location: /data/loki/cache
table_manager:
retention_deletes_enabled: true
retention_period: 300h # Make sure this is correctly a duration, e.g., "720h" for 30 days if needed
grafana:
enabled: false
sidecar:
datasources:
enabled: true
image:
tag: 7.5.17 # <--there is a bug in grafana https://github.com/grafana/loki/issues/8136
users:
default_theme: dark
Reference
https://community.grafana.com/t/gossip-ring-memberlist-no-private-ip-address-found/52209/7