fa452e4934
- Change instructions for deploying Prometheus stack and Pushgateway - Additional grouping to pushgateway to avoid overwriting of metrics in different pods / processes - Bugfix to ensure good retrieval of css en js files in eveai_app
80 lines
4.2 KiB
Markdown
80 lines
4.2 KiB
Markdown
# Pushgateway Grouping Keys (instance, namespace, process)
|
|
|
|
Goal: prevent metrics pushed by different Pods or worker processes from overwriting each other, while keeping Prometheus/Grafana queries simple.
|
|
|
|
Summary of decisions
|
|
- WORKER_ID source = OS process ID (PID)
|
|
- Always include namespace in grouping labels
|
|
|
|
What this changes
|
|
- Every push to Prometheus Pushgateway now includes a grouping_key with:
|
|
- instance = POD_NAME (fallback to HOSTNAME, then "dev")
|
|
- namespace = POD_NAMESPACE (fallback to ENVIRONMENT, then "dev")
|
|
- process = WORKER_ID (fallback to current PID)
|
|
- Prometheus will expose these as exported_instance, exported_namespace, and exported_process on the scraped series.
|
|
|
|
Code changes (already implemented)
|
|
- common/utils/business_event.py
|
|
- push_to_gateway(..., grouping_key={instance, namespace, process})
|
|
- Safe fallbacks ensure dev/test (Podman) keeps working with no K8s-specific env vars.
|
|
|
|
Kubernetes manifests (already implemented)
|
|
- All Deployments that push metrics set env vars via Downward API:
|
|
- POD_NAME from metadata.name
|
|
- POD_NAMESPACE from metadata.namespace
|
|
- Files updated:
|
|
- scaleway/manifests/base/applications/frontend/eveai-app/deployment.yaml
|
|
- scaleway/manifests/base/applications/frontend/eveai-api/deployment.yaml
|
|
- scaleway/manifests/base/applications/frontend/eveai-chat-client/deployment.yaml
|
|
- scaleway/manifests/base/applications/backend/eveai-workers/deployment.yaml
|
|
- scaleway/manifests/base/applications/backend/eveai-chat-workers/deployment.yaml
|
|
- scaleway/manifests/base/applications/backend/eveai-entitlements/deployment.yaml
|
|
|
|
No changes needed to secrets
|
|
- PUSH_GATEWAY_HOST/PORT remain provided via eveai-secrets; code composes PUSH_GATEWAY_URL internally.
|
|
|
|
How to verify
|
|
1) Pushgateway contains per-pod/process groups
|
|
- Port-forward Pushgateway (namespace monitoring):
|
|
- kubectl -n monitoring port-forward svc/monitoring-pushgateway-prometheus-pushgateway 9091:9091
|
|
- Inspect:
|
|
- curl -s http://127.0.0.1:9091/api/v1/metrics | jq '.[].labels'
|
|
- You should see labels including job (your service), instance (pod), namespace, process (pid).
|
|
|
|
2) Prometheus shows the labels as exported_*
|
|
- Port-forward Prometheus (namespace monitoring):
|
|
- kubectl -n monitoring port-forward svc/monitoring-prometheus 9090:9090
|
|
- Queries:
|
|
- label_values(eveai_llm_calls_total, exported_instance)
|
|
- label_values(eveai_llm_calls_total, exported_namespace)
|
|
- label_values(eveai_llm_calls_total, exported_process)
|
|
|
|
PromQL query patterns
|
|
- Hide per-process by aggregating away exported_process:
|
|
- sum without(exported_process) (rate(eveai_llm_calls_total[5m])) by (exported_job, exported_instance, exported_namespace)
|
|
- Service-level totals (hide instance and process):
|
|
- sum without(exported_instance, exported_process) (rate(eveai_llm_calls_total[5m])) by (exported_job, exported_namespace)
|
|
- Histogram example (p95 per service):
|
|
- histogram_quantile(0.95, sum without(exported_process) (rate(eveai_llm_duration_seconds_bucket[5m])) by (le, exported_job, exported_namespace))
|
|
|
|
Dev/Test (Podman) behavior
|
|
- No Kubernetes Downward API: POD_NAME/POD_NAMESPACE are not set.
|
|
- Fallbacks used by the code:
|
|
- instance = HOSTNAME if available, else "dev"
|
|
- namespace = ENVIRONMENT if available, else "dev"
|
|
- process = current PID
|
|
- This guarantees no crashes and still avoids process-level overwrites.
|
|
|
|
Operational notes
|
|
- Cardinality: adding process creates more series (one per worker). This is required to avoid data loss when multiple workers push concurrently. Dashboards should aggregate away exported_process unless you need per-worker detail.
|
|
- Batch jobs (future): use the same grouping and consider delete_from_gateway on successful completion to remove stale groups for that job/instance/process.
|
|
|
|
Troubleshooting
|
|
- If you still see overwriting:
|
|
- Confirm that instance, namespace, and process all appear in Pushgateway JSON labels for each group.
|
|
- Ensure that all pods set POD_NAME and POD_NAMESPACE (kubectl -n eveai-staging exec <pod> -- env | egrep "POD_NAME|POD_NAMESPACE").
|
|
- Verify that your app processes run push_to_gateway through the shared business_event wrapper.
|
|
|
|
Change log reference
|
|
- Implemented on 2025-09-26 by adding grouping_key in business_event push and env vars in Deployments.
|