eveAI/documentation/Production Setup/phase-8-application-services.md

Phase 8: Application Services (Staging)

This guide describes how to deploy EveAI application services to the Scaleway Kubernetes cluster, building on Phases 1–7 in cluster-install.md.

Prerequisites

• Ingress-NGINX running with external IP
• cert-manager installed and Certificate evie-staging-tls is READY (via HTTP ACME first, then HTTPS-only)
• External Secrets Operator installed; Kubernetes Secret eveai-secrets exists in namespace eveai-staging
• Verification service deployed and reachable via /verify
• Optional: Monitoring stack running, Pushgateway deployed or reachable; PUSH_GATEWAY_HOST/PORT available to apps (via eveai-secrets)

What we deploy (structure)

• Frontend (web) services
  • eveai-app → exposed at /admin
  • eveai-api → exposed at /api
  • eveai-chat-client → exposed at /client
• Backend worker services (internal)
  • eveai-workers (queue: embeddings)
  • eveai-chat-workers (queue: llm_interactions)
  • eveai-entitlements (queue: entitlements)
• Ops Jobs (manual DB ops)
  • 00-env-check
  • 02-db-bootstrap-ext
  • 03-db-migrate-public
  • 04-db-migrate-tenant
  • 05-seed-or-init-data
  • 06-verify-minimal

Manifests are under:

• scaleway/manifests/base/applications/frontend/
• scaleway/manifests/base/applications/backend/
• scaleway/manifests/base/applications/ops/jobs/
• Aggregate kustomization: scaleway/manifests/base/applications/kustomization.yaml

Step 1: Validate secrets

kubectl get secret eveai-secrets -n eveai-staging
kubectl get secret eveai-secrets -n eveai-staging -o jsonpath='{.data}' | jq 'keys'

Confirm presence of DB_, REDIS_, OPENAI_API_KEY, MISTRAL_API_KEY, JWT_SECRET_KEY, API_ENCRYPTION_KEY, MINIO_*, PUSH_GATEWAY_HOST, PUSH_GATEWAY_PORT.

Step 2: Deploy Ops Jobs (manual pre-deploy)

Run the DB ops scripts manually in order. Each manifest uses generateName; use kubectl create.

kubectl create -f scaleway/manifests/base/applications/ops/jobs/00-env-check-job.yaml
kubectl wait --for=condition=complete job -n eveai-staging -l job-type=env-check --timeout=600s

kubectl create -f scaleway/manifests/base/applications/ops/jobs/02-db-bootstrap-ext-job.yaml
kubectl wait --for=condition=complete job -n eveai-staging -l job-type=db-bootstrap-ext --timeout=1800s

kubectl create -f scaleway/manifests/base/applications/ops/jobs/03-db-migrate-public-job.yaml
kubectl wait --for=condition=complete job -n eveai-staging -l job-type=db-migrate-public --timeout=1800s

kubectl create -f scaleway/manifests/base/applications/ops/jobs/04-db-migrate-tenant-job.yaml
kubectl wait --for=condition=complete job -n eveai-staging -l job-type=db-migrate-tenant --timeout=3600s

kubectl create -f scaleway/manifests/base/applications/ops/jobs/05-seed-or-init-data-job.yaml
kubectl wait --for=condition=complete job -n eveai-staging -l job-type=db-seed-or-init --timeout=1800s

kubectl create -f scaleway/manifests/base/applications/ops/jobs/06-verify-minimal-job.yaml
kubectl wait --for=condition=complete job -n eveai-staging -l job-type=db-verify-minimal --timeout=900s

View logs:

kubectl -n eveai-staging get jobs
kubectl -n eveai-staging logs job/<created-job-name>

Step 3: Deploy backend workers

kubectl apply -k scaleway/manifests/base/applications/backend/

kubectl -n eveai-staging get deploy | egrep 'eveai-(workers|chat-workers|entitlements)'
# Optional: quick logs
kubectl -n eveai-staging logs deploy/eveai-workers --tail=100 || true
kubectl -n eveai-staging logs deploy/eveai-chat-workers --tail=100 || true
kubectl -n eveai-staging logs deploy/eveai-entitlements --tail=100 || true

Step 4: Deploy frontend services

kubectl apply -k scaleway/manifests/base/applications/frontend/

kubectl -n eveai-staging get deploy,svc | egrep 'eveai-(app|api|chat-client)'

Step 5: Verify Ingress routes

The HTTPS ingress has paths enabled for /admin, /api, /client. Verify:

kubectl -n eveai-staging describe ingress eveai-staging-ingress

curl -k https://evie-staging.askeveai.com/verify/health
curl -k https://evie-staging.askeveai.com/admin/healthz/ready
curl -k https://evie-staging.askeveai.com/api/healthz/ready
curl -k https://evie-staging.askeveai.com/client/healthz/ready

Resources and probes (staging defaults)

• Web (app, api, chat-client):
  • requests: 150m CPU, 256Mi RAM; limits: 500m CPU, 512Mi RAM; replicas: 1
  • readiness/liveness: GET /healthz/ready
• Workers:
  • eveai-workers: req 200m/512Mi, lim 1CPU/1Gi
  • eveai-chat-workers: req 500m/1Gi, lim 2CPU/3Gi
  • eveai-entitlements: req 100m/256Mi, lim 500m/512Mi

Pushgateway usage

• Ensure PUSH_GATEWAY_HOST and PUSH_GATEWAY_PORT are provided (e.g., pushgateway.monitoring.svc.cluster.local:9091), typically via eveai-secrets or a ConfigMap.
• Apps will continue to push business metrics; Prometheus scrapes the Pushgateway.

Bunny.net WAF (TODO)

• Configure Pull Zone for evie-staging.askeveai.com
• Set Origin to the LoadBalancer IP with HTTPS and Host header evie-staging.askeveai.com
• Define rate limits primarily on /api, looser on /client; enable bot filtering
• Only switch DNS (CNAME) to Bunny after TLS issuance completed directly against LoadBalancer

Troubleshooting

kubectl get all -n eveai-staging
kubectl get events -n eveai-staging --sort-by=.lastTimestamp
kubectl describe ingress eveai-staging-ingress -n eveai-staging
kubectl logs -n eveai-staging deploy/eveai-api --tail=200

Rollback / Cleanup

# Remove frontend/backend (keeps verification and other base resources)
kubectl delete -k scaleway/manifests/base/applications/frontend/
kubectl delete -k scaleway/manifests/base/applications/backend/

# Jobs are kept for history due to ttlSecondsAfterFinished; to delete immediately:
kubectl -n eveai-staging delete jobs --all