Deploy to Production

Production runs on AWS EKS with 2 replicas per service, managed data stores, and approval gates for deployments. Estimated cost is approximately $213/month.

Prerequisites

• Staging environment running and verified (see Deploy to Staging)
• All CI checks passing on main
• Staging has been running the candidate build for at least 24 hours
• Rollback plan documented (previous working image tags noted)

Release Checklist

Before promoting staging to production, verify:

• All CI checks pass on main
• Staging has been running the candidate build for at least 24 hours
• Health checks all green on staging (/health, /ready for every service)
• No error spikes in staging logs / monitoring
• Database migrations tested on staging
• Manual smoke tests pass (critical user flows)
• Performance baseline acceptable (no regressions)
• Rollback plan documented

Step 1: Provision Production Infrastructure

cd infra/terraform/environments/production

terraform init
terraform plan -out=plan.tfplan
# Review the plan carefully
terraform apply plan.tfplan

This creates:

• EKS cluster with managed node groups
• RDS db.r6g.large (PostgreSQL 16 + pgvector)
• ElastiCache r6g.large (Redis 7)
• CloudFront distributions for web and admin
• Route53 DNS records
• S3 buckets for backups

Step 2: Create a Release Tag

git checkout main
git pull origin main

# Create a release tag
git tag -a release/v1.2.0 -m "Release v1.2.0: <summary>"
git push origin release/v1.2.0

This triggers the cd-production.yml workflow:

1. Full regression test suite runs
2. Approval gate: a reviewer must approve in GitHub
3. Images are built, tagged with the version, and pushed to ECR
4. Production Kustomize overlay is updated
5. ArgoCD syncs production

Step 3: Monitor the Deployment

After ArgoCD reports sync complete:

# Check all pods are running and ready
kubectl get pods -n gospelib-production

# Hit production health endpoints
curl https://api.gospelib.com/health
curl https://api.gospelib.com/ready

# Check the web app
curl -I https://gospelib.com

# Monitor logs for errors
kubectl logs -f -l app=gospelib-gateway -n gospelib-production --since=5m
kubectl logs -f -l app=gospelib-content -n gospelib-production --since=5m

Step 4: Post-Release Verification

1. Monitor error rates and latency for 1 hour
2. Check Grafana dashboards for any anomalies
3. If stable, announce the release
4. If issues emerge, follow the Rollback Procedures

Frontend Deployment

Web and Admin apps deploy to AWS Amplify automatically:

App	Trigger	URL
Web	Push to main (apps/web/**)	gospelib.com
Admin	Push to main (apps/admin/**)	admin.gospelib.com

Mobile builds go through EAS:

Platform	Trigger	Distribution
iOS	Push to main (apps/mobile/**)	TestFlight → App Store
Android	Push to main (apps/mobile/**)	Internal track → Play Store

Database Migrations

PostgreSQL

Migrations use golang-migrate. Run before deploying new code:

# Check current version
migrate -source "file://services/auth/migrations" \
  -database "$PG_URL" version

# Apply pending migrations
migrate -source "file://services/auth/migrations" \
  -database "$PG_URL" up

warning

Always test migrations on staging first. Data-destructive down migrations (dropping columns/tables) cannot be undone.

FalkorDB

FalkorDB does not have traditional migrations. The ingest pipeline is the sole writer:

# Incremental ingest
kubectl create job ingest-manual-$(date +%s) \
  --from=cronjob/ingest-incremental \
  -n gospelib-production

Secrets Management

Production secrets are stored in AWS Secrets Manager and synced to Kubernetes via External Secrets Operator:

# infra/k8s/overlays/production/external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: gospelib-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: gospelib-secrets
  data:
    - secretKey: CLERK_SECRET_KEY
      remoteRef:
        key: gospelib/production/clerk-secret-key

Emergency Hotfix

For critical production bugs that cannot wait for the normal release cycle:

# 1. Create hotfix branch from the release tag
git checkout -b hotfix/critical-fix release/v1.2.3

# 2. Apply fix, commit, push
git push origin hotfix/critical-fix

# 3. Build and push the affected service image
docker build -t $ECR_URL/gospelib-gateway:hotfix-$GIT_SHA services/gateway/
docker push $ECR_URL/gospelib-gateway:hotfix-$GIT_SHA

# 4. Update production overlay directly
cd infra/k8s/overlays/production
kustomize edit set image gospelib-gateway=$ECR_URL/gospelib-gateway:hotfix-$GIT_SHA
kustomize build . | kubectl apply -f -

# 5. Tag the hotfix and merge back to main
git tag release/v1.2.4
git push origin release/v1.2.4
git checkout main && git merge hotfix/critical-fix && git push origin main

Cost Breakdown

Resource	Instance	Monthly Cost
EKS cluster	1 cluster	~$73
EC2 nodes	2× t3.medium	~$60
RDS PostgreSQL	db.r6g.large	~$50
ElastiCache Redis	cache.r6g.large	~$25
S3 + CloudFront	—	~$5
Total		~$213