edusav_cicd

Central pipeline repo for the Edusav polyrepo. Reusable Forgejo Actions workflows + config templates + a one-shot systemd bootstrap script. Lives only on Forgejo (git.edusav.com/edusav/edusav_cicd) — not mirrored to GitHub, because it references internal hostnames and ports.

See ~/edusav/cicd.md (private) for the full rollout plan; this README is the in-tree primer for anyone landing on the repo cold.

Layout

edusav_cicd/
├── .forgejo/workflows/
│   ├── go-service.yml            # build + deploy any Go multi-binary service
│   ├── bun-frontend.yml          # build + deploy Next.js standalone or Vite SPA
│   └── spring-boot.yml           # build + deploy the CDN JAR
├── deploy/
│   └── service-catalog.yml       # service → repo, binaries, ports, secrets, depends_on
├── bootstrap/
│   └── bootstrap-systemd.sh      # writes all forgejo-<env>-<svc>-<binary>.service units
└── configs/
    ├── edusav_IAM_service/{dev,stag}.yaml
    ├── edusav_gateway_service/{dev,stag}.yaml
    ├── edusav_trackhub_service/{dev,stag}.yaml
    ├── edusav_commerce_service/{dev,stag}.yaml
    ├── edusav_loyalty_service/{dev,stag}.yaml
    ├── edusav_wallet_service/{dev,stag}.yaml
    ├── edusav_messaging_service/{dev,stag}.yaml
    ├── edusav_payment_service/{dev,stag}.yaml
    ├── edusav_multimedia_service/{dev,stag}.yaml
    └── edusav_cdn_service/{dev,stag}.properties

How a service repo calls in

Each of the 13 service repos carries a ~12-line stub at .forgejo/workflows/ci.yml. Example for edusav_IAM_service:

name: ci
on:
  workflow_dispatch:
    inputs:
      target_env:
        type: choice
        options: [dev, stag]
        required: true

jobs:
  cicd:
    uses: edusav/edusav_cicd/.forgejo/workflows/go-service.yml@dev
    with:
      service-name: iam
      target-env: ${{ inputs.target_env }}
    secrets: inherit

service-name is the top-level key in deploy/service-catalog.yml. The reusable workflow looks up everything else (repo, binaries, ports, secrets list, host, needs_migration, depends_on) from there.

The three reusable workflows correspond to the catalog kind: field:

kind:	Workflow	Used by
go	go-service.yml	iam, gateway, trackhub-be, commerce, loyalty, wallet, messaging, payment, multimedia
bun-nextjs, bun-vite	bun-frontend.yml	front_commerce, fe_backofficeplatform, fe_trackhub
spring-boot	spring-boot.yml	cdn

How secrets flow

There is exactly one secret store: Forgejo org secrets at https://git.edusav.com/org/edusav/settings/actions/secrets.

• The runner reads every secret it needs at deploy time from ${{ secrets.X }}.
• The reusable workflow writes them onto the target host as a per-unit systemd drop-in at /etc/systemd/system/<unit>.service.d/secrets.conf, with one Environment="VAR=value" line per secret.
• The target-side config.yaml (or application-<env>.properties for cdn) at /opt/forgejo-<env>/<repo>/shared/ references those env-vars as ${VAR} placeholders. Viper (Go) / Spring Boot (cdn) expand them at process boot.
• No secret value is ever echoed to stdout, written to disk in the runner workspace, or committed to this repo.

Naming convention: <DOMAIN>_<FIELD>_<ENV>, e.g. DB_PASSWORD_DEV, TWILIO_AUTH_TOKEN_STAG. Every secret has both a _DEV and a _STAG variant even when the same value is used in both envs today — keeps the per-env split honest if a future rotation diverges.

Full secret list (~30 total)

Populate these in the Forgejo UI before the first workflow run. Drawn from ~/edusav/cicd_inventory.md:

# Infra (env-shared across services in the same env)
DB_PASSWORD_DEV               # already set to Bismillah123 on edustage
DB_PASSWORD_STAG              # already set to Bismillah123 on edustage
RABBITMQ_PASSWORD_DEV         # Bismillah123 on edustage (vhost /dev, user dev-edu)
RABBITMQ_PASSWORD_STAG        # Bismillah123 on edustage (vhost /stag, user stag-edu)
REDIS_PASSWORD_DEV            # empty string if Redis auth is off
REDIS_PASSWORD_STAG
ELASTICSEARCH_USER_DEV
ELASTICSEARCH_USER_STAG
ELASTICSEARCH_PASSWORD_DEV
ELASTICSEARCH_PASSWORD_STAG

# Cross-service shared (one value per env)
INTERNAL_API_KEY_DEV          # X-Internal-Api-Key; gateway + every service trust it
INTERNAL_API_KEY_STAG
JWT_SECRET_KEY_DEV            # IAM signs, gateway verifies — must match within env
JWT_SECRET_KEY_STAG

# IAM-specific (Google OAuth)
GOOGLE_OAUTH_CLIENT_ID_DEV
GOOGLE_OAUTH_CLIENT_ID_STAG
GOOGLE_OAUTH_CLIENT_SECRET_DEV
GOOGLE_OAUTH_CLIENT_SECRET_STAG

# MinIO (consumed by IAM, multimedia, messaging — two instances per env, distinct values)
MINIO_ACCESS_KEY_DEV
MINIO_ACCESS_KEY_STAG
MINIO_SECRET_KEY_DEV
MINIO_SECRET_KEY_STAG

# Messaging (Twilio — Account SID + Auth Token recorded 2026-05-19, see cicd.md §10)
TWILIO_ACCOUNT_SID_DEV
TWILIO_ACCOUNT_SID_STAG
TWILIO_AUTH_TOKEN_DEV
TWILIO_AUTH_TOKEN_STAG
TWILIO_WHATSAPP_FROM_DEV     # +14155238886 (Twilio Sandbox)
TWILIO_WHATSAPP_FROM_STAG

# Loyalty (hashid salt for short-link generation)
HASHID_SALT_DEV
HASHID_SALT_STAG

# CDN
CDN_INTERNAL_API_KEY_DEV
CDN_INTERNAL_API_KEY_STAG
AWS_S3_ACCESS_KEY_DEV         # if CDN's S3 is distinct from MinIO; otherwise reuse MINIO_*
AWS_S3_ACCESS_KEY_STAG
AWS_S3_SECRET_KEY_DEV
AWS_S3_SECRET_KEY_STAG

# Already-stored infra secrets (not in the per-service list, but the runner needs them)
GH_PAT                        # classic GitHub PAT, scope repo — for GOPRIVATE rewrites
DEPLOY_SSH_KEY                # ed25519 private key for adam@edustage + adam@edumm
EDUSTAGE_HOST                 # 10.10.0.101 (LAN)
EDUMM_HOST                    # 10.10.0.102 (LAN)

Payment's DOKU creds + RSA pairs are not Forgejo secrets — DOKU lives in DB payment_provider.credentials, RSA pairs live in /opt/forgejo-<env>/edusav_payment_service/shared/keys/.

Design decisions (locked 2026-05-19, don't relitigate)

• Shape C — plaintext templates with ${VAR} placeholders. No SOPS, no yq, no envsubst.
• Two files per service per env — full separation, even when 95% identical.
• Dead fields stay live with # DEAD per code audit 2026-05-19: <reason> — not stripped, not commented-out. Familiar shape for anyone who's read the upstream config.example.yaml; documented as cleanup-candidates.
• No upstream config.example.yaml cleanup PRs. Templates encode production truth (e.g. IAM permission_cache.*, payment security.rsa.*, multimedia rabbitmq.consumers.{dash_processing,video_dash_drm_processing} and cdn.*, messaging broadcast.batch_size). Example files can stay drifted.
• Per-env suffix on every secret name, even when the value is shared today.
• MinIO: two instances per env — MINIO_*_DEV and MINIO_*_STAG hold different values.

The per-service drift fixes and DEAD-block annotations come from a code audit (~/edusav/cicd_inventory.md) — read it before editing any template.

Bootstrap before the first deploy

Run from educicd as the git user — that account already has the Forgejo deploy key (~/.ssh/forgejo_deploy_ed25519) trusted by adam@edustage / adam@edumm, and LAN routing to both. Clone the repo into a scratch dir, then invoke the script with the LAN IPs of each target host:

sudo -u git -H bash -lc 'cd /tmp && rm -rf bootstrap-tmp && \
  git clone -b dev http://10.122.85.11:3000/edusav/edusav_cicd.git bootstrap-tmp && \
  cd bootstrap-tmp && \
  EDUSTAGE_HOST=10.10.0.101 EDUMM_HOST=10.10.0.102 \
  ./bootstrap/bootstrap-systemd.sh --env both'

The script walks deploy/service-catalog.yml and writes every forgejo-<env>-<svc>-<binary>.service unit on edustage + edumm, with the correct WorkingDirectory, ExecStart, and Wants=/After= ordering for the three gRPC startup dependencies (cicd.md §9). It is idempotent by sha256: each render is compared against the on-disk unit and skipped if identical, so re-running is cheap and only the hosts where something actually changed get a daemon-reload. The deploy workflow's secret drop-ins at .d/secrets.conf are not touched.

Useful flags:

• --env dev|stag|both (default both) — narrow to one env
• --service <name> (repeatable) — narrow to one or more catalog keys
• --host edustage|edumm|both (default both) — narrow to one host
• --dry-run — show what would be written without changing anything

Examples:

# What would change on dev across all services?
./bootstrap/bootstrap-systemd.sh --env dev --dry-run

# Rewrite only iam stag units on edustage
./bootstrap/bootstrap-systemd.sh --env stag --host edustage --service iam

# Run the full stag rollout (after the catalog is updated)
EDUSTAGE_HOST=10.10.0.101 EDUMM_HOST=10.10.0.102 \
  ./bootstrap/bootstrap-systemd.sh --env stag

The end-of-run summary line reports wrote=X skipped=Y missing=Z hosts_reloaded=N. missing counts units that didn't previously exist (a subset of wrote); skipped counts units whose sha matched. A clean re-run shows wrote=0.

Open operator-side actions

These don't block creating this repo but block the first successful deploy:

1. Populate the ~30 Forgejo org secrets listed above. Several known values (DB_PASSWORD_, RABBITMQ_PASSWORD_, TWILIO_*, EDUSTAGE_HOST, EDUMM_HOST, GH_PAT, DEPLOY_SSH_KEY) are recorded in cicd.md §10 and can be pasted. Missing values that need operator decisions:
   • INTERNAL_API_KEY_DEV / _STAG
   • JWT_SECRET_KEY_DEV / _STAG
   • GOOGLE_OAUTH_CLIENT_ID_DEV / _STAG + secret
   • ELASTICSEARCH_USER_DEV / _STAG + password
   • HASHID_SALT_DEV / _STAG
   • CDN_INTERNAL_API_KEY_DEV / _STAG
   • AWS_S3_ACCESS_KEY_* / AWS_S3_SECRET_KEY_* (or decide they reuse MINIO_*)
   • MINIO_ACCESS_KEY_DEV / _STAG + secret
2. Pick MinIO physical topology (cicd.md §14 item 1). The templates assume MinIO is reachable at 127.0.0.1:9000 on both edustage and edumm. Decide per env where MinIO actually runs and adjust host/port if it lives elsewhere.
3. Payment DOKU + RSA pairs (cicd.md §10 step 10a). DOKU credentials go into DB payment_provider.credentials JSONB per env. RSA keypairs go onto the target at /opt/forgejo-<env>/edusav_payment_service/shared/keys/.
4. Spring Boot active profile. The cdn workflow sets SPRING_PROFILES_ACTIVE=<env> in the unit drop-in. Make sure the JAR ships a matching application-<env>.properties loader path; the workflow already passes --spring.config.location=... so this should be fine.