> ## Documentation Index
> Fetch the complete documentation index at: https://velt.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Install Guide
> Copy-paste instructions for the AI coding agent that installs Velt full self-hosting in your Azure subscription.
Your AI coding agent runs the install. This page holds the instructions it follows, so you don't need to read them yourself.
1. Check the prerequisites and decide your inputs in [Get Started on Azure](/docs/self-hosting/full/azure/overview).
2. Copy the instructions below with the copy button.
3. Paste them into your agent along with the prompt from Get Started, then stay available for the steps that need a person.
The instructions mention companion runbooks (Validation, Upgrade, REST API, SAML and SCIM, Backup and Restore) that are not published on this site yet. Ask Velt for them.
## Agent instructions
````markdown Agent instructions (Azure) expandable theme={null}
# Velt Full Self-Hosting — Install Guide (Azure)
> **Doc set.** On AWS, use the [Install Guide (AWS)](https://docs.velt.dev/self-hosting/full/aws/install) instead.
> After install: the Validation runbook (acceptance evidence) and the Upgrade Guide; the REST API this deployment
> can serve is off by default (Phase 4.1 turns it on). Ask Velt for these runbooks.
> This file is the **executable install procedure** — paste it to an agent, or follow the commands yourself.
**Audience:** an AI coding agent with shell access, working for a Velt customer. A human
operator pastes this document (or its URL) to you as instructions. A human can also
follow it manually — every step is a plain command.
**Outcome:** the Velt portable backend running entirely inside the customer's own
Azure subscription:
1. **Substrate** — an AKS cluster, Azure Database for PostgreSQL Flexible Server (private,
customer-managed key), Key Vault, a Blob storage account, managed identities and an
Application Gateway ingress, created by Velt's signed Terraform module.
2. **Application** — Velt's signed container images (gateway, workers, migrate) and the
in-cluster Valkey cache, installed from Velt's signed Helm chart, pulled only from the
customer's own container registry.
3. **Console** — the Velt admin console served from the same ingress, signing in through the
customer's own identity provider (Entra ID or any OIDC provider).
4. **SDK** — the Velt client SDK bundle served from the same ingress, so the customer's pages
download nothing from Velt.
5. **A first workspace** — an API key, an organization and an admin user seeded into the
deployment's own database.
When finished, the customer's app, the console and the backend make **zero runtime requests
to any Velt-owned host**. Velt learns which version runs only from the receipt file the
operator chooses to send.
**Cloud scope:** Azure commercial (`AzureCloud`) and Azure Government (`AzureUSGovernment`).
Every command below is the same on both; the few values that differ are called out in boxes
marked **Azure Government**.
---
## To the agent executing this — read this first
- **Execute phases in order.** Each phase ends with a **Verify** block carrying a
*do not continue until* condition; do not proceed until it passes.
- **Maintain the state file** (`velt-selfhost-azure-state.json`, contract below) after
every step. If your session is interrupted, a fresh session resumes by reading this
guide plus the state file — never by memory.
- **Never run against any subscription other than `$SUBSCRIPTION_ID`.** Every resource
is prefixed by `DEPLOYMENT_NAME`; do not touch resources outside the deployment's
resource group.
- **All versions come from the release** (Phase 0). The images, the chart, the Terraform
module and the tools bundle are one release, pinned by version at the top of Phase 0.
Never substitute a different version for one component.
- **Verify before you use.** Every artifact — manifest, tools bundle, module, each image,
the chart — is verified with `cosign` against Velt's release identity before anything
is extracted, imported or installed. A verification failure is a **stop**, never a
retry: an artifact that does not verify is not a Velt release.
- **Human-in-the-loop moments are known in advance.** Show the operator the
"When you'll be needed" table before starting; when a moment arrives, hand them the
*exact* command or portal URL — never a vague ask.
- **Retryable errors (expected — retry with backoff, do NOT report as failures):**
- (a) `403 … Caller is not authorized … Assignment: (not found)` on the first Key Vault
key/secret create — the deployer's Key Vault Administrator grant is still propagating
(1–2 min). Wait 90 s and run `terraform apply` again (it is idempotent).
- (b) `Provider produced inconsistent result after apply: Root object was present, but
now absent` (VNet / Key Vault / identity), or `ParentResourceNotFound: SOA` on the
private DNS zone — Azure eventual consistency in a fresh region. Re-run the apply. If
a resource is in Terraform state but 404s in Azure, `terraform state rm `,
then apply again.
- (c) an `az` call failing with an expired token mid-run — `az login` again and re-run
the command (Terraform state and `az` are idempotent).
- (d) `az acr import` or an image pull failing with `connection refused` — transient;
retry the import.
- (e) the first `terraform apply` ending with a **404 on `azurerm_monitor_diagnostic_setting.appgw`**
("the Application Gateway was not found") after every other resource was created — the ingress
add-on creates the gateway asynchronously, and it can crash-loop on
`ApplicationGatewayInsufficientPermissionOnSubnet` for a minute or two while its subnet grant
propagates. Restart it and apply again (Phase 1.3 names the exact commands).
- **NON-retryable — needs an operator decision (Phase 0 and Troubleshooting):**
`LocationIsOfferRestricted` (PostgreSQL Flexible Server not permitted in this region for
this subscription) and `The VM size … is not allowed in your subscription in location`
— both are subscription-quota facts, resolved by changing region or SKU, never by retry.
The preflight in Phase 0 exists so you learn both BEFORE the apply.
- **Long waits are normal.** A full run is ~60–90 min of wall clock: the AKS + Postgres
apply is ~12–15 min; Helm pod startup ~2–3 min; the App Gateway's first program ~3–6 min.
Tell the operator when a long wait starts so the run does not look hung.
- **Shell quirks (bash/zsh):** in zsh, `$VAR:x` is a parameter modifier — write `${VAR}`
whenever a colon or word character follows. A pipeline's exit code reflects only its
last command — check the command you care about. `timeout` may be absent on macOS.
- If a command fails and is not retryable, check **Troubleshooting** at the end before
escalating to the operator.
- **Never print a secret.** The Postgres passwords, the DSNs, the ES256 private key and the
workspace auth token go into the operator's secret store; the state file and your report
carry **paths and names**, never values.
### State file contract — `velt-selfhost-azure-state.json`
Keep it in the working directory (or the operator's config repo). It is the single source
of truth for resume and for cross-phase handoffs. Shape:
```jsonc
{
"guideVersion": 4,
"release": { /* the full manifest fetched in Phase 0.1, including its tierB block.
manifestVerified: true when the manifest's cosign signature verified AND
the tierB block was present; false when the version had to come from
the chart's tag list instead (a weaker provenance claim — the final
report must say which one you got). */ },
"inputs": {
"subscriptionId": "", "tenantId": "", "cloud": "AzureCloud",
"location": "", "resourceGroup": "velt-portable", "deploymentName": "velt",
"publicHost": "",
"consoleAuthIssuer": "", "consoleAuthClientId": "",
"acrName": "",
"aksAdminGroupObjectId": "", "operatorEgressCidr": "",
"apiKey": "", "orgId": "", "adminUserIdHash": "", "consoleAdminEmail": "",
"allowedDomains": [], "privateCommentsEnabled": false,
"webhookV2Delivery": "", "redis": "in-chart|external", "alertEmailAddresses": []
},
"versions": {
"portableVersion": "", "consoleVersion": "", "sdkVersion": "", "schemaHead": ""
},
"phases": {
"0-preflight": "pending|done",
"1-substrate": "pending|in-progress|done",
"2-images": "…", "3-keys-secrets": "…", "4-helm": "…",
"5-ingress-dns-tls": "…", "6-smoke": "…", "7-acceptance": "…"
},
"artifacts": {
"toolsDir": "./velt-portable-tools",
"preflightReport": "", // written by portable-install-preflight.sh; "0-preflight" is done ONLY on exit 0
"moduleDir": "./terraform/azure",
"valuesFile": "", // values-.yaml
"receiptPath": "", // receipt.json from portable-receipt.sh (final report)
"wsEndpoint": "", "apiEndpoint": "", "consoleUrl": "",
"keyVaultId": "", "workloadIdentityClientId": "", "postgresSubnetCidr": "",
"appGatewayIp": "", "alertActionGroupId": "",
"jwtPrivateKeyLocation": "", // SECRET — a path or a vault reference, never inline
"jwtPublicKeySecretName": "velt-jwt-es256",
"jwtPrivateKeySecretName": "velt-jwt-es256-private",
"dsnSecretName": "velt-postgres-dsn", "dsnAdminSecretName": "velt-postgres-dsn-admin",
"appDbPasswordSecretName": "velt-app-db",
"tlsPath": "customer-secret|key-vault|cert-manager", "tlsSecretName": "velt-portable-tls",
"imageDigests": {} // from cosign / az acr: what the infosec sign-off covered
},
"humanSteps": {
"azLogin": "pending|done", "roleGrantConfirmed": "pending|done",
"regionConfirmed": "pending|done", "entraAppRegistration": "pending|done|n/a",
"infosecScanSignOff": "pending|done|waived", "dnsRecord": "pending|done",
"certificate": "pending|done", "signInTest": "pending|done",
"acceptanceApp": "pending|done|stand-in"
},
"log": [ { "ts": "…", "phase": "…", "note": "…" } ]
}
```
---
## Inputs and human-in-the-loop moments
Collect all of these **once, up front** — a run that stops halfway to ask for a
subscription id is a run the operator has to babysit.
### Inputs to collect from the operator (all up front, once)
| Input (placeholder used below) | Meaning | Example |
|---|---|---|
| `` → `SUBSCRIPTION_ID` | Azure subscription to deploy into | `0000…` |
| `` → `TENANT_ID` | Entra tenant id (the console's OIDC issuer is built from it) | `0000…` |
| `` → `CLOUD` | `AzureCloud` (commercial) or `AzureUSGovernment` | `AzureCloud` |
| `` → `LOCATION` | Azure region. MUST permit PostgreSQL Flexible Server for this subscription (Phase 0.5 checks). Hosts BOTH AKS and Postgres — single VNet, single region. | `canadacentral` |
| `` → `RESOURCE_GROUP` | Resource group name | `velt-portable` |
| `` → `DEPLOYMENT_NAME` | 3–16 character slug that prefixes every Azure resource name. **One-shot for 90 days:** a torn-down deployment's Key Vault stays soft-deleted under this name (the preflight's P10 refuses it) — suffix it (`acmevelt01`) rather than reuse it | `acmevelt01` |
| `` → `PUBLIC_HOST` | Public hostname the ingress serves. Builds the WS and API endpoints, the ingress host and the certificate name — keep them equal. Needs a DNS record (human moment #6), or use the free Azure DNS label in Phase 5.2. | `velt.acme.com` |
| `` → `CONSOLE_AUTH_ISSUER` | Console OIDC issuer. Entra commercial: `https://login.microsoftonline.com//v2.0`; Entra Government: `https://login.microsoftonline.us//v2.0`; or any OIDC provider's issuer | … |
| `` → `CONSOLE_AUTH_CLIENT_ID` | The console app registration's client id (human moment #4 if you cannot create app registrations) | … |
| `` → `ACR_NAME` | Azure Container Registry that will hold the verified Velt images. Globally unique, alphanumeric, 5–50 chars. | `acmeveltacr` |
| `` | Object id of the Entra security group whose members administer the cluster (REQUIRED — the cluster has no local accounts; a fresh tenant has none — Phase 0.3 creates one) | `0000…` |
| `` | The public IP range the operator runs `kubectl` from, as a CIDR (REQUIRED unless the operator chooses a private API server). Find it with `curl -4 https://api.ipify.org`; a home or VPN address changes, so give a range (`/24`) or expect to update `api_server_authorized_ip_ranges` later (the Operator notes at the end) | `203.0.113.7/32` |
| ``, ``, `` | The first workspace's API key, first organization id and the first admin user's id hash — strings the operator chooses; the SDK and REST calls later use these exact values | `acme-prod`, `acme`, `u-1` |
| `` | Email of the first console administrator — must equal the `email` (or Entra UPN) claim their IdP sign-in carries | `ops@acme.com` |
| `` | The domains the customer's pages are served from, comma separated — identify refuses a workspace that has none | `acme.com,app.acme.com` |
| `` | The browser origin (scheme + host [+ port]) of those pages, for CORS — list every origin, comma separated | `https://app.acme.com` |
| `` | The address Let's Encrypt (or your ACME CA) notifies about certificate expiry — only on the cert-manager TLS path (Phase 5.3); the console admin email is fine | `ops@acme.com` |
| `` | The first API key's auth token for the REST API (`x-velt-auth-token`): a value **you generate** (at least 24 characters, letters and digits) and store in the operator's secret store before Phase 4.5 seeds it | `openssl rand -base64 24 \| tr -d '/+='` |
| `` | **Optional — comment attachments.** The browser origins whose users attach files to comments; the storage account answers CORS for exactly these. Usually the same as ``. `https://` only (`http://localhost` / `http://127.0.0.1` for development), no path, no `*`. Leave empty to install without attachments (the SDK then hides the attachment button). | `https://app.acme.com` |
| Private comments? | Whether the customer uses Velt private comments (seeds `isPrivateCommentsEnabled`) | yes / no |
| Webhook V2 delivery | `""` (no advanced webhooks yet), `svix` (their own Svix account — needs a token) or `direct` (deliver from inside the cluster, no third party) | `direct` |
| Console SAML sign-in + SCIM | Whether console members sign in through the customer's SAML identity provider (Okta, Entra ID, Keycloak, Ping, ADFS), with or without the Entra OIDC sign-in above, and whether that IdP provisions the console roster over SCIM. Needs a SAML app in the IdP; can be turned on after the install ("Turning a feature on later") | no |
| Redis | in-chart Valkey (default; nothing to provision) or an external managed Redis URL | in-chart |
| Alert recipients | Who is told when a metric alert fires (persistent mode): email addresses for the action group the module creates, or the resource id of your own action group (your on-call) | `ops@acme.com` |
| Vulnerability scan policy | Does the organization require scanning third-party images before deployment? (human moment #5) | yes / no |
| TLS path | customer-supplied certificate (default), Key Vault certificate, or cert-manager + ACME (needs public egress) | customer-supplied |
### Features you can turn on — decide now or later
Every optional feature is off until it is turned on, and every one can be turned on after the
install with a `helm upgrade` plus whatever outside piece it needs. Decide on purpose rather
than inherit the defaults; the how-to for each is under "After the install → Turning a feature
on later".
| Feature | Default | Asked above as | Outside the chart it needs |
|---|---|---|---|
| REST API (server-to-server) | off | `rest-api` in `enabledModules` (Phase 4.1) | nothing — the API key's auth token from Phase 4.5 |
| Comment attachments | off | `` | a CORS rule on the storage account (Terraform `attachment_cors_origins`) |
| Advanced webhooks | off | Webhook V2 delivery | `direct`: nothing; `svix`: a Secret with the customer's Svix token |
| Console SAML sign-in + SCIM | off | Console SAML sign-in + SCIM | a SAML app in the customer's identity provider |
| Self-hosted SDK bundle | off | `sdk:` block (Phase 4.1) | the `velt-sdk-static` image imported into the registry; the app's `cdnHost` pin |
| Webhook payload encryption | off | — (decide later) | a Secret holding the encryption key |
| Deployment-wide token minting | off, and off is the safe choice | — (decide later) | a Secret holding the master token; only for internal services that mint tokens across several API keys |
| Private comments | per API key | Private comments? | nothing — the console (or the seed's `--project-config`) |
| FIPS mode | on in Azure Government, off in commercial | `fips_enabled` (Phase 1.2) | **an install-time choice**: the node pools' FIPS flag replaces the nodes when changed; the application's `fips.enabled` follows it |
| Node autoscaling | on: the pool starts at `aks_node_count` and the cluster autoscaler may grow it to `aks_node_max_count` (twice the count unless set) | — (Phase 1.2) | nothing to install (it is AKS's own); the VM family's vCPU quota for the ceiling (Phase 0.5) |
| Metric alerts | on in `persistent`, off in `sandbox` | Alert recipients | an action group with receivers — the module's (`alert_email_addresses`) or yours (`alert_action_group_id`) |
| PostgreSQL high availability | off | — (decide later) | `postgres_high_availability_mode` (Phase 1.2): `ZoneRedundant` needs a region with availability zones, `SameZone` does not; about doubles the database's cost |
### When you'll be needed (show the operator this table now)
| # | Moment | What the operator does | Time |
|---|---|---|---|
| 1 | Phase 0.3 | `az login` — you hand them the device-code URL and code if the session is headless | 2 min |
| 2 | Phase 0.3 | Confirm the deploying identity holds **Owner** (or **Contributor + User Access Administrator**) on the subscription. Contributor alone cannot create the Key Vault grant the module needs. | 2 min |
| 3 | Phase 0.5 | Confirm the chosen `LOCATION` (you propose one from the region checks) | 1 min |
| 4 | Phase 0.3 | Create the Entra console app registration and supply `CONSOLE_AUTH_CLIENT_ID`, IF your identity lacks the **Application Developer** role (otherwise you create it) | 5 min |
| 5 | Phase 2.2 | **Vulnerability-scan sign-off**: if organization policy requires scanning third-party images, review the scan of their registry copy and approve (or waive scanning explicitly) | 5–15 min |
| 6 | Phase 5.2 | Add the DNS A record pointing `PUBLIC_HOST` at the Application Gateway IP (not needed with the Azure DNS label) | 2 min + propagation |
| 7 | Phase 5.3 | Supply the TLS certificate for `PUBLIC_HOST` (customer-supplied path), or approve the Key Vault / cert-manager path | 5–30 min |
| 8 | Phase 7 | Sign in to the console in a browser; run the SDK loop in their app | 10 min |
| 9 | Phase 0.3 | Approve creating the Entra security group `velt-aks-admins` (or supply an existing group's object id), IF your identity may not create groups | 2 min |
| 10 | Phase 0.5 | Raise the vCPU quota when the subscription's limits are below the footprint — a family quota at **0** usually needs a Portal request or a Microsoft support ticket (hours to days): do this on day 1, or choose the small footprint | 5 min + waiting |
Expected total wall clock: **60–90 min** plus the human moments. Expected running cost of the
deployed stack is in **After the install** at the end — tell the operator up front that a
persistent deployment is a small always-on cluster plus a managed Postgres server,
and how to idle it.
### Tooling preflight (hard requirements — the preflight script checks the versioned ones)
| Tool | Why | Check | Install (macOS Homebrew; Linux: the vendor's package) |
|---|---|---|---|
| Azure CLI ≥ 2.60 | all provisioning | `az version` | `brew install azure-cli` |
| Terraform ≥ 1.6 | the substrate | `terraform version` | `brew tap hashicorp/tap && brew install hashicorp/tap/terraform` (the plain `terraform` formula no longer exists) |
| Helm 3.17+ or 4.x | the application (both are supported; the chart is installed by its digest, which Helm supports from 3.17) | `helm version` | `brew install helm` |
| kubectl | cluster operations | `kubectl version --client` | `brew install kubectl` |
| **Azure** kubelogin | Entra sign-in to the cluster (the cluster has no local accounts) | `kubelogin convert-kubeconfig --help` works | `brew install Azure/kubelogin/kubelogin` — NOT `brew install kubelogin`, which installs a different tool of the same name (int128's OIDC plugin) that Phase 1.4 cannot use |
| cosign | signature verification of every artifact (MANDATORY) | `cosign version` | `brew install cosign` |
| oras | pulling the manifest, tools bundle and module from the release registry | `oras version` | `brew install oras` |
| Node.js ≥ 18 | the update check (`portable-check-updates.js`) in the final report | `node --version` | `brew install node@22` |
| Docker daemon | running the provision CLI from the migrate image (keygen, token) | `docker info` | Docker Desktop / OrbStack / Colima |
| psql (optional) | lets the support bundle read the schema head when the database is reachable from the workstation — on this Azure path it is NOT (the server is private), so the receipt takes the head you read in Phase 4.4 with `--schema-head` instead | `psql --version` | skip it on Azure; `brew install libpq` elsewhere |
| curl, python3, openssl, jq, dig | verification, JSON reading, password generation, DNS checks (`dig` is on every macOS; Linux: `dnsutils` / `bind-utils`; without it use the `python3` one-liner shown in Phase 5). `python3` must RUN, not just be on the PATH: every digest step and the preflight's P9 use it, and P1 refuses one that does not run (a Mac without the Command Line Tools has a stub that only offers to install them) | `python3 -c 'import json'` | `xcode-select --install` (macOS), or `brew install jq` for jq |
Auth is probed in Phase 0.3 with real calls — a listed account is not proof of a working
session.
---
## Phase 0 — Release resolution, identity, region, preflight
### 0.0 The versions — set once, used everywhere
Every command in this guide reads these variables. Set them in the shell you run the
install from, and re-export them at the start of every fresh session (record them in
`versions` in the state file).
```bash
REG=us-docker.pkg.dev/velt-sdk/velt-releases
IDENTITY='^https://github\.com/snippyly/shared-firebase-function/\.github/workflows/container-release\.yml@refs/heads/main$'
ISSUER='https://token.actions.githubusercontent.com'
PORTABLE_V=0.1.13 # the portable release: images, chart, Terraform module, tools bundle
CONSOLE_V=0.1.19 # the console bundle — its own cadence, its own tag
SDK_V=6.0.24 # the SDK bundle — its own cadence; ALSO part of the served path
```
`PORTABLE_V` is the one you confirm from the manifest in 0.1 (the worked value above is the
release this guide was written for). `CONSOLE_V` and `SDK_V` come from the manifest too when
it pins them. When it does not (every manifest published before the release pipeline began pinning the SDK
omits `sdkStatic`), take the newest version a portable release published; only Velt's release job signs these images,
so the verify in Phase 2 still holds them to the same signer:
```bash
oras repo tags "$REG/velt-console-static" | grep -E '^[0-9]+\.[0-9]+\.[0-9]+$' | sort -V | tail -1 # CONSOLE_V
oras repo tags "$REG/velt-sdk-static" | grep -E '^[0-9]+\.[0-9]+\.[0-9]+$' | sort -V | tail -1 # SDK_V
```
Releases before `0.1.13` have no tools bundle. If the manifest pins an older
`portableVersion`, Velt support supplies the five scripts; everything else in this guide is
unchanged.
**The version numbers written into this guide are worked examples** (`0.1.13`, console
`0.1.19`, SDK `6.0.24`), not pins: the manifest you verify in 0.1 is the pin, and it is
usually newer than the examples. The registry may also hold a release newer than the one
the manifest pins (still being validated by Velt) — **a newer registry tag is not a release
you install**; the manifest is. Some steps below say "0.1.14 or newer" or "0.1.15 or newer":
they need a tools bundle, module or chart of at least that version, which the manifest's pin
tells you.
### 0.1 Resolve and VERIFY the release manifest
The umbrella manifest is the trust root. Its `tierB` block pins the portable release — every
artifact's reference and digest.
```bash
# MANDATORY — resolve the tag ONCE to its digest, verify that digest, then pull that digest.
# Only deploy a release that verifies.
# (The manifest is published by a different workflow than the images, so its identity differs.)
MANIFEST_DIGEST=$(oras manifest fetch --descriptor "$REG/velt-selfhost-manifest:latest" \
| python3 -c "import json,sys;print(json.load(sys.stdin)['digest'])")
cosign verify \
--certificate-identity-regexp '^https://github\.com/snippyly/shared-firebase-function/\.github/workflows/selfhost-umbrella-publish\.yml@refs/heads/main$' \
--certificate-oidc-issuer "$ISSUER" "$REG/velt-selfhost-manifest@$MANIFEST_DIGEST" \
&& oras pull -o . "$REG/velt-selfhost-manifest@$MANIFEST_DIGEST" # writes manifest.json
# Read the portable release pins.
python3 -c "import json;m=json.load(open('manifest.json'));print(json.dumps(m.get('tierB','<>'),indent=2))"
```
**Verify and fetch by one digest.** The registry is asked what a tag names every time a command
uses the tag, so a verification by tag followed by a pull by tag gets two answers, and only the
first was checked. Every step below that pulls or imports an artifact resolves the tag once
(`oras manifest fetch --descriptor`), verifies that digest, and pulls or imports that same
digest — the Helm chart included (Phase 4.3). Do not put the tag back in place of the digest.
When the block is present it pins `portableVersion`, `helmChart`, `module` (the Terraform
substrate blueprint), `tools` (the operator tools bundle — present from `0.1.13`) and
`images` (gateway / workers / migrate plus the optional `consoleStatic` and `sdkStatic`),
each by reference and cosign-verified digest. Set `PORTABLE_V` to `tierB.portableVersion`
and, when present, `CONSOLE_V` / `SDK_V` from the `consoleStatic` / `sdkStatic` tags. Use the
pinned references verbatim.
**A `tierB` block with no `tools` entry predates `0.1.13`** — the manifest is republished only
when Velt publishes a new umbrella, so it can lag the newest portable release. Treat it exactly
like an absent block: take the version from the chart's tag list below, record
`release.manifestVerified: false`, and tell the operator the manifest is behind (Velt republishes
it; the install does not wait for that). Do not install a version whose tools bundle does not
exist, and do not mix the manifest's older image pins with a newer chart.
**`tierB` is an optional block and may be absent** (a release without the portable deployment).
Do not assume either way — read it. When it is absent, or present without `tools`, the portable
version comes from the chart's own tag list in the release registry:
```bash
oras repo tags "$REG/velt-portable" | grep -E '^[0-9]+\.[0-9]+\.[0-9]+$' | sort -V | tail -1 # the newest portable release (signature tags sort last — filter them out)
```
Set `PORTABLE_V` to that value and record `release.manifestVerified: false` in the state
file — a chart tag is a weaker provenance claim than a verified manifest block, and the final
report must say which one the deployment got. Every artifact is still cosign-verified
individually below, so what is lost is the manifest's cross-component pin, not signing.
Store the whole manifest in the state file.
### 0.2 Pull and VERIFY the tools bundle
The five operator scripts this guide runs — the install preflight, the version receipt, the
support bundle, the upgrade script and the update check — ship as a signed artifact of the
same release. Pull them now, before anything else runs, so every later step uses the
release's own copies:
```bash
TOOLS_DIGEST=$(oras manifest fetch --descriptor "$REG/velt-portable-tools:$PORTABLE_V" \
| python3 -c "import json,sys;print(json.load(sys.stdin)['digest'])")
cosign verify --certificate-identity-regexp "$IDENTITY" --certificate-oidc-issuer "$ISSUER" "$REG/velt-portable-tools@$TOOLS_DIGEST" \
&& oras pull "$REG/velt-portable-tools@$TOOLS_DIGEST" && tar xzf "velt-portable-tools-$PORTABLE_V.tar.gz"
ls velt-portable-tools/
# portable-install-preflight.sh portable-receipt.sh portable-support-bundle.sh
# portable-upgrade.sh portable-check-updates.js README.md VERSION
cat velt-portable-tools/VERSION # must equal $PORTABLE_V
```
The archive unpacks to `./velt-portable-tools/`; the commands below call the scripts from
there. If `cosign verify` fails, STOP: do not extract the archive.
### 0.3 Sign in to Azure; identity and permissions
```bash
# Azure Government ONLY — before az login:
# az cloud set --name AzureUSGovernment
# Do not know the tenant id yet? Sign in once without it and read it: az login ; az account list -o table
az login --tenant "$TENANT_ID" # headless: add --use-device-code (hand the operator the URL + code)
az account set --subscription "$SUBSCRIPTION_ID"
az account show --query '{sub:id, name:name, tenant:tenantId, cloud:environmentName, user:user.name}' -o table
az ad signed-in-user show --query userPrincipalName -o tsv # the REAL identity: a guest prints …#EXT#@…, which `az account show` hides
export ARM_SUBSCRIPTION_ID="$SUBSCRIPTION_ID" # Terraform reads this
```
If device-code sign-in is refused with `AADSTS530035` (the tenant's security defaults or a
Conditional Access policy block the device-code flow), use the interactive browser sign-in
(`az login --tenant "$TENANT_ID"` without `--use-device-code`) from a machine with a browser,
or ask the tenant admin to allow device code for this user. **Check `user` in the output:** a
cached session for another account is common on a shared machine, and a guest account
(`…#EXT#@…`) holds roles under its guest identity, not its home one.
**The deploying identity needs more than Contributor** — **Owner**, or **Contributor +
User Access Administrator**, on the subscription. Two reasons, both hit on the very first
apply otherwise:
- The Key Vault is RBAC-authorized, and the module writes the customer-managed keys and the
Postgres password *into* it — a data-plane operation that subscription Owner alone does
not cover. The module therefore grants the deployer **Key Vault Administrator** on the
vault itself. **Creating that grant needs `Microsoft.Authorization/roleAssignments/write`,
which Contributor lacks.**
- **Do not pre-create that Key Vault Administrator grant by hand.** On a cold apply the
module creates it; a grant that already exists makes the apply fail with
`RoleAssignmentExists`.
```bash
# What roles does this identity hold on the subscription — directly, through a group, or
# inherited from a management group? (preflight P3 asks the same way)
az role assignment list --assignee "$(az ad signed-in-user show --query id -o tsv)" \
--scope "/subscriptions/$SUBSCRIPTION_ID" --include-groups --include-inherited \
--query '[].roleDefinitionName' -o tsv
```
Without `--include-groups --include-inherited` the list shows only direct assignments at that
exact scope, and a correctly privileged identity (Owner through a group, or on the management
group) is wrongly told to stop.
Two Entra questions decide human moments #4 and #9 below (may this identity create an app
registration? a security group?). Ask them with two Graph calls instead of guessing:
```bash
# Directory roles this identity holds (Global Administrator, Application Developer, Groups Administrator …):
az rest --method GET --url "https://graph.microsoft.com/v1.0/me/transitiveMemberOf/microsoft.graph.directoryRole" \
--query "value[].displayName" -o tsv
# Tenant-wide defaults when no role applies: may ordinary users create app registrations / security groups?
az rest --method GET --url "https://graph.microsoft.com/v1.0/policies/authorizationPolicy" \
--query "{appRegistrations:defaultUserRolePermissions.allowedToCreateApps, securityGroups:defaultUserRolePermissions.allowedToCreateSecurityGroups}" -o json
```
Either a role or a `true` default means you create the object yourself; otherwise it is the
operator's moment.
If the output lacks `Owner` and lacks the pair `Contributor` + `User Access Administrator`,
this is human moment #2: hand the operator the role to grant and stop until it is granted.
**The console app registration.** The console signs in through an OIDC app registration in
the operator's tenant. If your identity holds the Entra **Application Developer** role (or
higher), create it; otherwise this is human moment #4 — give the operator these exact
settings: a single-page application, redirect URI `https:///signup`, the
`openid profile email` scopes, and ask for the application (client) id.
```bash
# Only if you may create app registrations in this tenant:
CONSOLE_AUTH_CLIENT_ID=$(az ad app create --display-name "velt-console-$DEPLOYMENT_NAME" \
--sign-in-audience AzureADMyOrg --query appId -o tsv)
# The SPA redirect URI goes in with a Graph PATCH: `az ad app update --set spa.redirectUris=…` fails
# on a fresh app ("Couldn't find 'spa' in ''") on Azure CLI 2.90, and leaves the app without its URI.
APP_OBJECT_ID=$(az ad app show --id "$CONSOLE_AUTH_CLIENT_ID" --query id -o tsv)
az rest --method PATCH --url "https://graph.microsoft.com/v1.0/applications/$APP_OBJECT_ID" \
--headers 'Content-Type=application/json' \
--body '{"spa":{"redirectUris":["https://'"$PUBLIC_HOST"'/signup"]}}'
az ad app show --id "$CONSOLE_AUTH_CLIENT_ID" --query spa.redirectUris -o tsv # must print the URI — do not continue until it does
echo "$CONSOLE_AUTH_CLIENT_ID"
```
The console issuer for Entra is `https://login.microsoftonline.com//v2.0`
(Government: `https://login.microsoftonline.us//v2.0`). Any other OIDC provider
works the same way: its issuer URL and a client id for a single-page app with that redirect
URI. The issuer must be reachable **from the users' browsers**, not from the cluster.
**The AKS admin group.** The cluster is created with Entra ID as its identity provider and no
local accounts, so `aks_admin_group_object_ids` (Phase 1.2) must name an Entra security group
whose members administer the cluster. A fresh tenant has no groups. Find one, or create it:
```bash
az ad group list --query "[].{name:displayName,id:id}" -o table # any existing group to use?
# None suitable: create one and add the operator (needs the Groups Administrator or Global
# Administrator role, or the tenant setting that lets users create security groups):
AKS_ADMIN_GROUP_OBJECT_ID=$(az ad group create --display-name "velt-aks-admins" \
--mail-nickname "velt-aks-admins" --description "Administers the Velt AKS cluster" \
--query id -o tsv)
az ad group member add --group "$AKS_ADMIN_GROUP_OBJECT_ID" \
--member-id "$(az ad signed-in-user show --query id -o tsv)"
echo "$AKS_ADMIN_GROUP_OBJECT_ID"
```
If your identity may not create groups, this is human moment #9: ask the operator for an
existing group's object id, or for the group to be created with the operator as a member.
Record the id as ``.
### 0.4 Register the resource providers (once per subscription)
```bash
for NS in Microsoft.ContainerService Microsoft.DBforPostgreSQL Microsoft.KeyVault \
Microsoft.Storage Microsoft.Network Microsoft.ManagedIdentity \
Microsoft.Compute Microsoft.ContainerRegistry; do
az provider register --namespace "$NS"
done
# Registration is asynchronous — all eight must read Registered before the apply:
for NS in Microsoft.ContainerService Microsoft.DBforPostgreSQL Microsoft.KeyVault \
Microsoft.Storage Microsoft.Network Microsoft.ManagedIdentity \
Microsoft.Compute Microsoft.ContainerRegistry; do
printf '%-32s %s\n' "$NS" "$(az provider show --namespace "$NS" --query registrationState -o tsv)"
done
```
### 0.5 Region and quota — do this BEFORE the apply, not after it fails
PostgreSQL Flexible Server is gated per subscription **and** per region. An apply can bring
up the VNet, Key Vault, AKS and storage and then fail only on Postgres with
`LocationIsOfferRestricted` — roughly twelve minutes in. Postgres must share AKS's region
(private DNS zone + delegated subnet live in one VNet), so a restricted region means moving
the whole stack. Check first.
**Quota first, on day 1.** The module's persistent-mode default footprint is 3 nodes ×
`Standard_D4ds_v5` (4 vCPU each): it needs **Total Regional vCPUs ≥ 16** (12 + one node of
surge for a rolling upgrade) and **Standard DDSv5 Family vCPUs ≥ 16** in the region. A fresh
pay-as-you-go subscription commonly has a regional limit of **10** and several families at
**0**; a family at 0 usually cannot be raised self-service and needs a Portal request or a
Microsoft support ticket, which takes hours to days. Read the two numbers now (step 2), and
either file the request today (step 2b) or take the small footprint, which fits a 10-vCPU
subscription with no request.
**Scan several regions at once.** Each `az vm list-skus` call downloads the region's whole
SKU catalogue and takes one to two minutes with no output — one call per region, cached in a
file, answers every question below. On a fresh subscription it is common that Postgres is
restricted in one region, the default size is `NotAvailableForSubscription` in another and
the small size is not offered in a third, so run the loop over your candidates first and
pick the region where every line is green:
```bash
for LOC in westus3 centralus eastus2 westus2; do
SKUS="skus-$LOC.json"; [[ -s "$SKUS" ]] || az vm list-skus -l "$LOC" --resource-type virtualMachines -o json > "$SKUS" # 1–2 min each, once
PG=$(az postgres flexible-server list-skus -l "$LOC" -o json 2>/dev/null | python3 -c 'import json,sys; d=json.load(sys.stdin)[0]; print("ALLOWED" if [f["status"] for f in d["supportedFeatures"] if f["name"]=="OfferRestricted"]==["Disabled"] else "RESTRICTED")' 2>/dev/null || echo "RESTRICTED")
for SIZE in Standard_D4ds_v5 Standard_D2ds_v4; do
FAM=$(python3 -c "import json,sys; s=[x for x in json.load(open('$SKUS')) if x['name']=='$SIZE']; print(s[0]['family'] if s else '')")
RESTR=$(python3 -c "import json,sys; s=[x for x in json.load(open('$SKUS')) if x['name']=='$SIZE']; print(','.join(r['reasonCode'] for r in s[0].get('restrictions',[])) if s else 'NOT-OFFERED')")
QUOTA=$([[ -n "$FAM" ]] && az vm list-usage -l "$LOC" --query "[?name.value=='$FAM'].[currentValue,limit] | [0]" -o tsv 2>/dev/null | tr '\t' '/' || echo "n/a")
echo "$LOC postgres=$PG $SIZE family=${FAM:-none} restrictions=${RESTR:-none} familyQuota(used/limit)=${QUOTA:-n/a}"
done
done
```
A size whose `family` prints `none` is not offered there — the single-region commands below
then print nothing for the family line, which is the same answer, not a pass.
**1. Is Postgres offered to this subscription here?** One call, nothing to clean up:
```bash
az postgres flexible-server list-skus -l "$LOCATION" -o json | python3 -c '
import json,sys
d=json.load(sys.stdin)[0]
r=[f["status"] for f in d["supportedFeatures"] if f["name"]=="OfferRestricted"]
print("POSTGRES: " + ("ALLOWED" if r==["Disabled"] else "RESTRICTED (pick another region)"))
skus={s["name"] for e in d["supportedServerEditions"] for s in e["supportedServerSkus"]}
print("GP_Standard_D2ds_v5 (the module default): " + ("offered" if "Standard_D2ds_v5" in skus else "NOT offered — set postgres_sku"))'
```
**2. Compute quota for the AKS nodes.** The module's persistent-mode default is
**3 nodes × `Standard_D4ds_v5` (4 vCPU each)** — 12 vCPUs, plus one node of headroom for a
rolling upgrade surge (16). The cluster autoscaler may grow the pool to `aks_node_max_count`
(6 by default), so the ceiling with its surge is 28 vCPUs: the preflight refuses below 16 and
warns below 28, naming the value to request (`--max-nodes` if you set `aks_node_max_count`).
Read two numbers:
```bash
AKS_NODE_VM_SIZE=Standard_D4ds_v5 # change here if you override aks_node_vm_size below
AKS_VCPUS_PER_NODE=4
# The SKU's family (the quota is per family), then exactly the two lines that matter:
FAMILY=$(az vm list-skus -l "$LOCATION" --size "$AKS_NODE_VM_SIZE" --query "[?name=='$AKS_NODE_VM_SIZE'].family | [0]" -o tsv)
az vm list-usage -l "$LOCATION" \
--query "[?name.value=='cores' || name.value=='$FAMILY'].{quota:name.localizedValue, used:currentValue, limit:limit}" -o table
# Need: cores limit - used >= 3 × $AKS_VCPUS_PER_NODE + $AKS_VCPUS_PER_NODE (surge) = 16 for the default,
# and the family limit >= the same. A family limit of 0 = that size is unavailable to you here.
# Room to scale: 6 × $AKS_VCPUS_PER_NODE + $AKS_VCPUS_PER_NODE = 28 lets the autoscaler reach its default ceiling.
```
Then confirm the size is offered AND unrestricted — the two look alike when read carelessly
(an empty restriction list also comes back for a size the region does not offer at all):
```bash
az vm list-skus -l "$LOCATION" --size "$AKS_NODE_VM_SIZE" \
--query "[?name=='$AKS_NODE_VM_SIZE'].{size:name, restrictions:restrictions[].reasonCode}" -o json
# [] = NOT offered in this region — pick another size
# [{"size": …, "restrictions": []}] = offered and usable
# ["NotAvailableForSubscription"] = offered to others, restricted for you — pick another size
```
**2b. When the numbers do not fit, two ways out.**
*The small footprint — no request, fits a 10-vCPU subscription.* Any 2-vCPU **x86**
general-purpose size whose family has quota (the images are linux/amd64; ARM `_p*` sizes will
not run them): `Standard_D2ds_v4`, `Standard_D2ads_v7`, `Standard_D2s_v6` and their siblings.
3 nodes × 2 vCPU = 6, plus 2 for surge = 8. Each node has 8 GiB; the application's requests at
the floor are about 1 CPU and 3 GiB, so this is a comfortable evaluation and small-team
posture. Set in Phase 1.2:
```hcl
aks_node_vm_size = "Standard_D2ds_v4" # 2 vCPU x86; check its family quota the same way (Standard DDSv4 Family)
```
and pass `--vm-size Standard_D2ds_v4 --vcpus-per-node 2` to the preflight in 0.6. Move to the
default size later with a plan that changes only the node pool.
*The quota request — for the default size.* Self-service first; a family at 0 is usually
refused self-service and needs the Portal or a support ticket:
```bash
az extension add --name quota --upgrade -o none
SCOPE="/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Compute/locations/$LOCATION"
az quota create --scope "$SCOPE" --resource-name cores --resource-type dedicated --limit-object value=16
az quota create --scope "$SCOPE" --resource-name "$FAMILY" --resource-type dedicated --limit-object value=16
az quota request status list --scope "$SCOPE" -o table # Succeeded, InProgress, or Failed
```
`Failed` on a family: Portal → **Quotas → Compute**, the region, the family row → **Request
adjustment**; if that is refused too, **Help + support → New support request → Service and
subscription limits (quotas) → Compute-VM (cores-vCPUs)**, naming the region, the family and
16. Tell the operator this is human moment #10 and can take days; do not wait inside the
install — take the small footprint, or stop here and resume when it is granted.
Where only confidential sizes are offered, `Standard_DC2s_v3` is an x86 fallback — but its
family (`Standard DCSv3 Family`) is often at 0 too; read its quota exactly as above before
relying on it. Being on pay-as-you-go does not mean quota is unlimited — it means the
restrictions are different ones.
**3. Fallback — the empirical probe.** The capability list is almost always right; when it
disagrees with reality (an apply still fails on Postgres), find a permitted region by
creating a tiny throwaway server in each candidate and keeping the one that reaches
`Ready`. Everything it creates lives in one resource group that the last line deletes:
```bash
az group create -n velt-pg-probe -l eastus -o none
for R in eastus westus2 westus3 southcentralus canadacentral northcentralus; do
az postgres flexible-server create -g velt-pg-probe -n "probe-$R" -l "$R" \
--tier Burstable --sku-name Standard_B1ms --storage-size 32 --version 16 \
--admin-user probeadmin --admin-password "Pr0beX9z-$R-Q" \
--public-access None --yes >/tmp/probe-$R.log 2>&1 &
done; wait
grep -l 'OfferRestricted' /tmp/probe-*.log # the restricted regions
az group delete -n velt-pg-probe --yes --no-wait
```
(The preflight below runs this probe for you with `--probe-postgres`.) Human moment #3:
propose the region to the operator and record `regionConfirmed: done`.
> **Azure Government.** The same checks apply; regions are `usgovvirginia`, `usgovarizona`,
> `usgovtexas`. The default Postgres SKU and node size must be offered there — read the
> same two outputs.
### 0.6 Run the install preflight
The release's own preflight runs the checks above, and more, in order — cheapest and safest
first — and **stops at the first failure**. It mutates nothing, except the optional
`--probe-postgres`, which creates and deletes the one probe resource group shown above.
```bash
./velt-portable-tools/portable-install-preflight.sh \
--location "$LOCATION" --subscription-id "$SUBSCRIPTION_ID" \
--acr "$ACR_NAME" --manifest-ref "$REG/velt-selfhost-manifest:latest" \
--portable-version "$PORTABLE_V" --manifest-file ./manifest.json \
--deployment-name "$DEPLOYMENT_NAME" \
--vm-size "$AKS_NODE_VM_SIZE" --nodes 3 --vcpus-per-node "$AKS_VCPUS_PER_NODE" \
--report ./preflight-report.json # add --probe-postgres to run the empirical probe too
echo "exit $?"
```
`--portable-version`, `--manifest-file` and `--deployment-name` need a tools bundle of
**0.1.14 or newer** (they carry P9's manifest-pin comparison and P10's name check). A `0.1.13`
bundle answers `unknown argument` — drop those three flags there; the checks they add are the
ones you did by hand in 0.1 and the inputs table.
`--manifest-ref` takes the tag, as written: P9 resolves it and verifies that digest itself (a
bundle of 0.1.15 or older reads a `…@sha256:` reference as a tag and refuses it). A newer bundle
also checks that `./manifest.json` is the manifest it verified, and refuses when `latest` has
moved since 0.1 — repeat 0.1, then run the preflight again.
Exit `0` = every check passed. Exit `1` = a check failed and the report names it. Exit `2`
= usage error. There is no flag to continue past a failure. Record the report path in
`artifacts.preflightReport`.
| id | What it checks | When it fails |
|---|---|---|
| P1 | `az` ≥ 2.60, `terraform` ≥ 1.6, `helm` ≥ 3.17, `node` ≥ 18, a `python3` that runs, `kubectl`, `cosign`, `oras` and **Azure's** `kubelogin` on PATH (the other tool of that name is refused by name) | install or upgrade the named tool (the tooling table) |
| P2 | the `az` session's subscription equals `--subscription-id` | `az login` / `az account set` |
| P3 | the deployer holds **Owner**, or **Contributor + User Access Administrator**, on the subscription — directly, through a group, or inherited | human moment #2 (0.3) |
| P4 | the eight resource providers are `Registered` | re-run the loop in 0.4 and wait |
| P5 | PostgreSQL Flexible Server is offered in the region; with `--probe-postgres` also the empirical probe | pick another region (0.5) — Postgres must share AKS's region |
| P6 | the AKS node VM size has no restrictions in the region for this subscription | set `aks_node_vm_size` to a size the region offers (0.5) |
| P7 | the vCPU quota of that size's family leaves room for `--nodes × --vcpus-per-node` plus one surge node; a WARN (not a failure) when the autoscaler's ceiling (`--max-nodes`, twice `--nodes` unless given, plus the surge) does not fit, naming the value to request | request a quota increase, or a smaller size |
| P8 | `--acr`: the registry exists in this subscription, or its name is available | pick a globally unique alphanumeric name |
| P9 | `--manifest-ref`: the manifest's digest verifies under cosign with the release identity; with `--portable-version` and the pulled `manifest.json`, its `tierB` pin is compared with the release you install (equal: fine; older or without `tools`: the 0.1 fallback applies, `manifestVerified: false`; newer: refused — install the pinned one) | STOP on a signature failure — a manifest that does not verify is not a release; on "newer", set `PORTABLE_V` to the pinned version |
| P10 | `--deployment-name`: the fixed names derived from it are free — Key Vault `-kv` is neither live nor **soft-deleted** in this subscription, and the storage account name `blob` is available | pick another `DEPLOYMENT_NAME`, or recover and adopt the old vault (Troubleshooting: "soft-deleted Key Vault") |
P8, P9 and P10 report `skip` (never `pass`) when their input is not supplied — always supply
all three. P10 is the one a REINSTALL trips on: a persistent-mode Key Vault keeps its name for
90 days after a teardown and cannot be purged, and without this check the apply fails at the vault
about twelve minutes in, after the network, cluster identities and Key Vault grant are built.
**Verify (phase gate):** the manifest verified and is stored; `velt-portable-tools/VERSION`
equals `PORTABLE_V`; `az account show` names `SUBSCRIPTION_ID`; the role check printed
`Owner` or the Contributor + User Access Administrator pair; all eight providers read
`Registered`; the preflight exited `0`; every input in the table is recorded; the operator
acknowledged the human-moments table. **Do not continue until the preflight has exited 0.**
Mark `0-preflight: done`.
---
## Phase 1 — The substrate (Terraform)
### 1.1 Pull and VERIFY the Terraform module
The Azure substrate ships as a signed OCI artifact of the same release:
```bash
MODULE_DIGEST=$(oras manifest fetch --descriptor "$REG/velt-portable-module:$PORTABLE_V" \
| python3 -c "import json,sys;print(json.load(sys.stdin)['digest'])")
cosign verify --certificate-identity-regexp "$IDENTITY" \
--certificate-oidc-issuer "$ISSUER" "$REG/velt-portable-module@$MODULE_DIGEST" \
&& oras pull "$REG/velt-portable-module@$MODULE_DIGEST" \
&& tar xzf "velt-portable-module-$PORTABLE_V.tar.gz"
ls terraform/azure/ # the *.tf files and .terraform.lock.hcl — nothing else
```
The archive unpacks to `./terraform/azure/` — the commands below work verbatim from the
directory you extracted it in. It holds the `*.tf` files and `.terraform.lock.hcl`, which
pins the provider versions this release was tested with; no state, tfvars or plan file can
ride along.
**Terraform versions and platforms.** The module plans on Terraform 1.6 and every later
release. The lock file carries provider hashes for five platforms: `linux_amd64`,
`linux_arm64`, `darwin_amd64`, `darwin_arm64` and `windows_amd64`. On any other platform a
plain `terraform init` adds your platform's hash to the lock file; if you run
`init -lockfile=readonly`, add yours first:
```bash
terraform -chdir=terraform/azure providers lock -platform=_
```
### 1.2 Write `terraform/azure/terraform.tfvars`
This file holds identifiers; keep it out of any shared repository. Every value comes from
the inputs table:
```hcl
mode = "persistent" # the customer posture: stable names, 90-day retention, 3 × D4ds_v5 nodes
cloud = "AzureCloud" # or "AzureUSGovernment"
location = "" # confirmed in Phase 0.5
resource_group_name = ""
deployment_name = "" # 3-16 chars; prefixes every resource
# postgres_sku = "GP_Standard_D2ds_v5" # the default; change only if 0.5 showed it is not offered
# aks_node_vm_size = "Standard_D4ds_v5" # the default; override with an x86 size the region offers (0.5)
# aks_node_vm_size = "Standard_D2ds_v4" # the small footprint (6 vCPU + 2 surge) when the quota is 10 (0.5, step 2b)
# aks_node_count = 3 # the default in persistent mode
# aks_node_max_count = 6 # the cluster autoscaler's ceiling: twice aks_node_count unless set;
# equal to aks_node_count keeps a fixed pool
# Metric alerts (on in persistent): six alerts on Postgres, the nodes and the Application Gateway notify ONE action
# group — the one the module creates with these receivers, or yours (then no addresses here).
alert_email_addresses = [""]
# alert_action_group_id = "/subscriptions//resourceGroups//providers/Microsoft.Insights/actionGroups/"
# PostgreSQL: the point-in-time restore window (7..35 days) and optional high availability (about doubles its cost).
# postgres_backup_retention_days = 14 # unset keeps the server's own window (7 on a new server)
# postgres_high_availability_mode = "ZoneRedundant" # or "SameZone" where the region has no availability zones
console_auth_issuer = "" # e.g. https://login.microsoftonline.com//v2.0
console_auth_client_id = ""
public_host = ""
acr_name = ""
# Who operates the cluster, and from where. Persistent mode REQUIRES the first value and
# ONE of the next two: the cluster is created with Entra ID as its identity provider and
# local accounts disabled, so a plan that names no admins or no reachable API server is refused.
aks_admin_group_object_ids = [""]
api_server_authorized_ip_ranges = [""] # OR: private_cluster_enabled = true
# private_cluster_enabled = true # a private API server, reachable from inside the VNet only
# aks_network_policy = "cilium" # default: cilium in AzureCloud, azure in AzureUSGovernment — changing it later REPLACES the cluster
# fips_enabled = true # default: true in AzureUSGovernment, false in AzureCloud
# Comment attachments (optional; module 0.1.14 or newer). Browsers upload and download the files straight to the
# storage account with a short-lived signed URL, so the account must answer CORS for the app's origins. Empty (the
# default) = no CORS rule, and attachments stay off for browsers. Each entry is an origin: https://host[:port], no
# path, no "*". A 0.1.13 module has no such variable: either install without attachments, or add the rule by hand
# after the apply — `az storage cors add --services b --methods PUT GET HEAD OPTIONS --origins \
# --allowed-headers 'content-type,x-ms-*' --exposed-headers 'content-length,content-type,etag,last-modified,x-ms-*' \
# --max-age 3600 --account-name ` — and remove it from this file before the next apply on 0.1.14+.
# attachment_cors_origins = [""]
```
> **Azure Government.** `cloud = "AzureUSGovernment"`, a Government region, the
> `login.microsoftonline.us` issuer. `fips_enabled` and the Azure network-policy engine
> are the defaults there; set `fips.enabled: true` in the chart values (Phase 4) to match
> the `fips_enabled` output.
### 1.3 Apply
Tell the operator: **this takes 12–15 minutes**.
```bash
terraform -chdir=terraform/azure init
terraform -chdir=terraform/azure plan -out tf.plan
terraform -chdir=terraform/azure apply tf.plan
```
It creates: the resource group · a VNet with the AKS subnet, the Application Gateway subnet
and a delegated Postgres subnet · a private DNS zone and link · Key Vault (purge protection
on — the storage customer-managed key requires it) with the deployer's Key Vault
Administrator grant and two CMK keys · the Postgres administrator password as a Key Vault
secret · the **AKS cluster** (Entra ID RBAC, workload identity, OIDC issuer, the Application
Gateway ingress add-on, the network-policy engine, the cluster autoscaler on the node pool
between `aks_node_count` and `aks_node_max_count`) · **PostgreSQL Flexible Server 16**
(private, CMK, storage auto-grow, point-in-time restore — 7 days unless `postgres_backup_retention_days` says otherwise) · a storage account (CMK) with the `attachments` container · three
user-assigned identities (workload / Postgres CMK / storage CMK) with their role assignments
· the federated identity credential · **a Log Analytics workspace and five diagnostic
settings** (AKS, Postgres, Key Vault, Blob and the Application Gateway send their audit logs
there; `enable_diagnostics = false` in the tfvars opts out — the ingestion is a recurring
cost, see "After the install") · and the file `terraform/azure/out/velt-console-config.json`
(the console's runtime config, spliced into the chart values in Phase 4) · in persistent mode,
**an action group and six metric alerts** (Postgres storage and CPU, node CPU and memory, the
Application Gateway's unhealthy backends and 5xx answers; `enable_alerts = false` opts out).
Two things the plan prints that are not problems: a `Warning: Argument is deprecated` from
the module's own code (a provider deprecation, harmless on this provider range), and, on a
SECOND apply, two "changes" to attributes Azure defaulted after the first (the node pool's
`upgrade_settings`, a `Microsoft.Storage` service endpoint on the Postgres subnet) — apply
them; nothing is replaced.
Expected on a first apply and **retryable** (see the list at the top): the Key Vault 403
while the grant propagates — wait 90 s, apply again; `Provider produced inconsistent result
after apply` / `ParentResourceNotFound: SOA` — apply again; a **404 on
`azurerm_monitor_diagnostic_setting.appgw`** as the LAST resource (module 0.1.13 / 0.1.14) —
the ingress add-on creates the Application Gateway itself and may be crash-looping on
`ApplicationGatewayInsufficientPermissionOnSubnet` while its subnet grant propagates. A
0.1.15-or-newer module waits for that gateway itself before creating the setting: the apply
prints `appgw_ready: waiting for the ingress add-on to create the Application Gateway (n/90)`
every 15 s, for up to about 22 minutes, and the normal case ends in `appgw_ready: … is
Succeeded`. If it stops at the ceiling instead, the add-on has backed off past its own
retry — run the recipe below and apply again (the same recipe closes the 404 on the older
modules):
```bash
az aks get-credentials -g "$RESOURCE_GROUP" -n "${DEPLOYMENT_NAME}-aks" --overwrite-existing && kubelogin convert-kubeconfig -l azurecli
kubectl -n kube-system rollout restart deploy/ingress-appgw-deployment
kubectl -n kube-system rollout status deploy/ingress-appgw-deployment --timeout=5m
az network application-gateway list -g "MC_${RESOURCE_GROUP}_${DEPLOYMENT_NAME}-aks_${LOCATION}" --query "[].provisioningState" -o tsv # Succeeded, ~90 s later
terraform -chdir=terraform/azure plan -out tf.plan && terraform -chdir=terraform/azure apply tf.plan # completes the one resource
```
a Postgres `zone can only be
changed when exchanged with the standby zone` on a *second* apply does not occur on this
module (it ignores zone drift) — if you see it, you are not on the release's module.
**Not retryable:** `LocationIsOfferRestricted` (region) and `The VM size … is not allowed`
(SKU) — Phase 0.5 again, then re-plan. **Changing the network plugin or policy engine
replaces the cluster** — Terraform says so at plan time (`# forces replacement` on
`network_profile`); never apply such a plan over a cluster holding data without a
maintenance window.
### 1.4 Sign in to the cluster, read the outputs
```bash
az aks get-credentials -g "$RESOURCE_GROUP" -n "${DEPLOYMENT_NAME}-aks" --overwrite-existing
kubelogin convert-kubeconfig -l azurecli # Entra sign-in through your az session; no local accounts exist
kubectl get nodes -L kubernetes.io/arch -L kubernetes.io/os # 3 nodes Ready, ARCH amd64, OS linux (`-o wide` prints no architecture column)
terraform -chdir=terraform/azure output
# ws_endpoint, api_endpoint, ingress_class_name, key_vault_id, workload_identity_client_id,
# blob_endpoint, blob_container, postgres_subnet_cidr, fips_enabled, aks_oidc_issuer, app_gateway_id,
# alert_action_group_id
```
Record `wsEndpoint`, `apiEndpoint`, `keyVaultId`, `workloadIdentityClientId`,
`postgresSubnetCidr` and the `fips_enabled` value in the state file — Phase 4's values file
reads all of them. If you set `attachment_cors_origins`, record `blobEndpoint` and
`blobContainer` too (the `storage:` block in Phase 4.1).
**Verify (phase gate):** `terraform apply` ended with `Apply complete!`; `kubectl get nodes`
shows every node `Ready` on `amd64`; `terraform output` prints all the outputs above
non-empty; `terraform/azure/out/velt-console-config.json` exists and parses. **Do not
continue until `kubectl get nodes` works through Entra** — every later phase needs it.
Mark `1-substrate: done`.
---
## Phase 2 — Images: verify, import, scan
Velt's release registry is public-read, but your cluster must not pull from it at runtime:
verify every image against the release identity, then import it into your own registry.
The chart is pointed at your registry only.
### 2.1 Verify, then import
```bash
# 1. Your registry, attached to the cluster for pull auth (first time only).
az acr create -g "$RESOURCE_GROUP" -n "$ACR_NAME" --sku Basic -o none
az aks update -g "$RESOURCE_GROUP" -n "${DEPLOYMENT_NAME}-aks" --attach-acr "$ACR_NAME"
# 2. VERIFY FIRST — an unverified image must never reach your registry. Resolve each tag once,
# verify that digest, and import THAT digest (`--image` keeps the tag your values pull).
for IMG in velt-gateway velt-workers velt-migrate; do
DIGEST=$(oras manifest fetch --descriptor "$REG/$IMG:$PORTABLE_V" \
| python3 -c "import json,sys;print(json.load(sys.stdin)['digest'])")
cosign verify --certificate-identity-regexp "$IDENTITY" \
--certificate-oidc-issuer "$ISSUER" "$REG/$IMG@$DIGEST" \
&& az acr import -n "$ACR_NAME" --source "$REG/$IMG@$DIGEST" --image "$IMG:$PORTABLE_V"
done
# 3. The console SPA. Its tag is the CONSOLE version, not $PORTABLE_V.
CONSOLE_DIGEST=$(oras manifest fetch --descriptor "$REG/velt-console-static:$CONSOLE_V" \
| python3 -c "import json,sys;print(json.load(sys.stdin)['digest'])")
cosign verify --certificate-identity-regexp "$IDENTITY" \
--certificate-oidc-issuer "$ISSUER" "$REG/velt-console-static@$CONSOLE_DIGEST" \
&& az acr import -n "$ACR_NAME" --source "$REG/velt-console-static@$CONSOLE_DIGEST" \
--image "velt-console-static:$CONSOLE_V"
# 4. The SDK bundle host. Its tag is the SDK version. REQUIRED for a deployment whose pages
# must not download JavaScript from Velt (see the bundle axis, Phase 4.1 and 7.1).
SDK_DIGEST=$(oras manifest fetch --descriptor "$REG/velt-sdk-static:$SDK_V" \
| python3 -c "import json,sys;print(json.load(sys.stdin)['digest'])")
cosign verify --certificate-identity-regexp "$IDENTITY" \
--certificate-oidc-issuer "$ISSUER" "$REG/velt-sdk-static@$SDK_DIGEST" \
&& az acr import -n "$ACR_NAME" --source "$REG/velt-sdk-static@$SDK_DIGEST" --image "velt-sdk-static:$SDK_V"
# 5. Valkey (the Redis-compatible cache). The chart's in-chart cache would otherwise pull
# valkey/valkey:8 from DOCKER HUB — the one image that does not come from your registry.
# Import it so the cluster needs no public egress at runtime.
# (Skip only if you set redis.externalUrl to a managed Redis — that disables the in-chart cache.)
az acr import -n "$ACR_NAME" --source docker.io/valkey/valkey:8 --image valkey:8
```
Valkey is an upstream image, so there is no Velt signature to verify — pin it by digest if
your policy requires provenance, or use a managed Redis and skip it.
**Import by the verified digest, never by the tag.** `az acr import --source :` asks
Velt's registry for the tag again, so what it copies is not necessarily what `cosign verify`
checked; `--source @sha256:…` copies exactly the manifest that verified, under the tag
`--image` names. Record each image's digest for the report — each must be the digest that
verified above:
```bash
for IMG in velt-gateway velt-workers velt-migrate; do
az acr repository show -n "$ACR_NAME" --image "$IMG:$PORTABLE_V" --query digest -o tsv
done
```
### 2.2 Vulnerability scan + infosec sign-off (human moment #5) — BEFORE Phase 3
**Ask the operator whether their organization requires a vulnerability scan of third-party
images before deployment.** Most enterprises do, and a finding disputed mid-deploy costs more
than one found here. Scan **their** registry copy, not Velt's.
```bash
# Azure-native: Microsoft Defender for Containers assesses ACR images on push.
az security pricing create -n ContainerRegistry --tier Standard # if not already on
az acr repository show-tags -n "$ACR_NAME" --repository velt-gateway # then read findings in Defender for Cloud
# Or one-shot, no subscription change, against the registry copies — EVERY image you imported, not only the gateway:
az acr login -n "$ACR_NAME"
for IMG in velt-gateway:$PORTABLE_V velt-workers:$PORTABLE_V velt-migrate:$PORTABLE_V velt-console-static:$CONSOLE_V velt-sdk-static:$SDK_V valkey:8; do
trivy image --severity CRITICAL,HIGH --ignore-unfixed "$ACR_NAME.azurecr.io/$IMG"
done
```
**Findings on `valkey:8`** are expected sooner or later: it is the upstream Valkey image the
guide tells you to import, not a Velt build, and Velt's release process does not scan it. The
operator decides: accept them (pin the imported digest in `redis.image`, Phase 4.1, so the
accepted image is the one that runs), re-import a newer `valkey:8` and rescan, or use a
managed Redis instead (`redis.externalUrl`). Record the decision with the sign-off.
Every Velt image carries a **signed SPDX SBOM** as a cosign attestation, so a central
scanner can be handed the digest and verify offline:
```bash
cosign verify-attestation --type spdxjson \
--certificate-identity-regexp "$IDENTITY" --certificate-oidc-issuer "$ISSUER" \
"$REG/velt-gateway:$PORTABLE_V" | python3 -c "import json,sys,base64; \
print(json.loads(base64.b64decode(json.load(sys.stdin)['payload']))['predicate']['name'])"
```
Velt's release process fails on any FIXABLE critical or high finding in every published
image, so a clean scan is expected — the point of this step is the operator's **sign-off**,
not the discovery. Record `infosecScanSignOff: done` (or `waived`) and the digests.
**Verify (phase gate):** `cosign verify` printed a verified result for every image you
imported, and each imported digest is the one that verified; `az acr repository list -n "$ACR_NAME"` lists `velt-gateway`, `velt-workers`,
`velt-migrate`, `velt-console-static`, `velt-sdk-static` and `valkey` (unless external
Redis); the sign-off is recorded. **Do not continue with an image that did not verify.**
Mark `2-images: done`.
---
## Phase 3 — Keys and secrets
The chart hands secrets to the pods through `envFrom: secretRef`, so **the key inside each
Kubernetes Secret must be the environment-variable name the image reads** — a filename-style
key like `jwt-public-key.pem` never reaches the process. The exact key names are below;
do not rename them.
### 3.1 Generate the deployment's ES256 signing keypair
The provision CLI ships inside every release image. Run it from the **migrate** image
(the one whose normal shape is a short-lived process) — no source checkout, nothing to
install. Three details of the `docker run` are load-bearing:
- `--entrypoint node` — the CLI is invoked as a node script, bypassing the image's
server entrypoint;
- `-u $(id -u):$(id -g)` — the image runs as a non-root uid that cannot write a directory
you own on a Linux host, so the container runs as YOUR uid;
- `-v $PWD/keys:/keys` with `--out-dir /keys` — the keys are written to the **mount**;
written anywhere else they stay inside the `--rm` container and vanish with it.
```bash
mkdir -p keys
# dist/portable-runtime/src/provision-cli.js is where the image keeps its compiled CLI. A shell FUNCTION, not a
# variable: zsh (the macOS default) does not word-split an unquoted $PCLI, so `$PCLI keygen …` fails there with
# "no such file or directory: docker run …". The function works in bash and zsh alike.
pcli() { docker run --rm -u "$(id -u):$(id -g)" --entrypoint node -v "$PWD/keys:/keys" \
"$ACR_NAME.azurecr.io/velt-migrate:$PORTABLE_V" dist/portable-runtime/src/provision-cli.js "$@"; }
KEYS_OUT=/keys
# On an arm64 host (Apple Silicon), add --platform linux/amd64 after --rm: the images are amd64-only.
pcli keygen --out-dir "$KEYS_OUT" --secret-name velt-jwt-es256 --namespace velt
ls -l keys/ # jwt-private.pem (mode 600, owned by you), jwt-public.pem, jwt-public-key-secret.yaml
```
If `ls keys/` is empty, or keygen fails with `EACCES`, see Troubleshooting (a root-owned
`./keys` from an earlier run, or a missing `-u`).
### 3.2 Create the namespace and the secrets
```bash
kubectl create namespace velt
# 1. The JWT PUBLIC key — key MUST be VELT_JWT_PUBLIC_KEY (every pod verifies tokens with it).
kubectl -n velt create secret generic velt-jwt-es256 \
--from-file=VELT_JWT_PUBLIC_KEY=./keys/jwt-public.pem
# 2. The JWT PRIVATE key — key MUST be VELT_JWT_PRIVATE_KEY. Required whenever identify or
# the console is enabled (both are, below): identify signs the tokens it mints with it.
# Only the gateway mounts it.
kubectl -n velt create secret generic velt-jwt-es256-private \
--from-file=VELT_JWT_PRIVATE_KEY=./keys/jwt-private.pem
```
**Two database connection strings, three secrets.** The application is not the database
administrator:
- `velt-postgres-dsn-admin` — the server administrator (`veltadmin`, password in Key Vault).
**The migrate Job alone uses it**: it applies the schema and, on its first run, CREATES
the application role with data privileges only.
- `velt-postgres-dsn` — the application role (`velt_app`). The gateway and workers use it.
- `velt-app-db` — the application role's password, handed to the migrate Job so it can
create or update that role. The same value goes into the application DSN.
Both DSN secrets use the key `VELT_PG_DSN`; the password secret uses `VELT_APP_DB_PASSWORD`.
```bash
KV_NAME="${DEPLOYMENT_NAME}-kv" # persistent mode: no random suffix
PGPASS=$(az keyvault secret show --vault-name "$KV_NAME" --name "${DEPLOYMENT_NAME}-pg-password" --query value -o tsv)
APP_PW=$(openssl rand -base64 24 | tr -d '/+=') # letters and digits only: it is URL-embedded below
PG_HOST="${DEPLOYMENT_NAME}-pg.postgres.database.azure.com" # Azure Government: ${DEPLOYMENT_NAME}-pg.postgres.database.usgovcloudapi.net
kubectl -n velt create secret generic velt-postgres-dsn-admin --from-literal=VELT_PG_DSN=\
"postgres://veltadmin:${PGPASS}@${PG_HOST}:5432/postgres?sslmode=require"
kubectl -n velt create secret generic velt-app-db --from-literal=VELT_APP_DB_PASSWORD="${APP_PW}"
kubectl -n velt create secret generic velt-postgres-dsn --from-literal=VELT_PG_DSN=\
"postgres://velt_app:${APP_PW}@${PG_HOST}:5432/postgres?sslmode=require"
unset PGPASS APP_PW
```
The `velt_app` role does not exist until the migrate hook has run once (Phase 4) — that is
fine: the gateway and workers start only after the hook succeeds.
### 3.3 Put the private key away
`./keys/jwt-private.pem` is the deployment's signing key. Once both JWT secrets exist, the
cluster no longer needs the file. Keep ONE copy outside the cluster — you will mint a smoke
token from it in Phase 6, and Velt support may ask you to mint one later:
```bash
az keyvault secret set --vault-name "$KV_NAME" --name "${DEPLOYMENT_NAME}-jwt-private-key" \
--file ./keys/jwt-private.pem -o none
rm -r keys
```
Record `artifacts.jwtPrivateKeyLocation` as the Key Vault secret name (never the value).
Never commit the file anywhere.
**Verify (phase gate):**
```bash
kubectl -n velt get secret velt-jwt-es256 velt-jwt-es256-private velt-postgres-dsn velt-postgres-dsn-admin velt-app-db
kubectl -n velt get secret velt-jwt-es256 -o jsonpath='{.data}' | python3 -c "import json,sys;print(sorted(json.load(sys.stdin).keys()))"
# ['VELT_JWT_PUBLIC_KEY'] — the KEY name, exactly
```
All five secrets exist; the key names are `VELT_JWT_PUBLIC_KEY`, `VELT_JWT_PRIVATE_KEY`,
`VELT_PG_DSN` (both DSN secrets) and `VELT_APP_DB_PASSWORD`; the private key is in Key Vault
and `./keys` is gone. **Do not continue until the key-name check prints exactly
`VELT_JWT_PUBLIC_KEY`** — a wrong key name is a gateway that cannot start. Mark
`3-keys-secrets: done`.
---
## Phase 4 — Install the application (Helm)
### 4.1 The values file — complete
Write `values-.yaml`. Everything the chart needs is here; no image in this file comes
from anywhere but your own registry. Replace every `` from the inputs table and
the Phase 1 outputs.
```yaml
image:
registry: .azurecr.io # NO trailing slash — the chart adds the /
tag: "0.1.13" # $PORTABLE_V — also what the gateway's /version reports
serviceAccount:
annotations:
azure.workload.identity/client-id: ""
podLabels:
azure.workload.identity/use: "true"
ingress:
host: # the public_host you applied
className: azure-application-gateway # the ingress_class_name output
timeoutAnnotationKey: appgw.ingress.kubernetes.io/request-timeout # carries the WS idle timeout to App Gateway
tls:
secretName: velt-portable-tls # the customer-supplied certificate (Phase 5.3). Leave "" for the
# Key Vault path, and for the FIRST install on the cert-manager path.
secrets:
postgresDsnSecretName: velt-postgres-dsn # the APPLICATION role (Phase 3)
migratePostgresDsnSecretName: velt-postgres-dsn-admin # the administrator — migrate Job only
appDbPasswordSecretName: velt-app-db # the migrate Job creates/updates velt_app with it
jwtEs256SecretName: velt-jwt-es256
# svixSecretName: velt-svix # ONLY with webhookV2Delivery "" or "svix": your own Svix token, key VELT_SVIX_API_KEY
# REQUIRED whenever the console is enabled. `console.enabled` only deploys the SPA; without
# this block the gateway answers 501 on /console/session — the console loads, redirects to
# the IdP, comes back, and cannot sign anyone in.
# ⚠ Spelling: oidcIssuer / oidcClientId. Helm accepts `issuer:` / `clientId:` SILENTLY and
# leaves the real values empty. The chart's own render check can only fire once enabled is true.
consoleAuth:
enabled: true
oidcIssuer: # character for character the issuer of
# /.well-known/openid-configuration — see 4.2
oidcClientId:
jwtPrivateKeySecretName: velt-jwt-es256-private
# preferredUsernameFallback: true # Entra: see 4.2 — only if your chart lists this key
# REQUIRED for the SDK to mint self-hosted tokens. `allowedOrigins` is OPTIONAL: what identify
# requires is each workspace's allowed DOMAINS (seeded in 4.5). If you DO set allowedOrigins it
# narrows further, refuses "*" by design, and every origin listed must also be admitted by
# corsAllowedOrigins or the gateway refuses to boot.
identify:
enabled: true
wsEndpoint: wss:///ws
apiEndpoint: https:///api
# allowedOrigins: "https://"
# CORS — the browser origins of the customer's pages, exact (scheme + host [+ port]). EMPTY
# means browsers are blocked. A malformed entry is dropped with a warning in the gateway log.
corsAllowedOrigins: "https://,https://"
# Webhook V2 delivery — ONE choice per deployment. "" or "svix": deliver through your own Svix
# (then set secrets.svixSecretName). "direct": no Svix — the workers sign each event with Svix's
# own scheme (svix-id / svix-timestamp / svix-signature, a whsec_ secret your receiver verifies
# with the unchanged svix package) and POST it from inside the cluster to the endpoints you
# register through the console or the REST API. Nothing leaves to a third party. "direct"
# beside secrets.svixSecretName FAILS the render on purpose. Direct mode sends once and never
# retries; each attempt is a ledger row kept webhookV2DeliveryRetentionDays (7).
webhookV2Delivery: "" # "" | "svix" | "direct"
# webhookV2DeliveryRetentionDays: 7 # direct mode only; 0 keeps the ledger forever
# NetworkPolicy — enforced because Phase 1 created the cluster with a network-policy engine.
# Ingress restriction is ON (Redis only from the gateway and workers; workers and migrate
# accept nothing). Egress restriction is opt-in: list the Postgres subnet and everything the
# deployment calls out to — Blob, Entra / your IdP, Svix (not in direct mode), your email
# provider, your webhook receivers and permission provider. Leave `restricted: false` until
# you have that list; a policy that silently blocks a webhook is the failure you least want
# to debug.
networkPolicy:
enabled: true
egress:
restricted: false
postgresCidrs: [""] # e.g. ["10.10.16.0/24"]
allowCidrs: [] # the list above, once restricted: true
# FIPS mode for the APPLICATION. The node pools' FIPS is the substrate's switch (the
# fips_enabled output); the image ships the OpenSSL FIPS provider and refuses to start when
# this says true and FIPS is not active. Set it to match the output.
fips:
enabled: false # true on Azure Government / when fips_enabled is true
redis:
# Imported in Phase 2. Without this the chart pulls valkey/valkey:8 from Docker Hub.
image: .azurecr.io/valkey:8
# ALTERNATIVE: drop `image` and use a managed Redis instead; this disables the in-chart cache:
# externalUrl: rediss://.redis.cache.windows.net:6380
# The in-chart Redis ALWAYS requires a password. Nothing to set — the chart creates
# velt-redis-auth with a random one, keeps it across upgrades, and hands it to the pods.
# THE BUNDLE AXIS — not covered by `strict`, and the most consequential block on this page
# for a deployment that must not download JavaScript from Velt. The SDK's loader picks its
# bundle URL BEFORE velt.js exists, from {staging, develop, version, proxyDomain |
# proxyConfig.cdnHost}; `selfHosted` / `strict` are not loader inputs. Unpinned, it resolves
# https://cdn.velt.dev/lib/sdk@/velt.js. This block serves the bundle from YOUR ingress at
# /lib/sdk@/; the app-side pin (Phase 7.1) is what makes it used. Both halves or neither.
sdk:
enabled: true
version: "6.0.24" # $SDK_V — this is part of the served PATH
tag: "6.0.24" # the image tag; defaults to sdk.version
retainedVersions: [] # on a LATER SDK upgrade: the old version(s) to keep serving while apps repoint
# (chart 0.1.16+; one image serves one version — see the Upgrade Guide, Phase U5)
console:
enabled: true
tag: "" # $CONSOLE_V from the manifest (0.1 — the examples in this guide are older than the manifest)
config: {} # REPLACED by the splice in 4.2 — never leave it {} with enabled: true
# WHICH MODULES this deployment serves — the one list the console, the SDK and the gateway all
# read (chart 0.1.15 or newer; on 0.1.13 / 0.1.14 set `restApi.enabled: true` instead). `core` is the SDK
# data plane and is always on. Add `rest-api` for the customer REST API:
# a token-authenticated surface on the public ingress
# (`/api/v2/...`) that a deployment whose integrations all go through the SDK never needs; its
# credential is the API key's auth token you seed in Phase 4.5. Without `rest-api`, `/api/v2/...`
# answers a bare 404 and the console hides the REST features. No other module id is served on
# this deployment — listing one only makes the console show features that answer `unimplemented`.
enabledModules:
- core
# - rest-api
# Comment attachments (module and chart 0.1.14 or newer) — ONLY if you set attachment_cors_origins
# in Phase 1.2 (or added the CORS rule by hand on a 0.1.13 module); otherwise leave the whole block
# out (the attachment calls then answer `unimplemented` and the SDK hides the button). The gateway signs short-lived upload / download URLs for the storage account and
# the workers delete a deleted document's files; both use the workload identity above.
# storage:
# blobEndpoint: "" # https://.blob.core.windows.net/ (.usgovcloudapi.net on Government)
# container: "" # attachments
# tokenEndpoint: "https://login.microsoftonline.com//oauth2/v2.0/token" # Government: login.microsoftonline.us
# clientId: ""
# tenantId: ""
# # federatedTokenFile and sasTtlSecondsMax (3600, the longest a signed URL lives) keep their defaults
# Numbers of a million or more — restApi.maxBodyBytes, maxSnapshotBytes, dataRate.*, and any
# value under `env:` — may be written bare; the chart renders every numeric entry as its digits.
```
Defaults the file above overrides on purpose: `redis.image` defaults to Docker Hub,
`console.enabled` and `sdk.enabled` default to `false`, `networkPolicy` and `fips` to the
posture shown. A deployment that takes the defaults gets an egress dependency it did not ask
for, no console and no SDK host.
### 4.2 Splice the console config, check the issuer
Terraform wrote the console's runtime config in Phase 1. Replace the `console:` block's
`config: {}` with it rather than retyping it:
```bash
# Remove the placeholder console block from values-.yaml first (the last three lines of the
# `console:` block above), then append the generated one. JSON is valid YAML.
python3 - "$CONSOLE_V" <<'EOF' >> values-.yaml
import json, sys
cfg = json.load(open('terraform/azure/out/velt-console-config.json'))
print('console:')
print(' enabled: true')
print(f' tag: "{sys.argv[1]}"')
print(' config:', json.dumps(cfg))
EOF
# Or with yq: yq -o yaml '{"console":{"config": .}}' terraform/azure/out/velt-console-config.json
```
The generated config carries `enabledModules: ["core"]` from the module's own variable. Leave
it: the chart appends `rest-api` to the console's copy when your values turn the REST API on
(Phase 4.1), so the console shows what the gateway serves.
`auth.issuer` in that config must be reachable **from the browser**, not from the cluster —
the SPA fetches `{issuer}/.well-known/openid-configuration` to start the OIDC
authorization-code + PKCE flow. Set `console.enabled: false` if you are not deploying the
console; never `true` with an empty `config` — the chart fails the render on purpose,
because the SPA's fallback for a missing config is to boot on build defaults: a blank console
with no error.
**Two Entra checks before the install** (either one wrong is a console that cannot sign in):
```bash
# 1. The issuer, character for character. Both lines must print the same text; if they differ,
# use the SECOND one as consoleAuth.oidcIssuer AND as auth.issuer in the console config.
echo "$CONSOLE_AUTH_ISSUER"
curl -s "${CONSOLE_AUTH_ISSUER%/}/.well-known/openid-configuration" | jq -r '.issuer'
# 2. Does this release's chart carry the preferred_username fallback? An Entra member with no
# `mail` attribute has no `email` claim and signs in by UPN; releases that gate that fallback
# need the key set to true, or that member gets 401. The chart is resolved to its digest and verified first, as in 4.3.
CHART_DIGEST=$(oras manifest fetch --descriptor "$REG/velt-portable:$PORTABLE_V" \
| python3 -c "import json,sys;print(json.load(sys.stdin)['digest'])")
cosign verify --certificate-identity-regexp "$IDENTITY" --certificate-oidc-issuer "$ISSUER" "$REG/velt-portable@$CHART_DIGEST" \
&& helm show values "oci://$REG/velt-portable@$CHART_DIGEST" | grep -c preferredUsernameFallback
# 1 or more → add `preferredUsernameFallback: true` under consoleAuth in values-.yaml.
# 0 → the key does not exist on this release; the fallback is always on. Leave it out.
```
### 4.3 Verify the chart, then install
Tell the operator: **the install takes 3–5 minutes** (the schema migration runs first, then
the pods start).
```bash
CHART_DIGEST=$(oras manifest fetch --descriptor "$REG/velt-portable:$PORTABLE_V" \
| python3 -c "import json,sys;print(json.load(sys.stdin)['digest'])")
cosign verify --certificate-identity-regexp "$IDENTITY" \
--certificate-oidc-issuer "$ISSUER" "$REG/velt-portable@$CHART_DIGEST" \
&& helm install velt "oci://$REG/velt-portable@$CHART_DIGEST" -n velt -f values-.yaml --timeout 8m
```
The chart is installed by the digest that verified, like every other artifact. Do not add
`--version`: Helm 3.17 ignores it beside a digest, and newer Helm refuses the pair when the tag has
moved. Every later `helm upgrade` in this guide re-applies `$CHART_DIGEST`; in a new shell, set and
verify it again by running this block up to the `cosign verify` (everything before `&& helm install`).
The release is named `velt` in namespace `velt`; every resource is `velt-`
(`velt-gateway`, `velt-workers`, `velt-migrate`, `velt-redis`, `velt-console`, `velt-sdk`),
and that pair (`--release velt --namespace velt`) is what the receipt, support bundle and
upgrade commands take.
The **migrate** Job runs as a pre-install hook and applies every migration the release
carries, in order, against your Postgres before the application pods start; a migration
failure aborts the install loudly. Then the gateway, workers, Valkey, console and SDK pods
come up.
### 4.4 Verify the migrate hook actually applied the schema
A `Completed` migrate pod is necessary and NOT sufficient — read the **migration ledger**.
The runner commits each version's DDL and its `schema_migrations` row in one transaction, so
a recorded version means that version fully applied; a table count cannot tell you that.
```bash
kubectl -n velt get pods | grep migrate # Completed, not Error
kubectl -n velt logs job/velt-migrate | tail -20
# The schema head this chart applies is published ON the chart:
helm show chart "oci://$REG/velt-portable@$CHART_DIGEST" | grep schema-head
# velt.dev/schema-head: "126" ← an example: use the number YOUR chart prints
# The ledger's highest version must equal that number. Run the pod DETACHED, wait for it, then read its
# log: an attached `kubectl run --rm -i` can race the short-lived container and print NOTHING, which
# looks like an empty ledger (it is not — run again if you see only "try pressing enter").
kubectl -n velt run pgcheck --restart=Never --image=postgres:16-alpine \
--overrides='{"spec":{"containers":[{"name":"pgcheck","image":"postgres:16-alpine",
"envFrom":[{"secretRef":{"name":"velt-postgres-dsn"}}],
"command":["sh","-c","psql \"$VELT_PG_DSN\" -tAc \"select version, id, applied_at from schema_migrations order by version\""]}]}}'
kubectl -n velt wait pod/pgcheck --for=jsonpath='{.status.phase}'=Succeeded --timeout=120s
kubectl -n velt logs pgcheck | tee ledger.txt
kubectl -n velt delete pod pgcheck
SCHEMA_HEAD=$(awk -F'|' '{print $1}' ledger.txt | sort -n | tail -1); echo "ledger head: $SCHEMA_HEAD" # the final report's receipt takes it
```
The ledger carries one row per migration the release ships; **gaps in the numbering are
normal** (a version number reserved and never used is skipped). What must hold is
`max(version)` = the chart's `velt.dev/schema-head`. A chart with no such annotation is a
release older than `0.1.13`; then compare against the head Velt support names for that
release. The runner also announces **version skew** on its own — a database carrying versions
this image does not know about (a rollback mid-flight) is reported, never silently
re-applied; reconcile the image tag against the database before continuing if the log says so.
(`postgres:16-alpine` is pulled from Docker Hub for this one-off check; air-gapped clusters
import it into the registry first, like Valkey, and use that reference.)
### 4.5 Seed the first workspace (in-cluster)
The database is private, so run the provision CLI **inside the cluster**, from the migrate
image, with the DSN from the same secret the gateway uses — there is no second copy of it.
It is idempotent: re-running upserts, never duplicates.
Two credentials come out of this step, for two different ladders of the REST API — do not
confuse them:
- the **API key's auth token** (`x-velt-api-key` + `x-velt-auth-token`): what the REST API doc
calls "that apiKey's auth token", and what the console shows as the API key's **Auth Token**. You
choose it (`` in the inputs table) and the seed stores it hashed;
- the **workspace auth token** (`x-velt-workspace-id` + `x-velt-workspace-auth-token`): the
deployment-scoped credential of the `/v2/workspace/*` family. The seed GENERATES it and
prints it once — the console shows it as **Workspace Auth Token**.
A request on one ladder with the other ladder's token is refused with `403 Invalid Auth token.`.
```bash
# Inline JSON for --project-config and --identify-config is fine: neither carries a secret. The inner
# quotes are backslash-escaped because each value is spliced into the --overrides JSON string below.
PRIVATE_COMMENTS='{\"isPrivateCommentsEnabled\":true}' # or '{}' when the customer does not use private comments
# The identify configuration needs BOTH flags — a config without them is refused:
# requireAutoOrgUser true: a user who identifies into an organization is added to it (the hosted default for a
# new API key); false: add users to organizations through the REST API first
# requireJwtToken false: identify accepts a user without a token from your server (the hosted default); true:
# only users your server signs (needs your verification key as customerJwt)
IDENTIFY_CONFIG='{\"allowedOrigins\":[\"\"],\"requireAutoOrgUser\":true,\"requireJwtToken\":false}' # every browser origin of the app, comma separated inside the array
API_AUTH_TOKEN=$(az keyvault secret show --vault-name "$KV_NAME" --name "${DEPLOYMENT_NAME}-api-auth-token" --query value -o tsv) # , stored before this step
kubectl -n velt run velt-provision --restart=Never \
--image="$ACR_NAME.azurecr.io/velt-migrate:$PORTABLE_V" \
--overrides='{"spec":{"containers":[{"name":"velt-provision",
"image":"'"$ACR_NAME"'.azurecr.io/velt-migrate:'"$PORTABLE_V"'",
"envFrom":[{"secretRef":{"name":"velt-postgres-dsn"}}],
"command":["node","dist/portable-runtime/src/provision-cli.js","seed",
"--api-key","","--org-id","","--user-id-hash","",
"--console-admin-email","",
"--allowed-domains","",
"--identify-config","'"$IDENTIFY_CONFIG"'",
"--auth-token","'"$API_AUTH_TOKEN"'",
"--project-config","'"$PRIVATE_COMMENTS"'"]}]}}'
# The log holds the generated WORKSPACE auth token (one bare line, printed once) next to a node `pg`
# SSL-mode warning: capture it straight into the secret store, then delete the pod so the log is gone.
# ONE command: a failed seed, or a line that is not a token, stores nothing and keeps the pod so you
# can read its log (a re-seed prints no token and ends here too — expected; delete the pod).
kubectl -n velt wait pod/velt-provision --for=jsonpath='{.status.phase}'=Succeeded --timeout=120s \
&& ( umask 077; kubectl -n velt logs velt-provision | grep -vE '^\[|warning|Warning' | tail -1 > seed-workspace-token.txt ) \
&& grep -qE '^[A-Za-z0-9_-]{32,}$' seed-workspace-token.txt \
&& az keyvault secret set --vault-name "$KV_NAME" --name "${DEPLOYMENT_NAME}-workspace-auth-token" --file seed-workspace-token.txt -o none \
&& kubectl -n velt delete pod velt-provision \
|| echo "NOT stored — read: kubectl -n velt logs velt-provision; fix, then kubectl -n velt delete pod velt-provision"
rm -f seed-workspace-token.txt; unset API_AUTH_TOKEN
```
The pod runs detached and is read from its log on purpose: an attached `kubectl run --rm -i`
can race the container and lose the output, and a re-seed never prints the workspace token
again (it is stored hashed). If the log shows `workspace auth token already provisioned`, the
first run's token is the one in the secret store; re-seeding is otherwise idempotent.
**`--identify-config` matters beyond CORS:** a workspace seeded without one has no identify
configuration, and adding a second workspace in the console then fails with `cannot create a
second live apiKey … until every existing apiKey has a valid identify config` until the first
has one (Troubleshooting). Pass the app's origins here and the console is consistent from the
start.
What this seeds and why each flag matters:
- the API key row — without it the authorization evaluator denies everything;
- the first organization and the first admin user (`--user-id-hash` is the hash of the user
id your app will identify with);
- `--console-admin-email` — console members are matched by email; this is who can sign in;
- `--allowed-domains` — identify refuses a workspace that has none; add more later through
the console or the REST API;
- `--identify-config` — the workspace's identify configuration (its allowed browser origins);
the console requires one before a second workspace can be added;
- `--auth-token` — the API key's auth token for the REST API (`x-velt-auth-token`); without it
the REST API has no credential until one is created in the console;
- `--project-config` with `isPrivateCommentsEnabled: true` — **required when the customer
uses private comments**: the unseeded default is `false`, under which the SDK's viewer
does not build visibility tokens and a comment restricted to its own author disappears
from the author's view.
### 4.6 One port, two paths
The gateway serves WebSocket, REST and health on **one** port (8080). The chart probes
`/readyz` and `/healthz` on it and the Service maps both published ports (`ws:8080`,
`api:8081`) to it, so the ingress `/ws` vs `/api` split still works. A pod that hangs at
`0/1` with `connection refused` on its readiness probe is running a chart other than this
release's.
**Verify (phase gate):**
```bash
helm -n velt status velt | grep STATUS # deployed
kubectl -n velt get pods # velt-migrate-* Completed; gateway, workers, redis, console, sdk Running and Ready
```
`helm status` is `deployed`; the migrate pod is `Completed`; the ledger's `max(version)`
equals the chart's `velt.dev/schema-head`; the seed printed its completion and the auth token
was captured; every application pod is `Running` and Ready. **Do not continue until the
ledger head equals the chart annotation** — a half-applied schema surfaces as runtime errors
much later. Record `versions.schemaHead`. Mark `4-helm: done`.
---
## Phase 5 — Ingress, DNS and TLS (Application Gateway)
Phase 1 enabled the AKS **Application Gateway ingress add-on** in its own subnet. It programs
an Application Gateway from the chart's `Ingress` object; TLS terminates at the App Gateway,
and the gateway Service stays ClusterIP — there is no plaintext side door. Do this in THIS
order: the App Gateway's public IP does not exist until the add-on applies, and the
certificate's name must equal `PUBLIC_HOST`.
### 5.1 Read the Application Gateway IP; the first-sync restart
```bash
NODE_RG=$(az aks show -n "${DEPLOYMENT_NAME}-aks" -g "$RESOURCE_GROUP" --query nodeResourceGroup -o tsv)
az network application-gateway list -g "$NODE_RG" --query '[].{name:name, state:operationalState, prov:provisioningState}' -o table
az network public-ip list -g "$NODE_RG" --query "[?contains(name,'appgw')].ipAddress" -o tsv
kubectl -n velt get ingress # ADDRESS must show that IP
```
**Expected on a freshly created cluster:** the add-on reconciles while the App Gateway is
still `Stopped`/warming — the ingress shows an EMPTY ADDRESS and `:443` answers nothing, with
the add-on pod logging `Reset IP ... is in stopped state`. Once the table above reads
`Running` / `Succeeded`, force one reconcile and allow 3–6 minutes of
`provisioningState: Updating`:
```bash
kubectl -n kube-system rollout restart deploy/ingress-appgw-deployment
kubectl -n velt get ingress -w # ADDRESS appears, typically within a minute
```
If the add-on pod is in `CrashLoopBackOff` naming `ApplicationGatewayInsufficientPermissionOnSubnet`,
it is waiting on a role assignment the module created after the cluster; its own retry picks
it up as RBAC propagates — restart it with the same command rather than waiting out the
backoff ceiling.
Record `artifacts.appGatewayIp`.
### 5.2 DNS (human moment #6)
Create an **A record** for `PUBLIC_HOST` pointing at that IP. `PUBLIC_HOST` is the single
input that builds the WS/API endpoints, the ingress host AND the certificate's name — keep
them equal.
**No domain? Azure gives you a real one.** Every public IP takes a DNS name label, published
in public DNS as `