# Infra CLI

> Mirrored from [`iblai/infra-cli`](https://github.com/iblai/infra-cli) · [`README.md`](https://github.com/iblai/infra-cli/blob/main/README.md). This page is generated — edit it in the repository, not here.

Interactive CLI for provisioning and configuring the [ibl.ai](https://ibl.ai) platform on **AWS** or **GCP** (single-server). End-to-end infrastructure with Terraform, full application setup with Ansible. Can also bootstrap existing servers (any cloud or bare metal) without Terraform.

[AWS](https://aws.amazon.com)
[GCP](https://cloud.google.com)
[Terraform](https://www.terraform.io)
[Python 3.11+](https://www.python.org)
[License: MIT](https://github.com/iblai/infra-cli/blob/main/LICENSE)

> **Note:** This repository contains the installation and infrastructure provisioning tooling. Access to the ibl.ai Docker images and platform codebase requires a license. To get started, reach out at [ibl.ai/contact](https://ibl.ai/contact).

---

## Quick start

Five steps from zero to a running platform:

```bash
# 1. Install (needs Python 3.11+, Terraform on PATH)
git clone git@github.com:iblai/iblai-infra-ops.git && cd iblai-infra-ops
uv sync                        # AWS only
uv sync --extra gcp            # AWS + GCP support

# 2. Verify your cloud credentials have what's needed
uv run iblai infra permissions --check                                  # AWS
uv run iblai infra permissions --provider gcp --check --project <ID>    # GCP

# 3. Provision the infrastructure (interactive wizard — pick AWS or GCP)
uv run iblai infra provision

# 4. Install the platform on the new VM (Ansible over SSH)
uv run iblai infra setup <project-name>

# 5. Open the app
#    https://learn.<your-domain>
```

Prefer no prompts? Steps 3–4 also run from a `.env` file — see [Non-interactive](#4-non-interactive-provision--setup-env-file).

> **Deploying on Google Cloud?** The flow above is identical; read the short **[GCP guide](/developer/infra-cli/gcp)** first for GCP prerequisites (project, APIs, auth) and cert/DNS behavior.

## Prerequisites

| | Requirement |
|---|---|
| **Always** | Python 3.11+ · [uv](https://docs.astral.sh/uv/) (or pip) · [Terraform](https://developer.hashicorp.com/terraform/install) on PATH · a domain you control · GitHub access to the private ibl.ai packages (licensed) |
| **AWS** | An account with EC2, ELB, S3, ACM, Route53, IAM, STS permissions (`iblai infra permissions` prints the exact policy) |
| **GCP** | A project with billing; `compute` + `dns` APIs enabled; roles `compute.admin`, `dns.admin`, `iam.serviceAccountUser`; auth via `gcloud auth application-default login` or a service-account key. AWS S3 is still used for object storage — see the [GCP guide](/developer/infra-cli/gcp) |
| **Existing server** (bare metal / other cloud) | SSH access to the machine (Ubuntu 22.04, min 100 GB disk) and DNS you can point at it. No Terraform runs; only `iblai infra setup` is used. AWS credentials are still required — see below |

**AWS credentials are required on every deployment, including bare metal and GCP.** They are not for hosting. Setup takes two keys, one per job, so each can be scoped to only what it needs (give the same key twice if one does both):

- **ECR key** (required) — pulls the platform container images. It becomes the server's default AWS CLI profile, which every ECR login uses, and never reaches the platform's own settings.
- **S3 key** (optional) — the platform's AWS key, stored as the root-level `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` in `/ibl/secrets.yml`. DM S3 storage and SCORM uploads use it, and so do backup and monitoring alerts when an SNS topic is configured (it then needs `sns:Publish` too). Backup *uploads* go through the AWS CLI on the server — its default profile (the ECR key) unless `IBL_BACKUPS.EXTERNAL_STORAGE.S3_PROFILE` names another. Skipping the S3 key leaves whatever the server already has.

The platform keeps media, static assets, and backups in three S3 buckets. Provisioning on AWS creates them for you. On any other target, create them yourself before setup, named:

  ```
  <project>-<environment>-<domain-with-dots-as-dashes>-backups
  <project>-<environment>-<domain-with-dots-as-dashes>-dm-media
  <project>-<environment>-<domain-with-dots-as-dashes>-dm-static   # public-read
  ```

**Installed automatically** as Python dependencies: `ansible-core` (runs the setup), `boto3` (AWS SDK), and — with `--extra gcp` — the Google Cloud SDKs. Terraform is called as a subprocess and must be installed separately.

**What setup installs on the target server:** Docker Engine + compose, pyenv + Python 3.11.8, AWS CLI v2, and iblai-cli-ops (the platform management CLI — private repository; unauthenticated requests see a 404).

## Install

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh     # install uv if needed

git clone git@github.com:iblai/iblai-infra-ops.git
cd iblai-infra-ops
uv sync --extra gcp                                  # or plain `uv sync` for AWS only
```

Verify:

```bash
uv run iblai --version
terraform --version
```

Every command below can be run as `uv run iblai ...`, or just `iblai ...` after `source .venv/bin/activate`.

## Usage

Run `iblai infra` with no arguments for an interactive menu of all commands.

### 1. Check cloud permissions

```bash
iblai infra permissions [--check]                                  # AWS: IAM policy JSON + dry-run
iblai infra permissions --provider gcp [--check --project <ID>]    # GCP: roles + APIs + probe
```

### 2. Provision infrastructure

```bash
iblai infra provision
```

The wizard walks you through, validating as you go:

1. **Cloud** — AWS or GCP
2. **Credentials** — AWS: profile / keys / env vars · GCP: project ID, region/zone, ADC or key file
3. **Topology** — AWS: single-server, multi-server, or call-server (LiveKit) · GCP: single-server
4. **Project & compute** — name, environment, machine/instance type, disk (min **100 GB**; a 32 GB-RAM pick prints a heads-up — 64 GB recommended when AI features are on)
5. **Network & SSH** — CIDR, your IP for SSH access (auto-detected), SSH key (generate or provide)
6. **Domain & certificates** — AWS: Route53 + ACM / upload / none · GCP: Cloud DNS + Google-managed cert (validates asynchronously, 10–60 min) / upload / none
7. **WAF (optional, AWS single-server)** — AWS WAFv2 on the ALB; skippable, add later with `iblai infra waf enable`
8. **Review** — full summary before anything is created

Terraform then runs with live per-resource progress. The results panel shows the VM IP, SSH command, and app URL.

### 3. Set up the platform

```bash
iblai infra setup <name>       # a Terraform-provisioned environment (IP/domain/key auto-filled)
iblai infra setup              # any existing server (bare metal / other cloud) — prompts for everything
```

Prompts for: release tag of iblai-prod-images (the one version knob — the matching `iblai-cli-ops` version is resolved automatically from its pin), tenant platform name (blank = default), enable-AI toggle, optional integrations (SMTP / Stripe / Google SSO / Microsoft SSO — each off unless configured), GitHub PAT, AWS credentials (an ECR key, then an S3 key — separate, the same, or skipped), an LLM key (OpenAI or OpenRouter, optional), and super admin credentials.

**Everything optional can wait.** SMTP, SSO, Stripe and the LLM key are each skippable here — the platform comes up without them. Only the GitHub PAT, the ECR key and admin credentials are needed on the first run. Add any of them later with [`iblai infra configure <name>`](#7-optional-feature-toggles-post-provision) — no need to re-run setup.

**Installing from your own repos.** Three prompts control where the private packages come from. Press Enter to accept the defaults, or point them at a fork or per-deployment copy:

| Prompt | Default |
|---|---|
| `GitHub org owning the private packages` | `iblai` |
| `CLI ops repo (or repo/subdir for monorepo)` | `iblai-cli-ops` |
| `Prod images repo (or repo/subdir for monorepo)` | `iblai-prod-images` |

Both repo prompts also accept a `repo/subdir` path when the two packages live in one repository. The GitHub PAT needs **Contents: Read** on whichever repositories you name. The same three values are available as `GITHUB_ORG`, `CLI_OPS_REPO` and `PROD_IMAGES_REPO` in `.env.setup.example`.

The playbook runs 16 roles in phases:

| Phase | Roles | What it does |
|---|---|---|
| Host setup | `docker`, `awscli`, `python` | Docker + compose, AWS CLI v2, pyenv + Python 3.11.8 |
| Platform install | `ibl_cli_ops`, `ibl_platform` | Installs the pinned platform packages; configures domain, gateway, defaults |
| Core services | `ibl_dm`, `ibl_edx`, `ibl_spa` | Data Manager (Django/Postgres/Redis/Celery), Open edX (LMS/CMS/MySQL/Mongo/ES), and the SPAs the preset enables (Auth/LMS/OS on a fresh server) |
| Finalization | `integrations`, `admin_setup`, `data_seeding`, `ibl_tenant_platform` | OAuth/OIDC, edX↔DM sync, super admin, data seeding, optional tenant launch |
| Optional | `smtp_config`, `stripe_config`, `google_sso_config`, `microsoft_sso_config` | Each no-ops unless its trigger key is set |

**Monitoring.** The single-server preset enables Grafana, but setup does not start the monitoring stack (Prometheus, Alertmanager, Grafana and a health monitor). Start it on the server with `ibl utility prometheus start`.

### 4. Non-interactive provision + setup (`.env` file)

Same Terraform + Ansible, zero prompts. **Single-server only.**

```bash
# Provision — AWS
cp .env.provision.example .env && $EDITOR .env
iblai infra provision-env -f .env

# Provision — GCP (sets PROVIDER=gcp)
cp .env.provision.gcp.example .env && $EDITOR .env
iblai infra provision-env -f .env

# Set up the provisioned VM
cp .env.setup.example .env.setup && $EDITOR .env.setup
iblai infra setup-env <project-name> -f .env.setup
```

**Free-standing server** (no Terraform): omit the project name and add `TARGET_HOST`, `SSH_PRIVATE_KEY_PATH`, `BASE_DOMAIN`, `PROJECT_NAME` to `.env.setup`.

**Sample `.env` files** (every key documented inline — copy, edit, run):

| Sample | Used by |
|---|---|
| `.env.provision.example` | `provision-env` — fresh AWS single-server |
| `.env.provision.gcp.example` | `provision-env` — fresh GCP single-server |
| `.env.setup.example` | `setup-env` — platform install on a provisioned or free-standing VM |
| `.env.example` | `launch-env` — one-shot AMI launch (AWS) |

Populated `.env` files are gitignored; secrets are never persisted to `state.json` — they ride `--extra-vars` into Ansible at run time only.

**DM S3 buckets.** With `S3_STATIC_BUCKET` / `S3_MEDIA_BUCKET` set, `setup-env` first checks that static is public-read, media is private, and both have the CORS rules the browser needs (chat attachments upload straight from the browser to media). It stops if not; check or repair any time:

```bash
iblai infra s3 check -f .env.setup              # read-only; the CORS verdict is the browser's own preflight
iblai infra s3 fix -f .env.setup --yes          # adds what is missing, keeps existing rules and statements
iblai infra s3 fix <project-name> --profile admin --extra-origin https://tenant.example.org
```

`setup-env --fix-buckets` repairs and continues; `--skip-bucket-check` skips the check. Repairs need `s3:PutBucketCORS`, `s3:PutBucketPolicy` and `s3:PutBucketPublicAccessBlock`.

### 5. Re-setup an existing environment

```bash
iblai infra resetup <name>
```

Points a running environment at a new domain with fresh secrets: prompts for the new base domain, prod-images release tag, and credentials; rotates all secrets, syncs DB passwords, restarts services. No Terraform changes.

### 6. Verify DNS

```bash
iblai infra dns check <name>            # check once
iblai infra dns check <name> --watch    # re-check until everything resolves
```

Unless the deployment manages DNS itself (Route53 + ACM, or an existing Cloud DNS zone), you create the records — see the list printed after provisioning, also saved as `dns-records.txt` in the workspace. Provisioning offers to run this check straight away, and it can be re-run at any point while you wait for the records to appear.

Per subdomain it reports whether the name resolves and whether it points at **this** deployment's load balancer — a record resolving to some other host is flagged rather than passed. Certificate state is shown alongside, since an ACM or Google-managed certificate cannot validate until the records exist. Lookups go to public resolvers, so a stale local cache can't mask a failure. Exits non-zero while anything is unresolved.

`iblai infra setup <name>` runs the same check first and asks before installing against domains that don't resolve; `--skip-dns-check` bypasses it.

### 7. Optional feature toggles (post-provision)

Everything optional at setup can be added later against a running environment, without re-running setup:

```bash
iblai infra configure <name>       # menu of everything below
```

The environment must already be provisioned and set up. Only the relevant Ansible role re-runs, and it needs just the SSH key already recorded in the project — no GitHub token or AWS keys. Most changes are live immediately; the two exceptions are noted below.

#### Email (SMTP)

```bash
iblai infra smtp enable <name>               # prompts for host, port, credentials, sender
iblai infra smtp enable-env <name> -f .env   # non-interactive (SMTP_* keys)
iblai infra smtp disable <name> [--yes]
iblai infra smtp status <name>               # reads the live values off the server
```

⚠️ The only feature needing a restart — the settings reach the services as environment variables, so Data Manager and Open edX are recreated. The command asks first; `--no-restart` skips it (the change stays inert until they restart), `--yes` skips the prompt.

#### Google SSO

```bash
iblai infra sso google <name>
```

Register an OAuth client in Google Cloud Console first, with authorised redirect URI `https://learn.<your-domain>/auth/complete/google-oauth2/`. Prompts for client ID, secret, and an optional Workspace domain to restrict to. Live immediately.

#### Microsoft SSO

```bash
iblai infra sso microsoft <name> [--yes]
```

Register an application in Azure first, with redirect URI `https://learn.<your-domain>/auth/complete/azuread-oauth2/`. Prompts for client ID, secret and tenant ID.

⚠️ Restarts Open edX — it changes settings read only at boot. The command warns first.

#### Stripe billing

```bash
iblai infra stripe enable <name>
iblai infra stripe enable-env <name> -f .env   # non-interactive (STRIPE_* keys)
```

Prompts for test/live mode, secret and publishable keys, and optional pricing-table and webhook values. Live mode warns that real cards will be charged. Live immediately.

#### LLM API key

```bash
iblai infra llm set-key <name>
iblai infra llm set-key <name> --provider openrouter --api-key <key>   # non-interactive
```

Sets or rotates the credential the mentor service uses. Supported providers are `openai` (the default), `anthropic` and `openrouter`. An OpenAI or Anthropic key becomes the preferred provider, since the platform picks the preferred credential with no tie-break when several are marked. Live immediately.

An **OpenRouter** key is stored as the ibl.ai gateway key (the global credential `iblai`), beside any provider key rather than in place of it; it becomes the preferred one only when no other key is. The model catalogue is then synced so chat routes through the gateway straight away. Setup asks which key to provide (OpenAI or OpenRouter); `.env` files take `OPENAI_API_KEY` or `OPENROUTER_API_KEY`, and `launch` takes `--openai-key` or `--openrouter-key`.

**Why an OpenRouter key.** Mentors created by the platform, the default agent mentor among them, use the gateway models (`iblai-pro` / `iblai-fast`), which only an OpenRouter key serves. With an OpenRouter key chat works with no further setup.

With only an OpenAI key:

- **DM 4.412 and later** serve OpenAI's models with that key, but mentors on a gateway model need a default model to stand in for it. Set one with `ibl config set IBL_DM.DEFAULT_MENTOR_LLM_PROVIDER=openai IBL_DM.DEFAULT_MENTOR_LLM_MODEL=<model>` (ibl-cli-ops 7.24+), then `ibl render` and restart the DM.
- **Earlier DMs** mark most models, OpenAI's included, as served through the gateway at the daily catalogue sync, so chat stops working within a day of setup. Add an OpenRouter key.

#### Tenant platform

```bash
iblai infra platform create <name>
iblai infra platform create <name> --platform-name <tenant>
```

Creates a platform alongside the default `main` one, via the platform launcher so it comes up fully wired. **The tenant admin credentials are printed once and stored nowhere else — save them.**

#### Clone a SPA

```bash
iblai infra spa clone <name>                 # prompts for source, name and domain
iblai infra spa list <name>                  # what's deployed, with ports
iblai infra spa remove <name> --spa <clone>
```

Runs a copy of a deployed SPA alongside the original — its own port, its own domain — so it can be customised without touching the one the platform depends on. The copy starts with the source's **running** environment file, so it behaves identically until you change it:

```bash
# customise, then restart
/ibl/app/ibl-spa/<clone>/.env.<clone>
cd /ibl/app/ibl-spa/<clone> && docker compose up -d
```

Ports are allocated from 5060 (stock SPAs hold 5000–5009). The nginx block lands in `/etc/nginx/conf.d/custom_domains/`, which survives `ibl global-proxy` regeneration. Served over **HTTP** — point the domain at the server and put it behind whatever already terminates TLS. `remove` refuses the platform's own SPAs.

#### WAF (AWS single-server only)

```bash
iblai infra waf enable [<name>]               # WAFv2 on the ALB
iblai infra waf enable-env [<name>] -f .env   # non-interactive (WAF_ALLOWED_IPS)
iblai infra waf disable <name> [--yes]
iblai infra waf status [<name>]
```

The only one of these that changes infrastructure rather than platform config — it re-applies Terraform.

### 8. Launch from AMI (AWS, CI/CD)

One-shot Terraform + Ansible from a pre-built AMI:

```bash
iblai infra launch-env                       # .env-driven (cp .env.example .env first)

iblai infra launch \                         # fully flag-driven
  --ami-id $AMI_ID --domain $DOMAIN --hosted-zone-id $HOSTED_ZONE_ID \
  --aws-key-id $AWS_ACCESS_KEY_ID --aws-secret-key $AWS_SECRET_ACCESS_KEY \
  --ssh-public-key "$SSH_PUBLIC_KEY" --ssh-key $SSH_KEY_PATH \
  --git-token $GIT_TOKEN --vpn-ip $VPN_IP \
  --admin-email $ADMIN_EMAIL --admin-password $ADMIN_PASSWORD
```

See `iblai infra launch --help` for all options (instance type, `--platform-name`, SMTP/Stripe/SSO, `--enable-ai`, ...).

### 9. Service update (image updates, CI/CD)

Update container images and restart services — no provisioning, no secret rotation:

```bash
iblai infra service-update --host <ip> --ssh-key ~/.ssh/key.pem --git-token $GIT_TOKEN
```

Or launch a fresh AMI and swap it into an ALB target group — see `iblai infra service-update --help`.

### 10. Manage environments

```bash
iblai infra list                # all environments (cloud, type, status)
iblai infra status <name>       # details + outputs for one
iblai infra auth                # switch AWS credentials
iblai infra destroy <name>      # tear everything down
```

## Authentication

The CLI never silently auto-detects credentials — you always choose, interactively on first use, and the session is reused across commands.

**AWS** — pick one in the wizard: profiles from `~/.aws/` (type to filter), environment variables, or manual access keys.

**GCP** — two options; the wizard asks which:

*Option A — your Google account (local use):*

```bash
gcloud auth application-default login    # note: `gcloud auth login` alone is NOT enough
```

*Option B — service-account key (CI, or if your org blocks the browser flow above):*

```bash
PROJECT=<your-project-id>
gcloud iam service-accounts create infra-cli --project=$PROJECT
for R in roles/compute.admin roles/dns.admin roles/iam.serviceAccountUser; do
  gcloud projects add-iam-policy-binding $PROJECT \
    --member="serviceAccount:infra-cli@$PROJECT.iam.gserviceaccount.com" --role=$R --condition=None
done
gcloud iam service-accounts keys create ~/infra-cli-key.json \
  --iam-account="infra-cli@$PROJECT.iam.gserviceaccount.com"
```

Then choose **"Service-account key file"** in the wizard (or set `GCP_CREDENTIALS_FILE=~/infra-cli-key.json` in your `.env`). Keep the key file out of git; delete the service account when done.

## What gets created

**AWS (Terraform):** VPC with 2 public subnets · EC2 (Ubuntu 22.04, encrypted EBS) · Application Load Balancer (TLS 1.2/1.3) · ACM certificates (DNS-validated, auto-renewed) · security groups (SSH restricted to your IP) · 3 S3 buckets (backups, media, static) · Route53 A-records for all subdomains · optional WAFv2.

**GCP (Terraform):** VPC + regional subnet · Compute Engine VM (Ubuntu 22.04) · global external Application Load Balancer with a static IP · Google-managed SSL certificate (all subdomains, async validation) · firewall rules (SSH restricted to your IP, health-check ranges) · Cloud DNS A-records. Object storage stays on AWS S3 — details in the [GCP guide](/developer/infra-cli/gcp).

**Subdomains** — each gets a DNS A-record and is covered by the load balancer's TLS certificate. Replace `<your-domain>` with your base domain:

- `learn.<your-domain>`
- `preview.learn.<your-domain>`
- `studio.learn.<your-domain>`
- `apps.learn.<your-domain>`
- `meilisearch.learn.<your-domain>`
- `api.<your-domain>`
- `api.data.<your-domain>`
- `asgi.data.<your-domain>`
- `llm.data.<your-domain>`
- `base.manager.<your-domain>`
- `auth.<your-domain>`
- `os.<your-domain>`
- `lms.<your-domain>`
- `platform.<your-domain>`
- `monitor.<your-domain>`
- `flowise.<your-domain>`
- `prometheus.<your-domain>`

**Platform services (Ansible):** Open edX (LMS/CMS + MySQL/Redis/MongoDB/Elasticsearch/Forum) · Data Manager (Django/ASGI/Celery + PostgreSQL 16/Redis/Flowise) · Auth/Mentor/Skills SPAs · monitoring (Prometheus/Grafana) · nginx reverse proxy.

## Workspace

Terraform state, generated SSH keys, and project configuration live at `~/.iblai-infra/projects/<project-name>/`.

## Development

```bash
uv sync --extra dev --extra gcp
uv run pytest tests/ -v                                          # 1,084 tests
uv run pytest tests/ --cov=iblai_infra --cov-report=term-missing
```

```
iblai-infra-ops/
├── src/iblai_infra/
│   ├── cli.py                  # Typer CLI commands
│   ├── app.py                  # Provision wizard orchestrator
│   ├── models.py               # Pydantic models (the wizard ↔ Terraform contract)
│   ├── env_provision.py        # .env → InfraConfig (provision-env, AWS)
│   ├── gcp_env_provision.py    # .env → InfraConfig (provision-env, GCP)
│   ├── env_setup.py            # .env → SetupConfig  (setup-env)
│   ├── prompts/                # Interactive questionary prompts
│   ├── providers/              # aws.py + gcp.py — cloud SDK helpers
│   ├── features/               # Post-provision toggles (waf.py)
│   ├── terraform/              # Runner + templates
│   │   └── templates/
│   │       ├── aws/            # single-server (+ waf.tf), multi-server, call-server
│   │       └── gcp/            # single-server
│   └── ansible/                # Runner + playbooks + 16 roles
├── docs/                       # GCP guide, development notes
├── tests/                      # 1,084 tests
└── pyproject.toml
```

## License

Released under the [MIT License](https://github.com/iblai/infra-cli/blob/main/LICENSE).
