🛰️ Flux Cluster Orchestration

This repository is the Central Orchestration Hub for Jesse's homelab and engineering cluster. Built on k0s and managed via Flux CD, it acts as the "Glue" that links the cluster to various independent service repositories. It uses an "Operator + Instances" pattern where infrastructure operators provide capabilities and applications consume them through declarative manifests.

Config that lives in external systems — the identity provider, the DNS zone, the git forge — is managed with the same discipline via OpenTofu (tofu/, reconciled by tofu-controller). See docs/TERRAFORM.md. Day-to-day commands live in docs/OPERATIONS.md.

🗂️ Repository Structure

This repository uses a "Declarations at Root" pattern optimized for operator-managed infrastructure. Instead of base/overlay abstraction, we declare concrete instances directly across a two-layer bootstrap to ensure API compatibility.

.
├── ansible/                 # THE PROVISIONER (Gentoo-aware lifecycle)
│   ├── inventory/           # Node definitions and group variables
│   ├── local.yml            # Main playbook entry point
│   ├── playbooks/           # Targeted automation (bootstrap, ssh, uninstall)
│   └── roles/               # Modular logic (00-system, 10-wireguard, 20-k0s, 30-flux)
│
├── clusters/                   # THE ORCHESTRATION (Flux Engine)
│   └── production/             # The Flux root (The "Basement")
│       ├── flux-system/        # Flux engine manifests
│       ├── infrastructure.yaml # Reconciles ../../infrastructure (Layered DependsOn)
│       └── services.yaml       # Reconciles ../../services (DependsOn: infrastructure)
│
├── infrastructure/          # THE PLATFORM (Operators + Global Config)
│   ├── operators/           # LAYER 1: The "Capabilities" (Helm + CRDs)
│   │   ├── namespaces/      # Cluster-wide namespace definitions
│   │   ├── networking/      # Cilium & Traefik (repo.yaml/release.yaml)
│   │   ├── security/        # cert-manager Operator
│   │   ├── storage/         # CNPG & SeaweedFS Operators
│   │   └── management/      # Capacitor (Flux UI)
│   │
│   └── instances/           # LAYER 2: The "Implementation" (Custom Resources)
│       ├── networking/      
│       │   ├── cilium/      # L2-Policies, IP-Pools
│       │   └── coredns/     # ConfigMaps/Deployments
│       ├── security/        
│       │   └── cert-manager/ # Cloudflare Secrets, ClusterIssuers
│       ├── storage/         
│       │   ├── cnpg-operator/  # Cluster CRs, Database instances, barman backups
│       │   ├── local-path/     # StorageClasses
│       │   ├── manual-volumes/ # Static PV Tiers (Hot/Cold/Cloud)
│       │   └── seaweedfs/      # S3 Cluster instances
│       └── management/       # tofu-controller + sys-tf-* Terraform CRs
│
├── services/                # THE WORKLOADS (Applications consuming infrastructure)
│   ├── auth/                # authentik (identity provider)
│   ├── observability/       # kube-prometheus-stack, loki, alloy, tempo, gatus
│   ├── coms/                # gotify, apprise
│   ├── security/            # crowdsec, vaultwarden
│   ├── dev/                 # forgejo, woodpecker
│   ├── home/                # homarr
│   ├── ai/ data/ media/     # (declared, not yet populated)
│   └── kustomization.yaml   # Aggregator (each category explicitly listed)
│
├── tofu/                    # THE EXTERNAL CONFIG (OpenTofu via tofu-controller)
│   ├── authentik/           # OIDC/proxy apps, flows, groups, branding
│   ├── cloudflare/          # Public + email DNS records
│   ├── forgejo/             # OAuth apps, orgs, teams
│   └── grafana/             # Datasources / dashboards   -> see docs/TERRAFORM.md
│
├── docs/                    # Extended docs (OPERATIONS.md, TERRAFORM.md)
│
└── environment/             # THE WORKSHOP (Pre-GitOps experimentation)
    ├── dev/                 # Rapid iteration - breaking changes expected
    ├── stage/               # Pre-release validation - stability testing
    └── prod/                # Promoted workloads (use services/ instead)

🗄️ Infrastructure Components

Networking

• Cilium: CNI + BGP L2 announcements + Hubble observability
• Traefik: Ingress controller with CrowdSec integration
• CoreDNS: LAN DNS for *.bvoy.dev wildcard resolution

Storage

• CNPG: PostgreSQL operator with declarative database provisioning
• local-path-provisioner: Dynamic local storage on NVMe
• manual-volumes: Static PVs for tiered storage (hot/cold/cloud)
• SeaweedFS: S3-compatible distributed object storage

Security

• cert-manager: Automated TLS via Let's Encrypt + Cloudflare DNS-01
• CrowdSec: behavioural detection + firewall bouncer (in-cluster engine feeds the edge VPS)

Identity & Access

• authentik: OIDC / forward-auth identity provider (auth.bvoy.dev). Apps, flows, groups, and branding are managed as code in tofu/authentik/; client secrets are written back into app namespaces (oidc_secrets.tf).

Observability

• kube-prometheus-stack: Prometheus + Grafana + Alertmanager (metrics)
• Loki + Alloy: log aggregation (Alloy tails pod logs → Loki)
• Tempo: traces, with span-metrics/service-graph via the metrics generator
• Gatus: black-box uptime, alerting through Apprise → Gotify

🧭 Design Principles

1. Platform as Product

Infrastructure is not just "there"; it's a set of APIs. Networking, storage, and security are provided by operators. Applications consume these via native K8s objects (Ingress, PVC, Database CRs) rather than bespoke logic.

2. Declarative Database Provisioning

We use a "Database as Code" model via CNPG:

• App manifests declare a kind: Database with their schema needs.
• The platform automatically handles user creation, extensions, and password rotations.
• App teams never touch the database control plane or master credentials.

3. Environment Promotion Over Duplication

Workloads do not exist in multiple environments simultaneously to avoid configuration drift:

• environment/dev/: Rapid, manual experimentation.
• environment/stage/: Pre-release validation.
• services/: Production workloads with full GitOps lifecycle.

4. Explicit Over Implicit

Every resource is explicitly listed in a kustomization.yaml. We do not use automatic directory scanning (globbing). This ensures that a file's existence in the repo doesn't cause a deployment unless it is intentionally "hooked" into the orchestration tree.

5. Secrets Travel With Resources

Security is localized. To ensure atomicity and ease of deletion, SOPS-encrypted secrets live in the same directory as the resource that consumes them. No centralized "Secret Vault" is required for local cluster operations.

🏗️ The Orchestration Model

This repository follows a "Layered Operator + Instances" design. This ensures that the platform capabilities (the Operators) are fully established before the cluster attempts to declare the resources that depend on them.

The Two-Layer Pattern

Infrastructure is split into two distinct reconciliation phases:

1. Layer 1: The Operators (operators/)

   • Source: repo.yaml (HelmRepository) defines where the operator comes from.
   • Logic: release.yaml (HelmRelease) installs the operator and registers the Custom Resource Definitions (CRDs).
   • Purpose: Establishes the cluster's "vocabulary" (e.g., teaching K8s what a Database or IngressRoute is).

2. Layer 2: The Instances (instances/)

   • Declaration: Concrete Custom Resources (Database, IngressRoute, Certificate).
   • Method: Managed strictly via Native Kustomize and Flux Kustomizations.
   • Purpose: Implements the platform features using the vocabulary provided by Layer 1.

Kustomize-First Philosophy

We prefer Kustomize for all instance management because:

• Native SOPS Integration – Secrets live directly alongside the resources they protect.
• API Transparency – flux diff shows exactly what K8s sees, without hidden Helm template logic.
• Operator-Native – Modern operators are designed to reconcile plain YAML Custom Resources, making values.yaml abstractions redundant.

Example - CNPG Database Instance: Instead of a complex Helm chart, we declare the intent directly in instances/storage/cnpg-operator/databases/:

# databases/immich.sops.yaml (plain YAML, encrypted with SOPS)
---
apiVersion: v1
kind: Secret
metadata:
  name: immich-db-secret
  namespace: sys-postgres-ns
type: kubernetes.io/basic-auth
stringData:
  username: immich
  password: <encrypted>
---
apiVersion: postgresql.cnpg.io/v1
kind: Database
metadata:
  name: immich-db
  namespace: sys-postgres-ns
spec:
  cluster: { name: main-postgres }
  name: immich
  owner: immich

Helm Only When Necessary

We tactically use Helm within the operators/ layer for:

1. Operator Lifecycle – Most operators are distributed as Helm charts to handle complex RBAC and CRD injections efficiently.
2. Vendored Dependencies – Third-party services that do not yet support a native CRD-based management model.

The Rule: If it's a capability provider (Operator), use Helm. If it's a consumer of those capabilities (Instance), use Kustomize.

Why No base/ Directories?

Operators ARE the base. In this model, the "Base" is the API provided by the operator. When you install CNPG, you don't patch a template—you declare a kind: Database. This:

• Eliminates mental overhead: The file in instances is exactly what is running.
• Enables true pruning: Deleting a file in Git explicitly removes the resource from the cluster.
• Simplifies SOPS: No need to manage complex overlays just to inject a secret.

Service Lifecycle

1. Experiment – New workloads start in environment/dev/ as throwaway manifests.
2. Validate – Promising services move to environment/stage/ for integration testing.
3. Graduate – Stable services promote to services/ with a dedicated GitRepository and Kustomization.
4. Consume – Services reference infrastructure (StorageClasses, Ingresses, Databases) via standard Kubernetes lookups.

Decoupled Management

• Upgrading Infrastructure: Update the version in 00-operators/path/release.yaml. Flux upgrades the operator; the instances remain untouched.
• Scaling Workloads: Modify the instance in instances/. Flux applies the change only after validating the CRD exists.
• Rolling Back: Revert the Git commit. Flux prunes or rolls back to the previous state automatically.

🔀 Kustomize Aggregation Logic

Unlike standard file-crawling tools, Kustomize does not recurse directories automatically. This repository uses explicit aggregation:

• Explicit Mapping: Each functional directory contains a kustomization.yaml listing its specific sub-directories. This prevents "accidental deployments" of half-finished or experimental manifests.
• Single Source of Truth: Infrastructure and services are defined once, not duplicated across environments. The clusters/production/ entry point determines which components are active.

🏘️ Namespace Taxonomy

To keep kubectl get ns scannable, we use a consistent prefix system:

Prefix	Role	Purpose	Example
sys-	System	Platform operators and infrastructure controllers	sys-ingress-ns
lab-	Laboratory	Stable self-hosted services with SLAs	lab-nextcloud-ns
env-	Environment	Experimental workloads, may be destroyed	env-dev-api-ns
none	Operators	Third-party operator namespaces (e.g. cert-manager)	cnpg-system

🏷️ Naming Convention

Pattern: <namespace-prefix>-<component>-<resource-type>

Resources use consistent suffixes to instantly communicate their purpose:

Resource Type	Suffix	Example Metadata Name
Namespace	-ns	lab-immich-ns
GitRepository	-repo	lab-immich-repo
HelmRepository	-repo	traefik-repo
Kustomization	-ks	lab-immich-ks
HelmRelease	-release	traefik-release
Database	N/A	immich (Database CR)
Cluster	N/A	main-postgres (CNPG)

Why this matters: When troubleshooting with kubectl, names instantly reveal whether you're touching core infrastructure (sys-), a production service (lab-), or a dev experiment (env-).

🔐 Secrets Management (SOPS + Age)

Architecture:

• At Rest: All sensitive manifests are encrypted in Git using SOPS + Age (.sops.yaml extension)
• In Transit: Flux decrypts manifests in-memory during reconciliation using a private key stored in flux-system/sops-age
• In Use: Decrypted secrets are stored as native Kubernetes Secrets and consumed by workloads

What Gets Encrypted:

• Database credentials (databases/*.sops.yaml)
• API tokens (cloudflare-api-secret.sops.yaml)
• SSH keys (inventory/group_vars/all/sshkeys.yml)
• Age decryption keys (inventory/group_vars/all/agekeys.yml)

Key Insight: Secrets live alongside the resources they configure, not in a separate vault. The databases/immich.sops.yaml file contains both the Database CR definition and the root password.

💾 Storage Strategy

We use tiered storage to optimize cost vs. performance:

Tier	Use Case	Technology
Hot	Databases, active data	NVMe/SSD with local-path
Cold	Archives, backups	HDD with manual PVs
Cloud	Off-site replication	S3-compatible (SeaweedFS)

Applications declare storage requirements via StorageClass labels. The platform handles placement automatically based on the tier.

🛡️ Dependency Management

We use Flux's dependsOn to enforce layered bootstrapping:

1. Platform Foundation - The operators Kustomization installs all Helm charts and CRDs.
2. Platform Implementation - The instances Kustomization waits for the operators to be healthy before applying CRs.
3. Services Third - Applications wait for the entire infrastructure layer to be ready.

Example:

# clusters/production/infrastructure.yaml
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: sys-infra-instances-ks
spec:
  dependsOn:
    - name: sys-infra-operators-ks  # Instances wait for CRD registration
---
# clusters/production/services.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: lab-services-ks
spec:
  dependsOn:
    - name: sys-infra-instances-ks  # Services wait for databases/issuers to exist

This prevents dry-run failures where the Kube API would otherwise reject unknown resource types.

🌐 Network Architecture

This repository follows a Strict Hierarchical IP Schema where the network logic is mapped 1:1 to physical and virtual boundaries. The second and third octets define the Location, VLAN, and Function.

📍 Master Hierarchy (10.X.Y.Z)

• 10.1.Y.Z: Site A (Home Lab)
• 10.2.Y.Z: Site B (Cloud / VPS Gateway)
• 10.10.Y.Z: Virtual Overlay (Pods)
• 10.11.Y.Z: Virtual Overlay (Services)

🏗️ Home Site Breakdown (10.1.VLAN.ID)

VLAN	Subnet	Designation	Purpose
10	10.1.10.0/24	MGMT	Infrastructure mgmt (OpenWrt, Switches, IPMI)
20	10.1.20.0/24	SRV-K8S	Host IPs for K8s Nodes (The "Steel")
30	10.1.30.0/24	SRV-DATA	Storage Backend (NAS, SeaweedFS Replication)
40	10.1.40.0/24	TRUSTED	Personal devices & Management CLI
50	10.1.50.0/24	IOT	Untrusted Smart Home devices
99	10.1.99.0/24	DMZ	Public-facing LoadBalancers (Ingress)

☁️ Cluster Overlay (Cilium)

The cluster overlay is intentionally separated to avoid any conflict with physical site routing.

• Pod Subnet: 10.10.0.0/16
  • Carved into /24 per-node by Cilium.
• Service Subnet: 10.11.0.0/16
  • Virtual IPs for internal load balancing.

🛡️ Connectivity Flow

1. OpenWrt acts as the Core Router, handling Inter-VLAN routing and WireGuard encapsulation.
2. VPS Gateway (10.2.0.1) routes all 10.1.0.0/16 and 10.10.0.0/15 traffic through the tunnel (vpn_us_sea).
3. Cilium handles L2 Announcements on VLAN 99 for Ingress traffic, ensuring external requests land directly on the nodes.

🛠️ Operations

Day-to-day commands (provisioning, Flux, tofu-controller, SOPS, CNPG, deploying an SSO app, debugging) live in docs/OPERATIONS.md.

Quick reference — the reconcile tree:

flux reconcile kustomization sys-infra-operators-ks --with-source   # operators (CRDs)
flux reconcile kustomization sys-infra-instances-ks --with-source   # CRs + tofu-controller
flux reconcile kustomization lab-services-ks        --with-source   # apps
flux get helmreleases -A ; flux get all -A --status-selector ready=false

🔑 Security Note

This cluster uses an automated SSH key exchange (managed via 99-sshkeygen.yml). The unique public key generated by Flux during bootstrap must be added as a Deploy Key with Write Access in the Git provider settings to enable GitOps synchronization.

Why write access? Flux can commit back to the repository during automated updates (e.g., image automation). If you don't use this feature, read-only access is sufficient.

---

🌟 The North Star Vision

This repository embodies a "Platform as Product" philosophy:

• App teams interact with high-level APIs (Database CRs, Ingress annotations, StorageClasses)
• The platform handles complexity (TLS certificate provisioning, database user creation, backup scheduling)
• GitOps ensures every change is auditable, reversible, and declarative

The goal: A developer should deploy a new service by creating three files:

1. A GitRepository pointing to their application manifests
2. A Kustomization to sync it to the cluster
3. A Database CR to provision their schema (if needed)

Everything else—networking, secrets, monitoring, storage—should be automatic.

The philosophy: Kustomize for transparency, Operators for capabilities, Flux for orchestration. Simple primitives, powerful composition.

---

"Operators provide capabilities. Applications declare requirements. Flux ensures reality matches intent."