A Proxmox Container Born, Configured, and Destroyed From Code

By the end of part three this series had a complete written description of my cluster: every container definition under OpenTofu, every node role under Ansible, and two commands that continuously verify reality still matches the record. It sounds finished. It has a hole in it you could drive a truck through.

Nothing in that description has ever built anything. The containers were imported, not created; the roles were captured from configured nodes, not converged onto blank ones. A description that has never been used to build something is a hypothesis — and the whole promise of the series finale, rebuilding from zero, rests on it. So before trusting it with a disaster, I made it prove itself on something disposable: one small container, created by code, configured by code, verified, destroyed by code — while production didn’t feel a thing.

The shape of the proof

The loop has six stations, and the last one is the point: after the scratch container has lived and died, the production plan must still say No changes.

There’s a copy-pasteable version of the whole sequence in the scratch container lifecycle playbook — the prose below is the why behind each station.

Station 0: the write-credential problem

Here’s the first genuinely interesting obstacle. Part one put the whole safety posture on a read-only API token — the tool physically cannot modify the cluster. But a lifecycle test must create and destroy. Something has to hold write access, and the naive answer — just grant the existing token more permissions for an afternoon — turns out not to work, for a reason worth understanding.

Proxmox API tokens default to what it calls privilege separation. A separated token doesn’t inherit its user’s permissions; in the documentation’s words, its “effective permissions are calculated by intersecting user and token permissions,” and “privilege separated tokens can never have permissions on any given path that their associated user does not have.” Read that second sentence again: it’s an intersection, not a union.

I’d designed my automation user as a pure auditor on purpose, and this rule means that purity is enforced: no token minted under it can ever exceed read-only, no matter what ACLs the token gets. Which is the platform telling me the right answer. Don’t corrupt the auditor — mint a separate, disposable user for the test:

# a user that exists only for this test
pveum user add scratch@pve

# write access ONLY where the test lives: one VMID, its storage,
# its network zone, its node — plus read access to see the cluster
pveum acl modify /vms/9001 --users scratch@pve --roles PVEAdmin
pveum acl modify /storage/local-lvm --users scratch@pve --roles PVEAdmin
pveum acl modify /nodes/pvelab01 --users scratch@pve --roles PVEAdmin
pveum acl modify / --users scratch@pve --roles PVEAuditor

# its token — the secret is shown once; it lives in your shell only
pveum user token add scratch@pve tf --privsep 0

The blast radius of the entire experiment is now: container 9001, on one storage, on one node. The account holding it will be deleted within the hour. My real automation credential never changed at all — and that’s the pattern I’d recommend even if the intersection rule didn’t force it, because “temporarily upgraded and definitely remembered to downgrade later” is not a sentence with a good track record in a homelab.

Station 1: birth

The scratch container gets its own configuration directory with its own state — deliberately separate from production’s. State is the tool’s memory of what it manages; giving the throwaway experiment a separate memory means no command run here, however clumsy, can even see a production resource.

resource "proxmox_virtual_environment_container" "scratch" {
node_name    = "pvelab01"
vm_id        = 9001
description  = "scratch guest - safe to destroy"
unprivileged = true
started      = true

operating_system {
  template_file_id = "local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst"
  type             = "debian"
}

cpu    { cores = 1 }
memory { dedicated = 512 }
disk   { datastore_id = "local-lvm" size = 4 }

network_interface {
  name   = "eth0"
  bridge = "vmbr0"
}

initialization {
  hostname = "iac-scratch"
  ip_config {
    ipv4 { address = "dhcp" }
  }
  user_account {
    keys = [trimspace(file(pathexpand("~/.ssh/id_ed25519.pub")))]
  }
}
}

Two choices worth a sentence each. DHCP is deliberate: my gateway only routes clients it handed a lease to, so a static-IP scratch guest would sit there with no internet and fail its first apt update mysteriously — on my network, dhcp is load-bearing, not lazy. The SSH key injection is what lets Ansible in later without any password ceremony.

cd terraform/scratch
tofu apply

# Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

That one line — 1 added — is the first time in this entire series the code has created something. The hypothesis is now an experiment.

Station 2: the DHCP wait, done properly

A freshly booted container can exist before its network does — ask for its IP too early and there’s simply no lease yet. The tempting fix is sleep 10, which is how you end up with a script that works except when it doesn’t. The clean fix is a command that blocks until the condition is true: run the interface bring-up inside the container and let it return when the lease exists.

# on the node: blocks until eth0 is up with its lease
pct exec 9001 -- ifup eth0

# now this reliably has an answer
pct exec 9001 -- ip -4 addr show eth0

Station 3: a real converge, at last

Part three left an honest debt: the node roles had only ever been check-run against machines that were already configured. This is the repayment. The scratch playbook is small but each task is chosen to prove something, not just to do something: refresh the package cache (proves DNS, routing, and egress all work), install a package (proves the whole apt path), and write a marker file (proves file convergence).

ansible-playbook scratch.yml -i "10.0.0.201,"

# PLAY RECAP
# 10.0.0.201 : ok=3  changed=3  failed=0

The trailing comma is doing quiet work: it tells Ansible the -i value is a literal list of hosts rather than the path to an inventory file. And this recap line is the mirror image of part three’s — there, changed=0 was the goal, because the code was asserting an existing reality. Here changed=3 is the goal, because the code just made reality on a machine that had none. Same roles-and-tasks machinery, both directions proven.

I then did the thing you should always do after automation reports success: logged in over SSH and looked with my own eyes. The package was there; the marker file said what the code said it would say.

Stations 4 and 5: death and cleanup

The destroy is the station people flinch at, so it’s worth being precise about what the command can reach. OpenTofu’s documentation describes tofu destroy as the way “to destroy all remote objects managed by a particular OpenTofu configuration” — this configuration, this state. Our state contains exactly one object. Production lives in a different directory, in a different state, where every resource additionally carries prevent_destroy and the standing credential can’t write anyway. Three independent walls.

tofu destroy
# Destroy complete! Resources: 1 destroyed.

# trust, then verify against the platform itself:
pct list        # 9001 is gone

Then the identity follows its container into oblivion, and the last footprints get swept:

# deleting the user drops its ACLs and its token with it
pveum user delete scratch@pve
pveum acl list     # confirm: no scratch entries remain

# the scratch guest's host key is stale the moment it died
ssh-keygen -R 10.0.0.201

The known_hosts line is small but real hygiene: the next guest to get that DHCP address will present a different host key, and future-you doesn’t need a scary key-mismatch warning caused by a container that no longer exists.

Final station: tofu plan in the production directory. No changes. The entire lifecycle happened beside production, not near it.

What this proved — and the one thing it didn’t

The description can build. Create path: proven. Configure path on a blank machine: proven. Destroy path: proven, with the blast walls holding. The write-access pattern — disposable user, scoped ACLs, shell-only secret, delete after — is now a tested recipe rather than a theory, and it’s the same recipe I’ll reach for any time code needs temporary hands.

What it didn’t prove: scale and composition. One 512 MB container is not nine production guests with their real contents, and a lifecycle test says nothing about restoring the data inside those guests — that’s backup territory, and it always was. The finale composes everything this series has built — the imported definitions, the captured roles, the proven loop, and the backups — into the scenario that justifies all of it: the cluster is gone; rebuild it from zero.

Related posts:

• Turn a Hand-Built Proxmox Cluster Into Code Without Breaking It — part one: the read-only posture this test had to deliberately (and temporarily) step around
• Import a Live Proxmox Cluster Into OpenTofu With Zero Changes — part two: where production’s separate state and prevent_destroy walls were built
• Ansible for a Cluster You Built by Hand: Capture, Then Assert — part three: the roles whose create-path debt this post repays
• Rebuild Your Homelab From Zero: What Survives a Total Loss — part five: the disaster this rehearsal exists for
• Creating Your First Proxmox LXC Container: Step-by-Step — the same container birth, done by hand so you know what the code is doing
• Proxmox Security Hardening: SSH Keys, Firewall, and Locking Down a New Cluster — the wider access model the scoped-user pattern belongs to
• Proxmox Backup Server: Automated CT and VM Backups with Deduplication — what protects the part of production no lifecycle test can: the data