δΈ­ζ–‡ English

Same Repository, One Node on 9.2 and One Stuck on 9.1: Hunting the Fake Keyring That Silently Broke PVE Updates

Published: 2026-09-13 Β· Views --
ProxmoxVE PVE Troubleshooting Operations Linux Automation Windows 11 macOS Ubuntu 26.04

TL;DR

I run two Proxmox VE 9 nodes configured with exactly the same package repository (the pve-no-subscription mirror). One had already moved on to the latest 9.2.18; the other was stuck at 9.1.9 β€” and had been silently stuck for more than four months, with no error ever surfacing.

The root cause was neither the network nor the repository config: the GPG keyring file /etc/apt/trusted.gpg.d/proxmox-release-trixie.gpg had been created from the wrong download in the first place β€” it contained a “signature block” instead of a public key. Debian 13’s new signature verifier therefore refused to refresh the PVE package index every single time, then quietly kept using a four-month-old index.

The fix takes three steps: back up the broken file β†’ install the correct official key β†’ apt update again. No reinstall, no repository changes, five minutes end to end. This post walks through the full investigation and ships one-click fix scripts for Windows 11 / Ubuntu 26.04 / macOS 26, in both “run it yourself” and “hand it to an AI agent” flavors.

Original cover: a fake keyring locks a whole server out of updates

Figure 1: Original cover art. Same repository, two different version worlds β€” the culprit is a fake key sitting in the keyring.

1. Background: A Mundane “Version Alignment”

My home lab runs two physical PVE nodes hosting around twenty VMs: soft routers, databases, builders, reverse proxies. Both nodes were installed back in the PVE 9.0 days and point at the same pve-no-subscription mirror β€” a standard home-lab setup.

While aligning versions across both nodes, something odd showed up:

My first guess was “the sources must be configured differently” β€” the nodes were installed at different times, after all. That assumption died within minutes, and the real hunt began.

2. Symptoms: One Source, Two Realities

The full picture: same repo, two fates

Figure 2: The full picture. Node B sees 9.2.18; node A’s ceiling is welded shut at 9.1.9.

2.1 A Whole Release Series Apart

Running pveversion on both nodes tells the story at a glance:

pveversion: 9.1.9 vs 9.2.18

Figure 3: Real command output. Node A is on 9.1.9 (kernel 6.17 series); node B is on 9.2.18 (kernel 7.0 series).

2.2 apt Believes 9.1.9 Is the Latest

The decisive output is apt-cache policy pve-manager on node A:

apt-cache policy shows a locked candidate

Figure 4: Real command output (abridged). The Candidate is also 9.1.9 β€” and even 9.1.10 is missing from the version list.

Quick primer on reading this: Installed is what’s on disk, Candidate is what apt believes is available. A candidate of 9.1.9 means apt’s “shopping list” contains no 9.2 entries at all. The problem is not the install step β€” it’s the seeing step: apt has no idea 9.2.18 exists.

The obvious suspects fell one by one:

Suspect Check Verdict
Different sources? diff of sources.list and sources.list.d/ Identical
Mirror out of sync? Same mirror, yet node B sees 9.2.18 Ruled out
Version pinned? No pins in apt-cache policy, clean /etc/apt/preferences Ruled out
Disk full / apt lock? df -h fine, no locks Ruled out

3. Analysis: Timestamps Are the First Crime Scene

With the easy explanations exhausted, I switched angles: if apt can’t see new versions, when did it last refresh its shopping list? The index cache lives in /var/lib/apt/lists/, and file timestamps don’t lie:

Index timestamps: one frozen in April, one fresh

Figure 5: Real command output (filenames abridged). Node A’s PVE index is stuck at 2026-04-25; node B’s was refreshed in September.

First layer of the truth: node A’s PVE package index had not refreshed successfully for over four months. Late April 2026 is exactly when 9.1.9 was current β€” from that moment on, the PVE repository simply ceased to exist for this machine.

Which raises the question: why did nobody notice? Running apt-get update by hand reveals it:

The signature verification failure

Figure 6: Real command output. The Err: points at InRelease verification failing, missing key 24B30F06… β€” yet the whole command exits “successfully” with only W: warnings.

Every line of that error is informative:

The nastiest part is what the footnote says: only W: warnings, no E: errors, cheerful exit code. A cron job running apt update && apt upgrade would report success forever, upgrade nothing, and never make a sound.

3.1 Where Exactly Is the “Missing Key”?

For this node’s repository style, the verification key should live at /etc/apt/trusted.gpg.d/proxmox-release-trixie.gpg. Putting both nodes’ files side by side catches the culprit red-handed:

file(1) comparing both keyring files

Figure 7: Real command output. Node A’s “keyring” is actually a block starting with -----BEGIN PGP SIGNATURE-----; node B’s is a proper OpenPGP public key.

The whole story snaps into focus: when the PVE repository was first configured on node A, the key-download step saved the wrong file β€” the repository’s signature file went into the keyring directory instead of the public key. A signature block is not a key; the verifier cannot parse it, so every refresh reports a missing key. The broken file’s timestamp, April 26, marks the day of that “successful-looking” repository change β€” and the last time this node ever received an update from the PVE repository.

4. Root Cause: One Failed Verification Quietly Locks the Version

Here is the full chain as a diagram β€” a textbook case of silent degradation:

The apt verification flow

Figure 8: apt’s verification journey. With no matching public key in the keyring, sqv refuses the refresh and leaves nothing but a warning.

If GPG and keyrings feel abstract, here’s a restaurant analogy:

The restaurant stamp analogy

Figure 9: The restaurant analogy. Keyring = the book of official stamp samples; a broken keyring = a book with scrap paper inside, so nothing ever matches.

Mapping the metaphor to the mechanics:

  1. The repository publishes an InRelease index on every update β€” the full package list plus the vendor’s GPG signature (“a menu with an anti-forgery stamp”);
  2. The keyring is the local book of official stamp samples; every file under trusted.gpg.d is treated as a sample;
  3. The verifier (sqv) compares the menu’s stamp against the sample book at every refresh and rejects mismatches;
  4. Silent fallback: on rejection, apt neither aborts nor alerts β€” it just keeps serving last month’s menu (the stale index). That is the entire secret of “four months, nobody noticed”.

Root cause in one sentence: node A’s PVE keyring file contained the wrong content (a signature block posing as a key) β†’ every index refresh failed verification β†’ apt ran on April’s stale index β†’ the version ceiling was locked at 9.1.9.

5. The Fix: Swap the Key in Three Steps

The fix is refreshingly plain: put the correct public key back into the keyring, then refresh the index. The only question is where the correct key comes from. Both routes below are field-tested; I recommend them in this order:

The PVE repository ships an official package, proxmox-archive-keyring, containing all currently valid release keys. Package installs only verify checksums against the local index β€” no direct connectivity to the vendor’s overseas servers required, which makes this route ideal for networks where download.proxmox.com is hard to reach:

The missing key lives in the official keyring package

Figure 10: Real command output. The keyring shipped inside the official package contains exactly the fingerprint the error message asked for: 24B30F06….

apt-get install --reinstall -y proxmox-archive-keyring
install -m 0644 /usr/share/keyrings/proxmox-archive-keyring.gpg \
                /etc/apt/trusted.gpg.d/proxmox-release-trixie.gpg
apt-get update

5.2 Route 2 (Fallback): Copy from a Healthy Sibling Node

If your LAN has a healthy node with the same repository layout (mine happened to), copying its keyring file across is equally valid β€” same repository, same public key:

The full repair session

Figure 11: The actual repair. Back up the broken file β†’ copy the good key from the healthy node β†’ refresh the index β†’ the candidate jumps from 9.1.9 straight to 9.2.18.

Notes that apply to both routes:

By the way, the Proxmox wiki now recommends the deb822-style .sources files plus /usr/share/keyrings/proxmox-archive-keyring.gpg (referenced explicitly via Signed-By:), which is tidier than the global trusted.gpg.d keyring. I kept the minimal-change repair here and left the repository migration as a follow-up.

6. One-Click Fix Scripts

Too lazy to type? The logic is packaged into one-click scripts for three platforms. All of them use only SSH and the official repository β€” no third-party services β€” and all follow the same “check β†’ back up β†’ fix β†’ verify” flow, automatically skipping nodes whose keyring is already healthy:

The script’s four-step logic

Figure 12: Script flow. Check first, back up second, fix third, verify last β€” idempotent and safe to re-run.

Method 1: Run It Yourself

Ubuntu 26.04 / on the PVE node itself (fix-pve-keyring.sh) β€” runs locally on a node, or repairs a remote one when given a root@<node> argument:

#!/usr/bin/env bash
# fix-pve-keyring.sh β€” repair the version ceiling caused by PVE keyring failure
# Local:   sudo bash fix-pve-keyring.sh
# Remote:  bash fix-pve-keyring.sh root@<target-node>
set -euo pipefail

PAYLOAD=$(cat <<'EOS'
set -euo pipefail
KEY="/etc/apt/trusted.gpg.d/proxmox-release-trixie.gpg"
PKG="/usr/share/keyrings/proxmox-archive-keyring.gpg"

echo "== [1/5] Version before fix =="
pveversion || true
apt-cache policy pve-manager | sed -n '1,4p'

echo "== [2/5] Keyring health check =="
if file "$KEY" 2>/dev/null | grep -q "OpenPGP Public Key"; then
  echo "OK: keyring format is healthy. If updates still fail, investigate elsewhere."
else
  echo "BAD: keyring is not a valid public key file; repair needed."
fi

echo "== [3/5] Back up the original file =="
if [ -e "$KEY" ]; then
  cp -a "$KEY" "$KEY.bak.$(date +%Y%m%d%H%M%S)"
  echo "Backed up as $KEY.bak.*"
fi

echo "== [4/5] Restore the key from the official package =="
apt-get install --reinstall -y proxmox-archive-keyring \
  || echo "Reinstall failed; falling back to the local copy of the official key"
if [ ! -s "$PKG" ]; then
  echo "Official key file $PKG not found; see the manual route."; exit 1
fi
install -m 0644 "$PKG" "$KEY"

echo "== [5/5] Refresh the index and verify =="
apt-get update
echo "== Version after fix (Candidate should be the latest) =="
apt-cache policy pve-manager | sed -n '1,4p'
EOS
)

if [ "${1:-}" != "" ]; then
  ssh "$1" 'bash -s' <<<"$PAYLOAD"
else
  bash -c "$PAYLOAD"
fi

Windows 11 (fix-pve-keyring.ps1) β€” uses the built-in OpenSSH client; run .\fix-pve-keyring.ps1 -Target "root@<node>" in PowerShell:

# fix-pve-keyring.ps1 β€” one-click remote PVE keyring repair from Windows 11
param(
  [Parameter(Mandatory = $true)][string]$Target,  # e.g. root@<target-node>
  [int]$Port = 22
)

$payload = @'
set -euo pipefail
KEY="/etc/apt/trusted.gpg.d/proxmox-release-trixie.gpg"
PKG="/usr/share/keyrings/proxmox-archive-keyring.gpg"
echo "== [1/5] Version before fix =="
pveversion || true
apt-cache policy pve-manager | sed -n '1,4p'
echo "== [2/5] Keyring health check =="
file "$KEY" 2>/dev/null | grep -q "OpenPGP Public Key" \
  && echo "OK: keyring healthy" || echo "BAD: keyring broken, repairing"
echo "== [3/5] Back up the original file =="
[ -e "$KEY" ] && cp -a "$KEY" "$KEY.bak.$(date +%Y%m%d%H%M%S)"
echo "== [4/5] Restore the key from the official package =="
apt-get install --reinstall -y proxmox-archive-keyring \
  || echo "Reinstall failed; falling back to the local copy"
[ -s "$PKG" ] || { echo "$PKG not found"; exit 1; }
install -m 0644 "$PKG" "$KEY"
echo "== [5/5] Refresh the index and verify =="
apt-get update
apt-cache policy pve-manager | sed -n '1,4p'
'@

$payload | ssh -p $Port $Target "bash -s"
if ($LASTEXITCODE -eq 0) {
  Write-Host "Fix flow finished; confirm the Candidate above is current." -ForegroundColor Green
} else {
  Write-Host "Something failed; check the SSH connection and output above." -ForegroundColor Red
}

macOS 26 (fix-pve-keyring-macos.sh) β€” run bash fix-pve-keyring-macos.sh root@<node> in Terminal:

#!/usr/bin/env zsh
# fix-pve-keyring-macos.sh β€” one-click remote PVE keyring repair from macOS
set -euo pipefail
TARGET="${1:?[usage] bash fix-pve-keyring-macos.sh root@<target-node>}"

ssh "$TARGET" 'bash -s' <<'EOS'
set -euo pipefail
KEY="/etc/apt/trusted.gpg.d/proxmox-release-trixie.gpg"
PKG="/usr/share/keyrings/proxmox-archive-keyring.gpg"
echo "== [1/5] Version before fix =="
pveversion || true
apt-cache policy pve-manager | sed -n '1,4p'
echo "== [2/5] Keyring health check =="
file "$KEY" 2>/dev/null | grep -q "OpenPGP Public Key" \
  && echo "OK: keyring healthy" || echo "BAD: keyring broken, repairing"
echo "== [3/5] Back up the original file =="
[ -e "$KEY" ] && cp -a "$KEY" "$KEY.bak.$(date +%Y%m%d%H%M%S)"
echo "== [4/5] Restore the key from the official package =="
apt-get install --reinstall -y proxmox-archive-keyring \
  || echo "Reinstall failed; falling back to the local copy"
[ -s "$PKG" ] || { echo "$PKG not found"; exit 1; }
install -m 0644 "$PKG" "$KEY"
echo "== [5/5] Refresh the index and verify =="
apt-get update
apt-cache policy pve-manager | sed -n '1,4p'
EOS

On any platform, the fix is confirmed once the final Candidate jumps to 9.2.x (or whatever is current when you read this).

Method 2: Hand It to an Agent

Even easier: give the whole problem to an AI coding agent (ZCode, Claude Code, etc.) along with SSH access. This prompt is ready to paste:

Please fix a Proxmox VE server that cannot upgrade to the latest release. The likely
root cause is a broken keyring file /etc/apt/trusted.gpg.d/proxmox-release-trixie.gpg
(e.g. it contains a PGP signature block instead of a public key), which makes apt
signature verification fail and keeps the package index stale. Steps:

1. SSH into the node (connection details are in the session context / provided separately);
2. Record the current state: run pveversion and apt-cache policy pve-manager;
3. Inspect the keyring file with `file`; if it is not "OpenPGP Public Key",
   back it up with cp -a under /root/ first;
4. Run apt-get install --reinstall -y proxmox-archive-keyring, then copy
   /usr/share/keyrings/proxmox-archive-keyring.gpg over the keyring path above
   with mode 0644; if the reinstall fails but the local file exists, copy it directly;
5. Run apt-get update and confirm there are no Err:, Missing key, or
   unsupported filetype messages;
6. Report apt-cache policy pve-manager's Candidate as evidence of the fix.

Constraints: do NOT run dist-upgrade/full-upgrade, do NOT reboot the server,
and do NOT modify repository definitions.

After the agent finishes, verify its two pieces of evidence: no Err: lines in apt-get update, and a Candidate at the current release.

7. Lessons So It Never Happens Again

The five-item prevention checklist

Figure 13: The prevention checklist. The core idea: never equate “no errors” with “no problems”.

The most valuable lesson here is not about GPG itself β€” it’s about recognizing silent degradation as a failure mode:

  1. Run the three-step health check before any upgrade: scan apt-get update output for Err:/W:; check the candidate with apt-cache policy <pkg>; check index freshness with ls -l /var/lib/apt/lists/. Thirty seconds total, defeats 90% of “fake latest”.
  2. Never trust exit codes alone in automation: apt-get update exits successfully even when repository verification fails. Assert on output instead, e.g. apt-get update 2>&1 | grep -E '^(Err|W):' && exit 1.
  3. Source key material only from official channels: reinstalling the proxmox-archive-keyring package is the most reliable route; if you must download a key by hand, always confirm with file that you saved an OpenPGP Public Key β€” this very incident started with saving the wrong file.
  4. Let sibling nodes cross-check each other: when cluster members drift apart in versions or behavior, diff their repository configs and keyring directories first β€” the answer usually jumps out immediately.

8. Q&A

Q1: Why did the web UI look perfectly healthy while the problem sat there for four months?

A: On one hand, apt-get update only prints W: warnings when verification fails and still exits successfully, so cron-driven auto-updates never notice. On the other hand, the PVE web UI’s “Node β†’ Updates β†’ Repositories” panel does have a repository health check that flags signature problems β€” but most people (myself included, back then) rarely open that page. Add it to your routine inspection list.

Q2: Wouldn’t [trusted=yes] on the repository just make this go away?

A: That removes the stamp check entirely β€” whatever a man-in-the-middle injects gets installed. Never do this on production machines.

Q3: Would switching to a different mirror fix it?

A: No. The keyring is broken locally; every mirror on earth will fail verification the same way. Two nodes on the same mirror β€” one healthy, one stuck β€” is the proof.

Q4: Does 9.1 β†’ 9.2 need the official upgrade walkthrough?

A: No. Minor-version updates within the same major release are a plain apt full-upgrade. Only major upgrades (say 8 β†’ 9) require the official step-by-step procedure.

Q5: Should I upgrade immediately after the fix?

A: Technically you can, but it’s wiser to back up or snapshot important guests, check cluster health, and pick a maintenance window for apt full-upgrade; a reboot activates the new kernel. My node A hosts soft routers and databases, so the upgrade waited for a night window.

Q6: Does Debian 12 / PVE 8 have the same failure mode?

A: Yes, with different cosmetics. The older gpgv verifier reports NO_PUBKEY, but the mechanism is identical: broken keyring β†’ stale index β†’ locked version. The same repair philosophy (restore the official key, refresh the index) applies; just use the bookworm keyring filenames.

References

Comments

Sign in with GitHub to comment. Chinese and English versions share the same discussion. Discuss on GitHub

Views --