Brighid FS corruption 2026-09-19
Brighid filesystem corruption incident#
Date: September 19, 2026. Times below are PDT (UTC−07:00), unless noted.
Summary#
Brighid, an Arch Linux VM on the Proxmox host xeno, failed to boot after a package upgrade. The immediate cause was an ext4 directory metadata inconsistency on its root filesystem. Automatic filesystem checking refused to repair the inconsistency unattended, preventing the root filesystem from mounting. Manual filesystem repair restored bootability.
Subsequent inspection from an Arch ISO, after restoring the before_upgrade snapshot, reportedly found the same corruption. That snapshot was created at 14:33:38. Restoring it therefore reintroduced the damaged filesystem state. The original cause of the corruption remains undetermined.
The investigation was stopped at the user's request. No physical disk failure, pacman defect, or snapshot implementation defect was established.
Environment#
- Host: xeno, Proxmox VE 7.2-7; running kernel 5.15.39-3-pve.
- Host QEMU package: pve-qemu-kvm 6.2.0-11.
- Guest: Brighid, VM 102, Arch Linux; one vCPU and 16 GiB RAM.
- Virtual disk:
vms:vm-102-disk-0, 20 GiB, virtio-scsi-pci controller. - Root filesystem: ext4 on guest
/dev/sda1. - Filesystem UUID:
98d4e0fd-308a-4b45-b587-64e4b921cfcb. - Host storage: LVM thin pool
vms/vmson host/dev/sda, a single 2 TB Western Digital WD20EFRX-68EUZN0 hard drive. - Another Arch VM, Amalthus (105), upgraded and rebooted successfully.
Host /dev/sda is the physical disk; guest /dev/sda1 is a partition inside Brighid's virtual disk. They are not interchangeable repair targets.
Symptoms and diagnosis#
The VNC console initially appeared to stop at:
virt/tdx: TDX not supported by the host platformAfter enabling verbose output, the last visible kernel message was:
random: crng init doneNeither message identified the failure. Capturing serial console output exposed a failed systemd-fsck-root.service, followed by dependency failures for /sysroot and entry into initramfs emergency mode. A locked root account initially prevented access to the emergency shell.
A temporary SYSTEMD_SULOGIN_FORCE=1 kernel parameter allowed inspection. The checker reported:
/dev/sda1: '..' in /usr/include/nghttp2 (287383) is /usr/share/locale/es/LC_MESSAGES (266242), should be /usr/include (262146).
/dev/sda1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
fsck failed with exit status 4.The incorrect parent-directory entry was the actionable inconsistency. Messages saying extent trees could be narrower were optimization notices, not the boot blocker.
Evidence timeline#
Restores rolled back guest logs as well as files. This is a reconstruction from retained guest logs and the independent host journal, not a complete history of every upgrade attempt.
| Time | Observation |
|---|---|
| 14:33:11 | Retained earlier guest journal ends with QEMU guest-agent filesystem-freeze and freeze-hook messages. |
| 14:33:38–14:33:40 | Host creates before_upgrade; snapshot task reports success. |
| 14:54 onward | Host records repeated guest-agent ping timeouts for VM 102. |
| 15:18:13 | Host records a VM reset. |
| 15:36–15:46 | Guest shutdown attempt times out; concurrent stop attempts encounter a configuration lock. |
| 15:48 | Host boots again. |
| 15:58:45 | Host records successful rollback to before_upgrade. |
| 17:01:38 | Host records another successful rollback to before_upgrade. |
| 17:01:58 | Retained pre-repair guest boot begins. |
| 17:03:05 | Journald replaces a user journal described as corrupted or uncleanly shut down. This message does not distinguish the two causes. |
| 17:06:40 | Earliest retained explicit ext4 corruption report: directory block checksum failure for inode 287383, while accessed by pacman. |
| 17:07:05 | Retained pacman log records package transaction start. |
| 17:07:06 | Pacman records upgrading libnghttp2 from 1.68.0-1 to 1.70.0-1; ext4 errors recur. |
| 19:26:03 | Successful repaired guest boot begins. |
| Later, exact time unrecorded | User restores before_upgrade and runs fsck from an Arch ISO; reports the same corruption. |
The 17:06:40 error precedes package replacement in the retained transaction. comm pacman identifies the process encountering the corrupt directory, not the process that necessarily caused the corruption.
The snapshot inspection places the damage in the captured pre-upgrade state. It does not distinguish damage already present before capture from a defect occurring during capture. The first afternoon upgrade sequence was not fully reconstructed because restores erased guest history.
Recovery and validation#
- Enabled verbose boot output and serial console capture using the already configured serial socket and
qm terminal 102. - Entered the initramfs emergency shell using the temporary
SYSTEMD_SULOGIN_FORCE=1parameter. - Identified the ext4 root partition and the failed filesystem check.
- Recommended manual
e2fsck -f /dev/sda1with the filesystem unmounted. The user reported that manual repair fixed booting; the full repair transcript was not retained. - The repaired boot's supplied kernel excerpt showed successful ext4 mounting and no ext4 errors.
pacman -Qkk libnghttp2reported two altered headers:nghttp2.hhad size, modification-time, and SHA256 mismatches;nghttp2ver.hhad modification-time and SHA256 mismatches. Repairing filesystem structure did not establish package-file integrity. Reinstallation of these files was not confirmed during the investigation.- The user later reported matching corruption when checking the restored
before_upgradesnapshot from an Arch ISO. The proposed command wase2fsck -fn /dev/sda1, without mounting the partition. Only the user's matching-result report is available because access was through VNC. Exact options, mount state, and any skipped-journal-recovery warnings could not be independently checked from a saved transcript.
Return to the repaired snapshot after the final diagnostic restore was recommended but not confirmed. Removal of temporary diagnostic boot parameters was also not independently confirmed.
Host storage findings#
The supplied host journal contained no reported physical disk I/O failures or thin-pool exhaustion during the collected incident window.
LVM reported pool data usage of 10.69% and metadata usage of 15.97%. Brighid's thin-volume Data% of 99.38% describes allocated virtual disk blocks, not guest filesystem fullness. VG free space of zero means space was assigned to logical volumes; the thin pool itself had ample free space.
The SMART report showed:
- Overall health assessment: PASSED.
- Zero reallocated, pending, or offline-uncorrectable sectors.
- Zero reported uncorrectable errors and interface CRC errors.
- No ATA errors or pending defects logged.
- No self-tests logged.
- 60,533 power-on hours, approximately 6.9 years.
- Current reported temperature of 55°C and lifetime maximum of 61°C; the drive reports a specified maximum operating temperature of 60°C.
These results provide no direct evidence of physical disk failure, but cannot exclude silent corruption or faults elsewhere in the I/O path. Drive age and temperature are maintenance concerns, not proven causes. An extended SMART test was suggested; no test result was supplied.
Other observations#
pacman -Sccfailed to remove leftoverdownload-*directories with “Is a directory.” This cleanup behavior was not evidence of ext4 corruption. Its position relative to the final restore was not clarified.- Miniflux repeatedly failed a database migration because its database user was not the owner of the
hstoreextension. The log showed over a million restart attempts. No causal connection to filesystem corruption was found. - The old Proxmox/kernel/QEMU versions are relevant to future software-bug investigation, but no particular defect was identified.
Conclusion and remaining work#
Confirmed immediate cause: ext4 directory metadata corruption prevented automatic root-filesystem checking from completing and blocked boot. Manual repair restored bootability.
Based on the user's offline check, the before_upgrade snapshot contains the same corruption, explaining its recurrence after restoration. The original corruption mechanism remains unknown. Evidence does not justify attributing it to pacman, later recovery resets, snapshot creation, or the physical drive.
If investigation resumes, useful next steps are to examine an older backup, obtain an extended SMART test result, and verify/restore affected package files on the repaired system. Preserve before_upgrade as evidence.
Preserved evidence excerpts#
These excerpts are embedded so the report does not depend on retaining the temporary source files. Repeated messages and unrelated lines are omitted. Original long log lines are preserved rather than wrapped. Host journal timestamps below are UTC; guest journal timestamps are PDT. Pacman entries include their UTC offset.
Guest freeze, corruption, and unrelated service failure#
Source: /tmp/log.log.
Sep 19 14:33:06 brighid systemd[1]: miniflux.service: Scheduled restart job, restart counter is at 1250629.
Sep 19 14:33:06 brighid postgres[393408]: 2026-09-19 14:33:06.482 PDT [393408] ERROR: must be owner of extension hstore
Sep 19 14:33:06 brighid miniflux[393403]: [Migration v119] pq: must be owner of extension hstore
Sep 19 14:33:11 brighid qemu-ga[436]: info: guest-fsfreeze called
Sep 19 14:33:11 brighid qemu-ga[436]: info: executing fsfreeze hook with arg 'freeze'
Sep 19 17:01:59 archlinux kernel: EXT4-fs (sda1): mounted filesystem 98d4e0fd-308a-4b45-b587-64e4b921cfcb r/w with ordered data mode. Quota mode: none.
Sep 19 17:02:05 brighid kernel: EXT4-fs (sda1): re-mounted 98d4e0fd-308a-4b45-b587-64e4b921cfcb.
Sep 19 17:06:40 brighid kernel: EXT4-fs error (device sda1): __ext4_find_entry:1624: inode #287383: comm pacman: checksumming directory block 0
Sep 19 17:06:40 brighid kernel: EXT4-fs error (device sda1): __ext4_find_entry:1624: inode #287383: comm pacman: checksumming directory block 0
Sep 19 17:07:07 brighid kernel: EXT4-fs error (device sda1): htree_dirblock_to_tree:1051: inode #287383: comm pacman: Directory block failed checksumHost snapshot, reset, and rollback history#
Source: /tmp/xeno_incident.log. Subtract seven hours for PDT. Snapshot-task success indicates completion, not filesystem integrity.
Sep 19 21:33:27 xeno pvedaemon[1437250]: <root@pam> delete snapshot VM 102: before_upgrade
Sep 19 21:33:38 xeno pvedaemon[1437284]: <root@pam> snapshot VM 102: before_upgrade
Sep 19 21:33:40 xeno pvedaemon[1271]: <root@pam> end task UPID:xeno:0015EE64:27168DEE:6AAEFFB2:qmsnapshot:102:root@pam: OK
Sep 19 21:54:39 xeno pvedaemon[1437501]: VM 102 qmp command failed - VM 102 qmp command 'guest-ping' failed - got timeout
Sep 19 22:18:13 xeno pvedaemon[1428997]: <root@pam> starting task UPID:xeno:00160E70:271AA2AE:6AAF0A25:qmreset:102:root@pam:
Sep 19 22:18:13 xeno pvedaemon[1428997]: <root@pam> end task UPID:xeno:00160E70:271AA2AE:6AAF0A25:qmreset:102:root@pam: OK
Sep 19 22:36:17 xeno pvedaemon[1448367]: can't lock file '/var/lock/qemu-server/lock-102.conf' - got timeout
Sep 19 22:36:17 xeno pvedaemon[1437914]: <root@pam> end task UPID:xeno:001619AF:271C4636:6AAF0E57:qmstop:102:root@pam: can't lock file '/var/lock/qemu-server/lock-102.conf' - got timeout
Sep 19 22:46:00 xeno pvedaemon[1448362]: VM 102 qmp command failed - VM 102 qmp command 'guest-shutdown' failed - got timeout
Sep 19 22:46:00 xeno pvedaemon[1448362]: VM quit/powerdown failed
Sep 19 22:46:00 xeno pvedaemon[1446509]: <root@pam> end task UPID:xeno:001619AA:271C43B2:6AAF0E50:qmshutdown:102:root@pam: VM quit/powerdown failed
Sep 19 22:48:08 xeno kernel: Linux version 5.15.39-3-pve (build@proxmox) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #2 SMP PVE 5.15.39-3 (Wed, 27 Jul 2022 13:45:39 +0200) ()
Sep 19 22:58:38 xeno pvedaemon[3519]: <root@pam> snapshot VM 102: after_upgrade
Sep 19 22:58:38 xeno pvedaemon[1283]: <root@pam> end task UPID:xeno:00000DBF:0000FA6E:6AAF139E:qmsnapshot:102:root@pam: OK
Sep 19 22:58:45 xeno pvedaemon[3555]: <root@pam> rollback snapshot VM 102: before_upgrade
Sep 19 22:58:46 xeno pvedaemon[1282]: <root@pam> end task UPID:xeno:00000DE3:0000FD3D:6AAF13A5:qmrollback:102:root@pam: OK
Sep 20 00:01:19 xeno pvedaemon[14870]: <root@pam> delete snapshot VM 102: after_upgrade
Sep 20 00:01:29 xeno pvedaemon[14898]: <root@pam> snapshot VM 102: after_upgrade
Sep 20 00:01:30 xeno pvedaemon[1282]: <root@pam> end task UPID:xeno:00003A32:0006BBA9:6AAF2259:qmsnapshot:102:root@pam: OK
Sep 20 00:01:38 xeno pvedaemon[14931]: <root@pam> rollback snapshot VM 102: before_upgrade
Sep 20 00:01:38 xeno pvedaemon[1283]: <root@pam> end task UPID:xeno:00003A53:0006BEF1:6AAF2262:qmrollback:102:root@pam: OKPackage transaction#
Source: /tmp/pacman.log. The grep-generated source line-number prefixes have been removed; the log entry text is unchanged.
[2026-09-19T17:07:05-0700] [ALPM] transaction started
[2026-09-19T17:07:06-0700] [ALPM] upgraded libnghttp2 (1.68.0-1 -> 1.70.0-1)
[2026-09-19T17:07:08-0700] [ALPM] upgraded e2fsprogs (1.47.3-2 -> 1.47.4-1)
[2026-09-19T17:08:14-0700] [ALPM] upgraded pacman (7.1.0.r7.gb9f7d4a-1 -> 7.1.0.r9.g54d9411-2)Versions, storage allocation, snapshot task, and disk health#
Sources: /tmp/pv.log and /tmp/xeno_smart.log. Selected rows retain their original column spacing.
proxmox-ve: 7.2-1 (running kernel: 5.15.39-3-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
lvm2: 2.03.11-2.1
pve-qemu-kvm: 6.2.0-11
qemu-server: 7.2-3
/dev/nvme0n1p3 pve 237.97g 15.99g
/dev/sda vms <1.82t 0
VG LV Attr Pool Origin Data% Meta%
vms snap_vm-102-disk-0_after_upgrade Vri---tz-k vms
vms snap_vm-102-disk-0_before_upgrade Vri---tz-k vms
vms vm-102-disk-0 Vwi-aotz-- vms snap_vm-102-disk-0_before_upgrade 99.38
vms vms twi-aotz-- 10.69 15.97
snapshotting 'drive-scsi0' (vms:vm-102-disk-0)
Logical volume "snap_vm-102-disk-0_before_upgrade" created.
TASK OK
Device Model: WDC WD20EFRX-68EUZN0
Firmware Version: 82.00A82
Local Time is: Sat Sep 19 19:51:46 2026 PDT
SMART overall-health self-assessment test result: PASSED
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
9 Power_On_Hours -O--CK 018 018 000 - 60533
194 Temperature_Celsius -O---K 092 087 000 - 55
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 100 253 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
No Errors Logged
No self-tests have been logged. [To run self-tests, use: smartctl -t]
Current Temperature: 55 Celsius
Lifetime Min/Max Temperature: 19/61 Celsius
Min/Max recommended Temperature: 0/60 Celsius
0x05 0x058 1 60 --- Specified Maximum Operating Temperature
No Defects LoggedAdditional evidence supplied directly in the conversation#
The following excerpts were pasted by the user rather than read from the local evidence files.
Journal warning (PDT):
Sep 19 17:03:05 brighid systemd-journald[338]: File /var/log/journal/a05c07ec643d420aabb6716d54617e64/user-1000.journal corrupted or uncleanly shut down, renaming and replacing.Failed boot and inaccessible emergency console:
[FAILED] Failed to start File System Check on /dev/disk/by-uuid/98d4e0fd-308a-4b45-b587-64e4b921cfcb.
See 'systemctl status systemd-fsck-root.service' for details.
[DEPEND] Dependency failed for /sysroot.
[DEPEND] Dependency failed for Initrd Root File System.
[DEPEND] Dependency failed for Mountpoints Configured in the Real Root.
[ OK ] Started Emergency Shell.
[ OK ] Reached target Emergency Mode.
Cannot open access to console, the root account is locked.Filesystem-check service journal (UTC, as shown by its status output):
Sep 20 02:20:15 archlinux systemd-fsck[118]: /dev/sda1: '..' in /usr/include/nghttp2 (287383) is /usr/share/locale/es/LC_MESSAGES (266242), should be /usr/include (262146).
Sep 20 02:20:15 archlinux systemd-fsck[118]: /dev/sda1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
Sep 20 02:20:15 archlinux systemd-fsck[118]: (i.e., without -a or -p options)
Sep 20 02:20:15 archlinux systemd-fsck[113]: fsck failed with exit status 4.Filesystem identification:
/dev/sda1: UUID="98d4e0fd-308a-4b45-b587-64e4b921cfcb" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="adfbd697-01"Repaired boot (PDT):
Sep 19 19:26:04 archlinux kernel: EXT4-fs (sda1): mounted filesystem 98d4e0fd-308a-4b45-b587-64e4b921cfcb r/w with ordered data mode. Quota mode: none.
Sep 19 19:26:09 brighid kernel: EXT4-fs (sda1): re-mounted 98d4e0fd-308a-4b45-b587-64e4b921cfcb.Package ownership and verification after repair:
/usr/include/nghttp2/ is owned by libnghttp2 1.70.0-1
warning: libnghttp2: /usr/include/nghttp2/nghttp2.h (Modification time mismatch)
warning: libnghttp2: /usr/include/nghttp2/nghttp2.h (Size mismatch)
warning: libnghttp2: /usr/include/nghttp2/nghttp2.h (SHA256 checksum mismatch)
warning: libnghttp2: /usr/include/nghttp2/nghttp2ver.h (Modification time mismatch)
warning: libnghttp2: /usr/include/nghttp2/nghttp2ver.h (SHA256 checksum mismatch)
libnghttp2: 15 total files, 2 altered filesRepresentative cache-cleanup error (22 such directory errors were pasted):
error: could not remove /var/cache/pacman/pkg/download-ffym7V: Is a directoryOffline snapshot check: no transcript was available. The user's reports were: “the corruption was already there” and “the check result was the same.” This is a user-confirmed observation, not a preserved fsck log.