Five-node Proxmox and Ceph cluster on mixed Dell and HPE hardware

Context

I led the rebuild at a system integrator I have since left. The cluster was the team's internal lab: we ran test VMs and containers on it, and evaluated tools there before recommending them to customers. This page describes it as it stood when I left.

I designed the rebuild so the cluster keeps its data, and keeps serving it, when any one node fails. Everything below follows from that goal.

The hardware is five bare-metal servers, two Dell and three HPE, with seven drive bays each. One bay holds an SSD that boots Proxmox VE. The other six hold 1 TB HDDs, and every HDD belongs to Ceph: 30 disks, about 30 TB raw.

Constraints:

• Mixed vendors and disk models in one cluster
• SSDs are reserved for the hypervisor OS; Ceph gets HDDs only
• Internal lab: one site, no disaster recovery, no backup outside Ceph
• Hardened to CIS Level 1 plus a Proxmox hardening checklist

Architecture and failover

Compute: Proxmox VE with HA

All five nodes run Proxmox VE and host VMs and containers. Corosync gives each node one vote, so the cluster keeps quorum while three of five are up. One HA group covers every node.

When a node stops responding, the HA manager waits for it to be fenced: its local watchdog resets it, and IPMI confirms the power is off. Only then does Proxmox restart its VMs on the surviving nodes. Fencing first means a VM can never run on two nodes at once against the same disk.

Storage: Ceph on every node

Every node runs Ceph next to its VMs:

• One OSD per HDD, 30 in total, holding the data.
• A monitor per node. The five monitors hold the cluster map and keep quorum while three are up.
• Managers, which run the health checks and report metrics.
• Metadata servers for CephFS.

Proxmox stores VM and container disks as block devices, so those live on RBD. ISO images, templates, scripts and notes are files, so they live on CephFS. Each use has its own pool.

The fix for the first version's failure is erasure coding. Ceph splits each object into data chunks and parity chunks and spreads them across nodes, so no single node holds enough of an object to make it unreadable. When a node fails, Ceph reads every object from the other four and I/O keeps running. Half of raw capacity stays usable, about 15 TB. Each erasure-coded pool is paired with a small replicated pool, at three copies, for metadata.

Until the failed node returns, or Ceph rebuilds its chunks elsewhere, the pools run with less redundancy. A second node failure in that window puts data at risk.

Keeping nodes up: alerts before failures

Erasure coding handles a node that fails. The BMCs reduce how often that happens. iDRAC on the Dell servers and iLO on the HPE servers report failing parts over IPMI before they take a node down: a disk with rising SMART errors, a power supply or fan going bad, memory errors. A disk that predicts its own failure is drained and swapped while it still works.

The watchdog does a different job. It cannot prevent a fault, but when a node hangs it resets that node, so HA can safely move its VMs. The alerts shorten how long nodes are down; the watchdog shortens how long VMs are down.

Operations: logs, drift and the baseline

Every node sends system, Ceph and audit logs to ELK, alongside the BMC alerts. The team ran several ELK stacks side by side, partly to compare them before recommending one to customers.

An n8n workflow reads alerts and configuration-change events from ELK. It notifies the team when a service or node misbehaves, and an AI step writes a daily performance summary. When a node drifts from the CIS Level 1 baseline, n8n calls a CI/CD webhook, and the pipeline re-applies the baseline from git. The AI step reports and never writes a change to a node.

Try it: fail a node, catch a disk before it dies, cut power to the whole site, or break the CIS baseline and watch the pipeline restore it. Timings are illustrative, based on Proxmox HA and Ceph defaults.

Key decisions

Lifecycle

Ceph treats a missing disk as lost and starts copying its data to other disks to restore redundancy. On HDDs that copy takes hours. So planned work starts by telling Ceph the disk or node will be back, and every routine waits for storage to report healthy before it touches the next node or disk.

Failure modes

Results

How the 99.99% is calculated

Ceph stops serving only when two or more of the five nodes are down at the same moment. Assume each node is up 99.7% of the time, about 26 hours of downtime a year, and that nodes fail independently. Two of them are then down together about 0.009% of the time, roughly 47 minutes a year:

storage availability ≈ 1 − 10p²     p = 0.003  →  ≈ 99.991%

This is an estimate from the design, not a measured uptime. The BMC alerts are what keep p that low, because most failing parts get replaced before they take a node down.

The estimate leaves out failures that hit every node at once. The cluster runs 24/7 on one site, so a power cut stops all five nodes together, and so would losing the network they share or a bad change pushed to every node. A power cut means a full outage, not lost data: Ceph confirms a write only once enough chunks are on disk, so the cluster comes back with every write it acknowledged, as long as no disk or controller held writes in a volatile cache.

What I'd change

I would add a backup outside Ceph, such as Proxmox Backup Server. Erasure coding keeps data available through disk and node failures, but a bad command or a corrupted pool reaches every chunk at once, the same kind of event that destroyed the first version.

I would make site power the next fix: a UPS with a scripted clean shutdown turns a power cut from a crash into a planned stop.

I would give Ceph a small flash device per node for its metadata and write-ahead log, or move VM disks to a replicated pool. On HDD-only erasure coding, every write waits for the slowest disk, and mixed disk models widen that gap.

Once the ELK evaluation ends, I would consolidate to one stack and document which logs the AI summary may read.