Operating a Proxmox Cluster One Node at a Time

For most of my personal and professional career, I have primarily used the VMware ecosystem, including Workstation, Player, ESXi, and vSphere. In situations where it made sense, I also worked with Hyper-V.

With the Broadcom acquisition of VMware, many organizations have started evaluating alternatives and moving away from the platform. While there are many virtualization platforms available today, personally I found Proxmox to be a great fit for my environment.

Proxmox is well supported, well documented, and offers enterprise options for organizations that need them. The platform covers everything I need while also being built on Debian. Since my typical go-to operating systems are Debian and Ubuntu Server, this was a natural fit. I am already familiar with the distributions, where things are located, how to navigate the systems, and the various foot-guns that come from years of working with them.

My Infrastructure

Inside my 24U Dell rack, I currently have approximately six servers in operation, two Juniper switches, a Sophos appliance running pfSense, two UPS systems, and my ISP modem.

This environment has grown beyond a traditional homelab. While it is still personal infrastructure, it operates more like a small private cloud where I manage compute, networking, storage, backups, and the services running on top of it.

What Proxmox Offers Me

Running a Proxmox cluster across my servers provides me with:

• A centralized administration portal for all my compute resources.
• API access for automation and integrations, along with monitoring through LibreNMS.
• The ability to use Terraform to provision and spin up resources.
• High availability capabilities for critical services and the ability to migrate VMs between nodes with ease.
• PBS (Proxmox Backup Server) as a centralized backup solution across my servers for DR (Disaster Recovery).
• Proxmox VE accounts with TOTP authentication, allowing me to use a single account to manage everything across the cluster.
• Proxmox VE Firewall and SDN networking capabilities across the environment.
• Shared storage pools and centralized storage management.

Using Nginx, I have also created an upstream cluster containing all of my Proxmox servers. This allows me to use a single subdomain and access whichever server in the cluster is currently available.

This is a small quality-of-life improvement, but it makes managing the environment much easier. I no longer need to remember or bookmark each individual server URL. I have one entry point, and Nginx handles routing me to the available system.

Power Consumption & Cooling

While I run my own real infrastructure, mission-critical services, and have services with real users, my overall compute capacity is far beyond even my busiest hours and workloads.

If we frame this through the lens of AWS, you have options like Serverless/Lambda where you pay per request, or hosting static content through services like S3 and CloudFront where you pay based on storage, requests, and data transfer. These approaches can be extremely cost-effective depending on the workload and requirements.

Then you have EC2 On-Demand, where you have an always-on VM running 24/7. This can be a great solution because it avoids cold starts and provides consistent resources, but you are paying for that capacity regardless of whether it is actively being used.

That is essentially my current situation. I have all of these systems running, and I am footing the bill for the idle hours, peak workloads, and everything in between.

The cost is not just the electricity required to keep the equipment powered on. There is also the heat output generated by all of this equipment, which has a direct impact on cooling requirements and overall power consumption.

Reducing My Electrical Footprint

Over the last month or so, I've started an entire initiative to reduce my power consumption and cooling demands from all of my equipment.

Over the last several months, I reduced my always-on infrastructure down to one large server (32 cores, 256GB RAM, 38TB usable storage pool). The total average power draw was a pretty consistent ~600W. Prior to any changes, I was averaging ~800W.

Not including cooling, this equated to approximately 14.4 kWh per day, and in a 28-day billing cycle, approximately 403.2 kWh per month.

To further improve this, I opted to move away from 10Gb networking by powering down the Juniper EX4600 and moving back to my primary Juniper switch while utilizing LACP bonding. This change alone brought my overall power draw from around ~600W down to approximately ~450W.

Not including cooling, this equated to approximately 10.8 kWh per day, and in a 28-day billing cycle, approximately 302.4 kWh per month, which was a significant improvement.

In order to make this entire operation far more power efficient, I decided to go even further by moving all of my critical services (the most important workloads) onto a super low-power 2U platform and shutting down the larger server.

At this point, I am down to one modem, one Juniper switch, one small server, and two UPS systems, which together average approximately ~285W.

Power Reduction Summary

To visualize the impact of these changes, here is the progression of my always-on infrastructure power consumption.

This represents approximately a 52.5% reduction in always-on infrastructure power consumption, bringing the total from around 403.2 kWh/month down to approximately 191.5 kWh/month, not including cooling costs.

While I couldn't fit everything I needed onto the smaller system, I made the compromise to only power on my larger main server when needed and shut it back down once I am finished.

The Quorum Problem

The challenge with running a Proxmox cluster is that it is designed around multiple nodes being online. Once I started shutting down my larger nodes to reduce power consumption, I needed a way for my remaining node to safely operate by itself while still automatically starting the required services.

Just like in VMware, I set all my services to Auto-Start. However, the problem is when the entire cluster is essentially offline, Proxmox will not start the VMs due to quorum requirements.

My process would be to boot the server that had the services I needed, wait for it to fully boot, and then SSH into the server and manually run:

pvecm expected 1

Once running this command, all of the VMs and containers would auto start as expected. I would need to manually perform this step on every node I needed to boot and access while the rest of the cluster remained offline.

Since this isn't permanent, I finally decided to come up with a solution for this so I could just boot the node I needed and have everything come up automatically, minimizing the steps needed to get up and running.

The Solution: Automating Expected Votes

To solve this, I created a small systemd service that automatically runs pvecm expected 1 after the Proxmox cluster services are available. This removes the manual SSH step after every reboot and allows the node to continue with the normal VM and container startup process.

The script waits until the Proxmox cluster is online before applying the expected vote adjustment.

The Script

I created the following script at:

/root/setVote.sh

#!/usr/bin/env bash

# Wait until Corosync membership is available
while ! corosync-quorumtool 2>/dev/null | grep -q "Membership information"; do
    sleep 5
done

echo "Determining online votes..."

ONLINE_VOTES=$(corosync-quorumtool 2>/dev/null | awk '
/^[[:space:]]*[0-9x]+[[:space:]]+[0-9]+[[:space:]]/ {
    votes += $2
}
END {
    print votes
}')

echo "Online votes: $ONLINE_VOTES"

if [[ "$ONLINE_VOTES" =~ ^[0-9]+$ ]] && [ "$ONLINE_VOTES" -gt 0 ]; then
    echo "Setting expected votes to $ONLINE_VOTES"
    pvecm expected "$ONLINE_VOTES"
else
    echo "Unable to determine online votes"
    exit 1
fi

After creating the script, I made it executable:

chmod +x /root/setVote.sh

Systemd Service

To run this automatically after boot, I created a systemd service:

/etc/systemd/system/proxmox-expected-votes.service

[Unit]
Description=Set Proxmox Expected Votes
After=pve-cluster.service corosync.service network-online.target

[Service]
Type=oneshot
ExecStart=/root/setVote.sh

[Install]
WantedBy=multi-user.target

Enable the Service

After creating the service, systemd needs to be reloaded and the service enabled:

systemctl daemon-reload

systemctl enable proxmox-expected-votes.service

Going back to a normal operation, you can just disable the service from starting at all with:

systemctl disable proxmox-expected-votes.service

Testing

The service can be started immediately without waiting for a reboot:

systemctl start proxmox-expected-votes.service

To verify the service status:

systemctl status proxmox-expected-votes.service

Logs can be viewed with:

journalctl -u proxmox-expected-votes.service

Finally, verify the cluster quorum:

pvecm status

The expected result:

Votequorum information
----------------------
Expected votes:   1
Highest expected: 1
Total votes:      1
Quorum:           1
Flags:            Quorate

The Result

With this service in place, my workflow is now much simpler. I can power on the node I need, allow it to boot, and the required quorum adjustment happens automatically.

This removes the manual step of connecting over SSH after every reboot while still allowing me to maintain the benefits of a Proxmox cluster when multiple nodes are powered on.