Cluster not ready - no quorum? (500)

cluster not ready - no quorum? (500) or Cannot initialize CMAP service

Its suggested that you create a new storage where only the node which you want to separate has access. This can be an new export on your NFS or a new Ceph pool, to name a few examples. Its just important that the exact same storage does not gets accessed by multiple clusters. After setting this storage up move all data from the node and its VMs to it. Then you are ready to separate the node from the cluster.

Warning Ensure all shared resources are cleanly separated! You will run into conflicts and problems else.

First stop the corosync and the pve-cluster services on the node:

systemctl stop pve-cluster; systemctl stop corosync

Start the cluster filesystem again in local mode:

pmxcfs -l

Delete the corosync configuration files:

rm /etc/pve/corosync.conf; rm /etc/corosync/*

You can now start the filesystem again as normal service:

killall pmxcfs ;systemctl start pve-cluster

The node is now separated from the cluster. You can deleted it from a remaining node of the cluster with:

pvecm delnode oldnode

If the command failed, because the remaining node in the cluster lost quorum when the now separate node exited, you may set the expected votes to 1 as a workaround:

pvecm expected 1

And then repeat the pvecm delnode command.

Now switch back to the separated node, here delete all remaining files left from the old cluster. This ensures that the node can be added to another cluster again without problems.

rm /var/lib/corosync/*

As the configuration files from the other nodes are still in the cluster filesystem you may want to clean those up too. Remove simply the whole directory recursive from /etc/pve/nodes/NODENAME , but check three times that you used the correct one before deleting it.

Caution The nodes SSH keys are still in the authorized_key file, this means the nodes can still connect to each other with public key authentication. This should be fixed by removing the respective keys from the /etc/pve/priv/authorized_keys file.