The cluster stack
SSH
Install and configure
Firewall
Quorum
Start Pacemaker
Check cluster status
Cluster health check
Adding a resource

Getting Started

So, you’ve successfully installed crmsh on one or more machines, and now you want to configure a basic cluster. This guide is intended to provide step-by-step instructions for configuring Pacemaker with a single resource capable of failing over between a pair of nodes, and then builds on that base to cover some more advanced topics of cluster management.

Haven’t installed yet? Please follow the installation instructions before continuing this guide. Only crmsh and its dependencies need to be installed before following this guide.

Before continuing, make sure that this command executes successfully on all nodes, and returns a version number that is 2.1 or higher:

crm version
Example cluster

These are the machines used as an example in this guide. Please replace the references to these names and IP addresses to the values appropriate for your cluster:

Name IP

bob

10.0.0.3

alice

10.0.0.2

The cluster stack

The composition of the GNU/Linux cluster stack has changed somewhat over the years. The stack described here is the currently most common variant, but there are other ways of configuring these tools.

Simply put, a High Availability cluster is a set of machines (commonly referred to as nodes) with redundant capacity, such that if one or more of these machines experience failure of any kind, the other nodes in the cluster can take over the responsibilities previously handled by the failed node.

The cluster stack is a set of programs running on all of these nodes, communicating with each other over the network to monitor each other and deciding where, when and how resources are stopped, started or reconfigured.

The main component of the stack is Pacemaker, the software responsible for managing cluster resources, allocating them to cluster nodes according to the rules specified in the CIB.

The CIB is an XML document maintained by Pacemaker, which describes all cluster resources, their configuration and the constraints that decide where and how they are managed. This document is not edited directly, and with the help of crmsh it is possible to avoid exposure to the underlying XML at all.

Beneath Pacemaker in the stack sits Corosync, a cluster communication system. Corosync provides the communication capabilities and cluster membership functionality used by Pacemaker. Corosync is configured through the file /etc/corosync/corosync.conf. crmsh provides tools for configuring corosync similar to Pacemaker.

Aside from these two components, the stack also consists of a collection of Resource Agents. These are basically scripts that wrap software that the cluster needs to manage, providing a unified interface to configuration, supervision and management of the software. For example, there are agents that handle virtual IP resources, web servers, databases and filesystems.

crmsh is a command line tool which interfaces against all of these components, providing a unified interface for configuration and management of the whole cluster stack.

SSH

crmsh runs as a command line tool on any one of the cluster nodes. In order for to to control all cluster nodes, it needs to be able to execute commands remotely. crmsh does this by invoking ssh.

Configure /etc/hosts on each of the nodes so that the names of the other nodes map to the IP addresses of those nodes. For example in a cluster consisting of alice and bob, executing ping bob when logged in as root on alice should successfully locate bob on the network. Given the IP addresses of alice and bob above, the following should be entered into /etc/hosts on both nodes:

10.0.0.2      alice
10.0.0.3      bob

Once this is done, SSH keys need to be installed for password-less access between the nodes. This is something that will hopefully be automated in the future, but is unfortunately a manual step at this point in time.

The following commands should be executed as root on one of the nodes (in this example, alice):

# ensure that the ssh server is started
sudo systemctl start sshd
# create the shared key
mkdir -m 700 -p /root/.ssh
ssh-keygen -q -f /root/.ssh/id_rsa -C "Cluster Internal" -N ''
cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys

On the other nodes in the cluster (in this example, bob), execute the following as root:

mkdir -m 700 -p /root/.ssh
scp -oStrictHostKeyChecking=no \
    root@alice:'/root/.ssh/id_rsa*' /root/.ssh
cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys

This enables SSH to connect without prompting for a password between the nodes. Make sure this works before continuing with this guide.

Install and configure

To install the basic packages and configure systemd to manage Corosync and Pacemaker, the following command is provided by crmsh:

crm cluster init nodes=alice,bob

Firewall

If your cluster nodes have a firewall configured, you will need to open the following ports:

  • TCP: 5560

  • UDP: 5404, 5405

  • TCP: 21064 (if using DLM)

  • TCP: 30865 (if using csync2)

  • TCP: 7630 (if using the hawk web UI)

Tip
Issues with the firewall may manifest itself by cluster nodes being shown as UNCLEAN in the crm status output. If you see this, it is likely that a firewall rule is blocking cluster communications, or there may be some other problem with the network connection between the nodes.

Quorum

At this point, corosync needs to be configured for the particular cluster being created. First of all, quorum needs to taken into consideration. To make things as easy as possible, it is advisable to have a total number of nodes in the cluster that is not divisible by two. In other words, the number of nodes in the cluster should usually be either 3 or 5. This is to ensure quorum, that is, in the case of a loss of network connectivity between some subsets of nodes, one of the network partitions will either have more members than the others, or every node in the cluster will be isolated.

To configure corosync to manage quorum for you, you need to enable votequorum in the corosync configuration. To do this, the following command can be used:

crm corosync set quorum.provider corosync_votequorum

Corosync also needs to know the number of nodes required to be considered a majority vote. Usually, this should be set to 2:

crm corosync set quorum.expected_votes 2

After changing the quorum settings, the changes need to be propagated across the cluster and corosync needs to be restarted. To do this, the following sequence of commands can be used:

crm corosync push
crm cluster stop
crm cluster start
Note
Restarting the cluster is only necessary if the cluster has already been started.

Start Pacemaker

To start Corosync and Pacemaker, the following command can be used:

crm cluster start

Check cluster status

To see if Pacemaker is running, what nodes are part of the cluster and what resources are active, use the status command:

crm status

If this command fails or times out, there is some problem with Pacemaker or Corosync on the local machine. Perhaps some dependency is missing, a firewall is blocking cluster communication or some other unrelated problem has occurred. If this is the case, the cluster health command may be of use.

Cluster health check

To check the health status of the machines in the cluster, use the following command:

crm cluster health

This command will perform multiple diagnostics on all nodes in the cluster, and return information about low disk space, communication issues or problems with mismatching software versions between nodes, for example.

If no cluster has been configured or there is some fundamental problem with cluster communications, crmsh may be unable to figure out what nodes are part of the cluster. If this is the case, the list of nodes can be provided to the health command directly:

crm cluster health nodes=alice,bob

Adding a resource

To test the cluster and make sure it is working properly, we can configure a Dummy resource. The Dummy resource agent is a simple resource that doesn’t actually manage any software. It exposes a single numerical parameter called state which can be used to test the basic functionality of the cluster before introducing the complexities of actual resources.

To configure a Dummy resource, run the following command:

crm configure primitive p0 Dummy

This creates a new resource, gives it the name p0 and sets the agent for the resource to be the Dummy agent.

crm status should now show the p0 resource as started on one of the cluster nodes:

# crm status
Last updated: Wed Jul  2 21:49:26 2014
Last change: Wed Jul  2 21:49:19 2014
Stack: corosync
Current DC: alice (2) - partition with quorum
Version: 1.1.11-c3f1a7f
2 Nodes configured
1 Resources configured


Online: [ alice bob ]

 p0     (ocf::heartbeat:Dummy): Started alice

The resource can be stopped or started using the resource start and resource stop commands:

crm resource stop p0
crm resource start p0
Fork me on GitHub