In a Linux HA cluster setup on RHEL 6 that involves replication between sites using DRBD suddenly after reboot of one of the nodes it was imposible to start any cluster service from that node.
Surprinsingly there was no issue to do so from the other node.
While looking in the /var/log/messages on the affected node I saw the following log lines.
Apr 22 13:07:28 PRODB rgmanager[7586]: [lvm] HA LVM: Improper setup detected
Apr 22 13:07:28 PRODB rgmanager[7606]: [lvm] * initrd image needs to be newer than lvm.conf
It turns out that this is a known issue at RedHat and solved as solution 21622
Basically the issue is caused by a check that the rgmanager is doing, comparing the lvm.conf file from the initramfs kernel image with the filesystem file lvm.conf.
In case the check fails rgmanager will issue the above error.
As mentioned by RedHat:
NOTE: This warning will not cause a start operation to fail starting with resource-agents-3.9.2-21.el6 in RHEL 6 or rgmanager-2.0.52-37.el5 in RHEL 5.
To diagnose if this is the issue and to fix it follow the next steps:
STEP 1: Confirm that the lvm.conf copies are diferent
Execute the following commands:
# stat /etc/lvm/lvm.conf
# stat /boot/initramfs-$(uname -r).img
If the Modify: date is newer in case of the lvm.conf file that we can confirm the issue.
STEP 2: Back-up initramfs kernel image
Make a safety copy of the initrd/initramfs kernel image
# cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.$(date +%m-%d-%H%M%S).bak
STEP 3: Rebuild the initramfs kernel image
Execute the following to rebuild the image:
# dracut -f -v
Make sure that the above procedure is followed on all the cluster nodes and regenerate the initramfs.
[paypal_donation_button]