High Availability LVM :-
How to configure HA-LVM in RedHat
Cluster
Prerequisites:-
Actually we are going to
configure active-standby linux cluster so minimum two nodes are required for
configuring RHCS:-
Operating System :- Red Hat Enterprise
Linux Server release 6.7 (Santiago)
Hostname and IP of node1:- server1.example.com;
10.0.0.101/24
Hostname and IP of node2:- server2.example.com;
10.0.0.102/24
Third node will be used for management
purpose of cluster so luci is installed & shared iscsi storage server is
also configured on the same node.
Hostname and IP of node3:-
storage.example.com; 10.0.0.140/24
{ Note:- SELINUX is disabled }
Install Necessary Packages:-
On server1 & server2 install these
packages:-
#yum groupinstall "High
Availability" "Resilient Storage" -y
#yum -y install iscsi-initiator-utils
On storage node already iscsi server
is configured now install luci package on this node:-
#yum install luci –y
[root@server1 ~]# iscsiadm -m
discovery -t sendtargets -p storage.example.com
Starting iscsid: [ OK ]
10.0.0.140:3260,1
iqn.2015-12.com.example:share1
[root@server1 ~]# iscsiadm -m node
--targetname "iqn.2015-12.com.example:share1" --portal
"storage.example.com:3260" --login
Logging in to [iface: default, target:
iqn.2015-12.com.example:share1, portal: 10.0.0.140,3260] (multiple)
Login to [iface: default, target:
iqn.2015-12.com.example:share1, portal: 10.0.0.140,3260] successful.
[root@server1 ~]# service iscsi
restart
Stopping iscsi: [
OK ]
Starting iscsi:
[ OK ]
[root@server1 ~]# chkconfig iscsi on
Please confirm the iscsi disk is
attached or not
[root@server1 ~]# dmesg | tail
sd 34:0:0:1: [sdb] Mode Sense: 49 00
00 08
sd 34:0:0:1: [sdb] Write cache:
enabled, read cache: enabled, doesn't support DPO or FUA
sdb: unknown partition table
sd 34:0:0:1: [sdb] Attached SCSI disk
Please do the same steps on server2
:-
[root@server2 ~]# iscsiadm -m
discovery -t sendtargets -p storage.example.com
Starting iscsid:
[ OK ]
10.0.0.140:3260,1
iqn.2015-12.com.example:share1
[root@server2 ~]# iscsiadm -m node
--targetname "iqn.2015-12.com.example:share1" --portal
"storage.example.com:3260" --login
Logging in to [iface: default, target:
iqn.2015-12.com.example:share1, portal: 10.0.0.140,3260] (multiple)
Login to [iface: default, target:
iqn.2015-12.com.example:share1, portal: 10.0.0.140,3260] successful.
[root@server2 ~]# service iscsi
restart
Stopping iscsi:
[ OK ]
Starting iscsi:
[ OK ]
[root@server2 ~]# chkconfig iscsi on
[root@server2 ~]# dmesg | tail
sd 34:0:0:1: [sdb] Write Protect is
off
sd 34:0:0:1: [sdb] Mode Sense: 49 00
00 08
sd 34:0:0:1: [sdb] Write cache:
enabled, read cache: enabled, doesn't support DPO or FUA
sdb: unknown partition table
sd 34:0:0:1: [sdb] Attached SCSI disk
You have to assing the password for
ricci user on both the nodes:-
[root@server1 ~]# passwd --stdin ricci
Changing password for user ricci.
redhat
passwd: all authentication tokens
updated successfully.
Start the ricci service:-
[root@server1 ~]# /etc/init.d/ricci
start
Starting oddjobd:
[ OK ]
generating SSL certificates... done
Generating NSS database... done
Starting ricci:
[ OK ]
[root@server1 ~]# chkconfig ricci on
[root@server2 ~]# passwd --stdin ricci
Changing password for user ricci.
redhat
passwd: all authentication tokens
updated successfully.
[root@server2 ~]# /etc/init.d/ricci
start
Starting oddjobd:
[ OK ]
generating SSL certificates... done
Generating NSS database... done
Starting ricci: [ OK ]
[root@server2 ~]# chkconfig ricci on
Iptables Rules needs to add on both
the nodes:-
#iptables -I INPUT -m state --state
NEW -m multiport -p udp -s 10.0.0.0/24 -d 10.0.0.0/24 --dports 5404,5405 -j ACCEPTiptables
-I INPUT -m addrtype --dst-type MULTICAST -m state --state NEW -m multiport -p
udp -s 10.0.0.0/24 --dports 5404,5405 -j ACCEPT
#iptables -I INPUT -m state --state
NEW -p tcp -s 10.0.0.0/24 -d 10.0.0.0/24 --dport 21064 -j ACCEPT
#iptables -I INPUT -m state --state
NEW -p tcp -s 10.0.0.0/24 -d 10.0.0.0/24 --dport 11111 -j ACCEPT
#iptables -I INPUT -m state --state
NEW -p tcp -s 10.0.0.0/24 -d 10.0.0.0/24 --dport 16851 -j ACCEPT
#iptables -I INPUT -m state --state
NEW -p tcp -s 10.0.0.0/24 -d 10.0.0.0/24 --dport 8084 -j ACCEPT
#iptables -I INPUT -p igmp -j ACCEPT
#service iptables save ; service
iptables restart
Access Luci Console:-
Please enter username and password
Username:- root
Password:- our luci machine root user
password
After login Navigate the windows to "Manage Clusters" then click on Create
New pop-up will come here you need to
provide Cluster Name, Node’s hostname and ricci password
Here I have given mycluster as a
ClusterName,
Node Name:- server1.example.com &
server2.example.com password we already created for ricci user please give that
password, after that please check Enabled Shared Storage Support and Use
Locally Installed Packages
If everthing goes fine it will show
you below popup otherwise it will throw error
Now both the nodes has been added
successfully
Please make sure these services should
run on both the nodes:-
Cluster Daemons
|
|
Status
|
|
cman
|
Running
|
|
rgmanager
|
Running
|
|
Ricci
|
Running
|
|
Modclusterd
|
Running
|
|
Clvmd
|
Running
|
Now configure the HA-LVM with CLVM:-
Note:- locking_type parameter in
the global section of the /etc/lvm/lvm.conf file is set to the value
'3'.
[root@server1 ~]# fdisk -cu /dev/sdb
Device contains neither a valid DOS
partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk
identifier 0xb10ea483.
Changes will remain in memory only,
until you decide to write them.
After that, of course, the previous
content won't be recoverable.
Warning: invalid flag 0x0000 of
partition table 4 will be corrected by w(rite)
Command (m for help): p
Disk /dev/sdb: 8589 MB, 8589934592
bytes
64 heads, 32 sectors/track, 8192
cylinders, total 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512
bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes
/ 512 bytes
Disk identifier: 0xb10ea483
Device Boot Start End Blocks
Id System
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First sector (2048-16777215, default
2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G}
(2048-16777215, default 16777215):
Using default value 16777215
Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 8e
Changed system type of partition 1 to
8e (Linux LVM)
Command (m for help): p
Disk /dev/sdb: 8589 MB, 8589934592
bytes
64 heads, 32 sectors/track, 8192
cylinders, total 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512
bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes
/ 512 bytes
Disk identifier: 0xb10ea483
Device Boot Start End Blocks
Id System
/dev/sdb1 2048 16777215
8387584 8e Linux LVM
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition
table.
Syncing disks.
[root@server1 ~]# pvcreate /dev/sdb1
Physical volume "/dev/sdb1" successfully created
[root@server1 ~]# vgcreate -cy
cluster_vg /dev/sdb1
Clustered volume group "cluster_vg" successfully created
During the lvcreation you may get below error:-
[root@server1 ~]# lvcreate -L 7G -n cluster_lv cluster_vg
Error locking on node server2.example.com: Volume group for uuid not found: LWfm85PirK3ujvcShsSfOgP3PhFH4AU6KF8m0BcSfwU4dWD9DS19CMDh0aBJaagD
Failed to activate new LV
Solution:-
1) Make sure clvmd service is running on all the cluster nodes, if not start clvmd service using below command
# service clvmd start
2) Make sure the partition /dev/sdb1 is detected by the kernel and mapped on all the node
# partprobe ---> run this command on both the nodes server1 & server2
[root@server1 ~]# lvcreate -L 7G -n
cluster_lv cluster_vg
Logical volume "cluster_lv" created.
[root@server1 ~]# mkfs.ext4
/dev/mapper/cluster_vg-cluster_lv
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
458752 inodes, 1835008 blocks
91750 blocks (5.00%) reserved for the
super user
First data block=0
Maximum filesystem blocks=1879048192
56 block groups
32768 blocks per group, 32768
fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem
accounting information: done
This filesystem will be automatically
checked every 39 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
Add fencing to Cluster Select Fence
Devices,
{Note: Fencing is the disconnection of
a node from shared storage}
There are different types of fencing
devices available. Here I am using VMWARE virtual machine to build a cluster,
So I am selecting the option called VMware Fencing {SOAP Interface}
After selecting Vmware Fencing(SOAP
Interface)
Give desired name whatever you want, I
have given myfence as a Name, Address give your base machine IP Address {not
virtual machine ok}
Login :- login user of your base machine
Password:- password of that user
fence device created successfully
Step 1 – To get the UUID of your two
nodes please run the below command in terminal
[root@server1 ~]# fence_vmware_soap
--ip 192.168.58.1 --username "arun m" --password impel123 -z --action list | grep server1
server1,564d246b-949e-03fe-e3fc-14fc35ce9cb6
[root@server1 ~]# fence_vmware_soap
--ip 192.168.58.1 --username "arun m" --password impel123 -z --action list | grep server2
server2,564d8d2a-2054-1e05-22ce-b733e3f491d5
Step2:- Go back to the luci Manage
Clusters and click on select the firtnode server1.example.com
Then select Add Fence Method
Give any name here I have give
vmware_fence as a name, then click “Submit” your
changes
Now select “Add
Fence Instance”
In this section we are going to add your fencing device instance, select the option called myfence(VMware Fencing SOAP Interface)
[NOTE:- we have already configured this vmware fencing soap interface in Fence Devices tab ]
Next Give Vm name or any name {name
does not matter}
Next Give the UUID of server1
Next check the Use SSL option, then
click submit
Please do the same step for server2
also:-
Failover Domain Configuration:-
There are different types of failover
domains are there
Nodes in the ordered domain are
assigned a priority level from 1-100. Priority 1 being highest and 100 being
the lowest. A node with the highest priority will run the resource group. The
resource if it was running on node 2, will migrate to node 1 when it becomes
online.
After creating the whole cluster
setup, my cluster.conf file will look like the following
[root@server1 ~]# cat
/etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster
config_version="8" name="mycluster">
<clusternodes>
<clusternode
name="server1.example.com" nodeid="1"/>
<clusternode
name="server2.example.com" nodeid="2"/>
</clusternodes>
<cman expected_votes="1" two_node="1"/>
<rm>
<failoverdomains>
<failoverdomain
name="failoverdomain" nofailback="1"
ordered="1">
<failoverdomainnode name="server1.example.com"
priority="1"/>
<failoverdomainnode
name="server2.example.com" priority="2"/>
</failoverdomain>
</failoverdomains>
<resources>
<fs
device="/dev/cluster_vg/cluster_lv" force_unmount="1"
fsid="19408" fstype="ext4" mountpoint="/cluster"
name="myFS"/>
<lvm
lv_name="cluster_lv" name="lvm" self_fence="1"
vg_name="cluster_vg"/>
</resources>
<service
domain="failoverdomain" name="myservice"
recovery="relocate">
<lvm
ref="lvm"/>
<fs
ref="myFS"/>
</service>
</rm>
<fencedevices>
<fencedevice
agent="fence_vmware_soap" ipaddr="192.168.58.1"
login="arun m" name="myfence"
passwd="impel123"/>
</fencedevices>
</cluster>
[root@server1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_server1-lv_root
18G 5.6G
11G 35% /
tmpfs 1.9G 26M
1.9G 2% /dev/shm
/dev/sda1 477M 42M
411M 10% /boot
/dev/mapper/cluster_vg-cluster_lv
6.8G 16M
6.4G 1% /cluster
To test the cluster I am powering off
the server1:-
[root@server1 ~]# poweroff
[root@server1 ~]#
Broadcast message from
root@server1.example.com
(/dev/pts/0) at 20:39 ...
The system is going down for power off
NOW!
Verify server2 using df command LVM is
mounted on /cluster directory
[root@server2 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_server2-lv_root
18G 5.6G
11G 35% /
tmpfs 1.9G 26M
1.9G 2% /dev/shm
/dev/sda1 477M 42M
411M 10% /boot
[root@server2 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_server2-lv_root
18G 5.6G
11G 35% /
tmpfs 1.9G 26M
1.9G 2% /dev/shm
/dev/sda1 477M 42M
411M 10% /boot
/dev/mapper/cluster_vg-cluster_lv
6.8G 16M
6.4G 1% /cluster
