Red Hat Summit26.2. Installing Oracle 10gR2 Cluster Ready Services (CRS) with MPIO
There is a bug installing Oracle 10gR2 CRS services using MPIO(Multipath I/O). The bug has been reported to Oracle and the bug number and description states:
BUG 5005148 - CANNOT USE BLOCK DEVICES IN VOTING DISK AND OCR DURING 10G RAC INSTALLATION
BUG 5005148 - CANNOT USE BLOCK DEVICES IN VOTING DISK AND OCR DURING 10G RAC INSTALLATION
This has been fixed in Oracle 11g.
Before attempting to install CRS, ensure that all firewalls are disabled:
service iptables stop chkconfig iptables off
service iptables stop
chkconfig iptables off
There are workarounds to successfully install CRS using MPIO. MPIO has to be disabled during installation. After the Vipca portion of the CRS installation is complete, the MPIO can be turned back on, restart CRS services. In the following examples using MPIO, friendly names are used for ease of manageability. In this example Oracle 10g R2 CRS is using internal redundancy for the
OCR and VOTING disks.
Read
/usr/share/doc/device-mapper-multipath-0.4.5/Multipath-usage.txt for more information on configuring multipath.
First, determine which block devices will be associated with MPIO devices with your OCR or Voting Disks and save the output to a log file.
multipath -l > multipath.log
# multipath -l > multipath.log
ocr1 (36006016054141500beac25dcc436dc11)
[size=1 GB][features="0"][hwhandler="1 emc"]
\_ round-robin 0 [active]
\_ 1:0:1:0 sdc 8:32 [active]
\_ 2:0:1:0 sdn 8:208 [active]
ocr2 (36006016054141500a898dae7c436dc11)
[size=2 GB][features="0"][hwhandler="1 emc"]
\_ round-robin 0 [active]
\_ 1:0:1:1 sdd 8:48 [active]
\_ 2:0:1:1 sdo 8:224 [active]
vote1 (3600601605414150040a48bf2c436dc11)
[size=3 GB][features="0"][hwhandler="1 emc"]
\_ round-robin 0 [active]
\_ 1:0:1:2 sde 8:64 [active]
\_ 2:0:1:2 sdp 8:240 [active]
vote2 (36006016054141500c0fc0fffc436dc11)
[size=4 GB][features="0"][hwhandler="1 emc"]
\_ round-robin 0 [active]
\_ 1:0:1:4 sdg 8:96 [active]
\_ 2:0:1:4 sdr 65:16 [active]
vote3 (36006016054141500340b8309c536dc11)
[size=5 GB][features="0"][hwhandler="1 emc"]
\_ round-robin 0 [active]
\_ 1:0:1:5 sdh 8:112 [active]
\_ 2:0:1:5 sds 65:32 [active]
Delete raw links to Multipath Devices on all nodes:
rm -f /dev/raw/*
rm -f /dev/raw/*
Shut down multipath:
multipath -F #shuts off Multipath
multipath -F #shuts off Multipath
Looking at the multipath.log output file, you can see which block devices will used for Oracle CRS. To be sure you have the correct disk on all nodes, look at
/proc/partitons. In this installation example, each device has a unique size so it is easier to map. Now link the block devices to the raw devices on all nodes by modifying /etc/sysconfig/rawdevices. Run ll on the device to get the major minor number for a block device, as seen below.
ll /dev/sdc1 ll /dev/sdd1 ll /dev/sde1 ll /dev/sdg1 ll /dev/sdh1
# ll /dev/sdc1
brw-rw---- 1 root disk 8, 33 Oct 1 12:46 /dev/sdc1 #ocr1
# ll /dev/sdd1
brw-rw---- 1 root disk 8, 49 Oct 1 12:47 /dev/sdd1 #ocr2
# ll /dev/sde1
brw-rw---- 1 root disk 8, 65 Oct 1 12:47 /dev/sde1 #vote1
# ll /dev/sdg1
brw-rw---- 1 root disk 8, 97 Oct 1 12:48 /dev/sdg1 #vote2
# ll /dev/sdh1
brw-rw---- 1 root disk 8, 113 Oct 1 12:49 /dev/sdh1 #vote3
Change the configuration file
/etc/sysconfig/rawdevices using vi or your favorite editor and add the following block devices.
vi /etc/sysconfig/rawdevices
$ vi /etc/sysconfig/rawdevices
/dev/raw/raw1 8 33
/dev/raw/raw2 8 49
/dev/raw/raw3 8 65
/dev/raw/raw4 8 97
/dev/raw/raw5 8 113
On the other nodes run
blockdev so the partition table are read again and created:
blockdev --rereadpt /dev/sdc blockdev --rereadpt /dev/sdd blockdev --rereadpt /dev/sde blockdev --rereadpt /dev/sdg blockdev --rereadpt /dev/sdh
# blockdev --rereadpt /dev/sdc
# blockdev --rereadpt /dev/sdd
# blockdev --rereadpt /dev/sde
# blockdev --rereadpt /dev/sdg
# blockdev --rereadpt /dev/sdh
Double check on the other nodes that the minor numbers for the block devices match. The second number in the example below, 8 in this case, is the minor number and 1 is the major number.
ll /dev/sdh1
$ ll /dev/sdh1
brw-rw---- 1 root disk 8, 113 Oct 1 13:14 /dev/sdh1
Copy
/etc/sysconfig/rawdevices to other nodes then restart it. You will see the created raw devices in /dev/raw.
/etc/rc5.d/S56rawdevices start
# /etc/rc5.d/S56rawdevices start
Assigning devices:
/dev/raw/raw1 --> 8 33
/dev/raw/raw1: bound to major 8, minor 33
/dev/raw/raw2 --> 8 49
/dev/raw/raw2: bound to major 8, minor 49
/dev/raw/raw3 --> 8 65
/dev/raw/raw3: bound to major 8, minor 65
/dev/raw/raw4 --> 8 97
/dev/raw/raw4: bound to major 8, minor 97
/dev/raw/raw5 --> 8 113
/dev/raw/raw5: bound to major 8, minor 113
done
Now check that the permissions are set for the Oracle installation.
ll /dev/raw
# ll /dev/raw
total 0
crwxrwxrwx 1 oracle dba 162, 1 Oct 1 13:13 raw1
crwxrwxrwx 1 oracle dba 162, 2 Oct 1 13:13 raw2
crwxrwxrwx 1 oracle dba 162, 3 Oct 1 13:13 raw3
crwxrwxrwx 1 oracle dba 162, 4 Oct 1 13:13 raw4
crwxrwxrwx 1 oracle dba 162, 5 Oct 1 13:13 raw5
Now you can commence an Oracle Clusterware installation.
Note
If during the installation process, Oracle says the raw devices are busy, then Multipath was left on in one of the nodes.
After the
VIPCA portion of the Oracle CRS is complete, you need to comment out the raw devices you created and copy /etc/sysconfig/rawdevices to the other nodes:
#/dev/raw/raw1 8 33 #/dev/raw/raw2 8 49 #/dev/raw/raw3 8 65 #/dev/raw/raw4 8 97 #/dev/raw/raw5 8 113
#/dev/raw/raw1 8 33
#/dev/raw/raw2 8 49
#/dev/raw/raw3 8 65
#/dev/raw/raw4 8 97
#/dev/raw/raw5 8 113
Check to be sure the Oracle Notification Services has started on all nodes.
ps -ef |grep ons
# ps -ef |grep ons
oracle 20058 1 0 13:49 ? 00:00:00 /ora/crs/opmn/bin/ons -d
oracle 20059 20058 0 13:49 ? 00:00:00 /ora/crs/opmn/bin/ons -d
root 20812 28286 0 13:50 pts/2 00:00:00 grep ons
The installation is now complete. Now you can to turn off Oracle CRS and start using MPIO.
Note
It is best to do this process one node at a time. No need for reboots during this process.
Shutdown the Oracle CRS Process.
/ora/crs/bin/crsctl stop crs
# /ora/crs/bin/crsctl stop crs
Stopping resources.
Successfully stopped CRS resources
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
Delete the raw devices created for Oracle CRS installation.
rm -f /dev/raw/*
$ rm -f /dev/raw/*
Turn on Multipath I/O (MPIO) again.
multipath
$ multipath
Create a simple script in
/etc/rc5.d/ (before CRS Starts but after MPIO starts) to bind raw devices using MPIO.
num="1"
for i in `ls /dev/mpath/ocr?p? | sort`
do
raw /dev/raw/raw${num} $i
let "num = $num + 1"
done
num="3"
for i in `ls /dev/mpath/vote?p? | sort`
do
raw /dev/raw/raw${num} $i
let "num = $num + 1"
donenum="1"
for i in `ls /dev/mpath/ocr?p? | sort`
do
raw /dev/raw/raw${num} $i
let "num = $num + 1"
done
num="3"
for i in `ls /dev/mpath/vote?p? | sort`
do
raw /dev/raw/raw${num} $i
let "num = $num + 1"
done
Then run the script,
/etc/rc5.d/, that you just created.
/etc/rc5.d/S57local start
# /etc/rc5.d/S57local start
/dev/raw/raw1: bound to major 253, minor 25
/dev/raw/raw2: bound to major 253, minor 21
/dev/raw/raw3: bound to major 253, minor 22
/dev/raw/raw4: bound to major 253, minor 23
/dev/raw/raw5: bound to major 253, minor 24
Verify the correct permissions are all set.
ll /dev/raw
# ll /dev/raw
total 0
crwxrwxrwx 1 oracle dba 162, 1 Oct 1 13:58 raw1
crwxrwxrwx 1 oracle dba 162, 2 Oct 1 13:58 raw2
crwxrwxrwx 1 oracle dba 162, 3 Oct 1 13:58 raw3
crwxrwxrwx 1 oracle dba 162, 4 Oct 1 13:58 raw4
crwxrwxrwx 1 oracle dba 162, 5 Oct 1 13:58 raw5
Restart CRS and be sure the ONS processes all start. This ensures Oracle CRS is functional on this node.
/ora/crs/bin/crsctl start crs
# /ora/crs/bin/crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
Repeat for the process for the rest of the nodes. Now the Oracle CRS installation is complete using Multipath I/O.
Once complete, you can check the status of the nodes:
crs_stat -t
# crs_stat -t
Name Type Target State Host
------------------------------------------------------------------------
ora....t08.gsd application ONLINE ONLINE et-virt08
ora....t08.ons application ONLINE ONLINE et-virt08
ora....t08.vip application ONLINE ONLINE et-virt08
ora....t09.gsd application ONLINE ONLINE et-virt09
ora....t09.ons application ONLINE ONLINE et-virt09
ora....t09.vip application ONLINE ONLINE et-virt09
ora....t10.gsd application ONLINE ONLINE et-virt10
ora....t10.ons application ONLINE ONLINE et-virt10
ora....t10.vip application ONLINE ONLINE et-virt10
ora....t11.gsd application ONLINE ONLINE et-virt11
ora....t11.ons application ONLINE ONLINE et-virt11
ora....t11.vip application ONLINE ONLINE et-virt11
#
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}