Though this task is well documented (see [1] and [2]) there are still some pitfalls and quirks that can cause problems. Such as..
Should a particular task be done with the root or oracle user? Should it be performed on both nodes?
This is a step-by-step description how to perform the change of the public IP address in the Oracle two node RAC. The Oracle version is 10.2.0.3. The OS used in this example is Linux Centos 2.6.9-42.ELsmp
Note that due to the change of the private IP address the virtual IP address (VIP) may be required to change as well as it must remain in the same subnet as the public IP address.
The configuration
| Host | Current IP Address | Changed IP Address |
Private IP | hst1 | 192.168.1.101 | 192.168.2.121 |
Private IP | hst2 | 192.168.1.100 | 192.168.2.120 |
VIP | hst1-vip | 192.168.1.111 | 192.168.2.111 |
VIP | hst2-vip | 192.168.1.110 | 192.168.2.110 |
Note that the actual value of IP addresses and host names was changed.
The basic steps
The change of the private IP address is performed in the following basic steps
1) Shut down everything except the CRS stack
2) Change the public interface
3) Modify the VIP address
4) Shut down CRS
5) Modify IP address on OS level and reconfigure /etc/hosts, listener,..
6) Restart
The order of the steps is very important. E.g. step 3) can be performed only if the CRS stack is up. If you perform step 5) prematurely, CRS will go down automatically (probably trying some reboots beforehand). This will effectively block step 3) to be done.
Shut Down Everything Except the CRS Stack
The database and nodeapps (on all nodes) are stopped.
[oracle@hst1 ~]$ srvctl stop database -d mydb
[oracle@hst1 ~]$ srvctl stop nodeapps -n hst1
[oracle@hst1 ~]$ srvctl stop nodeapps -n hst2
Note that if you use ASM, it must be stopped as well.
After that we verify the status.
[oracle@hst1 ~]$ srvctl status database -d mydb
Instance MYDB1 is not running on node hst2
Instance MYDB2 is not running on node hst1
[oracle@hst1 ~]$ srvctl status nodeapps -n hst1
VIP is not running on node: hst1
GSD is not running on node: hst1
Listener is not running on node: hst1
ONS daemon is not running on node: hst1
[oracle@hst1 ~]$ srvctl status nodeapps -n hst2
VIP is not running on node: hst2
GSD is not running on node: hst2
Listener is not running on node: hst2
ONS daemon is not running on node: hst2
Everything is OK, we can go to the next step.
Change the Public Interface
First let us have a look on the actual status
[oracle@hst1 ~]$ oifcfg getif
eth0 192.168.1.0 global public
eth1 192.168.2.0 global cluster_interconnect
We need to change the interface eth0.
As there is no modify command, we will delete and redefine the interface.
[oraclu@hst1 ~]$ oifcfg delif -global eth0
[oraclu@hst1 ~]$ oifcfg setif -global eth0/192.168.2.0:public
the exact syntax of the setif command can be found in [3]
Note that the CRS installation user (here oraclu) must be used for this operation. Otherwise an error message is issued:
PROC-5: User does not have permission to perform a cluster registry operation on
this key. Authentication error [User [oraclu] does not match with initialized u
ser] [0]
PRIF-11: cluster registry error
Finally, we check if the action was done successfully
[oracle@hst1 ~]$ oifcfg getif
eth0 192.168.2.0 global public
eth1 192.168.2.0 global cluster_interconnect
Modify the VIP Address
As already mentioned, due to the fact that we changed the subnet of the public IP address, we must change the VIP address as well.
The following modify statement should be used.
[root@hst1 ~]# /appl/oracle/product/10.2.0/db_1/bin/srvctl modify nodeapps -n hst2 -A 192.168.2.110/255.255.255.0/eth0
[root@hst1 ~]# /appl/oracle/product/10.2.0/db_1/bin/srvctl modify nodeapps -n hst1 -A 192.168.2.111/255.255.255.0/eth0
Note that the root user should be used for this action.
The variable ORACLE_HOHE must be initialised.
Otherwise one of the following errors will be raised.
PRKO-2117 : This command should be executed as the system privilege user.
****ORACLE_HOME environment variable not set!
ORACLE_HOME should be set to the main
directory that contains Oracle products.
Set and export ORACLE_HOME, then re-run.
Shut Down CRS
The stop command must be performed on all nodes:
crsctl stop crs
Modify the IP Address on OS Level
Modify the public IP address (eth0). In Centos with application / system setting / network
Perform the change on all nodes.
Modify /etc/hosts and listener.ora files if required.
Restart
Reboot all nodes and verify the status.
Shortly after reboot we can see...
[oracle@hst2 ~]$ /sbin/ifconfig -a | egrep '(eth|Mask)'
eth0 Link encap:Ethernet HWaddr CA:FE:CA:FE:CA:FE
inet addr:192.168.2.120 Bcast:192.168.2.255 Mask:255.255.255.0
eth0:1 Link encap:Ethernet HWaddr CA:FE:CA:FE:CA:FE
inet addr:192.168.2.111 Bcast:192.168.2.255 Mask:255.255.255.0
eth0:2 Link encap:Ethernet HWaddr CA:FE:CA:FE:CA:FE
inet addr:192.168.2.110 Bcast:192.168.2.255 Mask:255.255.255.0
eth1 Link encap:Ethernet HWaddr CA:FE:CA:FE:CA:F1
inet addr:192.168.2.100 Bcast:192.168.2.255 Mask:255.255.255.0
inet addr:127.0.0.1 Mask:255.0.0.0
This means both VIP's eth0:1 and eth0:2 are switched to one host. This is not good, normally this appears if one host is down and the second host takes over the VIP.
Fortunately, after a few seconds we see..
[oracle@hst2 ~]$ /sbin/ifconfig -a | egrep '(eth|Mask)'
eth0 Link encap:Ethernet HWaddr CA:FE:CA:FE:CA:FE
inet addr:192.168.2.120 Bcast:192.168.2.255 Mask:255.255.255.0
eth0:2 Link encap:Ethernet HWaddr CA:FE:CA:FE:CA:FE
inet addr:192.168.2.110 Bcast:192.168.2.255 Mask:255.255.255.0
eth1 Link encap:Ethernet HWaddr CA:FE:CA:FE:CA:F1
inet addr:192.168.2.100 Bcast:192.168.2.255 Mask:255.255.255.0
inet addr:127.0.0.1 Mask:255.0.0.0
[oracle@hst1 ~]$ /sbin/ifconfig -a | egrep '(eth|Mask)'
eth0 Link encap:Ethernet HWaddr CA:FE:CA:FE:CA:F2
inet addr:192.168.2.121 Bcast:192.168.2.255 Mask:255.255.255.0
eth0:1 Link encap:Ethernet HWaddr CA:FE:CA:FE:CA:F2
inet addr:192.168.2.111 Bcast:192.168.2.255 Mask:255.255.255.0
eth1 Link encap:Ethernet HWaddr CA:FE:CA:FE:CA:F3
inet addr:192.168.2.101 Bcast:192.168.2.255 Mask:255.255.255.0
inet addr:127.0.0.1 Mask:255.0.0.0
I.e. the VIP's are distributed correctly.
The configuration of IP addresses is as required.
Let’s have a final look at the status of the RAC.
[oracle@hst1 ~]$ srvctl status database -d mydb
Instance MYDB1 is running on node hst2
Instance MYDB2 is running on node hst1
[oracle@hst1 ~]$ srvctl status nodeapps -n hst1
VIP is running on node: hst1
GSD is running on node: hst1
Listener is running on node: hst1
ONS daemon is running on node: hst1
[oracle@hst2 ~]$ srvctl status database -d mydb
Instance MYDB1 is running on node hst2
Instance MYDB2 is running on node hst1
[oracle@hst2 ~]$ srvctl status nodeapps -n hst1
VIP is running on node: hst1
GSD is running on node: hst1
Listener is running on node: hst1
ONS daemon is running on node: hst1
[root@hst1 ~]# /appl/oraclu/product/10.2.0/crs/bin/crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....B1.inst application ONLINE ONLINE hst2
ora....B2.inst application ONLINE ONLINE hst1
ora....DB1.srv application ONLINE ONLINE hst2
ora....DB2.srv application ONLINE ONLINE hst1
ora....BTAF.cs application ONLINE ONLINE hst2
ora.MYDB.db application ONLINE ONLINE hst2
ora....B2.lsnr application ONLINE ONLINE hst2
ora.hst2.gsd application ONLINE ONLINE hst2
ora.hst2.ons application ONLINE ONLINE hst2
ora.hst2.vip application ONLINE ONLINE hst2
ora....B1.lsnr application ONLINE ONLINE hst1
ora.hst1.gsd application ONLINE ONLINE hst1
ora.hst1.ons application ONLINE ONLINE hst1
ora.hst1.vip application ONLINE ONLINE hst1
Everything is up. We are ready!
References
[1] http://forums.oracle.com/forums/thread.jspa?messageID=2161448
[2] Metalink Note:276434.1
https://metalink.oracle.com/metalink/plsql/showdoc?db=NOT&id=276434.1&blackframe=1
[3] Metalink Note:283684.1
https://metalink.oracle.com/metalink/plsql/showdoc?db=NOT&id=283684.1&blackframe=1
(Account is required for Metalink access)
Jaromir D.B. Nemec is a freelancer specializing in data warehouse and integration solutions. He can be reached under http://www.db-nemec.com
