Monday, February 3, 2014

OCR and Voting Disk Recovery in 11g RAC Env

[root@node1 bin]# cat /etc/oracle/ocr.loc
#Device/file +OCR_DISK getting replaced by device /u02/ocr.dat
ocrconfig_loc=/u02/ocr.dat


[root@node1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3096
         Available space (kbytes) :     259024
         ID                       :  867552287
         Device/File Name         : /u02/ocr.dat
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

[oracle@node1 script]$ cluvfy comp ocr -n node1 -verbose

Verifying OCR integrity

Checking OCR integrity...

Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations


Checking OCR config file "/etc/oracle/ocr.loc"...

OCR config file "/etc/oracle/ocr.loc" check successful


Checking OCR location "/u02/ocr.dat"...

Check for OCR location "/u02/ocr.dat" successful

Check for compatible storage device for OCR location "/u02/ocr.dat"...

Check for compatible storage device for OCR location "/u02/ocr.dat" is successful...

Checking OCR device "/u02/ocr.dat" for sharedness...

OCR device "/u02/ocr.dat" is shared...


Checking size of the OCR location "/u02/ocr.dat" ...

Size check for OCR location "/u02/ocr.dat" successful...

NOTE:
This check does not verify the integrity of the OCR contents. Execute 'ocrcheck' as a privileged user to verify the contents of OCR.

OCR integrity check passed

Verification of OCR integrity was successful.



[root@node1 bin]# ./ocrconfig -showbackup

node1     2014/02/03 14:34:33     /u01/grid/cdata/node-scan/backup00.ocr

node1     2014/02/03 10:34:32     /u01/grid/cdata/node-scan/backup01.ocr

node1     2014/02/03 06:34:31     /u01/grid/cdata/node-scan/backup02.ocr

node1     2014/02/02 06:34:25     /u01/grid/cdata/node-scan/day.ocr

node1     2014/01/24 18:33:39     /u01/grid/cdata/node-scan/week.ocr

node2     2013/11/07 21:55:22     /u01/grid/cdata/node-scan/backup_20131107_215522.ocr

node2     2013/11/07 21:54:20     /u01/grid/cdata/node-scan/backup_20131107_215420.ocr

node2     2013/10/30 01:05:47     /u01/grid/cdata/node-scan/backup_20131030_010547.ocr

node2     2013/10/29 23:34:07     /u01/grid/cdata/node-scan/backup_20131029_233407.ocr
[root@node1 bin]# ./crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   813d3dffc3d44f94bfdb79a9ccdeb547 (/u02/vote.dat) []
 2. ONLINE   9ed751ab010b4fe5bfa5924111dd0c7e (/u02/vote01.dat) []
Located 2 voting disk(s).


----------------------------------------------------------



[oracle@node1 script]$ mv /u02/ocr.dat /u02/ocr_old.dat
[oracle@node1 script]$ ls -al /u02/ocr.dat
ls: /u02/ocr.dat: No such file or directory

[oracle@node1 script]$ mv /u02/vote.dat /u02/vote_old.dat
[oracle@node1 script]$ mv /u02/vote01.dat /u02/vote01_old.dat







-----------------------------------------------------------

[oracle@node1 script]$ ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]
[oracle@node1 script]$ cluvfy comp ocr -n node1 -verbose

Verifying OCR integrity

Checking OCR integrity...

Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations


Checking OCR config file "/etc/oracle/ocr.loc"...

OCR config file "/etc/oracle/ocr.loc" check successful


Checking OCR location "/u02/ocr.dat"...

ERROR:
PRVF-4178 : Check for OCR location "/u02/ocr.dat" failed on the following nodes:
        node1

        node1:No such file or directory
Check for compatible storage device for OCR location "/u02/ocr.dat"...


ERROR:
PRVF-10031 : OCR location "/u02/ocr.dat" is not on a compatible storage device for OCR ...

NOTE:
This check does not verify the integrity of the OCR contents. Execute 'ocrcheck' as a privileged user to verify the contents of OCR.

OCR integrity check failed

Verification of OCR integrity was unsuccessful on all the specified nodes.

-------------------------------------------------------------------



[root@node1 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@node1 bin]# ./crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node1'
CRS-2673: Attempting to stop 'ora.crsd' on 'node1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'node1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'node1'
CRS-2673: Attempting to stop 'ora.node1.vip' on 'node1'
CRS-2673: Attempting to stop 'ora.BACKUP.dg' on 'node1'
CRS-2673: Attempting to stop 'ora.OCR_DISK.dg' on 'node1'
CRS-2673: Attempting to stop 'ora.VDISK_DG.dg' on 'node1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'node1'
CRS-2673: Attempting to stop 'ora.node.db' on 'node1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'node1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'node1'
CRS-2673: Attempting to stop 'ora.cvu' on 'node1'
CRS-2677: Stop of 'ora.node1.vip' on 'node1' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'node1'
CRS-2677: Stop of 'ora.scan1.vip' on 'node1' succeeded
CRS-2677: Stop of 'ora.cvu' on 'node1' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.node2.vip' on 'node1'
CRS-2677: Stop of 'ora.node2.vip' on 'node1' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'node1' succeeded
CRS-2677: Stop of 'ora.node.db' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'node1'
CRS-2677: Stop of 'ora.BACKUP.dg' on 'node1' succeeded
CRS-2677: Stop of 'ora.OCR_DISK.dg' on 'node1' succeeded
CRS-2677: Stop of 'ora.VDISK_DG.dg' on 'node1' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'node1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'node1'
CRS-2677: Stop of 'ora.asm' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'node1'
CRS-2677: Stop of 'ora.ons' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'node1'
CRS-2677: Stop of 'ora.net1.network' on 'node1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'node1' has completed
CRS-2677: Stop of 'ora.crsd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'node1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'node1'
CRS-2673: Attempting to stop 'ora.evmd' on 'node1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'node1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'node1'
CRS-2677: Stop of 'ora.mdnsd' on 'node1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'node1' succeeded
CRS-2677: Stop of 'ora.asm' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'node1'
CRS-2677: Stop of 'ora.ctssd' on 'node1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'node1'
CRS-2677: Stop of 'ora.cssd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'node1'
CRS-2677: Stop of 'ora.gipcd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'node1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'node1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'node1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node1' has completed
CRS-4133: Oracle High Availability Services has been stopped.





[root@node1 bin]# ./crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'node1'
CRS-2676: Start of 'ora.mdnsd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'node1'
CRS-2676: Start of 'ora.gpnpd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'node1'
CRS-2672: Attempting to start 'ora.gipcd' on 'node1'
CRS-2676: Start of 'ora.cssdmonitor' on 'node1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'node1'
CRS-2672: Attempting to start 'ora.diskmon' on 'node1'
CRS-2676: Start of 'ora.diskmon' on 'node1' succeeded
CRS-2676: Start of 'ora.cssd' on 'node1' succeeded




[root@node1 bin]# ls -lhtr /u02/ocr.dat
-rw-r--r-- 1 root root 261M Feb  3 19:16 /u02/ocr.dat
[root@node1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2992
         Available space (kbytes) :     259128
         ID                       :  867552287
         Device/File Name         : /u02/ocr.dat
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

[root@node1 bin]# ./cluvfy comp ocr -n node1 -verbose
You must NOT be logged in as root (uid=0) when running ./cluvfy.
[root@node1 bin]# su - oracle

[oracle@node1 ~]$ cluvfy comp ocr -n node1 -verbose

Verifying OCR integrity

Checking OCR integrity...

Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations


Checking OCR config file "/etc/oracle/ocr.loc"...

OCR config file "/etc/oracle/ocr.loc" check successful


Checking OCR location "/u02/ocr.dat"...

ERROR:
PRVF-4178 : Check for OCR location "/u02/ocr.dat" failed on the following nodes:
        node1

        node1:Group of file "/u02/ocr.dat" did not match the expected value. [Expected = "oinstall" ; Found = "root"];Permissions of file "/u02/ocr.dat" did not match the expected value. [Expected = "0640" ; Found = "0644"]
Check for compatible storage device for OCR location "/u02/ocr.dat"...

Check for compatible storage device for OCR location "/u02/ocr.dat" is successful...

Checking OCR device "/u02/ocr.dat" for sharedness...

OCR device "/u02/ocr.dat" is shared...


Checking size of the OCR location "/u02/ocr.dat" ...

Size check for OCR location "/u02/ocr.dat" successful...

NOTE:
This check does not verify the integrity of the OCR contents. Execute 'ocrcheck' as a privileged user to verify the contents of OCR.

OCR integrity check failed

Verification of OCR integrity was unsuccessful on all the specified nodes.


[root@node1 bin]# chown root.oinstall /u02/ocr.dat
[root@node1 bin]# chmod 0640 /u02/ocr.dat


[root@node1 bin]# su - oracle
[oracle@node1 ~]$ cluvfy comp ocr -n node1 -verbose

Verifying OCR integrity

Checking OCR integrity...

Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations


Checking OCR config file "/etc/oracle/ocr.loc"...

OCR config file "/etc/oracle/ocr.loc" check successful


Checking OCR location "/u02/ocr.dat"...

Check for OCR location "/u02/ocr.dat" successful

Check for compatible storage device for OCR location "/u02/ocr.dat"...

Check for compatible storage device for OCR location "/u02/ocr.dat" is successful...

Checking OCR device "/u02/ocr.dat" for sharedness...

OCR device "/u02/ocr.dat" is shared...


Checking size of the OCR location "/u02/ocr.dat" ...

Size check for OCR location "/u02/ocr.dat" successful...

NOTE:
This check does not verify the integrity of the OCR contents. Execute 'ocrcheck' as a privileged user to verify the contents of OCR.

OCR integrity check passed

Verification of OCR integrity was successful.
[oracle@node1 ~]$




[root@node1 bin]# ./crsctl query css votedisk
Located 0 voting disk(s).

[root@node1 bin]# ./crsctl replace votedisk /u02/vote01.dat
CRS-4263: This operation is not allowed
CRS-4000: Command Replace failed, or completed with errors.

[root@node1 bin]# ./crsctl add css votedisk /u02/vote01.dat
Now formatting voting disk: /u02/vote01.dat.
CRS-4603: Successful addition of voting disk /u02/vote01.dat.

[root@node1 bin]# ./crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   3318e4665e104f8abf9baff060d0ee04 (/u02/vote01.dat) []
Located 1 voting disk(s).
[root@node1 bin]#



[root@node1 bin]# ./crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node1'
CRS-2673: Attempting to stop 'ora.cssd' on 'node1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'node1'
CRS-2677: Stop of 'ora.cssd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'node1'
CRS-2677: Stop of 'ora.mdnsd' on 'node1' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'node1'
CRS-2677: Stop of 'ora.gpnpd' on 'node1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@node1 bin]# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

[root@node1 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

No comments:

Post a Comment