Thursday, September 25, 2014

Error CRS-5016, CRS-2674: While restarting the ACFS Resource in clusterware

I met with an issue where the ACFS resource was not started then I tried to start it manually but no luck. 
Following are the scenriao for the same.

 [oracle@rac1 bin]./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.FRA.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.asm
               ONLINE  ONLINE       rac1                     Started
               ONLINE  ONLINE       rac2                     Started
ora.gsd
               OFFLINE OFFLINE      rac1
               OFFLINE OFFLINE      rac2
ora.net1.network
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.ons
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.registry.acfs
               ONLINE  OFFLINE      rac1
               ONLINE  ONLINE       rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac2
ora.cvu
      1        ONLINE  ONLINE       rac2
ora.oc4j
      1        ONLINE  ONLINE       rac2
ora.rac.db
      1        ONLINE  ONLINE       rac1                     Open
      2        ONLINE  ONLINE       rac2                     Open
ora.rac1.vip
      1        ONLINE  ONLINE       rac1
ora.rac2.vip
      1        ONLINE  ONLINE       rac2
ora.scan1.vip
      1        ONLINE  ONLINE       rac2
Then I tried to start the ACFS resource.

 [oracle@rac1 bin]./crsctl start res ora.registry.acfs -n rac1
CRS-2672: Attempting to start 'ora.registry.acfs' on 'rac1'
CRS-5016: Process "/u02/grid/bin/acfsregistrymount" spawned by agent "/u02/grid/bin/orarootagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u02/grid/log/rac1/agent/crsd/orarootagent_root/orarootagent_root.log"
CRS-2674: Start of 'ora.registry.acfs' on 'rac1' failed
CRS-4000: Command Start failed, or completed with errors.

Excerpts from alert log file
/u02/grid/log/rac1/agent/crsd/orarootagent_root/orarootagent_root.log

 2014-09-25 15:28:45.469: [ora.registry.acfs][2594167696] {1:41591:460} [check] getOracleHomeAttrib: oracle_home = /u02/grid
2014-09-25 15:28:45.675: [ora.registry.acfs][2594167696] {1:41591:460} [check] (:CLSN00010:)ACFS-9127: Not all ADVM/ACFS drivers have been loaded.
2014-09-25 15:28:45.675: [ora.registry.acfs][2594167696] {1:41591:460} [check] (:CLSN00010:)
2014-09-25 15:28:45.675: [ora.registry.acfs][2594167696] {1:41591:460} [check] (:CLSN00010:)Utils:execCmd scls_process_join() uret 1

 2014-09-25 15:28:45.675: [ora.registry.acfs][2594167696] {1:41591:460} [check] execCmd ret = 1
2014-09-25 15:28:45.675: [ora.registry.acfs][2594167696] {1:41591:460} [check] (:CLSN00010:)ACFS-9127: Not all ADVM/ACFS drivers have been loaded.
2014-09-25 15:28:45.675: [ora.registry.acfs][2594167696] {1:41591:460} [check] (:CLSN00010:)
2014-09-25 15:28:45.676: [ora.registry.acfs][2594167696] {1:41591:460} [check] RegistryAcfsAgent:: Check type: CLSAGFW_CHECK_AFTER_CMD
2014-09-25 15:28:45.676: [ora.registry.acfs][2594167696] {1:41591:460} [check] RegistryAcfsAgent:: Check Return: 1
2014-09-25 15:28:45.676: [    AGFW][2766052240] {1:41591:460} ora.registry.acfs rac1 1 state changed from: STARTING to: OFFLINE


Solution.

Lets cross-check whether the ACFSDriver has been loaded or not.

[oracle@rac1 bin]./acfsdriverstate -help
ACFS-9206: usage: acfsdriverstate [-orahome <ORACLE_HOME>] <installed | loaded | version | supported> [-s]
[oracle@rac1 bin]./acfsdriverstate -orahome /u02/grid/ supported
ACFS-9200: Supported
[oracle@rac1 bin]./acfsdriverstate -orahome /u02/grid/ installed
ACFS-9203: true
[oracle@rac1 bin]./acfsdriverstate -orahome /u02/grid/ loaded
ACFS-9204: false

Now it is pretty obvious that ACFS Drivers are not loaded.

Lets start them under user 'root'.


[root@rac1 bin]# ./acfsload start
ACFS-9391: Checking for existing ADVM/ACFS installation.
ACFS-9392: Validating ADVM/ACFS installation files for operating system.
ACFS-9393: Verifying ASM Administrator setup.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9154: Loading 'oracleoks.ko' driver.
ACFS-9154: Loading 'oracleadvm.ko' driver.
ACFS-9154: Loading 'oracleacfs.ko' driver.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9322: completed


[root@rac1 bin]# ./acfsdriverstate -orahome /u02/grid/ loaded
ACFS-9203: true

Now the ACFS Drivers have been loaded.

 Lets start the ACFS Resource.

 [root@rac1 bin]# ./crsctl start res ora.registry.acfs -n rac1
CRS-2672: Attempting to start 'ora.registry.acfs' on 'rac1'
CRS-2676: Start of 'ora.registry.acfs' on 'rac1' succeeded

[root@rac1 bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.FRA.dg
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.asm
               ONLINE  ONLINE       rac1                     Started
               ONLINE  ONLINE       rac2                     Started
ora.gsd
               OFFLINE OFFLINE      rac1
               OFFLINE OFFLINE      rac2
ora.net1.network
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.ons
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
ora.registry.acfs
               ONLINE  ONLINE       rac1
               ONLINE  ONLINE       rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac2
ora.cvu
      1        ONLINE  ONLINE       rac2
ora.oc4j
      1        ONLINE  ONLINE       rac2
ora.rac.db
      1        ONLINE  ONLINE       rac1                     Open
      2        ONLINE  ONLINE       rac2                     Open
ora.rac1.vip
      1        ONLINE  ONLINE       rac1
ora.rac2.vip
      1        ONLINE  ONLINE       rac2
ora.scan1.vip
      1        ONLINE  ONLINE       rac2

Posted by at
Labels:

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)