I met with an oracle error ORA-19625 while taking the database backup in my test machine. I would like to share my troubleshooting.
RMAN> backup database plus archivelog;
Starting backup at 05-APR-13
current log archived
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=140 device type=DISK
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup plus archivelog command at 04/05/2013 17:09:53
RMAN-06059: expected archived log not found, loss of archived log compromises recoverability
ORA-19625: error identifying file /u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_351_8olgowfo_.arc
ORA-27037: unable to obtain file status
Linux Error: 2: No such file or directory
Additional information: 3
I suspected that might be the archivelogs and backup sets are not validated so I crosschecked them but still no luck.
RMAN> crosscheck archivelog all;
released channel: ORA_DISK_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=140 device type=DISK
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_351_8olgowfo_.arc RECID=17 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_352_8olgox9p_.arc RECID=18 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_353_8olgp25d_.arc RECID=19 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_354_8olgw53l_.arc RECID=15 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_355_8olh45f3_.arc RECID=20 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_356_8omfjxq7_.arc RECID=16 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_02/o1_mf_1_357_8on1b4k6_.arc RECID=12 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_02/o1_mf_1_358_8ondcmf8_.arc RECID=11 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_02/o1_mf_1_359_8ondcrm5_.arc RECID=10 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_02/o1_mf_1_360_8ondd0cc_.arc RECID=13 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_361_8ot1mnqt_.arc RECID=8 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_362_8ot1nwn6_.arc RECID=9 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_363_8ot3d6ow_.arc RECID=5 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_364_8ot3djhk_.arc RECID=3 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_365_8ot3ds6x_.arc RECID=6 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_366_8ot5fh4q_.arc RECID=4 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_367_8ot5jg1h_.arc RECID=2 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_1_8ovboj52_.arc RECID=7 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_2_8owys7k3_.arc RECID=22 STAMP=811947009
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_2_8ovckbp9_.arc RECID=1 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_3_8owys95t_.arc RECID=23 STAMP=811947010
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_3_8owm3vbc_.arc RECID=14 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_4_8owys7cf_.arc RECID=21 STAMP=811947007
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_1_8ox0bt2y_.arc RECID=24 STAMP=811948594
Crosschecked 24 objects
RMAN> crosscheck backup;
using channel ORA_DISK_1
specification does not match any backup in the repository
RMAN> backup database;
Starting backup at 05-APR-13
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00002 name=/u01/app/oracle/oradata/rman/sysaux01.dbf
input datafile file number=00001 name=/u01/app/oracle/oradata/rman/system01.dbf
input datafile file number=00005 name=/u01/app/oracle/oradata/rman/example01.dbf
input datafile file number=00003 name=/u01/app/oracle/oradata/rman/undotbs01.dbf
input datafile file number=00004 name=/u01/app/oracle/oradata/users01.dbf
channel ORA_DISK_1: starting piece 1 at 05-APR-13
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 04/05/2013 13:52:49
ORA-19502: write error on file "/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_mf_nnndf_TAG20130405T134814_8ox266ph_.bkp", block number 25344 (block size=8192)
ORA-27072: File I/O error
Additional information: 4
Additional information: 25344
Additional information: 991232
Solution.
When I checked the disk size of the FRA which reside on "/u01" partition, it was completely full.
[oracle@oracle autobackup]$ df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 142G 103G 32G 77% /
/dev/sda3 29G 15G 13G 53% /home
/dev/sda1 193M 12M 172M 7% /boot
tmpfs 3.0G 1.1G 2.0G 35% /dev/shm
/dev/sda6 29G 27G 218M 100% /u01
shmfs 3.0G 1.1G 2.0G 35% /dev/shm
Now the problem is cleared to me. I quickly clean up the disk and problem get resolved.
After clean up task.
[oracle@oracle backupset]$ df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 142G 103G 32G 77% /
/dev/sda3 29G 15G 13G 53% /home
/dev/sda1 193M 12M 172M 7% /boot
tmpfs 3.0G 1.1G 2.0G 35% /dev/shm
/dev/sda6 29G 24G 3.2G 89% /u01
shmfs 3.0G 1.1G 2.0G 35% /dev/shm
Since I deleted the backup sets from the OS level so I crosschecked the archivelog and backup sets from RMAN prompt.
RMAN> crosscheck archivelog all;
released channel: ORA_DISK_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=140 device type=DISK
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_351_8olgowfo_.arc RECID=17 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_352_8olgox9p_.arc RECID=18 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_353_8olgp25d_.arc RECID=19 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_354_8olgw53l_.arc RECID=15 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_355_8olh45f3_.arc RECID=20 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_356_8omfjxq7_.arc RECID=16 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_02/o1_mf_1_357_8on1b4k6_.arc RECID=12 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_02/o1_mf_1_358_8ondcmf8_.arc RECID=11 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_02/o1_mf_1_359_8ondcrm5_.arc RECID=10 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_02/o1_mf_1_360_8ondd0cc_.arc RECID=13 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_361_8ot1mnqt_.arc RECID=8 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_362_8ot1nwn6_.arc RECID=9 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_363_8ot3d6ow_.arc RECID=5 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_364_8ot3djhk_.arc RECID=3 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_365_8ot3ds6x_.arc RECID=6 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_366_8ot5fh4q_.arc RECID=4 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_367_8ot5jg1h_.arc RECID=2 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_1_8ovboj52_.arc RECID=7 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_2_8owys7k3_.arc RECID=22 STAMP=811947009
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_2_8ovckbp9_.arc RECID=1 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_3_8owys95t_.arc RECID=23 STAMP=811947010
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_3_8owm3vbc_.arc RECID=14 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_4_8owys7cf_.arc RECID=21 STAMP=811947007
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_1_8ox0bt2y_.arc RECID=24 STAMP=811948594
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_2_8ox1pfg3_.arc RECID=25 STAMP=811949989
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_3_8ox24dow_.arc RECID=26 STAMP=811950436
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_4_8oxg0886_.arc RECID=27 STAMP=811962592
Crosschecked 27 objects
RMAN> crosscheck backup;
using channel ORA_DISK_1
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_mf_annnn_TAG20130405T133949_8ox1pfrz_.bkp RECID=4 STAMP=811949989
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_mf_annnn_TAG20130405T133949_8ox1qtz6_.bkp RECID=5 STAMP=811950034
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_mf_annnn_TAG20130405T133949_8ox1rb7g_.bkp RECID=6 STAMP=811950050
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_mf_annnn_TAG20130405T134716_8ox24f25_.bkp RECID=7 STAMP=811950437
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_mf_annnn_TAG20130405T134716_8ox25j8y_.bkp RECID=8 STAMP=811950472
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_mf_annnn_TAG20130405T134716_8ox25qgj_.bkp RECID=9 STAMP=811950479
Crosschecked 6 objects
Now time to take the backup of them.
RMAN> backup database plus archivelog;
Starting backup at 05-APR-13
current log archived
using channel ORA_DISK_1
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=4 RECID=27 STAMP=811962592
input archived log thread=1 sequence=5 RECID=28 STAMP=811962769
input archived log thread=1 sequence=6 RECID=29 STAMP=811962772
input archived log thread=1 sequence=7 RECID=30 STAMP=811962776
input archived log thread=1 sequence=8 RECID=31 STAMP=811962796
channel ORA_DISK_1: starting piece 1 at 05-APR-13
channel ORA_DISK_1: finished piece 1 at 05-APR-13
piece handle=/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_mf_annnn_TAG20130405T171316_8oxg6n9f_.bkp tag=TAG20130405T171316 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 05-APR-13
Starting backup at 05-APR-13
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00002 name=/u01/app/oracle/oradata/rman/sysaux01.dbf
input datafile file number=00001 name=/u01/app/oracle/oradata/rman/system01.dbf
input datafile file number=00005 name=/u01/app/oracle/oradata/rman/example01.dbf
input datafile file number=00003 name=/u01/app/oracle/oradata/rman/undotbs01.dbf
input datafile file number=00004 name=/u01/app/oracle/oradata/users01.dbf
channel ORA_DISK_1: starting piece 1 at 05-APR-13
channel ORA_DISK_1: finished piece 1 at 05-APR-13
piece handle=/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_mf_nnndf_TAG20130405T171317_8oxg6oks_.bkp tag=TAG20130405T171317 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:01:05
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
including current SPFILE in backup set
channel ORA_DISK_1: starting piece 1 at 05-APR-13
channel ORA_DISK_1: finished piece 1 at 05-APR-13
piece handle=/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 05-APR-13
Starting backup at 05-APR-13
current log archived
using channel ORA_DISK_1
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=9 RECID=32 STAMP=811962865
channel ORA_DISK_1: starting piece 1 at 05-APR-13
channel ORA_DISK_1: finished piece 1 at 05-APR-13
piece handle=/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 05-APR-13
The backup has been completed successfully.
Cheers.. :)
RMAN> backup database plus archivelog;
Starting backup at 05-APR-13
current log archived
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=140 device type=DISK
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup plus archivelog command at 04/05/2013 17:09:53
RMAN-06059: expected archived log not found, loss of archived log compromises recoverability
ORA-19625: error identifying file /u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_351_8olgowfo_.arc
ORA-27037: unable to obtain file status
Linux Error: 2: No such file or directory
Additional information: 3
I suspected that might be the archivelogs and backup sets are not validated so I crosschecked them but still no luck.
RMAN> crosscheck archivelog all;
released channel: ORA_DISK_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=140 device type=DISK
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_351_8olgowfo_.arc RECID=17 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_352_8olgox9p_.arc RECID=18 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_353_8olgp25d_.arc RECID=19 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_354_8olgw53l_.arc RECID=15 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_355_8olh45f3_.arc RECID=20 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_356_8omfjxq7_.arc RECID=16 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_02/o1_mf_1_357_8on1b4k6_.arc RECID=12 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_02/o1_mf_1_358_8ondcmf8_.arc RECID=11 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_02/o1_mf_1_359_8ondcrm5_.arc RECID=10 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_02/o1_mf_1_360_8ondd0cc_.arc RECID=13 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_361_8ot1mnqt_.arc RECID=8 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_362_8ot1nwn6_.arc RECID=9 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_363_8ot3d6ow_.arc RECID=5 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_364_8ot3djhk_.arc RECID=3 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_365_8ot3ds6x_.arc RECID=6 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_366_8ot5fh4q_.arc RECID=4 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_367_8ot5jg1h_.arc RECID=2 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_1_8ovboj52_.arc RECID=7 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_2_8owys7k3_.arc RECID=22 STAMP=811947009
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_2_8ovckbp9_.arc RECID=1 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_3_8owys95t_.arc RECID=23 STAMP=811947010
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_3_8owm3vbc_.arc RECID=14 STAMP=811946863
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_4_8owys7cf_.arc RECID=21 STAMP=811947007
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_1_8ox0bt2y_.arc RECID=24 STAMP=811948594
Crosschecked 24 objects
RMAN> crosscheck backup;
using channel ORA_DISK_1
specification does not match any backup in the repository
RMAN> backup database;
Starting backup at 05-APR-13
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00002 name=/u01/app/oracle/oradata/rman/sysaux01.dbf
input datafile file number=00001 name=/u01/app/oracle/oradata/rman/system01.dbf
input datafile file number=00005 name=/u01/app/oracle/oradata/rman/example01.dbf
input datafile file number=00003 name=/u01/app/oracle/oradata/rman/undotbs01.dbf
input datafile file number=00004 name=/u01/app/oracle/oradata/users01.dbf
channel ORA_DISK_1: starting piece 1 at 05-APR-13
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 04/05/2013 13:52:49
ORA-19502: write error on file "/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_mf_nnndf_TAG20130405T134814_8ox266ph_.bkp", block number 25344 (block size=8192)
ORA-27072: File I/O error
Additional information: 4
Additional information: 25344
Additional information: 991232
Solution.
When I checked the disk size of the FRA which reside on "/u01" partition, it was completely full.
[oracle@oracle autobackup]$ df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 142G 103G 32G 77% /
/dev/sda3 29G 15G 13G 53% /home
/dev/sda1 193M 12M 172M 7% /boot
tmpfs 3.0G 1.1G 2.0G 35% /dev/shm
/dev/sda6 29G 27G 218M 100% /u01
shmfs 3.0G 1.1G 2.0G 35% /dev/shm
Now the problem is cleared to me. I quickly clean up the disk and problem get resolved.
After clean up task.
[oracle@oracle backupset]$ df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 142G 103G 32G 77% /
/dev/sda3 29G 15G 13G 53% /home
/dev/sda1 193M 12M 172M 7% /boot
tmpfs 3.0G 1.1G 2.0G 35% /dev/shm
/dev/sda6 29G 24G 3.2G 89% /u01
shmfs 3.0G 1.1G 2.0G 35% /dev/shm
Since I deleted the backup sets from the OS level so I crosschecked the archivelog and backup sets from RMAN prompt.
RMAN> crosscheck archivelog all;
released channel: ORA_DISK_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=140 device type=DISK
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_351_8olgowfo_.arc RECID=17 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_352_8olgox9p_.arc RECID=18 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_353_8olgp25d_.arc RECID=19 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_354_8olgw53l_.arc RECID=15 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_355_8olh45f3_.arc RECID=20 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_01/o1_mf_1_356_8omfjxq7_.arc RECID=16 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_02/o1_mf_1_357_8on1b4k6_.arc RECID=12 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_02/o1_mf_1_358_8ondcmf8_.arc RECID=11 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_02/o1_mf_1_359_8ondcrm5_.arc RECID=10 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_02/o1_mf_1_360_8ondd0cc_.arc RECID=13 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_361_8ot1mnqt_.arc RECID=8 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_362_8ot1nwn6_.arc RECID=9 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_363_8ot3d6ow_.arc RECID=5 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_364_8ot3djhk_.arc RECID=3 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_365_8ot3ds6x_.arc RECID=6 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_366_8ot5fh4q_.arc RECID=4 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_367_8ot5jg1h_.arc RECID=2 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_1_8ovboj52_.arc RECID=7 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_2_8owys7k3_.arc RECID=22 STAMP=811947009
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_04/o1_mf_1_2_8ovckbp9_.arc RECID=1 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_3_8owys95t_.arc RECID=23 STAMP=811947010
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_3_8owm3vbc_.arc RECID=14 STAMP=811946863
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_4_8owys7cf_.arc RECID=21 STAMP=811947007
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_1_8ox0bt2y_.arc RECID=24 STAMP=811948594
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_2_8ox1pfg3_.arc RECID=25 STAMP=811949989
validation failed for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_3_8ox24dow_.arc RECID=26 STAMP=811950436
validation succeeded for archived log
archived log file name=/u01/app/oracle/fast_recovery_area/RMAN/archivelog/2013_04_05/o1_mf_1_4_8oxg0886_.arc RECID=27 STAMP=811962592
Crosschecked 27 objects
RMAN> crosscheck backup;
using channel ORA_DISK_1
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_mf_annnn_TAG20130405T133949_8ox1pfrz_.bkp RECID=4 STAMP=811949989
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_mf_annnn_TAG20130405T133949_8ox1qtz6_.bkp RECID=5 STAMP=811950034
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_mf_annnn_TAG20130405T133949_8ox1rb7g_.bkp RECID=6 STAMP=811950050
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_mf_annnn_TAG20130405T134716_8ox24f25_.bkp RECID=7 STAMP=811950437
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_mf_annnn_TAG20130405T134716_8ox25j8y_.bkp RECID=8 STAMP=811950472
crosschecked backup piece: found to be 'EXPIRED'
backup piece handle=/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_mf_annnn_TAG20130405T134716_8ox25qgj_.bkp RECID=9 STAMP=811950479
Crosschecked 6 objects
Now time to take the backup of them.
RMAN> backup database plus archivelog;
Starting backup at 05-APR-13
current log archived
using channel ORA_DISK_1
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=4 RECID=27 STAMP=811962592
input archived log thread=1 sequence=5 RECID=28 STAMP=811962769
input archived log thread=1 sequence=6 RECID=29 STAMP=811962772
input archived log thread=1 sequence=7 RECID=30 STAMP=811962776
input archived log thread=1 sequence=8 RECID=31 STAMP=811962796
channel ORA_DISK_1: starting piece 1 at 05-APR-13
channel ORA_DISK_1: finished piece 1 at 05-APR-13
piece handle=/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_mf_annnn_TAG20130405T171316_8oxg6n9f_.bkp tag=TAG20130405T171316 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 05-APR-13
Starting backup at 05-APR-13
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00002 name=/u01/app/oracle/oradata/rman/sysaux01.dbf
input datafile file number=00001 name=/u01/app/oracle/oradata/rman/system01.dbf
input datafile file number=00005 name=/u01/app/oracle/oradata/rman/example01.dbf
input datafile file number=00003 name=/u01/app/oracle/oradata/rman/undotbs01.dbf
input datafile file number=00004 name=/u01/app/oracle/oradata/users01.dbf
channel ORA_DISK_1: starting piece 1 at 05-APR-13
channel ORA_DISK_1: finished piece 1 at 05-APR-13
piece handle=/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_mf_nnndf_TAG20130405T171317_8oxg6oks_.bkp tag=TAG20130405T171317 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:01:05
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
including current SPFILE in backup set
channel ORA_DISK_1: starting piece 1 at 05-APR-13
channel ORA_DISK_1: finished piece 1 at 05-APR-13
piece handle=/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 05-APR-13
Starting backup at 05-APR-13
current log archived
using channel ORA_DISK_1
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=9 RECID=32 STAMP=811962865
channel ORA_DISK_1: starting piece 1 at 05-APR-13
channel ORA_DISK_1: finished piece 1 at 05-APR-13
piece handle=/u01/app/oracle/fast_recovery_area/RMAN/backupset/2013_04_05/o1_
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 05-APR-13
The backup has been completed successfully.
Cheers.. :)
No comments:
Post a Comment