0412 Afd Corrupted

[grid@hbyd-sds-db5 dev]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE hbyd-sds-db5 STABLE
ora.DATA.dg
ora.KFZC.dg
ONLINE INTERMEDIATE hbyd-sds-db5 STABLE
ora.LISTENER.lsnr
ora.OCR.dg
ora.chad
ONLINE OFFLINE hbyd-sds-db5 STABLE
ora.net1.network
ora.ons
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE hbyd-sds-db5 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE hbyd-sds-db6 169.254.252.195 192.
168.14.81,STABLE
ora.asm
1 ONLINE ONLINE hbyd-sds-db5 Started,STABLE
2 ONLINE ONLINE hbyd-sds-db6 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
ora.gisdb.db
1 ONLINE ONLINE hbyd-sds-db5 Open,HOME=/u01/app/o
racle/product/12.2.0
/dbhome,STABLE
/dbhome,STABLE
ora.hbyd-sds-db5.vip
ora.hbyd-sds-db6.vip
ora.kfzc.db
/dbhome,STABLE
/dbhome,STABLE
ora.mgmtdb
1 ONLINE ONLINE hbyd-sds-db6 Open,STABLE
ora.qosmserver
ora.scan1.vip
ora.yyxn.db
1 ONLINE OFFLINE hbyd-sds-db5 Instance Shutdown,ST
ARTING
2 ONLINE OFFLINE Instance Shutdown,ST
ABLE
--------------------------------------------------------------------------------
[grid@hbyd-sds-db5 dev]$ asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'hbyd-
sds-db5'
[grid@hbyd-sds-db5 dev]$ asmcmd dsget

parameter:/dev/asm*, AFD:*
profile:/dev/asm*,AFD:*
[grid@hbyd-sds-db5 dev]$ asmcmd dsset afd:*

ORA-02097: parameter cannot be modified because specified value is invalid
ORA-15014: path 'AFD:DATA8' is not in the discovery set (DBD ERROR: OCIStmtExecute)
[grid@hbyd-sds-db5 dev]$ asmcmd afd_lsdsk

--------------------------------------------------------------------------------
Label Filtering Path
================================================================================
DATA1 ENABLED /dev/asmdisk01
OCR1 ENABLED /dev/asmdisk09
[grid@hbyd-sds-db5 dev]$ kfed read /dev/asmdisk08|grep kfdhdb.grpname

kfdhdb.grpname: DATA ; 0x048: length=4
[grid@hbyd-sds-db5 dev]$ asmcmd lsdsk -G data

Path
AFD:DATA1
AFD:DATA2
AFD:DATA3
AFD:DATA4
AFD:DATA5
AFD:DATA6
AFD:DATA7
AFD:DATA8
[grid@hbyd-sds-db5 dev]$ asmcmd afd_scan

[grid@hbyd-sds-db5 dev]$ asmcmd afd_lsdsk
--------------------------------------------------------------------------------
Label Filtering Path
================================================================================
OCR1 ENABLED /dev/asmdisk09
[grid@hbyd-sds-db5 trace]$ cd /dev/oracleafd/disks/

[grid@hbyd-sds-db5 disks]$ ls -l
total 52
-rwxrwx--- 1 grid oinstall 15 Apr 3 2018 DATA1
-rwxrwx--- 1 grid oinstall 15 May 29 2018 DATA10
-rwxrwx--- 1 grid oinstall 15 Apr 3 2018 OCR1
[grid@hbyd-sds-db5 by-label]$ ls -l
total 0
lrwxrwxrwx 1 root root 12 Apr 12 15:42 DATA1 -> ../../scinib
lrwxrwxrwx 1 root root 12 Apr 12 15:42 DATA10 -> ../../scinik
lrwxrwxrwx 1 root root 12 Apr 12 15:42 DATA11 -> ../../scinil
lrwxrwxrwx 1 root root 12 Apr 12 15:42 DATA12 -> ../../scinim
lrwxrwxrwx 1 root root 12 Apr 12 15:42 DATA2 -> ../../scinic
lrwxrwxrwx 1 root root 12 Apr 12 15:42 DATA3 -> ../../scinid
lrwxrwxrwx 1 root root 12 Apr 12 15:42 DATA4 -> ../../scinie
lrwxrwxrwx 1 root root 12 Apr 12 15:42 DATA5 -> ../../scinif
lrwxrwxrwx 1 root root 12 Apr 12 15:42 DATA6 -> ../../scinig
lrwxrwxrwx 1 root root 12 Apr 12 15:42 DATA7 -> ../../scinih
lrwxrwxrwx 1 root root 12 Apr 12 15:42 DATA8 -> ../../scinia
lrwxrwxrwx 1 root root 12 Apr 12 15:42 DATA9 -> ../../scinij
lrwxrwxrwx 1 root root 12 Apr 12 15:42 OCR1 -> ../../scinii
# db alert log
[grid@hbyd-sds-db5 trace]$ tail -n 400 alert*.log|more
NOTE: Assigning number (1,3) to disk (AFD:DATA4)
SUCCESS: mounted group 1 (DATA)
NOTE: grp 1 disk 7: DATA8 path:AFD:DATA8
2019-04-12T15:30:21.957724+08:00
ALTER SYSTEM SET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=10.216.11.82)
(PORT=1521))' SCOPE=MEMORY SID='yyxn1';
2019-04-12T15:30:22.041511+08:00
NOTE: dependency between database yyxn and diskgroup resource ora.DATA.dg is
established
2019-04-12T15:30:22.091027+08:00
KSXPPING: KSXP selected for Ping
ALTER DATABASE MOUNT /* db agent *//* {1:65096:944} */
2019-04-12T15:30:22.281844+08:00
This instance was first to mount
2019-04-12T15:30:26.371476+08:00
Network throttle feature is disabled as mount time
2019-04-12T15:30:26.462076+08:00
Successful mount of redo thread 1, with mount id 3621090958
2019-04-12T15:30:26.647785+08:00
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Lost write protection disabled
Using STANDBY_ARCHIVE_DEST parameter default value as /backup/db5_arc/yyxn1arc
Completed: ALTER DATABASE MOUNT /* db agent *//* {1:65096:944} */
ALTER DATABASE OPEN /* db agent *//* {1:65096:944} */
This instance was first to open
Ping without log force is disabled:
not an Exadata system.
2019-04-12T15:30:27.468965+08:00
start recovery: pdb 0, passed in flags x4 (domain enable 0)
2019-04-12T15:30:27.469173+08:00
Beginning crash recovery of 2 threads
parallel recovery started with 32 processes
2019-04-12T15:30:27.585764+08:00
2019-04-12T15:30:27.586133+08:00
2019-04-12T15:30:27.586348+08:00
2019-04-12T15:30:27.586850+08:00
2019-04-12T15:30:27.587004+08:00
2019-04-12T15:30:27.587017+08:00
2019-04-12T15:30:27.587090+08:00
2019-04-12T15:30:27.587086+08:00
2019-04-12T15:30:27.587092+08:00
2019-04-12T15:30:27.587197+08:00
2019-04-12T15:30:27.587283+08:00
2019-04-12T15:30:27.587455+08:00
2019-04-12T15:30:27.587828+08:00
2019-04-12T15:30:27.587848+08:00
2019-04-12T15:30:27.587851+08:00
2019-04-12T15:30:27.587855+08:00
2019-04-12T15:30:27.587935+08:00
2019-04-12T15:30:27.588142+08:00
2019-04-12T15:30:27.588156+08:00
2019-04-12T15:30:27.588228+08:00
2019-04-12T15:30:27.588615+08:00
2019-04-12T15:30:27.588622+08:00
2019-04-12T15:30:27.588671+08:00
2019-04-12T15:30:27.588686+08:00
2019-04-12T15:30:27.588723+08:00
2019-04-12T15:30:27.588809+08:00
2019-04-12T15:30:27.588853+08:00
2019-04-12T15:30:27.589020+08:00
2019-04-12T15:30:27.589057+08:00
2019-04-12T15:30:27.589251+08:00
2019-04-12T15:30:27.589254+08:00
2019-04-12T15:30:27.589471+08:00
2019-04-12T15:30:28.005220+08:00
Started redo scan
2019-04-12T15:30:52.458691+08:00
ARCH: STARTING ARCH PROCESSES
Starting background process ARC0
2019-04-12T15:30:52.477328+08:00
ARC0 started with pid=9, OS id=3446
2019-04-12T15:30:52.494610+08:00
2019-04-12T15:30:52.511579+08:00
ARC0: Archival started
2019-04-12T15:30:52.527886+08:00
2019-04-12T15:30:52.528405+08:00
ARC2: Becoming a 'no FAL' ARCH
ARC2: Becoming the 'no SRL' ARCH
2019-04-12T15:30:52.538409+08:00
ARCH: STARTING ARCH PROCESSES COMPLETE
2019-04-12T15:30:52.560371+08:00
TT00: Gap Manager starting (PID:3454)
2019-04-12T15:31:19.197168+08:00
Decreasing number of high priority LMS from 4 to 0
2019-04-12T15:31:31.199969+08:00
AFDLIB IO Failed @ 0x7fdc84016698: issync=0 que=comp err=-5 sts=0x2e
dsk=0x890df0cd00000007 fst=3350926
AFDLIB IO Failed @ 0x7fdc89cb0588: issync=0 que=comp err=-5 sts=0x2e
dsk=0x890df0cd00000007 fst=3350528
2019-04-12T15:31:31.201139+08:00
Errors in file /u01/app/oracle/diag/rdbms/yyxn/yyxn1/trace/yyxn1_ora_64316.trc:
2019-04-12T15:31:31.201428+08:00
WARNING: Read Failed. group:1 disk:6 AU:409 offset:203776 size:1048576
path:AFD:DATA7
incarnation:0x3018cc68 asynchronous result:'I/O error'
subsys:/opt/oracle/extapi/64/asm/orcl/1/libafd12.so krq:0x7fdc84016508
bufp:0x7fdc81238000 osderr1:0xfffffffb osderr
2:0x2e
IO elapsed time: 21924000 usec Time waited on I/O: 0 usec
2019-04-12T15:31:31.202048+08:00
ORA-15186: ASMLIB error function = [kfk_asm_ioerror], error =
[18446744073709551611], mesg = [I/O error]
path:AFD:DATA7
subsys:/opt/oracle/extapi/64/asm/orcl/1/libafd12.so krq:0x7fdc89cb03f8
2:0x2e
NOTE: Suppressing further IO Read errors on group:1 disk:6
WARNING: failed to read mirror side 1 of virtual extent 176 logical extent 0 of
file 266 in group [1.377833179] from disk DAT
A7 allocation unit 409 reason error; if possible, will try another mirror side
2019-04-12T15:31:31.204400+08:00
ORA-00333: redo log read error block 1440142 count 6144
ORA-00312: online log 1 thread 1: '+DATA/YYXN/ONLINELOG/group_1.266.959084603'
ORA-15081: failed to submit an I/O operation to a disk
[18446744073709551611], mesg = [I/O error]
[18446744073709551611], mesg = [I/O error]
2019-04-12T15:31:53.129173+08:00
AFDLIB IO Failed @ 0x7fdc8143c4b8: issync=1 que=reqs err=-5 sts=0x2e
dsk=0x890df0cd00000007 fst=3350528
AFDLIB IO Failed @ 0x7fdc8143c4b8: issync=1 que=wait err=-5 sts=0x2e
dsk=0x890df0cd00000007 fst=3350528
2019-04-12T15:31:53.129490+08:00
2019-04-12T15:31:53.130035+08:00
[18446744073709551611], mesg = [I/O error]
2019-04-12T15:31:53.130373+08:00
Incomplete read from log member '+DATA/YYXN/ONLINELOG/group_1.266.959084603'.
Trying next member.
AFDLIB IO Failed @ 0x7fdc840160f0: issync=0 que=comp err=-5 sts=0x2e
dsk=0x890df0cd00000005 fst=3350926
dsk=0x890df0cd00000005 fst=3350528
2019-04-12T15:31:53.578536+08:00
path:AFD:DATA3
subsys:/opt/oracle/extapi/64/asm/orcl/1/libafd12.so krq:0x7fdc84015f60
bufp:0x7fdc80e38000 osderr1:0xfffffffb osderr
2:0x2e
2019-04-12T15:31:53.579231+08:00
[18446744073709551611], mesg = [I/O error]
path:AFD:DATA3
bufp:0x7fdc80e06400 osderr1:0xfffffffb osderr
2:0x2e
2019-04-12T15:31:53.580815+08:00
[18446744073709551611], mesg = [I/O error]
[18446744073709551611], mesg = [I/O error]
2019-04-12T15:33:55.053693+08:00
AFDLIB IO Failed @ 0x7fdc84016c40: issync=0 que=comp err=-5 sts=0x2e
dsk=0x890df0cd00000003 fst=483328
2019-04-12T15:33:55.053849+08:00
2019-04-12T15:33:55.054022+08:00
path:AFD:DATA2
subsys:/opt/oracle/extapi/64/asm/orcl/1/libafd12.so krq:0x7fdc84016ab0
bufp:0x7fdc81437c00 osderr1:0xfffffffb osderr
2:0x2e
WARNING: failed to read mirror side 1 of virtual extent 84 logical extent 0 of file
265 in group [1.377833179] from disk DATA
2 allocation unit 59 reason error; if possible, will try another mirror side
2019-04-12T15:33:55.055817+08:00
[18446744073709551611], mesg = [I/O error]
2019-04-12T15:34:16.981686+08:00
AFDLIB IO Failed @ 0x7fdc89c91aa8: issync=1 que=reqs err=-5 sts=0x2e
dsk=0x890df0cd00000003 fst=483328
AFDLIB IO Failed @ 0x7fdc89c91aa8: issync=1 que=wait err=-5 sts=0x2e
dsk=0x890df0cd00000003 fst=483328
2019-04-12T15:34:16.981970+08:00
2019-04-12T15:34:16.982119+08:00
path:AFD:DATA2
incarnation:0x3018cc64 synchronous result:'I/O error'
subsys:/opt/oracle/extapi/64/asm/orcl/1/libafd12.so krq:0x7fdc89c91918
2:0x2e
2019-04-12T15:34:16.983266+08:00
[18446744073709551611], mesg = [I/O error]
Trying next member.
AFDLIB IO Failed @ 0x7fdc84015b48: issync=0 que=comp err=-5 sts=0x2e
dsk=0x890df0cd00000003 fst=483330
2019-04-12T15:34:17.444867+08:00
2019-04-12T15:34:17.445660+08:00
[18446744073709551611], mesg = [I/O error]
2019-04-12T15:34:44.509653+08:00
dsk=0x890df0cd00000009 fst=7544832
2019-04-12T15:34:44.509809+08:00
2019-04-12T15:34:44.509968+08:00
path:AFD:DATA4
2:0x2e
2019-04-12T15:34:44.513075+08:00
[18446744073709551611], mesg = [I/O error]
2019-04-12T15:35:06.435689+08:00
AFDLIB IO Failed @ 0x7fdc89c91aa8: issync=1 que=reqs err=-5 sts=0x2e
dsk=0x890df0cd00000009 fst=7544832
AFDLIB IO Failed @ 0x7fdc89c91aa8: issync=1 que=wait err=-5 sts=0x2e
dsk=0x890df0cd00000009 fst=7544832
2019-04-12T15:35:06.435972+08:00
2019-04-12T15:35:06.436117+08:00
path:AFD:DATA4
subsys:/opt/oracle/extapi/64/asm/orcl/1/libafd12.so krq:0x7fdc89c91918
2:0x2e
2019-04-12T15:35:06.437640+08:00
[18446744073709551611], mesg = [I/O error]
Trying next member.
2019-04-12T15:35:28.365672+08:00
AFDLIB IO Failed @ 0x7fdc8143d518: issync=1 que=reqs err=-5 sts=0x2e
dsk=0x890df0cd00000005 fst=483330
AFDLIB IO Failed @ 0x7fdc8143d518: issync=1 que=wait err=-5 sts=0x2e
dsk=0x890df0cd00000005 fst=483330
2019-04-12T15:35:28.365959+08:00
2019-04-12T15:35:28.366109+08:00
path:AFD:DATA3
subsys:/opt/oracle/extapi/64/asm/orcl/1/libafd12.so krq:0x7fdc8143d388
2:0x2e
2019-04-12T15:35:28.367195+08:00
[18446744073709551611], mesg = [I/O error]
Trying next member.
AFDLIB IO Failed @ 0x7fdc84016698: issync=0 que=comp err=-5 sts=0x2e
dsk=0x890df0cd00000009 fst=7544834
2019-04-12T15:35:28.367765+08:00
path:AFD:DATA4
subsys:/opt/oracle/extapi/64/asm/orcl/1/libafd12.so krq:0x7fdc84016508
2:0x2e
2019-04-12T15:35:28.374394+08:00
Slave encountered ORA-10388 exception during crash recovery
2019-04-12T15:35:28.374442+08:00
2019-04-12T15:35:28.374602+08:00
2019-04-12T15:35:28.374654+08:00
[grid@hbyd-sds-db5 by-label]$ asmcmd -p

ASMCMD [+] > cd +DATA/YYXN/ONLINELOG
ASMCMD [+DATA/YYXN/ONLINELOG] > ls -l
Type Redund Striped Time Sys Name
ONLINELOG UNPROT COARSE APR 12 16:00:00 Y group_1.263.959084607
ASMCMD [+DATA/YYXN/ONLINELOG] >
ASMCMD [+DATA/YYXN/ONLINELOG] > cp group_1.266.959084603 /tmp/group_1.266.959084603

copying +DATA/YYXN/ONLINELOG/group_1.266.959084603 -> /tmp/group_1.266.959084603
ASMCMD-8016: copy source '+DATA/YYXN/ONLINELOG/group_1.266.959084603' and target
'/tmp/group_1.266.959084603' failed
ORA-19501: read error on file "+DATA/YYXN/ONLINELOG/group_1.266.959084603", block
number 1439745 (block size=512)
[18446744073709551611], mesg = [I/O error]
ORA-15120: ASM file name '/tmp/group_1.266.959084603' does not begin with the ASM
prefix character
ORA-06512: at "SYS.X$DBMS_DISKGROUP", line 603
ORA-06512: at line 3 (DBD ERROR: OCIStmtExecute)
ASMCMD [+DATA/YYXN/CONTROLFILE] > cp Current.261.958895763 /tmp/ctl.dbf

copying +DATA/YYXN/CONTROLFILE/Current.261.958895763 -> /tmp/ctl.dbf
ASMCMD [+DATA/YYXN/CONTROLFILE] >
ASMCMD [+DATA/YYXN/ONLINELOG] > cp group_9.304.959085021 /tmp/redo9.dbf

copying +DATA/YYXN/ONLINELOG/group_9.304.959085021 -> /tmp/redo9.dbf
ASMCMD [+DATA/YYXN/ONLINELOG] >



'/tmp/redo9.dbf' failed
[18446744073709551611], mesg = [I/O error]
ORA-15120: ASM file name '/tmp/redo9.dbf' does not begin with the ASM prefix
character
ASMCMD> cp group_2.264.959084969 /tmp/redo9.dbf

copying +data/yyxn/onlinelog/group_2.264.959084969 -> /tmp/redo9.dbf
ASMCMD-8016: copy source '+data/yyxn/onlinelog/group_2.264.959084969' and target
ORA-19501: read error on file "+data/yyxn/onlinelog/group_2.264.959084969", block
[18446744073709551611], mesg = [I/O error]
character
ASMCMD> cp group_2.265.959084965 /tmp/redo9.dbf

copying +data/yyxn/onlinelog/group_2.265.959084965 -> /tmp/redo9.dbf
ASMCMD-8016: copy source '+data/yyxn/onlinelog/group_2.265.959084965' and target
ORA-19501: read error on file "+data/yyxn/onlinelog/group_2.265.959084965", block
[18446744073709551611], mesg = [I/O error]
character

[18446744073709551611], mesg = [I/O error]
character

1* select group#,member,status from v$logfile
GROUP# MEMBER STATUS

---------- -------------------------------------------------- -------
2 +DATA/YYXN/ONLINELOG/group_2.265.959084965 <<<<
1 +DATA/YYXN/ONLINELOG/group_1.263.959084607
3 +DATA/YYXN/ONLINELOG/group_3.270.959084791 STALE <<<<
24 rows selected.
SQL> alter database drop logfile member

'+DATA/YYXN/ONLINELOG/group_1.266.959084603';
Database altered.
SQL> alter database drop logfile member

'+DATA/YYXN/ONLINELOG/group_3.270.959084791';
alter database drop logfile member '+DATA/YYXN/ONLINELOG/group_3.270.959084791'
*
ERROR at line 1:
ORA-01609: log 3 is the current log for thread 1 - cannot drop members
SQL> select group#,status,thread# from v$log;
GROUP# STATUS THREAD#

---------- ---------------- ----------
1 ACTIVE 1
2 ACTIVE 1
3 CURRENT 1
4 INACTIVE 1
5 INACTIVE 1
6 INACTIVE 1
7 INACTIVE 2
8 INACTIVE 2
9 INACTIVE 2
10 CURRENT 2
11 INACTIVE 2
12 INACTIVE 2
12 rows selected.
SQL> alter database clear unarchived logfile group 2;

alter database clear unarchived logfile group 2
*
ERROR at line 1:
ORA-01624: log 2 needed for crash recovery of instance yyxn1 (thread 1)
[/bin/bash]$tail -n 10000 alert_yyxn1.log|grep ORA-00312|sort|uniq

0412 Afd Corrupted

Transféré par

Informations du document

Description originale:

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

0412 Afd Corrupted

Transféré par

Droits d'auteur :

Formats disponibles

[grid@hbyd-sds-db5 dev]$ crsctl stat res -t

[grid@hbyd-sds-db5 dev]$ asmcmd dsget

[grid@hbyd-sds-db5 dev]$ asmcmd dsset afd:*

[grid@hbyd-sds-db5 dev]$ asmcmd afd_lsdsk

[grid@hbyd-sds-db5 dev]$ kfed read /dev/asmdisk08|grep kfdhdb.grpname

[grid@hbyd-sds-db5 dev]$ asmcmd lsdsk -G data

[grid@hbyd-sds-db5 dev]$ asmcmd afd_scan

[grid@hbyd-sds-db5 trace]$ cd /dev/oracleafd/disks/

[grid@hbyd-sds-db5 by-label]$ asmcmd -p

ASMCMD [+DATA/YYXN/ONLINELOG] > cp group_1.266.959084603 /tmp/group_1.266.959084603

ASMCMD [+DATA/YYXN/CONTROLFILE] > cp Current.261.958895763 /tmp/ctl.dbf

ASMCMD [+DATA/YYXN/ONLINELOG] > cp group_9.304.959085021 /tmp/redo9.dbf

ASMCMD [+DATA/YYXN/ONLINELOG] > cp group_9.304.959085021 /tmp/redo9.dbf

ASMCMD [+DATA/YYXN/ONLINELOG] > cp group_1.263.959084607 /tmp/redo9.dbf

ASMCMD [+DATA/YYXN/ONLINELOG] > cp group_1.266.959084603 /tmp/redo9.dbf

ASMCMD> cp group_2.264.959084969 /tmp/redo9.dbf

ASMCMD> cp group_2.265.959084965 /tmp/redo9.dbf

ASMCMD [+DATA/YYXN/ONLINELOG] > cp group_3.270.959084791 /tmp/redo9.dbf

ASMCMD [+DATA/YYXN/ONLINELOG] > cp group_3.269.959084795 /tmp/redo9.dbf

1* select group#,member,status from v$logfile

GROUP# MEMBER STATUS

SQL> alter database drop logfile member

SQL> alter database drop logfile member

SQL> select group#,status,thread# from v$log;

GROUP# STATUS THREAD#

SQL> alter database clear unarchived logfile group 2;

Vous aimerez peut-être aussi