E-mail:chf.dba@gmail.com
Title: Oracle 19c failure recovery
Author: DATABASE SOS©All rights reserved [without my consent, it may not be reproduced in any form, otherwise there is the right to further legal responsibility.]
Some customers found us. Their oracle 19c database started abnormally due to abnormal power failure. After a series of recovery, the problem could not be solved, and we asked for support. Through our Oracle Database Recovery Check Script , get The current database information is as follows:
The database version is 19C and the 19.5.0.0.191015 (30125133) patch is installed
After the database is successfully started, it will crash in a while
2020-03-10T01:44:41.018032+08:00 Pluggable database RACBAK opened read write 2020-03-10T01:44:41.018996+08:00 Pluggable database RAC opened read write 2020-03-10T01:44:51.244050+08:00 Completed: ALTER PLUGGABLE DATABASE ALL OPEN Starting background process CJQ0 Completed: ALTER DATABASE OPEN 2020-03-10T01:44:51.317085+08:00 CJQ0 started with pid=224, OS id=32581 2020-03-10T01:44:56.067043+08:00 Errors in file /opt/oracle/diag/rdbms/XFF/XFF/trace/XFF_j001_32588.trc (incident=1095281) (PDBNAME=RAC): ORA-00600: internal error code, arguments: [4193], [27733], [27754], [], [], [], [], [], [], [], [], [] RAC(4):Incident details in: /opt/oracle/diag/rdbms/XFF/XFF/incident/incdir_1095281/XFF_j001_32588_i1095281.trc RAC(4):Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. 2020-03-10T01:44:56.073112+08:00 RAC(4):***************************************************************** RAC(4):An internal routine has requested a dump of selected redo. RAC(4):This usually happens following a specific internal error, when RAC(4):analysis of the redo logs will help Oracle Support with the RAC(4):diagnosis. RAC(4):It is recommended that you retain all the redo logs generated (by RAC(4):all the instances) during the past 12 hours, in case additional RAC(4):redo dumps are required to help with the diagnosis. RAC(4):***************************************************************** 2020-03-10T01:44:56.079228+08:00 Errors in file /opt/oracle/diag/rdbms/XFF/XFF/trace/XFF_j002_32590.trc (incident=1095289) (PDBNAME=RAC): ORA-00600: internal error code, arguments: [4193], [2633], [2638], [], [], [], [], [], [], [], [], [] RAC(4):Incident details in: /opt/oracle/diag/rdbms/XFF/XFF/incident/incdir_1095289/XFF_j002_32590_i1095289.trc RAC(4):Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. 2020-03-10T01:44:56.085068+08:00 RAC(4):***************************************************************** RAC(4):An internal routine has requested a dump of selected redo. RAC(4):This usually happens following a specific internal error, when RAC(4):analysis of the redo logs will help Oracle Support with the RAC(4):diagnosis. RAC(4):It is recommended that you retain all the redo logs generated (by RAC(4):all the instances) during the past 12 hours, in case additional RAC(4):redo dumps are required to help with the diagnosis. RAC(4):***************************************************************** 2020-03-10T01:44:56.115765+08:00 Errors in file /opt/oracle/diag/rdbms/XFF/XFF/trace/XFF_j004_32594.trc (incident=1095305) (PDBNAME=RAC): ORA-00600: internal error code, arguments: [4193], [63532], [63537], [], [], [], [], [], [], [], [], [] RAC(4):Incident details in: /opt/oracle/diag/rdbms/XFF/XFF/incident/incdir_1095305/XFF_j004_32594_i1095305.trc RAC(4):Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. 2020-03-10T01:46:48.202213+08:00 RAC(4):Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0 RAC(4): Mem# 0: /opt/oracle/oradata/XFF/redo02.log RAC(4):Block recovery completed at rba 0.0.0, scn 0x0000000d3675e48e RAC(4):DDE: Problem Key 'ORA 600 [4193]' was completely flood controlled (0x6) Further messages for this problem key will be suppressed for up to 10 minutes 2020-03-10T01:46:48.384040+08:00 Errors in file /opt/oracle/diag/rdbms/XFF/XFF/trace/XFF_clmn_31741.trc: ORA-00600: internal error code, arguments: [4193], [27733], [27754], [], [], [], [], [], [], [], [], [] Errors in file /opt/oracle/diag/rdbms/XFF/XFF/trace/XFF_clmn_31741.trc (incident=1093505) (PDBNAME=CDB$ROOT): ORA-501 [] [] [] [] [] [] [] [] [] [] [] [] Incident details in: /opt/oracle/diag/rdbms/XFF/XFF/incident/incdir_1093505/XFF_clmn_31741_i1093505.trc 2020-03-10T01:46:49.264624+08:00 USER (ospid: 31741): terminating the instance due to ORA error 501 2020-03-10T01:46:49.280664+08:00 System state dump requested by (instance=1, osid=31741 (CLMN)), summary=[abnormal instance termination]. System State dumped to trace file /opt/oracle/diag/rdbms/XFF/XFF/trace/XFF_diag_31759.trc 2020-03-10T01:46:53.156926+08:00 ORA-00501: CLMN process terminated with error 2020-03-10T01:46:53.157103+08:00 Errors in file /opt/oracle/diag/rdbms/XFF/XFF/trace/XFF_diag_31759.trc: ORA-00501: CLMN process terminated with error 2020-03-10T01:46:53.157211+08:00 Dumping diagnostic data in directory=[cdmp_20200310014649], requested by (instance=1, osid=31741 (CLMN)), summary=[abnormal instance termination].
Judging by the error information, after the database is opened (especially after pdb 4 is opened), an ORA-600 4193 error is reported. Then the CLMN process is abnormal and the database crashes. For this type of failure, because of the pdb used, and because of the pdb The undo exception causes the database to crash after startup.You can perform special processing on pdb to prevent the database from crashing after startup.