很久以前的项目，系统已正常运行近5年左右了，最近数据库突然连接不上了，环境为oracle 11g 两节点的集群{rac1,rac2}，GI和db版本都是11.2.0.4 ，未开归档，操作系统rhel6.4X64
外部客户端都是链接失败，项目程序jdbc连接报“create connection error" 、"Connection reset"，sqlplus连接报”ORA-12547 丢失连接“，tnsping提示超时。
grid\diag\tnslsnr\rac1\listener\alert\log.xml中有以下错误信息，每次客户端连接都有。
<msg time='2023-07-17T20:59:12.226+08:00' org_id='oracle' comp_id='tnslsnr'
type='UNKNOWN' level='16' host_id='rac1'
host_addr='192.168.4.207'>
<txt>TNS-12518: TNS:listener could not hand off client connection
TNS-12547: TNS:lost contact
TNS-12560: TNS:protocol adapter error
TNS-00517: Lost contact
Linux Error: 32: Broken pipe
</txt>
</msg>
oracle\diag\rdbms\orcldb\orcldb1\alert\log.xml中有以下错误信息，每次客户端连接都有。
<msg time='2023-07-17T21:08:02.224+08:00' org_id='oracle' comp_id='rdbms'
type='UNKNOWN' level='16' host_id='rac1'
host_addr='192.168.4.207'>
<txt>
Fatal NI connect error 12547, connecting to:
(LOCAL=NO)
</txt>
</msg>
<msg time='2023-07-17T21:08:02.224+08:00' org_id='oracle' comp_id='rdbms'
type='UNKNOWN' level='16' host_id='rac1'
host_addr='192.168.4.207'>
<txt>
VERSION INFORMATION:
TNS for Linux: Version 11.2.0.4.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
</txt>
</msg>
<msg time='2023-07-17T21:08:02.225+08:00' org_id='oracle' comp_id='rdbms'
type='UNKNOWN' level='16' host_id='rac1'
host_addr='192.168.4.207'>
<txt> Time: 17-JUL-2023 21:08:02
</txt>
</msg>
<msg time='2023-07-17T21:08:02.225+08:00' org_id='oracle' comp_id='rdbms'
type='UNKNOWN' level='16' host_id='rac1'
host_addr='192.168.4.207'>
<txt> Tracing not turned on.
</txt>
</msg>
<msg time='2023-07-17T21:08:02.225+08:00' org_id='oracle' comp_id='rdbms'
type='UNKNOWN' level='16' host_id='rac1'
host_addr='192.168.4.207'>
<txt> Tns error struct:
</txt>
</msg>
<msg time='2023-07-17T21:08:02.225+08:00' org_id='oracle' comp_id='rdbms'
type='UNKNOWN' level='16' host_id='rac1'
host_addr='192.168.4.207'>
<txt> ns main err code: 12547
</txt>
</msg>
<msg time='2023-07-17T21:08:02.225+08:00' org_id='oracle' comp_id='rdbms'
type='UNKNOWN' level='16' host_id='rac1'
host_addr='192.168.4.207'>
<txt>
</txt>
</msg>
<msg time='2023-07-17T21:08:02.225+08:00' org_id='oracle' comp_id='rdbms'
type='UNKNOWN' level='16' host_id='rac1'
host_addr='192.168.4.207'>
<txt>TNS-12547: TNS:lost contact
</txt>
</msg>
<msg time='2023-07-17T21:08:02.225+08:00' org_id='oracle' comp_id='rdbms'
type='UNKNOWN' level='16' host_id='rac1'
host_addr='192.168.4.207'>
<txt> ns secondary err code: 12560
</txt>
</msg>
<msg time='2023-07-17T21:08:02.225+08:00' org_id='oracle' comp_id='rdbms'
type='UNKNOWN' level='16' host_id='rac1'
host_addr='192.168.4.207'>
<txt> nt main err code: 0
</txt>
</msg>
<msg time='2023-07-17T21:08:02.225+08:00' org_id='oracle' comp_id='rdbms'
type='UNKNOWN' level='16' host_id='rac1'
host_addr='192.168.4.207'>
<txt> nt secondary err code: 0
</txt>
</msg>
<msg time='2023-07-17T21:08:02.225+08:00' org_id='oracle' comp_id='rdbms'
type='UNKNOWN' level='16' host_id='rac1'
host_addr='192.168.4.207'>
<txt> nt OS err code: 0
</txt>
</msg>
<msg time='2023-07-17T21:08:02.234+08:00' org_id='oracle' comp_id='rdbms'
type='UNKNOWN' level='16' host_id='rac1'
host_addr='192.168.4.207' pid='10115'>
<txt>opiodr aborting process unknown ospid (10115) as a result of ORA-609
</txt>
</msg>
oracle\diag\rdbms\orcldb\orcldb1\trace\alert_orcldb1.log
Fatal NI connect error 12547, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for Linux: Version 11.2.0.4.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
Time: 13-JUL-2023 18:14:10
Tracing not turned on.
Tns error struct:
ns main err code: 12547
TNS-12547: TNS:lost contact
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
opiodr aborting process unknown ospid (8604) as a result of ORA-609
Completed: ALTER DATABASE OPEN /* db agent *//* {1:15042:143} */
Thu Jul 13 18:14:11 2023
Starting background process CJQ0
Thu Jul 13 18:14:11 2023
CJQ0 started with pid=48, OS id=8636
Thu Jul 13 18:14:15 2023
ORA-1653: unable to extend table SYS.SCHEDULER$_EVENT_LOG by 128 in tablespace SYSAUX
ORA-1653: unable to extend table SYS.SCHEDULER$_EVENT_LOG by 128 in tablespace SYSAUX
Errors in file /u01/oracle/diag/rdbms/orcldb/orcldb1/trace/orcldb1_j002_8721.trc:
ORA-01653: unable to extend table . by in tablespace
ORA-01653: unable to extend table SYS.SCHEDULER$_EVENT_LOG by 128 in tablespace SYSAUX
Thu Jul 13 18:14:15 2023
ORA-1653: unable to extend table SYS.SCHEDULER$_EVENT_LOG by 128 in tablespace SYSAUX
ORA-1653: unable to extend table SYS.SCHEDULER$_EVENT_LOG by 128 in tablespace SYSAUX
|
|
|
Fatal NI connect error 12547, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for Linux: Version 11.2.0.4.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
Time: 17-JUL-2023 21:10:11
Tracing not turned on.
Tns error struct:
ns main err code: 12547
TNS-12547: TNS:lost contact
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
opiodr aborting process unknown ospid (10552) as a result of ORA-609
/u01/11.2.0.4/grid/log\diag\tnslsnr\rac1\listener_scan1\alert\log.xml中有以下错误信息，每次客户端连接都有。
<msg time='2023-07-05T15:53:49.084+08:00' org_id='oracle' comp_id='tnslsnr'
type='UNKNOWN' level='16' host_id='rac1'
host_addr='::1'>
<txt>TNS-12582: TNS:invalid operation
TNS-12623: TNS:operation is illegal in this state
TNS-12509: TNS:listener failed to redirect client to service handler
TNS-12547: TNS:lost contact
TNS-12560: TNS:protocol adapter error
TNS-00517: Lost contact
Linux Error: 104: Connection reset by peer
</txt>
</msg>
奔溃中ING
