ORA-3136 WARNING: inbound connection timed out in alert log

as follow-up to https://joeyski.wordpress.com/2009/11/10/ora-3136-warning-inbound-connection-timed-out/

One client had a problem with its datawarehouse due to some batch process that failed and went unoticed for quite sometimes. After discovering that the data missing was during that some specified period we dug up the alert log and found this issues.

The problem started 6-Aug -10PM

Thread 1 advanced to log sequence 37449

Current log# 3 seq# 37449 mem# 0: /app/oracle/oradata/OLSG3DB/redo03.log

Thu Aug  6 22:10:13 2009

Errors in file /app/oracle/admin/OLSG3DB/bdump/olsg3db_arc1_1671288.trc:

ORA-19510: failed to set size of 182064 blocks for file
“/d8/oracle/oradata/OLSG3DB/flash_recovery_area/OLSG3DB/archivelog/1_37448_677257396.dbf”
(blocksize=512)

Failed to write the file in Flash_recovery_area
— FULL!

ORA-27037: unable to obtain file status

IBM AIX RISC System/6000 Error: 2: No such file or directory

Was not written that’s why ‘no such
file’

Additional information: 9

Thu Aug  6 22:10:13 2009

ARC1: Error 19510 Closing archive log file
‘/d8/oracle/oradata/OLSG3DB/flash_recovery_area/OLSG3DB/archivelog/1_37448_677257396.dbf’

ARC1: Failed to archive thread 1 sequence 37448 (19510)

ARCH: Archival stopped, error occurred. Will continue retrying

Thu Aug  6 22:10:13 2009

ORACLE Instance OLSG3DB – Archival Error

Thu Aug  6 22:10:13 2009

ORA-16038: log 2 sequence# 37448 cannot be archived

ORA-19510: failed to set size of  blocks for file “” (blocksize=)

ORA-00312: online log 2 thread 1: ‘/d2/oracle/oradata/OLSG3DB/redo02.log’

Thu Aug  6 22:10:13 2009

Errors in file /app/oracle/admin/OLSG3DB/bdump/olsg3db_arc1_1671288.trc:

ORA-16038: log 2 sequence# 37448 cannot be archived

ORA-19510: failed to set size of  blocks for file “” (blocksize=)

ORA-00312: online log 2 thread 1: ‘/d2/oracle/oradata/OLSG3DB/redo02.log’

Thu Aug  6 22:10:58 2009

Archiver process freed from errors. No longer stopped

Fri Aug  7 00:02:10 2009

DB_FLASHBACK_RETENTION_TARGET setting
kicks in some(small) space got freed, oracle is happy, can continue archiving
in Flashback_recovery_area

Thread 1 advanced to log sequence 37450

 Current log# 4 seq# 37450 mem# 0: /d1/oracle/oradata/OLSG3DB/redo04.log

Fri Aug  7 00:02:57 2009

Thread 1 advanced to log sequence 37451

 Current log# 5 seq# 37451 mem# 0: /d3/oracle/oradata/OLSG3DB/redo05.log

Fri Aug  7 00:04:47 2009

—-

—-

10AM work resumes, transactions peaks,
flashback_area is full again..

Thread 1 advanced to log sequence 37579

Current log# 3 seq# 37579 mem# 0: /app/oracle/oradata/OLSG3DB/redo03.log

Fri Aug  7 10:15:03 2009

Thread 1 cannot allocate new log, sequence 37580

Checkpoint not complete

Current log# 3 seq# 37579 mem# 0: /app/oracle/oradata/OLSG3DB/redo03.log

Fri Aug  7 10:31:54 2009

Thread 1 advanced to log sequence 37580

Current log# 4 seq# 37580 mem# 0: /d1/oracle/oradata/OLSG3DB/redo04.log

Fri Aug  7 10:31:54 2009

Oracle hangs because it can’t archive
redo logs, cannot process logins, hundreds of ORA
3136 follows
within timeframe 10:31:55!

WARNING: inbound connection timed out (ORA-3136)

Fri Aug  7 10:31:54 2009

WARNING: inbound connection timed out (ORA-3136)

Fri Aug  7 10:31:54 2009

..
..
..
..
WARNING: inbound connection timed out (ORA-3136)

Fri Aug  7 10:31:55 2009

WARNING: inbound connection timed out (ORA-3136)

Fri Aug  7 10:31:55 2009

WARNING: inbound connection timed out (ORA-3136)

Fri Aug  7 10:31:55 2009

WARNING: inbound connection timed out (ORA-3136)

Fri Aug  7 10:31:55 2009

WARNING: inbound connection timed out (ORA-3136)

Fri Aug  7 10:31:55 2009

WARNING: inbound connection timed out (ORA-3136)

Fri Aug  7 10:31:55 2009

WARNING: inbound connection timed out (ORA-3136)

Fri Aug  7 10:31:55 2009

WARNING: inbound connection timed out (ORA-3136)

Fri Aug  7 10:32:54 2009

Thread 1 advanced to log sequence 37581

 Current log# 5 seq# 37581 mem# 0: /d3/oracle/oradata/OLSG3DB/redo05.log

Fri Aug  7 10:33:56 2009

Thread 1 advanced to log sequence 37582

 Current log# 1 seq# 37582 mem# 0: /d0/oracle/oradata/OLSG3DB/redo01.log

Fri Aug  7 10:34:03 2009

Is this a manual shutdown or Oracle
died miserably?

Shutting down instance: further logons
disabled

Fri Aug  7 10:34:23 2009

Shutting down instance (immediate)

License high water mark = 414

Fri Aug  7 10:37:00 2009

sculkget: failed to lock /app/oracle/product/10.2.0/db_1/dbs/lkinstOLSG3DB
exclusive

sculkget: lock held by PID: 1806548

Oracle Instance Shutdown operation failed. Another process may be attempting
to startup or shutdown this Instance.

Failed to acquire instance startup/shutdown serialization primitive

Fri Aug  7 10:37:41 2009

Waiting for dispatcher ‘D000’ to shutdown

All dispatchers and shared servers shutdown

Fri Aug  7 10:37:48 2009

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

Fri Aug  7 10:37:53 2009

Process OS id : 1798242 alive after kill

Errors in file

Fri Aug  7 10:37:54 2009

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

Fri Aug  7 10:38:06 2009

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

Fri Aug  7 10:38:17 2009

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

Fri Aug  7 10:38:28 2009

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

Fri Aug  7 10:38:39 2009

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

Fri Aug  7 10:38:50 2009

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

Fri Aug  7 10:39:01 2009

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

Fri Aug  7 10:39:12 2009

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

Fri Aug  7 10:39:23 2009

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

PMON failed to acquire latch, see PMON dump

Fri Aug  7 10:39:29 2009

ALTER DATABASE CLOSE NORMAL

Fri Aug  7 10:39:29 2009

SMON: disabling tx recovery

SMON: disabling cache recovery

Fri Aug  7 10:40:10 2009

Shutting down archive processes

Archiving is disabled

Fri Aug  7 10:40:15 2009

ARCH shutting down

ARC2: Archival stopped

Fri Aug  7 10:40:20 2009

ARCH shutting down

ARC1: Archival stopped

Fri Aug  7 10:40:25 2009

ARCH shutting down

ARC0: Archival stopped

Fri Aug  7 10:40:28 2009

Thread 1 closed at log sequence 37582

Successful close of redo thread 1

Fri Aug  7 10:40:31 2009

Completed: ALTER DATABASE CLOSE NORMAL

Fri Aug  7 10:40:31 2009

ALTER DATABASE DISMOUNT

Completed: ALTER DATABASE DISMOUNT

ARCH: Archival disabled due to shutdown: 1089

Shutting down archive processes

Archiving is disabled

Archive process shutdown avoided: 0 active

ARCH: Archival disabled due to shutdown: 1089

Shutting down archive processes

Archiving is disabled

Archive process shutdown avoided: 0 active

Fri Aug  7 10:40:52 2009



So, we can conclude that the cause of ORA
3136 is the full FRA and not because of some network latency problems.

Advertisements

About this entry