Date & time of incident:
Tuesday, August 21, 2012 - 12:30
Post date:
Wednesday, August 22, 2012 - 09:49
Incident Description:
Yesterday (21 August) users have experienced intermittent "ORA-12520 connection problems with "TNS: listener could not find available handler for requested type of server" errors while connecting to the following services on ITCORE: cluman,cdb,remedy,sls,lemonops,tapelog.
Existing sessions were *not* affected by the problem.
The service was available again at 14:30. See below for more detailed explanations.
Service Element Affected:
General Purpose DB & App Platform
Specific Service detail:
ITCORE services cluman,cdb,remedy,sls,lemonops,tapelog
Impact:
Some applications linked to services are unavailable
Status:
Resolved
Resolution date:
Tue, Aug 21, 14:50
Posted by:
IT-DB
Unit responsible for resolution:
IT Department
Updates
Explanation (details):
Explanation (details):
This incident has been caused by the following chain of events:
- in the morning the password of a DB account has been changed;
- some client python processes running were still connecting using the old password;
- somehow the failed logins were leaving the connections in a bad state, which caused a lot of session waits for library cache locks.
The underlying reason for this is unclear, it could be either on the python driver side which is not able to properly close the connections in this case, or on the DB side where the listener did not understand what was going on;
- since 10:25 the waiting sessions kept on piling up, and at 12:30 saturated the maximum number of sessions available. Note that existing sessions were not affected by this problem;
- the waiting sessions were killed at 14:50 and this made the service 100% available again.