CERN Accelerating science

Problems with Drupal and AIS applications

 
Date & time of incident: 
Tuesday, April 24, 2012 - 17:15
Post date: 
Wednesday, April 4, 2012 - 16:47
Incident Description: 

There are problems affecting several Drupal ad AIS applications. It is linked with the planned intervention https://test-static-03.web.cern.ch/planned-intervention/firmware-upgrade-some-components-safe-host-controllers/24-04-2012

Upgrade of one controller was transparent, second node was done three weeks ago, and upgrade was also transparent. Same procedure was applied to second cluster, but upgrade on second cluster dbnasg403/dbnasg404 resulted on one controller freezing which has affected services running on it (drupal, nova, aisrac, hammercloud-atlas, AIS). Despite that we tried to minimize the freezing time some database services got affected. The downtime was less than 15 minutes. A postmortem and further analysis with Netapp is on-going (a call to NetApp has been opened).

We apologise for the inconvenience caused by this incident.

 

 

Service Element Affected: 
DB & App Platform for AIS
General Purpose DB & App Platform
Web Service
Impact: 
Service is unavailable
Status: 
Resolved
Resolution date: 
Tue, Apr 24, 17:35
Expected resolution or Next Update Time: 
Tuesday, April 24, 2012 - 19:00
Posted by: 
GS-AIS
Unit responsible for resolution: 
IT Department

Updates

Posted April 24, 2012 - 7:13pm

Second instance of AIS RAC prod was evicted by the clusterware, causing PPT services bound to instance 2 and not using service names to fail.

Instance 2  is back as of 17:44.

PPT services required application server restart and are back to normal operation at 18:45

IT-DB

Posted April 24, 2012 - 7:06pm

Third instance of production AIS RAC was overloaded and we have restarted it.

We are investigating for the root cause of the problem.

Services are back to normal (except PPT).

IT-DB