CERN Accelerating science

DB On Demand Service instances rebooting

 
Date & time of incident: 
Thursday, October 11, 2012 - 15:54
Post date: 
Thursday, October 11, 2012 - 16:02
Incident Description: 

Some DBOD instances were affected by sudden reboots because of the ongoing Virtualization layer problems (events from 14:18 to 14:36):
hc_cms, boinc, trac_svn, gt_pupp, lbcertif, mastro, nova, puppet, piwik, copvss

Workaround: 

Because of the ongoing VM layer problems, some of the underlying virtualization servers are rebooting spontaneously, and so do their virtual guests. This causes unaivalability during the reboot time until the services are restored.

Service Element Affected: 
Multiple Services
Any other affected service(s): 
 hc_cms, boinc, trac_svn, gt_pupp, lbcertif, mastro, nova, puppet, piwik, copvss
Impact: 
Service is unavailable
Status: 
Resolved
Resolution date: 
Wed, Oct 17, 10:00
Posted by: 
IT-DB
Unit responsible for resolution: 
IT Department

Updates

Posted October 17, 2012 - 10:39am

Some instances were moved to a different platform in order to reduce impact of the virtualization layer issues

Posted October 12, 2012 - 10:39am

At ~ 10:25 mastro, nova and trac_svn are still unavailable

Posted October 12, 2012 - 10:01am

A number of virtual services have rebooted again at 12/10/2012 09:48, affecting the
following DBOD instances:
lbcertif, boinc, gt_pupp, mastro, nova, trac_svn