CERN Accelerating science

CVICLR02FC cluster problems

 
Date & time of incident: 
Tuesday, June 18, 2013 - 10:00
Post date: 
Tuesday, June 18, 2013 - 15:05
Incident Description: 

We are again experiencing problems with the cluster CVICLR02FC that is hosting virtual machines for BE-CO

Due to this instability the VMs will get rebooted.

We are trying to solve the incident as soon as possible.

Service Element Affected: 
Multiple Services
Specific Service detail: 
CERN Virtualization Infrastructure
Impact: 
Service is degraded
Status: 
Resolved
Resolution date: 
Mon, Jun 24, 16:00
Expected resolution or Next Update Time: 
Tuesday, June 25, 2013 - 08:00
Posted by: 
IT-OIS
Unit responsible for resolution: 
IT Department

Updates

Posted June 25, 2013 - 8:48am

The machines are now running on new hardware and confirmed working by BE/CO.

 

Posted June 21, 2013 - 12:03pm

We continue to see problems on the cluster, so we will do a major intervention to evacuate the machines on to a different set of hypervisors.

Posted June 20, 2013 - 4:50pm

We will restart the complete cluster to reconnect to the updated storage members

Posted June 20, 2013 - 4:32pm

The cluster is still having problems. We are updating the firmware on the storage and evaluating possible alternatives for the VMs hosted.

Posted June 19, 2013 - 3:45pm

Update of the cluster finished

We have just update all the nodes in the cluster, and clean up all the duplicated entries in cluster database and vmm database.

It should recover the stability of the machines.