CERN Accelerating science

LCG router in the CC513 Vault incident

 
Date & time of incident: 
Sunday, February 10, 2013 - 11:01
Post date: 
Sunday, February 10, 2013 - 11:39
Incident Description: 

One LCG router in the Vault of B513 suddenly stopped working properly. A software reset of a linecard restored connectivity, but the exact cause is not yet understood; thus is incident is kept open.

All the following services were disconnected for ~20 minutes.

 

port-name Openlab 10G#S513-V-IP502-VLAN                          
 port-name Openlab 1Gb#S513-V-IP501-VLAN
 port-name CPU SP20#S513-V-IP180-VLAN
 port-name Disk - iSCSI SM39#S513-V-IP201-VLAN
 port-name CPU SP21#S513-V-IP181-VLAN
 port-name CPU SQ11#S513-V-IP111-VLAN
 port-name CPU SP15#S513-V-IP101-VLAN
 port-name Disk - iSCSI SM39#S513-V-IP202-VLAN
 port-name Tape#S513-V-IP401-VLAN
 port-name Tape#S513-V-IP402-VLAN
 port-name Tape#S513-V-IP404-VLAN
 port-name Tape#S513-V-IP405-VLAN
 port-name CPU SP14#S513-V-IP102-VLAN
 port-name CPU SP22#S513-V-IP182-VLAN
 port-name Tape#S513-V-IP406-VLAN
 port-name CPU SP23#S513-V-IP183-VLAN
 port-name CPU ST28#S513-V-IP148-VLAN
 port-name Tape#S513-V-IP407-VLAN
 port-name Tape#S513-V-IP408-VLAN
 port-name CPU SP13#S513-V-IP103-VLAN
 port-name CPU SP12#S513-V-IP104-VLAN
 port-name CPU SP11#S513-V-IP105-VLAN
 port-name CPU SP09#S513-V-IP106-VLAN
 port-name CPU SQ15#S513-V-IP107-VLAN                             
 port-name CPU SQ14#S513-V-IP108-VLAN
 port-name CPU SQ13#S513-V-IP109-VLAN
 port-name CPU SQ12#S513-V-IP110-VLAN
 port-name CPU SQ09#S513-V-IP112-VLAN
 port-name Disk - iSCSI SM39#S513-V-IP203-VLAN
 port-name Small Disk Server SL39#S513-V-IP301-VLAN
 port-name Small Disk Server SL39#S513-V-IP302-VLAN
 port-name Disk SL39#S513-V-IP204-VLAN
 port-name Disk SL39#S513-V-IP205-VLAN
 port-name Disk SL39#S513-V-IP206-VLAN
 port-name Disk SL39#S513-V-IP207-VLAN
 port-name Disk SL39#S513-V-IP208-VLAN
 port-name Disk SL39#S513-V-IP209-VLAN
 port-name Disk SL39#S513-V-IP210-VLAN
 port-name Disk SL39#S513-V-IP211-VLAN
 port-name 1Gb access for 10Gb disk servers SM39#S513-V-IP299-VLAN
 port-name CPU ST14#S513-V-IP122-VLAN
 port-name CPU ST13#S513-V-IP123-VLAN
 port-name CPU ST12#S513-V-IP124-VLAN
 port-name CPU ST11#S513-V-IP125-VLAN
 port-name CPU ST09#S513-V-IP126-VLAN
 port-name CPU ST08#S513-V-IP127-VLAN
 port-name CPU ST07#S513-V-IP128-VLAN                             
 port-name CPU ST06#S513-V-IP129-VLAN
 port-name CPU ST05#S513-V-IP130-VLAN
 port-name CPU ST20#S513-V-IP141-VLAN
 port-name CPU ST21#S513-V-IP142-VLAN
 port-name CPU ST22#S513-V-IP143-VLAN
 port-name CPU ST23#S513-V-IP144-VLAN
 port-name CPU ST24#S513-V-IP145-VLAN
 port-name Mid-Range Blades - Virtualisation SM32#S513-V-IP303-VLAN
 port-name Mid-Range Blades - Virtualisation SM34#S513-V-IP304-VLAN
 port-name Mid-Range Blades - General Purpose SJ33#S513-V-IP305-VLAN
 port-name Mid-Range Blades - General Purpose SJ34#S513-V-IP306-VLAN
 port-name Mid-Range Blades - General Purpose SJ35#S513-V-IP307-VLAN
 port-name Tapeservers at 10Gb in T522#S513-V-IP410-VLAN
 port-name Tapeservers at 10Gb in T521#S513-V-IP411-VLAN
 port-name CPU ST26#S513-V-IP146-VLAN
 port-name CPU ST29#S513-V-IP149-VLAN
 port-name CPU ST30#S513-V-IP150-VLAN
 port-name CPU Servers in SQ20#S513-V-IP161-VLAN
 port-name CPU Servers in SQ21#S513-V-IP162-VLAN
 port-name CPU Servers in SQ23#S513-V-IP163-VLAN
 port-name CPU Servers in SQ24#S513-V-IP164-VLAN
 port-name CPU SQ26#S513-V-IP165-VLAN
 port-name CPU SQ27#S513-V-IP166-VLAN                             
 port-name CPU SQ28#S513-V-IP167-VLAN
 port-name CPU SQ29#S513-V-IP168-VLAN
 port-name CPU SQ30#S513-V-IP169-VLAN
 port-name Bernd Test CPUs #1#S513-V-IP170-VLAN
 port-name Bernd Test CPUs #2#S513-V-IP171-VLAN
 port-name Bernd Test CPUs #3#S513-V-IP172-VLAN
 port-name Bernd Test CPUs #4#S513-V-IP173-VLAN
 port-name Bernd Test CPUs #5#S513-V-IP174-VLAN
 port-name Bernd Test CPUs #6#S513-V-IP175-VLAN
 port-name MidRange blades 10GbE OIS#S513-V-IP311-VLAN
 port-name MidRange blades 10GbE OIS#S513-V-IP312-VLAN
 port-name IPv6 tests LCG - HYPERV#S513-V-IP176-VLAN
 port-name CPU SP24#S513-V-IP184-VLAN
 port-name CPU SP26#S513-V-IP185-VLAN
 port-name CPU SP27#S513-V-IP186-VLAN
 port-name CPU SP28#S513-V-IP187-VLAN
 port-name CPU SP29#S513-V-IP188-VLAN
 port-name CPU SP30#S513-V-IP189-VLAN
 port-name Disk at 10Gb - SM40#S513-V-IP256-VLAN
 port-name Disk at 10Gb - SM42#S513-V-IP257-VLAN
 port-name Disk at 10Gb - SM44#S513-V-IP258-VLAN                 
 port-name OpenLAB Huawei NAS#S513-V-OX0-VLAN
 port-name CPU SE06#S513-V-IP177-VLAN
 port-name CPU SE09#S513-V-IP178-VLAN
 port-name CPU SE13#S513-V-IP179-VLAN
 

Service Element Affected: 
Multiple Services
Specific Service detail: 
port-name Openlab 10G#S513-V-IP502-VLAN port-name Openlab 1Gb#S513-V-IP501-VLAN port-name CPU SP20#S513-V-IP180-VLAN port-name Disk - iSCSI SM39#S513-V-IP201-VLAN port-name CPU SP21#S513-V-IP181-VLAN port-name CPU SQ11#S513-V-IP111-VLAN port-name CPU SP15#S513-V-IP101-VLAN port-name Disk - iSCSI SM39#S513-V-IP202-VLAN port-name Tape#S513-V-IP401-VLAN port-name Tape#S513-V-IP402-VLAN port-name Tape#S513-V-IP404-VLAN port-name Tape#S513-V-IP405-VLAN port-name CPU SP14#S513-V-IP102-VLAN port-name CPU SP22#S513-V-IP182-VLAN port-name Tape#S513-V-IP406-VLAN port-name CPU SP23#S513-V-IP183-VLAN port-name CPU ST28#S513-V-IP148-VLAN port-name Tape#S513-V-IP407-VLAN port-name Tape#S513-V-IP408-VLAN port-name CPU SP13#S513-V-IP103-VLAN port-name CPU SP12#S513-V-IP104-VLAN port-name CPU SP11#S513-V-IP105-VLAN port-name CPU SP09#S513-V-IP106-VLAN port-name CPU SQ15#S513-V-IP107-VLAN port-name CPU SQ14#S513-V-IP108-VLAN port-name CPU SQ13#S513-V-IP109-VLAN port-name CPU SQ12#S513-V-IP110-VLAN port-name CPU SQ09#S513-V-IP112-VLAN port-name Disk - iSCSI SM39#S513-V-IP203-VLAN port-name Small Disk Server SL39#S513-V-IP301-VLAN port-name Small Disk Server SL39#S513-V-IP302-VLAN port-name Disk SL39#S513-V-IP204-VLAN port-name Disk SL39#S513-V-IP205-VLAN port-name Disk SL39#S513-V-IP206-VLAN port-name Disk SL39#S513-V-IP207-VLAN port-name Disk SL39#S513-V-IP208-VLAN port-name Disk SL39#S513-V-IP209-VLAN port-name Disk SL39#S513-V-IP210-VLAN port-name Disk SL39#S513-V-IP211-VLAN port-name 1Gb access for 10Gb disk servers SM39#S513-V-IP299-VLAN port-name CPU ST14#S513-V-IP122-VLAN port-name CPU ST13#S513-V-IP123-VLAN port-name CPU ST12#S513-V-IP124-VLAN port-name CPU ST11#S513-V-IP125-VLAN port-name CPU ST09#S513-V-IP126-VLAN port-name CPU ST08#S513-V-IP127-VLAN port-name CPU ST07#S513-V-IP128-VLAN port-name CPU ST06#S513-V-IP129-VLAN port-name CPU ST05#S513-V-IP130-VLAN port-name CPU ST20#S513-V-IP141-VLAN port-name CPU ST21#S513-V-IP142-VLAN port-name CPU ST22#S513-V-IP143-VLAN port-name CPU ST23#S513-V-IP144-VLAN port-name CPU ST24#S513-V-IP145-VLAN port-name Mid-Range Blades - Virtualisation SM32#S513-V-IP303-VLAN port-name Mid-Range Blades - Virtualisation SM34#S513-V-IP304-VLAN port-name Mid-Range Blades - General Purpose SJ33#S513-V-IP305-VLAN port-name Mid-Range Blades - General Purpose SJ34#S513-V-IP306-VLAN port-name Mid-Range Blades - General Purpose SJ35#S513-V-IP307-VLAN port-name Tapeservers at 10Gb in T522#S513-V-IP410-VLAN port-name Tapeservers at 10Gb in T521#S513-V-IP411-VLAN port-name CPU ST26#S513-V-IP146-VLAN port-name CPU ST29#S513-V-IP149-VLAN port-name CPU ST30#S513-V-IP150-VLAN port-name CPU Servers in SQ20#S513-V-IP161-VLAN port-name CPU Servers in SQ21#S513-V-IP162-VLAN port-name CPU Servers in SQ23#S513-V-IP163-VLAN port-name CPU Servers in SQ24#S513-V-IP164-VLAN port-name CPU SQ26#S513-V-IP165-VLAN port-name CPU SQ27#S513-V-IP166-VLAN port-name CPU SQ28#S513-V-IP167-VLAN port-name CPU SQ29#S513-V-IP168-VLAN port-name CPU SQ30#S513-V-IP169-VLAN port-name Bernd Test CPUs #1#S513-V-IP170-VLAN port-name Bernd Test CPUs #2#S513-V-IP171-VLAN port-name Bernd Test CPUs #3#S513-V-IP172-VLAN port-name Bernd Test CPUs #4#S513-V-IP173-VLAN port-name Bernd Test CPUs #5#S513-V-IP174-VLAN port-name Bernd Test CPUs #6#S513-V-IP175-VLAN port-name MidRange blades 10GbE OIS#S513-V-IP311-VLAN port-name MidRange blades 10GbE OIS#S513-V-IP312-VLAN port-name IPv6 tests LCG - HYPERV#S513-V-IP176-VLAN port-name CPU SP24#S513-V-IP184-VLAN port-name CPU SP26#S513-V-IP185-VLAN port-name CPU SP27#S513-V-IP186-VLAN port-name CPU SP28#S513-V-IP187-VLAN port-name CPU SP29#S513-V-IP188-VLAN port-name CPU SP30#S513-V-IP189-VLAN port-name 100GbE L513-B-RBRMX-1/L513-V-RBRMX-1#S513-B-HA1-VLAN port-name 100GbE L513-B-RBRMX-2/L513-V-RBRMX-1#S513-B-HB1-VLAN port-name Disk at 10Gb - SM40#S513-V-IP256-VLAN port-name Disk at 10Gb - SM42#S513-V-IP257-VLAN port-name Disk at 10Gb - SM44#S513-V-IP258-VLAN port-name 100GbE L513-B-RBRMX-3/L513-V-RBRMX-1#S513-B-HC1-VLAN port-name OpenLAB Huawei NAS#S513-V-OX0-VLAN port-name CPU SE06#S513-V-IP177-VLAN port-name CPU SE09#S513-V-IP178-VLAN port-name CPU SE13#S513-V-IP179-VLAN
Impact: 
Service is unavailable
Status: 
Resolved
Resolution date: 
Sun, Feb 10, 11:50
Posted by: 
IT-CS
Unit responsible for resolution: 
IT Department

Updates

Posted February 10, 2013 - 11:49am

The router seems to be stable. The incident will be closed now.