CERN Accelerating science

Cleared Incidents 2011

Summary Date & time of incident Service Element Affected Impact Resolved on Post date
Drupal infrastructure outage Wed, Dec 28, 16:09 Web Service Service is unavailable Wednesday, December 28, 2011 - 18:03 Wed, Dec 28, 19:35
Service-Now unavailable Wed, Dec 21, 13:26 Service Management Service Service is unavailable Wednesday, December 21, 2011 - 15:55 Wed, Dec 21, 15:05
EOSATLAS namespace rebooted Mon, Dec 19, 17:05 Storage Service for Projects & Experiments Service is unavailable Monday, December 19, 2011 - 20:00 Mon, Dec 19, 17:28
Batch/LSF only available intermittently Sat, Dec 17, 18:00 Batch Service Service is unavailable Monday, December 19, 2011 - 17:49 Sun, Dec 18, 02:09
CERNT3 heavily degraded Fri, Dec 16, 17:34 Storage Service for Projects & Experiments Service is unavailable Friday, December 16, 2011 - 18:37 Fri, Dec 16, 17:37
LCGR database is experiencing problems Tue, Dec 6, 03:00 DB & Application Platform Service for Projects & Experiments Service is degraded Tuesday, December 6, 2011 - 12:30 Tue, Dec 6, 10:36
One of Atlas offline databases (ADCR) works with 2 of 3 nodes. Mon, Dec 5, 11:24 DB & Application Platform Service for Projects & Experiments Service is degraded Monday, December 5, 2011 - 11:45 Mon, Dec 5, 11:27
EOSATLAS failover Fri, Dec 2, 15:10 Storage Service for Projects & Experiments Service is unavailable Friday, December 2, 2011 - 15:36 Fri, Dec 2, 15:32
Ansys license server problem Thu, Dec 1, 07:25 Mechanical Design Software Service is unavailable Thursday, December 1, 2011 - 10:08 Thu, Dec 1, 08:47
Problem with SSO login for Service-now (SNOW tool) Wed, Nov 30, 07:00 Service Management Service Service is unavailable Wednesday, November 30, 2011 - 09:00 Wed, Nov 30, 09:04
Short interruption LCG services connected to L513-v-rftec-3 Fri, Nov 25, 17:25 Not specified Service is degraded Saturday, November 26, 2011 - 09:15 Fri, Nov 25, 17:33
User home folders c,g,h on DFS unavailable Thu, Nov 24, 03:40 Windows Desktop Service Service is unavailable Thursday, November 24, 2011 - 05:40 Thu, Nov 24, 07:13
ADCR database is unavailable Wed, Nov 23, 21:45 DB & App Platform for Accelerators Service is unavailable Thursday, November 24, 2011 - 08:55 Wed, Nov 23, 22:16
EOSCMS headnode hw problem Wed, Nov 23, 12:47 Storage Service for Projects & Experiments Service is degraded Wednesday, November 23, 2011 - 13:10 Wed, Nov 23, 14:30
AFS server afs106 down Wed, Nov 23, 10:45 Not specified Service is degraded Wednesday, November 23, 2011 - 11:08 Wed, Nov 23, 10:47
SRM-PUBLIC degraded, client-side timeouts Tue, Nov 22, 11:30 Storage Service for Projects & Experiments Service is degraded Tuesday, November 22, 2011 - 12:10 Tue, Nov 22, 12:09
HP print server down Tue, Nov 22, 11:20 Printing Service Service is unavailable Tuesday, November 22, 2011 - 11:58 Tue, Nov 22, 12:15
Services for offline DB atlas ADCR down for 25 min Tue, Nov 22, 09:30 DB & Application Platform Service for Projects & Experiments Service is unavailable Tuesday, November 22, 2011 - 09:55 Tue, Nov 22, 10:18
VOMS service on lcg-voms.cern.ch unavailable. Mon, Nov 21, 10:58 WLCG Proxy Services Service is degraded Monday, November 21, 2011 - 15:00 Mon, Nov 21, 15:18
DFS links un-available Fri, Nov 18, 17:58 Not specified Service is unavailable Friday, November 18, 2011 - 18:15 Fri, Nov 18, 18:41
EOSATLAS instabilities and emergency update Wed, Nov 16, 11:55 Storage Service for Projects & Experiments Service is degraded Wednesday, November 16, 2011 - 16:10 Wed, Nov 16, 13:38
EOSATLAS namespace reboot Wed, Nov 16, 09:16 Storage Service for Projects & Experiments Service is unavailable Wednesday, November 16, 2011 - 09:52 Wed, Nov 16, 10:06
web sites unvailable Wed, Nov 16, 05:30 Multiple Services Service is unavailable Wednesday, November 16, 2011 - 06:30 Wed, Nov 16, 06:22
External DNS problem Wed, Nov 16, 05:30 Network Infrastructure Service Service is degraded Wednesday, November 16, 2011 - 06:30 Wed, Nov 16, 07:12
SLS is degraded Thu, Nov 10, 09:52 IT Support Services for IT Operations Service is degraded Friday, November 11, 2011 - 13:55 Thu, Nov 10, 09:55
DFS Home server down Wed, Nov 9, 11:05 Not specified Service is unavailable Wednesday, November 9, 2011 - 11:20 Wed, Nov 9, 12:27
BATCH Service affected over the weekend Fri, Nov 4, 23:00 Batch Service Service is degraded Monday, November 7, 2011 - 17:30 Mon, Nov 7, 10:35
SRM-EOSATLAS (unable to authenticate CERN certificate) Thu, Nov 3, 14:30 Storage Service for Projects & Experiments Service is degraded Thursday, November 3, 2011 - 16:50 Thu, Nov 3, 17:07
Atlas offline DB services (ATLR) not available Tue, Nov 1, 10:00 DB & Application Platform Service for Projects & Experiments Service is unavailable Tuesday, November 1, 2011 - 10:30 Tue, Nov 1, 10:15
EOSCMS EONOET (unable to access quota space) glitch Fri, Oct 28, 16:03 Storage Service for Projects & Experiments Service is degraded Friday, October 28, 2011 - 16:19 Fri, Oct 28, 17:00
SRM-EOSATLAS out of file descriptors Tue, Oct 25, 12:45 Storage Service for Projects & Experiments Service is degraded Tuesday, October 25, 2011 - 13:45 Tue, Oct 25, 14:08
CMS VOMRS not syncing to VOMS Tue, Oct 25, 11:55 WLCG Proxy Services Service is degraded Wednesday, October 26, 2011 - 20:15 Wed, Oct 26, 12:07
Force10 Linecard crash impacting three services in the CC Tue, Oct 25, 11:24 Multiple Services Service is unavailable Tuesday, October 25, 2011 - 11:27 Tue, Oct 25, 11:45
EOSATLAS: cannot remove files via SRM Mon, Oct 24, 10:15 Storage Service for Projects & Experiments Service is degraded Monday, October 24, 2011 - 12:10 Mon, Oct 24, 12:23
devdb11 not available Fri, Oct 21, 11:03 DB & App Platform for EDMS Service is unavailable Friday, October 21, 2011 - 16:25 Fri, Oct 21, 11:11
DFS Home server Down Thu, Oct 20, 22:30 Not specified Service is unavailable Friday, October 21, 2011 - 05:30 Sat, Oct 1, 18:43
Partial disruption of GSM service Wed, Oct 19, 12:40 Mobile Telephone Service Service is degraded Wednesday, October 19, 2011 - 13:20 Wed, Oct 19, 13:38
One LCG distribution router in the vault crashed this morning at 11:35 Wed, Oct 12, 11:35 Network Infrastructure Service Service is unavailable Wednesday, October 12, 2011 - 11:45 Wed, Oct 12, 11:55
CET Access Problem Wed, Oct 12, 09:26 Financial Reporting Application Support Service is degraded Wednesday, October 12, 2011 - 10:30 Wed, Oct 12, 09:32
SRM-EOSATLAS stuck Mon, Oct 10, 19:30 Storage Service for Projects & Experiments Some applications linked to services are unavailable Monday, October 10, 2011 - 20:20 Mon, Oct 10, 20:32
AFS server problem Mon, Oct 10, 17:27 Storage Service for Projects & Experiments Service is degraded Monday, October 10, 2011 - 17:45 Mon, Oct 10, 17:28
Locks on Oracle HR and related applications Mon, Oct 10, 09:29 Multiple Services Service is degraded Monday, October 10, 2011 - 12:11 Mon, Oct 10, 09:31
High CPU load on the GPN router at SAFEHOST Mon, Oct 10, 09:00 Network Infrastructure Service Service is degraded Monday, October 10, 2011 - 12:00 Mon, Oct 10, 16:32
Spontaneous reboot of CMSR database node 3 Thu, Oct 6, 17:30 DB & Application Platform Service for Projects & Experiments Service is degraded Thursday, October 6, 2011 - 17:45 Thu, Oct 6, 17:38
Coupure de reseau sur plusieurs barraques au point 5 Wed, Oct 5, 17:00 Network Infrastructure Service Service is unavailable Wednesday, October 12, 2011 - 17:00 Thu, Oct 6, 09:00
Problems with the Service Desk tool (Service-now) Tue, Oct 4, 08:30 Process Application Support Service is unavailable Tuesday, October 4, 2011 - 10:55 Tue, Oct 4, 09:28
Accidental removal of Chrome browser Fri, Sep 30, 21:54 Windows Desktop Service Service is degraded Tuesday, October 4, 2011 - 10:00 Fri, Sep 30, 22:04
DFS Home server Down Thu, Sep 29, 09:20 Not specified Service is unavailable Thursday, September 29, 2011 - 09:40 Thu, Sep 29, 09:32
PDBR production database partially unavailable Thu, Sep 29, 00:15 DB & Application Platform Service for Projects & Experiments Service is degraded Thursday, September 29, 2011 - 00:35 Thu, Sep 29, 00:53
CMS offline production DB (CMSR) unavailable Tue, Sep 27, 14:05 DB & Application Platform Service for Projects & Experiments Service is unavailable Tuesday, September 27, 2011 - 15:10 Tue, Sep 27, 15:56
0513 S-0034: network problem since 20:08. Sat, Sep 24, 21:10 Under Investigation Some applications linked to services are unavailable Saturday, September 24, 2011 - 22:15 Sat, Sep 24, 21:22
SLS shows "grey" services Fri, Sep 23, 22:25 IT Support Services for IT Operations Service is unavailable Saturday, September 24, 2011 - 11:00 Mon, Sep 26, 09:51
CMSONR Database reduced availibility Fri, Sep 23, 05:30 DB & Application Platform Service for Projects & Experiments Service is degraded Friday, September 23, 2011 - 09:45 Fri, Sep 23, 08:06
AFS server problem Mon, Sep 19, 10:15 Storage Service for Projects & Experiments Service is degraded Monday, September 19, 2011 - 10:50 Mon, Sep 19, 10:41
Account Management Service unavailable Mon, Sep 19, 10:10 Multiple Services Service is degraded Wednesday, September 21, 2011 - 23:25 Mon, Sep 19, 10:12
Only for IT Service Managers: machine_excpetion alarm on kernel 2.6.18-274.el5 Fri, Sep 16, 10:43 Multiple Services Service is degraded Thursday, September 29, 2011 - 16:21 Fri, Sep 16, 11:05
EOSCMS files inconsistency Fri, Sep 16, 08:29 Storage Service for Projects & Experiments Service is degraded Friday, September 16, 2011 - 13:00 Fri, Sep 16, 11:36
Monitoring of OIS services down Wed, Sep 14, 09:30 Multiple Services Service is unavailable Wednesday, September 14, 2011 - 10:00 Wed, Sep 14, 09:45
High risk of unsafe web browsing Tue, Sep 13, 09:37 Multiple Services Service is degraded Tuesday, September 13, 2011 - 09:00 Tue, Sep 13, 09:43
Partial unavailability of LCG database Sun, Sep 11, 19:30 DB & Application Platform Service for Projects & Experiments Service is degraded Sunday, September 11, 2011 - 20:45 Sun, Sep 11, 21:30
CASTORPUBLIC problem Sat, Sep 10, 09:15 Storage Service for Projects & Experiments Service is unavailable Saturday, September 10, 2011 - 11:45 Sat, Sep 10, 11:29
Replication of ATLAS data to Tier1 sites UNAVAILABLE Wed, Aug 31, 12:35 DB & Application Platform Service for Projects & Experiments Service is unavailable Wednesday, August 31, 2011 - 16:30 Wed, Aug 31, 13:16
MyProxy Service Patch Applied Wed, Aug 31, 09:00 WLCG Proxy Services Service is degraded Wednesday, August 31, 2011 - 09:30 Wed, Aug 31, 09:42
Problem with EDH Application Mon, Aug 29, 10:15 Multiple Services Service is unavailable Monday, August 29, 2011 - 10:20 Mon, Aug 29, 11:15
Power cut in the Service Desk Wed, Aug 24, 12:35 Service Management Service Service is degraded Wednesday, August 24, 2011 - 12:54 Wed, Aug 24, 14:50
Power cut in the Service Desk Wed, Aug 24, 11:50 Service Management Service Service is degraded Wednesday, August 24, 2011 - 12:00 Wed, Aug 24, 12:22
DNS Load Balence Service is Faulty Wed, Aug 24, 10:00 IT Support Services for IT Operations Service is degraded Wednesday, August 24, 2011 - 14:00 Wed, Aug 24, 13:26
lxplus alias not load balancing correctly. Wed, Aug 24, 10:00 Interactive Linux Service Service is degraded Wednesday, August 24, 2011 - 14:00 Wed, Aug 24, 13:12
Two web servers down - some web sites are not reachable Wed, Aug 24, 08:25 Web Service Service is degraded Wednesday, August 24, 2011 - 09:20 Wed, Aug 24, 09:18
Virtual Machines hosting Physics Services unavailable Wed, Aug 24, 05:30 Server Hosting Service Service is degraded Wednesday, August 24, 2011 - 09:30 Wed, Aug 24, 09:17
webafs02 overloaded Mon, Aug 22, 15:30 Web Service Service is degraded Monday, August 22, 2011 - 16:30 Mon, Aug 22, 16:41
EDH CHIS declaration of health insurance situation spouse/partner is failing Mon, Aug 22, 14:54 Under Investigation Service is degraded Tuesday, August 30, 2011 - 13:00 Mon, Aug 22, 15:10
Account Management Service running slowly Mon, Aug 22, 12:26 Multiple Services Service is degraded Monday, August 22, 2011 - 17:23 Mon, Aug 22, 12:29
Major Power Cut Thu, Aug 18, 11:50 Multiple Services Service is degraded Thursday, August 18, 2011 - 14:40 Thu, Aug 18, 14:17
Unavailability of CMSONR database Thu, Aug 18, 11:45 DB & Application Platform Service for Projects & Experiments Service is unavailable Thursday, August 18, 2011 - 15:40 Thu, Aug 18, 12:17
Web sites hosted on WEBAFS02 not available Mon, Aug 15, 11:15 Web Service Service is degraded Monday, August 15, 2011 - 11:35 Mon, Aug 15, 11:21
AFS instability affecting several volumes Mon, Aug 15, 10:30 Storage Service for Projects & Experiments Some applications linked to services are unavailable Monday, August 15, 2011 - 11:35 Mon, Aug 15, 10:58
VOMS service for LHC VOs degraded. Wed, Aug 10, 09:00 WLCG Proxy Services Service is degraded Wednesday, August 10, 2011 - 17:30 Wed, Aug 10, 18:04
CASTOR central nameserver DB overload Tue, Aug 9, 19:30 Storage Service for Projects & Experiments Service is degraded Tuesday, August 9, 2011 - 20:30 Tue, Aug 9, 22:07
CASTORLHCB headnode failure Thu, Aug 4, 09:33 Storage Service for Projects & Experiments Service is degraded Thursday, August 4, 2011 - 09:45 Thu, Aug 4, 09:37
Emergency reboot of GPN core router Wed, Aug 3, 12:45 Network Infrastructure Service Service is degraded Wednesday, August 3, 2011 - 13:11 Wed, Aug 3, 12:45
LHCBR database problems Tue, Aug 2, 22:22 DB & Application Platform Service for Projects & Experiments Some applications linked to services are unavailable Tuesday, August 2, 2011 - 23:05 Tue, Aug 2, 23:25
Mysql backend not responding. Tue, Aug 2, 05:45 General Purpose DB & App Platform Service is unavailable Tuesday, August 2, 2011 - 07:00 Tue, Aug 2, 07:27
EOSCMS unavailable on 2011-07-28 Thu, Jul 28, 00:10 Storage Service for Projects & Experiments Service is unavailable Thursday, July 28, 2011 - 09:40 Thu, Jul 28, 10:03
lxplus logins are failing Mon, Jul 25, 14:22 Interactive Linux Service Service is unavailable Tuesday, July 26, 2011 - 10:00 Mon, Jul 25, 14:23
Drupal service unavailable. Wed, Jul 20, 23:00 Web Service Service is unavailable Thursday, July 21, 2011 - 09:50 Thu, Jul 21, 09:53
Homefolder of letter t unavailable Tue, Jul 19, 22:00 Multiple Services Service is unavailable Wednesday, July 20, 2011 - 10:00 Wed, Jul 20, 10:09
SharePoint un-available Tue, Jul 19, 09:20 Multiple Services Service is unavailable Tuesday, July 19, 2011 - 09:40 Tue, Jul 19, 09:35
Delivery of some e-mails from CERN to physik.uni-muenchen.de delayed or failed Mon, Jul 18, 14:00 E-Mail Service Service is degraded Thursday, July 21, 2011 - 11:00 Thu, Jul 21, 15:04
AIS Authorization password cannot be changed Mon, Jul 18, 11:57 Multiple Services Service is degraded Monday, July 18, 2011 - 19:00 Mon, Jul 18, 12:00
Lxbatch is currently down Sun, Jul 17, 15:43 Batch Service Service is unavailable Sunday, July 17, 2011 - 18:16 Sun, Jul 17, 15:46
Drupal-based websites - outages Tue, Jul 12, 16:55 Web Service Service is degraded Tuesday, July 12, 2011 - 17:30 Tue, Jul 12, 16:59
Mobile phone services Tue, Jul 12, 14:39 Mobile Telephone Service Service is degraded Tuesday, July 12, 2011 - 15:30 Tue, Jul 12, 15:06
Service-now unavailable Tue, Jul 12, 08:47 Service Management Service Service is unavailable Tuesday, July 12, 2011 - 08:55 Tue, Jul 12, 08:51
Problems to call CERN mobiles Mon, Jul 11, 09:30 Mobile Telephone Service Service is unavailable Monday, July 11, 2011 - 10:30 Mon, Jul 11, 11:58
CMSONR unavailability Sun, Jul 10, 18:20 DB & Application Platform Service for Projects & Experiments Service is unavailable Sunday, July 10, 2011 - 23:00 Sun, Jul 10, 19:15
Power cut Sun, Jul 10, 13:15 Multiple Services Service is degraded Sunday, July 10, 2011 - 23:40 Sun, Jul 10, 23:50
ALICE online database unavailable Sun, Jul 10, 12:10 DB & Application Platform Service for Projects & Experiments Service is unavailable Sunday, July 10, 2011 - 16:40 Sun, Jul 10, 13:09
LHCB online database unavailable Sun, Jul 10, 12:10 DB & Application Platform Service for Projects & Experiments Service is unavailable Sunday, July 10, 2011 - 13:45 Sun, Jul 10, 13:02
Interruption of 2 LCG network services at 14h30 Thu, Jul 7, 14:30 Network Service for Projects & Experiments Service is unavailable Thursday, July 7, 2011 - 14:35 Thu, Jul 7, 14:22
ATLAS offline database unavailability Thu, Jul 7, 10:15 DB & Application Platform Service for Projects & Experiments Service is unavailable Thursday, July 7, 2011 - 11:00 Thu, Jul 7, 11:00
SVN slow due to high load Wed, Jul 6, 12:32 GRID Development Service Service is degraded Wednesday, July 6, 2011 - 12:41 Wed, Jul 6, 12:36
Baan6 login problem Wed, Jul 6, 08:29 DB & App Platform for AIS Service is unavailable Wednesday, July 6, 2011 - 10:20 Wed, Jul 6, 08:33
Additional downtime for CASTOR CMS Tue, Jul 5, 16:42 Storage Service for Projects & Experiments Service is unavailable Tuesday, July 5, 2011 - 19:45 Tue, Jul 5, 16:44
LCG network infrastructure Tue, Jul 5, 12:45 Batch Service Service is degraded Friday, July 8, 2011 - 08:45 Tue, Jul 5, 14:28
LHCB online database unavailability Mon, Jul 4, 17:00 DB & Application Platform Service for Projects & Experiments Service is unavailable Monday, July 4, 2011 - 18:15 Mon, Jul 4, 17:28
MyProxy Service Access Blocked at CERN Perimeter Thu, Jun 30, 10:55 WLCG Proxy Services Service is degraded Thursday, June 30, 2011 - 12:05 Thu, Jun 30, 10:58
Oracle HR not working properly Thu, Jun 30, 09:50 Multiple Services Service is degraded Thursday, June 30, 2011 - 11:10 Thu, Jun 30, 09:51
Slow response from SVN Thu, Jun 30, 09:06 GRID Development Service Service is degraded Thursday, June 30, 2011 - 10:02 Thu, Jun 30, 09:11
CASTOR CMS degraded Tue, Jun 28, 16:00 Storage Service for Projects & Experiments Service is degraded Tuesday, June 28, 2011 - 18:00 Tue, Jun 28, 17:08
lxplus logins are failing Mon, Jun 27, 16:06 Interactive Linux Service All applications linked to service are unavailable Monday, June 27, 2011 - 19:30 Mon, Jun 27, 16:08
problems on S513-C-IP63 network service Mon, Jun 27, 10:45 Server Hosting Service Some applications linked to services are unavailable Monday, June 27, 2011 - 11:00 Mon, Jun 27, 14:36
SINDES problem which caused CCM alarms Sun, Jun 26, 10:00 Multiple Services Service is degraded Tuesday, June 28, 2011 - 13:00 Sun, Jun 26, 22:54
xrdcp processes stuck on lxbatch copying files from Castor Sat, Jun 25, 18:20 Under Investigation Service is degraded Tuesday, June 28, 2011 - 10:00 Sun, Jun 26, 13:13