Service Incidents

Central Git Service

Georgios Koloventzos — Wed, 09 Oct 2013 09:10:17 +0000

Kerberos authentication method is down.

Date & time of incident:

Wednesday, October 9, 2013 - 11:00

Resolution date:

Wednesday, October 9, 2013 - 11:27

Status:

Resolved

Service Element Affected:

Not specified

Impact:

Service is degraded

Workaround:

Please use the ldap authentication instead.

Posted by:

IT-PES

Unit responsible for resolution:

IT Department

CASTOR SRM degraded

Xavier Espinal Curull — Mon, 07 Oct 2013 09:23:35 +0000

Following CASTOR v14 nameserver update we are observing failures on SRM side, they are mainly coming from our internal test probe. Investigating.

Date & time of incident:

Monday, October 7, 2013 - 11:20

Resolution date:

Monday, October 7, 2013 - 12:30

Status:

Resolved

Service Element Affected:

CASTOR Service

Specific Service detail:

SRM services

Impact:

Service is degraded

Posted by:

IT-DSS

Unit responsible for resolution:

IT Department

Intermittent Problems accessing Git via kerberos

David Asbury — Thu, 10 Oct 2013 09:20:24 +0000

There are intermittent problems using kerberos access to the Git service.

Often the error: 500 Internal Server Error while accessing is returned.

We are working with IT/OIS group to understand the problem and find a lasting solution.

Date & time of incident:

Monday, October 7, 2013 - 10:59

Status:

Experts are working on a resolution

Service Element Affected:

Multiple Services

Impact:

Service is degraded

Workaround:

Often repeating the command is enough, but using LDAP authentication should avoid this problem. (See KB0002213 for information about Git access methods.)

Posted by:

IT-PES

Unit responsible for resolution:

IT Department

EOSPUBLIC outage

Jan Iven — Mon, 07 Oct 2013 06:31:34 +0000

The EOSPUBLIC namespace had crashed and needed manual intervention to start.

Date & time of incident:

Sunday, October 6, 2013 - 11:00

Resolution date:

Monday, October 7, 2013 - 08:15

Status:

Resolved

Service Element Affected:

EOS Service

Specific Service detail:

EOSPUBLIC

Impact:

Service is unavailable

Posted by:

IT-DSS

Unit responsible for resolution:

IT Department

Kerberos access to Git fails

Nils Hoimyr — Thu, 03 Oct 2013 15:31:25 +0000

Following an intervention to update server certificates on the central Git servers, Kerberos access to git fails on the central Git service.

Date & time of incident:

Thursday, October 3, 2013 - 17:28

Resolution date:

Thursday, October 3, 2013 - 17:40

Status:

Resolved

Service Element Affected:

Multiple Services

Specific Service detail:

Git service

Impact:

Service is degraded

Workaround:

Please use LDAP authentication (username/password) as a workaround to authenticate to git.cern.ch. Please refer to KB0002209 for more details.

Posted by:

IT-PES

Unit responsible for resolution:

IT Department

Public Information Screens General issue

Thomas Baron — Tue, 01 Oct 2013 07:28:54 +0000

Since midnight today, all public information screens on site are black due to a technical problem which is now fully understood. Experts are working on solving this issue as soon as possible.

Date & time of incident:

Tuesday, October 1, 2013 - 09:25

Resolution date:

Tuesday, October 1, 2013 - 16:30

Status:

Resolved

Expected resolution or Next Update Time:

Tuesday, October 1, 2013 - 18:00

Service Element Affected:

Public Information Display Service

Impact:

Service is unavailable

Posted by:

IT-CIS

Unit responsible for resolution:

IT Department

Commits to Puppet currently not possible

Juan Manuel Guijarro — Mon, 30 Sep 2013 09:41:31 +0000

Commits to the central Puppet service are not possible since the system which checks out the Git repository for the Puppet Masters is currently not available. We are investigating this problem.

Service is back at 12:10 am

Date & time of incident:

Monday, September 30, 2013 - 11:35

Resolution date:

Monday, September 30, 2013 - 12:00

Status:

Resolved

Expected resolution or Next Update Time:

Monday, September 30, 2013 - 12:00

Service Element Affected:

Multiple Services

Impact:

Service is degraded

Posted by:

IT-PES

Unit responsible for resolution:

IT Department

Filesystem full on a filer

Eric Cano — Tue, 01 Oct 2013 08:01:51 +0000

The filer's space monitoring did not distinguish between free space for live filesystem and free space for snapshots. The former slowly reached zero undetected. The full filesystem was resized immediately by the filer team. The monitoring was fixed during the day. During the incident, CATIA SmarTeam users could not check in their work. No data was lost.

Date & time of incident:

Monday, September 30, 2013 - 09:00

Resolution date:

Monday, September 30, 2013 - 09:47

Status:

Resolved

Service Element Affected:

Multiple Services

Specific Service detail:

Filer service

Any other affected service(s):

Mechanical Design Software Service

Impact:

Service is in read-only mode

Posted by:

IT-DSS

Unit responsible for resolution:

IT Department

Foreman ENC node returned 404s affecting puppet runs

Ben Dylan Jones — Thu, 26 Sep 2013 08:31:52 +0000

One of the foreman ENC nodes had a crashed Passenger process, causing ENC calls to return 404s. This meant puppet runs failed. Monitoring improvement necessary to fail httpd if Passenger has failed, so that the load balancer can do the right thing.

Date & time of incident:

Thursday, September 26, 2013 - 10:00

Resolution date:

Thursday, September 26, 2013 - 10:20

Status:

Resolved

Service Element Affected:

IT Operations Support Service

Specific Service detail:

Puppet

Impact:

Service is degraded

Posted by:

IT-PES

Unit responsible for resolution:

IT Department

Main CERN Web Site offline

Tim Bell — Sun, 22 Sep 2013 07:07:26 +0000

Following an issue with the Drupal infrastructure, the CERN main web site is offline.

Other Drupal sites were also affected from 08:30 to 08:45.

The issue is being investigated.

Date & time of incident:

Sunday, September 22, 2013 - 08:30

Resolution date:

Sunday, September 22, 2013 - 09:45

Status:

Resolved

Service Element Affected:

Drupal Service

Impact:

Service is unavailable

Posted by:

IT-OIS

Unit responsible for resolution:

IT Department

Virtual Machine creation failing on CERN private cloud

Jan Van Eldik — Fri, 20 Sep 2013 10:05:16 +0000

A configuration issue currently blocks Virtual Machine creation of the Cloud Infrastructure service.

The issue is being solved, full service should be restored by 14:00

Date & time of incident:

Friday, September 20, 2013 - 12:02

Resolution date:

Friday, September 20, 2013 - 14:00

Status:

Resolved

Service Element Affected:

Server Provisioning Service

Impact:

Service is degraded

Posted by:

IT-OIS

Unit responsible for resolution:

IT Department

Network Outage in B613

Computer Operations — Fri, 20 Sep 2013 08:08:33 +0000

Network outage due to an electrical problem. Experts are working on a resolution

Date & time of incident:

Friday, September 20, 2013 - 09:06

Resolution date:

Friday, September 20, 2013 - 14:00

Status:

Resolved

Service Element Affected:

CASTOR Service

Impact:

Service is unavailable

Posted by:

IT-CF

Unit responsible for resolution:

EN-EL (Electrical Distribution)

EOSALICE outage

Jan Iven — Wed, 18 Sep 2013 08:53:16 +0000

After this morning's network intervention and following an issue on the namespace, EOSALICE is unavailable.

Date & time of incident:

Wednesday, September 18, 2013 - 10:00

Resolution date:

Wednesday, September 18, 2013 - 11:40

Status:

Resolved

Service Element Affected:

EOS Service

Specific Service detail:

EOSALICE

Impact:

Service is unavailable

Posted by:

IT-DSS

Unit responsible for resolution:

IT Department

Some database services unavailable or service is degraded

Eric Grancher — Wed, 18 Sep 2013 06:42:47 +0000

Due to some side effect with the network intervention, a number of databases are currently not available or the service is degraded.

The responsible are working on the issue and will update this page.

The services have been fully restored at 12:13, apologies for the inconvenience.

Date & time of incident:

Wednesday, September 18, 2013 - 07:40

Resolution date:

Wednesday, September 18, 2013 - 12:13

Status:

Resolved

Expected resolution or Next Update Time:

Wednesday, September 18, 2013 - 12:00

Service Element Affected:

Accelerator Database Service

Impact:

Service is unavailable

Posted by:

IT-DB

Unit responsible for resolution:

IT Department

Service Consolidation Service affected by routers upgrade

Juan Manuel Guijarro — Wed, 18 Sep 2013 09:03:37 +0000

As a side effect of the computer center router upgrade this morning, several Virtual Machines hosted in the Service Consolidation Service crashed and had to be rebooted.

This was due to some of the Hypervisors needing to be recovered since they lost connection to their dedicated shared storage.

Date & time of incident:

Wednesday, September 18, 2013 - 05:00

Resolution date:

Wednesday, September 18, 2013 - 11:00

Status:

Resolved

Expected resolution or Next Update Time:

Wednesday, September 18, 2013 - 11:30

Service Element Affected:

Multiple Services

Impact:

Service is degraded

Posted by:

IT-PES

Unit responsible for resolution:

IT Department

Spam from Service-now - Our apologies

Nicole Cremel — Wed, 18 Sep 2013 08:57:03 +0000

On Monday 17 September, a human error triggered about 15000 “spam like” mail notifications from the Service Desk (Service-now tool).

The problem was understood and has been fixed within 4 mins.

Please accept our apologies if you have received this mail by mistake (and just ignore this message).

Note: the message sent by mistake was starting with

Dear LAS Web Interface,
We have created the following ticket concerning your issue: INC386648 "CVMFSProbe for lxplus0397".
...

Date & time of incident:

Tuesday, September 17, 2013 - 17:54

Resolution date:

Tuesday, September 17, 2013 - 17:55

Status:

Resolved

Service Element Affected:

Service Management Service

Specific Service detail:

Service-now

Impact:

Service is degraded

Posted by:

IT-DI

Unit responsible for resolution:

IT Department

Problems accessing https://pptevm.cern.ch/ppteu2

Artur Wiecek — Tue, 17 Sep 2013 09:11:07 +0000

Access to https://pptevm.cern.ch/ppteu2
is not possible at the moment due to authentication problems

Date & time of incident:

Tuesday, September 17, 2013 - 11:07

Resolution date:

Tuesday, September 17, 2013 - 12:05

Status:

Resolved

Service Element Affected:

Weblogic Service

Impact:

Service is unavailable

Posted by:

IT-DB

Unit responsible for resolution:

IT Department

ServiceNow access to ITIL cut for 15 mins

Zhechka Toteva — Fri, 13 Sep 2013 09:16:09 +0000

Due to misconfiguration, ITIL users did not have access to some information on their tickets for 15 mins.

Sorry for the inconvenience.

Date & time of incident:

Friday, September 13, 2013 - 10:28

Resolution date:

Friday, September 13, 2013 - 10:43

Status:

Resolved

Expected resolution or Next Update Time:

Friday, September 13, 2013 - 10:43

Service Element Affected:

Service Management Service

Impact:

Service is degraded

Posted by:

IT-DI

Unit responsible for resolution:

IT Department

Git DNS lookup failing

Nils Hoimyr — Thu, 12 Sep 2013 07:53:47 +0000

Git alias failing - git service unavailable.

Due to a problem with the Puppet configuration on the Git servers, the DNS alias lookup to the Git cluster failed since Wednesday evening. The problem has been solved Thursday morning, and further investigations with a root-cause analysis are on-going.

Sorry for the inconvenience caused by this interruption.

Date & time of incident:

Wednesday, September 11, 2013 - 19:00

Resolution date:

Thursday, September 12, 2013 - 09:00

Status:

Resolved

Service Element Affected:

Not specified

Specific Service detail:

Git distributed version control service (https://cern.service-now.com/service-portal/service-element.do?name=git-service)

Impact:

Service is unavailable

Posted by:

IT-PES

Unit responsible for resolution:

IT Department

afs240 (hosting atlas + some other afs projects) outage

Dan Van Der Ster — Wed, 11 Sep 2013 11:23:30 +0000

afs240, an AFS server hosting some atlas and other (svn, git, ...) project volumes, had a frozen fileserver process. Some volumes were unavailable during this time.

Date & time of incident:

Wednesday, September 11, 2013 - 12:14

Resolution date:

Wednesday, September 11, 2013 - 13:21

Status:

Resolved

Service Element Affected:

AFS Service

Impact:

Service is degraded

Posted by:

IT-DSS

Unit responsible for resolution:

IT Department