CERN Accelerating science

Intermittent Foreman problems

 
Date & time of incident: 
Friday, September 6, 2013 - 13:13
Post date: 
Friday, September 6, 2013 - 13:15
Incident Description: 

Intermittent problems with the Foreman service are affecting the Configuration infrastructure - we're looking into the problem.

Service Element Affected: 
IT Operations Support Service
Impact: 
Service is degraded
Status: 
Resolved
Resolution date: 
Tue, Sep 10, 15:00
Posted by: 
IT-PES
Unit responsible for resolution: 
IT Department

Updates

Posted September 12, 2013 - 3:55pm

Posted September 12, 2013 - 3:54pm

Problems on the connection to rubygems.org during the upgrade of Foreman to 1.2.2 to fix a CVE affecting our system rendered some parts of foreman unavailable at moments, including the ENC. Registration of machines failed at some points as well as tracked here  

https://its.cern.ch/jira/browse/AI-2969 is more or less related, as it was thought at first that Foreman was the root of that problem. Permissions on the /var/lib/puppet/yaml directory on some Puppet masters were not correct. That affected a very small subset of machines.

All issues mentioned above have been fixed.