CERN Accelerating science

Foreman performance problem

 
Date & time of incident: 
Wednesday, May 8, 2013 - 14:23
Post date: 
Wednesday, May 8, 2013 - 14:31
Incident Description: 

Due to performance problems affecting the Foreman service, the automatic registration of puppet managed virtual machines after boot process is sometimes failing. This performance problem may be also affecting Puppet runs on certain machines.

Foreman service managers are currently taking the necessary steps to increase the resources (number of nodes) which provide the service, as well as to reduce unnecessary workload on it.

Service Element Affected: 
Not specified
Impact: 
Service is degraded
Status: 
Resolved
Resolution date: 
Tue, May 14, 17:00
Expected resolution or Next Update Time: 
Wednesday, May 8, 2013 - 16:00
Posted by: 
IT-PES
Unit responsible for resolution: 
IT Department

Updates

Posted May 13, 2013 - 4:10pm

There is extra capacity running a new faster parser. We're upgrading one of the older workers, and will address the final worker tomorrow.

Posted May 10, 2013 - 5:06pm

This is still work in progress. The service is functional and a final update will be provided next week.

Posted May 8, 2013 - 2:53pm

We have disabled fact & report upload to foreman, and as such foreman is now usable, and puppet runs successful. We are working on adding extra capacity.