Intermitent storage connection problems are affecting Virtual Machines, which become randomly unavailable for a while then may become responsive again.
Update 26/9 08:00: the situation is slowly improving and getting more stable as the stress on the primary storage cell is being reduced (VM storage partially migrated to other storage cells, I/O intensive services such as antivirus scans disabled etc.), but not resolved yet.
The list of potentially affected VMs follows:
atlaslogbook
atlaslogbookt
batchmon02
bukowiecvm01
bukowiecvm02
cmscert
cmscollstat
cmsdasvm1
cmsdasvm2
cmsdasvm3
cmsdasvm4
cmsdasvm5
cmsdasvm6
cmslogbook
cmslogbookt
cmsperfpubvm
cmsperfvm
cmsperfvmdev
cmspubperfvm
cmssdtdev01
cmstrko2ovm
cmstrko2ovm02
creamtest001
dashboard24
dashboard25
dashboard26
dashboard27
dashboard28
dashboard29
dashboard30
dashboard31
evoportal
fts301
fts302
gridmsg003
historydqmweb
lcggenser3
lemon2build03
lemon2build04
lfcatlas01
lfclhcbro01
lfclhcbro02
lfclhcbrw01
lfclhcbrw02
lfclhcbrw03
lfcshared01
lfcshared02
lxcvm001
lxcvm002
lxcvm003
lxcvmfs01
lxdev61
lxdev62
lxdev63
lxdev64
lxlahey03
lxlic02
lxlic06
lxlic07
lxsvn01
mcwin01
musclefit
osadmin01
pcnds01
pcudsdev2
phedex-web-dev
sindesdev02
sindesdev03
slsdev02
smtarch02
tsmmsdev01
vmeos01
voatlas150
voatlas166
voatlas167
voatlas168
voatlas169
voatlas170
voatlas171
voatlas172
voatlas173
voatlas195
voatlas196
voatlas197
voatlas198
voatlas199
voatlas200
voatlas201
voatlas202
voatlas203
voatlas204
voatlas209
vocms01
vocms07
vocms12
vocms129
vocms130
vocms131
vocms132
vocms133
vocms134
vocms135
vocms137
vocms152
vocms153
vocms154
voms304
voms306
voms308
vona6101
vpcgiordano
Updates
There has been no new storage
There has been no new storage connection problems since 5 AM on 26/9. The situation seems stable enough to consider the incident closed. We'll keep working on reducing the stress on the VM storage subsystem to prevent similar problems in the future.