Azure VM outage

[dduportal]

Service(s)

cert.ci.jenkins.io, ci.jenkins.io, infra.ci.jenkins.io, trusted.ci.jenkins.io

Summary

Since Monday 9 September 2024 at 03:00pm UTC, we see an increased rate of failure to spin up Azure VM agents on ci.jenkins.io.

• Nothing in the Azure status site 🤦
• The Azure portal shows, in the "Azure Monitor" -> "Service Health" section, that we are subject to 2 incidents

🎯 Impact is on the Azure VM agents and sometimes on the Linux container agents when scaling up: builds looks slow because they are stuck waiting for agents to spin up

[dduportal]

Update:

• Impact on today's weekl release 2.476
  • trusted.ci is set up to retain its agents for 60 min (to reuse them) to accelerate build (puppet is disabled)
• Opened incident on status.jenkins.io - open Azure outage status#539
• Added a message on ci.jenkins.io:

[dduportal]

Problem is now gone as Azure teams added capacity and software fixes the 12 and 13 September

• Closing incident on status.jenkins.io: close Azure outage status#540
• Removing message from ci.jenkins.io