[Incident report] Multiple Greenlight instances unavailable

Tech circle Systems
1

Start date: 2021-09-30 14:20 UTC
End date: 2021-09-30 17:30 UTC

Description
The expiration of a root certificate caused some of our Greenlight instances to fail when trying to establish a secure connection with our BigBlueButton servers. When this happened, users on those instances saw an error message and were unable to access their rooms or their recordings. Affected instances were:

  • ceesc.meet.coop
  • foei.meet.coop
  • ournetworks.meet.coop
  • tor.meet.coop
  • ca.meet.coop

Other Greenlight instances were using a more recent version of the list of trusted root certificates that didn’t include this one.

Resolution
Once the issue was identified, we manually removed the offending certificate from the affected instances. Afterwards, the instances were updated to the more recent, unaffected version.

Next steps
We will work with the Technical Circle on two improvements:

  • Take steps to make sure that all our Greenlight instances are using the same, up-to-date versions.
  • Improve our monitoring to alert us when a Greenlight instance can’t communicate with the BBB backend.

Our apologies for the inconvenience!

2

Great, thank you for the report! Glad it was a relatively minor issue.

1 Like
3

Just a small point of clarification for those that may be affected by the same situation and need a fix/patch:

We did not remove the offending certificate from the client, instead we removed the ISRG Root X1 from the full chain provided by nginx. This forces the client to find a better non-expired path to root which bypasses the offending certificate.

1 Like