Issue:
We identified that due to a browser update of Chrome late December, microphones no longer connect to the server in Chrome.
Solution:
Temporary Workaround https://de2.meet.coop was deployed before the new year with open registrations running on version 2.3. This would allow members to create temporary rooms without delay for those that needed an immediate workaround.
Migration Process
Plans to migration to version 2.3 which was scheduled early 2022 are accelerated.
Maintenance window announced for Monday January 17, 2022
Successfully migrated of all users on January 17, 2022.
Successfully migrated of all recordings on 18, 2022.
Old de.meet.coop server has been shutdown January 20, 2022.
ca.meet.coop - In process.
Vendor back-orders have made upgrading this server difficult and we are looking at the best way to do an in-place upgrade with minimal downtime. We have mitigated the issue, but still working on finalizing the setup.
Migrated ca.meet.coop to Digital Ocean in Toronto temporarily (fully functional)
Re-deployed the physical server using the new tech stack
Great that these things are being fixed But a little alarming that such things are being broken by a routine upgrade of BBB versions?
Is this due to BBB changes (more worrying)? Or bcos our own devops patches, procedures and system configurations weren’t 100% watertight (a bit worrying, but a typical devops chellenge, that can(?) be resolved thro evolution and shared knowledge, as above)?
There are also other factors. Usually SaaS would have dedicated staff for building the product, rather than take software released by external parties (open source code), so there is more coordination between DevOps and Product.
Usually there are dedicated QA staff to check for regression in multiple environments. Here we have no QA staff to qualify releases.
If there is a production problem, such as the two above, the Product team is one Slack message away. Here our DevOps team has to figure out what what exactly went wrong, out of all possible causes.
The solution of this is staffing for “site / service reliability” and QA, what essentially comes down to money.
Thanks tech.circle! In sum: The community had been asking for upgrading to BBB 2.3 and you planned to do this early 2022. The Chrome induced audio connection error accelerated this upgrade. And in the process we got 100% control over our own infrastructure, now deployed with the code we have in our gitLab repositories. This process has been hard for all of us, especially for our users having a hard time during their meetings, and stressful for tech.circle, who has seen itself forced to run a migration and upgrade to a totally new infrastructure without a controlled testing process. Fortunately they have automated the deployment scripts so for next migrations, changes and upgrades can be done with much less effort.
When we get to more routine updates and upgrades, we can do this is a more controlled way. Where commercial software as a service (SaaS) operators have a development team and a quality assurance team, our QA process involves the user community - we have no paid QA staff as such. So this would look like this: tech.circle first deploys any proposed changes on a development server, then some of the active users test out the changes and when good, tech deploys on the production infrastructure.
The rest of the migration for the CA server has been complete. All meet.coop servers are running BBB version 2.3 and are on the correct servers now!
We also found that the new 2.3 server was not generating the downloadable MP4 recording files due to out dated scripts. We have corrected the scripts and generation of MP4 is now working. If you are missing a recording please e-mail us and we will manually re-generate the missing MP4.
Past rooms report error 404
Recordings entries showing up but links are broken
But a little alarming that such things are being broken by a routine upgrade of BBB versions?