Microphone Issues on Big Blue Button 2.2 (Resolved)

Issue:
We identified that due to a browser update of Chrome late December, microphones no longer connect to the server in Chrome.

Solution:

Temporary Workaround
https://de2.meet.coop was deployed before the new year with open registrations running on version 2.3. This would allow members to create temporary rooms without delay for those that needed an immediate workaround.

Migration Process
Plans to migration to version 2.3 which was scheduled early 2022 are accelerated.

de.meet.coop - Complete

  • Maintenance window announced for Monday January 17, 2022
  • Successfully migrated of all users on January 17, 2022.
  • Successfully migrated of all recordings on 18, 2022.
  • Old de.meet.coop server has been shutdown January 20, 2022.

ca.meet.coop - In process.
Vendor back-orders have made upgrading this server difficult and we are looking at the best way to do an in-place upgrade with minimal downtime. We have mitigated the issue, but still working on finalizing the setup.

  • Migrated ca.meet.coop to Digital Ocean in Toronto temporarily (fully functional)
  • Re-deployed the physical server using the new tech stack

Pending action:

  • Migrate Instance back to the physical server

Original thread

2 Likes

We found two issues after the migration

:white_check_mark: Past rooms report error 404

Cause: Old greenlight instance had /b/ prepended to each room, one instances do not.

Status: We patched the servers to accept both variations.

:white_check_mark: Rooms start without any presentation at all

Cause: Rooms with custom greenlight presentation could not find the uploaded presentations causing a BLACK presentation.

Status: We have restored all custom presentations back in greenlight. Rooms should load correctly now.

:white_check_mark: Recordings entries showing up but links are broken

Cause: Separation of Greenlight and BBB onto two different servers made the links invalid

Status: We updated all the links and the recordings are now working

:white_check_mark: Viewing recordings on the DE server yielded a error 500, but downloads work

Cause: Missing 2.3 player

Status: We added the 2.3 player and flagged to find out why it was missing.

Reference Thread:

Great that these things are being fixed :slight_smile: But a little alarming that such things are being broken by a routine upgrade of BBB versions?

Is this due to BBB changes (more worrying)? Or bcos our own devops patches, procedures and system configurations weren’t 100% watertight (a bit worrying, but a typical devops chellenge, that can(?) be resolved thro evolution and shared knowledge, as above)?

1 Like

There are also other factors. Usually SaaS would have dedicated staff for building the product, rather than take software released by external parties (open source code), so there is more coordination between DevOps and Product.

Usually there are dedicated QA staff to check for regression in multiple environments. Here we have no QA staff to qualify releases.

If there is a production problem, such as the two above, the Product team is one Slack message away. Here our DevOps team has to figure out what what exactly went wrong, out of all possible causes.

The solution of this is staffing for “site / service reliability” and QA, what essentially comes down to money.

3 Likes

This was not a routine upgrade but an overhaul of our core stack. We hope that these new changes will provide a more stable platform moving forward.

3 Likes

Thanks tech.circle! In sum: The community had been asking for upgrading to BBB 2.3 and you planned to do this early 2022. The Chrome induced audio connection error accelerated this upgrade. And in the process we got 100% control over our own infrastructure, now deployed with the code we have in our gitLab repositories. This process has been hard for all of us, especially for our users having a hard time during their meetings, and stressful for tech.circle, who has seen itself forced to run a migration and upgrade to a totally new infrastructure without a controlled testing process. Fortunately they have automated the deployment scripts so for next migrations, changes and upgrades can be done with much less effort.

When we get to more routine updates and upgrades, we can do this is a more controlled way. Where commercial software as a service (SaaS) operators have a development team and a quality assurance team, our QA process involves the user community - we have no paid QA staff as such. So this would look like this: tech.circle first deploys any proposed changes on a development server, then some of the active users test out the changes and when good, tech deploys on the production infrastructure.

4 Likes

Last update on this thread.

The rest of the migration for the CA server has been complete. All meet.coop servers are running BBB version 2.3 and are on the correct servers now!

We also found that the new 2.3 server was not generating the downloadable MP4 recording files due to out dated scripts. We have corrected the scripts and generation of MP4 is now working. If you are missing a recording please e-mail us and we will manually re-generate the missing MP4.

1 Like

is that a typo!? should it say …is now working?

also, when you say “all meet.coop servers” can you remind us which they are?
i.e. does this include demo.meet.coop?

Thanks

1 Like