Can I recover my recordings despite service being down?

Hello,

The service is down for a while, as I still get the 500 error. There are still a couple of essential recordings associated with my user account that I have not yet downloaded on my local device.

Is there a way to go and download these? And if so, how?

Thank you

@kawaiipunk do you know the answer to the above? :backhand_index_pointing_up:

Hi folks,

So here’s the current situation. Me and Nick have been working on this problem a lot.

Firstly the comms blast that @wouter tried to send out went wrong and wasn’t sent via email on the OpenCollective due to a confusing UI change: Announcement to meet.coop members 2025-10-31 - Open Collective

We desperately need help with comms on this issue. It’s really hard to switch between being a sysadmin and doing comms. I’m trying my best to communicate on the forum but it needs to be more than that. @Graham is super low on capacity at the moment due to personal reasons.

We now have off-site encrypted backups of all the data. It’s quite a lot of data.

This is basically all the recordings from 2021 onwards.

A major difficulty is that without the Greenlight service running, the metadata for those files isn’t human readable and relies on a long hash which in some unclear way, links the recording files with a user account in the Greenlight GUI.

However we can’t spin up the old Big Blue Button and Greenlight apps because they are stuck on an old version which is insecure and we don’t have access to one of the servers still (ca.meet.coop).

We also have users demanding that we take their data offline and delete it as it was stored on insecure servers.

The current plan (its been a long a winding road to get here) is to spin up two new deployments of Big Blue Button and Greenlight on the old 2.7.x version (which is still supported but our new server is on the 3.x release) and import the databases of the old servers so that everyone can still login and archive or delete their folders. Maybe we have that online for one month.

We’re not even convinced of the feasibility of this because of the extremely channelling situation of not having access to ca.meet.coop.

The other option is try and manually reunite members with their recordings, which is a really long and arduous process and we’d have to view and therefore be really cautious with folks data.

The final option is to just say that no, we can’t recover your recordings and them delete them.

So none of these options are easy. We’re open to feedback on what the best route is but ultimately @Graham is acting as our line manager here as so has the authority to make a call on this.

I’m sorry there isn’t better news. Everyone is trying their best. It’s just a really challenging situation. If there is any way you can still support meet.coop it would be helpful.

1 Like

The above plan has been developed somewhat. Here is the first draft of copy for the plan that @Graham requested to unlock any further labour hours from me and Nick. This is currently Work in Progress but I am sharing early for community feedback and knowledge sharing.

We have a plan to reunite our users with their old recordings

ColloCall recommended following steps:

  • Drop the new Greenlight 3

  • Setup a Greenlight 2

  • Import all data (database, recordings)

  • Migrate to Greenlight 3

We responded to CollCall expanding this plan with additional details. We’ll share that with you now:

  • We want to keep the current live de.meet.coop hosted by ColloCall as-is. This is what our customers are currently using.

  • We have two old servers that hold recordings.

  • We can provide the entire /var/bigbluebutton/ directory for both of those servers.

  • The old de.meet.coop /var/bigbluebutton/ directory is 834 GB.

  • The old ca.meet.coop /var/bigbluebutton/ directory is 133 GB.

  • We have a central keycloak server for SSO that we will continue to manage and host ourselves.

  • We have full control over meet.coop DNS records.

Based on what we know, we only wish to proceed with the second and third actions in the ColloCall recommendations above:

  • Set up a Greenlight 2 server (on new hosting)

  • Import all data (recordings, data files, Keycloak)

Regarding the fourth step ColloCall recommended, we won’t need to migrate the to Greenlight 3 as we will spin these recordings servers down after a few months of meet.coop users being able to access their recordings.

However, because we have two old servers, we believe it will be easier for meet.coop users if ColloCall spin up two Greenlight 2.7.x servers so that the users can access a simulacrum of the old setup.

We think we should name the servers:

Based on our research, we believe that the previous Greenlight 2 servers were migrated off of separate databases in 2022. None of the database instances we’ve found have been modified since then, and do not appear functional. All of the BigBlueButton servers appear to store data in trees of structured text files (XML, JSON, that sort of thing).

Therefore, our understanding is that the meet.coop Keycloak authentication server has superseded any SQL database for either Greenlight or BigBlueButton and therefore the /var/bigbluebutton directory will be sufficient to mimic the old servers.

If everything goes as we hope, we will have a situation where the meet.coop users will be able to log in to both recordings.de.meet.coop and recordings.ca.meet.coop with their existing Keycloak accounts, in order to access and download their recordings. Then after a period time (3 months), we spin down recordings.de.meet.coop and recordings.ca.meet.coop.

We are seeking feedback from ColloCall about the feasibility of this plan and how much it’s going to cost meet.coop.

We have some costs from our friends at ColloCall:

  • 500 EUR per month it’s up (lets give one month for recording retrieval?)
  • 714 EUR one off labour cost

I personally think this is a small price to pay for giving our members access to past recordings and gaining a bit of that trust back.

Canadian server recordings

As I understand, the conversation above is about recordings on the German server? Correct?

In which case, do we also need a plan to restore recordings from the Canadian server? Can this be done ‘offline’, prior to the CA server being restored to use? In which case, where might these recordings be accessed: on the functioning German server? Can SSO solve this one for us?

@mikemh under the current plan, the recordings will become available under two new temporary Greenlight servers hosted by ColloCall:

recordings.ca.meet.coop

recordings.de.meet.coop

These will stay up for a month (costs us by the month to have them up), allowing users logging in SSO to retrieve or delete their recordings.

After this is done, the old de and ca servers will be wiped and then eventually the off-site encrypted backups we took of the old servers.

those do not currently resolve, so i assume this is a plan for the future, right?

1 Like

Is it possible to transfer recordings in BBB’s full-feature playback mode (ie richer than mp4 format), from a server that goes extinct, to one that will survive? Or is the only option to download a recording in mp4 or lose it (or delete it)?

The recordings will be available in whatever formats Greenlight supports. This isn’t a custom solution, we’d just be making the recordings available in as similar way to they were originally via the application GUI.

:+1: thanks. And will Keycloak SSO mean that recordings from either CA or GE server will be available? For just a month before being ‘lost forever’?

Sorry to be dumb :frowning:

1 Like

Yes, both past de and ca recordings will be available.

The reason we can’t keep them online for longer is that it costs quite a bit to have more servers online and also there was never any clean up of old recordings so even recordings from lapsed members are available. This is a possible GDPR issue.

We also received a deletion request from a now ex-member that we haven’t yet been able to fulfil so we need to comply with that ASAP.

1 Like