An update from me. Not the most pleasant news so brace yourselves. Please feel free to give comments and feedback in this thread.
Please note I have a new email as well support at tech care dot coop as me and my fellow worker start to formalise this type of work into a new co-op called Tech Care Co-op.
Budget
My work has stalled. My 10 hour discovery budget has run out and I have pitched to @Graham a plan for sorting all the tech issues with meet.coop but I haven’t received any feedback on it. It will be a significant cost to the meet.coop OpenCollective but will get us into a sustainable and professional place to keep meet.coop running into the future.
I have estimated I will need 30 hours of budget to deliver the plan I have come up with (1500 GBP).
Discovery
I had to make some of the discovery results private due to major security issues discovered in the platform. I didn’t receive much feedback on this and didn’t feel like the urgency of the situation was really addressed properly so therefore I am resorting to public disclosure in the interests of transparency with the core team and the community.
Currently the servers are running vulnerable unsupported versions of Ubuntu 18.06 and Big Blue Button 2.7.x. They have not been maintained for years.
There is some energy from folks in the community such as @Graham, @wouter, @mikemh and David Jacovkis (@Graham has been my main facilitator supporting my work) to get things to a better place but the communications have been very dysfunctional and getting the access needed to carry out the work has been extremely demotivating where I’m often waiting weeks to a month for a reply via email or Matrix. I’m sure these folks are doing the best they can (and I give my thanks for that) but we need to step up efforts to get to a better place and towards being a functional co-op so situations like this don’t happen again.
Rescue plan
I submit an expense for 30 hours labour at 50 GBP per hour to the OpenCollective which the community consents to and is approved.
I still need access to sso.meet.coop (necessary for migration).
I still need access to the meet.coop DNS records (necessary for migration).
Contract with the German co-op ColloCall to migrate ca.meet.coop and de.meet.coop to new hosts with up to date instances of Big Blue Button which they will maintain going forwards. This will be a significant expense but is the right move I think. They will provide a professional service.
My co-op Autonomic and WebArchitects, will be contracted to investigate, update and maintain sso.meet.coop to ensure that server is maintained professionally.
The user base is encouraged to download their own recordings from the servers as an extra layer of backup.
The user base is kept up to date on progress, maintenance windows and downtime.
Timeline
I would like to see all of this delivered by the end of November due to the urgency of the situation. It is certainly achievable but only if the remainder of the access issues are sorted promptly.
I have already made contact with ColloCall and will try and get a sync meeting with them next week to brief them on the situation so they can prepare their qoute.
I thought it was important to increase the level of transparency of the situation in order to keep the userbase community up to date. I think we should proceed with more honesty and transparency on the bad situation we are in. I think it will empower and motivate all of us to work on solving this with more community accountability.
I hope folks who I mentioned appreciate that I hold no ill will towards any of you about the current situation. It has been a bit frustrating at times but I am committed to helping out and I want to see meet.coop get to a sustainable place. Hopefully the critiques given are both constructive and comradely.
Thanks for this @kawaiipunk, Webarchitects has only been providing Discourse, Nextcloud and WordPress hosting for the project for some years now, we stepped back from everything else, I hope your proposed rescue plan can be enacted, according to OpenCollective there is money to pay you for this.
This is easy to sort out, @Graham should I grant @kawaiipunk access to the repo?
Thanks, @kawaiipunk for giving the requested status update here. It is certainly time to act urgently.
Those 30 hours will be well invested IMHO. I think also thst it was in accordance with the whole idea of getting you involved, @kawaiipunk ? @Graham although somewhat missing, is the one to know.
What you’ll miss is the access to the SSO server. It was set up by Yurko of the @hypha team IIRC, but possibly in collaboration with someone from Autonomic? David Jacovkis will know and should have access as well.
Personally I should still have an admin account on the sso server, but that won’t help you really, is it?
I will keep an eye out to help where I can.
Thanks all, for rescuing this collective project.
Note that I’m not involved in any formal role any longer since almost 3 years, but of course this project goes to my heart.
It maybe most appropriate to organise a community vote about @kawaiipunk ‘s proposal. Graham as coordinator would be the best to organise that, but in his absence I’d be willing to organise that.
Hi. Apologies if I haven’t responded to anything that i should have. I’m not aware of having stuff on this in my queue, so I think this is probably down to a miscommunication at some point. I though that we were waiting for some costings? I’m keen to progress this, and happy that @wouter is interested ot be involved. Let’s set up a call to discuss agree the plan.
Hi Graham, good to see your reply. Thanks. I’m ok to join a call, although for the collective governance of meet.coop I’d think a community vote on the proposed work & budget would (also) be important (most people won’t join a call, but hopefully a few more would join on online vote in the forum). That gives us some idea of how much people still care?
Apart, there is some server access still lacking, who helps with that? @dvdjaco maybe?
Im with what’s being proposed by @kawaiipunk and endorsed by @wouter and @Graham . If there’s a call I’ll come into it. I agree with Wouter that a community decision should be made - at this point it looks like all the community is in this thread! But…
To be done with haste? @kawaiipunk d’you have the capacity right now?
Pinging @anarcat on this, they were asking in matrix.
Ok, I have been chatting with @anarcat who has helped advise and clarify. I think we should move decisively and with haste.
I have taken an encrypted backup of bbb.ca.meet.coop and bbb.de.meet.coop using Restic to the Autonomic Borgbase account. I need to run it again to include some extra data but all the important client data will be there soon.
Let’s send out comms to all clients/members that they need to download any recordings from the servers in the next few days. Maybe a deadline of Wednesday 22nd at 12 UTC would be appropriate. That would be an extra layer of backups for client data.
ColloCall are asking if I can meet on the week of the 28th Oct. I’m pushing them a little to see if they can meet sooner. I’d love to get new servers spun up asap.
The most security minded diligent option here which is to lock down the current servers. Possibly to the extent that aren’t accessible via anything but ssh (I currently don’t have dashboard access to boot the servers back up if we do boot them down). I don’t know if that’s acceptable to the community. It’s possible service downtime would be 1-4 weeks before we can get everything ship shape.
The less nuclear option is to remove and secure the client data e.g. recordings from the vulnerable servers. That’s the most valuable data we have. It would be a pain to reunite the recordings with their owners as the folder and file names are all long hashes, but it is possible if anyone didn’t get the comms blast and deadline as suggested above.
In the meantime, one thing we can do is get the new Web Archs VPS span up via @chris and get Autonomic to spin up a fresh Keycloak install ready for the migration from sso.meet.coop (when I get access).
I can also submit my invoice to the OpenCollective in anticipation of the outcome of the decision about the plan.
Can I suggest that a small temporary committee is formed to make the decision at a meeting rather than wait for input from the entire community. It won’t be a reversible decision but it will be transparent and accountable to the community going forwards.
i think a good compromise would be to shutdown everything but a skeleton crew of services: keep SSH and a webserver up that keeps a splash page that announces the maintenance. that reduces the attack surface dramatically.
another option is to setup SSO authentication at the webserver level (and not the BBB/application level, which has a larger attack surface) which would protect both the recordings and the applications, at the cost of blocking access to non-registered users.
I don’t see the need for a community vote personally. This is an operational matter, and unless anyone here objects it’s my view that we should just get on wth this. It was always the plan that I’d discussed with @kawaiipunk - it’s just got stalled for the reasons they have outlined.
I had a good idea over breakfast. Let’s get Autonomic to spin up a Jitsi Meet server at jitsi.meet.coop in the meantime that our client base can use. No SSO or anything, just open.
So at least we can provide some sort of service during this transitory period to tide us over. We could have that spun up today on Hetzner.
What about other instances {example}.meet.coop which may or may not use sso.meet.coop?
I don’t have a full list of where are sub domains service lives right now but meet.coop has only five of it’s own servers (not including the Web Archs managed infra which are maintained) so those instances will live on one of those five and are affected by this issue.
Hi folks, I now have DNS access thanks to @chris and others.
I span up https://jitsi.meet.coop/ for our client base to use running Debian 13 and the latest Jitsi Meet Docker image. Autonomic will maintain that VPS for the time being.
Apologies if I am jumping ahead on this, I just want stuff to happen.
I have draft a email to send out to all our members via OpenCollective updates:
https://pad.autonomic.zone/s/TRuNxe0aV# (link not working in Discourse for some reason, you’ll have to copy and paste). If anyone can help with proofing that @Graham@mikemh@wouter that would be super helpful.
We just need to decide whether we will be pulling down the servers or not and if so, what deadline we should add.