the hedgedoc link doesn’t work:
You are not allowed to access this page. Maybe try logging in?
1 Like
note that there’s significant risk of abuse with that, but if you’re used to that, go for it.
https://pad.autonomic.zone/s/TRuNxe0aV works for me, see above, where Discourse even converted it to a clickable link, complete with title. 
LGTM, btw, good job. I’d keep the “TLDR” acronym out of there though.
Are you planning on taking the servers offline? the message doesn’t mention that possibility…
Whilst I appreciate your desire to take action @kawaiipunk , and I agree that action does need to be taken, I do feel that the language and tone in your messages and the draft message that you have written to send to paying supporters is unhelpful. If I were a paying supporter, on receipt of this I would most likely cease my use of the service and stop my financial contributions, hastening the end of this project rather than giving it a decent chance of moving into a more sustainable position. I urge you not to send this.
Given what’s been undertaken already my suggested approach - remembering that as far as we know the existing servers have not been compromised despite the vulnerabilities, and have been online in this state for many months - is as follows:
- We issue a measured statement to users to let them know that we need to take action to upgrade our servers in light of known security concerns.
- We ask that they download any recordings that they may wish to keep, letting them know that we have a backup but can’t guarantee that. We give them a week or two to do that.
- We then take the servers offline. Creating new servers could start sooner of course.
- We provide a stopgap service until the new platform is in place.
We don’t point users to this thread - and I’d personally prefer if this thread was made private, as it represents a security risk in its own right.
We avoid using terms like ‘collapse’, ‘critical’ and ‘predicament’ etc, and we aim to reassure our supporters that things are in hand and under control.
Questions:
- If we can set up a jitsi server in short order can we not set up a BBB server that people can use in the interim? GitHub - bigbluebutton/bbb-install: BASH script to install BigBlueButton in 30 minutes. suggests that this is not necessarily a major time -consuming task, and while it might not be ideal it’s at least the platform they are used to?
- Do we actually need to to run both the Canadian and the German server? Might it be simpler and perhaps cheaper (i.e no need for SSO) to use a single well located but higher spec server
I’ll be happy to edit/draft the comms - this is what I do, but I can’t do it right now as I’m coming to this off the back of 10 hours of hard work. Let’s de-escalate, please.
You’re welcome to edit the draft @Graham. I am not intending to send it without approval, will leave it to others in the co-op, there isn’t a rush.
1 Like
the cat is kind of out of the bag here…
also, it’s kind of naive to imagine an attacker wouldn’t already know about the vulnerability of the server, it’s kind of plastered all over the place. just a simple nmap -v -A ``tor.meet.coop tells me that:
- it runs
OpenSSH 7.6p1 Ubuntu 4ubuntu0.7 - it runs
nginx/1.21.5 - it runs a Prometheus node exporter (1.3.1) on port 9100 which exposes even more information:
- it was built with go1.17.3
- the server hasn’t been rebooted in 576 days
- it’s a VM running under qemu 1.14.0-0-g155821a1990b
- it runs Ubuntu 18.04.6 LTS (Bionic Beaver)
- the kernel
Linux 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023
From there, it is relatively trivial to figure out a couple of exploits that could at least take the server down, I would bet. But I haven’t actually performed the security audit required to assess the criticality of this.
I would be extremely worried by any server running a two year old Linux kernel, to say the least.
Finally, as a client of meet.coop, i really appreciate the transparency here, it is long overdue.
2 Likes
Thanks @kawaiipunk to take the initiative to draft a message for the community (later I could access it!). Maybe @Graham you could adjust the message to make it a bit less alarming, though @anarcat is right that we are in a critical situation - transparency is needed, and especially urgent action.
Also nice that we have a jitsi backup service now. Of course it would be been nicer if that’d be BBB, but given this as a breakfast action by @kawaiipunk I think we can only be grateful.
I’m happy with all of you taking care. If there would be further collective action, like a community meeting or so to draw lessons from where we are, how we got her, how we may move forward, or possibly to make some collective decisions, then I’m happy to join in.
I’ve edited the message text, re-ordering for sense and modifying the language a little so as to avoid setting unnecesary hares running, and I’ve added a suggested date and time when the current servers will be taken down. giving users a few days to download recordings should they wish to. This time and date is a suggestion, and as I’m not actively involved in the work it’s not my place to set this timeline, but it does need to be communicated.
Feel free to make further edits as needed.
Thanks to all for moving this forward.
2 Likes
let’s keep this ball rolling… @kawaiipunk can you check the date and are you happy with the modified version of the announcement? When we’re set, I can publish the message in the OC news announcement so it gets published there and distributed to all our OC contributors.
Hello all, just to add a user perspective, I think the transparency is good because I had the feeling that meet.coop was ailing (and not really communicating) but now I know more about the critical status.
Does this situation mean that the services are not usable for some time? I can’t login to the German server right now.
Wish you all the best with recoveries.
P.s.: I haven’t seen any user announcement in my inbox..
Hola!
Just a quick note to say that Remix the commons may provide some €€ to support the work of @kawaiipunk on meet.coop. We’ll discuss that next week (monthly circle). I hope others users will join.
2 Likes
Hi this is a new error but I fear that may have been me as I was tinkering with https://de.meet.coop/ and I accidentally restarted the Docker containers for Greenlight. I sent a panicky email to some of the crew but then managed to get the containers going again. This could be tricky to troubleshoot as I don’t have access to sso.meet.coop. Will see what is possible.
If you need somewhere to meet in the meantime, please use our Jitsi if you are able. We’ll try and get this fixed.
2 Likes
Thanks! This would be super useful. Is anyone a member of social.coop e.g. @andrewe I noticed there is a vote to donate some funds in the next month. Could we put forward meet.coop?
1 Like
Hi little update from me. I’ve been really ill with flu for the last week so apologies for the lack of update.
Here is my current todo list:
- Submit invoice to OpenCollective for 30 hours labour to deliver this migration
- Try and poke again for access to sso.meet.coop
- Try and troubleshoot the de.meet.coop issue
- Try and setup a meeting with ColloCall asap (they offered some slots but I was ill)
- Ask Autonomic (the co-op where my day job is) to spin up a Keycloak Single Sign On server on a WebArchitects VPS for a future migration. Make sure we have a good amount of users added there for beach factor (need a list/policy?).
Quite a lot there 
Some more details, the current plan is as @Graham said to fallback to one really beefy dedicated server with ColloCall and pay them to manage that.
The goal is that every component of the infra will be managed by a responsive and reliable tech worker co-op, as the Web Archs stuff has been this whole time. This will leave the members and community to focus on all the organising/business/creative stuff.
The main blocker right now is ssh/root access to sso.meet.coop and ca.meet.coop. Without this access, it’s not possibly to fully complete the migration.
1 Like
@wouter @Graham the message draft is looking brilliant now, much more professional. Pls send out on OC asap.
1 Like
Happy that you recovered, fully I hope, @kawaiipunk ? Following your instructions I published the announcement at the OC portal.
Observation: Possibly the distribution by OC through email has changed or isn’t operative any longer? That is the impression I got through their interface, and also … there are no announcement emails in my box. I’ll post it in the matrix channel to make sure it is distributed more widely.
1 Like
Agree, looks good. I took a final pass a few weeks ago (I believe I made the last two revisions).
I am also waiting to hear back from Open Collective on this. I sent a social.coop member update but am unsure if it was sent/delivered via email.
1 Like
Hey folks,
Latest update from me.
SSO server
@Graham is going to give me access to the meet.coop Digital Ocean team so I can try and find a way into the sso.meet.coop VPS. As far as I know, there’s custom code there that we need that syncs Keycloak with OpenCollective membership and we need the Keycloak data of our customers at the very least.
ColloCall
ColloCall haven’t replied to my email with questions since 30/10. I sent them a boop today. They sent us a quote but there’s a few options I wasn’t clear on.
Keycloak
@chris has spun up a VPS for the new sso.meet.coop server and I will ask Autonomic to deploy the new Keycloak app asap ready for the data migration from the old server.
Name for our new server
Any idea for a domain name for the new server? A few ideas to start off:
I can make a poll for a few options if they are commented here and we can decide.
Invoices
I put my invoice in for the 30 hours labour for the migration: 30 hours labour to faciliate infrastructure migration · Expense #273396 - Open Collective
Web Architects will send theirs soon for the new VPS server. They are consolidating the price for the apps they already host. Total TBC.
There is a expense from @gcotnoir submitted on Oct 9, 2025: Montreal server January to May 2025 · Expense #268915 - Open Collective. I would however be totally frank and say that if that invoice if the agreement for that fee is meant to include the labour of applying application and OS upgrades then the work has not been carried out from my perspective and upgrades haven’t been applied to a professional standard for the past two years. I will leave to others to approve this one.
Reminders
If you need a secure way to meet in the meantime, please use: https://jitsi.meet.coop/