On Decentralisation and The Online Meeting Cooperative

When we started early May, we first wanted to share a BBB server between three cooperatives - femProcomuns, Collective.tools and Webarchitects, but why would we limit this to just our own three coops? We agreed that also other actors from the Commons and Social and Solidarity Economy would be very welcome to join us and we would abide by the Cooperative Principles - see the home page at https://www.org.meet.coop/ We mentioned the idea to people around us and the ball started rolling.

The second weekly meeting (Thursdays at 15h CEST) we had people from Europe but also from other continents, in particular from Hypha.coop and Koumbit both from Canada and Yasu from Japan joining in. The idea of building The Online Meeting Cooperative as a network of nodes became clear. Based on those discussions - in the meetings and over email initially - @chris drafted a Roadmap. In this roadmap we envision different nodes in different continents. Each node may have one or more servers and at least one organisation taking care for the node and enabling people and organisations to become a member and contribute resources (money, time, servers mainly) to make the node and global network sustainable.

Traditional companies like Zoom, Inc have one legal body, owning, controlling and exploiting the services it develops for its customers, in a top-down fashion. We try to do this bottom-up, initiated by commons cooperative actors who take responsibility for the project and supported by organisations and people who want this to exist and benefit from the service. Zoom and likes aim to maximise their value at the stock exchange at whatever means it may take (keeping users personal data hostage and abusing them). Our mission is therefore totally different as we aim to provide an ethical service in a bottom-up way to the community and can do this as long as we the community are all involved in making it happen. Let’s discuss some of the challenges of such more decentralised model: Organisationally, financially, technically, legally.

Organisationally
Being decentralised means to make use of the organisations we already have. We intent to avoid to set up a new legal entity and instead connect the existing entities to each other. We designed different types of membership and contributions: Producer members and Consumer members, mainly for organisations, but we also encourage individuals to make a contribution as individual members. Financial contributions can be made through either one of the cooperatives that have taken responsibility to channel the collective funds, let us call them the “custodian members”. (if anyone has a better name, please shout!). Non-financial contributions can be made by contributing time and resources to the project. Taking responsibility and working on relevant tasks helps the project and therefore should be valued. We run time registration tool. Do request an account to track your contribution. Valuable non-financial contributions can be counted toward a membership contribution - we should discuss how to validate this collectively.

We aim for the community to be as open and self-organised as possible, the use of an open forum like this one is important, but also the choice of a public wiki, git repositories and other tools.

Financially
Being decentralised means you can contribute financially through different organisations, the custodian members. We started with three custodians: femProcomuns, Webarchitects and Collective.tools and it seems that Hypha.coop is willing to join that group. This means they take care for their local node and contribute to the larger whole and be solidarity with other nodes in need. Here it is important to note that membership of The Online Meeting Cooperative is formally done by joining anyone of the custodians and make your contribution through them.

Technically
The Roadmap already includes a list of servers that are in the air. We envision the key service to be accessible through https://meet.coop and have nodes with subdomains (e.g. Australasia → nz.meet.coop, Europe → de.meet.coop, North America → ca.meet.coop, etc). Hoever ideally we can have all users access the service through one and only frontend. There are several loadbalancing solutions, but the first choice seems to be the official loadbalancing solution by the BBB project: Scalite. However the load balancing documentation is based on having several servers in one data centre with a shared file system. The GreenLight user interface can be the frontend to the loadbalancer but we’re not sure how recordings made at one server can be accessed on another. Apparently it isn’t designed for load balancing a global network of servers. Here more research needs to be done and we should see whether this is a solvable issue or whether we can come up with a suitable solution.

If we manage to have one user interface - and thus one database with user accounts and rooms, the load balancer finds the most appropriate server (based on proximity and most capacity available) and opens the room there. This would be a huge advantage to the users and would also allow expanding the network. Imagine if one organisation has a dedicated BBB server and wants to make it available part of the time to the network (when they don’t use it), that would increase overall capacity greatly. At the same time in exchange for networking once’s server, we could lower their management costs and increase capacity at peak moments. There’s a business case there…

In any case, the aim is to have loadbalancing in the beta stage (September) and work on improving it until Production stage in January.

Be it noted that so far we agreed on one key centralised aspect of the design of The Online Meeting Cooperative: We use one name and a (set of) shared domain names, in order to facilitate community building. But there’s possibly also a second: We aim for user accounts to be unique and have users access the services with one account. While BBB isn’t designed for a federated network, this could result in one centralised user account and room database connecting to the different servers in the network according to their load and proximity. There’s also the idea of redesigning BBB and making federation possible at this level. Those are some aspects to be researched in more depth.

Legally
Being decentralised means no single legal entity, that’s generally a challenge to the status quo, but there are existing solutions for that. We can see ourselves as a joint venture or consortium and sign an agreement, especially between the custodians and with the producer members. Do we have examples we can take inspiration from?

General assembly: members should have a vote in the cooperative. We’ll have to agree that anyone joining according to the membership procedure will have a vote. There are different types of memberships, do we need to have weighted votes? An individual vote or an organisational vote may count differently. Also workers and consumers have a different weight in multistakeholder cooperatives such as Webarchitects or femProcomuns.

There’s also policies on privacy and personal data and conditions of use. How do we deal with local differences, i.e. if Webarchitects were to be responsible, UK regulations would apply, if femProcomuns, Spanish ones. We have different examples here, CommonsCloud: Conditions of use, Data protection, Cookies.

The Conditions of Use should include a Licensing Policy. From the start we are committed to share knowledge under free licenses, to the point of enabling people to replicate servers and the organisation as such. Software, documentation and content in general is licensed under free-as-in-freedom licenses. The software repos already have their GPL and other free licenses applied; the wiki should get an appropriate license. Possibly CC BY-SA, int’l.

Multiculturalism, gender and diversity
English isn’t the language for all, DevOps shouldn’t be male dominated and the world is tremendously inequal. We seek inputs to build our community as diverse as possible with the limitations that each of us has. How can we effectively become a multilingual community? How can we cater for income differences in the North and between the North and the Global South? How can we foster gender equality?

Of course there’s much more to say about decentralisation. Looking forward to your comments!

2 Likes

Thanks for posting this @wouter, it’s very helpful, I wonder if we should have a version of it as a page on the public facing meet.coop website?

I think that for the planned beta stage of the project, September 2020 to the end of December 2020, it is not realistic to be able to have designed and implemented a system for syncing accounts and recordings between BigBlueButtons instances on different continents in a secure manner, I think that this is something that we should see if we can implement for the production stage in 2021. The production front facing system, if we can develop it, would not only need to ensure that user databases are synced between servers and recordings are synced between servers (or made available across servers) but would need to assign requests to open rooms based on availability of servers, capacity of servers and also geolocation of clients to find the best server to open their room on, the development of the code to do this isn’t going to be a trivial task…

I would suggest that for the beta stage we should aim to have several autonomous Online Meeting Co-operative BigBlueButton instances in different countries, for example es.meet.coop, ca.meet.coop, uk.meet.coop, nz.meet.coop, se.meet.coop etc and these instances won’t share user accounts or meeting recordings between them.

I have created a list of our current servers on the wiki.

1 Like

Of course this won’t be trivial, I understand. Some comments though:

If we could have one frontend at http://meet.coop and behind it the loadbalancer, there’d be no synching of accounts and rooms needed. True, it would be a centralised aspect of the network which we should be careful with. But Scalite seems to be designed for that function: upon request by users at the frontend, a room is opened on the server with less load or most available capacity.

What you and Machiel also brought up is the possible problem of the recordings, in that they will be stored at the server where a room session is running and how will Scalite be able to connect to recordings on different servers?

But one frontend - GreenLight so far - which has the database of user accounts and rooms makes much sense, so users at different nodes can invite, share and manage rooms together. The loadbalancing should take care of the BBB server.

Maybe the recordings database can be centralised or replicated in such way that the GreenLight frontend always connects to the recordings wherever they’re stored?

Exactly, there would have to be a central file server and a central PostgreSQL database, these things won’t work well unless we have some really clever solution, plus how would people pick which continent they want their conference to be hosted on?

I really think that for the beta stage separate BBB server instances (which might well be multiple severs themselves with a scalelite front end), for example es.meet.coop, ca.meet.coop , uk.meet.coop , nz.meet.coop , se.meet.coop, perhaps with a central SSO so people can open rooms on a server of their choosing, is the best we can realistically aim for.

A single interface for users would be something to aim for the production stage, January 2021.

Thanks for laying that out @wouter :firecracker:

Tough, yep. Each Greenlight front-end can only see the rooms/recordings that it makes, which is tied into the index database of scalalite and the shared storage volume of the underyling BBB instances. I guess there would need to be some source code changes to allow for swapping out some of this stateful logic to allow for distributing rooms/recordings. It seem like the bbb-setup mailing list is the place to dig further…

I guess we’re leading towards a central LDAP server auth which Greenlight supports. Another option is that I see there is also functional but WIP (:see_no_evil:) work for Greenlight supporting OpenID Connect. This could allow for something like a central Keycloak SSO which can be themed out of the box and administered through a web interface. I’ve also seen that Keycloak can plug-in existing LDAP providers for user federation.

EDIT: there was also talk of commonscloud / cc-onboarding · GitLab :+1:

1 Like

The current design for the Open 2020 conference session on digital tools infrastructure is here:
https://notepad.diglife.coop/s/rk3LqJrhU#
You’ll see there are things in this agenda that relate to meet.coop governance and development, and I hope colleagues here will contribute in the session as indicated here.

This is a different tack from the operational emphasis of some of what @wouter and others wrote above. But I think it joins up, and is something of strategic concern. (If you feel it doesn’t fit here, do suggest relocating this to a more suitable thread?)

From a governance perspective, while I agree that a decentralised approach doesn’t demand new legal entities, I do think it requires clear agreements, and a commitment to transparency, especially around contributions and costs.

2 Likes

In that scenario we would have just 1 Greenlight + Scalite frontend and multiple BBB servers spread out over the globe. That scenario doesn’t need user account synching, as there’d be just one GL user db. The challenge Chris and others have suggested might come with the recordings, that will be at different BBB servers. We don’t know yet how that works through Scalite.
This scenario also has the challenge mentioned to adjust Scalite in such way that it chooses the server to open a room not only based on available server capacity but also based on proximity to a user defined preferred location. Imagine That a Canadian Coop selects their room to be opened preferably on or near Montreal, the loadbalancer then should have some to be developed algorithm to make he most appropriate server choice.

I guess we’re leading towards a central LDAP server auth which Greenlight supports. Another option is that I see there is also functional but WIP (:see_no_evil:) work for Greenlight supporting OpenID Connect. This could allow for something like a central Keycloak SSO which can be themed out of the box and administered through a web interface. I’ve also seen that Keycloak can plug-in existing LDAP providers for user federation.

Possibly we can have a combination of login methods, which GreenLight seems to offer. We’d like to give our current CommonsCloud userbase the option to login with our LDAP authentication service, and I guess others may have their own services.

EDIT: there was also talk of commonscloud / cc-onboarding · GitLab
The CommonsCloud onboarding software has some particular feats that might appeal to us here:

  1. its a web based user account management in LDAP, so all current systems we’re running can authenticate against it
  2. it has users, collectives (and collective managers) and services. The user journey starts with the user requesting an account, specifying which primary collective it belongs to. the manager of that collective then validate the registration request and then can give the user access to the services managed by that collective. This helps to keep user management at the level of the collectives. Admins don’t need to intervene.
  3. the directory structure has been designed to allow for federation between different installs. It hasn’t been tested though. But the idea is that user1@femprocomuns.coop of the CommonsCloud node in Barcelona could connect to services of another node (if publicly available or if invited) with her same account.
    If you’d be interested, we can do a little session on how the CC user management works. Be aware that the UI shall be improved, especially to handle large numbers of users. And also that we can’t currently dedicate much time to needed improvements.

exactly, @Graham, we need to be sure to keep transparency - so far the producer members have gotten access to the cloud.meet.coop NextCloud instance in order to share ongoing financial income and expenses, and we have an overall commitment to work in the open through the wiki and forum and register time in time.meet.coop. Does that suffice so far, or what would you suggest?

On the clear legal agreements, how would we best do that? Are there inspiring examples of initiatives with similar values having agreed some kind of consortium agreements where people and organisations can join and leave as members? I have some examples from European Commission funded R&D consortia (of which we have been part), but they’re really catered for commitment towards the EC.

I think that running multiple Greenlight containers in front of one Scalite server with multiple BigBluebutton backends (with all the servers in the same datacentre) might be the way to go, however we should check that this is possible with the developers!

Chris, I’m just wondering why would you want multiple GL containers? Then we’d have problems to sync user accounts and so. But you will have good reasoning for this I’m sure.

Because they could be running on different domain names and have different branding and use different authentication methods — this could be an expensive option for big organisations who want to have full control of user accounts and rooms?

1 Like

4 posts were split to a new topic: Systems architecture

Any update on the problem of putting globally distributed BBB servers behind one loadbalancer and front-end (with an option for a separate front-end for high-paying customers that want it)? I’m not sure exactly how it would work organizationally yet, but I’m still keen to put together an Aotearoa node using CatalystCloud.nz (perhaps Catalyst themselves might be interested in becoming an operational member of meet.coop).

1 Like