Manyverse and Scuttlebutt: A human-centric technology stack for social applications
Are you aware the web is dying in the stranglehold of big tech, from which you'd like to move away, but feel you don't have an alternative? If you are ready for a completely different paradigm, Manyverse and Scuttlebutt may be your thing.
The web began dying in 2014. This provocative proposition was the key premise of an article that went viral in 2017, and not without good reasons. Its author, Andre 'Staltz' Medeiros, seems to have hit a nerve, noting himself that he managed to raise more awareness than what he was hoping for.
Medeiros is a successful web developer, and he understands the technological underpinnings, as well as the social and other consequences of ongoing developments. Things such as the FCC's decision to kill Net Neutrality, or W3C's turn in favor of DRM, and the ever-increasing concentration of power in the hands of big tech.
That's just business as usual, some might argue. It would be hard to imagine how power structures in the real world would not eventually come to be reflected in the online world. Not everybody has to like this, or settle for this though. Medeiros decided to put his money where his mouth is, and work on a solution. The result, based on the idea of reflecting social structures in the real world online, has just been unveiled.
It can only happen, he went on to add, if the web takes a courageous step toward its next level. For Medeiros, this means a return to the source of "peer-to-peer technologists [who] brought to the world several revolutionary technologies: USENET, Napster, BitTorrent, Kazaa, Skype, Bitcoin, Ethereum, and actually even the web itself."
Medeiros quit his job in order to "join a group of peer-to-peer programmers and help build technology that can rescue our digital freedoms." Their plan, as he puts it, is to build the mobile mesh web that works with or without internet access, to reach four billion people currently offline. And their weapon of choice goes by the unlikely name of Scuttlebutt.
Scuttlebutt is a peer-to-peer data gossiping protocol. In other words, a way to spread and synchronize data in a distributed network without central control.
Scuttlebutt was originally developed by Dominic Tarr, a computer programmer with an off-the-grid lifestyle. In order to deal with the challenges this posed to his need for connecting, he came up with a protocol based on the premises of sporadic connectivity and a structure that reflects that of social connections in the real world.
The term "Scuttlebutt" comes from the original water-cooler gossip, and has nautical roots. Unlike most data protocols, Scuttlebutt does not work on the assumption of constant connectivity and centralized services: Data is stored locally, and synchronized between contacts (or friends, to use the social network terminology). Data is also encrypted, hence Scuttlebutt is also referred to as SSB, or Secure Scuttlebutt.
Manyverse, and many hard questions
There are many tricky parts to this idea, starting with how this synchronization works. But before we get to those, a few words on functionality, and Manyverse.
Until recently, the tools and applications built on top of Scuttlebutt were rather raw. This is where Medeiros comes in. He built a mobile application called Manyverse as a front-end for Scuttlebutt, with the goal of popularizing it. First off, a mobile application did not exist until now, so this was an obvious area for improvement, but there is more to this.
Medeiros said he wants to make Manyverse be the branding "gift wrap" to Scuttlebutt: "Even though I and many others have learned to like the name Scuttlebutt, a lot of people are put off simply by the name, and I think that's an unfortunate detail to hinder adoption, so Manyverse aims to be the frontdoor to the Scuttleverse."
So, here come the tricky parts. It sounds good to be able to store everything on your local device, but sooner or later that local device storage will be insufficient. Especially considering the primary use case at the moment seems to be off the grid and less technologically developed settings, one can imagine low-end devices quickly filling up.
A mechanism to offload older content to some secondary storage (think hot/warm/cold storage) would be a requirement for this to scale going forward. Data replication could involve user storage (disk), or even commercial cloud storage. Given Scuttlebutt applies encryption, privacy should not be a problem -- although of course access control may be.
And, of course, there is always the scenario of devices crashing, getting lost, etc. Scuttlebutt deals with this by prompting users to create a new identity, reconnect with their contacts and start from scratch.
Answering hard questions
Medeiros addressed these topics in turn, starting with the issue of limited storage: "Storage being limited on end devices is a disadvantage for SSB. But recently, we began seeing this as an advantage as well. Storage as a precious resource means each user will prefer to store only people worth storing. We see it as the digital form of hospitality."
As far as offloading older content is concerned, Medeiros mentioned he did an analysis of what takes space in his local SSB, and it turned out that the majority was blobs (images, video, etc) -- unsurprisingly, we might add. Medeiros noted this can easily be purged, also using an automated strategy, although at the moment users have to delete files themselves. Medeiros mentioned some alternatives, such as auto-deleting large blobs first.
This way, the smaller a blob is, the longer it'll stay stored, and Medeiros noted this has the social effect of encouraging people to respect others' storage limits and prefer small blobs. Even though blobs can be deleted, there is a core log file required for the application to function. Medeiros noted its size is not unreasonable, but after some point it will make sense to also prune this file.
In the SSB community, the automatic creation of new accounts has been discussed, for instance, one account for each year. Creating accounts is cheap, Medeiros said, because they are just a crypto keypair, and on the UI level you can aggregate all sub-accounts under one "virtual" account.
"This way, you can choose whether to keep old sub-account from previous years, or just let them eventually get forgotten. Not all of this is implemented yet, but it's conceptually simple, the protocol supports it," he said.
When it comes to losing accounts, Medeiros said the key identity issue is addressed using a SSB subproject called Dark Crystal. "Overall, the downside would be the re-download experience, but we are quite positive about this use case being much more hassle-free than recovering a corporate account with passwords, 2FA, SMS verification, secret questions, etc," he said.
Synchronization and Pubs
So, synchronization. How does it work? As this is based on synchronizing logs, would it not be very inefficient having to go through the entire log in a serial way when syncing with a friend to check whether each update is relevant for you?
In the SSB world this is called the "initial sync," and Medeiros acknowledged it's currently the worst part of the user experience that affects all the apps. He did add, however, they are getting better at it:
"One of the ongoing projects is doing prioritized replications so that you first get data from close friends, and can easily browse through their content while other secondary syncs are happening in the background.
Some time ago Dominic [Tarr] worked on improving the sync performance throughout the distributed network by applying an algorithm called Epidemic Broadcast Trees. In practice, only the initial sync is a pain, the week-by-week or day-by-day update experience is quite light."
Being in the same LAN with a friend is one way to synchronize, but there are more. The overall strategy is to have as many modes of connectivity as possible, said Medeiros: LAN (implemented), DHT-based direct P2P (implemented but will get better), Bluetooth (in development), "room servers" (not designed yet), and Pubs.
Public Peers, aka Pubs, are the closest thing to a server in SSB at the moment. To sync across the internet, Pub nodes run at public IPs and follow users. They are essentially mail-bots that improve uptime and availability. The Scuttlebut community runs some Pubs, and anybody can create and introduce their own.
The role of Pubs in the network is paramount. In real life, there is no such thing as a pure p2p network -- some nodes have more resources than others. In theory, Scuttlebutt can work in a purely p2p way, but only for very basic and limited scenarios, both storage-wise and connectivity-wise.
The question is not why pubs are needed, but what are the incentives and implications for anyone running pubs. Some people may volunteer to run pubs. But when the load becomes so heavy, they cannot use the machines that run pubs for anything else, and they have to foot huge bills. What then? How can the network be sustainable if there is no reward/incentive mechanism for people to chip in?
The incentive game
Medeiros said most people in Scuttlebutt don't worry about the Pub incentive game. But he does: "I have been thinking a lot about how to solve this. The more modes of connectivity, the better, because if one mode doesn't work or is not available, people still have ways of sharing updates. In the worst case, they'll have to periodically meet each other in a common LAN. I think this realistic to expect for folks in remote communities or underprivileged countries."
Medeiros elaborated on different ways of connectivity, yet acknowledged that pubs are the only solution that does not require the online presence of a friend. He noted, however, that pubs are fungible. Therefore, he does not think the incentive game is negative.
"I could easily setup a pub temporarily, just to mirror my content for a while and make it easier for one friend or group of friends to get in contact with me. In fact, I have done exactly that, and I'm never afraid of shutting down a pub, because I know nothing is lost when I do that," he said.
But what happens if, for example, illegal content posted on the network is shared on the internet through a pub? Medeiros said they encourage connecting only with trusted peers, including trusted pubs: "So if illegal content happens to get replicated to your computer, then it's first and foremost a breach of trust with implications to the community."
The technical answer to this, noted Medeiros, is that pubs can be shut down easily and the data wiped, so there shouldn't be permanent "service provider" concerns regarding that.
"Overall when it comes to illegal content, SSB is similar to other local databases such as Picture Gallery mobile apps or USB sticks, since their sharing is also often done through Bluetooth or USB cable. SSB is above all a local database, not a network," he said.
"I take as inspiration WikiMedia and Mozilla Foundation, which work somewhat in the intersection of 'product' development and advocacy/activism. Product startups have a run-way, and if they can't fly before that, their closed source product cannot be easily continued by the community, because infrastructure and DevOps is a challenge.
Our situation with open source and no infrastructure differs significantly from that. With the examples of other free and open source software out there, I believe we can build something significant in the span of five years or more."
Instead of a product like Facebook built by Zuckerberg, Medeiros said, people should think of him as Linus Torvalds, and Manyverse as Linux, with the goal being to support a grassroots movement that can create multiple "distributions."
As a protocol, SSB has some interesting properties, although it has not been systematically analyzed, as the focus is on implementation. As an approach, Manyverse is a departure from the average social networking or messaging app. It looks like it still has some way to go, but if you are ready for a change of scenery, this could be your thing.
Photos: From facial recognition to connected toys, a trip inside the invisible big data revolution