Dec 6, 2019 7:00 AM

You Can Have Collaborative Software That’s Wary of the Cloud

Research lab Ink & Switch wants to harness the benefits of productivity and communication without forcing users to give up control of their data.

Cursor puts together folder made of squares

A few weeks ago, as a damp winter chill settled on San Francisco, Peter van Hardenberg decided it was time for some eggnog. He knew of an excellent homespun recipe from a former colleague at Heroku, a company that helps startups build cloud-based apps. And if he recalled correctly, it was stored on one of Heroku’s many cloud servers. But when Van Hardenberg typed in the relevant URL, he found the cupboard was bare—lost to some long-ago server maintenance. “It had just fallen off the internet,” he says.

Such are the daily glitches of life in the cloud. Cloud computing has made essential tools, like Google Docs and Slack, possible. But it comes with compromises. The smart toaster can’t toast without Wi-Fi. The music skips when you enter the subway tunnel, and then the entire library disappears when the startup goes out of business. “We’ve forgotten what it’s like to have software that works,” Van Hardenberg says. But for software companies, the model of hoarding software and data on remote servers works just fine. It’s lucrative, in fact. Hand over enough of your data, and you’ll eventually need a membership to access it. Or else it’s probably being used for advertising. None of your data is truly yours.

Van Hardenberg and his colleagues at Ink & Switch, a private research lab that includes other Heroku alums, want to provide an alternative to that model. They call the effort “local-first” software. (Van Hardenberg considers it a form of penance for having built a company based on offering cloud services.) Local-first reflects a yearning, in part, for the days when software came in a cardboard box. Back then, you installed it on your computer, where it remained safely ensconced, along with your files. But the point isn’t to ditch the cloud entirely so much as deemphasize it, says Martin Kleppmann, a Cambridge University researcher who works with Ink & Switch. It’s “local-first,” not “local-only.” The idea is to marry local storage of software and data with certain things the cloud does well, like collaboration.

Files are stored across the devices of invited collaborators, rather than on a corporate server or in the cloud. It’s “decentralized,” in other words. That word is often synonymous with blockchain. And there are overlaps, spiritually, in the desire to avoid centralized authority. But blockchains are based on a lack of trust with other users. They use a computationally expensive process called consensus to ensure everyone agrees on a common state of affairs, without anyone taking advantage. Local-first software is based on trust. The point is collaboration among friendly parties. “The consensus I need is what’s on my computer,” Van Hardenberg says.

Local-first software uses a leaner technology called conflict-free replicated data types, or CRDTs, first fleshed out by researchers in France and Portugal in 2011. The concept is similar to Git, a tool programmers use to manage software development on platforms like Github. But instead of manually merging changes, as Git requires, CRDTs do it automatically. When Kleppmann came upon the idea a few years ago, the technology was being used primarily in a few backend databases and in academic applications. But he was intrigued by the concept. So he set out to make it more useful to developers, designing a JavaScript library called Automerge that made CRDTs more flexible and efficient. The hope is to get the software to a point where developers want to use it to build a local version of Slack or Trello—Kleppmann personally wants a local-first Evernote.

The Ink & Switch team has used Automerge for a handful of prototypes. Van Hardenberg shows me an app called Pushpin. It’s a little bit like Pinterest, a board where users can share images and notes, along with websites pulled from Chrome using a plugin and a simple chat function. He had been pleasantly surprised by how well the CRDTs worked. “It was like the Wright brothers,” he says. “We’re really flying. This magical feeling of freedom and independence.”

Sure, the application was a fairly low bar. There’s little data on a message board app, and the potential operations aren’t too complex. Right now, the technology struggles under the load of more than a couple of megabytes—plenty for chats and editing documents. But higher-order software for collaborating on photos or design files can reach the scale of gigabytes—1,000 times larger, or more. But Kleppmann hopes an ongoing rewrite of the Automerge innards could get it close, enabling those larger-scale apps.

The technology brings a few other unique challenges. “It turns out that CRDTs are really easy to implement badly,” Kleppmann says. Traditional databases are designed to be set in stone; you can trust that the document in front of you is the latest version. That’s the beauty of having one centralized copy. Not so with CRDTs, which involves trade-offs to permit editing in real time. As multiple users make changes to a file, they need to have faith that the changes and conflicts will eventually resolve. There’s always a slight lag as the document catches up to the latest edits. It’s barely perceptible for something like Pushpin, but it would get more noticeable in more complex apps.

Usually that resolves itself just fine. But there’s still work to be done on edge cases—deciding which actions to prioritize when two users do things at the same time. Together with the difficulties of implementing CRDTs, that could lead to problems, says Emin Gun Sirer, a professor of computer science at Cornell. “I would very much worry about these kinds of solutions in settings where immediacy and consistency are very important,” he says. In other words, you might not want, say, to store vital medical records as CRDTs, lest the changes fail to take hold quickly when they’re handed off from one ER doctor to another. Sirer is a fan of the concept, so long as developers are careful and use it in the right contexts. By avoiding data collection, CRDTs are good for privacy, he notes, especially when they’re combined with secure peer-to-peer communications.

Ink & Switch is still deciding whether to push apps like Pushpin out of beta. The lab’s primary aim is research, not building products. But the team hopes its work encourages other developers to run with the idea. Local-first is clearly starting from an underdog position, technologically. “The big question is whether these apps provide sufficiently compelling features to get people to switch from cloud apps,” says Matei Zaharia, a professor of computer science at Stanford and chief technologist at Databricks. For now, at least, the technology can’t replicate the large-scale collaborations you can enjoy on, say, Google Docs.

Another challenge is getting the economics right. “Entrepreneurs want to know, what gun do I have to hold to my users’ heads to make them pay?” Van Hardenberg says. Cloud software offers an easy answer by locking in user data. For local-first, a strategy might involve paying for software up front, like we used to do at Best Buy, or subscriptions—though that might be close to many current cloud business models. But who knows, with users increasingly fed up with cloud-based glitches and frustrated with big tech, maybe they’ll be ready to pay for something a bit more artisanal—an alternative that gives us more control.