FWIW, it's straightforward to see how this could be accomplished with Sandstorm's model, since user data is very clearly divided into grains (app instances / documents) with clear owners, and these grains can easily be transferred between servers. Today, you can click "download backup" to get a zip, then upload it to some other server, and end up with exactly the same grain reproduced there. We plan to streamline this further, to allow mass transfers and synchronizing grains between servers. It's a major goal for us that it should be trivial to move between hosts, so that you're never stuck with a host that you don't like.
Good to know. Just to re-iterate one point, the ideal is that the server does not have a hold over the data (by say withholding download of backup data), or that the data is held tied to a certain server.
Server and data (encrypted) should just be commodities, and the domain DNS is what should describe ownership. That's why I would like the focus to be on federation than easy import/export.
Realistically, if you want to run code on the server, the server needs some control over the data. There are some distributed systems lately that really only distribute storage while all computation happens client-side. That doesn't work well if you want real-time collaboration (the latency is too high) or multi-user permission constraints. Not to mention, it requires rewriting the world. So, I'm skeptical of those approaches. I'd rather make it easy to get your data out when you need/want to.
But it's certainly an opinion that is open to debate.
> To ensure low latency, I'm OK with the protocol allowing a server to cache and render recent data, while a background service syncs to the de-centralized store.
This would be a help for the UI from the protocol, but the ideal IMO should be something that normal SaaS companies/products hold you on for -- they have your data hostage.
I understand that for sandstorm above many not be possible as you'd have to re-write a lot, and this feature does not hold sufficient priority for the relevant code re-write to happen.
That's too bad though -- I was expecting this need of mine to be provided by the project. Still I wish you the best of luck as you're head and shoulders the better option above any of the current SaaS apps for a personal clouds.
Understood. The exciting thing about sandstorm to me was (and is) that it promised a lay-person easy server and cloud data management. To make this viable, the user would only need to pay for compute and storage.
However, compute and storage providers must not be able to hold user data hostage. Yes, the ideal is hard to acheive, but, for example, nntp had shown decades ago that data storage federation is possible. It's about coming up with the right protocol.
To ensure low latencies, I'm OK with the protocol allowing a server to cache and render recent data, while a
background service syncs to the de-centralized store.
NNTP didn't maintain any kind of privacy of data, though. Once you layer that in, the complexities are tenfold.
It's the issue that arose when I was working on Appleseed. People really wanted a fully decentralized social network, but they wanted to maintain the kind of lock & key access that silos provide.
Federation allows you to have a modicum of control over data, but you still have the "message passing" problem, where once you distribute the information across the network, you don't control what people do with it anymore. This exists, to a lesser extent, in a silo as it concerns the users, but with federation, it concerns the hosts as well.
So people want it fully decentralized, but also encrypted, but also available easily, but also secure, but also user friendly with a decent UX (discoverability, speed, etc).
That's a whole lot you have to balance. It is about coming up with the right protocol, but that's a much bigger problem than that sentence broadcasts.
I think most projects start trying to solve all these problems, but then settle on one aspect of it.
I don't think any project balances all of them in any meaningful way, but if any were ever able to pull it off, they'd change the internet as we know it, forever.
I wrote a paper back in 2010 about a proposal for having browsers handle private keys, and a decentralized social network could then store encrypted data, which would be decrypted at the browser level. But in order to maintain that kind of trust, it would have to be baked into the browser directly. A plugin wouldn't be trustworthy enough and wouldn't have the necessary install density, and client-side JS is out of the question.
And then there's the UX question to how users handle private keys. Do you start to expect everyday users who just want to browse their social network to juggle private keys, move them across devices, etc. And what happens when a user wants to search the network? Even if it's possible, I can't think of a way that isn't extraordinarily slow.
But that's one way you could handle encrypted data so that the servers involved don't have any idea what they're storing. Then you could split it into chunks and spread it all over the place.
There's just so many moving parts, and all the projects working on it are underfunded and undermanned to solve just the single aspect they're focused on, which isn't even the whole package.
I'm not saying it's impossible. It's just extraordinarily difficult.
Of the projects I'm aware of (I'm sure there are many I'm not) - the one that seems to hold the most promise of realizing a working set/trade-off of these features is probably ipfs[i,f]. At it's core it's "just" about a p2p content-addressable web - but with a naming scheme and client-side public key based crypto -- it should be possible to build both an "open" and a "closed" web on top of it.
It's not there yet, though. But as I said, of current projects, that have people doing real work, and is in some sort of alpha state -- I think it holds the greatest promise.
Just getting the features there is going to take a while, not to mention stitching together a user experience and user interface that actually makes it usable in an easy and secure fashion for regular users...
[f] The faq repository: https://github.com/ipfs/faq/issues (note a lot of discussion and open questions - ie: not all these have a clear "best answer")
