I don't really understand what this is offering beyond Cloudflare's recent release of running SQLite in durable objects: https://blog.cloudflare.com/sqlite-in-durable-objects/. Is it about providing an external interface to Cloudflare's SQLite databases?
If that's the case there's libsql (https://github.com/tursodatabase/libsql) which already provides HTTP clients, embedded replicas (essentially good old SQLite but with replication support), self-hosted SQLite server and is maintained by a company using it for their own product.
iirc they are both powered by the same engine to stream and replicate the WAL. I believe R2 is now implemented as a Durable Object backed by SQLite now.
Still think there is a lot we can add to StarbaseDB to make the developer experience on SQLite databases better, and personally I think it starts with a better external interface. Provide a frictionless way to get started. Then figuring out how to improve developer experiences on how they interface with databases.
Is it auto-accessible REST endpoints? Easy to implement websocket support for the database? Data replication for scaling reads? Offline data syncing? A lot of potential wins for a layer like this to build on.
Not a bad idea. There'll always be someone who says "but I can already do that with X" and then someone who replies to that with "but I can already do that with Y" and in the end when you're done trying to stick to the purist laws you've ran out of time and motivation and your project goes into the Pile of Dead Projects.
Or, we have easy to use, enjoyable interfaces and let ourselves use them, and actually finish things.
From what I could understand of their architecture [1], you may lose data when the machine running DO fails.
DO implements a very comprehensive strategy to mitigate data loss in DO backed by SQLite. But Starbase seems to add a layer of query caching on the application level, which seems to be naively implemented (I might be misunderstanding, please correct if I'm wrong).
We don't cache query responses in the Starbase application layer. When we originally set out to build this, we were under the assumption we had to handle the ACID transactional support ourselves. Thankfully someone from Cloudflare shed some light for us as you can see in our Github issues and open PR's :)
Just fyi, the person that replied wasn't just "someone" at Cloudflare. It was Kenton Varda (kentonv here). He's the creator of Cloudflare workers, he's an incredible engineer.
Had no idea this was put on HN today. Author of StarbaseDB here, will get a bug fix for the blog redirection pushed out soon! Thanks for bringing this up.
Really glad Kenton chimed in there, as it wasn't clear in the docs and I was about to implement a similar pseudo transaction pattern as this.
Pretty excited about this tech, it really flips the script on the concept of a DB per user / sharding approach which used to be costly at the schema/physical db level.
This is very appealing. If the authors are reading:
What is the architecture overview? Does each worker have a copy or do all of them work on the same DB files? How does it handle concurrent workers and db locking? Is SQLite native locking mechanism sufficient in this application?
Multiple workers would connect to the same durable object and write to it, basically, the Durable Object takes care of all the writes in a single thread, so there's no issues with concurrent modifications.
We probably need to build out some sort of data replication to scale reads though Durable Objects are limited to 1000 RPS.
I have this old school expectation that I should be able to understand what your website/product/project is about but reading the first few sentences in the "above the fold" section of your homepage. I don't want to play a game or click below the fold "read more" links to try and find out what this web content has to do with the domain name "starbasedb".
Perhaps they actually know their intended (seemingly game developer?) audience that I give them credit for? But I do have plans (admittedly only very vague plans right now) to use one of the available "SQLite from javascript in a webpage querying the SQLite file in an object store like S3 without a web backend" techniques or projects. Maybe they'd work for me with my non game oriented use case? I don't know, it seems like they don't want me to find out...
Appreciate the candidness to your response here. Honestly we put this brand and website together in two days and we wanted to have some fun with it. That said there is a lot more we can do to make the website more informative, and we'll be sure to improve it as we go. Expect more blog posts, tutorials and implementation guides this week.
ELI5: Does "Open source, scale-to-zero, HTTP SQLite database built on top of Cloudflare Durable Objects" mean that you should pay and/or use Cloudflare for this? Or CDOs are generally available outside Cloudflare?
No, there's no traditional database-level locking here. The Durable Object itself is single-threaded, so you don't have to worry about multiple things trying to change the same data at the same time.
As far as scaling yeah there are still some challenges there. I mentioned in another response there's a limit of 1K RPS, and the single-threadedness could become a bottle-neck. I think there are probably clever workarounds here that we still want to explore, for instance each Durable Object could handle a seperate piece of state and you would effectively have a level of state-based sharding.
May I just say; hell yeah on the brand identity! - finally someone has the guts to actually have an identity instead og pastel based standard b2b shit.
Last time I saw that was zendesk with their Buddha's and weird tonal audio devices.
Edited for corrected semi-colon because I'm haunted by a grammar-nazi.
`Isolation. Due to the nature of our queue and query executions, everything is synchronous by nature so no two queries can be ran concurrently making them run in an isolated nature by default. Check.`
I'm not sure if I misunderstood something, but using a global queue/lock to enforce isolation properties seems to be just a little disingenuous for a dbms. It works, but it's odd to talk about isolation in a system with zero concurrency. It's like saying a bicycle gets infinite miles per gallon in fuel efficiency.
This has the same name as an Elon Musk project. I can see the point of reusing names (this is a clever spin after all), but I’d expect the association to evoke quite some negative reactions.
