Pedantically, RESTful APIs do not maintain session state on the server. This is important for scalability.
(Whether "pure" REST is appropriate for a given API, or will be adopted if it is, is another question.)
If too much is done on the client-side, it loses some of the benefits of webapps, such as easier to pirate the working code; so the competitive advantage needs to be in the data (or in the server side processing) rather than the client app itself.
EDIT There's a general problem with RESTful APIs: they are fixed and can't be adapted to every use. You might end up doing a lot of transformation on the client side to assemble the info you want, and it might take several calls, causing latency. What's needed is something like relations and SQL, so you can extract the data in whatever structure you need, taking into account latency (assembled on the server, the message short though including oft-needed related data, and minimizing slow network trips). An obvious solution is for the server to just accept SQL directly - i.e. for that to be your RESTful API.
This doesn't sound right - it's certainly not a conventional RESTful API - but what exactly is the problem with it?
I think we don't have it yet, because there isn't a need for it in these early days - because, so far, it's common for one server to be used by just one client application, so that they are specifically customized for that one use-case. But what will happen when several different client applications want to use the same server? Secondly, these APIs are new, and there has been little time for them to change much, so the problem of compatibility with old clients hasn't arisen much.
SQL solves both these problems, of different application and of changes, but these problem don't exist so far. And I guess it's not certain that they will exist - if clients are always silently upgraded; if for a second app you just create a second RESTful API for it (so the integration occurs further back, at the database, instead of at the API).
> There's a general problem with RESTful APIs: they are fixed and can't be adapted to every use. You might end up doing a lot of transformation on the client side to assemble the info you want, and it might take several calls, causing latency
This isn't REST as such, in fact you have the same issue with any service API.
Solutions include creating composite resources, see Subbu's (excellent) REST/HTTP book which discusses it.
Its also worth noting that more granular resources can have advantages particularly when mutability is considered. So if you combine mutable data with immutable data, and in addition the mutable data changes to very different schedules, then taking advantage of HTTP caching becomes more difficult. So if you create more granular resources but some of them are very cacheable then you can actually in some cases get less latency.
> SQL solves both these problems, of different application and of changes, but these problem don't exist so far.
I haven't had time to play with it but ql.io is designed to address these issues and might be worth a look:
Nailed it. Composite resources and good use of caching with fine-grained resources can go a long way.
HTTP is a pretty sophisticated protocol, when all is said and done. That means there is a learning curve in using it effectively. However, a good BaaS takes care of that for you, including providing default client libraries that know how to take advantage of the API properly.
(Full-disclosure: my company, spire.io, has recently launched a REST API BaaS that attempts to address exactly these concerns.)
I don't think I was clear enough on that point - I didn't mean that the session state itself would be accessed as a REST resource.
I imagine the session state would consist of client side Javascript (since this service would be ideal for the type of single page apps served well by Backbone.js and the like). The javascript would probably submit time-expiring authentication tokens with each request. That way, session could be persisted client side, while maintaining permissioned access to the CRUD actions on the service side.
EDIT (to respond to 6ren's edit):
I agree that, in a production environment, the flexibility of the rapid prototyping could, in many cases, turn into rigidity when faced with just a RESTful API.
But not necessarily in all cases, especially when implementing a single page app style site, where you are more likely to persist data client side in the background for when you need it.
Ultimately, I think the tradeoff might be worth in some circumstances. And given the apparent success of the many mobile BaaS solutions, it seems to be often enough to capitalize on.
On 6ren's rigidity argument, relations and SQL are still not enough to guarantee you'll always be able to get the data you need; notably, when the data model changes. Backward compatibility is important and we might have they only solution that guarantees it.
> relations and SQL are still not enough to guarantee you'll always be able to get the data you need; notably, when the data model changes
What data model changes are you thinking of? Relations and SQL will cope with reorganizations/restructurings of the data, provided the information content is the same. But won't cope if the information is different (e.g. info is deleted or added).
> "Only ChronicDB can run new applications and unmodified old applications against a master database without breaking anything." [from your link]
I like your points about undoable DELETE (do modern RDB's really lack that?); but the above quote is striking because it was the stated motivation for relational databases in Codd's 1970 paper: http://www.seas.upenn.edu/~zives/03f/cis550/codd.pdf
[from abstract] Activities of users
at terminals and most application programs should remain
unaffected when the internal representation of data is changed
and even when some aspects of the external representation
are changed.
[from conclusion]
In Section 1 a relational model of data is proposed as a
basis for protecting users of formatted data systems from
the potentially disruptive changes in data representation
caused by growth in the data bank and changes in traffic.
The paper discusses derivability, and how some relations (such as those needed by old applications) can be derived from named relations using a high level language (now SQL). The idea was that you can change the database schema yet still provide the old relations (this mechanism is now called "views").
I'm not saying ChronicDB has no role, just that this one particular aspect, in isolation, doesn't seem new. But if it facilitates back-compatibility in contexts where RDB doesn't work well, it could be a big win. Back-compatibility, like integration, is always in demand.
EDIT looking at the "schema migration" example (http://chronicdb.com/system/files/images/agility_full_snap_g...), it looks like a join, with renaming, and a type change... is the new thing offered in this example that updates are bidirectional (especially for the type change)? Or is it that both old and new relations have the same name, "customers"?
You are right, this aspect is not new. More like half-baked.
I'm not sure what you mean by bidirectional. If this suggests having a separate copy for the old data and the new data then, in ChronicDB at least, no they are not bidirectional. Which is what one wants ideally: data consistency.
If you mean that updates on the old version are immediately reflected in the new version, then yes.
Basically, there's a single version of the truth, while old and new relations can have different names, for all DML operations.
Yes, the second one. So that from the old app's point of view, nothing has changed: it can create, read, update, delete as before. A single version of the truth etc.
SQL's "CREATE VIEW" does this already (note: not all views can support CRUD, e.g. if you changed an aggregate column, like COUNT, there is no clear meaning for the affect on the underlying tables. If you increased count, what rows should it add? If you decreased count, which ones should it delete? The problem with aggregates is that the mapping from the underlying tables to the view isn't 1:1, you can't run the mapping backwards to reconstruct the original).
1. How is SQL's solution "half-baked"?
2. What does ChronicDB add?
I mean, back-compatibility is valuable, but what is the problem you see with the existing solution?
1. Thank you for the link on views being updateable with some (natural) restrictions (http://news.ycombinator.com/item?id=3406952). Given this feature, SQL offers the primitives needed. Even if this feature is missing from various databases at the moment, it should be available eventually.
But updateable views are still not what you want! See below.
2. What ChronicDB adds in terms of backward compatibility is, effectively, automatic creation and management of these views for every schema change.
Rather than manually write compatibility views and rewrite/review an application to ensure it always uses these views, programmers instead access plain tables, not views. In fact ChronicDB creates no views at all!
You want to issue SQL queries that don't break. You don't want to manage views to do so. It's a subtle, yet major point. No database to date offers such automated versioning.
ChronicDB of course adds other features we wanted to have from our database (reversibility, data versioning, zero-downtime relocation) again in the spirit of saving the programmer/DBA from doing anything manually.
2. Yes, I agree, nice to have the "view" created automatically when you change schema.
Codd's idea was also that applications needn't be aware of the change (a view appears as a plain table)... but (in contrast to your approach) the views were managed by the DBA who was making the schema changes. So I think this feature was present from the very beginning, being the initial motivation for relational databases. [I looked up MySQL because that's the DB I have access to, but it's a general feature:
http://en.wikipedia.org/wiki/View_(database)#Read-only_vs._u... ]
I would expect automatic versioning to be available too, perhaps only in the older and more expensive RDBs. I've come across quite a few research papers on schema evolution in relational databases - some from big RDB vendors. I don't recall any on automated versioning, but I wasn't looking for that. So this is just my guess.
However, if you're not seeing databases with automated versioning, then (even if some highend RDBs do have it) there will be customers who also aren't seeing it (or it's not applicable e.g. it's too expensive). A market for you. And if you can make it truly painless (or even just less painful), that's really something.
I don't see any problem with your other features. The one feature of mapping for back-compatibility is of interest to me, which lead me to study Codd's original paper in great detail over a few weeks. That's why I've been hassling you on this issue.
This problem has already been solved; it's called OData (http://www.odata.org/). Personally, I've been using it with Microsoft's new Web Api to great benefit.
It provides a very rich query API on top of basic http, and as you can see works naturally with REST services. I'm not aware of how other frameworks provide OData support, but some quick googling reveals that most web frameworks have odata libraries available.
It's an incredibly bad idea to allow SQL directly, for obvious reasons (would require executing user provided SQL among other things.)
I believe that having a single REST api that serves multiple front-ends (web, mobile, thick client etc) is the future as it removes the need to maintain an entirely different codebase just for a web interface. Indeed, many major websites have moved to this model.
> It's an incredibly bad idea to allow SQL directly, for obvious reasons (would require executing user provided SQL among other things.)
Could you elaborate on this? I've been thinking about providing SQL access to a data-heavy service, but I keep hearing that you never should.
(Almost) all servers have granular access-control, views can further provide limited views of the data, SQL itself is mostly declarative, which makes it possible to analyze the queries before running them, and logging and setting limits on long-running queries is standard.
Well, the first problem you'll run into is, which SQL implementation do you use? I think pretty much all discussion of the topic stops here since there are so many differences between the actual SQL standard and what the varios modern RDBMS' actually use.
The only real way past this hurdle is to create an intermediate SQL parser that uses your own interpretation of the standard, and at this point you may as well just use OData.
I'm not sure how valuable a discussion about just using the SQL implementation that comes with your RDMBS is, as doing this defeats all the effort we put into making our front ends ignorant of the underlying schema by locking the front-end into a specific SQL implementation for queries.
Seems like at least some SQL-like flexibility could be added by using custom keywords in the Accept header. There's an example of using `depth` parameter for specifying how far server should dig into related resources[1].
Surely other keywords could be added as well (e.g., specifying a set of resource fields, ordering, etc). A RESTful solution, although probably not quite elegant.
If too much is done on the client-side, it loses some of the benefits of webapps, such as easier to pirate the working code; so the competitive advantage needs to be in the data (or in the server side processing) rather than the client app itself.
EDIT There's a general problem with RESTful APIs: they are fixed and can't be adapted to every use. You might end up doing a lot of transformation on the client side to assemble the info you want, and it might take several calls, causing latency. What's needed is something like relations and SQL, so you can extract the data in whatever structure you need, taking into account latency (assembled on the server, the message short though including oft-needed related data, and minimizing slow network trips). An obvious solution is for the server to just accept SQL directly - i.e. for that to be your RESTful API.
This doesn't sound right - it's certainly not a conventional RESTful API - but what exactly is the problem with it?
I think we don't have it yet, because there isn't a need for it in these early days - because, so far, it's common for one server to be used by just one client application, so that they are specifically customized for that one use-case. But what will happen when several different client applications want to use the same server? Secondly, these APIs are new, and there has been little time for them to change much, so the problem of compatibility with old clients hasn't arisen much.
SQL solves both these problems, of different application and of changes, but these problem don't exist so far. And I guess it's not certain that they will exist - if clients are always silently upgraded; if for a second app you just create a second RESTful API for it (so the integration occurs further back, at the database, instead of at the API).