P.S - my current favourite solution would be a 2 phase commit when the mobile client does 2nd API call to commit the booking. Server rolls back the booking if no commit received within N seconds/minutes.

Not sure if this is a recommended approach, but I did something similar recently.

I create a guid on the client, and send this with the request. If I get a connection error or no response, I re-query with my client side id to see if the transaction succeeded.

I'm sure there are downsides to this, however for now it's simple and seems to be working well.

Cheers, I think this might be what we settle on (or similar).

One slight tweak would be to do a "retry" using the guid, but the server knows to send result of last success if the guid matches a completed transaction.

You can't protect against this. Any confirmation messages are subject to the same problem. Look up the FLP paper for proof.

This is really a user interface problem; your user needs to think of a transaction during a network outage as being in an "unknown" state rather than a "succeeded" or "failed" state.

Big thanks for that. I'll give it a read.

We've hit problems with ack in the past. So I thought we could add more acks to increase confidence. But it's never 100% certainty. I'm sure this paper will help understand the issues.

Could you theoretically wait to commit database transactions until after the client has sent acknowledgement of the response?

Kind of, yup.

That said, one detail I left out is that this transaction is distributed across 4 separate systems, so it's not quite as simple as a database lock. And I wouldn't do a long db lock unless it was some kind of pessimistic offline lock :)

But we're mostly worrying about the mobile app acknowledging the response so the same principle stands - we need the mobile app to ACK :)