We have built something like this on top of a distributed in-memory database. The changelog of the distributed in-memory database is the 'source of truth' that downstream clients would like to consume. The research problem is that the changelog is batches of transactions that complete within a given epoch (time). Transactions may execute in parallel on different data nodes within an epoch and there is no global ordering over them (no global time, only logical time). Downstream clients, however, may require an ordering over the transactions, so we worked on a solution to this problem when the database stores metadata for a filesystem. Our solution ensures we don't have to serialize the transactions to ensure a global ordering, and provides strong eventual consistency to clients.
Can you elaborate on the solution? Do you use an aggregator node, or have you massaged real-time reqs to allow for an aggregation period on the client side? I'm super curious to hear how you proceeded!
No, we're using a table as a 'queue' in the db. Client are decoupled and if our middleware is offline, it can restart and catch up by draining the table. Transactions over the table ensure the consistency and integrity of the 'queue'.
We provide at-least-once semantics to downstream apps, by exploiting transactions in the DB. Actually, we have one sink in the DB itself and we get exactly once semantics for that. The work in under submission with anonymous reviewing, so can't elaborate massively on everything else. Performance numbers are good, though, a sigle server can forward more than 10k ops/sec from the database changelog to a downstream db used for freetext search.
