Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

You can query your writes in Redis, so it won't be lost if your database goes down.

https://piwik.org/faq/how-to/faq_19738/



Note to anybody reading this but not following the link: parent poster means "queue your writes in Redis" and not "query."


I'm sorry, I am not a native speaker and it was already late.


Cos your Redis will never go down? Or need to be upgraded? Patched?


How does snowplow solve this problem? By writing to disk?


Snowplow is implemented as a unidirectional data pipeline:

    tracking -> collection -> enrichment -> storage
Between each step there is typically some kind of persistent queue (mostly S3/Kinesis), and data won't be lost if a downstream component is not operational. Examples:

* If your event collector is unavailable, raw events will be cached in the tracker in localStorage, SQLite or similar

* If your Redshift database is read-only for maintenance, enriched events will be held back until Redshift is writeable again


There is of course the opportunity for a collector to be down, but the aim is to keep those components super simple and rely on really durable storage (Kinesis, S3) managed by someone else.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: