Very wide range depending on the company. Startups and the like $200K/yr+ and equity. The equivalent at Google is $500K - ?.
As an aside, Staff Engineers have to ride this strange line of being tech leads, managers, architects, and ICs. In my experience, one part or all parts will suffer. Will Larson's book[1] dances well around the ambiguity, but basically when you take a Staff Engineer job, it'll be unclear what you're signing up for.
How do you differentiate background and worker processes? I’m currently running VMs for background processes processing pub sub topics. Is that not something that could go on cloud run?
When an instance isn't handling a request it either gets killed almost immediately, or if even if you have min idle instances configured, its CPU get's throttled to nothing. So you can't have goroutines or some such doing anything outside of the request context. It's my team's one gripe with Cloud Run. It'd be great to have an option for long-lived instances.
Like I said, even if the instance is "alive", if it's not currently handling a request it's CPU is throttled to nothing so you can't trust any background process will finish its work.
In a previous company we were scraping customers web pages to load customer and product information because the number of systems and teams involved to connect to their CDP would take months to get up and running. A few problems we had to solve was the web pages changing underneath us and some pages not being formatted the same. We ended up having to detect those changes server-side to alert us to update the scraper. How do you all deal with these use-cases?
Also, we tried to leverage: https://www.diffbot.com/ but the lack-of-accuracy/lack-of-complete-data + cost never justified it's usage.
Yes, very good question (I've answered a similar one on detecting whether the page has finished loading).
It's a deceptively hard problem. Essentially, what we do is fingerprint the element. If the page changes, it boils down to how effective our fingerprinting and search algorithms are, to find the element if it has moved or changed.
The algorithms behind that are good enough for most use-cases now, but it's something we're continuously iterating on.
Eg: Start out looking at nanoparticle functionalisation, then stumble on a very useful sensor application or targeted molecular delivery.
Eg: Work on autonomous air sensor, then find simple off shoot application for tuberculosis using a completely different technology.
Eg: Build a medical laser for transdermal drug delivery then find applications in dental work and scar tissue removal.
Eg: Design a flexible heterogeneous compute cluster for hpc application, then get customers who want it for the low administration cost.
You start by bringing some very competent and diverse people together to tackle some tricky and new problem. Then after a while you'll have found a bunch of peripheral questions and applications for one thing or other you create in the project.
People talking and sharing experience and knowledge will almost always find new stuff out in the dark. If you're working on the edge of knowledge there is almost always interesting stuff hiding just outside the light of your personal flashlight. Bringing people together in multidisciplinary projects is amazing.
I think that while it is a common situation in small/scrappy companies, unless you want to get good at scrapping together tech solution, this may not be the best experience for you right now.
I was fortunate enough to start at one of the big four out of school and work on some cool projects. I eventually went to work at a few startups, where it was more about being a good plumber (similar to what you are doing) than being a good coder. Many
of the junior engineers got burnt out by trying to build and create solutions (where do I make algorithms?!?) around these 3rd party systems but the truth is, that wasn't what the business needed. Since I had already had my fix for creating some complex software, I did okay. I probably stayed at certain startups too long but I did get better at knowing when to build vs buy and how to build+buy+glue.
What helped me develop that knowledge was actually spending years building so I knew the cost of design/implementation/support/extension.
