Celery is the worst background task framework, except for all the others.
There are bugs and issues, but because so many people are using it, you’re rarely the first to stumble upon a problem. We processed double-digit millions of messages daily with Celery + RabbitMQ without major obstacles. Regardless of what people say, it should be your first go-to.
Celery has way too much magic crammed into it, it is very annoying to debug, and produces interesting bugs. Celery is/was also a "pickle-first" API and this almost always turns out to be the wrong choice. As a rule of thumb, persisting pickles is a really bad idea. Trying to hide IPC / make-believe that it's not there tends to be a bad idea. Trying to hide interfaces between components tends to be a bad idea. Celery combines all of these bad ideas into one blob. The last time I looked the code was also a huge mess, even for old-guard-pythonic-code standards.
I think Celery has a lot of magic happening under it. When the abstractions are so high, it's important they never leak and you don't see anything below the turtles you are supposed to see.
I often prefer designing around explicit queues and building workers/dispatchers. One queuing system I miss is the old Google App Engine one - you set up the queue, the URL it calls with the payload (in your own app), the rate it should use, and that's it.
Very relatable to a recent interview experience I had with a popular freelance platform for the backend developer position.
I never worked at a FAANG-ish company, and in the course of my 10-year career I spent most of my efforts on stopping the organizations from building the wrong thing in the first place, not on "making things scaleable" from the get-go. My view is that if you have product-market fit, you can throw money on the problem for a very, very long time and do just fine, so everyone in the org should focus on achieving PMF as soon as possible.
The question of "How would you scale a Django service to 10M requests per day" came up, and my answer to just scale components vertically and purchase stronger servers obviously was not satisfactory.
Quite honestly, I haven't. I've been playing with temporal, and since it's picking up pace, I thought to have an option to have Temporal in place of celery.
Type hints in Python add a great amount of visual noise to the code, and I actively avoid them wherever possible. If static typing is a must, use a language where static typing is not an afterthought, and let Python be Python.
Would you rather deal with a little visual noise or a runtime exception that you could've caught before code got to production? For me it's about tradeoffs, and so far the tradeoff has been well worth it.
Indeed, I almost can't read untyped python code these days. It just feels like "what the hell is going on here?" and "what is this object?" ever so often. Sorry to say that but most people who write python just aren't good API designers, or software engineers in general, and type hints can at least help others get a vague idea of what the intent was.
If you're not using Vim or Emacs, and not very proficient at touch typing, you're almost certainly limited by your typing speed.
Coding is an iterative process, regardless of whether you're handcrafting the code or using AI -- you need to move your thoughts / code / prompts from your head to the computer. You have to use the keyboard to do this. You have to do this over and over again, interleaving thinking with typing, and if you're fumbling for the mouse or smashing those arrow keys, your thinking is blocked.
Posthog's newsletter is one of the few I re-subscribed to after a general newsletter purge, top content.
Regarding the superday controversy: my best interviewing experience was a conversational technical interview followed by a paid-either-way-the-decision-went 2 week project. I did quite a bit of competitive programming so leetcode interviews are not a dealbreaker for me, but I feel there's just too much at stake for 1-2h coding exercise, and projects allow you to showcase all of your skills, not just speedcoding.
Of course they should still code, otherwise they're not engineering managers, they're project/product managers, and good engineers will not follow those people.
Whenever we talk about EM duties, it's always a list of fuzzy, empty words:
> Owning the team's strategy and roadmap, and ensuring efficient execution.
This is project management.
> Making decisions to ensure that the team is working on the right things and saying no to the things that don't matter.
This is product management.
> Dealing with fires, escalations, and other crises that pop up all of the time.
How can an EM that doesn't code deal with fires? The only thing they can do is pull the sleeve of someone else who does code, and then, what, hang behind their shoulder until the fire is put out?
> Building a strong culture within the team so that people are engaged, challenged, and motivated.
This is meaningless. We're not children.
> Mentoring and coaching your reports so they get better and can have more work delegated to them, thus increasing output further.
Mentoring them at what? If an EM doesn't code, how can they mentor in an area that's relevant to the mentee (i.e. a coding engineer)? They can coach them on moving Jira tickets more effectively, or playing office politics better?
> Managing the team's stakeholders so they can offer their steer to the team early and often.
Again, product management.
> Actively performance managing the team so that superstars can continue to shine and underperformers can be coached or exited.
Combination of meaningless and project management. In order to be able to evaluate someone's performance, you need to know their work. If you don't, the only metric you have is number of tickets, or "velocity" or whichever other bullshit metric you use because you're not in the trenches.
> Building close working relationships with other teams so that smooth collaboration happens across the organization, leading to a better and more cohesive product.
Office politics.
The only one that make sense is hiring and retaining great people, but you can't do either without being a technical person.
EMs that don't code making technical decisions is a showcase for divorcing decision making from suffering the consequences of those decisions. And having teams with EMs + tech leads + team leads etc. is just making things worse by diluting the responsibility.
It would be nice to include the generated sql queries along with the code samples though. I've been on a similar path recently and being able to see the queries was really helpful (even the ones that failed!).
There are bugs and issues, but because so many people are using it, you’re rarely the first to stumble upon a problem. We processed double-digit millions of messages daily with Celery + RabbitMQ without major obstacles. Regardless of what people say, it should be your first go-to.