One thing I would point out is that the simplicity of StackOverflow's hardware stack means that it would also be very simple to host in a cloud provider like AWS.
I would also point out that StackOverflow isn't an incredibly good performer compared to a number of other websites. Check it out on PageSpeed Insights. I also used a couple other tools that suggest that it loads slower in some regions than others.
I can get 60k IOPS on a nanode at $5/mo rate. You actually get a random amount between 20k-60k depending on the type of machine they provision your nanode on.
You want 60k IOPS on AWS? Be prepared to pay like 4-5k/mo. Want multiple systems with it so you can cluster a reasonably performance database? Pay for each and you need to special request the ability to go past 100k IOPS in a region.
Want the privilege of a dedicated vCPU? It's $1500/mo per region just to turn on the feature.
I tried to use AWS just as you mention here and the cost was 100-1000x what it would cost me at akamai cloud or Hetzner. All these fees I was unfamiliar with popped out and the cost was crazy high.
The system I tried to provision was functionally similar to the bottom end dedicated akamai cloud offering and it was going to cost 15-20k/mo instead of $72. Over 200x. And that was just the hourly provisioned rate, not including egress and other bits...
These high costs are Amazon's way of telling you that you aren't architecting for the cloud. In my ~10 years doing cloud infrastrcuture on AWS I've never worked at a company had to purchase dedicated IOPS or vCPU.
It's a real problem if you build on a VPS as your base. I was asked to prep such a system for scale on AWS and choked on the pricing. So we settled for what they offer at reasonable pricing and if further growth is needed, will need to hit eject on AWS and just go elsewhere.
I recommend starting elsewhere and save the migration of you just use EC2.
Most places will let you scale a VPS quite well and there is a logical handoff to your own hardware where you can scale to a system with 512 vCPU and TBs of ram for less than Amazon is charging for step two.
But the base configuration, for the money, is pretty sad compared to what you get else where. Scaling up is horrendously expensive compared to other providers.
Why would you bother with AWS for this I don't even know. There is no draw to AWS if you are only using EC2. You couple your architecture to their specialized services and go all in and eat the platform risk, and long term cost.
Or... don't even bother with AWS. It makes no sense to use just EC2.
I would not give much on those gambled page speed analysis. It comes down to: is ist fast enough? The answer clearly was yes during all those years for the users.
> One thing I would point out is that the simplicity of StackOverflow's hardware stack means that it would also be very simple to host in a cloud provider like AWS.
Yes, of course. Every cloud provider also does PaaS, but often it is also not the most cost effective option - often it is the most expensive one!
The cheapest option is often the option where toy get reliant on cloud-specific sdks, serverless, etc.
Yeah, I have to strongly disagree with that. Sure, AWS can be stupidly expensive, however it can also save you a lot of money.
A few years ago, I was the VPoE for a small-ish startup in Denmark. I was the only ops person, and I was able to provide 100% uptime for a year, including a migration from k8s clusters, AWS AZ outages and whatnot. During that time, I also reduced our monthly spend from $15k to $5k. This was a fully auto-scaling, redundant and highly available system.
When I started, every night I got woken up by alerts (working alongside the CEO and another engineer to fix things). By the time I had stabilised things, the only thing waking us up were our providers having outages.
I used to run a similar business earlier, without AWS. We owned our own hardware and had ops people going to data centres. I can guarantee you we did not spend less than $12-13k/month (AWS fee + my salary) in that company. Think closer to $100-150k/month.
AWS can be cheaper when you factor in the cost of employees. It can also be stupidly expensive when you use it the wrong way.
Another example: I host a small service that gets very seldom, seldom use. Maybe 20-50 people discover it and use it per month. I have the backend running on a Lambda, and it costs me about $0.5 per month. It took me 20 minutes to write the CloudFormation for that and push it through my CI pipeline to have it deployed. There is no way I could get cheaper hosting, uptime/availability, and faster time to market than with a Lambda.
If and when that service becomes more used where it warrants running full time, I’ll rewrite the request handling and throw it in my k8s cluster. But until then, I do believe this is the cheapest solution (for me).
> ... I also reduced our monthly spend from $15k to $5k. This was a fully auto-scaling, redundant and highly available system.
I believe you, but most people don't do this, sadly. You very clearly _get_ Cloud, but most don't, and financial optimisations often come much later down the line.
I have a similar story, though from almost 10 years ago, so maybe things have changed. As VPoE for a small startup with mobile messaging apps (something like a couple million MAU and ~200M messages/month), I orchestrated a lift-and-shift from a top tier managed hosting provider to AWS EC2. The result was about a 60% drop in monthly spend on hosting. We used almost nothing except S3 and EC2, with some provisioned IOPs for our SQL backend.
That said, it was a long time ago, and I'm sure competition and prices have changed dramatically.
Yeah that Lambda setup you mention is terribly cheap, and it is one of the very few extremely attractive services pricewise - but many people get lured into using other AWS services that aren't cheap, even though Amazon makes people think so by blurring the total price by splitting it into hours, calculating egress separately, dependencies on services that are billable by the hour (it's not obvious when you start that a private network has the cost of an NAT GW attached to it). Using only Lambda is really cost effective but frankly, few people resist the temptation and use it the way you described.
> it's not obvious when you start that a private network has the cost of an NAT GW attached to it
It’s not actually required.
And I don’t mean the completely unrealistic “if you run stuff on your private network that never needs to connect to the internet, then you don’t need a NAT gateway!”. I just mean you can implement routing to the internet in cheaper ways.
If you don’t need 100Gbps highly available outbound traffic routing like a NAT gateway provides… you run a t3.nano instance doing forwarding and NAT that will sustain 30Mbps and burst to 5Gbps for $3.80/mo. Just on raw bandwidth, unless you need more than a t3.large can push (~500Mbps continuous, 5Gbps burst), it’s cheaper to run an EC2 instance as your gateway rather than use a NAT gateway.
> Check it out on PageSpeed Insights. I also used a couple other tools that suggest that it loads slower in some regions than others.
And yet Stack Overflow has such strong SEO that outright cloning their content on a malware page will get you top 10 results.
I think people seriously overestimate the value of Web Core Vitals. At the end of the day a website is usually trying to deliver some value to a visitor: if you deliver more value by rejecting complexity, but sacrifice some speed to do it: users will still value the end result, and search engines will reward you.
I would also point out that StackOverflow isn't an incredibly good performer compared to a number of other websites. Check it out on PageSpeed Insights. I also used a couple other tools that suggest that it loads slower in some regions than others.