I heard about that a few times already, but I never reached a point where my ecs setup combined with other aws services was not enough. If you had everything with just stuff in kubernetes wouldn’t it still be a pretty big deal to migrate?
As a deployment model I bet ECS, fargate and lambda are great. But at the scale of my projects (small) I like being able to run a copy of the full infrastructure (or as much as possible) locally and reuse as much as possible from "prod".
And regarding Kubernetes migrations, once you've made sure you have network and DNS connectivity cross cluster it's essentially just replacing the CSI and LoadBalancer controller. Then the actual data migration there's no magic bullet, depends on what you run.
The USP for Kubernetes is that it's essentially the same no matter where you run it since everything conforms to the Kubernetes API spec.
If you don't want or need local development LARPing prod then anything goes.
So for stuff like secrets management, buckets, api gateways and such, you deploy those services to k8s? And if you don’t mind, is maintaining those services cost effective? I’m asking because I’m always looking to do the trade off of money per time
Kubernetes already has simple secrets, good enough for me.
I would provision buckets with Terraform/tofu, we just use ingress so idk about API gateways.
The eye opener for me was "I can just do this in Kubernetes", which is pretty much always true (though not always right).
Kubernetes + Prometheus + Grafana (with friends), cert-manager, CSI, LB and some CNI you have something resembling what I'd use from $cloud provider.
Deploying K3s is really easy, it can definitely be a time-sink when you're learning but the knowledge transfers really well.
You also don't really need all Kubernetes features to use it, you can deploy K3s on a single VM and run your pods with hostnetworking and local path mounts, essentially turning it into a fancy docker-compose which you can grow with instead of throw out.
I value FOSS and being able to run "anywhere" with the same tools. K8s and Postgres gets me there, I haven't worked on any "web scale" projects though but I know both can scale pretty high.
I personally reached the multi-cloud situation a handful of times - it's particularly common if you're doing anything on-prem, or with customers on certain regions (eg China). If you're truly married to AWS/GCP/whatever stack (fargate, lambda, etc), it's literally impossible to migrate. The successful migrations / multi-cloud setups I've seen all used the cloud provider's specific features sparingly (eg S3 encapsulated on a single library, so migrating is simple), etc
But I agree if you're doing something simpler, sticking to a single provider is fine
