Disclosure: I used to work for GCP and launched Preemptible VMs.
Congrats! Can I suggest charging more?
IIUC, your business plan is $10/month for the company, regardless of number of users?
You probably save a company that $10 in a day or less for one GPU: One A100 is ~$3/hr on demand and ~$.90/hr as Preemptible, saving over $2/hr.
Said another way, your pitch is to recover a lot of the 70% discount that they aren't going to do themselves. If you were a managed training service, you could pitch yourself as "half the price of AWS or GCP" and keep the 20%+ margin with both parties being happy. (The problem is that pass through billing makes that obvious, you need to support lots of bucket security and IAM controls, etc.).
Fwiw, I would also branch out into inference! Preemptible and Spot T4s are commonly used for heavy image models, but many people pay full price. Inference that takes X ms can easily be handled "without errors" in the shutdown time. The risk is handling all the capacity swings.
Thanks for the feedback, yeah at this stage we really just wanted to find out if we can build something useful for the community. Agreed on the pricing suggestion.
Also, interesting point about inference. I'm not sure though how common it is for companies to need GPUs for inference. Because if you can have a CPU based inference model, which I thought was most common, it's probably not a big usecase?
I've got some comment somewhere on HN that says exactly that "try CPU inference first, it's pretty good".
The need to reach for a T4 comes when someone is doing a big model on images or video and wants sub-second response time. (Think some of the stuff on Snapchat, etc.)
i've worked with people who have needed GPU powered inference on AWS. Training had to happen on p3's but inference happened on g4's. The pricing is lower (and depending on the use case, often the overall cost) and spot savings are usually less dramatic as these instances can be very in demand.
Congrats! Can I suggest charging more?
IIUC, your business plan is $10/month for the company, regardless of number of users?
You probably save a company that $10 in a day or less for one GPU: One A100 is ~$3/hr on demand and ~$.90/hr as Preemptible, saving over $2/hr.
Said another way, your pitch is to recover a lot of the 70% discount that they aren't going to do themselves. If you were a managed training service, you could pitch yourself as "half the price of AWS or GCP" and keep the 20%+ margin with both parties being happy. (The problem is that pass through billing makes that obvious, you need to support lots of bucket security and IAM controls, etc.).
Fwiw, I would also branch out into inference! Preemptible and Spot T4s are commonly used for heavy image models, but many people pay full price. Inference that takes X ms can easily be handled "without errors" in the shutdown time. The risk is handling all the capacity swings.