For VERY small use cases I would recommend a VPS, however because you're suggesting that you want a lot of storage then I would recommend at least a dedicated server but I would keep buying your own hardware in the back of your mind. I'll talk about this below.
VPS: If you're suggesting hosting a few websites and a small amount of email then a VPS is fine. Prometeus is my favourite for reliability and price. However bare in mind that with a VPS you're host can look into your files whenever they like (mounting the image/device if Xen/KVM) and just entering your container if it's OpenVZ. So for sensitive data I don't recommend a VPS. Also you may be put under restrictions if you max out your CPU and memory usage for long periods of time.
Dedicated Server: This is good for large storage and if you want unrestricted CPU and memory usage. You can obviously virtualize your own servers onto this and you can also run Windows and Linux (+ OS X). This has all the benefits of having your own physical box/colocation plus you get free hardware replacements.
Own hardware: If you're happy with just relying on your network connection then you can host it at home with a small quarter rack and just put a rack mount server in there. There are some benefits with this. If you're just torrenting, hosting email and a light web server or even just for testing you can get an old server off eBay or craigslist for next-to-nothing and you have the benefits of owning the hardware and putting in as many HDDs as you like. If you're coloing then you also get a very good network connection and reliable uptime, and a public-facing IP address. Oh and having your own physical hardware is cool, and colocation is even more cool and fun to do!
I don't have any experience with colocation, but it's incredibly interesting to me. I actually love the idea of having my own hardware. I actually was searching for storage units that had network drops (couldn't find any). The noir lover in me gets off on the idea of a 5x5 storage unit with just a small 4U cabinet against the back wall.
It's as simple as doing a search for 'data center near X'. Call up their sales office, and ask them how much per month for ping, power, and cooling. You may also be able to find VARs that lease out racks 1 or 2U at a time for cheap as well. Shop around, there's HUGE variance in pricing in this space. It also couldn't hurt to let the salesdroids know you're shopping for best price as well. I doubt there's a ton of wiggle room for a handful of U, but if you're talking a half rack or more they're generally willing to begrudgingly work with you on price.
VPS: If you're suggesting hosting a few websites and a small amount of email then a VPS is fine. Prometeus is my favourite for reliability and price. However bare in mind that with a VPS you're host can look into your files whenever they like (mounting the image/device if Xen/KVM) and just entering your container if it's OpenVZ. So for sensitive data I don't recommend a VPS. Also you may be put under restrictions if you max out your CPU and memory usage for long periods of time.
Dedicated Server: This is good for large storage and if you want unrestricted CPU and memory usage. You can obviously virtualize your own servers onto this and you can also run Windows and Linux (+ OS X). This has all the benefits of having your own physical box/colocation plus you get free hardware replacements.
Own hardware: If you're happy with just relying on your network connection then you can host it at home with a small quarter rack and just put a rack mount server in there. There are some benefits with this. If you're just torrenting, hosting email and a light web server or even just for testing you can get an old server off eBay or craigslist for next-to-nothing and you have the benefits of owning the hardware and putting in as many HDDs as you like. If you're coloing then you also get a very good network connection and reliable uptime, and a public-facing IP address. Oh and having your own physical hardware is cool, and colocation is even more cool and fun to do!