Absolutely my favourite recent development in the Python world. I won't be mistaken if I say that many of us have built half-arse versions of this on top of urllib(2) multiple times in the past. :-)
The support for keep-alive is particularly welcome, huge kudos to the author.
Question to users of other languages: do equivalents exist for PHP, Perl, C# or Java? Ruby folks rave about restclient, but is there anything else?
httparty does a few of the same tricks, but I still find its API a bit obtrusive. At least it automatically parses JSON! (my library does this too, if an appropriate library is loaded)
In addition to a humane interface, it will soon be backed by a Ragel-generated HTTP parser:
Looks very promising. I see it currently uses Net::HTTP under the hood - do you plan to keep it that way, like Httparty and Faraday, or are you going to do away with Net::HTTP altogether? (The comment saying "this is temporary" next to requiring 'net/https' is encouraging... :))
Zed had two parsers: a server parser as part of Mongrel and a client parser as part of RFuzz. I actually used the client parser from RFuzz in my Rev/Cool.io libraries. I talked to Zed about his plans for these, he wanted to do a unified parser that can handle both HTTP requests and responses (since the only difference between the two is the first line) but he never got around to it. Both projects are now unmaintained.
Carl is working on a comprehensive event-driven HTTP (and Websockets) framework for Clojure. I'm just borrowing his parser and re-wrapping it in Ruby. I prefer to use Carl's as he's (at least for now) actively maintaining it, as opposed to using Zed's abandonware.
No, no it's not. (Http)WebRequest IS AWFUL. I built RestSharp because HWR is so bad. WebClient is closer to requests, but still horribly awful if you know anything about HTTP.
The new System.Net.Http.HttpClient in .NET 4.5 is closer to Requests and not entirely bad, if a little rigid.
- It's too low level for quick stuff
- It handles HTTP basic auth wrong
- It throws an exception, AN EXCEPTION, for any response that isn't 2xx
- Async is an ugly mess
- In love with streams
- Horribly inconsistent across framework profiles
- ServicePointManager.Expect100Continue
- Inconsistent header handling and restrictions
- Terrible cookie API
I think it's fine for simple scenarios (i.e. plain GET), but it breaks down in a hurry and the fact that you have to read out of the response stream is real overhead.
> I think it's fine for simple scenarios (i.e. plain GET)
So's python's urllib2, the point of `request` is that as soon as you go even slightly beyond very basic GET and POST, it breaks down quickly in a horrible mess.
I'm not your parent poster but I personally dislike that you have to instantiate a separated Response object, and that you have to fetch a stream, even for tiny responses. In those Python/Ruby frameworks it's just one line. I always end up writing a wrapper just to change it.
And this one is a pet peeve, but I hate that verbose stateful OO idiom where you instantiate an object (with a factory) and then use setters to change properties, instead of setting it all in a single command. I don't know if other .NET HTTP libraries change that, though...
Just about every library & language has a Request-like wrapper. It's nothing unique to Node, and certainly nothing unique to client-side JavaScript where native XHR is a nightmare.
This looks so nice. Is there any upcoming support for HTTP caching? I know it is not the easiest thing to do because you will probably need to make several cache backends (file, db, memcached, redis, etc.) to keep everyone happy.
Also I couldn't tell if there is keep-alive support, is there?
Cool! Is there anything I can look at to see how async I/O will work? Does all the async support depend on gevent monkey patching (as requests/async.py would suggest)?
Does that also mean that it will have better support for timeouts? I'm not interested in socket timeouts, more in a 'finish this in N seconds or just give up' thing.
For that reason I am using pycurl now for some stuff where I had previously used the requests library. Which has a really really horrible API :-/
The most interesting thing about caching is that you also get support for 304/Etags/Last-Modified - which is amazingly useful for any application that might hit the same URL more than once. I'd vote for that logic in requests core, with caching as a key/value interface (ship an in-memory or file-based default, but make it easy to hook in memcached/redis/etc ala Django).
I don't mean to badmouth Requests or Python in general, it certainly looks like an improvement, but can anyone enlighten me as to why an API that wraps an HTTP request with concise methods is #1 on HN, twice? This looks more like a good answer to SO question than "news".
This looks like all the other wrapper methods you spend 5 minutes doing, once, when you start a new codebase?
It's #1 because people vote it up; people vote it up because it interests them or they'd like to see more of it. That's the pattern for every link here.
We may agree or disagree with the consensus of the masses, but the masses make the front page.
The philosophy of Python is that those wrapper methods shouldn't be necessary.
I'm glad when language quirks and libraries get linked, it's good for discussion. (And more informative than a good third of the stuff that gets linked.)
I came here with the same question. Considering I come from PHP & JavaScript where the native APIs are awful, wrapping is a necessary step for sanity. Interesting that Python didn't have this until now.
Python has always had it to some degree, but merging these sorts of things into the core language is what major releases are for. No language is perfect. I think the reason you see this upvoted so much is there are so many Python programmers who want to see this on the roadmap for future versions of Python (and it probably will be.)
The design of this library reminds me of Apple's design... While a lot of Python is more like Google, or Microsoft -
they just don't get it. (Admittedly, parts of Python are very well-designed, but I still see a lot of room for improvement...)
I think LWP is more equivalent to Python's urllib2 (ie. both big beasts!). So this i believe is closer to what LWP::UserAgent (https://metacpan.org/module/LWP::UserAgent) does.
Apparently 113,798 people found a reason to download the 3.2.2 Windows installers (32 and 64 bit) from python.org through only the first 10 days of the month of October. http://www.python.org/webstats/usage_201110.html
(doesn't include mention of the 3.2 source tarball downloads, which are under 5,000, most likely since people get that from their distro anyway)
Forgive me if I've missed this, but does this library have support for multipart/form-data POSTing? That's a capability in the python library that's sorely missing - especially in 2.X.
Although Requests looks very nice, I've wondered why most people wouldn't simply use pycurl - http://pycurl.sourceforge.net/ ? It's stable and you get all the power that cURL has (which is a lot), and the API is fairly simple - although maybe not as pythonic as the Requests API is.
On a side note: last time I checked urllib2 (which actually drives Requests) could not do SSL requests over a proxy, which gave me the feeling that this implementation still isn't very mature.
I love Requests, but it would be really nice to have some Python libraries with cert-verified SSL support. I've hacked SSL support on top of urllib2 for my projects at work (https://github.com/josephturnerjr/urllib2.VerifiedHTTPS), but it'd be really nice if libraries started prioritizing secure communications as a first-order requirement.
Then I will be porting my projects over to Requests very soon :)
It's pretty easy to implement, as it already works at the lower (socket) level with the ssl module, it just hasn't made it up the stack into urllib2 (or Requests..yet!).
If you're using Twisted (or Tornado, ugh) then there are builtin HTTP clients which are async in all the right ways and provide similar high-level APIs for making requests. If you're using gevent, then this is what you want. If you're using eventlet, you should switch to gevent or learn Twisted.
So yeah, this is suboptimal, but not the worst thing in the world. It should really be documented though.
Excellent library but they need to fix this piece of misinformation: "A project that is released as GPL cannot be used in any commercial product without the product itself also being offered as open source."
This is false. A project released as GPL can also be released on another (commercial) license.
If Python is going to include urllib or urllib2 in the standard distro, why not make it good? Http is foundational to (web) programming. I am shocked at how poor the standard httpclient implementations are in most environments.
Oh, what I would give for a "mechanize for humans" built on this! I love mechanize, but I dream of not having to go back to its one page documentation and the source code every time I want to do something.
How does this compare (feature wise) to the poster library? Are they about the same? I've used poster some, but not requests... I'll have to try it now. Just curious how it compares to poster.
Given that there are a great deal of service APIs in the wild this is going to lower the barrier of entry for many less experienced developers looking to prototype their ideas.
"Requests is an ISC Licensed HTTP library, written in Python, for human beings.
Most existing Python modules for sending HTTP requests are extremely verbose and cumbersome. Python’s builtin urllib2 module provides most of the HTTP capabilities you should need, but the api is thoroughly broken. It requires an enormous amount of work (even method overrides) to perform the simplest of tasks."
The support for keep-alive is particularly welcome, huge kudos to the author.
Question to users of other languages: do equivalents exist for PHP, Perl, C# or Java? Ruby folks rave about restclient, but is there anything else?