A very difficult language to learn is navajo[1], in fact it was used during the WWII by americans to communicate and it was one of the few ciphers that wasn't broke by enemies.
Having no native teachers available increased the difficulty, and the WWII usage was against non-native English speakers. That said, the Native American languages that I know of are grammatically quite different from English.
In that specific case there were less than 30 people who knew that language in the US during WWII, and the navajo language was also completely different from other native languages.
UPDATE: I mean 30 people outside the navajo tribe.
I can attest that this language is very hard to learn. It is an incredibly fun language and very VERY descriptive. The hardest part is actually speaking the language itself, since we're not used to using some of the extra sounds that Navajo uses.
Also, single words literally translate into English phrases or sentences.
The language hasn't done much for me these past few years, until I started learning Chinese. It is freaky how similar the languages sound. And I don't mean linguistically, of course. Learning Chinese is thankfully easier for me since I can replicate the phonetics without trouble.
[1] http://en.wikipedia.org/wiki/Navajo_language