There are many historical examples where a smaller force has taken over large populations. You need some sort of military advantage to do that. Britain, Spain or going further back the Romans.
These involve components though that are unacceptable in modern warfare. e.g. the threat and execution of collective punishment against large populations. The smaller force doesn't need to subdue the entire population at once, just a chunk at a time.
For less extreme measures you would effectively need to put in place martial law and martial rule that governs every aspect of the populations' lives. In this case you need enough force on the ground all the time to enforce it and those forces are going to take losses in a guerilla warfare situation (e.g. the Russians in Afghanistan). Especially if someone is supporting the opposing side...
In Afghanistan it's a mix of issues:
1. The occupying force has low tolerance for casualties and is in defensive mode.
2. The population has a long history of overthrowing occupation and therefore assumes the same will happen here.
3. Everyone can see the lack of commitment for the occupying force. This is not a "win at all costs" situation.
4. The Taliban is supported by external entities fighting a proxy war.
5. The Taliban is, at least in theory, in a "win at all costs" mode.
The problem with being the good guys is that it's hard to win against guerilla warfare on foreign soil without being evil.
Here are some questions I have about it:
Why didn't the Allied occupation of Europe in WWI and WWII suffer from guerilla warfare? Were we more willing to accept casualties? Were we more ruthless about collective punishment? Is the culture of Europe such that Europeans don't do guerilla war? If that's the case, why did the colonists fight a guerilla war in the American Revolution, since they were culturally European?
I'm assuming the occupation of Japan went well because of the shock of being nuked and the surrender of the Emporer, if neither of those things had happened, the aftermath would have likely not turned out as smoothly for the occupiers...
The problem with being the good guys is that it's hard to win against guerilla warfare on foreign soil without being evil.
The Malay insurgence seemed to be beaten fairly well.
Why didn't the Allied occupation of Europe in WWI and WWII suffer from guerilla warfare?
Which Allied occupation of Europe would that have been? In WW2, everyone wanted the Nazi's out, plus there was the immediate threat of Russian invasion in most areas.
In WW1 the allies didn't deploy major forces in Axis countries for any significant period of time.
Additionally there were pretty significant numbers of fighting-age men killed prior to the end of WW1 & WW2. That left less potential guerilla fighters.
Is the culture of Europe such that Europeans don't do guerilla war?
One of the first large scale examples of semi-organised guerilla war was the Peninsular War, where Spain and Portugal (and later the British) forced Napoleon's French forces out of the Iberian Peninsula.
The name "guerilla war" comes from the Spainish use of it during that war.
Also, see guerilla resistance movements in France, Poland and the Balkans during WW2.
I'm assuming the occupation of Japan went well because of the shock of being nuked and the surrender of the Emporer
That, and the lack of potential fighters, and the fact the Allies pretty much kept the Japanese power structures as they were during the war (they were too worried about the Russians to change anything). The Japanese didn't have anything to rebel against.
The USSR occupation of Eastern Europe / Warsaw pact countries after WW2 did suffer from guerilla warfare, with some squads lasting until 1960'ies. And that was even with the full "good practice" of lots of boots on the ground, resettled bureaucrats, total control of whole economy with cushy positions for local collaborators, and no fear of using collective punishment.
For example, if some village peasants feed the guerilla troops (willingly or not), and you shoot or deport anyone suspected of that, not bothering about possible false positives while rewarding snitches/collaborators; and using mass media to make sure that everyone knows the consequences - then in a bunch of years that will reduce support for insurgents. But western troops under media coverage can't really do that effectively.
Also, using Afghan army to make peace in Afghanistan seems ineffective - there are well known benefits to classical approach used by USSR, earlier imperial Russia, Ottoman empire and Roman empire; you conscript troops from the occupied areas and send them to pacify other occupied areas with different and preferably alien/hostile culture. Conscripting Afghan soldiers to pacify Iraq, and conscripting Iraqi soldiers to pacify Afghanistan would reduce the risk of corruption, fraternizing, and defection. Again, USA probably isn't willing to do that.
I don't have an answer, but the Japanese bit at the end is fascinating. I read a book (may have been a history of the Kokoda Trail) which discussed the Japanese soldiers reaction to being captured. The expectation was to be executed or tortured. Once it became clear this wasn't going to happen, full cooperation usually occurred as the culture and loss signified the defeat by a superior leader and force, and allegiances shifted. I have likely jumbled this a bit, but the gist of it was that the Japanese soldiers reaction to loss was quite different to the European soldiers.
"These involve components though that are unacceptable in modern warfare. e.g. the threat and execution of collective punishment against large populations. The smaller force doesn't need to subdue the entire population at once, just a chunk at a time."
Not by far. The Romans would kill a large portion of a rebellious city/village (or all) and the rest would be shipped away as slaves. You wouldn't want to mess with them.
The Taliban is part of the local population. They will definitely intimidate, kill, terrorize but it's not the same.
EDIT: Just to be clear... I'm not proposing the US behave like the Romans used to. If anything, given the limitations it should have been more careful in the way it applies force.
Another fine example were the Mongols. They achieved a lot with very few soldiers. They also left mountains of skulls (literally) and cities where they murdered everything (including animals) in their wake.
Suffice to say, after a while nobody wanted to fight the Mongols. The subjugated did what they were told, because anything was better than the alternative: they might keep your prettiest daughters as sex slaves (if the daughters were lucky), kill you and all other members of your family, kill your dogs, kill your cows, kill the chickens, and then burn down the entire city for kicks. Maybe they even killed the rats.
Then they'd send back a scouting party a couple days later to execute any survivors who someone escaped the first massacre.
These involve components though that are unacceptable in modern warfare. e.g. the threat and execution of collective punishment against large populations. The smaller force doesn't need to subdue the entire population at once, just a chunk at a time.
For less extreme measures you would effectively need to put in place martial law and martial rule that governs every aspect of the populations' lives. In this case you need enough force on the ground all the time to enforce it and those forces are going to take losses in a guerilla warfare situation (e.g. the Russians in Afghanistan). Especially if someone is supporting the opposing side...
In Afghanistan it's a mix of issues:
1. The occupying force has low tolerance for casualties and is in defensive mode.
2. The population has a long history of overthrowing occupation and therefore assumes the same will happen here.
3. Everyone can see the lack of commitment for the occupying force. This is not a "win at all costs" situation.
4. The Taliban is supported by external entities fighting a proxy war.
5. The Taliban is, at least in theory, in a "win at all costs" mode.