It's not just the US government that has crazy, weird, inefficient technology. I'd be surprised if every government wasn't like this. The biggest IT failure in the world was the UK's attempt at healthcare computing that cost 12Bn GBP and didn't deliver a functioning system.
I work as a contractor for the Australian government. I personally know of multiple project failures in the 10s of millions of AUD range and a few in the hundreds of millions. These stories don't even make the news.
I've worked at small companies, research institutes, universities and now in government. I've not worked big corporate but have heard that it is similar, although more efficient than government. Size means you get less feedback on what is really useful.
Government fundamentally lacks feedback on what really matters. In the US the department of health cannot be driven out of business by another department that does what is important 10% better. In private industry that discipline and feedback makes things work.
If you build a widget X and it isn't something that people want you go under. That doesn't happen in government. If you build a donkey but it's the donkey they paid for it could be in service for 20 years.
It's hard to see how to make it all better. Perhaps trying to keep components small and having multiple groups build them and select the best might help. Then at least 2+ groups would have to compete to build a better system.
Thinking a little more about the situation, consider that the majority of startups fail. I would not be surprised, if one were to take a look at the success rates of internal projects in large companies, to learn that those are fairly low as well. So if the private sector is more efficient than the public sector in IT, the differences are probably more subtle than one would at first think. In both cases you have lots of capital being spent on projects that won't come to fruition. Perhaps the incentives in the private sector are a little better aligned towards a successful outcome.
Yes, it's believed that most of those internal projects fail. Canceled, declare victory because of the internal political stakes but quietly not used (very much), delivered with a fraction of the original features, etc.
For some time I've thought this was one of the primary attractions of offshoring: if you must maintain the pretense of developing new programs and systems, it's a cheaper way to inevitably fail....
I kind of like the idea of every government function has two independent providers that compete for funding, and citizens could choose which provider to use.
I'm sure that system would blow up in some other way, however.
I think you'd need a lot more than 2. It's hard to get meaningful competition from only a smaller number because it ends up being much easier and profitble to collude (see telcos/cable companies/etc).
Also, it might be worth mentioning where I work Agile development is used. It does help some but it isn't a silver bullet.
Also, I didn't explicitly say so but small companies are more efficient. If they are not, they tend to go insolvent. That's what is so good about them.
I work as a contractor for the Australian government. I personally know of multiple project failures in the 10s of millions of AUD range and a few in the hundreds of millions. These stories don't even make the news.
I've worked at small companies, research institutes, universities and now in government. I've not worked big corporate but have heard that it is similar, although more efficient than government. Size means you get less feedback on what is really useful.
Government fundamentally lacks feedback on what really matters. In the US the department of health cannot be driven out of business by another department that does what is important 10% better. In private industry that discipline and feedback makes things work.
If you build a widget X and it isn't something that people want you go under. That doesn't happen in government. If you build a donkey but it's the donkey they paid for it could be in service for 20 years.
It's hard to see how to make it all better. Perhaps trying to keep components small and having multiple groups build them and select the best might help. Then at least 2+ groups would have to compete to build a better system.