It's interesting to see how IT is perceived as a cost-center and not essential to the business in healthcare; yet everything crumbles and collapses when there's an IT outage.
These organizations might not be culturally accustomed to have IT at the core of their business/mission, but it very much is. They might not value engineering skills and people in IT, but they have evolved an absolute dependency on those over the years.
The issue here is cultural, not technical. These ransomware attacks, breaches and outages are completely self-imposed. They can end anytime as soon as the hospital wants it. All they have to do is value and acknowledge IT as a fundamental pillar of their organization, comp it appropriately and make sure to run on hardware that's patched and safe to use. Really no different than washing hands before a procedure to stop infections. Else the cycle will endlessly repeat itself.
If you look at the details of the post, it wasn't the hospital but the billing department that was shut down. Rest of the hospital was fine.
So in theory the hospital could have operated but they couldnt bill insurers or government entities.
I'm looking forward to the day that AI replaces the meatsacks in these departments for how awful and poorly run they typically are.
>it wasn't the hospital but the billing department that was shut down. Rest of the hospital was fine. So in theory the hospital could have operated but they couldnt bill insurers or government entities.
A ransomware attack hit SMP Health in 2021. The attack halted the hospital’s ability to submit claims to insurers, Medicare or Medicaid for months, sending it into a financial spiral, Burt said.
So, they struggled to handle this for two years before they finally shut down.
This ransomware attack happened in 2021, and they were unable to bill Medicare and insurers for months. This is just like any other hospital closing for financial reasons
It's interesting that most do not seem to be much worried about the insane vulnerabilities and general lack of security in a lot of our critical infrastructure. You'd think people should be happy about ransomware attacks exposing how weak it all is, because that's concerning and needs to be worked on. These guys are financially incentivized to pen test even companies and institutions who don't take security seriously. Which includes a lot of public ones too.
What a Russian ransomware firm can do, sophisticated state actors manage with ease.
Damn you all are all in on victim blaming. Don't get me wrong, they clearly should have invested more in security, backups, etc, but a fucking hospital is closing come on.
Sometimes investigators know the actual actors involved with such attacks. 15 years ago I worked for Microsoft and went to a security presentation where the team had done some brilliant work and collaborated with law enforcement over months and had specifically identified an individual associated with a cyber attack. The individual was out of reach because of weak computer crime laws in the attacker’s country and poor diplomatic relations.
The FBI was co-presenting and touting the staggering amounts of manpower and funds dedicated to this investigation.
I leaned over to a friend of mine and said something to the effect of “with the state of organized crime in that country we could spend a few tens of thousands of dollars for a permanent solution”.
I always think back to that comment. Attribution in computer crime is hard, but attackers get away with impunity because they either hide behind anonymity or behind jurisdictional issues. Should we just let that go or use more pragmatic solutions?
Are you really advocating the government use extrajudicial assassinations whenever they have difficulties charging someone they accuse of committing a crime?
It was really a thought experiment rather than advocacy of action. The question I have is, is it right to let someone continue to hurt people just because we don't have an agreed on process to stop them?
> That sort of thing isn't the FBI's style though.
Agreed
It's very much Mossad's style - I don't know which US agency outsources the job to them, but if someone pisses of the US spook community enough, it does get ordered. I don't think they get pissed over what happens to a hospital in any country in the world, however.
But folks are probably doing this because of terrible job prospects in their country. If you pay their government to off this one person, you'd probably need to off five more next week.
FWIW, I don't work in a hospital, but I have relatively recently taken on as a client a small rural practice (8-15 employee range) and it's helped further open my eyes and give me more perspective. Big places should at least have dependably competent teams in principle, even if they have tons of expensive legacy, lots more strain/demand, and more perverse incentives and so on. But I think these challenges go up and down the stack in the medical world, and I don't think the available options are as friendly or as good as they should be. This place still only has quite limited internet options (though at long last fiber may come in the next few years), having everything be cloud-based isn't realistic on an unreliable 10 Mbps DSL connection. They had no "IT", just random locals (like us in fairness), which can be decent but also can be... less optimal (one set of intermittent issues they'd been having for years I traced to the previous guys who installed their ethernet not bothering to actually qualify it). And what the doctor had been able to figure out himself. It's a bunch of practitioners and then one front desk person who does all the billing and such and that's it. There was a mixture of windows 7/vista and Mac OS 10.10/10.11. But also a $30k panoramic Xray machine and a bunch of other smaller but still expensive systems that obviously they needed to have work and can't just "oh yeah just buy a new one!" This is a critical local resource in an area where there are towns with a total population of as little as 65.
It was doable on a budget with some work to put together a vastly improved setup for them with data going to a ZFS system with constant snapshotting and then that too getting further backups in a few ways to off site, but it took way more technical knowledge/metaknowledge then it should have. If I think about all the various components, this could and should be a lot easier. And even with what we went with I still can't say I'm truly satisfied. Like, why don't TrueNAS or everyone else in the space long since have watch dogs for signs of ransomware? Encrypted files are readily discernible from unencrypted files by computers, so why isn't there just a GUI option to "alert on threshold of change in encrypted files|filesystem entropy"? Or even "radical deviation in snapshot size"? The key thing about ransomware is deviation from norms, by definition it must suddenly make changes to a massive set of files in a way totally different from typical patterns. If all of a sudden a NAS/SAN observes any hints of that why in 2023 isn't that universally a red alert? Why is Webauthn not everywhere for admin pages? Hiding everything behind an internal VPN is good practice, but further layers would be good too. Etc etc, no doubt endless stuff I'm not even thinking of or don't know myself.
It's a huge societal drain to lose medical services over such events and fantastically inefficient. The cost to ensure everyone has at least some final fallback WORM recovery or the like, even if a month old, has to be a tiny fraction of the damage that even a single hospital shutdown (or slowdown even) causes. Somehow it's not getting priced in and responded to in a uniform manner. I really feel like I should be long since 100% out of a job on this sort of thing because there should be easy options for non-technical types, and that government should have universal help and guides for all medical providers from sole practitioner on up nationwide to get into a known good state.
Realistically, small independent provider organizations like that are going to fade away. Between IT issues (advanced persistent security threats), insurance costs, and regulatory compliance requirements it now takes greater economies of scale just to continue operating. Many of those small practices have already been rolled up into larger integrated care delivery organizations and that trend will only continue. I'm not claiming that this is necessarily a good thing, just that it's inevitable given the incentives in the US healthcare system.
Starlink may be a good option for giving rural medical facilities reliable access to cloud services.
Thanks cryptocurrency! What a useful technology. Wasting electricity, enabling wild financial speculation, and best of all shutting down critical public institutions like hospitals by enabling rampant cyber crime.
Is there anything this miracle technology can’t do?
Can you please not post generic flamewar comments to HN? Regardless of who's right or wrong, the millionth variation of this snarky dismissal is exactly what we're not going for here.
Not to defend crypto, but I can replace the word "cryptocurrency" with, say, "computers" and your sentence is equally true. It's pointless to blame technology for how it's used. Why not blame the people who run randomware operations, or the hospital administrators who don't backup their data? If a ransomware attack is enough to cause the hospital to shut down, that means they were a bad glitch away for shutting down, anyway.
> If a ransomware attack is enough to cause the hospital to shut down, that means they were a bad glitch away for shutting down, anyway.
I'm going to disagree here. A bad glitch isn't likely to encrypted entire systems and prevent any recovery of data. While a bad glitch could cause complete data loss, ransomware attacks are far more common.
You're not disagreeing with me. You've agreed that a glitch that leads to complete data loss is possible. If you still don't backup then you're gambling the operability of the hospital that such a glitch will never happen and that a ransomware attack will never happen.
>You've agreed that a glitch that leads to complete data loss is possible.
Complete data loss of one instance (whether "instance" is a single computer or single site as in the case of a fire/flood). But random glitches, mistakes, or events without any intention behind them are very different from hostile actors. They don't work to spread everywhere. They don't actively work to hide themselves. Random glitches, mistakes, or events don't try to install rootkits and persist themselves.
>If you still don't backup
Ransomware operators aren't stupid either. We've long since passed the point where "just have a backup" naively is enough, ransomware outfits will actively go after backups if it can, trying to move laterally throughout the victim organization, as well as do things like keep a key around long after everything is encrypted so it can stay transparent for awhile. The idea being that even if backups it can't touch are happening, by the time the trigger is pulled and all keys deleted weeks/months of data will still have successfully locked up with the backup systems dutifully backing it up. For many people, individuals even not just businesses, "we can restore but everything for the past 1-2 months will be lost" represents major expense/pain. Particularly in a medical setting.
It's not that there aren't solutions to that too, and yes I'd hope a big organization would cover it, but afaik it's not as easy as I wish it was. I don't think the challenge of malicious for-profit actors should be trivialized in the way you seem to have been doing. And the value function of computers vs crypto is astronomical, I don't think it's reasonable to equate the two as you originally did either.
>It's not that there aren't solutions to that too, and yes I'd hope a big organization would cover it, but afaik it's not as easy as I wish it was. I don't think the challenge of malicious for-profit actors should be trivialized in the way you seem to have been doing.
Yes, it's difficult. Such is the world we live in. It doesn't make sense to blame cryptocurrencies for that, and not the people who didn't do their job. The existence of cryptocurrencies and of ransomware is something that was outside of their control. Their backup policy wasn't.
>And the value function of computers vs crypto is astronomical, I don't think it's reasonable to equate the two as you originally did either.
You misunderstood my point. That, in balance, computers are much more valuable than cryptocurrencies is a matter of opinion (which I agree with). A Luddite could argue that if the hospital hadn't been using computers for billing, the attack would have been impossible, and could have a point that using computers instead of paper ultimately created more problems than it solved. All that changes is that the technology being criticized was computers rather than cryptocurrencies. But both arguments miss the point that neither computers nor cryptocurrencies are the problem, but rather how they're used.
>least because cryptocurrencies are impossible without computers which means by definition any value they offer is a subset of the value of computers.
Oh. In that case the energy waste, the scams, the ransomware, etc. are not harms arising from cryptocurrencies, but merely a special case of the harm arising from computers. I mean, I was keeping the two disconnected in favor of computers, but if you want to lump them together then by all means do so.
The “glitch” in most ransomeware and AP fraud attacks is an employee clicking on something or believing something they shouldn’t have. LinkedIn facilitates this, because it makes it easy to understand the org chart of a company and target newer employees with urgent, personalized requests from the “CEO.” One person at a place where I recently worked actually fell for it and was at a store buying a SECOND round of gift cards with his company card before he finally got a twinge and pinged someone to see if—perhaps—this might not be legit.
Training can only do so much. People have no attention span these days, and all warnings go in one ear and out the other, even in they are well-intentioned people (as this employee was).
> Do we need to legislate having big "FRAUD WARNING" signs on every gift card display, just like we need "do not use in bathtub" labels on hair dryers?
The market has provided these without legislation, at least near me.
Over a sufficiently long time scale, every organization will eventually have a disgruntled employee who intentionally tries to sabotage their systems out of spite or revenge or even as part of a criminal embezzlement scheme. Competent IT managers need to recognize and guard against that threat. Smaller hospitals may not have the scale or financial resources to hire such people so the only practical solution is to outsource their entire IT infrastructure to a larger vendor that can bring in the necessary competencies. It's sad that the situation has reached this point but we have to face the reality that IT security threats will only continue getting worse.
Hospitals are particularly vulnerable to ransom ware.
I don’t think you grasp the hodgepodge of obsolete computer systems that are attached to computer networks to run lab equipment, MRI machines, process pharmacy prescriptions, and manage a patient’s chart. Keeping all of this secure and properly backed up is pretty much impossible.
Seriously, I’ve seen machines in use running everything from ancient versions of Sun Solaris to Windows 95.
First, it's not impossible, it's just difficult. And however difficult it is, if those systems are necessary to maintain operation, then they're mission-critical and it's someone's job to make sure that they continue to operate.
Second, a ransomware attack isn't going to attack stuff like medical equipment, which wouldn't hold any important files anyway. It's going to attack workstations and servers likely running some version of Windows. If your IT staff can't keep that stuff backed up then you don't have IT staff.
It’s impossible to patch systems that are out of support and the security patch literally doesn’t exist. In some situations the original vendor themselves don’t even exist anymore.
Also regarding the servers and workstations, recovering from ransom ware isn’t as simple as just restoring from a backup. You’re still looking at days of effort restoring backups, reimaging workstations, and validating that everything works.
A vulnerable system that can't be patched shouldn't be in a position where that vulnerability can be exploited, if it's mission-critical. If it's mission-critical, it's vulnerable and can't be patched, and has to be exposed, then it's not fit for purpose and has to be replaced, or have a replacement (i.e. a backup) on stand-by.
Yes, restoring a backup requires effort. But surely less effort than losing the data altogether.
> I don’t think you grasp the hodgepodge of obsolete computer systems that are attached to computer networks to run lab equipment, MRI machines, process pharmacy prescriptions, and manage a patient’s chart. Keeping all of this secure and properly backed up is pretty much impossible.
> Seriously, I’ve seen machines in use running everything from ancient versions of Sun Solaris to Windows 95.
And yet the parking lot will be full of BMW and Porsches. There's definitely money coming-in; why not spend it on making sure the infra can perform when needed?
The hospitals themselves operate on relatively thin margins usually. Insurance companies, labor costs, pharmaceuticals, blood products, single use sterile items, housekeeping, taking care of uninsured patients that can’t pay the bills, and other regulatory requirements are extremely costly and eat up most of the budget. Reinvesting in IT infrastructure isn’t something that is a priority.
Crypto does have a valid case for paying people for work across country boundaries when financial systems to do this are unreliable or non-existent. (Or has prohibitive fees.)
Before anyone says paying criminals for illegal drugs is a victimless crime, remember that a significant portion of the illegal drug trade is run by the cartels and other organized crime organizations which do victimize people.
I'm already paying criminals every time I pay my taxes. Hell, even when I fill up my car with gas, or when I buy meds. If you really want to live ethically and not pay criminals, you have to be autonomous.
The real crime here is preventing me from opening my third eye!
Or honest people who live in countries that have a lot of criminals and/or have a corrupt government.
Life is extremely tough for legitimate businesses in countries such as Nigeria, due to the government's incompetence (some say active collusion) in dealing with fraud. Banks generally won't touch them, and even if they do, transactions are often delayed for weeks.
I get paid in USDC for programming gigs and third party exchanges in Iran convert it to Rials for me.
The third party exchanges resell crypto to other Iranians who want to use it to buy sanctioned digital goods.
It is the widespread use of crypto that makes this possible if it wasn't mainstream then there wouldn't be a difference between getting paid in crypto and a Google Play gift card.
Or people deemed to not deserve access to services (the entire nation of Russia recently). Or people who were cut off from payment processing, like Wikileaks.
If crypto was, at any point, not a tool of the rich elite of those countries then immediately, it would be banned.
Crypto isn't magic -- indeed, it's vastly more sensitive to state control, since anyone in control over any local part of the network can completely disable the system.
I mean it's actively used in countries with high inflation and bad currency management. Just because it's not used in your Wal-Mart doesn't mean it doesn't exist.
I generally disagree with your popular comparison with,
There is building a knife, and there is building a knife that is optimized to cut human flesh, with supportive ergonomics.
I'm sorry, but if you think cryptocurrency invented crime, I have no idea what to say. Ransomware existed welllll before widespread adoption of cryptocurrency, and some of the big ones still use the old infrastructure. Like Moneypak. Even Cryptolocker was mostly paid through vouchers.
These organizations might not be culturally accustomed to have IT at the core of their business/mission, but it very much is. They might not value engineering skills and people in IT, but they have evolved an absolute dependency on those over the years.
The issue here is cultural, not technical. These ransomware attacks, breaches and outages are completely self-imposed. They can end anytime as soon as the hospital wants it. All they have to do is value and acknowledge IT as a fundamental pillar of their organization, comp it appropriately and make sure to run on hardware that's patched and safe to use. Really no different than washing hands before a procedure to stop infections. Else the cycle will endlessly repeat itself.