This is one reason why I'm so frustrated by the "cancer research is not reproducible" news stories that get out.
Not because cancer studies actually are reproducible, but rather because the reason they are not so often reproducible often comes down to the high complexity of the disease.
You grow your HCC1139 cell line in your lab, with your own supply of reagents, and you might get very different gene expression profiles of a lab in a different country growing HCC1139 cell lines on their own supply of reagents. There's so many subtleties in what is going on underneath in these systems that getting them right is insanely hard.
We don't know all the inputs, all the variables, that influence results. Getting the same result in one lab three times doesn't mean that another lab will get exactly the same results. But until we publish the results, we won't know what does and doesn't reproduce, unless there's that stake in the ground and a concerted effort to spend the highly limited time focused on pushing forward the boundaries of knowledge, we won't know.
So even if a broad range of hundreds of wet lab techniques, across thousands of experimental systems, doesn't always get the same result in different hands, unless we start to push at the amount of detail going on we'll never understand how to start to treat all these things.
Anyway, that's my rant, and I can understand why outsiders are concerned about individual cancer research experiments not reproducing at a very high rate, but I think it needs to be placed in this sort of context, where we only have measurement tools for a tiny fraction of what's going on, and need to still discover all the unknown unknowns.
I completely agree, but I also think there is some truth to the related statement: 'cancer research often isn't conducted in a way that is actually useful'!
For example, in-vivo tumor experiments in mice can yield completely different results depending on exactly where the tumor was implanted. E.g. a 'lung cancer mouse model' may have the lung cancer injected just under the skin, also known as subcutaneous tumor models, instead of in the lung! Entirely because it's a lot more efficient + yields more trustable data, but the results are often deeply disconnected from how the tumor would naturally grow + respond to drugs within its host organ.
If your experimental system is so delicate that it can't be reproduced in another location (by expert hands), is it really a good experimental system? To me it sounds like a terrible experimental system that is brittle and does not generalize.
Let me give an example of a good experimental system: restriction endonucleases.
You can download the original papers where EcoRI, BamH, etc, were isolated, purified, and demonstrated to show sequence-specific DNA cutting. Any student in any lab in the world can pull those papers- some 40+ years after they were published- and reproduce them using trivial old techniques and map any random plasmid with it. It might work a little worse in an incredibly hot lab, or a very cold one, but you'll still get all the activity you need to demonstrate that you purified a restriction endonuclease.
There are a lot of researchers out there who have something in their lab that they use to generate papers, but who are not actually discovering anything that is robust and generalizable- and those are required steps for treating people with cancer reliably at scale. Many results are irreproducible because the authors were incompetent or lying (or outright fooling themselves and others without knowing).
Thanks for your posts! I've been very impressed with your ability to both be at the leading edge of knowledge and communicate the parts that are most interesting for a broad technical audience, it's an impressive skill.
Not because cancer studies actually are reproducible, but rather because the reason they are not so often reproducible often comes down to the high complexity of the disease.
You grow your HCC1139 cell line in your lab, with your own supply of reagents, and you might get very different gene expression profiles of a lab in a different country growing HCC1139 cell lines on their own supply of reagents. There's so many subtleties in what is going on underneath in these systems that getting them right is insanely hard.
We don't know all the inputs, all the variables, that influence results. Getting the same result in one lab three times doesn't mean that another lab will get exactly the same results. But until we publish the results, we won't know what does and doesn't reproduce, unless there's that stake in the ground and a concerted effort to spend the highly limited time focused on pushing forward the boundaries of knowledge, we won't know.
So even if a broad range of hundreds of wet lab techniques, across thousands of experimental systems, doesn't always get the same result in different hands, unless we start to push at the amount of detail going on we'll never understand how to start to treat all these things.
Anyway, that's my rant, and I can understand why outsiders are concerned about individual cancer research experiments not reproducing at a very high rate, but I think it needs to be placed in this sort of context, where we only have measurement tools for a tiny fraction of what's going on, and need to still discover all the unknown unknowns.