How to Spot and Avoid Bad Science Takes
--
The bar is on the ground at this point, but Jordan Peterson, Candace Owens, and David Frum still can’t make it over.
I normally talk about science without talking about current events because I’d rather not be crucified online by people who think in terms of bumper stickers. Being hit with all three takes in the same week was too painful, though. To ease my pain, I want to use these takes as a starting point to clear up some misconceptions about science. This article is not about politics, but about the specific arguments presented here. Even if these people are on your team, I want to at least make the case here that their arguments are incorrect.
Why Cover These Takes?
Climate change and COVID disinformation campaigns are arguably the worst forms of science denial out there, but I haven’t mentioned them. I’m not an expert and others have responded better than I could. With that being said, I have done my research. On COVID, I’ve found that
- COVID is far deadlier than the vaccine. Around 2% of people who get COVID die, while the death rate for all the vaccines is at most 0.0022%. To get that number, they’re including literally everyone who had any kind of adverse reaction to the vaccine and die afterward.
- COVID has far more frequent and far more debilitating symptoms. Remember when the FDA and CDC recommended a pause on the J&J vaccine after 5 out of ~10,000,000 people had blood clots? Sounds pretty dangerous, doesn’t it? Well, if you think that’s dangerous, you should think COVID is incredibly dangerous. You’re 8–10 times more likely to get a blood clot from COVID. If you do this comparison for every symptom (including death), you’ll find that the vaccines are far safer than COVID.
- All vaccines greatly reduce the likelihood of getting infected, getting seriously ill, dying, or infecting others.
You won’t find me talking about COVID, though. My ability to distinguish between good and bad statistics does not make me qualified to talk about the mechanics of a cytokine storm. If I were to write about COVID, I could end up saying something incorrect and be used as part of the disinformation campaign, so I don’t.
So Why These Takes?
On the other hand, I know a thing or two about the specific takes I’m responding to in this article and I can provide a response to these takes that you won’t be able to find elsewhere.
The Fable of the Indestructible Fortress
A company of knights had been sent to siege a supposedly indestructible fortress. When they get to where the location, they could only find an enemy knight sitting on a stone in a clearing.
“You there! Where are your men?” said the leader of the company.
“Our indestructible fortresses only need one knight to maintain.”
“If you’re so confident, why don’t you tell us where it is?”
“It’s all around you,” replied the enemy knight. The knights looked around and saw nothing but a forest.
“Are you mocking us?”
“Yes, but I am telling you the truth. Our king realized that fortresses made with stone and timber could be destroyed, but no attack could destroy the rubble. Breaking broken stone leaves you with broken stone. The broken stone remains.”
“So where is the rubble that is this fortress?”
“It would be a waste of effort and resources to build something just to break it down. The rocks, trees, and ground are sufficient.”
“What if we cut down the trees?”
“We’ll still have the stumps.”
“And if we remove the stumps?”
“The ground remains.”
“And if we dig up the ground?”
“We’ll use the ground under it.”
“And if we dug to the other side of the Earth?”
“You can try, but I’ll still have the air, and maybe some water.” The company looked around at each other in disbelief. The enemy knight broke the silence.
“I too was stunned when I heard the idea, but it’s quite ingenious. Now, if you would kindly put on some indestructible shackles, we — ” He fell to the ground with an arrow sticking out of his chest.
“He was starting to annoy me,” said an archer of the company.
What is Science?
My memory of when I was a kid is foggy and distant, but I remember some know-it-all kids.² Every so often someone would claim to be smarter than someone else. To prove which of the kids were smarter, one of them would ask the other kid a question³ like
- Who was the first president?
- What are the three primary colors?
- What is the ground state energy of a Quantum Harmonic Oscillator?
You know, normal kid stuff. Sometimes, a kid, which we’ll call the liar, would respond “I know what the answer is, but I want to see if you know it.” The other kid would then say the answer because kids are stupid. The know-it-all would say, “Yeah, that’s right,” before asking a question of their own. With a response like that, the liar could get the credit for answering the question without answering the question. You couldn’t prove that they didn’t know the answer in the same way you couldn’t destroy a “fortress” that consisted of whatever was nearby.
Falsifiable Claims
Instead of leaving things vague, waiting until the answer comes out, and claiming it knew it all along, science makes claims about the world that anyone can test. Scientists recognize the inanity of the indestructible fortress, so they build a real fortress. The fortress may not be able to weather any attack, but scientists understand that the goal is to build the strongest fortresses they can to protect humanity from a cold, indifferent universe. To do so, they’ll tell everyone else exactly what they would need to do to knock it down so they can get rid of the weak links. To give an example, I’d like to list a few predictions made by different theories.
- The Modern Evolutionary Synthesis gives precise predictions for which fossils you’ll find in which geological layers along with which DNA you’ll find in which animals.
- The Modern Evolutionary Synthesis also predicts that all animals can be arranged in a hierarchy based on morphological similarity that will exactly match the hierarchy given by genetic similarity.
- Nuclear Physics gives precise decay rates for isotopes.
- Condensed Matter Physics predicts that you can make a diode by sandwiching positively and negatively-doped silicon.
- Newton’s Law of Gravity predicts the paths massive bodies will take.
- General Relativity also predicts the path massive bodies will take.
If you find a fossil or a planet out of place, you will have blown a hole in whatever part of the theory made the prediction. And this ability to blow holes in theories isn’t some hypothetical. We’ve done it before with Newtonian Gravity, half a dozen models of the atom, Classical Mechanics, etc.
A New Theory
To be clear, whatever you propose should also explain whatever the old theory was able to explain. It is possible to knock down a single tower while most of the fortress remains standing. For example, General Relativity and Quantum Mechanics include Classical Mechanics as an approximation in some limit. If someone proposes a new theory after they’ve “debunked” the old theory without showing how the new theory explains and predicts everything the old theory explained and predicted, they’re charlatans not to be trusted.
David Frum
With all that out of the way, I’m going to start with Frum’s take because it’s the least harmful and the least serious. It’s a Seinfeld-style joke with a bad premise that other people have told so often that it has shown up in scientific papers and books on climatology¹.
It’s Just a Joke, so You Shouldn’t Criticize it, but it’s Funny Because it’s True
If you want your fans to defend your take, couch it in a joke. Your fans will take you seriously while insulting others for doing the same. To get around this glaring double standard, I want to make it clear that this entire section is a joke about the premise that “meteorologists make inaccurate predictions that other professions wouldn’t allow.” If you want to criticize the premise, then it’s just a joke and you need to lighten up. If you don’t think it’s a joke, then it’s obvious that you don’t like having your worldview destroyed with facts and logic. If you don’t think it’s funny, you’re being too sensitive.
Throwing Stones in a Glass House
I want to avoid talking about politics as much as possible in this article, but I have to mention that David Frum made several infamous predictions about the Iraq War and WMDs and kept his job. Regardless of your political views, his claim that the fighting would be over in a few weeks was off by years.
I’ll Have Six Milk, Please
A problem becomes clear if you rewrite the tweet with different units.
0.0000158 – 0.000095 miles of snow predicted tomorrow? Can you imagine if any other profession allowed itself a similar range?
“Your plane will depart tomorrow between 1:00:00.158 and 1:00:00.95.”
“We’ve bought you a dress between size 1.0000158 and 1.000095.”
“Best to limit your daily wine intake to between 1.0000158 and 1.000095 glasses.”
You might argue that I’m not using the natural units for snowfall, but there isn’t really a set of natural units for snowfall. It’s more along the lines of “At X inches of snow, Y will happen,” where Y is something like certain roads closing, schools having a snow day, etc. There’s also more to the weather than the number of inches in a snowfall.
Dresses, Cable Companies, and Astrophysics
As many women pointed out in the responses to the Tweet, dress sizes are inaccurate. As many people pointed out, cable companies sending someone over to repair your TV actually do come at some time between 1:00 PM and 6:00 PM. Lastly, astrophysicists work with large enough quantities that they only need to be accurate within a few orders of magnitude.
They can get away with this because there’s no practical difference between 10⁷⁵ years and 10⁸⁰ years.
Nonlinear Coupled Second-Order Partial Differential Equations
The fundamental equations that govern fluid flow are the Navier-Stokes Equations and the corresponding continuity equation. In Cartesian coordinates, they look like
where everything but the ∂, x, y, z, and μ is a function of position and time. As you can imagine, they’re quite difficult to solve. To give you an idea, here’s a quick table for reference:
If you look with a keen eye, you may notice that every word that could make solving the Navier-Stokes equations harder is in the title of the section. They’ve been around for over a century, but we don’t even know if they’re guaranteed to have physical solutions for all possible initial conditions. I don’t know about you, but I’m going to cut the weathermen some slack.
Chaos Theory and the Weather
Everything I’ve said in the previous section is difficult to deal with, but we do have methods of approximating solutions, i.e. integration methods with a small time step. Unfortunately, these equations have a larger problem: they’re chaotic. Slight differences in the initial conditions (a complete description of the system, where “complete” can mean different things) of chaotic systems will eventually lead to the systems having vastly different behavior. All the words in the previous sentence are doing some heavy lifting. I would recommend checking out the video below to get a better idea of what makes a system chaotic.
In a practical sense, we can only know our initial conditions of the weather with enough certainty to make predictions about the local weather that are accurate up to a few weeks in the future.
Ensemble Methods
The chaotic nature of weather means we have to use ensemble forecasts. Ensemble forecasts run a large number of simulations with changes in the initial conditions based on the certainty of our observations. For example, if we are 90% certain that the air pressure is between 1 atm and 1.01 atm, then 90% of our simulations will have the initial air pressure between 1 atm and 1.01 atm. Once we’ve run all the simulations, we go through and see how many of them had less than 1 inch of snow, how many of them had less than 2 inches of snow, etc. Doing so will get us an estimate of the probability distribution for receiving x inches of snow.
How to Report Data to the Public?
So what should we report to the public? In an ideal world, we would give them the raw probability distribution. People would then take the action that maximized their expected utils. We do not live in an ideal world. People are awful at probability and statistics. If they understood probability, casinos wouldn’t be able to make a profit and we wouldn’t have multiple unjust convictions because someone brought up a bad statistical argument in the courtroom.
So maybe we try reporting just the average? The average isn’t always a useful measure. The average human has approximately one testicle and one fallopian tube, but the likelihood of meeting this average human is almost zero. In the case of snow, if there’s a 90% chance of not getting snow and a 10% chance of getting two feet of snow, then reporting that we’re going to get 2.4 inches of snow will be inaccurate. If concrete can settle properly as long as there are less than 4 inches of snow (I’m making up a number for this example.), then people will pour concrete thinking they’re safe even though there’s a 10% chance the concrete will fail. They’d be better off waiting until next week.
So maybe we only report snow if the risk is above a threshold? If we set the threshold too high, people would get mad at meteorologists because a low-risk snowstorm would eventually hit. People will pour concrete and have it ruined. If we set the threshold too low, people would still get mad at meteorologists for overreacting. People will delay construction projects because they refuse to pour concrete unless it’s safe or outright ignore the meteorologists.
All in all, the best thing they can do is give a simplified probability distribution, like “there’s a 90% chance snow will be between 1 and 6 inches,” which is what they do.
Candace Owens
David Frum is in this article because I happened to see his take the same week as the other two, it’s funny that he of all people would say it in a public forum on the internet, it allows me to bring up a good point about the difficulty of reporting data to the public, and meteorologists have a difficult job. As annoying as it is, it’s the least harmful of the three takes. The idea that the moon landing was faked is an order of magnitude more harmful. Every moon landing conspiracy has been debunked in a million ways, with everyone from the USSR during the Cold War to Mythbusters providing their own unique debunking.
Keeping a Secret
You would think that at least one out of the 400,000 people involved in the moon landing or a member of a foreign government that agreed that the moon landing happened would have cracked by now. That argument alone is enough to disprove the conspiracy theory.
Fuel Tanks
Rockets don’t carry the fuel up with them. They shoot it out the back. That’s how rockets work. By the time Apollo 11 reached low Earth orbit, most of the initial mass was left in the atmosphere, especially since they use multi-stage rockets that drop not just the fuel, but also parts of the rocket. You can find out how much mass you need to get a certain mass going some speed using the Tsiolkovsky Rocket Equation. Deriving the equation was one of the exercises I mentioned in the second article in my series on Quantum Mechanics because it’s not too hard to derive on your own from what I said in the article.
Notice how the change in velocity only depends on the ratio between the initial mass and the final mass. In other words, double the final mass means you have to double the initial mass to get the same change in velocity.
Erasing Originals
Until recently, being able to store any amount of video for long periods of time was cost-prohibitive. VHS and Betamax came out in the 70s after the first moon landing. Studios frequently wiped tapes to reuse and save money, like they did with Doctor Who. Some form of wiping happened even up to the 90s, like how Konami lost the source code for Silent Hill 2. With that being said, the footage was viewed live around the world and recorded by a lot of people. The only difference between the footage we have and the footage we lost is that the footage we lost is of higher quality. You can also look at the reports NASA released on the tapes.
Live Broadcast from the Moon
I can’t really debunk her claim about the live broadcast from the moon being impossible because she hasn’t given a reason why. Is it because the signal couldn’t get to where it needed to get to? As long as radio waves have a clear line of sight, enough power, a decent mechanism to focus the signal, and a decent mechanism to receive the signal, they can transmit data over long distances. The Unified S-Band had all that covered. Furthermore, you can also read the specs for the lunar lander to verify that it had enough power to do what it needed to do.
History of Moon Landing Conspiracies
It’s also interesting to look into the history of the conspiracy theory, which includes some guy making up some nonsense and then FOX News airing an hour-long documentary about how the moon landing was faked.
The General Problem
Every scientific claim is accompanied by some way you can disprove the claim, as I’ve listed above, but Candace is not making scientific claims. What experiment could I perform or math could I do that would support the claim that the fuel tank was too big? What is the maximum size of a fuel tank we could use? Does it depend on the rocket size? How could I show that the live broadcast was impossible? How high quality must the video be for it to count as evidence? How much power would I need to broadcast a reliable signal from the moon? Is there some fundamental limit on how far a signal can be sent? How much signal loss should we expect? How much error correction should we send to cancel out the noise? Do we have the bandwidth to cover the error correction? What is the exact series of steps we would need to fake the moon landing? Was video technology advanced enough to fake the moon landing? With science, you can answer all these questions exactly. Her arguments are rubble, indestructible only because they are whatever she finds around her.
Jordan Peterson
Jordan Peterson’s argument is the worst not just because climate change will lead to a large number of small collapses until one finally breaks the camel’s back, but also because his statement promotes a fundamental misunderstanding of science bordering on the most asinine presuppositionalism (“If you don’t know everything, then you can’t know anything.”). I won’t cover whether climate change is real in this article, but I will respond to the claims Jordan Peterson made.
Cases are Difficult to Predict, but Averages Aren’t
Earlier in the article, I talked about how predicting the weather is quite difficult can be difficult to explain why we should cut meteorologists some slack. You might therefore be thinking that since climate technically includes the entire Earth, it might be harder to predict. It’s not. For every problem, there are three scales to a problem (I haven’t heard these terms elsewhere, so I might be making them up):
- The microscale, where the size of the system is small enough to fully model. Modeling a quantum system containing a few individual atoms would be on the microscale or the airflow in a PC is a microscale problem.
- The mesoscale, where the size of the system is too large to model directly, but not large enough to use statistical methods. Modeling the weather is a mesoscale problem.
- The macroscale, where the size of the system is so large that statistical methods allow you to make macroscopic predictions. Modeling the climate is a macroscopic problem.
The microscale is easy because you only have to consider a few objects, e.g. the sun and the Earth. The macroscale is easy because the system gets so large that deviations from statistical predictions are not noticeable. For example, even if we were to approximate atoms as hard spheres governed by Classical Mechanics, we would have to use every computer on Earth just to store the position, velocity, and orientation of all the molecules in a few liters of air. On the other hand, the Ideal Gas Law, which is a statement about all the molecules in the gas, is
You might be wondering how we can take all the behaviors of a large number of atoms that are acting effectively randomly and still come up with such a simple equation. You can find the specific answer to this question in one of my previous articles, but the general idea is simple: cases are difficult to predict, but averages aren’t. I can’t predict the number of hairs on your head, but I can tell you that if you were to pick a million people randomly and calculate the average number of hairs on their head, you would get roughly 100,000.
All Models are Wrong, but Some are Useful
The title of this section is a quote from the statistician George Box, but the idea’s been around for quite a while. When Newton derived Kepler’s Laws, he ignored all masses except the sun and the planet in his calculations. Newton also did not incorporate the effects of Relativity into his calculations. Newton’s model was wrong… but not by much. For example, calculating the precession of Mercury using Newtonian Mechanics while including the other masses in the solar system will give you an answer that’s off by 0.012° (a.k.a. 43 arcseconds) per century. I mean, sure we can do better with General Relativity, but being off by one ten-thousandth of a degree per year is pretty good.
To put it in perspective, if we were to draw an image of the solar system predicted by Newtonian Mechanics and the solar system predicted by General Relativity after a century, we would need a resolution of around 1,000,000 by 1,000,000 pixels to see a difference (I’m playing with astronomer-style numbers, so I’m just trying to get in the ballpark.). Every other planet has significantly less precession, so you would only see a deviation with Mercury.
In other words, if I were a sailor using the planets to navigate or trying to send someone to the moon, Newtonian Mechanics gives a good enough answer. But how could it? It doesn’t take everything into account, and therefore it must be wrong. The answer’s pretty simple: some factors matter more than others. While other bodies in the solar system and Relativity do affect the orbits of the planets, the sun has the largest effect on everything in the solar system by far.
This is a feature of science. You get a complex model with a lot of parameters, figure out which parameters have the largest effect, and use those parameters in your simplified model. If you need, you can add the parameters you’ve ignored back into your model with techniques like Perturbation Theory. To answer Jordan Peterson’s question, we start with a complex, but complete model and choose the variables that have the greatest effect.
If an Amateur Can Think of a Problem, So Can an Expert
During the heyday of Creationism, I watched a lot of videos and read a lot of articles debunking Creationists. I’ve seen their tactics. None of them are clever, but they can be quite effective if you already buy into what they’re selling. By far, the least clever trick is to just claim scientists haven’t even considered X, and therefore the entire argument is wrong.
This argument is deceptively simple, so let’s see an example where the argument holds up. If someone were to warn of the solar system tearing itself apart, but you found that they forgot to consider the gravitational pull of the sun, then their argument would fall apart. Likewise, if someone were to claim that they can’t understand how a rocket could fly to the moon because the fuel tank is too big without considering basic physics, then their argument would fall apart.
So what’s wrong with the argument? To understand, let’s look at a quote from a scientific paper about how you can determine how much each factor affects the weather.
If more realistic models … also have the property that a few of the eigenvalues of MM^T are much larger than the remaining, a study based upon a small ensemble of initial errors should … give a reasonable estimate of the growth rate of random error…. It would appear, then, that the best use could be made of computational time by choosing only a small number of error fields for superposition upon a particular initial state.
There’s a bit of science and math terminology being thrown around, but the basic idea is that each eigenvalue corresponds to some input to the system, and the size of the eigenvalue determines how much of an effect the input has on the system.
I’m not bringing this paper up because it’s the most technically advanced or even the most directly applicable to predicting the climate, as it focuses on the weather (though it’s not like they’re just going to forget to include the methods brought up in this paper in climate models). I’m bringing it up because I have a specific question I want to ask you: When do you think this paper was published? Write your answer down somewhere or text yourself or do whatever so you can’t change your answer. Got it? Good. I don’t want to have the answer where you can see it, so I’ll just link to the paper.
As you can imagine, we’ve made a lot of progress since that paper was published, which you can read about in the book Predictability of Weather and Climate mentioned in the footnotes. Jordan Peterson is decades behind the science, and he’s not alone. Creationists would bring up some problems with the science, like how radiocarbon dating would give inaccurate results for the ages of mollusks and other underwater creatures. When you look into the matter, you find that scientists have known about this problem for decades and have already addressed it (e.g. the reservoir effect affects how much carbon-14 is in certain animals, which affects the age you get from radiocarbon dating). Creationists who make claims that scientists haven’t considered X are either repeating a lie they heard or deliberately lying.
Given the number of people who incorrectly say “scientists haven’t considered X” when they have considered X decades ago, I want to propose a rule of thumb: If someone claims scientists haven’t considered something that an amateur would consider, then you can bet that scientists have considered it decades ago. To be clear, I’m not saying that it’s impossible for an amateur to come up with something the scientific community hasn’t or that the scientific community’s response must be correct. All I’m saying is that if I have thousands of people with years of training in fields with which most people have little to no experience and the explicit job to come up with accurate models, they’ve probably already considered whatever you’re considering.
Conclusion
I’m not posting this article with the intent of changing the minds of anyone I’ve mentioned in this article. Candace Owens said as much and Jordan Peterson got his talking points from Exxon. Instead, I wanted to use their ignorance to illustrate several points:
- Reporting data to the public is difficult.
- Units are important.
- If your claim doesn’t come with a way to disprove it, it is unscientific.
- The distinction between microscale, mesoscale, and macroscale models.
- All models are wrong, but some models are useful.
- If someone claims scientists haven’t considered something that an amateur would consider, then you can bet that scientists have considered it decades ago.
If you want to get a response from me, you’ll have to give me a clear standard of evidence or argument that I could provide to disprove whatever you’re saying, just as I have done in this article. With that being said, I don’t plan on writing articles like these frequently, as I’d much rather write about interesting things like Quantum Mechanics.
Shameful Self Promotion
Unlike most of my articles, I have neither submitted it to a publication nor sent it out to my email subscribers because I’m willing to bet that most of them subscribed for the Quantum Mechanics series, and I don’t want to throw current events in their face. Since I can’t rely on the normal boosts from the algorithm, I ask that you share this article with people who might be interested.
Footnotes
¹Palmer, T.N. “Predictability of weather and climate: from theory to practice.” Predictability of Weather and Climate. Cambridge, 2006. 1–29. It’s an entire book, but I’m only citing the first article.
²I might have even been one, but I’ve learned that making others feel stupid is stupid if you want to have friends. I’d much rather share knowledge than lord it over others.
³Coming up with questions like these is a bad way to prove you’re smarter than someone else because intelligence is less about the random facts you know and more about what you can do with your knowledge. Textbooks are filled to the brim with facts, but all they can do is lie around in the hope that someone uses them. Furthermore, everyone can come up with questions that nobody they know could answer. For a deeper dive, I recommend reading the book Mindwise: Why We Misunderstand What Others Think, Believe, Feel, and Want by Nicholas Epley. It can also help you to better understand others.
