There is weak research, improper research, and misleading research.

Then there is stupid research, which often incorporates all of the above.

We recently encountered what can charitably be described as stupid gun research of a variety where common sense appears to be a foreign concept to the authors.

JAMA That In Your Pipe And Smoke It

Yet again the Journal of the American Medical Association committed criminology malpractice. In this case, the corpse of intellectual reason was slayed by a paper titled Direct Exposure to Mass Shootings Among US Adults. ¹ We would normally just put a new entry on our Bad Research Roster page, which you should reference any time someone lobs a “study” at you. But this piece of work (or piece of something) was so deeply defective and comical, it calls for a blog entry to not only inform you, but teach how to detect industrial-grade buncombe.

The Hardheaded Headline

The paper claims that that 7% of surveyed adults had “been present on the scene where 4 or more people were shot.”

Seven percent is a big number. Asians and Native Americans combined make up a little more than 7% of the population. So, imagine every single person of those races being “present” at what JAMA classifies as a “mass shooting.” ²

JAMA’s reported survey was of adults, so let’s make the case even worse for the authors of this paper, who should seriously consider self-exile to the woodshed. In the USA, this would extrapolate to over 18 million adults having been present at a “mass shooting,” which is perfectly improbable. The state of New York has 19 million residents. So, the authors claim that nearly the equivalent of the population of the Empire State has been “present” at a mass shooting.

Honest Injure?

The paper also concludes  that 2% of the people surveyed (regardless of being at a mass shooting or not) report being injured at the event.

That is about 361,620 adults. That is roughly the population of Cleveland, Ohio. That’s a lot of bloodshed.

But it isn’t.

Foremost, the authors included all injuries at the “scene,” including “being trampled, or other causes of injury.” So, if you were at the “scene” of a mass shooting and  you sprained your ankle running away, you were added to the 2% figure and thus helped them make a headline few reporters would ever read deeply enough to debunk  on their own.

But it gets even worse, and even ob-scene.

This Paper Is An Ugly Crime Scene

When you read a headline about 7% of the population having “direct exposure” to a mass shooting, you think “direct” would mean being at the event itself.

Sorry, but believing that would disqualify you from publishing in the Journal of the American Medical Association.

Per the survey design, the authors classified being “present” at a mass shooting to include “or you could hear the gunfire.” We think it is not being nitpicky to classify hearing gunshots a block away as not “being present” at a mass shooting or having “direct exposure” to the event. The lack of physical presence is the giveaway.

Synthetic People

So where did the author find their survey respondents? They don’t completely disclose this.

The respondents were obtained via an online survey panel, which is an acceptable approach providing it is a public resource (it was) and large enough. Their approach, though, was synthetically suspicious.

Most well-designed surveys merely oversample the population at large, then weight the responses by demographics (age, race, etc.) to scale overall responses to match the population at large.

They didn’t do that. Instead, they created a “matched sample… based on selecting individuals to be interviewed [that] matched the target sample from a large pool of online opt-in panelists.” One might claim this is economy, an attempt to get rational results while polling fewer people. But the selection criteria are not completely, or even well described.

But self-selection exists, and in this paper, it is a bit of a sham. After synthetically modeling their matched sample, they sent survey invitations clearly telling possible (and possibly preselected) potential respondents that the survey was about “Exposure to Gun Violence in the United States.”

People who have been exposed to any type of gun violence are vastly more likely to take the bait than those who have not. This is self-selection on steroids. Yes, respondents  to a company doing random digit dialing, the time-tested mode of survey randomization, do deselect from surveys. But that is vastly different than people reacting to provocative text like “Exposure to Gun Violence.”

It  is worth noting the authors do not mention self-selection biasing in the section subtitled “Sampling, Quality Checks, Generalizability, and Non-Random Selection.”

The [Rotten] Core Problem

Those who work in the computer fields know well  the old adage “garbage in, garbage out.” Herein we have a “garbage in” problem.

In our Journalists Gun Issue Guide, we have a page on which data sources are reliable and which ones are questionable. In that latter category resides the Gun Violence Archive (GVA).

That this study cites GVA shows another case where the extrapolated numbers expose unbelievability.

In a strangely worded passage, the authors note that “Estimates from the Gun Violence Archive… led us to generate a sample of 10,000 individuals residing in the US.”  How this unreliable source generated a sample is not described in the paper or errata sheet.

To give you an idea of where this source may have polluted the author’s understanding, they themselves note that GVA estimates about 500 mass shootings a year in the US,  or about 1.4 a day. Keeping in mind that their own definition of “mass shooting” is 4+ people catching a bullet, the estimate is improbable.

Lifetime Tonnage

One factor to bear in mind is that the survey responses are “lifetime” recollections, not annualized data.

Is it possible that 18 million adults have been present at a “mass shooting” at some point in their lives? Unlikely.

U.S. life expectancy is 77 years. If we halve that (to use the midpoint of all ages) after subtracting those under age 18 (since the study specified “adults”), we have a 29 year average range for “a lifetime” of remembrances.

This roughly equates to 620,690 people (slightly more than all the folk in Memphis, Tennessee) a year experiencing a “mass shooting” in their lifetimes to get to that 18,000,000 adults mark.

Perspective Ignored

What the authors did was to sail past basic order-of-magnitude sanity-checking to ensure their writing and ’rithmetic wasn’t risible. When you are presented with scary headlines, ask what the extrapolated numbers are. If they are inconceivable, then the paper is ill-conceived.