Bad Research Roster

List of Critiqued Papers

 

Costs of Fatal and Nonfatal Firearm Injuries in the U.S., 2019 and 2020

DATE: 2023
AUTHORS: Miller, Barnett, Florence, Harrison, Dahlberg, Mercy
PUBLICATION: American Journal or Preventative Medicine

OVERVIEW:

Attempts to calculate an annual “social cost” from gun injuries and fatalities.

CRITIQUE:

  • Includes assumed costs for both:
    • Quality of life lost (e.g., “pain and suffering”)
    • Estimates for the “value of a statistical life”, which is an assumed price society puts on the value of longer life outcomes.
  • “The bulk of the cost associated with firearm injuries does not come from the medical costs but rather from lost VSL and reduced quality of life.”

The Era of Progress on Gun Mortality: State Gun Regulations and Gun Deaths from 1991 to 2016

DATE: 2023
AUTHORS: Sharkeya, Kang
PUBLICATION: Epidemiology

OVERVIEW:

Time-series correlation test of the raw number of “restrictive” gun laws in states to total gun deaths.

CRITIQUE:

  • Amalgamated all modes of gun deaths (homicides, suicides, accidents)
  • Failed to incorporate 24 states passing habitual offender laws at the start of the analysis period
  • Used suicide measures to proxy gun ownership rates, then measured suicide rates (circular math)
  • Opened with very biased language (i.e., “The United States has a gun problem. The United States has more private guns per capita and higher levels of household gun ownership than any other developed country.”) indicating an “intent of findings”
  • Used political party affiliation of each state’s governor as a weighting factor
  • Used in-state gun manufacturing data, which given that some states have no significant manufacturing is a false proxy
  • Made the assumption that state governments would not add strict laws when large gun makers are resident, which is demonstrable incorrect given the number that have left states for that very reason (i.e., Smith & Wesson, Beretta, Remington) which means another false proxy
  • Failed to incorporate pre-period abnormally high gun violence rates in some states (i.e., California) in trend analysis, nor do they appear to population weight this (California, with abnormally high pre-1993 gun violence rates fell fast after passing a habitual offender law and their population makes the reductions artificially large)

NOTE: The specifically weighted California as +20 for restrictive states, and Kansas as -2. But in 2019 (pre-pandemic) those states had nearly identical gun homicide rates, 3.2 and 3.4 respectively.

Guns in the Home and Risk of a Violent Death in the Home: Findings from a National Study

DATE: November 2004
AUTHORS: Dahlberg, Ikeda, Kresnow
PUBLICATION: American Journal of Epidemiology

OVERVIEW:

Follow-back survey to evaluate if firearms in the home increases risk

CRITIQUE:

  • Failed to evaluate all people who (a) had firearms in the home but (b) died outside of the home (e.g., seniors in hospitals)
  • Only evaluated deaths
  • Failed to evaluate possible lives saved by defense gun use
  • Does not evaluate harm caused by non-firearm means (e.g., stabbings, etc.) which helps mask homes with criminal activities (i.e., occupied by gang members, etc.)

More Guns, More Unintended Consequences: The Effects of Right-To-Carry On Criminal Behavior And Policing In Us Cities

DATE: June 2022
AUTHORS: Donohue, Cai, Bondy, Cook
PUBLICATION: National Bureau of Economic Research

OVERVIEW:

Statistical modeling of a few cities to explore if changes in concealed carry laws affected crime.

CRITIQUE:

  • A small and seemingly cherry-picked sample of 47 cities, not a comprehensive 50 state or 3,143 counties. The list of cities if not provided in the paper.
  • “… our focus on major cities …” and then “… This paper is the first of which we are aware that provides credible causal estimates of the effects of RTC laws on gun theft or policing.” The two statements are oppositional and self-negating.
  • Major metro also induces the problem that metro cities are often in areas where obtaining a carry permit is difficult of impossible, even if it is technically “shall-issue” (e.g., Washington D.C.).
  • An atypical set of control variables not normally associated with criminology (e.g., percentage of the population made up of female headed households, the percentage of people living alone, etc.) and which cannot conceivably map to the probability of carrying firearms legally in public.
  • “We also removed all observations for a particular city-crime if that city-crime was missing more than 15 observations.” Inappropriate methodology. It negates entire classes of crime, and if even a few of the tiny number of cities being reviewed are involved, then it significantly skews estimates.
  • Authors used “estimated value” of stolen guns as a measurement value, though the “value” of a gun is crime is not measured by retail or street sales prices.
    • In the measurement period, crime “clearance rates” (cases closed or prosecuted) rose significantly in cities that had no change and fell in those that had a change

The Relation Between State Gun Laws and the Incidence and Severity of Mass Public Shootings in the United States, 1976–2018

DATE: October 2020
AUTHORS: Siegal, Goder-Reiser, Duwe, Rocque, Fox, Fridel
PUBLICATION: Law and Human Behavior

OVERVIEW:

A generalized estimating equations logistic regression to examine the relationship between eight state firearm laws and the likelihood of a mass public shooting

CRITIQUE:

  • “We sought to investigate mass public shootings, a small subset of all mass shootings. The sample size for analysis was therefore unavoidably small (N  155 events), resulting in fairly wide confidence intervals on many of our point estimates and making it difficult to conclude that laws we found to be unassociated with mass public shootings do not affect these events … we cannot infer causality from this study.”
  • Did not evaluate the two leading factors in MPS lethality, planning and “cattle pen scenarios”.
  • “Other factors associated with the occurrence of a mass public shooting were population, unemployment rate, divorce rate, firearm homicide rate, and suicide rate.”

Easiness of Legal Access to Concealed Firearm Permits and Homicide Rates in the United States

DATE: December 2017
AUTHORS: Siegel, Xuan, Ross, Galea, Kalesan, Fleegler, Goss
PUBLICATION: American Journal of Public Health

OVERVIEW:

Used a quasi-experimental panel design to measure raw homicide counts after passage of concealed carry legislation

CRITIQUE:

  • Measured changes in raw number of homicides, not homicide rates, the latter of which measures probabilities of harm (the former does not).
  • Started measurements the year after passage, ignoring what the developing situation might have been beforehand and which might have stimulated demand for the legislation (i.e., homicides were already rising).
  • Used log transformation for homicide counts, which is inappropriate for high-quality source data, which they used.
  • Restricted analysis to years 1991 through 2002, despite collecting data through 2015.
  • Quoting the paper, “these results should be interpreted with caution because of the possibility that they reflect a reverse association. That is, it is possible that the adoption of shall-issue concealed carry laws is associated with higher baseline homicide rates so that we are picking up not a causal effect of these laws on homicide but a systematic difference in baseline homicide rates between states that do or do not have these laws.”

Homicide Deaths Among Adult Cohabitants of Handgun Owners in California, 2004 to 2016

DATE: April  2022
AUTHORS: Studdert, Zhang,Holsinger, Prince
PUBLICATION: Annals of Internal Medicine

OVERVIEW:

Used multiple data sources in California (only) to model is cohabitating with a handgun owner is an increased risk.

CRITIQUE:

  • Such California cohabitants had an overall homicide rate of 3.1 / 100,000 (2019, per FBI) but California’s overall homicide rate was 4.3. Authors did not disclose this.
  • Excluded many people for inappropriate reasons (e.g., did not include people who had previously owned guns before cohabitation).
  • Did not test against localized violence rates, which is a key indicator of both the likelihood of homicides but also the primary motivation to acquire guns (e.g., gun buying in rough neighborhoods is a response to crime, and in-home firearm homicides were the largest measurement in this study).
  • Though documented in the study, the authors failed to note the astounding difference in firearm homicide counts between handgun-owning and non-handgun owning households.
Handgun Owning Households Non-handgun Households
Overall homicides 47 1,448
In-home firearm homicides 29 418
Away-from-home firearm homicides 12 929

Analysis of “Stand Your Ground” Self-defense Laws and Statewide Rates of Homicides and Firearm Homicides

DATE: February 21, 2022
AUTHORS: Esposti, Wiebe, Gasparrini
PUBLICATION: JAMA Netw Open

OVERVIEW:

Claims that states with “stand your ground” laws saw “8% to 11% national increase in monthly rates of homicide and firearm homicide”.

CRITIQUE:

  • Study was in the COIVID-19 pandemic year (“Data were analyzed from November 2019 to December 2020”) which is not time series and is also polluted by unusual circumstances.

Public Mass Shooters and Firearms: A Cross-National Study of 171 Countries

DATE: 2016
AUTHORS: Lankford
PUBLICATION: Violence and Victims, Volume 31, Number 2

OVERVIEW:

Claims that the United States has more mass public shootings (MPSs) that all other nations, 31% of global count.

CRITIQUE:

  • Used English only datasets from American law enforcement, one of which was only concerning MPSs in America.
  • Used “active shooter” datasets, which is different than MPSs.
  • Use a unique definition of a MPS (included only attacks by a single perpetrator).
  • Included “pre-internet” we searches going back to 1966, periods where media capture of events was non-existent.
  • Did not include searches in languages other than English.

Association of Changes in Missouri Firearm Laws With Adolescent and Young Adult Suicides by Firearms

DATE: November, 2020
AUTHORS: Bhatt, Cheng, Morris, Beyer, Chestnut, Steigerwalt, Metzner
PUBLICATION: Journal of the American Medical Association

OVERVIEW:

Claims changes in Missouri’s “permit to purchase” pistol and concealed carry laws led to an increase in suicide.

CRITIQUE:

  • Synthetic modeling to produce non existent suicide rates.
  • Assumption in the model were thin and not typical of criminology standards.
  • Used a aggregate of shall-issue, may-issue and no-issue states to model the effects of CCWs to suicides.
  • Uneven (7 and 3) year before/after snapshot.
  • Circular logic on home gun ownership and suicide rates (using suicide rates to measure gun ownership, the then model suicides).

Handgun Ownership and Suicide in California

DATE: June, 2020
AUTHORS: Wintemute, Studdert, Zhang, Swanson, Prince, Rodden, Holsinger, Spittal, Miller
PUBLICATION: New England Journal of Medicine

OVERVIEW:

Claims an “greatly elevated” association between handgun ownership and firearm suicide.

CRITIQUE:

  • Based on incomplete integrated data that at best indexed 61% of the target population.
  • Omitted gun ownership that preceded the study (i.e., just looked at guns bought during the study period).
  • Failed to compare age-based observations of both gun ownership and gun suicide.
  • Did not explore the well documented intersection of age/illness and suicide.
  • Despite geocoding data, failed to report on regional/cultural differences that contribute to suicide.
  • No investigation into suicide ideation before gun acquisition.
  • Incomplete and inappropriate control proxies.

State-level Estimates of Household Firearm Ownership

DATE: April, 2020
AUTHORS: Schell, Peterson, Vegetabile, Scherling, Smart, Morral
ORGANIZATION: Rand

OVERVIEW:

Claims declining household gun owner rates in the United States.

CRITIQUE:

  • Used existing low-ball baseline survey as the anchor for all analysis.
    • Baseline includes all households, including those of resident aliens, legal or otherwise.
    • Baseline survey is known by respondents to be a government project and thus may have false negative reporting.
    • Baseline recorded in three clustered years, so the baseline itself has no trending use and thus makes for a very weak comparison.
  • Uses proxies for validation that have serious data quality assumptions and possible defects. These include:
    • Hunting licenses (which fail metro/non-metro population density imbalances).
    • Magazine subscriptions covering Internet era of massively declining magazine subscription rates.
    • Background checks which do not account for preexisting supply of guns.
  • Uses one proxy (suicides) with an incomplete assessment of urban/rural/sex combinations which would skew numbers low.
  • Uses other gun ownership surveys for proxy validation, but omits surveys with much higher reported gun ownership rates (possible cherry-picked set of surveys).

Full review at http://www.gunfacts.info/blog/gun-ownership-randomness/.

Evidence concerning the regulation of firearms design, sale, and carrying on fatal mass shootings in the United States

DATE: February, 2020
AUTHORS: Webster, McCourt, Crifasi, Booty, Stuart
ORGANIZATION: John Hopkins University Bloomberg School of Public Health

OVERVIEW:

Claims handgun licencing and large capacity magazine law reduce mass shootings.

CRITIQUE:

  • Used incorrect statistical testing approach
  • Amalgamated mass public and private shootings
  • Controlled for gang and narcotics shootings, but there are known huge gaps therein
  • Use a debated proxy for gun ownership rates
  • Included the year 2017 as the last year studied, which was an unusual year for mass shootings
  • Their full report included include control variables of dubious (and undocumented) fragility
  • Cited a “retracted” study that in turn …
  • Caused this study to omit five states (Florida, Kansas, Kentucky, Montana, and Nebraska) from analysis

Criminologist Gary Kleck published a paper 1 reviewing this study which contained these (and other) critical observations:

  • “The authors stressed the findings based on the flawed methods (which incorrectly omitted year fixed effects), and deemphasized those based on the more correct methods.”
  • “The authors, however, never explain why or how [large capacity magazines] possession or use would affect either whether a prospective mass shooter would carry out a MS or the number of people killed.”
  • “Their main results indicated a significant negative association of [large capacity magazine] bans with the rate of [mass shooting] incidents, but no significant association with the number of [mass shooting] fatalities.”
  • “The authors’ sole justification for their extraordinarily poor choice of control variables was that these same irrelevant variables had been used in prior studies, all co-authored by Webster, and all of similarly primitive quality.”
  • “The authors chose to define the purchaser licensing law as one applying to handguns, even though they claimed that many [mass shootings] are committed using assault rifles.” [Editor’s note: the publicly available mass shooting database all show that rifles play a minority role in mass shootings]
  • “Many of the authors’ own findings regarding purchaser licensing show no significant association with [mass shootings]. Their estimates of the law’s effects are highly inconsistent across the different sets of methods reported as part of their sensitivity checks.”

Extreme Risk Protection Orders Intended to Prevent Mass Shootings

DATE: August, 2019
AUTHORS: Wintemute, Pear, Schleimer, Pallin, Sohl, Kravitz-Wirtz, Tomsich
ORGANIZATION: UC Davis Violence Prevention Research Program, University of California Firearm Violence Research Center

OVERVIEW:

Reviews a subset of California based “red-flag” gun confiscation episodes with an eye toward mass public shootings.

CRITIQUE:

  • Author discloses that “It is impossible to know whether violence would have occurred had GVROs not been issued, and we make no claim of a causal relationship.”
  • Examined 21 instances from 10 of California’s 58 counties. In this period there were 414 “gun violence restraining orders” (GVROs).
  • Made two baseless assumptions:
    • “Suspects … had exhibited behavior suggesting … an intent [to commit a mass shooting]
    • “[T]he subject had or would soon have access to firearms.” but were not actually in possession.
  • Of the 52 firearms seized, 26 came from just one suspect.

State Gun Laws and Pediatric Firearm-Related Mortality

DATE: August, 2019
AUTHORS: Goyal, Badolato, Patel, Iqbal, Parikh, McCarter
ORGANIZATION: Children’s National Health System, Washington, District of Columbia; Department of Pediatrics, School of Medicine and Health Sciences, The George Washington University, Washington, District of Columbia, Pediatrics, Rockville, Maryland

OVERVIEW:

Uses an arbitrary ranking of the strictness of state gun laws and included adults (age 18+) to reach generate invalid statements.

CRITIQUE:

  • Used the “Gun Law Scorecards” from the Brady Campaign to Prevent Gun Violence, a gun control advocacy group, as the measure of strictness for gun laws. This scorecard has routinely been critiqued for having no basis for its rankings, instead appearing to positively rank gunslaw based on perceived political desirability.
  • Authors used non-universal and non-child focused variables for analysis. In other words, the arbitrary list was augmented with invalid data pools. “Secondary exposure variables included individual laws previously associated with lower mortality rates in the total population of adults and children.” Their list of these secondary laws – universal background checks for firearm purchase, universal background checks for ammunition purchase, and identification requirement  for firearms (microstamping, ballistic fingerprinting – are of debatable value with conflicting assessments by criminologists.
  • The publication routinely mentions children, but examines firearm deaths and injuries to people upwards of 21 years in age. This, of course, includes the most active cross-section of active street gang members, ages 13-21.
    • To a certain degree, they admit to this oddity by saying “Although the intent of injury may differ across the pediatric age group, we chose to focus this study across the entire pediatric age spectrum …”
  • They excluded states with extremely low firearm death rates, which would likely include states with high gun ownership but low urbanality, and thus low gang participation rates.
  • Gun ownership data was via a YouGov online survey, which is of questionable robustness. “… it is possible that this estimate is inaccurate.”

Association Between Connecticut’s Permit-to-Purchase Handgun Law and Homicides

DATE: April, 2015
AUTHORS: Rudolph, Stuart, Vernick, Webster
ORGANIZATION: Center for Gun Policy and Research, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, Maryland

OVERVIEW:

Uses proxy data and synthetic analysis to create hypothetical crime data to test if Connecticut’s Permit-to-Purchase (PTP) law lowered homicides

Overall Homicides Rates - Connecticut, Surrounding States, U.S. 1989 - 2014CRITIQUE:

  • “Compare[ed] Connecticut’s homicide rates after the law change to rates we would have expected if the law had not been adopted, using the synthetic control method.”
  • “Use[ed] longitudinal data … to predict pre-law homicide trends in Connecticut.” (in other words, did not use actual data).
  • Only looked at changes after the law passed which masked the fact that homicide rates were already dropping.
  • Ignores a steep increase in Connecticut homicides three years before passage.
  • One of the control states was California, which two years prior had a homicide rate 40% above national average.

Criminal Use of Assault Weapons and High-Capacity Semiautomatic Firearms: an Updated Examination of Local and National Sources

DATE: June, 2018
AUTHORS: Koper, Johnson, Nichols, Ayers, Mullins
ORGANIZATION: Journal of Empirical Legal Studies

OVERVIEW:

Uses proxy data on a small sample of metropolitan areas to see if criminals use “assault weapons” or large capacity magazines (LCM)

CRITIQUE:

  1. “… criminal use of these weapons was approximated by examining and triangulating across several local and national data sources on guns used in different types of crimes.”
  2. Looked at only 10 cities, some seemingly cherry-picked.
  3. The cities selected had populations between 124,000 and 684,500. Major metro areas, where homicides are disproportionately high due to street gangs, are not being evaluated in this study.
  4. Cities in southern states are not represented, with the exception of Richmond Virginia. Given the dynamics of Florida, Texas and Arizona, this is a big omission.
  5. Includes guns confiscated by police but possibly not used in the actual commission of a crime.
  6. Inconsistent definition of “assault weapon”. “AWs were defined based on the weapons that have most commonly been identified as such based on the old federal ban, current state laws, and the recently proposed federal ban.”
  7. Irrational data control used for “assault weapons”. “… an additional ceiling estimate of AW use was calculated based on the prevalence of semiautomatic rifles.” This is problematic since (a) many AWs are not riles and (b) most rifles are not AWs.
  8. Only three of the cities studied, two in the northeast, had data on LCM.
  9. Arbitrarily defined “large capacity magazine” as one holding more than 10 rounds.
  10. The prevalence of LCMs was fictional. “use of LCM firearms was approximated based on recoveries of semiautomatics that are commonly manufactured and sold with LCMs.”

Right-to-Carry Laws and Violent Crime: A Comprehensive Assessment Using Panel Data and a State-Level Synthetic Control Analysis

DATE: April, 2019
AUTHORS: Donohue, Aneja, Weber
ORGANIZATION: Journal of Empirical Legal Studies

OVERVIEW:

Paper uses an odd selection of control variables along with synthetic modeling of data, based on select assumptions, to generate crime statistics that do not actually exist.

CRITIQUE:

  1. “Distill[ed] from an array of different panel data regressions.” This is a opaque way of saying the core data is not from a standard source, such as FBI Uniform Crime Statistics.
  2. “Using two major sets of explanatory variables.” These external variables “model” the panel data to further remove it from the actual raw data.
  3. “By comparing the actual crime pattern for [right to carry]-adopting states with the estimated synthetic controls …” In short, comparing real data to manufactured data, which was created based on assumptions and synthetic modeling.
  4. In the first analysis section, authors compared changes in violent crime rates from 1977-2014:
    1. This is a 37 year period that starts during an era of massively rising violent crime.
    2. The period also begins before states started adopting right to carry in 1988 (Florida being the first notable conversion state).
    3. In this section there was no control for the year of enactment for each state, and thus the before/after effects for states.
  5. In the modeled section of the study:
    1. Omission of very common control variables, such as population density (which discounts the effects of larger cities with inner-city crime), violent crime arrest rates, per capita income, and more.

State gun laws, gun ownership, and mass shootings in the US: cross sectional time series

DATE: January 28, 2019
AUTHORS: Reeping, Cerdá, Kalesan, Wiebe, Galea, Branas
ORGANIZATION: Columbia University, Langone School of Medicine, University of Pennsylvania, Boston University

OVERVIEW:

The paper uses a proxy for the relative degree of gun control in the sundry states. They also use an academic paper to estimate gun ownership rates in each state.

CRITIQUE:

  1. The documents used to measure the relative degree of gun control law are based on an arbitrary and linear scale.This is a critical flaw.
    1. The authors of that document said in email “The primary criteria in determining each state’s rating was how much (or little) a state regulated a traveler’s ability to carry a loaded firearm preferably a handgun) without a recognized license.”
    2. But otherwise, no weighting was given any law based on the degree of restrictiveness or the alleged efficacy of the law.
  2. The rate of gun ownership comes from a relatively old review of the available measures.
    1. In this paper, the author notes that “there are no known measures that are valid indicators of trends in gun levels, making credible longitudinal research on the subject impossible at present.”
  3. The authors used firearm suicide rates as their proxy for gun ownership, which runs the hazard of not controlling for cultural and mental health intervention variations between states (e.g., it has been shown that people who are isolated and live in cultures of self-reliance both have higher overall suicide rates and gun ownership).
    1. This is a critical issue as there is strong correlation between mental health and mass shooters.

What Do We Know About the Association Between Firearm Legislation and Firearm-Related Injuries

PUBLICATION: Epidemiologic Reviews
DATE: February, 2016
AUTHORS: Santaella-Tenorio, Cerdá, Villaveces, Galea
ORGANIZATION: Johns Hopkins Bloomberg School of Public Health

OVERVIEW:

This is a review of literature spanning 64 years (1950 through 2014). The papers were drawn from three databases (PubMed, Scopus and Web of Knowledge) and exclusions made using Guide to Community Preventive Services.

CRITIQUE:

  1. The databases come from the medical sciences field (PubMed), others that are heavily populated with non-criminology publications (Scopus) and social sciences (heavily on Web of Knowledge). The sources are light on criminology, though not exclusionary.
  2. The use of the Guide to Community Preventive Services is concerning. Their review processes “Use methods that address the specific needs of public health.”
    1. The “criteria for including and excluding studies” is not transparent. The selection table is references to a web site that obscures the data.
  3. Some routinely decried studies (Ayers/Donohue) are included as authoritative.
  4. Amalgamated U.S. and international papers, obscuring cultural differences.
  5. The authors admit to a number of problems with their review:
    1. Studies where simultaneous legislation (gun control plus “get tough on criminal” laws) obscure the cause of changes in violence.
    2. Limited confidence in studies due to design and execution.

Firearm Legislation and Firearm Mortality in the USA: a cross-sectional, state-level study

This study is famous for making the claim that nationalizing three gun control laws would reduce gun deaths by 90%.

PUBLICATION: The Lancet
DATE: March, 2016
AUTHORS: Kalesan, Mobily, Keiser, Fagan, Galea
ORGANIZATION: Various

OVERVIEW:

An epidemiology approach to criminology and the alleged cause/effect of firearm legislation. Highly critiqued by many. In fact, an accompanying commentary in the same Lancet edition listed several methodology exceptions.

CRITIQUE:

  1. Used epidemiology measures not used in criminology (specifically, incident rate ratios as opposed to straight crime rates).
  2. Cross-sectional study using only one external control variable (unemployment rates). Cross sectional studies can be valid is a large number potential confounding variables). A time series study base before and after enactment of laws would have been robust.
  3. They used one year, and some have accused them of cherry picking the year (2009 for laws, 2010 for deaths) since crime data well before and after that year was available.
  4. State-level gun ownership data was not transparent. It was gathered by YouGov, an “international internet-based market research firm, headquartered in the UK”. A scan of their site failed to show this poll, how it was conducted and thus there is no transparency on ownership rates.
  5. There were passages in the text that were transparently anti-gun in nature, often echoing word-by-word memes from con control organizations. “Unfettered sales from unlicensed dealers”, a non-existent category. “Firearm ownership rates are reported to have fallen” which we debunked and showed gun ownership rates depend on residency status.
  6. A wild claim that a national universal background check law would lower gun death rates by 56% despite there being only a 10% total difference in firearm homicides in 2010 between states with and without universal background checks.
  7. The basic claim, that 90% of gun deaths could be eliminated via background checks for guns and ammo and firearm identification, is absurd since over 60% of gun deaths are by suicide, and nearly all of those are done with legally owned guns.
  8. Calculation modeling was used as a predictor of gun violence, a poorly documented “firearm risk profile” for each state, “predicted” how nationalizing laws would affect gun deaths.

Survey Research and Self-Defense Gun Use: An Examination of Extreme Overestimates

PUBLICATION: Journal of Criminal Law & Criminology
DATE: Summer 1997
AUTHORS: David Hemenway
ORGANIZATION: Harvard

OVERVIEW:

This paper claims to refute the conclusions of Gary Kleck and Marc Gertz, whereby Kleck/Gertz concluded the now certified claim of 2.5 million defensive gun uses (DGUs) every year. Yet, Hemenway’s paper never makes a point. There is a lot of speculation, odd yet inappropriate examples, and outright inaccuracies. In the end though, no refute is actually made and no statistical proof is offered.

CRITIQUE:

Hemenway is listed as a Professor of Health Policy at the Harvard School. This is another instance of medical academics dabbling inexpertly into criminology. Indeed, Hemenway inaccurately used an epidemiology truth table in this paper to incorrectly explain survey response accuracy methods. The major problems with this “refute” include:

  1. Many wild assumptions and assertions, most made without proof points (“… its conclusions cannot be accepted as valid”, “likelihood of [many different things]”, “interviewers presumably knew”, “interviewers appear to have …”, etc.)
  2. Inaccurate methods. Of note was the use of a 4×4 truth table to suggest inaccuracies in a 1×2 survey matrix (the 4×4 table is quite appropriate for epidemiology when medical tests can and do produce both false positives and negatives, but not in disproving a set of survey yes/no responses).
  3. Several inaccurate quotes or summaries of Kleck’s work (e.g. Kleck determined guns prevented 400,000 murders when in fact it was 400,000 serious violent crimes).
  4. Some incomprehensibly odd comparisons. The worst one was comparing possible over-reporting of DGUs to over-reporting of UFO encounters, ignoring the basic difference between an actual, experienced event and one requiring speculative interpretation (“Using a gun in self-defense, like having contact with an alien …”).
  5. An odd reliance on the National Crime Victimization Survey (NCVS), whose measure of DGUs is a low-side outlier when contrasted to criminology and media DGU surveys (it would appear that Hemenway wishes to believe the NCVS is accurate, despite the well-known reporting bias induced via personal interviews [face-to-face] in the respondent’s home).
  6. Though Hemenway is responding to a specific Kleck paper, he does not mention how in Kleck’s book Targeting Guns that the 2.5 million number is replicated in several other criminology and media surveys, and how the NCVS tallies are not.
  7. Hemenway ignores that DGUs prevent victimization, and relies instead on only counting DGUs that occurred during/after victimization. In short, he used oranges to talk about apples.
  8. In one very odd passage, he confuses individual DGUs with household DGUs, apparently not understanding the design of the original survey, which was to measure individual DGUs.

Household Firearm Ownership and Rates of Suicide Across the 50 United States

PUBLICATION: Journal of TRAUMA
DATE: April 2007
AUTHORS: David Hemenway, Matthew Miller, Steven Lippmann, Deborah Azrael

OVERVIEW:

This paper claims that states with higher hoursehold firearm ownership rates have higher suicides rates, and that the association is “significant”.

CRITIQUE:

Foremost, the study is unbalanced. To get the numbers that compared 15 “high ownership” states with a mere six “low ownership” states. They did this to compare roughly the same gross number of people, but this leads to stacking the deck in terms of the type of state cultures.

The cultural aspect is important. Many of the “high ownership” states are very rural, low population density, and have strongly “independent” cultures – cultures where people make their own decisions, including life and death ones. The “high ownership” states included Wyoming, the Dakotas, Montana, Alaska, etc. It should be noted that some of these rural states (West Virginia being one) have unusual drug and alcohol abuse rates, which is a key determinant for suicides.

As important, of the mere six states in the “low ownership” set, the authors included Hawaii, which has a profoundly different culture and is unique by its geographic isolation.

Target on Trafficking – New York Crime Gun Analysis

PUBLICATION: Office of the Attorney General, State of New York
DATE: October 2016

Overview:

Using Bureau of Alcohol, Tobacco and Firearms (BATF) crime gun trace data, along with questionable ranking systems and other methodology mistakes, the report attempted to identify several states other than New York as the source for crime guns. The report blames the bulk of New York’s crime gun sources on the mythical Iron Pipeline.

CRITIQUE:

  • The report skips by the most obvious issue, which is that most New York State crime guns come from New York. According to the 2013 BATF “Firearms Recovered and Traced” report, New York was the source for 31% of all its crime guns, despite having very stringent gun control laws.
    • In fact, crime guns from all Iron Pipeline states were a mere 37% more than from New York alone.
    • Interestingly, the report claims that only 1/2 of crime guns come from out of state, which by proxy means just as many come from inside the state.
  • Aside: New York has a “universal” background check system, so it is surprising that the number of New York sourced crime guns is as high as it is, indicating that these laws do not work.
  • No attempt was made to pair guns from out of state with normal interstate relocation. In other words, many or most of these guns may have been legally acquired and brought into the state by people moving there.
  • The report uses an arbitrary scoring system to predict if the gun was purposefully trafficked. However …
    • No statistical justification was used when selecting the scoring ratios.
    • For one key measurement (date of retail), nearly half of the BATF trace record were not available, and another 12% of crime guns could not be linked to any state. In short, the report disposed of half the relevant data.
    • 80% of the score for an assumed trafficked gun is a short time to crime (time from retailing of the gun to recovery at a crime scene). This presents many problems:
      • There is no statistical justification for this being the proper weighting.
      • They cite an ancient study that covered only 27 cities as proof of this assumption, and that study did not conclusively show the that a short time to crime was proof of trafficking.
      • Only 10% of the weighting was given to “border crossing”, meaning the key element – interstate movement – was the lowest criteria.
      • In the report, they admit that 1/2 of the trace records lacked sufficient classification data, and those were not given a score. In other words, the report blindly dismissed 1/2 of the data.
  • Even though the report weighted time to crime heavily, the authors admit that a mere 19% of recovered crime guns fit their definition for a short import period. In other words, less than 1/5th of crime guns were allegedly trafficked.
  • The report admits that a small number of high density cities are responsible for nearly all traced crime guns, with NYC and the adjoining part of Long Island being the majority. But the report does not investigate why this very tight cluster is such a huge problem. If removed from the analysis, New York state’s numbers would likely equal national averages.
  • The authors discredit themselves by citing Michael Bloomberg’s Mayors Against Illegal Guns and the Law Center to Prevent Gun Violence activist organization as an authoritative source for defining “supplier” states.
    • The latter is most important. The report cites the LCAGV as providing an “objective measure of the strength of gun safety laws”. Yet the LCAGV does not cite their own scoring system in the scorecard for state gun laws. There’s is an arbitrary definition, and not at all “objective”.

School shootings during 2013–2015 in the USA

PUBLICATION: Injury Prevention
DATE: December 2016

Overview:

Claims that states with firearm purchase background laws have a lower rate of school shootings.

CRITIQUE:

  • Foremost, school shootings are pretty rare. The authors about 51 per year, and of course these include gang incidents. They may also have included incidents near (not on) campus.
  • The control variables (mental health expenditures, K-12 education expenditure) were unusual and not standard for typical criminology research (poverty rates, population density, etc.).
  • Used epidemiology statistics, not criminology statistics.
  • The authors note that “urbanicity” (the impact of living in urban areas) had a greater covarience to school shootings than negative association with background checks.

Handgun Legislation and Changes in Statewide Overall Suicide Rates

PUBLICATION: American Journal of Public Health Research
DATE: April 2017

Overview:

Claims that states with “universal” background checks and mandatory waiting periods have lower overall suicide rates.

CRITIQUE:

The authors performed a study, comparing all 50 states and the District of Columbia. They evaluated the change in suicide rates between 2013 and 2014 (one year). To quote their article, “… statewide suicide rate changes between 2013 and 2014 …”

  • Foremost, a one year period cannot predict time-series effects. This is a major defect.
  • Aligned with this is that most time series studies examine the before/after effects of a law. The authors admit “… To our knowledge, no states changed the status on any of these laws during 2013 and 2014.”
  • Though they tested with a significant number of variables, they omitted many known to be associated with suicide rates, such as economic situation changes (i.e., layoffs, bankruptcies, etc.), familial support and abandonment rates, mental health access, and more.
    • Quite oddly, they included “elevation above sea level” as a comparison variable.
    • Interestingly, the period under study was during recover from the Great Recession, a time of very uneven economic recovery. To not fully analyze the effect economic variable changes (or lack thereof) were having on suicide rates is serious.

Lead exposure at firing ranges—a review

PUBLICATION: Environmental Health
DATE: April, 2017
AUTHORS: Mark A. S. Laidlaw, Gabriel Filippelli, Howard Mielke, Brian Gulson, Andrew S. Ball

OVERVIEW:

Claims that blood lead levels (BLL) in employees at firing ranges is dangerous.

CRITIQUE:

  1. The BLL range for employees at United States firing ranges was between 16.7 and 30.3 µg/dL, which is within all regulatory safety ranges.
  2. Many of the high scores were for shooting instructors and their blood samples taken immediately after training (in other words, not a average daily BLL).
  3. The firing ranges examined included overseas where common building codes and material handling procedures vary.

Right-To-Carry Laws and Violent Crime: A Comprehensive Assessment Using Panel Data and a State-Level Synthetic Controls Analysis

PUBLICATION: Working paper
DATE: July 2017
AUTHORS: John J. Donohue, Abhay Aneja, and Kyle D. Weber

OVERVIEW:

Claims Right To carry (RTC) laws increase violent crime from 13-15%.

CRITIQUE:

  1. Uses mathematical modeling to “predict” what crime rates might have been without RTC.
  2. Uses limited pairing of non-RTC states (2-4 states) with study states.
  3. Control states often have no cultural, population or geographical similarity. For example, Texas was studies by comparing it with California, Nebraska and Wisconsin.
  4. Studied only aggregate violent crime, despite RTC being a public function and certain forms of violence are not generally public (i.e., rape).

In-State and Interstate Associations Between Gun Shows and Firearm Deaths and Injuries

PUBLICATION: Annals of Internal Medicine
DATE: October 24, 2017
AUTHORS: Matthay, Galin, Rudolph, Farkas, Wintemute, Ahern

OVERVIEW:

Claims Nevada gun shows lead to more firearm injuries and deaths in California.

CRITIQUE:

  1. Raw gun injury data (presented in the paper) shows no change in California gun injury/deaths.
  2. Even after seemingly inappropriate statistical “adjustments”, the change in rates of firearm death an injuries were tiny (rate ratio variances commonly of less than one percent).
  3. Use control variables (seasonality and at-risk populations without:
    1. Defining the criteria for these variables.
    2. Reporting their assumptions concerning these.
    3. Reporting the constants used, which had direct affects on their modeling.
  4. Only looked at if Nevada gun shows affected California gun injuries, but not the other way (California affecting Nevada).
  5. Does not verify that firearms acquired at the gun shows were used in homicides, suicides or accidents.
  6. Used “difference in differences” statistical modeling to simulate experimental data:
    1. Inappropriate for criminology review with “as occurred” data.
    2. Approach subject to a variety of biases.
    3. Lack of broad set of confounding variables used to test regressions.
  7. No controls for suspect injury demographics (alcoholics, gang members, mentally ill, etc.) who are more prone to firearm accidents and homicides.
  8. Did not appear to factor-in that Nevada gun shows disallow non-Nevada residents to exit guns shows with acquired arms, and that California FFLs are in attendance to facilitate California regulations on waiting periods, etc.

State Firearm laws and Interstate Firearm Deaths from Homicide and Suicide in the United States

PUBLICATION: JAMA Internal Medicine
DATE: March 12, 2018
AUTHORS: Kaufman, Morrison, Branas, Wiebe

OVERVIEW:

Claims that “stronger” firearm laws are associated with lower suicide rates.

CRITIQUE:

  1. Paper “received a waiver of review” for unexplained reasons.
  2. Regardless of methodology concerns, the variability in their calculated Incidence Rate Ratios (IRRs) was
  3. The variance in IRR for “firearm” events and “non-firearm” events was commonly small (2-6% range).
  4. Inexplicably, they could not account for “causal relationship between state policies and firearm deaths.”
  5. Used distance as a proxy for firearm migration despite stating explicitly that the FBI notes this to not be a barrier (guns travel quite far as people move from state to state).
  6. Excluded Washington D.C. for no “applicable state laws” though city laws are well known.
  7. “Calculated counts of firearm homicides” instead of using the standard FBI UCR homicide tables.
  8. They began with an assumption of what firearm laws are associated with firearm death or interstate movement of firearms. 2
  9. They created their own scoring system for six categories of laws, and assigned points without scaling criteria.
  10. In many instances, their validation of efficacy came from irrational covariance assumptions (e.g., gun availability causes suicides).
  11. Made an incorrect assumption about interstate proximity in illegal/legal firearm migration.
  12. They clustered counties by the number of laws in effect, not their alleged efficacy (e.g., states with three marginal laws would be better that states with one strong law).
  13. Used epidemiology modeling instead standard statistical processes used in criminology.
  14. They had a poorly explained substitution of a model from 2010 with one from 2012
  15. They tested three different models, but published only on one.

Notes:

  1. The Hazards of Data-dredging in Gun Control Research:
    Comment on Webster et al. (2020), Kleck, 2020 ↩
  2. We will note, in passing, that the roster of laws by state we provided to them by a gun control advocacy group who publishes their own rating system for laws, which begs questions of legitimacy. ↩