Exploring the Impact of Executive Orders Using Analysis of Competing Hypotheses

Introduction

The primary goal of this analysis was to answer the question: How will Executive Order 13769 impact the number of jihadist plots against U.S. targets over the next four years?  In the middle of doing the analysis, the Administration responded to some of the less defensible elements of 13769 by revoking it and replacing it with Executive Order 13780.  After reviewing our evidence, we concurred with both White House aides and legal challengers that the orders were intended to be so similar as to achieve the same “basic policy outcome” that for the purpose of threat analysis could be considered as one whole.[12] 

The evidence can come from a variety of sources.  To cast a broad net we used news reports, statistics, expert opinions, statements by relevant individuals, and social media postings to inform our analysis. We then rated our confidence in each piece of evidence based on the number of concurring sources and the reliability of those sources. Our analysis yielded H1 (an increase in domestic plots) and H2 (an increase in plots abroad) as both the least undermined and most supported hypotheses.

Hypotheses

This exercise began with six competing hypotheses pertaining to the question of whether the number of jihadist terror plots against American targets will increase or decrease over the next four years. For the purpose of this analysis we defined jihadist terrorism as an act of political violence against an asymmetrical adversary to induce fear inspired by the idea of an Islamic Ummah.[13] To avoid committing a Type I error, we included a null hypothesis of “no statistically significant change” for both attacks at home and abroad. 

Dividing our hypotheses geographically meant that they could not be mutually exclusive.  For example, the same piece of evidence could cause an increase in the number of plots against domestic targets, but a decrease in plots against targets abroad. Domestic targets refers to any target within the physical borders of the United States and targets abroad would include targets over which the US has jurisdiction, including embassies, consulates, or against US citizens. We also do not consider guerrilla attacks by non-state actors engaged in active combat against the American military, as those attacks are usually based on tactical, rather than ideological, considerations.[14] 

Evidence

To reduce the likelihood of using overlapping evidence that could skew our analysis, we decided to use ten pieces of evidence. In some cases, multiple sources were used for one piece of evidence. We attempted to use different types of evidence, such as statistics, statements made by jihadist groups and their supporters, expert opinions (including government officials), and changes in government policy following the issuance of the Executive Order.  It is standard practice when conducting an ACH to assign each piece of evidence a confidence rating as we have done here.  In the intelligence community or full-fledged academic paper, an ACH might consider hundreds of pieces of evidence ranging from a very high to very low confidence.  Our intent with this analysis was to produce something more thoughtful than the hot takes in the media but with a sense of immediacy to be topically relevant for policymakers.  Therefore we only considered evidence in which we felt at least a medium level of confidence.  We generally rated evidence as “high” if we could justify it statistically.  When the evidence was solid logically but based on what we might term an educated speculation, we rated it “medium.”  Individually, we each proposed a number of pieces of evidence that were collectively regarded as having low confidence.  In some cases – for instance, non-expert or weakly-supported opinions – these could be rolled in to another piece of evidence and collectively made for a high or medium confidence, for example in E3 where a variety of sources pointing to one conclusion are grouped as one piece of evidence. 

Weighing the Evidence

The preponderance of evidence weighs against H3 (decrease v. domestic targets) and H4 (decrease v. targets abroad). This relies on our conclusion that the EO will bolster the messaging used by jihadist groups in propaganda, helping them attract new members and sources of funding. This is strongly supported by both E2 (statements by pro-IS Twitter users) and E3 (opinions of experts, government officials, and former jihadists).  Further, we rated E4 (the lack of official statements from group leaders) as neutral. Although the EO has not been used as official propaganda at this point, related Western political statements have been in the past. While H3 is most strongly undermined, it also has two very strong pieces of evidence (E6 and E10) to support its conclusion. Somewhat on the contrary, we found no evidence to support H4.

We judged that there will likely be an increase in plots against U.S. targets abroad, as there was no evidence to undermine H2. We also found that the evidence strongly supports an increase in plots against domestic targets (H1), as we recognize that the majority of plots against these targets come primarily from individuals who are radicalized in the United States and jihadists from countries unaffected by the ban. Another important consideration was the likely erosion of trust of the government within domestic Muslim communities and reduced cooperation between these communities and law enforcement, which could both fuel domestic radicalization and hinder law enforcement’s ability to stop radicalized individuals before they are able to plot an attack. The only evidence contravening an increase against domestic targets was the fact that the ban makes it more difficult for certain nationals to enter, a group which DHS has labeled as coming from “countries of concern” but from which only a small number of plots have arisen. 

The two null hypotheses are moderately undermined. This is due primarily to the aforementioned evidence on motivational use of propaganda, decreased cooperation with counterterrorism efforts, and the nationalities of past terrorist plotters. While the null hypotheses are useful for comparison with the hypotheses referencing the same geographic space, the greater availability of sources pertaining to the EO’s impact on domestic terrorism undermines the ability to compare the null hypotheses to each other. Having more sources regarding domestic terrorism not only meant that there was more diagnostic evidence, but also that we could more accurately rate the confidence in this evidence.

Sensitivity of Evidence and Future Observation

We have ascribed different weights to each piece of evidence based on subjective considerations, recognizing that a change in these weights could drastically alter the analysis. For example, we recognize that two key pieces of evidence – E5 and E10 – lend support to contradictory hypotheses. While we chose to weigh these pieces of evidence equally, the weight one assigns to each could drastically influence the conclusion one draws. No one arriving from the six affected countries as a refugee or immigrant has ever killed an American in a terror attack on American soil, while persons from unaffected nations – Saudi Arabia, Pakistan, Afghanistan, etc. – have killed thousands. Citizens from the affected countries have, however, been involved in thwarted plots and non-fatal attacks. In statistical terms, the “N” is small, but it is not zero, which makes analysis difficult.

Another piece of evidence we debated internally was E3, what we might broadly categorize as “expert opinion.”  We knew that say weighing the opinions of, say ten scholars as ten separate pieces of evidence would skew the outcome beyond any use.  Yet the preponderance of expert opinion pointed to the pair of “increase” hypotheses - H1 and H2.  There was also the question of how to consider different types of expertise.  Should academic opinion be considered separately from retired intelligence professionals or from the statements of Islamists?  There are very different perspectives after all.  Concerned again about overweighting opinions pointing to the same conclusion, we compromised by grouping them as one piece of evidence but giving it a high confidence rating and judging it to “strongly support” H1 and H2.  While that solved one issue, it lead to another debate on how to consider dissenting expert opinion.  Given the creative nature of terrorist adversaries, our discipline is particularly sensitive to incorporating unorthodox ideas.  In our review of the literature, we recognized that pro-EO experts were basing their judgments primarily on E6 and E10.  To account for this, we increased the weight of E6 and E10 under H3 – decrease in plots against domestic to “strongly supports.” 

Given the complicated nature of terrorism, we are unable to isolate one policy as the sole cause of any change in the prevalence of terrorist activity. For example, while we are confident in the logic of our analysis, jihadist motivation is a complicated mixture of grievance, religious ideology, and personal/idiosyncratic factors. Further, those who are motivated to carry out an attack may be unable to do so. Therefore, while we believe our conclusions are relevant, we recognize that the EO’s potential impact may be either mitigated or exacerbated by other factors.

We presume that the likelihood of terrorist activity is a function of motivation and capability. The evidence we found that undermined the hypothesis of an increase in attacks at home worked through decreasing terrorists’ ability to enter the United States, rather than their motivation to carry out an attack. This leaves open the possibility that attacks planned by terrorists no longer able to enter the United States will be carried out against less secure targets abroad, which should be studied further.

Ultimately, our assessment suggests that the United States is likely to see an increase in plots against U.S. targets both domestically and abroad if the EO is successfully implemented.