In training the next generation of scholars and practitioners, START offers its students a chance to publish their work on this blog.

Exploring the Impact of Executive Orders Using Analysis of Competing Hypotheses

Author

Sally Barth, Megan Rutter, Steve Hoodjer and Nolan Quinn

As student interns at START, we employ a number of analytical tools to fulfill our mission of providing data-driven research to public-sector clients, as well as the general public. One tool the Advanced Research Interns worked with this semester was the Analysis of Competing Hypotheses (ACH) methodology. The intelligence community uses ACH methodology when there are multiple potential explanations for adversary behavior or several possible outcomes from actions taken by an ally. It is particularly useful for controversial issues because it requires analysts to leave an “audit trail” of what they have considered and how they arrived at their hypothesis (or hypotheses), rather than jumping to a conclusion and finding support for it later.[1]  The end conclusion is rarely stated as definitive – analysts are often dealing with an uncertain future or opaque adversary decision making – but rather it is stated as which hypothesis or group of hypotheses are most supported and least undermined by the available evidence. This method requires the research team to collaborate and agree on each piece of evidence, along with their subsequent weighting and scoring against each possible hypothesis. This exercise pushed our team to discuss, debate, and work through many slight disagreements to help develop our collaborative research skills for the future and create our group ACH. Here, we walk you through how we completed this ACH on the “travel ban” executive orders and an explanation of our conclusions.

 

Analysis of Competing Hypotheses

 

How will the Executive Orders 13769 and 13780 – Protecting the Nation from Foreign Terrorist Entry - impact the number of jihadist plots against U.S. targets over the next four years?

 

Over the next four years…

 

H0 (A) - There will be no statistically significant change in the number of jihadist plots against U.S. targets at home.

H0 (B) - There will be no statistically significant change in the number of jihadist plots against U.S. targets abroad.

H1 - There will be a statistically significant increase in the number of jihadist plots against U.S. targets at home.

 

H2 - There will be a statistically significant increase in the number of jihadist plots against U.S. targets abroad.

 

H3 - There will be a statistically significant decrease in the number of jihadist plots against U.S. targets at home.

 

H4 - There will be a statistically significant decrease in the number of jihadist plots against U.S. targets abroad.

 

 

Confidence Rating

H0 (A) – no change v. domestic targets

H0 (B) – no change v. targets abroad

H1 – Increase v. domestic targets

H2 – Increase v. targets abroad

H3 – Decrease v. domestic targets

H4 – Decrease v. targets abroad

E1 – The majority of publicly known terror plots against U.S. domestic targets are from homegrown radicals (citizens or legal residents)[2]

High

+

N/A

+

N/A

-

N/A

E2 – Grassroots pro- IS twitter users have used the ban in support of extremist ideology[3]

High

-

-

+

+

--

--

E3 – Several former jihadists,

terrorism experts, and government officials say groups like AQ and IS could effectively use the travel ban in their recruitment efforts[4]

High

-

-

++

++

--

--

E4

On an official level, no proof that AQ and IS have used travel ban for recruitment purposes[5]

High

+

+

0

0

0

0

E5

There have not been successful fatal attacks by refugees, immigrants, or 2nd gen. from the 7 banned countries[6]

High

+

N/A

?

N/A

-

N/A

E6 – Ban prevents people  from ‘countries of concern’ to get into the US[7]

Medium

-

+

-

+

++

-

E7

The ban will likely undermine alliances with Muslim partners abroad, upon whom we rely for CT cooperation[8]

Medium

-

-

+

++

-

-

E8 – Loss of visa interviews will remove a key source of intelligence.[9]

Medium

-

-

+

+

-

-

E9 – Ban may reduce cooperation with law enforcement among domestic Muslim communities, a key source of information for domestic CT efforts.[10]

Medium

-

-

++

+

--

-

E10 – There have been some terrorist plots in the US by people from or descended from these 6 countries of concern.[11] 

High

-

N/A

-

N/A

++

N/A

 

 

H0

Home

H0 Abroad

Increase Home

Increase Abroad

Decrease Home

Decrease Abroad

Evidence Undermined

 

7

5

2

0

10

8

Evidence Supported

 

3

2

8

8

4

0

?

 

0

0

1

0

0

0

N/A

 

0

3

0

3

0

3

neutral

 

0

0

1

1

1

1

Net Score

 

-4

-3

6

8

-6

-8

 

 

Hypothesis Support Rating Key

 

?

Unknown

N/A

Not Applicable

--

Strongly Undermines

-

Undermines

0

Neutral

+

Supports

++

Strongly Supports

 

Introduction

The primary goal of this analysis was to answer the question: How will Executive Order 13769 impact the number of jihadist plots against U.S. targets over the next four years?  In the middle of doing the analysis, the Administration responded to some of the less defensible elements of 13769 by revoking it and replacing it with Executive Order 13780.  After reviewing our evidence, we concurred with both White House aides and legal challengers that the orders were intended to be so similar as to achieve the same “basic policy outcome” that for the purpose of threat analysis could be considered as one whole.[12] 

The evidence can come from a variety of sources.  To cast a broad net we used news reports, statistics, expert opinions, statements by relevant individuals, and social media postings to inform our analysis. We then rated our confidence in each piece of evidence based on the number of concurring sources and the reliability of those sources. Our analysis yielded H1 (an increase in domestic plots) and H2 (an increase in plots abroad) as both the least undermined and most supported hypotheses.

 

Hypotheses

This exercise began with six competing hypotheses pertaining to the question of whether the number of jihadist terror plots against American targets will increase or decrease over the next four years. For the purpose of this analysis we defined jihadist terrorism as an act of political violence against an asymmetrical adversary to induce fear inspired by the idea of an Islamic Ummah.[13] To avoid committing a Type I error, we included a null hypothesis of “no statistically significant change” for both attacks at home and abroad. 

Dividing our hypotheses geographically meant that they could not be mutually exclusive.  For example, the same piece of evidence could cause an increase in the number of plots against domestic targets, but a decrease in plots against targets abroad. Domestic targets refers to any target within the physical borders of the United States and targets abroad would include targets over which the US has jurisdiction, including embassies, consulates, or against US citizens. We also do not consider guerrilla attacks by non-state actors engaged in active combat against the American military, as those attacks are usually based on tactical, rather than ideological, considerations.[14] 

     

Evidence

To reduce the likelihood of using overlapping evidence that could skew our analysis, we decided to use ten pieces of evidence. In some cases, multiple sources were used for one piece of evidence. We attempted to use different types of evidence, such as statistics, statements made by jihadist groups and their supporters, expert opinions (including government officials), and changes in government policy following the issuance of the Executive Order.  It is standard practice when conducting an ACH to assign each piece of evidence a confidence rating as we have done here.  In the intelligence community or full-fledged academic paper, an ACH might consider hundreds of pieces of evidence ranging from a very high to very low confidence.  Our intent with this analysis was to produce something more thoughtful than the hot takes in the media but with a sense of immediacy to be topically relevant for policymakers.  Therefore we only considered evidence in which we felt at least a medium level of confidence.  We generally rated evidence as “high” if we could justify it statistically.  When the evidence was solid logically but based on what we might term an educated speculation, we rated it “medium.”  Individually, we each proposed a number of pieces of evidence that were collectively regarded as having low confidence.  In some cases – for instance, non-expert or weakly-supported opinions – these could be rolled in to another piece of evidence and collectively made for a high or medium confidence, for example in E3 where a variety of sources pointing to one conclusion are grouped as one piece of evidence. 

 

Weighing the Evidence

The preponderance of evidence weighs against H3 (decrease v. domestic targets) and H4 (decrease v. targets abroad). This relies on our conclusion that the EO will bolster the messaging used by jihadist groups in propaganda, helping them attract new members and sources of funding. This is strongly supported by both E2 (statements by pro-IS Twitter users) and E3 (opinions of experts, government officials, and former jihadists).  Further, we rated E4 (the lack of official statements from group leaders) as neutral. Although the EO has not been used as official propaganda at this point, related Western political statements have been in the past. While H3 is most strongly undermined, it also has two very strong pieces of evidence (E6 and E10) to support its conclusion. Somewhat on the contrary, we found no evidence to support H4.

We judged that there will likely be an increase in plots against U.S. targets abroad, as there was no evidence to undermine H2. We also found that the evidence strongly supports an increase in plots against domestic targets (H1), as we recognize that the majority of plots against these targets come primarily from individuals who are radicalized in the United States and jihadists from countries unaffected by the ban. Another important consideration was the likely erosion of trust of the government within domestic Muslim communities and reduced cooperation between these communities and law enforcement, which could both fuel domestic radicalization and hinder law enforcement’s ability to stop radicalized individuals before they are able to plot an attack. The only evidence contravening an increase against domestic targets was the fact that the ban makes it more difficult for certain nationals to enter, a group which DHS has labeled as coming from “countries of concern” but from which only a small number of plots have arisen. 

The two null hypotheses are moderately undermined. This is due primarily to the aforementioned evidence on motivational use of propaganda, decreased cooperation with counterterrorism efforts, and the nationalities of past terrorist plotters. While the null hypotheses are useful for comparison with the hypotheses referencing the same geographic space, the greater availability of sources pertaining to the EO’s impact on domestic terrorism undermines the ability to compare the null hypotheses to each other. Having more sources regarding domestic terrorism not only meant that there was more diagnostic evidence, but also that we could more accurately rate the confidence in this evidence.

 

Sensitivity of Evidence and Future Observation

We have ascribed different weights to each piece of evidence based on subjective considerations, recognizing that a change in these weights could drastically alter the analysis. For example, we recognize that two key pieces of evidence – E5 and E10 – lend support to contradictory hypotheses. While we chose to weigh these pieces of evidence equally, the weight one assigns to each could drastically influence the conclusion one draws. No one arriving from the six affected countries as a refugee or immigrant has ever killed an American in a terror attack on American soil, while persons from unaffected nations – Saudi Arabia, Pakistan, Afghanistan, etc. – have killed thousands. Citizens from the affected countries have, however, been involved in thwarted plots and non-fatal attacks. In statistical terms, the “N” is small, but it is not zero, which makes analysis difficult.

Another piece of evidence we debated internally was E3, what we might broadly categorize as “expert opinion.”  We knew that say weighing the opinions of, say ten scholars as ten separate pieces of evidence would skew the outcome beyond any use.  Yet the preponderance of expert opinion pointed to the pair of “increase” hypotheses - H1 and H2.  There was also the question of how to consider different types of expertise.  Should academic opinion be considered separately from retired intelligence professionals or from the statements of Islamists?  There are very different perspectives after all.  Concerned again about overweighting opinions pointing to the same conclusion, we compromised by grouping them as one piece of evidence but giving it a high confidence rating and judging it to “strongly support” H1 and H2.  While that solved one issue, it lead to another debate on how to consider dissenting expert opinion.  Given the creative nature of terrorist adversaries, our discipline is particularly sensitive to incorporating unorthodox ideas.  In our review of the literature, we recognized that pro-EO experts were basing their judgments primarily on E6 and E10.  To account for this, we increased the weight of E6 and E10 under H3 – decrease in plots against domestic to “strongly supports.” 

Given the complicated nature of terrorism, we are unable to isolate one policy as the sole cause of any change in the prevalence of terrorist activity. For example, while we are confident in the logic of our analysis, jihadist motivation is a complicated mixture of grievance, religious ideology, and personal/idiosyncratic factors. Further, those who are motivated to carry out an attack may be unable to do so. Therefore, while we believe our conclusions are relevant, we recognize that the EO’s potential impact may be either mitigated or exacerbated by other factors.

We presume that the likelihood of terrorist activity is a function of motivation and capability. The evidence we found that undermined the hypothesis of an increase in attacks at home worked through decreasing terrorists’ ability to enter the United States, rather than their motivation to carry out an attack. This leaves open the possibility that attacks planned by terrorists no longer able to enter the United States will be carried out against less secure targets abroad, which should be studied further.

Ultimately, our assessment suggests that the United States is likely to see an increase in plots against U.S. targets both domestically and abroad if the EO is successfully implemented.

           

 

[1] Heuer, Richards J.  1999.  Psychology of Intelligence Analysis.  Washington, DC:  Center for the Study of Intelligence.  95.

[2] Bergen, Peter; Ford, Albert; Sims, Alyssa; and Sterman, David.  2017.  “Terrorism in America After 9/11:  Part II Who are the Terrorists?”  New America Foundation.  Washington, DC.  https://www.newamerica.org/in-depth/terrorism-in-america/who-are-terrorists/.  Majority Staff.  2017.  “Terror Threat Snapshot:  February 2017.”  U.S. House of Representatives, Homeland Security Committee.  Washington, DC.  https://homeland.house.gov/wp-content/uploads/2017/02/Terror-Threat-Snapshot-February-2017.pdf.  Crenshaw, Martha; Dahl, Erik; and Wilson, Margaret.  2017.  “Jihadist Plots in the United States January 1993-February 2016:  Interim Findings.”  The Consortium for the Study of Terrorism and Responses to Terrorism (START).  College Park, MD.  http://www.start.umd.edu/pubs/START_FailedFoiled_JihadistPlotsInterimFindings_Infographic_Jan2017.pdf

 

[3] Kassierer, Alex.  2017.  “Jihadists React to President Trump’s Executive Order.”  Flashpoint Intel, January 31.  https://www.flashpoint-intel.com/blog-jihadists-react-executive-order/.  Warrick, Joby.  2017.  “Jihadist Groups Hail Trump’s Travel Ban as a Victory.”  Washington Post, January 29.  https://www.washingtonpost.com/world/national-security/jihadist-groups-hail-trumps-travel-ban-as-a-victory/2017/01/29/50908986-e66d-11e6-b82f-687d6e6a3e7c_story.html?utm_term=.14d1c1252b5f

[4] Byman, Daniel L. 2017.  “Why Trump’s Policies Will Increase Terrorism – and Why Trump Might Benefit as a Result.”  January 30.  Brookings Institute.  Washington, DC.  https://www.brookings.edu/blog/markaz/2017/01/30/why-trumps-policies-will-increase-terrorism-and-why-trump-might-benefit-as-a-result/.  Stern, Jessica.  2017.  “Trump’s Executive Order Will Make us Less Safe.”  Boston Globe, January 30.  https://www.bostonglobe.com/opinion/2017/01/30/trump-executive-order-will-make-less-safe/BPSK9KWyU3NwHtZg6l2jiK/story.html.  Mozes, N.  2017.  “Arab World Split Over President Trump’s Executive Order Suspending Entry of Citizens From Arab and Islamic Countries Into the U.S.”  February 7.  The Middle East Media Research Institute (MEMRI), Washington, DC.  https://www.memri.org/reports/arab-world-split-over-president-trumps-executive-order-suspending-entry-citizens-arab-and.  Miller, Greg and Missy Ryan.  2017.  “Officials Worry That U.S. Counterterrorism Defenses Will be Weakened by Trump Actions.”  The Atlantic, January 29.  Miller, Greg and Missy Ryan.  2017.  “Officials Worry That U.S. Counterterrorism Defenses Will be Weakened by Trump Actions.”  Washington Post, January 29.   https://www.washingtonpost.com/world/national-security/officials-worry-that-us-counterterrorism-defenses-will-be-weakened-by-trump-actions/2017/01/29/1f045074-e644-11e6-b82f-687d6e6a3e7c_story.html?utm_term=.9a783d4d0c95.  

[5] Miller, Greg.  2017.  “The Islamic State has been Oddly Quiet About Trump.”  Washington Post, January 28.  https://www.washingtonpost.com/news/worldviews/wp/2017/01/28/the-islamic-state-has-been-oddly-quiet-about-trump/?tid=a_inl&utm_term=.ff10363a125c.  Valverde, Miriam.  2017.  “No Proof ISIS Leaders Using Donald Trump’s Travel Ban for Recruitment.”  Politifact, February 7.  http://www.politifact.com/truth-o-meter/statements/2017/feb/07/seth-moulton/mostly-false-claim-isis-already-using-trumps-execu/.   

 

[6] Nowrasteh, Alex.  2017.  “Guide to Trump’s Executive Order to Limit Migration for ‘National Security’ Reasons.”  January 26.  CATO Institute, Washington, DC.  https://www.cato.org/blog/guide-trumps-executive-order-limit-migration-national-security-reasons.  Levenson, Eric.  2017.  “How Many Fatal Terror Attacks Have Refugees Carried out in the U.S.?  None.”  CNN, February 29.  http://www.cnn.com/2017/01/29/us/refugee-terrorism-trnd/.   

[7] Department of Homeland Security Press Office.  2016.  “DHS Announced Further Travel Restrictions for the Visa Waiver Program.”  United States Department of Homeland Security, February 18.  https://www.dhs.gov/news/2016/02/18/dhs-announces-further-travel-restrictions-visa-waiver-program.  Carafano, James Jay.  2017.  “Trump, Refugees, and the Truth.”  February 5.  Heritage Foundation, Washington, DC.  http://www.heritage.org/immigration/commentary/trump-refugees-and-the-truth.

[8] Kheel, Rebecca.  2017.  “Trump Travel Order Complicates ISIS Fight in Iraq.”  The Hill, February 1.  http://thehill.com/policy/defense/317244-trump-travel-order-complicates-isis-fight-in-iraq

[9] Toosi, Nahal.  2017.  “Trump’s Travel Ban Could Endanger U.S. Intelligence Gathering.”  Politico, February 4.  http://www.politico.com/story/2017/02/trump-visa-ban-intelligence-gathering-234616

[10] Bergen, Peter and Andrew Lebovich.  2011.  “Analysis:  1 in 5 Terror Cases Started with Tips From Muslims.”  CNN, March 10.  http://www.cnn.com/2011/POLITICS/03/09/bergen.king.hearing/.  Beutel, Alejandro.  2012.  “Data on Post-9/11 Terrorism in the United States.”  June 1.  Muslim Public Affairs Council, Washington, DC.  http://www.mpac.org/assets/docs/publications/MPAC-Post-911-Terrorism-Data.pdf.  Gottinger, Paul.  2017.  “American Muslims Stop More Terror Attacks then the NSA.”  ShadowProof, January 27.  https://shadowproof.com/2017/01/27/american-muslims-stop-terror-attacks-nsa/.    

[11] Smith, Lee.  2017.  “Trump’s Travel Ban Addressed Real Problems.”  February 13.  Hudson Institute, Washington, DC.  https://hudson.org/research/13352-trump-s-travel-ban-addressed-real-problems.  Narayan, Chandrika and Steve Visser.  2016.  “ISIS Wing Claims Responsibility for Minnesota Mall Attack.”  CNN, September 18.  http://www.cnn.com/2016/09/18/us/minnesota-mall-stabbing/.

[12] Burns, Alexander.  2017.  “Hawaii Sues to Block Trump Travel Ban; First Challenge to Order.”  New York Times, March 8.  https://www.nytimes.com/2017/03/08/us/trump-travel-ban-hawaii.html

[13] Bockstette, Carsten. 2008. Jihadist Terrorist Use of Strategic Communication Management Techniques.  Garmisch-Partenkirchen, Germany:  George C. Marshall Center. 8.

[14] For an in-depth discussion of the difficulty and importance of separating definitions of terrorism and guerrilla warfare based on context and target type, see:  Ganor, Boaz.  2010.  “Defining Terrorism – Is One Man’s Terrorist Another Man’s Freedom Fighter?”  January 1.  The International Institute for Counter-Terrorism, Herzliya, Israel.  https://www.ict.org.il/Article/1123/Defining-Terrorism-Is-One-Mans-Terrorist-Another-Mans-Freedom-Fighter