The Realities of Research Work: A semester reflection

This continues our series of student reflections and analysis authored by our research team.

The Realities of Research Work: A semester reflection

Stephanie Sorich

When I pictured my first time participating on a research team, I pictured something grandiose: Beautiful Mind-styled webs of ideas pasted around on the walls, philosophical discussion about the social consequences of our results, and a glamorous story to tell my friends and family at home. I ended up with something different.

Grunt work. Lots and lots of grunt work.

At least, that’s how it felt originally. Coming onto this project, the first few days were mostly just filled with confusion over the newness of everything. Combine this with the feeling of combing through source files for the very first time and the tedious feeling of trying to navigate the team’s drive. After week one, I was kicking myself for not taking ice skating instead.

There’s a difference between the research that you conduct in class and the research that is conducted by projects like this one. In class, the data collection process is quick and usually painless, and after a few weeks you’ve got a ten page paper to prove you did the work. I loved the feeling of accomplishment, and after three years in school, I began to dream about the actual impact I could have one day.

But the issue with projects of this size is that the work doesn’t last a few weeks. Long after I have graduated, this project will finally see its database completed. The odds of me being there to see the fruits of all of our labor is very small. What I am here for is the day in and day out of the project: coding and recoding, searching the internet for any source I can find more credible than a random local newspaper, and talking out the nitty gritty details. As a big picture thinker, this was a huge change of pace for me.

I know that the job of coding the cases is obviously essential to the project. But especially as someone who expected to waltz into the room and become the star of the research world, there’s a little something to be desired. However, as a senior, I know that this won’t be the last time I feel this way. As I move into my first job (which is hopefully in research as well) I know that I’ll be starting at the bottom of the totem pole. There’s a good chance that I’ll be in a supporting role for a long time before I see myself becoming the professional that I want to be one day.

You’ve got to start at the bottom to make it to the top. So here are a few pieces of advice from someone who just finished their first semester on a research project:

  1. Get used to having 20 tabs open at once. Find a case, check it with the spreadsheet, check it with another spreadsheet, file the case files, have I enticed you yet?
  2. Know that the way you see things isn’t always going to be the right way to see things. In my case, it rarely was. Take advantage of the bright minds around you and learn how to look at things from different angles.
  3. Don’t take offense at someone pointing out your mistakes. Especially at the beginning, being the best you can be means taking criticism. When you give criticism, you don’t make it personal, so don’t take it personal when you’re on the other side.
  4. ASK QUESTIONS. Being the person who think they’ve got it all down when they don’t is far more embarrassing than having the humility to admit that you need help. All of your teammates will also be grateful that they don’t have to go back and fix your mistakes.
  5. Make yourself useful. There will be days where, in the course of your normal work, you find mistakes. Don’t be the one who turns a blind eye, be the one who goes in and fixes it.

Now I’m obviously not speaking as an expert. My one semester is basically a minute in comparison to years that real professionals spend. But as the college equivalent of a grandfather in rocking chair, know that starting at the bottom is okay and it will eventually help you as you move up. I think that this experience has made me a better teammate, a better learner, and an overall better worker. Research isn’t always the drama associated with changing the world, sometimes it really is just the drama of disagreeing with your coding partner.

Remote Coding: a Future Idea for the Prosecution Project

This continues our series of student reflections and analysis authored by our research team.

Remote Coding: a Future Idea for the Prosecution Project

Meekael Hailu

As I have been with the Prosecution Project now for roughly 5 months, I feel like I have developed a great feel for the overall workings of the project itself, although I have always wanted to challenge myself to learn more. As we round out the end of the semester, my mind begins to wonder how far this project will move into the future, and what exactly we can do as a coding operation to excel its development.

As I began to think about the possible steps of growth the project can take, I thought of coding expansion. With a proper technique and educational foundation in how to properly code in this project, I fundamentally believe that this project can expand past just the institutional confines wherever Dr.Loadenthal is teaching at a particular time. With a fully-fleshed out application process that is selective, I feel as if that students at various institutions all across the country can join to be a part of the project, just as long as they meet the requirements to join.

This vision would require coding training, from start to finish, to be completely online. There could be some coding quizzes during the training process to verify that the instructions that are being given are being applied correctly. In addition to that if possible, the individual in the HR position could hold weekly or bi-weekly video calls with the newly brought-on coders to check in with and to help them throughout the process of being trained in how to properly contribute to the database.

In addition to that, for this future network that would be built remotely at various institutions across the country, there would be multiple assigned readings that would act as lecture replacements for the people who cannot physically attend a classroom-style instruction for the project. An example of the possible readings that could replace what we would learn in class. One example of this being utilized could be from the book titled Empirical Political Analysis.

This book breaks down the basics that will be needed for the coding process, especially for beginner coders that might be brought onto this project in the future. It states:

“Whether the subject of the research is elections, political advertisements, news accounts, organizations, or anything else, you must be aware of the importance of the selection process and its implications for the meaning and usefulness of your research” (Rich, 2018).

This would be a brilliant piece of literature to require for the onboarding process. This is an idea I feel is something that with the proper preparation can be done with flying colors. This project produces the more hands are involved in the process itself.

With this idea, obviously, there would be more stringent policies for misuse of the project’s information and database. This ties back into the necessary component of having meetings with the coders to develop some sort of accountability system for coders that are not being actively monitored in a traditional classroom sense. An incentive that can be offered for academically proficient students that we would want as a part of this project is academic credit at their respective institutions if they offered that for students.

Also, there can eventually be individuals at these various institutions who have created their own the Prosecution Project chapter, and there can be a designed hierarchy and specific criteria that must be met for individuals who are filling those positions. For example, if someone has just joined, they would be classified as a beginner coder and be subject to the guidelines that we apply to them, and there could also be new positions that are developed in tandem with the remote coder system to ensure that the coding process runs seamlessly from beginning to end. A position like the one just described could be called a “data comber” someone who combs through the data that is submitted to us remotely just to make sure all the stringent criteria that are applied to them is met.

The longer I am involved with tPP, the more excited I become with where it seems to end up going. I am happy I was able to be part of such an organization, and I know developments like these are only a certain amount of time away.


Rich, R. C. (2018). Empirical political analysis: quantitative and qualitative research methods. New York, NY: Routledge.

Reflections from a first-semester team member

This continues our series of student reflections and analysis authored by our research team.

Reflections from a first-semester team member

Emma Lovejoy

I came on board with tPP in August 2019, and I think the project is amazing.  The issues we are investigating, the information we’re accessing, and the things our data can be used for in the future, all really exciting things.  As a brand-new coder though, there have been challenges with developing confidence in my work.

The biggest hurdle has been cultivating any sense of ownership: I don’t have an academic background that’s very applicable, and picking up the mechanics of coding and information management didn’t come naturally to me.  The process of doing so didn’t leave a lot of time or space for getting a working understanding of the data we already have.

Even after months looking at our spreadsheets, I don’t know how clearly I could talk about the data as it looks today.  That disconnect, while minor in the abstract, in practice has caused my work to come slower than it could, since every decision I come across (from coding to excluding cases) feels like something I’m not qualified to do.  It’s imposter syndrome, in a word, but I think there are ways to help new members prevent and overcome it in years to come.

In our team’s closing discussions we debated a restructuring of the roles for future student participants as a possible the first step.  Having smaller student groups with distinct goals and responsibilities, such as finding cases, updating existing cases, coding from scratch, and even document management with no data entry, would allow people to self-select roles that compliment their skill sets.  I really think allowing specialization like this without first requiring a certain volume of coding would help students to approach the project with confidence, rather than having to ‘wait and see’ if they find something to do that they enjoy and are good at.  While there is tremendous value in having diverse skills and proficiencies, being able to gravitate toward things that are instinctively interesting might provide a more accessible starting place.

A consistent, perhaps self-paced training program has also come up in brainstorming for the future.  There’s definitely something to be said for everyone receiving the same orienting information when they come on board, and creating a digital program would allow more hands-on practice before actually touching a new case. I think it’s likely that extra practice at whatever task one’s focus will be on would greatly reduce unnecessary mistakes that come of second guessing, and to some degree increase productivity for the same reason: choices are faster and easier to make when you’re confident in your understanding.

I’m sure at this point someone has decided I’m just complaining, and that if I’d spent more time with the data or the manual or the codebook I wouldn’t have struggled with these things.  And that may well be true for people with a background in data science.  But, coming to it as a new skill to learn, and primed in a much more narrative style of research, it’s taken me longer to get accustomed to the simultaneous speed and specificity with which we need to address each case. I think the changes I’ve mentioned, as well as any number of ideas that have and will come up in the brainstorming process, will help to streamline the work of tPP as it continues to grow, and new people continue to come on board.

On the hardship of managing big data

This continues our series of student reflections and analysis authored by our research team.

On the hardship of managing big data

Megan Roques

As the semester is coming to an end, it is essential to sit back and review the work that has been done. One of the tasks the class has taken on has been reviewing the validity of all of the project’s case files. The cases are separated by year and each student signed up to review one year worth of cases.

Before starting this assignment, I believed that the time spent on this task could be used in adding more cases to the database or scraping documents to find more cases. However, I was wrong. I did not take into account the hardship of management of big data.

It would be impossible for one person to go through all the folders in a timely manner. Additionally, keeping the source files up to date should be a task that is done often. Sometimes, a document that may have been accessible 3 years ago may not be available now because of a company shutting down. This instance could happen when finding sources from non-governmental entities such as finding an article from a local newspaper’s website. Once, a document goes missing, it should be a priority to replace this document with another one that contains the same (or even more!) information.

Most of the document replacing process was self-explanatory such as checking to see if the case was actually in our records or adding more documents if there are not enough. One of the steps, however, peaked my interest: make sure there is an official document. This step was kind of shocking because I have previously coded a few cases where the only information available are from news sources or journal entries. The official information of the case may be difficult to obtain because the records may be sealed, the case may be a state case so the information is not easily available, or the case is very recent (happened in the past ~12 months) and processes like appeals or additional sentencing are still in process.

While going through my assigned folder, it was fascinating how despite the new information that gets released over time our codes remain the same. Even in cases where the official documents get added to our files such as the Randy Linn case, the case is still “correct” based on the guidelines set forth by the project. Some of the minor details that get updated are things like middle names or the case is added to a group case. You might be wondering why the members don’t wait for the official documents to be released in order to fully code a case. It may seem time-consuming and redundant to look for more files after a case is already in the database.

The way the information-gathering process is set up to deal with large amounts of data. This process is best described by scholars like Jensen (2018) as scraping information from other databases, governmental files (such as the FBI), and social media sites like Twitter. Then, the scraped information serves as our “supplementary data sources” and the hunting for official documents begins. These supplementary data sources are key to coding a case because news sources or social media websites may give one insight about the “why” of the offender which is left out of indictments and charging documents. Therefore, the news articles and journal entries written about an offender can be more meaningful to the project than the official documents. However, the gathering of the official documents is still an important task since it serves as confirmation for sentencing and charging details.

Throughout this activity I have learned the value of different information sources and how they each have their own impact on the project. It really helped me realize that often not all the information that you would like is available for one particular case. Therefore, it is important to monitor certain cases so when official documents or more details are released one is able to retrieve those details.


Jensen, Michael J. “Big Data: Methods for Collection and Analysis.” In Theory and Methods in Political Science, edited by Vivian Lowndes, David Marsh, and Gerry Stoker, 4th ed., 306-20. London, UK: Palgrave Macmillan UK, 2018.

Growth Means Change

This continues our series of student reflections and analysis authored by our research team.

Growth Means Change

Morgan Demboski

As the semester comes to a close, the project is now close to its fourth year. Since its birth, the Prosecution Project has evolved and improved through a lot of trial and error and collaborative brainstorming. On our last day of class for the Fall 2019 semester, the team split into small groups and discussed problems we found with the project, as well as came up with possible solutions to these problems. Being lucky enough to have been apart of this project since Fall 2017, I have witnessed how this project has grown, expanded, and changed for the better. However, one problem we continuously run into is keeping uniformity in the cases as the codebook, manual, and variables are altered. 

We run into new cases daily that make us reevaluate an aspect of our codebook. For example, until this semester, we had no ‘Threat/Harassment’ option underneath the ‘Tactic’ variable. We had the option of ‘Bomb threat/Hoax’ or ‘Armed intimidation/standoff;’ however, threatening a person/property for a social or political motive was not included in the tactics.

Furthermore, in the variable ‘Completion of Crime,’ we had the option of ‘Planned but not Attempted,’ ‘Attempted,’ ‘Carried through,’ and ‘Unknown,’ but we had no option in which a threat fell under. We were confronted with many cases that involved threats, such as William Patrick Syring from Arlington, VA who threatened employees of the Arab American Institute (AAI), and Rick Lynn Simmons from Grand Rapids, MI who left a racist, threatening voicemail to Democratic presidential candidate, Senator Cory Booker. As a result, we added the option ‘Threat/Harassment’ to the ‘Tactic’ variable and ‘Threat’ to the ‘Completion of Crime’ variable. 

This is one example of how our codebook and variables change constantly. However, when the codebook or something in the variables is altered, the cases that have already been completed are often left un-updated. Before we had the new variable, cases involving threats were always coded as ‘Other’ (recently changed to ‘Uncategorized’) in ‘Tactic,’ so when we added ‘Threat/Harassment’ as an option, these past cases needed to be changed. With the almost weekly modifications in coding language, variables, and formats, a lot of these older cases tend to slip through the cracks. To deal with this, my small group came up with an idea that we believe would help create more uniformity in coding past, present, and future cases. 

Currently, the project is organized into a class of students, a handful of independent coders, and a small group of senior team members. A large majority of the class, along with the independent coders, work on adding cases and creating new case starters, coding and expanding on uncompleted cases, and updating pending or completed cases. We have assignments throughout the semester that help us keep on track, with a total of around 30 case starters per student due by the end of the semester, around 30 completely coded cases per each student pair due by the end of the semester, and other assignments such as scraping source files or swapping out source documents. 

Though we have completed a lot of work this semester, our group suggests that it would be more efficient and effective to create different categories of coders in which students are organized into at the beginning of the semester. The first category is New Case Coders or coders who work solely on adding cases/case starters. They would focus on looking at Twitter and social media posts, the news, and other sources to find cases that have not yet been included in our database. They would then create a case starter, simply including the Date, Date Descriptor, Case ID, Name, and a Short Narrative. This role is great for members who have very little to no experience in coding with the project, and for members who prefer researching and finding new cases instead of deciphering specific variables.

We named the second category of coders Expanding Coders. Members in this category would focus on expanding those case starters and filling in the necessary variables. Working off of the source document and basic information that the New Case Coder found, the Expanding Coder will finish the case and prepare it to be checked and finalized by a senior coder. The third category, the category I believe to be most important at this stage in the project, is the Updating Coders. The members in this category will focus solely on updating previous and already completed cases with new information and new modifications to the codebook and manual. This would include updating pending cases with new information as well as updating finalized cases with newly found information or changes in variable language or codebook definitions.

The last category of team members, which already exists in the current structure, is the steering team, or senior members. This is a group of team members who have extensive experience with the project and have a deep understanding of the coding process. These members manage the more logistical side of the project, along with Project Director Dr. Michael Loadenthal, but they also have the important task of checking finished cases for mistakes and discrepancies and moving them to the completed (For Official Use Only) spreadsheet. 

After discussing our experiences with the project, each of my small group members had a preference of which category they would like to be a part of. We think that others in the project also prefer starting cases over updating and vice versa. By splitting people into their preferred categories, people can focus on a specific area rather than having to juggle both new cases and updating old cases. The first week of class (or training) can be spent teaching students what each role does and giving a small assignment that introduces the different roles so that students can decide their preferred job. New members to the team will generally be New Case Coders so that they can become familiar with the structure of the project and the coding process. Senior coders are typically designated before the year starts, and the rest of the team is welcome to choose whether they want to focus on adding new cases or updating old ones. By narrowing the duties of the members, each person can focus on perfecting their area and not worry about whether the other tasks need to be done. 

This is not to say that the members will work solely on new cases or old cases, there are a lot of tasks in the project still needing to be completed that are unrelated to coding entirely. As the project expands and more cases are added, our team has to reorganize and find ways to create uniformity among the thousands of cases in our database. Our small group’s suggestion was one among many offered by the team, each containing novel aspects that will grow and improve the project. We have excelled so far at making modifications to meet the fast-paced growth of the database, and we will continue to come up with ways to meet the changing needs of the team and the project as a whole. 

On the Topic of Efficiency 

This continues our series of student reflections and analysis authored by our research team.

On the Topic of Efficiency 

Emily Ashner

“Even Too Much Water Can be Lethal”- Edward Tenner

In a database filled with thousands of cases, and reaching far beyond in the amount of sources to back these up, a system of some sort of efficiency must be in place. The 39 page manual, 24 page codebook, an assembly line to quickly scrape documents, and other processes are ever-changing and updating to best organize and analyze data. As no project is error-free, especially one as new as tPP, this week’s work is being done to ensure each case has reputable sources and match with the designated formatting for files sources.

Processes such as this simply ensure that the project becomes more valid and consistent. However, it also points out room to further clarify the system to ensure efficiency. Taking quite a few hours to look through the individual cases of 2009, I found many cases lacking the requirements of sources outlined in the manual, likely never uploaded. This is a sort of expected problem- not necessarily detrimental to the project. Yet, it raises a few questions of different nature. 

What does this say about efficiency of technological systems? 

As we have seen the advancement of technology simply explode over the last few decades and journey into the fourth technological revolution, we have to question if there are underlying errors in the system- possibly analogous to our source document errors on the project.

As described by Jesus Mena, the use of neural networks is growing. These networks are another form of technological efficiency, like algorithms for example, that are rooted in a biological action. They reflect the way in which the brain organizes patterns and learning and use this idea to build a system that can organize data and make predictions. Mena describes this connection to the brain, seen below:

While I do not doubt the extensive research that went in to develop this system in conjunction with our understanding of human processes, problems arise as our knowledge expands in this area. Learning, as suggested, is rooted in synaptic changes- also known as plasticity. The brain is incredibly talented at accounting for deficits and adapting need be. Another way this is done is through modulation, which is shorter in time and allows for fast changes to a behavior done by one area. This is where I fear networks may not truly reflect this idea. As Mena suggests, using networks will help in organizing and creating predictive models of criminal feature, for example. Can the network truly change as society does- what if the system is multipart and can modulate? While organization seems to be exemplified, there comes a point where this type of efficiency may show error that will be hard to return from. 

Is efficiency always the best thing? 

As with everything else, there are critiques to systems incredibly efficient. As Edward Tenner describes in The Efficiency Paradox: What Big Data Can’t Do, “even an excess of water can be lethal.” Tenner describes many concerns of efficiency, but one of interest is the idea of hyperfocus. Hyperfocus has increased our skill to recognize large patterns and pictures, but cannot focus on small details. This is of concern to tPP and criminal proceedings. Is there a point where we generalize too much? An issue with prosecution is found in stereotypes, could this happen if begin to build systems that start to generalize patterns? 

Does tPP have appropriate error in process?

We must not forget one of the goals of this project is to provide students with skills and an opportunity for creative growth. Amongst error in  provides opportunity for innovation. Finding flaws in the process allows for students to really think about the project and how to build systems. In going through all of the cases from 2009 I was able to think about the process, but I also learned a great deal about a lot of cases I had never been exposed to prior. Priority should absolutely be based in producing results that are valid, but maybe a structure that is only 90% efficient is not a flaw at all. 

Works cited:

Mena, Jesus. Investigative Data Mining for Security and Criminal Detection. 1 edition. Boston, MA: Butterworth-Heinemann, 2002. [Excerpt, Chapter 6, “Neural Networks: Classifying Patterns”].

Tenner, Edward. The Efficiency Paradox: What Big Data Can’t Do. New York: Vintage Books, a division of Penguin Random House LLC, 2019.

Criminal gangs and Terrorism

In late November 2019, federal, state and local law enforcement led a series of arrests and seizures in Georgia. According to a typical news account:

A multi-agency investigation in north Georgia has led to several federal charges and 13 arrests across two states in connection with a major criminal organization…two search warrants were executed…led to arrests of people from both Georgia and Tennessee. Charges included Racketeering Influence Corrupt Organization (RICO) Act violations, violation of the Street Gang Terrorism and Prevention Act, firearms possession counts and numerous counts tied to illegal drugs.

Among the arrests were people that the ARDEO statement suggests were members of or associated with the Aryan Brotherhood, a criminal organization “that is involved in violence, firearms, and transportation of illegal narcotics.”

This case highlights an inclusion challenge for tPP. News accounts consistently refers to the Aryan Brotherhood as a CRIMINAL organization. However, the charge rhetorically labels  gang activity TERRORISM, and the Brotherhood is an organization that can produce organized political violence and has a socio-political agenda.

Do we include the case based on the group, since the Aryan Brotherhood has been known organizationally to support political violence? Do we include because state speech labels the crime as terroristic? Do we exclude because  the defendants are labeled as criminals and not social or political actors?
To make the question more complex, in recent weeks, President Trump has threatened to begin designating Mexican drugs cartels as foreign terrorist organizations (FTOs). According to the New York Times:

President Trump revived this idea [of narco-terrorism] in an interview last week with the former Fox News host Bill O’Reilly. “Are you going to designate those cartels in Mexico as terror groups and start hitting them with drones and things like that?” Mr. O’Reilly asked. “I don’t want to say what I’m going to do, but they will be designated,” Mr. Trump responded. “I have been working on that for the last 90 days.” He was referring to putting some of the cartels on the State Department’s list of foreign terrorist organizations.

If this were to occur, it would have major impacts on (not only tPP but also) US domestic policing policy, national security policy, and questions of legality and counter-terrorism measures.

Input and Output: Opportunities Afforded to tPP Team Members

This continues our series of student reflections and analysis authored by our research team.

Input and Output: Opportunities Afforded to tPP Team Members

Izzy Bielamowicz

Over the course of my time with tPP, I have been offered numerous opportunities by our executive director Dr. Michael Loadenthal and by the project itself. These opportunities have led to invaluable experiences for my professional career development. Beyond the direct impact my work with tPP has on my resume, the tangential exposure which I have garnered as a result of my involvement with the project is unequivocal. Of course, these opportunities are not handed out for free. With meticulous and quality work, I and many other tPP members have dedicated themselves to the project, leveraging a hard work ethic to further personal development. tPP is arguably the most profitable experience I have listed on my resume in terms of both personal and career development. tPP has taught me how to be an effective team member in pairs, small groups, and large groups; has led to infinite leadership opportunities; and has even given me the chance to submit my research on the dataset for publication.

In terms of teamwork, tPP has a complex hierarchy of members in which pairings and groupings evolve. At the most basic structural level, each semester has given me a new, consistent partner for coding purposes. Working one-on-one with another member of the team is difficult at times as disagreement often occurs during the coding process (specifically among contentious variables such as ideological affiliation, tactic, and completion of crime). These disagreements are unacceptable without resolution, so compromise is a necessity – an extremely crucial skill to possess, professionally.

At a more profound level, I have co-authored elaborate research papers with team members for the duration of entire semesters (about four months), learning the value of patience, negotiation, and tenacity. Furthermore, team work has permeated my time with the project in terms of planning for the team in a collaborative environment such as round-table meeting for the general body and the steering committee.

With the opportunity to be a member of the Steering Committee as the prime example, the leadership opportunities which I have been offered throughout my time with tPP are endless. As a member of the Steering Committee, I am able to see the project from a new perspective. Available to help guide and mentor new members, inclusion in important decision-making and project updates, and service as a reference are just a few of the many responsibilities associated with this position. Having this experience listed on my professional resume will open many doors for me, but over a year of loyalty and commitment to tPP granted me the opportunity to fill this roll. Action, reaction; hard work, payoff; input, output.

Other leadership opportunities bestowed onto tPP members include representing the project at conferences and other university sponsored events, and communicating with the project’s expert advisory board – building a professional network along the way. The most beneficial of all of my tPP experiences; however, are the publishing opportunities I have been awarded throughout my time with the project. Advantaging the connections I have through tPP – from Dr. Loadenthal to the expansive advisory board – I have been able to submit various works to multiple publishers, including leading journals on the subject of Critical Terrorism Studies. Receiving feedback from reviewers and editors has been a unique and insightful opportunity to improve my writing and prepare for a professional career. Having works reviewed by journals and being on a project which has produced publications is extremely impressive on a resume for undergraduates, and differentiates me and others working with the team from competition when applying and interviewing for potential job positions.

I am beyond grateful for the opportunities tPP has given me during my course of work on the project. From improving research and writing ability to affording unparalleled leadership opportunities, the weight this project has on my academic experience is insurmountable. I have gained a unique skillset as a result of my involvement with the project, and the magnitude this impact has had on my personal and professional life is uncontested. I will continue to give my best to tPP, and I rest assured that the project will continue to give back to me.