{"id":402195,"date":"2024-06-29T16:20:56","date_gmt":"2024-06-29T16:20:56","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=402195"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=402195","title":{"rendered":"Crime, Race and Lethal Force in the USA \u2014 Part 2<\/span>"},"content":{"rendered":"
<\/div>\n
\n
\n
\n
\n
\"image\"<\/div>\n

In the previous part<\/a> of this article, I talked about the research background, goals, assumptions, source data, and used tools. Today, without further ado, let’s say together\u2026
<\/a> <\/p>\n

Chocks Away!
<\/h2>\n

We start by importing the required packages and defining the root folder where the source data sit:<\/p>\n

import pandas as pd, numpy as np  # root folder path (change for your own!) ROOT_FOLDER = r'c:\\_PROG_\\Projects\\us_crimes' <\/code><\/pre>\n
  <\/p>\n
Lethal Force Fatalities
  <\/h2>\n
  First let’s look into the use of lethal force data. Load the CSV into a new DataFrame:<\/p>\n
# FENC source CSV FENC_FILE = ROOT_FOLDER + '\\\\fatal_enc_db.csv'  # read to DataFrame df_fenc = pd.read_csv(FENC_FILE, sep=';', header=0, usecols=[\"Date (Year)\", \"Subject's race with imputations\", \"Cause of death\", \"Intentional Use of Force (Developing)\", \"Location of death (state)\"]) <\/code><\/pre>\n
  You will note that not all the fields are loaded, but only those we’ll need in the research: year, victim race (with imputations), cause of death (not used now but may come in useful later on), intentional use of force flag, and the state where the death took place.<\/p>\n
  It’s worth understanding what \u00absubject’s race with imputations\u00bb means. The fact is, the official \/ media sources that FENC uses to glean data don’t always report the victim’s race, resulting in data gaps. To compensate these gaps, the FENC community involves third-party experts who estimate the race by the other available data (with some degree of error). You can read more on this on the FENC website<\/a> or see notes in the original Excel spreadsheet<\/a> (sheet 2).<\/p>\n
  We’ll then give the columns handier titles and drop the rows with missing data:<\/p>\n
df_fenc.columns = ['Race', 'State', 'Cause', 'UOF', 'Year'] df_fenc.dropna(inplace=True) <\/code><\/pre>\n
  Now we hava to unify the race categories with those used in the crime and population datasets we are going to match with, since these datasets use somewhat different racial classifications. The FENC database, for one, singles out the Hispanic\/Latino ethnicity, as well as Asian\/Pacific Islanders and Middle Easterns. But in this research, we’re focusing on Blacks and Whites only. So we must make some aggregation \/ renaming:<\/p>\n
df_fenc = df_fenc.replace({'Race': {'European-American\/White': 'White',                            'African-American\/Black': 'Black',                            'Hispanic\/Latino': 'White', 'Native American\/Alaskan': 'American Indian',                           'Asian\/Pacific Islander': 'Asian', 'Middle Eastern': 'Asian',                           'NA': 'Unknown', 'Race unspecified': 'Unknown'}}, value=None) <\/code><\/pre>\n
  We are leaving only White (now including Hispanic\/Latino) and Black victims:<\/p>\n
df_fenc = df_fenc.loc[df_fenc['Race'].isin(['White', 'Black'])] <\/code><\/pre>\n
  What’s the purpose of the UOF (Use Of Force) field? For this research, we want to analyze only those cases when the police (or other law enforcement agencies) intentionally<\/i> used lethal force. We leave out cases when the death was the result of suicide (for example, when sieged by the police) or pursuit and crash in a vehicle. This constraint follows from two criteria: <\/p>\n
  1) the circumstances of deaths not directly resulting from use of force don’t normally allow of a transparent cause-and-effect link between the acts of the law enforcement officers and the ensuing death (one example could be when a man dies from a heart attack when held at gun-point by a police officer; another common example is when a suspect being arrested shoots him\/herself in the head); 
  2) it is only intentional use of force that counts in official statistics; thus, for instance, the future FBI database I mentioned in the previous part of the article will collect only such cases.<\/p>\n
  So to leave only intentional use of force cases:<\/p>\n
df_fenc = df_fenc.loc[df_fenc['UOF'].isin(['Deadly force', 'Intentional use of force'])] <\/code><\/pre>\n
  For convenience we’ll add the full state names. I made a separate CSV file<\/a> for that purpose, which we’re now merging with our data:<\/p>\n
df_state_names = pd.read_csv(ROOT_FOLDER + '\\\\us_states.csv', sep=';', header=0) df_fenc = df_fenc.merge(df_state_names, how='inner', left_on='State', right_on='state_abbr') <\/code><\/pre>\n
  Type df_fenc.head()<\/code> to peek at the resulting dataset:<\/p>\n
\n\n\n\n\n\n\n\n\n\n
<\/th>\nRace<\/th>\nState<\/th>\nCause<\/th>\nUOF<\/th>\nYear<\/th>\nstate_name<\/th>\nstate_abbr<\/th>\n<\/tr>\n<\/thead>\n
0<\/th>\nBlack<\/td>\nGA<\/td>\nGunshot<\/td>\nDeadly force<\/td>\n2000<\/td>\nGeorgia<\/td>\nGA<\/td>\n<\/tr>\n
1<\/th>\nBlack<\/td>\nGA<\/td>\nGunshot<\/td>\nDeadly force<\/td>\n2000<\/td>\nGeorgia<\/td>\nGA<\/td>\n<\/tr>\n
2<\/th>\nBlack<\/td>\nGA<\/td>\nGunshot<\/td>\nDeadly force<\/td>\n2000<\/td>\nGeorgia<\/td>\nGA<\/td>\n<\/tr>\n
3<\/th>\nBlack<\/td>\nGA<\/td>\nGunshot<\/td>\nDeadly force<\/td>\n2000<\/td>\nGeorgia<\/td>\nGA<\/td>\n<\/tr>\n
4<\/th>\nBlack<\/td>\nGA<\/td>\nGunshot<\/td>\nDeadly force<\/td>\n2000<\/td>\nGeorgia<\/td>\nGA<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
  Since we’re not going to investigate the individual cases, let’s aggregate the data by years and victim races:<\/p>\n
# group by year and race ds_fenc_agg = df_fenc.groupby(['Year', 'Race']).count()['Cause'] df_fenc_agg = ds_fenc_agg.unstack(level=1) # cast numericals to UINT16 to save memory df_fenc_agg = df_fenc_agg.astype('uint16') <\/code><\/pre>\n
  The resulting table is indexed by years (2000 \u2014 2020) and contains two columns: ‘White’ (number of white victims) and ‘Black’ (number of black victims). Let’s take a look at the corresponding plot:<\/p>\n
plt = df_fenc_agg.plot(xticks=df_fenc_agg.index, color=['olive', 'g']) plt.set_xticklabels(df_fenc_agg.index, rotation='vertical') plt.set_xlabel('') plt.set_ylabel('Number of police victims') plt <\/code><\/pre>\n
  <\/a><\/p>\n
  Intermediate conclusion:<\/b>  <\/p>\n
White police victims outnumber black victims in absolute figures.  <\/p><\/blockquote>\n
  The average difference factor between the two is about 2.4. It’s not a far guess that this is due to the difference between the population of the two races in the US. Well, let’s look at per capita values then.<\/p>\n
  Load the population data:<\/p>\n
# population CSV file (1991 - 2018 data points) POP_FILE = ROOT_FOLDER + '\\\\us_pop_1991-2018.csv' df_pop = pd.read_csv(POP_FILE, index_col=0, dtype='int64') <\/code><\/pre>\n
  Then merge the data with our dataset:<\/p>\n
# take only Black and White population for 2000 - 2018 df_pop = df_pop.loc[2000:2018, ['White_pop', 'Black_pop']]  # join dataframes and drop rows with missing values df_fenc_agg = df_fenc_agg.join(df_pop) df_fenc_agg.dropna(inplace=True)  # cast population numbers to integer type df_fenc_agg = df_fenc_agg.astype({'White_pop': 'uint32', 'Black_pop': 'uint32'}) <\/code><\/pre>\n
  OK. Finally, create two new columns with per capita (per million) values dividing the abosulte victim counts by the respective race population and multiplying by one million:<\/p>\n
df_fenc_agg['White_promln'] = df_fenc_agg['White'] * 1e6 \/ df_fenc_agg['White_pop'] df_fenc_agg['Black_promln'] = df_fenc_agg['Black'] * 1e6 \/ df_fenc_agg['Black_pop'] <\/code><\/pre>\n
  Let’s see what we get:<\/p>\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
<\/th>\nBlack<\/th>\nWhite<\/th>\nWhite_pop<\/th>\nBlack_pop<\/th>\nWhite_promln<\/th>\nBlack_promln<\/th>\n<\/tr>\n
Year<\/th>\n<\/th>\n<\/th>\n<\/th>\n<\/th>\n<\/th>\n<\/th>\n<\/tr>\n<\/thead>\n
2000<\/th>\n148<\/td>\n291<\/td>\n218756353<\/td>\n35410436<\/td>\n1.330247<\/td>\n4.179559<\/td>\n<\/tr>\n
2001<\/th>\n158<\/td>\n353<\/td>\n219843871<\/td>\n35758783<\/td>\n1.605685<\/td>\n4.418495<\/td>\n<\/tr>\n
2002<\/th>\n161<\/td>\n363<\/td>\n220931389<\/td>\n36107130<\/td>\n1.643044<\/td>\n4.458953<\/td>\n<\/tr>\n
2003<\/th>\n179<\/td>\n388<\/td>\n222018906<\/td>\n36455476<\/td>\n1.747599<\/td>\n4.910099<\/td>\n<\/tr>\n
2004<\/th>\n157<\/td>\n435<\/td>\n223106424<\/td>\n36803823<\/td>\n1.949742<\/td>\n4.265861<\/td>\n<\/tr>\n
2005<\/th>\n181<\/td>\n452<\/td>\n224193942<\/td>\n37152170<\/td>\n2.016112<\/td>\n4.871855<\/td>\n<\/tr>\n
2006<\/th>\n212<\/td>\n460<\/td>\n225281460<\/td>\n37500517<\/td>\n2.041890<\/td>\n5.653255<\/td>\n<\/tr>\n
2007<\/th>\n219<\/td>\n449<\/td>\n226368978<\/td>\n37848864<\/td>\n1.983487<\/td>\n5.786171<\/td>\n<\/tr>\n
2008<\/th>\n213<\/td>\n442<\/td>\n227456495<\/td>\n38197211<\/td>\n1.943229<\/td>\n5.576323<\/td>\n<\/tr>\n
2009<\/th>\n249<\/td>\n478<\/td>\n228544013<\/td>\n38545558<\/td>\n2.091501<\/td>\n6.459888<\/td>\n<\/tr>\n
2010<\/th>\n219<\/td>\n506<\/td>\n229397472<\/td>\n38874625<\/td>\n2.205778<\/td>\n5.633495<\/td>\n<\/tr>\n
2011<\/th>\n290<\/td>\n577<\/td>\n230838975<\/td>\n39189528<\/td>\n2.499578<\/td>\n7.399936<\/td>\n<\/tr>\n
2012<\/th>\n302<\/td>\n632<\/td>\n231992377<\/td>\n39623138<\/td>\n2.724227<\/td>\n7.621809<\/td>\n<\/tr>\n
2013<\/th>\n310<\/td>\n693<\/td>\n232969901<\/td>\n39919371<\/td>\n2.974633<\/td>\n7.765653<\/td>\n<\/tr>\n
2014<\/th>\n264<\/td>\n704<\/td>\n233963128<\/td>\n40379066<\/td>\n3.009021<\/td>\n6.538041<\/td>\n<\/tr>\n
2015<\/th>\n272<\/td>\n729<\/td>\n234940100<\/td>\n40695277<\/td>\n3.102919<\/td>\n6.683822<\/td>\n<\/tr>\n
2016<\/th>\n269<\/td>\n723<\/td>\n234644039<\/td>\n40893369<\/td>\n3.081263<\/td>\n6.578084<\/td>\n<\/tr>\n
2017<\/th>\n265<\/td>\n743<\/td>\n235507457<\/td>\n41393491<\/td>\n3.154889<\/td>\n6.401973<\/td>\n<\/tr>\n
2018<\/th>\n265<\/td>\n775<\/td>\n236173020<\/td>\n41617764<\/td>\n3.281493<\/td>\n6.367473<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
  The two rightmost columns now contain per million victim counts for both races. Time to visualize that:<\/p>\n
plt = df_fenc_agg.loc[:, ['White_promln', 'Black_promln']].plot(xticks=df_fenc_agg.index, color=['g', 'olive']) plt.set_xticklabels(df_fenc_agg.index, rotation='vertical') plt.set_xlabel('') plt.set_ylabel('Number of police victims\\nper 1 mln. within race') plt <\/code><\/pre>\n
  <\/a><\/p>\n
  We’ll also display the basic stats for this data by running:<\/p>\n
df_fenc_agg.loc[:, ['White_promln', 'Black_promln']].describe() <\/code><\/pre>\n
\n\n\n\n\n\n\n\n\n\n\n\n\n
<\/th>\nWhite_promln<\/th>\nBlack_promln<\/th>\n<\/tr>\n<\/thead>\n
count<\/th>\n19.000000<\/td>\n19.000000<\/td>\n<\/tr>\n
mean<\/b><\/th>\n2.336123<\/b><\/td>\n5.872145<\/b><\/td>\n<\/tr>\n
std<\/b><\/th>\n0.615133<\/b><\/td>\n1.133677<\/b><\/td>\n<\/tr>\n
min<\/th>\n1.330247<\/td>\n4.179559<\/td>\n<\/tr>\n
25%<\/th>\n1.946485<\/td>\n4.890977<\/td>\n<\/tr>\n
50%<\/th>\n2.091501<\/td>\n5.786171<\/td>\n<\/tr>\n
75%<\/th>\n2.991827<\/td>\n6.558062<\/td>\n<\/tr>\n
max<\/th>\n3.281493<\/td>\n7.765653<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
  Intermediate conclusions:<\/b>  <\/p>\n
\n
    \n
  1. Lethal force results on average in 5.9 per one million Black deaths and 2.3 per one million White deaths (Black victim count is 2.6 greater in unit values).<\/li>\n
  2. Data deviation (scatter) for Blacks is 1.8 higher than for Whites \u2014 you can see that the green curve representing White victims is considerably smoother.<\/li>\n
  3. Black victims peaked in 2013 at 7.7 per million; White victims peaked in 2018 at 3.3 per million.<\/li>\n
  4. White victims grow continuously from year to year (by 0.1 \u2014 0.2 per million on average), while Black victims rolled back to their 2009 level after a climax in 2011 \u2014 2013.<\/li>\n<\/ol>\n
      <\/p><\/blockquote>\n
      Thus, we can answer our first question<\/b>:<\/p>\n
       \u2014 Can one say the police kill Blacks more frequently than Whites?<\/i><\/b>
       \u2014 Yes, it is a correct inference. Blacks are 2.6 times more likely to meet death by the hands of law enforcement agencies than Whites.<\/b><\/p>\n
      Bearing in mind this inference, let’s go ahead and look at the crime data to see if (and how) they are related to lethal force fatalities and races.<\/p>\n
    Crime Data
      <\/h2>\n
      Let’s load our crime CSV:<\/p>\n
    CRIMES_FILE = ROOT_FOLDER + '\\\\culprits_victims.csv' df_crimes = pd.read_csv(CRIMES_FILE, sep=';', header=0,                          index_col=0, usecols=['Year', 'Offense', 'Offender\/Victim', 'White',                                         'White pro capita', 'Black', 'Black pro capita']) <\/code><\/pre>\n
      Again, as before, we’re using only the relevant fields: year, offense type, offender \/ victim classifier and offense counts for each race (absolute \u2014 ‘White’, ‘Black’ and per capita \u2014 ‘White pro capita’, ‘Black pro capita’).<\/p>\n
      Let’s look what we have here (with df_crimes.head()<\/code>):<\/p>\n
    \n\n\n\n\n\n\n\n\n\n\n
    <\/th>\nOffense<\/th>\nOffender\/Victim<\/th>\nBlack<\/th>\nWhite<\/th>\nBlack pro capita<\/th>\nWhite pro capita<\/th>\n<\/tr>\n
    Year<\/th>\n<\/th>\n<\/th>\n<\/th>\n<\/th>\n<\/th>\n<\/th>\n<\/tr>\n<\/thead>\n
    1991<\/th>\nAll Offenses<\/td>\nOffender<\/td>\n490<\/td>\n598<\/td>\n1.518188e-05<\/td>\n2.861673e-06<\/td>\n<\/tr>\n
    1991<\/th>\nAll Offenses<\/td>\nOffender<\/td>\n4<\/td>\n4<\/td>\n1.239337e-07<\/td>\n1.914160e-08<\/td>\n<\/tr>\n
    1991<\/th>\nAll Offenses<\/td>\nOffender<\/td>\n508<\/td>\n122<\/td>\n1.573958e-05<\/td>\n5.838195e-07<\/td>\n<\/tr>\n
    1991<\/th>\nAll Offenses<\/td>\nOffender<\/td>\n155<\/td>\n176<\/td>\n4.802432e-06<\/td>\n8.422314e-07<\/td>\n<\/tr>\n
    1991<\/th>\nAll Offenses<\/td>\nOffender<\/td>\n13<\/td>\n19<\/td>\n4.027846e-07<\/td>\n9.092270e-08<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
      We won’t need data on offense victims so far, so get rid of them:<\/p>\n
    # leave only offenders df_crimes1 = df_crimes.loc[df_crimes['Offender\/Victim'] == 'Offender'] # leave only 2000 - 2018 data years and remove redundant columns df_crimes1 = df_crimes1.loc[2000:2018, ['Offense', 'White', 'White pro capita', 'Black', 'Black pro capita']] <\/code><\/pre>\n
      Here’s the resulting dataset (1295 rows * 5 columns):<\/p>\n
    \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
    <\/th>\nOffense<\/th>\nWhite<\/th>\nWhite pro capita<\/th>\nBlack<\/th>\nBlack pro capita<\/th>\n<\/tr>\n
    Year<\/th>\n<\/th>\n<\/th>\n<\/th>\n<\/th>\n<\/th>\n<\/tr>\n<\/thead>\n
    2000<\/th>\nAll Offenses<\/td>\n679<\/td>\n0.000003<\/td>\n651<\/td>\n0.000018<\/td>\n<\/tr>\n
    2000<\/th>\nAll Offenses<\/td>\n11458<\/td>\n0.000052<\/td>\n30199<\/td>\n0.000853<\/td>\n<\/tr>\n
    2000<\/th>\nAll Offenses<\/td>\n4439<\/td>\n0.000020<\/td>\n3188<\/td>\n0.000090<\/td>\n<\/tr>\n
    2000<\/th>\nAll Offenses<\/td>\n10481<\/td>\n0.000048<\/td>\n5153<\/td>\n0.000146<\/td>\n<\/tr>\n
    2000<\/th>\nAll Offenses<\/td>\n746<\/td>\n0.000003<\/td>\n63<\/td>\n0.000002<\/td>\n<\/tr>\n
    …<\/th>\n…<\/td>\n…<\/td>\n…<\/td>\n…<\/td>\n…<\/td>\n<\/tr>\n
    2018<\/th>\nLarceny Theft Offenses<\/td>\n1961<\/td>\n0.000008<\/td>\n1669<\/td>\n0.000040<\/td>\n<\/tr>\n
    2018<\/th>\nLarceny Theft Offenses<\/td>\n48616<\/td>\n0.000206<\/td>\n30048<\/td>\n0.000722<\/td>\n<\/tr>\n
    2018<\/th>\nDrugs Narcotic Offenses<\/td>\n555974<\/td>\n0.002354<\/td>\n223398<\/td>\n0.005368<\/td>\n<\/tr>\n
    2018<\/th>\nDrugs Narcotic Offenses<\/td>\n305052<\/td>\n0.001292<\/td>\n63785<\/td>\n0.001533<\/td>\n<\/tr>\n
    2018<\/th>\nWeapon Law Violation<\/td>\n70034<\/td>\n0.000297<\/td>\n58353<\/td>\n0.001402<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
      Now we need to convert the per capita (per 1 person) values to per million values (in keeping with the unit data we use throughout the research). Just multiply the per capita columns by one million:<\/p>\n
    df_crimes1['White_promln'] = df_crimes1['White pro capita'] * 1e6 df_crimes1['Black_promln'] = df_crimes1['Black pro capita'] * 1e6 <\/code><\/pre>\n
      To see the whole picture \u2014 how crimes committed by Whites and Blacks are distributed across the offense types, let’s aggregate the absolute crime counts by years:<\/p>\n
    df_crimes_agg = df_crimes1.groupby(['Offense']).sum().loc[:, ['White', 'Black']] <\/code><\/pre>\n
    \n\n\n\n\n\n\n\n\n\n\n\n\n
    <\/th>\nWhite<\/th>\nBlack<\/th>\n<\/tr>\n
    Offense<\/th>\n<\/th>\n<\/th>\n<\/tr>\n<\/thead>\n
    All Offenses<\/th>\n44594795<\/td>\n22323144<\/td>\n<\/tr>\n
    Assault Offenses<\/th>\n12475830<\/td>\n7462272<\/td>\n<\/tr>\n
    Drugs Narcotic Offenses<\/th>\n9624596<\/td>\n3453140<\/td>\n<\/tr>\n
    Larceny Theft Offenses<\/th>\n9563917<\/td>\n4202235<\/td>\n<\/tr>\n
    Murder And Nonnegligent Manslaughter<\/th>\n28913<\/td>\n39617<\/td>\n<\/tr>\n
    Sex Offenses<\/th>\n833088<\/td>\n319366<\/td>\n<\/tr>\n
    Weapon Law Violation<\/th>\n829485<\/td>\n678861<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
      Or in a graph:<\/p>\n
    plt = df_crimes_agg.plot.barh(color=['g', 'olive']) plt.set_ylabel('') plt.set_xlabel('Number of offenses (sum for 2000-2018)') <\/code><\/pre>\n
      <\/a><\/p>\n
      We can observe here that:<\/p>\n