We are in the third year of our “Documenting Hate” project, a partnership with more than 170 newsrooms. The project has included creating a database of reported potential hate crimes and bias incidents, from harassment to murder, and telling the stories of the people and communities affected by these acts. We’re also tracking media coverage of hate crimes around the country.

Last weekend, 22 people were killed in El Paso, Texas, by a young man the authorities suspect of harboring a violent hatred for immigrants and Latinos. A mass shooting in Gilroy, California, days earlier left three dead and is now being investigated as a possible act of domestic terrorism.

Thousands of hate crimes happen every year in the United States, in which people are targeted for the way they look, the religion they practice, the people they love and the languages they speak.

The scores of newsrooms involved in the “Documenting Hate” project have reported on these acts, and they will continue to. Here, then, we publish a kind of primer to the project.

What Is a Hate Crime?

The FBI defines a hate crime as a “criminal offense against a person or property motivated in whole or in part by an offender’s bias against a race, religion, disability, sexual orientation, ethnicity, gender, or gender identity.” That means that it is a crime — such as a robbery, assault or vandalism — that investigators determine was motivated entirely or partially by prejudice.

How Are Hate Crimes Defined by Law?

First, there are laws that allow for the prosecution of hate crimes.

There are numerous federal laws, including the Civil Rights Act of 1968, the Violent Crime Control and Law Enforcement Act of 1994 and the Matthew Shepard and James Byrd Jr. Hate Crimes Prevention Act of 2009. These laws allow the government to prosecute suspects for hate crimes and to increase penalties.

Many states have their own hate crime laws, and some of them cover additional categories, such as nationality or the homeless. Arkansas, Georgia, Indiana, South Carolina and Wyoming have no such state laws.

But even if states have laws on the books, it doesn’t mean they’re effective; for example, Utah’s bias crime statute was called “worthless” by the Salt Lake County district attorney. (Last week, the Legislature voted to beef up the law, and the governor is expected to sign the bill.)

And some states only cover some protected classes; for example, some states don’t include LGBTQ victims as a protected class in their hate crime statutes.

Then there are laws that define how to track hate crimes. The Hate Crime Statistics Act of 1990 established the collection of hate crime data by the Department of Justice. Some state hate crime laws require data collection, while others don’t. These crimes don’t need to have been prosecuted as hate crimes, they simply need to have evidence that they were motivated by bias.

How Many Hate Crimes Happen Every Year?

The short answer is: We don’t really know, because the data is so poor.

The hate crime data collection process hasn’t been entirely effective, in part because local police departments aren’t required to send their numbers to the federal government. As a result, some of them don’t send anything. Hawaii, for example, doesn’t report at all.

The FBI reported more than 7,100 hate crime incidents in 2017, the most recent year for which data is available. But the Bureau of Justice Statistics conducts a hate crimes survey and estimates there could be up to 250,000 hate crimes a year.

Even when police send numbers to the FBI, the data is sometimes questionable. Eighty-seven percent of police agencies that sent their data to the FBI in 2017 reported zero hate crimes. After reviewing more than 2,000 police reports, our partner BuzzFeed News found 15 hate-related assaults in 10 cities that reported no hate crimes to the FBI.

And in many states, local police send hate crime data to state agencies, who then submit the data to the FBI, but we found cases were this process did not function properly.

“The current statistics are a complete and utter joke,” Roy Austin, former deputy assistant attorney general in the Department of Justice’s civil rights division, told ProPublica.

Why Aren’t Police Doing a Better Job Tracking and Investigating Hate Crimes?

First, there are missing cases that police records don’t capture, because more than half of victims don’t report to police. That’s due to a variety of reasons, such as mistrust in police, a fear of not being taken seriously or uncertainty if they experienced a crime.

We know that some police departments don’t send any data to the FBI, or claim to have no hate crimes. Another issue is that some police departments don’t do a good job investigating and tracking hate crimes. We’ve found some victims who argue that the police don’t take hate crimes seriously enough, sometimes resisting taking a report or lacking the basic knowledge about what box to check on a police report. We found that some police departments marked crimes as anti-heterosexual when they were actually anti-gay or not even hate crimes. That means that sometimes, even if police are tracking hate crimes, the data they send to the FBI is flawed.

Another key problem is that police training on hate crimes varies very widely. Only a dozen states have statutes requiring police academies to provide hate crimes training, and even if recruits do get instruction, it’s sometimes for as little as a half an hour.

To improve hate crime tracking, “You have to have a combination of training, executive leadership, and some kind of infrastructure that is sustained and continuing,” Brian Levin, a hate crimes researcher, told ProPublica.

What About Prosecutions?

Hate crimes are notoriously difficult to prosecute, because attorneys must prove the defendant’s intent was based on bias. Adding hate crime charges can introduce layers of complexity to otherwise straightforward cases of assault or vandalism. A ProPublica investigation found that of the nearly 1,000 hate crime cases reported to police in Texas from 2010 to 2015, fewer than 10 were successfully prosecuted.

Some cities, such as Boston and New York, have dedicated bias crime units that provide expertise and added scrutiny to hate crime cases in order to build evidence for the prosecution. But in many places, it’s usually up to local police to investigate these crimes.

Sometimes, the FBI will send investigators in order to prosecute cases in federal court. But a News21 analysis found that just 100 hate crimes were prosecuted as federal crimes from January 2010 to July 2018.