Gaggle Drops LGBTQ Keywords from Student Surveillance Tool Following Bias Concerns

Get stories like these delivered straight to your inbox. Sign up for The 74 Newsletter

Digital monitoring company Gaggle says it will no longer flag students who use words like “gay” and “lesbian” in school assignments and chat messages, a significant policy shift that follows accusations its software facilitated discrimination of LGBTQ teens in a quest to keep them safe.

A spokesperson for the company, which describes itself as supporting student safety and well-being, cited a societal shift toward greater acceptance of LGBTQ youth — rather than criticism of its product — as the impetus for the change as part of a “continuous evaluation and updating process.”

The company, which uses artificial intelligence and human content moderators to sift through billions of student communications each year, has long defended its use of LGBTQ-specific keywords to identify students who might hurt themselves or others. In arguing the targeted monitoring is necessary to save lives, executives have pointed to the prevalence of bullying against LGBTQ youth and data indicating they’re significantly more likely to consider suicide than their straight and cisgender classmates. 

But in practice, Gaggle’s critics argued, the keywords put LGBTQ students at a heightened risk of scrutiny by school officials and, on some occasions, the police. Nearly a third of LGBTQ students said they or someone they know experienced nonconsensual disclosure of their sexual orientation or gender identity — often called outing — as a result of digital activity monitoring, according to a national survey released in August by the nonprofit Center for Democracy and Technology. The survey encompassed the impacts of multiple monitoring companies who contract with school districts, such as GoGuardian, Gaggle, Securly and Bark. 

Gaggle’s decision to remove several LGBTQ-specific keywords, including “queer” and “bisexual,” from its dictionary of words that trigger alerts was first reported in a recent VICE News documentary. It follows extensive reporting by The 74 into the company’s business practices and sometimes negative effects on students who are caught in its surveillance dragnet. 

Though Gaggle’s software is generally limited to monitoring school-issued accounts, including those by Google and Microsoft, the company recently acknowledged it can scan through photos on students’ personal cell phones if they plug them into district laptops.

The keyword shift comes at a particularly perilous moment, as Republican lawmakers in multiple states push bills targeting LGBTQ youth. Legislation has looked to curtail classroom instruction about sexual orientation and gender identity, ban books and classroom curricula featuring LGBTQ themes and prohibit transgender students from receiving gender-affirming health care, participating in school athletics and using restroom facilities that match their gender identities. Such a hostile political climate and pandemic-era disruptions, a recent youth survey by The Trevor Project revealed, has contributed to an uptick in LGBTQ youth who have seriously considered suicide. 

The U.S. Education Department received 453 discrimination complaints involving students’ sexual orientation or gender identity last year, according to data provided to The 74 by its civil rights office. That’s a significant increase from previous years, including in 2021 when federal officials received 249 such complaints. The Trump administration took a less aggressive tack on civil rights enforcement and complaints dwindled. In 2018, the Education Department received just 57 complaints related to sexual orientation or gender identity discrimination.

The increase in discrimination allegations involving sexual orientation or gender identity are part of a record spike in civil rights complaints overall, according to data obtained by The New York Times. The total number of complaints for 2021-22 grew to 19,000, a historic high and more than double the previous year. 

In September, The 74 revealed that Gaggle had donated $25,000 to The Trevor Project, the nonprofit that released the recent youth survey and whose advocacy is focused on suicide prevention among LGBTQ youth. The arrangement was framed on Gaggle’s website as a collaboration to “improve mental health outcomes for LGBTQ young people.” 

The revelation was met with swift backlash on social media, with multiple Trevor Project supporters threatening to halt future donations. Within hours, the group announced it had returned the donation, acknowledging concerns about Gaggle “having a role in negatively impacting LGBTQ students.” 

The Trevor Project didn’t respond to requests for comment on Gaggle’s decision to pull certain LGBTQ-specific keywords from its systems. 

In a statement to The 74, Gaggle spokesperson Paget Hetherington said the company regularly modifies the keywords its software uses to trigger a human review of students’ digital communications. Certain LGBTQ-specific words, she said, are no longer relevant to the 24-year-old company’s efforts to protect students from abuse and were purged late last year.

“At points in time in the not-too-distant past, those words were weaponized by bullies to harass and target members of the LGBTQ+ community, so as part of an effective methodology to combat that discriminatory harassment and violence, those words were once effective tools to help identify dangerous situations,” Hetherington said. “Thankfully, over the past two decades, our society evolved and began a period of widespread acceptance, especially among the K-12 student population that Gaggle serves. With that evolution and acceptance, it has become increasingly rare to see those words used in the negative, harassing context they once were; hence, our decision to take these off our word/phrases list.”

Hetherington said Gaggle will continue to monitor students’ use of the words “faggot,” “lesbo,” and others that are “commonly used as slurs.” A previous review by The 74 found that Gaggle regularly flagged students for harmless speech, like profanity in fictional articles submitted to a school’s literary magazine, and students’ private journals. 

Anti-LGBTQ activists have used surveillance to target their opponents for generations, and privacy advocates warn that in the era of “Don’t Say Gay” laws and abortion bans, information gleaned from Gaggle and similar services could be weaponized against students.

Gaggle executives have minimized privacy concerns and claim the tool saved more than 1,400 lives last school year. That statistic hasn’t been independently verified and there’s a dearth of research to suggest digital monitoring is an effective school-safety tool. A recent survey found a majority of parents and teachers believe the benefits of student monitoring outweigh privacy concerns. The Vice News documentary included the perspective of a high school student who was flagged by Gaggle for writing a paper titled “Essay on the Reasons Why I Want to Kill Myself but Can’t/Didn’t.” Adults wouldn’t have known she was struggling without Gaggle, she said. 

“I do think that it’s helpful in some ways,” the student said, “but I also kind of think that it’s — I wouldn’t say an invasion of privacy — but if obviously something gets flagged and a person who it wasn’t intended for reads through that, I think that’s kind of uncomfortable.” 

Student surveillance critic Evan Greer, director of the nonprofit digital rights group Fight for the Future, said the tweaks to Gaggle’s keyword dictionary are unlikely to have a significant effect on LGBTQ teens and blasted the company’s stated justification for the move as being “out of touch” with the state of anti-LGBTQ harassment in schools. Meanwhile, Greer said that LGBTQ youth frequently refer to each other using “reclaimed slurs,” reappropriating words that are generally considered derogatory and remain in Gaggle’s dictionary. 

“This is just like lipstick on a pig — no offense to pigs — but I don’t see how this actually in any meaningful way mitigates the potential for this software to nonconsensually out LGBTQ students to administrators,” Greer said. “I don’t see how it prevents the software from being used to invade the privacy of students in a wide range of other circumstances.”

Gaggle and its competitors — including GoGuardian, Bark and Securly — have faced similar scrutiny in Washington. In April, Democratic Sens. Elizabeth Warren and Ed Markey argued in a report that the tools could be misused to discipline students and warned they could be used disproportionately against students of color and LGBTQ youth. 

In a letter to the lawmakers, Gaggle founder and CEO Jeff Patterson said the company cannot test the potential for bias in its system because the software flags student communications anonymously and the company has “no context or background on students,” including their race or sexual orientation. They also said their monitoring services are not meant to be used as a disciplinary tool. 

In the survey released last summer by the Center for Democracy and Technology, however, 78% of teachers reported that digital monitoring tools were used to discipline students. Black and Hispanic students reported being far more likely than white students to get into trouble because of online monitoring. 

In October, the White House cautioned school districts against the “continuous surveillance” of students if monitoring tools are likely to trample students’ rights. It also directed the Education Department to issue guidance to districts on the safe use of artificial intelligence. The guidance is expected to be released early this year.

As an increasing number of districts implement Gaggle for bullying prevention efforts, surveillance critic Greer said the company has failed to consider how adults can cause harm.

“There is now a very visible far-right movement attacking LGBTQ kids, and particularly trans kids and teenagers,” Greer said. “If anything, queer kids are more in the crosshairs today than they were a year ago or two years ago — and that’s why this surveillance is so dangerous.”

If you are in crisis, please call the National Suicide Prevention Lifeline at 1-800-273-TALK (8255), or contact the Crisis Text Line by texting TALK to 741741. For LGBTQ mental health support, contact The Trevor Project’s toll-free support line at 866-488-7386.

Get stories like these delivered straight to your inbox. Sign up for The 74 Newsletter