Instagram is testing its anti-bullying tool, an AI that recognizes text often associated with bullying.
The photo and video sharing app borrowed the AI tool DeepText from Facebook for the task. In 2016, its primary job was to seek out spam. From 2017 until now, the team has been training the AI to find and block offensive comments, such as racial slurs.
Comments on Instagram are read by AI. And if it detects that the commenter is bullying the user, it will show a notice reading: “Are you sure you want to post this? Learn more.”
Once the commenter taps “Learn More,” they will be redirected to a notice that says: “We are asking people to rethink comments that seem similar to others that have been reported.”
Even though the commenter can ignore the message and proceed with its posting, tests results conducted by Instagram show that the notice encourages some people to undo the hurtful post. The team shares the impact of the users rethinking their actions is a significant result in terms of the bullying trend.
Instagram has been an essential tool in people’s social lives. Since June 2018, Instagram has 1 billion monthly active users. And with the integration of Facebook’s messaging system, the app has evolved into more than a photo and video sharing tool.
Instagram users do not just share photos and videos; they are having conversations through the comment section and direct messaging feature. People share a glimpse of their lives through Instagram Live Stories.
The app’s features enabled people to communicate efficiently, but it also made bullying accessible. Instagram caters to 80% of teens, and more than half of them have been bullied using the app.
Chief Executive Adam Mosseri discussed Instagram’s advocacy to tackle online bullying in the F8 Facebook Developers conference held in San Jose, California.
In an interview with Time Magazine, Mosseri emphasizes Instagram’s commitment to this advocacy. Using AI to address bullying may mean the loss of customers after some time.
“We will make decisions that mean people use Instagram less,” Mosseri says, “if it keeps people more safe.”
How did Instagram do it?
How did the engineers use DeepText to recognize bullying? Well, engineers started by building a training set. Moderators spent hours sorting through thousands of comments determining which statements are considered bullying. After this information has been fed to machine learning, the AI learns the patterns and detects the bullying comments.
One of the challenges in looking for the patterns in the text is the constant evolution of language. Slangs come up every day. So, it is always crucial for researchers to read and contextualize each post accurately.
However, the team did not stop training the AI in analyzing text. Name-calling has been the primary form of bullying, but with modern technology, bullying has taken in the way of photos and videos.
Altering a picture or posting embarrassing videos are one of the modern ways of bullying. The team scoured bullying photos and videos and saw patterns.
Patterns like up-skirt photos or photos with a split frame with a human on one side and an animal on the other emerged. Another is locating red X’s drawn across a person’s face.
But aside from name-calling and embarrassing photos and videos, hate pages are also a new form of bullying. Users set up fake accounts to pretend to be another person or degrade another person through hate campaigns.
The team has also set up to train DeepText to check the relationship between accounts to identify if there is bullying. For example, if a user often posts the same text on another user’s photos, it may be a red flag.
According to Yoav Shapira, Instagram’s lead engineer, their researchers write weekly reports on their findings on bullying interactions on the app, and most of the time, they are discovering new forms of bullying that they didn’t consider before.
Aside from the DeepText tool, Instagram is also working on another feature called Restrict. The feature enables users to filter and block abusive comments. If a user blocks another, the user will also not see any messages the blocked one sends. Commenters who are blocked will not learn that the user blocked them.
Currently, testing of the AI features will be limited to English-speaking users before implementing it globally. What’s clear is that Instagram is doing its best to lessen instances of bullying on its platform.