How AI is Changing Trust and Safety
By Chirag Mahapatra
Chirag Mahapatra works at Trooly, a startup using machine learning to provide trust ratings based on data available from public sources such as websites, social media, and crime records.
Trust is fundamental to human interactions. It is necessary for trade, politics and social bonds. It is essential for the internet. In the last 20 years we have had three distinct explosions of internet activity, each with its own set of trust issues.
- From 2000 to 2005 we witnessed the growth of e-commerce in platforms like Ebay, Amazon and Paypal. These companies suffered from money laundering and fraud problems.
- From 2005 to 2010 there was a rapid growth of social networks like Facebook, Linkedin and Twitter. These companies faced problems in online abuse, hate speech, fake accounts and, more recently, fake content.
- Since 2010, there has been a growth of companies like Uber and Airbnb which promote peer-based interactions. These companies face some of the issues mentioned above along with new problems in guest vandalism, prostitution, sexual harassment and in cases of homicide.
The main purpose for online trust is safety. The danger from an online interaction grows as the interaction becomes more personal. One of the deadliest shooting attacks last year was by an Uber driver in Kalamazoo, Michigan. A list of incidents attributed to ride sharing companies is aggregated here.
Most of the leading companies have taken steps to tackle trust and safety concerns. These steps have ranged from setting policy such as protection insurance, requiring background checks and manual review of transactions. But the strongest of these tools is arguably machine learning because it is impossible to manually review all the transactions on any given platform.
As I see it, the rise of machine learning has led to three main changes. First, companies have become more proactive with respect to trust and safety. In the early stages of peer-to-peer economy, most companies only banned users after they committed a bad action. However, when the losses from fraud, damage and potential litigation started adding up, many companies have introduced stricter policies, banning users even for suspicious activities. For example, a company might ban all users with an IP address originating from Nigeria. This new level of vigilance can cause a lot of false positives or false alarms. In cases where there might be a risk to person or company, one can tune the algorithm to accept more false positives. In other cases where the risk is only a small financial loss (promotion abuse, etc.), companies might be willing to accept false negatives. Machine learning allows us to make the right tradeoff depending on the scenario.
Second, there has been an emergence of startups building industry-wide models in trust and safety space. Previously, companies relied only on their own data to make decisions. Now, they can supplement it with external data. Sift Science and Onfido are two examples of such startups. These companies have worked with a wide variety of clients in different industries and have been able to tune their own model to detect many patterns.
For example, when processing an email address startups providing trust solutions look at features such as first letter capitalization, presence of digits in the email, and use of a burner email domain. According to their research, the presence of the above features is 6x, 4x and 9x more likely to correspond to a fraudulent user. Sift Science provides solutions in payment fraud, account takeover and content, promotion and account abuse; Onfido provides solutions in identity verification and background checks. By using these external solutions companies leading P2P economy have been able to reduce friction for good users, reduce loss from fraud, and increase the conversion rate while, at the same time saving their fraud teams time and money.
Finally, the most positive impact of AI has been in the reduction of bias. Researchers have shown that humans make decisions based on cognitive, social and personal biases. For example, Airbnb guests have been known to discriminate against guests with “black sounding names”, Asians and even the disabled. These biases have even crept into our machine learning models.
Research by Microsoft Research shows that when you apply a neural network technique called word2vec on articles from Google news, it will encode gender based stereotypes like ‘father:doctor::mother:nurse’ and ‘man:computer programmer::woman:homemaker’. These biases tend to most impact minorities and people of color. However, once this bias has been identified, it is much easier to fix the AI, or de-bias the algorithm. Companies are now taking proactive steps to reduce bias and discrimination on their platforms. The most prominent example is that of Airbnb which has taken significant steps to promote inclusion.
Today there is a great deal of publicly available data in the form of web sites, social media, watch lists, crime lists, etc. However, there is an enormous challenge in extracting insights from this data. I have worked on using publicly available data to augment identity verification, abuse detection, background checks and employment screening solutions. This has been challenging because every platform is different and each have their own set of problems. I believe that, as our online interactions grow, we need to invest in a trust layer to reduce and, one day hopefully eliminate bad incidents. Machine learning will play a huge role in building this layer.
