For an introduction to GenAI and a broader exploration of the risks and barriers around public sector adoption, check out our recent blog post on Generative AI in Government.
The development and deployment of GenAI models has rapidly gained traction in the tech landscape over the past year. The rise in investment for GenAI companies, as evidenced by the growth from $612.8 million in 2022 to a staggering $2.3 billion in funding in 2023, indicates that investors are confident in the technology’s transformative potential. Similarly, tech companies are making their own GenAI tools publicly available, such as Google's conversational AI, Bard, Adobe’s Generative Fill functionalities or OpenAI's image generator, DALL-E.
While GenAI’s emerging opportunities are worth exploring, it is essential for us to think beyond the hype and examine the potential adverse effects of this groundbreaking technology to determine how to best intervene - whether through product, policy, or other approaches to safeguard citizens online. This is particularly critical as since the release of OpenAI’s ChatGPT-3 in November 2022, GenAI has been made available to the general public, demanding careful consideration to ensure that individuals are employing it in a safe and responsible manner. How can we measure its impact on users' online experiences? More specifically, how can we ensure that the content produced by GenAI is both safe for its users and not misused to produce or perpetuate harm?
With the aim of deepening our shared understanding of GenAI’s impact on online safety, PUBLIC conducted extensive research during which we conducted desk research and reviewed 40+ literary sources from academia, government, industry and civil society. Here are our key findings
Through our research around risks associated with the use of GenAI, we observed that the misuse or misappropriation of this technology’s capabilities can impact a wide variety of online harms, ranging from mis-disinformation or fraud to intimate image-based abuse or Child Sexual Abuse Material (CSAM) production. The impact of GenAI on online harms is two-fold, with both malicious actors leveraging the technology to carry out online harms operations and users who may harm themselves through their interactions with GenAI.
While traditional AI is associated with risks such as bias in decision making, GenAI’s content generation abilities add new potential risks by producing material harmful in nature or used for malicious operations. Our research has identified a few characteristics of GenAI that, despite not being inherently harmful, can be leveraged to amplify or create harm.
GenAI tools can produce synthetic media, which is defined as content that has been generated or manipulated to appear to be based on reality, despite being artificial. Although synthetic media existed prior to the emergence of GenAI (i.e. manually manipulated pictures created using a photo editing software) GenAI enables the average citizen to create synthetic media more easily, and perhaps more convincingly. Currently GenAI is able to produce high-quality outputs that can convincingly fool humans - sometimes even AI detectors themselves - into thinking the AI-generated content is of human origin. For instance, research shows that humans cannot distinguish AI-generated tweets with tweets written by real Twitter users.
The ability to blur the distinction between AI-generated content and authentic content production can therefore be leveraged by malicious actors to intentionally deceive targeted individuals or communities. AI-powered voice-cloning technologies are already enabling fraudsters to impersonate relatives or professionals and carry out more convincing scamming operations. Through its ability to produce high-quality synthetic media, GenAI can enable malicious actors to produce harmful realistic material and further perpetuate harms towards populations online.
GenAI can be leveraged to automate production of AI generated content at a near-instantaneous speed, allowing users to complete tasks at a much faster rate and with increased efficiency. This ability can encourage malicious actors to propagate harm at a large scale and with minimum time input, therefore amplifying the spread and scope of online harms. Such capabilities are particularly harmful in scaling fraudulent operations, mis- and disinformation or harassment against targeted individuals.
Recently, research by NewsGuard has uncovered over 380 unreliable AI-generated news websites that publish AI-written news in a range of languages and themes. While some authors of these websites seemingly leveraged GenAI’s ability to automate content production at pace to generate revenue from programmatic advertising, others used automated AI-generation processes to knowingly propagate harm. The research also identified state-funded disinformation websites run by AI-text generators to back false claims. Through its ability to generate content fast and at scale, GenAI can enable malicious actors to increase the scope of their harm with limited time input, particularly for mis- and disinformation operations.
GenAI models enable malicious actors with limited technical expertise to engage in harmful online activities. First, GenAI powered conversational agents can enable more and more actors to educate themselves in how to better carry out online harm operations. Voicing this concern, Europol outlines that while information provided by such agents is already “freely available on the internet, the possibility to use the model to provide specific steps by asking contextual questions means it is significantly easier for malicious actors to understand and subsequently carry out various types of crime”. Secondly, GenAI lowers technical barriers to producing both high-quality and scaled synthetic content. From simple prompts, users can produce complex content such as voice cloning or deep fake imagery and scale their abuse operations, for example by using ChatGPT to generate fake social media content to promote a fraudulent investment offer.
Replicated versions of the now offline website DeepNude can generate synthetic non-consensual intimate imagery without requiring for its users to have any knowledge of coding or of altering images manually. In addition, in the US, the FBI has issued a public service announcement warning of the rise of reports from ‘victims, including minor children and non-consenting adults, whose photos or videos were altered into explicit content’ and linking it to recent AI-enabled technology advancements that lower accessibility barriers to producing harmful content. The low technical barrier to entry enables a wide range of malicious actors to leverage GenAI models to generate and spread types of online harms, especially in conducting identity fraud, scamming, synthetic intimate image-based abuse, and large-scale influence operations which would have otherwise required refined technical knowledge.
It is important to note that the above characteristics are a few select of many we identified during our research. We anticipate the landscape of GenAI’s potential misuse for online harm will continue to expand beyond the characteristics outlined above. As such, we will continue to reflect on these use cases and their prevalence overtime, and urge our readers to do so as well.
Even when using GenAI without ill-intent, vulnerabilities within this technology can produce harmful unintended outputs.
Due to risks in bias and hallucinations both users and developers must pay attention to the quality and reliability of AI-generated content before sharing it or using it to inform. Additionally, it is worth noting that users’ interactions with GenAI can sometimes be unclear, causing the AI to misunderstand queries and give inaccurate or unreliable answers. This means that alongside working towards a safe development and deployment of GenAI models, user awareness is key to enhance the accuracy and effectiveness of the generated outputs.
When we prompted ChatGPT to explain what online safety harms could arise with the use of GenAI, it responded with concerns around content generation, including “deepfake generation, enabling fabricated content that deceives and defames individuals…and phishing” as well as specific harm types, including “propaganda…cyberbullying”. Our research has found that this is just a small part of the picture.
At PUBLIC, we are building an evidence base around GenAI-enabled threats to ultimately inform decision-making and opportunities in targeted interventions for both policymakers and safety tech providers. Our research demonstrated how GenAI can act as a catalyst for online harms, showing the importance of acknowledging that this technology’s potential - both for helping society and for harming it - is still unfolding. We encourage interested stakeholders in the Trust and Safety ecosystem to engage with us in developing a coherent understanding of GenAI-enabled threats to help government, regulators, and industry stakeholders navigate this rapidly changing landscape in both positive usage and mitigating harms.
If you would like to learn more about this issue, or are interested in assessing GenAI risks to online safety, please reach out to us at email@example.com.
Insights from a roundtable between PUBLIC, UK Government, Ofcom, investors and industry experts on what’s next for the online safety tech sector.
Explore PUBLIC's pivotal role in shaping the Online Safety Act, supporting government agencies, and pioneering evidence-based online safety solutions, in this new interview with Dan Fitter.
In this blog, we share our views on the challenges to AADC compliance based on recent user research with online platforms and regulators.
The metaverse is more than just a tech buzzword, it will likely have an impact on the way we socialise, play, work, and learn.
This month we kicked off our brand new GovStart programme - taking on a cohort of 34 startups across a number of different sectors. In this blog post we delve into the solutions of our 7 Privacy, Security and Online Safety startups spanning areas such as Disinformation detection, AI auditing and identity verification.
As part of our advisory work in online safety, PUBLIC is delivering a range of projects to improve child safety online. This blog outlines our view of the landscape, its challenges and potential solutions, through a Safety Tech lens. Safety Tech is technology that has been designed to facilitate safer online experiences, and to protect users from harmful content, contact, or conduct.
Hearing from 3 leaders in the field: Sasha Havlicek (ISD), John-Orr Hanna (CRISP), and Lyric Jain (Logically), this blog offers a series of takeaways from the Defence Disrupted panel discussion, alongside PUBLIC’s own insights on trends and challenges from our current work with leading public sector organisations.
A Safer Future: Enhancing Citizens’ Trust in a Digital World
Nominet and PUBLIC partnered to research and tackle key illegal online harms faced by young people when using the internet. This blogpost provides a final update on the research, focusing on systemic opportunities, and outlines next steps for interested stakeholders.
The long-term societal impacts of false narratives propagated by foreign adversaries has yet to be completely understood and quantified, the development and anticipated launch of the Government’s Online Safety Bill in the UK provides an opportune moment to consider counter-disinformation interventions.