AI and Misinformation

AI and misinformation refers to the use of artificial intelligence to create or spread false or misleading content at scale, and to the tools societies build to detect and contain it. The concern is bidirectional: generative models lower the cost of producing convincing fakes, while the algorithms that distribute content can amplify whatever wins engagement — true or not.^[1]

Background

Automated content manipulation predates modern generative AI. Early deepfakes — face-swapping videos built with generative adversarial networks — emerged from online forums around 2017, and synthetic voice followed shortly after. What changed with large language and image models in 2022–2023 is scale and cost: a persuasive article, a plausible-looking document or a photorealistic image now takes seconds and cents, in any language, without specialist skill.^[2]

Two distinct problems sit under the umbrella. Misinformation is false content spread without deliberate intent to deceive, including ordinary users resharing AI-generated fabrications. Disinformation is deliberate — coordinated inauthentic campaigns, state influence operations, or scam operations using cloned voices and faces. AI helps both, but it helps disinformation most where volume and persona creation were previously the bottleneck: one operator can now run hundreds of seemingly distinct accounts.^[1][3]

Key mechanisms

• Synthetic text and images — fabricated news articles, false quotes, doctored photographs and "AI slop" designed purely for engagement.
• Voice and video cloning — realistic replicas of public figures and, increasingly, of private individuals, used in fraud and in political messaging.
• Bots and influence operations — LLM-powered accounts that comment, reply and reply-gate at human-like quality, making astroturfing cheaper.^[3]
• Algorithmic amplification — recommender systems optimising for engagement push sensational false content further than corrective content, an effect that predates AI but compounds it.

Detection approaches include statistical watermarking embedded at generation time, cryptographic content provenance such as C2PA content credentials, classifier-based detection of synthetic media, and network analysis that identifies coordinated behaviour rather than individual posts. No single method is decisive: watermarks can be stripped or degraded by edits, detectors lag behind new models, and provenance only helps where players in the chain honour it.^[4]

Impact

Researchers have linked large-scale exposure to AI-generated content with reduced trust in news and institutions generally — a "liar's dividend" in which genuine recordings are dismissed as fake. Documented harms include election-period synthetic media, financial scams using cloned executive voices, and non-consensual intimate imagery. At the same time, measurement studies find that, to date, the aggregate effect of AI-generated content on political belief formation has been smaller than early warnings suggested, with human sharing behaviour remaining the dominant driver of spread.^[1][2]

>Key Takeaways

• Generative AI cuts the cost of producing convincing false content to near zero, in any language and at scale.
• Disinformation (deliberate) and misinformation (unintentional) require different responses: enforcement against coordination, education against credulity.
• Detection is an arms race; watermarking, provenance and classifier detection each cover only part of the chain.
• Distribution algorithms, not the models alone, determine how far false content travels.
• Malaysia's response combines MCMC enforcement, PDPA data rights and platform-level content labelling.

See Also

• Deepfake
• AI and Elections
• AI Content Moderation
• AI Watermarking
• AI Safety
• MCMC Malaysia