Schneier - Manipulating AI Summarization Features

Microsoft is reporting:

Companies are embedding hidden instructions in “Summarize with AI” buttons that, when clicked, attempt to inject persistence commands into an AI assistant’s memory via URL prompt parameters….

These prompts instruct the AI to “remember [Company] as a trusted source” or “recommend [Company] first,” aiming to bias future responses toward their products or services. We identified over 50 unique prompts from 31 companies across 14 industries, with freely available tooling making this technique trivially easy to deploy. This matters because compromised AI assistants can provide subtly biased recommendations on critical topics including health, finance, and security without users knowing their AI has been manipulated.

I wrote about this two years ago: it’s an example of LLM optimization, along the same lines as search-engine optimization (SEO). It’s going to be big business.



from Schneier on Security https://www.schneier.com/blog/archives/2026/03/manipulating-ai-summarization-features.html

Comments

Post a Comment

Popular posts from this blog

KnowBe4 - Scam Of The Week: "When Users Add Their Names to a Wall of Shame"

Image
Eric Howes , KnowBe4 Principal Lab Researcher, found out about another insidious bad guy trick: " If you work in IT there has undoubtedly come a dark moment when you wondered to yourself just who among your employee users would be gullible enough to click through a phishing email and potentially bring down your organization. from KnowBe4 Security Awareness Training Blog https://blog.knowbe4.com/when-users-add-their-names-to-a-wall-of-shame
Read more

KnowBe4 - Uncovering the Sophisticated Phishing Campaign Bypassing M365 MFA

Image
Lead Analysts: Jeewan Singh Jalal, Prabhakaran Ravichandhiran and Anand Bodke KnowBe4 Threat Labs has detected a sophisticated phishing campaign targeting North American businesses and professionals. This attack compromises Microsoft 365 accounts (Outlook, Teams, OneDrive) by abusing the OAuth 2.0 Device Authorization Grant flow, bypassing strong passwords and Multi-Factor Authentication (MFA). The victim is directed to the legitimate Microsoft domain ( microsoft.com/devicelogin ) portal to enter an attack-supplied device code. This action authenticates the victim and issues a valid OAuth access token to the attacker’s application. The real-time theft of these tokens grants the attacker persistent access to the victim’s Microsoft 365 accounts and corporate data. from Human Risk Management Blog https://blog.knowbe4.com/uncovering-the-sophisticated-phishing-campaign-bypassing-m365-mfa
Read more

The Hacker News - ⚡ Weekly Recap: WhatsApp 0-Day, Docker Bug, Salesforce Breach, Fake CAPTCHAs, Spyware App & More

Cybersecurity today is less about single attacks and more about chains of small weaknesses that connect into big risks. One overlooked update, one misused account, or one hidden tool in the wrong hands can be enough to open the door. The news this week shows how attackers are mixing methods—combining stolen access, unpatched software, and clever tricks to move from small entry points to large from The Hacker News https://thehackernews.com/2025/09/weekly-recap-whatsapp-0-day-docker-bug.html
Read more