When an assistant opens a help page, what has it been given: information about the product, or instructions about how to behave? It can receive both in the same page. A trusted brand tells you whose material you are reading. You still have to decide which parts may inform the answer and which parts are trying to direct the work.
Redact’s investigation of WhatsApp help pages reports hidden instructions in material delivered to AI readers. Its dated update records WhatsApp’s explanation that notes intended for its support chatbot had appeared because of a bug and had been removed. The dispute concerns the origin and purpose of those notes; the operational issue is broader. Retrieval can bring another party’s behavioral directions into an assistant’s context alongside ordinary reference material.
Think of a reference book beside a work order. The book can explain how a product works; the work order says what you have been asked to do. A paragraph in the book does not become a new work order just because it sounds authoritative. Enterprise retrieval needs that distinction to remain visible when both arrive as text.
Review the input the assistant receives
A screenshot is not a complete record of a retrieval operation. A tool might extract hidden text, metadata, accessibility content, surrounding navigation, or fragments from several pages. The important input for an evaluation is the exact representation returned to the model.
Choose a representative support or policy task and preserve that representation. Compare it with the user-visible page, then identify which differences are expected and which could affect the answer. This is a concrete test of the retrieval pipeline, rather than an assumption that the model saw what a human reviewer saw.
Trace a passage back to the page and the time it was retrieved. That connection gives a reviewer somewhere to begin when an answer goes wrong. Mix passages from several locations into an unlabeled block and you lose the starting points. The team then has to work out where an instruction entered before it can decide how to correct the answer.
Authority is a separate property
An organization may authorize a support assistant to apply an internal policy. That does not mean every external page describing the subject can add new restrictions to the assistant’s behavior. The relevant question is who is allowed to issue the instruction, not whether the wording sounds professional or helpful.
Represent that distinction in the system design. Task instructions, user requests, approved organizational rules, and retrieved material should have clear roles. A retrieved page should be treated as evidence about its subject. Requests embedded in it to suppress sources, change reporting, or take unrelated actions need to remain source content rather than operating instructions.
The model’s response is only one place to enforce the distinction. Tool access and application logic should also reflect the actual authorization. An assistant that reads a page should not thereby acquire permission to send a message, change an account, or withhold an independently required escalation route.
Evaluate changes in the answer
A useful test compares the same user question against a clean source and a deliberately altered version. Keep the factual material the same while introducing text that attempts to change the assistant’s behavior. Then measure what changes: source selection, omitted information, escalation, and proposed actions.
Do not score only whether the final sentence sounds plausible. A response can be fluent and factually unobjectionable in what it includes while selectively leaving out something the user needs. Review omissions as well as false statements, and preserve the trace needed to distinguish a retrieval failure from an instruction-following failure.
These are suggested evaluation methods, not a claim that this particular investigation proves a universal failure rate. The reported case establishes a reason to test the boundary in the actual application. Different tools, extraction methods, model instructions, and permissions may produce different outcomes.
Make correction practical
Suppose a reviewer finds directions that should never have governed the answer. Which source versions contained them, and which version of the assistant used those sources? Keep the retrieval record so the team can identify the affected work and rerun representative cases. A prompt change may help the next answer. The people responsible for earlier answers also need a way to learn what those answers were based on.
Keep the human review focused on the decision the assistant is supporting. For a customer-support application, that means the answer, the evidence provided, and the available escalation path. For an operational agent, it also means the actions attempted and the authority under which they were attempted.
Keep an approved reference answer for the test, including what the assistant must disclose and when it should ask the user for direction. Have a reviewer assess the clean and altered cases without being told which is which. This makes selective omission easier to notice and gives the team a repeatable example to retain after a fix. The exercise should remain isolated from real customer accounts and actual outbound actions.
Open a real retrieved payload and test a conflicting instruction in an isolated case. Keep the source’s useful information, and check whether its attempt to redirect the assistant stays on the evidence side of the boundary. A source can be current, valuable, and authoritative about its own product without acquiring authority over the task. Keep the reference book beside the work order, with the distinction intact.