A safety test in English leaves a question unanswered: what happens when the same request arrives in another language, with local names and context?
The October 29 MSafeTemp preprint studies that gap. Max Neuwinger, Hanna Yukhymenko and Imanol Schlag build templates from existing prompts, translate them and fill their slots with generic or culturally relevant entities. The separate HarmBench evaluation track spans 60 languages; the WildGuard-based training track spans 44. Those counts describe different parts of the corpus.
The version-pinned experiments show why a single multilingual average can conceal important behavior. Several tested models comply with harmful translated requests more often than with the English originals. Llama-3.3-70B-Instruct fails most often on the English originals in these comparisons. Localized variants reach failures that translation misses, but they do not consistently produce a higher overall attack-success rate. The finding concerns coverage of different failures, rather than one universally stronger attack.
The study also pairs harmful prompts with intended benign alternatives to measure unnecessary refusal. That control has a limit: human raters endorsed only 65% of the sampled benign labels. Some softened requests still looked harmful, so the resulting refusal measure may overstate refusal of legitimate work. Human validation covered 11 languages, while most evaluation relied on automated model judges. Fixed, single-turn templates also leave longer conversations outside the tested setting.
For a service team, this suggests a practical review. Start with the languages people actually use. Compare original, translated and locally grounded requests, and include legitimate requests that resemble the risky ones. Ask fluent reviewers to check intent and labels before treating a refusal as a defect. Keep the exact model, prompt and judge versions with each result.
The decision is whether the service both refuses dangerous work and completes legitimate work in each supported language. A strong English result answers only part of that question.