Wednesday, October 21

Where capability meets the work.

Enterprise Ops4 min read

Agent memory needs a relevance test

MemDev-Bench examines memory across real repository histories. Its findings make applicability and preserved process information more useful design questions than the amount of material an agent remembers.

As a leader: Evaluate memory against current repository state and measure the failures introduced by irrelevant past experience.

As an individual: Attach scope, source, and validity conditions to a reusable lesson rather than storing an unqualified instruction.

Read the analysis →

Enterprise Ops4 min read

Unattended agent changes need a real way back

A new preprint proposes measuring both verifier strength and recoverability before allowing an agent to improve itself unattended. An apparently successful approval loop is not enough evidence.

As a leader: Permit unattended changes only where the verifier is tested and the exact prior artifact is retained.

As an individual: Test restoring the saved version, then separately inspect consequences that restoration cannot undo.

Enterprise Ops4 min read

More reviewing agents can mean more false alarms

A controlled financial-document review study finds tradeoffs when work is divided among agents. Enterprises should measure misses, false alarms, and total review effort before adopting a team architecture.

As a leader: Compare a proposed agent team against a simple baseline using the same documents, requirements, and cost accounting.

As an individual: Track false alarms separately from caught defects so a busier reviewer does not automatically look better.

The boundaries that need to hold.

Security4 min read

Task completion needs an independent authorization check

AuthorizationBench examines whether agents stay inside delegated scope even when they can finish the task. The enterprise decision is where permission is enforced, observed, and reviewed.

As a leader: Make delegated scope an explicit acceptance criterion and enforce consequential actions at the tool boundary.

As an individual: Separate preparing a proposed action from executing it, and preserve the confirmation that authorizes execution.

• KPHarness: Co-Evolving Knowledge and Probing for Long-Horizon Gaming Harnesses ↗

  arXiv:2610.27417v1 Announce Type: new Abstract: A failed gaming trajectory reveals that an agent needs intervention, but not necessarily how its execution harness should change. Effective harness adaptation therefore requires gathering diagnostic evidence, rather than merely proposing alternative recovery rules. We present KPHarness, a framework that adapts the external execution harness of a frozen vision-language-action model through knowledge-probing co-evolution. Recovery knowledge supplies context-specific intervention hypotheses that guide active probing; probe outcomes reveal when these hypotheses apply and provide evidence for refining both recovery knowledge and later probe selection. This reciprocal process allows the harness to improve not only which interventions it applies, but also how it investigates failures. KPHarness separates fast policy execution from a slow research loop that translates accumulated knowledge and probing evidence into bounded harness revisions. Each revision is compared with the incumbent under matched sandbox conditions and accepted only when the paired evaluation shows improvement. Recoveries supported by further validation are retained for cross-task reuse, while immediate interaction feedback and delayed episode-success credit update action- and knowledge-probing policies during adaptation. Experiments on Minecraft tasks spanning mining, combat, and crafting show improved task coverage over bare policy execution, a manually specified fixed harness, and knowledge-only harness evolution with non-adaptive probing. Evaluation on previously unseen tasks further shows improved task completion with the evolved harness, knowledge bank, and probing policies held fixed,

• Harness Self-Evolution for Long-Horizon Terminal Agents ↗

  arXiv:2610.27435v1 Announce Type: new Abstract: Long-horizon terminal agents may solve a task in one run yet fail across repeated runs. Stochastic errors can propagate through actions and context, creating a gap between capability and reliable execution. We present RiseHarness, a framework for iteratively improving agent harnesses with model weights fixed. It focuses on flipping tasks with both successful and failed runs under the same configuration, using successful runs as references for failure analysis. RiseHarness combines agent trajectories with internal harness traces from a runtime monitor to examine how model actions interact with command execution, feedback, context management, and completion decisions. A Harness Handbook links this evidence to harness behaviors and source code, helping identify recurring problems in mechanisms shared across tasks. RiseHarness then searches other harness frameworks for relevant implementations and adapts them as design references to the current model and execution protocol. Candidate changes undergo behavioral checks and repeated task evaluation; versions with measured gains seed the next iteration. This loop integrates suitable mechanisms from different frameworks into a single harness and uses new execution evidence to guide further changes. We evaluate task success and reliability across models, ablate harness execution evidence and external research, and test transfer to unseen benchmarks with the same model. RiseHarness treats a specific model as the unit of harness adaptation, studying how changes to shared interaction mechanisms can make existing capabilities more reliable across tasks.

• Ask to Distill: Learning to Synthesize Data for On-Policy Distillation ↗

  arXiv:2610.27450v1 Announce Type: new Abstract: On-policy distillation (OPD) can succeed with surprisingly few training questions, motivating a view of questions as carriers of distributional knowledge transfer. How can we assess whether a newly generated question is valuable for distillation, and learn to generate better questions---without access to its correct answer? We introduce Ask-to-Distill (\method), a framework that treats synthetic questions as distribution elicitors and optimizes the teacher supervision they expose along student-generated trajectories. A student-initialized synthesizer is optimized using a distributional surrogate for question value, computed from a frozen student--teacher pair without reference answers, correctness rewards, or per-candidate student updates. The reward combines teacher--student disagreement with teacher probability mass on the student's candidate support, together with length and diversity controls. The generated questions are filtered and deduplicated into a prompt-only bank for standard OPD. Across mathematical reasoning, code generation, and agentic tasks, our method improves average performance and teacher-gap recovery over unoptimized student-generated prompts and KL-filtered real questions, while matching or exceeding real-bank OPD in the evaluated settings. Signal analyses further show that optimized questions can elicit denser supervision and more total signal from shorter student responses, supporting elicited distributional supervision as an objective for synthetic distillation data. Our code, data and models are publicly available at https://github.com/GasolSun36/Ask_to_Distill.

• Make delegated scope an explicit acceptance criterion and enforce consequential actions at the tool boundary.
• Evaluate memory against current repository state and measure the failures introduced by irrelevant past experience.
• Permit unattended changes only where the verifier is tested and the exact prior artifact is retained.