A sales table can tell you the size of a discount. It may take a policy memo to tell you whether staff purchases count toward revenue. If the policy changed last month, even a well-written query can give the wrong answer for the period you asked about.
SynthCorp, a research prototype from Jiani Zhang and colleagues, makes cases like that testable. It simulates routine work and disruptions, such as a policy or schema change, on one calendar. Those events produce tables, documents and a catalog together. The system records what each event changed, so the researchers can trace the reference answer for a question back to the simulated events and check the program that computes it.
The paper describes seven generated data estates and 256 tasks. In 205 of them, the query needed evidence from a document or catalog. In the authors’ model comparisons, furnishing one retrieved document helped when one memo contained the relevant rule. On larger or more dispersed collections, agents that could look up several sources before answering generally did better. The results measure performance inside the simulation. An agent that does well there still needs to prove it can find the right rule in your own systems, with the documents and access it will actually have. Benchmark construction and comparisons
A data team can borrow the test idea without recreating the entire company. Pick a familiar metric and ask for the same measure just before and just after a known rule change. Have the data owner supply the two expected answers and the dated source for each. Grade the policy the agent chose as well as the number it returned.