EVOKE

Workshop Proposal · HCOMP and CI 2026

EVOKE

Eliciting Verifiable, Operational Knowledge from Experts

Turning what experts know into formal representations a machine can check: constraints, optimization models, logic, programs, typed specifications, ontologies, and probabilistic models.

Call for participation Workshop format
Venue Virginia Tech, near Washington DC
Date September 27, 2026
Submission deadline July 23, 2026
Mode Full-day, in person, with a hybrid option
Participants Expected attendance: 40 to 60

Overview

We have endless data and powerful AI.We still lack knowledge which is provably true and useful.

EVOKE focuses on drawing expert knowledge into forms a machine can reason over and check. It turns on the one question that separates knowledge from data: how do we know what we captured is correct and useful?

When the target is a formal representation, that question becomes measurable. A solver returns ground truth. A verifier certifies a result. A contradiction surfaces in the expert's own claims, one they never knew they held.

Large language models suddenly made it cheap to elicit knowledge from an expert, taking the first steps towards resolving the bottleneck that stalled this field for decades. But, they capture it as prose, with no guarantee it holds up, and they fail silently: the most dangerous output is one that looks right but is subtly wrong.

The core question

"How do we know that knowledge elicited from experts is correct and useful?"

This question breaks into three sub-questions EVOKE is built to answer:

01

Can an expert's claims be turned into a form a machine can reason over correctly?

02

Can we measure the correctness, completeness, and usefulness of these claims against ground truth?

03

Can we build systems which fail loudly, surfacing ambiguity and contradiction, instead of quietly producing answers that look right but are wrong?

Rationale and objectives

Getting what experts know into a machine just got far easier — but it is far from solved.

Expert systems and hand-built knowledge bases stalled because getting what a person knows into a usable form was slow and brittle. That barrier just dropped, and the work is now spread across communities that rarely share a room.

Knowledge engineering, expert judgment, human-AI interaction, operations research, neuro-symbolic AI, natural language processing, formal methods, decision analysis: each holds part of the answer, and EVOKE is built to put them in conversation around shared evaluation.

  1. Bring scattered communities together around a shared language for elicitation, soundness, and usefulness.
  2. Define shared evaluation that uses solvers and verifiers as ground truth, and borrow calibration and aggregation from the study of expert judgment.
  3. Launch an open shared task and dataset of elicited problems that a solver can check.
  4. Produce a workshop report, a position paper on evaluation standards, and scope a journal special issue and a recurring EVOKE series.

Format and activities

A full-day workshop built around eliciting and verifying concrete information.

EVOKE is an in-person workshop with a hybrid option so remote authors and speakers can take part. The shared-task session is hands-on by design: every attendee should help elicit and verify something before the day is out.

Two keynotes

One on learning theory and educability; one on integrating language models with formal verifiers and solvers.

Talks from experts

Classic human-elicitation traditions (knowledge acquisition, structured expert judgment) paired with the new LLM-and-solver frontier.

Contributor Program

Full papers, short papers, position papers, demos, datasets, benchmarks, and lightning talks in a single track.

Interactive shared task

A live, scored elicitation challenge: elicit a formal problem from an unreliable narrator and score it against a solver.

Panel

Can extracted knowledge be proven useful — and is elicitation its own skill, or something that bigger AI models will simply absorb?

Data clinic

Industry participants bring real elicitation problems, with demos running through breaks and an open repository for shared artifacts.

Call for participation

Bring an expert, a verifier, and a hard question.

EVOKE invites research, position, demo, and dataset papers on turning expert knowledge into forms a machine can reason over correctly.

Submissions are 2 to 8 pages excluding references. Contribute as an author, work together on the shared task, or participate in discussions.

Submission deadline July 23, 2026
Notifications August 7, 2026
Workshop Sept. 27, 2026

Shared task

A live, scored challenge run during the workshop: elicit a formal problem from an unreliable narrator, then score it against a solver. It is evaluation that can fail loudly, reveal contradictions, and show the moment human knowledge becomes something a machine can check.

Topics in scope

Formal representations

Hard and soft constraints, optimization and scheduling models, logic, programs, typed specifications, ontologies, knowledge graphs, probabilistic models, and priors.

LLM-mediated elicitation

Multi-turn dialogue, mixed initiative, clarifying questions, underspecification, ambiguity, contradiction, tacit knowledge, and constraints inferred from behavior.

Evaluation and benchmarks

Correctness, completeness, faithfulness, usefulness, calibration, known answers, unreliable narrators, and metrics for elicitation quality and efficiency.

Solvers and verifiers

Autoformalization, neuro-symbolic methods, constraint programming, SAT, SMT, MILP, theorem provers, and soundness-preserving pipelines.

Educability

Teaching versus learning, belief choice and belief verification, robust logic, and the sample and teaching efficiency of elicitation.

Trust and accountability

Decision traces, auditability, bias in elicitation, aggregation across experts, provenance, and governance of machine-captured knowledge.

Application areas

Robotics, operations, compliance, agentic AI, and vertical AI.

Organizers

Organizing committee

The organizing committee is balanced across academia and industry, research area, and demographics; at least two organizers will attend in person.

Praveen Paritosh

MLCommons

Works on knowledge representation, crowdsourcing, and evaluation. Co-founded data-centric benchmarking efforts at MLCommons, including DataPerf and the DMLR initiative.

Vinay K. Chaudhri

Knowledge representation and reasoning, and AI knowledge infrastructure

Building the knowledge infrastructure for modern AI through knowledge representation, reasoning, and knowledge engineering. Former SRI International Scientist and Stanford instructor

Ram Bala

Santa Clara University

Professor of AI and analytics whose work spans operations research, optimization, and analytics for data-driven decision-making.

Clifton McFate

AI researcher

Specializes in knowledge representation, natural language understanding, and neuro-symbolic reasoning; built business-logic elicitation systems at Elemental Cognition.

Outcomes and outreach

EVOKE is built to leave lasting infrastructure behind, not just a day of talks.

Planned outcomes include a workshop report, an open benchmark, a dataset, a public leaderboard, a community position paper on evaluation standards for elicited knowledge, and scoping for a journal special issue and a recurring EVOKE series.

Outreach will use HCOMP and CI channels, knowledge-engineering and Semantic Web lists including K-CAP, operations research venues including INFORMS, CPAIOR, and AI4OPT, formal-methods, neuro-symbolic, NLP, human-AI interaction, and CSCW communities, plus direct outreach to applied-AI companies and labs.

References

Intellectual foundations

  1. Valiant, L. (2024). The Importance of Being Educable: A New Theory of Human Uniqueness. Princeton University Press.
  2. Valiant, L. (2024). The Parameters of Educability. arXiv:2412.09480.
  3. Valiant, L. (2000). Robust Logics. Artificial Intelligence, 117.
  4. Valiant, L. (1984). A Theory of the Learnable. Communications of the ACM, 27(11).
  5. Feigenbaum, E. (1984). Knowledge engineering: the applied side of artificial intelligence. Annals of the New York Academy of Sciences.
  6. Cooke, R. M. (1991). Experts in Uncertainty: Opinion and Subjective Probability in Science. Oxford University Press.
  7. Horvitz, E. (1999). Principles of mixed-initiative user interfaces. CHI.
  8. Lawless, C., Schoeffer, J., Le, L., et al. (2024). "I Want It That Way": Enabling Interactive Decision Support Using Large Language Models and Constraint Programming. ACM Transactions on Interactive Intelligent Systems, 14(3).
  9. Ahmadi Teshnizi, A., Gao, W., and Udell, M. (2024). OptiMUS: Optimization Modeling Using MIP Solvers and Large Language Models. arXiv:2310.06116.
  10. Cosler, M., Hahn, C., Mendoza, D., Schmitt, F., and Trippel, C. (2023). nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Models. CAV.
  11. Vaccaro, M., Almaatouq, A., and Malone, T. (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour.