Skip to content

Interoperability

ASB is the provenance and specification layer that feeds benchmark formats, not a competitor to them.

An ASB artifact starts from a paper and keeps the link to it: the research question, the reported finding, verbatim evidence, the inputs, outputs, tools and skills, an evaluation plan and the research path behind the result. Formats for analyses, research objects, benchmark tasks and agent skills can be written from that material. This section lists those bridges, what each one carries and its status.

  • A task card records one bounded task extracted from a published paper.
  • ASB Schema produces ASTRA documents and is upstream of them for published research: the paper's tasks, decisions and evidence become an ASTRA analysis.
  • ASB Schema is upstream of Terminal-Bench, whose tasks need an oracle and a verifier that a card does not carry; the card supplies the provenance, the reference result and a draft task statement.

Bridge status

existing
Code of the reference pipeline writes or reads the format today.
feasible
The two formats line up field by field; a converter is a short piece of new code.
needs new code
A new emitter, together with content the card does not carry (for example an oracle and a verifier), written for each task.

Bridges

Format Description Direction Status
ASTRA astra.yaml One scientific analysis: inputs, outputs, decisions, options, insights destination existing
ASTRA run_trace.yaml The build of a run, recorded as ASTRA decisions destination existing
RO-Crate 1.1 A directory of research data with one JSON-LD metadata file destination existing
Process Run Crate RO-Crate profile for the execution of tools as CreateAction entities destination feasible
ARA One agent-native research artifact per paper, with a trace/ layer destination existing
CWL, Nextflow, Snakemake Workflow languages destination existing
Harbor and Terminal-Bench A task directory with an instruction, an environment, an oracle and a verifier that writes a reward destination needs new code
ScienceAgentBench Data-driven tasks whose output is a Python program destination existing
PaperBench Replication of a whole paper graded by a rubric tree destination and source existing
MLE-bench Machine-learning competitions graded against a leaderboard destination needs new code
Inspect Evaluation framework: dataset, solver and scorer destination feasible
BixBench, CORE-Bench, DiscoveryBench Question-and-answer items about published analyses destination feasible
Agent Skills SKILL.md A skill directory with a front-matter name and description destination existing
MCP Skills extension Skills served by a Model Context Protocol server destination feasible
schema.org, PROV-O, Dublin Core, EDAM, P-Plan Shared vocabularies source of terms existing
SSSOM A table format for ontology mappings destination existing
Claim ledgers and nanopublications Verbatim claims grouped into results; RDF assertions with provenance destination and source existing
Zenodo and DataCite Archived deposits with DOIs destination existing

What the format carries

Properties that make a card a good source for these formats:

Property Slots
Typed origin of the task provenance_source (published, preprint, draft_paper, local_project), source_package, crossref_doi
Verbatim evidence with a stated reason evidence_snippets (snippet_id, source_path, reason, text)
Question and finding as a pair research_question, finding
Declared relation to the paper task_kind (reproduction, analysis, component_reconstruction, extension)
Graded executability reproducibility_tier, run_command, run_cwd, run_timeout_seconds
Evaluation plan in two parts evaluation_strategy (direct_checks, expert_review), landmark_outputs, expected_artifact_name
Binding to released skills and tools tools, skills; ASBSkill, ASBTool, SkillCollection
Chained tasks input_from
Research object as packaging unit SciTaskCapsule with RO-Crate metadata
The research path, abandoned branches included ExplorationTree with support_level on each step
Release and credit on the collection license, doi, lead_curators, roles on SkillCollection
One schema language, generated forms LinkML sources; JSON Schema and a JSON-LD context generated from them; 72 exact and 120 close mappings

A card leaves to the destination format what belongs to it: the execution environment, an oracle solution, hidden answers, canary strings and per-task license and authorship.

Pages