CSOAI · GSPC suite · 12 benchmarks

GSPC-DET

the detector interop axis spec — protocol only

Measure whether one producer's watermark is readable by another's detector, as an N x M matrix.

Statutory anchor

EU Code of Practice on Transparency — interoperability due 2 February 2027

How it is graded

Deterministically. A regex extracts the label, scored by macro-F1 — no model judges another model. A response with no readable label is reported as UNMEASURED and is excluded from the denominator; it is never scored as a wrong answer. That distinction separates "the model was wrong" from "the model never answered", and it is enforced in the harness code, not just claimed in prose.

What may be quoted

This axis is a published protocol, not a measured result. No score exists yet, and none is implied.

At a glance

axisdetector interop

stateSPEC

items—

gradingdeterministic

licenceApache-2.0

The dataset Run the harness CSOAI

DetBench — the protocol

A published protocol, not a measured matrix. No score exists yet. By 2 February 2027 the ~190 signatories of the EU Code of Practice must make watermark detection interoperable, and the Code concedes common evaluation standards have yet to emerge.

id	criterion	predicate	question	labels	why
det-001	effectiveness	detects_own_marking	Given an asset marked by provider P, does P's own detector return DETECTED?	DETECTED, NOT_DETECTED, ERROR	Baseline. A detector that cannot read its own marking fails before interoperability is in question.
det-002	interoperability	detects_foreign_marking	Given an asset marked by provider Q, does P's detector return DETECTED?	DETECTED, NOT_DETECTED, ERROR	This is the Commitment 2 obligation. It is the one nobody currently measures.
det-003	interoperability	agrees_on_negative	Given an unmarked asset, do P and Q both return NOT_DETECTED?	AGREE, DISAGREE, ERROR	False positives are the failure mode that destroys trust in a detector network.
det-004	robustness	survives_transform	After a declared transform (recompress, crop, resize, screenshot, re-encode), does the detector still return DETECTED?	SURVIVES, DESTROYED, ERROR	Shares its transform ladder with GSPC provenance (ProvBench), so the two axes are comparable.
det-005	reliability	stable_across_runs	Across k repeated runs on the identical asset, is the verdict identical every time?	STABLE, UNSTABLE, ERROR	A non-deterministic detector cannot support a compliance claim.
det-006	effectiveness	declares_absence_honestly	Where no marking is present, does the detector say so rather than guessing?	DECLARES_ABSENT, GUESSES, ERROR	The UNMEASURED discipline, applied to detectors.

6 predicates · csoai/gspc-det · published as data so a harness can consume it before any measurement exists · measurement, not certification.