Skip to content
Back to the Standard
Reproducibility Seal

The signed envelope that proves a score is reproducible.

Every score the SongForgeAI API emits ships with an ed25519 reproducibility seal. The seal is a small JSON object listing the rubric version, the model, the temperature, and the build SHA that produced the score. The signature proves the seal was issued by a SongForgeAI deploy and hasn’t been tampered with. This page is the canonical specification for that contract: every field, the verification flow, and the guarantees the seal makes (and doesn’t).

What the seal guarantees

  • The score, the rubric version, the model id, the temperature, and the build SHA are bound together by the signature. Changing any one of them after the fact invalidates the signature.
  • The signature was produced by the holder of the private key paired with the public key at /.well-known/songforgeai-pubkey.json. Only SongForgeAI deploys hold the private key.
  • The seal travels alongside the score, not behind a paywall or an API call. Anyone with the public key can verify it offline.

What the seal does NOT guarantee

  • That the rubric is correct. The seal says “this score came from this rubric.” It does not say the rubric’s metric weights are optimal. Disagreeing with the rubric is the point of publishing it; the seal lets that disagreement happen against a fixed, citeable artifact.
  • That an LLM running the rubric is the same as a human. The seal documents the runtime; the underlying eval is still Claude scoring against the published rubric. Real inter-rater agreement statistics from a 30-rater human cohort are an upcoming v2 deliverable; the rubric is specifically designed so that a human grader, a different model, or an academic team can apply it and we can compare results.
  • That the score will be identical on a re-run. Anthropic’s API has bounded nondeterminism (~1-2 points at temp 0.7). The seal proves the rubric + runtime; it doesn’t override the underlying stochasticity.

Field schema

The seal is the JSON object below the seal key in every score response. The signature in the signature field covers the canonical-JSON serialization of this object; field order is sorted alphabetically before signing.

  • rubricVersionstring (semver)

    The version of the published rubric the score was computed against. Matches the version field in scoring-standard.json. Consumers verifying a score should reject the seal when this is more than one MINOR off from their expected version.

  • modelstring

    Anthropic model ID that produced the eval. Currently `claude-sonnet-4-20250514`; documented on the public model card at /scoring/standard/model-card and pinned in src/lib/model-registry.ts. Required for reproducibility — running the rubric against a different model is a different scoring system.

  • temperaturenumber

    Sampling temperature used by the eval. Pinned at 0.7 for the Lyric Scoring Standard v1.x. Documented on the model card.

  • buildShastring (git sha)

    The deploy SHA of the SongForgeAI codebase that produced the score. Lets a reviewer trace back to the exact prompt code, banned-terms list version, and overlay set in effect. Reproducibility goes deeper than rubric+model — it includes the prompt fragments and the cliche scanner.

  • buildinteger

    Monotonic build number paired with the SHA. Easier to read in the UI than a 7-char SHA; same provenance content.

  • exampleIdstring (optional)

    When present, identifies the specific scored artifact (corpus entry id, calibration sample id, or song id). Lets a reviewer correlate the seal with the lyric body it covers.

  • scoredAtstring (ISO 8601 timestamp)

    When the score was produced. Combined with rubricVersion, the consumer can verify the score is current relative to a rubric MINOR bump.

  • compositeScorenumber (0-100)

    The composite output of the rubric on this lyric. Signed alongside the rubric+model+build context so changing the score after the fact would invalidate the signature.

  • gradestring (S+ / S / A+ / A / B+ / B / C+ / C / D+ / D / F)

    The letter grade derived from compositeScore via the published threshold table (scoreToGrade in @songforgeai/scoring-rubric). Included for downstream consumers who index on grade rather than raw composite.

Public key

The Ed25519 public key is published at:

GET https://songforgeai.com/.well-known/songforgeai-pubkey.json

The file is static, CDN-cacheable, and pins the current public key in hex format. Fetch once and pin in your consumer; the key only rotates on a major-version bump and any rotation is announced in the rubric changelog with a 30-day overlap window.

Verification flow

  1. 1

    Receive a score response with a seal

    Every successful POST to /api/v1/score returns a JSON body containing the score plus a `seal` field (object) and a `signature` field (base64). The seal is the data that was signed; the signature is the proof.

  2. 2

    Fetch the public key

    The Ed25519 public key is published at /.well-known/songforgeai-pubkey.json. The file is static, cacheable, and CDN-served. Fetch it once and pin it; rotate handling is documented at the bottom of this page.

  3. 3

    Run ed25519 verification

    Compute the ed25519-verify against (data = canonical JSON of the seal object, signature = the base64-decoded signature, public key = the hex-decoded public key). Pass = the score came from the claimed system. Fail = the seal has been tampered with, mis-signed, or signed with a different key.

  4. 4

    Cross-reference rubric version

    Even after a green signature check, confirm seal.rubricVersion matches the rubric your consumer was built against. Mismatched MINORs may have differing metric definitions; mismatched MAJORs are a different rubric entirely.

  5. 5

    (Optional) Cross-reference build SHA

    For deeper reproducibility, look up the build SHA in the public engineering log at /engineering. The deploy log lets a reviewer trace exactly which prompt code, cliche list, and per-subgenre overlays were active when the score was produced.

Try it

The browser verifier at /verify runs the full ed25519 check on a paste-able seal + signature + public key. No account, no API call. The example fixture button loads a known-good seal so you can confirm the verifier is wired correctly before checking your own scores.

Open the verifier

Implementer notes

  • The reference TypeScript verifier is in src/lib/seal-signer.ts of the SongForgeAI repo. Same function the /verify page calls in-browser.
  • The @songforgeai/scoring-rubric npm package re-exports the ReproducibilitySeal type so consumers can type the seal field without duplicating the interface.
  • For Python or R consumers, the canonical-JSON + ed25519-verify recipe is documented inline above. Any standard ed25519 library (libsodium, PyNaCl, sodium-r) handles step 3.
  • Suspicious or reproducibly-failing seals can be reported at support@songforgeai.com. Operator-confirmed bugs in the seal pipeline result in a public engineering-log entry within 7 days.
Specification published under CC BY 4.0; same license as the rubric and the corpus. Cite as “Lyric Scoring Standard v1.2.0 — Reproducibility Seal Specification (B1960).” Back to the standard
More from the standard