Task preparation

Tasks are prepared as batches (JSON). Understanding who can prepare them, what types exist, and what to include ensures evaluations run smoothly.

Who can prepare a task and how

Administrators (or users with upload permission) can create and upload batches via the Datasets page.
Batches are uploaded as JSON; the schema is the same for MT and ASR except for language fields and the type of input (text vs. audio URL/path).

Types of tasks: MT and ASR

MT: each task has a source segment (input) and one or more translation hypotheses (model outputs).
ASR: each task has an audio input (URL or path) and one or more transcription hypotheses. ASR evaluations are batch-based.

Required fields

Top level:

dataset_name, dataset_domain, batch_name, tasks (array).
MT: add source_language and target_language. ASR: add language.

Each task: id, input (source text for MT, audio URL or path for ASR), models (array). Each model: output (text), model (string identifier), rate (number, use 0 when preparing), rank (number, use 0).

Optional fields and default values

rating_guideline: if not provided, the default 1–5 scale is used (Critical, Major, Minor, Neutral, Kudos).
domains: if not provided, no domain dropdown is shown; evaluators can still complete the task.
task_models_shuffles: if not provided, model identifiers in the batch are shown as-is.
Per-task reference: optional; evaluators can add a reference in the UI when needed.

MT batch example

input is the source segment; each model output is the translation. rate and rank are set by evaluators; use 0 in the upload.

{
  "dataset_name": "Swahili-English News",
  "dataset_domain": "news",
  "batch_name": "Batch 001",
  "source_language": "sw",
  "target_language": "en",
  "tasks": [
    {
      "id": "task-1",
      "input": "Habari za leo zinasema joto litaendelea.",
      "models": [
        { "output": "Today's news says the heat will continue.", "model": "Model A", "rate": 0, "rank": 0 },
        { "output": "The news today say heat will continue.", "model": "Model B", "rate": 0, "rank": 0 }
      ]
    }
  ]
}

ASR batch example

input must be a URL or path to the audio. Use language instead of source_language and target_language.

{
  "dataset_name": "Amharic Read Speech",
  "dataset_domain": "speech",
  "batch_name": "ASR Batch 001",
  "language": "am",
  "tasks": [
    {
      "id": "task-1",
      "input": "https://example.com/audio/sample1.wav",
      "models": [
        { "output": "Transcribed text from model A.", "model": "Model A", "rate": 0, "rank": 0 },
        { "output": "Transcribed text from model B.", "model": "Model B", "rate": 0, "rank": 0 }
      ]
    }
  ]
}