Orchestrator

Graph

Node Configuration

Run with Parameters

Pipeline JSON

{
  "edges": [
    {
      "to": "pairwise_judge",
      "from": "__start__"
    },
    {
      "to": "__end__",
      "from": "pairwise_judge"
    }
  ],
  "nodes": [
    {
      "name": "pairwise_judge",
      "type": "llm",
      "config": {
        "llm": {
          "model": "gpt-4o",
          "temperature": 0
        },
        "output_schema": {
          "type": "object",
          "title": "GenericPairwisePreferenceResult",
          "required": [
            "preference",
            "criteria_winners",
            "rationale",
            "confidence"
          ],
          "properties": {
            "rationale": {
              "type": "string",
              "description": "Brief justification tied to CRITERIA"
            },
            "confidence": {
              "enum": [
                "low",
                "medium",
                "high"
              ],
              "type": "string",
              "description": "How clear the preference was"
            },
            "preference": {
              "enum": [
                "a",
                "b",
                "tie"
              ],
              "type": "string",
              "description": "Overall preferred candidate"
            },
            "criteria_winners": {
              "type": "array",
              "items": {
                "type": "object",
                "required": [
                  "criterion",
                  "winner"
                ],
                "properties": {
                  "note": {
                    "type": "string"
                  },
                  "winner": {
                    "enum": [
                      "a",
                      "b",
                      "tie"
                    ],
                    "type": "string"
                  },
                  "criterion": {
                    "type": "string"
                  }
                }
              },
              "description": "Per-criterion preference aligned to CRITERIA"
            }
          }
        },
        "system_prompt": "You compare two model outputs for the same task (shadow / A-B experiment).\n\nThe human message contains:\n- TASK: what the model was asked to do\n- CONTEXT: optional source material or inputs\n- CRITERIA: ordered list of evaluation priorities (first matters most)\n- CANDIDATE_A / CANDIDATE_B: the two outputs\n- Optional EXPERIMENT: opaque metadata (model ids, prompt versions) \u2014 do not prefer a side because of these labels\n\nPick preference = a | b | tie using CRITERIA order. Prefer factual grounding in CONTEXT when present. Do not rewrite candidates.\n\nThis blueprint is a skeleton: replace the system prompt / schema when a domain (summarization, relevancy, classification, \u2026) needs tighter rubrics."
      }
    }
  ],
  "description": "Skeleton: generic pairwise preference for shadow experiments (model type, prompt variant, temperature, etc.). Seed messages with TASK, CONTEXT (optional), CANDIDATE_A, CANDIDATE_B, and CRITERIA. Swap A/B upstream to mitigate position bias. Specialize per domain rather than overloading this graph."
}