Orchestrator

Graph

Node Configuration

Run with Parameters

Pipeline JSON

{
  "edges": [
    {
      "to": "pairwise_judge",
      "from": "__start__"
    },
    {
      "to": "__end__",
      "from": "pairwise_judge"
    }
  ],
  "nodes": [
    {
      "name": "pairwise_judge",
      "type": "llm",
      "config": {
        "llm": {
          "model": "gpt-4o",
          "temperature": 0
        },
        "output_schema": {
          "type": "object",
          "title": "SummarizationPairwisePreferenceResult",
          "required": [
            "preference",
            "dimension_winners",
            "rationale",
            "decisive_differences"
          ],
          "properties": {
            "rationale": {
              "type": "string",
              "description": "Brief evidence-backed comparison"
            },
            "preference": {
              "enum": [
                "a",
                "b",
                "tie"
              ],
              "type": "string",
              "description": "Overall preferred candidate"
            },
            "dimension_winners": {
              "type": "object",
              "required": [
                "faithfulness",
                "coverage",
                "instruction_adherence",
                "clarity"
              ],
              "properties": {
                "clarity": {
                  "enum": [
                    "a",
                    "b",
                    "tie"
                  ],
                  "type": "string"
                },
                "coverage": {
                  "enum": [
                    "a",
                    "b",
                    "tie"
                  ],
                  "type": "string"
                },
                "faithfulness": {
                  "enum": [
                    "a",
                    "b",
                    "tie"
                  ],
                  "type": "string"
                },
                "instruction_adherence": {
                  "enum": [
                    "a",
                    "b",
                    "tie"
                  ],
                  "type": "string"
                }
              }
            },
            "decisive_differences": {
              "type": "array",
              "items": {
                "type": "string"
              },
              "description": "Concrete differences that drove the preference"
            }
          }
        },
        "system_prompt": "You compare two summarization candidates for the same task.\n\nThe human message contains:\n- SOURCE: original document (or excerpt)\n- PROMPT: instructions given to the summarizer(s)\n- SUMMARY_A: first candidate\n- SUMMARY_B: second candidate\n- Optional LABELS: opaque ids for the systems that produced A and B (ignore for preference; do not favor a label)\n\nDecide which summary better satisfies PROMPT given SOURCE. Prefer faithfulness over fluency. If both are equally good or equally bad, choose tie.\n\nDo not rewrite either summary. Do not invent SOURCE content.\n\nCompare on:\n- faithfulness (supported by SOURCE)\n- coverage (PROMPT-required points)\n- instruction_adherence\n- clarity / concision\n\nReturn preference = a | b | tie, plus per-dimension winners and a short rationale."
      }
    }
  ],
  "description": "Skeleton: pairwise preference judge for two summarization candidates (shadow / prompt / model A-B tests). Seed messages with SOURCE, PROMPT, SUMMARY_A, SUMMARY_B, and optional labels for which system produced each. Prefer randomizing A/B assignment upstream to reduce position bias."
}