Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Models

{
  "by": "Jimmc414",
  "descendants": 4,
  "id": 40215100,
  "kids": [
    40216933,
    40216434,
    40216381,
    40217318
  ],
  "score": 47,
  "time": 1714504816,
  "title": "Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Models",
  "type": "story",
  "url": "https://arxiv.org/abs/2404.18796"
}

{
  "author": "Pat Verga",
  "date": "2024-04-29T12:00:00.000Z",
  "description": "As Large Language Models (LLMs) have become more advanced, they have outpaced our abilities to accurately evaluate their quality. Not only is finding data to adequately probe particular model properties difficult, but evaluating the correctness of a model’s freeform generation alone is a challenge. To address this, many evaluations now rely on using LLMs themselves as judges to score the quality of outputs from other LLMs. Evaluations most commonly use a single large model like GPT4. While this method has grown in popularity, it is costly, has been shown to introduce intramodel bias, and in this work, we find that very large models are often unnecessary. We propose instead to evaluate models using a Panel of LLm evaluators (PoLL). Across three distinct judge settings and spanning six different datasets, we find that using a PoLL composed of a larger number of smaller models outperforms a single large judge, exhibits less intra-model bias due to its composition of disjoint model families, and does so while being over seven times less expensive.",
  "image": "https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png",
  "logo": null,
  "publisher": "arXiv.org",
  "title": "Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models",
  "url": "https://arxiv.org/abs/2404.18796v2"
}

{
  "url": "https://arxiv.org/abs/2404.18796",
  "title": "Replacing Judges with Juries: Evaluating LLM Generations with a...",
  "description": "As Large Language Models (LLMs) have become more advanced, they have outpaced our abilities to accurately evaluate their quality. Not only is finding data to adequately probe particular model...",
  "links": [
    "https://arxiv.org/abs/2404.18796v2",
    "https://arxiv.org/abs/2404.18796"
  ],
  "image": "https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png",
  "content": "<div>\n  <div>\n    <p>\n      </p><h2>Computer Science &gt; Computation and Language</h2>\n    <p></p>\n    <p><strong>arXiv:2404.18796</strong> (cs)\n    </p>\n<div>\n                <p>View a PDF of the paper titled Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models, by Pat Verga and 8 other authors</p>\n    <p><a target=\"_blank\" href=\"https://arxiv.org/pdf/2404.18796\">View PDF</a>\n    <a target=\"_blank\" href=\"https://arxiv.org/html/2404.18796v2\">HTML (experimental)</a></p><blockquote>\n            <span>Abstract:</span>As Large Language Models (LLMs) have become more advanced, they have outpaced our abilities to accurately evaluate their quality. Not only is finding data to adequately probe particular model properties difficult, but evaluating the correctness of a model's freeform generation alone is a challenge. To address this, many evaluations now rely on using LLMs themselves as judges to score the quality of outputs from other LLMs. Evaluations most commonly use a single large model like GPT4. While this method has grown in popularity, it is costly, has been shown to introduce intramodel bias, and in this work, we find that very large models are often unnecessary. We propose instead to evaluate models using a Panel of LLm evaluators (PoLL). Across three distinct judge settings and spanning six different datasets, we find that using a PoLL composed of a larger number of smaller models outperforms a single large judge, exhibits less intra-model bias due to its composition of disjoint model families, and does so while being over seven times less expensive.\n    </blockquote>\n  </div>\n    <div>\n      <h2>Submission history</h2><p> From: Pat Verga [<a target=\"_blank\" href=\"https://arxiv.org/show-email/0e344dbe/2404.18796\">view email</a>]      <br />            <strong><a target=\"_blank\" href=\"https://arxiv.org/abs/2404.18796v1\">[v1]</a></strong>\n        Mon, 29 Apr 2024 15:33:23 UTC (7,795 KB)<br />\n    <strong>[v2]</strong>\n        Wed, 1 May 2024 15:37:11 UTC (7,795 KB)<br />\n</p></div>\n  </div>\n<div>    <div>\n      <p><a></a>\n      <span>Full-text links:</span></p><h2>Access Paper:</h2>\n      <ul>\n  <p>\nView a PDF of the paper titled Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models, by Pat Verga and 8 other authors</p><li><a target=\"_blank\" href=\"https://arxiv.org/pdf/2404.18796\">View PDF</a></li><li><a target=\"_blank\" href=\"https://arxiv.org/html/2404.18796v2\">HTML (experimental)</a></li><li><a target=\"_blank\" href=\"https://arxiv.org/src/2404.18796\">TeX Source\n </a></li></ul>\n    </div>\n        <div><p>\n    Current browse context: </p><p>cs.CL</p>\n    </div>\n<p><span>export BibTeX citation</span>\n</p>\n<div>\n  <p></p><h3>Bookmark</h3><p></p><p><a target=\"_blank\" href=\"http://www.bibsonomy.org/BibtexHandler?requTask=upload&amp;url=https://arxiv.org/abs/2404.18796&amp;description=Replacing%20Judges%20with%20Juries:%20Evaluating%20LLM%20Generations%20with%20a%20Panel%20of%20Diverse%20Models\" title=\"Bookmark on BibSonomy\">\n    <img src=\"https://arxiv.org/static/browse/0.3.4/images/icons/social/bibsonomy.png\" alt=\"BibSonomy logo\" />\n  </a>\n  <a target=\"_blank\" href=\"https://reddit.com/submit?url=https://arxiv.org/abs/2404.18796&amp;title=Replacing%20Judges%20with%20Juries:%20Evaluating%20LLM%20Generations%20with%20a%20Panel%20of%20Diverse%20Models\" title=\"Bookmark on Reddit\">\n    <img src=\"https://arxiv.org/static/browse/0.3.4/images/icons/social/reddit.png\" alt=\"Reddit logo\" />\n  </a>\n</p></div>  </div>\n<div><p>\n    <label>Bibliographic Tools</label></p><div>\n      <h2>Bibliographic and Citation Tools</h2>\n      <div>\n          <p><label>\n              <span></span>\n              <span>Bibliographic Explorer Toggle</span>\n            </label>\n          </p>\n        </div>\n    </div>\n    <p>\n    <label>Code, Data, Media</label></p><div>\n      <h2>Code, Data and Media Associated with this Article</h2>\n    </div>\n      <p>\n      <label>Demos</label></p><div>\n        <h2>Demos</h2>\n      </div>\n      <p>\n      <label>Related Papers</label></p><div>\n        <h2>Recommenders and Search Tools</h2>\n      </div>\n      <p>\n      <label>\n        About arXivLabs\n      </label></p><div>\n            <h2>arXivLabs: experimental projects with community collaborators</h2>\n            <p>arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.</p>\n            <p>Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.</p>\n            <p>Have an idea for a project that will add value for arXiv's community? <a target=\"_blank\" href=\"https://info.arxiv.org/labs/index.html\"><strong>Learn more about arXivLabs</strong></a>.</p>\n          </div>\n    </div>\n</div>",
  "author": "",
  "favicon": "https://arxiv.org/static/browse/0.3.4/images/icons/favicon-16x16.png",
  "source": "arxiv.org",
  "published": "",
  "ttr": 77,
  "type": "website"
}