The Matrix: A Bayesian learning model for LLMs

{
  "by": "smaddox",
  "descendants": 0,
  "id": 40249915,
  "score": 3,
  "time": 1714756229,
  "title": "The Matrix: A Bayesian learning model for LLMs",
  "type": "story",
  "url": "https://arxiv.org/abs/2402.03175"
}

{
  "author": "Siddhartha Dalal",
  "date": "2024-02-05T12:00:00.000Z",
  "description": "In this paper, we introduce a Bayesian learning model to understand the behavior of Large Language Models (LLMs). We explore the optimization metric of LLMs, which is based on predicting the next token, and develop a novel model grounded in this principle. Our approach involves constructing an ideal generative text model represented by a multinomial transition probability matrix with a prior, and we examine how LLMs approximate this matrix. We discuss the continuity of the mapping between embeddings and multinomial distributions, and present the Dirichlet approximation theorem to approximate any prior. Additionally, we demonstrate how text generation by LLMs aligns with Bayesian learning principles and delve into the implications for in-context learning, specifically explaining why in-context learning emerges in larger models where prompts are considered as samples to be updated. Our findings indicate that the behavior of LLMs is consistent with Bayesian Learning, offering new insights into their functioning and potential applications.",
  "image": "https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png",
  "logo": "https://logo.clearbit.com/arxiv.org",
  "publisher": "arXiv.org",
  "title": "The Matrix: A Bayesian learning model for LLMs",
  "url": "https://arxiv.org/abs/2402.03175v1"
}

{
  "url": "https://arxiv.org/abs/2402.03175",
  "title": "The Matrix: A Bayesian learning model for LLMs",
  "description": "In this paper, we introduce a Bayesian learning model to understand the behavior of Large Language Models (LLMs). We explore the optimization metric of LLMs, which is based on predicting the next...",
  "links": [
    "https://arxiv.org/abs/2402.03175v1",
    "https://arxiv.org/abs/2402.03175"
  ],
  "image": "https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png",
  "content": "<div>\n  <div>\n    <p>\n      </p><h2>Computer Science &gt; Machine Learning</h2>\n    <p></p>\n    <p><strong>arXiv:2402.03175</strong> (cs)\n    </p>\n<div>\n                <p>View a PDF of the paper titled The Matrix: A Bayesian learning model for LLMs, by Siddhartha Dalal and Vishal Misra</p>\n    <p><a target=\"_blank\" href=\"https://arxiv.org/pdf/2402.03175\">View PDF</a></p><blockquote>\n            <span>Abstract:</span>In this paper, we introduce a Bayesian learning model to understand the behavior of Large Language Models (LLMs). We explore the optimization metric of LLMs, which is based on predicting the next token, and develop a novel model grounded in this principle. Our approach involves constructing an ideal generative text model represented by a multinomial transition probability matrix with a prior, and we examine how LLMs approximate this matrix. We discuss the continuity of the mapping between embeddings and multinomial distributions, and present the Dirichlet approximation theorem to approximate any prior. Additionally, we demonstrate how text generation by LLMs aligns with Bayesian learning principles and delve into the implications for in-context learning, specifically explaining why in-context learning emerges in larger models where prompts are considered as samples to be updated. Our findings indicate that the behavior of LLMs is consistent with Bayesian Learning, offering new insights into their functioning and potential applications.\n    </blockquote>\n  </div>\n    <div>\n      <h2>Submission history</h2><p> From: Vishal Misra [<a target=\"_blank\" href=\"https://arxiv.org/show-email/b5133c63/2402.03175\">view email</a>]      <br />    <strong>[v1]</strong>\n        Mon, 5 Feb 2024 16:42:10 UTC (305 KB)<br />\n</p></div>\n  </div>\n<div>    <div>\n      <p><a></a>\n      <span>Full-text links:</span></p><h2>Access Paper:</h2>\n      <ul>\n  <p>\nView a PDF of the paper titled The Matrix: A Bayesian learning model for LLMs, by Siddhartha Dalal and Vishal Misra</p><li><a target=\"_blank\" href=\"https://arxiv.org/pdf/2402.03175\">View PDF</a></li><li><a target=\"_blank\" href=\"https://arxiv.org/src/2402.03175\">TeX Source</a></li><li><a target=\"_blank\" href=\"https://arxiv.org/format/2402.03175\">Other Formats</a></li></ul>\n    </div>\n        <div><p>\n    Current browse context: </p><p>cs.LG</p>\n    </div>\n<p>\n    <span>export BibTeX citation</span>\n</p>\n<div>\n  <p></p><h3>Bookmark</h3><p></p><p><a target=\"_blank\" href=\"http://www.bibsonomy.org/BibtexHandler?requTask=upload&amp;url=https://arxiv.org/abs/2402.03175&amp;description=The%20Matrix:%20A%20Bayesian%20learning%20model%20for%20LLMs\" title=\"Bookmark on BibSonomy\">\n    <img src=\"https://arxiv.org/static/browse/0.3.4/images/icons/social/bibsonomy.png\" alt=\"BibSonomy logo\" />\n  </a>\n  <a target=\"_blank\" href=\"https://reddit.com/submit?url=https://arxiv.org/abs/2402.03175&amp;title=The%20Matrix:%20A%20Bayesian%20learning%20model%20for%20LLMs\" title=\"Bookmark on Reddit\">\n    <img src=\"https://arxiv.org/static/browse/0.3.4/images/icons/social/reddit.png\" alt=\"Reddit logo\" />\n  </a>\n</p></div>  </div>\n<div><p>\n    <label>Bibliographic Tools</label></p><div>\n      <h2>Bibliographic and Citation Tools</h2>\n      <div>\n          <p><label>\n              <span></span>\n              <span>Bibliographic Explorer Toggle</span>\n            </label>\n          </p>\n        </div>\n    </div>\n    <p>\n    <label>Code, Data, Media</label></p><div>\n      <h2>Code, Data and Media Associated with this Article</h2>\n    </div>\n      <p>\n      <label>Demos</label></p><div>\n        <h2>Demos</h2>\n      </div>\n      <p>\n      <label>Related Papers</label></p><div>\n        <h2>Recommenders and Search Tools</h2>\n        <div>\n            <p><label>\n                <span></span>\n                <span>IArxiv recommender toggle</span>\n              </label>\n            </p>\n          </div>\n      </div>\n      <p>\n      <label>\n        About arXivLabs\n      </label></p><div>\n            <h2>arXivLabs: experimental projects with community collaborators</h2>\n            <p>arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.</p>\n            <p>Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.</p>\n            <p>Have an idea for a project that will add value for arXiv's community? <a target=\"_blank\" href=\"https://info.arxiv.org/labs/index.html\"><strong>Learn more about arXivLabs</strong></a>.</p>\n          </div>\n    </div>\n</div>",
  "author": "",
  "favicon": "https://arxiv.org/static/browse/0.3.4/images/icons/favicon-16x16.png",
  "source": "arxiv.org",
  "published": "",
  "ttr": 69,
  "type": "website"
}