{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Python source\n", "\n", "> Source code for `llms_txt` Python module, containing helpers to create and use llms.txt files" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "#| default_exp core" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "#| export\n", "import re" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "#| hide\n", "from nbdev.showdoc import *\n", "import nbdev; nbdev.nbdev_export()" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "#| export\n", "from fastcore.utils import *\n", "from fastcore.xml import *\n", "from fastcore.script import *\n", "import httpx\n", "from urllib.parse import urlparse" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Introduction" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "%ai reset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The llms.txt file spec is for files located in the path `llms.txt` of a website (or, optionally, in a subpath). `llms-sample.txt` is a simple example. A file following the spec contains the following sections as markdown, in the specific order:\n", "\n", "- An H1 with the name of the project or site. This is the only required section\n", "- A blockquote with a short summary of the project, containing key information necessary for understanding the rest of the file\n", "- Zero or more markdown sections (e.g. paragraphs, lists, etc) of any type, except headings, containing more detailed information about the project and how to interpret the provided files\n", "- Zero or more markdown sections delimited by H2 headers, containing \"file lists\" of URLs where further detail is available\n", " - Each \"file list\" is a markdown list, containing a required markdown hyperlink `[name](url)`, then optionally a `:` and notes about the file.\n", "\n", "Here's the start of a sample llms.txt file we'll use for testing:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "# FastHTML\n", "\n", "> FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore's `FT` \"FastTags\" into a library for creating server-rendered hypermedia applications.\n", "\n", "Remember:\n", "\n", "- Use `serve()` for running uvicorn (`if __name__ == \"__main__\"` is not needed since it's automatic)\n", "- When a title is needed with a response, use `Titled`; note that that already wraps children in `Container`, and already includes both the meta title as well as the H1 element\n" ] } ], "source": [ "samp = Path('llms-sample.txt').read_text()\n", "print(samp[:480])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Reading" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll implement `parse_llms_file` to pull out the sections of llms.txt into a simple data structure." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "#| export\n", "def opt_re(s):\n", " \"Pattern to optionally match `s`\"\n", " return f'(?:{s})?'\n", "\n", "def named_re(nm, pat):\n", " \"Pattern to match `pat` in a named capture group\"\n", " return f'(?P<{nm}>{pat})'\n", "\n", "def search(pat, txt, flags=0):\n", " \"Dictionary of matched groups in `pat` within `txt`\"\n", " res = re.search(pat, txt, flags=flags)\n", " return res.groupdict() if res else None" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll work \"outside in\" so we can test the innermost matches as we go." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Parse links" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "link = '- [FastHTML quick start](https://docs.fastht.ml/tutorials/quickstart_for_web_devs.html.md): A brief overview of FastHTML features'" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "%%aip 0\n", "Parse the first part of `link` into a dict" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'title': 'internal docs - ed'}" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "title = named_re('title', r'[^\\]]+')\n", "pat = fr'-\\s*\\[{title}\\]'\n", "search(pat, samp)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "%%aip 0\n", "Do the next bit." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'title': 'internal docs - ed', 'url': 'https://llmstxt.org/ed.html'}" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "url = named_re('url', r'[^\\)]+')\n", "pat += fr'\\({url}\\)'\n", "search(pat, samp)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "%%aip 0\n", "Do the final bit. Note it's optional." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'title': 'FastHTML quick start',\n", " 'url': 'https://docs.fastht.ml/tutorials/quickstart_for_web_devs.html.md',\n", " 'desc': 'A brief overview of FastHTML features'}" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "desc = named_re('desc', r'.*')\n", "pat += opt_re(fr':\\s*{desc}')\n", "search(pat, link)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "%%aip 0\n", "Combine those sections into a function `parse_link(txt)`" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "#| export\n", "def parse_link(txt):\n", " \"Parse a link section from llms.txt\"\n", " title = named_re('title', r'[^\\]]+')\n", " url = named_re('url', r'[^\\)]+')\n", " desc = named_re('desc', r'.*')\n", " desc_pat = opt_re(fr\":\\s*{desc}\")\n", " pat = fr'-\\s*\\[{title}\\]\\({url}\\){desc_pat}'\n", " return re.search(pat, txt).groupdict()" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'title': 'FastHTML quick start',\n", " 'url': 'https://docs.fastht.ml/tutorials/quickstart_for_web_devs.html.md',\n", " 'desc': 'A brief overview of FastHTML features'}" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "parse_link(link)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'title': 'foo', 'url': 'http://foo', 'desc': None}" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "parse_link('-[foo](http://foo)')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Parse sections" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "sections = '''First bit.\n", "\n", "## S1\n", "\n", "-[foo](http://foo)\n", "- [foo2](http://foo2): stuff\n", "\n", "## S2\n", "\n", "- [foo3](http://foo3)'''" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'First bit.\\n\\n'" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "start,*rest = re.split(fr'^##\\s*(.*?$)', sections, flags=re.MULTILINE)\n", "start" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['S1',\n", " '\\n\\n-[foo](http://foo)\\n- [foo2](http://foo2): stuff\\n\\n',\n", " 'S2',\n", " '\\n\\n- [foo3](http://foo3)']" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "rest" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "%%aip 0\n", "Concisely create a dict from the pairs in `rest`." ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'S1': '\\n\\n-[foo](http://foo)\\n- [foo2](http://foo2): stuff\\n\\n',\n", " 'S2': '\\n\\n- [foo3](http://foo3)'}" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d = dict(chunked(rest, 2))\n", "d" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'-[foo](http://foo)\\n- [foo2](http://foo2): stuff'" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "links = d['S1']\n", "links.strip()" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "%%aip 0\n", "Parse `links` into a list of links. There can be multiple newlines between them." ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "#| export\n", "def _parse_links(links):\n", " return [parse_link(l) for l in re.split(r'\\n+', links.strip()) if l.strip()]" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'title': 'foo', 'url': 'http://foo', 'desc': None},\n", " {'title': 'foo2', 'url': 'http://foo2', 'desc': 'stuff'}]" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "_parse_links(links)" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [], "source": [ "%%aip 0\n", "Create a function that uses the above steps to parse an llms.txt into `start` and a dict with keys like `d` and parsed list of links as values." ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [], "source": [ "#| export\n", "def _parse_llms(txt):\n", " start,*rest = re.split(fr'^##\\s*(.*?$)', txt, flags=re.MULTILINE)\n", " d = dict(chunked(rest, 2))\n", " sects = {k: _parse_links(v) for k,v in d.items()}\n", " return start.strip(),sects" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'# FastHTML\\n\\n> FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore\\'s `FT` \"FastTags\" into a library for creating server-rendered hypermedia applications.\\n\\nRemember:\\n\\n- Use `serve()` for running uvicorn (`if __name__ == \"__main__\"` is not needed since it\\'s automatic)\\n- When a title is needed with a response, use `Titled`; note that that already wraps children in `Container`, and already includes both the meta title as well as the H1 element.'" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "start, sects = _parse_llms(samp)\n", "start" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [], "source": [ "title = named_re('title', r'.+?$')\n", "summ = named_re('summary', '.+?$')\n", "summ_pat = opt_re(fr\"^>\\s*{summ}$\")\n", "info = named_re('info', '.*')" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'title': 'FastHTML',\n", " 'summary': 'FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore\\'s `FT` \"FastTags\" into a library for creating server-rendered hypermedia applications.',\n", " 'info': 'Remember:\\n\\n- Use `serve()` for running uvicorn (`if __name__ == \"__main__\"` is not needed since it\\'s automatic)\\n- When a title is needed with a response, use `Titled`; note that that already wraps children in `Container`, and already includes both the meta title as well as the H1 element.'}" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pat = fr'^#\\s*{title}\\n+{summ_pat}\\n+{info}'\n", "search(pat, start, (re.MULTILINE|re.DOTALL))" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [], "source": [ "%%aip 0\n", "Let's finish it off!" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [], "source": [ "#| export\n", "def parse_llms_file(txt):\n", " \"Parse llms.txt file contents in `txt` to an `AttrDict`\"\n", " start,sects = _parse_llms(txt)\n", " title = named_re('title', r'.+?$')\n", " summ = named_re('summary', '.+?$')\n", " summ_pat = opt_re(fr\"^>\\s*{summ}$\")\n", " info = named_re('info', '.*')\n", " pat = fr'^#\\s*{title}\\n+{summ_pat}\\n+{info}'\n", " d = search(pat, start, (re.MULTILINE|re.DOTALL))\n", " d['sections'] = sects\n", " return dict2obj(d)" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore\\'s `FT` \"FastTags\" into a library for creating server-rendered hypermedia applications.'" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "llmsd = parse_llms_file(samp)\n", "llmsd.summary" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(#1) [{'title': 'Todo list application', 'url': 'https://raw.githubusercontent.com/AnswerDotAI/fasthtml/main/examples/adv_app.py', 'desc': 'Detailed walk-thru of a complete CRUD app in FastHTML showing idiomatic use of FastHTML and HTMX patterns.'}]" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "llmsd.sections.Examples" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## XML conversion" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For some LLMs such as Claude, XML format is preferred, so we'll provide a function to create that format." ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [], "source": [ "#| export\n", "from fastcore.xml import Sections,Project,Doc" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [], "source": [ "#|export\n", "def _local_docs_pth(cfg): return cfg.config_path/'_proc'/cfg.doc_path\n", "def _get_config(): return Config.find('settings.ini')\n", "\n", "def get_doc_content(url):\n", " \"Fetch content from local file if in nbdev repo.\"\n", " if (cfg:=_get_config()) and url.startswith(cfg.doc_host):\n", " relative_path = urlparse(url).path.lstrip('/')\n", " local_path = _local_docs_pth(cfg) / relative_path\n", " if local_path.exists(): return local_path.read_text()\n", " return httpx.get(url).text" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [], "source": [ "#| export\n", "def _doc(kw):\n", " \"Create a `Doc` FT object with the text retrieved from `url` as the child, and `kw` as attrs.\"\n", " url = kw.pop('url')\n", " txt = get_doc_content(url)\n", " re_comment = re.compile('^$', flags=re.MULTILINE)\n", " re_base64_img = re.compile(r']*src=\"data:image/[^\"]*\"[^>]*>')\n", " txt = '\\n'.join([o for o in txt.splitlines() if not re_comment.search(o) and not re_base64_img.search(o)])\n", " return Doc(txt, **kw)" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [], "source": [ "#| export\n", "def _section(nm, items, n_workers=None):\n", " \"Create a section containing a `Doc` object for each child.\"\n", " return ft(nm, *parallel(_doc, items, n_workers=n_workers, threadpool=True))" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [], "source": [ "#| export\n", "def mk_ctx(d, optional=True, n_workers=None):\n", " \"Create a `Project` with a `Section` for each H2 part in `d`, optionally skipping the 'optional' section.\"\n", " skip = '' if optional else 'Optional'\n", " sections = [_section(k, v, n_workers=n_workers) for k,v in d.sections.items() if k!=skip]\n", " return Project(title=d.title, summary=d.summary)(d.info, *sections)" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Remember:\n", "\n", "- Use `serve()` for running uvic...\n" ] } ], "source": [ "ctx = mk_ctx(llmsd)\n", "print(to_xml(ctx, do_escape=False)[:260]+'...')" ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [], "source": [ "#| export\n", "def get_sizes(ctx):\n", " \"Get the size of each section of the LLM context\"\n", " return {o.tag:{p.title:len(p.children[0]) for p in o.children} for o in ctx.children if hasattr(o,'tag')}" ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'docs': {'internal docs - ed': 34464,\n", " 'FastHTML quick start': 27383,\n", " 'HTMX reference': 26812,\n", " 'Starlette quick guide': 7936},\n", " 'examples': {'Todo list application': 18558},\n", " 'optional': {'Starlette full documentation': 48331}}" ] }, "execution_count": 44, "metadata": {}, "output_type": "execute_result" } ], "source": [ "get_sizes(ctx)" ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "164662" ] }, "execution_count": 45, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Path('../fasthtml.md').write_text(to_xml(ctx, do_escape=False))" ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [], "source": [ "#| export\n", "def create_ctx(txt, optional=False, n_workers=None):\n", " \"A `Project` with a `Section` for each H2 part in `txt`, optionally skipping the 'optional' section.\"\n", " d = parse_llms_file(txt)\n", " ctx = mk_ctx(d, optional=optional, n_workers=n_workers)\n", " return to_xml(ctx, do_escape=False)" ] }, { "cell_type": "code", "execution_count": 47, "metadata": {}, "outputs": [], "source": [ "#| export\n", "@call_parse\n", "def llms_txt2ctx(\n", " fname:str, # File name to read\n", " optional:bool_arg=False, # Include 'optional' section?\n", " n_workers:int=None, # Number of threads to use for parallel downloading\n", " save_nbdev_fname:str=None #save output to nbdev `{docs_path}` instead of emitting to stdout\n", "):\n", " \"Print a `Project` with a `Section` for each H2 part in file read from `fname`, optionally skipping the 'optional' section.\"\n", " ctx = create_ctx(Path(fname).read_text(), optional=optional, n_workers=n_workers)\n", " if save_nbdev_fname and (cfg:=_get_config()):\n", " (_local_docs_pth(cfg) / save_nbdev_fname).mk_write(ctx)\n", " else: print(ctx)" ] }, { "cell_type": "code", "execution_count": 48, "metadata": {}, "outputs": [], "source": [ "!llms_txt2ctx llms-sample.txt > ../fasthtml.md" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Export -" ] }, { "cell_type": "code", "execution_count": 49, "metadata": {}, "outputs": [], "source": [ "#|hide\n", "#|eval: false\n", "from nbdev import nbdev_export\n", "nbdev_export()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" } }, "nbformat": 4, "nbformat_minor": 4 }