912 lines
23 KiB
Text
912 lines
23 KiB
Text
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Python source\n",
|
|
"\n",
|
|
"> Source code for `llms_txt` Python module, containing helpers to create and use llms.txt files"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| default_exp core"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"import re"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| hide\n",
|
|
"from nbdev.showdoc import *\n",
|
|
"import nbdev; nbdev.nbdev_export()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"from fastcore.utils import *\n",
|
|
"from fastcore.xml import *\n",
|
|
"from fastcore.script import *\n",
|
|
"import httpx\n",
|
|
"from urllib.parse import urlparse"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Introduction"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%ai reset"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"The llms.txt file spec is for files located in the path `llms.txt` of a website (or, optionally, in a subpath). `llms-sample.txt` is a simple example. A file following the spec contains the following sections as markdown, in the specific order:\n",
|
|
"\n",
|
|
"- An H1 with the name of the project or site. This is the only required section\n",
|
|
"- A blockquote with a short summary of the project, containing key information necessary for understanding the rest of the file\n",
|
|
"- Zero or more markdown sections (e.g. paragraphs, lists, etc) of any type, except headings, containing more detailed information about the project and how to interpret the provided files\n",
|
|
"- Zero or more markdown sections delimited by H2 headers, containing \"file lists\" of URLs where further detail is available\n",
|
|
" - Each \"file list\" is a markdown list, containing a required markdown hyperlink `[name](url)`, then optionally a `:` and notes about the file.\n",
|
|
"\n",
|
|
"Here's the start of a sample llms.txt file we'll use for testing:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"# FastHTML\n",
|
|
"\n",
|
|
"> FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore's `FT` \"FastTags\" into a library for creating server-rendered hypermedia applications.\n",
|
|
"\n",
|
|
"Remember:\n",
|
|
"\n",
|
|
"- Use `serve()` for running uvicorn (`if __name__ == \"__main__\"` is not needed since it's automatic)\n",
|
|
"- When a title is needed with a response, use `Titled`; note that that already wraps children in `Container`, and already includes both the meta title as well as the H1 element\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"samp = Path('llms-sample.txt').read_text()\n",
|
|
"print(samp[:480])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Reading"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"We'll implement `parse_llms_file` to pull out the sections of llms.txt into a simple data structure."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"def opt_re(s):\n",
|
|
" \"Pattern to optionally match `s`\"\n",
|
|
" return f'(?:{s})?'\n",
|
|
"\n",
|
|
"def named_re(nm, pat):\n",
|
|
" \"Pattern to match `pat` in a named capture group\"\n",
|
|
" return f'(?P<{nm}>{pat})'\n",
|
|
"\n",
|
|
"def search(pat, txt, flags=0):\n",
|
|
" \"Dictionary of matched groups in `pat` within `txt`\"\n",
|
|
" res = re.search(pat, txt, flags=flags)\n",
|
|
" return res.groupdict() if res else None"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"We'll work \"outside in\" so we can test the innermost matches as we go."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Parse links"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"link = '- [FastHTML quick start](https://docs.fastht.ml/tutorials/quickstart_for_web_devs.html.md): A brief overview of FastHTML features'"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%%aip 0\n",
|
|
"Parse the first part of `link` into a dict"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'title': 'FastHTML quick start'}"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"title = named_re('title', r'[^\\]]+')\n",
|
|
"pat = fr'-\\s*\\[{title}\\]'\n",
|
|
"search(pat, samp)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%%aip 0\n",
|
|
"Do the next bit."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'title': 'FastHTML quick start',\n",
|
|
" 'url': 'https://docs.fastht.ml/tutorials/quickstart_for_web_devs.html.md'}"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"url = named_re('url', r'[^\\)]+')\n",
|
|
"pat += fr'\\({url}\\)'\n",
|
|
"search(pat, samp)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%%aip 0\n",
|
|
"Do the final bit. Note it's optional."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'title': 'FastHTML quick start',\n",
|
|
" 'url': 'https://docs.fastht.ml/tutorials/quickstart_for_web_devs.html.md',\n",
|
|
" 'desc': 'A brief overview of FastHTML features'}"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"desc = named_re('desc', r'.*')\n",
|
|
"pat += opt_re(fr':\\s*{desc}')\n",
|
|
"search(pat, link)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%%aip 0\n",
|
|
"Combine those sections into a function `parse_link(txt)`"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"def parse_link(txt):\n",
|
|
" \"Parse a link section from llms.txt\"\n",
|
|
" title = named_re('title', r'[^\\]]+')\n",
|
|
" url = named_re('url', r'[^\\)]+')\n",
|
|
" desc = named_re('desc', r'.*')\n",
|
|
" desc_pat = opt_re(fr\":\\s*{desc}\")\n",
|
|
" pat = fr'-\\s*\\[{title}\\]\\({url}\\){desc_pat}'\n",
|
|
" return re.search(pat, txt).groupdict()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'title': 'FastHTML quick start',\n",
|
|
" 'url': 'https://docs.fastht.ml/tutorials/quickstart_for_web_devs.html.md',\n",
|
|
" 'desc': 'A brief overview of FastHTML features'}"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"parse_link(link)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'title': 'foo', 'url': 'http://foo', 'desc': None}"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"parse_link('-[foo](http://foo)')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Parse sections"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"sections = '''First bit.\n",
|
|
"\n",
|
|
"## S1\n",
|
|
"\n",
|
|
"-[foo](http://foo)\n",
|
|
"- [foo2](http://foo2): stuff\n",
|
|
"\n",
|
|
"## S2\n",
|
|
"\n",
|
|
"- [foo3](http://foo3)'''"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"'First bit.\\n\\n'"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"start,*rest = re.split(fr'^##\\s*(.*?$)', sections, flags=re.MULTILINE)\n",
|
|
"start"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"['S1',\n",
|
|
" '\\n\\n-[foo](http://foo)\\n- [foo2](http://foo2): stuff\\n\\n',\n",
|
|
" 'S2',\n",
|
|
" '\\n\\n- [foo3](http://foo3)']"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"rest"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%%aip 0\n",
|
|
"Concisely create a dict from the pairs in `rest`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'S1': '\\n\\n-[foo](http://foo)\\n- [foo2](http://foo2): stuff\\n\\n',\n",
|
|
" 'S2': '\\n\\n- [foo3](http://foo3)'}"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"d = dict(chunked(rest, 2))\n",
|
|
"d"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"'-[foo](http://foo)\\n- [foo2](http://foo2): stuff'"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"links = d['S1']\n",
|
|
"links.strip()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%%aip 0\n",
|
|
"Parse `links` into a list of links. There can be multiple newlines between them."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"def _parse_links(links):\n",
|
|
" return [parse_link(l) for l in re.split(r'\\n+', links.strip()) if l.strip()]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"[{'title': 'foo', 'url': 'http://foo', 'desc': None},\n",
|
|
" {'title': 'foo2', 'url': 'http://foo2', 'desc': 'stuff'}]"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"_parse_links(links)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%%aip 0\n",
|
|
"Create a function that uses the above steps to parse an llms.txt into `start` and a dict with keys like `d` and parsed list of links as values."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"def _parse_llms(txt):\n",
|
|
" start,*rest = re.split(fr'^##\\s*(.*?$)', txt, flags=re.MULTILINE)\n",
|
|
" d = dict(chunked(rest, 2))\n",
|
|
" sects = {k: _parse_links(v) for k,v in d.items()}\n",
|
|
" return start.strip(),sects"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"'# FastHTML\\n\\n> FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore\\'s `FT` \"FastTags\" into a library for creating server-rendered hypermedia applications.\\n\\nRemember:\\n\\n- Use `serve()` for running uvicorn (`if __name__ == \"__main__\"` is not needed since it\\'s automatic)\\n- When a title is needed with a response, use `Titled`; note that that already wraps children in `Container`, and already includes both the meta title as well as the H1 element.'"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"start, sects = _parse_llms(samp)\n",
|
|
"start"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"title = named_re('title', r'.+?$')\n",
|
|
"summ = named_re('summary', '.+?$')\n",
|
|
"summ_pat = opt_re(fr\"^>\\s*{summ}$\")\n",
|
|
"info = named_re('info', '.*')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'title': 'FastHTML',\n",
|
|
" 'summary': 'FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore\\'s `FT` \"FastTags\" into a library for creating server-rendered hypermedia applications.',\n",
|
|
" 'info': 'Remember:\\n\\n- Use `serve()` for running uvicorn (`if __name__ == \"__main__\"` is not needed since it\\'s automatic)\\n- When a title is needed with a response, use `Titled`; note that that already wraps children in `Container`, and already includes both the meta title as well as the H1 element.'}"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"pat = fr'^#\\s*{title}\\n+{summ_pat}\\n+{info}'\n",
|
|
"search(pat, start, (re.MULTILINE|re.DOTALL))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%%aip 0\n",
|
|
"Let's finish it off!"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"def parse_llms_file(txt):\n",
|
|
" \"Parse llms.txt file contents in `txt` to an `AttrDict`\"\n",
|
|
" start,sects = _parse_llms(txt)\n",
|
|
" title = named_re('title', r'.+?$')\n",
|
|
" summ = named_re('summary', '.+?$')\n",
|
|
" summ_pat = opt_re(fr\"^>\\s*{summ}$\")\n",
|
|
" info = named_re('info', '.*')\n",
|
|
" pat = fr'^#\\s*{title}\\n+{summ_pat}\\n+{info}'\n",
|
|
" d = search(pat, start, (re.MULTILINE|re.DOTALL))\n",
|
|
" d['sections'] = sects\n",
|
|
" return dict2obj(d)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"'FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore\\'s `FT` \"FastTags\" into a library for creating server-rendered hypermedia applications.'"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"llmsd = parse_llms_file(samp)\n",
|
|
"llmsd.summary"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"(#1) [{'title': 'Todo list application', 'url': 'https://raw.githubusercontent.com/AnswerDotAI/fasthtml/main/examples/adv_app.py', 'desc': 'Detailed walk-thru of a complete CRUD app in FastHTML showing idiomatic use of FastHTML and HTMX patterns.'}]"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"llmsd.sections.Examples"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## XML conversion"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"For some LLMs such as Claude, XML format is preferred, so we'll provide a function to create that format."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"from fastcore.xml import Sections,Project,Doc"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"def find_root_dir():\n",
|
|
" \"Find the root directory of the nbdev project by looking for settings.ini\"\n",
|
|
" path = Path.cwd()\n",
|
|
" while path != path.parent:\n",
|
|
" if (path / 'settings.ini').exists(): return path\n",
|
|
" path = path.parent\n",
|
|
" return None"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#|export\n",
|
|
"def get_doc_content(url):\n",
|
|
" \"Fetch content from local file if in nbdev repo.\"\n",
|
|
" root_dir = find_root_dir()\n",
|
|
" if root_dir:\n",
|
|
" config = Config(root_dir, 'settings.ini')\n",
|
|
" doc_host = config.get('doc_host')\n",
|
|
" if doc_host and url.startswith(doc_host):\n",
|
|
" parsed_url = urlparse(url)\n",
|
|
" relative_path = parsed_url.path.lstrip('/')\n",
|
|
" local_path = root_dir / '_docs' / relative_path\n",
|
|
" if local_path.exists():\n",
|
|
" with open(local_path, 'r') as f: return f.read()\n",
|
|
" # If not a local file or file doesn't exist, fetch from URL\n",
|
|
" return httpx.get(url).text"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"def _doc(kw):\n",
|
|
" \"Create a `Doc` FT object with the text retrieved from `url` as the child, and `kw` as attrs.\"\n",
|
|
" url = kw.pop('url')\n",
|
|
" txt = get_doc_content(url)\n",
|
|
" re_comment = re.compile('^<!--.*-->$', flags=re.MULTILINE)\n",
|
|
" re_base64_img = re.compile(r'<img[^>]*src=\"data:image/[^\"]*\"[^>]*>')\n",
|
|
" txt = '\\n'.join([o for o in txt.splitlines() if not re_comment.search(o) and not re_base64_img.search(o)])\n",
|
|
" return Doc(txt, **kw)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"def _section(nm, items, n_workers=None):\n",
|
|
" \"Create a section containing a `Doc` object for each child.\"\n",
|
|
" return ft(nm, *parallel(_doc, items, n_workers=n_workers, threadpool=True))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"def mk_ctx(d, optional=True, n_workers=None):\n",
|
|
" \"Create a `Project` with a `Section` for each H2 part in `d`, optionally skipping the 'optional' section.\"\n",
|
|
" skip = '' if optional else 'Optional'\n",
|
|
" sections = [_section(k, v, n_workers=n_workers) for k,v in d.sections.items() if k!=skip]\n",
|
|
" return Project(title=d.title, summary=d.summary)(d.info, *sections)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"<project title=\"FastHTML\" summary='FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore's `FT` \"FastTags\" into a library for creating server-rendered hypermedia applications.'>\n",
|
|
"Remember:\n",
|
|
"\n",
|
|
"- Use `serve()` for running uvi...\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"ctx = mk_ctx(llmsd)\n",
|
|
"print(to_xml(ctx, do_escape=False)[:260]+'...')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"def get_sizes(ctx):\n",
|
|
" \"Get the size of each section of the LLM context\"\n",
|
|
" return {o.tag:{p.title:len(p.children[0]) for p in o.children} for o in ctx.children if hasattr(o,'tag')}"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'docs': {'FastHTML quick start': 27376,\n",
|
|
" 'HTMX reference': 26427,\n",
|
|
" 'Starlette quick guide': 7936},\n",
|
|
" 'examples': {'Todo list application': 18558},\n",
|
|
" 'optional': {'Starlette full documentation': 48331}}"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"get_sizes(ctx)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"129814"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"Path('../fasthtml.md').write_text(to_xml(ctx, do_escape=False))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"def create_ctx(txt, optional=False, n_workers=None):\n",
|
|
" \"A `Project` with a `Section` for each H2 part in `txt`, optionally skipping the 'optional' section.\"\n",
|
|
" d = parse_llms_file(txt)\n",
|
|
" ctx = mk_ctx(d, optional=optional, n_workers=n_workers)\n",
|
|
" return to_xml(ctx, do_escape=False)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"@call_parse\n",
|
|
"def llms_txt2ctx(\n",
|
|
" fname:str, # File name to read\n",
|
|
" optional:bool_arg=False, # Include 'optional' section?\n",
|
|
" n_workers:int=None # Number of threads to use for parallel downloading\n",
|
|
"):\n",
|
|
" \"Print a `Project` with a `Section` for each H2 part in file read from `fname`, optionally skipping the 'optional' section.\"\n",
|
|
" print(create_ctx(Path(fname).read_text(), optional=optional, n_workers=n_workers))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!llms_txt2ctx llms-sample.txt > ../fasthtml.md"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Export -"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#|hide\n",
|
|
"#|eval: false\n",
|
|
"from nbdev import nbdev_export\n",
|
|
"nbdev_export()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "python3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 4
|
|
}
|