# Python source


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

## Introduction

The llms.txt file spec is for files located in the path `llms.txt` of a
website (or, optionally, in a subpath). `llms-sample.txt` is a simple
example. A file following the spec contains the following sections as
markdown, in the specific order:

- An H1 with the name of the project or site. This is the only required
  section
- A blockquote with a short summary of the project, containing key
  information necessary for understanding the rest of the file
- Zero or more markdown sections (e.g. paragraphs, lists, etc) of any
  type, except headings, containing more detailed information about the
  project and how to interpret the provided files
- Zero or more markdown sections delimited by H2 headers, containing
  “file lists” of URLs where further detail is available
  - Each “file list” is a markdown list, containing a required markdown
    hyperlink `[name](url)`, then optionally a `:` and notes about the
    file.

Here’s the start of a sample llms.txt file we’ll use for testing:

``` python
samp = Path('llms-sample.txt').read_text()
print(samp[:480])
```

    # FastHTML

    > FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore's `FT` "FastTags" into a library for creating server-rendered hypermedia applications.

    Remember:

    - Use `serve()` for running uvicorn (`if __name__ == "__main__"` is not needed since it's automatic)
    - When a title is needed with a response, use `Titled`; note that that already wraps children in `Container`, and already includes both the meta title as well as the H1 element

## Reading

We’ll implement `parse_llms_file` to pull out the sections of llms.txt
into a simple data structure.

------------------------------------------------------------------------

### search

``` python
def search(
    pat, txt, flags:int=0
):
```

*Dictionary of matched groups in `pat` within `txt`*

------------------------------------------------------------------------

### named_re

``` python
def named_re(
    nm, pat
):
```

*Pattern to match `pat` in a named capture group*

------------------------------------------------------------------------

### opt_re

``` python
def opt_re(
    s
):
```

*Pattern to optionally match `s`*

We’ll work “outside in” so we can test the innermost matches as we go.

### Parse links

``` python
link = '- [FastHTML quick start](https://fastht.ml/docs/tutorials/quickstart_for_web_devs.html.md): A brief overview of FastHTML features'
```

``` python
title = named_re('title', r'[^\]]+')
pat =  fr'-\s*\[{title}\]'
search(pat, samp)
```

    {'title': 'FastHTML quick start'}

``` python
url = named_re('url', r'[^\)]+')
pat += fr'\({url}\)'
search(pat, samp)
```

    {'title': 'FastHTML quick start',
     'url': 'https://fastht.ml/docs/tutorials/quickstart_for_web_devs.html.md'}

``` python
desc = named_re('desc', r'.*')
pat += opt_re(fr':\s*{desc}')
search(pat, link)
```

    {'title': 'FastHTML quick start',
     'url': 'https://fastht.ml/docs/tutorials/quickstart_for_web_devs.html.md',
     'desc': 'A brief overview of FastHTML features'}

------------------------------------------------------------------------

### parse_link

``` python
def parse_link(
    txt
):
```

*Parse a link section from llms.txt*

``` python
parse_link(link)
```

    {'title': 'FastHTML quick start',
     'url': 'https://fastht.ml/docs/tutorials/quickstart_for_web_devs.html.md',
     'desc': 'A brief overview of FastHTML features'}

``` python
parse_link('-[foo](http://foo)')
```

    {'title': 'foo', 'url': 'http://foo', 'desc': None}

### Parse sections

``` python
sections = '''First bit.

## S1

-[foo](http://foo)
- [foo2](http://foo2): stuff

## S2

- [foo3](http://foo3)'''
```

``` python
start,*rest = re.split(fr'^##\s*(.*?$)', sections, flags=re.MULTILINE)
start
```

    'First bit.\n\n'

``` python
rest
```

    ['S1',
     '\n\n-[foo](http://foo)\n- [foo2](http://foo2): stuff\n\n',
     'S2',
     '\n\n- [foo3](http://foo3)']

``` python
d = dict(chunked(rest, 2))
d
```

    {'S1': '\n\n-[foo](http://foo)\n- [foo2](http://foo2): stuff\n\n',
     'S2': '\n\n- [foo3](http://foo3)'}

``` python
links = d['S1']
links.strip()
```

    '-[foo](http://foo)\n- [foo2](http://foo2): stuff'

``` python
_parse_links(links)
```

    [{'title': 'foo', 'url': 'http://foo', 'desc': None},
     {'title': 'foo2', 'url': 'http://foo2', 'desc': 'stuff'}]

``` python
start, sects = _parse_llms(samp)
start
```

    '# FastHTML\n\n> FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore\'s `FT` "FastTags" into a library for creating server-rendered hypermedia applications.\n\nRemember:\n\n- Use `serve()` for running uvicorn (`if __name__ == "__main__"` is not needed since it\'s automatic)\n- When a title is needed with a response, use `Titled`; note that that already wraps children in `Container`, and already includes both the meta title as well as the H1 element.'

``` python
title = named_re('title', r'.+?$')
summ = named_re('summary', '.+?$')
summ_pat = opt_re(fr"^>\s*{summ}$")
info = named_re('info', '.*')
```

``` python
pat = fr'^#\s*{title}\n+{summ_pat}\n+{info}'
search(pat, start, (re.MULTILINE|re.DOTALL))
```

    {'title': 'FastHTML',
     'summary': 'FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore\'s `FT` "FastTags" into a library for creating server-rendered hypermedia applications.',
     'info': 'Remember:\n\n- Use `serve()` for running uvicorn (`if __name__ == "__main__"` is not needed since it\'s automatic)\n- When a title is needed with a response, use `Titled`; note that that already wraps children in `Container`, and already includes both the meta title as well as the H1 element.'}

------------------------------------------------------------------------

### parse_llms_file

``` python
def parse_llms_file(
    txt
):
```

*Parse llms.txt file contents in `txt` to an `AttrDict`*

``` python
llmsd = parse_llms_file(samp)
llmsd.summary
```

    'FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore\'s `FT` "FastTags" into a library for creating server-rendered hypermedia applications.'

``` python
llmsd.sections.Examples
```

    [{'title': 'Todo list application', 'url': 'https://raw.githubusercontent.com/AnswerDotAI/fasthtml/main/examples/adv_app.py', 'desc': 'Detailed walk-thru of a complete CRUD app in FastHTML showing idiomatic use of FastHTML and HTMX patterns.'}]

## XML conversion

For some LLMs such as Claude, XML format is preferred, so we’ll provide
a function to create that format.

------------------------------------------------------------------------

### get_doc_content

``` python
def get_doc_content(
    url
):
```

*Fetch content from local file if in nbdev repo.*

------------------------------------------------------------------------

### mk_ctx

``` python
def mk_ctx(
    d, optional:bool=True, n_workers:NoneType=None
):
```

*Create a `Project` with a `Section` for each H2 part in `d`, optionally
skipping the ‘optional’ section.*

``` python
ctx = mk_ctx(llmsd)
print(to_xml(ctx, do_escape=False)[:260]+'...')
```

    {'title': 'FastHTML quick start', 'url': 'https://fastht.ml/docs/tutorials/quickstart_for_web_devs.html.md', 'desc': 'A brief overview of FastHTML features'}{'title': 'HTMX reference', 'url': 'https://raw.githubusercontent.com/bigskysoftware/htmx/master/www/content/reference.md', 'desc': 'Brief description of all HTMX attributes, CSS classes, headers, events, extensions, js lib methods, and config options'}

    {'title': 'Starlette quick guide', 'url': 'https://gist.githubusercontent.com/jph00/e91192e9bdc1640f5421ce3c904f2efb/raw/61a2774912414029edaf1a55b506f0e283b93c46/starlette-quick.md', 'desc': {}}
    {'title': 'Todo list application', 'url': 'https://raw.githubusercontent.com/AnswerDotAI/fasthtml/main/examples/adv_app.py', 'desc': 'Detailed walk-thru of a complete CRUD app in FastHTML showing idiomatic use of FastHTML and HTMX patterns.'}
    {'title': 'Starlette full documentation', 'url': 'https://gist.githubusercontent.com/jph00/809e4a4808d4510be0e3dc9565e9cbd3/raw/9b717589ca44cedc8aaf00b2b8cacef922964c0f/starlette-sml.md', 'desc': 'A subset of the Starlette documentation useful for FastHTML development.'}
    <project title="FastHTML" summary='FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore&#39;s `FT` "FastTags" into a library for creating server-rendered hypermedia applications.'>Remember:

    - Use `serve()` for running uvic...

------------------------------------------------------------------------

### get_sizes

``` python
def get_sizes(
    ctx
):
```

*Get the size of each section of the LLM context*

``` python
get_sizes(ctx)
```

    {'docs': {'FastHTML quick start': 35321,
      'HTMX reference': 28365,
      'Starlette quick guide': 7936},
     'examples': {'Todo list application': 16032},
     'optional': {'Starlette full documentation': 48331}}

``` python
Path('../fasthtml.md').write_text(to_xml(ctx, do_escape=False))
```

    137125

------------------------------------------------------------------------

### create_ctx

``` python
def create_ctx(
    txt, optional:bool=False, n_workers:NoneType=None
):
```

*A `Project` with a `Section` for each H2 part in `txt`, optionally
skipping the ‘optional’ section.*

------------------------------------------------------------------------

### llms_txt2ctx

``` python
def llms_txt2ctx(
    fname:str, # File name to read
    optional:bool_arg=False, # Include 'optional' section?
    n_workers:int=None, # Number of threads to use for parallel downloading
    save_nbdev_fname:str=None, # save output to nbdev `{docs_path}` instead of emitting to stdout
):
```

*Print a `Project` with a `Section` for each H2 part in file read from
`fname`, optionally skipping the ‘optional’ section.*

``` python
!llms_txt2ctx llms-sample.txt > ../fasthtml.md
```