Merge pull request #163 from niklasmohrin/hyperfine

2026-08-09 09:49:10 +02:00 · 2021-07-30 15:46:03 +02:00 · 2021-07-30 15:46:03 +02:00 · 123c809630
commit 123c809630
parent 187214e39e d1d36a961a
2 changed files with 180 additions and 31 deletions
--- a/README.md
+++ b/README.md
@ -31,38 +31,37 @@ High level project goals:

 - [x] Download and cache pages
 - [x] Don't require a network connection for anything besides updating the cache
- [x] Command line interface similar or equivalent to the [NodeJS client][tldr-node-client]
+- [x] Command line interface similar or equivalent to the [NodeJS client][node-gh]
+- [x] Comply with the [tldr client specification][client-spec]
+- [x] Advanced highlighting and configuration
 - [x] Be fast

-A tool like `tldr` should be as frictionless as possible to use. It should be
-easy to invoke (just `tldr tar`, not using another subcommand like `tldr find
-tar`) and it should show the output as fast as possible.
+A tool like `tldr` should be as frictionless as possible to use and show the
+output as fast as possible.

-tealdeer reaches these goals. During a (highly non-scientific) test (see
-[#38](https://github.com/dbrgn/tealdeer/issues/38) for details), I tested the
-invocation speed of `tldr <command>` for a few of the existing clients:
+We think that `tealdeer` reaches these goals. We put together a (more or less)
+reproducible benchmark that compiles a handful of clients from source and
+measures the execution times on a cold disk cache. The benchmarking is run in a
+Docker container using sharkdp's [`hyperfine`][hyperfine-gh]
+([Dockerfile][benchmark-dockerfile]).

-| Client | Times (ms) | Avg of 5 (ms) |
-| --- | --- | --- |
-| [Tealdeer](https://github.com/dbrgn/tealdeer/) | `15/11/5/5/11` | `9.4` (100%) |
-| [C client](https://github.com/tldr-pages/tldr-cpp-client) | `11/5/12/11/15` | `10.8` (115%) |
-| [Bash client](https://github.com/pepa65/tldr-bash-client) | `15/19/22/25/24` | `21.0` (223%) |
-| [Go client by k3mist](https://github.com/k3mist/tldr/) | `98/96/100/95/101` | `98.8` (1'051%) |
-| [Python client](https://github.com/lord63/tldr.py) | `152/148/151/158/140` | `149.8` (1'594%) |
-| [NodeJS client](https://github.com/tldr-pages/tldr-node-client) | `169/171/170/170/170` | `170.0` (1'809%) |
-
-tealdeer was the winner here, although the C client and the Bash client are in
-the same speed class. Interpreted languages are clearly much slower to invoke,
-a delay of 170 milliseconds is definitely noticeable and increases friction for
-the user.
-
-These are the clients I tried but failed to compile or run:
-[Haskell client](https://github.com/psibi/tldr-hs),
-[Ruby client](https://github.com/YellowApple/tldrb),
-[Perl client](https://github.com/skaji/perl-tldr),
-[Go client by anoopengineer](https://github.com/anoopengineer/tldr/),
-[PHP client](https://github.com/BrainMaestro/tldr-php).
+| Client (50 runs, 29.07.2021)      | Programming Language | Mean in ms | Deviation in ms | Comments                |
+| :---:                             | :---:                | :---:      | :---:           | :---:                   |
+| `tealdeer`                        | Rust                 | 16.2       | 1.2             |                         |
+| [`tldr-c`][c-gh]                  | C                    | 35.7       | 3.2             |                         |
+| [`tldr-hs`][hs-gh]                | Haskell              | 23.6       | 1.0             | no example highlighting |
+| [`fast-tldr`][fast-tldr-gh]       | Haskell              | 15.2       | 0.7             | no example highlighting |
+| [`tldr-bash`][bash-gh]            | Bash                 | 28.3       | 1.1             |                         |
+| [`tldr-python-client`][python-gh] | Python               | 97.9       | 3.5             |                         |
+| [`outfieldr`][outfieldr-gh]       | Zig                  | 4.7        | 0.5             | no user configuration   |
+| [`tldr-node-client`][node-gh]     | Javascript / Node    | 533.1      | 12.5            |                         |

+As you can see, `tealdeer` is not the fastest of the tested clients anymore. We
+strive for useful features and code quality over raw performance, even if that
+means that we don't come out on top in this friendly competition. That said, we
+are still optimizing the code, for example when the `outfieldr` developers
+[suggested to switch][outfieldr-comment-tls] to a native TLS implementation
+instead of the native libraries.

 ## Development

@ -113,7 +112,18 @@ be dual licensed as above, without any additional terms or conditions.
 Thanks to @severen for coming up with the name "tealdeer"!


-[tldr-node-client]: https://github.com/tldr-pages/tldr-node-client
+[node-gh]: https://github.com/tldr-pages/tldr-node-client
+[c-gh]: https://github.com/tldr-pages/tldr-c-client
+[hs-gh]: https://github.com/psibi/tldr-hs
+[fast-tldr-gh]: https://github.com/gutjuri/fast-tldr
+[bash-gh]: https://4e4.win/tldr
+[outfieldr-gh]: https://gitlab.com/ve-nt/outfieldr
+[python-gh]: https://github.com/tldr-pages/tldr-python-client
+
+[benchmark-dockerfile]: https://github.com/dbrgn/tealdeer/blob/master/benchmarks/Dockerfile
+[client-spec]: https://github.com/tldr-pages/tldr/blob/main/CLIENT-SPECIFICATION.md
+[hyperfine-gh]: https://github.com/sharkdp/hyperfine
+[outfieldr-comment-tls]: https://github.com/dbrgn/tealdeer/issues/129#issuecomment-833596765

 <!-- Badges -->
 [github-actions]: https://github.com/dbrgn/tealdeer/actions?query=branch%3Amaster
--- a/benchmarks/Dockerfile
+++ b/benchmarks/Dockerfile
@ -0,0 +1,139 @@
+# Benchmark Dockerfile for tealdeer
+#
+# To run the benchmarks, execute
+#
+#     docker build -t tldr-benchmark .
+#     docker run --privileged --rm -it tldr-benchmark
+#
+# as root in the directory of this Dockerfile. This will build the compared
+# clients and benchmark them with `hyperfine` at the end.
+#
+# The `--privileged` flag is needed to drop the disk caches before every run. If
+# you want to test with hot caches or don't want to use this flag, you will have
+# to remove the `--prepare` line from the `hyperfine` command at the end of this
+# file and rebuild the image.
+
+################################################################################
+
+FROM rust AS tealdeer-builder
+
+WORKDIR /build
+RUN git clone https://github.com/dbrgn/tealdeer.git \
+        && cd tealdeer \
+        && cargo build --release \
+        && mkdir /build-outputs \
+        && cp target/release/tldr /build-outputs/tealdeer
+
+################################################################################
+
+FROM ubuntu:latest AS tldr-c-builder
+
+WORKDIR /build
+RUN apt-get update && apt-get install -y build-essential git && rm -rf /var/lib/apt/lists/*
+RUN git clone https://github.com/tldr-pages/tldr-c-client.git \
+        && cd tldr-c-client \
+        && DEBIAN_FRONTEND=noninteractive ./deps.sh \
+        && make \
+        && mkdir /build-outputs /deps \
+        && cp tldr /build-outputs/tldr-c \
+        && cp deps.sh /deps/tldr-c-deps.sh
+
+################################################################################
+
+FROM haskell AS haskell-builder
+
+WORKDIR /build
+
+RUN git clone https://github.com/psibi/tldr-hs.git \
+        && cd tldr-hs \
+        && stack build --install-ghc
+
+RUN git clone https://github.com/gutjuri/fast-tldr \
+        && cd fast-tldr \
+        && stack build --install-ghc
+
+RUN mkdir /build-outputs \
+        && find tldr-hs/.stack-work/dist -type f -iname tldr -exec mv '{}' /build-outputs/tldr-hs \; \
+        && find fast-tldr/.stack-work/dist -type f -iname tldr -exec mv '{}' /build-outputs/fast-tldr \;
+
+################################################################################
+
+FROM node:slim AS node-builder
+
+WORKDIR /build-outputs
+RUN npm install tldr \
+        && cp $(which node) . \
+        && echo './node -- ./node_modules/.bin/tldr "$@"' > tldr-node \
+        && chmod +x tldr-node
+
+################################################################################
+
+FROM euantorano/zig:0.8.0 AS zig-builder
+
+WORKDIR /build
+RUN apk add git \
+        && git clone https://gitlab.com/ve-nt/outfieldr.git \
+        && cd outfieldr \
+        && git submodule init \
+        && git submodule update \
+        && zig build -Drelease-safe \
+        && mkdir /build-outputs \
+        && cp bin/tldr /build-outputs/outfieldr
+
+################################################################################
+
+FROM ubuntu:latest AS benchmark
+
+ENV LANG="en_US.UTF-8"
+
+WORKDIR /deps
+RUN apt-get update && apt-get install -y wget unzip python3 python3-venv && rm -rf /var/lib/apt/lists/*
+COPY --from=tldr-c-builder /deps/* ./
+RUN for file in *; do DEBIAN_FRONTEND=noninteractive sh $file; done
+
+WORKDIR /clients
+COPY --from=tealdeer-builder /build-outputs/* ./
+COPY --from=tldr-c-builder /build-outputs/* ./
+COPY --from=haskell-builder /build-outputs/* ./
+RUN wget -qO tldr-bash https://4e4.win/tldr && chmod +x tldr-bash
+COPY --from=node-builder /build-outputs/node /build-outputs/tldr-node ./
+COPY --from=node-builder /build-outputs/node_modules/ ./node_modules/
+COPY --from=zig-builder /build-outputs/* ./
+
+# python is really hard to isolate in a package, using pyinstaller didn't really work either, so for now we just use it like this
+RUN python3 -m venv tldr-python \
+        && cd tldr-python \
+        && bash -c 'source bin/activate; pip install wheel; pip install tldr; deactivate' \
+        && cd .. \
+        && echo '#!/bin/bash' > tldr-python.bash \
+        && echo 'source tldr-python/bin/activate; tldr $@' >> tldr-python.bash \
+        && chmod +x tldr-python.bash
+
+# Update all the individual caches
+RUN bash -c 'mkdir -p /caches/{tealdeer,tldr-c,tldr-hs,fast-tldr,tldr-bash,tldr-node,tldr-python,outfieldr/.local/share}' \
+        && TEALDEER_CACHE_DIR=/caches/tealdeer ./tealdeer -u \
+        && TLDR_CACHE_DIR=/caches/tldr-c ./tldr-c -u \
+        && XDG_DATA_HOME=/caches/tldr-hs ./tldr-hs -u \
+        && XDG_DATA_HOME=/caches/fast-tldr ./fast-tldr -u \
+        && XDG_DATA_HOME=/caches/tldr-bash ./tldr-bash -u \
+        && HOME=/caches/tldr-node ./tldr-node -u \
+        && HOME=/caches/tldr-python ./tldr-python.bash -u \
+        && HOME=/caches/outfieldr ./outfieldr -u
+
+WORKDIR /tools
+RUN wget -q https://github.com/sharkdp/hyperfine/releases/download/v1.11.0/hyperfine_1.11.0_amd64.deb && dpkg -i hyperfine_1.11.0_amd64.deb
+
+ENV PAGE="tar"
+WORKDIR /clients
+CMD hyperfine \
+        --warmup 10 \
+        --runs 50 \
+        --prepare 'sync; echo 3 | tee /proc/sys/vm/drop_caches' \
+        "TEALDEER_CACHE_DIR=/caches/tealdeer ./tealdeer $PAGE" \
+        "TLDR_CACHE_DIR=/caches/tldr-c ./tldr-c $PAGE" \
+        "XDG_DATA_HOME=/caches/tldr-hs ./tldr-hs $PAGE" \
+        "XDG_DATA_HOME=/caches/fast-tldr ./fast-tldr $PAGE" \
+        "XDG_DATA_HOME=/caches/tldr-bash TLDR_LESS=0 ./tldr-bash $PAGE" \
+        "HOME=/caches/tldr-python ./tldr-python.bash $PAGE" \
+        "HOME=/caches/outfieldr ./outfieldr $PAGE" \
+        "HOME=/caches/tldr-node ./tldr-node $PAGE"