mirror of
https://github.com/borgbackup/borg.git
synced 2026-09-02 14:43:21 +02:00
ea9e1eb55dNew docs/internals/chunker.rst documenting all supported chunkers twice: - a plain-language section for users: what the chunker choice affects, the chunk-size fingerprinting threat model, an at-a-glance comparison table, and concrete recommendations per use case (borg 1.x dedup compat, speed, untrusted repository storage, disk images); - a technical section for readers with a crypto background: the FSWC model, why direct-decision keyed rolling hashes are broken (key recovery from observed boundaries), the UHF-then-PRF construction and its epsilon-AU requirement, the three universal hashes with their collision bounds (rabin-aes ~2^-55 probabilistic, goldilocks-aes ~2^-58, toeplitz-aes exactly 2^-64 unconditional), why the fixed 64-byte window is essential (including the rotation/cancellation algebra that also explains the classic 4095-vs-4096 buzhash window), rejected alternatives, measured performance, proof-bound limits and the wider-digest upgrade path, plus references. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
49 lines
1.8 KiB
ReStructuredText
49 lines
1.8 KiB
ReStructuredText
.. include:: global.rst.inc
|
|
.. _internals:
|
|
|
|
Internals
|
|
=========
|
|
|
|
The internals chapter describes and analyzes most of the inner workings
|
|
of Borg.
|
|
|
|
Borg uses a low-level, key-value store, the :ref:`repository`, and
|
|
implements a more complex data structure on top of it, which is made
|
|
up of the :ref:`manifest <manifest>`, :ref:`archives <archive>`,
|
|
:ref:`items <item>` and data :ref:`chunks`.
|
|
|
|
Each repository can hold multiple :ref:`archives <archive>`, which
|
|
represent individual backups that contain a full archive of the files
|
|
specified when the backup was performed.
|
|
|
|
Deduplication is performed globally across all data in the repository
|
|
(multiple backups and even multiple hosts), both on data and file
|
|
metadata, using :ref:`chunks` created by the chunker using a
|
|
content-defined chunking algorithm - the Gear rolling hash of FastCDC_
|
|
("fastcdc" chunker, the default) or Buzhash_ ("buzhash" and "buzhash64"
|
|
chunker) - or a simpler fixed block size algorithm ("fixed" chunker).
|
|
|
|
To perform the repository-wide deduplication, a hash of each
|
|
chunk is checked against the :ref:`chunks index <index>`, which is a
|
|
hash table of all chunks that already exist.
|
|
|
|
.. figure:: internals/structure.png
|
|
:figwidth: 100%
|
|
:width: 100%
|
|
|
|
Layers in Borg. At the very top, commands are implemented, using
|
|
a data access layer provided by the Archive and Item classes.
|
|
The "key" object provides both compression and authenticated
|
|
encryption used by the data access layer. The "key" object represents
|
|
the sole trust boundary in Borg.
|
|
The lowest layer is the repository accessed via class Repository.
|
|
Repository uses ``borgstore`` internally.
|
|
|
|
.. toctree::
|
|
:caption: Internals contents
|
|
|
|
internals/security
|
|
internals/data-structures
|
|
internals/chunker
|
|
internals/packs
|
|
internals/frontends
|