upstream/borgbackup
1
0
Fork 0
mirror of https://github.com/borgbackup/borg.git synced 2026-02-21 17:00:14 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

Merge pull request #6463 from ThomasWaldmann/new-crypto

Browse source 
new AEAD crypto with session keys
This commit is contained in:
TW 2022-03-26 17:27:58 +01:00 committed by GitHub
commit fbdeaa89bc
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
20 changed files with 569 additions and 308 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

35
docs/faq.rst
View file

@ -89,7 +89,7 @@ Also, you must not run borg against multiple instances of the same repo
(which is an issue if they happen to be not the same).
See :issue:`4272` for an example.
- Encryption security issues if you would update repo and copy-of-repo
independently, due to AES counter reuse.
independently, due to AES counter reuse (when using legacy encryption modes).
See also: :ref:`faq_corrupt_repo`
@ -246,6 +246,8 @@ then use ``tar`` to perform the comparison:
My repository is corrupt, how can I restore from an older copy of it?
---------------------------------------------------------------------
Note: this is only required for repos using legacy encryption modes.
If your repositories are encrypted and have the same ID, the recommended method
is to delete the corrupted repository, but keep its security info, and then copy
the working repository to the same location:
@ -473,8 +475,11 @@ Security
.. _borg_security_critique:
Isn't BorgBackup's AES-CTR crypto broken?
-----------------------------------------
Isn't BorgBackup's legacy AES-CTR-based crypto broken?
------------------------------------------------------
Note: in borg 1.3 new AEAD cipher based modes with session keys were added,
solving the issues of the legacy modes.
If a nonce (counter) value is reused, AES-CTR mode crypto is broken.
@ -713,6 +718,8 @@ Please disclose security issues responsibly.
How important are the nonce files?
------------------------------------
This only applies to repositories using legacy encryption modes.
Borg uses :ref:`AES-CTR encryption <borg_security_critique>`. An
essential part of AES-CTR is a sequential counter that must **never**
repeat. If the same value of the counter is used twice in the same repository,
@ -881,14 +888,14 @@ What's the expected backup performance?
---------------------------------------
Compared to simply copying files (e.g. with ``rsync``), Borg has more work to do.
This can make creation of the first archive slower, but saves time
This can make creation of the first archive slower, but saves time
and disk space on subsequent runs. Here what Borg does when you run ``borg create``:
- Borg chunks the file (using the relatively expensive buzhash algorithm)
- It then computes the "id" of the chunk (hmac-sha256 (often slow, except
- It then computes the "id" of the chunk (hmac-sha256 (often slow, except
if your CPU has sha256 acceleration) or blake2b (fast, in software))
- Then it checks whether this chunk is already in the repo (local hashtable lookup,
fast). If so, the processing of the chunk is completed here. Otherwise it needs to
- Then it checks whether this chunk is already in the repo (local hashtable lookup,
fast). If so, the processing of the chunk is completed here. Otherwise it needs to
process the chunk:
- Compresses (the default lz4 is super fast)
- Encrypts (AES, usually fast if your CPU has AES acceleration as usual
@ -896,9 +903,9 @@ and disk space on subsequent runs. Here what Borg does when you run ``borg creat
- Authenticates ("signs") using hmac-sha256 or blake2b (see above),
- Transmits to repo. If the repo is remote, this usually involves an SSH connection
(does its own encryption / authentication).
- Stores the chunk into a key/value store (the key is the chunk id, the value
- Stores the chunk into a key/value store (the key is the chunk id, the value
is the data). While doing that, it computes a CRC32 of the data (repo low-level
checksum, used by borg check --repository) and also updates the repo index
checksum, used by borg check --repository) and also updates the repo index
(another hashtable).
Subsequent backups are usually very fast if most files are unchanged and only
@ -928,14 +935,14 @@ If you feel your Borg backup is too slow somehow, here is what you can do:
- Make sure Borg has enough RAM (depends on how big your repo is / how many
files you have)
- Use one of the blake2 modes for --encryption except if you positively know
- Use one of the blake2 modes for --encryption except if you positively know
your CPU (and openssl) accelerates sha256 (then stay with hmac-sha256).
- Don't use any expensive compression. The default is lz4 and super fast.
Uncompressed is often slower than lz4.
- Just wait. You can also interrupt it and start it again as often as you like,
it will converge against a valid "completed" state (see ``--checkpoint-interval``,
maybe use the default, but in any case don't make it too short). It is starting
from the beginning each time, but it is still faster then as it does not store
from the beginning each time, but it is still faster then as it does not store
data into the repo which it already has there from last checkpoint.
- If you don’t need additional file attributes, you can disable them with ``--noflags``,
``--noacls``, ``--noxattrs``. This can lead to noticable performance improvements
@ -945,12 +952,12 @@ If you feel that Borg "freezes" on a file, it could be in the middle of processi
large file (like ISOs or VM images). Borg < 1.2 announces file names *after* finishing
with the file. This can lead to displaying the name of a small file, while processing the
next (larger) file. For very big files this can lead to the progress display show some
previous short file for a long time while it processes the big one. With Borg 1.2 this
previous short file for a long time while it processes the big one. With Borg 1.2 this
was changed to announcing the filename before starting to process it.
To see what files have changed and take more time processing, you can also add
``--list --filter=AME --stats`` to your ``borg create`` call to produce more log output,
including a file list (with file status characters) and also some statistics at
``--list --filter=AME --stats`` to your ``borg create`` call to produce more log output,
including a file list (with file status characters) and also some statistics at
the end of the backup.
Then you do the backup and look at the log output:

28
docs/internals/data-structures.rst
View file

@ -865,6 +865,31 @@ Encryption
.. seealso:: The :ref:`borgcrypto` section for an in-depth review.
AEAD modes
~~~~~~~~~~
Uses modern AEAD ciphers: AES-OCB or CHACHA20-POLY1305.
For each borg invocation, a new sessionkey is derived from the borg key material
and the 48bit IV starts from 0 again (both ciphers internally add a 32bit counter
to our IV, so we'll just count up by 1 per chunk).
The chunk layout is best seen at the bottom of this diagram:
.. figure:: encryption-aead.png
:figwidth: 100%
:width: 100%
No special IV/counter management is needed here due to the use of session keys.
A 48 bit IV is way more than needed: If you only backed up 4kiB chunks (2^12B),
the IV would "limit" the data encrypted in one session to 2^(12+48)B == 2.3 exabytes,
meaning you would run against other limitations (RAM, storage, time) way before that.
In practice, chunks are usually bigger, for big files even much bigger, giving an
even higher limit.
Legacy modes
~~~~~~~~~~~~
AES_-256 is used in CTR mode (so no need for padding). A 64 bit initialization
vector is used, a MAC is computed on the encrypted chunk
and both are stored in the chunk. Encryption and MAC use two different keys.
@ -884,6 +909,9 @@ To reduce payload size, only 8 bytes of the 16 bytes nonce is saved in the
payload, the first 8 bytes are always zeros. This does not affect security but
limits the maximum repository capacity to only 295 exabytes (2**64 * 16 bytes).
Both modes
~~~~~~~~~~
Encryption keys (and other secrets) are kept either in a key file on the client
('keyfile' mode) or in the repository config on the server ('repokey' mode).
In both cases, the secrets are generated from random and then encrypted by a

BIN
docs/internals/encryption-aead.odg Normal file
View file

Binary file not shown.

BIN
docs/internals/encryption-aead.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 156 KiB

90
docs/internals/security.rst
View file

@ -124,7 +124,88 @@ prompt is a set BORG_PASSPHRASE. See issue :issue:`2169` for details.
Encryption
----------
Encryption is currently based on the Encrypt-then-MAC construction,
AEAD modes
~~~~~~~~~~
Modes: --encryption (repokey|keyfile)-[blake2-](aes-ocb|chacha20-poly1305)
Supported: borg 1.3+
Encryption with these modes is based on AEAD ciphers (authenticated encryption
with associated data) and session keys.
Depending on the chosen mode (see :ref:`borg_init`) different AEAD ciphers are used:
- AES-256-OCB - super fast, single-pass algorithm IF you have hw accelerated AES.
- chacha20-poly1305 - very fast, purely software based AEAD cipher.
The chunk ID is derived via a MAC over the plaintext (mac key taken from borg key):
- HMAC-SHA256 - super fast IF you have hw accelerated SHA256.
- Blake2b - very fast, purely software based algorithm.
For each borg invocation, a new session id is generated by `os.urandom`_.
From that session id, the initial key material (ikm, taken from the borg key)
and an application and cipher specific salt, borg derives a session key via HKDF.
For each session key, IVs (nonces) are generated by a counter which increments for
each encrypted message.
Session::
sessionid = os.urandom(24)
ikm = enc_key || enc_hmac_key
salt = "borg-session-key-CIPHERNAME"
sessionkey = HKDF(ikm, sessionid, salt)
message_iv = 0
Encryption::
id = MAC(id_key, data)
compressed = compress(data)
header = type-byte || 00h || message_iv || sessionid
aad = id || header
message_iv++
encrypted, auth_tag = AEAD_encrypt(session_key, message_iv, compressed, aad)
authenticated = header || auth_tag || encrypted
Decryption::
# Given: input *authenticated* data and a *chunk-id* to assert
type-byte, past_message_iv, past_sessionid, auth_tag, encrypted = SPLIT(authenticated)
ASSERT(type-byte is correct)
past_key = HKDF(ikm, past_sessionid, salt)
decrypted = AEAD_decrypt(past_key, past_message_iv, authenticated)
decompressed = decompress(decrypted)
ASSERT( CONSTANT-TIME-COMPARISON( chunk-id, MAC(id_key, decompressed) ) )
Notable:
- More modern and often faster AEAD ciphers instead of self-assembled stuff.
- Due to the usage of session keys, IVs (nonces) do not need special care here as
they did for the legacy encryption modes.
- The id is now also input into the authentication tag computation.
This strongly associates the id with the written data (== associates the key with
the value). When later reading the data for some id, authentication will only
succeed if what we get was really written by us for that id.
Legacy modes
~~~~~~~~~~~~
Modes: --encryption (repokey|keyfile)-[blake2]
Supported: all borg versions, blake2 since 1.1
DEPRECATED. We strongly suggest you use the safer AEAD modes, see above.
Encryption with these modes is based on the Encrypt-then-MAC construction,
which is generally seen as the most robust way to create an authenticated
encryption scheme from encryption and message authentication primitives.
@ -137,7 +218,7 @@ in the future.
Depending on the chosen mode (see :ref:`borg_init`) different primitives are used:
- The actual encryption is currently always AES-256 in CTR mode. The
- Legacy encryption modes use AES-256 in CTR mode. The
counter is added in plaintext, since it is needed for decryption,
and is also tracked locally on the client to avoid counter reuse.
@ -253,7 +334,7 @@ Implementations used
We do not implement cryptographic primitives ourselves, but rely
on widely used libraries providing them:
- AES-CTR and HMAC-SHA-256 from OpenSSL 1.0 / 1.1 are used,
- AES-CTR, AES-OCB, CHACHA20-POLY1305 and HMAC-SHA-256 from OpenSSL 1.1 are used,
which is also linked into the static binaries we provide.
We think this is not an additional risk, since we don't ever
use OpenSSL's networking, TLS or X.509 code, but only their
@ -268,7 +349,8 @@ on widely used libraries providing them:
Implemented cryptographic constructions are:
- Encrypt-then-MAC based on AES-256-CTR and either HMAC-SHA-256
- AEAD modes: AES-OCB and CHACHA20-POLY1305 are straight from OpenSSL.
- Legacy modes: Encrypt-then-MAC based on AES-256-CTR and either HMAC-SHA-256
or keyed BLAKE2b256 as described above under Encryption_.
- Encrypt-and-MAC based on AES-256-CTR and HMAC-SHA-256
as described above under `Offline key security`_.

1
docs/quickstart.rst
View file

@ -387,6 +387,7 @@ For automated backups the passphrase can be specified using the
A backup inside of the backup that is encrypted with that key/passphrase
won't help you with that, of course.
Only applies to repos using legacy encryption modes:
In case you lose your repository and the security information, but have an
older copy of it to restore from, don't use that later for creating new
backups – you would run into security issues (reuse of nonce counter

13
docs/usage/init.rst
View file

@ -4,16 +4,19 @@ Examples
~~~~~~~~
::
# Local repository, repokey encryption, BLAKE2b (often faster, since Borg 1.1)
$ borg init --encryption=repokey-blake2 /path/to/repo
# Local repository, recommended repokey AEAD crypto modes
$ borg init --encryption=repokey-aes-ocb /path/to/repo
$ borg init --encryption=repokey-chacha20-poly1305 /path/to/repo
$ borg init --encryption=repokey-blake2-aes-ocb /path/to/repo
$ borg init --encryption=repokey-blake2-chacha20-poly1305 /path/to/repo
# Local repository (no encryption)
# Local repository (no encryption), not recommended
$ borg init --encryption=none /path/to/repo
# Remote repository (accesses a remote borg via ssh)
# repokey: stores the (encrypted) key into <REPO_DIR>/config
$ borg init --encryption=repokey-blake2 user@hostname:backup
$ borg init --encryption=repokey-aes-ocb user@hostname:backup
# Remote repository (accesses a remote borg via ssh)
# keyfile: stores the (encrypted) key into ~/.config/borg/keys/
$ borg init --encryption=keyfile user@hostname:backup
$ borg init --encryption=keyfile-aes-ocb user@hostname:backup

6
src/borg/archive.py
View file

@ -1789,7 +1789,7 @@ class ArchiveChecker:
def add_callback(chunk):
id_ = self.key.id_hash(chunk)
cdata = self.key.encrypt(chunk)
cdata = self.key.encrypt(id_, chunk)
add_reference(id_, len(chunk), len(cdata), cdata)
return id_
@ -1811,7 +1811,7 @@ class ArchiveChecker:
def replacement_chunk(size):
chunk = Chunk(None, allocation=CH_ALLOC, size=size)
chunk_id, data = cached_hash(chunk, self.key.id_hash)
cdata = self.key.encrypt(data)
cdata = self.key.encrypt(chunk_id, data)
csize = len(cdata)
return chunk_id, size, csize, cdata
@ -1998,7 +1998,7 @@ class ArchiveChecker:
archive.items = items_buffer.chunks
data = msgpack.packb(archive.as_dict())
new_archive_id = self.key.id_hash(data)
cdata = self.key.encrypt(data)
cdata = self.key.encrypt(new_archive_id, data)
add_reference(new_archive_id, len(data), len(cdata), cdata)
self.manifest.archives[info.name] = (new_archive_id, info.ts)
pi.finish()

117
src/borg/archiver.py
View file

@ -604,9 +604,9 @@ class Archiver:
if not is_libressl:
tests.extend([
("aes-256-ocb", lambda: AES256_OCB(
None, key_256, iv=key_96, header_len=1, aad_offset=1).encrypt(random_10M, header=b'X')),
key_256, iv=key_96, header_len=1, aad_offset=1).encrypt(random_10M, header=b'X')),
("chacha20-poly1305", lambda: CHACHA20_POLY1305(
None, key_256, iv=key_96, header_len=1, aad_offset=1).encrypt(random_10M, header=b'X')),
key_256, iv=key_96, header_len=1, aad_offset=1).encrypt(random_10M, header=b'X')),
])
for spec, func in tests:
print(f"{spec:<24} {size:<10} {timeit(func, number=100):.3f}s")
@ -4184,22 +4184,19 @@ class Archiver:
Encryption mode TLDR
++++++++++++++++++++
The encryption mode can only be configured when creating a new repository -
you can neither configure it on a per-archive basis nor change the
encryption mode of an existing repository.
The encryption mode can only be configured when creating a new repository - you can
neither configure it on a per-archive basis nor change the mode of an existing repository.
This example will likely NOT give optimum performance on your machine (performance
tips will come below):
Use ``repokey``::
::
borg init --encryption repokey /path/to/repo
Or ``repokey-blake2`` depending on which is faster on your client machines (see below)::
borg init --encryption repokey-blake2 /path/to/repo
Borg will:
1. Ask you to come up with a passphrase.
2. Create a borg key (which contains 3 random secrets. See :ref:`key_files`).
2. Create a borg key (which contains some random secrets. See :ref:`key_files`).
3. Encrypt the key with your passphrase.
4. Store the encrypted borg key inside the repository directory (in the repo config).
This is why it is essential to use a secure passphrase.
@ -4235,79 +4232,53 @@ class Archiver:
You can change your passphrase for existing repos at any time, it won't affect
the encryption/decryption key or other secrets.
More encryption modes
+++++++++++++++++++++
Choosing an encryption mode
+++++++++++++++++++++++++++
Only use ``--encryption none`` if you are OK with anyone who has access to
your repository being able to read your backups and tamper with their
contents without you noticing.
Depending on your hardware, hashing and crypto performance may vary widely.
The easiest way to find out about what's fastest is to run ``borg benchmark cpu``.
If you want "passphrase and having-the-key" security, use ``--encryption keyfile``.
The key will be stored in your home directory (in ``~/.config/borg/keys``).
`repokey` modes: if you want ease-of-use and "passphrase" security is good enough -
the key will be stored in the repository (in ``repo_dir/config``).
If you do **not** want to encrypt the contents of your backups, but still
want to detect malicious tampering use ``--encryption authenticated``.
`keyfile` modes: if you rather want "passphrase and having-the-key" security -
the key will be stored in your home directory (in ``~/.config/borg/keys``).
If ``BLAKE2b`` is faster than ``SHA-256`` on your hardware, use ``--encryption authenticated-blake2``,
``--encryption repokey-blake2`` or ``--encryption keyfile-blake2``. Note: for remote backups
the hashing is done on your local machine.
The following table is roughly sorted in order of preference, the better ones are
in the upper part of the table, in the lower part is the old and/or unsafe(r) stuff:
.. nanorst: inline-fill
+----------+---------------+------------------------+--------------------------+
| Hash/MAC | Not encrypted | Not encrypted, | Encrypted (AEAD w/ AES) |
| | no auth | but authenticated | and authenticated |
+----------+---------------+------------------------+--------------------------+
| SHA-256 | none | `authenticated` | repokey |
| | | | keyfile |
+----------+---------------+------------------------+--------------------------+
| BLAKE2b | n/a | `authenticated-blake2` | `repokey-blake2` |
| | | | `keyfile-blake2` |
+----------+---------------+------------------------+--------------------------+
+---------------------------------+---------------+---------------+------------------+-------+
|**mode (* = keyfile or repokey)**|**ID-Hash** |**Encryption** |**Authentication**|**V>=**|
+---------------------------------+---------------+---------------+------------------+-------+
| ``*-blake2-chacha20-poly1305`` | BLAKE2b | CHACHA20 | POLY1305 | 1.3 |
+---------------------------------+---------------+---------------+------------------+-------+
| ``*-chacha20-poly1305`` | HMAC-SHA-256 | CHACHA20 | POLY1305 | 1.3 |
+---------------------------------+---------------+---------------+------------------+-------+
| ``*-blake2-aes-ocb`` | BLAKE2b | AES256-OCB | AES256-OCB | 1.3 |
+---------------------------------+---------------+---------------+------------------+-------+
| ``*-aes-ocb`` | HMAC-SHA-256 | AES256-OCB | AES256-OCB | 1.3 |
+---------------------------------+---------------+---------------+------------------+-------+
| ``*-blake2`` | BLAKE2b | AES256-CTR | BLAKE2b | 1.1 |
+---------------------------------+---------------+---------------+------------------+-------+
| ``*`` | HMAC-SHA-256 | AES256-CTR | HMAC-SHA256 | any |
+---------------------------------+---------------+---------------+------------------+-------+
| authenticated-blake2 | BLAKE2b | none | BLAKE2b | 1.1 |
+---------------------------------+---------------+---------------+------------------+-------+
| authenticated | HMAC-SHA-256 | none | HMAC-SHA256 | 1.1 |
+---------------------------------+---------------+---------------+------------------+-------+
| none | SHA-256 | none | none | any |
+---------------------------------+---------------+---------------+------------------+-------+
.. nanorst: inline-replace
Modes `marked like this` in the above table are new in Borg 1.1 and are not
backwards-compatible with Borg 1.0.x.
`none` mode uses no encryption and no authentication. You're advised to NOT use this mode
as it would expose you to all sorts of issues (DoS, confidentiality, tampering, ...) in
case of malicious activity in the repository.
On modern Intel/AMD CPUs (except very cheap ones), AES is usually
hardware-accelerated.
BLAKE2b is faster than SHA256 on Intel/AMD 64-bit CPUs
(except AMD Ryzen and future CPUs with SHA extensions),
which makes `authenticated-blake2` faster than `none` and `authenticated`.
On modern ARM CPUs, NEON provides hardware acceleration for SHA256 making it faster
than BLAKE2b-256 there. NEON accelerates AES as well.
Hardware acceleration is always used automatically when available.
`repokey` and `keyfile` use AES-CTR-256 for encryption and HMAC-SHA256 for
authentication in an encrypt-then-MAC (EtM) construction. The chunk ID hash
is HMAC-SHA256 as well (with a separate key).
These modes are compatible with Borg 1.0.x.
`repokey-blake2` and `keyfile-blake2` are also authenticated encryption modes,
but use BLAKE2b-256 instead of HMAC-SHA256 for authentication. The chunk ID
hash is a keyed BLAKE2b-256 hash.
These modes are new and *not* compatible with Borg 1.0.x.
`authenticated` mode uses no encryption, but authenticates repository contents
through the same HMAC-SHA256 hash as the `repokey` and `keyfile` modes (it uses it
as the chunk ID hash). The key is stored like `repokey`.
This mode is new and *not* compatible with Borg 1.0.x.
`authenticated-blake2` is like `authenticated`, but uses the keyed BLAKE2b-256 hash
from the other blake2 modes.
This mode is new and *not* compatible with Borg 1.0.x.
`none` mode uses no encryption and no authentication. It uses SHA256 as chunk
ID hash. This mode is not recommended, you should rather consider using an authenticated
or authenticated/encrypted mode. This mode has possible denial-of-service issues
when running ``borg create`` on contents controlled by an attacker.
Use it only for new repositories where no encryption is wanted **and** when compatibility
with 1.0.x is important. If compatibility with 1.0.x is not important, use
`authenticated-blake2` or `authenticated` instead.
This mode is compatible with Borg 1.0.x.
If you do **not** want to encrypt the contents of your backups, but still want to detect
malicious tampering use an `authenticated` mode. It's like `repokey` minus encryption.
""")
subparser = subparsers.add_parser('init', parents=[common_parser], add_help=False,
description=self.do_init.__doc__, epilog=init_epilog,

4
src/borg/cache.py
View file

@ -942,7 +942,7 @@ class LocalCache(CacheStatsMixin):
refcount = self.seen_chunk(id, size)
if refcount and not overwrite:
return self.chunk_incref(id, stats)
data = self.key.encrypt(chunk)
data = self.key.encrypt(id, chunk)
csize = len(data)
self.repository.put(id, data, wait=wait)
self.chunks.add(id, 1, size, csize)
@ -1107,7 +1107,7 @@ Chunk index: {0.total_unique_chunks:20d} unknown"""
refcount = self.seen_chunk(id, size)
if refcount:
return self.chunk_incref(id, stats, size=size)
data = self.key.encrypt(chunk)
data = self.key.encrypt(id, chunk)
csize = len(data)
self.repository.put(id, data, wait=wait)
self.chunks.add(id, 1, size, csize)

12
src/borg/constants.py
View file

@ -111,6 +111,8 @@ class KeyBlobStorage:
class KeyType:
# legacy crypto
# upper 4 bits are ciphersuite, 0 == legacy AES-CTR
KEYFILE = 0x00
# repos with PASSPHRASE mode could not be created any more since borg 1.0, see #97.
# in borg 1.3 all of its code and also the "borg key migrate-to-repokey" command was removed.
@ -123,6 +125,16 @@ class KeyType:
BLAKE2REPO = 0x05
BLAKE2AUTHENTICATED = 0x06
AUTHENTICATED = 0x07
# new crypto
# upper 4 bits are ciphersuite, lower 4 bits are keytype
AESOCBKEYFILE = 0x10
AESOCBREPO = 0x11
CHPOKEYFILE = 0x20
CHPOREPO = 0x21
BLAKE2AESOCBKEYFILE = 0x30
BLAKE2AESOCBREPO = 0x31
BLAKE2CHPOKEYFILE = 0x40
BLAKE2CHPOREPO = 0x41
REPOSITORY_README = """This is a Borg Backup repository.

229
src/borg/crypto/key.py
View file

@ -23,7 +23,7 @@ from ..platform import SaveFile
from .nonces import NonceManager
from .low_level import AES, bytes_to_long, long_to_bytes, bytes_to_int, num_cipher_blocks, hmac_sha256, blake2b_256, hkdf_hmac_sha512
from .low_level import AES256_CTR_HMAC_SHA256, AES256_CTR_BLAKE2b
from .low_level import AES256_CTR_HMAC_SHA256, AES256_CTR_BLAKE2b, AES256_OCB, CHACHA20_POLY1305
class UnsupportedPayloadError(Error):
@ -156,8 +156,9 @@ class KeyBase:
def id_hash(self, data):
"""Return HMAC hash using the "id" HMAC key
"""
raise NotImplementedError
def encrypt(self, chunk):
def encrypt(self, id, data):
pass
def decrypt(self, id, data, decompress=True):
@ -263,8 +264,8 @@ class PlaintextKey(KeyBase):
def id_hash(self, data):
return sha256(data).digest()
def encrypt(self, chunk):
data = self.compressor.compress(chunk)
def encrypt(self, id, data):
data = self.compressor.compress(data)
return b''.join([self.TYPE_STR, data])
def decrypt(self, id, data, decompress=True):
@ -335,12 +336,12 @@ class AESKeyBase(KeyBase):
PAYLOAD_OVERHEAD = 1 + 32 + 8 # TYPE + HMAC + NONCE
CIPHERSUITE = AES256_CTR_HMAC_SHA256
CIPHERSUITE = None # override in derived class
logically_encrypted = True
def encrypt(self, chunk):
data = self.compressor.compress(chunk)
def encrypt(self, id, data):
data = self.compressor.compress(data)
next_iv = self.nonce_manager.ensure_reservation(self.cipher.next_iv(),
self.cipher.block_count(len(data)))
return self.cipher.encrypt(data, header=self.TYPE_STR, iv=next_iv)
@ -382,7 +383,9 @@ class AESKeyBase(KeyBase):
self.nonce_manager = NonceManager(self.repository, nonce)
class FlexiKeyBase(AESKeyBase):
class FlexiKey:
FILE_ID = 'BORG_KEY'
@classmethod
def detect(cls, repository, manifest_data):
key = cls(repository)
@ -405,12 +408,6 @@ class FlexiKeyBase(AESKeyBase):
key._passphrase = passphrase
return key
def find_key(self):
raise NotImplementedError
def load(self, target, passphrase):
raise NotImplementedError
def _load(self, key_data, passphrase):
cdata = a2b_base64(key_data)
data = self.decrypt_key_file(cdata, passphrase)
@ -488,18 +485,6 @@ class FlexiKeyBase(AESKeyBase):
logger.info('Keep this key safe. Your data will be inaccessible without it.')
return key
def save(self, target, passphrase, create=False):
raise NotImplementedError
def get_new_target(self, args):
raise NotImplementedError
class FlexiKey(ID_HMAC_SHA_256, FlexiKeyBase):
TYPES_ACCEPTABLE = {KeyType.KEYFILE, KeyType.REPO, KeyType.PASSPHRASE}
FILE_ID = 'BORG_KEY'
def sanity_check(self, filename, id):
file_id = self.FILE_ID.encode() + b' '
repo_id = hexlify(id)
@ -624,40 +609,43 @@ class FlexiKey(ID_HMAC_SHA_256, FlexiKeyBase):
raise TypeError('Unsupported borg key storage type')
class KeyfileKey(FlexiKey):
class KeyfileKey(ID_HMAC_SHA_256, AESKeyBase, FlexiKey):
TYPES_ACCEPTABLE = {KeyType.KEYFILE, KeyType.REPO, KeyType.PASSPHRASE}
TYPE = KeyType.KEYFILE
NAME = 'key file'
ARG_NAME = 'keyfile'
STORAGE = KeyBlobStorage.KEYFILE
CIPHERSUITE = AES256_CTR_HMAC_SHA256
class RepoKey(FlexiKey):
class RepoKey(ID_HMAC_SHA_256, AESKeyBase, FlexiKey):
TYPES_ACCEPTABLE = {KeyType.KEYFILE, KeyType.REPO, KeyType.PASSPHRASE}
TYPE = KeyType.REPO
NAME = 'repokey'
ARG_NAME = 'repokey'
STORAGE = KeyBlobStorage.REPO
CIPHERSUITE = AES256_CTR_HMAC_SHA256
class Blake2FlexiKey(ID_BLAKE2b_256, FlexiKey):
class Blake2KeyfileKey(ID_BLAKE2b_256, AESKeyBase, FlexiKey):
TYPES_ACCEPTABLE = {KeyType.BLAKE2KEYFILE, KeyType.BLAKE2REPO}
CIPHERSUITE = AES256_CTR_BLAKE2b
class Blake2KeyfileKey(Blake2FlexiKey):
TYPE = KeyType.BLAKE2KEYFILE
NAME = 'key file BLAKE2b'
ARG_NAME = 'keyfile-blake2'
STORAGE = KeyBlobStorage.KEYFILE
CIPHERSUITE = AES256_CTR_BLAKE2b
class Blake2RepoKey(Blake2FlexiKey):
class Blake2RepoKey(ID_BLAKE2b_256, AESKeyBase, FlexiKey):
TYPES_ACCEPTABLE = {KeyType.BLAKE2KEYFILE, KeyType.BLAKE2REPO}
TYPE = KeyType.BLAKE2REPO
NAME = 'repokey BLAKE2b'
ARG_NAME = 'repokey-blake2'
STORAGE = KeyBlobStorage.REPO
CIPHERSUITE = AES256_CTR_BLAKE2b
class AuthenticatedKeyBase(FlexiKey):
class AuthenticatedKeyBase(AESKeyBase, FlexiKey):
STORAGE = KeyBlobStorage.REPO
# It's only authenticated, not encrypted.
@ -676,8 +664,8 @@ class AuthenticatedKeyBase(FlexiKey):
if manifest_data is not None:
self.assert_type(manifest_data[0])
def encrypt(self, chunk):
data = self.compressor.compress(chunk)
def encrypt(self, id, data):
data = self.compressor.compress(data)
return b''.join([self.TYPE_STR, data])
def decrypt(self, id, data, decompress=True):
@ -690,7 +678,7 @@ class AuthenticatedKeyBase(FlexiKey):
return data
class AuthenticatedKey(AuthenticatedKeyBase):
class AuthenticatedKey(ID_HMAC_SHA_256, AuthenticatedKeyBase):
TYPE = KeyType.AUTHENTICATED
TYPES_ACCEPTABLE = {TYPE}
NAME = 'authenticated'
@ -704,8 +692,173 @@ class Blake2AuthenticatedKey(ID_BLAKE2b_256, AuthenticatedKeyBase):
ARG_NAME = 'authenticated-blake2'
# ------------ new crypto ------------
class AEADKeyBase(KeyBase):
"""
Chunks are encrypted and authenticated using some AEAD ciphersuite
Layout: suite:4 keytype:4 reserved:8 messageIV:48 sessionID:192 auth_tag:128 payload:... [bits]
^-------------------- AAD ----------------------------^
Offsets:0 1 2 8 32 48 [bytes]
suite: 1010b for new AEAD crypto, 0000b is old crypto
keytype: see constants.KeyType (suite+keytype)
reserved: all-zero, for future use
messageIV: a counter starting from 0 for all new encrypted messages of one session
sessionID: 192bit random, computed once per session (the session key is derived from this)
auth_tag: authentication tag output of the AEAD cipher (computed over payload and AAD)
payload: encrypted chunk data
"""
PAYLOAD_OVERHEAD = 1 + 1 + 6 + 24 + 16 # [bytes], see Layout
CIPHERSUITE = None # override in subclass
logically_encrypted = True
MAX_IV = 2 ** 48 - 1
def encrypt(self, id, data):
# to encrypt new data in this session we use always self.cipher and self.sessionid
data = self.compressor.compress(data)
reserved = b'\0'
iv = self.cipher.next_iv()
if iv > self.MAX_IV: # see the data-structures docs about why the IV range is enough
raise IntegrityError("IV overflow, should never happen.")
iv_48bit = iv.to_bytes(6, 'big')
header = self.TYPE_STR + reserved + iv_48bit + self.sessionid
return self.cipher.encrypt(data, header=header, iv=iv, aad=id)
def decrypt(self, id, data, decompress=True):
# to decrypt existing data, we need to get a cipher configured for the sessionid and iv from header
self.assert_type(data[0], id)
iv_48bit = data[2:8]
sessionid = data[8:32]
iv = int.from_bytes(iv_48bit, 'big')
cipher = self._get_cipher(sessionid, iv)
try:
payload = cipher.decrypt(data, aad=id)
except IntegrityError as e:
raise IntegrityError(f"Chunk {bin_to_hex(id)}: Could not decrypt [{str(e)}]")
if not decompress:
return payload
data = self.decompress(payload)
self.assert_id(id, data)
return data
def init_from_random_data(self):
data = os.urandom(100)
self.enc_key = data[0:32]
self.enc_hmac_key = data[32:64]
self.id_key = data[64:96]
self.chunk_seed = bytes_to_int(data[96:100])
# Convert to signed int32
if self.chunk_seed & 0x80000000:
self.chunk_seed = self.chunk_seed - 0xffffffff - 1
def _get_session_key(self, sessionid):
assert len(sessionid) == 24 # 192bit
key = hkdf_hmac_sha512(
ikm=self.enc_key + self.enc_hmac_key,
salt=sessionid,
info=b'borg-session-key-' + self.CIPHERSUITE.__name__.encode(),
output_length=32
)
return key
def _get_cipher(self, sessionid, iv):
assert isinstance(iv, int)
key = self._get_session_key(sessionid)
cipher = self.CIPHERSUITE(key=key, iv=iv, header_len=1+1+6+24, aad_offset=0)
return cipher
def init_ciphers(self, manifest_data=None, iv=0):
# in every new session we start with a fresh sessionid and at iv == 0, manifest_data and iv params are ignored
self.sessionid = os.urandom(24)
self.cipher = self._get_cipher(self.sessionid, iv=0)
class AESOCBKeyfileKey(ID_HMAC_SHA_256, AEADKeyBase, FlexiKey):
TYPES_ACCEPTABLE = {KeyType.AESOCBKEYFILE, KeyType.AESOCBREPO}
TYPE = KeyType.AESOCBKEYFILE
NAME = 'key file AES-OCB'
ARG_NAME = 'keyfile-aes-ocb'
STORAGE = KeyBlobStorage.KEYFILE
CIPHERSUITE = AES256_OCB
class AESOCBRepoKey(ID_HMAC_SHA_256, AEADKeyBase, FlexiKey):
TYPES_ACCEPTABLE = {KeyType.AESOCBKEYFILE, KeyType.AESOCBREPO}
TYPE = KeyType.AESOCBREPO
NAME = 'repokey AES-OCB'
ARG_NAME = 'repokey-aes-ocb'
STORAGE = KeyBlobStorage.REPO
CIPHERSUITE = AES256_OCB
class CHPOKeyfileKey(ID_HMAC_SHA_256, AEADKeyBase, FlexiKey):
TYPES_ACCEPTABLE = {KeyType.CHPOKEYFILE, KeyType.CHPOREPO}
TYPE = KeyType.CHPOKEYFILE
NAME = 'key file ChaCha20-Poly1305'
ARG_NAME = 'keyfile-chacha20-poly1305'
STORAGE = KeyBlobStorage.KEYFILE
CIPHERSUITE = CHACHA20_POLY1305
class CHPORepoKey(ID_HMAC_SHA_256, AEADKeyBase, FlexiKey):
TYPES_ACCEPTABLE = {KeyType.CHPOKEYFILE, KeyType.CHPOREPO}
TYPE = KeyType.CHPOREPO
NAME = 'repokey ChaCha20-Poly1305'
ARG_NAME = 'repokey-chacha20-poly1305'
STORAGE = KeyBlobStorage.REPO
CIPHERSUITE = CHACHA20_POLY1305
class Blake2AESOCBKeyfileKey(ID_BLAKE2b_256, AEADKeyBase, FlexiKey):
TYPES_ACCEPTABLE = {KeyType.BLAKE2AESOCBKEYFILE, KeyType.BLAKE2AESOCBREPO}
TYPE = KeyType.BLAKE2AESOCBKEYFILE
NAME = 'key file Blake2b AES-OCB'
ARG_NAME = 'keyfile-blake2-aes-ocb'
STORAGE = KeyBlobStorage.KEYFILE
CIPHERSUITE = AES256_OCB
class Blake2AESOCBRepoKey(ID_BLAKE2b_256, AEADKeyBase, FlexiKey):
TYPES_ACCEPTABLE = {KeyType.BLAKE2AESOCBKEYFILE, KeyType.BLAKE2AESOCBREPO}
TYPE = KeyType.BLAKE2AESOCBREPO
NAME = 'repokey Blake2b AES-OCB'
ARG_NAME = 'repokey-blake2-aes-ocb'
STORAGE = KeyBlobStorage.REPO
CIPHERSUITE = AES256_OCB
class Blake2CHPOKeyfileKey(ID_BLAKE2b_256, AEADKeyBase, FlexiKey):
TYPES_ACCEPTABLE = {KeyType.BLAKE2CHPOKEYFILE, KeyType.BLAKE2CHPOREPO}
TYPE = KeyType.BLAKE2CHPOKEYFILE
NAME = 'key file Blake2b ChaCha20-Poly1305'
ARG_NAME = 'keyfile-blake2-chacha20-poly1305'
STORAGE = KeyBlobStorage.KEYFILE
CIPHERSUITE = CHACHA20_POLY1305
class Blake2CHPORepoKey(ID_BLAKE2b_256, AEADKeyBase, FlexiKey):
TYPES_ACCEPTABLE = {KeyType.BLAKE2CHPOKEYFILE, KeyType.BLAKE2CHPOREPO}
TYPE = KeyType.BLAKE2CHPOREPO
NAME = 'repokey Blake2b ChaCha20-Poly1305'
ARG_NAME = 'repokey-blake2-chacha20-poly1305'
STORAGE = KeyBlobStorage.REPO
CIPHERSUITE = CHACHA20_POLY1305
AVAILABLE_KEY_TYPES = (
PlaintextKey,
KeyfileKey, RepoKey, AuthenticatedKey,
Blake2KeyfileKey, Blake2RepoKey, Blake2AuthenticatedKey,
# new crypto
AESOCBKeyfileKey, AESOCBRepoKey,
CHPOKeyfileKey, CHPORepoKey,
Blake2AESOCBKeyfileKey, Blake2AESOCBRepoKey,
Blake2CHPOKeyfileKey, Blake2CHPORepoKey,
)

219
src/borg/crypto/low_level.pyx
View file

@ -42,7 +42,7 @@ from cpython cimport PyMem_Malloc, PyMem_Free
from cpython.buffer cimport PyBUF_SIMPLE, PyObject_GetBuffer, PyBuffer_Release
from cpython.bytes cimport PyBytes_FromStringAndSize
API_VERSION = '1.2_01'
API_VERSION = '1.3_01'
cdef extern from "openssl/crypto.h":
int CRYPTO_memcmp(const void *a, const void *b, size_t len)
@ -77,9 +77,9 @@ cdef extern from "openssl/evp.h":
int EVP_DecryptFinal_ex(EVP_CIPHER_CTX *ctx, unsigned char *out, int *outl)
int EVP_CIPHER_CTX_ctrl(EVP_CIPHER_CTX *ctx, int type, int arg, void *ptr)
int EVP_CTRL_GCM_GET_TAG
int EVP_CTRL_GCM_SET_TAG
int EVP_CTRL_GCM_SET_IVLEN
int EVP_CTRL_AEAD_GET_TAG
int EVP_CTRL_AEAD_SET_TAG
int EVP_CTRL_AEAD_SET_IVLEN
const EVP_MD *EVP_sha256() nogil
@ -152,7 +152,7 @@ class UNENCRYPTED:
self.header_len = header_len
self.set_iv(iv)
def encrypt(self, data, header=b'', iv=None):
def encrypt(self, data, header=b'', iv=None, aad=None):
"""
IMPORTANT: it is called encrypt to satisfy the crypto api naming convention,
but this does NOT encrypt and it does NOT compute and store a MAC either.
@ -162,7 +162,7 @@ class UNENCRYPTED:
assert self.iv is not None, 'iv needs to be set before encrypt is called'
return header + data
def decrypt(self, envelope):
def decrypt(self, envelope, aad=None):
"""
IMPORTANT: it is called decrypt to satisfy the crypto api naming convention,
but this does NOT decrypt and it does NOT verify a MAC either, because data
@ -184,10 +184,10 @@ class UNENCRYPTED:
cdef class AES256_CTR_BASE:
# Layout: HEADER + MAC 32 + IV 8 + CT (same as attic / borg < 1.2 IF HEADER = TYPE_BYTE, no AAD)
# Layout: HEADER + MAC 32 + IV 8 + CT (same as attic / borg < 1.3 IF HEADER = TYPE_BYTE, no AAD)
cdef EVP_CIPHER_CTX *ctx
cdef unsigned char *enc_key
cdef unsigned char enc_key[32]
cdef int cipher_blk_len
cdef int iv_len, iv_len_short
cdef int aad_offset
@ -235,7 +235,7 @@ cdef class AES256_CTR_BASE:
"""
raise NotImplementedError
def encrypt(self, data, header=b'', iv=None):
def encrypt(self, data, header=b'', iv=None, aad=None):
"""
encrypt data, compute mac over aad + iv + cdata, prepend header.
aad_offset is the offset into the header where aad starts.
@ -252,7 +252,7 @@ cdef class AES256_CTR_BASE:
ilen + self.cipher_blk_len) # play safe, 1 extra blk
if not odata:
raise MemoryError
cdef int olen
cdef int olen = 0
cdef int offset
cdef Py_buffer idata = ro_buffer(data)
cdef Py_buffer hdata = ro_buffer(header)
@ -264,15 +264,12 @@ cdef class AES256_CTR_BASE:
offset += self.mac_len
self.store_iv(odata+offset, self.iv)
offset += self.iv_len_short
rc = EVP_EncryptInit_ex(self.ctx, EVP_aes_256_ctr(), NULL, self.enc_key, self.iv)
if not rc:
if not EVP_EncryptInit_ex(self.ctx, EVP_aes_256_ctr(), NULL, self.enc_key, self.iv):
raise CryptoError('EVP_EncryptInit_ex failed')
rc = EVP_EncryptUpdate(self.ctx, odata+offset, &olen, <const unsigned char*> idata.buf, ilen)
if not rc:
if not EVP_EncryptUpdate(self.ctx, odata+offset, &olen, <const unsigned char*> idata.buf, ilen):
raise CryptoError('EVP_EncryptUpdate failed')
offset += olen
rc = EVP_EncryptFinal_ex(self.ctx, odata+offset, &olen)
if not rc:
if not EVP_EncryptFinal_ex(self.ctx, odata+offset, &olen):
raise CryptoError('EVP_EncryptFinal_ex failed')
offset += olen
self.mac_compute(<const unsigned char *> hdata.buf+aoffset, alen,
@ -285,19 +282,18 @@ cdef class AES256_CTR_BASE:
PyBuffer_Release(&hdata)
PyBuffer_Release(&idata)
def decrypt(self, envelope):
def decrypt(self, envelope, aad=None):
"""
authenticate aad + iv + cdata, decrypt cdata, ignore header bytes up to aad_offset.
"""
cdef int ilen = len(envelope)
cdef int hlen = self.header_len
assert hlen == self.header_len
cdef int aoffset = self.aad_offset
cdef int alen = hlen - aoffset
cdef unsigned char *odata = <unsigned char *>PyMem_Malloc(ilen + self.cipher_blk_len) # play safe, 1 extra blk
if not odata:
raise MemoryError
cdef int olen
cdef int olen = 0
cdef int offset
cdef unsigned char mac_buf[32]
assert sizeof(mac_buf) == self.mac_len
@ -311,14 +307,12 @@ cdef class AES256_CTR_BASE:
if not EVP_DecryptInit_ex(self.ctx, EVP_aes_256_ctr(), NULL, self.enc_key, iv):
raise CryptoError('EVP_DecryptInit_ex failed')
offset = 0
rc = EVP_DecryptUpdate(self.ctx, odata+offset, &olen,
<const unsigned char*> idata.buf+hlen+self.mac_len+self.iv_len_short,
ilen-hlen-self.mac_len-self.iv_len_short)
if not rc:
if not EVP_DecryptUpdate(self.ctx, odata+offset, &olen,
<const unsigned char*> idata.buf+hlen+self.mac_len+self.iv_len_short,
ilen-hlen-self.mac_len-self.iv_len_short):
raise CryptoError('EVP_DecryptUpdate failed')
offset += olen
rc = EVP_DecryptFinal_ex(self.ctx, odata+offset, &olen)
if rc <= 0:
if not EVP_DecryptFinal_ex(self.ctx, odata+offset, &olen):
raise CryptoError('EVP_DecryptFinal_ex failed')
offset += olen
self.blocks += self.block_count(offset)
@ -335,8 +329,7 @@ cdef class AES256_CTR_BASE:
if isinstance(iv, int):
iv = iv.to_bytes(self.iv_len, byteorder='big')
assert isinstance(iv, bytes) and len(iv) == self.iv_len
for i in range(self.iv_len):
self.iv[i] = iv[i]
self.iv = iv
self.blocks = 0 # how many AES blocks got encrypted with this IV?
def next_iv(self):
@ -360,7 +353,7 @@ cdef class AES256_CTR_BASE:
cdef class AES256_CTR_HMAC_SHA256(AES256_CTR_BASE):
cdef unsigned char *mac_key
cdef unsigned char mac_key[32]
def __init__(self, mac_key, enc_key, iv=None, header_len=1, aad_offset=1):
assert isinstance(mac_key, bytes) and len(mac_key) == 32
@ -377,7 +370,7 @@ cdef class AES256_CTR_HMAC_SHA256(AES256_CTR_BASE):
const unsigned char *data2, int data2_len,
unsigned char *mac_buf):
data = data1[:data1_len] + data2[:data2_len]
mac = hmac.HMAC(self.mac_key, data, hashlib.sha256).digest()
mac = hmac.digest(self.mac_key[:self.mac_len], data, 'sha256')
for i in range(self.mac_len):
mac_buf[i] = mac[i]
@ -390,7 +383,7 @@ cdef class AES256_CTR_HMAC_SHA256(AES256_CTR_BASE):
cdef class AES256_CTR_BLAKE2b(AES256_CTR_BASE):
cdef unsigned char *mac_key
cdef unsigned char mac_key[128]
def __init__(self, mac_key, enc_key, iv=None, header_len=1, aad_offset=1):
assert isinstance(mac_key, bytes) and len(mac_key) == 128
@ -423,15 +416,16 @@ ctypedef const EVP_CIPHER * (* CIPHER)()
cdef class _AEAD_BASE:
# Layout: HEADER + MAC 16 + IV 12 + CT
# new crypto used in borg >= 1.3
# Layout: HEADER + MAC 16 + CT
cdef CIPHER cipher
cdef EVP_CIPHER_CTX *ctx
cdef unsigned char *enc_key
cdef unsigned char key[32]
cdef int cipher_blk_len
cdef int iv_len
cdef int aad_offset
cdef int header_len
cdef int header_len_expected
cdef int mac_len
cdef unsigned char iv[12]
cdef long long blocks
@ -441,83 +435,87 @@ cdef class _AEAD_BASE:
"""check whether library requirements for this ciphersuite are satisfied"""
raise NotImplemented # override / implement in child class
def __init__(self, mac_key, enc_key, iv=None, header_len=1, aad_offset=1):
assert mac_key is None
assert isinstance(enc_key, bytes) and len(enc_key) == 32
def __init__(self, key, iv=None, header_len=0, aad_offset=0):
"""
init AEAD crypto
:param key: 256bit encrypt-then-mac key
:param iv: 96bit initialisation vector / nonce
:param header_len: expected length of header
:param aad_offset: where in the header the authenticated data starts
"""
assert isinstance(key, bytes) and len(key) == 32
self.iv_len = sizeof(self.iv)
self.header_len = 1
self.header_len_expected = header_len
assert aad_offset <= header_len
self.aad_offset = aad_offset
self.header_len = header_len
self.mac_len = 16
self.enc_key = enc_key
self.key = key
if iv is not None:
self.set_iv(iv)
else:
self.blocks = -1 # make sure set_iv is called before encrypt
def __cinit__(self, mac_key, enc_key, iv=None, header_len=1, aad_offset=1):
def __cinit__(self, key, iv=None, header_len=0, aad_offset=0):
self.ctx = EVP_CIPHER_CTX_new()
def __dealloc__(self):
EVP_CIPHER_CTX_free(self.ctx)
def encrypt(self, data, header=b'', iv=None):
def encrypt(self, data, header=b'', iv=None, aad=b''):
"""
encrypt data, compute mac over aad + iv + cdata, prepend header.
aad_offset is the offset into the header where aad starts.
encrypt data, compute auth tag over aad + header + cdata.
return header + auth tag + cdata.
aad_offset is the offset into the header where the authenticated header part starts.
aad is additional authenticated data, which won't be included in the returned data,
but only used for the auth tag computation.
"""
if iv is not None:
self.set_iv(iv)
assert self.blocks == 0, 'iv needs to be set before encrypt is called'
# AES-GCM, AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit (12Byte)
# AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit (12Byte)
# IV we provide, thus we must not encrypt more than 2^32 cipher blocks with same IV).
block_count = self.block_count(len(data))
if block_count > 2**32:
raise ValueError('too much data, would overflow internal 32bit counter')
cdef int ilen = len(data)
cdef int hlen = len(header)
assert hlen == self.header_len
assert hlen == self.header_len_expected
cdef int aoffset = self.aad_offset
cdef int alen = hlen - aoffset
cdef unsigned char *odata = <unsigned char *>PyMem_Malloc(hlen + self.mac_len + self.iv_len +
cdef int aadlen = len(aad)
cdef unsigned char *odata = <unsigned char *>PyMem_Malloc(hlen + self.mac_len +
ilen + self.cipher_blk_len)
if not odata:
raise MemoryError
cdef int olen
cdef int olen = 0
cdef int offset
cdef Py_buffer idata = ro_buffer(data)
cdef Py_buffer hdata = ro_buffer(header)
cdef Py_buffer aadata = ro_buffer(aad)
try:
offset = 0
for i in range(hlen):
odata[offset+i] = header[i]
offset += hlen
offset += self.mac_len
self.store_iv(odata+offset, self.iv)
rc = EVP_EncryptInit_ex(self.ctx, self.cipher(), NULL, NULL, NULL)
if not rc:
if not EVP_EncryptInit_ex(self.ctx, self.cipher(), NULL, NULL, NULL):
raise CryptoError('EVP_EncryptInit_ex failed')
if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_GCM_SET_IVLEN, self.iv_len, NULL):
if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_AEAD_SET_IVLEN, self.iv_len, NULL):
raise CryptoError('EVP_CIPHER_CTX_ctrl SET IVLEN failed')
rc = EVP_EncryptInit_ex(self.ctx, NULL, NULL, self.enc_key, self.iv)
if not rc:
if not EVP_EncryptInit_ex(self.ctx, NULL, NULL, self.key, self.iv):
raise CryptoError('EVP_EncryptInit_ex failed')
rc = EVP_EncryptUpdate(self.ctx, NULL, &olen, <const unsigned char*> hdata.buf+aoffset, alen)
if not rc:
if not EVP_EncryptUpdate(self.ctx, NULL, &olen, <const unsigned char*> aadata.buf, aadlen):
raise CryptoError('EVP_EncryptUpdate failed')
if not EVP_EncryptUpdate(self.ctx, NULL, &olen, odata+offset, self.iv_len):
if not EVP_EncryptUpdate(self.ctx, NULL, &olen, <const unsigned char*> hdata.buf+aoffset, alen):
raise CryptoError('EVP_EncryptUpdate failed')
offset += self.iv_len
rc = EVP_EncryptUpdate(self.ctx, odata+offset, &olen, <const unsigned char*> idata.buf, ilen)
if not rc:
if not EVP_EncryptUpdate(self.ctx, odata+offset, &olen, <const unsigned char*> idata.buf, ilen):
raise CryptoError('EVP_EncryptUpdate failed')
offset += olen
rc = EVP_EncryptFinal_ex(self.ctx, odata+offset, &olen)
if not rc:
if not EVP_EncryptFinal_ex(self.ctx, odata+offset, &olen):
raise CryptoError('EVP_EncryptFinal_ex failed')
offset += olen
if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_GCM_GET_TAG, self.mac_len, odata+hlen):
if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_AEAD_GET_TAG, self.mac_len, odata + hlen):
raise CryptoError('EVP_CIPHER_CTX_ctrl GET TAG failed')
self.blocks = block_count
return odata[:offset]
@ -525,53 +523,50 @@ cdef class _AEAD_BASE:
PyMem_Free(odata)
PyBuffer_Release(&hdata)
PyBuffer_Release(&idata)
PyBuffer_Release(&aadata)
def decrypt(self, envelope):
def decrypt(self, envelope, aad=b''):
"""
authenticate aad + iv + cdata, decrypt cdata, ignore header bytes up to aad_offset.
authenticate aad + header + cdata (from envelope), ignore header bytes up to aad_offset.,
return decrypted cdata.
"""
# AES-GCM, AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit (12Byte)
# AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit (12Byte)
# IV we provide, thus we must not decrypt more than 2^32 cipher blocks with same IV):
approx_block_count = self.block_count(len(envelope)) # sloppy, but good enough for borg
if approx_block_count > 2**32:
raise ValueError('too much data, would overflow internal 32bit counter')
cdef int ilen = len(envelope)
cdef int hlen = self.header_len
assert hlen == self.header_len
cdef int hlen = self.header_len_expected
cdef int aoffset = self.aad_offset
cdef int alen = hlen - aoffset
cdef int aadlen = len(aad)
cdef unsigned char *odata = <unsigned char *>PyMem_Malloc(ilen + self.cipher_blk_len)
if not odata:
raise MemoryError
cdef int olen
cdef int olen = 0
cdef int offset
cdef Py_buffer idata = ro_buffer(envelope)
cdef Py_buffer aadata = ro_buffer(aad)
try:
if not EVP_DecryptInit_ex(self.ctx, self.cipher(), NULL, NULL, NULL):
raise CryptoError('EVP_DecryptInit_ex failed')
iv = self.fetch_iv(<unsigned char *> idata.buf+hlen+self.mac_len)
self.set_iv(iv)
if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_GCM_SET_IVLEN, self.iv_len, NULL):
if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_AEAD_SET_IVLEN, self.iv_len, NULL):
raise CryptoError('EVP_CIPHER_CTX_ctrl SET IVLEN failed')
if not EVP_DecryptInit_ex(self.ctx, NULL, NULL, self.enc_key, iv):
if not EVP_DecryptInit_ex(self.ctx, NULL, NULL, self.key, self.iv):
raise CryptoError('EVP_DecryptInit_ex failed')
if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_GCM_SET_TAG, self.mac_len, <unsigned char *> idata.buf+hlen):
raise CryptoError('EVP_CIPHER_CTX_ctrl SET TAG failed')
rc = EVP_DecryptUpdate(self.ctx, NULL, &olen, <const unsigned char*> idata.buf+aoffset, alen)
if not rc:
if not EVP_DecryptUpdate(self.ctx, NULL, &olen, <const unsigned char*> aadata.buf, aadlen):
raise CryptoError('EVP_DecryptUpdate failed')
if not EVP_DecryptUpdate(self.ctx, NULL, &olen,
<const unsigned char*> idata.buf+hlen+self.mac_len, self.iv_len):
if not EVP_DecryptUpdate(self.ctx, NULL, &olen, <const unsigned char*> idata.buf+aoffset, alen):
raise CryptoError('EVP_DecryptUpdate failed')
offset = 0
rc = EVP_DecryptUpdate(self.ctx, odata+offset, &olen,
<const unsigned char*> idata.buf+hlen+self.mac_len+self.iv_len,
ilen-hlen-self.mac_len-self.iv_len)
if not rc:
if not EVP_DecryptUpdate(self.ctx, odata+offset, &olen,
<const unsigned char*> idata.buf+hlen+self.mac_len,
ilen-hlen-self.mac_len):
raise CryptoError('EVP_DecryptUpdate failed')
offset += olen
rc = EVP_DecryptFinal_ex(self.ctx, odata+offset, &olen)
if rc <= 0:
if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_AEAD_SET_TAG, self.mac_len, <unsigned char *> idata.buf + hlen):
raise CryptoError('EVP_CIPHER_CTX_ctrl SET TAG failed')
if not EVP_DecryptFinal_ex(self.ctx, odata+offset, &olen):
# a failure here means corrupted or tampered tag (mac) or data.
raise IntegrityError('Authentication / EVP_DecryptFinal_ex failed')
offset += olen
@ -580,6 +575,7 @@ cdef class _AEAD_BASE:
finally:
PyMem_Free(odata)
PyBuffer_Release(&idata)
PyBuffer_Release(&aadata)
def block_count(self, length):
return num_cipher_blocks(length, self.cipher_blk_len)
@ -590,71 +586,48 @@ cdef class _AEAD_BASE:
if isinstance(iv, int):
iv = iv.to_bytes(self.iv_len, byteorder='big')
assert isinstance(iv, bytes) and len(iv) == self.iv_len
for i in range(self.iv_len):
self.iv[i] = iv[i]
self.iv = iv
self.blocks = 0 # number of cipher blocks encrypted with this IV
def next_iv(self):
# call this after encrypt() to get the next iv (int) for the next encrypt() call
# AES-GCM, AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit
# AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit
# (12 byte) IV we provide, thus we only need to increment the IV by 1.
iv = int.from_bytes(self.iv[:self.iv_len], byteorder='big')
return iv + 1
cdef fetch_iv(self, unsigned char * iv_in):
return iv_in[0:self.iv_len]
cdef store_iv(self, unsigned char * iv_out, unsigned char * iv):
cdef int i
for i in range(self.iv_len):
iv_out[i] = iv[i]
def extract_iv(self, envelope):
offset = self.header_len + self.mac_len
return bytes_to_long(envelope[offset:offset+self.iv_len])
cdef class _AES_BASE(_AEAD_BASE):
def __init__(self, *args, **kwargs):
self.cipher_blk_len = 16
super().__init__(*args, **kwargs)
cdef class _CHACHA_BASE(_AEAD_BASE):
def __init__(self, *args, **kwargs):
self.cipher_blk_len = 64
super().__init__(*args, **kwargs)
cdef class AES256_OCB(_AES_BASE):
cdef class AES256_OCB(_AEAD_BASE):
@classmethod
def requirements_check(cls):
if is_libressl:
raise ValueError('AES OCB is not implemented by LibreSSL (yet?).')
def __init__(self, mac_key, enc_key, iv=None, header_len=1, aad_offset=1):
def __init__(self, key, iv=None, header_len=0, aad_offset=0):
self.requirements_check()
self.cipher = EVP_aes_256_ocb
super().__init__(mac_key, enc_key, iv=iv, header_len=header_len, aad_offset=aad_offset)
self.cipher_blk_len = 16
super().__init__(key, iv=iv, header_len=header_len, aad_offset=aad_offset)
cdef class CHACHA20_POLY1305(_CHACHA_BASE):
cdef class CHACHA20_POLY1305(_AEAD_BASE):
@classmethod
def requirements_check(cls):
if is_libressl:
raise ValueError('CHACHA20-POLY1305 is not implemented by LibreSSL (yet?).')
def __init__(self, mac_key, enc_key, iv=None, header_len=1, aad_offset=1):
def __init__(self, key, iv=None, header_len=0, aad_offset=0):
self.requirements_check()
self.cipher = EVP_chacha20_poly1305
super().__init__(mac_key, enc_key, iv=iv, header_len=header_len, aad_offset=aad_offset)
self.cipher_blk_len = 64
super().__init__(key, iv=iv, header_len=header_len, aad_offset=aad_offset)
cdef class AES:
"""A thin wrapper around the OpenSSL EVP cipher API - for legacy code, like key file encryption"""
cdef CIPHER cipher
cdef EVP_CIPHER_CTX *ctx
cdef unsigned char *enc_key
cdef unsigned char enc_key[32]
cdef int cipher_blk_len
cdef int iv_len
cdef unsigned char iv[16]
@ -721,7 +694,7 @@ cdef class AES:
if not EVP_DecryptUpdate(self.ctx, odata, &olen, <const unsigned char*> idata.buf, ilen):
raise Exception('EVP_DecryptUpdate failed')
offset += olen
if EVP_DecryptFinal_ex(self.ctx, odata+offset, &olen) <= 0:
if not EVP_DecryptFinal_ex(self.ctx, odata+offset, &olen):
# this error check is very important for modes with padding or
# authentication. for them, a failure here means corrupted data.
# CTR mode does not use padding nor authentication.
@ -742,8 +715,7 @@ cdef class AES:
if isinstance(iv, int):
iv = iv.to_bytes(self.iv_len, byteorder='big')
assert isinstance(iv, bytes) and len(iv) == self.iv_len
for i in range(self.iv_len):
self.iv[i] = iv[i]
self.iv = iv
self.blocks = 0 # number of cipher blocks encrypted with this IV
def next_iv(self):
@ -752,7 +724,6 @@ cdef class AES:
return iv + self.blocks
def hmac_sha256(key, data):
return hmac.digest(key, data, 'sha256')
@ -779,7 +750,7 @@ def hkdf_hmac_sha512(ikm, salt, info, output_length):
# Step 1. HKDF-Extract (ikm, salt) -> prk
if salt is None:
salt = bytes(64)
prk = hmac.HMAC(salt, ikm, hashlib.sha512).digest()
prk = hmac.digest(salt, ikm, 'sha512')
# Step 2. HKDF-Expand (prk, info, output_length) -> output key
n = ceil(output_length / digest_length)
@ -787,6 +758,6 @@ def hkdf_hmac_sha512(ikm, salt, info, output_length):
output = b''
for i in range(n):
msg = t_n + info + (i + 1).to_bytes(1, 'little')
t_n = hmac.HMAC(prk, msg, hashlib.sha512).digest()
t_n = hmac.digest(prk, msg, 'sha512')
output += t_n
return output[:output_length]

2
src/borg/helpers/checks.py
View file

@ -31,7 +31,7 @@ def check_extension_modules():
raise ExtensionModuleError
if compress.API_VERSION != '1.2_02':
raise ExtensionModuleError
if borg.crypto.low_level.API_VERSION != '1.2_01':
if borg.crypto.low_level.API_VERSION != '1.3_01':
raise ExtensionModuleError
if item.API_VERSION != '1.2_01':
raise ExtensionModuleError

2
src/borg/helpers/manifest.py
View file

@ -261,4 +261,4 @@ class Manifest:
self.tam_verified = True
data = self.key.pack_and_authenticate_metadata(manifest.as_dict())
self.id = self.key.id_hash(data)
self.repository.put(self.MANIFEST_ID, self.key.encrypt(data))
self.repository.put(self.MANIFEST_ID, self.key.encrypt(self.MANIFEST_ID, data))

2
src/borg/selftest.py
View file

@ -29,7 +29,7 @@ SELFTEST_CASES = [
ChunkerTestCase,
]
SELFTEST_COUNT = 35
SELFTEST_COUNT = 36
class SelfTestResult(TestResult):

12
src/borg/testsuite/archiver.py
View file

@ -36,7 +36,7 @@ from ..cache import Cache, LocalCache
from ..chunker import has_seek_hole
from ..constants import * # NOQA
from ..crypto.low_level import bytes_to_long, num_cipher_blocks
from ..crypto.key import FlexiKeyBase, RepoKey, KeyfileKey, Passphrase, TAMRequiredError
from ..crypto.key import FlexiKey, RepoKey, KeyfileKey, Passphrase, TAMRequiredError
from ..crypto.keymanager import RepoIdMismatch, NotABorgKeyFile
from ..crypto.file_integrity import FileIntegrityError
from ..helpers import Location, get_security_dir
@ -2882,7 +2882,7 @@ class ArchiverTestCase(ArchiverTestCaseBase):
def raise_eof(*args):
raise EOFError
with patch.object(FlexiKeyBase, 'create', raise_eof):
with patch.object(FlexiKey, 'create', raise_eof):
self.cmd('init', '--encryption=repokey', self.repository_location, exit_code=1)
assert not os.path.exists(self.repository_location)
@ -3806,7 +3806,7 @@ class ArchiverCheckTestCase(ArchiverTestCaseBase):
'version': 1,
})
archive_id = key.id_hash(archive)
repository.put(archive_id, key.encrypt(archive))
repository.put(archive_id, key.encrypt(archive_id, archive))
repository.commit(compact=False)
self.cmd('check', self.repository_location, exit_code=1)
self.cmd('check', '--repair', self.repository_location, exit_code=0)
@ -3894,7 +3894,7 @@ class ManifestAuthenticationTest(ArchiverTestCaseBase):
def spoof_manifest(self, repository):
with repository:
_, key = Manifest.load(repository, Manifest.NO_OPERATION_CHECK)
repository.put(Manifest.MANIFEST_ID, key.encrypt(msgpack.packb({
repository.put(Manifest.MANIFEST_ID, key.encrypt(Manifest.MANIFEST_ID, msgpack.packb({
'version': 1,
'archives': {},
'config': {},
@ -3907,7 +3907,7 @@ class ManifestAuthenticationTest(ArchiverTestCaseBase):
repository = Repository(self.repository_path, exclusive=True)
with repository:
manifest, key = Manifest.load(repository, Manifest.NO_OPERATION_CHECK)
repository.put(Manifest.MANIFEST_ID, key.encrypt(msgpack.packb({
repository.put(Manifest.MANIFEST_ID, key.encrypt(Manifest.MANIFEST_ID, msgpack.packb({
'version': 1,
'archives': {},
'timestamp': (datetime.utcnow() + timedelta(days=1)).strftime(ISO_FORMAT),
@ -3929,7 +3929,7 @@ class ManifestAuthenticationTest(ArchiverTestCaseBase):
manifest = msgpack.unpackb(key.decrypt(None, repository.get(Manifest.MANIFEST_ID)))
del manifest[b'tam']
repository.put(Manifest.MANIFEST_ID, key.encrypt(msgpack.packb(manifest)))
repository.put(Manifest.MANIFEST_ID, key.encrypt(Manifest.MANIFEST_ID, msgpack.packb(manifest)))
repository.commit(compact=False)
output = self.cmd('list', '--debug', self.repository_location)
assert 'archive1234' in output

56
src/borg/testsuite/crypto.py
View file

@ -91,11 +91,10 @@ class CryptoTestCase(BaseTestCase):
def test_AE(self):
# used in legacy-like layout (1 type byte, no aad)
mac_key = None
enc_key = b'X' * 32
iv = 0
key = b'X' * 32
iv_int = 0
data = b'foo' * 10
header = b'\x23'
header = b'\x23' + iv_int.to_bytes(12, 'big')
tests = [
# (ciphersuite class, exp_mac, exp_cdata)
]
@ -111,11 +110,11 @@ class CryptoTestCase(BaseTestCase):
for cs_cls, exp_mac, exp_cdata in tests:
# print(repr(cs_cls))
# encrypt/mac
cs = cs_cls(mac_key, enc_key, iv, header_len=1, aad_offset=1)
cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=1)
hdr_mac_iv_cdata = cs.encrypt(data, header=header)
hdr = hdr_mac_iv_cdata[0:1]
mac = hdr_mac_iv_cdata[1:17]
iv = hdr_mac_iv_cdata[17:29]
iv = hdr_mac_iv_cdata[1:13]
mac = hdr_mac_iv_cdata[13:29]
cdata = hdr_mac_iv_cdata[29:]
self.assert_equal(hexlify(hdr), b'23')
self.assert_equal(hexlify(mac), exp_mac)
@ -123,23 +122,22 @@ class CryptoTestCase(BaseTestCase):
self.assert_equal(hexlify(cdata), exp_cdata)
self.assert_equal(cs.next_iv(), 1)
# auth/decrypt
cs = cs_cls(mac_key, enc_key, header_len=len(header), aad_offset=1)
cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=1)
pdata = cs.decrypt(hdr_mac_iv_cdata)
self.assert_equal(data, pdata)
self.assert_equal(cs.next_iv(), 1)
# auth-failure due to corruption (corrupted data)
cs = cs_cls(mac_key, enc_key, header_len=len(header), aad_offset=1)
cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=1)
hdr_mac_iv_cdata_corrupted = hdr_mac_iv_cdata[:29] + b'\0' + hdr_mac_iv_cdata[30:]
self.assert_raises(IntegrityError,
lambda: cs.decrypt(hdr_mac_iv_cdata_corrupted))
def test_AEAD(self):
# test with aad
mac_key = None
enc_key = b'X' * 32
iv = 0
key = b'X' * 32
iv_int = 0
data = b'foo' * 10
header = b'\x12\x34\x56'
header = b'\x12\x34\x56' + iv_int.to_bytes(12, 'big')
tests = [
# (ciphersuite class, exp_mac, exp_cdata)
]
@ -155,11 +153,11 @@ class CryptoTestCase(BaseTestCase):
for cs_cls, exp_mac, exp_cdata in tests:
# print(repr(cs_cls))
# encrypt/mac
cs = cs_cls(mac_key, enc_key, iv, header_len=3, aad_offset=1)
cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=1)
hdr_mac_iv_cdata = cs.encrypt(data, header=header)
hdr = hdr_mac_iv_cdata[0:3]
mac = hdr_mac_iv_cdata[3:19]
iv = hdr_mac_iv_cdata[19:31]
iv = hdr_mac_iv_cdata[3:15]
mac = hdr_mac_iv_cdata[15:31]
cdata = hdr_mac_iv_cdata[31:]
self.assert_equal(hexlify(hdr), b'123456')
self.assert_equal(hexlify(mac), exp_mac)
@ -167,16 +165,38 @@ class CryptoTestCase(BaseTestCase):
self.assert_equal(hexlify(cdata), exp_cdata)
self.assert_equal(cs.next_iv(), 1)
# auth/decrypt
cs = cs_cls(mac_key, enc_key, header_len=len(header), aad_offset=1)
cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=1)
pdata = cs.decrypt(hdr_mac_iv_cdata)
self.assert_equal(data, pdata)
self.assert_equal(cs.next_iv(), 1)
# auth-failure due to corruption (corrupted aad)
cs = cs_cls(mac_key, enc_key, header_len=len(header), aad_offset=1)
cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=1)
hdr_mac_iv_cdata_corrupted = hdr_mac_iv_cdata[:1] + b'\0' + hdr_mac_iv_cdata[2:]
self.assert_raises(IntegrityError,
lambda: cs.decrypt(hdr_mac_iv_cdata_corrupted))
def test_AEAD_with_more_AAD(self):
# test giving extra aad to the .encrypt() and .decrypt() calls
key = b'X' * 32
iv_int = 0
data = b'foo' * 10
header = b'\x12\x34'
tests = []
if not is_libressl:
tests += [AES256_OCB, CHACHA20_POLY1305]
for cs_cls in tests:
# encrypt/mac
cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=0)
hdr_mac_iv_cdata = cs.encrypt(data, header=header, aad=b'correct_chunkid')
# successful auth/decrypt (correct aad)
cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=0)
pdata = cs.decrypt(hdr_mac_iv_cdata, aad=b'correct_chunkid')
self.assert_equal(data, pdata)
# unsuccessful auth (incorrect aad)
cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=0)
self.assert_raises(IntegrityError,
lambda: cs.decrypt(hdr_mac_iv_cdata, aad=b'incorrect_chunkid'))
# These test vectors come from https://www.kullo.net/blog/hkdf-sha-512-test-vectors/
# who claims to have verified these against independent Python and C++ implementations.

47
src/borg/testsuite/key.py
View file

@ -8,7 +8,8 @@ import pytest
from ..crypto.key import bin_to_hex
from ..crypto.key import PlaintextKey, AuthenticatedKey, RepoKey, KeyfileKey, \
Blake2KeyfileKey, Blake2RepoKey, Blake2AuthenticatedKey
Blake2KeyfileKey, Blake2RepoKey, Blake2AuthenticatedKey, \
AESOCBKeyfileKey, AESOCBRepoKey, CHPOKeyfileKey, CHPORepoKey
from ..crypto.key import ID_HMAC_SHA_256, ID_BLAKE2b_256
from ..crypto.key import TAMRequiredError, TAMInvalid, TAMUnsupportedSuiteError, UnsupportedManifestError
from ..crypto.key import identify_key
@ -80,6 +81,8 @@ class TestKey:
Blake2KeyfileKey,
Blake2RepoKey,
Blake2AuthenticatedKey,
AESOCBKeyfileKey, AESOCBRepoKey,
CHPOKeyfileKey, CHPORepoKey,
))
def key(self, request, monkeypatch):
monkeypatch.setenv('BORG_PASSPHRASE', 'test')
@ -111,18 +114,21 @@ class TestKey:
def test_plaintext(self):
key = PlaintextKey.create(None, None)
chunk = b'foo'
assert hexlify(key.id_hash(chunk)) == b'2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae'
assert chunk == key.decrypt(key.id_hash(chunk), key.encrypt(chunk))
id = key.id_hash(chunk)
assert hexlify(id) == b'2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae'
assert chunk == key.decrypt(id, key.encrypt(id, chunk))
def test_keyfile(self, monkeypatch, keys_dir):
monkeypatch.setenv('BORG_PASSPHRASE', 'test')
key = KeyfileKey.create(self.MockRepository(), self.MockArgs())
assert key.cipher.next_iv() == 0
manifest = key.encrypt(b'ABC')
chunk = b'ABC'
id = key.id_hash(chunk)
manifest = key.encrypt(id, chunk)
assert key.cipher.extract_iv(manifest) == 0
manifest2 = key.encrypt(b'ABC')
manifest2 = key.encrypt(id, chunk)
assert manifest != manifest2
assert key.decrypt(None, manifest) == key.decrypt(None, manifest2)
assert key.decrypt(id, manifest) == key.decrypt(id, manifest2)
assert key.cipher.extract_iv(manifest2) == 1
iv = key.cipher.extract_iv(manifest)
key2 = KeyfileKey.detect(self.MockRepository(), manifest)
@ -131,7 +137,8 @@ class TestKey:
assert len({key2.id_key, key2.enc_key, key2.enc_hmac_key}) == 3
assert key2.chunk_seed != 0
chunk = b'foo'
assert chunk == key2.decrypt(key.id_hash(chunk), key.encrypt(chunk))
id = key.id_hash(chunk)
assert chunk == key2.decrypt(id, key.encrypt(id, chunk))
def test_keyfile_nonce_rollback_protection(self, monkeypatch, keys_dir):
monkeypatch.setenv('BORG_PASSPHRASE', 'test')
@ -139,9 +146,11 @@ class TestKey:
with open(os.path.join(get_security_dir(repository.id_str), 'nonce'), "w") as fd:
fd.write("0000000000002000")
key = KeyfileKey.create(repository, self.MockArgs())
data = key.encrypt(b'ABC')
chunk = b'ABC'
id = key.id_hash(chunk)
data = key.encrypt(id, chunk)
assert key.cipher.extract_iv(data) == 0x2000
assert key.decrypt(None, data) == b'ABC'
assert key.decrypt(id, data) == chunk
def test_keyfile_kfenv(self, tmpdir, monkeypatch):
keyfile = tmpdir.join('keyfile')
@ -152,7 +161,7 @@ class TestKey:
assert keyfile.exists()
chunk = b'ABC'
chunk_id = key.id_hash(chunk)
chunk_cdata = key.encrypt(chunk)
chunk_cdata = key.encrypt(chunk_id, chunk)
key = KeyfileKey.detect(self.MockRepository(), chunk_cdata)
assert chunk == key.decrypt(chunk_id, chunk_cdata)
keyfile.remove()
@ -209,18 +218,20 @@ class TestKey:
def test_roundtrip(self, key):
repository = key.repository
plaintext = b'foo'
encrypted = key.encrypt(plaintext)
id = key.id_hash(plaintext)
encrypted = key.encrypt(id, plaintext)
identified_key_class = identify_key(encrypted)
assert identified_key_class == key.__class__
loaded_key = identified_key_class.detect(repository, encrypted)
decrypted = loaded_key.decrypt(None, encrypted)
decrypted = loaded_key.decrypt(id, encrypted)
assert decrypted == plaintext
def test_decrypt_decompress(self, key):
plaintext = b'123456789'
encrypted = key.encrypt(plaintext)
assert key.decrypt(None, encrypted, decompress=False) != plaintext
assert key.decrypt(None, encrypted) == plaintext
id = key.id_hash(plaintext)
encrypted = key.encrypt(id, plaintext)
assert key.decrypt(id, encrypted, decompress=False) != plaintext
assert key.decrypt(id, encrypted) == plaintext
def test_assert_id(self, key):
plaintext = b'123456789'
@ -240,7 +251,8 @@ class TestKey:
assert AuthenticatedKey.id_hash is ID_HMAC_SHA_256.id_hash
assert len(key.id_key) == 32
plaintext = b'123456789'
authenticated = key.encrypt(plaintext)
id = key.id_hash(plaintext)
authenticated = key.encrypt(id, plaintext)
# 0x07 is the key TYPE, \x0000 identifies no compression.
assert authenticated == b'\x07\x00\x00' + plaintext
@ -250,7 +262,8 @@ class TestKey:
assert Blake2AuthenticatedKey.id_hash is ID_BLAKE2b_256.id_hash
assert len(key.id_key) == 128
plaintext = b'123456789'
authenticated = key.encrypt(plaintext)
id = key.id_hash(plaintext)
authenticated = key.encrypt(id, plaintext)
# 0x06 is the key TYPE, 0x0000 identifies no compression.
assert authenticated == b'\x06\x00\x00' + plaintext

2
src/borg/testsuite/remote.py
View file

@ -165,7 +165,7 @@ class TestRepositoryCache:
def _put_encrypted_object(self, key, repository, data):
id_ = key.id_hash(data)
repository.put(id_, key.encrypt(data))
repository.put(id_, key.encrypt(id_, data))
return id_
@pytest.fixture