Commit graph

1408 commits

Author SHA1 Message Date
Elmar Hoffmann
3e50ca8b63 make constants for files cache mode more clear (#6724)
* make constants for files cache mode more clear

Traditionally, DEFAULT_FILES_CACHE_MODE_UI and DEFAULT_FILES_CACHE_MODE
were - as the naming scheme implies - the same setting, one being the UI
representation as given to the --files-cache command line option and the
other being the same default value in the internal representation.

It happended that the actual value used in borg create always comes from
DEFAULT_FILES_CACHE_MODE_UI (because that does have the --files-cache
option) whereas for all other commands (that do not use the files cache) it
comes from DEFAULT_FILES_CACHE_MODE.

PR #5777 then abused this fact to implement the optimisation to skip loading
of the files cache in those other commands by changing the value of
DEFAULT_FILES_CACHE_MODE to disabled.
This however also changes the meaning of that variable and thus redesignates
it to something not matching the original naming anymore.

Anyone not aware of this change and the intention behind it looking at the
code would have a hard time figuring this out and be easily mislead.

This does away with the confusion making the code more maintainable by
renaming DEFAULT_FILES_CACHE_MODE to FILES_CACHE_MODE_DISABLED, making the
new intention of that internal default clear.

* make constant for files cache mode UI default match naming scheme
2022-05-30 15:34:08 +02:00
Andrea Gelmini
3bdf4ba829 Fix typos (#6688)
fix typos

Co-authored-by: Thomas Waldmann <tw@waldmann-edv.de>
2022-05-15 23:41:57 +02:00
Thomas Waldmann
88cf0a05a1 mount -o versions: give clear error msg instead of crashing
it does not make sense to request versions view if you only
look at 1 archive, but the code shall not crash in that case
as it did, but give a clear error msg.
2022-05-15 00:45:59 +02:00
TW
64c4f9696a
Merge pull request #6684 from ThomasWaldmann/fix-archive-progress-json-1.1
show_progress: add finished=true/false to archive_progress json
2022-05-08 19:46:07 +02:00
Thomas Waldmann
25cc85e78b show_progress: add finished=true/false to archive_progress json, fixes #6570
also:
- remove empty values from final json
- add test
2022-05-08 18:38:45 +02:00
Thomas Waldmann
f0349fcc54 check that borg does not require pytest for normal usage, fixes #6563
also: move the note about this to the very top of the affected modules.
2022-05-07 18:55:04 +02:00
Thomas Waldmann
2311dac835 give a more helpful error msg for unsupported key format, fixes #6561 2022-04-24 22:28:58 +02:00
Thomas Waldmann
dbe29b5849 give a more helpful error msg for unsupported encrypted key format, fixes #6561 2022-04-24 22:28:46 +02:00
TW
c2f59c9409
Merge pull request #6631 from ThomasWaldmann/fix-info-authenticated-1.1
info: fix authenticated mode repo to show "Encrypted: No", fixes #6462
2022-04-19 23:38:32 +02:00
Thomas Waldmann
ab2ba210da info: fix authenticated mode repo to show "Encrypted: No", fixes #6462 2022-04-19 22:09:45 +02:00
TW
6804d2c017
Merge pull request #6626 from ThomasWaldmann/issue-5673-1.1
do not load files cache for commands not using it, fixes #5673
2022-04-19 21:58:27 +02:00
Thomas Waldmann
9f2beff23b fix transaction rollback: use files cache filename as found in txn.active, fixes #6353 2022-04-19 21:19:21 +02:00
Thomas Waldmann
0ea5171ae6 do not load files cache for commands not using it, fixes #5673 2022-04-19 21:11:52 +02:00
TW
dc4aebde71
diff: support presence change for blkdev, chrdev and fifo items (1.1-maint) (#6625)
diff: support presence change for blkdev, chrdev and fifo items

fixes #6483
2022-04-19 20:55:59 +02:00
TW
0c79bd8e4a
Merge pull request #6621 from ThomasWaldmann/secure-erase-msg-1.1
remove stray punctuation from secure-erase message
2022-04-18 10:01:23 +02:00
TW
8cfd6fc5d7
Merge pull request #6617 from ThomasWaldmann/invalid-repo-msg-1.1
better error msg for defect or unsupported repo configs, fixes #6566
2022-04-18 09:52:11 +02:00
Jakub Wilk
91aab05228 remove stray punctuation from secure-erase message 2022-04-18 09:34:06 +02:00
Thomas Waldmann
8e48ca70f4 better error msg for defect or unsupported repo configs, fixes #6566 2022-04-18 09:29:04 +02:00
Thomas Waldmann
6defc50790 load_key: no key is same as empty key, fixes #6441
when migrating from repokey to keyfile, we just store an empty key into the repo config,
because we do not have a "delete key" RPC api. thus, empty key means "there is no key".

here we fix load_key, so that it does not behave differently for no key and empty key:
in both cases, it just returns an empty value.

additionally, we strip the value we get from the config, so whitespace does not matter.

All callers now check for the repokey not being empty, otherwise RepoKeyNotFoundError
is raised.
2022-04-12 19:50:36 +02:00
Thomas Waldmann
a0722a92ec check archives: improve error handling for corrupt archive metadata block
this is similar to #4777.

borg check must not crash if an archive metadata block does not decrypt.

Instead, report the archive_id, remove the archive from the manifest and skip to the next archive.
2022-04-07 01:35:35 +02:00
Thomas Waldmann
3f9ae96c7b check archive: make robust_iterator more robust, fixes #4777
borg check must not crash if an archive metadata chunk does not decrypt.

Instead, report the chunk and skip to the next one.
2022-04-07 01:22:34 +02:00
Björn Ketelaars
bf5c590769 1.1 - Fix OpenBSD symlink mode test failure (#2055)
OpenBSD does not have `lchmod()` causing `os.lchmod` to be unavailable
on this platform. As a result ArchiverTestCase::test_basic_functionality
fails when run manually (#2055).

OpenBSD does have `fchmodat()`, which has a flag that makes it behave
like `lchmod()`. In Python this can be used via `os.chmod(path, mode,
follow_symlinks=False)`.

As of Python 3.3 `os.lchmod(path, mode)` is equivalent to
`os.chmod(path, mode, follow_symlinks=False)`. As such, switching to the
latter is preferred as it enables more platforms to do the right thing.
2022-04-05 18:29:53 +02:00
TW
8497981a72
Merge pull request #6527 from ThomasWaldmann/fix-ipv6-url-parsing-1.1
fix scp repo url parsing for ip v6 addrs, fixes #6526
2022-04-04 20:39:44 +02:00
Thomas Waldmann
20a103ccb9 use same host regex for ssh and scp style, refactor/clean up
although bug #6526 did not show with ssh style URLs, we should
not have different regexes for the host part for ssh and scp style.

thus i extracted the host_re from both and also cleaned up a bit.
2022-04-03 21:38:39 +02:00
Thomas Waldmann
9324f31e60 fix scp repo url parsing for ip v6 addrs, fixes #6526
added a negative lookahead/lookbehind to make sure an ipv6 addr
(enclosed in square brackets) does not get badly matched by the
regex part intended for hostnames and ipv4 addrs only.

the other part of that regex which is actually intended to match
ipv6 addrs only matches if they are enclosed in square brackets.

also added tests for ssh and scp style repo URLs with ipv6 addrs
in brackets.

also: made regex more readable, putting these 2 cases on separate lines.
2022-04-03 21:31:50 +02:00
aspargas2
9f281c266d delete: add repository id and location to prompt
backport of 19c7b85
2022-03-29 22:22:33 -04:00
Thomas Waldmann
f8cb7147a7 use hmac.compare_digest instead of ==, fixes 6470 2022-03-26 18:55:48 +01:00
James Buren
67a2d3a296 1.1: Make switch fallthrough explicit
This includes a backport of #6426

Testing:
```
git clean -e /.envrc -e /.direnv -fdx \
&& CFLAGS=-Werror=implicit-fallthrough pip -v install -e .
```
2022-03-10 07:48:23 +03:00
Thomas Waldmann
988a27c44c ensure_dir: respect umask for created directory modes, fixes #6400
we tried to be very private / secure here, but that created the issue
that a less secure umask (like e.g. 0o007) just did not work.

to make the umask work, we must start from 0o777 mode and let the
umask do its work, like e.g. 0o777 & ~0o007 --> 0o770.

with borg's default umask of 0o077, it usually ends up being 0o700,
so only permissions for the user (not group, not others).
2022-03-09 17:23:36 +01:00
Thomas Waldmann
6a5718691f SaveFile: respect umask for final file mode, fixes #6400 2022-03-09 17:20:43 +01:00
Thomas Waldmann
15a57cd0bd fix "useless trailing comma" cython warnings 2022-03-08 21:31:17 +01:00
Andrey Bienkowski
e5e0da0360 ‘PyUnicode_AsUnicode’ is deprecated
Fix compilation warnings:
```
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/opt/hostedtoolcache/Python/3.9.10/x64/include/python3.9 -c src/borg/platform/posix.c -o build/temp.linux-x86_64-3.9/src/borg/platform/posix.o
src/borg/platform/posix.c: In function ‘__pyx_pf_4borg_8platform_5posix_2swidth’:
src/borg/platform/posix.c:1572:3: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
 1572 |   __pyx_t_2 = __Pyx_PyUnicode_AsUnicode(__pyx_v_s); if (unlikely((!__pyx_t_2) && PyErr_Occurred())) __PYX_ERR(0, 19, __pyx_L1_error)
      |   ^~~~~~~~~
In file included from /opt/hostedtoolcache/Python/3.9.10/x64/include/python3.9/unicodeobject.h:1026,
                 from /opt/hostedtoolcache/Python/3.9.10/x64/include/python3.9/Python.h:93,
                 from src/borg/platform/posix.c:19:
/opt/hostedtoolcache/Python/3.9.10/x64/include/python3.9/cpython/unicodeobject.h:580:45: note: declared here
  580 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
      |                                             ^~~~~~~~~~~~~~~~~~~
```

Ref: https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_AsUnicode
2022-03-07 21:40:47 +03:00
Thomas Waldmann
a257edbe3a compact: not "freeable", but "maybe freeable"
e.g. if there is a ton of DELs in a segment, they all are maybe freeable,
but only if we also got rid of the respective PUTs (see also #6289).
2022-02-23 21:14:23 +03:00
Thomas Waldmann
57475f4b72 compact: derive freed space from quota use before/after, fixes #5679
due to the way quota accounting is done, this is likely not
100% precise, but much better than selling the hints as the truth.

Backport notes: 1.1 did not report how much space was freed so
I have added the message.
2022-02-23 21:02:46 +03:00
Thomas Waldmann
a653e216d9 SaveFile: fix race conditions
Thanks to Andrey Bienkowski (@hexagonrecursion) for reporting this and writing reproducer code.

Changes:
- use different, randomly (but recognizably) named temp files while writing (securely made by os.mkstemp())
- make sure temp files are cleaned up in normal and error conditions
- SyncFile can now get corresponding pair of path + open os-level fd
- cleaned up: fd now means os-level fd, f means python-file-like object
- fixed a caller of SaveFile
2022-02-21 00:04:46 +03:00
TW
794907d4d2
Merge pull request #6312 from hexagonrecursion/bp-unlink
Backport:  safe_unlink (was: truncate_and_unlink)
2022-02-20 21:37:02 +01:00
Andrey Bienkowski
a26a6e80f2 Write a test for safe_unlink 2022-02-20 21:31:23 +03:00
Thomas Waldmann
0adcbca699 rename truncate_and_unlink to safe_unlink
it usually does not truncate any more,
only under "disk full" circumstances and only if there is only one hardlink.
2022-02-20 21:31:23 +03:00
Thomas Waldmann
cb6faf6828 safer truncate_and_unlink implementation
the previous implementation caused collateral damage on hardlink-copies of a repository,
see: https://github.com/borgbackup/borg/discussions/6286
2022-02-20 21:31:23 +03:00
TW
28d22a9501
Merge pull request #6326 from hexagonrecursion/bp-setdefault
Backport: implement more standard hashindex.setdefault behaviour
2022-02-20 16:52:26 +01:00
Thomas Waldmann
07ae21e94d implement more standard hashindex.setdefault behaviour
the .get() like behaviour (== returning the value) was missing.

it's still not 100% like dict.setdefault, because there is no
default value None. but None doesn't make sense here, because we
usually need a N-tuple matching the hash table's value format.

note: this "bug" (or unusual implementation) was without consequences,
      because hashindex.setdefault is not used anywhere in borg, so
      it was also not used in a wrong way anywhere.

https://docs.python.org/3/library/stdtypes.html#dict.setdefault
2022-02-20 18:37:31 +03:00
Andrey Bienkowski
d9b27561a2 Backport: docs: link to borg_placeholders 2022-02-20 17:33:53 +03:00
Andrey Bienkowski
dffac711bb Backport: docs: link to borg_patterns 2022-02-20 17:30:57 +03:00
Andrey Bienkowski
a8e13dd64c Backport: docs: use same phrasing in misc. help texts 2022-02-20 17:19:04 +03:00
TW
0624d7f18f
Merge pull request #6309 from hexagonrecursion/bp-tldr
Backport: Doc: borg init: explain the encryption modes better
2022-02-19 17:19:29 +01:00
Laurent
54ee7fb7cc docs: explain the difference between a path that ends with or without a slash (#6297)
docs: explain the difference between a path that ends with or without a slash
2022-02-19 13:58:49 +03:00
Andrey Bienkowski
ded2a76b31 Backport: Doc: borg init: explain the encryption modes better
This is a backport of #6184
2022-02-19 13:39:21 +03:00
Tomás Andrighetti
345f01413e Exclude directories in is_hardlink_master 2022-02-13 18:53:36 -03:00
Andrey Bienkowski
8bba72b3aa Fix indentation
This fixes an indentation error in src/borg/archiver.py that caused
the HTML help to look bad (the tail end of the page ended up
*inside* of a code block instead of after it)
2022-02-08 18:02:05 +03:00
bobthebadguy
2cdf0df14d Clarify usage of patternfile roots (#6242) 2022-02-08 17:37:25 +03:00