locking: fix ExclusiveLock race condition bug, fixes#4923
- ExclusiveLock is now based on os.rename instead of os.mkdir.
- catch FileNotFoundError observed under race condition in ExclusiveLock.release()
and .kill_stale_lock()
- added TestExclusiveLock.test_race_condition() which reveals issue #4923
- updated docs
- locking: use "raise LockTimeout from None" for prettier traceback
Co-authored-by: Thomas Portmann <thomas@portmann.org>
Co-authored-by: Thomas Waldmann <tw@waldmann-edv.de>
The umask value is NOT transmitted from client to server any more,
so the borg client can not influence the borg server umask any more.
If one wants to have a specific umask on the server side, one needs to
use a ssh forced command in .ssh/authorized_keys file.
OTOH, as the default value is 077 (in general, for client as well as for
the server) and the server does not take the value from the client any more,
there usually should be no need to give it on the server side, IF you are
happy with the default value.
Running 'borg key import' on a keyfile repository with the BORG_KEY_FILE
environment variable set works correctly if the BORG_KEY_FILE file
already exists. However, the command crashes if the BORG_KEY_FILE file
does not exist:
$ BORG_KEY_FILE=newborgkey borg key import /home/strager/borg-backups/straglum borgkey
Local Exception
Traceback (most recent call last):
[snip]
File "[snip]/borg/crypto/key.py", line 713, in sanity_check
with open(filename, 'rb') as fd:
FileNotFoundError: [Errno 2] No such file or directory: '[snip]/newborgkey'
Platform: Linux straglum 5.0.0-25-generic #26~18.04.1-Ubuntu SMP Thu Aug 1 13:51:02 UTC 2019 x86_64
Linux: debian buster/sid
Borg: 1.1.11 Python: CPython 3.7.7 msgpack: 0.5.6
PID: 15306 CWD: /home/strager/Projects/borg
sys.argv: ['[snip]/borg', 'key', 'import', '/home/strager/borg-backups/straglum', 'borgkey']
SSH_ORIGINAL_COMMAND: None
Make 'borg key import' not require the BORG_KEY_FILE file to already
exist.
This commit does not change the behavior of 'borg key import' without
BORG_KEY_FILE. This commit also does not change the behavior of 'borg
key import' on a repokey repository.
general.rst and man_intro.rst both included usage_general.rst.inc, which resulted in three Sphinx warning "WARNING: duplicate label".
To prevent this we move all sections of usage_general into own include files and add a second usage_general file without the three labels.
Fix various code blocks in the docs
- rst markup: put codeblock markup on separate line to make it better visible / separate it from normal text colons.
- borg help texts in archiver.py: put codeblock markup instead of colon - that way it looks like a single colon when using the cli help and also works as code block markup.
add a push-style example to borg-create docs
When I browsed the readthedocs and manpages while setting up Borg for
myself for the first time, I saw multiple hints that pushing backups via
SSH is supported (e.g. the existence of the `borg serve` command and the
`--rsh` option), but I never found a clear example. I only found
<https://borgbackup.readthedocs.io/en/stable/usage/general.html#repository-urls>
after being pointed to it by @ThomasWaldmann.
Therefore, an explicit example in the `borg create` documentation seems
appropriate. I put it next to the pull-style example using sshfs which
serves a similar usecase.
Add "SSH Configuration" section
Add "SSH Configuration" section to "borg serve" documentation, to outline ssh/sshd configuration to prevent borg serve keeping a lock on a repo in the event the ssh connection is abnormally disconnected.
In response to issues #3988, #636 and #4485 (and probably others).
drop BORG_HOSTNAME_IS_UNIQUE (please use BORG_HOST_ID if needed)
borg now always assumes it has a unique hostid - either automatically
from fqdn plus uuid.getnode() or overridden via BORG_HOST_ID.
This command works similarly to "git config" - it parses repo and
cache configs to get, set, and delete values. It only works on local
repos so a malicious client can't e.g. override their storage quota
or reset the append_only flag.
Add tests for borg config
Add documentation for borg config
Change manual config edits -> borg config
There were a couple places in the documentation where it was advised
to edit the repository or cache config file, a process that is stream-
lined by borg config.
rename authenticated to authenticated-blake2, consistent with the other
blake2 key modes
add authenticated mode that fills the blank and is consistent with the
other "unqualified" key modes
this refactors umount code we already used for the testsuite into the platform module's namespace.
also, it exposes that functionality via the cli api, so users can use it via "borg umount <mountpoint>",
which is more consistent than using borg to mount and fusermount -u (or umount) to un-mount.
This fixes#1247. It also regenerates the usage documentation, so that
styling fixex in that section (as well as other existing changes) make
it into the files in docs/.
Use with caution: permanent data loss by specifying incorrect patterns
is easily possible. Make a dry run to make sure you got everything right.
borg recreate has many uses:
- Can selectively remove files/dirs from old archives, e.g. to free
space or purging picturarum biggus dickus from history
- Recompress data
- Rechunkify data, to have upgraded Attic / Borg 0.xx archives deduplicate
with Borg 1.x archives. (Or to experiment with chunker-params for
specific use cases
It is interrupt- and resumable.
Chunks are not freed on-the-fly.
Rationale:
Makes only sense when rechunkifying, but logic on which new chunks to
free what input chunks is complicated and *very* delicate.
Future TODOs:
- Refactor tests using py.test fixtures
-- would require porting ArchiverTestCase to py.test: many changes,
this changeset is already borderline too large.
- Possibly add a --target option to not replace the source archive
-- with the target possibly in another Repo
(better than "cp" due to full integrity checking, and deduplication
at the target)
- Detect and skip (unless --always-recompress) already recompressed chunks
Fixes#787#686#630#70 (and probably some I overlooked)
Also see #757 and #770
- Group options
- Nicer list of options in Sphinx
- Deduplicate 'Common options'
(including --help)
The latter is done by explicitly declaring --help in the common_parser,
which is then inherited by the sub-parsers; no change in observable
behaviour.
- Archives now have a new metadata field 'comment'
- 'info' command shows a comment if it's present
- 'create' command now has option '--comment' for adding comments to archives.
- A new command 'comment' is added for modifying the comments on existing
archives.
Resolves#842.
Previously, on 'borg diff', the output always had first the modifications, then
additions, and finally removals. Output may be easier to follow if the various
kinds of changes are interleaved. This commit is a simple solution that first
collects the output lines and sorts them by file path before printing. This new
behavior is optional and disabled by default. It can be enabled with '--sort'
command line option.
This option will be especially useful after the planned multi-threading changes
arrive. Multi-threading may shuffle the archive order of files making diff
output hard to follow without sorting.
Resolves#797.
- add archiver.main_mount()
- provide borgfs behaviour when the monolithic binary is called via a
symlink called borgfs
- docs: update usage of mount subcommand, provide examples for borgfs and
add symlink creation to standalone binary installation
- run build_usage
- add entry point in setup.py
- patch helpers.py:get_keys_dir() to allow mounting fstab entries with
"user" option set
Without this, setuid() called at some point by mount changes the HOME
environment variable to '/root' and os.expanduser('~') would return
'/root' as well, thus the mount would fail with
PermissionError: [Errno 13] Permission denied: '/root/.config'
After setuid(), the HOME variable stays intact, so we still can
explicitly query USER's home.
Also, os.path.expanduser() behaves differently for '~' and '~someuser'
as parameters: when called with an explicit username, the possibly set
environment variable HOME is no longer respected. So we have to check if
it is set and only expand the user's home directory if HOME is unset.
- add myself to AUTHORS
the generation of those files was causing us way too much pain to
justify automatically generating them all the time.
those will have to be re-generated with `build_api` or `build_usage`
as appropriate, for example when function signatures or commandline
flags change.
see #384
