Commit graph

720 commits

Author SHA1 Message Date
enkore
a013bd7d75 Merge pull request #2687 from enkore/docs/formatting-i2650
More consistent docs formatting
2017-06-17 12:38:48 +02:00
Marian Beermann
bd701e58c5 docs: backticks for option values 2017-06-17 12:20:25 +02:00
Marian Beermann
48642d787a docs: double backticks for --options 2017-06-17 12:18:52 +02:00
Marian Beermann
1f5ddb6572 document pattern denial of service 2017-06-17 11:59:56 +02:00
enkore
2b13607f46 init: shaext is supported in openssl and better on ryzen than b2 2017-06-16 11:44:23 +02:00
enkore
5eeaf7a67c Merge pull request #2660 from enkore/issue/2659
cache sync: check Operation.READ compatibility with manifest
2017-06-16 09:48:09 +02:00
enkore
b6a4cf19bc Merge pull request #2676 from enkore/f/fusecache
fuse: redo ItemCache
2017-06-16 09:43:51 +02:00
Marian Beermann
b2a4ae6bc2 lrucache: use explicit sentinel instead of None
just in case someone wants to cache a big pile of nothing
2017-06-16 00:41:38 +02:00
Marian Beermann
2766693706 fuse: update comments 2017-06-15 23:50:17 +02:00
Marian Beermann
faf2d0b537 chunker: fix invalid use of types
With the argument specified as unsigned char *, Cython emits
code in the Python wrapper to convert string-like objects to
unsigned char* (essentially PyBytes_AS_STRING).

Because the len(data) call is performed on a cdef'd string-ish type,
Cython emits a strlen() call, on the result of PyBytes_AS_STRING.

This is not correct, since embedded null bytes are entirely possible.

Incidentally, the code generated by Cython was also not correct,
since the Clang Static Analyzer found a path of execution where
passing arguments in a weird way from Python resulted in strlen(NULL).

Formulated like this, Cython emits essentially:

c_buzhash(
 PyBytes_AS_STRING(data),
 PyObject_Length(data),
 ...
)

which is correct.
2017-06-14 19:16:36 +02:00
Marian Beermann
3b928a4558 fuse: refactor ItemCache 2017-06-14 13:58:09 +02:00
Marian Beermann
9fd79a9e56 fuse: decrypted cache 2017-06-14 13:35:07 +02:00
Marian Beermann
f04119c246 fuse: ItemCache on top of object cache 2017-06-14 13:35:07 +02:00
Marian Beermann
ec532304d2 fuse: remove unnecessary normpaths 2017-06-14 13:34:34 +02:00
enkore
8e477414ee Merge pull request #2675 from enkore/f/fusecache
fuse: instrumentation
2017-06-14 09:31:55 +02:00
Marian Beermann
ff05895b7e fuse: don't keep all Archive() instances around
they're only needed inside process_archive, and not needed in general
for pending_archives.
2017-06-13 23:16:28 +02:00
Marian Beermann
879f72f227 fuse: log process_archive timing
the easier alternative to "/bin/time stat mountpoint/<archive>/..."
2017-06-13 23:16:28 +02:00
Marian Beermann
c791921951 fuse: instrument caches
note: signal processing can be arbitrarily delayed;
Python processes signals as soon as the code returns into the interpreter
loop, which doesn't happen unless libfuse returns control, i.e.
some request has been sent to the file system.
2017-06-13 23:16:28 +02:00
enkore
ae162307f7 Merge pull request #2672 from enkore/f/fastinfo
info: use CacheSynchronizer & HashIndex.stats_against
2017-06-13 22:51:03 +02:00
Thomas Waldmann
ccd066f0af FUSE: fix negative uid/gid crash, fixes #2674
they could come into archives e.g. when backing up
external drives under cygwin.
2017-06-13 19:51:37 +02:00
Marian Beermann
e189a4d302 info: use CacheSynchronizer & HashIndex.stats_against 2017-06-13 14:34:10 +02:00
enkore
fd1efbac90 Merge pull request #2671 from enkore/f/chunkerub
chunker: don't do uint32_t >> 32
2017-06-13 13:13:33 +02:00
Marian Beermann
944a4abd58 chunker: don't do uint32_t >> 32 2017-06-13 11:42:43 +02:00
Marian Beermann
4490a8bbc3 cache sync: don't do memcpy(..., 0, 0)
!ctx->buf => ctx->tail - ctx->head == 0
2017-06-13 11:16:04 +02:00
Thomas Waldmann
0b00c14c27 don't write to disk with --stdout, fixes #2645
if we always give stdout to extract_item(), it gets into the stdout-
processing branch which only emits data from items that have chunks
and does nothing for items which don't.
2017-06-13 01:12:14 +02:00
Marian Beermann
c9c227f2ca cache sync: check Operation.READ compatibility with manifest 2017-06-12 23:46:49 +02:00
Marian Beermann
78cbf695c4 cache sync: suppress GCC C90/C99 int literal warning
warning: this decimal constant is unsigned only in ISO C90

Raised by GCC 4.9.2 on PowerPC.

The warning is bogus here due to the immediate explicit cast; newer
versions don't emit it.
2017-06-12 10:53:55 +02:00
enkore
148a8a855b Merge pull request #2663 from enkore/f/fcs-changes
cache sync changes
2017-06-11 21:27:25 +02:00
Marian Beermann
783a5926d6 cache sync: introduce BORG_NO_PYTHON
textshell edition
2017-06-11 20:23:17 +02:00
Marian Beermann
3c6372f841 cache sync: convert incoming integers to uint64_t 2017-06-11 20:20:00 +02:00
Marian Beermann
f786211b12 RepositoryCache: truncate+unlink errored file 2017-06-11 20:20:00 +02:00
enkore
49d74e12fc Merge pull request #2661 from enkore/f/consistent-key-names
normalize authenticated key modes
2017-06-11 20:01:32 +02:00
Marian Beermann
221dc1c4c7 normalize authenticated key modes
rename authenticated to authenticated-blake2, consistent with the other
blake2 key modes

add authenticated mode that fills the blank and is consistent with the
other "unqualified" key modes
2017-06-11 18:07:09 +02:00
Marian Beermann
5f5371f0b1 implement --glob-archives/-a 2017-06-11 12:15:12 +02:00
Marian Beermann
49ca3dca33 cache sync: move assert() behind declarations 2017-06-11 11:42:39 +02:00
textshell
86363dcd4b Merge pull request #2648 from textshell/feature/mandatory-features-master
Add minimal version of in repository mandatory feature flags. (master)
2017-06-10 17:50:28 +02:00
enkore
33a7331bda Merge pull request #2643 from enkore/f/experimental-patterns
mark --pattern, --patterns-from as experimental
2017-06-10 17:15:22 +02:00
enkore
13f396d5ad Merge pull request #2638 from enkore/f/fastcachesync-minify
Compact chunks.archive.d
2017-06-10 17:13:25 +02:00
enkore
d06ee5648c Merge pull request #2572 from enkore/f/fastcachesync
Improve cache sync speed
2017-06-10 17:12:51 +02:00
Martin Hostettler
e63808a63f Add tests for cache compatibility code. 2017-06-10 11:43:33 +02:00
Martin Hostettler
b8ad8b84da Cache: Wipe cache if compatibility is not sure
Add detection of possibly incompatible combinations
of the borg versions maintaining the cache and the featues used.
2017-06-10 11:42:48 +02:00
Martin Hostettler
005068dd6d Improve robustness of monkey patching borg.constants.PBKDF2_ITERATIONS. And add lots of warnings. 2017-06-10 11:42:42 +02:00
Marian Beermann
5cab72035a mark --pattern, --patterns-from as experimental 2017-06-10 10:29:02 +02:00
Marian Beermann
827c478500 nanorst for "borg help TOPIC" 2017-06-10 10:29:02 +02:00
Marian Beermann
0e31f78dd6 cache sync: avoid "l" and such as a variable name, note vanilla changes 2017-06-10 10:17:28 +02:00
Marian Beermann
5eb43b8464 cache sync: give overview of the source's structure 2017-06-10 10:17:28 +02:00
Marian Beermann
310a71e4f0 cache sync: use ro_buffer to accept bytes, memoryview, ... 2017-06-10 10:17:28 +02:00
Martin Hostettler
1176a1c4e4 permit manifest version 2 as well 1 one. 2017-06-09 23:22:18 +02:00
Martin Hostettler
193e8bcaef Add tests for mandatory repository feature flags. 2017-06-09 23:20:07 +02:00
Martin Hostettler
fd0250d34a Add minimal version of in repository mandatory feature flags.
This should allow us to make sure older borg versions can be cleanly
prevented from doing operations that are no longer safe because of
repository format evolution. This allows more fine grained control than
just incrementing the manifest version. So for example a change that
still allows new archives to be created but would corrupt the repository
when an old version tries to delete an archive or check the repository
would add the new feature to the check and delete set but leave it out
of the write set.

This is somewhat inspired by ext{2,3,4} which uses sets for
compat (everything except fsck), ro-compat (may only be accessed
read-only by older versions) and features (refuse all access).
2017-06-09 23:13:31 +02:00