From 3d5a8016e6909db84a251b4ffeae9d28e781b95e Mon Sep 17 00:00:00 2001 From: Marian Beermann Date: Mon, 17 Jul 2017 00:06:02 +0200 Subject: [PATCH 1/5] docs: streamline installation: file systems --- docs/installation.rst | 29 ----------------------------- docs/usage_general.rst.inc | 10 ++++++++++ 2 files changed, 10 insertions(+), 29 deletions(-) diff --git a/docs/installation.rst b/docs/installation.rst index fb43f7213..c2f842f49 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -25,35 +25,6 @@ There are different ways to install |project_name|: Pre-Installation Considerations ------------------------------- -Repository File System -~~~~~~~~~~~~~~~~~~~~~~ - -- |project_name| stores data only 3 directory levels deep and uses short file and - directory names. -- |project_name| requires read and write permissions on the repository file system. -- |project_name| stores backup metadata and data into so-called segment files. The - target size of these files and also the count of these files per directory is set - in the :ref:`config-file`. -- |project_name| uses a generic and very portable mkdir-based `locking`_ mechanism. - POSIX locks, NFS locks, windows file locks, lockf(), flock() and hardlinks are - **not** used. -- Hardlinks are only required when performing an in-place upgrade of an Attic - repository. -- A journaling file system is strongly recommended. More information can be - found in :ref:`file-systems`. -- |project_name| requires the following file system operations: - - - create, open, read, write, seek, close, rename, delete - - link - when upgrading an Attic repo in-place - - listdir, stat - - fsync on files and directories to ensure data is written onto storage media - (some file systems do not support fsync on directories, which Borg accommodates for) - -:ref:`data-structures` contains additional information about how |project_name| -manages data. - -.. _locking: https://en.wikipedia.org/wiki/File_locking#Lock_files - (G)LIBC requirements -------------------- diff --git a/docs/usage_general.rst.inc b/docs/usage_general.rst.inc index 1aea4298d..77301b02b 100644 --- a/docs/usage_general.rst.inc +++ b/docs/usage_general.rst.inc @@ -236,6 +236,16 @@ If you are suspicious whether your Borg repository is still consistent and readable after one of the failures mentioned above occurred, run ``borg check --verify-data`` to make sure it is consistent. +.. rubric:: Requirements for Borg repository file systems + +- Long file names +- At least three directory levels with short names +- Typically, file sizes up to a few hundred MB. + Large repositories may require large files (>2 GB). +- Up to 10000 files per directory +- mkdir(2) should be atomic, since it is used for locking +- Hardlinks are needed for :ref:`borg_upgrade` ``--inplace`` + Units ~~~~~ From ab773956c19eb65ded62855740433b393b49ae25 Mon Sep 17 00:00:00 2001 From: Marian Beermann Date: Mon, 17 Jul 2017 00:10:05 +0200 Subject: [PATCH 2/5] docs: streamline installation (2) I can't give a good reason why anyone would care about this here. This is right before distribution packages, which are per definition compatible (sometimes tested, even). Instead, mention it where it applies. --- docs/installation.rst | 37 +++++++++++++------------------------ 1 file changed, 13 insertions(+), 24 deletions(-) diff --git a/docs/installation.rst b/docs/installation.rst index c2f842f49..026aad1c3 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -20,26 +20,6 @@ There are different ways to install |project_name|: have the latest code or use revision control (each release is tagged). -.. _installation-requirements: - -Pre-Installation Considerations -------------------------------- - -(G)LIBC requirements --------------------- - -Borg uses some filesytem functions from Python's `os` standard library module -with `follow_symlinks=False`. These are implemented since quite a while with -the non-symlink-following (g)libc functions like e.g. `lstat` or `lutimes` -(not: `stat` or `utimes`). - -Some stoneage systems (like RHEL/CentOS 5) and also Python interpreter binaries -compiled to be able to run on such systems (like Python installed via Anaconda) -might miss these functions and Borg won't be able to work correctly. -This issue will be detected early and Borg will abort with a fatal error. - -For the Borg binaries, there are additional (g)libc requirements, see below. - .. _distribution-package: Distribution Package @@ -110,8 +90,8 @@ Standalone Binary |project_name| binaries (generated with `pyinstaller`_) are available on the releases_ page for the following platforms: -* **Linux**: glibc >= 2.13 (ok for most supported Linux releases). Maybe older - glibc versions also work, if they are compatible to 2.13. +* **Linux**: glibc >= 2.13 (ok for most supported Linux releases). + Older glibc releases are untested and may not work. * **Mac OS X**: 10.10 (does not work with older OS X releases) * **FreeBSD**: 10.2 (unknown whether it works for older releases) @@ -214,6 +194,14 @@ For example, ntfs-3g on Linux isn't able to convey NTFS ACLs. From Source ----------- +.. note:: + + Some older Linux systems (like RHEL/CentOS 5) and Python interpreter binaries + compiled to be able to run on such systems (like Python installed via Anaconda) + might miss functions required by Borg. + + This issue will be detected early and Borg will abort with a fatal error. + Dependencies ~~~~~~~~~~~~ @@ -224,7 +212,7 @@ following dependencies first: the default Python version on most systems, it is usually available as an optional install. * OpenSSL_ >= 1.0.0, plus development headers. -* libacl_ (that pulls in libattr_ also), both plus development headers. +* libacl_ (which depends on libattr_), both plus development headers. * liblz4_, plus development headers. * some Python dependencies, pip will automatically install them for you * optionally, the llfuse_ Python package is required if you wish to mount an @@ -232,7 +220,8 @@ following dependencies first: * optionally libb2_. If it is not found a bundled implementation is used instead. If you have troubles finding the right package names, have a look at the -distribution specific sections below and also at the Vagrantfile in our repo. +distribution specific sections below or the Vagrantfile in the git repository, +which contains installation scripts for a number of operating systems. In the following, the steps needed to install the dependencies are listed for a selection of platforms. If your distribution is not covered by these From 735339c20949f528228ac1b8c16f73016d36e764 Mon Sep 17 00:00:00 2001 From: Marian Beermann Date: Mon, 17 Jul 2017 00:11:30 +0200 Subject: [PATCH 3/5] docs: streamline installation: platforms table --- docs/installation.rst | 72 -------------------------------------- docs/usage_general.rst.inc | 72 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+), 72 deletions(-) diff --git a/docs/installation.rst b/docs/installation.rst index 026aad1c3..de7a582aa 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -117,78 +117,6 @@ the old version using the same steps as shown above. .. _pyinstaller: http://www.pyinstaller.org .. _releases: https://github.com/borgbackup/borg/releases -.. _platforms: - -Features & platforms --------------------- - -Besides regular file and directory structures, |project_name| can preserve - -* Symlinks (stored as symlink, the symlink is not followed) -* Special files: - - * Character and block device files (restored via mknod) - * FIFOs ("named pipes") - * Special file *contents* can be backed up in ``--read-special`` mode. - By default the metadata to create them with mknod(2), mkfifo(2) etc. is stored. -* Hardlinked regular files, devices, FIFOs (considering all items in the same archive) -* Timestamps in nanosecond precision: mtime, atime, ctime -* Permissions: - - * IDs of owning user and owning group - * Names of owning user and owning group (if the IDs can be resolved) - * Unix Mode/Permissions (u/g/o permissions, suid, sgid, sticky) - -On some platforms additional features are supported: - -.. Yes/No's are grouped by reason/mechanism/reference. - -+------------------+----------+-----------+------------+ -| Platform | ACLs | xattr | Flags | -| | [#acls]_ | [#xattr]_ | [#flags]_ | -+==================+==========+===========+============+ -| Linux x86 | Yes | Yes | Yes [1]_ | -+------------------+ | | | -| Linux PowerPC | | | | -+------------------+ | | | -| Linux ARM | | | | -+------------------+----------+-----------+------------+ -| Mac OS X | Yes | Yes | Yes (all) | -+------------------+----------+-----------+ | -| FreeBSD | Yes | Yes | | -+------------------+----------+-----------+ | -| OpenBSD | n/a | n/a | | -+------------------+----------+-----------+ | -| NetBSD | n/a | No [2]_ | | -+------------------+----------+-----------+------------+ -| Solaris 11 | No [3]_ | n/a | -+------------------+ | | -| OpenIndiana | | | -+------------------+----------+-----------+------------+ -| Windows (cygwin) | No [4]_ | No | No | -+------------------+----------+-----------+------------+ - -Some Distributions (e.g. Debian) run additional tests after each release, these -are not reflected here. - -Other Unix-like operating systems may work as well, but have not been tested at all. - -Note that most of the platform-dependent features also depend on the file system. -For example, ntfs-3g on Linux isn't able to convey NTFS ACLs. - -.. [1] Only "nodump", "immutable", "compressed" and "append" are supported. - Feature request :issue:`618` for more flags. -.. [2] Feature request :issue:`1332` -.. [3] Feature request :issue:`1337` -.. [4] Cygwin tries to map NTFS ACLs to permissions with varying degress of success. - -.. [#acls] The native access control list mechanism of the OS. This normally limits access to - non-native ACLs. For example, NTFS ACLs aren't completely accessible on Linux with ntfs-3g. -.. [#xattr] extended attributes; key-value pairs attached to a file, mainly used by the OS. - This includes resource forks on Mac OS X. -.. [#flags] aka *BSD flags*. The Linux set of flags [1]_ is portable across platforms. - The BSDs define additional flags. - .. _source-install: From Source diff --git a/docs/usage_general.rst.inc b/docs/usage_general.rst.inc index 77301b02b..94b92a3c1 100644 --- a/docs/usage_general.rst.inc +++ b/docs/usage_general.rst.inc @@ -354,3 +354,75 @@ Network (only for client/server operation): operations used for transaction support also go over the connection. If you backup multiple sources to one target repository, additional traffic happens for cache resynchronization. + +.. _platforms: + +Support for file metadata +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Besides regular file and directory structures, |project_name| can preserve + +* Symlinks (stored as symlink, the symlink is not followed) +* Special files: + + * Character and block device files (restored via mknod) + * FIFOs ("named pipes") + * Special file *contents* can be backed up in ``--read-special`` mode. + By default the metadata to create them with mknod(2), mkfifo(2) etc. is stored. +* Hardlinked regular files, devices, FIFOs (considering all items in the same archive) +* Timestamps in nanosecond precision: mtime, atime, ctime +* Permissions: + + * IDs of owning user and owning group + * Names of owning user and owning group (if the IDs can be resolved) + * Unix Mode/Permissions (u/g/o permissions, suid, sgid, sticky) + +On some platforms additional features are supported: + +.. Yes/No's are grouped by reason/mechanism/reference. + ++------------------+----------+-----------+------------+ +| Platform | ACLs | xattr | Flags | +| | [#acls]_ | [#xattr]_ | [#flags]_ | ++==================+==========+===========+============+ +| Linux x86 | Yes | Yes | Yes [1]_ | ++------------------+ | | | +| Linux PowerPC | | | | ++------------------+ | | | +| Linux ARM | | | | ++------------------+----------+-----------+------------+ +| Mac OS X | Yes | Yes | Yes (all) | ++------------------+----------+-----------+ | +| FreeBSD | Yes | Yes | | ++------------------+----------+-----------+ | +| OpenBSD | n/a | n/a | | ++------------------+----------+-----------+ | +| NetBSD | n/a | No [2]_ | | ++------------------+----------+-----------+------------+ +| Solaris 11 | No [3]_ | n/a | ++------------------+ | | +| OpenIndiana | | | ++------------------+----------+-----------+------------+ +| Windows (cygwin) | No [4]_ | No | No | ++------------------+----------+-----------+------------+ + +Some Distributions (e.g. Debian) run additional tests after each release, these +are not reflected here. + +Other Unix-like operating systems may work as well, but have not been tested at all. + +Note that most of the platform-dependent features also depend on the file system. +For example, ntfs-3g on Linux isn't able to convey NTFS ACLs. + +.. [1] Only "nodump", "immutable", "compressed" and "append" are supported. + Feature request :issue:`618` for more flags. +.. [2] Feature request :issue:`1332` +.. [3] Feature request :issue:`1337` +.. [4] Cygwin tries to map NTFS ACLs to permissions with varying degress of success. + +.. [#acls] The native access control list mechanism of the OS. This normally limits access to + non-native ACLs. For example, NTFS ACLs aren't completely accessible on Linux with ntfs-3g. +.. [#xattr] extended attributes; key-value pairs attached to a file, mainly used by the OS. + This includes resource forks on Mac OS X. +.. [#flags] aka *BSD flags*. The Linux set of flags [1]_ is portable across platforms. + The BSDs define additional flags. From cc93cbb202204fb7384edae95d2951dc8174d5f4 Mon Sep 17 00:00:00 2001 From: Marian Beermann Date: Mon, 17 Jul 2017 00:14:58 +0200 Subject: [PATCH 4/5] docs: usage; remove strange blurb about tested platforms --- docs/usage_general.rst.inc | 27 ++++++++++----------------- 1 file changed, 10 insertions(+), 17 deletions(-) diff --git a/docs/usage_general.rst.inc b/docs/usage_general.rst.inc index 94b92a3c1..1fdb96165 100644 --- a/docs/usage_general.rst.inc +++ b/docs/usage_general.rst.inc @@ -360,21 +360,21 @@ Network (only for client/server operation): Support for file metadata ~~~~~~~~~~~~~~~~~~~~~~~~~ -Besides regular file and directory structures, |project_name| can preserve +Besides regular file and directory structures, Borg can preserve -* Symlinks (stored as symlink, the symlink is not followed) -* Special files: +* symlinks (stored as symlink, the symlink is not followed) +* special files: - * Character and block device files (restored via mknod) + * character and block device files (restored via mknod) * FIFOs ("named pipes") - * Special file *contents* can be backed up in ``--read-special`` mode. + * special file *contents* can be backed up in ``--read-special`` mode. By default the metadata to create them with mknod(2), mkfifo(2) etc. is stored. -* Hardlinked regular files, devices, FIFOs (considering all items in the same archive) -* Timestamps in nanosecond precision: mtime, atime, ctime -* Permissions: +* hardlinked regular files, devices, FIFOs (considering all items in the same archive) +* timestamps in nanosecond precision: mtime, atime, ctime +* permissions: * IDs of owning user and owning group - * Names of owning user and owning group (if the IDs can be resolved) + * names of owning user and owning group (if the IDs can be resolved) * Unix Mode/Permissions (u/g/o permissions, suid, sgid, sticky) On some platforms additional features are supported: @@ -385,11 +385,7 @@ On some platforms additional features are supported: | Platform | ACLs | xattr | Flags | | | [#acls]_ | [#xattr]_ | [#flags]_ | +==================+==========+===========+============+ -| Linux x86 | Yes | Yes | Yes [1]_ | -+------------------+ | | | -| Linux PowerPC | | | | -+------------------+ | | | -| Linux ARM | | | | +| Linux | Yes | Yes | Yes [1]_ | +------------------+----------+-----------+------------+ | Mac OS X | Yes | Yes | Yes (all) | +------------------+----------+-----------+ | @@ -406,9 +402,6 @@ On some platforms additional features are supported: | Windows (cygwin) | No [4]_ | No | No | +------------------+----------+-----------+------------+ -Some Distributions (e.g. Debian) run additional tests after each release, these -are not reflected here. - Other Unix-like operating systems may work as well, but have not been tested at all. Note that most of the platform-dependent features also depend on the file system. From b3e0dd37619e980d3319c5adb487da32f3bb9908 Mon Sep 17 00:00:00 2001 From: Marian Beermann Date: Sun, 23 Jul 2017 16:03:45 +0200 Subject: [PATCH 5/5] docs: file system requirements, update segs per dir --- docs/usage_general.rst.inc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/usage_general.rst.inc b/docs/usage_general.rst.inc index 1fdb96165..3c6191438 100644 --- a/docs/usage_general.rst.inc +++ b/docs/usage_general.rst.inc @@ -242,7 +242,7 @@ and readable after one of the failures mentioned above occurred, run - At least three directory levels with short names - Typically, file sizes up to a few hundred MB. Large repositories may require large files (>2 GB). -- Up to 10000 files per directory +- Up to 1000 files per directory (10000 for repositories initialized with Borg 1.0) - mkdir(2) should be atomic, since it is used for locking - Hardlinks are needed for :ref:`borg_upgrade` ``--inplace``