Commit graph

1517 commits

Author SHA1 Message Date
Louis
64583a0f86
Merge pull request #48985 from nextcloud/backport/47986/stable28
[stable28] perf(ObjectStoreStorage): Improve (slow) move on same object bucket
2024-10-31 15:57:22 +01:00
Christoph Fiehe
dec5a5fe9b perf(ObjectStoreStorage): Improve (slow) move on same object bucket
This commit fixes the issue #47856. When you upload a file into a group folder and when you use a single S3 bucket as primary storage, the final move operation hangs for a long time. In the background, Nextcloud initiates a copy-delete sequence from the bucket into the bucket, with causes a lot unnecessary overhead. Nextcloud thinks that the file must be imported to another storage and does not recognize that everything is done on the same object bucket. In that case, the import step can be completely skipped, which saves time, network bandwidth and reduces the load on the object storage.

The behavior improves a lot with https://github.com/nextcloud/server/pull/46013. However, there are still some put messages that are being sent to the object storage when you use an object storage as primary storage and upload files into a group folder.

Co-authored-by: Kate <26026535+provokateurin@users.noreply.github.com>
Signed-off-by: Christoph Fiehe <c.fiehe@eurodata.de>
2024-10-29 15:53:48 +00:00
Daniel Calviño Sánchez
e18de3aac0 fix: Fix copying or moving from shared groupfolders
When copying or moving between two local storages the source path (on
disk) to copy or move from is got from the unjailed path of the source
storage. However, if the source storage has more than one jail getting
the unjailed path resolves the most external jail, but the source path
needs to be got from the most internal jail instead (the one closer to
the local storage).

This can happen, for example, with a shared groupfolder: in that case
there is an external jail for the shared storage, and one internal jail
for the groupfolder storage wrapped by the shared storage.

Signed-off-by: Daniel Calviño Sánchez <danxuliu@gmail.com>
2024-10-29 10:36:33 +00:00
Julius Knorr
6b870fd8fa fix: Allow overriding shouldApplyQuota check from child classes
Signed-off-by: Julius Knorr <jus@bitgrid.net>
2024-10-09 16:41:29 +00:00
Robin Appelman
49a756b9b9 fix: rework move into object store to better preserve fileids
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-10-08 18:14:26 +02:00
Robin Appelman
99d62bf368 fix: preserve fileid when moving from objectstore to non-objectstore
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-10-08 18:14:26 +02:00
Robin Appelman
5d36da3717 fix: ensure source folder is removed from cache when moving to objectstore
otherwise this causes confusion down the line as it's contents will be moved to the new cache

Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-10-08 18:14:26 +02:00
Robin Appelman
6421581c00 fix: ensure parent folder exists when writing a file to object storage
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-10-08 18:14:26 +02:00
Robin Appelman
78637be4cd fix: verify that parent exists in cache when inserting
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-10-08 18:14:26 +02:00
Robin Appelman
0dacf09779 fix: fix object store id for test object store
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-10-08 18:14:26 +02:00
Robin Appelman
e0f7dec5b6 fix: cleanup objectstore file_put_content
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-10-08 18:14:26 +02:00
Robin Appelman
e794d04f57 fix: create intermediate directories for objectstore moveFromStorage
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-10-08 18:14:26 +02:00
Robin Appelman
fe8ac42c90 fix: get source file before moving the cache item in objectstore moveFromStorage
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-10-08 18:14:26 +02:00
Daniel Kesselberg
a22ee3ebb0
fix(s3): suppress php deprecation warning on php8
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2024-09-26 12:40:19 +02:00
John Molakvoæ
063a726280 chore: improve hash_file php usage in Local Storage
Co-authored-by: Ferdinand Thiessen <opensource@fthiessen.de>
Signed-off-by: John Molakvoæ <skjnldsv@users.noreply.github.com>
2024-09-16 15:16:56 +00:00
Louis Chemineau
3dc0952551 fix(files): Ensure that the hash method does not return null
To match beececf660/lib/private/Files/View.php (L1050)

- Fix https://github.com/nextcloud/server/issues/44110

Signed-off-by: Louis Chemineau <louis@chmn.me>
2024-09-16 15:16:56 +00:00
John Molakvoæ
46a15f2915
Merge pull request #47861 from nextcloud/backport/47860/stable28 2024-09-15 22:09:21 +02:00
provokateurin
6e1be06e97 fix(Storage\Local): Do not call getSourcePath() on SplFileInfo
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-15 14:51:49 +00:00
provokateurin
aff9322287 fix(files): Adjust Cache::searchQuery() parameter name to match interface
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-13 17:00:21 +00:00
Julius Knorr
fa1f0022fa
fix: Only write once to template instead of create/copy
Signed-off-by: Julius Knorr <jus@bitgrid.net>
2024-09-10 10:13:20 +02:00
provokateurin
e77ac5c8e1 fix(files): Create non-existent parents of mountpoints
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-04 18:01:36 +00:00
Julius Härtl
53f92e68e6 fix: Pass the mountpoint target user to storages without owner
Storages that do not have a dedicated owner (e.g. groupfolders, external
storages) currently always assume the current session user as the owner.
This leads to several issues when there is no user session but a node is
obtained through a user folder.

In order to have the correct user available we need to pass the user
that is used to setup a mountpoint along to the storage layer as we
generally assume that an owner is available for those.

Signed-off-by: Julius Härtl <jus@bitgrid.net>
2024-09-03 12:23:37 +02:00
Robin Appelman
c2af3e04b0 fix: use mountpoint from storage to find the encryption keys
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-08-20 14:18:16 +00:00
Robin Appelman
52da5f715f feat: store the mountpoint of storages in the mount options
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-08-20 14:18:16 +00:00
Côme Chilliet
a59176b7dc fix(encryption): Fix mountpoint check to accept if several are found
There is no strong requirement to have only one mount for a given
 storage id. Also the error in this case would be misleading.

Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2024-08-07 18:59:42 +00:00
Ferdinand Thiessen
e102c752f9 fix: OCP\Files\Node\Folder::search was not setting the owner
The owner was not set on the file info causing e.g. webdav searches to never return the known owner.

Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2024-07-30 11:05:22 +00:00
Julius Härtl
ce6768027e fix: Do not return partial file info if we have a cache entry
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2024-07-25 17:40:12 +00:00
Ferdinand Thiessen
1f1d625212 fix: FileInfo from View should have the correct name of a mountpoint
If a mountpoint is returned from a `View` the name must match the view scoped name
and not the global name.
For example group folders have a name like `1` or `2` (the ID),
but the name of the FileInfo returned fro mthe View should be the mount point name,
like `First groupfolder` or similar.

Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2024-07-24 13:25:05 +00:00
Git'Fellow
18ffec158c fix(s3): Don't wait indefinitely for S3 to return
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

fix: lint

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

fix: use AwsException

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

fix: Throw on connection failure

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

fix: Wrap all in try catch block

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

fix: use RequestTimeout error message

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

log: use OCP Server class

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

fix: Handle connect timeout only

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

fix: Handle errors more generically

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
2024-07-09 07:50:39 +00:00
Andy Scherzinger
e1af6af627
Merge pull request #45839 from nextcloud/backport/45649/stable28
[stable28] fix(ObjectStore): Fix regression when hostname is empty
2024-06-13 18:28:50 +02:00
Andy Scherzinger
bdf8dc3737
Merge pull request #45857 from nextcloud/backport/45172/stable28
[stable28] feat: add option to disable scanner transactions
2024-06-13 18:26:49 +02:00
Robin Appelman
e93017d308
feat: add option to disable scanner transactions
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-06-13 14:11:18 +02:00
Robin Appelman
4314997a2a fix: get child ids for folder in a separate query during move
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-06-13 09:45:20 +00:00
Josh
d285adee2a fix(ObjectStore): handle empty S3 hostname
Fixes #45637

The support for s3-accelerate added in #44496 introduced a regression in AWS S3 environments when `hostname` is blank (which is a valid configuration w/ AWS since the hostname gets auto-generated).

Signed-off-by: Josh <josh.t.richards@gmail.com>
2024-06-13 11:22:58 +02:00
Louis Chemineau
536d1cc26a fix(files): Use isRetryable to catch retryable exceptions
Signed-off-by: Louis Chemineau <louis@chmn.me>
2024-06-12 08:14:11 +00:00
John Molakvoæ
378833e343
Merge pull request #45528 from nextcloud/backport/45014/stable28 2024-06-07 12:01:04 +02:00
Josh
5ebd5626c5
Merge pull request #45059 from nextcloud/backport/44496/stable28
[stable28] feat: support "s3-accelerate" endpoint
2024-06-03 17:19:37 -04:00
Git'Fellow
4df77c6fa1 Try create folder first and check if it exists after
Co-authored-by: Benjamin Gaussorgues <benjamin.gaussorgues@nextcloud.com>
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
2024-05-30 18:30:37 +00:00
Git'Fellow
fe459f9d74 fix(files): Try to create dir only if it not exists
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
2024-05-30 18:30:37 +00:00
Christoph Wurst
66898f82f1 perf(db): Sort data for IN before chunking
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2024-05-29 08:39:44 -04:00
Git'Fellow
91c1d0bf7e
Fix backport
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
2024-05-28 11:30:00 +02:00
Robin Appelman
d1e53c64cc fix: forbid moving a folder into a subfolder of itself
Signed-off-by: Robin Appelman <robin@icewind.nl>

[skip ci]
2024-05-27 12:31:49 +00:00
Josh
b62901131f fix(Files): Change how scanner diffs for changed metadata
Fixes #43408 

Signed-off-by: Josh <josh.t.richards@gmail.com>
2024-05-13 09:06:41 +00:00
Robin Appelman
2f0b200665 feat: add option to enable s3 acceleration
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-05-10 10:03:40 -04:00
Joas Schilling
0095ce7723
fix(CI): Suppress psalm UndefinedClass
Signed-off-by: Joas Schilling <coding@schilljs.com>
2024-04-18 09:32:35 +02:00
Joas Schilling
1dcc48ac4f fix(files): Also skip cross storage move with access control
Signed-off-by: Joas Schilling <coding@schilljs.com>
2024-04-17 14:32:55 +00:00
Robin Appelman
4bd5b49077 fix: use proper jailed patch in watcher
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-04-10 01:21:01 +02:00
Julius Härtl
5ec8f0b075 perf: Avoid updating the folder size if we know the size difference
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2024-04-05 12:57:39 +00:00
Louis Chemineau
ad2b367dbd feat(files)): Add retry logic to cover deadlock situations when moving many files
Signed-off-by: Louis Chemineau <louis@chmn.me>
2024-03-28 13:01:38 +00:00
Stephan Orbaugh
08444f45f1
Merge pull request #44320 from nextcloud/backport/39990/stable28
[stable28] add some recrusive detection/prevention
2024-03-25 15:27:27 +01:00