Commit graph

73 commits

Author SHA1 Message Date
Robin Appelman
3e94ee391f fix: create intermediate directories for objectstore moveFromStorage
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-18 13:54:55 +02:00
Robin Appelman
879fcc4253 fix: get source file before moving the cache item in objectstore moveFromStorage
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-18 13:54:55 +02:00
Stephan Orbaugh
f25acfb799
Merge pull request #47986 from cfiehe/fix_move_on_same_bucket
perf(ObjectStoreStorage): Improve (slow) move on same object bucket
2024-09-17 16:35:19 +02:00
provokateurin
d2d1e3284d
fix(Storage): Fix method docs inheritance
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-17 10:20:38 +02:00
Côme Chilliet
492e6997d8
chore: Fix psalm issues, put back private versions of getter in private Storage interface
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2024-09-16 18:11:38 +02:00
Christoph Fiehe
9597072ada perf(ObjectStoreStorage): Improve (slow) move on same object bucket
This commit fixes the issue #47856. When you upload a file into a group folder and when you use a single S3 bucket as primary storage, the final move operation hangs for a long time. In the background, Nextcloud initiates a copy-delete sequence from the bucket into the bucket, with causes a lot unnecessary overhead. Nextcloud thinks that the file must be imported to another storage and does not recognize that everything is done on the same object bucket. In that case, the import step can be completely skipped, which saves time, network bandwidth and reduces the load on the object storage.

The behavior improves a lot with https://github.com/nextcloud/server/pull/46013. However, there are still some put messages that are being sent to the object storage when you use an object storage as primary storage and upload files into a group folder.

Co-authored-by: Kate <26026535+provokateurin@users.noreply.github.com>
Signed-off-by: Christoph Fiehe <c.fiehe@eurodata.de>
2024-09-16 15:11:09 +02:00
Julius Härtl
6d2253e692
Merge pull request #46013 from nextcloud/obj-store-move-from-storage-preserve-fileid
fix: write object to the correct urn when moving from another storage to object store
2024-08-26 13:47:27 +02:00
Daniel Kesselberg
af6de04e9e
style: update codestyle for coding-standard 1.2.3
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2024-08-25 19:34:58 +02:00
Robin Appelman
bd740ac0b0
fix: write object to the correct urn when moving from another storage to object store
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-08-23 15:22:16 +02:00
Andy Scherzinger
dae7c159f7
chore: Add SPDX header
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
2024-05-24 13:11:22 +02:00
Côme Chilliet
ec5133b739 fix: Apply new coding standard to all files
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2024-04-02 14:16:21 +02:00
Joas Schilling
951c330d39
Revert "don't perform an explicit setup when using the file cache" 2024-03-07 14:38:02 +01:00
Robin Appelman
4fba4cd14c fix: fix user folder init
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-03-06 17:52:24 +01:00
Côme Chilliet
c0ce272e9c chore: Migrate away from OC::$server->getLogger
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2024-02-13 17:32:30 +01:00
Thomas Citharel
30798eb6c2
refactor(objectstorage): cleanup types
Signed-off-by: Thomas Citharel <tcit@tcit.fr>
2024-02-02 14:59:21 +01:00
Thomas Citharel
3be3dbdb3b
refactor(objectstorage): move $user property from ObjectStoreStorage class to HomeObjectStoreStorage
Signed-off-by: Thomas Citharel <tcit@tcit.fr>
2024-02-02 14:59:21 +01:00
Ferdinand Thiessen
5172baaf8b
fix(ObjectStore): Make copying behavior consistent with local storage
Drop file permissions on copy like we do on local storage.

Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2023-11-17 12:54:37 +01:00
Julius Härtl
88a96e4db7 fix: Only store unencrypted_size if path should be encrypted
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-09-06 07:25:28 +02:00
Robin Appelman
de85610385 use source cache when listing folder during recursive copy
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-06-19 16:40:15 +02:00
Robin Appelman
cc0ad25c98
don't always check if we need to setup the object store root
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-05-23 17:44:40 +02:00
Robin Appelman
2ea41dab93
repair -1 folder sizes for object store background scan
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-05-10 19:33:26 +02:00
Simon L
85c9e75f33
Merge pull request #32314 from nextcloud/object-mkdir-logging
additional logging when mkdir fails for object storage
2023-04-17 23:42:08 +02:00
Robin Appelman
c6be96d04f
additional logging when mkdir fails for object storage
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-04-05 14:52:19 +02:00
Robin Appelman
d63fa2d991
improve objectstore rmdir handling
remove folder entries as they are cleared instead of in one go afterwards

otherwise they stick around if some of the child entries can't be removed

Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-04-03 18:28:29 +02:00
Julius Härtl
e23aa8883e
feat(s3): Use multipart upload for chunked uploading
This allows to stream file chunks directly to S3 during upload.

Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-03-08 14:00:04 +01:00
Côme Chilliet
f5c361cf44
composer run cs:fix
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2023-01-20 11:45:08 +01:00
Maxence Lange
cc639a8877 compare cached filesize on download
Signed-off-by: Maxence Lange <maxence@artificial-owl.com>
2022-12-01 15:42:15 -01:00
Robin Appelman
a5036a973c allow disabling object store write check
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-10-26 18:45:59 +00:00
Carl Schwan
a5ea677370 Rename file1 and file2 to source and target in Storage abstraction
Signed-off-by: Carl Schwan <carl@carlschwan.eu>
2022-10-18 17:21:47 +02:00
Julius Härtl
0f0e74a78c Cleanup temporary files after finishing the write to object storage
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2022-06-02 12:57:18 +00:00
Carl Schwan
bfa60aaf27
Fix permissions when copying from ObjectStorage
Make sure that when a user copy a file from a directory they don't have
all permissions to a directory where they have more permissions, the
permissions are correctly set to the one from the parent taget folder.

This was caused by the ObjectStoreStorage::copyFromStorage using
the jailed storage and cache entry instead of the unjailed one like other
storages (the local one).
Steps to reproduce

+ Use object storage
+ Create a groupfolder with one group having full permission and another one
  who can just read files.
+ With an user who is in the second group, copy a file from the groupfolder to
  the home folder of this user.
+ The file in the home folder of the user will be read only and can't be deleted
  even though it is in their home folder and they are the owner. In oc_filecache,
  the permissions stored for this file are 1 (READ)

Signed-off-by: Carl Schwan <carl@carlschwan.eu>
2021-10-28 13:29:50 +02:00
Julius Härtl
21db463bc9
Reset checksum when writing files to object store
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2021-06-21 12:04:49 +02:00
John Molakvoæ (skjnldsv)
215aef3cbd
Update php licenses
Signed-off-by: John Molakvoæ (skjnldsv) <skjnldsv@protonmail.com>
2021-06-04 22:02:41 +02:00
Robin Appelman
ef6f2e68f0
explicitly close source stream on object store upload even if count wrapper isn't needed
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-04-29 17:01:19 +02:00
Robin Appelman
effb7dc8ba
set mimetype for objects uploaded to object storages
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-04-21 15:25:58 +02:00
Robin Appelman
ad636ae12d
use returned file id for object store copy
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-03-12 17:22:40 +01:00
Robin Appelman
c87b1a50d9 apply object store copy optimization when 'cross storage' copy is within the same object store
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-03-09 20:31:27 +01:00
Christoph Wurst
9ce3ea3368
Update license headers
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-12-30 14:07:05 +01:00
Tigran Mkrtchyan
4f2dc18f58 storage: update IStorage#file_put_contents docs to match usage
The current phpdoc of IStorage#file_put_contents doesnt corresponds to
it's actual usage in code, e.g.

Signed-off-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
2020-12-21 19:04:18 +01:00
Robin Appelman
9d4848e863
use in objectstore copy
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-11-24 15:16:58 +01:00
Roeland Jago Douma
bb06b6cce4
Fix reading empty files from objectstorage
Since we try to do range requests this will fail hard.
However since empty files are not that interesting to read anyways we
just read from an emptry memory stream.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-09-09 20:45:13 +02:00
Roeland Jago Douma
9fdeed8cc0
Run stat less often for objectstorages
When we want to get the permissions we now do stat at least 5 times for
each entry. Which is a bit much. Especially since the permssions are all
just in the database already.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-09-08 14:38:36 +02:00
Roeland Jago Douma
789b33aba4
Only update the filecache entry once the file has been written to S3
If we already update before we have no way to revert if the upload
fails.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-08-31 12:28:04 +02:00
Roeland Jago Douma
6ffd7173f9
Don't lose filecache entry on s3 overwrite error
If the object store errors we should not always delete the filecache
entry. As this might lead to people losing access to their files.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-08-31 12:25:20 +02:00
Christoph Wurst
734c62bee0
Format code according to PSR2
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:56:50 +02:00
Christoph Wurst
caff1023ea
Format control structures, classes, methods and function
To continue this formatting madness, here's a tiny patch that adds
unified formatting for control structures like if and loops as well as
classes, their methods and anonymous functions. This basically forces
the constructs to start on the same line. This is not exactly what PSR2
wants, but I think we can have a few exceptions with "our" style. The
starting of braces on the same line is pracrically standard for our
code.

This also removes and empty lines from method/function bodies at the
beginning and end.

Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 14:19:56 +02:00
Christoph Wurst
14c996d982
Use elseif instead of else if
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 10:35:09 +02:00
Christoph Wurst
1a9330cd69
Update the license headers for Nextcloud 19
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-03-31 14:52:54 +02:00
Christoph Wurst
b80ebc9674
Use the short array syntax, everywhere
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-03-26 16:34:56 +01:00
Roeland Jago Douma
9a35e789b4
Get correct mimetype on objectstores
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-03-11 11:14:12 +01:00