Carl Schwan
b6313f68d3
perf(s3): Expose pre-signed urls for S3
...
This is faster than going back to nextcloud to download the files.
This is an opt-in setting that can be enabled by setting
use_presigned_url in the object store config.
Additionally add support for the proxy config which is needed in a
docker setup. See https://github.com/juliusknorr/nextcloud-docker-dev/pull/431
Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>
2025-12-09 11:21:35 +01:00
Daniel Kesselberg
f977a7fec6
fix(s3): make data integrity protections opt-in
...
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2025-12-02 21:03:26 +01:00
provokateurin
2ab29a69e0
feat(ObjectStore): Allow overriding arguments per bucket
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2025-12-01 16:06:05 +01:00
provokateurin
0c3a872908
feat(CertificateManager): Add option to specify the default certificates bundle path
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2025-12-01 10:47:23 +01:00
Kent Delante
96672e1c4d
feat: emit an event when an S3 bucket is created
...
Signed-off-by: Kent Delante <kent.delante@proton.me>
2025-11-24 19:37:18 +08:00
Andy Scherzinger
9e2f07b7d4
Merge pull request #56395 from nextcloud/carl/cache-bucket-exists
...
perf(s3): Cache whether bucket exists
2025-11-24 12:13:27 +01:00
Carl Schwan
a228ffa641
refactor(s3): Readability
...
Co-authored-by: Kate <26026535+provokateurin@users.noreply.github.com>
Signed-off-by: Carl Schwan <carl@carlschwan.eu>
2025-11-20 16:50:14 +01:00
nfebe
3030783785
feat(objectstore): add configurable S3 retry attempts
...
Add retriesMaxAttempts parameter to S3 objectstore configuration
to allow customization of AWS SDK retry behavior for handling
unreliable network conditions or proxy issues.
Defaults to 5 retries (AWS SDK default) if not specified.
Signed-off-by: nfebe <fenn25.fn@gmail.com>
2025-11-19 16:17:43 +01:00
Robin Appelman
a7d130c598
Revert "fix(ObjectStoreStorage): Encode original-path in writeStream by @frabe1579"
...
This reverts commit 4b4b39e7ec
2025-11-18 19:53:17 +01:00
Robin Appelman
8194be795d
fix: encode s3 metadata as base64 if needed
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-11-18 16:06:57 +01:00
Carl Schwan
67d0307e66
perf(s3): Cache whether bucket exists
...
Otherwise, we call doesBucketExist all the time which does a network
request to the S3 server adding some non-trivial latency when creating a
S3 connection object.
Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>
2025-11-12 14:17:48 +01:00
Kate
e3edf43e28
Merge pull request #56096 from nextcloud/fix/56077/s3-configure-checksum-defaults
CodeQL Advanced / Analyze (actions) (push) Waiting to run
CodeQL Advanced / Analyze (javascript-typescript) (push) Waiting to run
Integration sqlite / changes (push) Waiting to run
Integration sqlite / integration-sqlite (master, 8.4, main, --tags ~@large files_features) (push) Blocked by required conditions
Integration sqlite / integration-sqlite (master, 8.4, main, capabilities_features) (push) Blocked by required conditions
Integration sqlite / integration-sqlite (master, 8.4, main, collaboration_features) (push) Blocked by required conditions
Integration sqlite / integration-sqlite (master, 8.4, main, comments_features) (push) Blocked by required conditions
Integration sqlite / integration-sqlite (master, 8.4, main, dav_features) (push) Blocked by required conditions
Integration sqlite / integration-sqlite (master, 8.4, main, features) (push) Blocked by required conditions
Integration sqlite / integration-sqlite (master, 8.4, main, federation_features) (push) Blocked by required conditions
Integration sqlite / integration-sqlite (master, 8.4, main, file_conversions) (push) Blocked by required conditions
Integration sqlite / integration-sqlite (master, 8.4, main, filesdrop_features) (push) Blocked by required conditions
Integration sqlite / integration-sqlite (master, 8.4, main, ldap_features) (push) Blocked by required conditions
Integration sqlite / integration-sqlite (master, 8.4, main, openldap_features) (push) Blocked by required conditions
Integration sqlite / integration-sqlite (master, 8.4, main, openldap_numerical_features) (push) Blocked by required conditions
Integration sqlite / integration-sqlite (master, 8.4, main, remoteapi_features) (push) Blocked by required conditions
Integration sqlite / integration-sqlite (master, 8.4, main, routing_features) (push) Blocked by required conditions
Integration sqlite / integration-sqlite (master, 8.4, main, setup_features) (push) Blocked by required conditions
Integration sqlite / integration-sqlite (master, 8.4, main, sharees_features) (push) Blocked by required conditions
Integration sqlite / integration-sqlite (master, 8.4, main, sharing_features) (push) Blocked by required conditions
Integration sqlite / integration-sqlite (master, 8.4, main, theming_features) (push) Blocked by required conditions
Integration sqlite / integration-sqlite (master, 8.4, main, videoverification_features) (push) Blocked by required conditions
Integration sqlite / integration-sqlite-summary (push) Blocked by required conditions
Psalm static code analysis / static-code-analysis (push) Waiting to run
Psalm static code analysis / static-code-analysis-security (push) Waiting to run
Psalm static code analysis / static-code-analysis-ocp (push) Waiting to run
Psalm static code analysis / static-code-analysis-ncu (push) Waiting to run
2025-11-03 11:08:33 +01:00
Stephan Orbaugh
ef21c79368
Merge pull request #55717 from DaanSelen/fix-issue-55665
...
fix(ObjectStoreStorage): fix s3 path encoding (issue causing 55665)
2025-11-03 10:58:53 +01:00
Fiehe Christoph
b87add2711
fix(s3): expose request_checksum_calculation and response_checksum_validation
...
Fix https://github.com/nextcloud/server/issues/56077
This commit makes the configuration settings 'request_checksum_calculation' and 'response_checksum_validation' of the S3Client from the AWS SDK for PHP configurable.
Signed-off-by: Fiehe Christoph <c.fiehe@eurodata.de>
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2025-10-30 11:57:32 +01:00
Daan Selen
4b4b39e7ec
fix(ObjectStoreStorage): Encode original-path in writeStream by @frabe1579
...
Signed-off-by: Daan Selen <dselen@systemec.nl>
2025-10-15 11:53:39 +02:00
Thomas Citharel
5c8bdad828
fix(objectstore): add named parameter for like query
...
to avoid 'object::%' to be considered as a column
(`la colonne \\u00ab object::% \\u00bb n'existe pas` on PG)
Signed-off-by: Thomas Citharel <tcit@tcit.fr>
2025-10-13 16:18:18 +02:00
Louis Chmn
00ec57efd6
fix(S3): Only append streams if non-seekable
...
Later, when we rewind the stream in `writeMultiPart` during retry, both streams were rewinded, so the resulting stream was bigger than expected.
Inspired by https://github.com/aws/aws-sdk-php/blob/master/src/S3/ObjectUploader.php#L136-L146
Signed-off-by: Louis Chmn <louis@chmn.me>
2025-10-10 09:54:05 +02:00
Carl Schwan
b0357663b9
perf(preview): Optimize migration and simplify DB layout
...
* Simplify migration by not moving the actual files and just updating
the DB
* Don't store the storageid in the preview table as it is not needed
* Start adding tests
Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>
2025-10-06 13:37:15 +02:00
Carl Schwan
13c35c0f17
perf(preview): Migrate previews to the new optimized table
...
Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>
2025-10-06 13:37:15 +02:00
Carl Schwan
18fbacdd8d
perf(preview): Split preview data to new table
...
The new oc_previews table is optimized for storing previews and should
decrease significantly the space taken by previews in the filecache
table.
This attend to reuse the IObjectStore abstraction over S3/Swift/Azure
but currently only support one single bucket configuration.
Signed-off-by: Carl Schwan <carl.schwan@nextclound.com>
2025-10-06 13:37:15 +02:00
yemkareems
200f9dc39f
fix: small typo fix
...
Signed-off-by: yemkareems <yemkareems@gmail.com>
2025-09-29 09:12:38 +05:30
provokateurin
98081dfd42
feat(ObjectStore): Add option to limit the total size
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2025-09-10 08:15:28 +02:00
Robin Appelman
0c1d5e8c54
fix: ensure all object store configuration have distict bucket names
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-09-05 18:45:32 +02:00
Robin Appelman
02f4a82088
fix: make bucket mapper work with new multi-object-store config
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-28 17:51:25 +02:00
Robin Appelman
b3c53c7436
feat: allow object store configuration aliases for easier migrations
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-14 13:06:51 +02:00
Robin Appelman
2d4bba7b0c
feat: add command to get user objectstore config mappings
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-14 13:06:51 +02:00
Robin Appelman
385dd36ff8
feat: allow configuring multiple objectstore configurations
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-14 13:06:51 +02:00
Robin Appelman
97efc95efc
fix: better object store write error propagation
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-05 19:17:58 +02:00
Robin Appelman
83b8a390cd
fix: always do stream counting for object store upload
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-05 19:17:58 +02:00
Robin Appelman
398b106f0c
fix: validate written size for s3 multipart uploads
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-05 19:17:58 +02:00
Ferdinand Thiessen
5981b7eb51
chore: apply new CSFixer rules
...
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
# Conflicts:
# apps/settings/lib/SetupChecks/PhpOpcacheSetup.php
2025-07-01 16:26:50 +02:00
Robin Appelman
2e1c316582
fix: remove double cache write in ObjectStoreStorage::touch
...
the file_put_contents already puts all of those values into the cache
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-06-18 22:22:51 +02:00
Kent Delante
ea98e4b3f5
Merge pull request #53419 from nextcloud/leftybournes/fix/object_storage_large_uploads
...
fix(s3): retry failed multipart uploads with decreased concurrency
2025-06-18 21:07:06 +08:00
Daniel Calviño Sánchez
d7ae952dc8
fix(ObjectStore): Make S3 "connect_timeout" option configurable
...
The hardcoded connection timeout of 5 seconds may not be enough in some
cases, so now it is got from the ObjectStore arguments in Nextcloud
configuration, falling back to 5 if not set.
The connection timeout is set in seconds, but decimal precision can be
used for subsecond accuracy (for example, 4.2 for 4200 milliseconds).
Signed-off-by: Daniel Calviño Sánchez <danxuliu@gmail.com>
2025-06-16 20:17:50 +02:00
Kent Delante
06b5ce5859
fix(s3): retry failed multipart uploads with decreased concurrency
...
Signed-off-by: Kent Delante <kent.delante@proton.me>
2025-06-10 17:39:59 +08:00
Louis
f4cb78b905
Merge pull request #52242 from nextcloud/artonge/fix/copy_subfolders_s3
2025-06-02 11:55:54 +02:00
Robin Appelman
5b3f4cd8fe
perf: set the folder size in the same query as we create it
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-05-28 15:52:00 +02:00
Louis Chemineau
1f8b823992
fix(S3): Use original folder size during copy
...
This prevent having copied folders with a wrongly set size of 0KB.
- Fix https://github.com/nextcloud/server/issues/51916
Signed-off-by: Louis Chemineau <louis@chmn.me>
2025-05-28 15:52:00 +02:00
Robin Appelman
7599162c7b
feat: move primary object store configuration to a single place
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-05-28 15:32:31 +02:00
Kate
9592390070
Merge pull request #51779 from nextcloud/object-store-filename
2025-05-06 15:21:01 +02:00
Anna Larch
49baa914cf
fix(objectstorage): add retry attempts to S3 connection
...
Signed-off-by: Anna Larch <anna@nextcloud.com>
2025-04-23 12:15:06 +02:00
Louis Chemineau
8fdf2a7eae
fix: Transfer ownership with S3 as primary
...
When using S3 as primary storage, transferring ownership with the `--move` option fail with the following error:
`SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '8-45b963397aa40d4a0063e0d85e4fe7a1' for key 'fs_storage_path_hash'`
The `--move` option moves the entire home folder from one account to another.
The error means that the move failed because the destination folder already exist in `oc_filecache`.
- With S3 as primary storage, folders only exists as entries in `oc_filecache`.
- With S3 as primary storage, `moveFromStorage(...)` only moves the cache entry, as nothing needs to be moved on disk. This cache move does not delete potentially pre-existing destination folder.
- With Local storage, `moveFromStorage(...)` calls `rename(...)` which delete pre-existing folder.
- `transfer(...)`: 687a4d9ac7/apps/files/lib/Service/OwnershipTransferService.php (L112)687a4d9ac7/lib/private/Files/SetupManager.php (L261-L262)687a4d9ac7/lib/private/Files/ObjectStore/ObjectStoreStorage.php (L91-L135)687a4d9ac7/lib/private/Files/ObjectStore/ObjectStoreStorage.php (L635-L636)
2025-04-01 13:53:56 +02:00
Robin Appelman
8aae332622
feat: store original storage id and path in object store metadata
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-03-31 15:10:58 +02:00
Robin Appelman
94114b99f7
feat: more generic way of passing metadata to object storage backends for new objects
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-03-31 14:55:18 +02:00
Robin Appelman
fcde776683
feat: add command to list objects
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-03-31 14:30:40 +02:00
Robin Appelman
c3bc362f48
feat: add command to get object metadata
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-03-31 14:30:40 +02:00
Hector Manuel
12a92a2e8b
Merge branch 'master' into feat/support-aws-session-token
2025-03-04 18:27:54 +01:00
Benjamin Gaussorgues
9bdee76535
perf(files): faster query to fetch incomplete directories
...
Signed-off-by: Benjamin Gaussorgues <benjamin.gaussorgues@nextcloud.com>
2025-02-13 10:11:02 +01:00
Hector Valcarcel
6b4c859a41
feat(object_store): Add support for session token in AWS credentials
...
- Pass session token, either null or with value, to the AWS Credentials constructor
Signed-off-by: Hector Valcarcel <hmvalcarcel@gmail.com>
2025-02-06 16:24:19 +01:00
Richard Steinmetz
f1c025dfd3
fix(s3): treat empty sse_c_key as not set
...
Co-authored-by: Christian Becker <christian@dabecka.de>
Signed-off-by: Richard Steinmetz <richard@steinmetz.cloud>
2025-01-28 22:59:59 +01:00
