upstream/nextcloud
1
0
Fork 0
mirror of https://github.com/nextcloud/server.git synced 2026-02-23 18:05:02 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions 12
81915 commits 675 branches 1186 tags 6.7 GiB
Commit graph

310 commits

Author SHA1 Message Date
nfebe
8d5dcbce79
feat(objectstore): add configurable S3 retry attempts 
Add retriesMaxAttempts parameter to S3 objectstore configuration
to allow customization of AWS SDK retry behavior for handling
unreliable network conditions or proxy issues.

Defaults to 5 retries (AWS SDK default) if not specified.

Signed-off-by: nfebe <fenn25.fn@gmail.com>
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
 2025-12-03 12:53:39 +01:00
provokateurin
e67c8e739e feat(ObjectStore): Allow overriding arguments per bucket 
Signed-off-by: provokateurin <kate@provokateurin.de>
 2025-12-01 16:44:41 +00:00
provokateurin
3140daa1aa
feat(CertificateManager): Add option to specify the default certificates bundle path 
Signed-off-by: provokateurin <kate@provokateurin.de>
 2025-12-01 11:57:26 +01:00
Kent Delante
8706dadcf0
Merge pull request #56616 from nextcloud/backport/56565/stable31
Some checks are pending
Integration sqlite / changes (push) Waiting to run
Details
Integration sqlite / integration-sqlite (stable31, 8.1, stable31, --tags ~@large files_features) (push) Blocked by required conditions
Details
Integration sqlite / integration-sqlite (stable31, 8.1, stable31, capabilities_features) (push) Blocked by required conditions
Details
Integration sqlite / integration-sqlite (stable31, 8.1, stable31, collaboration_features) (push) Blocked by required conditions
Details
Integration sqlite / integration-sqlite (stable31, 8.1, stable31, comments_features) (push) Blocked by required conditions
Details
Integration sqlite / integration-sqlite (stable31, 8.1, stable31, dav_features) (push) Blocked by required conditions
Details
Integration sqlite / integration-sqlite (stable31, 8.1, stable31, features) (push) Blocked by required conditions
Details
Integration sqlite / integration-sqlite (stable31, 8.1, stable31, federation_features) (push) Blocked by required conditions
Details
Integration sqlite / integration-sqlite (stable31, 8.1, stable31, file_conversions) (push) Blocked by required conditions
Details
Integration sqlite / integration-sqlite (stable31, 8.1, stable31, files_reminders) (push) Blocked by required conditions
Details
Integration sqlite / integration-sqlite (stable31, 8.1, stable31, filesdrop_features) (push) Blocked by required conditions
Details
Integration sqlite / integration-sqlite (stable31, 8.1, stable31, ldap_features) (push) Blocked by required conditions
Details
Integration sqlite / integration-sqlite (stable31, 8.1, stable31, openldap_features) (push) Blocked by required conditions
Details
Integration sqlite / integration-sqlite (stable31, 8.1, stable31, openldap_numerical_features) (push) Blocked by required conditions
Details
Integration sqlite / integration-sqlite (stable31, 8.1, stable31, remoteapi_features) (push) Blocked by required conditions
Details
Integration sqlite / integration-sqlite (stable31, 8.1, stable31, setup_features) (push) Blocked by required conditions
Details
Integration sqlite / integration-sqlite (stable31, 8.1, stable31, sharees_features) (push) Blocked by required conditions
Details
Integration sqlite / integration-sqlite (stable31, 8.1, stable31, sharing_features) (push) Blocked by required conditions
Details
Integration sqlite / integration-sqlite (stable31, 8.1, stable31, theming_features) (push) Blocked by required conditions
Details
Integration sqlite / integration-sqlite (stable31, 8.1, stable31, videoverification_features) (push) Blocked by required conditions
Details
Integration sqlite / integration-sqlite-summary (push) Blocked by required conditions
Details
Psalm static code analysis / static-code-analysis (push) Waiting to run
Details
Psalm static code analysis / static-code-analysis-security (push) Waiting to run
Details
Psalm static code analysis / static-code-analysis-ocp (push) Waiting to run
Details
Psalm static code analysis / static-code-analysis-ncu (push) Waiting to run
Details 
[stable31] feat: emit an event when an S3 bucket is created
 2025-11-27 10:26:42 +08:00
Kent Delante
bf4c094741 feat: emit an event when an S3 bucket is created 
Signed-off-by: Kent Delante <kent.delante@proton.me>
 2025-11-25 20:16:21 +08:00
Robin Appelman
d1a760d618
fix: ensure all object store configuration have distict bucket names 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-11-20 15:36:14 +01:00
Robin Appelman
6d487ac990
fix: make bucket mapper work with new multi-object-store config 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-11-20 15:36:13 +01:00
Robin Appelman
f21dcac9ef
feat: allow object store configuration aliases for easier migrations 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-11-20 15:36:11 +01:00
Robin Appelman
832c77ab52
feat: add command to get user objectstore config mappings 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-11-20 15:16:43 +01:00
Robin Appelman
ff74df3528
feat: allow configuring multiple objectstore configurations 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-11-20 15:16:39 +01:00
Louis Chmn
3cfc34ecfa fix(S3): Only append streams if non-seekable 
Later, when we rewind the stream in `writeMultiPart` during retry, both streams were rewinded, so the resulting stream was bigger than expected.

Inspired by https://github.com/aws/aws-sdk-php/blob/master/src/S3/ObjectUploader.php#L136-L146

Signed-off-by: Louis Chmn <louis@chmn.me>
 2025-10-10 12:43:19 +00:00
Robin Appelman
6842d0746d
Merge pull request #55288 from nextcloud/backport/52816/stable31 
[stable31] feat: move primary object store configuration to a single place
 2025-09-30 10:35:42 +02:00
yemkareems
d615109126 fix: small typo fix 
Signed-off-by: yemkareems <yemkareems@gmail.com>
 2025-09-29 12:24:43 +00:00
Robin Appelman
7162c3e8ed
feat: move primary object store configuration to a single place 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-09-24 18:39:21 +02:00
Robin Appelman
05e64418a7
fix: better object store write error propagation 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-08-07 13:48:43 +02:00
Robin Appelman
d1af0f34a2
fix: always do stream counting for object store upload 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-08-07 13:48:42 +02:00
Robin Appelman
902cb3dbb9
fix: validate written size for s3 multipart uploads 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-08-07 13:48:42 +02:00
Robin Appelman
e483387189
feat: more generic way of passing metadata to object storage backends for new objects 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-08-07 13:48:37 +02:00
Kent Delante
f40d14abac fix(s3): retry failed multipart uploads with decreased concurrency 
Signed-off-by: Kent Delante <kent.delante@proton.me>
 2025-06-20 18:24:16 +08:00
Daniel Calviño Sánchez
019b535868 fix(ObjectStore): Make S3 "connect_timeout" option configurable 
The hardcoded connection timeout of 5 seconds may not be enough in some
cases, so now it is got from the ObjectStore arguments in Nextcloud
configuration, falling back to 5 if not set.

The connection timeout is set in seconds, but decimal precision can be
used for subsecond accuracy (for example, 4.2 for 4200 milliseconds).

Signed-off-by: Daniel Calviño Sánchez <danxuliu@gmail.com>
 2025-06-17 03:30:55 +00:00
Robin Appelman
edfb7b2393 perf: set the folder size in the same query as we create it 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-06-02 09:56:29 +00:00
Louis Chemineau
9d26571eb2 fix(S3): Use original folder size during copy 
This prevent having copied folders with a wrongly set size of 0KB.

- Fix https://github.com/nextcloud/server/issues/51916

Signed-off-by: Louis Chemineau <louis@chmn.me>
 2025-06-02 09:56:29 +00:00
Robin Appelman
780766b63c fix: keep IObjectStoreMetaData in private namespace for backports 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-05-16 10:41:27 +02:00
Robin Appelman
f0cab800ab feat: add command to list objects 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-05-16 10:41:27 +02:00
Robin Appelman
a324dbec2e feat: add command to get object metadata 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-05-16 10:41:27 +02:00
Anna Larch
d393e98c9e fix(objectstorage): add retry attempts to S3 connection 
Signed-off-by: Anna Larch <anna@nextcloud.com>
 2025-04-23 18:40:23 +00:00
Louis Chemineau
27ed921ad5 fix: Transfer ownership with S3 as primary 
When using S3 as primary storage, transferring ownership with the `--move` option fail with the following error:

`SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '8-45b963397aa40d4a0063e0d85e4fe7a1' for key 'fs_storage_path_hash'`

The `--move` option moves the entire home folder from one account to another.
The error means that the move failed because the destination folder already exist in `oc_filecache`.

- With S3 as primary storage, folders only exists as entries in `oc_filecache`.
- With S3 as primary storage, `moveFromStorage(...)` only moves the cache entry, as nothing needs to be moved on disk. This cache move does not delete potentially pre-existing destination folder.
- With Local storage, `moveFromStorage(...)` calls `rename(...)` which delete pre-existing folder.

- `transfer(...)`: 687a4d9ac7/apps/files/lib/Service/OwnershipTransferService.php (L112)
- `oneTimeUserSetup(...)`: 687a4d9ac7/lib/private/Files/SetupManager.php (L261-L262)
- `mkdir(...)`: 687a4d9ac7/lib/private/Files/ObjectStore/ObjectStoreStorage.php (L91-L135)
- `moveFromStorage(...)`: 687a4d9ac7/lib/private/Files/ObjectStore/ObjectStoreStorage.php (L635-L636)

Delete pre-existing folder in `moveFromStorage(...)`

Signed-off-by: Louis Chemineau <louis@chmn.me>
 2025-04-16 10:19:17 +02:00
Benjamin Gaussorgues
d056e4dc74 perf(files): faster query to fetch incomplete directories 
Signed-off-by: Benjamin Gaussorgues <benjamin.gaussorgues@nextcloud.com>
 2025-02-13 14:24:44 +00:00
Richard Steinmetz
936ef804cd fix(s3): treat empty sse_c_key as not set 
Co-authored-by: Christian Becker <christian@dabecka.de>
Signed-off-by: Richard Steinmetz <richard@steinmetz.cloud>
 2025-01-29 17:35:52 +00:00
Ferdinand Thiessen
26c93519f2 fix: Harden files scanner for invalid null access 
Co-authored-by: Ferdinand Thiessen <opensource@fthiessen.de>
Co-authored-by: Kate <26026535+provokateurin@users.noreply.github.com>
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
 2025-01-28 19:49:03 +00:00
Robin Appelman
5896f8c37f
chore: cleanup leftover debug statement 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-12-09 15:40:01 +01:00
Robin Appelman
6cf66f95ce
fix: get object size from stream where possible 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-12-05 15:16:22 +01:00
Robin Appelman
ca547bb913
fix: don't perform the extra buffering in s3 stream write when the stream size is known 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-12-05 14:07:40 +01:00
Robin Appelman
7da8951c57
fix: use php://temp instead of php://memory for multi-part upload buffer 
this should reduce potential memory issues if the limit is set very high

Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-12-05 14:07:37 +01:00
Robin Appelman
df3a9e221d
fix: throw correct exception type when we can't verify if an s3 bucket exists 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-11-26 16:49:55 +01:00
provokateurin
0de4843b73 refactor(Storage): Align all Storage constructors 
Signed-off-by: provokateurin <kate@provokateurin.de>
 2024-10-23 13:24:18 +02:00
provokateurin
f28e74b7a8
refactor(Storage): Make all parameter types strong types 
Signed-off-by: provokateurin <kate@provokateurin.de>
 2024-10-07 15:00:05 +02:00
Robin Appelman
3e12e1e789
fix: rework move into object store to better preserve fileids 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-09-27 14:46:08 +02:00
Robin Appelman
888d06dff9
fix: preserve fileid when moving from objectstore to non-objectstore 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-09-27 14:45:42 +02:00
Robin Appelman
da21acfb3f
fix: ensure source folder is removed from cache when moving to objectstore 
otherwise this causes confusion down the line as it's contents will be moved to the new cache

Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-09-27 14:45:42 +02:00
provokateurin
7cdccd058f
fix(Storage): Fix IStorage return types 
Signed-off-by: provokateurin <kate@provokateurin.de>
 2024-09-26 18:13:03 +02:00
Robin Appelman
47d281865e perf: reuse cache info we already have when moving to object store 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-09-19 19:06:06 +02:00
provokateurin
9836e9b164
chore(deps): Update nextcloud/coding-standard to v1.3.1 
Signed-off-by: provokateurin <kate@provokateurin.de>
 2024-09-19 14:21:20 +02:00
Robin Appelman
3c0c81bb94 fix: ensure parent folder exists when writing a file to object storage 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-09-18 14:11:27 +02:00
Robin Appelman
e0fb25f846 fix: fix object store id for test object store 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-09-18 13:54:55 +02:00
Robin Appelman
60f6347c97 fix: cleanup objectstore file_put_content 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-09-18 13:54:55 +02:00
Robin Appelman
3e94ee391f fix: create intermediate directories for objectstore moveFromStorage 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-09-18 13:54:55 +02:00
Robin Appelman
879fcc4253 fix: get source file before moving the cache item in objectstore moveFromStorage 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-09-18 13:54:55 +02:00
Stephan Orbaugh
f25acfb799
Merge pull request #47986 from cfiehe/fix_move_on_same_bucket 
perf(ObjectStoreStorage): Improve (slow) move on same object bucket
 2024-09-17 16:35:19 +02:00
provokateurin
d2d1e3284d
fix(Storage): Fix method docs inheritance 
Signed-off-by: provokateurin <kate@provokateurin.de>
 2024-09-17 10:20:38 +02:00