Commit graph

304 commits

Author SHA1 Message Date
Robin Appelman
3b655e6de7 fix: fix mimetype not being updated when changing file extention on object store
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-12-11 07:54:24 +00:00
Andy Scherzinger
77f00ae812
Merge pull request #45681 from nextcloud/test-scanner-no-transactions-27
[27] feat: add option to disable scanner transactions
2024-06-13 18:33:33 +02:00
Robin Appelman
0396fe3816 fix: get child ids for folder in a separate query during move
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-06-13 09:44:20 +00:00
Louis Chemineau
3da39308cd fix(files): Use isRetryable to catch retryable exceptions
Signed-off-by: Louis Chemineau <louis@chmn.me>
2024-06-12 08:13:20 +00:00
Robin Appelman
6c4e74b6ec
feat: add option to disable scanner transactions
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-06-05 19:18:50 +02:00
Christoph Wurst
b226039fe9 perf(db): Sort data for IN before chunking
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2024-05-25 20:50:56 -04:00
Julius Härtl
fb08724486
perf: Avoid updating the folder size if we know the size difference
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2024-04-09 15:45:05 +02:00
Robin Appelman
93cc59c5df
fix: use proper jailed patch in watcher
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-04-09 13:20:38 +02:00
Louis Chemineau
ee14e2f965 feat(files)): Add retry logic to cover deadlock situations when moving many files
Signed-off-by: Louis Chemineau <louis@chmn.me>
2024-03-28 15:29:03 +01:00
Benjamin Gaussorgues
92df765457
Merge pull request #43246 from nextcloud/backport/41327/stable27 2024-02-09 10:31:11 +01:00
Christoph Wurst
d4e01fb0a1 fix(files): Don't attempt to format a partial cache entry
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2024-02-06 12:52:45 +00:00
Robin Appelman
7c8d71d0a0 also updated parent etags when a changed etag is detected during scanning
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-01-31 18:28:05 +00:00
Robin Appelman
90c57a6700 dont reuse etag for folders marked explicitly unscanned
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-01-31 18:28:05 +00:00
Git'Fellow
6b496ab164
fix: reduce memory consumption of scans
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

Fix lint

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
2024-01-16 09:29:09 +01:00
Robin Appelman
204aedc606 optimize cache jail creation
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-11-07 13:59:32 +00:00
Côme Chilliet
e0eb9a8ce9 Store size in int|float for 32bit support
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2023-09-25 15:14:31 +00:00
Julius Härtl
fd6990a33b
fix: Only read unencrypted_size when file is actually encrypted
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-09-12 15:05:16 +02:00
Robin Appelman
5ebceaa54c
reuse dependencies from wrapped cache
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-09-07 16:22:00 +02:00
Ferdinand Thiessen
752523f874 fix: Prevent PHP warnings when optional CacheEntry attributes are unset
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2023-08-17 09:48:53 +00:00
Robin Appelman
a5cd892caa select the fileid first when looking for incomplete files
this seems to improve mariadbs index selection

Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-08-14 18:58:47 +00:00
John Molakvoæ
0638f58632 fix: simplify sourceData check
Co-authored-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
Signed-off-by: John Molakvoæ <skjnldsv@users.noreply.github.com>
2023-08-05 10:34:12 +00:00
Josh Richards
d6fe5acbec Catch more invalid cache source storage paths
OC\Files\Cache\Cache::get can return string|false|null, not just string|false.

- nextcloud/server#26270 added handling of false, but null is needed too.
- Well, or we change the default $resullt to false, but I'm not sure if that has other ramifications and the real need here is to simply catch situations where the cache source storage path is not valid for whatever reason

Related: nextcloud/server#19009

Signed-off-by: Josh Richards <josh.t.richards@gmail.com>
2023-08-05 10:34:11 +00:00
Arthur Schiwon
66a7064db3
fix: include invisible tags for admins
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-07-06 21:52:51 +02:00
Arthur Schiwon
99442b60d5 fix: favorites view and universal search against tags
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-07-06 17:29:12 +00:00
Arthur Schiwon
7d8d586983 fix: change if with conditionless else to switch; and a parameter value
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-07-06 17:29:12 +00:00
Arthur Schiwon
7bfe6f2cc0 use efficient tag retrieval on DAV report request
- uses DAV search approach against valid files joined by systemtag selector
- reduced table join for tag/systemtag search
- supports pagination
- no changes to the output formats or similar

Example request body:

<?xml version="1.0"?>
<oc:filter-files xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns" xmlns:nc="http://nextcloud.org/ns" xmlns:ocs="http://open-collaboration-services.org/ns">
  <d:prop>
    <d:getcontentlength/>
    <d:getcontenttype/>
    <d:getetag/>
    <d:getlastmodified/>
    <d:resourcetype/>
    <nc:face-detections/>
    <nc:file-metadata-size/>
    <nc:has-preview/>
    <nc:realpath/>
    <oc:favorite/>
    <oc:fileid/>
    <oc:permissions/>
    <nc:nbItems/>
  </d:prop>
  <oc:filter-rules>
    <oc:systemtag>32</oc:systemtag>
  </oc:filter-rules>
  <d:limit>
    <d:nresults>50</d:nresults>
    <nc:firstresult>0</nc:firstresult>
  </d:limit>
</oc:filter-files>

Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-07-06 17:29:11 +00:00
Arthur Schiwon
dd2dfc89fd
Merge pull request #38976 from nextcloud/backport/38625/stable27
[stable27] fix: expect interface, not a specific implementation
2023-07-04 22:07:29 +02:00
Robin Appelman
b219a17cc6 file scanner performance improvements
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-06-26 15:44:16 +00:00
Arthur Schiwon
09c4ba257c refactor: declare getMount() and getMountsIn() at IRootFolder
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-06-23 23:04:44 +00:00
Arthur Schiwon
f724760f67 chore: ugly type juggling
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-06-23 23:04:44 +00:00
Arthur Schiwon
c04b1ed857 fix: expect interface, not a specific implementation
- fixes a regression when deleting folders while music app was enabled,
  for a LazyRoot was passed to this method.

Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-06-23 23:04:44 +00:00
Julius Härtl
ce8e21d2c5
fix: execute throws docrine exceptions not our wrapped ones
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-05-24 20:45:51 +02:00
Côme Chilliet
8362eea14e
Merge pull request #38196 from nextcloud/fix/fix-32bits-freespace-and-sizes
Get rid of more int casts in file size manipulations
2023-05-15 17:05:46 +02:00
Côme Chilliet
ae525e1935
Fix return types in phpdoc
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2023-05-15 12:24:42 +02:00
Julius Härtl
e18f97fc95
Merge pull request #37709 from nextcloud/bugfix/deadlock 2023-05-13 13:13:04 +02:00
Côme Chilliet
a90581b405
Get rid of more int casts in file size manipulations
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2023-05-11 12:46:16 +02:00
Arthur Schiwon
b6c034ac57
Merge pull request #37961 from nextcloud/poc/noid/systemtags-perf
SystemTags endpoint to return tags used by a user with meta data
2023-05-11 10:16:03 +02:00
Robin Appelman
2ea41dab93
repair -1 folder sizes for object store background scan
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-05-10 19:33:26 +02:00
Arthur Schiwon
df662f50bd
refactor: remove where specification from SELECT getter
- search constraints are now fully in control of
  SystemTagsInFilesDetector::detectAssignedSystemTagsIn(), avoids
  duplication of a WHERE statement

Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-05-10 18:48:06 +02:00
Arthur Schiwon
dbfd2f936a
refactor: remove SystemTag logic from Folder into QuerySearchHelper
- adds OC\SystemTag\SystemTagsInFilesDetector where the search logic is
  moved to

Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-05-09 23:51:52 +02:00
Arthur Schiwon
2489a2d63a
feat: specify media type via url path: systemtags-current/$mediaType
- only the media part of the mime type can be search, but not the full
  mime type. It can be added, should it become necessary.
- thus fixes previously hardcoded selector for image/ types
- also fixes a return type hint
- adds a return type hint

Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-05-09 23:51:51 +02:00
Arthur Schiwon
6bff6a5667
PoC: SystemTags endpoint to return tags used by a user with meta data
Target case is photos app: when visiting the tags category, all systemtags
of the whole cloud are retrieved. In subequent steps the next tag is
requested until the browser view is filled with tag tiles (i.e. previews
are requested just as well).

With this approach, we incorpoate the dav search and look for user related
tags that are used by them, and already returns the statistics (number of
files tagged with the respective tag) as well as a file id for the purpose
to load the preview. This defaults to the file with the highest id.

Call:
curl -s -u 'user:password' \
  'https://my.nc.srv/remote.php/dav/systemtags-current' \
  -X PROPFIND -H 'Accept: text/plain' \
  -H 'Accept-Language: en-US,en;q=0.5'  -H 'Depth: 1' \
  -H 'Content-Type: text/plain;charset=UTF-8' \
  --data @/home/doe/request-systemtag-props.xml

With request-systemtag-props.xml:
<?xml version="1.0" encoding="UTF-8"?>
<d:propfind xmlns:d="DAV:">
        <d:prop xmlns:oc="http://owncloud.org/ns" xmlns:nc="http://nextcloud.org/ns">
                <oc:id/>
                <oc:display-name/>
                <oc:user-visible/>
                <oc:user-assignable/>
                <oc:can-assign/>
                <nc:files-assigned/>
                <nc:reference-fileid/>
        </d:prop>
</d:propfind>

Example output:
  …
  <d:response>
    <d:href>/master/remote.php/dav/systemtags/84</d:href>
    <d:propstat>
      <d:prop>
        <oc:id>84</oc:id>
        <oc:display-name>Computer</oc:display-name>
        <oc:user-visible>true</oc:user-visible>
        <oc:user-assignable>true</oc:user-assignable>
        <oc:can-assign>true</oc:can-assign>
        <nc:files-assigned>42</nc:files-assigned>
        <nc:reference-fileid>924022</nc:reference-fileid>
      </d:prop>
      <d:status>HTTP/1.1 200 OK</d:status>
    </d:propstat>
  </d:response>
  <d:response>
    <d:href>/remote.php/dav/systemtags/97</d:href>
    <d:propstat>
      <d:prop>
        <oc:id>97</oc:id>
        <oc:display-name>Bear</oc:display-name>
        <oc:user-visible>true</oc:user-visible>
        <oc:user-assignable>true</oc:user-assignable>
        <oc:can-assign>true</oc:can-assign>
        <nc:files-assigned>1</nc:files-assigned>
        <nc:reference-fileid>923422</nc:reference-fileid>
      </d:prop>
      <d:status>HTTP/1.1 200 OK</d:status>
    </d:propstat>
  </d:response>
  …

Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-05-09 23:51:48 +02:00
Julius Härtl
3bdd770129
fix: Check for wrapped retriable exceptions
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-05-02 19:42:11 +02:00
Robin Appelman
4953296b9e
Merge pull request #36857 from nextcloud/scan-repair-unencrypted-size
fix unencrypted_size for folders when scanning the filesystem with encryption enabled
2023-04-24 15:24:50 +02:00
Robin Appelman
0ac7230b7f
Merge pull request #37820 from nextcloud/parent-storage-mtime-ignore-error
ignore errors while trying to update parent storage_mtime
2023-04-21 19:11:27 +02:00
Robin Appelman
4b52dafbf9
ignore errors while trying to update parent storage_mtime
in the worst case this should only cause an extra rescan later

Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-04-20 17:16:03 +02:00
Côme Chilliet
b294edad80
Merge branch 'master' into enh/type-iconfig-getter-calls
Signed-off-by: Côme Chilliet <91878298+come-nc@users.noreply.github.com>
2023-04-20 16:52:38 +02:00
Robin Appelman
b1f352c614
improve handling of files we can't access in the scanner
instead of erroring, remove the items from the cache.

this situation can be triggered if a user has access to a file but looses it afterwards

Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-04-05 14:44:55 +02:00
Côme Chilliet
426c0341ff
Use typed version of IConfig::getSystemValue as much as possible
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2023-04-05 12:50:08 +02:00
Robin Appelman
5267a74027
fix type hints
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-04-04 17:05:45 +02:00