Commit graph

78 commits

Author SHA1 Message Date
Scott Bishel
82b8d4dc07
MM-55966 - Update ArrayFromJSON to use LimitedReader (#25510)
* update ArrayFromJSON to use LimitedReader

* update for bad merge

* fix lint errors

* update test code

* update unit tests

* update unit tests

* fix unit tests

* use consts, other cleanup

* add non sorting duplicate check

* set config to default value, then config setting if available

* fix lint errors

* fixes and debugs

* fix log test

* remove setting from Client, add unlimited Parser to client

* a couple more fixes

* another fix

* rename some variables

* remove superflous call

* check for valid MaximumPayloadSize

* update language file

* fix for e2e-tests

* update util function to return error

* lint fix

* update config property name to include unit

* fix for unit test

* add new config to telemetry

* call function to create LimitedReader

* Deprecate old function, use new function name

* return new AppError on failed parse

* return new AppError on failed parse

* return new AppError on failed parse

* add constant for i18n valid constants

* Update server/public/model/utils_test.go

Co-authored-by: Miguel de la Cruz <mgdelacroix@gmail.com>

* Apply suggestions from code review

Co-authored-by: Miguel de la Cruz <mgdelacroix@gmail.com>

* update error variable, remove unnecessary check

* Update function names

* fix errors from merge

* update unit test to create unique ids

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Miguel de la Cruz <mgdelacroix@gmail.com>
2024-01-09 10:04:16 -07:00
Doug Lauder
241e8edc2e
Ping new shared channels remotes immediately (#25850)
* option for auto inviting plugin to all shared channels.

* auto-invite remotes to shared channels when flag set

* fix unit test

* immediately ping new remotes; fix unique siteurl bug

* make i18n-extract

* fix translations

* fix merge conflicts

* make modules-tidy

* revert accidental go.mod change

* revert accidental go.sum changes

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2024-01-09 12:35:05 +01:00
Doug Lauder
000aa0d891
Plugin API for Shared Channels: support auto invite (#25834)
* option for auto inviting plugin to all shared channels.

* auto-invite remotes to shared channels when flag set

* fix unit test

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2024-01-08 18:12:18 +01:00
Claudio Costa
077221a940
Revert "MM-55295: Improve DB performance of some APIs (#25318)" (#25852)
This reverts commit ec88ab4ee9.
2024-01-05 10:05:37 -06:00
Ben Schumacher
edc305716f
[MM-56284] Remove non-generic StoreResult (#25750) 2024-01-04 12:30:21 +01:00
Doug Lauder
2d1135ca46
Shared channels plugin APIs for MS Teams plugin (#25805)
New plugin APIs and hooks for accessing Shared Channels service via plugin. 

- RegisterPluginForSharedChannels(opts model.RegisterPluginOpts) (remoteID string, err error)
- UnregisterPluginForSharedChannels(pluginID string) error
- ShareChannel(sc *model.SharedChannel) (*model.SharedChannel, error)
- UpdateSharedChannel(sc *model.SharedChannel) (*model.SharedChannel, error)
- UnshareChannel(channelID string) (unshared bool, err error)
- UpdateSharedChannelCursor(channelID, remoteID string, cusror model.GetPostsSinceForSyncCursor) error
- SyncSharedChannel(channelID string) error
- InviteRemoteToChannel(channelID string, remoteID string, userID string) error
- UninviteRemoteFromChannel(channelID string, remoteID string) error

Hooks
- OnSharedChannelsSyncMsg(msg *model.SyncMsg, rc *model.RemoteCluster) (model.SyncResponse, error)
- OnSharedChannelsPing(rc *model.RemoteCluster) bool
2023-12-22 17:00:27 -05:00
Ibrahim Serdar Acikgoz
0f3553c8ab
shared/filestore/localstore.go: remove unnecessary warning (#25801) 2023-12-22 19:26:57 +01:00
Ben Schumacher
1fdddfe678
[MM-53269] Add configuration setting for integration requests timeout (#23805)
Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-12-18 16:07:00 +01:00
Nick Misasi
f0a336ba07
Feature: Wrangler (#23602)
* Migrate feature/wrangler to mono-repo

* Add wrangler files

* Fix linters, types, etc

* Fix snapshots

* Fix playwright

* Fix pipelines

* Fix more pipeline

* Fixes for pipelines

* More changes for pipeline

* Fix types

* Add support for a feature flag, but leave it defaulted on for spinwick usage for now

* Update snapshot

* fix js error when removing last value of multiselect, support CSV marshaling to string array for textsetting

* Fix linter

* Remove TODO

* Remove another TODO

* fix tests

* Fix i18n

* Add server tests

* Fix linter

* Fix linter

* Use proper icon for dot menu

* Update snapshot

* Add Cypress UI tests for various entrypoints to move thread modal, split SCSS out from forward post into its own thing

* clean up

* fix linter

* More cleanup

* Revert files to master

* Fix linter for e2e tests

* Make ForwardPostChannelSelect channel types configurable with a prop

* Add missing return

* Fixes from PR feedback

* First batch of PR Feedback

* Another batch of PR changes

* Fix linter

* Update snapshots

* Wrangler system messages are translated to each user's locale

* Initially translate Wrangler into system locale rather than initiating user

* More fixes for PR Feedback

* Fix some server tests

* More updates with master. Fixes around pipelines. Enforce Enterprise license on front/back end

* Add tests for dot_menu

* More pipeline fixes

* Fix e2etests prettier

* Update cypress tests, change occurrences of 'Wrangler' with 'Move Thread'

* Fix linter

* Remove enterprise lock

* A couple more occurrences of wrangler strings, and one more enterprise lock

* Fix server tests

* Fix i18n

* Fix e2e linter

* Feature flag shouldn't be on by default

* Enable move threads feature in smoke tests (#25657)

* enable move threads feature
* add @prod tag

* Fix move_thread_from_public_channel e2e test

* Fix e2e style

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: yasserfaraazkhan <attitude3cena.yf@gmail.com>
2023-12-11 15:27:34 -05:00
Doug Lauder
8bf9e4c481
Prep for MS Teams plugin API (#25565)
* - columns added to ShareChannelRemotes: lastpostcreateat, lastpostupdateat
- SyncMsg and SyncResponse moved to `model` package
- field added to RemoteCluster struct: PluginID

* sync new posts before updated posts to ensure post order in MS Teams

* add plugid to remoteclusters table and store

* don't sync history by default
2023-12-04 13:10:20 -05:00
Ben Schumacher
b2ec1ff8ae
[MM-55595] Use annotated logger in search layer (#25468) 2023-12-04 18:34:57 +01:00
Doug Lauder
ce7259ddc5
fix mung of usernames on collision in shared channels; tilde is not a valid character (#25575) 2023-12-01 17:33:13 +01:00
Doug Lauder
57a87d8ca5
Fix username collision for shared channels sync update (#25578)
* fix username collision for shared channels sync update

* ensure both return types are handled

* remove unused interface
2023-12-01 17:32:14 +01:00
Ben Schumacher
c395ec6245
[MM-55990] Logger improvements for tests (#25467) 2023-11-30 10:47:04 +01:00
Ben Schumacher
e99ded1635
[MM-55480] Update logr to v2.0.21 (#25431) 2023-11-23 10:30:08 +01:00
Devin Binnie
1bd72bdb99
[MM-55014][MM-55015] Add last login timestamp for users, add materialized view and refresh job to keep track of post stats for Postgres (#25152)
* [MM-55014][MM-55015] Add last login timestamp for users, add materialized view and refresh job for Postgres

* Check fixes

* Fix type issue

* Add verification that lastlogin was updated

* PR feedback

* Morge'd

* Morge'd again

* Merge'd

* Update admin setting strings

* WIP

* PR feedback

* Oops

* Fix i18n

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2023-11-14 11:26:27 -05:00
Agniva De Sarker
ec88ab4ee9
MM-55295: Improve DB performance of some APIs (#25318)
Load tests show that channelstore.GetMember and
channelstore.GetMembersForUser are among the chief
queries that take up CPU in the DB.

In this PR, we attempt some strategic optimizations to
reduce/optimize calls to channelstore.GetMember

1. Optimize  `(a *App) HasPermissionToChannel`

We replace GetChannelMember with GetAllChannelMembersForUser
because it's cache backed. So although it gets more data,
it does not hit the DB and saves some latency.

2. Optimize getPostsForChannelAroundLastUnread

We repace getChannelMember with getChannelMemberOnly
which is a lite version of the store call which queries
just the ChannelMembers table. This is because
in the app layer, we just use the LastViewedAt attribute.
Therefore, there is no reason to join with 5 tables when
a single table can do the job.

3. Optimize publishWebsocketEventForPermalinkPost

We use GetAllChannelMembersById instead of GetChannelMembersPage
which again joins with a lot of other tables.

4. Optimize countMentionsFromPost

Again, we replace GetChannelMember which is a costly call joining
multiple tables, with GetAllChannelMembersNotifyPropsForChannel
which is cache-backed and gives us just what we need in the app
layer - notify props.

```release-note
Make small optimizations in several DB calls:
- App.HasPermissionToChannel
- getPostsForChannelAroundLastUnread
- publishWebsocketEventForPermalinkPost
- countMentionsFromPost
```

https://mattermost.atlassian.net/browse/MM-55295

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2023-11-13 21:15:57 +05:30
Soumyadip Moni
3e3087955b
make use of GenericStoreResult in telemetry.go (#25388) 2023-11-13 11:11:17 +05:30
Ben Schumacher
612e4458ef
[MM-55409] Enable revive linter on enterprise code (#25391) 2023-11-10 15:24:12 +01:00
Ben Schumacher
b0bf5d0765
[MM-55053] Use request-specific logger in channels/app/file.go (#25037) 2023-11-07 10:04:16 +01:00
Ben Schumacher
486e836b83
Fix racy test issues (#24971) 2023-11-06 12:26:17 +01:00
Catena cyber
04e7b6bc9e
Applies perfsprint linter (#24999)
Replacing usages of fmt.Sprint with faster alternatives

Signed-off-by: Philippe Antoine <contact@catenacyber.fr>
2023-11-03 11:18:18 +01:00
Ben Schumacher
274adbf077
Reduce usage of mlog.Any (#25180) 2023-11-01 12:57:13 +01:00
ludvigbolin
4547d9ab0e
[GH24626] Remove ExperimentalTimezone as this is always true (#24657)
* Remove all occurrences of ExperimentalTimezone in /server

This removes all uses of ExperimentalTimezone and makes all conditions using this setting being either removed (== false) or always used (== true)

* Remove timezone parameter for getChannelMemberByGroup

As timezone is always enabled this function no longer has a second parameter. The endpoint will always be called with includeTimezones set to true

* Remove all uses of ExperimentalTimezone in webapp

All conditions dependent of this are adjusted to always be true if experimentalTimezone should be true and vice versa

* Change all tests checking for correct usage of ExperimentalTimezone

As this parameter is no longer used, the tests which check for correct behaviour of this is no longer needed

* Fix type errors

Tests regarding types does now pass

'npm run check-types' will run successfully

* fix lint issues

Fix all current lint errors and now lint test goes through

* Fix test regarding profile popover

enableTimezone now is gone and all profile_popover.test.tsx can run successfully.

Added state.entities.users.profiles[] as it before threw an error without it.

* Fix tests where "April 1 2019" was changed to "April 01 2019"

Now it will show "April 1 2019" and "timeZone" will only show if its other than empty ("").

* Fix test where date was set to NaN

Now date will show correctly and will not be NaN

* fix minor test case that failed before

* fix linting in server

* Delete tests that are not valid with timeZoneEnabled removed

tests in advanced_create_comment and advanced_create_post had timeZoneEnabled
set to false as default. The tests that use this are now changed to once which are meant to have timeZoneEnabled set to true.

Adjust jest function in advanced_create_post to get correct stacktrace

* Parameter deleted

isTimeZoneEnabled is always true and therefore the parameter can be deleted

* adjust to prettier standard

* Remove all occurrences of ExperimentalTimezone in /server

This removes all uses of ExperimentalTimezone and makes all conditions using this setting being either removed (== false) or always used (== true)

* Remove timezone parameter for getChannelMemberByGroup

As timezone is always enabled this function no longer has a second parameter. The endpoint will always be called with includeTimezones set to true

* Remove all uses of ExperimentalTimezone in webapp

All conditions dependent of this are adjusted to always be true if experimentalTimezone should be true and vice versa

* Change all tests checking for correct usage of ExperimentalTimezone

As this parameter is no longer used, the tests which check for correct behaviour of this is no longer needed

* Fix type errors

Tests regarding types does now pass

'npm run check-types' will run successfully

* fix lint issues

Fix all current lint errors and now lint test goes through

* Fix test regarding profile popover

enableTimezone now is gone and all profile_popover.test.tsx can run successfully.

Added state.entities.users.profiles[] as it before threw an error without it.

* Fix tests where "April 1 2019" was changed to "April 01 2019"

Now it will show "April 1 2019" and "timeZone" will only show if its other than empty ("").

* Fix test where date was set to NaN

Now date will show correctly and will not be NaN

* fix minor test case that failed before

* fix linting in server

* Delete tests that are not valid with timeZoneEnabled removed

tests in advanced_create_comment and advanced_create_post had timeZoneEnabled
set to false as default. The tests that use this are now changed to once which are meant to have timeZoneEnabled set to true.

Adjust jest function in advanced_create_post to get correct stacktrace

* Parameter deleted

isTimeZoneEnabled is always true and therefore the parameter can be deleted

* adjust to prettier standard

---------

Co-authored-by: Harrison Healey <harrisonmhealey@gmail.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
2023-10-30 15:31:05 -04:00
Ben Schumacher
c7461751f2
Use request.CTX instead of *request.Context (#24877)
* Use request.CTX instead of *request.Context

* Fix tests
2023-10-30 16:33:37 +01:00
Ben Schumacher
5d3ba7483b
[MM-53057] Disallow malformed channel names for DMs (#24404) 2023-10-24 15:08:14 +02:00
Ben Schumacher
e864f9cfee
[MM-55052] Improve file extraction logging (#25036) 2023-10-23 20:25:40 +02:00
Jesse Hallam
2ba91e43b6
Remove support for unused product interfaces (#24965)
We no longer rely on commands and hooks for products, so simplify this code.
2023-10-17 11:55:41 -03:00
Ben Schumacher
0d5a8b8841
Add whitespace linter (#24855) 2023-10-11 10:13:36 +02:00
Jesse Hallam
47bfa2b66b
Pass a logger instead of embedding on job (#24650)
* pass a logger instead of embedding on job

* leverage mlog.Millis

* use worker logger with HandleJobPanic

* rely on existing LogClone instead

* guard Job.LogClone against nil Job

* s/workername/worker_name

* Revert "rely on existing LogClone instead"

This reverts commit 17303cbac9.

* Revert "guard Job.LogClone against nil Job"

This reverts commit f1ae22dee5.
2023-10-09 11:04:55 -03:00
Ben Schumacher
aad25be4e1
[MM-54434] Use job.Logger to capture ldap logs (#24493) 2023-10-06 22:43:21 +02:00
Ibrahim Serdar Acikgoz
53a0f8da27
Add limiting ability to log fields (#24251) 2023-09-28 19:28:40 +03:00
Ben Schumacher
30b12f199b
[MM-54132] Use annotated logger for log messages from jobs (#24275) 2023-09-07 08:50:22 +02:00
Ben Cooke
791ee40568
[MM-53291] Data retention improvements (#24253)
* adding new migration for RetentionIdsForDeletion, changing logic for deleting orphaned reactions. Updating delete user and channel endpoints to remove respective reactions
2023-09-06 08:25:27 -04:00
Ben Schumacher
b2f36c7cdf
[MM-54318] Add file storage information to support package (#24474)
* Reorder stats in support package struct

* Add file storage information to support package

* Add clusterID

* Change method name to DriverName

* Ordering

* Fix test
2023-09-06 09:21:25 +02:00
M-ZubairAhmed
bac317dfe5
MM-54181 : Clean up code around deprecated Gfycat (#24343)
* first pass of removal

* first pass of removal

* Update admin_console.js

* rev beta

* remove more

* Update initial_state.ts

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2023-09-05 16:40:37 -04:00
Felipe Martin
9f7521d003
Move request package into public/shared folder (#24420)
* move request package to public/shared

* updated app-layers

* update app layer

* remove original package location
2023-09-05 09:47:30 +02:00
Harrison Healey
f2fcf3d839
MM-53107 Add setting to limit the max number of markdown nodes (#24414)
* MM-53107 Add setting to limit the max number of markdown nodes

* Add extra validation for a couple config fields

* Add new setting to telemetry
2023-08-31 14:36:24 -04:00
Agniva De Sarker
b84236d555
MM-49627: Suppress transformation of embedded images via image proxy (#24401)
There was no check for embedded images which made "copy text" paste
the siteURL instead of the actual image content.

This does not change the behavior of actually rendering the image
which is how other sites behave as well.

https://mattermost.atlassian.net/browse/MM-49627
```release-note
NONE
```
2023-08-30 22:33:42 +05:30
Matthew Straughn
2e449413e8
fixed initialism errors in accordance with golang conventions in 7 files. (#24287) 2023-08-22 10:15:51 +03:00
Ben Schumacher
3ee5432664
[MM-53968] Includes mattermost-plugin-api into the mono repo (#24235)
Include https://github.com/mattermost/mattermost-plugin-api into the mono repo

Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com>
Co-authored-by: Michael Kochell <mjkochell@gmail.com>
Co-authored-by: Alejandro García Montoro <alejandro.garciamontoro@gmail.com>
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
Co-authored-by: Alex Dovenmuehle <alex.dovenmuehle@mattermost.com>
Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
Co-authored-by: Christopher Poile <cpoile@gmail.com>
Co-authored-by: İlker Göktuğ Öztürk <ilkergoktugozturk@gmail.com>
Co-authored-by: Shota Gvinepadze <wineson@gmail.com>
Co-authored-by: Ali Farooq <ali.farooq0@pm.me>
Co-authored-by: Maria A Nunez <maria.nunez@mattermost.com>
Co-authored-by: Daniel Espino García <larkox@gmail.com>
Co-authored-by: Christopher Speller <crspeller@gmail.com>
Co-authored-by: Alex Dovenmuehle <adovenmuehle@gmail.com>
Co-authored-by: Szymon Gibała <szymongib@gmail.com>
Co-authored-by: Lev <1187448+levb@users.noreply.github.com>
Co-authored-by: Jason Frerich <jason.frerich@mattermost.com>
Co-authored-by: Agniva De Sarker <agnivade@yahoo.co.in>
Co-authored-by: Artur M. Wolff <artur.m.wolff@gmail.com>
Co-authored-by: Madhav Hugar <16546715+madhavhugar@users.noreply.github.com>
Co-authored-by: Joe <security.joe@pm.me>
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
Co-authored-by: José Peso <trilopin@users.noreply.github.com>
2023-08-21 09:50:30 +02:00
Scott Bishel
faea229921
MM-53688 Remove boards dataretention (#24231)
* remove boards data-retention

* remove another variable

* update snapshots, to remove boards

* need to keep config settings until focalboard is updated

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2023-08-15 08:27:45 -06:00
Jesse Hallam
e39b485c4b
Fail vs. fatal on store startup (#24170)
If the store fails to initialize (e.g. run a migration), it would `log.Fatal` and then `os.Exit`. Unfortunately, this trips up `TestMain`, which happily keeps running tests, now guaranteed to fail.

Avoid this by instead returning an error from the store initialization, handling appropriately at the layer above.
2023-08-04 23:05:01 -03:00
Julien Tant
d1691833ad
[MM-53881] Make sure export filestore won't use bifrost (#24169) 2023-08-02 09:44:20 -07:00
Jesse Hallam
933b0eb89c
MM-53877: fix export config settings (#24167)
* wip test

* test and use Export* config correctly
2023-08-02 10:24:52 -03:00
Agniva De Sarker
6d6e589c11
MM-53747: Create job to encode older image paths (#24073)
Bifrost now encodes all image paths. Due to this
one-way translation, we need to encode all the older
image paths as well.

After this is done, we can remove the double-lookup.

https://mattermost.atlassian.net/browse/MM-53747

```release-note
NONE
```

Co-authored-by: Mattermost Build <build@mattermost.com>
2023-07-25 08:38:35 +05:30
Agniva De Sarker
feeb0cd2ef
MM-53747: Bifrost: write original paths (#24079)
Now that we have identified the true fix
from Bifrost side, we will slowly revert
the changes that have happened.

Starting with this first fix, we will revert
to writing out non-encoded paths while still
doing the double read. This will break uploading
files with + for a short while, and then
the new Bifrost fix should fix that as well.

https://mattermost.atlassian.net/browse/MM-53747

```release-note
NONE
```
2023-07-21 08:59:59 +05:30
Julien Tant
077c16ef61
[MM-53454] Add export file settings + slash command for public link (#23915)
Co-authored-by: Mattermost Build <build@mattermost.com>
2023-07-19 13:01:39 -07:00
Agniva De Sarker
c1dfdb867a
MM-53709: Handle older non-encoded paths in bifrost (#24058)
We do a check to see if a non-encoded path is present
or not, and in that case, choose not to encode it.

This accrues an additional StatFile call. But after
we have encoded all paths to the new style, we will
get rid of this.

https://mattermost.atlassian.net/browse/MM-53709

```release-note
NONE
```
2023-07-19 22:32:04 +05:30
Ben Cooke
bf0e0ed45a
[MM-53466] Remove concurrency from data retention delete batching (#23927)
* Add TimeBetweenBatches config value for data retention

* adding feature flag for data retention concurrency
---------

Co-authored-by: Julien Tant <julien@craftyx.fr>
Co-authored-by: Mattermost Build <build@mattermost.com>
2023-07-05 11:09:03 -04:00