Commit graph

49 commits

Author SHA1 Message Date
Ibrahim Serdar Acikgoz
084006c0ea
[MM-61758] Burn on read feature (#34703)
Some checks are pending
API / build (push) Waiting to run
Server CI / Compute Go Version (push) Waiting to run
Server CI / Check mocks (push) Blocked by required conditions
Server CI / Check go mod tidy (push) Blocked by required conditions
Server CI / check-style (push) Blocked by required conditions
Server CI / Check serialization methods for hot structs (push) Blocked by required conditions
Server CI / Vet API (push) Blocked by required conditions
Server CI / Check migration files (push) Blocked by required conditions
Server CI / Generate email templates (push) Blocked by required conditions
Server CI / Check store layers (push) Blocked by required conditions
Server CI / Check mmctl docs (push) Blocked by required conditions
Server CI / Postgres with binary parameters (push) Blocked by required conditions
Server CI / Postgres (push) Blocked by required conditions
Server CI / Postgres (FIPS) (push) Blocked by required conditions
Server CI / Generate Test Coverage (push) Blocked by required conditions
Server CI / Run mmctl tests (push) Blocked by required conditions
Server CI / Run mmctl tests (FIPS) (push) Blocked by required conditions
Server CI / Build mattermost server app (push) Blocked by required conditions
Web App CI / check-lint (push) Waiting to run
Web App CI / check-i18n (push) Blocked by required conditions
Web App CI / check-types (push) Blocked by required conditions
Web App CI / test (platform) (push) Blocked by required conditions
Web App CI / test (mattermost-redux) (push) Blocked by required conditions
Web App CI / test (channels shard 1/4) (push) Blocked by required conditions
Web App CI / test (channels shard 2/4) (push) Blocked by required conditions
Web App CI / test (channels shard 3/4) (push) Blocked by required conditions
Web App CI / test (channels shard 4/4) (push) Blocked by required conditions
Web App CI / upload-coverage (push) Blocked by required conditions
Web App CI / build (push) Blocked by required conditions
* Add read receipt store for burn on read message types

* update mocks

* fix invalidation target

* have consistent case on index creation

* Add temporary posts table

* add mock

* add transaction support

* reflect review comments

* wip: Add reveal endpoint

* user check error id instead

* wip: Add ws events and cleanup for burn on read posts

* add burn endpoint for explicitly burning messages

* add translations

* Added logic to associate files of BoR post with the post

* Added test

* fixes

* disable pinning posts and review comments

* MM-66594 - Burn on read UI integration (#34647)

* MM-66244 - add BoR visual components to message editor

* MM-66246 - BoR visual indicator for sender and receiver

* MM-66607 - bor - add timer countdown and autodeletion

* add the system console max time to live config

* use the max expire at and create global scheduler to register bor messages

* use seconds for BoR config values in BE

* implement the read by text shown in the tooltip logic

* unestack the posts from same receiver and BoR  and fix styling

* avoid opening reply RHS

* remove unused dispatchers

* persis the BoR label in the drafts

* move expiration value to metadata

* adjust unit tests to metadata insted of props

* code clean up and some performance improvements; add period grace for deletion too

* adjust migration serie number

* hide bor messages when config is off

* performance improvements on post component and code clean up

* keep bor existing post functionality if config is disabled

* Add read receipt store for burn on read message types

* Add temporary posts table

* add transaction support

* reflect review comments

* wip: Add reveal endpoint

* user check error id instead

* wip: Add ws events and cleanup for burn on read posts

* avoid reacting to unrevealed bor messages

* adjust migration number

* Add read receipt store for burn on read message types

* have consistent case on index creation

* Add temporary posts table

* add mock

* add transaction support

* reflect review comments

* wip: Add reveal endpoint

* user check error id instead

* wip: Add ws events and cleanup for burn on read posts

* add burn endpoint for explicitly burning messages

* adjust post reveal and type with backend changes

* use real config values, adjust icon usage and style

* adjust the delete from from sender and receiver

* improve self deleting logic by placing in badge, use burn endpoint

* adjust websocket events handling for the read by sender label information

* adjust styling for concealed and error state

* update burn-on-read post event handling for improved recipient tracking and multi-device sync

* replace burn_on_read with type in database migrations and model

* remove burn_on_read metadata from PostMetadata and related structures

* Added logic to associate files of BoR post with the post

* Added test

* adjust migration name and fix linter

* Add read receipt store for burn on read message types

* update mocks

* have consistent case on index creation

* Add temporary posts table

* add mock

* add transaction support

* reflect review comments

* wip: Add reveal endpoint

* user check error id instead

* wip: Add ws events and cleanup for burn on read posts

* add burn endpoint for explicitly burning messages

* Added logic to associate files of BoR post with the post

* Added test

* disable pinning posts and review comments

* show attachment on bor reveal

* remove unused translation

* Enhance burn-on-read post handling and refine previous post ID retrieval logic

* adjust the returning chunk to work with bor messages

* read temp post from master db

* read from master

* show the copy link button to the sender

* revert unnecessary check

* restore correct json tag

* remove unused error handling  and clarify burn-on-read comment

* improve type safety and use proper selectors

* eliminate code duplication in deletion handler

* optimize performance and add documentation

* delete bor message for sender once all receivers reveal it

* add burn on read to scheduled posts

* add feature enable check

* use master to avoid  all read recipients race condition

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
Co-authored-by: Harshil Sharma <harshilsharma63@gmail.com>

* squash migrations into single file

* add configuration for the scheduler

* don't run messagehasbeenposted hook

* remove parallel tests on burn on read

* add clean up for closing opened modals from previous tests

* simplify delete menu item rendering

* add cleanup step to close open modals after each test to prevent pollution

* streamline delete button visibility logic for Burn on Read posts

* improve reliability of closing post menu and modals by using body ESC key

---------

Co-authored-by: Harshil Sharma <harshilsharma63@gmail.com>
Co-authored-by: Pablo Vélez <pablovv2012@gmail.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
2025-12-11 07:59:50 +01:00
Elias Nahum
1022cd44c0
MM-65756 Database Migrations, Indexes and Methods for Auto-Translation (#34047)
Some checks failed
API / build (push) Has been cancelled
Server CI / Compute Go Version (push) Has been cancelled
Web App CI / check-lint (push) Has been cancelled
Web App CI / check-i18n (push) Has been cancelled
Web App CI / check-types (push) Has been cancelled
Web App CI / test (push) Has been cancelled
Web App CI / build (push) Has been cancelled
Server CI / Check mocks (push) Has been cancelled
Server CI / Check go mod tidy (push) Has been cancelled
Server CI / check-style (push) Has been cancelled
Server CI / Check serialization methods for hot structs (push) Has been cancelled
Server CI / Vet API (push) Has been cancelled
Server CI / Check migration files (push) Has been cancelled
Server CI / Generate email templates (push) Has been cancelled
Server CI / Check store layers (push) Has been cancelled
Server CI / Check mmctl docs (push) Has been cancelled
Server CI / Postgres with binary parameters (push) Has been cancelled
Server CI / Postgres (push) Has been cancelled
Server CI / Postgres (FIPS) (push) Has been cancelled
Server CI / Generate Test Coverage (push) Has been cancelled
Server CI / Run mmctl tests (push) Has been cancelled
Server CI / Run mmctl tests (FIPS) (push) Has been cancelled
Server CI / Build mattermost server app (push) Has been cancelled
* AutoTranslate config settings

* comment out Agents provider

* Add auto translate timeout config validation

* i18n messages for autotranslation config validation

* fix test

* validate url for libreTranslate

* Feedback review

* Admin Console UI for Auto-Translation

* fix admin console conditional section display

* i18n

* removed unintentional change

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* update admin.general.localization.autoTranslateProviderDescription newline

* fix lint

* Fix types

* UX feedback review

* fix typo in i18n

* Fix AutoTranslation feature flag

* feedback review

* Fix test default values

* feedback review

* re-add isHidden property to feature discovery

* Database Migrations, Indexes and Methods for Auto-Translation

* i18n

* fix retrylayer and storetest

* Fix search query

* fix lint

* remove the request.CTX and modify Translation model

* fix lint and external url

* Add settings to playwright

* Add empty as a valid value for the Provider

* Update jsonb queries

* Fix queries and add model methods

* fix go lint

* go lint fix 2

* fix db migrations

* feedback review + store cache

* increase migration number

* cleanup autotranslation store cache

* use NULL as objectType for posts

* fix bad merge

* fix tests

* add missing i18n

* Switch prop bags column to boolean

* fix lint

* fix tests

* Remove database search

* use Builder methods

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: BenCookie95 <benkcooke@gmail.com>
2025-11-22 09:32:01 +08:00
Ben Cooke
3aad6b0448
Add support for resource parameter with OAuth (#33743) 2025-11-11 15:24:42 -05:00
Ben Cooke
a79ac96b50
OAuth public client support through DCR and PKCE support for public/confidential clients (#33664)
* public client support along with PKCE for public/confidential clients

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2025-11-11 17:43:37 +00:00
Ben Cooke
a9c9953439
Authorization metadata endpoint and Dynamic Client Registration of Confidential OAuth Apps (#33642)
Some checks are pending
API / build (push) Waiting to run
Server CI / Compute Go Version (push) Waiting to run
Server CI / Check mocks (push) Blocked by required conditions
Server CI / Check go mod tidy (push) Blocked by required conditions
Server CI / check-style (push) Blocked by required conditions
Server CI / Check serialization methods for hot structs (push) Blocked by required conditions
Server CI / Vet API (push) Blocked by required conditions
Server CI / Check migration files (push) Blocked by required conditions
Server CI / Generate email templates (push) Blocked by required conditions
Server CI / Check store layers (push) Blocked by required conditions
Server CI / Check mmctl docs (push) Blocked by required conditions
Server CI / Postgres with binary parameters (push) Blocked by required conditions
Server CI / Postgres (push) Blocked by required conditions
Server CI / Postgres (FIPS) (push) Blocked by required conditions
Server CI / Generate Test Coverage (push) Blocked by required conditions
Server CI / Run mmctl tests (push) Blocked by required conditions
Server CI / Run mmctl tests (FIPS) (push) Blocked by required conditions
Server CI / Build mattermost server app (push) Blocked by required conditions
Web App CI / check-lint (push) Waiting to run
Web App CI / check-i18n (push) Waiting to run
Web App CI / check-types (push) Waiting to run
Web App CI / test (push) Waiting to run
Web App CI / build (push) Waiting to run
* initial DCR and metadata implementation

* check for duplicate registrations

* tests and other cleanup

* dcr fixes

* tidy up unused DCR fields

* remove initial access token support

* remove duplicate client checks

* remove unused store function

* remove restrictive redirect url checks

* create some constants for endpoints

* surface support for implicit grant and add system console setting

* fix frontend issues with DCR clients

* rate limiting the DCR endpoint

* lint

* lint and cleanup

* remove storage of grants, responses and methods. Just enforce in the code

* fix lint and tests

* docs and test

* accidentally removed comments

* fix mock

* translations

* do not advertise public client capability

* validate supplied token_endpoint_auth_method

* fix pr comments

* updates

* add metadata endpoint to docs

* add definition

* lint

* fix client4

* fix client methods

* fix client again

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2025-11-11 14:27:18 +00:00
Harshil Sharma
3265054ad5
Migrate content flagging settings to database (#33989)
Some checks failed
API / build (push) Waiting to run
Server CI / Compute Go Version (push) Waiting to run
Server CI / Check mocks (push) Blocked by required conditions
Server CI / Check go mod tidy (push) Blocked by required conditions
Server CI / check-style (push) Blocked by required conditions
Server CI / Check serialization methods for hot structs (push) Blocked by required conditions
Server CI / Vet API (push) Blocked by required conditions
Server CI / Check migration files (push) Blocked by required conditions
Server CI / Generate email templates (push) Blocked by required conditions
Server CI / Check store layers (push) Blocked by required conditions
Server CI / Check mmctl docs (push) Blocked by required conditions
Server CI / Postgres with binary parameters (push) Blocked by required conditions
Server CI / Postgres (push) Blocked by required conditions
Server CI / Postgres (FIPS) (push) Blocked by required conditions
Server CI / Generate Test Coverage (push) Blocked by required conditions
Server CI / Run mmctl tests (push) Blocked by required conditions
Server CI / Run mmctl tests (FIPS) (push) Blocked by required conditions
Server CI / Build mattermost server app (push) Blocked by required conditions
Web App CI / check-lint (push) Waiting to run
Web App CI / check-i18n (push) Waiting to run
Web App CI / check-types (push) Waiting to run
Web App CI / test (push) Waiting to run
Web App CI / build (push) Waiting to run
Migration-assist Sync / Check if migration-assist have been synced (push) Has been cancelled
* lint fix

* CI

* added new migration mocks

* Used setup for tests

* some comment

* Removed unnecesseery nil check

* Form validation

* WIP tests

* WIP tests

* WIP tests

* fix: mock content flagging config selector with correct reasons format

Co-authored-by: aider (anthropic/claude-sonnet-4-20250514) <aider@aider.chat>

* fix: add mock for getContentFlaggingConfig in flag post modal test

Co-authored-by: aider (anthropic/claude-sonnet-4-20250514) <aider@aider.chat>

* Updated error code order in API docs

* removed empty files

* Added tests

* lint fixes

* minor tweak

* lint fix

* type fix

* fixed test

* nit

* test enhancements

* API WIP

* API WIP

* creating values

* creating content flagging channel and properties

* Able to save properties

* Added another property field

* WIP

* WIP

* Added validations

* Added data validations and hidden post if confifgured to

* lint fixes

* Added API spec

* Added some tests

* Added tests for getContentReviewBot

* test: add comprehensive tests for getContentReviewChannels function

* Added more app layer tests

* Added TestCanFlagPost

* test: Add comprehensive tests for FlagPost function

* Added all app layer tests

* Removed a file that was reamoved downstream

* test: add content flagging test file

* test: add comprehensive tests for FlagContentRequest.IsValid method

* Added model tests

* test: add comprehensive tests for SqlPropertyValueStore.CreateMany

* test: add comprehensive tests for flagPost() API function

* Added API tests

* linter fix

* WIP

* sent post flagging confirmation message

* fixed i18n nissues

* fixed i18n nissues

* CI

* WIP

* WIP

* Added API call

* test: add test for Client4.flagPost API call in FlagPostModal

* fix: remove userEvent.setup() from flag post modal test

* test: wrap submit button click in act for proper state updates

* Updated tests

* lint fix

* Updated test

* fix: reset contentFlaggingGroupId for test isolation in content flagging tests

* removed cached group ID

* removed debug log

* CI

* Updated to allow special characters in comments

* Handled empty comment

* Created getContentFlaggingFields API

* created getPostPropertyValues API

* Used finally

* WIP

* Created useContentFlaggingFields hook

* WIP

* WIP

* Added option to retain data for reviewers

* Displayed deleted post's preview

* DIsplayed all properties

* Adding field name i18n

* WIP - managing i18n able texts

* Finished displaying all fields

* Manual cleanup

* lint fixes

* team role filter logic fix

* Fixed tests

* created new API to fetch flagged posts

* lint fix

* Added new client methods

* test: add comprehensive tests for content flagging APIs

* Added new API tests

* fixed openapi spec

* Fixed DataSpillageReport tests

* Fixed PostMarkdown test

* Fixed PostPreviewPropertyRenderer test

* Added metadata to card renderer

* test fixes

* Added no comment placeholder

* Added view detail button

* Created RemoveFlaggedMessageConfirmationModal modal

* Added key and remove flag request modal

* IMplemented delete flagged post

* Handled edge cases of deleting flagged post

* keep message

* UI integration

* Added WS event for post report update and handled deleted files of flagged post

* Added error handling in keep/remove forms

* i18n fixes

* Fixed test

* Updated OpenAPI specs

* fixed types

* fixed types

* refactoring

* refactor: improve test mocking for data spillage report component

* test mock updates

* Fixed tests

* Updated reducer

* not resetting mocks

* Added migrations for content flagging tables

* Created new structure

* review fixes

* Used correct ot name

* WIP

* review fixes

* review fixes

* Added new property translations

* CI

* CI

* CI

* Improved test

* fixed test

* CI

* New UI component

* WIP

* Updated settings APIs

* cached DB data

* used cached reviewer data

* Updated tests

* Lint fixes

* test: add tests for saveContentFlaggingSettings and getContentFlaggingSettings APIs

* test fix

* test: add tests for SaveContentFlaggingConfig and GetContentFlaggingConfigReviewerIDs

* Updated tests

* test: add content flagging test for local cache layer

* test: add comprehensive tests for content flagging store cache

* Updated tests

* lint fix

* Updated mobile text

* Added content flagging SQL store mocks

* Added API specs for new APIs

* fixed tests

* feat: add TestContentFlaggingStore function for content flagging store testing

* feat: add comprehensive tests for content flagging store

* Added SQL store tests

* test: add content flagging test for local cache layer

* test: add tests for content flagging store caching

* Added cache layer tests

* Updated tests

* Fixed

* Handled JSON error

* fixes

* fixes

* Fixed retry layer test

* fixerdf i18n

* Fixed test

* CI

* building index concurrently

* CI

* fixed a test

* CI

* cleanup

* Integrate flag post api (#33798)

* WIP

* WIP

* Added API call

* test: add test for Client4.flagPost API call in FlagPostModal

* fix: remove userEvent.setup() from flag post modal test

* test: wrap submit button click in act for proper state updates

* Updated tests

* lint fix

* CI

* Updated to allow special characters in comments

* Handled empty comment

* Used finally

* CI

* Fixed test

* Spillage card integration (#33832)

* Created getContentFlaggingFields API

* created getPostPropertyValues API

* WIP

* Created useContentFlaggingFields hook

* WIP

* WIP

* Added option to retain data for reviewers

* Displayed deleted post's preview

* DIsplayed all properties

* Adding field name i18n

* WIP - managing i18n able texts

* Finished displaying all fields

* Manual cleanup

* lint fixes

* team role filter logic fix

* Fixed tests

* created new API to fetch flagged posts

* lint fix

* Added new client methods

* test: add comprehensive tests for content flagging APIs

* Added new API tests

* fixed openapi spec

* Fixed DataSpillageReport tests

* Fixed PostMarkdown test

* Fixed PostPreviewPropertyRenderer test

* Added metadata to card renderer

* test fixes

* Added no comment placeholder

* Fixed test

* refactor: improve test mocking for data spillage report component

* test mock updates

* Updated reducer

* not resetting mocks

* WIP

* review fixes

* CI

* Fixed

* fixes

* Content flagging actions implementation (#33852)

* Added view detail button

* Created RemoveFlaggedMessageConfirmationModal modal

* Added key and remove flag request modal

* IMplemented delete flagged post

* Handled edge cases of deleting flagged post

* keep message

* UI integration

* Added WS event for post report update and handled deleted files of flagged post

* Added error handling in keep/remove forms

* i18n fixes

* Updated OpenAPI specs

* fixed types

* fixed types

* refactoring

* Fixed tests

* review fixes

* Added new property translations

* Improved test

* fixed test

* CI

* fixes

* CI

* fixed a test

* CI

* Review fixes

---------

Co-authored-by: aider (anthropic/claude-sonnet-4-20250514) <aider@aider.chat>
2025-10-13 12:24:01 +05:30
Agniva De Sarker
9dd8c056e7
MM-63368: Remove MySQL (#33458)
https://mattermost.atlassian.net/browse/MM-63368

```release-note
Remove MySQL support from the codebase entirely.
```
2025-07-22 20:40:55 +05:30
catalintomai
85391de22a
MM-57326: [Shared Channels] Message priority, acknowledgement and persistent notifications need to be synced (#30736) 2025-06-16 02:30:21 +02:00
catalintomai
fa1c77d9b0
MM-52600: [Shared Channels] Shared channels do not sync channel membership (#30976) 2025-06-15 10:07:56 +02:00
catalintomai
c46ed6c681
MM-62751: [Shared Channels] Allow remote users to be discoverable in the create DM/GM modal (#30918) 2025-06-13 16:51:12 +02:00
Devin Binnie
25a4839a9e
Automatic channel category sorting (#30866)
* Automatic channel category sorting

* Fix types

* AIed

* Fix issue where categories are updated for all users

* Move all logic to server, clean up

* PR feedback

* Fix lint

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2025-06-11 14:29:36 -04:00
Ibrahim Serdar Acikgoz
1cf2f08108
[MM-64437] Hotfix on attribute view creation error (#31225) 2025-05-30 11:35:31 +02:00
Ibrahim Serdar Acikgoz
a344b3225b
[MM-61756] Attribute Based Access Control - Phase 1 (#30785)
Attribute Based Access Control - Base
* MM-63662

* MM-63919

* MM-63954

* MM-63955 

* MM-63425

* MM-63426

* MM-63458

* MM-63459

* MM-63603

* MM-63845

* MM-64146

* MM-64199

* MM-64201

* MM-64233

* MM-64247

* MM-64268

---------

Co-authored-by: Harshil Sharma <harshilsharma63@gmail.com>
Co-authored-by: Pablo Andrés Vélez Vidal <pablovv2012@gmail.com>
Co-authored-by: abhijit-singh <abhijitsingh0702@gmail.com>
Co-authored-by: Harrison Healey <harrisonmhealey@gmail.com>
2025-05-15 11:33:08 +02:00
Agniva De Sarker
131cf039bb
MM-63756: Added index to sidebarchannels table (#30724)
The (s SqlChannelStore) getSidebarCategoriesT gets called quite frequently.
- Team switch
- WS reconnect
- Category created
- Category updated
- Category deleted

Of these 1 and 2 are probably the most commonly called sources. Based on that,
the sidebarChannels table is not that well-optimized. Even though
the query time might be reasonable, without an index, it has to churn a lot of
DB CPU for a sequential scan.

We add a new index to optimize this.

CREATE INDEX idx_sidebarchannels_categoryid ON sidebarchannels(categoryid);

```
Before:
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Sort  (cost=40854.18..40854.19 rows=4 width=193) (actual time=251.635..251.646 rows=204 loops=1)
   Sort Key: sidebarcategories.sortorder, sidebarchannels.sortorder
   Sort Method: quicksort  Memory: 65kB
   Buffers: shared hit=1203 read=23668
   ->  Nested Loop  (cost=8.87..40854.14 rows=4 width=193) (actual time=251.345..251.455 rows=204 loops=1)
         Buffers: shared hit=1203 read=23668
         ->  Nested Loop  (cost=0.41..9.47 rows=1 width=54) (actual time=0.068..0.074 rows=1 loops=1)
               Buffers: shared hit=5
               ->  Seq Scan on teams  (cost=0.00..1.03 rows=1 width=27) (actual time=0.024..0.026 rows=1 loops=1)
                     Filter: (((id)::text = '3ee5y5ok6jgxicrmqstdnghmfr'::text) AND (deleteat = 0))
                     Rows Removed by Filter: 1
                     Buffers: shared hit=1
               ->  Index Scan using teammembers_pkey on teammembers  (cost=0.41..8.43 rows=1 width=27) (actual time=0.039..0.043 rows=1 loops=1)
                     Index Cond: (((teamid)::text = '3ee5y5ok6jgxicrmqstdnghmfr'::text) AND ((userid)::text = 'tc3p1yqw67d8idcp3g98awexqe'::text))
                     Filter: (deleteat = 0)
                     Buffers: shared hit=4
         ->  Hash Right Join  (cost=8.45..40844.62 rows=4 width=193) (actual time=251.274..251.361 rows=204 loops=1)
               Hash Cond: ((sidebarchannels.categoryid)::text = (sidebarcategories.id)::text)
               Buffers: shared hit=1198 read=23668
               ->  Seq Scan on sidebarchannels  (cost=0.00..37514.77 rows=1265277 width=100) (actual time=0.043..99.345 rows=1265444 loops=1)
                     Buffers: shared hit=1194 read=23668
               ->  Hash  (cost=8.44..8.44 rows=1 width=158) (actual time=0.047..0.047 rows=6 loops=1)
                     Buckets: 1024  Batches: 1  Memory Usage: 10kB
                     Buffers: shared hit=4
                     ->  Index Scan using idx_sidebarcategories_userid_teamid on sidebarcategories  (cost=0.42..8.44 rows=1 width=158) (actual time=0.029..0.037 rows=6 loops=1)
                           Index Cond: (((userid)::text = 'tc3p1yqw67d8idcp3g98awexqe'::text) AND ((teamid)::text = '3ee5y5ok6jgxicrmqstdnghmfr'::text))
                           Buffers: shared hit=4
 Planning:
   Buffers: shared hit=9
 Planning Time: 1.215 ms
 Execution Time: 251.755 ms
(31 rows)

After:
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Sort  (cost=1544.53..1544.54 rows=4 width=192) (actual time=0.834..0.859 rows=204 loops=1)
   Sort Key: sidebarcategories.sortorder, sidebarchannels.sortorder
   Sort Method: quicksort  Memory: 65kB
   Buffers: shared hit=58
   ->  Nested Loop Left Join  (cost=8.53..1544.49 rows=4 width=192) (actual time=0.066..0.252 rows=204 loops=1)
         Buffers: shared hit=58
         ->  Nested Loop  (cost=0.83..17.93 rows=1 width=157) (actual time=0.042..0.098 rows=6 loops=1)
               Buffers: shared hit=34
               ->  Nested Loop  (cost=0.42..9.48 rows=1 width=157) (actual time=0.030..0.049 rows=6 loops=1)
                     Buffers: shared hit=10
                     ->  Index Scan using idx_sidebarcategories_userid_teamid on sidebarcategories  (cost=0.42..8.44 rows=1 width=157) (actual time=0.018..0.022 rows=6 loops=1)
                           Index Cond: (((userid)::text = 'tc3p1yqw67d8idcp3g98awexqe'::text) AND ((teamid)::text = '3ee5y5ok6jgxicrmqstdnghmfr'::text))
                           Buffers: shared hit=4
                     ->  Seq Scan on teams  (cost=0.00..1.03 rows=1 width=27) (actual time=0.002..0.003 rows=1 loops=6)
                           Filter: (((id)::text = '3ee5y5ok6jgxicrmqstdnghmfr'::text) AND (deleteat = 0))
                           Rows Removed by Filter: 1
                           Buffers: shared hit=6
               ->  Index Scan using teammembers_pkey on teammembers  (cost=0.41..8.43 rows=1 width=27) (actual time=0.007..0.007 rows=1 loops=6)
                     Index Cond: (((teamid)::text = '3ee5y5ok6jgxicrmqstdnghmfr'::text) AND ((userid)::text = 'tc3p1yqw67d8idcp3g98awexqe'::text))
                     Filter: (deleteat = 0)
                     Buffers: shared hit=24
         ->  Bitmap Heap Scan on sidebarchannels  (cost=7.69..1522.35 rows=421 width=100) (actual time=0.012..0.017 rows=34 loops=6)
               Recheck Cond: ((categoryid)::text = (sidebarcategories.id)::text)
               Heap Blocks: exact=6
               Buffers: shared hit=24
               ->  Bitmap Index Scan on idx_sidebarchannels_categoryid  (cost=0.00..7.58 rows=421 width=0) (actual time=0.010..0.010 rows=34 loops=6)
                     Index Cond: ((categoryid)::text = (sidebarcategories.id)::text)
                     Buffers: shared hit=18
 Planning:
   Buffers: shared hit=18
 Planning Time: 0.543 ms
 Execution Time: 0.968 ms
(32 rows)
```

I have also looked at potentially re-ordering the JOINs to make
sidebarchannels and sidebarcategories JOIN earlier, but that didn't give
a major benefit.

Also looked at adding a compound index with (categoryid, sortorder) to improve
sorting performance, but that didn't give a major benefit from what the single
column index already gives.

The `completePopulatingCategoryChannelsT` query also partially benefits
from this. But the Postgres optimizer sometimes selects the index on categoryId
and sometimes on ChannelId, both giving equivalent performance. So there's no major
improvement there, but at the same time, no regression as well.

```
Original:
[bigdb] # EXPLAIN (ANALYZE, BUFFERS) SELECT Id FROM ChannelMembers LEFT JOIN Channels ON Channels.Id=ChannelMembers.ChannelId WHERE (ChannelMembers.UserId = 'tc3p1yqw67d8idcp3g98awexqe' AND Channels.Type IN ('D'
                                                                                                                                                                                                                ,'G') AND Channels.DeleteAt = 0 AND NOT EXISTS ( SELECT 1 FROM SidebarChannels JOIN SidebarCategories on SidebarChannels.CategoryId=SidebarCategories.Id WHERE (SidebarChannels.ChannelId = ChannelMembers.ChannelI
                                                                                                                                                                                                                                                                                                                                                                                d AND SidebarCategories.UserId = 'tc3p1yqw67d8idcp3g98awexqe' AND SidebarCategories.TeamId = '3ee5y5ok6jgxicrmqstdnghmfr') )) ORDER BY DisplayName ASC;
                                                                                            QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Sort  (cost=5864.68..5865.84 rows=463 width=40) (actual time=9.008..9.022 rows=39 loops=1)
   Sort Key: channels.displayname
   Sort Method: quicksort  Memory: 27kB
   Buffers: shared hit=2112
   ->  Nested Loop Anti Join  (cost=1.96..5844.18 rows=463 width=40) (actual time=0.188..8.932 rows=39 loops=1)
         Buffers: shared hit=2112
         ->  Nested Loop  (cost=0.99..3476.66 rows=463 width=67) (actual time=0.159..7.952 rows=39 loops=1)
               Buffers: shared hit=1956
               ->  Index Only Scan using idx_channelmembers_user_id_channel_id_last_viewed_at on channelmembers  (cost=0.56..40.78 rows=470 width=27) (actual time=0.036..0.467 rows=437 loops=1)
                     Index Cond: (userid = 'tc3p1yqw67d8idcp3g98awexqe'::text)
                     Heap Fetches: 45
                     Buffers: shared hit=208
               ->  Memoize  (cost=0.43..7.69 rows=1 width=40) (actual time=0.016..0.016 rows=0 loops=437)
                     Cache Key: channelmembers.channelid
                     Cache Mode: logical
                     Hits: 0  Misses: 437  Evictions: 0  Overflows: 0  Memory Usage: 42kB
                     Buffers: shared hit=1748
                     ->  Index Scan using channels_pkey on channels  (cost=0.42..7.68 rows=1 width=40) (actual time=0.015..0.015 rows=0 loops=437)
                           Index Cond: ((id)::text = (channelmembers.channelid)::text)
                           Filter: ((type = ANY ('{D,G}'::channel_type[])) AND (deleteat = 0))
                           Rows Removed by Filter: 1
                           Buffers: shared hit=1748
         ->  Nested Loop  (cost=0.97..5.10 rows=1 width=27) (actual time=0.023..0.023 rows=0 loops=39)
               Buffers: shared hit=156
               ->  Index Only Scan using sidebarchannels_pkey on sidebarchannels  (cost=0.55..4.56 rows=1 width=92) (actual time=0.022..0.022 rows=0 loops=39)
                     Index Cond: (channelid = (channelmembers.channelid)::text)
                     Heap Fetches: 0
                     Buffers: shared hit=156
               ->  Index Scan using sidebarcategories_pkey on sidebarcategories  (cost=0.42..0.48 rows=1 width=65) (never executed)
                     Index Cond: ((id)::text = (sidebarchannels.categoryid)::text)
                     Filter: (((userid)::text = 'tc3p1yqw67d8idcp3g98awexqe'::text) AND ((teamid)::text = '3ee5y5ok6jgxicrmqstdnghmfr'::text))
 Planning:
   Buffers: shared hit=48 dirtied=1
 Planning Time: 2.222 ms
 Execution Time: 9.142 ms
(35 rows)

New:
[bigdb] # EXPLAIN (ANALYZE, BUFFERS) SELECT Id FROM ChannelMembers LEFT JOIN Channels ON Channels.Id=ChannelMembers.ChannelId WHERE (ChannelMembers.UserId = 'tc3p1yqw67d8idcp3g98awexqe' AND Channels.Type IN ('D'
                                                                                                                                                                                                                ,'G') AND Channels.DeleteAt = 0 AND NOT EXISTS ( SELECT 1 FROM SidebarChannels JOIN SidebarCategories on SidebarChannels.CategoryId=SidebarCategories.Id WHERE (SidebarChannels.ChannelId = ChannelMembers.ChannelId AND SidebarCategories.UserId = 'tc3p1yqw67d8idcp3g98awexqe' AND SidebarCategories.TeamId = '3ee5y5ok6jgxicrmqstdnghmfr') )) ORDER BY DisplayName ASC;
                                                                                            QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Sort  (cost=5059.95..5061.11 rows=463 width=40) (actual time=12.072..12.086 rows=39 loops=1)
   Sort Key: channels.displayname
   Sort Method: quicksort  Memory: 27kB
   Buffers: shared hit=1984
   ->  Nested Loop Anti Join  (cost=9.10..5039.45 rows=463 width=40) (actual time=0.751..12.009 rows=39 loops=1)
         Join Filter: ((sidebarchannels.channelid)::text = (channelmembers.channelid)::text)
         Rows Removed by Join Filter: 7839
         Buffers: shared hit=1984
         ->  Nested Loop  (cost=0.99..3476.66 rows=463 width=67) (actual time=0.161..7.579 rows=39 loops=1)
               Buffers: shared hit=1956
               ->  Index Only Scan using idx_channelmembers_user_id_channel_id_last_viewed_at on channelmembers  (cost=0.56..40.78 rows=470 width=27) (actual time=0.036..0.449 rows=437 loops=1)
                     Index Cond: (userid = 'tc3p1yqw67d8idcp3g98awexqe'::text)
                     Heap Fetches: 45
                     Buffers: shared hit=208
               ->  Memoize  (cost=0.43..7.69 rows=1 width=40) (actual time=0.016..0.016 rows=0 loops=437)
                     Cache Key: channelmembers.channelid
                     Cache Mode: logical
                     Hits: 0  Misses: 437  Evictions: 0  Overflows: 0  Memory Usage: 42kB
                     Buffers: shared hit=1748
                     ->  Index Scan using channels_pkey on channels  (cost=0.42..7.68 rows=1 width=40) (actual time=0.014..0.014 rows=0 loops=437)
                           Index Cond: ((id)::text = (channelmembers.channelid)::text)
                           Filter: ((type = ANY ('{D,G}'::channel_type[])) AND (deleteat = 0))
                           Rows Removed by Filter: 1
                           Buffers: shared hit=1748
         ->  Materialize  (cost=8.11..1535.03 rows=4 width=27) (actual time=0.003..0.046 rows=201 loops=39)
               Buffers: shared hit=28
               ->  Nested Loop  (cost=8.11..1535.01 rows=4 width=27) (actual time=0.099..0.383 rows=201 loops=1)
                     Buffers: shared hit=28
                     ->  Index Scan using idx_sidebarcategories_userid_teamid on sidebarcategories  (cost=0.42..8.44 rows=1 width=65) (actual time=0.047..0.057 rows=6 loops=1)
                           Index Cond: (((userid)::text = 'tc3p1yqw67d8idcp3g98awexqe'::text) AND ((teamid)::text = '3ee5y5ok6jgxicrmqstdnghmfr'::text))
                           Buffers: shared hit=4
                     ->  Bitmap Heap Scan on sidebarchannels  (cost=7.69..1522.35 rows=421 width=92) (actual time=0.028..0.040 rows=34 loops=6)
                           Recheck Cond: ((categoryid)::text = (sidebarcategories.id)::text)
                           Heap Blocks: exact=6
                           Buffers: shared hit=24
                           ->  Bitmap Index Scan on idx_sidebarchannels_categoryid  (cost=0.00..7.58 rows=421 width=0) (actual time=0.023..0.023 rows=34 loops=6)
                                 Index Cond: ((categoryid)::text = (sidebarcategories.id)::text)
                                 Buffers: shared hit=18
 Planning:
   Buffers: shared hit=51
 Planning Time: 2.240 ms
 Execution Time: 12.210 ms
(42 rows)
```

Analysis on MySQL for completion:
```
Before:
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| -> Sort: SidebarCategories.SortOrder, SidebarChannels.SortOrder  (actual time=277.675..277.675 rows=4 loops=1)
    -> Stream results  (cost=138558.36 rows=1287808) (actual time=242.506..277.650 rows=4 loops=1)
        -> Left hash join (<hash>(SidebarChannels.CategoryId)=<hash>(SidebarCategories.Id)), extra conditions: (SidebarChannels.CategoryId = SidebarCategories.Id)  (cost=138558.36 rows=1287808) (actual time=242.498..277.626 rows=4 loops=1)
            -> Index lookup on SidebarCategories using idx_sidebarcategories_userid_teamid (UserId='qdggj9pyobgkjpj8htwzizks1r', TeamId='xmh7bupzajnudqf3h4mm76qapy')  (cost=1.40 rows=4) (actual time=0.092..0.094 rows=4 loops=1)
            -> Hash
                -> Table scan on SidebarChannels  (cost=8394.55 rows=321952) (actual time=0.123..106.334 rows=300002 loops=1)
 |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

After:
----------------------------------------------------------------+
| -> Sort: SidebarCategories.SortOrder, SidebarChannels.SortOrder  (actual time=0.739..0.742 rows=4 loops=1)
    -> Stream results  (cost=6.80 rows=7) (actual time=0.468..0.703 rows=4 loops=1)
        -> Nested loop left join  (cost=6.80 rows=7) (actual time=0.456..0.673 rows=4 loops=1)
            -> Index lookup on SidebarCategories using idx_sidebarcategories_userid_teamid (UserId='qdggj9pyobgkjpj8htwzizks1r', TeamId='xmh7bupzajnudqf3h4mm76qapy')  (cost=4.38 rows=4) (actual time=0.302..0.313 rows=4 loops=1)
            -> Index lookup on SidebarChannels using idx_sidebarchannels_categoryid (CategoryId=SidebarCategories.Id)  (cost=0.48 rows=2) (actual time=0.085..0.087 rows=0 loops=4)
 |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
```

Timing wise, it takes around 2s to add the index on a table with 1.2M rows
for Postgres. And it takes around 5s on MySQL on a table with 300K rows.
It looks like it takes longer on MySQL, but since both migrations are
non-locking, it should be fine.

https://mattermost.atlassian.net/browse/MM-63756
```release-note
NONE
```
2025-04-24 12:11:28 +05:30
Ibrahim Serdar Acikgoz
10b1f4c5ac
[MM-63428] add access control policy store (#30597) 2025-04-02 13:39:28 +02:00
Harshil Sharma
6e738f489f
Channel banner sql migrations (#30274)
* Adde MySQL and Postgres migrations

* Replaced select * with column names

* removed all * from channel SQL store

* cleanup

* Fixed a duplicate column

* cleanup

* Added migrations and store support

* WIP

* used channelname slice in a missed place

* Handled patch

* Added app level tests

* Added API layer tests

* Added API layer tests

* WIP

* converted to query builder

* cleanupo

* added not null and default constraints

* Fixed test

* fixed file name

* review fixes

* review fixes

* updated migration file

* fixed text

* Review fixes
2025-02-25 14:52:15 +05:30
Julien Tant
632a60b332
[MM-62553]+[MM-62554] Property Architecture: cursor based pagination (#30119)
* refactor: Replace pagination with cursor-based pagination for custom profile attributes

* remove pagination loop on property value retrieval for CPA

* add migrations to optimize pagination on property fields and values

* adapt test to remove pagination check

* update migrations list

* postgres: drop index concurrently

* concurrent index manipulation must be done outside of a Tx

* fix: Correct SQL index drop syntax from "OM" to "ON" in migration files

* test: Add CountForGroup test cases for property field store

* refactor: Add CountForGroup method to PropertyFieldStore interface and implementations

* Fix style and i18n

* feat: Add optional deleted property field filtering to CountForGroup method

* refactor: Update CountForGroup to support optional deleted property fields

* test: Add comprehensive tests for CountForGroup with includeDeleted parameter

* adapt test + gen layers

* rename property service method and set the includeDelete to false

* refactor: Remove redundant constant and use CustomProfileAttributesFieldLimit directly

* fix tests

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2025-02-13 15:23:50 +00:00
Agniva De Sarker
ae9e6174e5
[AI assisted]: Improve system console statistics performance (#29899)
```release-note
NONE
```

Co-authored-by: Matthew Birtch <mattbirtch@gmail.com>
2025-02-04 21:54:01 +05:30
Miguel de la Cruz
ecdce71fc4
Adds the main Property System Architecture components (#29644)
* Adds the main Property System Architecture components

This change adds the necessary migrations for the Property Groups,
Fields and Values tables to be created, the store layer and a Property
Service that can be used from the app layer.

* Update property field type to use user instead of person

* Update PropertyFields to allow for unique nondeleted fields and remove redundant indexes

* Update PropertyValues to allow for unique nondeleted fields and remove redundant indexes

* Use StringMap instead of the map[string]any on property fields

* Add i18n strings

* Revert "Use StringMap instead of the map[string]any on property fields"

This reverts commit e2735ab0f8.

* Cast JSON binary data to string and add todo note for StringMap use

* Add mocks to the retrylayer tests

* Cast JSON binary data to string in property value store

* Check for binary parameter instead of casting to string for JSON data

* Check property field type is one of the allowed ones

* Avoid reusing err variable to be explicit about the returned value

* Merge Property System Migrations into one file

* Adds NOT NULL to timestamps at the DB level

* Update stores to use tableSelectQuery instead of a slice var

* Update PropertyField model translations to be more explicit and avoid repetition

* Update PropertyValue model translations to be more explicit and avoid repetition

* Use ExecBuilder instead of ToSql&Exec

* Update property field errors to add context

* Ensure PerPage is greater than zero

* Update store errors to give more context

* Use ExecBuilder in the property stores where possible

* Add an on conflict suffix to the group register to avoid race conditions

* Remove badly used translation string

* Remove unused get in register group method

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2025-01-13 11:41:44 +00:00
Harshil Sharma
e281b3f37e
Feature scheduled messages (#28932)
* Create scheduled post api (#27920)

* Added migration files for Postgres

* Added migrations for MySQL

* Added store method

* Added API and store tests

* Renamed migration after syncing with master

* Added app layer tests

* API is ready

* API is ready

* API is ready

* Renamed migration after syncing with master

* Updated migration list

* Fixed retry layer tests

* Allowed posts with empty messages

* Review fixes

* Reverted an incorrect change

* Renamed migration and fixed ID assignment

* CI

* Send post button changes (#28019)

* added Split button

* WIP

* Added core menu options

* WIP

* WIP

* WIP

* Handled displaying error in creating scheduled post

* lint fixes

* webapp i18n fix

* Review fixes

* Fixed a webapp test

* A few more fixes

* Removed a duplicate comment

* Scheduled post job (#28088)

* Added the job function

* Added query for fetching scheduled posts for pricessing

* WIP

* WIP

* WIP

* WIP

* WIP

* WIP

* Reafactoring of scheduled post job

* Lint fixes

* Updated i18n files

* FInishing touches

* Added tests for GetScheduledPosts

* Added tests for PermanentlyDeleteScheduledPosts

* Updated all layer

* Some changes as discussed with team

* Added tests for UpdatedScheduledPost

* Code review refactoring

* Added job test

* MM-60120 - Custom time selection (#28120)

* Added a common date time picker modal and used it for post reminder

* Added a common date time picker modal and used it for post reminderggp

* Added modal for custom schedule time and fixed TZ issue

* WIP

* Removed event from useSubmit hook

* Removed event from useSubmit hook

* Added timezone handling

* fixed type error

* Updated i18n strings

* Minor cleanup

* updated snapshots

* review fixes

* Handled event

* Supported for having a DM thread open in RHS while in a regular channel

* Review fixes

* MM-60136 - Scheduled messages tab (#28133)

* WIP

* WIP

* Created Tabs and Tab wrapper with added styling

* Added API to get scheduled posts

* WIP

* Displated scheduled post count

* i18n fix

* Added tests

* Handled asetting active tab absed on URL:

* Reverted unintended change

* Added API to client ad OpenAPI specs

* Renamed file

* Adding fileinfo to schedule posts

* Partial review fixes

* Made get scheduled post API return posts by teamID

* review fixes

* Moved scheduled post redux code to MM-redux package

* Usedd selector factory

* WIP:

* WIP:

* Lint fix

* Fixed an incorrect openapi spec file

* Removed redundent permission check

* Clreaed scheduled post data on logout

* Removed unused i18n string:

* lint fix

* Render scheduled posts (#28208)

* WIP

* WIP

* Created Tabs and Tab wrapper with added styling

* Added API to get scheduled posts

* WIP

* Displated scheduled post count

* i18n fix

* Added tests

* Handled asetting active tab absed on URL:

* Reverted unintended change

* Added API to client ad OpenAPI specs

* Renamed file

* Created common component for draft list item

* WIP

* WIP

* Adding fileinfo to schedule posts

* Basic rendering

* Added count badge to tabs

* WIP

* Made the Drafts LHS iteam appear if no drafts exist but scheduled posts do

* Fixed icon size

* Partial review fixes

* Made get scheduled post API return posts by teamID

* Handled initial vs team switch load

* Displayed scheduled date in panel header

* Added error message and error indiocator

* WIP

* review fixes

* WIP Adding error reason tag

* Added error codes

* Moved scheduled post redux code to MM-redux package

* Usedd selector factory

* WIP:

* WIP:

* Lint fix

* Fixed an incorrect openapi spec file

* Removed redundent permission check

* Clreaed scheduled post data on logout

* Removed unused i18n string:

* lint fix

* Opened rescheduling modal

* Updated graphic for empty state of schduled post list

* Added delete scheduled post option and modal

* Badge and timezone fix

* WIP:

* Added send now confirmation modal

* lint

* Webapp i18n fix

* Fixed webapp test

* Fixed a bug where DM/GM scheduled posts weren't immideatly showing up in UI

* Minor fixes

* WIP

* Review fixes

* Review fixes

* Optimisations

* Fixed reducer name

* Moment optimizatin

* Updated route check

* MM-60144 - added API to update a scheduled post (#28248)

* WIP

* Added api and ap layer for update scheduled post ̛̦̄

* Added API to OpenAI specs, Go client and TS client

* removed permissio check

* Added tests

* Fixed tests

* Added PreUpdate method on scheduled post model

* MM-60131 - Reschedule post integration (#28281)

* Handled rescheduling post in webapp

* Added error handling

* MM-60146 - Delete scheduled post api (#28265)

* WIP

* Added api and ap layer for update scheduled post ̛̦̄

* Added API to OpenAI specs, Go client and TS client

* removed permissio check

* Added tests

* Fixed tests

* Added PreUpdate method on scheduled post model

* Added delete scheduled post API

* Added API to Go client and OpenAPI specs

* Added API to TS client

* Added tests

* CI

* Rmeoved two incorrect code comments

* MM-60653 - Integrated delete scheduled post API (#28296)

* Integrated delete scheduled apost API

* Lint fix

* Review fixes

* Excluded draft checks from scheduled posts (#28370)

* Excluded draft checks from scheduled posts

* Added a removed todo

* MM-60125 - Scheduled post channel indicator (#28320)

* Integrated delete scheduled apost API

* Lint fix

* Added state for storing scheduled posts by channel ID

* Refactored redux store to store scheudled posts by ID, thens tore IDs everywhere

* Refactored redux store to store scheudled posts by ID, thens tore IDs everywhere

* WIP

* Added scheduled post indiocator

* Handled single and multiple scheudled posts

* Review fixes

* Fixed styling and handled center channel, RHS and threads view

* Lint fix

* i18n fix

* Fixed a cycling dependency

* Lint fix

* Added some more comments

* Updated styling

* Review fixes

* Added common component for remote user time and scheduled post indicator

* Updated scheduled post count

* Minor change

* Moved CSS code around

* Fixed a bug where files in scheduled post didn't show up until refresh (#28359)

---------

Co-authored-by: Daniel Espino García <larkox@gmail.com>

* Scheduled post config (#28485)

* Added config

* Added config on server and webapp side

* Added config check in server and webapp

* Added license check

* Added license check

* Added placeholder help text

* Added license check to job

* Fixed job test

* Review fixes

* Updated English text

* Review fixes

* MM-60118 - Added index on ScheduledPosts table (#28579)

* Added index

* Updated indexes

* Scheduled posts misc fixes (#28625)

* Added detailed logging for scheduled post job

* Limited scheduled posts processing to 24 hours

* Marked old scheduled posts as unable to send

* Added t5ests

* converted some logs to trace level

* Fixed a bug causing error message to show up on deleting a scheduled post in a deleted thread (#28630)

* Fixed scheduled posts link in RHS (#28659)

* Fixed scheduled posts link in RHS

* Review fixes

* Fix permission name in scheduled posts by team (#28580)

* Fix permission name

* fix wording

---------

Co-authored-by: Mattermost Build <build@mattermost.com>

* FIxed width of generic modal header to fix browser channel modal (#28639)

* Only consider error-free scheduled posts for indicator in channel and RHS (#28683)

* Show only errro free scheudled posts in post box indicator

* Fixed a bug to handle no scheduled posts

* Fixed draft and scheudled post UI in mobile view (#28680)

* MM-60873 and MM-60872 - Fixed a bug with updating scheduled posts (#28656)

* Fixed a bug with updating scheduled posts

* Better selectors

* MOved shceuled post message length validation to app layer

* MM-60732 - Scheduled posts channel link now takes you to the first scheduled post in channel/thread in list (#28768)

* Ordered scheudle dposts by schgeudled at nad create at

* Ordered in client

* Added scroll to target

* Removed classname prop

* Fixed tests

* Added doc

* Import fix

* MM-60961 - Fixed a bug where API used incoming create at date for scheduled post (#28703)

* Fixed a bug where API used incoming create at date for scheduled post

* Stopped sending created at value for scheduled post

* MM-60785 - Fixed a bug where scheduled posts of channel we are no longer member of didn't show up (#28637)

* Fixed a bug where scheduled posts of channel we are no longer member of didn't show up

* Added a comment

* CI

* Used data loader to optimise laoding missing channels

* Minor refactoring

* MM-60963 - Added common checks for post and scheduled posts (#28713)

* Added commen checks for post and scheuled posts

* Sanitised scheduled posts

* Fixed tests

* Splitted post checks into app and context functions

* Added checks on scheduiled posts job as well:

* i18n fix

* Fixed a test

* Renamed a func

* removed duplicate check

* Scheduled posts UI fixes (#28828)

* Fixed send button and time picker borders

* Fixed center alignment of time picker

* Removed on for today and tomorrow

* Lint fix

* Date time modal hover state fix

* Badge fix

* Fixed a mnerge issue

* Scheduled Post send now and add schedule on draft (#28851)

* Added send now option on scheduled posts

* Minor refactoring

* WIP

* WIP

* WIP

* Lint fix

* i18n fix

* Snapshot update

* Review fixes

* Scheduled post inline editing (#28893)

* Added send now option on scheduled posts

* Minor refactoring

* WIP

* WIP

* WIP

* Lint fix

* i18n fix

* Snapshot update

* Displayed editing component in scheduled post

* Added handling for updating scheduled post

* Handle events

* Fixed escape key issue in scheudled post editing

* Fixes

* Displayed error message for editing error

* Don't show mention warning

* Handled dev mode (#28918)

* MInor fixes

* client fix

* Fixes

* CI

* Removed dev mode behaviour temperorily (#29008)

---------

Co-authored-by: Daniel Espino García <larkox@gmail.com>
Co-authored-by: Eva Sarafianou <eva.sarafianou@gmail.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
2024-11-04 11:39:35 +05:30
Julien Tant
1909206e16
[MM-59069] Make sure OTP are actual One Time Password (#28074)
Automatic Merge
2024-09-17 00:44:32 +02:00
Miguel de la Cruz
f8202309ce
Adds logical deletes to shared channel remotes and remote clusters (#28159)
* Adds logical deletes to shared channel remotes and remote clusters

Instead of physically deleting the shared channel remote and remote
clusters records when a channel is unshared, a remote uninvited or a
remote cluster is deleted, now those have a logical `DeleteAt` field
that is set.

This allows us to safely restore shared channels between two remote
clusters (as of now resetting the cursor without backfilling their
contents) and to know which connections were established in the past
and now are severed.

* Delete the index in remoteclusters before adding the new column

* Fix bad error check
2024-09-12 13:55:11 +02:00
Miguel de la Cruz
cd99c4068d
Removes the backslash in migrations.list (#27882)
The use of the backslash was causing issues between different versions
of make.
2024-08-09 11:17:15 +02:00
Miguel de la Cruz
59873a2a50
Adds default team to the remote cluster entity (#27863)
* Adds default team to the remote cluster entity

A new DefaultTeamId field is added to the RemoteCluster entity and its
endpoints, and used when receiving channel invites to choose in which
team to create a new channel.

This will be later extended with the ability for the system admin to
manually accept invites, choosing which team to create the channel on
each. This use case will be triggered when the DefaultTeamId field is
empty, which now simply chooses the first team it finds in the
database as a fallback.

* Fix migrations list

* Fixes channelinvite test case

* Fix i18n

* Fix migration list
2024-08-08 16:36:27 +02:00
Scott Bishel
99881b819a
MM-58548 Remove manage_team permissions from System Console Ancillary permissions (#27395)
* remove manage team permissions from sysconsole_write_user_management_chanels and sysconsole_write_user_management_groups

* update migrations, add unit tests

* run migrations-extract

* make updating ancillary permissions a post

* update file names

* add new api to doc, update body to just be []string

* revert moving ancillary permissions to post

* fix queries after final testing

* Update channel.go

* Update channel.go

* Update 000124_remove_manage_team_permission.up.sql

* Update 000124_remove_manage_team_permission.up.sql

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2024-07-09 08:21:29 -06:00
Scott Bishel
71c25fb316
MM-58525 Fix upload file permissions (#27298)
* tie create_post and upload_file permissions together

* update tests

* update file name

* update migration to do in batches

* Update 000122_remove_upload_file_permission.up.sql

* fix formatting

* simplify migrations, fix regex issue

* update file names for recent merge

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2024-06-24 10:47:52 -06:00
Harrison Healey
5defc75e46
MM-57913 Increase maximum length of Preferences.Value column (#27297)
* MM-57913 Increase maximum length of Preferences.Value column

* Add missing column name to migration

* Lowercased table and column names
2024-06-19 10:39:37 -04:00
Ibrahim Serdar Acikgoz
179d28ee3f
[MM-58341] db/migrations: mysql-8.4 fix for 000027_create_status (#27080) 2024-06-05 18:52:53 +02:00
Nick Misasi
f1019d076e
[CLD-7567] Deprecate Self Serve: Second Pass (#26853)
* Deprecate Self Serve: First Pass

* Fix ci

* Fix more ci

* Remmove outdated server tests

* Fix a missed spot opening purchase modal in Self Hosted

* Fix i18n

* Clean up some more server code, fix webapp test

* Fix alignment of button

* Fix linter

* Fix i18n server side

* Deprecate in product true up

* Add back translation

* Remove client functions

* Put back client functions

* webapp deprecation

* Deprecate Self Serve: Second Pass

* Fix various pipeline issues

* Fix linter

* Fix pipelines

* Fix handlers_test.go

* Fix console.error around hostedCustomer in reducer

* PICKY LINTER PLEASE

* Fix webapp tests, various other fixes for the CI pipelines

* Fix i18n

* Updates to accomadate enterprise code removal

* Fix mocks

* More removal

* Fix

* Adjustments from PR

* Fixes for QA Feedback

* Update

* Add migrations to remove true up review history

* Fix migrations check

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: maria.nunez <maria.nunez@mattermost.com>
2024-05-02 09:15:15 -04:00
Elias Nahum
7e9cd04a8b
Channel Bookmarks (#25449)
* create ChannelBookmarks table

* ChannelBookmark model

* channel bookamrks Store layer

* add GetBookmarksForAllChannelByIdSince

* add channel bookmarks to test store

* Add channel bookmarks to app layer

* remove index for createAt in channel bookmarks migrations

* remove createAt from select channel bookmark query and enable store delete bookmark test

* update reponse of UpdateBookmark

* rename db migration files

* channel bookmarks store update sort order

* channel bookmarks app layer update sort order

* fix lint & tests

* Fix lint and introduce util functions to insert / remove from slice

* remove model etag

* i18n

* defer remove file info after test run

* Fix tests passing the request context

* fix migrations

* fix TestRetry

* Add bookmark permissions (#25560)

* Adds channel bookmarks permissions

* Fix linter

* Remove unnecessary empty lines

* Remove scss change as it's not necessary anymore

* Fix mock store

* Fix mock store and add role entry

* Fix test

* Adds cypress test and update permissions migration to update admin roles

* Adds channel bookmarks roles to default admin roles

* Adds bookmark permissions to default role permissions constant in webapp

* Update mmctl test

* Update permission test after normalising the roles

* fix store tests

* fix app layer tests

* Add new bookmark endpoint (#25624)

* Adds channel bookmarks api scaffold and create endpoint

* Applies review comments to the API docs

* Adds websocket test to create channel bookmark

---------

Co-authored-by: Mattermost Build <build@mattermost.com>

* MM-54426 exclude Channel Bookmarks files from data retention (#25656)

* Augment channel APIs to include bookmarks (#25567)

* update files docs for server 9.4

* Adds update channel bookmark endpoint (#25653)

* Adds update channel bookmark sort order endpoint (#25686)

* Adds update channel bookmark endpoint

* Updates edit app method to return the right deleted bookmark and adds tests

* Adds the update channel bookmark sort order endpoint

* Fix repeated test after merge

* Assign right permissions to each test

* Update store and app layer to return specific errors and add tests

* Adds delete channel bookmark endpoint (#25693)

* Updates edit app method to return the right deleted bookmark and adds tests

* Fix repeated test after merge

* Updates edit app method to return the right deleted bookmark and adds tests

* Adds delete channel bookmark endpoint

* Adds list channel bookmarks endpoint (#25700)

* Add channel moderation to bookmarks (#25716)

* fix migrations index

* fix getChannelsForTeamForUser

* fix getChannelsForTeamForUser

* fix bad merge client4

* fix file api with bookmark permission

* add ChannelBookmarks feature flag

* add missing translations

* Set DB column for type as enum

* use custom type for bookmark query using sqlx

* use transaction when saving bookmark

* return NewErrNotFound instead of Sql.ErrNoRows

* use squirrel for IN query

* add a limit of 1K for records in GetBookmarksForAllChannelByIdSince

* UpdateSortOrder with one single query instead of multiple updates

* fix shadow declaration

* fix channel bookmarks permission string definition in admin console

* fix another shadow declaration

* Fix model conversion

* add SplitSliceInChunks

* remove include bookmarks in channels api

* Cap amount of bookmarks per channel

* add etag back to get channels

* feedback review

* update file info when replacing a bookmark file

* return 501 not implemented when the license is not available

* add detail message when getting channel member on bookmark api

* start audit before permission check on create bookmark api

* use require.Eventuallyf for testing WS events

* remove unnecessary log in app layer

* use require instead of assert to avoid panics

* enforce limit when querying bookmarks since

* prevent to create/update bookmark if file is already attached

* fix lint

* delete file when a bookmark is deleted

* Dot allow to set a fileId and a url at the same time to a bookmark

* fix query to delete a file that belongs to a bookmark

* do not patch the bookmark type

* Server side FeatureFlag check (#26145)

* use ff in server, set ff to false

* turn on FF for unit tests

* defer unset FF for unit tests

* turn ff on for testing

* only allow attaching files that were uploaded for bookmark

* Set feature flag off as default

* fix lint

* update email templates as PR failed

* revert templates

* force the assignment of ID when creating a bookmark

* Fix unit tests

---------

Co-authored-by: Miguel de la Cruz <miguel@mcrx.me>
Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Caleb Roseland <caleb@calebroseland.com>
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
2024-03-12 08:36:05 -06:00
Doug Lauder
d56dc9d0ce
Add Options field to RemoteClusters (#25771)
* add Option flag to RemoteClusters

* add Options column to RemoteClusters table
2023-12-19 13:01:35 -05:00
Devin Binnie
109f4643c6
[MM-55017] Add API method to get users for Admin Reporting (#25499)
* Add store method to get reporting data

* Some store changes

* Added app layer

* Added API call, some miscellaneous fixes

* Fix lint

* Fix serialized check

* Add API docs

* Fix user store tests leaking users

* Fix test

* PR feedback

* Add filtering for role/team/activated user, filter out bot users

* Fix mock

* Fix test

* Oops

* Switch to using struct filter

* More PR feedback

* Fix gen

* Fix test

* Fix API docs

* Fix test

* Fix possible SQL injection, some query optimization

* Fix migrations

* Oops

* Add role to API

* Fix check

* Add Client4 API call for load testing

* Fix test

* Update server/channels/store/storetest/user_store.go

Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>

* PR feedback

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
2023-12-08 10:30:08 -05:00
Doug Lauder
8bf9e4c481
Prep for MS Teams plugin API (#25565)
* - columns added to ShareChannelRemotes: lastpostcreateat, lastpostupdateat
- SyncMsg and SyncResponse moved to `model` package
- field added to RemoteCluster struct: PluginID

* sync new posts before updated posts to ensure post order in MS Teams

* add plugid to remoteclusters table and store

* don't sync history by default
2023-12-04 13:10:20 -05:00
Felipe Martin
b40366dbdf
[MM-55028] Added OAuthOutgoingConnection store (#25221)
* OAuthOutgoingConnection model

* added store

* make generated

* add missing license headers

* fix receiver name

* i18n

* i18n sorting

* update migrations from master

* make migrations-extract

* update retrylayer tests

* replaced sql query with id pagination

* fixed flaky tests

* missing columns

* missing columns on save/update

* typo

* improved tests

* remove enum from mysql colum

* add password credentials to store

* renamed migrations

* model change suggestions

* refactor test functionsn

* migration typo

* refactor store table names

* updated sanitize test

* oauthoutgoingconnection -> outgoingoauthconnection

* signature change

* i18n update

* granttype typo

* uppercase typo

* lowercase store name
2023-11-20 09:42:07 -05:00
Devin Binnie
1bd72bdb99
[MM-55014][MM-55015] Add last login timestamp for users, add materialized view and refresh job to keep track of post stats for Postgres (#25152)
* [MM-55014][MM-55015] Add last login timestamp for users, add materialized view and refresh job for Postgres

* Check fixes

* Fix type issue

* Add verification that lastlogin was updated

* PR feedback

* Morge'd

* Morge'd again

* Merge'd

* Update admin setting strings

* WIP

* PR feedback

* Oops

* Fix i18n

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2023-11-14 11:26:27 -05:00
Ibrahim Serdar Acikgoz
1cd439f8e0
db/migrations: drop nextsyncat and description columns from shared remotes (#25064) 2023-10-31 15:25:16 +03:00
Ben Cooke
791ee40568
[MM-53291] Data retention improvements (#24253)
* adding new migration for RetentionIdsForDeletion, changing logic for deleting orphaned reactions. Updating delete user and channel endpoints to remove respective reactions
2023-09-06 08:25:27 -04:00
Devin Binnie
a3b194581f
[MM-37984] Allow Desktop App to authenticate via external providers outside of the app on supported servers (#24140)
* [MM-37984] Allow Desktop App to authenticate via external providers outside of the app on supported servers

* PR feedback

* Add support for mattermost-dev protocol for development use

* Update server/channels/db/migrations/postgres/000110_create_desktop_tokens.up.sql

* Fix silly typo

* Update server/channels/db/migrations/postgres/000110_create_desktop_tokens.up.sql

* Remove storage of client token, only validate it on the client

* Update migrations

* Add concurrently create index

* Remove CONCURRENTLY for now

* Fix issue with changing history

* Remove old migration

* Use idempotent statement to drop old index

* Remove reference to old table
2023-08-30 11:21:43 -04:00
Agniva De Sarker
555009aa82
Bumping up migration number to 111 (#24341)
PR https://github.com/mattermost/mattermost/pull/24039 was reverted
which left an erroneous migration 110. So to workaround this,
we create a new migration with number 111.

```release-note
NONE
```
2023-08-23 21:15:53 +05:30
Agniva De Sarker
ce92ab7e1c
Update vacuuming parameters for some tables (#23997)
After some analysis of vacuuming statistics of our large
Postgres customers, it was observed that the default vacuuming
thresholds for some tables are not optimal enough for our workload.

We need more frequent vacuuming for some tables. The analysis
showed that the top 4 tables that needed rectification
were posts, preferences, threadmemberships and fileinfo.

So we reduce the threshold by half for those tables, triggering
autovacuum and autoanalyze doubly often.

```release-note
NONE
```
2023-08-02 21:18:18 +05:30
Jesse Hallam
89e65257a6
Remove command palette POC (#24044)
* Revert "POC: Cross-team recent search (#20027)"

This reverts commit aa59c28b04, preserving
a few code tidyings unrelated to the original PR.

* Revert "Add feature flag for command palette (#20011)"

This reverts commit c78c5ce3f3.
2023-07-24 10:31:06 -03:00
Devin Binnie
823ef27d6c
Revert "[MM-37984] Allow Desktop App to authenticate via external providers outside of the app on supported servers (#23795)" (#24039)
This reverts commit abdf4e58c3.
2023-07-17 10:54:53 -04:00
Devin Binnie
abdf4e58c3
[MM-37984] Allow Desktop App to authenticate via external providers outside of the app on supported servers (#23795)
* WIP

* Add rate limiting for desktop token API

* Missing mocks

* Style fixes

* Update snapshots

* Maybe use an actual redirect link :P

* Refactoring for tests

* Add tests for server

* Fix lint issue

* Fix tests

* Fix lint

* Add front-end screen component

* Component logic

* Style changes

* Quick style fix

* Lint fixes

* Initial PR feedback

* Enable logging into the browser as well when completing the login process

* Refactor to push more logic to the other component

* Remove unnecessary helper code

* Fix i18n

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2023-07-12 09:25:05 -04:00
Doug Lauder
7116e9267a
fix typo in index name for idx_teammembers_create_at (#23632) 2023-06-06 10:50:16 -04:00
Vishal
9399ce8637
[MM-47751][MM-48102] MPA: Send Persistent Notifications (#21619)
* MM-46410: adds urgency on mention counts

We have introduced priority for posts in
https://github.com/mattermost/mattermost-webapp/pull/10951.
We do need to color the mention badges in the webapp with a prominent
color when a mention is posted in an urgent message.
A thread has urgent mentions if the root post is marked as urgent, and
the replies contain mentions to the user viewing the thread.

This PR adds two columns, urgentmentioncount, and isurgent, in
channelmembers, and threads tables respectively.
Furthermore when asking for team/thread mention counts, we also return
urgent mention counts for the user.

* Adds PostAcknowledgements table and apis

* job init and fetch mentions

* add-migrations

* delete-expired

* send-notifications

* Fetches post priority in batches

* stop-notifications

* stop-notification-on-reply

* MM-47750: Adds PostAcknowledgements table and apis

- Adds post acknowledgement api/app/store methods to be able to save and
delete post acknowledgements by users.
- Adds wesbsocket events for acknowledgement created/deleted
- Returns post acknowledgements in the post's metadata

* add-license-check

* add-pagination

* delete on channel and team

* validate guests

* add configs

* move create priority post check from app to api

* Add desktop notifications

* check status

* use config in job

* add IsUrgent check

* Add last-sent-at

* validate max recipients

* Update lastSentAt

* Validate min. recipient

* send email notification only once

* remove email notifications

* use latest time from config to run job

* Add notifications counter

* publish events to mentioned users only

* pickup license updates in scheduler

* don't allow post owner to stop notifications

* follow normal notifications behaviour

* Validates persistent notifications interval

* move logic of handling valid and expired posts into sql

* Adds persistent notifications in the webapp

---------

Co-authored-by: koox00 <3829551+koox00@users.noreply.github.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
2023-05-18 23:44:12 +05:30
Ben Cooke
5adbf6c612
[MM-52288] Oauth2 postgres migration issue (#23036)
* fixing migrations that caused some oauth2 issues
2023-05-02 12:32:30 -04:00
Kyriakos Z
a24111f9bd
MM-45009: Delete ThreadMemberships from "left" channels (#22559)
* MM-50550: Filter out threads from "left" channels v2

Currently leaving a channel doesn't affect the thread memberships of
that user/channel combination.
This PR aims to filter out all threads from those channels for the user.

Adds a DeleteAt column in the ThreadMemberships table, and filter out
all thread memberships that are "deleted".
Each time a user leaves a channel all thread memberships are going to be
marked as deleted, and when a user joins a channel again all those
existing thread memberships will be re-instantiated.

Adds a migration to mark all existing thread memberships as deleted
depending on whether there exists a channel membership for that
channel/user.

* Added migration files into list

* Fixes tests

* Fixes case where DeleteAt would be null

* Guard thread API endpoints with appropriate perms

* Deletes ThreadMembership rows upon leaving channel

* Minor style changes

* Use NoTranslation error

* Refactors tests

* Adds API tests to assert permissions on Team

* Adds tests, and fixes migrations

* Fixes test description

* Fix test

* Removes check on DM/GMs

* Change the MySQL query in the migration

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2023-04-19 15:20:34 +03:00
Agniva De Sarker
56b18ca7bf
MM-45956: Optimize FileInfo stats query (#22603)
* MM-45956: Optimize FileInfo stats query

We Denormalize Post.ChannelId on FileInfo.ChannelId

```release-note
The file info stats query is now optimized by denormalizing the channelID column into the table itself. This will speed up the query to get the file count for a channel on clicking the RHS.

Migration times:
On a MySQL 8.0.31 DB with
1405 rows in FileInfo and 11M posts, it took around 0.3s

On a Postgres 12.14 DB with
1731 rows in FileInfo and 11M posts, it took around 0.27s
```

https://mattermost.atlassian.net/browse/MM-45956
2023-03-23 22:14:04 +05:30
Doug Lauder
c943ed6859
Mono repo -> Master (#22553)
Combines the following repositories into one:

https://github.com/mattermost/mattermost-server
https://github.com/mattermost/mattermost-webapp
https://github.com/mattermost/focalboard
https://github.com/mattermost/mattermost-plugin-playbooks
2023-03-22 17:22:27 -04:00