mattermost/server/channels/app/plugin_test.go

1291 lines
41 KiB
Go
Raw Permalink Normal View History

// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
// See LICENSE.txt for license information.
package app
import (
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
"archive/tar"
"compress/gzip"
"crypto/sha256"
"encoding/base64"
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
"encoding/json"
MM-16261: Synchronize plugins in HA (#11657) * MM-16272 - Synchronize plugins across cluster (#11611) * MM-16272 - Synchronize plugins across cluster * Adding a test * MM-16272 - Fixed tests * MM-16272 - PR feedback * MM-16270 - Plugin Sync (#11615) * Initial implementation for plugin synch with file store. WIP * Removed ListAll implementation. Used ListDirectory and change localstore to be consistent and return all items (files and folders) from directory * Refactored plugin filestore operations out of main install/remove plugin * Fixing error handling details * Changes to use structured logging * More logging fixes * Wording and comments improvements * Error handling and control flow improvements * Changed managed flag check to use os.stat * Added file store plugin dir and filename consts * Replaced FileRead to use a the FileReader in PluginSync * Minor styling and PR feedback changes * Minor error handling improvements * Added unit test for SyncPlugins. Changed SyncPlugins to use plugins environment to list available plugins * PR Feedback improvements * Minor err handling fix * Removing FileStorePath from PluginEventData (#11644) * Fix plugin path (#11654) * tweak path, logging Fix an issue not finding the plugins folder in S3. Tweak logging messages to add additional clarity. * Removing FileExists check when Syncing plugins. Updated localstore to not return an error when directory does not exist * PR Feedback * Install prepackaged plugins locally only (#11656) * s/uninstall/remove * Updated ClusterMessage comment * Updated PluginSync to test against s3 + local storage
2019-07-18 14:05:53 -04:00
"fmt"
"io"
MM-16261: Synchronize plugins in HA (#11657) * MM-16272 - Synchronize plugins across cluster (#11611) * MM-16272 - Synchronize plugins across cluster * Adding a test * MM-16272 - Fixed tests * MM-16272 - PR feedback * MM-16270 - Plugin Sync (#11615) * Initial implementation for plugin synch with file store. WIP * Removed ListAll implementation. Used ListDirectory and change localstore to be consistent and return all items (files and folders) from directory * Refactored plugin filestore operations out of main install/remove plugin * Fixing error handling details * Changes to use structured logging * More logging fixes * Wording and comments improvements * Error handling and control flow improvements * Changed managed flag check to use os.stat * Added file store plugin dir and filename consts * Replaced FileRead to use a the FileReader in PluginSync * Minor styling and PR feedback changes * Minor error handling improvements * Added unit test for SyncPlugins. Changed SyncPlugins to use plugins environment to list available plugins * PR Feedback improvements * Minor err handling fix * Removing FileStorePath from PluginEventData (#11644) * Fix plugin path (#11654) * tweak path, logging Fix an issue not finding the plugins folder in S3. Tweak logging messages to add additional clarity. * Removing FileExists check when Syncing plugins. Updated localstore to not return an error when directory does not exist * PR Feedback * Install prepackaged plugins locally only (#11656) * s/uninstall/remove * Updated ClusterMessage comment * Updated PluginSync to test against s3 + local storage
2019-07-18 14:05:53 -04:00
"os"
"path/filepath"
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
"sort"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mattermost/mattermost/server/public/model"
"github.com/mattermost/mattermost/server/public/plugin"
"github.com/mattermost/mattermost/server/public/shared/mlog"
"github.com/mattermost/mattermost/server/v8/channels/testlib"
"github.com/mattermost/mattermost/server/v8/channels/utils/fileutils"
)
func getHashedKey(key string) string {
hash := sha256.New()
hash.Write([]byte(key))
return base64.StdEncoding.EncodeToString(hash.Sum(nil))
}
func TestPluginKeyValueStore(t *testing.T) {
mainHelper.Parallel(t)
th := Setup(t)
pluginID := "testpluginid"
defer func() {
assert.Nil(t, th.App.DeletePluginKey(pluginID, "key"))
assert.Nil(t, th.App.DeletePluginKey(pluginID, "key2"))
assert.Nil(t, th.App.DeletePluginKey(pluginID, "key3"))
assert.Nil(t, th.App.DeletePluginKey(pluginID, "key4"))
}()
assert.Nil(t, th.App.SetPluginKey(pluginID, "key", []byte("test")))
ret, err := th.App.GetPluginKey(pluginID, "key")
assert.Nil(t, err)
assert.Equal(t, []byte("test"), ret)
// Test inserting over existing entries
assert.Nil(t, th.App.SetPluginKey(pluginID, "key", []byte("test2")))
ret, err = th.App.GetPluginKey(pluginID, "key")
assert.Nil(t, err)
assert.Equal(t, []byte("test2"), ret)
// Test getting non-existent key
ret, err = th.App.GetPluginKey(pluginID, "notakey")
assert.Nil(t, err)
assert.Nil(t, ret)
// Test deleting non-existent keys.
assert.Nil(t, th.App.DeletePluginKey(pluginID, "notrealkey"))
// Verify behaviour for the old approach that involved storing the hashed keys.
hashedKey2 := getHashedKey("key2")
kv := &model.PluginKeyValue{
PluginId: pluginID,
Key: hashedKey2,
Value: []byte("test"),
ExpireAt: 0,
}
_, nErr := th.App.Srv().Store().Plugin().SaveOrUpdate(kv)
assert.NoError(t, nErr)
// Test fetch by keyname (this key does not exist but hashed key will be used for lookup)
ret, err = th.App.GetPluginKey(pluginID, "key2")
assert.Nil(t, err)
assert.Equal(t, kv.Value, ret)
// Test fetch by hashed keyname
ret, err = th.App.GetPluginKey(pluginID, hashedKey2)
assert.Nil(t, err)
assert.Equal(t, kv.Value, ret)
// Test ListKeys
assert.Nil(t, th.App.SetPluginKey(pluginID, "key3", []byte("test3")))
assert.Nil(t, th.App.SetPluginKey(pluginID, "key4", []byte("test4")))
list, err := th.App.ListPluginKeys(pluginID, 0, 1)
assert.Nil(t, err)
assert.Equal(t, []string{"key"}, list)
list, err = th.App.ListPluginKeys(pluginID, 1, 1)
assert.Nil(t, err)
assert.Equal(t, []string{"key3"}, list)
list, err = th.App.ListPluginKeys(pluginID, 0, 4)
assert.Nil(t, err)
assert.Equal(t, []string{"key", "key3", "key4", hashedKey2}, list)
list, err = th.App.ListPluginKeys(pluginID, 0, 2)
assert.Nil(t, err)
assert.Equal(t, []string{"key", "key3"}, list)
list, err = th.App.ListPluginKeys(pluginID, 1, 2)
assert.Nil(t, err)
assert.Equal(t, []string{"key4", hashedKey2}, list)
list, err = th.App.ListPluginKeys(pluginID, 2, 2)
assert.Nil(t, err)
assert.Equal(t, []string{}, list)
// List Keys bad input
list, err = th.App.ListPluginKeys(pluginID, 0, 0)
assert.Nil(t, err)
assert.Equal(t, []string{"key", "key3", "key4", hashedKey2}, list)
list, err = th.App.ListPluginKeys(pluginID, 0, -1)
assert.Nil(t, err)
assert.Equal(t, []string{"key", "key3", "key4", hashedKey2}, list)
list, err = th.App.ListPluginKeys(pluginID, -1, 1)
assert.Nil(t, err)
assert.Equal(t, []string{"key"}, list)
list, err = th.App.ListPluginKeys(pluginID, -1, 0)
assert.Nil(t, err)
assert.Equal(t, []string{"key", "key3", "key4", hashedKey2}, list)
}
2017-12-01 10:07:32 -05:00
func TestPluginKeyValueStoreCompareAndSet(t *testing.T) {
mainHelper.Parallel(t)
th := Setup(t)
pluginID := "testpluginid"
defer func() {
assert.Nil(t, th.App.DeletePluginKey(pluginID, "key"))
}()
// Set using Set api for key2
assert.Nil(t, th.App.SetPluginKey(pluginID, "key2", []byte("test")))
ret, err := th.App.GetPluginKey(pluginID, "key2")
assert.Nil(t, err)
assert.Equal(t, []byte("test"), ret)
// Attempt to insert value for key2
updated, err := th.App.CompareAndSetPluginKey(pluginID, "key2", nil, []byte("test2"))
assert.Nil(t, err)
assert.False(t, updated)
ret, err = th.App.GetPluginKey(pluginID, "key2")
assert.Nil(t, err)
assert.Equal(t, []byte("test"), ret)
// Insert new value for key
updated, err = th.App.CompareAndSetPluginKey(pluginID, "key", nil, []byte("test"))
assert.Nil(t, err)
assert.True(t, updated)
ret, err = th.App.GetPluginKey(pluginID, "key")
assert.Nil(t, err)
assert.Equal(t, []byte("test"), ret)
// Should fail to insert again
updated, err = th.App.CompareAndSetPluginKey(pluginID, "key", nil, []byte("test3"))
assert.Nil(t, err)
assert.False(t, updated)
ret, err = th.App.GetPluginKey(pluginID, "key")
assert.Nil(t, err)
assert.Equal(t, []byte("test"), ret)
// Test updating using incorrect old value
updated, err = th.App.CompareAndSetPluginKey(pluginID, "key", []byte("oldvalue"), []byte("test3"))
assert.Nil(t, err)
assert.False(t, updated)
ret, err = th.App.GetPluginKey(pluginID, "key")
assert.Nil(t, err)
assert.Equal(t, []byte("test"), ret)
// Test updating using correct old value
updated, err = th.App.CompareAndSetPluginKey(pluginID, "key", []byte("test"), []byte("test2"))
assert.Nil(t, err)
assert.True(t, updated)
ret, err = th.App.GetPluginKey(pluginID, "key")
assert.Nil(t, err)
assert.Equal(t, []byte("test2"), ret)
}
func TestPluginKeyValueStoreSetWithOptionsJSON(t *testing.T) {
mainHelper.Parallel(t)
pluginID := "testpluginid"
t.Run("storing a value without providing options works", func(t *testing.T) {
th := Setup(t)
result, err := th.App.SetPluginKeyWithOptions(pluginID, "key", []byte("value-1"), model.PluginKVSetOptions{})
assert.True(t, result)
assert.Nil(t, err)
// and I can get it back!
ret, err := th.App.GetPluginKey(pluginID, "key")
assert.Nil(t, err)
assert.Equal(t, []byte(`value-1`), ret)
})
t.Run("test that setting it atomic when it doesn't match doesn't change anything", func(t *testing.T) {
th := Setup(t)
err := th.App.SetPluginKey(pluginID, "key", []byte("value-1"))
require.Nil(t, err)
result, err := th.App.SetPluginKeyWithOptions(pluginID, "key", []byte("value-3"), model.PluginKVSetOptions{
Atomic: true,
OldValue: []byte("value-2"),
})
assert.False(t, result)
assert.Nil(t, err)
// test that the value didn't change
ret, err := th.App.GetPluginKey(pluginID, "key")
assert.Nil(t, err)
assert.Equal(t, []byte(`value-1`), ret)
})
t.Run("test the atomic change with the proper old value", func(t *testing.T) {
th := Setup(t)
err := th.App.SetPluginKey(pluginID, "key", []byte("value-2"))
require.Nil(t, err)
result, err := th.App.SetPluginKeyWithOptions(pluginID, "key", []byte("value-3"), model.PluginKVSetOptions{
Atomic: true,
OldValue: []byte("value-2"),
})
assert.True(t, result)
assert.Nil(t, err)
// test that the value did change
ret, err := th.App.GetPluginKey(pluginID, "key")
assert.Nil(t, err)
assert.Equal(t, []byte(`value-3`), ret)
})
t.Run("when new value is nil and old value matches with the current, it should delete the currently set value", func(t *testing.T) {
th := Setup(t)
// first set a value.
result, err := th.App.SetPluginKeyWithOptions(pluginID, "nil-test-key-2", []byte("value-1"), model.PluginKVSetOptions{})
require.Nil(t, err)
require.True(t, result)
// now it should delete the set value.
result, err = th.App.SetPluginKeyWithOptions(pluginID, "nil-test-key-2", nil, model.PluginKVSetOptions{
Atomic: true,
OldValue: []byte("value-1"),
})
assert.Nil(t, err)
assert.True(t, result)
ret, err := th.App.GetPluginKey(pluginID, "nil-test-key-2")
assert.Nil(t, err)
assert.Nil(t, ret)
})
t.Run("when new value is nil and there is a value set for the key already, it should delete the currently set value", func(t *testing.T) {
th := Setup(t)
// first set a value.
result, err := th.App.SetPluginKeyWithOptions(pluginID, "nil-test-key-3", []byte("value-1"), model.PluginKVSetOptions{})
require.Nil(t, err)
require.True(t, result)
// now it should delete the set value.
result, err = th.App.SetPluginKeyWithOptions(pluginID, "nil-test-key-3", nil, model.PluginKVSetOptions{})
assert.Nil(t, err)
assert.True(t, result)
// verify a nil value is returned
ret, err := th.App.GetPluginKey(pluginID, "nil-test-key-3")
assert.Nil(t, err)
assert.Nil(t, ret)
// verify the row is actually gone
list, err := th.App.ListPluginKeys(pluginID, 0, 1)
assert.Nil(t, err)
assert.Empty(t, list)
})
t.Run("when old value is nil and there is no value set for the key before, it should set the new value", func(t *testing.T) {
th := Setup(t)
result, err := th.App.SetPluginKeyWithOptions(pluginID, "nil-test-key-4", []byte("value-1"), model.PluginKVSetOptions{
Atomic: true,
OldValue: nil,
})
assert.Nil(t, err)
assert.True(t, result)
ret, err := th.App.GetPluginKey(pluginID, "nil-test-key-4")
assert.Nil(t, err)
assert.Equal(t, []byte("value-1"), ret)
})
t.Run("test that value is set and unset with ExpireInSeconds", func(t *testing.T) {
th := Setup(t)
result, err := th.App.SetPluginKeyWithOptions(pluginID, "key", []byte("value-1"), model.PluginKVSetOptions{
ExpireInSeconds: 1,
})
assert.True(t, result)
assert.Nil(t, err)
// test that the value is set
ret, err := th.App.GetPluginKey(pluginID, "key")
assert.Nil(t, err)
assert.Equal(t, []byte(`value-1`), ret)
// test that the value is not longer
time.Sleep(1500 * time.Millisecond)
ret, err = th.App.GetPluginKey(pluginID, "key")
assert.Nil(t, err)
assert.Nil(t, ret)
})
}
MM-8622: Improved plugin error reporting (#8737) * allow `Wait()`ing on the supervisor In the event the plugin supervisor shuts down a plugin for crashing too many times, the new `Wait()` interface allows the `ActivatePlugin` to accept a callback function to trigger when `supervisor.Wait()` returns. If the supervisor shuts down normally, this callback is invoked with a nil error, otherwise any error reported by the supervisor is passed along. * improve plugin activation/deactivation logic Avoid triggering activation of previously failed-to-start plugins just becase something in the configuration changed. Now, intelligently compare the global enable bit as well as the each individual plugin's enabled bit. * expose store to manipulate PluginStatuses * expose API to fetch plugin statuses * keep track of whether or not plugin sandboxing is supported * transition plugin statuses * restore error on plugin activation if already active * don't initialize test plugins until successfully loaded * emit websocket events when plugin statuses change * skip pruning if already initialized * MM-8622: maintain plugin statuses in memory Switch away from persisting plugin statuses to the database, and maintain in memory instead. This will be followed by a cluster interface to query the in-memory status of plugin statuses from all cluster nodes. At the same time, rename `cluster_discovery_id` on the `PluginStatus` model object to `cluster_id`. * MM-8622: aggregate plugin statuses across cluster * fetch cluster plugin statuses when emitting websocket notification * address unit test fixes after rebasing * relax (poor) racey unit test re: supervisor.Wait() * make store-mocks
2018-05-23 14:26:35 -04:00
func TestGetPluginStatusesDisabled(t *testing.T) {
mainHelper.Parallel(t)
th := Setup(t)
MM-8622: Improved plugin error reporting (#8737) * allow `Wait()`ing on the supervisor In the event the plugin supervisor shuts down a plugin for crashing too many times, the new `Wait()` interface allows the `ActivatePlugin` to accept a callback function to trigger when `supervisor.Wait()` returns. If the supervisor shuts down normally, this callback is invoked with a nil error, otherwise any error reported by the supervisor is passed along. * improve plugin activation/deactivation logic Avoid triggering activation of previously failed-to-start plugins just becase something in the configuration changed. Now, intelligently compare the global enable bit as well as the each individual plugin's enabled bit. * expose store to manipulate PluginStatuses * expose API to fetch plugin statuses * keep track of whether or not plugin sandboxing is supported * transition plugin statuses * restore error on plugin activation if already active * don't initialize test plugins until successfully loaded * emit websocket events when plugin statuses change * skip pruning if already initialized * MM-8622: maintain plugin statuses in memory Switch away from persisting plugin statuses to the database, and maintain in memory instead. This will be followed by a cluster interface to query the in-memory status of plugin statuses from all cluster nodes. At the same time, rename `cluster_discovery_id` on the `PluginStatus` model object to `cluster_id`. * MM-8622: aggregate plugin statuses across cluster * fetch cluster plugin statuses when emitting websocket notification * address unit test fixes after rebasing * relax (poor) racey unit test re: supervisor.Wait() * make store-mocks
2018-05-23 14:26:35 -04:00
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.Enable = false
})
MM-8622: Improved plugin error reporting (#8737) * allow `Wait()`ing on the supervisor In the event the plugin supervisor shuts down a plugin for crashing too many times, the new `Wait()` interface allows the `ActivatePlugin` to accept a callback function to trigger when `supervisor.Wait()` returns. If the supervisor shuts down normally, this callback is invoked with a nil error, otherwise any error reported by the supervisor is passed along. * improve plugin activation/deactivation logic Avoid triggering activation of previously failed-to-start plugins just becase something in the configuration changed. Now, intelligently compare the global enable bit as well as the each individual plugin's enabled bit. * expose store to manipulate PluginStatuses * expose API to fetch plugin statuses * keep track of whether or not plugin sandboxing is supported * transition plugin statuses * restore error on plugin activation if already active * don't initialize test plugins until successfully loaded * emit websocket events when plugin statuses change * skip pruning if already initialized * MM-8622: maintain plugin statuses in memory Switch away from persisting plugin statuses to the database, and maintain in memory instead. This will be followed by a cluster interface to query the in-memory status of plugin statuses from all cluster nodes. At the same time, rename `cluster_discovery_id` on the `PluginStatus` model object to `cluster_id`. * MM-8622: aggregate plugin statuses across cluster * fetch cluster plugin statuses when emitting websocket notification * address unit test fixes after rebasing * relax (poor) racey unit test re: supervisor.Wait() * make store-mocks
2018-05-23 14:26:35 -04:00
_, err := th.App.GetPluginStatuses()
require.NotNil(t, err)
require.EqualError(t, err, "GetPluginStatuses: Plugins have been disabled. Please check your logs for details.")
MM-8622: Improved plugin error reporting (#8737) * allow `Wait()`ing on the supervisor In the event the plugin supervisor shuts down a plugin for crashing too many times, the new `Wait()` interface allows the `ActivatePlugin` to accept a callback function to trigger when `supervisor.Wait()` returns. If the supervisor shuts down normally, this callback is invoked with a nil error, otherwise any error reported by the supervisor is passed along. * improve plugin activation/deactivation logic Avoid triggering activation of previously failed-to-start plugins just becase something in the configuration changed. Now, intelligently compare the global enable bit as well as the each individual plugin's enabled bit. * expose store to manipulate PluginStatuses * expose API to fetch plugin statuses * keep track of whether or not plugin sandboxing is supported * transition plugin statuses * restore error on plugin activation if already active * don't initialize test plugins until successfully loaded * emit websocket events when plugin statuses change * skip pruning if already initialized * MM-8622: maintain plugin statuses in memory Switch away from persisting plugin statuses to the database, and maintain in memory instead. This will be followed by a cluster interface to query the in-memory status of plugin statuses from all cluster nodes. At the same time, rename `cluster_discovery_id` on the `PluginStatus` model object to `cluster_id`. * MM-8622: aggregate plugin statuses across cluster * fetch cluster plugin statuses when emitting websocket notification * address unit test fixes after rebasing * relax (poor) racey unit test re: supervisor.Wait() * make store-mocks
2018-05-23 14:26:35 -04:00
}
func TestGetPluginStatuses(t *testing.T) {
mainHelper.Parallel(t)
th := Setup(t)
MM-8622: Improved plugin error reporting (#8737) * allow `Wait()`ing on the supervisor In the event the plugin supervisor shuts down a plugin for crashing too many times, the new `Wait()` interface allows the `ActivatePlugin` to accept a callback function to trigger when `supervisor.Wait()` returns. If the supervisor shuts down normally, this callback is invoked with a nil error, otherwise any error reported by the supervisor is passed along. * improve plugin activation/deactivation logic Avoid triggering activation of previously failed-to-start plugins just becase something in the configuration changed. Now, intelligently compare the global enable bit as well as the each individual plugin's enabled bit. * expose store to manipulate PluginStatuses * expose API to fetch plugin statuses * keep track of whether or not plugin sandboxing is supported * transition plugin statuses * restore error on plugin activation if already active * don't initialize test plugins until successfully loaded * emit websocket events when plugin statuses change * skip pruning if already initialized * MM-8622: maintain plugin statuses in memory Switch away from persisting plugin statuses to the database, and maintain in memory instead. This will be followed by a cluster interface to query the in-memory status of plugin statuses from all cluster nodes. At the same time, rename `cluster_discovery_id` on the `PluginStatus` model object to `cluster_id`. * MM-8622: aggregate plugin statuses across cluster * fetch cluster plugin statuses when emitting websocket notification * address unit test fixes after rebasing * relax (poor) racey unit test re: supervisor.Wait() * make store-mocks
2018-05-23 14:26:35 -04:00
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.Enable = true
})
pluginStatuses, err := th.App.GetPluginStatuses()
require.Nil(t, err)
require.NotNil(t, pluginStatuses)
}
MM-16261: Synchronize plugins in HA (#11657) * MM-16272 - Synchronize plugins across cluster (#11611) * MM-16272 - Synchronize plugins across cluster * Adding a test * MM-16272 - Fixed tests * MM-16272 - PR feedback * MM-16270 - Plugin Sync (#11615) * Initial implementation for plugin synch with file store. WIP * Removed ListAll implementation. Used ListDirectory and change localstore to be consistent and return all items (files and folders) from directory * Refactored plugin filestore operations out of main install/remove plugin * Fixing error handling details * Changes to use structured logging * More logging fixes * Wording and comments improvements * Error handling and control flow improvements * Changed managed flag check to use os.stat * Added file store plugin dir and filename consts * Replaced FileRead to use a the FileReader in PluginSync * Minor styling and PR feedback changes * Minor error handling improvements * Added unit test for SyncPlugins. Changed SyncPlugins to use plugins environment to list available plugins * PR Feedback improvements * Minor err handling fix * Removing FileStorePath from PluginEventData (#11644) * Fix plugin path (#11654) * tweak path, logging Fix an issue not finding the plugins folder in S3. Tweak logging messages to add additional clarity. * Removing FileExists check when Syncing plugins. Updated localstore to not return an error when directory does not exist * PR Feedback * Install prepackaged plugins locally only (#11656) * s/uninstall/remove * Updated ClusterMessage comment * Updated PluginSync to test against s3 + local storage
2019-07-18 14:05:53 -04:00
func TestPluginSync(t *testing.T) {
mainHelper.Parallel(t)
Always require signatures for prepackaged plugins (#31785) * Always require signatures for prepackaged plugins We have always required signatures for packages installed via the marketplace -- whether remotely satisfied, or sourced from the prepackaged plugin cache. However, prepackaged plugins discovered and automatically installed on startup did not require a valid signature. Since we already ship signatures for all Mattermost-authored prepackaged plugins, it's easy to simply start requiring this. Distributions of Mattermost that bundle their own prepackaged plugins will have to include their own signatures. This in turn requires distributing and configuring Mattermost with a custom public key via `PluginSettings.SignaturePublicKeyFiles`. Note that this enhanced security is neutered with a deployment that uses a file-based `config.json`, as any exploit that allows appending to the prepackaged plugins cache probably also allows modifying `config.json` to register a new public key. A [database-based config](https://docs.mattermost.com/configure/configuration-in-your-database.html) is recommended. Finally, we already support an optional setting `PluginSettings.RequirePluginSignature` to always require a plugin signature, although this effectively disables plugin uploads and requires extra effort to deploy the corresponding signature. In environments where only prepackaged plugins are used, this setting is ideal. Fixes: https://mattermost.atlassian.net/browse/MM-64627 * setup dev key, expect no plugins if sig fails * Fix shadow variable errors in test helpers Pre-declare signaturePublicKey variable in loops to avoid shadowing the outer err variable used in error handling. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Replace PrepackagedPlugin.Signature with SignaturePath for memory efficiency - Changed PrepackagedPlugin struct to use SignaturePath string instead of Signature []byte - Updated buildPrepackagedPlugin to use file descriptor instead of reading signature into memory - Modified plugin installation and persistence to read from signature file paths - Updated all tests to check SignaturePath instead of Signature field - Removed unused bytes import from plugin.go This change reduces memory usage by storing file paths instead of signature data in memory while maintaining the same security verification functionality. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
2025-06-24 14:11:02 -04:00
path, _ := fileutils.FindDir("tests")
th := SetupConfig(t, func(cfg *model.Config) {
cfg.PluginSettings.SignaturePublicKeyFiles = []string{
filepath.Join(path, "development-private-key.asc"),
}
})
MM-16261: Synchronize plugins in HA (#11657) * MM-16272 - Synchronize plugins across cluster (#11611) * MM-16272 - Synchronize plugins across cluster * Adding a test * MM-16272 - Fixed tests * MM-16272 - PR feedback * MM-16270 - Plugin Sync (#11615) * Initial implementation for plugin synch with file store. WIP * Removed ListAll implementation. Used ListDirectory and change localstore to be consistent and return all items (files and folders) from directory * Refactored plugin filestore operations out of main install/remove plugin * Fixing error handling details * Changes to use structured logging * More logging fixes * Wording and comments improvements * Error handling and control flow improvements * Changed managed flag check to use os.stat * Added file store plugin dir and filename consts * Replaced FileRead to use a the FileReader in PluginSync * Minor styling and PR feedback changes * Minor error handling improvements * Added unit test for SyncPlugins. Changed SyncPlugins to use plugins environment to list available plugins * PR Feedback improvements * Minor err handling fix * Removing FileStorePath from PluginEventData (#11644) * Fix plugin path (#11654) * tweak path, logging Fix an issue not finding the plugins folder in S3. Tweak logging messages to add additional clarity. * Removing FileExists check when Syncing plugins. Updated localstore to not return an error when directory does not exist * PR Feedback * Install prepackaged plugins locally only (#11656) * s/uninstall/remove * Updated ClusterMessage comment * Updated PluginSync to test against s3 + local storage
2019-07-18 14:05:53 -04:00
testCases := []struct {
Description string
ConfigFunc func(cfg *model.Config)
}{
{
"local",
func(cfg *model.Config) {
cfg.FileSettings.DriverName = model.NewPointer(model.ImageDriverLocal)
MM-16261: Synchronize plugins in HA (#11657) * MM-16272 - Synchronize plugins across cluster (#11611) * MM-16272 - Synchronize plugins across cluster * Adding a test * MM-16272 - Fixed tests * MM-16272 - PR feedback * MM-16270 - Plugin Sync (#11615) * Initial implementation for plugin synch with file store. WIP * Removed ListAll implementation. Used ListDirectory and change localstore to be consistent and return all items (files and folders) from directory * Refactored plugin filestore operations out of main install/remove plugin * Fixing error handling details * Changes to use structured logging * More logging fixes * Wording and comments improvements * Error handling and control flow improvements * Changed managed flag check to use os.stat * Added file store plugin dir and filename consts * Replaced FileRead to use a the FileReader in PluginSync * Minor styling and PR feedback changes * Minor error handling improvements * Added unit test for SyncPlugins. Changed SyncPlugins to use plugins environment to list available plugins * PR Feedback improvements * Minor err handling fix * Removing FileStorePath from PluginEventData (#11644) * Fix plugin path (#11654) * tweak path, logging Fix an issue not finding the plugins folder in S3. Tweak logging messages to add additional clarity. * Removing FileExists check when Syncing plugins. Updated localstore to not return an error when directory does not exist * PR Feedback * Install prepackaged plugins locally only (#11656) * s/uninstall/remove * Updated ClusterMessage comment * Updated PluginSync to test against s3 + local storage
2019-07-18 14:05:53 -04:00
},
},
{
"s3",
func(cfg *model.Config) {
s3Host := os.Getenv("CI_MINIO_HOST")
if s3Host == "" {
s3Host = "localhost"
MM-16261: Synchronize plugins in HA (#11657) * MM-16272 - Synchronize plugins across cluster (#11611) * MM-16272 - Synchronize plugins across cluster * Adding a test * MM-16272 - Fixed tests * MM-16272 - PR feedback * MM-16270 - Plugin Sync (#11615) * Initial implementation for plugin synch with file store. WIP * Removed ListAll implementation. Used ListDirectory and change localstore to be consistent and return all items (files and folders) from directory * Refactored plugin filestore operations out of main install/remove plugin * Fixing error handling details * Changes to use structured logging * More logging fixes * Wording and comments improvements * Error handling and control flow improvements * Changed managed flag check to use os.stat * Added file store plugin dir and filename consts * Replaced FileRead to use a the FileReader in PluginSync * Minor styling and PR feedback changes * Minor error handling improvements * Added unit test for SyncPlugins. Changed SyncPlugins to use plugins environment to list available plugins * PR Feedback improvements * Minor err handling fix * Removing FileStorePath from PluginEventData (#11644) * Fix plugin path (#11654) * tweak path, logging Fix an issue not finding the plugins folder in S3. Tweak logging messages to add additional clarity. * Removing FileExists check when Syncing plugins. Updated localstore to not return an error when directory does not exist * PR Feedback * Install prepackaged plugins locally only (#11656) * s/uninstall/remove * Updated ClusterMessage comment * Updated PluginSync to test against s3 + local storage
2019-07-18 14:05:53 -04:00
}
s3Port := os.Getenv("CI_MINIO_PORT")
if s3Port == "" {
s3Port = "9000"
MM-16261: Synchronize plugins in HA (#11657) * MM-16272 - Synchronize plugins across cluster (#11611) * MM-16272 - Synchronize plugins across cluster * Adding a test * MM-16272 - Fixed tests * MM-16272 - PR feedback * MM-16270 - Plugin Sync (#11615) * Initial implementation for plugin synch with file store. WIP * Removed ListAll implementation. Used ListDirectory and change localstore to be consistent and return all items (files and folders) from directory * Refactored plugin filestore operations out of main install/remove plugin * Fixing error handling details * Changes to use structured logging * More logging fixes * Wording and comments improvements * Error handling and control flow improvements * Changed managed flag check to use os.stat * Added file store plugin dir and filename consts * Replaced FileRead to use a the FileReader in PluginSync * Minor styling and PR feedback changes * Minor error handling improvements * Added unit test for SyncPlugins. Changed SyncPlugins to use plugins environment to list available plugins * PR Feedback improvements * Minor err handling fix * Removing FileStorePath from PluginEventData (#11644) * Fix plugin path (#11654) * tweak path, logging Fix an issue not finding the plugins folder in S3. Tweak logging messages to add additional clarity. * Removing FileExists check when Syncing plugins. Updated localstore to not return an error when directory does not exist * PR Feedback * Install prepackaged plugins locally only (#11656) * s/uninstall/remove * Updated ClusterMessage comment * Updated PluginSync to test against s3 + local storage
2019-07-18 14:05:53 -04:00
}
s3Endpoint := fmt.Sprintf("%s:%s", s3Host, s3Port)
cfg.FileSettings.DriverName = model.NewPointer(model.ImageDriverS3)
cfg.FileSettings.AmazonS3AccessKeyId = model.NewPointer(model.MinioAccessKey)
cfg.FileSettings.AmazonS3SecretAccessKey = model.NewPointer(model.MinioSecretKey)
cfg.FileSettings.AmazonS3Bucket = model.NewPointer(model.MinioBucket)
cfg.FileSettings.AmazonS3PathPrefix = model.NewPointer("")
cfg.FileSettings.AmazonS3Endpoint = model.NewPointer(s3Endpoint)
cfg.FileSettings.AmazonS3Region = model.NewPointer("")
cfg.FileSettings.AmazonS3SSL = model.NewPointer(false)
MM-16261: Synchronize plugins in HA (#11657) * MM-16272 - Synchronize plugins across cluster (#11611) * MM-16272 - Synchronize plugins across cluster * Adding a test * MM-16272 - Fixed tests * MM-16272 - PR feedback * MM-16270 - Plugin Sync (#11615) * Initial implementation for plugin synch with file store. WIP * Removed ListAll implementation. Used ListDirectory and change localstore to be consistent and return all items (files and folders) from directory * Refactored plugin filestore operations out of main install/remove plugin * Fixing error handling details * Changes to use structured logging * More logging fixes * Wording and comments improvements * Error handling and control flow improvements * Changed managed flag check to use os.stat * Added file store plugin dir and filename consts * Replaced FileRead to use a the FileReader in PluginSync * Minor styling and PR feedback changes * Minor error handling improvements * Added unit test for SyncPlugins. Changed SyncPlugins to use plugins environment to list available plugins * PR Feedback improvements * Minor err handling fix * Removing FileStorePath from PluginEventData (#11644) * Fix plugin path (#11654) * tweak path, logging Fix an issue not finding the plugins folder in S3. Tweak logging messages to add additional clarity. * Removing FileExists check when Syncing plugins. Updated localstore to not return an error when directory does not exist * PR Feedback * Install prepackaged plugins locally only (#11656) * s/uninstall/remove * Updated ClusterMessage comment * Updated PluginSync to test against s3 + local storage
2019-07-18 14:05:53 -04:00
},
},
}
for _, testCase := range testCases {
t.Run(testCase.Description, func(t *testing.T) {
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.Enable = true
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
testCase.ConfigFunc(cfg)
MM-16261: Synchronize plugins in HA (#11657) * MM-16272 - Synchronize plugins across cluster (#11611) * MM-16272 - Synchronize plugins across cluster * Adding a test * MM-16272 - Fixed tests * MM-16272 - PR feedback * MM-16270 - Plugin Sync (#11615) * Initial implementation for plugin synch with file store. WIP * Removed ListAll implementation. Used ListDirectory and change localstore to be consistent and return all items (files and folders) from directory * Refactored plugin filestore operations out of main install/remove plugin * Fixing error handling details * Changes to use structured logging * More logging fixes * Wording and comments improvements * Error handling and control flow improvements * Changed managed flag check to use os.stat * Added file store plugin dir and filename consts * Replaced FileRead to use a the FileReader in PluginSync * Minor styling and PR feedback changes * Minor error handling improvements * Added unit test for SyncPlugins. Changed SyncPlugins to use plugins environment to list available plugins * PR Feedback improvements * Minor err handling fix * Removing FileStorePath from PluginEventData (#11644) * Fix plugin path (#11654) * tweak path, logging Fix an issue not finding the plugins folder in S3. Tweak logging messages to add additional clarity. * Removing FileExists check when Syncing plugins. Updated localstore to not return an error when directory does not exist * PR Feedback * Install prepackaged plugins locally only (#11656) * s/uninstall/remove * Updated ClusterMessage comment * Updated PluginSync to test against s3 + local storage
2019-07-18 14:05:53 -04:00
})
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
env := th.App.GetPluginsEnvironment()
require.NotNil(t, env)
MM-16261: Synchronize plugins in HA (#11657) * MM-16272 - Synchronize plugins across cluster (#11611) * MM-16272 - Synchronize plugins across cluster * Adding a test * MM-16272 - Fixed tests * MM-16272 - PR feedback * MM-16270 - Plugin Sync (#11615) * Initial implementation for plugin synch with file store. WIP * Removed ListAll implementation. Used ListDirectory and change localstore to be consistent and return all items (files and folders) from directory * Refactored plugin filestore operations out of main install/remove plugin * Fixing error handling details * Changes to use structured logging * More logging fixes * Wording and comments improvements * Error handling and control flow improvements * Changed managed flag check to use os.stat * Added file store plugin dir and filename consts * Replaced FileRead to use a the FileReader in PluginSync * Minor styling and PR feedback changes * Minor error handling improvements * Added unit test for SyncPlugins. Changed SyncPlugins to use plugins environment to list available plugins * PR Feedback improvements * Minor err handling fix * Removing FileStorePath from PluginEventData (#11644) * Fix plugin path (#11654) * tweak path, logging Fix an issue not finding the plugins folder in S3. Tweak logging messages to add additional clarity. * Removing FileExists check when Syncing plugins. Updated localstore to not return an error when directory does not exist * PR Feedback * Install prepackaged plugins locally only (#11656) * s/uninstall/remove * Updated ClusterMessage comment * Updated PluginSync to test against s3 + local storage
2019-07-18 14:05:53 -04:00
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
t.Run("new bundle in the file store", func(t *testing.T) {
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.RequirePluginSignature = false
})
MM-16261: Synchronize plugins in HA (#11657) * MM-16272 - Synchronize plugins across cluster (#11611) * MM-16272 - Synchronize plugins across cluster * Adding a test * MM-16272 - Fixed tests * MM-16272 - PR feedback * MM-16270 - Plugin Sync (#11615) * Initial implementation for plugin synch with file store. WIP * Removed ListAll implementation. Used ListDirectory and change localstore to be consistent and return all items (files and folders) from directory * Refactored plugin filestore operations out of main install/remove plugin * Fixing error handling details * Changes to use structured logging * More logging fixes * Wording and comments improvements * Error handling and control flow improvements * Changed managed flag check to use os.stat * Added file store plugin dir and filename consts * Replaced FileRead to use a the FileReader in PluginSync * Minor styling and PR feedback changes * Minor error handling improvements * Added unit test for SyncPlugins. Changed SyncPlugins to use plugins environment to list available plugins * PR Feedback improvements * Minor err handling fix * Removing FileStorePath from PluginEventData (#11644) * Fix plugin path (#11654) * tweak path, logging Fix an issue not finding the plugins folder in S3. Tweak logging messages to add additional clarity. * Removing FileExists check when Syncing plugins. Updated localstore to not return an error when directory does not exist * PR Feedback * Install prepackaged plugins locally only (#11656) * s/uninstall/remove * Updated ClusterMessage comment * Updated PluginSync to test against s3 + local storage
2019-07-18 14:05:53 -04:00
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
fileReader, err := os.Open(filepath.Join(path, "testplugin.tar.gz"))
require.NoError(t, err)
defer fileReader.Close()
MM-16261: Synchronize plugins in HA (#11657) * MM-16272 - Synchronize plugins across cluster (#11611) * MM-16272 - Synchronize plugins across cluster * Adding a test * MM-16272 - Fixed tests * MM-16272 - PR feedback * MM-16270 - Plugin Sync (#11615) * Initial implementation for plugin synch with file store. WIP * Removed ListAll implementation. Used ListDirectory and change localstore to be consistent and return all items (files and folders) from directory * Refactored plugin filestore operations out of main install/remove plugin * Fixing error handling details * Changes to use structured logging * More logging fixes * Wording and comments improvements * Error handling and control flow improvements * Changed managed flag check to use os.stat * Added file store plugin dir and filename consts * Replaced FileRead to use a the FileReader in PluginSync * Minor styling and PR feedback changes * Minor error handling improvements * Added unit test for SyncPlugins. Changed SyncPlugins to use plugins environment to list available plugins * PR Feedback improvements * Minor err handling fix * Removing FileStorePath from PluginEventData (#11644) * Fix plugin path (#11654) * tweak path, logging Fix an issue not finding the plugins folder in S3. Tweak logging messages to add additional clarity. * Removing FileExists check when Syncing plugins. Updated localstore to not return an error when directory does not exist * PR Feedback * Install prepackaged plugins locally only (#11656) * s/uninstall/remove * Updated ClusterMessage comment * Updated PluginSync to test against s3 + local storage
2019-07-18 14:05:53 -04:00
_, appErr := th.App.WriteFile(fileReader, getBundleStorePath("testplugin"))
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
checkNoError(t, appErr)
MM-16261: Synchronize plugins in HA (#11657) * MM-16272 - Synchronize plugins across cluster (#11611) * MM-16272 - Synchronize plugins across cluster * Adding a test * MM-16272 - Fixed tests * MM-16272 - PR feedback * MM-16270 - Plugin Sync (#11615) * Initial implementation for plugin synch with file store. WIP * Removed ListAll implementation. Used ListDirectory and change localstore to be consistent and return all items (files and folders) from directory * Refactored plugin filestore operations out of main install/remove plugin * Fixing error handling details * Changes to use structured logging * More logging fixes * Wording and comments improvements * Error handling and control flow improvements * Changed managed flag check to use os.stat * Added file store plugin dir and filename consts * Replaced FileRead to use a the FileReader in PluginSync * Minor styling and PR feedback changes * Minor error handling improvements * Added unit test for SyncPlugins. Changed SyncPlugins to use plugins environment to list available plugins * PR Feedback improvements * Minor err handling fix * Removing FileStorePath from PluginEventData (#11644) * Fix plugin path (#11654) * tweak path, logging Fix an issue not finding the plugins folder in S3. Tweak logging messages to add additional clarity. * Removing FileExists check when Syncing plugins. Updated localstore to not return an error when directory does not exist * PR Feedback * Install prepackaged plugins locally only (#11656) * s/uninstall/remove * Updated ClusterMessage comment * Updated PluginSync to test against s3 + local storage
2019-07-18 14:05:53 -04:00
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
appErr = th.App.SyncPlugins()
checkNoError(t, appErr)
MM-16261: Synchronize plugins in HA (#11657) * MM-16272 - Synchronize plugins across cluster (#11611) * MM-16272 - Synchronize plugins across cluster * Adding a test * MM-16272 - Fixed tests * MM-16272 - PR feedback * MM-16270 - Plugin Sync (#11615) * Initial implementation for plugin synch with file store. WIP * Removed ListAll implementation. Used ListDirectory and change localstore to be consistent and return all items (files and folders) from directory * Refactored plugin filestore operations out of main install/remove plugin * Fixing error handling details * Changes to use structured logging * More logging fixes * Wording and comments improvements * Error handling and control flow improvements * Changed managed flag check to use os.stat * Added file store plugin dir and filename consts * Replaced FileRead to use a the FileReader in PluginSync * Minor styling and PR feedback changes * Minor error handling improvements * Added unit test for SyncPlugins. Changed SyncPlugins to use plugins environment to list available plugins * PR Feedback improvements * Minor err handling fix * Removing FileStorePath from PluginEventData (#11644) * Fix plugin path (#11654) * tweak path, logging Fix an issue not finding the plugins folder in S3. Tweak logging messages to add additional clarity. * Removing FileExists check when Syncing plugins. Updated localstore to not return an error when directory does not exist * PR Feedback * Install prepackaged plugins locally only (#11656) * s/uninstall/remove * Updated ClusterMessage comment * Updated PluginSync to test against s3 + local storage
2019-07-18 14:05:53 -04:00
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
// Check if installed
pluginStatus, err := env.Statuses()
require.NoError(t, err)
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
require.Len(t, pluginStatus, 1)
require.Equal(t, pluginStatus[0].PluginId, "testplugin")
})
MM-16261: Synchronize plugins in HA (#11657) * MM-16272 - Synchronize plugins across cluster (#11611) * MM-16272 - Synchronize plugins across cluster * Adding a test * MM-16272 - Fixed tests * MM-16272 - PR feedback * MM-16270 - Plugin Sync (#11615) * Initial implementation for plugin synch with file store. WIP * Removed ListAll implementation. Used ListDirectory and change localstore to be consistent and return all items (files and folders) from directory * Refactored plugin filestore operations out of main install/remove plugin * Fixing error handling details * Changes to use structured logging * More logging fixes * Wording and comments improvements * Error handling and control flow improvements * Changed managed flag check to use os.stat * Added file store plugin dir and filename consts * Replaced FileRead to use a the FileReader in PluginSync * Minor styling and PR feedback changes * Minor error handling improvements * Added unit test for SyncPlugins. Changed SyncPlugins to use plugins environment to list available plugins * PR Feedback improvements * Minor err handling fix * Removing FileStorePath from PluginEventData (#11644) * Fix plugin path (#11654) * tweak path, logging Fix an issue not finding the plugins folder in S3. Tweak logging messages to add additional clarity. * Removing FileExists check when Syncing plugins. Updated localstore to not return an error when directory does not exist * PR Feedback * Install prepackaged plugins locally only (#11656) * s/uninstall/remove * Updated ClusterMessage comment * Updated PluginSync to test against s3 + local storage
2019-07-18 14:05:53 -04:00
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
t.Run("bundle removed from the file store", func(t *testing.T) {
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.RequirePluginSignature = false
})
MM-16368 - Plugin Signing (#13017) * [MM-18757] POST handler for `/plugins/marketplace` (#12372) * Implement installMarketplacePlugin * Add InstallMarketplacePlugin endpoint * Fix go.mod * merge with master * Fix go.mod * Fix plugin tests * Move get plugin to marketplace client * Fix stylistic concerns * Add trailing newline to the go.mod * [MM-16586] Add plugin signature settings (#12390) * MM-17149 - Extend config.json for marketplace settings (#11933) * MM-17149 - Extend config.json for marketplace settings * Renamed MarketplaceUrl, tracking default marketplace url * Added EnableMarketplace to the client config * Revert "Added EnableMarketplace to the client config" This reverts commit 0f982c4c661c2cd9bb96264e9a01a2363c40d9c5. * MM-17149 - Added EnableMarketplace to the client config (#11958) * Added EnableMarketplace to the client config * Moved EnableMarketplace setting out of limited client configuration * Add public key settings to the config.json * Rename PublicKeys to SignaturePublicKeyFiles * Change filepath.Split to Base * Remove additional prints * Force extention of a public key file * Remove config validation * Remove error on delete * Remove config cloning * Add error messages * Add plugin public key tests * Rename extension to PluginSignaturePublicKeyFileExtention * Remove EnforceVerification * Change []*PublicKeyDescription to []string * Change .asc extension to .plugin.asc * Change ordering of public methods * Change plugin key commands * Update examples in the plugin key commands * Remove forcing extention * Add verify signature in settings * Fix tabbing * Fix naming * Remove unused text * Remove unused text * Update command examples * Fix unit tests * Change errors.New to errors.Wrap * Fix verbose flag * Change .asc to .gpg * Fix } * Change AddPublicKey signature * Change public.key extension * Add plugin public key command tests * Update en.json * Bootstrap the public keys * Update en.json * Fix en.json * Fix en.json * Bootstrap hard-coded public key * Remove unused texts in en.json * Change file to name * Add license header * Update development public key * Remove writeFile method * Remove .plugin.asc extension * Rename publiKey to mattermostPublicKey * Remove init_public_keys string * GolangCI * Closing file handlers * Fixed test that was installing nps plugin * [MM-19798] Implement plugin signature verification (#12768) * MM-17149 - Extend config.json for marketplace settings (#11933) * MM-17149 - Extend config.json for marketplace settings * Renamed MarketplaceUrl, tracking default marketplace url * Added EnableMarketplace to the client config * Revert "Added EnableMarketplace to the client config" This reverts commit 0f982c4c661c2cd9bb96264e9a01a2363c40d9c5. * MM-17149 - Added EnableMarketplace to the client config (#11958) * Added EnableMarketplace to the client config * Moved EnableMarketplace setting out of limited client configuration * Add public key settings to the config.json * Rename PublicKeys to SignaturePublicKeyFiles * Change filepath.Split to Base * Remove additional prints * Force extention of a public key file * Remove config validation * Remove error on delete * Remove config cloning * Add error messages * Add plugin public key tests * Rename extension to PluginSignaturePublicKeyFileExtention * Remove EnforceVerification * Change []*PublicKeyDescription to []string * Change .asc extension to .plugin.asc * Change ordering of public methods * Change plugin key commands * Update examples in the plugin key commands * Remove forcing extention * Add verify signature in settings * Fix tabbing * Fix naming * Remove unused text * Remove unused text * Update command examples * Fix unit tests * Change errors.New to errors.Wrap * Fix verbose flag * Change .asc to .gpg * Fix } * Change AddPublicKey signature * Change public.key extension * Add plugin public key command tests * Update en.json * Bootstrap the public keys * Update en.json * Fix en.json * Fix en.json * Bootstrap hard-coded public key * Remove unused texts in en.json * Change file to name * Add license header * Implement plugin signature verification * Remove benburker openpgp * Update en.json * Update development public key * Add support of multiple signatures in filestore * Update en.json * Run go mod vendor * Fix style * Remove writeFile method * Remove .plugin.asc extension * Rename publiKey to mattermostPublicKey * Verify plugin with mattermost public key * Remove init_public_keys string * Add InstallPluginWithSignature method and Refactor * Add signature verification on claster notification * Remove armored signature headers * Add error strings * Fix en.json * Change signatureStorePath * Implement minor fixes * Refactor plugin install methods * Add installPlugin method to uploadPlugin * Update en.json * Refactor installPlugin * Limit number of signatures * Close signatures * Fix helper function * Fix fromReadCloseSeekerToReadSeeker * Cleaned up ReadCloseSeeker for signatures * Remove signature truncation on FS * GolangCI * Add tests for armored signatures and plugin uploads * Fix nil slice issue * Fix TestPluginSync * Fixed tests * Return io.ReadSeeker from downloadFromUrl * Add log for the found plugins in the file store * Remove logging plugin detection info * [MM-20134] Consume and store single-signature for each plugin (#13081) * Consume and store single-signature for each plugin * Fix en.json * Remove saveSignature method * Remove public key hash * PR Feedback * refactored config * PR feedback
2019-11-18 19:02:41 -05:00
appErr := th.App.RemoveFile(getBundleStorePath("testplugin"))
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
checkNoError(t, appErr)
appErr = th.App.SyncPlugins()
checkNoError(t, appErr)
// Check if removed
pluginStatus, err := env.Statuses()
require.NoError(t, err)
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
require.Empty(t, pluginStatus)
})
t.Run("plugin signatures required, no signature", func(t *testing.T) {
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.RequirePluginSignature = true
})
pluginFileReader, err := os.Open(filepath.Join(path, "testplugin.tar.gz"))
require.NoError(t, err)
defer pluginFileReader.Close()
_, appErr := th.App.WriteFile(pluginFileReader, getBundleStorePath("testplugin"))
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
checkNoError(t, appErr)
appErr = th.App.SyncPlugins()
checkNoError(t, appErr)
pluginStatus, err := env.Statuses()
require.NoError(t, err)
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
require.Len(t, pluginStatus, 0)
})
t.Run("plugin signatures required, wrong signature", func(t *testing.T) {
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.RequirePluginSignature = true
})
signatureFileReader, err := os.Open(filepath.Join(path, "testplugin2.tar.gz.sig"))
require.NoError(t, err)
defer signatureFileReader.Close()
_, appErr := th.App.WriteFile(signatureFileReader, getSignatureStorePath("testplugin"))
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
checkNoError(t, appErr)
appErr = th.App.SyncPlugins()
checkNoError(t, appErr)
pluginStatus, err := env.Statuses()
require.NoError(t, err)
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
require.Len(t, pluginStatus, 0)
})
t.Run("plugin signatures required, correct signature", func(t *testing.T) {
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.RequirePluginSignature = true
})
signatureFileReader, err := os.Open(filepath.Join(path, "testplugin.tar.gz.sig"))
require.NoError(t, err)
defer signatureFileReader.Close()
Always require signatures for prepackaged plugins (#31785) * Always require signatures for prepackaged plugins We have always required signatures for packages installed via the marketplace -- whether remotely satisfied, or sourced from the prepackaged plugin cache. However, prepackaged plugins discovered and automatically installed on startup did not require a valid signature. Since we already ship signatures for all Mattermost-authored prepackaged plugins, it's easy to simply start requiring this. Distributions of Mattermost that bundle their own prepackaged plugins will have to include their own signatures. This in turn requires distributing and configuring Mattermost with a custom public key via `PluginSettings.SignaturePublicKeyFiles`. Note that this enhanced security is neutered with a deployment that uses a file-based `config.json`, as any exploit that allows appending to the prepackaged plugins cache probably also allows modifying `config.json` to register a new public key. A [database-based config](https://docs.mattermost.com/configure/configuration-in-your-database.html) is recommended. Finally, we already support an optional setting `PluginSettings.RequirePluginSignature` to always require a plugin signature, although this effectively disables plugin uploads and requires extra effort to deploy the corresponding signature. In environments where only prepackaged plugins are used, this setting is ideal. Fixes: https://mattermost.atlassian.net/browse/MM-64627 * setup dev key, expect no plugins if sig fails * Fix shadow variable errors in test helpers Pre-declare signaturePublicKey variable in loops to avoid shadowing the outer err variable used in error handling. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Replace PrepackagedPlugin.Signature with SignaturePath for memory efficiency - Changed PrepackagedPlugin struct to use SignaturePath string instead of Signature []byte - Updated buildPrepackagedPlugin to use file descriptor instead of reading signature into memory - Modified plugin installation and persistence to read from signature file paths - Updated all tests to check SignaturePath instead of Signature field - Removed unused bytes import from plugin.go This change reduces memory usage by storing file paths instead of signature data in memory while maintaining the same security verification functionality. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
2025-06-24 14:11:02 -04:00
_, appErr := th.App.WriteFile(signatureFileReader, getSignatureStorePath("testplugin"))
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
checkNoError(t, appErr)
appErr = th.App.SyncPlugins()
checkNoError(t, appErr)
pluginStatus, err := env.Statuses()
require.NoError(t, err)
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
require.Len(t, pluginStatus, 1)
require.Equal(t, pluginStatus[0].PluginId, "testplugin")
appErr = th.App.DeletePublicKey("pub_key")
checkNoError(t, appErr)
appErr = th.App.ch.RemovePlugin("testplugin")
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
checkNoError(t, appErr)
MM-16368 - Plugin Signing (#13017) * [MM-18757] POST handler for `/plugins/marketplace` (#12372) * Implement installMarketplacePlugin * Add InstallMarketplacePlugin endpoint * Fix go.mod * merge with master * Fix go.mod * Fix plugin tests * Move get plugin to marketplace client * Fix stylistic concerns * Add trailing newline to the go.mod * [MM-16586] Add plugin signature settings (#12390) * MM-17149 - Extend config.json for marketplace settings (#11933) * MM-17149 - Extend config.json for marketplace settings * Renamed MarketplaceUrl, tracking default marketplace url * Added EnableMarketplace to the client config * Revert "Added EnableMarketplace to the client config" This reverts commit 0f982c4c661c2cd9bb96264e9a01a2363c40d9c5. * MM-17149 - Added EnableMarketplace to the client config (#11958) * Added EnableMarketplace to the client config * Moved EnableMarketplace setting out of limited client configuration * Add public key settings to the config.json * Rename PublicKeys to SignaturePublicKeyFiles * Change filepath.Split to Base * Remove additional prints * Force extention of a public key file * Remove config validation * Remove error on delete * Remove config cloning * Add error messages * Add plugin public key tests * Rename extension to PluginSignaturePublicKeyFileExtention * Remove EnforceVerification * Change []*PublicKeyDescription to []string * Change .asc extension to .plugin.asc * Change ordering of public methods * Change plugin key commands * Update examples in the plugin key commands * Remove forcing extention * Add verify signature in settings * Fix tabbing * Fix naming * Remove unused text * Remove unused text * Update command examples * Fix unit tests * Change errors.New to errors.Wrap * Fix verbose flag * Change .asc to .gpg * Fix } * Change AddPublicKey signature * Change public.key extension * Add plugin public key command tests * Update en.json * Bootstrap the public keys * Update en.json * Fix en.json * Fix en.json * Bootstrap hard-coded public key * Remove unused texts in en.json * Change file to name * Add license header * Update development public key * Remove writeFile method * Remove .plugin.asc extension * Rename publiKey to mattermostPublicKey * Remove init_public_keys string * GolangCI * Closing file handlers * Fixed test that was installing nps plugin * [MM-19798] Implement plugin signature verification (#12768) * MM-17149 - Extend config.json for marketplace settings (#11933) * MM-17149 - Extend config.json for marketplace settings * Renamed MarketplaceUrl, tracking default marketplace url * Added EnableMarketplace to the client config * Revert "Added EnableMarketplace to the client config" This reverts commit 0f982c4c661c2cd9bb96264e9a01a2363c40d9c5. * MM-17149 - Added EnableMarketplace to the client config (#11958) * Added EnableMarketplace to the client config * Moved EnableMarketplace setting out of limited client configuration * Add public key settings to the config.json * Rename PublicKeys to SignaturePublicKeyFiles * Change filepath.Split to Base * Remove additional prints * Force extention of a public key file * Remove config validation * Remove error on delete * Remove config cloning * Add error messages * Add plugin public key tests * Rename extension to PluginSignaturePublicKeyFileExtention * Remove EnforceVerification * Change []*PublicKeyDescription to []string * Change .asc extension to .plugin.asc * Change ordering of public methods * Change plugin key commands * Update examples in the plugin key commands * Remove forcing extention * Add verify signature in settings * Fix tabbing * Fix naming * Remove unused text * Remove unused text * Update command examples * Fix unit tests * Change errors.New to errors.Wrap * Fix verbose flag * Change .asc to .gpg * Fix } * Change AddPublicKey signature * Change public.key extension * Add plugin public key command tests * Update en.json * Bootstrap the public keys * Update en.json * Fix en.json * Fix en.json * Bootstrap hard-coded public key * Remove unused texts in en.json * Change file to name * Add license header * Implement plugin signature verification * Remove benburker openpgp * Update en.json * Update development public key * Add support of multiple signatures in filestore * Update en.json * Run go mod vendor * Fix style * Remove writeFile method * Remove .plugin.asc extension * Rename publiKey to mattermostPublicKey * Verify plugin with mattermost public key * Remove init_public_keys string * Add InstallPluginWithSignature method and Refactor * Add signature verification on claster notification * Remove armored signature headers * Add error strings * Fix en.json * Change signatureStorePath * Implement minor fixes * Refactor plugin install methods * Add installPlugin method to uploadPlugin * Update en.json * Refactor installPlugin * Limit number of signatures * Close signatures * Fix helper function * Fix fromReadCloseSeekerToReadSeeker * Cleaned up ReadCloseSeeker for signatures * Remove signature truncation on FS * GolangCI * Add tests for armored signatures and plugin uploads * Fix nil slice issue * Fix TestPluginSync * Fixed tests * Return io.ReadSeeker from downloadFromUrl * Add log for the found plugins in the file store * Remove logging plugin detection info * [MM-20134] Consume and store single-signature for each plugin (#13081) * Consume and store single-signature for each plugin * Fix en.json * Remove saveSignature method * Remove public key hash * PR Feedback * refactored config * PR feedback
2019-11-18 19:02:41 -05:00
})
MM-16261: Synchronize plugins in HA (#11657) * MM-16272 - Synchronize plugins across cluster (#11611) * MM-16272 - Synchronize plugins across cluster * Adding a test * MM-16272 - Fixed tests * MM-16272 - PR feedback * MM-16270 - Plugin Sync (#11615) * Initial implementation for plugin synch with file store. WIP * Removed ListAll implementation. Used ListDirectory and change localstore to be consistent and return all items (files and folders) from directory * Refactored plugin filestore operations out of main install/remove plugin * Fixing error handling details * Changes to use structured logging * More logging fixes * Wording and comments improvements * Error handling and control flow improvements * Changed managed flag check to use os.stat * Added file store plugin dir and filename consts * Replaced FileRead to use a the FileReader in PluginSync * Minor styling and PR feedback changes * Minor error handling improvements * Added unit test for SyncPlugins. Changed SyncPlugins to use plugins environment to list available plugins * PR Feedback improvements * Minor err handling fix * Removing FileStorePath from PluginEventData (#11644) * Fix plugin path (#11654) * tweak path, logging Fix an issue not finding the plugins folder in S3. Tweak logging messages to add additional clarity. * Removing FileExists check when Syncing plugins. Updated localstore to not return an error when directory does not exist * PR Feedback * Install prepackaged plugins locally only (#11656) * s/uninstall/remove * Updated ClusterMessage comment * Updated PluginSync to test against s3 + local storage
2019-07-18 14:05:53 -04:00
})
}
}
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
// See https://github.com/mattermost/mattermost-server/issues/19189
func TestChannelsPluginsInit(t *testing.T) {
mainHelper.Parallel(t)
th := Setup(t)
runNoPanicTest := func(t *testing.T) {
path, _ := fileutils.FindDir("tests")
require.NotPanics(t, func() {
th.Server.Channels().initPlugins(th.Context, path, path)
})
}
t.Run("no panics when plugins enabled", func(t *testing.T) {
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.Enable = true
})
runNoPanicTest(t)
})
t.Run("no panics when plugins disabled", func(t *testing.T) {
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.Enable = false
})
runNoPanicTest(t)
})
}
func TestSyncPluginsActiveState(t *testing.T) {
mainHelper.Parallel(t)
th := Setup(t)
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.Enable = true
})
env := th.App.GetPluginsEnvironment()
require.NotNil(t, env)
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.RequirePluginSignature = false
})
path, _ := fileutils.FindDir("tests")
fileReader, err := os.Open(filepath.Join(path, "testplugin.tar.gz"))
require.NoError(t, err)
defer fileReader.Close()
_, appErr := th.App.WriteFile(fileReader, getBundleStorePath("testplugin"))
checkNoError(t, appErr)
// Sync with file store so the plugin environment has access to this plugin.
appErr = th.App.SyncPlugins()
checkNoError(t, appErr)
// Verify the plugin was installed and set to deactivated.
pluginStatus, err := env.Statuses()
require.NoError(t, err)
require.Len(t, pluginStatus, 1)
require.Equal(t, pluginStatus[0].PluginId, "testplugin")
require.Equal(t, pluginStatus[0].State, model.PluginStateNotRunning)
// Enable plugin by setting setting config. This implicitly calls SyncPluginsActiveState through a config listener.
th.App.UpdateConfig(func(cfg *model.Config) {
cfg.PluginSettings.PluginStates["testplugin"] = &model.PluginState{Enable: true}
})
// Verify the plugin was activated due to config change.
pluginStatus, err = env.Statuses()
require.NoError(t, err)
require.Len(t, pluginStatus, 1)
require.Equal(t, pluginStatus[0].PluginId, "testplugin")
require.Equal(t, pluginStatus[0].State, model.PluginStateRunning)
// Disable plugin by setting config. This implicitly calls SyncPluginsActiveState through a config listener.
th.App.UpdateConfig(func(cfg *model.Config) {
cfg.PluginSettings.PluginStates["testplugin"] = &model.PluginState{Enable: false}
})
// Verify the plugin was deactivated due to config change.
pluginStatus, err = env.Statuses()
require.NoError(t, err)
require.Len(t, pluginStatus, 1)
require.Equal(t, pluginStatus[0].PluginId, "testplugin")
require.Equal(t, pluginStatus[0].State, model.PluginStateNotRunning)
}
func TestPluginPanicLogs(t *testing.T) {
mainHelper.Parallel(t)
t.Run("should panic", func(t *testing.T) {
th := Setup(t).InitBasic(t)
tearDown, _, _ := SetAppEnvironmentWithPlugins(t, []string{
`
package main
import (
"github.com/mattermost/mattermost/server/public/plugin"
"github.com/mattermost/mattermost/server/public/model"
)
type MyPlugin struct {
plugin.MattermostPlugin
}
func (p *MyPlugin) MessageWillBePosted(c *plugin.Context, post *model.Post) (*model.Post, string) {
panic("some text from panic")
return nil, ""
}
func main() {
plugin.ClientMain(&MyPlugin{})
}
`,
}, th.App, th.NewPluginAPI)
post := &model.Post{
UserId: th.BasicUser.Id,
ChannelId: th.BasicChannel.Id,
Message: "message_",
CreateAt: model.GetMillis() - 10000,
}
_, err := th.App.CreatePost(th.Context, post, th.BasicChannel, model.CreatePostFlags{SetOnline: true})
assert.Nil(t, err)
th.TestLogger.Flush()
// We shutdown plugins first so that the read on the log buffer is race-free.
th.App.ch.ShutDownPlugins()
tearDown()
testlib.AssertLog(t, th.LogBuffer, mlog.LvlDebug.Name, "panic: some text from panic")
})
}
func TestPluginStatusActivateError(t *testing.T) {
mainHelper.Parallel(t)
t.Run("should return error from OnActivate in plugin statuses", func(t *testing.T) {
th := Setup(t).InitBasic(t)
pluginSource := `
package main
import (
"errors"
"github.com/mattermost/mattermost/server/public/plugin"
)
type MyPlugin struct {
plugin.MattermostPlugin
}
func (p *MyPlugin) OnActivate() error {
return errors.New("sample error")
}
func main() {
plugin.ClientMain(&MyPlugin{})
}
`
tearDown, _, _ := SetAppEnvironmentWithPlugins(t, []string{pluginSource}, th.App, th.NewPluginAPI)
defer tearDown()
env := th.App.GetPluginsEnvironment()
pluginStatus, err := env.Statuses()
require.NoError(t, err)
require.Len(t, pluginStatus, 1)
require.Equal(t, "sample error", pluginStatus[0].Error)
})
}
type byID []*plugin.PrepackagedPlugin
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
func (a byID) Len() int { return len(a) }
func (a byID) Swap(i, j int) { a[i], a[j] = a[j], a[i] }
func (a byID) Less(i, j int) bool { return a[i].Manifest.Id < a[j].Manifest.Id }
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
type pluginStatusById model.PluginStatuses
func (a pluginStatusById) Len() int { return len(a) }
func (a pluginStatusById) Swap(i, j int) { a[i], a[j] = a[j], a[i] }
func (a pluginStatusById) Less(i, j int) bool { return a[i].PluginId < a[j].PluginId }
func TestProcessPrepackagedPlugins(t *testing.T) {
mainHelper.Parallel(t)
Always require signatures for prepackaged plugins (#31785) * Always require signatures for prepackaged plugins We have always required signatures for packages installed via the marketplace -- whether remotely satisfied, or sourced from the prepackaged plugin cache. However, prepackaged plugins discovered and automatically installed on startup did not require a valid signature. Since we already ship signatures for all Mattermost-authored prepackaged plugins, it's easy to simply start requiring this. Distributions of Mattermost that bundle their own prepackaged plugins will have to include their own signatures. This in turn requires distributing and configuring Mattermost with a custom public key via `PluginSettings.SignaturePublicKeyFiles`. Note that this enhanced security is neutered with a deployment that uses a file-based `config.json`, as any exploit that allows appending to the prepackaged plugins cache probably also allows modifying `config.json` to register a new public key. A [database-based config](https://docs.mattermost.com/configure/configuration-in-your-database.html) is recommended. Finally, we already support an optional setting `PluginSettings.RequirePluginSignature` to always require a plugin signature, although this effectively disables plugin uploads and requires extra effort to deploy the corresponding signature. In environments where only prepackaged plugins are used, this setting is ideal. Fixes: https://mattermost.atlassian.net/browse/MM-64627 * setup dev key, expect no plugins if sig fails * Fix shadow variable errors in test helpers Pre-declare signaturePublicKey variable in loops to avoid shadowing the outer err variable used in error handling. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Replace PrepackagedPlugin.Signature with SignaturePath for memory efficiency - Changed PrepackagedPlugin struct to use SignaturePath string instead of Signature []byte - Updated buildPrepackagedPlugin to use file descriptor instead of reading signature into memory - Modified plugin installation and persistence to read from signature file paths - Updated all tests to check SignaturePath instead of Signature field - Removed unused bytes import from plugin.go This change reduces memory usage by storing file paths instead of signature data in memory while maintaining the same security verification functionality. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
2025-06-24 14:11:02 -04:00
testsPath, found := fileutils.FindDir("tests")
require.True(t, found, "failed to find tests directory")
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
setup := func(t *testing.T) *TestHelper {
t.Helper()
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
Always require signatures for prepackaged plugins (#31785) * Always require signatures for prepackaged plugins We have always required signatures for packages installed via the marketplace -- whether remotely satisfied, or sourced from the prepackaged plugin cache. However, prepackaged plugins discovered and automatically installed on startup did not require a valid signature. Since we already ship signatures for all Mattermost-authored prepackaged plugins, it's easy to simply start requiring this. Distributions of Mattermost that bundle their own prepackaged plugins will have to include their own signatures. This in turn requires distributing and configuring Mattermost with a custom public key via `PluginSettings.SignaturePublicKeyFiles`. Note that this enhanced security is neutered with a deployment that uses a file-based `config.json`, as any exploit that allows appending to the prepackaged plugins cache probably also allows modifying `config.json` to register a new public key. A [database-based config](https://docs.mattermost.com/configure/configuration-in-your-database.html) is recommended. Finally, we already support an optional setting `PluginSettings.RequirePluginSignature` to always require a plugin signature, although this effectively disables plugin uploads and requires extra effort to deploy the corresponding signature. In environments where only prepackaged plugins are used, this setting is ideal. Fixes: https://mattermost.atlassian.net/browse/MM-64627 * setup dev key, expect no plugins if sig fails * Fix shadow variable errors in test helpers Pre-declare signaturePublicKey variable in loops to avoid shadowing the outer err variable used in error handling. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Replace PrepackagedPlugin.Signature with SignaturePath for memory efficiency - Changed PrepackagedPlugin struct to use SignaturePath string instead of Signature []byte - Updated buildPrepackagedPlugin to use file descriptor instead of reading signature into memory - Modified plugin installation and persistence to read from signature file paths - Updated all tests to check SignaturePath instead of Signature field - Removed unused bytes import from plugin.go This change reduces memory usage by storing file paths instead of signature data in memory while maintaining the same security verification functionality. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
2025-06-24 14:11:02 -04:00
th := SetupConfig(t, func(cfg *model.Config) {
cfg.PluginSettings.SignaturePublicKeyFiles = []string{
filepath.Join(testsPath, "development-private-key.asc"),
}
})
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
// Make a prepackaged_plugins directory for use with the tests.
err := os.Mkdir(filepath.Join(th.tempWorkspace, prepackagedPluginsDir), os.ModePerm)
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
require.NoError(t, err)
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.Enable = true
*cfg.PluginSettings.EnableRemoteMarketplace = false
})
return th
}
initPlugins := func(t *testing.T, th *TestHelper) ([]*plugin.PrepackagedPlugin, []*plugin.PrepackagedPlugin) {
t.Helper()
appErr := th.App.ch.syncPlugins()
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
require.Nil(t, appErr)
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
err := th.App.ch.processPrepackagedPlugins(filepath.Join(th.tempWorkspace, prepackagedPluginsDir))
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
require.NoError(t, err)
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
env := th.App.GetPluginsEnvironment()
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
plugins, transitionalPlugins := env.PrepackagedPlugins(), env.TransitionallyPrepackagedPlugins()
th.App.ch.persistTransitionallyPrepackagedPlugins()
return plugins, transitionalPlugins
}
copyAsPrepackagedPlugin := func(t *testing.T, th *TestHelper, filename string) {
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
t.Helper()
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
err := testlib.CopyFile(filepath.Join(testsPath, filename), filepath.Join(th.tempWorkspace, prepackagedPluginsDir, filename))
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
require.NoError(t, err)
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
err = testlib.CopyFile(filepath.Join(testsPath, fmt.Sprintf("%s.sig", filename)), filepath.Join(th.tempWorkspace, prepackagedPluginsDir, fmt.Sprintf("%s.sig", filename)))
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
require.NoError(t, err)
}
copyAsFilestorePlugin := func(t *testing.T, th *TestHelper, bundleFilename, pluginID string) {
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
t.Helper()
err := testlib.CopyFile(filepath.Join(testsPath, bundleFilename), filepath.Join(*th.App.Config().FileSettings.Directory, fmt.Sprintf("plugins/%s.tar.gz", pluginID)))
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
require.NoError(t, err)
err = testlib.CopyFile(filepath.Join(testsPath, fmt.Sprintf("%s.sig", bundleFilename)), filepath.Join(*th.App.Config().FileSettings.Directory, fmt.Sprintf("plugins/%s.tar.gz.sig", pluginID)))
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
require.NoError(t, err)
}
expectPrepackagedPlugin := func(t *testing.T, pluginID, version string, actual *plugin.PrepackagedPlugin) {
t.Helper()
require.Equal(t, pluginID, actual.Manifest.Id)
Always require signatures for prepackaged plugins (#31785) * Always require signatures for prepackaged plugins We have always required signatures for packages installed via the marketplace -- whether remotely satisfied, or sourced from the prepackaged plugin cache. However, prepackaged plugins discovered and automatically installed on startup did not require a valid signature. Since we already ship signatures for all Mattermost-authored prepackaged plugins, it's easy to simply start requiring this. Distributions of Mattermost that bundle their own prepackaged plugins will have to include their own signatures. This in turn requires distributing and configuring Mattermost with a custom public key via `PluginSettings.SignaturePublicKeyFiles`. Note that this enhanced security is neutered with a deployment that uses a file-based `config.json`, as any exploit that allows appending to the prepackaged plugins cache probably also allows modifying `config.json` to register a new public key. A [database-based config](https://docs.mattermost.com/configure/configuration-in-your-database.html) is recommended. Finally, we already support an optional setting `PluginSettings.RequirePluginSignature` to always require a plugin signature, although this effectively disables plugin uploads and requires extra effort to deploy the corresponding signature. In environments where only prepackaged plugins are used, this setting is ideal. Fixes: https://mattermost.atlassian.net/browse/MM-64627 * setup dev key, expect no plugins if sig fails * Fix shadow variable errors in test helpers Pre-declare signaturePublicKey variable in loops to avoid shadowing the outer err variable used in error handling. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Replace PrepackagedPlugin.Signature with SignaturePath for memory efficiency - Changed PrepackagedPlugin struct to use SignaturePath string instead of Signature []byte - Updated buildPrepackagedPlugin to use file descriptor instead of reading signature into memory - Modified plugin installation and persistence to read from signature file paths - Updated all tests to check SignaturePath instead of Signature field - Removed unused bytes import from plugin.go This change reduces memory usage by storing file paths instead of signature data in memory while maintaining the same security verification functionality. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
2025-06-24 14:11:02 -04:00
require.NotEmpty(t, actual.SignaturePath, "testplugin has no signature")
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
require.Equal(t, version, actual.Manifest.Version)
}
expectPluginInFilestore := func(t *testing.T, th *TestHelper, pluginID, version string) {
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
t.Helper()
bundlePath := filepath.Join(*th.App.Config().FileSettings.Directory, fmt.Sprintf("plugins/%s.tar.gz", pluginID))
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
require.FileExists(t, bundlePath)
require.FileExists(t, filepath.Join(*th.App.Config().FileSettings.Directory, fmt.Sprintf("plugins/%s.tar.gz.sig", pluginID)))
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
// Verify the version recorded in the Manifest
f, err := os.Open(bundlePath)
require.NoError(t, err)
uncompressedStream, err := gzip.NewReader(f)
require.NoError(t, err)
tarReader := tar.NewReader(uncompressedStream)
var manifest model.Manifest
for {
header, err := tarReader.Next()
if err == io.EOF {
break
}
require.NoError(t, err)
t.Log(header.Name)
if header.Typeflag == tar.TypeReg && filepath.Base(header.Name) == "plugin.json" {
manifestReader := json.NewDecoder(tarReader)
err = manifestReader.Decode(&manifest)
require.NoError(t, err)
break
}
}
require.Equal(t, pluginID, manifest.Id, "failed to find manifest")
require.Equal(t, version, manifest.Version)
}
expectPluginNotInFilestore := func(t *testing.T, th *TestHelper, pluginID string) {
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
t.Helper()
require.NoFileExists(t, filepath.Join(*th.App.Config().FileSettings.Directory, fmt.Sprintf("plugins/%s.tar.gz", pluginID)))
require.NoFileExists(t, filepath.Join(*th.App.Config().FileSettings.Directory, fmt.Sprintf("plugins/%s.tar.gz.sig", pluginID)))
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
}
expectPluginStatus := func(t *testing.T, pluginID, version string, actual *model.PluginStatus) {
t.Helper()
require.Equal(t, pluginID, actual.PluginId)
require.Equal(t, version, actual.Version)
}
t.Run("single plugin automatically installed since enabled", func(t *testing.T) {
th := setup(t)
env := th.App.GetPluginsEnvironment()
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.AutomaticPrepackagedPlugins = true
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
cfg.PluginSettings.PluginStates["testplugin"] = &model.PluginState{Enable: true}
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
})
copyAsPrepackagedPlugin(t, th, "testplugin.tar.gz")
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
plugins, transitionalPlugins := initPlugins(t, th)
require.Len(t, plugins, 1, "expected one prepackaged plugin")
expectPrepackagedPlugin(t, "testplugin", "0.0.1", plugins[0])
require.Empty(t, transitionalPlugins)
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
pluginStatus, err := env.Statuses()
require.NoError(t, err)
require.Len(t, pluginStatus, 1)
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
expectPluginStatus(t, "testplugin", "0.0.1", pluginStatus[0])
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
expectPluginNotInFilestore(t, th, "testplugin")
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
})
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
t.Run("single plugin, not automatically installed since not enabled", func(t *testing.T) {
th := setup(t)
env := th.App.GetPluginsEnvironment()
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.AutomaticPrepackagedPlugins = true
})
copyAsPrepackagedPlugin(t, th, "testplugin.tar.gz")
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
plugins, transitionalPlugins := initPlugins(t, th)
require.Len(t, plugins, 1, "expected one prepackaged plugin")
expectPrepackagedPlugin(t, "testplugin", "0.0.1", plugins[0])
require.Empty(t, transitionalPlugins)
pluginStatus, err := env.Statuses()
require.NoError(t, err)
require.Empty(t, pluginStatus, 0)
expectPluginNotInFilestore(t, th, "testplugin")
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
})
t.Run("single plugin, not automatically installed despite enabled since automatic prepackaged plugins disabled", func(t *testing.T) {
th := setup(t)
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
env := th.App.GetPluginsEnvironment()
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.AutomaticPrepackagedPlugins = false
cfg.PluginSettings.PluginStates["testplugin"] = &model.PluginState{Enable: true}
})
copyAsPrepackagedPlugin(t, th, "testplugin.tar.gz")
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
plugins, transitionalPlugins := initPlugins(t, th)
require.Len(t, plugins, 1, "expected one prepackaged plugin")
expectPrepackagedPlugin(t, "testplugin", "0.0.1", plugins[0])
require.Empty(t, transitionalPlugins)
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
pluginStatus, err := env.Statuses()
require.NoError(t, err)
require.Empty(t, pluginStatus, 0)
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
expectPluginNotInFilestore(t, th, "testplugin")
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
})
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
t.Run("multiple plugins, some automatically installed since enabled", func(t *testing.T) {
th := setup(t)
env := th.App.GetPluginsEnvironment()
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.AutomaticPrepackagedPlugins = true
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
cfg.PluginSettings.PluginStates["testplugin2"] = &model.PluginState{Enable: true}
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
})
copyAsPrepackagedPlugin(t, th, "testplugin.tar.gz")
copyAsPrepackagedPlugin(t, th, "testplugin2.tar.gz")
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
plugins, transitionalPlugins := initPlugins(t, th)
require.Len(t, plugins, 2, "expected two prepackaged plugins")
sort.Sort(byID(plugins))
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
expectPrepackagedPlugin(t, "testplugin", "0.0.1", plugins[0])
expectPrepackagedPlugin(t, "testplugin2", "1.2.3", plugins[1])
require.Empty(t, transitionalPlugins)
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
pluginStatus, err := env.Statuses()
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
require.NoError(t, err)
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
require.Len(t, pluginStatus, 1)
expectPluginStatus(t, "testplugin2", "1.2.3", pluginStatus[0])
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
expectPluginNotInFilestore(t, th, "testplugin")
expectPluginNotInFilestore(t, th, "testplugin2")
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
})
t.Run("multiple plugins, one previously installed, all now installed", func(t *testing.T) {
th := setup(t)
env := th.App.GetPluginsEnvironment()
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.AutomaticPrepackagedPlugins = true
})
copyAsFilestorePlugin(t, th, "testplugin.tar.gz", "testplugin")
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
th.App.UpdateConfig(func(cfg *model.Config) {
cfg.PluginSettings.PluginStates["testplugin"] = &model.PluginState{Enable: true}
cfg.PluginSettings.PluginStates["testplugin2"] = &model.PluginState{Enable: true}
})
copyAsPrepackagedPlugin(t, th, "testplugin.tar.gz")
copyAsPrepackagedPlugin(t, th, "testplugin2.tar.gz")
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
plugins, transitionalPlugins := initPlugins(t, th)
require.Len(t, plugins, 2, "expected two prepackaged plugins")
sort.Sort(byID(plugins))
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
expectPrepackagedPlugin(t, "testplugin", "0.0.1", plugins[0])
expectPrepackagedPlugin(t, "testplugin2", "1.2.3", plugins[1])
require.Empty(t, transitionalPlugins)
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
pluginStatus, err := env.Statuses()
require.NoError(t, err)
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
sort.Sort(pluginStatusById(pluginStatus))
require.Len(t, pluginStatus, 2)
expectPluginStatus(t, "testplugin", "0.0.1", pluginStatus[0])
expectPluginStatus(t, "testplugin2", "1.2.3", pluginStatus[1])
expectPluginInFilestore(t, th, "testplugin", "0.0.1")
expectPluginNotInFilestore(t, th, "testplugin2")
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
})
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
t.Run("multiple plugins, one previously installed and now upgraded, all now installed", func(t *testing.T) {
th := setup(t)
env := th.App.GetPluginsEnvironment()
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.AutomaticPrepackagedPlugins = true
})
copyAsFilestorePlugin(t, th, "testplugin.tar.gz", "testplugin")
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
th.App.UpdateConfig(func(cfg *model.Config) {
cfg.PluginSettings.PluginStates["testplugin"] = &model.PluginState{Enable: true}
cfg.PluginSettings.PluginStates["testplugin2"] = &model.PluginState{Enable: true}
})
copyAsPrepackagedPlugin(t, th, "testplugin-v0.0.2.tar.gz")
copyAsPrepackagedPlugin(t, th, "testplugin2.tar.gz")
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
plugins, transitionalPlugins := initPlugins(t, th)
err := th.App.ch.processPrepackagedPlugins(filepath.Join(th.tempWorkspace, prepackagedPluginsDir))
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
require.NoError(t, err)
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
require.Len(t, plugins, 2, "expected two prepackaged plugins")
sort.Sort(byID(plugins))
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
expectPrepackagedPlugin(t, "testplugin", "0.0.2", plugins[0])
expectPrepackagedPlugin(t, "testplugin2", "1.2.3", plugins[1])
require.Empty(t, transitionalPlugins)
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
pluginStatus, err := env.Statuses()
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
require.NoError(t, err)
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
sort.Sort(pluginStatusById(pluginStatus))
require.Len(t, pluginStatus, 2)
expectPluginStatus(t, "testplugin", "0.0.2", pluginStatus[0])
expectPluginStatus(t, "testplugin2", "1.2.3", pluginStatus[1])
expectPluginInFilestore(t, th, "testplugin", "0.0.1")
expectPluginNotInFilestore(t, th, "testplugin2")
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
})
t.Run("multiple plugins, one previously installed but prepackaged is older, all now installed", func(t *testing.T) {
th := setup(t)
env := th.App.GetPluginsEnvironment()
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.AutomaticPrepackagedPlugins = true
})
copyAsFilestorePlugin(t, th, "testplugin-v0.0.2.tar.gz", "testplugin")
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
th.App.UpdateConfig(func(cfg *model.Config) {
cfg.PluginSettings.PluginStates["testplugin"] = &model.PluginState{Enable: true}
cfg.PluginSettings.PluginStates["testplugin2"] = &model.PluginState{Enable: true}
})
copyAsPrepackagedPlugin(t, th, "testplugin.tar.gz")
copyAsPrepackagedPlugin(t, th, "testplugin2.tar.gz")
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
plugins, transitionalPlugins := initPlugins(t, th)
require.Len(t, plugins, 2, "expected two prepackaged plugins")
sort.Sort(byID(plugins))
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
expectPrepackagedPlugin(t, "testplugin", "0.0.1", plugins[0])
expectPrepackagedPlugin(t, "testplugin2", "1.2.3", plugins[1])
require.Empty(t, transitionalPlugins)
pluginStatus, err := env.Statuses()
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
require.NoError(t, err)
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
sort.Sort(pluginStatusById(pluginStatus))
require.Len(t, pluginStatus, 2)
expectPluginStatus(t, "testplugin", "0.0.2", pluginStatus[0])
expectPluginStatus(t, "testplugin2", "1.2.3", pluginStatus[1])
expectPluginInFilestore(t, th, "testplugin", "0.0.2")
expectPluginNotInFilestore(t, th, "testplugin2")
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
})
t.Run("multiple plugins, not automatically installed despite enabled since automatic prepackaged plugins disabled", func(t *testing.T) {
th := setup(t)
env := th.App.GetPluginsEnvironment()
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.AutomaticPrepackagedPlugins = false
cfg.PluginSettings.PluginStates["testplugin"] = &model.PluginState{Enable: true}
cfg.PluginSettings.PluginStates["testplugin2"] = &model.PluginState{Enable: true}
})
copyAsPrepackagedPlugin(t, th, "testplugin.tar.gz")
copyAsPrepackagedPlugin(t, th, "testplugin2.tar.gz")
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
plugins, transitionalPlugins := initPlugins(t, th)
require.Len(t, plugins, 2, "expected two prepackaged plugins")
sort.Sort(byID(plugins))
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
expectPrepackagedPlugin(t, "testplugin", "0.0.1", plugins[0])
expectPrepackagedPlugin(t, "testplugin2", "1.2.3", plugins[1])
require.Empty(t, transitionalPlugins)
pluginStatus, err := env.Statuses()
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
require.NoError(t, err)
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
require.Len(t, pluginStatus, 0)
expectPluginNotInFilestore(t, th, "testplugin")
expectPluginNotInFilestore(t, th, "testplugin2")
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
})
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
t.Run("removing a prepackaged plugin leaves it disabled", func(t *testing.T) {
th := setup(t)
env := th.App.GetPluginsEnvironment()
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.AutomaticPrepackagedPlugins = true
cfg.PluginSettings.PluginStates["testplugin"] = &model.PluginState{Enable: true}
cfg.PluginSettings.PluginStates["testplugin2"] = &model.PluginState{Enable: true}
})
copyAsPrepackagedPlugin(t, th, "testplugin.tar.gz")
copyAsPrepackagedPlugin(t, th, "testplugin2.tar.gz")
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
plugins, transitionalPlugins := initPlugins(t, th)
require.Len(t, plugins, 2, "expected two prepackaged plugins")
sort.Sort(byID(plugins))
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
expectPrepackagedPlugin(t, "testplugin", "0.0.1", plugins[0])
expectPrepackagedPlugin(t, "testplugin2", "1.2.3", plugins[1])
require.Empty(t, transitionalPlugins)
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
pluginStatus, err := env.Statuses()
require.NoError(t, err)
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
sort.Sort(pluginStatusById(pluginStatus))
require.Len(t, pluginStatus, 2)
require.Equal(t, "testplugin", pluginStatus[0].PluginId)
require.Equal(t, "testplugin2", pluginStatus[1].PluginId)
appErr := th.App.ch.RemovePlugin("testplugin")
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
checkNoError(t, appErr)
pluginStatus, err = env.Statuses()
require.NoError(t, err)
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
require.Len(t, pluginStatus, 1)
require.Equal(t, "testplugin2", pluginStatus[0].PluginId)
plugins, transitionalPlugins = initPlugins(t, th)
require.Len(t, plugins, 2, "expected two prepackaged plugins")
sort.Sort(byID(plugins))
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
expectPrepackagedPlugin(t, "testplugin", "0.0.1", plugins[0])
expectPrepackagedPlugin(t, "testplugin2", "1.2.3", plugins[1])
require.Empty(t, transitionalPlugins)
pluginStatus, err = env.Statuses()
require.NoError(t, err)
require.Len(t, pluginStatus, 1)
expectPluginStatus(t, "testplugin2", "1.2.3", pluginStatus[0])
expectPluginNotInFilestore(t, th, "testplugin")
expectPluginNotInFilestore(t, th, "testplugin2")
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
})
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
t.Run("single transitional plugin automatically installed and persisted since enabled", func(t *testing.T) {
th := setup(t)
env := th.App.GetPluginsEnvironment()
oldTransitionallyPrepackagedPlugins := append([]string{}, transitionallyPrepackagedPlugins...)
transitionallyPrepackagedPlugins = append(transitionallyPrepackagedPlugins, "testplugin")
defer func() {
transitionallyPrepackagedPlugins = oldTransitionallyPrepackagedPlugins
}()
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
th.App.UpdateConfig(func(cfg *model.Config) {
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
*cfg.PluginSettings.AutomaticPrepackagedPlugins = true
cfg.PluginSettings.PluginStates["testplugin"] = &model.PluginState{Enable: true}
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
})
copyAsPrepackagedPlugin(t, th, "testplugin.tar.gz")
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
plugins, transitionalPlugins := initPlugins(t, th)
require.Empty(t, plugins)
require.Len(t, transitionalPlugins, 1)
expectPrepackagedPlugin(t, "testplugin", "0.0.1", transitionalPlugins[0])
pluginStatus, err := env.Statuses()
require.NoError(t, err)
require.Len(t, pluginStatus, 1)
expectPluginStatus(t, "testplugin", "0.0.1", pluginStatus[0])
expectPluginInFilestore(t, th, "testplugin", "0.0.1")
expectPluginNotInFilestore(t, th, "testplugin2")
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
})
t.Run("single transitional plugin not persisted since already in filestore", func(t *testing.T) {
th := setup(t)
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
env := th.App.GetPluginsEnvironment()
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
oldTransitionallyPrepackagedPlugins := append([]string{}, transitionallyPrepackagedPlugins...)
transitionallyPrepackagedPlugins = append(transitionallyPrepackagedPlugins, "testplugin")
defer func() {
transitionallyPrepackagedPlugins = oldTransitionallyPrepackagedPlugins
}()
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.AutomaticPrepackagedPlugins = true
cfg.PluginSettings.PluginStates["testplugin"] = &model.PluginState{Enable: true}
})
copyAsFilestorePlugin(t, th, "testplugin.tar.gz", "testplugin")
copyAsPrepackagedPlugin(t, th, "testplugin.tar.gz")
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
plugins, transitionalPlugins := initPlugins(t, th)
require.Empty(t, plugins)
require.Empty(t, transitionalPlugins)
pluginStatus, err := env.Statuses()
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
require.NoError(t, err)
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
require.Len(t, pluginStatus, 1)
expectPluginStatus(t, "testplugin", "0.0.1", pluginStatus[0])
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
expectPluginInFilestore(t, th, "testplugin", "0.0.1")
expectPluginNotInFilestore(t, th, "testplugin2")
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
})
t.Run("single transitional plugin persisted since newer than filestore", func(t *testing.T) {
th := setup(t)
env := th.App.GetPluginsEnvironment()
oldTransitionallyPrepackagedPlugins := append([]string{}, transitionallyPrepackagedPlugins...)
transitionallyPrepackagedPlugins = append(transitionallyPrepackagedPlugins, "testplugin")
defer func() {
transitionallyPrepackagedPlugins = oldTransitionallyPrepackagedPlugins
}()
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.AutomaticPrepackagedPlugins = true
cfg.PluginSettings.PluginStates["testplugin"] = &model.PluginState{Enable: true}
})
copyAsFilestorePlugin(t, th, "testplugin.tar.gz", "testplugin")
copyAsPrepackagedPlugin(t, th, "testplugin-v0.0.2.tar.gz")
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
plugins, transitionalPlugins := initPlugins(t, th)
require.Empty(t, plugins)
require.Len(t, transitionalPlugins, 1)
expectPrepackagedPlugin(t, "testplugin", "0.0.2", transitionalPlugins[0])
pluginStatus, err := env.Statuses()
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
require.NoError(t, err)
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
require.Len(t, pluginStatus, 1)
expectPluginStatus(t, "testplugin", "0.0.2", pluginStatus[0])
expectPluginInFilestore(t, th, "testplugin", "0.0.2")
expectPluginNotInFilestore(t, th, "testplugin2")
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
})
t.Run("transitional plugins persisted only once", func(t *testing.T) {
th := setup(t)
env := th.App.GetPluginsEnvironment()
oldTransitionallyPrepackagedPlugins := append([]string{}, transitionallyPrepackagedPlugins...)
transitionallyPrepackagedPlugins = append(transitionallyPrepackagedPlugins, "testplugin")
defer func() {
transitionallyPrepackagedPlugins = oldTransitionallyPrepackagedPlugins
}()
th.App.UpdateConfig(func(cfg *model.Config) {
*cfg.PluginSettings.AutomaticPrepackagedPlugins = true
cfg.PluginSettings.PluginStates["testplugin"] = &model.PluginState{Enable: true}
})
copyAsPrepackagedPlugin(t, th, "testplugin.tar.gz")
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
plugins, transitionalPlugins := initPlugins(t, th)
require.Empty(t, plugins)
require.Len(t, transitionalPlugins, 1)
expectPrepackagedPlugin(t, "testplugin", "0.0.1", transitionalPlugins[0])
appErr := th.App.ch.RemovePlugin("testplugin")
require.Nil(t, appErr)
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
pluginStatus, err := env.Statuses()
require.NoError(t, err)
MM-53355: install transitionally prepackaged plugins to filestore (#24225) * move plugin signature verification to caller The semantics for when plugin signature validation is required are unique to the caller, so move this logic there instead of masking it, thus simplifying some of the downstream code. * support transitionally prepacked plugins Transitionally prepackaged plugins are prepackaged plugins slated for unpackaging in some future release. Like prepackaged plugins, they automatically install or upgrade if the server is configured to enable that plugin, but unlike prepackaged plugins they don't add to the marketplace to allow for offline installs. In fact, if unlisted from the marketplace and not already enabled via `config.json`, a transitionally prepackaged plugin is essentially hidden. To ensure a smooth transition in the future release when this plugin is no longer prepackaged at all, transitionally prepackaged plugins are persisted to the filestore as if they had been installed by the enduser. On the next restart, even while the plugin is still transitionally prepackaged, the version in the filestore will take priority. It remains possible for a transitionally prepackaged plugin to upgrade (and once again persist) if we ship a newer version before dropping it altogether. Some complexity arises in a multi-server cluster, primarily because we don't want to deal with multiple servers writing the same object to the filestore. This is probably fine for S3, but has undefined semantics for regular filesystems, especially with some customers backing their files on any number of different fileshare technologies. To simplify the complexity, only the cluster leader persists transitionally prepackaged plugins. Unfortunately, this too is complicated, since on upgrade to the first version with the transitionally prepackaged plugin, there is no guarantee that server will be the leader. In fact, as all nodes restart, there is no guarantee that any newly started server will start as the leader. So the persistence has to happen in a job-like fashion. The migration system might work, except we want the ability to run this repeatedly as we add to (or update) these transitionally prepackaged plugins. We also want to minimize the overhead required from the server to juggle any of this. As a consequence, the persistence of transitionally prepackaged plugins occurs on every cluster leader change. Each server will try at most once to persist its collection of transitionally prepackaged plugins, and newly started servers will see the plugins in the filestore and skip this step altogether. The current set of transitionally prepackaged plugins include the following, but this is expected to change: * focalboard * complete list of transitionally prepackaged plugins * update plugin_install.go docs * updated test plugins * unit test transitionally prepackged plugins * try restoring original working directory * Apply suggestions from code review Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com> * clarify processPrepackagedPlugins comment --------- Co-authored-by: Michael Kochell <6913320+mickmister@users.noreply.github.com>
2023-08-17 11:46:57 -04:00
require.Empty(t, pluginStatus, 0)
th.App.ch.persistTransitionallyPrepackagedPlugins()
expectPluginNotInFilestore(t, th, "testplugin")
expectPluginNotInFilestore(t, th, "testplugin2")
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
})
}
func TestGetPluginStateOverride(t *testing.T) {
mainHelper.Parallel(t)
th := Setup(t)
t.Run("no override", func(t *testing.T) {
overrides, value := th.App.ch.getPluginStateOverride("focalboard")
require.False(t, overrides)
require.False(t, value)
})
t.Run("apps override", func(t *testing.T) {
t.Run("without enabled flag", func(t *testing.T) {
overrides, value := th.App.ch.getPluginStateOverride("com.mattermost.apps")
require.True(t, overrides)
require.False(t, value)
})
t.Run("with enabled flag set to true", func(t *testing.T) {
mainHelper.Parallel(t)
th2 := SetupConfig(t, func(cfg *model.Config) {
cfg.FeatureFlags.AppsEnabled = true
})
overrides, value := th2.App.ch.getPluginStateOverride("com.mattermost.apps")
require.False(t, overrides)
require.False(t, value)
})
t.Run("with enabled flag set to false", func(t *testing.T) {
mainHelper.Parallel(t)
th2 := SetupConfig(t, func(cfg *model.Config) {
cfg.FeatureFlags.AppsEnabled = false
})
overrides, value := th2.App.ch.getPluginStateOverride("com.mattermost.apps")
require.True(t, overrides)
require.False(t, value)
})
})
}