Add AI development guidance for Mattermost contributors working on the
Python plugin gRPC infrastructure. Includes directory structure, key
components, proto organization, common patterns, and instructions for
adding new API methods and hooks.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add ARCHITECTURE.md documenting the Python plugin gRPC system including:
- High-level overview and ASCII system architecture diagram
- Component layer documentation (Protocol, Go Infrastructure, Python SDK)
- Process lifecycle (loading, environment variables, shutdown)
- Communication flow diagrams (hooks, API calls, ServeHTTP streaming)
- Key design decisions (embedded AppError, 64KB chunks, APIServerRegistrar)
- Complete file reference table mapping functionality to source files
- Extension guide for adding new API methods and hooks
This documentation enables internal Mattermost engineers to understand,
debug, and extend the Python plugin infrastructure.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add python-proto-gen and proto-gen-all targets to server/public/Makefile
for complete gRPC code generation workflow:
- python-proto-gen: Generates Python gRPC code by calling generate_protos.py
- proto-gen-all: Runs both Go and Python proto generation in sequence
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add TestPythonAPIServerStartup: verify server starts and registrar called
- Add TestPythonAPIServerLifecycle: verify cleanup stops server properly
- Add TestPythonCommandWithAPIServer: verify env var not set when apiImpl is nil
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add apiServerCleanup call to supervisor Shutdown method
- Cleanup happens AFTER Python process terminates for graceful disconnect
- Fix test assertion to check for relative executable path
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add startAPIServer function to start gRPC server on random port
- Add APIServerRegistrar type to break import cycle with apiserver package
- Modify WithCommandFromManifest to accept apiImpl and registrar
- Update configurePythonCommand to start API server and set MATTERMOST_PLUGIN_API_TARGET env var
- Add apiServerCleanup field to supervisor struct for cleanup on shutdown
- Add SetAPIServerRegistrar method to Environment for dependency injection
- Wire up API server registrar in app/plugin.go
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Go side:
- Log hooks returned by Implemented()
- Log each hook name -> ID mapping
- Log OnActivate implementation status
- Log OnActivate call flow
Python side:
- Log Implemented() return value
- Log OnActivate gRPC receipt and handler invocation
This is temporary debug logging to diagnose why OnActivate
isn't being called for Python plugins.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
OnActivate, ServeHTTP, MessageWillBePosted, MessageWillBeUpdated, and
ServeMetrics are handled specially and not in client_rpc_generated.go,
but they still need to be in the hookNameToId map for the Python plugin
Implemented() mechanism to work.
Without this, Python plugins could report implementing OnActivate but
the Go side wouldn't recognize the hook name and wouldn't call it.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When pluginInfo.Path is relative (e.g., "plugins/com.mattermost.hello-python"),
the venv interpreter path was also relative. exec.Command needs an absolute
path to find the executable when cmd.Dir is set.
Fix: Convert the venv path to absolute using filepath.Abs before returning.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When cmd.Dir is set, command arguments are resolved relative to that
directory. The code was passing the full path (including plugin dir)
as the script argument while also setting cmd.Dir to the plugin dir,
causing Python to look for the script at a duplicated path like:
plugins/com.mattermost.hello-python/plugins/com.mattermost.hello-python/plugin.py
Fix: Pass just the executable name (e.g., "plugin.py") to the command
since cmd.Dir is already set to the plugin directory. The full path
is still used for the existence check before command creation.
Also removes debug logging that was added for diagnosis.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add integration tests demonstrating complete Python plugin lifecycle:
- TestPythonPluginIntegration: Tests startup, hooks, ServeHTTP, shutdown
- TestPythonPluginServeHTTP: Tests HTTP streaming with large responses
- TestPythonPluginEnvironmentIntegration: Tests Environment activation/deactivation
- TestPythonPluginCrashRecovery: Tests crash detection and restart
- TestPythonPluginImplementsChecking: Tests hook implementation tracking
All tests use fake Python interpreters (compiled Go binaries) that implement
the PluginHooks gRPC service to validate the Go-side integration without
requiring actual Python.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add TestPythonPluginHookDispatch that verifies end-to-end hook dispatch
for Python plugins. The test:
- Creates a fake Python plugin with PluginHooks gRPC service
- Verifies hooks are properly wired through the supervisor
- Tests Implemented(), OnActivate, OnDeactivate, and MessageHasBeenPosted
Also update existing Phase 5 tests to implement PluginHooks service:
- TestPythonSupervisor_HealthCheckSuccess
- TestPythonPluginEnvironmentActivation
- TestPythonSupervisor_Restart
These tests now reflect the new architecture where Python plugins
have hooks wired via hooksGRPCClient.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Remove Phase 5 limitation that skipped hook dispensing for Python plugins.
Now Python plugins use hooksGRPCClient to receive hook invocations through
the same infrastructure as Go plugins.
Changes:
- Extract gRPC connection from go-plugin's GRPCClient
- Create hooksGRPCClient adapter using the connection
- Wrap in hooksTimerLayer for metrics collection
- Populate implemented hooks array from gRPC client
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implement the hooksGRPCClient adapter that implements the plugin.Hooks
interface by delegating to a gRPC PluginHooksClient. This enables Python
plugins to receive hook invocations through the same infrastructure as
Go plugins.
Key features:
- Task 1: Core adapter structure with constructor and lifecycle hooks
(OnActivate, OnDeactivate, OnConfigurationChange, Implemented)
- Task 2: ServeHTTP with bidirectional streaming (64KB chunks)
- Task 3: All remaining hook methods including message, user, channel,
team, command, WebSocket, and miscellaneous hooks
Technical details:
- Uses context.Background() with 30s timeout for gRPC calls
- Checks implemented array before making gRPC calls
- Converts between model types and protobuf types
- ServeHTTP uses bidirectional streaming for request/response body
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Create shell scripts for consistent test invocation across CI environments:
Python script (python-sdk/scripts/run_integration_tests.sh):
- Auto-detects virtual environment or uses system Python
- Installs package in editable mode if not present
- Runs test_integration_e2e.py with verbose output
- Supports passing extra pytest arguments
Go script (server/public/pluginapi/grpc/scripts/run_integration_tests.sh):
- Runs from the correct module directory
- Filters to TestPythonPlugin and TestIntegration tests
- Disables test caching for clean CI runs
- Supports passing extra go test arguments
Both scripts:
- Are executable (chmod +x)
- Print diagnostic information (working dir, version)
- Exit with appropriate error codes
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Create integration tests that prove Go gRPC server and Python plugin
communicate correctly via simulated in-memory gRPC communication:
- TestPythonPluginLifecycle: Full lifecycle (activate, deactivate)
- TestPythonPluginAPICall: Round-trip plugin -> API server callback
- TestPythonPluginHookChain: MessageWillBePosted modification/rejection
- TestPythonPluginActivationFailure: Error propagation from plugin
- TestPythonPluginAPIErrorPropagation: AppError -> gRPC status mapping
- TestIntegrationConcurrentHookCalls: Concurrent hook invocations
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Replace props.runtime test cases with Server.Runtime field tests:
- python runtime from manifest field (no .py extension)
- go runtime explicit (should not be Python)
- python with full config (PythonVersion, ManifestPython)
- Keep .py extension fallback test for backward compatibility
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Replace transitional props.runtime hack with proper manifest field detection.
Now prioritizes Server.Runtime="python" over .py extension fallback.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add TestManifestUnmarshalPython function with test cases for:
- YAML Python plugin manifest parsing with all fields
- JSON Python plugin manifest parsing with all fields
- Backward compatibility for Go plugins without runtime field
- Python plugin with minimal config (just runtime, no python section)
Verifies Runtime, PythonVersion, and Python struct fields parse correctly
in both JSON and YAML formats.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Update ServeHTTPCaller to support true streaming responses:
- Full-duplex operation: plugin can respond before request body is consumed
- Flush support: best-effort flushing when underlying writer supports http.Flusher
- Status code validation (100-999 range) to prevent server panics
- Early response handling with proper goroutine cleanup
- Proper error responses for invalid status codes and plugin failures
Key changes:
- Add logger to ServeHTTPCaller for error logging
- Add context cancellation to sendRequest for early response termination
- Add receiveFirstResponse to handle first message with body chunk + flush
- Add writeResponseHeaders with status code validation
- Add flush() method for best-effort HTTP flushing
- Update streamResponseBody to handle flush flag
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add flush field to ServeHTTPResponse for best-effort HTTP flush support.
Update ServeHTTPResponse and ServeHTTPResponseInit with comprehensive
documentation of streaming invariants, message ordering, and status
code validation rules.
Changes:
- Add flush boolean field (field 4) to ServeHTTPResponse
- Document response streaming invariants (init-once, header locking)
- Document status code validation (100-999 range, 0 defaults to 200)
- Regenerate Go and Python protobuf code
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add tests for both Go and Python ServeHTTP implementations covering:
- Body chunking behavior (small, exact, multiple, large bodies)
- Context cancellation during body streaming
- Header conversion utilities
- HTTPRequest/HTTPResponseWriter helper classes
- Request body assembly from multiple chunks
- Error handling (500 for handler errors, 404 for missing handler)
Go tests:
- TestChunking_* for body chunking scenarios
- TestConvertHTTPHeaders for header conversion
- TestBuildRequestInit for request metadata building
- TestWriteResponseHeaders for response header writing
- TestContextCancellation_DuringBodyRead for cancellation
Python tests:
- TestHTTPRequest for request wrapper class
- TestHTTPResponseWriter for response writer class
- TestHeaderConversion for proto<->dict conversion
- TestServeHTTPServicer for gRPC servicer behavior
- TestChunkingBehavior for body assembly
- TestCancellation for request cancellation
Also:
- Fixed sendRequest to properly set body_complete flag on EOF
- Added ServeHTTP to HookName enum
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add ServeHTTPCaller that streams HTTP requests to Python plugins
and receives streaming responses.
Key features:
- Bidirectional streaming for efficient large body transfer
- 64KB chunked request body streaming (no full buffering)
- Context cancellation propagation from HTTP client disconnect
- HTTPHeader conversion supporting multi-value headers
- PluginContext passed through to Python handler
Implementation details:
- sendRequest: streams body chunks with completion flag
- receiveResponseInit: gets status code and headers
- streamResponseBody: writes response chunks to ResponseWriter
- buildRequestInit: converts http.Request to protobuf init
Note: Response streaming improvements deferred to 08-02.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add bidirectional streaming RPC for ServeHTTP hook to support
efficient HTTP request/response transfer between Go and Python.
Key changes:
- New hooks_http.proto with ServeHTTPRequest/Response messages
- HTTPHeader message for multi-value header support
- ServeHTTPRequestInit with full request metadata
- ServeHTTPResponseInit for status and headers
- Body chunks with completion flag for streaming
- Updated hooks.proto with ServeHTTP streaming RPC
- Regenerated Go and Python code
Design decisions:
- 64KB default chunk size per gRPC best practices
- First message carries metadata, subsequent messages carry body
- body_complete flag signals end of stream
- Headers as repeated HTTPHeader for multi-value support
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add end-to-end test for restart behavior:
1. Activate healthy fake Python plugin
2. Verify health check succeeds
3. Kill process via supervisor.client.Kill() to simulate crash
4. Verify health check fails after crash
5. Call RestartPlugin and verify health check succeeds again
Uses polling loops instead of arbitrary sleeps for reliability.
Proves Environment.RestartPlugin can recover crashed Python plugins.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add tests for:
- Handshake timeout: fake interpreter blocks forever without printing
handshake, verifies supervisor times out within StartTimeout
- Invalid handshake: fake interpreter prints netrpc protocol instead of
grpc, verifies protocol mismatch error
- Malformed handshake: fake interpreter prints incomplete handshake line,
verifies parsing error
These tests use compiled Go binaries as fake "Python interpreters" to
validate supervisor error handling without requiring real Python.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- TestPythonSupervisor_HealthCheckSuccess: validates end-to-end spawning
of Python-style plugin with gRPC health check using fake interpreter
- TestPythonPluginEnvironmentActivation: validates Environment can
activate/deactivate Python plugins without panicking on nil hooks
- Update Environment.Activate to use WithCommandFromManifest which
handles both Go (netrpc) and Python (gRPC) plugins
The fake interpreter is a compiled Go binary that:
1. Starts a gRPC server on random port
2. Registers health service with "plugin" status SERVING
3. Prints go-plugin handshake line
4. Blocks until killed
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- startPluginServer: skip OnActivate when Hooks() is nil
- Deactivate: guard OnDeactivate call against nil hooks
- Shutdown: guard OnDeactivate call against nil hooks
- Add informative log when Python plugin activates without hooks
This allows Phase 5 process supervision to work while hook dispatch
is deferred to Phase 7.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Detect Python plugins in newSupervisor via isPythonPlugin helper
- Configure AllowedProtocols to include ProtocolGRPC for Python plugins
- Increase StartTimeout to 10s for Python interpreter startup
- Skip Dispense("hooks") for Python plugins (Phase 5 supervision only)
- Leave sup.hooks nil - hook dispatch deferred to Phase 7
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add helper functions for Python plugin support in the supervisor:
- isPythonPlugin: detects Python plugins via .py extension or props.runtime
- findPythonInterpreter: discovers venv-first Python with PATH fallback
- sanitizePythonScriptPath: validates script paths preventing traversal
- buildPythonCommand: creates exec.Cmd with proper working dir and WaitDelay
- WithCommandFromManifest: unified option for Go and Python plugin commands
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implement all File-related gRPC RPC handlers:
- GetFileInfo, GetFileInfos, GetFile
- GetFileLink, ReadFile, UploadFile
- CopyFileInfos, SetFileSearchableContent
Add comprehensive unit tests for:
- GetFileInfo success and not found cases
- GetFile success and error cases
- ReadFile success and not found cases
- UploadFile success and permission error
- CopyFileInfos and SetFileSearchableContent
- GetFileInfos with options
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add conversion functions between model.Team/TeamMember/TeamUnread/TeamStats
and their protobuf equivalents. Includes:
- teamToProto/teamFromProto for Team conversion
- teamMemberToProto/teamMemberFromProto for TeamMember conversion
- teamUnreadToProto for TeamUnread conversion
- teamStatsToProto for TeamStats conversion
- teamMembersWithErrorToProto for graceful member creation
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add pb <-> model conversion functions for Post, File, and KV Store types:
convert_post.go:
- Post, PostMetadata, PostEmbed, Reaction conversions
- PostList and PostSearchResults conversions
- SearchParams and SearchParameter conversions
convert_file.go:
- FileInfo conversions
- GetFileInfosOptions conversion
convert_kv.go:
- PluginKVSetOptions conversion
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Enhance package documentation to explain how subsequent plans (04-02
through 04-04) should extend the APIServer with new RPC implementations.
Includes code example and key patterns for error handling and model
conversion.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add in-memory gRPC testing infrastructure using bufconn. Includes:
- testHarness struct for easy test setup/teardown
- Smoke tests for GetServerVersion and IsEnterpriseReady RPCs
- Comprehensive error conversion tests for all HTTP->gRPC mappings
- Tests use plugintest.API mock from the existing codebase
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Create the APIServer struct that wraps plugin.API and embeds
UnimplementedPluginAPIServer for incremental implementation.
Includes Register helper and smoke RPCs (GetServerVersion,
IsEnterpriseReady) to validate the wiring.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add hooks_command.proto to Makefile proto mappings
- Update hooks_command.proto to import api_remaining.proto for shared types
(CommandArgs, CommandResponse, PluginClusterEvent)
- Define new types: WebSocketRequest, SyncMsgJson, SyncResponse, RemoteCluster
- Generated Go code compiles successfully
- Total hooks in PluginHooks service: 41 (excluding deferred ServeHTTP/ServeMetrics)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
