Remove metric metadata persistence (keeping only OTel resource/scope
features) and refactor the native metadata subsystem from hardcoded
resource/scope handling to a generic kind framework.
The kind framework uses Go generics for type-safe hot paths
(Versioned[V], MemStore[V], KindOps[V]) and a KindDescriptor interface
for runtime dispatch at serialization boundaries (WAL, Parquet, head
commit/replay).
Key changes:
- Remove metric metadata persistence feature and related API endpoints
- Add Versioned[V] generic container and MemStore[V] generic store
- Add KindDescriptor interface and global kind registry
- Implement resourceKindDescriptor and scopeKindDescriptor
- Replace memSeries resource/scope fields with kindMeta []kindMetaEntry
- Use kind-aware iteration in compaction and DB.SeriesMetadata()
- Update WAL replay to use CommitToSeries via kind registry
- Wire WAL encode/decode functions via init() to break import cycle
- Fix silent data corruption when RefResolver fails on read or write
- Provide type aliases for backward compatibility
Adding a new metadata kind now requires implementing KindOps +
KindDescriptor and registering in init(), instead of modifying 12+ files.
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Add support for persisting OTel resource/scope attributes alongside time series
data in TSDB, enabling the info() PromQL function to enrich metrics with
resource context without relying on target_info metric scraping.
Storage layer:
- Add ResourceUpdater interface with UpdateResource() to storage.Appender
- Add ResourceContext to AppendV2Options for passing resource attributes
- Add ResourceQuerier interface for querying stored resource attributes
- Add EntityData type for passing OTel entity information through storage
TSDB seriesmetadata subsystem:
- Add Entity type with OTel entity data model (type, ID, description)
- Add ResourceVersion with versioned identifying/descriptive attributes
- Add VersionedResource for tracking attribute changes over time
- Add MemResourceStore for in-memory resource storage with versioning
- Extend Parquet schema with resource namespace, nested attribute lists,
and entity rows
- Add zstd compression, row sorting, and footer metadata to Parquet writer
OTLP integration:
- Extend CombinedAppender to extract and persist resource attributes from
OTLP ResourceMetrics during ingestion
PromQL info() function:
- Rewrite to use stored resource attributes instead of fetching target_info
series, with attribute name translation via LabelNamer mappings
Web API:
- Add /api/v1/resources endpoint for querying stored resource attributes
- Support format=attributes for autocomplete use cases
- Add OpenAPI documentation for the new endpoint
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
* Add OpenAPI 3.2 specification generation for Prometheus HTTP API
This commit introduces an OpenAPI specification for the Prometheus API.
After testing multiple code-generation servers with built-in APIs, this
implementation uses an independent spec file outside of the critical path.
This spec file is tested with a framework present in this pull request.
The specification helps clients know which parameters they can use and is
served at /api/v1/openapi.yaml. The spec file will evolve with the
Prometheus API and has the same version number.
Downstream projects can tune the APIs presented in the spec file with
configuration options using the IncludePaths setting for path filtering.
In the future, there is room to generate a server from this spec file
(e.g. with interfaces), but this is out of scope for this pull request.
Architecture:
- Core OpenAPI infrastructure (openapi.go): Dynamic spec building,
caching, and thread-safe spec generation
- Schema definitions (openapi_schemas.go): Complete type definitions
for all API request and response types
- Path specifications (openapi_paths.go): Endpoint definitions with
parameters, request bodies, and response schemas
- Examples (openapi_examples.go): Realistic request/response examples
- Helper functions (openapi_helpers.go): Reusable builders for common
OpenAPI structures
Testing:
- Comprehensive test suite with golden file validation
- Test helpers package for API testing infrastructure
- OpenAPI compliance validation utilities
The golden file captures the complete specification for snapshot testing.
Update with: go test -run TestOpenAPIGolden -update-openapi-spec
REVIEWERS: The most important thing to check would be the OpenAPI golden
file (web/api/v1/testdata/openapi_golden.yaml). Test scenarios are important
as they test the actual OpenAPI spec validity.
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
* Add OpenAPI 3.1 support with version selection
Add support for both OpenAPI 3.1 and 3.2 specifications with version
selection via openapi_version query parameter. Defaults to 3.1 for
broader compatibility
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
* Enhance OpenAPI examples and add helper functions
- Add timestampExamples helper for consistent time formatting
- Add exampleMap helper to simplify example creation
- Improve example summaries with query details
- Add matrix result example for range vector queries
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
* web/api: Add AtST method to test helper iterators
Implement the AtST() method required by chunkenc.Iterator interface
for FakeSeriesIterator and FakeHistogramSeriesIterator test helpers.
The method returns 0 as these test helpers don't use start timestamps
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
* OpenAPI: Add minimum coverage test
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
* OpenAPI: Improve examples handling
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
---------
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
