Remove the per-series nativeMeta struct (kindMetaEntry slice, stableHash
cache, GetKindMeta/SetKindMeta methods) from memSeries, saving ~360 bytes
per OTLP series (~10GB for 30M series).
Resource/scope metadata is now committed directly to the shared MemStore
via new CommitResourceToStore/CommitScopeToStore functions, which use
SetVersionedWithDiff to atomically check-and-commit. The filter and commit
steps are merged into commitAndFilterResources/commitAndFilterScopes,
which write to MemStore before WAL logging and skip WAL writes when content
is unchanged (detected by comparing version counts before/after commit).
WAL replay also commits directly to MemStore instead of going through
per-series storage.
Committing to MemStore before WAL is safe because MemStore is in-memory
only and rebuilt from WAL on crash recovery.
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Add GetIndexedResourceAttrs() on MemSeriesMetadata and Head for reading
the current set of indexed descriptive resource attributes. Document
replace-not-mutate semantics on SetIndexedResourceAttrs.
Add HashFilter to WriterOptions so callers can skip entries by
labelsHash before version iteration and content hash computation. Wire
it into WriteFileWithOptions and buildResourceAttrIndexRows. In
compaction, set HashFilter from the labelsHashToRef map for early
rejection.
Replace the global seen map in buildResourceAttrIndexRows with per-series
dedup (keyed by contentHash only since seriesRef is constant within a
callback), reducing memory from total-attrs×series to max-attrs-per-series.
Add streaming Parquet reader functions (OpenParquetFile,
StreamVersionedResourcesFromFile, StreamResourceAttrIndexFromFile) for
row-group-at-a-time reads without materializing the entire file into
memory. Functional options support series ref filtering, ref resolution,
time range filtering, and emit callbacks.
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Add EnableResourceAttrIndex option to control whether the resource
attribute inverted index is built in memory and written to Parquet.
When disabled, the index is not created but unique attribute name
tracking (used for autocomplete) still works.
This allows downstream projects like Mimir to disable the index
per-tenant when Parquet-native filtering on attr_key/attr_value
columns is preferred over the in-memory inverted index.
Prometheus defaults to enabled, preserving current behavior.
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Add content-addressed deduplication to MemStore so that versions with
identical content share map/slice pointers from a single canonical entry.
At Mimir scale (30M series sharing ~1K unique OTel resources), this
reduces per-version memory from ~1500B to ~72B, saving ~85GB.
The new ContentDedupOps[V] interface is an optional extension of KindOps
detected via type assertion. Kinds that implement ContentHash and ThinCopy
get automatic dedup; kinds that don't are unaffected.
MemStore maintains a 256-way sharded content table with double-checked
locking. Set, SetVersioned, and SetVersionedWithDiff intern versions after
deep copies. InternVersion is exposed for per-series interning from the
head commit and WAL replay paths.
Also skip WAL writes for unchanged resource/scope content via
filterUnchangedResources/filterUnchangedScopes, avoiding redundant WAL
I/O when only time ranges change.
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
When native metadata is disabled (the default), every memSeries paid
32 bytes for the kindMeta slice header (24B) and stableHash (8B) even
though both fields were unused. At 100M series this wastes 3.2 GB.
Introduce a nativeMeta struct holding both fields, stored as a single
*nativeMeta pointer on memSeries (8 bytes). Lazy-allocated on first
SetKindMeta call. Saves 24 bytes per series when native metadata is
not in use.
Also adds getStableHash() helper to DRY up the three identical
lazy-compute patterns, and correctly moves stableHash from the
"immutable before Mutex" section to after the Mutex (it was always
computed under the series lock).
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
CommitResourceDirect and CommitScopeDirect previously copied each map
2-3 times: NewEntity/NewResourceVersion/NewScopeVersion deep-copied
from the caller's buffers, then copyResourceVersion/CopyScopeVersion
or AddOrExtend(ops.Copy) copied again when storing.
Construct versions directly with exactly one maps.Clone per map and
take ownership of the result. On first insert, store the owned version
directly. On existing series, inline the extend-or-append logic to
avoid the redundant copy through AddOrExtend.
CommitToSeries (cold path, WAL replay) now delegates to the Direct
functions, eliminating the duplicated logic.
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Replace monolithic allResources/allScopes slices with chunked flushing
using checkpointFlushChunkSize (10K records per chunk). Peak allocation
drops from O(total_records) — potentially gigabytes for 100M series —
to O(checkpointFlushChunkSize) ≈ 2 MB per chunk.
cp.Log() and enc.Resources()/enc.Scopes() are stateless and support
multiple calls, so chunked flushing preserves identical checkpoint
semantics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Add a fast-path in SetVersionedWithDiff that skips MergeVersioned when
the incoming versioned has exactly one version that equals the current
(latest) version. In this common case (~90% of commits in steady state),
just extend the time range in-place via UpdateTimeRange — zero
allocations vs ~12 from the MergeVersioned code path.
This mirrors the AddOrExtend pattern in versioned.go. The caller
(updateSharedResourceMetadata) uses old vs cur to compute index diffs;
since the fast-path only extends time ranges (not attribute content),
the index update is still correct — effectively a no-op.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Apply the same content-addressed dedup pattern used for resources to
scopes during WAL checkpoint. Replace allScopesMap (full RefScope per
series per version) with scopeContentTable + scopeRefToContent.
At 100M series sharing 50 unique scopes: memory drops from ~20 GB
(100M × ~200B full records) to ~2.4 GB (50 × 200B content + 100M × 24B
mappings).
Also rename resourceMapping → contentMapping since the type is now
shared between resources and scopes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Replace single resourceAttrIndexMu RWMutex with 256-way sharded
attrIndexStripe, eliminating the contention bottleneck when 256
ingester goroutines update resources concurrently.
Each "key\x00value" index key routes to a single stripe via xxhash.
Per-key operations (Lookup, sortedInsert, sortedRemove) lock only
that stripe. Global operations (Build, SetIndexedResourceAttrs)
iterate all stripes or use a dedicated mutex.
The indexedResourceAttrs config field gets its own indexedResourceAttrsMu
since config changes are rare and independent of per-key operations.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Three small improvements for Mimir integration:
1. Head.SetIndexedResourceAttrs() enables per-tenant runtime
reconfiguration of which descriptive resource attributes are
indexed. Thread-safe via resourceAttrIndexMu.
2. Replace EnableBloomFilters bool with BloomFilterFormat enum
(BloomFilterNone, BloomFilterParquetNative). Reserved slot for
future BloomFilterSidecar format for independent store-gateway
caching.
3. UniqueAttrNameReader interface (added in prior commit) is already
checked via type assertion in HeadAndOOOQuerier.IterUniqueAttributeNames
for O(1) cached name lookup.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Three related improvements for distributed-scale metadata:
1. Lazy block metadata resolution: Defer Parquet loading from OpenBlock
to Block.SeriesMetadata() via sync.Once, avoiding the cost of
resolving all seriesRef → labelsHash at block open time. File size
is captured via stat for correct Size() before first access.
2. BlockMeta extension: Add BlockSeriesMetadata to meta.json with
namespace row counts and indexed resource attribute names. Populated
during compaction via WriteStats. Enables Mimir store-gateway to
pre-plan queries without opening the Parquet file.
3. Streaming layered reader: Replace map[uint64]*VersionedResource with
map[uint64]struct{} hash-set in IterVersionedResources/Scopes, using
O(1) point lookups for the ~1% shared hashes during base iteration.
Saves ~40% memory per entry (16 bytes vs 56+ bytes).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Many series share identical resource content (same OTLP resource with
many metrics). Instead of storing full RefResource records per series
ref, checkpoint now stores unique content once in a content table
(contentHash → canonical record) and maps refs to content hashes with
time ranges. This dramatically reduces checkpoint memory when N series
share K unique resources (K << N): from O(N×500B) to O(K×500B + N×24B).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Accept a reusable []string buffer in encodeMap to avoid allocating
make([]string, 0, len(m)) per call. The buffer grows to max-key-count
and is reused across all map encodes in a single Resources() or
Scopes() call.
At 10K series/batch with ~9 maps per resource (identifying, descriptive,
+ entity maps), this eliminates ~90K allocations per batch.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Add CommitResourceDirect/CommitScopeDirect and CollectResourceDirect/
CollectScopeDirect functions that operate on concrete types instead of
going through the KindDescriptor interface. This avoids boxing
ResourceCommitData/ScopeCommitData structs into interface{} on every
commit call (one heap allocation per series per batch).
The generic KindDescriptor.CommitToSeries is retained for cold paths
(WAL replay, compaction). headAppenderBase.commitResources/commitScopes
now call the direct functions, and Head gets type-safe
updateSharedResourceMetadata/updateSharedScopeMetadata methods.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Add a grow-only UniqueResourceAttrNames cache to MemSeriesMetadata that
tracks all resource attribute names seen. This avoids O(N_series) full
scans in IterUniqueAttributeNames, reducing it to O(1) via the
UniqueAttrNameReader interface (checked by type assertion).
The cache is populated during BuildResourceAttrIndex (full build) and
UpdateResourceAttrIndex (incremental updates). Both HeadAndOOOQuerier
and HeadAndOOOChunkQuerier use the fast path when available.
Also optimize enrichVectorWithResourceAttrs in promql/info.go: reuse
the allAttrs map via clear() instead of allocating per series per
timestamp.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Replace map[uint64]struct{} inner maps in the resource attribute
inverted index with sorted []uint64 slices using copy-on-write
semantics. This reduces memory ~4x (8 bytes/entry vs ~50 bytes/entry
for hash map overhead) and eliminates snapshot copies on read.
sortedInsert/sortedRemove create new slices on mutation, so readers
holding old slices are safe from concurrent changes. LookupResourceAttr
returns the live slice directly (zero-copy).
The layered reader uses two-pointer sorted merge for union operations.
The web API uses two-pointer sorted intersection for filter intersection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Replace the single-mutex MemStore with 256-way sharding (matching the
stripeSeries pattern in tsdb/head.go). Each stripe has its own RWMutex
and map, reducing lock contention under high concurrency with many
ingester goroutines.
snapshotEntries() iterates stripes sequentially, holding each stripe's
RLock briefly before releasing it. Len/TotalEntries/TotalVersions
aggregate across all stripes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
The in-memory inverted index and Parquet resource_attr_index rows
previously indexed ALL resource attributes. At scale this produces
excessive index data. This change makes indexing selective:
- Identifying attributes (service.name, etc.) are always indexed
- Descriptive attributes are only indexed if explicitly configured
via IndexedResourceAttrs (e.g., k8s.namespace.name)
Additionally, dedicated top-level AttrKey/AttrValue columns are added
to resource_attr_index Parquet rows, enabling Parquet-native column
filtering (column stats, bloom filters) at the store-gateway layer
without loading the full index into memory.
Backward compatibility:
- Old readers ignore unknown AttrKey/AttrValue columns
- New readers fall back to IdentifyingAttrs[0] for old files
- Default behavior (nil IndexedResourceAttrs) indexes only
identifying attributes
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Fix critical data race where LookupResourceAttr returned a live inner
map reference after releasing RLock, causing concurrent mutation by
writers. Return a snapshot copy instead.
Reduce sequential mutex acquisitions in the resource commit path from 6
to 4 by introducing SetVersionedWithDiff on MemStore, which atomically
stores data and returns old/new state in a single lock acquisition.
Remove the redundant deep copy before SetVersioned (which creates its
own copy internally).
Replace lock-held iteration in MemStore.Iter/IterVersioned with a
snapshot-then-iterate pattern: take a shallow copy of entry pointers
under the lock, then iterate without holding it. Prevents writer
starvation on large stores.
Optimize layered reader iteration to avoid O(N_head) cross-reader point
lookups by collecting top entries into a map first, then merging inline
during base iteration.
Replace concat+sort in MergeVersioned with a two-pointer merge since
both inputs are already sorted by MinTime. Eliminates one allocation and
the O(n log n) sort.
Use numeric hash pairs instead of string-keyed structs for dedup in
buildResourceAttrIndexRows, reducing per-entry memory from ~100B to 16B.
Cache labels.StableHash on memSeries to avoid recomputation on every
metadata commit (labels are immutable after creation).
Optimize compaction ref resolver to only scan index entries that have
metadata, with early exit when all hashes are resolved.
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
- Incremental inverted index on head: add resourceAttrIndexMu, Update/Remove
methods, integrate with updateSharedMetadata/cleanupSharedMetadata
- Per-block inverted index in Parquet: NamespaceResourceAttrIndex namespace,
EnableInvertedIndex writer option, write/read index rows, enable in compaction
- Layered metadata reader: blocks-only cache (no TTL) + live head overlay,
replacing 30s head TTL with always-current head data
- Eliminate fallback label scan: remove O(ALL_SERIES) q.Select() fallback in
/resources and /resources/series endpoints
- Safety cap: maxMetadataResults=500k on resource metadata API responses
- Convenience reader options: WithResourceAttrIndexOnly, WithResourceData,
WithScopeData, WithFullResourceData for Mimir store-gateway
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The /resources/series endpoint previously did an O(N) full scan of all
series with resource metadata on every request. Add an inverted index
on MemSeriesMetadata that maps "key\x00value" → set of labelsHashes,
covering both Identifying and Descriptive attributes across all versions.
BuildResourceAttrIndex() is called once in buildSeriesMetadata() after
merging blocks + head. Since the merged reader is cached for 30s, the
index build cost is amortized across many API requests.
The handler now intersects candidate sets from the index per filter,
then verifies each candidate with GetVersionedResource + time range +
attribute checks. Falls back to the original full scan if the index
is nil (reader without index support).
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Remove scope (name, version, schema_url, attributes) and entity (type,
attributes) filter support from the /resources/series reverse-lookup
endpoint. This is an architectural choice for simplicity — the endpoint
now exclusively accepts resource.attr=key:value filters.
Matched series still include scope versions in the response as
supplementary data, but scopes and entities are no longer filterable
criteria.
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Address performance and correctness issues in /api/v1/resources and
/api/v1/resources/series identified during architectural review:
- Add LabelsForHash() on Reader interface to eliminate O(ALL_SERIES)
__name__=~".+" scan for hash-to-labels resolution. Labels are captured
at block open time and during head collection, providing O(1) lookup.
- Add context.Context to all metadata iteration methods (IterKind,
IterResources, IterVersionedResources, IterVersionedScopes) with
cancellation checks every 100 iterations, allowing cancelled HTTP
requests to short-circuit metadata scans.
- Cache DB.SeriesMetadata() results with two-level invalidation: block
set changes detected via sorted ULID fingerprint, head staleness via
30-second TTL.
- Make limit behavior deterministic: collect all results, sort by
labels.Compare, then truncate with a warning. Fix limit >= 0 bug
(changed to limit > 0) that caused zero results when no limit set.
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Add a new GET /api/v1/resources/series endpoint that performs reverse
lookup of time series by OTel metadata criteria. Given resource
attribute, scope, or entity filters, the endpoint returns all series
that have matching metadata along with their full version history.
Query parameters:
- resource.attr=key:value (repeatable, AND logic)
- scope.name, scope.version, scope.schema_url (exact match)
- scope.attr=key:value (repeatable, AND logic)
- entity.type, entity.attr=key:value (repeatable, AND logic)
- match[] (optional PromQL label matchers for pre-filtering)
- start, end (optional time range to filter versions)
- limit (optional max results)
This complements the existing /api/v1/resources endpoint which does
forward lookup (given series, return their attributes).
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Remove metric metadata persistence (keeping only OTel resource/scope
features) and refactor the native metadata subsystem from hardcoded
resource/scope handling to a generic kind framework.
The kind framework uses Go generics for type-safe hot paths
(Versioned[V], MemStore[V], KindOps[V]) and a KindDescriptor interface
for runtime dispatch at serialization boundaries (WAL, Parquet, head
commit/replay).
Key changes:
- Remove metric metadata persistence feature and related API endpoints
- Add Versioned[V] generic container and MemStore[V] generic store
- Add KindDescriptor interface and global kind registry
- Implement resourceKindDescriptor and scopeKindDescriptor
- Replace memSeries resource/scope fields with kindMeta []kindMetaEntry
- Use kind-aware iteration in compaction and DB.SeriesMetadata()
- Update WAL replay to use CommitToSeries via kind registry
- Wire WAL encode/decode functions via init() to break import cycle
- Fix silent data corruption when RefResolver fails on read or write
- Provide type aliases for backward compatibility
Adding a new metadata kind now requires implementing KindOps +
KindDescriptor and registering in init(), instead of modifying 12+ files.
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Introduce content-addressed normalization that eliminates cross-series
duplication of OTel resource and scope data. Instead of one full row
per series per resource version, use content-addressed table rows keyed
by xxhash plus lightweight mapping rows that reference them with
per-series time ranges. 100 series sharing the same resource now produce
1 table row + 100 mapping rows instead of 100 full copies.
New namespaces: resource_table, resource_mapping, scope_table,
scope_mapping. Metric metadata rows are unchanged. The in-memory model
stays denormalized; normalization is purely a Parquet I/O optimization.
Add distributed-scale Parquet features for store-gateway integration:
- Namespace-partitioned row groups: each namespace is written as a
separate row group via Flush boundaries, enabling selective reads.
- Bloom filters: optional split-block bloom filters on labels_hash and
content_hash columns for row group skipping without deserialization.
- Configurable row group size limits: MaxRowsPerRowGroup splits large
namespaces into multiple row groups for bounded memory on read.
- io.ReaderAt read API: ReadSeriesMetadataFromReaderAt decouples from
*os.File, enabling objstore.Bucket-backed readers.
- Namespace filtering: WithNamespaceFilter skips non-matching row
groups using Parquet column index min/max bounds.
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Add support for persisting OTel resource/scope attributes alongside time series
data in TSDB, enabling the info() PromQL function to enrich metrics with
resource context without relying on target_info metric scraping.
Storage layer:
- Add ResourceUpdater interface with UpdateResource() to storage.Appender
- Add ResourceContext to AppendV2Options for passing resource attributes
- Add ResourceQuerier interface for querying stored resource attributes
- Add EntityData type for passing OTel entity information through storage
TSDB seriesmetadata subsystem:
- Add Entity type with OTel entity data model (type, ID, description)
- Add ResourceVersion with versioned identifying/descriptive attributes
- Add VersionedResource for tracking attribute changes over time
- Add MemResourceStore for in-memory resource storage with versioning
- Extend Parquet schema with resource namespace, nested attribute lists,
and entity rows
- Add zstd compression, row sorting, and footer metadata to Parquet writer
OTLP integration:
- Extend CombinedAppender to extract and persist resource attributes from
OTLP ResourceMetrics during ingestion
PromQL info() function:
- Rewrite to use stored resource attributes instead of fetching target_info
series, with attribute name translation via LabelNamer mappings
Web API:
- Add /api/v1/resources endpoint for querying stored resource attributes
- Support format=attributes for autocomplete use cases
- Add OpenAPI documentation for the new endpoint
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Change readXOR2ControlFast to return (uint8, bool) instead of (uint8, error)
to avoid allocating io.EOF on the fast path. Refactor encodeJoint to skip
computing vbits when the value is a stale NaN. Add TestXOR2DecodeFunctionsAcrossPadding
to exercise decodeValue, decodeValueKnownNonZero, and decodeNewLeadingTrailing
across all 64 bit-buffer alignments.
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
The docs for these functions previously described them as acting on
"each histogram sample," which was ambiguous. Add "native" to clarify
they only operate on native histogram samples, not classic histograms.
This distinction was originally documented but lost when the
experimental feature warnings were removed.
Signed-off-by: Jeremy Rickards <jeremy.rickards@sap.com>
Port the following optimizations from the roidelapluie/xor2 branch to
the ST-aware XOR2 implementation on main:
bstream.go:
- Add writeBitsFast() as a writeBits variant that handles the partial
last byte inline to avoid per-byte writeByte calls and writes
complete bytes directly to the stream slice; used only by XOR2 to
leave the shared writeBits unchanged for other encoders
- Add readXOR2ControlFast() for inlinable hot-path control decoding
that avoids buffer refills for the common 4-bit cases
- Add readUvarint()/readVarint() methods that use direct method calls
instead of io.ByteReader interface dispatch, reducing GC pressure
from interior pointer references in findObject
xor2.go:
- Switch all writeBits calls to writeBitsFast
- Use readXOR2ControlFast() + readXOR2Control() fallback in Next()
- Use it.br.readVarint()/readUvarint() instead of binary.ReadVarint/
ReadUvarint to avoid GC overhead from interface dispatch
- Add 3-bit fast path in decodeValue() to read the full value control
prefix in one buffer peek rather than up to three separate bit reads
- Add combined 1+sz bit fast path in decodeValueKnownNonZero() to
fold the control bit and value bits into a single buffer operation
- Add 11-bit combined read in decodeNewLeadingTrailing() to read
leading (5 bits) and sigbits (6 bits) together
- Pre-compute the value XOR delta in encodeJoint() and pass it to
writeVDeltaKnownNonZero(delta uint64) to avoid recomputation
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
When using ManagedIdentity authentication with system-assigned identity,
the client_id field is intentionally left empty. However, the current code
unconditionally sets options.ID = azidentity.ClientID(cfg.ClientID), which
passes an empty string instead of nil. The Azure SDK treats an empty
ClientID as a request for a user-assigned identity with an empty client ID,
rather than falling back to system-assigned identity.
Fix by only setting options.ID when cfg.ClientID is non-empty, matching the
pattern already used in storage/remote/azuread/azuread.go.
Fixes#16634
Signed-off-by: Ogulcan Aydogan <ogulcanaydogan@hotmail.com>
[hcloud.Server.Datacenter] is deprecated and will be removed after 1 July 2026. Use [hcloud.Server.Location] instead.
See https://docs.hetzner.cloud/changelog#2025-12-16-phasing-out-datacenters
Changes to Hetzner meta labels:
- `__meta_hetzner_datacenter`
- is deprecated for the role `robot` but kept for backward compatibility. Using `__meta_hetzner_robot_datacenter` is preferred.
- is deprecated for the role `hcloud` and will stop working after the 1 July 2026.
- `__meta_hetzner_hcloud_datacenter_location` label
- is deprecated but kept for backward compatibility, the same data is available in the [`hcloud.Server.Location`](https://pkg.go.dev/github.com/hetznercloud/hcloud-go/v2/hcloud#Server) struct.
- using `__meta_hetzner_hcloud_location` is preferred.
- `__meta_hetzner_hcloud_datacenter_location_network_zone`
- is deprecated but kept for backward compatibility, the same data is available in the [`hcloud.Server.Location`](https://pkg.go.dev/github.com/hetznercloud/hcloud-go/v2/hcloud#Server) struct.
- using `__meta_hetzner_hcloud_location_network_zone` is preferred.
- `__meta_hetzner_hcloud_location`
- replacement label for `__meta_hetzner_hcloud_datacenter_location`
- `__meta_hetzner_hcloud_location_network_zone`
- replacement label for `__meta_hetzner_hcloud_datacenter_location_network_zone`
- `__meta_hetzner_robot_datacenter`
- replacement label for `__meta_hetzner_datacenter` with the role `robot`.
Signed-off-by: Jonas Lammler <jonas.lammler@hetzner-cloud.de>
* feat(tsdb/chunkenc): add float chunk format with start timestamp support
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* optimize code path and layout
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* make new format usable in head
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* fix issue with seeking to last sample again
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* fix iterator benchmark for chunks not supporting ST
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* reduce footprint of the xoroptst chunk iterator object
It was 80 bytes with a lot of padding compared to the 56 bytes of the
original xor chunk iterator. Made it 64 bytes, tightly packed.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Fix benchmark expectations on ST in interator
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* add inclusive delta test case
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* make testcases independent of order
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* drop unused code
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Drop commented out line
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* documentation
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Small simplification in the doc
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Add delta st inclusive test case for random vt
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Switch to delta of difference of st to prev t
from delta of delta of st.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Write ST after T and V so we can write a single bit on the second sample
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* verify chunk sample len function
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Reduce size of first st stored a little
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* test the case where st equals the t
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* add st equal t to bechmarks
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* test(chunkenc): test that appender can contonue chunks
Test that initializing a chunk appender from an existing chunk
works correctly.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* fix(chunkenc): bug in initializing appender on existing chunk
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Add cases with jitter in the start time as well
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* tsdb: ST-in-WAL: Counter implementation and benchmarks (#17671)
Initial implementation of https://github.com/prometheus/prometheus/issues/17790.
Only implements ST-per-sample for Counters. Tests and benchmarks updated.
Note: This increases the size of the RefSample object for all users, whether st-per-sample is turned on or not.
Signed-off-by: Owen Williams <owen.williams@grafana.com>
* refactor: sed enableStStorage/enableSTStorage
Signed-off-by: bwplotka <bwplotka@gmail.com>
* feat[scrape]: add ST parsing support to scrape AppenderV2 flow (#18103)
Signed-off-by: bwplotka <bwplotka@gmail.com>
* feat(tsdb): change head opt EnableSTStorage to atomic (#18107)
In downstream projects this needs to be set dynamically per tenant.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Merge pull request #18108 from prometheus/bwplotka/fix
scrape: add tests for ST appending; add warnings for ST feature flag users around _created drop
* refact(tsdb): trivial rename (#18109)
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* fix(tsdb): missing passing head option to wal/wbl write (#18113)
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* feat(tsdb): allow using ST capable XOR chunks - retain format on read (#18013)
* feat(tsdb): allow appending to ST capable XOR chunk optionally
Only for float samples as of now. Supports for in-order and out-of-order
samples.
Make sure that on readout the ST capable chunks are returned automatically.
When the chunks are returned as is, this is trivially true.
When a chunk needs to be re-coded due to deletion (tombstone) markers,
we take the encoding of the original chunk.
When a chunk needs to be created from overlapping chunks, we observe
whether ST is zero or not and create the new chunk based on that.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* fix test after merge
Signed-off-by: bwplotka <bwplotka@gmail.com>
* feat: RW2 sending ST support
Signed-off-by: bwplotka <bwplotka@gmail.com>
tmp
Signed-off-by: bwplotka <bwplotka@gmail.com>
* tests: test ST in a cheapest way possible
Signed-off-by: bwplotka <bwplotka@gmail.com>
* tests: add bench CLI recommended invokations
Signed-off-by: bwplotka <bwplotka@gmail.com>
* fixed tests after rebase
Signed-off-by: bwplotka <bwplotka@gmail.com>
* feat(chunkenc): replace xoroptst chunk encoding with xor2
XOR2 is based on https://github.com/prometheus/prometheus/pull/18238
With additional ST support.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* feat: add compliance RW sender test for agent
Signed-off-by: bwplotka <bwplotka@gmail.com>
* feat(agent): add support for appending ST
Signed-off-by: bwplotka <bwplotka@gmail.com>
* replace stray xoroptst words
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
* post merge conflict fixes
Signed-off-by: bwplotka <bwplotka@gmail.com>
* feat(tsdb): register st_storage in feature API
Register the st-storage feature flag in the feature registry via
the TSDB options, consistent with how other TSDB features like
exemplar_storage and delayed_compaction are registered.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Coded with Claude Sonnet 4.6.
* Document xor2-encoding feature flag
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Add xor2-encoding feature flag
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Update CHANGELOG
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Fix linting
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Remove setting of xor2 encoding option in db open
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Fix tests
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Fix linting
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Update feature flag description
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Update comments and feature flag description
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Update documentation for st-storage feature
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* st: disconnect st-storage with xor2-encoding given planned experiments (#18316)
* st: disconnect st-storage with xor2-encoding given planned experiments
Signed-off-by: bwplotka <bwplotka@gmail.com>
* Update docs/feature_flags.md
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
* Update docs/feature_flags.md
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
* Update docs/feature_flags.md
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
* Update docs/feature_flags.md
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
---------
Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
---------
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Signed-off-by: Aleksandr Smirnov <5targazer@mail.ru>
Signed-off-by: Mohammad Abbasi <mohammad.v184@gmail.com>
Signed-off-by: matt-gp <small_minority@hotmail.com>
Signed-off-by: Ian Kerins <git@isk.haus>
Signed-off-by: SuperQ <superq@gmail.com>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Signed-off-by: ffgan <sudoemt@gmail.com>
Signed-off-by: Patryk Prus <p@trykpr.us>
Signed-off-by: Owen Williams <owen.williams@grafana.com>
Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: 3Juhwan <13selfesteem91@naver.com>
Signed-off-by: Sammy Tran <sammyqtran@gmail.com>
Signed-off-by: Casie Chen <casie.chen@grafana.com>
Signed-off-by: Dan Cech <dcech@grafana.com>
Signed-off-by: kakabisht <kakabisht07@gmail.com>
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: Divyansh Mishra <divyanshmishra@Divyanshs-MacBook-Air-3.local>
Signed-off-by: Varun Chawla <varun_6april@hotmail.com>
Signed-off-by: Martin Valiente Ainz <64830185+tinitiuset@users.noreply.github.com>
Signed-off-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
Signed-off-by: Björn Rabenstein <github@rabenste.in>
Signed-off-by: beorn7 <beorn@grafana.com>
Signed-off-by: Sayuru <71478576+samaras3@users.noreply.github.com>
Signed-off-by: Matt <small_minority@hotmail.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
Co-authored-by: Bryan Boreham <bjboreham@gmail.com>
Co-authored-by: Sasha <103973965+crush-on-anechka@users.noreply.github.com>
Co-authored-by: Mohammad Abbasi <mohammad.v184@gmail.com>
Co-authored-by: matt-gp <small_minority@hotmail.com>
Co-authored-by: Ian Kerins <git@isk.haus>
Co-authored-by: SuperQ <superq@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Julien <291750+roidelapluie@users.noreply.github.com>
Co-authored-by: ffgan <sudoemt@gmail.com>
Co-authored-by: Patryk Prus <p@trykpr.us>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Joe Adams <github@joeadams.io>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Owen Williams <owen.williams@grafana.com>
Co-authored-by: 3Juhwan <13selfesteem91@naver.com>
Co-authored-by: Casie Chen <casie.chen@grafana.com>
Co-authored-by: Dan Cech <dcech@grafana.com>
Co-authored-by: hridyesh bisht <41201308+kakabisht@users.noreply.github.com>
Co-authored-by: zenador <zenador@users.noreply.github.com>
Co-authored-by: Divyansh Mishra <divyanshmishra@Divyanshs-MacBook-Air-3.local>
Co-authored-by: Varun Chawla <varun_6april@hotmail.com>
Co-authored-by: Martin Valiente Ainz <64830185+tinitiuset@users.noreply.github.com>
Co-authored-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>
Co-authored-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Co-authored-by: Linas Medžiūnas <linasm@users.noreply.github.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
Co-authored-by: beorn7 <beorn@grafana.com>
Co-authored-by: Sayuru <71478576+samaras3@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Carrie Edwards <edwrdscarrie@gmail.com>
