The docs for these functions previously described them as acting on
"each histogram sample," which was ambiguous. Add "native" to clarify
they only operate on native histogram samples, not classic histograms.
This distinction was originally documented but lost when the
experimental feature warnings were removed.
Signed-off-by: Jeremy Rickards <jeremy.rickards@sap.com>
When using ManagedIdentity authentication with system-assigned identity,
the client_id field is intentionally left empty. However, the current code
unconditionally sets options.ID = azidentity.ClientID(cfg.ClientID), which
passes an empty string instead of nil. The Azure SDK treats an empty
ClientID as a request for a user-assigned identity with an empty client ID,
rather than falling back to system-assigned identity.
Fix by only setting options.ID when cfg.ClientID is non-empty, matching the
pattern already used in storage/remote/azuread/azuread.go.
Fixes#16634
Signed-off-by: Ogulcan Aydogan <ogulcanaydogan@hotmail.com>
[hcloud.Server.Datacenter] is deprecated and will be removed after 1 July 2026. Use [hcloud.Server.Location] instead.
See https://docs.hetzner.cloud/changelog#2025-12-16-phasing-out-datacenters
Changes to Hetzner meta labels:
- `__meta_hetzner_datacenter`
- is deprecated for the role `robot` but kept for backward compatibility. Using `__meta_hetzner_robot_datacenter` is preferred.
- is deprecated for the role `hcloud` and will stop working after the 1 July 2026.
- `__meta_hetzner_hcloud_datacenter_location` label
- is deprecated but kept for backward compatibility, the same data is available in the [`hcloud.Server.Location`](https://pkg.go.dev/github.com/hetznercloud/hcloud-go/v2/hcloud#Server) struct.
- using `__meta_hetzner_hcloud_location` is preferred.
- `__meta_hetzner_hcloud_datacenter_location_network_zone`
- is deprecated but kept for backward compatibility, the same data is available in the [`hcloud.Server.Location`](https://pkg.go.dev/github.com/hetznercloud/hcloud-go/v2/hcloud#Server) struct.
- using `__meta_hetzner_hcloud_location_network_zone` is preferred.
- `__meta_hetzner_hcloud_location`
- replacement label for `__meta_hetzner_hcloud_datacenter_location`
- `__meta_hetzner_hcloud_location_network_zone`
- replacement label for `__meta_hetzner_hcloud_datacenter_location_network_zone`
- `__meta_hetzner_robot_datacenter`
- replacement label for `__meta_hetzner_datacenter` with the role `robot`.
Signed-off-by: Jonas Lammler <jonas.lammler@hetzner-cloud.de>
* feat(tsdb/chunkenc): add float chunk format with start timestamp support
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* optimize code path and layout
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* make new format usable in head
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* fix issue with seeking to last sample again
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* fix iterator benchmark for chunks not supporting ST
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* reduce footprint of the xoroptst chunk iterator object
It was 80 bytes with a lot of padding compared to the 56 bytes of the
original xor chunk iterator. Made it 64 bytes, tightly packed.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Fix benchmark expectations on ST in interator
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* add inclusive delta test case
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* make testcases independent of order
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* drop unused code
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Drop commented out line
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* documentation
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Small simplification in the doc
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Add delta st inclusive test case for random vt
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Switch to delta of difference of st to prev t
from delta of delta of st.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Write ST after T and V so we can write a single bit on the second sample
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* verify chunk sample len function
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Reduce size of first st stored a little
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* test the case where st equals the t
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* add st equal t to bechmarks
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* test(chunkenc): test that appender can contonue chunks
Test that initializing a chunk appender from an existing chunk
works correctly.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* fix(chunkenc): bug in initializing appender on existing chunk
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Add cases with jitter in the start time as well
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* tsdb: ST-in-WAL: Counter implementation and benchmarks (#17671)
Initial implementation of https://github.com/prometheus/prometheus/issues/17790.
Only implements ST-per-sample for Counters. Tests and benchmarks updated.
Note: This increases the size of the RefSample object for all users, whether st-per-sample is turned on or not.
Signed-off-by: Owen Williams <owen.williams@grafana.com>
* refactor: sed enableStStorage/enableSTStorage
Signed-off-by: bwplotka <bwplotka@gmail.com>
* feat[scrape]: add ST parsing support to scrape AppenderV2 flow (#18103)
Signed-off-by: bwplotka <bwplotka@gmail.com>
* feat(tsdb): change head opt EnableSTStorage to atomic (#18107)
In downstream projects this needs to be set dynamically per tenant.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Merge pull request #18108 from prometheus/bwplotka/fix
scrape: add tests for ST appending; add warnings for ST feature flag users around _created drop
* refact(tsdb): trivial rename (#18109)
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* fix(tsdb): missing passing head option to wal/wbl write (#18113)
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* feat(tsdb): allow using ST capable XOR chunks - retain format on read (#18013)
* feat(tsdb): allow appending to ST capable XOR chunk optionally
Only for float samples as of now. Supports for in-order and out-of-order
samples.
Make sure that on readout the ST capable chunks are returned automatically.
When the chunks are returned as is, this is trivially true.
When a chunk needs to be re-coded due to deletion (tombstone) markers,
we take the encoding of the original chunk.
When a chunk needs to be created from overlapping chunks, we observe
whether ST is zero or not and create the new chunk based on that.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* fix test after merge
Signed-off-by: bwplotka <bwplotka@gmail.com>
* feat: RW2 sending ST support
Signed-off-by: bwplotka <bwplotka@gmail.com>
tmp
Signed-off-by: bwplotka <bwplotka@gmail.com>
* tests: test ST in a cheapest way possible
Signed-off-by: bwplotka <bwplotka@gmail.com>
* tests: add bench CLI recommended invokations
Signed-off-by: bwplotka <bwplotka@gmail.com>
* fixed tests after rebase
Signed-off-by: bwplotka <bwplotka@gmail.com>
* feat(chunkenc): replace xoroptst chunk encoding with xor2
XOR2 is based on https://github.com/prometheus/prometheus/pull/18238
With additional ST support.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* feat: add compliance RW sender test for agent
Signed-off-by: bwplotka <bwplotka@gmail.com>
* feat(agent): add support for appending ST
Signed-off-by: bwplotka <bwplotka@gmail.com>
* replace stray xoroptst words
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
* post merge conflict fixes
Signed-off-by: bwplotka <bwplotka@gmail.com>
* feat(tsdb): register st_storage in feature API
Register the st-storage feature flag in the feature registry via
the TSDB options, consistent with how other TSDB features like
exemplar_storage and delayed_compaction are registered.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Coded with Claude Sonnet 4.6.
* Document xor2-encoding feature flag
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Add xor2-encoding feature flag
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Update CHANGELOG
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Fix linting
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Remove setting of xor2 encoding option in db open
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Fix tests
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Fix linting
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Update feature flag description
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Update comments and feature flag description
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Update documentation for st-storage feature
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* st: disconnect st-storage with xor2-encoding given planned experiments (#18316)
* st: disconnect st-storage with xor2-encoding given planned experiments
Signed-off-by: bwplotka <bwplotka@gmail.com>
* Update docs/feature_flags.md
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
* Update docs/feature_flags.md
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
* Update docs/feature_flags.md
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
* Update docs/feature_flags.md
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
---------
Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
---------
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Signed-off-by: Aleksandr Smirnov <5targazer@mail.ru>
Signed-off-by: Mohammad Abbasi <mohammad.v184@gmail.com>
Signed-off-by: matt-gp <small_minority@hotmail.com>
Signed-off-by: Ian Kerins <git@isk.haus>
Signed-off-by: SuperQ <superq@gmail.com>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Signed-off-by: ffgan <sudoemt@gmail.com>
Signed-off-by: Patryk Prus <p@trykpr.us>
Signed-off-by: Owen Williams <owen.williams@grafana.com>
Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: 3Juhwan <13selfesteem91@naver.com>
Signed-off-by: Sammy Tran <sammyqtran@gmail.com>
Signed-off-by: Casie Chen <casie.chen@grafana.com>
Signed-off-by: Dan Cech <dcech@grafana.com>
Signed-off-by: kakabisht <kakabisht07@gmail.com>
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: Divyansh Mishra <divyanshmishra@Divyanshs-MacBook-Air-3.local>
Signed-off-by: Varun Chawla <varun_6april@hotmail.com>
Signed-off-by: Martin Valiente Ainz <64830185+tinitiuset@users.noreply.github.com>
Signed-off-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
Signed-off-by: Björn Rabenstein <github@rabenste.in>
Signed-off-by: beorn7 <beorn@grafana.com>
Signed-off-by: Sayuru <71478576+samaras3@users.noreply.github.com>
Signed-off-by: Matt <small_minority@hotmail.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
Co-authored-by: Bryan Boreham <bjboreham@gmail.com>
Co-authored-by: Sasha <103973965+crush-on-anechka@users.noreply.github.com>
Co-authored-by: Mohammad Abbasi <mohammad.v184@gmail.com>
Co-authored-by: matt-gp <small_minority@hotmail.com>
Co-authored-by: Ian Kerins <git@isk.haus>
Co-authored-by: SuperQ <superq@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Julien <291750+roidelapluie@users.noreply.github.com>
Co-authored-by: ffgan <sudoemt@gmail.com>
Co-authored-by: Patryk Prus <p@trykpr.us>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Joe Adams <github@joeadams.io>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Owen Williams <owen.williams@grafana.com>
Co-authored-by: 3Juhwan <13selfesteem91@naver.com>
Co-authored-by: Casie Chen <casie.chen@grafana.com>
Co-authored-by: Dan Cech <dcech@grafana.com>
Co-authored-by: hridyesh bisht <41201308+kakabisht@users.noreply.github.com>
Co-authored-by: zenador <zenador@users.noreply.github.com>
Co-authored-by: Divyansh Mishra <divyanshmishra@Divyanshs-MacBook-Air-3.local>
Co-authored-by: Varun Chawla <varun_6april@hotmail.com>
Co-authored-by: Martin Valiente Ainz <64830185+tinitiuset@users.noreply.github.com>
Co-authored-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>
Co-authored-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Co-authored-by: Linas Medžiūnas <linasm@users.noreply.github.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
Co-authored-by: beorn7 <beorn@grafana.com>
Co-authored-by: Sayuru <71478576+samaras3@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Adding scape on shutdown
Signed-off-by: avilevy <avilevy@google.com>
* scrape: replace skipOffsetting to make the test offset deterministic instead of skipping it entirely
Signed-off-by: avilevy <avilevy@google.com>
* renamed calculateScrapeOffset to getScrapeOffset
Signed-off-by: avilevy <avilevy@google.com>
* test(scrape): refactor time-based manager tests to use synctest
Addresses PR feedback to remove flaky, time-based sleeping in the scrape manager tests.
Add TestManager_InitialScrapeOffset and TestManager_ScrapeOnShutdown to use the testing/synctest package, completely eliminating real-world time.Sleep delays and making the assertions 100% deterministic.
- Replaced httptest.Server with net.Pipe and a custom startFakeHTTPServer helper to ensure all network I/O remains durably blocked inside the synctest bubble.
- Leveraged the skipOffsetting option to eliminate random scrape jitter, making the time-travel math exact and predictable.
- Using skipOffsetting also safely bypasses the global singleflight DNS lookup in setOffsetSeed, which previously caused cross-bubble panics in synctest.
- Extracted shared boilerplate into a setupSynctestManager helper to keep the test cases highly readable and data-driven.
Signed-off-by: avilevy <avilevy@google.com>
* Clarify use cases in InitialScrapeOffset comment
Signed-off-by: avilevy <avilevy@google.com>
* test(scrape): use httptest for mock server to respect context cancellation
- Replaced manual HTTP string formatting over `net.Pipe` with `httptest.NewUnstartedServer`.
- Implemented an in-memory `pipeListener` to allow the server to handle `net.Pipe` connections directly. This preserves `synctest` time isolation without opening real OS ports.
- Added explicit `r.Context().Done()` handling in the mock HTTP handler to properly simulate aborted requests and scrape timeouts.
- Validates that the request context remains active and is not prematurely cancelled during `ScrapeOnShutdown` scenarios.
- Renamed `skipOffsetting` to `skipJitterOffsetting`.
- Addressed other PR comments.
Signed-off-by: avilevy <avilevy@google.com>
* tmp
Signed-off-by: bwplotka <bwplotka@gmail.com>
* exp2
Signed-off-by: bwplotka <bwplotka@gmail.com>
* fix
Signed-off-by: bwplotka <bwplotka@gmail.com>
* scrape: fix scrapeOnShutdown context bug and refactor test helpers
The scrapeOnShutdown feature was failing during manager shutdown because
the scrape pool context was being cancelled before the final shutdown
scrapes could execute. Fix this by delaying context cancellation
in scrapePool.stop() until after all scrape loops have stopped.
In addition:
- Added test cases to verify scrapeOnShutdown works with InitialScrapeOffset.
- Refactored network test helper functions from manager_test.go to
helpers_test.go.
- Addressed other comments.
Signed-off-by: avilevy <avilevy@google.com>
* Update scrape/scrape.go
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: avilevy18 <105948922+avilevy18@users.noreply.github.com>
---------
Signed-off-by: avilevy <avilevy@google.com>
Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: avilevy18 <105948922+avilevy18@users.noreply.github.com>
Co-authored-by: bwplotka <bwplotka@gmail.com>
* tsdb: mmap histogram chunks during WAL replay
The float sample path in processWALSamples calls mmapChunks when a new
chunk is created during WAL replay, but the histogram path was missing
this call. Without it, histogram head chunks accumulate as a linked
list in memory rather than being mmapped, causing unnecessary memory
growth during long WAL replays.
---------
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Register the st-storage feature flag in the feature registry via
the TSDB options, consistent with how other TSDB features like
exemplar_storage and delayed_compaction are registered.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Coded with Claude Sonnet 4.6.
