prometheus/tsdb/agent
Bartlomiej Plotka 2ba3046c80
Merge float st-storage implementation (PROM-60) and initial xor2-encoding (#18062)
* feat(tsdb/chunkenc): add float chunk format with start timestamp support


Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* optimize code path and layout

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* make new format usable in head

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* fix issue with seeking to last sample again

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* fix iterator benchmark for chunks not supporting ST

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* reduce footprint of the xoroptst chunk iterator object

It was 80 bytes with a lot of padding compared to the 56 bytes of the
original xor chunk iterator. Made it 64 bytes, tightly packed.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Fix benchmark expectations on ST in interator

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* add inclusive delta test case

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* make testcases independent of order

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* drop unused code

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Drop commented out line

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* documentation

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Small simplification in the doc

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Add delta st inclusive test case for random vt

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Switch to delta of difference of st to prev t

from delta of delta of st.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Write ST after T and V so we can write a single bit on the second sample

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* verify chunk sample len function

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Reduce size of first st stored a little

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* test the case where st equals the t

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* add st equal t to bechmarks

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* test(chunkenc): test that appender can contonue chunks

Test that initializing a chunk appender from an existing chunk
works correctly.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* fix(chunkenc): bug in initializing appender on existing chunk

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Add cases with jitter in the start time as well


Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* tsdb: ST-in-WAL: Counter implementation and benchmarks (#17671)

Initial implementation of https://github.com/prometheus/prometheus/issues/17790.
Only implements ST-per-sample for Counters. Tests and benchmarks updated.

Note: This increases the size of the RefSample object for all users, whether st-per-sample is turned on or not.

Signed-off-by: Owen Williams <owen.williams@grafana.com>

* refactor: sed enableStStorage/enableSTStorage

Signed-off-by: bwplotka <bwplotka@gmail.com>

* feat[scrape]: add ST parsing support to scrape AppenderV2 flow (#18103)

Signed-off-by: bwplotka <bwplotka@gmail.com>

* feat(tsdb): change head opt EnableSTStorage to atomic (#18107)

In downstream projects this needs to be set dynamically per tenant.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Merge pull request #18108 from prometheus/bwplotka/fix

scrape: add tests for ST appending; add warnings for ST feature flag users around _created drop

* refact(tsdb): trivial rename (#18109)

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* fix(tsdb): missing passing head option to wal/wbl write (#18113)

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* feat(tsdb): allow using ST capable XOR chunks - retain format on read (#18013)

* feat(tsdb): allow appending to ST capable XOR chunk optionally

Only for float samples as of now.  Supports for in-order and out-of-order
samples.

Make sure that on readout the ST capable chunks are returned automatically.
When the chunks are returned as is, this is trivially true.
When a chunk needs to be re-coded due to deletion (tombstone) markers,
we take the encoding of the original chunk.
When a chunk needs to be created from overlapping chunks, we observe
whether ST is zero or not and create the new chunk based on that.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* fix test after merge

Signed-off-by: bwplotka <bwplotka@gmail.com>

* feat: RW2 sending ST support

Signed-off-by: bwplotka <bwplotka@gmail.com>

tmp

Signed-off-by: bwplotka <bwplotka@gmail.com>

* tests: test ST in a cheapest way possible

Signed-off-by: bwplotka <bwplotka@gmail.com>

* tests: add bench CLI recommended invokations

Signed-off-by: bwplotka <bwplotka@gmail.com>

* fixed tests after rebase

Signed-off-by: bwplotka <bwplotka@gmail.com>

* feat(chunkenc): replace xoroptst chunk encoding with xor2

XOR2 is based on https://github.com/prometheus/prometheus/pull/18238
With additional ST support.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* feat: add compliance RW sender test for agent

Signed-off-by: bwplotka <bwplotka@gmail.com>

* feat(agent): add support for appending ST

Signed-off-by: bwplotka <bwplotka@gmail.com>

* replace stray xoroptst words

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* post merge conflict fixes

Signed-off-by: bwplotka <bwplotka@gmail.com>

* feat(tsdb): register st_storage in feature API

Register the st-storage feature flag in the feature registry via
the TSDB options, consistent with how other TSDB features like
exemplar_storage and delayed_compaction are registered.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Coded with Claude Sonnet 4.6.

* Document xor2-encoding feature flag

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>

* Add xor2-encoding feature flag

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>

* Update CHANGELOG

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>

* Fix linting

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>

* Remove setting of xor2 encoding option in db open

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>

* Fix tests

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>

* Fix linting

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>

* Update feature flag description

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>

* Update comments and feature flag description

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>

* Update documentation for st-storage feature

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>

* st: disconnect st-storage with xor2-encoding given planned experiments (#18316)

* st: disconnect st-storage with xor2-encoding given planned experiments

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Update docs/feature_flags.md

Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Update docs/feature_flags.md

Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Update docs/feature_flags.md

Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Update docs/feature_flags.md

Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

---------

Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>

---------

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Signed-off-by: Aleksandr Smirnov <5targazer@mail.ru>
Signed-off-by: Mohammad Abbasi <mohammad.v184@gmail.com>
Signed-off-by: matt-gp <small_minority@hotmail.com>
Signed-off-by: Ian Kerins <git@isk.haus>
Signed-off-by: SuperQ <superq@gmail.com>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Signed-off-by: ffgan <sudoemt@gmail.com>
Signed-off-by: Patryk Prus <p@trykpr.us>
Signed-off-by: Owen Williams <owen.williams@grafana.com>
Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: 3Juhwan <13selfesteem91@naver.com>
Signed-off-by: Sammy Tran <sammyqtran@gmail.com>
Signed-off-by: Casie Chen <casie.chen@grafana.com>
Signed-off-by: Dan Cech <dcech@grafana.com>
Signed-off-by: kakabisht <kakabisht07@gmail.com>
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: Divyansh Mishra <divyanshmishra@Divyanshs-MacBook-Air-3.local>
Signed-off-by: Varun Chawla <varun_6april@hotmail.com>
Signed-off-by: Martin Valiente Ainz <64830185+tinitiuset@users.noreply.github.com>
Signed-off-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
Signed-off-by: Björn Rabenstein <github@rabenste.in>
Signed-off-by: beorn7 <beorn@grafana.com>
Signed-off-by: Sayuru <71478576+samaras3@users.noreply.github.com>
Signed-off-by: Matt <small_minority@hotmail.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
Co-authored-by: Bryan Boreham <bjboreham@gmail.com>
Co-authored-by: Sasha <103973965+crush-on-anechka@users.noreply.github.com>
Co-authored-by: Mohammad Abbasi <mohammad.v184@gmail.com>
Co-authored-by: matt-gp <small_minority@hotmail.com>
Co-authored-by: Ian Kerins <git@isk.haus>
Co-authored-by: SuperQ <superq@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Julien <291750+roidelapluie@users.noreply.github.com>
Co-authored-by: ffgan <sudoemt@gmail.com>
Co-authored-by: Patryk Prus <p@trykpr.us>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Joe Adams <github@joeadams.io>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Owen Williams <owen.williams@grafana.com>
Co-authored-by: 3Juhwan <13selfesteem91@naver.com>
Co-authored-by: Casie Chen <casie.chen@grafana.com>
Co-authored-by: Dan Cech <dcech@grafana.com>
Co-authored-by: hridyesh bisht <41201308+kakabisht@users.noreply.github.com>
Co-authored-by: zenador <zenador@users.noreply.github.com>
Co-authored-by: Divyansh Mishra <divyanshmishra@Divyanshs-MacBook-Air-3.local>
Co-authored-by: Varun Chawla <varun_6april@hotmail.com>
Co-authored-by: Martin Valiente Ainz <64830185+tinitiuset@users.noreply.github.com>
Co-authored-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>
Co-authored-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Co-authored-by: Linas Medžiūnas <linasm@users.noreply.github.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
Co-authored-by: beorn7 <beorn@grafana.com>
Co-authored-by: Sayuru <71478576+samaras3@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Carrie Edwards <edwrdscarrie@gmail.com>
2026-03-19 10:51:40 +01:00
..
db.go Merge float st-storage implementation (PROM-60) and initial xor2-encoding (#18062) 2026-03-19 10:51:40 +01:00
db_append_v2.go feat(agent): add support for appending ST 2026-03-10 12:27:48 +00:00
db_append_v2_test.go feat(agent): add support for appending ST 2026-03-10 12:27:48 +00:00
db_test.go Merge branch 'main' into start-time-main-sync 2026-03-12 08:28:15 +00:00
series.go Use Set in GetOrSet to enforce proper lock ordering 2026-02-25 16:52:07 -05:00
series_test.go Prevent duplicate SeriesRefs from being lost in db stripeSeries 2026-02-25 06:03:53 -05:00