* feat(tsdb/chunkenc): add float chunk format with start timestamp support
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* optimize code path and layout
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* make new format usable in head
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* fix issue with seeking to last sample again
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* fix iterator benchmark for chunks not supporting ST
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* reduce footprint of the xoroptst chunk iterator object
It was 80 bytes with a lot of padding compared to the 56 bytes of the
original xor chunk iterator. Made it 64 bytes, tightly packed.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Fix benchmark expectations on ST in interator
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* add inclusive delta test case
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* make testcases independent of order
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* drop unused code
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Drop commented out line
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* documentation
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Small simplification in the doc
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Add delta st inclusive test case for random vt
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Switch to delta of difference of st to prev t
from delta of delta of st.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Write ST after T and V so we can write a single bit on the second sample
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* verify chunk sample len function
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Reduce size of first st stored a little
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* test the case where st equals the t
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* add st equal t to bechmarks
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* test(chunkenc): test that appender can contonue chunks
Test that initializing a chunk appender from an existing chunk
works correctly.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* fix(chunkenc): bug in initializing appender on existing chunk
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Add cases with jitter in the start time as well
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* tsdb: ST-in-WAL: Counter implementation and benchmarks (#17671)
Initial implementation of https://github.com/prometheus/prometheus/issues/17790.
Only implements ST-per-sample for Counters. Tests and benchmarks updated.
Note: This increases the size of the RefSample object for all users, whether st-per-sample is turned on or not.
Signed-off-by: Owen Williams <owen.williams@grafana.com>
* refactor: sed enableStStorage/enableSTStorage
Signed-off-by: bwplotka <bwplotka@gmail.com>
* feat[scrape]: add ST parsing support to scrape AppenderV2 flow (#18103)
Signed-off-by: bwplotka <bwplotka@gmail.com>
* feat(tsdb): change head opt EnableSTStorage to atomic (#18107)
In downstream projects this needs to be set dynamically per tenant.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Merge pull request #18108 from prometheus/bwplotka/fix
scrape: add tests for ST appending; add warnings for ST feature flag users around _created drop
* refact(tsdb): trivial rename (#18109)
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* fix(tsdb): missing passing head option to wal/wbl write (#18113)
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* feat(tsdb): allow using ST capable XOR chunks - retain format on read (#18013)
* feat(tsdb): allow appending to ST capable XOR chunk optionally
Only for float samples as of now. Supports for in-order and out-of-order
samples.
Make sure that on readout the ST capable chunks are returned automatically.
When the chunks are returned as is, this is trivially true.
When a chunk needs to be re-coded due to deletion (tombstone) markers,
we take the encoding of the original chunk.
When a chunk needs to be created from overlapping chunks, we observe
whether ST is zero or not and create the new chunk based on that.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* fix test after merge
Signed-off-by: bwplotka <bwplotka@gmail.com>
* feat: RW2 sending ST support
Signed-off-by: bwplotka <bwplotka@gmail.com>
tmp
Signed-off-by: bwplotka <bwplotka@gmail.com>
* tests: test ST in a cheapest way possible
Signed-off-by: bwplotka <bwplotka@gmail.com>
* tests: add bench CLI recommended invokations
Signed-off-by: bwplotka <bwplotka@gmail.com>
* fixed tests after rebase
Signed-off-by: bwplotka <bwplotka@gmail.com>
* feat(chunkenc): replace xoroptst chunk encoding with xor2
XOR2 is based on https://github.com/prometheus/prometheus/pull/18238
With additional ST support.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* feat: add compliance RW sender test for agent
Signed-off-by: bwplotka <bwplotka@gmail.com>
* feat(agent): add support for appending ST
Signed-off-by: bwplotka <bwplotka@gmail.com>
* replace stray xoroptst words
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
* post merge conflict fixes
Signed-off-by: bwplotka <bwplotka@gmail.com>
* feat(tsdb): register st_storage in feature API
Register the st-storage feature flag in the feature registry via
the TSDB options, consistent with how other TSDB features like
exemplar_storage and delayed_compaction are registered.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Coded with Claude Sonnet 4.6.
* Document xor2-encoding feature flag
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Add xor2-encoding feature flag
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Update CHANGELOG
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Fix linting
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Remove setting of xor2 encoding option in db open
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Fix tests
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Fix linting
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Update feature flag description
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Update comments and feature flag description
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Update documentation for st-storage feature
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* st: disconnect st-storage with xor2-encoding given planned experiments (#18316)
* st: disconnect st-storage with xor2-encoding given planned experiments
Signed-off-by: bwplotka <bwplotka@gmail.com>
* Update docs/feature_flags.md
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
* Update docs/feature_flags.md
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
* Update docs/feature_flags.md
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
* Update docs/feature_flags.md
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
---------
Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
---------
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Signed-off-by: Aleksandr Smirnov <5targazer@mail.ru>
Signed-off-by: Mohammad Abbasi <mohammad.v184@gmail.com>
Signed-off-by: matt-gp <small_minority@hotmail.com>
Signed-off-by: Ian Kerins <git@isk.haus>
Signed-off-by: SuperQ <superq@gmail.com>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Signed-off-by: ffgan <sudoemt@gmail.com>
Signed-off-by: Patryk Prus <p@trykpr.us>
Signed-off-by: Owen Williams <owen.williams@grafana.com>
Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: 3Juhwan <13selfesteem91@naver.com>
Signed-off-by: Sammy Tran <sammyqtran@gmail.com>
Signed-off-by: Casie Chen <casie.chen@grafana.com>
Signed-off-by: Dan Cech <dcech@grafana.com>
Signed-off-by: kakabisht <kakabisht07@gmail.com>
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: Divyansh Mishra <divyanshmishra@Divyanshs-MacBook-Air-3.local>
Signed-off-by: Varun Chawla <varun_6april@hotmail.com>
Signed-off-by: Martin Valiente Ainz <64830185+tinitiuset@users.noreply.github.com>
Signed-off-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
Signed-off-by: Björn Rabenstein <github@rabenste.in>
Signed-off-by: beorn7 <beorn@grafana.com>
Signed-off-by: Sayuru <71478576+samaras3@users.noreply.github.com>
Signed-off-by: Matt <small_minority@hotmail.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
Co-authored-by: Bryan Boreham <bjboreham@gmail.com>
Co-authored-by: Sasha <103973965+crush-on-anechka@users.noreply.github.com>
Co-authored-by: Mohammad Abbasi <mohammad.v184@gmail.com>
Co-authored-by: matt-gp <small_minority@hotmail.com>
Co-authored-by: Ian Kerins <git@isk.haus>
Co-authored-by: SuperQ <superq@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Julien <291750+roidelapluie@users.noreply.github.com>
Co-authored-by: ffgan <sudoemt@gmail.com>
Co-authored-by: Patryk Prus <p@trykpr.us>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Joe Adams <github@joeadams.io>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Owen Williams <owen.williams@grafana.com>
Co-authored-by: 3Juhwan <13selfesteem91@naver.com>
Co-authored-by: Casie Chen <casie.chen@grafana.com>
Co-authored-by: Dan Cech <dcech@grafana.com>
Co-authored-by: hridyesh bisht <41201308+kakabisht@users.noreply.github.com>
Co-authored-by: zenador <zenador@users.noreply.github.com>
Co-authored-by: Divyansh Mishra <divyanshmishra@Divyanshs-MacBook-Air-3.local>
Co-authored-by: Varun Chawla <varun_6april@hotmail.com>
Co-authored-by: Martin Valiente Ainz <64830185+tinitiuset@users.noreply.github.com>
Co-authored-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>
Co-authored-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Co-authored-by: Linas Medžiūnas <linasm@users.noreply.github.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
Co-authored-by: beorn7 <beorn@grafana.com>
Co-authored-by: Sayuru <71478576+samaras3@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Carrie Edwards <edwrdscarrie@gmail.com>
Initial implementation of https://github.com/prometheus/prometheus/issues/17790.
Only implements ST-per-sample for Counters. Tests and benchmarks updated.
Note: This increases the size of the RefSample object for all users, whether st-per-sample is turned on or not.
Signed-off-by: Owen Williams <owen.williams@grafana.com>
We have separate pools for Appender and AppenderV2 objects, and must not
put another kind of object into them.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Partially fixes https://github.com/prometheus/prometheus/issues/17416 by
renaming all CT* names to ST* in the whole codebase except RW2 (this is
done in separate
[PR](https://github.com/prometheus/prometheus/pull/17411)) and
PrometheusProto exposition proto.
```
CreatedTimestamp -> StartTimestamp
CreatedTimeStamp -> StartTimestamp
created_timestamp -> start_timestamp
CT -> ST
ct -> st
```
Signed-off-by: bwplotka <bwplotka@gmail.com>
Reduce the resolution of histograms as needed and ignore invalid
schemas while emitting a warning log.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
See
https://pkg.go.dev/golang.org/x/tools/gopls/internal/analysis/modernize
for details.
This ran into a few issues (arguably bugs in the modernize tool),
which I will fix in the next commit, so that we have transparency what
was done automatically.
Beyond those hiccups, I believe all the changes applied are
legitimate. Even where there might be no tangible direct gain, I would
argue it's still better to use the "modern" way to avoid micro
discussions in tiny style PRs later.
Signed-off-by: beorn7 <beorn@grafana.com>
Right now Prometheus only logs when these operations are completed.
It's a bit surprising to see suddenly a message saying "I was busy doing X for the past N minutes"
so let's add a message when the operation starts, so it's easier to understand what Prometheus was doing at any point in time
when reading logs.
Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com>
This removes the stringlabels build tag, makes that implementation the default one, and moves the old labels implementation under the slicelabels build tag.
Fixes#16064.
Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com>
Renames the head's deleted map to walExpiries, and creates entries for any
duplicate series records encountered during WAL replay, with the expiry set
to the highest current WAL segment number. Any subsequent WAL
checkpoints will see the duplicate series entry in the walExpiries map, and
keep the series record until the last WAL segment that could contain its
samples is deleted.
Other considerations:
WBL: series records aren't written to the WBL, so there are no duplicates to deal with
agent mode: has its own WAL replay logic that handles duplicate series records differently, and is outside the scope of this PR
Rationales:
* metadata-wal-records might be deprecated and replaced going forward: https://github.com/prometheus/prometheus/issues/15911
* PRW 2.0 works without metadata just fine (although it sends untyped metrics as expected).
Signed-off-by: bwplotka <bwplotka@gmail.com>
Remove the 2 minute timeout as the default is 2 hours and wouldn't
interfere. With the test. Otherwise the extra samples combined with
race detection can push the test over 2 minutes and make it fail.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
