upstream/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus.git synced 2026-03-21 18:10:42 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 3
prometheus/tsdb
History
Bartlomiej Plotka 2ba3046c80
Merge float st-storage implementation (PROM-60) and initial xor2-encoding (#18062) 
* feat(tsdb/chunkenc): add float chunk format with start timestamp support


Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* optimize code path and layout

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* make new format usable in head

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* fix issue with seeking to last sample again

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* fix iterator benchmark for chunks not supporting ST

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* reduce footprint of the xoroptst chunk iterator object

It was 80 bytes with a lot of padding compared to the 56 bytes of the
original xor chunk iterator. Made it 64 bytes, tightly packed.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Fix benchmark expectations on ST in interator

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* add inclusive delta test case

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* make testcases independent of order

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* drop unused code

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Drop commented out line

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* documentation

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Small simplification in the doc

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Add delta st inclusive test case for random vt

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Switch to delta of difference of st to prev t

from delta of delta of st.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Write ST after T and V so we can write a single bit on the second sample

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* verify chunk sample len function

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Reduce size of first st stored a little

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* test the case where st equals the t

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* add st equal t to bechmarks

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* test(chunkenc): test that appender can contonue chunks

Test that initializing a chunk appender from an existing chunk
works correctly.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* fix(chunkenc): bug in initializing appender on existing chunk

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Add cases with jitter in the start time as well


Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* tsdb: ST-in-WAL: Counter implementation and benchmarks (#17671)

Initial implementation of https://github.com/prometheus/prometheus/issues/17790.
Only implements ST-per-sample for Counters. Tests and benchmarks updated.

Note: This increases the size of the RefSample object for all users, whether st-per-sample is turned on or not.

Signed-off-by: Owen Williams <owen.williams@grafana.com>

* refactor: sed enableStStorage/enableSTStorage

Signed-off-by: bwplotka <bwplotka@gmail.com>

* feat[scrape]: add ST parsing support to scrape AppenderV2 flow (#18103)

Signed-off-by: bwplotka <bwplotka@gmail.com>

* feat(tsdb): change head opt EnableSTStorage to atomic (#18107)

In downstream projects this needs to be set dynamically per tenant.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Merge pull request #18108 from prometheus/bwplotka/fix

scrape: add tests for ST appending; add warnings for ST feature flag users around _created drop

* refact(tsdb): trivial rename (#18109)

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* fix(tsdb): missing passing head option to wal/wbl write (#18113)

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* feat(tsdb): allow using ST capable XOR chunks - retain format on read (#18013)

* feat(tsdb): allow appending to ST capable XOR chunk optionally

Only for float samples as of now.  Supports for in-order and out-of-order
samples.

Make sure that on readout the ST capable chunks are returned automatically.
When the chunks are returned as is, this is trivially true.
When a chunk needs to be re-coded due to deletion (tombstone) markers,
we take the encoding of the original chunk.
When a chunk needs to be created from overlapping chunks, we observe
whether ST is zero or not and create the new chunk based on that.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* fix test after merge

Signed-off-by: bwplotka <bwplotka@gmail.com>

* feat: RW2 sending ST support

Signed-off-by: bwplotka <bwplotka@gmail.com>

tmp

Signed-off-by: bwplotka <bwplotka@gmail.com>

* tests: test ST in a cheapest way possible

Signed-off-by: bwplotka <bwplotka@gmail.com>

* tests: add bench CLI recommended invokations

Signed-off-by: bwplotka <bwplotka@gmail.com>

* fixed tests after rebase

Signed-off-by: bwplotka <bwplotka@gmail.com>

* feat(chunkenc): replace xoroptst chunk encoding with xor2

XOR2 is based on https://github.com/prometheus/prometheus/pull/18238
With additional ST support.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* feat: add compliance RW sender test for agent

Signed-off-by: bwplotka <bwplotka@gmail.com>

* feat(agent): add support for appending ST

Signed-off-by: bwplotka <bwplotka@gmail.com>

* replace stray xoroptst words

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* post merge conflict fixes

Signed-off-by: bwplotka <bwplotka@gmail.com>

* feat(tsdb): register st_storage in feature API

Register the st-storage feature flag in the feature registry via
the TSDB options, consistent with how other TSDB features like
exemplar_storage and delayed_compaction are registered.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Coded with Claude Sonnet 4.6.

* Document xor2-encoding feature flag

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>

* Add xor2-encoding feature flag

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>

* Update CHANGELOG

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>

* Fix linting

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>

* Remove setting of xor2 encoding option in db open

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>

* Fix tests

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>

* Fix linting

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>

* Update feature flag description

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>

* Update comments and feature flag description

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>

* Update documentation for st-storage feature

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>

* st: disconnect st-storage with xor2-encoding given planned experiments (#18316)

* st: disconnect st-storage with xor2-encoding given planned experiments

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Update docs/feature_flags.md

Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Update docs/feature_flags.md

Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Update docs/feature_flags.md

Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Update docs/feature_flags.md

Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

---------

Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>

---------

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Signed-off-by: Aleksandr Smirnov <5targazer@mail.ru>
Signed-off-by: Mohammad Abbasi <mohammad.v184@gmail.com>
Signed-off-by: matt-gp <small_minority@hotmail.com>
Signed-off-by: Ian Kerins <git@isk.haus>
Signed-off-by: SuperQ <superq@gmail.com>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Signed-off-by: ffgan <sudoemt@gmail.com>
Signed-off-by: Patryk Prus <p@trykpr.us>
Signed-off-by: Owen Williams <owen.williams@grafana.com>
Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: 3Juhwan <13selfesteem91@naver.com>
Signed-off-by: Sammy Tran <sammyqtran@gmail.com>
Signed-off-by: Casie Chen <casie.chen@grafana.com>
Signed-off-by: Dan Cech <dcech@grafana.com>
Signed-off-by: kakabisht <kakabisht07@gmail.com>
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: Divyansh Mishra <divyanshmishra@Divyanshs-MacBook-Air-3.local>
Signed-off-by: Varun Chawla <varun_6april@hotmail.com>
Signed-off-by: Martin Valiente Ainz <64830185+tinitiuset@users.noreply.github.com>
Signed-off-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
Signed-off-by: Björn Rabenstein <github@rabenste.in>
Signed-off-by: beorn7 <beorn@grafana.com>
Signed-off-by: Sayuru <71478576+samaras3@users.noreply.github.com>
Signed-off-by: Matt <small_minority@hotmail.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
Co-authored-by: Bryan Boreham <bjboreham@gmail.com>
Co-authored-by: Sasha <103973965+crush-on-anechka@users.noreply.github.com>
Co-authored-by: Mohammad Abbasi <mohammad.v184@gmail.com>
Co-authored-by: matt-gp <small_minority@hotmail.com>
Co-authored-by: Ian Kerins <git@isk.haus>
Co-authored-by: SuperQ <superq@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Julien <291750+roidelapluie@users.noreply.github.com>
Co-authored-by: ffgan <sudoemt@gmail.com>
Co-authored-by: Patryk Prus <p@trykpr.us>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Joe Adams <github@joeadams.io>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Owen Williams <owen.williams@grafana.com>
Co-authored-by: 3Juhwan <13selfesteem91@naver.com>
Co-authored-by: Casie Chen <casie.chen@grafana.com>
Co-authored-by: Dan Cech <dcech@grafana.com>
Co-authored-by: hridyesh bisht <41201308+kakabisht@users.noreply.github.com>
Co-authored-by: zenador <zenador@users.noreply.github.com>
Co-authored-by: Divyansh Mishra <divyanshmishra@Divyanshs-MacBook-Air-3.local>
Co-authored-by: Varun Chawla <varun_6april@hotmail.com>
Co-authored-by: Martin Valiente Ainz <64830185+tinitiuset@users.noreply.github.com>
Co-authored-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>
Co-authored-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Co-authored-by: Linas Medžiūnas <linasm@users.noreply.github.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
Co-authored-by: beorn7 <beorn@grafana.com>
Co-authored-by: Sayuru <71478576+samaras3@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Carrie Edwards <edwrdscarrie@gmail.com>
2026-03-19 10:51:40 +01:00
..
agent Merge float st-storage implementation (PROM-60) and initial xor2-encoding (#18062) 2026-03-19 10:51:40 +01:00
chunkenc Fix linting 2026-03-12 11:07:02 -07:00
chunks Merge branch 'main' into st-main-sync2 2026-02-25 13:54:25 +00:00
compression tsdb: ST-in-WAL: Counter implementation and benchmarks (#17671) 2026-02-12 13:17:50 -05:00
docs feat(chunkenc): replace xoroptst chunk encoding with xor2 2026-03-06 14:35:06 +01:00
encoding Remove copyright date from headers (#17785) 2026-01-05 13:46:21 +01:00
fileutil chore: fix emptyStringTest issues from gocritic (#18226) 2026-03-04 08:24:50 +01:00
goversion Remove copyright date from headers (#17785) 2026-01-05 13:46:21 +01:00
index tsdb: Optimize LabelValues for sparse intersections (Fixes #14551) 2026-02-17 13:35:31 -08:00
record Merge branch 'main' into start-time-main-sync 2026-03-12 08:28:15 +00:00
testdata tsdb: Delete blocks atomically; Remove tmp blocks on start; Added test. (#7772) 2020-08-11 06:56:08 +01:00
tombstones Refactor various tsdb sub-packages (#17847) 2026-01-13 13:38:58 +00:00
tsdbutil tsdb/wlog: Remove any temproary checkpoints when creating a Checkpoint (#17598) 2026-02-17 09:23:54 +01:00
wlog post merge conflict fixes 2026-03-12 09:03:08 +00:00
.gitignore Moving tsdb into its own subdirectory 2019-08-13 13:58:49 +05:30
block.go Merge pull request #18180 from atilsensalduz/bugfix/tsdb-writeMetaFile-fd-leak 2026-02-24 12:25:00 +00:00
block_test.go auto update head sample use in tests 2026-01-14 13:15:13 +01:00
blockwriter.go Remove copyright date from headers (#17785) 2026-01-05 13:46:21 +01:00
blockwriter_test.go Remove copyright date from headers (#17785) 2026-01-05 13:46:21 +01:00
CHANGELOG.md Rename default branch to main 2021-02-22 20:28:02 +01:00
compact.go tsdb: Migrate multi-errors to errors package (#17768) 2026-02-04 10:41:57 +01:00
compact_test.go chore(lint): enable wg.Go 2026-02-17 15:21:51 +01:00
db.go Remove setting of xor2 encoding option in db open 2026-03-12 11:08:33 -07:00
db_append_v2_test.go Fix linting 2026-03-12 12:26:42 -07:00
db_test.go Merge remote-tracking branch 'origin/main' into feature/start-time 2026-03-02 14:47:44 -05:00
example_test.go Remove copyright date from headers (#17785) 2026-01-05 13:46:21 +01:00
exemplar.go tsdb: fix rare case of OOO exemplar insertion pointing to the exemplar being deleted (#17865) 2026-01-15 08:49:37 +01:00
exemplar_test.go chore: enable staticcheck linter and update golangci-lint to 2.10.1 2026-02-18 15:58:16 +00:00
head.go Update comments and feature flag description 2026-03-13 07:43:28 -07:00
head_append.go Add xor2-encoding feature flag 2026-03-12 11:07:00 -07:00
head_append_v2.go Add xor2-encoding feature flag 2026-03-12 11:07:00 -07:00
head_append_v2_test.go Add xor2-encoding feature flag 2026-03-12 11:07:00 -07:00
head_bench_test.go tsdb: add auto-cleanup to newTestHead and remove redundant cleanup calls (#17890) 2026-01-19 12:57:05 +01:00
head_dedupelabels.go Remove copyright date from headers (#17785) 2026-01-05 13:46:21 +01:00
head_other.go Remove copyright date from headers (#17785) 2026-01-05 13:46:21 +01:00
head_read.go tsdb: Add StaleHead and GC for stale series in the Head block 2026-01-23 17:59:41 -08:00
head_read_test.go feat(tsdb): allow using ST capable XOR chunks - retain format on read (#18013) 2026-02-20 09:15:51 +01:00
head_test.go Merge branch 'main' into feature/start-time 2026-03-17 13:06:25 +01:00
head_wal.go Merge float st-storage implementation (PROM-60) and initial xor2-encoding (#18062) 2026-03-19 10:51:40 +01:00
isolation.go Remove copyright date from headers (#17785) 2026-01-05 13:46:21 +01:00
isolation_test.go chore(lint): enable wg.Go 2026-02-17 15:21:51 +01:00
label_values_bench_test.go tsdb: Optimize LabelValues for sparse intersections (Fixes #14551) 2026-02-17 13:35:31 -08:00
mocks_test.go Remove copyright date from headers (#17785) 2026-01-05 13:46:21 +01:00
ooo_head.go Add xor2-encoding feature flag 2026-03-12 11:07:00 -07:00
ooo_head_read.go Add xor2-encoding feature flag 2026-03-12 11:07:00 -07:00
ooo_head_read_test.go tsdb: add auto-cleanup to newTestHead and remove redundant cleanup calls (#17890) 2026-01-19 12:57:05 +01:00
ooo_head_test.go Add xor2-encoding feature flag 2026-03-12 11:07:00 -07:00
ooo_isolation.go Remove copyright date from headers (#17785) 2026-01-05 13:46:21 +01:00
ooo_isolation_test.go Remove copyright date from headers (#17785) 2026-01-05 13:46:21 +01:00
querier.go feat(tsdb): allow using ST capable XOR chunks - retain format on read (#18013) 2026-02-20 09:15:51 +01:00
querier_bench_test.go Remove copyright date from headers (#17785) 2026-01-05 13:46:21 +01:00
querier_test.go feat(tsdb): allow using ST capable XOR chunks - retain format on read (#18013) 2026-02-20 09:15:51 +01:00
README.md Fixed broken link in tsdb README.md 2022-10-07 16:20:20 +00:00
repair.go tsdb: Migrate multi-errors to errors package (#17768) 2026-02-04 10:41:57 +01:00
repair_test.go Remove copyright date from headers (#17785) 2026-01-05 13:46:21 +01:00
testutil.go Remove copyright date from headers (#17785) 2026-01-05 13:46:21 +01:00
tsdbblockutil.go Remove copyright date from headers (#17785) 2026-01-05 13:46:21 +01:00

README.md

TSDB

GoPkg

This directory contains the Prometheus TSDB (Time Series DataBase) library, which handles storage and querying of all Prometheus v2 data.

Documentation

External resources

A series of blog posts explaining different components of TSDB: