1499 commits

Author	SHA1	Message	Date
bwplotka	09cb9bbd77	fix: fix OOO appendFloat; reuse logic between v1 and v2 ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-01-27 13:53:59 +00:00
bwplotka	9c7561fb6a	fix: optimizations to AppenderV2 ... Signed-off-by: bwplotka <bwplotka@gmail.com> fix: 48B alloc optimization to AppenderV2 Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-01-27 12:32:18 +00:00
Ganesh Vernekar	3e4a094dbb	Add stale_series_compaction_threshold config file option ... Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>	2026-01-23 18:12:34 -08:00
Ganesh Vernekar	4f3de8da29	tsdb: Add unit tests for stale series compaction ... Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>	2026-01-23 18:07:34 -08:00
Ganesh Vernekar	43dc23afe7	tsdb: Clear stale series from the Head during WAL replay ... Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>	2026-01-23 18:02:45 -08:00
Ganesh Vernekar	43e69388df	tsdb: Add stale series compaction support in the DB ... Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>	2026-01-23 18:00:08 -08:00
Ganesh Vernekar	9b444b57af	tsdb: Add StaleHead and GC for stale series in the Head block ... Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>	2026-01-23 17:59:41 -08:00
Bartlomiej Plotka	bec70227f1	feat(scrape)[PART5b]: Add AppenderV2 support to scrape.NewManager constructor (#17872) ... * feat(scrape)[PART5b]: Add AppenderV2 support to scrape.NewManager optionally to V1 Signed-off-by: bwplotka <bwplotka@gmail.com> * Update scrape/manager.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * fixes after rebase Signed-off-by: bwplotka <bwplotka@gmail.com> * Apply suggestions from code review Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>	2026-01-23 09:04:05 +00:00
yy	0e77dcfdb9	fix: typo in error message (#17878) ... Signed-off-by: Yohei Yamamoto <yhymmt123@gmail.com>	2026-01-20 12:14:02 +01:00
Arve Knudsen	572f247b4d	tsdb: add auto-cleanup to newTestHead and remove redundant cleanup calls (#17890) ... Add automatic cleanup to newTestHeadWithOptions so that heads created with newTestHead are automatically closed when the test ends. This simplifies test code by removing the need for manual cleanup in most cases. Changes: - Add t.Cleanup in newTestHeadWithOptions immediately after creating the head, using _ = h.Close() to handle double-close gracefully - Remove redundant t.Cleanup, defer, and explicit Close calls from tests that use newTestHead - Add cleanup for heads created with NewHead directly in restart patterns (e.g., restartHeadAndVerifySeriesCounts, startHead) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2026-01-19 12:57:05 +01:00
Arve Knudsen	dd85d7ca97	tsdb: fix memory leaks in buffer pools (#17879) ... * tsdb: fix memory leaks in buffer pools Clear reference fields when returning buffers to pools to avoid retaining data after the buffer is reused. Affected pools: - refSeriesPool: clear Labels - histogramsPool: clear H pointer - floatHistogramsPool: clear FH pointer - metadataPool: clear Unit and Help strings Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Clear slices Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Clear metadata buffer Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Remove tests Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2026-01-19 11:14:31 +00:00
Bartlomiej Plotka	c4b0da94db	Merge pull request #17835 from prometheus/bwplotka/a2-storage-support ... feat(storage)[PART4b]: add AppenderV2 to the rest of storage.Storage implementations + mock exemplar fix	2026-01-16 13:04:14 +00:00
George Krajcsovits	ef350b2b54	Merge pull request #17840 from prometheus/krajo/st-in-chunks ... feat(tsdb): new Append parameter and AtST interface for chunks	2026-01-16 10:32:53 +01:00
Julius Hinze	e49d11c88d	tsdb: fix rare case of OOO exemplar insertion pointing to the exemplar being deleted (#17865) ... Signed-off-by: Julius Hinze <julius.hinze@grafana.com>	2026-01-15 08:49:37 +01:00
Julius Hinze	ccb7468b09	tsdb: fix grow/shrink nextIndex calculation (#17863) ... Signed-off-by: Julius Hinze <julius.hinze@grafana.com>	2026-01-14 16:44:50 +01:00
George Krajcsovits	06a59346fe	Update tsdb/chunkenc/chunk.go ... Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com> Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>	2026-01-14 16:41:03 +01:00
bwplotka	49c3aea56d	feat(storage)[PART4b]: add AppenderV2 to the rest of storage.Storage impl ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-01-14 13:54:27 +00:00
György Krajcsovits	adf734db7a	update remaining tests ... Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2026-01-14 13:15:16 +01:00
György Krajcsovits	a5ac0bff1d	update ooo_head.go but only with TODOs ... Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2026-01-14 13:15:15 +01:00
György Krajcsovits	28dca34f4f	auto update head sample use in tests ... find . -name "*.go" -type f -exec sed -E -i \ 's/([^[:alpha:]]sample\{)([^,{:]+,[^,]+,[^,]+,[^,]+\})/\10, \2/g' {} + I've omitted tsdb/ooo_head.go from the commit because I'm also adding todo there. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2026-01-14 13:15:13 +01:00
György Krajcsovits	f616689f09	tsdb/head.go: add start timestamp to sample type used in tests ... Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2026-01-14 13:15:12 +01:00
György Krajcsovits	70bc06718d	feat(tsdb): new AppenderV2 and AtST interface for chunks ... No implementation yet. Just to test the shape of the interface. AtST is implemented for trivial cases, anything else is hard coded to return 0. Ref: https://github.com/prometheus/prometheus/issues/17791 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2026-01-14 13:15:09 +01:00
Ben Kochie	72a23934ad	Refactor various tsdb sub-packages (#17847) ... Migrate various tsdb related packages from `tsdb/errors` to the standard library `errors` package. Signed-off-by: SuperQ <superq@gmail.com>	2026-01-13 13:38:58 +00:00
Bryan Boreham	da754cd240	Merge remote-tracking branch 'origin/release-3.9' into merge-3.9-into-main	2026-01-07 15:51:44 +00:00
Bryan Boreham	f1719fa1d4	[BUGFIX] Agent: fix crash from invalid type in pool (#17802) ... We have separate pools for Appender and AppenderV2 objects, and must not put another kind of object into them. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2026-01-07 14:01:02 +00:00
Julius Hinze	22463b1e9f	tsdb: add support for OOO exemplars in CircularExemplarStorage (#17469) ... * tsdb: add support for OOO exemplars in CircularExemplarStorage Doubly linked exemplar storage resize. Split exemplar buffer resize into shrink and grow functions. Skip duplicate OOO exemplars, re-initialize emptied index after deleting its last exemplar. Signed-off-by: Julius Hinze <julius.hinze@grafana.com>	2026-01-07 13:25:50 +01:00
Patryk Prus	dcda4840a7	tsdb/index: export sentinel errors for size limit failures (#17773) ... * tsdb/index: export sentinel errors for size limit failures --------- Signed-off-by: Patryk Prus <p@trykpr.us> Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>	2026-01-06 14:07:23 +01:00
Ben Kochie	e14795bbf4	Remove copyright date from headers (#17785) ... Remove copyright dates from various files as part of [PROM-50]. [PROM-50]: https://github.com/prometheus/proposals/blob/main/proposals/0050-remove-copyright-dates.md Signed-off-by: SuperQ <superq@gmail.com>	2026-01-05 13:46:21 +01:00
Jarkko Pöyry	e4b6d443fc	tsdb: fix handle leak on mmap failure on MS Windows (#17310) ... Signed-off-by: Jarkko Pöyry <jarkko.poyry@metaplay.io>	2025-12-21 20:55:02 +01:00
Bartlomiej Plotka	1c0537dc02	Merge pull request #17677 from prometheus/bwplotka/a2-agent ... refactor(tsdb/agent)[PART3]: add AppenderV2 support to agent	2025-12-17 14:27:51 +00:00
bwplotka	96ff5b8f9c	addressed comment ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-17 05:47:28 +00:00
bwplotka	bab7614d1b	fix: ensure no race on lset ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-17 05:39:07 +00:00
bwplotka	fc019d6628	refactor(scrape): DRY-ed getOrCreate flow ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-17 05:29:30 +00:00
Charles Korn	69aa3ac67b	tsdb: correctly log error in headAppenderV2.appendExemplars ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2025-12-17 10:04:14 +11:00
NamanParlecha	c94101d023	TSDB: Option to configure TSDB Block Reload Interval (#16728) ... Add --storage.tsdb.block-reload-interval flag to configure TSDB block reload interval. --------- Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> Signed-off-by: NamanParlecha <namanparlecha@gmail.com> Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>	2025-12-15 09:31:17 +01:00
bwplotka	ad367b504b	refactor(tsdb/agent)[PART3]: add AppenderV2 support to agent ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-13 05:17:39 +00:00
bwplotka	763b935b45	refactor(tsdb/agent): 1:1 copy of db.go and db_test.go for starting point ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-12 15:20:42 +00:00
Bryan Boreham	41665a4a55	[CHORE] TSDB: Remove unused LabelValueFor function ... The last use of this was removed 4 years ago. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2025-12-12 13:44:13 +00:00
Bryan Boreham	583bc01cc9	Refactor: TSDB: small improvement to Postings (#17661) ... * [TESTS] TSDB: Check that ListPostings are sorted `newListPostings` is a convenient place to do this; move it into the test file because it's not needed for anything else. Also simplify the existing `SliceIsSorted` check. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * [COMMENTS] TSDB: Document that Postings are ordered. The description of `Seek()` implies they are, but it's better to be explicit. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * [REFACTOR] Unexport ListPostings type It's only used within the one package, and it would be surprising if some downstream code did rely on this type, given all of its members are unexported. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> --------- Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2025-12-11 08:14:35 +00:00
Julien	f73aba34cd	Merge pull request #17427 from roidelapluie/roidelapluie/ffapi ... API: Add a /api/v1/features endpoint	2025-12-10 10:14:03 +01:00
Julien Pivotto	a5671a002f	API: Add a /api/v1/features endpoint ... Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>	2025-12-09 16:13:14 +01:00
Bartlomiej Plotka	be419d80dc	Merge pull request #17629 from prometheus/bwplotka/a2-tsdb ... refactor(appenderV2)[PART1]: add AppenderV2 interface; add TSDB AppenderV2 implementation	2025-12-09 11:41:00 +00:00
bwplotka	e7e45090e4	refactor(appenderV2): port TSDB non-head tests ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-09 10:39:45 +00:00
bwplotka	0b70a07572	refactor(appenderV2): add TSDB AppenderV2 implementation ... Signed-off-by: bwplotka <bwplotka@gmail.com> tmp Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-09 10:39:43 +00:00
bwplotka	a41e1144dd	refactor(appenderV2): 1:1 copy of db_test.go to db_append_v2_test.go (starting point) ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-09 09:53:39 +00:00
bwplotka	efdfb8fed6	refactor(appenderV2): 1:1 copy of head append test files for v2 (starting point) ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-09 09:53:39 +00:00
bwplotka	129650df9d	refactor(appenderV2): 1:1 copy of head_append.go -> head_append_v2.go (starting point) ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-09 09:53:39 +00:00
dongjiang	3239723098	Update golangci-lint and add modernize check (#17640) ... * add modernize check Signed-off-by: dongjiang1989 <dongjiang1989@126.com> * fix golangci lint Signed-off-by: dongjiang1989 <dongjiang1989@126.com> --------- Signed-off-by: dongjiang1989 <dongjiang1989@126.com>	2025-12-05 09:29:10 +01:00
Bartlomiej Plotka	f6ca7145ca	refactor(tsdb): use one test newTestDB constructor (#17638) ... For tests only, we had various ways of opening DB. Reduced to one instead of: * Open * newTestDB * newTestDBOpts * openTestDB This so https://github.com/prometheus/prometheus/pull/17629 is smaller and bit easier. Also for test maintainability and consistency. Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-03 07:55:48 +00:00
Łukasz Mierzwa	8a1086a128	feat: Add flag that blocks lvl 1 compactions until upload is confirmed in an external JSON file (#17435) ... * Delay compactions until Thanos uploads all blocks Using Thanos sidecar with Prometheus requires us to disable TSDB compactions on Prometheus side by setting --storage.tsdb.min-block-duration and --storage.tsdb.max-block-duration to the same value. See https://thanos.io/tip/components/sidecar.md. The main problem this avoids is that Prometheus might compact given block before Thanos uploads it, creating a gap in Thanos metrics. Thanos does not upload compacted blocks because that would upload the same sample multiple times. You can tell Thanos to upload compacted blocks but that is aimed at one time migrations. This patch creates a bridge between Thanos and Prometheus by allowing Prometheus to read the shipper file Thanos creates, where it tracks which blocks were already uploaded, and using that data delays compaction of blocks until they are marked as uploaded by Thanos. Thanks to this both services can coordinate with each other (in a way) and we can stop disabling compaction on Prometheus side when Thanos uploads are enabled. The reason to have this is that disabling compactions have very dramatic performance cost. Since most time series exist for longer than a single block duration (2h by default) large chunks of block index will reference the same series, so 10 * 2h blocks will each have an index that is usually fairly big and is almost the same for all 10 blocks. Compaction de-duplicates the index so merging 10 blocks together would leave us with a single index that is around the same size as each of these 10 2h blocks would have (plus some extra for series that only exists in some blocks, but not all). Every range query that iterates over all 10 blocks would then have to read each index and so we're doing 10x more work then if we had a single compacted block. Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> * Rename structs and functions to make this more generic Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> * Address review comments Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> * Cache UploadMeta for 1 minute Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> --------- Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com>	2025-12-02 10:39:45 +00:00