982 commits

Author	SHA1	Message	Date
Bartlomiej Plotka	2597a12080	st: Add a hidden 'st-storage' feature flag for PROM-60 (#17907) ... Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	2026-01-28 09:05:54 +00:00
Arve Knudsen	a5f86c3fb6	cmd/prometheus: fix flaky TestQueryLog race condition (#17933) ... Add waitForQueryLog helper that polls for query log entries to appear before asserting, rather than reading the file immediately after making a query. This fixes a race condition where the query log wasn't flushed to disk before the test read the file. The helper uses a 5 second timeout with 100ms polling intervals, which is generous enough to handle slow CI environments while keeping the test responsive. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2026-01-27 09:02:16 +00:00
Ganesh Vernekar	3e4a094dbb	Add stale_series_compaction_threshold config file option ... Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>	2026-01-23 18:12:34 -08:00
Bartlomiej Plotka	bec70227f1	feat(scrape)[PART5b]: Add AppenderV2 support to scrape.NewManager constructor (#17872) ... * feat(scrape)[PART5b]: Add AppenderV2 support to scrape.NewManager optionally to V1 Signed-off-by: bwplotka <bwplotka@gmail.com> * Update scrape/manager.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * fixes after rebase Signed-off-by: bwplotka <bwplotka@gmail.com> * Apply suggestions from code review Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>	2026-01-23 09:04:05 +00:00
Bartlomiej Plotka	0d116b0994	tests(teststorage): Close Storage in the helper (#17902) ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-01-23 08:41:35 +00:00
wangyazhou	629bd3c5d5	Fix logger syntax for timingsLogger (#17870) ... Fix logger syntax for timingsLogger Signed-off-by: wangyazhou <564631192@qq.com>	2026-01-20 12:15:42 +01:00
Julius Volz	1d3d98ed16	Merge pull request #17644 from prometheus/binop-fill-modifier ... PromQL: Add `fill*()` binop modifiers to provide default values for missing series	2026-01-19 20:05:52 +01:00
Julius Volz	d3b6e61487	Put binop fill modifiers behind a feature flag ... Signed-off-by: Julius Volz <julius.volz@gmail.com>	2026-01-15 11:18:48 +01:00
Julius Volz	d6aa6a3fc7	Add new fill modifiers to features test data ... Signed-off-by: Julius Volz <julius.volz@gmail.com>	2026-01-15 07:56:23 +01:00
bwplotka	49c3aea56d	feat(storage)[PART4b]: add AppenderV2 to the rest of storage.Storage impl ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-01-14 13:54:27 +00:00
Ben Kochie	d9ed026658	Refractor promtool errors (#17842) ... Replace use of `tsdb/errors` with standard library `errors`. Signed-off-by: SuperQ <superq@gmail.com>	2026-01-13 06:35:11 +00:00
Arthur Silva Sens	1e317d0098	Add configuration option to control extra-scrape-metrics (#17606)	2026-01-06 09:00:49 -03:00
Ben Kochie	e14795bbf4	Remove copyright date from headers (#17785) ... Remove copyright dates from various files as part of [PROM-50]. [PROM-50]: https://github.com/prometheus/proposals/blob/main/proposals/0050-remove-copyright-dates.md Signed-off-by: SuperQ <superq@gmail.com>	2026-01-05 13:46:21 +01:00
Aditya Tiwari	146080186d	promtool: Add --lint flag to check metrics command to allow disabling linting (#17669) ... * promtool: allow cardinality with metrics linting and add --lint to check metrics Signed-off-by: ADITYA TIWARI <adityatiwari342005@gmail.com> * fix/ci: Simplify test case variable declaration Remove unnecessary variable declaration in test cases. Signed-off-by: ADITYA TIWARI <142050150+ADITYATIWARI342005@users.noreply.github.com> * promtool: avoid Tee for --lint=none Signed-off-by: ADITYA TIWARI <adityatiwari342005@gmail.com> * promtool: validate at least one feature enabled in check metrics addresses feedback to ensure the command does something useful now fails with clear error when both --lint=none and no --extended flag. Signed-off-by: ADITYA TIWARI <adityatiwari342005@gmail.com> --------- Signed-off-by: ADITYA TIWARI <adityatiwari342005@gmail.com> Signed-off-by: ADITYA TIWARI <142050150+ADITYATIWARI342005@users.noreply.github.com>	2025-12-18 10:19:06 +00:00
Bartlomiej Plotka	1c0537dc02	Merge pull request #17677 from prometheus/bwplotka/a2-agent ... refactor(tsdb/agent)[PART3]: add AppenderV2 support to agent	2025-12-17 14:27:51 +00:00
Bryan Boreham	6f18cc3074	Merge pull request #13409 from smallfish/feature/promtool-dump-series ... promtool: add dump-series	2025-12-16 11:51:06 +00:00
NamanParlecha	c94101d023	TSDB: Option to configure TSDB Block Reload Interval (#16728) ... Add --storage.tsdb.block-reload-interval flag to configure TSDB block reload interval. --------- Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> Signed-off-by: NamanParlecha <namanparlecha@gmail.com> Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>	2025-12-15 09:31:17 +01:00
bwplotka	ad367b504b	refactor(tsdb/agent)[PART3]: add AppenderV2 support to agent ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-13 05:17:39 +00:00
Julien	f73aba34cd	Merge pull request #17427 from roidelapluie/roidelapluie/ffapi ... API: Add a /api/v1/features endpoint	2025-12-10 10:14:03 +01:00
Julien Pivotto	a5671a002f	API: Add a /api/v1/features endpoint ... Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>	2025-12-09 16:13:14 +01:00
dongjiang	3239723098	Update golangci-lint and add modernize check (#17640) ... * add modernize check Signed-off-by: dongjiang1989 <dongjiang1989@126.com> * fix golangci lint Signed-off-by: dongjiang1989 <dongjiang1989@126.com> --------- Signed-off-by: dongjiang1989 <dongjiang1989@126.com>	2025-12-05 09:29:10 +01:00
Julien	0279e14d4a	Merge pull request #17636 from roidelapluie/roidelapluie/starttime ... Add start_timestamp field for unit tests	2025-12-04 10:47:24 +01:00
Julien Pivotto	4620c8ac71	Simplify StartTime assignment in unit test setup. ... Remove redundant IsZero check since promqltest.LazyLoader already handles zero StartTime by defaulting to Unix epoch. Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>	2025-12-04 10:14:09 +01:00
Jan Fajerski	ac5bdd66d7	Merge pull request #17634 from jan--f/sync-release-3.8 ... Sync branch release-3.8	2025-12-03 21:26:45 +01:00
Julien Pivotto	1a853e23db	Add start_timestamp field for unit tests. ... This commit adds support for configuring a custom start timestamp for Prometheus unit tests, allowing tests to use realistic timestamps instead of starting at Unix epoch 0. Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>	2025-12-03 11:20:23 +01:00
Jan Fajerski	fae20b73ff	Merge branch 'release-3.8' into sync-release-3.8 ... Conflicts: storage/remote/write_handler.go storage/remote/write_handler_test.go Pick `main` Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	2025-12-02 14:24:53 +01:00
Łukasz Mierzwa	8a1086a128	feat: Add flag that blocks lvl 1 compactions until upload is confirmed in an external JSON file (#17435) ... * Delay compactions until Thanos uploads all blocks Using Thanos sidecar with Prometheus requires us to disable TSDB compactions on Prometheus side by setting --storage.tsdb.min-block-duration and --storage.tsdb.max-block-duration to the same value. See https://thanos.io/tip/components/sidecar.md. The main problem this avoids is that Prometheus might compact given block before Thanos uploads it, creating a gap in Thanos metrics. Thanos does not upload compacted blocks because that would upload the same sample multiple times. You can tell Thanos to upload compacted blocks but that is aimed at one time migrations. This patch creates a bridge between Thanos and Prometheus by allowing Prometheus to read the shipper file Thanos creates, where it tracks which blocks were already uploaded, and using that data delays compaction of blocks until they are marked as uploaded by Thanos. Thanks to this both services can coordinate with each other (in a way) and we can stop disabling compaction on Prometheus side when Thanos uploads are enabled. The reason to have this is that disabling compactions have very dramatic performance cost. Since most time series exist for longer than a single block duration (2h by default) large chunks of block index will reference the same series, so 10 * 2h blocks will each have an index that is usually fairly big and is almost the same for all 10 blocks. Compaction de-duplicates the index so merging 10 blocks together would leave us with a single index that is around the same size as each of these 10 2h blocks would have (plus some extra for series that only exists in some blocks, but not all). Every range query that iterates over all 10 blocks would then have to read each index and so we're doing 10x more work then if we had a single compacted block. Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> * Rename structs and functions to make this more generic Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> * Address review comments Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> * Cache UploadMeta for 1 minute Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> --------- Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com>	2025-12-02 10:39:45 +00:00
Ayoub Mrini	e7999528fa	fix(test): make TestRemoteWrite_ReshardingWithoutDeadlock more reliable and re-enable it (#17490) ... Improve test stability by waiting for the relevant metrics to appear on /metrics before the first check on the desired shard count. Increase the scrape interval to avoid timeouts, as 100 ms may be insufficient for Prometheus to scrape itself in some environments (e.g., CI). Have Prometheus scrape itself multiple times to increase the volume of data sent and help fill the queue more quickly. Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2025-11-25 21:30:42 +00:00
Jan Fajerski	26f8c92de8	test: skip TestRemoteWrite_ReshardingWithoutDeadlock temporarily as flaky (#17534) (#17543) ... (cherry picked from commit 35c3232a2e) Signed-off-by: machine424 <ayoubmrini424@gmail.com> Signed-off-by: Jan Fajerski <jfajersk@redhat.com> Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com>	2025-11-17 15:00:12 +00:00
beorn7	be4efd740c	cmd: Make feature flag native-histograms a no-op. ... Signed-off-by: beorn7 <beorn@grafana.com>	2025-11-16 23:22:02 +01:00
Ayoub Mrini	35c3232a2e	test: skip TestRemoteWrite_ReshardingWithoutDeadlock temporarily as flaky (#17534) ... Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2025-11-14 08:42:20 +00:00
Bartlomiej Plotka	f50ff0a40a	feat: rename CreatedTimestamp to StartTimestamp (#17523) ... Partially fixes https://github.com/prometheus/prometheus/issues/17416 by renaming all CT* names to ST* in the whole codebase except RW2 (this is done in separate [PR](https://github.com/prometheus/prometheus/pull/17411)) and PrometheusProto exposition proto. ``` CreatedTimestamp -> StartTimestamp CreatedTimeStamp -> StartTimestamp created_timestamp -> start_timestamp CT -> ST ct -> st ``` Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-11-13 14:17:51 +00:00
Bryan Boreham	e17742902b	lint ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2025-11-10 18:28:13 +00:00
Bryan Boreham	11cf858166	Small refactor: use clearer name formatSeriesSetLabelsToJSON ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2025-11-10 18:18:08 +00:00
Bryan Boreham	89b92f7880	[TEST] Promtool: test dump series together with dump samples ... Add data files for series tests. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2025-11-10 18:18:08 +00:00
Bryan Boreham	beea578b20	Make dump-series a --format flag on the dump command ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2025-11-10 18:07:23 +00:00
smallfish	5560397a70	promtool: add dump-series ... Signed-off-by: smallfish <smallfish.xy@gmail.com>	2025-11-10 17:13:33 +00:00
Ben Kochie	204249fcb5	Update golangci-lint (#17478) ... * Update golangci-lint to v2.6.0 * Fixup various linting issues. * Fixup deprecations. * Add exception for `labels.MetricName` deprecation. Signed-off-by: SuperQ <superq@gmail.com>	2025-11-05 13:47:34 +01:00
Minh Nguyen	30992dd032	[RW2] Fix: Only update metadata to WAL when metadata-wal-records feature is enabled (#17470) ... * add feature check when UpdateMetadata Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * add appendMetadata boolean to write_hander Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> --------- Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	2025-11-04 08:16:57 +00:00
Ben Kochie	48956f60d7	Update modernize (#17471) ... Apply additional Go modernize tool improvements. Signed-off-by: SuperQ <superq@gmail.com>	2025-11-04 05:13:49 +00:00
Minh Nguyen	9f93c2d2e1	promtool: Add Remote Write 2.0 support to push metrics command (#17417) ... * add feature flag for remote write v2 Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * change from number to protobuf_message Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix test Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix name Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * run make cli-documentation Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix help Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * run make cli-documentation Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> --------- Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	2025-10-31 11:38:40 +00:00
György Krajcsovits	b8192127ee	Merge remote-tracking branch 'origin/release-3.7' into krajo/merge-3.7.3-to-main ... # Conflicts: # CHANGELOG.md # storage/remote/queue_manager_test.go	2025-10-30 09:21:25 +01:00
Ayoub Mrini	6806b68f93	[release-3.7] fix: Remote-write: revert changes in the queue resharding logic (#17412) ... * Revert "chore: deprecate prometheus_remote_storage_{samples,exemplars,histograms}_in_total and prometheus_remote_storage_highest_timestamp_in_seconds" This reverts commit ba14bc49db. Signed-off-by: machine424 <ayoubmrini424@gmail.com> * Revert "storage/remote: compute highestTimestamp and dataIn at QueueManager level" This reverts commit 184c7eb918. Signed-off-by: machine424 <ayoubmrini424@gmail.com> * fix(remote-write): bring back the per queue metrics Signed-off-by: machine424 <ayoubmrini424@gmail.com> * test(remote): add TestRemoteWrite_ReshardingWithoutDeadlock to reproduce the sharding scale up deadlock Signed-off-by: machine424 <ayoubmrini424@gmail.com> --------- Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2025-10-29 14:04:09 +00:00
György Krajcsovits	bb8b611d2c	Merge remote-tracking branch 'origin/release-3.7' into krajo/merge-release-3.7-pre3-to-main ... Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2025-10-29 14:32:42 +01:00
Arve Knudsen	92ccadc96d	promtool: generate rulesLintConfig when --lint=none is set (#17414) ... * Correctly generate rulesLintConfig when --lint=none is set (#17399) If I run promtool check config --lint=none I get: ``` Checking rules.yml FAILED: rules.yml: unset nameValidationScheme ``` This is because passing --lint=none stops newConfigLintConfig from generating rulesLintConfig which is needed for validation. It means that defaults are used then, one of which is unset value for metric name validation, causing this error. Fix this by handling --lint=none case correctly and still generating rulesLintConfig. --------- Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	2025-10-28 14:34:52 +01:00
Łukasz Mierzwa	0e187f5542	Correctly generate rulesLintConfig when --lint=none is set (#17399) ... If I run promtool check config --lint=none I get: ``` Checking rules.yml FAILED: rules.yml: unset nameValidationScheme ``` This is becuase passing --lint=none stops newConfigLintConfig from generating rulesLintConfig which is needed for validation. It means that defaults are used then, one of which is unset value for metric name validation, causing this error. Fix this by handling --lint=none case correctly and still generating rulesLintConfig. Fixes #17398. Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com>	2025-10-28 13:07:10 +01:00
Minh Nguyen	ad4b59c504	tsdb: Deprecate retention flags; add tsdb.retention runtime configuration (#17026) ... * Move storage from CL to config file Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * Fix .md Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * run make cli-documentation Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * run make cli-documentation Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * nit_fixed Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * add test and update configuration.md Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix lint Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> --------- Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	2025-10-27 14:51:33 +00:00
Minh Nguyen	6bb367970e	feat(promtool): add RW2 support to promtool push metrics using client_golang library (#17280) ... * Add WriteProto method and tests for promtool metrics This commit adds: 1. WriteProto method to storage/remote/client.go that handles marshaling and compression of protobuf messages 2. Updated parseAndPushMetrics in cmd/promtool/metrics.go to use the new WriteProto method 3. Comprehensive tests for PushMetrics functionality The WriteProto method provides a cleaner API for sending protobuf messages without manually handling marshaling and compression. Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * use Write method from exp/api/remote Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix lint Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix test Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * nit fixed Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix lint Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> --------- Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	2025-10-27 13:56:48 +00:00
Minh Nguyen	f070e35358	[RW]: Adopt client_golang/exp/api/remote types for receiving RW1 and RW2 (#17197) ... Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> # Conflicts: # storage/remote/write_handler.go * add comment Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix failling test Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * nit_fixing Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix comment Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> --------- Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	2025-10-24 10:31:34 +01:00
beorn7	ad7d1aed99	Phase out native histogram feature flag ... The detailed plan for this is laid out in https://github.com/prometheus/prometheus/issues/16572 . This commit adds a global and local scrape config option `scrape_native_histograms`, which has to be set to true to ingest native histograms. To ease the transition, the feature flag is changed to simply set the default of `scrape_native_histograms` to true. Further implications: - The default scrape protocols now depend on the `scrape_native_histograms` setting. - Everywhere else, histograms are now "on by default". Documentation beyond the one for the feature flag and the scrape config are deliberately left out. See https://github.com/prometheus/prometheus/pull/17232 for that. Signed-off-by: beorn7 <beorn@grafana.com>	2025-10-15 14:50:52 +02:00