1748 commits

Author	SHA1	Message	Date
bwplotka	3cf43337dc	post merge conflict fixes ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-03-12 09:03:08 +00:00
bwplotka	c133a969af	Merge branch 'main' into start-time-main-sync	2026-03-12 08:28:15 +00:00
Bartlomiej Plotka	a73202012b	tsdb/wlog[PERF]: optimize WAL watcher reads (up to 540x less B/op; 13000x less allocs/op) (#18250) ... See the detailed analysis https://docs.google.com/document/d/1efVAMcEw7-R_KatHHcobcFBlNsre-DoThVHI8AO2SDQ/edit?tab=t.0 I ran extensive benchmarks using synthetic data as well as real WAL segments pulled from the prombench runs. All benchmarks are here https://github.com/prometheus/prometheus/compare/bwplotka/wal-reuse?expand=1 * optimization(tsdb/wlog): reuse Ref* buffers across WAL watchers' reads Signed-off-by: bwplotka <bwplotka@gmail.com> * optimization(tsdb/wlog): avoid expensive error wraps Signed-off-by: bwplotka <bwplotka@gmail.com> * optimization(tsdb/wlog): reuse array for filtering Signed-off-by: bwplotka <bwplotka@gmail.com> * fmt Signed-off-by: bwplotka <bwplotka@gmail.com> * lint fix Signed-off-by: bwplotka <bwplotka@gmail.com> * tsdb/record: add test for clear() on histograms Signed-off-by: bwplotka <bwplotka@gmail.com> * updated WriteTo with what's currently expected Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-03-11 09:17:13 +00:00
Bartlomiej Plotka	d448f3f970	Merge pull request #18218 from prometheus/utilrecord ... tests(util/testwal): Move WAL record generation to separate package for reuse	2026-03-05 10:07:57 +01:00
Matthieu MOREL	45b9329e68	chore: fix emptyStringTest issues from gocritic (#18226) ... Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2026-03-04 08:24:50 +01:00
Kyle Eckhart	897ba10d11	remote write: fix sent_batch_duration_seconds measuring before the request is sent (#18214) ... * remote write: fix sent_batch_duration_seconds measuring before the request was sent Signed-off-by: Kyle Eckhart <kgeckhart@users.noreply.github.com>	2026-03-03 10:57:55 -08:00
bwplotka	596830ee72	tests(util/testwal): Move WAL record generation to separate package for reuse ... Signed-off-by: bwplotka <bwplotka@gmail.com> tmp Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-03-03 10:06:21 +00:00
bwplotka	0ad8516ce0	fixed tests after rebase ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-02-25 19:15:22 +00:00
bwplotka	f27ca31bed	tests: add bench CLI recommended invokations ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-02-25 19:05:49 +00:00
bwplotka	c2eac549d5	tests: test ST in a cheapest way possible ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-02-25 19:05:49 +00:00
bwplotka	ba1b87f51f	feat: RW2 sending ST support ... Signed-off-by: bwplotka <bwplotka@gmail.com> tmp Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-02-25 19:01:45 +00:00
bwplotka	8f3a6020d8	Merge branch 'main' into st-main-sync2	2026-02-25 13:54:25 +00:00
Bartlomiej Plotka	333e0dc188	tests: reinforce and optimize queue_manager_test createTimeSeries (#18179) ... * tests: fix createTimeSeries so it does not create unnecessary load 16M samples 4k series Signed-off-by: bwplotka <bwplotka@gmail.com> * addressed comments Signed-off-by: bwplotka <bwplotka@gmail.com> * Apply suggestions from code review Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>	2026-02-25 13:22:13 +00:00
bwplotka	56c46af6a6	Merge branch 'main' into st-f-main	2026-02-23 10:00:39 +00:00
George Krajcsovits	223f016c44	feat(tsdb): allow using ST capable XOR chunks - retain format on read (#18013) ... * feat(tsdb): allow appending to ST capable XOR chunk optionally Only for float samples as of now. Supports for in-order and out-of-order samples. Make sure that on readout the ST capable chunks are returned automatically. When the chunks are returned as is, this is trivially true. When a chunk needs to be re-coded due to deletion (tombstone) markers, we take the encoding of the original chunk. When a chunk needs to be created from overlapping chunks, we observe whether ST is zero or not and create the new chunk based on that. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2026-02-20 09:15:51 +01:00
Bartlomiej Plotka	1e2529f8d6	Merge pull request #18084 from veeceey/fix/issue-16645-otlp-too-old-sample ... fix: handle ErrTooOldSample as 400 in OTLP and v2 histogram write paths	2026-02-17 09:11:18 +00:00
Arve Knudsen	4fb6ce4d9a	fix: prevent stale exemplars leaking to histogram _sum/_count across data points (#18056) ... In addHistogramDataPoints, exemplars assigned to the +Inf bucket of one data point were carried over into the _sum and _count Append calls of the next data point via the shared appOpts. Clear appOpts.Exemplars at the start of each loop iteration to restore the nil-exemplar semantics that existed before the AppenderV2 migration. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2026-02-15 10:47:38 +01:00
Varun Chawla	e72bc1381c	fix: handle ErrTooOldSample as 400 Bad Request in OTLP and v2 histogram write paths ... The OTLP write handler and the PRW v2 histogram append path were missing ErrTooOldSample from their error type checks, causing these errors to fall through to the default case and return HTTP 500 Internal Server Error. This triggered unnecessary retries in OTLP clients like the Python SDK. The PRW v1 write handler (line 115) and the PRW v2 sample append path (line 377) already correctly handle ErrTooOldSample as a 400, and this change makes the remaining paths consistent. Also adds ErrTooOldSample to the v1 sample/histogram log checks so these errors are properly logged instead of silently returned. Fixes #16645 Signed-off-by: Varun Chawla <varun_6april@hotmail.com>	2026-02-14 02:49:49 -08:00
Owen Williams	b57f5b59b3	tsdb: ST-in-WAL: Counter implementation and benchmarks (#17671) ... Initial implementation of https://github.com/prometheus/prometheus/issues/17790. Only implements ST-per-sample for Counters. Tests and benchmarks updated. Note: This increases the size of the RefSample object for all users, whether st-per-sample is turned on or not. Signed-off-by: Owen Williams <owen.williams@grafana.com>	2026-02-12 13:17:50 -05:00
Ganesh Vernekar	0ede19e2ee	Replace deprecated google.CredentialsFromJSON with option.WithAuthCredentialsFile ... Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>	2026-02-10 13:03:14 -08:00
Bartlomiej Plotka	5e46e77754	refactor: use Appender mock for otlptranslator tests (#17999) ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-02-06 09:51:40 +00:00
Bartlomiej Plotka	c8e7f4e2a6	tests: Unify TestDiskFillingUpAfterDisablingOOO and avoid hiding errors (#18017) ... * tests: Unify TestDiskFillingUpAfterDisablingOOO and avoid hiding errors Signed-off-by: bwplotka <bwplotka@gmail.com> * addressed comments Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-02-05 16:11:35 +00:00
Bartlomiej Plotka	7769495a4a	refactor: switch OTLP handler to AppendableV2 (#17996) ... * refactor: switch OTLP handler to AppendableV2 Signed-off-by: bwplotka <bwplotka@gmail.com> * addressed comments Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-02-03 16:44:40 +00:00
Arve Knudsen	3c44ca757d	storage: fix ToError() to return nil for empty partial errors (#18002) ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2026-02-03 16:58:02 +01:00
Bartlomiej Plotka	44d772b4e7	refactor: use Appender mock for tests (#17992) ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-02-02 15:56:11 +00:00
Bartlomiej Plotka	076369fad0	refactor: move OTLP handler to separate file (#17990) ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-02-02 14:36:00 +00:00
Bartlomiej Plotka	9657c23c37	storage: optimized fanoutAppenderV2 (#17976) ... * storage: add BenchmarkFanoutAppenderV2 Signed-off-by: bwplotka <bwplotka@gmail.com> * fix: optimized fanoutAppenderV2 Signed-off-by: bwplotka <bwplotka@gmail.com> * optimized more Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-02-02 07:04:30 +00:00
bwplotka	6c18bba5c6	Revert "storage: add BenchmarkFanoutAppenderV2" ... This reverts commit 91b0f353b0.	2026-01-30 15:54:52 +00:00
bwplotka	91b0f353b0	storage: add BenchmarkFanoutAppenderV2 ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-01-30 15:53:59 +00:00
Arve Knudsen	dc34b90f93	otlptranslator: fix silently swallowed error in addSumNumberDataPoints (#17954) ... The createAttributes error was incorrectly returning nil instead of err, causing errors to be silently discarded. This could lead to silent data loss for sum metrics during OTLP ingestion. Fixes #17953 Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2026-01-28 13:58:50 +01:00
Callum Styan	97e7ef802c	remote write: simplify readability of timeseries filtering by using the slices package (#14318) ... * simplify readability of timeseries filtering by using the slices package Signed-off-by: Callum Styan <callumstyan@gmail.com> * ensure that BenchmarkBuildTimeSeries doesn't account for the building of the actual proto in the benchmark results, we only care about the buildTimeSeries call Signed-off-by: Callum Styan <callumstyan@gmail.com> --------- Signed-off-by: Callum Styan <callumstyan@gmail.com>	2026-01-28 11:47:34 +00:00
Ganesh Vernekar	9eb78735cf	Merge pull request #16929 from prometheus/codesome/stale-series-compaction ... tsdb: Early compaction of stale series Closes #13616 Based on https://github.com/prometheus/proposals/pull/55 Stale series tracking was added in #16925. This PR compacts the stale series into its own block before the normal compaction hits. Here is how the settings: stale_series_compaction_threshold: As soon as the ratio of stale series in the head block crosses StaleSeriesImmediateCompactionThreshold, TSDB performs a stale series compaction and puts all the stale series into a block and removed it from the head, but it does not remove it from the WAL. (technically this condition is checked every minute and not exactly immediate) Additional details WAL replay: after a stale series compaction, tombstones are added with (MinInt64, MaxInt64) for all these stale series. During WAL replay we add a special condition where when we find such tombstone, it immediately removes the series from the memory instead of storing the tombstone. This is required so that we don't spike up memory during WAL replay and also don't keep the compacted stale series in the memory. Head block truncation ignores this block via the added metadata, similar to out-of-order blocks.	2026-01-24 15:18:17 -08:00
Arve Knudsen	2332962c4b	otlptranslator: filter __name__ from OTLP attributes to prevent duplicates (#17917) ... * otlptranslator: filter __name__ from OTLP attributes to prevent duplicates OTLP metrics can have a __name__ attribute which, when combined with the metric name passed via extras, creates duplicate __name__ labels. This commit implements filtering out of any __name__ metric attribute from OTLP. Also rename TestCreateAttributes to TestPrometheusConverter_createAttributes for consistency, and add test cases for __name__, __type__, and __unit__ OTLP metric attributes. --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2026-01-24 16:44:08 +01:00
Ganesh Vernekar	4f3de8da29	tsdb: Add unit tests for stale series compaction ... Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>	2026-01-23 18:07:34 -08:00
Bartlomiej Plotka	0d116b0994	tests(teststorage): Close Storage in the helper (#17902) ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-01-23 08:41:35 +00:00
Ben Kochie	1a0c56993c	Use standard library errors in otlptranslator (#17894) ... Remove direct use of `go.uber.org/multierr` in favor of standard library `errors` package. Signed-off-by: SuperQ <superq@gmail.com>	2026-01-20 10:13:40 +01:00
Bartlomiej Plotka	c4b0da94db	Merge pull request #17835 from prometheus/bwplotka/a2-storage-support ... feat(storage)[PART4b]: add AppenderV2 to the rest of storage.Storage implementations + mock exemplar fix	2026-01-16 13:04:14 +00:00
bwplotka	f61a83bcd6	fix: appenderV2 mock exemplar appendErr injection before appending it ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-01-16 10:47:00 +00:00
Arve Knudsen	4afa76d083	OTLP: label caching for OTLP-to-Prometheus conversion to reduce allocations and improve latency (#17860) ... * otlptranslator: add label caching for OTLP-to-Prometheus conversion Add per-request caching to reduce redundant computation and allocations during OTLP metric conversion: 1. Per-request label sanitization cache: Cache sanitized label names within a request to avoid repeated string allocations for commonly repeated labels like __name__, job, instance. 2. Resource-level label caching: Precompute and cache job, instance, promoted resource attributes, and external labels once per ResourceMetrics boundary instead of for each datapoint. 3. Scope-level label caching: Precompute and cache scope metadata labels (otel_scope_name, otel_scope_version, etc.) once per ScopeMetrics boundary. 4. LabelNamer instance caching: Reuse the LabelNamer struct across datapoints within the same resource context. These optimizations significantly reduce allocations and improve latency for OTLP ingestion workloads with many datapoints per resource/scope. --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>	2026-01-16 10:30:05 +00:00
George Krajcsovits	ef350b2b54	Merge pull request #17840 from prometheus/krajo/st-in-chunks ... feat(tsdb): new Append parameter and AtST interface for chunks	2026-01-16 10:32:53 +01:00
bwplotka	1d3c6210f3	add extra fanout test ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-01-15 14:13:58 +00:00
Arve Knudsen	a85a8998c9	otlptranslator: fix BenchmarkPrometheusConverter_FromMetrics (#17866) ... The benchmark was passing appendMetadata=false to NewCombinedAppender, which caused UpdateMetadata to never be called on the underlying noOpAppender. This resulted in app.metadata always being 0, failing the assertion that metadata count should be positive. Fix by enabling metadata appending in the benchmark. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2026-01-15 09:33:16 +01:00
bwplotka	8a2921e385	addressed feedback ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-01-14 13:57:48 +00:00
bwplotka	49c3aea56d	feat(storage)[PART4b]: add AppenderV2 to the rest of storage.Storage impl ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-01-14 13:54:27 +00:00
György Krajcsovits	6137de085e	test ListSeriesIterator with ST ... Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2026-01-14 13:15:19 +01:00
György Krajcsovits	5ecc0e7062	test that ChainSampleIterator passes on the AtST call ... Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2026-01-14 13:15:18 +01:00
György Krajcsovits	8067b3d60a	add test coverage for buffer.go ... I've checked that commenting out any one of the new lines produces an error. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2026-01-14 13:15:17 +01:00
György Krajcsovits	6647e512ad	update ExpandSamples ... Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2026-01-14 13:15:14 +01:00
György Krajcsovits	a00c0d6a66	auto update f/h/fh sample init with positional fields ... find . -name "*.go" -type f \ -exec sed -E -i 's/((f|h|fh)Sample\{)([^,{:]+,[^,]+\})/\10, \3/g' {} + Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2026-01-14 13:15:11 +01:00
György Krajcsovits	1e77d9ded8	storage/buffer.go: add ST to sample types and iterators ... Also fix the single multi line fSample definition to be one liner. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2026-01-14 13:15:10 +01:00