774 commits

Author	SHA1	Message	Date
Bartlomiej Plotka	7769495a4a	refactor: switch OTLP handler to AppendableV2 (#17996) ... * refactor: switch OTLP handler to AppendableV2 Signed-off-by: bwplotka <bwplotka@gmail.com> * addressed comments Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-02-03 16:44:40 +00:00
Bartlomiej Plotka	44d772b4e7	refactor: use Appender mock for tests (#17992) ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-02-02 15:56:11 +00:00
Bartlomiej Plotka	076369fad0	refactor: move OTLP handler to separate file (#17990) ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-02-02 14:36:00 +00:00
Arve Knudsen	dc34b90f93	otlptranslator: fix silently swallowed error in addSumNumberDataPoints (#17954) ... The createAttributes error was incorrectly returning nil instead of err, causing errors to be silently discarded. This could lead to silent data loss for sum metrics during OTLP ingestion. Fixes #17953 Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2026-01-28 13:58:50 +01:00
Callum Styan	97e7ef802c	remote write: simplify readability of timeseries filtering by using the slices package (#14318) ... * simplify readability of timeseries filtering by using the slices package Signed-off-by: Callum Styan <callumstyan@gmail.com> * ensure that BenchmarkBuildTimeSeries doesn't account for the building of the actual proto in the benchmark results, we only care about the buildTimeSeries call Signed-off-by: Callum Styan <callumstyan@gmail.com> --------- Signed-off-by: Callum Styan <callumstyan@gmail.com>	2026-01-28 11:47:34 +00:00
Arve Knudsen	2332962c4b	otlptranslator: filter __name__ from OTLP attributes to prevent duplicates (#17917) ... * otlptranslator: filter __name__ from OTLP attributes to prevent duplicates OTLP metrics can have a __name__ attribute which, when combined with the metric name passed via extras, creates duplicate __name__ labels. This commit implements filtering out of any __name__ metric attribute from OTLP. Also rename TestCreateAttributes to TestPrometheusConverter_createAttributes for consistency, and add test cases for __name__, __type__, and __unit__ OTLP metric attributes. --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2026-01-24 16:44:08 +01:00
Ben Kochie	1a0c56993c	Use standard library errors in otlptranslator (#17894) ... Remove direct use of `go.uber.org/multierr` in favor of standard library `errors` package. Signed-off-by: SuperQ <superq@gmail.com>	2026-01-20 10:13:40 +01:00
Bartlomiej Plotka	c4b0da94db	Merge pull request #17835 from prometheus/bwplotka/a2-storage-support ... feat(storage)[PART4b]: add AppenderV2 to the rest of storage.Storage implementations + mock exemplar fix	2026-01-16 13:04:14 +00:00
Arve Knudsen	4afa76d083	OTLP: label caching for OTLP-to-Prometheus conversion to reduce allocations and improve latency (#17860) ... * otlptranslator: add label caching for OTLP-to-Prometheus conversion Add per-request caching to reduce redundant computation and allocations during OTLP metric conversion: 1. Per-request label sanitization cache: Cache sanitized label names within a request to avoid repeated string allocations for commonly repeated labels like __name__, job, instance. 2. Resource-level label caching: Precompute and cache job, instance, promoted resource attributes, and external labels once per ResourceMetrics boundary instead of for each datapoint. 3. Scope-level label caching: Precompute and cache scope metadata labels (otel_scope_name, otel_scope_version, etc.) once per ScopeMetrics boundary. 4. LabelNamer instance caching: Reuse the LabelNamer struct across datapoints within the same resource context. These optimizations significantly reduce allocations and improve latency for OTLP ingestion workloads with many datapoints per resource/scope. --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>	2026-01-16 10:30:05 +00:00
George Krajcsovits	ef350b2b54	Merge pull request #17840 from prometheus/krajo/st-in-chunks ... feat(tsdb): new Append parameter and AtST interface for chunks	2026-01-16 10:32:53 +01:00
Arve Knudsen	a85a8998c9	otlptranslator: fix BenchmarkPrometheusConverter_FromMetrics (#17866) ... The benchmark was passing appendMetadata=false to NewCombinedAppender, which caused UpdateMetadata to never be called on the underlying noOpAppender. This resulted in app.metadata always being 0, failing the assertion that metadata count should be positive. Fix by enabling metadata appending in the benchmark. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2026-01-15 09:33:16 +01:00
bwplotka	8a2921e385	addressed feedback ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-01-14 13:57:48 +00:00
bwplotka	49c3aea56d	feat(storage)[PART4b]: add AppenderV2 to the rest of storage.Storage impl ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-01-14 13:54:27 +00:00
György Krajcsovits	70bc06718d	feat(tsdb): new AppenderV2 and AtST interface for chunks ... No implementation yet. Just to test the shape of the interface. AtST is implemented for trivial cases, anything else is hard coded to return 0. Ref: https://github.com/prometheus/prometheus/issues/17791 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2026-01-14 13:15:09 +01:00
George Krajcsovits	66bdc88013	fix(remote_read): NHCB not returned over remote read samples (#17794) ... NHCB is native histograms with custom buckets. prompb is used for both remote write 1.0 and remote read. We do not support NHCB over remote write 1.0 , however we should absolutely support it for remote read. Prometheus remote write 1.0 client already refuses to send NHCB. Prometheus remote write 1.0 server accepts NHCB, but doesn't store custom values, corrupting the result. I'm now handling NHCB correctly, instead of refusing or corrupting. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2026-01-07 08:44:57 +01:00
Ben Kochie	e14795bbf4	Remove copyright date from headers (#17785) ... Remove copyright dates from various files as part of [PROM-50]. [PROM-50]: https://github.com/prometheus/proposals/blob/main/proposals/0050-remove-copyright-dates.md Signed-off-by: SuperQ <superq@gmail.com>	2026-01-05 13:46:21 +01:00
dependabot[bot]	87401302f6	chore(deps): bump google.golang.org/api from 0.257.0 to 0.258.0 (#17761) ... Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.257.0 to 0.258.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.257.0...v0.258.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-version: 0.258.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2026-01-04 15:47:58 +00:00
Bryan Boreham	950f756e9e	Merge remote-tracking branch 'origin/release-3.8' into merge-3.8.1-to-main ... Tweak RW2 change for consistency with older lines. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2025-12-19 09:33:41 +00:00
bwplotka	da253bddf5	fix: ensure remote PRWv1 write handler does not send false 0 response headers ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-15 12:27:35 +00:00
bwplotka	cd98ded6ea	test: add regression test against remote write handler bad response stats ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-12 22:43:51 +00:00
Julien	93edf912a7	Merge pull request #17630 from zjumathcode/main ... chore: Fix function name typo in createBatchSpan comment	2025-12-02 16:23:48 +01:00
Ben Edmunds	0e682a70a6	RW2: Allow custom scope in azuread (#17483) ... Signed-off-by: Ben Edmunds <sammybenblue2@gmail.com>	2025-12-02 11:45:23 +00:00
zjumathcode	0ac2221a20	chore: Fix function name typo in createBatchSpan comment ... Signed-off-by: zjumathcode <pai314159@2980.com>	2025-12-01 19:14:19 +08:00
Björn Rabenstein	b8d19543b8	Add histogram validation in remote-read and during reducing resolution (#17561) ... ReduceResolution is currently called before validation during ingestion. This will cause a panic if there are not enough buckets in the histogram. If there are too many buckets, the spurious buckets are ignored, and therefore the error in the input histogram is masked. Furthermore, invalid negative offsets might cause problems, too. Therefore, we need to do some minimal validation in reduceResolution. Fortunately, it is easy and shouldn't slow things down. Sadly, it requires to return errors, which triggers a bunch of code changes. Even here is a bright side, we can get rud of a few panics. (Remember: Don't panic!) In different news, we haven't done a full validation of histograms read via remote-read. This is not so much a security concern (as you can throw off Prometheus easily by feeding it bogus data via remote-read) but more that remote-read sources might be makeshift and could accidentally create invalid histograms. We really don't want to panic in that case. So this commit does not only add a check of the spans and buckets as needed for resolution reduction but also a full validation during remote-read. Signed-off-by: beorn7 <beorn@grafana.com>	2025-11-21 00:22:24 +01:00
Bartlomiej Plotka	3bcc88b053	prw2: Move Remote Write 2.0 CT to be per Sample; Rename to ST (start timestamp) (#17411) ... Relates to https://github.com/prometheus/prometheus/issues/16944#issuecomment-3164760343 Signed-off-by: bwplotka <bwplotka@gmail.com> (cherry picked from commit cefefc6897)	2025-11-20 08:26:04 +00:00
Minh Nguyen	5087a25848	Remote Write Receive Fix: Remove duplicate labels when type-and-unit-label feature is on (#17546) ... * drop extra label from receiver Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * used constant Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> --------- Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	2025-11-18 09:37:09 +00:00
Bartlomiej Plotka	cefefc6897	prw2: Move Remote Write 2.0 CT to be per Sample; Rename to ST (start timestamp) (#17411) ... Relates to https://github.com/prometheus/prometheus/issues/16944#issuecomment-3164760343 Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-11-17 14:59:40 +00:00
Laurent Dufresne	d99f8dacc4	chore: remove dead code (#17542) ... Signed-off-by: Laurent Dufresne <laurent.dufresne@grafana.com>	2025-11-17 10:37:55 +01:00
Bartlomiej Plotka	f50ff0a40a	feat: rename CreatedTimestamp to StartTimestamp (#17523) ... Partially fixes https://github.com/prometheus/prometheus/issues/17416 by renaming all CT* names to ST* in the whole codebase except RW2 (this is done in separate [PR](https://github.com/prometheus/prometheus/pull/17411)) and PrometheusProto exposition proto. ``` CreatedTimestamp -> StartTimestamp CreatedTimeStamp -> StartTimestamp created_timestamp -> start_timestamp CT -> ST ct -> st ``` Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-11-13 14:17:51 +00:00
Bartlomiej Plotka	675bafe2fb	Merge pull request #17441 from pipiland2612/refactor_queue_manger ... Refactor part of queue_manger.go by creating struct to reuse some common function	2025-11-13 15:07:11 +01:00
Minh Nguyen	7ebff91cfd	OTLP Receiver: Only update metadata to WAL when metadata-wal-records feature is enabled (#17472) ... OTLP Receiver: Only update metadata to WAL when metadata-wal-records feature is enabled. --------- Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	2025-11-13 09:53:12 +01:00
Ben Kochie	204249fcb5	Update golangci-lint (#17478) ... * Update golangci-lint to v2.6.0 * Fixup various linting issues. * Fixup deprecations. * Add exception for `labels.MetricName` deprecation. Signed-off-by: SuperQ <superq@gmail.com>	2025-11-05 13:47:34 +01:00
Minh Nguyen	30992dd032	[RW2] Fix: Only update metadata to WAL when metadata-wal-records feature is enabled (#17470) ... * add feature check when UpdateMetadata Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * add appendMetadata boolean to write_hander Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> --------- Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	2025-11-04 08:16:57 +00:00
Ben Kochie	48956f60d7	Update modernize (#17471) ... Apply additional Go modernize tool improvements. Signed-off-by: SuperQ <superq@gmail.com>	2025-11-04 05:13:49 +00:00
Minh Nguyen	784ec0a792	update test to test both v1 and v2 (#17467) ... Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	2025-11-03 09:22:46 +00:00
pipiland2612	704afd8529	add timeSeriesAgeChecker to refactor filter code ... Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	2025-10-31 23:19:53 +02:00
pipiland2612	9e6a626dae	create timeSeriesStats to reduce return variable ... Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	2025-10-31 22:17:45 +02:00
pipiland2612	e1cb29bf8a	create common struct and function to DRY ... Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	2025-10-31 21:55:14 +02:00
Minh Nguyen	c8f1de18a7	[RW2] Fix type and unit labels propagation in Remote Write v2 receiver to prioritize type-and-unit-labels feature (#17387) ... * fix Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix nits & update docs Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix docs Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> --------- Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	2025-10-31 08:59:03 +00:00
György Krajcsovits	b8192127ee	Merge remote-tracking branch 'origin/release-3.7' into krajo/merge-3.7.3-to-main ... # Conflicts: # CHANGELOG.md # storage/remote/queue_manager_test.go	2025-10-30 09:21:25 +01:00
Laurent Dufresne	7621eb772c	histogram: Add Error type for all histogram errors ... `histogram.Error` becomes the generic wrapper type for all histogram errors. This makes it easier and less error prone when adding new errors to check if an error is an histogram error as well as making it less error prone to convert the errors. This change the type of those specific sentinel errors from error to `histogram.Error`, but it should almost never matter. e.g., `errors.Is(err, ErrHistogram...)` would still work out of the box. Signed-off-by: Laurent Dufresne <laurent.dufresne@grafana.com>	2025-10-30 08:45:34 +01:00
Ayoub Mrini	6806b68f93	[release-3.7] fix: Remote-write: revert changes in the queue resharding logic (#17412) ... * Revert "chore: deprecate prometheus_remote_storage_{samples,exemplars,histograms}_in_total and prometheus_remote_storage_highest_timestamp_in_seconds" This reverts commit ba14bc49db. Signed-off-by: machine424 <ayoubmrini424@gmail.com> * Revert "storage/remote: compute highestTimestamp and dataIn at QueueManager level" This reverts commit 184c7eb918. Signed-off-by: machine424 <ayoubmrini424@gmail.com> * fix(remote-write): bring back the per queue metrics Signed-off-by: machine424 <ayoubmrini424@gmail.com> * test(remote): add TestRemoteWrite_ReshardingWithoutDeadlock to reproduce the sharding scale up deadlock Signed-off-by: machine424 <ayoubmrini424@gmail.com> --------- Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2025-10-29 14:04:09 +00:00
Arve Knudsen	c36e966bf8	OTLP: de-duplicate target_info samples with conflicting timestamps (#17400) ... Add logic to the target_info metric generation in the OTLP endpoint, so that any samples with the same timestamp for the same (target_info) series are de-duplicated. It comes out of a user's bug report about duplicated target_info samples in Grafana Mimir (which uses the Prometheus target_info generation logic). If I'm not mistaken, duplicate target_info samples should stem from multiple resources in the same OTLP request being translated to the same target_info label set. It shouldn't be caused by a Prometheus bug.	2025-10-28 14:13:43 +00:00
Minh Nguyen	6bb367970e	feat(promtool): add RW2 support to promtool push metrics using client_golang library (#17280) ... * Add WriteProto method and tests for promtool metrics This commit adds: 1. WriteProto method to storage/remote/client.go that handles marshaling and compression of protobuf messages 2. Updated parseAndPushMetrics in cmd/promtool/metrics.go to use the new WriteProto method 3. Comprehensive tests for PushMetrics functionality The WriteProto method provides a cleaner API for sending protobuf messages without manually handling marshaling and compression. Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * use Write method from exp/api/remote Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix lint Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix test Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * nit fixed Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix lint Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> --------- Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	2025-10-27 13:56:48 +00:00
Minh Nguyen	f070e35358	[RW]: Adopt client_golang/exp/api/remote types for receiving RW1 and RW2 (#17197) ... Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> # Conflicts: # storage/remote/write_handler.go * add comment Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix failling test Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * nit_fixing Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix comment Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> --------- Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	2025-10-24 10:31:34 +01:00
Julius Hinze	05612757b4	prometheusremotewrite: fix require.equal argument order (#17391) ... Signed-off-by: Julius Hinze <julius.hinze@grafana.com>	2025-10-23 15:13:32 +02:00
Arve Knudsen	ef42c088ba	OTLP: Add configuration parameters to control label name translation (#17345) ... As a follow-up to #17344, add two configuration parameters for controlling label name translation, both defaulting to on for backwards compatibility (currently these behaviours are hardcoded as enabled): * otlp.label_name_underscore_sanitization => Prefix label names starting with a single underscore with key_ when translating OTel attribute names * otlp.label_name_preserve_multiple_underscores => Keep multiple consecutive underscores in label names when translating OTel attribute names Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2025-10-22 08:27:35 +02:00
György Krajcsovits	ea398c15e8	Merge branch 'release-3.7' into krajo/merge-release-3071-to-main	2025-10-17 10:45:55 +02:00
Arve Knudsen	99d0967133	Fix lint failure ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2025-10-16 16:56:18 +02:00
Arve Knudsen	f5804e7cf2	Remove configuration parameters ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2025-10-16 16:37:24 +02:00