From 92ae68764954b3606a899f5f3ecd16846cda7324 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicki=20K=C5=99=C3=AD=C5=BEek?= Date: Fri, 21 Nov 2025 14:35:57 +0100 Subject: [PATCH 1/2] Set stricter respdiff:recent-named threshold This job is testing the current BIND implementation against the latest released version. Unless there has been a behaviour change, there should be no difference. In practice, there is a small number of differences caused by upstream discrepencies. Some of those cause "upstream unstable" answers which are excluded from the results, but statistically, some of those will end up being detected as differences on the resolver under test. Currently, there seems to be about 300 upstream unstable answers with typically around 50-60 differences. Setting the threshold to 0.1 should be stable enough to pass if there are no changes, yet sensitive enough to detect even fairly small changes to behaviour. --- .gitlab-ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 90b2b9d834..196ce3f807 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -2027,7 +2027,7 @@ respdiff:recent-named: CC: gcc CFLAGS: "${CFLAGS_COMMON} -DISC_TRACK_PTHREADS_OBJECTS" EXTRA_CONFIGURE: "-Doptimization=g" - MAX_DISAGREEMENTS_PERCENTAGE: "0.3" + MAX_DISAGREEMENTS_PERCENTAGE: "0.1" # Performance tests From bcc4369b0bf243433ca5334cdce3982a15ce4027 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nicki=20K=C5=99=C3=AD=C5=BEek?= Date: Fri, 21 Nov 2025 15:05:36 +0100 Subject: [PATCH 2/2] Increase the threshold for respdiff-third-party There are multiple reasons for the increased amount of differences we've been seeing lately and for the raise of the threshold: 1. Recent hardening against cache poisoning (CVE-2025-40778) have uncovered a few edge cases where the domain can't be properly resolved with the new protections in place, but those are issues with upstream configuration and DNS setup. 2. The same hardening magnified some behaviour differences between 9.21 and older versions. Some misconfigured domains, which can be resolved with BIND 9.20 and older are no longer resolvable in 9.21+. This can be again attributed to upstream DNS misconfiguration. See #5649. 3. A change in the respdiff CI job to include timeouts in the comparison, or rather, increasing the timeouts to resolve the previously timed out queries, which are typically failures. With the previous job configuration, those were omitted from comparison, because they were timeouts. Now, there should be no timeouts, but there is a slight increase in the amount of differences for the threshold evaluation. --- .gitlab-ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 196ce3f807..ddb03c1f32 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -1998,7 +1998,7 @@ respdiff-third-party: CC: gcc CFLAGS: "${CFLAGS_COMMON}" EXTRA_CONFIGURE: "-Doptimization=g" - MAX_DISAGREEMENTS_PERCENTAGE: "0.3" + MAX_DISAGREEMENTS_PERCENTAGE: "0.4" script: - bash respdiff.sh -s third_party -q "${PWD}/100k_mixed.txt" -c 1 -w "${PWD}/rspworkdir" "${CI_PROJECT_DIR}" - cd ../.. && ninja -C build clean >/dev/null 2>&1