The dns_adb cleaning is little bit muddled as it mixes the "TTL"
based cleaning (.expire_v4 and .expire_v6 for adbname, .expires for
adbentry) with overmem cleaning.
Rewrite the LRU based cleaning to use SIEVE algorithm and to be overmem
cleaning only with a requirement to always cleanup at least 2-times the
size of the newly added entry.
Qpzone employs a locking strategy where rwlocks are grouped into
buckets, and each zone gets 17 buckets.
This strategy is suboptimal in two ways:
- If named is serving a single zone or a zone is the majority of the
traffic, this strategy pretty much guarantees contention when using
more than a dozen threads.
- If named is serving many small zones, it causes substantial memory
usage.
This commit switches the locking to a global table initialized at start
time. This should have three effects:
- Performance should improve in the single zone case, since now we are
selecting from a bigger pool of locks.
- Memory consumption should go down significantly in the many zone
cases.
- Performance should not degrade substantially in the many zone cases.
The reason for this is that, while we could have substantially more
zones than locks, we can query/edit only O(num threads) at the same
time. So by making the global table much bigger than the expected
number of threads, we can limit contention.
Merge branch 'alessio/global-qpzone-lock-table' into 'main'
See merge request isc-projects/bind9!10446
Qpzone employs a locking strategy where rwlocks are grouped into
buckets, and each zone gets 17 buckets.
This strategy is suboptimal in two ways:
- If named is serving a single zone or a zone is the majority of the
traffic, this strategy pretty much guarantees contention when using
more than a dozen threads.
- If named is serving many small zones, it causes substantial memory
usage.
This commit switches the locking to a global table initialized at start
time. This should have three effects:
- Performance should improve in the single zone case, since now we are
selecting from a bigger pool of locks.
- Memory consumption should go down significantly in the many zone
cases.
- Performance should not degrade substantially in the many zone cases.
The reason for this is that, while we could have substantially more
zones than locks, we can query/edit only O(num threads) at the same
time. So by making the global table much bigger than the expected
number of threads, we can limit contention.
In the current implementation, the resigning heap is part of the zone
database. This leads to a cycle, as the database has a reference to its
nodes, but each node needs a reference to the database.
This MR splits the resigning heap into its own separate struct, in order
to help breaking the cycle.
Merge branch 'alessio/split-qpzone-heap-from-qpdb' into 'main'
See merge request isc-projects/bind9!10706
In the current implementation, the resigning heap is part of the zone
database. This leads to a cycle, as the database has a reference to its
nodes, but each node needs a reference to the database.
This MR splits the resigning heap into its own separate struct, in order
to help breaking the cycle.
Recovering the node lock from a pointer to the header and a pointer to
the db is a common operation. This commit abstracts it away into a
function, so that the node lock selection logic may be modified more
easily.
A query for a zone that was not yet loaded may yield an unexpected result such as a CNAME or DNAME, triggering an assertion failure. This has been fixed.
Closes#5357
Merge branch '5357-resume-qmin-cname' into 'main'
See merge request isc-projects/bind9!10562
Print optionally a bit more details not passed to event in case
dns_view_findzonecut returns unexpected result. Result would be
visible later in foundevent, but found fname would be lost. Print it
into the log.
When authoritative zone is loaded when query minimization query for the
same zone is already pending, it might receive unexpected result codes.
Normally DNS_R_CNAME would follow to query_cname after processing sent
events, but dns_view_findzonecut does not fill CNAME target into
event->foundevent. Usual lookup via query_lookup would always have that
filled.
Ideally we would restart the query with unmodified search name, if
unexpected change from recursing to local zone cut were detected. Until
dns_view_findzonecut is modified to export zone/cache source of the cut,
at least fail queries which went into unexpected state.
The man page for :iscman:`named-makejournal` was erroneously not
installed when building from a source tarball. This has been fixed.
See #5379
Merge branch '5379-fix-named-makejournal-man-page-installation' into 'main'
See merge request isc-projects/bind9!10709
The man page for named-makejournal is erroneously not installed when
building from a source tarball. Add that man page to the appropriate
lists in the build system so that it is installed both when building
from a Git repository and from a source tarball.
System tests' after_script missed the PYTHON environmental variable
setup.
$ find -name 'tsan.*' -exec "$PYTHON" util/parse_tsan.py {} \;
find: '': No such file or directory
Merge branch 'mnowak/fix-parse_tsan-invocation' into 'main'
See merge request isc-projects/bind9!10683
System tests' after_script missed the PYTHON environmental variable
setup.
$ find -name 'tsan.*' -exec "$PYTHON" util/parse_tsan.py {} \;
find: '': No such file or directory
The ADB memory cleaning is opportunistic even when we are under
memory pressure (in the overmem condition). Split the opportunistic
LRU cleaning and overmem cleaning and make the overmem cleaning
always cleanup double of the newly allocated adbname/adbentry to
ensure we never allocate more memory than the assigned limit.
Merge branch 'ondrej/enforce-memory-cleanup-in-ADB-when-overmem' into 'main'
See merge request isc-projects/bind9!10637
The purge_stale_names()/purge_stale_entries() is opportunistic even when
we are under memory pressure (overmem). Split the opportunistic LRU
cleaning and overmem cleaning. This makes the stale purging much
simpler as we don't have to try that hard and makes the overmem cleaning
always cleanup double the amount of the newly allocated ADB name/entry.
Use the native shared library extension when build loadable
libaries. For most platforms this is ".so" but for Darwin it
is ".dylib".
Closes#5375
Merge branch '5375-use-native-shared-library-extension' into 'main'
See merge request isc-projects/bind9!10588
Tweak various system test which have been unstable in the past weeks.
Closes#5406
Merge branch 'nicki/improve-system-test-stability' into 'main'
See merge request isc-projects/bind9!10690
The code which checks for both IPv4 and IPv6 mixed usage is inherently
unstable, since the address family is chosen randomly for each
connection.
Closes#5406
It's possible to use pytest.mark.flaky, which achieves the exact same
thing as our custom-defined isctest.mark.flaky -- attempts to rerun the
test on failure, but only is flaky package is available.
The test_kasp_case[secondary.kasp] can sometimes fail on freebsd13. It
appears the test gets stuck on some operation which should be very
quick, but for some reason takes at least a few seconds, causing the
cb_ixfr_is_signed() function to time out.
In one of the cases I investigated, it wasn't a query/response that
caused a timeout, but rather some operation in between. The test
attempts to read from a keyfile/statefile, but I see no reason why that
should block.
In any case, try to increase the timeout for the verification, as that
shouldn't hurt. Also allow the test to be re-run on freebsd13, as it's
likely to be caused by some odd behaviour on that platform -- the issue
doesn't appear anywhere else.
The check "unix socket message counts" sometimes fails with "dnstap
output file smaller than expected". This only happens on freebsd13 and
can't be reproduced easily. There was an attempt to decrease the
required file size in the past, but apparently, the issue can still
occur.
The serve_stale test has some inherent instabilities affecting many
different checks. While the failure rate isn't too high (about four
failures in past three weeks of nightlies), it gets ignored, because the
test has been unstable for a very long time.
This removes a leftover check which should've been removed in a prior
change (see #5244). The softhsm2 failures when attempting to delete the
token should be ignored.
Previously, the one-second sleep was unreliable, as it didn't properly
indicate that the rndc reconfig has been processed. The "test 'rndc
reconfig' with a broken config" check would sometimes fail under TSAN
in CI, because the previous rndc reconfig was still ongoing, and the
subsequent rndc reconfig was ignored.
These tests have been unstable under TSAN in the past, but it appears
that the same failure mode can happen outside of TSAN tests as well.
These tests have produced 12 failures combined in the past three weeks
in nightlies.
The fetchlimit test has failed 8 times in the nightly CI over the past
three weeks. That makes the overall failure rate somewhere around 1 %,
which isn't a lot, but is still annoying when lots of testing is going
on.
Rndc test "test 'rndc reconfig' with a broken config" was failing
intermittently.
Wait for 'running' to be logged rather than just using 'sleep 1' before
calling 'rndc reconfig' a second time to get the expected error message
rather than 'reconfig request ignored: already running'.
Closes#5408
Merge branch '5408-rndc-test-second-rndc-reconfig-happens-too-soon' into 'main'
See merge request isc-projects/bind9!10687
Rndc test "test 'rndc reconfig' with a broken config" was failing
intermittently.
Wait for 'running' to be logged rather than just using 'sleep 1' before
calling 'rndc reconfig' a second time to get the expected error message
rather than 'reconfig request ignored: already running'.
There are many system tests where we set `dnssec-validation yes;` only
to also set `trust-anchors { };` which effectively disables the
validation.
This MR replaces this convoluted setup with just `dnssec-validation no;`.
Merge branch 'stepan/empty-trust-anchors-in-system-tests' into 'main'
See merge request isc-projects/bind9!10684
There are many system tests where we set `dnssec-validation yes;` only
to also set `trust-anchors { };` which effectively disables the
validation.
This commit replaces this convoluted setup with just
`dnssec-validation no;`.
On MRs it uses the merge target as the reference.
In schedules it uses the latest released version for this branch as the reference.
This MR lays the ground work for using respdiff on non-standard configurations (like ECS) in the public repo, see https://gitlab.isc.org/isc-private/bind9/-/merge_requests/807#note_573140.
To reduce the future hassle when maintaining the -S version, most of the work (including an added job, so we know that it actually works) is done here.
Merge branch 'stepan/respdiff-against-merge-target-or-last-release' into 'main'
See merge request isc-projects/bind9!10664
There are three adbname flags that are used to identify different
types of adbname lookups when hashing rather than using multiple
hash tables. Separate these to their own structure element as these
need to be able to be read without locking the adbname structure.
Closes#5404
Merge branch '5404-seperate-out-adbname-type-flags' into 'main'
See merge request isc-projects/bind9!10677
There are three adbname flags that are used to identify different
types of adbname lookups when hashing rather than using multiple
hash tables. Separate these to their own structure element as these
need to be able to be read without locking the adbname structure.
In specific circumstances the :iscman:`named` resolver process could
terminate unexpectedly when stale answers were enabled and the
``stale-answer-client-timeout 0`` configuration option was used.
This has been fixed.
See isc-projects/bind9#5372
Merge branch '5372-security-serve-stale-crash-on-insist-unreachable' into 'v9.21.10-release'
See merge request isc-private/bind9!808
If ns__query_start() is called because of a chained query (e.g.
after encountering a CNAME), a previously set DNS_DBFIND_STALETIMEOUT
flag on the query's 'dboptions' field can cause an assertion
failure if the new query's 'stalefirst' value is not true (e.g. if the
target qname is an authoritative zone for the server). Reset the
DNS_DBFIND_STALETIMEOUT flag in the query_lookup() function before
evaluating the 'stalefirst' value, and make sure to assign a fresh
value to the `stalefirst' flag instead of conditionally assigning it
only if the value is 'true'.
Mark unstable unit tests with `flaky` test suite. Execute the stable
separately in CI. Allow the flaky ones to be re-executed once in case
they fail.
Closes#5385
Merge branch '5385-rerun-flaky-unit-tests' into 'main'
See merge request isc-projects/bind9!10665
