The named configuration files used in the "geoip2" system test cause a
rather large number of views (6-8) to be set up in each tested named
instance. Each view has its own cache.
Commit aa72c31422 caused the RBT hash
table to be pre-allocated to a size derived from "max-cache-size", so
that it never needs to be rehashed. The size of that hash table is not
expected to be significant enough to cause memory use issues in typical
conditions even for large "max-cache-size" settings.
However, these two factors combined can cause memory exhaustion issues
in GitLab CI, where we run multiple "instances" of the test suite in
parallel on the same runner, each test suite executes multiple system
tests concurrently, and each system test may potentially start multiple
named instances at the same time. In practice, this problem currently
only seems to be affecting the "geoip2" system test, which is failing
intermittently due to named instances used by that test getting killed
by oom-killer.
Prevent the "geoip2" system test from failing intermittently by setting
"max-cache-size" in named configuration files used in that test to a low
value in order to keep memory usage at bay even with a large number of
views configured.
(cherry picked from commit 4292d5bdfe)
In 9.17 we introduced 'primaries' as a synonym for 'masters' in the
configuration file. This synonym has not been backported so change
the serve-stale test to make use of the 'masters' keyword.
Add a fifth named (ns5) that runs with `stale-cache-enable no;` and
check that there are no stale records in the cache.
(cherry picked from commit abc2ab9223)
The current serve-stale implementation in BIND 9 stores all received
records in the cache for a max-stale-ttl interval (default 12 hours).
This allows DNS operators to turn the serve-stale answers in an event of
large authoritative DNS outage. The caching of the stale answers needs
to be enabled before the outage happens or the feature would be
otherwise useless.
The negative consequence of the default setting is the inevitable
cache-bloat that happens for every and each DNS operator running named.
In this MR, a new configuration option `stale-cache-enable` is
introduced that allows the operators to selectively enable or disable
the serve-stale feature of BIND 9 based on their decision.
The newly introduced option has been disabled by default,
e.g. serve-stale is disabled in the default configuration and has to be
enabled if required.
(cherry picked from commit ce53db34d6)
Running system tests with root privileges is potentially dangerous.
Only allow it when explicitly requested (by building with
--enable-developer).
(cherry picked from commit 3ef106f69d)
There were several problems with rbt hashtable implementation:
1. Our internal hashing function returns uint64_t value, but it was
silently truncated to unsigned int in dns_name_hash() and
dns_name_fullhash() functions. As the SipHash 2-4 higher bits are
more random, we need to use the upper half of the return value.
2. The hashtable implementation in rbt.c was using modulo to pick the
slot number for the hash table. This has several problems because
modulo is: a) slow, b) oblivious to patterns in the input data. This
could lead to very uneven distribution of the hashed data in the
hashtable. Combined with the single-linked lists we use, it could
really hog-down the lookup and removal of the nodes from the rbt
tree[a]. The Fibonacci Hashing is much better fit for the hashtable
function here. For longer description, read "Fibonacci Hashing: The
Optimization that the World Forgot"[b] or just look at the Linux
kernel. Also this will make Diego very happy :).
3. The hashtable would rehash every time the number of nodes in the rbt
tree would exceed 3 * (hashtable size). The overcommit will make the
uneven distribution in the hashtable even worse, but the main problem
lies in the rehashing - every time the database grows beyond the
limit, each subsequent rehashing will be much slower. The mitigation
here is letting the rbt know how big the cache can grown and
pre-allocate the hashtable to be big enough to actually never need to
rehash. This will consume more memory at the start, but since the
size of the hashtable is capped to `1 << 32` (e.g. 4 mio entries), it
will only consume maximum of 32GB of memory for hashtable in the
worst case (and max-cache-size would need to be set to more than
4TB). Calling the dns_db_adjusthashsize() will also cap the maximum
size of the hashtable to the pre-computed number of bits, so it won't
try to consume more gigabytes of memory than available for the
database.
FIXME: What is the average size of the rbt node that gets hashed? I
chose the pagesize (4k) as initial value to precompute the size of
the hashtable, but the value is based on feeling and not any real
data.
For future work, there are more places where we use result of the hash
value modulo some small number and that would benefit from Fibonacci
Hashing to get better distribution.
Notes:
a. A doubly linked list should be used here to speedup the removal of
the entries from the hashtable.
b. https://probablydance.com/2018/06/16/fibonacci-hashing-the-optimization-that-the-world-forgot-or-a-better-alternative-to-integer-modulo/
(cherry picked from commit e24bc324b4)
Check that resign interval is actually in days rather than hours
by checking that RRSIGs are all within the allowed day range.
(cherry picked from commit 11ecf7901b)
While the creation and publication times of the various keys
in this policy are nearly at the same time there is a chance that
one key is created a second later than the other.
The `set_keytimes_algorithm_policy` mistakenly set the keytimes
for KEY3 based of the "published" time from KEY2.
(cherry picked from commit 24e07ae98e)
This test ensures that named will correctly shutdown
when receiving multiple control connections after processing
of either "rncd stop" or "kill -SIGTERM" commands.
Before the fix, named was crashing due to a race condition happening
between two threads, one running shutdown logic in named/server.c
and other handling control logic in controlconf.c.
This test tries to reproduce the above scenario by issuing multiple
queries to a target named instance, issuing either rndc stop or kill
-SIGTERM command to the same named instance, then starting multiple rndc
status connections to ensure it is not crashing anymore.
(cherry picked from commit 042e509753)
Implement the 'rndc dnssec -status' command that will output
some information about the key states, such as which policy is
used for the zone, what keys are in use, and when rollover is
scheduled.
Add loose testing in the kasp system test, the actual times are
already tested via key file inspection.
(cherry picked from commit 19ce9ec1d4)
Make yaml.load_all() use yaml.SafeLoader to address a warning currently
emitted when bin/tests/system/dnstap/ydump.py is run:
ydump.py:28: YAMLLoadWarning: calling yaml.load_all() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
for l in yaml.load_all(f.stdout):
The wait until zones are signed after rndc reconfig is broken
because the zones are already signed before the reconfig. Fix
by having a different way to ensure the signing of the zone is
complete. This does require a call to the "wait_for_done_signing"
function after each "check_keys" call after the ns6 reconfig.
The "wait_for_done_signing" looks for a (newly added) debug log
message that named will output if it is done signing with a certain
key.
(cherry picked from commit a47192ed5b)
Add a note why we don't have a test case for the issue.
It is tricky to write a good test case for this if our tools are
not allowed to create signatures for unsupported algorithms.
(cherry picked from commit c6345fffe9)
these keywords were added to the parser as synonyms for "master"
and "slave" but were never hooked in to the configuration of named,
so they were ignored. this has been fixed and the option is now
checked for correctness.
(cherry picked from commit ba31b189b4)
DS records only belong at delegation points and if present
at the zone apex are invariably the result of administrative
errors. Additionally they can't be queried for with modern
resolvers as the parent servers will be queried.
(cherry picked from commit 35a58d30c9)
Originally, the default value for max-stale-ttl was 1 week, which could
and in some scenarios lead to cache exhaustion on a busy resolvers.
Picking the default value will always be juggling between value that's
useful (e.g. keeping the already cached records after they have already
expired and the upstream name servers are down) and not bloating the
cache too much (e.g. keeping everything for a very long time). The new
default reflects what we think is a reasonable to time to react on both
sides (upstream authoritative and downstream recursive).
(cherry picked from commit 13fd3ecfab)
The logic in `keymgr_key_has_successor(key, keyring)` is flawed, it
returns true if there is any key in the keyring that has a successor,
while what we really want here is to make sure that the given key
has a successor in the given keyring.
Rather than relying on `keymgr_key_exists_with_state`, walk the
list of keys in the keyring and check if the key is a successor of
the given predecessor key.
(cherry picked from commit 0d578097ef)
The usage of 'date -d' in the kasp system test is not portable,
replace with a python script. Also remove some leftover
"set_keytime 'yes'" calls.
(cherry picked from commit 5b3decaf48)
This improves keytime testing on algorithm rollover. It now
tests for specific times, and also tests for SyncPublish and
Removed keytimes.
(cherry picked from commit 61c1040ae5)
This improves keytime testing on CSK rollover. It now
tests for specific times, and also tests for SyncPublish and
Removed keytimes.
Since an "active key" for ZSK and KSK means something
different, this makes it tricky to decide when a CSK is
active. An "active key" intuitively means the key is signing
so we say a CSK is active when it is creating zone signatures.
This change means a lot of timings for the CSK rollover tests
need to be adjusted.
The keymgr code needs a slight change on calculating the
prepublication time: For a KSK we need to include the parent
registration delay, but for CSK we look at the zone signing
property and stick with the ZSK prepublication calculation.
(cherry picked from commit e233433772)
This improves keytime testing on KSK rollover. It now
tests for specific times, and also tests for SyncPublish and
Removed keytimes.
(cherry picked from commit 649d0833ce)
Registration delay is not part of the Iret retire interval, thus
removed from the calculation when setting the Delete time metadata.
Include the registration delay in prepublication time, because
we need to prepublish the key sooner than just the Ipub
publication interval.
(cherry picked from commit 50bbbb76a8)
This improves keytime testing on ZSK rollover. It now
tests for specific times, and also tests for SyncPublish and
Removed keytimes.
(cherry picked from commit e01fcbbaf8)
This improves keytime testing for enabling DNSSEC. It now
tests for specific times, and also tests for SyncPublish.
(cherry picked from commit cf51c87fad)
This commit adds testing keytiming metadata. In order to facilitate
this, the kasp system test undergoes a few changes:
1. When finding a key file, rather than only saving the key ID,
also save the base filename and creation date with `key_save`.
These can be used later to set expected key times.
2. Add a test function `set_addkeytime` that takes a key, which
keytiming to update, a datetime in keytiming format, and a number
(seconds) to add, and sets the new time in the given keytime
parameter of the given key. This is used to set the expected key
times.
3. Split `check_keys` in `check_keys` and `check_keytimes`. First we
need to find the keyfile before we can check the keytimes.
We need to retrieve the creation date (and sometimes other
keytimes) to determine the other expected key times.
4. Add helper functions to set the expected key times per policy.
This avoids lots of duplication.
Check for keytimes for the first test cases (all that do not cover
rollovers).
(cherry picked from commit f8e34b57b4)
After removing dnssec-settime calls that set key rollover
relationship, we can adjust the counts in test output filenames.
Also fix a couple of more wrong counts in output filenames.
(cherry picked from commit 8204e31f0e)
Using dnssec-setttime after dnssec-keygen in the kasp system test
can lead to off by one second failures, so reduce the usage of
dnssec-settime in the setup scripts. This commit deals with
setting the key rollover relationship (predecessor/successor).
(cherry picked from commit 5a590c47a5)
