The shotgun performance tests are executed in a different repository, in
a couple of different pipelines. To hide away the complexity, this
script takes the pipeline ID of the triggered pipeline and then takes
care of the rest - waits for the pipeline to finish, locates the child
pipeline and the relevant results. The output from this script is a
convenient link to the charts with the results once they're available.
GitLab also has a mechanism which can wait for another pipeline.
However, it can't be utilized here, since there are variables which
need to be passed in when the pipeline is triggered (like protocol to be
tested, load, runtime etc.). This isn't currently supported by the
GitLab feature.
(cherry picked from commit ff32421a64)
All changes in this commit were automated using the command:
shfmt -w -i 2 -ci -bn . $(find . -name "*.sh.in")
By default, only *.sh and files without extension are checked, so
*.sh.in files have to be added additionally. (See mvdan/sh#944)
(manually replayed commit 4cb8b13987)
At times, a problem might occur where a test is not responding,
especially in the CI, determining the specific test responsible can be
difficult. Fortunately, when running tests with the pytest runner,
pytest sets the PYTEST_CURRENT_TEST environment variable to the current
test nodeid and stage. Afterward, the variable can be examined to
identify the test that has stopped responding.
The monitoring script needs to be started in the background. Still, the
shell executor used for BSD and FIPS testing can't handle the background
process cleanly, and the script step will wait for the background
process for the entire duration of the background process (currently
3000 seconds). Therefore, run the monitoring script only when the Docker
executor is used where this is not a problem.
(cherry picked from commit 35792b1700)
util/parse_tsan.py builds tables of mutexes, threads, and pointers it
finds in the TSAN report provided to it as a command-line argument and
then replaces all mentions of each of these entities so that they are
numbered sequentially in the processed report. For example, this line:
Cycle in lock order graph: M0 (...) => M5 (...) => M9 (...) => M0
is expected to become:
Cycle in lock order graph: M1 (...) => M2 (...) => M3 (...) => M1
Problems arise when the gaps between mutex/thread identifiers present on
a single line are smaller than the total number of mutexes/threads found
by the script so far. For example, the following line:
Cycle in lock order graph: M0 (...) => M1 (...) => M2 (...) => M0
first gets turned into:
Cycle in lock order graph: M1 (...) => M1 (...) => M2 (...) => M1
and then into:
Cycle in lock order graph: M2 (...) => M2 (...) => M2 (...) => M2
In other words, lines like this become garbled due to information loss.
The problem stems from the fact that the numbering scheme the script
uses for identifying mutexes and threads is exactly the same as the one
used by TSAN itself. Update util/parse_tsan.py so that it uses
zero-padded numbers instead, making the "overlapping" demonstrated above
impossible.
(cherry picked from commit 7f0790c82f)
The script will start the named process configured as both an
authoritative and recursive server for each pairwise ./configure
configuration. The test is considered successful if the named process
runs until the 5-second timeout is triggered, and there is no named.lock
file present, indicating that named did not crash on shutdown.
(cherry picked from commit a708c2f93d)
Ensure that no .gitignore file in the tree contains patterns matching
files tracked by Git by adding a simple script that detects such
situations and running that script as part of the "misc" job in the
"precheck" stage of every GitLab CI pipeline.
(cherry picked from commit 5b28da7ea0)
The checkbashisms script reports errors like this one:
script util/check-line-length.sh does not appear to have a #! interpreter line;
you may get strange results
(cherry picked from commit 9e68997cbb)
Fedora 33 doesn't support RSASHA1 in future mode. There is no easy
check for this other than by attempting to perform a verification
using known good signatures. We don't attempt to sign with RSASHA1
as that would not work in FIPS mode. RSASHA1 is verify only.
The test vectors were generated using OpenSSL 3.0 and
util/gen-rsa-sha-vectors.c. Rerunning will generate a new set of
test vectors as the private key is not preserved.
e.g.
cc util/gen-rsa-sha-vectors.c -I /opt/local/include \
-L /opt/local/lib -lcrypto
(cherry picked from commit cd3f00874f63a50954cebb78edac8f580a27c0de)
We have had perpetual problem with Sphinx implicitly double-including
files. To avoid that problem all files with name suffix .inc.rst are now
ignored by Sphinx, and writter can conveniently include them without
modifying conf.py for each and every file.
(cherry picked from commit 1322372a0c)
When the parse tsan files have text extension they can be viewed
directly in the GitLab web UI without downloading them locally.
(cherry picked from commit 80582073a5)
The util/check-changes script has two modes of operation - more relaxed
release branch mode and strict development branch mode. When we forked
the v9_18 branch, the stricter mode stayed enabled.
Disable the strict CHANGES file checking suitable only for development
branch.
Also make the script more verbose to identify which patch is being
processed and check for failures in spatch standard error output.
(cherry picked from commit 48c44fe6d4)
This commit converts the license handling to adhere to the REUSE
specification. It specifically:
1. Adds used licnses to LICENSES/ directory
2. Add "isc" template for adding the copyright boilerplate
3. Changes all source files to include copyright and SPDX license
header, this includes all the C sources, documentation, zone files,
configuration files. There are notes in the doc/dev/copyrights file
on how to add correct headers to the new files.
4. Handle the rest that can't be modified via .reuse/dep5 file. The
binary (or otherwise unmodifiable) files could have license places
next to them in <foo>.license file, but this would lead to cluttered
repository and most of the files handled in the .reuse/dep5 file are
system test files.
The copyright handling has been long obsolete, the works is covered as
whole by the COPYING/LICENSE file even if a specific file doesn't have
a copyright header.
The important thing to remember here is that any work is covered by a
copyright law and by explicitly giving it license we provide extra
rights to the users of the works.
All signed zone files present in bin/tests/system/inline/ns8 should
contain the unsigned serial number in the raw-format header. Add a
check to ensure that is the case. Extend the dnssec-signzone command
line in ns8/sign.sh with the -L option to allow the zones initially
signed there to pass the newly added check. Add another zone to the
configuration for the ns8 named instance to ensure the check also passes
when multiple zones are inline-signed by a single named instance.
Add library ctor and dtor for isc_os compilation unit which initializes
the numbers of the CPUs and also checks whether L1 cacheline size is
really 64 if the sysconf() call is available.
This sets as many server options as possible at once to detect
cut-and-paste bugs when implementing new server options in peer.c.
Most of the accessor functions are similar and it is easy to miss
updating a macro name or structure element name when adding new
accessor functions.
checkconf/setup.sh is there to minimise the difference to branches
with optional server options where the list is updated at runtime.
The TCP connection reset test starts mock UDP and TCP server which
always returns empty DNS answer with TC bit set over UDP and resets the
TCP connection after five seconds.
When tested without the fix, the DNS query to 10.53.0.2 times out and
the ns2 server hangs at shutdown.
This commit adds a new system-test: transport-acl system test. It is
intended to test the new, extended syntax for ACLs, the one where port
or transport protocol can be specified. Currently, it includes the
tests only using allow-transfer statement, as this extended syntax is
used only there, at least for now.
This is almost minimal prototype to show how to use python-hypothesis
library in a system test. It does not fully replace existing shell-based
system test for wildcards.
OpenSSL 3.0.0 deprecates the ERR_get_error_line_data() function.
Use ERR_get_error_all() instead of ERR_get_error_line_data() and create
a shim to use the old variant for the older OpenSSL versions which don't
have the newer ERR_get_error_all().
The dst_key_pubcompare() and dst_key_compare() didn't have a unit test,
add the unit tests which test comparing the same keys, different keys,
and, where possible, similar keys with a manually altered parameter.
dst_key_pubcompare() internally uses the *_todns() functions of the
lib/dns/openssl*_link.c modules.
dst_key_compare() internally uses the *_compare() functions of the
lib/dns/openssl*_link.c modules.
The __builtin_expect() can be used to provide the compiler with branch
prediction information. The Gcc manual says[1] on the subject:
In general, you should prefer to use actual profile feedback for
this (-fprofile-arcs), as programmers are notoriously bad at
predicting how their programs actually perform.
Stop using __builtin_expect() and ISC_LIKELY() and ISC_UNLIKELY() macros
to provide the branch prediction information as the performance testing
shows that named performs better when the __builtin_expect() is not
being used.
1. https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html#index-_005f_005fbuiltin_005fexpect
the logfileconfig system test did not conform to the style of
other tests, and was difficult to read and maintain. it has
been cleaned up and simplifeid in several ways:
- named.args used when appropriate so that named can be started with
specified command line arguments, instead of having it launched
directly from tests.sh
- unused root zone removed from named configuration
- an existing directory used instead of using 'mkdir' to create one
- dnssec-validation disabled to stop the server sending unnecessary queries
incidental fix: removed leftover debugging printfs from logconf.c.
Remove the dynamic registration of result codes. Convert isc_result_t
from unsigned + #defines into 32-bit enum type in grand unified
<isc/result.h> header. Keep the existing values of the result codes
even at the expense of the description and identifier tables being
unnecessary large.
Additionally, add couple of:
switch (result) {
[...]
default:
break;
}
statements where compiler now complains about missing enum values in the
switch statement.
as libdns is no longer exported, it's not necessary to have
init and shutdown functions. the only purpose they served
was to create a private mctx and run dst_lib_init(), which
can be called directly instead.
- serve-stale: dig wasn't always running in background when it should.
some of the serve-stale test cases are based on groups of dig calls
running simultaneously in the background: the test pauses and resumes
running after 'wait'. in some cases the final call to dig in a group
wasn't in the background, and this sometimes caused delays that
affected later test results. in another case, a test was simplified
and made more reliable by running dig in the foreground removing a
sleep.
- serve-stale: The extension of the dig timeout period from 10 to 11
seconds in commit 5307bf64ce was left undone in a few places and has
now been completed.
- serve-stale: Resolver-query-timeout was set incorrectly. a comment
above a test case in serve-stale/tests.sh says: "We configured a long
value of 30 seconds for resolver-query-timeout," but
resolver-query-timeout was actually set to 10, not 30. this is now
fixed.
- rpz: Force retransfer of the fast-expire zone, to ensure it's fully
loaded in ns3; previously it could have been left unloaded if ns5
wasn't up yet when ns3 attempted the zone transfer.
- statistics: The TCP4SendErr counter is incremented when a TCP dispatch
is canceled while sending. depending on test timing, this may have
happened by the time the statistics are dumped. worked around by
ignoring that stat couunter when checking for errors.
- hooks: Add a prereq.sh script to prevent running under TSAN.
- zero: Disabled the servfail cache so that SERVFAIL is reported only
when there actually is a failure, not repeatedly every time the same
query is sent.
Continuing the effort to move all uses of the isc_socket API into
dispatch.c, this commit removes the dns_tcpmsg module entirely, as
dispatch was its only caller, and moves the parts of its functionality
that were being used into the dispatch module.
This code will be removed when we switch to using netmgr TCPDNS.
This commit adds support for loading DH-parameters (Diffie-Hellman
parameters) via the new "dhparam-file" option within "tls" clause. In
particular, Diffie-Hellman parameters are needed to enable the range
of forward-secrecy enabled cyphers for TLSv1.2, which are getting
silently disabled otherwise.
Previously, the Makefiles for mysql and mysqldyn DLZ modules were
generated from autoconf to get CFLAGS and LIBS for MariaDB or MySQL
libraries. The static Makefiles uses a simpler method by calling
`mysql_config` directly from the Makefile.
The old-style DLZ drivers were already marked as no longer actively
maintained and expected to be removed eventually. With the new automake
build system, the old-style DLZ drivers were not updated, and instead of
putting an effort into something that's not being maintained, let's
rather remove the unmaintained code.
Closes: #2814
The native PKCS#11 support has been removed in favour of better
maintained, more performance and easier to use OpenSSL PKCS#11 engine
from the OpenSC project.
The previous versions of BIND 9 exported its internal libraries so that
they can be used by third-party applications more easily. Certain
library functions were altered from specific BIND-only behavior to more
generic behavior when used by other applications.
This commit removes the function isc_lib_register() that was used by
external applications to enable the functionality.
