Caching prevents server upgrades being detected in a timely manner
and it can also prevent DNSSEC responses being requested.
(cherry picked from commit 90b2f94d9b)
TSAN reports a lock-order-inversion (potential deadlock) issue in
add_trace_entry().
While it is true that in one case a lock in the 'isc_mem_t' structure is
locked first, and then a lock in the 'FILE' structure is locked second,
and in the the second case it is the other way around, this isn't an
issue, because those are 'FILE' structures for totally different files,
used in different parts of the code.
Closes#5266
Backport of MR !10355
Merge branch 'backport-5266-freebsd-suppress-tsan-lock-order-inversion-false-positive-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!10371
TSAN reports a lock-order-inversion (potential deadlock) issue in
add_trace_entry():
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock)
Cycle in lock order graph: M0001 (0x000000000001) => M0002 (0x000000000002) => M0001
Mutex M0002 acquired here while holding mutex M0001 in main thread:
#0 _pthread_mutex_lock /usr/src/contrib/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1342:3
#1 add_trace_entry lib/isc/mem.c:210:2
#2 isc__mem_get lib/isc/mem.c:606:2
#3 isc_buffer_allocate lib/isc/./include/isc/buffer.h:1080:23
#4 pushandgrow lib/isc/lex.c:321:3
#5 isc_lex_gettoken lib/isc/lex.c:445:22
#6 cfg_gettoken lib/isccfg/parser.c:3490:11
#7 cfg_parse_mapbody lib/isccfg/parser.c:2230:3
#8 cfg_parse_obj lib/isccfg/parser.c:247:11
#9 parse2 lib/isccfg/parser.c:628:11
#10 cfg_parse_file lib/isccfg/parser.c:668:11
#11 load_configuration bin/named/server.c:8069:13
#12 run_server bin/named/server.c:9518:2
#13 isc__async_cb lib/isc/async.c:110:3
#14 uv__async_io /tmp/libuv-1.50.0/src/unix/async.c:208:5
#15 uv__io_poll /tmp/libuv-1.50.0/src/unix/kqueue.c:369:9
#16 uv_run /tmp/libuv-1.50.0/src/unix/core.c:460:5
#17 loop_thread lib/isc/loop.c:327:6
#18 thread_body lib/isc/thread.c:89:8
#19 isc_thread_main lib/isc/thread.c:124:2
#20 isc_loopmgr_run lib/isc/loop.c:513:2
#21 main bin/named/main.c:1469:2
Mutex M0001 previously acquired by the same thread here:
#0 _pthread_mutex_lock /usr/src/contrib/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1342:3
#1 _flockfile /usr/src/lib/libc/stdio/_flock_stub.c:72:3
#2 cfg_gettoken lib/isccfg/parser.c:3490:11
#3 cfg_parse_mapbody lib/isccfg/parser.c:2230:3
#4 cfg_parse_obj lib/isccfg/parser.c:247:11
#5 parse2 lib/isccfg/parser.c:628:11
#6 cfg_parse_file lib/isccfg/parser.c:668:11
#7 load_configuration bin/named/server.c:8069:13
#8 run_server bin/named/server.c:9518:2
#9 isc__async_cb lib/isc/async.c:110:3
#10 uv__async_io /tmp/libuv-1.50.0/src/unix/async.c:208:5
#11 uv__io_poll /tmp/libuv-1.50.0/src/unix/kqueue.c:369:9
#12 uv_run /tmp/libuv-1.50.0/src/unix/core.c:460:5
#13 loop_thread lib/isc/loop.c:327:6
#14 thread_body lib/isc/thread.c:89:8
#15 isc_thread_main lib/isc/thread.c:124:2
#16 isc_loopmgr_run lib/isc/loop.c:513:2
#17 main bin/named/main.c:1469:2
Mutex M0001 acquired here while holding mutex M0002 in main thread:
#0 _pthread_mutex_lock /usr/src/contrib/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1342:3
#1 _flockfile /usr/src/lib/libc/stdio/_flock_stub.c:72:3
#2 print_active lib/isc/mem.c:629:3
#3 isc_mem_stats lib/isc/mem.c:694:2
#4 main bin/named/main.c:1498:4
Mutex M0002 previously acquired by the same thread here:
#0 _pthread_mutex_lock /usr/src/contrib/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1342:3
#1 isc_mem_stats lib/isc/mem.c:668:2
#2 main bin/named/main.c:1498:4
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) lib/isc/mem.c:210:2 in add_trace_entry
In the first stack frame ('M0001'->'M0002' lock order) cfg_gettoken()
uses flockfile() to lock 'M0001' for the 'FILE' object associated with
the configuration file (e.g. the configuration file itself and
whatever it includes, like a zone database), then it locks a memory
context mutex M0002.
In the other stack frmae ('M0002'->'M0001' lock order) isc_mem_stats()
locks a memory context mutex M0002, then it uses fprintf(), which
internally locks a 'M0001' mutex with flockfile() to write into the
'named.memstats' memory statistics file.
While it is true that in one case a lock in the 'isc_mem_t' structure is
locked first, and then a lock in the 'FILE' structure is locked second,
and in the the second case it is the other way around, this isn't an
issue, because those are 'FILE' structures for totally different files,
used in different parts of the code.
It was also manually confirmed that 'named.memstats' doesn't get
processed by cfg_gettoken(), and is used only in the second stack
frame's code flow when named is exiting.
(cherry picked from commit 33dff63cbb)
TSAN reports a data race in FreeBSD's memset(), called by its
__crt_calloc() memory allocation function. There is a very similar
bug report [1] in FreeBSD bug tracker, and an existing code-review [2]
that tries to address an issue, the description of which is very
similar to what we are seeing.
Suppress this report by adding its signature to '.tsan-suppress'.
[1] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=282794
[2] https://reviews.freebsd.org/D28536?id=86694Closes#5267
Backport of MR !10353
Merge branch 'backport-5267-freebsd-suppress-tsan-data-race-false-positive-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!10368
TSAN reports a data race in FreeBSD's memset(), called by its
__crt_calloc() memory allocation function. There is a very similar
bug report [1] in FreeBSD bug tracker, and an existing code-review [2]
that tries to address an issue, the description of which is very
similar to what we are seeing.
Suppress this report by adding its signature to '.tsan-suppress'.
[1] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=282794
[2] https://reviews.freebsd.org/D28536?id=86694
(cherry picked from commit 40a91f51d7)
For pipelines in the private repository, add an optional manual job,
which allows the current branch to be pushed into the specified
customer's git repository. This can be useful to provide patch previews
for early testing.
For tags created in a private repository, add a manual job which pushes
the created tag to all entitled customers.
Backport of MR !10323
Merge branch 'backport-nicki/ci-customer-git-automation-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!10357
For pipelines in the private repository, add an optional manual job,
which allows the current branch to be pushed into the specified
customer's git repository. This can be useful to provide patch previews
for early testing.
For tags created in a private repository, add a manual job which pushes
the created tag to all entitled customers.
(cherry picked from commit 378b412e94)
When query responses timed out, the resolver could incorrectly increase the regular responses counters, even if no response was received. This has been fixed.
Closes#5193
Backport of MR !10227
Merge branch 'backport-5193-resolver-statistics-counters-fix-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!10288
Add a test to check that the timed out responses do not skew the
normal responses statistics counters, and that they do update the
timeouts counter.
(cherry picked from commit 0c7fa8d572)
When a response times out the fctx_cancelquery() function
incorrectly calculates it in the 'dns_resstatscounter_queryrtt5'
counter (i.e. >=1600 ms). To avoid this, the rctx_timedout()
function should make sure that 'rctx->finish' is NULL. And in order
to adjust the RTT values for the timed out server, 'rctx->no_response'
should be true. Update the rctx_timedout() function to make those
changes.
(cherry picked from commit 830e548111)
The resquery_response() function increases the response counter without
checking if the response was successful. Increase the counter only when
the result indicates success.
(cherry picked from commit 12e7dfa397)
These changelogs meant for -S edition were introduced to avoid rebase
conflicts. However, the same result can be achieved by linking the -S
changelogs directly from their open-source variants, rather than
including the -S changelogs directly in changelog.rst.
Mattermost discussion: https://mattermost.isc.org/isc/pl/1gioy1gatifttborqx5qhc9paw
Merge branch 'nicki/remove-s-specific-changelog-files-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!10337
These changelogs meant for -S edition were introduced to avoid rebase
conflicts. However, the same result can be achieved by linking the -S
changelogs directly from their open-source variants, rather than
including the -S changelogs directly in changelog.rst.
Previously, in some cases, the resolver could return rdatasets of type CNAME or DNAME without the result code being set to `DNS_R_CNAME` or `DNS_R_DNAME`. This could trigger an assertion failure in the ADB. The resolver error has been fixed.
Closes#5201
Backport of MR !10172
Backport of MR !10178
Merge branch 'backport-5201-adb-cname-error-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!10330
the fix in commit 1edbbc32b4 was incomplete; the wrong
event result could also be set in cache_name() and validated().
(cherry picked from commit 9ebeb60174)
when the caching of a negative record failed because of the
presence of a positive one, ncache_adderesult() could override
this to ISC_R_SUCCESS. this could cause CNAME and DNAME responses
to be handled incorrectly. ncache_adderesult() now sets the result
code correctly in such cases.
(cherry picked from commit 1edbbc32b4)
Otherwise curl 8.13 rejects the line with:
I:Check HTTP/1.1 keep-alive with truncated stream (21)
curl: option --header: error encountered when reading a file
curl: try 'curl --help' or 'curl --manual' for more information
Also, see https://github.com/curl/curl/pull/16572.
Closes#5249
Backport of MR !10319
Merge branch 'backport-5249-statschannel-limit-http-header-size-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!10322
Otherwise curl 8.13 rejects the line with:
I:Check HTTP/1.1 keep-alive with truncated stream (21)
curl: option --header: error encountered when reading a file
curl: try 'curl --help' or 'curl --manual' for more information
Also, see https://github.com/curl/curl/pull/16572.
(cherry picked from commit bcb931f4a8)
All DNSKEY keys are able to authenticate. The `DNS_KEYTYPE_NOAUTH` (and `DNS_KEYTYPE_NOCONF`) flags were defined for the KEY rdata type, and are not applicable to DNSKEY. Previously, however, because the DNSKEY implementation was built on top of KEY, the `_NOAUTH` flag prevented authentication in DNSKEYs as well. This has been corrected.
Closes#5240
Backport of MR !10261
Merge branch 'backport-5240-ignore-noauth-flag-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!10316
All DNSKEY keys are able to authenticate. The DNS_KEYTYPE_NOAUTH
(and DNS_KEYTYPE_NOCONF) flags were defined for the KEY rdata type,
and are not applicable to DNSKEY.
Previously, because the DNSKEY implementation was built on top of
KEY, the NOAUTH flag prevented authentication in DNSKEYs as well.
This has been corrected.
(cherry picked from commit 5c21576f82)
Use enums for DNS_KEYFLAG_, DNS_KEYTYPE_, DNS_KEYOWNER_, DNS_KEYALG_,
and DNS_KEYPROTO_ values.
Remove values that are never used.
Eliminate the obsolete DNS_KEYFLAG_SIGNATORYMASK. Instead, add three
more RESERVED bits for the key flag values that it covered but which
were never used.
(cherry picked from commit fee1ba40df)
This commit simplifies code flow in the tls_cycle_input() and makes
the incoming data processing similar to that in TCP DNS. In
particular, now we decipher all the the incoming data before making a
single isc__nm_process_sock_buffer() call. Previously we would try to
decipher data bit-by-bit before trying to process the deciphered bit
via isc__nm_process_sock_buffer(). Doing like before made the code
much less predictable, in particular in the areas like when reading is
paused or resumed.
The newer approach also allowed us to get rid of some old kludges.
Closes#5247
Merge branch '5247-unexpected-callbacks' into 'bind-9.18'
See merge request isc-projects/bind9!10294
This commit simplifies code flow in the tls_cycle_input() and makes
the incoming data processing similar to that in TCP DNS. In
particular, now we decipher all the the incoming data before making a
single isc__nm_process_sock_buffer() call. Previously we would try to
decipher data bit-by-bit before trying to process the deciphered bit
via isc__nm_process_sock_buffer(). Doing like before made the code
much less predictable, in particular in the areas like when reading is
paused or resumed.
The newer approach also allowed us to get rid of some old kludges.
The false positive rate is about 10-20 % when evaluating shotgun results
from a single run. Attempt to reduce the false positive rate by allowing
a re-run of failed jobs.
Backport of MR !10271
Merge branch 'backport-nicki/ci-shotgun-reduce-false-positives-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!10280
The false positive rate is about 10-20 % when evaluating shotgun results
from a single run. Attempt to reduce the false positive rate by allowing
a re-run of failed jobs.
While there is a slight risk that barely noticable decreases in
performance might slip by more easily in MRs, they'd still likely pop up
during nightly or pre-release testing.
Also increase the tolerance threshold for DoH latency comparisons, as
those tests often experience increased jitter in the tail end latencies.
(cherry picked from commit 5eab352478)
When `-T cookiealwaysvalid` is passed to `named`, DNS cookie checks for
the incoming queries always pass, given they are structurally correct.
Backport of MR !10232
Merge branch 'backport-aram/new-named-minus-T-option-of-cookiealwaysvalid-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!10265
When -T cookiealwaysvalid is passed to named, DNS cookie checks for
the incoming queries always pass, given they are structurally correct.
(cherry picked from commit 807ef8545d)
Add missing locks in dns_zone_getxfrsource4 et al. Addresses CID 468706, 468708, 468741, 468742, 468785, and 468778.
Cleanup dns_zone_setxfrsource4 et al to now return void.
Remove double copies with dns_zone_getprimaryaddr and dns_zone_getsourceaddr.
Closes#4933
Backport of MR !9485
Merge branch 'backport-4933-add-missing-locks-when-returning-addresses-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!10260
Add missing locks in dns_zone_getxfrsource4 et al. Addresses CID
468706, 468708, 468741, 468742, 468785 and 468778.
Cleanup dns_zone_setxfrsource4 et al to now return void.
Remove double copies with dns_zone_getprimaryaddr and dns_zone_getsourceaddr.
(cherry picked from commit d0a59277fb)
The `I:checking that lifting the limit will allow everything to get
cached (20)` test was failing due to the TTL of the records being
too short for the elapsed time of the test. Raise the TTL to fix
this and adjust other tests as needed.
Closes#5206
Backport of MR !10177
Merge branch 'backport-5206-tune-last-sub-test-of-reclimit-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!10250
The 'I:checking that lifting the limit will allow everything to get
cached (20)' test was failing due to the TTL of the records being
too short for the elapsed time of the test. Raise the TTL to fix
this and adjust other tests as needed.
(cherry picked from commit 1a58bd2113)
From technical reasons --with-readline=libedit is not being tested on
FreeBSD anymore as it's hard to have anchors both unified and specific.
(cherry picked from commit e0df774ca0)
ZONEMD digests RRSIG records and potentially digests SIG record. Add digests
methods for both record types.
Closes#5219
Backport of MR !10217
Merge branch 'backport-5219-add-digest-methods-for-sig-and-rrsig-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!10219
ZONEMD needs to be able to digest SIG and RRSIG records. The signer
field can be compressed in SIG so we need to call dns_name_digest().
While for RRSIG the records the signer field is not compressed the
canonical form has the signer field downcased (RFC 4034, 6.2). This
also implies that compare_rrsig needs to downcase the signer field
during comparison.
(cherry picked from commit 006c5990ce)
