upstream/bind9
1
0
Fork 0
mirror of https://github.com/isc-projects/bind9.git synced 2026-02-26 11:32:01 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions
bind9/lib/dns
History
Ondřej Surý aa72c31422 Fix the rbt hashtable and grow it when setting max-cache-size 
There were several problems with rbt hashtable implementation:

1. Our internal hashing function returns uint64_t value, but it was
   silently truncated to unsigned int in dns_name_hash() and
   dns_name_fullhash() functions.  As the SipHash 2-4 higher bits are
   more random, we need to use the upper half of the return value.

2. The hashtable implementation in rbt.c was using modulo to pick the
   slot number for the hash table.  This has several problems because
   modulo is: a) slow, b) oblivious to patterns in the input data.  This
   could lead to very uneven distribution of the hashed data in the
   hashtable.  Combined with the single-linked lists we use, it could
   really hog-down the lookup and removal of the nodes from the rbt
   tree[a].  The Fibonacci Hashing is much better fit for the hashtable
   function here.  For longer description, read "Fibonacci Hashing: The
   Optimization that the World Forgot"[b] or just look at the Linux
   kernel.  Also this will make Diego very happy :).

3. The hashtable would rehash every time the number of nodes in the rbt
   tree would exceed 3 * (hashtable size).  The overcommit will make the
   uneven distribution in the hashtable even worse, but the main problem
   lies in the rehashing - every time the database grows beyond the
   limit, each subsequent rehashing will be much slower.  The mitigation
   here is letting the rbt know how big the cache can grown and
   pre-allocate the hashtable to be big enough to actually never need to
   rehash.  This will consume more memory at the start, but since the
   size of the hashtable is capped to `1 << 32` (e.g. 4 mio entries), it
   will only consume maximum of 32GB of memory for hashtable in the
   worst case (and max-cache-size would need to be set to more than
   4TB).  Calling the dns_db_adjusthashsize() will also cap the maximum
   size of the hashtable to the pre-computed number of bits, so it won't
   try to consume more gigabytes of memory than available for the
   database.

   FIXME: What is the average size of the rbt node that gets hashed?  I
   chose the pagesize (4k) as initial value to precompute the size of
   the hashtable, but the value is based on feeling and not any real
   data.

For future work, there are more places where we use result of the hash
value modulo some small number and that would benefit from Fibonacci
Hashing to get better distribution.

Notes:
a. A doubly linked list should be used here to speedup the removal of
   the entries from the hashtable.
b. https://probablydance.com/2018/06/16/fibonacci-hashing-the-optimization-that-the-world-forgot-or-a-better-alternative-to-integer-modulo/

(cherry picked from commit e24bc324b4)
2020-07-30 11:57:24 +02:00
..
include Fix the rbt hashtable and grow it when setting max-cache-size 2020-07-30 11:57:24 +02:00
rdata Check walking the hip rendezvous servers. 2020-07-24 15:24:49 +10:00
tests Check walking the hip rendezvous servers. 2020-07-24 15:24:49 +10:00
win32 Fix the rbt hashtable and grow it when setting max-cache-size 2020-07-30 11:57:24 +02:00
.gitignore 4394. [func] Add rndc command "dnstap-reopen" to close and 2016-06-24 09:37:04 +10:00
acl.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
adb.c Fix the rbt hashtable and grow it when setting max-cache-size 2020-07-30 11:57:24 +02:00
api prep 9.16.5 2020-07-15 23:10:55 +02:00
badcache.c Fix the rbt hashtable and grow it when setting max-cache-size 2020-07-30 11:57:24 +02:00
byaddr.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
cache.c Fix the rbt hashtable and grow it when setting max-cache-size 2020-07-30 11:57:24 +02:00
callbacks.c Merge branch '46-enforce-clang-format-rules' into 'master' 2020-02-14 08:45:59 +00:00
catz.c change "expr == false" to "!expr" in conditionals 2020-05-25 17:04:04 -07:00
client.c remove redundant rctx != NULL check 2020-07-06 10:30:25 +10:00
clientinfo.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
compress.c change 'expr == true' to 'expr' in conditionals 2020-05-25 17:03:59 -07:00
db.c Fix the rbt hashtable and grow it when setting max-cache-size 2020-07-30 11:57:24 +02:00
dbiterator.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
dbtable.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
diff.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
dispatch.c change 'expr == true' to 'expr' in conditionals 2020-05-25 17:03:59 -07:00
dlz.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
dns64.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
dnsrps.c Fix the rbt hashtable and grow it when setting max-cache-size 2020-07-30 11:57:24 +02:00
dnssec.c Use the new sorting rules to regroup #include headers 2020-03-11 08:55:12 +00:00
dnstap.c Fix rndc dnstap -roll N 2020-06-29 22:30:01 +00:00
dnstap.proto fix spelling errors reported by Fossies. 2020-02-21 07:05:31 +00:00
ds.c Stop leaking OpenSSL types and defines in the isc/md.h 2020-04-28 15:24:07 +02:00
dst_api.c Move dst key printtime in separate function 2020-07-01 09:57:44 +02:00
dst_internal.h Finish refactoring after the removal of --with-ecdsa and --with-eddsa. 2020-05-01 06:54:26 +02:00
dst_openssl.h Use the new sorting rules to regroup #include headers 2020-03-11 08:55:12 +00:00
dst_parse.c Use the new sorting rules to regroup #include headers 2020-03-11 08:55:12 +00:00
dst_parse.h Merge branch '46-enforce-clang-format-rules' into 'master' 2020-02-14 08:45:59 +00:00
dst_pkcs11.h Merge branch '46-just-use-clang-format-to-reformat-sources' into 'master' 2020-02-12 14:51:18 +00:00
dst_result.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
dyndb.c change 'expr == true' to 'expr' in conditionals 2020-05-25 17:03:59 -07:00
ecdb.c Merge branch '46-enforce-clang-format-rules' into 'master' 2020-02-14 08:45:59 +00:00
ecs.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
fixedname.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
forward.c Merge branch '46-enforce-clang-format-rules' into 'master' 2020-02-14 08:45:59 +00:00
gen-unix.h Use the new sorting rules to regroup #include headers 2020-03-11 08:55:12 +00:00
gen-win32.h Add C11 localtime_r and gmtime_r shims for Windows 2020-03-17 15:33:24 -07:00
gen.c Add C11 localtime_r and gmtime_r shims for Windows 2020-03-17 15:33:24 -07:00
geoip2.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
gssapi_link.c Use the new sorting rules to regroup #include headers 2020-03-11 08:55:12 +00:00
gssapictx.c Use the new sorting rules to regroup #include headers 2020-03-11 08:55:12 +00:00
hmac_link.c Stop leaking OpenSSL types and defines in the isc/md.h 2020-04-28 15:24:07 +02:00
ipkeylist.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
iptable.c Merge branch '46-enforce-clang-format-rules' into 'master' 2020-02-14 08:45:59 +00:00
journal.c Merge branch '46-enforce-clang-format-rules' into 'master' 2020-02-14 08:45:59 +00:00
kasp.c Fix race condition dnssec-policy with views 2020-03-09 16:25:35 +01:00
key.c Use the new sorting rules to regroup #include headers 2020-03-11 08:55:12 +00:00
keydata.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
keymgr.c Check return value of dst_key_getbool() 2020-07-14 17:48:21 +02:00
keytable.c The dsset returned by dns_keynode_dsset needs to be thread safe. 2020-06-11 16:09:43 +10:00
Kyuafile regen master 2017-12-29 01:44:18 +00:00
lib.c change "expr == false" to "!expr" in conditionals 2020-05-25 17:04:04 -07:00
log.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
lookup.c Merge branch '46-enforce-clang-format-rules' into 'master' 2020-02-14 08:45:59 +00:00
Makefile.in Resolve the overlinking of the system libraries 2020-05-11 09:49:54 +02:00
mapapi improve calculation of database size 2020-03-12 00:38:37 -07:00
master.c Reject primary zones with an DS record at the zone apex. 2020-06-04 16:06:07 +02:00
masterdump.c change "expr == false" to "!expr" in conditionals 2020-05-25 17:04:04 -07:00
message.c Add +yaml support for EDE 2020-06-05 10:50:58 +10:00
name.c Fix the rbt hashtable and grow it when setting max-cache-size 2020-07-30 11:57:24 +02:00
ncache.c Merge branch '46-enforce-clang-format-rules' into 'master' 2020-02-14 08:45:59 +00:00
nsec.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
nsec3.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
nta.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
openssl_link.c Use the new sorting rules to regroup #include headers 2020-03-11 08:55:12 +00:00
openssldh_link.c Remove unnecessary forward declarations. 2020-05-01 06:54:26 +02:00
opensslecdsa_link.c Add initial support for ECDSA keys via OpenSSL PKCS#11 engine 2020-05-01 14:30:04 +02:00
openssleddsa_link.c Add engine support to OpenSSL EdDSA implementation. 2020-05-01 16:25:56 +02:00
opensslrsa_link.c Simplify error handling 2020-05-01 14:30:04 +02:00
order.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
peer.c Merge branch '46-enforce-clang-format-rules' into 'master' 2020-02-14 08:45:59 +00:00
pkcs11.c Use the new sorting rules to regroup #include headers 2020-03-11 08:55:12 +00:00
pkcs11ecdsa_link.c Use switch instead of if when evaluating curves 2020-05-01 06:54:27 +02:00
pkcs11eddsa_link.c Update EdDSA implementation to PKCS#11 v3.0. 2020-05-01 08:00:52 +02:00
pkcs11rsa_link.c Remove unnecessary forward declarations. 2020-05-01 06:54:26 +02:00
portlist.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
private.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
rbt.c Fix the rbt hashtable and grow it when setting max-cache-size 2020-07-30 11:57:24 +02:00
rbtdb.c Fix the rbt hashtable and grow it when setting max-cache-size 2020-07-30 11:57:24 +02:00
rbtdb.h Merge branch '46-add-curly-braces' into 'master' 2020-02-13 21:28:35 +00:00
rcode.c Use the new sorting rules to regroup #include headers 2020-03-11 08:55:12 +00:00
rdata.c Adjust range limit of unknown meta types 2020-07-08 13:44:47 +10:00
rdatalist.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
rdatalist_p.h Merge branch '46-just-use-clang-format-to-reformat-sources' into 'master' 2020-02-12 14:51:18 +00:00
rdataset.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
rdatasetiter.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
rdataslab.c improve calculation of database size 2020-03-12 00:38:37 -07:00
request.c Merge branch '46-enforce-clang-format-rules' into 'master' 2020-02-14 08:45:59 +00:00
resolver.c Fix idle timeout for connected TCP sockets 2020-07-30 11:16:09 +02:00
result.c Reject primary zones with an DS record at the zone apex. 2020-06-04 16:06:07 +02:00
rootns.c adjust the clang-format penalties to reduce string breaking 2020-02-17 14:43:46 -08:00
rpz.c Fix rpz wildcard name matching 2020-07-27 17:02:16 -03:00
rriterator.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
rrl.c Fix the rbt hashtable and grow it when setting max-cache-size 2020-07-30 11:57:24 +02:00
sdb.c Fix the rbt hashtable and grow it when setting max-cache-size 2020-07-30 11:57:24 +02:00
sdlz.c Fix the rbt hashtable and grow it when setting max-cache-size 2020-07-30 11:57:24 +02:00
soa.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
spnego.asn1 Remove $Id markers, Principal Author and Reviewed tags from the full source tree 2018-05-11 13:17:46 +02:00
spnego.c Address compilation warnings on FreeBSD 11.4 2020-06-29 12:03:01 +02:00
spnego.h Merge branch '46-add-curly-braces' into 'master' 2020-02-13 21:28:35 +00:00
spnego_asn1.c Merge branch '46-enforce-clang-format-rules' into 'master' 2020-02-14 08:45:59 +00:00
spnego_asn1.pl Remove $Id markers, Principal Author and Reviewed tags from the full source tree 2018-05-11 13:17:46 +02:00
ssu.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
ssu_external.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
stats.c Replace hard coded value with constant 2020-04-03 10:04:24 +02:00
tcpmsg.c Merge branch '46-enforce-clang-format-rules' into 'master' 2020-02-14 08:45:59 +00:00
time.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
timer.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
tkey.c change 'expr == true' to 'expr' in conditionals 2020-05-25 17:03:59 -07:00
tsec.c Use the new sorting rules to regroup #include headers 2020-03-11 08:55:12 +00:00
tsig.c change "expr == false" to "!expr" in conditionals 2020-05-25 17:04:04 -07:00
tsig_p.h Merge branch '46-just-use-clang-format-to-reformat-sources' into 'master' 2020-02-12 14:51:18 +00:00
ttl.c Merge branch '46-enforce-clang-format-rules' into 'master' 2020-02-14 08:45:59 +00:00
update.c change "expr == false" to "!expr" in conditionals 2020-05-25 17:04:04 -07:00
validator.c The validator could fail when select_signing_key/get_dst_key failed 2020-06-25 22:42:43 +10:00
version.c Merge branch '46-just-use-clang-format-to-reformat-sources' into 'master' 2020-02-12 14:51:18 +00:00
view.c Fix the rbt hashtable and grow it when setting max-cache-size 2020-07-30 11:57:24 +02:00
xfrin.c change "expr == false" to "!expr" in conditionals 2020-05-25 17:04:04 -07:00
zone.c kasp tests: fix wait for reconfig done 2020-06-29 08:09:40 +02:00
zone_p.h Merge branch '46-enforce-clang-format-rules' into 'master' 2020-02-14 08:45:59 +00:00
zonekey.c Merge branch 'each-style-tweak' into 'master' 2020-02-14 05:35:29 +00:00
zoneverify.c The dsset returned by dns_keynode_dsset needs to be thread safe. 2020-06-11 16:09:43 +10:00
zt.c Merge branch '46-enforce-clang-format-rules' into 'master' 2020-02-14 08:45:59 +00:00