Commit graph

10740 commits

Author SHA1 Message Date
Ilya Shipitsin
b8888ab557 CLEANUP: assorted typo fixes in the code and comments
This is 15th iteration of typo fixes
2021-01-06 17:32:03 +01:00
Amaury Denoyelle
10d5c3172b BUG/MINOR: srv: do not cleanup idle conns if pool max is null
If a server is configured to not have any idle conns, returns immediatly
from srv_cleanup_connections. This avoids a segfault when a server is
configured with pool-max-conn to 0.

This should be backported up to 2.2.
2021-01-06 16:57:17 +01:00
Amaury Denoyelle
e3c4192962 BUG/MINOR: srv: do not init address if backend is disabled
Do not proceed on init_addr if the backend of the server is marked as
disabled. When marked as disabled, the server is not fully initialized
and some operation must be avoided to prevent segfault. It is correct
because there is no way to activate a disabled backend.

This fixes the github issue #1031.
This should be backported to 2.2.
2021-01-06 16:57:17 +01:00
Ilya Shipitsin
1e9a66603f CLEANUP: assorted typo fixes in the code and comments
This is 14th iteration of typo fixes
2021-01-06 16:26:50 +01:00
Frédéric Lécaille
153d4a89d0 BUG/MINOR: quic: NULL pointer dereferences when building post handshake frames.
The second one was detected by cppcheck contrary to the first one.
Fixes issue #1032.
Thank you to Ilya for having reported this.
2021-01-06 13:59:05 +01:00
Christopher Faulet
de79cd28ec BUG/MINOR: stats: Make stat_l variable used to dump a stat line thread local
Since ee63d4bd6 ("MEDIUM: stats: integrate static proxies stats in new
stats"), all dumped stats for a given domain, the default ones and the
modules ones, are merged in a signle array to dump them in a generic way.
For this purpose, the stat_l global variable is allocated at startup to
store a line of stats before the dump, i.e. all stats of an entity
(frontend, backend, listener, server or dns nameserver). But this variable
is not thread safe.  If stats are retrieved concurrently by several clients
on different threads, the same variable is used. This leads to corrupted
stats output.

To fix the bug, the stat_l variable is now thread local.

This patch should probably solve issues #972 and #992. It must be backported
to 2.3.
2021-01-06 10:34:12 +01:00
Thayne McCombs
24da7e1aa6 BUG/MEDIUM: server: srv_set_addr_desc() crashes when a server has no address
GitHub Issue #1026 reported a crash during configuration check for the
following example config:

    backend 0
    server 0 0
    server 0 0

HAProxy crashed in srv_set_addr_desc() due to a NULL pointer dereference
caused by `sa2str` returning NULL for an `AF_UNSPEC` address (`0`).

Check to make sure the address key is non-null before using it for
comparison or inserting it into the tree.

The crash was introduced in commit 92149f9a8 ("MEDIUM: stick-tables: Add
srvkey option to stick-table") which not in any released version so no
backport is needed.

Cc: Tim Duesterhus <tim@bastelstu.be>
2021-01-06 09:19:15 +01:00
Christopher Faulet
8d4977ae86 BUG/MINOR: tcpcheck: Report a L7OK if the last evaluated rule is a send rule
When all rules of a tcpcheck ruleset are successfully evaluated, the right
check status must always be reported. It is true if the last evaluated rule
is an expect or a connect rule. But not if it is a send rule. In this
situation, nothing more is done until the check timeout expiration and a
L7TOUT is reported instead of a L7OK.

Now, by default, when all rules were successfully evaluated, a L7OK is
reported. When the last evaluated rule is an expect or a connect, the
behavior remains unchanged.

This patch should fix the issue #1027. It must be backported as far as 2.2.
2021-01-05 17:31:49 +01:00
William Dauchy
afb9368221 CLEANUP: spoe: fix typo on var_check_arg comment
there was an extra `s` added to the `var_check_arg` function

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-01-05 17:23:32 +01:00
Tim Duesterhus
c294284e33 CLEANUP: Reduce scope of hdr_age in http_action_store_cache()
This is only required to process the `age` header.
2021-01-05 17:05:58 +01:00
Tim Duesterhus
e2fff10a19 CLEANUP: Reduce scope of header_name in http_action_store_cache()
This variable is only needed deeply nested in a single location and clang's
static analyzer complains about a dead initialization. Reduce the scope to
satisfy clang and the human that reads the function.
2021-01-05 17:05:58 +01:00
Willy Tarreau
8f7efcddd6 CLEANUP: mworker: remove duplicate pointer tests in cfg_parse_program()
As reported in issue #1017, there are two harmless duplicate tests in
cfg_parse_program(), one made of a "if" using the same condition as the
loop it's in, and the other one being a null test before a free. This
just removes them. No backport is needed.
2021-01-05 15:58:37 +01:00
Tim Duesterhus
5ce5a1586d BUG/MINOR: cfgparse: Fail if the strdup() for rule->be.name for use_backend fails
This patch fixes GitHub issue #1024.

I could track the `strdup` back to commit
3a1f5fda10 which is 1.9-dev8. It's probably not
worth the effort to backport it across this refactoring.

This patch should be backported to 1.9+.
2021-01-05 11:37:41 +01:00
Willy Tarreau
b6fc524f05 MINOR: ssl: make tlskeys_list_get_next() take a list element
As reported in issue #1010, gcc-11 as of 2021-01-05 is overzealous in
its -Warray-bounds check as it considers that a cast of a global struct
accesses the entire struct even if only one specific element is accessed.
This instantly breaks all lists making use of container_of() to build
their iterators as soon as the starting point is known if the next
element is retrieved from the list head in a way that is visible to the
compiler's optimizer, because it decides that accessing the list's next
element dereferences the list as a larger struct (which it does not).

The temporary workaround consisted in disabling -Warray-bounds, but this
warning is traditionally quite effective at spotting real bugs, and we
actually have is a single occurrence of this issue in the whole code.

By changing the tlskeys_list_get_next() function to take a list element
as the starting point instead of the current element, we can avoid
the starting point issue but this requires to change all call places
to write hideous casts made of &((struct blah*)ref)->list. At the
moment we only have two such call places, the first one being used to
initialize the list (which is the one causing the warning) and which
is thus easy to simplify, and the second one for which we already have
an aliased pointer to the reference that is still valid at the call
place, and given the original pointer also remained unchanged, we can
safely use this alias, and this is safer than leaving a cast there.

Let's make this change now while it's still easy.

The generated code only changed in function cli_io_handler_tlskeys_files()
due to register allocation and the change of variable scope between the
old one and the new one.
2021-01-05 11:15:45 +01:00
Tim Duesterhus
cb8b281c02 CLEANUP: ssl: Remove useless local variable in tlskeys_list_get_next()
`getnext` was only used to fill `ref` at the beginning of the function. Both
have the same type. Replace the parameter name by `ref` to remove the useless
local variable.
2021-01-05 10:25:20 +01:00
Tim Duesterhus
2c7bb33144 CLEANUP: ssl: Remove useless loop in tlskeys_list_get_next()
This loop was always exited in the first iteration by `return`.
2021-01-05 10:24:36 +01:00
Frédéric Lécaille
242fb1b639 MINOR: quic: Drop packets with STREAM frames with wrong direction.
A server initiates streams with odd-numbered stream IDs.
Also add useful traces when parsing STREAM frames.
2021-01-04 12:31:28 +01:00
Frédéric Lécaille
129a351a3f BUG/MINOR: quic: Wrong STREAM frames parsing.
After having re-read the RFC, we noticed there are two bugs in the STREAM
frame parser. When the OFF bit (0x04) in the frame type is not set
we must set the offset to 0 (it was not set at all). When the LEN bit (0x02)
is not set we must extend the length of the data field to the end of the packet
(it was not set at all).
2021-01-04 12:31:28 +01:00
Frédéric Lécaille
50044adc60 MINOR: quic: Pass quic_conn struct to frame parsers.
This is only for debugging purposes.
2021-01-04 12:31:28 +01:00
Frédéric Lécaille
ea60499912 BUG/MINOR: quic: Possible CRYPTO frame building errors.
This is issue is due to the fact that when we call the function
responsible of building CRYPTO frames to fill a buffer, the Length
field of this packet did not take into an account the trailing 16 bytes for
the AEAD tag. Furthermore, the remaining <room> available in this buffer
was not decremented by the CRYPTO frame length, but only by the CRYPTO data length
of this frame.
2021-01-04 12:31:28 +01:00
Frédéric Lécaille
6c1e36ce55 CLEANUP: quic: Remove useless QUIC event trace definitions.
Remove QUIC_EV_CONN_E* event trace macros which were defined for  errors.
Replace QUIC_EV_CONN_ECHPKT by QUIC_EV_CONN_BCFRMS used in qc_build_cfrms()
2021-01-04 12:31:28 +01:00
Frédéric Lécaille
d341fc3609 CLEANUP: qpack: Wrong comment about the draft for QPACK static header table.
This came with a "copy and paste" from the definition for HPACK.
2021-01-04 12:31:28 +01:00
Frédéric Lécaille
164096eb76 MINOR: qpack: Add static header table definitions for QPACK.
As HPACK, QPACK makes usage of a static header table.
2021-01-04 12:31:28 +01:00
Tim Duesterhus
e5ff14100a CLEANUP: Compare the return value of XXXcmp() functions with zero
According to coding-style.txt it is recommended to use:

`strcmp(a, b) == 0` instead of `!strcmp(a, b)`

So let's do this.

The change was performed by running the following (very long) coccinelle patch
on src/:

    @@
    statement S;
    expression E;
    expression F;
    @@

      if (
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) != 0
      )
    (
      S
    |
      { ... }
    )

    @@
    statement S;
    expression E;
    expression F;
    @@

      if (
    - !
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) == 0
      )
    (
      S
    |
      { ... }
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    G &&
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) != 0
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    G ||
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) != 0
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) != 0
    && G
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) != 0
    || G
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    G &&
    - !
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) == 0
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    G ||
    - !
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) == 0
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    - !
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) == 0
    && G
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    - !
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) == 0
    || G
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    - !
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) == 0
    )
2021-01-04 10:09:02 +01:00
Tim Duesterhus
f89d43a381 MINOR: lua: Use consistent error message 'memory allocation failed'
Other locations in the configuration parser use 'memory allocation failed', so
use this one as well.
2021-01-03 20:37:16 +01:00
Tim Duesterhus
621e74afd1 BUG/MINOR: lua: Fix memory leak error cases in hlua_config_prepend_path
In case of an error `p` is not properly freed.

Minor leak during configuration parsing in out of memory situations, no
backport needed.
2021-01-03 20:37:16 +01:00
Tim Duesterhus
a7ebffef66 BUG/MINOR: sink: Return an allocation failure in __sink_new if strdup() fails
This patch fixes GitHub issue #1023.

The function was introduced in commit 99c453d ("MEDIUM: ring: new
section ring to declare custom ring buffers."), which first appeared
in 2.2-dev9. The fix should be backported to 2.2+.
2021-01-03 20:35:45 +01:00
Thayne McCombs
92149f9a82 MEDIUM: stick-tables: Add srvkey option to stick-table
This allows using the address of the server rather than the name of the
server for keeping track of servers in a backend for stickiness.

The peers code was also extended to support feeding the dictionary using
this key instead of the name.

Fixes #814
2020-12-31 10:04:54 +01:00
Tim Duesterhus
dc38bc4a1a BUG/MEDIUM: cache: Fix hash collision in accept-encoding handling for Vary
This patch fixes GitHub Issue #988. Commit ce9e7b2521
was not sufficient, because it fell back to a hash comparison if the bitmap
of known encodings was not acceptable instead of directly returning the the
cached response is not compatible.

This patch also extends the reg-test to test the hash collision that was
mentioned in #988.

Vary handling is 2.4, no backport needed.
2020-12-31 09:39:08 +01:00
Remi Tricot-Le Breton
e6cc5b5974 MINOR: cache: Replace the "process-vary" option's expected values
Replace the <0/1> expected values of the process-vary option by a more
usual <on/off> pair.
2020-12-24 17:18:00 +01:00
Remi Tricot-Le Breton
42efffd7f6 MINOR: cache: Remove redundant test in http_action_req_cache_use
The suppressed check is fully covered by the next one and can then be
removed.
2020-12-24 17:18:00 +01:00
Remi Tricot-Le Breton
ce9e7b2521 MEDIUM: cache: Manage a subset of encodings in accept-encoding normalizer
The accept-encoding normalizer now explicitely manages a subset of
encodings which will all have their own bit in the encoding bitmap
stored in the cache entry. This way two requests with the same primary
key will be served the same cache entry if they both explicitely accept
the stored response's encoding, even if their respective secondary keys
are not the same and do not match the stored response's one.
The actual hash of the accept-encoding will still be used if the
response's encoding is unmanaged.
The encoding matching and the encoding weight parsing are done for every
subpart of the accept-encoding values, and a bitmap of accepted
encodings is built for every request. It is then tested upon any stored
response that has the same primary key until one with an accepted
encoding is found.
The specific "identity" and "*" accept-encoding values are managed too.
When storing a response in the key, we also parse the content-encoding
header in order to only set the response's corresponding encoding's bit
in its cache_entry encoding bitmap.

This patch fixes GitHub issue #988.
It does not need to be backported.
2020-12-24 17:18:00 +01:00
Remi Tricot-Le Breton
56e46cb393 MINOR: http: Add helper functions to trim spaces and tabs
Add two helper functions that trim leading or trailing spaces and
horizontal tabs from an ist string.
2020-12-24 17:18:00 +01:00
Remi Tricot-Le Breton
6a34b2b65d MINOR: cache: Add specific secondary key comparison mechanism
Add the possibility to define custom comparison functions for every
sub-part of the secondary key hash instead of using a global memcmp.
2020-12-24 17:18:00 +01:00
Remi Tricot-Le Breton
e4421dec7e BUG/MINOR: cache: Manage multiple headers in accept-encoding normalization
The accept-encoding part of the secondary key (vary) was only built out
of the first occurrence of the header. So if a client had two
accept-encoding headers, gzip and br for instance, the key would have
been built out of the gzip string. So another client that only managed
gzip would have been sent the cached resource, even if it was a br resource.
The http_find_header function is now called directly by the normalizers
so that they can manage multiple headers if needed.
A request that has more than 16 encodings will be considered as an
illegitimate request and its response will not be stored.

This fixes GitHub issue #987.

It does not need any backport.
2020-12-24 17:18:00 +01:00
Remi Tricot-Le Breton
2b5c5cbef6 MINOR: cache: Avoid storing responses whose secondary key was not correctly calculated
If any of the secondary hash normalizing functions raises an error, the
secondary hash will be unusable. In this case, the response will not be
stored anymore.
2020-12-24 17:18:00 +01:00
Remi Tricot-Le Breton
bba2912758 MINOR: cache: Refactoring of secondary_key building functions
The two secondary_key building functions (prebuild_full_key
and build_key) have roughly the same content so their code can
be mutualized.
2020-12-24 17:18:00 +01:00
Frédéric Lécaille
f63921fc24 MINOR: quic: Add traces for quic_packet_encrypt().
Add traces to have an idea why this function may fail. In fact
in never fails when the passed parameters are correct, especially the
lengths. This is not the case when a packet is not correctly built
before being encrypted.
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
133e8a7146 MINOR: quic: make a packet build fails when qc_build_frm() fails.
Even if the size of frames built by qc_build_frm() are computed so that
not to overflow a buffer, do not rely on this and always makes a packet
build fails if we could not build a frame.
Also add traces to have an idea where qc_build_frm() fails.
Fixes a memory leak in qc_build_phdshk_apkt().
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
f7e0b8d6ae MINOR: quic: Add traces for in flght ack-eliciting packet counter.
Add trace for this counter. Also shorten its variable name (->ifae_pkts).
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
b4e17386cb MINOR: quic: Update the initial salt to that of draft-29.
This salt is ued at leat up to draft-32. At this date ngtcp2 always
uses this salt even if it started the draft-33 development.

Note that when the salt is not correct, we cannot remove the header
protection. In this case the packet number length is wrong.
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
47c433fdcb MINOR: quic: Display the SSL alert in ->ssl_send_alert() callback.
At least displays the SSL alert error code passed to ->ssl_send_alert()
QUIC BIO method and the SSL encryption level. This function is newly called
when using picoquic client with a recent version of BoringSSL (Nov 19 2020).
This is not the case with OpenSSL with 32 as QUIC draft implementation.
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
26c49d9eb0 MINOR: quic: Add traces to congestion avoidance NewReno callback.
These traces are missing and are useful do diagnose issue in the congestion
avoidance callback for NewReno algorithm.
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
0c14020f11 MINOR: quic: Code reordering to help in reviewing/modifying.
Reorder by increasing type the switch/case in qc_parse_pkt_frms()
which is the high level frame parser.
Add new STREAM_X frame types to support some tests with ngtcp2 client.
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
f7fe9659f0 MINOR: quic: Flag RX packet as ack-eliciting from the generic parser.
Add ->flags to the QUIC frame parser as this has been done for the builder so
that to flag RX packets as ack-eliciting at low level. This should also be
helpful to maintain the code if we have to add new flags to RX packets.
Remove the statements which does the same thing as higher level in
qc_parse_pkt_frms().
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
04ffb66bc9 MINOR: quic: Make usage of the congestion control window.
Remove ->ifcdata which was there to control the CRYPTO data sent to the
peer so that not to saturate its reception buffer. This was a sort
of flow control.
Add ->prep_in_flight counter to the QUIC path struct to control the
number of bytes prepared to be sent so that not to saturare the
congestion control window. This counter is increased each time a
packet was built. This has nothing to see with ->in_flight which
is the real in flight number of bytes which have really been sent.
We are olbiged to maintain two such counters to know how many bytes
of data we can prepared before sending them.
Modify traces consequently which were useful to diagnose issues about
the congestion control window usage.
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
c5e72b9868 MINOR: quic: Attempt to make trace more readable
As there is a lot of information in this protocol, this is not
easy to make the traces readable. We remove here a few of them and
shorten some line shortening the variable names.
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
8090b51e92 MAJOR: quic: Make usage of ebtrees to store QUIC ACK ranges.
Store QUIC ACK ranges in ebtrees in place of lists with a 0(n) time complexity
for insertion.
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
0a76901926 MINOR: cfgparse: QUIC default server transport parameters init.
This patch is there to initialize the default transport parameters for QUIC
as a preparation for one of the QUIC next steps to come: fully support QUIC
protocol for haproxy servers.
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
026a7921a5 MINOR: quic: QUIC socket management finalization.
Implement ->accept_conn() callback for QUIC listener sockets.
Note that this patch also implements quic_session_accept() function
which is similar to session_accept_fd() without calling conn_complete_session()
at this time because we do not have any real QUIC mux.
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
e9473c7833 MINOR: ssl: QUIC transport parameters parsing.
This patch modifies the TLS ClientHello message callback so that to parse the QUIC
client transport parameters.
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
ec216523f7 MINOR: ssl: SSL CTX initialization modifications for QUIC.
Makes TLS/TCP and QUIC share the same CTX initializer so that not to modify the
caller which is an XPRT callback used both by the QUIC xprt and the SSL xprt over
TCP.
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
f46c10cfb1 MINOR: server: Add QUIC definitions to servers.
This patch adds QUIC structs to server struct so that to make the QUIC code
compile. Also initializes the ebtree to store the connections by connection
IDs.
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
884f2e9f43 MINOR: listener: Add QUIC info to listeners and receivers.
This patch adds a quic_transport_params struct to bind_conf struct
used for the listeners. This is to store the QUIC transport parameters
for the listeners. Also initializes them when calling str2listener().
Before str2sa_range() it's too early to figure we're going to speak QUIC,
and after it's too late as listeners are already created. So it seems that
doing it in str2listener() when the protocol is discovered is the best
place.

Also adds two ebtrees to the underlying receivers to store the connection
by connections IDs (one for the original connection IDs, and another
one for the definitive connection IDs which really identify the connections.

However it doesn't seem normal that it is stored in the receiver nor the
listener. There should be a private context in the listener so that
protocols can store internal information. This element should in
fact be the listener handle.

Something still feels wrong, and probably we'll have to make QUIC and
SSL co-exist: a proof of this is that there's some explicit code in
bind_parse_ssl() to prevent the "ssl" keyword from replacing the xprt.
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
a7e7ce957d MINOR: quic: Import C source code files for QUIC protocol.
This patch imports all the C files for QUIC protocol implementation with few
modifications from 20200720-quic branch of quic-dev repository found at
https://github.com/haproxytech/quic-dev.

Traces were implemented to help with the development.
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
10caf65634 MINOR: tools: Add support for QUIC addresses parsing.
Add "quic4" and "quic6" keywords to str2sa_range() to parse QUIC IPv4
and IPv6 addresses respectively.
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
e50afbd4e4 MINOR: cfgparse: Do not modify the QUIC xprt when parsing "ssl".
When parsing "ssl" keyword for TLS bindings, we must not use the same xprt as the one
for TLS/TCP connections. So, do not modify the QUIC xprt which will be initialized
when parsing QUIC addresses wich "ssl" bindings.
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
901ee2f37b MINOR: ssl: Export definitions required by QUIC.
QUIC needs to initialize its BIO and SSL session the same way as for SSL over TCP
connections. It needs also to use the same ClientHello callback.
This patch only exports functions and variables shared between QUIC and SSL/TCP
connections.
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
5aa92411fb MINOR: ssl_sock: Initialize BIO and SSL objects outside of ssl_sock_init()
This patch extraces the code which initializes the BIO and SSL session
objects so that to reuse it elsewhere later for QUIC conections which
only needs SSL and BIO objects at th TLS layer stack level to work.
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
70da889d57 MINOR: quic: Redefine control layer callbacks which are QUIC specific.
We add src/quic_sock.c QUIC specific socket management functions as callbacks
for the control layer: ->accept_conn, ->default_iocb and ->rx_listening.
accept_conn() will have to be defined. The default I/O handler only recvfrom()
the datagrams received. Furthermore, ->rx_listening callback always returns 1 at
this time but should returns 0 when reloading the processus.
2020-12-23 11:57:26 +01:00
Frédéric Lécaille
ca42b2c9d3 MINOR: protocol: Create proto_quic QUIC protocol layer.
As QUIC is a connection oriented protocol, this file is almost a copy of
proto_tcp without TCP specific features. To suspend/resume a QUIC receiver
we proceed the same way as for proto_udp receivers.

With the recent updates to the listeners, we don't need a specific set of
quic*_add_listener() functions, the default ones are sufficient. The fields
declaration were reordered to make the various layers more visible like in
other protocols.

udp_suspend_receiver/udp_resume_receiver are up-to-date (the check for INHERITED
is present) and the code being UDP-specific, it's normal to use UDP here.
Note that in the future we might more reasily reference stacked layers so that
there's no more need for specifying the pointer here.
2020-12-23 11:57:26 +01:00
Dragan Dosen
04bf0cc086 MINOR: sample: add the xxh3 converter
This patch adds support for the XXH3 variant of hash function that
generates a 64-bit hash.
2020-12-23 06:39:21 +01:00
Dragan Dosen
6bfe425679 CLEANUP: xxhash: remove the unused src/xxhash.c
The source file src/xxhash.c is removed, as we use XXH_INLINE_ALL.
2020-12-23 06:39:21 +01:00
Dragan Dosen
967e7e79af MEDIUM: xxhash: use the XXH3 functions to generate 64-bit hashes
Replace the XXH64() function calls with the XXH3 variant function
XXH3_64bits_withSeed() where possible.
2020-12-23 06:39:21 +01:00
Dragan Dosen
de37443e64 IMPORT: xxhash: update to v0.8.0 that introduces stable XXH3 variant
A new XXH3 variant of hash functions shows a noticeable improvement in
performance (especially on small data), and also brings 128-bit support,
better inlining and streaming capabilities.

Performance comparison is available here:

  https://github.com/Cyan4973/xxHash/wiki/Performance-comparison
2020-12-23 06:39:21 +01:00
Amaury Denoyelle
6d975f0af6 MINOR: check: do not ignore a connection header for http-check send
Allow the user to specify a custom Connection header for http-check
send. This is useful for example to implement a websocket upgrade check.

If no connection header has been set, a 'Connection: close' header is
automatically appended to allow the server to close the connection
immediately after the request/response.

Update the documentation related to http-check send.

This fixes the github issue #1009.
2020-12-22 14:22:44 +01:00
Tim Duesterhus
12a08d8849 BUG/MEDIUM: mux_h2: Add missing braces in h2_snd_buf()around trace+wakeup
This is a regression in 7838a79ba ("MEDIUM: mux-h2/trace: add lots of traces
all over the code"). The issue was found using -Wmisleading-indentation.
This patch fixes GitHub issue #1015.

The impact of this bug is that it could in theory cause occasional delays
on some long responses for connections having otherwise no traffic.

This patch should be backported to 2.1+, the commit was first tagged in
v2.1-dev2.
2020-12-22 09:02:11 +01:00
Ilya Shipitsin
f38a01884a CLEANUP: assorted typo fixes in the code and comments
This is 13n iteration of typo fixes
2020-12-21 11:24:48 +01:00
Baptiste Assmann
949a7f6459 BUG/MINOR: dns: SRV records ignores duplicated AR records
This bug happens when a service has multiple records on the same host
and the server provides the A/AAAA resolution in the response as AR
(Additional Records).

In such condition, the first occurence of the host will be taken from
the Additional section, while the second (and next ones) will be process
by an independent resolution task (like we used to do before 2.2).
This can lead to a situation where the "synchronisation" of the
resolution may diverge, like described in github issue #971.

Because of this behavior, HAProxy mixes various type of requests to
resolve the full list of servers: SRV+AR for all "first" occurences and
A/AAAA for all other occurences of an existing hostname.
IE: with the following type of response:

   ;; ANSWER SECTION:
   _http._tcp.be2.tld.     3600    IN      SRV     5 500 80 A2.tld.
   _http._tcp.be2.tld.     3600    IN      SRV     5 500 86 A3.tld.
   _http._tcp.be2.tld.     3600    IN      SRV     5 500 80 A1.tld.
   _http._tcp.be2.tld.     3600    IN      SRV     5 500 85 A3.tld.

   ;; ADDITIONAL SECTION:
   A2.tld.                 3600    IN      A       192.168.0.2
   A3.tld.                 3600    IN      A       192.168.0.3
   A1.tld.                 3600    IN      A       192.168.0.1
   A3.tld.                 3600    IN      A       192.168.0.3

the first A3 host is resolved using the Additional Section and the
second one through a dedicated A request.

When linking the SRV records to their respective Additional one, a
condition was missing (chek if said SRV record is already attached to an
Additional one), leading to stop processing SRV only when the target
SRV field matches the Additional record name. Hence only the first
occurence of a target was managed by an additional record.
This patch adds a condition in this loop to ensure the record being
parsed is not already linked to an Additional Record. If so, we can
carry on the parsing to find a possible next one with the same target
field value.

backport status: 2.2 and above
2020-12-21 11:19:09 +01:00
Ilya Shipitsin
af204881a3 BUILD: ssl: fine guard for SSL_CTX_get0_privatekey call
SSL_CTX_get0_privatekey is openssl/boringssl specific function present
since openssl-1.0.2, let us define readable guard for it, not depending
on HA_OPENSSL_VERSION
2020-12-21 11:17:36 +01:00
Willy Tarreau
c7ead07b9c CLEANUP: debug: mark the RNG's seed as unsigned
Since commit 8a069eb9a ("MINOR: debug: add a trivial PRNG for scheduler
stress-tests"), 32-bit gcc 4.7 emits this warning when parsing the
initial seed for the debugger's RNG (2463534242):

  src/debug.c:46:1: warning: this decimal constant is unsigned only in ISO C90 [enabled by default]

Let's mark it explicitly unsigned.
2020-12-18 16:31:08 +01:00
Christopher Faulet
0c366a8761 BUG/MEDIUM: mux-h1: Handle h1_process() failures on a pipelined request
On frontend side, when a conn-stream is detached from a H1 connection, the
H1 stream is destroyed and if we already have some data to parse (a
pipelined request), we process these data immedialtely calling
h1_process(). Then we adjust the H1 connection timeout. But h1_process() may
fail and release the H1 connection. For instance, a parsing error may be
reported. Thus, when that happens, we must not use anymore the H1 connection
and exit.

This patch must be backported as far as the 2.2. This bug can impact the 2.3
and the 2.2, in theory, if h1 stream creation fails. But, concretly, it only
fails on the 2.4 because the requests are now parsed at this step.
2020-12-18 15:13:58 +01:00
Christopher Faulet
fac0f8f029 CLEANUP: mux-h2: Rename h2c_frt_handle_data() to be generic
h2c_frt_handle_data() is now used to parse DATA frames on the frontend and
the backend side. Thus it is renamed into h2c_handle_data().
2020-12-18 15:05:57 +01:00
Christopher Faulet
142854b1da CLEANUP: mux-h2: Rename h2s_frt_make_resp_data() to be generic
h2s_frt_make_resp_data() is now used to emit DATA frames on the frontend and
the backend side. Thus it is renamed into h2s_make_data().
2020-12-18 15:05:57 +01:00
Christopher Faulet
198ef8b1de BUG/MEDIUM: http-ana: Never for sending data in TUNNEL mode
When a channel is set in TUNNEL mode, we now always set the CF_NEVER_WAIT flag,
to be sure to never wait for sending data. It is important because in TUNNEL
mode, we have no idea if more data are expected or not. Setting this flag
prevent the MSG_MORE flag to be set on the connection.

It is only a problem with the HTX, since the 2.2. On previous versions, the
MSG_MORE flag is only set on the mux initiative. In fact, the problem arises
because there is an ambiguity in tunnel mode about the HTX_FL_EOI flag. In this
mode, from the mux point of view, while the SHUTR is not received more data are
expected. But from the channel point of view, we want to send data asap.

At short term, this fix is good enough and is valid anyway. But for the long
term more reliable solution must be found. At least, the to_forward field must
regain its original meaning.

This patch must be backported as far as 2.2.
2020-12-18 15:05:57 +01:00
Christopher Faulet
3e1748bbf3 BUG/MINOR: mux-h1: Don't set CS_FL_EOI too early for protocol upgrade requests
When a protocol upgrade request is received, once parsed, it is waiting for
the response in the DONE state. But we must not set the flag CS_FL_EOI
because we don't know if a protocol upgrade will be performed or not.

Now, it is set on the response path, if both sides reached the DONE
state. If a protocol upgrade is finally performed, both side are switched in
TUNNEL state. Thus the CS_FL_EOI flag is not set.

If backported, this patch must be adapted because for now it relies on last
2.4-dev changes. It may be backported as far as 2.0.
2020-12-18 15:05:57 +01:00
Christopher Faulet
c75668ebff BUG/MINOR: http: Establish a tunnel for all 2xx responses to a CONNECT
As stated in the rfc7231, section 4.3.6, an HTTP tunnel via a CONNECT method
is successfully established if the server replies with any 2xx status
code. However, only 200 responses are considered as valid. With this patch,
any 2xx responses are now considered to estalish the tunnel.

This patch may be backported on demand to all stable versions and adapted
for the legacy HTTP. It works this way since a very long time and nobody
complains.
2020-12-18 15:05:57 +01:00
Miroslav Zagorac
7f8314c8d1 MINOR: opentracing: add ARGC_OT enum
Due to the addition of the OpenTracing filter it is necessary to define
ARGC_OT enum.  This value is used in the functions fmt_directive() and
smp_resolve_args().
2020-12-16 15:49:53 +01:00
Miroslav Zagorac
6deab79d59 MINOR: vars: replace static functions with global ones
The OpenTracing filter uses several internal HAProxy functions to work
with variables and therefore requires two static local HAProxy functions,
var_accounting_diff() and var_clear(), to be declared global.

In fact, the var_clear() function was not originally defined as static,
but it lacked a declaration.
2020-12-16 14:20:08 +01:00
Remi Tricot-Le Breton
5853c0c0d5 MINOR: cache: Add a max-secondary-entries cache option
This new option allows to tune the maximum number of simultaneous
entries with the same primary key in the cache (secondary entries).
When we try to store a response in the cache and there are already
max-secondary-entries living entries in the cache, the storage will
fail (but the response will still be sent to the client).
It defaults to 10 and does not have a maximum number.
2020-12-15 16:35:09 +01:00
Remi Tricot-Le Breton
73be796462 MEDIUM: cache: Avoid going over duplicates lists too often
The secondary entry counter cannot be updated without going over all the
items of a duplicates list periodically. In order to avoid doing it too
often and to impact the cache's performances, a timestamp is added to
the cache_entry. It will store the timestamp (with second precision) of
the last iteration over the list (actually the last call of the
clear_expired_duplicates function). This way, this function will not be
called more than once per second for a given duplicates list.
2020-12-15 16:35:09 +01:00
Remi Tricot-Le Breton
65904e4f07 MEDIUM: cache: Add a secondary entry counter and insertion limitation
Add an arbitrary maximum number of secondary entries per primary hash
(10 for now) to the cache. This prevents the cache from being filled
with duplicates of the same resource.
This works thanks to an entry counter that is kept in one of the
duplicates of the list (the last one).
When an entry is added to the list, the ebtree's implementation ensures
that it will be added to the end of the existing list so the only thing
to do to keep the counter updated is to get the previous counter from
the second to last entry.
Likewise, when an entry is explicitely deleted, we update the counter
from the list's last item.
2020-12-15 16:35:09 +01:00
Ilya Shipitsin
ec60909871 BUILD: SSL: fine guard for SSL_CTX_add_server_custom_ext call
SSL_CTX_add_server_custom_ext is openssl specific function present
since openssl-1.0.2, let us define readable guard for it, not depending
on HA_OPENSSL_VERSION
2020-12-15 16:13:35 +01:00
Remi Tricot-Le Breton
964caaff0e BUG/MAJOR: cache: Crash because of disabled entry not removed from the tree
The cache entries are now added into the tree even when they are not
complete yet. If we realized while trying to add a response's payload
that the shctx was full, the entry was disabled through the
disable_cache_entry function, which cleared the key field of the entry's
node, but without actually removing it from the tree. So the shctx row
could be stolen from the entry and the row's content be rewritten while
a lookup in the tree would still find a reference to the old entry. This
caused a random crash in case of cache saturation and row reuse.
This patch adds the missing removal of the node from the tree next to
the reset of the key in disable_cache_entry.

This bug was introduced by commit 3243447 ("MINOR: cache: Add entry
to the tree as soon as possible")

It does not need to be backported.
2020-12-15 15:31:30 +01:00
William Lallemand
a55685bfea BUG/MEDIUM: ssl/crt-list: bad behavior with "commit ssl cert"
In issue #1004, it was reported that it is not possible to remove
correctly a certificate after updating it when it came from a crt-list.

Indeed the "commit ssl cert" command on the CLI does not update the list
of ckch_inst in the crtlist_entry. Because of this, the "del ssl
crt-list" command does not remove neither the instances nor the SNIs
because they were never linked to the crtlist_entry.

This patch fixes the issue by inserting the ckch_inst in the
crtlist_entry once generated.

Must be backported as far as 2.2.
2020-12-15 15:13:21 +01:00
Christopher Faulet
cc043f66b7 BUG/MEDIUM: mux-h1: Fix a deadlock when a 408 error is pending for a client
When a frontend H1 connection timed out waiting for the next request, a 408
error message is returned to the client. It is performed into the H1C task
process function, h1_timeout_task(), and under the idle connection takeover
lock. If the 408 error message cannot be sent immediately, we wait for a
next retry. In this case, the lock must be released.

This bug was introduced by the commit c4bfa59f1d ("MAJOR: mux-h1: Create the
client stream as later as possible") and is specific to the 2.4-DEV. No
backport needed.
2020-12-14 10:06:13 +01:00
Christopher Faulet
cb33d3ac7f BUG/MEDIUM: lb-leastconn: Reposition a server using the right eweight
Depending on the context, the current eweight or the next one must be used
to reposition a server in the tree. When the server state is updated, for
instance its weight, the next eweight must be used because it is not yet
committed. However, when the server is used, on normal conditions, the
current eweight must be used.

In fact, it is only a bug on the 1.8. On newer versions, the changes on a
server are performed synchronously. But it is safer to rely on the right
eweight value to avoid any futur bugs.

On the 1.8, it is important to do so, because the server state is updated
and committed inside the rendez-vous point. Thus, the next server state may
be unsync with the current state for a short time, waiting all threads join
the rendez-vous point. It is especially a problem if the next eweight is set
to 0. Because otherwise, it must not be used to reposition the server in the
tree, leading to a divide by 0.

This patch must be backported as far as 1.8.
2020-12-14 09:52:34 +01:00
Willy Tarreau
746b0515a4 MEDIUM: connection: make use of the control layer check_events/ignore_events
This changes the subscribe/unsubscribe functions to rely on the control
layer's check_events/ignore_events. At the moment only the socket version
of these functions is present so the code should basically be the same.
2020-12-11 17:06:11 +01:00
Willy Tarreau
472125bc04 MINOR: protocol: add a pair of check_events/ignore_events functions at the ctrl layer
Right now the connection subscribe/unsubscribe code needs to manipulate
FDs, which is not compatible with QUIC. In practice what we need there
is to be able to either subscribe or wake up depending on readiness at
the moment of subscription.

This commit introduces two new functions at the control layer, which are
provided by the socket code, to check for FD readiness or subscribe to it
at the control layer. For now it's not used.
2020-12-11 17:02:50 +01:00
Willy Tarreau
2ded48dd27 MINOR: connection: make conn_sock_drain() use the control layer's ->drain()
Now we don't touch the fd anymore there, instead we rely on the ->drain()
provided by the control layer. As such the function was renamed to
conn_ctrl_drain().
2020-12-11 16:26:01 +01:00
Willy Tarreau
427c846cc9 MINOR: protocol: add a ->drain() function at the connection control layer
This is what we need to drain pending incoming data from an connection.
The code was taken from conn_sock_drain() without the connection-specific
stuff. It still takes a connection for now for API simplicity.
2020-12-11 16:26:00 +01:00
Willy Tarreau
586f71b43f REORG: connection: move the socket iocb (conn_fd_handler) to sock.c
conn_fd_handler() is 100% specific to socket code. It's about time
it moves to sock.c which manipulates socket FDs. With it comes
conn_fd_check() which tests for the socket's readiness. The ugly
connection status check at the end of the iocb was moved to an inlined
function in connection.h so that if we need it for other socket layers
it's not too hard to reuse.

The code was really only moved and not changed at all.
2020-12-11 16:26:00 +01:00
Willy Tarreau
827fee7406 MINOR: connection: remove sock-specific code from conn_sock_send()
The send() loop present in this function and the error handling is already
present in raw_sock_from_buf(). Let's rely on it instead and stop touching
the FD from this place. The send flag was changed to use a more agnostic
CO_SFL_*. The name was changed to "conn_ctrl_send()" to remind that it's
meant to be used to send at the lowest level.
2020-12-11 16:25:11 +01:00
Amaury Denoyelle
f7719a25db MINOR: stream: add timeout sample fetches
Add cur_server_timeout and cur_tunnel_timeout.

These sample fetches return the current timeout value for a stream. This
is useful to retrieve the value of a timeout which was changed via a
set-timeout rule.
2020-12-11 12:01:07 +01:00
Amaury Denoyelle
12bada5662 MINOR: stream: add sample fetches
Prepare the possibility to register sample fetches on the stream.

This commit is necessary to implement sample fetches to retrieve the
current timeout values.
2020-12-11 12:01:07 +01:00
Amaury Denoyelle
d91d779618 MINOR: backend: add timeout sample fetches
Add be_server_timeout and be_tunnel_timeout.

These sample fetches return the configuration value for server or tunnel
timeout on the backend side.
2020-12-11 12:01:07 +01:00
Amaury Denoyelle
da184d5306 MINOR: frontend: add client timeout sample fetch
Add a sample fetch named fe_client_timeout to return the configuration
value for the client timeout on a frontend.
2020-12-11 12:01:07 +01:00
Amaury Denoyelle
8d22823ade MEDIUM: http_act: define set-timeout server/tunnel action
Add a new http-request action 'set-timeout [server/tunnel]'. This action
can be used to update the server or tunnel timeout of a stream. It takes
two parameters, the timeout name to update and the new timeout value.
This rule is only valid for a proxy with backend capabilities. The
timeout value cannot be null. A sample expression can also be used
instead of a plain value.
2020-12-11 12:01:07 +01:00
Amaury Denoyelle
fb50443517 MEDIUM: stream: support a dynamic tunnel timeout
Allow the modification of the tunnel timeout on the stream side.
Use a new field in the stream for the tunnel timeout. It is initialized
by the tunnel timeout from backend unless it has already been set by a
set-timeout tunnel rule.
2020-12-11 12:01:07 +01:00
Amaury Denoyelle
90d3d882e3 MEDIUM: stream: support a dynamic server timeout
Allow the modification of the timeout server value on the stream side.
Do not apply the default backend server timeout in back_establish if it
is already defined. This is the case if a set-timeout server rule has
been executed.
2020-12-11 12:01:07 +01:00
Amaury Denoyelle
b715078821 MINOR: stream: prepare the hot refresh of timeouts
Define a stream function to allow to update the timeouts.
This commit is in preparation for the support of dynamic timeouts with
the set-timeout rule.
2020-12-11 12:01:07 +01:00
Christopher Faulet
82635a0fc1 BUG/MINOR: tools: Reject size format not starting by a digit
parse_size_err() function is now more strict on the size format. The first
character must be a digit. Otherwise an error is returned. Thus "size k" is
now rejected.

This patch must be backported to all stable versions.
2020-12-11 12:01:07 +01:00
Christopher Faulet
c20ad0d8db BUG/MINOR: tools: make parse_time_err() more strict on the timer validity
First, an error is now reported if the first character is not a digit. Thus,
"timeout client s" triggers an error now. Then 'u' is also rejected
now. 'us' is valid and should be used set the timer in microseconds. However
'u' alone is not a valid unit. It was just ignored before (default to
milliseconds). Now, it is an error. Finally, a warning is reported if the
end of the text is not reached after the timer parsing. This warning will
probably be switched to an error in a futur version.

This patch must be backported to all stable versions.
2020-12-11 12:01:04 +01:00
Christopher Faulet
cad5f5e1ed MINOR: tcpcheck: Only wait for more payload data on HTTP expect rules
For HTTP expect rules, if the buffer is not empty, it is guarantee that all
responses headers are received, with the start-line. Thus, except for
payload matching, there is no reason to wait for more data from the moment
the htx message is not empty.

This patch may be backported as far as 2.2.
2020-12-11 11:48:15 +01:00
Christopher Faulet
c878f56f7c BUG/MINOR: tcpcheck: Don't rearm the check timeout on each read
The check timeout is used to limit a health-check execution. By default
inter timeout is used. But when defined the check timeout is used. In this
case, the inter timeout (or connect timeout) is used for the connection
establishment only. And the check timeout for the health-check
execution. Thus, it must be set after a successfull connect. It means it is
rearm at the end of each connect rule.

This patch with the previous one (BUG/MINOR: http-check: Use right condition
to consider HTX message as full) should solve the issue #991. It must be
backported as far as 2.2. On the 2.3 and 2.2, there are 2 places were the
connection establishement is handled. The check timeout must be set on both.
2020-12-11 11:48:15 +01:00
Christopher Faulet
3f527197cd BUG/MINOR: http-check: Use right condition to consider HTX message as full
When an HTTP expect rule is evaluated, we must know if more data is expected
or not to wait if the matching fails. If the whole response is received or
if the HTX message is full, we must not wait. In this context,
htx_free_data_space() must be used instead of htx_free_space(). The fisrt
one count down the block size. Otherwise at the edge, when only the block
size remains free (8 bytes), we may think there is some place for more data
while the mux is unable to add more block.

This bug explains the loop described on the GH issue #991. It should be
backported as far as 2.2.
2020-12-11 11:48:15 +01:00
Willy Tarreau
8b250ba738 CLEANUP: connection: open-code conn_cond_update_polling() and update the comment
This last call to conn_cond_update_polling() is now totally misleading as
the function only stops polling in case of unrecoverable connection error.
Let's open-code the test to make it more prominent and explain what we're
trying to do there. It's even almost certain this code is never executed
anymore, as the only remaining case should be a mux's wake function setting
CO_FL_ERROR without disabling the polling, but they need to be audited first
to make sure this is the case.
2020-12-11 11:19:24 +01:00
Willy Tarreau
f7e4a6fc07 MINOR: checks: don't call conn_cond_update_polling() anymore
This was a leftover of the pre-mux v1.8-dev3 era. It makes no sense anymore
to try to disable polling on a connection we don't own, it's the mux's job
and it's properly done upon shutdowns and closes.
2020-12-11 11:11:06 +01:00
Willy Tarreau
30bd4efb1b MINOR: checks: use cs_drain_and_close() instead of draining the connection
As explained in previous commit, the situation is absurd as we try to
cleanly drain pending data before impolitely shutting down, and it could
be counter productive on real muxes. Let's use cs_drain_and_close() instead.
2020-12-11 11:09:29 +01:00
Willy Tarreau
7d7b11cf93 MINOR: mux-pt: take care of CS_SHR_DRAIN in shutr()
When the shutr() requests CS_SHR_DRAIN and there's no particular shutr
implemented on the underlying transport layer, we must drain pending data.
This is what happens when cs_drain_and_close() is called. It is important
for TCP checks to drain large responses and close cleanly.
2020-12-11 11:07:19 +01:00
Willy Tarreau
a5ea751922 MINOR: stream-int: don't touch polling anymore on shutdown
Not only it's become totally useless with muxes, in addition it's
dangerous to play with the mux's FD while shutting a stream down for
writes. It's already done *if necessary* by the cs_shutw() code at the
mux layer. Fortunately it doesn't seem to have any impact, most likely
the polling updates used to immediately revert this operation.
2020-12-11 10:29:11 +01:00
Willy Tarreau
5a1d439225 CLEANUP: connection: use fd_stop_both() instead of conn_stop_polling()
conn_stop_polling() in fact only calls fd_stop_both() after checking
that the ctrl layer is ready. It's the case in conn_fd_check() so
let's get rid of this next-to-last user of this function.
2020-12-11 09:56:53 +01:00
Remi Tricot-Le Breton
e3e1e5f34b MINOR: cache: Dump secondary entries in "show cache"
The duplicated entries (in case of vary) were not taken into account by
the "show cache" command. They are now dumped too.
A new "vary" column is added to the output. It contains the complete
seocndary key (in hex format).
2020-12-10 15:59:49 +01:00
Willy Tarreau
29885f0308 MINOR: udp: export udp_suspend_receiver() and udp_resume_receiver()
QUIC will rely on UDP at the receiver level, and will need these functions
to suspend/resume the receivers. In the future, protocol chaining may
simplify this.
2020-12-08 18:10:18 +01:00
Willy Tarreau
de471c4655 MINOR: protocol: add a set of ctrl_init/ctrl_close methods for setup/teardown
Currnetly conn_ctrl_init() does an fd_insert() and conn_ctrl_close() does an
fd_delete(). These are the two only short-term obstacles against using a
non-fd handle to set up a connection. Let's have pur these into the protocol
layer, along with the other connection-level stuff so that the generic
connection code uses them instead. This will allow to define new ones for
other protocols (e.g. QUIC).

Since we only support regular sockets at the moment, the code was placed
into sock.c and shared with proto_tcp, proto_uxst and proto_sockpair.
2020-12-08 15:50:56 +01:00
Willy Tarreau
b366c9a59a CLEANUP: protocol: group protocol struct members by usage
For the sake of an improved readability, let's group the protocol
field members according to where they're supposed to be defined:
  - connection layer (note: for now even UDP needs one)
  - binding layer
  - address family
  - socket layer
Nothing else was changed.
2020-12-08 14:58:24 +01:00
Willy Tarreau
b9b2fd7cf4 MINOR: protocol: export protocol definitions
The various protocols were made static since there was no point in
exporting them in the past. Nowadays with QUIC relying on UDP we'll
significantly benefit from UDP being exported and more generally from
being able to declare some functions as being the same as other
protocols'.

In an ideal world it should not be these protocols which should be
exported, but the intermediary levels:
  - socket layer (sock.c only right now), already exported as functions
    but nothing structured at the moment ;
  - family layer (sock_inet, sock_unix, sockpair etc): already structured
    and exported
  - binding layer (the part that relies on the receiver): currently fused
    within the protocol
  - connectiong layer (the part that manipulates connections): currently
    fused within the protocol
  - protocol (connection's control): shouldn't need to be exposed
    ultimately once the elements above are in an easily sharable way.
2020-12-08 14:54:08 +01:00
Willy Tarreau
f9ad06cb26 MINOR: protocol: remove the redundant ->sock_domain field
This field used to be needed before commit 2b5e0d8b6 ("MEDIUM: proto_udp:
replace last AF_CUST_UDP* with AF_INET*") as it was used as a protocol
entry selector. Since this commit it's always equal to the socket family's
value so it's entirely redundant. Let's remove it now to simplify the
protocol definition a little bit.
2020-12-08 12:13:54 +01:00
Christopher Faulet
c43fca0139 BUG/MINOR: stream: Don't use input buffer after the ownership xfer
At the end of stream_new(), once the input buffer is transfer to the request
channel, it must not be used anymore. The previous patch (16df178b6 "BUG/MEDIUM:
stream: Xfer the input buffer to a fully created stream") was pushed to quickly.

No backport needed.
2020-12-04 17:22:50 +01:00
Christopher Faulet
16df178b6e BUG/MEDIUM: stream: Xfer the input buffer to a fully created stream
The input buffer passed as argument to create a new stream must not be
transferred when the request channel is initialized because the channel
flags are not set at this stage. In addition, the API is a bit confusing
regarding the buffer owner when an error occurred. The caller remains the
owner, but reading the code it is not obvious.

So, first of all, to avoid any ambiguities, comments are added on the
calling chain to make it clear. The buffer owner is the caller if any error
occurred. And the ownership is transferred to the stream on success.

Then, to make things simple, the ownership is transferred at the end of
stream_new(), in case of success. And the input buffer is updated to point
on BUF_NULL. Thus, in all cases, if the caller try to release it calling
b_free() on it, it is not a problem. Of course, it remains the caller
responsibility to release it on error.

The patch fixes a bug introduced by the commit 26256f86e ("MINOR: stream:
Pass an optional input buffer when a stream is created"). No backport is
needed.
2020-12-04 17:15:03 +01:00
William Lallemand
b7fdfdfd92 MEDIUM: ssl: fatal error with bundle + openssl < 1.1.1
Since HAProxy 2.3, OpenSSL 1.1.1 is a requirement for using a
multi-certificate bundle in the configuration. This patch emits a fatal
error when HAProxy tries to load a bundle with an older version of
HAProxy.

This problem was encountered by an user in issue #990.

This must be backported in 2.3.
2020-12-04 15:45:02 +01:00
Willy Tarreau
d1f250f87b MINOR: listener: now use a generic add_listener() function
With the removal of the family-specific port setting, all protocol had
exactly the same implementation of ->add(). A generic one was created
with the name "default_add_listener" so that all other ones can now be
removed. The API was slightly adjusted so that the protocol and the
listener are passed instead of the listener and the port.

Note that all protocols continue to provide this ->add() method instead
of routinely calling default_add_listener() from create_listeners(). This
makes sure that any non-standard protocol will still be able to intercept
the listener addition if needed.

This could be backported to 2.3 along with the few previous patches on
listners as a pure code cleanup.
2020-12-04 15:08:00 +01:00
Willy Tarreau
07400c56bb MINOR: listener: automatically set the port when creating listeners
In create_listeners() we iterate over a port range and call the
protocol's ->add() function to add a new listener on the specified
port. Only tcp4/tcp6/udp4/udp6 support a port, the other ones ignore
it. Now that we can rely on the address family to properly set the
port, better do it this way directly from create_listeners() and
remove the family-specific case from the protocol layer.
2020-12-04 15:08:00 +01:00
Willy Tarreau
73bed9ff13 MINOR: protocol: add a ->set_port() helper to address families
At various places we need to set a port on an IPv4 or IPv6 address, and
it requires casts that are easy to get wrong. Let's add a new set_port()
helper to the address family to assist in this. It will be directly
accessible from the protocol and will make the operation seamless.
Right now this is only implemented for sock_inet as other families do
not need a port.
2020-12-04 15:08:00 +01:00
Christopher Faulet
c31bc724d4 MINOR: h1-htx/http-ana: Set BODYLESS flag on message in TUNNEL state
When a H1 message is parsed, if the parser state is switched to TUNNEL mode
just after the header parsing, the BODYLESS flag is set on the HTX
start-line. By transitivity, the corresponding flag is set on the message in
HTTP analysers.  Thus it is possible to rely on it to not wait for the
request body.
2020-12-04 14:41:49 +01:00
Christopher Faulet
2a40854244 MINOR: http-ana: Properly set message flags from the start-line flags
CNT_LEN and TE_CHNK flags must be set on the message only when the
corresponding flag is set on the HTX start-line. Before, when the transfer
length was known XFER_LEN set), the HTTP_MSGF_TE_CHNK was the default. But
it is not appropriate. Now, it is only set if the message is chunked. Thus,
it is now possible to have a known transfer length without CNT_LEN or
TE_CHNK.

In addition, the BODYLESS flags may be set, independently on XFER_LEN one.
2020-12-04 14:41:49 +01:00
Christopher Faulet
6ad06066cd CLEANUP: connection: Remove CS_FL_READ_PARTIAL flag
Since the recent refactoring of the H1 multiplexer, this flag is no more
used. Thus it is removed.
2020-12-04 14:41:49 +01:00
Christopher Faulet
da831fa068 CLEANUP: http-ana: Remove TX_WAIT_NEXT_RQ unsued flag
This flags is now unused. It was used in REQ_WAIT_HTTP analyser, when a
stream was waiting for a request, to set the keep-alive timeout or to avoid
to send HTTP errors to client.
2020-12-04 14:41:49 +01:00
Christopher Faulet
8bebd2fe52 MEDIUM: http-ana: Don't process partial or empty request anymore
It is now impossible to start the HTTP request processing in the stream
analysers with a partial or empty request message. The mux-h2 was already
waiting of the request headers before creating the stream. Now the mux-h1
does the same. All errors (aborts, timeout or invalid requests) waiting for
the request headers are now handled by the multiplexers. So there is no
reason to still handle them in the REQ_WAIT_HTTP (http_wait_for_request)
analyser.

To ensure there is no ambiguity, a BUG_ON() was added to exit if a partial
request is received in this analyser.
2020-12-04 14:41:49 +01:00
Christopher Faulet
2afd874704 CLEANUP: htx: Remove HTX_FL_UPGRADE unsued flag
Now the H1 to H2 upgrade is handled before the stream
creation. HTX_FL_UPGRADE flag is now unused.
2020-12-04 14:41:49 +01:00
Christopher Faulet
4a8779f808 MINOR: http-ana: Remove useless update of t_idle duration of the stream
Becaues the stream is now created after the request headers parsing, the
idle duration from the session is always up-to-date.
2020-12-04 14:41:49 +01:00
Christopher Faulet
3ced1d1db4 CLEANUP: mux-h1: Rename H1C_F_CS_* flags and reorder H1C flags
H1C_F_CS_* flags are renamed into H1C_F_ST_*. They reflect the connection
state. So "ST" is well suited. "CS" is confusing because it is also the
abbreviation for conn-stream.

In addition, H1C flags are reordered.
2020-12-04 14:41:49 +01:00
Christopher Faulet
c4bfa59f1d MAJOR: mux-h1: Create the client stream as later as possible
This is the reason for all previous patches. The conn-stream and the
associated stream are created as later as possible. It only concerns the
frontend connections. But it means the request headers, and possibly the
first data block, are received and parsed before the conn-stream
creation. To do so, an embryonic H1 stream, with no conn-stream, is
created. The result of this "early parsing" is stored in its rx buffer, used
to fill the request channel when the stream is created. During this step,
some HTTP errors may be returned by the mux. It must also handle
http-request/keep-alive timeouts. A significative change is about H1 to H2
upgrade. It happens very early now, and no H1 stream are created (and thus
of course no conn-stream).

The most important part of this patch is located to the h1_process()
function. Because it must trigger the parsing when there is no H1
stream. h1_recv() function has also been simplified.
2020-12-04 14:41:49 +01:00
Christopher Faulet
c18fc234d9 MINOR: mux-h1: Add functions to send HTTP errors from the mux
For now, this part is unsued. But this patch adds functions to handle errors
on idle and embryonic H1 connections and send corresponding HTTP error
messages to the client (400, 408 or 500). Thanks to previous patches, these
functions take care to update the right stats counters, but also the
counters tracked by the session.

A field to store the HTTP error code has been added in the H1C structure. It
is used for error retransmits, if any, and to get it in http logs. It is
used to return the mux exit status code when the MUX_EXIT_STATUS ctl
parameter is requested.
2020-12-04 14:41:49 +01:00
Christopher Faulet
ce5e6bcb04 MINOR: logs: Get the multiplexer exist status when no stream is provided
When a log message is emitted from the session level, by a multiplexer,
there is no stream. Thus for HTTP session, there no status code and the
termination flags are not correctly set.

Thanks to previous patch, the HTTP status code is deduced from the mux exist
status, using the MUX_EXIT_STATE ctl param. This is only done for HTTP
frontends. If it is defined ( != 0), it is used to deduce the termination
flags.
2020-12-04 14:41:49 +01:00
Christopher Faulet
4c8ad84232 MINOR: mux: Add a ctl parameter to get the exit status of the multiplexers
The ctl param MUX_EXIT_STATUS can be request to get the exit status of a
multiplexer. For instance, it may be an HTTP status code or an H2 error. For
now, 0 is always returned. When the mux h1 will be able to return HTTP
errors itself, this ctl param will be used to get the HTTP status code from
the logs.

the mux_exit_status enum has been created to map internal mux exist status
to generic one. Thus there is 5 possible status for now: success, invalid
error, timeout error, internal error and unknown.
2020-12-04 14:41:49 +01:00
Christopher Faulet
84600631cd MINOR: stick-tables: Add functions to update some values of a tracked counter
The cumulative numbers of http requests, http errors, bytes received and
sent and their respective rates for a tracked counters are now updated using
specific stream independent functions. These functions are used by the
stream but the aim is to allow the session to do so too. For now, there is
no reason to perform these updates from the session, except from the mux-h2
maybe. But, the mux-h1, on the frontend side, will be able to return some
errors to the client, before the stream creation. In this case, it will be
mandatory to update counters tracked at the session level.
2020-12-04 14:41:49 +01:00
Christopher Faulet
dbe57794c4 MINOR: mux-h1: Add a idle expiration date on the H1 connection
An idle expiration date is added on the H1 connection with the function to
set it depending on connection state. First, there is no idle timeout on
backend connections, For idle frontend connections, the http-request or
keep-alive timeout are used depending on which timeout is defined and if it
is the first request or not. For embryonic connections, the http-request is
always used, if defined. For attached or shutted down connections, no idle
timeout is applied.

For now the idle expiration date is never set and the h1_set_idle_expiration
function remains unused.
2020-12-04 14:41:49 +01:00
Christopher Faulet
5d3c93cd43 MINOR: mux-h1: Process next request for IDLE connection only
When the conn-stream is detached for a H1 connection, there is no reason to
subscribe for reads or process pending input data if the connection is not
idle. Because, it means a shutdown is pending.
2020-12-04 14:41:49 +01:00
Christopher Faulet
adcd789d92 MINOR: mux-h1: Rework h1_refresh_timeout to be easier to read
Conditions to set a timeout on the H1C task have been simplified or at least
changed to rely on H1 connection flags. Now, following rules are used :

 * the shutdown timeout is applied on dead (not alive) or shutted down
   connections.

 * The client/server timeout is applied if there are still some pending
   outgoing data.

 * The client timeout is applied on alive frontend connections with no
   conn-stream. It means on idle or embryionic frontend connections.

 * For all other connections (backend or attached connections), no timeout
   is applied. For frontend or backend attached connections, the timeout is
   handled by the application layer. For idle backend connections, there is
   no timeout.
2020-12-04 14:41:49 +01:00
Christopher Faulet
3c82d8b328 MINOR: mux-h1: Rework how shutdowns are handled
We now only rely on one flag to notify a shutdown. The shutdown is performed
at the connection level when there are no more pending outgoing data. So, it
means it is performed immediately if the output buffer is empty. Otherwise
it is deferred after the outgoing data are sent.

This simplify a bit the mux because there is now only one flag to check.
2020-12-04 14:41:49 +01:00
Christopher Faulet
119ac870ce MINOR: mux-h1: Disable reads if an error was reported on the H1 stream
Don't try to read more data if a parsing or a formatting error was reported
on the H1 stream. There is no reason to continue to process the messages for
the current connection in this case. If a parsing error occurs, it means the
input is invalid. If a formatting error occurs, it is an internal error and
it is probably safer to give up.
2020-12-04 14:41:49 +01:00
Christopher Faulet
295b8d1649 MINOR: mux-h1: Reset more H1C flags when a H1 stream is destroyed
When a H1 stream is destroyed, all dynamic flags on the H1 connection are
reset to be sure to leave it in a clean state.
2020-12-04 14:41:49 +01:00
Christopher Faulet
c1c66a4759 MINOR: mux-h1: rework the h1_timeout_task() function
Mainly to make it easier to read. First of all, when a H1 connection is
still there, we check if the connection was stolen by another thread or
not. If yes we release the task and leave. Then we check if the task is
expired or not. Only expired tasks are considered. Finally, if a conn-stream
is still attached to the connection (H1C_F_CS_ATTACHED flag set), we
return. Otherwise, the task and the H1 connection are released.
2020-12-04 14:41:48 +01:00
Christopher Faulet
bb8baf477d MINOR: mux-h1: Add embryonic and attached states on the H1 connection
Be prepared to have a H1 connection in one of the following states :

 * A H1 connection waiting for a new message with no H1 stream.
   H1C_F_CS_IDLE flag is set.

 * A H1 connection processing a new message with a H1 stream but no
   conn-stream attached. H1C_F_CS_EMBRYONIC flag is set

 * A H1 connection with a H1 stream and a conn-stream attached.
   H1C_F_CS_ATTACHED flag is set.

 * A H1 connection with no H1 stream, waiting to be released. No flag is set.

These flags are mutually exclusives. When none is set, it means the
connection will be released ASAP, just remaining outgoing data must be sent
before. For now, the second state (H1C_F_CS_EMBRYONIC) is transient.
2020-12-04 14:41:48 +01:00
Christopher Faulet
a583af6333 MINOR: mux-h1: Don't set CS flags in internal parsing functions
Now, only h1_process_input() function set or unset the conn-stream
flags. This way, internal parsing functions don't rely anymore on the
conn-stream.
2020-12-04 14:41:48 +01:00
Christopher Faulet
d17ad8214f MINOR: mux-h1: Add a rxbuf into the H1 stream
For now this buffer is not used. But it will be used to parse the headers,
and possibly the first block of data, when no stream is attached to the H1
connection. The aim is to use it to create the stream, thanks to recent
changes on the streams creation api.
2020-12-04 14:41:48 +01:00
Christopher Faulet
2f0ec66613 MINOR: mux-h1: Split front/back h1 stream creation in 2 functions
Dedicated functions are now used to create frontend and backend H1
streams. h1c_frt_stream_new() is now used to create frontend H1 streams and
h1c_bck_stream_new() to create backend ones. Both rely on h1s_new() function
to allocate the stream itself. It is a bit easier to add specific processing
depending we are on the frontend or the backend side.
2020-12-04 14:41:48 +01:00
Christopher Faulet
60ef12c80b MINOR: mux-h1: Separate parsing and formatting errors at H1 stream level
Instead of using H1S flags to report an error on the request or the
response, independently it is a parsing or a formatting error, we now use a
flag to report parsing errors and another one to report formatting
ones. This simplify the message parsing. It is also easier to figure out
what error happened when one of this flag is set. The side may be deduced
checking the H1C_F_IS_BACK flag.
2020-12-04 14:41:48 +01:00
Christopher Faulet
0a799aa3d6 MINOR: mux-h1: Introduce H1C_F_IS_BACK flag on the H1 connection
This flag is only set on the backend side and is tested instead of calling
conn_is_back() function.
2020-12-04 14:41:48 +01:00
Christopher Faulet
ae635766f6 MEDIUM: mux-h1: Use a h1c flag to block reads when splicing is in-progress
Instead of using 2 flags on the H1 stream (H1S_F_BUF_FLUSH and
H1S_F_SPLICED_DATA), we now only use one flag on the H1 connection
(H1C_F_WANT_SPLICE) to notify we want to use splicing or we are using
splicing. This flag blocks the calls to rcv_buf() connection callback.

It is a bit easier to set the H1 connection capability to receive data in
its input buffer instead of relying on the H1 stream.
2020-12-04 14:41:48 +01:00
Christopher Faulet
089acd5b0d MINOR: mux-h1: Add a flag to disable reads to wait opposite side
H1C_F_WAIT_OPPOSITE must be set on the H1 conenction to don't read more data
because we must be sync with the opposite side. This flag replaces the
H1C_F_IN_BUSY flag. Its name is a bit explicit. It is automatically set on
the backend side when the mux is created. It is safe to do so because at
this stage, the request has not yet been sent to the server. This way, in
h1_recv_allowed(), a test on this flag is enough to block the reads instead
of testing the H1 stream state on the backend side.
2020-12-04 14:41:48 +01:00
Christopher Faulet
26256f86e1 MINOR: stream: Pass an optional input buffer when a stream is created
It is now possible to set the buffer used by the channel request buffer when
a stream is created. It may be useful if input data are already received,
instead of waiting the first call to the mux rcv_buf() callback. This change
is mandatory to support H1 connection with no stream attached.

For now, the multiplexers don't pass any buffer. BUF_NULL is thus used to
call stream_create_from_cs().
2020-12-04 14:41:48 +01:00
Christopher Faulet
3b536a3131 MINOR: mux-h1: Don't provide anymore timing info using cs_info structure
The cs_info are now unused. The stream uses the session to get these
info. So we can safely remove it from the mux-h1.
2020-12-04 14:41:48 +01:00
Christopher Faulet
15e525f495 MINOR: stream: Don't retrieve anymore timing info from the mux csinfo
These info are only provided by the mux-h1. But, thanks to previous patches,
we can get them from the session directly. There is no need to retrieve them
from the mux anymore.
2020-12-04 14:41:48 +01:00
Christopher Faulet
7a6c513246 MINOR: stream: Always get idle duration from the session
Since the idle duration provided by the session is always up-to-date, there
is no more reason to rely on the multiplexer cs_info to set it to the
stream.
2020-12-04 14:41:48 +01:00
Christopher Faulet
dd78921c66 MINOR: logs: Use session idle duration when no stream is provided
When a log message is emitted from the session, using sess_log() function,
there is no stream available. In this case, instead of deducing the idle
duration from the accept date, we use the one provided by the session. 0 is
used if it is undefined (i.e set to -1).
2020-12-04 14:41:48 +01:00
Christopher Faulet
42849b047a MINOR: mux-h1: Reset session dates and durations info when the CS is detached
These info are reset for the next transaction, if the connection is kept
alive. From the stream point of view, it should be the same a new
connection, except there is no handshake. Thus the handshake duration is set
to 0.
2020-12-04 14:41:48 +01:00
Christopher Faulet
4e74155466 MINOR: mux-h1: Update session idle duration when data are received
The session idle duration is set if not already done when data are
received. For now, this value is still unused.
2020-12-04 14:41:48 +01:00
Christopher Faulet
d517396f8e MINOR: session: Add the idle duration field into the session
The idle duration between two streams is added to the session structure. It
is not necessarily pertinent on all protocols. In fact, it is only defined
for H1 connections. It is the duration between two H1 transactions. But the
.get_cs_info() callback function on the multiplexers only exists because
this duration is missing at the session level. So it is a simplification
opportunity for a really low cost.

To reduce the cost, a hole in the session structure is filled by moving
.srv_list field at the end of the structure.
2020-12-04 14:41:48 +01:00
Christopher Faulet
268c92e2f8 BUG/MINOR: mux-h1: Handle keep-alive timeout for idle frontend connections
IDLE frontend connections have no stream attached. The stream is only
created when new data are received, when the parsing of the next request
starts. Thus the keep-alive timeout, handled into the HTTP analysers, is not
considered while nothing is received. But this is especially when this
timeout must be considered. Concretely the http-keep-alive is ignored while
no data are received. Only the client timeout is used. It will only be
considered on incomplete requests, if the http-request timeout is not set.

To fix the bug, the http-keep-alive timeout must be handled at the mux
level, for IDLE frontend connection only.

This patch should fix the issue #984. It must be backported as far as
2.2. On prior versions, the stream is created earlier. So, it is not a
problem, except if this behavior changes of course (it was an optim of the
2.2, but don't remember the commit).
2020-12-04 14:41:48 +01:00
Willy Tarreau
7da02dd308 BUG/MINOR: listener: use sockaddr_in6 for IPv6
A copy-paste bug between {tcp,udp}{4,6}_add_listener() resulted in
using a struct sockaddr_in to set the TCP/UDP port while it ought to
be a struct sockaddr_in6. Fortunately, the port has the same offset
(2) in both so it was harmless. A cleaner way to proceed would be
to have a set_port function exported by the address family layer.

This needs to be backported to 2.3.
2020-12-04 14:28:23 +01:00
Willy Tarreau
186f37674c BUG/MINOR: lua-thread: close all states on deinit
It seems to me that lua_close() must be called on all states at deinit
time, not just the first two ones. This is likely a remnant of commit
59f11be43 ("MEDIUM: lua-thread: Add the lua-load-per-thread directive").
There should likely be some memory leak reports when using Lua without
this fix, though none were observed for now.

No backport is needed as this was merged into 2.4-dev.
2020-12-04 12:00:11 +01:00
Thierry Fournier
aafc777854 BUG/MEDIUM: lua-thread: some parts must be initialized once
Lua dedicated TCP, HTTP and SSL socket and proxies must be initialized
once. Right now, they are initialized from the Lua init state, but since
commit 59f11be43 ("MEDIUM: lua-thread: Add the lua-load-per-thread
directive") this function is called one time per lua context. This
caused some fields to be cleared and overwritten, and pre-allocated
object to be lost. This is why the address sanitizer detected memory
leaks from the socket_ssl server initialization.

Let's move all the state-independent part of the function to the
hlua_init() function to avoid this.

No backport is needed, this is only 2.4-dev.
2020-12-04 11:55:05 +01:00
Remi Tricot-Le Breton
51058d64a6 MINOR: cache: Consider invalid Age values as stale
Do not store responses that have an invalid age header (non numerical,
negative ...).
2020-12-04 10:21:56 +01:00
Remi Tricot-Le Breton
72cffaf440 MEDIUM: cache: Remove cache entry in case of POST on the same resource
In case of successful unsafe method on a stored resource, the cached entry
must be invalidated (see RFC7234#4.4).
A "non-error response" is one with a 2xx (Successful) or 3xx (Redirection)
status code.
This implies that the primary hash must now be calculated on requests
that have an unsafe method (POST or PUT for instance) so that we can
disable the corresponding entries when we process the response.
2020-12-04 10:21:56 +01:00
Remi Tricot-Le Breton
fcea374fdf MINOR: cache: Add extra "cache-control" value checks
The Cache-Control max-age and s-maxage directives should be followed by
a positive numerical value (see RFC 7234#5.2.1.1). According to the
specs, a sender "should not" generate a quoted-string value but we will
still accept this format.
2020-12-04 10:21:56 +01:00
Remi Tricot-Le Breton
795e1412b0 MINOR: cache: Do not store stale entry
When a response has an Age header (filled in by another cache on the
message's path) that is greater than its defined maximum age (extracted
either from cache-control directives or an expires header), it is
already stale and should not be cached.
2020-12-04 10:21:56 +01:00
David Carlier
2d0493af49 BUILD/MINOR: haproxy DragonFlyBSD affinity build update.
sched_setaffinity supported by this platform.
2020-12-02 22:43:57 +01:00
Thierry Fournier
46278ff828 MINOR: lua-thread: Add verbosity in errors
Because lua-load-per-thread could not load the same code for each thread,
this patch displays the state-id associated with the error.
2020-12-02 21:53:16 +01:00
Thierry Fournier
59f11be436 MEDIUM: lua-thread: Add the lua-load-per-thread directive
The goal is to allow execution of one main lua state per thread.

This patch contains the main job. The lua init is done using these
steps:
 - "lua-load-per-thread" loads the lua code in the first thread
 - it creates the structs
 - it stores loaded files
 - the 1st step load is completed (execution of hlua_post_init)
   and now, we known the number of threads
 - we initilize lua states for all remaining threads
 - for each one, we load the lua file
 - for each one, we execute post-init

Once all is loaded, we control consistency of functions references.
The rules are:
 - a function reference cannot be in the shared lua state and in
   a per-thread lua state at the same time.
 - if a function reference is declared in a per-thread lua state, it
   must be declared in all per-thread lua states
2020-12-02 21:53:16 +01:00
Thierry Fournier
c749259dff MINOR: lua-thread: Store each function reference and init reference in array
The goal is to allow execution of one main lua state per thread.

The array introduces storage of one reference per thread, because each
lua state can have different reference id for a same function. A function
returns the preferred state id according to configuration and current
thread id.
2020-12-02 21:53:16 +01:00
Thierry Fournier
021d986ecc MINOR: lua-thread: Replace state_from by state_id
The goal is to allow execution of one main lua state per thread.

"state_from" is a pointer to the parent lua state. "state_id"
is the index of the parent state id in the reference lua states
array. "state_id" is better because the lock is a "== 0" test
which is quick than pointer comparison. In other way, the state_id
index could index other things the the Lua state concerned. I
think to the function references.
2020-12-02 21:53:16 +01:00
Thierry Fournier
62a22aa23f MINOR: lua-thread: Replace "struct hlua_function" allocation by dedicated function
The goal is to allow execution of one main lua state per thread.

This function will initialize the struct with other things than 0.
With this function helper, the initialization is centralized and
it prevents mistakes. This patch also keeps a reference to each
declared function in a list. It will be useful in next patches to
control consistency of declared references.
2020-12-02 21:53:16 +01:00
Thierry Fournier
afc63e2cb1 MINOR: lua-thread: Replace global gL var with an array of states
The goal is to allow execution of one main lua state per thread.

The array of states is initialized at the max number of thread +1.
We define the index 0 is the common state shared by all threads
and should be locked. Other index index are dedicated to each
one thread. The old gL now becomes hlua_states[0].
2020-12-02 21:53:16 +01:00
Thierry Fournier
7cbe5046e8 MEDIUM: lua-thread: Apply lock only if the parent state is the main thread
The goal is to allow execution of one main lua state per thread.

This patch opens the way to addition of a per-thread dedicated lua state.
By passing the hlua we can figure the original state that's been used
and decide to lock or not.
2020-12-02 21:53:16 +01:00
Thierry Fournier
3c539327f4 MEDIUM: lua-thread: No longer use locked context in initialization parts
The goal is to allow execution of one main lua state per thread.

Stop using locks in init part, we will use only in parts where
the parent lua state is known, so we could take decision about lock
according with the lua parent state.
2020-12-02 21:53:16 +01:00
Thierry Fournier
ecb83c24c4 MINOR: lua-thread: Add the "thread" core variable
The goal is to allow execution of one main lua state per thread.

This commit introduces this variable in the core. Lua state initialized
by thread will have access to this variable, which reports the executing
thread. 0 indicates the shared thread. Programs which must be executed
only once can check for core.thread <= 1.
2020-12-02 21:53:16 +01:00
Thierry Fournier
b8cef175bd MINOR: lua-thread: Split hlua_post_init() function in two parts
The goal is to allow execution of one main lua state per thread.

This function will be called for each initialized lua state, so
one per thread. The split transforms the lua state variable from
global to local.
2020-12-02 21:53:16 +01:00
Thierry Fournier
c93c15cf8c MINOR: lua-thread: Split hlua_load function in two parts
The goal is to allow execution of one main lua state per thread.

This function will be called once per thread, using different Lua
states. This patch prepares the work.
2020-12-02 21:53:16 +01:00
Thierry Fournier
75fc02956b MINOR: lua-thread: make hlua_ctx_init() get L from its caller
The goal is to allow execution of one main lua state per thread.

The function hlua_ctx_init() now gets the original lua state from
its caller. This allows the initialisation of lua_thread (coroutines)
from any master lua state.

The parent lua state is stored in the hlua struct.

This patch is a temporary transition, it will be modified later.
2020-12-02 21:53:16 +01:00
Thierry Fournier
1eac28f5fc MINOR: lua-thread: Split hlua_init() function in two parts
The goal is to allow execution of one main lua state per thread.

This is a preparative work in order to init more than one stack
in the lua-thread objective.
2020-12-02 21:53:16 +01:00
Thierry Fournier
ad5345fed7 MINOR: lua-thread: Replace embedded struct hlua_function by a pointer
The goal is to allow execution of one main lua state per thread.

Because this struct will be filled after the configuration parser, we
cannot copy the content. The actual state of the Haproxy code doesn't
justify this change, it is an update preparing next steps.
2020-12-02 21:53:16 +01:00
Thierry Fournier
92689e651e MINOR: lua-thread: Stop usage of struct hlua for the global lua state
The goal is to no longer use "struct hlua" with global main lua_state.

The usage of the "struct hlua" is no longer required. This patch replaces
this struct by another one.

Now, the usage of runtime Lua phase is separated from the start lua phase.
2020-12-02 21:53:16 +01:00
Thierry Fournier
4234dbd03b MINOR: lua-thread: Use NULL context for main lua state
The goal is to no longer use "struct hlua" with global main lua_state.

This patch returns NULL value when some code tries go get the hlua struct
associated with a task through hlua_gethlua(). This functions is useful
only during runtime because the struct hlua contains only runtime states.

Some Lua functions allowed to yield are called from init environment.
I'm not sure this is a good practice. Maybe it will be clever to
disallow calling this kind of functions.
2020-12-02 21:53:16 +01:00
Thierry Fournier
9eb3230b7c MINOR: lua-thread: hlua_ctx_renew() is never called with main gL lua state
The goal is no longer using "struct hlua" with global main lua_state.

if somewhere in the code, hlua_ctx_renew() is called with a global Lua
context, we have a serious bug. A crash is better than working with
this bug, so this patch remove a useless control.

In other way, this control were used during hlua_post_init() function.
The function hlua_post_init() used a call to the runtime hlua_ctx_resume()
function. This call no longer exists.
2020-12-02 21:53:16 +01:00
Thierry Fournier
670db24329 MEDIUM: lua-thread: make hlua_post_init() no longer use the runtime execution function
The goal is to no longer use "struct hlua" with global main lua_state.

The hlua_post_init() is executed during start phase, it does not require
yielding nor any advanced runtime error processing. Let's simplify this
by re-implementing the code using lower-level functions which directly
take a state and not an hlua anymore.
2020-12-02 21:53:16 +01:00
Thierry Fournier
3fb9e5133a MINOR: lua-thread: remove struct hlua from function hlua_prepend_path()
The goal is to no longer use "struct hlua" with global main lua_state
and directly take the state instead.

This patch removes the implicit dependency to this struct with
the function hlua_prepend_path()
2020-12-02 21:53:16 +01:00
Willy Tarreau
cdb53465f4 MEDIUM: lua-thread: use atomics for memory accounting
Let's switch memory accounting to atomics so that the allocator function
may safely be used from concurrent Lua states.

Given that this function is extremely hot on the call path, we try to
optimize it for the most common case, which is:
  - no limit
  - there's enough memory

The accounting is what is particuarly expensive in threads since all
CPUs compete for a cache line, so when the limit is not used, we don't
want to use accounting. However we need to preserve it during the boot
phase until we may parse a "tune.lua.maxmem" value. For this, we turn
the unlimited "0" value to ~0 at the end of the boot phase to mark the
definite end of accounting. The function then detects this value and
directly jumps to realloc() in this case.

When the limit is enforced however, we use a CAS to check and reserve
our share of memory, and we roll back on failure. The CAS is used both
for increments and decrements so that a single operation is enough to
update the counters.
2020-12-02 21:53:16 +01:00
Willy Tarreau
d36c7fa5ec MINOR: lua: simplify hlua_alloc() to only rely on realloc()
The function really has the semantics of a realloc() except that it
also passes the old size to help with accounting. No need to special
case the free or malloc, realloc does everything we need.
2020-12-02 21:53:16 +01:00
Emeric Brun
fdabf49548 BUG/MAJOR: ring: tcp forward on ring can break the reader counter.
If the session is not established, the applet handler could leave
with the applet detached from the ring. At next call, the attach
counter will be decreased again causing unpredectable behavior.

This patch should be backported on branches >=2.2
2020-12-02 20:17:19 +01:00
Frédéric Lécaille
fd1831499e BUG/MINOR: trace: Wrong displayed trace level
With commit a1f12746b ("MINOR: traces: add a new level "error" below
the "user" level") a new trace level was inserted, resulting in
shifting all exiting ones by one. But the levels reported in the
__trace() function were not updated accordingly, resulting in the
TRACE_LEVEL_DEVELOPER not to be properly reported anymore. This
patch fixes it by extending the number of levels to 6.

No backport is needed.
2020-12-02 17:44:40 +01:00
Remi Tricot-Le Breton
3243447f83 MINOR: cache: Add entry to the tree as soon as possible
When many concurrent requests targeting the same resource were seen, the
cache could sometimes be filled by too many partial responses resulting
in the impossibility to cache a single one of them. This happened
because the actual tree insertion happened only after all the payload of
every response was seen. So until then, every response was added to the
cache because none of the streams knew that a similar request/response
was already being treated.
This patch consists in adding the cache_entry as soon as possible in the
tree (right after the first packet) so that the other responses do not
get cached as well (if they have the same primary key).
A "complete" flag is also added to the cache_entry so that we know if
all the payload is already stored in the entry or if it is still being
processed.
2020-12-02 16:38:42 +01:00
Remi Tricot-Le Breton
8bb72aa82f MINOR: cache: Improve accept_encoding_normalizer
Turn the "Accept-Encoding" value to lower case before processing it.
Calculate the CRC on every token instead of a sorted concatenation of
them all (in order to avoir copying them) then XOR all the CRCs into a
single hash (while ignoring duplicates).
2020-12-02 16:32:54 +01:00
Thierry Fournier
f67442efdb BUG/MINOR: lua: warn when registering action, conv, sf, cli or applet multiple times
Lua allows registering multiple sample-fetches, converters, action, cli,
applet/services with the same name. This is absolutely useless since only
the first registration will be used. This patch sends a warning if the case
is encountered.

This pach could be backported until 1.8, with the 3 associated patches:
 - MINOR: actions: Export actions lookup functions
 - MINOR: actions: add a function returning a service pointer from its name
 - MINOR: cli: add a function to look up a CLI service description
2020-12-02 09:45:18 +01:00
Thierry Fournier
a51a1fd174 MINOR: cli: add a function to look up a CLI service description
This function will be useful to check if the keyword is already registered.
Also add a define for the max number of args.

This will be needed by a next patch to fix a bug and will have to be
backported.
2020-12-02 09:45:18 +01:00
Thierry Fournier
87e539906b MINOR: actions: add a function returning a service pointer from its name
This function simply calls action_lookup() on the private service_keywords,
to look up a service name. This will be used to detect double registration
of a same service from Lua.

This will be needed by a next patch to fix a bug and will have to be
backported.
2020-12-02 09:45:18 +01:00
Thierry Fournier
7a71a6d9d2 MINOR: actions: Export actions lookup functions
These functions will be useful to check if a keyword is already registered.
This will be needed by a next patch to fix a bug, and will need to be
backported.
2020-12-02 09:45:18 +01:00
Thierry Fournier
2f05cc6f86 BUG/MINOR: lua: Some lua init operation are processed unsafe
Operation luaL_openlibs() and lua_prepend path are processed whithout
the safe context, so in case of failure Haproxy aborts or stops without
error message.

This patch could be backported until 1.8
2020-12-02 09:45:18 +01:00
Thierry Fournier
13d08b73eb BUG/MINOR: lua: Post init register function are not executed beyond the first one
Just because if the first init is a success we return success in place
of continuing the loop.

This patch could be backported until 1.8
2020-12-02 09:45:18 +01:00