Commit graph

6002 commits

Author SHA1 Message Date
Christopher Faulet
13045f0eae MINOR: stream-int-conn-stream: Move si_update_* in conn-stream scope
si_update_rx(), si_update_tx() and si_update() are renamed cs_update_rx(),
cs_upate_tx() and cs_update() and updated to manipulate a conn-stream
instead of a stream-interface.
2022-04-13 15:10:15 +02:00
Christopher Faulet
9ffddd5ca5 REORG: conn-stream: Move cs_app_ops in conn_stream.c
Callback functions to perform shutdown for reads and writes and to trigger
I/O calls are now moved in conn_stream.c.
2022-04-13 15:10:15 +02:00
Christopher Faulet
19bd728642 REORG: conn-stream: Move cs_shut* and cs_chk* in cs_utils
cs_shutr(), cs_shutw(), cs_chk_rcv() and cs_chk_snd() are moved in
cs_utils.h
2022-04-13 15:10:15 +02:00
Christopher Faulet
dde33046bd REORG: stream-int: Move si_is_conn_error() in the header file
To ease next changes, this function is moved in the header file. It is a
transient commit.
2022-04-13 15:10:15 +02:00
Christopher Faulet
9b7a9b400d REORG: stream-int: Export si_cs_recv(), si_cs_send() and si_cs_process()
It is a transient commit. It should ease next changes about the conn-stream
refactoring. At the end these functions will be moved in the conn-stream
scope.
2022-04-13 15:10:15 +02:00
Christopher Faulet
aa91d6292b MINOR: stream-int/connection: Move conn_si_send_proxy() in the connection scope
conn_si_send_proxy() function is renamed conn_send_proxy() and moved in
connection.c
2022-04-13 15:10:15 +02:00
Christopher Faulet
64b8d33577 MINOR: connection: unconst mux's get_fist_cs() callback function
This change is mandatory for next commits.
2022-04-13 15:10:15 +02:00
Christopher Faulet
3704663e5f MINOR: applet: Use the CS to register and release applets instead of SI
si_register_applet() and si_applet_release() are renamed
cs_register_applet() and cs_applet_release() and now manipulate a
conn-stream instead of a stream-inteface.
2022-04-13 15:10:15 +02:00
Christopher Faulet
0c6a64cd5f MEDIUM: stream-int/conn-stream: Move si_ops in the conn-stream scope
The si_ops structure is renamed to cs_app_ops and the callback functions are
changed to manipulate a conn-stream instead of a stream-interface..
2022-04-13 15:10:15 +02:00
Christopher Faulet
da098e6c17 MINOR: stream-int/conn-stream: Move si_shut* and si_chk* in conn-stream scope
si_shutr(), si_shutw(), si_chk_rcv() and si_chk_snd() are moved in the
conn-stream scope and renamed, respectively, cs_shutr(), cs_shutw(),
cs_chk_rcv(), cs_chk_snd() and manipulate a conn-stream instead of a
stream-interface.
2022-04-13 15:10:15 +02:00
Christopher Faulet
69ef6c9ef4 MINOR: conn-stream: Rename CS functions dedicated to connections
Some conn-stream functions are only used when there is a connection. Thus,
they was renamed with "cs_conn_" prefix. In addition, we expect to have a
connection, so a BUG_ON is added to be sure the functions are never called
in another context.
2022-04-13 15:10:15 +02:00
Christopher Faulet
2f35e7b6ab MEDIUM: stream-int/conn-stream: Handle I/O subscriptions in the conn-stream
wait_event structure is moved in the conn-stream. The tasklet is only
created if the conn-stream is attached to a mux and released when the mux is
detached. This implies a subtle change. In stream_int_chk_rcv() function,
the wakeup of the tasklet was removed because there is no longer tasklet at
this stage (stream_int_chk_rcv() is a callback function of si_embedded_ops).
2022-04-13 15:10:15 +02:00
Christopher Faulet
070b91bc11 MEDIUM: conn-stream: Be prepared to fail to attach a cs to a mux
To be able to move wait_event from the stream-interface to the conn-stream,
we must be prepare to handle errors when a mux is attached to a conn-stream.
Indeed, the wait_event's tasklet will be allocated when both a mux and a
stream will be both attached to a stream. So, we must be prepared to handle
allocation errors.
2022-04-13 15:10:15 +02:00
Christopher Faulet
0797656ead MINOR: conn-stream/connection: Move SHR/SHW modes in the connection scope
These flags only concerns the connection part. In addition, it is required
for a next commit, to avoid circular deps. Thus CS_SHR_* and CS_SHW_* were
renamed with the "CO_" prefix.
2022-04-13 15:10:15 +02:00
Christopher Faulet
e39a4dfdf0 MINOR: stream-int/conn-stream: Move si_conn_ready() in the conn-stream scope
si_conn_ready() is renamed cs_conn_ready() and handle a conn-stream insted
of a stream-interface. The function is now in cs_utils.h.
2022-04-13 15:10:15 +02:00
Christopher Faulet
0a4dcb65ff MINOR: stream-int/backend: Move si_connect() in the backend scope
si_connect() is moved in backend.c and renamed as do_connect_server(). In
addition, the function now manipulate a stream instead of a
stream-interface.
2022-04-13 15:10:15 +02:00
Christopher Faulet
9125f3cc77 MINOR: stream-int/stream: Move si_retnclose() in the stream scope
si_retnclose() is used to send a reply to a client before closing. There is
no use on the server side, in spite of the function is generic. Thus, it is
renamed stream_retnclose() and moved into the stream scope. The function now
handle a stream and explicitly send a message to the client.
2022-04-13 15:10:15 +02:00
Christopher Faulet
62e757470a MEDIUM: stream-int/conn-stream: Move stream-interface state in the conn-stream
The stream-interface state (SI_ST_*) is now in the conn-stream. It is a
mechanical replacement for now. Nothing special. SI_ST_* and SI_SB_* were
renamed accordingly. Utils functions to manipulate these infos were moved
under the conn-stream scope.

But it could be good to keep in mind that this part should be
reworked. Indeed, at the CS level, we only need to know if it is ready to
receive or to send. The state of conn-stream from INI to EST is only used on
the server side. The client CS is immediately set to EST. Thus current
SI_ST_* states should probably be moved to the stream to reflect the server
connection state during the establishment stage.
2022-04-13 15:10:15 +02:00
Christopher Faulet
50264b41c8 MEDIUM: stream-int: Move SI err_type in the stream
Only the server side is concerned by the stream-interface error type. It is
useless to have an err_type field on the client side. So, it is now move to
the stream. SI_ET_* are renames STRM_ET_* and moved in stream-t.h header
file.
2022-04-13 15:10:14 +02:00
Christopher Faulet
a70a3548bc MINOR: stream: Only save previous connection state for the server side
The previous connection state on the client side was only used for debugging
purpose to report client close. But this may be handled when the client
stream-interface is switched from SI_ST_DIS to SI_ST_CLO.

So, there only remains the previous connection state on the server side that
is used by the stream, in process_stream(), to be able to set the correct
termination flags. Thus, instead of keeping this info in the
stream-interface for only one side, the info is now stored in the stream
itself.
2022-04-13 15:10:14 +02:00
Christopher Faulet
78ed7f247b CLEANUP: stream-int: Remove unused SI_FL_CLEAN_ABRT flag
This flag is unused. So remove it to be able to remove the stream-interface.
2022-04-13 15:10:14 +02:00
Christopher Faulet
d139138bbc MINOR: stream-int: Remove SI_FL_SRC_ADDR to rely on stream flags instead
Flag to get the source ip/port with getsockname is now handled at the stream
level. Thus SI_FL_SRC_ADDR stream-int flag is replaced by SF_SRC_ADDR stream
flag.
2022-04-13 15:10:14 +02:00
Christopher Faulet
a728518c15 MINOR: stream-int: Remove SI_FL_INDEP_STR to rely on CS flags instead
Flag to consider a stream as indepenent is now handled at the conn-stream
level. Thus SI_FL_INDEP_STR stream-int flag is replaced by CS_FL_INDEP_STR
conn-stream flags.
2022-04-13 15:10:14 +02:00
Christopher Faulet
974da9f8a4 MINOR: stream-int: Remove SI_FL_DONT_WAKE to rely on CS flags instead
Flag to not wake the stream up on I/O is now handled at the conn-stream
level. Thus SI_FL_DONT_WAKE stream-int flag is replaced by CS_FL_DONT_WAKE
conn-stream flags.
2022-04-13 15:10:14 +02:00
Christopher Faulet
8abe712749 MINOR: stream-int: Remove SI_FL_NOLINGER/NOHALF to rely on CS flags instead
Flags to disable lingering and half-close are now handled at the conn-stream
level. Thus SI_FL_NOLINGER and SI_FL_NOHALF stream-int flags are replaced by
CS_FL_NOLINGER and CS_FL_NOHALF conn-stream flags.
2022-04-13 15:10:14 +02:00
Christopher Faulet
9a52123800 MINOR: stream-int: Remove SI_FL_KILL_CON to rely on conn-stream endpoint only
Instead of setting a stream-interface flag to then set the corresponding
conn-stream endpoint flag, we now only rely the conn-stream endoint. Thus
SI_FL_KILL_CON is replaced by CS_EP_KILL_CONN.

In addition si_must_kill_conn() is replaced by cs_must_kill_conn().
2022-04-13 15:10:14 +02:00
Christopher Faulet
7b5ca8f457 MINOR: channel: Use conn-streams as channel producer and consumer
chn_prod() and chn_cons() now return a conn-stream instead of a
stream-interface.
2022-04-13 15:10:14 +02:00
Christopher Faulet
6cd56d5a69 MEDIUM: conn-stream: Use endpoint error instead of conn-stream error
Instead of relying on the conn-stream error, via CS_FL_ERR flags, we now
directly use the error at the endpoint level with the flag CS_EP_ERROR. It
should be safe to do so. But we must be careful because it is still possible
that an error is processed too early. Anyway, a conn-stream has always a
valid endpoint, maybe detached from any endpoint, but valid.
2022-04-13 15:10:14 +02:00
Christopher Faulet
af642df3b8 MINOR: stream-int/conn-stream: Report error to the CS instead of the SI
SI_FL_ERR is removed and replaced by CS_FL_ERROR. It is a transient patch
because the idea is to rely on the endpoint to handle errors at this
level. But if for any reason it is not possible, the stream-interface flags
will still be replaced.
2022-04-13 15:10:14 +02:00
Christopher Faulet
ae024ced03 MEDIUM: stream-int/stream: Use connect expiration instead of SI expiration
The expiration date in the stream-interface was only used on the server side
to set the connect, queue or turn-around timeout. It was checked on the
frontend stream-interface, but never used concretely. So it was removed and
replaced by a connect expiration date in the stream itself. Thus, SI_FL_EXP
flag in stream-interfaces is replaced by a stream flag, SF_CONN_EXP.
2022-04-13 15:10:14 +02:00
Christopher Faulet
1d9877700e MINOR: stream-int/conn-stream: Move half-close timeout in the conn-stream
The half-close timeout (hcto) is now part of the conn-stream. It is a step
closer to the stream-interface removal.
2022-04-13 15:10:14 +02:00
Christopher Faulet
8da67aae3e MEDIUM: stream-int/conn-stream: Move src/dst addresses in the conn-stream
The source and destination addresses at the applicative layer are moved from
the stream-interface to the conn-stream. This simplifies a bit the code and
it is a logicial step to remove the stream-interface.
2022-04-13 15:10:14 +02:00
Christopher Faulet
731c8e6cf9 MINOR: stream: Simplify retries counter calculation
The conn_retries counter was set to the max value and decremented at each
connection retry. Thus the counter reflected the number of retries left and
not the real number of retries. All calculations of redispatch or reporting
of number of retries experienced were made using subtracts from the
configured retries, which was complicated and didn't bring any benefit.

Now, this counter is set to 0 and incremented at each retry. We know we've
reached the maximum allowed connection retries by comparing it to the
configured value. In all other cases, we directly use the counter.

This patch should address the feature request #1608.
2022-04-13 15:10:14 +02:00
Christopher Faulet
909f318259 MINOR: stream-int/stream: Move conn_retries counter in the stream
The conn_retries counter may be moved into the stream structure. It only
concerns the connection establishment. The frontend stream-interface does not
use it. So it is a logical change.
2022-04-13 15:10:14 +02:00
Christopher Faulet
e05bf9e413 MINOR: stream-int/txn: Move buffer for L7 retries in the HTTP transaction
The L7 retries only concerns the stream when a server connection is
established. Thus instead of storing the L7 buffer into the
stream-interface, it may be moved to the stream. And because it is only
available for HTTP streams, it may be moved in the HTTP transaction.

Associated flags are also moved into the HTTP transaction.
2022-04-13 15:10:14 +02:00
Christopher Faulet
908628c4c0 MEDIUM: tree-wide: Use CS util functions instead of SI ones
At many places, we now use the new CS functions to get a stream or a channel
from a conn-stream instead of using the stream-interface API. It is the
first step to reduce the scope of the stream-interfaces. The main change
here is about the applet I/O callback functions. Before the refactoring, the
stream-interface was the appctx owner. Thus, it was heavily used. Now, as
far as possible,the conn-stream is used. Of course, it remains many calls to
the stream-interface API.
2022-04-13 15:10:14 +02:00
Christopher Faulet
1cf9ef25a9 MINOR: conn-stream: Add header file with util functions related to conn-streams
cs_utils.h header file will contain all util functions related to the
conn_streams. For now, few functions were added, all are equivalent to SI
functions. Idea is to progressively replace SI functions by CS ones.
2022-04-13 15:10:14 +02:00
Christopher Faulet
3099511571 MINOR: conn-stream: Add ISBACK conn-stream flag
CS_FL_ISBACK is a new flag, set on backend conn-streams. We must just be
careful to preserve this flag when the endpoint is detached from the
conn-stream.
2022-04-13 15:10:14 +02:00
Christopher Faulet
b041b23ae4 MEDIUM: conn-stream: Move remaning flags from CS to endpoint
All old flags CS_FL_* are now moved in the endpoint scope and renamed
CS_EP_* accordingly. It is a systematic replacement. There is no true change
except for the health-check and the endpoint reset. Here it is a bit special
because the same conn-stream is reused. Thus, we must handle endpoint
allocation errors. To do so, cs_reset_endp() has been adapted.

Thanks to this last change, it will now be possible to simplify the
multiplexer and probably the applets too. A review must also be performed to
remove some flags in the channel or the stream-interface. The HTX will
probably be simplified too. Finally, there is now some place in the
conn-stream to move info from the stream-interface.
2022-04-13 15:10:14 +02:00
Christopher Faulet
9ec2f4dc7c MAJOR: conn-stream: Share endpoint struct between the CS and the mux/applet
The conn-stream endpoint is now shared between the conn-stream and the
applet or the multiplexer. If the mux or the applet is created first, it is
responsible to also create the endpoint and share it with the conn-stream.
If the conn-stream is created first, it is the opposite.

When the endpoint is only owned by an applet or a mux, it is called an
orphan endpoint (there is no conn-stream). When it is only owned by a
conn-stream, it is called a detached endpoint (there is no mux/applet).

The last entity that owns an endpoint is responsible to release it. When a
mux or an applet is detached from a conn-stream, the conn-stream
relinquishes the endpoint to recreate a new one. This way, the endpoint
state is never lost for the mux or the applet.
2022-04-13 15:10:14 +02:00
Christopher Faulet
cb2fa368e9 REORG: applet: Uninline appctx_new function
appctx_new() is moved in the C file and appctx_init() is now private.
2022-04-13 15:10:14 +02:00
Christopher Faulet
a9e8b3979d MEDIUM: conn-stream: Pre-allocate endpoint to create CS from muxes and applets
It is a transient commit to prepare next changes. Now, when a conn-stream is
created from an applet or a multiplexer, an endpoint is always provided. In
addition, the API to create a conn-stream was specialized to have one
function per type.

The next step will be to share the endpoint structure.
2022-04-13 15:10:14 +02:00
Christopher Faulet
b669d684c0 MEDIUM: conn-stream: Be able to pass endpoint to create a conn-stream
It is a transient commit to prepare next changes. It is possible to pass a
pre-allocated endpoint to create a new conn-stream. If it is NULL, a new
endpoint is created, otherwise the existing one is used. There no more
change at the conn-stream level.

In the applets, all conn-stream are created with no pre-allocated
endpoint. But for multiplexers, an endpoint is systematically created before
creating the conn-stream.
2022-04-13 15:10:14 +02:00
Christopher Faulet
e9e4820288 MINOR: conn-stream: Move some CS flags to the endpoint
Some CS flags, only related to the endpoint, are moved into the endpoint
struct. More will probably moved later. Those ones are not critical. So it
is pretty safe to move them now and this will ease next changes.
2022-04-13 15:10:14 +02:00
Christopher Faulet
db90f2aa9f MEDIUM: conn-stream: Add an endpoint structure in the conn-stream
Group the endpoint target of a conn-stream, its context and the associated
flags in a dedicated structure in the conn-stream. It is not inlined in the
conn-stream structure. There is a dedicated pool.

For now, there is no complexity. It is just an indirection to get the
endpoint or its context. But the purpose of this structure is to be able to
share a refcounted context between the mux and the conn-stream. This way, it
will be possible to preserve it when the mux is detached from the
conn-stream.
2022-04-13 15:10:14 +02:00
Christopher Faulet
bb772d09f5 REORG: Initialize the conn-stream by hand in cs_init()
The function cs_init() is only called by cs_new(). The conn-stream
initialization will be reviewed. It is easier to do it in cs_new() instead
of using a dedicated function. cs_new() is pretty simple, there is no reason
to split the code in this case.
2022-04-13 15:10:14 +02:00
Christopher Faulet
9388204db1 MAJOR: conn-stream: Invert conn-stream endpoint and its context
This change is only significant for the multiplexer part. For the applets,
the context and the endpoint are the same. Thus, there is no much change. For
the multiplexer part, the connection was used to set the conn-stream
endpoint and the mux's stream was the context. But it is a bit strange
because once a mux is installed, it takes over the connection. In a
wonderful world, the connection should be totally hidden behind the mux. The
stream-interface and, in a lesser extent, the stream, still access the
connection because that was inherited from the pre-multiplexer era.

Now, the conn-stream endpoint is the mux's stream (an opaque entity for the
conn-stream) and the connection is the context. Dedicated functions have
been added to attached an applet or a mux to a conn-stream.
2022-04-13 15:10:14 +02:00
Christopher Faulet
2479e5f775 MEDIUM: applet: Set the appctx owner during allocation
The appctx owner is now always a conn-stream. Thus, it can be set during the
appctx allocation. But, to do so, the conn-stream must be created first. It
is not a problem on the server side because the conn-stream is created with
the stream. On the client side, we must take care to create the conn-stream
first.

This change should ease other changes about the applets bootstrapping.
2022-04-13 15:10:13 +02:00
Christopher Faulet
81a40f630e MINOR: conn-stream: Add flags to set the type of the endpoint
This patch is mandatory to invert the endpoint and the context in the
conn-stream. There is no common type (at least for now) for the entity
representing a mux (h1s, h2s...), thus we must set its type when the
endpoint is attached to a conn-stream. There is 2 types for the conn-stream
endpoints: the mux (CS_FL_ENDP_MUX) and the applet (CS_FL_ENDP_APP).
2022-04-13 15:10:13 +02:00
Christopher Faulet
4aa1d2838c MINOR: applet: Make .init callback more generic
For now there is no much change. Only the appctx is passed as argument when
the .init callback function is called. And it is not possible to yield at
this stage. It is not a problem because the feature is not used. Only the
lua defines this callback function for the lua TCP/HTTP services. The idea
is to be able to use it for all applets to initialize the appctx context.
2022-04-13 15:10:13 +02:00
Willy Tarreau
3a0a0d6cc1 BUILD: ssl: add an unchecked version of __conn_get_ssl_sock_ctx()
First gcc, then now coverity report possible null derefs in situations
where we know these cannot happen since we call the functions in
contexts that guarantee the existence of the connection and the method
used. Let's introduce an unchecked version of the function for such
cases, just like we had to do with objt_*. This allows us to remove the
ALREADY_CHECKED() statements (which coverity doesn't see), and addresses
github issues #1643, #1644, #1647.
2022-04-12 07:33:26 +02:00
Willy Tarreau
784b868c97 MEDIUM: quic: move conn->qc into conn->handle
It was supposed to be there, and probably was not placed there due to
historic limitations in listener_accept(), but now there does not seem
to be a remaining valid reason for keeping the quic_conn out of the
handle. In addition in new_quic_cli_conn() the handle->fd was incorrectly
set to the listener's FD.
2022-04-11 19:33:04 +02:00
Willy Tarreau
de827958a2 MEDIUM: ssl: improve retrieval of ssl_sock_ctx and SSL detection
Historically there was a single way to have an SSL transport on a
connection, so detecting if the transport layer was SSL and a context
was present was sufficient to detect SSL. With QUIC, things have changed
because QUIC also relies on SSL, but the context is embedded inside the
quic_conn and the transport layer doesn't match expectations outside,
making it difficult to detect that SSL is in use over the connection.

The approach taken here to improve this consists in adding a new method
at the transport layer, get_ssl_sock_ctx(), to retrieve this often needed
ssl_sock_ctx, and to use this to detect the presence of SSL. This will
even allow some simplifications and cleanups to be made in the SSL code
itself, and QUIC will be able to provide one to export its ssl_sock_ctx.
2022-04-11 19:33:04 +02:00
Willy Tarreau
cdf7c8e543 MINOR: quic-sock: provide a pair of get_src/get_dst functions
These functions will allow the connection layer to retrieve a quic_conn's
source or destination when possible. The quic_conn holds the peer's address
but not the local one, and the sockets API doesn't always makes that easy
for datagrams. Thus for frontend connection what we're doing here is to
retrieve the listener's address when the destination address is desired.

Now it finally becomes possible to fetch the source and destination using
"src" and "dst", and to pass an incoming connection's endpoints via the
proxy protocol.
2022-04-11 19:33:04 +02:00
Willy Tarreau
e151609110 MINOR: protocol: add get_src() and get_dst() at the protocol level
Right now the proto_fam descriptor provides a family-specific
get_src() and get_dst() pair of calls to retrieve a socket's source
or destination address. However this only works for connected mode
sockets. QUIC provides its own stream protocol, which relies on a
datagram protocol underneath, so the get_src()/get_dst() at that
protocol's family will not work, and QUIC would need to provide its
own.

This patch implements get_src() and get_dst() at the protocol level
from a connection, and makes sure that conn_get_src()/conn_get_dst()
will automatically use them if defined before falling back to the
family's pair of functions.
2022-04-11 19:33:04 +02:00
Willy Tarreau
987c08a5e2 MINOR: connection: rearrange conn_get_src/dst to be a bit more extensible
We'll want conn_get_src/dst to support other means of retrieving these
respective IP addresses, but the functions as they're designed are a bit
too restrictive for now.

This patch arranges them to have a default error fallback allowing to
test different mechanisms. In addition we now make sure the underlying
protocol is of type stream before calling the family's get_src/dst as
it makes no sense to do that on dgram sockets for example.
2022-04-11 19:33:04 +02:00
Willy Tarreau
07ecfc5e88 MEDIUM: connection: panic when calling FD-specific functions on FD-less conns
Certain functions cannot be called on an FD-less conn because they are
normally called as part of the protocol-specific setup/teardown sequence.
Better place a few BUG_ON() to make sure none of them is called in other
situations. If any of them would trigger in ambiguous conditions, it would
always be possible to replace it with an error.
2022-04-11 19:31:47 +02:00
Willy Tarreau
e22267971b MINOR: connection: skip FD-based syscalls for FD-less connections
Some syscalls at the TCP level act directly on the FD. Some of them
are used by TCP actions like set-tos, set-mark, silent-drop, others
try to retrieve TCP info, get the source or destination address. These
ones must not be called with an invalid FD coming from an FD-less
connection, so let's add the relevant tests for this. It's worth
noting that all these ones already have fall back plans (do nothing,
error, or switch to alternate implementation).
2022-04-11 19:31:47 +02:00
Willy Tarreau
83a966d025 MINOR: connection: add conn_fd() to retrieve the FD only when it exists
There are plenty of places (particularly in debug code) where we try to
dump the connection's FD only when the connection is defined. That's
already a pain but now it gets one step further with QUIC because we do
*not* want to dump this FD in this case.

conn_fd() checks if the connection exists, is ready and is not fd-less,
and returns the FD only in this case, otherwise returns -1. This aims at
simplifying most of these conditions.
2022-04-11 19:31:47 +02:00
Willy Tarreau
c78a9698ef MINOR: connection: add a new flag CO_FL_FDLESS on fd-less connections
QUIC connections do not use a file descriptor, instead they use the
quic equivalent which is the quic_conn. A number of our historical
functions at the connection level continue to unconditionally touch
the file descriptor and this may have consequences once QUIC starts
to be used.

This patch adds a new flag on QUIC connections, CO_FL_FDLESS, to
mention that the connection doesn't have a file descriptor, hence the
FD-based API must never be used on them.

From now on it will be possible to intrument existing functions to
panic when this flag is present.
2022-04-11 19:31:47 +02:00
William Lallemand
d7bfbe2333 BUILD: ssl: add USE_ENGINE and disable the openssl engine by default
The OpenSSL engine API is deprecated starting with OpenSSL 3.0.

In order to have a clean build this feature is now disabled by default.
It can be reactivated with USE_ENGINE=1 on the build line.
2022-04-11 18:41:24 +02:00
Remi Tricot-Le Breton
b5d968d9b2 MEDIUM: global: Add a "close-spread-time" option to spread soft-stop on time window
The new 'close-spread-time' global option can be used to spread idle and
active HTTP connction closing after a SIGUSR1 signal is received. This
allows to limit bursts of reconnections when too many idle connections
are closed at once. Indeed, without this new mechanism, in case of
soft-stop, all the idle connections would be closed at once (after the
grace period is over), and all active HTTP connections would be closed
by appending a "Connection: close" header to the next response that goes
over it (or via a GOAWAY frame in case of HTTP2).

This patch adds the support of this new option for HTTP as well as HTTP2
connections. It works differently on active and idle connections.

On active connections, instead of sending systematically the GOAWAY
frame or adding the 'Connection: close' header like before once the
soft-stop has started, a random based on the remainder of the close
window is calculated, and depending on its result we could decide to
keep the connection alive. The random will be recalculated for any
subsequent request/response on this connection so the GOAWAY will still
end up being sent, but we might wait a few more round trips. This will
ensure that goaways are distributed along a longer time window than
before.

On idle connections, a random factor is used when determining the expire
field of the connection's task, which should naturally spread connection
closings on the time window (see h2c_update_timeout).

This feature request was described in GitHub issue #1614.
This patch should be backported to 2.5. It depends on "BUG/MEDIUM:
mux-h2: make use of http-request and keep-alive timeouts" which
refactorized the timeout management of HTTP2 connections.
2022-04-08 18:15:21 +02:00
Frédéric Lécaille
8c7927c6dd MINOR: quic_tls: Make key update use of reusable cipher contexts
We modify the key update feature implementation to support reusable cipher contexts
as this is done for the other cipher contexts for packet decryption and encryption.
To do so we attach a context to the quic_tls_kp struct and initialize it each time
the underlying secret key is updated. Same thing when we rotate the secrets keys,
we rotate the contexts as the same time.
2022-04-08 15:38:29 +02:00
Frédéric Lécaille
f2f4a4eee5 MINOR: quic_tls: Stop hardcoding cipher IV lengths
For QUIC AEAD usage, the number of bytes for the IVs is always 12.
2022-04-08 15:38:29 +02:00
Frédéric Lécaille
f4605748f4 MINOR: quic_tls: Add reusable cipher contexts to QUIC TLS contexts
Add ->ctx new member field to quic_tls_secrets struct to store the cipher context
for each QUIC TLS context TX/RX parts.
Add quic_tls_rx_ctx_init() and quic_tls_tx_ctx_init() functions to initialize
these cipher context for RX and TX parts respectively.
Make qc_new_isecs() call these two functions to initialize the cipher contexts
of the Initial secrets. Same thing for ha_quic_set_encryption_secrets() to
initialize the cipher contexts of the subsequent derived secrets (ORTT, Handshake,
1RTT).
Modify quic_tls_decrypt() and quic_tls_encrypt() to always use the same cipher
context without allocating it each time they are called.
2022-04-08 15:38:29 +02:00
Amaury Denoyelle
b515b0af1d MEDIUM: quic: report closing state for the MUX
Define a new API to notify the MUX from the quic-conn when the
connection is about to be closed. This happens in the following cases :
- on idle timeout
- on CONNECTION_CLOSE emission or reception

The MUX wake callback is called on these conditions. The quic-conn
QUIC_FL_NOTIFY_CLOSE is set to only report once. On the MUX side,
connection flags CO_FL_SOCK_RD_SH|CO_FL_SOCK_WR_SH are set to interrupt
future emission/reception.

This patch is the counterpart to
  "MEDIUM: mux-quic: report CO_FL_ERROR on send".
Now the quic-conn is able to report its closing, which may be translated
by the MUX into a CO_FL_ERROR on the connection for the upper layer.
This allows the MUX to properly react to the QUIC closing mechanism for
both idle-timeout and closing/draining states.
2022-04-07 10:37:45 +02:00
Amaury Denoyelle
fe035eca3a MEDIUM: mux-quic: report errors on conn-streams
Complete the error reporting. For each attached streams, if CO_FL_ERROR
is set, mark them with CS_FL_ERR_PENDING|CS_FL_ERROR. This will notify
the upper layer to trigger streams detach and release of the MUX.

This reporting is implemented in a new function qc_wake_some_streams(),
called by qc_wake(). This ensures that a lower-layer error is quickly
reported to the individual streams.
2022-04-07 10:37:45 +02:00
Amaury Denoyelle
198d35f9c6 MINOR: mux-quic: define is_active app-ops
Add a new app layer operation is_active. This can be used by the MUX to
check if the connection can be considered as active or not. This is used
inside qcc_is_dead as a first check.

For example on HTTP/3, if there is at least one bidir client stream
opened the connection is active. This explicitly ignore the uni streams
used for control and qpack as they can never be closed during the
connection lifetime.
2022-04-07 10:23:10 +02:00
Amaury Denoyelle
06890aaa91 MINOR: mux-quic: adjust timeout to accelerate closing
Improve timeout handling on the MUX. When releasing a stream, first
check if the connection can be considered as dead and should be freed
immediatly. This allows to liberate resources faster when possible.

If the connection is still active, ensure there is no attached
conn-stream before scheduling the timeout. To do this, add a nb_cs field
in the qcc structure.
2022-04-07 10:23:10 +02:00
Amaury Denoyelle
846cc046ae MINOR: mux-quic: factorize conn-stream attach
Provide a new function qc_attach_cs. This must be used by the app layer
when a conn-stream can be instantiated. This will simplify future
development.
2022-04-07 10:10:23 +02:00
Amaury Denoyelle
6057b4090e CLEANUP: mux-quic: remove unused QC_CF_CC_RECV
This flag was used to notify the MUX about a CONNECTION_CLOSE frame
reception. It is now unused on the MUX side and can be removed. A new
mechanism to detect quic-conn closing will be soon implemented.
2022-04-07 10:10:23 +02:00
Amaury Denoyelle
db71e3bd09 BUG/MEDIUM: quic: ensure quic-conn survives to the MUX
Rationalize the lifetime of the quic-conn regarding with the MUX. The
quic-conn must not be freed if the MUX is still allocated.

This simplify the MUX code when accessing the quic-conn and removed
possible segfaults.

To implement this, if the quic-conn timer expired, the quic-conn is
released only if the MUX is not allocated. Else, the quic-conn is
flagged with QUIC_FL_CONN_EXP_TIMER. The MUX is then responsible
to call quic_close() which will free the flagged quic-conn.
2022-04-07 10:10:22 +02:00
Frédéric Lécaille
59bf255806 MINOR: quic: Add closing connection state
New received packets after sending CONNECTION_CLOSE frame trigger a new
CONNECTION_CLOSE frame to be sent. Each time such a frame is sent we
increase the number of packet required to send another CONNECTION_CLOSE
frame.
Rearm only one time the idle timer when sending a CONNECTION_CLOSE frame.
2022-04-06 15:52:35 +02:00
Frédéric Lécaille
47756809fb MINOR: quic: Add draining connection state.
As soon as we receive a CONNECTION_CLOSE frame, we must stop sending packets.
We add QUIC_FL_CONN_DRAINING connection flag to do so.
2022-04-06 15:52:35 +02:00
Frédéric Lécaille
eb2a2da67c BUG/MINOR: quic: Missing TX packet deallocations
Ensure all TX packets are deallocated. There may be remaining ones which
will never be acknowledged or deemed lost.
2022-04-01 16:26:06 +02:00
Frédéric Lécaille
b823bb7f7f MINOR: quic: Add traces about list of frames
This should be useful to have an idea of the list of frames which could be built
towards the list of available frames when building packets.
Same thing about the frames which could not be built because of a lack of room
in the TX buffer.
2022-04-01 16:26:06 +02:00
Frédéric Lécaille
b002145e9f MEDIUM: quic: Send ACK frames asap
Due to a erroneous interpretation of the RFC 9000 (quic-transport), ACKs frames
were always sent only after having received two ack-eliciting packets.
This could trigger useless retransmissions for tail packets on the peer side.
For now on, we send as soon as possible ACK frames as soon as we have ACK to send,
in the same packets as the ack-eliciting frame packets, and we also send ACK
frames after having received 2 ack-eliciting packets since the last time we sent
an ACK frame with other ack-eliciting frames.
2022-04-01 16:22:52 +02:00
Frédéric Lécaille
205e4f359e CLEANUP: quic: Remove all atomic operations on packet number spaces
As such variables are handled by the QUIC connection I/O handler which runs
always on the thread, there is no need to continue to use such atomic operations
2022-04-01 16:22:47 +02:00
Frédéric Lécaille
fc79006c92 CLEANUP: quic: Remove all atomic operations on quic_conn struct
As the QUIC connections are always handled by the same thread there is no need
anymore to continue to use atomic operations on such variables.
2022-04-01 16:22:44 +02:00
Amaury Denoyelle
d8e680cbaf MEDIUM: mux-quic: remove qcs tree node
The new qc_stream_desc type has a tree node for storage. Thus, we can
remove the node in the qcs structure.

When initializing a new stream, it is stored into the qcc streams_by_id
tree. When the MUX releases it, it will freed as soon as its buffer is
emptied. Before this, the quic-conn is responsible to store it inside
its own streams_by_id tree.
2022-03-30 16:26:59 +02:00
Amaury Denoyelle
7272cd76fc MEDIUM: quic: move transport fields from qcs to qc_conn_stream
Move the xprt-buf and ack related fields from qcs to the qc_stream_desc
structure. In exchange, qcs has a pointer to the low-level stream. For
each new qcs, a qc_stream_desc is automatically allocated.

This simplify the transport layer by removing qcs/mux manipulation
during ACK frame parsing. An additional check is done to not notify the
MUX on sending if the stream is already released : this case may now
happen on retransmission.

To complete this change, the quic_stream frame now references the
quic_stream instance instead of a qcs.
2022-03-30 16:19:48 +02:00
Amaury Denoyelle
5c3859c509 MINOR: quic: implement stream descriptor for transport layer
Currently, the mux qcs streams manage the Tx buffering, even after
sending it to the transport layer. Buffers are emptied when
acknowledgement are treated by the transport layer. This complicates the
MUX liberation and we may loose some data after the MUX free.

Change this paradigm by moving the buffering on the transport layer. For
this goal, a new type is implemented as low-level stream at the
transport layer, as a counterpart of qcs mux instances. This structure
is called qc_stream_desc. This will allow to free the qcs/qcc instances
without having to wait for acknowledge reception.

For the moment, the quic-conn is responsible to store the qc_stream_desc
in a new tree named streams_by_id. This will sligthly change in the next
commits to remove the qcs node which has a similar purpose :
qc_stream_desc instances will be shared between the qcc MUX and the
quic-conn.

This patch only introduces the new type definition and the function to
manipulate it. The following commit will bring the rearchitecture in the
qcs structure.
2022-03-30 16:16:07 +02:00
Amaury Denoyelle
cbc13b71c6 MINOR: mux-quic: define release app-ops
Define a new callback release inside qcc_app_ops. It is called when the
qcc MUX is freed via qc_release. This will allows to implement cleaning
on the app layer.
2022-03-30 16:12:18 +02:00
Amaury Denoyelle
dccbd733f0 MINOR: mux-quic: reorganize qcs free
Regroup some cleaning operations inside a new function qcs_free. This
can be used for all streams, both through qcs_destroy and with
uni-directional streams.
2022-03-30 16:12:18 +02:00
Amaury Denoyelle
50742294f5 MINOR: mux-quic: return qcs instance from qcc_get_qcs
Refactoring on qcc_get_qcs : return the qcs instance instead of the tree
node. This is useful to hide some eb64_entry macros for better
readability.
2022-03-30 16:12:18 +02:00
William Lallemand
30fcca18a5 MINOR: ssl/lua: CertCache.set() allows to update an SSL certificate file
The CertCache.set() function allows to update an SSL certificate file
stored in the memory of the HAProxy process. This function does the same
as "set ssl cert" + "commit ssl cert" over the CLI.

This could be used to update the crt and key, as well as the OCSP, the
SCTL, and the OSCP issuer.

The implementation does yield every 10 ckch instances, the same way the
"commit ssl cert" do.
2022-03-30 14:56:10 +02:00
William Lallemand
26654e7a59 MINOR: ssl: add "crt" in the cert_exts array
The cert_exts array does handle "crt" the default way, however
you might stil want to look for these extensions in the array.
2022-03-30 14:55:53 +02:00
William Lallemand
e60c7d6e59 MINOR: ssl: export ckch_inst_rebuild()
ckch_inst_rebuild() will be needed to regenerate the ckch instances from
the lua code, we need to export it.
2022-03-30 12:18:16 +02:00
William Lallemand
aaacc7e8ad MINOR: ssl: move the cert_exts and the CERT_TYPE enum
Move the cert_exts declaration and the CERT_TYPE enum in the .h in order
to reuse them in another file.
2022-03-30 12:18:16 +02:00
William Lallemand
3b5a3a6c03 MINOR: ssl: split the cert commit io handler
Extract the code that replace the ckch_store and its dependencies into
the ckch_store_replace() function.

This function must be used under the global ckch lock.
It frees everything related to the old ckch_store.
2022-03-30 12:18:16 +02:00
Willy Tarreau
2100b383ab MINOR: action: add a function to dump the list of actions for a ruleset
The new function dump_act_rules() now dumps the list of actions supported
by a ruleset. These actions are alphanumerically sorted first so that the
produced output is easy to compare.
2022-03-30 11:19:22 +02:00
Willy Tarreau
3ff476e9ef MINOR: tools: add strordered() to check whether strings are ordered
When trying to sort sets of strings, it's often needed to required to
compare 3 strings to see if the chosen one fits well between the two
others. That's what this function does, in addition to being able to
ignore extremities when they're NULL (typically for the first iteration
for example).
2022-03-30 10:02:56 +02:00
Willy Tarreau
29d799d591 MINOR: sample: list registered sample converter functions
Similar to the sample fetch keywords, let's also list the converter
keywords. They're much simpler since there's no compatibility matrix.
Instead the input and output types are listed. This is called by
dump_registered_keywords() for the "cnv" keywords class.
2022-03-29 18:01:37 +02:00
Willy Tarreau
f78813f74f MINOR: samples: add a function to list register sample fetch keywords
New function smp_dump_fetch_kw lists registered sample fetch keywords
with their compatibility matrix, mandatory and optional argument types,
and output types. It's called from dump_registered_keywords() with class
"smp".
2022-03-29 18:01:37 +02:00
Willy Tarreau
6ff7d1b9a5 MINOR: acl: add a function to dump the list of known ACL keywords
New function acl_dump_kwd() dumps the registered ACL keywords and their
sample-fetch equivalent to stdout. It's called by dump_registered_keywords()
for keyword class "acl".
2022-03-29 18:01:37 +02:00
Willy Tarreau
06d0e2e034 MINOR: cli: add a new keyword dump function
New function cli_list_keywords() scans the list of registered CLI keywords
and dumps them on stdout. It's now called from dump_registered_keywords()
for the class "cli".

Some keywords are valid for the master, they'll be suffixed with
"[MASTER]". Others are valid for the worker, they'll have "[WORKER]".
Those accessible only in expert mode will show "[EXPERT]" and the
experimental ones will show "[EXPERIM]".
2022-03-29 18:01:37 +02:00
Willy Tarreau
ca1acd6080 MINOR: config: add a function to dump all known config keywords
All registered config keywords that are valid in the config parser are
dumped to stdout organized like the regular sections (global, listen,
etc). Some keywords that are known to only be valid in frontends or
backends will be suffixed with [FE] or [BE].

All regularly registered "bind" and "server" keywords are also dumped,
one per "bind" or "server" line. Those depending on ssl are listed after
the "ssl" keyword. Doing so required to export the listener and server
keyword lists that were static.

The function is called from dump_registered_keywords() for keyword
class "cfg".
2022-03-29 18:01:32 +02:00
Willy Tarreau
76871a4f8c MINOR: management: add some basic keyword dump infrastructure
It's difficult from outside haproxy to detect the supported keywords
and syntax. Interestingly, many of our modern keywords are enumerated
since they're registered from constructors, so it's not very hard to
enumerate most of them.

This patch creates some basic infrastructure to support dumping existing
keywords from different classes on stdout. The format will differ depending
on the classes, but the idea is that the output could easily be passed to
a script that generates some simple syntax highlighting rules, completion
rules for editors, syntax checkers or config parsers.

The principle chosen here is that if "-dK" is passed on the command-line,
at the end of the parsing the registered keywords will be dumped for the
requested classes passed after "-dK". Special name "help" will show known
classes, while "all" will execute all of them. The reason for doing that
after the end of the config processor is that it will also enumerate
internally-generated keywords, Lua or even those loaded from external
code (e.g. if an add-on is loaded using LD_PRELOAD). A typical way to
call this with a valid config would be:

    ./haproxy -dKall -q -c -f /path/to/config

If there's no config available, feeding /dev/null will also do the job,
though it will not be able to detect dynamically created keywords, of
course.

This patch also updates the management doc.

For now nothing but the help is listed, various subsystems will follow
in subsequent patches.
2022-03-29 17:55:54 +02:00
Amaury Denoyelle
0c2d964280 REORG: quic: use a dedicated quic_loss.c
Move all inline functions with trace from quic_loss.h to a dedicated
object file. This let to remove the TRACE_SOURCE macro definition
outside of the include file.

This change is required to be able to define another TRACE_SOUCE inside
the mux_quic.c for a dedicated trace module.
2022-03-25 14:45:45 +01:00
Amaury Denoyelle
777969c163 BUILD: quic: add missing includes
Complete the include list for the files quic_loss.h and quic_sock.c.
2022-03-25 14:45:45 +01:00