To make haload work correctly with hq_interop (h0/QUIC), we need
to support its specific stream layout. This patch adds the inline helper
hq_interop_strm_bytes_in() which correctly extracts the <bytes_in> value
from either __sc_strm(sc) or __sc_hldstream(sc) depending on the object
type, preventing haload crashes when it uses hq_interop.
This patch introduces the haload target to the Makefile. It defines
the HALOAD_OBJS list, which includes the standard objects along with
the specific haload_init.o, haload.o, and hbuf.o files.
The build process follows the same pattern as the haterm tool,
allowing haload to be compiled as a standalone binary.
Export _srv_parse_kw() and srv_postinit() so they can be called from
haload (to come), which needs to configure servers using HAProxy's configuration
parser keywords.
This patch exports sc_new() by removing its static storage class and
adding its prototype to include/haproxy/stconn.h.
This is required to allow external modules, such as the upcoming haload
benchmarking tool, to allocate and initialize new stream connectors
from a stream endpoint descriptor (sedesc).
This patch introduces the sc_hastream() and __sc_hastream() inline
helpers to retrieve a haload stream context (struct hastream) from
a stream connector.
These functions allow the stconn layer to safely access haload-specific
stream data when the application type is OBJ_TYPE_HXLOAD.
This patch introduces the OBJ_TYPE_HXLOAD object type to represent
haload stream objects (struct hastream).
It also adds the associated inline helper functions objt_hastream()
and __objt_hastream() to allow safe casting and retrieval of
hastream contexts from a generic object pointer, following the
standard container_of pattern.
haload is a client-side HTTP benchmarking tool designed to manage
concurrent HTTP streams.
This patch defines the hldstream C structure, which serves as the
core object to represent a haload HTTP stream for all the HTTP protocol.
It will be used by the upcoming haload module to handle specialized
stream contexts.
haload is the successor to the h1load HTTP benchmarking tool.
This patch adds haload stream definitions as arguments for the TRACE API.
These will be used by the upcoming haload module, which will handle
hldstream struct objects instead of regular stream structs.
Introduce the new <no_listener_mode> global variable to define a new operating mode
for haproxy. This variable can be set to 1 to allow haproxy to start without
any listeners. Without such a setting, haproxy refuses to start without listener.
During the initialization cycle, setting this variable to 1 ensures that the
lack of configured listeners is no longer treated as a fatal error. This allows
programs based on haproxy source code to initialize the stack and use its
features even without a frontend. This will be the case for haload.
Add a new lightweight hbuf API to buffer formatted strings, similar to the
existing buffer API (struct buffer), extracting the code which already does this
in haterm_init.c. This is used by haterm to build its configuration in memory
(fileless mode). And this will be used by haload to do the same thing.
Update haterm to use this new API.
Note: hstream_str_buf_append() has been renamed to hbuf_str_append().
hpack_dht_insert() has three call sites for hpack_dht_defrag(). Two of
them (lines 293 and 306) correctly check for a NULL return and bail out
with -1. The third (line 353, data-space defrag path) assigns the return
value to dht and immediately dereferences it without a NULL check.
When pool_head_hpack_tbl is exhausted, hpack_dht_alloc() returns NULL,
hpack_dht_defrag() propagates it, and line 354 dereferences NULL+0x0a
(offsetof wrap), crashing the worker with SIGSEGV.
Add a NULL check consistent with the two other call sites.
This must be backported to all stable versions.
Reported-by: Tristan (@TristanInSec)
When 0RTT is enabled, a temporary buffer for early data is used. We read
from it first when the mux asks for data, and then we free it when it is
empty, but that is not right, because maybe we have more early data to
receive, and then we no longer have any buffer to store them, and that
will eventually end up with the connection closed in error.
To fix that, as long as we haven't received all the early data yet, just
reset the buffer, instead of freeing it.
This should fix github issue #3416
This should be backported up to 2.8.
QUIC flow control on bidirectional streams ensure that the peer cannot
emit more than what haproxy has allowed, which guarantees that buffering
is under controlled on the receiver side.
This limit is first announced on the transport parameter via
initial_max_stream_data_bidi_remote which is derived from configuration
value. QUIC MUX calculation for its streams is then directly based on
the transport parameter.
This mechanism works as expected on the frontend side, as in this case
all exchanges occur on remote streams opened by the opposite side.
However, this is not working as expected on the backend side, as in this
case transfers occur on streams opened locally by haproxy as the client.
Thus, configuration has no impact on backend side rxbuf which remains
set to a single buffer, causing important latency when retrieving large
objects.
This patch removes this limitation on the backend side by adjusting
quic_transport_params_init(). If <server> parameter is false, limitation
is set for initial_max_stream_data_bidi_local TP.
This must be backported up to 3.3.
The file specified by 'h1-case-adjust-file' directive is only loaded during
post-parsing. However when a relative path is used, the corresponding
absoulte path was not resolve during parsing. So the file could be loaded
relatively from the wrong location leading to a configuration error. It may
happen if several configuration files are used or if several
"default-config" are used. The last "default" location was always used.
To fix the issue, the absolute path of the file is now resolved when the
directive is parsed.
This patch should fix the issue #3415. It must be backported to all
versions.
with L7 retries are configured, when the max number of retries is reached
the error must be reported to the client. However, when it was an abort on a
reused connections, the client connection is silently closed. While it is
expected without L7 retries, to let the client retries on its own, it is
unexepcted with L7 retries.
So let's fix it by ignoring the SF_SRV_REUSED flag on the stream when a L7
retry fails. This way, a 502/425 will be reported to the client.
This patch should help to fix the issue #3414. It must be backported to all
supported versions.
A memset used for debug was left when "keep-query" option was added. Let's
remove it. This bug should be harmless but it consumes extra CPU for
nothing.
This patch should be backported as far as 3.2.
Whne the endpoint is shut (an applet or a mux), at least one of the SHW
flags must be set (SE_SHW_SILENT or SE_SHW_NORMAL). It is mandatory for
muxes to perform the shutdown. Otherwise, the shutdown could be ignored.
So let's add a BUG_ON() on it to be sure this never happen.
This patch documents the behavior where the internal HTTP client sets
the response status to 0 when an error is encountered by the stream
(SF_ERR_MASK).
This allows users to distinguish between an HTTP status code returned
by a remote server and an internal error generated by HAProxy (e.g.
connection timeout, connection refused, etc.).
With the httpclient it's difficult to get if the HTTP status code was
returned by the actual server or if it's the internal proxy that
generate the error.
This patch changes the behavior by setting the status to 0 when an error
is get by the stream.
There were already valid cases when the status was 0 on some error, so
that should not really change the error path in the scripts.
Refactor the Lua HTTP client to defer initialization. core.httpclient()
no longer initializes the internal HTTP client immediately. Instead,
initialization now occurs within hlua_httpclient_send() when a request
method (e.g., get, put, head) is invoked.
The HTTPClient class now serves as a factory for accessing methods, while
a new class, HTTPClientRequest, has been introduced to represent individual
requests and manage the HTTP client lifecycle.
This change allows multiple requests to be executed using a single
HTTP client instance:
local hc = core.httpclient()
local res1 = hc:get({url = "...", headers = ...})
local res2 = hc:post({url = "...", headers = ...})
local res3 = hc:put({url = "...", headers = ...})
This refactor maintains backward compatibility, as existing scripts that
instantiate a new core.httpclient() for every request will continue to
work as expected.
Move the lua httpclient code from hlua.c to http_client.c
The code is almost the same but the registering of the class which is
done in hlua_http_client_init_state(), from REGISTER_HLUA_STATE_INIT()
check_args() calls have been replaced by hlua_check_args().
hlua_httpclient_destroy_all() is exported so it can be called in hlua.c.
hlua_httpclient_table_to_hdrs() is made static.
hlua_pusherror() and check_args() are being exported.
check_args() is now a macro to hlua_check_args() so it's not confusing
when called outside hlua.c.
When a dynamic server is added with consistent hash balancing on the
backend, its lb_nodes elements are allocated and associated with a
calculated server key. This operation is performed in add server handler
via srv_alloc_lb(). By default, the server key is based on its ID.
However, automatic server ID is calculated later in add server handler,
which means the initial lb_nodes are not valid.
This could cause load balacing issue but in fact this is not directly
visible as the server key is recalculated when the dynamic server is
enabled via chash_queue_dequeue_srv() : all server lb_nodes are dequeued
and requeued with the now proper key.
Thus, "add server" handler must be corrected as it is buggy when
considering it alone. The simplest solution of the current patch is to
initialize server ID before srv_alloc_lb() is invoked. There is no issue
as handler runs under thread isolation so there is no risk of multiple
servers manipulating the same ID. Server insertion in proxy ID tree is
still performed at the end of the handler when all fallible operation
are completed.
The fact that server key is recalculated when the server is set to ready
state is a side effect of the following patch which was introduced in
3.0. What this means though is that users of older releases are facing a
bigger issue, with load-balancing not working as expected. Thus,
this patch is even more crucial for 2.8 and older releases.
faa8c3e024
MEDIUM: lb-chash: Deterministic node hashes based on server address
This should fix github issue #3413. Thanks to Joao Morais for is
analysis on the problem.
This must be backported to all stable releases.
When we receive a PUSH_PROMISE frame while we don't expect it, flag it
as a connection error, do not just set ret to H3_ERR_ID_ERROR, as it
would just be considered the number of bytes we read, and could lead to
random corruption. This should only happen with backend connections.
This should be backported whenever commit 4a8bb2fe5 is backported.
Now that there is no longer a shared wake queue, chances are if a shared task
is scheduled, it will always end up on the same thread. In
wake_expired_tasks(), when a task has to be waken up, randomly look to
three other threads, and if the runqueue of the current thread is at least
two time bigger than the runqueue of one of the other threads, then give
that task to that thread, so that our load gets reduced.
If we're giving the task to another thread, then we have to add the
TASK_RUNNING flag until we waked it up, otherwise the other thread could
just run it, if it gets waken up from another path, and free it while
we're still not done with it.
2 times has been chosen somewhat arbitrarily, and may be tweaked at a
later date if deemed not optimal.
Modify task_instant_wakeup() to use __task_set_state_and_tid().
It uses the new ownership behavior, but that's okay because
task_instant_wakeup() was not used anywhere.
Totally remove the per-thread group wait queue. This was potentially a
source of contention, because there were only a global lock for all
those wait queues.
Instead, for shared tasks, there is now the concept of ownership for the
task. When a task is in the wait queue, run queue, or is running on that
particular thread, the task's tid is set to -2 - thread_tid, and only
that thread will be responsible for it until it is no longer running,
and in none of its queue.
When a shared task is scheduled to be run at a later time, if its
current tid is -1, then the current thread will take ownership, and put
it in its own wait queue. If it is already owned, then TASK_WOKEN_WQ is
added to the task's state, and a task_wakeup() is done, so that the
owner thread will add it in its wait queue.
If there is any owner, then a task_wakeup() will just add the task to
the owner's runqueue, otherwise the current thread will become the
owner.
Introduce a new function, __task_get_current_owner, that returns the
owner of a task based on its current tid.
-1 means there is no current owner, otherwise either the tid is >= 0, in
which case it will just return it, or it's < -1, in which case it will
return -2 - tid, the tid of the thread with the current ownership.
Introduce __task_get_new_tid_field(), that provides the tid to be used
for a task.
For shared task, to mark temporary ownership of a task, instead of -1,
the tid will be set to -2-tid, tid being the tid of the current thread.
Introduce a new function, __task_set_state_and_tid, that atomically can
set a task's state and its tid. This will be used later, as the tid will
be used to indicate task ownership even for shared tasks.
This Lua script automates dns-01 ACME challenges using the Gandi LiveDNS
API v5. It subscribes to the ACME_DEPLOY event to set the required
_acme-challenge TXT record via the Gandi REST API, signals HAProxy that
the challenge is ready using ACME.challenge_ready(), then cleans up the
TXT record once the certificate is issued on ACME_NEWCERT.
The API key is read from the GANDI_API_KEY environment variable at
startup. Zone discovery is automatic: the script probes parent zones
from longest to shortest until Gandi accepts the record, which handles
both apex and wildcard certificates transparently.
Add EVENT_HDL_SUB_ACME_DEPLOY to the ACME family. It is published in
the dns-01 challenge path after the TXT record information has been
prepared, carrying the certificate store name, domain, account
thumbprint, dns_record value, and optionally the provider and vars
strings.
Lua subscribers using core.event_sub() receive the event data as an
AcmeEvent object, which is the same class used for ACME_NEWCERT and
carries the fields relevant to the event type.
Add a new EVENT_HDL_SUB_ACME_NEWCERT event type in the ACME family.
It is published after a new certificate has been successfully fetched
and installed. The event carries the certificate store name, allowing
subscribers to act on newly available certificates.
Lua subscribers using core.event_sub() receive the event data as an
AcmeEvent object with a crtname field containing the certificate store
name.
Using ha_diag_warning() to report the number of threads created resulted
in warnings being counted and possibly an error being fired when combined
with -dW:
$ printf "global\nstats socket /tmp/sock1\n" | ./haproxy -dD -dW -c -f /dev/stdin; echo $?
[NOTICE] (10406) : haproxy version is 3.5-dev0-5091ac-35
[NOTICE] (10406) : path to executable is ./haproxy
[DIAG] (10406) : Created 20 threads split into 2 groups
[ALERT] (10406) : Some warnings were found and 'zero-warning' is set. Aborting.
1
Now that we have ha_diag_notice(), let's use it:
$ printf "global\nstats socket /tmp/sock1\n" | ./haproxy -dD -dW -c -f /dev/stdin; echo $?
[DIAG] (10513) : Created 20 threads split into 2 groups
0
It would make sense to backport this to 3.2 because it helps validate configs
against diag warnings without triggering a false positive. It depends on
this previous patch:
MINOR: errors: add ha_diag_notice() to report diag-level notifications
Right now the only way to report info that is only displayed in diag
mode with -dD is to use ha_diag_warning(). The problem is that this is
then counted as a warning and may result in errors when combined with
-dW, as happens for the CPU topology info:
$ printf "global\nstats socket /tmp/sock1\n" | ./haproxy -dD -dW -c -f /dev/stdin; echo $?
[NOTICE] (10406) : haproxy version is 3.5-dev0-5091ac-35
[NOTICE] (10406) : path to executable is ./haproxy
[DIAG] (10406) : Created 20 threads split into 2 groups
[ALERT] (10406) : Some warnings were found and 'zero-warning' is set. Aborting.
1
We need another level. This commit introduces ha_diag_notice() which only
emits a notification that doesn't count as a warning. Note that we could
even introduce an info level and revisit various messages so that notice
only reports certain events while info is for anything (like versions
above). That could be a future improvement.
The Linux tls module requires a socket to be in TCP_ESTABLISHED state
before we can enable the TLS ULP on the socket, if the socket is in any
other state, then the setsockopt() call will fail, and we won't use
kTLS on that socket.
To make sure we're not doing it too early, defer it until the TLS
handshake is done, which means the TCP connection is established.
This should be backported up to 3.3.
Signed-off-by: Karol Kucharski <kkucharski@fastlogic.pl>
Add a new ACME global Lua table with a challenge_ready(crt, dns) method
that wraps acme_challenge_ready(). It marks the ACME challenge for domain
<dns> in certificate <crt> as ready and returns the number of remaining
challenges, or 0 when all challenges are ready and validation has been
triggered. A Lua error is raised if the certificate or domain is not found.
The ACME table is registered for each lua_State via the new
REGISTER_HLUA_STATE_INIT() mechanism.
__LJMP, WILL_LJMP() and MAY_LJMP() were defined locally in hlua.c,
making them unavailable to other modules that implement Lua bindings.
Move them to include/haproxy/hlua.h so they can be used outside of
hlua.c.
Add a registration mechanism so that modules outside of hlua.c can hook
into each lua_State creation. Modules call hap_register_hlua_state_init()
(or the REGISTER_HLUA_STATE_INIT() macro) with a callback of the form:
int my_init(lua_State *L, char **errmsg);
The callback returns an ERR_* code. ERR_ALERT and ERR_WARN trigger
ha_alert()/ha_warning() respectively; any other non-zero errmsg is
emitted via ha_notice(). ERR_FATAL or ERR_ABORT cause exit(1).
Registered entries are freed in hlua_deinit().
Output HTTP/3 header traces on the backend side. As previous commit,
this relies on h3_trace_header() function.
Extra calls are added for fields extracted from the request start-line
which produce an HTTP/3 pseudo-header.
