upstream/haproxy
1
0
Fork 0
mirror of https://github.com/haproxy/haproxy.git synced 2026-04-07 10:08:38 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 8
haproxy/include/haproxy/thread.h

566 lines
22 KiB
C
Raw Normal View History

MEDIUM: threads: Add hathreads header file This file contains all functions and macros used to deal with concurrency in HAProxy. It contains all high-level function to do atomic operation (HA_ATOMIC_*). Note, for now, we rely on "__atomic" GCC builtins to do atomic operation. So HAProxy can be compiled with the thread support iff these builtins are available. It also contains wrappers around plocks to use spin or read/write locks. These wrappers are used to abstract the internal representation of the locking system and to add information to help debugging, when compiled with suitable options. To add extra info on locks, you need to add DEBUG=-DDEBUG_THREAD or DEBUG=-DDEBUG_FULL compilation option. In addition to timing info on locks, we keep info on where a lock was acquired the last time (function name, file and line). There are also the thread id and a flag to know if it is still locked or not. This will be useful to debug deadlocks.
2017-10-12 10:09:09 -04:00
/*
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
* include/haproxy/thread.h
* definitions, macros and inline functions used by threads.
MEDIUM: threads: Add hathreads header file This file contains all functions and macros used to deal with concurrency in HAProxy. It contains all high-level function to do atomic operation (HA_ATOMIC_*). Note, for now, we rely on "__atomic" GCC builtins to do atomic operation. So HAProxy can be compiled with the thread support iff these builtins are available. It also contains wrappers around plocks to use spin or read/write locks. These wrappers are used to abstract the internal representation of the locking system and to add information to help debugging, when compiled with suitable options. To add extra info on locks, you need to add DEBUG=-DDEBUG_THREAD or DEBUG=-DDEBUG_FULL compilation option. In addition to timing info on locks, we keep info on where a lock was acquired the last time (function name, file and line). There are also the thread id and a flag to know if it is still locked or not. This will be useful to debug deadlocks.
2017-10-12 10:09:09 -04:00
*
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
* Copyright (C) 2017 Christopher Faulet - cfaulet@haproxy.com
* Copyright (C) 2020 Willy Tarreau - w@1wt.eu
MEDIUM: threads: Add hathreads header file This file contains all functions and macros used to deal with concurrency in HAProxy. It contains all high-level function to do atomic operation (HA_ATOMIC_*). Note, for now, we rely on "__atomic" GCC builtins to do atomic operation. So HAProxy can be compiled with the thread support iff these builtins are available. It also contains wrappers around plocks to use spin or read/write locks. These wrappers are used to abstract the internal representation of the locking system and to add information to help debugging, when compiled with suitable options. To add extra info on locks, you need to add DEBUG=-DDEBUG_THREAD or DEBUG=-DDEBUG_FULL compilation option. In addition to timing info on locks, we keep info on where a lock was acquired the last time (function name, file and line). There are also the thread id and a flag to know if it is still locked or not. This will be useful to debug deadlocks.
2017-10-12 10:09:09 -04:00
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation, version 2.1
* exclusively.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public
* License along with this library; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
*/
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
#ifndef _HAPROXY_THREAD_H
#define _HAPROXY_THREAD_H
MEDIUM: threads: Add hathreads header file This file contains all functions and macros used to deal with concurrency in HAProxy. It contains all high-level function to do atomic operation (HA_ATOMIC_*). Note, for now, we rely on "__atomic" GCC builtins to do atomic operation. So HAProxy can be compiled with the thread support iff these builtins are available. It also contains wrappers around plocks to use spin or read/write locks. These wrappers are used to abstract the internal representation of the locking system and to add information to help debugging, when compiled with suitable options. To add extra info on locks, you need to add DEBUG=-DDEBUG_THREAD or DEBUG=-DDEBUG_FULL compilation option. In addition to timing info on locks, we keep info on where a lock was acquired the last time (function name, file and line). There are also the thread id and a flag to know if it is still locked or not. This will be useful to debug deadlocks.
2017-10-12 10:09:09 -04:00
REORG: include: update all files to use haproxy/api.h or api-t.h if needed All files that were including one of the following include files have been updated to only include haproxy/api.h or haproxy/api-t.h once instead: - common/config.h - common/compat.h - common/compiler.h - common/defaults.h - common/initcall.h - common/tools.h The choice is simple: if the file only requires type definitions, it includes api-t.h, otherwise it includes the full api.h. In addition, in these files, explicit includes for inttypes.h and limits.h were dropped since these are now covered by api.h and api-t.h. No other change was performed, given that this patch is large and affects 201 files. At least one (tools.h) was already freestanding and didn't get the new one added.
2020-05-27 06:58:42 -04:00
#include <haproxy/api.h>
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
#include <haproxy/thread-t.h>
REORG: includes: create tinfo.h for the thread_info struct The thread_info struct is convenient to store various per-thread info without having to resort to a painful thread_local storage which is slow and painful to initialize. The problem is, by having this one in thread.h it's very difficult to add more entries there because everyone already includes thread.h so conversely thread.h cannot reference certain types. There's no point in having this there, instead let's create a new pair of files, tinfo{,-t}.h, which declare the structure. This way it will become possible to extend them with other includes and have certain files store their own types there.
2020-06-29 03:57:23 -04:00
#include <haproxy/tinfo.h>
MINOR: thread: implement ha_thread_relax() At some places we're using a painful ifdef to decide whether to use sched_yield() or pl_cpu_relax() to relax in loops, this is hardly exportable. Let's move this to ha_thread_relax() instead and une this one only.
2019-05-17 10:33:13 -04:00
MEDIUM: threads: Add hathreads header file This file contains all functions and macros used to deal with concurrency in HAProxy. It contains all high-level function to do atomic operation (HA_ATOMIC_*). Note, for now, we rely on "__atomic" GCC builtins to do atomic operation. So HAProxy can be compiled with the thread support iff these builtins are available. It also contains wrappers around plocks to use spin or read/write locks. These wrappers are used to abstract the internal representation of the locking system and to add information to help debugging, when compiled with suitable options. To add extra info on locks, you need to add DEBUG=-DDEBUG_THREAD or DEBUG=-DDEBUG_FULL compilation option. In addition to timing info on locks, we keep info on where a lock was acquired the last time (function name, file and line). There are also the thread id and a flag to know if it is still locked or not. This will be useful to debug deadlocks.
2017-10-12 10:09:09 -04:00
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
/* Note: this file mainly contains 5 sections:
* - a small common part, which also corresponds to the common API
* - one used solely when USE_THREAD is *not* set
* - one used solely when USE_THREAD is set
* - one used solely when USE_THREAD is set WITHOUT debugging
* - one used solely when USE_THREAD is set WITH debugging
*
MINOR: threads: move "nbthread" parsing to hathreads.c The purpose is to make sure that all variables which directly depend on this nbthread argument are set at the right moment. For now only all_threads_mask needs to be set. It used to be set while calling thread_sync_init() which is called too late for certain checks. The same function handles threads and non-threads, which removes the need for some thread-specific knowledge from cfgparse.c.
2018-07-30 04:34:35 -04:00
*/
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
/* Generic exports */
int parse_nbthread(const char *arg, char **err);
CLEANUP: thread: uninline ha_tkill/ha_tkillall/ha_cpu_relax() These ones are rarely used or only to waste CPU cycles waiting, and are the last ones requiring system includes in thread.h. Let's uninline them and move them to thread.c.
2021-10-06 17:33:20 -04:00
void ha_tkill(unsigned int thr, int sig);
void ha_tkillall(int sig);
void ha_thread_relax(void);
MEDIUM: threads: detect incomplete CPU bindings It's very easy to mess up with some cpu-map directives and to leave some thread unbound. Let's add a test that checks that either all threads are bound or none are bound, but that we do not face the intermediary situation where some are pinned and others are left wandering around, possibly on the same CPUs as bound ones. Note that this should not be backported, or maybe turned into a notice only, as it appears that it will easily catch invalid configs and that may break updates for some users.
2023-09-04 10:53:20 -04:00
int thread_detect_binding_discrepancies(void);
MEDIUM: threads: detect excessive thread counts vs cpu-map This detects when there are more threads bound via cpu-map than CPUs enabled in cpu-map, or when there are more total threads than the total number of CPUs available at boot (for unbound threads) and configured for bound threads. In this case, a warning is emitted to explain the problems it will cause, and explaining how to address the situation. Note that some configurations will not be detected as faulty because the algorithmic complexity to resolve all arrangements grows in O(N!). This means that having 3 threads on 2 CPUs and one thread on 2 CPUs will not be detected as it's 4 threads for 4 CPUs. But at least configs such as T0:(1,4) T1:(1,4) T2:(2,4) T3:(3,4) will not trigger a warning since they're valid.
2023-09-04 11:36:20 -04:00
int thread_detect_more_than_cpus(void);
MEDIUM: threads: automatically assign threads to groups This takes care of unassigned threads groups and places unassigned threads there, in a more or less balanced way. Too sparse allocations may still fail though. For now with a maximum group number fixed to 1 nothing can really fail.
2021-09-27 04:10:26 -04:00
int thread_map_to_groups();
CLEANUP: listener/thread: remove now unused bind_conf's bind_tgroup/bind_thread Not needed anymore since last commit, let's get rid of it.
2023-02-02 11:01:10 -05:00
int thread_resolve_group_mask(struct thread_set *ts, int defgrp, char **err);
MEDIUM: thread: start to detect thread groups and threads min/max By mutually refining the thread count and group count, we can try to detect the most suitable setup for the current machine. Taskset is implicitly handled correctly. tgroups automatically adapt to the configured number of threads. cpu-map manages to limit tgroups to the smallest supported value. The thread-limit is enforced. Just like in cfgparse, if the thread count was forced to a higher value, it's reduced and a warning is emitted. But if it was not set, the thr_max value is bound to this limit so that further calculations respect it. We continue to default to the max number of available threads and 1 tgroup by default, with the limit. This normally allows to get rid of that test in check_config_validity().
2023-07-18 08:04:10 -04:00
void thread_detect_count(void);
MINOR: thread: add a simple thread_set API The purpose is to be able to store large thread sets, defined by ranges that may cross group boundaries, as well as define lists of groups and masks. The thread_set struct implements the storage, and the parser is in parse_thread_set(), with a focus on "bind" lines, but not only.
2023-01-31 13:27:48 -05:00
int parse_thread_set(const char *arg, struct thread_set *ts, char **err);
MINOR: threads/cli: display the lock history on "show threads" This will display the lock labels and modes for each non-empty step at the end of "show threads" when these are defined. This allows to emit up to the last 8 locking operation for each thread on 64 bit machines.
2025-04-28 09:19:35 -04:00
const char *lock_label(enum lock_label label);
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
extern int thread_cpus_enabled_at_boot;
MINOR: threads: add a "stuck" flag to the thread_info struct This flag is constantly cleared by the scheduler and will be set by the watchdog timer to detect stuck threads. It is also set by the "show threads" command so that it is easy to spot if the situation has evolved between two subsequent calls : if the first "show threads" shows no stuck thread and the second one shows such a stuck thread, it indicates that this thread didn't manage to make any forward progress since the previous call, which is extremely suspicious.
2019-05-22 01:06:44 -04:00
MEDIUM: threads: Add hathreads header file This file contains all functions and macros used to deal with concurrency in HAProxy. It contains all high-level function to do atomic operation (HA_ATOMIC_*). Note, for now, we rely on "__atomic" GCC builtins to do atomic operation. So HAProxy can be compiled with the thread support iff these builtins are available. It also contains wrappers around plocks to use spin or read/write locks. These wrappers are used to abstract the internal representation of the locking system and to add information to help debugging, when compiled with suitable options. To add extra info on locks, you need to add DEBUG=-DDEBUG_THREAD or DEBUG=-DDEBUG_FULL compilation option. In addition to timing info on locks, we keep info on where a lock was acquired the last time (function name, file and line). There are also the thread id and a flag to know if it is still locked or not. This will be useful to debug deadlocks.
2017-10-12 10:09:09 -04:00
#ifndef USE_THREAD
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
/********************** THREADS DISABLED ************************/
MINOR: threads: add more consistency between certain variables in no-thread case When threads are disabled, some variables such as tid and tid_bit are still checked everywhere, the MAX_THREADS_MASK macro is ~0UL while MAX_THREADS is 1, and the all_threads_mask variable is replaced with a macro forced to zero. The compiler cannot optimize away all this code involving checks on tid and tid_bit, and we end up in special cases where all_threads_mask has to be specifically tested for being zero or not. It is not even certain the code paths are always equivalent when testing without threads and with nbthread 1. Let's change this to make sure we always present a single thread when threads are disabled, and have the relevant values declared as constants so that the compiler can optimize all the tests away. Now we have MAX_THREADS_MASK set to 1, all_threads_mask set to 1, tid set to zero and tid_bit set to 1. Doing just this has removed 4 kB of code in the no-thread case. A few checks for all_threads_mask==0 have been removed since it never happens anymore.
2018-08-01 13:12:20 -04:00
/* Only way found to replace variables with constants that are optimized away
* at build time.
*/
enum { tid_bit = 1UL };
enum { tid = 0 };
MINOR: threads: add the current group ID in thread-local "tgid" variable This is the equivalent of "tid" for ease of access. In the future if we make th_cfg a pure thread-local array (not a pointer), it may make sense to move it there.
2021-09-30 02:00:11 -04:00
enum { tgid = 1 };
MINOR: threads: add a MAX_THREADS define instead of LONGBITS This one allows not to inflate some structures when threads are disabled. Now struct global is 1.4 kB instead of 33 kB. Should be backported to 1.8 for ease of backporting of upcoming patches.
2018-01-20 12:19:22 -05:00
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
#define HA_SPIN_INIT(l) do { /* do nothing */ } while(0)
#define HA_SPIN_DESTROY(l) do { /* do nothing */ } while(0)
#define HA_SPIN_LOCK(lbl, l) do { /* do nothing */ } while(0)
#define HA_SPIN_TRYLOCK(lbl, l) ({ 0; })
#define HA_SPIN_UNLOCK(lbl, l) do { /* do nothing */ } while(0)
MINOR: threads: add a thread-local thread_info pointer "ti" Since we're likely to access this thread_info struct more frequently in the future, let's reserve the thread-local symbol to access it directly and avoid always having to combine thread_info and tid. This pointer is set when tid is set.
2019-05-20 12:57:53 -04:00
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
#define HA_RWLOCK_INIT(l) do { /* do nothing */ } while(0)
#define HA_RWLOCK_DESTROY(l) do { /* do nothing */ } while(0)
#define HA_RWLOCK_WRLOCK(lbl, l) do { /* do nothing */ } while(0)
BUILD: threads: Rename SPIN/RWLOCK macros using HA_ prefix This remove any name conflicts, especially on Solaris.
2017-11-07 04:42:54 -05:00
#define HA_RWLOCK_TRYWRLOCK(lbl, l) ({ 0; })
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
#define HA_RWLOCK_WRUNLOCK(lbl, l) do { /* do nothing */ } while(0)
#define HA_RWLOCK_RDLOCK(lbl, l) do { /* do nothing */ } while(0)
BUILD: threads: Rename SPIN/RWLOCK macros using HA_ prefix This remove any name conflicts, especially on Solaris.
2017-11-07 04:42:54 -05:00
#define HA_RWLOCK_TRYRDLOCK(lbl, l) ({ 0; })
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
#define HA_RWLOCK_RDUNLOCK(lbl, l) do { /* do nothing */ } while(0)
MEDIUM: threads: Add hathreads header file This file contains all functions and macros used to deal with concurrency in HAProxy. It contains all high-level function to do atomic operation (HA_ATOMIC_*). Note, for now, we rely on "__atomic" GCC builtins to do atomic operation. So HAProxy can be compiled with the thread support iff these builtins are available. It also contains wrappers around plocks to use spin or read/write locks. These wrappers are used to abstract the internal representation of the locking system and to add information to help debugging, when compiled with suitable options. To add extra info on locks, you need to add DEBUG=-DDEBUG_THREAD or DEBUG=-DDEBUG_FULL compilation option. In addition to timing info on locks, we keep info on where a lock was acquired the last time (function name, file and line). There are also the thread id and a flag to know if it is still locked or not. This will be useful to debug deadlocks.
2017-10-12 10:09:09 -04:00
MINOR: threads: add the transitions to/from the seek state Since our locks are based on progressive locks, we support the upgradable seek lock that is compatible with readers and upgradable to a write lock. The main purpose is to take it while seeking down a tree for modification while other threads may seek the same tree for an input (e.g. compute the next event date). The newly supported operations are: HA_RWLOCK_SKLOCK(lbl,l) pl_take_s(l) /* N --> S */ HA_RWLOCK_SKTOWR(lbl,l) pl_stow(l) /* S --> W */ HA_RWLOCK_WRTOSK(lbl,l) pl_wtos(l) /* W --> S */ HA_RWLOCK_SKTORD(lbl,l) pl_stor(l) /* S --> R */ HA_RWLOCK_WRTORD(lbl,l) pl_wtor(l) /* W --> R */ HA_RWLOCK_SKUNLOCK(lbl,l) pl_drop_s(l) /* S --> N */ HA_RWLOCK_TRYSKLOCK(lbl,l) (!pl_try_s(l)) /* N -?> S */ HA_RWLOCK_TRYRDTOSK(lbl,l) (!pl_try_rtos(l)) /* R -?> S */ Existing code paths are left unaffected so this patch doesn't affect any running code.
2020-10-16 10:53:46 -04:00
#define HA_RWLOCK_SKLOCK(lbl,l) do { /* do nothing */ } while(0)
#define HA_RWLOCK_SKTOWR(lbl,l) do { /* do nothing */ } while(0)
#define HA_RWLOCK_WRTOSK(lbl,l) do { /* do nothing */ } while(0)
#define HA_RWLOCK_SKTORD(lbl,l) do { /* do nothing */ } while(0)
#define HA_RWLOCK_WRTORD(lbl,l) do { /* do nothing */ } while(0)
#define HA_RWLOCK_SKUNLOCK(lbl,l) do { /* do nothing */ } while(0)
#define HA_RWLOCK_TRYSKLOCK(lbl,l) ({ 0; })
#define HA_RWLOCK_TRYRDTOSK(lbl,l) ({ 0; })
MINOR: threads: Add HA_RWLOCK_TRYRDTOWR() Add HA_RWLOCK_TRYRDTOWR(), that tries to upgrade a lock from reader to writer, and fails if any seeker or writer already holds it.
2025-03-26 08:11:30 -04:00
#define HA_RWLOCK_TRYRDTOWR(lbl,l) ({ 0; })
MINOR: threads: add the transitions to/from the seek state Since our locks are based on progressive locks, we support the upgradable seek lock that is compatible with readers and upgradable to a write lock. The main purpose is to take it while seeking down a tree for modification while other threads may seek the same tree for an input (e.g. compute the next event date). The newly supported operations are: HA_RWLOCK_SKLOCK(lbl,l) pl_take_s(l) /* N --> S */ HA_RWLOCK_SKTOWR(lbl,l) pl_stow(l) /* S --> W */ HA_RWLOCK_WRTOSK(lbl,l) pl_wtos(l) /* W --> S */ HA_RWLOCK_SKTORD(lbl,l) pl_stor(l) /* S --> R */ HA_RWLOCK_WRTORD(lbl,l) pl_wtor(l) /* W --> R */ HA_RWLOCK_SKUNLOCK(lbl,l) pl_drop_s(l) /* S --> N */ HA_RWLOCK_TRYSKLOCK(lbl,l) (!pl_try_s(l)) /* N -?> S */ HA_RWLOCK_TRYRDTOSK(lbl,l) (!pl_try_rtos(l)) /* R -?> S */ Existing code paths are left unaffected so this patch doesn't affect any running code.
2020-10-16 10:53:46 -04:00
BUG/MINOR: signals: ha_sigmask macro for multithreading The behavior of sigprocmask in an multithreaded environment is undefined. The new macro ha_sigmask() calls either pthreads_sigmask() or sigprocmask() if haproxy was built with thread support or not. This should be backported to 1.8.
2018-06-07 05:23:40 -04:00
#define ha_sigmask(how, set, oldset) sigprocmask(how, set, oldset)
MEDIUM: threads: replace ha_set_tid() with ha_set_thread() ha_set_tid() was randomly used either to explicitly set thread 0 or to set any possibly incomplete thread during boot. Let's replace it with a pointer to a valid thread or NULL for any thread. This allows us to check that the designated threads are always valid, and to ignore the thread 0's mapping when setting it to NULL, and always use group 0 with it during boot. The initialization code is also cleaner, as we don't pass ugly casts of a thread ID to a pointer anymore.
2021-09-28 03:43:11 -04:00
/* Sets the current thread to a valid one described by <thr>, or to any thread
* and any group if NULL (e.g. for use during boot where they're not totally
* initialized).
*/
static inline void ha_set_thread(const struct thread_info *thr)
MINOR: threads: add more consistency between certain variables in no-thread case When threads are disabled, some variables such as tid and tid_bit are still checked everywhere, the MAX_THREADS_MASK macro is ~0UL while MAX_THREADS is 1, and the all_threads_mask variable is replaced with a macro forced to zero. The compiler cannot optimize away all this code involving checks on tid and tid_bit, and we end up in special cases where all_threads_mask has to be specifically tested for being zero or not. It is not even certain the code paths are always equivalent when testing without threads and with nbthread 1. Let's change this to make sure we always present a single thread when threads are disabled, and have the relevant values declared as constants so that the compiler can optimize all the tests away. Now we have MAX_THREADS_MASK set to 1, all_threads_mask set to 1, tid set to zero and tid_bit set to 1. Doing just this has removed 4 kB of code in the no-thread case. A few checks for all_threads_mask==0 have been removed since it never happens anymore.
2018-08-01 13:12:20 -04:00
{
MEDIUM: threads: replace ha_set_tid() with ha_set_thread() ha_set_tid() was randomly used either to explicitly set thread 0 or to set any possibly incomplete thread during boot. Let's replace it with a pointer to a valid thread or NULL for any thread. This allows us to check that the designated threads are always valid, and to ignore the thread 0's mapping when setting it to NULL, and always use group 0 with it during boot. The initialization code is also cleaner, as we don't pass ugly casts of a thread ID to a pointer anymore.
2021-09-28 03:43:11 -04:00
if (thr) {
ti = thr;
tg = ti->tg;
th_ctx = &ha_thread_ctx[ti->tid];
} else {
ti = &ha_thread_info[0];
tg = &ha_tgroup_info[0];
th_ctx = &ha_thread_ctx[0];
}
MINOR: threads: add more consistency between certain variables in no-thread case When threads are disabled, some variables such as tid and tid_bit are still checked everywhere, the MAX_THREADS_MASK macro is ~0UL while MAX_THREADS is 1, and the all_threads_mask variable is replaced with a macro forced to zero. The compiler cannot optimize away all this code involving checks on tid and tid_bit, and we end up in special cases where all_threads_mask has to be specifically tested for being zero or not. It is not even certain the code paths are always equivalent when testing without threads and with nbthread 1. Let's change this to make sure we always present a single thread when threads are disabled, and have the relevant values declared as constants so that the compiler can optimize all the tests away. Now we have MAX_THREADS_MASK set to 1, all_threads_mask set to 1, tid set to zero and tid_bit set to 1. Doing just this has removed 4 kB of code in the no-thread case. A few checks for all_threads_mask==0 have been removed since it never happens anymore.
2018-08-01 13:12:20 -04:00
}
BUG/MINOR: signals: ha_sigmask macro for multithreading The behavior of sigprocmask in an multithreaded environment is undefined. The new macro ha_sigmask() calls either pthreads_sigmask() or sigprocmask() if haproxy was built with thread support or not. This should be backported to 1.8.
2018-06-07 05:23:40 -04:00
MEDIUM: threads: add a stronger thread_isolate_full() call The current principle of running under isolation was made to access sensitive data while being certain that no other thread was using them in parallel, without necessarily having to place locks everywhere. The main use case are "show sess" and "show fd" which run over long chains of pointers. The thread_isolate() call relies on the "harmless" bit that indicates for a given thread that it's not currently doing such sensitive things, which is advertised using thread_harmless_now() and which ends usings thread_harmless_end(), which also waits for possibly concurrent threads to complete their work if they took this opportunity for starting something tricky. As some system calls were notoriously slow (e.g. mmap()), a bunch of thread_harmless_now() / thread_harmless_end() were placed around them to let waiting threads do their work while such other threads were not able to modify memory contents. But this is not sufficient for performing memory modifications. One such example is the server deletion code. By modifying memory, it not only requires that other threads are not playing with it, but are not either in the process of touching it. The fact that a pool_alloc() or pool_free() on some structure may call thread_harmless_now() and let another thread start to release the same object's memory is not acceptable. This patch introduces the concept of "idle threads". Threads entering the polling loop are idle, as well as those that are waiting for all others to become idle via the new function thread_isolate_full(). Once thread_isolate_full() is granted, the thread is not idle anymore, and it is released using thread_release() just like regular isolation. Its users have to keep in mind that across this call nothing is granted as another thread might have performed shared memory modifications. But such users are extremely rare and are actually expecting this from their peers as well. Note that that in case of backport, this patch depends on previous patch: MINOR: threads: make thread_release() not wait for other ones to complete
2021-08-04 05:44:17 -04:00
static inline void thread_idle_now()
{
BUG/MINOR: threads: set threads_idle and threads_harmless even with no threads Some signal handlers rely on these to decide about the level of detail to provide in dumps, so let's properly fill the info about entering/leaving idle. Note that for consistency with other tests we're using bitops with t->ltid_bit, while we could simply assign 0/1 to the fields. But it makes the code more readable and the whole difference is only 88 bytes on a 3MB executable. This bug is not important, and while older versions are likely affected as well, it's not worth taking the risk to backport this in case it would wake up an obscure bug.
2025-04-15 03:03:35 -04:00
tg_ctx->threads_idle |= ti->ltid_bit;
MEDIUM: threads: add a stronger thread_isolate_full() call The current principle of running under isolation was made to access sensitive data while being certain that no other thread was using them in parallel, without necessarily having to place locks everywhere. The main use case are "show sess" and "show fd" which run over long chains of pointers. The thread_isolate() call relies on the "harmless" bit that indicates for a given thread that it's not currently doing such sensitive things, which is advertised using thread_harmless_now() and which ends usings thread_harmless_end(), which also waits for possibly concurrent threads to complete their work if they took this opportunity for starting something tricky. As some system calls were notoriously slow (e.g. mmap()), a bunch of thread_harmless_now() / thread_harmless_end() were placed around them to let waiting threads do their work while such other threads were not able to modify memory contents. But this is not sufficient for performing memory modifications. One such example is the server deletion code. By modifying memory, it not only requires that other threads are not playing with it, but are not either in the process of touching it. The fact that a pool_alloc() or pool_free() on some structure may call thread_harmless_now() and let another thread start to release the same object's memory is not acceptable. This patch introduces the concept of "idle threads". Threads entering the polling loop are idle, as well as those that are waiting for all others to become idle via the new function thread_isolate_full(). Once thread_isolate_full() is granted, the thread is not idle anymore, and it is released using thread_release() just like regular isolation. Its users have to keep in mind that across this call nothing is granted as another thread might have performed shared memory modifications. But such users are extremely rare and are actually expecting this from their peers as well. Note that that in case of backport, this patch depends on previous patch: MINOR: threads: make thread_release() not wait for other ones to complete
2021-08-04 05:44:17 -04:00
}
static inline void thread_idle_end()
{
BUG/MINOR: threads: set threads_idle and threads_harmless even with no threads Some signal handlers rely on these to decide about the level of detail to provide in dumps, so let's properly fill the info about entering/leaving idle. Note that for consistency with other tests we're using bitops with t->ltid_bit, while we could simply assign 0/1 to the fields. But it makes the code more readable and the whole difference is only 88 bytes on a 3MB executable. This bug is not important, and while older versions are likely affected as well, it's not worth taking the risk to backport this in case it would wake up an obscure bug.
2025-04-15 03:03:35 -04:00
tg_ctx->threads_idle &= ~ti->ltid_bit;
MEDIUM: threads: add a stronger thread_isolate_full() call The current principle of running under isolation was made to access sensitive data while being certain that no other thread was using them in parallel, without necessarily having to place locks everywhere. The main use case are "show sess" and "show fd" which run over long chains of pointers. The thread_isolate() call relies on the "harmless" bit that indicates for a given thread that it's not currently doing such sensitive things, which is advertised using thread_harmless_now() and which ends usings thread_harmless_end(), which also waits for possibly concurrent threads to complete their work if they took this opportunity for starting something tricky. As some system calls were notoriously slow (e.g. mmap()), a bunch of thread_harmless_now() / thread_harmless_end() were placed around them to let waiting threads do their work while such other threads were not able to modify memory contents. But this is not sufficient for performing memory modifications. One such example is the server deletion code. By modifying memory, it not only requires that other threads are not playing with it, but are not either in the process of touching it. The fact that a pool_alloc() or pool_free() on some structure may call thread_harmless_now() and let another thread start to release the same object's memory is not acceptable. This patch introduces the concept of "idle threads". Threads entering the polling loop are idle, as well as those that are waiting for all others to become idle via the new function thread_isolate_full(). Once thread_isolate_full() is granted, the thread is not idle anymore, and it is released using thread_release() just like regular isolation. Its users have to keep in mind that across this call nothing is granted as another thread might have performed shared memory modifications. But such users are extremely rare and are actually expecting this from their peers as well. Note that that in case of backport, this patch depends on previous patch: MINOR: threads: make thread_release() not wait for other ones to complete
2021-08-04 05:44:17 -04:00
}
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
static inline void thread_harmless_now()
{
BUG/MINOR: threads: set threads_idle and threads_harmless even with no threads Some signal handlers rely on these to decide about the level of detail to provide in dumps, so let's properly fill the info about entering/leaving idle. Note that for consistency with other tests we're using bitops with t->ltid_bit, while we could simply assign 0/1 to the fields. But it makes the code more readable and the whole difference is only 88 bytes on a 3MB executable. This bug is not important, and while older versions are likely affected as well, it's not worth taking the risk to backport this in case it would wake up an obscure bug.
2025-04-15 03:03:35 -04:00
tg_ctx->threads_harmless |= ti->ltid_bit;
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
}
MINOR: thread: add is_thread_harmless() to know if a thread already is harmless The harmless status is not re-entrant, so sometimes for signal handling it can be useful to know if we're already harmless or not. Let's add a function doing that, and make the debugger use it instead of manipulating the harmless mask.
2022-07-01 11:11:03 -04:00
static inline int is_thread_harmless()
{
BUG/MINOR: threads: set threads_idle and threads_harmless even with no threads Some signal handlers rely on these to decide about the level of detail to provide in dumps, so let's properly fill the info about entering/leaving idle. Note that for consistency with other tests we're using bitops with t->ltid_bit, while we could simply assign 0/1 to the fields. But it makes the code more readable and the whole difference is only 88 bytes on a 3MB executable. This bug is not important, and while older versions are likely affected as well, it's not worth taking the risk to backport this in case it would wake up an obscure bug.
2025-04-15 03:03:35 -04:00
return !!(tg_ctx->threads_harmless & ti->ltid_bit);
MINOR: thread: add is_thread_harmless() to know if a thread already is harmless The harmless status is not re-entrant, so sometimes for signal handling it can be useful to know if we're already harmless or not. Let's add a function doing that, and make the debugger use it instead of manipulating the harmless mask.
2022-07-01 11:11:03 -04:00
}
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
static inline void thread_harmless_end()
{
BUG/MINOR: threads: set threads_idle and threads_harmless even with no threads Some signal handlers rely on these to decide about the level of detail to provide in dumps, so let's properly fill the info about entering/leaving idle. Note that for consistency with other tests we're using bitops with t->ltid_bit, while we could simply assign 0/1 to the fields. But it makes the code more readable and the whole difference is only 88 bytes on a 3MB executable. This bug is not important, and while older versions are likely affected as well, it's not worth taking the risk to backport this in case it would wake up an obscure bug.
2025-04-15 03:03:35 -04:00
tg_ctx->threads_harmless &= ~ti->ltid_bit;
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
}
BUILD: thread: implement thread_harmless_end_sig() for threadless builds Building without thread support was broken in 2.8-dev2 with commit 7e70bfc8c ("MINOR: threads: add a thread_harmless_end() version that doesn't wait") that forgot to define the function for the threadless cases. No backport is needed.
2023-03-22 05:26:58 -04:00
static inline void thread_harmless_end_sig()
{
BUG/MINOR: threads: set threads_idle and threads_harmless even with no threads Some signal handlers rely on these to decide about the level of detail to provide in dumps, so let's properly fill the info about entering/leaving idle. Note that for consistency with other tests we're using bitops with t->ltid_bit, while we could simply assign 0/1 to the fields. But it makes the code more readable and the whole difference is only 88 bytes on a 3MB executable. This bug is not important, and while older versions are likely affected as well, it's not worth taking the risk to backport this in case it would wake up an obscure bug.
2025-04-15 03:03:35 -04:00
tg_ctx->threads_harmless &= ~ti->ltid_bit;
BUILD: thread: implement thread_harmless_end_sig() for threadless builds Building without thread support was broken in 2.8-dev2 with commit 7e70bfc8c ("MINOR: threads: add a thread_harmless_end() version that doesn't wait") that forgot to define the function for the threadless cases. No backport is needed.
2023-03-22 05:26:58 -04:00
}
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
static inline void thread_isolate()
{
}
MEDIUM: threads: add a stronger thread_isolate_full() call The current principle of running under isolation was made to access sensitive data while being certain that no other thread was using them in parallel, without necessarily having to place locks everywhere. The main use case are "show sess" and "show fd" which run over long chains of pointers. The thread_isolate() call relies on the "harmless" bit that indicates for a given thread that it's not currently doing such sensitive things, which is advertised using thread_harmless_now() and which ends usings thread_harmless_end(), which also waits for possibly concurrent threads to complete their work if they took this opportunity for starting something tricky. As some system calls were notoriously slow (e.g. mmap()), a bunch of thread_harmless_now() / thread_harmless_end() were placed around them to let waiting threads do their work while such other threads were not able to modify memory contents. But this is not sufficient for performing memory modifications. One such example is the server deletion code. By modifying memory, it not only requires that other threads are not playing with it, but are not either in the process of touching it. The fact that a pool_alloc() or pool_free() on some structure may call thread_harmless_now() and let another thread start to release the same object's memory is not acceptable. This patch introduces the concept of "idle threads". Threads entering the polling loop are idle, as well as those that are waiting for all others to become idle via the new function thread_isolate_full(). Once thread_isolate_full() is granted, the thread is not idle anymore, and it is released using thread_release() just like regular isolation. Its users have to keep in mind that across this call nothing is granted as another thread might have performed shared memory modifications. But such users are extremely rare and are actually expecting this from their peers as well. Note that that in case of backport, this patch depends on previous patch: MINOR: threads: make thread_release() not wait for other ones to complete
2021-08-04 05:44:17 -04:00
static inline void thread_isolate_full()
{
}
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
static inline void thread_release()
{
}
static inline unsigned long thread_isolated()
{
return 1;
}
REORG: thread: move the thread init/affinity/stop to thread.c haproxy.c still has to deal with pthread-specific low-level stuff that is OS-dependent. We should not have to deal with this there, and we do not need to access pthread anywhere else. Let's move these 3 functions to thread.c and keep empty inline ones for when threads are disabled.
2021-10-06 16:22:40 -04:00
static inline void setup_extra_threads(void *(*handler)(void *))
{
}
static inline void wait_for_threads_completion()
{
}
static inline void set_thread_cpu_affinity()
{
}
REORG: thread: move ha_get_pthread_id() to thread.c It's the last function which directly accesses the pthread_t, let's move it to thread.c and leave a static inline for non-thread.
2021-10-06 16:44:28 -04:00
static inline unsigned long long ha_get_pthread_id(unsigned int thr)
{
return 0;
}
MINOR: thread: define cshared type Define a new type "struct cshared". This can be used as a tool to manipulate a global counter with thread-safety ensured. Each thread would declare its thread-local cshared type, which would point to a global counter. Each thread can then add/substract value to their owned thread-local cshared instance via cshared_add(). If the difference exceed a configured limit, either positively or negatively, the global counter is updated and thread-local instance is reset to 0. Each thread can safely read the global counter value using cshared_read().
2025-04-25 05:37:07 -04:00
static inline void cshared_init(struct cshared *ctr, uint64_t *var, int lim)
{
ctr->global = var;
ctr->diff = 0;
ctr->lim = 0;
}
static inline void cshared_add(struct cshared *ctr, int diff)
{
ctr->global += diff;
}
static inline uint64_t cshared_read(struct cshared *ctr)
{
return *ctr->global;
}
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
#else /* !USE_THREAD */
/********************** THREADS ENABLED ************************/
MEDIUM: threads: Add hathreads header file This file contains all functions and macros used to deal with concurrency in HAProxy. It contains all high-level function to do atomic operation (HA_ATOMIC_*). Note, for now, we rely on "__atomic" GCC builtins to do atomic operation. So HAProxy can be compiled with the thread support iff these builtins are available. It also contains wrappers around plocks to use spin or read/write locks. These wrappers are used to abstract the internal representation of the locking system and to add information to help debugging, when compiled with suitable options. To add extra info on locks, you need to add DEBUG=-DDEBUG_THREAD or DEBUG=-DDEBUG_FULL compilation option. In addition to timing info on locks, we keep info on where a lock was acquired the last time (function name, file and line). There are also the thread id and a flag to know if it is still locked or not. This will be useful to debug deadlocks.
2017-10-12 10:09:09 -04:00
MINOR: threads: inline the wait function for pthread_rwlock emulation When using pthread_rwlock emulation, contention is reported on pl_wait_unlock_long(). This is really not convenient to analyse what is happening. Now plock supports inlining the wait call for just the lorw functions by enabling PLOCK_LORW_INLINE_WAIT. Let's do this so that now the wait time will be precisely reported as either pthread_rwlock_rdlock() or pthread_rwlock_wrlock() depending on the contended function, but no more on pl_wait_unlock_long(), which will still be reported for all other locks.
2023-08-16 16:51:37 -04:00
#define PLOCK_LORW_INLINE_WAIT
MEDIUM: threads: Add hathreads header file This file contains all functions and macros used to deal with concurrency in HAProxy. It contains all high-level function to do atomic operation (HA_ATOMIC_*). Note, for now, we rely on "__atomic" GCC builtins to do atomic operation. So HAProxy can be compiled with the thread support iff these builtins are available. It also contains wrappers around plocks to use spin or read/write locks. These wrappers are used to abstract the internal representation of the locking system and to add information to help debugging, when compiled with suitable options. To add extra info on locks, you need to add DEBUG=-DDEBUG_THREAD or DEBUG=-DDEBUG_FULL compilation option. In addition to timing info on locks, we keep info on where a lock was acquired the last time (function name, file and line). There are also the thread id and a flag to know if it is still locked or not. This will be useful to debug deadlocks.
2017-10-12 10:09:09 -04:00
#include <import/plock.h>
CLEANUP: tree-wide: fix prototypes for functions taking no arguments. "f(void)" is the correct and preferred form for a function taking no argument, while some places use the older "f()". These were reported by clang's -Wmissing-prototypes, for example: src/cpuset.c:111:5: warning: no previous prototype for function 'ha_cpuset_size' [-Wmissing-prototypes] int ha_cpuset_size() include/haproxy/cpuset.h:42:5: note: this declaration is not a prototype; add 'void' to make it a prototype for a zero-parameter function int ha_cpuset_size(); ^ void This aggregate patch fixes this for the following functions: ha_backtrace_to_stderr(), ha_cpuset_size(), ha_panic(), ha_random64(), ha_thread_dump_all_to_trash(), get_exec_path(), check_config_validity(), mworker_child_nb(), mworker_cli_proxy_(create|stop)(), mworker_cleantasks(), mworker_cleanlisteners(), mworker_ext_launch_all(), mworker_reload(), mworker_(env|proc_list)_to_(proc_list|env)(), mworker_(un|)block_signals(), proxy_adjust_all_maxconn(), proxy_destroy_all_defaults(), get_tainted(), pool_total_(allocated|used)(), thread_isolate(_full|)(), thread(_sync|)_release(), thread_harmless_till_end(), thread_cpu_mask_forced(), dequeue_all_listeners(), next_timer_expiry(), wake_expired_tasks(), process_runnable_tasks(), init_acl(), init_buffer(), (de|)init_log_buffers(), (de|)init_pollers(), fork_poller(), pool_destroy_all(), pool_evict_from_local_caches(), pool_total_failures(), dump_pools_to_trash(), cfg_run_diagnostics(), tv_init_(process|thread)_date(), __signal_process_queue(), deinit_signals(), haproxy_unblock_signals()
2021-09-12 06:49:33 -04:00
void thread_harmless_till_end(void);
void thread_isolate(void);
void thread_isolate_full(void);
void thread_release(void);
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
void ha_spin_init(HA_SPINLOCK_T *l);
void ha_rwlock_init(HA_RWLOCK_T *l);
REORG: thread: move the thread init/affinity/stop to thread.c haproxy.c still has to deal with pthread-specific low-level stuff that is OS-dependent. We should not have to deal with this there, and we do not need to access pthread anywhere else. Let's move these 3 functions to thread.c and keep empty inline ones for when threads are disabled.
2021-10-06 16:22:40 -04:00
void setup_extra_threads(void *(*handler)(void *));
void wait_for_threads_completion();
void set_thread_cpu_affinity();
REORG: thread: move ha_get_pthread_id() to thread.c It's the last function which directly accesses the pthread_t, let's move it to thread.c and leave a static inline for non-thread.
2021-10-06 16:44:28 -04:00
unsigned long long ha_get_pthread_id(unsigned int thr);
MEDIUM: threads: Adds a set of functions to handle sync-point A sync-point is a protected area where you have the warranty that no concurrency access is possible. It is implementated as a thread barrier to enter in the sync-point and another one to exit from it. Inside the sync-point, all threads that must do some syncrhonous processing will be called one after the other while all other threads will wait. All threads will then exit from the sync-point at the same time. A sync-point will be evaluated only when necessary because it is a costly operation. To limit the waiting time of each threads, we must have a mechanism to wakeup all threads. This is done with a pipe shared by all threads. By writting in this pipe, we will interrupt all threads blocked on a poller. The pipe is then flushed before exiting from the sync-point.
2017-10-19 05:59:15 -04:00
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
extern volatile unsigned int rdv_requests;
extern volatile unsigned int isolated_thread;
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
extern THREAD_LOCAL unsigned int tid; /* The thread id */
MINOR: threads: add the current group ID in thread-local "tgid" variable This is the equivalent of "tid" for ease of access. In the future if we make th_cfg a pure thread-local array (not a pointer), it may make sense to move it there.
2021-09-30 02:00:11 -04:00
extern THREAD_LOCAL unsigned int tgid; /* The thread group id (starts at 1) */
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
BUG/MINOR: signals: ha_sigmask macro for multithreading The behavior of sigprocmask in an multithreaded environment is undefined. The new macro ha_sigmask() calls either pthreads_sigmask() or sigprocmask() if haproxy was built with thread support or not. This should be backported to 1.8.
2018-06-07 05:23:40 -04:00
#define ha_sigmask(how, set, oldset) pthread_sigmask(how, set, oldset)
MEDIUM: threads: replace ha_set_tid() with ha_set_thread() ha_set_tid() was randomly used either to explicitly set thread 0 or to set any possibly incomplete thread during boot. Let's replace it with a pointer to a valid thread or NULL for any thread. This allows us to check that the designated threads are always valid, and to ignore the thread 0's mapping when setting it to NULL, and always use group 0 with it during boot. The initialization code is also cleaner, as we don't pass ugly casts of a thread ID to a pointer anymore.
2021-09-28 03:43:11 -04:00
/* Sets the current thread to a valid one described by <thr>, or to any thread
* and any group if NULL (e.g. for use during boot where they're not totally
* initialized).
MINOR: threads: make tg point to the current thread's group A the "tg" thread-local variable now always points to the current thread group. It's pre-initializd to the first one during boot and is set to point to the thread's one by ha_set_tid(). This last one takes care of checking whether the thread group was assigned or not because it may be called during boot before threads are initialized.
2021-09-28 02:58:49 -04:00
*/
MEDIUM: threads: replace ha_set_tid() with ha_set_thread() ha_set_tid() was randomly used either to explicitly set thread 0 or to set any possibly incomplete thread during boot. Let's replace it with a pointer to a valid thread or NULL for any thread. This allows us to check that the designated threads are always valid, and to ignore the thread 0's mapping when setting it to NULL, and always use group 0 with it during boot. The initialization code is also cleaner, as we don't pass ugly casts of a thread ID to a pointer anymore.
2021-09-28 03:43:11 -04:00
static inline void ha_set_thread(const struct thread_info *thr)
MINOR: threads: add more consistency between certain variables in no-thread case When threads are disabled, some variables such as tid and tid_bit are still checked everywhere, the MAX_THREADS_MASK macro is ~0UL while MAX_THREADS is 1, and the all_threads_mask variable is replaced with a macro forced to zero. The compiler cannot optimize away all this code involving checks on tid and tid_bit, and we end up in special cases where all_threads_mask has to be specifically tested for being zero or not. It is not even certain the code paths are always equivalent when testing without threads and with nbthread 1. Let's change this to make sure we always present a single thread when threads are disabled, and have the relevant values declared as constants so that the compiler can optimize all the tests away. Now we have MAX_THREADS_MASK set to 1, all_threads_mask set to 1, tid set to zero and tid_bit set to 1. Doing just this has removed 4 kB of code in the no-thread case. A few checks for all_threads_mask==0 have been removed since it never happens anymore.
2018-08-01 13:12:20 -04:00
{
MEDIUM: threads: replace ha_set_tid() with ha_set_thread() ha_set_tid() was randomly used either to explicitly set thread 0 or to set any possibly incomplete thread during boot. Let's replace it with a pointer to a valid thread or NULL for any thread. This allows us to check that the designated threads are always valid, and to ignore the thread 0's mapping when setting it to NULL, and always use group 0 with it during boot. The initialization code is also cleaner, as we don't pass ugly casts of a thread ID to a pointer anymore.
2021-09-28 03:43:11 -04:00
if (thr) {
MINOR: tinfo: remove the global thread ID bit (tid_bit) Each thread has its own local thread id and its own global thread id, in addition to the masks corresponding to each. Once the global thread ID can go beyond 64 it will not be possible to have a global thread Id bit anymore, so better start to remove it and use only the local one from the struct thread_info.
2022-06-14 04:43:01 -04:00
BUG_ON(!thr->ltid_bit);
MEDIUM: threads: replace ha_set_tid() with ha_set_thread() ha_set_tid() was randomly used either to explicitly set thread 0 or to set any possibly incomplete thread during boot. Let's replace it with a pointer to a valid thread or NULL for any thread. This allows us to check that the designated threads are always valid, and to ignore the thread 0's mapping when setting it to NULL, and always use group 0 with it during boot. The initialization code is also cleaner, as we don't pass ugly casts of a thread ID to a pointer anymore.
2021-09-28 03:43:11 -04:00
BUG_ON(!thr->tg);
MINOR: tinfo: add the tgid to the thread_info struct At several places we're dereferencing the thread group just to catch the group number, and this will become even more required once we start to use per-group contexts. Let's just add the tgid in the thread_info struct to make this easier.
2022-06-28 04:49:57 -04:00
BUG_ON(!thr->tgid);
MEDIUM: threads: replace ha_set_tid() with ha_set_thread() ha_set_tid() was randomly used either to explicitly set thread 0 or to set any possibly incomplete thread during boot. Let's replace it with a pointer to a valid thread or NULL for any thread. This allows us to check that the designated threads are always valid, and to ignore the thread 0's mapping when setting it to NULL, and always use group 0 with it during boot. The initialization code is also cleaner, as we don't pass ugly casts of a thread ID to a pointer anymore.
2021-09-28 03:43:11 -04:00
ti = thr;
tg = thr->tg;
tid = thr->tid;
MINOR: tinfo: add the tgid to the thread_info struct At several places we're dereferencing the thread group just to catch the group number, and this will become even more required once we start to use per-group contexts. Let's just add the tgid in the thread_info struct to make this easier.
2022-06-28 04:49:57 -04:00
tgid = thr->tgid;
MEDIUM: threads: replace ha_set_tid() with ha_set_thread() ha_set_tid() was randomly used either to explicitly set thread 0 or to set any possibly incomplete thread during boot. Let's replace it with a pointer to a valid thread or NULL for any thread. This allows us to check that the designated threads are always valid, and to ignore the thread 0's mapping when setting it to NULL, and always use group 0 with it during boot. The initialization code is also cleaner, as we don't pass ugly casts of a thread ID to a pointer anymore.
2021-09-28 03:43:11 -04:00
th_ctx = &ha_thread_ctx[tid];
MEDIUM: tinfo: add a dynamic thread-group context The thread group info is not sufficient to represent a thread group's current state as it's read-only. We also need something comparable to the thread context to represent the aggregate state of the threads in that group. This patch introduces ha_tgroup_ctx[] and tg_ctx for this. It's indexed on the group id and must be cache-line aligned. The thread masks that were global and that do not need to remain global were moved there (want_rdv, harmless, idle). Given that all the masks placed there now become group-specific, the associated thread mask (tid_bit) now switches to the thread's local bit (ltid_bit). Both are the same for nbtgroups 1 but will differ for other values. There's also a tg_ctx pointer in the thread so that it can be reached from other threads.
2022-06-27 10:02:24 -04:00
tg_ctx = &ha_tgroup_ctx[tgid-1];
MEDIUM: threads: replace ha_set_tid() with ha_set_thread() ha_set_tid() was randomly used either to explicitly set thread 0 or to set any possibly incomplete thread during boot. Let's replace it with a pointer to a valid thread or NULL for any thread. This allows us to check that the designated threads are always valid, and to ignore the thread 0's mapping when setting it to NULL, and always use group 0 with it during boot. The initialization code is also cleaner, as we don't pass ugly casts of a thread ID to a pointer anymore.
2021-09-28 03:43:11 -04:00
} else {
MINOR: threads: add the current group ID in thread-local "tgid" variable This is the equivalent of "tid" for ease of access. In the future if we make th_cfg a pure thread-local array (not a pointer), it may make sense to move it there.
2021-09-30 02:00:11 -04:00
tgid = 1;
MEDIUM: threads: replace ha_set_tid() with ha_set_thread() ha_set_tid() was randomly used either to explicitly set thread 0 or to set any possibly incomplete thread during boot. Let's replace it with a pointer to a valid thread or NULL for any thread. This allows us to check that the designated threads are always valid, and to ignore the thread 0's mapping when setting it to NULL, and always use group 0 with it during boot. The initialization code is also cleaner, as we don't pass ugly casts of a thread ID to a pointer anymore.
2021-09-28 03:43:11 -04:00
tid = 0;
ti = &ha_thread_info[0];
tg = &ha_tgroup_info[0];
th_ctx = &ha_thread_ctx[0];
MEDIUM: tinfo: add a dynamic thread-group context The thread group info is not sufficient to represent a thread group's current state as it's read-only. We also need something comparable to the thread context to represent the aggregate state of the threads in that group. This patch introduces ha_tgroup_ctx[] and tg_ctx for this. It's indexed on the group id and must be cache-line aligned. The thread masks that were global and that do not need to remain global were moved there (want_rdv, harmless, idle). Given that all the masks placed there now become group-specific, the associated thread mask (tid_bit) now switches to the thread's local bit (ltid_bit). Both are the same for nbtgroups 1 but will differ for other values. There's also a tg_ctx pointer in the thread so that it can be reached from other threads.
2022-06-27 10:02:24 -04:00
tg_ctx = &ha_tgroup_ctx[0];
MEDIUM: threads: replace ha_set_tid() with ha_set_thread() ha_set_tid() was randomly used either to explicitly set thread 0 or to set any possibly incomplete thread during boot. Let's replace it with a pointer to a valid thread or NULL for any thread. This allows us to check that the designated threads are always valid, and to ignore the thread 0's mapping when setting it to NULL, and always use group 0 with it during boot. The initialization code is also cleaner, as we don't pass ugly casts of a thread ID to a pointer anymore.
2021-09-28 03:43:11 -04:00
}
MINOR: threads: add more consistency between certain variables in no-thread case When threads are disabled, some variables such as tid and tid_bit are still checked everywhere, the MAX_THREADS_MASK macro is ~0UL while MAX_THREADS is 1, and the all_threads_mask variable is replaced with a macro forced to zero. The compiler cannot optimize away all this code involving checks on tid and tid_bit, and we end up in special cases where all_threads_mask has to be specifically tested for being zero or not. It is not even certain the code paths are always equivalent when testing without threads and with nbthread 1. Let's change this to make sure we always present a single thread when threads are disabled, and have the relevant values declared as constants so that the compiler can optimize all the tests away. Now we have MAX_THREADS_MASK set to 1, all_threads_mask set to 1, tid set to zero and tid_bit set to 1. Doing just this has removed 4 kB of code in the no-thread case. A few checks for all_threads_mask==0 have been removed since it never happens anymore.
2018-08-01 13:12:20 -04:00
}
MEDIUM: threads: add a stronger thread_isolate_full() call The current principle of running under isolation was made to access sensitive data while being certain that no other thread was using them in parallel, without necessarily having to place locks everywhere. The main use case are "show sess" and "show fd" which run over long chains of pointers. The thread_isolate() call relies on the "harmless" bit that indicates for a given thread that it's not currently doing such sensitive things, which is advertised using thread_harmless_now() and which ends usings thread_harmless_end(), which also waits for possibly concurrent threads to complete their work if they took this opportunity for starting something tricky. As some system calls were notoriously slow (e.g. mmap()), a bunch of thread_harmless_now() / thread_harmless_end() were placed around them to let waiting threads do their work while such other threads were not able to modify memory contents. But this is not sufficient for performing memory modifications. One such example is the server deletion code. By modifying memory, it not only requires that other threads are not playing with it, but are not either in the process of touching it. The fact that a pool_alloc() or pool_free() on some structure may call thread_harmless_now() and let another thread start to release the same object's memory is not acceptable. This patch introduces the concept of "idle threads". Threads entering the polling loop are idle, as well as those that are waiting for all others to become idle via the new function thread_isolate_full(). Once thread_isolate_full() is granted, the thread is not idle anymore, and it is released using thread_release() just like regular isolation. Its users have to keep in mind that across this call nothing is granted as another thread might have performed shared memory modifications. But such users are extremely rare and are actually expecting this from their peers as well. Note that that in case of backport, this patch depends on previous patch: MINOR: threads: make thread_release() not wait for other ones to complete
2021-08-04 05:44:17 -04:00
/* Marks the thread as idle, which means that not only it's not doing anything
* dangerous, but in addition it has not started anything sensitive either.
* This essentially means that the thread currently is in the poller, thus
* outside of any execution block. Needs to be terminated using
* thread_idle_end(). This is needed to release a concurrent call to
* thread_isolate_full().
*/
static inline void thread_idle_now()
{
MEDIUM: tinfo: add a dynamic thread-group context The thread group info is not sufficient to represent a thread group's current state as it's read-only. We also need something comparable to the thread context to represent the aggregate state of the threads in that group. This patch introduces ha_tgroup_ctx[] and tg_ctx for this. It's indexed on the group id and must be cache-line aligned. The thread masks that were global and that do not need to remain global were moved there (want_rdv, harmless, idle). Given that all the masks placed there now become group-specific, the associated thread mask (tid_bit) now switches to the thread's local bit (ltid_bit). Both are the same for nbtgroups 1 but will differ for other values. There's also a tg_ctx pointer in the thread so that it can be reached from other threads.
2022-06-27 10:02:24 -04:00
HA_ATOMIC_OR(&tg_ctx->threads_idle, ti->ltid_bit);
MEDIUM: threads: add a stronger thread_isolate_full() call The current principle of running under isolation was made to access sensitive data while being certain that no other thread was using them in parallel, without necessarily having to place locks everywhere. The main use case are "show sess" and "show fd" which run over long chains of pointers. The thread_isolate() call relies on the "harmless" bit that indicates for a given thread that it's not currently doing such sensitive things, which is advertised using thread_harmless_now() and which ends usings thread_harmless_end(), which also waits for possibly concurrent threads to complete their work if they took this opportunity for starting something tricky. As some system calls were notoriously slow (e.g. mmap()), a bunch of thread_harmless_now() / thread_harmless_end() were placed around them to let waiting threads do their work while such other threads were not able to modify memory contents. But this is not sufficient for performing memory modifications. One such example is the server deletion code. By modifying memory, it not only requires that other threads are not playing with it, but are not either in the process of touching it. The fact that a pool_alloc() or pool_free() on some structure may call thread_harmless_now() and let another thread start to release the same object's memory is not acceptable. This patch introduces the concept of "idle threads". Threads entering the polling loop are idle, as well as those that are waiting for all others to become idle via the new function thread_isolate_full(). Once thread_isolate_full() is granted, the thread is not idle anymore, and it is released using thread_release() just like regular isolation. Its users have to keep in mind that across this call nothing is granted as another thread might have performed shared memory modifications. But such users are extremely rare and are actually expecting this from their peers as well. Note that that in case of backport, this patch depends on previous patch: MINOR: threads: make thread_release() not wait for other ones to complete
2021-08-04 05:44:17 -04:00
}
/* Ends the harmless period started by thread_idle_now(), i.e. the thread is
* about to restart engaging in sensitive operations. This must not be done on
* a thread marked harmless, as it could cause a deadlock between another
* thread waiting for idle again and thread_harmless_end() in this thread.
*
* The right sequence is thus:
* thread_idle_now();
* thread_harmless_now();
* poll();
* thread_harmless_end();
* thread_idle_end();
*/
static inline void thread_idle_end()
{
MEDIUM: tinfo: add a dynamic thread-group context The thread group info is not sufficient to represent a thread group's current state as it's read-only. We also need something comparable to the thread context to represent the aggregate state of the threads in that group. This patch introduces ha_tgroup_ctx[] and tg_ctx for this. It's indexed on the group id and must be cache-line aligned. The thread masks that were global and that do not need to remain global were moved there (want_rdv, harmless, idle). Given that all the masks placed there now become group-specific, the associated thread mask (tid_bit) now switches to the thread's local bit (ltid_bit). Both are the same for nbtgroups 1 but will differ for other values. There's also a tg_ctx pointer in the thread so that it can be reached from other threads.
2022-06-27 10:02:24 -04:00
HA_ATOMIC_AND(&tg_ctx->threads_idle, ~ti->ltid_bit);
MEDIUM: threads: add a stronger thread_isolate_full() call The current principle of running under isolation was made to access sensitive data while being certain that no other thread was using them in parallel, without necessarily having to place locks everywhere. The main use case are "show sess" and "show fd" which run over long chains of pointers. The thread_isolate() call relies on the "harmless" bit that indicates for a given thread that it's not currently doing such sensitive things, which is advertised using thread_harmless_now() and which ends usings thread_harmless_end(), which also waits for possibly concurrent threads to complete their work if they took this opportunity for starting something tricky. As some system calls were notoriously slow (e.g. mmap()), a bunch of thread_harmless_now() / thread_harmless_end() were placed around them to let waiting threads do their work while such other threads were not able to modify memory contents. But this is not sufficient for performing memory modifications. One such example is the server deletion code. By modifying memory, it not only requires that other threads are not playing with it, but are not either in the process of touching it. The fact that a pool_alloc() or pool_free() on some structure may call thread_harmless_now() and let another thread start to release the same object's memory is not acceptable. This patch introduces the concept of "idle threads". Threads entering the polling loop are idle, as well as those that are waiting for all others to become idle via the new function thread_isolate_full(). Once thread_isolate_full() is granted, the thread is not idle anymore, and it is released using thread_release() just like regular isolation. Its users have to keep in mind that across this call nothing is granted as another thread might have performed shared memory modifications. But such users are extremely rare and are actually expecting this from their peers as well. Note that that in case of backport, this patch depends on previous patch: MINOR: threads: make thread_release() not wait for other ones to complete
2021-08-04 05:44:17 -04:00
}
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
/* Marks the thread as harmless. Note: this must be true, i.e. the thread must
* not be touching any unprotected shared resource during this period. Usually
* this is called before poll(), but it may also be placed around very slow
* calls (eg: some crypto operations). Needs to be terminated using
* thread_harmless_end().
*/
static inline void thread_harmless_now()
{
MEDIUM: tinfo: add a dynamic thread-group context The thread group info is not sufficient to represent a thread group's current state as it's read-only. We also need something comparable to the thread context to represent the aggregate state of the threads in that group. This patch introduces ha_tgroup_ctx[] and tg_ctx for this. It's indexed on the group id and must be cache-line aligned. The thread masks that were global and that do not need to remain global were moved there (want_rdv, harmless, idle). Given that all the masks placed there now become group-specific, the associated thread mask (tid_bit) now switches to the thread's local bit (ltid_bit). Both are the same for nbtgroups 1 but will differ for other values. There's also a tg_ctx pointer in the thread so that it can be reached from other threads.
2022-06-27 10:02:24 -04:00
HA_ATOMIC_OR(&tg_ctx->threads_harmless, ti->ltid_bit);
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
}
MINOR: thread: add is_thread_harmless() to know if a thread already is harmless The harmless status is not re-entrant, so sometimes for signal handling it can be useful to know if we're already harmless or not. Let's add a function doing that, and make the debugger use it instead of manipulating the harmless mask.
2022-07-01 11:11:03 -04:00
/* Returns non-zero if the current thread is already harmless */
static inline int is_thread_harmless()
{
return !!(HA_ATOMIC_LOAD(&tg_ctx->threads_harmless) & ti->ltid_bit);
}
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
/* Ends the harmless period started by thread_harmless_now(). Usually this is
* placed after the poll() call. If it is discovered that a job was running and
* is relying on the thread still being harmless, the thread waits for the
* other one to finish.
*/
static inline void thread_harmless_end()
{
while (1) {
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
HA_ATOMIC_AND(&tg_ctx->threads_harmless, ~ti->ltid_bit);
if (likely(_HA_ATOMIC_LOAD(&rdv_requests) == 0))
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
break;
thread_harmless_till_end();
}
}
MINOR: threads: add a thread_harmless_end() version that doesn't wait thread_harmless_end() needs to wait for rdv_requests to disappear so that we're certain to respect a harmless promise that possibly allowed another thread to proceed under isolation. But this doesn't work in a signal handler because a thread could be interrupted by the debug handler while already waiting for isolation and with rdv_request>0. As such this function could cause a deadlock in such a signal handler. Let's implement a specific variant for this, thread_harmless_end_sig(), that just resets the thread's bit and doesn't wait. It must of course not be used past a check point that would allow the isolation requester to return and see the thread as temporarily harmless then turning back on its promise. This will be needed to fix a race in the debug handler.
2023-01-19 12:43:48 -05:00
/* Ends the harmless period started by thread_harmless_now(), but without
* waiting for isolated requests. This is meant to be used from signal handlers
* which might be called recursively while a thread already requested an
* isolation that must be ignored. It must not be used past a checkpoint where
* another thread could return and see the current thread as harmless before
* this call (or this could validate an isolation request by accident).
*/
static inline void thread_harmless_end_sig()
{
HA_ATOMIC_AND(&tg_ctx->threads_harmless, ~ti->ltid_bit);
}
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
/* an isolated thread has its ID in isolated_thread */
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
static inline unsigned long thread_isolated()
{
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
return _HA_ATOMIC_LOAD(&isolated_thread) == tid;
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
}
MEDIUM: threads: keep history of taken locks with DEBUG_THREAD > 0 by only storing a word in each thread context, we can keep the history of all taken/dropped locks by label. This is expected to be very cheap and to permit to store up to 8 consecutive lock operations in 64 bits. That should significantly help detect recursive locks as well as figure what thread was likely to hinder another one waiting for a lock. For now we only store the final state of the lock, we don't store the attempt to get it. It's just a matter of space since we already need 4 ops (rd,sk,wr,un) which take 2 bits, leaving max 64 labels. We're already around 45. We could also multiply by 5 and still keep 8 bits total per lock, that would limit us to 51 locks max. It seems that most of the time if we get a watchdog panic, anyway the victim thread will be perfectly located so that we don't need a specific value for this. Another benefit is that we perform a single memory write per lock.
2025-04-28 03:42:58 -04:00
/* locking levels, for history and debugging */
#define _LK_UN 0
#define _LK_RD 1
#define _LK_SK 2
#define _LK_WR 3
#if (DEBUG_THREAD < 1) && !defined(DEBUG_FULL)
MINOR: thread: add a lock level information in the thread_ctx The new lock_level field indicates the number of cumulated locks that are held by the current thread. It's fed as soon as DEBUG_THREAD is at least 1. In addition, thread_isolate() adds 128, so that it's even possible to check for combinations of both. The value is also reported in thread dumps (warnings and panics).
2025-09-11 02:29:25 -04:00
#define _lock_wait(_LK_, bal, lbl, expr) do { (void)(expr); } while (0)
#define _lock_cond(_LK_, bal, lbl, expr) ({ typeof(expr) _expr = (expr); _expr; })
MEDIUM: threads: keep history of taken locks with DEBUG_THREAD > 0 by only storing a word in each thread context, we can keep the history of all taken/dropped locks by label. This is expected to be very cheap and to permit to store up to 8 consecutive lock operations in 64 bits. That should significantly help detect recursive locks as well as figure what thread was likely to hinder another one waiting for a lock. For now we only store the final state of the lock, we don't store the attempt to get it. It's just a matter of space since we already need 4 ops (rd,sk,wr,un) which take 2 bits, leaving max 64 labels. We're already around 45. We could also multiply by 5 and still keep 8 bits total per lock, that would limit us to 51 locks max. It seems that most of the time if we get a watchdog panic, anyway the victim thread will be perfectly located so that we don't need a specific value for this. Another benefit is that we perform a single memory write per lock.
2025-04-28 03:42:58 -04:00
#else
DEBUG: threads: merge successive idempotent lock operations in history In order to make the lock history a bit more useful, let's try to merge adjacent lock/unlock sequences that don't change anything for other threads. For this we can replace the last unlock with the new operation on the same label, and even just not store it if it was the same as the one before the unlock, since in the end it's the same as if the unlock had not been done. Now loops that used to be filled with "R:LISTENER U:LISTENER" show more useful info such as: S:IDLE_CONNS U:IDLE_CONNS S:PEER U:PEER S:IDLE_CONNS U:IDLE_CONNS R:LISTENER U:LISTENER U:STK_TABLE W:STK_SESS U:STK_SESS R:STK_TABLE U:STK_TABLE W:STK_SESS U:STK_SESS R:STK_TABLE R:STK_TABLE U:STK_TABLE W:STK_SESS U:STK_SESS W:STK_TABLE_UPDT U:STK_TABLE_UPDT S:PEER It's worth noting that it can sometimes induce confusion when recursive locks of the same label are used (a few exist on peers or stick-tables), as in such a case the two operations would be needed. However these ones are already undebuggable, so instead they will just have to be renamed to make sure they use a distinct label.
2025-05-05 12:24:40 -04:00
/* principle: each lock operation takes 8 bits, 6 of which (the highest) are
* the lock label, and two of which (the lowest) are the operation (_LK_*).
* In order to preserve as much usable history as possible, we try to merge
* repetitions:
* - if a lock is taken just after it was released, the release is erased
* from history and replace with the new operation ;
* - if, when replacing an unlock, the new operation is the same as the
* one before the unlock, then the new one is not added.
* This means that sequences like "R:foo U:foo R:foo" just become "R:foo",
* but that those like "R:foo U:foo W:foo U:foo" become "R:foo W:foo U:foo".
*/
#define _lock_wait_common(_LK_, lbl) do { \
ulong _lck = ((lbl + 1) << 2) + _LK_; \
if ((uint8_t)th_ctx->lock_history == (uint8_t)(((lbl + 1) << 2) + _LK_UN)) { \
/* re-lock of just unlocked, try to compact and possibly merge with n-2 */ \
th_ctx->lock_history >>= 8; \
if ((uint8_t)th_ctx->lock_history != (uint8_t)_lck) \
th_ctx->lock_history = (th_ctx->lock_history << 8) + _lck; \
} \
else \
th_ctx->lock_history = (th_ctx->lock_history << 8) + _lck; \
} while (0)
MINOR: thread: add a lock level information in the thread_ctx The new lock_level field indicates the number of cumulated locks that are held by the current thread. It's fed as soon as DEBUG_THREAD is at least 1. In addition, thread_isolate() adds 128, so that it's even possible to check for combinations of both. The value is also reported in thread dumps (warnings and panics).
2025-09-11 02:29:25 -04:00
#define _lock_wait(_LK_, bal, lbl, expr) do { \
MINOR: activity: collect time spent waiting on a lock for each task When DEBUG_THREAD > 0, and if task profiling is enabled, then each locking attempt will measure the time it takes to obtain the lock, then add that time to a thread_ctx accumulator that the scheduler will then retrieve to update the current task's sched_activity entry. The value will then appear avearaged over the number of calls in the lkw_avg column of "show profiling tasks", such as below: Tasks activity over 48.298 sec till 0.000 sec ago: function calls cpu_tot cpu_avg lkw_avg lat_avg h1_io_cb 3200170 26.81s 8.377us - 32.73us <- sock_conn_iocb@src/sock.c:1099 tasklet_wakeup sc_conn_io_cb 1657841 1.645s 992.0ns - 853.0us <- sc_app_chk_rcv_conn@src/stconn.c:844 tasklet_wakeup process_stream 1600450 49.16s 30.71us 1.936us 1.392ms <- sc_notify@src/stconn.c:1206 task_wakeup process_stream 1600321 7.770m 291.3us 209.1us 901.6us <- stream_new@src/stream.c:585 task_wakeup sc_conn_io_cb 1599928 7.975s 4.984us - 65.77us <- h1_wake_stream_for_recv@src/mux_h1.c:3633 tasklet_wakeup task_process_applet 997609 46.37s 46.48us 16.80us 113.0us <- sc_app_chk_snd_applet@src/stconn.c:1043 appctx_wakeup process_table_expire 922074 48.79s 52.92us 7.275us 181.1us <- run_tasks_from_lists@src/task.c:670 task_queue stktable_add_pend_updates 705423 1.511s 2.142us - 56.81us <- stktable_add_pend_updates@src/stick_table.c:869 tasklet_wakeup task_process_applet 683511 34.75s 50.84us 18.37us 153.3us <- __process_running_peer_sync@src/peers.c:3579 appctx_wakeup h1_io_cb 535395 198.1ms 370.0ns 72.00ns 930.4us <- h1_takeover@src/mux_h1.c:5659 tasklet_wakeup It now makes it pretty obvious which tasks (hence call chains) spend their time waiting on a lock and for what share of their execution time.
2025-09-11 04:47:35 -04:00
uint64_t lock_start = 0; \
extern uint64_t now_mono_time(void); \
if (_LK_ != _LK_UN) { \
MINOR: thread: add a lock level information in the thread_ctx The new lock_level field indicates the number of cumulated locks that are held by the current thread. It's fed as soon as DEBUG_THREAD is at least 1. In addition, thread_isolate() adds 128, so that it's even possible to check for combinations of both. The value is also reported in thread dumps (warnings and panics).
2025-09-11 02:29:25 -04:00
th_ctx->lock_level += bal; \
MEDIUM: activity: apply and use new finegrained task profiling settings In continuity of previous patch, this one makes use of the new profiling flags. For this, based on the global "profiling" setting, when switching profiling on, we set or clear two flags on the thread context, TH_FL_TASK_PROFILING_L and TH_FL_TASK_PROFILING_M to indicate whether lock profiling and/or malloc profiling are desired when profiling is enabled. These flags are checked along with TH_FL_TASK_PROFILING to decide when to collect time around a lock or a malloc. And by default we're back to the behavior of 3.2 in that neither lock nor malloc times are collected anymore. This is sufficient to see the CPU usage spent in the VDSO to significantly drop from 22% to 2.2% on a highly loaded system. This should be backported to 3.3 along with the previous patch.
2026-02-10 08:17:36 -05:00
if (unlikely((th_ctx->flags & (TH_FL_TASK_PROFILING|TH_FL_TASK_PROFILING_L)) == \
(TH_FL_TASK_PROFILING|TH_FL_TASK_PROFILING_L))) \
MINOR: activity: collect time spent waiting on a lock for each task When DEBUG_THREAD > 0, and if task profiling is enabled, then each locking attempt will measure the time it takes to obtain the lock, then add that time to a thread_ctx accumulator that the scheduler will then retrieve to update the current task's sched_activity entry. The value will then appear avearaged over the number of calls in the lkw_avg column of "show profiling tasks", such as below: Tasks activity over 48.298 sec till 0.000 sec ago: function calls cpu_tot cpu_avg lkw_avg lat_avg h1_io_cb 3200170 26.81s 8.377us - 32.73us <- sock_conn_iocb@src/sock.c:1099 tasklet_wakeup sc_conn_io_cb 1657841 1.645s 992.0ns - 853.0us <- sc_app_chk_rcv_conn@src/stconn.c:844 tasklet_wakeup process_stream 1600450 49.16s 30.71us 1.936us 1.392ms <- sc_notify@src/stconn.c:1206 task_wakeup process_stream 1600321 7.770m 291.3us 209.1us 901.6us <- stream_new@src/stream.c:585 task_wakeup sc_conn_io_cb 1599928 7.975s 4.984us - 65.77us <- h1_wake_stream_for_recv@src/mux_h1.c:3633 tasklet_wakeup task_process_applet 997609 46.37s 46.48us 16.80us 113.0us <- sc_app_chk_snd_applet@src/stconn.c:1043 appctx_wakeup process_table_expire 922074 48.79s 52.92us 7.275us 181.1us <- run_tasks_from_lists@src/task.c:670 task_queue stktable_add_pend_updates 705423 1.511s 2.142us - 56.81us <- stktable_add_pend_updates@src/stick_table.c:869 tasklet_wakeup task_process_applet 683511 34.75s 50.84us 18.37us 153.3us <- __process_running_peer_sync@src/peers.c:3579 appctx_wakeup h1_io_cb 535395 198.1ms 370.0ns 72.00ns 930.4us <- h1_takeover@src/mux_h1.c:5659 tasklet_wakeup It now makes it pretty obvious which tasks (hence call chains) spend their time waiting on a lock and for what share of their execution time.
2025-09-11 04:47:35 -04:00
lock_start = now_mono_time(); \
} \
MEDIUM: threads: keep history of taken locks with DEBUG_THREAD > 0 by only storing a word in each thread context, we can keep the history of all taken/dropped locks by label. This is expected to be very cheap and to permit to store up to 8 consecutive lock operations in 64 bits. That should significantly help detect recursive locks as well as figure what thread was likely to hinder another one waiting for a lock. For now we only store the final state of the lock, we don't store the attempt to get it. It's just a matter of space since we already need 4 ops (rd,sk,wr,un) which take 2 bits, leaving max 64 labels. We're already around 45. We could also multiply by 5 and still keep 8 bits total per lock, that would limit us to 51 locks max. It seems that most of the time if we get a watchdog panic, anyway the victim thread will be perfectly located so that we don't need a specific value for this. Another benefit is that we perform a single memory write per lock.
2025-04-28 03:42:58 -04:00
(void)(expr); \
MINOR: activity: collect time spent with a lock held for each task When DEBUG_THREAD > 0 and task profiling enabled, we'll now measure the time spent with at least one lock held for each task. The time is collected by locking operations when locks are taken raising the level to one, or released resetting the level. An accumulator is updated in the thread_ctx struct that is collected by the scheduler when the task returns, and updated in the sched_activity entry of the related task. This allows to observe figures like this one: Tasks activity over 259.516 sec till 0.000 sec ago: function calls cpu_tot cpu_avg lkw_avg lkd_avg lat_avg h1_io_cb 15466589 2.574m 9.984us - - 33.45us <- sock_conn_iocb@src/sock.c:1099 tasklet_wakeup sc_conn_io_cb 8047994 8.325s 1.034us - - 870.1us <- sc_app_chk_rcv_conn@src/stconn.c:844 tasklet_wakeup process_stream 7734689 4.356m 33.79us 1.990us 1.641us 1.554ms <- sc_notify@src/stconn.c:1206 task_wakeup process_stream 7734292 46.74m 362.6us 278.3us 132.2us 972.0us <- stream_new@src/stream.c:585 task_wakeup sc_conn_io_cb 7733158 46.88s 6.061us - - 68.78us <- h1_wake_stream_for_recv@src/mux_h1.c:3633 tasklet_wakeup task_process_applet 6603593 4.484m 40.74us 16.69us 34.00us 96.47us <- sc_app_chk_snd_applet@src/stconn.c:1043 appctx_wakeup task_process_applet 4761796 3.420m 43.09us 18.79us 39.28us 138.2us <- __process_running_peer_sync@src/peers.c:3579 appctx_wakeup process_table_expire 4710662 4.880m 62.16us 9.648us 53.95us 158.6us <- run_tasks_from_lists@src/task.c:671 task_queue stktable_add_pend_updates 4171868 6.786s 1.626us - 1.487us 47.94us <- stktable_add_pend_updates@src/stick_table.c:869 tasklet_wakeup h1_io_cb 2871683 1.198s 417.0ns 70.00ns 69.00ns 1.005ms <- h1_takeover@src/mux_h1.c:5659 tasklet_wakeup process_peer_sync 2304957 5.368s 2.328us - 1.156us 68.54us <- stktable_add_pend_updates@src/stick_table.c:873 task_wakeup process_peer_sync 1388141 3.174s 2.286us - 1.130us 52.31us <- run_tasks_from_lists@src/task.c:671 task_queue stktable_add_pend_updates 463488 3.530s 7.615us 2.000ns 7.134us 771.2us <- stktable_touch_with_exp@src/stick_table.c:654 tasklet_wakeup Here we see that almost the entirety of stktable_add_pend_updates() is spent under a lock, that 1/3 of the execution time of process_stream() was performed under a lock and that 2/3 of it was spent waiting for a lock (this is related to the 10 track-sc present in this config), and that the locking time in process_peer_sync() has now significantly reduced. This is more visible with "show profiling tasks aggr": Tasks activity over 475.354 sec till 0.000 sec ago: function calls cpu_tot cpu_avg lkw_avg lkd_avg lat_avg h1_io_cb 25742539 3.699m 8.622us 11.00ns 10.00ns 188.0us sc_conn_io_cb 22565666 1.475m 3.920us - - 473.9us process_stream 21665212 1.195h 198.6us 140.6us 67.08us 1.266ms task_process_applet 16352495 11.31m 41.51us 17.98us 36.55us 112.3us process_peer_sync 7831923 17.15s 2.189us - 1.107us 41.27us process_table_expire 6878569 6.866m 59.89us 9.359us 51.91us 151.8us stktable_add_pend_updates 6602502 14.77s 2.236us - 2.060us 119.8us h1_timeout_task 801 703.4us 878.0ns - - 185.7us srv_cleanup_toremove_conns 347 12.43ms 35.82us 240.0ns 70.00ns 1.924ms accept_queue_process 142 1.384ms 9.743us - - 340.6us srv_cleanup_idle_conns 74 475.0us 6.418us 896.0ns 5.667us 114.6us
2025-09-11 05:26:40 -04:00
if (_LK_ == _LK_UN) { \
MINOR: thread: add a lock level information in the thread_ctx The new lock_level field indicates the number of cumulated locks that are held by the current thread. It's fed as soon as DEBUG_THREAD is at least 1. In addition, thread_isolate() adds 128, so that it's even possible to check for combinations of both. The value is also reported in thread dumps (warnings and panics).
2025-09-11 02:29:25 -04:00
th_ctx->lock_level += bal; \
MEDIUM: activity: apply and use new finegrained task profiling settings In continuity of previous patch, this one makes use of the new profiling flags. For this, based on the global "profiling" setting, when switching profiling on, we set or clear two flags on the thread context, TH_FL_TASK_PROFILING_L and TH_FL_TASK_PROFILING_M to indicate whether lock profiling and/or malloc profiling are desired when profiling is enabled. These flags are checked along with TH_FL_TASK_PROFILING to decide when to collect time around a lock or a malloc. And by default we're back to the behavior of 3.2 in that neither lock nor malloc times are collected anymore. This is sufficient to see the CPU usage spent in the VDSO to significantly drop from 22% to 2.2% on a highly loaded system. This should be backported to 3.3 along with the previous patch.
2026-02-10 08:17:36 -05:00
if (th_ctx->lock_level == 0 &&\
unlikely((th_ctx->flags & (TH_FL_TASK_PROFILING|TH_FL_TASK_PROFILING_L)) == \
(TH_FL_TASK_PROFILING|TH_FL_TASK_PROFILING_L))) \
MINOR: activity: collect time spent with a lock held for each task When DEBUG_THREAD > 0 and task profiling enabled, we'll now measure the time spent with at least one lock held for each task. The time is collected by locking operations when locks are taken raising the level to one, or released resetting the level. An accumulator is updated in the thread_ctx struct that is collected by the scheduler when the task returns, and updated in the sched_activity entry of the related task. This allows to observe figures like this one: Tasks activity over 259.516 sec till 0.000 sec ago: function calls cpu_tot cpu_avg lkw_avg lkd_avg lat_avg h1_io_cb 15466589 2.574m 9.984us - - 33.45us <- sock_conn_iocb@src/sock.c:1099 tasklet_wakeup sc_conn_io_cb 8047994 8.325s 1.034us - - 870.1us <- sc_app_chk_rcv_conn@src/stconn.c:844 tasklet_wakeup process_stream 7734689 4.356m 33.79us 1.990us 1.641us 1.554ms <- sc_notify@src/stconn.c:1206 task_wakeup process_stream 7734292 46.74m 362.6us 278.3us 132.2us 972.0us <- stream_new@src/stream.c:585 task_wakeup sc_conn_io_cb 7733158 46.88s 6.061us - - 68.78us <- h1_wake_stream_for_recv@src/mux_h1.c:3633 tasklet_wakeup task_process_applet 6603593 4.484m 40.74us 16.69us 34.00us 96.47us <- sc_app_chk_snd_applet@src/stconn.c:1043 appctx_wakeup task_process_applet 4761796 3.420m 43.09us 18.79us 39.28us 138.2us <- __process_running_peer_sync@src/peers.c:3579 appctx_wakeup process_table_expire 4710662 4.880m 62.16us 9.648us 53.95us 158.6us <- run_tasks_from_lists@src/task.c:671 task_queue stktable_add_pend_updates 4171868 6.786s 1.626us - 1.487us 47.94us <- stktable_add_pend_updates@src/stick_table.c:869 tasklet_wakeup h1_io_cb 2871683 1.198s 417.0ns 70.00ns 69.00ns 1.005ms <- h1_takeover@src/mux_h1.c:5659 tasklet_wakeup process_peer_sync 2304957 5.368s 2.328us - 1.156us 68.54us <- stktable_add_pend_updates@src/stick_table.c:873 task_wakeup process_peer_sync 1388141 3.174s 2.286us - 1.130us 52.31us <- run_tasks_from_lists@src/task.c:671 task_queue stktable_add_pend_updates 463488 3.530s 7.615us 2.000ns 7.134us 771.2us <- stktable_touch_with_exp@src/stick_table.c:654 tasklet_wakeup Here we see that almost the entirety of stktable_add_pend_updates() is spent under a lock, that 1/3 of the execution time of process_stream() was performed under a lock and that 2/3 of it was spent waiting for a lock (this is related to the 10 track-sc present in this config), and that the locking time in process_peer_sync() has now significantly reduced. This is more visible with "show profiling tasks aggr": Tasks activity over 475.354 sec till 0.000 sec ago: function calls cpu_tot cpu_avg lkw_avg lkd_avg lat_avg h1_io_cb 25742539 3.699m 8.622us 11.00ns 10.00ns 188.0us sc_conn_io_cb 22565666 1.475m 3.920us - - 473.9us process_stream 21665212 1.195h 198.6us 140.6us 67.08us 1.266ms task_process_applet 16352495 11.31m 41.51us 17.98us 36.55us 112.3us process_peer_sync 7831923 17.15s 2.189us - 1.107us 41.27us process_table_expire 6878569 6.866m 59.89us 9.359us 51.91us 151.8us stktable_add_pend_updates 6602502 14.77s 2.236us - 2.060us 119.8us h1_timeout_task 801 703.4us 878.0ns - - 185.7us srv_cleanup_toremove_conns 347 12.43ms 35.82us 240.0ns 70.00ns 1.924ms accept_queue_process 142 1.384ms 9.743us - - 340.6us srv_cleanup_idle_conns 74 475.0us 6.418us 896.0ns 5.667us 114.6us
2025-09-11 05:26:40 -04:00
th_ctx->locked_total += now_mono_time() - th_ctx->lock_start_date; \
MEDIUM: activity: apply and use new finegrained task profiling settings In continuity of previous patch, this one makes use of the new profiling flags. For this, based on the global "profiling" setting, when switching profiling on, we set or clear two flags on the thread context, TH_FL_TASK_PROFILING_L and TH_FL_TASK_PROFILING_M to indicate whether lock profiling and/or malloc profiling are desired when profiling is enabled. These flags are checked along with TH_FL_TASK_PROFILING to decide when to collect time around a lock or a malloc. And by default we're back to the behavior of 3.2 in that neither lock nor malloc times are collected anymore. This is sufficient to see the CPU usage spent in the VDSO to significantly drop from 22% to 2.2% on a highly loaded system. This should be backported to 3.3 along with the previous patch.
2026-02-10 08:17:36 -05:00
} else if (unlikely((th_ctx->flags & (TH_FL_TASK_PROFILING|TH_FL_TASK_PROFILING_L)) == \
(TH_FL_TASK_PROFILING|TH_FL_TASK_PROFILING_L))) { \
MINOR: activity: collect time spent with a lock held for each task When DEBUG_THREAD > 0 and task profiling enabled, we'll now measure the time spent with at least one lock held for each task. The time is collected by locking operations when locks are taken raising the level to one, or released resetting the level. An accumulator is updated in the thread_ctx struct that is collected by the scheduler when the task returns, and updated in the sched_activity entry of the related task. This allows to observe figures like this one: Tasks activity over 259.516 sec till 0.000 sec ago: function calls cpu_tot cpu_avg lkw_avg lkd_avg lat_avg h1_io_cb 15466589 2.574m 9.984us - - 33.45us <- sock_conn_iocb@src/sock.c:1099 tasklet_wakeup sc_conn_io_cb 8047994 8.325s 1.034us - - 870.1us <- sc_app_chk_rcv_conn@src/stconn.c:844 tasklet_wakeup process_stream 7734689 4.356m 33.79us 1.990us 1.641us 1.554ms <- sc_notify@src/stconn.c:1206 task_wakeup process_stream 7734292 46.74m 362.6us 278.3us 132.2us 972.0us <- stream_new@src/stream.c:585 task_wakeup sc_conn_io_cb 7733158 46.88s 6.061us - - 68.78us <- h1_wake_stream_for_recv@src/mux_h1.c:3633 tasklet_wakeup task_process_applet 6603593 4.484m 40.74us 16.69us 34.00us 96.47us <- sc_app_chk_snd_applet@src/stconn.c:1043 appctx_wakeup task_process_applet 4761796 3.420m 43.09us 18.79us 39.28us 138.2us <- __process_running_peer_sync@src/peers.c:3579 appctx_wakeup process_table_expire 4710662 4.880m 62.16us 9.648us 53.95us 158.6us <- run_tasks_from_lists@src/task.c:671 task_queue stktable_add_pend_updates 4171868 6.786s 1.626us - 1.487us 47.94us <- stktable_add_pend_updates@src/stick_table.c:869 tasklet_wakeup h1_io_cb 2871683 1.198s 417.0ns 70.00ns 69.00ns 1.005ms <- h1_takeover@src/mux_h1.c:5659 tasklet_wakeup process_peer_sync 2304957 5.368s 2.328us - 1.156us 68.54us <- stktable_add_pend_updates@src/stick_table.c:873 task_wakeup process_peer_sync 1388141 3.174s 2.286us - 1.130us 52.31us <- run_tasks_from_lists@src/task.c:671 task_queue stktable_add_pend_updates 463488 3.530s 7.615us 2.000ns 7.134us 771.2us <- stktable_touch_with_exp@src/stick_table.c:654 tasklet_wakeup Here we see that almost the entirety of stktable_add_pend_updates() is spent under a lock, that 1/3 of the execution time of process_stream() was performed under a lock and that 2/3 of it was spent waiting for a lock (this is related to the 10 track-sc present in this config), and that the locking time in process_peer_sync() has now significantly reduced. This is more visible with "show profiling tasks aggr": Tasks activity over 475.354 sec till 0.000 sec ago: function calls cpu_tot cpu_avg lkw_avg lkd_avg lat_avg h1_io_cb 25742539 3.699m 8.622us 11.00ns 10.00ns 188.0us sc_conn_io_cb 22565666 1.475m 3.920us - - 473.9us process_stream 21665212 1.195h 198.6us 140.6us 67.08us 1.266ms task_process_applet 16352495 11.31m 41.51us 17.98us 36.55us 112.3us process_peer_sync 7831923 17.15s 2.189us - 1.107us 41.27us process_table_expire 6878569 6.866m 59.89us 9.359us 51.91us 151.8us stktable_add_pend_updates 6602502 14.77s 2.236us - 2.060us 119.8us h1_timeout_task 801 703.4us 878.0ns - - 185.7us srv_cleanup_toremove_conns 347 12.43ms 35.82us 240.0ns 70.00ns 1.924ms accept_queue_process 142 1.384ms 9.743us - - 340.6us srv_cleanup_idle_conns 74 475.0us 6.418us 896.0ns 5.667us 114.6us
2025-09-11 05:26:40 -04:00
uint64_t now = now_mono_time(); \
if (lock_start) \
th_ctx->lock_wait_total += now - lock_start; \
if (th_ctx->lock_level == 1) \
th_ctx->lock_start_date = now; \
} \
DEBUG: threads: don't keep lock label "OTHER" in the per-thread history Most threads are filled with "R:OTHER U:OTHER" in their history. Since anything non-important can use other it's not observable but it pollutes the history. Let's just drop OTHER entirely during the recording.
2025-05-05 12:10:57 -04:00
if (lbl != OTHER_LOCK) \
DEBUG: threads: merge successive idempotent lock operations in history In order to make the lock history a bit more useful, let's try to merge adjacent lock/unlock sequences that don't change anything for other threads. For this we can replace the last unlock with the new operation on the same label, and even just not store it if it was the same as the one before the unlock, since in the end it's the same as if the unlock had not been done. Now loops that used to be filled with "R:LISTENER U:LISTENER" show more useful info such as: S:IDLE_CONNS U:IDLE_CONNS S:PEER U:PEER S:IDLE_CONNS U:IDLE_CONNS R:LISTENER U:LISTENER U:STK_TABLE W:STK_SESS U:STK_SESS R:STK_TABLE U:STK_TABLE W:STK_SESS U:STK_SESS R:STK_TABLE R:STK_TABLE U:STK_TABLE W:STK_SESS U:STK_SESS W:STK_TABLE_UPDT U:STK_TABLE_UPDT S:PEER It's worth noting that it can sometimes induce confusion when recursive locks of the same label are used (a few exist on peers or stick-tables), as in such a case the two operations would be needed. However these ones are already undebuggable, so instead they will just have to be renamed to make sure they use a distinct label.
2025-05-05 12:24:40 -04:00
_lock_wait_common(_LK_, lbl); \
MEDIUM: threads: keep history of taken locks with DEBUG_THREAD > 0 by only storing a word in each thread context, we can keep the history of all taken/dropped locks by label. This is expected to be very cheap and to permit to store up to 8 consecutive lock operations in 64 bits. That should significantly help detect recursive locks as well as figure what thread was likely to hinder another one waiting for a lock. For now we only store the final state of the lock, we don't store the attempt to get it. It's just a matter of space since we already need 4 ops (rd,sk,wr,un) which take 2 bits, leaving max 64 labels. We're already around 45. We could also multiply by 5 and still keep 8 bits total per lock, that would limit us to 51 locks max. It seems that most of the time if we get a watchdog panic, anyway the victim thread will be perfectly located so that we don't need a specific value for this. Another benefit is that we perform a single memory write per lock.
2025-04-28 03:42:58 -04:00
} while (0)
MINOR: thread: add a lock level information in the thread_ctx The new lock_level field indicates the number of cumulated locks that are held by the current thread. It's fed as soon as DEBUG_THREAD is at least 1. In addition, thread_isolate() adds 128, so that it's even possible to check for combinations of both. The value is also reported in thread dumps (warnings and panics).
2025-09-11 02:29:25 -04:00
#define _lock_cond(_LK_, bal, lbl, expr) ({ \
MEDIUM: threads: keep history of taken locks with DEBUG_THREAD > 0 by only storing a word in each thread context, we can keep the history of all taken/dropped locks by label. This is expected to be very cheap and to permit to store up to 8 consecutive lock operations in 64 bits. That should significantly help detect recursive locks as well as figure what thread was likely to hinder another one waiting for a lock. For now we only store the final state of the lock, we don't store the attempt to get it. It's just a matter of space since we already need 4 ops (rd,sk,wr,un) which take 2 bits, leaving max 64 labels. We're already around 45. We could also multiply by 5 and still keep 8 bits total per lock, that would limit us to 51 locks max. It seems that most of the time if we get a watchdog panic, anyway the victim thread will be perfectly located so that we don't need a specific value for this. Another benefit is that we perform a single memory write per lock.
2025-04-28 03:42:58 -04:00
typeof(expr) _expr = (expr); \
MINOR: activity: collect time spent with a lock held for each task When DEBUG_THREAD > 0 and task profiling enabled, we'll now measure the time spent with at least one lock held for each task. The time is collected by locking operations when locks are taken raising the level to one, or released resetting the level. An accumulator is updated in the thread_ctx struct that is collected by the scheduler when the task returns, and updated in the sched_activity entry of the related task. This allows to observe figures like this one: Tasks activity over 259.516 sec till 0.000 sec ago: function calls cpu_tot cpu_avg lkw_avg lkd_avg lat_avg h1_io_cb 15466589 2.574m 9.984us - - 33.45us <- sock_conn_iocb@src/sock.c:1099 tasklet_wakeup sc_conn_io_cb 8047994 8.325s 1.034us - - 870.1us <- sc_app_chk_rcv_conn@src/stconn.c:844 tasklet_wakeup process_stream 7734689 4.356m 33.79us 1.990us 1.641us 1.554ms <- sc_notify@src/stconn.c:1206 task_wakeup process_stream 7734292 46.74m 362.6us 278.3us 132.2us 972.0us <- stream_new@src/stream.c:585 task_wakeup sc_conn_io_cb 7733158 46.88s 6.061us - - 68.78us <- h1_wake_stream_for_recv@src/mux_h1.c:3633 tasklet_wakeup task_process_applet 6603593 4.484m 40.74us 16.69us 34.00us 96.47us <- sc_app_chk_snd_applet@src/stconn.c:1043 appctx_wakeup task_process_applet 4761796 3.420m 43.09us 18.79us 39.28us 138.2us <- __process_running_peer_sync@src/peers.c:3579 appctx_wakeup process_table_expire 4710662 4.880m 62.16us 9.648us 53.95us 158.6us <- run_tasks_from_lists@src/task.c:671 task_queue stktable_add_pend_updates 4171868 6.786s 1.626us - 1.487us 47.94us <- stktable_add_pend_updates@src/stick_table.c:869 tasklet_wakeup h1_io_cb 2871683 1.198s 417.0ns 70.00ns 69.00ns 1.005ms <- h1_takeover@src/mux_h1.c:5659 tasklet_wakeup process_peer_sync 2304957 5.368s 2.328us - 1.156us 68.54us <- stktable_add_pend_updates@src/stick_table.c:873 task_wakeup process_peer_sync 1388141 3.174s 2.286us - 1.130us 52.31us <- run_tasks_from_lists@src/task.c:671 task_queue stktable_add_pend_updates 463488 3.530s 7.615us 2.000ns 7.134us 771.2us <- stktable_touch_with_exp@src/stick_table.c:654 tasklet_wakeup Here we see that almost the entirety of stktable_add_pend_updates() is spent under a lock, that 1/3 of the execution time of process_stream() was performed under a lock and that 2/3 of it was spent waiting for a lock (this is related to the 10 track-sc present in this config), and that the locking time in process_peer_sync() has now significantly reduced. This is more visible with "show profiling tasks aggr": Tasks activity over 475.354 sec till 0.000 sec ago: function calls cpu_tot cpu_avg lkw_avg lkd_avg lat_avg h1_io_cb 25742539 3.699m 8.622us 11.00ns 10.00ns 188.0us sc_conn_io_cb 22565666 1.475m 3.920us - - 473.9us process_stream 21665212 1.195h 198.6us 140.6us 67.08us 1.266ms task_process_applet 16352495 11.31m 41.51us 17.98us 36.55us 112.3us process_peer_sync 7831923 17.15s 2.189us - 1.107us 41.27us process_table_expire 6878569 6.866m 59.89us 9.359us 51.91us 151.8us stktable_add_pend_updates 6602502 14.77s 2.236us - 2.060us 119.8us h1_timeout_task 801 703.4us 878.0ns - - 185.7us srv_cleanup_toremove_conns 347 12.43ms 35.82us 240.0ns 70.00ns 1.924ms accept_queue_process 142 1.384ms 9.743us - - 340.6us srv_cleanup_idle_conns 74 475.0us 6.418us 896.0ns 5.667us 114.6us
2025-09-11 05:26:40 -04:00
if (_expr == 0) { \
MINOR: thread: add a lock level information in the thread_ctx The new lock_level field indicates the number of cumulated locks that are held by the current thread. It's fed as soon as DEBUG_THREAD is at least 1. In addition, thread_isolate() adds 128, so that it's even possible to check for combinations of both. The value is also reported in thread dumps (warnings and panics).
2025-09-11 02:29:25 -04:00
th_ctx->lock_level += bal; \
MEDIUM: activity: apply and use new finegrained task profiling settings In continuity of previous patch, this one makes use of the new profiling flags. For this, based on the global "profiling" setting, when switching profiling on, we set or clear two flags on the thread context, TH_FL_TASK_PROFILING_L and TH_FL_TASK_PROFILING_M to indicate whether lock profiling and/or malloc profiling are desired when profiling is enabled. These flags are checked along with TH_FL_TASK_PROFILING to decide when to collect time around a lock or a malloc. And by default we're back to the behavior of 3.2 in that neither lock nor malloc times are collected anymore. This is sufficient to see the CPU usage spent in the VDSO to significantly drop from 22% to 2.2% on a highly loaded system. This should be backported to 3.3 along with the previous patch.
2026-02-10 08:17:36 -05:00
if (unlikely((th_ctx->flags & (TH_FL_TASK_PROFILING|TH_FL_TASK_PROFILING_L)) == \
(TH_FL_TASK_PROFILING|TH_FL_TASK_PROFILING_L))) { \
MINOR: activity: collect time spent with a lock held for each task When DEBUG_THREAD > 0 and task profiling enabled, we'll now measure the time spent with at least one lock held for each task. The time is collected by locking operations when locks are taken raising the level to one, or released resetting the level. An accumulator is updated in the thread_ctx struct that is collected by the scheduler when the task returns, and updated in the sched_activity entry of the related task. This allows to observe figures like this one: Tasks activity over 259.516 sec till 0.000 sec ago: function calls cpu_tot cpu_avg lkw_avg lkd_avg lat_avg h1_io_cb 15466589 2.574m 9.984us - - 33.45us <- sock_conn_iocb@src/sock.c:1099 tasklet_wakeup sc_conn_io_cb 8047994 8.325s 1.034us - - 870.1us <- sc_app_chk_rcv_conn@src/stconn.c:844 tasklet_wakeup process_stream 7734689 4.356m 33.79us 1.990us 1.641us 1.554ms <- sc_notify@src/stconn.c:1206 task_wakeup process_stream 7734292 46.74m 362.6us 278.3us 132.2us 972.0us <- stream_new@src/stream.c:585 task_wakeup sc_conn_io_cb 7733158 46.88s 6.061us - - 68.78us <- h1_wake_stream_for_recv@src/mux_h1.c:3633 tasklet_wakeup task_process_applet 6603593 4.484m 40.74us 16.69us 34.00us 96.47us <- sc_app_chk_snd_applet@src/stconn.c:1043 appctx_wakeup task_process_applet 4761796 3.420m 43.09us 18.79us 39.28us 138.2us <- __process_running_peer_sync@src/peers.c:3579 appctx_wakeup process_table_expire 4710662 4.880m 62.16us 9.648us 53.95us 158.6us <- run_tasks_from_lists@src/task.c:671 task_queue stktable_add_pend_updates 4171868 6.786s 1.626us - 1.487us 47.94us <- stktable_add_pend_updates@src/stick_table.c:869 tasklet_wakeup h1_io_cb 2871683 1.198s 417.0ns 70.00ns 69.00ns 1.005ms <- h1_takeover@src/mux_h1.c:5659 tasklet_wakeup process_peer_sync 2304957 5.368s 2.328us - 1.156us 68.54us <- stktable_add_pend_updates@src/stick_table.c:873 task_wakeup process_peer_sync 1388141 3.174s 2.286us - 1.130us 52.31us <- run_tasks_from_lists@src/task.c:671 task_queue stktable_add_pend_updates 463488 3.530s 7.615us 2.000ns 7.134us 771.2us <- stktable_touch_with_exp@src/stick_table.c:654 tasklet_wakeup Here we see that almost the entirety of stktable_add_pend_updates() is spent under a lock, that 1/3 of the execution time of process_stream() was performed under a lock and that 2/3 of it was spent waiting for a lock (this is related to the 10 track-sc present in this config), and that the locking time in process_peer_sync() has now significantly reduced. This is more visible with "show profiling tasks aggr": Tasks activity over 475.354 sec till 0.000 sec ago: function calls cpu_tot cpu_avg lkw_avg lkd_avg lat_avg h1_io_cb 25742539 3.699m 8.622us 11.00ns 10.00ns 188.0us sc_conn_io_cb 22565666 1.475m 3.920us - - 473.9us process_stream 21665212 1.195h 198.6us 140.6us 67.08us 1.266ms task_process_applet 16352495 11.31m 41.51us 17.98us 36.55us 112.3us process_peer_sync 7831923 17.15s 2.189us - 1.107us 41.27us process_table_expire 6878569 6.866m 59.89us 9.359us 51.91us 151.8us stktable_add_pend_updates 6602502 14.77s 2.236us - 2.060us 119.8us h1_timeout_task 801 703.4us 878.0ns - - 185.7us srv_cleanup_toremove_conns 347 12.43ms 35.82us 240.0ns 70.00ns 1.924ms accept_queue_process 142 1.384ms 9.743us - - 340.6us srv_cleanup_idle_conns 74 475.0us 6.418us 896.0ns 5.667us 114.6us
2025-09-11 05:26:40 -04:00
if (_LK_ == _LK_UN && th_ctx->lock_level == 0) \
th_ctx->locked_total += now_mono_time() - th_ctx->lock_start_date; \
else if (_LK_ != _LK_UN && th_ctx->lock_level == 1) \
th_ctx->lock_start_date = now_mono_time(); \
} \
} \
DEBUG: threads: don't keep lock label "OTHER" in the per-thread history Most threads are filled with "R:OTHER U:OTHER" in their history. Since anything non-important can use other it's not observable but it pollutes the history. Let's just drop OTHER entirely during the recording.
2025-05-05 12:10:57 -04:00
if (lbl != OTHER_LOCK && !_expr) \
DEBUG: threads: merge successive idempotent lock operations in history In order to make the lock history a bit more useful, let's try to merge adjacent lock/unlock sequences that don't change anything for other threads. For this we can replace the last unlock with the new operation on the same label, and even just not store it if it was the same as the one before the unlock, since in the end it's the same as if the unlock had not been done. Now loops that used to be filled with "R:LISTENER U:LISTENER" show more useful info such as: S:IDLE_CONNS U:IDLE_CONNS S:PEER U:PEER S:IDLE_CONNS U:IDLE_CONNS R:LISTENER U:LISTENER U:STK_TABLE W:STK_SESS U:STK_SESS R:STK_TABLE U:STK_TABLE W:STK_SESS U:STK_SESS R:STK_TABLE R:STK_TABLE U:STK_TABLE W:STK_SESS U:STK_SESS W:STK_TABLE_UPDT U:STK_TABLE_UPDT S:PEER It's worth noting that it can sometimes induce confusion when recursive locks of the same label are used (a few exist on peers or stick-tables), as in such a case the two operations would be needed. However these ones are already undebuggable, so instead they will just have to be renamed to make sure they use a distinct label.
2025-05-05 12:24:40 -04:00
_lock_wait_common(_LK_, lbl); \
MEDIUM: threads: keep history of taken locks with DEBUG_THREAD > 0 by only storing a word in each thread context, we can keep the history of all taken/dropped locks by label. This is expected to be very cheap and to permit to store up to 8 consecutive lock operations in 64 bits. That should significantly help detect recursive locks as well as figure what thread was likely to hinder another one waiting for a lock. For now we only store the final state of the lock, we don't store the attempt to get it. It's just a matter of space since we already need 4 ops (rd,sk,wr,un) which take 2 bits, leaving max 64 labels. We're already around 45. We could also multiply by 5 and still keep 8 bits total per lock, that would limit us to 51 locks max. It seems that most of the time if we get a watchdog panic, anyway the victim thread will be perfectly located so that we don't need a specific value for this. Another benefit is that we perform a single memory write per lock.
2025-04-28 03:42:58 -04:00
_expr; \
})
#endif
MINOR: thread: define cshared type Define a new type "struct cshared". This can be used as a tool to manipulate a global counter with thread-safety ensured. Each thread would declare its thread-local cshared type, which would point to a global counter. Each thread can then add/substract value to their owned thread-local cshared instance via cshared_add(). If the difference exceed a configured limit, either positively or negatively, the global counter is updated and thread-local instance is reset to 0. Each thread can safely read the global counter value using cshared_read().
2025-04-25 05:37:07 -04:00
/* Init a shared counter <ctr> which references global value <var>. Update are
* performed each time the shared counter exceed <lim>, either on the positive
* or negative value.
*/
static inline void cshared_init(struct cshared *ctr, uint64_t *var, int lim)
{
ctr->global = var;
ctr->diff = 0;
ctr->lim = lim;
}
/* Add <diff>, which may be positive or negative, to <ctr> shared counter. */
static inline void cshared_add(struct cshared *ctr, int diff)
{
ctr->diff += diff;
if (ctr->diff <= -(ctr->lim) || ctr->diff >= ctr->lim) {
HA_ATOMIC_ADD(ctr->global, ctr->diff);
ctr->diff = 0;
}
}
/* Atomically get current global value from <ctr> shared counter. */
static inline uint64_t cshared_read(struct cshared *ctr)
{
return HA_ATOMIC_LOAD(ctr->global);
}
MINOR: threads: turn the full lock debugging to DEBUG_THREAD=2 At level 1 it now does nothing. This is reserved for some subsequent patches which will implement lighter debugging.
2025-04-28 03:05:02 -04:00
#if (DEBUG_THREAD < 2) && !defined(DEBUG_FULL)
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
/* Thread debugging is DISABLED, these are the regular locking functions */
#define HA_SPIN_INIT(l) ({ (*l) = 0; })
#define HA_SPIN_DESTROY(l) ({ (*l) = 0; })
MINOR: thread: add a lock level information in the thread_ctx The new lock_level field indicates the number of cumulated locks that are held by the current thread. It's fed as soon as DEBUG_THREAD is at least 1. In addition, thread_isolate() adds 128, so that it's even possible to check for combinations of both. The value is also reported in thread dumps (warnings and panics).
2025-09-11 02:29:25 -04:00
#define HA_SPIN_LOCK(lbl, l) _lock_wait(_LK_SK, 1, lbl, pl_take_s(l))
#define HA_SPIN_TRYLOCK(lbl, l) _lock_cond(_LK_SK, 1, lbl, !pl_try_s(l))
#define HA_SPIN_UNLOCK(lbl, l) _lock_wait(_LK_UN, -1, lbl, pl_drop_s(l))
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
#define HA_RWLOCK_INIT(l) ({ (*l) = 0; })
#define HA_RWLOCK_DESTROY(l) ({ (*l) = 0; })
MINOR: thread: add a lock level information in the thread_ctx The new lock_level field indicates the number of cumulated locks that are held by the current thread. It's fed as soon as DEBUG_THREAD is at least 1. In addition, thread_isolate() adds 128, so that it's even possible to check for combinations of both. The value is also reported in thread dumps (warnings and panics).
2025-09-11 02:29:25 -04:00
#define HA_RWLOCK_WRLOCK(lbl,l) _lock_wait(_LK_WR, 1, lbl, pl_take_w(l))
#define HA_RWLOCK_TRYWRLOCK(lbl,l) _lock_cond(_LK_WR, 1, lbl, !pl_try_w(l))
#define HA_RWLOCK_WRUNLOCK(lbl,l) _lock_wait(_LK_UN, -1, lbl, pl_drop_w(l))
#define HA_RWLOCK_RDLOCK(lbl,l) _lock_wait(_LK_RD, 1, lbl, pl_take_r(l))
#define HA_RWLOCK_TRYRDLOCK(lbl,l) _lock_cond(_LK_RD, 1, lbl, (!pl_try_r(l)))
#define HA_RWLOCK_RDUNLOCK(lbl,l) _lock_wait(_LK_UN, -1, lbl, pl_drop_r(l))
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
MINOR: threads: add the transitions to/from the seek state Since our locks are based on progressive locks, we support the upgradable seek lock that is compatible with readers and upgradable to a write lock. The main purpose is to take it while seeking down a tree for modification while other threads may seek the same tree for an input (e.g. compute the next event date). The newly supported operations are: HA_RWLOCK_SKLOCK(lbl,l) pl_take_s(l) /* N --> S */ HA_RWLOCK_SKTOWR(lbl,l) pl_stow(l) /* S --> W */ HA_RWLOCK_WRTOSK(lbl,l) pl_wtos(l) /* W --> S */ HA_RWLOCK_SKTORD(lbl,l) pl_stor(l) /* S --> R */ HA_RWLOCK_WRTORD(lbl,l) pl_wtor(l) /* W --> R */ HA_RWLOCK_SKUNLOCK(lbl,l) pl_drop_s(l) /* S --> N */ HA_RWLOCK_TRYSKLOCK(lbl,l) (!pl_try_s(l)) /* N -?> S */ HA_RWLOCK_TRYRDTOSK(lbl,l) (!pl_try_rtos(l)) /* R -?> S */ Existing code paths are left unaffected so this patch doesn't affect any running code.
2020-10-16 10:53:46 -04:00
/* rwlock upgrades via seek locks */
MINOR: thread: add a lock level information in the thread_ctx The new lock_level field indicates the number of cumulated locks that are held by the current thread. It's fed as soon as DEBUG_THREAD is at least 1. In addition, thread_isolate() adds 128, so that it's even possible to check for combinations of both. The value is also reported in thread dumps (warnings and panics).
2025-09-11 02:29:25 -04:00
#define HA_RWLOCK_SKLOCK(lbl,l) _lock_wait(_LK_SK, 1, lbl, pl_take_s(l)) /* N --> S */
#define HA_RWLOCK_SKTOWR(lbl,l) _lock_wait(_LK_WR, 0, lbl, pl_stow(l)) /* S --> W */
#define HA_RWLOCK_WRTOSK(lbl,l) _lock_wait(_LK_SK, 0, lbl, pl_wtos(l)) /* W --> S */
#define HA_RWLOCK_SKTORD(lbl,l) _lock_wait(_LK_RD, 0, lbl, pl_stor(l)) /* S --> R */
#define HA_RWLOCK_WRTORD(lbl,l) _lock_wait(_LK_RD, 0, lbl, pl_wtor(l)) /* W --> R */
#define HA_RWLOCK_SKUNLOCK(lbl,l) _lock_wait(_LK_UN, -1, lbl, pl_drop_s(l)) /* S --> N */
#define HA_RWLOCK_TRYSKLOCK(lbl,l) _lock_cond(_LK_SK, 1, lbl, !pl_try_s(l)) /* N -?> S */
#define HA_RWLOCK_TRYRDTOSK(lbl,l) _lock_cond(_LK_SK, 0, lbl, !pl_try_rtos(l)) /* R -?> S */
#define HA_RWLOCK_TRYRDTOWR(lbl, l) _lock_cond(_LK_WR, 0, lbl, !pl_try_rtow(l)) /* R -?> W */
MINOR: threads: add the transitions to/from the seek state Since our locks are based on progressive locks, we support the upgradable seek lock that is compatible with readers and upgradable to a write lock. The main purpose is to take it while seeking down a tree for modification while other threads may seek the same tree for an input (e.g. compute the next event date). The newly supported operations are: HA_RWLOCK_SKLOCK(lbl,l) pl_take_s(l) /* N --> S */ HA_RWLOCK_SKTOWR(lbl,l) pl_stow(l) /* S --> W */ HA_RWLOCK_WRTOSK(lbl,l) pl_wtos(l) /* W --> S */ HA_RWLOCK_SKTORD(lbl,l) pl_stor(l) /* S --> R */ HA_RWLOCK_WRTORD(lbl,l) pl_wtor(l) /* W --> R */ HA_RWLOCK_SKUNLOCK(lbl,l) pl_drop_s(l) /* S --> N */ HA_RWLOCK_TRYSKLOCK(lbl,l) (!pl_try_s(l)) /* N -?> S */ HA_RWLOCK_TRYRDTOSK(lbl,l) (!pl_try_rtos(l)) /* R -?> S */ Existing code paths are left unaffected so this patch doesn't affect any running code.
2020-10-16 10:53:46 -04:00
MINOR: threads: turn the full lock debugging to DEBUG_THREAD=2 At level 1 it now does nothing. This is reserved for some subsequent patches which will implement lighter debugging.
2025-04-28 03:05:02 -04:00
#else /* (DEBUG_THREAD < 2) && !defined(DEBUG_FULL) */
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
/* Thread debugging is ENABLED, these are the instrumented functions */
#define __SPIN_INIT(l) ({ (*l) = 0; })
#define __SPIN_DESTROY(l) ({ (*l) = 0; })
#define __SPIN_LOCK(l) pl_take_s(l)
BUILD: thread: add parenthesis around values of locking macros clang just failed on fd.c with this error: src/fd.c:491:9: error: logical not is only applied to the left hand side of this comparison [-Werror,-Wlogical-not-parentheses] while (HA_SPIN_TRYLOCK(OTHER_LOCK, &log_lock) != 0) { ^ ~~ That's because this expands to this: while (!pl_try_s(&log_lock) != 0) { Let's just add parenthesis in the TRYLOCK macros to avoid this. This may need to be backported if commit df187875d ("BUG/MEDIUM: log: don't hold the log lock during writev() on a file descriptor") is backported as well as it seems to be the first one to trigger it.
2020-06-12 05:42:25 -04:00
#define __SPIN_TRYLOCK(l) (!pl_try_s(l))
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
#define __SPIN_UNLOCK(l) pl_drop_s(l)
#define __RWLOCK_INIT(l) ({ (*l) = 0; })
#define __RWLOCK_DESTROY(l) ({ (*l) = 0; })
#define __RWLOCK_WRLOCK(l) pl_take_w(l)
BUILD: thread: add parenthesis around values of locking macros clang just failed on fd.c with this error: src/fd.c:491:9: error: logical not is only applied to the left hand side of this comparison [-Werror,-Wlogical-not-parentheses] while (HA_SPIN_TRYLOCK(OTHER_LOCK, &log_lock) != 0) { ^ ~~ That's because this expands to this: while (!pl_try_s(&log_lock) != 0) { Let's just add parenthesis in the TRYLOCK macros to avoid this. This may need to be backported if commit df187875d ("BUG/MEDIUM: log: don't hold the log lock during writev() on a file descriptor") is backported as well as it seems to be the first one to trigger it.
2020-06-12 05:42:25 -04:00
#define __RWLOCK_TRYWRLOCK(l) (!pl_try_w(l))
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
#define __RWLOCK_WRUNLOCK(l) pl_drop_w(l)
#define __RWLOCK_RDLOCK(l) pl_take_r(l)
BUILD: thread: add parenthesis around values of locking macros clang just failed on fd.c with this error: src/fd.c:491:9: error: logical not is only applied to the left hand side of this comparison [-Werror,-Wlogical-not-parentheses] while (HA_SPIN_TRYLOCK(OTHER_LOCK, &log_lock) != 0) { ^ ~~ That's because this expands to this: while (!pl_try_s(&log_lock) != 0) { Let's just add parenthesis in the TRYLOCK macros to avoid this. This may need to be backported if commit df187875d ("BUG/MEDIUM: log: don't hold the log lock during writev() on a file descriptor") is backported as well as it seems to be the first one to trigger it.
2020-06-12 05:42:25 -04:00
#define __RWLOCK_TRYRDLOCK(l) (!pl_try_r(l))
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
#define __RWLOCK_RDUNLOCK(l) pl_drop_r(l)
MINOR: threads: add the transitions to/from the seek state Since our locks are based on progressive locks, we support the upgradable seek lock that is compatible with readers and upgradable to a write lock. The main purpose is to take it while seeking down a tree for modification while other threads may seek the same tree for an input (e.g. compute the next event date). The newly supported operations are: HA_RWLOCK_SKLOCK(lbl,l) pl_take_s(l) /* N --> S */ HA_RWLOCK_SKTOWR(lbl,l) pl_stow(l) /* S --> W */ HA_RWLOCK_WRTOSK(lbl,l) pl_wtos(l) /* W --> S */ HA_RWLOCK_SKTORD(lbl,l) pl_stor(l) /* S --> R */ HA_RWLOCK_WRTORD(lbl,l) pl_wtor(l) /* W --> R */ HA_RWLOCK_SKUNLOCK(lbl,l) pl_drop_s(l) /* S --> N */ HA_RWLOCK_TRYSKLOCK(lbl,l) (!pl_try_s(l)) /* N -?> S */ HA_RWLOCK_TRYRDTOSK(lbl,l) (!pl_try_rtos(l)) /* R -?> S */ Existing code paths are left unaffected so this patch doesn't affect any running code.
2020-10-16 10:53:46 -04:00
/* rwlock upgrades via seek locks */
#define __RWLOCK_SKLOCK(l) pl_take_s(l) /* N --> S */
#define __RWLOCK_SKTOWR(l) pl_stow(l) /* S --> W */
#define __RWLOCK_WRTOSK(l) pl_wtos(l) /* W --> S */
#define __RWLOCK_SKTORD(l) pl_stor(l) /* S --> R */
#define __RWLOCK_WRTORD(l) pl_wtor(l) /* W --> R */
#define __RWLOCK_SKUNLOCK(l) pl_drop_s(l) /* S --> N */
#define __RWLOCK_TRYSKLOCK(l) (!pl_try_s(l)) /* N -?> S */
#define __RWLOCK_TRYRDTOSK(l) (!pl_try_rtos(l)) /* R -?> S */
MINOR: threads: Add HA_RWLOCK_TRYRDTOWR() Add HA_RWLOCK_TRYRDTOWR(), that tries to upgrade a lock from reader to writer, and fails if any seeker or writer already holds it.
2025-03-26 08:11:30 -04:00
#define __RWLOCK_TRYRDTOWR(l) (!pl_try_rtow(l)) /* R -?> W */
MINOR: threads: add the transitions to/from the seek state Since our locks are based on progressive locks, we support the upgradable seek lock that is compatible with readers and upgradable to a write lock. The main purpose is to take it while seeking down a tree for modification while other threads may seek the same tree for an input (e.g. compute the next event date). The newly supported operations are: HA_RWLOCK_SKLOCK(lbl,l) pl_take_s(l) /* N --> S */ HA_RWLOCK_SKTOWR(lbl,l) pl_stow(l) /* S --> W */ HA_RWLOCK_WRTOSK(lbl,l) pl_wtos(l) /* W --> S */ HA_RWLOCK_SKTORD(lbl,l) pl_stor(l) /* S --> R */ HA_RWLOCK_WRTORD(lbl,l) pl_wtor(l) /* W --> R */ HA_RWLOCK_SKUNLOCK(lbl,l) pl_drop_s(l) /* S --> N */ HA_RWLOCK_TRYSKLOCK(lbl,l) (!pl_try_s(l)) /* N -?> S */ HA_RWLOCK_TRYRDTOSK(lbl,l) (!pl_try_rtos(l)) /* R -?> S */ Existing code paths are left unaffected so this patch doesn't affect any running code.
2020-10-16 10:53:46 -04:00
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
#define HA_SPIN_INIT(l) __spin_init(l)
#define HA_SPIN_DESTROY(l) __spin_destroy(l)
MINOR: thread: add a lock level information in the thread_ctx The new lock_level field indicates the number of cumulated locks that are held by the current thread. It's fed as soon as DEBUG_THREAD is at least 1. In addition, thread_isolate() adds 128, so that it's even possible to check for combinations of both. The value is also reported in thread dumps (warnings and panics).
2025-09-11 02:29:25 -04:00
#define HA_SPIN_LOCK(lbl, l) _lock_wait(_LK_SK, 1, lbl, __spin_lock(lbl, l, __func__, __FILE__, __LINE__))
#define HA_SPIN_TRYLOCK(lbl, l) _lock_cond(_LK_SK, 1, lbl, __spin_trylock(lbl, l, __func__, __FILE__, __LINE__))
#define HA_SPIN_UNLOCK(lbl, l) _lock_wait(_LK_UN, -1, lbl, __spin_unlock(lbl, l, __func__, __FILE__, __LINE__))
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
#define HA_RWLOCK_INIT(l) __ha_rwlock_init((l))
#define HA_RWLOCK_DESTROY(l) __ha_rwlock_destroy((l))
MINOR: thread: add a lock level information in the thread_ctx The new lock_level field indicates the number of cumulated locks that are held by the current thread. It's fed as soon as DEBUG_THREAD is at least 1. In addition, thread_isolate() adds 128, so that it's even possible to check for combinations of both. The value is also reported in thread dumps (warnings and panics).
2025-09-11 02:29:25 -04:00
#define HA_RWLOCK_WRLOCK(lbl,l) _lock_wait(_LK_WR, 1, lbl, __ha_rwlock_wrlock(lbl, l, __func__, __FILE__, __LINE__))
#define HA_RWLOCK_TRYWRLOCK(lbl,l) _lock_cond(_LK_WR, 1, lbl, __ha_rwlock_trywrlock(lbl, l, __func__, __FILE__, __LINE__))
#define HA_RWLOCK_WRUNLOCK(lbl,l) _lock_wait(_LK_UN, -1, lbl, __ha_rwlock_wrunlock(lbl, l, __func__, __FILE__, __LINE__))
#define HA_RWLOCK_RDLOCK(lbl,l) _lock_wait(_LK_RD, 1, lbl, __ha_rwlock_rdlock(lbl, l))
#define HA_RWLOCK_TRYRDLOCK(lbl,l) _lock_cond(_LK_RD, 1, lbl, __ha_rwlock_tryrdlock(lbl, l))
#define HA_RWLOCK_RDUNLOCK(lbl,l) _lock_wait(_LK_UN, -1, lbl, __ha_rwlock_rdunlock(lbl, l))
#define HA_RWLOCK_SKLOCK(lbl,l) _lock_wait(_LK_SK, 1, lbl, __ha_rwlock_sklock(lbl, l, __func__, __FILE__, __LINE__))
#define HA_RWLOCK_SKTOWR(lbl,l) _lock_wait(_LK_WR, 0, lbl, __ha_rwlock_sktowr(lbl, l, __func__, __FILE__, __LINE__))
#define HA_RWLOCK_WRTOSK(lbl,l) _lock_wait(_LK_SK, 0, lbl, __ha_rwlock_wrtosk(lbl, l, __func__, __FILE__, __LINE__))
#define HA_RWLOCK_SKTORD(lbl,l) _lock_wait(_LK_RD, 0, lbl, __ha_rwlock_sktord(lbl, l, __func__, __FILE__, __LINE__))
#define HA_RWLOCK_WRTORD(lbl,l) _lock_wait(_LK_RD, 0, lbl, __ha_rwlock_wrtord(lbl, l, __func__, __FILE__, __LINE__))
#define HA_RWLOCK_SKUNLOCK(lbl,l) _lock_wait(_LK_UN, -1, lbl, __ha_rwlock_skunlock(lbl, l, __func__, __FILE__, __LINE__))
#define HA_RWLOCK_TRYSKLOCK(lbl,l) _lock_cond(_LK_SK, 1, lbl, __ha_rwlock_trysklock(lbl, l, __func__, __FILE__, __LINE__))
#define HA_RWLOCK_TRYRDTOSK(lbl,l) _lock_cond(_LK_RD, 0, lbl, __ha_rwlock_tryrdtosk(lbl, l, __func__, __FILE__, __LINE__))
#define HA_RWLOCK_TRYRDTOWR(lbl,l) _lock_cond(_LK_WR, 0, lbl, __ha_rwlock_tryrdtowr(lbl, l, __func__, __FILE__, __LINE__))
MINOR: threads: add the transitions to/from the seek state Since our locks are based on progressive locks, we support the upgradable seek lock that is compatible with readers and upgradable to a write lock. The main purpose is to take it while seeking down a tree for modification while other threads may seek the same tree for an input (e.g. compute the next event date). The newly supported operations are: HA_RWLOCK_SKLOCK(lbl,l) pl_take_s(l) /* N --> S */ HA_RWLOCK_SKTOWR(lbl,l) pl_stow(l) /* S --> W */ HA_RWLOCK_WRTOSK(lbl,l) pl_wtos(l) /* W --> S */ HA_RWLOCK_SKTORD(lbl,l) pl_stor(l) /* S --> R */ HA_RWLOCK_WRTORD(lbl,l) pl_wtor(l) /* W --> R */ HA_RWLOCK_SKUNLOCK(lbl,l) pl_drop_s(l) /* S --> N */ HA_RWLOCK_TRYSKLOCK(lbl,l) (!pl_try_s(l)) /* N -?> S */ HA_RWLOCK_TRYRDTOSK(lbl,l) (!pl_try_rtos(l)) /* R -?> S */ Existing code paths are left unaffected so this patch doesn't affect any running code.
2020-10-16 10:53:46 -04:00
MEDIUM: threads: Add hathreads header file This file contains all functions and macros used to deal with concurrency in HAProxy. It contains all high-level function to do atomic operation (HA_ATOMIC_*). Note, for now, we rely on "__atomic" GCC builtins to do atomic operation. So HAProxy can be compiled with the thread support iff these builtins are available. It also contains wrappers around plocks to use spin or read/write locks. These wrappers are used to abstract the internal representation of the locking system and to add information to help debugging, when compiled with suitable options. To add extra info on locks, you need to add DEBUG=-DDEBUG_THREAD or DEBUG=-DDEBUG_FULL compilation option. In addition to timing info on locks, we keep info on where a lock was acquired the last time (function name, file and line). There are also the thread id and a flag to know if it is still locked or not. This will be useful to debug deadlocks.
2017-10-12 10:09:09 -04:00
/* Following functions are used to collect some stats about locks. We wrap
* pthread functions to known how much time we wait in a lock. */
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
void show_lock_stats();
void __ha_rwlock_init(struct ha_rwlock *l);
void __ha_rwlock_destroy(struct ha_rwlock *l);
void __ha_rwlock_wrlock(enum lock_label lbl, struct ha_rwlock *l,
const char *func, const char *file, int line);
int __ha_rwlock_trywrlock(enum lock_label lbl, struct ha_rwlock *l,
const char *func, const char *file, int line);
void __ha_rwlock_wrunlock(enum lock_label lbl,struct ha_rwlock *l,
const char *func, const char *file, int line);
void __ha_rwlock_rdlock(enum lock_label lbl,struct ha_rwlock *l);
int __ha_rwlock_tryrdlock(enum lock_label lbl,struct ha_rwlock *l);
void __ha_rwlock_rdunlock(enum lock_label lbl,struct ha_rwlock *l);
void __ha_rwlock_wrtord(enum lock_label lbl, struct ha_rwlock *l,
const char *func, const char *file, int line);
void __ha_rwlock_wrtosk(enum lock_label lbl, struct ha_rwlock *l,
const char *func, const char *file, int line);
void __ha_rwlock_sklock(enum lock_label lbl, struct ha_rwlock *l,
const char *func, const char *file, int line);
void __ha_rwlock_sktowr(enum lock_label lbl, struct ha_rwlock *l,
const char *func, const char *file, int line);
void __ha_rwlock_sktord(enum lock_label lbl, struct ha_rwlock *l,
const char *func, const char *file, int line);
void __ha_rwlock_skunlock(enum lock_label lbl,struct ha_rwlock *l,
const char *func, const char *file, int line);
int __ha_rwlock_trysklock(enum lock_label lbl, struct ha_rwlock *l,
const char *func, const char *file, int line);
int __ha_rwlock_tryrdtosk(enum lock_label lbl, struct ha_rwlock *l,
const char *func, const char *file, int line);
MINOR: threads: Add HA_RWLOCK_TRYRDTOWR() Add HA_RWLOCK_TRYRDTOWR(), that tries to upgrade a lock from reader to writer, and fails if any seeker or writer already holds it.
2025-03-26 08:11:30 -04:00
int __ha_rwlock_tryrdtowr(enum lock_label lbl, struct ha_rwlock *l,
const char *func, const char *file, int line);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
void __spin_init(struct ha_spinlock *l);
void __spin_destroy(struct ha_spinlock *l);
void __spin_lock(enum lock_label lbl, struct ha_spinlock *l,
const char *func, const char *file, int line);
int __spin_trylock(enum lock_label lbl, struct ha_spinlock *l,
const char *func, const char *file, int line);
void __spin_unlock(enum lock_label lbl, struct ha_spinlock *l,
const char *func, const char *file, int line);
MEDIUM: threads: Add hathreads header file This file contains all functions and macros used to deal with concurrency in HAProxy. It contains all high-level function to do atomic operation (HA_ATOMIC_*). Note, for now, we rely on "__atomic" GCC builtins to do atomic operation. So HAProxy can be compiled with the thread support iff these builtins are available. It also contains wrappers around plocks to use spin or read/write locks. These wrappers are used to abstract the internal representation of the locking system and to add information to help debugging, when compiled with suitable options. To add extra info on locks, you need to add DEBUG=-DDEBUG_THREAD or DEBUG=-DDEBUG_FULL compilation option. In addition to timing info on locks, we keep info on where a lock was acquired the last time (function name, file and line). There are also the thread id and a flag to know if it is still locked or not. This will be useful to debug deadlocks.
2017-10-12 10:09:09 -04:00
#endif /* DEBUG_THREAD */
#endif /* USE_THREAD */
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
#endif /* _HAPROXY_THREAD_H */
Reference in a new issue Copy permalink