upstream/haproxy
1
0
Fork 0
mirror of https://github.com/haproxy/haproxy.git synced 2026-02-17 01:28:32 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions 10
haproxy/src/thread.c

1769 lines
54 KiB
C
Raw Normal View History

MEDIUM: threads: Add hathreads header file This file contains all functions and macros used to deal with concurrency in HAProxy. It contains all high-level function to do atomic operation (HA_ATOMIC_*). Note, for now, we rely on "__atomic" GCC builtins to do atomic operation. So HAProxy can be compiled with the thread support iff these builtins are available. It also contains wrappers around plocks to use spin or read/write locks. These wrappers are used to abstract the internal representation of the locking system and to add information to help debugging, when compiled with suitable options. To add extra info on locks, you need to add DEBUG=-DDEBUG_THREAD or DEBUG=-DDEBUG_FULL compilation option. In addition to timing info on locks, we keep info on where a lock was acquired the last time (function name, file and line). There are also the thread id and a flag to know if it is still locked or not. This will be useful to debug deadlocks.
2017-10-12 10:09:09 -04:00
/*
* functions about threads.
*
* Copyright (C) 2017 Christopher Fauet - cfaulet@haproxy.com
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version
* 2 of the License, or (at your option) any later version.
*
*/
MAJOR: threads: enable one thread per CPU by default Threads have long matured by now, still for most users their usage is not trivial. It's about time to enable them by default on platforms where we know the number of CPUs bound. This patch does this, it counts the number of CPUs the process is bound to upon startup, and enables as many threads by default. Of course, "nbthread" still overrides this, but if it's not set the default behaviour is to start one thread per CPU. The default number of threads is reported in "haproxy -vv". Simply using "taskset -c" is now enough to adjust this number of threads so that there is no more need for playing with cpu-map. And thanks to the previous patches on the listener, the vast majority of configurations will not need to duplicate "bind" lines with the "process x/y" statement anymore either, so a simple config will automatically adapt to the number of processors available.
2019-01-26 08:27:06 -05:00
#define _GNU_SOURCE
MEDIUM: threads: Adds a set of functions to handle sync-point A sync-point is a protected area where you have the warranty that no concurrency access is possible. It is implementated as a thread barrier to enter in the sync-point and another one to exit from it. Inside the sync-point, all threads that must do some syncrhonous processing will be called one after the other while all other threads will wait. All threads will then exit from the sync-point at the same time. A sync-point will be evaluated only when necessary because it is a costly operation. To limit the waiting time of each threads, we must have a mechanism to wakeup all threads. This is done with a pipe shared by all threads. By writting in this pipe, we will interrupt all threads blocked on a poller. The pipe is then flushed before exiting from the sync-point.
2017-10-19 05:59:15 -04:00
#include <unistd.h>
MINOR: threads: move "nbthread" parsing to hathreads.c The purpose is to make sure that all variables which directly depend on this nbthread argument are set at the right moment. For now only all_threads_mask needs to be set. It used to be set while calling thread_sync_init() which is called too late for certain checks. The same function handles threads and non-threads, which removes the need for some thread-specific knowledge from cfgparse.c.
2018-07-30 04:34:35 -04:00
#include <stdlib.h>
MEDIUM: threads: Adds a set of functions to handle sync-point A sync-point is a protected area where you have the warranty that no concurrency access is possible. It is implementated as a thread barrier to enter in the sync-point and another one to exit from it. Inside the sync-point, all threads that must do some syncrhonous processing will be called one after the other while all other threads will wait. All threads will then exit from the sync-point at the same time. A sync-point will be evaluated only when necessary because it is a costly operation. To limit the waiting time of each threads, we must have a mechanism to wakeup all threads. This is done with a pipe shared by all threads. By writting in this pipe, we will interrupt all threads blocked on a poller. The pipe is then flushed before exiting from the sync-point.
2017-10-19 05:59:15 -04:00
CLEANUP: thread: uninline ha_tkill/ha_tkillall/ha_cpu_relax() These ones are rarely used or only to waste CPU cycles waiting, and are the last ones requiring system includes in thread.h. Let's uninline them and move them to thread.c.
2021-10-06 17:33:20 -04:00
#include <signal.h>
#include <unistd.h>
#ifdef _POSIX_PRIORITY_SCHEDULING
#include <sched.h>
#endif
MINOR: thread: use a dedicated static pthread_t array in thread.c This removes the thread identifiers from struct thread_info and moves them only in static array in thread.c since it's now the only file that needs to touch it. It's also the only file that needs to include pthread.h, beyond haproxy.c which needs it to start the poll loop. As a result, much less system includes are needed and the LoC reduced by around 3%.
2021-10-06 16:53:51 -04:00
#ifdef USE_THREAD
# include <pthread.h>
#endif
MAJOR: threads: enable one thread per CPU by default Threads have long matured by now, still for most users their usage is not trivial. It's about time to enable them by default on platforms where we know the number of CPUs bound. This patch does this, it counts the number of CPUs the process is bound to upon startup, and enables as many threads by default. Of course, "nbthread" still overrides this, but if it's not set the default behaviour is to start one thread per CPU. The default number of threads is reported in "haproxy -vv". Simply using "taskset -c" is now enough to adjust this number of threads so that there is no more need for playing with cpu-map. And thanks to the previous patches on the listener, the vast majority of configurations will not need to duplicate "bind" lines with the "process x/y" statement anymore either, so a simple config will automatically adapt to the number of processors available.
2019-01-26 08:27:06 -05:00
#ifdef USE_CPU_AFFINITY
REORG: thread: move the thread init/affinity/stop to thread.c haproxy.c still has to deal with pthread-specific low-level stuff that is OS-dependent. We should not have to deal with this there, and we do not need to access pthread anywhere else. Let's move these 3 functions to thread.c and keep empty inline ones for when threads are disabled.
2021-10-06 16:22:40 -04:00
# include <sched.h>
# if defined(__FreeBSD__) || defined(__DragonFly__)
# include <sys/param.h>
# ifdef __FreeBSD__
# include <sys/cpuset.h>
# endif
# include <pthread_np.h>
# endif
# ifdef __APPLE__
# include <mach/mach_types.h>
# include <mach/thread_act.h>
# include <mach/thread_policy.h>
# endif
# include <haproxy/cpuset.h>
MINOR: threads: Implement thread_cpus_enabled() for FreeBSD. Use cpuset_getaffinity() to implement thread_cpus_enabled() on FreeBSD, so that we can know the number of CPUs available, and automatically launch as much threads if nbthread isn't specified.
2019-04-10 18:06:47 -04:00
#endif
REORG: include: move cfgparse.h to haproxy/cfgparse.h There's no point splitting the file in two since only cfgparse uses the types defined there. A few call places were updated and cleaned up. All of them were in C files which register keywords. There is nothing left in common/ now so this directory must not be used anymore.
2020-06-04 18:00:29 -04:00
#include <haproxy/cfgparse.h>
REORG: time: move time-keeping code and variables to clock.c There is currently a problem related to time keeping. We're mixing the functions to perform calculations with the os-dependent code needed to retrieve and adjust the local time. This patch extracts from time.{c,h} the parts that are solely dedicated to time keeping. These are the "now" or "before_poll" variables for example, as well as the various now_*() functions that make use of gettimeofday() and clock_gettime() to retrieve the current time. The "tv_*" functions moved there were also more appropriately renamed to "clock_*". Other parts used to compute stolen time are in other files, they will have to be picked next.
2021-10-08 03:33:24 -04:00
#include <haproxy/clock.h>
CLEANUP: include: tree-wide alphabetical sort of include files This patch fixes all the leftovers from the include cleanup campaign. There were not that many (~400 entries in ~150 files) but it was definitely worth doing it as it revealed a few duplicates.
2020-06-09 03:07:15 -04:00
#include <haproxy/fd.h>
#include <haproxy/global.h>
BUILD: thread: include log.h from thread.c It's needed for ha_alert(). Probably that a separate file for error reporting at boot would be useful.
2021-05-08 14:33:02 -04:00
#include <haproxy/log.h>
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
#include <haproxy/thread.h>
REORG: tools: split common/standard.h into haproxy/tools{,-t}.h And also rename standard.c to tools.c. The original split between tools.h and standard.h dates from version 1.3-dev and was mostly an accident. This patch moves the files back to what they were expected to be, and takes care of not changing anything else. However this time tools.h was split between functions and types, because it contains a small number of commonly used macros and structures (e.g. name_desc) which in turn cause the massive list of includes of tools.h to conflict with the callers. They remain the ugliest files of the whole project and definitely need to be cleaned and split apart. A few types are defined there only for functions provided there, and some parts are even OS-specific and should move somewhere else, such as the symbol resolution code.
2020-06-03 12:09:46 -04:00
#include <haproxy/tools.h>
MEDIUM: threads: Add hathreads header file This file contains all functions and macros used to deal with concurrency in HAProxy. It contains all high-level function to do atomic operation (HA_ATOMIC_*). Note, for now, we rely on "__atomic" GCC builtins to do atomic operation. So HAProxy can be compiled with the thread support iff these builtins are available. It also contains wrappers around plocks to use spin or read/write locks. These wrappers are used to abstract the internal representation of the locking system and to add information to help debugging, when compiled with suitable options. To add extra info on locks, you need to add DEBUG=-DDEBUG_THREAD or DEBUG=-DDEBUG_FULL compilation option. In addition to timing info on locks, we keep info on where a lock was acquired the last time (function name, file and line). There are also the thread id and a flag to know if it is still locked or not. This will be useful to debug deadlocks.
2017-10-12 10:09:09 -04:00
MINOR: threads: introduce a minimalistic notion of thread-group This creates a struct tgroup_info which knows the thread ID of the first thread in a group, and the number of threads in it. For now there's only one thread group supported in the configuration, but it may be forced to other values for development purposes by defining MAX_TGROUPS, and it's enabled even when threads are disabled and will need to remain accessible during boot to keep a simple enough internal API. For the purpose of easing the configurations which do not specify a thread group, we're starting group numbering at 1 so that thread group 0 can be "undefined" (i.e. for "bind" lines or when binding tasks). The goal will be to later move there some global items that must be made per-group.
2021-09-13 12:11:26 -04:00
struct tgroup_info ha_tgroup_info[MAX_TGROUPS] = { };
THREAD_LOCAL const struct tgroup_info *tg = &ha_tgroup_info[0];
BUILD/MEDIUM: threads: rename thread_info struct to ha_thread_info On Darwin, the thread_info name exists as a standard function thus we need to rename our array to ha_thread_info to fix this conflict.
2019-09-13 00:03:12 -04:00
struct thread_info ha_thread_info[MAX_THREADS] = { };
MINOR: thread: make "ti" a const pointer and clean up thread_info a bit We want to make sure that the current thread_info accessed via "ti" will remain constant, so that we don't accidentally place new variable parts there and so that the compiler knows that info retrieved from there is not expected to have changed between two function calls. Only a few init locations had to be adjusted to use the array and the rest is unaffected.
2021-10-01 10:29:27 -04:00
THREAD_LOCAL const struct thread_info *ti = &ha_thread_info[0];
MEDIUM: threads: Add hathreads header file This file contains all functions and macros used to deal with concurrency in HAProxy. It contains all high-level function to do atomic operation (HA_ATOMIC_*). Note, for now, we rely on "__atomic" GCC builtins to do atomic operation. So HAProxy can be compiled with the thread support iff these builtins are available. It also contains wrappers around plocks to use spin or read/write locks. These wrappers are used to abstract the internal representation of the locking system and to add information to help debugging, when compiled with suitable options. To add extra info on locks, you need to add DEBUG=-DDEBUG_THREAD or DEBUG=-DDEBUG_FULL compilation option. In addition to timing info on locks, we keep info on where a lock was acquired the last time (function name, file and line). There are also the thread id and a flag to know if it is still locked or not. This will be useful to debug deadlocks.
2017-10-12 10:09:09 -04:00
MEDIUM: tinfo: add a dynamic thread-group context The thread group info is not sufficient to represent a thread group's current state as it's read-only. We also need something comparable to the thread context to represent the aggregate state of the threads in that group. This patch introduces ha_tgroup_ctx[] and tg_ctx for this. It's indexed on the group id and must be cache-line aligned. The thread masks that were global and that do not need to remain global were moved there (want_rdv, harmless, idle). Given that all the masks placed there now become group-specific, the associated thread mask (tid_bit) now switches to the thread's local bit (ltid_bit). Both are the same for nbtgroups 1 but will differ for other values. There's also a tg_ctx pointer in the thread so that it can be reached from other threads.
2022-06-27 10:02:24 -04:00
struct tgroup_ctx ha_tgroup_ctx[MAX_TGROUPS] = { };
THREAD_LOCAL struct tgroup_ctx *tg_ctx = &ha_tgroup_ctx[0];
REORG: thread/sched: move the task_per_thread stuff to thread_ctx The scheduler contains a lot of stuff that is thread-local and not exclusively tied to the scheduler. Other parts (namely thread_info) contain similar thread-local context that ought to be merged with it but that is even less related to the scheduler. However moving more data into this structure isn't possible since task.h is high level and cannot be included everywhere (e.g. activity) without causing include loops. In the end, it appears that the task_per_thread represents most of the per-thread context defined with generic types and should simply move to tinfo.h so that everyone can use them. The struct was renamed to thread_ctx and the variable "sched" was renamed to "th_ctx". "sched" used to be initialized manually from run_thread_poll_loop(), now it's initialized by ha_set_tid() just like ti, tid, tid_bit. The memset() in init_task() was removed in favor of a bss initialization of the array, so that other subsystems can put their stuff in this array. Since the tasklet array has TL_CLASSES elements, the TL_* definitions was moved there as well, but it's not a problem. The vast majority of the change in this patch is caused by the renaming of the structures.
2021-10-01 05:30:33 -04:00
struct thread_ctx ha_thread_ctx[MAX_THREADS] = { };
THREAD_LOCAL struct thread_ctx *th_ctx = &ha_thread_ctx[0];
MEDIUM: threads: Add hathreads header file This file contains all functions and macros used to deal with concurrency in HAProxy. It contains all high-level function to do atomic operation (HA_ATOMIC_*). Note, for now, we rely on "__atomic" GCC builtins to do atomic operation. So HAProxy can be compiled with the thread support iff these builtins are available. It also contains wrappers around plocks to use spin or read/write locks. These wrappers are used to abstract the internal representation of the locking system and to add information to help debugging, when compiled with suitable options. To add extra info on locks, you need to add DEBUG=-DDEBUG_THREAD or DEBUG=-DDEBUG_FULL compilation option. In addition to timing info on locks, we keep info on where a lock was acquired the last time (function name, file and line). There are also the thread id and a flag to know if it is still locked or not. This will be useful to debug deadlocks.
2017-10-12 10:09:09 -04:00
#ifdef USE_THREAD
MINOR: thread: add a new all_tgroups_mask variable to know about active tgroups In order to kill all_threads_mask we'll need to have an equivalent for the thread groups. The all_tgroups_mask does just this, it keeps one bit set per enabled group.
2022-06-24 09:55:11 -04:00
volatile unsigned long all_tgroups_mask __read_mostly = 1; // nbtgroup 1 assumed by default
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
volatile unsigned int rdv_requests = 0; // total number of threads requesting RDV
volatile unsigned int isolated_thread = ~0; // ID of the isolated thread, or ~0 when none
MINOR: threads: add the current group ID in thread-local "tgid" variable This is the equivalent of "tid" for ease of access. In the future if we make th_cfg a pure thread-local array (not a pointer), it may make sense to move it there.
2021-09-30 02:00:11 -04:00
THREAD_LOCAL unsigned int tgid = 1; // thread ID starts at 1
MINOR: threads: add more consistency between certain variables in no-thread case When threads are disabled, some variables such as tid and tid_bit are still checked everywhere, the MAX_THREADS_MASK macro is ~0UL while MAX_THREADS is 1, and the all_threads_mask variable is replaced with a macro forced to zero. The compiler cannot optimize away all this code involving checks on tid and tid_bit, and we end up in special cases where all_threads_mask has to be specifically tested for being zero or not. It is not even certain the code paths are always equivalent when testing without threads and with nbthread 1. Let's change this to make sure we always present a single thread when threads are disabled, and have the relevant values declared as constants so that the compiler can optimize all the tests away. Now we have MAX_THREADS_MASK set to 1, all_threads_mask set to 1, tid set to zero and tid_bit set to 1. Doing just this has removed 4 kB of code in the no-thread case. A few checks for all_threads_mask==0 have been removed since it never happens anymore.
2018-08-01 13:12:20 -04:00
THREAD_LOCAL unsigned int tid = 0;
MAJOR: threads: enable one thread per CPU by default Threads have long matured by now, still for most users their usage is not trivial. It's about time to enable them by default on platforms where we know the number of CPUs bound. This patch does this, it counts the number of CPUs the process is bound to upon startup, and enables as many threads by default. Of course, "nbthread" still overrides this, but if it's not set the default behaviour is to start one thread per CPU. The default number of threads is reported in "haproxy -vv". Simply using "taskset -c" is now enough to adjust this number of threads so that there is no more need for playing with cpu-map. And thanks to the previous patches on the listener, the vast majority of configurations will not need to duplicate "bind" lines with the "process x/y" statement anymore either, so a simple config will automatically adapt to the number of processors available.
2019-01-26 08:27:06 -05:00
int thread_cpus_enabled_at_boot = 1;
MINOR: thread: use a dedicated static pthread_t array in thread.c This removes the thread identifiers from struct thread_info and moves them only in static array in thread.c since it's now the only file that needs to touch it. It's also the only file that needs to include pthread.h, beyond haproxy.c which needs it to start the poll loop. As a result, much less system includes are needed and the LoC reduced by around 3%.
2021-10-06 16:53:51 -04:00
static pthread_t ha_pthread[MAX_THREADS] = { };
MINOR: threads: add more consistency between certain variables in no-thread case When threads are disabled, some variables such as tid and tid_bit are still checked everywhere, the MAX_THREADS_MASK macro is ~0UL while MAX_THREADS is 1, and the all_threads_mask variable is replaced with a macro forced to zero. The compiler cannot optimize away all this code involving checks on tid and tid_bit, and we end up in special cases where all_threads_mask has to be specifically tested for being zero or not. It is not even certain the code paths are always equivalent when testing without threads and with nbthread 1. Let's change this to make sure we always present a single thread when threads are disabled, and have the relevant values declared as constants so that the compiler can optimize all the tests away. Now we have MAX_THREADS_MASK set to 1, all_threads_mask set to 1, tid set to zero and tid_bit set to 1. Doing just this has removed 4 kB of code in the no-thread case. A few checks for all_threads_mask==0 have been removed since it never happens anymore.
2018-08-01 13:12:20 -04:00
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
/* Marks the thread as harmless until the last thread using the rendez-vous
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
* point quits. Given that we can wait for a long time, sched_yield() is
BUG/MEDIUM: thread: Fix a deadlock if an isolated thread is marked as harmless If an isolated thread is marked as harmless, it will loop forever in thread_harmless_till_end() waiting no threads are isolated anymore. It never happens because the current thread is isolated. To fix the bug, we exclude the current thread for the test. We now wait for all other threads to leave the rendez-vous point. This bug only seems to occurr if HAProxy is compiled with DEBUG_UAF, when pool_gc() is called. pool_gc() isolates the current thread, while pool_free_area() set the thread as harmless when munmap is called. This patch must be backported as far as 2.0.
2021-03-25 09:11:36 -04:00
* used when available to offer the CPU resources to competing threads if
* needed.
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
*/
void thread_harmless_till_end()
{
MEDIUM: tinfo: add a dynamic thread-group context The thread group info is not sufficient to represent a thread group's current state as it's read-only. We also need something comparable to the thread context to represent the aggregate state of the threads in that group. This patch introduces ha_tgroup_ctx[] and tg_ctx for this. It's indexed on the group id and must be cache-line aligned. The thread masks that were global and that do not need to remain global were moved there (want_rdv, harmless, idle). Given that all the masks placed there now become group-specific, the associated thread mask (tid_bit) now switches to the thread's local bit (ltid_bit). Both are the same for nbtgroups 1 but will differ for other values. There's also a tg_ctx pointer in the thread so that it can be reached from other threads.
2022-06-27 10:02:24 -04:00
_HA_ATOMIC_OR(&tg_ctx->threads_harmless, ti->ltid_bit);
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
while (_HA_ATOMIC_LOAD(&rdv_requests) != 0) {
CLEANUP: thread: fix fantaisist indentation of thread_harmless_till_end() Probably due to a copy-paste, there were two indent levels in this function since its introduction in 1.9 by commit 60b639ccb ("MEDIUM: hathreads: implement a more flexible rendez-vous point"). Let's fix this.
2021-08-04 04:33:57 -04:00
ha_thread_relax();
}
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
}
/* Isolates the current thread : request the ability to work while all other
MINOR: threads: make thread_release() not wait for other ones to complete The original intent of making thread_release() wait for other requesters to proceed was more of a fairness trade, guaranteeing that a thread that was granted an access to the CPU would be in turn giving back once its job is done. But this is counter-productive as it forces such threads to spin instead of going back to the poller, and it prevents us from implementing multiple levels of guarantees, as a thread_release() call could spin waiting for another requester to pass while that requester expects stronger guarantees than the current thread may be able to offer. Let's just remove that wait period and let the thread go back to the poller, a-la "race to idle". While in theory it could possibly slightly increase the perceived latency of concurrent slow operations like "show fd" or "show sess", it is not the case at all in tests, probably because the time needed to reach the poller remains extremely low anyway.
2021-08-04 05:22:07 -04:00
* threads are harmless, as defined by thread_harmless_now() (i.e. they're not
* going to touch any visible memory area). Only returns once all of them are
MEDIUM: tinfo: add a dynamic thread-group context The thread group info is not sufficient to represent a thread group's current state as it's read-only. We also need something comparable to the thread context to represent the aggregate state of the threads in that group. This patch introduces ha_tgroup_ctx[] and tg_ctx for this. It's indexed on the group id and must be cache-line aligned. The thread masks that were global and that do not need to remain global were moved there (want_rdv, harmless, idle). Given that all the masks placed there now become group-specific, the associated thread mask (tid_bit) now switches to the thread's local bit (ltid_bit). Both are the same for nbtgroups 1 but will differ for other values. There's also a tg_ctx pointer in the thread so that it can be reached from other threads.
2022-06-27 10:02:24 -04:00
* harmless, with the current thread's bit in &tg_ctx->threads_harmless cleared.
MINOR: threads: make thread_release() not wait for other ones to complete The original intent of making thread_release() wait for other requesters to proceed was more of a fairness trade, guaranteeing that a thread that was granted an access to the CPU would be in turn giving back once its job is done. But this is counter-productive as it forces such threads to spin instead of going back to the poller, and it prevents us from implementing multiple levels of guarantees, as a thread_release() call could spin waiting for another requester to pass while that requester expects stronger guarantees than the current thread may be able to offer. Let's just remove that wait period and let the thread go back to the poller, a-la "race to idle". While in theory it could possibly slightly increase the perceived latency of concurrent slow operations like "show fd" or "show sess", it is not the case at all in tests, probably because the time needed to reach the poller remains extremely low anyway.
2021-08-04 05:22:07 -04:00
* Needs to be completed using thread_release().
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
*/
void thread_isolate()
{
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
uint tgrp, thr;
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
MEDIUM: tinfo: add a dynamic thread-group context The thread group info is not sufficient to represent a thread group's current state as it's read-only. We also need something comparable to the thread context to represent the aggregate state of the threads in that group. This patch introduces ha_tgroup_ctx[] and tg_ctx for this. It's indexed on the group id and must be cache-line aligned. The thread masks that were global and that do not need to remain global were moved there (want_rdv, harmless, idle). Given that all the masks placed there now become group-specific, the associated thread mask (tid_bit) now switches to the thread's local bit (ltid_bit). Both are the same for nbtgroups 1 but will differ for other values. There's also a tg_ctx pointer in the thread so that it can be reached from other threads.
2022-06-27 10:02:24 -04:00
_HA_ATOMIC_OR(&tg_ctx->threads_harmless, ti->ltid_bit);
MEDIUM: threads: Use the new _HA_ATOMIC_* macros. Use the new _HA_ATOMIC_* macros and add barriers where needed.
2019-03-08 12:51:17 -05:00
__ha_barrier_atomic_store();
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
_HA_ATOMIC_INC(&rdv_requests);
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
/* wait for all threads to become harmless. They cannot change their
* mind once seen thanks to rdv_requests above, unless they pass in
BUG/MEDIUM: threads: fix a tiny race in thread_isolate() Aurlien found a tiny race in thread_isolate() that can allow a thread that was running under isolation to continue running while another one enters isolation. The reason is that the check for harmless is only done before winning the CAS, but since the previously isolated thread doesn't wait for !rdv_request in thread_release(), it can effectively continue its activities while the next one believes it's isolated. A proper solution consists in looping once again in thread_isolate() to recheck (and wait) for all threads to be isolated once the CAS is won. The issue was introduced in 2.7 by commit 598cf3f22 ("MAJOR: threads: change thread_isolate to support inter-group synchronization") so the fix needs to be backported there.
2023-05-27 07:45:01 -04:00
* front of us. For this reason we proceed in 4 steps:
* 1) wait for all threads to declare themselves harmless
* 2) try to grab the isolated_thread exclusivity
* 3) verify again that all threads are harmless, since another one
* that was isolating between 1 and 2 could have dropped its
* harmless state there.
* 4) drop harmless flag (which also has the benefit of leaving
* all other threads wait on reads instead of writes.
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
*/
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
while (1) {
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
for (tgrp = 0; tgrp < global.nbtgroups; tgrp++) {
BUG/MINOR: thread: always reload threads_enabled in loops A few loops waiting for threads to synchronize such as thread_isolate() rightfully filter the thread masks via the threads_enabled field that contains the list of enabled threads. However, it doesn't use an atomic load on it. Before 2.7, the equivalent variables were marked as volatile and were always reloaded. In 2.7 they're fields in ha_tgroup_ctx[], and the risk that the compiler keeps them in a register inside a loop is not null at all. In practice when ha_thread_relax() calls sched_yield() or an x86 PAUSE instruction, it could be verified that the variable is always reloaded. If these are avoided (e.g. architecture providing neither solution), it's visible in asm code that the variables are not reloaded. In this case, if a thread exists just between the moment the two values are read, the loop could spin forever. This patch adds the required _HA_ATOMIC_LOAD() on the relevant threads_enabled fields. It must be backported to 2.7.
2023-01-19 13:14:18 -05:00
do {
ulong te = _HA_ATOMIC_LOAD(&ha_tgroup_info[tgrp].threads_enabled);
ulong th = _HA_ATOMIC_LOAD(&ha_tgroup_ctx[tgrp].threads_harmless);
if ((th & te) == te)
break;
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
ha_thread_relax();
BUG/MINOR: thread: always reload threads_enabled in loops A few loops waiting for threads to synchronize such as thread_isolate() rightfully filter the thread masks via the threads_enabled field that contains the list of enabled threads. However, it doesn't use an atomic load on it. Before 2.7, the equivalent variables were marked as volatile and were always reloaded. In 2.7 they're fields in ha_tgroup_ctx[], and the risk that the compiler keeps them in a register inside a loop is not null at all. In practice when ha_thread_relax() calls sched_yield() or an x86 PAUSE instruction, it could be verified that the variable is always reloaded. If these are avoided (e.g. architecture providing neither solution), it's visible in asm code that the variables are not reloaded. In this case, if a thread exists just between the moment the two values are read, the loop could spin forever. This patch adds the required _HA_ATOMIC_LOAD() on the relevant threads_enabled fields. It must be backported to 2.7.
2023-01-19 13:14:18 -05:00
} while (1);
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
}
BUG/MEDIUM: threads: fix a tiny race in thread_isolate() Aurlien found a tiny race in thread_isolate() that can allow a thread that was running under isolation to continue running while another one enters isolation. The reason is that the check for harmless is only done before winning the CAS, but since the previously isolated thread doesn't wait for !rdv_request in thread_release(), it can effectively continue its activities while the next one believes it's isolated. A proper solution consists in looping once again in thread_isolate() to recheck (and wait) for all threads to be isolated once the CAS is won. The issue was introduced in 2.7 by commit 598cf3f22 ("MAJOR: threads: change thread_isolate to support inter-group synchronization") so the fix needs to be backported there.
2023-05-27 07:45:01 -04:00
/* all other ones are harmless. isolated_thread will contain
* ~0U if no other one competes, !=tid if another one got it,
* tid if the current thread already grabbed it on the previous
* round.
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
*/
thr = _HA_ATOMIC_LOAD(&isolated_thread);
BUG/MEDIUM: threads: fix a tiny race in thread_isolate() Aurlien found a tiny race in thread_isolate() that can allow a thread that was running under isolation to continue running while another one enters isolation. The reason is that the check for harmless is only done before winning the CAS, but since the previously isolated thread doesn't wait for !rdv_request in thread_release(), it can effectively continue its activities while the next one believes it's isolated. A proper solution consists in looping once again in thread_isolate() to recheck (and wait) for all threads to be isolated once the CAS is won. The issue was introduced in 2.7 by commit 598cf3f22 ("MAJOR: threads: change thread_isolate to support inter-group synchronization") so the fix needs to be backported there.
2023-05-27 07:45:01 -04:00
if (thr == tid)
break; // we won and we're certain everyone is harmless
/* try to win the race against others */
if (thr != ~0U || !_HA_ATOMIC_CAS(&isolated_thread, &thr, tid))
ha_thread_relax();
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
}
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
/* the thread is no longer harmless as it runs */
_HA_ATOMIC_AND(&tg_ctx->threads_harmless, ~ti->ltid_bit);
/* the thread is isolated until it calls thread_release() which will
* 1) reset isolated_thread to ~0;
* 2) decrement rdv_requests.
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
*/
}
MEDIUM: threads: add a stronger thread_isolate_full() call The current principle of running under isolation was made to access sensitive data while being certain that no other thread was using them in parallel, without necessarily having to place locks everywhere. The main use case are "show sess" and "show fd" which run over long chains of pointers. The thread_isolate() call relies on the "harmless" bit that indicates for a given thread that it's not currently doing such sensitive things, which is advertised using thread_harmless_now() and which ends usings thread_harmless_end(), which also waits for possibly concurrent threads to complete their work if they took this opportunity for starting something tricky. As some system calls were notoriously slow (e.g. mmap()), a bunch of thread_harmless_now() / thread_harmless_end() were placed around them to let waiting threads do their work while such other threads were not able to modify memory contents. But this is not sufficient for performing memory modifications. One such example is the server deletion code. By modifying memory, it not only requires that other threads are not playing with it, but are not either in the process of touching it. The fact that a pool_alloc() or pool_free() on some structure may call thread_harmless_now() and let another thread start to release the same object's memory is not acceptable. This patch introduces the concept of "idle threads". Threads entering the polling loop are idle, as well as those that are waiting for all others to become idle via the new function thread_isolate_full(). Once thread_isolate_full() is granted, the thread is not idle anymore, and it is released using thread_release() just like regular isolation. Its users have to keep in mind that across this call nothing is granted as another thread might have performed shared memory modifications. But such users are extremely rare and are actually expecting this from their peers as well. Note that that in case of backport, this patch depends on previous patch: MINOR: threads: make thread_release() not wait for other ones to complete
2021-08-04 05:44:17 -04:00
/* Isolates the current thread : request the ability to work while all other
* threads are idle, as defined by thread_idle_now(). It only returns once
* all of them are both harmless and idle, with the current thread's bit in
MEDIUM: tinfo: add a dynamic thread-group context The thread group info is not sufficient to represent a thread group's current state as it's read-only. We also need something comparable to the thread context to represent the aggregate state of the threads in that group. This patch introduces ha_tgroup_ctx[] and tg_ctx for this. It's indexed on the group id and must be cache-line aligned. The thread masks that were global and that do not need to remain global were moved there (want_rdv, harmless, idle). Given that all the masks placed there now become group-specific, the associated thread mask (tid_bit) now switches to the thread's local bit (ltid_bit). Both are the same for nbtgroups 1 but will differ for other values. There's also a tg_ctx pointer in the thread so that it can be reached from other threads.
2022-06-27 10:02:24 -04:00
* &tg_ctx->threads_harmless and idle_mask cleared. Needs to be completed using
MEDIUM: threads: add a stronger thread_isolate_full() call The current principle of running under isolation was made to access sensitive data while being certain that no other thread was using them in parallel, without necessarily having to place locks everywhere. The main use case are "show sess" and "show fd" which run over long chains of pointers. The thread_isolate() call relies on the "harmless" bit that indicates for a given thread that it's not currently doing such sensitive things, which is advertised using thread_harmless_now() and which ends usings thread_harmless_end(), which also waits for possibly concurrent threads to complete their work if they took this opportunity for starting something tricky. As some system calls were notoriously slow (e.g. mmap()), a bunch of thread_harmless_now() / thread_harmless_end() were placed around them to let waiting threads do their work while such other threads were not able to modify memory contents. But this is not sufficient for performing memory modifications. One such example is the server deletion code. By modifying memory, it not only requires that other threads are not playing with it, but are not either in the process of touching it. The fact that a pool_alloc() or pool_free() on some structure may call thread_harmless_now() and let another thread start to release the same object's memory is not acceptable. This patch introduces the concept of "idle threads". Threads entering the polling loop are idle, as well as those that are waiting for all others to become idle via the new function thread_isolate_full(). Once thread_isolate_full() is granted, the thread is not idle anymore, and it is released using thread_release() just like regular isolation. Its users have to keep in mind that across this call nothing is granted as another thread might have performed shared memory modifications. But such users are extremely rare and are actually expecting this from their peers as well. Note that that in case of backport, this patch depends on previous patch: MINOR: threads: make thread_release() not wait for other ones to complete
2021-08-04 05:44:17 -04:00
* thread_release(). By doing so the thread also engages in being safe against
* any actions that other threads might be about to start under the same
* conditions. This specifically targets destruction of any internal structure,
* which implies that the current thread may not hold references to any object.
*
* Note that a concurrent thread_isolate() will usually win against
* thread_isolate_full() as it doesn't consider the idle_mask, allowing it to
* get back to the poller or any other fully idle location, that will
* ultimately release this one.
*/
void thread_isolate_full()
{
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
uint tgrp, thr;
MEDIUM: threads: add a stronger thread_isolate_full() call The current principle of running under isolation was made to access sensitive data while being certain that no other thread was using them in parallel, without necessarily having to place locks everywhere. The main use case are "show sess" and "show fd" which run over long chains of pointers. The thread_isolate() call relies on the "harmless" bit that indicates for a given thread that it's not currently doing such sensitive things, which is advertised using thread_harmless_now() and which ends usings thread_harmless_end(), which also waits for possibly concurrent threads to complete their work if they took this opportunity for starting something tricky. As some system calls were notoriously slow (e.g. mmap()), a bunch of thread_harmless_now() / thread_harmless_end() were placed around them to let waiting threads do their work while such other threads were not able to modify memory contents. But this is not sufficient for performing memory modifications. One such example is the server deletion code. By modifying memory, it not only requires that other threads are not playing with it, but are not either in the process of touching it. The fact that a pool_alloc() or pool_free() on some structure may call thread_harmless_now() and let another thread start to release the same object's memory is not acceptable. This patch introduces the concept of "idle threads". Threads entering the polling loop are idle, as well as those that are waiting for all others to become idle via the new function thread_isolate_full(). Once thread_isolate_full() is granted, the thread is not idle anymore, and it is released using thread_release() just like regular isolation. Its users have to keep in mind that across this call nothing is granted as another thread might have performed shared memory modifications. But such users are extremely rare and are actually expecting this from their peers as well. Note that that in case of backport, this patch depends on previous patch: MINOR: threads: make thread_release() not wait for other ones to complete
2021-08-04 05:44:17 -04:00
MEDIUM: tinfo: add a dynamic thread-group context The thread group info is not sufficient to represent a thread group's current state as it's read-only. We also need something comparable to the thread context to represent the aggregate state of the threads in that group. This patch introduces ha_tgroup_ctx[] and tg_ctx for this. It's indexed on the group id and must be cache-line aligned. The thread masks that were global and that do not need to remain global were moved there (want_rdv, harmless, idle). Given that all the masks placed there now become group-specific, the associated thread mask (tid_bit) now switches to the thread's local bit (ltid_bit). Both are the same for nbtgroups 1 but will differ for other values. There's also a tg_ctx pointer in the thread so that it can be reached from other threads.
2022-06-27 10:02:24 -04:00
_HA_ATOMIC_OR(&tg_ctx->threads_idle, ti->ltid_bit);
_HA_ATOMIC_OR(&tg_ctx->threads_harmless, ti->ltid_bit);
MEDIUM: threads: add a stronger thread_isolate_full() call The current principle of running under isolation was made to access sensitive data while being certain that no other thread was using them in parallel, without necessarily having to place locks everywhere. The main use case are "show sess" and "show fd" which run over long chains of pointers. The thread_isolate() call relies on the "harmless" bit that indicates for a given thread that it's not currently doing such sensitive things, which is advertised using thread_harmless_now() and which ends usings thread_harmless_end(), which also waits for possibly concurrent threads to complete their work if they took this opportunity for starting something tricky. As some system calls were notoriously slow (e.g. mmap()), a bunch of thread_harmless_now() / thread_harmless_end() were placed around them to let waiting threads do their work while such other threads were not able to modify memory contents. But this is not sufficient for performing memory modifications. One such example is the server deletion code. By modifying memory, it not only requires that other threads are not playing with it, but are not either in the process of touching it. The fact that a pool_alloc() or pool_free() on some structure may call thread_harmless_now() and let another thread start to release the same object's memory is not acceptable. This patch introduces the concept of "idle threads". Threads entering the polling loop are idle, as well as those that are waiting for all others to become idle via the new function thread_isolate_full(). Once thread_isolate_full() is granted, the thread is not idle anymore, and it is released using thread_release() just like regular isolation. Its users have to keep in mind that across this call nothing is granted as another thread might have performed shared memory modifications. But such users are extremely rare and are actually expecting this from their peers as well. Note that that in case of backport, this patch depends on previous patch: MINOR: threads: make thread_release() not wait for other ones to complete
2021-08-04 05:44:17 -04:00
__ha_barrier_atomic_store();
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
_HA_ATOMIC_INC(&rdv_requests);
MEDIUM: threads: add a stronger thread_isolate_full() call The current principle of running under isolation was made to access sensitive data while being certain that no other thread was using them in parallel, without necessarily having to place locks everywhere. The main use case are "show sess" and "show fd" which run over long chains of pointers. The thread_isolate() call relies on the "harmless" bit that indicates for a given thread that it's not currently doing such sensitive things, which is advertised using thread_harmless_now() and which ends usings thread_harmless_end(), which also waits for possibly concurrent threads to complete their work if they took this opportunity for starting something tricky. As some system calls were notoriously slow (e.g. mmap()), a bunch of thread_harmless_now() / thread_harmless_end() were placed around them to let waiting threads do their work while such other threads were not able to modify memory contents. But this is not sufficient for performing memory modifications. One such example is the server deletion code. By modifying memory, it not only requires that other threads are not playing with it, but are not either in the process of touching it. The fact that a pool_alloc() or pool_free() on some structure may call thread_harmless_now() and let another thread start to release the same object's memory is not acceptable. This patch introduces the concept of "idle threads". Threads entering the polling loop are idle, as well as those that are waiting for all others to become idle via the new function thread_isolate_full(). Once thread_isolate_full() is granted, the thread is not idle anymore, and it is released using thread_release() just like regular isolation. Its users have to keep in mind that across this call nothing is granted as another thread might have performed shared memory modifications. But such users are extremely rare and are actually expecting this from their peers as well. Note that that in case of backport, this patch depends on previous patch: MINOR: threads: make thread_release() not wait for other ones to complete
2021-08-04 05:44:17 -04:00
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
/* wait for all threads to become harmless. They cannot change their
* mind once seen thanks to rdv_requests above, unless they pass in
BUG/MEDIUM: threads: fix a tiny race in thread_isolate() Aurlien found a tiny race in thread_isolate() that can allow a thread that was running under isolation to continue running while another one enters isolation. The reason is that the check for harmless is only done before winning the CAS, but since the previously isolated thread doesn't wait for !rdv_request in thread_release(), it can effectively continue its activities while the next one believes it's isolated. A proper solution consists in looping once again in thread_isolate() to recheck (and wait) for all threads to be isolated once the CAS is won. The issue was introduced in 2.7 by commit 598cf3f22 ("MAJOR: threads: change thread_isolate to support inter-group synchronization") so the fix needs to be backported there.
2023-05-27 07:45:01 -04:00
* front of us. For this reason we proceed in 4 steps:
* 1) wait for all threads to declare themselves harmless
* 2) try to grab the isolated_thread exclusivity
* 3) verify again that all threads are harmless, since another one
* that was isolating between 1 and 2 could have dropped its
* harmless state there.
* 4) drop harmless flag (which also has the benefit of leaving
* all other threads wait on reads instead of writes.
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
*/
MEDIUM: threads: add a stronger thread_isolate_full() call The current principle of running under isolation was made to access sensitive data while being certain that no other thread was using them in parallel, without necessarily having to place locks everywhere. The main use case are "show sess" and "show fd" which run over long chains of pointers. The thread_isolate() call relies on the "harmless" bit that indicates for a given thread that it's not currently doing such sensitive things, which is advertised using thread_harmless_now() and which ends usings thread_harmless_end(), which also waits for possibly concurrent threads to complete their work if they took this opportunity for starting something tricky. As some system calls were notoriously slow (e.g. mmap()), a bunch of thread_harmless_now() / thread_harmless_end() were placed around them to let waiting threads do their work while such other threads were not able to modify memory contents. But this is not sufficient for performing memory modifications. One such example is the server deletion code. By modifying memory, it not only requires that other threads are not playing with it, but are not either in the process of touching it. The fact that a pool_alloc() or pool_free() on some structure may call thread_harmless_now() and let another thread start to release the same object's memory is not acceptable. This patch introduces the concept of "idle threads". Threads entering the polling loop are idle, as well as those that are waiting for all others to become idle via the new function thread_isolate_full(). Once thread_isolate_full() is granted, the thread is not idle anymore, and it is released using thread_release() just like regular isolation. Its users have to keep in mind that across this call nothing is granted as another thread might have performed shared memory modifications. But such users are extremely rare and are actually expecting this from their peers as well. Note that that in case of backport, this patch depends on previous patch: MINOR: threads: make thread_release() not wait for other ones to complete
2021-08-04 05:44:17 -04:00
while (1) {
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
for (tgrp = 0; tgrp < global.nbtgroups; tgrp++) {
BUG/MINOR: thread: always reload threads_enabled in loops A few loops waiting for threads to synchronize such as thread_isolate() rightfully filter the thread masks via the threads_enabled field that contains the list of enabled threads. However, it doesn't use an atomic load on it. Before 2.7, the equivalent variables were marked as volatile and were always reloaded. In 2.7 they're fields in ha_tgroup_ctx[], and the risk that the compiler keeps them in a register inside a loop is not null at all. In practice when ha_thread_relax() calls sched_yield() or an x86 PAUSE instruction, it could be verified that the variable is always reloaded. If these are avoided (e.g. architecture providing neither solution), it's visible in asm code that the variables are not reloaded. In this case, if a thread exists just between the moment the two values are read, the loop could spin forever. This patch adds the required _HA_ATOMIC_LOAD() on the relevant threads_enabled fields. It must be backported to 2.7.
2023-01-19 13:14:18 -05:00
do {
ulong te = _HA_ATOMIC_LOAD(&ha_tgroup_info[tgrp].threads_enabled);
ulong th = _HA_ATOMIC_LOAD(&ha_tgroup_ctx[tgrp].threads_harmless);
ulong id = _HA_ATOMIC_LOAD(&ha_tgroup_ctx[tgrp].threads_idle);
if ((th & id & te) == te)
break;
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
ha_thread_relax();
BUG/MINOR: thread: always reload threads_enabled in loops A few loops waiting for threads to synchronize such as thread_isolate() rightfully filter the thread masks via the threads_enabled field that contains the list of enabled threads. However, it doesn't use an atomic load on it. Before 2.7, the equivalent variables were marked as volatile and were always reloaded. In 2.7 they're fields in ha_tgroup_ctx[], and the risk that the compiler keeps them in a register inside a loop is not null at all. In practice when ha_thread_relax() calls sched_yield() or an x86 PAUSE instruction, it could be verified that the variable is always reloaded. If these are avoided (e.g. architecture providing neither solution), it's visible in asm code that the variables are not reloaded. In this case, if a thread exists just between the moment the two values are read, the loop could spin forever. This patch adds the required _HA_ATOMIC_LOAD() on the relevant threads_enabled fields. It must be backported to 2.7.
2023-01-19 13:14:18 -05:00
} while (1);
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
}
BUG/MEDIUM: threads: fix a tiny race in thread_isolate() Aurlien found a tiny race in thread_isolate() that can allow a thread that was running under isolation to continue running while another one enters isolation. The reason is that the check for harmless is only done before winning the CAS, but since the previously isolated thread doesn't wait for !rdv_request in thread_release(), it can effectively continue its activities while the next one believes it's isolated. A proper solution consists in looping once again in thread_isolate() to recheck (and wait) for all threads to be isolated once the CAS is won. The issue was introduced in 2.7 by commit 598cf3f22 ("MAJOR: threads: change thread_isolate to support inter-group synchronization") so the fix needs to be backported there.
2023-05-27 07:45:01 -04:00
/* all other ones are harmless and idle. isolated_thread will
* contain ~0U if no other one competes, !=tid if another one
* got it, tid if the current thread already grabbed it on the
* previous round.
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
*/
thr = _HA_ATOMIC_LOAD(&isolated_thread);
BUG/MEDIUM: threads: fix a tiny race in thread_isolate() Aurlien found a tiny race in thread_isolate() that can allow a thread that was running under isolation to continue running while another one enters isolation. The reason is that the check for harmless is only done before winning the CAS, but since the previously isolated thread doesn't wait for !rdv_request in thread_release(), it can effectively continue its activities while the next one believes it's isolated. A proper solution consists in looping once again in thread_isolate() to recheck (and wait) for all threads to be isolated once the CAS is won. The issue was introduced in 2.7 by commit 598cf3f22 ("MAJOR: threads: change thread_isolate to support inter-group synchronization") so the fix needs to be backported there.
2023-05-27 07:45:01 -04:00
if (thr == tid)
break; // we won and we're certain everyone is harmless
if (thr != ~0U || !_HA_ATOMIC_CAS(&isolated_thread, &thr, tid))
ha_thread_relax();
MEDIUM: threads: add a stronger thread_isolate_full() call The current principle of running under isolation was made to access sensitive data while being certain that no other thread was using them in parallel, without necessarily having to place locks everywhere. The main use case are "show sess" and "show fd" which run over long chains of pointers. The thread_isolate() call relies on the "harmless" bit that indicates for a given thread that it's not currently doing such sensitive things, which is advertised using thread_harmless_now() and which ends usings thread_harmless_end(), which also waits for possibly concurrent threads to complete their work if they took this opportunity for starting something tricky. As some system calls were notoriously slow (e.g. mmap()), a bunch of thread_harmless_now() / thread_harmless_end() were placed around them to let waiting threads do their work while such other threads were not able to modify memory contents. But this is not sufficient for performing memory modifications. One such example is the server deletion code. By modifying memory, it not only requires that other threads are not playing with it, but are not either in the process of touching it. The fact that a pool_alloc() or pool_free() on some structure may call thread_harmless_now() and let another thread start to release the same object's memory is not acceptable. This patch introduces the concept of "idle threads". Threads entering the polling loop are idle, as well as those that are waiting for all others to become idle via the new function thread_isolate_full(). Once thread_isolate_full() is granted, the thread is not idle anymore, and it is released using thread_release() just like regular isolation. Its users have to keep in mind that across this call nothing is granted as another thread might have performed shared memory modifications. But such users are extremely rare and are actually expecting this from their peers as well. Note that that in case of backport, this patch depends on previous patch: MINOR: threads: make thread_release() not wait for other ones to complete
2021-08-04 05:44:17 -04:00
}
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
/* we're not idle nor harmless anymore at this point. Other threads
* waiting on this condition will need to wait until out next pass to
* the poller, or our next call to thread_isolate_full().
MEDIUM: threads: add a stronger thread_isolate_full() call The current principle of running under isolation was made to access sensitive data while being certain that no other thread was using them in parallel, without necessarily having to place locks everywhere. The main use case are "show sess" and "show fd" which run over long chains of pointers. The thread_isolate() call relies on the "harmless" bit that indicates for a given thread that it's not currently doing such sensitive things, which is advertised using thread_harmless_now() and which ends usings thread_harmless_end(), which also waits for possibly concurrent threads to complete their work if they took this opportunity for starting something tricky. As some system calls were notoriously slow (e.g. mmap()), a bunch of thread_harmless_now() / thread_harmless_end() were placed around them to let waiting threads do their work while such other threads were not able to modify memory contents. But this is not sufficient for performing memory modifications. One such example is the server deletion code. By modifying memory, it not only requires that other threads are not playing with it, but are not either in the process of touching it. The fact that a pool_alloc() or pool_free() on some structure may call thread_harmless_now() and let another thread start to release the same object's memory is not acceptable. This patch introduces the concept of "idle threads". Threads entering the polling loop are idle, as well as those that are waiting for all others to become idle via the new function thread_isolate_full(). Once thread_isolate_full() is granted, the thread is not idle anymore, and it is released using thread_release() just like regular isolation. Its users have to keep in mind that across this call nothing is granted as another thread might have performed shared memory modifications. But such users are extremely rare and are actually expecting this from their peers as well. Note that that in case of backport, this patch depends on previous patch: MINOR: threads: make thread_release() not wait for other ones to complete
2021-08-04 05:44:17 -04:00
*/
MEDIUM: tinfo: add a dynamic thread-group context The thread group info is not sufficient to represent a thread group's current state as it's read-only. We also need something comparable to the thread context to represent the aggregate state of the threads in that group. This patch introduces ha_tgroup_ctx[] and tg_ctx for this. It's indexed on the group id and must be cache-line aligned. The thread masks that were global and that do not need to remain global were moved there (want_rdv, harmless, idle). Given that all the masks placed there now become group-specific, the associated thread mask (tid_bit) now switches to the thread's local bit (ltid_bit). Both are the same for nbtgroups 1 but will differ for other values. There's also a tg_ctx pointer in the thread so that it can be reached from other threads.
2022-06-27 10:02:24 -04:00
_HA_ATOMIC_AND(&tg_ctx->threads_idle, ~ti->ltid_bit);
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
_HA_ATOMIC_AND(&tg_ctx->threads_harmless, ~ti->ltid_bit);
BUG/MEDIUM: threads: fix a tiny race in thread_isolate() Aurlien found a tiny race in thread_isolate() that can allow a thread that was running under isolation to continue running while another one enters isolation. The reason is that the check for harmless is only done before winning the CAS, but since the previously isolated thread doesn't wait for !rdv_request in thread_release(), it can effectively continue its activities while the next one believes it's isolated. A proper solution consists in looping once again in thread_isolate() to recheck (and wait) for all threads to be isolated once the CAS is won. The issue was introduced in 2.7 by commit 598cf3f22 ("MAJOR: threads: change thread_isolate to support inter-group synchronization") so the fix needs to be backported there.
2023-05-27 07:45:01 -04:00
/* the thread is isolated until it calls thread_release() which will
* 1) reset isolated_thread to ~0;
* 2) decrement rdv_requests.
*/
MEDIUM: threads: add a stronger thread_isolate_full() call The current principle of running under isolation was made to access sensitive data while being certain that no other thread was using them in parallel, without necessarily having to place locks everywhere. The main use case are "show sess" and "show fd" which run over long chains of pointers. The thread_isolate() call relies on the "harmless" bit that indicates for a given thread that it's not currently doing such sensitive things, which is advertised using thread_harmless_now() and which ends usings thread_harmless_end(), which also waits for possibly concurrent threads to complete their work if they took this opportunity for starting something tricky. As some system calls were notoriously slow (e.g. mmap()), a bunch of thread_harmless_now() / thread_harmless_end() were placed around them to let waiting threads do their work while such other threads were not able to modify memory contents. But this is not sufficient for performing memory modifications. One such example is the server deletion code. By modifying memory, it not only requires that other threads are not playing with it, but are not either in the process of touching it. The fact that a pool_alloc() or pool_free() on some structure may call thread_harmless_now() and let another thread start to release the same object's memory is not acceptable. This patch introduces the concept of "idle threads". Threads entering the polling loop are idle, as well as those that are waiting for all others to become idle via the new function thread_isolate_full(). Once thread_isolate_full() is granted, the thread is not idle anymore, and it is released using thread_release() just like regular isolation. Its users have to keep in mind that across this call nothing is granted as another thread might have performed shared memory modifications. But such users are extremely rare and are actually expecting this from their peers as well. Note that that in case of backport, this patch depends on previous patch: MINOR: threads: make thread_release() not wait for other ones to complete
2021-08-04 05:44:17 -04:00
}
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
/* Cancels the effect of thread_isolate() by resetting the ID of the isolated
* thread and decrementing the number of RDV requesters. This immediately allows
* other threads to expect to be executed, though they will first have to wait
* for this thread to become harmless again (possibly by reaching the poller
* again).
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
*/
void thread_release()
{
MAJOR: threads: change thread_isolate to support inter-group synchronization thread_isolate() and thread_isolate_full() were relying on a set of thread masks for all threads in different states (rdv, harmless, idle). This cannot work anymore when the number of threads increases beyond LONGBITS so we need to change the mechanism. What is done here is to have a counter of requesters and the number of the current isolated thread. Threads which want to isolate themselves increment the request counter and wait for all threads to be marked harmless (or idle) by scanning all groups and watching the respective masks. This is possible because threads cannot escape once they discover this counter, unless they also want to isolate and possibly pass first. Once all threads are harmless, the requesting thread tries to self-assign the isolated thread number, and if it fails it loops back to checking all threads. If it wins it's guaranted to be alone, and can drop its harmless bit, so that other competing threads go back to the loop waiting for all threads to be harmless. The benefit of proceeding this way is that there's very little write contention on the thread number (none during work), hence no cache line moves between caches, thus frozen threads do not slow down the isolated one. Once it's done, the isolated thread resets the thread number (hence lets another thread take the place) and decrements the requester count, thus possibly releasing all harmless threads. With this change there's no more need for any global mask to synchronize any thread, and we only need to loop over a number of groups to check 64 threads at a time per iteration. As such, tinfo's threads_want_rdv could be dropped. This was tested with 64 threads spread into 2 groups, running 64 tasks (from the debug dev command), 20 "show sess" (thread_isolate()), 20 "add server blah/blah" (thread_isolate()), and 20 "del server blah/blah" (thread_isolate_full()). The load remained very low (limited by external socat forks) and no stuck nor starved thread was found.
2022-07-01 09:08:37 -04:00
HA_ATOMIC_STORE(&isolated_thread, ~0U);
HA_ATOMIC_DEC(&rdv_requests);
MEDIUM: hathreads: implement a more flexible rendez-vous point The current synchronization point enforces certain restrictions which are hard to workaround in certain areas of the code. The fact that the critical code can only be called from the sync point itself is a problem for some callback-driven parts. The "show fd" command for example is fragile regarding this. Also it is expensive in terms of CPU usage because it wakes every other thread just to be sure all of them join to the rendez-vous point. It's a problem because the sleeping threads would not need to be woken up just to know they're doing nothing. Here we implement a different approach. We keep track of harmless threads, which are defined as those either doing nothing, or doing harmless things. The rendez-vous is used "for others" as a way for a thread to isolate itself. A thread then requests to be alone using thread_isolate() when approaching the dangerous area, and then waits until all other threads are either doing the same or are doing something harmless (typically polling). The function only returns once the thread is guaranteed to be alone, and the critical section is terminated using thread_release().
2018-08-02 04:16:17 -04:00
}
MEDIUM: threads: Adds a set of functions to handle sync-point A sync-point is a protected area where you have the warranty that no concurrency access is possible. It is implementated as a thread barrier to enter in the sync-point and another one to exit from it. Inside the sync-point, all threads that must do some syncrhonous processing will be called one after the other while all other threads will wait. All threads will then exit from the sync-point at the same time. A sync-point will be evaluated only when necessary because it is a costly operation. To limit the waiting time of each threads, we must have a mechanism to wakeup all threads. This is done with a pipe shared by all threads. By writting in this pipe, we will interrupt all threads blocked on a poller. The pipe is then flushed before exiting from the sync-point.
2017-10-19 05:59:15 -04:00
REORG: thread: move the thread init/affinity/stop to thread.c haproxy.c still has to deal with pthread-specific low-level stuff that is OS-dependent. We should not have to deal with this there, and we do not need to access pthread anywhere else. Let's move these 3 functions to thread.c and keep empty inline ones for when threads are disabled.
2021-10-06 16:22:40 -04:00
/* Sets up threads, signals and masks, and starts threads 2 and above.
* Does nothing when threads are disabled.
*/
void setup_extra_threads(void *(*handler)(void *))
{
sigset_t blocked_sig, old_sig;
int i;
/* ensure the signals will be blocked in every thread */
sigfillset(&blocked_sig);
sigdelset(&blocked_sig, SIGPROF);
sigdelset(&blocked_sig, SIGBUS);
sigdelset(&blocked_sig, SIGFPE);
sigdelset(&blocked_sig, SIGILL);
sigdelset(&blocked_sig, SIGSEGV);
pthread_sigmask(SIG_SETMASK, &blocked_sig, &old_sig);
/* Create nbthread-1 thread. The first thread is the current process */
MINOR: thread: use a dedicated static pthread_t array in thread.c This removes the thread identifiers from struct thread_info and moves them only in static array in thread.c since it's now the only file that needs to touch it. It's also the only file that needs to include pthread.h, beyond haproxy.c which needs it to start the poll loop. As a result, much less system includes are needed and the LoC reduced by around 3%.
2021-10-06 16:53:51 -04:00
ha_pthread[0] = pthread_self();
REORG: thread: move the thread init/affinity/stop to thread.c haproxy.c still has to deal with pthread-specific low-level stuff that is OS-dependent. We should not have to deal with this there, and we do not need to access pthread anywhere else. Let's move these 3 functions to thread.c and keep empty inline ones for when threads are disabled.
2021-10-06 16:22:40 -04:00
for (i = 1; i < global.nbthread; i++)
MEDIUM: threads: replace ha_set_tid() with ha_set_thread() ha_set_tid() was randomly used either to explicitly set thread 0 or to set any possibly incomplete thread during boot. Let's replace it with a pointer to a valid thread or NULL for any thread. This allows us to check that the designated threads are always valid, and to ignore the thread 0's mapping when setting it to NULL, and always use group 0 with it during boot. The initialization code is also cleaner, as we don't pass ugly casts of a thread ID to a pointer anymore.
2021-09-28 03:43:11 -04:00
pthread_create(&ha_pthread[i], NULL, handler, &ha_thread_info[i]);
REORG: thread: move the thread init/affinity/stop to thread.c haproxy.c still has to deal with pthread-specific low-level stuff that is OS-dependent. We should not have to deal with this there, and we do not need to access pthread anywhere else. Let's move these 3 functions to thread.c and keep empty inline ones for when threads are disabled.
2021-10-06 16:22:40 -04:00
}
/* waits for all threads to terminate. Does nothing when threads are
* disabled.
*/
void wait_for_threads_completion()
{
int i;
/* Wait the end of other threads */
for (i = 1; i < global.nbthread; i++)
MINOR: thread: use a dedicated static pthread_t array in thread.c This removes the thread identifiers from struct thread_info and moves them only in static array in thread.c since it's now the only file that needs to touch it. It's also the only file that needs to include pthread.h, beyond haproxy.c which needs it to start the poll loop. As a result, much less system includes are needed and the LoC reduced by around 3%.
2021-10-06 16:53:51 -04:00
pthread_join(ha_pthread[i], NULL);
REORG: thread: move the thread init/affinity/stop to thread.c haproxy.c still has to deal with pthread-specific low-level stuff that is OS-dependent. We should not have to deal with this there, and we do not need to access pthread anywhere else. Let's move these 3 functions to thread.c and keep empty inline ones for when threads are disabled.
2021-10-06 16:22:40 -04:00
#if defined(DEBUG_THREAD) || defined(DEBUG_FULL)
show_lock_stats();
#endif
}
/* Tries to set the current thread's CPU affinity according to the cpu_map */
void set_thread_cpu_affinity()
{
#if defined(USE_CPU_AFFINITY)
/* no affinity setting for the master process */
if (master)
return;
/* Now the CPU affinity for all threads */
MEDIUM: cpu-map: replace the process number with the thread group number The principle remains the same, but instead of having a single process and ignoring extra ones, now we set the affinity masks for the respective threads of all groups. The doc was updated with a few extra examples.
2022-07-08 03:38:30 -04:00
if (ha_cpuset_count(&cpu_map[tgid - 1].thread[ti->ltid])) {/* only do this if the thread has a THREAD map */
REORG: thread: move the thread init/affinity/stop to thread.c haproxy.c still has to deal with pthread-specific low-level stuff that is OS-dependent. We should not have to deal with this there, and we do not need to access pthread anywhere else. Let's move these 3 functions to thread.c and keep empty inline ones for when threads are disabled.
2021-10-06 16:22:40 -04:00
# if defined(__APPLE__)
/* Note: this API is limited to the first 32/64 CPUs */
MEDIUM: cpu-map: replace the process number with the thread group number The principle remains the same, but instead of having a single process and ignoring extra ones, now we set the affinity masks for the respective threads of all groups. The doc was updated with a few extra examples.
2022-07-08 03:38:30 -04:00
unsigned long set = cpu_map[tgid - 1].thread[ti->ltid].cpuset;
REORG: thread: move the thread init/affinity/stop to thread.c haproxy.c still has to deal with pthread-specific low-level stuff that is OS-dependent. We should not have to deal with this there, and we do not need to access pthread anywhere else. Let's move these 3 functions to thread.c and keep empty inline ones for when threads are disabled.
2021-10-06 16:22:40 -04:00
int j;
while ((j = ffsl(set)) > 0) {
thread_affinity_policy_data_t cpu_set = { j - 1 };
thread_port_t mthread;
MINOR: thread: use a dedicated static pthread_t array in thread.c This removes the thread identifiers from struct thread_info and moves them only in static array in thread.c since it's now the only file that needs to touch it. It's also the only file that needs to include pthread.h, beyond haproxy.c which needs it to start the poll loop. As a result, much less system includes are needed and the LoC reduced by around 3%.
2021-10-06 16:53:51 -04:00
mthread = pthread_mach_thread_np(ha_pthread[tid]);
REORG: thread: move the thread init/affinity/stop to thread.c haproxy.c still has to deal with pthread-specific low-level stuff that is OS-dependent. We should not have to deal with this there, and we do not need to access pthread anywhere else. Let's move these 3 functions to thread.c and keep empty inline ones for when threads are disabled.
2021-10-06 16:22:40 -04:00
thread_policy_set(mthread, THREAD_AFFINITY_POLICY, (thread_policy_t)&cpu_set, 1);
set &= ~(1UL << (j - 1));
}
# else
MEDIUM: cpu-map: replace the process number with the thread group number The principle remains the same, but instead of having a single process and ignoring extra ones, now we set the affinity masks for the respective threads of all groups. The doc was updated with a few extra examples.
2022-07-08 03:38:30 -04:00
struct hap_cpuset *set = &cpu_map[tgid - 1].thread[ti->ltid];
REORG: thread: move the thread init/affinity/stop to thread.c haproxy.c still has to deal with pthread-specific low-level stuff that is OS-dependent. We should not have to deal with this there, and we do not need to access pthread anywhere else. Let's move these 3 functions to thread.c and keep empty inline ones for when threads are disabled.
2021-10-06 16:22:40 -04:00
MINOR: thread: use a dedicated static pthread_t array in thread.c This removes the thread identifiers from struct thread_info and moves them only in static array in thread.c since it's now the only file that needs to touch it. It's also the only file that needs to include pthread.h, beyond haproxy.c which needs it to start the poll loop. As a result, much less system includes are needed and the LoC reduced by around 3%.
2021-10-06 16:53:51 -04:00
pthread_setaffinity_np(ha_pthread[tid], sizeof(set->cpuset), &set->cpuset);
REORG: thread: move the thread init/affinity/stop to thread.c haproxy.c still has to deal with pthread-specific low-level stuff that is OS-dependent. We should not have to deal with this there, and we do not need to access pthread anywhere else. Let's move these 3 functions to thread.c and keep empty inline ones for when threads are disabled.
2021-10-06 16:22:40 -04:00
# endif
}
#endif /* USE_CPU_AFFINITY */
}
REORG: thread: move ha_get_pthread_id() to thread.c It's the last function which directly accesses the pthread_t, let's move it to thread.c and leave a static inline for non-thread.
2021-10-06 16:44:28 -04:00
/* Retrieves the opaque pthread_t of thread <thr> cast to an unsigned long long
* since POSIX took great care of not specifying its representation, making it
* hard to export for post-mortem analysis. For this reason we copy it into a
* union and will use the smallest scalar type at least as large as its size,
* which will keep endianness and alignment for all regular sizes. As a last
* resort we end up with a long long ligned to the first bytes in memory, which
* will be endian-dependent if pthread_t is larger than a long long (not seen
* yet).
*/
unsigned long long ha_get_pthread_id(unsigned int thr)
{
union {
pthread_t t;
unsigned long long ll;
unsigned int i;
unsigned short s;
unsigned char c;
} u = { 0 };
MINOR: thread: use a dedicated static pthread_t array in thread.c This removes the thread identifiers from struct thread_info and moves them only in static array in thread.c since it's now the only file that needs to touch it. It's also the only file that needs to include pthread.h, beyond haproxy.c which needs it to start the poll loop. As a result, much less system includes are needed and the LoC reduced by around 3%.
2021-10-06 16:53:51 -04:00
u.t = ha_pthread[thr];
REORG: thread: move ha_get_pthread_id() to thread.c It's the last function which directly accesses the pthread_t, let's move it to thread.c and leave a static inline for non-thread.
2021-10-06 16:44:28 -04:00
if (sizeof(u.t) <= sizeof(u.c))
return u.c;
else if (sizeof(u.t) <= sizeof(u.s))
return u.s;
else if (sizeof(u.t) <= sizeof(u.i))
return u.i;
return u.ll;
}
MINOR: threads: implement ha_tkill() and ha_tkillall() These functions are used respectively to signal one thread or all threads. When multithreading is disabled, it's always the current thread which is signaled.
2019-05-22 02:43:34 -04:00
/* send signal <sig> to thread <thr> */
void ha_tkill(unsigned int thr, int sig)
{
MINOR: thread: use a dedicated static pthread_t array in thread.c This removes the thread identifiers from struct thread_info and moves them only in static array in thread.c since it's now the only file that needs to touch it. It's also the only file that needs to include pthread.h, beyond haproxy.c which needs it to start the poll loop. As a result, much less system includes are needed and the LoC reduced by around 3%.
2021-10-06 16:53:51 -04:00
pthread_kill(ha_pthread[thr], sig);
MINOR: threads: implement ha_tkill() and ha_tkillall() These functions are used respectively to signal one thread or all threads. When multithreading is disabled, it's always the current thread which is signaled.
2019-05-22 02:43:34 -04:00
}
/* send signal <sig> to all threads. The calling thread is signaled last in
* order to allow all threads to synchronize in the handler.
*/
void ha_tkillall(int sig)
{
unsigned int thr;
for (thr = 0; thr < global.nbthread; thr++) {
BUG/MINOR: thread: use the correct thread's group in ha_tkillall() In ha_tkillall(), the current thread's group was used to check for the thread being running instead of using the target thread's group mask. Most of the time it would not have any effect unless some groups are uneven where it can lead to incomplete thread dumps for example. No backport is needed, this is purely 2.7.
2022-07-15 02:27:56 -04:00
if (!(ha_thread_info[thr].tg->threads_enabled & ha_thread_info[thr].ltid_bit))
MINOR: threads: implement ha_tkill() and ha_tkillall() These functions are used respectively to signal one thread or all threads. When multithreading is disabled, it's always the current thread which is signaled.
2019-05-22 02:43:34 -04:00
continue;
if (thr == tid)
continue;
MINOR: thread: use a dedicated static pthread_t array in thread.c This removes the thread identifiers from struct thread_info and moves them only in static array in thread.c since it's now the only file that needs to touch it. It's also the only file that needs to include pthread.h, beyond haproxy.c which needs it to start the poll loop. As a result, much less system includes are needed and the LoC reduced by around 3%.
2021-10-06 16:53:51 -04:00
pthread_kill(ha_pthread[thr], sig);
MINOR: threads: implement ha_tkill() and ha_tkillall() These functions are used respectively to signal one thread or all threads. When multithreading is disabled, it's always the current thread which is signaled.
2019-05-22 02:43:34 -04:00
}
raise(sig);
}
CLEANUP: thread: uninline ha_tkill/ha_tkillall/ha_cpu_relax() These ones are rarely used or only to waste CPU cycles waiting, and are the last ones requiring system includes in thread.h. Let's uninline them and move them to thread.c.
2021-10-06 17:33:20 -04:00
void ha_thread_relax(void)
{
#ifdef _POSIX_PRIORITY_SCHEDULING
sched_yield();
#else
pl_cpu_relax();
#endif
}
CLEANUP: threads: don't register an initcall when not debugging It's a bit overkill to register an initcall to call a function to set a lock to zero when not debugging, let's just declare the lock as pre-initialized to zero.
2020-10-18 04:20:59 -04:00
/* these calls are used as callbacks at init time when debugging is on */
MINOR: thread: provide a set of lock initialisers This patch adds ha_spin_init() and ha_rwlock_init() which are used as a callback to initialise locks at boot time. They perform exactly the same as HA_SPIN_INIT() or HA_RWLOCK_INIT() but from within a real function.
2018-11-25 13:28:23 -05:00
void ha_spin_init(HA_SPINLOCK_T *l)
{
HA_SPIN_INIT(l);
}
CLEANUP: threads: don't register an initcall when not debugging It's a bit overkill to register an initcall to call a function to set a lock to zero when not debugging, let's just declare the lock as pre-initialized to zero.
2020-10-18 04:20:59 -04:00
/* these calls are used as callbacks at init time when debugging is on */
MINOR: thread: provide a set of lock initialisers This patch adds ha_spin_init() and ha_rwlock_init() which are used as a callback to initialise locks at boot time. They perform exactly the same as HA_SPIN_INIT() or HA_RWLOCK_INIT() but from within a real function.
2018-11-25 13:28:23 -05:00
void ha_rwlock_init(HA_RWLOCK_T *l)
{
HA_RWLOCK_INIT(l);
}
BUG/MINOR: init: properly detect NUMA bindings on large systems The NUMA detection code tries not to interfer with any taskset the user could have specified in init scripts. For this it compares the number of CPUs available with the number the process is bound to. However, the CPU count is retrieved after being applied an upper bound of MAX_THREADS, so if the machine has more than 64 CPUs, the comparison always fails and makes haproxy think the user has already enforced a binding, and it does not pin it anymore to a single NUMA node. This can be verified by issuing: $ socat /path/to/sock - <<< "show info" | grep thread On a dual 48-CPU machine it reports 64, implying that threads are allowed to run on the second socket: Nbthread: 64 With this fix, the function properly reports 96, and the output shows 48, indicating that a single NUMA node was used: Nbthread: 48 Of course nothing is changed when "no numa-cpu-mapping" is specified: Nbthread: 64 This can be backported to 2.4.
2023-03-09 04:12:06 -05:00
/* returns the number of CPUs the current process is enabled to run on,
* regardless of any MAX_THREADS limitation.
*/
MAJOR: threads: enable one thread per CPU by default Threads have long matured by now, still for most users their usage is not trivial. It's about time to enable them by default on platforms where we know the number of CPUs bound. This patch does this, it counts the number of CPUs the process is bound to upon startup, and enables as many threads by default. Of course, "nbthread" still overrides this, but if it's not set the default behaviour is to start one thread per CPU. The default number of threads is reported in "haproxy -vv". Simply using "taskset -c" is now enough to adjust this number of threads so that there is no more need for playing with cpu-map. And thanks to the previous patches on the listener, the vast majority of configurations will not need to duplicate "bind" lines with the "process x/y" statement anymore either, so a simple config will automatically adapt to the number of processors available.
2019-01-26 08:27:06 -05:00
static int thread_cpus_enabled()
{
int ret = 1;
#ifdef USE_CPU_AFFINITY
#if defined(__linux__) && defined(CPU_COUNT)
cpu_set_t mask;
if (sched_getaffinity(0, sizeof(mask), &mask) == 0)
ret = CPU_COUNT(&mask);
MINOR: threads: Implement thread_cpus_enabled() for FreeBSD. Use cpuset_getaffinity() to implement thread_cpus_enabled() on FreeBSD, so that we can know the number of CPUs available, and automatically launch as much threads if nbthread isn't specified.
2019-04-10 18:06:47 -04:00
#elif defined(__FreeBSD__) && defined(USE_CPU_AFFINITY)
cpuset_t cpuset;
if (cpuset_getaffinity(CPU_LEVEL_CPUSET, CPU_WHICH_PID, -1,
sizeof(cpuset), &cpuset) == 0)
ret = CPU_COUNT(&cpuset);
BUG/MINOR: threads: Fixes the number of possible cpus report for Mac. There is no low level api to achieve same as Linux/FreeBSD, we rely on CPUs available. Without this, the number of threads is just 1 for Mac while having 8 cores in my M1. Backporting to 2.1 should be enough if that's possible. Signed-off-by: David CARLIER <devnexen@gmail.com>
2021-01-15 03:09:56 -05:00
#elif defined(__APPLE__)
ret = (int)sysconf(_SC_NPROCESSORS_ONLN);
MAJOR: threads: enable one thread per CPU by default Threads have long matured by now, still for most users their usage is not trivial. It's about time to enable them by default on platforms where we know the number of CPUs bound. This patch does this, it counts the number of CPUs the process is bound to upon startup, and enables as many threads by default. Of course, "nbthread" still overrides this, but if it's not set the default behaviour is to start one thread per CPU. The default number of threads is reported in "haproxy -vv". Simply using "taskset -c" is now enough to adjust this number of threads so that there is no more need for playing with cpu-map. And thanks to the previous patches on the listener, the vast majority of configurations will not need to duplicate "bind" lines with the "process x/y" statement anymore either, so a simple config will automatically adapt to the number of processors available.
2019-01-26 08:27:06 -05:00
#endif
#endif
ret = MAX(ret, 1);
return ret;
}
MINOR: thread: implement the detection of forced cpu affinity Create a function thread_cpu_mask_forced. Its purpose is to report if a restrictive cpu mask is active for the current proces, for example due to a taskset invocation. It is only implemented for the linux platform currently.
2021-03-31 10:57:39 -04:00
/* Returns 1 if the cpu set is currently restricted for the process else 0.
* Currently only implemented for the Linux platform.
*/
int thread_cpu_mask_forced()
{
#if defined(__linux__)
const int cpus_avail = sysconf(_SC_NPROCESSORS_ONLN);
return cpus_avail != thread_cpus_enabled();
#else
return 0;
#endif
}
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
/* Below come the lock-debugging functions */
#if defined(DEBUG_THREAD) || defined(DEBUG_FULL)
struct lock_stat lock_stats[LOCK_LABELS];
/* this is only used below */
static const char *lock_label(enum lock_label label)
{
switch (label) {
case TASK_RQ_LOCK: return "TASK_RQ";
case TASK_WQ_LOCK: return "TASK_WQ";
case LISTENER_LOCK: return "LISTENER";
case PROXY_LOCK: return "PROXY";
case SERVER_LOCK: return "SERVER";
case LBPRM_LOCK: return "LBPRM";
case SIGNALS_LOCK: return "SIGNALS";
case STK_TABLE_LOCK: return "STK_TABLE";
case STK_SESS_LOCK: return "STK_SESS";
case APPLETS_LOCK: return "APPLETS";
case PEER_LOCK: return "PEER";
case SHCTX_LOCK: return "SHCTX";
case SSL_LOCK: return "SSL";
case SSL_GEN_CERTS_LOCK: return "SSL_GEN_CERTS";
case PATREF_LOCK: return "PATREF";
case PATEXP_LOCK: return "PATEXP";
case VARS_LOCK: return "VARS";
case COMP_POOL_LOCK: return "COMP_POOL";
case LUA_LOCK: return "LUA";
case NOTIF_LOCK: return "NOTIF";
case SPOE_APPLET_LOCK: return "SPOE_APPLET";
case DNS_LOCK: return "DNS";
case PID_LIST_LOCK: return "PID_LIST";
case EMAIL_ALERTS_LOCK: return "EMAIL_ALERTS";
case PIPES_LOCK: return "PIPES";
case TLSKEYS_REF_LOCK: return "TLSKEYS_REF";
case AUTH_LOCK: return "AUTH";
case LOGSRV_LOCK: return "LOGSRV";
case DICT_LOCK: return "DICT";
case PROTO_LOCK: return "PROTO";
case QUEUE_LOCK: return "QUEUE";
case CKCH_LOCK: return "CKCH";
case SNI_LOCK: return "SNI";
case SSL_SERVER_LOCK: return "SSL_SERVER";
case SFT_LOCK: return "SFT";
case IDLE_CONNS_LOCK: return "IDLE_CONNS";
MINOR: ssl: Add a lock to the OCSP response tree The tree that contains OCSP responses is never locked despite being used at runtime for OCSP stapling as well as the CLI through "set ssl cert" and "set ssl ocsp-response" commands. Everything works though because the certificate_ocsp structure is refcounted and the tree's entries are cleaned up when SSL_CTXs are destroyed (thanks to an ex_data entry in which the certificate_ocsp pointer is stored). This new lock will come to use when the OCSP auto update mechanism is fully implemented because this new feature will be based on another tree that stores the same certificate_ocsp members and updates their contents periodically.
2022-12-20 05:11:02 -05:00
case OCSP_LOCK: return "OCSP";
MEDIUM: quic: use a global CID trees list Previously, quic_connection_id were stored in a per-thread tree list. Datagram were first dispatched to the correct thread using the encoded TID before a tree lookup was done. Remove these trees and replace it with a global trees list of 256 entries. A CID is using the list index corresponding to its first byte. On datagram dispatch, CID is lookup on its tree and TID is retrieved using new member quic_connection_id.tid. As such, a read-write lock protects each list instances. With 256 entries, it is expected that contention should be reduced. A new structure quic_cid_tree served as a tree container associated with its read-write lock. An API is implemented to ensure lock safety for insert/lookup/delete operation. This patch is a step forward to be able to break the affinity between a CID and a TID encoded thread. This is required to be able to migrate a quic_conn after accept to select thread based on their load. This should be backported up to 2.7 after a period of observation.
2023-04-18 05:10:54 -04:00
case QC_CID_LOCK: return "QC_CID";
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
case OTHER_LOCK: return "OTHER";
case DEBUG1_LOCK: return "DEBUG1";
case DEBUG2_LOCK: return "DEBUG2";
case DEBUG3_LOCK: return "DEBUG3";
case DEBUG4_LOCK: return "DEBUG4";
case DEBUG5_LOCK: return "DEBUG5";
case LOCK_LABELS: break; /* keep compiler happy */
};
/* only way to come here is consecutive to an internal bug */
abort();
}
void show_lock_stats()
{
int lbl;
for (lbl = 0; lbl < LOCK_LABELS; lbl++) {
if (!lock_stats[lbl].num_write_locked &&
!lock_stats[lbl].num_seek_locked &&
!lock_stats[lbl].num_read_locked) {
fprintf(stderr,
"Stats about Lock %s: not used\n",
lock_label(lbl));
continue;
}
fprintf(stderr,
"Stats about Lock %s: \n",
lock_label(lbl));
if (lock_stats[lbl].num_write_locked)
fprintf(stderr,
BUILD: thead: Fix several 32 bits compilation issues with uint64_t variables Cast uint64_t as ullong and difference between two uint64_t as llong.
2023-02-24 03:47:07 -05:00
"\t # write lock : %llu\n"
"\t # write unlock: %llu (%lld)\n"
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
"\t # wait time for write : %.3f msec\n"
"\t # wait time for write/lock: %.3f nsec\n",
BUILD: thead: Fix several 32 bits compilation issues with uint64_t variables Cast uint64_t as ullong and difference between two uint64_t as llong.
2023-02-24 03:47:07 -05:00
(ullong)lock_stats[lbl].num_write_locked,
(ullong)lock_stats[lbl].num_write_unlocked,
(llong)(lock_stats[lbl].num_write_unlocked - lock_stats[lbl].num_write_locked),
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
(double)lock_stats[lbl].nsec_wait_for_write / 1000000.0,
lock_stats[lbl].num_write_locked ? ((double)lock_stats[lbl].nsec_wait_for_write / (double)lock_stats[lbl].num_write_locked) : 0);
if (lock_stats[lbl].num_seek_locked)
fprintf(stderr,
BUILD: thead: Fix several 32 bits compilation issues with uint64_t variables Cast uint64_t as ullong and difference between two uint64_t as llong.
2023-02-24 03:47:07 -05:00
"\t # seek lock : %llu\n"
"\t # seek unlock : %llu (%lld)\n"
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
"\t # wait time for seek : %.3f msec\n"
"\t # wait time for seek/lock : %.3f nsec\n",
BUILD: thead: Fix several 32 bits compilation issues with uint64_t variables Cast uint64_t as ullong and difference between two uint64_t as llong.
2023-02-24 03:47:07 -05:00
(ullong)lock_stats[lbl].num_seek_locked,
(ullong)lock_stats[lbl].num_seek_unlocked,
(llong)(lock_stats[lbl].num_seek_unlocked - lock_stats[lbl].num_seek_locked),
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
(double)lock_stats[lbl].nsec_wait_for_seek / 1000000.0,
lock_stats[lbl].num_seek_locked ? ((double)lock_stats[lbl].nsec_wait_for_seek / (double)lock_stats[lbl].num_seek_locked) : 0);
if (lock_stats[lbl].num_read_locked)
fprintf(stderr,
BUILD: thead: Fix several 32 bits compilation issues with uint64_t variables Cast uint64_t as ullong and difference between two uint64_t as llong.
2023-02-24 03:47:07 -05:00
"\t # read lock : %llu\n"
"\t # read unlock : %llu (%lld)\n"
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
"\t # wait time for read : %.3f msec\n"
"\t # wait time for read/lock : %.3f nsec\n",
BUILD: thead: Fix several 32 bits compilation issues with uint64_t variables Cast uint64_t as ullong and difference between two uint64_t as llong.
2023-02-24 03:47:07 -05:00
(ullong)lock_stats[lbl].num_read_locked,
(ullong)lock_stats[lbl].num_read_unlocked,
(llong)(lock_stats[lbl].num_read_unlocked - lock_stats[lbl].num_read_locked),
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
(double)lock_stats[lbl].nsec_wait_for_read / 1000000.0,
lock_stats[lbl].num_read_locked ? ((double)lock_stats[lbl].nsec_wait_for_read / (double)lock_stats[lbl].num_read_locked) : 0);
}
}
void __ha_rwlock_init(struct ha_rwlock *l)
{
memset(l, 0, sizeof(struct ha_rwlock));
__RWLOCK_INIT(&l->lock);
}
void __ha_rwlock_destroy(struct ha_rwlock *l)
{
__RWLOCK_DESTROY(&l->lock);
memset(l, 0, sizeof(struct ha_rwlock));
}
void __ha_rwlock_wrlock(enum lock_label lbl, struct ha_rwlock *l,
const char *func, const char *file, int line)
{
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
ulong tbit = (ti && ti->ltid_bit) ? ti->ltid_bit : 1;
struct ha_rwlock_state *st = &l->info.st[tgid-1];
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
uint64_t start_time;
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
if ((st->cur_readers | st->cur_seeker | st->cur_writer) & tbit)
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
abort();
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_OR(&st->wait_writers, tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
start_time = now_mono_time();
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
__RWLOCK_WRLOCK(&l->lock);
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
HA_ATOMIC_ADD(&lock_stats[lbl].nsec_wait_for_write, (now_mono_time() - start_time));
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
HA_ATOMIC_INC(&lock_stats[lbl].num_write_locked);
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
st->cur_writer = tbit;
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
l->info.last_location.function = func;
l->info.last_location.file = file;
l->info.last_location.line = line;
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_AND(&st->wait_writers, ~tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
}
int __ha_rwlock_trywrlock(enum lock_label lbl, struct ha_rwlock *l,
const char *func, const char *file, int line)
{
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
ulong tbit = (ti && ti->ltid_bit) ? ti->ltid_bit : 1;
struct ha_rwlock_state *st = &l->info.st[tgid-1];
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
uint64_t start_time;
int r;
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
if ((st->cur_readers | st->cur_seeker | st->cur_writer) & tbit)
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
abort();
/* We set waiting writer because trywrlock could wait for readers to quit */
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_OR(&st->wait_writers, tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
start_time = now_mono_time();
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
r = __RWLOCK_TRYWRLOCK(&l->lock);
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
HA_ATOMIC_ADD(&lock_stats[lbl].nsec_wait_for_write, (now_mono_time() - start_time));
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
if (unlikely(r)) {
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_AND(&st->wait_writers, ~tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
return r;
}
HA_ATOMIC_INC(&lock_stats[lbl].num_write_locked);
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
st->cur_writer = tbit;
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
l->info.last_location.function = func;
l->info.last_location.file = file;
l->info.last_location.line = line;
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_AND(&st->wait_writers, ~tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
return 0;
}
void __ha_rwlock_wrunlock(enum lock_label lbl,struct ha_rwlock *l,
const char *func, const char *file, int line)
{
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
ulong tbit = (ti && ti->ltid_bit) ? ti->ltid_bit : 1;
struct ha_rwlock_state *st = &l->info.st[tgid-1];
if (unlikely(!(st->cur_writer & tbit))) {
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
/* the thread is not owning the lock for write */
abort();
}
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
st->cur_writer = 0;
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
l->info.last_location.function = func;
l->info.last_location.file = file;
l->info.last_location.line = line;
__RWLOCK_WRUNLOCK(&l->lock);
HA_ATOMIC_INC(&lock_stats[lbl].num_write_unlocked);
}
void __ha_rwlock_rdlock(enum lock_label lbl,struct ha_rwlock *l)
{
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
ulong tbit = (ti && ti->ltid_bit) ? ti->ltid_bit : 1;
struct ha_rwlock_state *st = &l->info.st[tgid-1];
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
uint64_t start_time;
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
if ((st->cur_readers | st->cur_seeker | st->cur_writer) & tbit)
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
abort();
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_OR(&st->wait_readers, tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
start_time = now_mono_time();
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
__RWLOCK_RDLOCK(&l->lock);
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
HA_ATOMIC_ADD(&lock_stats[lbl].nsec_wait_for_read, (now_mono_time() - start_time));
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
HA_ATOMIC_INC(&lock_stats[lbl].num_read_locked);
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_OR(&st->cur_readers, tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_AND(&st->wait_readers, ~tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
}
int __ha_rwlock_tryrdlock(enum lock_label lbl,struct ha_rwlock *l)
{
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
ulong tbit = (ti && ti->ltid_bit) ? ti->ltid_bit : 1;
struct ha_rwlock_state *st = &l->info.st[tgid-1];
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
int r;
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
if ((st->cur_readers | st->cur_seeker | st->cur_writer) & tbit)
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
abort();
/* try read should never wait */
r = __RWLOCK_TRYRDLOCK(&l->lock);
if (unlikely(r))
return r;
HA_ATOMIC_INC(&lock_stats[lbl].num_read_locked);
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_OR(&st->cur_readers, tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
return 0;
}
void __ha_rwlock_rdunlock(enum lock_label lbl,struct ha_rwlock *l)
{
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
ulong tbit = (ti && ti->ltid_bit) ? ti->ltid_bit : 1;
struct ha_rwlock_state *st = &l->info.st[tgid-1];
if (unlikely(!(st->cur_readers & tbit))) {
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
/* the thread is not owning the lock for read */
abort();
}
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_AND(&st->cur_readers, ~tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
__RWLOCK_RDUNLOCK(&l->lock);
HA_ATOMIC_INC(&lock_stats[lbl].num_read_unlocked);
}
void __ha_rwlock_wrtord(enum lock_label lbl, struct ha_rwlock *l,
const char *func, const char *file, int line)
{
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
ulong tbit = (ti && ti->ltid_bit) ? ti->ltid_bit : 1;
struct ha_rwlock_state *st = &l->info.st[tgid-1];
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
uint64_t start_time;
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
if ((st->cur_readers | st->cur_seeker) & tbit)
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
abort();
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
if (!(st->cur_writer & tbit))
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
abort();
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_OR(&st->wait_readers, tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
start_time = now_mono_time();
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
__RWLOCK_WRTORD(&l->lock);
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
HA_ATOMIC_ADD(&lock_stats[lbl].nsec_wait_for_read, (now_mono_time() - start_time));
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
HA_ATOMIC_INC(&lock_stats[lbl].num_read_locked);
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_OR(&st->cur_readers, tbit);
HA_ATOMIC_AND(&st->cur_writer, ~tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
l->info.last_location.function = func;
l->info.last_location.file = file;
l->info.last_location.line = line;
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_AND(&st->wait_readers, ~tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
}
void __ha_rwlock_wrtosk(enum lock_label lbl, struct ha_rwlock *l,
const char *func, const char *file, int line)
{
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
ulong tbit = (ti && ti->ltid_bit) ? ti->ltid_bit : 1;
struct ha_rwlock_state *st = &l->info.st[tgid-1];
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
uint64_t start_time;
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
if ((st->cur_readers | st->cur_seeker) & tbit)
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
abort();
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
if (!(st->cur_writer & tbit))
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
abort();
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_OR(&st->wait_seekers, tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
start_time = now_mono_time();
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
__RWLOCK_WRTOSK(&l->lock);
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
HA_ATOMIC_ADD(&lock_stats[lbl].nsec_wait_for_seek, (now_mono_time() - start_time));
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
HA_ATOMIC_INC(&lock_stats[lbl].num_seek_locked);
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_OR(&st->cur_seeker, tbit);
HA_ATOMIC_AND(&st->cur_writer, ~tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
l->info.last_location.function = func;
l->info.last_location.file = file;
l->info.last_location.line = line;
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_AND(&st->wait_seekers, ~tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
}
void __ha_rwlock_sklock(enum lock_label lbl, struct ha_rwlock *l,
const char *func, const char *file, int line)
{
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
ulong tbit = (ti && ti->ltid_bit) ? ti->ltid_bit : 1;
struct ha_rwlock_state *st = &l->info.st[tgid-1];
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
uint64_t start_time;
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
if ((st->cur_readers | st->cur_seeker | st->cur_writer) & tbit)
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
abort();
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_OR(&st->wait_seekers, tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
start_time = now_mono_time();
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
__RWLOCK_SKLOCK(&l->lock);
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
HA_ATOMIC_ADD(&lock_stats[lbl].nsec_wait_for_seek, (now_mono_time() - start_time));
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
HA_ATOMIC_INC(&lock_stats[lbl].num_seek_locked);
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_OR(&st->cur_seeker, tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
l->info.last_location.function = func;
l->info.last_location.file = file;
l->info.last_location.line = line;
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_AND(&st->wait_seekers, ~tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
}
void __ha_rwlock_sktowr(enum lock_label lbl, struct ha_rwlock *l,
const char *func, const char *file, int line)
{
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
ulong tbit = (ti && ti->ltid_bit) ? ti->ltid_bit : 1;
struct ha_rwlock_state *st = &l->info.st[tgid-1];
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
uint64_t start_time;
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
if ((st->cur_readers | st->cur_writer) & tbit)
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
abort();
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
if (!(st->cur_seeker & tbit))
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
abort();
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_OR(&st->wait_writers, tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
start_time = now_mono_time();
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
__RWLOCK_SKTOWR(&l->lock);
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
HA_ATOMIC_ADD(&lock_stats[lbl].nsec_wait_for_write, (now_mono_time() - start_time));
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
HA_ATOMIC_INC(&lock_stats[lbl].num_write_locked);
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_OR(&st->cur_writer, tbit);
HA_ATOMIC_AND(&st->cur_seeker, ~tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
l->info.last_location.function = func;
l->info.last_location.file = file;
l->info.last_location.line = line;
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_AND(&st->wait_writers, ~tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
}
void __ha_rwlock_sktord(enum lock_label lbl, struct ha_rwlock *l,
const char *func, const char *file, int line)
{
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
ulong tbit = (ti && ti->ltid_bit) ? ti->ltid_bit : 1;
struct ha_rwlock_state *st = &l->info.st[tgid-1];
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
uint64_t start_time;
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
if ((st->cur_readers | st->cur_writer) & tbit)
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
abort();
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
if (!(st->cur_seeker & tbit))
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
abort();
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_OR(&st->wait_readers, tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
start_time = now_mono_time();
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
__RWLOCK_SKTORD(&l->lock);
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
HA_ATOMIC_ADD(&lock_stats[lbl].nsec_wait_for_read, (now_mono_time() - start_time));
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
HA_ATOMIC_INC(&lock_stats[lbl].num_read_locked);
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_OR(&st->cur_readers, tbit);
HA_ATOMIC_AND(&st->cur_seeker, ~tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
l->info.last_location.function = func;
l->info.last_location.file = file;
l->info.last_location.line = line;
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_AND(&st->wait_readers, ~tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
}
void __ha_rwlock_skunlock(enum lock_label lbl,struct ha_rwlock *l,
const char *func, const char *file, int line)
{
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
ulong tbit = (ti && ti->ltid_bit) ? ti->ltid_bit : 1;
struct ha_rwlock_state *st = &l->info.st[tgid-1];
if (!(st->cur_seeker & tbit))
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
abort();
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_AND(&st->cur_seeker, ~tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
l->info.last_location.function = func;
l->info.last_location.file = file;
l->info.last_location.line = line;
__RWLOCK_SKUNLOCK(&l->lock);
HA_ATOMIC_INC(&lock_stats[lbl].num_seek_unlocked);
}
int __ha_rwlock_trysklock(enum lock_label lbl, struct ha_rwlock *l,
const char *func, const char *file, int line)
{
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
ulong tbit = (ti && ti->ltid_bit) ? ti->ltid_bit : 1;
struct ha_rwlock_state *st = &l->info.st[tgid-1];
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
uint64_t start_time;
int r;
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
if ((st->cur_readers | st->cur_seeker | st->cur_writer) & tbit)
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
abort();
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_OR(&st->wait_seekers, tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
start_time = now_mono_time();
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
r = __RWLOCK_TRYSKLOCK(&l->lock);
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
HA_ATOMIC_ADD(&lock_stats[lbl].nsec_wait_for_seek, (now_mono_time() - start_time));
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
if (likely(!r)) {
/* got the lock ! */
HA_ATOMIC_INC(&lock_stats[lbl].num_seek_locked);
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_OR(&st->cur_seeker, tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
l->info.last_location.function = func;
l->info.last_location.file = file;
l->info.last_location.line = line;
}
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_AND(&st->wait_seekers, ~tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
return r;
}
int __ha_rwlock_tryrdtosk(enum lock_label lbl, struct ha_rwlock *l,
const char *func, const char *file, int line)
{
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
ulong tbit = (ti && ti->ltid_bit) ? ti->ltid_bit : 1;
struct ha_rwlock_state *st = &l->info.st[tgid-1];
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
uint64_t start_time;
int r;
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
if ((st->cur_writer | st->cur_seeker) & tbit)
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
abort();
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
if (!(st->cur_readers & tbit))
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
abort();
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_OR(&st->wait_seekers, tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
start_time = now_mono_time();
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
r = __RWLOCK_TRYRDTOSK(&l->lock);
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
HA_ATOMIC_ADD(&lock_stats[lbl].nsec_wait_for_seek, (now_mono_time() - start_time));
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
if (likely(!r)) {
/* got the lock ! */
HA_ATOMIC_INC(&lock_stats[lbl].num_seek_locked);
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_OR(&st->cur_seeker, tbit);
HA_ATOMIC_AND(&st->cur_readers, ~tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
l->info.last_location.function = func;
l->info.last_location.file = file;
l->info.last_location.line = line;
}
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_AND(&st->wait_seekers, ~tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
return r;
}
void __spin_init(struct ha_spinlock *l)
{
memset(l, 0, sizeof(struct ha_spinlock));
__SPIN_INIT(&l->lock);
}
void __spin_destroy(struct ha_spinlock *l)
{
__SPIN_DESTROY(&l->lock);
memset(l, 0, sizeof(struct ha_spinlock));
}
void __spin_lock(enum lock_label lbl, struct ha_spinlock *l,
const char *func, const char *file, int line)
{
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
ulong tbit = (ti && ti->ltid_bit) ? ti->ltid_bit : 1;
struct ha_spinlock_state *st = &l->info.st[tgid-1];
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
uint64_t start_time;
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
if (unlikely(st->owner & tbit)) {
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
/* the thread is already owning the lock */
abort();
}
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_OR(&st->waiters, tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
start_time = now_mono_time();
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
__SPIN_LOCK(&l->lock);
MINOR: thread/debug: replace nsec_now() with now_mono_time() The two functions do exactly the same except that the second one is already provided by time.h and still defined if not available.
2021-10-05 12:48:23 -04:00
HA_ATOMIC_ADD(&lock_stats[lbl].nsec_wait_for_write, (now_mono_time() - start_time));
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
HA_ATOMIC_INC(&lock_stats[lbl].num_write_locked);
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
st->owner = tbit;
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
l->info.last_location.function = func;
l->info.last_location.file = file;
l->info.last_location.line = line;
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
HA_ATOMIC_AND(&st->waiters, ~tbit);
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
}
int __spin_trylock(enum lock_label lbl, struct ha_spinlock *l,
const char *func, const char *file, int line)
{
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
ulong tbit = (ti && ti->ltid_bit) ? ti->ltid_bit : 1;
struct ha_spinlock_state *st = &l->info.st[tgid-1];
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
int r;
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
if (unlikely(st->owner & tbit)) {
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
/* the thread is already owning the lock */
abort();
}
/* try read should never wait */
r = __SPIN_TRYLOCK(&l->lock);
if (unlikely(r))
return r;
HA_ATOMIC_INC(&lock_stats[lbl].num_write_locked);
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
st->owner = tbit;
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
l->info.last_location.function = func;
l->info.last_location.file = file;
l->info.last_location.line = line;
return 0;
}
void __spin_unlock(enum lock_label lbl, struct ha_spinlock *l,
const char *func, const char *file, int line)
{
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
ulong tbit = (ti && ti->ltid_bit) ? ti->ltid_bit : 1;
struct ha_spinlock_state *st = &l->info.st[tgid-1];
if (unlikely(!(st->owner & tbit))) {
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
/* the thread is not owning the lock */
abort();
}
MEDIUM: debug/threads: make the lock debugging take tgroups into account Since we have to use masks to verify owners/waiters, we have no other option but to have them per group. This definitely inflates the size of the locks, but this is only used for extreme debugging anyway so that's not dramatic. Thus as of now, all masks in the lock stats are local bit masks, derived from ti->ltid_bit. Since at boot ltid_bit might not be set, we just take care of this situation (since some structs are initialized under look during boot), and use bit 0 from group 0 only.
2022-07-15 11:53:10 -04:00
st->owner = 0;
REORG: thread: uninline the lock-debugging code The lock-debugging code in thread.h has no reason to be inlined. the functions are quite fat and perform a lot of operations so there's no saving keeping them inlined. Worse, most of them are in fact not inlined, resulting in a significantly bigger executable. This patch moves all this part from thread.h to thread.c. The functions are still exported in thread.h of course. This results in ~166kB less code: text data bss dec hex filename 3165938 99424 897376 4162738 3f84b2 haproxy-before 2991987 99424 897376 3988787 3cdd33 haproxy-after In addition the build time with thread debugging enabled has shrunk from 19.2 to 17.7s thanks to much less code to be parsed in thread.h that is included virtually everywhere.
2021-10-05 12:39:27 -04:00
l->info.last_location.function = func;
l->info.last_location.file = file;
l->info.last_location.line = line;
__SPIN_UNLOCK(&l->lock);
HA_ATOMIC_INC(&lock_stats[lbl].num_write_unlocked);
}
#endif // defined(DEBUG_THREAD) || defined(DEBUG_FULL)
MINOR: thread: provide an alternative to pthread's rwlock Since version 1.1.0, OpenSSL's libcrypto ignores the provided locking mechanism and uses pthread's rwlocks instead. The problem is that for some code paths (e.g. async engines) this results in a huge amount of syscalls on systems facing a bit of contention, to the point where more than 80% of the CPU can be spent in the system dealing with spinlocks just for futex_wake(). This patch provides an alternative by redefining the relevant pthread rwlocks from the low-overhead version of the progressive rw locks. This way there will be no more syscalls in case of contention, and CPU will be burnt in userland. Doing this saves massive amounts of CPU, where the locks only take 12-15% vs 80% before, which allows SSL to work much faster on large thread counts (e.g. 24 or more). The tryrdlock and trywrlock variants have been implemented using a CAS since their goal is only to succeed on no contention and never to wait. The pthread_rwlock API is complete except that the timed versions of the rdlock and wrlock do not wait and simply fall back to trylock versions. Since the gains have only been observed with async engines for now, this option remains disabled by default. It can be enabled at build time using USE_PTHREAD_EMULATION=1.
2022-07-10 04:58:57 -04:00
#if defined(USE_PTHREAD_EMULATION)
/* pthread rwlock emulation using plocks (to avoid expensive futexes).
* these are a direct mapping on Progressive Locks, with the exception that
* since there's a common unlock operation in pthreads, we need to know if
* we need to unlock for reads or writes, so we set the topmost bit to 1 when
* a write lock is acquired to indicate that a write unlock needs to be
* performed. It's not a problem since this bit will never be used given that
* haproxy won't support as many threads as the plocks.
*
* The storage is the pthread_rwlock_t cast as an ulong
*/
int pthread_rwlock_init(pthread_rwlock_t *restrict rwlock, const pthread_rwlockattr_t *restrict attr)
{
ulong *lock = (ulong *)rwlock;
*lock = 0;
return 0;
}
int pthread_rwlock_destroy(pthread_rwlock_t *rwlock)
{
ulong *lock = (ulong *)rwlock;
*lock = 0;
return 0;
}
int pthread_rwlock_rdlock(pthread_rwlock_t *rwlock)
{
pl_lorw_rdlock((unsigned long *)rwlock);
return 0;
}
int pthread_rwlock_tryrdlock(pthread_rwlock_t *rwlock)
{
return !!pl_cmpxchg((unsigned long *)rwlock, 0, PLOCK_LORW_SHR_BASE);
}
int pthread_rwlock_timedrdlock(pthread_rwlock_t *restrict rwlock, const struct timespec *restrict abstime)
{
return pthread_rwlock_tryrdlock(rwlock);
}
int pthread_rwlock_wrlock(pthread_rwlock_t *rwlock)
{
pl_lorw_wrlock((unsigned long *)rwlock);
return 0;
}
int pthread_rwlock_trywrlock(pthread_rwlock_t *rwlock)
{
return !!pl_cmpxchg((unsigned long *)rwlock, 0, PLOCK_LORW_EXC_BASE);
}
int pthread_rwlock_timedwrlock(pthread_rwlock_t *restrict rwlock, const struct timespec *restrict abstime)
{
return pthread_rwlock_trywrlock(rwlock);
}
int pthread_rwlock_unlock(pthread_rwlock_t *rwlock)
{
pl_lorw_unlock((unsigned long *)rwlock);
return 0;
}
#endif // defined(USE_PTHREAD_EMULATION)
BUILD: threads: better workaround for late loading of libgcc_s Commit 77b98220e ("BUG/MINOR: threads: work around a libgcc_s issue with chrooting") tried to address an issue with libgcc_s being loaded too late. But it turns out that the symbol used there isn't present on armhf, thus it breaks the build. Given that the issue manifests itself during pthread_exit(), the safest and most portable way to test this is to call pthread_exit(). For this we create a dummy thread which exits, during the early boot. This results in the relevant library to be loaded if needed, making sure that a later call to pthread_exit() will still work. It was tested to work fine under linux on the following platforms: glibc: - armhf - aarch64 - x86_64 - sparc64 - ppc64le musl: - mipsel Just running the code under strace easily shows the call in the dummy thread, for example here on armhf: $ strace -fe trace=file ./haproxy -v 2>&1 | grep gcc_s [pid 23055] open("/lib/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3 The code was isolated so that it's easy to #ifdef it out if needed. This should be backported where the patch above is backported (likely 2.0).
2020-09-09 11:07:54 -04:00
/* Depending on the platform and how libpthread was built, pthread_exit() may
* involve some code in libgcc_s that would be loaded on exit for the first
* time, causing aborts if the process is chrooted. It's harmless bit very
* dirty. There isn't much we can do to make sure libgcc_s is loaded only if
* needed, so what we do here is that during early boot we create a dummy
* thread that immediately exits. This will lead to libgcc_s being loaded
* during boot on the platforms where it's required.
*/
static void *dummy_thread_function(void *data)
{
pthread_exit(NULL);
return NULL;
}
static inline void preload_libgcc_s(void)
{
pthread_t dummy_thread;
BUG/MINOR: thread: add a check for pthread_create preload_libgcc_s() use pthread_create to create a thread and then call pthread_join to use it, but it doesn't check if the option is successful. So add a check to aviod potential crash.
2023-05-26 04:44:34 -04:00
if (pthread_create(&dummy_thread, NULL, dummy_thread_function, NULL) == 0)
pthread_join(dummy_thread, NULL);
BUILD: threads: better workaround for late loading of libgcc_s Commit 77b98220e ("BUG/MINOR: threads: work around a libgcc_s issue with chrooting") tried to address an issue with libgcc_s being loaded too late. But it turns out that the symbol used there isn't present on armhf, thus it breaks the build. Given that the issue manifests itself during pthread_exit(), the safest and most portable way to test this is to call pthread_exit(). For this we create a dummy thread which exits, during the early boot. This results in the relevant library to be loaded if needed, making sure that a later call to pthread_exit() will still work. It was tested to work fine under linux on the following platforms: glibc: - armhf - aarch64 - x86_64 - sparc64 - ppc64le musl: - mipsel Just running the code under strace easily shows the call in the dummy thread, for example here on armhf: $ strace -fe trace=file ./haproxy -v 2>&1 | grep gcc_s [pid 23055] open("/lib/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3 The code was isolated so that it's easy to #ifdef it out if needed. This should be backported where the patch above is backported (likely 2.0).
2020-09-09 11:07:54 -04:00
}
REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h This splits the hathreads.h file into types+macros and functions. Given that most users of this file used to include it only to get the definition of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h (i.e. types and macros). All the thread management was left to haproxy/thread.h. It's worth noting the drop of the trailing "s" in the name, to remove the permanent confusion that arises between this one and the system implementation (no "s") and the makefile's option (no "s"). For consistency, src/hathreads.c was also renamed thread.c. A number of files were updated to only include thread-t which is the one they really needed. Some future improvements are possible like replacing empty inlined functions with macros for the thread-less case, as building at -O0 disables inlining and causes these ones to be emitted. But this really is cosmetic.
2020-05-28 09:29:19 -04:00
static void __thread_init(void)
MEDIUM: threads: Add hathreads header file This file contains all functions and macros used to deal with concurrency in HAProxy. It contains all high-level function to do atomic operation (HA_ATOMIC_*). Note, for now, we rely on "__atomic" GCC builtins to do atomic operation. So HAProxy can be compiled with the thread support iff these builtins are available. It also contains wrappers around plocks to use spin or read/write locks. These wrappers are used to abstract the internal representation of the locking system and to add information to help debugging, when compiled with suitable options. To add extra info on locks, you need to add DEBUG=-DDEBUG_THREAD or DEBUG=-DDEBUG_FULL compilation option. In addition to timing info on locks, we keep info on where a lock was acquired the last time (function name, file and line). There are also the thread id and a flag to know if it is still locked or not. This will be useful to debug deadlocks.
2017-10-12 10:09:09 -04:00
{
MINOR: threads: make MAX_THREADS configurable at build time There's some value in being able to limit MAX_THREADS, either to save precious resources in embedded environments, or to protect certain deployments against accidently incorrect settings. With this patch, if MAX_THREADS is defined at build time, it will be used. However, given that LONGBITS is not a macro but is defined according to sizeof(long), we can't check the value range at build time and instead we need to perform the check at early boot time. However, the compiler is able to optimize away the constant comparisons and doesn't even emit the check code when values are correct. The output message regarding threading support was improved to report the number of threads.
2019-01-26 07:35:03 -05:00
char *ptr = NULL;
BUILD: threads: better workaround for late loading of libgcc_s Commit 77b98220e ("BUG/MINOR: threads: work around a libgcc_s issue with chrooting") tried to address an issue with libgcc_s being loaded too late. But it turns out that the symbol used there isn't present on armhf, thus it breaks the build. Given that the issue manifests itself during pthread_exit(), the safest and most portable way to test this is to call pthread_exit(). For this we create a dummy thread which exits, during the early boot. This results in the relevant library to be loaded if needed, making sure that a later call to pthread_exit() will still work. It was tested to work fine under linux on the following platforms: glibc: - armhf - aarch64 - x86_64 - sparc64 - ppc64le musl: - mipsel Just running the code under strace easily shows the call in the dummy thread, for example here on armhf: $ strace -fe trace=file ./haproxy -v 2>&1 | grep gcc_s [pid 23055] open("/lib/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3 The code was isolated so that it's easy to #ifdef it out if needed. This should be backported where the patch above is backported (likely 2.0).
2020-09-09 11:07:54 -04:00
preload_libgcc_s();
BUG/MINOR: threads: work around a libgcc_s issue with chrooting Sander Hoentjen reported another issue related to libgcc_s in issue #671. What happens is that when the old process quits, pthread_exit() calls something from libgcc_s.so after the process was chrooted, and this is the first call to that library, causing an attempt to load it. In a chroot, this fails, thus libthread aborts. The behavior widely differs between operating systems because some decided to use a static build for this library. In 2.2 this was resolved as a side effect of a workaround for the same issue with the backtrace() call, which is also in libgcc_s. This was in commit 0214b45 ("MINOR: debug: call backtrace() once upon startup"). But backtraces are not necessarily enabled, and we need something for older versions. By inspecting a significant number of ligcc_s on various gcc versions and platforms, it appears that a few functions have been present since gcc 3.0, one of which, _Unwind_Find_FDE() has no side effect (it only returns a pointer). What this patch does is that in the thread initialization code, if built with gcc >= 3.0, a call to this function is made in order to make sure that libgcc_s is loaded at start up time and that there will be no need to load it upon exit. An easy way to check which libs are loaded under Linux is : $ strace -e trace=openat ./haproxy -v With this patch applied, libgcc_s now appears during init. Sander confirmed that this patch was enough to put an end to the core dumps on exit in 2.0, so this patch should be backported there, and maybe as far as 1.8.
2020-09-02 02:04:35 -04:00
MAJOR: threads: enable one thread per CPU by default Threads have long matured by now, still for most users their usage is not trivial. It's about time to enable them by default on platforms where we know the number of CPUs bound. This patch does this, it counts the number of CPUs the process is bound to upon startup, and enables as many threads by default. Of course, "nbthread" still overrides this, but if it's not set the default behaviour is to start one thread per CPU. The default number of threads is reported in "haproxy -vv". Simply using "taskset -c" is now enough to adjust this number of threads so that there is no more need for playing with cpu-map. And thanks to the previous patches on the listener, the vast majority of configurations will not need to duplicate "bind" lines with the "process x/y" statement anymore either, so a simple config will automatically adapt to the number of processors available.
2019-01-26 08:27:06 -05:00
thread_cpus_enabled_at_boot = thread_cpus_enabled();
BUG/MINOR: init: properly detect NUMA bindings on large systems The NUMA detection code tries not to interfer with any taskset the user could have specified in init scripts. For this it compares the number of CPUs available with the number the process is bound to. However, the CPU count is retrieved after being applied an upper bound of MAX_THREADS, so if the machine has more than 64 CPUs, the comparison always fails and makes haproxy think the user has already enforced a binding, and it does not pin it anymore to a single NUMA node. This can be verified by issuing: $ socat /path/to/sock - <<< "show info" | grep thread On a dual 48-CPU machine it reports 64, implying that threads are allowed to run on the second socket: Nbthread: 64 With this fix, the function properly reports 96, and the output shows 48, indicating that a single NUMA node was used: Nbthread: 48 Of course nothing is changed when "no numa-cpu-mapping" is specified: Nbthread: 64 This can be backported to 2.4.
2023-03-09 04:12:06 -05:00
thread_cpus_enabled_at_boot = MIN(thread_cpus_enabled_at_boot, MAX_THREADS);
MAJOR: threads: enable one thread per CPU by default Threads have long matured by now, still for most users their usage is not trivial. It's about time to enable them by default on platforms where we know the number of CPUs bound. This patch does this, it counts the number of CPUs the process is bound to upon startup, and enables as many threads by default. Of course, "nbthread" still overrides this, but if it's not set the default behaviour is to start one thread per CPU. The default number of threads is reported in "haproxy -vv". Simply using "taskset -c" is now enough to adjust this number of threads so that there is no more need for playing with cpu-map. And thanks to the previous patches on the listener, the vast majority of configurations will not need to duplicate "bind" lines with the "process x/y" statement anymore either, so a simple config will automatically adapt to the number of processors available.
2019-01-26 08:27:06 -05:00
MINOR: threads: report the number of thread groups in build options haproxy -vv shows the number of threads but didn't report the number of groups, let's add it.
2022-08-06 10:44:55 -04:00
memprintf(&ptr, "Built with multi-threading support (MAX_TGROUPS=%d, MAX_THREADS=%d, default=%d).",
MAX_TGROUPS, MAX_THREADS, thread_cpus_enabled_at_boot);
MINOR: threads: make MAX_THREADS configurable at build time There's some value in being able to limit MAX_THREADS, either to save precious resources in embedded environments, or to protect certain deployments against accidently incorrect settings. With this patch, if MAX_THREADS is defined at build time, it will be used. However, given that LONGBITS is not a macro but is defined according to sizeof(long), we can't check the value range at build time and instead we need to perform the check at early boot time. However, the compiler is able to optimize away the constant comparisons and doesn't even emit the check code when values are correct. The output message regarding threading support was improved to report the number of threads.
2019-01-26 07:35:03 -05:00
hap_register_build_opts(ptr, 1);
MEDIUM: threads: Add hathreads header file This file contains all functions and macros used to deal with concurrency in HAProxy. It contains all high-level function to do atomic operation (HA_ATOMIC_*). Note, for now, we rely on "__atomic" GCC builtins to do atomic operation. So HAProxy can be compiled with the thread support iff these builtins are available. It also contains wrappers around plocks to use spin or read/write locks. These wrappers are used to abstract the internal representation of the locking system and to add information to help debugging, when compiled with suitable options. To add extra info on locks, you need to add DEBUG=-DDEBUG_THREAD or DEBUG=-DDEBUG_FULL compilation option. In addition to timing info on locks, we keep info on where a lock was acquired the last time (function name, file and line). There are also the thread id and a flag to know if it is still locked or not. This will be useful to debug deadlocks.
2017-10-12 10:09:09 -04:00
#if defined(DEBUG_THREAD) || defined(DEBUG_FULL)
memset(lock_stats, 0, sizeof(lock_stats));
#endif
}
BUILD: thread: use initcall instead of a constructor The constructor present there could be replaced with an initcall. This one is set at level STG_PREPARE because it also zeroes the lock_stats, and it's a bit odd that it could possibly have been scheduled to run after other constructors that might already preset some of these locks by accident.
2022-04-25 13:23:17 -04:00
INITCALL0(STG_PREPARE, __thread_init);
MEDIUM: threads: Add hathreads header file This file contains all functions and macros used to deal with concurrency in HAProxy. It contains all high-level function to do atomic operation (HA_ATOMIC_*). Note, for now, we rely on "__atomic" GCC builtins to do atomic operation. So HAProxy can be compiled with the thread support iff these builtins are available. It also contains wrappers around plocks to use spin or read/write locks. These wrappers are used to abstract the internal representation of the locking system and to add information to help debugging, when compiled with suitable options. To add extra info on locks, you need to add DEBUG=-DDEBUG_THREAD or DEBUG=-DDEBUG_FULL compilation option. In addition to timing info on locks, we keep info on where a lock was acquired the last time (function name, file and line). There are also the thread id and a flag to know if it is still locked or not. This will be useful to debug deadlocks.
2017-10-12 10:09:09 -04:00
BUILD: thread: properly report multi-thread support When refactoring the build option strings in 1.9, the thread support was placed outside of the ifdef block resulting in threads always being mentioned even if that was not true. Let's fix this and also mention when threads are disabled to help troubleshooting.
2018-12-15 10:48:14 -05:00
#else
CLEANUP: thread: uninline ha_tkill/ha_tkillall/ha_cpu_relax() These ones are rarely used or only to waste CPU cycles waiting, and are the last ones requiring system includes in thread.h. Let's uninline them and move them to thread.c.
2021-10-06 17:33:20 -04:00
/* send signal <sig> to thread <thr> (send to process in fact) */
void ha_tkill(unsigned int thr, int sig)
{
raise(sig);
}
/* send signal <sig> to all threads (send to process in fact) */
void ha_tkillall(int sig)
{
raise(sig);
}
void ha_thread_relax(void)
{
#ifdef _POSIX_PRIORITY_SCHEDULING
sched_yield();
#endif
}
BUILD: thread: properly report multi-thread support When refactoring the build option strings in 1.9, the thread support was placed outside of the ifdef block resulting in threads always being mentioned even if that was not true. Let's fix this and also mention when threads are disabled to help troubleshooting.
2018-12-15 10:48:14 -05:00
REGISTER_BUILD_OPTS("Built without multi-threading support (USE_THREAD not set).");
MINOR: threads: move "nbthread" parsing to hathreads.c The purpose is to make sure that all variables which directly depend on this nbthread argument are set at the right moment. For now only all_threads_mask needs to be set. It used to be set while calling thread_sync_init() which is called too late for certain checks. The same function handles threads and non-threads, which removes the need for some thread-specific knowledge from cfgparse.c.
2018-07-30 04:34:35 -04:00
#endif // USE_THREAD
MEDIUM: threads: automatically assign threads to groups This takes care of unassigned threads groups and places unassigned threads there, in a more or less balanced way. Too sparse allocations may still fail though. For now with a maximum group number fixed to 1 nothing can really fail.
2021-09-27 04:10:26 -04:00
/* scans the configured thread mapping and establishes the final one. Returns <0
* on failure, >=0 on success.
*/
int thread_map_to_groups()
{
int t, g, ut, ug;
int q, r;
MINOR: thread: add a new all_tgroups_mask variable to know about active tgroups In order to kill all_threads_mask we'll need to have an equivalent for the thread groups. The all_tgroups_mask does just this, it keeps one bit set per enabled group.
2022-06-24 09:55:11 -04:00
ulong m __maybe_unused;
MEDIUM: threads: automatically assign threads to groups This takes care of unassigned threads groups and places unassigned threads there, in a more or less balanced way. Too sparse allocations may still fail though. For now with a maximum group number fixed to 1 nothing can really fail.
2021-09-27 04:10:26 -04:00
ut = ug = 0; // unassigned threads & groups
for (t = 0; t < global.nbthread; t++) {
if (!ha_thread_info[t].tg)
ut++;
}
for (g = 0; g < global.nbtgroups; g++) {
if (!ha_tgroup_info[g].count)
ug++;
MINOR: tinfo: replace the tgid with tgid_bit in tgroup_info Now that the tgid is accessible from the thread, it's pointless to have it in the group, and it was only set but never used. However we'll soon frequently need the mask corresponding to the group ID and the risk of getting it wrong with the +1 or to shift 1 instead of 1UL is important, so let's store the tgid_bit there.
2022-06-28 11:48:07 -04:00
ha_tgroup_info[g].tgid_bit = 1UL << g;
MEDIUM: threads: automatically assign threads to groups This takes care of unassigned threads groups and places unassigned threads there, in a more or less balanced way. Too sparse allocations may still fail though. For now with a maximum group number fixed to 1 nothing can really fail.
2021-09-27 04:10:26 -04:00
}
if (ug > ut) {
ha_alert("More unassigned thread-groups (%d) than threads (%d). Please reduce thread-groups\n", ug, ut);
return -1;
}
/* look for first unassigned thread */
for (t = 0; t < global.nbthread && ha_thread_info[t].tg; t++)
;
/* assign threads to empty groups */
for (g = 0; ug && ut; ) {
/* due to sparse thread assignment we can end up with more threads
* per group on last assigned groups than former ones, so we must
* always try to pack the maximum remaining ones together first.
*/
q = ut / ug;
r = ut % ug;
if ((q + !!r) > MAX_THREADS_PER_GROUP) {
BUG/MINOR: thread: report thread and group counts in the correct order In case too many thread groups are needed for the threads, we emit an error indicating the problem. Unfortunately the threads and groups counts were reversed. This can be backported to 2.6.
2023-03-09 05:39:51 -05:00
ha_alert("Too many remaining unassigned threads (%d) for thread groups (%d). Please increase thread-groups or make sure to keep thread numbers contiguous\n", ut, ug);
MEDIUM: threads: automatically assign threads to groups This takes care of unassigned threads groups and places unassigned threads there, in a more or less balanced way. Too sparse allocations may still fail though. For now with a maximum group number fixed to 1 nothing can really fail.
2021-09-27 04:10:26 -04:00
return -1;
}
/* thread <t> is the next unassigned one. Let's look for next
* unassigned group, we know there are some left
*/
while (ut >= ug && ha_tgroup_info[g].count)
g++;
/* group g is unassigned, try to fill it with consecutive threads */
while (ut && ut >= ug && ha_tgroup_info[g].count < q + !!r &&
(!ha_tgroup_info[g].count || t == ha_tgroup_info[g].base + ha_tgroup_info[g].count)) {
if (!ha_tgroup_info[g].count) {
/* assign new group */
ha_tgroup_info[g].base = t;
ug--;
}
ha_tgroup_info[g].count++;
MINOR: tinfo: add the tgid to the thread_info struct At several places we're dereferencing the thread group just to catch the group number, and this will become even more required once we start to use per-group contexts. Let's just add the tgid in the thread_info struct to make this easier.
2022-06-28 04:49:57 -04:00
ha_thread_info[t].tgid = g + 1;
MEDIUM: threads: automatically assign threads to groups This takes care of unassigned threads groups and places unassigned threads there, in a more or less balanced way. Too sparse allocations may still fail though. For now with a maximum group number fixed to 1 nothing can really fail.
2021-09-27 04:10:26 -04:00
ha_thread_info[t].tg = &ha_tgroup_info[g];
MEDIUM: tinfo: add a dynamic thread-group context The thread group info is not sufficient to represent a thread group's current state as it's read-only. We also need something comparable to the thread context to represent the aggregate state of the threads in that group. This patch introduces ha_tgroup_ctx[] and tg_ctx for this. It's indexed on the group id and must be cache-line aligned. The thread masks that were global and that do not need to remain global were moved there (want_rdv, harmless, idle). Given that all the masks placed there now become group-specific, the associated thread mask (tid_bit) now switches to the thread's local bit (ltid_bit). Both are the same for nbtgroups 1 but will differ for other values. There's also a tg_ctx pointer in the thread so that it can be reached from other threads.
2022-06-27 10:02:24 -04:00
ha_thread_info[t].tg_ctx = &ha_tgroup_ctx[g];
MEDIUM: threads: automatically assign threads to groups This takes care of unassigned threads groups and places unassigned threads there, in a more or less balanced way. Too sparse allocations may still fail though. For now with a maximum group number fixed to 1 nothing can really fail.
2021-09-27 04:10:26 -04:00
ut--;
/* switch to next unassigned thread */
while (++t < global.nbthread && ha_thread_info[t].tg)
;
}
}
if (ut) {
ha_alert("Remaining unassigned threads found (%d) because all groups are in use. Please increase 'thread-groups', reduce 'nbthreads' or remove or extend 'thread-group' enumerations.\n", ut);
return -1;
}
MINOR: threads: set the tid, ltid and their bit in thread_cfg This will be a convenient way to communicate the thread ID and its local ID in the group, as well as their respective bits when creating the threads or when only a pointer is given.
2021-09-28 02:53:11 -04:00
for (t = 0; t < global.nbthread; t++) {
ha_thread_info[t].tid = t;
ha_thread_info[t].ltid = t - ha_thread_info[t].tg->base;
ha_thread_info[t].ltid_bit = 1UL << ha_thread_info[t].ltid;
}
MINOR: thread: add a new all_tgroups_mask variable to know about active tgroups In order to kill all_threads_mask we'll need to have an equivalent for the thread groups. The all_tgroups_mask does just this, it keeps one bit set per enabled group.
2022-06-24 09:55:11 -04:00
m = 0;
MINOR: tinfo: add the mask of enabled threads in each group In order to replace the global "all_threads_mask" we'll need to have an equivalent per group. Take this opportunity for calling it threads_enabled and make sure which ones are counted there (in case in the future we allow to stop some).
2022-06-24 09:18:49 -04:00
for (g = 0; g < global.nbtgroups; g++) {
ha_tgroup_info[g].threads_enabled = nbits(ha_tgroup_info[g].count);
BUG/MEDIUM: thread: consider secondary threads as idle+harmless during boot idle and harmless bits in the tgroup_ctx structure were not explicitly set during boot. | struct tgroup_ctx ha_tgroup_ctx[MAX_TGROUPS] = { }; As the structure is first statically initialized, .threads_harmless and .threads_idle are automatically zero- initialized by the compiler. Unfortulately, this means that such threads are not considered idle nor harmless by thread_isolate(_full)() functions until they enter the polling loop (thread_harmless_now() and thread_idle_now() are respectively called before entering the polling loop) Because of this, any attempt to call thread_isolate() or thread_isolate_full() during a startup phase with nbthreads >= 2 will cause thread_isolate to loop until every secondary threads make it through their first polling loop. If the startup phase is aborted during boot (ie: "-c" option to check the configuration), secondary threads may be initialized but will never be started (ie: they won't enter the polling loop), thus thread_isolate() could would loop forever in such cases. We can easily reveal the bug with this patch reproducer: | diff --git a/src/haproxy.c b/src/haproxy.c | index e91691658..0b733f6ee 100644 | --- a/src/haproxy.c | +++ b/src/haproxy.c | @@ -2317,6 +2317,10 @@ static void init(int argc, char **argv) | if (pr || px) { | /* At least one peer or one listener has been found */ | qfprintf(stdout, "Configuration file is valid\n"); | + printf("haproxy will loop...\n"); | + thread_isolate(); | + printf("we will never reach this\n"); | + thread_release(); | deinit_and_exit(0); | } | qfprintf(stdout, "Configuration file has no error but will not start (no listener) => exit(2).\n"); Now we start haproxy with a valid config: $> haproxy -c -f valid.conf Configuration file is valid haproxy will loop... ^C ------------------------------------------------------------------------------ This did not cause any issue so far because no early deinit paths require full thread isolation. But this may change when new features or requirements are introduced, so we should fix this before it becomes a real issue. To fix this, we explicitly assign .threads_harmless and .threads_idle to .threads_enabled value in thread_map_to_groups() function during boot. This is the proper place to do this since as long as .threads_enabled is not explicitly set, its default value is also 0 (zero-initialized by the compiler) code snippet from thread_isolate() function: ulong te = _HA_ATOMIC_LOAD(&ha_tgroup_info[tgrp].threads_enabled); ulong th = _HA_ATOMIC_LOAD(&ha_tgroup_ctx[tgrp].threads_harmless); if ((th & te) == te) break; Thus thread_isolate(_full()) won't be looping forever in thread_isolate() even if it were to be used before thread_map_to_groups() is executed. No backport needed unless this is a requirement.
2023-01-27 09:13:28 -05:00
/* for now, additional threads are not started, so we should
* consider them as harmless and idle.
* This will get automatically updated when such threads are
* started in run_thread_poll_loop()
* Without this, thread_isolate() and thread_isolate_full()
* will fail to work as long as secondary threads did not enter
* the polling loop at least once.
*/
ha_tgroup_ctx[g].threads_harmless = ha_tgroup_info[g].threads_enabled;
ha_tgroup_ctx[g].threads_idle = ha_tgroup_info[g].threads_enabled;
MINOR: thread: add a new all_tgroups_mask variable to know about active tgroups In order to kill all_threads_mask we'll need to have an equivalent for the thread groups. The all_tgroups_mask does just this, it keeps one bit set per enabled group.
2022-06-24 09:55:11 -04:00
if (!ha_tgroup_info[g].count)
continue;
m |= 1UL << g;
MINOR: tinfo: add the mask of enabled threads in each group In order to replace the global "all_threads_mask" we'll need to have an equivalent per group. Take this opportunity for calling it threads_enabled and make sure which ones are counted there (in case in the future we allow to stop some).
2022-06-24 09:18:49 -04:00
}
MINOR: thread: add a new all_tgroups_mask variable to know about active tgroups In order to kill all_threads_mask we'll need to have an equivalent for the thread groups. The all_tgroups_mask does just this, it keeps one bit set per enabled group.
2022-06-24 09:55:11 -04:00
#ifdef USE_THREAD
all_tgroups_mask = m;
#endif
MEDIUM: threads: automatically assign threads to groups This takes care of unassigned threads groups and places unassigned threads there, in a more or less balanced way. Too sparse allocations may still fail though. For now with a maximum group number fixed to 1 nothing can really fail.
2021-09-27 04:10:26 -04:00
return 0;
}
MEDIUM: listener/config: make the "thread" parser rely on thread_sets Instead of reading and storing a single group and a single mask for a "thread" directive on a bind line, we now store the complete range in a thread set that's stored in the bind_conf. The bind_parse_thread() function now just calls parse_thread_set() to complete the current set, which starts empty, and thread_resolve_group_mask() was updated to support retrieving thread group numbers or absolute thread numbers directly from the pre-filled thread_set, and continue to feed bind_tgroup and bind_thread. The CLI parsers which were pre-initialized to set the bind_tgroup to 1 cannot do it anymore as it would prevent one from restricting the thread set. Instead check_config_validity() now detects the CLI frontend and passes the info down to thread_resolve_group_mask() that will automatically use only the group 1's threads for these listeners. The same is done for the peers listeners for now. At this step it's already possible to start with all previous valid configs as well as extended ones supporting comma-delimited thread sets. In addition the parser already accepts large ranges spanning multiple groups, but since the underlying listeners infrastructure is not read, for now we're maintaining a specific check against this at the higher level of the config validity check. The patch is a bit large because thread resolution is performed in multiple steps, so we need to adjust all of them at once to preserve functional and technical consistency.
2023-01-31 13:31:27 -05:00
/* Converts a configuration thread set based on either absolute or relative
* thread numbers into a global group+mask. This is essentially for use with
* the "thread" directive on "bind" lines, where "thread 4-6,10-12" might be
* turned to "2/1-3,4/1-3". It cannot be used before the thread mapping above
* was completed and the thread group numbers configured. The thread_set is
* replaced by the resolved group-based one. It is possible to force a single
* default group for unspecified sets instead of enabling all groups by passing
CLEANUP: listener/thread: remove now unused bind_conf's bind_tgroup/bind_thread Not needed anymore since last commit, let's get rid of it.
2023-02-02 11:01:10 -05:00
* this group's non-zero value to defgrp.
MINOR: threads: add a new function to resolve config groups and masks In the configuration sometimes we'll omit a thread group number to designate a global thread number range, and sometimes we'll mention the group and designate IDs within that group. The operation is more complex than it seems due to the need to check for ranges spanning between multiple groups and determining groups from threads from bit masks and remapping bit masks between local/global. This patch adds a function to perform this operation, it takes a group and mask on input and updates them on output. It's designed to be used by "bind" lines but will likely be usable at other places if needed. For situations where specified threads do not exist in the group, we have the choice in the code between silently fixing the thread set or failing with a message. For now the better option seems to return an error, but if it turns out to be an issue we can easily change that in the future. Note that it should only happen with "x/even" when group x only has one thread.
2021-09-29 12:59:47 -04:00
*
* Returns <0 on failure, >=0 on success.
*/
CLEANUP: listener/thread: remove now unused bind_conf's bind_tgroup/bind_thread Not needed anymore since last commit, let's get rid of it.
2023-02-02 11:01:10 -05:00
int thread_resolve_group_mask(struct thread_set *ts, int defgrp, char **err)
MINOR: threads: add a new function to resolve config groups and masks In the configuration sometimes we'll omit a thread group number to designate a global thread number range, and sometimes we'll mention the group and designate IDs within that group. The operation is more complex than it seems due to the need to check for ranges spanning between multiple groups and determining groups from threads from bit masks and remapping bit masks between local/global. This patch adds a function to perform this operation, it takes a group and mask on input and updates them on output. It's designed to be used by "bind" lines but will likely be usable at other places if needed. For situations where specified threads do not exist in the group, we have the choice in the code between silently fixing the thread set or failing with a message. For now the better option seems to return an error, but if it turns out to be an issue we can easily change that in the future. Note that it should only happen with "x/even" when group x only has one thread.
2021-09-29 12:59:47 -04:00
{
BUILD: thread: fix build warnings with older gcc compilers The "{ 0 }" form to initialize an empty structure triggers build warnings on gcc 4.8, let's use the more common "{ }" instead.
2023-02-04 04:49:01 -05:00
struct thread_set new_ts = { };
MEDIUM: listener/config: make the "thread" parser rely on thread_sets Instead of reading and storing a single group and a single mask for a "thread" directive on a bind line, we now store the complete range in a thread set that's stored in the bind_conf. The bind_parse_thread() function now just calls parse_thread_set() to complete the current set, which starts empty, and thread_resolve_group_mask() was updated to support retrieving thread group numbers or absolute thread numbers directly from the pre-filled thread_set, and continue to feed bind_tgroup and bind_thread. The CLI parsers which were pre-initialized to set the bind_tgroup to 1 cannot do it anymore as it would prevent one from restricting the thread set. Instead check_config_validity() now detects the CLI frontend and passes the info down to thread_resolve_group_mask() that will automatically use only the group 1's threads for these listeners. The same is done for the peers listeners for now. At this step it's already possible to start with all previous valid configs as well as extended ones supporting comma-delimited thread sets. In addition the parser already accepts large ranges spanning multiple groups, but since the underlying listeners infrastructure is not read, for now we're maintaining a specific check against this at the higher level of the config validity check. The patch is a bit large because thread resolution is performed in multiple steps, so we need to adjust all of them at once to preserve functional and technical consistency.
2023-01-31 13:31:27 -05:00
ulong mask, imask;
uint g;
MINOR: threads: add a new function to resolve config groups and masks In the configuration sometimes we'll omit a thread group number to designate a global thread number range, and sometimes we'll mention the group and designate IDs within that group. The operation is more complex than it seems due to the need to check for ranges spanning between multiple groups and determining groups from threads from bit masks and remapping bit masks between local/global. This patch adds a function to perform this operation, it takes a group and mask on input and updates them on output. It's designed to be used by "bind" lines but will likely be usable at other places if needed. For situations where specified threads do not exist in the group, we have the choice in the code between silently fixing the thread set or failing with a message. For now the better option seems to return an error, but if it turns out to be an issue we can easily change that in the future. Note that it should only happen with "x/even" when group x only has one thread.
2021-09-29 12:59:47 -04:00
MINOR: thread: keep a bitmask of enabled groups in thread_set We're only checking for 0, 1, or >1 groups enabled there, and we'll soon need to be more precise and know quickly which groups are non-empty. Let's just replace the count with a mask of enabled groups. This will allow to quickly spot the presence of any such group in a set.
2023-03-01 05:24:29 -05:00
if (!ts->grps) {
MINOR: threads: add a new function to resolve config groups and masks In the configuration sometimes we'll omit a thread group number to designate a global thread number range, and sometimes we'll mention the group and designate IDs within that group. The operation is more complex than it seems due to the need to check for ranges spanning between multiple groups and determining groups from threads from bit masks and remapping bit masks between local/global. This patch adds a function to perform this operation, it takes a group and mask on input and updates them on output. It's designed to be used by "bind" lines but will likely be usable at other places if needed. For situations where specified threads do not exist in the group, we have the choice in the code between silently fixing the thread set or failing with a message. For now the better option seems to return an error, but if it turns out to be an issue we can easily change that in the future. Note that it should only happen with "x/even" when group x only has one thread.
2021-09-29 12:59:47 -04:00
/* unspecified group, IDs are global */
MEDIUM: listener/config: make the "thread" parser rely on thread_sets Instead of reading and storing a single group and a single mask for a "thread" directive on a bind line, we now store the complete range in a thread set that's stored in the bind_conf. The bind_parse_thread() function now just calls parse_thread_set() to complete the current set, which starts empty, and thread_resolve_group_mask() was updated to support retrieving thread group numbers or absolute thread numbers directly from the pre-filled thread_set, and continue to feed bind_tgroup and bind_thread. The CLI parsers which were pre-initialized to set the bind_tgroup to 1 cannot do it anymore as it would prevent one from restricting the thread set. Instead check_config_validity() now detects the CLI frontend and passes the info down to thread_resolve_group_mask() that will automatically use only the group 1's threads for these listeners. The same is done for the peers listeners for now. At this step it's already possible to start with all previous valid configs as well as extended ones supporting comma-delimited thread sets. In addition the parser already accepts large ranges spanning multiple groups, but since the underlying listeners infrastructure is not read, for now we're maintaining a specific check against this at the higher level of the config validity check. The patch is a bit large because thread resolution is performed in multiple steps, so we need to adjust all of them at once to preserve functional and technical consistency.
2023-01-31 13:31:27 -05:00
if (thread_set_is_empty(ts)) {
/* all threads of all groups, unless defgrp is set and
* we then set it as the only group.
*/
for (g = defgrp ? defgrp-1 : 0; g < (defgrp ? defgrp : global.nbtgroups); g++) {
new_ts.rel[g] = ha_tgroup_info[g].threads_enabled;
MINOR: thread: keep a bitmask of enabled groups in thread_set We're only checking for 0, 1, or >1 groups enabled there, and we'll soon need to be more precise and know quickly which groups are non-empty. Let's just replace the count with a mask of enabled groups. This will allow to quickly spot the presence of any such group in a set.
2023-03-01 05:24:29 -05:00
if (new_ts.rel[g])
new_ts.grps |= 1UL << g;
MINOR: threads: add a new function to resolve config groups and masks In the configuration sometimes we'll omit a thread group number to designate a global thread number range, and sometimes we'll mention the group and designate IDs within that group. The operation is more complex than it seems due to the need to check for ranges spanning between multiple groups and determining groups from threads from bit masks and remapping bit masks between local/global. This patch adds a function to perform this operation, it takes a group and mask on input and updates them on output. It's designed to be used by "bind" lines but will likely be usable at other places if needed. For situations where specified threads do not exist in the group, we have the choice in the code between silently fixing the thread set or failing with a message. For now the better option seems to return an error, but if it turns out to be an issue we can easily change that in the future. Note that it should only happen with "x/even" when group x only has one thread.
2021-09-29 12:59:47 -04:00
}
} else {
MEDIUM: listener/config: make the "thread" parser rely on thread_sets Instead of reading and storing a single group and a single mask for a "thread" directive on a bind line, we now store the complete range in a thread set that's stored in the bind_conf. The bind_parse_thread() function now just calls parse_thread_set() to complete the current set, which starts empty, and thread_resolve_group_mask() was updated to support retrieving thread group numbers or absolute thread numbers directly from the pre-filled thread_set, and continue to feed bind_tgroup and bind_thread. The CLI parsers which were pre-initialized to set the bind_tgroup to 1 cannot do it anymore as it would prevent one from restricting the thread set. Instead check_config_validity() now detects the CLI frontend and passes the info down to thread_resolve_group_mask() that will automatically use only the group 1's threads for these listeners. The same is done for the peers listeners for now. At this step it's already possible to start with all previous valid configs as well as extended ones supporting comma-delimited thread sets. In addition the parser already accepts large ranges spanning multiple groups, but since the underlying listeners infrastructure is not read, for now we're maintaining a specific check against this at the higher level of the config validity check. The patch is a bit large because thread resolution is performed in multiple steps, so we need to adjust all of them at once to preserve functional and technical consistency.
2023-01-31 13:31:27 -05:00
/* some absolute threads are set, we must remap them to
* relative ones. Each group cannot have more than
* LONGBITS threads, thus it spans at most two absolute
* blocks.
*/
for (g = 0; g < global.nbtgroups; g++) {
uint block = ha_tgroup_info[g].base / LONGBITS;
uint base = ha_tgroup_info[g].base % LONGBITS;
mask = ts->abs[block] >> base;
BUILD: thread: silence a build warning when threads are disabled When threads are disabled, the compiler complains that we might be accessing tg->abs[] out of bounds since the array is of size 1. It cannot know that the condition to do this is never met, and given that it's not in a fast path, we can make it more obvious.
2023-03-22 05:28:50 -04:00
if (base &&
(block + 1) < sizeof(ts->abs) / sizeof(ts->abs[0]) &&
ha_tgroup_info[g].count > (LONGBITS - base))
MEDIUM: listener/config: make the "thread" parser rely on thread_sets Instead of reading and storing a single group and a single mask for a "thread" directive on a bind line, we now store the complete range in a thread set that's stored in the bind_conf. The bind_parse_thread() function now just calls parse_thread_set() to complete the current set, which starts empty, and thread_resolve_group_mask() was updated to support retrieving thread group numbers or absolute thread numbers directly from the pre-filled thread_set, and continue to feed bind_tgroup and bind_thread. The CLI parsers which were pre-initialized to set the bind_tgroup to 1 cannot do it anymore as it would prevent one from restricting the thread set. Instead check_config_validity() now detects the CLI frontend and passes the info down to thread_resolve_group_mask() that will automatically use only the group 1's threads for these listeners. The same is done for the peers listeners for now. At this step it's already possible to start with all previous valid configs as well as extended ones supporting comma-delimited thread sets. In addition the parser already accepts large ranges spanning multiple groups, but since the underlying listeners infrastructure is not read, for now we're maintaining a specific check against this at the higher level of the config validity check. The patch is a bit large because thread resolution is performed in multiple steps, so we need to adjust all of them at once to preserve functional and technical consistency.
2023-01-31 13:31:27 -05:00
mask |= ts->abs[block + 1] << (LONGBITS - base);
mask &= nbits(ha_tgroup_info[g].count);
mask &= ha_tgroup_info[g].threads_enabled;
/* now the mask exactly matches the threads to be enabled
* in this group.
*/
new_ts.rel[g] |= mask;
MINOR: thread: keep a bitmask of enabled groups in thread_set We're only checking for 0, 1, or >1 groups enabled there, and we'll soon need to be more precise and know quickly which groups are non-empty. Let's just replace the count with a mask of enabled groups. This will allow to quickly spot the presence of any such group in a set.
2023-03-01 05:24:29 -05:00
if (new_ts.rel[g])
new_ts.grps |= 1UL << g;
MINOR: threads: add a new function to resolve config groups and masks In the configuration sometimes we'll omit a thread group number to designate a global thread number range, and sometimes we'll mention the group and designate IDs within that group. The operation is more complex than it seems due to the need to check for ranges spanning between multiple groups and determining groups from threads from bit masks and remapping bit masks between local/global. This patch adds a function to perform this operation, it takes a group and mask on input and updates them on output. It's designed to be used by "bind" lines but will likely be usable at other places if needed. For situations where specified threads do not exist in the group, we have the choice in the code between silently fixing the thread set or failing with a message. For now the better option seems to return an error, but if it turns out to be an issue we can easily change that in the future. Note that it should only happen with "x/even" when group x only has one thread.
2021-09-29 12:59:47 -04:00
}
}
} else {
MEDIUM: listener/config: make the "thread" parser rely on thread_sets Instead of reading and storing a single group and a single mask for a "thread" directive on a bind line, we now store the complete range in a thread set that's stored in the bind_conf. The bind_parse_thread() function now just calls parse_thread_set() to complete the current set, which starts empty, and thread_resolve_group_mask() was updated to support retrieving thread group numbers or absolute thread numbers directly from the pre-filled thread_set, and continue to feed bind_tgroup and bind_thread. The CLI parsers which were pre-initialized to set the bind_tgroup to 1 cannot do it anymore as it would prevent one from restricting the thread set. Instead check_config_validity() now detects the CLI frontend and passes the info down to thread_resolve_group_mask() that will automatically use only the group 1's threads for these listeners. The same is done for the peers listeners for now. At this step it's already possible to start with all previous valid configs as well as extended ones supporting comma-delimited thread sets. In addition the parser already accepts large ranges spanning multiple groups, but since the underlying listeners infrastructure is not read, for now we're maintaining a specific check against this at the higher level of the config validity check. The patch is a bit large because thread resolution is performed in multiple steps, so we need to adjust all of them at once to preserve functional and technical consistency.
2023-01-31 13:31:27 -05:00
/* groups were specified */
for (g = 0; g < MAX_TGROUPS; g++) {
imask = ts->rel[g];
if (!imask)
continue;
if (g >= global.nbtgroups) {
memprintf(err, "'thread' directive references non-existing thread group %u", g+1);
return -1;
}
MINOR: threads: add a new function to resolve config groups and masks In the configuration sometimes we'll omit a thread group number to designate a global thread number range, and sometimes we'll mention the group and designate IDs within that group. The operation is more complex than it seems due to the need to check for ranges spanning between multiple groups and determining groups from threads from bit masks and remapping bit masks between local/global. This patch adds a function to perform this operation, it takes a group and mask on input and updates them on output. It's designed to be used by "bind" lines but will likely be usable at other places if needed. For situations where specified threads do not exist in the group, we have the choice in the code between silently fixing the thread set or failing with a message. For now the better option seems to return an error, but if it turns out to be an issue we can easily change that in the future. Note that it should only happen with "x/even" when group x only has one thread.
2021-09-29 12:59:47 -04:00
MEDIUM: listener/config: make the "thread" parser rely on thread_sets Instead of reading and storing a single group and a single mask for a "thread" directive on a bind line, we now store the complete range in a thread set that's stored in the bind_conf. The bind_parse_thread() function now just calls parse_thread_set() to complete the current set, which starts empty, and thread_resolve_group_mask() was updated to support retrieving thread group numbers or absolute thread numbers directly from the pre-filled thread_set, and continue to feed bind_tgroup and bind_thread. The CLI parsers which were pre-initialized to set the bind_tgroup to 1 cannot do it anymore as it would prevent one from restricting the thread set. Instead check_config_validity() now detects the CLI frontend and passes the info down to thread_resolve_group_mask() that will automatically use only the group 1's threads for these listeners. The same is done for the peers listeners for now. At this step it's already possible to start with all previous valid configs as well as extended ones supporting comma-delimited thread sets. In addition the parser already accepts large ranges spanning multiple groups, but since the underlying listeners infrastructure is not read, for now we're maintaining a specific check against this at the higher level of the config validity check. The patch is a bit large because thread resolution is performed in multiple steps, so we need to adjust all of them at once to preserve functional and technical consistency.
2023-01-31 13:31:27 -05:00
/* some relative threads are set. Keep only existing ones for this group */
mask = nbits(ha_tgroup_info[g].count);
MINOR: threads: add a new function to resolve config groups and masks In the configuration sometimes we'll omit a thread group number to designate a global thread number range, and sometimes we'll mention the group and designate IDs within that group. The operation is more complex than it seems due to the need to check for ranges spanning between multiple groups and determining groups from threads from bit masks and remapping bit masks between local/global. This patch adds a function to perform this operation, it takes a group and mask on input and updates them on output. It's designed to be used by "bind" lines but will likely be usable at other places if needed. For situations where specified threads do not exist in the group, we have the choice in the code between silently fixing the thread set or failing with a message. For now the better option seems to return an error, but if it turns out to be an issue we can easily change that in the future. Note that it should only happen with "x/even" when group x only has one thread.
2021-09-29 12:59:47 -04:00
if (!(mask & imask)) {
/* no intersection between the thread group's
* threads and the bind line's.
*/
#ifdef THREAD_AUTO_ADJUST_GROUPS
unsigned long new_mask = 0;
while (imask) {
new_mask |= imask & mask;
MINOR: thread: add a simple thread_set API The purpose is to be able to store large thread sets, defined by ranges that may cross group boundaries, as well as define lists of groups and masks. The thread_set struct implements the storage, and the parser is in parse_thread_set(), with a focus on "bind" lines, but not only.
2023-01-31 13:27:48 -05:00
imask >>= ha_tgroup_info[g].count;
MINOR: threads: add a new function to resolve config groups and masks In the configuration sometimes we'll omit a thread group number to designate a global thread number range, and sometimes we'll mention the group and designate IDs within that group. The operation is more complex than it seems due to the need to check for ranges spanning between multiple groups and determining groups from threads from bit masks and remapping bit masks between local/global. This patch adds a function to perform this operation, it takes a group and mask on input and updates them on output. It's designed to be used by "bind" lines but will likely be usable at other places if needed. For situations where specified threads do not exist in the group, we have the choice in the code between silently fixing the thread set or failing with a message. For now the better option seems to return an error, but if it turns out to be an issue we can easily change that in the future. Note that it should only happen with "x/even" when group x only has one thread.
2021-09-29 12:59:47 -04:00
}
imask = new_mask;
#else
MINOR: thread: add a simple thread_set API The purpose is to be able to store large thread sets, defined by ranges that may cross group boundaries, as well as define lists of groups and masks. The thread_set struct implements the storage, and the parser is in parse_thread_set(), with a focus on "bind" lines, but not only.
2023-01-31 13:27:48 -05:00
memprintf(err, "'thread' directive only references threads not belonging to group %u", g+1);
MINOR: threads: add a new function to resolve config groups and masks In the configuration sometimes we'll omit a thread group number to designate a global thread number range, and sometimes we'll mention the group and designate IDs within that group. The operation is more complex than it seems due to the need to check for ranges spanning between multiple groups and determining groups from threads from bit masks and remapping bit masks between local/global. This patch adds a function to perform this operation, it takes a group and mask on input and updates them on output. It's designed to be used by "bind" lines but will likely be usable at other places if needed. For situations where specified threads do not exist in the group, we have the choice in the code between silently fixing the thread set or failing with a message. For now the better option seems to return an error, but if it turns out to be an issue we can easily change that in the future. Note that it should only happen with "x/even" when group x only has one thread.
2021-09-29 12:59:47 -04:00
return -1;
#endif
}
MINOR: thread: add a simple thread_set API The purpose is to be able to store large thread sets, defined by ranges that may cross group boundaries, as well as define lists of groups and masks. The thread_set struct implements the storage, and the parser is in parse_thread_set(), with a focus on "bind" lines, but not only.
2023-01-31 13:27:48 -05:00
new_ts.rel[g] = imask & mask;
MINOR: thread: keep a bitmask of enabled groups in thread_set We're only checking for 0, 1, or >1 groups enabled there, and we'll soon need to be more precise and know quickly which groups are non-empty. Let's just replace the count with a mask of enabled groups. This will allow to quickly spot the presence of any such group in a set.
2023-03-01 05:24:29 -05:00
if (new_ts.rel[g])
new_ts.grps |= 1UL << g;
MINOR: threads: add a new function to resolve config groups and masks In the configuration sometimes we'll omit a thread group number to designate a global thread number range, and sometimes we'll mention the group and designate IDs within that group. The operation is more complex than it seems due to the need to check for ranges spanning between multiple groups and determining groups from threads from bit masks and remapping bit masks between local/global. This patch adds a function to perform this operation, it takes a group and mask on input and updates them on output. It's designed to be used by "bind" lines but will likely be usable at other places if needed. For situations where specified threads do not exist in the group, we have the choice in the code between silently fixing the thread set or failing with a message. For now the better option seems to return an error, but if it turns out to be an issue we can easily change that in the future. Note that it should only happen with "x/even" when group x only has one thread.
2021-09-29 12:59:47 -04:00
}
}
MINOR: thread: add a simple thread_set API The purpose is to be able to store large thread sets, defined by ranges that may cross group boundaries, as well as define lists of groups and masks. The thread_set struct implements the storage, and the parser is in parse_thread_set(), with a focus on "bind" lines, but not only.
2023-01-31 13:27:48 -05:00
/* update the thread_set */
MINOR: tinfo: make thread_set functions return nth group/mask instead of first thread_set_first_group() and thread_set_first_tmask() were modified and renamed to instead return the number and mask of the nth group. Passing zero continues to return the first one, but it will be more convenient to use this way when building shards.
2023-02-27 05:27:38 -05:00
if (!thread_set_nth_group(&new_ts, 0)) {
MINOR: thread: add a simple thread_set API The purpose is to be able to store large thread sets, defined by ranges that may cross group boundaries, as well as define lists of groups and masks. The thread_set struct implements the storage, and the parser is in parse_thread_set(), with a focus on "bind" lines, but not only.
2023-01-31 13:27:48 -05:00
memprintf(err, "'thread' directive only references non-existing threads");
return -1;
}
*ts = new_ts;
return 0;
}
/* Parse a string representing a thread set in one of the following forms:
*
* - { "all" | "odd" | "even" | <abs_num> [ "-" <abs_num> ] }[,...]
* => these are (lists of) absolute thread numbers
*
* - <tgnum> "/" { "all" | "odd" | "even" | <rel_num> [ "-" <rel_num> ][,...]
* => these are (lists of) per-group relative thread numbers. All numbers
* must be lower than or equal to LONGBITS. When multiple list elements
* are provided, each of them must contain the thread group number.
*
* Minimum value for a thread or group number is always 1. Maximum value for an
* absolute thread number is MAX_THREADS, maximum value for a relative thread
* number is MAX_THREADS_PER_GROUP, an maximum value for a thread group is
* MAX_TGROUPS. "all", "even" and "odd" will be bound by MAX_THREADS and/or
* MAX_THREADS_PER_GROUP in any case. In ranges, a missing digit before "-"
* is implicitly 1, and a missing digit after "-" is implicitly the highest of
* its class. As such "-" is equivalent to "all", allowing to build strings
* such as "${MIN}-${MAX}" where both MIN and MAX are optional.
*
* It is not valid to mix absolute and relative numbers. As such:
* - all valid (all absolute threads)
* - 12-19,24-31 valid (abs threads 12 to 19 and 24 to 31)
* - 1/all valid (all 32 or 64 threads of group 1)
* - 1/1-4,1/8-10,2/1 valid
* - 1/1-4,8-10 invalid (mixes relatve "1/1-4" with absolute "8-10")
* - 1-4,8-10,2/1 invalid (mixes absolute "1-4,8-10" with relative "2/1")
* - 1/odd-4 invalid (mixes range with boundary)
*
* The target thread set is *completed* with supported threads, which means
* that it's the caller's responsibility for pre-initializing it. If the target
* thread set is NULL, it's not updated and the function only verifies that the
* input parses.
*
* On success, it returns 0. otherwise it returns non-zero with an error
* message in <err>.
*/
int parse_thread_set(const char *arg, struct thread_set *ts, char **err)
{
const char *set;
const char *sep;
int v, min, max, tg;
int is_rel;
/* search for the first delimiter (',', '-' or '/') to decide whether
* we're facing an absolute or relative form. The relative form always
* starts with a number followed by a slash.
*/
for (sep = arg; isdigit((uchar)*sep); sep++)
;
is_rel = (/*sep > arg &&*/ *sep == '/'); /* relative form */
/* from there we have to cut the thread spec around commas */
set = arg;
tg = 0;
while (*set) {
/* note: we can't use strtol() here because "-3" would parse as
* (-3) while we want to stop before the "-", so we find the
* separator ourselves and rely on atoi() whose value we may
* ignore depending where the separator is.
*/
for (sep = set; isdigit((uchar)*sep); sep++)
;
if (sep != set && *sep && *sep != '/' && *sep != '-' && *sep != ',') {
memprintf(err, "invalid character '%c' in thread set specification: '%s'.", *sep, set);
return -1;
}
v = (sep != set) ? atoi(set) : 0;
/* Now we know that the string is made of an optional series of digits
* optionally followed by one of the delimiters above, or that it
* starts with a different character.
*/
/* first, let's search for the thread group (digits before '/') */
if (tg || !is_rel) {
/* thread group already specified or not expected if absolute spec */
if (*sep == '/') {
if (tg)
memprintf(err, "redundant thread group specification '%s' for group %d", set, tg);
else
memprintf(err, "group-relative thread specification '%s' is not permitted after a absolute thread range.", set);
return -1;
}
} else {
/* this is a group-relative spec, first field is the group number */
if (sep == set && *sep == '/') {
memprintf(err, "thread group number expected before '%s'.", set);
return -1;
}
if (*sep != '/') {
memprintf(err, "absolute thread specification '%s' is not permitted after a group-relative thread range.", set);
return -1;
}
if (v < 1 || v > MAX_TGROUPS) {
memprintf(err, "invalid thread group number '%d', permitted range is 1..%d in '%s'.", v, MAX_TGROUPS, set);
return -1;
}
tg = v;
/* skip group number and go on with set,sep,v as if
* there was no group number.
*/
set = sep + 1;
continue;
}
/* Now 'set' starts at the min thread number, whose value is in v if any,
* and preset the max to it, unless the range is filled at once via "all"
* (stored as 1:0), "odd" (stored as) 1:-1, or "even" (stored as 1:-2).
* 'sep' points to the next non-digit which may be set itself e.g. for
* "all" etc or "-xx".
*/
if (!*set) {
/* empty set sets no restriction */
min = 1;
max = is_rel ? MAX_THREADS_PER_GROUP : MAX_THREADS;
}
else {
if (sep != set && *sep && *sep != '-' && *sep != ',') {
CLEANUP: assorted typo fixes in the code and comments This is 35th iteration of typo fixes
2023-04-01 06:26:42 -04:00
// Only delimiters are permitted around digits.
MINOR: thread: add a simple thread_set API The purpose is to be able to store large thread sets, defined by ranges that may cross group boundaries, as well as define lists of groups and masks. The thread_set struct implements the storage, and the parser is in parse_thread_set(), with a focus on "bind" lines, but not only.
2023-01-31 13:27:48 -05:00
memprintf(err, "invalid character '%c' in thread set specification: '%s'.", *sep, set);
return -1;
}
/* for non-digits, find next delim */
for (; *sep && *sep != '-' && *sep != ','; sep++)
;
min = max = 1;
if (sep != set) {
/* non-empty first thread */
if (isteq(ist2(set, sep-set), ist("all")))
max = 0;
else if (isteq(ist2(set, sep-set), ist("odd")))
max = -1;
else if (isteq(ist2(set, sep-set), ist("even")))
max = -2;
else if (v)
min = max = v;
else
max = min = 0; // throw an error below
}
if (min < 1 || min > MAX_THREADS || (is_rel && min > MAX_THREADS_PER_GROUP)) {
memprintf(err, "invalid first thread number '%s', permitted range is 1..%d, or 'all', 'odd', 'even'.",
set, is_rel ? MAX_THREADS_PER_GROUP : MAX_THREADS);
return -1;
}
/* is this a range ? */
if (*sep == '-') {
if (min != max) {
memprintf(err, "extraneous range after 'all', 'odd' or 'even': '%s'.", set);
return -1;
}
/* this is a seemingly valid range, there may be another number */
for (set = ++sep; isdigit((uchar)*sep); sep++)
;
v = atoi(set);
if (sep == set) { // no digit: to the max
max = is_rel ? MAX_THREADS_PER_GROUP : MAX_THREADS;
if (*sep && *sep != ',')
max = 0; // throw an error below
} else
max = v;
if (max < 1 || max > MAX_THREADS || (is_rel && max > MAX_THREADS_PER_GROUP)) {
memprintf(err, "invalid last thread number '%s', permitted range is 1..%d.",
set, is_rel ? MAX_THREADS_PER_GROUP : MAX_THREADS);
return -1;
}
}
/* here sep points to the first non-digit after the thread spec,
* must be a valid delimiter.
*/
if (*sep && *sep != ',') {
memprintf(err, "invalid character '%c' after thread set specification: '%s'.", *sep, set);
return -1;
}
}
/* store values */
if (ts) {
if (is_rel) {
/* group-relative thread numbers */
MINOR: thread: keep a bitmask of enabled groups in thread_set We're only checking for 0, 1, or >1 groups enabled there, and we'll soon need to be more precise and know quickly which groups are non-empty. Let's just replace the count with a mask of enabled groups. This will allow to quickly spot the presence of any such group in a set.
2023-03-01 05:24:29 -05:00
ts->grps |= 1UL << (tg - 1);
MINOR: thread: add a simple thread_set API The purpose is to be able to store large thread sets, defined by ranges that may cross group boundaries, as well as define lists of groups and masks. The thread_set struct implements the storage, and the parser is in parse_thread_set(), with a focus on "bind" lines, but not only.
2023-01-31 13:27:48 -05:00
if (max >= min) {
for (v = min; v <= max; v++)
BUG/MEDIUM: thread: fix extraneous shift in the thread_set parser Aurlien reported a bug making a statement such as "thread 2-2" fail for a config made of exactly 2 threads. What happens is that the parser for the "thread" keyword scans a range of thread numbers from either 1..64 or 0,-1,-2 for special values, and presets the bit masks accordingly in the thread set, except that due to the 1..64 range, the shift length must be reduced by one. Not doing this causes empty masks for single-bit values that are exactly equal to the number of threads in the group and fails to properly parse. No backport is needed as this was introduced in 2.8-dev3 by commit bef43dfa6 ("MINOR: thread: add a simple thread_set API").
2023-02-06 12:01:50 -05:00
ts->rel[tg - 1] |= 1UL << (v - 1);
MINOR: thread: add a simple thread_set API The purpose is to be able to store large thread sets, defined by ranges that may cross group boundaries, as well as define lists of groups and masks. The thread_set struct implements the storage, and the parser is in parse_thread_set(), with a focus on "bind" lines, but not only.
2023-01-31 13:27:48 -05:00
} else {
memset(&ts->rel[tg - 1],
(max == 0) ? 0xff /* all */ : (max == -1) ? 0x55 /* odd */: 0xaa /* even */,
sizeof(ts->rel[tg - 1]));
}
} else {
/* absolute thread numbers */
if (max >= min) {
for (v = min; v <= max; v++)
BUG/MEDIUM: thread: fix extraneous shift in the thread_set parser Aurlien reported a bug making a statement such as "thread 2-2" fail for a config made of exactly 2 threads. What happens is that the parser for the "thread" keyword scans a range of thread numbers from either 1..64 or 0,-1,-2 for special values, and presets the bit masks accordingly in the thread set, except that due to the 1..64 range, the shift length must be reduced by one. Not doing this causes empty masks for single-bit values that are exactly equal to the number of threads in the group and fails to properly parse. No backport is needed as this was introduced in 2.8-dev3 by commit bef43dfa6 ("MINOR: thread: add a simple thread_set API").
2023-02-06 12:01:50 -05:00
ts->abs[(v - 1) / LONGBITS] |= 1UL << ((v - 1) % LONGBITS);
MINOR: thread: add a simple thread_set API The purpose is to be able to store large thread sets, defined by ranges that may cross group boundaries, as well as define lists of groups and masks. The thread_set struct implements the storage, and the parser is in parse_thread_set(), with a focus on "bind" lines, but not only.
2023-01-31 13:27:48 -05:00
} else {
memset(&ts->abs,
(max == 0) ? 0xff /* all */ : (max == -1) ? 0x55 /* odd */: 0xaa /* even */,
sizeof(ts->abs));
}
}
}
set = *sep ? sep + 1 : sep;
tg = 0;
}
return 0;
MINOR: threads: add a new function to resolve config groups and masks In the configuration sometimes we'll omit a thread group number to designate a global thread number range, and sometimes we'll mention the group and designate IDs within that group. The operation is more complex than it seems due to the need to check for ranges spanning between multiple groups and determining groups from threads from bit masks and remapping bit masks between local/global. This patch adds a function to perform this operation, it takes a group and mask on input and updates them on output. It's designed to be used by "bind" lines but will likely be usable at other places if needed. For situations where specified threads do not exist in the group, we have the choice in the code between silently fixing the thread set or failing with a message. For now the better option seems to return an error, but if it turns out to be an issue we can easily change that in the future. Note that it should only happen with "x/even" when group x only has one thread.
2021-09-29 12:59:47 -04:00
}
MINOR: config: use a standard parser for the "nbthread" keyword Probably because of some copy-paste from "nbproc", "nbthread" used to be parsed in cfgparse instead of using a registered parser. Let's fix this to clean up the code base now.
2021-09-22 05:55:22 -04:00
/* Parse the "nbthread" global directive, which takes an integer argument that
* contains the desired number of threads.
MINOR: threads: move "nbthread" parsing to hathreads.c The purpose is to make sure that all variables which directly depend on this nbthread argument are set at the right moment. For now only all_threads_mask needs to be set. It used to be set while calling thread_sync_init() which is called too late for certain checks. The same function handles threads and non-threads, which removes the need for some thread-specific knowledge from cfgparse.c.
2018-07-30 04:34:35 -04:00
*/
MINOR: config: use a standard parser for the "nbthread" keyword Probably because of some copy-paste from "nbproc", "nbthread" used to be parsed in cfgparse instead of using a registered parser. Let's fix this to clean up the code base now.
2021-09-22 05:55:22 -04:00
static int cfg_parse_nbthread(char **args, int section_type, struct proxy *curpx,
const struct proxy *defpx, const char *file, int line,
char **err)
MINOR: threads: move "nbthread" parsing to hathreads.c The purpose is to make sure that all variables which directly depend on this nbthread argument are set at the right moment. For now only all_threads_mask needs to be set. It used to be set while calling thread_sync_init() which is called too late for certain checks. The same function handles threads and non-threads, which removes the need for some thread-specific knowledge from cfgparse.c.
2018-07-30 04:34:35 -04:00
{
long nbthread;
char *errptr;
MINOR: config: use a standard parser for the "nbthread" keyword Probably because of some copy-paste from "nbproc", "nbthread" used to be parsed in cfgparse instead of using a registered parser. Let's fix this to clean up the code base now.
2021-09-22 05:55:22 -04:00
if (too_many_args(1, args, err, NULL))
return -1;
MEDIUM: thread: Restric nbthread/thread-group(s) to very first global sections nbhread, thead-group and thread-groups directives must only be defined in very first global sections. It means no other section must have been parsed before. Indeed, some parts of the configuratio depends on the value of these settings and it is undefined to change them after.
2022-11-18 09:52:58 -05:00
if (non_global_section_parsed == 1) {
memprintf(err, "'%s' not allowed if a non-global section was previously defined. This parameter must be declared in the first global section", args[0]);
return -1;
}
MINOR: config: use a standard parser for the "nbthread" keyword Probably because of some copy-paste from "nbproc", "nbthread" used to be parsed in cfgparse instead of using a registered parser. Let's fix this to clean up the code base now.
2021-09-22 05:55:22 -04:00
nbthread = strtol(args[1], &errptr, 10);
if (!*args[1] || *errptr) {
memprintf(err, "'%s' passed a missing or unparsable integer value in '%s'", args[0], args[1]);
return -1;
MINOR: threads: move "nbthread" parsing to hathreads.c The purpose is to make sure that all variables which directly depend on this nbthread argument are set at the right moment. For now only all_threads_mask needs to be set. It used to be set while calling thread_sync_init() which is called too late for certain checks. The same function handles threads and non-threads, which removes the need for some thread-specific knowledge from cfgparse.c.
2018-07-30 04:34:35 -04:00
}
#ifndef USE_THREAD
if (nbthread != 1) {
MINOR: config: use a standard parser for the "nbthread" keyword Probably because of some copy-paste from "nbproc", "nbthread" used to be parsed in cfgparse instead of using a registered parser. Let's fix this to clean up the code base now.
2021-09-22 05:55:22 -04:00
memprintf(err, "'%s' specified with a value other than 1 while HAProxy is not compiled with threads support. Please check build options for USE_THREAD", args[0]);
return -1;
MINOR: threads: move "nbthread" parsing to hathreads.c The purpose is to make sure that all variables which directly depend on this nbthread argument are set at the right moment. For now only all_threads_mask needs to be set. It used to be set while calling thread_sync_init() which is called too late for certain checks. The same function handles threads and non-threads, which removes the need for some thread-specific knowledge from cfgparse.c.
2018-07-30 04:34:35 -04:00
}
#else
if (nbthread < 1 || nbthread > MAX_THREADS) {
MINOR: config: use a standard parser for the "nbthread" keyword Probably because of some copy-paste from "nbproc", "nbthread" used to be parsed in cfgparse instead of using a registered parser. Let's fix this to clean up the code base now.
2021-09-22 05:55:22 -04:00
memprintf(err, "'%s' value must be between 1 and %d (was %ld)", args[0], MAX_THREADS, nbthread);
return -1;
MINOR: threads: move "nbthread" parsing to hathreads.c The purpose is to make sure that all variables which directly depend on this nbthread argument are set at the right moment. For now only all_threads_mask needs to be set. It used to be set while calling thread_sync_init() which is called too late for certain checks. The same function handles threads and non-threads, which removes the need for some thread-specific knowledge from cfgparse.c.
2018-07-30 04:34:35 -04:00
}
MEDIUM: threads: Add hathreads header file This file contains all functions and macros used to deal with concurrency in HAProxy. It contains all high-level function to do atomic operation (HA_ATOMIC_*). Note, for now, we rely on "__atomic" GCC builtins to do atomic operation. So HAProxy can be compiled with the thread support iff these builtins are available. It also contains wrappers around plocks to use spin or read/write locks. These wrappers are used to abstract the internal representation of the locking system and to add information to help debugging, when compiled with suitable options. To add extra info on locks, you need to add DEBUG=-DDEBUG_THREAD or DEBUG=-DDEBUG_FULL compilation option. In addition to timing info on locks, we keep info on where a lock was acquired the last time (function name, file and line). There are also the thread id and a flag to know if it is still locked or not. This will be useful to debug deadlocks.
2017-10-12 10:09:09 -04:00
#endif
MINOR: config: use a standard parser for the "nbthread" keyword Probably because of some copy-paste from "nbproc", "nbthread" used to be parsed in cfgparse instead of using a registered parser. Let's fix this to clean up the code base now.
2021-09-22 05:55:22 -04:00
HA_DIAG_WARNING_COND(global.nbthread,
MINOR: global: add a new "thread-groups" directive This is used to configure the number of thread groups. For now it can only be 1.
2021-09-22 06:07:23 -04:00
"parsing [%s:%d] : '%s' is already defined and will be overridden.\n",
file, line, args[0]);
MINOR: config: use a standard parser for the "nbthread" keyword Probably because of some copy-paste from "nbproc", "nbthread" used to be parsed in cfgparse instead of using a registered parser. Let's fix this to clean up the code base now.
2021-09-22 05:55:22 -04:00
global.nbthread = nbthread;
return 0;
MINOR: threads: move "nbthread" parsing to hathreads.c The purpose is to make sure that all variables which directly depend on this nbthread argument are set at the right moment. For now only all_threads_mask needs to be set. It used to be set while calling thread_sync_init() which is called too late for certain checks. The same function handles threads and non-threads, which removes the need for some thread-specific knowledge from cfgparse.c.
2018-07-30 04:34:35 -04:00
}
MINOR: config: use a standard parser for the "nbthread" keyword Probably because of some copy-paste from "nbproc", "nbthread" used to be parsed in cfgparse instead of using a registered parser. Let's fix this to clean up the code base now.
2021-09-22 05:55:22 -04:00
MINOR: global: add a new "thread-group" directive This registers a mapping of threads to groups by enumerating for each thread what group it belongs to, and marking the group as assigned. It takes care of checking for redefinitions, overlaps, and holes. It supports both individual numbers and ranges. The thread group is referenced from the thread config.
2021-09-27 07:55:10 -04:00
/* Parse the "thread-group" global directive, which takes an integer argument
* that designates a thread group, and a list of threads to put into that group.
*/
static int cfg_parse_thread_group(char **args, int section_type, struct proxy *curpx,
const struct proxy *defpx, const char *file, int line,
char **err)
{
char *errptr;
long tnum, tend, tgroup;
int arg, tot;
MEDIUM: thread: Restric nbthread/thread-group(s) to very first global sections nbhread, thead-group and thread-groups directives must only be defined in very first global sections. It means no other section must have been parsed before. Indeed, some parts of the configuratio depends on the value of these settings and it is undefined to change them after.
2022-11-18 09:52:58 -05:00
if (non_global_section_parsed == 1) {
memprintf(err, "'%s' not allowed if a non-global section was previously defined. This parameter must be declared in the first global section", args[0]);
return -1;
}
MINOR: global: add a new "thread-group" directive This registers a mapping of threads to groups by enumerating for each thread what group it belongs to, and marking the group as assigned. It takes care of checking for redefinitions, overlaps, and holes. It supports both individual numbers and ranges. The thread group is referenced from the thread config.
2021-09-27 07:55:10 -04:00
tgroup = strtol(args[1], &errptr, 10);
if (!*args[1] || *errptr) {
memprintf(err, "'%s' passed a missing or unparsable integer value in '%s'", args[0], args[1]);
return -1;
}
if (tgroup < 1 || tgroup > MAX_TGROUPS) {
memprintf(err, "'%s' thread-group number must be between 1 and %d (was %ld)", args[0], MAX_TGROUPS, tgroup);
return -1;
}
/* look for a preliminary definition of any thread pointing to this
* group, and remove them.
*/
if (ha_tgroup_info[tgroup-1].count) {
ha_warning("parsing [%s:%d] : '%s %ld' was already defined and will be overridden.\n",
file, line, args[0], tgroup);
for (tnum = ha_tgroup_info[tgroup-1].base;
tnum < ha_tgroup_info[tgroup-1].base + ha_tgroup_info[tgroup-1].count;
tnum++) {
MINOR: tinfo: add the tgid to the thread_info struct At several places we're dereferencing the thread group just to catch the group number, and this will become even more required once we start to use per-group contexts. Let's just add the tgid in the thread_info struct to make this easier.
2022-06-28 04:49:57 -04:00
if (ha_thread_info[tnum-1].tg == &ha_tgroup_info[tgroup-1]) {
MINOR: global: add a new "thread-group" directive This registers a mapping of threads to groups by enumerating for each thread what group it belongs to, and marking the group as assigned. It takes care of checking for redefinitions, overlaps, and holes. It supports both individual numbers and ranges. The thread group is referenced from the thread config.
2021-09-27 07:55:10 -04:00
ha_thread_info[tnum-1].tg = NULL;
MINOR: tinfo: add the tgid to the thread_info struct At several places we're dereferencing the thread group just to catch the group number, and this will become even more required once we start to use per-group contexts. Let's just add the tgid in the thread_info struct to make this easier.
2022-06-28 04:49:57 -04:00
ha_thread_info[tnum-1].tgid = 0;
MEDIUM: tinfo: add a dynamic thread-group context The thread group info is not sufficient to represent a thread group's current state as it's read-only. We also need something comparable to the thread context to represent the aggregate state of the threads in that group. This patch introduces ha_tgroup_ctx[] and tg_ctx for this. It's indexed on the group id and must be cache-line aligned. The thread masks that were global and that do not need to remain global were moved there (want_rdv, harmless, idle). Given that all the masks placed there now become group-specific, the associated thread mask (tid_bit) now switches to the thread's local bit (ltid_bit). Both are the same for nbtgroups 1 but will differ for other values. There's also a tg_ctx pointer in the thread so that it can be reached from other threads.
2022-06-27 10:02:24 -04:00
ha_thread_info[tnum-1].tg_ctx = NULL;
MINOR: tinfo: add the tgid to the thread_info struct At several places we're dereferencing the thread group just to catch the group number, and this will become even more required once we start to use per-group contexts. Let's just add the tgid in the thread_info struct to make this easier.
2022-06-28 04:49:57 -04:00
}
MINOR: global: add a new "thread-group" directive This registers a mapping of threads to groups by enumerating for each thread what group it belongs to, and marking the group as assigned. It takes care of checking for redefinitions, overlaps, and holes. It supports both individual numbers and ranges. The thread group is referenced from the thread config.
2021-09-27 07:55:10 -04:00
}
ha_tgroup_info[tgroup-1].count = ha_tgroup_info[tgroup-1].base = 0;
}
tot = 0;
for (arg = 2; args[arg] && *args[arg]; arg++) {
tend = tnum = strtol(args[arg], &errptr, 10);
if (*errptr == '-')
tend = strtol(errptr + 1, &errptr, 10);
if (*errptr || tnum < 1 || tend < 1 || tnum > MAX_THREADS || tend > MAX_THREADS) {
memprintf(err, "'%s %ld' passed an unparsable or invalid thread number '%s' (valid range is 1 to %d)", args[0], tgroup, args[arg], MAX_THREADS);
return -1;
}
for(; tnum <= tend; tnum++) {
if (ha_thread_info[tnum-1].tg == &ha_tgroup_info[tgroup-1]) {
ha_warning("parsing [%s:%d] : '%s %ld': thread %ld assigned more than once on the same line.\n",
file, line, args[0], tgroup, tnum);
} else if (ha_thread_info[tnum-1].tg) {
ha_warning("parsing [%s:%d] : '%s %ld': thread %ld was previously assigned to thread group %ld and will be overridden.\n",
file, line, args[0], tgroup, tnum,
(long)(ha_thread_info[tnum-1].tg - &ha_tgroup_info[0] + 1));
}
if (!ha_tgroup_info[tgroup-1].count) {
ha_tgroup_info[tgroup-1].base = tnum-1;
ha_tgroup_info[tgroup-1].count = 1;
}
else if (tnum >= ha_tgroup_info[tgroup-1].base + ha_tgroup_info[tgroup-1].count) {
ha_tgroup_info[tgroup-1].count = tnum - ha_tgroup_info[tgroup-1].base;
}
else if (tnum < ha_tgroup_info[tgroup-1].base) {
ha_tgroup_info[tgroup-1].count += ha_tgroup_info[tgroup-1].base - tnum-1;
ha_tgroup_info[tgroup-1].base = tnum - 1;
}
MINOR: tinfo: add the tgid to the thread_info struct At several places we're dereferencing the thread group just to catch the group number, and this will become even more required once we start to use per-group contexts. Let's just add the tgid in the thread_info struct to make this easier.
2022-06-28 04:49:57 -04:00
ha_thread_info[tnum-1].tgid = tgroup;
MINOR: global: add a new "thread-group" directive This registers a mapping of threads to groups by enumerating for each thread what group it belongs to, and marking the group as assigned. It takes care of checking for redefinitions, overlaps, and holes. It supports both individual numbers and ranges. The thread group is referenced from the thread config.
2021-09-27 07:55:10 -04:00
ha_thread_info[tnum-1].tg = &ha_tgroup_info[tgroup-1];
MEDIUM: tinfo: add a dynamic thread-group context The thread group info is not sufficient to represent a thread group's current state as it's read-only. We also need something comparable to the thread context to represent the aggregate state of the threads in that group. This patch introduces ha_tgroup_ctx[] and tg_ctx for this. It's indexed on the group id and must be cache-line aligned. The thread masks that were global and that do not need to remain global were moved there (want_rdv, harmless, idle). Given that all the masks placed there now become group-specific, the associated thread mask (tid_bit) now switches to the thread's local bit (ltid_bit). Both are the same for nbtgroups 1 but will differ for other values. There's also a tg_ctx pointer in the thread so that it can be reached from other threads.
2022-06-27 10:02:24 -04:00
ha_thread_info[tnum-1].tg_ctx = &ha_tgroup_ctx[tgroup-1];
MINOR: global: add a new "thread-group" directive This registers a mapping of threads to groups by enumerating for each thread what group it belongs to, and marking the group as assigned. It takes care of checking for redefinitions, overlaps, and holes. It supports both individual numbers and ranges. The thread group is referenced from the thread config.
2021-09-27 07:55:10 -04:00
tot++;
}
}
if (ha_tgroup_info[tgroup-1].count > tot) {
memprintf(err, "'%s %ld' assigned sparse threads, only contiguous supported", args[0], tgroup);
return -1;
}
if (ha_tgroup_info[tgroup-1].count > MAX_THREADS_PER_GROUP) {
memprintf(err, "'%s %ld' assigned too many threads (%d, max=%d)", args[0], tgroup, tot, MAX_THREADS_PER_GROUP);
return -1;
}
return 0;
}
MINOR: global: add a new "thread-groups" directive This is used to configure the number of thread groups. For now it can only be 1.
2021-09-22 06:07:23 -04:00
/* Parse the "thread-groups" global directive, which takes an integer argument
* that contains the desired number of thread groups.
*/
static int cfg_parse_thread_groups(char **args, int section_type, struct proxy *curpx,
const struct proxy *defpx, const char *file, int line,
char **err)
{
long nbtgroups;
char *errptr;
if (too_many_args(1, args, err, NULL))
return -1;
MEDIUM: thread: Restric nbthread/thread-group(s) to very first global sections nbhread, thead-group and thread-groups directives must only be defined in very first global sections. It means no other section must have been parsed before. Indeed, some parts of the configuratio depends on the value of these settings and it is undefined to change them after.
2022-11-18 09:52:58 -05:00
if (non_global_section_parsed == 1) {
memprintf(err, "'%s' not allowed if a non-global section was previously defined. This parameter must be declared in the first global section", args[0]);
return -1;
}
MINOR: global: add a new "thread-groups" directive This is used to configure the number of thread groups. For now it can only be 1.
2021-09-22 06:07:23 -04:00
nbtgroups = strtol(args[1], &errptr, 10);
if (!*args[1] || *errptr) {
memprintf(err, "'%s' passed a missing or unparsable integer value in '%s'", args[0], args[1]);
return -1;
}
#ifndef USE_THREAD
if (nbtgroups != 1) {
memprintf(err, "'%s' specified with a value other than 1 while HAProxy is not compiled with threads support. Please check build options for USE_THREAD", args[0]);
return -1;
}
#else
if (nbtgroups < 1 || nbtgroups > MAX_TGROUPS) {
memprintf(err, "'%s' value must be between 1 and %d (was %ld)", args[0], MAX_TGROUPS, nbtgroups);
return -1;
}
#endif
HA_DIAG_WARNING_COND(global.nbtgroups,
"parsing [%s:%d] : '%s' is already defined and will be overridden.\n",
file, line, args[0]);
global.nbtgroups = nbtgroups;
return 0;
}
MINOR: config: use a standard parser for the "nbthread" keyword Probably because of some copy-paste from "nbproc", "nbthread" used to be parsed in cfgparse instead of using a registered parser. Let's fix this to clean up the code base now.
2021-09-22 05:55:22 -04:00
/* config keyword parsers */
static struct cfg_kw_list cfg_kws = {ILH, {
{ CFG_GLOBAL, "nbthread", cfg_parse_nbthread, 0 },
MINOR: global: add a new "thread-group" directive This registers a mapping of threads to groups by enumerating for each thread what group it belongs to, and marking the group as assigned. It takes care of checking for redefinitions, overlaps, and holes. It supports both individual numbers and ranges. The thread group is referenced from the thread config.
2021-09-27 07:55:10 -04:00
{ CFG_GLOBAL, "thread-group", cfg_parse_thread_group, 0 },
MINOR: global: add a new "thread-groups" directive This is used to configure the number of thread groups. For now it can only be 1.
2021-09-22 06:07:23 -04:00
{ CFG_GLOBAL, "thread-groups", cfg_parse_thread_groups, 0 },
MINOR: config: use a standard parser for the "nbthread" keyword Probably because of some copy-paste from "nbproc", "nbthread" used to be parsed in cfgparse instead of using a registered parser. Let's fix this to clean up the code base now.
2021-09-22 05:55:22 -04:00
{ 0, NULL, NULL }
}};
INITCALL1(STG_REGISTER, cfg_register_keywords, &cfg_kws);
Reference in a new issue Copy permalink