upstream/haproxy
1
0
Fork 0
mirror of https://github.com/haproxy/haproxy.git synced 2026-02-26 19:30:30 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions 11
haproxy/src/lb_fwlc.c

376 lines
11 KiB
C
Raw Normal View History

[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
/*
* Fast Weighted Least Connection load balancing algorithm.
*
* Copyright 2000-2009 Willy Tarreau <w@1wt.eu>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version
* 2 of the License, or (at your option) any later version.
*
*/
REORG: include: move server.h to haproxy/server{,-t}.h extern struct dict server_name_dict was moved from the type file to the main file. A handful of inlined functions were moved at the bottom of the file. Call places were updated to use server-t.h when relevant, or to simply drop the entry when not needed.
2020-06-04 17:20:13 -04:00
#include <import/eb32tree.h>
REORG: include: update all files to use haproxy/api.h or api-t.h if needed All files that were including one of the following include files have been updated to only include haproxy/api.h or haproxy/api-t.h once instead: - common/config.h - common/compat.h - common/compiler.h - common/defaults.h - common/initcall.h - common/tools.h The choice is simple: if the file only requires type definitions, it includes api-t.h, otherwise it includes the full api.h. In addition, in these files, explicit includes for inttypes.h and limits.h were dropped since these are now covered by api.h and api-t.h. No other change was performed, given that this patch is large and affects 201 files. At least one (tools.h) was already freestanding and didn't get the new one added.
2020-05-27 06:58:42 -04:00
#include <haproxy/api.h>
CLEANUP: include: tree-wide alphabetical sort of include files This patch fixes all the leftovers from the include cleanup campaign. There were not that many (~400 entries in ~150 files) but it was definitely worth doing it as it revealed a few duplicates.
2020-06-09 03:07:15 -04:00
#include <haproxy/backend.h>
REORG: include: move queue.h to haproxy/queue{,-t}.h Nothing outstanding here. A number of call places were not justified and removed.
2020-06-04 16:59:39 -04:00
#include <haproxy/queue.h>
REORG: include: move server.h to haproxy/server{,-t}.h extern struct dict server_name_dict was moved from the type file to the main file. A handful of inlined functions were moved at the bottom of the file. Call places were updated to use server-t.h when relevant, or to simply drop the entry when not needed.
2020-06-04 17:20:13 -04:00
#include <haproxy/server-t.h>
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
/* Remove a server from a tree. It must have previously been dequeued. This
* function is meant to be called when a server is going down or has its
* weight disabled.
BUG/MEDIUM: lb/threads: always properly lock LB algorithms on maintenance operations Since commit 3ff577e ("MAJOR: server: make server state changes synchronous again"), srv_update_status() calls the various maintenance operations of the LB algorithms (->set_server_up, ->set_server_down, ->update_server_weight()). These ones are called with a single thread guaranteed by the rendez-vous point, so the fact that they're lacking some locks has no effect. However we'll need to remove the rendez-vous point so we have to take care of properly locking all the LB algos. The comments have been properly updated on the various functions to mention their locking expectations. All these functions are called with the server lock held, and all of them now support concurrent calls by using the lbprm's lock. This fix doesn't need to be backported at the moment, though if any check-specific issue surfaced in 1.8, it could make sense to reuse it.
2018-08-21 13:44:53 -04:00
*
* The server's lock and the lbprm's lock must be held.
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
*/
static inline void fwlc_remove_from_tree(struct server *s)
{
s->lb_tree = NULL;
}
BUG/MEDIUM: lb/threads: always properly lock LB algorithms on maintenance operations Since commit 3ff577e ("MAJOR: server: make server state changes synchronous again"), srv_update_status() calls the various maintenance operations of the LB algorithms (->set_server_up, ->set_server_down, ->update_server_weight()). These ones are called with a single thread guaranteed by the rendez-vous point, so the fact that they're lacking some locks has no effect. However we'll need to remove the rendez-vous point so we have to take care of properly locking all the LB algos. The comments have been properly updated on the various functions to mention their locking expectations. All these functions are called with the server lock held, and all of them now support concurrent calls by using the lbprm's lock. This fix doesn't need to be backported at the moment, though if any check-specific issue surfaced in 1.8, it could make sense to reuse it.
2018-08-21 13:44:53 -04:00
/* simply removes a server from a tree.
*
OPTIM: lb-leastconn: do not take the server lock on take_conn/drop_conn The operations are only an insert and a delete into the LB tree, which doesn't require the server's lock at all as the lbprm lock is already held. Let's drop it. Just for the sake of cleanness, given that the served and nbpend values used to be atomically updated, we'll use an atomic load to read them.
2021-02-17 10:14:00 -05:00
* The lbprm's lock must be held.
BUG/MEDIUM: lb/threads: always properly lock LB algorithms on maintenance operations Since commit 3ff577e ("MAJOR: server: make server state changes synchronous again"), srv_update_status() calls the various maintenance operations of the LB algorithms (->set_server_up, ->set_server_down, ->update_server_weight()). These ones are called with a single thread guaranteed by the rendez-vous point, so the fact that they're lacking some locks has no effect. However we'll need to remove the rendez-vous point so we have to take care of properly locking all the LB algos. The comments have been properly updated on the various functions to mention their locking expectations. All these functions are called with the server lock held, and all of them now support concurrent calls by using the lbprm's lock. This fix doesn't need to be backported at the moment, though if any check-specific issue surfaced in 1.8, it could make sense to reuse it.
2018-08-21 13:44:53 -04:00
*/
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
static inline void fwlc_dequeue_srv(struct server *s)
{
eb32_delete(&s->lb_node);
}
BUG/MEDIUM: lb-leastconn: Reposition a server using the right eweight Depending on the context, the current eweight or the next one must be used to reposition a server in the tree. When the server state is updated, for instance its weight, the next eweight must be used because it is not yet committed. However, when the server is used, on normal conditions, the current eweight must be used. In fact, it is only a bug on the 1.8. On newer versions, the changes on a server are performed synchronously. But it is safer to rely on the right eweight value to avoid any futur bugs. On the 1.8, it is important to do so, because the server state is updated and committed inside the rendez-vous point. Thus, the next server state may be unsync with the current state for a short time, waiting all threads join the rendez-vous point. It is especially a problem if the next eweight is set to 0. Because otherwise, it must not be used to reposition the server in the tree, leading to a divide by 0. This patch must be backported as far as 1.8.
2020-12-11 09:36:01 -05:00
/* Queue a server in its associated tree, assuming the <eweight> is >0.
MINOR: lb: make the leastconn algorithm more accurate The leastconn algorithm queues available servers based on their weighted current load. But this results in an inaccurate load balancing when weights differ and the load is very low, because what matters is not the load before picking the server but the load resulting from picking the server. At the very least, it must be granted that servers with the highest weight are always picked first when no server has any connection. This patch addresses this by simply adding one to the current connections count when queuing the server, since this is the load the server will have once picked. This finally allows to bridge the gap that existed between the "leastconn" and the "first" algorithms.
2018-12-14 02:33:28 -05:00
* Servers are sorted by (#conns+1)/weight. To ensure maximum accuracy,
* we use (#conns+1)*SRV_EWGHT_MAX/eweight as the sorting key. The reason
* for using #conns+1 is to sort by weights in case the server is picked
* and not before it is picked. This provides a better load accuracy for
* low connection counts when weights differ and makes sure the round-robin
BUG/MINOR: lb/leastconn: ignore the server weights for empty servers As discussed in issue #178, the change brought around 1.9-dev11 by commit 1eb6c55808 ("MINOR: lb: make the leastconn algorithm more accurate") causes some harm in the situation it tried to improve. By always applying the server's weight even for no connection, we end up always picking the same servers for the first connections, so under a low load, if servers only have either 0 or 1 connections, in practice the same servers will always be picked. This patch partially restores the original behaviour but still keeping the spirit of the aforementioned patch. Now what is done is that servers with no connections will always be picked first, regardless of their weight, so they will effectively follow round-robin. Only servers with one connection or more will see an accurate weight applied. This patch was developed and tested by @malsumis and @jaroslawr who reported the initial issue. It should be backported to 2.0 and 1.9.
2019-09-06 11:04:04 -04:00
* applies between servers of highest weight first. However servers with no
* connection are always picked first so that under low loads, it's not
* always the single server with the highest weight that gets picked.
BUG/MEDIUM: lb/threads: always properly lock LB algorithms on maintenance operations Since commit 3ff577e ("MAJOR: server: make server state changes synchronous again"), srv_update_status() calls the various maintenance operations of the LB algorithms (->set_server_up, ->set_server_down, ->update_server_weight()). These ones are called with a single thread guaranteed by the rendez-vous point, so the fact that they're lacking some locks has no effect. However we'll need to remove the rendez-vous point so we have to take care of properly locking all the LB algos. The comments have been properly updated on the various functions to mention their locking expectations. All these functions are called with the server lock held, and all of them now support concurrent calls by using the lbprm's lock. This fix doesn't need to be backported at the moment, though if any check-specific issue surfaced in 1.8, it could make sense to reuse it.
2018-08-21 13:44:53 -04:00
*
BUG/MEDIUM: lb-leastconn: Reposition a server using the right eweight Depending on the context, the current eweight or the next one must be used to reposition a server in the tree. When the server state is updated, for instance its weight, the next eweight must be used because it is not yet committed. However, when the server is used, on normal conditions, the current eweight must be used. In fact, it is only a bug on the 1.8. On newer versions, the changes on a server are performed synchronously. But it is safer to rely on the right eweight value to avoid any futur bugs. On the 1.8, it is important to do so, because the server state is updated and committed inside the rendez-vous point. Thus, the next server state may be unsync with the current state for a short time, waiting all threads join the rendez-vous point. It is especially a problem if the next eweight is set to 0. Because otherwise, it must not be used to reposition the server in the tree, leading to a divide by 0. This patch must be backported as far as 1.8.
2020-12-11 09:36:01 -05:00
* NOTE: Depending on the calling context, we use s->next_eweight or
* s->cur_eweight. The next value is used when the server state is updated
* (because the weight changed for instance). During this step, the server
* state is not yet committed. The current value is used to reposition the
* server in the tree. This happens when the server is used.
*
OPTIM: lb-leastconn: do not take the server lock on take_conn/drop_conn The operations are only an insert and a delete into the LB tree, which doesn't require the server's lock at all as the lbprm lock is already held. Let's drop it. Just for the sake of cleanness, given that the served and nbpend values used to be atomically updated, we'll use an atomic load to read them.
2021-02-17 10:14:00 -05:00
* The lbprm's lock must be held.
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
*/
BUG/MEDIUM: lb-leastconn: Reposition a server using the right eweight Depending on the context, the current eweight or the next one must be used to reposition a server in the tree. When the server state is updated, for instance its weight, the next eweight must be used because it is not yet committed. However, when the server is used, on normal conditions, the current eweight must be used. In fact, it is only a bug on the 1.8. On newer versions, the changes on a server are performed synchronously. But it is safer to rely on the right eweight value to avoid any futur bugs. On the 1.8, it is important to do so, because the server state is updated and committed inside the rendez-vous point. Thus, the next server state may be unsync with the current state for a short time, waiting all threads join the rendez-vous point. It is especially a problem if the next eweight is set to 0. Because otherwise, it must not be used to reposition the server in the tree, leading to a divide by 0. This patch must be backported as far as 1.8.
2020-12-11 09:36:01 -05:00
static inline void fwlc_queue_srv(struct server *s, unsigned int eweight)
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
{
OPTIM: lb-leastconn: do not take the server lock on take_conn/drop_conn The operations are only an insert and a delete into the LB tree, which doesn't require the server's lock at all as the lbprm lock is already held. Let's drop it. Just for the sake of cleanness, given that the served and nbpend values used to be atomically updated, we'll use an atomic load to read them.
2021-02-17 10:14:00 -05:00
unsigned int inflight = _HA_ATOMIC_LOAD(&s->served) + _HA_ATOMIC_LOAD(&s->nbpend);
MINOR: leastconn: take the queue length into account when queuing servers When servers are queued into the leastconn tree, it's important to also consider their queue length. There could be some servers with lots of queued requests that we don't want to hammer with extra connections. In order not to add extra stress to the LB algorithm, we don't update the value when adding to the queue, only when updating the connection count (i.e. picking from the queue or releasing a connection). This will be sufficient to significantly improve the fairness in such situations.
2020-10-22 11:41:45 -04:00
BUG/MEDIUM: lb-leastconn: Reposition a server using the right eweight Depending on the context, the current eweight or the next one must be used to reposition a server in the tree. When the server state is updated, for instance its weight, the next eweight must be used because it is not yet committed. However, when the server is used, on normal conditions, the current eweight must be used. In fact, it is only a bug on the 1.8. On newer versions, the changes on a server are performed synchronously. But it is safer to rely on the right eweight value to avoid any futur bugs. On the 1.8, it is important to do so, because the server state is updated and committed inside the rendez-vous point. Thus, the next server state may be unsync with the current state for a short time, waiting all threads join the rendez-vous point. It is especially a problem if the next eweight is set to 0. Because otherwise, it must not be used to reposition the server in the tree, leading to a divide by 0. This patch must be backported as far as 1.8.
2020-12-11 09:36:01 -05:00
s->lb_node.key = inflight ? (inflight + 1) * SRV_EWGHT_MAX / eweight : 0;
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
eb32_insert(s->lb_tree, &s->lb_node);
}
/* Re-position the server in the FWLC tree after it has been assigned one
* connection or after it has released one. Note that it is possible that
* the server has been moved out of the tree due to failed health-checks.
BUG/MEDIUM: lb/threads: always properly lock LB algorithms on maintenance operations Since commit 3ff577e ("MAJOR: server: make server state changes synchronous again"), srv_update_status() calls the various maintenance operations of the LB algorithms (->set_server_up, ->set_server_down, ->update_server_weight()). These ones are called with a single thread guaranteed by the rendez-vous point, so the fact that they're lacking some locks has no effect. However we'll need to remove the rendez-vous point so we have to take care of properly locking all the LB algos. The comments have been properly updated on the various functions to mention their locking expectations. All these functions are called with the server lock held, and all of them now support concurrent calls by using the lbprm's lock. This fix doesn't need to be backported at the moment, though if any check-specific issue surfaced in 1.8, it could make sense to reuse it.
2018-08-21 13:44:53 -04:00
*
MINOR: lb/api: let callers of take_conn/drop_conn tell if they have the lock The two algos defining these functions (first and leastconn) do not need the server's lock. However it's already present in pendconn_process_next_strm() so the API must be updated so that the functions may take it if needed and that the callers indicate whether they already own it. As such, the call places (backend.c and stream.c) now do not take it anymore, queue.c was unchanged since it's already held, and both "first" and "leastconn" were updated to take it if not already held. A quick test on the "first" algo showed a jump from 432 to 565k rps by just dropping the lock in stream.c!
2021-02-17 10:01:37 -05:00
* <locked> must reflect the server's lock ownership. The lbprm's lock will
* be used.
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
*/
MINOR: lb/api: let callers of take_conn/drop_conn tell if they have the lock The two algos defining these functions (first and leastconn) do not need the server's lock. However it's already present in pendconn_process_next_strm() so the API must be updated so that the functions may take it if needed and that the callers indicate whether they already own it. As such, the call places (backend.c and stream.c) now do not take it anymore, queue.c was unchanged since it's already held, and both "first" and "leastconn" were updated to take it if not already held. A quick test on the "first" algo showed a jump from 432 to 565k rps by just dropping the lock in stream.c!
2021-02-17 10:01:37 -05:00
static void fwlc_srv_reposition(struct server *s, int locked)
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
{
OPTIM: lb-leastconn: do not unlink the server if it did not change Due to the two-phase server reservation, there are 3 calls to fwlc_srv_reposition() per request, one during assign_server() to reserve the slot, one in connect_server() to commit it, and one in process_stream() to release it. However only one of the first two will change the key, so it's needlessly costly to take the lock, remove a server and insert it again at the same place when we can already figure we ought not to even have taken the lock. Furthermore, even when the server needs to move, there can be quite some contention on the lbprm lock forcing the thread to wait. During this time the served and nbpend server values might have changed, just like the lb_node.key itself. Thus we measure the values again under the lock before updating the tree. Measurements have shown that under contention with 16 servers and 16 threads, 50% of the updates can be avoided there. This patch makes the function compute the new key and compare it to the current one before deciding to move the entry (and does it again under the lock forthe second test). This removes between 40 and 50% of the tree updates depending on the thread contention and the number of servers. The performance gain due to this (on 16 threads) was: 16 servers: 415 krps -> 440 krps (6%, contention on lbprm) 4 servers: 554 krps -> 714 krps (+29%, little contention) One point worth thinking about is that it's not logic to update the tree 2-3 times per request while it's only read once. half to 2/3 of these updates are not needed. An experiment consisting in subscribing the server to a list and letting the readers reinsert them on the fly showed further degradation instead of an improvement. A better approach would probably consist in avoinding writes to shared cache lines by having the leastconn nodes distinct from the servers, with one node per value, and having them hold an mt-list with all the servers having that number of connections. The connection count tree would then be read-mostly instead of facing heavy writes, and most write operations would be performed on 1-3 list heads which are way cheaper to migrate than a tree node, and do not require updating the last two updated neighbors' cache lines.
2021-02-17 10:26:55 -05:00
unsigned int inflight = _HA_ATOMIC_LOAD(&s->served) + _HA_ATOMIC_LOAD(&s->nbpend);
unsigned int new_key = inflight ? (inflight + 1) * SRV_EWGHT_MAX / s->cur_eweight : 0;
/* some calls will be made for no change (e.g connect_server() after
* assign_server(). Let's check that first.
*/
if (s->lb_node.node.leaf_p && s->lb_node.key == new_key)
return;
MINOR: backend: replace the lbprm lock with an rwlock It was previously a spinlock, and it happens that a number of LB algos only lock it for lookups, without performing any modification. Let's first turn it to an rwlock and w-lock it everywhere. This is strictly identical. It was carefully checked that every HA_SPIN_LOCK() was turned to HA_RWLOCK_WRLOCK() and that HA_SPIN_UNLOCK() was turned to HA_RWLOCK_WRUNLOCK() on this lock. _INIT and _DESTROY were updated too.
2020-10-17 12:48:47 -04:00
HA_RWLOCK_WRLOCK(LBPRM_LOCK, &s->proxy->lbprm.lock);
BUG/MEDIUM: lb_fwlc: Don't test the server's lb_tree from outside the lock In the function fwlc_srv_reposition(), the server's lb_tree is tested from outside the lock. So it is possible to remove it after the test and then call eb32_insert() in fwlc_queue_srv() with a NULL root pointer, which is invalid. Moving the test in the scope of the lock fixes the bug. This issue was reported on Github, issue #126. This patch must be backported to 2.0, 1.9 and 1.8.
2019-06-19 04:50:38 -04:00
if (s->lb_tree) {
OPTIM: lb-leastconn: do not unlink the server if it did not change Due to the two-phase server reservation, there are 3 calls to fwlc_srv_reposition() per request, one during assign_server() to reserve the slot, one in connect_server() to commit it, and one in process_stream() to release it. However only one of the first two will change the key, so it's needlessly costly to take the lock, remove a server and insert it again at the same place when we can already figure we ought not to even have taken the lock. Furthermore, even when the server needs to move, there can be quite some contention on the lbprm lock forcing the thread to wait. During this time the served and nbpend server values might have changed, just like the lb_node.key itself. Thus we measure the values again under the lock before updating the tree. Measurements have shown that under contention with 16 servers and 16 threads, 50% of the updates can be avoided there. This patch makes the function compute the new key and compare it to the current one before deciding to move the entry (and does it again under the lock forthe second test). This removes between 40 and 50% of the tree updates depending on the thread contention and the number of servers. The performance gain due to this (on 16 threads) was: 16 servers: 415 krps -> 440 krps (6%, contention on lbprm) 4 servers: 554 krps -> 714 krps (+29%, little contention) One point worth thinking about is that it's not logic to update the tree 2-3 times per request while it's only read once. half to 2/3 of these updates are not needed. An experiment consisting in subscribing the server to a list and letting the readers reinsert them on the fly showed further degradation instead of an improvement. A better approach would probably consist in avoinding writes to shared cache lines by having the leastconn nodes distinct from the servers, with one node per value, and having them hold an mt-list with all the servers having that number of connections. The connection count tree would then be read-mostly instead of facing heavy writes, and most write operations would be performed on 1-3 list heads which are way cheaper to migrate than a tree node, and do not require updating the last two updated neighbors' cache lines.
2021-02-17 10:26:55 -05:00
/* we might have been waiting for a while on the lock above
* so it's worth testing again because other threads are very
* likely to have released a connection or taken one leading
* to our target value (50% of the case in measurements).
*/
inflight = _HA_ATOMIC_LOAD(&s->served) + _HA_ATOMIC_LOAD(&s->nbpend);
new_key = inflight ? (inflight + 1) * SRV_EWGHT_MAX / s->cur_eweight : 0;
if (!s->lb_node.node.leaf_p || s->lb_node.key != new_key) {
eb32_delete(&s->lb_node);
s->lb_node.key = new_key;
eb32_insert(s->lb_tree, &s->lb_node);
}
BUG/MEDIUM: lb_fwlc: Don't test the server's lb_tree from outside the lock In the function fwlc_srv_reposition(), the server's lb_tree is tested from outside the lock. So it is possible to remove it after the test and then call eb32_insert() in fwlc_queue_srv() with a NULL root pointer, which is invalid. Moving the test in the scope of the lock fixes the bug. This issue was reported on Github, issue #126. This patch must be backported to 2.0, 1.9 and 1.8.
2019-06-19 04:50:38 -04:00
}
MINOR: backend: replace the lbprm lock with an rwlock It was previously a spinlock, and it happens that a number of LB algos only lock it for lookups, without performing any modification. Let's first turn it to an rwlock and w-lock it everywhere. This is strictly identical. It was carefully checked that every HA_SPIN_LOCK() was turned to HA_RWLOCK_WRLOCK() and that HA_SPIN_UNLOCK() was turned to HA_RWLOCK_WRUNLOCK() on this lock. _INIT and _DESTROY were updated too.
2020-10-17 12:48:47 -04:00
HA_RWLOCK_WRUNLOCK(LBPRM_LOCK, &s->proxy->lbprm.lock);
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
}
/* This function updates the server trees according to server <srv>'s new
* state. It should be called when server <srv>'s status changes to down.
* It is not important whether the server was already down or not. It is not
* important either that the new state is completely down (the caller may not
* know all the variables of a server's state).
BUG/MEDIUM: lb/threads: always properly lock LB algorithms on maintenance operations Since commit 3ff577e ("MAJOR: server: make server state changes synchronous again"), srv_update_status() calls the various maintenance operations of the LB algorithms (->set_server_up, ->set_server_down, ->update_server_weight()). These ones are called with a single thread guaranteed by the rendez-vous point, so the fact that they're lacking some locks has no effect. However we'll need to remove the rendez-vous point so we have to take care of properly locking all the LB algos. The comments have been properly updated on the various functions to mention their locking expectations. All these functions are called with the server lock held, and all of them now support concurrent calls by using the lbprm's lock. This fix doesn't need to be backported at the moment, though if any check-specific issue surfaced in 1.8, it could make sense to reuse it.
2018-08-21 13:44:53 -04:00
*
* The server's lock must be held. The lbprm's lock will be used.
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
*/
static void fwlc_set_server_status_down(struct server *srv)
{
struct proxy *p = srv->proxy;
MINOR: server: use functions to detect state changes and to update them Detecting that a server's status has changed is a bit messy, as well as it is to commit the status changes. We'll have to add new conditions soon and we'd better avoid to multiply the number of touched locations with the high risk of forgetting them. This commit introduces : - srv_lb_status_changed() to report if the status changed from the previously committed one ; - svr_lb_commit_status() to commit the current status The function is now used by all load-balancing algorithms.
2014-05-13 13:27:31 -04:00
if (!srv_lb_status_changed(srv))
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
return;
MEDIUM: check: server states and weight propagation re-work The server state and weight was reworked to handle "pending" values updated by checks/CLI/LUA/agent. These values are commited to be propagated to the LB stack. In further dev related to multi-thread, the commit will be handled into a sync point. Pending values are named using the prefix 'next_' Current values used by the LB stack are named 'cur_'
2017-08-31 08:41:55 -04:00
if (srv_willbe_usable(srv))
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
goto out_update_state;
MINOR: backend: replace the lbprm lock with an rwlock It was previously a spinlock, and it happens that a number of LB algos only lock it for lookups, without performing any modification. Let's first turn it to an rwlock and w-lock it everywhere. This is strictly identical. It was carefully checked that every HA_SPIN_LOCK() was turned to HA_RWLOCK_WRLOCK() and that HA_SPIN_UNLOCK() was turned to HA_RWLOCK_WRUNLOCK() on this lock. _INIT and _DESTROY were updated too.
2020-10-17 12:48:47 -04:00
HA_RWLOCK_WRLOCK(LBPRM_LOCK, &p->lbprm.lock);
BUG/MEDIUM: lb/threads: always properly lock LB algorithms on maintenance operations Since commit 3ff577e ("MAJOR: server: make server state changes synchronous again"), srv_update_status() calls the various maintenance operations of the LB algorithms (->set_server_up, ->set_server_down, ->update_server_weight()). These ones are called with a single thread guaranteed by the rendez-vous point, so the fact that they're lacking some locks has no effect. However we'll need to remove the rendez-vous point so we have to take care of properly locking all the LB algos. The comments have been properly updated on the various functions to mention their locking expectations. All these functions are called with the server lock held, and all of them now support concurrent calls by using the lbprm's lock. This fix doesn't need to be backported at the moment, though if any check-specific issue surfaced in 1.8, it could make sense to reuse it.
2018-08-21 13:44:53 -04:00
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
MEDIUM: check: server states and weight propagation re-work The server state and weight was reworked to handle "pending" values updated by checks/CLI/LUA/agent. These values are commited to be propagated to the LB stack. In further dev related to multi-thread, the commit will be handled into a sync point. Pending values are named using the prefix 'next_' Current values used by the LB stack are named 'cur_'
2017-08-31 08:41:55 -04:00
if (!srv_currently_usable(srv))
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
/* server was already down */
goto out_update_backend;
REORG/MEDIUM: server: split server state and flags in two different variables Till now, the server's state and flags were all saved as a single bit field. It causes some difficulties because we'd like to have an enum for the state and separate flags. This commit starts by splitting them in two distinct fields. The first one is srv->state (with its counter-part srv->prev_state) which are now enums, but which still contain bits (SRV_STF_*). The flags now lie in their own field (srv->flags). The function srv_is_usable() was updated to use the enum as input, since it already used to deal only with the state. Note that currently, the maintenance mode is still in the state for simplicity, but it must move as well.
2014-05-13 09:54:22 -04:00
if (srv->flags & SRV_F_BACKUP) {
MEDIUM: check: server states and weight propagation re-work The server state and weight was reworked to handle "pending" values updated by checks/CLI/LUA/agent. These values are commited to be propagated to the LB stack. In further dev related to multi-thread, the commit will be handled into a sync point. Pending values are named using the prefix 'next_' Current values used by the LB stack are named 'cur_'
2017-08-31 08:41:55 -04:00
p->lbprm.tot_wbck -= srv->cur_eweight;
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
p->srv_bck--;
if (srv == p->lbprm.fbck) {
/* we lost the first backup server in a single-backup
* configuration, we must search another one.
*/
struct server *srv2 = p->lbprm.fbck;
do {
srv2 = srv2->next;
} while (srv2 &&
REORG/MEDIUM: server: split server state and flags in two different variables Till now, the server's state and flags were all saved as a single bit field. It causes some difficulties because we'd like to have an enum for the state and separate flags. This commit starts by splitting them in two distinct fields. The first one is srv->state (with its counter-part srv->prev_state) which are now enums, but which still contain bits (SRV_STF_*). The flags now lie in their own field (srv->flags). The function srv_is_usable() was updated to use the enum as input, since it already used to deal only with the state. Note that currently, the maintenance mode is still in the state for simplicity, but it must move as well.
2014-05-13 09:54:22 -04:00
!((srv2->flags & SRV_F_BACKUP) &&
MEDIUM: check: server states and weight propagation re-work The server state and weight was reworked to handle "pending" values updated by checks/CLI/LUA/agent. These values are commited to be propagated to the LB stack. In further dev related to multi-thread, the commit will be handled into a sync point. Pending values are named using the prefix 'next_' Current values used by the LB stack are named 'cur_'
2017-08-31 08:41:55 -04:00
srv_willbe_usable(srv2)));
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
p->lbprm.fbck = srv2;
}
} else {
MEDIUM: check: server states and weight propagation re-work The server state and weight was reworked to handle "pending" values updated by checks/CLI/LUA/agent. These values are commited to be propagated to the LB stack. In further dev related to multi-thread, the commit will be handled into a sync point. Pending values are named using the prefix 'next_' Current values used by the LB stack are named 'cur_'
2017-08-31 08:41:55 -04:00
p->lbprm.tot_wact -= srv->cur_eweight;
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
p->srv_act--;
}
fwlc_dequeue_srv(srv);
fwlc_remove_from_tree(srv);
out_update_backend:
/* check/update tot_used, tot_weight */
update_backend_weight(p);
MINOR: backend: replace the lbprm lock with an rwlock It was previously a spinlock, and it happens that a number of LB algos only lock it for lookups, without performing any modification. Let's first turn it to an rwlock and w-lock it everywhere. This is strictly identical. It was carefully checked that every HA_SPIN_LOCK() was turned to HA_RWLOCK_WRLOCK() and that HA_SPIN_UNLOCK() was turned to HA_RWLOCK_WRUNLOCK() on this lock. _INIT and _DESTROY were updated too.
2020-10-17 12:48:47 -04:00
HA_RWLOCK_WRUNLOCK(LBPRM_LOCK, &p->lbprm.lock);
BUG/MEDIUM: lb/threads: always properly lock LB algorithms on maintenance operations Since commit 3ff577e ("MAJOR: server: make server state changes synchronous again"), srv_update_status() calls the various maintenance operations of the LB algorithms (->set_server_up, ->set_server_down, ->update_server_weight()). These ones are called with a single thread guaranteed by the rendez-vous point, so the fact that they're lacking some locks has no effect. However we'll need to remove the rendez-vous point so we have to take care of properly locking all the LB algos. The comments have been properly updated on the various functions to mention their locking expectations. All these functions are called with the server lock held, and all of them now support concurrent calls by using the lbprm's lock. This fix doesn't need to be backported at the moment, though if any check-specific issue surfaced in 1.8, it could make sense to reuse it.
2018-08-21 13:44:53 -04:00
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
out_update_state:
MINOR: server: use functions to detect state changes and to update them Detecting that a server's status has changed is a bit messy, as well as it is to commit the status changes. We'll have to add new conditions soon and we'd better avoid to multiply the number of touched locations with the high risk of forgetting them. This commit introduces : - srv_lb_status_changed() to report if the status changed from the previously committed one ; - svr_lb_commit_status() to commit the current status The function is now used by all load-balancing algorithms.
2014-05-13 13:27:31 -04:00
srv_lb_commit_status(srv);
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
}
/* This function updates the server trees according to server <srv>'s new
* state. It should be called when server <srv>'s status changes to up.
* It is not important whether the server was already down or not. It is not
* important either that the new state is completely UP (the caller may not
* know all the variables of a server's state). This function will not change
* the weight of a server which was already up.
BUG/MEDIUM: lb/threads: always properly lock LB algorithms on maintenance operations Since commit 3ff577e ("MAJOR: server: make server state changes synchronous again"), srv_update_status() calls the various maintenance operations of the LB algorithms (->set_server_up, ->set_server_down, ->update_server_weight()). These ones are called with a single thread guaranteed by the rendez-vous point, so the fact that they're lacking some locks has no effect. However we'll need to remove the rendez-vous point so we have to take care of properly locking all the LB algos. The comments have been properly updated on the various functions to mention their locking expectations. All these functions are called with the server lock held, and all of them now support concurrent calls by using the lbprm's lock. This fix doesn't need to be backported at the moment, though if any check-specific issue surfaced in 1.8, it could make sense to reuse it.
2018-08-21 13:44:53 -04:00
*
* The server's lock must be held. The lbprm's lock will be used.
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
*/
static void fwlc_set_server_status_up(struct server *srv)
{
struct proxy *p = srv->proxy;
MINOR: server: use functions to detect state changes and to update them Detecting that a server's status has changed is a bit messy, as well as it is to commit the status changes. We'll have to add new conditions soon and we'd better avoid to multiply the number of touched locations with the high risk of forgetting them. This commit introduces : - srv_lb_status_changed() to report if the status changed from the previously committed one ; - svr_lb_commit_status() to commit the current status The function is now used by all load-balancing algorithms.
2014-05-13 13:27:31 -04:00
if (!srv_lb_status_changed(srv))
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
return;
MEDIUM: check: server states and weight propagation re-work The server state and weight was reworked to handle "pending" values updated by checks/CLI/LUA/agent. These values are commited to be propagated to the LB stack. In further dev related to multi-thread, the commit will be handled into a sync point. Pending values are named using the prefix 'next_' Current values used by the LB stack are named 'cur_'
2017-08-31 08:41:55 -04:00
if (!srv_willbe_usable(srv))
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
goto out_update_state;
MINOR: backend: replace the lbprm lock with an rwlock It was previously a spinlock, and it happens that a number of LB algos only lock it for lookups, without performing any modification. Let's first turn it to an rwlock and w-lock it everywhere. This is strictly identical. It was carefully checked that every HA_SPIN_LOCK() was turned to HA_RWLOCK_WRLOCK() and that HA_SPIN_UNLOCK() was turned to HA_RWLOCK_WRUNLOCK() on this lock. _INIT and _DESTROY were updated too.
2020-10-17 12:48:47 -04:00
HA_RWLOCK_WRLOCK(LBPRM_LOCK, &p->lbprm.lock);
BUG/MEDIUM: lb/threads: always properly lock LB algorithms on maintenance operations Since commit 3ff577e ("MAJOR: server: make server state changes synchronous again"), srv_update_status() calls the various maintenance operations of the LB algorithms (->set_server_up, ->set_server_down, ->update_server_weight()). These ones are called with a single thread guaranteed by the rendez-vous point, so the fact that they're lacking some locks has no effect. However we'll need to remove the rendez-vous point so we have to take care of properly locking all the LB algos. The comments have been properly updated on the various functions to mention their locking expectations. All these functions are called with the server lock held, and all of them now support concurrent calls by using the lbprm's lock. This fix doesn't need to be backported at the moment, though if any check-specific issue surfaced in 1.8, it could make sense to reuse it.
2018-08-21 13:44:53 -04:00
MEDIUM: check: server states and weight propagation re-work The server state and weight was reworked to handle "pending" values updated by checks/CLI/LUA/agent. These values are commited to be propagated to the LB stack. In further dev related to multi-thread, the commit will be handled into a sync point. Pending values are named using the prefix 'next_' Current values used by the LB stack are named 'cur_'
2017-08-31 08:41:55 -04:00
if (srv_currently_usable(srv))
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
/* server was already up */
goto out_update_backend;
REORG/MEDIUM: server: split server state and flags in two different variables Till now, the server's state and flags were all saved as a single bit field. It causes some difficulties because we'd like to have an enum for the state and separate flags. This commit starts by splitting them in two distinct fields. The first one is srv->state (with its counter-part srv->prev_state) which are now enums, but which still contain bits (SRV_STF_*). The flags now lie in their own field (srv->flags). The function srv_is_usable() was updated to use the enum as input, since it already used to deal only with the state. Note that currently, the maintenance mode is still in the state for simplicity, but it must move as well.
2014-05-13 09:54:22 -04:00
if (srv->flags & SRV_F_BACKUP) {
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
srv->lb_tree = &p->lbprm.fwlc.bck;
MEDIUM: check: server states and weight propagation re-work The server state and weight was reworked to handle "pending" values updated by checks/CLI/LUA/agent. These values are commited to be propagated to the LB stack. In further dev related to multi-thread, the commit will be handled into a sync point. Pending values are named using the prefix 'next_' Current values used by the LB stack are named 'cur_'
2017-08-31 08:41:55 -04:00
p->lbprm.tot_wbck += srv->next_eweight;
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
p->srv_bck++;
if (!(p->options & PR_O_USE_ALL_BK)) {
if (!p->lbprm.fbck) {
/* there was no backup server anymore */
p->lbprm.fbck = srv;
} else {
/* we may have restored a backup server prior to fbck,
* in which case it should replace it.
*/
struct server *srv2 = srv;
do {
srv2 = srv2->next;
} while (srv2 && (srv2 != p->lbprm.fbck));
if (srv2)
p->lbprm.fbck = srv;
}
}
} else {
srv->lb_tree = &p->lbprm.fwlc.act;
MEDIUM: check: server states and weight propagation re-work The server state and weight was reworked to handle "pending" values updated by checks/CLI/LUA/agent. These values are commited to be propagated to the LB stack. In further dev related to multi-thread, the commit will be handled into a sync point. Pending values are named using the prefix 'next_' Current values used by the LB stack are named 'cur_'
2017-08-31 08:41:55 -04:00
p->lbprm.tot_wact += srv->next_eweight;
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
p->srv_act++;
}
/* note that eweight cannot be 0 here */
BUG/MEDIUM: lb-leastconn: Reposition a server using the right eweight Depending on the context, the current eweight or the next one must be used to reposition a server in the tree. When the server state is updated, for instance its weight, the next eweight must be used because it is not yet committed. However, when the server is used, on normal conditions, the current eweight must be used. In fact, it is only a bug on the 1.8. On newer versions, the changes on a server are performed synchronously. But it is safer to rely on the right eweight value to avoid any futur bugs. On the 1.8, it is important to do so, because the server state is updated and committed inside the rendez-vous point. Thus, the next server state may be unsync with the current state for a short time, waiting all threads join the rendez-vous point. It is especially a problem if the next eweight is set to 0. Because otherwise, it must not be used to reposition the server in the tree, leading to a divide by 0. This patch must be backported as far as 1.8.
2020-12-11 09:36:01 -05:00
fwlc_queue_srv(srv, srv->next_eweight);
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
out_update_backend:
/* check/update tot_used, tot_weight */
update_backend_weight(p);
MINOR: backend: replace the lbprm lock with an rwlock It was previously a spinlock, and it happens that a number of LB algos only lock it for lookups, without performing any modification. Let's first turn it to an rwlock and w-lock it everywhere. This is strictly identical. It was carefully checked that every HA_SPIN_LOCK() was turned to HA_RWLOCK_WRLOCK() and that HA_SPIN_UNLOCK() was turned to HA_RWLOCK_WRUNLOCK() on this lock. _INIT and _DESTROY were updated too.
2020-10-17 12:48:47 -04:00
HA_RWLOCK_WRUNLOCK(LBPRM_LOCK, &p->lbprm.lock);
BUG/MEDIUM: lb/threads: always properly lock LB algorithms on maintenance operations Since commit 3ff577e ("MAJOR: server: make server state changes synchronous again"), srv_update_status() calls the various maintenance operations of the LB algorithms (->set_server_up, ->set_server_down, ->update_server_weight()). These ones are called with a single thread guaranteed by the rendez-vous point, so the fact that they're lacking some locks has no effect. However we'll need to remove the rendez-vous point so we have to take care of properly locking all the LB algos. The comments have been properly updated on the various functions to mention their locking expectations. All these functions are called with the server lock held, and all of them now support concurrent calls by using the lbprm's lock. This fix doesn't need to be backported at the moment, though if any check-specific issue surfaced in 1.8, it could make sense to reuse it.
2018-08-21 13:44:53 -04:00
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
out_update_state:
MINOR: server: use functions to detect state changes and to update them Detecting that a server's status has changed is a bit messy, as well as it is to commit the status changes. We'll have to add new conditions soon and we'd better avoid to multiply the number of touched locations with the high risk of forgetting them. This commit introduces : - srv_lb_status_changed() to report if the status changed from the previously committed one ; - svr_lb_commit_status() to commit the current status The function is now used by all load-balancing algorithms.
2014-05-13 13:27:31 -04:00
srv_lb_commit_status(srv);
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
}
/* This function must be called after an update to server <srv>'s effective
* weight. It may be called after a state change too.
BUG/MEDIUM: lb/threads: always properly lock LB algorithms on maintenance operations Since commit 3ff577e ("MAJOR: server: make server state changes synchronous again"), srv_update_status() calls the various maintenance operations of the LB algorithms (->set_server_up, ->set_server_down, ->update_server_weight()). These ones are called with a single thread guaranteed by the rendez-vous point, so the fact that they're lacking some locks has no effect. However we'll need to remove the rendez-vous point so we have to take care of properly locking all the LB algos. The comments have been properly updated on the various functions to mention their locking expectations. All these functions are called with the server lock held, and all of them now support concurrent calls by using the lbprm's lock. This fix doesn't need to be backported at the moment, though if any check-specific issue surfaced in 1.8, it could make sense to reuse it.
2018-08-21 13:44:53 -04:00
*
* The server's lock must be held. The lbprm's lock will be used.
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
*/
static void fwlc_update_server_weight(struct server *srv)
{
int old_state, new_state;
struct proxy *p = srv->proxy;
MINOR: server: use functions to detect state changes and to update them Detecting that a server's status has changed is a bit messy, as well as it is to commit the status changes. We'll have to add new conditions soon and we'd better avoid to multiply the number of touched locations with the high risk of forgetting them. This commit introduces : - srv_lb_status_changed() to report if the status changed from the previously committed one ; - svr_lb_commit_status() to commit the current status The function is now used by all load-balancing algorithms.
2014-05-13 13:27:31 -04:00
if (!srv_lb_status_changed(srv))
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
return;
/* If changing the server's weight changes its state, we simply apply
* the procedures we already have for status change. If the state
* remains down, the server is not in any tree, so it's as easy as
* updating its values. If the state remains up with different weights,
* there are some computations to perform to find a new place and
* possibly a new tree for this server.
*/
MEDIUM: check: server states and weight propagation re-work The server state and weight was reworked to handle "pending" values updated by checks/CLI/LUA/agent. These values are commited to be propagated to the LB stack. In further dev related to multi-thread, the commit will be handled into a sync point. Pending values are named using the prefix 'next_' Current values used by the LB stack are named 'cur_'
2017-08-31 08:41:55 -04:00
old_state = srv_currently_usable(srv);
new_state = srv_willbe_usable(srv);
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
if (!old_state && !new_state) {
MINOR: server: use functions to detect state changes and to update them Detecting that a server's status has changed is a bit messy, as well as it is to commit the status changes. We'll have to add new conditions soon and we'd better avoid to multiply the number of touched locations with the high risk of forgetting them. This commit introduces : - srv_lb_status_changed() to report if the status changed from the previously committed one ; - svr_lb_commit_status() to commit the current status The function is now used by all load-balancing algorithms.
2014-05-13 13:27:31 -04:00
srv_lb_commit_status(srv);
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
return;
}
else if (!old_state && new_state) {
fwlc_set_server_status_up(srv);
return;
}
else if (old_state && !new_state) {
fwlc_set_server_status_down(srv);
return;
}
MINOR: backend: replace the lbprm lock with an rwlock It was previously a spinlock, and it happens that a number of LB algos only lock it for lookups, without performing any modification. Let's first turn it to an rwlock and w-lock it everywhere. This is strictly identical. It was carefully checked that every HA_SPIN_LOCK() was turned to HA_RWLOCK_WRLOCK() and that HA_SPIN_UNLOCK() was turned to HA_RWLOCK_WRUNLOCK() on this lock. _INIT and _DESTROY were updated too.
2020-10-17 12:48:47 -04:00
HA_RWLOCK_WRLOCK(LBPRM_LOCK, &p->lbprm.lock);
BUG/MEDIUM: lb/threads: always properly lock LB algorithms on maintenance operations Since commit 3ff577e ("MAJOR: server: make server state changes synchronous again"), srv_update_status() calls the various maintenance operations of the LB algorithms (->set_server_up, ->set_server_down, ->update_server_weight()). These ones are called with a single thread guaranteed by the rendez-vous point, so the fact that they're lacking some locks has no effect. However we'll need to remove the rendez-vous point so we have to take care of properly locking all the LB algos. The comments have been properly updated on the various functions to mention their locking expectations. All these functions are called with the server lock held, and all of them now support concurrent calls by using the lbprm's lock. This fix doesn't need to be backported at the moment, though if any check-specific issue surfaced in 1.8, it could make sense to reuse it.
2018-08-21 13:44:53 -04:00
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
if (srv->lb_tree)
fwlc_dequeue_srv(srv);
REORG/MEDIUM: server: split server state and flags in two different variables Till now, the server's state and flags were all saved as a single bit field. It causes some difficulties because we'd like to have an enum for the state and separate flags. This commit starts by splitting them in two distinct fields. The first one is srv->state (with its counter-part srv->prev_state) which are now enums, but which still contain bits (SRV_STF_*). The flags now lie in their own field (srv->flags). The function srv_is_usable() was updated to use the enum as input, since it already used to deal only with the state. Note that currently, the maintenance mode is still in the state for simplicity, but it must move as well.
2014-05-13 09:54:22 -04:00
if (srv->flags & SRV_F_BACKUP) {
MEDIUM: check: server states and weight propagation re-work The server state and weight was reworked to handle "pending" values updated by checks/CLI/LUA/agent. These values are commited to be propagated to the LB stack. In further dev related to multi-thread, the commit will be handled into a sync point. Pending values are named using the prefix 'next_' Current values used by the LB stack are named 'cur_'
2017-08-31 08:41:55 -04:00
p->lbprm.tot_wbck += srv->next_eweight - srv->cur_eweight;
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
srv->lb_tree = &p->lbprm.fwlc.bck;
} else {
MEDIUM: check: server states and weight propagation re-work The server state and weight was reworked to handle "pending" values updated by checks/CLI/LUA/agent. These values are commited to be propagated to the LB stack. In further dev related to multi-thread, the commit will be handled into a sync point. Pending values are named using the prefix 'next_' Current values used by the LB stack are named 'cur_'
2017-08-31 08:41:55 -04:00
p->lbprm.tot_wact += srv->next_eweight - srv->cur_eweight;
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
srv->lb_tree = &p->lbprm.fwlc.act;
}
BUG/MEDIUM: lb-leastconn: Reposition a server using the right eweight Depending on the context, the current eweight or the next one must be used to reposition a server in the tree. When the server state is updated, for instance its weight, the next eweight must be used because it is not yet committed. However, when the server is used, on normal conditions, the current eweight must be used. In fact, it is only a bug on the 1.8. On newer versions, the changes on a server are performed synchronously. But it is safer to rely on the right eweight value to avoid any futur bugs. On the 1.8, it is important to do so, because the server state is updated and committed inside the rendez-vous point. Thus, the next server state may be unsync with the current state for a short time, waiting all threads join the rendez-vous point. It is especially a problem if the next eweight is set to 0. Because otherwise, it must not be used to reposition the server in the tree, leading to a divide by 0. This patch must be backported as far as 1.8.
2020-12-11 09:36:01 -05:00
fwlc_queue_srv(srv, srv->next_eweight);
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
update_backend_weight(p);
MINOR: backend: replace the lbprm lock with an rwlock It was previously a spinlock, and it happens that a number of LB algos only lock it for lookups, without performing any modification. Let's first turn it to an rwlock and w-lock it everywhere. This is strictly identical. It was carefully checked that every HA_SPIN_LOCK() was turned to HA_RWLOCK_WRLOCK() and that HA_SPIN_UNLOCK() was turned to HA_RWLOCK_WRUNLOCK() on this lock. _INIT and _DESTROY were updated too.
2020-10-17 12:48:47 -04:00
HA_RWLOCK_WRUNLOCK(LBPRM_LOCK, &p->lbprm.lock);
BUG/MEDIUM: lb/threads: always properly lock LB algorithms on maintenance operations Since commit 3ff577e ("MAJOR: server: make server state changes synchronous again"), srv_update_status() calls the various maintenance operations of the LB algorithms (->set_server_up, ->set_server_down, ->update_server_weight()). These ones are called with a single thread guaranteed by the rendez-vous point, so the fact that they're lacking some locks has no effect. However we'll need to remove the rendez-vous point so we have to take care of properly locking all the LB algos. The comments have been properly updated on the various functions to mention their locking expectations. All these functions are called with the server lock held, and all of them now support concurrent calls by using the lbprm's lock. This fix doesn't need to be backported at the moment, though if any check-specific issue surfaced in 1.8, it could make sense to reuse it.
2018-08-21 13:44:53 -04:00
MINOR: server: use functions to detect state changes and to update them Detecting that a server's status has changed is a bit messy, as well as it is to commit the status changes. We'll have to add new conditions soon and we'd better avoid to multiply the number of touched locations with the high risk of forgetting them. This commit introduces : - srv_lb_status_changed() to report if the status changed from the previously committed one ; - svr_lb_commit_status() to commit the current status The function is now used by all load-balancing algorithms.
2014-05-13 13:27:31 -04:00
srv_lb_commit_status(srv);
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
}
/* This function is responsible for building the trees in case of fast
* weighted least-conns. It also sets p->lbprm.wdiv to the eweight to
* uweight ratio. Both active and backup groups are initialized.
*/
void fwlc_init_server_tree(struct proxy *p)
{
struct server *srv;
struct eb_root init_head = EB_ROOT;
p->lbprm.set_server_status_up = fwlc_set_server_status_up;
p->lbprm.set_server_status_down = fwlc_set_server_status_down;
p->lbprm.update_server_eweight = fwlc_update_server_weight;
p->lbprm.server_take_conn = fwlc_srv_reposition;
p->lbprm.server_drop_conn = fwlc_srv_reposition;
p->lbprm.wdiv = BE_WEIGHT_SCALE;
for (srv = p->srv; srv; srv = srv->next) {
MEDIUM: check: server states and weight propagation re-work The server state and weight was reworked to handle "pending" values updated by checks/CLI/LUA/agent. These values are commited to be propagated to the LB stack. In further dev related to multi-thread, the commit will be handled into a sync point. Pending values are named using the prefix 'next_' Current values used by the LB stack are named 'cur_'
2017-08-31 08:41:55 -04:00
srv->next_eweight = (srv->uweight * p->lbprm.wdiv + p->lbprm.wmult - 1) / p->lbprm.wmult;
MINOR: server: use functions to detect state changes and to update them Detecting that a server's status has changed is a bit messy, as well as it is to commit the status changes. We'll have to add new conditions soon and we'd better avoid to multiply the number of touched locations with the high risk of forgetting them. This commit introduces : - srv_lb_status_changed() to report if the status changed from the previously committed one ; - svr_lb_commit_status() to commit the current status The function is now used by all load-balancing algorithms.
2014-05-13 13:27:31 -04:00
srv_lb_commit_status(srv);
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
}
recount_servers(p);
update_backend_weight(p);
p->lbprm.fwlc.act = init_head;
p->lbprm.fwlc.bck = init_head;
/* queue active and backup servers in two distinct groups */
for (srv = p->srv; srv; srv = srv->next) {
MEDIUM: check: server states and weight propagation re-work The server state and weight was reworked to handle "pending" values updated by checks/CLI/LUA/agent. These values are commited to be propagated to the LB stack. In further dev related to multi-thread, the commit will be handled into a sync point. Pending values are named using the prefix 'next_' Current values used by the LB stack are named 'cur_'
2017-08-31 08:41:55 -04:00
if (!srv_currently_usable(srv))
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
continue;
REORG/MEDIUM: server: split server state and flags in two different variables Till now, the server's state and flags were all saved as a single bit field. It causes some difficulties because we'd like to have an enum for the state and separate flags. This commit starts by splitting them in two distinct fields. The first one is srv->state (with its counter-part srv->prev_state) which are now enums, but which still contain bits (SRV_STF_*). The flags now lie in their own field (srv->flags). The function srv_is_usable() was updated to use the enum as input, since it already used to deal only with the state. Note that currently, the maintenance mode is still in the state for simplicity, but it must move as well.
2014-05-13 09:54:22 -04:00
srv->lb_tree = (srv->flags & SRV_F_BACKUP) ? &p->lbprm.fwlc.bck : &p->lbprm.fwlc.act;
BUG/MEDIUM: lb-leastconn: Reposition a server using the right eweight Depending on the context, the current eweight or the next one must be used to reposition a server in the tree. When the server state is updated, for instance its weight, the next eweight must be used because it is not yet committed. However, when the server is used, on normal conditions, the current eweight must be used. In fact, it is only a bug on the 1.8. On newer versions, the changes on a server are performed synchronously. But it is safer to rely on the right eweight value to avoid any futur bugs. On the 1.8, it is important to do so, because the server state is updated and committed inside the rendez-vous point. Thus, the next server state may be unsync with the current state for a short time, waiting all threads join the rendez-vous point. It is especially a problem if the next eweight is set to 0. Because otherwise, it must not be used to reposition the server in the tree, leading to a divide by 0. This patch must be backported as far as 1.8.
2020-12-11 09:36:01 -05:00
fwlc_queue_srv(srv, srv->next_eweight);
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
}
}
/* Return next server from the FWLC tree in backend <p>. If the tree is empty,
* return NULL. Saturated servers are skipped.
BUG/MEDIUM: lb/threads: always properly lock LB algorithms on maintenance operations Since commit 3ff577e ("MAJOR: server: make server state changes synchronous again"), srv_update_status() calls the various maintenance operations of the LB algorithms (->set_server_up, ->set_server_down, ->update_server_weight()). These ones are called with a single thread guaranteed by the rendez-vous point, so the fact that they're lacking some locks has no effect. However we'll need to remove the rendez-vous point so we have to take care of properly locking all the LB algos. The comments have been properly updated on the various functions to mention their locking expectations. All these functions are called with the server lock held, and all of them now support concurrent calls by using the lbprm's lock. This fix doesn't need to be backported at the moment, though if any check-specific issue surfaced in 1.8, it could make sense to reuse it.
2018-08-21 13:44:53 -04:00
*
CLEANUP: backend: fix incorrect comments on locking conditions for lb functions The leastconn and roundrobin functions mention that the server's lock must be held while this is not true at all and it is not used either. The "first" algo doesn't mention anything about the need for locking, so let's mention that it uses the lbprm lock.
2021-06-01 10:58:31 -04:00
* The lbprm's lock will be used in R/O mode. The server's lock is not used.
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
*/
struct server *fwlc_get_next_server(struct proxy *p, struct server *srvtoavoid)
{
struct server *srv, *avoided;
struct eb32_node *node;
srv = avoided = NULL;
MINOR: lb/leastconn: only take a read lock in fwlc_get_next_server() This function doesn't change the tree, it only looks for the first usable server, so let's do that under a read lock to limit the situations like the ones described in issue #881 where finding a usable server when dealing with lots of saturated ones can be expensive. At least threads will now be able to look up in parallel. It's interesting to note that s->served is not incremented during the server choice, nor is the server repositionned. So right now already, nothing prevents multiple threads from picking the same server. This will not cause a significant imbalance anyway given that the server will automatically be repositionned at the right place, but this might be something to improve in the future if it doesn't come with too high a cost. It also looks like the way a server's weight is updated could be revisited so that the write lock gets tighter at the expense of a short part of inconsistency between weights and servers still present in the tree.
2020-10-17 13:32:09 -04:00
HA_RWLOCK_RDLOCK(LBPRM_LOCK, &p->lbprm.lock);
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
if (p->srv_act)
node = eb32_first(&p->lbprm.fwlc.act);
MEDIUM: threads/lb: Make LB algorithms (lb_*.c) thread-safe A lock for LB parameters has been added inside the proxy structure and atomic operations have been used to update server variables releated to lb. The only significant change is about lb_map. Because the servers status are updated in the sync-point, we can call recalc_server_map function synchronously in map_set_server_status_up/down function.
2017-06-09 08:17:53 -04:00
else if (p->lbprm.fbck) {
srv = p->lbprm.fbck;
goto out;
}
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
else if (p->srv_bck)
node = eb32_first(&p->lbprm.fwlc.bck);
MEDIUM: threads/lb: Make LB algorithms (lb_*.c) thread-safe A lock for LB parameters has been added inside the proxy structure and atomic operations have been used to update server variables releated to lb. The only significant change is about lb_map. Because the servers status are updated in the sync-point, we can call recalc_server_map function synchronously in map_set_server_status_up/down function.
2017-06-09 08:17:53 -04:00
else {
srv = NULL;
goto out;
}
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
while (node) {
/* OK, we have a server. However, it may be saturated, in which
* case we don't want to reconsider it for now, so we'll simply
* skip it. Same if it's the server we try to avoid, in which
* case we simply remember it for later use if needed.
*/
struct server *s;
s = eb32_entry(node, struct server, lb_node);
MEDIUM: fwlc: re-enable per-server queuing up to maxqueue Leastconn has the nice propery of being able to sort servers by their current usage. It's really a shame to force all requests into the backend queue when the algo would be able to also consider their current queue. In order not to change existing behavior but extend it, this patch allows leastconn to elect servers which are already full if they have an explicitly configured maxqueue setting above zero and their queue hasn't reached that threshold. This will significantly reduce the pressure in the backend queue when queuing a lot with lots of servers. A test on 8 threads with 100 servers configured with maxconn 1 jumped from 165krps to 330krps with maxqueue 15 with this patch. This partially undoes commit 82cd5c13a ("OPTIM: backend: skip LB when we know the backend is full") but allows to scale much better even by setting a single-digit maxqueue value. Some better heuristics could be used to maintain the behavior of the bypass in the patch above, consisting in keeping it if it's known that there is no server with a configured maxqueue in the farm (or in the backend).
2020-10-22 11:19:07 -04:00
if (!s->maxconn || s->served + s->nbpend < srv_dynamic_maxconn(s) + s->maxqueue) {
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
if (s != srvtoavoid) {
srv = s;
break;
}
avoided = s;
}
node = eb32_next(node);
}
if (!srv)
srv = avoided;
MEDIUM: threads/lb: Make LB algorithms (lb_*.c) thread-safe A lock for LB parameters has been added inside the proxy structure and atomic operations have been used to update server variables releated to lb. The only significant change is about lb_map. Because the servers status are updated in the sync-point, we can call recalc_server_map function synchronously in map_set_server_status_up/down function.
2017-06-09 08:17:53 -04:00
out:
MINOR: lb/leastconn: only take a read lock in fwlc_get_next_server() This function doesn't change the tree, it only looks for the first usable server, so let's do that under a read lock to limit the situations like the ones described in issue #881 where finding a usable server when dealing with lots of saturated ones can be expensive. At least threads will now be able to look up in parallel. It's interesting to note that s->served is not incremented during the server choice, nor is the server repositionned. So right now already, nothing prevents multiple threads from picking the same server. This will not cause a significant imbalance anyway given that the server will automatically be repositionned at the right place, but this might be something to improve in the future if it doesn't come with too high a cost. It also looks like the way a server's weight is updated could be revisited so that the write lock gets tighter at the expense of a short part of inconsistency between weights and servers still present in the tree.
2020-10-17 13:32:09 -04:00
HA_RWLOCK_RDUNLOCK(LBPRM_LOCK, &p->lbprm.lock);
[CLEANUP] backend: move LB algos to individual files It was becoming painful to have all the LB algos in backend.c. Let's move them to their own files. A few hashing functions still need be broken in two parts, one for the contents and one for the map position.
2009-10-01 05:19:37 -04:00
return srv;
}
/*
* Local variables:
* c-indent-level: 8
* c-basic-offset: 8
* End:
*/
Reference in a new issue Copy permalink