mirror of
https://github.com/redis/redis.git
synced 2026-04-27 09:06:55 -04:00
64a40b20d9
## Introduction
Redis introduced IO Thread in 6.0, allowing IO threads to handle client
request reading, command parsing and reply writing, thereby improving
performance. The current IO thread implementation has a few drawbacks.
- The main thread is blocked during IO thread read/write operations and
must wait for all IO threads to complete their current tasks before it
can continue execution. In other words, the entire process is
synchronous. This prevents the efficient utilization of multi-core CPUs
for parallel processing.
- When the number of clients and requests increases moderately, it
causes all IO threads to reach full CPU utilization due to the busy wait
mechanism used by the IO threads. This makes it challenging for us to
determine which part of Redis has reached its bottleneck.
- When IO threads are enabled with TLS and io-threads-do-reads, a
disconnection of a connection with pending data may result in it being
assigned to multiple IO threads simultaneously. This can cause race
conditions and trigger assertion failures. Related issue:
redis#12540
Therefore, we designed an asynchronous IO threads solution. The IO
threads adopt an event-driven model, with the main thread dedicated to
command processing, meanwhile, the IO threads handle client read and
write operations in parallel.
## Implementation
### Overall
As before, we did not change the fact that all client commands must be
executed on the main thread, because Redis was originally designed to be
single-threaded, and processing commands in a multi-threaded manner
would inevitably introduce numerous race and synchronization issues. But
now each IO thread has independent event loop, therefore, IO threads can
use a multiplexing approach to handle client read and write operations,
eliminating the CPU overhead caused by busy-waiting.
the execution process can be briefly described as follows:
the main thread assigns clients to IO threads after accepting
connections, IO threads will notify the main thread when clients
finish reading and parsing queries, then the main thread processes
queries from IO threads and generates replies, IO threads handle
writing reply to clients after receiving clients list from main thread,
and then continue to handle client read and write events.
### Each IO thread has independent event loop
We now assign each IO thread its own event loop. This approach
eliminates the need for the main thread to perform the costly
`epoll_wait` operation for handling connections (except for specific
ones). Instead, the main thread processes requests from the IO threads
and hands them back once completed, fully offloading read and write
events to the IO threads.
Additionally, all TLS operations, including handling pending data, have
been moved entirely to the IO threads. This resolves the issue where
io-threads-do-reads could not be used with TLS.
### Event-notified client queue
To facilitate communication between the IO threads and the main thread,
we designed an event-notified client queue. Each IO thread and the main
thread have two such queues to store clients waiting to be processed.
These queues are also integrated with the event loop to enable handling.
We use pthread_mutex to ensure the safety of queue operations, as well
as data visibility and ordering, and race conditions are minimized, as
each IO thread and the main thread operate on independent queues,
avoiding thread suspension due to lock contention. And we implemented an
event notifier based on `eventfd` or `pipe` to support event-driven
handling.
### Thread safety
Since the main thread and IO threads can execute in parallel, we must
handle data race issues carefully.
**client->flags**
The primary tasks of IO threads are reading and writing, i.e.
`readQueryFromClient` and `writeToClient`. However, IO threads and the
main thread may concurrently modify or access `client->flags`, leading
to potential race conditions. To address this, we introduced an io-flags
variable to record operations performed by IO threads, thereby avoiding
race conditions on `client->flags`.
**Pause IO thread**
In the main thread, we may want to operate data of IO threads, maybe
uninstall event handler, access or operate query/output buffer or resize
event loop, we need a clean and safe context to do that. We pause IO
thread in `IOThreadBeforeSleep`, do some jobs and then resume it. To
avoid thread suspended, we use busy waiting to confirm the target
status. Besides we use atomic variable to make sure memory visibility
and ordering. We introduce these functions to pause/resume IO Threads as
below.
```
pauseIOThread, resumeIOThread
pauseAllIOThreads, resumeAllIOThreads
pauseIOThreadsRange, resumeIOThreadsRange
```
Testing has shown that `pauseIOThread` is highly efficient, allowing the
main thread to execute nearly 200,000 operations per second during
stress tests. Similarly, `pauseAllIOThreads` with 8 IO threads can
handle up to nearly 56,000 operations per second. But operations
performed between pausing and resuming IO threads must be quick;
otherwise, they could cause the IO threads to reach full CPU
utilization.
**freeClient and freeClientAsync**
The main thread may need to terminate a client currently running on an
IO thread, for example, due to ACL rule changes, reaching the output
buffer limit, or evicting a client. In such cases, we need to pause the
IO thread to safely operate on the client.
**maxclients and maxmemory-clients updating**
When adjusting `maxclients`, we need to resize the event loop for all IO
threads. Similarly, when modifying `maxmemory-clients`, we need to
traverse all clients to calculate their memory usage. To ensure safe
operations, we pause all IO threads during these adjustments.
**Client info reading**
The main thread may need to read a client’s fields to generate a
descriptive string, such as for the `CLIENT LIST` command or logging
purposes. In such cases, we need to pause the IO thread handling that
client. If information for all clients needs to be displayed, all IO
threads must be paused.
**Tracking redirect**
Redis supports the tracking feature and can even send invalidation
messages to a connection with a specified ID. But the target client may
be running on IO thread, directly manipulating the client’s output
buffer is not thread-safe, and the IO thread may not be aware that the
client requires a response. In such cases, we pause the IO thread
handling the client, modify the output buffer, and install a write event
handler to ensure proper handling.
**clientsCron**
In the `clientsCron` function, the main thread needs to traverse all
clients to perform operations such as timeout checks, verifying whether
they have reached the soft output buffer limit, resizing the
output/query buffer, or updating memory usage. To safely operate on a
client, the IO thread handling that client must be paused.
If we were to pause the IO thread for each client individually, the
efficiency would be very low. Conversely, pausing all IO threads
simultaneously would be costly, especially when there are many IO
threads, as clientsCron is invoked relatively frequently.
To address this, we adopted a batched approach for pausing IO threads.
At most, 8 IO threads are paused at a time. The operations mentioned
above are only performed on clients running in the paused IO threads,
significantly reducing overhead while maintaining safety.
### Observability
In the current design, the main thread always assigns clients to the IO
thread with the least clients. To clearly observe the number of clients
handled by each IO thread, we added the new section in INFO output. The
`INFO THREADS` section can show the client count for each IO thread.
```
# Threads
io_thread_0:clients=0
io_thread_1:clients=2
io_thread_2:clients=2
```
Additionally, in the `CLIENT LIST` output, we also added a field to
indicate the thread to which each client is assigned.
`id=244 addr=127.0.0.1:41870 laddr=127.0.0.1:6379 ... resp=2 lib-name=
lib-ver= io-thread=1`
## Trade-off
### Special Clients
For certain special types of clients, keeping them running on IO threads
would result in severe race issues that are difficult to resolve.
Therefore, we chose not to offload these clients to the IO threads.
For replica, monitor, subscribe, and tracking clients, main thread may
directly write them a reply when conditions are met. Race issues are
difficult to resolve, so we have them processed in the main thread. This
includes the Lua debug clients as well, since we may operate connection
directly.
For blocking client, after the IO thread reads and parses a command and
hands it over to the main thread, if the client is identified as a
blocking type, it will be remained in the main thread. Once the blocking
operation completes and the reply is generated, the client is
transferred back to the IO thread to send the reply and wait for event
triggers.
### Clients Eviction
To support client eviction, it is necessary to update each client’s
memory usage promptly during operations such as read, write, or command
execution. However, when a client operates on an IO thread, it is not
feasible to update the memory usage immediately due to the risk of data
races. As a result, memory usage can only be updated either in the main
thread while processing commands or in the `ClientsCron` periodically.
The downside of this approach is that updates might experience a delay
of up to one second, which could impact the precision of memory
management for eviction.
To avoid incorrectly evicting clients. We adopted a best-effort
compensation solution, when we decide to eviction a client, we update
its memory usage again before evicting, if the memory used by the client
does not decrease or memory usage bucket is not changed, then we will
evict it, otherwise, not evict it.
However, we have not completely solved this problem. Due to the delay in
memory usage updates, it may lead us to make incorrect decisions about
the need to evict clients.
### Defragment
In the majority of cases we do NOT use the data from argv directly in
the db.
1. key names
We store a copy that we allocate in the main thread, see `sdsdup()` in
`dbAdd()`.
2. hash key and value
We store key as hfield and store value as sds, see `hfieldNew()` and
`sdsdup()` in `hashTypeSet()`.
3. other datatypes
They don't even use SDS, so there is no reference issues.
But in some cases client the data from argv may be retain by the main
thread.
As a result, during fragmentation cleanup, we need to move allocations
from the IO thread’s arena to the main thread’s arena. We always
allocate new memory in the main thread’s arena, but the memory released
by IO threads may not yet have been reclaimed. This ultimately causes
the fragmentation rate to be higher compared to creating and allocating
entirely within a single thread.
The following cases below will lead to memory allocated by the IO thread
being kept by the main thread.
1. string related command: `append`, `getset`, `mset` and `set`.
If `tryObjectEncoding()` does not change argv, we will keep it directly
in the main thread, see the code in `tryObjectEncoding()`(specifically
`trimStringObjectIfNeeded()`)
2. block related command.
the key names will be kept in `c->db->blocking_keys`.
3. watch command
the key names will be kept in `c->db->watched_keys`.
4. [s]subscribe command
channel name will be kept in `serverPubSubChannels`.
5. script load command
script will be kept in `server.lua_scripts`.
7. some module API: `RM_RetainString`, `RM_HoldString`
Those issues will be handled in other PRs.
## Testing
### Functional Testing
The commit with enabling IO Threads has passed all TCL tests, but we did
some changes:
**Client query buffer**: In the original code, when using a reusable
query buffer, ownership of the query buffer would be released after the
command was processed. However, with IO threads enabled, the client
transitions from an IO thread to the main thread for processing. This
causes the ownership release to occur earlier than the command
execution. As a result, when IO threads are enabled, the client's
information will never indicate that a shared query buffer is in use.
Therefore, we skip the corresponding query buffer tests in this case.
**Defragment**: Add a new defragmentation test to verify the effect of
io threads on defragmentation.
**Command delay**: For deferred clients in TCL tests, due to clients
being assigned to different threads for execution, delays may occur. To
address this, we introduced conditional waiting: the process proceeds to
the next step only when the `client list` contains the corresponding
commands.
### Sanitizer Testing
The commit passed all TCL tests and reported no errors when compiled
with the `fsanitizer=thread` and `fsanitizer=address` options enabled.
But we made the following modifications: we suppressed the sanitizer
warnings for clients with watched keys when updating `client->flags`, we
think IO threads read `client->flags`, but never modify it or read the
`CLIENT_DIRTY_CAS` bit, main thread just only modifies this bit, so
there is no actual data race.
## Others
### IO thread number
In the new multi-threaded design, the main thread is primarily focused
on command processing to improve performance. Typically, the main thread
does not handle regular client I/O operations but is responsible for
clients such as replication and tracking clients. To avoid breaking
changes, we still consider the main thread as the first IO thread.
When the io-threads configuration is set to a low value (e.g., 2),
performance does not show a significant improvement compared to a
single-threaded setup for simple commands (such as SET or GET), as the
main thread does not consume much CPU for these simple operations. This
results in underutilized multi-core capacity. However, for more complex
commands, having a low number of IO threads may still be beneficial.
Therefore, it’s important to adjust the `io-threads` based on your own
performance tests.
Additionally, you can clearly monitor the CPU utilization of the main
thread and IO threads using `top -H -p $redis_pid`. This allows you to
easily identify where the bottleneck is. If the IO thread is the
bottleneck, increasing the `io-threads` will improve performance. If the
main thread is the bottleneck, the overall performance can only be
scaled by increasing the number of shards or replicas.
---------
Co-authored-by: debing.sun <debing.sun@redis.com>
Co-authored-by: oranagra <oran@redislabs.com>
920 lines
39 KiB
Tcl
920 lines
39 KiB
Tcl
proc test_memory_efficiency {range} {
|
||
r flushall
|
||
set rd [redis_deferring_client]
|
||
set base_mem [s used_memory]
|
||
set written 0
|
||
for {set j 0} {$j < 10000} {incr j} {
|
||
set key key:$j
|
||
set val [string repeat A [expr {int(rand()*$range)}]]
|
||
$rd set $key $val
|
||
incr written [string length $key]
|
||
incr written [string length $val]
|
||
incr written 2 ;# A separator is the minimum to store key-value data.
|
||
}
|
||
for {set j 0} {$j < 10000} {incr j} {
|
||
$rd read ; # Discard replies
|
||
}
|
||
|
||
set current_mem [s used_memory]
|
||
set used [expr {$current_mem-$base_mem}]
|
||
set efficiency [expr {double($written)/$used}]
|
||
return $efficiency
|
||
}
|
||
|
||
start_server {tags {"memefficiency external:skip"}} {
|
||
foreach {size_range expected_min_efficiency} {
|
||
32 0.15
|
||
64 0.25
|
||
128 0.35
|
||
1024 0.75
|
||
16384 0.82
|
||
} {
|
||
test "Memory efficiency with values in range $size_range" {
|
||
set efficiency [test_memory_efficiency $size_range]
|
||
assert {$efficiency >= $expected_min_efficiency}
|
||
}
|
||
}
|
||
}
|
||
|
||
run_solo {defrag} {
|
||
proc wait_for_defrag_stop {maxtries delay} {
|
||
wait_for_condition $maxtries $delay {
|
||
[s active_defrag_running] eq 0
|
||
} else {
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
puts [r info memory]
|
||
puts [r info stats]
|
||
puts [r memory malloc-stats]
|
||
fail "defrag didn't stop."
|
||
}
|
||
}
|
||
|
||
proc test_active_defrag {type} {
|
||
if {[string match {*jemalloc*} [s mem_allocator]] && [r debug mallctl arenas.page] <= 8192} {
|
||
test "Active defrag main dictionary: $type" {
|
||
r config set hz 100
|
||
r config set activedefrag no
|
||
r config set active-defrag-threshold-lower 5
|
||
r config set active-defrag-cycle-min 65
|
||
r config set active-defrag-cycle-max 75
|
||
r config set active-defrag-ignore-bytes 2mb
|
||
r config set maxmemory 100mb
|
||
r config set maxmemory-policy allkeys-lru
|
||
|
||
populate 700000 asdf1 150
|
||
populate 100 asdf1 150 0 false 1000
|
||
populate 170000 asdf2 300
|
||
populate 100 asdf2 300 0 false 1000
|
||
|
||
assert {[scan [regexp -inline {expires\=([\d]*)} [r info keyspace]] expires=%d] > 0}
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
set frag [s allocator_frag_ratio]
|
||
if {$::verbose} {
|
||
puts "frag $frag"
|
||
}
|
||
assert {$frag >= 1.4}
|
||
|
||
r config set latency-monitor-threshold 5
|
||
r latency reset
|
||
r config set maxmemory 110mb ;# prevent further eviction (not to fail the digest test)
|
||
set digest [debug_digest]
|
||
catch {r config set activedefrag yes} e
|
||
if {[r config get activedefrag] eq "activedefrag yes"} {
|
||
# Wait for the active defrag to start working (decision once a
|
||
# second).
|
||
wait_for_condition 50 100 {
|
||
[s total_active_defrag_time] ne 0
|
||
} else {
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
puts [r info memory]
|
||
puts [r info stats]
|
||
puts [r memory malloc-stats]
|
||
fail "defrag not started."
|
||
}
|
||
|
||
# This test usually runs for a while, during this interval, we test the range.
|
||
assert_range [s active_defrag_running] 65 75
|
||
r config set active-defrag-cycle-min 1
|
||
r config set active-defrag-cycle-max 1
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
assert_range [s active_defrag_running] 1 1
|
||
r config set active-defrag-cycle-min 65
|
||
r config set active-defrag-cycle-max 75
|
||
|
||
# Wait for the active defrag to stop working.
|
||
wait_for_defrag_stop 2000 100
|
||
|
||
# Test the fragmentation is lower.
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
set frag [s allocator_frag_ratio]
|
||
set max_latency 0
|
||
foreach event [r latency latest] {
|
||
lassign $event eventname time latency max
|
||
if {$eventname == "active-defrag-cycle"} {
|
||
set max_latency $max
|
||
}
|
||
}
|
||
if {$::verbose} {
|
||
puts "frag $frag"
|
||
set misses [s active_defrag_misses]
|
||
set hits [s active_defrag_hits]
|
||
puts "hits: $hits"
|
||
puts "misses: $misses"
|
||
puts "max latency $max_latency"
|
||
puts [r latency latest]
|
||
puts [r latency history active-defrag-cycle]
|
||
}
|
||
assert {$frag < 1.1}
|
||
# due to high fragmentation, 100hz, and active-defrag-cycle-max set to 75,
|
||
# we expect max latency to be not much higher than 7.5ms but due to rare slowness threshold is set higher
|
||
if {!$::no_latency} {
|
||
assert {$max_latency <= 30}
|
||
}
|
||
}
|
||
# verify the data isn't corrupted or changed
|
||
set newdigest [debug_digest]
|
||
assert {$digest eq $newdigest}
|
||
r save ;# saving an rdb iterates over all the data / pointers
|
||
|
||
# if defrag is supported, test AOF loading too
|
||
if {[r config get activedefrag] eq "activedefrag yes" && $type eq "standalone"} {
|
||
test "Active defrag - AOF loading" {
|
||
# reset stats and load the AOF file
|
||
r config resetstat
|
||
r config set key-load-delay -25 ;# sleep on average 1/25 usec
|
||
r debug loadaof
|
||
r config set activedefrag no
|
||
# measure hits and misses right after aof loading
|
||
set misses [s active_defrag_misses]
|
||
set hits [s active_defrag_hits]
|
||
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
set frag [s allocator_frag_ratio]
|
||
set max_latency 0
|
||
foreach event [r latency latest] {
|
||
lassign $event eventname time latency max
|
||
if {$eventname == "while-blocked-cron"} {
|
||
set max_latency $max
|
||
}
|
||
}
|
||
if {$::verbose} {
|
||
puts "AOF loading:"
|
||
puts "frag $frag"
|
||
puts "hits: $hits"
|
||
puts "misses: $misses"
|
||
puts "max latency $max_latency"
|
||
puts [r latency latest]
|
||
puts [r latency history "while-blocked-cron"]
|
||
}
|
||
# make sure we had defrag hits during AOF loading
|
||
assert {$hits > 100000}
|
||
# make sure the defragger did enough work to keep the fragmentation low during loading.
|
||
# we cannot check that it went all the way down, since we don't wait for full defrag cycle to complete.
|
||
assert {$frag < 1.4}
|
||
# since the AOF contains simple (fast) SET commands (and the cron during loading runs every 1024 commands),
|
||
# it'll still not block the loading for long periods of time.
|
||
if {!$::no_latency} {
|
||
assert {$max_latency <= 40}
|
||
}
|
||
}
|
||
} ;# Active defrag - AOF loading
|
||
}
|
||
r config set appendonly no
|
||
r config set key-load-delay 0
|
||
|
||
test "Active defrag eval scripts: $type" {
|
||
r flushdb
|
||
r script flush sync
|
||
r config set hz 100
|
||
r config set activedefrag no
|
||
wait_for_defrag_stop 500 100
|
||
r config resetstat
|
||
r config set active-defrag-threshold-lower 5
|
||
r config set active-defrag-cycle-min 65
|
||
r config set active-defrag-cycle-max 75
|
||
r config set active-defrag-ignore-bytes 1500kb
|
||
r config set maxmemory 0
|
||
|
||
set n 50000
|
||
|
||
# Populate memory with interleaving script-key pattern of same size
|
||
set dummy_script "--[string repeat x 400]\nreturn "
|
||
set rd [redis_deferring_client]
|
||
for {set j 0} {$j < $n} {incr j} {
|
||
set val "$dummy_script[format "%06d" $j]"
|
||
$rd script load $val
|
||
$rd set k$j $val
|
||
}
|
||
for {set j 0} {$j < $n} {incr j} {
|
||
$rd read ; # Discard script load replies
|
||
$rd read ; # Discard set replies
|
||
}
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
if {$::verbose} {
|
||
puts "used [s allocator_allocated]"
|
||
puts "rss [s allocator_active]"
|
||
puts "frag [s allocator_frag_ratio]"
|
||
puts "frag_bytes [s allocator_frag_bytes]"
|
||
}
|
||
assert_lessthan [s allocator_frag_ratio] 1.05
|
||
|
||
# Delete all the keys to create fragmentation
|
||
for {set j 0} {$j < $n} {incr j} { $rd del k$j }
|
||
for {set j 0} {$j < $n} {incr j} { $rd read } ; # Discard del replies
|
||
$rd close
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
if {$::verbose} {
|
||
puts "used [s allocator_allocated]"
|
||
puts "rss [s allocator_active]"
|
||
puts "frag [s allocator_frag_ratio]"
|
||
puts "frag_bytes [s allocator_frag_bytes]"
|
||
}
|
||
assert_morethan [s allocator_frag_ratio] 1.4
|
||
|
||
catch {r config set activedefrag yes} e
|
||
if {[r config get activedefrag] eq "activedefrag yes"} {
|
||
|
||
# wait for the active defrag to start working (decision once a second)
|
||
wait_for_condition 50 100 {
|
||
[s total_active_defrag_time] ne 0
|
||
} else {
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
puts [r info memory]
|
||
puts [r info stats]
|
||
puts [r memory malloc-stats]
|
||
fail "defrag not started."
|
||
}
|
||
|
||
# wait for the active defrag to stop working
|
||
wait_for_defrag_stop 500 100
|
||
|
||
# test the fragmentation is lower
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
if {$::verbose} {
|
||
puts "used [s allocator_allocated]"
|
||
puts "rss [s allocator_active]"
|
||
puts "frag [s allocator_frag_ratio]"
|
||
puts "frag_bytes [s allocator_frag_bytes]"
|
||
}
|
||
assert_lessthan_equal [s allocator_frag_ratio] 1.05
|
||
}
|
||
# Flush all script to make sure we don't crash after defragging them
|
||
r script flush sync
|
||
} {OK}
|
||
|
||
test "Active defrag big keys: $type" {
|
||
r flushdb
|
||
r config set hz 100
|
||
r config set activedefrag no
|
||
wait_for_defrag_stop 500 100
|
||
r config resetstat
|
||
r config set active-defrag-max-scan-fields 1000
|
||
r config set active-defrag-threshold-lower 5
|
||
r config set active-defrag-cycle-min 65
|
||
r config set active-defrag-cycle-max 75
|
||
r config set active-defrag-ignore-bytes 2mb
|
||
r config set maxmemory 0
|
||
r config set list-max-ziplist-size 5 ;# list of 10k items will have 2000 quicklist nodes
|
||
r config set stream-node-max-entries 5
|
||
r hmset hash h1 v1 h2 v2 h3 v3
|
||
r lpush list a b c d
|
||
r zadd zset 0 a 1 b 2 c 3 d
|
||
r sadd set a b c d
|
||
r xadd stream * item 1 value a
|
||
r xadd stream * item 2 value b
|
||
r xgroup create stream mygroup 0
|
||
r xreadgroup GROUP mygroup Alice COUNT 1 STREAMS stream >
|
||
|
||
# create big keys with 10k items
|
||
set rd [redis_deferring_client]
|
||
for {set j 0} {$j < 10000} {incr j} {
|
||
$rd hset bighash $j [concat "asdfasdfasdf" $j]
|
||
$rd lpush biglist [concat "asdfasdfasdf" $j]
|
||
$rd zadd bigzset $j [concat "asdfasdfasdf" $j]
|
||
$rd sadd bigset [concat "asdfasdfasdf" $j]
|
||
$rd xadd bigstream * item 1 value a
|
||
}
|
||
for {set j 0} {$j < 50000} {incr j} {
|
||
$rd read ; # Discard replies
|
||
}
|
||
|
||
# create some small items (effective in cluster-enabled)
|
||
r set "{bighash}smallitem" val
|
||
r set "{biglist}smallitem" val
|
||
r set "{bigzset}smallitem" val
|
||
r set "{bigset}smallitem" val
|
||
r set "{bigstream}smallitem" val
|
||
|
||
|
||
set expected_frag 1.5
|
||
if {$::accurate} {
|
||
# scale the hash to 1m fields in order to have a measurable the latency
|
||
for {set j 10000} {$j < 1000000} {incr j} {
|
||
$rd hset bighash $j [concat "asdfasdfasdf" $j]
|
||
}
|
||
for {set j 10000} {$j < 1000000} {incr j} {
|
||
$rd read ; # Discard replies
|
||
}
|
||
# creating that big hash, increased used_memory, so the relative frag goes down
|
||
set expected_frag 1.3
|
||
}
|
||
|
||
# add a mass of string keys
|
||
for {set j 0} {$j < 500000} {incr j} {
|
||
$rd setrange $j 150 a
|
||
}
|
||
for {set j 0} {$j < 500000} {incr j} {
|
||
$rd read ; # Discard replies
|
||
}
|
||
assert_equal [r dbsize] 500015
|
||
|
||
# create some fragmentation
|
||
for {set j 0} {$j < 500000} {incr j 2} {
|
||
$rd del $j
|
||
}
|
||
for {set j 0} {$j < 500000} {incr j 2} {
|
||
$rd read ; # Discard replies
|
||
}
|
||
assert_equal [r dbsize] 250015
|
||
|
||
# start defrag
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
set frag [s allocator_frag_ratio]
|
||
if {$::verbose} {
|
||
puts "frag $frag"
|
||
}
|
||
assert {$frag >= $expected_frag}
|
||
r config set latency-monitor-threshold 5
|
||
r latency reset
|
||
|
||
set digest [debug_digest]
|
||
catch {r config set activedefrag yes} e
|
||
if {[r config get activedefrag] eq "activedefrag yes"} {
|
||
# wait for the active defrag to start working (decision once a second)
|
||
wait_for_condition 50 100 {
|
||
[s total_active_defrag_time] ne 0
|
||
} else {
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
puts [r info memory]
|
||
puts [r info stats]
|
||
puts [r memory malloc-stats]
|
||
fail "defrag not started."
|
||
}
|
||
|
||
# wait for the active defrag to stop working
|
||
wait_for_defrag_stop 500 100
|
||
|
||
# test the fragmentation is lower
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
set frag [s allocator_frag_ratio]
|
||
set max_latency 0
|
||
foreach event [r latency latest] {
|
||
lassign $event eventname time latency max
|
||
if {$eventname == "active-defrag-cycle"} {
|
||
set max_latency $max
|
||
}
|
||
}
|
||
if {$::verbose} {
|
||
puts "frag $frag"
|
||
set misses [s active_defrag_misses]
|
||
set hits [s active_defrag_hits]
|
||
puts "hits: $hits"
|
||
puts "misses: $misses"
|
||
puts "max latency $max_latency"
|
||
puts [r latency latest]
|
||
puts [r latency history active-defrag-cycle]
|
||
}
|
||
assert {$frag < 1.1}
|
||
# due to high fragmentation, 100hz, and active-defrag-cycle-max set to 75,
|
||
# we expect max latency to be not much higher than 7.5ms but due to rare slowness threshold is set higher
|
||
if {!$::no_latency} {
|
||
assert {$max_latency <= 30}
|
||
}
|
||
}
|
||
# verify the data isn't corrupted or changed
|
||
set newdigest [debug_digest]
|
||
assert {$digest eq $newdigest}
|
||
r save ;# saving an rdb iterates over all the data / pointers
|
||
} {OK}
|
||
|
||
test "Active defrag pubsub: $type" {
|
||
r flushdb
|
||
r config set hz 100
|
||
r config set activedefrag no
|
||
wait_for_defrag_stop 500 100
|
||
r config resetstat
|
||
r config set active-defrag-threshold-lower 5
|
||
r config set active-defrag-cycle-min 65
|
||
r config set active-defrag-cycle-max 75
|
||
r config set active-defrag-ignore-bytes 1500kb
|
||
r config set maxmemory 0
|
||
|
||
# Populate memory with interleaving pubsub-key pattern of same size
|
||
set n 50000
|
||
set dummy_channel "[string repeat x 400]"
|
||
set rd [redis_deferring_client]
|
||
set rd_pubsub [redis_deferring_client]
|
||
for {set j 0} {$j < $n} {incr j} {
|
||
set channel_name "$dummy_channel[format "%06d" $j]"
|
||
$rd_pubsub subscribe $channel_name
|
||
$rd_pubsub read ; # Discard subscribe replies
|
||
$rd_pubsub ssubscribe $channel_name
|
||
$rd_pubsub read ; # Discard ssubscribe replies
|
||
# Pub/Sub clients are handled in the main thread, so their memory is
|
||
# allocated there. Using the SETBIT command avoids the main thread
|
||
# referencing argv from IO threads.
|
||
$rd setbit k$j [expr {[string length $channel_name] * 8}] 1
|
||
$rd read ; # Discard set replies
|
||
}
|
||
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
if {$::verbose} {
|
||
puts "used [s allocator_allocated]"
|
||
puts "rss [s allocator_active]"
|
||
puts "frag [s allocator_frag_ratio]"
|
||
puts "frag_bytes [s allocator_frag_bytes]"
|
||
}
|
||
assert_lessthan [s allocator_frag_ratio] 1.05
|
||
|
||
# Delete all the keys to create fragmentation
|
||
for {set j 0} {$j < $n} {incr j} { $rd del k$j }
|
||
for {set j 0} {$j < $n} {incr j} { $rd read } ; # Discard del replies
|
||
$rd close
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
if {$::verbose} {
|
||
puts "used [s allocator_allocated]"
|
||
puts "rss [s allocator_active]"
|
||
puts "frag [s allocator_frag_ratio]"
|
||
puts "frag_bytes [s allocator_frag_bytes]"
|
||
}
|
||
assert_morethan [s allocator_frag_ratio] 1.35
|
||
|
||
catch {r config set activedefrag yes} e
|
||
if {[r config get activedefrag] eq "activedefrag yes"} {
|
||
|
||
# wait for the active defrag to start working (decision once a second)
|
||
wait_for_condition 50 100 {
|
||
[s total_active_defrag_time] ne 0
|
||
} else {
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
puts [r info memory]
|
||
puts [r info stats]
|
||
puts [r memory malloc-stats]
|
||
fail "defrag not started."
|
||
}
|
||
|
||
# wait for the active defrag to stop working
|
||
wait_for_defrag_stop 500 100
|
||
|
||
# test the fragmentation is lower
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
if {$::verbose} {
|
||
puts "used [s allocator_allocated]"
|
||
puts "rss [s allocator_active]"
|
||
puts "frag [s allocator_frag_ratio]"
|
||
puts "frag_bytes [s allocator_frag_bytes]"
|
||
}
|
||
assert_lessthan_equal [s allocator_frag_ratio] 1.05
|
||
}
|
||
|
||
# Publishes some message to all the pubsub clients to make sure that
|
||
# we didn't break the data structure.
|
||
for {set j 0} {$j < $n} {incr j} {
|
||
set channel "$dummy_channel[format "%06d" $j]"
|
||
r publish $channel "hello"
|
||
assert_equal "message $channel hello" [$rd_pubsub read]
|
||
$rd_pubsub unsubscribe $channel
|
||
$rd_pubsub read
|
||
r spublish $channel "hello"
|
||
assert_equal "smessage $channel hello" [$rd_pubsub read]
|
||
$rd_pubsub sunsubscribe $channel
|
||
$rd_pubsub read
|
||
}
|
||
$rd_pubsub close
|
||
}
|
||
|
||
test "Active Defrag HFE: $type" {
|
||
r flushdb
|
||
r config set hz 100
|
||
r config set activedefrag no
|
||
wait_for_defrag_stop 500 100
|
||
r config resetstat
|
||
# TODO: Lower the threshold after defraging the ebuckets.
|
||
# Now just to ensure that the reference is updated correctly.
|
||
r config set active-defrag-threshold-lower 12
|
||
r config set active-defrag-cycle-min 65
|
||
r config set active-defrag-cycle-max 75
|
||
r config set active-defrag-ignore-bytes 1500kb
|
||
r config set maxmemory 0
|
||
r config set hash-max-listpack-value 512
|
||
r config set hash-max-listpack-entries 10
|
||
|
||
# Populate memory with interleaving hash field of same size
|
||
set n 3000
|
||
set fields 16 ;# make all the fields in an eblist.
|
||
set dummy_field "[string repeat x 400]"
|
||
set rd [redis_deferring_client]
|
||
for {set i 0} {$i < $n} {incr i} {
|
||
for {set j 0} {$j < $fields} {incr j} {
|
||
$rd hset h$i f$j $dummy_field
|
||
$rd hexpire h$i 9999999 FIELDS 1 f$j
|
||
$rd set "k$i$j" $dummy_field
|
||
}
|
||
}
|
||
for {set j 0} {$j < [expr $n*$fields]} {incr j} {
|
||
$rd read ; # Discard hset replies
|
||
$rd read ; # Discard hexpire replies
|
||
$rd read ; # Discard set replies
|
||
}
|
||
|
||
# Coverage for listpackex.
|
||
r hset h_lpex f0 $dummy_field
|
||
r hexpire h_lpex 9999999 FIELDS 1 f0
|
||
assert_encoding listpackex h_lpex
|
||
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
if {$::verbose} {
|
||
puts "used [s allocator_allocated]"
|
||
puts "rss [s allocator_active]"
|
||
puts "frag [s allocator_frag_ratio]"
|
||
puts "frag_bytes [s allocator_frag_bytes]"
|
||
}
|
||
assert_lessthan [s allocator_frag_ratio] 1.05
|
||
|
||
# Delete all the keys to create fragmentation
|
||
for {set i 0} {$i < $n} {incr i} {
|
||
for {set j 0} {$j < $fields} {incr j} {
|
||
r del "k$i$j"
|
||
}
|
||
}
|
||
$rd close
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
if {$::verbose} {
|
||
puts "used [s allocator_allocated]"
|
||
puts "rss [s allocator_active]"
|
||
puts "frag [s allocator_frag_ratio]"
|
||
puts "frag_bytes [s allocator_frag_bytes]"
|
||
}
|
||
assert_morethan [s allocator_frag_ratio] 1.35
|
||
|
||
catch {r config set activedefrag yes} e
|
||
if {[r config get activedefrag] eq "activedefrag yes"} {
|
||
|
||
# wait for the active defrag to start working (decision once a second)
|
||
wait_for_condition 50 100 {
|
||
[s total_active_defrag_time] ne 0
|
||
} else {
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
puts [r info memory]
|
||
puts [r info stats]
|
||
puts [r memory malloc-stats]
|
||
fail "defrag not started."
|
||
}
|
||
|
||
# wait for the active defrag to stop working
|
||
wait_for_defrag_stop 500 100
|
||
|
||
# test the fragmentation is lower
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
if {$::verbose} {
|
||
puts "used [s allocator_allocated]"
|
||
puts "rss [s allocator_active]"
|
||
puts "frag [s allocator_frag_ratio]"
|
||
puts "frag_bytes [s allocator_frag_bytes]"
|
||
}
|
||
assert_lessthan_equal [s allocator_frag_ratio] 1.5
|
||
}
|
||
}
|
||
|
||
test "Active defrag for argv retained by the main thread from IO thread: $type" {
|
||
r flushdb
|
||
r config set hz 100
|
||
r config set activedefrag no
|
||
wait_for_defrag_stop 500 100
|
||
r config resetstat
|
||
set io_threads [lindex [r config get io-threads] 1]
|
||
if {$io_threads == 1} {
|
||
r config set active-defrag-threshold-lower 5
|
||
} else {
|
||
r config set active-defrag-threshold-lower 10
|
||
}
|
||
r config set active-defrag-cycle-min 65
|
||
r config set active-defrag-cycle-max 75
|
||
r config set active-defrag-ignore-bytes 1000kb
|
||
r config set maxmemory 0
|
||
|
||
# Create some clients so that they are distributed among different io threads.
|
||
set clients {}
|
||
for {set i 0} {$i < 8} {incr i} {
|
||
lappend clients [redis_client]
|
||
}
|
||
|
||
# Populate memory with interleaving key pattern of same size
|
||
set dummy "[string repeat x 400]"
|
||
set n 10000
|
||
for {set i 0} {$i < [llength $clients]} {incr i} {
|
||
set rr [lindex $clients $i]
|
||
for {set j 0} {$j < $n} {incr j} {
|
||
$rr set "k$i-$j" $dummy
|
||
}
|
||
}
|
||
|
||
# If io-threads is enable, verify that memory allocation is not from the main thread.
|
||
if {$io_threads != 1} {
|
||
# At least make sure that bin 448 is created in the main thread's arena.
|
||
r set k dummy
|
||
r del k
|
||
|
||
# We created 10000 string keys of 400 bytes each for each client, so when the memory
|
||
# allocation for the 448 bin in the main thread is significantly smaller than this,
|
||
# we can conclude that the memory allocation is not coming from it.
|
||
set malloc_stats [r memory malloc-stats]
|
||
if {[regexp {(?s)arenas\[0\]:.*?448[ ]+[\d]+[ ]+([\d]+)[ ]} $malloc_stats - allocated]} {
|
||
# Ensure the allocation for bin 448 in the main thread’s arena
|
||
# is far less than 4375k (10000 * 448 bytes).
|
||
assert_lessthan $allocated 200000
|
||
} else {
|
||
fail "Failed to get the main thread's malloc stats."
|
||
}
|
||
}
|
||
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
if {$::verbose} {
|
||
puts "used [s allocator_allocated]"
|
||
puts "rss [s allocator_active]"
|
||
puts "frag [s allocator_frag_ratio]"
|
||
puts "frag_bytes [s allocator_frag_bytes]"
|
||
}
|
||
assert_lessthan [s allocator_frag_ratio] 1.05
|
||
|
||
# Delete keys with even indices to create fragmentation.
|
||
for {set i 0} {$i < [llength $clients]} {incr i} {
|
||
set rd [lindex $clients $i]
|
||
for {set j 0} {$j < $n} {incr j 2} {
|
||
$rd del "k$i-$j"
|
||
}
|
||
}
|
||
for {set i 0} {$i < [llength $clients]} {incr i} {
|
||
[lindex $clients $i] close
|
||
}
|
||
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
if {$::verbose} {
|
||
puts "used [s allocator_allocated]"
|
||
puts "rss [s allocator_active]"
|
||
puts "frag [s allocator_frag_ratio]"
|
||
puts "frag_bytes [s allocator_frag_bytes]"
|
||
}
|
||
assert_morethan [s allocator_frag_ratio] 1.4
|
||
|
||
catch {r config set activedefrag yes} e
|
||
if {[r config get activedefrag] eq "activedefrag yes"} {
|
||
|
||
# wait for the active defrag to start working (decision once a second)
|
||
wait_for_condition 50 100 {
|
||
[s total_active_defrag_time] ne 0
|
||
} else {
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
puts [r info memory]
|
||
puts [r info stats]
|
||
puts [r memory malloc-stats]
|
||
fail "defrag not started."
|
||
}
|
||
|
||
# wait for the active defrag to stop working
|
||
wait_for_defrag_stop 500 100
|
||
|
||
# test the fragmentation is lower
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
if {$::verbose} {
|
||
puts "used [s allocator_allocated]"
|
||
puts "rss [s allocator_active]"
|
||
puts "frag [s allocator_frag_ratio]"
|
||
puts "frag_bytes [s allocator_frag_bytes]"
|
||
}
|
||
|
||
if {$io_threads == 1} {
|
||
assert_lessthan_equal [s allocator_frag_ratio] 1.05
|
||
} else {
|
||
# TODO: When multithreading is enabled, argv may be created in the io thread
|
||
# and kept in the main thread, which can cause fragmentation to become worse.
|
||
assert_lessthan_equal [s allocator_frag_ratio] 1.1
|
||
}
|
||
}
|
||
}
|
||
|
||
if {$type eq "standalone"} { ;# skip in cluster mode
|
||
test "Active defrag big list: $type" {
|
||
r flushdb
|
||
r config set hz 100
|
||
r config set activedefrag no
|
||
wait_for_defrag_stop 500 100
|
||
r config resetstat
|
||
r config set active-defrag-max-scan-fields 1000
|
||
r config set active-defrag-threshold-lower 5
|
||
r config set active-defrag-cycle-min 65
|
||
r config set active-defrag-cycle-max 75
|
||
r config set active-defrag-ignore-bytes 2mb
|
||
r config set maxmemory 0
|
||
r config set list-max-ziplist-size 5 ;# list of 500k items will have 100k quicklist nodes
|
||
|
||
# create big keys with 10k items
|
||
set rd [redis_deferring_client]
|
||
|
||
set expected_frag 1.5
|
||
# add a mass of list nodes to two lists (allocations are interlaced)
|
||
set val [string repeat A 100] ;# 5 items of 100 bytes puts us in the 640 bytes bin, which has 32 regs, so high potential for fragmentation
|
||
set elements 500000
|
||
for {set j 0} {$j < $elements} {incr j} {
|
||
$rd lpush biglist1 $val
|
||
$rd lpush biglist2 $val
|
||
}
|
||
for {set j 0} {$j < $elements} {incr j} {
|
||
$rd read ; # Discard replies
|
||
$rd read ; # Discard replies
|
||
}
|
||
|
||
# create some fragmentation
|
||
r del biglist2
|
||
|
||
# start defrag
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
set frag [s allocator_frag_ratio]
|
||
if {$::verbose} {
|
||
puts "frag $frag"
|
||
}
|
||
|
||
assert {$frag >= $expected_frag}
|
||
r config set latency-monitor-threshold 5
|
||
r latency reset
|
||
|
||
set digest [debug_digest]
|
||
catch {r config set activedefrag yes} e
|
||
if {[r config get activedefrag] eq "activedefrag yes"} {
|
||
# wait for the active defrag to start working (decision once a second)
|
||
wait_for_condition 50 100 {
|
||
[s total_active_defrag_time] ne 0
|
||
} else {
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
puts [r info memory]
|
||
puts [r info stats]
|
||
puts [r memory malloc-stats]
|
||
fail "defrag not started."
|
||
}
|
||
|
||
# wait for the active defrag to stop working
|
||
wait_for_defrag_stop 500 100
|
||
|
||
# test the fragmentation is lower
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
set misses [s active_defrag_misses]
|
||
set hits [s active_defrag_hits]
|
||
set frag [s allocator_frag_ratio]
|
||
set max_latency 0
|
||
foreach event [r latency latest] {
|
||
lassign $event eventname time latency max
|
||
if {$eventname == "active-defrag-cycle"} {
|
||
set max_latency $max
|
||
}
|
||
}
|
||
if {$::verbose} {
|
||
puts "used [s allocator_allocated]"
|
||
puts "rss [s allocator_active]"
|
||
puts "frag_bytes [s allocator_frag_bytes]"
|
||
puts "frag $frag"
|
||
puts "misses: $misses"
|
||
puts "hits: $hits"
|
||
puts "max latency $max_latency"
|
||
puts [r latency latest]
|
||
puts [r latency history active-defrag-cycle]
|
||
puts [r memory malloc-stats]
|
||
}
|
||
assert {$frag < 1.1}
|
||
# due to high fragmentation, 100hz, and active-defrag-cycle-max set to 75,
|
||
# we expect max latency to be not much higher than 7.5ms but due to rare slowness threshold is set higher
|
||
if {!$::no_latency} {
|
||
assert {$max_latency <= 30}
|
||
}
|
||
|
||
# in extreme cases of stagnation, we see over 20m misses before the tests aborts with "defrag didn't stop",
|
||
# in normal cases we only see 100k misses out of 500k elements
|
||
assert {$misses < $elements}
|
||
}
|
||
# verify the data isn't corrupted or changed
|
||
set newdigest [debug_digest]
|
||
assert {$digest eq $newdigest}
|
||
r save ;# saving an rdb iterates over all the data / pointers
|
||
r del biglist1 ;# coverage for quicklistBookmarksClear
|
||
} {1}
|
||
|
||
test "Active defrag edge case: $type" {
|
||
# there was an edge case in defrag where all the slabs of a certain bin are exact the same
|
||
# % utilization, with the exception of the current slab from which new allocations are made
|
||
# if the current slab is lower in utilization the defragger would have ended up in stagnation,
|
||
# kept running and not move any allocation.
|
||
# this test is more consistent on a fresh server with no history
|
||
start_server {tags {"defrag"} overrides {save ""}} {
|
||
r flushdb
|
||
r config set hz 100
|
||
r config set activedefrag no
|
||
wait_for_defrag_stop 500 100
|
||
r config resetstat
|
||
r config set active-defrag-max-scan-fields 1000
|
||
r config set active-defrag-threshold-lower 5
|
||
r config set active-defrag-cycle-min 65
|
||
r config set active-defrag-cycle-max 75
|
||
r config set active-defrag-ignore-bytes 1mb
|
||
r config set maxmemory 0
|
||
set expected_frag 1.3
|
||
|
||
r debug mallctl-str thread.tcache.flush VOID
|
||
# fill the first slab containing 32 regs of 640 bytes.
|
||
for {set j 0} {$j < 32} {incr j} {
|
||
r setrange "_$j" 600 x
|
||
r debug mallctl-str thread.tcache.flush VOID
|
||
}
|
||
|
||
# add a mass of keys with 600 bytes values, fill the bin of 640 bytes which has 32 regs per slab.
|
||
set rd [redis_deferring_client]
|
||
set keys 640000
|
||
for {set j 0} {$j < $keys} {incr j} {
|
||
$rd setrange $j 600 x
|
||
}
|
||
for {set j 0} {$j < $keys} {incr j} {
|
||
$rd read ; # Discard replies
|
||
}
|
||
|
||
# create some fragmentation of 50%
|
||
set sent 0
|
||
for {set j 0} {$j < $keys} {incr j 1} {
|
||
$rd del $j
|
||
incr sent
|
||
incr j 1
|
||
}
|
||
for {set j 0} {$j < $sent} {incr j} {
|
||
$rd read ; # Discard replies
|
||
}
|
||
|
||
# create higher fragmentation in the first slab
|
||
for {set j 10} {$j < 32} {incr j} {
|
||
r del "_$j"
|
||
}
|
||
|
||
# start defrag
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
set frag [s allocator_frag_ratio]
|
||
if {$::verbose} {
|
||
puts "frag $frag"
|
||
}
|
||
|
||
assert {$frag >= $expected_frag}
|
||
|
||
set digest [debug_digest]
|
||
catch {r config set activedefrag yes} e
|
||
if {[r config get activedefrag] eq "activedefrag yes"} {
|
||
# wait for the active defrag to start working (decision once a second)
|
||
wait_for_condition 50 100 {
|
||
[s total_active_defrag_time] ne 0
|
||
} else {
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
puts [r info memory]
|
||
puts [r info stats]
|
||
puts [r memory malloc-stats]
|
||
fail "defrag not started."
|
||
}
|
||
|
||
# wait for the active defrag to stop working
|
||
wait_for_defrag_stop 500 100
|
||
|
||
# test the fragmentation is lower
|
||
after 120 ;# serverCron only updates the info once in 100ms
|
||
set misses [s active_defrag_misses]
|
||
set hits [s active_defrag_hits]
|
||
set frag [s allocator_frag_ratio]
|
||
if {$::verbose} {
|
||
puts "frag $frag"
|
||
puts "hits: $hits"
|
||
puts "misses: $misses"
|
||
}
|
||
assert {$frag < 1.1}
|
||
assert {$misses < 10000000} ;# when defrag doesn't stop, we have some 30m misses, when it does, we have 2m misses
|
||
}
|
||
|
||
# verify the data isn't corrupted or changed
|
||
set newdigest [debug_digest]
|
||
assert {$digest eq $newdigest}
|
||
r save ;# saving an rdb iterates over all the data / pointers
|
||
}
|
||
} ;# standalone
|
||
}
|
||
}
|
||
}
|
||
|
||
start_cluster 1 0 {tags {"defrag external:skip cluster"} overrides {appendonly yes auto-aof-rewrite-percentage 0 save "" loglevel debug}} {
|
||
test_active_defrag "cluster"
|
||
}
|
||
|
||
start_server {tags {"defrag external:skip standalone"} overrides {appendonly yes auto-aof-rewrite-percentage 0 save "" loglevel debug}} {
|
||
test_active_defrag "standalone"
|
||
}
|
||
} ;# run_solo
|