Historically we have not distinguished between kernel wirings and user
wirings for accounting purposes. User wirings (via mlock(2)) were
subject to a global limit on the number of wired pages, so if large
swaths of physical memory were wired by the kernel, as happens with
the ZFS ARC among other things, the limit could be exceeded, causing
user wirings to fail.
The change adds a new counter, v_user_wire_count, which counts the
number of virtual pages wired by user processes via mlock(2) and
mlockall(2). Only user-wired pages are subject to the system-wide
limit which helps provide some safety against deadlocks. In
particular, while sources of kernel wirings typically support some
backpressure mechanism, there is no way to reclaim user-wired pages
shorting of killing the wiring process. The limit is exported as
vm.max_user_wired, renamed from vm.max_wired, and changed from u_int
to u_long.
The choice to count virtual user-wired pages rather than physical
pages was done for simplicity. There are mechanisms that can cause
user-wired mappings to be destroyed while maintaining a wiring of
the backing physical page; these make it difficult to accurately
track user wirings at the physical page layer.
The change also closes some holes which allowed user wirings to succeed
even when they would cause the system limit to be exceeded. For
instance, mmap() may now fail with ENOMEM in a process that has called
mlockall(MCL_FUTURE) if the new mapping would cause the user wiring
limit to be exceeded.
Note that bhyve -S is subject to the user wiring limit, which defaults
to 1/3 of physical RAM. Users that wish to exceed the limit must tune
vm.max_user_wired.
Reviewed by: kib, ngie (mlock() test changes)
Tested by: pho (earlier version)
MFC after: 45 days
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D19908
Initialize `oldlen` to the size of the value, instead of leaving the value
unitialized. Leaving it unitialized seems to work by accident on amd64 when
running 64-bit programs, but not on i386.
This matches patterns in use in other programs.
PR: 237458
Approved by: emaste (mentor; implicit)
MFC after: 1 week
Tested on: ^/head (amd64), ^/stable/11 (i386)
In short, the prior code was far too simplistic when it came to calling recv(2)
and failed intermittently (or in the case of Jenkins, deterministically).
Handle short recv(2)s by checking the return code and incrementing the window
into the buffer by the number of received bytes. If the number of received
bytes <= 0, then bail out of the loop, and test the total number of received
bytes vs the expected number of bytes sent for equality, and base whether or
not the test passes/fails on that fact.
Remove the expected failure, now that the hdtr testcases deterministically pass
on my host after this change [1].
PR: 234809 [1], 235200
Reviewed by: asomers
Approved by: emaste (mentor)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D19188
ci.FreeBSD.org does not have access to a DNS resolver/network (unlike my test
VM), so in order for the test to pass on the host, it needs to avoid the DNS
lookup by using the numeric host address representation.
PR: 235200
Reviewed by: asomers, lwhsu
Approved by: emaste (mentor)
MFC after: 2 weeks
MFC with: r343362, r343365, r343367-r343368, r343461
Differential Revision: https://reviews.freebsd.org/D19026
The return value (`err`) should be checked; not the `errno` value.
PR: 235200
Approved by: emaste (mentor)
Reviewed by: asomers, lwhsu
MFC after: 28 days
MFC with: r343362, r343365, r343367-r343368
Differential Revision: https://reviews.freebsd.org/D18969
I should have only changed the format qualifier with the `size_t` value,
`length`, not the other [`off_t`] value, `dest_file_size`.
MFC after: 1 month
MFC with: r343362, r343365, r343367
Approved by: emaste (mentor; implicit)
Reported by: gcc 8.x
gcc 8.x is more pedantic than clang 7.x with format strings and the tests
passed `void*` variables while supplying `%s` (which is technically
incorrect).
Make the affected `void*` variables use `char*` storage instead to address
this issue, as the compiler will upcast the values to `char*`.
MFC after: 1 month
MFC with: r343362
Approved by: emaste (mentor; implicit)
Reviewed by: asomers
Differential Revision: https://reviews.freebsd.org/D18934
These testcases exercise a number of functional requirements for sendfile(2).
The testcases use IPv4 and IPv6 domain sockets with TCP, and were confirmed
functional on UFS and ZFS. UDP address family sockets cannot be used per the
sendfile(2) contract, thus using UDP sockets is outside the scope of
testing the syscall in positive cases. As seen in
`:s_negative_udp_socket_test`, UDP is used to test the sendfile(2) contract
to ensure that EINVAL is returned by sendfile(2).
The testcases added explicitly avoid testing out `SF_SYNC` due to the
complexity of verifying that support. However, this is a good next logical
item to verify.
The `hdtr_positive*` testcases work to a certain degree (the header
testcases pass), but the trailer testcases do not work (it is an expected
failure). In particular, the value received by the mock server doesn't match
the expected value, and instead looks something like the following (using
python array notation):
`trailer[:]message[1:]`
instead of:
`message[:]trailer[:]`
This makes me think there's a buffer overrun issue or problem with the
offset somewhere in the sendfile(2) system call, but I need to do some
other testing first to verify that the code is indeed sane, and my
assumptions/code isn't buggy.
The `sbytes_negative` testcases that check `sbytes` being set to an
invalid value resulting in `EFAULT` fails today as the other change
(which checks `copyout(9)`) has not been committed [1]. Thus, it
should remain an expected failure (see bug 232210 for more details
on this item).
Next steps for testing sendfile(2):
1. Fix the header/trailer testcases so that they pass.
2. Setup if_tap interface and test with it, instead of using "localhost", per
@asomers's suggestion.
3. Handle short recv(2)'s in `server_cat(..)`.
4. Add `SF_SYNC` support.
5. Add some more negative tests outside the scope of the functional contract.
MFC after: 1 month
Reviewed by: asomers
Approved by: emaste (mentor)
PR: 232210
Sponsored by: Netflix, Inc
Differential Revision: https://reviews.freebsd.org/D18625
Previously, libc.so would initialize its notion of the break address
using _end, a special symbol emitted by the static linker following
the bss section. Compatibility issues between lld and ld.bfd could
cause the wrong definition of _end (libc.so's definition rather than
that of the executable) to be used, breaking the brk()/sbrk()
interface.
Avoid this problem and future interoperability issues by simply not
relying on _end. Instead, modify the break() system call to return
the kernel's view of the current break address, and have libc
initialize its state using an extra syscall upon the first use of the
interface. As a side effect, this appears to fix brk()/sbrk() usage
in executables run with rtld direct exec, since the kernel and libc.so
no longer maintain separate views of the process' break address.
PR: 228574
Reviewed by: kib (previous version)
MFC after: 2 months
Differential Revision: https://reviews.freebsd.org/D15663
Add a clock_nanosleep() syscall, as specified by POSIX.
Make nanosleep() a wrapper around it.
Attach the clock_nanosleep test from NetBSD. Adjust it for the
FreeBSD behavior of updating rmtp only when interrupted by a signal.
I believe this to be POSIX-compliant, since POSIX mentions the rmtp
parameter only in the paragraph about EINTR. This is also what
Linux does. (NetBSD updates rmtp unconditionally.)
Copy the whole nanosleep.2 man page from NetBSD because it is complete
and closely resembles the POSIX description. Edit, polish, and reword it
a bit, being sure to keep any relevant text from the FreeBSD page.
Reviewed by: kib, ngie, jilles
MFC after: 3 weeks
Relnotes: yes
Sponsored by: Dell EMC
Differential Revision: https://reviews.freebsd.org/D10020
This contains some new testcases in /usr/tests/...:
- .../lib/libc
- .../lib/libthr
- .../lib/msun
- .../sys/kern
Tested on: amd64, i386
MFC after: 1 month
This uses the same fix as r294894 did for the mlock test. The code from
that commit is moved into a common object file which PROGS supports
building first.
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D8689
it to lib/libc/tests/sys/Makefile [*]
Even though make -VPACKAGE and make -n install seem to do the right thing,
the effects are a bit different, depending on the build host.
MFC after: 1 week
Obtained from: HardenedBSD (af602f0db) [*]
Reported by: Oliver Pinter <oliver.pinter@hardenedbsd.org> [*]
Sponsored by: Dell EMC Isilon
- Some of the lib/libc and lib/thr tests fail
- lib/msun/exp_test:exp2_values now passes with clang 3.8.0
The Makefiles in contrib/netbsd-tests were pruned as they have no value
Sponsored by: EMC / Isilon Storage Division
after r298107
Summary of changes:
- Replace all instances of FILES/TESTS with ${PACKAGE}FILES. This ensures that
namespacing is kept with FILES appropriately, and that this shouldn't need
to be repeated if the namespace changes -- only the definition of PACKAGE
needs to be changed
- Allow PACKAGE to be overridden by callers instead of forcing it to always be
`tests`. In the event we get to the point where things can be split up
enough in the base system, it would make more sense to group the tests
with the blocks they're a part of, e.g. byacc with byacc-tests, etc
- Remove PACKAGE definitions where possible, i.e. where FILES wasn't used
previously.
- Remove unnecessary TESTSPACKAGE definitions; this has been elided into
bsd.tests.mk
- Remove unnecessary BINDIRs used previously with ${PACKAGE}FILES;
${PACKAGE}FILESDIR is now automatically defined in bsd.test.mk.
- Fix installation of files under data/ subdirectories in lib/libc/tests/hash
and lib/libc/tests/net/getaddrinfo
- Remove unnecessary .include <bsd.own.mk>s (some opportunistic cleanup)
Document the proposed changes in share/examples/tests/tests/... via examples
so it's clear that ${PACKAGES}FILES is the suggested way forward in terms of
replacing FILES. share/mk/bsd.README didn't seem like the appropriate method
of communicating that info.
MFC after: never probably
X-MFC with: r298107
PR: 209114
Relnotes: yes
Tested with: buildworld, installworld, checkworld; buildworld, packageworld
Sponsored by: EMC / Isilon Storage Division
netbsd-tests.test.mk (r289151)
- Eliminate explicit OBJTOP/SRCTOP setting
- Convert all ad hoc NetBSD test integration over to netbsd-tests.test.mk
- Remove unnecessary TESTSDIR setting
- Use SRCTOP where possible for clarity
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Divison
Each issue has a PR open to track. This workaround allows us to run the
tests to investigate the failures and avoid any new regressions.
PR: 202304, 202305, 202307
Reviewed by: ngie
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3378
500 new testcases
Various TODOs have been sprinkled around the Makefiles for items that even need
to be ported (missing features), testcases have issues with building/linking, or
issues at runtime.
A variant of this code has been tested extensively on amd64 and i386
10-STABLE/11-CURRENT for several months without issue. It builds on other
architectures, but the code will remain off until I have prove it works on
virtual hardware or real hardware on other architectures
In collaboration with: pho, Casey Peel <casey.peel@isilon.com>
Sponsored by: EMC / Isilon Storage Division
