This change adds a new counter that tracks the total number
of permanently allocated pages.
Differential Revision: https://reviews.freebsd.org/D46978
Reviewed by: alc, markj
Add _init to the function names of the functions that initialize
iterators for swblks.
Reported by: alc, markj
Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D46974
Use reserved work for kvo_flags.
Mark such object with KVMO_FLAG_SYSVSHM.
Provide segment key in kvo_vn_fileid, vnode never can back shm mapping.
Provide sequence number in kvo_vn_fsid_freebsd11.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D46959
Take advantage of a nearby 2-byte hole to avoid growing the struct.
This way, only the offsets of "flags" and "pg_color" change. Bump
__FreeBSD_version since some out-of-tree kernel modules may access these
fields, though I haven't found any examples so far.
Reviewed by: kib
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D35905
Allocate the 'zero_region' page using VM_ALLOC_NOFREE since
it never gets released.
Differential Revision: https://reviews.freebsd.org/D46885
Reviewed by: alc, markj, kib
Replace calls to pctrie lookup and remove functions, which always
begin with a search from the pctrie root, with the use of pctrie
iterators to traverse and remove items from the pctrie of swapblks
without duplicating searches. Take care to reset iterators after
reacquiring an object lock, since with the lock released they could be
invalidated.
Reviewed by: alc, markj, kib
Tested by: pho (previous versions)
Differential Revision: https://reviews.freebsd.org/D46620
Replace the use of page lookups and page next pointers in swap_pager.c
with vm_page iterators.
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D46704
Undo a change in swap_pager_swapoff_object that assumed a swap block
would be valid after reacquiring a lock, when that validity cannot be
assumed.
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D46753
Allocate the 'bogus_page' page using VM_ALLOC_NOFREE since
it never gets released.
Differential Revision: https://reviews.freebsd.org/D46699
Reviewed by: alc, markj, kib
Instead of iterating over object->memq to free pages, use a callback
in VM_RADIX_PCTRIE_RECLAIM_CALLBACK to do it.
Reviewed by: rlibby
Differential Revision: https://reviews.freebsd.org/D45588
Add to the vm_radix and vm_page interfaces methods to use pctrie
iterators with vm_radix tries.
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D46663
Clarify comments about what happens to source blocks in
swp_pager_meta_transfer. No functional changes.
Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D46682
Function swp_pager_meta_transfer expects that after dropping and
reacquiring an object lock, the swap block it's processing still
exists, and has not been removed from the trie and freed. Rewrite to
avoid depending on that, by scrubbing, removing and freeing it before
releasing the lock.
Reviewed by: alc, markj
Differential Revision: https://reviews.freebsd.org/D46629
Combine two loops, each iterating over the same array of pages to
initialize them, into a single loop.
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D46609
Define wrapper functions for the pctrie operations specific to
swap_pager, to hide some verbose details. Separate the meta_transfer
and meta_free functions into separate functions.
Reviewed by: kib (previous version)
Tested by: pho (previous version)
Differential Revision: https://reviews.freebsd.org/D46315
Make vm_phys_alloc_freelist_pages static. There are no longer any
callers outside of vm_phys.c.
Reviewed by: alc, markj
Differential Revision: https://reviews.freebsd.org/D46539
Callers of cdev_pager_free_page in the kernel always have object->type
== OBJT_MGTDEVICE. Define a function for them to call that skips the
runtime type check in cdev_pager_free.
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D46389
Function swp_pager_meta_transfer uses 'pindex' as the start address of
the swblk in calculating which page to lookup in order to count freed
pages. However, the lookup for a swblk at 'pindex' or greater may
produce one greater than 'pindex', given by sb->p, and that's the
value that should be used to compute a page adddress.
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D46234
Rewrite vm_pageout_cluster to eliminate redundant variables and
duplicated code.
Remove tests on pindex to check for object boundary conditions, since
the page_next and page_prev functions return NULL at the object
boundaries. Fix an alignment error that could happen if pindex is
aligned, and the first of vm_pageout_page_count flushable pages, and
the page at pindex-1 is also flushable.
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D46217
Ensure that pmap_init() properly initialized pagesizes[]. In part, we
are making this change to document the requirement that the non-zero
elements of pagesizes[] must be in ascending order.
Reviewed by: kib, markj
Attempting to reduce vm_pageout_page_count at startup when the machine
has less than 8MB of physical memory is pointless, since we haven't run
on machines with so little memory in ages.
Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D46206
The array passed to vm_pageout_flush, and constructed in a middle-out
fashion, can never use array element zero. Shrink the array by one,
and reduce indices by one, to save that bit of stack space. In the
vm_object version, make the accounting look more like the pageout
version.
Reported by: alc
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D46208
Pass VM_NOFREE_IMPORT_ORDER to vm_phys_alloc_pages instead of
VM_LEVEL_0_ORDER when allocating a higher-order page for
the NOFREE page allocator.
Reported by: alc
Fixes: a8693e8
This patch adds a new KVA arena for separating M_NEVERFREED allocations.
Separating KVAs for pages that are never freed should facilitate
superpage promotion in the kernel.
Differential Revision: https://reviews.freebsd.org/D45997
Reviewed by: alc, kib, markj
Tested by: alc
This patch adds a reservation-aware bump allocator intended for
allocating NOFREE pages. The main goal of this change is to reduce the
long-term fragmentation issues caused by pages that are never freed during runtime.
The `vm_page_alloc_nofree_domain` routine hands out 0-order pages from
a preallocated superpage. Once an active NOFREE superpage fills up, the
routine will try to allocate a new one and discard the old one.
This routine will get invoked whenever VM_ALLOC_NOFREE is passed to
vm_page_alloc_noobj or vm_page_alloc.
Differential Revision: https://reviews.freebsd.org/D45863
Reviewed by: alc, kib, markj
Tested by: alc
This patch adds two additional vm_page flags to distinguish pages that
never get released while the system is running (e.g. UMA_ZONE_NOFREE slabs).
Differential Revision: https://reviews.freebsd.org/D45970
Reviewed by: alc, kib, markj
Tested by: alc
This patch adds an additional malloc(9) flag to distinguish allocations
that are never freed during runtime.
Differential Revision: https://reviews.freebsd.org/D45045
Reviewed by: alc, kib, markj
Tested by: alc
vm_pageout_cluster prepares an array for passing to vm_pageout_flush
by starting in the middle of a double-sized array and working out from
the middle. Using the same technique in vm_object_page_collect_flush
saves one loop that traverses a piece of linked list, and 80 bytes of
amd64 binary code.
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D46173
Remove the NO_SWAPPING option. There is still some code in
vm_swapout.c, but it relates to RACCT handling. Remove the option and
make compilation of vm_swapout.c conditional on RACCT.
Tested by: pho
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D46130
- The kernel stack objects do not need to be pageable, so use OBJT_PHYS
objects instead. The main difference is that mappings do not require
PV entries.
- Make some externally visible functions, relating to kernel thread
stack internals, private to vm_glue.c, as their external consumers are
now gone.
Tested by: pho
Reviewed by: alc, kib
Differential Revision: https://reviews.freebsd.org/D46119
- Remove most checks of the P_INMEM flag.
- Some uses remain since a few userspace tools, e.g., ps(1) and top(1)
expect the flag to be set. These can be cleaned up but the code has
most likely been copy-pasted elsewhere and while linger for a long
time.
Tested by: pho
Reviewed by: alc, imp, kib
Differential Revision: https://reviews.freebsd.org/D46117
- Remove the IS_SWAPPED thread inhibitor state.
- Remove all uses of TD_IS_SWAPPED() in the kernel.
- Remove the TDF_CANSWAP flag.
- Remove the P_SWAPPINGOUT and P_SWAPPINGIN flags.
Tested by: pho
Reviewed by: alc, imp, kib
Differential Revision: https://reviews.freebsd.org/D46115
- Modify PHOLD() to no longer fault in the process.
- Remove _PHOLD_LITE(), which is now the same as _PHOLD(), fix up
consumers.
- Remove faultin() and its callees.
Tested by: pho
Reviewed by: imp, kib
Differential Revision: https://reviews.freebsd.org/D46114
After mi_startup() finishes, thread0 becomes the "swapper", whose
responsibility is to swap threads back in on demand. Now that threads
can't be swapped out, there is no use for this thread. Just sleep
forever once sysinits are finished; thread_exit() doesn't work because
thread0 is allocated statically. The thread could be repurposed if that
would be useful.
Tested by: pho
Reviewed by: alc, imp, kib
Differential Revision: https://reviews.freebsd.org/D46113
Where vm_fault calls vm_page_next, replace it with a use of TAILQ_NEXT
and a KASSERT. This avoids needless computation in a NODEBUG kernel
and makes the error checking clearer in a GENERIC kernel.
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D46168
Once upon a time, I created vm_page_alloc_freelist{,_domain}() to
support faster allocation of pages that were mapped by the partial
direct map on 32-bit MIPS. At the time, I expected that these
functions might find other uses too, but those other uses never
materialized. So, these functions have not been used for some time
now. Instead, people use the more general vm_page_alloc_contig().
Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D46063
It has simply been an alias for the kernel_arena for many years now.
Enough time has passed to retire it. Any out-of-tree kernel modules
that directly use kmem_arena should switch to kernel_arena.
Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D46057
It is supposed to be used for ref_count manipulations when the pages
are owned by an object, but ref_count is used for something else than
the wiring, e.g. PTE population count on the page table page.
Reviewed by: markj
Sponsored by: Advanced Micro Devices (AMD)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D45910
Update pagesizes[] to include the L3 ATTR_CONTIGUOUS (L3C) page size,
which is 64KB when the base page size is 4KB and 2MB when the base page
size is 16KB.
Add support for L3C pages to shm_create_largepage().
Add support for creating L3C page mappings to pmap_enter(psind=1).
Add support for reporting L3C page mappings to mincore(2) and
procstat(8).
Update vm_fault_soft_fast() and vm_fault_populate() to handle multiple
superpage sizes.
Declare arm64 as supporting two superpage reservation sizes, and
simulate two superpage reservation sizes, updating the vm_page's psind
field to reflect the correct page size from pagesizes[]. (The next
patch in this series will replace this simulation. This patch is
already big enough.)
Co-authored-by: Eliot Solomon <ehs3@rice.edu>
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D45766
