Summary:
It can be useful to see a summary of CPU caches on bootup. This is done
for most platforms already, so add this to arm64, in the form of (taken
from Apple M1 pro test):
L1 cache: 192KB (instruction), 128KB (data)
L2 cache: 12288KB (unified)
This is printed out per-CPU, only under bootverbose.
Future refinements could instead determine if a cache level is shared
with other cores (L2 is shared among cores on some SoCs, for instance),
and perform a better calculation to the full true cache sizes. For
instance, it's known that the M1 pro, on which this test was done, has 2
12MB L2 clusters, for a total of 24MB. Seeing each CPU with 12288KB L2
would make one think that there's 12MB * NCPUs, for possibly 120MB
cache, which is incorrect.
Sponsored by: Juniper Networks, Inc.
Reviewed by: #arm64, andrew
Differential Revision: https://reviews.freebsd.org/D35366
(cherry picked from commit 139ba152c9)
To enable it user-space needs to call feenableexcept().
FPE_FLTIDO has been added as the IDF bit can't be mapped to any existing
FPE code.
Reviewed by: andrew@
Differential revision: https://reviews.freebsd.org/D35247
MFC after: 2 weeks
(cherry picked from commit 6e2caba7a1)
This includes adding support for NT_ARM_VFP for 32-bit binaries
running under aarch64 kernels both for ptrace(), and coredumps via the
kernel and gcore.
Reviewed by: andrew, markj
Sponsored by: University of Cambridge, Google, Inc.
Differential Revision: https://reviews.freebsd.org/D34448
(cherry picked from commit add00c381e)
Move the common kernel function signatures from machine/reg.h to a new
sys/reg.h. This is in preperation for adding PT_GETREGSET to ptrace(2).
Reviewed by: imp, markj
Sponsored by: DARPA, AFRL (original work)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D19830
(cherry picked from commit b792434150)
Logical and ('&&') was used to join two conditions instead of logical
or ('||') causing some store instructions to not be recognized.
Reported by: GCC 9 -Wparentheses
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D31949
(cherry picked from commit 8753039a8f)
A different exception is raised when we hit a 32bits breakpoint, rather than
a 64bits one, so handle those as well when COMPAT_FREEBSD32 is defined.
This should fix SIGBUS at least when using breakpoints with thumb2 code.
PR: 256468
MFC After: 1 week
(cherry picked from commit 2734050154)
When a DMA request using bounce pages completes, a swi is triggered to
schedule pending DMA requests using the just-freed bounce pages. For
a long time this bus_dma swi has been tied to a "virtual memory" swi
(swi_vm). However, all of the swi_vm implementations are the same and
consist of checking a flag (busdma_swi_pending) which is always true
and if set calling busdma_swi. I suspect this dates back to the
pre-SMPng days and that the intention was for swi_vm to serve as a
mux. However, in the current scheme there's no need for the mux.
Instead, remove swi_vm and vm_ih. Each bus_dma implementation that
uses bounce pages is responsible for creating its own swi (busdma_ih)
which it now schedules directly. This swi invokes busdma_swi directly
removing the need for busdma_swi_pending.
One consequence is that the swi now works on RISC-V which had previously
failed to invoke busdma_swi from swi_vm.
Reviewed by: imp, kib
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D33447
(cherry picked from commit 254e4e5b77)
The header exports the following:
- Definition of struct tcb.
- Helpers to get/set the tcb for the current thread.
- TLS_TCB_SIZE (size of TCB)
- TLS_TCB_ALIGN (alignment of TCB)
- TLS_VARIANT_I or TLS_VARIANT_II
- TLS_DTV_OFFSET (bias of pointers in dtv[])
- TLS_TP_OFFSET (bias of "thread pointer" relative to TCB)
Note that TLS_TP_OFFSET does not account for if the unbiased thread
pointer points to the start of the TCB (arm and x86) or the end of the
TCB (MIPS, PowerPC, and RISC-V).
Note also that for amd64, the struct tcb does not include the unused
tcb_spare field included in the current structure in libthr. libthr
does not use this field, and the existing calls in libc and rtld that
allocate a TCB for amd64 assume it is the size of 3 Elf_Addr's (and
thus do not allocate room for tcb_spare).
A <sys/_tls_variant_i.h> header is used by architectures using
Variant I TLS which uses a common struct tcb.
Reviewed by: kib (older version of x86/tls.h), jrtc27
Sponsored by: The University of Cambridge, Google Inc.
Differential Revision: https://reviews.freebsd.org/D33351
For stable/13 only, sys/arm/include/tls.h includes support for
ARM_TP_ADDRESS which is not present in main.
(cherry picked from commit 1a62e9bc00)
To support cc -pg on arm64 we need to implement .mcount. As clang and
gcc think it is function like it just needs to load the arguments
to _mcount and call it.
On gcc the first argument is passed in x0, however this is missing on
clang so we need to load it from the stack. As it's the caller return
address this will be at a known location.
PR: 262709
Reviewed by: emaste (earlier version)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34634
(cherry picked from commit 16d5f9a164)
On arm64 we can select how strongly we order device memory. Currently
we use the strongest type of non-Gathering, non-Reordering, no Early
write acknowledgement. This is equivalent to VM_MEMATTR_SO in the 32-bit
arm code.
Create a new memory type to remove the no Early write acknowledgement
option to create a memory attribute that is equivalent to the arm
VM_MEMATTR_DEVICE.
Keep the the old nGnRnE memory as what we provide for VM_MEMATTR_DEVICE
until we can test nGnRE on more hardware. A method for dynamically
switching back may be needed as at least one vendor is known to have
broken nGnRE memory.
Sponsored by: Innovate UK
(cherry picked from commit 13ec5a6da0)
This was one of the last registers to not be in alphabetical order in
armreg.h. Fix this to make it easier to find.
Sponsored by: Innovate UK
(cherry picked from commit a1b4e4fa9a)
They are in a different order to the TCR_TG1 values but appear to have
been copied incorrectly.
While here use TCR_TG0_4K in locore.S to make it explicit the userspace
page size is 4K.
Sponsored by: The FreeBSD Foundation
(cherry picked from commit f62e099ed5)
Following ARMARM sec D5.2.11, which says:
> Where an instruction results in an update to a System register,
> as is the case with the AT * address translation instructions,
> explicit synchronization must be performed before the result is
> guaranteed to be visible to subsequent direct reads of the
> PAR_EL1.
Reviewed By: andrew
MFC after: 3 weeks
Sponsored by: Ampere Computing
Differential Revision: https://reviews.freebsd.org/D34665
(cherry picked from commit 7be7bd6775)
We assume the pointer returned from get_pcpu will be consistent even
if the thread is moved to a new CPU. Fix this by partially reverting
63c858a04d to make get_pcpu a function again.
Sponsored by: The FreeBSD Foundation
(cherry picked from commit ed30663426)
On some systems (e.g. Lenovo ThinkPad X240, Apple MacBookPro12,1)
the SMBIOS entry point is not found in the <0xFFFFF space.
Follow the SMBIOS spec and use the EFI Configuration Table for
locating the entry point on EFI systems.
Reviewed by: rpokala, dab
MFC after: 1 week
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D29276
(cherry picked from commit a29bff7a52)
We should clear the single step flag when entering a signal hander and
set it when returning. This fixes the ptrace__PT_STEP_with_signal test.
While here add support for userspace to set the single step bit as on
x86. This can be used by userspace for self tracing.
Reviewed by: kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34170
(cherry picked from commit 31cf95cec7)
While here clean up the names for the naming convention of the other
registers in this file.
Reviewed by: kib, mhorne (earlier version)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34060
(cherry picked from commit 664640ba6c)
To trace leaf asm functions we can insert a single nop instruction as
the first instruction in a function and trigger off this.
Reviewed by: gnn
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D28132
(cherry picked from commit 28d945204e)
These ones were unambiguous cases where the Foundation was the only
listed copyright holder (in the associated license block).
Sponsored by: The FreeBSD Foundation
(cherry picked from commit 9feff969a0)
Now we have a macro to descrive the arm64 TTBR registers use it in the
ASID_TO_OPERAND pmap macro.
Sponsored by: The FreeBSD Foundation
(cherry picked from commit 459e0b92e8)
When writing to memory on arm64 we may be trying to be accessing a
read-only page. In this case try to access via the DMAP region to
get a writable location.
While here simplify writing data in DDB and stop trashing the size as
it is passed into the cache handling functions.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32053
(cherry picked from commit 3d2533f5c2)
We only need to include sys/_atomic_subword.h on arm64 to provide
atomic_testandset_acq_long. Add an implementation in the arm64 atomic.h
based on the existing atomic_testandset macro.
Reviewed by: kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33587
(cherry picked from commit 02c16e2174)
With the update to llvm 13 we are able to tell the compiler it can find
the SSP canary relative to the register that holds the userspace stack
pointer. As this is unused in most of the kernel it can be used here
to point to a per-thread SSP canary.
As the kernel could be built with an old toolchain, e.g. when upgrading
from 13, add a warning that the options was enabled but the compiler
doesn't support it to both the build and kernel boot.
Discussed with: emaste
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33079
(cherry picked from commit ae92ace05f)
To minimise NUMA traffic allocate the pcpu, dpcpu, and boot stacks in
the correct domain when possible.
Submitted by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32338
(cherry picked from commit a90ebeb5fe)
We do not consider the space reserved for the pcb to be part of the
total kstack size, so it should not be included in the calculation of
the used stack size.
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
(cherry picked from commit b02908b051)
Make sys/reg.h includable on aarch64 by making machine/reg.h
self-contained: Include sys/_types.h and use __uint* instead of uint*.
Sponsored by: Netflix
(cherry picked from commit 8ee8271e22)
The minidump code is written assuming that certain global state will not
change, and rightly so, since it executes from a kernel debugger
context. In order to support taking minidumps of a live system, we
should allow copies of relevant global state that is likely to change to
be passed as parameters to the minidumpsys() function.
This patch does the work of parameterizing this function, by adding a
struct minidumpstate argument. For now, this struct allows for copies of
the kernel message buffer, and the bitset that tracks which pages should
be dumped (vm_page_dump). Follow-up changes will actually make use of
these arguments.
Notably, dump_avail[] does not need a snapshot, since it is not expected
to change after system initialization.
The existing minidumpsys() definitions are renamed, and a thin MI
wrapper is added to kern_dump.c, which handles the construction of
the state struct. Thus, calling minidumpsys() remains as simple as
before.
Reviewed by: kib, markj, jhb
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D31989
(cherry picked from commit 1adebe3cd6)
Make it easy to define interceptors for new sanitizer runtimes, rather
than assuming KCSAN. Lay a bit of groundwork for KASAN and KMSAN.
When a sanitizer is compiled in, atomic(9) and bus_space(9) definitions
in atomic_san.h are used by default instead of the inline
implementations in the platform's atomic.h. These definitions are
implemented in the sanitizer runtime, which includes
machine/{atomic,bus}.h with SAN_RUNTIME defined to pull in the actual
implementations.
No functional change intended.
Sponsored by: The FreeBSD Foundation
(cherry picked from commit 3ead60236f)
Support changing the protection of preloaded kernel modules by
implementing pmap_change_prot on arm64 and calling it from
preload_protect.
Reviewed by: alc (previous version)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32026
(cherry picked from commit a85ce4ad72)
Add the table page table attributes on arm64 and use them to add
restrictions to the block and page entries below them. This ensures
we are unable to increase the permissions in these last level entries
without also changing them in the upper levels.
Use the attributes to ensure the kernel can't execute from userspace
memory and vice versa, userspace has no access to read or write kernel
memory, and that the DMAP region is non-executable.
Reviewed by: alc, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32081
(cherry picked from commit f6de51d3e0)
The implementation of the progress bar is simple, but duplicated for
most minidump implementations. Extract the common bits to kern_dump.c.
Ensure that the bar is reset with each subsequent dump; this was only
done on some platforms previously.
Reviewed by: markj
MFC after: 2 weeks
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D31885
(cherry picked from commit ab4ed843a3)
These are needed to signal to the kernel when a Reliability,
Availability, and Serviceability (RAS) exception has triggered.
Reviewed by: mhorne
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31477
(cherry picked from commit 17b6ee9613)
Use these to simplify the definition of the user_regs array.
Reviewed by: imp, markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31333
(cherry picked from commit 10f6680faa)
The Arm Architecture Envelope Model is a simulator that models the
architecture rather than any specific implementation. Add its part ID
macro and add it to the list of Arm CPUs we can decode.
Sponsored by: The FreeBSD Foundation
(cherry picked from commit 2531f067ea)
On arm64 we should use bit 55 of the address to decide if aan address
is a user or kernel address. Add a new macro with this check and a
second to ensure the address is in teh canonical form, i.e.
the top bits are all zero or all one.
This will help with supporting future cpu features, including Top
Byte Ignore, Pointer Authentication, and Memory Tagging.
Reviewed by: kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31179
(cherry picked from commit b7a78d573a)
They are valid as of the ARMv8.7 XML.
While here remove SCTLR_RES0 as it's unused and depends on which CPU
the kernel is running on and switch to shifted values as they are
easier to compare with the documentation.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31120
(cherry picked from commit 5484e6d9cc)
They are valid as of the ARMv8.7 XML.
While here switch to use shifted values as they are easier to compare
with values in the Arm Reference Manual.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31093
(cherry picked from commit 3c1bfadaf9)
