Switch to INTR_ROOT_COUNT as this name better describes its purpose.
Remove the default INTR_ROOT_IRQ from the core. Define it (redundantly)
in each architecture's header, but now placed alongside its sibling
values (if defined by the platform, e.g. arm64 INTR_ROOT_FIQ).
Reviewed by: mhorne
Pull Request: https://github.com/freebsd/freebsd-src/pull/1280
In order to match reality, allow using these functions with pointers on
const objects, and bring us closer to C11.
Remove the '+' modifier in the atomic_load_acq_64_i586()'s inline asm
statement's constraint for '*p' (the value to load). CMPXCHG8B always
writes back some value, even when the value exchange does not happen in
which case what was read is written back. atomic_load_acq_64_i586()
further takes care of the operation atomically writing back the same
value that was read in any case. All in all, this makes the inline
asm's write back undetectable by any other code, whether executing on
other CPUs or code on the same CPU before and after the call to
atomic_load_acq_64_i586(), except for the fact that CMPXCHG8B will
trigger a #GP(0) if the memory address is part of a read-only mapping.
This unfortunate property is however out of scope of the C abstract
machine, and in particular independent of whether the 'uint64_t' pointed
to is declared 'const' or not.
Approved by: markj (mentor)
MFC after: 5 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D46887
Switch from creating the msr instructions to manage PAN to use the
"msr pan, #1" instruction directly. When this was added clang didn't
have support to assemble the instructions. This appears to have been
added to clang 13 which is sufficiently old enough.
Binutils releases from around the same time appear to have added this
instruction so any modern gcc should also support this instruction.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D47817
To allow for it to be more selective when we enable it, e.g. if the
CPU has an erratum that prevents us from doing so, move the check for
setting TCR_HD to C.
Reviewed by: alc
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D47808
Sometimes we need defines from this file in assembler code. Today we do
the heavyweight approach of using genassym for that. However, they are
just #defines, so in the future we want to include sys/intr.h to pick up
the needed constants in exception.S.
PR: 283041
Sponsored by: Netflix
Reviewed by: mmel, andrew
Differential Revision: https://reviews.freebsd.org/D47846
For current architectures, these are just aliases for the existing
operation on the relevant scalar integer.
Reviewed by: imp, kib
Obtained from: CheriBSD
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D47631
This reverts commit 536c8d948e. The
change seemed fine on the surface, but converting to an enum has raised
some concerns due to the asm <-> C interface. Back it out and let
someone else deal with it later if they'd like to.
Further context about the concerns can be found in D47279.
This is a function that calls into the EFI Runtime Services, but can
handle a fault. To support this add a handler in assembly that can
restore the kernel state on an exception and return a failure to the
caller.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46817
When we receive an exception while in a spinlock we shouldn't enable
interrupts. When entering a spinlock we disable interrupts so enabling
them here could cause surprising results.
The three cases that could cause this are:
1. A break-before-make sequence
2. Accessing possibly unmapped code with a fault handler
3. Buggy code
1 and 2 are supported later in the data abort handler, and 3 should be
fixed when found.
Reviewed by: mmel, kib, markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46816
sys/intr.h originally started life as an extract of arm's intr.h, and
this include was dropped in its place. Changes in flight want to add
some MD definitions that we'll use in the more MI parts of INTRNG.
Let's formally reverse the dependency now since this is way more
common in general. All of the includes switched in this change that I
spot-checked were in-fact wanting declarations historically included in
sys/intr.h anyways.
Reviewed by: andrew, imp, jrtc27, mhorne, mmel, olce
Differential Revision: https://reviews.freebsd.org/D47002
uint32_t is handy for directly interfacing with assembly-language. For
the C portion, enum is much handier. In particular there is no need to
count the number of roots by hand. This also works better for being
able to build kernels with varying numbers of roots.
Switch to INTR_ROOT_COUNT as this better matches the purpose of the
value. Switch to root_type, rather than rootnum for similar reasons.
Remove the default from the core. Better to require the architectures
to declare the type since they will routinely deviate and a default
chosen now will likely be suboptimal.
Leave intr_irq_handler() taking a register type as that better matches
for interfacing with assembly-language.
Rather than keeping the entire machine interrupt header assembly-safe,
switch to use of the existing genassym.c program to generate them. This
will be handier in the long-term, most of the header never needs to be
exposed to assembly-language.
Use the same method to decode CTR_EL0 as for the ID registers. This
will allow us to create a common view over all CPUs.
This will also allow us to create a common view for userspace and the
kernel if we detect a difference on some CPUs, or to handle errata.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D47124
In preperation for supporting non-4 bit field width to identcpu.c,
e.g. for ctr_el0, add the width of the existing ID register fields.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D47118
The VPIPT and AIVIVT cache types are reserved from Armv8.0. Remove
them as nothing will report these values.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D47117
The VPIPT I-cache policy has been removed from the architecture.
Remove support for it from pmap. The vmm side was not imported so
calls to pmap_invalidate_vpipt_icache would cause the kernel to branch
to a NULL pointer.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D47116
Report when SVE is present and allow it to be used by calling
sve_restore_state on an SVE exception from userspace.
Reviewed by: kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43310
As with floating point instructions don't trap SVE instructions to the
hypervisor. This lets us handle then in the kernel.
Reviewed by: imp (earlier version)
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43303
Add the NT_ARM_SVE note type and use it to access the SVE registers
from ptrace. This allows userspace to modify the full SVE register
values.
Try to follow the Linux semantics to allow debuggers to use this with
minimal changes.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43309
To allow for user space to read the SVE vector length add a sysarch
handler to return the value to userspace.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43308
Use the new extended register support in the arm64 signal frame to
handle the SVE registers.
As these registers alias the VFP registers we use the floating point
register values to restore the lower bits of the SVE registers. This
is to support software that doesn't understand SVE to continue working.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43307
Add initial kernel support for SVE. This detects if SVE is present on
all CPUs, and if so allows for the use of SVE in the future.
As the SVE registers are a superset of the VFP registers we don't need
to restore the VFP registers when SVE is enabled.
Ths interface to enable SVE is provided, but not used until SVE is
supported in signals and with ptrace.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43306
arm64 supports FIQs, fast interrupt requests, which are required by
Apple silicon which hardwires the ARM timers to FIQs. This is needed by
the upcoming Apple Interrupt Controller. Based on work by andrew@ and
kevans@ in https://reviews.freebsd.org/D40161.
Signed-off-by: Ayrton Munoz <a.munoz3327@gmail.com>
Co-authored-by: Kyle Evans <kevans@FreeBSD.org>
Co-authored-by: Andrew Turner <andrew@FreeBSD.org>
Reviewed-by: imp,mmel,mhorne
Pull-Request: https://github.com/freebsd/freebsd-src/pull/1363
Add a new macro that enables all CPTR_EL2 traps. This helps ensure we
trap all extensions we don't support.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46516
The TTA field moves depending on the HCR_EL2.E2H field. Add a macro to
hold the E2H == 1 case.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46515
Rename register fields that are only valid when HCR_EL2.E2H == 1. Some
fields move around depending on the value of the E2H field.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46514
When LPA2 is enabled the shareability attribute in the page table are
replaces with output address bits. To support a larger physical address
space make this attribute dynamic so we only set it when appropriate.
Reviewed by: alc, kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46394
ATTR_SH(ATTR_SH_IS) will soon be dynamic as the field is moved out of
the page tables in FEAT_LPA2. When this happens ATTR_DEFAULT will just
be ATTR_AF.
Rather than keeping ATTR_DEFAULT with one attribute remove it.
Reviewed by: alc, kib, markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46466
These are only used in one place so expand them there.
While here always set TCR_SH0_IS and TCR_SH1_IS. There is no advantage
to not set them in a UP kernel.
Reviewed by: alc, kib, markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46393
This will be used to support FEAT_LPA2 to allow more than 48 bits of
physical address space.
Reviewed by: alc, kib, markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46392
vmm.h is required for VM_MAX_SUFFIXLEN. vmm_snapshot.h is required for
struct vm_snapshot_meta.
This is a prerequisite for including vmm_dev.h in the headers parsed by
libsysdecode.
Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D46485
The maximum VM name length has changed since the arm64 vmm code was
forked. For now, just sync with the amd64 definitions, since they
permit longer VM names. As arm64/vmm is implemented only in main, I
won't bother dealing with backward compat.
Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D46484
This file contains the vmm device file implementation. Most of this
code is not machine-dependent and so shouldn't be duplicated this way.
Move most of it into a generic dev/vmm/vmm_dev.c. This will make it
easier to introduce a cdev-based interface for VM creation, which in
turn makes it possible to implement support for running bhyve as an
unprivileged user.
Machine-dependent ioctls continue to be handled in machine-dependent
code. To make the split a bit easier to handle, introduce a pair of
tables which define MI and MD ioctls. Each table entry can set flags
which determine which locks need to be held in order to execute the
handler. vmmdev_ioctl() now looks up the ioctl in one of the tables,
acquires locks and either handles the ioctl directly or calls
vmmdev_machdep_ioctl() to handle it.
No functional change intended. There is a lot of churn in this change
but the underlying logic in the ioctl handlers is the same. For now,
vmm_dev.h is still mostly separate, even though some parts could be
merged in principle. This would involve changing include paths for
userspace, though.
Reviewed by: corvink, jhb
Differential Revision: https://reviews.freebsd.org/D46431
This was added to not use the physical timer when E2H was set. As we
now use the correct timer in this case we can remove this extra check.
Tested by: kevans
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46086
Teach the virtual timer about the cnthctl_el2 field layout under VHE.
As with non-VHE we need to trap the physical timer and not trap the
virtual timer.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46074
When targeting Armv8.1 we can assume FEAT_LSE is available and can use
the atomic instructions this provides without needing to check for
support first.
Reviewed by: imp, markj, emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46159
As of 9e6544dd6e contigfree(9) is no longer
needed and should not be used anymore. We leave a wrapper for 3rd party
code in at least 15.x but remove (almost) all other cases from the tree.
This leaves one use of contigfree(9) untouched; that was the original
trigger for 9e6544dd6e and is handled in D45813 (to be committed
seperately later).
Sponsored by: The FreeBSD Foundation
Reviewed by: markj, kib
Tested by: pho (10h stress test run)
Differential Revision: https://reviews.freebsd.org/D46099
Before entering the kernel exception level ensure sctlr_el2 and
sctlr_el1 are in a known state. The EOS flag needs to be set to ensure
an eret instruction is a context synchronization event.
Set spcr_el1 when entering the kernel from EL1 and use an eret
instruction to return to the caller. This ensures the CPU pstate is
consistent with the value in spcr_el1 as it is the only way to set it
directly.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45528
When entering the kernel with the E2H field set the layout of the
cnthctl_el2 register changes. Use the correct field locations to enable
access to the counter and timer registers from EL1.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45529
When adding VHE support we may need to access these registers using
the _EL12 or _EL1 name depending on if VHE is enabled or not.
Add the registers we will likely need to access from VHE so these can
be updated in the guest switcher code.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45525
All architectures enable NEW_PCIB in DEFAULTS (arm being the most recent
to do so in 121be55599 (arm: Set NEW_PCIB in DEFAULTS rather than a
subset of kernel configs")), so it's time we removed the legacy code
that no longer sees much testing and has a significant maintenance
burden.
Reviewed by: jhb, andrew, emaste
Differential Revision: https://reviews.freebsd.org/D32954
