If a counter more than overflows just as we read it on switch out then,
if using sampling mode, we will negate this small value to give a huge
reload count, and if we later switch back in that context we will
validate that value against pm_reloadcount and panic an INVARIANTS
kernel with:
panic: [pmc,1470] pmcval outside of expected range cpu=2 ri=16 pmcval=fffff292 pm_reloadcount=10000
or similar. Presumably in a non-INVARIANTS kernel we will instead just
use the provided value as the reload count, which would lead to the
overflow not happing for a very long time (e.g. 78 minutes for a 48-bit
counter incrementing at an averate rate of 1GHz).
Instead, clamp the reload count to 0 (which corresponds precisely to the
value we would compute if it had just overflowed and no more), which
will result in hwpmc using the full original reload count again. This is
the approach used by core for Intel (for both fixed and programmable
counters).
As part of this, armv7 and arm64 are made conceptually simpler; rather
than skipping modifying the overflow count for sampling mode counters so
it's always kept as ~0, those special cases are removed so it's always
applicable and the concatentation of it and the hardware counter can
always be viewed as a 64-bit counter, which also makes them look more
like other architectures.
Whilst here, fix an instance of UB (shifting a 1 into the sign bit) for
amd in its sign-extension code.
Reviewed by: andrew, mhorne, kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D33654
(cherry picked from commit e74c7ffcb1)
Performance counters and overflow interrupts are assumed to be disabled
by default, but this is not guaranteed. Ensure we disable both during
per-cpu initialization, before enabling the PMU. Otherwise, some systems
(such as the Ampere eMAG) would experience an interrupt storm upon
loading the hwpmc module.
Reviewed by: br
MFC after: 5 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32854
(cherry picked from commit b826cc3caf)
As in commit 831850d8b0, this routine can trigger false positives, so
exclude it from instrumentation.
Reported by: pho
Sponsored by: The FreeBSD Foundation
(cherry picked from commit 5d243d41b1)
Looking for "tsc-tsc" in the pmu tables will fail every time. Instead,
make this an alias for the static TSC event defined in pmc_events.h.
This fixes 'pmcstat -s cycles' on Intel and AMD.
Reviewed by: emaste
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32197
(cherry picked from commit 937539e0a3)
It was written for Nehalem and Westmere, with minor but incomplete
updates for Sandy Bridge in 78d763a29b. The uncore architecture
changed significantly with this generation, bringing new layouts and
locations for some MSRs.
Misprogramming these MSRs in ucp_start_pmc() may panic the system, and
this is trivially reproducible via pmcstat(8) on at least Broadwell and
Haswell. Disable the class on these CPUs until it can be updated more
completely and leave a TODO comment detailing some of the work required.
Note that the nclasses value for Broadwell was already incorrect and
doesn't need changing.
The result is that any uncore events listed by pmcstat -L will no longer
be allocatable, but this is already the case for newer generations of
Intel CPUs.
PR: 253687
Reported by: Zhenlei Huang <zlei.huang@gmail.com>
Reviewed by: kib
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31389
(cherry picked from commit 4f35e8cba2)
8cc3815f:
hwpmc_arm64: accept raw event codes for PMC_OP_PMCALLOCATE
Make it possible to specify event codes without an offset of
PMC_EV_ARMV8_FIRST, by setting a machine-dependent flag. This is
required to make use of event definitions from pmu-events.
Reviewed by: ray (slightly earlier version)
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3060228dd6730:
libpmc: enable pmu_utils on arm64
This allows supported libpmc to query/select from the pmu-events table,
which may have a more complete set of events than what we define
manually. A future update to these definitions should greatly improve
this support. The alias table is empty for now, until this future import
is complete.
Add the Foundation's copyright for recent work on this file.
Reviewed by: ray (slightly earlier version)
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3060327ea55fc:
libpmc/hwpmc: fix issues with arm64 pmu-events support
Due to a mis-merge, the changes committed to libpmc never called
pmu_parse_event(), or set pm->pm_ev. However, this field shouldn't be
used to carry the actual pmc event code anyway, as it is expected to
contain the index into the pmu event array (otherwise, it breaks event
name lookup in pmclog_get_event()). Add a new MD field,
pm_md.pm_md_config, to pass the raw event code to arm64_allocate_pmc().
Additionally, the change made to pmc_md_op_pmcallocate was incorrect, as
this is a union, not a struct. Restore the proper padding size.
Reviewed by: luporl, ray, andrew
Fixes: 28dd6730a5 ("libpmc: enable pmu_utils on arm64")
Fixes: 8cc3815f02 ("hwpmc_arm64: accept raw event codes...")
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31221
(cherry picked from commit 8cc3815f02)
(cherry picked from commit 28dd6730a5)
(cherry picked from commit 27ea55fc65)
This will be used to detect supported pmu events. The expected format is
the MIDR register with the revision and variant fields masked. See also:
lib/libpmc/pmu-events/arch/arm64/mapfile.csv.
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D30601
(cherry picked from commit 5867cccdc4)
The lower pmc event bits were masked off to find the PMC event ID.
The doesn't work when there are more events. Switch it to use the
offser relative to the first event while also checking the ID is
in the expected range.
Reviewed by: gnn, ray
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D29600
(cherry picked from commit 24b2f4ea49)
Virtual PMCs could be running on multiple CPUs so this needs to be
a per-CPU value.
Submitted by: rwatson (earlier version)
Reviewed by: gnn
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D27973
When testing hwpmc on arm64 we found the counter could overflow while
reading the event count. Handle this case in the armv7 code by also
checking if the overflow bit is set and incrementing the overflow
cound as needed.
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D27969
We increment the overflow count when receiving an overflow interrupt
with special care to check if it happens while reading the event counter.
Sponsored by: Innovate UK
This change adds support for POWER8 and POWER9 PMCs (bare metal and
pseries).
All PowerISA 2.07B non-random events are supported.
Implementation was based on that of PPC970.
Reviewed by: jhibbits
Sponsored by: Eldorado Research Institute (eldorado.org.br)
Differential Revision: https://reviews.freebsd.org/D26110
And add a _74XX suffix to 74XX SPRs.
This is a preparation for adding support to POWER8/9 PMCs, which have most
SPRs equal to 970 ones.
Reviewed by: jhibbits
Sponsored by: Eldorado Research Institute (eldorado.org.br)
Differential Revision: https://reviews.freebsd.org/D26532
This adds support for the Cortex-A76 and Neoverse-N1 PMU counters to pmc.
While here add more PMCR_IDCODE values and check the implementers code is
correct before setting the PMU type.
Reviewed by: bz, emaste (looks reasonable to me)
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D25959
These functions were introduced before UMA started ensuring that freed
memory gets placed in domain-local caches. They no longer serve any
purpose since UMA now provides their functionality by default. Remove
them to simplyify the kernel memory allocator interfaces a bit.
Reviewed by: cem, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D25937
Use the existing PMC_CPUID_LEN to size pmc_cpuid in the kernel and various
buffers for reading it in libpmc. This avoids some extra syscalls and
malloc/frees.
While in here, use strlcpy to copy a user-provided cpuid string instead of
memcpy, to make sure we terminate the buffer.
Reviewed by: mav
MFC after: 1 week
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D25679
pmc_cpuid was uninitialized for most AMD processor families. We can still
populate this string for unimplemented families.
Also added a CPUID_TO_STEPPING macro and converted existing code to use it.
Reviewed by: mav
MFC after: 2 weeks
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D25673
It follows the equivalent Linux change to be able to differentiate
skylakex and cascadelakex, sharing the same model but not stepping.
This fixes skylakex handling broken by r363144.
MFC after: 6 days
Recognize new micro-architecture in hwpmc_intel driver. Based on Intel
document 325462-071US. Tested with tools/test/hwpmc/pmctest.py
on Atom E3930 SoC.
Submitted by: Dawid Gorecki <dgr@semihalf.com>
Reviewed by: kib
Obtained from: Semihalf
Sponsored by: Stormshield
Differential Revision: https://reviews.freebsd.org/D24310
r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are
still not MPSAFE (or already are but aren’t properly marked).
Use it in preparation for a general review of all nodes.
This is non-functional change that adds annotations to SYSCTL_NODE and
SYSCTL_PROC nodes using one of the soon-to-be-required flags.
Mark all obvious cases as MPSAFE. All entries that haven't been marked
as MPSAFE before are by default marked as NEEDGIANT
Approved by: kib (mentor, blanket)
Commented by: kib, gallatin, melifaro
Differential Revision: https://reviews.freebsd.org/D23718
To make the PMC tool pmcstat working properly on Hygon platform, add
support for Hygon Dhyana family 18h by using the PMC initialization
code path of AMD family 17h.
Submitted by: Pu Wen <puwen@hygon.cn>
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D23562
Use the KPI to tweak MSRs in mitigation code.
Reviewed by: markj, scottl
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D22431
- amd_intr() does not account for the offset (0x200) in the counter
MSR address and ends up accessing invalid regions while reading
counter value after the 4th counter (0xC001000[8,9,..]) and
erroneously updates the counter values for counters [1-4].
- amd_intr() should only check core pmcs for interrupts since
other types of pmcs (L3,DF) cannot generate interrupts.
- fix pmc NMI's being ignored due to NMI latency on newer AMD processors
Note that this fixes a kernel panic due to GPFs accessing MSRs on
higher core count AMD cpus (seen on both Rome 7502P, and
Threadripper 2990WX 32-core CPUs)
Discussed with: markj
Submitted by: Shreyank Amartya
Differential Revision: https://reviews.freebsd.org/D21553
In case the implementation ever changes from using a chain of next pointers,
then changing the macro definition will be necessary, but changing all the
files that iterate over vm_map entries will not.
Drop a counter in vm_object.c that would have an effect only if the
vm_map entry count was wrong.
Discussed with: alc
Reviewed by: markj
Tested by: pho (earlier version)
Differential Revision: https://reviews.freebsd.org/D21882
BERI stands for Bluespec Extensible RISC Implementation, based on MIPS.
BERI has not implemented standard MIPS perfomance monitoring counters,
instead it provides statistical counters.
BERI statcounters have a several limitations:
- They can't be written
- They don't support start/stop operation
- None of hardware interrupt is provided on a counter overflow.
So make it separate to hwpmc_mips module and support process/system
counting mode only.
Sponsored by: DARPA, AFRL
The forthcoming microcode update will fix a TSX bug by clobbering PMC3
when TSX instructions are executed (even speculatively). There is an
alternate mode where CPU executes all TSX instructions by aborting
them, in which case PMC3 is still available to OS. Any code that
correctly uses TSX must be ready to handle abort anyway.
Since it is believed that FreeBSD population of hwpmc(4) users is
significantly larger than the population of TSX users, switch the
microcode into TSX abort mode whenever a pmc is allocated, and back to
bug avoidance mode when the last pmc is deallocated.
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
The hwpmc pcpu sample buffer is prone to head of line blocking
when waiting for user process to return to user space and
collect a pending callchain. If more than one tick has elapsed
between the time the sample entry was marked for collection and
the time that the hardclock pmc handler runs to copy the records
to a larger temporary buffer, mark the sample entry as not in
use.
This changes reduces the number of samples marked as not valid
when collecting under load from ~99.5% to 5-20%.
Reported by: mjg@
MFC after: 3 days
Remove malloc_domain(9) and most other _domain KPIs added in r327900.
The new functions allow the caller to specify a general NUMA domain
selection policy, rather than specifically requesting an allocation from
a specific domain. The latter policy tends to interact poorly with
M_WAITOK, resulting in situations where a caller is blocked indefinitely
because the specified domain is depleted. Most existing consumers of
the _domain KPIs are converted to instead use a DOMAINSET_PREF() policy,
in which we fall back to other domains to satisfy the allocation
request.
This change also defines a set of DOMAINSET_FIXED() policies, which
only permit allocations from the specified domain.
Discussed with: gallatin, jeff
Reported and tested by: pho (previous version)
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D17418
