This routine returns a monotonic count of the number of nanoseconds elapsed
since the previous call. On arm64 it uses the generic system timer. The
implementation multiplies the counter value by 10**9 then divides by the counter
frequency, but this multiplication can overflow. This can result in trace
records with non-monotonic timestamps, which breaks libdtrace's temporal
ordering algorithm.
An easy fix is to reverse the order of operations, since the counter frequency
will in general be smaller than 10**9. (In fact, it's mandated to be 1Ghz in
ARMv9, which makes life simple.) However, this can give a fair bit of error.
Adopt the calculation used on amd64, with tweaks to handle frequencies as low as
1MHz: the ARM generic timer documentation suggests that ARMv8 timers are
typically in the 1MHz-50MHz range, which is true on arm64 systems that I have
access to.
MFC after: 2 weeks
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D49244
(cherry picked from commit 36ae5ce2f2)
On arm64, the FBT provider treats tail calls as return probes. Ignoring
the question of whether this is really correct, the implementation is
wrong: instr is a pointer to uint32_t, so the removed multiplication by
the instruction size is wrong. As a result, FBT would create return
probes for intra-function branches.
MFC after: 2 weeks
Sponsored by: Innovate UK
(cherry picked from commit 4da070ce6c)
Remove some unused variables and reduce the scope of some others.
No functional change intended.
MFC after: 1 week
Sponsored by: Innovate UK
(cherry picked from commit 8384a19adc)
- Don't allow FBT and kinst to instrument the KMSAN runtime.
- When fetching data from the traced thread's stack, mark it as
initialized. It may well be uninitialized, but as dtrace permits
arbitrary inspection of kernel memory, it isn't very useful to raise
KMSAN reports.
- Mark data copied in from userspace as initialized, as we do for
copyin() etc. using interceptors.
MFC after: 2 weeks
(cherry picked from commit fdeb273d49)
We were previously allocating MAXCPU structures for several purposes,
but this is generally unnecessary and is quite excessive, especially
after MAXCPU was bumped to 1024 on amd64 and arm64. We already are
careful to allocate only as many per-CPU tracing buffers as are needed;
extend this to other allocations.
For example, in a 2-vCPU VM, the size of a consumer state structure
drops from 64KB to 128B. The size of the per-consumer `dts_buffer` and
`dts_aggbuffer` arrays shrink similarly. Ditto for pre-allocations of
local and global D variable storage space.
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D47667
(cherry picked from commit 5d12db2daf)
- Use a fresh context when entering dtrace_invop() via a breakpoint
exception.
- Mark the #BP trapframe as initialized.
MFC after: 2 weeks
(cherry picked from commit cc3da1955c)
It is not needed after commit 7e80fd5ef397. No functional change
intended.
Reviewed by: avg
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D46675
(cherry picked from commit 06e7bc83f6)
FBT refuses to create probes in modules which depend on dtrace(all), but
dtrace_test is a convenient place to add functions specifically for
testing dtrace.
The dependency on dtraceall is not needed, so just remove it. In fact,
it can be useful to test SDT probe creation by loading dtrace_test with
and without dtraceall loaded.
Reviewed by: avg
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D46673
(cherry picked from commit 5bd7b976c1)
dtrace_getarg() previously walked the call stack looking for a frame
matching the dtrace_invop_callsite symbol, in order to look for a
trapframe corresponding to an invop (i.e., FBT or kinst) probe. Commit
3ba8e9dc4a broke this in some cases by breaking the expected alignment
of the dtrace_invop_callsite symbol.
Rather than groveling around the stack to find invop probe arguments,
simply use the trapframe reference saved by dtrace_invop(). This is
simpler and less fragile.
Reported by: avg
Reviewed by: avg
MFC after: 2 weeks
Fixes: 3ba8e9dc4a ("dtrace/amd64: Implement emulation of call instructions")
Differential Revision: https://reviews.freebsd.org/D46672
(cherry picked from commit 9aabab09c4)
This was done in the original DTrace import, presumably because that
made it a bit easier to handle includes. However, this can cause
dtrace_getpcstack() to be inlined into dtrace_probe(), resulting in a
missing frame in stack traces since dtrace_getpcstack() takes care to
bump "aframes" to account for its own stack frame.
To avoid this, compile dtrace_isa.c separately on all platforms. Add
requisite includes.
MFC after: 2 weeks
Sponsored by: Innovate UK
(cherry picked from commit 82283cad12)
For invop providers (i.e., fbt and kinst) we can simply reach into the
invop trapframe to fetch argument registers for arguments 0-7; for
argument 8 and beyond we have to read the value off of the stack.
Reviewed by: Domagoj Stolfa, avg
MFC after: 2 weeks
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D45649
(cherry picked from commit bae00433f0)
profile provider probes fire in the context of a timer interrupt. Thus,
the "regs" action can make use of the interrupt trap frame to get
register values when the interrupt happened in kernel mode. Make that
trap frame available when possible so that "regs" works more or less as
it already does with the fbt and kinst providers.
MFC after: 1 week
(cherry picked from commit 7d35b38972)
fbt computes incorrect instruction position for AArch64 kernel module symbol.
The issue is with the for loop, it does an extra increment of instr pointer
after the required instruction is found. Hence, a wrong instruction is
targeted for patching.
Signed-off-by: Igor Ostapenko <pm@igoro.pro>
Fixes: 980746e5cb ("fbt: simplify arm64 function-prologue parsing")
Reviewed by: markj
Pull Request: https://github.com/freebsd/freebsd-src/pull/855
MFC after: 1 week
(cherry picked from commit b4db386f9f)
This reverts commit 22508c8b6c.
The t_kinst_curprobe field is no longer needed by kinst.
Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D41031
Using per-CPU and per-thread trampolines is expensive and error-prone,
since we're rewriting the same memory blocks constantly. Per-probe
trampolines solve this problem by giving each probe its own block of
executable memory, which more or less remains the same after the initial
write.
What this patch does, is get rid of the initialization code which
allocates a trampoline for each thread, and instead let each port of
kinst allocate a trampoline for each new probe created. It also sets up
the infrastructure needed to support the new trampoline scheme.
This change is not currently supported on amd64, as the amd64 port needs
further changes to work, so this is a temporary/gradual patch to fix the
riscv and arm64 ports.
Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40962
Currently kinst checks if only the first instruction is 'push %rbp',
essentially excluding functions that do push RBP, but not in the first
instruction. This patch modifies kinst to check for 'push %rbp', as
well, as a following 'pop %rbp', anywhere in the function. This behavior
also matches that of FBT.
Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40283
Needed by the forthcoming RISC-V and ARM64 ports.
Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40872
Use unprivileged loads to access user memory. Without this, the
accesses trap and various dtrace actions such as ustack() fail.
Reviewed by: andrew
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40540
It was used in one place and was added specifically to support dtrace
stack unwinding code. Write an equivalent expression using struct
unwind_state instead. No functional change intended.
Reviewed by: andrew
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40538
The forthcoming RISC-V and ARM64 ports of kinst introduce a new field
named "t_kinst_curprobe", so "t_kinst" (which points to a trampoline)
becomes a misleading name.
No functional change intended.
Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40507
Centralize KINST_TRAMP_FILL_PATTERN and KINST_TRAMP_FILL_SIZE to reduce
redefinitions, and use the architecture-dependent kinst_patchval_t as
their size.
Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40406
match_opcode() is defined in FBT, kinst, and dtrace_subr.c. The function
prologue-checking functions are defined in FBT and kinst.
Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40335
Tracing memcpy() would crash the kernel, because we'd also trace the
memcpy() calls from kinst_invop(). To fix this, introduce kinst_memcpy()
whose arguments are 'volatile', so that we avoid having the compiler
replace it with a regular memcpy().
Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40284
Exclude functions that are not safe-to-trace.
Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
ifferential Revision: https://reviews.freebsd.org/D39229
The current implementation and comment was specific to amd64. Even
though in the case of kinst's supported architectures (RISC-V and ARM64)
VM_MIN_KERNEL_ADDRESS is equal to KERNBASE, it's better to be explicit.
Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40266
