Since IOMMU map entries store a reference to the domain in which they
reside, there is no need to pass the domain to iommu_gas_free_entry(),
iommu_gas_free_space(), and iommu_gas_free_region().
Push down the acquisition and release of the IOMMU domain lock into
iommu_gas_free_space() and iommu_gas_free_region().
Both of these changes allow for simplifications in the callers of the
functions without really complicating the functions themselves.
Moreover, the latter change eliminates the direct use of the IOMMU
domain lock from the x86-specific DMAR code.
Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D35995
(cherry picked from commit 4670f90846)
Replace the DMAR unit's tlb_flush TAILQ by a custom list implementation
that enables dmar_qi_task() to dequeue entries without holding the DMAR
lock.
Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D35951
(cherry picked from commit 42736dc44d)
Merge lowermatch and uppermatch into find_space. Eliminate uppermatch
recursion. Merge match_insert into match_one and eliminate some
redundant calculation. Move some initialization out of find_space and
into map (and out from under a lock).
Reviewed by: kib (previous version), alc
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D35440
(cherry picked from commit e0e8d0c8d6)
If iommu_gas_match_one has to adjust for a boundary crossing, its
check against maxaddr includes 'offset' in its calculation, to ensure
that the allocated memory does not exceed the max address. However, if
there's no boundary crossing adjustment, then the maxaddr check
disregards 'offset'. Fix that.
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D34978
(cherry picked from commit 11fced21cc)
Eliminate a possible case of use-after-free in an error handling path
after a mapping failure. Specifically, eliminate IOMMU_MAP_ENTRY_QI_NF
and instead perform the IOTLB invalidation synchronously. Otherwise,
when iommu_domain_unload_entry() is called and told not to free the
IOMMU map entry, the caller could free the entry before dmar_qi_task()
is finished with it.
Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D35878
(cherry picked from commit 8bc3673847)
Eliminate the unroll_entry field from struct iommu_map_entry, shrinking
the struct by 16 bytes on 64-bit architectures.
Reviewed by: kib
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D35769
(cherry picked from commit db0110a536)
In the extremely unlikely case that the iommu_gas_map_region() call in
bus_dma_iommu_load_ident() failed, we would attempt to unmap the failed
entry twice, first in iommu_gas_map_region(), and a second time in the
caller. Once is enough, and twice is problematic because it leads to a
second RB_REMOVE call on the same tree node. Like it or not, RB_TREE
does not handle that possibility.
Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D35869
(cherry picked from commit dfabdacb27)
The loop iteration in iommu_gas_lowermatch checks the bound
a->common->lowaddr twice per loop iteration. Rewrite to test only once
per iteration. Do not worry about passing to iommu_gas_match_one a
range wholly beyond lowaddr. Since that function checks the upper end
of the range against lowaddr, it'll get rejected there.
Reviewed by: alc
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D35522
(cherry picked from commit 70b5d8fa0f)
Define simple functions for alignment and boundary checks and use them
everywhere instead of having slightly different implementations
scattered about. Define them in vm_extern.h and use them where
possible where vm_extern.h is included.
Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D33685
(cherry picked from commit c606ab59e7)
Introduce fine-grained locking on the dmamap's list of map entries,
replacing the use of the domain lock. This is not the most significant
source of lock contention, but it is the easiest to address.
Reviewed by: kib
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D35557
(cherry picked from commit eeb46578c2)
As of 19bb5a7244, the IOMMU_MF_CANSPLIT case in iommu_gas_match_one()
must take into account the specified offset. Otherwise, the recently
changed end calculation in iommu_gas_match_insert() could produce an
end address that crosses the specified boundary by one page.
Reviewed by: dougm
MFC with: 19bb5a7244
(cherry picked from commit 7b39a9bc1d)
Release the domain lock when iommu_gas_reserve_region_extend()'s call to
iommu_gas_reserve_region_locked() fails.
MFC after: 2 weeks
(cherry picked from commit 0ba1d86082)
Since OFF_TO_IDX() inherently truncates the given value, there is no
need to perform trunc_page() on it.
MFC after: 2 weeks
(cherry picked from commit 32e82bcc15)
As of 19bb5a7244, the size passed to iommu_gas_map is no longer
required to be a multiple of the CPU page size.
MFC after: 2 weeks
(cherry picked from commit 164491fb03)
If iommu_gas_find_space() ever called iommu_gas_uppermatch(), and it
succeeded in allocating space, then the subsequent KASSERT would be
triggered. Change that KASSERT to accept either success or ENOMEM.
MFC after: 1 week
(cherry picked from commit e6775534ae)
Change the recursive implementation to one that uses parent pointers
to walk back up the rb-tree, to slightly improve performance.
Reviewed by: alc, kib
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D35486
(cherry picked from commit f979ad0030)
Reverse the order of the search for a free space in lowermatch, to
make it a first-fit search. Iommu_gas_match_one always allocates from
the beginning of the free gap discovered from searching the tree, so
the current code isn't really allocating in a reverse first-fit
anyway, and making the search first-fit reduces the number of iommu
page table pages that are used.
Reported by: alc
Reviewed by: alc, kib
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D35458
(cherry picked from commit 975715b788)
Let the caller to iommu_map pass the size parameter without rounding
it up to a multiple of page size. Let iommu_map round it up when
necessary, which is not all of the time, so that in some cases less
space is reserved.
Reviewed by: alc, kib (previous version)
Tested by: pho, br
Discussed with: andrew
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D35424
(cherry picked from commit 19bb5a7244)
iommu_gas_lowermatch looks right, then left, then right again in its
search for free space. Change to a more straightforward last-fit
search that touches fewer tree nodes and improves performance.
Reported by: wxzhu@rice.edu
Reviewed by: alc, kib
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D35439
(cherry picked from commit 3003117253)
In iommu_gas_lowermatch and iommu_gas_uppermatch, a subtree search is
quickly terminated if the largest available free space in the subtree
is below a limit, where that limit is related to the size of the
allocation request. However, that limit is too small; it does not
account for both of the guard pages that will surround the allocated
space, but only for one of them. Consequently, it permits the search
to proceed through nodes that cannot produce a successful allocation
for all the requested space. Fix that limit to improve search
performance.
Reviewed by: alc, kib
Submitted by: Weixi Zhu (wxzhu@rice.edu)
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D35414
(cherry picked from commit b831865fe3)
iommu_bus_dmamap_load_something1 includes code for handling the
possibility of splitting a buffer that is needlessly complex.
Simplify it.
Reviewed by: alc, kib
MFC after: 3 weeks
Tested by: pho (previous revisions)
Differential Revision: https://reviews.freebsd.org/D35232
(cherry picked from commit 04e86ae357)
These ones were unambiguous cases where the Foundation was the only
listed copyright holder (in the associated license block).
Sponsored by: The FreeBSD Foundation
(cherry picked from commit 9feff969a0)
PCI memory address space is shared between memory-mapped devices (MMIO)
and host memory (which may be remapped by an IOMMU). Device accesses to
an address within a memory aperture in a PCIe root port will be treated
as peer-to-peer and not forwarded to an IOMMU. To avoid this, reserve
the address space of the root port's memory apertures in the address
space used by the IOMMU for remapping.
Reviewed by: kib, tychon
Discussed with: Anton Rang <rang@acm.org>
Tested by: tychon
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D27503
The entry->flags field is initialized in iommu_gas_init_domain().
Reviewed by: kib
Sponsored by: Innovate DSbD
Differential Revision: https://reviews.freebsd.org/D27235
This is needed on arm64 for the interface between iommu framework
and iommu controller drivers.
Reviewed by: kib
Sponsored by: Innovate DSbD
Differential Revision: https://reviews.freebsd.org/D27229
o Add iommu_unmap_msi() to release the msi GAS entry.
o Provide default implementations for iommu init/deinit methods.
Reviewed by: kib
Sponsored by: Innovate DSbD
Differential Revision: https://reviews.freebsd.org/D26906
This allows the interrupt controller driver only need a small change to
create a map for the page the device will write to raise an interrupt.
Submitted by: andrew
Reviewed by: kib
Sponsored by: Innovate DSbD
Differential Revision: https://reviews.freebsd.org/D26705
This function isn't ACPI dependent and we may use it on FDT systems
as well.
o Don't repeat the function declaration, include iommu.h instead.
Reviewed by: andrew, kib
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D26584
It could be used in various IOMMU platforms, not only DMAR.
Reviewed by: kib
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D26373
so we don't ifdef for every arch in busdma_iommu.c;
o No need to include specialreg.h for x86, remove it.
Requested by: andrew
Reviewed by: kib
Sponsored by: DARPA/AFRL
Differential Revision: https://reviews.freebsd.org/D25957
These functions were introduced before UMA started ensuring that freed
memory gets placed in domain-local caches. They no longer serve any
purpose since UMA now provides their functionality by default. Remove
them to simplyify the kernel memory allocator interfaces a bit.
Reviewed by: cem, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D25937
so x86 can support Intel DMAR and AMD IOMMU simultaneously.
Reviewed by: kib
Sponsored by: DARPA/AFRL
Differential Revision: https://reviews.freebsd.org/D25894
