2005-01-05 15:17:21 -05:00
|
|
|
/*-
|
1997-04-26 07:46:25 -04:00
|
|
|
* ----------------------------------------------------------------------------
|
|
|
|
|
* "THE BEER-WARE LICENSE" (Revision 42):
|
|
|
|
|
* <phk@FreeBSD.org> wrote this file. As long as you retain this notice you
|
|
|
|
|
* can do whatever you want with this stuff. If we meet some day, and you think
|
|
|
|
|
* this stuff is worth it, you can buy me a beer in return. Poul-Henning Kamp
|
|
|
|
|
* ----------------------------------------------------------------------------
|
|
|
|
|
*
|
1999-08-27 21:08:13 -04:00
|
|
|
* $FreeBSD$
|
1997-04-26 07:46:25 -04:00
|
|
|
*
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#ifndef _MACHINE_SMP_H_
|
|
|
|
|
#define _MACHINE_SMP_H_
|
|
|
|
|
|
2003-11-17 03:58:16 -05:00
|
|
|
#ifdef _KERNEL
|
|
|
|
|
|
|
|
|
|
#ifdef SMP
|
|
|
|
|
|
|
|
|
|
#ifndef LOCORE
|
|
|
|
|
|
2015-12-07 12:41:20 -05:00
|
|
|
#include <x86/x86_smp.h>
|
2003-11-17 03:58:16 -05:00
|
|
|
|
2015-12-07 12:41:20 -05:00
|
|
|
extern int pmap_pcid_enabled;
|
|
|
|
|
extern int invpcid_works;
|
For amd64 non-PCID machines, and for i386 machines with support for
the PG_G global pte flag, pmap_invalidate_all() fails to flush global
TLB entries [*]. This is because TLB shootdown handler for such
configs reloads CR3, and on i386 pmap_invalidate_all() does the same
for the initiating CPU. Note that current code does not issue total
invalidation requests for the kernel_pmap.
Rename amd64 function invltlb_globpcid() to invltlb_glob(), it is not
specific for PCID for quite some time, and implement the same
functionality for i386. Use the function instead of invltlb() in
shootdown handlers and in i386 pmap_invalidate_all(), but only for the
kernel pmap (which maps pages with the PG_G attribute set), which
takes care of PG_G TLB entries on flush.
To detect the affected pmap in i386 TLB shootdown handler, pmap should
be passed to the smp_masked_invltlb() function, which makes amd64 and
i386 TLB shootdown code almost identical. Merge the code under x86/.
Noted by: jhb [*]
Reviewed by: cem, jhb, pho
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D4346
2015-12-03 06:14:14 -05:00
|
|
|
|
2003-11-17 03:58:16 -05:00
|
|
|
/* global symbols in mpboot.S */
|
|
|
|
|
extern char mptramp_start[];
|
|
|
|
|
extern char mptramp_end[];
|
|
|
|
|
extern u_int32_t mptramp_pagetables;
|
|
|
|
|
|
|
|
|
|
/* IPI handlers */
|
|
|
|
|
inthand_t
|
Rewrite amd64 PCID implementation to follow an algorithm described in
the Vahalia' "Unix Internals" section 15.12 "Other TLB Consistency
Algorithms". The same algorithm is already utilized by the MIPS pmap
to handle ASIDs.
The PCID for the address space is now allocated per-cpu during context
switch to the thread using pmap, when no PCID on the cpu was ever
allocated, or the current PCID is invalidated. If the PCID is reused,
bit 63 of %cr3 can be set to avoid TLB flush.
Each cpu has PCID' algorithm generation count, which is saved in the
pmap pcpu block when pcpu PCID is allocated. On invalidation, the
pmap generation count is zeroed, which signals the context switch code
that already allocated PCID is no longer valid. The implication is
the TLB shootdown for the given cpu/address space, due to the
allocation of new PCID.
The pm_save mask is no longer has to be tracked, which (significantly)
reduces the targets of the TLB shootdown IPIs. Previously, pm_save
was reset only on pmap_invalidate_all(), which made it accumulate the
cpuids of all processors on which the thread was scheduled between
full TLB shootdowns.
Besides reducing the amount of TLB shootdowns and removing atomics to
update pm_saves in the context switch code, the algorithm is much
simpler than the maintanence of pm_save and selection of the right
address space in the shootdown IPI handler.
Reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 3 weeks
2015-05-09 15:11:01 -04:00
|
|
|
IDTVEC(invltlb_pcid), /* TLB shootdowns - global, pcid */
|
|
|
|
|
IDTVEC(invltlb_invpcid),/* TLB shootdowns - global, invpcid */
|
2015-12-07 12:41:20 -05:00
|
|
|
IDTVEC(justreturn); /* interrupt CPU with minimum overhead */
|
2003-11-17 03:58:16 -05:00
|
|
|
|
2014-03-11 06:03:29 -04:00
|
|
|
void invltlb_pcid_handler(void);
|
Rewrite amd64 PCID implementation to follow an algorithm described in
the Vahalia' "Unix Internals" section 15.12 "Other TLB Consistency
Algorithms". The same algorithm is already utilized by the MIPS pmap
to handle ASIDs.
The PCID for the address space is now allocated per-cpu during context
switch to the thread using pmap, when no PCID on the cpu was ever
allocated, or the current PCID is invalidated. If the PCID is reused,
bit 63 of %cr3 can be set to avoid TLB flush.
Each cpu has PCID' algorithm generation count, which is saved in the
pmap pcpu block when pcpu PCID is allocated. On invalidation, the
pmap generation count is zeroed, which signals the context switch code
that already allocated PCID is no longer valid. The implication is
the TLB shootdown for the given cpu/address space, due to the
allocation of new PCID.
The pm_save mask is no longer has to be tracked, which (significantly)
reduces the targets of the TLB shootdown IPIs. Previously, pm_save
was reset only on pmap_invalidate_all(), which made it accumulate the
cpuids of all processors on which the thread was scheduled between
full TLB shootdowns.
Besides reducing the amount of TLB shootdowns and removing atomics to
update pm_saves in the context switch code, the algorithm is much
simpler than the maintanence of pm_save and selection of the right
address space in the shootdown IPI handler.
Reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 3 weeks
2015-05-09 15:11:01 -04:00
|
|
|
void invltlb_invpcid_handler(void);
|
2014-03-11 06:27:57 -04:00
|
|
|
int native_start_all_aps(void);
|
2003-11-17 03:58:16 -05:00
|
|
|
|
|
|
|
|
#endif /* !LOCORE */
|
|
|
|
|
#endif /* SMP */
|
|
|
|
|
|
|
|
|
|
#endif /* _KERNEL */
|
1997-04-26 07:46:25 -04:00
|
|
|
#endif /* _MACHINE_SMP_H_ */
|
