Merge tag 'x86-fsgsbase-2020-08-04' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip (4da9f330) · Commits · 戴 / test

Documentation/admin-guide/kernel-parameters.txt

+2 −0

Original line number	Diff line number	Diff line
		@@ -3084,6 +3084,8 @@
		no5lvl [X86-64] Disable 5-level paging mode. Forces
		kernel to use 4-level paging instead.

		nofsgsbase [X86] Disables FSGSBASE instructions.

		no_console_suspend
		[HW] Never suspend the console
		Disable suspending of consoles during suspend and

Documentation/x86/x86_64/fsgs.rst

0 → 100644

+199 −0

Original line number	Diff line number	Diff line
		.. SPDX-License-Identifier: GPL-2.0

		Using FS and GS segments in user space applications
		===================================================

		The x86 architecture supports segmentation. Instructions which access
		memory can use segment register based addressing mode. The following
		notation is used to address a byte within a segment:

		Segment-register:Byte-address

		The segment base address is added to the Byte-address to compute the
		resulting virtual address which is accessed. This allows to access multiple
		instances of data with the identical Byte-address, i.e. the same code. The
		selection of a particular instance is purely based on the base-address in
		the segment register.

		In 32-bit mode the CPU provides 6 segments, which also support segment
		limits. The limits can be used to enforce address space protections.

		In 64-bit mode the CS/SS/DS/ES segments are ignored and the base address is
		always 0 to provide a full 64bit address space. The FS and GS segments are
		still functional in 64-bit mode.

		Common FS and GS usage
		------------------------------

		The FS segment is commonly used to address Thread Local Storage (TLS). FS
		is usually managed by runtime code or a threading library. Variables
		declared with the '__thread' storage class specifier are instantiated per
		thread and the compiler emits the FS: address prefix for accesses to these
		variables. Each thread has its own FS base address so common code can be
		used without complex address offset calculations to access the per thread
		instances. Applications should not use FS for other purposes when they use
		runtimes or threading libraries which manage the per thread FS.

		The GS segment has no common use and can be used freely by
		applications. GCC and Clang support GS based addressing via address space
		identifiers.

		Reading and writing the FS/GS base address
		------------------------------------------

		There exist two mechanisms to read and write the FS/GS base address:

		- the arch_prctl() system call

		- the FSGSBASE instruction family

		Accessing FS/GS base with arch_prctl()
		--------------------------------------

		The arch_prctl(2) based mechanism is available on all 64-bit CPUs and all
		kernel versions.

		Reading the base:

		arch_prctl(ARCH_GET_FS, &fsbase);
		arch_prctl(ARCH_GET_GS, &gsbase);

		Writing the base:

		arch_prctl(ARCH_SET_FS, fsbase);
		arch_prctl(ARCH_SET_GS, gsbase);

		The ARCH_SET_GS prctl may be disabled depending on kernel configuration
		and security settings.

		Accessing FS/GS base with the FSGSBASE instructions
		---------------------------------------------------

		With the Ivy Bridge CPU generation Intel introduced a new set of
		instructions to access the FS and GS base registers directly from user
		space. These instructions are also supported on AMD Family 17H CPUs. The
		following instructions are available:

		=============== ===========================
		RDFSBASE %reg Read the FS base register
		RDGSBASE %reg Read the GS base register
		WRFSBASE %reg Write the FS base register
		WRGSBASE %reg Write the GS base register
		=============== ===========================

		The instructions avoid the overhead of the arch_prctl() syscall and allow
		more flexible usage of the FS/GS addressing modes in user space
		applications. This does not prevent conflicts between threading libraries
		and runtimes which utilize FS and applications which want to use it for
		their own purpose.

		FSGSBASE instructions enablement
		^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
		The instructions are enumerated in CPUID leaf 7, bit 0 of EBX. If
		available /proc/cpuinfo shows 'fsgsbase' in the flag entry of the CPUs.

		The availability of the instructions does not enable them
		automatically. The kernel has to enable them explicitly in CR4. The
		reason for this is that older kernels make assumptions about the values in
		the GS register and enforce them when GS base is set via
		arch_prctl(). Allowing user space to write arbitrary values to GS base
		would violate these assumptions and cause malfunction.

		On kernels which do not enable FSGSBASE the execution of the FSGSBASE
		instructions will fault with a #UD exception.

		The kernel provides reliable information about the enabled state in the
		ELF AUX vector. If the HWCAP2_FSGSBASE bit is set in the AUX vector, the
		kernel has FSGSBASE instructions enabled and applications can use them.
		The following code example shows how this detection works::

		#include <sys/auxv.h>
		#include <elf.h>

		/* Will be eventually in asm/hwcap.h */
		#ifndef HWCAP2_FSGSBASE
		#define HWCAP2_FSGSBASE (1 << 1)
		#endif

		....

		unsigned val = getauxval(AT_HWCAP2);

		if (val & HWCAP2_FSGSBASE)
		printf("FSGSBASE enabled\n");

		FSGSBASE instructions compiler support
		^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

		GCC version 4.6.4 and newer provide instrinsics for the FSGSBASE
		instructions. Clang 5 supports them as well.

		=================== ===========================
		_readfsbase_u64() Read the FS base register
		_readfsbase_u64() Read the GS base register
		_writefsbase_u64() Write the FS base register
		_writegsbase_u64() Write the GS base register
		=================== ===========================

		To utilize these instrinsics <immintrin.h> must be included in the source
		code and the compiler option -mfsgsbase has to be added.

		Compiler support for FS/GS based addressing
		-------------------------------------------

		GCC version 6 and newer provide support for FS/GS based addressing via
		Named Address Spaces. GCC implements the following address space
		identifiers for x86:

		========= ====================================
		__seg_fs Variable is addressed relative to FS
		__seg_gs Variable is addressed relative to GS
		========= ====================================

		The preprocessor symbols __SEG_FS and __SEG_GS are defined when these
		address spaces are supported. Code which implements fallback modes should
		check whether these symbols are defined. Usage example::

		#ifdef __SEG_GS

		long data0 = 0;
		long data1 = 1;

		long __seg_gs *ptr;

		/* Check whether FSGSBASE is enabled by the kernel (HWCAP2_FSGSBASE) */
		....

		/* Set GS base to point to data0 */
		_writegsbase_u64(&data0);

		/* Access offset 0 of GS */
		ptr = 0;
		printf("data0 = %ld\n", *ptr);

		/* Set GS base to point to data1 */
		_writegsbase_u64(&data1);
		/* ptr still addresses offset 0! */
		printf("data1 = %ld\n", *ptr);


		Clang does not provide the GCC address space identifiers, but it provides
		address spaces via an attribute based mechanism in Clang 2.6 and newer
		versions:

		==================================== =====================================
		__attribute__((address_space(256)) Variable is addressed relative to GS
		__attribute__((address_space(257)) Variable is addressed relative to FS
		==================================== =====================================

		FS/GS based addressing with inline assembly
		-------------------------------------------

		In case the compiler does not support address spaces, inline assembly can
		be used for FS/GS based addressing mode::

		mov %fs:offset, %reg
		mov %gs:offset, %reg

		mov %reg, %fs:offset
		mov %reg, %gs:offset

Documentation/x86/x86_64/index.rst

+1 −0

Original line number	Diff line number	Diff line
		@@ -14,3 +14,4 @@ x86_64 Support
		fake-numa-for-cpusets
		cpu-hotplug-spec
		machinecheck
		fsgs

arch/x86/entry/calling.h

+40 −0

Original line number	Diff line number	Diff line
		@@ -6,6 +6,7 @@
		#include <asm/percpu.h>
		#include <asm/asm-offsets.h>
		#include <asm/processor-flags.h>
		#include <asm/inst.h>

		/*

		@@ -341,6 +342,12 @@ For 32-bit we have the following conventions - kernel is built with
		#endif
		.endm

		.macro SAVE_AND_SET_GSBASE scratch_reg:req save_reg:req
		rdgsbase \save_reg
		GET_PERCPU_BASE \scratch_reg
		wrgsbase \scratch_reg
		.endm

		#else /* CONFIG_X86_64 */
		# undef UNWIND_HINT_IRET_REGS
		# define UNWIND_HINT_IRET_REGS
		@@ -351,3 +358,36 @@ For 32-bit we have the following conventions - kernel is built with
		call stackleak_erase
		#endif
		.endm

		#ifdef CONFIG_SMP

		/*
		* CPU/node NR is loaded from the limit (size) field of a special segment
		* descriptor entry in GDT.
		*/
		.macro LOAD_CPU_AND_NODE_SEG_LIMIT reg:req
		movq $__CPUNODE_SEG, \reg
		lsl \reg, \reg
		.endm

		/*
		* Fetch the per-CPU GSBASE value for this processor and put it in @reg.
		* We normally use %gs for accessing per-CPU data, but we are setting up
		* %gs here and obviously can not use %gs itself to access per-CPU data.
		*/
		.macro GET_PERCPU_BASE reg:req
		ALTERNATIVE \
		"LOAD_CPU_AND_NODE_SEG_LIMIT \reg", \
		"RDPID \reg", \
		X86_FEATURE_RDPID
		andq $VDSO_CPUNODE_MASK, \reg
		movq __per_cpu_offset(, \reg, 8), \reg
		.endm

		#else

		.macro GET_PERCPU_BASE reg:req
		movq pcpu_unit_offsets(%rip), \reg
		.endm

		#endif /* CONFIG_SMP */

arch/x86/entry/entry_64.S

+107 −32

Original line number	Diff line number	Diff line
		@@ -38,6 +38,7 @@
		#include <asm/frame.h>
		#include <asm/trapnr.h>
		#include <asm/nospec-branch.h>
		#include <asm/fsgsbase.h>
		#include <linux/err.h>

		#include "calling.h"
		@@ -426,10 +427,7 @@ SYM_CODE_START(\asmsym)
		testb $3, CS-ORIG_RAX(%rsp)
		jnz .Lfrom_usermode_switch_stack_\@

		/*
		* paranoid_entry returns SWAPGS flag for paranoid_exit in EBX.
		* EBX == 0 -> SWAPGS, EBX == 1 -> no SWAPGS
		*/
		/* paranoid_entry returns GS information for paranoid_exit in EBX. */
		call paranoid_entry

		UNWIND_HINT_REGS
		@@ -458,10 +456,7 @@ SYM_CODE_START(\asmsym)
		UNWIND_HINT_IRET_REGS offset=8
		ASM_CLAC

		/*
		* paranoid_entry returns SWAPGS flag for paranoid_exit in EBX.
		* EBX == 0 -> SWAPGS, EBX == 1 -> no SWAPGS
		*/
		/* paranoid_entry returns GS information for paranoid_exit in EBX. */
		call paranoid_entry
		UNWIND_HINT_REGS

		@@ -798,24 +793,21 @@ SYM_CODE_END(xen_failsafe_callback)
		#endif /* CONFIG_XEN_PV */

		/*
		* Save all registers in pt_regs, and switch gs if needed.
		* Use slow, but surefire "are we in kernel?" check.
		* Return: ebx=0: need swapgs on exit, ebx=1: otherwise
		* Save all registers in pt_regs. Return GSBASE related information
		* in EBX depending on the availability of the FSGSBASE instructions:
		*
		* FSGSBASE R/EBX
		* N 0 -> SWAPGS on exit
		* 1 -> no SWAPGS on exit
		*
		* Y GSBASE value at entry, must be restored in paranoid_exit
		*/
		SYM_CODE_START_LOCAL(paranoid_entry)
		UNWIND_HINT_FUNC
		cld
		PUSH_AND_CLEAR_REGS save_ret=1
		ENCODE_FRAME_POINTER 8
		movl $1, %ebx
		movl $MSR_GS_BASE, %ecx
		rdmsr
		testl %edx, %edx
		js 1f /* negative -> in kernel */
		SWAPGS
		xorl %ebx, %ebx

		1:
		/*
		* Always stash CR3 in %r14. This value will be restored,
		* verbatim, at exit. Needed if paranoid_entry interrupted
		@@ -825,9 +817,51 @@ SYM_CODE_START_LOCAL(paranoid_entry)
		* This is also why CS (stashed in the "iret frame" by the
		* hardware at entry) can not be used: this may be a return
		* to kernel code, but with a user CR3 value.
		*
		* Switching CR3 does not depend on kernel GSBASE so it can
		* be done before switching to the kernel GSBASE. This is
		* required for FSGSBASE because the kernel GSBASE has to
		* be retrieved from a kernel internal table.
		*/
		SAVE_AND_SWITCH_TO_KERNEL_CR3 scratch_reg=%rax save_reg=%r14

		/*
		* Handling GSBASE depends on the availability of FSGSBASE.
		*
		* Without FSGSBASE the kernel enforces that negative GSBASE
		* values indicate kernel GSBASE. With FSGSBASE no assumptions
		* can be made about the GSBASE value when entering from user
		* space.
		*/
		ALTERNATIVE "jmp .Lparanoid_entry_checkgs", "", X86_FEATURE_FSGSBASE

		/*
		* Read the current GSBASE and store it in %rbx unconditionally,
		* retrieve and set the current CPUs kernel GSBASE. The stored value
		* has to be restored in paranoid_exit unconditionally.
		*
		* The MSR write ensures that no subsequent load is based on a
		* mispredicted GSBASE. No extra FENCE required.
		*/
		SAVE_AND_SET_GSBASE scratch_reg=%rax save_reg=%rbx
		ret

		.Lparanoid_entry_checkgs:
		/* EBX = 1 -> kernel GSBASE active, no restore required */
		movl $1, %ebx
		/*
		* The kernel-enforced convention is a negative GSBASE indicates
		* a kernel value. No SWAPGS needed on entry and exit.
		*/
		movl $MSR_GS_BASE, %ecx
		rdmsr
		testl %edx, %edx
		jns .Lparanoid_entry_swapgs
		ret

		.Lparanoid_entry_swapgs:
		SWAPGS

		/*
		* The above SAVE_AND_SWITCH_TO_KERNEL_CR3 macro doesn't do an
		* unconditional CR3 write, even in the PTI case. So do an lfence
		@@ -835,6 +869,8 @@ SYM_CODE_START_LOCAL(paranoid_entry)
		*/
		FENCE_SWAPGS_KERNEL_ENTRY

		/* EBX = 0 -> SWAPGS required on exit */
		xorl %ebx, %ebx
		ret
		SYM_CODE_END(paranoid_entry)

		@@ -845,22 +881,44 @@ SYM_CODE_END(paranoid_entry)
		*
		* We may be returning to very strange contexts (e.g. very early
		* in syscall entry), so checking for preemption here would
		* be complicated. Fortunately, we there's no good reason
		* to try to handle preemption here.
		* be complicated. Fortunately, there's no good reason to try
		* to handle preemption here.
		*
		* On entry, ebx is "no swapgs" flag (1: don't need swapgs, 0: need it)
		* R/EBX contains the GSBASE related information depending on the
		* availability of the FSGSBASE instructions:
		*
		* FSGSBASE R/EBX
		* N 0 -> SWAPGS on exit
		* 1 -> no SWAPGS on exit
		*
		* Y User space GSBASE, must be restored unconditionally
		*/
		SYM_CODE_START_LOCAL(paranoid_exit)
		UNWIND_HINT_REGS
		testl %ebx, %ebx /* swapgs needed? */
		jnz .Lparanoid_exit_no_swapgs
		/* Always restore stashed CR3 value (see paranoid_entry) */
		RESTORE_CR3 scratch_reg=%rbx save_reg=%r14
		SWAPGS_UNSAFE_STACK
		/*
		* The order of operations is important. RESTORE_CR3 requires
		* kernel GSBASE.
		*
		* NB to anyone to try to optimize this code: this code does
		* not execute at all for exceptions from user mode. Those
		* exceptions go through error_exit instead.
		*/
		RESTORE_CR3 scratch_reg=%rax save_reg=%r14

		/* Handle the three GSBASE cases */
		ALTERNATIVE "jmp .Lparanoid_exit_checkgs", "", X86_FEATURE_FSGSBASE

		/* With FSGSBASE enabled, unconditionally restore GSBASE */
		wrgsbase %rbx
		jmp restore_regs_and_return_to_kernel
		.Lparanoid_exit_no_swapgs:
		/* Always restore stashed CR3 value (see paranoid_entry) */
		RESTORE_CR3 scratch_reg=%rbx save_reg=%r14

		.Lparanoid_exit_checkgs:
		/* On non-FSGSBASE systems, conditionally do SWAPGS */
		testl %ebx, %ebx
		jnz restore_regs_and_return_to_kernel

		/* We are returning to a context with user GSBASE */
		SWAPGS_UNSAFE_STACK
		jmp restore_regs_and_return_to_kernel
		SYM_CODE_END(paranoid_exit)

		@@ -1266,10 +1324,27 @@ end_repeat_nmi:
		/* Always restore stashed CR3 value (see paranoid_entry) */
		RESTORE_CR3 scratch_reg=%r15 save_reg=%r14

		testl %ebx, %ebx /* swapgs needed? */
		/*
		* The above invocation of paranoid_entry stored the GSBASE
		* related information in R/EBX depending on the availability
		* of FSGSBASE.
		*
		* If FSGSBASE is enabled, restore the saved GSBASE value
		* unconditionally, otherwise take the conditional SWAPGS path.
		*/
		ALTERNATIVE "jmp nmi_no_fsgsbase", "", X86_FEATURE_FSGSBASE

		wrgsbase %rbx
		jmp nmi_restore

		nmi_no_fsgsbase:
		/* EBX == 0 -> invoke SWAPGS */
		testl %ebx, %ebx
		jnz nmi_restore

		nmi_swapgs:
		SWAPGS_UNSAFE_STACK

		nmi_restore:
		POP_REGS

Admin message