From a1c7570cedd03372812a5b693732880867babbca Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 12:01:24 +0300 Subject: x86/mm/pat: Acquire init_mm write lock on collapse to avoid UAF x86 implements page attribute modification using its Change Page Attributes (CPA) mechanism. This tracks properties of ranges such as cache mode through x86 page attributes, and as part of that logic manipulates kernel page tables. Since commit: 41d88484c71c ("x86/mm/pat: restore large ROX pages after fragmentation") ranges of kernel page table entries can be collapsed into huge page table entries as part of this logic. As part of this collapse, it frees the page tables which the collapsed entries previously pointed to, and it does so without any relevant locks being held to preclude concurrent kernel page table walkers. The only way this code can be reached is if CPA_COLLAPSE is specified, and this is only set in set_memory_rox() via: set_memory_rox() -> change_page_attr_set_clr() -> cpa_flush() -> cpa_collapse_large_pages() Notable users of this are execmem and BPF when manipulating executable mappings. However, this is problematic for ptdump as it walks ranges it does not own and thus runs the risk of a use-after-free on page tables freed underneath it. In addition, concurrent CPA collapse operations are possible which can also cause races. Resolve the issue by acquiring the mmap write lock on init_mm across the whole operation. It is safe to acquire a sleeping lock as all the callers invoke set_memory_rox() from process context and in any case, change_page_attr_set_clr() calls vm_unmap_alias() which ultimately takes a mutex, disallowing atomic context here. Fixes: 41d88484c71c ("x86/mm/pat: restore large ROX pages after fragmentation") Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Mike Rapoport (Microsoft) Signed-off-by: Dave Hansen Signed-off-by: Ingo Molnar Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Kiryl Shutsemau (Meta) Reviewed-by: David Hildenbrand (Arm) Reviewed-by: Dave Hansen Reviewed-by: Will Deacon Reviewed-by: David Carlier Tested-by: Atish Patra Tested-by: Nikunj A Dadhania Cc:stable@vger.kernel.org Link: https://patch.msgid.link/20260813-cpa-fixes-v2-1-39b4ff90f91d@kernel.org --- arch/x86/mm/pat/set_memory.c | 15 ++++++++++++++- include/linux/mmap_lock.h | 2 ++ 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index c38faf39ce15..4abddd763882 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -22,6 +22,7 @@ #include #include #include +#include #include #include @@ -409,7 +410,7 @@ static void __cpa_flush_tlb(void *data) static int collapse_large_pages(unsigned long addr, struct list_head *pgtables); -static void cpa_collapse_large_pages(struct cpa_data *cpa) +static void __cpa_collapse_large_pages(struct cpa_data *cpa) { unsigned long start, addr, end; struct ptdesc *ptdesc, *tmp; @@ -443,6 +444,18 @@ static void cpa_collapse_large_pages(struct cpa_data *cpa) } } +static void cpa_collapse_large_pages(struct cpa_data *cpa) +{ + /* + * Take the mmap write lock on init_mm to: + * - Avoid a use-after-free if raced by ptdump (which takes its own + * write lock on init_mm). + * - Serialise concurrent CPA walkers. + */ + scoped_guard(mmap_write_lock, &init_mm) + __cpa_collapse_large_pages(cpa); +} + static void cpa_flush(struct cpa_data *cpa, int cache) { unsigned int i; diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index bec0eab6ef03..b8a13b8d36a4 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -630,6 +630,8 @@ static inline void mmap_read_unlock(struct mm_struct *mm) DEFINE_GUARD(mmap_read_lock, struct mm_struct *, mmap_read_lock(_T), mmap_read_unlock(_T)) DEFINE_GUARD_COND(mmap_read_lock, _try, mmap_read_trylock(_T)) +DEFINE_GUARD(mmap_write_lock, struct mm_struct *, + mmap_write_lock(_T), mmap_write_unlock(_T)) static inline void mmap_read_unlock_non_owner(struct mm_struct *mm) { -- cgit v1.2.3 From d5d8b8662e6e5a565b47a0388640e88402f23274 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 12:01:25 +0300 Subject: x86/mm/pat: Acquire init_mm read lock on attribute changes to avoid UAF A previous commit protected against races between ptdump and CPA collapse, however one still exists between attribute changes and collapse as reported by Denis V. Lunev (linked). When an attribute change arises, a lockless page table walker obtains a PTE entry, which is later written to via set_pte_atomic(): ... -> change_page_attr_set_clr() -> __change_page_attr_set_clr() -> __change_page_attr() -> _lookup_address_cpa() -> lookup_address_in_pgd_attr() -> [ lockless page table walker ] -> set_pte_atomic() There is nothing preventing a concurrent CPA collapse which can free the PTE that was retrieved here, resulting in a use-after-free. With the mmap write lock taken on init_mm over CPA collapse, resolve this race by acquiring an mmap read lock on init_mm over __change_page_attr_set_clr(). This locks across the whole operation over which the walk and the PTE entry write occurs, solving the race. It is safe to do this here, as no spinlocks are held upon entry to __change_page_attr_set_clr(). However, the lock must not be held over an allocation, as allocation can trigger reclaim and shrinkers may call into CPA recursively, making deadlocks possible (init_mm -> ... -> fs_reclaim -> init_mm). A page table is allocated when a huge page needs to be split: -> change_page_attr_set_clr() -> __change_page_attr_set_clr() -> __change_page_attr() -> split_large_page() [ pagetable_alloc() ] -> __split_large_page() Avoid deadlocks by dropping the mmap lock across pagetable_alloc() in split_large_page() and track whether this is needed by adding a new 'init_mm_read_locked' flag to struct cpa_data. This is safe as __split_large_page() (called with locks re-established) revalidates that the page table entry is the same as it was prior to the locks being dropped and __change_page_attr() repeats the entire page table walk whenever a split occurs, so concurrent split and collapse are accounted for. Concurrent ptdump is also safe as the lock is only dropped over page table allocation during which time the page table has not yet been modified. The CPA_COLLAPSE flag is only set by set_memory_rox(), which exclusively operates upon vmalloc ranges, and on x86 only within the module mapping space. This is important, because some callers directly invoke __change_page_attr_set_clr(), bypassing this lock. However, none of these operate within the module mapping space. * cpa_process_alias() - a recursive helper called by __change_page_attr_set_clr(). * __set_memory_enc_pgtable() - operates on the direct mapping and (via __vmbus_establish_gpadl()) the vmalloc mapping space. * __set_pages_[n]p() - called by set_direct_map_[invalid, default, valid]_noflush(), __kernel_map_pages() - operates on the direct map. * kernel_[un]map_pages_in_pgd() - operates on EFI ranges. This work is based upon Denis V. Lunev's excellent analysis of the bug with gratitude. [ dhansen: move to imperative voice in changelog ] Fixes: 41d88484c71c ("x86/mm/pat: restore large ROX pages after fragmentation") Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Mike Rapoport (Microsoft) Signed-off-by: Dave Hansen Signed-off-by: Ingo Molnar Tested-by: Atish Patra Tested-by: Nikunj A Dadhania Link: https://lore.kernel.org/all/20260626163213.2284080-1-den@openvz.org/ Cc:stable@vger.kernel.org Link: https://patch.msgid.link/20260813-cpa-fixes-v2-2-39b4ff90f91d@kernel.org --- arch/x86/mm/pat/set_memory.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index 4abddd763882..cb5d6d6f71a4 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -50,7 +50,8 @@ struct cpa_data { unsigned int flags; unsigned int force_split : 1, force_static_prot : 1, - force_flush_all : 1; + force_flush_all : 1, + init_mm_read_locked : 1; struct page **pages; }; @@ -1240,7 +1241,11 @@ static int split_large_page(struct cpa_data *cpa, pte_t *kpte, struct ptdesc *ptdesc; spin_unlock(&cpa_lock); + if (cpa->init_mm_read_locked) + mmap_read_unlock(&init_mm); ptdesc = pagetable_alloc(GFP_KERNEL, 0); + if (cpa->init_mm_read_locked) + mmap_read_lock(&init_mm); spin_lock(&cpa_lock); if (!ptdesc) return -ENOMEM; @@ -2134,7 +2139,11 @@ static int change_page_attr_set_clr(unsigned long *addr, int numpages, cpa.curpage = 0; cpa.force_split = force_split; - ret = __change_page_attr_set_clr(&cpa, 1); + /* Avoid race with concurrent CPA collapse. */ + cpa.init_mm_read_locked = true; + scoped_guard(mmap_read_lock, &init_mm) + ret = __change_page_attr_set_clr(&cpa, 1); + cpa.init_mm_read_locked = false; /* * Check whether we really changed something: -- cgit v1.2.3 From 1587d3394e254639cc36516256031334095e6ef3 Mon Sep 17 00:00:00 2001 From: Pedro Falcato Date: Thu, 13 Aug 2026 12:01:26 +0300 Subject: x86/alternatives: Exclude text poking against change_page_attr() From time to time, the following BUG can be observed in the x86 alternatives patching code [0]: > kernel BUG at arch/x86/kernel/alternative.c:2576! > Oops: invalid opcode: 0000 [#1] SMP NOPTI > CPU: 0 UID: 0 PID: 355 Comm: (udev-worker) Not tainted 7.1.3-1-default #1 PREEMPT(full) openSUSE Tumbleweed 8c1795b03ec64f997e57a8ad38b1161e3b98da64 > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS unknown 02/02/2022 > RIP: 0010:__text_poke+0x2aa/0x450 > Call Trace: > > smp_text_poke_batch_finish+0x2a7/0x320 > __static_call_transform+0xb7/0x220 > arch_static_call_transform+0x5b/0xb0 > __static_call_init+0xe9/0x270 > static_call_module_notify+0x11f/0x150 > notifier_call_chain+0x61/0xe0 > blocking_notifier_call_chain_robust+0x63/0xc0 > load_module+0x1c92/0x20c0 > init_module_from_file+0xd8/0x140 > idempotent_init_module+0x100/0x2f0 > __x64_sys_finit_module+0x71/0xe0 > do_syscall_64+0xe1/0x610 > entry_SYSCALL_64_after_hwframe+0x76/0x7e which matches the following BUG_ON() in alternative.c: /* * If something went wrong, crash and burn since recovery paths are not * implemented. */ BUG_ON(!pages[0] || (cross_page_boundary && !pages[1])); This can happen if vmalloc_to_page() fails, for any reason. Such can happen if text poking races with CPA, which can possibly result in the collapsing of page tables (or breaking of PMD hugepages). It is not a problem for most users of vmalloc_to_page() (they solely own the vmalloc'd range) but, when CONFIG_ARCH_HAS_EXECMEM_ROX=y, various modules own a single execmem vmalloc range, and can call set_memory_*() in parallel on it. This can happen to race against __text_poke and cause havoc in vmalloc_to_page(). Fix it by excluding against CPA using the init_mm mmap read lock. [ dhansen: Fix up SoB ordering. The actual code flow here was: Pedro=>Lorenzo=>Mike=>Me which is reflected in the SoB chain now. I *believe* Mike simply picked up Lorenzo's update to Pedro's post from the Link ] Fixes: 64f6a4e10c05 ("x86: re-enable EXECMEM_ROX support") Reported-by: Jiri Slaby Reported-by: Steffen Dirkwinkel Signed-off-by: Pedro Falcato Signed-off-by: Lorenzo Stoakes (ARM) Co-developed-by: Lorenzo Stoakes (ARM) Signed-off-by: Mike Rapoport (Microsoft) Signed-off-by: Dave Hansen Signed-off-by: Ingo Molnar Tested-by: Jiri Slaby Tested-by: Atish Patra Tested-by: Nikunj A Dadhania Cc: stable@vger.kernel.org Link: https://bugzilla.opensuse.org/show_bug.cgi?id=1271202 [0] Link: https://lore.kernel.org/linux-mm/555ea1d43a12c30a8f1eaf10c899b3790d728f33.camel@dirkwinkel.cc/ Link: https://patch.msgid.link/20260813-cpa-fixes-v2-3-39b4ff90f91d@kernel.org --- arch/x86/kernel/alternative.c | 39 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 3 deletions(-) diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 91b1cdd16569..add62db3e82c 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -6,6 +6,9 @@ #include #include #include +#include +#include +#include #include #include @@ -2372,6 +2375,38 @@ static void text_poke_memset(void *dst, const void *src, size_t len) typedef void text_poke_f(void *dst, const void *src, size_t len); +static void __poke_vmalloc_pages(struct page **pages, void *addr, + bool cross_page_boundary) +{ + pages[0] = vmalloc_to_page(addr); + if (cross_page_boundary) + pages[1] = vmalloc_to_page(addr + PAGE_SIZE); +} + +static void poke_vmalloc_pages(struct page **pages, void *addr, + bool cross_page_boundary) +{ + if (in_dbg_master()) { + /* + * If called from kgdb cannot sleep, but all other CPUs stopped + * anyway so safe to proceed without locks + */ + __poke_vmalloc_pages(pages, addr, cross_page_boundary); + } else { + /* + * execmem ROX ranges are shared between modules and can be + * collapsed to huge PMD entries, and this collapse can happen + * concurrently with a racing set_memory_rox(). + * + * Prevent vmalloc_to_page() from racing by acquiring an + * init_mm read lock which pairs with the init_mm write lock in + * cpa_collapse_large_pages(). + */ + guard(mmap_read_lock)(&init_mm); + __poke_vmalloc_pages(pages, addr, cross_page_boundary); + } +} + static void *__text_poke(text_poke_f func, void *addr, const void *src, size_t len) { bool cross_page_boundary = offset_in_page(addr) + len > PAGE_SIZE; @@ -2389,9 +2424,7 @@ static void *__text_poke(text_poke_f func, void *addr, const void *src, size_t l BUG_ON(!after_bootmem); if (!core_kernel_text((unsigned long)addr)) { - pages[0] = vmalloc_to_page(addr); - if (cross_page_boundary) - pages[1] = vmalloc_to_page(addr + PAGE_SIZE); + poke_vmalloc_pages(pages, addr, cross_page_boundary); } else { pages[0] = virt_to_page(addr); WARN_ON(!PageReserved(pages[0])); -- cgit v1.2.3 From 9e4a3ec3411bb6bb59e3c1f29b75609f1e87aac4 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 12:01:27 +0300 Subject: x86/mm/pat: Allocate split page tables as kernel page tables A PTE is allocated directly without going through the standard page table allocation routines (such as pte_alloc_one_kernel()) when the CPA code splits a large page (__split_large_page()). This means the page table constructor is never called nor is the page table marked as a kernel page table. The former results in the folio associated with the page table not being marked as a page table (__pagetable_ctor() is never called thus neither is __folio_set_pgtable()) nor are statistics updated to reflect it (lruvec_stat_add_folio() is never called). The latter issue of failing to mark the page table as a kernel page table (ptdesc_set_kernel() is never called) is far more problematic. Since commit: 5ba2f0a15564 ("mm: introduce deferred freeing for kernel page tables") kernel page table freeing has been batched and since the subsequent commit: e37d5a2d60a3 ("iommu/sva: invalidate stale IOTLB entries for kernel address space") IOTLB cache entries for kernel page tables have been invalidated upon being freed. Since split page tables are freed without this invalidation, the IOTLB can contain stale entries for them. Resolve the issue by using the ordinary PTE allocation API at split time. This results in these kernel page tables invoking a page table constructor, and thus requires a page table destructor. Destructors are not always present, like for early allocated direct map page tables). Conditionally call pagetable_dtor_free() if the PG_table folio flag for the ptdesc is set, otherwise we free the page table via pagetable_free(). Regardless of which path is taken page tables marked as kernel page tables, which now includes split page tables, take the correct route through pagetable_free_kernel(). There is a user-visible side effect in that split page tables will appear in nr_page_table_pages in /proc/vmstat (as do other kernel page tables allocated after early boot), however this is a positive change. This issue started being markedly problematic after commit: 5ba2f0a15564 ("mm: introduce deferred freeing for kernel page tables") so choose this as the Fixes target. [ dhansen: rephrase in imperative mood ] Fixes: 5ba2f0a15564 ("mm: introduce deferred freeing for kernel page tables") Signed-off-by: Lorenzo Stoakes (ARM) Signed-off-by: Mike Rapoport (Microsoft) Signed-off-by: Dave Hansen Signed-off-by: Ingo Molnar Acked-by: Vishal Moola Tested-by: Atish Patra Tested-by: Nikunj A Dadhania Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260813-cpa-fixes-v2-4-39b4ff90f91d@kernel.org --- arch/x86/mm/pat/set_memory.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index cb5d6d6f71a4..4652487b5572 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -441,7 +441,15 @@ static void __cpa_collapse_large_pages(struct cpa_data *cpa) list_for_each_entry_safe(ptdesc, tmp, &pgtables, pt_list) { list_del(&ptdesc->pt_list); - pagetable_free(ptdesc); + /* + * Only early alloc'd direct map should not be flagged PG_table + * here and those shouldn't be collapsed. However be abundantly + * cautious and handle the !PG_table case too. + */ + if (PageTable((ptdesc_page(ptdesc)))) + pagetable_dtor_free(ptdesc); + else + pagetable_free(ptdesc); } } @@ -1134,11 +1142,10 @@ set: static int __split_large_page(struct cpa_data *cpa, pte_t *kpte, unsigned long address, - struct ptdesc *ptdesc) + pte_t *pbase) { unsigned long lpaddr, lpinc, ref_pfn, pfn, pfninc = 1; - struct page *base = ptdesc_page(ptdesc); - pte_t *pbase = (pte_t *)page_address(base); + struct page *base = virt_to_page(pbase); unsigned int i, level; pgprot_t ref_prot; bool nx, rw; @@ -1238,20 +1245,20 @@ __split_large_page(struct cpa_data *cpa, pte_t *kpte, unsigned long address, static int split_large_page(struct cpa_data *cpa, pte_t *kpte, unsigned long address) { - struct ptdesc *ptdesc; + pte_t *pte; spin_unlock(&cpa_lock); if (cpa->init_mm_read_locked) mmap_read_unlock(&init_mm); - ptdesc = pagetable_alloc(GFP_KERNEL, 0); + pte = pte_alloc_one_kernel(&init_mm); if (cpa->init_mm_read_locked) mmap_read_lock(&init_mm); spin_lock(&cpa_lock); - if (!ptdesc) + if (!pte) return -ENOMEM; - if (__split_large_page(cpa, kpte, address, ptdesc)) - pagetable_free(ptdesc); + if (__split_large_page(cpa, kpte, address, pte)) + pte_free_kernel(&init_mm, pte); return 0; } -- cgit v1.2.3 From f7491d7c81db0e7c304a7bd757a76d2fbeaff80e Mon Sep 17 00:00:00 2001 From: Vernon Yang Date: Thu, 3 Sep 2026 11:16:08 +0800 Subject: x86/mm: Fix user-space data loss with MADV_FREE and THP Some of users of Polars (a data analytics library) have lost production data from this bug. They seem to have just the right combination of huge pages, MADV_FREE and heavy reclaim pressure. pmd_modify() masks the old value with (_HPAGE_CHG_MASK & ~_PAGE_DIRTY), silently discarding the hardware dirty bit. The subsequent pmd_mksaveddirty() call is supposed to transfer _PAGE_DIRTY into _PAGE_SAVED_DIRTY when write-protecting, but the dirty bit was already stripped from the value, so there is nothing left to transfer. Contrast with pte_modify(), which keeps _PAGE_DIRTY_BITS in its mask, and pud_modify(), which keeps _HPAGE_CHG_MASK untouched: pmd_modify() is the odd one out. Any pmd_modify() on a writable, dirty PMD loses the dirty state. One visible consequence is data loss with MADV_FREE on PMD-mapped THP: memset(buf, 0x5A, size); // PMD-mapped THP, PMD dirty madvise(buf, size, MADV_FREE); // PMD cleaned but left writable, // folio marked lazyfree memset(buf, 0x5A, size); // hardware sets _PAGE_DIRTY again mprotect(buf, size, PROT_READ); // pmd_modify() drops the dirty bit mprotect(buf, size, PROT_READ|PROT_WRITE); // ... memory pressure ... Reclaim (e.g. under memcg pressure) then finds the lazyfree folio with no dirty bit set anywhere and frees it in __discard_anon_folio_pmd_locked(), even though the data was rewritten after MADV_FREE; subsequent reads fault in fresh zero pages. NUMA hinting alone can trigger the same loss, as do_huge_pmd_numa_page() restores the PMD through pmd_modify() as well. PMD-mapped file THPs are affected too: mprotect()/NUMA hinting dropping the dirty bit means rewritten data is never written back. Fix it by keeping _PAGE_DIRTY in the preserved mask, exactly like pte_modify() and pud_modify() do. The existing pmd_mksaveddirty()/pmd_clear_saveddirty() pair then performs the hardware-dirty <-> saved-dirty transition based on the write bit, preserving the shadow-stack encoding rules. Fixes: bb3aadf7d446 ("x86/mm: Start actually marking _PAGE_SAVED_DIRTY") Closes: https://lore.kernel.org/r/CAJxLxMUGu1-L+O_nAONOwOXnS=cNbNApCWqdthRjd76LThtSPg@mail.gmail.com/ Reported-by: Orson Peters Signed-off-by: Vernon Yang Signed-off-by: Dave Hansen Signed-off-by: Ingo Molnar Reviewed-by: Rick Edgecombe Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260903031608.1194238-1-vernon2gm@gmail.com --- arch/x86/include/asm/pgtable.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index d5f4917c1edc..d551120a7c88 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -806,7 +806,7 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) pmdval_t val = pmd_val(pmd), oldval = val; pmd_t pmd_result; - val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY); + val &= _HPAGE_CHG_MASK; val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK); -- cgit v1.2.3 From 5a5d26f2cfe13467166219f6bf58099326912ddb Mon Sep 17 00:00:00 2001 From: Soheil Hassas Yeganeh Date: Mon, 31 Aug 2026 14:48:44 +0000 Subject: x86/cfi: Fix FineIBT hash offset in cfi_get_func_hash() The switch of the FineIBT preamble from "subl $hash, %r10d" to the shorter "subl $hash, %eax" moved the hash immediate from offset 7 to offset 5 of the preamble. fineibt_preamble_hash was updated to match, but the open-coded offset in cfi_get_func_hash() was missed and it still reads the hash at offset 7. cfi_get_func_hash() is used by the BPF JIT to give a struct_ops trampoline the CFI hash of the stub function it stands in for. With FineIBT the trampoline now gets the upper half of the real hash followed by the first two bytes of the next instruction, so the first indirect call from the kernel into a struct_ops program, tcp_init_congestion_control() calling ->init() of a BPF congestion control for example, fails the FineIBT check and the kernel dies with a CFI failure. Move the FineIBT preamble template and its offset defines above cfi_get_func_hash() and use fineibt_preamble_hash there, so every reader of the preamble shares one definition of its layout. The CFI_FINEIBT arm is only built with CONFIG_FINEIBT, the only configuration in which cfi_mode can take that value. cfi_get_func_arity() does not need the same treatment: the __bhi_args call whose displacement it reads still ends at the function address. Fixes: 85a2d4a890dc ("x86,ibt: Use UDB instead of 0xEA") Assisted-by: LLM Signed-off-by: Soheil Hassas Yeganeh Signed-off-by: Peter Zijlstra (Intel) Cc: stable@vger.kernel.org # 6.18+ Link: https://patch.msgid.link/20260831-b4-x86-cfi-fineibt-func-hash-v1-1-6ffc0af5c4ec@gmail.com --- arch/x86/kernel/alternative.c | 72 +++++++++++++++++++++++-------------------- 1 file changed, 38 insertions(+), 34 deletions(-) diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index add62db3e82c..741d8767ddf8 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -1201,6 +1201,41 @@ static bool cfi_debug __ro_after_init; bool cfi_bhi __ro_after_init = false; #endif +#ifdef CONFIG_FINEIBT +/* + * : + * 0: f3 0f 1e fa endbr64 + * 4: 2d 78 56 34 12 sub $0x12345678, %eax + * 9: 2e 0f 85 03 00 00 00 jne,pn 13 + * 10: 0f 1f 40 d6 nopl -0x2a(%rax) + * + * Note that the JNE target is the 0xD6 byte inside the NOPL, this decodes as + * UDB on x86_64 and raises #UD. + */ +asm( ".pushsection .rodata \n" + "fineibt_preamble_start: \n" + " endbr64 \n" + " subl $0x12345678, %eax \n" + "fineibt_preamble_bhi: \n" + " cs jne.d32 fineibt_preamble_start+0x13 \n" + "#fineibt_func: \n" + " nopl -42(%rax) \n" + "fineibt_preamble_end: \n" + ".popsection\n" +); + +extern u8 fineibt_preamble_start[]; +extern u8 fineibt_preamble_bhi[]; +extern u8 fineibt_preamble_end[]; + +#define fineibt_preamble_size (fineibt_preamble_end - fineibt_preamble_start) +#define fineibt_preamble_bhi (fineibt_preamble_bhi - fineibt_preamble_start) +#define fineibt_preamble_ud 0x13 +#define fineibt_preamble_hash 5 + +#define fineibt_prefix_size (fineibt_preamble_size - ENDBR_INSN_SIZE) +#endif /* CONFIG_FINEIBT */ + #ifdef CONFIG_CFI u32 cfi_get_func_hash(void *func) { @@ -1208,9 +1243,11 @@ u32 cfi_get_func_hash(void *func) func -= cfi_get_offset(); switch (cfi_mode) { +#ifdef CONFIG_FINEIBT case CFI_FINEIBT: - func += 7; + func += fineibt_preamble_hash; break; +#endif case CFI_KCFI: func += 1; break; @@ -1366,39 +1403,6 @@ early_param("cfi", cfi_parse_cmdline); * anyway. */ -/* - * : - * 0: f3 0f 1e fa endbr64 - * 4: 2d 78 56 34 12 sub $0x12345678, %eax - * 9: 2e 0f 85 03 00 00 00 jne,pn 13 - * 10: 0f 1f 40 d6 nopl -0x2a(%rax) - * - * Note that the JNE target is the 0xD6 byte inside the NOPL, this decodes as - * UDB on x86_64 and raises #UD. - */ -asm( ".pushsection .rodata \n" - "fineibt_preamble_start: \n" - " endbr64 \n" - " subl $0x12345678, %eax \n" - "fineibt_preamble_bhi: \n" - " cs jne.d32 fineibt_preamble_start+0x13 \n" - "#fineibt_func: \n" - " nopl -42(%rax) \n" - "fineibt_preamble_end: \n" - ".popsection\n" -); - -extern u8 fineibt_preamble_start[]; -extern u8 fineibt_preamble_bhi[]; -extern u8 fineibt_preamble_end[]; - -#define fineibt_preamble_size (fineibt_preamble_end - fineibt_preamble_start) -#define fineibt_preamble_bhi (fineibt_preamble_bhi - fineibt_preamble_start) -#define fineibt_preamble_ud 0x13 -#define fineibt_preamble_hash 5 - -#define fineibt_prefix_size (fineibt_preamble_size - ENDBR_INSN_SIZE) - /* * : * 0: b8 78 56 34 12 mov $0x12345678, %eax -- cgit v1.2.3 From f65d38155aef069c897a64643a03db237dd1e0c8 Mon Sep 17 00:00:00 2001 From: David Laight Date: Mon, 3 Aug 2026 10:47:01 +0100 Subject: x86/div64: Fix addition of large constants in mul_u64_add_u64_div_u64() Adding constants over 2^31 fails to compile because the ADD instruction only supports 32bit signed immediates. Replace the "irm" constraint with "erm" so that the compiler loads large constants into a register. Found by a patch to drivers/iio/frequency/ad9910.c [ bp: Massage commit message. ] Fixes: 6480241f31f5 ("lib: add mul_u64_add_u64_div_u64() and mul_u64_u64_div_u64_roundup()") Signed-off-by: David Laight Signed-off-by: Borislav Petkov (AMD) Reviewed-by: H. Peter Anvin Link: https://patch.msgid.link/20260803094702.3852-2-david.laight.linux@gmail.com --- arch/x86/include/asm/div64.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/include/asm/div64.h b/arch/x86/include/asm/div64.h index 30fd06ede751..8a2d343f977e 100644 --- a/arch/x86/include/asm/div64.h +++ b/arch/x86/include/asm/div64.h @@ -111,7 +111,7 @@ static inline u64 mul_u64_add_u64_div_u64(u64 rax, u64 mul, u64 add, u64 div) if (!statically_true(!add)) asm ("addq %[add], %[lo]; adcq $0, %[hi]" : - [lo] "+r" (rax), [hi] "+r" (rdx) : [add] "irm" (add)); + [lo] "+r" (rax), [hi] "+r" (rdx) : [add] "erm" (add)); asm ("divq %[div]" : "+a" (rax), "+d" (rdx) : [div] "rm" (div)); -- cgit v1.2.3 From 27600805e62f800bacf990354632eae4e487d34c Mon Sep 17 00:00:00 2001 From: Yazen Ghannam Date: Thu, 3 Sep 2026 10:43:25 -0500 Subject: x86/amd_node: Fix PCI device reference counting in amd_smn_init() The local "root" pointer is a temporary variable used during the device search. Therefore, refcount related to the search iterators should be cleaned up after the search is complete. Use the __free() cleanup macro to ensure the refcount is decremented when the temporary pointer goes out of scope. Additionally, increment the refcount when caching a root pointer. This ensures the in-use refcount is separate from the temporary search refcounting. Finally, drop the redundant "root = NULL" before the second search loop. The pci_get_class() iterator always decrements the refcount of its "from" argument, so the first loop can only fall through with "root" already NULL. Fixes: 0a4b61d9c2e4 ("x86/amd_node: Fix AMD root device caching") Closes: https://sashiko.dev/#/patchset/20260806160159.230453-1-jason.andryuk%40amd.com Reported-by: Sashiko Assisted-by: LLM Signed-off-by: Yazen Ghannam Signed-off-by: Borislav Petkov (AMD) Reviewed-by: Mario Limonciello (AMD) Cc: Link: https://patch.msgid.link/20260903154325.74343-1-yazen.ghannam@amd.com --- arch/x86/kernel/amd_node.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/arch/x86/kernel/amd_node.c b/arch/x86/kernel/amd_node.c index 762585775b5a..b7926ba3610a 100644 --- a/arch/x86/kernel/amd_node.c +++ b/arch/x86/kernel/amd_node.c @@ -251,7 +251,7 @@ __setup("amd_smn_debugfs_enable", amd_smn_enable_dfs); static int __init amd_smn_init(void) { u16 count, num_roots, roots_per_node, node, num_nodes; - struct pci_dev *root; + struct pci_dev *root __free(pci_dev_put) = NULL; if (!cpu_feature_enabled(X86_FEATURE_ZEN)) return 0; @@ -262,7 +262,6 @@ static int __init amd_smn_init(void) return 0; num_roots = 0; - root = NULL; while ((root = get_next_root(root))) { pci_dbg(root, "Reserving PCI config space\n"); @@ -299,14 +298,13 @@ static int __init amd_smn_init(void) count = 0; node = 0; - root = NULL; while (node < num_nodes && (root = get_next_root(root))) { /* Use one root for each node and skip the rest. */ if (count++ % roots_per_node) continue; pci_dbg(root, "is root for AMD node %u\n", node); - amd_roots[node++] = root; + amd_roots[node++] = pci_dev_get(root); } if (enable_dfs) { -- cgit v1.2.3