From d230991493b521eeff39f32434fddcbcdb109eb0 Mon Sep 17 00:00:00 2001 From: Johannes Weiner Date: Mon, 29 Jun 2026 12:33:37 -0400 Subject: mm: mempolicy: fix automatic numa balancing for shmem Neha reports that mapped shmem aren't considered for NUMA balancing, noting convergence problems and bandwidth bottlenecking for cachelib based workloads on tiered memory systems. Looking at the code and going through the git history, this doesn't actually seem intentional: Commit fc3147245d19 ("mm: numa: Limit NUMA scanning to migrate-on-fault VMAs") added a vma_policy_mof() gate to task_numa_work() so VMAs whose policy lacks MPOL_F_MOF are skipped from NUMA balancing scans. The motivation was a real usecase: Oracle was pinning shared segments with mbind(MPOL_BIND) so trapping faults was both expensive and pointless. The handling of NULL from vm_ops->get_policy, however, treated "user explicitly opted out" the same as "user never specified anything." For VMAs whose shared policy is absent - the common case for shmem - the scan was disabled too. This issue is old. It probably hurts less in conventional NUMA. But it's very noticeable on tiered systems, where entire tmpfs workingsets can get stuck on lower-bandwidth memory. Fix this by having vma_policy_mof() use __get_vma_policy() directly, and thereby handle the fallback to task policy (-> preferred_node_policy() has MPOL_F_MOF per default). Every other consumer of vm_ops->get_policy already handles it this way, the scan-eligibility check was the outlier. This preserves Mel's intended fix: don't scan stuff the user explicitly pinned. But allow default policy vmas to participate in balancing. Link: https://lore.kernel.org/20260629163337.1264881-1-hannes@cmpxchg.org Fixes: fc3147245d19 ("mm: numa: Limit NUMA scanning to migrate-on-fault VMAs") Signed-off-by: Johannes Weiner Reported-by: Neha Gholkar Tested-by: Neha Gholkar Reviewed-by: Gregory Price Acked-by: David Hildenbrand (Arm) Acked-by: Balbir Singh Cc: Alistair Popple Cc: Byungchul Park Cc: "Huang, Ying" Cc: Joshua Hahn Cc: Matthew Brost Cc: Rakie Kim Cc: Zi Yan Cc: Signed-off-by: Andrew Morton --- mm/mempolicy.c | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 501e0b80d7da..5720f7f54d94 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2060,24 +2060,15 @@ struct mempolicy *get_vma_policy(struct vm_area_struct *vma, bool vma_policy_mof(struct vm_area_struct *vma) { struct mempolicy *pol; + pgoff_t ilx; + bool mof; - if (vma->vm_ops && vma->vm_ops->get_policy) { - bool ret = false; - pgoff_t ilx; /* ignored here */ - - pol = vma->vm_ops->get_policy(vma, vma->vm_start, &ilx); - if (pol && (pol->flags & MPOL_F_MOF)) - ret = true; - mpol_cond_put(pol); - - return ret; - } - - pol = vma->vm_policy; + pol = __get_vma_policy(vma, vma->vm_start, &ilx); if (!pol) pol = get_task_policy(current); - - return pol->flags & MPOL_F_MOF; + mof = pol->flags & MPOL_F_MOF; + mpol_cond_put(pol); + return mof; } bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone) -- cgit v1.2.3 From 644ad84870ab503f5619d21cf3d37efb134e82de Mon Sep 17 00:00:00 2001 From: Hajime Tazaki Date: Thu, 2 Jul 2026 10:25:46 +0900 Subject: mm: nommu: point to the write iterator upon split_vma When a user invokes munmap(2) on a partial region allocated by mmap(), the kernel may split the original region if necessary and shrink it to the correct size. At the beginning of vmi_shrink_vma(), the unused part is cleared; however, an assertion is triggered if the shrink occurs after split_vma(). This commit fixes the issue by correctly configuring the pointer to the iterator at the end of split_vma(). This bug was detected using the Linux Test Project (LTP) test linked below, running on a nommu UML (User-Mode Linux) environment (via an out-of-tree extension to UML). Here is a minimal reproducible chunk of code for this issue: void *addr; size_t pagesize = getpagesize(); addr = mmap(NULL, pagesize * 4, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); munmap(addr + pagesize * 1, pagesize); This is the console output with CONFIG_DEBUG_MAPLE_TREE=y. nommu: WARN at __mas_set_range:791 (1) MAS: tree=0000000091c23b08 enode=0000000065057663 (ma_active) Store Type: node_store [9/9] index=70af8000 last=ffffffffffffffff min=0 max=ffffffffffffffff sheaf=0000000000000000, request 0 depth=0, flags=0 maple_tree(0000000091c23b08) flags 307, height 1 root 0000000083394c06 0-ffffffffffffffff: node 0000000010c90bd6 depth 0 type 1 parent 0000000050e1ddf8 contents: 0000000000000000 707A 7FFF 00000000eb0ac2b5 707AFFFF 0000000000000000 7093FFFF 0000000045ead616 7095FFFF 0000000000000000 7096CFFF 000 00000681c7151 7096FFFF 0000000000000000 70AF3FFF 000000006c78b9e9 70AF4FFF 000000001914ab0b 70AF7FFF 00000000000 00000 FFFFFFFFFFFFFFFF 0000000000000000 0 0000000000000000 0 0000000000000000 0 0000000000000000 0 0000000000000 000 0 00000000bca8be4f 0-707a7fff: 0000000000000000 707a8000-707affff: 00000000eb0ac2b5 707b0000-7093ffff: 0000000000000000 70940000-7095ffff: 0000000045ead616 70960000-7096cfff: 0000000000000000 7096d000-7096ffff: 00000000681c7151 70970000-70af3fff: 0000000000000000 70af4000-70af4fff: 000000006c78b9e9 70af5000-70af7fff: 000000001914ab0b 70af8000-ffffffffffffffff: 0000000000000000 nommu: Pass: 796 Run:797 Link: https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/syscalls/mseal/mseal02.c Link: https://lore.kernel.org/20260702012546.665383-1-thehajime@gmail.com Signed-off-by: Hajime Tazaki Cc: Jann Horn Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Pedro Falcato Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- mm/nommu.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/nommu.c b/mm/nommu.c index 277f663e1c5b..498e01ee40b0 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -1393,6 +1393,10 @@ static int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, setup_vma_to_mm(vma, mm); setup_vma_to_mm(new, mm); vma_iter_store_new(vmi, new); + + /* vmi should point lower address */ + if (new_below) + vma_next(vmi); mm->map_count++; return 0; -- cgit v1.2.3 From 9b5e4809806cb300cc163b26fa70dfd36e3577b3 Mon Sep 17 00:00:00 2001 From: Ethan Nelson-Moore Date: Wed, 10 Jun 2026 15:09:04 -0700 Subject: maple_tree: remove undocumented CONFIG_MAPLE_RCU_DISABLED macro consults the macro CONFIG_MAPLE_RCU_DISABLED to determine whether to disable the mt_in_rcu() function (by making it always return false). This macro is not reachable via Kconfig, despite its name, and is not documented anywhere. Remove it to avoid polluting the CONFIG_* namespace. Discovered while searching for CONFIG_* symbols referenced in code but not defined in any Kconfig file. Link: https://lore.kernel.org/20260610220905.99860-1-enelsonmoore@gmail.com Signed-off-by: Ethan Nelson-Moore Acked-by: SeongJae Park Reviewed-by: Liam Howlett Reviewed-by: Alice Ryhl Cc: Andrew Ballance Signed-off-by: Andrew Morton --- include/linux/maple_tree.h | 4 ---- 1 file changed, 4 deletions(-) diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h index 4a5631906aff..1b3014377105 100644 --- a/include/linux/maple_tree.h +++ b/include/linux/maple_tree.h @@ -11,7 +11,6 @@ #include #include #include -/* #define CONFIG_MAPLE_RCU_DISABLED */ /* * Allocated nodes are mutable until they have been inserted into the tree, @@ -864,9 +863,6 @@ static inline void mt_init(struct maple_tree *mt) static inline bool mt_in_rcu(struct maple_tree *mt) { -#ifdef CONFIG_MAPLE_RCU_DISABLED - return false; -#endif return mt->ma_flags & MT_FLAGS_USE_RCU; } -- cgit v1.2.3 From 1eba458a54d073aa8bd62627ad0e9264ef0f51e3 Mon Sep 17 00:00:00 2001 From: Kaitao Cheng Date: Tue, 7 Jul 2026 17:03:31 +0800 Subject: mm/kconfig: drop redundant memory hotplug dependencies MHP_MEMMAP_ON_MEMORY is defined inside the MEMORY_HOTPLUG block, and MEMORY_HOTPLUG already depends on SPARSEMEM_VMEMMAP. Keep the explicit MEMORY_HOTPLUG dependency for local readability, but drop the redundant SPARSEMEM_VMEMMAP dependency. ZONE_DEVICE depends on MEMORY_HOTREMOVE, which depends on MEMORY_HOTPLUG. MEMORY_HOTPLUG in turn depends on SPARSEMEM_VMEMMAP. Drop the direct MEMORY_HOTPLUG and SPARSEMEM_VMEMMAP dependencies from ZONE_DEVICE. This does not change the set of valid configurations. Link: https://lore.kernel.org/20260707090331.52971-1-kaitao.cheng@linux.dev Signed-off-by: Kaitao Cheng Acked-by: David Hildenbrand (Arm) Acked-by: Muchun Song Reviewed-by: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Michal Hocko Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- mm/Kconfig | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/mm/Kconfig b/mm/Kconfig index 3185500ce7b7..d28dde592de4 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -588,7 +588,7 @@ config MEMORY_HOTREMOVE config MHP_MEMMAP_ON_MEMORY def_bool y - depends on MEMORY_HOTPLUG && SPARSEMEM_VMEMMAP + depends on MEMORY_HOTPLUG depends on ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE endif # MEMORY_HOTPLUG @@ -1227,9 +1227,7 @@ config ZONE_DMA32 config ZONE_DEVICE bool "Device memory (pmem, HMM, etc...) hotplug support" - depends on MEMORY_HOTPLUG depends on MEMORY_HOTREMOVE - depends on SPARSEMEM_VMEMMAP select XARRAY_MULTI help -- cgit v1.2.3 From 092836fedd82cdafc6e2085c4b7a7878bfa3ca1d Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Thu, 9 Jul 2026 10:13:34 +0530 Subject: mm: standardize printing for pgtable entries Bad page map reporting currently stores page table entry values in an unsigned long long and prints them with fixed 64-bit-oriented format strings. This is inconsistent across call sites and does not work well for architectures where page table entry values are not naturally represented as 64-bit values, such as 32-bit or 128-bit entries. Introduce a common helper to convert raw page table entry values into a fixed-width hexadecimal string based on the actual entry size. Use it for bad page map reporting and for dumping the page table walk in __print_bad_page_map_pgtable(). Pass page table entry values to the reporting path as raw bytes together with their size, instead of forcing them through an unsigned long long. It keeps the printed output consistent and avoids truncation or misleading formatting for non-64-bit page table entries. Link: https://lore.kernel.org/20260709044334.1741263-1-anshuman.khandual@arm.com Signed-off-by: David Hildenbrand (Arm) Co-developed-by: Anshuman Khandual Signed-off-by: Anshuman Khandual Cc: Andriy Shevchenko Cc: David Hildenbrand Cc: Hugh Dickins Cc: Matthew Wilcox (Oracle) Cc: Ryan Roberts Signed-off-by: Andrew Morton --- mm/memory.c | 102 ++++++++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 79 insertions(+), 23 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index be743a9c6606..d2f14ba2261c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -519,9 +519,52 @@ static bool is_bad_page_map_ratelimited(void) return false; } +static void ptval_bytes_to_hex_str(char *buf, size_t buf_size, const void *entry, size_t entry_size) +{ + if (WARN_ON_ONCE(buf_size < entry_size * 2 + 1)) { + snprintf(buf, buf_size, "overflow"); + return; + } + + switch (entry_size) { + case sizeof(u32): + snprintf(buf, buf_size, "%08x", *(const u32 *)entry); + break; + case sizeof(u64): + snprintf(buf, buf_size, "%016llx", *(const u64 *)entry); + break; +#if defined(__SIZEOF_INT128__) + case sizeof(u128): + snprintf(buf, buf_size, "%016llx%016llx", + (unsigned long long)(*(const u128 *)entry >> 64), + (unsigned long long)*(const u128 *)entry); + break; +#endif + default: + snprintf(buf, buf_size, "unsupported"); + break; + } +} + +#define ptval_to_str(buf, val) \ + do { \ + auto __val = (val); \ + \ + ptval_bytes_to_hex_str((buf), sizeof(buf), &__val, sizeof(__val)); \ + } while (0) + +#if defined(__SIZEOF_INT128__) +#define PTVAL_STR_MAX (32 + 1) /* Max 128-bit value in hex + NUL */ +#else +#define PTVAL_STR_MAX (16 + 1) /* Max 64-bit value in hex + NUL */ +#endif + static void __print_bad_page_map_pgtable(struct mm_struct *mm, unsigned long addr) { - unsigned long long pgdv, p4dv, pudv, pmdv; + char pgd_str[PTVAL_STR_MAX]; + char p4d_str[PTVAL_STR_MAX]; + char pud_str[PTVAL_STR_MAX]; + char pmd_str[PTVAL_STR_MAX]; p4d_t p4d, *p4dp; pud_t pud, *pudp; pmd_t pmd, *pmdp; @@ -532,34 +575,34 @@ static void __print_bad_page_map_pgtable(struct mm_struct *mm, unsigned long add * see locking requirements for print_bad_page_map(). */ pgdp = pgd_offset(mm, addr); - pgdv = pgd_val(*pgdp); + ptval_to_str(pgd_str, pgd_val(*pgdp)); if (!pgd_present(*pgdp) || pgd_leaf(*pgdp)) { - pr_alert("pgd:%08llx\n", pgdv); + pr_alert("pgd:%s\n", pgd_str); return; } p4dp = p4d_offset(pgdp, addr); p4d = p4dp_get(p4dp); - p4dv = p4d_val(p4d); + ptval_to_str(p4d_str, p4d_val(p4d)); if (!p4d_present(p4d) || p4d_leaf(p4d)) { - pr_alert("pgd:%08llx p4d:%08llx\n", pgdv, p4dv); + pr_alert("pgd:%s p4d:%s\n", pgd_str, p4d_str); return; } pudp = pud_offset(p4dp, addr); pud = pudp_get(pudp); - pudv = pud_val(pud); + ptval_to_str(pud_str, pud_val(pud)); if (!pud_present(pud) || pud_leaf(pud)) { - pr_alert("pgd:%08llx p4d:%08llx pud:%08llx\n", pgdv, p4dv, pudv); + pr_alert("pgd:%s p4d:%s pud:%s\n", pgd_str, p4d_str, pud_str); return; } pmdp = pmd_offset(pudp, addr); pmd = pmdp_get(pmdp); - pmdv = pmd_val(pmd); + ptval_to_str(pmd_str, pmd_val(pmd)); /* * Dumping the PTE would be nice, but it's tricky with CONFIG_HIGHPTE, @@ -567,8 +610,7 @@ static void __print_bad_page_map_pgtable(struct mm_struct *mm, unsigned long add * doing another map would be bad. print_bad_page_map() should * already take care of printing the PTE. */ - pr_alert("pgd:%08llx p4d:%08llx pud:%08llx pmd:%08llx\n", pgdv, - p4dv, pudv, pmdv); + pr_alert("pgd:%s p4d:%s pud:%s pmd:%s\n", pgd_str, p4d_str, pud_str, pmd_str); } /* @@ -584,10 +626,11 @@ static void __print_bad_page_map_pgtable(struct mm_struct *mm, unsigned long add * page table lock. */ static void print_bad_page_map(struct vm_area_struct *vma, - unsigned long addr, unsigned long long entry, struct page *page, - enum pgtable_level level) + unsigned long addr, const void *entry, size_t entry_size, + struct page *page, enum pgtable_level level) { struct address_space *mapping; + char entry_str[PTVAL_STR_MAX]; pgoff_t index; if (is_bad_page_map_ratelimited()) @@ -596,8 +639,9 @@ static void print_bad_page_map(struct vm_area_struct *vma, mapping = vma->vm_file ? vma->vm_file->f_mapping : NULL; index = linear_page_index(vma, addr); - pr_alert("BUG: Bad page map in process %s %s:%08llx", current->comm, - pgtable_level_to_str(level), entry); + ptval_bytes_to_hex_str(entry_str, sizeof(entry_str), entry, entry_size); + pr_alert("BUG: Bad page map in process %s %s:%s", current->comm, + pgtable_level_to_str(level), entry_str); __print_bad_page_map_pgtable(vma->vm_mm, addr); if (page) dump_page(page, "bad page map"); @@ -627,8 +671,13 @@ static inline bool pgtable_level_has_pxx_special(enum pgtable_level level) } } -#define print_bad_pte(vma, addr, pte, page) \ - print_bad_page_map(vma, addr, pte_val(pte), page, PGTABLE_LEVEL_PTE) +static void print_bad_pte(struct vm_area_struct *vma, unsigned long addr, + pte_t pte, struct page *page) +{ + auto entry = pte_val(pte); + + print_bad_page_map(vma, addr, &entry, sizeof(entry), page, PGTABLE_LEVEL_PTE); +} /** * __vm_normal_page() - Get the "struct page" associated with a page table entry. @@ -636,8 +685,9 @@ static inline bool pgtable_level_has_pxx_special(enum pgtable_level level) * @addr: The address where the page table entry is mapped. * @pfn: The PFN stored in the page table entry. * @special: Whether the page table entry is marked "special". - * @level: The page table level for error reporting purposes only. * @entry: The page table entry value for error reporting purposes only. + * @entry_size: The size of @entry. + * @level: The page table level for error reporting purposes only. * * "Special" mappings do not wish to be associated with a "struct page" (either * it doesn't exist, or it exists but they don't want to touch it). In this @@ -697,7 +747,7 @@ static inline bool pgtable_level_has_pxx_special(enum pgtable_level level) */ static inline struct page *__vm_normal_page(struct vm_area_struct *vma, unsigned long addr, unsigned long pfn, bool special, - unsigned long long entry, enum pgtable_level level) + const void *entry, size_t entry_size, enum pgtable_level level) { if (pgtable_level_has_pxx_special(level)) { if (unlikely(special)) { @@ -710,7 +760,7 @@ static inline struct page *__vm_normal_page(struct vm_area_struct *vma, if (is_zero_pfn(pfn) || is_huge_zero_pfn(pfn)) return NULL; - print_bad_page_map(vma, addr, entry, NULL, level); + print_bad_page_map(vma, addr, entry, entry_size, NULL, level); return NULL; } /* @@ -741,7 +791,7 @@ static inline struct page *__vm_normal_page(struct vm_area_struct *vma, if (unlikely(pfn > highest_memmap_pfn)) { /* Corrupted page table entry. */ - print_bad_page_map(vma, addr, entry, NULL, level); + print_bad_page_map(vma, addr, entry, entry_size, NULL, level); return NULL; } /* @@ -767,8 +817,10 @@ static inline struct page *__vm_normal_page(struct vm_area_struct *vma, struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr, pte_t pte) { + auto entry = pte_val(pte); + return __vm_normal_page(vma, addr, pte_pfn(pte), pte_special(pte), - pte_val(pte), PGTABLE_LEVEL_PTE); + &entry, sizeof(entry), PGTABLE_LEVEL_PTE); } /** @@ -809,8 +861,10 @@ struct folio *vm_normal_folio(struct vm_area_struct *vma, unsigned long addr, struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t pmd) { + auto entry = pmd_val(pmd); + return __vm_normal_page(vma, addr, pmd_pfn(pmd), pmd_special(pmd), - pmd_val(pmd), PGTABLE_LEVEL_PMD); + &entry, sizeof(entry), PGTABLE_LEVEL_PMD); } /** @@ -850,8 +904,10 @@ struct folio *vm_normal_folio_pmd(struct vm_area_struct *vma, struct page *vm_normal_page_pud(struct vm_area_struct *vma, unsigned long addr, pud_t pud) { + auto entry = pud_val(pud); + return __vm_normal_page(vma, addr, pud_pfn(pud), pud_special(pud), - pud_val(pud), PGTABLE_LEVEL_PUD); + &entry, sizeof(entry), PGTABLE_LEVEL_PUD); } #endif -- cgit v1.2.3 From 0b4268ac77fa6e2c05fc8b90441b0832b211385d Mon Sep 17 00:00:00 2001 From: Kaitao Cheng Date: Sun, 12 Jul 2026 17:33:26 +0800 Subject: mm/kconfig: drop redundant dependency wrappers Some mm Kconfig entries repeat dependencies that are already expressed by their surrounding blocks or menus. The zsmalloc allocator options menu already depends on ZSMALLOC, so the outer if ZSMALLOC block does not add any extra constraint. MEMORY_HOTREMOVE and MHP_MEMMAP_ON_MEMORY are both inside the if MEMORY_HOTPLUG block, so their local depends on MEMORY_HOTPLUG entries are redundant. PTE_MARKER_UFFD_WP is the only entry under if USERFAULTFD. Move the USERFAULTFD dependency into the symbol itself and combine it with the architecture support dependency. This keeps the same visibility and defaults while avoiding duplicate dependency expressions. Link: https://lore.kernel.org/20260712093326.8313-1-kaitao.cheng@linux.dev Signed-off-by: Kaitao Cheng Suggested-by: Julian Braha Reviewed-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- mm/Kconfig | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/mm/Kconfig b/mm/Kconfig index d28dde592de4..060190e12bce 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -125,8 +125,6 @@ config ZSWAP_COMPRESSOR_DEFAULT config ZSMALLOC tristate -if ZSMALLOC - menu "Zsmalloc allocator options" depends on ZSMALLOC @@ -161,8 +159,6 @@ config ZSMALLOC_CHAIN_SIZE endmenu -endif - menu "Slab allocator options" config SLUB @@ -583,12 +579,10 @@ endchoice config MEMORY_HOTREMOVE bool "Allow for memory hot remove" - depends on MEMORY_HOTPLUG select MIGRATION config MHP_MEMMAP_ON_MEMORY def_bool y - depends on MEMORY_HOTPLUG depends on ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE endif # MEMORY_HOTPLUG @@ -1393,17 +1387,15 @@ menuconfig USERFAULTFD Enable the userfaultfd() system call that allows to intercept and handle page faults in userland. -if USERFAULTFD config PTE_MARKER_UFFD_WP bool "Userfaultfd write protection support for shmem/hugetlbfs" default y - depends on HAVE_ARCH_USERFAULTFD_WP + depends on USERFAULTFD && HAVE_ARCH_USERFAULTFD_WP help Allows to create marker PTEs for userfaultfd write protection purposes. It is required to enable userfaultfd write protection on file-backed memory types like shmem and hugetlbfs. -endif # USERFAULTFD # multi-gen LRU { config LRU_GEN -- cgit v1.2.3 From 0bd14001eb264247d565a5a44a71675df273640d Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:18 +0100 Subject: mm/vma: introduce VMA anon page offset field and add helpers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Patch series "mm/rmap: index MAP_PRIVATE file-backed folios by anonymous pgoff", v5. In memory management we've managed to manufacture a great deal of confusion around the concept of anonymous memory. We have: 1. 'Pure anon' memory - anonymous VMAs whose folios are anonymous and swap-backed (thus for reclaim purposes, treated as anonymous). These are simple enough. 2. shmem - file-backed VMAs, file-backed folios (from rmap perspective) so present in the page cache and mapped by an address_space object, but whose folios are also swap-backed (thus treated as anonymous for reclaim purposes). 3. MAP_PRIVATE-mapped /dev/zero - a strange beast whose VMAs have vma->vm_file set, but which clears vma->vm_ops to satisfy vma_is_anonymous(), resulting in VMAs that were mmap()'d referencing a file, but are in every other sense anonymous, including the folios. 4. Other MAP_PRIVATE-file backed mappings - These possess file-backed VMAs and have file-backed folios until CoW'd, at which point those CoW'd folios are anonymous. This series fixes issue 3. In order for us to traverse VMAs using the reverse mapping, we require two fields - folio->mapping and folio->index. The first tells the rmap code where to look for VMAs, and the second tells it at which offset the folio starts within the referenced object. For anonymous folios, folio->mapping points at an anon_vma object. For file-backed folios, it points at an address_space. And: * For file-backed folios folio->index is simply the page offset of the start of the folio within the file. * For anonymous folios belonging to pure anon mappings, folio->index is equal to the anonymous page offset of the folio. * For anonymous folios belonging to file-backed mappings (i.e. CoW'd folios of a MAP_PRIVATE file-backed mapping), folio->index is equal to the file page offset. This series establishes a new anonymous page offset property of VMAs to allow us to map anonymous folios at their anonymous page offset, consistent with pure anon. The purpose of doing so is to lay the foundations for the scalable CoW work. This is necessary because scalable CoW looks in the maple tree for the VMA located at folio->index << PAGE_SHIFT, before falling back to looking up tracked remaps if necessary. The MAP_PRIVATE file-backed case means that folio indices will very often conflict with one another and this remap tracking becomes substantially more contended, and of course the fast path can never be used. This also makes it possible, in future, to unshare anonymously mapped folios with deep fork hierarchies on remap, eliminating the need for remap tracking in the vast majority of cases. Similar to page offset of pure anonymous VMAs, we update the anonymous page offset of unfaulted file-backed VMAs on remap, but do not once CoW'd (i.e. vma->anon_vma is non-NULL). Overall, there is little impact on mergeability, which remains exactly the same for pure anonymous and shared file-backed mappings, with the only impact being on MAP_PRIVATE-mapped file-backed mappings, which must now match on anonymous page offset as well as file page offset to be merged. To fail to merge like this would require CoW'ing the mapping, then finding another VMA with identical file and compatible page offset to remap next to. This is therefore very much an edge case that should have very little impact (and which scalable CoW may very well address in any case). This patch (of 16): Establish fields in vm_area_struct to store the anonymous page offset of VMAs. Initially, the anonymous page offset of a VMA is vma->vm_start >> PAGE_SHIFT. When a VMA is remapped to new_address its anonymous page offset is either updated to new_address >> PAGE_SHIFT if unfaulted or, if faulted, remains equal to the anonymous page offset it had when first faulted. Currently, anonymous folios belonging to CoW'd MAP_PRIVATE-mapped file-backed VMAs are tracked by their file offsets. By adding anonymous offset as a property of VMAs, we can now track them by their anonymous page offset instead. By tracking this, we provide the means by which to eliminate this inconsistency, and more importantly lay the foundations for future work for the scalable CoW anonymous rmap rework. This patch simply adds the fields and some simple helpers. Subsequent patches will update mm code to make use of these fields correctly. The fields chosen are packed in the VMA such that, for 64-bit kernel builds, no additional space is taken up. The first field is present on cacheline 0 containing key VMA fields, and the second on cacheline 3, which contains file-backed reverse mapping fields. Given the relative time spent accessing reverse mapping fields as well as updating them, there shouldn't be any performance impact here from false sharing. Update the VMA userland tests to account for this change. No callsites are updated yet, so no functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-0-c21581c0c3c8@kernel.org Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-1-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Reviewed-by: Gregory Price (Meta) Reviewed-by: Xu Xin Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- include/linux/mm.h | 59 +++++++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 12 +++++++++ mm/vma.h | 14 ++++++++++ mm/vma_init.c | 1 + tools/testing/vma/include/dup.h | 26 ++++++++++++++++++ 5 files changed, 112 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 87feaa5a2b78..df78847f5f07 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4393,6 +4393,65 @@ static inline pgoff_t vma_last_pgoff(const struct vm_area_struct *vma) return vma_end_pgoff(vma) - 1; } +/** + * vma_start_anon_pgoff() - Get the anonymous page offset of the start of @vma + * @vma: The VMA whose anonymous page offset is required. + * + * If unfaulted, then this is vma->vm_start >> PAGE_SHIFT, if faulted then the + * anonymous page offset at the time of first fault. + * + * If the VMA is anonymous, this returns the same value as vma_start_pgoff(). + * + * This value is used for tracking MAP_PRIVATE file-backed mappings by their + * anonymous page offset. + * + * Returns: The anonymous page offset of the start of @vma. + */ +static inline pgoff_t vma_start_anon_pgoff(const struct vm_area_struct *vma) +{ + pgoff_t pgoff = 0; + +#ifdef CONFIG_64BIT + pgoff += vma->__vm_anon_pgoff_hi; + pgoff <<= 32; +#endif + pgoff += vma->__vm_anon_pgoff_lo; + return pgoff; +} + +/** + * vma_end_anon_pgoff() - Get the anonymous page offset of the exclusive end of + * @vma. + * @vma: The VMA whose end anonymous page offset is required. + * + * This returns the anonymous exclusive end page offset of @vma, which is useful + * for expressing page offset ranges. + * + * See the description of vma_start_anon_pgoff() for a description of VMA + * anonymous page offsets. + * + * Returns: The exclusive end anonymous page offset of @vma. + */ +static inline pgoff_t vma_end_anon_pgoff(const struct vm_area_struct *vma) +{ + return vma_start_anon_pgoff(vma) + vma_pages(vma); +} + +/** + * vma_last_anon_pgoff() - Get the anonymous page offset of the last page in + * @vma. + * @vma: The VMA whose last anonymous page offset is required. + * + * See the description of vma_start_anon_pgoff() for a description of VMA + * anonymous page offsets. + * + * Returns: The last anonymous page offset of @vma. + */ +static inline pgoff_t vma_last_anon_pgoff(const struct vm_area_struct *vma) +{ + return vma_end_anon_pgoff(vma) - 1; +} + static inline unsigned long vma_desc_size(const struct vm_area_desc *desc) { return desc->end - desc->start; diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 939b5ea8c9e0..ebf0d912be7d 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -967,6 +967,11 @@ struct vm_area_struct { */ unsigned int vm_lock_seq; #endif + /* + * Low 32-bits of anonymous page offset. + * See vma_start_anon_pgoff() comment for details. + */ + unsigned int __vm_anon_pgoff_lo; /* * A file's MAP_PRIVATE vma can be in both i_mmap tree and anon_vma * list, after a COW of one of the file pages. A MAP_SHARED vma @@ -1041,6 +1046,13 @@ struct vm_area_struct { #ifdef CONFIG_DEBUG_LOCK_ALLOC struct lockdep_map vmlock_dep_map; #endif +#endif +#ifdef CONFIG_64BIT + /* + * High 32-bits of anonymous page offset. + * See vma_start_anon_pgoff() comment for details. + */ + unsigned int __vm_anon_pgoff_hi; #endif /* * For areas with an address space and backing store, diff --git a/mm/vma.h b/mm/vma.h index 0bc7d521e976..54ed7c744e3b 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -283,6 +283,20 @@ static inline void vma_set_pgoff(struct vm_area_struct *vma, pgoff_t pgoff) vma->vm_pgoff = pgoff; } +static inline void __vma_set_anon_pgoff(struct vm_area_struct *vma, pgoff_t pgoff) +{ +#ifdef CONFIG_64BIT + vma->__vm_anon_pgoff_hi = pgoff >> 32; +#endif + vma->__vm_anon_pgoff_lo = pgoff & GENMASK(31, 0); +} + +static inline void vma_set_anon_pgoff(struct vm_area_struct *vma, pgoff_t pgoff) +{ + vma_assert_can_modify(vma); + __vma_set_anon_pgoff(vma, pgoff); +} + static inline void vma_add_pgoff(struct vm_area_struct *vma, pgoff_t delta) { vma_assert_can_modify(vma); diff --git a/mm/vma_init.c b/mm/vma_init.c index 715feee283f0..baa7e82f47e3 100644 --- a/mm/vma_init.c +++ b/mm/vma_init.c @@ -51,6 +51,7 @@ static void vm_area_init_from(const struct vm_area_struct *src, dest->vm_end = src->vm_end; dest->anon_vma = src->anon_vma; dest->vm_pgoff = vma_start_pgoff(src); + __vma_set_anon_pgoff(dest, vma_start_anon_pgoff(src)); dest->vm_file = src->vm_file; dest->vm_private_data = src->vm_private_data; vm_flags_init(dest, src->vm_flags); diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index cdeb53bbdd1b..17f94e5de569 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -577,6 +577,7 @@ struct vm_area_struct { */ unsigned int vm_lock_seq; #endif + unsigned int __vm_anon_pgoff_lo; /* * A file's MAP_PRIVATE vma can be in both i_mmap tree and anon_vma @@ -612,6 +613,9 @@ struct vm_area_struct { #ifdef CONFIG_PER_VMA_LOCK /* Unstable RCU readers are allowed to read this. */ refcount_t vm_refcnt; +#endif +#ifdef CONFIG_64BIT + unsigned int __vm_anon_pgoff_hi; #endif /* * For areas with an address space and backing store, @@ -1320,6 +1324,28 @@ static inline pgoff_t vma_end_pgoff(const struct vm_area_struct *vma) return vma_start_pgoff(vma) + vma_pages(vma); } +static inline pgoff_t vma_start_anon_pgoff(const struct vm_area_struct *vma) +{ + pgoff_t pgoff = 0; + +#ifdef CONFIG_64BIT + pgoff += vma->__vm_anon_pgoff_hi; + pgoff <<= 32; +#endif + pgoff += vma->__vm_anon_pgoff_lo; + return pgoff; +} + +static inline pgoff_t vma_end_anon_pgoff(const struct vm_area_struct *vma) +{ + return vma_start_anon_pgoff(vma) + vma_pages(vma); +} + +static inline pgoff_t vma_last_anon_pgoff(const struct vm_area_struct *vma) +{ + return vma_end_anon_pgoff(vma) - 1; +} + static inline int vfs_mmap_prepare(struct file *file, struct vm_area_desc *desc) { return file->f_op->mmap_prepare(desc); -- cgit v1.2.3 From 51943a18ad4bd6ff8baea2da7b8cce2f86f1a959 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:19 +0100 Subject: mm: provide vma_[flags_]is_cow_mapping() and remove is_cow_mapping() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All remaining callers of is_cow_mapping() are invoking it in the form of is_cow_mapping(vma->vm_flags) or an indirected version of this. Therefore, provide a helper - vma_is_cow_mapping() to directly test the VMA. Additionally provide a new helper vma_flags_is_cow_mapping() which performs the check using the new vma_flags_t type, and share this logic between vma_is_cow_mapping() and vma_desc_is_cow_mapping(). With these changes, no callers of is_cow_mapping() remain, so remove it. Also update the userland VMA tests to reflect the change. No functional change intended. [akpm@linux-foundation.org: fix kerneldoc comment typo, per Lorenzo] Link: https://lore.kernel.org/aob1goSSPH6sTN9y@gremlin Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-2-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Gregory Price (Meta) Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- arch/s390/mm/gmap_helpers.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 4 +- drivers/gpu/drm/drm_gem_shmem_helper.c | 2 +- drivers/gpu/drm/panthor/panthor_gem.c | 2 +- drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +- drivers/gpu/drm/xe/xe_device.c | 2 +- fs/proc/task_mmu.c | 2 +- include/linux/mm.h | 71 ++++++++++++++++++++++++++++++--- kernel/events/uprobes.c | 2 +- mm/gup.c | 2 +- mm/huge_memory.c | 8 ++-- mm/hugetlb.c | 2 +- mm/internal.h | 2 +- mm/memory.c | 25 ++++++------ mm/mempolicy.c | 2 +- tools/testing/vma/include/dup.h | 11 +++++ 16 files changed, 105 insertions(+), 36 deletions(-) diff --git a/arch/s390/mm/gmap_helpers.c b/arch/s390/mm/gmap_helpers.c index 4bf7c9012feb..cd5fded159c0 100644 --- a/arch/s390/mm/gmap_helpers.c +++ b/arch/s390/mm/gmap_helpers.c @@ -200,7 +200,7 @@ static int find_zeropage_pte_entry(pte_t *pte, unsigned long addr, * currently only works in COW mappings, which is also where * mm_forbids_zeropage() is checked. */ - if (!is_cow_mapping(walk->vma->vm_flags)) + if (!vma_is_cow_mapping(walk->vma)) return -EFAULT; *found_addr = addr; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c index 6a0699746fbc..0c7309080a7a 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c @@ -377,9 +377,9 @@ static int amdgpu_gem_object_mmap(struct drm_gem_object *obj, struct vm_area_str /* Workaround for Thunk bug creating PROT_NONE,MAP_PRIVATE mappings * for debugger access to invisible VRAM. Should have used MAP_SHARED * instead. Clearing VM_MAYWRITE prevents the mapping from ever - * becoming writable and makes is_cow_mapping(vm_flags) false. + * becoming writable and makes vma_is_cow_mapping(vma) false. */ - if (is_cow_mapping(vma->vm_flags) && + if (vma_is_cow_mapping(vma) && !(vma->vm_flags & VM_ACCESS_FLAGS)) vm_flags_clear(vma, VM_MAYWRITE); diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c b/drivers/gpu/drm/drm_gem_shmem_helper.c index 06d019d51d3e..177d0e0b9334 100644 --- a/drivers/gpu/drm/drm_gem_shmem_helper.c +++ b/drivers/gpu/drm/drm_gem_shmem_helper.c @@ -753,7 +753,7 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_struct return ret; } - if (is_cow_mapping(vma->vm_flags)) + if (vma_is_cow_mapping(vma)) return -EINVAL; dma_resv_lock(shmem->base.resv, NULL); diff --git a/drivers/gpu/drm/panthor/panthor_gem.c b/drivers/gpu/drm/panthor/panthor_gem.c index 770556353968..d2eec46f7abe 100644 --- a/drivers/gpu/drm/panthor/panthor_gem.c +++ b/drivers/gpu/drm/panthor/panthor_gem.c @@ -761,7 +761,7 @@ static int panthor_gem_mmap(struct drm_gem_object *obj, struct vm_area_struct *v return ret; } - if (is_cow_mapping(vma->vm_flags)) + if (vma_is_cow_mapping(vma)) return -EINVAL; if (!refcount_inc_not_zero(&bo->cmap.mmap_count)) { diff --git a/drivers/gpu/drm/ttm/ttm_bo_vm.c b/drivers/gpu/drm/ttm/ttm_bo_vm.c index 88babf435ac2..872bf444b1f0 100644 --- a/drivers/gpu/drm/ttm/ttm_bo_vm.c +++ b/drivers/gpu/drm/ttm/ttm_bo_vm.c @@ -489,7 +489,7 @@ static const struct vm_operations_struct ttm_bo_vm_ops = { int ttm_bo_mmap_obj(struct vm_area_struct *vma, struct ttm_buffer_object *bo) { /* Enforce no COW since would have really strange behavior with it. */ - if (is_cow_mapping(vma->vm_flags)) + if (vma_is_cow_mapping(vma)) return -EINVAL; drm_gem_object_get(&bo->base); diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c index 9d119c95a569..de5fdf49d729 100644 --- a/drivers/gpu/drm/xe/xe_device.c +++ b/drivers/gpu/drm/xe/xe_device.c @@ -330,7 +330,7 @@ static int xe_pci_barrier_mmap(struct file *filp, if (vma->vm_end - vma->vm_start > SZ_4K) return -EINVAL; - if (is_cow_mapping(vma->vm_flags)) + if (vma_is_cow_mapping(vma)) return -EINVAL; if (vma->vm_flags & (VM_READ | VM_EXEC)) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 817e3e0f9194..5c54aebe2118 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1693,7 +1693,7 @@ static inline bool pte_is_pinned(struct vm_area_struct *vma, unsigned long addr, if (!pte_write(pte)) return false; - if (!is_cow_mapping(vma->vm_flags)) + if (!vma_is_cow_mapping(vma)) return false; if (likely(!mm_flags_test(MMF_HAS_PINNED, vma->vm_mm))) return false; diff --git a/include/linux/mm.h b/include/linux/mm.h index df78847f5f07..20361b4344ea 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2271,17 +2271,76 @@ void unpin_user_pages(struct page **pages, unsigned long npages); void unpin_user_folio(struct folio *folio, unsigned long npages); void unpin_folios(struct folio **folios, unsigned long nfolios); -static inline bool is_cow_mapping(vm_flags_t flags) +/** + * vma_flags_is_cow_mapping() - Do these VMA flags imply a CoW mapping? + * @flags: The VMA flags to check. + * + * Mappings which could be CoW'd (subject to Copy-On-Write faults) are + * described as CoW mappings. + * + * All mappings backed by anonymous folios (all anonymous mappings and most + * MAP_PRIVATE-file backed ranges) are CoW mappings. + * + * All other mappings (including all MAP_SHARED mappings) are non-CoW. + * + * The criteria are !VMA_SHARED_BIT, VMA_MAYWRITE_BIT. + * + * VMA_MAYWRITE_BIT is checked instead of VMA_WRITE_BIT to account for both + * future mprotect() calls which can render a read-only mapping writable, and + * GUP with FOLL_FORCE (e.g. ptrace) which can CoW a read-only mapping. + * + * - No anonymous mapping can ever clear VMA_MAYWRITE_BIT. + * + * - Writes to anonymous mappings do not immediately result in CoW faults but + * may do so after the process is forked or if a read is followed by a + * write. + * + * - Writes to MAP_PRIVATE file-backed mappings result in CoW faults and may + * do so again after fork. + * + * - MAP_SHARED mappings of a file opened read-only are transformed into + * VMA_MAYSHARE_BIT, !VMA_SHARED_BIT, !VMA_MAYWRITE_BIT mappings, so remain + * non-CoW. + * + * - Drivers may clear VMA_MAYWRITE_BIT but do so at mmap() time and cannot + * mark themselves anonymous. Having cleared this flag it is not valid for + * them to leave the VMA_WRITE_BIT flag set. + * + * As a consequence, the anonymous reverse mapping only tracks CoW mappings. + * + * Returns: true if the flags indicate a CoW mapping, otherwise false. + */ +static inline bool vma_flags_is_cow_mapping(const vma_flags_t *flags) +{ + return vma_flags_test(flags, VMA_MAYWRITE_BIT) && + !vma_flags_test(flags, VMA_SHARED_BIT); +} + +/** + * vma_is_cow_mapping() - Is this VMA a CoW mapping? + * @vma: The VMA to check. + * + * See vma_flags_is_cow_mapping() for details. + * + * Returns: true if the VMA is a CoW mapping, otherwise false. + */ +static inline bool vma_is_cow_mapping(const struct vm_area_struct *vma) { - return (flags & (VM_SHARED | VM_MAYWRITE)) == VM_MAYWRITE; + return vma_flags_is_cow_mapping(&vma->flags); } +/** + * vma_desc_is_cow_mapping() - Is this VMA descriptor a CoW mapping? + * @desc: The VMA descriptor to check. + * + * See vma_flags_is_cow_mapping() for details. + * + * Returns: true if the VMA descriptor describes a CoW mapping, otherwise + * false. + */ static inline bool vma_desc_is_cow_mapping(struct vm_area_desc *desc) { - const vma_flags_t *flags = &desc->vma_flags; - - return vma_flags_test(flags, VMA_MAYWRITE_BIT) && - !vma_flags_test(flags, VMA_SHARED_BIT); + return vma_flags_is_cow_mapping(&desc->vma_flags); } #ifndef CONFIG_MMU diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index ae2f3b9f8d50..eb0d11092fb3 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -513,7 +513,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, uprobe = container_of(auprobe, struct uprobe, arch); - if (WARN_ON_ONCE(!is_cow_mapping(vma->vm_flags))) + if (WARN_ON_ONCE(!vma_is_cow_mapping(vma))) return -EINVAL; /* diff --git a/mm/gup.c b/mm/gup.c index 99902c15703b..8ea3de60e82d 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1236,7 +1236,7 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags) * Anon pages in shared mappings are surprising: now * just reject it. */ - if (!is_cow_mapping(vm_flags)) + if (!vma_is_cow_mapping(vma)) return -EFAULT; } } else if (!(vm_flags & VM_READ)) { diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 644d6905b49c..ff13b57d9d56 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1681,7 +1681,7 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, unsigned long pfn, BUG_ON(!(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP))); BUG_ON((vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)) == (VM_PFNMAP|VM_MIXEDMAP)); - BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags)); + BUG_ON((vma->vm_flags & VM_PFNMAP) && vma_is_cow_mapping(vma)); pfnmap_setup_cachemode_pfn(pfn, &pgprot); @@ -1789,7 +1789,7 @@ vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, unsigned long pfn, BUG_ON(!(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP))); BUG_ON((vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)) == (VM_PFNMAP|VM_MIXEDMAP)); - BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags)); + BUG_ON((vma->vm_flags & VM_PFNMAP) && vma_is_cow_mapping(vma)); pfnmap_setup_cachemode_pfn(pfn, &pgprot); @@ -1931,7 +1931,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, * applied special bit, or we made the PRIVATE mapping be * able to wrongly write to the backend MMIO. */ - VM_WARN_ON_ONCE(is_cow_mapping(src_vma->vm_flags) && pmd_write(pmd)); + VM_WARN_ON_ONCE(vma_is_cow_mapping(src_vma) && pmd_write(pmd)); goto set_pmd; } @@ -2052,7 +2052,7 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, * TODO: once we support anonymous pages, use * folio_try_dup_anon_rmap_*() and split if duplicating fails. */ - if (is_cow_mapping(vma->vm_flags) && pud_write(pud)) { + if (vma_is_cow_mapping(vma) && pud_write(pud)) { pudp_set_wrprotect(src_mm, addr, src_pud); pud = pud_wrprotect(pud); } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 79e5c3b3e850..49bf325325c0 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4898,7 +4898,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, pte_t *src_pte, *dst_pte, entry; struct folio *pte_folio; unsigned long addr; - bool cow = is_cow_mapping(src_vma->vm_flags); + bool cow = vma_is_cow_mapping(src_vma); struct hstate *h = hstate_vma(src_vma); unsigned long sz = huge_page_size(h); unsigned long npages = pages_per_huge_page(h); diff --git a/mm/internal.h b/mm/internal.h index 68db5abd0a4c..a75e7641ef49 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1353,7 +1353,7 @@ static inline bool gup_must_unshare(struct vm_area_struct *vma, * ... because we only care about writable private ("COW") * mappings where we have to break COW early. */ - return is_cow_mapping(vma->vm_flags); + return vma_is_cow_mapping(vma); } /* Paired with a memory barrier in folio_try_share_anon_rmap_*(). */ diff --git a/mm/memory.c b/mm/memory.c index d2f14ba2261c..396d7b9059e6 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -780,7 +780,7 @@ static inline struct page *__vm_normal_page(struct vm_area_struct *vma, /* Only CoW'ed anon folios are "normal". */ if (pfn == index) return NULL; - if (!is_cow_mapping(vma->vm_flags)) + if (!vma_is_cow_mapping(vma)) return NULL; } } @@ -1002,7 +1002,6 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, pte_t *dst_pte, pte_t *src_pte, struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, unsigned long addr, int *rss) { - vm_flags_t vm_flags = dst_vma->vm_flags; pte_t orig_pte = ptep_get(src_pte); softleaf_t entry = softleaf_from_pte(orig_pte); pte_t pte = orig_pte; @@ -1026,7 +1025,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, rss[mm_counter(folio)]++; if (!softleaf_is_migration_read(entry) && - is_cow_mapping(vm_flags)) { + vma_is_cow_mapping(dst_vma)) { /* * COW mappings require pages in both parent and child * to be set to read. A previously exclusive entry is @@ -1067,7 +1066,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, * save and restore device driver state). */ if (softleaf_is_device_private_write(entry) && - is_cow_mapping(vm_flags)) { + vma_is_cow_mapping(dst_vma)) { entry = make_readable_device_private_entry( swp_offset(entry)); pte = swp_entry_to_pte(entry); @@ -1082,7 +1081,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, * exclusive entries currently only support private writable * (ie. COW) mappings. */ - VM_BUG_ON(!is_cow_mapping(src_vma->vm_flags)); + VM_BUG_ON(!vma_is_cow_mapping(src_vma)); if (try_restore_exclusive_pte(src_vma, addr, src_pte, orig_pte)) return -EBUSY; return -ENOENT; @@ -1181,7 +1180,7 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma, } /* If it's a COW mapping, write protect it both processes. */ - if (is_cow_mapping(src_vma->vm_flags) && writable) { + if (vma_is_cow_mapping(src_vma) && writable) { wrprotect_ptes(src_mm, addr, src_pte, nr); pte = pte_wrprotect(pte); } @@ -1602,9 +1601,9 @@ copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) * We need to invalidate the secondary MMU mappings only when * there could be a permission downgrade on the ptes of the * parent mm. And a permission downgrade will only happen if - * is_cow_mapping() returns true. + * vma_is_cow_mapping() returns true. */ - is_cow = is_cow_mapping(src_vma->vm_flags); + is_cow = vma_is_cow_mapping(src_vma); if (is_cow) { mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_PAGE, @@ -2437,7 +2436,7 @@ static bool vm_mixed_zeropage_allowed(struct vm_area_struct *vma) if (mm_forbids_zeropage(vma->vm_mm)) return false; /* zeropages in COW mappings are common and unproblematic. */ - if (is_cow_mapping(vma->vm_flags)) + if (vma_is_cow_mapping(vma)) return true; /* Mappings that do not allow for writable PTEs are unproblematic. */ if (!(vma->vm_flags & (VM_WRITE | VM_MAYWRITE))) @@ -2888,7 +2887,7 @@ vm_fault_t vmf_insert_pfn_prot(struct vm_area_struct *vma, unsigned long addr, BUG_ON(!(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP))); BUG_ON((vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)) == (VM_PFNMAP|VM_MIXEDMAP)); - BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags)); + BUG_ON((vma->vm_flags & VM_PFNMAP) && vma_is_cow_mapping(vma)); BUG_ON((vma->vm_flags & VM_MIXEDMAP) && pfn_valid(pfn)); if (addr < vma->vm_start || addr >= vma->vm_end) @@ -3300,7 +3299,7 @@ static int remap_pfn_range_prepare_vma(struct vm_area_struct *vma, unsigned long size) { const unsigned long end = addr + PAGE_ALIGN(size); - const bool is_cow = is_cow_mapping(vma->vm_flags); + const bool is_cow = vma_is_cow_mapping(vma); int err; err = get_remap_pgoff(is_cow, addr, end, vma->vm_start, vma->vm_end, @@ -6800,7 +6799,7 @@ static vm_fault_t sanitize_fault_flags(struct vm_area_struct *vma, * FAULT_FLAG_UNSHARE only applies to COW mappings. Let's * just treat it like an ordinary read-fault otherwise. */ - if (!is_cow_mapping(vma->vm_flags)) + if (!vma_is_cow_mapping(vma)) *flags &= ~FAULT_FLAG_UNSHARE; } else if (*flags & FAULT_FLAG_WRITE) { /* Write faults on read-only mappings are impossible ... */ @@ -6808,7 +6807,7 @@ static vm_fault_t sanitize_fault_flags(struct vm_area_struct *vma, return VM_FAULT_SIGSEGV; /* ... and FOLL_FORCE only applies to COW mappings. */ if (WARN_ON_ONCE(!(vma->vm_flags & VM_WRITE) && - !is_cow_mapping(vma->vm_flags))) + !vma_is_cow_mapping(vma))) return VM_FAULT_SIGSEGV; } #ifdef CONFIG_PER_VMA_LOCK diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 5720f7f54d94..3498a5651d50 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -844,7 +844,7 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma, return false; /* Also skip shared copy-on-write folios */ - if (is_cow_mapping(vma->vm_flags) && folio_maybe_mapped_shared(folio)) + if (vma_is_cow_mapping(vma) && folio_maybe_mapped_shared(folio)) return false; /* Folios are pinned and can't be migrated */ diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index 17f94e5de569..af2fd3f607b5 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -1162,6 +1162,17 @@ static inline bool vma_is_shared_maywrite(struct vm_area_struct *vma) return is_shared_maywrite(&vma->flags); } +static inline bool vma_flags_is_cow_mapping(const vma_flags_t *flags) +{ + return vma_flags_test(flags, VMA_MAYWRITE_BIT) && + !vma_flags_test(flags, VMA_SHARED_BIT); +} + +static inline bool vma_is_cow_mapping(const struct vm_area_struct *vma) +{ + return vma_flags_is_cow_mapping(&vma->flags); +} + static inline struct vm_area_struct *vma_next(struct vma_iterator *vmi) { /* -- cgit v1.2.3 From 7e6543d1f939eaaca008c13395e52cbd07605cb0 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:20 +0100 Subject: mm: introduce linear_anon_page_index() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This function provides the anonymous equivalent of linear_page_index(), instead offsetting based on the anonymous page offset of the VMA. It is valid only for anonymous or MAP_PRIVATE file-backed mappings, in other words CoW mappings. For pure anon VMAs, this will be equal to linear_page_index(). Assert that both of these invariants are true in linear_anon_page_index() and implement the algorithm in __linear_anon_page_index(). Note that MAP_PRIVATE-/dev/zero mappings will satisfy vma_is_anonymous() but not fulfill this invariant, so when asserting this we check vma->vm_file to account for this. We do not update callsites yet, so no functional change intended. Also const-ify vma_is_anonymous() to make it compatible with the const-ified linear_anon_page_index(). While we're here, update linear_page_index() to be more succinct. VMA userland tests are also updated accordingly. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-3-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Reviewed-by: Gregory Price (Meta) Acked-by: David Hildenbrand (Arm) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- include/linux/mm.h | 2 +- include/linux/pagemap.h | 40 +++++++++++++++++++++++++++++++++++++--- tools/testing/vma/include/dup.h | 25 ++++++++++++++++++++++++- 3 files changed, 62 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 20361b4344ea..dd09c438fa23 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1556,7 +1556,7 @@ static inline void vma_desc_set_anonymous(struct vm_area_desc *desc) desc->vm_ops = NULL; } -static inline bool vma_is_anonymous(struct vm_area_struct *vma) +static inline bool vma_is_anonymous(const struct vm_area_struct *vma) { return !vma->vm_ops; } diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index c6fc783aaee5..0adfa6605653 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -1094,10 +1094,44 @@ static inline pgoff_t linear_page_delta(const struct vm_area_struct *vma, static inline pgoff_t linear_page_index(const struct vm_area_struct *vma, const unsigned long address) { - pgoff_t pgoff; + return linear_page_delta(vma, address) + vma_start_pgoff(vma); +} + +static inline pgoff_t __linear_anon_page_index(const struct vm_area_struct *vma, + const unsigned long address) +{ + return linear_page_delta(vma, address) + vma_start_anon_pgoff(vma); +} + +/** + * linear_anon_page_index() - Determine the absolute anonymous page offset of + * @address within @vma. + * @vma: An anonymous or MAP_PRIVATE file-backed VMA in which @address resides. + * @address: The address whose absolute page offset is required. + * + * This returns the anonymous page offset of @address, which is the page offset + * the address possessed at the time the VMA was first faulted. + * + * For anonymous mappings, this returns the same value as linear_page_index(). + * + * For MAP_PRIVATE file-backed mappings, this returns the anonymous page offset + * of @address, which is the page offset the address possessed at the time the + * VMA was first faulted. + * + * It is not valid to call this function for shared file-backed mappings. + * + * Returns: The absolute anonymous page offset of @address within @vma. + */ +static inline pgoff_t linear_anon_page_index(const struct vm_area_struct *vma, + const unsigned long address) +{ + const pgoff_t pgoff = __linear_anon_page_index(vma, address); + + VM_WARN_ON_ONCE(!vma_is_cow_mapping(vma)); + /* Account for MAP_PRIVATE-/dev/zero which is only semi-anonymous. */ + if (vma_is_anonymous(vma) && !vma->vm_file) + VM_WARN_ON_ONCE(pgoff != linear_page_index(vma, address)); - pgoff = linear_page_delta(vma, address); - pgoff += vma_start_pgoff(vma); return pgoff; } diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index af2fd3f607b5..4655aecffaf3 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -1428,7 +1428,7 @@ static inline void vma_iter_set(struct vma_iterator *vmi, unsigned long addr) mas_set(&vmi->mas, addr); } -static inline bool vma_is_anonymous(struct vm_area_struct *vma) +static inline bool vma_is_anonymous(const struct vm_area_struct *vma) { return !vma->vm_ops; } @@ -1621,3 +1621,26 @@ static inline pgprot_t vma_get_page_prot(const struct vm_area_struct *vma) { return vma_flags_to_page_prot(vma->flags); } + +static inline pgoff_t __linear_anon_page_index(const struct vm_area_struct *vma, + const unsigned long address) +{ + pgoff_t pgoff; + + pgoff = linear_page_delta(vma, address); + pgoff += vma_start_anon_pgoff(vma); + return pgoff; +} + +static inline pgoff_t linear_anon_page_index(const struct vm_area_struct *vma, + const unsigned long address) +{ + const pgoff_t pgoff = __linear_anon_page_index(vma, address); + + VM_WARN_ON_ONCE(!vma_is_cow_mapping(vma)); + /* Account for MAP_PRIVATE-/dev/zero which is only semi-anonymous. */ + if (vma_is_anonymous(vma) && !vma->vm_file) + VM_WARN_ON_ONCE(pgoff != linear_page_index(vma, address)); + + return pgoff; +} -- cgit v1.2.3 From c02fe674feaf3529b4d8c808c363aa1bcf955979 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:21 +0100 Subject: mm: abstract vma_address() and introduce vma_anon_address() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduce __vma_address() which abstracts the VMA start page offset field as pgoff_start, then update vma_address() to use it. Then introduce vma_anon_address() which does the equivalent of vma_address(), only using the anonymous page offset of the VMA rather than the file-backed one. Also add an assert to ensure that the function is not called for mappings which are file-backed but not MAP_PRIVATE to ensure it is only used in the correct places. This will be necessary for determining the address of a folio's index within a VMA when the folio belongs to a MAP_PRIVATE file-backed VMA but has been CoW'd, and thus is anonymous, once the anonymous VMA page offset field is used for the reverse mapping. No callers are updated, so no functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-4-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Gregory Price (Meta) Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- mm/internal.h | 49 +++++++++++++++++++++++++++++++++++++------------ 1 file changed, 37 insertions(+), 12 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index a75e7641ef49..761738d61fe0 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1017,19 +1017,9 @@ void mlock_drain_remote(int cpu); extern pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma); -/** - * vma_address - Find the virtual address a page range is mapped at - * @vma: The vma which maps this object. - * @pgoff: The page offset within its object. - * @nr_pages: The number of pages to consider. - * - * If any page in this range is mapped by this VMA, return the first address - * where any of these pages appear. Otherwise, return -EFAULT. - */ -static inline unsigned long vma_address(const struct vm_area_struct *vma, - pgoff_t pgoff, unsigned long nr_pages) +static inline unsigned long __vma_address(const struct vm_area_struct *vma, + pgoff_t pgoff, pgoff_t pgoff_start, unsigned long nr_pages) { - const pgoff_t pgoff_start = vma_start_pgoff(vma); unsigned long address; if (pgoff >= pgoff_start) { @@ -1047,6 +1037,41 @@ static inline unsigned long vma_address(const struct vm_area_struct *vma, return address; } +/** + * vma_address - Find the virtual address a page range is mapped at. + * @vma: The vma which maps this object. + * @pgoff: The page offset within its object. + * @nr_pages: The number of pages to consider. + * + * If any page in this range is mapped by this VMA, return the first address + * where any of these pages appear. Otherwise, return -EFAULT. + */ +static inline unsigned long vma_address(const struct vm_area_struct *vma, + pgoff_t pgoff, unsigned long nr_pages) +{ + return __vma_address(vma, pgoff, vma_start_pgoff(vma), nr_pages); +} + +/** + * vma_anon_address - Find the address an anonymous folio with index @pgoff_anon + * is mapped at. + * @vma: The vma which maps this object. + * @pgoff_anon: The anonymous page index belonging to the folio. + * @nr_pages: The number of pages to consider. + * + * This is only valid for anonymous or MAP_PRIVATE-mapped file-backed VMAs. + * + * Returns: If any page in this range is mapped by this VMA, return the first + * address where any of these pages appear. Otherwise, return -EFAULT. + */ +static inline unsigned long vma_anon_address(const struct vm_area_struct *vma, + pgoff_t pgoff_anon, unsigned long nr_pages) +{ + VM_WARN_ON_ONCE(!vma_is_cow_mapping(vma)); + + return __vma_address(vma, pgoff_anon, vma_start_anon_pgoff(vma), nr_pages); +} + /* * Then at what user virtual address will none of the range be found in vma? * Assumes that vma_address() already returned a good starting address. -- cgit v1.2.3 From 2a8de2d1d6a5f10251e89c77b547fd3243c8c3a8 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:22 +0100 Subject: mm: update print_bad_page_map() to show anon index if appropriate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit If the VMA is a CoW mapping page offset may differ from anon page offset, indicating different positions in the relevant rmap trees. Update print_bad_page_map() to reflect that - if the mapping is non-CoW or the indexes match, then output only one index as before, otherwise output both with (file) or (anon) suffixes to reflect which is which. It's not possible to give only one index as there is no folio available to perform folio_test_anon() upon (the page table entry is bad so this is unavailable). This is potentially useful debugging information and matches the existing page offset provided. Use the raw __linear_anon_page_index() function so as to always output this value regardless of whether the mapping is file-backed or not and to avoid asserts that shouldn't apply here. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-5-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Reviewed-by: Gregory Price (Meta) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- mm/memory.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 396d7b9059e6..c54943302553 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -631,13 +631,14 @@ static void print_bad_page_map(struct vm_area_struct *vma, { struct address_space *mapping; char entry_str[PTVAL_STR_MAX]; - pgoff_t index; + pgoff_t index, anon_index; if (is_bad_page_map_ratelimited()) return; mapping = vma->vm_file ? vma->vm_file->f_mapping : NULL; index = linear_page_index(vma, addr); + anon_index = __linear_anon_page_index(vma, addr); ptval_bytes_to_hex_str(entry_str, sizeof(entry_str), entry, entry_size); pr_alert("BUG: Bad page map in process %s %s:%s", current->comm, @@ -645,8 +646,14 @@ static void print_bad_page_map(struct vm_area_struct *vma, __print_bad_page_map_pgtable(vma->vm_mm, addr); if (page) dump_page(page, "bad page map"); - pr_alert("addr:%px vm_flags:%08lx anon_vma:%px mapping:%px index:%lx\n", - (void *)addr, vma->vm_flags, vma->anon_vma, mapping, index); + pr_alert("addr:%px vm_flags:%08lx anon_vma:%px mapping:%px", + (void *)addr, vma->vm_flags, vma->anon_vma, mapping); + if (!vma_is_cow_mapping(vma) || index == anon_index) { + pr_cont(" index:%lx\n", index); + } else { + pr_cont(" index:%lx (file) %lx (anon)\n", index, anon_index); + } + pr_alert("file:%pD fault:%ps mmap:%ps mmap_prepare: %ps read_folio:%ps\n", vma->vm_file, vma->vm_ops ? vma->vm_ops->fault : NULL, -- cgit v1.2.3 From dba10745e9b4dcf2d8b6a0c44a23eec4f58365f0 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:23 +0100 Subject: mm: introduce and use vma_filebacked_address() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit In cases where we know that the VMA is file-backed, use vma_filebacked_address() rather than vma_address(). This lays the foundation for using the anonymous page offset via vma_anon_address(). Also add an assert to ensure that the VMA whose address is required is not anonymous. No functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-6-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Tested-by: syzbot@syzkaller.appspotmail.com Acked-by: David Hildenbrand (Arm) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Gregory Price (Meta) Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- mm/internal.h | 18 ++++++++++++++++++ mm/memory-failure.c | 4 ++-- mm/page_vma_mapped.c | 6 +++++- mm/rmap.c | 10 ++++++---- 4 files changed, 31 insertions(+), 7 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 761738d61fe0..f6475d77024d 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1037,6 +1037,24 @@ static inline unsigned long __vma_address(const struct vm_area_struct *vma, return address; } +/** + * vma_filebacked_address - Find the virtual address a file-backed page range is + * mapped at. + * @vma: The vma which maps this object. + * @pgoff: The page offset within its object. + * @nr_pages: The number of pages to consider. + * + * Returns: If any page in this range is mapped by this VMA, return the first + * address where any of these pages appear. Otherwise, return -EFAULT. + */ +static inline unsigned long vma_filebacked_address(const struct vm_area_struct *vma, + pgoff_t pgoff, unsigned long nr_pages) +{ + VM_WARN_ON_ONCE(vma_is_anonymous(vma)); + + return __vma_address(vma, pgoff, vma_start_pgoff(vma), nr_pages); +} + /** * vma_address - Find the virtual address a page range is mapped at. * @vma: The vma which maps this object. diff --git a/mm/memory-failure.c b/mm/memory-failure.c index aaf14608b30e..a8b03e2920ba 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -620,7 +620,7 @@ static void add_to_kill_fsdax(struct task_struct *tsk, const struct page *p, struct vm_area_struct *vma, struct list_head *to_kill, pgoff_t pgoff) { - unsigned long addr = vma_address(vma, pgoff, 1); + unsigned long addr = vma_filebacked_address(vma, pgoff, 1); __add_to_kill(tsk, p, vma, to_kill, addr); } @@ -2265,7 +2265,7 @@ static void add_to_kill_pgoff(struct task_struct *tsk, } /* Check for pgoff not backed by struct page */ - tk->addr = vma_address(vma, pgoff, 1); + tk->addr = vma_filebacked_address(vma, pgoff, 1); tk->size_shift = PAGE_SHIFT; if (tk->addr == -EFAULT) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index d7670ba4147b..081e483cc7bf 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -356,6 +356,7 @@ unsigned long page_mapped_in_vma(const struct page *page, struct vm_area_struct *vma) { const struct folio *folio = page_folio(page); + const pgoff_t pgoff = page_pgoff(folio, page); struct page_vma_mapped_walk pvmw = { .pfn = page_to_pfn(page), .nr_pages = 1, @@ -363,7 +364,10 @@ unsigned long page_mapped_in_vma(const struct page *page, .flags = PVMW_SYNC, }; - pvmw.address = vma_address(vma, page_pgoff(folio, page), 1); + if (folio_test_anon(folio)) + pvmw.address = vma_address(vma, pgoff, 1); + else + pvmw.address = vma_filebacked_address(vma, pgoff, 1); if (pvmw.address == -EFAULT) goto out; if (!page_vma_mapped_walk(&pvmw)) diff --git a/mm/rmap.c b/mm/rmap.c index 1f72d279ba68..bf618e4678d3 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -865,14 +865,15 @@ unsigned long page_address_in_vma(const struct folio *folio, if (!vma->anon_vma || !anon_vma || vma->anon_vma->root != anon_vma->root) return -EFAULT; + /* KSM folios don't reach here because of the !anon_vma check */ + return vma_address(vma, page_pgoff(folio, page), 1); } else if (!vma->vm_file) { return -EFAULT; } else if (vma->vm_file->f_mapping != folio->mapping) { return -EFAULT; } - /* KSM folios don't reach here because of the !anon_vma check */ - return vma_address(vma, page_pgoff(folio, page), 1); + return vma_filebacked_address(vma, page_pgoff(folio, page), 1); } /* @@ -1321,7 +1322,7 @@ int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, if (invalid_mkclean_vma(vma, NULL)) return 0; - pvmw.address = vma_address(vma, pgoff, nr_pages); + pvmw.address = vma_filebacked_address(vma, pgoff, nr_pages); VM_BUG_ON_VMA(pvmw.address == -EFAULT, vma); return page_vma_mkclean_one(&pvmw); @@ -3098,7 +3099,8 @@ static void __rmap_walk_file(struct folio *folio, struct address_space *mapping, } lookup: mapping_rmap_tree_foreach(vma, mapping, pgoff_start, pgoff_end) { - unsigned long address = vma_address(vma, pgoff_start, nr_pages); + unsigned long address = vma_filebacked_address(vma, pgoff_start, + nr_pages); VM_BUG_ON_VMA(address == -EFAULT, vma); cond_resched(); -- cgit v1.2.3 From 9998bc06d75bfbb17b8ff7f83183d133923c5829 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:24 +0100 Subject: mm/vma: fix self-merge check in copy_vma() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The existing logic is very confusing so improve things. Firstly rename the confusing faulted_in_anon_vma variable to can_self_merge and update this when the page offset is updated. What is being checked for is a 'self-merge' - that is between the VMA being remapped and its prior VMA (remember that this is copy_vma() - if a non-MREMAP_DONTUNMAP remap the original VMA is only removed afterwards). This can happen if the VMA is moved immediately adjacent to itself, either before or after it: |----------------|----------------| | | | v | v |...............||---------------||...............| | new || old || new | |...............||---------------||---------------| In these cases the old VMA is simply expanded to cover the new range. It is also possible for the move to both self-merge and merge with a prior VMA if it is placed between a preceding VMA and its old self: |---------------| | | v | |---------------||...............||---------------| | prev || new || old | |---------------||...............||---------------| In this case, the old VMA is removed and 'prev' is expanded and replaces it. Since copy_vma_and_data() which calls copy_vma() intends to reference the old VMA after the merge, it must have this pointer updated. This kind of self-merge is not possible with a succeeding merge, as the merge always prefers to expand the preceding VMA if possible. copy_vma() accounts for this by explicitly checking to see if a self-merge occurred and updating the vmap pointer if so. However it incorrect did so even for a subsequent merge (this is simply a noop so it had no impact). So change this to only check for the case which matters - a backwards merge - and rearrange the parameters to make it clearer we're doing that - i.e. check new_vma->vm_start < old_vma_start (having already renamed vma_start to old_vma_start to make it clear this is the previous VMA). Also update the existing wall-of-text comment to be a lot clearer. While we're here, replace the VM_BUG_ON_VMA() with a VM_WARN_ON_ONCE_VMA() and update the VMA userland tests accordingly. No functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-7-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Gregory Price (Meta) Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- mm/vma.c | 35 ++++++++++++++++------------------- tools/testing/vma/vma_internal.h | 1 + 2 files changed, 17 insertions(+), 19 deletions(-) diff --git a/mm/vma.c b/mm/vma.c index a325376e62ea..ecefc9e63070 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -1911,10 +1911,10 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, bool *need_rmap_locks) { struct vm_area_struct *vma = *vmap; - unsigned long vma_start = vma->vm_start; + unsigned long old_vma_start = vma->vm_start; struct mm_struct *mm = vma->vm_mm; struct vm_area_struct *new_vma; - bool faulted_in_anon_vma = true; + bool can_self_merge = false; VMA_ITERATOR(vmi, mm, addr); VMG_VMA_STATE(vmg, &vmi, NULL, vma, addr, addr + len); @@ -1924,7 +1924,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, */ if (unlikely(vma_is_anonymous(vma) && !vma->anon_vma)) { pgoff = addr >> PAGE_SHIFT; - faulted_in_anon_vma = false; + can_self_merge = true; } /* @@ -1944,24 +1944,21 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, new_vma = vma_merge_copied_range(&vmg); if (new_vma) { - /* - * Source vma may have been merged into new_vma - */ - if (unlikely(vma_start >= new_vma->vm_start && - vma_start < new_vma->vm_end)) { + /* Self-merged and VMA replaced. */ + if (unlikely(new_vma->vm_start < old_vma_start && + new_vma->vm_end > old_vma_start)) { /* - * The only way we can get a vma_merge with - * self during an mremap is if the vma hasn't - * been faulted in yet and we were allowed to - * reset the dst vma->vm_pgoff to the - * destination address of the mremap to allow - * the merge to happen. mremap must change the - * vm_pgoff linearity between src and dst vmas - * (in turn preventing a vma_merge) to be - * safe. It is only safe to keep the vm_pgoff - * linear if there are no pages mapped yet. + * The only way a VMA can both self-merge and be + * replaced is if the remap places the new VMA + * immediately prior to its old self ('next') and + * immediately after another VMA ('prev') causing the + * next to be removed and prev to be expanded to cover + * the entire range. + * + * This should only be possible if the page offset was + * updated, i.e. the VMA is unfaulted. */ - VM_BUG_ON_VMA(faulted_in_anon_vma, new_vma); + VM_WARN_ON_ONCE_VMA(!can_self_merge, new_vma); *vmap = vma = new_vma; } *need_rmap_locks = diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index 4f6c5666ac07..8a48b231aa7a 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -53,6 +53,7 @@ typedef __bitwise unsigned int vm_fault_t; #define VM_WARN_ON(_expr) (WARN_ON(_expr)) #define VM_WARN_ON_ONCE(_expr) (WARN_ON_ONCE(_expr)) +#define VM_WARN_ON_ONCE_VMA(_expr, _vma) (WARN_ON_ONCE(_expr)) #define VM_WARN_ON_VMG(_expr, _vmg) (WARN_ON(_expr)) #define VM_BUG_ON(_expr) (BUG_ON(_expr)) #define VM_BUG_ON_VMA(_expr, _vma) (BUG_ON(_expr)) -- cgit v1.2.3 From 746b9e0a4777e8e883d70b4292a6c3d8c435712c Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:25 +0100 Subject: tools/testing/vma: add tests for copy_vma() self-merge MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Assert that a VMA can be moved backwards, forwards and between a preceding VMA and its old self. In the cases in which the VMA merges only with itself expect that to be achieved by expanding its old self, so assert that these function correctly. However in the case of a merge between a preceding VMA and itself the original VMA is removed, so assert that the preceding VMA replaces the one passed in as vmap and the merge is as expected. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-8-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Gregory Price (Meta) Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- tools/testing/vma/tests/vma.c | 46 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/tools/testing/vma/tests/vma.c b/tools/testing/vma/tests/vma.c index 754a2da06321..0d40d7ba2181 100644 --- a/tools/testing/vma/tests/vma.c +++ b/tools/testing/vma/tests/vma.c @@ -33,7 +33,51 @@ static bool test_copy_vma(void) struct mm_struct mm = {}; bool need_locks = false; VMA_ITERATOR(vmi, &mm, 0); - struct vm_area_struct *vma, *vma_new, *vma_next; + struct vm_area_struct *vma, *vma_prev, *vma_new, *vma_next, *vma_orig; + + /* Move forwards, adjacent to old self - self-merge. */ + + vma = alloc_and_link_vma(&mm, 0x1000, 0x2000, 1, vma_flags); + vma_set_anonymous(vma); + vma_orig = vma; + vma_new = copy_vma(&vma, 0x2000, 0x1000, 1, &need_locks); + ASSERT_EQ(vma_new, vma_orig); + ASSERT_EQ(vma, vma_orig); + ASSERT_EQ(vma_new->vm_start, 0x1000); + ASSERT_EQ(vma_new->vm_end, 0x3000); + + cleanup_mm(&mm, &vmi); + + /* Move backwards, adjacent to old self - self-merge. */ + + vma = alloc_and_link_vma(&mm, 0x2000, 0x3000, 2, vma_flags); + vma_set_anonymous(vma); + vma_orig = vma; + vma_new = copy_vma(&vma, 0x1000, 0x1000, 2, &need_locks); + ASSERT_EQ(vma_new, vma_orig); + ASSERT_EQ(vma, vma_orig); + ASSERT_EQ(vma_new->vm_start, 0x1000); + ASSERT_EQ(vma_new->vm_end, 0x3000); + + cleanup_mm(&mm, &vmi); + + /* + * Move backwards between prior VMA and old self - self-merge and vma + * updated to a new VMA. + */ + + vma_prev = alloc_and_link_vma(&mm, 0x1000, 0x2000, 1, vma_flags); + vma_set_anonymous(vma_prev); + vma = alloc_and_link_vma(&mm, 0x3000, 0x4000, 3, vma_flags); + vma_set_anonymous(vma); + vma_orig = vma; + vma_new = copy_vma(&vma, 0x2000, 0x1000, 3, &need_locks); + ASSERT_NE(vma_new, vma_orig); + ASSERT_EQ(vma_new, vma); + ASSERT_EQ(vma_new->vm_start, 0x1000); + ASSERT_EQ(vma_new->vm_end, 0x4000); + + cleanup_mm(&mm, &vmi); /* Move backwards and do not merge. */ -- cgit v1.2.3 From 6a993c7fbc3e99431e148eb261c9b2e38525fce4 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:26 +0100 Subject: mm: propagate VMA anonymous page offset on map, remap, split + merge MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit We must correctly update VMA anonymous page offset state on all VMA operations that would result in it changing, with special attention given to remapping. We cover most cases by simply updating vma_set_range() to do so (with a new anonymous page offset parameter), but also notably must update the merging and mapping logic to propagate this parameter correctly. The remap logic remains the same - we may update the anonymous page offset if the VMA is unfaulted, but now this applies to MAP_PRIVATE file-backed mappings too, so we update the code to reflect this. Note that we use __linear_anon_page_index() upon remap as the VMA may be shared, in order that we update the field consistently regardless of VMA type. Similarly, pass through anon page offset to the merge logic, updating the vma_merge_struct struct to propagate it, and also use __linear_anon_page_index() to obtain the anonymous page index so it can be safely used for both shared and MAP_PRIVATE file-backed mappings. In copy_vma(), the anonymous page offset is updated regardless of whether the mapping is a CoW mapping or not. This is both to keep the anonymous page offset consistent even for non-CoW mappings (it is set so should at least remain correct) and makes the logic cleaner. A self-merge however remains permitted only for mappings which can have a populated vma->anon_vma and do not require alignment on a separate file offset - that is pure anonymous VMAs, so only set can_self_merge if vma_is_anonymous(). Finally, we update insert_vm_struct() to correctly set the anonymous page offset on insertion of a VMA. We simply ensure state is correctly propagated here, so no functional changes are intended. Also update VMA userland tests to reflect this change. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-9-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Gregory Price (Meta) Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- mm/mremap.c | 6 ++-- mm/vma.c | 52 ++++++++++++++++++---------- mm/vma.h | 75 ++++++++++++++++++++++++----------------- mm/vma_exec.c | 2 +- tools/testing/vma/shared.c | 3 +- tools/testing/vma/tests/merge.c | 4 ++- tools/testing/vma/tests/vma.c | 10 +++--- 7 files changed, 95 insertions(+), 57 deletions(-) diff --git a/mm/mremap.c b/mm/mremap.c index b64aa1f6e07e..9ea1707eafa5 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -1265,7 +1265,9 @@ static void unmap_source_vma(struct vma_remap_struct *vrm) static int copy_vma_and_data(struct vma_remap_struct *vrm, struct vm_area_struct **new_vma_ptr) { - const unsigned long new_pgoff = linear_page_index(vrm->vma, vrm->addr); + const pgoff_t new_pgoff = linear_page_index(vrm->vma, vrm->addr); + const pgoff_t new_anon_pgoff = + __linear_anon_page_index(vrm->vma, vrm->addr); struct vm_area_struct *vma = vrm->vma; struct vm_area_struct *new_vma; unsigned long moved_len; @@ -1273,7 +1275,7 @@ static int copy_vma_and_data(struct vma_remap_struct *vrm, PAGETABLE_MOVE(pmc, NULL, NULL, vrm->addr, vrm->new_addr, vrm->old_len); new_vma = copy_vma(&vma, vrm->new_addr, vrm->new_len, new_pgoff, - &pmc.need_rmap_locks); + new_anon_pgoff, &pmc.need_rmap_locks); if (!new_vma) { vrm_uncharge(vrm); *new_vma_ptr = NULL; diff --git a/mm/vma.c b/mm/vma.c index ecefc9e63070..e35b04ac12cb 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -18,6 +18,7 @@ struct mmap_state { unsigned long addr; unsigned long end; pgoff_t pgoff; + pgoff_t anon_pgoff; unsigned long pglen; union { vm_flags_t vm_flags; @@ -46,13 +47,14 @@ struct mmap_state { bool file_doesnt_need_get :1; }; -#define MMAP_STATE(name, mm_, vmi_, addr_, len_, pgoff_, vma_flags_, file_) \ +#define MMAP_STATE(name, mm_, vmi_, addr_, len_, pgoff_, anon_pgoff_, vma_flags_, file_) \ struct mmap_state name = { \ .mm = mm_, \ .vmi = vmi_, \ .addr = addr_, \ .end = (addr_) + (len_), \ .pgoff = pgoff_, \ + .anon_pgoff = anon_pgoff_, \ .pglen = PHYS_PFN(len_), \ .vma_flags = vma_flags_, \ .file = file_, \ @@ -67,6 +69,7 @@ struct mmap_state { .end = (map_)->end, \ .vma_flags = (map_)->vma_flags, \ .pgoff = (map_)->pgoff, \ + .anon_pgoff = (map_)->anon_pgoff, \ .file = (map_)->file, \ .prev = (map_)->prev, \ .middle = vma_, \ @@ -82,10 +85,11 @@ static void __vma_set_range(struct vm_area_struct *vma, unsigned long start, } static void vma_set_range(struct vm_area_struct *vma, unsigned long start, - unsigned long end, pgoff_t pgoff) + unsigned long end, pgoff_t pgoff, pgoff_t anon_pgoff) { __vma_set_range(vma, start, end); vma_set_pgoff(vma, pgoff); + vma_set_anon_pgoff(vma, anon_pgoff); } /* Was this VMA ever forked from a parent, i.e. maybe contains CoW mappings? */ @@ -812,7 +816,8 @@ static int commit_merge(struct vma_merge_struct *vmg) */ vma_adjust_trans_huge(vma, vmg->start, vmg->end, vmg->__adjust_middle_start ? vmg->middle : NULL); - vma_set_range(vma, vmg->start, vmg->end, vmg_start_pgoff(vmg)); + vma_set_range(vma, vmg->start, vmg->end, vmg_start_pgoff(vmg), + vmg_start_anon_pgoff(vmg)); vmg_adjust_set_range(vmg); vma_iter_store_overwrite(vmg->vmi, vmg->target); @@ -982,6 +987,7 @@ static __must_check struct vm_area_struct *vma_merge_existing_range( vmg->start = prev->vm_start; vmg->end = next->vm_end; vmg->pgoff = vma_start_pgoff(prev); + vmg->anon_pgoff = vma_start_anon_pgoff(prev); /* * We already ensured anon_vma compatibility above, so now it's @@ -1000,6 +1006,7 @@ static __must_check struct vm_area_struct *vma_merge_existing_range( */ vmg->start = prev->vm_start; vmg->pgoff = vma_start_pgoff(prev); + vmg->anon_pgoff = vma_start_anon_pgoff(prev); if (!vmg->__remove_middle) vmg->__adjust_middle_start = true; @@ -1022,12 +1029,14 @@ static __must_check struct vm_area_struct *vma_merge_existing_range( if (vmg->__remove_middle) { vmg->end = next->vm_end; vmg->pgoff = vma_start_pgoff(next) - pglen; + vmg->anon_pgoff = vma_start_anon_pgoff(next) - pglen; } else { /* We shrink middle and expand next. */ vmg->__adjust_next_start = true; vmg->start = middle->vm_start; vmg->end = start; vmg->pgoff = vma_start_pgoff(middle); + vmg->anon_pgoff = vma_start_anon_pgoff(middle); } err = dup_anon_vma(next, middle, &anon_dup); @@ -1137,6 +1146,7 @@ struct vm_area_struct *vma_merge_new_range(struct vma_merge_struct *vmg) vmg->start = prev->vm_start; vmg->target = prev; vmg->pgoff = vma_start_pgoff(prev); + vmg->anon_pgoff = vma_start_anon_pgoff(prev); /* * If this merge would result in removal of the next VMA but we @@ -1908,7 +1918,7 @@ static int vma_link(struct mm_struct *mm, struct vm_area_struct *vma) */ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, unsigned long addr, unsigned long len, pgoff_t pgoff, - bool *need_rmap_locks) + pgoff_t anon_pgoff, bool *need_rmap_locks) { struct vm_area_struct *vma = *vmap; unsigned long old_vma_start = vma->vm_start; @@ -1919,12 +1929,16 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, VMG_VMA_STATE(vmg, &vmi, NULL, vma, addr, addr + len); /* - * If anonymous vma has not yet been faulted, update new pgoff - * to match new location, to increase its chance of merging. + * If a vma has not yet been faulted, update its anonymous pgoff to + * match the new location to increase its chance of merging. */ - if (unlikely(vma_is_anonymous(vma) && !vma->anon_vma)) { - pgoff = addr >> PAGE_SHIFT; - can_self_merge = true; + if (!vma->anon_vma) { + anon_pgoff = addr >> PAGE_SHIFT; + + if (vma_is_anonymous(vma)) { + pgoff = anon_pgoff; + can_self_merge = true; + } } /* @@ -1940,6 +1954,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, return NULL; /* should never get here */ vmg.pgoff = pgoff; + vmg.anon_pgoff = anon_pgoff; vmg.next = vma_iter_next_rewind(&vmi, NULL); new_vma = vma_merge_copied_range(&vmg); @@ -1955,8 +1970,8 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, * next to be removed and prev to be expanded to cover * the entire range. * - * This should only be possible if the page offset was - * updated, i.e. the VMA is unfaulted. + * This should only be possible if the anonymous page + * offset was updated, i.e. the VMA is unfaulted. */ VM_WARN_ON_ONCE_VMA(!can_self_merge, new_vma); *vmap = vma = new_vma; @@ -1967,7 +1982,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, new_vma = vm_area_dup(vma); if (!new_vma) goto out; - vma_set_range(new_vma, addr, addr + len, pgoff); + vma_set_range(new_vma, addr, addr + len, pgoff, anon_pgoff); if (vma_dup_policy(vma, new_vma)) goto out_free_vma; if (anon_vma_clone(new_vma, vma, VMA_OP_REMAP)) @@ -2609,7 +2624,7 @@ static int __mmap_new_vma(struct mmap_state *map, struct vm_area_struct **vmap, if (is_anon) vma_set_anonymous(vma); - vma_set_range(vma, map->addr, map->end, map->pgoff); + vma_set_range(vma, map->addr, map->end, map->pgoff, map->anon_pgoff); vma->flags = map->vma_flags; vma->vm_page_prot = map->page_prot; @@ -2798,7 +2813,8 @@ static unsigned long __mmap_region(struct file *file, unsigned long addr, struct vm_area_struct *vma = NULL; bool have_mmap_prepare = file && file->f_op->mmap_prepare; VMA_ITERATOR(vmi, mm, addr); - MMAP_STATE(map, mm, &vmi, addr, len, pgoff, vma_flags, file); + const pgoff_t anon_pgoff = addr >> PAGE_SHIFT; + MMAP_STATE(map, mm, &vmi, addr, len, pgoff, anon_pgoff, vma_flags, file); struct vm_area_desc desc = { .mm = mm, .file = file, @@ -2943,6 +2959,7 @@ int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, unsigned long addr, unsigned long len, vma_flags_t vma_flags) { struct mm_struct *mm = current->mm; + const pgoff_t pgoff = addr >> PAGE_SHIFT; /* * Check against address space limits by the changed size @@ -2967,7 +2984,7 @@ int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, * occur after forking, so the expand will only happen on new VMAs. */ if (vma && vma->vm_end == addr) { - VMG_STATE(vmg, mm, vmi, addr, addr + len, vma_flags, PHYS_PFN(addr)); + VMG_STATE(vmg, mm, vmi, addr, addr + len, vma_flags, pgoff, pgoff); vmg.prev = vma; /* vmi is positioned at prev, which this mode expects. */ @@ -2987,7 +3004,7 @@ int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, goto unacct_fail; vma_set_anonymous(vma); - vma_set_range(vma, addr, addr + len, addr >> PAGE_SHIFT); + vma_set_range(vma, addr, addr + len, pgoff, pgoff); vma->flags = vma_flags; vma->vm_page_prot = vm_get_page_prot(vma_flags_to_legacy(vma_flags)); vma_start_write(vma); @@ -3379,6 +3396,7 @@ int insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma) WARN_ON_ONCE(vma->anon_vma); vma_set_pgoff(vma, vma->vm_start >> PAGE_SHIFT); } + vma_set_anon_pgoff(vma, vma->vm_start >> PAGE_SHIFT); if (vma_link(mm, vma)) { if (vma_test(vma, VMA_ACCOUNT_BIT)) @@ -3434,7 +3452,7 @@ struct vm_area_struct *__install_special_mapping( vma->vm_ops = ops; vma->vm_private_data = priv; - vma_set_range(vma, addr, addr + len, 0); + vma_set_range(vma, addr, addr + len, 0, addr >> PAGE_SHIFT); ret = insert_vm_struct(mm, vma); if (ret) diff --git a/mm/vma.h b/mm/vma.h index 54ed7c744e3b..024fabe63560 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -104,6 +104,7 @@ struct vma_merge_struct { unsigned long start; unsigned long end; pgoff_t pgoff; + pgoff_t anon_pgoff; union { /* Temporary while VMA flags are being converted. */ @@ -237,11 +238,6 @@ static inline bool vmg_nomem(struct vma_merge_struct *vmg) return vmg->state == VMA_MERGE_ERROR_NOMEM; } -static inline pgoff_t vmg_start_pgoff(const struct vma_merge_struct *vmg) -{ - return vmg->pgoff; -} - static inline pgoff_t vmg_pages(const struct vma_merge_struct *vmg) { const unsigned long size = vmg->end - vmg->start; @@ -249,6 +245,11 @@ static inline pgoff_t vmg_pages(const struct vma_merge_struct *vmg) return size >> PAGE_SHIFT; } +static inline pgoff_t vmg_start_pgoff(const struct vma_merge_struct *vmg) +{ + return vmg->pgoff; +} + static inline pgoff_t vmg_end_pgoff(const struct vma_merge_struct *vmg) { return vmg_start_pgoff(vmg) + vmg_pages(vmg); @@ -283,6 +284,16 @@ static inline void vma_set_pgoff(struct vm_area_struct *vma, pgoff_t pgoff) vma->vm_pgoff = pgoff; } +static inline pgoff_t vmg_start_anon_pgoff(const struct vma_merge_struct *vmg) +{ + return vmg->anon_pgoff; +} + +static inline pgoff_t vmg_end_anon_pgoff(const struct vma_merge_struct *vmg) +{ + return vmg_start_anon_pgoff(vmg) + vmg_pages(vmg); +} + static inline void __vma_set_anon_pgoff(struct vm_area_struct *vma, pgoff_t pgoff) { #ifdef CONFIG_64BIT @@ -301,44 +312,48 @@ static inline void vma_add_pgoff(struct vm_area_struct *vma, pgoff_t delta) { vma_assert_can_modify(vma); vma_set_pgoff(vma, vma_start_pgoff(vma) + delta); + vma_set_anon_pgoff(vma, vma_start_anon_pgoff(vma) + delta); } static inline void vma_sub_pgoff(struct vm_area_struct *vma, pgoff_t delta) { vma_assert_can_modify(vma); vma_set_pgoff(vma, vma_start_pgoff(vma) - delta); -} + vma_set_anon_pgoff(vma, vma_start_anon_pgoff(vma) - delta); +} + +#define VMG_STATE(name, mm_, vmi_, start_, end_, vma_flags_, pgoff_, anon_pgoff_) \ + struct vma_merge_struct name = { \ + .mm = mm_, \ + .vmi = vmi_, \ + .start = start_, \ + .end = end_, \ + .vma_flags = vma_flags_, \ + .pgoff = pgoff_, \ + .anon_pgoff = anon_pgoff_, \ + .state = VMA_MERGE_START, \ + } -#define VMG_STATE(name, mm_, vmi_, start_, end_, vma_flags_, pgoff_) \ +#define VMG_VMA_STATE(name, vmi_, prev_, vma_, start_, end_) \ struct vma_merge_struct name = { \ - .mm = mm_, \ + .mm = vma_->vm_mm, \ .vmi = vmi_, \ + .prev = prev_, \ + .middle = vma_, \ + .next = NULL, \ .start = start_, \ .end = end_, \ - .vma_flags = vma_flags_, \ - .pgoff = pgoff_, \ + .vm_flags = vma_->vm_flags, \ + .pgoff = linear_page_index(vma_, start_), \ + .anon_pgoff = __linear_anon_page_index(vma_, start_), \ + .file = vma_->vm_file, \ + .anon_vma = vma_->anon_vma, \ + .policy = vma_policy(vma_), \ + .uffd_ctx = vma_->vm_userfaultfd_ctx, \ + .anon_name = anon_vma_name(vma_), \ .state = VMA_MERGE_START, \ } -#define VMG_VMA_STATE(name, vmi_, prev_, vma_, start_, end_) \ - struct vma_merge_struct name = { \ - .mm = vma_->vm_mm, \ - .vmi = vmi_, \ - .prev = prev_, \ - .middle = vma_, \ - .next = NULL, \ - .start = start_, \ - .end = end_, \ - .vm_flags = vma_->vm_flags, \ - .pgoff = linear_page_index(vma_, start_), \ - .file = vma_->vm_file, \ - .anon_vma = vma_->anon_vma, \ - .policy = vma_policy(vma_), \ - .uffd_ctx = vma_->vm_userfaultfd_ctx, \ - .anon_name = anon_vma_name(vma_), \ - .state = VMA_MERGE_START, \ - } - #ifdef CONFIG_DEBUG_VM_MAPLE_TREE void validate_mm(struct mm_struct *mm); #else @@ -520,7 +535,7 @@ void unlink_file_vma_batch_add(struct unlink_vma_file_batch *vb, struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, unsigned long addr, unsigned long len, pgoff_t pgoff, - bool *need_rmap_locks); + pgoff_t anon_pgoff, bool *need_rmap_locks); struct anon_vma *find_mergeable_anon_vma(struct vm_area_struct *vma); diff --git a/mm/vma_exec.c b/mm/vma_exec.c index 7af1260689b9..586c52155942 100644 --- a/mm/vma_exec.c +++ b/mm/vma_exec.c @@ -41,7 +41,7 @@ int relocate_vma_down(struct vm_area_struct *vma, unsigned long shift) unsigned long new_end = old_end - shift; VMA_ITERATOR(vmi, mm, new_start); VMG_STATE(vmg, mm, &vmi, new_start, old_end, EMPTY_VMA_FLAGS, - vma_start_pgoff(vma)); + vma_start_pgoff(vma), vma_start_anon_pgoff(vma)); struct vm_area_struct *next; struct mmu_gather tlb; PAGETABLE_MOVE(pmc, vma, vma, old_start, new_start, length); diff --git a/tools/testing/vma/shared.c b/tools/testing/vma/shared.c index bea9ea6db02a..4a39c9d50489 100644 --- a/tools/testing/vma/shared.c +++ b/tools/testing/vma/shared.c @@ -23,7 +23,8 @@ struct vm_area_struct *alloc_vma(struct mm_struct *mm, vma->vm_start = start; vma->vm_end = end; - vma->vm_pgoff = pgoff; + vma_set_pgoff(vma, pgoff); + vma_set_anon_pgoff(vma, start >> PAGE_SHIFT); vma->flags = vma_flags; vma_assert_detached(vma); diff --git a/tools/testing/vma/tests/merge.c b/tools/testing/vma/tests/merge.c index e357accc8499..48418b82b01d 100644 --- a/tools/testing/vma/tests/merge.c +++ b/tools/testing/vma/tests/merge.c @@ -45,6 +45,7 @@ void vmg_set_range(struct vma_merge_struct *vmg, unsigned long start, vmg->start = start; vmg->end = end; vmg->pgoff = pgoff; + vmg->anon_pgoff = start >> PAGE_SHIFT; vmg->vma_flags = vma_flags; vmg->just_expand = false; @@ -108,6 +109,7 @@ static bool test_simple_merge(void) .end = 0x2000, .vma_flags = vma_flags, .pgoff = 1, + .anon_pgoff = 1, }; ASSERT_FALSE(attach_vma(&mm, vma_left)); @@ -1431,7 +1433,7 @@ static bool test_expand_only_mode(void) struct mm_struct mm = {}; VMA_ITERATOR(vmi, &mm, 0); struct vm_area_struct *vma_prev, *vma; - VMG_STATE(vmg, &mm, &vmi, 0x5000, 0x9000, vma_flags, 5); + VMG_STATE(vmg, &mm, &vmi, 0x5000, 0x9000, vma_flags, 5, 5); /* * Place a VMA prior to the one we're expanding so we assert that we do diff --git a/tools/testing/vma/tests/vma.c b/tools/testing/vma/tests/vma.c index 0d40d7ba2181..c8ef7b8cd46b 100644 --- a/tools/testing/vma/tests/vma.c +++ b/tools/testing/vma/tests/vma.c @@ -40,7 +40,7 @@ static bool test_copy_vma(void) vma = alloc_and_link_vma(&mm, 0x1000, 0x2000, 1, vma_flags); vma_set_anonymous(vma); vma_orig = vma; - vma_new = copy_vma(&vma, 0x2000, 0x1000, 1, &need_locks); + vma_new = copy_vma(&vma, 0x2000, 0x1000, 1, 1, &need_locks); ASSERT_EQ(vma_new, vma_orig); ASSERT_EQ(vma, vma_orig); ASSERT_EQ(vma_new->vm_start, 0x1000); @@ -53,7 +53,7 @@ static bool test_copy_vma(void) vma = alloc_and_link_vma(&mm, 0x2000, 0x3000, 2, vma_flags); vma_set_anonymous(vma); vma_orig = vma; - vma_new = copy_vma(&vma, 0x1000, 0x1000, 2, &need_locks); + vma_new = copy_vma(&vma, 0x1000, 0x1000, 2, 2, &need_locks); ASSERT_EQ(vma_new, vma_orig); ASSERT_EQ(vma, vma_orig); ASSERT_EQ(vma_new->vm_start, 0x1000); @@ -71,7 +71,7 @@ static bool test_copy_vma(void) vma = alloc_and_link_vma(&mm, 0x3000, 0x4000, 3, vma_flags); vma_set_anonymous(vma); vma_orig = vma; - vma_new = copy_vma(&vma, 0x2000, 0x1000, 3, &need_locks); + vma_new = copy_vma(&vma, 0x2000, 0x1000, 3, 3, &need_locks); ASSERT_NE(vma_new, vma_orig); ASSERT_EQ(vma_new, vma); ASSERT_EQ(vma_new->vm_start, 0x1000); @@ -82,7 +82,7 @@ static bool test_copy_vma(void) /* Move backwards and do not merge. */ vma = alloc_and_link_vma(&mm, 0x3000, 0x5000, 3, vma_flags); - vma_new = copy_vma(&vma, 0, 0x2000, 0, &need_locks); + vma_new = copy_vma(&vma, 0, 0x2000, 0, 3, &need_locks); ASSERT_NE(vma_new, vma); ASSERT_EQ(vma_new->vm_start, 0); ASSERT_EQ(vma_new->vm_end, 0x2000); @@ -95,7 +95,7 @@ static bool test_copy_vma(void) vma = alloc_and_link_vma(&mm, 0, 0x2000, 0, vma_flags); vma_next = alloc_and_link_vma(&mm, 0x6000, 0x8000, 6, vma_flags); - vma_new = copy_vma(&vma, 0x4000, 0x2000, 4, &need_locks); + vma_new = copy_vma(&vma, 0x4000, 0x2000, 4, 4, &need_locks); vma_assert_attached(vma_new); ASSERT_EQ(vma_new, vma_next); -- cgit v1.2.3 From 50c5f35a64aad82e3fd22e26f2b9c9c7395fa206 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:27 +0100 Subject: mm/rmap: track whether the page VMA mapped pgoff is anonymous MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update the page_vma_mapped_walk structure to track whether the pgoff being tracked is an anonymous pgoff or not and update the comments to reflect this. This is necessary in order to determine the correct VMA page offset in vma_address_end() when pvmw->nr_pages > 1. Also document that pvmw->pgoff is meaningless for pvmw->nr_pages == 1 and for KSM. Do not set this field where pgoff is not specified. This is laying the groundwork for eventually using anonymous page offsets as the index for all anonymous folios. No functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-10-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Gregory Price (Meta) Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- include/linux/rmap.h | 4 +++- mm/rmap.c | 2 ++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 8dc0871e5f00..0574537a355c 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -864,13 +864,14 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, struct page_vma_mapped_walk { unsigned long pfn; unsigned long nr_pages; - pgoff_t pgoff; + pgoff_t pgoff; /* Only meaningful if nr_pages > 1 and not a KSM walk */ struct vm_area_struct *vma; unsigned long address; pmd_t *pmd; pte_t *pte; spinlock_t *ptl; unsigned int flags; + bool pgoff_is_anon : 1; }; #define DEFINE_FOLIO_VMA_WALK(name, _folio, _vma, _address, _flags) \ @@ -881,6 +882,7 @@ struct page_vma_mapped_walk { .vma = _vma, \ .address = _address, \ .flags = _flags, \ + .pgoff_is_anon = folio_test_anon(_folio), \ } static inline void page_vma_mapped_walk_done(struct page_vma_mapped_walk *pvmw) diff --git a/mm/rmap.c b/mm/rmap.c index bf618e4678d3..1b23ac709f45 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1240,6 +1240,7 @@ static bool mapping_wrprotect_range_one(struct folio *folio, .vma = vma, .address = address, .flags = PVMW_SYNC, + .pgoff_is_anon = false, }; state->cleaned += page_vma_mkclean_one(&pvmw); @@ -1317,6 +1318,7 @@ int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, .pgoff = pgoff, .vma = vma, .flags = PVMW_SYNC, + .pgoff_is_anon = false, }; if (invalid_mkclean_vma(vma, NULL)) -- cgit v1.2.3 From e7006fbd608f490f1a687df9d724de4454f8e164 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:28 +0100 Subject: mm: clean up vma_address_end() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit vma_address_end() is a confusing function with a lot of moving parts so clean it up prior to extending it for anon page indexed mappings. Const-ify some variables and establish pgoff_vma_start and pgoff_end variables to clearly identify the page offset for the start of the VMA and the end of the page offset range specified by the page walk. This simplifies the function significantly and lays the groundwork for a future change to update this function to account for anonymously page indexed folios. No functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-11-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Gregory Price (Meta) Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- mm/internal.h | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index f6475d77024d..a38b5f7896b2 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1091,22 +1091,24 @@ static inline unsigned long vma_anon_address(const struct vm_area_struct *vma, } /* - * Then at what user virtual address will none of the range be found in vma? + * At what user virtual address will none of the range be found in vma? * Assumes that vma_address() already returned a good starting address. */ static inline unsigned long vma_address_end(struct page_vma_mapped_walk *pvmw) { - struct vm_area_struct *vma = pvmw->vma; - pgoff_t pgoff; + const pgoff_t pgoff_end = pvmw->pgoff + pvmw->nr_pages; + const struct vm_area_struct *vma = pvmw->vma; + pgoff_t pgoff_vma_start; unsigned long address; /* Common case, plus ->pgoff is invalid for KSM */ if (pvmw->nr_pages == 1) return pvmw->address + PAGE_SIZE; - pgoff = pvmw->pgoff + pvmw->nr_pages; + pgoff_vma_start = vma_start_pgoff(vma); + address = vma->vm_start + - ((pgoff - vma_start_pgoff(vma)) << PAGE_SHIFT); + ((pgoff_end - pgoff_vma_start) << PAGE_SHIFT); /* Check for address beyond vma (or wrapped through 0?) */ if (address < vma->vm_start || address > vma->vm_end) address = vma->vm_end; -- cgit v1.2.3 From 8e658ecc3be21e43573e6639af2d1c536bbfe67d Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:29 +0100 Subject: mm/huge_memory: update remove_migration_pmd() to accept a folio MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This function does not need to accept a page and requiring it to is unnecessary and misleading. make_[writable, readable]_device_private_entry() must be passed a PMD-aligned PFN as they immediately used to obtain a softleaf PMD entry and the same argument applies to folio_add_[anon, file]_rmap_pmd(). While we are here, update a VM_BUG_ON() to a VM_WARN_ON_ONCE(). No functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-12-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Gregory Price (Meta) Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- include/linux/swapops.h | 6 +++--- mm/huge_memory.c | 16 +++++++--------- mm/migrate.c | 2 +- 3 files changed, 11 insertions(+), 13 deletions(-) diff --git a/include/linux/swapops.h b/include/linux/swapops.h index c956bc445ee0..1f3ff3b93e16 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -325,8 +325,8 @@ struct page_vma_mapped_walk; extern int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, struct page *page); -extern void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, - struct page *new); +void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, + struct folio *folio); extern void pmd_migration_entry_wait(struct mm_struct *mm, pmd_t *pmd); @@ -346,7 +346,7 @@ static inline int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, } static inline void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, - struct page *new) + struct folio *folio) { BUILD_BUG(); } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index ff13b57d9d56..2822190daf2b 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -5077,9 +5077,8 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, return 0; } -void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) +void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct folio *folio) { - struct folio *folio = page_folio(new); struct vm_area_struct *vma = pvmw->vma; struct mm_struct *mm = vma->vm_mm; unsigned long address = pvmw->address; @@ -5115,11 +5114,9 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) swp_entry_t entry; if (pmd_write(pmde)) - entry = make_writable_device_private_entry( - page_to_pfn(new)); + entry = make_writable_device_private_entry(folio_pfn(folio)); else - entry = make_readable_device_private_entry( - page_to_pfn(new)); + entry = make_readable_device_private_entry(folio_pfn(folio)); pmde = softleaf_to_pmd(entry); if (pmd_swp_soft_dirty(*pvmw->pmd)) @@ -5134,11 +5131,12 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) if (!softleaf_is_migration_read(entry)) rmap_flags |= RMAP_EXCLUSIVE; - folio_add_anon_rmap_pmd(folio, new, vma, haddr, rmap_flags); + folio_add_anon_rmap_pmd(folio, &folio->page, vma, haddr, rmap_flags); } else { - folio_add_file_rmap_pmd(folio, new, vma); + folio_add_file_rmap_pmd(folio, &folio->page, vma); } - VM_BUG_ON(pmd_write(pmde) && folio_test_anon(folio) && !PageAnonExclusive(new)); + VM_WARN_ON_ONCE(pmd_write(pmde) && folio_test_anon(folio) && + !PageAnonExclusive(&folio->page)); set_pmd_at(mm, haddr, pvmw->pmd, pmde); /* No need to invalidate - it was non-present before */ diff --git a/mm/migrate.c b/mm/migrate.c index 8aaafcea7bc1..9e32af3fe303 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -372,7 +372,7 @@ static bool remove_migration_pte(struct folio *folio, if (!pvmw.pte) { VM_BUG_ON_FOLIO(folio_test_hugetlb(folio) || !folio_test_pmd_mappable(folio), folio); - remove_migration_pmd(&pvmw, new); + remove_migration_pmd(&pvmw, folio); continue; } #endif -- cgit v1.2.3 From 5f653f8b7a3469524b1dd9fbd4b803baa34f14a7 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:30 +0100 Subject: mm/migrate: calculate large folio page index using PFN MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rather than having to figure out the page index to use using linear_page_index(), calculate it using PFN. This is a more natural fit as the linear page index is immaterial to determining the folio page index. Derive the page index from the offset between migration entry PFN and folio PFN - pvmw.pfn (set via DEFINE_FOLIO_VMA_WALK() which uses folio_pfn() to obtain it). Additionally remove a not so useful comment and clean the code layout up. No functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-13-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Suggested-by: David Hildenbrand (Arm) Acked-by: David Hildenbrand (Arm) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Gregory Price (Meta) Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- mm/migrate.c | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 9e32af3fe303..15b45832bcfa 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -356,16 +356,11 @@ static bool remove_migration_pte(struct folio *folio, while (page_vma_mapped_walk(&pvmw)) { rmap_t rmap_flags = RMAP_NONE; - pte_t old_pte; - pte_t pte; + unsigned long idx = 0; softleaf_t entry; struct page *new; - unsigned long idx = 0; - - /* pgoff is invalid for ksm pages, but they are never large */ - if (folio_test_large(folio) && !folio_test_hugetlb(folio)) - idx = linear_page_index(vma, pvmw.address) - pvmw.pgoff; - new = folio_page(folio, idx); + pte_t old_pte; + pte_t pte; #ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES /* PMD-mapped THP migration entry */ @@ -381,14 +376,18 @@ static bool remove_migration_pte(struct folio *folio, pvmw.pte); else old_pte = ptep_get(pvmw.pte); + + entry = softleaf_from_pte(old_pte); + if (folio_test_large(folio) && !folio_test_hugetlb(folio)) + idx = softleaf_to_pfn(entry) - pvmw.pfn; + if (rmap_walk_arg->map_unused_to_zeropage && try_to_map_unused_to_zeropage(&pvmw, folio, old_pte, idx)) continue; folio_get(folio); + new = folio_page(folio, idx); pte = mk_pte(new, READ_ONCE(vma->vm_page_prot)); - - entry = softleaf_from_pte(old_pte); if (!softleaf_is_migration_young(entry)) pte = pte_mkold(pte); if (folio_test_dirty(folio) && softleaf_is_migration_dirty(entry)) -- cgit v1.2.3 From 93c0c8dc87f6eb9f6ce71c4ac379bef88965192c Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:31 +0100 Subject: mm/rmap: use anon pgoff to track MAP_PRIVATE file-backed anon folios MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Currently anonymous folios belonging to CoW'd MAP_PRIVATE file-backed mappings are indexed by their page offset within the file in which they were originally mapped. This differs from anonymous folios belonging to pure anon mappings which are indexed by their anonymous page offset (the address at which they'd belong in the VMA when first faulted). This change fixes this inconsistency, always indexing anonymous folios by their anonymous page offset regardless of the VMA to which they belong. The foundations have been laid such that we need only switch this functionality on such by: * Using linear_anon_page_index() in __folio_set_anon() to assign the folio's index to the anonymous linear index rather than the file-backed one. * Otherwise using linear_anon_page_index() in all instances where anonymous folios are being referenced or manipulated. * Replacing vma_address() with vma_filebacked_address() or vma_anon_address() as appropriate. * Updating the merging logic to check that anonymous page offsets are aligned as well as filebacked ones for MAP_PRIVATE file-backed VMAs, introducing needs_adjacent_anon_pgoff() to figure out when this is required. * Updating linear_folio_page_index() to invoke linear_anon_page_index() if the folio is anonymous. * Updating vma_address_end() to use the VMA's anonymous page offset when pvmw->pgoff is anonymous. * Correcting folio_within_range() to use anonymous page offset for anonymous folios. This will have no impact on merging of anonymous VMAs, whose page offset and anonymous page offset are identical, nor will it impact shared file-backed VMAs, which will continue to be merged based on the file-backed page offset. However, MAP_PRIVATE file-backed mappings must now be aligned on anonymous page offset as well. In most instances this should have no impact on merging of file-backed mappings, which are usually not merged all that often, let alone MAP_PRIVATE mapped ones, and rarely remapped and faulted before being moved back in place (the case in which a merge may now fail). One subtle impact of this change is in NUMA interleaving - since commit 88c91dc58582 ("mempolicy: migration attempt to match interleave nodes"), migration heuristically tries to maintain interleaving behaviour matching the policy using folio indices. When doing migration of CoW'd MAP_PRIVATE-file backed ranges, the 'base' upon which the interleaving behaviour is performed will vary for these ranges. However the commit notes that ranges spanning multiple VMAs will already cause varying bases, and that this is an acceptable approximation. It is very unlikely real world use-cases will be impacted by this (MAP_PRIVATE file-backed mappings are already an edge case), and all that will happen is that such ranges will cause interleaving to be rotated over the CoW'd range, with little to no impact. This commit lays the foundations for future scalable CoW work which needs to track some remaps, meaning that most remap tracking can be avoided, and in nearly all cases the anonymous page offset will be able to be used to quickly find the VMA in an mm. Note that the need_rmap_locks check doesn't need to be updated, as any remapping will offset both the anonymous and file-backed page offset, so it suffices to check only one. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-14-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Gregory Price (Meta) Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- mm/huge_memory.c | 2 +- mm/internal.h | 27 ++++++++------------------- mm/interval_tree.c | 4 ++-- mm/ksm.c | 6 +++--- mm/page_vma_mapped.c | 2 +- mm/rmap.c | 12 ++++++------ mm/userfaultfd.c | 4 ++-- mm/vma.c | 32 +++++++++++++++++++++++++++++++- 8 files changed, 54 insertions(+), 35 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 2822190daf2b..a8174d1d3848 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2930,7 +2930,7 @@ int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm } folio_move_anon_rmap(src_folio, dst_vma); - src_folio->index = linear_page_index(dst_vma, dst_addr); + src_folio->index = linear_anon_page_index(dst_vma, dst_addr); _dst_pmd = folio_mk_pmd(src_folio, dst_vma->vm_page_prot); /* Follow mremap() behavior and treat the entry dirty after the move */ diff --git a/mm/internal.h b/mm/internal.h index a38b5f7896b2..16750b130ec4 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -945,7 +945,8 @@ folio_within_range(struct folio *folio, struct vm_area_struct *vma, return false; pgoff_folio = folio_pgoff(folio); - pgoff_vma_start = vma_start_pgoff(vma); + pgoff_vma_start = folio_test_anon(folio) ? + vma_start_anon_pgoff(vma) : vma_start_pgoff(vma); if (start < vma->vm_start) start = vma->vm_start; @@ -1056,23 +1057,8 @@ static inline unsigned long vma_filebacked_address(const struct vm_area_struct * } /** - * vma_address - Find the virtual address a page range is mapped at. - * @vma: The vma which maps this object. - * @pgoff: The page offset within its object. - * @nr_pages: The number of pages to consider. - * - * If any page in this range is mapped by this VMA, return the first address - * where any of these pages appear. Otherwise, return -EFAULT. - */ -static inline unsigned long vma_address(const struct vm_area_struct *vma, - pgoff_t pgoff, unsigned long nr_pages) -{ - return __vma_address(vma, pgoff, vma_start_pgoff(vma), nr_pages); -} - -/** - * vma_anon_address - Find the address an anonymous folio with index @pgoff_anon - * is mapped at. + * vma_anon_address - Find the virtual address an anonymous page range is mapped + * at. * @vma: The vma which maps this object. * @pgoff_anon: The anonymous page index belonging to the folio. * @nr_pages: The number of pages to consider. @@ -1105,7 +1091,10 @@ static inline unsigned long vma_address_end(struct page_vma_mapped_walk *pvmw) if (pvmw->nr_pages == 1) return pvmw->address + PAGE_SIZE; - pgoff_vma_start = vma_start_pgoff(vma); + if (pvmw->pgoff_is_anon) + pgoff_vma_start = vma_start_anon_pgoff(vma); + else + pgoff_vma_start = vma_start_pgoff(vma); address = vma->vm_start + ((pgoff_end - pgoff_vma_start) << PAGE_SHIFT); diff --git a/mm/interval_tree.c b/mm/interval_tree.c index 3ae9e106d3af..7bbbf15cfbf0 100644 --- a/mm/interval_tree.c +++ b/mm/interval_tree.c @@ -83,12 +83,12 @@ mapping_rmap_tree_iter_next(struct vm_area_struct *vma, static pgoff_t avc_start_pgoff(struct anon_vma_chain *avc) { - return vma_start_pgoff(avc->vma); + return vma_start_anon_pgoff(avc->vma); } static pgoff_t avc_last_pgoff(struct anon_vma_chain *avc) { - return vma_last_pgoff(avc->vma); + return vma_last_anon_pgoff(avc->vma); } INTERVAL_TREE_DEFINE(struct anon_vma_chain, rb, pgoff_t, rb_subtree_last, diff --git a/mm/ksm.c b/mm/ksm.c index b4142746777e..b5854dc14a2e 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1625,7 +1625,7 @@ static int try_to_merge_with_ksm_page(struct ksm_rmap_item *rmap_item, * stable_tree, break_cow() will clean it up. */ rmap_item->anon_vma = vma->anon_vma; - rmap_item->linear_page_index = linear_page_index(vma, rmap_item->address); + rmap_item->linear_page_index = linear_anon_page_index(vma, rmap_item->address); get_anon_vma(vma->anon_vma); out: mmap_read_unlock(mm); @@ -3152,7 +3152,7 @@ struct folio *ksm_might_need_to_copy(struct folio *folio, return folio; /* no need to copy it */ } else if (!anon_vma) { return folio; /* no need to copy it */ - } else if (folio->index == linear_page_index(vma, addr) && + } else if (folio->index == linear_anon_page_index(vma, addr) && anon_vma->root == vma->anon_vma->root) { return folio; /* still no need to copy it */ } @@ -3222,7 +3222,7 @@ again: /* * Currently, KSM folios are always small folios, so it's * sufficient to search for a single page. We can simply use - * the linear_page_index of the original de-duplicate + * the linear_anon_page_index of the original de-duplicate * anonymous page that we remembered in the rmap_item while * de-duplicating. Note that mremap() always de-duplicates KSM * folios: so if there was mremap() in our parent or our child, diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 081e483cc7bf..4e964545e5e8 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -365,7 +365,7 @@ unsigned long page_mapped_in_vma(const struct page *page, }; if (folio_test_anon(folio)) - pvmw.address = vma_address(vma, pgoff, 1); + pvmw.address = vma_anon_address(vma, pgoff, 1); else pvmw.address = vma_filebacked_address(vma, pgoff, 1); if (pvmw.address == -EFAULT) diff --git a/mm/rmap.c b/mm/rmap.c index 1b23ac709f45..34ceeb600111 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -866,7 +866,7 @@ unsigned long page_address_in_vma(const struct folio *folio, vma->anon_vma->root != anon_vma->root) return -EFAULT; /* KSM folios don't reach here because of the !anon_vma check */ - return vma_address(vma, page_pgoff(folio, page), 1); + return vma_anon_address(vma, page_pgoff(folio, page), 1); } else if (!vma->vm_file) { return -EFAULT; } else if (vma->vm_file->f_mapping != folio->mapping) { @@ -1485,7 +1485,7 @@ static void __folio_set_anon(struct folio *folio, struct vm_area_struct *vma, */ anon_vma = (void *) anon_vma + FOLIO_MAPPING_ANON; WRITE_ONCE(folio->mapping, (struct address_space *) anon_vma); - folio->index = linear_page_index(vma, address); + folio->index = linear_anon_page_index(vma, address); } /** @@ -1512,8 +1512,8 @@ static void __page_check_anon_rmap(const struct folio *folio, */ VM_BUG_ON_FOLIO(folio_anon_vma(folio)->root != vma->anon_vma->root, folio); - VM_BUG_ON_PAGE(page_pgoff(folio, page) != linear_page_index(vma, address), - page); + VM_BUG_ON_PAGE(page_pgoff(folio, page) != + linear_anon_page_index(vma, address), page); } static __always_inline void __folio_add_anon_rmap(struct folio *folio, @@ -3038,10 +3038,10 @@ static void rmap_walk_anon(struct folio *folio, pgoff_end = pgoff_start + folio_nr_pages(folio) - 1; anon_rmap_tree_foreach(avc, anon_vma, pgoff_start, pgoff_end) { struct vm_area_struct *vma = avc->vma; - unsigned long address = vma_address(vma, pgoff_start, + const unsigned long address = vma_anon_address(vma, pgoff_start, folio_nr_pages(folio)); - VM_BUG_ON_VMA(address == -EFAULT, vma); + VM_WARN_ON_ONCE_VMA(address == -EFAULT, vma); cond_resched(); if (rwc->invalid_vma && rwc->invalid_vma(vma, rwc->arg)) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index de4cc2483562..23fb68fce000 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -1352,7 +1352,7 @@ static long move_present_ptes(struct mm_struct *mm, } folio_move_anon_rmap(src_folio, dst_vma); - src_folio->index = linear_page_index(dst_vma, dst_addr); + src_folio->index = linear_anon_page_index(dst_vma, dst_addr); orig_dst_pte = folio_mk_pte(src_folio, dst_vma->vm_page_prot); /* Set soft dirty bit so userspace can notice the pte was moved */ @@ -1428,7 +1428,7 @@ static int move_swap_pte(struct mm_struct *mm, struct vm_area_struct *dst_vma, */ if (src_folio) { folio_move_anon_rmap(src_folio, dst_vma); - src_folio->index = linear_page_index(dst_vma, dst_addr); + src_folio->index = linear_anon_page_index(dst_vma, dst_addr); } else { /* * Check if the swap entry is cached after acquiring the src_pte diff --git a/mm/vma.c b/mm/vma.c index e35b04ac12cb..35e7a64855fa 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -204,6 +204,25 @@ static void init_multi_vma_prep(struct vma_prepare *vp, vp->skip_vma_uprobe = true; } +/* + * Does this merge require that adjacent VMAs must have adjacent anonymous page + * offsets in addition to having adjacent vma->vm_pgoff? + * + * This is only required for MAP_PRIVATE-file backed mappings as the page offset + * for pure anonymous VMAs is equal to the anonymous page offset. + * + * Read-only shared mappings (with VMA_SHARED_BIT cleared) are always unfaulted + * so automatically have correct anonymous page offset (as it is always updated + * on remap). + * + * 'Special' mappings in the sense of VDSO, VVAR etc. have !file but would in + * any case not be candidates for merge nor be mergeable. + */ +static bool needs_adjacent_anon_pgoff(const struct vma_merge_struct *vmg) +{ + return vmg->file && vma_flags_is_cow_mapping(&vmg->vma_flags); +} + /* * Return true if we can merge this (vma_flags,anon_vma,file,vm_pgoff) * in front of (at a lower virtual address and file offset than) the vma. @@ -225,6 +244,9 @@ static bool can_vma_merge_before(struct vma_merge_struct *vmg) return false; if (vmg_end_pgoff(vmg) != vma_start_pgoff(vmg->next)) return false; + if (needs_adjacent_anon_pgoff(vmg) && + vmg_end_anon_pgoff(vmg) != vma_start_anon_pgoff(vmg->next)) + return false; return true; } @@ -245,6 +267,9 @@ static bool can_vma_merge_after(struct vma_merge_struct *vmg) return false; if (vma_end_pgoff(vmg->prev) != vmg_start_pgoff(vmg)) return false; + if (needs_adjacent_anon_pgoff(vmg) && + vma_end_anon_pgoff(vmg->prev) != vmg_start_anon_pgoff(vmg)) + return false; return true; } @@ -2048,7 +2073,12 @@ static int anon_vma_compatible(struct vm_area_struct *a, struct vm_area_struct * if (!vma_flags_empty(&diff)) return false; /* Page offset must align. */ - return vma_end_pgoff(a) == vma_start_pgoff(b); + if (vma_end_pgoff(a) != vma_start_pgoff(b)) + return false; + /* Only reached from anon path, so either MAP_PRIVATE file or anon. */ + if (vma_end_anon_pgoff(a) != vma_start_anon_pgoff(b)) + return false; + return true; } /* -- cgit v1.2.3 From 6b7460ad1af8b10cc6b629649979d9d0d844046e Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:32 +0100 Subject: tools/testing/vma: expand VMA merge tests to assert anon pgoff MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Now we have introduced the VMA anonymous page offset attribute and update it when VMAs are manipulated, update VMA merge tests to assert that the anonymous page offset is as expected. Also update instances where we could use vma_start_pgoff() to do so. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-15-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Gregory Price (Meta) Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- tools/testing/vma/tests/merge.c | 45 ++++++++++++++++++++++++++++++++--------- 1 file changed, 36 insertions(+), 9 deletions(-) diff --git a/tools/testing/vma/tests/merge.c b/tools/testing/vma/tests/merge.c index 48418b82b01d..acaab282939c 100644 --- a/tools/testing/vma/tests/merge.c +++ b/tools/testing/vma/tests/merge.c @@ -121,6 +121,7 @@ static bool test_simple_merge(void) ASSERT_EQ(vma->vm_start, 0); ASSERT_EQ(vma->vm_end, 0x3000); ASSERT_EQ(vma_start_pgoff(vma), 0); + ASSERT_EQ(vma_start_anon_pgoff(vma), 0); ASSERT_FLAGS_SAME_MASK(&vma->flags, vma_flags); detach_free_vma(vma); @@ -153,6 +154,7 @@ static bool test_simple_modify(void) ASSERT_EQ(vma->vm_start, 0x1000); ASSERT_EQ(vma->vm_end, 0x2000); ASSERT_EQ(vma_start_pgoff(vma), 1); + ASSERT_EQ(vma_start_anon_pgoff(vma), 1); /* * Now walk through the three split VMAs and make sure they are as @@ -165,6 +167,7 @@ static bool test_simple_modify(void) ASSERT_EQ(vma->vm_start, 0); ASSERT_EQ(vma->vm_end, 0x1000); ASSERT_EQ(vma_start_pgoff(vma), 0); + ASSERT_EQ(vma_start_anon_pgoff(vma), 0); detach_free_vma(vma); vma_iter_clear(&vmi); @@ -174,6 +177,7 @@ static bool test_simple_modify(void) ASSERT_EQ(vma->vm_start, 0x1000); ASSERT_EQ(vma->vm_end, 0x2000); ASSERT_EQ(vma_start_pgoff(vma), 1); + ASSERT_EQ(vma_start_anon_pgoff(vma), 1); detach_free_vma(vma); vma_iter_clear(&vmi); @@ -183,6 +187,7 @@ static bool test_simple_modify(void) ASSERT_EQ(vma->vm_start, 0x2000); ASSERT_EQ(vma->vm_end, 0x3000); ASSERT_EQ(vma_start_pgoff(vma), 2); + ASSERT_EQ(vma_start_anon_pgoff(vma), 2); detach_free_vma(vma); mtree_destroy(&mm.mm_mt); @@ -212,6 +217,7 @@ static bool test_simple_expand(void) ASSERT_EQ(vma->vm_start, 0); ASSERT_EQ(vma->vm_end, 0x3000); ASSERT_EQ(vma_start_pgoff(vma), 0); + ASSERT_EQ(vma_start_anon_pgoff(vma), 0); detach_free_vma(vma); mtree_destroy(&mm.mm_mt); @@ -234,6 +240,7 @@ static bool test_simple_shrink(void) ASSERT_EQ(vma->vm_start, 0); ASSERT_EQ(vma->vm_end, 0x1000); ASSERT_EQ(vma_start_pgoff(vma), 0); + ASSERT_EQ(vma_start_anon_pgoff(vma), 0); detach_free_vma(vma); mtree_destroy(&mm.mm_mt); @@ -346,6 +353,7 @@ static bool __test_merge_new(bool is_sticky, bool a_is_sticky, bool b_is_sticky, ASSERT_EQ(vma->vm_start, 0); ASSERT_EQ(vma->vm_end, 0x5000); ASSERT_EQ(vma_start_pgoff(vma), 0); + ASSERT_EQ(vma_start_anon_pgoff(vma), 0); ASSERT_EQ(vma->anon_vma, &dummy_anon_vma); ASSERT_TRUE(vma_write_started(vma)); ASSERT_EQ(mm.map_count, 3); @@ -367,6 +375,7 @@ static bool __test_merge_new(bool is_sticky, bool a_is_sticky, bool b_is_sticky, ASSERT_EQ(vma->vm_start, 0x6000); ASSERT_EQ(vma->vm_end, 0x9000); ASSERT_EQ(vma_start_pgoff(vma), 6); + ASSERT_EQ(vma_start_anon_pgoff(vma), 6); ASSERT_EQ(vma->anon_vma, &dummy_anon_vma); ASSERT_TRUE(vma_write_started(vma)); ASSERT_EQ(mm.map_count, 3); @@ -387,6 +396,7 @@ static bool __test_merge_new(bool is_sticky, bool a_is_sticky, bool b_is_sticky, ASSERT_EQ(vma->vm_start, 0); ASSERT_EQ(vma->vm_end, 0x9000); ASSERT_EQ(vma_start_pgoff(vma), 0); + ASSERT_EQ(vma_start_anon_pgoff(vma), 0); ASSERT_EQ(vma->anon_vma, &dummy_anon_vma); ASSERT_TRUE(vma_write_started(vma)); ASSERT_EQ(mm.map_count, 2); @@ -407,6 +417,7 @@ static bool __test_merge_new(bool is_sticky, bool a_is_sticky, bool b_is_sticky, ASSERT_EQ(vma->vm_start, 0xa000); ASSERT_EQ(vma->vm_end, 0xc000); ASSERT_EQ(vma_start_pgoff(vma), 0xa); + ASSERT_EQ(vma_start_anon_pgoff(vma), 0xa); ASSERT_EQ(vma->anon_vma, &dummy_anon_vma); ASSERT_TRUE(vma_write_started(vma)); ASSERT_EQ(mm.map_count, 2); @@ -426,6 +437,7 @@ static bool __test_merge_new(bool is_sticky, bool a_is_sticky, bool b_is_sticky, ASSERT_EQ(vma->vm_start, 0); ASSERT_EQ(vma->vm_end, 0xc000); ASSERT_EQ(vma_start_pgoff(vma), 0); + ASSERT_EQ(vma_start_anon_pgoff(vma), 0); ASSERT_EQ(vma->anon_vma, &dummy_anon_vma); ASSERT_TRUE(vma_write_started(vma)); ASSERT_EQ(mm.map_count, 1); @@ -446,6 +458,7 @@ static bool __test_merge_new(bool is_sticky, bool a_is_sticky, bool b_is_sticky, ASSERT_EQ(vma->vm_start, 0); ASSERT_EQ(vma->vm_end, 0xc000); ASSERT_EQ(vma_start_pgoff(vma), 0); + ASSERT_EQ(vma_start_anon_pgoff(vma), 0); ASSERT_EQ(vma->anon_vma, &dummy_anon_vma); detach_free_vma(vma); @@ -642,7 +655,8 @@ static bool test_vma_merge_with_close(void) ASSERT_EQ(vmg.state, VMA_MERGE_SUCCESS); ASSERT_EQ(vma_prev->vm_start, 0); ASSERT_EQ(vma_prev->vm_end, 0x5000); - ASSERT_EQ(vma_prev->vm_pgoff, 0); + ASSERT_EQ(vma_start_pgoff(vma_prev), 0); + ASSERT_EQ(vma_start_anon_pgoff(vma_prev), 0); ASSERT_EQ(cleanup_mm(&mm, &vmi), 2); @@ -753,7 +767,8 @@ static bool test_vma_merge_with_close(void) ASSERT_EQ(vmg.state, VMA_MERGE_SUCCESS); ASSERT_EQ(vma_prev->vm_start, 0); ASSERT_EQ(vma_prev->vm_end, 0x5000); - ASSERT_EQ(vma_prev->vm_pgoff, 0); + ASSERT_EQ(vma_start_pgoff(vma_prev), 0); + ASSERT_EQ(vma_start_anon_pgoff(vma_prev), 0); ASSERT_EQ(cleanup_mm(&mm, &vmi), 2); @@ -808,6 +823,7 @@ static bool test_vma_merge_new_with_close(void) ASSERT_EQ(vma->vm_start, 0); ASSERT_EQ(vma->vm_end, 0x5000); ASSERT_EQ(vma_start_pgoff(vma), 0); + ASSERT_EQ(vma_start_anon_pgoff(vma), 0); ASSERT_EQ(vma->vm_ops, &vm_ops); ASSERT_TRUE(vma_write_started(vma)); ASSERT_EQ(mm.map_count, 2); @@ -863,11 +879,13 @@ static bool __test_merge_existing(bool prev_is_sticky, bool middle_is_sticky, bo ASSERT_EQ(vmg.state, VMA_MERGE_SUCCESS); ASSERT_EQ(vma_next->vm_start, 0x3000); ASSERT_EQ(vma_next->vm_end, 0x9000); - ASSERT_EQ(vma_next->vm_pgoff, 3); + ASSERT_EQ(vma_start_pgoff(vma_next), 3); + ASSERT_EQ(vma_start_anon_pgoff(vma_next), 3); ASSERT_EQ(vma_next->anon_vma, &dummy_anon_vma); ASSERT_EQ(vma->vm_start, 0x2000); ASSERT_EQ(vma->vm_end, 0x3000); ASSERT_EQ(vma_start_pgoff(vma), 2); + ASSERT_EQ(vma_start_anon_pgoff(vma), 2); ASSERT_TRUE(vma_write_started(vma)); ASSERT_TRUE(vma_write_started(vma_next)); ASSERT_EQ(mm.map_count, 2); @@ -897,7 +915,8 @@ static bool __test_merge_existing(bool prev_is_sticky, bool middle_is_sticky, bo ASSERT_EQ(vmg.state, VMA_MERGE_SUCCESS); ASSERT_EQ(vma_next->vm_start, 0x2000); ASSERT_EQ(vma_next->vm_end, 0x9000); - ASSERT_EQ(vma_next->vm_pgoff, 2); + ASSERT_EQ(vma_start_pgoff(vma_next), 2); + ASSERT_EQ(vma_start_anon_pgoff(vma_next), 2); ASSERT_EQ(vma_next->anon_vma, &dummy_anon_vma); ASSERT_TRUE(vma_write_started(vma_next)); ASSERT_EQ(mm.map_count, 1); @@ -929,11 +948,13 @@ static bool __test_merge_existing(bool prev_is_sticky, bool middle_is_sticky, bo ASSERT_EQ(vmg.state, VMA_MERGE_SUCCESS); ASSERT_EQ(vma_prev->vm_start, 0); ASSERT_EQ(vma_prev->vm_end, 0x6000); - ASSERT_EQ(vma_prev->vm_pgoff, 0); + ASSERT_EQ(vma_start_pgoff(vma_prev), 0); + ASSERT_EQ(vma_start_anon_pgoff(vma_prev), 0); ASSERT_EQ(vma_prev->anon_vma, &dummy_anon_vma); ASSERT_EQ(vma->vm_start, 0x6000); ASSERT_EQ(vma->vm_end, 0x7000); ASSERT_EQ(vma_start_pgoff(vma), 6); + ASSERT_EQ(vma_start_anon_pgoff(vma), 6); ASSERT_TRUE(vma_write_started(vma_prev)); ASSERT_TRUE(vma_write_started(vma)); ASSERT_EQ(mm.map_count, 2); @@ -964,7 +985,8 @@ static bool __test_merge_existing(bool prev_is_sticky, bool middle_is_sticky, bo ASSERT_EQ(vmg.state, VMA_MERGE_SUCCESS); ASSERT_EQ(vma_prev->vm_start, 0); ASSERT_EQ(vma_prev->vm_end, 0x7000); - ASSERT_EQ(vma_prev->vm_pgoff, 0); + ASSERT_EQ(vma_start_pgoff(vma_prev), 0); + ASSERT_EQ(vma_start_anon_pgoff(vma_prev), 0); ASSERT_EQ(vma_prev->anon_vma, &dummy_anon_vma); ASSERT_TRUE(vma_write_started(vma_prev)); ASSERT_EQ(mm.map_count, 1); @@ -996,7 +1018,8 @@ static bool __test_merge_existing(bool prev_is_sticky, bool middle_is_sticky, bo ASSERT_EQ(vmg.state, VMA_MERGE_SUCCESS); ASSERT_EQ(vma_prev->vm_start, 0); ASSERT_EQ(vma_prev->vm_end, 0x9000); - ASSERT_EQ(vma_prev->vm_pgoff, 0); + ASSERT_EQ(vma_start_pgoff(vma_prev), 0); + ASSERT_EQ(vma_start_anon_pgoff(vma_prev), 0); ASSERT_EQ(vma_prev->anon_vma, &dummy_anon_vma); ASSERT_TRUE(vma_write_started(vma_prev)); ASSERT_EQ(mm.map_count, 1); @@ -1126,7 +1149,8 @@ static bool test_anon_vma_non_mergeable(void) ASSERT_EQ(vmg.state, VMA_MERGE_SUCCESS); ASSERT_EQ(vma_prev->vm_start, 0); ASSERT_EQ(vma_prev->vm_end, 0x7000); - ASSERT_EQ(vma_prev->vm_pgoff, 0); + ASSERT_EQ(vma_start_pgoff(vma_prev), 0); + ASSERT_EQ(vma_start_anon_pgoff(vma_prev), 0); ASSERT_TRUE(vma_write_started(vma_prev)); ASSERT_FALSE(vma_write_started(vma_next)); @@ -1157,7 +1181,8 @@ static bool test_anon_vma_non_mergeable(void) ASSERT_EQ(vmg.state, VMA_MERGE_SUCCESS); ASSERT_EQ(vma_prev->vm_start, 0); ASSERT_EQ(vma_prev->vm_end, 0x7000); - ASSERT_EQ(vma_prev->vm_pgoff, 0); + ASSERT_EQ(vma_start_pgoff(vma_prev), 0); + ASSERT_EQ(vma_start_anon_pgoff(vma_prev), 0); ASSERT_TRUE(vma_write_started(vma_prev)); ASSERT_FALSE(vma_write_started(vma_next)); @@ -1419,6 +1444,7 @@ static bool test_merge_extend(void) ASSERT_EQ(vma->vm_start, 0); ASSERT_EQ(vma->vm_end, 0x4000); ASSERT_EQ(vma_start_pgoff(vma), 0); + ASSERT_EQ(vma_start_anon_pgoff(vma), 0); ASSERT_TRUE(vma_write_started(vma)); ASSERT_EQ(mm.map_count, 1); @@ -1459,6 +1485,7 @@ static bool test_expand_only_mode(void) ASSERT_EQ(vma->vm_start, 0x3000); ASSERT_EQ(vma->vm_end, 0x9000); ASSERT_EQ(vma_start_pgoff(vma), 3); + ASSERT_EQ(vma_start_anon_pgoff(vma), 3); ASSERT_TRUE(vma_write_started(vma)); ASSERT_EQ(vma_iter_addr(&vmi), 0x3000); vma_assert_attached(vma); -- cgit v1.2.3 From fb580e196497738d98dbad5b8459913435b376e7 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:33 +0100 Subject: tools/testing/selftests/mm: test anonymous page offset merge behaviour MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit While maintaining anonymous page offsets for VMAs has no impact for most merge cases, it does impact MAP_PRIVATE-mapped file-backed mappings which happen to have matching page offset but not matching anonymous page offset. Assert this behaviour by attempting to map an unfaulted MAP_PRIVATE-memfd region with a faulted one with compatible file page offsets but incompatible anonymous page offsets. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-16-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Gregory Price (Meta) Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- tools/testing/selftests/mm/merge.c | 57 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/tools/testing/selftests/mm/merge.c b/tools/testing/selftests/mm/merge.c index 519e5ac02db7..52b8727b6628 100644 --- a/tools/testing/selftests/mm/merge.c +++ b/tools/testing/selftests/mm/merge.c @@ -1305,6 +1305,63 @@ TEST_F(merge, merge_vmas_with_mseal) ASSERT_EQ(procmap->query.vma_end, (unsigned long)ptr + 2 * page_size); } +TEST_F(merge, anon_and_page_offset_mismatch_memfd) +{ + struct procmap_fd *procmap = &self->procmap; + unsigned int page_size = self->page_size; + char *carveout = self->carveout; + char *ptr, *ptr2; + int fd; + + /* Create a 10 page memfd descriptor. */ + fd = memfd_create("anon_page_offset_test", MFD_CLOEXEC); + ASSERT_NE(fd, -1); + ASSERT_EQ(ftruncate(fd, 10 * page_size), 0); + + /* Map a region using the memfd at page offset 0. */ + ptr = mmap(carveout, 5 * page_size, PROT_READ | PROT_WRITE, + MAP_FIXED | MAP_PRIVATE, fd, 0); + ASSERT_NE(ptr, MAP_FAILED); + + /* + * Map another separately and trigger a CoW fault at page offset 5: + * + * |-----------| |---------| + * | unfaulted | | faulted | + * |-----------| |---------| + */ + ptr2 = mmap(&carveout[10 * page_size], 5 * page_size, + PROT_READ | PROT_WRITE, MAP_FIXED | MAP_PRIVATE, + fd, 5 * page_size); + ASSERT_NE(ptr2, MAP_FAILED); + ptr2[0] = 'x'; + + /* + * Now move it in place: + * + * |----------| + * | | + * v | + * |-----------| |---------| + * | unfaulted | | faulted | + * |-----------| |---------| + * + * Because the anonymous page offset of the faulted region is now + * &carveout[10 * page_size], despite the two regions being mergeable + * due to file page offset, they are NOT mergeable due to anonymous + * page offset. + */ + ptr2 = sys_mremap(ptr2, 5 * page_size, 5 * page_size, + MREMAP_MAYMOVE | MREMAP_FIXED, + &carveout[5 * page_size]); + ASSERT_NE(ptr2, MAP_FAILED); + + /* Assert that they did not merge. */ + ASSERT_TRUE(find_vma_procmap(procmap, ptr)); + ASSERT_EQ(procmap->query.vma_start, (unsigned long)ptr); + ASSERT_EQ(procmap->query.vma_end, (unsigned long)ptr + 5 * page_size); +} + TEST_F(merge_with_fork, mremap_faulted_to_unfaulted_prev) { struct procmap_fd *procmap = &self->procmap; -- cgit v1.2.3 From 3541a2b06ecd78ba333188df04368dcf97273d6a Mon Sep 17 00:00:00 2001 From: Breno Leitao Date: Mon, 20 Jul 2026 06:23:45 -0700 Subject: mm/kmemleak: report RCU-tasks quiescent states during the scan kmemleak_scan() can run for ages on large debug kernels. It was causing some soft-lockups which I got fixed with commit 3175fcfec8b16baeb ("mm/kmemleak: avoid soft lockup when scanning task stacks") with our beloved cond_resched(). I've got the fix above deployed in the Meta fleet, and now I am seeing: INFO: rcu_tasks detected stalls on tasks: task:kmemleak state:R ... nvcsw: 274/274 holdout: 1 idle_cpu: -1/3 scan_block scan_gray_list kmemleak_scan and, worse, blocks the callers waiting on that grace period. Here a BPF struct_ops map free, which waits via synchronize_rcu_mult(call_rcu, call_rcu_tasks), is stuck long enough to also trip the hung task check: INFO: task kworker/...:bpf_map_free_deferred blocked for 122 seconds __wait_rcu_gp bpf_struct_ops_map_free Then I've learned that cond_resched() is not an RCU-tasks quiescent state, so, we need to use stronger primitives. Use cond_resched_tasks_rcu_qs() at the scan reschedule points so the scan reports an RCU-tasks quiescent state as it proceeds. Inspired by commit b96285e10aad ("tracing: Have osnoise_main() add a quiescent state for task rcu"). Link: https://lore.kernel.org/20260720-kmemleak_rcu_task-v1-1-5b460ade777d@debian.org Fixes: c4b28963fd79 ("mm/kmemleak: rely on rcu for task stack scanning") Signed-off-by: Breno Leitao Reviewed-by: Paul E. McKenney Reviewed-by: SJ Park Reviewed-by: Catalin Marinas Cc: Breno Leitao Cc: Puranjay Mohan Cc: Signed-off-by: Andrew Morton --- mm/kmemleak.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/mm/kmemleak.c b/mm/kmemleak.c index e96e9efd19b0..0a6045c857d6 100644 --- a/mm/kmemleak.c +++ b/mm/kmemleak.c @@ -1571,7 +1571,7 @@ static int scan_large_block(void *start, void *end) if (scan_block(start, next, NULL)) return 1; start = next; - cond_resched(); + cond_resched_tasks_rcu_qs(); } return 0; @@ -1608,7 +1608,7 @@ static void scan_object(struct kmemleak_object *object) scan_block(start, end, object); raw_spin_unlock_irqrestore(&object->lock, flags); - cond_resched(); + cond_resched_tasks_rcu_qs(); raw_spin_lock_irqsave(&object->lock, flags); if (!(object->flags & OBJECT_ALLOCATED)) break; @@ -1630,7 +1630,7 @@ static void scan_object(struct kmemleak_object *object) break; raw_spin_unlock_irqrestore(&object->lock, flags); - cond_resched(); + cond_resched_tasks_rcu_qs(); raw_spin_lock_irqsave(&object->lock, flags); } while (object->flags & OBJECT_ALLOCATED); } else { @@ -1658,7 +1658,7 @@ static void scan_gray_list(void) */ object = list_entry(gray_list.next, typeof(*object), gray_list); while (&object->gray_list != &gray_list) { - cond_resched(); + cond_resched_tasks_rcu_qs(); /* may add new objects to the list */ if (!scan_should_stop()) @@ -1693,7 +1693,7 @@ static void kmemleak_cond_resched(struct kmemleak_object *object) raw_spin_unlock_irq(&kmemleak_lock); rcu_read_unlock(); - cond_resched(); + cond_resched_tasks_rcu_qs(); rcu_read_lock(); raw_spin_lock_irq(&kmemleak_lock); @@ -1738,7 +1738,7 @@ static void kmemleak_scan_task_stacks(void) } put_task_struct(p); } - cond_resched(); + cond_resched_tasks_rcu_qs(); } while (pid && !stop); } @@ -1915,7 +1915,7 @@ static void kmemleak_scan(void) struct page *page = pfn_to_online_page(pfn); if (!(pfn & 63)) - cond_resched(); + cond_resched_tasks_rcu_qs(); if (!page) continue; -- cgit v1.2.3 From 3bf07ce8058bcab03ea1cb9cd11ec1d6b2ee5af0 Mon Sep 17 00:00:00 2001 From: Baolin Wang Date: Mon, 20 Jul 2026 19:12:00 +0800 Subject: mm: vmscan: convert folio_referenced() to use vma_flags_t Patch series "promote mapped executable folios after first usage for MGLRU", v4. Now MGLRU's protection of mapped executable file folios is less reliable. Follow the classical LRU's logic, promoting mapped executable file folios after their first usage to give executable code a better chance to stay in memory and improve workload performance (See patch 2 for more details). This patch (of 3): Replace use of the legacy vm_flags_t flags with vma_flags_t values for folio_referenced() and related logic. This is also a preparation for the following changes. No functional changes. Link: https://lore.kernel.org/cover.1784509721.git.baolin.wang@linux.alibaba.com Link: https://lore.kernel.org/2bd39e16ec19e3e3c4716aa9a1a25775c26cac57.1784509721.git.baolin.wang@linux.alibaba.com Signed-off-by: Baolin Wang Acked-by: David Hildenbrand (Arm) Acked-by: Johannes Weiner Reviewed-by: Barry Song Reviewed-by: Kairui Song Reviewed-by: Axel Rasmussen Cc: Harry Yoo Cc: Jann Horn Cc: Lance Yang Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Rik van Riel Cc: Shakeel Butt Cc: Vlastimil Babka Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton --- include/linux/rmap.h | 7 +++---- mm/rmap.c | 19 +++++++++++-------- mm/vmscan.c | 14 +++++++------- 3 files changed, 21 insertions(+), 19 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 0574537a355c..0b332770abee 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -843,7 +843,7 @@ static inline int folio_try_share_anon_rmap_pmd(struct folio *folio, * Called from mm/vmscan.c to handle paging out */ int folio_referenced(struct folio *, int is_locked, - struct mem_cgroup *memcg, vm_flags_t *vm_flags); + struct mem_cgroup *memcg, vma_flags_t *vma_flags); void try_to_migrate(struct folio *folio, enum ttu_flags flags); void try_to_unmap(struct folio *, enum ttu_flags flags); @@ -977,10 +977,9 @@ struct anon_vma *folio_lock_anon_vma_read(const struct folio *folio, #define anon_vma_prepare(vma) (0) static inline int folio_referenced(struct folio *folio, int is_locked, - struct mem_cgroup *memcg, - vm_flags_t *vm_flags) + struct mem_cgroup *memcg, vma_flags_t *vma_flags) { - *vm_flags = 0; + vma_flags_clear_all(vma_flags); return 0; } diff --git a/mm/rmap.c b/mm/rmap.c index 34ceeb600111..14f2f9b07572 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -908,7 +908,7 @@ out: struct folio_referenced_arg { int mapcount; int referenced; - vm_flags_t vm_flags; + vma_flags_t vma_flags; struct mem_cgroup *memcg; }; @@ -927,7 +927,7 @@ static bool folio_referenced_one(struct folio *folio, address = pvmw.address; nr = 1; - if (vma->vm_flags & VM_LOCKED) { + if (vma_test(vma, VMA_LOCKED_BIT)) { ptes++; pra->mapcount--; @@ -948,7 +948,7 @@ static bool folio_referenced_one(struct folio *folio, /* Restore the mlock which got missed */ mlock_vma_folio(folio, vma); page_vma_mapped_walk_done(&pvmw); - pra->vm_flags |= VM_LOCKED; + vma_flags_set(&pra->vma_flags, VMA_LOCKED_BIT); return false; /* To break the loop */ } @@ -1016,8 +1016,11 @@ static bool folio_referenced_one(struct folio *folio, referenced++; if (referenced) { + vma_flags_t vma_flags = vma->flags; + pra->referenced++; - pra->vm_flags |= vma->vm_flags & ~VM_LOCKED; + vma_flags_clear(&vma_flags, VMA_LOCKED_BIT); + vma_flags_set_mask(&pra->vma_flags, vma_flags); } if (!pra->mapcount) @@ -1055,7 +1058,7 @@ static bool invalid_folio_referenced_vma(struct vm_area_struct *vma, void *arg) * @folio: The folio to test. * @is_locked: Caller holds lock on the folio. * @memcg: target memory cgroup - * @vm_flags: A combination of all the vma->vm_flags which referenced the folio. + * @vma_flags: A combination of all the vma->flags which referenced the folio. * * Quick test_and_clear_referenced for all mappings of a folio, * @@ -1063,7 +1066,7 @@ static bool invalid_folio_referenced_vma(struct vm_area_struct *vma, void *arg) * the function bailed out due to rmap lock contention. */ int folio_referenced(struct folio *folio, int is_locked, - struct mem_cgroup *memcg, vm_flags_t *vm_flags) + struct mem_cgroup *memcg, vma_flags_t *vma_flags) { bool we_locked = false; struct folio_referenced_arg pra = { @@ -1079,7 +1082,7 @@ int folio_referenced(struct folio *folio, int is_locked, }; VM_WARN_ON_ONCE_FOLIO(folio_is_zone_device(folio), folio); - *vm_flags = 0; + vma_flags_clear_all(vma_flags); if (!pra.mapcount) return 0; @@ -1093,7 +1096,7 @@ int folio_referenced(struct folio *folio, int is_locked, } rmap_walk(folio, &rwc); - *vm_flags = pra.vm_flags; + vma_flags_set_mask(vma_flags, pra.vma_flags); if (we_locked) folio_unlock(folio); diff --git a/mm/vmscan.c b/mm/vmscan.c index 8bd0bea62767..206213e56ec8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -861,16 +861,16 @@ static enum folio_references folio_check_references(struct folio *folio, struct scan_control *sc) { int referenced_ptes, referenced_folio; - vm_flags_t vm_flags; + vma_flags_t vma_flags; referenced_ptes = folio_referenced(folio, 1, sc->target_mem_cgroup, - &vm_flags); + &vma_flags); /* * The supposedly reclaimable folio was found to be in a VM_LOCKED vma. * Let the folio, now marked Mlocked, be moved to the unevictable list. */ - if (vm_flags & VM_LOCKED) + if (vma_flags_test(&vma_flags, VMA_LOCKED_BIT)) return FOLIOREF_ACTIVATE; /* @@ -914,7 +914,7 @@ static enum folio_references folio_check_references(struct folio *folio, /* * Activate file-backed executable folios after first usage. */ - if ((vm_flags & VM_EXEC) && folio_is_file_lru(folio)) + if (vma_flags_test(&vma_flags, VMA_EXEC_BIT) && folio_is_file_lru(folio)) return FOLIOREF_ACTIVATE; return FOLIOREF_KEEP; @@ -2065,7 +2065,7 @@ static void shrink_active_list(unsigned long nr_to_scan, { unsigned long nr_taken; unsigned long nr_scanned; - vm_flags_t vm_flags; + vma_flags_t vma_flags; LIST_HEAD(l_hold); /* The folios which were snipped off */ LIST_HEAD(l_active); LIST_HEAD(l_inactive); @@ -2109,7 +2109,7 @@ static void shrink_active_list(unsigned long nr_to_scan, /* Referenced or rmap lock contention: rotate */ if (folio_referenced(folio, 0, sc->target_mem_cgroup, - &vm_flags) != 0) { + &vma_flags) != 0) { /* * Identify referenced, file-backed active folios and * give them one more trip around the active list. So @@ -2119,7 +2119,7 @@ static void shrink_active_list(unsigned long nr_to_scan, * IO, plus JVM can create lots of anon VM_EXEC folios, * so we ignore them here. */ - if ((vm_flags & VM_EXEC) && folio_is_file_lru(folio)) { + if (vma_flags_test(&vma_flags, VMA_EXEC_BIT) && folio_is_file_lru(folio)) { nr_rotated += folio_nr_pages(folio); list_add(&folio->lru, &l_active); continue; -- cgit v1.2.3 From b64727d264782a5441aa625eba60b3c53fd2842e Mon Sep 17 00:00:00 2001 From: Baolin Wang Date: Mon, 20 Jul 2026 19:12:01 +0800 Subject: mm: vmscan: add a helper to identify file-backed executable folios Add a helper to identify file-backed executable folios to avoid duplicate code. No functional changes. Link: https://lore.kernel.org/2ee74f9f98ac45a2f0db0ceb018e086bdb671d17.1784509721.git.baolin.wang@linux.alibaba.com Signed-off-by: Baolin Wang Acked-by: Johannes Weiner Reviewed-by: Axel Rasmussen Reviewed-by: David Hildenbrand (Arm) Reviewed-by: Kairui Song Reviewed-by: Barry Song Cc: Harry Yoo Cc: Jann Horn Cc: Lance Yang Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Rik van Riel Cc: Shakeel Butt Cc: Vlastimil Babka Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton --- mm/vmscan.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 206213e56ec8..f2628b9187f8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -268,6 +268,12 @@ static int sc_swappiness(struct scan_control *sc, struct mem_cgroup *memcg) } #endif +static inline bool is_exec_file_folio(const struct folio *folio, + const vma_flags_t *vma_flags) +{ + return vma_flags_test(vma_flags, VMA_EXEC_BIT) && folio_is_file_lru(folio); +} + static void set_task_reclaim_state(struct task_struct *task, struct reclaim_state *rs) { @@ -914,7 +920,7 @@ static enum folio_references folio_check_references(struct folio *folio, /* * Activate file-backed executable folios after first usage. */ - if (vma_flags_test(&vma_flags, VMA_EXEC_BIT) && folio_is_file_lru(folio)) + if (is_exec_file_folio(folio, &vma_flags)) return FOLIOREF_ACTIVATE; return FOLIOREF_KEEP; @@ -2119,7 +2125,7 @@ static void shrink_active_list(unsigned long nr_to_scan, * IO, plus JVM can create lots of anon VM_EXEC folios, * so we ignore them here. */ - if (vma_flags_test(&vma_flags, VMA_EXEC_BIT) && folio_is_file_lru(folio)) { + if (is_exec_file_folio(folio, &vma_flags)) { nr_rotated += folio_nr_pages(folio); list_add(&folio->lru, &l_active); continue; -- cgit v1.2.3 From 0ee06ee38aed1a9949510624a1b4abdc7e39815a Mon Sep 17 00:00:00 2001 From: Baolin Wang Date: Mon, 20 Jul 2026 19:12:02 +0800 Subject: mm: mglru: promote mapped executable folios after first usage Classical LRU protects mapped executable file folios through commit 8cab4754d24a0 ("vmscan: make mapped executable pages the first class citizen") and commit c909e99364c8 ("vmscan: activate executable pages after first usage"), giving executable code a better chance to stay in memory, avoiding IO thrashing and improving workload performance. However, MGLRU's protection of mapped executable file folios is less reliable. Although shrink_folio_list() checks references, the access flag of mapped executable file folios may have already been checked and cleared by lru_gen_look_around() or walk_mm(). Additionally, folio_update_gen() or lru_gen_set_refs() only sets the 'PG_referenced' flag for mapped executable file folios, which causes shrink_folio_list() to ignore the first usage of these mapped executable file folios and reclaim them easily. Follow the classical LRU's logic, promoting mapped executable file folios after their first usage in folio_update_gen() and lru_gen_set_refs(), giving executable code a better chance to stay in memory. On my 32-core Arm machine, with the memcg limit set to 2G, running 'make -j32' to build kernel showed some improvement in sys time. base patched 9248.543s 7861.579s Link: https://lore.kernel.org/f57d94b1d85bb3d620d89bb739128f0d929bf9c6.1784509721.git.baolin.wang@linux.alibaba.com Signed-off-by: Baolin Wang Acked-by: Johannes Weiner Reviewed-by: Axel Rasmussen Cc: Barry Song Cc: David Hildenbrand Cc: Harry Yoo Cc: Jann Horn Cc: Kairui Song Cc: Lance Yang Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Rik van Riel Cc: Shakeel Butt Cc: Vlastimil Babka Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton --- mm/vmscan.c | 43 +++++++++++++++++++++++++++---------------- 1 file changed, 27 insertions(+), 16 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index f2628b9187f8..d65ccea92756 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -841,10 +841,16 @@ enum folio_references { * with PG_active set. In contrast, the aging (page table walk) path uses * folio_update_gen(). */ -static bool lru_gen_set_refs(struct folio *folio) +static bool lru_gen_set_refs(struct folio *folio, const vma_flags_t *vma_flags) { /* see the comment on LRU_REFS_FLAGS */ if (!folio_test_referenced(folio) && !folio_test_workingset(folio)) { + /* Activate file-backed executable folios after first usage. */ + if (is_exec_file_folio(folio, vma_flags)) { + set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset)); + return true; + } + set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced)); return false; } @@ -857,7 +863,7 @@ static bool lru_gen_set_refs(struct folio *folio) return true; } #else -static bool lru_gen_set_refs(struct folio *folio) +static bool lru_gen_set_refs(struct folio *folio, const vma_flags_t *vma_flags) { return false; } @@ -892,7 +898,7 @@ static enum folio_references folio_check_references(struct folio *folio, if (!referenced_ptes) return FOLIOREF_RECLAIM; - return lru_gen_set_refs(folio) ? FOLIOREF_ACTIVATE : FOLIOREF_KEEP; + return lru_gen_set_refs(folio, &vma_flags) ? FOLIOREF_ACTIVATE : FOLIOREF_KEEP; } referenced_folio = folio_test_clear_referenced(folio); @@ -3208,14 +3214,19 @@ static bool positive_ctrl_err(struct ctrl_pos *sp, struct ctrl_pos *pv) ******************************************************************************/ /* promote pages accessed through page tables */ -static int folio_update_gen(struct folio *folio, int gen) +static int folio_update_gen(struct folio *folio, int gen, const vma_flags_t *vma_flags) { unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f); VM_WARN_ON_ONCE(gen >= MAX_NR_GENS); - /* see the comment on LRU_REFS_FLAGS */ - if (!folio_test_referenced(folio) && !folio_test_workingset(folio)) { + /* + * See the comment on LRU_REFS_FLAGS, and activate file-backed + * executable folios after first usage to avoid typical IO + * thrashing from reclaiming. + */ + if (!folio_test_referenced(folio) && !folio_test_workingset(folio) && + !is_exec_file_folio(folio, vma_flags)) { set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced)); return -1; } @@ -3448,8 +3459,8 @@ static bool suitable_to_scan(int total, int young) return young * n >= total; } -static void walk_update_folio(struct lru_gen_mm_walk *walk, struct folio *folio, - int new_gen, bool dirty) +static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area_struct *vma, + struct folio *folio, int new_gen, bool dirty) { int old_gen; @@ -3462,10 +3473,10 @@ static void walk_update_folio(struct lru_gen_mm_walk *walk, struct folio *folio, folio_mark_dirty(folio); if (walk) { - old_gen = folio_update_gen(folio, new_gen); + old_gen = folio_update_gen(folio, new_gen, &vma->flags); if (old_gen >= 0 && old_gen != new_gen) update_batch_size(walk, folio, old_gen, new_gen); - } else if (lru_gen_set_refs(folio)) { + } else if (lru_gen_set_refs(folio, &vma->flags)) { old_gen = folio_lru_gen(folio); if (old_gen >= 0 && old_gen != new_gen) folio_activate(folio); @@ -3538,7 +3549,7 @@ restart: continue; if (last != folio) { - walk_update_folio(walk, last, gen, dirty); + walk_update_folio(walk, args->vma, last, gen, dirty); last = folio; dirty = false; @@ -3551,7 +3562,7 @@ restart: walk->mm_stats[MM_LEAF_YOUNG] += nr; } - walk_update_folio(walk, last, gen, dirty); + walk_update_folio(walk, args->vma, last, gen, dirty); last = NULL; if (i < PTRS_PER_PTE && get_next_vma(PMD_MASK, PAGE_SIZE, args, &start, &end)) @@ -3629,7 +3640,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area goto next; if (last != folio) { - walk_update_folio(walk, last, gen, dirty); + walk_update_folio(walk, vma, last, gen, dirty); last = folio; dirty = false; @@ -3643,7 +3654,7 @@ next: i = i > MIN_LRU_BATCH ? 0 : find_next_bit(bitmap, MIN_LRU_BATCH, i) + 1; } while (i <= MIN_LRU_BATCH); - walk_update_folio(walk, last, gen, dirty); + walk_update_folio(walk, vma, last, gen, dirty); lazy_mmu_mode_disable(); spin_unlock(ptl); @@ -4278,7 +4289,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr) continue; if (last != folio) { - walk_update_folio(walk, last, gen, dirty); + walk_update_folio(walk, vma, last, gen, dirty); last = folio; dirty = false; @@ -4290,7 +4301,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr) young += nr; } - walk_update_folio(walk, last, gen, dirty); + walk_update_folio(walk, vma, last, gen, dirty); lazy_mmu_mode_disable(); -- cgit v1.2.3 From 1b7c8fe294a6bf913832e998f519029cad47dbcb Mon Sep 17 00:00:00 2001 From: Ridong Chen Date: Thu, 23 Jul 2026 11:24:33 +0800 Subject: memcg: move mem_cgroup_swappiness and vm_swappiness to mm/swap.h Patch series "mm: vmscan: fix node reclaim ignoring swappiness parameter", v4. The per-node proactive reclaim interface (/sys/devices/system/node/nodeX/reclaim) accepts a swappiness parameter, but it is silently ignored when CONFIG_MEMCG is disabled. The root cause is that sc_swappiness() has separate implementations for CONFIG_MEMCG and !CONFIG_MEMCG, and the latter never checks proactive_swappiness. Patch 1 moves mem_cgroup_swappiness() and vm_swappiness out of the public include/linux/swap.h into the mm-private mm/swap.h, and makes the helper handle both CONFIG_MEMCG and !CONFIG_MEMCG in a single inline function. This is a prerequisite for unifying sc_swappiness(). Patch 2 consolidates sc_swappiness() into a single definition that works regardless of CONFIG_MEMCG, fixing the node reclaim swappiness bug. This patch (of 2): The per-memcg swappiness knob is v1-only; v2 always uses global vm_swappiness and ignores the per-cgroup field. Both mem_cgroup_swappiness() and vm_swappiness are only used within mm/ (memcontrol.c, memcontrol-v1.c, vmscan.c), so move them out of the public include/linux/swap.h into the mm-private mm/swap.h. This keeps unrelated declarations out of include/linux/swap.h. Guard memcg->swappiness with CONFIG_MEMCG_V1 as well, so v2-only kernels drop the unused field. No functional change for v1; v2-only kernels drop the unused field. Link: https://lore.kernel.org/20260723032434.2016749-1-ridong.chen@linux.dev Link: https://lore.kernel.org/20260723032434.2016749-2-ridong.chen@linux.dev Signed-off-by: Ridong Chen Acked-by: Johannes Weiner Reviewed-by: Barry Song Reviewed-by: Song Hu Acked-by: Shakeel Butt Cc: Axel Rasmussen Cc: Baoquan He Cc: Chris Li Cc: David Hildenbrand Cc: Davidlohr Bueso Cc: Kairui Song Cc: Kemeng Shi Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Muchun Song Cc: Nhat Pham Cc: Roman Gushchin Cc: Wei Xu Cc: Yuanchu Xie Cc: Qi Zheng Signed-off-by: Andrew Morton --- include/linux/memcontrol.h | 4 ++-- include/linux/swap.h | 19 ------------------- mm/memcontrol.c | 4 ++-- mm/swap.h | 14 ++++++++++++++ 4 files changed, 18 insertions(+), 23 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 8170bb8066a2..f619e24fc3bb 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -239,8 +239,6 @@ struct mem_cgroup { */ bool oom_group; - int swappiness; - /* memory.events and memory.events.local */ struct cgroup_file events_file; struct cgroup_file events_local_file; @@ -318,6 +316,8 @@ struct mem_cgroup { /* List of events which userspace want to receive */ struct list_head event_list; spinlock_t event_list_lock; + + int swappiness; #endif /* CONFIG_MEMCG_V1 */ struct mem_cgroup_per_node *nodeinfo[]; diff --git a/include/linux/swap.h b/include/linux/swap.h index 696ed01709c2..330a420fd6de 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -309,7 +309,6 @@ static inline bool lru_cache_disabled(void) } extern unsigned long shrink_all_memory(unsigned long nr_pages); -extern int vm_swappiness; long remove_mapping(struct address_space *mapping, struct folio *folio); #if defined(CONFIG_SYSFS) && defined(CONFIG_NUMA) @@ -468,25 +467,7 @@ static inline int add_swap_extent(struct swap_info_struct *sis, } #endif /* CONFIG_SWAP */ #ifdef CONFIG_MEMCG -static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg) -{ - /* Cgroup2 doesn't have per-cgroup swappiness */ - if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) - return READ_ONCE(vm_swappiness); - - /* root ? */ - if (mem_cgroup_disabled() || mem_cgroup_is_root(memcg)) - return READ_ONCE(vm_swappiness); - - return READ_ONCE(memcg->swappiness); -} - void lru_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent, int nid); -#else -static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg) -{ - return READ_ONCE(vm_swappiness); -} #endif #if defined(CONFIG_SWAP) && defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 69b37f63a307..ec92cb40156d 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -63,6 +63,7 @@ #include #include #include "internal.h" +#include "swap.h" #include "swap_table.h" #include #include @@ -4176,11 +4177,10 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) #endif page_counter_set_high(&memcg->swap, PAGE_COUNTER_MAX); if (parent) { - WRITE_ONCE(memcg->swappiness, mem_cgroup_swappiness(parent)); - page_counter_init(&memcg->memory, &parent->memory, memcg_on_dfl); page_counter_init(&memcg->swap, &parent->swap, false); #ifdef CONFIG_MEMCG_V1 + WRITE_ONCE(memcg->swappiness, mem_cgroup_swappiness(parent)); memcg->memory.track_failcnt = !memcg_on_dfl; memcg->memsw.track_failcnt = !memcg_on_dfl; WRITE_ONCE(memcg->oom_kill_disable, READ_ONCE(parent->oom_kill_disable)); diff --git a/mm/swap.h b/mm/swap.h index b51ad3071a73..2ccf8cf7f6c1 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -4,6 +4,8 @@ #include /* for atomic_long_t */ #include /* for PAGE_SHIFT */ +#include /* for mem_cgroup_swappiness() */ + struct mempolicy; struct swap_iocb; struct swap_memcg_table; @@ -76,6 +78,18 @@ enum swap_cluster_flags { CLUSTER_FLAG_MAX, }; +extern int vm_swappiness; + +static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg) +{ +#ifdef CONFIG_MEMCG_V1 + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys) && + !mem_cgroup_disabled() && !mem_cgroup_is_root(memcg)) + return READ_ONCE(memcg->swappiness); +#endif + return READ_ONCE(vm_swappiness); +} + #ifdef CONFIG_SWAP #include /* for swp_offset */ #include /* for bio_end_io_t */ -- cgit v1.2.3 From 8a905195850d383c0465ab5bdd3c91d94269b242 Mon Sep 17 00:00:00 2001 From: Ridong Chen Date: Thu, 23 Jul 2026 11:24:34 +0800 Subject: mm: vmscan: fix node reclaim ignoring swappiness parameter sc_swappiness() had two separate definitions depending on CONFIG_MEMCG. The !CONFIG_MEMCG variant simply returned vm_swappiness, ignoring the proactive_swappiness value passed through scan_control. This caused the swappiness parameter written to /sys/devices/system/node/nodeX/reclaim to have no effect when CONFIG_MEMCG is disabled. Fix this by consolidating sc_swappiness() into a single definition that checks sc->proactive_swappiness first, then falls back to mem_cgroup_swappiness() which already handles both CONFIG_MEMCG and !CONFIG_MEMCG. Before fix (swappiness=max ignored, mostly file pages reclaimed): # cat /proc/sys/vm/swappiness 60 # cat /proc/vmstat | grep pgsteal pgsteal_kswapd 0 pgsteal_direct 0 pgsteal_khugepaged 0 pgsteal_proactive 1840 pgsteal_anon 25 pgsteal_file 1815 # echo "64M swappiness=max" > /sys/devices/system/node/node0/reclaim # cat /proc/vmstat | grep pgsteal pgsteal_kswapd 0 pgsteal_direct 0 pgsteal_khugepaged 0 pgsteal_proactive 18013 pgsteal_anon 337 pgsteal_file 17676 After fix (swappiness=max honored, anon pages reclaimed as expected): # cat /proc/vmstat | grep pgsteal pgsteal_kswapd 0 pgsteal_direct 0 pgsteal_khugepaged 0 pgsteal_proactive 0 pgsteal_anon 0 pgsteal_file 0 # echo "64M swappiness=max" > /sys/devices/system/node/node0/reclaim # cat /proc/vmstat | grep pgsteal pgsteal_kswapd 0 pgsteal_direct 0 pgsteal_khugepaged 0 pgsteal_proactive 16283 pgsteal_anon 16283 pgsteal_file 0 Link: https://lore.kernel.org/20260723032434.2016749-3-ridong.chen@linux.dev Fixes: b980077899ea ("mm: introduce per-node proactive reclaim interface") Signed-off-by: Ridong Chen Acked-by: Johannes Weiner Reviewed-by: Barry Song Acked-by: Qi Zheng Tested-by: Song Hu Reviewed-by: Song Hu Acked-by: Shakeel Butt Cc: Axel Rasmussen Cc: Baoquan He Cc: Chris Li Cc: David Hildenbrand Cc: Davidlohr Bueso Cc: Kairui Song Cc: Kemeng Shi Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Muchun Song Cc: Nhat Pham Cc: Roman Gushchin Cc: Wei Xu Cc: Yuanchu Xie Cc: [6.17+] Signed-off-by: Andrew Morton --- mm/vmscan.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index d65ccea92756..5bad065a38f1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -199,6 +199,13 @@ struct scan_control { */ int vm_swappiness = 60; +static int sc_swappiness(struct scan_control *sc, struct mem_cgroup *memcg) +{ + if (sc->proactive && sc->proactive_swappiness) + return *sc->proactive_swappiness; + return mem_cgroup_swappiness(memcg); +} + #ifdef CONFIG_MEMCG /* Returns true for reclaim through cgroup limits or cgroup interfaces. */ @@ -239,13 +246,6 @@ static bool writeback_throttling_sane(struct scan_control *sc) #endif return false; } - -static int sc_swappiness(struct scan_control *sc, struct mem_cgroup *memcg) -{ - if (sc->proactive && sc->proactive_swappiness) - return *sc->proactive_swappiness; - return mem_cgroup_swappiness(memcg); -} #else static bool cgroup_reclaim(struct scan_control *sc) { @@ -261,11 +261,6 @@ static bool writeback_throttling_sane(struct scan_control *sc) { return true; } - -static int sc_swappiness(struct scan_control *sc, struct mem_cgroup *memcg) -{ - return READ_ONCE(vm_swappiness); -} #endif static inline bool is_exec_file_folio(const struct folio *folio, -- cgit v1.2.3 From a69797fb36452865252f10c8ac9ef6781d07e3d7 Mon Sep 17 00:00:00 2001 From: Usama Arif Date: Mon, 27 Jul 2026 09:23:23 -0700 Subject: mm/vmstat, mm/memcontrol: add _monotonic vmstat readers Patch series "mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost", v5. The anon/file scan balance heuristic in get_scan_count() is fed by two scalars in struct lruvec (anon_cost, file_cost) that every reclaim producer updates under lruvec->lru_lock. The cost-recording work itself is trivial, but it both contends for and contributes to contention on lru_lock - which is often a contention point on memory-pressured workloads. Specifically: - shrink_inactive_list() re-acquires lru_lock at function exit just to call lru_note_cost_unlock_irq(). - shrink_active_list() does the same after rotation accounting. - workingset_refault() takes folio_lruvec_lock_irq() purely to record the refault cost. - prepare_scan_control() snapshots anon_cost/file_cost under lru_lock. - lru_note_cost_unlock_irq() itself walks parent_lruvec() and re-acquires lru_lock on every ancestor, multiplying the cost of every update by memcg-hierarchy depth. This series removes those producer-side acquisitions entirely. The rotation inputs become per-LRU PGROTATE_{ANON,FILE} vmstat counters. NR_VMSCAN_WRITE already captures reclaim-driven pageout at writeout(); charge it through lruvec_stat_mod_folio() so it is available per lruvec and aggregated through the memcg hierarchy. Reclaim does not submit filesystem folios for writeback from this path, so pageout contributes only to anon cost. WORKINGSET_RESTORE_* already captures the refault input. PGROTATE_* are also useful independently of scan balancing. They are cumulative base-page events, not unique-page counts. Classic inactive reclaim records scan work that does not produce immediate reclaim or demotion, while active reclaim records referenced executable file folios retained on the active list. MGLRU records initially isolated pages that remain unreclaimed after its retry passes. Read alongside pgscan_* and pgsteal_*, their deltas identify which LRU type is consuming reclaim CPU without producing immediate yield. Unlike the existing pgrotated event, they do not imply a move to the inactive-list tail. prepare_scan_control() reads the raw cost signals without lru_lock: anon = PGROTATE_ANON + (NR_VMSCAN_WRITE + WORKINGSET_RESTORE_ANON) * SWAP_CLUSTER_MAX file = PGROTATE_FILE + WORKINGSET_RESTORE_FILE * SWAP_CLUSTER_MAX It folds the deltas into a per-lruvec accumulator. A dedicated per-lruvec cost_lock, not touched by isolate_lru_folios(), move_folios_to_lru(), or folio_add_lru(), serialises the accumulator RMW and the lrusize/4 halving check. Hierarchy aggregation is implicit in rstat propagation, so the parent_lruvec() walk and the lru_reparent_memcg() cost-splice both disappear. Moving accumulation and decay to the reclaim side also improves the cost model across reclaim gaps. With producer-side decay, events that happen while reclaim is idle still age each other before reclaim ever samples the costs. If a workload refaults a large anon set and then a smaller file set before reclaim runs again, the later file activity can age the earlier anon activity out of the cost model. The new scheme observes the whole between-reclaim delta and decays anon and file proportionally, so the scan-balance history better represents what happened since the last reclaim pass. Trade-offs: - Cost reads see rstat-aggregated values that can lag until periodic / reader-triggered flushing. - Per-lruvec footprint grows by 4 unsigned longs + a spinlock (a struct lru_cost { count, last_rotated, last_io } per side), which is a small cost. - NR_VMSCAN_WRITE now also updates the folio's lruvec/memcg stat, adding memcg stat accounting to the reclaim writeout path while preserving the existing node-level total. == Numbers == Tested on a 176-core, 256 GB host. The benchmark drives sustained swap-out/refault inside a tight memcg using vm-scalability/usemem: usemem -n 16 --prealloc --prefault --random $((256*1024*1024)) run inside a two-level memcg with memory.max=512M on the leaf (4 GB anon working set has to fit in 512 MB -> continuous shrink_inactive_list + workingset_refault). A 16 GB swap file is used. Measurement is a 30 s `perf lock record -a` window over otherwise-idle hardware. Workload rates are identical on both kernels (the bench drives the same memory pressure): baseline patched delta pgscan_direct / s 172,662 171,817 ~0% pgsteal_direct / s 67,162 66,306 ~0% workingset_refault_anon / s 40,696 39,830 ~0% perf lock contention (total wait per 30 s window): Lock Name Before After % change shrink_lruvec+0x770 722.84 ms 0 -100% (eliminated) (= lru_note_cost_unlock_irq) workingset_refault+0x167 385.26 ms 0 -100% (eliminated) (= lru_note_cost_refault) shrink_node+0x4ad 689.43 ms 26.95 ms -96% shrink_active_list 208.34 ms 15.97 ms -92% lru_add_drain_cpu+0x34 1.96 s 917.71 ms -53% Total LRU lock wait ~4.23 s ~1.66 s -61% The two specific contention sites the patch removes (shrink_lruvec+0x770 = lru_note_cost_unlock_irq; workingset_refault+0x167 = lru_note_cost_refault) are completely absent from the patched perf-lock-contention output. Secondary reductions in shrink_node, shrink_active_list, lru_add_drain_cpu and pgrefill/pgactivate look like knock-on effects from removing the cost-recording overhead and the parent_lruvec walk. The remaining ~1.66 s of LRU lock wait on the patched kernel is dominated by the per-CPU pagevec drain (lru_add_drain_cpu) and the main reclaim path in shrink_lruvec. The numbers above can be reproduced using the script in [1]. This patch (of 3): lruvec_page_state(), node_page_state(), and global_node_page_state() all clamp negative reads to zero on CONFIG_SMP so that a transient per-CPU delta skew presents as zero pages rather than as a garbage unsigned value. This is the right behaviour for non-monotonic page-count readers. It is however incorrect for callers that snapshot a monotonically- incremented event counter and compute a delta from two samples. Once the underlying signed long wraps past LONG_MAX, the clamped read drops to zero while the previously-recorded snapshot still holds the pre-wrap value; the unsigned subtraction then underflows into a ~2^31 spurious delta for 32-bit architecture and corrupts the caller's accumulator. Add non-clamping siblings that return the underlying state value cast to unsigned long: global_node_page_state_monotonic() node_page_state_monotonic() lruvec_page_state_monotonic() With both samples read via the _monotonic variant, unsigned modular subtraction stays correct across a signed-long wraparound as long as the true growth between two samples fits in unsigned long (< 2^32 on 32-bit, < 2^64 on 64-bit); the 32-bit bound is the practically-reachable one that motivates this helper. The variants are only safe for monotonically-incremented counters. Non-monotonic page-count readers must keep using the existing clamped helpers so transient negative reads still present as zero. This is a prerequisite for a later patch which replaces the producer-side anon_cost/file_cost accumulators with a read-side accumulator in prepare_scan_control() that samples monotonic per-LRU vmstat counters (PGROTATE_*, NR_VMSCAN_WRITE, WORKINGSET_RESTORE_*) via lruvec_page_state_monotonic() and folds their unsigned modular deltas into lruvec->cost[].count. Link: https://lore.kernel.org/20260727162550.2032-1-usama.arif@linux.dev Link: https://lore.kernel.org/20260727162550.2032-2-usama.arif@linux.dev Link: https://gist.github.com/uarif1/a4eb33a86c5b2d7bbc55b42f0956e884 [1] Signed-off-by: Usama Arif Acked-by: Johannes Weiner Acked-by: Shakeel Butt Acked-by: Vlastimil Babka (SUSE) Cc: Axel Rasmussen Cc: Baoquan He Cc: Chris Li Cc: David Hildenbrand Cc: David Rientjes Cc: Kairui Song Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Nhat Pham Cc: Roman Gushchin Cc: Suren Baghdasaryan Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton --- include/linux/memcontrol.h | 8 ++++++++ include/linux/vmstat.h | 16 ++++++++++++++++ mm/memcontrol.c | 36 ++++++++++++++++++++++++++++++++++++ mm/vmstat.c | 11 +++++++++++ 4 files changed, 71 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index f619e24fc3bb..e78bc98ab229 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -947,6 +947,8 @@ unsigned long memcg_page_state_output(struct mem_cgroup *memcg, int item); bool memcg_stat_item_valid(int idx); bool memcg_vm_event_item_valid(enum vm_event_item idx); unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx); +unsigned long lruvec_page_state_monotonic(struct lruvec *lruvec, + enum node_stat_item idx); unsigned long lruvec_page_state_local(struct lruvec *lruvec, enum node_stat_item idx); @@ -1399,6 +1401,12 @@ static inline unsigned long lruvec_page_state(struct lruvec *lruvec, return node_page_state(lruvec_pgdat(lruvec), idx); } +static inline unsigned long lruvec_page_state_monotonic(struct lruvec *lruvec, + enum node_stat_item idx) +{ + return node_page_state_monotonic(lruvec_pgdat(lruvec), idx); +} + static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, enum node_stat_item idx) { diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index 3c9c266cf782..fb8c76289e02 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -194,6 +194,19 @@ unsigned long global_node_page_state_pages(enum node_stat_item item) return x; } +/* + * Non-clamping variant of global_node_page_state() intended for callers that + * snapshot a monotonically-incremented counter and subtract two samples. + * Returns the raw wrapping value so that unsigned modular subtraction stays + * correct across a signed-long overflow (a real hazard on 32-bit) that the + * clamp in global_node_page_state() would otherwise turn into a huge spurious + * delta. Do NOT use for non-monotonic page-count reads. + */ +static inline unsigned long global_node_page_state_monotonic(enum node_stat_item item) +{ + return (unsigned long)atomic_long_read(&vm_node_stat[item]); +} + static inline unsigned long global_node_page_state(enum node_stat_item item) { VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); @@ -259,11 +272,14 @@ extern unsigned long node_page_state(struct pglist_data *pgdat, enum node_stat_item item); extern unsigned long node_page_state_pages(struct pglist_data *pgdat, enum node_stat_item item); +extern unsigned long node_page_state_monotonic(struct pglist_data *pgdat, + enum node_stat_item item); extern void fold_vm_numa_events(void); #else #define sum_zone_node_page_state(node, item) global_zone_page_state(item) #define node_page_state(node, item) global_node_page_state(item) #define node_page_state_pages(node, item) global_node_page_state_pages(item) +#define node_page_state_monotonic(node, item) global_node_page_state_monotonic(item) static inline void fold_vm_numa_events(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ec92cb40156d..d804f8d07581 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -508,6 +508,42 @@ unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx) return x; } +/** + * lruvec_page_state_monotonic - non-clamping lruvec stat read for delta sampling + * @lruvec: the LRU vector to read from + * @idx: the node_stat_item to read + * + * Returns the raw state[idx] value cast to unsigned long, skipping the + * clamp-negative-to-zero step in lruvec_page_state(). Intended for callers + * that snapshot a monotonically-incremented counter and subtract two + * samples: unsigned modular arithmetic then yields the correct delta across + * a signed-long wraparound (a real hazard on 32-bit) that the clamp would + * otherwise turn into a huge spurious delta. + * + * Do NOT use for non-monotonic page-count reads where a transient negative + * reading from per-CPU delta skew must present as zero. + * + * XXX: This helper (and its node/global peers) exists because some + * monotonically-incremented event counters are stored in + * enum node_stat_item. + */ +unsigned long lruvec_page_state_monotonic(struct lruvec *lruvec, + enum node_stat_item idx) +{ + struct mem_cgroup_per_node *pn; + int i; + + if (mem_cgroup_disabled()) + return node_page_state_monotonic(lruvec_pgdat(lruvec), idx); + + i = memcg_stats_index(idx); + if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx)) + return 0; + + pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); + return (unsigned long)READ_ONCE(pn->lruvec_stats->state[i]); +} + unsigned long lruvec_page_state_local(struct lruvec *lruvec, enum node_stat_item idx) { diff --git a/mm/vmstat.c b/mm/vmstat.c index 3b5cb1031f72..507118474c03 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1025,6 +1025,17 @@ unsigned long node_page_state(struct pglist_data *pgdat, return node_page_state_pages(pgdat, item); } + +/* + * Non-clamping variant of node_page_state() intended for callers that + * snapshot a monotonically-incremented counter and subtract two samples. + * See global_node_page_state_monotonic() for the rationale. + */ +unsigned long node_page_state_monotonic(struct pglist_data *pgdat, + enum node_stat_item item) +{ + return (unsigned long)atomic_long_read(&pgdat->vm_stat[item]); +} #endif /* -- cgit v1.2.3 From 1b089def0fb8833ec4b331b61908f29d6b491ccb Mon Sep 17 00:00:00 2001 From: Usama Arif Date: Mon, 27 Jul 2026 09:23:24 -0700 Subject: mm/vmscan: add pgrotate_anon and pgrotate_file vmstat counters Reclaim can spend substantial work on an LRU type without immediately reclaiming or demoting a corresponding amount of memory. Record this work in PGROTATE_ANON and PGROTATE_FILE. For classic LRU reclaim: - Inactive-list reclaim adds nr_scanned - nr_reclaimed to the corresponding anon/file counter when isolation succeeds. - Active-list reclaim adds referenced executable file folios that are retained on the active list to PGROTATE_FILE. Active anon reclaim does not contribute this component. For MGLRU, add the number of initially isolated pages that remain unreclaimed after both the initial and retry passes to the counter for the selected anon/file type. These counters are distinct from the existing pgrotated vm event. pgrotated records an actual move to the inactive-list tail, primarily after reclaim-marked writeback completes or failed invalidation leaves a folio for accelerated reclaim. PGROTATE_ANON and PGROTATE_FILE measure reclaim cost and do not imply that a folio moved to an LRU tail. A subsequent patch will consume these counters for anon/file scan balancing. Link: https://lore.kernel.org/20260727162550.2032-3-usama.arif@linux.dev Signed-off-by: Usama Arif Acked-by: Shakeel Butt Acked-by: Johannes Weiner Reviewed-by: Vlastimil Babka (SUSE) Cc: Axel Rasmussen Cc: Baoquan He Cc: Chris Li Cc: David Hildenbrand Cc: David Rientjes Cc: Kairui Song Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Nhat Pham Cc: Roman Gushchin Cc: Suren Baghdasaryan Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton --- include/linux/mmzone.h | 2 ++ mm/memcontrol.c | 2 ++ mm/vmscan.c | 14 +++++++++++++- mm/vmstat.c | 2 ++ 4 files changed, 19 insertions(+), 1 deletion(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 0507193b3ae3..1cc5ea506b7c 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -323,6 +323,8 @@ enum node_stat_item { PGSCAN_PROACTIVE, PGSCAN_ANON, PGSCAN_FILE, + PGROTATE_ANON, + PGROTATE_FILE, PGREFILL, #ifdef CONFIG_HUGETLB_PAGE NR_HUGETLB, diff --git a/mm/memcontrol.c b/mm/memcontrol.c index d804f8d07581..011bb3b0346b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -425,6 +425,8 @@ static const unsigned int memcg_node_stat_items[] = { PGSCAN_PROACTIVE, PGSCAN_ANON, PGSCAN_FILE, + PGROTATE_ANON, + PGROTATE_FILE, PGREFILL, #ifdef CONFIG_HUGETLB_PAGE NR_HUGETLB, diff --git a/mm/vmscan.c b/mm/vmscan.c index 5bad065a38f1..fa2c5cf577af 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2038,6 +2038,9 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, item = PGSTEAL_KSWAPD + reclaimer_offset(sc); mod_lruvec_state(lruvec, item, nr_reclaimed); mod_lruvec_state(lruvec, PGSTEAL_ANON + file, nr_reclaimed); + if (nr_scanned > nr_reclaimed) + mod_lruvec_state(lruvec, PGROTATE_ANON + file, + nr_scanned - nr_reclaimed); lruvec_lock_irq(lruvec); lru_note_cost_unlock_irq(lruvec, file, stat.nr_pageout, @@ -2147,6 +2150,8 @@ static void shrink_active_list(unsigned long nr_to_scan, count_vm_events(PGDEACTIVATE, nr_deactivate); count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate); mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); + if (nr_rotated) + mod_lruvec_state(lruvec, PGROTATE_ANON + file, nr_rotated); lruvec_lock_irq(lruvec); lru_note_cost_unlock_irq(lruvec, file, 0, nr_rotated); @@ -4828,7 +4833,8 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, struct reclaim_stat stat; struct lru_gen_mm_walk *walk; int scanned, reclaimed; - int isolated = 0, type, type_scanned; + int isolated = 0, nr_isolated = 0, type, type_scanned; + unsigned long total_reclaimed = 0; bool skip_retry = false; struct mem_cgroup *memcg = lruvec_memcg(lruvec); struct pglist_data *pgdat = lruvec_pgdat(lruvec); @@ -4840,6 +4846,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, scanned = isolate_folios(nr_to_scan, lruvec, sc, swappiness, &list, &isolated, &type, &type_scanned); + nr_isolated = isolated; /* Scanning may have emptied the oldest gen, flush it */ if (scanned) @@ -4852,6 +4859,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, retry: reclaimed = shrink_folio_list(&list, pgdat, sc, &stat, false, memcg); sc->nr_reclaimed += reclaimed; + total_reclaimed += reclaimed; /* Retry pass is only meant for clean folios without new isolation */ if (isolated) handle_reclaim_writeback(isolated, pgdat, sc, &stat); @@ -4903,6 +4911,10 @@ retry: goto retry; } + if (nr_isolated > total_reclaimed) + mod_lruvec_state(lruvec, PGROTATE_ANON + type, + nr_isolated - total_reclaimed); + return scanned; } diff --git a/mm/vmstat.c b/mm/vmstat.c index 507118474c03..7d6e61a01f51 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1301,6 +1301,8 @@ const char * const vmstat_text[] = { [I(PGSCAN_PROACTIVE)] = "pgscan_proactive", [I(PGSCAN_ANON)] = "pgscan_anon", [I(PGSCAN_FILE)] = "pgscan_file", + [I(PGROTATE_ANON)] = "pgrotate_anon", + [I(PGROTATE_FILE)] = "pgrotate_file", [I(PGREFILL)] = "pgrefill", #ifdef CONFIG_HUGETLB_PAGE [I(NR_HUGETLB)] = "nr_hugetlb", -- cgit v1.2.3 From 7b9f4e5f81013bcb0d16bf54b349ee323ce0ac01 Mon Sep 17 00:00:00 2001 From: Usama Arif Date: Mon, 27 Jul 2026 09:23:25 -0700 Subject: mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost The anon/file scan balance in get_scan_count() is driven by two scalars in struct lruvec, anon_cost and file_cost, accumulated by every reclaim producer under lruvec->lru_lock. The acquisition sites for cost work specifically are: - shrink_inactive_list() re-takes lru_lock at function exit purely to call lru_note_cost_unlock_irq() with (nr_pageout, nr_scanned - nr_reclaimed). One acquisition per inactive shrink. - shrink_active_list() does the same with (0, nr_rotated). One acquisition per active shrink. - workingset_refault() takes the lock via folio_lruvec_lock_irq() purely to record the refault cost. One acquisition per refault. - prepare_scan_control() takes lru_lock just to snapshot the two scalars into sc->{anon,file}_cost. - lru_note_cost_unlock_irq() itself walks parent_lruvec and re-acquires lru_lock on each ancestor to propagate the update, adding O(memcg-depth) acquisitions per producer call. This hurts because lru_lock is already a heavy contention point on memory-heavy workloads: every isolate_lru_folios(), move_folios_to_lru() and folio_add_lru() takes it. The cost work itself is trivial (two scalar bumps and one comparison), but it contends with and causes contention for actual LRU manipulation. The parent_lruvec() walk also multiplies cost-update overhead by memcg hierarchy depth. The balance formula for anon and file, respectively, is this: cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated Instead of recording cost and running averaging logic directly when these events occur, snapshot running vmstat counters once per reclaim cycle and derive the balance from event deltas since the last run. Use PGROTATE_* from the preceding patch for the rotation input. WORKINGSET_RESTORE_* and NR_VMSCAN_WRITE provide the remaining event counters. Charge NR_VMSCAN_WRITE through lruvec stats so all inputs can be sampled per lruvec and aggregated through the memcg hierarchy. This is overall cheaper and has fewer lock acquisition sites. Moving accumulation and decay to the reclaim side also improves the cost model across reclaim gaps. With producer-side decay, events that happen while reclaim is idle still age each other before reclaim ever samples the costs. If a workload refaults a large anon set and then a smaller file set before reclaim runs again, the later file activity can age the earlier anon activity out of the cost model. The new scheme observes the whole between-reclaim delta and decays anon and file proportionally, so the scan-balance history better represents what happened since the last reclaim pass. A dedicated per-lruvec spinlock, cost_lock, serialises the delta extraction, the cost->count update and the halving loop against concurrent reclaimers in the same memcg+node. NR_VMSCAN_WRITE is accounted at writeout(), so reclaim_stat.nr_pageout is no longer needed and is removed. memcg-v1's memory.stat anon_cost/file_cost is now sourced from cost[].count instead of the removed lruvec anon_cost/file_cost fields. The reported values only refresh when prepare_scan_control() runs and are bounded at ~lrusize/4 by the halving loop; the scan-balance signal they express is unchanged. Under pure MGLRU the scan-balance signal itself is not consumed (both prepare_scan_control() and get_scan_count() are short-circuited on the MGLRU paths, and MGLRU's own type/tier selection comes from read_ctrl_pos() on lrugen->{avg_refaulted,avg_total,refaulted,evicted}, not from anon_cost/file_cost). NR_VMSCAN_WRITE naturally covers writeout from either reclaim implementation. The preceding patch also bumps PGROTATE_{ANON,FILE} from evict_folios(), so rotation-driven reclaim work is accounted consistently across both implementations. Link: https://lore.kernel.org/20260727162550.2032-4-usama.arif@linux.dev Signed-off-by: Usama Arif Acked-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Vlastimil Babka (SUSE) Cc: Axel Rasmussen Cc: Baoquan He Cc: Chris Li Cc: David Hildenbrand Cc: David Rientjes Cc: Kairui Song Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Nhat Pham Cc: Roman Gushchin Cc: Suren Baghdasaryan Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton --- include/linux/mmzone.h | 13 +++++++-- include/linux/vmstat.h | 1 - mm/folio.c | 69 ------------------------------------------- mm/internal.h | 3 -- mm/memcontrol-v1.c | 4 +-- mm/memcontrol.c | 1 + mm/mmzone.c | 1 + mm/vmscan.c | 79 +++++++++++++++++++++++++++++++++++++++++--------- mm/workingset.c | 5 ---- 9 files changed, 81 insertions(+), 95 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 1cc5ea506b7c..158c1fba2393 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -757,6 +757,12 @@ void lru_gen_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent, #endif /* CONFIG_LRU_GEN */ +struct lru_cost { + unsigned long count; + unsigned long last_rotated; + unsigned long last_io; +}; + struct lruvec { struct list_head lists[NR_LRU_LISTS]; /* per lruvec lru_lock for memcg */ @@ -765,9 +771,12 @@ struct lruvec { * These track the cost of reclaiming one LRU - file or anon - * over the other. As the observed cost of reclaiming one LRU * increases, the reclaim scan balance tips toward the other. + * Updated and decayed at prepare_scan_control() time; cost_lock + * serialises that update. */ - unsigned long anon_cost; - unsigned long file_cost; + struct lru_cost cost[ANON_AND_FILE]; + /* Protects cost[]. */ + spinlock_t cost_lock; /* Non-resident age, driven by LRU movement */ atomic_long_t nonresident_age; /* Refaults at the time of last reclaim cycle */ diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index fb8c76289e02..5b31d8e7ae40 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -20,7 +20,6 @@ struct reclaim_stat { unsigned nr_congested; unsigned nr_writeback; unsigned nr_immediate; - unsigned nr_pageout; unsigned nr_activate[ANON_AND_FILE]; unsigned nr_ref_keep; unsigned nr_unmap_fail; diff --git a/mm/folio.c b/mm/folio.c index d2937600cf72..a9e328c3f21b 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -265,73 +265,6 @@ void folio_rotate_reclaimable(struct folio *folio) folio_batch_add_and_move(folio, lru_move_tail); } -void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file, - unsigned int nr_io, unsigned int nr_rotated) - __releases(lruvec->lru_lock) - __releases(rcu) -{ - unsigned long cost; - - /* - * Reflect the relative cost of incurring IO and spending CPU - * time on rotations. This doesn't attempt to make a precise - * comparison, it just says: if reloads are about comparable - * between the LRU lists, or rotations are overwhelmingly - * different between them, adjust scan balance for CPU work. - */ - cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated; - if (!cost) { - spin_unlock_irq(&lruvec->lru_lock); - rcu_read_unlock(); - return; - } - - for (;;) { - unsigned long lrusize; - - /* Record cost event */ - if (file) - lruvec->file_cost += cost; - else - lruvec->anon_cost += cost; - - /* - * Decay previous events - * - * Because workloads change over time (and to avoid - * overflow) we keep these statistics as a floating - * average, which ends up weighing recent refaults - * more than old ones. - */ - lrusize = lruvec_page_state(lruvec, NR_INACTIVE_ANON) + - lruvec_page_state(lruvec, NR_ACTIVE_ANON) + - lruvec_page_state(lruvec, NR_INACTIVE_FILE) + - lruvec_page_state(lruvec, NR_ACTIVE_FILE); - - if (lruvec->file_cost + lruvec->anon_cost > lrusize / 4) { - lruvec->file_cost /= 2; - lruvec->anon_cost /= 2; - } - - spin_unlock_irq(&lruvec->lru_lock); - lruvec = parent_lruvec(lruvec); - if (!lruvec) { - rcu_read_unlock(); - break; - } - spin_lock_irq(&lruvec->lru_lock); - } -} - -void lru_note_cost_refault(struct folio *folio) -{ - struct lruvec *lruvec; - - lruvec = folio_lruvec_lock_irq(folio); - lru_note_cost_unlock_irq(lruvec, folio_is_file_lru(folio), - folio_nr_pages(folio), 0); -} - static void lru_activate(struct lruvec *lruvec, struct folio *folio) { long nr_pages = folio_nr_pages(folio); @@ -1162,8 +1095,6 @@ void lru_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent, int child_lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid)); parent_lruvec = mem_cgroup_lruvec(parent, NODE_DATA(nid)); - parent_lruvec->anon_cost += child_lruvec->anon_cost; - parent_lruvec->file_cost += child_lruvec->file_cost; for_each_lru(lru) lruvec_reparent_lru(child_lruvec, parent_lruvec, lru, nid); diff --git a/mm/internal.h b/mm/internal.h index 16750b130ec4..07f60ca0b201 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -41,9 +41,6 @@ void workingset_refault(struct folio *folio, void *shadow); void workingset_activation(struct folio *folio); /* mm/folio.c */ -void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file, - unsigned int nr_io, unsigned int nr_rotated); -void lru_note_cost_refault(struct folio *folio); void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma); static inline bool folio_may_be_lru_cached(struct folio *folio) diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c index 2dc599484d00..835fc8e51184 100644 --- a/mm/memcontrol-v1.c +++ b/mm/memcontrol-v1.c @@ -2287,8 +2287,8 @@ void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) for_each_online_pgdat(pgdat) { mz = memcg->nodeinfo[pgdat->node_id]; - anon_cost += mz->lruvec.anon_cost; - file_cost += mz->lruvec.file_cost; + anon_cost += mz->lruvec.cost[WORKINGSET_ANON].count; + file_cost += mz->lruvec.cost[WORKINGSET_FILE].count; } seq_buf_printf(s, "anon_cost %lu\n", anon_cost); seq_buf_printf(s, "file_cost %lu\n", file_cost); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 011bb3b0346b..65057b59b097 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -399,6 +399,7 @@ static const unsigned int memcg_node_stat_items[] = { NR_SHMEM_THPS, NR_FILE_THPS, NR_ANON_THPS, + NR_VMSCAN_WRITE, NR_VMALLOC, NR_KERNEL_STACK_KB, NR_PAGETABLE, diff --git a/mm/mmzone.c b/mm/mmzone.c index 59dc3f2076a6..9cc9ef588580 100644 --- a/mm/mmzone.c +++ b/mm/mmzone.c @@ -79,6 +79,7 @@ void lruvec_init(struct lruvec *lruvec) memset(lruvec, 0, sizeof(struct lruvec)); spin_lock_init(&lruvec->lru_lock); + spin_lock_init(&lruvec->cost_lock); zswap_lruvec_state_init(lruvec); for_each_lru(lru) diff --git a/mm/vmscan.c b/mm/vmscan.c index fa2c5cf577af..3a6701143620 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -670,7 +670,7 @@ static pageout_t pageout(struct folio *folio, struct address_space *mapping, folio_clear_reclaim(folio); trace_mm_vmscan_write_folio(folio); - node_stat_add_folio(folio, NR_VMSCAN_WRITE); + lruvec_stat_mod_folio(folio, NR_VMSCAN_WRITE, folio_nr_pages(folio)); return PAGE_SUCCESS; } @@ -1413,8 +1413,6 @@ retry: sc->nr_scanned -= (nr_pages - 1); nr_pages = 1; } - stat->nr_pageout += nr_pages; - if (folio_test_writeback(folio)) goto keep; if (folio_test_dirty(folio)) @@ -2042,9 +2040,6 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, mod_lruvec_state(lruvec, PGROTATE_ANON + file, nr_scanned - nr_reclaimed); - lruvec_lock_irq(lruvec); - lru_note_cost_unlock_irq(lruvec, file, stat.nr_pageout, - nr_scanned - nr_reclaimed); handle_reclaim_writeback(nr_taken, pgdat, sc, &stat); trace_mm_vmscan_lru_shrink_inactive(pgdat->node_id, nr_scanned, nr_reclaimed, &stat, sc->priority, file); @@ -2153,8 +2148,6 @@ static void shrink_active_list(unsigned long nr_to_scan, if (nr_rotated) mod_lruvec_state(lruvec, PGROTATE_ANON + file, nr_rotated); - lruvec_lock_irq(lruvec); - lru_note_cost_unlock_irq(lruvec, file, 0, nr_rotated); trace_mm_vmscan_lru_shrink_active(pgdat->node_id, nr_taken, nr_activate, nr_deactivate, nr_rotated, sc->priority, file); } @@ -2287,8 +2280,10 @@ enum scan_balance { static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) { - unsigned long file; + struct lru_cost *anon_cost, *file_cost; struct lruvec *target_lruvec; + unsigned long lrusize; + unsigned long file; if (lru_gen_enabled() && !lru_gen_switching()) return; @@ -2304,11 +2299,69 @@ static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) /* * Determine the scan balance between anon and file LRUs. + * + * The cost model is based on rotations, refaults and + * reclaim-driven writes (anon only) on each side. + * + * These event counters are monotonic, so each reclaim cycle + * the delta since the last scan is extracted and incorporated + * into a decaying average. This ensures currency, as workloads + * change over time, and avoids overflow in the calculations. + * + * Use lruvec_page_state_monotonic() so unsigned subtraction + * yields the correct delta across a signed-long wraparound of + * the underlying counter (a real hazard on 32-bit that the + * clamp in lruvec_page_state() would otherwise turn into a huge + * spurious delta). */ - spin_lock_irq(&target_lruvec->lru_lock); - sc->anon_cost = target_lruvec->anon_cost; - sc->file_cost = target_lruvec->file_cost; - spin_unlock_irq(&target_lruvec->lru_lock); + spin_lock(&target_lruvec->cost_lock); + + for (int f = 0; f <= 1; f++) { + struct lru_cost *cost = &target_lruvec->cost[f]; + unsigned long rotated, io, nr_rotated, nr_io; + + rotated = lruvec_page_state_monotonic(target_lruvec, + PGROTATE_ANON + f); + io = lruvec_page_state_monotonic(target_lruvec, + WORKINGSET_RESTORE_BASE + f); + if (f == WORKINGSET_ANON) + io += lruvec_page_state_monotonic(target_lruvec, + NR_VMSCAN_WRITE); + + nr_rotated = rotated - cost->last_rotated; + nr_io = io - cost->last_io; + + /* + * Reflect the relative cost of incurring IO and spending + * CPU time on rotations. This doesn't attempt to make a + * precise comparison, it just says: if reloads are about + * comparable between the LRU lists, or rotations are + * overwhelmingly different between them, adjust scan + * balance for CPU work. + */ + cost->count += nr_io * SWAP_CLUSTER_MAX + nr_rotated; + + cost->last_rotated = rotated; + cost->last_io = io; + } + + anon_cost = &target_lruvec->cost[WORKINGSET_ANON]; + file_cost = &target_lruvec->cost[WORKINGSET_FILE]; + + lrusize = lruvec_page_state(target_lruvec, NR_INACTIVE_ANON) + + lruvec_page_state(target_lruvec, NR_ACTIVE_ANON) + + lruvec_page_state(target_lruvec, NR_INACTIVE_FILE) + + lruvec_page_state(target_lruvec, NR_ACTIVE_FILE); + + while (anon_cost->count + file_cost->count > lrusize / 4) { + anon_cost->count /= 2; + file_cost->count /= 2; + } + + sc->anon_cost = anon_cost->count; + sc->file_cost = file_cost->count; + + spin_unlock(&target_lruvec->cost_lock); /* * Target desirable inactive:active list ratios for the anon diff --git a/mm/workingset.c b/mm/workingset.c index f351798e723a..7ac2b88c80ae 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -584,11 +584,6 @@ void workingset_refault(struct folio *folio, void *shadow) /* Folio was active prior to eviction */ if (workingset) { folio_set_workingset(folio); - /* - * XXX: Move to folio_add_lru() when it supports new vs - * putback - */ - lru_note_cost_refault(folio); mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + file, nr); } out: -- cgit v1.2.3 From dde75313eed0b014c437f48dd75c0308b592cbf9 Mon Sep 17 00:00:00 2001 From: Sergey Senozhatsky Date: Wed, 29 Jul 2026 13:57:35 +0900 Subject: zram: set default primary compressor in zram_destroy_comps() Patch series "zram: fix zram issues reported by sashiko". Sashiko drove by and reported [1] a couple of zram issues: a possible BUG_ON() in zlib code due to missing winbits range validation and one possible NULL-ptr dereference in zcomp. Both are low risk yet still worth fixing. This patch (of 2): zram_destroy_comps() resets all compressors and leaves them set to NULL, including the primary one, which is invalid device state, as now comp_algorithm_show()->strcmp() can be called on a NULL compressor. Set default primary compressor in zram_destroy_comps(). Link: https://lore.kernel.org/20260729045745.775973-2-senozhatsky@chromium.org Fixes: 486fd58af7ac ("zram: don't free statically defined names") Link: https://sashiko.dev/#/patchset/20260728092935.31139-1-haoqinhuang7@gmail.com [1] Signed-off-by: Sergey Senozhatsky Cc: Minchan Kim Cc: Haoqin Huang Cc: Signed-off-by: Andrew Morton --- drivers/block/zram/zram_drv.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 4bfe63a5225d..cfa98846ac48 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2828,6 +2828,7 @@ static void zram_destroy_comps(struct zram *zram) zram->comp_algs[prio] = NULL; zram_comp_params_reset(zram); + comp_algorithm_set(zram, ZRAM_PRIMARY_COMP, default_compressor); } static void zram_reset_device(struct zram *zram) @@ -2845,8 +2846,6 @@ static void zram_reset_device(struct zram *zram) zram_destroy_comps(zram); memset(&zram->stats, 0, sizeof(zram->stats)); reset_bdev(zram); - - comp_algorithm_set(zram, ZRAM_PRIMARY_COMP, default_compressor); } static ssize_t disksize_store(struct device *dev, struct device_attribute *attr, -- cgit v1.2.3 From ec7607ac4717ff521c9c1e9d8271c26293345513 Mon Sep 17 00:00:00 2001 From: Sergey Senozhatsky Date: Wed, 29 Jul 2026 13:57:36 +0900 Subject: zram: validate deflate params We must validate user-supplied deflate winbits before we pass it to zlib_deflate_workspacesize(), which triggers BUG_ON() if winbits value is outside of valid ranges. Link: https://lore.kernel.org/20260729045745.775973-3-senozhatsky@chromium.org Fixes: dc75a0d93bd5 ("zram: support deflate-specific params") Link: https://sashiko.dev/#/patchset/20260728092935.31139-1-haoqinhuang7@gmail.com Signed-off-by: Sergey Senozhatsky Cc: Minchan Kim Cc: Haoqin Huang Cc: Signed-off-by: Andrew Morton --- drivers/block/zram/backend_deflate.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/drivers/block/zram/backend_deflate.c b/drivers/block/zram/backend_deflate.c index f92a52a720d1..b3f7d08b49d9 100644 --- a/drivers/block/zram/backend_deflate.c +++ b/drivers/block/zram/backend_deflate.c @@ -24,8 +24,16 @@ static int deflate_setup_params(struct zcomp_params *params) { if (params->level == ZCOMP_PARAM_NOT_SET) params->level = Z_DEFAULT_COMPRESSION; - if (params->deflate.winbits == ZCOMP_PARAM_NOT_SET) + if (params->deflate.winbits == ZCOMP_PARAM_NOT_SET) { params->deflate.winbits = DEFLATE_DEF_WINBITS; + } else { + s32 wb = params->deflate.winbits; + + if ((wb < -15 || wb > -9) && (wb < 9 || wb > 15)) { + pr_err("invalid deflate winbits: %d\n", wb); + return -EINVAL; + } + } return 0; } -- cgit v1.2.3 From 9477820c63cbf4d97114238f3d1ff10dfd6bee3f Mon Sep 17 00:00:00 2001 From: Guopeng Zhang Date: Fri, 24 Jul 2026 10:18:05 +0800 Subject: mm: memcg: stop reclaim when a limit update is superseded kernfs serializes file operations only per open file, so separate open files can update the same memory.high or memory.max file concurrently. Both handlers store the new limit before synchronous reclaim, but continue to use the writer's local target in the reclaim loop. If another writer raises or removes the limit, the first writer can continue reclaiming toward a stale target. For memory.max, this can leave the writer looping indefinitely once reclaim retries are exhausted. The OOM path sees sufficient margin under the current limit and returns true without killing, while the writer still compares usage against its stale target and records another OOM event. Check the current limit at the start of each reclaim iteration and stop if it no longer matches the writer's target. Reproducer: Populate a cgroup with anonymous memory and disable swapping. Lower memory.max from one open file, then restore it to "max" through another open file after the new limit becomes visible. Without the patch, the first writer remains blocked and repeatedly increments the OOM event counter. With the patch, it returns normally. This was not motivated by a reported production workload. We found it through automated randomized testing for our cgroup observability work and reduced it to the reproducer above. Link: https://lore.kernel.org/20260724021805.1234583-1-guopeng.zhang@linux.dev Fixes: 8c8c383c04f6 ("mm: memcontrol: try harder to set a new memory.high") Fixes: b6e6edcfa405 ("mm: memcontrol: reclaim and OOM kill when shrinking memory.max below usage") Signed-off-by: Guopeng Zhang Acked-by: Tao Cui Acked-by: Johannes Weiner Cc: Michal Hocko Cc: Muchun Song Cc: Roman Gushchin Cc: Shakeel Butt Cc: Signed-off-by: Andrew Morton --- mm/memcontrol.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 65057b59b097..0dd847af69a2 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4839,6 +4839,9 @@ static ssize_t memory_high_write(struct kernfs_open_file *of, unsigned long nr_pages = page_counter_read(&memcg->memory); unsigned long reclaimed; + if (high != READ_ONCE(memcg->memory.high)) + break; + if (nr_pages <= high) break; @@ -4894,6 +4897,9 @@ static ssize_t memory_max_write(struct kernfs_open_file *of, for (;;) { unsigned long nr_pages = page_counter_read(&memcg->memory); + if (max != READ_ONCE(memcg->memory.max)) + break; + if (nr_pages <= max) break; -- cgit v1.2.3 From f74e6dff5440f662bced614b723a7d8f2b05919d Mon Sep 17 00:00:00 2001 From: Sergey Senozhatsky Date: Thu, 30 Jul 2026 16:51:46 +0900 Subject: Documentation: zram: correct algo parameters configuration documentation zram has always reset all previously set parameters for the given algorithm in comp_params_store(). Make documentation more clear and explicitly state that all relevant/necessary parameters should be set in one configuration write. Link: https://lore.kernel.org/20260730075158.1339787-1-senozhatsky@chromium.org Signed-off-by: Sergey Senozhatsky Cc: Jonathan Corbet Cc: Minchan Kim Signed-off-by: Andrew Morton --- Documentation/admin-guide/blockdev/zram.rst | 34 ++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/Documentation/admin-guide/blockdev/zram.rst b/Documentation/admin-guide/blockdev/zram.rst index 2f6bbfd991fe..148b7cf3b924 100644 --- a/Documentation/admin-guide/blockdev/zram.rst +++ b/Documentation/admin-guide/blockdev/zram.rst @@ -109,14 +109,41 @@ path to the `dict` along with other parameters:: #pass path to pre-trained zstd dictionary echo "algo=zstd dict=/etc/dictionary" > /sys/block/zram0/algorithm_params + #pass path to pre-trained zstd dictionary and compression level + echo "algo=zstd level=8 dict=/etc/dictionary" > \ + /sys/block/zram0/algorithm_params + #same, but using algorithm priority + echo "algo=zstd priority=1" > /sys/block/zram0/recomp_algorithm echo "priority=1 dict=/etc/dictionary" > \ /sys/block/zram0/algorithm_params - #pass path to pre-trained zstd dictionary and compression level +Each write to `algorithm_params` replaces the entire set of parameters of +the corresponding algorithm, parameters that are not listed in the write +are reset to their default values. Configure all of the parameters of an +algorithm in one write:: + + #WRONG: the second write resets level back to its default value + echo "algo=zstd level=8" > /sys/block/zram0/algorithm_params + echo "algo=zstd dict=/etc/dictionary" > /sys/block/zram0/algorithm_params + + #RIGHT echo "algo=zstd level=8 dict=/etc/dictionary" > \ /sys/block/zram0/algorithm_params +Select the compression algorithm before configuring its parameters. The +parameters of one algorithm are not necessarily valid for another one, so +changing the algorithm of a particular priority resets that priority's +parameters:: + + #WRONG: comp_algorithm write resets the previously configured level + echo "level=8" > /sys/block/zram0/algorithm_params + echo zstd > /sys/block/zram0/comp_algorithm + + #RIGHT + echo zstd > /sys/block/zram0/comp_algorithm + echo "algo=zstd level=8" > /sys/block/zram0/algorithm_params + Parameters are algorithm specific: not all algorithms support pre-trained dictionaries, not all algorithms support `level`. Furthermore, for certain algorithms `level` controls the compression level (the higher the value the @@ -124,6 +151,11 @@ better the compression ratio, it even can take negatives values for some algorithms), for other algorithms `level` is acceleration level (the higher the value the lower the compression ratio). +Parameters are handed over to the compression algorithm when the device is +initialised, hence invalid parameters (or parameters that the selected +algorithm does not support) are reported by the `disksize` write, and not +by the `algorithm_params` write that has configured them. + Set Disksize ============ -- cgit v1.2.3 From 6b0d1083364fc8e7cc2f7d1f93ee3ee78f4d52f7 Mon Sep 17 00:00:00 2001 From: Shakeel Butt Date: Tue, 28 Jul 2026 19:46:12 -0700 Subject: memcg: bypass the reclaim and oom killer for dying tasks once oom_reaper is done At Meta, we are seeing instances where an OOM killed job is stuck in the exit path for several hours. In one particular case, the job was stuck for more than 8 hours and I had to manually remove the memory.max limits to allow the process to exit. The job was a single process job and had ~55 GiB memory.max and zswap enabled. It had almost 0 anon in memory and ~111 GiB in zswap compressed to ~51 GiB zswap pool (i.e. almost all of memory.current was zswap). Nothing was left on the LRUs to reclaim. On further inspection, I observed ~20k threads of that process stuck with the following stack: [<0>] mem_cgroup_out_of_memory+0x4e/0xa0 [<0>] charge_memcg+0x8bf/0x990 [<0>] mem_cgroup_swapin_charge_folio+0x4e/0x80 [<0>] __read_swap_cache_async+0x10c/0x260 [<0>] swapin_readahead+0x116/0x3f0 [<0>] do_swap_page+0x13c/0x1ce0 [<0>] handle_mm_fault+0x61d/0x11f0 [<0>] do_user_addr_fault+0x3e7/0x6d0 [<0>] exc_page_fault+0x8f/0x110 [<0>] asm_exc_page_fault+0x22/0x30 [<0>] __get_user_8+0x14/0x20 [<0>] futex_cleanup+0x27/0x1c0 [<0>] futex_exit_release+0x47/0x60 [<0>] do_exit+0x107/0x940 [<0>] do_group_exit+0x81/0xa0 [<0>] get_signal+0x2b1/0x6e0 [<0>] arch_do_signal_or_restart+0x1a/0x1c0 [<0>] exit_to_user_mode_loop+0xa8/0x1c0 [<0>] do_syscall_64+0x152/0x250 [<0>] entry_SYSCALL_64_after_hwframe+0x4b/0x53 In addition the dmesg was filled with "Out of memory and no killable processes..." messages. I have no idea why oom reaper was not able to reap/unmap the process. My guess is that since oom reaper tries to acquire mmap_lock in read mode limited number of times and then gives up, there might be a thread of that process which had mmap_lock in write mode at that time. My initial suspicion was the futex_cleanup and kernel page fault causing infinite fault and charge retries but that was put to rest in previous discussions happened on similar problem [1]. My current theory is that it is just a simple slow serialization behind the oom_lock. Unlike page allocator, memcg charge code takes the oom_lock without the "try". Though memcg oom code uses mutex_lock_killable(), note that in the call stack get_signal() consumes SIGKILL (or sigdelset(SIGKILL)) before calling do_group_exit(). So this mutex_lock_killable() is just a mutex_lock() here. Therefore 10s of thousands of threads are waiting on oom_lock and one by one they get -EFAULT from get_user() in the futex cleanup code and bails out. Discussion from [1] led to commit a75ffa26122b ("memcg, oom: do not bypass oom killer for dying tasks") which routes dying tasks into the OOM path precisely so the oom_reaper can reap their mm and free the memory asynchronously. But the reaper is best-effort and one-shot: if it cannot take mmap_lock for read (e.g. a sibling thread holds it for write) it sets MMF_OOM_SKIP and never retries, leaving only the glacial oom_lock-serialized synchronous drain. Once MMF_OOM_SKIP is set there is no more asynchronous reclaim coming for the mm, so a dying task charging against it has nothing left to wait for: it frees its memory only once it finishes exiting. Running reclaim and the (no-victim) OOM killer for it is then pointless, and doing it for 10s of thousands of exiting threads is what serializes them behind oom_lock. So before reclaim, if current is an OOM victim whose reaper is done, fail the charge. Reproduced with 20k threads, each parking a robust futex head on its own zswapped page, OOM-group-killed while a sibling holds mmap_lock for write so the reaper gives up and sets MMF_OOM_SKIP. Tested on next-20260728 and baseline show ~90 seconds exit time while with the patch the exit time reduced to ~3 seconds. Link: https://lore.kernel.org/20260729024612.3369005-1-shakeel.butt@linux.dev Link: https://lore.kernel.org/7a4e5591f45df455e6a485fc5400989569d3d22d.camel@surriel.com/ [1] Signed-off-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Michal Hocko Cc: David Rientjes Cc: Muchun Song Cc: Nhat Pham Cc: Rik van Riel Cc: Roman Gushchin Cc: Suren Baghdasaryan Cc: Signed-off-by: Andrew Morton --- mm/memcontrol.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0dd847af69a2..1d3339520809 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2693,6 +2693,19 @@ retry: if (!gfpflags_allow_blocking(gfp_mask)) goto nomem; + /* + * OOM victim still needs to charge memory to exit. OOM reaper should + * help but it might fail on mmap_lock contention. If the victim is a + * large thread group then all exiting threads might compete on oom_lock + * just to learn that there is nothing really killable anymore. Bail + * out early and fail the charge to expedite their exit. They are + * considered fully reclaimed by the oom reaper and they shouldn't + * contribute further charges. + */ + if (tsk_is_oom_victim(current) && + mm_flags_test(MMF_OOM_SKIP, current->signal->oom_mm)) + goto nomem; + __memcg_memory_event(mem_over_limit, MEMCG_MAX, allow_spinning); raised_max_event = true; -- cgit v1.2.3 From 62e39381b7804eb94a43bd1e0b0c216aad3e5f08 Mon Sep 17 00:00:00 2001 From: Alexander Gordeev Date: Mon, 20 Jul 2026 16:16:33 +0200 Subject: mm: use proper PTE accessor in move_ptes() Follow the pattern established by commit c33c794828f2 ("mm: ptep_get() conversion") and use the proper PTE accessor instead of a direct pointer dereference. Link: https://lore.kernel.org/20260720141633.501799-1-agordeev@linux.ibm.com Fixes: b36b701bbcd9 ("mm: expose abnormal new_pte during move_ptes") Signed-off-by: Alexander Gordeev Acked-by: David Hildenbrand (Arm) Cc: Ryan Roberts Signed-off-by: Andrew Morton --- mm/mremap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/mremap.c b/mm/mremap.c index 9ea1707eafa5..e8df5cdb0ac9 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -264,7 +264,7 @@ static int move_ptes(struct pagetable_move_control *pmc, for (; old_addr < old_end; old_ptep += nr_ptes, old_addr += nr_ptes * PAGE_SIZE, new_ptep += nr_ptes, new_addr += nr_ptes * PAGE_SIZE) { - VM_WARN_ON_ONCE(!pte_none(*new_ptep)); + VM_WARN_ON_ONCE(!pte_none(ptep_get(new_ptep))); nr_ptes = 1; max_nr_ptes = (old_end - old_addr) >> PAGE_SHIFT; -- cgit v1.2.3 From 5120b1e048d48596ffaec1a8412012a91adba73b Mon Sep 17 00:00:00 2001 From: Guillaume Morin Date: Tue, 28 Jul 2026 21:29:03 +0200 Subject: hugetlb: only adjust reservation during unmapping if mapcount is 0 Since df7a6d1f6405, __unmap_hugepage_range can adjust reservations. In the case of folio mapped in both a parent and a child, if the parent unmaps the range first, the reservation adjustment will result in an underflow of the reserved count. Once the child unmaps the range, the count is restored. Change __unmap_hugepage_range() to check the mapcount before adjusting the reservation. Link: https://lore.kernel.org/all/alEJkwn5VlTTH_ZX@bender.morinfr.org/ Link: https://lore.kernel.org/amkC_1Ya6OiUoiLZ@bender.morinfr.org Fixes: df7a6d1f6405 ("mm/hugetlb: restore the reservation if needed") Signed-off-by: Guillaume Morin Reviewed-by: Breno Leitao Reviewed-by: Rik van Riel Cc: Muchun Song Cc: David Hildenbrand Cc: Oscar Salvador Cc: Signed-off-by: Andrew Morton --- mm/hugetlb.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 49bf325325c0..7d14511c20ad 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5218,6 +5218,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, bool adjust_reservation; unsigned long last_addr_mask; + i_mmap_assert_write_locked(vma->vm_file->f_mapping); WARN_ON(!is_vm_hugetlb_page(vma)); BUG_ON(start & ~huge_page_mask(h)); BUG_ON(end & ~huge_page_mask(h)); @@ -5309,7 +5310,10 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, /* * Restore the reservation for anonymous page, otherwise the - * backing page could be stolen by someone. + * backing page could be stolen by someone. Restore only on the + * last unmap, otherwise the owner could empty its resv map + * while the folio is still mapped by a child. Note that holding + * i_mmap_lock_write is needed to check the number of mappings. * If there we are freeing a surplus, do not set the restore * reservation bit. */ @@ -5317,7 +5321,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, spin_lock_irq(&hugetlb_lock); if (!h->surplus_huge_pages && __vma_private_lock(vma) && - folio_test_anon(folio)) { + !folio_mapped(folio) && folio_test_anon(folio)) { folio_set_hugetlb_restore_reserve(folio); /* Reservation to be adjusted after the spin lock */ adjust_reservation = true; -- cgit v1.2.3 From 7a39f03bc9da3499c2423758f353ab72d23faa16 Mon Sep 17 00:00:00 2001 From: Pratyush Mallick Date: Fri, 31 Jul 2026 19:37:05 +0000 Subject: mm/page_reporting: add page_reporting_delay_ms module parameter Free page reporting currently hardcodes a 2-second interval between reports. This rigid delay cannot accommodate diverse guest workloads. This patch introduces a module parameter, page_reporting_delay_ms (default: 2000), allowing users to tune the reporting rate: - Lower values enable aggressive memory reclamation by returning unused pages to the host immediately. - Higher values help batch pages during spiky allocation/free churn, reducing hypercalls and nested page fault overheads. Setting the delay to 0 is safe and execution is strictly gated by: - reporting is only triggered by high-order page frees. - expensive hypercalls are bounded by a slot capacity watermark check before proceeding. Link: https://lore.kernel.org/20260731193705.2902728-1-pratmal@google.com Signed-off-by: Pratyush Mallick Reviewed-by: SJ Park Acked-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Cc: Anshuman Khandual Cc: Brendan Jackman Cc: Greg Thelen Cc: Johannes Weiner Cc: Jonathan Corbet Cc: Liam R. Howlett Cc: Michal Hocko Cc: Mike Rapoport Cc: SeongJae Park Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Zi Yan Signed-off-by: Andrew Morton --- Documentation/admin-guide/kernel-parameters.txt | 6 ++++++ mm/page_reporting.c | 28 ++++++++++++++++--------- 2 files changed, 24 insertions(+), 10 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index b5493a7f8f22..364c2dce8e70 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -4810,6 +4810,12 @@ Kernel parameters Adjust the minimal page reporting order. The page reporting is disabled when it exceeds MAX_PAGE_ORDER. + page_reporting.page_reporting_delay_ms= + [KNL] Free page reporting delay in milliseconds + Format: + Adjust the delay in milliseconds between free page + reporting intervals. Default is 2000 (2 seconds). + panic= [KNL] Kernel behaviour on panic: delay timeout > 0: seconds before rebooting timeout = 0: wait forever diff --git a/mm/page_reporting.c b/mm/page_reporting.c index 1cce8729696e..de587be17801 100644 --- a/mm/page_reporting.c +++ b/mm/page_reporting.c @@ -48,7 +48,11 @@ MODULE_PARM_DESC(page_reporting_order, "Set page reporting order"); */ EXPORT_SYMBOL_GPL(page_reporting_order); -#define PAGE_REPORTING_DELAY (2 * HZ) +static unsigned int page_reporting_delay_ms = 2 * MSEC_PER_SEC; +module_param(page_reporting_delay_ms, uint, 0644); +MODULE_PARM_DESC(page_reporting_delay_ms, + "Set page reporting delay in milliseconds"); + static struct page_reporting_dev_info __rcu *pr_dev_info __read_mostly; enum { @@ -57,6 +61,13 @@ enum { PAGE_REPORTING_ACTIVE }; +/* schedule work for page reporting */ +static void page_reporting_schedule_work(struct page_reporting_dev_info *prdev) +{ + queue_delayed_work(system_freezable_wq, &prdev->work, + msecs_to_jiffies(page_reporting_delay_ms)); +} + /* request page reporting */ static void __page_reporting_request(struct page_reporting_dev_info *prdev) @@ -77,12 +88,10 @@ __page_reporting_request(struct page_reporting_dev_info *prdev) return; /* - * Delay the start of work to allow a sizable queue to build. For - * now we are limiting this to running no more than once every - * couple of seconds. + * Delay the start of work to allow a sizable queue to build. + * We limit this based on page_reporting_delay_ms. */ - queue_delayed_work(system_freezable_wq, &prdev->work, - PAGE_REPORTING_DELAY); + page_reporting_schedule_work(prdev); } /* notify prdev of free page reporting request */ @@ -337,13 +346,12 @@ static void page_reporting_process(struct work_struct *work) err_out: /* * If the state has reverted back to requested then there may be - * additional pages to be processed. We will defer for 2s to allow - * more pages to accumulate. + * additional pages to be processed. We will defer by + * page_reporting_delay_ms to allow more pages to accumulate. */ state = atomic_cmpxchg(&prdev->state, state, PAGE_REPORTING_IDLE); if (state == PAGE_REPORTING_REQUESTED) - queue_delayed_work(system_freezable_wq, &prdev->work, - PAGE_REPORTING_DELAY); + page_reporting_schedule_work(prdev); } static DEFINE_MUTEX(page_reporting_mutex); -- cgit v1.2.3 From 8380671909bfcdd44818abf6b93f82bd3669bc8c Mon Sep 17 00:00:00 2001 From: Sang-Heon Jeon Date: Sat, 1 Aug 2026 01:47:55 +0900 Subject: mm/sparse: correct init section annotations The !SPARSEMEM_EXTREME stub of sparse_index_init() has no annotation but the SPARSEMEM_EXTREME variant is __meminit. So mark the stub __meminit too. mminit_validate_memmodel_limits() is only called by memory_present(), which is __init. So mark it __init. sparse_usagebuf and sparse_usagebuf_end are only used by sparse_init_early_section(), sparse_usage_init() and sparse_usage_fini(), which are all __init. So mark them __initdata. Link: https://lore.kernel.org/20260731164758.1210668-1-ekffu200098@gmail.com Signed-off-by: Sang-Heon Jeon Reviewed-by: Andrew Morton Acked-by: David Hildenbrand (Arm) Acked-by: Mike Rapoport (Microsoft) Reviewed-by: Anshuman Khandual Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- mm/sparse.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/mm/sparse.c b/mm/sparse.c index 704a9dec2b9a..67fa192d4289 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -104,7 +104,7 @@ int __meminit sparse_index_init(unsigned long section_nr, int nid) return 0; } #else /* !SPARSEMEM_EXTREME */ -int sparse_index_init(unsigned long section_nr, int nid) +int __meminit sparse_index_init(unsigned long section_nr, int nid) { return 0; } @@ -127,7 +127,7 @@ static inline int sparse_early_nid(struct mem_section *section) } /* Validate the physical addressing limitations of the model */ -static void __meminit mminit_validate_memmodel_limits(unsigned long *start_pfn, +static void __init mminit_validate_memmodel_limits(unsigned long *start_pfn, unsigned long *end_pfn) { unsigned long max_sparsemem_pfn = (DIRECT_MAP_PHYSMEM_END + 1) >> PAGE_SHIFT; @@ -249,8 +249,8 @@ void __weak __meminit vmemmap_populate_print_last(void) { } -static void *sparse_usagebuf __meminitdata; -static void *sparse_usagebuf_end __meminitdata; +static void *sparse_usagebuf __initdata; +static void *sparse_usagebuf_end __initdata; /* * Helper function that is used for generic section initialization, and -- cgit v1.2.3 From 7822fa5f4f647d05d31eea3edd01382c6183e4cd Mon Sep 17 00:00:00 2001 From: Wilson Felipe Pereira Date: Fri, 31 Jul 2026 05:37:06 +0000 Subject: mm: zswap: drop list_lru param from zswap_lru_add() and _del() Since zswap_lru_add() and zswap_lru_del() are only called with the global zswap_list_lru, remove the redundant list_lru argument and use zswap_list_lru directly. Link: https://lore.kernel.org/20260731053721.1412304-1-wfelipe@google.com Signed-off-by: Wilson Felipe Pereira Acked-by: Johannes Weiner Acked-by: Yosry Ahmed Reviewed-by: SJ Park Acked-by: Nhat Pham Cc: Chengming Zhou Signed-off-by: Andrew Morton --- mm/zswap.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 761cd699e0a3..a810524c7621 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -665,7 +665,7 @@ static inline int entry_to_nid(struct zswap_entry *entry) return page_to_nid(virt_to_page(entry)); } -static void zswap_lru_add(struct list_lru *list_lru, struct zswap_entry *entry) +static void zswap_lru_add(struct zswap_entry *entry) { int nid = entry_to_nid(entry); struct mem_cgroup *memcg; @@ -684,11 +684,11 @@ static void zswap_lru_add(struct list_lru *list_lru, struct zswap_entry *entry) rcu_read_lock(); memcg = mem_cgroup_from_entry(entry); /* will always succeed */ - list_lru_add(list_lru, &entry->lru, nid, memcg); + list_lru_add(&zswap_list_lru, &entry->lru, nid, memcg); rcu_read_unlock(); } -static void zswap_lru_del(struct list_lru *list_lru, struct zswap_entry *entry) +static void zswap_lru_del(struct zswap_entry *entry) { int nid = entry_to_nid(entry); struct mem_cgroup *memcg; @@ -696,7 +696,7 @@ static void zswap_lru_del(struct list_lru *list_lru, struct zswap_entry *entry) rcu_read_lock(); memcg = mem_cgroup_from_entry(entry); /* will always succeed */ - list_lru_del(list_lru, &entry->lru, nid, memcg); + list_lru_del(&zswap_list_lru, &entry->lru, nid, memcg); rcu_read_unlock(); } @@ -764,7 +764,7 @@ static void zswap_entry_cache_free(struct zswap_entry *entry) */ static void zswap_entry_free(struct zswap_entry *entry) { - zswap_lru_del(&zswap_list_lru, entry); + zswap_lru_del(entry); zs_free(entry->pool->zs_pool, entry->handle); zswap_pool_put(entry->pool); if (entry->objcg) { @@ -1461,7 +1461,7 @@ static bool zswap_store_page(struct page *page, entry->referenced = true; if (entry->length) { INIT_LIST_HEAD(&entry->lru); - zswap_lru_add(&zswap_list_lru, entry); + zswap_lru_add(entry); } return true; -- cgit v1.2.3 From 34a00895d032a414830d41106a09329ae6c251b6 Mon Sep 17 00:00:00 2001 From: Arvind Yadav Date: Tue, 28 Jul 2026 11:58:32 +0530 Subject: mm/migrate_device: clear stale mapping after freeing swapcache __migrate_device_pages() reads the folio mapping before calling folio_free_swap(). When folio_free_swap() succeeds, the folio is removed from the swap cache, but the saved mapping still points to swap_space. Passing the stale mapping to folio_migrate_mapping() makes it use the mapped-folio path for a folio that is no longer in swapcache. It can then operate on swap_space.i_pages with invalid reference accounting, eventually triggering a folio reference count BUG. After a successful split, nr still contains the number of pages in the original large folio, although each resulting page is now a separate order-0 folio. Reset nr to 1 so each split folio is processed separately, including its own swapcache removal and mapping lookup. Refresh the saved mapping after folio_free_swap() so the current folio state is used during migration. Link: https://lore.kernel.org/20260728062832.1107127-1-arvind.yadav@intel.com Fixes: df263d9a7dff ("mm/migrate_device: try to handle swapcache pages") Signed-off-by: Arvind Yadav Reviewed-by: Zi Yan Reviewed-by: Balbir Singh Cc: David Hildenbrand Cc: Matthew Brost Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Signed-off-by: Andrew Morton --- mm/migrate_device.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 18d097c38853..9a346162c688 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -1193,6 +1193,13 @@ static void __migrate_device_pages(unsigned long *src_pfns, MIGRATE_PFN_COMPOUND); goto next; } + + /* + * reset nr so that only first after-split folio + * is processed below + */ + VM_WARN_ON_ONCE(folio_test_large(folio)); + nr = 1; } else if ((src_pfns[i] & MIGRATE_PFN_MIGRATE) && (dst_pfns[i] & MIGRATE_PFN_COMPOUND) && !(src_pfns[i] & MIGRATE_PFN_COMPOUND)) { @@ -1232,6 +1239,12 @@ static void __migrate_device_pages(unsigned long *src_pfns, folio = page_folio(migrate_pfn_to_page(src_pfns[i+j])); newfolio = page_folio(migrate_pfn_to_page(dst_pfns[i+j])); + /* + * folio_free_swap() removed the folio from the swap + * cache. Refresh the saved mapping before migration. + */ + mapping = folio_mapping(folio); + r = folio_migrate_mapping(mapping, newfolio, folio, extra_cnt); if (r) src_pfns[i+j] &= ~MIGRATE_PFN_MIGRATE; -- cgit v1.2.3 From c299a2285d9d8bda4da024455de65e3d00de6f17 Mon Sep 17 00:00:00 2001 From: Zi Yan Date: Tue, 4 Aug 2026 17:04:27 -0400 Subject: mm/huge_memory: use folio's memcg inside __folio_split() Patch series "Honor XA_FLAGS_ACCOUNT in xas_split_alloc() and charge to folio's memcg", v3. __GFP_ACCOUNT is needed for xarray node allocation accounting when XA_FLAGS_ACCOUNT is set. Commit 7b785645e8f13 ("mm: fix page cache convergence regression") fixed a workingset regression with it. xas_split_alloc() does not have it and needs to be fixed. In addition, based on Sashiko's review[1] and Johannes' confirmation[2], to charge the right memcg, folio's memcg needs to be active during folio split. Add that before adding __GFP_ACCOUNT. There is no workingset convergence regression related to missing __GFP_ACCOUNT in xas_split_alloc() and the impact to userspace should be minor. This patch (of 2): During a pagecache folio split, an xarray node allocation can happen and needs to charge at folio's memcg instead of folio split invoker's memcg, because for example folio split can happen during reclaim and reclaim's active memcg might not be folio's memcg. Switch to folio's memcg at the beginning and switch back afterwards. Link: https://lore.kernel.org/20260804-add-gfp_account-to-xas_split_alloc-v3-0-38cb3ff325c5@nvidia.com Link: https://lore.kernel.org/20260804-add-gfp_account-to-xas_split_alloc-v3-1-38cb3ff325c5@nvidia.com Link: https://sashiko.dev/#/patchset/20260727-add-gfp_account-to-xas_split_alloc-v1-1-9fae6bf64838%40nvidia.com?part=1 [1] Link: https://lore.kernel.org/all/amtcBZ-_QVRgCd6b@cmpxchg.org/ [2] Fixes: 6b24ca4a1a8d ("mm: Use multi-index entries in the page cache") Signed-off-by: Zi Yan Suggested-by: Johannes Weiner Reviewed-by: Baolin Wang Acked-by: Lorenzo Stoakes (ARM) Acked-by: Johannes Weiner Cc: Barry Song Cc: David Hildenbrand Cc: Dev Jain Cc: Lance Yang Cc: Liam R. Howlett Cc: Matthew Wilcox (Oracle) Cc: Ryan Roberts Cc: William Kucharski Cc: Signed-off-by: Andrew Morton --- mm/huge_memory.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a8174d1d3848..ced400f72d43 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -4105,34 +4105,42 @@ static int __folio_split(struct folio *folio, unsigned int new_order, XA_STATE(xas, &folio->mapping->i_pages, folio->index); struct folio *end_folio = folio_next(folio); bool is_anon = folio_test_anon(folio); + struct mem_cgroup *memcg, *old_memcg; struct address_space *mapping = NULL; struct anon_vma *anon_vma = NULL; int old_order = folio_order(folio); struct folio *new_folio, *next; int nr_shmem_dropped = 0; enum ttu_flags ttu_flags = 0; - int ret; pgoff_t end = 0; + int ret; VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio); VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio); if (folio != page_folio(split_at) || folio != page_folio(lock_at)) { ret = -EINVAL; - goto out; + goto out_no_memcg; } if (new_order >= old_order) { ret = -EINVAL; - goto out; + goto out_no_memcg; } ret = folio_check_splittable(folio, new_order, split_type); if (ret) { VM_WARN_ONCE(ret == -EINVAL, "Tried to split an unsplittable folio"); - goto out; + goto out_no_memcg; } + /* + * switch to folio's memcg as xarray node allocation can happen and + * needs to charge to it. + */ + memcg = get_mem_cgroup_from_folio(folio); + old_memcg = set_active_memcg(memcg); + if (is_anon) { /* * The caller does not necessarily hold an mmap_lock that would @@ -4275,6 +4283,10 @@ out_unlock: if (mapping) i_mmap_unlock_read(mapping); out: + /* restore to caller's old_memcg */ + set_active_memcg(old_memcg); + mem_cgroup_put(memcg); +out_no_memcg: xas_destroy(&xas); if (is_pmd_order(old_order)) count_vm_event(!ret ? THP_SPLIT_PAGE : THP_SPLIT_PAGE_FAILED); -- cgit v1.2.3 From 789763523fb43cdc328de5cb5dcd19240ccf90d8 Mon Sep 17 00:00:00 2001 From: Zi Yan Date: Tue, 4 Aug 2026 17:04:28 -0400 Subject: xarray: honor XA_FLAGS_ACCOUNT in xas_split_alloc() XArray operations that allocate xa_nodes, such as xas_nomem() and xas_alloc(), add __GFP_ACCOUNT when the array has XA_FLAGS_ACCOUNT set. This charges the allocated memory and avoids the workingset convergence issue described by commit 7b785645e8f13 ("mm: fix page cache convergence regression"). xas_split_alloc() does not add _GFP_ACCOUNT when XA_FLAGS_ACCOUNT is present. Fix it. Link: https://lore.kernel.org/20260804-add-gfp_account-to-xas_split_alloc-v3-2-38cb3ff325c5@nvidia.com Fixes: 6b24ca4a1a8d ("mm: Use multi-index entries in the page cache") Signed-off-by: Zi Yan Reviewed-by: Lorenzo Stoakes (ARM) Acked-by: Johannes Weiner Cc: Baolin Wang Cc: Barry Song Cc: David Hildenbrand Cc: Dev Jain Cc: Lance Yang Cc: Liam R. Howlett Cc: Matthew Wilcox (Oracle) Cc: Ryan Roberts Cc: William Kucharski Cc: Signed-off-by: Andrew Morton --- lib/xarray.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/xarray.c b/lib/xarray.c index 9a8b4916540c..bfe7bef80f34 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -1053,6 +1053,9 @@ void xas_split_alloc(struct xa_state *xas, void *entry, unsigned int order, if (xas->xa_shift + XA_CHUNK_SHIFT > order) return; + if (xas->xa->xa_flags & XA_FLAGS_ACCOUNT) + gfp |= __GFP_ACCOUNT; + do { struct xa_node *node; -- cgit v1.2.3 From 078e1a0fc41a42baaf113383b52dab8874c0b967 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Fri, 31 Jul 2026 22:27:53 +0200 Subject: mm/gup: fix always draining LRU caches in collect_longterm_unpinnable_folios() folio_may_be_lru_cached() is currently only true for small folios, and for small folios FOLL_PIN adds GUP_PIN_COUNTING_BIAS references instead of 1 in try_grab_folio()/try_grab_folio_fast(). Consequently, our folio_ref_count(folio) != folio_expected_ref_count(folio) + 1 check in collect_longterm_unpinnable_folios() will currently always identify "reference mismatch" and first drain the local LRU cache to then drain the LRU cache on all CPUs, as collect_longterm_unpinnable_folios() is really called after pinning the folios with FOLL_PIN. Add a comment because the current code is not quite intuitive: we used to drain only to make sure the folio_isolate_lru() would succeed. But then we also started draining to make later migration more reliable. We'll refactor that code soon a bit, to also make it usable in other context where we really want to remove any references from LRU caches. Let's add CC stable, because having an easy way for excessive LRU cache draining on all CPUs does not sound right. In common scenarios we don't expect to ever have to drain. Link: https://lore.kernel.org/20260731-check_and_migrate_movable_folios-v1-1-e0002d7b791e@kernel.org Fixes: 98c6d259319e ("mm/gup: check ref_count instead of lru before migration") Fixes: a09a8a1fbb37 ("mm/gup: local lru_add_drain() to avoid lru_add_drain_all()") Signed-off-by: David Hildenbrand (Arm) Acked-by: Hugh Dickins Cc: Ackerley Tng Cc: Jason Gunthorpe Cc: John Hubbard Cc: Kiryl Shutsemau Cc: Peter Xu Cc: Signed-off-by: Andrew Morton --- mm/gup.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 8ea3de60e82d..bce275c7dbb6 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2273,6 +2273,7 @@ static unsigned long collect_longterm_unpinnable_folios( for (folio = pofs_get_folio(pofs, i); folio; folio = pofs_next_folio(folio, pofs, &i)) { + const int pin_refs = folio_has_pincount(folio) ? 1 : GUP_PIN_COUNTING_BIAS; if (folio_is_longterm_pinnable(folio)) continue; @@ -2287,15 +2288,20 @@ static unsigned long collect_longterm_unpinnable_folios( continue; } + /* + * We drain not only to make the folio_isolate_lru() succeed, + * but also to remove any other folio references from LRU + * caches. + */ if (drained == 0 && folio_may_be_lru_cached(folio) && folio_ref_count(folio) != - folio_expected_ref_count(folio) + 1) { + folio_expected_ref_count(folio) + pin_refs) { lru_add_drain(); drained = 1; } if (drained == 1 && folio_may_be_lru_cached(folio) && folio_ref_count(folio) != - folio_expected_ref_count(folio) + 1) { + folio_expected_ref_count(folio) + pin_refs) { lru_add_drain_all(); drained = 2; } -- cgit v1.2.3 From fc6415a384f026f48b414a48588c0970c060679f Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Thu, 6 Aug 2026 20:09:06 +0200 Subject: mm/gup: factor out LRU cache draining for folio into lru_cache_drain_for_folio() KVM with guest_memfd wants to remove any folio references due to LRU caches, as it really must only allow to convert folios from shared to private when there are no unexpected folio references (e.g., from GUP references). So, to drive the refcount down, it needs a way to flush the LRU caches. Let's factor out what we have in lru_cache_drain_for_folio(). Document it, and also mention that concurrent folio (un)mapping might, in theory, miss detecting LRU cache references. Keep obtaining the expected refcount twice to minimize the possibility. For the current and future user that should work, and we don't really have a better alternative: we could detect if the mapcount changed, but it would still be racy and add more complexity with questionable benefit. Maybe there is a chance to avoid the draining entirely in the future, by avoiding extra references from the LRU cache: Hugh thinks there might be a way. But for the time being, this handling is unfortunately required. Make folio_may_be_lru_cached() accept a const pointer so lru_cache_drain_for_folio() can accept a const pointer as well. Link: https://lore.kernel.org/20260806-lru_cache_drain_for_folio-v1-1-c6287d295e99@kernel.org Signed-off-by: David Hildenbrand (Arm) Reviewed-by: Fuad Tabba Cc: Ackerley Tng Cc: Baoquan He Cc: Barry Song Cc: Chris Li Cc: Jason Gunthorpe Cc: John Hubbard Cc: Kairui Song Cc: Kemeng Shi Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Nhat Pham Cc: Peter Xu Cc: Sean Christopherson Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- include/linux/swap.h | 8 ++++++++ mm/folio.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ mm/gup.c | 15 ++------------- mm/internal.h | 2 +- 4 files changed, 57 insertions(+), 14 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 330a420fd6de..b4b1c0a84c8b 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -298,6 +298,14 @@ void folio_add_lru(struct folio *folio); void folio_mark_accessed(struct folio *folio); void lru_add_drain_all(void); +enum lru_cache_drained { + LRU_CACHE_NOT_DRAINED, + LRU_CACHE_DRAINED, + LRU_CACHE_DRAINED_ALL, +}; +void lru_cache_drain_for_folio(const struct folio *folio, + unsigned int extra_refs, enum lru_cache_drained *drained); + /* linux/mm/folio-compat.c */ void mark_page_accessed(struct page *page); diff --git a/mm/folio.c b/mm/folio.c index a9e328c3f21b..59c477120b9a 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -881,6 +881,52 @@ void lru_add_drain_all(void) } #endif /* CONFIG_SMP */ +/** + * lru_cache_drain_for_folio() - drain LRU caches if the caches might hold + * folio references + * @folio: The folio. + * @extra_refs: Extra folio references held by the caller. + * @drained: Drain status for batch folio processing. + * + * Drain LRU caches if the caches might hold folio references. Start + * with a local LRU cache drain, to then drain LRU caches on all CPUs if + * local draining was insufficient. + * + * This function detects LRU cache references by comparing the folio refcount + * with the sum of the expected folio refcount + extra references held by the + * caller. Note that we cannot rely on PG_lru to reliably detect all LRU + * cache references, and there are rare scenarios (concurrent folio (un)mapping) + * where this function might miss detecting LRU cache references. + * + * If @drained is not NULL, the function will avoid re-draining LRU caches + * when processing multiple folios in a row. In that case, the variable + * @drained points at must be initialized to LRU_CACHE_NOT_DRAINED before + * the first invocation by the caller. + */ +void lru_cache_drain_for_folio(const struct folio *folio, + unsigned int extra_refs, enum lru_cache_drained *drained) +{ + if (!folio_may_be_lru_cached(folio)) + return; + + if (!drained || *drained == LRU_CACHE_NOT_DRAINED) { + if (folio_ref_count(folio) == + folio_expected_ref_count(folio) + extra_refs) + return; + lru_add_drain(); + if (drained) + *drained = LRU_CACHE_DRAINED; + } + if (!drained || *drained == LRU_CACHE_DRAINED) { + if (folio_ref_count(folio) == + folio_expected_ref_count(folio) + extra_refs) + return; + lru_add_drain_all(); + if (drained) + *drained = LRU_CACHE_DRAINED_ALL; + } +} + atomic_t lru_disable_count = ATOMIC_INIT(0); /* diff --git a/mm/gup.c b/mm/gup.c index bce275c7dbb6..eb898ea1ee22 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2266,9 +2266,9 @@ static unsigned long collect_longterm_unpinnable_folios( struct list_head *movable_folio_list, struct pages_or_folios *pofs) { + enum lru_cache_drained drained = LRU_CACHE_NOT_DRAINED; unsigned long collected = 0; struct folio *folio; - int drained = 0; long i = 0; for (folio = pofs_get_folio(pofs, i); folio; @@ -2293,18 +2293,7 @@ static unsigned long collect_longterm_unpinnable_folios( * but also to remove any other folio references from LRU * caches. */ - if (drained == 0 && folio_may_be_lru_cached(folio) && - folio_ref_count(folio) != - folio_expected_ref_count(folio) + pin_refs) { - lru_add_drain(); - drained = 1; - } - if (drained == 1 && folio_may_be_lru_cached(folio) && - folio_ref_count(folio) != - folio_expected_ref_count(folio) + pin_refs) { - lru_add_drain_all(); - drained = 2; - } + lru_cache_drain_for_folio(folio, pin_refs, &drained); if (!folio_isolate_lru(folio)) continue; diff --git a/mm/internal.h b/mm/internal.h index 07f60ca0b201..38b1165212c9 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -43,7 +43,7 @@ void workingset_activation(struct folio *folio); /* mm/folio.c */ void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma); -static inline bool folio_may_be_lru_cached(struct folio *folio) +static inline bool folio_may_be_lru_cached(const struct folio *folio) { /* * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting. -- cgit v1.2.3 From 272b0d84b17f72f6396254dbaa6264f2f74a7997 Mon Sep 17 00:00:00 2001 From: Artem Lytkin Date: Sat, 1 Aug 2026 14:49:15 +0300 Subject: mm/vmalloc: make vm_struct.nr_pages an unsigned long vm_struct::nr_pages is an unsigned int, and the file keeps deriving byte counts from it as nr_pages << PAGE_SHIFT. A shift is evaluated in the type of its promoted left operand, so those are 32-bit arithmetic and wrap at 4 GiB of bytes, which is 2^20 pages. Every site depends on a cast being remembered; vmap() has one, two recent commits did not. vread_iter() then computes a size of zero for a 4 GiB VM_ALLOC area and /proc/kcore returns it as zeros while reporting a successful read, which drgn, crash or gdb cannot tell from real memory, and the vrealloc() grow-in-place check declines a request that would have fit. Widen the field so the class of bug goes away instead of one site at a time. Everything feeding or consuming it widens too: vm_area_alloc_pages() and its accumulators, nr_small_pages, new_nr_pages and old_nr_pages, the index range of vm_area_free_pages(), and three page indexes that were plain int. Five casts go. Two prints needed fixing as well, %u in vmalloc_dump_obj() and %d for the unsigned field in vmalloc_info_show(). No bug report behind this, I found it reading the code. The 4 GiB wrap needs only a machine with over 4 GiB of memory. Neither larger threshold is a practical concern: 2^32 pages, where the field itself truncates, is 16 TiB and beyond what hardware can populate, and 2^31, where the plain int indexes break, is 8 TiB and larger than anything in the tree asks for. The int *nr cursor in the mapping path is unchanged and is separate work. Users outside mm/vmalloc.c need no change either. Those handing the count to a narrower parameter cannot drive it near 2^31, and kho_preserve_vmalloc() stores it into a 32-bit ABI field that still receives the same low bits; above 2^32 pages the truncation just moves out of vm_struct into that store. sizeof(struct vm_struct) on x86-64 stays 72 bytes with CONFIG_HAVE_ARCH_HUGE_VMALLOC=n and goes from 72 to 80 with it enabled, both inside the kmalloc-96 bucket it already comes from. Link: https://lore.kernel.org/20260801114915.115224-1-iprintercanon@gmail.com Fixes: 0bca23804632 ("mm/vmalloc: use physical page count in vread_iter() for VM_ALLOC areas") Fixes: d57ac904ffdc ("mm/vmalloc: use physical page count for vrealloc() grow-in-place check") Signed-off-by: Artem Lytkin Suggested-by: Andrew Morton Reviewed-by: Uladzislau Rezki (Sony) Assisted-by: Claude:claude-fable-5 Cc: Matthew Wilcox (Oracle) Cc: Cc: Signed-off-by: Andrew Morton --- include/linux/vmalloc.h | 2 +- mm/vmalloc.c | 58 ++++++++++++++++++++++++------------------------- 2 files changed, 29 insertions(+), 31 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index e4d8d0a9f30f..aed121d729b0 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -62,7 +62,7 @@ struct vm_struct { #ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC unsigned int page_order; #endif - unsigned int nr_pages; + unsigned long nr_pages; phys_addr_t phys_addr; const void *caller; unsigned long requested_size; diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 5c0c2d0d6ae7..72d7f0d81c05 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3361,7 +3361,7 @@ struct vm_struct *remove_vm_area(const void *addr) static inline void set_area_direct_map(const struct vm_struct *area, int (*set_direct_map)(struct page *page)) { - int i; + unsigned long i; /* HUGE_VMALLOC passes small pages to set_direct_map */ for (i = 0; i < area->nr_pages; i++) @@ -3377,7 +3377,7 @@ static void vm_reset_perms(struct vm_struct *area) unsigned long start = ULONG_MAX, end = 0; unsigned int page_order = vm_area_page_order(area); int flush_dmap = 0; - int i; + unsigned long i; /* * Find the start and end range of the direct mappings to make sure that @@ -3450,10 +3450,10 @@ void vfree_atomic(const void *addr) * Caller is responsible for unmapping (vunmap_range) and KASAN * poisoning before calling this. */ -static void vm_area_free_pages(struct vm_struct *vm, unsigned int start_idx, - unsigned int end_idx) +static void vm_area_free_pages(struct vm_struct *vm, unsigned long start_idx, + unsigned long end_idx) { - unsigned int i; + unsigned long i; if (!(vm->flags & VM_MAP_PUT_PAGES)) { for (i = start_idx; i < end_idx; i++) @@ -3665,12 +3665,12 @@ static inline gfp_t vmalloc_gfp_adjust(gfp_t flags, const bool large) return flags; } -static inline unsigned int +static inline unsigned long vm_area_alloc_pages(gfp_t gfp, int nid, - unsigned int order, unsigned int nr_pages, struct page **pages) + unsigned int order, unsigned long nr_pages, struct page **pages) { - unsigned int nr_allocated = 0; - unsigned int nr_remaining = nr_pages; + unsigned long nr_allocated = 0; + unsigned long nr_remaining = nr_pages; unsigned int max_attempt_order = MAX_PAGE_ORDER; struct page *page; int i; @@ -3718,7 +3718,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid, if (!order) { while (nr_allocated < nr_pages) { unsigned int nr, nr_pages_request; - int i; + unsigned long i; /* * A maximum allowed request is hard-coded and is 100 @@ -3726,7 +3726,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid, * long preemption off scenario in the bulk-allocator * so the range is [1:100]. */ - nr_pages_request = min(100U, nr_pages - nr_allocated); + nr_pages_request = min(100UL, nr_pages - nr_allocated); /* memory allocation should consider mempolicy, we can't * wrongly use nearest node when nid == NUMA_NO_NODE, @@ -3872,12 +3872,12 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, unsigned long addr = (unsigned long)area->addr; unsigned long size = get_vm_area_size(area); unsigned long array_size; - unsigned int nr_small_pages = size >> PAGE_SHIFT; + unsigned long nr_small_pages = size >> PAGE_SHIFT; unsigned int page_order; unsigned int flags; int ret; - array_size = (unsigned long)nr_small_pages * sizeof(struct page *); + array_size = nr_small_pages * sizeof(struct page *); /* __GFP_NOFAIL and "noblock" flags are mutually exclusive. */ if (!gfpflags_allow_blocking(gfp_mask)) @@ -4375,7 +4375,7 @@ void *vrealloc_node_align_noprof(const void *p, size_t size, unsigned long align } if (size <= old_size) { - unsigned int new_nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT; + unsigned long new_nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT; /* Zero out "freed" memory, potentially for future realloc. */ if (want_init_on_free() || want_init_on_alloc(flags)) @@ -4404,7 +4404,7 @@ void *vrealloc_node_align_noprof(const void *p, size_t size, unsigned long align !(vm->flags & (VM_FLUSH_RESET_PERMS | VM_USERMAP)) && gfp_has_io_fs(flags)) { unsigned long addr = (unsigned long)kasan_reset_tag(p); - unsigned int old_nr_pages = vm->nr_pages; + unsigned long old_nr_pages = vm->nr_pages; /* * Use the node lock to synchronize with concurrent @@ -4417,16 +4417,13 @@ void *vrealloc_node_align_noprof(const void *p, size_t size, unsigned long align spin_unlock(&vn->busy.lock); /* Notify kmemleak of the reduced allocation size before unmapping. */ - kmemleak_free_part( - (void *)addr + ((unsigned long)new_nr_pages - << PAGE_SHIFT), - (unsigned long)(old_nr_pages - new_nr_pages) - << PAGE_SHIFT); + kmemleak_free_part((void *)addr + + (new_nr_pages << PAGE_SHIFT), + (old_nr_pages - new_nr_pages) + << PAGE_SHIFT); - vunmap_range(addr + ((unsigned long)new_nr_pages - << PAGE_SHIFT), - addr + ((unsigned long)old_nr_pages - << PAGE_SHIFT)); + vunmap_range(addr + (new_nr_pages << PAGE_SHIFT), + addr + (old_nr_pages << PAGE_SHIFT)); vm_area_free_pages(vm, new_nr_pages, old_nr_pages); } @@ -5250,7 +5247,7 @@ bool vmalloc_dump_obj(void *object) struct vmap_area *va; struct vmap_node *vn; unsigned long addr; - unsigned int nr_pages; + unsigned long nr_pages; addr = PAGE_ALIGN((unsigned long) object); vn = addr_to_node(addr); @@ -5270,7 +5267,7 @@ bool vmalloc_dump_obj(void *object) nr_pages = vm->nr_pages; spin_unlock(&vn->busy.lock); - pr_cont(" %u-page vmalloc region starting at %#lx allocated at %pS\n", + pr_cont(" %lu-page vmalloc region starting at %#lx allocated at %pS\n", nr_pages, addr, caller); return true; @@ -5288,16 +5285,17 @@ bool vmalloc_dump_obj(void *object) static void show_numa_info(struct seq_file *m, struct vm_struct *v, unsigned int *counters) { - unsigned int nr; unsigned int step = 1U << vm_area_page_order(v); + unsigned long i; + unsigned int nr; if (!counters) return; memset(counters, 0, nr_node_ids * sizeof(unsigned int)); - for (nr = 0; nr < v->nr_pages; nr += step) - counters[page_to_nid(v->pages[nr])] += step; + for (i = 0; i < v->nr_pages; i += step) + counters[page_to_nid(v->pages[i])] += step; for_each_node_state(nr, N_HIGH_MEMORY) if (counters[nr]) seq_printf(m, " N%u=%u", nr, counters[nr]); @@ -5355,7 +5353,7 @@ static int vmalloc_info_show(struct seq_file *m, void *p) seq_printf(m, " %pS", v->caller); if (v->nr_pages) - seq_printf(m, " pages=%d", v->nr_pages); + seq_printf(m, " pages=%lu", v->nr_pages); if (v->phys_addr) seq_printf(m, " phys=%pa", &v->phys_addr); -- cgit v1.2.3 From c310a8932a3107c9bc8f01d473e9d085f8aa9c98 Mon Sep 17 00:00:00 2001 From: Eric Biggers Date: Mon, 3 Aug 2026 11:04:26 -0700 Subject: mm/swap: reject swapon() on filesystem-level encrypted files ext4 and f2fs don't prevent filesystem-level encrypted files from being set up directly as swap files. In this case, encryption is bypassed. No one should be doing this, vs. the methods of encrypted swap that actually do work (such as swapping to a dm-crypt device, or swapping to a loopback device on top of a filesystem-level encrypted file). Nevertheless, to prevent user error, make swapon() explicitly reject this case. Document this behavior in fscrypt.rst as well. Link: https://lore.kernel.org/20260803180426.3123-1-ebiggers@kernel.org Fixes: 9bd8212f981e ("ext4 crypto: add encryption policy and password salt support") Fixes: f424f664f0e8 ("f2fs crypto: add encryption policy and password salt support") Signed-off-by: Eric Biggers Reviewed-by: Baoquan He Reviewed-by: Muhammad Usama Anjum Reviewed-by: "Darrick J. Wong" Cc: Barry Song Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Signed-off-by: Andrew Morton --- Documentation/filesystems/fscrypt.rst | 4 ++++ mm/swapfile.c | 7 +++++++ 2 files changed, 11 insertions(+) diff --git a/Documentation/filesystems/fscrypt.rst b/Documentation/filesystems/fscrypt.rst index c0dd35f1af12..e4882b73120e 100644 --- a/Documentation/filesystems/fscrypt.rst +++ b/Documentation/filesystems/fscrypt.rst @@ -1238,6 +1238,10 @@ astute users may notice some differences in behavior: - DAX (Direct Access) is not supported on encrypted files. +- Encrypted files cannot be used directly as swap files. To swap to + an encrypted file, set up a loopback device on top of it. + Alternatively, encrypted swap can use a dm-crypt device. + - The maximum length of an encrypted symlink is 2 bytes shorter than the maximum length of an unencrypted symlink. For example, on an EXT4 filesystem with a 4K block size, unencrypted symlinks can be up diff --git a/mm/swapfile.c b/mm/swapfile.c index 4e07d457e261..d7f749ad60c2 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -3668,6 +3668,13 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) error = -EBUSY; goto bad_swap_unlock_inode; } + if (IS_ENCRYPTED(inode)) { + pr_warn_once( + "Filesystem-level encrypted swapfile '%s' is unsupported. Create a loop device over it, or use dm-crypt\n", + name->name); + error = -EINVAL; + goto bad_swap_unlock_inode; + } /* * The swap subsystem needs a major overhaul to support this. -- cgit v1.2.3 From 8282cb36d021835e6574c393df6a4aed1bf6ada3 Mon Sep 17 00:00:00 2001 From: Jianlin Shi Date: Thu, 6 Aug 2026 16:27:34 +0800 Subject: mm/page_alloc: only update lowmem_reserve_ratio on sysctl write lowmem_reserve_ratio_sysctl_handler() ignores the return value of proc_dointvec_minmax() and always calls setup_per_zone_lowmem_reserve(), even for read operations. Fix three issues: 1. Propagate errors from proc_dointvec_minmax() instead of always returning success. For example, writing non-integer garbage to the sysctl now returns an error instead of silently succeeding with unchanged values. 2. Only call setup_per_zone_lowmem_reserve() when the sysctl is actually written, matching the write-only refresh pattern of min_free_kbytes and watermark_scale_factor handlers. 3. On write, parse into a temporary ratio[] array and only copy into sysctl_lowmem_reserve_ratio[] and refresh derived state after the full vector is validated. This avoids leaving the ratio array partially updated while skipping setup when proc_dointvec_minmax() returns an error on a later element (suggested by Andrew Morton). Drop the manual "< 1 -> 0" sanitization loop and set .extra1 = SYSCTL_ZERO on the ctl_table entry so proc_dointvec_minmax() enforces the minimum on write; negative values now return -EINVAL instead of being silently coerced to 0 (suggested by Vlastimil Babka). [akpm@linux-foundation.org: add comment, per hannes] Link: https://lore.kernel.org/anSRGASe5FIrqwlg@cmpxchg.org Link: https://lore.kernel.org/linux-mm/tencent_FFD4F4D728AAE8A8AE0AF277A59854A29A06@qq.com/ Link: https://lore.kernel.org/tencent_A860C873956A52E26AD8D309A308A241BA08@qq.com Signed-off-by: Jianlin Shi Acked-by: Vlastimil Babka (SUSE) Cc: Brendan Jackman Cc: Johannes Weiner Cc: Michal Hocko Cc: Suren Baghdasaryan Cc: Zi Yan Cc: Joel Granados Signed-off-by: Andrew Morton --- mm/page_alloc.c | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 083cbcb5bdde..12fac9084c48 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6853,8 +6853,8 @@ static int sysctl_min_slab_ratio_sysctl_handler(const struct ctl_table *table, i /* * lowmem_reserve_ratio_sysctl_handler - just a wrapper around - * proc_dointvec() so that we can call setup_per_zone_lowmem_reserve() - * whenever sysctl_lowmem_reserve_ratio changes. + * proc_dointvec_minmax() so that we can call + * setup_per_zone_lowmem_reserve() when the sysctl is written. * * The reserve ratio obviously has absolutely no relation with the * minimum watermarks. The lowmem reserve ratio can only make sense @@ -6863,16 +6863,27 @@ static int sysctl_min_slab_ratio_sysctl_handler(const struct ctl_table *table, i static int lowmem_reserve_ratio_sysctl_handler(const struct ctl_table *table, int write, void *buffer, size_t *length, loff_t *ppos) { - int i; + struct ctl_table tmp = *table; + int ratio[ARRAY_SIZE(sysctl_lowmem_reserve_ratio)]; + int rc; - proc_dointvec_minmax(table, write, buffer, length, ppos); + if (!write) + return proc_dointvec_minmax(table, write, buffer, length, ppos); - for (i = 0; i < MAX_NR_ZONES; i++) { - if (sysctl_lowmem_reserve_ratio[i] < 1) - sysctl_lowmem_reserve_ratio[i] = 0; - } + /* + * proc_dointvec_max() works incrementally. Use a buffer and only set + * the values if all of them parse cleanly. + */ + memcpy(ratio, sysctl_lowmem_reserve_ratio, sizeof(ratio)); + tmp.data = ratio; + + rc = proc_dointvec_minmax(&tmp, write, buffer, length, ppos); + if (rc) + return rc; + memcpy(sysctl_lowmem_reserve_ratio, ratio, sizeof(ratio)); setup_per_zone_lowmem_reserve(); + return 0; } @@ -6971,6 +6982,7 @@ static const struct ctl_table page_alloc_sysctl_table[] = { .maxlen = sizeof(sysctl_lowmem_reserve_ratio), .mode = 0644, .proc_handler = lowmem_reserve_ratio_sysctl_handler, + .extra1 = SYSCTL_ZERO, }, #ifdef CONFIG_NUMA { -- cgit v1.2.3 From 32c9625638c2fda23b9301be5184b8abf84c92a7 Mon Sep 17 00:00:00 2001 From: Daniel Palmer Date: Sat, 23 May 2026 22:04:45 +0900 Subject: tmpfs/ramfs: let memfd_create() work on nommu Currently trying to use memfd_create() on nommu returns an error with errno set to EFBIG. The manpage memfd_create() doesn't have EFBIG as a possible error value. Doing some digging this is coming from 0 getting passed as newsize to ramfs_nommu_expand_for_mapping() and that getting into get_order() and there "The result is undefined if the size is 0". Whatever comes out of get_order() is then used in the following logic and that results in the EFBIG that causes the syscall to fail and the errno in userspace. If newsize is 0 there is nothing to do so just return. Roughly tested on m68k nommu by creating a process, creating an memfd, forking another process, mmap()ing the memfd in the child, writing into the mapping, then mmap()ing in the parent and checking that the right data is there. Link: https://lore.kernel.org/20260523130445.1101818-1-daniel@thingy.jp Signed-off-by: Daniel Palmer Acked-by: Lorenzo Stoakes Cc: "Liam R. Howlett" Cc: Al Viro Cc: Christian Brauner Signed-off-by: Andrew Morton --- fs/ramfs/file-nommu.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/fs/ramfs/file-nommu.c b/fs/ramfs/file-nommu.c index 2f79bcb89d2e..fb471bf88ab7 100644 --- a/fs/ramfs/file-nommu.c +++ b/fs/ramfs/file-nommu.c @@ -69,6 +69,9 @@ int ramfs_nommu_expand_for_mapping(struct inode *inode, size_t newsize) gfp_t gfp = mapping_gfp_mask(inode->i_mapping); /* make various checks */ + if (!newsize) + return 0; + order = get_order(newsize); if (unlikely(order > MAX_PAGE_ORDER)) return -EFBIG; -- cgit v1.2.3 From 4e7e499b750f0ba1806a9f60ad091364c3382ad1 Mon Sep 17 00:00:00 2001 From: Pratyush Mallick Date: Mon, 3 Aug 2026 22:17:31 +0000 Subject: selftests/mm: rename local_config.h to local_config.h_gen Patch series "selftests/mm: use pattern matching in .gitignore", v4. The current selftests/mm/.gitignore hardcodes each generated test binary by name, which requires manual updates every time a new test is added. This series switches to a pattern-matching approach (similar to KVM selftests), ignoring everything by default and allowing specific source extensions. To accommodate this without tracking generated headers, local_config.h is renamed to local_config.h_gen. This patch (of 2): Because local_config.h is a generated build artifact, un-ignoring all .h files in .gitignore causes it to incorrectly show up as an untracked file in git status. Rename it to local_config.h_gen so it no longer matches the !*.h inclusion rule, preparing for a subsequent patch that switches .gitignore to a pattern-matching approach. Update Makefile, check_config.sh, and affected test sources (cow.c, gup_longterm.c) accordingly. Link: https://lore.kernel.org/20260803221732.3651981-1-pratmal@google.com Link: https://lore.kernel.org/20260803221732.3651981-2-pratmal@google.com Signed-off-by: Pratyush Mallick Reviewed-by: Lorenzo Stoakes Acked-by: Mike Rapoport (Microsoft) Suggested-by: David Hildenbrand Cc: Jason Gunthorpe Cc: John Hubbard Cc: "Liam R. Howlett" Cc: Michal Hocko Cc: Peter Xu Cc: SeongJae Park Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Yosry Ahmed Signed-off-by: Andrew Morton --- tools/testing/selftests/mm/Makefile | 6 +++--- tools/testing/selftests/mm/check_config.sh | 2 +- tools/testing/selftests/mm/cow.c | 2 +- tools/testing/selftests/mm/gup_longterm.c | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile index 277a141d662e..0f31d850707d 100644 --- a/tools/testing/selftests/mm/Makefile +++ b/tools/testing/selftests/mm/Makefile @@ -5,7 +5,7 @@ # script so kunit knows to run it, and add it to the list below. # If you do not YOUR TESTS WILL NOT RUN IN THE CI. -LOCAL_HDRS += $(selfdir)/mm/local_config.h $(top_srcdir)/mm/gup_test.h +LOCAL_HDRS += $(selfdir)/mm/local_config.h_gen $(top_srcdir)/mm/gup_test.h LOCAL_HDRS += $(selfdir)/mm/mseal_helpers.h include local_config.mk @@ -261,11 +261,11 @@ $(OUTPUT)/migration: LDLIBS += -lnuma $(OUTPUT)/rmap: LDLIBS += -lnuma -local_config.mk local_config.h: check_config.sh +local_config.mk local_config.h_gen: check_config.sh $(call msg,CHK,config,$@) $(Q)CC="$(CC)" CFLAGS="$(CFLAGS)" ./check_config.sh -EXTRA_CLEAN += local_config.mk local_config.h +EXTRA_CLEAN += local_config.mk local_config.h_gen ifeq ($(IOURING_EXTRA_LIBS),) all: warn_missing_liburing diff --git a/tools/testing/selftests/mm/check_config.sh b/tools/testing/selftests/mm/check_config.sh index 32beaefe279e..1c603261e93d 100755 --- a/tools/testing/selftests/mm/check_config.sh +++ b/tools/testing/selftests/mm/check_config.sh @@ -4,7 +4,7 @@ # Probe for libraries and create header files to record the results. Both C # header files and Makefile include fragments are created. -OUTPUT_H_FILE=local_config.h +OUTPUT_H_FILE=local_config.h_gen OUTPUT_MKFILE=local_config.mk tmpname=$(mktemp) diff --git a/tools/testing/selftests/mm/cow.c b/tools/testing/selftests/mm/cow.c index 0c627ea89ff7..7fa2d97ca9b2 100644 --- a/tools/testing/selftests/mm/cow.c +++ b/tools/testing/selftests/mm/cow.c @@ -21,7 +21,7 @@ #include #include -#include "local_config.h" +#include "local_config.h_gen" #ifdef LOCAL_CONFIG_HAVE_LIBURING #include #endif /* LOCAL_CONFIG_HAVE_LIBURING */ diff --git a/tools/testing/selftests/mm/gup_longterm.c b/tools/testing/selftests/mm/gup_longterm.c index c03b4f8910c0..510de93be681 100644 --- a/tools/testing/selftests/mm/gup_longterm.c +++ b/tools/testing/selftests/mm/gup_longterm.c @@ -21,7 +21,7 @@ #include #include -#include "local_config.h" +#include "local_config.h_gen" #ifdef LOCAL_CONFIG_HAVE_LIBURING #include #endif /* LOCAL_CONFIG_HAVE_LIBURING */ -- cgit v1.2.3 From 8bc69b8d209f2c08b8a311e63b910b8a718eee79 Mon Sep 17 00:00:00 2001 From: Warren Xiong Date: Tue, 4 Aug 2026 20:16:58 +0800 Subject: selftests/mm: read memory information without popen read_memory_info() invokes two shell pipelines to obtain MemFree and Hugepagesize from /proc/meminfo. It does not check whether popen() returns NULL before passing the result to fgets(), and it does not call pclose() when fgets() fails. Open /proc/meminfo directly and obtain both values in a single pass. This removes the unchecked NULL path, closes the file on all paths, and avoids dependencies on external commands. The compaction test continues to pass after this change. Link: https://lore.kernel.org/1785845818-3131-1-git-send-email-warren.xiong@ugreen.com Signed-off-by: Warren Xiong Acked-by: David Hildenbrand (Arm) Acked-by: Mike Rapoport (Microsoft) Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- tools/testing/selftests/mm/compaction_test.c | 38 +++++++++++++++------------- 1 file changed, 21 insertions(+), 17 deletions(-) diff --git a/tools/testing/selftests/mm/compaction_test.c b/tools/testing/selftests/mm/compaction_test.c index 5b582588e015..30d4ace7155a 100644 --- a/tools/testing/selftests/mm/compaction_test.c +++ b/tools/testing/selftests/mm/compaction_test.c @@ -29,30 +29,34 @@ struct map_list { int read_memory_info(unsigned long *memfree, unsigned long *hugepagesize) { - char buffer[256] = {0}; - char *cmd = "cat /proc/meminfo | grep -i memfree | grep -o '[0-9]*'"; - FILE *cmdfile = popen(cmd, "r"); + char buffer[256]; + int found = 0; + FILE *file; + int ret = -1; - if (!(fgets(buffer, sizeof(buffer), cmdfile))) { - ksft_print_msg("Failed to read meminfo: %s\n", strerror(errno)); + file = fopen("/proc/meminfo", "r"); + if (!file) { + ksft_print_msg("Failed to open /proc/meminfo: %s\n", + strerror(errno)); return -1; } - pclose(cmdfile); - - *memfree = atoll(buffer); - cmd = "cat /proc/meminfo | grep -i hugepagesize | grep -o '[0-9]*'"; - cmdfile = popen(cmd, "r"); - - if (!(fgets(buffer, sizeof(buffer), cmdfile))) { - ksft_print_msg("Failed to read meminfo: %s\n", strerror(errno)); - return -1; + while (fgets(buffer, sizeof(buffer), file) && found != 2) { + if (sscanf(buffer, "MemFree: %lu kB", memfree) == 1 || + sscanf(buffer, "Hugepagesize: %lu kB", hugepagesize) == 1) + found++; } - pclose(cmdfile); - *hugepagesize = atoll(buffer); + if (ferror(file)) + ksft_print_msg("Failed to read /proc/meminfo: %s\n", + strerror(errno)); + else if (found != 2) + ksft_print_msg("Failed to parse /proc/meminfo\n"); + else + ret = 0; - return 0; + fclose(file); + return ret; } int prereq(void) -- cgit v1.2.3 From 2bee308f3adbd09aa7f6b01fd2271de36538973c Mon Sep 17 00:00:00 2001 From: Pratyush Mallick Date: Mon, 3 Aug 2026 22:17:32 +0000 Subject: selftests/mm: use pattern matching in .gitignore The current .gitignore hardcodes each generated test binary by name, requiring updates every time a new test is added. Switch to the pattern-matching approach similar to KVM:selftests. Ignore everything by default and then allow source extensions (.c, .h, .sh) and tracked non-source files. Note that local_config.h was renamed to local_config.h_gen in a previous patch so that un-ignoring *.h files does not cause generated build artifacts to become untracked. [akpm@linux-foundation.org: fix botched merge resolution] Link: https://lore.kernel.org/20260803221732.3651981-3-pratmal@google.com Signed-off-by: Pratyush Mallick Reviewed-by: Lorenzo Stoakes Acked-by: Mike Rapoport (Microsoft) Suggested-by: Yosry Ahmed Reviewed-by: SJ Park Acked-by: David Hildenbrand (Arm) Cc: Jason Gunthorpe Cc: John Hubbard Cc: "Liam R. Howlett" Cc: Michal Hocko Cc: Peter Xu Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- tools/testing/selftests/mm/.gitignore | 76 +++++------------------------------ 1 file changed, 9 insertions(+), 67 deletions(-) diff --git a/tools/testing/selftests/mm/.gitignore b/tools/testing/selftests/mm/.gitignore index 9ccd9e1447e6..fcd892ed21e3 100644 --- a/tools/testing/selftests/mm/.gitignore +++ b/tools/testing/selftests/mm/.gitignore @@ -1,68 +1,10 @@ # SPDX-License-Identifier: GPL-2.0-only -cow -hugepage-mmap -hugepage-mremap -hugepage-shm -hugepage-vmemmap -hugetlb-mmap -hugetlb-mremap -hugetlb-shm -hugetlb-vmemmap -hugetlb-madvise -hugetlb-read-hwpoison -hugetlb-soft-offline -khugepaged -map_hugetlb -map_populate -thuge-gen -compaction_test -memory-failure -migration -mlock2-tests -mrelease_test -mremap_dontunmap -mremap_test -on-fault-limit -transhuge-stress -pagemap_ioctl -pfnmap -process_madv -*.tmp* -protection_keys -protection_keys_32 -protection_keys_64 -madv_populate -uffd-stress -uffd-unit-tests -uffd-wp-mremap -mlock-intersect-test -mlock-random-test -virtual_address_range -gup_test -va_128TBswitch -map_fixed_noreplace -write_to_hugetlbfs -hmm-tests -memfd_secret -soft-dirty -split_huge_page_test -ksm_tests -local_config.h -local_config.mk -ksm_functional_tests -mdwe_test -gup_longterm -mkdirty -va_high_addr_switch -hugetlb_fault_after_madv -hugetlb_madv_vs_map -mseal_test -droppable -hugetlb_dio -pkey_sighandler_tests_32 -pkey_sighandler_tests_64 -guard-regions -merge -prctl_thp_disable -rmap -folio_split_race_test +* +!/**/ +!*.c +!*.h +!*.sh +!.gitignore +!Makefile +!config +!settings -- cgit v1.2.3 From 3c37cac718fa407b6656d925c4437ed5410fcdda Mon Sep 17 00:00:00 2001 From: Longlong Xia Date: Wed, 5 Aug 2026 21:27:34 +0800 Subject: mm/ksm: avoid missing ksmd wakeups in ksm_enter __ksm_enter() decides whether ksmd needs a wakeup by checking if the mm slot list is empty before inserting the new slot. The empty check is currently outside ksm_mmlist_lock. Another CPU can remove the last slot and let ksmd go back to sleep after the unlocked check, while this CPU inserts a new slot and skips the wakeup based on the stale result. Take ksm_mmlist_lock before checking the list so the empty-to-nonempty transition and the insertion are observed as one critical section. Link: https://lore.kernel.org/20260805132736.1063408-1-xialonglong2025@163.com Fixes: 6e15838425ac ("ksm: keep quiet while list empty") Signed-off-by: Longlong Xia Acked-by: David Hildenbrand (Arm) Reviewed-by: Andrew Morton Cc: Chengming Zhou Cc: Izik Eidus Cc: xu xin Signed-off-by: Andrew Morton --- mm/ksm.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/mm/ksm.c b/mm/ksm.c index b5854dc14a2e..14dd6a6e8e6d 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -3060,10 +3060,9 @@ int __ksm_enter(struct mm_struct *mm) slot = &mm_slot->slot; + spin_lock(&ksm_mmlist_lock); /* Check ksm_run too? Would need tighter locking */ needs_wakeup = list_empty(&ksm_mm_head.slot.mm_node); - - spin_lock(&ksm_mmlist_lock); mm_slot_insert(mm_slots_hash, mm, slot); /* * When KSM_RUN_MERGE (or KSM_RUN_STOP), -- cgit v1.2.3 From a44ab4bd1ec0432d686c25f9c160379ffa1e694c Mon Sep 17 00:00:00 2001 From: Hongfu Li Date: Wed, 5 Aug 2026 18:59:07 +0800 Subject: ksm: update comments and docs to reference folio->mapping The KSM code already stores and checks the stable node key via folio->mapping, but the comment in ksm_get_folio() and the reverse mapping documentation in ksm.rst still refer to page->mapping. This is a pure wording update to match the folio-based implementation. No functional change is intended. Link: https://lore.kernel.org/20260805105927.41987-1-hongfu.li@linux.dev Signed-off-by: Hongfu Li Acked-by: David Hildenbrand (Arm) Reviewed-by: Xu Xin Reviewed-by: Dongliang Mu Cc: Alex Shi Cc: Chengming Zhou Cc: David Hildenbrand Cc: Jonathan Corbet Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Yanteng Si Signed-off-by: Andrew Morton --- Documentation/mm/ksm.rst | 4 ++-- Documentation/translations/zh_CN/mm/ksm.rst | 4 ++-- mm/ksm.c | 7 +++---- 3 files changed, 7 insertions(+), 8 deletions(-) diff --git a/Documentation/mm/ksm.rst b/Documentation/mm/ksm.rst index 2806e3e4a10e..2b4f72f1f953 100644 --- a/Documentation/mm/ksm.rst +++ b/Documentation/mm/ksm.rst @@ -24,13 +24,13 @@ tree. If a KSM page is shared between less than ``max_page_sharing`` VMAs, the node of the stable tree that represents such KSM page points to a -list of struct ksm_rmap_item and the ``page->mapping`` of the +list of struct ksm_rmap_item and the ``folio->mapping`` of the KSM page points to the stable tree node. When the sharing passes this threshold, KSM adds a second dimension to the stable tree. The tree node becomes a "chain" that links one or more "dups". Each "dup" keeps reverse mapping information for a KSM -page with ``page->mapping`` pointing to that "dup". +page with ``folio->mapping`` pointing to that "dup". Every "chain" and all "dups" linked into a "chain" enforce the invariant that they represent the same write protected memory content, diff --git a/Documentation/translations/zh_CN/mm/ksm.rst b/Documentation/translations/zh_CN/mm/ksm.rst index f0f458753d0c..822c7a289671 100644 --- a/Documentation/translations/zh_CN/mm/ksm.rst +++ b/Documentation/translations/zh_CN/mm/ksm.rst @@ -31,10 +31,10 @@ KSM维护着稳定树中的KSM页的逆映射信息。 当KSM页面的共享数小于 ``max_page_sharing`` 的虚拟内存区域(VMAs)时，则代表了 KSM页的稳定树其中的节点指向了一个ksm_rmap_item结构体类型的列表。同时，这个KSM页 -的 ``page->mapping`` 指向了该稳定树节点。 +的 ``folio->mapping`` 指向了该稳定树节点。 如果共享数超过了阈值，KSM将给稳定树添加第二个维度。稳定树就变成链接一个或多 -个稳定树"副本"的"链"。每个副本都保留KSM页的逆映射信息，其中 ``page->mapping`` +个稳定树"副本"的"链"。每个副本都保留KSM页的逆映射信息，其中 ``folio->mapping`` 指向该"副本"。 每个链以及链接到该链中的所有"副本"强制不变的是，它们代表了相同的写保护内存 diff --git a/mm/ksm.c b/mm/ksm.c index 14dd6a6e8e6d..49d48d1e0998 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -959,10 +959,9 @@ enum ksm_get_folio_flags { * seconds or even minutes: much too unresponsive. So instead we use a * "keyhole reference": access to the ksm page from the stable node peeps * out through its keyhole to see if that page still holds the right key, - * pointing back to this stable node. This relies on freeing a PageAnon - * page to reset its page->mapping to NULL, and relies on no other use of - * a page to put something that might look like our key in page->mapping. - * is on its way to being freed; but it is an anomaly to bear in mind. + * pointing back to this stable node. This relies on freeing an anon + * folio to reset its mapping to NULL, and relies on no other use of a + * folio to put something that might look like our key in its mapping. */ static struct folio *ksm_get_folio(struct ksm_stable_node *stable_node, enum ksm_get_folio_flags flags) -- cgit v1.2.3 From b9183788a2def7b26785eccc8c23dba2bbf9e5b1 Mon Sep 17 00:00:00 2001 From: "Uladzislau Rezki (Sony)" Date: Sun, 2 Aug 2026 12:46:27 +0200 Subject: mm/vmalloc: do not warn on -ENOMEM from va_alloc() Since vmalloc() accepts non-blocking GFP flags, allocation requests may fail when callers pass restrictive GFP masks. va_clip() may return -ENOMEM when its GFP_NOWAIT fallback allocation fails during NE_FIT_TYPE splitting. This is an expected failure, so va_alloc() should return the error without triggering a kernel splat. Link: https://lore.kernel.org/20260802104627.63892-1-urezki@gmail.com Signed-off-by: Uladzislau Rezki (Sony) Reported-by: syzbot+61c997e6be1d9bb300ba@syzkaller.appspotmail.com Closes: https://lore.kernel.org/6a6d3cbd.6ce73036.24301b.000e.GAE@google.com Reviewed-by: Anshuman Khandual Reviewed-by: Baoquan He Signed-off-by: Andrew Morton --- mm/vmalloc.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 72d7f0d81c05..bea9f76ed7e7 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1840,8 +1840,10 @@ va_alloc(struct vmap_area *va, /* Update the free vmap_area. */ ret = va_clip(root, head, va, nva_start_addr, size); - if (WARN_ON_ONCE(ret)) + if (ret) { + WARN_ON_ONCE(ret != -ENOMEM); return ret; + } return nva_start_addr; } @@ -1914,12 +1916,9 @@ preload_this_cpu_lock(spinlock_t *lock, gfp_t gfp_mask, int node) /* * Preload this CPU with one extra vmap_area object. It is used - * when fit type of free area is NE_FIT_TYPE. It guarantees that - * a CPU that does an allocation is preloaded. - * - * We do it in non-atomic context, thus it allows us to use more - * permissive allocation masks to be more stable under low memory - * condition and high memory pressure. + * when fit type of free area is NE_FIT_TYPE. It is best effort + * pre-loading. If it fails va_clip() may return -ENOMEM from its + * GFP_NOWAIT fallback. */ if (!this_cpu_read(ne_fit_preload_node)) va = kmem_cache_alloc_node(vmap_area_cachep, gfp_mask, node); -- cgit v1.2.3 From 6fd3e592c09dd19d5ba47e45dc5f38837629d1ba Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 4 Aug 2026 11:08:59 +0100 Subject: mm: add some missing includes to mm-local headers There are a number of internal headers local to mm/ which reference functions and data types without including the relevant headers. mm/vma.h is a special case that intentionally does not include additional headers, but the others are not. This breaks tooling like clangd (which is where I noticed this), though the build is OK due to the C files including the headers happening to include required dependencies. It's better to be explicit about dependencies anyway, so add the missing includes and fix clangd as a bonus. Link: https://lore.kernel.org/20260804-fix-some-local-headers-v1-1-a7beb173c116@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: Vlastimil Babka (SUSE) Acked-by: David Hildenbrand (Arm) Acked-by: Zi Yan Reviewed-by: Barry Song Reviewed-by: Baoquan He Cc: Chris Li Cc: Johannes Weiner Cc: Kairui Song Cc: Kemeng Shi Cc: Liam R. Howlett Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Nhat Pham Cc: Oscar Salvador Cc: Roman Gushchin Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: "Uladzislau Rezki (Sony)" Signed-off-by: Andrew Morton --- mm/cma.h | 1 + mm/hugetlb_cma.h | 2 ++ mm/memcontrol-v1.h | 1 + mm/pgalloc-track.h | 3 +++ mm/shuffle.h | 2 ++ mm/swap.h | 1 + mm/vmalloc.h | 2 ++ 7 files changed, 12 insertions(+) diff --git a/mm/cma.h b/mm/cma.h index c70180c36559..ab6d39898ea5 100644 --- a/mm/cma.h +++ b/mm/cma.h @@ -2,6 +2,7 @@ #ifndef __MM_CMA_H__ #define __MM_CMA_H__ +#include #include #include diff --git a/mm/hugetlb_cma.h b/mm/hugetlb_cma.h index 3aa483573d17..730b2b4965b6 100644 --- a/mm/hugetlb_cma.h +++ b/mm/hugetlb_cma.h @@ -2,6 +2,8 @@ #ifndef _LINUX_HUGETLB_CMA_H #define _LINUX_HUGETLB_CMA_H +#include + #ifdef CONFIG_CMA void hugetlb_cma_free_frozen_folio(struct folio *folio); struct folio *hugetlb_cma_alloc_frozen_folio(int order, gfp_t gfp_mask, diff --git a/mm/memcontrol-v1.h b/mm/memcontrol-v1.h index 0f703f239c80..1e394269c613 100644 --- a/mm/memcontrol-v1.h +++ b/mm/memcontrol-v1.h @@ -4,6 +4,7 @@ #define __MM_MEMCONTROL_V1_H #include +#include /* Cgroup v1 and v2 common declarations */ diff --git a/mm/pgalloc-track.h b/mm/pgalloc-track.h index e9e879de8649..1a6de1358a21 100644 --- a/mm/pgalloc-track.h +++ b/mm/pgalloc-track.h @@ -2,6 +2,9 @@ #ifndef _LINUX_PGALLOC_TRACK_H #define _LINUX_PGALLOC_TRACK_H +#include +#include + #if defined(CONFIG_MMU) static inline p4d_t *p4d_alloc_track(struct mm_struct *mm, pgd_t *pgd, unsigned long address, diff --git a/mm/shuffle.h b/mm/shuffle.h index 61bbcddeeee6..11bec7521ab8 100644 --- a/mm/shuffle.h +++ b/mm/shuffle.h @@ -2,7 +2,9 @@ // Copyright(c) 2018 Intel Corporation. All rights reserved. #ifndef _MM_SHUFFLE_H #define _MM_SHUFFLE_H + #include +#include #define SHUFFLE_ORDER MAX_PAGE_ORDER diff --git a/mm/swap.h b/mm/swap.h index 2ccf8cf7f6c1..4e4c291bbfde 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -5,6 +5,7 @@ #include /* for atomic_long_t */ #include /* for PAGE_SHIFT */ #include /* for mem_cgroup_swappiness() */ +#include /* for MAX_SWAPFILES_SHIFT, struct swap_info_struct */ struct mempolicy; struct swap_iocb; diff --git a/mm/vmalloc.h b/mm/vmalloc.h index dcfe30eaa80c..8866ddcff668 100644 --- a/mm/vmalloc.h +++ b/mm/vmalloc.h @@ -5,6 +5,8 @@ #ifndef __MM_VMALLOC_H #define __MM_VMALLOC_H +#include + #ifdef CONFIG_MMU void __init vmalloc_init(void); int __must_check vmap_pages_range_noflush(unsigned long addr, unsigned long end, -- cgit v1.2.3 From 34568000f3c9aa3a14073f20857d79983345a7df Mon Sep 17 00:00:00 2001 From: Sang-Heon Jeon Date: Wed, 5 Aug 2026 00:11:41 +0900 Subject: mm/Kconfig: make FLATMEM depend on !NUMA Patch series "mm/page_ext: remove pgdat_page_ext_init()", v2. pgdat_page_ext_init() has no effect on FLATMEM. The pgdat is always the zero-initialized contig_page_data, because no architecture supports FLATMEM + NUMA. That constraint is only implicit in the arch Kconfig files. So patch 1 makes it explicit in mm/Kconfig, and patch 2 removes pgdat_page_ext_init(). No functional change. This patch (of 2): FLATMEM + NUMA is not supported by any architecture and fails to build. The constraint is only implicit in the arch Kconfig files. So make it explicit in mm/Kconfig. No functional change. Link: https://lore.kernel.org/20260804151145.3419768-1-ekffu200098@gmail.com Link: https://lore.kernel.org/20260804151145.3419768-2-ekffu200098@gmail.com Signed-off-by: Sang-Heon Jeon Suggested-by: Zi Yan Acked-by: Zi Yan Acked-by: David Hildenbrand (Arm) Acked-by: Mike Rapoport (Microsoft) Cc: Johannes Weiner Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- mm/Kconfig | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index 060190e12bce..331daf7fcfab 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -390,6 +390,7 @@ choice config FLATMEM_MANUAL bool "Flat Memory" + depends on !NUMA depends on !ARCH_SPARSEMEM_ENABLE || ARCH_FLATMEM_ENABLE help This option is best suited for non-NUMA systems with @@ -424,6 +425,7 @@ config SPARSEMEM config FLATMEM def_bool y + depends on !NUMA depends on !SPARSEMEM || FLATMEM_MANUAL # -- cgit v1.2.3 From 0ddb8bb85b98ff59f4643b7e4500e45f650dddcf Mon Sep 17 00:00:00 2001 From: Sang-Heon Jeon Date: Wed, 5 Aug 2026 00:11:42 +0900 Subject: mm/page_ext: remove pgdat_page_ext_init() pgdat_page_ext_init() sets pgdat->node_page_ext to NULL only on FLATMEM. FLATMEM depends on !NUMA, so the pgdat is always the zero-initialized contig_page_data and the store has no effect. So remove the call site, the unused function and its declaration. No functional change. Link: https://lore.kernel.org/20260804151145.3419768-3-ekffu200098@gmail.com Signed-off-by: Sang-Heon Jeon Acked-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Johannes Weiner Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- include/linux/page_ext.h | 5 ----- mm/mm_init.c | 1 - mm/page_ext.c | 9 --------- 3 files changed, 15 deletions(-) diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h index f23d4b218da0..79c53ec45dfa 100644 --- a/include/linux/page_ext.h +++ b/include/linux/page_ext.h @@ -55,7 +55,6 @@ struct page_ext { extern bool early_page_ext; extern unsigned long page_ext_size; -extern void pgdat_page_ext_init(struct pglist_data *pgdat); static inline bool early_page_ext_enabled(void) { @@ -202,10 +201,6 @@ static inline bool early_page_ext_enabled(void) return false; } -static inline void pgdat_page_ext_init(struct pglist_data *pgdat) -{ -} - static inline void page_ext_init(void) { } diff --git a/mm/mm_init.c b/mm/mm_init.c index 711f821f7b3c..e9c4204b73ad 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1394,7 +1394,6 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat) for (i = 0; i < NR_VMSCAN_THROTTLE; i++) init_waitqueue_head(&pgdat->reclaim_wait[i]); - pgdat_page_ext_init(pgdat); lruvec_init(&pgdat->__lruvec); } diff --git a/mm/page_ext.c b/mm/page_ext.c index e2e92bd27ebd..b679a8c1f7d7 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -164,11 +164,6 @@ void __init page_ext_init_flatmem_late(void) invoke_init_callbacks(); } -void __meminit pgdat_page_ext_init(struct pglist_data *pgdat) -{ - pgdat->node_page_ext = NULL; -} - static struct page_ext *lookup_page_ext(const struct page *page) { unsigned long pfn = page_to_pfn(page); @@ -494,10 +489,6 @@ oom: panic("Out of memory"); } -void __meminit pgdat_page_ext_init(struct pglist_data *pgdat) -{ -} - #endif /** -- cgit v1.2.3 From 45214458d6b50124afef3187f6352adeddf74d6f Mon Sep 17 00:00:00 2001 From: Haoqin Huang Date: Tue, 4 Aug 2026 17:38:37 +0800 Subject: zram: do not release zstd global params from error paths Patch series "zram: fix zstd error paths and add parameter validation", v6, Patch 1 removes zstd_release_params() from both zstd_create() and zstd_setup_params() error paths -- the former is a layering violation in a per-CPU callback, the latter is redundant as zcomp_init() already calls release_params() on setup failure. Patch 2 rejects zero-size dictionaries and prints distinct error messages for sz < 0 (returns the original error code) and sz == 0 ("empty file"). Currently errors are silently swallowed. Patch 3 adds pr_fmt to each backend file so that pr_err() messages are auto-prefixed with the algorithm name. Patch 4 validates dict and level parameters in each backend's .setup_params(), rejecting unsupported combinations and out-of-range levels. Patch 5 resets per-priority params on algorithm change before init. This patch (of 5): zstd_setup_params() creates global cdict and ddict stored in params->drv_data, shared across all per-CPU contexts. The per-CPU zstd_create() error path called zstd_release_params(), which freed those globally-shared objects. This is a layering violation: a per-CPU callback should only clean up its own context, not release resources owned by the compression lifecycle. zstd_setup_params() called zstd_release_params() on its own error path as well, but zcomp_init() already calls release_params() when setup fails, so this is redundant. Remove zstd_release_params() from both error paths. Link: https://lore.kernel.org/20260804093841.67920-1-haoqinhuang7@gmail.com Link: https://lore.kernel.org/20260804093841.67920-2-haoqinhuang7@gmail.com Signed-off-by: Haoqin Huang Signed-off-by: Rongwei Wang Reviewed-by: Sergey Senozhatsky Tested-by: Sergey Senozhatsky Cc: David Sterba Cc: Jens Axboe Cc: Minchan Kim Cc: Nick Terrell Signed-off-by: Andrew Morton --- drivers/block/zram/backend_zstd.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend_zstd.c index d00b548056dc..5fabc3e7e975 100644 --- a/drivers/block/zram/backend_zstd.c +++ b/drivers/block/zram/backend_zstd.c @@ -85,7 +85,6 @@ static int zstd_setup_params(struct zcomp_params *params) return 0; error: - zstd_release_params(params); return -EINVAL; } @@ -161,7 +160,6 @@ static int zstd_create(struct zcomp_params *params, struct zcomp_ctx *ctx) return 0; error: - zstd_release_params(params); zstd_destroy(ctx); return -EINVAL; } -- cgit v1.2.3 From 6dc404d433adf09045565054aecf85714db95b46 Mon Sep 17 00:00:00 2001 From: Haoqin Huang Date: Tue, 4 Aug 2026 17:38:38 +0800 Subject: zram: reject zero-size dictionary kernel_read_file_from_path() already rejects empty files (i_size <= 0) and returns -EINVAL, but the current implementation only checks for sz < 0 without logging any information. Use sz == 0 to reject the zero-size case and print distinct error messages for each failure type. Link: https://lore.kernel.org/20260804093841.67920-3-haoqinhuang7@gmail.com Signed-off-by: Haoqin Huang Signed-off-by: Rongwei Wang Reviewed-by: Sergey Senozhatsky Tested-by: Sergey Senozhatsky Cc: David Sterba Cc: Jens Axboe Cc: Minchan Kim Cc: Nick Terrell Signed-off-by: Andrew Morton --- drivers/block/zram/zram_drv.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index cfa98846ac48..f73e30b61067 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1700,8 +1700,16 @@ static int comp_params_store(struct zram *zram, u32 prio, s32 level, INT_MAX, NULL, READING_POLICY); - if (sz < 0) + if (sz < 0) { + pr_err("failed to load dictionary %s (err=%zd)\n", + dict_path, sz); + return sz; + } + if (sz == 0) { + pr_err("failed to load dictionary %s (empty file)\n", + dict_path); return -EINVAL; + } } zram->params[prio].dict_sz = sz; -- cgit v1.2.3 From 70922d5ef84a5863ac80d4b13f574cb9e461a716 Mon Sep 17 00:00:00 2001 From: Haoqin Huang Date: Tue, 4 Aug 2026 17:38:39 +0800 Subject: zram: add pr_fmt to backend files Add pr_fmt to each backend so that pr_err() messages are auto-prefixed with the algorithm name. While at it, tweak the deflate winbits pr_err to avoid a duplicated "deflate" prefix. Link: https://lore.kernel.org/20260804093841.67920-4-haoqinhuang7@gmail.com Signed-off-by: Haoqin Huang Signed-off-by: Rongwei Wang Reviewed-by: Sergey Senozhatsky Tested-by: Sergey Senozhatsky Cc: David Sterba Cc: Jens Axboe Cc: Minchan Kim Cc: Nick Terrell Signed-off-by: Andrew Morton --- drivers/block/zram/backend_842.c | 2 ++ drivers/block/zram/backend_deflate.c | 4 +++- drivers/block/zram/backend_lz4.c | 4 ++++ drivers/block/zram/backend_lz4hc.c | 4 ++++ drivers/block/zram/backend_lzo.c | 2 ++ drivers/block/zram/backend_lzorle.c | 2 ++ drivers/block/zram/backend_zstd.c | 2 ++ 7 files changed, 19 insertions(+), 1 deletion(-) diff --git a/drivers/block/zram/backend_842.c b/drivers/block/zram/backend_842.c index 10d9d5c60f53..d9b8a6bba2cb 100644 --- a/drivers/block/zram/backend_842.c +++ b/drivers/block/zram/backend_842.c @@ -1,5 +1,7 @@ // SPDX-License-Identifier: GPL-2.0-or-later +#define pr_fmt(fmt) "842: " fmt + #include #include #include diff --git a/drivers/block/zram/backend_deflate.c b/drivers/block/zram/backend_deflate.c index b3f7d08b49d9..ee26e6c9282f 100644 --- a/drivers/block/zram/backend_deflate.c +++ b/drivers/block/zram/backend_deflate.c @@ -1,5 +1,7 @@ // SPDX-License-Identifier: GPL-2.0-or-later +#define pr_fmt(fmt) "deflate: " fmt + #include #include #include @@ -30,7 +32,7 @@ static int deflate_setup_params(struct zcomp_params *params) s32 wb = params->deflate.winbits; if ((wb < -15 || wb > -9) && (wb < 9 || wb > 15)) { - pr_err("invalid deflate winbits: %d\n", wb); + pr_err("invalid winbits %d\n", wb); return -EINVAL; } } diff --git a/drivers/block/zram/backend_lz4.c b/drivers/block/zram/backend_lz4.c index c449d511ba86..6d58956ed5b2 100644 --- a/drivers/block/zram/backend_lz4.c +++ b/drivers/block/zram/backend_lz4.c @@ -1,3 +1,7 @@ +// SPDX-License-Identifier: GPL-2.0-or-later + +#define pr_fmt(fmt) "lz4: " fmt + #include #include #include diff --git a/drivers/block/zram/backend_lz4hc.c b/drivers/block/zram/backend_lz4hc.c index f6a336acfe20..c0c3715087c8 100644 --- a/drivers/block/zram/backend_lz4hc.c +++ b/drivers/block/zram/backend_lz4hc.c @@ -1,3 +1,7 @@ +// SPDX-License-Identifier: GPL-2.0-or-later + +#define pr_fmt(fmt) "lz4hc: " fmt + #include #include #include diff --git a/drivers/block/zram/backend_lzo.c b/drivers/block/zram/backend_lzo.c index 4c906beaae6b..84330dea6af5 100644 --- a/drivers/block/zram/backend_lzo.c +++ b/drivers/block/zram/backend_lzo.c @@ -1,5 +1,7 @@ // SPDX-License-Identifier: GPL-2.0-or-later +#define pr_fmt(fmt) "lzo: " fmt + #include #include #include diff --git a/drivers/block/zram/backend_lzorle.c b/drivers/block/zram/backend_lzorle.c index 10640c96cbfc..b3b03a008b64 100644 --- a/drivers/block/zram/backend_lzorle.c +++ b/drivers/block/zram/backend_lzorle.c @@ -1,5 +1,7 @@ // SPDX-License-Identifier: GPL-2.0-or-later +#define pr_fmt(fmt) "lzo-rle: " fmt + #include #include #include diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend_zstd.c index 5fabc3e7e975..fb61acdaef67 100644 --- a/drivers/block/zram/backend_zstd.c +++ b/drivers/block/zram/backend_zstd.c @@ -1,5 +1,7 @@ // SPDX-License-Identifier: GPL-2.0-or-later +#define pr_fmt(fmt) "zstd: " fmt + #include #include #include -- cgit v1.2.3 From 7b0f677c7bd539bd5695b14f9a195e257c3b4463 Mon Sep 17 00:00:00 2001 From: Haoqin Huang Date: Tue, 4 Aug 2026 17:38:40 +0800 Subject: zram: validate parameters in each backend's setup_params Dict and level parameters are silently accepted even for backends that do not support them. Validate these parameters in each backend's .setup_params() to reject unsupported combinations and out-of-range levels with a specific error message. Link: https://lore.kernel.org/20260804093841.67920-5-haoqinhuang7@gmail.com Signed-off-by: Haoqin Huang Signed-off-by: Rongwei Wang Reviewed-by: Sergey Senozhatsky Tested-by: Sergey Senozhatsky Cc: David Sterba Cc: Jens Axboe Cc: Minchan Kim Cc: Nick Terrell Signed-off-by: Andrew Morton --- drivers/block/zram/backend_842.c | 8 ++++++++ drivers/block/zram/backend_deflate.c | 13 ++++++++++++- drivers/block/zram/backend_lz4.c | 6 +++++- drivers/block/zram/backend_lz4hc.c | 12 +++++++++++- drivers/block/zram/backend_lzo.c | 8 ++++++++ drivers/block/zram/backend_lzorle.c | 8 ++++++++ drivers/block/zram/backend_zstd.c | 7 ++++++- 7 files changed, 58 insertions(+), 4 deletions(-) diff --git a/drivers/block/zram/backend_842.c b/drivers/block/zram/backend_842.c index d9b8a6bba2cb..3846a04c69d7 100644 --- a/drivers/block/zram/backend_842.c +++ b/drivers/block/zram/backend_842.c @@ -15,6 +15,14 @@ static void release_params_842(struct zcomp_params *params) static int setup_params_842(struct zcomp_params *params) { + if (params->dict_sz) { + pr_err("dictionary is not supported\n"); + return -EOPNOTSUPP; + } + if (params->level != ZCOMP_PARAM_NOT_SET) { + pr_err("compression level is not supported\n"); + return -EOPNOTSUPP; + } return 0; } diff --git a/drivers/block/zram/backend_deflate.c b/drivers/block/zram/backend_deflate.c index ee26e6c9282f..f71b11bcac78 100644 --- a/drivers/block/zram/backend_deflate.c +++ b/drivers/block/zram/backend_deflate.c @@ -24,8 +24,19 @@ static void deflate_release_params(struct zcomp_params *params) static int deflate_setup_params(struct zcomp_params *params) { - if (params->level == ZCOMP_PARAM_NOT_SET) + if (params->dict_sz) { + pr_err("dictionary is not supported\n"); + return -EOPNOTSUPP; + } + + if (params->level == ZCOMP_PARAM_NOT_SET) { params->level = Z_DEFAULT_COMPRESSION; + } else if (params->level < Z_DEFAULT_COMPRESSION || + params->level > Z_BEST_COMPRESSION) { + pr_err("invalid compression level %d\n", params->level); + return -EINVAL; + } + if (params->deflate.winbits == ZCOMP_PARAM_NOT_SET) { params->deflate.winbits = DEFLATE_DEF_WINBITS; } else { diff --git a/drivers/block/zram/backend_lz4.c b/drivers/block/zram/backend_lz4.c index 6d58956ed5b2..1e28104ad964 100644 --- a/drivers/block/zram/backend_lz4.c +++ b/drivers/block/zram/backend_lz4.c @@ -32,8 +32,12 @@ static int lz4_setup_params(struct zcomp_params *params) LZ4_stream_t *dict_stream; int ret; - if (params->level == ZCOMP_PARAM_NOT_SET) + if (params->level == ZCOMP_PARAM_NOT_SET) { params->level = LZ4_ACCELERATION_DEFAULT; + } else if (params->level < LZ4_ACCELERATION_DEFAULT) { + pr_err("invalid compression level %d\n", params->level); + return -EINVAL; + } if (!params->dict || !params->dict_sz) return 0; diff --git a/drivers/block/zram/backend_lz4hc.c b/drivers/block/zram/backend_lz4hc.c index c0c3715087c8..d8aa01bb258f 100644 --- a/drivers/block/zram/backend_lz4hc.c +++ b/drivers/block/zram/backend_lz4hc.c @@ -22,8 +22,18 @@ static void lz4hc_release_params(struct zcomp_params *params) static int lz4hc_setup_params(struct zcomp_params *params) { - if (params->level == ZCOMP_PARAM_NOT_SET) + if (params->level == ZCOMP_PARAM_NOT_SET) { params->level = LZ4HC_DEFAULT_CLEVEL; + } else if (params->level < 1 || params->level > LZ4HC_MAX_CLEVEL) { + /* + * Use < 1 rather than < LZ4HC_MIN_CLEVEL here because + * LZ4HC_compress_generic() only clamps levels below 1 + * (levels 1 and 2 are valid). LZ4HC_MIN_CLEVEL (3) is + * advisory and not enforced by the library. + */ + pr_err("invalid compression level %d\n", params->level); + return -EINVAL; + } return 0; } diff --git a/drivers/block/zram/backend_lzo.c b/drivers/block/zram/backend_lzo.c index 84330dea6af5..d83f92cf757c 100644 --- a/drivers/block/zram/backend_lzo.c +++ b/drivers/block/zram/backend_lzo.c @@ -14,6 +14,14 @@ static void lzo_release_params(struct zcomp_params *params) static int lzo_setup_params(struct zcomp_params *params) { + if (params->dict_sz) { + pr_err("dictionary is not supported\n"); + return -EOPNOTSUPP; + } + if (params->level != ZCOMP_PARAM_NOT_SET) { + pr_err("compression level is not supported\n"); + return -EOPNOTSUPP; + } return 0; } diff --git a/drivers/block/zram/backend_lzorle.c b/drivers/block/zram/backend_lzorle.c index b3b03a008b64..1b120d062c92 100644 --- a/drivers/block/zram/backend_lzorle.c +++ b/drivers/block/zram/backend_lzorle.c @@ -14,6 +14,14 @@ static void lzorle_release_params(struct zcomp_params *params) static int lzorle_setup_params(struct zcomp_params *params) { + if (params->dict_sz) { + pr_err("dictionary is not supported\n"); + return -EOPNOTSUPP; + } + if (params->level != ZCOMP_PARAM_NOT_SET) { + pr_err("compression level is not supported\n"); + return -EOPNOTSUPP; + } return 0; } diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend_zstd.c index fb61acdaef67..08da3810cffd 100644 --- a/drivers/block/zram/backend_zstd.c +++ b/drivers/block/zram/backend_zstd.c @@ -60,8 +60,13 @@ static int zstd_setup_params(struct zcomp_params *params) return -ENOMEM; params->drv_data = zp; - if (params->level == ZCOMP_PARAM_NOT_SET) + if (params->level == ZCOMP_PARAM_NOT_SET) { params->level = zstd_default_clevel(); + } else if (params->level < zstd_min_clevel() || + params->level > zstd_max_clevel()) { + pr_err("invalid compression level %d\n", params->level); + goto error; + } zp->cprm = zstd_get_params(params->level, PAGE_SIZE); -- cgit v1.2.3 From 702c5a799db20e49fe67cdfa27bac65374ad00ab Mon Sep 17 00:00:00 2001 From: Haoqin Huang Date: Tue, 4 Aug 2026 17:38:41 +0800 Subject: zram: reset per-priority params when changing algorithm before init Parameters validated against one algorithm may be invalid for another (e.g. lz4 accepts level=65535 but zstd does not). Although algorithm changes are blocked after disksize is set, they are allowed before device initialization. Reset per-priority params on algorithm change so that stale parameters do not silently carry over. Link: https://lore.kernel.org/20260804093841.67920-6-haoqinhuang7@gmail.com Signed-off-by: Haoqin Huang Signed-off-by: Rongwei Wang Reviewed-by: Sergey Senozhatsky Tested-by: Sergey Senozhatsky Cc: David Sterba Cc: Jens Axboe Cc: Minchan Kim Cc: Nick Terrell Signed-off-by: Andrew Morton --- drivers/block/zram/zram_drv.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index f73e30b61067..56183c827e1b 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1652,6 +1652,17 @@ static void comp_algorithm_set(struct zram *zram, u32 prio, const char *alg) zram->comp_algs[prio] = alg; } +static void comp_params_reset(struct zram *zram, u32 prio) +{ + struct zcomp_params *params = &zram->params[prio]; + + vfree(params->dict); + params->level = ZCOMP_PARAM_NOT_SET; + params->deflate.winbits = ZCOMP_PARAM_NOT_SET; + params->dict_sz = 0; + params->dict = NULL; +} + static int __comp_algorithm_store(struct zram *zram, u32 prio, const char *buf) { const char *alg; @@ -1672,20 +1683,10 @@ static int __comp_algorithm_store(struct zram *zram, u32 prio, const char *buf) } comp_algorithm_set(zram, prio, alg); + comp_params_reset(zram, prio); return 0; } -static void comp_params_reset(struct zram *zram, u32 prio) -{ - struct zcomp_params *params = &zram->params[prio]; - - vfree(params->dict); - params->level = ZCOMP_PARAM_NOT_SET; - params->deflate.winbits = ZCOMP_PARAM_NOT_SET; - params->dict_sz = 0; - params->dict = NULL; -} - static int comp_params_store(struct zram *zram, u32 prio, s32 level, const char *dict_path, struct deflate_params *deflate_params) -- cgit v1.2.3 From 894913e2d35c46ff19a77530907771ae57862b96 Mon Sep 17 00:00:00 2001 From: Longlong Xia Date: Tue, 4 Aug 2026 14:59:18 +0800 Subject: zram: fix out-of-bounds access in writeback_store() Patch series "zram: fix stale scan bounds after reinitialization". Both writeback_store() and read_block_state() derive their table scan bounds from zram->disksize before acquiring dev_lock. If the device is reset and reinitialized with a smaller disksize between that read and lock acquisition, the bound can describe the old table while the scan operates on the new one. This can lead to out-of-bounds slot accesses. Move both bound calculations under dev_lock so each bound remains consistent with the table throughout its scan. Keep the fixes separate because the affected interfaces originate from different commits and can be backported independently. This patch (of 2): writeback_store() calculates the table scan bounds before taking dev_lock. A reset followed by reconfiguration with a smaller disksize can therefore replace zram->table while writeback_store() is waiting for the lock. Once it acquires the lock, it sees an initialized device but scans the new table using the old upper bound, resulting in an out-of-bounds access. Calculate the number of pages while holding dev_lock so the scan bound matches the table protected by the lock. Link: https://lore.kernel.org/20260804065919.3970386-1-xialonglong2025@163.com Link: https://lore.kernel.org/20260804065919.3970386-2-xialonglong2025@163.com Fixes: a939888ec38b ("zram: support idle/huge page writeback") Assisted-by: Codex:gpt-5.6-sol Signed-off-by: Longlong Xia Reviewed-by: Sergey Senozhatsky Cc: Jens Axboe Cc: Minchan Kim Cc: Signed-off-by: Andrew Morton --- drivers/block/zram/zram_drv.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 56183c827e1b..2be5c20e3f14 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1235,8 +1235,8 @@ static ssize_t writeback_store(struct device *dev, const char *buf, size_t len) { struct zram *zram = dev_to_zram(dev); - u64 nr_pages = zram->disksize >> PAGE_SHIFT; - unsigned long lo = 0, hi = nr_pages; + u64 nr_pages; + unsigned long lo = 0, hi; struct zram_pp_ctl *pp_ctl = NULL; struct zram_wb_ctl *wb_ctl = NULL; char *args, *param, *val; @@ -1250,6 +1250,9 @@ static ssize_t writeback_store(struct device *dev, if (!zram->backing_dev) return -ENODEV; + nr_pages = zram->disksize >> PAGE_SHIFT; + hi = nr_pages; + pp_ctl = init_pp_ctl(); if (!pp_ctl) return -ENOMEM; -- cgit v1.2.3 From 391f057f44a51cc9418da5cba78b014324174264 Mon Sep 17 00:00:00 2001 From: Longlong Xia Date: Tue, 4 Aug 2026 14:59:19 +0800 Subject: zram: fix out-of-bounds access in read_block_state() read_block_state() calculates nr_pages before taking dev_lock. If the device is reset and reinitialized with a smaller disksize before lock acquisition, nr_pages still describes the old table. The subsequent loop can then call slot_lock() past the end of the newly allocated table. Read disksize after acquiring dev_lock and checking that the device is initialized. The read lock then keeps the table and its bound stable for the duration of the scan. Link: https://lore.kernel.org/20260804065919.3970386-3-xialonglong2025@163.com Fixes: c0265342bff4 ("zram: introduce zram memory tracking") Assisted-by: Codex:gpt-5.6-sol Signed-off-by: Longlong Xia Reviewed-by: Sergey Senozhatsky Cc: Jens Axboe Cc: Minchan Kim Cc: Signed-off-by: Andrew Morton --- drivers/block/zram/zram_drv.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 2be5c20e3f14..82b78e6e55b2 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1543,7 +1543,7 @@ static ssize_t read_block_state(struct file *file, char __user *buf, char *kbuf; ssize_t index, written = 0; struct zram *zram = file->private_data; - unsigned long nr_pages = zram->disksize >> PAGE_SHIFT; + unsigned long nr_pages; kbuf = kvmalloc(count, GFP_KERNEL); if (!kbuf) @@ -1555,6 +1555,8 @@ static ssize_t read_block_state(struct file *file, char __user *buf, return -EINVAL; } + nr_pages = zram->disksize >> PAGE_SHIFT; + for (index = *ppos; index < nr_pages; index++) { int copied; -- cgit v1.2.3 From bb3e3c5c2d63fa55566a4b0647fffd68172ee17f Mon Sep 17 00:00:00 2001 From: Ye Liu Date: Wed, 5 Aug 2026 17:31:07 +0800 Subject: mm: debug_page_alloc: fix type mismatch for debug_guardpage_minorder The debug_guardpage_minorder local variable is declared as unsigned int, but debug_guardpage_minorder_setup() uses unsigned long and kstrtoul() to parse the value. Use kstrtouint() with unsigned int local variable to match the actual type of _debug_guardpage_minorder. Also fix the format specifier from %lu to %u accordingly. Link: https://lore.kernel.org/20260805093108.2352900-1-ye.liu@linux.dev Signed-off-by: Ye Liu Reviewed-by: Andrew Morton Cc: Johannes Weiner Cc: Michal Hocko Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Zi Yan Signed-off-by: Andrew Morton --- mm/debug_page_alloc.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/debug_page_alloc.c b/mm/debug_page_alloc.c index 6a26eca546c3..41e3d1f1ad96 100644 --- a/mm/debug_page_alloc.c +++ b/mm/debug_page_alloc.c @@ -20,14 +20,14 @@ early_param("debug_pagealloc", early_debug_pagealloc); static int __init debug_guardpage_minorder_setup(char *buf) { - unsigned long res; + unsigned int res; - if (kstrtoul(buf, 10, &res) < 0 || res > MAX_PAGE_ORDER / 2) { + if (kstrtouint(buf, 10, &res) < 0 || res > MAX_PAGE_ORDER / 2) { pr_err("Bad debug_guardpage_minorder value: %s\n", buf); return 0; } _debug_guardpage_minorder = res; - pr_info("Setting debug_guardpage_minorder to %lu\n", res); + pr_info("Setting debug_guardpage_minorder to %u\n", res); return 0; } early_param("debug_guardpage_minorder", debug_guardpage_minorder_setup); -- cgit v1.2.3 From afff109c2f8b35b88ea783d345c1067a311a57d8 Mon Sep 17 00:00:00 2001 From: Abhishek Bapat Date: Wed, 5 Aug 2026 17:29:52 +0000 Subject: alloc_tag: expose boot-time compression configuration Currently, userspace has limited visibility into the exact active runtime state of memory allocation profiling and its page extension compression ('sysctl.vm.mem_profiling={0|1|never}[,compressed]'). While reading the sysctl provides basic on/off status, it is currently impossible for userspace to natively determine whether page-tag compression was successfully enabled without scraping dmesg boot logs. Add a new read-only sysctl representing how compression was configured at boot time. Link: https://lore.kernel.org/c795f8089f82841e8a6e00d7ca286da2b23aeb7b.1785950530.git.abhishekbapat@google.com Signed-off-by: Abhishek Bapat Acked-by: Suren Baghdasaryan Cc: Hao Ge Signed-off-by: Andrew Morton --- Documentation/mm/allocation-profiling.rst | 11 +++++++++++ mm/alloc_tag.c | 6 ++++++ 2 files changed, 17 insertions(+) diff --git a/Documentation/mm/allocation-profiling.rst b/Documentation/mm/allocation-profiling.rst index 5389d241176a..e928aa3e4e1e 100644 --- a/Documentation/mm/allocation-profiling.rst +++ b/Documentation/mm/allocation-profiling.rst @@ -43,6 +43,17 @@ sysctl: warnings produced by allocations made while profiling is disabled and freed when it's enabled. + /proc/sys/vm/mem_profiling_compressed + + 1: Page alloc tag compression is enabled. + + 0: Page alloc tag compression is disabled. + + This reflects a static boot-time configuration of how page allocation tags are + stored (in page flags when compression is enabled and in page_ext when disabled). + Toggling ``mem_profiling`` at runtime does not change the state of + ``mem_profiling_compressed``. + Runtime info: /proc/allocinfo diff --git a/mm/alloc_tag.c b/mm/alloc_tag.c index e93e7fec1f06..b60ee89704cc 100644 --- a/mm/alloc_tag.c +++ b/mm/alloc_tag.c @@ -961,6 +961,12 @@ static const struct ctl_table memory_allocation_profiling_sysctls[] = { .mode = 0644, .proc_handler = proc_mem_profiling_handler, }, + { + .procname = "mem_profiling_compressed", + .data = &mem_profiling_compressed, + .mode = 0444, + .proc_handler = proc_do_static_key, + }, }; static void __init sysctl_init(void) -- cgit v1.2.3 From e73aeb8a411e5327da9c0746b2f93924ab081113 Mon Sep 17 00:00:00 2001 From: Muchun Song Date: Wed, 5 Aug 2026 10:25:36 +0800 Subject: mm/sparse: keep mem_section_usage_size() internal mem_section_usage_size() is only needed by sparsemem implementation code after commit ae751d567baa ("mm/bootmem_info: stop marking mem_section_usage as MIX_SECTION_INFO"), so keeping the declaration in mmzone.h now exposes the helper to all mmzone.h users for no reason. Move the helper to sparse.h so sparse.c and sparse-vmemmap.c can share it through the internal header. While doing so, calculate the allocation size with struct_size_t(), which ties the expression to the pageblock_flags trailing array instead of open-coding the struct header plus bitmap size. Link: https://lore.kernel.org/20260805022536.1206575-1-songmuchun@bytedance.com Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- include/linux/mmzone.h | 1 - mm/sparse.c | 10 ---------- mm/sparse.h | 6 ++++++ 3 files changed, 6 insertions(+), 11 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 158c1fba2393..94f9c3ff5416 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2069,7 +2069,6 @@ static inline struct mem_section *__nr_to_section(unsigned long nr) #endif return &mem_section[root][nr & SECTION_ROOT_MASK]; } -extern size_t mem_section_usage_size(void); /* * We use the lower bits of the mem_map pointer to store a little bit of diff --git a/mm/sparse.c b/mm/sparse.c index 67fa192d4289..7c15406e77f5 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -213,16 +213,6 @@ static void __init memblocks_present(void) memory_present(nid, start, end); } -static unsigned long usemap_size(void) -{ - return BITS_TO_LONGS(SECTION_BLOCKFLAGS_BITS) * sizeof(unsigned long); -} - -size_t mem_section_usage_size(void) -{ - return sizeof(struct mem_section_usage) + usemap_size(); -} - #ifdef CONFIG_SPARSEMEM_VMEMMAP unsigned long __init section_map_size(void) { diff --git a/mm/sparse.h b/mm/sparse.h index 95aa031213f2..3b744667a7e6 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -47,6 +47,12 @@ static inline void __section_mark_present(struct mem_section *ms, ms->section_mem_map |= SECTION_MARKED_PRESENT; } + +static inline size_t mem_section_usage_size(void) +{ + return struct_size_t(struct mem_section_usage, pageblock_flags, + BITS_TO_LONGS(SECTION_BLOCKFLAGS_BITS)); +} #else static inline void sparse_init(void) {} #endif /* CONFIG_SPARSEMEM */ -- cgit v1.2.3 From 8be7c167be5792ce9c9fcb784b4cd086624feed2 Mon Sep 17 00:00:00 2001 From: Ye Liu Date: Wed, 5 Aug 2026 10:15:55 +0800 Subject: mm/show_mem: fix format string inconsistencies and type mismatches Fix five format string issues in show_free_areas() and __show_mem(): 1-2. reserved_highatomic and free_highatomic: %luKB -> %lukB The uppercase "KB" is inconsistent with all other fields in the same output block and with /proc/meminfo convention. 3. local_pcp: %ukB -> %lukB with explicit (unsigned long) cast per_cpu_pages.count is int, so K(count) yields int. Using %u was a signed/unsigned mismatch. Cast to unsigned long and use %lu for consistency with all other K() usages in the file. 4. total pagecache pages: %ld -> %lu global_node_page_state() returns unsigned long. Using %ld is a signedness mismatch caught by gcc -Wformat-signedness. 5. hwpoisoned pages: %lu -> %ld atomic_long_read() returns long (signed). Using %lu is a signedness mismatch caught by gcc -Wformat-signedness. Verified with: make KCFLAGS="-Wformat -Wformat-signedness" mm/show_mem.o Link: https://lore.kernel.org/20260805021556.1908807-1-ye.liu@linux.dev Signed-off-by: Ye Liu Acked-by: Vlastimil Babka (SUSE) Acked-by: Johannes Weiner Cc: Michal Hocko Cc: Suren Baghdasaryan Cc: Zi Yan Signed-off-by: Andrew Morton --- mm/show_mem.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/show_mem.c b/mm/show_mem.c index d1288b4c2b64..b938cbcd774a 100644 --- a/mm/show_mem.c +++ b/mm/show_mem.c @@ -309,8 +309,8 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask, " min:%lukB" " low:%lukB" " high:%lukB" - " reserved_highatomic:%luKB" - " free_highatomic:%luKB" + " reserved_highatomic:%lukB" + " free_highatomic:%lukB" " active_anon:%lukB" " inactive_anon:%lukB" " active_file:%lukB" @@ -323,7 +323,7 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask, " mlocked:%lukB" " bounce:%lukB" " free_pcp:%lukB" - " local_pcp:%ukB" + " local_pcp:%lukB" " free_cma:%lukB" "\n", zone->name, @@ -350,7 +350,7 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask, K(zone_page_state(zone, NR_MLOCK)), 0UL, K(free_pcp), - K(this_cpu_read(zone->per_cpu_pageset->count)), + K((unsigned long)this_cpu_read(zone->per_cpu_pageset->count)), K(zone_page_state(zone, NR_FREE_CMA_PAGES))); printk("lowmem_reserve[]:"); for (i = 0; i < MAX_NR_ZONES; i++) @@ -400,7 +400,7 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask, hugetlb_show_meminfo_node(nid); } - printk("%ld total pagecache pages\n", global_node_page_state(NR_FILE_PAGES)); + printk("%lu total pagecache pages\n", global_node_page_state(NR_FILE_PAGES)); show_swap_cache_info(); } @@ -430,7 +430,7 @@ void __show_mem(unsigned int filter, const nodemask_t *nodemask, printk("%lu pages cma reserved\n", totalcma_pages); #endif #ifdef CONFIG_MEMORY_FAILURE - printk("%lu pages hwpoisoned\n", atomic_long_read(&num_poisoned_pages)); + printk("%ld pages hwpoisoned\n", atomic_long_read(&num_poisoned_pages)); #endif #ifdef CONFIG_MEM_ALLOC_PROFILING static DEFINE_SPINLOCK(mem_alloc_profiling_spinlock); -- cgit v1.2.3 From 184bf187c45ba6c1141aa7fe10bf10f85d5a7634 Mon Sep 17 00:00:00 2001 From: Sergey Senozhatsky Date: Thu, 6 Aug 2026 12:16:32 +0900 Subject: zram: switch to unsigned long indexing zram has always used "unsigned int" for (page) index calculations, which unnecessarily limited max zram disksize. Switch to "unsigned long" and permit much larger zram devices. Link: https://lore.kernel.org/20260806031640.536615-1-senozhatsky@chromium.org Signed-off-by: Sergey Senozhatsky Suggested-by: Andrew Morton Co-developed-by: Longlong Xia Cc: Minchan Kim Signed-off-by: Andrew Morton --- drivers/block/zram/zram_drv.c | 137 ++++++++++++++++++++++++------------------ 1 file changed, 77 insertions(+), 60 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 82b78e6e55b2..d09fdca49cbd 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -56,7 +56,7 @@ static size_t huge_class_size; static const struct block_device_operations zram_devops; -static void slot_free(struct zram *zram, u32 index); +static void slot_free(struct zram *zram, unsigned long index); /* * entry locking rules: @@ -70,7 +70,7 @@ static void slot_free(struct zram *zram, u32 index); * 4) Use TRY lock variant when in atomic context * - must check return value and handle locking failers */ -static __must_check bool slot_trylock(struct zram *zram, u32 index) +static __must_check bool slot_trylock(struct zram *zram, unsigned long index) { unsigned long *lock = &zram->table[index].__lock; @@ -83,7 +83,7 @@ static __must_check bool slot_trylock(struct zram *zram, u32 index) return false; } -static void slot_lock(struct zram *zram, u32 index) +static void slot_lock(struct zram *zram, unsigned long index) { unsigned long *lock = &zram->table[index].__lock; @@ -92,7 +92,7 @@ static void slot_lock(struct zram *zram, u32 index) lock_acquired(&zram->table_lock_map, _RET_IP_); } -static void slot_unlock(struct zram *zram, u32 index) +static void slot_unlock(struct zram *zram, unsigned long index) { unsigned long *lock = &zram->table[index].__lock; @@ -110,55 +110,56 @@ static inline struct zram *dev_to_zram(struct device *dev) return (struct zram *)dev_to_disk(dev)->private_data; } -static unsigned long get_slot_handle(struct zram *zram, u32 index) +static unsigned long get_slot_handle(struct zram *zram, unsigned long index) { return zram->table[index].handle; } -static void set_slot_handle(struct zram *zram, u32 index, unsigned long handle) +static void set_slot_handle(struct zram *zram, unsigned long index, + unsigned long handle) { zram->table[index].handle = handle; } -static bool test_slot_flag(struct zram *zram, u32 index, +static bool test_slot_flag(struct zram *zram, unsigned long index, enum zram_pageflags flag) { return zram->table[index].attr.flags & BIT(flag); } -static void set_slot_flag(struct zram *zram, u32 index, +static void set_slot_flag(struct zram *zram, unsigned long index, enum zram_pageflags flag) { zram->table[index].attr.flags |= BIT(flag); } -static void clear_slot_flag(struct zram *zram, u32 index, +static void clear_slot_flag(struct zram *zram, unsigned long index, enum zram_pageflags flag) { zram->table[index].attr.flags &= ~BIT(flag); } -static size_t get_slot_size(struct zram *zram, u32 index) +static size_t get_slot_size(struct zram *zram, unsigned long index) { return zram->table[index].attr.flags & (BIT(ZRAM_FLAG_SHIFT) - 1); } -static void set_slot_size(struct zram *zram, u32 index, size_t size) +static void set_slot_size(struct zram *zram, unsigned long index, size_t size) { unsigned long flags = zram->table[index].attr.flags >> ZRAM_FLAG_SHIFT; zram->table[index].attr.flags = (flags << ZRAM_FLAG_SHIFT) | size; } -static inline bool slot_allocated(struct zram *zram, u32 index) +static inline bool slot_allocated(struct zram *zram, unsigned long index) { return get_slot_size(zram, index) || test_slot_flag(zram, index, ZRAM_SAME) || test_slot_flag(zram, index, ZRAM_WB); } -static inline void set_slot_comp_priority(struct zram *zram, u32 index, - u32 prio) +static inline void set_slot_comp_priority(struct zram *zram, + unsigned long index, u32 prio) { prio &= ZRAM_COMP_PRIORITY_MASK; /* @@ -170,14 +171,14 @@ static inline void set_slot_comp_priority(struct zram *zram, u32 index, zram->table[index].attr.flags |= (prio << ZRAM_COMP_PRIORITY_BIT1); } -static inline u32 get_slot_comp_priority(struct zram *zram, u32 index) +static inline u32 get_slot_comp_priority(struct zram *zram, unsigned long index) { u32 prio = zram->table[index].attr.flags >> ZRAM_COMP_PRIORITY_BIT1; return prio & ZRAM_COMP_PRIORITY_MASK; } -static void mark_slot_accessed(struct zram *zram, u32 index) +static void mark_slot_accessed(struct zram *zram, unsigned long index) { clear_slot_flag(zram, index, ZRAM_IDLE); clear_slot_flag(zram, index, ZRAM_PP_SLOT); @@ -284,7 +285,7 @@ static void release_pp_ctl(struct zram *zram, struct zram_pp_ctl *ctl) } static bool place_pp_slot(struct zram *zram, struct zram_pp_ctl *ctl, - u32 index) + unsigned long index) { struct zram_pp_slot *pps; u32 bid; @@ -418,7 +419,7 @@ static void mark_idle(struct zram *zram, ktime_t cutoff) { int is_idle = 1; unsigned long nr_pages = zram->disksize >> PAGE_SHIFT; - int index; + unsigned long index; for (index = 0; index < nr_pages; index++) { /* @@ -485,8 +486,9 @@ static ssize_t idle_store(struct device *dev, struct device_attribute *attr, #define INVALID_BDEV_BLOCK (~0UL) static int read_from_zspool_raw(struct zram *zram, struct page *page, - u32 index); -static int read_from_zspool(struct zram *zram, struct page *page, u32 index); + unsigned long index); +static int read_from_zspool(struct zram *zram, struct page *page, + unsigned long index); struct zram_wb_ctl { /* idle list is accessed only by the writeback task, no concurency */ @@ -522,7 +524,7 @@ struct zram_rb_req { /* error status (sync read) */ int error; }; - u32 index; + unsigned long index; }; #define FOUR_K(x) ((x) * (1 << (PAGE_SHIFT - 12))) @@ -910,7 +912,7 @@ static void zram_account_writeback_submit(struct zram *zram) static int zram_writeback_complete(struct zram *zram, struct zram_wb_req *req) { - u32 index = req->pps->index; + unsigned long index = req->pps->index; int err; err = blk_status_to_errno(req->bio.bi_status); @@ -1032,7 +1034,7 @@ static int zram_writeback_slots(struct zram *zram, struct zram_wb_req *req = NULL; struct zram_pp_slot *pps; int ret = 0, err = 0; - u32 index = 0; + unsigned long index = 0; while ((pps = select_pp_slot(ctl))) { if (zram->wb_limit_enable && !zram->bd_wb_limit) { @@ -1198,7 +1200,7 @@ static void scan_slots_for_writeback(struct zram *zram, u32 mode, unsigned long lo, unsigned long hi, struct zram_pp_ctl *ctl) { - u32 index = lo; + unsigned long index = lo; while (index < hi) { bool ok = true; @@ -1235,7 +1237,7 @@ static ssize_t writeback_store(struct device *dev, const char *buf, size_t len) { struct zram *zram = dev_to_zram(dev); - u64 nr_pages; + unsigned long nr_pages; unsigned long lo = 0, hi; struct zram_pp_ctl *pp_ctl = NULL; struct zram_wb_ctl *wb_ctl = NULL; @@ -1336,7 +1338,8 @@ out: return ret; } -static int decompress_bdev_page(struct zram *zram, struct page *page, u32 index) +static int decompress_bdev_page(struct zram *zram, struct page *page, + unsigned long index) { struct zcomp_strm *zstrm; unsigned int size; @@ -1378,7 +1381,7 @@ static void zram_deferred_decompress(struct work_struct *w) struct zram_rb_req *req = container_of(w, struct zram_rb_req, work); struct page *page = bio_first_page_all(req->bio); struct zram *zram = req->zram; - u32 index = req->index; + unsigned long index = req->index; int ret; ret = decompress_bdev_page(zram, page, index); @@ -1429,7 +1432,7 @@ static void zram_async_read_endio(struct bio *bio) } static int read_from_bdev_async(struct zram *zram, struct page *page, - u32 index, unsigned long blk_idx, + unsigned long index, unsigned long blk_idx, struct bio *parent) { struct zram_rb_req *req; @@ -1479,8 +1482,8 @@ static void zram_sync_read(struct work_struct *w) * chained IO with parent IO in same context, it's a deadlock. To avoid that, * use a worker thread context. */ -static int read_from_bdev_sync(struct zram *zram, struct page *page, u32 index, - unsigned long blk_idx) +static int read_from_bdev_sync(struct zram *zram, struct page *page, + unsigned long index, unsigned long blk_idx) { struct zram_rb_req req; @@ -1499,8 +1502,9 @@ static int read_from_bdev_sync(struct zram *zram, struct page *page, u32 index, return decompress_bdev_page(zram, page, index); } -static int read_from_bdev(struct zram *zram, struct page *page, u32 index, - unsigned long blk_idx, struct bio *parent) +static int read_from_bdev(struct zram *zram, struct page *page, + unsigned long index, unsigned long blk_idx, + struct bio *parent) { atomic64_inc(&zram->stats.bd_reads); if (!parent) { @@ -1512,8 +1516,9 @@ static int read_from_bdev(struct zram *zram, struct page *page, u32 index, } #else static inline void reset_bdev(struct zram *zram) {}; -static int read_from_bdev(struct zram *zram, struct page *page, u32 index, - unsigned long blk_idx, struct bio *parent) +static int read_from_bdev(struct zram *zram, struct page *page, + unsigned long index, unsigned long blk_idx, + struct bio *parent) { return -EIO; } @@ -1541,7 +1546,8 @@ static ssize_t read_block_state(struct file *file, char __user *buf, size_t count, loff_t *ppos) { char *kbuf; - ssize_t index, written = 0; + unsigned long index; + ssize_t written = 0; struct zram *zram = file->private_data; unsigned long nr_pages; @@ -1565,7 +1571,7 @@ static ssize_t read_block_state(struct file *file, char __user *buf, goto next; copied = snprintf(kbuf + written, count, - "%12zd %12u.%06d %c%c%c%c%c%c\n", + "%12lu %12u.%06d %c%c%c%c%c%c\n", index, zram->table[index].attr.ac_time, 0, test_slot_flag(zram, index, ZRAM_SAME) ? 's' : '.', test_slot_flag(zram, index, ZRAM_WB) ? 'w' : '.', @@ -1972,8 +1978,8 @@ static ssize_t debug_stat_show(struct device *dev, static void zram_meta_free(struct zram *zram, u64 disksize) { - size_t num_pages = disksize >> PAGE_SHIFT; - size_t index; + unsigned long num_pages = disksize >> PAGE_SHIFT; + unsigned long index; if (!zram->table) return; @@ -1990,7 +1996,7 @@ static void zram_meta_free(struct zram *zram, u64 disksize) static bool zram_meta_alloc(struct zram *zram, u64 disksize) { - size_t num_pages; + unsigned long num_pages; num_pages = disksize >> PAGE_SHIFT; zram->table = vzalloc(array_size(num_pages, sizeof(*zram->table))); @@ -2013,7 +2019,7 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize) return true; } -static void slot_free(struct zram *zram, u32 index) +static void slot_free(struct zram *zram, unsigned long index) { unsigned long handle; @@ -2067,7 +2073,7 @@ out: } static int read_same_filled_page(struct zram *zram, struct page *page, - u32 index) + unsigned long index) { void *mem; @@ -2078,7 +2084,7 @@ static int read_same_filled_page(struct zram *zram, struct page *page, } static int read_incompressible_page(struct zram *zram, struct page *page, - u32 index) + unsigned long index) { unsigned long handle; void *src, *dst; @@ -2093,7 +2099,8 @@ static int read_incompressible_page(struct zram *zram, struct page *page, return 0; } -static int read_compressed_page(struct zram *zram, struct page *page, u32 index) +static int read_compressed_page(struct zram *zram, struct page *page, + unsigned long index) { struct zcomp_strm *zstrm; unsigned long handle; @@ -2118,7 +2125,8 @@ static int read_compressed_page(struct zram *zram, struct page *page, u32 index) } #if defined CONFIG_ZRAM_WRITEBACK -static int read_from_zspool_raw(struct zram *zram, struct page *page, u32 index) +static int read_from_zspool_raw(struct zram *zram, struct page *page, + unsigned long index) { struct zcomp_strm *zstrm; unsigned long handle; @@ -2150,7 +2158,8 @@ static int read_from_zspool_raw(struct zram *zram, struct page *page, u32 index) * Reads (decompresses if needed) a page from zspool (zsmalloc). * Corresponding ZRAM slot should be locked. */ -static int read_from_zspool(struct zram *zram, struct page *page, u32 index) +static int read_from_zspool(struct zram *zram, struct page *page, + unsigned long index) { if (test_slot_flag(zram, index, ZRAM_SAME) || !get_slot_handle(zram, index)) @@ -2162,8 +2171,8 @@ static int read_from_zspool(struct zram *zram, struct page *page, u32 index) return read_incompressible_page(zram, page, index); } -static int zram_read_page(struct zram *zram, struct page *page, u32 index, - struct bio *parent) +static int zram_read_page(struct zram *zram, struct page *page, + unsigned long index, struct bio *parent) { int ret; @@ -2185,7 +2194,7 @@ static int zram_read_page(struct zram *zram, struct page *page, u32 index, /* Should NEVER happen. Return bio error if it does. */ if (WARN_ON(ret < 0)) - pr_err("Decompression failed! err=%d, page=%u\n", ret, index); + pr_err("Decompression failed! err=%d, page=%lu\n", ret, index); return ret; } @@ -2195,7 +2204,7 @@ static int zram_read_page(struct zram *zram, struct page *page, u32 index, * always expects a full page for the output. */ static int zram_bvec_read_partial(struct zram *zram, struct bio_vec *bvec, - u32 index, int offset) + unsigned long index, int offset) { struct page *page = alloc_page(GFP_NOIO); int ret; @@ -2210,7 +2219,7 @@ static int zram_bvec_read_partial(struct zram *zram, struct bio_vec *bvec, } static int zram_bvec_read(struct zram *zram, struct bio_vec *bvec, - u32 index, int offset, struct bio *bio) + unsigned long index, int offset, struct bio *bio) { if (is_partial_io(bvec)) return zram_bvec_read_partial(zram, bvec, index, offset); @@ -2218,7 +2227,7 @@ static int zram_bvec_read(struct zram *zram, struct bio_vec *bvec, } static int write_same_filled_page(struct zram *zram, unsigned long fill, - u32 index) + unsigned long index) { slot_lock(zram, index); slot_free(zram, index); @@ -2233,7 +2242,7 @@ static int write_same_filled_page(struct zram *zram, unsigned long fill, } static int write_incompressible_page(struct zram *zram, struct page *page, - u32 index) + unsigned long index) { unsigned long handle; void *src; @@ -2273,7 +2282,8 @@ static int write_incompressible_page(struct zram *zram, struct page *page, return 0; } -static int zram_write_page(struct zram *zram, struct page *page, u32 index) +static int zram_write_page(struct zram *zram, struct page *page, + unsigned long index) { int ret = 0; unsigned long handle; @@ -2340,7 +2350,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) * This is a partial IO. Read the full page before writing the changes. */ static int zram_bvec_write_partial(struct zram *zram, struct bio_vec *bvec, - u32 index, int offset) + unsigned long index, int offset) { struct page *page = alloc_page(GFP_NOIO); int ret; @@ -2358,7 +2368,7 @@ static int zram_bvec_write_partial(struct zram *zram, struct bio_vec *bvec, } static int zram_bvec_write(struct zram *zram, struct bio_vec *bvec, - u32 index, int offset) + unsigned long index, int offset) { if (is_partial_io(bvec)) return zram_bvec_write_partial(zram, bvec, index, offset); @@ -2426,8 +2436,9 @@ next: * * Corresponding ZRAM slot should be locked. */ -static int recompress_slot(struct zram *zram, u32 index, struct page *page, - u64 *num_recomp_pages, u32 threshold, u32 prio) +static int recompress_slot(struct zram *zram, unsigned long index, + struct page *page, u64 *num_recomp_pages, + u32 threshold, u32 prio) { struct zcomp_strm *zstrm = NULL; unsigned long handle_old; @@ -2679,7 +2690,7 @@ out: static void zram_bio_discard(struct zram *zram, struct bio *bio) { size_t n = bio->bi_iter.bi_size; - u32 index = bio->bi_iter.bi_sector >> SECTORS_PER_PAGE_SHIFT; + unsigned long index = bio->bi_iter.bi_sector >> SECTORS_PER_PAGE_SHIFT; u32 offset = (bio->bi_iter.bi_sector & (SECTORS_PER_PAGE - 1)) << SECTOR_SHIFT; @@ -2720,7 +2731,7 @@ static void zram_bio_read(struct zram *zram, struct bio *bio) struct bvec_iter iter = bio->bi_iter; do { - u32 index = iter.bi_sector >> SECTORS_PER_PAGE_SHIFT; + unsigned long index = iter.bi_sector >> SECTORS_PER_PAGE_SHIFT; u32 offset = (iter.bi_sector & (SECTORS_PER_PAGE - 1)) << SECTOR_SHIFT; struct bio_vec bv = bio_iter_iovec(bio, iter); @@ -2751,7 +2762,7 @@ static void zram_bio_write(struct zram *zram, struct bio *bio) struct bvec_iter iter = bio->bi_iter; do { - u32 index = iter.bi_sector >> SECTORS_PER_PAGE_SHIFT; + unsigned long index = iter.bi_sector >> SECTORS_PER_PAGE_SHIFT; u32 offset = (iter.bi_sector & (SECTORS_PER_PAGE - 1)) << SECTOR_SHIFT; struct bio_vec bv = bio_iter_iovec(bio, iter); @@ -2865,6 +2876,7 @@ static void zram_reset_device(struct zram *zram) static ssize_t disksize_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { + unsigned long num_pages; u64 disksize; struct zcomp *comp; struct zram *zram = dev_to_zram(dev); @@ -2882,6 +2894,11 @@ static ssize_t disksize_store(struct device *dev, struct device_attribute *attr, } disksize = PAGE_ALIGN(disksize); + num_pages = disksize >> PAGE_SHIFT; + /* Slots are addressed by an unsigned long index */ + if (!num_pages || ((u64)num_pages << PAGE_SHIFT) != disksize) + return -EINVAL; + if (!zram_meta_alloc(zram, disksize)) return -ENOMEM; -- cgit v1.2.3 From e4ce743a8f3a8ac1428e220e4d0311f39d64ff87 Mon Sep 17 00:00:00 2001 From: Baolin Wang Date: Thu, 6 Aug 2026 11:34:12 +0800 Subject: selftests: mm: extend the check_huge() to support mTHP check Patch series "add anon mTHP collapse test cases", v3. This patch (of 4): To support checking for various sized mTHPs during mTHP collapse, extend the check_huge() function prototype to accept two new parameters specifying the address range and mTHP size, in preparation for the following patches. No functional changes. Link: https://lore.kernel.org/cover.1785985999.git.baolin.wang@linux.alibaba.com Link: https://lore.kernel.org/e5039cbc70f8de853e6c21048d65803a5fe41042.1785985999.git.baolin.wang@linux.alibaba.com Signed-off-by: Baolin Wang Reviewed-by: Nico Pache (Red Hat) Tested-by: Nico Pache (Red Hat) Acked-by: Zi Yan Acked-by: Kiryl Shutsemau (Meta) Cc: Barry Song Cc: David Hildenbrand Cc: Dev Jain Cc: Lance Yang Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Ryan Roberts Signed-off-by: Andrew Morton --- tools/testing/selftests/mm/folio_split_race_test.c | 2 +- tools/testing/selftests/mm/khugepaged.c | 66 ++++++++++++---------- tools/testing/selftests/mm/pagemap_ioctl.c | 2 +- tools/testing/selftests/mm/prctl_thp_disable.c | 2 +- tools/testing/selftests/mm/soft-dirty.c | 2 +- tools/testing/selftests/mm/split_huge_page_test.c | 14 ++--- tools/testing/selftests/mm/uffd-common.c | 4 +- tools/testing/selftests/mm/vm_util.c | 6 +- tools/testing/selftests/mm/vm_util.h | 6 +- 9 files changed, 56 insertions(+), 48 deletions(-) diff --git a/tools/testing/selftests/mm/folio_split_race_test.c b/tools/testing/selftests/mm/folio_split_race_test.c index 6329e37fff4c..45b84f7b364e 100644 --- a/tools/testing/selftests/mm/folio_split_race_test.c +++ b/tools/testing/selftests/mm/folio_split_race_test.c @@ -182,7 +182,7 @@ static uint64_t run_iteration(void) for (i = 0; i < TOTAL_PAGES; i++) fill_page(mmap_base, i); - if (!check_huge_shmem(mmap_base, NR_PMD_PAGE, pmd_pagesize)) + if (!check_huge_shmem(mmap_base, FILE_SIZE, NR_PMD_PAGE, pmd_pagesize)) ksft_exit_fail_msg("No shmem THP is allocated\n"); if (pthread_barrier_init(&ctl.barrier, NULL, NUM_READER_THREADS + 1) != 0) diff --git a/tools/testing/selftests/mm/khugepaged.c b/tools/testing/selftests/mm/khugepaged.c index 10e8dedcb087..c02d00846a79 100644 --- a/tools/testing/selftests/mm/khugepaged.c +++ b/tools/testing/selftests/mm/khugepaged.c @@ -51,7 +51,7 @@ struct mem_ops { void *(*setup_area)(int nr_hpages); void (*cleanup_area)(void *p, unsigned long size); void (*fault)(void *p, unsigned long start, unsigned long end); - bool (*check_huge)(void *addr, int nr_hpages); + bool (*check_huge)(void *addr, size_t len, int nr_hpages, unsigned long hpage_size); const char *name; }; @@ -276,7 +276,7 @@ static void *alloc_hpage(struct mem_ops *ops) ksft_print_msg("Allocate huge page..."); if (madvise_collapse_retry(p, hpage_pmd_size)) ksft_exit_fail_perror("madvise(MADV_COLLAPSE)"); - if (!ops->check_huge(p, 1)) + if (!ops->check_huge(p, hpage_pmd_size, 1, hpage_pmd_size)) ksft_exit_fail_perror("madvise(MADV_COLLAPSE)"); if (madvise(p, hpage_pmd_size, MADV_HUGEPAGE)) ksft_exit_fail_perror("madvise(MADV_HUGEPAGE)"); @@ -310,9 +310,10 @@ static void anon_fault(void *p, unsigned long start, unsigned long end) fill_memory(p, start, end); } -static bool anon_check_huge(void *addr, int nr_hpages) +static bool anon_check_huge(void *addr, size_t len, int nr_hpages, + unsigned long hpage_size) { - return check_huge_anon(addr, nr_hpages, hpage_pmd_size); + return check_huge_anon(addr, len, nr_hpages, hpage_size); } static void *file_setup_area_common(int nr_hpages, enum file_setup_ops setup) @@ -412,13 +413,14 @@ static void file_fault_write(void *p, unsigned long start, unsigned long end) ksft_exit_fail_perror("madvise(MADV_POPULATE_WRITE)"); } -static bool file_check_huge(void *addr, int nr_hpages) +static bool file_check_huge(void *addr, size_t len, int nr_hpages, + unsigned long hpage_size) { switch (finfo.type) { case VMA_FILE: - return check_huge_file(addr, nr_hpages, hpage_pmd_size); + return check_huge_file(addr, len, nr_hpages, hpage_size); case VMA_SHMEM: - return check_huge_shmem(addr, nr_hpages, hpage_pmd_size); + return check_huge_shmem(addr, len, nr_hpages, hpage_size); default: exit(EXIT_FAILURE); return false; @@ -448,9 +450,10 @@ static void shmem_cleanup_area(void *p, unsigned long size) close(finfo.fd); } -static bool shmem_check_huge(void *addr, int nr_hpages) +static bool shmem_check_huge(void *addr, size_t len, int nr_hpages, + unsigned long hpage_size) { - return check_huge_shmem(addr, nr_hpages, hpage_pmd_size); + return check_huge_shmem(addr, len, nr_hpages, hpage_size); } static struct mem_ops __anon_ops = { @@ -533,7 +536,7 @@ static void __madvise_collapse(const char *msg, char *p, int nr_hpages, ret = madvise_collapse_retry(p, nr_hpages * hpage_pmd_size); if (((bool)ret) == expect) fail("Fail: Bad return value"); - else if (!ops->check_huge(p, expect ? nr_hpages : 0)) + else if (!ops->check_huge(p, nr_hpages * hpage_pmd_size, expect ? nr_hpages : 0, hpage_pmd_size)) fail("Fail: check_huge()"); else success("OK"); @@ -545,7 +548,7 @@ static void madvise_collapse(const char *msg, char *p, int nr_hpages, struct mem_ops *ops, bool expect) { /* Sanity check */ - if (!ops->check_huge(p, 0)) + if (!ops->check_huge(p, nr_hpages * hpage_pmd_size, 0, hpage_pmd_size)) ksft_exit_fail_msg("Unexpected huge page\n"); __madvise_collapse(msg, p, nr_hpages, ops, expect); } @@ -554,11 +557,12 @@ static void madvise_collapse(const char *msg, char *p, int nr_hpages, static bool wait_for_scan(const char *msg, char *p, int nr_hpages, struct mem_ops *ops) { + size_t len = nr_hpages * hpage_pmd_size; int full_scans; int timeout = 6; /* 3 seconds */ /* Sanity check */ - if (!ops->check_huge(p, 0)) + if (!ops->check_huge(p, len, 0, hpage_pmd_size)) ksft_exit_fail_msg("Unexpected huge page\n"); madvise(p, nr_hpages * hpage_pmd_size, MADV_HUGEPAGE); @@ -568,7 +572,7 @@ static bool wait_for_scan(const char *msg, char *p, int nr_hpages, ksft_print_msg("%s...", msg); while (timeout--) { - if (ops->check_huge(p, nr_hpages)) + if (ops->check_huge(p, len, nr_hpages, hpage_pmd_size)) break; if (thp_read_num("khugepaged/full_scans") >= full_scans) break; @@ -582,6 +586,8 @@ static bool wait_for_scan(const char *msg, char *p, int nr_hpages, static void khugepaged_collapse(const char *msg, char *p, int nr_hpages, struct mem_ops *ops, bool expect) { + size_t len = nr_hpages * hpage_pmd_size; + /* * read&write file collapse fails since khugepaged does not flush * the target dirty folios @@ -605,7 +611,7 @@ static void khugepaged_collapse(const char *msg, char *p, int nr_hpages, if (ops != &__anon_ops) ops->fault(p, 0, nr_hpages * hpage_pmd_size); - if (ops->check_huge(p, expect ? nr_hpages : 0)) + if (ops->check_huge(p, len, expect ? nr_hpages : 0, hpage_pmd_size)) success("OK"); else fail("Fail"); @@ -634,7 +640,7 @@ static void alloc_at_fault(void) p = alloc_mapping(1); *p = 1; ksft_print_msg("Allocate huge page on fault..."); - if (check_huge_anon(p, 1, hpage_pmd_size)) + if (check_huge_anon(p, hpage_pmd_size, 1, hpage_pmd_size)) success("OK"); else fail("Fail"); @@ -643,7 +649,7 @@ static void alloc_at_fault(void) madvise(p, page_size, MADV_DONTNEED); ksft_print_msg("Split huge PMD on MADV_DONTNEED..."); - if (check_huge_anon(p, 0, hpage_pmd_size)) + if (check_huge_anon(p, hpage_pmd_size, 0, hpage_pmd_size)) success("OK"); else fail("Fail"); @@ -815,7 +821,7 @@ static void collapse_single_pte_entry_compound(struct collapse_context *c, struc madvise(p, hpage_pmd_size, MADV_NOHUGEPAGE); ksft_print_msg("Split huge page leaving single PTE mapping compound page..."); madvise(p + page_size, hpage_pmd_size - page_size, MADV_DONTNEED); - if (ops->check_huge(p, 0)) + if (ops->check_huge(p, hpage_pmd_size, 0, hpage_pmd_size)) success("OK"); else fail("Fail"); @@ -836,7 +842,7 @@ static void collapse_full_of_compound(struct collapse_context *c, struct mem_ops ksft_print_msg("Split huge page leaving single PTE page table full of compound pages..."); madvise(p, page_size, MADV_NOHUGEPAGE); madvise(p, hpage_pmd_size, MADV_NOHUGEPAGE); - if (ops->check_huge(p, 0)) + if (ops->check_huge(p, hpage_pmd_size, 0, hpage_pmd_size)) success("OK"); else fail("Fail"); @@ -858,7 +864,7 @@ static void collapse_compound_extreme(struct collapse_context *c, struct mem_ops for (i = 0; i < hpage_pmd_nr; i++) { madvise(BASE_ADDR, hpage_pmd_size, MADV_HUGEPAGE); ops->fault(BASE_ADDR, 0, hpage_pmd_size); - if (!ops->check_huge(BASE_ADDR, 1)) + if (!ops->check_huge(BASE_ADDR, hpage_pmd_size, 1, hpage_pmd_size)) ksft_exit_fail_msg("Failed to allocate huge page\n"); madvise(BASE_ADDR, hpage_pmd_size, MADV_NOHUGEPAGE); @@ -881,7 +887,7 @@ static void collapse_compound_extreme(struct collapse_context *c, struct mem_ops ops->cleanup_area(BASE_ADDR, hpage_pmd_size); ops->fault(p, 0, hpage_pmd_size); - if (!ops->check_huge(p, 1)) + if (!ops->check_huge(p, hpage_pmd_size, 1, hpage_pmd_size)) success("OK"); else fail("Fail"); @@ -903,7 +909,7 @@ static void collapse_fork(struct collapse_context *c, struct mem_ops *ops) ksft_print_msg("Allocate small page..."); ops->fault(p, 0, page_size); - if (ops->check_huge(p, 0)) + if (ops->check_huge(p, hpage_pmd_size, 0, hpage_pmd_size)) success("OK"); else fail("Fail"); @@ -911,7 +917,7 @@ static void collapse_fork(struct collapse_context *c, struct mem_ops *ops) ksft_print_msg("Share small page over fork()..."); if (!fork()) { /* Do not touch settings on child exit */ - if (ops->check_huge(p, 0)) + if (ops->check_huge(p, hpage_pmd_size, 0, hpage_pmd_size)) success("OK"); else fail("Fail"); @@ -929,7 +935,7 @@ static void collapse_fork(struct collapse_context *c, struct mem_ops *ops) exit_status = WEXITSTATUS(wstatus); ksft_print_msg("Check if parent still has small page..."); - if (ops->check_huge(p, 0)) + if (ops->check_huge(p, hpage_pmd_size, 0, hpage_pmd_size)) success("OK"); else fail("Fail"); @@ -947,7 +953,7 @@ static void collapse_fork_compound(struct collapse_context *c, struct mem_ops *o ksft_print_msg("Share huge page over fork()..."); if (!fork()) { /* Do not touch settings on child exit */ - if (ops->check_huge(p, 1)) + if (ops->check_huge(p, hpage_pmd_size, 1, hpage_pmd_size)) success("OK"); else fail("Fail"); @@ -955,7 +961,7 @@ static void collapse_fork_compound(struct collapse_context *c, struct mem_ops *o ksft_print_msg("Split huge page PMD in child process..."); madvise(p, page_size, MADV_NOHUGEPAGE); madvise(p, hpage_pmd_size, MADV_NOHUGEPAGE); - if (ops->check_huge(p, 0)) + if (ops->check_huge(p, hpage_pmd_size, 0, hpage_pmd_size)) success("OK"); else fail("Fail"); @@ -976,7 +982,7 @@ static void collapse_fork_compound(struct collapse_context *c, struct mem_ops *o exit_status = WEXITSTATUS(wstatus); ksft_print_msg("Check if parent still has huge page..."); - if (ops->check_huge(p, 1)) + if (ops->check_huge(p, hpage_pmd_size, 1, hpage_pmd_size)) success("OK"); else fail("Fail"); @@ -995,7 +1001,7 @@ static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_ops ksft_print_msg("Share huge page over fork()..."); if (!fork()) { /* Do not touch settings on child exit */ - if (ops->check_huge(p, 1)) + if (ops->check_huge(p, hpage_pmd_size, 1, hpage_pmd_size)) success("OK"); else fail("Fail"); @@ -1003,7 +1009,7 @@ static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_ops ksft_print_msg("Trigger CoW on page %d of %d...", hpage_pmd_nr - max_ptes_shared - 1, hpage_pmd_nr); ops->fault(p, 0, (hpage_pmd_nr - max_ptes_shared - 1) * page_size); - if (ops->check_huge(p, 0)) + if (ops->check_huge(p, hpage_pmd_size, 0, hpage_pmd_size)) success("OK"); else fail("Fail"); @@ -1016,7 +1022,7 @@ static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_ops hpage_pmd_nr - max_ptes_shared, hpage_pmd_nr); ops->fault(p, 0, (hpage_pmd_nr - max_ptes_shared) * page_size); - if (ops->check_huge(p, 0)) + if (ops->check_huge(p, hpage_pmd_size, 0, hpage_pmd_size)) success("OK"); else fail("Fail"); @@ -1034,7 +1040,7 @@ static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_ops exit_status = WEXITSTATUS(wstatus); ksft_print_msg("Check if parent still has huge page..."); - if (ops->check_huge(p, 1)) + if (ops->check_huge(p, hpage_pmd_size, 1, hpage_pmd_size)) success("OK"); else fail("Fail"); diff --git a/tools/testing/selftests/mm/pagemap_ioctl.c b/tools/testing/selftests/mm/pagemap_ioctl.c index 1b2dffcc999b..cfd1987339c1 100644 --- a/tools/testing/selftests/mm/pagemap_ioctl.c +++ b/tools/testing/selftests/mm/pagemap_ioctl.c @@ -1085,7 +1085,7 @@ static void unpopulated_written_test(const char *name, char *mem, long size, memset(mem, 1, size); if (use_thp && (madvise(mem, size, MADV_COLLAPSE) || - !check_huge_anon(mem, size / hpage_size, hpage_size))) { + !check_huge_anon(mem, size, size / hpage_size, hpage_size))) { ksft_test_result_skip("%s could not form a THP\n", name); goto out; } diff --git a/tools/testing/selftests/mm/prctl_thp_disable.c b/tools/testing/selftests/mm/prctl_thp_disable.c index d8d9d1de57b8..82c6e96ea6eb 100644 --- a/tools/testing/selftests/mm/prctl_thp_disable.c +++ b/tools/testing/selftests/mm/prctl_thp_disable.c @@ -67,7 +67,7 @@ static int test_mmap_thp(enum thp_collapse_type madvise_buf, size_t pmdsize) /* HACK: make sure we have a separate VMA that we can check reliably. */ mprotect(mem, pmdsize, PROT_READ); - ret = check_huge_anon(mem, 1, pmdsize); + ret = check_huge_anon(mem, pmdsize, 1, pmdsize); munmap(mmap_mem, mmap_size); return ret; } diff --git a/tools/testing/selftests/mm/soft-dirty.c b/tools/testing/selftests/mm/soft-dirty.c index fb1864a68e1c..e198facf78bb 100644 --- a/tools/testing/selftests/mm/soft-dirty.c +++ b/tools/testing/selftests/mm/soft-dirty.c @@ -103,7 +103,7 @@ static void test_hugepage(int pagemap_fd, int pagesize) for (i = 0; i < hpage_len; i++) map[i] = (char)i; - if (check_huge_anon(map, 1, hpage_len)) { + if (check_huge_anon(map, hpage_len, 1, hpage_len)) { ksft_test_result_pass("Test %s huge page allocation\n", __func__); clear_softdirty(); diff --git a/tools/testing/selftests/mm/split_huge_page_test.c b/tools/testing/selftests/mm/split_huge_page_test.c index 32b991472f74..4cc70873a674 100644 --- a/tools/testing/selftests/mm/split_huge_page_test.c +++ b/tools/testing/selftests/mm/split_huge_page_test.c @@ -296,7 +296,7 @@ static void verify_rss_anon_split_huge_page_all_zeroes(char *one_page, int nr_hp unsigned long rss_anon_before, rss_anon_after; size_t i; - if (!check_huge_anon(one_page, nr_hpages, pmd_pagesize)) + if (!check_huge_anon(one_page, nr_hpages * pmd_pagesize, nr_hpages, pmd_pagesize)) ksft_exit_fail_msg("No THP is allocated\n"); rss_anon_before = rss_anon(); @@ -311,7 +311,7 @@ static void verify_rss_anon_split_huge_page_all_zeroes(char *one_page, int nr_hp if (one_page[i] != (char)0) ksft_exit_fail_msg("%ld byte corrupted\n", i); - if (!check_huge_anon(one_page, 0, pmd_pagesize)) + if (!check_huge_anon(one_page, nr_hpages * pmd_pagesize, 0, pmd_pagesize)) ksft_exit_fail_msg("Still AnonHugePages not split\n"); rss_anon_after = rss_anon(); @@ -347,7 +347,7 @@ static void split_pmd_thp_to_order(int order) for (i = 0; i < len; i++) one_page[i] = (char)i; - if (!check_huge_anon(one_page, 4, pmd_pagesize)) + if (!check_huge_anon(one_page, 4 * pmd_pagesize, 4, pmd_pagesize)) ksft_exit_fail_msg("No THP is allocated\n"); /* split all THPs */ @@ -366,7 +366,7 @@ static void split_pmd_thp_to_order(int order) (pmd_order + 1))) ksft_exit_fail_msg("Unexpected THP split\n"); - if (!check_huge_anon(one_page, 0, pmd_pagesize)) + if (!check_huge_anon(one_page, 4 * pmd_pagesize, 0, pmd_pagesize)) ksft_exit_fail_msg("Still AnonHugePages not split\n"); ksft_test_result_pass("Split huge pages to order %d successful\n", order); @@ -393,7 +393,7 @@ static void split_pte_mapped_thp(void) for (i = 0; i < thp_area_size; i++) thp_area[i] = (char)i; - if (!check_huge_anon(thp_area, nr_thps, pmd_pagesize)) { + if (!check_huge_anon(thp_area, nr_thps * pmd_pagesize, nr_thps, pmd_pagesize)) { ksft_test_result_skip("Not all THPs allocated\n"); goto out; } @@ -657,7 +657,7 @@ static int create_pagecache_thp_and_fd(const char *testfile, size_t fd_size, force_read_pages(*addr, fd_size / pmd_pagesize, pmd_pagesize); - if (!check_huge_file(*addr, fd_size / pmd_pagesize, pmd_pagesize)) { + if (!check_huge_file(*addr, fd_size, fd_size / pmd_pagesize, pmd_pagesize)) { ksft_print_msg("No large pagecache folio generated, please provide a filesystem supporting large folio\n"); munmap(*addr, fd_size); close(*fd); @@ -735,7 +735,7 @@ static void split_thp_in_pagecache_to_order_at(size_t fd_size, goto out; } - if (!check_huge_file(addr, 0, pmd_pagesize)) { + if (!check_huge_file(addr, fd_size, 0, pmd_pagesize)) { ksft_print_msg("Still FilePmdMapped not split\n"); err = EXIT_FAILURE; goto out; diff --git a/tools/testing/selftests/mm/uffd-common.c b/tools/testing/selftests/mm/uffd-common.c index f48f5d4594ab..1fb967ef4985 100644 --- a/tools/testing/selftests/mm/uffd-common.c +++ b/tools/testing/selftests/mm/uffd-common.c @@ -194,7 +194,9 @@ static void shmem_alias_mapping(uffd_global_test_opts_t *gopts, __u64 *start, static void shmem_check_pmd_mapping(uffd_global_test_opts_t *gopts, void *p, int expect_nr_hpages) { - if (!check_huge_shmem(gopts->area_dst_alias, expect_nr_hpages, + size_t len = expect_nr_hpages * read_pmd_pagesize(); + + if (!check_huge_shmem(gopts->area_dst_alias, len, expect_nr_hpages, read_pmd_pagesize())) err("Did not find expected %d number of hugepages", expect_nr_hpages); diff --git a/tools/testing/selftests/mm/vm_util.c b/tools/testing/selftests/mm/vm_util.c index ef1ea11981a7..ed7b4eae3f3c 100644 --- a/tools/testing/selftests/mm/vm_util.c +++ b/tools/testing/selftests/mm/vm_util.c @@ -247,17 +247,17 @@ err_out: return thp == (nr_hpages * (hpage_size >> 10)); } -bool check_huge_anon(void *addr, int nr_hpages, uint64_t hpage_size) +bool check_huge_anon(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) { return __check_huge(addr, "AnonHugePages: ", nr_hpages, hpage_size); } -bool check_huge_file(void *addr, int nr_hpages, uint64_t hpage_size) +bool check_huge_file(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) { return __check_huge(addr, "FilePmdMapped:", nr_hpages, hpage_size); } -bool check_huge_shmem(void *addr, int nr_hpages, uint64_t hpage_size) +bool check_huge_shmem(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) { return __check_huge(addr, "ShmemPmdMapped:", nr_hpages, hpage_size); } diff --git a/tools/testing/selftests/mm/vm_util.h b/tools/testing/selftests/mm/vm_util.h index 7799154b67ee..565570b2cf8b 100644 --- a/tools/testing/selftests/mm/vm_util.h +++ b/tools/testing/selftests/mm/vm_util.h @@ -90,9 +90,9 @@ void clear_softdirty(void); bool check_for_pattern(FILE *fp, const char *pattern, char *buf, size_t len); uint64_t read_pmd_pagesize(void); unsigned long rss_anon(void); -bool check_huge_anon(void *addr, int nr_hpages, uint64_t hpage_size); -bool check_huge_file(void *addr, int nr_hpages, uint64_t hpage_size); -bool check_huge_shmem(void *addr, int nr_hpages, uint64_t hpage_size); +bool check_huge_anon(void *addr, size_t len, int nr_hpages, uint64_t hpage_size); +bool check_huge_file(void *addr, size_t len, int nr_hpages, uint64_t hpage_size); +bool check_huge_shmem(void *addr, size_t len, int nr_hpages, uint64_t hpage_size); int64_t allocate_transhuge(void *ptr, int pagemap_fd); int pageflags_get(unsigned long pfn, int kpageflags_fd, uint64_t *flags); -- cgit v1.2.3 From 6995150ede2805914b2fbc3a2ba674d06784c04b Mon Sep 17 00:00:00 2001 From: Baolin Wang Date: Thu, 6 Aug 2026 11:34:13 +0800 Subject: selftests: mm: move gather_after_split_folio_orders() into vm_util.c file Move gather_after_split_folio_orders() to vm_util.c as a helper function in preparation for implementing checks for mTHP collapse. While we are at it, rename this function to indicate that it is not only used for large folio splits. No functional changes. Link: https://lore.kernel.org/30a0a99556adf11c2bf97aa08d6da4830bb43f6f.1785985999.git.baolin.wang@linux.alibaba.com Signed-off-by: Baolin Wang Reviewed-by: Nico Pache (Red Hat) Tested-by: Nico Pache (Red Hat) Reviewed-by: Zi Yan Acked-by: Kiryl Shutsemau (Meta) Cc: Barry Song Cc: David Hildenbrand Cc: Dev Jain Cc: Lance Yang Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Ryan Roberts Signed-off-by: Andrew Morton --- tools/testing/selftests/mm/split_huge_page_test.c | 125 +--------------------- tools/testing/selftests/mm/vm_util.c | 119 ++++++++++++++++++++ tools/testing/selftests/mm/vm_util.h | 2 + 3 files changed, 122 insertions(+), 124 deletions(-) diff --git a/tools/testing/selftests/mm/split_huge_page_test.c b/tools/testing/selftests/mm/split_huge_page_test.c index 4cc70873a674..86a603692826 100644 --- a/tools/testing/selftests/mm/split_huge_page_test.c +++ b/tools/testing/selftests/mm/split_huge_page_test.c @@ -104,129 +104,6 @@ fail: return false; } -static int vaddr_pageflags_get(char *vaddr, int pagemap_fd, int kpageflags_fd, - uint64_t *flags) -{ - unsigned long pfn; - - pfn = pagemap_get_pfn(pagemap_fd, vaddr); - - /* non-present PFN */ - if (pfn == -1UL) - return 1; - - if (pageflags_get(pfn, kpageflags_fd, flags)) - return -1; - - return 0; -} - -/* - * gather_after_split_folio_orders - scan through [vaddr_start, len) and record - * folio orders - * - * @vaddr_start: start vaddr - * @len: range length - * @pagemap_fd: file descriptor to /proc//pagemap - * @kpageflags_fd: file descriptor to /proc/kpageflags - * @orders: output folio order array - * @nr_orders: folio order array size - * - * gather_after_split_folio_orders() scan through [vaddr_start, len) and check - * all folios within the range and record their orders. All order-0 pages will - * be recorded. Non-present vaddr is skipped. - * - * NOTE: the function is used to check folio orders after a split is performed, - * so it assumes [vaddr_start, len) fully maps to after-split folios within that - * range. - * - * Return: 0 - no error, -1 - unhandled cases - */ -static int gather_after_split_folio_orders(char *vaddr_start, size_t len, - int pagemap_fd, int kpageflags_fd, int orders[], int nr_orders) -{ - uint64_t page_flags = 0; - int cur_order = -1; - char *vaddr; - - if (pagemap_fd == -1 || kpageflags_fd == -1) - return -1; - if (!orders) - return -1; - if (nr_orders <= 0) - return -1; - - for (vaddr = vaddr_start; vaddr < vaddr_start + len;) { - char *next_folio_vaddr; - int status; - - status = vaddr_pageflags_get(vaddr, pagemap_fd, kpageflags_fd, - &page_flags); - if (status < 0) - return -1; - - /* skip non present vaddr */ - if (status == 1) { - vaddr += psize(); - continue; - } - - /* all order-0 pages with possible false postive (non folio) */ - if (!(page_flags & (KPF_COMPOUND_HEAD | KPF_COMPOUND_TAIL))) { - orders[0]++; - vaddr += psize(); - continue; - } - - /* skip non thp compound pages */ - if (!(page_flags & KPF_THP)) { - vaddr += psize(); - continue; - } - - /* vpn points to part of a THP at this point */ - if (page_flags & KPF_COMPOUND_HEAD) - cur_order = 1; - else { - vaddr += psize(); - continue; - } - - next_folio_vaddr = vaddr + (1UL << (cur_order + pshift())); - - if (next_folio_vaddr >= vaddr_start + len) - break; - - while ((status = vaddr_pageflags_get(next_folio_vaddr, - pagemap_fd, kpageflags_fd, - &page_flags)) >= 0) { - /* - * non present vaddr, next compound head page, or - * order-0 page - */ - if (status == 1 || - (page_flags & KPF_COMPOUND_HEAD) || - !(page_flags & (KPF_COMPOUND_HEAD | KPF_COMPOUND_TAIL))) { - if (cur_order < nr_orders) { - orders[cur_order]++; - cur_order = -1; - vaddr = next_folio_vaddr; - } - break; - } - - cur_order++; - next_folio_vaddr = vaddr + (1UL << (cur_order + pshift())); - } - - if (status < 0) - return status; - } - if (cur_order > 0 && cur_order < nr_orders) - orders[cur_order]++; - return 0; -} - static int check_after_split_folio_orders(char *vaddr_start, size_t len, int pagemap_fd, int kpageflags_fd, int orders[], int nr_orders) { @@ -240,7 +117,7 @@ static int check_after_split_folio_orders(char *vaddr_start, size_t len, ksft_exit_fail_msg("Cannot allocate memory for vaddr_orders"); memset(vaddr_orders, 0, sizeof(int) * nr_orders); - status = gather_after_split_folio_orders(vaddr_start, len, pagemap_fd, + status = gather_folio_orders(vaddr_start, len, pagemap_fd, kpageflags_fd, vaddr_orders, nr_orders); if (status) ksft_exit_fail_msg("gather folio info failed\n"); diff --git a/tools/testing/selftests/mm/vm_util.c b/tools/testing/selftests/mm/vm_util.c index ed7b4eae3f3c..5a427f494cd8 100644 --- a/tools/testing/selftests/mm/vm_util.c +++ b/tools/testing/selftests/mm/vm_util.c @@ -194,6 +194,125 @@ err_out: return rss_anon; } +static int vaddr_pageflags_get(char *vaddr, int pagemap_fd, int kpageflags_fd, + uint64_t *flags) +{ + unsigned long pfn; + + pfn = pagemap_get_pfn(pagemap_fd, vaddr); + + /* non-present PFN */ + if (pfn == -1UL) + return 1; + + if (pageflags_get(pfn, kpageflags_fd, flags)) + return -1; + + return 0; +} + +/* + * gather_folio_orders - scan through [vaddr_start, len) and record + * folio orders + * + * @vaddr_start: start vaddr + * @len: range length + * @pagemap_fd: file descriptor to /proc//pagemap + * @kpageflags_fd: file descriptor to /proc/kpageflags + * @orders: output folio order array + * @nr_orders: folio order array size + * + * gather_folio_orders() scan through [vaddr_start, len) and check + * all folios within the range and record their orders. All order-0 pages will + * be recorded. Non-present vaddr is skipped. + * + * Return: 0 - no error, -1 - unhandled cases + */ +int gather_folio_orders(char *vaddr_start, size_t len, + int pagemap_fd, int kpageflags_fd, int orders[], int nr_orders) +{ + uint64_t page_flags = 0; + int cur_order = -1; + char *vaddr; + + if (pagemap_fd == -1 || kpageflags_fd == -1) + return -1; + if (!orders) + return -1; + if (nr_orders <= 0) + return -1; + + for (vaddr = vaddr_start; vaddr < vaddr_start + len;) { + char *next_folio_vaddr; + int status; + + status = vaddr_pageflags_get(vaddr, pagemap_fd, kpageflags_fd, + &page_flags); + if (status < 0) + return -1; + + /* skip non present vaddr */ + if (status == 1) { + vaddr += psize(); + continue; + } + + /* all order-0 pages with possible false postive (non folio) */ + if (!(page_flags & (KPF_COMPOUND_HEAD | KPF_COMPOUND_TAIL))) { + orders[0]++; + vaddr += psize(); + continue; + } + + /* skip non thp compound pages */ + if (!(page_flags & KPF_THP)) { + vaddr += psize(); + continue; + } + + /* vpn points to part of a THP at this point */ + if (page_flags & KPF_COMPOUND_HEAD) + cur_order = 1; + else { + vaddr += psize(); + continue; + } + + next_folio_vaddr = vaddr + (1UL << (cur_order + pshift())); + + if (next_folio_vaddr >= vaddr_start + len) + break; + + while ((status = vaddr_pageflags_get(next_folio_vaddr, + pagemap_fd, kpageflags_fd, + &page_flags)) >= 0) { + /* + * non present vaddr, next compound head page, or + * order-0 page + */ + if (status == 1 || + (page_flags & KPF_COMPOUND_HEAD) || + !(page_flags & (KPF_COMPOUND_HEAD | KPF_COMPOUND_TAIL))) { + if (cur_order < nr_orders) { + orders[cur_order]++; + cur_order = -1; + vaddr = next_folio_vaddr; + } + break; + } + + cur_order++; + next_folio_vaddr = vaddr + (1UL << (cur_order + pshift())); + } + + if (status < 0) + return status; + } + if (cur_order > 0 && cur_order < nr_orders) + orders[cur_order]++; + return 0; +} + char *__get_smap_entry(void *addr, const char *pattern, char *buf, size_t len) { int ret; diff --git a/tools/testing/selftests/mm/vm_util.h b/tools/testing/selftests/mm/vm_util.h index 565570b2cf8b..9a49af88702e 100644 --- a/tools/testing/selftests/mm/vm_util.h +++ b/tools/testing/selftests/mm/vm_util.h @@ -95,6 +95,8 @@ bool check_huge_file(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) bool check_huge_shmem(void *addr, size_t len, int nr_hpages, uint64_t hpage_size); int64_t allocate_transhuge(void *ptr, int pagemap_fd); int pageflags_get(unsigned long pfn, int kpageflags_fd, uint64_t *flags); +int gather_folio_orders(char *vaddr_start, size_t len, + int pagemap_fd, int kpageflags_fd, int orders[], int nr_orders); int uffd_register(int uffd, void *addr, uint64_t len, bool miss, bool wp, bool minor); -- cgit v1.2.3 From 6dedaf0d46a96cb659ba57a959f5493dbab0de31 Mon Sep 17 00:00:00 2001 From: Baolin Wang Date: Thu, 6 Aug 2026 11:34:14 +0800 Subject: selftests: mm: implement the mTHP-sized hugepage check helpers Implement mTHP-sized hugepage checking helpers using gather_folio_orders(). Also rename the existing PMD-sized huge page check function to __check_pmd_huge() for clarity. Link: https://lore.kernel.org/56b16691f605426b33b5cf47319233de6127a6b3.1785985999.git.baolin.wang@linux.alibaba.com Signed-off-by: Baolin Wang Reviewed-by: Nico Pache (Red Hat) Tested-by: Nico Pache (Red Hat) Acked-by: Kiryl Shutsemau (Meta) Reviewed-by: Zi Yan Cc: Barry Song Cc: David Hildenbrand Cc: Dev Jain Cc: Lance Yang Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Ryan Roberts Signed-off-by: Andrew Morton --- tools/testing/selftests/mm/vm_util.c | 76 ++++++++++++++++++++++++++++++++++-- 1 file changed, 72 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/mm/vm_util.c b/tools/testing/selftests/mm/vm_util.c index 5a427f494cd8..13b5cff7dfe3 100644 --- a/tools/testing/selftests/mm/vm_util.c +++ b/tools/testing/selftests/mm/vm_util.c @@ -15,6 +15,9 @@ #define SMAP_FILE_PATH "/proc/self/smaps" #define STATUS_FILE_PATH "/proc/self/status" #define MAX_LINE_LENGTH 500 +#define PAGEMAP_PATH "/proc/self/pagemap" +#define KPAGEFLAGS_PATH "/proc/kpageflags" +#define MAX_NR_ORDERS 20 unsigned int __page_size; unsigned int __page_shift; @@ -348,7 +351,7 @@ err_out: return entry; } -bool __check_huge(void *addr, char *pattern, int nr_hpages, +static bool __check_pmd_huge(void *addr, char *pattern, int nr_hpages, uint64_t hpage_size) { char buffer[MAX_LINE_LENGTH]; @@ -366,19 +369,84 @@ err_out: return thp == (nr_hpages * (hpage_size >> 10)); } +static bool check_large_folios(void *addr, size_t len, int nr_hpages, + uint64_t hpage_size) +{ + int order = 0, pagesize = getpagesize(); + unsigned int nr_pages = hpage_size / pagesize; + int orders[MAX_NR_ORDERS], status; + int pagemap_fd, kpageflags_fd; + bool ret = false; + + if (!nr_pages) + ksft_exit_fail_msg("invalid hugepage size\n"); + + order = 31 - __builtin_clz(nr_pages); + if (!order || order >= MAX_NR_ORDERS) + ksft_exit_fail_msg("invalid order\n"); + + memset(orders, 0, sizeof(int) * MAX_NR_ORDERS); + pagemap_fd = open(PAGEMAP_PATH, O_RDONLY); + if (pagemap_fd == -1) + ksft_exit_fail_msg("read pagemap fail\n"); + + kpageflags_fd = open(KPAGEFLAGS_PATH, O_RDONLY); + if (kpageflags_fd == -1) { + close(pagemap_fd); + ksft_exit_fail_msg("read kpageflags fail\n"); + } + + status = gather_folio_orders(addr, len, pagemap_fd, + kpageflags_fd, orders, MAX_NR_ORDERS); + if (status) + goto out; + + if (orders[order] == nr_hpages) + ret = true; + +out: + close(pagemap_fd); + close(kpageflags_fd); + return ret; +} + bool check_huge_anon(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) { - return __check_huge(addr, "AnonHugePages: ", nr_hpages, hpage_size); + uint64_t pmd_pagesize = read_pmd_pagesize(); + + if (!pmd_pagesize) + ksft_exit_fail_msg("reading PMD pagesize failed\n"); + + if (hpage_size == pmd_pagesize) + return __check_pmd_huge(addr, "AnonHugePages: ", nr_hpages, hpage_size); + + return check_large_folios(addr, len, nr_hpages, hpage_size); } bool check_huge_file(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) { - return __check_huge(addr, "FilePmdMapped:", nr_hpages, hpage_size); + uint64_t pmd_pagesize = read_pmd_pagesize(); + + if (!pmd_pagesize) + ksft_exit_fail_msg("reading PMD pagesize failed\n"); + + if (hpage_size == pmd_pagesize) + return __check_pmd_huge(addr, "FilePmdMapped:", nr_hpages, hpage_size); + + return check_large_folios(addr, len, nr_hpages, hpage_size); } bool check_huge_shmem(void *addr, size_t len, int nr_hpages, uint64_t hpage_size) { - return __check_huge(addr, "ShmemPmdMapped:", nr_hpages, hpage_size); + uint64_t pmd_pagesize = read_pmd_pagesize(); + + if (!pmd_pagesize) + ksft_exit_fail_msg("reading PMD pagesize failed\n"); + + if (hpage_size == pmd_pagesize) + return __check_pmd_huge(addr, "ShmemPmdMapped:", nr_hpages, hpage_size); + + return check_large_folios(addr, len, nr_hpages, hpage_size); } int64_t allocate_transhuge(void *ptr, int pagemap_fd) -- cgit v1.2.3 From 76f134aabb623dfcec5c9cd9bece23365dd04cfd Mon Sep 17 00:00:00 2001 From: Baolin Wang Date: Thu, 6 Aug 2026 11:34:15 +0800 Subject: selftests: mm: add mTHP collapse test cases Added a new command 'mthp_khugepaged' for mTHP collapse, along with the '-c' parameter to specify the collapse order. Additionally, added mTHP collapse test cases for 'collapse_full', 'collapse_empty', and 'collapse_single_mthp' for anonymous folios. All khugepaged test cases passed. Link: https://lore.kernel.org/f260058520214a9611922a96326bc54ba282fb73.1785985999.git.baolin.wang@linux.alibaba.com Signed-off-by: Baolin Wang Reviewed-by: Nico Pache (Red Hat) Tested-by: Nico Pache (Red Hat) Acked-by: Kiryl Shutsemau (Meta) Acked-by: Zi Yan Cc: Barry Song Cc: David Hildenbrand Cc: Dev Jain Cc: Lance Yang Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Ryan Roberts Signed-off-by: Andrew Morton --- tools/testing/selftests/mm/khugepaged.c | 137 +++++++++++++++++++++++++----- tools/testing/selftests/mm/run_vmtests.sh | 2 + 2 files changed, 120 insertions(+), 19 deletions(-) diff --git a/tools/testing/selftests/mm/khugepaged.c b/tools/testing/selftests/mm/khugepaged.c index c02d00846a79..8f221c792a28 100644 --- a/tools/testing/selftests/mm/khugepaged.c +++ b/tools/testing/selftests/mm/khugepaged.c @@ -26,9 +26,11 @@ #define BASE_ADDR ((void *)(1UL << 30)) static unsigned long hpage_pmd_size; +static int hpage_pmd_order; static unsigned long page_size; static int hpage_pmd_nr; static int anon_order; +static int collapse_order; #define PID_SMAPS "/proc/self/smaps" #define TEST_FILE "collapse_test_file" @@ -69,6 +71,7 @@ struct collapse_context { }; static struct collapse_context *khugepaged_context; +static struct collapse_context *mthp_khugepaged_context; static struct collapse_context *madvise_context; struct file_info { @@ -554,25 +557,25 @@ static void madvise_collapse(const char *msg, char *p, int nr_hpages, } #define TICK 500000 -static bool wait_for_scan(const char *msg, char *p, int nr_hpages, - struct mem_ops *ops) +static bool wait_for_scan(const char *msg, char *p, size_t len, + int nr_hpages, int collap_order, struct mem_ops *ops) { - size_t len = nr_hpages * hpage_pmd_size; + unsigned long hpage_size = page_size << collap_order; int full_scans; int timeout = 6; /* 3 seconds */ /* Sanity check */ - if (!ops->check_huge(p, len, 0, hpage_pmd_size)) + if (!ops->check_huge(p, len, 0, hpage_size)) ksft_exit_fail_msg("Unexpected huge page\n"); - madvise(p, nr_hpages * hpage_pmd_size, MADV_HUGEPAGE); + madvise(p, len, MADV_HUGEPAGE); /* Wait until the second full_scan completed */ full_scans = thp_read_num("khugepaged/full_scans") + 2; ksft_print_msg("%s...", msg); while (timeout--) { - if (ops->check_huge(p, len, nr_hpages, hpage_pmd_size)) + if (ops->check_huge(p, len, nr_hpages, hpage_size)) break; if (thp_read_num("khugepaged/full_scans") >= full_scans) break; @@ -595,7 +598,7 @@ static void khugepaged_collapse(const char *msg, char *p, int nr_hpages, if (!is_tmpfs(ops) && ops == &__read_write_file_write_ops) expect = false; - if (wait_for_scan(msg, p, nr_hpages, ops)) { + if (wait_for_scan(msg, p, len, nr_hpages, hpage_pmd_order, ops)) { if (expect) fail("Timeout"); else @@ -617,12 +620,62 @@ static void khugepaged_collapse(const char *msg, char *p, int nr_hpages, fail("Fail"); } +static void mthp_khugepaged_collapse(const char *msg, char *p, int nr_hpages, + struct mem_ops *ops, bool expect) +{ + unsigned long hpage_size = page_size << collapse_order; + struct thp_settings settings = *thp_current_settings(); + /* mTHP collpase only allocates PMD sized memory */ + size_t len = hpage_pmd_size; + + /* Set mTHP setting for mTHP collapse */ + if (ops == &__anon_ops) { + settings.thp_enabled = THP_NEVER; + settings.hugepages[collapse_order].enabled = THP_MADVISE; + } + + thp_push_settings(&settings); + + if (wait_for_scan(msg, p, len, nr_hpages, collapse_order, ops)) { + if (expect) + fail("Timeout"); + else + success("OK"); + + /* Restore THP settings for mTHP collapse. */ + thp_pop_settings(); + return; + } + + /* + * For file and shmem memory, khugepaged only retracts pte entries after + * putting the new hugepage in the page cache. The hugepage must be + * subsequently refaulted to install the pmd mapping for the mm. + */ + if (ops != &__anon_ops) + ops->fault(p, 0, nr_hpages * hpage_size); + + if (ops->check_huge(p, len, expect ? nr_hpages : 0, hpage_size)) + success("OK"); + else + fail("Fail"); + + /* Restore THP settings for mTHP collapse. */ + thp_pop_settings(); +} + static struct collapse_context __khugepaged_context = { .collapse = &khugepaged_collapse, .enforce_pte_scan_limits = true, .name = "khugepaged", }; +static struct collapse_context __mthp_khugepaged_context = { + .collapse = &mthp_khugepaged_collapse, + .enforce_pte_scan_limits = true, + .name = "mthp_khugepaged", +}; + static struct collapse_context __madvise_context = { .collapse = &madvise_collapse, .enforce_pte_scan_limits = false, @@ -661,10 +714,17 @@ static void alloc_at_fault(void) static void collapse_full(struct collapse_context *c, struct mem_ops *ops) { void *p; - int nr_hpages = 4; + int nr_pmds = 4, nr_hpages = 4; unsigned long size = nr_hpages * hpage_pmd_size; - p = ops->setup_area(nr_hpages); + /* Only try 1 PMD sized range for mTHP collapse. */ + if (c == &__mthp_khugepaged_context) { + nr_pmds = 1; + nr_hpages = 1 << (hpage_pmd_order - collapse_order); + size = hpage_pmd_size; + } + + p = ops->setup_area(nr_pmds); ops->fault(p, 0, size); c->collapse("Collapse multiple fully populated PTE table", p, nr_hpages, ops, true); @@ -676,10 +736,31 @@ static void collapse_full(struct collapse_context *c, struct mem_ops *ops) static void collapse_empty(struct collapse_context *c, struct mem_ops *ops) { + int nr_hpages = 1; + void *p; + + if (c == &__mthp_khugepaged_context) + nr_hpages = 1 << (hpage_pmd_order - collapse_order); + + p = ops->setup_area(1); + c->collapse("Do not collapse empty PTE table", p, nr_hpages, ops, false); + ops->cleanup_area(p, hpage_pmd_size); + ksft_test_result_report(exit_status, "%s\n", __func__); +} + +static void collapse_single_mthp(struct collapse_context *c, struct mem_ops *ops) +{ + unsigned long hpage_size = page_size << collapse_order; void *p; p = ops->setup_area(1); - c->collapse("Do not collapse empty PTE table", p, 1, ops, false); + /* + * Only fault collapse_order sized ranges, and only check 1 + * collapse_order sized huge page. + */ + ops->fault(p, 0, hpage_size); + c->collapse("Collapse PTE table with half PTE entries present", + p, 1, ops, true); ops->cleanup_area(p, hpage_pmd_size); ksft_test_result_report(exit_status, "%s\n", __func__); } @@ -1081,8 +1162,8 @@ static void madvise_retracted_page_tables(struct collapse_context *c, ops->fault(p, 0, size); /* Let khugepaged collapse and leave pmd cleared */ - if (wait_for_scan("Collapse and leave PMD cleared", p, nr_hpages, - ops)) { + if (wait_for_scan("Collapse and leave PMD cleared", p, size, nr_hpages, + hpage_pmd_order, ops)) { fail("Timeout"); return; } @@ -1098,17 +1179,19 @@ static void usage(void) { fprintf(stderr, "\nUsage: ./khugepaged [OPTIONS] [dir]\n\n"); fprintf(stderr, "\t\t: :\n"); - fprintf(stderr, "\t\t: [all|khugepaged|madvise]\n"); + fprintf(stderr, "\t\t: [all|khugepaged|mthp_khugepaged|madvise]\n"); fprintf(stderr, "\t\t: [all|anon|file|shmem]\n"); fprintf(stderr, "\n\t\"file,all\" mem_type requires [dir] argument\n"); fprintf(stderr, "\n\t\"file,all\" mem_type requires a file system\n"); fprintf(stderr, "\twith PMD-sized large folio support\n"); fprintf(stderr, "\n\tif [dir] is a (sub)directory of a tmpfs mount, tmpfs must be\n"); fprintf(stderr, "\tmounted with huge=advise option for khugepaged tests to work\n"); + fprintf(stderr, "\n\tmthp_khugepaged only supports anon mem_type now.\n"); fprintf(stderr, "\n\tSupported Options:\n"); fprintf(stderr, "\t\t-h: This help message.\n"); fprintf(stderr, "\t\t-s: mTHP size, expressed as page order.\n"); fprintf(stderr, "\t\t Defaults to 0. Use this size for anon or shmem allocations.\n"); + fprintf(stderr, "\t\t-c: collapse order for mTHP collapse, expressed as page order.\n"); exit(1); } @@ -1118,11 +1201,14 @@ static void parse_test_type(int argc, char **argv) char *buf; const char *token; - while ((opt = getopt(argc, argv, "s:h")) != -1) { + while ((opt = getopt(argc, argv, "s:c:h")) != -1) { switch (opt) { case 's': anon_order = atoi(optarg); break; + case 'c': + collapse_order = atoi(optarg); + break; case 'h': default: usage(); @@ -1148,6 +1234,10 @@ static void parse_test_type(int argc, char **argv) madvise_context = &__madvise_context; } else if (!strcmp(token, "khugepaged")) { khugepaged_context = &__khugepaged_context; + } else if (!strcmp(token, "mthp_khugepaged")) { + mthp_khugepaged_context = &__mthp_khugepaged_context; + if (collapse_order <= 0 || collapse_order >= hpage_pmd_order) + usage(); } else if (!strcmp(token, "madvise")) { madvise_context = &__madvise_context; } else { @@ -1163,14 +1253,20 @@ static void parse_test_type(int argc, char **argv) read_write_file_write_ops = &__read_write_file_write_ops; anon_ops = &__anon_ops; shmem_ops = &__shmem_ops; + if (mthp_khugepaged_context) + usage(); } else if (!strcmp(buf, "anon")) { anon_ops = &__anon_ops; } else if (!strcmp(buf, "file")) { read_only_file_ops = &__read_only_file_ops; read_write_file_read_ops = &__read_write_file_read_ops; read_write_file_write_ops = &__read_write_file_write_ops; + if (mthp_khugepaged_context) + usage(); } else if (!strcmp(buf, "shmem")) { shmem_ops = &__shmem_ops; + if (mthp_khugepaged_context) + usage(); } else { usage(); } @@ -1213,7 +1309,6 @@ static int nr_test_cases; int main(int argc, char **argv) { - int hpage_pmd_order; struct thp_settings default_settings = { .thp_enabled = THP_MADVISE, .thp_defrag = THP_DEFRAG_ALWAYS, @@ -1239,10 +1334,6 @@ int main(int argc, char **argv) if (!thp_is_enabled()) ksft_exit_skip("Transparent Hugepages not available\n"); - parse_test_type(argc, argv); - - setbuf(stdout, NULL); - page_size = getpagesize(); hpage_pmd_size = read_pmd_pagesize(); if (!hpage_pmd_size) @@ -1250,6 +1341,10 @@ int main(int argc, char **argv) hpage_pmd_nr = hpage_pmd_size / page_size; hpage_pmd_order = __builtin_ctz(hpage_pmd_nr); + parse_test_type(argc, argv); + + setbuf(stdout, NULL); + default_settings.khugepaged.max_ptes_none = hpage_pmd_nr - 1; default_settings.khugepaged.max_ptes_swap = hpage_pmd_nr / 8; default_settings.khugepaged.max_ptes_shared = hpage_pmd_nr / 2; @@ -1267,6 +1362,7 @@ int main(int argc, char **argv) TEST(collapse_full, khugepaged_context, read_write_file_read_ops); TEST(collapse_full, khugepaged_context, read_write_file_write_ops); TEST(collapse_full, khugepaged_context, shmem_ops); + TEST(collapse_full, mthp_khugepaged_context, anon_ops); TEST(collapse_full, madvise_context, anon_ops); TEST(collapse_full, madvise_context, read_only_file_ops); TEST(collapse_full, madvise_context, read_write_file_read_ops); @@ -1274,8 +1370,11 @@ int main(int argc, char **argv) TEST(collapse_full, madvise_context, shmem_ops); TEST(collapse_empty, khugepaged_context, anon_ops); + TEST(collapse_empty, mthp_khugepaged_context, anon_ops); TEST(collapse_empty, madvise_context, anon_ops); + TEST(collapse_single_mthp, mthp_khugepaged_context, anon_ops); + TEST(collapse_single_pte_entry, khugepaged_context, anon_ops); TEST(collapse_single_pte_entry, khugepaged_context, read_only_file_ops); TEST(collapse_single_pte_entry, khugepaged_context, read_write_file_read_ops); diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh index 687d115e3bd8..d09f9f6a384e 100755 --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -410,6 +410,8 @@ CATEGORY="thp" run_test ./khugepaged all:shmem CATEGORY="thp" run_test ./khugepaged -s 4 all:shmem +CATEGORY="thp" run_test ./khugepaged -c 4 mthp_khugepaged:anon + # Try to create XFS if not provided if [ -z "${SPLIT_HUGE_PAGE_TEST_XFS_PATH}" ]; then if test_selected "thp"; then -- cgit v1.2.3 From 4b82a0b91be5f8cf7a3a46d6fa3931224150b6b4 Mon Sep 17 00:00:00 2001 From: Hongfu Li Date: Thu, 6 Aug 2026 11:08:50 +0800 Subject: selftests/mm: drop duplicate test_seal_mprotect_two_vma_with_gap() call mseal_test main() invokes test_seal_mprotect_two_vma_with_gap() twice. The second run repeats all assertions with no benefit. Drop the duplicate call. Link: https://lore.kernel.org/20260806030850.76077-1-hongfu.li@linux.dev Signed-off-by: Hongfu Li Reviewed-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Reviewed-by: SJ Park Reviewed-by: Anshuman Khandual Cc: Liam R. Howlett Cc: Michal Hocko Cc: Mike Rapoport Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- tools/testing/selftests/mm/mseal_test.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tools/testing/selftests/mm/mseal_test.c b/tools/testing/selftests/mm/mseal_test.c index 93c2e13094d4..1a05e6921fed 100644 --- a/tools/testing/selftests/mm/mseal_test.c +++ b/tools/testing/selftests/mm/mseal_test.c @@ -1876,7 +1876,7 @@ int main(void) if (!pkey_supported()) ksft_print_msg("PKEY not supported\n"); - ksft_set_plan(88); + ksft_set_plan(87); test_seal_addseal(); test_seal_unmapped_start(); @@ -1913,7 +1913,6 @@ int main(void) test_seal_mprotect_partial_mprotect(false); test_seal_mprotect_partial_mprotect(true); - test_seal_mprotect_two_vma_with_gap(); test_seal_mprotect_two_vma_with_gap(); test_seal_mprotect_merge(false); -- cgit v1.2.3 From 54cc9b38470905e7fdf4a8786e9b1b85ac045f0a Mon Sep 17 00:00:00 2001 From: Ye Liu Date: Thu, 6 Aug 2026 08:45:55 +0800 Subject: mm: debug_page_alloc: fix NULL buf in debug_guardpage_minorder_setup If the kernel command line includes "debug_guardpage_minorder" without an equals sign (i.e., no value is provided), the early parameter parser passes a NULL buf pointer to the setup function. kstrtouint() does not perform a NULL check on its input and calls directly into kstrtoull() which dereferences s[0] unconditionally, leading to a NULL pointer dereference and early boot crash. Additionally, the error path's pr_err("%s", buf) would also crash with a NULL format argument. Link: https://lore.kernel.org/20260806004556.2633049-1-ye.liu@linux.dev Fixes: c0a32fc5a2e4 ("mm: more intensive memory corruption debugging") Signed-off-by: Ye Liu Acked-by: Zi Yan Reviewed-by: John Hubbard Reviewed-by: Andrew Morton Cc: Johannes Weiner Cc: Michal Hocko Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- mm/debug_page_alloc.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/debug_page_alloc.c b/mm/debug_page_alloc.c index 41e3d1f1ad96..fd2664c3c86a 100644 --- a/mm/debug_page_alloc.c +++ b/mm/debug_page_alloc.c @@ -22,8 +22,8 @@ static int __init debug_guardpage_minorder_setup(char *buf) { unsigned int res; - if (kstrtouint(buf, 10, &res) < 0 || res > MAX_PAGE_ORDER / 2) { - pr_err("Bad debug_guardpage_minorder value: %s\n", buf); + if (!buf || kstrtouint(buf, 10, &res) < 0 || res > MAX_PAGE_ORDER / 2) { + pr_err("Bad debug_guardpage_minorder value: %s\n", buf ?: "(missing)"); return 0; } _debug_guardpage_minorder = res; -- cgit v1.2.3 From dc924f0f85afd5a211357e2f2b670772853d1ce3 Mon Sep 17 00:00:00 2001 From: Audra Mitchell Date: Thu, 6 Aug 2026 11:00:34 -0400 Subject: selftests/mm/vm_util.c: correct __pagemap_scan_get_categories return value Currently __pagemap_scan_get_categories returns the result from the ioctl call which should be an int, not uint64_t. The ioctl may return -1 on error, which will be interpreted as UINT64_MAX. Adjust the return type to use the correct value. Link: https://lore.kernel.org/20260806150339.1824251-2-audra@redhat.com Signed-off-by: Audra Mitchell Reviewed-by: Liam R. Howlett (Oracle) Acked-by: David Hildenbrand (Arm) Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- tools/testing/selftests/mm/vm_util.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/mm/vm_util.c b/tools/testing/selftests/mm/vm_util.c index 13b5cff7dfe3..4fe4a5a610d1 100644 --- a/tools/testing/selftests/mm/vm_util.c +++ b/tools/testing/selftests/mm/vm_util.c @@ -34,7 +34,7 @@ uint64_t pagemap_get_entry(int fd, char *start) return entry; } -static uint64_t __pagemap_scan_get_categories(int fd, char *start, struct page_region *r) +static int __pagemap_scan_get_categories(int fd, char *start, struct page_region *r) { struct pm_scan_arg arg; @@ -58,7 +58,7 @@ static uint64_t __pagemap_scan_get_categories(int fd, char *start, struct page_r static uint64_t pagemap_scan_get_categories(int fd, char *start) { struct page_region r; - long ret; + int ret; ret = __pagemap_scan_get_categories(fd, start, &r); if (ret < 0) -- cgit v1.2.3 From 36a9799b2c86fd0f86d54f3eec8ea37ac467be25 Mon Sep 17 00:00:00 2001 From: Zhan Xusheng Date: Wed, 5 Aug 2026 15:05:29 +0800 Subject: maple_tree: remove unused mas_is_root_limits() The last callers of mas_is_root_limits() were removed by commit b8852ef30c67 ("maple_tree: remove maple big node and subtree structs"), together with the maple subtree state (mast_*) code that used it. As a static inline it does not trigger -Wunused-function, so it went unnoticed. Remove it. No functional change. Link: https://lore.kernel.org/20260805070529.4118794-1-zhanxusheng@xiaomi.com Signed-off-by: Zhan Xusheng Reviewed-by: Liam R. Howlett (Oracle) Cc: Alice Ryhl Cc: Andrew Ballance Signed-off-by: Andrew Morton --- lib/maple_tree.c | 5 ----- 1 file changed, 5 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 529acc056e55..b48bc4064ad2 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -356,11 +356,6 @@ static __always_inline bool mte_is_root(const struct maple_enode *node) return ma_is_root(mte_to_node(node)); } -static inline bool mas_is_root_limits(const struct ma_state *mas) -{ - return !mas->min && mas->max == ULONG_MAX; -} - static __always_inline bool mt_is_alloc(struct maple_tree *mt) { return (mt->ma_flags & MT_FLAGS_ALLOC_RANGE); -- cgit v1.2.3 From 20d4d490bc0ca5fbfa3f99e1e8e7d398b7971ccb Mon Sep 17 00:00:00 2001 From: Henry Elderman Date: Fri, 7 Aug 2026 11:19:58 +0200 Subject: mm/execmem: fix fallback_end description in kernel-doc The kernel-doc for struct execmem_range incorrectly describes @fallback_end as "start". Correct it to "end". Link: https://lore.kernel.org/20260807091958.4735-1-henry.elderman.edu+linux@gmail.com Signed-off-by: Henry Elderman Reviewed-by: Mike Rapoport (Microsoft) Signed-off-by: Andrew Morton --- include/linux/execmem.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/execmem.h b/include/linux/execmem.h index 7de229134e30..1bd34925d1aa 100644 --- a/include/linux/execmem.h +++ b/include/linux/execmem.h @@ -89,7 +89,7 @@ static inline int execmem_restore_rox(void *ptr, size_t size) { return 0; } * @end: address space end (inclusive) * @fallback_start: start of the secondary address space range for fallback * allocations on architectures that require it - * @fallback_end: start of the secondary address space (inclusive) + * @fallback_end: end of the secondary address space (inclusive) * @pgprot: permissions for memory in this address space * @alignment: alignment required for text allocations * @flags: options for memory allocations for this range -- cgit v1.2.3 From 4004c130c358b1561a55323b0e747f20f5133f7b Mon Sep 17 00:00:00 2001 From: Muhammad Usama Anjum Date: Mon, 27 Jul 2026 10:52:17 +0100 Subject: selftests/mm: skip COW tmpfile cases when fallocate() is unsupported Patch series "selftests/mm: Handle unsupported and transient test conditions", v3. Several MM selftests report failures when the test environment lacks an underlying prerequisite, such as fallocate() support, MADV_REMOVE, local page-cache semantics, or swap. This series converts those unsupported cases to SKIP while preserving failures for unexpected errors. It also allows migration tests to retry transient move_pages() failures. This patch (of 4): The tmpfile-backed COW cases allocate a one-page file with fallocate() before exercising private and shared mappings. When the filesystem backing tmpfile() does not implement fallocate(), setup fails with EOPNOTSUPP and no COW behavior is exercised. This occurs when the temporary directory resides on a filesystem with limited allocation support, such as NFSv3. Reporting a failure adds noise because the test prerequisite is absent rather than the COW implementation being broken. Report EOPNOTSUPP as a skip. Continue treating every other fallocate() error as a failure so unexpected setup regressions remain visible. Link: https://lore.kernel.org/20260727095225.372655-1-usama.anjum@arm.com Link: https://lore.kernel.org/20260727095225.372655-2-usama.anjum@arm.com Fixes: f8664f3c4a08 ("selftests/vm: cow: basic COW tests for non-anonymous pages") Signed-off-by: Muhammad Usama Anjum Tested-by: Sarthak Sharma Acked-by: Usama Arif Acked-by: David Hildenbrand (Arm) Cc: Baolin Wang Cc: Barry Song Cc: Dev Jain Cc: Lance Yang Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Naoya Horiguchi Cc: Nico Pache Cc: Ryan Roberts Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Zi Yan Signed-off-by: Andrew Morton --- tools/testing/selftests/mm/cow.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/mm/cow.c b/tools/testing/selftests/mm/cow.c index 7fa2d97ca9b2..8aa5249d9bef 100644 --- a/tools/testing/selftests/mm/cow.c +++ b/tools/testing/selftests/mm/cow.c @@ -1718,8 +1718,13 @@ static void run_with_tmpfile(non_anon_test_fn fn, const char *desc) /* File consists of a single page filled with zeroes. */ if (fallocate(fd, 0, 0, pagesize)) { - ksft_perror("fallocate() failed"); - log_test_result(KSFT_FAIL); + if (errno == EOPNOTSUPP) { + ksft_print_msg("fallocate() not supported by filesystem\n"); + log_test_result(KSFT_SKIP); + } else { + ksft_perror("fallocate() failed"); + log_test_result(KSFT_FAIL); + } goto close; } -- cgit v1.2.3 From e5220e4d934f8c06ef7734ed56bdd9b62307ea9c Mon Sep 17 00:00:00 2001 From: Muhammad Usama Anjum Date: Mon, 27 Jul 2026 10:52:18 +0100 Subject: selftests/mm: skip guard hole-punch test if MADV_REMOVE is unsupported The hole_punch case verifies that guard regions survive MADV_REMOVE and that the backing range is punched out. MADV_REMOVE delegates the hole punch to the backing filesystem, which may reject the operation with EOPNOTSUPP. That result means the test cannot establish the state whose guard semantics it intends to validate. Treating the missing filesystem capability as a guard-region failure creates a false regression. Unmap the range and skip only when MADV_REMOVE fails with EOPNOTSUPP. Preserve the assertion for all other errors so failures on supported configurations remain visible. Link: https://lore.kernel.org/20260727095225.372655-3-usama.anjum@arm.com Signed-off-by: Muhammad Usama Anjum Tested-by: Sarthak Sharma Acked-by: Usama Arif Acked-by: David Hildenbrand (Arm) Cc: Baolin Wang Cc: Barry Song Cc: Dev Jain Cc: Lance Yang Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Naoya Horiguchi Cc: Nico Pache Cc: Ryan Roberts Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Zi Yan Signed-off-by: Andrew Morton --- tools/testing/selftests/mm/guard-regions.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/mm/guard-regions.c b/tools/testing/selftests/mm/guard-regions.c index b21df3040b1c..5c8ec3ca75d7 100644 --- a/tools/testing/selftests/mm/guard-regions.c +++ b/tools/testing/selftests/mm/guard-regions.c @@ -1912,7 +1912,7 @@ TEST_F(guard_regions, hole_punch) { const unsigned long page_size = self->page_size; char *ptr; - int i; + int i, ret; if (variant->backing == ANON_BACKED) SKIP(return, "Truncation test specific to file-backed"); @@ -1944,8 +1944,12 @@ TEST_F(guard_regions, hole_punch) } /* Now hole punch the guarded region. */ - ASSERT_EQ(madvise(&ptr[3 * page_size], 4 * page_size, - MADV_REMOVE), 0); + ret = madvise(&ptr[3 * page_size], 4 * page_size, MADV_REMOVE); + if (ret == -1 && errno == EOPNOTSUPP) { + ASSERT_EQ(munmap(ptr, 10 * page_size), 0); + SKIP(return, "MADV_REMOVE not supported by filesystem"); + } + ASSERT_EQ(ret, 0); /* Ensure guard regions remain. */ for (i = 0; i < 10; i++) { -- cgit v1.2.3 From e14e52a7ce02f6b62cecbf5c61425b4137422df2 Mon Sep 17 00:00:00 2001 From: Muhammad Usama Anjum Date: Mon, 27 Jul 2026 10:52:20 +0100 Subject: selftests/mm: skip hard dirty page-cache test on NFS The hard dirty_pagecache variant uses MADV_HWPOISON to exercise recovery of a dirty file-backed page. The recovery path records -EIO in the address_space mapping, which NFS later reports when the test closes the file. This makes the test fail after the hwpoison checks have completed. Skip this variant when the test file is on NFS. Keep the hard clean-page and both soft-offline variants enabled because they use folio removal, invalidation, or migration rather than recording a delayed writeback error. The unsupported-filesystem path in clean_pagecache() also returns without closing the opened test file. Close the descriptor before skipping there and in dirty_pagecache(). Link: https://lore.kernel.org/20260727095225.372655-5-usama.anjum@arm.com Signed-off-by: Muhammad Usama Anjum Reviewed-by: Miaohe Lin Cc: Baolin Wang Cc: Barry Song Cc: David Hildenbrand Cc: Dev Jain Cc: Lance Yang Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Naoya Horiguchi Cc: Nico Pache Cc: Ryan Roberts Cc: Sarthak Sharma Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Usama Arif Cc: Vlastimil Babka Cc: Zi Yan Signed-off-by: Andrew Morton --- tools/testing/selftests/mm/memory-failure.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/mm/memory-failure.c b/tools/testing/selftests/mm/memory-failure.c index 1a5a32e22cce..f3cb578b1609 100644 --- a/tools/testing/selftests/mm/memory-failure.c +++ b/tools/testing/selftests/mm/memory-failure.c @@ -287,8 +287,10 @@ TEST_F(memory_failure, clean_pagecache) if (fd < 0) SKIP(return, "failed to open test file.\n"); fs_type = get_fs_type(fd); - if (!fs_type || fs_type == TMPFS_MAGIC) + if (!fs_type || fs_type == TMPFS_MAGIC) { + close(fd); SKIP(return, "unsupported filesystem :%x\n", fs_type); + } addr = mmap(0, self->page_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); @@ -327,8 +329,16 @@ TEST_F(memory_failure, dirty_pagecache) if (fd < 0) SKIP(return, "failed to open test file.\n"); fs_type = get_fs_type(fd); - if (!fs_type || fs_type == TMPFS_MAGIC) + /* + * MADV_HARD poisoning of dirty page-cache data records an expected + * -EIO in the file mapping. NFS reports this error on close(), so + * skip this variant. + */ + if (!fs_type || fs_type == TMPFS_MAGIC || + (fs_type == NFS_SUPER_MAGIC && variant->type == MADV_HARD)) { + close(fd); SKIP(return, "unsupported filesystem :%x\n", fs_type); + } addr = mmap(0, self->page_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); -- cgit v1.2.3 From 746c94b7cb7900327a6ca7c1fcbfdd0243729253 Mon Sep 17 00:00:00 2001 From: Muhammad Usama Anjum Date: Mon, 27 Jul 2026 10:52:21 +0100 Subject: selftests/mm: retry migration failures for the full runtime move_pages() is best effort and can temporarily fail when concurrent faults race with page unmapping. A busy shared-anon workload can exhaust the current 100 retries long before the intended 20-second runtime and produce a false failure. Use the full runtime as the retry window. Since the initial page location is unknown, require it to reach both alternating NUMA targets to confirm that cross-node migration made progress despite transient contention. Link: https://lore.kernel.org/20260727095225.372655-6-usama.anjum@arm.com Signed-off-by: Muhammad Usama Anjum Cc: Baolin Wang Cc: Barry Song Cc: David Hildenbrand Cc: Dev Jain Cc: Lance Yang Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Naoya Horiguchi Cc: Nico Pache Cc: Ryan Roberts Cc: Sarthak Sharma Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Usama Arif Cc: Vlastimil Babka Cc: Zi Yan Signed-off-by: Andrew Morton --- tools/testing/selftests/mm/migration.c | 37 ++++++++++++++++++---------------- 1 file changed, 20 insertions(+), 17 deletions(-) diff --git a/tools/testing/selftests/mm/migration.c b/tools/testing/selftests/mm/migration.c index 29f7492453d4..f19d53c69576 100644 --- a/tools/testing/selftests/mm/migration.c +++ b/tools/testing/selftests/mm/migration.c @@ -7,7 +7,7 @@ #include "kselftest_harness.h" #include "hugepage_settings.h" -#include +#include #include #include #include @@ -20,7 +20,6 @@ #define TWOMEG (2<<20) #define RUNTIME (20) -#define MAX_RETRIES 100 #define ALIGN(x, a) (((x) + (a - 1)) & (~((a) - 1))) HUGETLB_SETUP_DEFAULT_PAGES(1) @@ -110,7 +109,7 @@ int migrate(uint64_t *ptr, int n1, int n2) int ret, tmp; int status = 0; struct timespec ts1, ts2; - int failures = 0; + int success = 0; if (clock_gettime(CLOCK_MONOTONIC, &ts1)) return -1; @@ -119,29 +118,33 @@ int migrate(uint64_t *ptr, int n1, int n2) if (clock_gettime(CLOCK_MONOTONIC, &ts2)) return -1; - if (ts2.tv_sec - ts1.tv_sec >= RUNTIME) - return 0; + if (ts2.tv_sec - ts1.tv_sec >= RUNTIME) { + /* Reaching both targets verifies a cross-node move. */ + if (success >= 2) + return 0; + else + return -2; + } ret = move_pages(0, 1, (void **) &ptr, &n2, &status, MPOL_MF_MOVE_ALL); - if (ret) { - if (ret > 0) { - /* Migration is best effort; try again */ - if (++failures < MAX_RETRIES) - continue; - printf("Didn't migrate %d pages\n", ret); - } - else - perror("Couldn't migrate pages"); + if (ret < 0) { + perror("Couldn't migrate pages"); + return ret; + } + /* Migration is best effort. Try again */ + if (ret > 0 || status < 0) + continue; + if (status != n2) { + printf("Page is on node %d instead of target node %d\n", + status, n2); return -2; } - failures = 0; + success++; tmp = n2; n2 = n1; n1 = tmp; } - - return 0; } void *access_mem(void *ptr) -- cgit v1.2.3 From dc8458f43fe964d8ade74c9b0fce54fe71d156de Mon Sep 17 00:00:00 2001 From: Hao Jia Date: Thu, 6 Aug 2026 15:09:42 +0800 Subject: mm/zswap: fix global shrinker when memory cgroup is disabled MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Patch series "mm/zswap: Fixes and improves the zswap shrink", v4. This series fixes and improves the zswap global shrinker (shrink_worker()): Patch 1: Fix missing global shrinker when memory cgroup is disabled. Patch 2: Extend shrink_memcg() to support batch writeback and thereby improving the writeback efficiency in the shrink_worker() and zswap_store() paths. This patch (of 2): Zswap writeback when the global pool limit is hit fails when memory cgroup is disabled. The pool remains full until it is organically drained by swapins or memory freeing, leading to zswap store failures and pages bypassing getting written directly to the backing swap device, causing LRU inversion (hotter pages with higher fault latency). This happens because mem_cgroup_iter() always returns NULL when memory cgroups are disabled. As a result, the global shrinker shrink_worker() repeatedly takes empty walks. After MAX_RECLAIM_RETRIES failed attempts, the worker gives up without writing back any pages. Therefore, when memory cgroup is disabled, fall through with the !memcg branch and shrink the root memcg directly. With memcg disabled, shrink_memcg() only returns -ENOENT when the root LRU is empty, which means the total pages are already below thr. In the absence of heavy concurrent zswap stores, the loop then safely bails out via the zswap_total_pages() <= thr check; otherwise, it will resume shrinking the memcg after processing the reschedule check. For any other return value from shrink_memcg(), the loop is guaranteed to terminate, either after MAX_RECLAIM_RETRIES failures or once the threshold is met. This is a potential performance regression for people using zswap without memcg that was introduced by the commit in "Fixes". Link: https://lore.kernel.org/20260806070943.95542-1-jiahao.kernel@gmail.com Link: https://lore.kernel.org/20260806070943.95542-2-jiahao.kernel@gmail.com Fixes: a65b0e7607cc ("zswap: make shrinking memcg-aware") Signed-off-by: Hao Jia Suggested-by: Nhat Pham Acked-by: Nhat Pham Acked-by: Yosry Ahmed Reported-by: Yosry Ahmed Cc: Chengming Zhou Cc: Johannes Weiner Cc: Michal Hocko Cc: Michal Koutný Cc: Muchun Song Cc: Roman Gushchin Cc: Shakeel Butt Cc: Tejun Heo Cc: Signed-off-by: Andrew Morton --- mm/zswap.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index a810524c7621..cf824e2a6b38 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1356,11 +1356,12 @@ static void shrink_worker(struct work_struct *w) } while (memcg && !mem_cgroup_tryget_online(memcg)); spin_unlock(&zswap_shrink_lock); - if (!memcg) { - /* - * Continue shrinking without incrementing failures if - * we found candidate memcgs in the last tree walk. - */ + /* + * A NULL memcg ends a full hierarchy pass (except when memcg is + * disabled, where it is always NULL: fall through to the root LRU). + * Count a failure only if the last pass found no candidates. + */ + if (!memcg && !mem_cgroup_disabled()) { if (!attempts && ++failures == MAX_RECLAIM_RETRIES) break; @@ -1379,7 +1380,7 @@ static void shrink_worker(struct work_struct *w) * and failures. */ if (ret == -ENOENT) - continue; + goto resched; ++attempts; if (ret && ++failures == MAX_RECLAIM_RETRIES) -- cgit v1.2.3 From 6f34b4126b8bb7b24c2d6b6541d51d9655287fb4 Mon Sep 17 00:00:00 2001 From: Hao Jia Date: Thu, 6 Aug 2026 15:09:43 +0800 Subject: mm/zswap: support batch writeback in shrink_memcg() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Currently, shrink_memcg() writes back at most one entry per-node during its traversal. This makes shrink_worker() inefficient, as it must repeatedly re-enter shrink_memcg() to make any substantial progress. Under high memory pressure, this can cause the writeback speed to be too slow to keep up with refaults, leading to zswap store failures and forcing pages to skip zswap and go directly to disk, which results in an LRU inversion. To address this, extend the per-node scan budget in shrink_memcg() from a single entry to up to SWAP_CLUSTER_MAX pages, enabling batch writeback for both the shrink_worker() and zswap_store() paths. Test Setup: - Total memory: 32 GB, 1 NUMA node. - zswap settings: accept_threshold_percent=50, shrinker_enabled=N. Test Case 1: Set max_pool_percent=1, allocate 512MB of anonymous pages, and fill them with random data (to avoid compression). Then, use cgroup memory.reclaim to force a large amount of anonymous pages into zswap. At an interval of 2ms, allocate a 4K anonymous page where the first 4 bytes are random numbers and the rest are zeros, and then trigger reclamation of this 4K page through cgroup memory.reclaim. When the pool threshold is reached, shrink_memcg() will be triggered. The test data after running for 120s is as follows: Baseline Patched shrink_worker wakeups 5,363 169 shrink_memcg calls 11,373,201 350,703 written_back pages 40,212 40,241 zswap_store calls 161,190 163,753 store succeeded (ret=1) 102,743 117,183 store rejected (ret=0) 58,447 46,570 store reject rate ~36% ~28% pool_limit_hit delta 55,826 33,760 pswpout 98,659 86,811 pswpin 2 0 Test Case 2: We evaluated the following two sub-configurations using stress-ng inside a cgroup capped at memory.max=1G for 120 seconds: Test Case 2a (max_pool_percent=1): Continuously triggers the global zswap pool limit, thereby waking up shrink_worker() to perform asynchronous shrinking. Test Case 2b (zswap.max=320M, max_pool_percent=50): Continuously triggers the cgroup's zswap.max limit, thereby invoking synchronous shrinking. Command executed for both setups: bash -c 'echo $$ > /sys/fs/cgroup/zswaptest/cgroup.procs ; \ exec stress-ng --vm 4 --vm-bytes 4G --vm-keep --vm-method rand-set -t \ 120s -q' Test Case 2a (max_pool_percent=1): Baseline Patched shrink_worker wakeups 5,640 1,308 shrink_memcg calls 8,481,500 3,140,972 written_back pages 260 468,216 zswap_store calls 2,742,756 2,011,269 store succeeded (ret=1) 934,640 947,988 store rejected (ret=0) 1,808,116 1,063,281 store reject rate ~66% ~52% pool_limit_hit delta 1,181,310 196,882 pswpout 1,808,376 1,531,497 pswpin 4,288,497 3,635,365 Test Case 2b (zswap.max=320M, max_pool_percent=50): Baseline Patched shrink_worker wakeups 0 0 shrink_memcg calls 687,608 54,002 written_back pages 639,176 846,663 zswap_store calls 1,224,222 1,228,548 store succeeded (ret=1) 992,816 1,208,123 store rejected (ret=0) 231,431 20,425 store reject rate ~19% ~2% pool_limit_hit delta 0 0 pswpout 870,745 867,360 pswpin 1,707,823 1,216,814 Under identical workloads and runtimes, batched zswap shrinking exhibits a significant reduction in both shrink_worker() wakeups and shrink_memcg() calls. Furthermore, the sharp drop in both pswpin and zswap_store() rejections demonstrates that batching zswap shrink operations effectively mitigates zswap_store() failures caused by hitting the pool limit. This significantly prevents pages from bypassing zswap and falling back directly to disk, thereby reducing LRU inversion. Link: https://lore.kernel.org/20260806070943.95542-3-jiahao.kernel@gmail.com Signed-off-by: Hao Jia Suggested-by: Yosry Ahmed Suggested-by: Johannes Weiner Acked-by: Yosry Ahmed Acked-by: Nhat Pham Cc: Chengming Zhou Cc: Michal Hocko Cc: Michal Koutný Cc: Muchun Song Cc: Roman Gushchin Cc: Shakeel Butt Cc: Tejun Heo Signed-off-by: Andrew Morton --- mm/zswap.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index cf824e2a6b38..0b9435b4f57c 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1275,6 +1275,14 @@ static struct shrinker *zswap_alloc_shrinker(void) return shrinker; } +/* + * Scan up to SWAP_CLUSTER_MAX pages on each per-node zswap LRU of @memcg + * and write back the reclaimable ones. + * + * Return: 0 if at least one entry was written back, -EAGAIN if entries + * were scanned but none could be written back, or -ENOENT if @memcg has + * writeback disabled, is a zombie cgroup, or has empty zswap LRUs. + */ static int shrink_memcg(struct mem_cgroup *memcg) { int nid, shrunk = 0, scanned = 0; @@ -1290,13 +1298,14 @@ static int shrink_memcg(struct mem_cgroup *memcg) return -ENOENT; for_each_node_state(nid, N_NORMAL_MEMORY) { - unsigned long nr_to_walk = 1; + unsigned long nr_to_walk = SWAP_CLUSTER_MAX; shrunk += list_lru_walk_one(&zswap_list_lru, nid, memcg, &shrink_memcg_cb, NULL, &nr_to_walk); - scanned += 1 - nr_to_walk; + scanned += SWAP_CLUSTER_MAX - nr_to_walk; } + /* Nothing was scanned: every LRU under @memcg was empty. */ if (!scanned) return -ENOENT; -- cgit v1.2.3 From 3774c56cc38b9ddcdf46dd717d4de954b40d8986 Mon Sep 17 00:00:00 2001 From: "Mike Rapoport (Microsoft)" Date: Thu, 6 Aug 2026 09:58:38 +0300 Subject: drivers/base, mm: move arch_numa.c to mm/ arch_numa.c implements boot time discovery and initialization of NUMA topology on architectures that select GENERIC_ARCH_NUMA (currently arm64 and riscv). Since this is step in the initialization of the memory management subsystem, it's logical to have arch_numa.c in mm/ alongside numa.c, numa_memblks.c and numa_emulation.c. Move arch_numa.c to mm/ and add its F: entry to "MEMBLOCK AND MEMORY MANAGEMENT INITIALIZATION" in MAINTAINERS. Link: https://lore.kernel.org/20260806-arch-numa-v1-1-968ec128121e@kernel.org Signed-off-by: Mike Rapoport (Microsoft) Acked-by: Greg Kroah-Hartman Acked-by: Lorenzo Stoakes (ARM) Acked-by: Vlastimil Babka (SUSE) Acked-by: David Hildenbrand (Arm) Acked-by: Danilo Krummrich Cc: Albert Ou Cc: Alexandre Ghiti Cc: Catalin Marinas Cc: Liam R. Howlett Cc: Michal Hocko Cc: Palmer Dabbelt Cc: "Rafael J. Wysocki" Cc: Suren Baghdasaryan Cc: Will Deacon Signed-off-by: Andrew Morton --- MAINTAINERS | 1 + drivers/base/Kconfig | 7 - drivers/base/Makefile | 1 - drivers/base/arch_numa.c | 375 ----------------------------------------------- mm/Kconfig | 7 + mm/Makefile | 1 + mm/arch_numa.c | 375 +++++++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 384 insertions(+), 383 deletions(-) delete mode 100644 drivers/base/arch_numa.c create mode 100644 mm/arch_numa.c diff --git a/MAINTAINERS b/MAINTAINERS index e0ea1b915305..06271e742d32 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -16880,6 +16880,7 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/rppt/memblock.git fixes F: Documentation/core-api/boot-time-mm.rst F: include/linux/kho/abi/memblock.h F: include/linux/memblock.h +F: mm/arch_numa.c F: mm/memblock.c F: mm/memtest.c F: mm/mm_init.c diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig index f7d385cbd3ba..1d93bfc34490 100644 --- a/drivers/base/Kconfig +++ b/drivers/base/Kconfig @@ -239,13 +239,6 @@ config GENERIC_ARCH_TOPOLOGY appropriate scaling, sysfs interface for reading capacity values at runtime. -config GENERIC_ARCH_NUMA - bool - select NUMA_MEMBLKS - help - Enable support for generic NUMA implementation. Currently, RISC-V - and ARM64 use it. - config FW_DEVLINK_SYNC_STATE_TIMEOUT bool "sync_state() behavior defaults to timeout instead of strict" help diff --git a/drivers/base/Makefile b/drivers/base/Makefile index 8074a10183dc..435710f643a5 100644 --- a/drivers/base/Makefile +++ b/drivers/base/Makefile @@ -25,7 +25,6 @@ obj-$(CONFIG_PINCTRL) += pinctrl.o obj-$(CONFIG_DEV_COREDUMP) += devcoredump.o obj-$(CONFIG_GENERIC_MSI_IRQ) += platform-msi.o obj-$(CONFIG_GENERIC_ARCH_TOPOLOGY) += arch_topology.o -obj-$(CONFIG_GENERIC_ARCH_NUMA) += arch_numa.o obj-$(CONFIG_ACPI) += physical_location.o obj-y += test/ diff --git a/drivers/base/arch_numa.c b/drivers/base/arch_numa.c deleted file mode 100644 index 442ea239bba7..000000000000 --- a/drivers/base/arch_numa.c +++ /dev/null @@ -1,375 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -/* - * NUMA support, based on the x86 implementation. - * - * Copyright (C) 2015 Cavium Inc. - * Author: Ganapatrao Kulkarni - */ - -#define pr_fmt(fmt) "NUMA: " fmt - -#include -#include -#include -#include -#include - -#include - -static int cpu_to_node_map[NR_CPUS] = { [0 ... NR_CPUS-1] = NUMA_NO_NODE }; - -bool numa_off; - -static __init int numa_parse_early_param(char *opt) -{ - if (!opt) - return -EINVAL; - if (str_has_prefix(opt, "off")) - numa_off = true; - if (!strncmp(opt, "fake=", 5)) - return numa_emu_cmdline(opt + 5); - - return 0; -} -early_param("numa", numa_parse_early_param); - -cpumask_var_t node_to_cpumask_map[MAX_NUMNODES]; -EXPORT_SYMBOL(node_to_cpumask_map); - -#ifdef CONFIG_DEBUG_PER_CPU_MAPS - -/* - * Returns a pointer to the bitmask of CPUs on Node 'node'. - */ -const struct cpumask *cpumask_of_node(int node) -{ - - if (node == NUMA_NO_NODE) - return cpu_all_mask; - - if (WARN_ON(node < 0 || node >= nr_node_ids)) - return cpu_none_mask; - - if (WARN_ON(node_to_cpumask_map[node] == NULL)) - return cpu_online_mask; - - return node_to_cpumask_map[node]; -} -EXPORT_SYMBOL(cpumask_of_node); - -#endif - -#ifndef CONFIG_NUMA_EMU -static void numa_update_cpu(unsigned int cpu, bool remove) -{ - int nid = cpu_to_node(cpu); - - if (nid == NUMA_NO_NODE) - return; - - if (remove) - cpumask_clear_cpu(cpu, node_to_cpumask_map[nid]); - else - cpumask_set_cpu(cpu, node_to_cpumask_map[nid]); -} - -void numa_add_cpu(unsigned int cpu) -{ - numa_update_cpu(cpu, false); -} - -void numa_remove_cpu(unsigned int cpu) -{ - numa_update_cpu(cpu, true); -} -#endif - -void numa_clear_node(unsigned int cpu) -{ - numa_remove_cpu(cpu); - set_cpu_numa_node(cpu, NUMA_NO_NODE); -} - -/* - * Allocate node_to_cpumask_map based on number of available nodes - * Requires node_possible_map to be valid. - * - * Note: cpumask_of_node() is not valid until after this is done. - * (Use CONFIG_DEBUG_PER_CPU_MAPS to check this.) - */ -static void __init setup_node_to_cpumask_map(void) -{ - int node; - - /* setup nr_node_ids if not done yet */ - if (nr_node_ids == MAX_NUMNODES) - setup_nr_node_ids(); - - /* allocate and clear the mapping */ - for (node = 0; node < nr_node_ids; node++) { - alloc_bootmem_cpumask_var(&node_to_cpumask_map[node]); - cpumask_clear(node_to_cpumask_map[node]); - } - - /* cpumask_of_node() will now work */ - pr_debug("Node to cpumask map for %u nodes\n", nr_node_ids); -} - -/* - * Set the cpu to node and mem mapping - */ -void numa_store_cpu_info(unsigned int cpu) -{ - set_cpu_numa_node(cpu, cpu_to_node_map[cpu]); -} - -void __init early_map_cpu_to_node(unsigned int cpu, int nid) -{ - /* fallback to node 0 */ - if (nid < 0 || nid >= MAX_NUMNODES || numa_off) - nid = 0; - - cpu_to_node_map[cpu] = nid; - - /* - * We should set the numa node of cpu0 as soon as possible, because it - * has already been set up online before. cpu_to_node(0) will soon be - * called. - */ - if (!cpu) - set_cpu_numa_node(cpu, nid); -} - -#ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA -unsigned long __per_cpu_offset[NR_CPUS] __read_mostly; -EXPORT_SYMBOL(__per_cpu_offset); - -int early_cpu_to_node(int cpu) -{ - return cpu_to_node_map[cpu]; -} - -static int __init pcpu_cpu_distance(unsigned int from, unsigned int to) -{ - return node_distance(early_cpu_to_node(from), early_cpu_to_node(to)); -} - -void __init setup_per_cpu_areas(void) -{ - unsigned long delta; - unsigned int cpu; - int rc = -EINVAL; - - if (pcpu_chosen_fc != PCPU_FC_PAGE) { - /* - * Always reserve area for module percpu variables. That's - * what the legacy allocator did. - */ - rc = pcpu_embed_first_chunk(PERCPU_MODULE_RESERVE, - PERCPU_DYNAMIC_RESERVE, PAGE_SIZE, - pcpu_cpu_distance, - early_cpu_to_node); -#ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK - if (rc < 0) - pr_warn("PERCPU: %s allocator failed (%d), falling back to page size\n", - pcpu_fc_names[pcpu_chosen_fc], rc); -#endif - } - -#ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK - if (rc < 0) - rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, early_cpu_to_node); -#endif - if (rc < 0) - panic("Failed to initialize percpu areas (err=%d).", rc); - - delta = (unsigned long)pcpu_base_addr - (unsigned long)__per_cpu_start; - for_each_possible_cpu(cpu) - __per_cpu_offset[cpu] = delta + pcpu_unit_offsets[cpu]; -} -#endif - -/* - * Initialize NODE_DATA for a node on the local memory - */ -static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn) -{ - if (start_pfn >= end_pfn) - pr_info("Initmem setup node %d []\n", nid); - - alloc_node_data(nid); - - NODE_DATA(nid)->node_id = nid; - NODE_DATA(nid)->node_start_pfn = start_pfn; - NODE_DATA(nid)->node_spanned_pages = end_pfn - start_pfn; -} - -static int __init numa_register_nodes(void) -{ - int nid; - - /* Check the validity of the memblock/node mapping */ - if (!memblock_validate_numa_coverage(0)) - return -EINVAL; - - /* Finally register nodes. */ - for_each_node_mask(nid, numa_nodes_parsed) { - unsigned long start_pfn, end_pfn; - - get_pfn_range_for_nid(nid, &start_pfn, &end_pfn); - setup_node_data(nid, start_pfn, end_pfn); - node_set_online(nid); - } - - /* Setup online nodes to actual nodes*/ - node_possible_map = numa_nodes_parsed; - - return 0; -} - -static int __init numa_init(int (*init_func)(void)) -{ - int ret; - - ret = numa_memblks_init(init_func, /* memblock_force_top_down */ false); - if (ret < 0) - goto out_free_distance; - - if (nodes_empty(numa_nodes_parsed)) { - pr_info("No NUMA configuration found\n"); - ret = -EINVAL; - goto out_free_distance; - } - - ret = numa_register_nodes(); - if (ret < 0) - goto out_free_distance; - - setup_node_to_cpumask_map(); - - return 0; -out_free_distance: - numa_reset_distance(); - return ret; -} - -/** - * dummy_numa_init() - Fallback dummy NUMA init - * - * Used if there's no underlying NUMA architecture, NUMA initialization - * fails, or NUMA is disabled on the command line. - * - * Must online at least one node (node 0) and add memory blocks that cover all - * allowed memory. It is unlikely that this function fails. - * - * Return: 0 on success, -errno on failure. - */ -static int __init dummy_numa_init(void) -{ - phys_addr_t start = memblock_start_of_DRAM(); - phys_addr_t end = memblock_end_of_DRAM() - 1; - int ret; - - if (numa_off) - pr_info("NUMA disabled\n"); /* Forced off on command line. */ - pr_info("Faking a node at [mem %pap-%pap]\n", &start, &end); - - ret = numa_add_memblk(0, start, end + 1); - if (ret) { - pr_err("NUMA init failed\n"); - return ret; - } - node_set(0, numa_nodes_parsed); - - numa_off = true; - return 0; -} - -#ifdef CONFIG_ACPI_NUMA -static int __init arch_acpi_numa_init(void) -{ - int ret; - - ret = acpi_numa_init(); - if (ret) { - pr_debug("Failed to initialise from firmware\n"); - return ret; - } - - return srat_disabled() ? -EINVAL : 0; -} -#else -static int __init arch_acpi_numa_init(void) -{ - return -EOPNOTSUPP; -} -#endif - -/** - * arch_numa_init() - Initialize NUMA - * - * Try each configured NUMA initialization method until one succeeds. The - * last fallback is dummy single node config encompassing whole memory. - */ -void __init arch_numa_init(void) -{ - if (!numa_off) { - if (!acpi_disabled && !numa_init(arch_acpi_numa_init)) - return; - if (acpi_disabled && !numa_init(of_numa_init)) - return; - } - - numa_init(dummy_numa_init); -} - -#ifdef CONFIG_NUMA_EMU -void __init numa_emu_update_cpu_to_node(int *emu_nid_to_phys, - unsigned int nr_emu_nids) -{ - int i, j; - - /* - * Transform cpu_to_node_map table to use emulated nids by - * reverse-mapping phys_nid. The maps should always exist but fall - * back to zero just in case. - */ - for (i = 0; i < ARRAY_SIZE(cpu_to_node_map); i++) { - if (cpu_to_node_map[i] == NUMA_NO_NODE) - continue; - for (j = 0; j < nr_emu_nids; j++) - if (cpu_to_node_map[i] == emu_nid_to_phys[j]) - break; - cpu_to_node_map[i] = j < nr_emu_nids ? j : 0; - } -} - -u64 __init numa_emu_dma_end(void) -{ - return memblock_start_of_DRAM() + SZ_4G; -} - -void debug_cpumask_set_cpu(unsigned int cpu, int node, bool enable) -{ - struct cpumask *mask; - - if (node == NUMA_NO_NODE) - return; - - mask = node_to_cpumask_map[node]; - if (!cpumask_available(mask)) { - pr_err("node_to_cpumask_map[%i] NULL\n", node); - dump_stack(); - return; - } - - if (enable) - cpumask_set_cpu(cpu, mask); - else - cpumask_clear_cpu(cpu, mask); - - pr_debug("%s cpu %d node %d: mask now %*pbl\n", - enable ? "numa_add_cpu" : "numa_remove_cpu", - cpu, node, cpumask_pr_args(mask)); -} -#endif /* CONFIG_NUMA_EMU */ diff --git a/mm/Kconfig b/mm/Kconfig index 331daf7fcfab..8a24c130d008 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1463,6 +1463,13 @@ config NUMA_EMU into virtual nodes when booted with "numa=fake=N", where N is the number of nodes. This is only useful for debugging. +config GENERIC_ARCH_NUMA + bool + select NUMA_MEMBLKS + help + Enable support for generic NUMA implementation. Currently, RISC-V + and ARM64 use it. + config ARCH_HAS_USER_SHADOW_STACK bool help diff --git a/mm/Makefile b/mm/Makefile index ab37ef428d98..e7245cb88c66 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -122,6 +122,7 @@ obj-$(CONFIG_CMA) += cma.o obj-$(CONFIG_NUMA) += numa.o obj-$(CONFIG_NUMA_MEMBLKS) += numa_memblks.o obj-$(CONFIG_NUMA_EMU) += numa_emulation.o +obj-$(CONFIG_GENERIC_ARCH_NUMA) += arch_numa.o obj-$(CONFIG_BALLOON) += balloon.o obj-$(CONFIG_PAGE_EXTENSION) += page_ext.o obj-$(CONFIG_PAGE_TABLE_CHECK) += page_table_check.o diff --git a/mm/arch_numa.c b/mm/arch_numa.c new file mode 100644 index 000000000000..442ea239bba7 --- /dev/null +++ b/mm/arch_numa.c @@ -0,0 +1,375 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * NUMA support, based on the x86 implementation. + * + * Copyright (C) 2015 Cavium Inc. + * Author: Ganapatrao Kulkarni + */ + +#define pr_fmt(fmt) "NUMA: " fmt + +#include +#include +#include +#include +#include + +#include + +static int cpu_to_node_map[NR_CPUS] = { [0 ... NR_CPUS-1] = NUMA_NO_NODE }; + +bool numa_off; + +static __init int numa_parse_early_param(char *opt) +{ + if (!opt) + return -EINVAL; + if (str_has_prefix(opt, "off")) + numa_off = true; + if (!strncmp(opt, "fake=", 5)) + return numa_emu_cmdline(opt + 5); + + return 0; +} +early_param("numa", numa_parse_early_param); + +cpumask_var_t node_to_cpumask_map[MAX_NUMNODES]; +EXPORT_SYMBOL(node_to_cpumask_map); + +#ifdef CONFIG_DEBUG_PER_CPU_MAPS + +/* + * Returns a pointer to the bitmask of CPUs on Node 'node'. + */ +const struct cpumask *cpumask_of_node(int node) +{ + + if (node == NUMA_NO_NODE) + return cpu_all_mask; + + if (WARN_ON(node < 0 || node >= nr_node_ids)) + return cpu_none_mask; + + if (WARN_ON(node_to_cpumask_map[node] == NULL)) + return cpu_online_mask; + + return node_to_cpumask_map[node]; +} +EXPORT_SYMBOL(cpumask_of_node); + +#endif + +#ifndef CONFIG_NUMA_EMU +static void numa_update_cpu(unsigned int cpu, bool remove) +{ + int nid = cpu_to_node(cpu); + + if (nid == NUMA_NO_NODE) + return; + + if (remove) + cpumask_clear_cpu(cpu, node_to_cpumask_map[nid]); + else + cpumask_set_cpu(cpu, node_to_cpumask_map[nid]); +} + +void numa_add_cpu(unsigned int cpu) +{ + numa_update_cpu(cpu, false); +} + +void numa_remove_cpu(unsigned int cpu) +{ + numa_update_cpu(cpu, true); +} +#endif + +void numa_clear_node(unsigned int cpu) +{ + numa_remove_cpu(cpu); + set_cpu_numa_node(cpu, NUMA_NO_NODE); +} + +/* + * Allocate node_to_cpumask_map based on number of available nodes + * Requires node_possible_map to be valid. + * + * Note: cpumask_of_node() is not valid until after this is done. + * (Use CONFIG_DEBUG_PER_CPU_MAPS to check this.) + */ +static void __init setup_node_to_cpumask_map(void) +{ + int node; + + /* setup nr_node_ids if not done yet */ + if (nr_node_ids == MAX_NUMNODES) + setup_nr_node_ids(); + + /* allocate and clear the mapping */ + for (node = 0; node < nr_node_ids; node++) { + alloc_bootmem_cpumask_var(&node_to_cpumask_map[node]); + cpumask_clear(node_to_cpumask_map[node]); + } + + /* cpumask_of_node() will now work */ + pr_debug("Node to cpumask map for %u nodes\n", nr_node_ids); +} + +/* + * Set the cpu to node and mem mapping + */ +void numa_store_cpu_info(unsigned int cpu) +{ + set_cpu_numa_node(cpu, cpu_to_node_map[cpu]); +} + +void __init early_map_cpu_to_node(unsigned int cpu, int nid) +{ + /* fallback to node 0 */ + if (nid < 0 || nid >= MAX_NUMNODES || numa_off) + nid = 0; + + cpu_to_node_map[cpu] = nid; + + /* + * We should set the numa node of cpu0 as soon as possible, because it + * has already been set up online before. cpu_to_node(0) will soon be + * called. + */ + if (!cpu) + set_cpu_numa_node(cpu, nid); +} + +#ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA +unsigned long __per_cpu_offset[NR_CPUS] __read_mostly; +EXPORT_SYMBOL(__per_cpu_offset); + +int early_cpu_to_node(int cpu) +{ + return cpu_to_node_map[cpu]; +} + +static int __init pcpu_cpu_distance(unsigned int from, unsigned int to) +{ + return node_distance(early_cpu_to_node(from), early_cpu_to_node(to)); +} + +void __init setup_per_cpu_areas(void) +{ + unsigned long delta; + unsigned int cpu; + int rc = -EINVAL; + + if (pcpu_chosen_fc != PCPU_FC_PAGE) { + /* + * Always reserve area for module percpu variables. That's + * what the legacy allocator did. + */ + rc = pcpu_embed_first_chunk(PERCPU_MODULE_RESERVE, + PERCPU_DYNAMIC_RESERVE, PAGE_SIZE, + pcpu_cpu_distance, + early_cpu_to_node); +#ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK + if (rc < 0) + pr_warn("PERCPU: %s allocator failed (%d), falling back to page size\n", + pcpu_fc_names[pcpu_chosen_fc], rc); +#endif + } + +#ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK + if (rc < 0) + rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, early_cpu_to_node); +#endif + if (rc < 0) + panic("Failed to initialize percpu areas (err=%d).", rc); + + delta = (unsigned long)pcpu_base_addr - (unsigned long)__per_cpu_start; + for_each_possible_cpu(cpu) + __per_cpu_offset[cpu] = delta + pcpu_unit_offsets[cpu]; +} +#endif + +/* + * Initialize NODE_DATA for a node on the local memory + */ +static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn) +{ + if (start_pfn >= end_pfn) + pr_info("Initmem setup node %d []\n", nid); + + alloc_node_data(nid); + + NODE_DATA(nid)->node_id = nid; + NODE_DATA(nid)->node_start_pfn = start_pfn; + NODE_DATA(nid)->node_spanned_pages = end_pfn - start_pfn; +} + +static int __init numa_register_nodes(void) +{ + int nid; + + /* Check the validity of the memblock/node mapping */ + if (!memblock_validate_numa_coverage(0)) + return -EINVAL; + + /* Finally register nodes. */ + for_each_node_mask(nid, numa_nodes_parsed) { + unsigned long start_pfn, end_pfn; + + get_pfn_range_for_nid(nid, &start_pfn, &end_pfn); + setup_node_data(nid, start_pfn, end_pfn); + node_set_online(nid); + } + + /* Setup online nodes to actual nodes*/ + node_possible_map = numa_nodes_parsed; + + return 0; +} + +static int __init numa_init(int (*init_func)(void)) +{ + int ret; + + ret = numa_memblks_init(init_func, /* memblock_force_top_down */ false); + if (ret < 0) + goto out_free_distance; + + if (nodes_empty(numa_nodes_parsed)) { + pr_info("No NUMA configuration found\n"); + ret = -EINVAL; + goto out_free_distance; + } + + ret = numa_register_nodes(); + if (ret < 0) + goto out_free_distance; + + setup_node_to_cpumask_map(); + + return 0; +out_free_distance: + numa_reset_distance(); + return ret; +} + +/** + * dummy_numa_init() - Fallback dummy NUMA init + * + * Used if there's no underlying NUMA architecture, NUMA initialization + * fails, or NUMA is disabled on the command line. + * + * Must online at least one node (node 0) and add memory blocks that cover all + * allowed memory. It is unlikely that this function fails. + * + * Return: 0 on success, -errno on failure. + */ +static int __init dummy_numa_init(void) +{ + phys_addr_t start = memblock_start_of_DRAM(); + phys_addr_t end = memblock_end_of_DRAM() - 1; + int ret; + + if (numa_off) + pr_info("NUMA disabled\n"); /* Forced off on command line. */ + pr_info("Faking a node at [mem %pap-%pap]\n", &start, &end); + + ret = numa_add_memblk(0, start, end + 1); + if (ret) { + pr_err("NUMA init failed\n"); + return ret; + } + node_set(0, numa_nodes_parsed); + + numa_off = true; + return 0; +} + +#ifdef CONFIG_ACPI_NUMA +static int __init arch_acpi_numa_init(void) +{ + int ret; + + ret = acpi_numa_init(); + if (ret) { + pr_debug("Failed to initialise from firmware\n"); + return ret; + } + + return srat_disabled() ? -EINVAL : 0; +} +#else +static int __init arch_acpi_numa_init(void) +{ + return -EOPNOTSUPP; +} +#endif + +/** + * arch_numa_init() - Initialize NUMA + * + * Try each configured NUMA initialization method until one succeeds. The + * last fallback is dummy single node config encompassing whole memory. + */ +void __init arch_numa_init(void) +{ + if (!numa_off) { + if (!acpi_disabled && !numa_init(arch_acpi_numa_init)) + return; + if (acpi_disabled && !numa_init(of_numa_init)) + return; + } + + numa_init(dummy_numa_init); +} + +#ifdef CONFIG_NUMA_EMU +void __init numa_emu_update_cpu_to_node(int *emu_nid_to_phys, + unsigned int nr_emu_nids) +{ + int i, j; + + /* + * Transform cpu_to_node_map table to use emulated nids by + * reverse-mapping phys_nid. The maps should always exist but fall + * back to zero just in case. + */ + for (i = 0; i < ARRAY_SIZE(cpu_to_node_map); i++) { + if (cpu_to_node_map[i] == NUMA_NO_NODE) + continue; + for (j = 0; j < nr_emu_nids; j++) + if (cpu_to_node_map[i] == emu_nid_to_phys[j]) + break; + cpu_to_node_map[i] = j < nr_emu_nids ? j : 0; + } +} + +u64 __init numa_emu_dma_end(void) +{ + return memblock_start_of_DRAM() + SZ_4G; +} + +void debug_cpumask_set_cpu(unsigned int cpu, int node, bool enable) +{ + struct cpumask *mask; + + if (node == NUMA_NO_NODE) + return; + + mask = node_to_cpumask_map[node]; + if (!cpumask_available(mask)) { + pr_err("node_to_cpumask_map[%i] NULL\n", node); + dump_stack(); + return; + } + + if (enable) + cpumask_set_cpu(cpu, mask); + else + cpumask_clear_cpu(cpu, mask); + + pr_debug("%s cpu %d node %d: mask now %*pbl\n", + enable ? "numa_add_cpu" : "numa_remove_cpu", + cpu, node, cpumask_pr_args(mask)); +} +#endif /* CONFIG_NUMA_EMU */ -- cgit v1.2.3 From 28b13c3c4c6592da8dd59a7b06655e39665ee624 Mon Sep 17 00:00:00 2001 From: Bart Van Assche Date: Fri, 7 Aug 2026 10:13:10 -0700 Subject: mm: make VM_FAULT_RESULT_TRACE compatible with sparse Fix the following sparse warnings that appear while building f2fs: ./include/trace/events/f2fs.h:1469:1: warning: incorrect type in initializer (different base types) ./include/trace/events/f2fs.h:1469:1: expected unsigned long mask ./include/trace/events/f2fs.h:1469:1: got restricted vm_fault_t Link: https://lore.kernel.org/e56c9e2aead04f79192c3110de80d846e41e3791.1786122711.git.bvanassche@acm.org Signed-off-by: Bart Van Assche Acked-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Reviewed-by: Anshuman Khandual Signed-off-by: Andrew Morton --- include/linux/mm_types.h | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index ebf0d912be7d..6d815f6440c9 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1718,20 +1718,20 @@ enum vm_fault_reason { VM_FAULT_SIGSEGV | VM_FAULT_HWPOISON | \ VM_FAULT_HWPOISON_LARGE | VM_FAULT_FALLBACK) -#define VM_FAULT_RESULT_TRACE \ - { VM_FAULT_OOM, "OOM" }, \ - { VM_FAULT_SIGBUS, "SIGBUS" }, \ - { VM_FAULT_MAJOR, "MAJOR" }, \ - { VM_FAULT_HWPOISON, "HWPOISON" }, \ - { VM_FAULT_HWPOISON_LARGE, "HWPOISON_LARGE" }, \ - { VM_FAULT_SIGSEGV, "SIGSEGV" }, \ - { VM_FAULT_NOPAGE, "NOPAGE" }, \ - { VM_FAULT_LOCKED, "LOCKED" }, \ - { VM_FAULT_RETRY, "RETRY" }, \ - { VM_FAULT_FALLBACK, "FALLBACK" }, \ - { VM_FAULT_DONE_COW, "DONE_COW" }, \ - { VM_FAULT_NEEDDSYNC, "NEEDDSYNC" }, \ - { VM_FAULT_COMPLETED, "COMPLETED" } +#define VM_FAULT_RESULT_TRACE \ + { (__force u32)VM_FAULT_OOM, "OOM" }, \ + { (__force u32)VM_FAULT_SIGBUS, "SIGBUS" }, \ + { (__force u32)VM_FAULT_MAJOR, "MAJOR" }, \ + { (__force u32)VM_FAULT_HWPOISON, "HWPOISON" }, \ + { (__force u32)VM_FAULT_HWPOISON_LARGE, "HWPOISON_LARGE" }, \ + { (__force u32)VM_FAULT_SIGSEGV, "SIGSEGV" }, \ + { (__force u32)VM_FAULT_NOPAGE, "NOPAGE" }, \ + { (__force u32)VM_FAULT_LOCKED, "LOCKED" }, \ + { (__force u32)VM_FAULT_RETRY, "RETRY" }, \ + { (__force u32)VM_FAULT_FALLBACK, "FALLBACK" }, \ + { (__force u32)VM_FAULT_DONE_COW, "DONE_COW" }, \ + { (__force u32)VM_FAULT_NEEDDSYNC, "NEEDDSYNC" }, \ + { (__force u32)VM_FAULT_COMPLETED, "COMPLETED" } struct vm_special_mapping { const char *name; /* The name, e.g. "[vdso]". */ -- cgit v1.2.3 From 34e0849142c317eed68f3a4818dd3808fb1c47db Mon Sep 17 00:00:00 2001 From: Sourav Panda Date: Fri, 7 Aug 2026 04:00:03 +0000 Subject: mm/hugetlb_cma: support percentage-based hugetlb_cma reservation Currently, hugetlb_cma reservation only supports absolute sizes (e.g., hugetlb_cma=2G or hugetlb_cma=0:1G,1:1G). This can be restrictive in heterogeneous environments or when deploying common kernel command lines across machines with different memory capacities. Add support for percentage-based hugetlb_cma reservation (e.g., hugetlb_cma=20% or hugetlb_cma=0:20%,1:10%). The percentage is calculated against the total memory (for global settings) or against the node-specific memory (for node-specific settings) using memblock APIs during early boot. Link: https://lore.kernel.org/20260807040003.2156630-1-souravpanda@google.com Signed-off-by: Sourav Panda Acked-by: Usama Arif Cc: David Hildenbrand Cc: David Rientjes Cc: Frank van der Linden Cc: Greg Thelen Cc: Muchun Song Cc: Oscar Salvador Cc: Suren Baghdasaryan Signed-off-by: Andrew Morton --- Documentation/admin-guide/kernel-parameters.txt | 10 +- mm/hugetlb_cma.c | 142 ++++++++++++++++++++++-- 2 files changed, 142 insertions(+), 10 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 364c2dce8e70..1af62cd16c9d 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2064,8 +2064,14 @@ Kernel parameters hugetlb_cma= [HW,CMA,EARLY] The size of a CMA area used for allocation of gigantic hugepages. Or using node format, the size of a CMA area per node can be specified. - Format: nn[KMGTPE] or (node format) - :nn[KMGTPE][,:nn[KMGTPE]] + The size can be an absolute value (e.g., 2G) or a + percentage of the total memory or node memory (e.g., 20%). + Percentage-derived sizes are rounded down to a multiple of + the architecture's gigantic hugepage size and may become + zero. + Format: nn[KMGTPE] or nn% or (node format) + :nn[KMGTPE][,:nn[KMGTPE]] or + :nn%[,:nn%] The size must be a multiple of the gigantic page size. When using node format, this applies to each per-node size. diff --git a/mm/hugetlb_cma.c b/mm/hugetlb_cma.c index 4dfce68b354a..db0680e82847 100644 --- a/mm/hugetlb_cma.c +++ b/mm/hugetlb_cma.c @@ -9,6 +9,9 @@ #include #include +#include +#include +#include #include "internal.h" #include "hugetlb_cma.h" @@ -18,6 +21,28 @@ static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES] __initdata; static bool hugetlb_cma_only __ro_after_init; static unsigned long hugetlb_cma_size __ro_after_init; +static unsigned int hugetlb_cma_percent __initdata; +static unsigned int hugetlb_cma_percent_in_node[MAX_NUMNODES] __initdata; + +#ifdef CONFIG_NUMA +static phys_addr_t __init memblock_node_memory_size(int nid) +{ + struct memblock_region *reg; + phys_addr_t size = 0; + + for_each_mem_region(reg) { + if (reg->nid == nid) + size += reg->size; + } + return size; +} +#else +static phys_addr_t __init memblock_node_memory_size(int nid) +{ + return memblock_phys_mem_size(); +} +#endif + void hugetlb_cma_free_frozen_folio(struct folio *folio) { WARN_ON_ONCE(!cma_release_frozen(hugetlb_cma[folio_nid(folio)], @@ -90,14 +115,31 @@ static int __init cmdline_parse_hugetlb_cma(char *p) break; if (s[count] == ':') { + char *next; + if (tmp >= MAX_NUMNODES) break; nid = array_index_nospec(tmp, MAX_NUMNODES); + hugetlb_cma_size = 0; + hugetlb_cma_percent = 0; + s += count + 1; - tmp = memparse(s, &s); - hugetlb_cma_size_in_node[nid] = tmp; - hugetlb_cma_size += tmp; + tmp = memparse(s, &next); + if (*next == '%') { + if (tmp > 100) { + pr_warn("hugetlb_cma: invalid percentage %lu for node %d\n", + tmp, nid); + break; + } + hugetlb_cma_percent_in_node[nid] = tmp; + hugetlb_cma_size_in_node[nid] = 0; + s = next + 1; + } else { + hugetlb_cma_size_in_node[nid] = tmp; + hugetlb_cma_percent_in_node[nid] = 0; + s = next; + } /* * Skip the separator if have one, otherwise @@ -108,7 +150,28 @@ static int __init cmdline_parse_hugetlb_cma(char *p) else break; } else { - hugetlb_cma_size = memparse(p, &p); + char *next; + + tmp = memparse(p, &next); + if (*next == '%') { + if (tmp > 100) { + pr_warn("hugetlb_cma: invalid percentage %lu\n", tmp); + } else { + hugetlb_cma_percent = tmp; + hugetlb_cma_size = 0; + for (nid = 0; nid < MAX_NUMNODES; nid++) { + hugetlb_cma_size_in_node[nid] = 0; + hugetlb_cma_percent_in_node[nid] = 0; + } + } + } else { + hugetlb_cma_size = tmp; + hugetlb_cma_percent = 0; + for (nid = 0; nid < MAX_NUMNODES; nid++) { + hugetlb_cma_size_in_node[nid] = 0; + hugetlb_cma_percent_in_node[nid] = 0; + } + } break; } } @@ -134,8 +197,36 @@ void __init hugetlb_cma_reserve(void) { unsigned long size, reserved, per_node, order, gigantic_page_size; bool node_specific_cma_alloc = false; + bool has_node_specific_param = false; int nid; + for (nid = 0; nid < MAX_NUMNODES; nid++) { + if (hugetlb_cma_size_in_node[nid] || hugetlb_cma_percent_in_node[nid]) { + has_node_specific_param = true; + break; + } + } + + if (has_node_specific_param) { + hugetlb_cma_size = 0; + for (nid = 0; nid < MAX_NUMNODES; nid++) { + if (hugetlb_cma_percent_in_node[nid]) { + phys_addr_t node_gfp_mem = memblock_node_memory_size(nid); + u64 s; + + s = mul_u64_u32_div((u64)node_gfp_mem, + hugetlb_cma_percent_in_node[nid], + 100); + + hugetlb_cma_size_in_node[nid] = s; + } + hugetlb_cma_size += hugetlb_cma_size_in_node[nid]; + } + } else if (hugetlb_cma_percent) { + hugetlb_cma_size = mul_u64_u32_div((u64)memblock_phys_mem_size(), + hugetlb_cma_percent, 100); + } + if (!hugetlb_cma_size) return; @@ -154,6 +245,32 @@ void __init hugetlb_cma_reserve(void) VM_WARN_ON(order <= MAX_PAGE_ORDER); gigantic_page_size = PAGE_SIZE << order; + if (hugetlb_cma_percent) { + unsigned long orig_size = hugetlb_cma_size; + + hugetlb_cma_size = ALIGN_DOWN(hugetlb_cma_size, PAGE_SIZE << order); + if (orig_size && !hugetlb_cma_size) + pr_warn("hugetlb_cma: reservation size rounded down to 0 from %lu MiB (%u%%)\n", + orig_size / SZ_1M, hugetlb_cma_percent); + } else if (has_node_specific_param) { + hugetlb_cma_size = 0; + for (nid = 0; nid < MAX_NUMNODES; nid++) { + if (hugetlb_cma_percent_in_node[nid]) { + unsigned long orig_size = hugetlb_cma_size_in_node[nid]; + + hugetlb_cma_size_in_node[nid] = + ALIGN_DOWN(hugetlb_cma_size_in_node[nid], + PAGE_SIZE << order); + if (orig_size && !hugetlb_cma_size_in_node[nid]) + pr_warn("hugetlb_cma: reservation size rounded down to 0 from %lu MiB (%u%%) on node %d\n", + orig_size / SZ_1M, + hugetlb_cma_percent_in_node[nid], + nid); + } + hugetlb_cma_size += hugetlb_cma_size_in_node[nid]; + } + } + hugetlb_bootmem_set_nodes(); for (nid = 0; nid < MAX_NUMNODES; nid++) { @@ -194,8 +311,13 @@ void __init hugetlb_cma_reserve(void) per_node = DIV_ROUND_UP(hugetlb_cma_size, nodes_weight(hugetlb_bootmem_nodes)); per_node = round_up(per_node, gigantic_page_size); - pr_info("hugetlb_cma: reserve %lu MiB, up to %lu MiB per node\n", - hugetlb_cma_size / SZ_1M, per_node / SZ_1M); + if (hugetlb_cma_percent) + pr_info("hugetlb_cma: reserve %lu MiB (%u%%), up to %lu MiB per node\n", + hugetlb_cma_size / SZ_1M, hugetlb_cma_percent, + per_node / SZ_1M); + else + pr_info("hugetlb_cma: reserve %lu MiB, up to %lu MiB per node\n", + hugetlb_cma_size / SZ_1M, per_node / SZ_1M); } reserved = 0; @@ -230,8 +352,12 @@ void __init hugetlb_cma_reserve(void) } reserved += size; - pr_info("hugetlb_cma: reserved %lu MiB on node %d\n", - size / SZ_1M, nid); + if (hugetlb_cma_percent_in_node[nid]) + pr_info("hugetlb_cma: reserved %lu MiB (%u%%) on node %d\n", + size / SZ_1M, hugetlb_cma_percent_in_node[nid], nid); + else + pr_info("hugetlb_cma: reserved %lu MiB on node %d\n", + size / SZ_1M, nid); if (reserved >= hugetlb_cma_size) break; -- cgit v1.2.3 From 4050b5b0b60c93160a816d68ea1eb9ae92739a73 Mon Sep 17 00:00:00 2001 From: Hongfu Li Date: Fri, 7 Aug 2026 09:35:55 +0800 Subject: selftests/mm: fix read_file() return value check read_file() returns 0 on open/read failures and never returns negative values. Existing < 0 error checks never trigger, so read failures are silently ignored. Check for zero return to detect read_file() failures. Also fix misleading error message in get_finfo(). The error string incorrectly references read_num when reading uevent files. Link: https://lore.kernel.org/20260807013555.36525-1-hongfu.li@linux.dev Fixes: e0c13f9761df ("khugepaged: add self test") Signed-off-by: Hongfu Li Acked-by: David Hildenbrand (Arm) Cc: Baolin Wang Cc: Barry Song Cc: Dev Jain Cc: Lance Yang Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Ryan Roberts Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Zi Yan Signed-off-by: Andrew Morton --- tools/testing/selftests/mm/khugepaged.c | 4 ++-- tools/testing/selftests/mm/vm_util.c | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/mm/khugepaged.c b/tools/testing/selftests/mm/khugepaged.c index 8f221c792a28..d3a53673e1f9 100644 --- a/tools/testing/selftests/mm/khugepaged.c +++ b/tools/testing/selftests/mm/khugepaged.c @@ -141,8 +141,8 @@ static void get_finfo(const char *dir) major(path_stat.st_dev), minor(path_stat.st_dev)) >= sizeof(path)) ksft_exit_fail_msg("%s: Pathname is too long\n", __func__); - if (read_file(path, buf, sizeof(buf)) < 0) - ksft_exit_fail_perror("read_file(read_num)"); + if (!read_file(path, buf, sizeof(buf))) + ksft_exit_fail_perror("read_file(uevent)"); if (strstr(buf, "DEVTYPE=disk")) { /* Found it */ if (snprintf(finfo.dev_queue_read_ahead_path, diff --git a/tools/testing/selftests/mm/vm_util.c b/tools/testing/selftests/mm/vm_util.c index 4fe4a5a610d1..4821a3563036 100644 --- a/tools/testing/selftests/mm/vm_util.c +++ b/tools/testing/selftests/mm/vm_util.c @@ -942,7 +942,7 @@ unsigned long read_num(const char *path) { char buf[21]; - if (read_file(path, buf, sizeof(buf)) < 0) + if (!read_file(path, buf, sizeof(buf))) ksft_exit_fail_perror("read_file()"); return strtoul(buf, NULL, 10); -- cgit v1.2.3 From 22709abff9d0e3b0c61434cad58a9f7e86d68384 Mon Sep 17 00:00:00 2001 From: Ethan Nelson-Moore Date: Wed, 10 Jun 2026 18:22:44 -0700 Subject: mm: fix CONFIG_STACK_GROWSUP typo in tools/testing/vma/include/dup.h Commit 2b6a3f061f11 ("mm: declare VMA flags by bit") significantly refactored the header file include/linux/mm.h. In that step, it introduced a typo in an ifdef, referring to a non-existing config option STACK_GROWS_UP, whereas the actual config option is called STACK_GROWSUP. Commit 40a4af52e047 ("mm: fix CONFIG_STACK_GROWSUP typo in mm.h") fixed this typo in the mm.h header file, but did not update the copy of the code in tools/testing/vma/include/dup.h. Update this copy as well. Commit message adapted from the above-referenced fix to mm.h. Link: https://lore.kernel.org/20260611012258.432043-1-enelsonmoore@gmail.com Signed-off-by: Ethan Nelson-Moore Reviewed-by: Lorenzo Stoakes Cc: Alice Ryhl Cc: Jann Horn Cc: Liam R. Howlett Cc: Pedro Falcato Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- tools/testing/vma/include/dup.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index 4655aecffaf3..4c58487b764e 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -243,7 +243,7 @@ enum { #define VM_NOHUGEPAGE INIT_VM_FLAG(NOHUGEPAGE) #define VM_MERGEABLE INIT_VM_FLAG(MERGEABLE) #define VM_STACK INIT_VM_FLAG(STACK) -#ifdef CONFIG_STACK_GROWS_UP +#ifdef CONFIG_STACK_GROWSUP #define VM_STACK_EARLY INIT_VM_FLAG(STACK_EARLY) #define VMA_STACK_EARLY mk_vma_flags(VMA_STACK_EARLY_BIT) #else -- cgit v1.2.3 From 1d581ab2348cdbb6d4d0a467382926b68e374ec9 Mon Sep 17 00:00:00 2001 From: Suren Baghdasaryan Date: Wed, 8 Jul 2026 18:01:23 +0000 Subject: alloc_tag: add ioctl to /proc/allocinfo Patch series "alloc_tag: introduce IOCTL-based filtering for MAP", v8. Currently, memory allocation profiling data is primarily exposed through /proc/allocinfo. While useful for manual inspection, this text-based interface poses challenges for production monitoring and large-scale analysis: 1. Userspace must parse large amounts of text to extract specific fields. 2. To find specific tags, userspace must read the entire dataset, requiring many context switches and high data copying. 3. The kernel currently aggregates per-CPU counters for every allocation size, even those the user intends to filter out immediately. This series introduces a new IOCTL-based binary interface for allocinfo that supports kernel-side filtering. By allowing the user to specify a filter mask, we significantly reduce the work performed in-kernel and the amount of data transferred to userspace. The IOCTL mechanism was chosen for allocinfo to address the per-CPU counter aggregation bottleneck. A traditional read() operation must report the total allocation count and sizes for every code tag in the system. Doing so requires iterating across all CPUs to sum their per-CPU counters for thousands of tags, which introduces substantial runtime overhead. The IOCTL interface allows userspace to push selective filtering criteria directly into the kernel before the per-CPU counter aggregation. The kernel aggregates per-CPU counters only for a small subset of tags that match the filter. This results in significant performance improvement. Beyond fast filtered retrieval, the IOCTL foundation allows introducing a context capture mechanism in the future to capture the context for specific allocations. Performance measurements were conducted on an Intel Xeon Platinum 8481C (224 CPUs) with caches dropped before each run. The IOCTL mechanism shows a ~20x performance improvement for filtered queries. The kernel avoids the expensive per-CPU counter aggregation (alloc_tag_read) for any tags that fail the initial string or location filters. Scenario 1: Specific File Filtering (arch/x86/events/rapl.c) 1. Traditional (cat /proc/allocinfo | grep): 22ms (sys) 2. IOCTL Interface: 1ms (sys) Scenario 2: Compound Filtering (Filename + Size) 1. Traditional: (cat ... | grep | awk): 21ms (sys) 2. IOCTL Interface: 1ms (sys) Scenario 3: Size-Based Filtering (min_size = 1MB) 1. Traditional: (cat ... | awk): 21ms (sys) 2. IOCTL Interface: 14ms (sys) This patch (of 6): Add the following ioctl commands for /proc/allocinfo file: ALLOCINFO_IOC_CONTENT_ID - gets content identifier which can be used to check whether the file content has changed specifically due to module load/unload. Every time a module is loaded / unloaded, the returned value will be different. By comparing the identifier value at the beginning and at the end of the content retrieval operation, users can validate retrieved information for consistency. ALLOCINFO_IOC_GET_AT - gets the record at the specified position. This is the position of a record in /proc/allocinfo. ALLOCINFO_IOC_GET_NEXT - gets the record next to the last retrieved one. If no records were previously retrieved, returns the first record. Note, function file and module names often have the same prefixes, therefore when filtering for them, we compare the last 64 characters to minimize the chances of name collisions. [akpm@linux-foundation.org: include compat.h, per Suren] Closes: https://lore.kernel.org/oe-kbuild-all/202607091820.qbjlGhKK-lkp@intel.com/ Link: https://lore.kernel.org/cover.1783532853.git.abhishekbapat@google.com Link: https://lore.kernel.org/15596de2607ef13e7c77c6d74763f4ae992ec475.1783532853.git.abhishekbapat@google.com Signed-off-by: Suren Baghdasaryan Signed-off-by: Abhishek Bapat Acked-by: Hao Ge Cc: Jonathan Corbet Cc: Kent Overstreet Cc: Sourav Panda Signed-off-by: Andrew Morton --- Documentation/mm/allocation-profiling.rst | 5 + Documentation/userspace-api/ioctl/ioctl-number.rst | 2 + MAINTAINERS | 1 + include/linux/codetag.h | 2 + include/uapi/linux/alloc_tag.h | 65 ++++++ lib/codetag.c | 18 ++ mm/alloc_tag.c | 239 ++++++++++++++++++++- 7 files changed, 330 insertions(+), 2 deletions(-) create mode 100644 include/uapi/linux/alloc_tag.h diff --git a/Documentation/mm/allocation-profiling.rst b/Documentation/mm/allocation-profiling.rst index e928aa3e4e1e..b2ebcef8af6f 100644 --- a/Documentation/mm/allocation-profiling.rst +++ b/Documentation/mm/allocation-profiling.rst @@ -57,6 +57,11 @@ sysctl: Runtime info: /proc/allocinfo + Profiling data can be retrieved either by reading `/proc/allocinfo` directly as + text or programmatically via `ioctl()` calls defined in ``. + The ioctl interface supports structured binary data extraction as well as filtering + by module name, function, file, line number, accuracy, or allocation size limits. + Example output:: root@moria-kvm:~# sort -g /proc/allocinfo|tail|numfmt --to=iec diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst index 3f0ef1e27eb0..2fc53093752d 100644 --- a/Documentation/userspace-api/ioctl/ioctl-number.rst +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst @@ -346,6 +346,8 @@ Code Seq# Include File Comments 0xA5 20-2F linux/surface_aggregator/dtx.h Microsoft Surface DTX driver +0xA6 00-0F uapi/linux/alloc_tag.h Memory allocation profiling + 0xAA 00-3F linux/uapi/linux/userfaultfd.h 0xAB 00-1F linux/nbd.h 0xAC 00-1F linux/raw.h diff --git a/MAINTAINERS b/MAINTAINERS index 06271e742d32..557e5fd32073 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -16940,6 +16940,7 @@ S: Maintained F: Documentation/mm/allocation-profiling.rst F: include/linux/alloc_tag.h F: include/linux/pgalloc_tag.h +F: include/uapi/linux/alloc_tag.h F: mm/alloc_tag.c MEMORY MANAGEMENT - BALLOON diff --git a/include/linux/codetag.h b/include/linux/codetag.h index ddae7484ca45..a25a085c2df1 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -77,6 +77,8 @@ struct codetag_iterator { void codetag_lock_module_list(struct codetag_type *cttype); bool codetag_trylock_module_list(struct codetag_type *cttype); void codetag_unlock_module_list(struct codetag_type *cttype); +unsigned long codetag_get_content_id(struct codetag_type *cttype); +unsigned int codetag_get_count(struct codetag_type *cttype); struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype); struct codetag *codetag_next_ct(struct codetag_iterator *iter); diff --git a/include/uapi/linux/alloc_tag.h b/include/uapi/linux/alloc_tag.h new file mode 100644 index 000000000000..ee6a023cbaf4 --- /dev/null +++ b/include/uapi/linux/alloc_tag.h @@ -0,0 +1,65 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* + * alloc_tag IOCTL API definition + * + * Copyright (C) 2026 Google, LLC. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _UAPI_ALLOC_TAG_H +#define _UAPI_ALLOC_TAG_H + +#include + +/* + * Function, file and module names often have the same prefixes, therefore + * when filtering by these criteria, we compare the last 64 characters to + * minimize the chances of name collisions + */ +#define ALLOCINFO_STR_SIZE 64 + +struct allocinfo_content_id { + __u64 id; +}; + +struct allocinfo_tag { + /* Longer names are trimmed */ + char modname[ALLOCINFO_STR_SIZE]; + char function[ALLOCINFO_STR_SIZE]; + char filename[ALLOCINFO_STR_SIZE]; + __u64 lineno; +}; + +/* The alignment ensures 32-bit compatible interfaces are not broken */ +struct allocinfo_counter { + __u64 bytes; + __u64 calls; + __u8 accurate; +} __attribute__((aligned(8))); + +struct allocinfo_tag_data { + struct allocinfo_tag tag; + struct allocinfo_counter counter; +}; + +struct allocinfo_get_at { + __u64 pos; /* input */ + struct allocinfo_tag_data data; +}; + +#define _ALLOCINFO_IOC_CONTENT_ID 0 +#define _ALLOCINFO_IOC_GET_AT 1 +#define _ALLOCINFO_IOC_GET_NEXT 2 + +#define ALLOCINFO_IOC_BASE 0xA6 +#define ALLOCINFO_IOC_CONTENT_ID _IOR(ALLOCINFO_IOC_BASE, _ALLOCINFO_IOC_CONTENT_ID, \ + struct allocinfo_content_id) +#define ALLOCINFO_IOC_GET_AT _IOWR(ALLOCINFO_IOC_BASE, _ALLOCINFO_IOC_GET_AT, \ + struct allocinfo_get_at) +#define ALLOCINFO_IOC_GET_NEXT _IOR(ALLOCINFO_IOC_BASE, _ALLOCINFO_IOC_GET_NEXT, \ + struct allocinfo_tag_data) + +#endif /* _UAPI_ALLOC_TAG_H */ diff --git a/lib/codetag.c b/lib/codetag.c index 4001a7ea6675..a9cda4c962a3 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -19,6 +19,8 @@ struct codetag_type { struct codetag_type_desc desc; /* generates unique sequence number for module load */ unsigned long next_mod_seq; + /* bumped on every module load and unload */ + unsigned long content_id; }; struct codetag_range { @@ -50,6 +52,20 @@ void codetag_unlock_module_list(struct codetag_type *cttype) up_read(&cttype->mod_lock); } +unsigned long codetag_get_content_id(struct codetag_type *cttype) +{ + lockdep_assert_held(&cttype->mod_lock); + + return cttype->content_id; +} + +unsigned int codetag_get_count(struct codetag_type *cttype) +{ + lockdep_assert_held(&cttype->mod_lock); + + return cttype->count; +} + struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype) { struct codetag_iterator iter = { @@ -204,6 +220,7 @@ static int codetag_module_init(struct codetag_type *cttype, struct module *mod) down_write(&cttype->mod_lock); cmod->mod_seq = ++cttype->next_mod_seq; + ++cttype->content_id; mod_id = idr_alloc(&cttype->mod_idr, cmod, 0, 0, GFP_KERNEL); if (mod_id >= 0) { if (cttype->desc.module_load) { @@ -368,6 +385,7 @@ void codetag_unload_module(struct module *mod) cttype->count -= range_size(cttype, &cmod->range); idr_remove(&cttype->mod_idr, mod_id); kfree(cmod); + ++cttype->content_id; } up_write(&cttype->mod_lock); if (found && cttype->desc.free_section_mem) diff --git a/mm/alloc_tag.c b/mm/alloc_tag.c index b60ee89704cc..b2ac166880ac 100644 --- a/mm/alloc_tag.c +++ b/mm/alloc_tag.c @@ -5,6 +5,8 @@ #include #include #include +#include +#include #include #include #include @@ -14,6 +16,7 @@ #include #include #include +#include #include "internal.h" #include "page_alloc.h" @@ -59,6 +62,10 @@ struct allocinfo_private { struct codetag_iterator iter; struct codetag_iterator reported_iter; bool print_header; + /* ioctl uses a separate iterator not to interfere with reads */ + struct codetag_iterator ioctl_iter; + bool positioned; /* seq_open_private() sets to 0 */ + struct mutex ioctl_lock; }; static void *allocinfo_start(struct seq_file *m, loff_t *pos) @@ -142,6 +149,235 @@ static const struct seq_operations allocinfo_seq_op = { .show = allocinfo_show, }; +/* + * Initializes seq_file operations and allocates private state when opening + * the /proc/allocinfo procfs entry. + */ +static int allocinfo_open(struct inode *inode, struct file *file) +{ + int ret; + + ret = seq_open_private(file, &allocinfo_seq_op, + sizeof(struct allocinfo_private)); + if (!ret) { + struct seq_file *m = file->private_data; + struct allocinfo_private *priv = m->private; + + mutex_init(&priv->ioctl_lock); + } + return ret; +} + +/* + * Cleans up the seq_file state and frees up the private state allocated in + * allocinfo_open() when closing the /proc/allocinfo file descriptor. + */ +static int allocinfo_release(struct inode *inode, struct file *file) +{ + struct seq_file *m = file->private_data; + struct allocinfo_private *priv = m->private; + + mutex_destroy(&priv->ioctl_lock); + return seq_release_private(inode, file); +} + +/* + * Returns a pointer to the suffix of a string so that its length fits within + * ALLOCINFO_STR_SIZE, preserving the trailing characters. + * Function, file and module names often have the same prefixes, therefore + * when filtering by these criteria, we compare the last 64 characters to + * minimize the chances of name collisions + */ +static const char *allocinfo_str(const char *str) +{ + size_t len = strlen(str); + + /* Keep an extra space for the trailing NULL. */ + if (len >= ALLOCINFO_STR_SIZE) + str += (len - ALLOCINFO_STR_SIZE) + 1; + return str; +} + +/* Copy a string and trim from the beginning if it's too long */ +static void allocinfo_copy_str(char *dest, const char *src) +{ + strscpy_pad(dest, allocinfo_str(src), ALLOCINFO_STR_SIZE); +} + +/* + * Populates the UAPI allocinfo_tag_data structure with active runtime + * profiling counters extracted from the given kernel codetag. + */ +static void allocinfo_to_params(struct codetag *ct, + struct allocinfo_tag_data *data) +{ + struct alloc_tag *tag = ct_to_alloc_tag(ct); + struct alloc_tag_counters counter = alloc_tag_read(tag); + + if (ct->modname) + allocinfo_copy_str(data->tag.modname, ct->modname); + else + data->tag.modname[0] = '\0'; + allocinfo_copy_str(data->tag.function, ct->function); + allocinfo_copy_str(data->tag.filename, ct->filename); + data->tag.lineno = ct->lineno; + data->counter.bytes = counter.bytes; + data->counter.calls = counter.calls; + data->counter.accurate = !alloc_tag_is_inaccurate(tag); +} + +/* + * Retrieves the unique content ID representing the current allocation tag module + * layout, allowing userspace to detect if modules were loaded / unloaded. + */ +static int allocinfo_ioctl_get_content_id(struct seq_file *m, void __user *arg) +{ + struct allocinfo_content_id params; + + codetag_lock_module_list(alloc_tag_cttype); + params.id = codetag_get_content_id(alloc_tag_cttype); + codetag_unlock_module_list(alloc_tag_cttype); + if (copy_to_user(arg, ¶ms, sizeof(params))) + return -EFAULT; + + return 0; +} + +/* + * Seeks the ioctl iterator to the specified 0-indexed tag position, reads its + * profiling data and returns it to userspace. + */ +static int allocinfo_ioctl_get_at(struct seq_file *m, void __user *arg) +{ + struct allocinfo_private *priv; + struct codetag *ct; + __u64 pos; + struct allocinfo_get_at params = {0}; + + if (copy_from_user(¶ms, arg, sizeof(params))) + return -EFAULT; + + priv = m->private; + pos = params.pos; + + mutex_lock(&priv->ioctl_lock); + codetag_lock_module_list(alloc_tag_cttype); + + if (pos >= codetag_get_count(alloc_tag_cttype)) { + codetag_unlock_module_list(alloc_tag_cttype); + mutex_unlock(&priv->ioctl_lock); + return -ENOENT; + } + + /* Find the codetag */ + priv->ioctl_iter = codetag_get_ct_iter(alloc_tag_cttype); + ct = codetag_next_ct(&priv->ioctl_iter); + while (ct && pos--) + ct = codetag_next_ct(&priv->ioctl_iter); + if (ct) { + allocinfo_to_params(ct, ¶ms.data); + priv->positioned = true; + } + + codetag_unlock_module_list(alloc_tag_cttype); + mutex_unlock(&priv->ioctl_lock); + + if (!ct) + return -ENOENT; + + if (copy_to_user(arg, ¶ms, sizeof(params))) + return -EFAULT; + + return 0; +} + +/* + * Advances the ioctl iterator to the next allocation tag in the sequence and + * returns its profiling data to userspace. + */ +static int allocinfo_ioctl_get_next(struct seq_file *m, void __user *arg) +{ + struct allocinfo_private *priv; + struct codetag *ct; + struct allocinfo_tag_data params; + int ret = 0; + + memset(¶ms, 0, sizeof(params)); + priv = m->private; + + mutex_lock(&priv->ioctl_lock); + codetag_lock_module_list(alloc_tag_cttype); + + if (!priv->positioned) { + priv->ioctl_iter = codetag_get_ct_iter(alloc_tag_cttype); + priv->positioned = true; + } + + ct = codetag_next_ct(&priv->ioctl_iter); + if (ct) + allocinfo_to_params(ct, ¶ms); + + if (!ct) { + priv->positioned = false; + ret = -ENOENT; + } + codetag_unlock_module_list(alloc_tag_cttype); + mutex_unlock(&priv->ioctl_lock); + + if (ret == 0) { + if (copy_to_user(arg, ¶ms, sizeof(params))) + return -EFAULT; + } + return ret; +} + +/* + * Entry point ioctl function for /proc/allocinfo routing requests to fetch the + * layout content ID, seek to a specific tag, or read sequential tags. + */ +static long allocinfo_ioctl(struct file *file, unsigned int cmd, + unsigned long __arg) +{ + void __user *arg = (void __user *)__arg; + int ret; + + switch (cmd) { + case ALLOCINFO_IOC_CONTENT_ID: + ret = allocinfo_ioctl_get_content_id(file->private_data, arg); + break; + case ALLOCINFO_IOC_GET_AT: + ret = allocinfo_ioctl_get_at(file->private_data, arg); + break; + case ALLOCINFO_IOC_GET_NEXT: + ret = allocinfo_ioctl_get_next(file->private_data, arg); + break; + default: + ret = -ENOIOCTLCMD; + break; + } + + return ret; +} + +#ifdef CONFIG_COMPAT +static long allocinfo_compat_ioctl(struct file *file, unsigned int cmd, + unsigned long arg) +{ + return allocinfo_ioctl(file, cmd, (unsigned long)compat_ptr(arg)); +} +#endif + +static const struct proc_ops allocinfo_proc_ops = { + .proc_open = allocinfo_open, + .proc_read_iter = seq_read_iter, + .proc_lseek = seq_lseek, + .proc_release = allocinfo_release, + .proc_ioctl = allocinfo_ioctl, +#ifdef CONFIG_COMPAT + .proc_compat_ioctl = allocinfo_compat_ioctl, +#endif +}; + size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sleep) { struct codetag_iterator iter; @@ -999,8 +1235,7 @@ static int __init alloc_tag_init(void) return 0; } - if (!proc_create_seq_private(ALLOCINFO_FILE_NAME, 0400, NULL, &allocinfo_seq_op, - sizeof(struct allocinfo_private), NULL)) { + if (!proc_create(ALLOCINFO_FILE_NAME, 0400, NULL, &allocinfo_proc_ops)) { pr_err("Failed to create %s file\n", ALLOCINFO_FILE_NAME); shutdown_mem_profiling(false); return -ENOMEM; -- cgit v1.2.3 From 5732a4e4c18acac152768df9ce48e057232edbf4 Mon Sep 17 00:00:00 2001 From: Abhishek Bapat Date: Wed, 8 Jul 2026 18:01:24 +0000 Subject: alloc_tag: add ioctl filters to /proc/allocinfo Extend the capability of the IOCTL mechanism to filter allocations based on tag's module name, function name, file name and line number. Link: https://lore.kernel.org/6a6100c0c58cb2911f39126b9fe177a8c17db16f.1783532853.git.abhishekbapat@google.com Signed-off-by: Abhishek Bapat Acked-by: Hao Ge Acked-by: Suren Baghdasaryan Cc: Jonathan Corbet Cc: Kent Overstreet Cc: Sourav Panda Signed-off-by: Andrew Morton --- include/uapi/linux/alloc_tag.h | 26 +++++++++++++++- mm/alloc_tag.c | 68 +++++++++++++++++++++++++++++++++++++++--- 2 files changed, 89 insertions(+), 5 deletions(-) diff --git a/include/uapi/linux/alloc_tag.h b/include/uapi/linux/alloc_tag.h index ee6a023cbaf4..13e9b5916bf5 100644 --- a/include/uapi/linux/alloc_tag.h +++ b/include/uapi/linux/alloc_tag.h @@ -45,8 +45,32 @@ struct allocinfo_tag_data { struct allocinfo_counter counter; }; +enum { + ALLOCINFO_FILTER_MODNAME, + ALLOCINFO_FILTER_FUNCTION, + ALLOCINFO_FILTER_FILENAME, + ALLOCINFO_FILTER_LINENO, + __ALLOCINFO_FILTER_LAST = ALLOCINFO_FILTER_LINENO +}; + +#define ALLOCINFO_FILTER_MASK_MODNAME (1 << ALLOCINFO_FILTER_MODNAME) +#define ALLOCINFO_FILTER_MASK_FUNCTION (1 << ALLOCINFO_FILTER_FUNCTION) +#define ALLOCINFO_FILTER_MASK_FILENAME (1 << ALLOCINFO_FILTER_FILENAME) +#define ALLOCINFO_FILTER_MASK_LINENO (1 << ALLOCINFO_FILTER_LINENO) + +#define ALLOCINFO_FILTER_MASKS \ + ((1 << (__ALLOCINFO_FILTER_LAST + 1)) - 1) + +struct allocinfo_filter { + __u64 mask; /* bitmask of the filter fields used */ + struct allocinfo_tag fields; +}; + struct allocinfo_get_at { - __u64 pos; /* input */ + /* inputs */ + __u64 pos; + struct allocinfo_filter filter; + /* output */ struct allocinfo_tag_data data; }; diff --git a/mm/alloc_tag.c b/mm/alloc_tag.c index b2ac166880ac..d7ed0034c49f 100644 --- a/mm/alloc_tag.c +++ b/mm/alloc_tag.c @@ -62,6 +62,7 @@ struct allocinfo_private { struct codetag_iterator iter; struct codetag_iterator reported_iter; bool print_header; + struct allocinfo_filter filter; /* ioctl uses a separate iterator not to interfere with reads */ struct codetag_iterator ioctl_iter; bool positioned; /* seq_open_private() sets to 0 */ @@ -204,6 +205,12 @@ static void allocinfo_copy_str(char *dest, const char *src) strscpy_pad(dest, allocinfo_str(src), ALLOCINFO_STR_SIZE); } +/* Compare two strings and only consider the trimmed suffix if s1 is too long */ +static int allocinfo_cmp_str(const char *str, const char *template) +{ + return strncmp(allocinfo_str(str), template, ALLOCINFO_STR_SIZE); +} + /* * Populates the UAPI allocinfo_tag_data structure with active runtime * profiling counters extracted from the given kernel codetag. @@ -243,6 +250,40 @@ static int allocinfo_ioctl_get_content_id(struct seq_file *m, void __user *arg) return 0; } +/* + * Verifies whether a given codetag satisfies the active filtering criteria by + * matching its characteristics against the specified filter. + */ +static bool matches_filter(struct codetag *ct, struct allocinfo_filter *filter) +{ + if (!filter || !filter->mask) + return true; + + if (filter->mask & ALLOCINFO_FILTER_MASK_MODNAME) { + /* user wants to filter by modname but ct->modname is NULL */ + if (!ct->modname) { + /* validate if user was attempting to filter for built-in allocations */ + if (filter->fields.modname[0] != '\0') + return false; + } else if (allocinfo_cmp_str(ct->modname, filter->fields.modname)) + return false; + } + + if ((filter->mask & ALLOCINFO_FILTER_MASK_FUNCTION) && + ct->function && allocinfo_cmp_str(ct->function, filter->fields.function)) + return false; + + if ((filter->mask & ALLOCINFO_FILTER_MASK_FILENAME) && + ct->filename && allocinfo_cmp_str(ct->filename, filter->fields.filename)) + return false; + + if ((filter->mask & ALLOCINFO_FILTER_MASK_LINENO) && + ct->lineno != filter->fields.lineno) + return false; + + return true; +} + /* * Seeks the ioctl iterator to the specified 0-indexed tag position, reads its * profiling data and returns it to userspace. @@ -251,29 +292,46 @@ static int allocinfo_ioctl_get_at(struct seq_file *m, void __user *arg) { struct allocinfo_private *priv; struct codetag *ct; - __u64 pos; struct allocinfo_get_at params = {0}; + __u64 skip_count; if (copy_from_user(¶ms, arg, sizeof(params))) return -EFAULT; + if (params.filter.mask & ~ALLOCINFO_FILTER_MASKS) + return -EINVAL; + priv = m->private; - pos = params.pos; mutex_lock(&priv->ioctl_lock); codetag_lock_module_list(alloc_tag_cttype); - if (pos >= codetag_get_count(alloc_tag_cttype)) { + if (params.pos >= codetag_get_count(alloc_tag_cttype)) { codetag_unlock_module_list(alloc_tag_cttype); mutex_unlock(&priv->ioctl_lock); return -ENOENT; } + skip_count = params.pos; + + if (params.filter.mask) + priv->filter = params.filter; + else + priv->filter.mask = 0; + /* Find the codetag */ priv->ioctl_iter = codetag_get_ct_iter(alloc_tag_cttype); ct = codetag_next_ct(&priv->ioctl_iter); - while (ct && pos--) + + while (ct) { + if (matches_filter(ct, &priv->filter)) { + if (skip_count == 0) + break; + skip_count--; + } ct = codetag_next_ct(&priv->ioctl_iter); + } + if (ct) { allocinfo_to_params(ct, ¶ms.data); priv->positioned = true; @@ -314,6 +372,8 @@ static int allocinfo_ioctl_get_next(struct seq_file *m, void __user *arg) } ct = codetag_next_ct(&priv->ioctl_iter); + while (ct && !matches_filter(ct, &priv->filter)) + ct = codetag_next_ct(&priv->ioctl_iter); if (ct) allocinfo_to_params(ct, ¶ms); -- cgit v1.2.3 From 6f6769ea88f89116e8d66a4ac77056e2e3b377c5 Mon Sep 17 00:00:00 2001 From: Abhishek Bapat Date: Wed, 8 Jul 2026 18:01:25 +0000 Subject: alloc_tag: add size-based filtering to ioctl Extend the allocinfo filtering mechanism to allow users to filter tags based on the total number of bytes allocated [min_size, max_size]. The size range is inclusive. Filtering by size involves retrieving allocinfo per-CPU counters, which is an expensive operation. Hence, the performance of size-based filtering will be worse than other filters. Link: https://lore.kernel.org/0a7653b70ae0d64e967fbea0e933bc35f8ac656e.1783532853.git.abhishekbapat@google.com Signed-off-by: Abhishek Bapat Acked-by: Hao Ge Acked-by: Suren Baghdasaryan Cc: Jonathan Corbet Cc: Kent Overstreet Cc: Sourav Panda Signed-off-by: Andrew Morton --- include/uapi/linux/alloc_tag.h | 8 +++++- mm/alloc_tag.c | 64 +++++++++++++++++++++++++++++++++--------- 2 files changed, 58 insertions(+), 14 deletions(-) diff --git a/include/uapi/linux/alloc_tag.h b/include/uapi/linux/alloc_tag.h index 13e9b5916bf5..0de5fc180790 100644 --- a/include/uapi/linux/alloc_tag.h +++ b/include/uapi/linux/alloc_tag.h @@ -50,13 +50,17 @@ enum { ALLOCINFO_FILTER_FUNCTION, ALLOCINFO_FILTER_FILENAME, ALLOCINFO_FILTER_LINENO, - __ALLOCINFO_FILTER_LAST = ALLOCINFO_FILTER_LINENO + ALLOCINFO_FILTER_MIN_SIZE, + ALLOCINFO_FILTER_MAX_SIZE, + __ALLOCINFO_FILTER_LAST = ALLOCINFO_FILTER_MAX_SIZE }; #define ALLOCINFO_FILTER_MASK_MODNAME (1 << ALLOCINFO_FILTER_MODNAME) #define ALLOCINFO_FILTER_MASK_FUNCTION (1 << ALLOCINFO_FILTER_FUNCTION) #define ALLOCINFO_FILTER_MASK_FILENAME (1 << ALLOCINFO_FILTER_FILENAME) #define ALLOCINFO_FILTER_MASK_LINENO (1 << ALLOCINFO_FILTER_LINENO) +#define ALLOCINFO_FILTER_MASK_MIN_SIZE (1 << ALLOCINFO_FILTER_MIN_SIZE) +#define ALLOCINFO_FILTER_MASK_MAX_SIZE (1 << ALLOCINFO_FILTER_MAX_SIZE) #define ALLOCINFO_FILTER_MASKS \ ((1 << (__ALLOCINFO_FILTER_LAST + 1)) - 1) @@ -64,6 +68,8 @@ enum { struct allocinfo_filter { __u64 mask; /* bitmask of the filter fields used */ struct allocinfo_tag fields; + __u64 min_size; + __u64 max_size; }; struct allocinfo_get_at { diff --git a/mm/alloc_tag.c b/mm/alloc_tag.c index d7ed0034c49f..04b640d74bcc 100644 --- a/mm/alloc_tag.c +++ b/mm/alloc_tag.c @@ -211,16 +211,20 @@ static int allocinfo_cmp_str(const char *str, const char *template) return strncmp(allocinfo_str(str), template, ALLOCINFO_STR_SIZE); } +/* Fetch the per-CPU counters */ +static inline struct alloc_tag_counters allocinfo_prefetch_counters(struct codetag *ct) +{ + return alloc_tag_read(ct_to_alloc_tag(ct)); +} + /* * Populates the UAPI allocinfo_tag_data structure with active runtime * profiling counters extracted from the given kernel codetag. */ static void allocinfo_to_params(struct codetag *ct, - struct allocinfo_tag_data *data) + struct allocinfo_tag_data *data, + struct alloc_tag_counters *counters) { - struct alloc_tag *tag = ct_to_alloc_tag(ct); - struct alloc_tag_counters counter = alloc_tag_read(tag); - if (ct->modname) allocinfo_copy_str(data->tag.modname, ct->modname); else @@ -228,9 +232,9 @@ static void allocinfo_to_params(struct codetag *ct, allocinfo_copy_str(data->tag.function, ct->function); allocinfo_copy_str(data->tag.filename, ct->filename); data->tag.lineno = ct->lineno; - data->counter.bytes = counter.bytes; - data->counter.calls = counter.calls; - data->counter.accurate = !alloc_tag_is_inaccurate(tag); + data->counter.bytes = counters->bytes; + data->counter.calls = counters->calls; + data->counter.accurate = !alloc_tag_is_inaccurate(ct_to_alloc_tag(ct)); } /* @@ -254,7 +258,9 @@ static int allocinfo_ioctl_get_content_id(struct seq_file *m, void __user *arg) * Verifies whether a given codetag satisfies the active filtering criteria by * matching its characteristics against the specified filter. */ -static bool matches_filter(struct codetag *ct, struct allocinfo_filter *filter) +static bool matches_filter(struct codetag *ct, struct allocinfo_filter *filter, + struct alloc_tag_counters *counters, + bool *fetched_counters) { if (!filter || !filter->mask) return true; @@ -281,6 +287,19 @@ static bool matches_filter(struct codetag *ct, struct allocinfo_filter *filter) ct->lineno != filter->fields.lineno) return false; + if (filter->mask & (ALLOCINFO_FILTER_MASK_MIN_SIZE | ALLOCINFO_FILTER_MASK_MAX_SIZE)) { + if (!*fetched_counters) { + *counters = allocinfo_prefetch_counters(ct); + *fetched_counters = true; + } + if ((filter->mask & ALLOCINFO_FILTER_MASK_MIN_SIZE) && + counters->bytes < filter->min_size) + return false; + if ((filter->mask & ALLOCINFO_FILTER_MASK_MAX_SIZE) && + counters->bytes > filter->max_size) + return false; + } + return true; } @@ -294,6 +313,8 @@ static int allocinfo_ioctl_get_at(struct seq_file *m, void __user *arg) struct codetag *ct; struct allocinfo_get_at params = {0}; __u64 skip_count; + struct alloc_tag_counters counters; + bool fetched_counters; if (copy_from_user(¶ms, arg, sizeof(params))) return -EFAULT; @@ -301,6 +322,11 @@ static int allocinfo_ioctl_get_at(struct seq_file *m, void __user *arg) if (params.filter.mask & ~ALLOCINFO_FILTER_MASKS) return -EINVAL; + if ((params.filter.mask & ALLOCINFO_FILTER_MASK_MIN_SIZE) && + (params.filter.mask & ALLOCINFO_FILTER_MASK_MAX_SIZE) && + params.filter.min_size > params.filter.max_size) + return -EINVAL; + priv = m->private; mutex_lock(&priv->ioctl_lock); @@ -324,7 +350,8 @@ static int allocinfo_ioctl_get_at(struct seq_file *m, void __user *arg) ct = codetag_next_ct(&priv->ioctl_iter); while (ct) { - if (matches_filter(ct, &priv->filter)) { + fetched_counters = false; + if (matches_filter(ct, &priv->filter, &counters, &fetched_counters)) { if (skip_count == 0) break; skip_count--; @@ -333,7 +360,9 @@ static int allocinfo_ioctl_get_at(struct seq_file *m, void __user *arg) } if (ct) { - allocinfo_to_params(ct, ¶ms.data); + if (!fetched_counters) + counters = allocinfo_prefetch_counters(ct); + allocinfo_to_params(ct, ¶ms.data, &counters); priv->positioned = true; } @@ -359,6 +388,8 @@ static int allocinfo_ioctl_get_next(struct seq_file *m, void __user *arg) struct codetag *ct; struct allocinfo_tag_data params; int ret = 0; + struct alloc_tag_counters counters; + bool fetched_counters; memset(¶ms, 0, sizeof(params)); priv = m->private; @@ -372,11 +403,18 @@ static int allocinfo_ioctl_get_next(struct seq_file *m, void __user *arg) } ct = codetag_next_ct(&priv->ioctl_iter); - while (ct && !matches_filter(ct, &priv->filter)) + while (ct) { + fetched_counters = false; + if (matches_filter(ct, &priv->filter, &counters, &fetched_counters)) + break; ct = codetag_next_ct(&priv->ioctl_iter); - if (ct) - allocinfo_to_params(ct, ¶ms); + } + if (ct) { + if (!fetched_counters) + counters = allocinfo_prefetch_counters(ct); + allocinfo_to_params(ct, ¶ms, &counters); + } if (!ct) { priv->positioned = false; ret = -ENOENT; -- cgit v1.2.3 From 33588e0b81df2972922f2591767a4e3c16813ab9 Mon Sep 17 00:00:00 2001 From: Abhishek Bapat Date: Wed, 8 Jul 2026 18:01:26 +0000 Subject: alloc_tag: add accuracy based filtering to ioctl Extend the allocinfo filtering mechanism to allow users to filter tags based on their accuracy. [abhishekbapat@google.com: move `inaccurate` filtering criteria from `struct allocinfo_tag` to `struct allocinfo_filter`] Link: https://lore.kernel.org/e4e49ec4a5960292aeeb9e196526c18dc95228a2.1785867739.git.abhishekbapat@google.com Link: https://lore.kernel.org/396a5e4bc3b2990223ab355f2cd3ceb6aa15499e.1783532853.git.abhishekbapat@google.com Signed-off-by: Abhishek Bapat Acked-by: Hao Ge Acked-by: Suren Baghdasaryan Cc: Jonathan Corbet Cc: Kent Overstreet Cc: Sourav Panda Signed-off-by: Andrew Morton --- include/uapi/linux/alloc_tag.h | 4 ++++ mm/alloc_tag.c | 8 ++++++++ 2 files changed, 12 insertions(+) diff --git a/include/uapi/linux/alloc_tag.h b/include/uapi/linux/alloc_tag.h index 0de5fc180790..e3ad94444864 100644 --- a/include/uapi/linux/alloc_tag.h +++ b/include/uapi/linux/alloc_tag.h @@ -50,6 +50,7 @@ enum { ALLOCINFO_FILTER_FUNCTION, ALLOCINFO_FILTER_FILENAME, ALLOCINFO_FILTER_LINENO, + ALLOCINFO_FILTER_INACCURATE, ALLOCINFO_FILTER_MIN_SIZE, ALLOCINFO_FILTER_MAX_SIZE, __ALLOCINFO_FILTER_LAST = ALLOCINFO_FILTER_MAX_SIZE @@ -59,6 +60,7 @@ enum { #define ALLOCINFO_FILTER_MASK_FUNCTION (1 << ALLOCINFO_FILTER_FUNCTION) #define ALLOCINFO_FILTER_MASK_FILENAME (1 << ALLOCINFO_FILTER_FILENAME) #define ALLOCINFO_FILTER_MASK_LINENO (1 << ALLOCINFO_FILTER_LINENO) +#define ALLOCINFO_FILTER_MASK_INACCURATE (1 << ALLOCINFO_FILTER_INACCURATE) #define ALLOCINFO_FILTER_MASK_MIN_SIZE (1 << ALLOCINFO_FILTER_MIN_SIZE) #define ALLOCINFO_FILTER_MASK_MAX_SIZE (1 << ALLOCINFO_FILTER_MAX_SIZE) @@ -70,6 +72,8 @@ struct allocinfo_filter { struct allocinfo_tag fields; __u64 min_size; __u64 max_size; + /* filter criteria only; see allocinfo_counter.accurate for actual accuracy */ + __u64 inaccurate; }; struct allocinfo_get_at { diff --git a/mm/alloc_tag.c b/mm/alloc_tag.c index 04b640d74bcc..b33410310477 100644 --- a/mm/alloc_tag.c +++ b/mm/alloc_tag.c @@ -262,6 +262,8 @@ static bool matches_filter(struct codetag *ct, struct allocinfo_filter *filter, struct alloc_tag_counters *counters, bool *fetched_counters) { + bool inaccurate; + if (!filter || !filter->mask) return true; @@ -287,6 +289,12 @@ static bool matches_filter(struct codetag *ct, struct allocinfo_filter *filter, ct->lineno != filter->fields.lineno) return false; + if (filter->mask & ALLOCINFO_FILTER_MASK_INACCURATE) { + inaccurate = !!(ct->flags & CODETAG_FLAG_INACCURATE); + if (inaccurate != !!(filter->inaccurate)) + return false; + } + if (filter->mask & (ALLOCINFO_FILTER_MASK_MIN_SIZE | ALLOCINFO_FILTER_MASK_MAX_SIZE)) { if (!*fetched_counters) { *counters = allocinfo_prefetch_counters(ct); -- cgit v1.2.3 From 2f252a7a6c90c94be042c10ca3b85f37038d1070 Mon Sep 17 00:00:00 2001 From: Abhishek Bapat Date: Wed, 8 Jul 2026 18:01:27 +0000 Subject: kselftest: alloc_tag: add kselftest for ioctl interface Introduce a kselftest to verify the new IOCTL-based interface for /proc/allocinfo. The test covers: 1. Validation of the filename filter. 2. Validation of the function filter. The first test validates the functionality of the filename filter. Using "mm/memory.c" as the candidate filename filter, it retrieves filtered entries from both procfs and ioctl and matches the first VEC_MAX_ENTRIES entries. The second test validates the functionality of the function filter. It uses "dup_mm" as the candidate function as we do not expect this function name to change frequently and hence won't be needing to modify this test often. Note that both the tests match line no, function name and file name fields. Bytes allocated and calls are not matched as those values may change in the time when the data is being read from procfs and ioctl and hence can lead to false negatives. [abhishekbapat@google.com: fix a typo in the selftest] Link: https://lore.kernel.org/e4e49ec4a5960292aeeb9e196526c18dc95228a2.1785867739.git.abhishekbapat@google.com Closes: https://sashiko.dev/#/patchset/cover.1783532853.git.abhishekbapat@google.com Link: https://lore.kernel.org/e2a3795677a14aeab249758ba570cd5e98402032.1783532853.git.abhishekbapat@google.com Signed-off-by: Abhishek Bapat Tested-by: Hao Ge Acked-by: Hao Ge Cc: Jonathan Corbet Cc: Kent Overstreet Cc: Sourav Panda Cc: Suren Baghdasaryan Signed-off-by: Andrew Morton --- MAINTAINERS | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/alloc_tag/Makefile | 8 + .../selftests/alloc_tag/allocinfo_ioctl_test.c | 334 +++++++++++++++++++++ 4 files changed, 344 insertions(+) create mode 100644 tools/testing/selftests/alloc_tag/Makefile create mode 100644 tools/testing/selftests/alloc_tag/allocinfo_ioctl_test.c diff --git a/MAINTAINERS b/MAINTAINERS index 557e5fd32073..4899b81bd839 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -16942,6 +16942,7 @@ F: include/linux/alloc_tag.h F: include/linux/pgalloc_tag.h F: include/uapi/linux/alloc_tag.h F: mm/alloc_tag.c +F: tools/testing/selftests/alloc_tag/ MEMORY MANAGEMENT - BALLOON M: Andrew Morton diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index 5528682a3a91..2cc63e4134fb 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -1,5 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 TARGETS += acct +TARGETS += alloc_tag TARGETS += alsa TARGETS += amd-pstate TARGETS += arm64 diff --git a/tools/testing/selftests/alloc_tag/Makefile b/tools/testing/selftests/alloc_tag/Makefile new file mode 100644 index 000000000000..c4637f69e9c2 --- /dev/null +++ b/tools/testing/selftests/alloc_tag/Makefile @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: GPL-2.0 + +TEST_GEN_PROGS := allocinfo_ioctl_test + +CFLAGS += -Wall +CFLAGS += $(KHDR_INCLUDES) + +include ../lib.mk diff --git a/tools/testing/selftests/alloc_tag/allocinfo_ioctl_test.c b/tools/testing/selftests/alloc_tag/allocinfo_ioctl_test.c new file mode 100644 index 000000000000..3614ee9b46fb --- /dev/null +++ b/tools/testing/selftests/alloc_tag/allocinfo_ioctl_test.c @@ -0,0 +1,334 @@ +// SPDX-License-Identifier: GPL-2.0-only + +/* kselftest for allocinfo ioctl + * allocinfo ioctl retrieves allocinfo data through ioctl + * Copyright (C) 2026 Google, Inc. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "../kselftest.h" + +#define MAX_LINE_LEN 512 +#define ALLOCINFO_PROC "/proc/allocinfo" + +enum ioctl_ret { + IOCTL_SUCCESS = 0, + IOCTL_FAILURE = 1, + IOCTL_INVALID_DATA = 2, +}; + +#define VEC_MAX_ENTRIES 32 + +struct allocinfo_tag_data_vec { + struct allocinfo_tag_data tag[VEC_MAX_ENTRIES]; + __u64 count; +}; + +static inline int __allocinfo_get_content_id(int dev_fd, struct allocinfo_content_id *params) +{ + return ioctl(dev_fd, ALLOCINFO_IOC_CONTENT_ID, params); +} + +static inline int __allocinfo_get_at(int dev_fd, struct allocinfo_get_at *params) +{ + return ioctl(dev_fd, ALLOCINFO_IOC_GET_AT, params); +} + +static inline int __allocinfo_get_next(int dev_fd, struct allocinfo_tag_data *params) +{ + return ioctl(dev_fd, ALLOCINFO_IOC_GET_NEXT, params); +} + +static bool match_entry(const struct allocinfo_tag_data *procfs_entry, + const struct allocinfo_tag_data *tag_data, + bool match_bytes, bool match_calls, bool match_lineno, + bool match_function, bool match_filename) +{ + if (match_bytes && tag_data->counter.bytes != procfs_entry->counter.bytes) { + ksft_print_msg("size retrieved through ioctl does not match procfs\n"); + return false; + } + + if (match_calls && tag_data->counter.calls != procfs_entry->counter.calls) { + ksft_print_msg("call count retrieved through ioctl does not match procfs\n"); + return false; + } + + if (match_lineno && tag_data->tag.lineno != procfs_entry->tag.lineno) { + ksft_print_msg("lineno retrieved through ioctl does not match procfs\n"); + return false; + } + + if (match_function && + strncmp(tag_data->tag.function, procfs_entry->tag.function, ALLOCINFO_STR_SIZE)) { + ksft_print_msg("function retrieved through ioctl does not match procfs\n"); + return false; + } + + if (match_filename && + strncmp(tag_data->tag.filename, procfs_entry->tag.filename, ALLOCINFO_STR_SIZE)) { + ksft_print_msg("filename retrieved through ioctl does not match procfs\n"); + return false; + } + return true; +} + +static bool match_entries(const struct allocinfo_tag_data_vec *procfs_entries, + const struct allocinfo_tag_data_vec *tags, + bool match_bytes, bool match_calls, bool match_lineno, + bool match_function, bool match_filename) +{ + __u64 i; + + if (procfs_entries->count != tags->count) { + ksft_print_msg("Entry count mismatch. ioctl entries: %llu, proc entries: %llu\n", + tags->count, procfs_entries->count); + return false; + } + for (i = 0; i < procfs_entries->count; i++) { + if (!match_entry(&procfs_entries->tag[i], &tags->tag[i], + match_bytes, match_calls, match_lineno, + match_function, match_filename)) { + ksft_print_msg("%lluth entry does not match.\n", i); + return false; + } + } + return true; +} + +static const char *allocinfo_str(const char *str) +{ + size_t len = strlen(str); + + if (len >= ALLOCINFO_STR_SIZE) + str += (len - ALLOCINFO_STR_SIZE) + 1; + return str; +} + +static void allocinfo_copy_str(char *dest, const char *src) +{ + strncpy(dest, allocinfo_str(src), ALLOCINFO_STR_SIZE - 1); + dest[ALLOCINFO_STR_SIZE - 1] = '\0'; +} + +static int get_filtered_procfs_entries(struct allocinfo_tag_data_vec *procfs_entries, + const struct allocinfo_filter *filter) +{ + FILE *fp = fopen(ALLOCINFO_PROC, "r"); + char line[MAX_LINE_LEN]; + int matches; + struct allocinfo_tag_data procfs_entry; + + if (!fp) { + ksft_print_msg("Failed to open " ALLOCINFO_PROC " for reading\n"); + return 1; + } + memset(procfs_entries, 0, sizeof(*procfs_entries)); + while (fgets(line, sizeof(line), fp) && procfs_entries->count < VEC_MAX_ENTRIES) { + char filename[MAX_LINE_LEN]; + char function[MAX_LINE_LEN]; + + memset(&procfs_entry, 0, sizeof(procfs_entry)); + matches = sscanf(line, "%llu %llu %[^:]:%llu func:%s", + &procfs_entry.counter.bytes, + &procfs_entry.counter.calls, + filename, + &procfs_entry.tag.lineno, + function); + + if (matches != 5) + continue; + + allocinfo_copy_str(procfs_entry.tag.filename, filename); + allocinfo_copy_str(procfs_entry.tag.function, function); + + if (filter->mask & ALLOCINFO_FILTER_MASK_FILENAME) { + if (strncmp(procfs_entry.tag.filename, + filter->fields.filename, ALLOCINFO_STR_SIZE)) + continue; + } + if (filter->mask & ALLOCINFO_FILTER_MASK_FUNCTION) { + if (strncmp(procfs_entry.tag.function, + filter->fields.function, ALLOCINFO_STR_SIZE)) + continue; + } + if (filter->mask & ALLOCINFO_FILTER_MASK_LINENO) { + if (procfs_entry.tag.lineno != filter->fields.lineno) + continue; + } + if (filter->mask & ALLOCINFO_FILTER_MASK_MIN_SIZE) { + if (procfs_entry.counter.bytes < filter->min_size) + continue; + } + if (filter->mask & ALLOCINFO_FILTER_MASK_MAX_SIZE) { + if (procfs_entry.counter.bytes > filter->max_size) + continue; + } + + memcpy(&procfs_entries->tag[procfs_entries->count++], &procfs_entry, + sizeof(procfs_entry)); + } + fclose(fp); + return 0; +} + +static enum ioctl_ret get_filtered_ioctl_entries(struct allocinfo_tag_data_vec *tags, + const struct allocinfo_filter *filter, + __u64 start_pos) +{ + int fd = open(ALLOCINFO_PROC, O_RDONLY); + + if (fd < 0) { + ksft_print_msg("Failed to open " ALLOCINFO_PROC " for IOCTL\n"); + return IOCTL_FAILURE; + } + + struct allocinfo_content_id start_cont_id, end_cont_id; + struct allocinfo_get_at get_at_params; + const int max_retries = 10; + int retry_count = 0; + int status; + + /* + * __allocinfo_get_content_id may return different values if a kernel module was loaded + * between the two calls. If that happens, the data gathered cannot be considered consistent + * and hence needs to be fetched again to avoid flakiness. + */ + do { + if (__allocinfo_get_content_id(fd, &start_cont_id)) { + ksft_print_msg("allocinfo_get_content_id failed\n"); + status = IOCTL_FAILURE; + break; + } + + memset(tags, 0, sizeof(*tags)); + memset(&get_at_params, 0, sizeof(get_at_params)); + memcpy(&get_at_params.filter, filter, sizeof(*filter)); + get_at_params.pos = start_pos; + if (__allocinfo_get_at(fd, &get_at_params)) { + ksft_print_msg("allocinfo_get_at failed\n"); + status = IOCTL_FAILURE; + break; + } + memcpy(&tags->tag[tags->count++], &get_at_params.data, sizeof(get_at_params.data)); + + while (tags->count < VEC_MAX_ENTRIES && + __allocinfo_get_next(fd, &tags->tag[tags->count]) == 0) + tags->count++; + + if (__allocinfo_get_content_id(fd, &end_cont_id)) { + ksft_print_msg("allocinfo_get_content_id failed\n"); + status = IOCTL_FAILURE; + break; + } + + if (start_cont_id.id == end_cont_id.id) { + status = IOCTL_SUCCESS; + } else { + ksft_print_msg("allocinfo_get_content_id mismatch, retrying...\n"); + status = IOCTL_INVALID_DATA; + } + } while (status == IOCTL_INVALID_DATA && retry_count++ < max_retries); + + close(fd); + return status; +} + +static int run_filter_test(const struct allocinfo_filter *filter) +{ + struct allocinfo_tag_data_vec *tags = malloc(sizeof(*tags)); + struct allocinfo_tag_data_vec *procfs_entries = malloc(sizeof(*procfs_entries)); + int ioctl_status; + int ret = KSFT_PASS; + + if (!tags || !procfs_entries) { + ksft_print_msg("Memory allocation failed.\n"); + ret = KSFT_FAIL; + goto exit; + } + + if (get_filtered_procfs_entries(procfs_entries, filter)) { + ksft_print_msg("Error retrieving entries from " ALLOCINFO_PROC "\n"); + ret = KSFT_SKIP; + goto exit; + } + + if (procfs_entries->count == 0) { + ksft_print_msg("No entries found in " ALLOCINFO_PROC ", skipping test\n"); + ret = KSFT_SKIP; + goto exit; + } + + ioctl_status = get_filtered_ioctl_entries(tags, filter, 0); + if (ioctl_status == IOCTL_INVALID_DATA) { + ksft_print_msg("Trouble retrieving valid IOCTL entries, skipping.\n"); + ret = KSFT_SKIP; + goto exit; + } + if (ioctl_status == IOCTL_FAILURE) { + ksft_print_msg("Error retrieving IOCTL entries.\n"); + ret = KSFT_FAIL; + goto exit; + } + + if (!match_entries(procfs_entries, tags, false, false, true, true, true)) + ret = KSFT_FAIL; + +exit: + free(tags); + free(procfs_entries); + return ret; +} + +static int test_filename_filter(void) +{ + struct allocinfo_filter filter; + const char *target_filename = "mm/memory.c"; + + memset(&filter, 0, sizeof(filter)); + filter.mask |= ALLOCINFO_FILTER_MASK_FILENAME; + strncpy(filter.fields.filename, target_filename, ALLOCINFO_STR_SIZE); + + return run_filter_test(&filter); +} + +static int test_function_filter(void) +{ + struct allocinfo_filter filter; + const char *target_function = "dup_mm"; + + memset(&filter, 0, sizeof(filter)); + filter.mask |= ALLOCINFO_FILTER_MASK_FUNCTION; + strncpy(filter.fields.function, target_function, ALLOCINFO_STR_SIZE); + + return run_filter_test(&filter); +} + +int main(int argc, char *argv[]) +{ + int ret; + + ksft_set_plan(2); + + ret = test_filename_filter(); + if (ret == KSFT_SKIP) + ksft_test_result_skip("Skipping test_filename_filter\n"); + else + ksft_test_result(ret == KSFT_PASS, "test_filename_filter\n"); + + ret = test_function_filter(); + if (ret == KSFT_SKIP) + ksft_test_result_skip("Skipping test_function_filter\n"); + else + ksft_test_result(ret == KSFT_PASS, "test_function_filter\n"); + + ksft_finished(); +} -- cgit v1.2.3 From 923690d8099349ec46c8930c0e73b462ea41632b Mon Sep 17 00:00:00 2001 From: Abhishek Bapat Date: Wed, 8 Jul 2026 18:01:28 +0000 Subject: kselftest: alloc_tag: extend the allocinfo ioctl kselftest Add the following 2 scenarios to the allocinfo ioctl kselftest: 1. Validate size based filtering 2. Validate lineno based filtering The first test uses "do_init_module" as the candidate function for the test. This is because the associated site will only allocate memory when a kernel module is loaded. The return value of get_content_id() changes every time modules are loaded or unloaded. Hence, as long as get_content_id() values at the start and the end of the test are the same, the memory allocated by the do_init_module call site should also remain the same. Consequently, the test can assume consistency between the value returned by the ioctl and the procfs resulting in less flakiness. Link: https://lore.kernel.org/e5171926b48802531284c1cb5f04734017141341.1783532853.git.abhishekbapat@google.com Signed-off-by: Abhishek Bapat Tested-by: Hao Ge Acked-by: Hao Ge Cc: Jonathan Corbet Cc: Kent Overstreet Cc: Sourav Panda Cc: Suren Baghdasaryan Signed-off-by: Andrew Morton --- .../selftests/alloc_tag/allocinfo_ioctl_test.c | 216 ++++++++++++++++++++- 1 file changed, 215 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/alloc_tag/allocinfo_ioctl_test.c b/tools/testing/selftests/alloc_tag/allocinfo_ioctl_test.c index 3614ee9b46fb..74fd64b2370c 100644 --- a/tools/testing/selftests/alloc_tag/allocinfo_ioctl_test.c +++ b/tools/testing/selftests/alloc_tag/allocinfo_ioctl_test.c @@ -5,6 +5,7 @@ * Copyright (C) 2026 Google, Inc. */ +#include #include #include #include @@ -312,11 +313,212 @@ static int test_function_filter(void) return run_filter_test(&filter); } +static int test_size_filter(void) +{ + int fd; + struct allocinfo_tag_data_vec *tags = malloc(sizeof(*tags)); + struct allocinfo_tag_data_vec *procfs_entries = malloc(sizeof(*procfs_entries)); + struct allocinfo_filter filter; + int ret = KSFT_PASS; + __u64 target_size, i, pos; + struct allocinfo_tag_data *found_tag = NULL; + const char *target_function = "do_init_module"; + struct allocinfo_content_id start_cont_id, end_cont_id; + int retry = 0; + const int max_retries = 10; + + if (!tags || !procfs_entries) { + ksft_print_msg("Memory allocation failed.\n"); + ret = KSFT_FAIL; + goto freemem; + } + + fd = open(ALLOCINFO_PROC, O_RDONLY); + if (fd < 0) { + ksft_print_msg("Failed to open " ALLOCINFO_PROC ": %s\n", strerror(errno)); + ret = KSFT_SKIP; + goto freemem; + } + + do { + found_tag = NULL; + pos = 0; + + if (__allocinfo_get_content_id(fd, &start_cont_id)) { + ksft_print_msg("allocinfo_get_content_id failed\n"); + ret = KSFT_FAIL; + goto exit; + } + + memset(&filter, 0, sizeof(filter)); + filter.mask |= ALLOCINFO_FILTER_MASK_FUNCTION; + strncpy(filter.fields.function, target_function, ALLOCINFO_STR_SIZE); + + if (get_filtered_procfs_entries(procfs_entries, &filter)) { + ksft_print_msg("Error retrieving entries from " ALLOCINFO_PROC "\n"); + ret = KSFT_SKIP; + goto exit; + } + + if (procfs_entries->count == 0) { + ksft_print_msg("Function %s not found in procfs\n", target_function); + ret = KSFT_SKIP; + goto exit; + } + + target_size = procfs_entries->tag[0].counter.bytes; + + memset(&filter, 0, sizeof(filter)); + filter.mask |= ALLOCINFO_FILTER_MASK_MIN_SIZE | ALLOCINFO_FILTER_MASK_MAX_SIZE; + filter.min_size = target_size; + filter.max_size = target_size; + + while (1) { + struct allocinfo_get_at get_at_params; + + memset(&get_at_params, 0, sizeof(get_at_params)); + memcpy(&get_at_params.filter, &filter, sizeof(filter)); + get_at_params.pos = pos; + + if (__allocinfo_get_at(fd, &get_at_params)) + break; + + tags->count = 0; + memcpy(&tags->tag[tags->count++], &get_at_params.data, + sizeof(get_at_params.data)); + + while (tags->count < VEC_MAX_ENTRIES && + __allocinfo_get_next(fd, &tags->tag[tags->count]) == 0) + tags->count++; + + for (i = 0; i < tags->count; i++) { + if (strcmp(tags->tag[i].tag.function, target_function) == 0) { + found_tag = &tags->tag[i]; + break; + } + } + + if (found_tag || tags->count < VEC_MAX_ENTRIES) + break; + + pos += tags->count; + } + + if (__allocinfo_get_content_id(fd, &end_cont_id)) { + ksft_print_msg("allocinfo_get_content_id failed\n"); + ret = KSFT_FAIL; + goto exit; + } + + if (start_cont_id.id == end_cont_id.id) + break; + + ksft_print_msg("Module load detected during size verification, retrying...\n"); + } while (retry++ < max_retries); + + if (start_cont_id.id == end_cont_id.id && !found_tag) { + ksft_print_msg("Entry with function %s not found in IOCTL results\n", + target_function); + ret = KSFT_FAIL; + } else if (start_cont_id.id != end_cont_id.id) { + ksft_print_msg("Failed to match content_ids for procfs and IOCTL, skipping...\n"); + ret = KSFT_SKIP; + } else if (found_tag && found_tag->counter.bytes != target_size) { + ksft_print_msg("IOCTL entry size %llu does not match target size %llu\n", + found_tag->counter.bytes, target_size); + ret = KSFT_FAIL; + } + +exit: + close(fd); +freemem: + free(tags); + free(procfs_entries); + return ret; +} + +static int test_lineno_filter(void) +{ + struct allocinfo_tag_data_vec *tags = malloc(sizeof(*tags)); + struct allocinfo_tag_data_vec *procfs_entries = malloc(sizeof(*procfs_entries)); + struct allocinfo_filter filter; + enum ioctl_ret ioctl_status; + int ret = KSFT_PASS; + __u64 target_lineno, i; + struct allocinfo_tag_data *target_tag; + bool found = false; + + if (!tags || !procfs_entries) { + ksft_print_msg("Memory allocation failed.\n"); + ret = KSFT_FAIL; + goto exit; + } + + memset(&filter, 0, sizeof(filter)); + + if (get_filtered_procfs_entries(procfs_entries, &filter)) { + ksft_print_msg("Error retrieving entries from " ALLOCINFO_PROC "\n"); + ret = KSFT_SKIP; + goto exit; + } + if (procfs_entries->count == 0) { + ksft_print_msg("Could not retrieve procfs entries\n"); + ret = KSFT_SKIP; + goto exit; + } + /* + * We depend on the procfs results to determine the line number for the filter before + * making the ioctl query. Hence, we cannot reuse run_filter_test here. + */ + target_tag = &procfs_entries->tag[0]; + target_lineno = target_tag->tag.lineno; + + filter.mask |= ALLOCINFO_FILTER_MASK_LINENO; + filter.fields.lineno = target_lineno; + + ioctl_status = get_filtered_ioctl_entries(tags, &filter, 0); + if (ioctl_status == IOCTL_INVALID_DATA) { + ksft_print_msg("Trouble retrieving valid IOCTL entries, skipping.\n"); + ret = KSFT_SKIP; + goto exit; + } + if (ioctl_status == IOCTL_FAILURE) { + ksft_print_msg("Error retrieving IOCTL entries.\n"); + ret = KSFT_FAIL; + goto exit; + } + + for (i = 0; i < tags->count; i++) { + if (tags->tag[i].tag.lineno != target_lineno) { + ksft_print_msg("IOCTL entry %llu has incorrect lineno %llu.\n", + i, tags->tag[i].tag.lineno); + ret = KSFT_FAIL; + goto exit; + } + + if (strncmp(tags->tag[i].tag.function, target_tag->tag.function, + ALLOCINFO_STR_SIZE) == 0 && + strncmp(tags->tag[i].tag.filename, target_tag->tag.filename, + ALLOCINFO_STR_SIZE) == 0) + found = true; + } + + if (!found) { + ksft_print_msg("Original procfs entry not found in IOCTL lineno filter results.\n"); + ret = KSFT_FAIL; + } + +exit: + free(tags); + free(procfs_entries); + return ret; +} + int main(int argc, char *argv[]) { int ret; - ksft_set_plan(2); + ksft_set_plan(4); ret = test_filename_filter(); if (ret == KSFT_SKIP) @@ -330,5 +532,17 @@ int main(int argc, char *argv[]) else ksft_test_result(ret == KSFT_PASS, "test_function_filter\n"); + ret = test_size_filter(); + if (ret == KSFT_SKIP) + ksft_test_result_skip("Skipping test_size_filter\n"); + else + ksft_test_result(ret == KSFT_PASS, "test_size_filter\n"); + + ret = test_lineno_filter(); + if (ret == KSFT_SKIP) + ksft_test_result_skip("Skipping test_lineno_filter\n"); + else + ksft_test_result(ret == KSFT_PASS, "test_lineno_filter\n"); + ksft_finished(); } -- cgit v1.2.3 From a44730dd05a3f3a663f79fde62d6731393046655 Mon Sep 17 00:00:00 2001 From: Zhiling Zou Date: Fri, 31 Jul 2026 11:22:51 +0800 Subject: mm: shmem: reject page-aligned fallocate end overflow shmem_fallocate() validates offset + len with inode_newsize_ok(), but then rounds that end offset up to a page boundary before entering the preallocation loop. For a valid request ending at MAX_LFS_FILESIZE, such as offset = 0 and len = LLONG_MAX, adding PAGE_SIZE - 1 to the validated end can overflow the signed loff_t used for the rounded end calculation. If that wrapped value is then converted into a page index, shmem_fallocate() can enter the folio allocation loop with an invalid range. Use check_add_overflow() when calculating the page-aligned end, and fail before entering the allocation loop if the rounded end cannot be represented. Link: https://lore.kernel.org/1929a466735dcbb9438936ff50b7a4fc2332a8a4.1785377919.git.zhilinz@nebusec.ai Fixes: e2d12e22c59c ("tmpfs: support fallocate preallocation") Signed-off-by: Zhiling Zou Reported-by: Vega Reviewed-by: Baolin Wang Cc: Hugh Dickins Signed-off-by: Andrew Morton --- mm/shmem.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/mm/shmem.c b/mm/shmem.c index 774f4b18ff5c..8ea776e52823 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -3617,6 +3617,7 @@ static long shmem_fallocate(struct file *file, int mode, loff_t offset, struct shmem_inode_info *info = SHMEM_I(inode); struct shmem_falloc shmem_falloc; pgoff_t start, index, end, undo_fallocend; + loff_t aligned_end; int error; if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE)) @@ -3673,8 +3674,15 @@ static long shmem_fallocate(struct file *file, int mode, loff_t offset, goto out; } + /* Check for wraparound */ + if (check_add_overflow(offset + len, (loff_t)PAGE_SIZE - 1, + &aligned_end)) { + error = -EFBIG; + goto out; + } + start = offset >> PAGE_SHIFT; - end = (offset + len + PAGE_SIZE - 1) >> PAGE_SHIFT; + end = aligned_end >> PAGE_SHIFT; /* Try to avoid a swapstorm if len is impossible to satisfy */ if (sbinfo->max_blocks && end - start > sbinfo->max_blocks) { error = -ENOSPC; -- cgit v1.2.3 From a8efc69a65fbef61bed0923c0de3425d74817c1c Mon Sep 17 00:00:00 2001 From: Christoph Hellwig Date: Mon, 13 Jul 2026 11:33:38 +0200 Subject: shmem: provide a shmem_write_folio wrapper Patch series "better block swap batching and a different take on swap_ops v5". This series makes use of the swap_iocb for block as well so that it doesn't do inefficient single-bio I/O, and then rebases the swap_ops from Baoquan on top of the now very different method structure. When running doing kernels builds, which is a workload that doesn't really do much THP anonymous memory it still gets 2x clustering for writeout and 1.2x for reading back swap in. The overall times do not actually change, though. This patch (of 7): Provide a wrapper for the shmem abuses in drm to prepare for swap I/O refactoring by keeping swap_iocb handling entirely contained in mm/. Link: https://lore.kernel.org/20260713093350.2154226-1-hch@lst.de Link: https://lore.kernel.org/20260713093350.2154226-2-hch@lst.de Signed-off-by: Christoph Hellwig Reviewed-by: Baoquan He Reviewed-by: Nhat Pham Reviewed-by: Baolin Wang Acked-by: Chris Li Reviewed-by: Kairui Song Cc: Kemeng Shi Cc: Barry Song Cc: Youngjun Park Signed-off-by: Andrew Morton --- drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 2 +- drivers/gpu/drm/ttm/ttm_backup.c | 2 +- include/linux/shmem_fs.h | 5 +---- mm/shmem.c | 7 ++++++- mm/swap.h | 4 ++++ 5 files changed, 13 insertions(+), 7 deletions(-) diff --git a/drivers/gpu/drm/i915/gem/i915_gem_shmem.c b/drivers/gpu/drm/i915/gem/i915_gem_shmem.c index 06543ae60706..ef9440166295 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_shmem.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_shmem.c @@ -325,7 +325,7 @@ void __shmem_writeback(size_t size, struct address_space *mapping) if (folio_mapped(folio)) folio_redirty_for_writepage(&wbc, folio); else - error = shmem_writeout(folio, NULL, NULL); + error = shmem_write_folio(folio); } } diff --git a/drivers/gpu/drm/ttm/ttm_backup.c b/drivers/gpu/drm/ttm/ttm_backup.c index 3c067aadc52d..0c2d53a13b2a 100644 --- a/drivers/gpu/drm/ttm/ttm_backup.c +++ b/drivers/gpu/drm/ttm/ttm_backup.c @@ -160,7 +160,7 @@ ttm_backup_backup_folio(struct file *backup, struct folio *folio, if (writeback && !folio_mapped(to_folio) && folio_clear_dirty_for_io(to_folio)) { folio_set_reclaim(to_folio); - ret = shmem_writeout(to_folio, NULL, NULL); + ret = shmem_write_folio(to_folio); if (!folio_test_writeback(to_folio)) folio_clear_reclaim(to_folio); if (ret == AOP_WRITEPAGE_ACTIVATE) diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h index e729b9b0e38d..5663dff53186 100644 --- a/include/linux/shmem_fs.h +++ b/include/linux/shmem_fs.h @@ -12,8 +12,6 @@ #include #include -struct swap_iocb; - /* inode in-kernel data */ #ifdef CONFIG_TMPFS_QUOTA @@ -123,8 +121,7 @@ static inline bool shmem_mapping(const struct address_space *mapping) void shmem_unlock_mapping(struct address_space *mapping); struct page *shmem_read_mapping_page_gfp(struct address_space *mapping, pgoff_t index, gfp_t gfp_mask); -int shmem_writeout(struct folio *folio, struct swap_iocb **plug, - struct list_head *folio_list); +int shmem_write_folio(struct folio *folio); void shmem_truncate_range(struct inode *inode, loff_t start, uoff_t end); int shmem_unuse(unsigned int type); diff --git a/mm/shmem.c b/mm/shmem.c index 8ea776e52823..d245e01416e9 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1751,7 +1751,12 @@ redirty: folio_mark_dirty(folio); return AOP_WRITEPAGE_ACTIVATE; /* Return with folio locked */ } -EXPORT_SYMBOL_GPL(shmem_writeout); + +int shmem_write_folio(struct folio *folio) +{ + return shmem_writeout(folio, NULL, NULL); +} +EXPORT_SYMBOL_GPL(shmem_write_folio); #if defined(CONFIG_NUMA) && defined(CONFIG_TMPFS) static void shmem_show_mpol(struct seq_file *seq, struct mempolicy *mpol) diff --git a/mm/swap.h b/mm/swap.h index 4e4c291bbfde..276b7975a9dc 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -486,4 +486,8 @@ static inline unsigned int folio_swap_flags(struct folio *folio) } #endif /* CONFIG_SWAP */ + +int shmem_writeout(struct folio *folio, struct swap_iocb **plug, + struct list_head *folio_list); + #endif /* _MM_SWAP_H */ -- cgit v1.2.3 From 8f29aa226f82d8f9e7cf75f0ed5964960c7fe682 Mon Sep 17 00:00:00 2001 From: Christoph Hellwig Date: Mon, 13 Jul 2026 11:33:39 +0200 Subject: mm/swap: introduce struct swap_io_ctx Generalize the context currently provided by double pointers to struct swap_iocb to an on-stack context. This cleans up the code and prepares for adding more fields and supporting batching multiple folios into a single bio for block-based swap as well. This new swap_io_ctx is required for all functions using it, the old way of allowing a NULL iocb for some callers is removed to keep the interface consistent. To reduce code duplication caused by this, a new swap_cache_read_folio_sync helper is added to consolidate the code to call swap_cache_read_folio with a local swap_io_ctx. The unpug helpers are renamed to use the submit wording as they are generalized. Link: https://lore.kernel.org/20260713093350.2154226-3-hch@lst.de Signed-off-by: Christoph Hellwig Reviewed-by: Nhat Pham Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Youngjun Park Signed-off-by: Andrew Morton --- mm/madvise.c | 16 +++++++-------- mm/page_io.c | 60 ++++++++++++++++++++++++++++++--------------------------- mm/shmem.c | 13 +++++++++---- mm/swap.h | 35 +++++++++++++++------------------ mm/swap_state.c | 53 +++++++++++++++++++++++++++++--------------------- mm/vmscan.c | 15 +++++++-------- mm/zswap.c | 4 +++- 7 files changed, 105 insertions(+), 91 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index bf9ce199935a..07a21ca31bad 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -188,7 +188,7 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start, unsigned long end, struct mm_walk *walk) { struct vm_area_struct *vma = walk->private; - struct swap_iocb *splug = NULL; + struct swap_io_ctx ctx = {}; pte_t *ptep = NULL; spinlock_t *ptl; unsigned long addr; @@ -212,15 +212,15 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start, pte_unmap_unlock(ptep, ptl); ptep = NULL; - folio = read_swap_cache_async(entry, GFP_HIGHUSER_MOVABLE, - vma, addr, &splug); + folio = read_swap_cache_async(&ctx, entry, GFP_HIGHUSER_MOVABLE, + vma, addr); if (folio) folio_put(folio); } if (ptep) pte_unmap_unlock(ptep, ptl); - swap_read_unplug(splug); + swap_read_submit(&ctx); cond_resched(); return 0; @@ -238,7 +238,7 @@ static void shmem_swapin_range(struct vm_area_struct *vma, XA_STATE(xas, &mapping->i_pages, linear_page_index(vma, start)); pgoff_t end_index = linear_page_index(vma, end) - 1; struct folio *folio; - struct swap_iocb *splug = NULL; + struct swap_io_ctx ctx = {}; rcu_read_lock(); xas_for_each(&xas, folio, end_index) { @@ -257,15 +257,15 @@ static void shmem_swapin_range(struct vm_area_struct *vma, xas_pause(&xas); rcu_read_unlock(); - folio = read_swap_cache_async(entry, mapping_gfp_mask(mapping), - vma, addr, &splug); + folio = read_swap_cache_async(&ctx, entry, + mapping_gfp_mask(mapping), vma, addr); if (folio) folio_put(folio); rcu_read_lock(); } rcu_read_unlock(); - swap_read_unplug(splug); + swap_read_submit(&ctx); } #endif /* CONFIG_SWAP */ diff --git a/mm/page_io.c b/mm/page_io.c index b23f494fcc83..fe24a49e034c 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -248,7 +248,7 @@ static void swap_zeromap_folio_clear(struct folio *folio) * We may have stale swap cache pages in memory: notice * them here and get rid of the unnecessary final write. */ -int swap_writeout(struct folio *folio, struct swap_iocb **swap_plug) +int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio) { int ret = 0; @@ -295,7 +295,7 @@ int swap_writeout(struct folio *folio, struct swap_iocb **swap_plug) } rcu_read_unlock(); - __swap_writepage(folio, swap_plug); + __swap_writepage(ctx, folio); return 0; out_unlock: folio_unlock(folio); @@ -390,9 +390,9 @@ static void sio_write_complete(struct kiocb *iocb, long ret) mempool_free(sio, sio_pool); } -static void swap_writepage_fs(struct folio *folio, struct swap_iocb **swap_plug) +static void swap_writepage_fs(struct swap_io_ctx *ctx, struct folio *folio) { - struct swap_iocb *sio = swap_plug ? *swap_plug : NULL; + struct swap_iocb *sio = ctx->sio; struct swap_info_struct *sis = __swap_entry_to_info(folio->swap); struct file *swap_file = sis->swap_file; loff_t pos = swap_dev_pos(folio->swap); @@ -403,7 +403,7 @@ static void swap_writepage_fs(struct folio *folio, struct swap_iocb **swap_plug) if (sio) { if (sio->iocb.ki_filp != swap_file || sio->iocb.ki_pos + sio->len != pos) { - swap_write_unplug(sio); + swap_write_submit(ctx); sio = NULL; } } @@ -418,12 +418,11 @@ static void swap_writepage_fs(struct folio *folio, struct swap_iocb **swap_plug) bvec_set_folio(&sio->bvecs[sio->nr_bvecs], folio, folio_size(folio), 0); sio->len += folio_size(folio); sio->nr_bvecs += 1; - if (sio->nr_bvecs == ARRAY_SIZE(sio->bvecs) || !swap_plug) { - swap_write_unplug(sio); + if (sio->nr_bvecs == ARRAY_SIZE(sio->bvecs)) { + swap_write_submit(ctx); sio = NULL; } - if (swap_plug) - *swap_plug = sio; + ctx->sio = sio; } static void swap_writepage_bdev_sync(struct folio *folio, @@ -463,7 +462,7 @@ static void swap_writepage_bdev_async(struct folio *folio, submit_bio(bio); } -void __swap_writepage(struct folio *folio, struct swap_iocb **swap_plug) +void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio) { struct swap_info_struct *sis = __swap_entry_to_info(folio->swap); @@ -474,7 +473,7 @@ void __swap_writepage(struct folio *folio, struct swap_iocb **swap_plug) * is safe. */ if (data_race(sis->flags & SWP_FS_OPS)) - swap_writepage_fs(folio, swap_plug); + swap_writepage_fs(ctx, folio); /* * ->flags can be updated non-atomically, * but that will never affect SWP_SYNCHRONOUS_IO, so the data_race @@ -486,16 +485,20 @@ void __swap_writepage(struct folio *folio, struct swap_iocb **swap_plug) swap_writepage_bdev_async(folio, sis); } -void swap_write_unplug(struct swap_iocb *sio) +void swap_write_submit(struct swap_io_ctx *ctx) { + struct swap_iocb *sio = ctx->sio; struct iov_iter from; - struct address_space *mapping = sio->iocb.ki_filp->f_mapping; int ret; + if (!sio) + return; + iov_iter_bvec(&from, ITER_SOURCE, sio->bvecs, sio->nr_bvecs, sio->len); - ret = mapping->a_ops->swap_rw(&sio->iocb, &from); + ret = sio->iocb.ki_filp->f_mapping->a_ops->swap_rw(&sio->iocb, &from); if (ret != -EIOCBQUEUED) sio_write_complete(&sio->iocb, ret); + ctx->sio = NULL; } static void sio_read_complete(struct kiocb *iocb, long ret) @@ -587,18 +590,16 @@ static bool swap_read_folio_zeromap(struct folio *folio) return true; } -static void swap_read_folio_fs(struct folio *folio, struct swap_iocb **plug) +static void swap_read_folio_fs(struct swap_io_ctx *ctx, struct folio *folio) { struct swap_info_struct *sis = __swap_entry_to_info(folio->swap); - struct swap_iocb *sio = NULL; + struct swap_iocb *sio = ctx->sio; loff_t pos = swap_dev_pos(folio->swap); - if (plug) - sio = *plug; if (sio) { if (sio->iocb.ki_filp != sis->swap_file || sio->iocb.ki_pos + sio->len != pos) { - swap_read_unplug(sio); + swap_read_submit(ctx); sio = NULL; } } @@ -613,12 +614,11 @@ static void swap_read_folio_fs(struct folio *folio, struct swap_iocb **plug) bvec_set_folio(&sio->bvecs[sio->nr_bvecs], folio, folio_size(folio), 0); sio->len += folio_size(folio); sio->nr_bvecs += 1; - if (sio->nr_bvecs == ARRAY_SIZE(sio->bvecs) || !plug) { - swap_read_unplug(sio); + if (sio->nr_bvecs == ARRAY_SIZE(sio->bvecs)) { + swap_read_submit(ctx); sio = NULL; } - if (plug) - *plug = sio; + ctx->sio = sio; } static void swap_read_folio_bdev_sync(struct folio *folio, @@ -658,7 +658,7 @@ static void swap_read_folio_bdev_async(struct folio *folio, submit_bio(bio); } -void swap_read_folio(struct folio *folio, struct swap_iocb **plug) +void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio) { struct swap_info_struct *sis = __swap_entry_to_info(folio->swap); bool synchronous = sis->flags & SWP_SYNCHRONOUS_IO; @@ -693,7 +693,7 @@ void swap_read_folio(struct folio *folio, struct swap_iocb **plug) zswap_folio_swapin(folio); if (data_race(sis->flags & SWP_FS_OPS)) { - swap_read_folio_fs(folio, plug); + swap_read_folio_fs(ctx, folio); } else if (synchronous) { swap_read_folio_bdev_sync(folio, sis); } else { @@ -708,14 +708,18 @@ finish: delayacct_swapin_end(); } -void __swap_read_unplug(struct swap_iocb *sio) +void swap_read_submit(struct swap_io_ctx *ctx) { + struct swap_iocb *sio = ctx->sio; struct iov_iter from; - struct address_space *mapping = sio->iocb.ki_filp->f_mapping; int ret; + if (!sio) + return; + iov_iter_bvec(&from, ITER_DEST, sio->bvecs, sio->nr_bvecs, sio->len); - ret = mapping->a_ops->swap_rw(&sio->iocb, &from); + ret = sio->iocb.ki_filp->f_mapping->a_ops->swap_rw(&sio->iocb, &from); if (ret != -EIOCBQUEUED) sio_read_complete(&sio->iocb, ret); + ctx->sio = NULL; } diff --git a/mm/shmem.c b/mm/shmem.c index d245e01416e9..2e4dacdcce11 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1597,13 +1597,13 @@ start_over: /** * shmem_writeout - Write the folio to swap + * @ctx: swap I/O context * @folio: The folio to write - * @plug: swap plug * @folio_list: list to put back folios on split * * Move the folio from the page cache to the swap cache. */ -int shmem_writeout(struct folio *folio, struct swap_iocb **plug, +int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio, struct list_head *folio_list) { struct address_space *mapping = folio->mapping; @@ -1715,7 +1715,7 @@ try_split: shmem_delete_from_page_cache(folio, swp_to_radix_entry(folio->swap)); BUG_ON(folio_mapped(folio)); - error = swap_writeout(folio, plug); + error = swap_writeout(ctx, folio); if (error != AOP_WRITEPAGE_ACTIVATE) { /* folio has been unlocked */ return error; @@ -1754,7 +1754,12 @@ redirty: int shmem_write_folio(struct folio *folio) { - return shmem_writeout(folio, NULL, NULL); + struct swap_io_ctx ctx = {}; + int err; + + err = shmem_writeout(&ctx, folio, NULL); + swap_write_submit(&ctx); + return err; } EXPORT_SYMBOL_GPL(shmem_write_folio); diff --git a/mm/swap.h b/mm/swap.h index 276b7975a9dc..70f8e287f140 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -91,6 +91,10 @@ static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg) return READ_ONCE(vm_swappiness); } +struct swap_io_ctx { + struct swap_iocb *sio; +}; + #ifdef CONFIG_SWAP #include /* for swp_offset */ #include /* for bio_end_io_t */ @@ -253,17 +257,11 @@ extern void __swap_cluster_free_entries(struct swap_info_struct *si, /* linux/mm/page_io.c */ int sio_pool_init(void); -struct swap_iocb; -void swap_read_folio(struct folio *folio, struct swap_iocb **plug); -void __swap_read_unplug(struct swap_iocb *plug); -static inline void swap_read_unplug(struct swap_iocb *plug) -{ - if (unlikely(plug)) - __swap_read_unplug(plug); -} -void swap_write_unplug(struct swap_iocb *sio); -int swap_writeout(struct folio *folio, struct swap_iocb **swap_plug); -void __swap_writepage(struct folio *folio, struct swap_iocb **swap_plug); +void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio); +void swap_read_submit(struct swap_io_ctx *ctx); +void swap_write_submit(struct swap_io_ctx *ctx); +int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio); +void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio); /* linux/mm/swap_state.c */ extern struct address_space swap_space __read_mostly; @@ -330,9 +328,8 @@ void __swap_cache_replace_folio(struct swap_cluster_info *ci, void show_swap_cache_info(void); void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry, int nr); -struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, - struct vm_area_struct *vma, unsigned long addr, - struct swap_iocb **plug); +struct folio *read_swap_cache_async(struct swap_io_ctx *ctx, swp_entry_t entry, + gfp_t gfp_mask, struct vm_area_struct *vma, unsigned long addr); struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, struct mempolicy *mpol, pgoff_t ilx); struct folio *swapin_readahead(swp_entry_t entry, gfp_t flag, @@ -348,7 +345,6 @@ static inline unsigned int folio_swap_flags(struct folio *folio) } #else /* CONFIG_SWAP */ -struct swap_iocb; static inline struct swap_cluster_info *swap_cluster_lock( struct swap_info_struct *si, pgoff_t offset, bool irq) { @@ -394,11 +390,11 @@ static inline void folio_put_swap(struct folio *folio, struct page *page) { } -static inline void swap_read_folio(struct folio *folio, struct swap_iocb **plug) +static inline void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio) { } -static inline void swap_write_unplug(struct swap_iocb *sio) +static inline void swap_write_submit(struct swap_io_ctx *ctx) { } @@ -440,8 +436,7 @@ static inline void swap_update_readahead(struct folio *folio, { } -static inline int swap_writeout(struct folio *folio, - struct swap_iocb **swap_plug) +static inline int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio) { return 0; } @@ -487,7 +482,7 @@ static inline unsigned int folio_swap_flags(struct folio *folio) #endif /* CONFIG_SWAP */ -int shmem_writeout(struct folio *folio, struct swap_iocb **plug, +int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio, struct list_head *folio_list); #endif /* _MM_SWAP_H */ diff --git a/mm/swap_state.c b/mm/swap_state.c index 1444d20a40e9..5be825911e64 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -638,9 +638,9 @@ void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma, } } -static struct folio *swap_cache_read_folio(swp_entry_t entry, gfp_t gfp, - struct mempolicy *mpol, pgoff_t ilx, - struct swap_iocb **plug, bool readahead) +static struct folio *swap_cache_read_folio(struct swap_io_ctx *ctx, + swp_entry_t entry, gfp_t gfp, struct mempolicy *mpol, + pgoff_t ilx, bool readahead) { struct folio *folio; @@ -654,7 +654,7 @@ static struct folio *swap_cache_read_folio(swp_entry_t entry, gfp_t gfp, if (IS_ERR_OR_NULL(folio)) return NULL; - swap_read_folio(folio, plug); + swap_read_folio(ctx, folio); if (readahead) { folio_set_readahead(folio); count_vm_event(SWAP_RA); @@ -682,6 +682,7 @@ static struct folio *swap_cache_read_folio(swp_entry_t entry, gfp_t gfp, struct folio *swapin_sync(swp_entry_t entry, gfp_t gfp, unsigned long orders, struct vm_fault *vmf, struct mempolicy *mpol, pgoff_t ilx) { + struct swap_io_ctx ctx = {}; struct folio *folio; do { @@ -694,7 +695,8 @@ struct folio *swapin_sync(swp_entry_t entry, gfp_t gfp, unsigned long orders, if (IS_ERR(folio)) return folio; - swap_read_folio(folio, NULL); + swap_read_folio(&ctx, folio); + swap_read_submit(&ctx); return folio; } @@ -704,9 +706,8 @@ struct folio *swapin_sync(swp_entry_t entry, gfp_t gfp, unsigned long orders, * A failure return means that either the page allocation failed or that * the swap entry is no longer in use. */ -struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, - struct vm_area_struct *vma, unsigned long addr, - struct swap_iocb **plug) +struct folio *read_swap_cache_async(struct swap_io_ctx *ctx, swp_entry_t entry, + gfp_t gfp_mask, struct vm_area_struct *vma, unsigned long addr) { struct swap_info_struct *si; struct mempolicy *mpol; @@ -718,13 +719,24 @@ struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, return NULL; mpol = get_vma_policy(vma, addr, 0, &ilx); - folio = swap_cache_read_folio(entry, gfp_mask, mpol, ilx, plug, false); + folio = swap_cache_read_folio(ctx, entry, gfp_mask, mpol, ilx, false); mpol_cond_put(mpol); put_swap_device(si); return folio; } +static struct folio *swap_cache_read_folio_sync(swp_entry_t entry, gfp_t gfp, + struct mempolicy *mpol, pgoff_t ilx) +{ + struct swap_io_ctx ctx = {}; + struct folio *folio; + + folio = swap_cache_read_folio(&ctx, entry, gfp, mpol, ilx, false); + swap_read_submit(&ctx); + return folio; +} + static unsigned int __swapin_nr_pages(unsigned long prev_offset, unsigned long offset, int hits, @@ -813,8 +825,8 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, unsigned long start_offset, end_offset; unsigned long mask; struct swap_info_struct *si = __swap_entry_to_info(entry); + struct swap_io_ctx ctx = {}; struct blk_plug plug; - struct swap_iocb *splug = NULL; swp_entry_t ra_entry; mask = swapin_nr_pages(offset) - 1; @@ -833,17 +845,16 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, for (offset = start_offset; offset <= end_offset ; offset++) { /* Ok, do the async read-ahead now */ ra_entry = swp_entry(swp_type(entry), offset); - folio = swap_cache_read_folio(ra_entry, gfp_mask, mpol, ilx, - &splug, offset != entry_offset); + folio = swap_cache_read_folio(&ctx, ra_entry, gfp_mask, mpol, + ilx, offset != entry_offset); if (!folio) continue; folio_put(folio); } blk_finish_plug(&plug); - swap_read_unplug(splug); + swap_read_submit(&ctx); skip: - /* The page was likely read above, so no need for plugging here */ - return swap_cache_read_folio(entry, gfp_mask, mpol, ilx, NULL, false); + return swap_cache_read_folio_sync(entry, gfp_mask, mpol, ilx); } static int swap_vma_ra_win(struct vm_fault *vmf, unsigned long *start, @@ -903,8 +914,8 @@ static int swap_vma_ra_win(struct vm_fault *vmf, unsigned long *start, static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, struct mempolicy *mpol, pgoff_t targ_ilx, struct vm_fault *vmf) { + struct swap_io_ctx ctx = {}; struct blk_plug plug; - struct swap_iocb *splug = NULL; struct folio *folio; pte_t *pte = NULL, pentry; int win; @@ -943,8 +954,8 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, if (!si) continue; } - folio = swap_cache_read_folio(entry, gfp_mask, mpol, ilx, - &splug, addr != vmf->address); + folio = swap_cache_read_folio(&ctx, entry, gfp_mask, mpol, ilx, + addr != vmf->address); if (si) put_swap_device(si); if (!folio) @@ -954,12 +965,10 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, if (pte) pte_unmap(pte); blk_finish_plug(&plug); - swap_read_unplug(splug); + swap_read_submit(&ctx); skip: /* The folio was likely read above, so no need for plugging here */ - folio = swap_cache_read_folio(targ_entry, gfp_mask, mpol, targ_ilx, - NULL, false); - return folio; + return swap_cache_read_folio_sync(targ_entry, gfp_mask, mpol, targ_ilx); } /** diff --git a/mm/vmscan.c b/mm/vmscan.c index 3a6701143620..4742297693fe 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -617,8 +617,8 @@ typedef enum { /* * pageout is called by shrink_folio_list() for each dirty folio. */ -static pageout_t pageout(struct folio *folio, struct address_space *mapping, - struct swap_iocb **plug, struct list_head *folio_list) +static pageout_t pageout(struct swap_io_ctx *ctx, struct address_space *mapping, + struct folio *folio, struct list_head *folio_list) { int res; @@ -654,9 +654,9 @@ static pageout_t pageout(struct folio *folio, struct address_space *mapping, * the split out folios get added back to folio_list. */ if (shmem_mapping(mapping)) - res = shmem_writeout(folio, plug, folio_list); + res = shmem_writeout(ctx, folio, folio_list); else - res = swap_writeout(folio, plug); + res = swap_writeout(ctx, folio); if (res < 0) handle_write_error(mapping, folio, res); @@ -1066,7 +1066,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, unsigned int nr_reclaimed = 0, nr_demoted = 0; unsigned int pgactivate = 0; bool do_demote_pass; - struct swap_iocb *plug = NULL; + struct swap_io_ctx ctx = {}; folio_batch_init(&free_folios); memset(stat, 0, sizeof(*stat)); @@ -1394,7 +1394,7 @@ retry: * starts and then write it out here. */ try_to_unmap_flush_dirty(); - switch (pageout(folio, mapping, &plug, folio_list)) { + switch (pageout(&ctx, mapping, folio, folio_list)) { case PAGE_KEEP: goto keep_locked; case PAGE_ACTIVATE: @@ -1582,8 +1582,7 @@ keep: list_splice(&ret_folios, folio_list); count_vm_events(PGACTIVATE, pgactivate); - if (plug) - swap_write_unplug(plug); + swap_write_submit(&ctx); return nr_reclaimed; } diff --git a/mm/zswap.c b/mm/zswap.c index 0b9435b4f57c..c33d496bfdb2 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -992,6 +992,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, struct folio *folio; struct mempolicy *mpol; struct swap_info_struct *si; + struct swap_io_ctx ctx = {}; int ret = 0; /* try to allocate swap cache folio */ @@ -1049,7 +1050,8 @@ static int zswap_writeback_entry(struct zswap_entry *entry, folio_set_reclaim(folio); /* start writeback */ - __swap_writepage(folio, NULL); + __swap_writepage(&ctx, folio); + swap_write_submit(&ctx); out: if (ret) { -- cgit v1.2.3 From dda8fb68b5907d0a3c6216e5e31e45adca582b68 Mon Sep 17 00:00:00 2001 From: Christoph Hellwig Date: Mon, 13 Jul 2026 11:33:40 +0200 Subject: mm/swap: also use struct swap_iocb for block I/O Block I/O benefits from batching just as much as remote file systems. Extend struct swap_iocb to support building a bio on the fly as well, and rewrite the block based swap code for it. This especially benefits submit_bio based drivers that do not have the block plugging available, but also saves allocating extra bios for blk-mq drivers. Add a pre-allocated bio to struct swap_iocb in a union with kiocb used for file system based swap so that struct swap_iocb can be used for all swap I/O, and initialize the pool for it unconditionally. Various low-level bdev and fs functions are now replaced with a unified can_merge/add/submit scheme. Note that the block based swap code now uses the same memcg-based check previously added for file system based swap as well. Link: https://lore.kernel.org/20260713093350.2154226-4-hch@lst.de Signed-off-by: Christoph Hellwig Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Youngjun Park Signed-off-by: Andrew Morton --- mm/page_io.c | 526 +++++++++++++++++++++++++++------------------------------- mm/swap.h | 1 + mm/swapfile.c | 9 +- 3 files changed, 254 insertions(+), 282 deletions(-) diff --git a/mm/page_io.c b/mm/page_io.c index fe24a49e034c..0195c25a77eb 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -28,54 +28,6 @@ #include "swap.h" #include "swap_table.h" -static void __end_swap_bio_write(struct bio *bio) -{ - struct folio *folio = bio_first_folio_all(bio); - - if (bio->bi_status) { - /* - * We failed to write the page out to swap-space. - * Re-dirty the page in order to avoid it being reclaimed. - * Also print a dire warning that things will go BAD (tm) - * very quickly. - * - * Also clear PG_reclaim to avoid folio_rotate_reclaimable() - */ - folio_mark_dirty(folio); - pr_alert_ratelimited("Write-error on swap-device (%u:%u:%llu)\n", - MAJOR(bio_dev(bio)), MINOR(bio_dev(bio)), - (unsigned long long)bio->bi_iter.bi_sector); - folio_clear_reclaim(folio); - } - folio_end_writeback(folio); -} - -static void end_swap_bio_write(struct bio *bio) -{ - __end_swap_bio_write(bio); - bio_put(bio); -} - -static void __end_swap_bio_read(struct bio *bio) -{ - struct folio *folio = bio_first_folio_all(bio); - - if (bio->bi_status) { - pr_alert_ratelimited("Read-error on swap-device (%u:%u:%llu)\n", - MAJOR(bio_dev(bio)), MINOR(bio_dev(bio)), - (unsigned long long)bio->bi_iter.bi_sector); - } else { - folio_mark_uptodate(folio); - } - folio_unlock(folio); -} - -static void end_swap_bio_read(struct bio *bio) -{ - __end_swap_bio_read(bio); - bio_put(bio); -} - int generic_swapfile_activate(struct swap_info_struct *sis, struct file *swap_file, sector_t *span) @@ -316,18 +268,36 @@ static inline void count_swpout_vm_event(struct folio *folio) } #if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP) +static struct cgroup_subsys_state *folio_memcg_blkg_css(struct folio *folio) +{ + return cgroup_e_css(folio_memcg(folio)->css.cgroup, &io_cgrp_subsys); +} + +static bool folio_blkg_can_merge(struct folio *folio, struct folio *prev_folio) +{ + bool can_merge = true; + + if (folio_memcg_charged(folio) != folio_memcg_charged(prev_folio)) + return false; + if (folio_memcg_charged(folio)) { + rcu_read_lock(); + if (folio_memcg_blkg_css(folio) != + folio_memcg_blkg_css(prev_folio)) + can_merge = false; + rcu_read_unlock(); + } + return can_merge; +} + static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio) { struct cgroup_subsys_state *css; - struct mem_cgroup *memcg; if (!folio_memcg_charged(folio)) return; - rcu_read_lock(); - memcg = folio_memcg(folio); - css = cgroup_e_css(memcg->css.cgroup, &io_cgrp_subsys); - if (!css || !css_tryget(css)) + css = folio_memcg_blkg_css(folio); + if (css && !css_tryget(css)) css = NULL; rcu_read_unlock(); @@ -336,11 +306,18 @@ static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio) css_put(css); } #else +static bool folio_blkg_can_merge(struct folio *folio, struct folio *prev_folio) +{ + return true; +} #define bio_associate_blkg_from_page(bio, folio) do { } while (0) #endif /* CONFIG_MEMCG && CONFIG_BLK_CGROUP */ struct swap_iocb { - struct kiocb iocb; + union { + struct kiocb iocb; + struct bio bio; + }; struct bio_vec bvecs[SWAP_CLUSTER_MAX]; int nr_bvecs; int len; @@ -360,171 +337,70 @@ int sio_pool_init(void) return 0; } -static void sio_write_complete(struct kiocb *iocb, long ret) +static bool swap_can_merge(struct swap_io_ctx *ctx, struct folio *folio, + int rw) { - struct swap_iocb *sio = container_of(iocb, struct swap_iocb, iocb); - struct page *page = sio->bvecs[0].bv_page; - int p; + struct swap_info_struct *sis = __swap_entry_to_info(folio->swap); + struct bio_vec *last_bv = &ctx->sio->bvecs[ctx->sio->nr_bvecs - 1]; + struct folio *prev_folio = bvec_folio(last_bv); + size_t prev_folio_size = folio_size(prev_folio); - if (ret != sio->len) { - /* - * In the case of swap-over-nfs, this can be a - * temporary failure if the system has limited - * memory for allocating transmit buffers. - * Mark the page dirty and avoid - * folio_rotate_reclaimable but rate-limit the - * messages. - */ - pr_err_ratelimited("Write error %ld on dio swapfile (%llu)\n", - ret, swap_dev_pos(page_swap_entry(page))); - for (p = 0; p < sio->nr_bvecs; p++) { - page = sio->bvecs[p].bv_page; - set_page_dirty(page); - ClearPageReclaim(page); - } - } + if (ctx->sis != sis) + return false; - for (p = 0; p < sio->nr_bvecs; p++) - end_page_writeback(sio->bvecs[p].bv_page); + if (sis->flags & SWP_FS_OPS) { + if (swap_dev_pos(folio->swap) != + swap_dev_pos(prev_folio->swap) + prev_folio_size) + return false; + } else { + if (swap_folio_sector(folio) != + swap_folio_sector(prev_folio) + + (prev_folio_size >> SECTOR_SHIFT)) + return false; + if (rw == WRITE && !folio_blkg_can_merge(folio, prev_folio)) + return false; + } - mempool_free(sio, sio_pool); + return true; } -static void swap_writepage_fs(struct swap_io_ctx *ctx, struct folio *folio) +static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw) { - struct swap_iocb *sio = ctx->sio; struct swap_info_struct *sis = __swap_entry_to_info(folio->swap); - struct file *swap_file = sis->swap_file; - loff_t pos = swap_dev_pos(folio->swap); + struct swap_iocb *sio = ctx->sio; - count_swpout_vm_event(folio); - folio_start_writeback(folio); - folio_unlock(folio); - if (sio) { - if (sio->iocb.ki_filp != swap_file || - sio->iocb.ki_pos + sio->len != pos) { + if (sio && !swap_can_merge(ctx, folio, rw)) { + if (rw == WRITE) swap_write_submit(ctx); - sio = NULL; - } + else + swap_read_submit(ctx); + sio = ctx->sio; } + if (!sio) { - sio = mempool_alloc(sio_pool, GFP_NOIO); - init_sync_kiocb(&sio->iocb, swap_file); - sio->iocb.ki_complete = sio_write_complete; - sio->iocb.ki_pos = pos; + ctx->sis = sis; + ctx->sio = sio = mempool_alloc(sio_pool, GFP_NOIO); sio->nr_bvecs = 0; sio->len = 0; } bvec_set_folio(&sio->bvecs[sio->nr_bvecs], folio, folio_size(folio), 0); sio->len += folio_size(folio); - sio->nr_bvecs += 1; - if (sio->nr_bvecs == ARRAY_SIZE(sio->bvecs)) { - swap_write_submit(ctx); - sio = NULL; + if (++sio->nr_bvecs == ARRAY_SIZE(sio->bvecs)) { + if (rw == WRITE) + swap_write_submit(ctx); + else + swap_read_submit(ctx); } - ctx->sio = sio; } -static void swap_writepage_bdev_sync(struct folio *folio, - struct swap_info_struct *sis) -{ - struct bio_vec bv; - struct bio bio; - - bio_init(&bio, sis->bdev, &bv, 1, REQ_OP_WRITE | REQ_SWAP); - bio.bi_iter.bi_sector = swap_folio_sector(folio); - bio_add_folio_nofail(&bio, folio, folio_size(folio), 0); - - bio_associate_blkg_from_page(&bio, folio); - count_swpout_vm_event(folio); - - folio_start_writeback(folio); - folio_unlock(folio); - - submit_bio_wait(&bio); - __end_swap_bio_write(&bio); -} - -static void swap_writepage_bdev_async(struct folio *folio, - struct swap_info_struct *sis) +void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio) { - struct bio *bio; - - bio = bio_alloc(sis->bdev, 1, REQ_OP_WRITE | REQ_SWAP, GFP_NOIO); - bio->bi_iter.bi_sector = swap_folio_sector(folio); - bio->bi_end_io = end_swap_bio_write; - bio_add_folio_nofail(bio, folio, folio_size(folio), 0); + VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio); - bio_associate_blkg_from_page(bio, folio); count_swpout_vm_event(folio); folio_start_writeback(folio); folio_unlock(folio); - submit_bio(bio); -} - -void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio) -{ - struct swap_info_struct *sis = __swap_entry_to_info(folio->swap); - - VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio); - /* - * ->flags can be updated non-atomically, - * but that will never affect SWP_FS_OPS, so the data_race - * is safe. - */ - if (data_race(sis->flags & SWP_FS_OPS)) - swap_writepage_fs(ctx, folio); - /* - * ->flags can be updated non-atomically, - * but that will never affect SWP_SYNCHRONOUS_IO, so the data_race - * is safe. - */ - else if (data_race(sis->flags & SWP_SYNCHRONOUS_IO)) - swap_writepage_bdev_sync(folio, sis); - else - swap_writepage_bdev_async(folio, sis); -} - -void swap_write_submit(struct swap_io_ctx *ctx) -{ - struct swap_iocb *sio = ctx->sio; - struct iov_iter from; - int ret; - - if (!sio) - return; - - iov_iter_bvec(&from, ITER_SOURCE, sio->bvecs, sio->nr_bvecs, sio->len); - ret = sio->iocb.ki_filp->f_mapping->a_ops->swap_rw(&sio->iocb, &from); - if (ret != -EIOCBQUEUED) - sio_write_complete(&sio->iocb, ret); - ctx->sio = NULL; -} - -static void sio_read_complete(struct kiocb *iocb, long ret) -{ - struct swap_iocb *sio = container_of(iocb, struct swap_iocb, iocb); - int p; - - if (ret == sio->len) { - for (p = 0; p < sio->nr_bvecs; p++) { - struct folio *folio = bvec_folio(&sio->bvecs[p]); - - count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN); - count_memcg_folio_events(folio, PSWPIN, folio_nr_pages(folio)); - folio_mark_uptodate(folio); - folio_unlock(folio); - } - count_vm_events(PSWPIN, sio->len >> PAGE_SHIFT); - } else { - for (p = 0; p < sio->nr_bvecs; p++) { - struct folio *folio = bvec_folio(&sio->bvecs[p]); - - folio_unlock(folio); - } - pr_alert_ratelimited("Read-error on swap-device\n"); - } - mempool_free(sio, sio_pool); + swap_add_folio(ctx, folio, WRITE); } /* @@ -590,74 +466,6 @@ static bool swap_read_folio_zeromap(struct folio *folio) return true; } -static void swap_read_folio_fs(struct swap_io_ctx *ctx, struct folio *folio) -{ - struct swap_info_struct *sis = __swap_entry_to_info(folio->swap); - struct swap_iocb *sio = ctx->sio; - loff_t pos = swap_dev_pos(folio->swap); - - if (sio) { - if (sio->iocb.ki_filp != sis->swap_file || - sio->iocb.ki_pos + sio->len != pos) { - swap_read_submit(ctx); - sio = NULL; - } - } - if (!sio) { - sio = mempool_alloc(sio_pool, GFP_KERNEL); - init_sync_kiocb(&sio->iocb, sis->swap_file); - sio->iocb.ki_pos = pos; - sio->iocb.ki_complete = sio_read_complete; - sio->nr_bvecs = 0; - sio->len = 0; - } - bvec_set_folio(&sio->bvecs[sio->nr_bvecs], folio, folio_size(folio), 0); - sio->len += folio_size(folio); - sio->nr_bvecs += 1; - if (sio->nr_bvecs == ARRAY_SIZE(sio->bvecs)) { - swap_read_submit(ctx); - sio = NULL; - } - ctx->sio = sio; -} - -static void swap_read_folio_bdev_sync(struct folio *folio, - struct swap_info_struct *sis) -{ - struct bio_vec bv; - struct bio bio; - - bio_init(&bio, sis->bdev, &bv, 1, REQ_OP_READ); - bio.bi_iter.bi_sector = swap_folio_sector(folio); - bio_add_folio_nofail(&bio, folio, folio_size(folio), 0); - /* - * Keep this task valid during swap readpage because the oom killer may - * attempt to access it in the page fault retry time check. - */ - get_task_struct(current); - count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN); - count_memcg_folio_events(folio, PSWPIN, folio_nr_pages(folio)); - count_vm_events(PSWPIN, folio_nr_pages(folio)); - submit_bio_wait(&bio); - __end_swap_bio_read(&bio); - put_task_struct(current); -} - -static void swap_read_folio_bdev_async(struct folio *folio, - struct swap_info_struct *sis) -{ - struct bio *bio; - - bio = bio_alloc(sis->bdev, 1, REQ_OP_READ, GFP_KERNEL); - bio->bi_iter.bi_sector = swap_folio_sector(folio); - bio->bi_end_io = end_swap_bio_read; - bio_add_folio_nofail(bio, folio, folio_size(folio), 0); - count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN); - count_memcg_folio_events(folio, PSWPIN, folio_nr_pages(folio)); - count_vm_events(PSWPIN, folio_nr_pages(folio)); - submit_bio(bio); -} - void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio) { struct swap_info_struct *sis = __swap_entry_to_info(folio->swap); @@ -691,14 +499,7 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio) /* We have to read from slower devices. Increase zswap protection. */ zswap_folio_swapin(folio); - - if (data_race(sis->flags & SWP_FS_OPS)) { - swap_read_folio_fs(ctx, folio); - } else if (synchronous) { - swap_read_folio_bdev_sync(folio, sis); - } else { - swap_read_folio_bdev_async(folio, sis); - } + swap_add_folio(ctx, folio, READ); finish: if (workingset) { @@ -708,18 +509,189 @@ finish: delayacct_swapin_end(); } -void swap_read_submit(struct swap_io_ctx *ctx) +static void swap_write_end(struct swap_iocb *sio, bool failed) +{ + int p; + + for (p = 0; p < sio->nr_bvecs; p++) { + struct page *page = sio->bvecs[p].bv_page; + + if (failed) { + set_page_dirty(page); + ClearPageReclaim(page); + } + end_page_writeback(page); + } + mempool_free(sio, sio_pool); +} + +static void swap_fs_write_complete(struct kiocb *iocb, long ret) +{ + struct swap_iocb *sio = container_of(iocb, struct swap_iocb, iocb); + bool failed = ret != sio->len; + + if (failed) { + struct page *page = sio->bvecs[0].bv_page; + + /* + * In the case of swap-over-nfs, this can be a temporary failure + * if the system has limited memory for allocating transmit + * buffers. Mark the page dirty and avoid + * folio_rotate_reclaimable but rate-limit the messages. + */ + pr_err_ratelimited("Write error %ld on dio swapfile (%llu)\n", + ret, swap_dev_pos(page_swap_entry(page))); + } + + swap_write_end(sio, failed); +} + +static void end_swap_bio_write(struct bio *bio) +{ + struct swap_iocb *sio = container_of(bio, struct swap_iocb, bio); + bool failed = !!bio->bi_status; + + if (failed) + pr_alert_ratelimited("Write-error on swap-device (%u:%u:%llu)\n", + MAJOR(bio_dev(bio)), MINOR(bio_dev(bio)), + (unsigned long long)bio->bi_iter.bi_sector); + bio_uninit(bio); + swap_write_end(sio, failed); +} + +static void swap_read_end(struct swap_iocb *sio, bool failed) +{ + int p; + + for (p = 0; p < sio->nr_bvecs; p++) { + struct folio *folio = bvec_folio(&sio->bvecs[p]); + + if (!failed) { + count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN); + count_memcg_folio_events(folio, PSWPIN, + folio_nr_pages(folio)); + folio_mark_uptodate(folio); + } + folio_unlock(folio); + } + + if (!failed) + count_vm_events(PSWPIN, sio->len >> PAGE_SHIFT); + + mempool_free(sio, sio_pool); +} + +static void swap_fs_read_complete(struct kiocb *iocb, long ret) +{ + struct swap_iocb *sio = container_of(iocb, struct swap_iocb, iocb); + bool failed = ret != sio->len; + + if (failed) + pr_alert_ratelimited("Read-error on swap-device\n"); + swap_read_end(sio, failed); +} + +static void swap_bio_read_end_io(struct bio *bio) +{ + struct swap_iocb *sio = container_of(bio, struct swap_iocb, bio); + bool failed = !!bio->bi_status; + + if (failed) + pr_alert_ratelimited("Read-error on swap-device (%u:%u:%llu)\n", + MAJOR(bio_dev(bio)), MINOR(bio_dev(bio)), + (unsigned long long)bio->bi_iter.bi_sector); + bio_uninit(bio); + swap_read_end(sio, failed); +} + +static void swap_bdev_submit_write(struct swap_io_ctx *ctx) { struct swap_iocb *sio = ctx->sio; - struct iov_iter from; + struct bio *bio = &sio->bio; + + bio_init(bio, ctx->sis->bdev, sio->bvecs, ARRAY_SIZE(sio->bvecs), + REQ_OP_WRITE | REQ_SWAP); + bio->bi_iter.bi_size = sio->len; + bio->bi_iter.bi_sector = swap_folio_sector(bio_first_folio_all(bio)); + bio_associate_blkg_from_page(bio, bio_first_folio_all(bio)); + + if (ctx->sis->flags & SWP_SYNCHRONOUS_IO) { + submit_bio_wait(bio); + end_swap_bio_write(bio); + } else { + bio->bi_end_io = end_swap_bio_write; + submit_bio(bio); + } +} + +static void swap_bdev_submit_read(struct swap_io_ctx *ctx) +{ + struct swap_iocb *sio = ctx->sio; + struct bio *bio = &sio->bio; + + bio_init(bio, ctx->sis->bdev, sio->bvecs, ARRAY_SIZE(sio->bvecs), + REQ_OP_READ); + bio->bi_iter.bi_size = sio->len; + bio->bi_iter.bi_sector = swap_folio_sector(bio_first_folio_all(bio)); + + if (ctx->sis->flags & SWP_SYNCHRONOUS_IO) { + /* + * Keep this task valid during swap readpage because the oom + * killer may attempt to access it in the page fault retry + * time check. + */ + get_task_struct(current); + submit_bio_wait(bio); + swap_bio_read_end_io(bio); + put_task_struct(current); + } else { + bio->bi_end_io = swap_bio_read_end_io; + submit_bio(bio); + } +} + +static void swap_fs_submit(struct swap_io_ctx *ctx, int rw) +{ + struct swap_iocb *sio = ctx->sio; + struct iov_iter iter; int ret; - if (!sio) - return; + init_sync_kiocb(&sio->iocb, ctx->sis->swap_file); + sio->iocb.ki_pos = swap_dev_pos(bvec_folio(&sio->bvecs[0])->swap); + if (rw == WRITE) + sio->iocb.ki_complete = swap_fs_write_complete; + else + sio->iocb.ki_complete = swap_fs_read_complete; - iov_iter_bvec(&from, ITER_DEST, sio->bvecs, sio->nr_bvecs, sio->len); - ret = sio->iocb.ki_filp->f_mapping->a_ops->swap_rw(&sio->iocb, &from); + iov_iter_bvec(&iter, rw == WRITE ? ITER_SOURCE : ITER_DEST, + sio->bvecs, sio->nr_bvecs, sio->len); + ret = sio->iocb.ki_filp->f_mapping->a_ops->swap_rw(&sio->iocb, &iter); if (ret != -EIOCBQUEUED) - sio_read_complete(&sio->iocb, ret); + sio->iocb.ki_complete(&sio->iocb, ret); +} + +void swap_write_submit(struct swap_io_ctx *ctx) +{ + if (!ctx->sio) + return; + + if (ctx->sis->flags & SWP_FS_OPS) + swap_fs_submit(ctx, WRITE); + else + swap_bdev_submit_write(ctx); + ctx->sio = NULL; + ctx->sis = NULL; +} + +void swap_read_submit(struct swap_io_ctx *ctx) +{ + if (!ctx->sio) + return; + + if (ctx->sis->flags & SWP_FS_OPS) + swap_fs_submit(ctx, READ); + else + swap_bdev_submit_read(ctx); ctx->sio = NULL; + ctx->sis = NULL; } diff --git a/mm/swap.h b/mm/swap.h index 70f8e287f140..86b2a241b734 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -93,6 +93,7 @@ static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg) struct swap_io_ctx { struct swap_iocb *sio; + struct swap_info_struct *sis; }; #ifdef CONFIG_SWAP diff --git a/mm/swapfile.c b/mm/swapfile.c index d7f749ad60c2..be75c995c49b 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -2959,6 +2959,10 @@ static int setup_swap_extents(struct swap_info_struct *sis, struct inode *inode = mapping->host; int ret; + ret = sio_pool_init(); + if (ret) + return ret; + if (S_ISBLK(inode->i_mode)) { ret = add_swap_extent(sis, 0, sis->max, 0); *span = sis->pages; @@ -2970,11 +2974,6 @@ static int setup_swap_extents(struct swap_info_struct *sis, if (ret < 0) return ret; sis->flags |= SWP_ACTIVATED; - if ((sis->flags & SWP_FS_OPS) && - sio_pool_init() != 0) { - destroy_swap_extents(sis, swap_file); - return -ENOMEM; - } return ret; } -- cgit v1.2.3 From 4e915b16ded278e57519e54793ab84b4a094fa83 Mon Sep 17 00:00:00 2001 From: Christoph Hellwig Date: Mon, 13 Jul 2026 11:33:41 +0200 Subject: mm/swap: remove count_swpout_vm_event There is only one caller left, so merge it into that. Link: https://lore.kernel.org/20260713093350.2154226-5-hch@lst.de Signed-off-by: Christoph Hellwig Reviewed-by: Baoquan He Reviewed-by: Nhat Pham Cc: Baolin Wang Cc: Barry Song Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Youngjun Park Signed-off-by: Andrew Morton --- mm/page_io.c | 24 ++++++++++-------------- 1 file changed, 10 insertions(+), 14 deletions(-) diff --git a/mm/page_io.c b/mm/page_io.c index 0195c25a77eb..56f21e49572e 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -254,19 +254,6 @@ out_unlock: return ret; } -static inline void count_swpout_vm_event(struct folio *folio) -{ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - if (unlikely(folio_test_pmd_mappable(folio))) { - count_memcg_folio_events(folio, THP_SWPOUT, 1); - count_vm_event(THP_SWPOUT); - } -#endif - count_mthp_stat(folio_order(folio), MTHP_STAT_SWPOUT); - count_memcg_folio_events(folio, PSWPOUT, folio_nr_pages(folio)); - count_vm_events(PSWPOUT, folio_nr_pages(folio)); -} - #if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP) static struct cgroup_subsys_state *folio_memcg_blkg_css(struct folio *folio) { @@ -397,7 +384,16 @@ void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio) { VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio); - count_swpout_vm_event(folio); +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + if (unlikely(folio_test_pmd_mappable(folio))) { + count_memcg_folio_events(folio, THP_SWPOUT, 1); + count_vm_event(THP_SWPOUT); + } +#endif + count_mthp_stat(folio_order(folio), MTHP_STAT_SWPOUT); + count_memcg_folio_events(folio, PSWPOUT, folio_nr_pages(folio)); + count_vm_events(PSWPOUT, folio_nr_pages(folio)); + folio_start_writeback(folio); folio_unlock(folio); swap_add_folio(ctx, folio, WRITE); -- cgit v1.2.3 From 563597895e65113b4dafea6cacd1df23b0cf6660 Mon Sep 17 00:00:00 2001 From: Christoph Hellwig Date: Mon, 13 Jul 2026 11:33:42 +0200 Subject: mm/swap: use swap_ops to register swap device's methods This simplifies codes and makes logic clearer. And also makes later any new swap device type being added easier to handle. Currently there are two types of swap devices: fs and bdev. [hch@lst.de: updated for the new submit and can_merge abstraction] Link: https://lore.kernel.org/20260713093350.2154226-6-hch@lst.de Signed-off-by: Baoquan He Signed-off-by: Christoph Hellwig Suggested-by: Chris Li Reviewed-by: Nhat Pham Cc: Baolin Wang Cc: Barry Song Cc: Kairui Song Cc: Kemeng Shi Cc: Youngjun Park Signed-off-by: Andrew Morton --- include/linux/swap.h | 1 + mm/page_io.c | 68 +++++++++++++++++++++++++++++++++------------------- mm/swap.h | 10 ++++++++ mm/swapfile.c | 4 ++++ 4 files changed, 58 insertions(+), 25 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index b4b1c0a84c8b..5979b1427368 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -276,6 +276,7 @@ struct swap_info_struct { struct work_struct reclaim_work; /* reclaim worker */ struct list_head discard_clusters; /* discard clusters list */ struct plist_node avail_list; /* entry in swap_avail_head */ + const struct swap_ops *ops; }; static inline swp_entry_t page_swap_entry(struct page *page) diff --git a/mm/page_io.c b/mm/page_io.c index 56f21e49572e..c36b44ffe947 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -334,21 +334,7 @@ static bool swap_can_merge(struct swap_io_ctx *ctx, struct folio *folio, if (ctx->sis != sis) return false; - - if (sis->flags & SWP_FS_OPS) { - if (swap_dev_pos(folio->swap) != - swap_dev_pos(prev_folio->swap) + prev_folio_size) - return false; - } else { - if (swap_folio_sector(folio) != - swap_folio_sector(prev_folio) + - (prev_folio_size >> SECTOR_SHIFT)) - return false; - if (rw == WRITE && !folio_blkg_can_merge(folio, prev_folio)) - return false; - } - - return true; + return sis->ops->can_merge(folio, prev_folio, prev_folio_size, rw); } static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw) @@ -646,6 +632,23 @@ static void swap_bdev_submit_read(struct swap_io_ctx *ctx) } } +static bool swap_bdev_can_merge(struct folio *folio, struct folio *prev_folio, + size_t prev_folio_size, int rw) +{ + if (swap_folio_sector(folio) != + swap_folio_sector(prev_folio) + (prev_folio_size >> SECTOR_SHIFT)) + return false; + if (rw == WRITE && !folio_blkg_can_merge(folio, prev_folio)) + return false; + return true; +} + +const struct swap_ops swap_bdev_ops = { + .submit_write = swap_bdev_submit_write, + .submit_read = swap_bdev_submit_read, + .can_merge = swap_bdev_can_merge, +}; + static void swap_fs_submit(struct swap_io_ctx *ctx, int rw) { struct swap_iocb *sio = ctx->sio; @@ -666,15 +669,34 @@ static void swap_fs_submit(struct swap_io_ctx *ctx, int rw) sio->iocb.ki_complete(&sio->iocb, ret); } +static void swap_fs_submit_write(struct swap_io_ctx *ctx) +{ + swap_fs_submit(ctx, WRITE); +} + +static void swap_fs_submit_read(struct swap_io_ctx *ctx) +{ + swap_fs_submit(ctx, READ); +} + +static bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio, + size_t prev_folio_size, int rw) +{ + return swap_dev_pos(folio->swap) == + swap_dev_pos(prev_folio->swap) + prev_folio_size; +} + +const struct swap_ops swap_fs_ops = { + .submit_write = swap_fs_submit_write, + .submit_read = swap_fs_submit_read, + .can_merge = swap_fs_can_merge, +}; + void swap_write_submit(struct swap_io_ctx *ctx) { if (!ctx->sio) return; - - if (ctx->sis->flags & SWP_FS_OPS) - swap_fs_submit(ctx, WRITE); - else - swap_bdev_submit_write(ctx); + ctx->sis->ops->submit_write(ctx); ctx->sio = NULL; ctx->sis = NULL; } @@ -683,11 +705,7 @@ void swap_read_submit(struct swap_io_ctx *ctx) { if (!ctx->sio) return; - - if (ctx->sis->flags & SWP_FS_OPS) - swap_fs_submit(ctx, READ); - else - swap_bdev_submit_read(ctx); + ctx->sis->ops->submit_read(ctx); ctx->sio = NULL; ctx->sis = NULL; } diff --git a/mm/swap.h b/mm/swap.h index 86b2a241b734..ffc36695d4ac 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -96,6 +96,13 @@ struct swap_io_ctx { struct swap_info_struct *sis; }; +struct swap_ops { + bool (*can_merge)(struct folio *folio, struct folio *prev_folio, + size_t prev_folio_size, int rw); + void (*submit_write)(struct swap_io_ctx *ctx); + void (*submit_read)(struct swap_io_ctx *ctx); +}; + #ifdef CONFIG_SWAP #include /* for swp_offset */ #include /* for bio_end_io_t */ @@ -483,6 +490,9 @@ static inline unsigned int folio_swap_flags(struct folio *folio) #endif /* CONFIG_SWAP */ +extern const struct swap_ops swap_bdev_ops; +extern const struct swap_ops swap_fs_ops; + int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio, struct list_head *folio_list); diff --git a/mm/swapfile.c b/mm/swapfile.c index be75c995c49b..ad623dae483b 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -2963,6 +2963,8 @@ static int setup_swap_extents(struct swap_info_struct *sis, if (ret) return ret; + sis->ops = &swap_bdev_ops; + if (S_ISBLK(inode->i_mode)) { ret = add_swap_extent(sis, 0, sis->max, 0); *span = sis->pages; @@ -2973,6 +2975,8 @@ static int setup_swap_extents(struct swap_info_struct *sis, ret = mapping->a_ops->swap_activate(sis, swap_file, span); if (ret < 0) return ret; + if (sis->flags & SWP_FS_OPS) + sis->ops = &swap_fs_ops; sis->flags |= SWP_ACTIVATED; return ret; } -- cgit v1.2.3 From 0df74c11587941b35596d1e8990dcab06bdbfeb5 Mon Sep 17 00:00:00 2001 From: Christoph Hellwig Date: Mon, 13 Jul 2026 11:33:43 +0200 Subject: mm/swap: remove SWP_FS_OPS Provide a swap_fs_activate helper that directly sets up swap_fs_ops, and a flag in struct swap_ops to indicate of NOFS swapping is allowed. Link: https://lore.kernel.org/20260713093350.2154226-7-hch@lst.de Signed-off-by: Christoph Hellwig Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Youngjun Park Signed-off-by: Andrew Morton --- Documentation/filesystems/locking.rst | 5 +++-- Documentation/filesystems/vfs.rst | 4 ++-- fs/nfs/file.c | 4 +--- fs/smb/client/file.c | 4 +--- include/linux/swap.h | 6 +++++- mm/page_io.c | 10 +++++++++- mm/swap.h | 22 ++++++++++------------ mm/swapfile.c | 2 -- mm/vmscan.c | 15 +++++++-------- 9 files changed, 38 insertions(+), 34 deletions(-) diff --git a/Documentation/filesystems/locking.rst b/Documentation/filesystems/locking.rst index 08d01bc62c31..1a50d41a39a1 100644 --- a/Documentation/filesystems/locking.rst +++ b/Documentation/filesystems/locking.rst @@ -355,13 +355,14 @@ should perform any validation and preparation necessary to ensure that writes can be performed with minimal memory allocation. It should call add_swap_extent(), or the helper iomap_swapfile_activate(), and return the number of extents added. If IO should be submitted through -->swap_rw(), it should set SWP_FS_OPS, otherwise IO will be submitted +->swap_rw(), it should call swap_fs_activate, otherwise IO will be submitted directly to the block device ``sis->bdev``. ->swap_deactivate() will be called in the sys_swapoff() path after ->swap_activate() returned success. -->swap_rw will be called for swap IO if SWP_FS_OPS was set by ->swap_activate(). +->swap_rw will be called for swap IO if swap_fs_activate was called by +->swap_activate(). file_lock_operations ==================== diff --git a/Documentation/filesystems/vfs.rst b/Documentation/filesystems/vfs.rst index 7c753148af88..e7677423a20f 100644 --- a/Documentation/filesystems/vfs.rst +++ b/Documentation/filesystems/vfs.rst @@ -977,7 +977,7 @@ cache in your filesystem. The following members are defined: can be performed with minimal memory allocation. It should call add_swap_extent(), or the helper iomap_swapfile_activate(), and return the number of extents added. If IO should be submitted - through ->swap_rw(), it should set SWP_FS_OPS, otherwise IO will + through ->swap_rw(), it should call swap_fs_activate, otherwise IO will be submitted directly to the block device ``sis->bdev``. ``swap_deactivate`` @@ -985,7 +985,7 @@ cache in your filesystem. The following members are defined: successful. ``swap_rw`` - Called to read or write swap pages when SWP_FS_OPS is set. + Called to read or write swap pages when swap_fs_activate was called. The File Object =============== diff --git a/fs/nfs/file.c b/fs/nfs/file.c index a0d8f1c1cf10..851d93a09988 100644 --- a/fs/nfs/file.c +++ b/fs/nfs/file.c @@ -597,7 +597,7 @@ static int nfs_swap_activate(struct swap_info_struct *sis, struct file *file, ret = rpc_clnt_swap_activate(clnt); if (ret) return ret; - ret = add_swap_extent(sis, 0, sis->max, 0); + ret = swap_fs_activate(sis); if (ret < 0) { rpc_clnt_swap_deactivate(clnt); return ret; @@ -607,8 +607,6 @@ static int nfs_swap_activate(struct swap_info_struct *sis, struct file *file, if (cl->rpc_ops->enable_swap) cl->rpc_ops->enable_swap(inode); - - sis->flags |= SWP_FS_OPS; return ret; } diff --git a/fs/smb/client/file.c b/fs/smb/client/file.c index b279a44be729..7f2924ce2881 100644 --- a/fs/smb/client/file.c +++ b/fs/smb/client/file.c @@ -3451,9 +3451,7 @@ static int cifs_swap_activate(struct swap_info_struct *sis, * but we could add call to grab a byte range lock to prevent others * from reading or writing the file */ - - sis->flags |= SWP_FS_OPS; - return add_swap_extent(sis, 0, sis->max, 0); + return swap_fs_activate(sis); } static void cifs_swap_deactivate(struct file *file) diff --git a/include/linux/swap.h b/include/linux/swap.h index 5979b1427368..8dd68733c955 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -202,7 +202,6 @@ enum { SWP_SOLIDSTATE = (1 << 4), /* blkdev seeks are cheap */ SWP_BLKDEV = (1 << 6), /* its a block device */ SWP_ACTIVATED = (1 << 7), /* set after swap_activate success */ - SWP_FS_OPS = (1 << 8), /* swapfile operations go through fs */ SWP_AREA_DISCARD = (1 << 9), /* single-time swap area discards */ SWP_PAGE_DISCARD = (1 << 10), /* freed swap page-cluster discards */ SWP_STABLE_WRITES = (1 << 11), /* no overwrite PG_writeback pages */ @@ -343,6 +342,7 @@ extern void __meminit kswapd_stop(int nid); #ifdef CONFIG_SWAP +int swap_fs_activate(struct swap_info_struct *sis); int add_swap_extent(struct swap_info_struct *sis, unsigned long start_page, unsigned long nr_pages, sector_t start_block); int generic_swapfile_activate(struct swap_info_struct *, struct file *, @@ -468,6 +468,10 @@ static inline bool folio_free_swap(struct folio *folio) return false; } +static inline int swap_fs_activate(struct swap_info_struct *sis) +{ + return -EINVAL; +} static inline int add_swap_extent(struct swap_info_struct *sis, unsigned long start_page, unsigned long nr_pages, sector_t start_block) diff --git a/mm/page_io.c b/mm/page_io.c index c36b44ffe947..cea438b66bce 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -686,12 +686,20 @@ static bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio, swap_dev_pos(prev_folio->swap) + prev_folio_size; } -const struct swap_ops swap_fs_ops = { +static const struct swap_ops swap_fs_ops = { + .flags = SWAP_OPS_F_REQUIRE_NOFS, .submit_write = swap_fs_submit_write, .submit_read = swap_fs_submit_read, .can_merge = swap_fs_can_merge, }; +int swap_fs_activate(struct swap_info_struct *sis) +{ + sis->ops = &swap_fs_ops; + return add_swap_extent(sis, 0, sis->max, 0); +} +EXPORT_SYMBOL_GPL(swap_fs_activate); + void swap_write_submit(struct swap_io_ctx *ctx) { if (!ctx->sio) diff --git a/mm/swap.h b/mm/swap.h index ffc36695d4ac..1a78578fd067 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -96,7 +96,17 @@ struct swap_io_ctx { struct swap_info_struct *sis; }; +/* + * SWAP_OPS_F_REQUIRE_NOFS: + * When set, all reclaim operations must operated as GFS_NOFS and not + * just GFP_NOIO, as GFP_NOIO allocations could recourse into the + * file system backing this swap file. + */ +#define SWAP_OPS_F_REQUIRE_NOFS (1U << 0) + struct swap_ops { + unsigned int flags; + bool (*can_merge)(struct folio *folio, struct folio *prev_folio, size_t prev_folio_size, int rw); void (*submit_write)(struct swap_io_ctx *ctx); @@ -347,11 +357,6 @@ struct folio *swapin_sync(swp_entry_t entry, gfp_t flag, unsigned long orders, void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma, unsigned long addr); -static inline unsigned int folio_swap_flags(struct folio *folio) -{ - return __swap_entry_to_info(folio->swap)->flags; -} - #else /* CONFIG_SWAP */ static inline struct swap_cluster_info *swap_cluster_lock( struct swap_info_struct *si, pgoff_t offset, bool irq) @@ -482,16 +487,9 @@ static inline void __swap_cache_replace_folio(struct swap_cluster_info *ci, struct folio *old, struct folio *new) { } - -static inline unsigned int folio_swap_flags(struct folio *folio) -{ - return 0; -} - #endif /* CONFIG_SWAP */ extern const struct swap_ops swap_bdev_ops; -extern const struct swap_ops swap_fs_ops; int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio, struct list_head *folio_list); diff --git a/mm/swapfile.c b/mm/swapfile.c index ad623dae483b..dacef34a3ed7 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -2975,8 +2975,6 @@ static int setup_swap_extents(struct swap_info_struct *sis, ret = mapping->a_ops->swap_activate(sis, swap_file, span); if (ret < 0) return ret; - if (sis->flags & SWP_FS_OPS) - sis->ops = &swap_fs_ops; sis->flags |= SWP_ACTIVATED; return ret; } diff --git a/mm/vmscan.c b/mm/vmscan.c index 4742297693fe..3194da7dcc79 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1040,16 +1040,15 @@ static bool may_enter_fs(struct folio *folio, gfp_t gfp_mask) { if (gfp_mask & __GFP_FS) return true; - if (!folio_test_swapcache(folio) || !(gfp_mask & __GFP_IO)) - return false; /* - * We can "enter_fs" for swap-cache with only __GFP_IO - * providing this isn't SWP_FS_OPS. - * ->flags can be updated non-atomically, - * but that will never affect SWP_FS_OPS, so the data_race - * is safe. + * We can "enter_fs" for swap-cache with only __GFP_IO unless backed by + * a swapfile that requires GFP_NOFS I/O. */ - return !data_race(folio_swap_flags(folio) & SWP_FS_OPS); + if (folio_test_swapcache(folio) && (gfp_mask & __GFP_IO) && + !(__swap_entry_to_info(folio->swap)->ops->flags & + SWAP_OPS_F_REQUIRE_NOFS)) + return true; + return false; } /* -- cgit v1.2.3 From c01e6df60e7be422ee5ce5e2a76d43fb05fab4c2 Mon Sep 17 00:00:00 2001 From: Christoph Hellwig Date: Mon, 13 Jul 2026 11:33:44 +0200 Subject: mm/vmstat: add NRSWP{IN,OUT} counters Count how many swap I/Os we cause. Due to batching this can be different than the current counter number of pages written/read, and tracking this information is useful to see how efficient the batching is. The counters are added at the end of enum vm_event_item and the vmstat_text array under the assumption that the order of fields in /proc/vmstat is an ABI. If that is not the case, they could be grouped with the other swap counters. Link: https://lore.kernel.org/20260713093350.2154226-8-hch@lst.de Signed-off-by: Christoph Hellwig Reviewed-by: Nhat Pham Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Youngjun Park Signed-off-by: Andrew Morton --- include/linux/vm_event_item.h | 4 ++++ mm/page_io.c | 2 ++ mm/vmstat.c | 6 +++++- 3 files changed, 11 insertions(+), 1 deletion(-) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 03fe95f5a020..2628ccda076a 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -175,6 +175,10 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, KSTACK_REST, #endif #endif /* CONFIG_DEBUG_STACK_USAGE */ +#ifdef CONFIG_SWAP + NRSWPIN, + NRSWPOUT, +#endif /* CONFIG_SWAP */ NR_VM_EVENT_ITEMS }; diff --git a/mm/page_io.c b/mm/page_io.c index cea438b66bce..e4fa7ffffe8b 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -704,6 +704,7 @@ void swap_write_submit(struct swap_io_ctx *ctx) { if (!ctx->sio) return; + count_vm_events(NRSWPOUT, 1); ctx->sis->ops->submit_write(ctx); ctx->sio = NULL; ctx->sis = NULL; @@ -713,6 +714,7 @@ void swap_read_submit(struct swap_io_ctx *ctx) { if (!ctx->sio) return; + count_vm_events(NRSWPIN, 1); ctx->sis->ops->submit_read(ctx); ctx->sio = NULL; ctx->sis = NULL; diff --git a/mm/vmstat.c b/mm/vmstat.c index 7d6e61a01f51..cb57714539fb 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1502,7 +1502,11 @@ const char * const vmstat_text[] = { #if THREAD_SIZE > 65536 [I(KSTACK_REST)] = "kstack_rest", #endif -#endif +#endif /* CONFIG_DEBUG_STACK_USAGE */ +#ifdef CONFIG_SWAP + [I(NRSWPIN)] = "nrswpin", + [I(NRSWPOUT)] = "nrswpout", +#endif /* CONFIG_SWAP */ #undef I #endif /* CONFIG_VM_EVENT_COUNTERS */ }; -- cgit v1.2.3 From b090524f775b412d12c34b71d6d39fe46aa72e30 Mon Sep 17 00:00:00 2001 From: Hongfu Li Date: Fri, 17 Jul 2026 15:11:04 +0800 Subject: mm/swap: fix swap_cluster_lock() !CONFIG_SWAP stub signature mismatch The !CONFIG_SWAP stub for swap_cluster_lock() has mismatched prototype: it has an extra unused irq argument and uses pgoff_t instead of unsigned long for offset. All callers are under CONFIG_SWAP so the extra parameter is dead. Delete the unused stub function entirely. Link: https://lore.kernel.org/20260717071104.73467-1-hongfu.li@linux.dev Signed-off-by: Hongfu Li Reviewed-by: Baoquan He Acked-by: Kairui Song Cc: Barry Song Cc: Chris Li Cc: Hongfu Li Cc: Kemeng Shi Cc: Nhat Pham Signed-off-by: Andrew Morton --- mm/swap.h | 5 ----- 1 file changed, 5 deletions(-) diff --git a/mm/swap.h b/mm/swap.h index 1a78578fd067..48379b2ab202 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -358,11 +358,6 @@ void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma, unsigned long addr); #else /* CONFIG_SWAP */ -static inline struct swap_cluster_info *swap_cluster_lock( - struct swap_info_struct *si, pgoff_t offset, bool irq) -{ - return NULL; -} static inline struct swap_cluster_info *swap_cluster_get_and_lock( struct folio *folio) -- cgit v1.2.3 From 1f2b4b28aafeeeded553812c70f4a8bd054ac38a Mon Sep 17 00:00:00 2001 From: Yunzhao Li Date: Thu, 2 Jul 2026 11:07:35 -0700 Subject: mm/zswap: use ratelimited stats flush in zswap_shrinker_count() zswap_shrinker_count() calls mem_cgroup_flush_stats(), which takes the global cgroup rstat lock synchronously. On machines with many CPUs and NUMA nodes, this creates severe lock contention in the kswapd reclaim path: - Multiple kswapd threads (one per NUMA node) run concurrently. - do_shrink_slab() invokes zswap_shrinker_count() for each memcg-aware shrinker pass. - Each call flushes the full cgroup rstat hierarchy under the global lock. On AMD EPYC 9684X machines (96 cores, 192 threads, 12 NUMA nodes) running production workloads with zswap enabled, perf shows 2.88% of kernel cycles in osq_lock contention from this path: 2.88% [k] osq_lock --__mutex_lock.constprop.0 --__cgroup_rstat_lock --cgroup_rstat_flush_locked --cgroup_rstat_flush --zswap_shrinker_count do_shrink_slab shrink_slab shrink_node balance_pgdat kswapd 84% of kswapd kernel cycles are spent in shrink_slab -> zswap_shrinker_count -> cgroup_rstat_flush, not in actual page reclaim (shrink_lruvec). Controlled A/B on identical hardware and workload: shrinker=Y: 2.88% osq_lock, memory PSI 1.58% shrinker=N: 0.00% osq_lock, memory PSI 0.57% eBPF-based rstat lock wait measurement across 8 production metals confirms the contention splits cleanly along shrinker enablement: shrinker=Y: 50-250x more contended lock acquisitions (248/s vs 1.1/s) shrinker=N: baseline lock wait (0.0017 s/s vs 1.04 s/s) zswap_shrinker_count() only produces a heuristic estimate, scaled by compression ratio via mult_frac(). The actual writeback happens in zswap_shrinker_scan(). Slightly stale stats are acceptable here. Switch to mem_cgroup_flush_stats_ratelimited(), which only flushes if the periodic 2-second flusher is one full cycle late. This matches the approach already used in prepare_scan_control() (mm/vmscan.c) for the same reclaim path. After applying this patch, rstat flush latency and lock wait time on shrinker=Y machines dropped to the same level as shrinker=N controls, while the zswap shrinker continues to function (pool size remains bounded under the max_pool_percent cap). Previously discussed: - Chengming Zhou (Dec 2023): rstat contention from zswap_shrinker_count [1] - Shakeel Butt (Aug 2024): zswap_shrinker_count still uses sync flush [2] - Yosry Ahmed (Aug 2024): suggested eliminating in-kernel flushers [3] - Jesper Dangaard Brouer (Sep 2024): cgroup/rstat V11 patch [4] Link: https://lore.kernel.org/20260702180908.150136-1-yunzhao@cloudflare.com Link: https://lore.kernel.org/linux-mm/20231206103935.3440502-1-zhouchengming@bytedance.com/ [1] Link: https://lore.kernel.org/linux-mm/CALvZod7LFxLCxVpOFH8b2Ppm8T40HPGMKQwX_=NPCWB_mFW+oQ@mail.gmail.com/ [2] Link: https://lore.kernel.org/linux-mm/CAJD7tkYvFyOSX+rP_FKGBhxvZiCDxtpsNp-c5CGOA-4Bq9oXSg@mail.gmail.com/ [3] Link: https://lore.kernel.org/linux-mm/172616070094.2055617.17676042522679701515.stgit@firesoul/ [4] Suggested-by: Jesper Dangaard Brouer Signed-off-by: Jesper Dangaard Brouer Signed-off-by: Yunzhao Li Tested-by: Yunzhao Li Acked-by: Johannes Weiner Acked-by: Jesper Dangaard Brouer Acked-by: Nhat Pham Cc: Chengming Zhou Cc: Shakeel Butt Cc: Yosry Ahmed Cc: Yunzhao Li Cc: Sourav Panda Signed-off-by: Andrew Morton --- mm/zswap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/zswap.c b/mm/zswap.c index c33d496bfdb2..9f777a48b106 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1219,7 +1219,7 @@ static unsigned long zswap_shrinker_count(struct shrinker *shrinker, * Without memcg, use the zswap pool-wide metrics. */ if (!mem_cgroup_disabled()) { - mem_cgroup_flush_stats(memcg); + mem_cgroup_flush_stats_ratelimited(memcg); nr_backing = memcg_page_state(memcg, MEMCG_ZSWAP_B) >> PAGE_SHIFT; nr_stored = memcg_page_state(memcg, MEMCG_ZSWAPPED); } else { -- cgit v1.2.3 From 097492865fbef3af59200e31fef9148ecea80aad Mon Sep 17 00:00:00 2001 From: Hongfu Li Date: Mon, 10 Aug 2026 17:32:15 +0800 Subject: mm/cma: remove stray newline from auto-generated CMA area name When no name is supplied, cma_new_area() generates names with format "cma%d\n", introducing an unintended newline character ('\n') in the CMA name. Most CMA regions are created with explicit names, so this path is seldom hit. The newline only creates cosmetic noise in debug logs, traces and debugfs with no functional impact. Link: https://lore.kernel.org/20260810093215.91419-1-hongfu.li@linux.dev Signed-off-by: Hongfu Li Acked-by: David Hildenbrand (Arm) Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: SJ Park Reviewed-by: Anshuman Khandual Cc: Liam R. Howlett Cc: Michal Hocko Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- mm/cma.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/cma.c b/mm/cma.c index a7929c758df1..a10ea37a261d 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -242,7 +242,7 @@ static int __init cma_new_area(const char *name, phys_addr_t size, if (name) strscpy(cma->name, name); else - snprintf(cma->name, CMA_MAX_NAME, "cma%d\n", cma_area_count); + snprintf(cma->name, CMA_MAX_NAME, "cma%d", cma_area_count); cma->available_count = cma->count = size >> PAGE_SHIFT; cma->order_per_bit = order_per_bit; -- cgit v1.2.3 From 8790303cbaac52a11dfed4aab261f8ea60682525 Mon Sep 17 00:00:00 2001 From: Hui Su Date: Sat, 8 Aug 2026 11:14:59 +0800 Subject: kasan: fix cache shrink race with CPU hotplug kasan_quarantine_remove_cache() first invokes per_cpu_remove_cache() on all online CPUs. Each callback moves objects belonging to the cache from cpu_quarantine to the CPU's shrink_qlist, where they can later be freed from task context. kmem_cache_destroy() invokes the quarantine removal path while holding cpus_read_lock(), but kmem_cache_shrink() does not. The latter can therefore race with CPU offlining as follows: kmem_cache_shrink() CPU hotplug ------------------- ----------- on_each_cpu() CPU1 moves objects to CPU1's shrink_qlist on_each_cpu() returns CPU1 goes offline kasan_cpu_offline() drains cpu_quarantine leaves shrink_qlist untouched for_each_online_cpu() skips CPU1 The objects left on CPU1's shrink_qlist are not returned to the slab allocator. This may prevent kmem_cache_shrink() from releasing slabs that would otherwise become empty. If CPU1 remains offline, a later kmem_cache_destroy() also skips the list and can report that the cache still contains objects. An intermittent occurrence was observed with a virtio-9p filesystem. The mount and umount commands both returned 0, but the kernel logged the following during the userspace-triggered teardown: [ 2994.380134][ T111] BUG 9p-fcall-cache-1 (Tainted: G B ): Objects remaining on __kmem_cache_shutdown() [ 2994.381140][ T111] Object 0xff11000004361118 @offset=4376 [ 2994.381607][ T111] Allocated in p9_fcall_init+0x201/0x400 age=19564 cpu=1 pid=104 [ 2994.382591][ T111] p9_fcall_init+0x201/0x400 [ 2994.382810][ T111] p9_tag_alloc+0x12f/0x700 [ 2994.382982][ T111] p9_client_prepare_req+0x102/0x3e0 [ 2994.383165][ T111] p9_client_rpc+0x1ab/0xa50 [ 2994.383334][ T111] p9_client_getattr_dotl+0xb0/0x1a0 [ 2994.383515][ T111] v9fs_vfs_getattr_dotl+0x115/0x360 [ 2994.383719][ T111] vfs_getattr_nosec+0x22c/0x3a0 [ 2994.383910][ T111] vfs_statx+0xd7/0x170 [ 2994.384062][ T111] vfs_fstatat+0x45/0x80 [ 2994.384215][ T111] __do_sys_newfstatat+0x84/0xe0 [ 2994.384386][ T111] do_syscall_64+0x115/0x6a0 [ 2994.384566][ T111] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 2994.399720][ T111] WARNING: mm/slub.c:1244 at __kmem_cache_shutdown+0x363/0x500, CPU#0: busybox/111 [ 2994.405655][ T111] Call Trace: [ 2994.406325][ T111] kmem_cache_destroy+0x73/0x1b0 [ 2994.406630][ T111] p9_client_destroy+0x271/0x3c0 [ 2994.407210][ T111] v9fs_session_close+0x3c/0x260 [ 2994.407409][ T111] v9fs_kill_super+0x48/0x90 [ 2994.407584][ T111] deactivate_locked_super+0xa3/0x160 [ 2994.407778][ T111] cleanup_mnt+0x1dd/0x3e0 Thus, a successful umount left objects in the 9p fcall cache and prevented the cache from being destroyed cleanly. Per-CPU shrink_qlist storage exists for every possible CPU, and each list is protected by its own raw spinlock. Iterate over possible CPUs so that a list populated before its CPU went offline is drained as well. for_each_possible_cpu() can do more work than for_each_online_cpu(), but this change only affects CONFIG_KASAN_GENERIC kernels. The extra work is limited to cache shrink and cache destruction paths and does not affect the normal allocation/free fast path. It adds one raw-spinlock-protected scan of each possible CPU's shrink list. These lists are normally empty; a non-empty list is traversed to remove objects belonging to the cache being shrunk or destroyed. Link: https://lore.kernel.org/20260808031459.3032812-1-sh_def@163.com Fixes: 07d067e4f2ce ("kasan: fix sleeping function called from invalid context on RT kernel") Signed-off-by: Hui Su Reviewed-by: Andrey Ryabinin Cc: Alexander Potapenko Cc: Andrey Konovalov Cc: Dmitry Vyukov Cc: Vincenzo Frascino Cc: "Zhang, Qiang1" Cc: Signed-off-by: Andrew Morton --- mm/kasan/quarantine.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/mm/kasan/quarantine.c b/mm/kasan/quarantine.c index 6958aa713c67..16f4e67beee8 100644 --- a/mm/kasan/quarantine.c +++ b/mm/kasan/quarantine.c @@ -355,7 +355,12 @@ void kasan_quarantine_remove_cache(struct kmem_cache *cache) */ on_each_cpu(per_cpu_remove_cache, cache, 1); - for_each_online_cpu(cpu) { + /* + * A CPU can go offline after on_each_cpu() returns, leaving cache + * objects on that CPU's shrink list. Scan all possible CPUs to + * drain those lists. + */ + for_each_possible_cpu(cpu) { sq = per_cpu_ptr(&shrink_qlist, cpu); raw_spin_lock_irqsave(&sq->lock, flags); qlist_move_cache(&sq->qlist, &to_free, cache); -- cgit v1.2.3 From 534b19bbb67717eea2272573bc0ff7ba4ba109c1 Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Mon, 10 Aug 2026 13:31:14 +0200 Subject: mm/gup_test: keep longterm pin state per file The pin longterm test currently stores its data globally, shared among multiple concurrent users of the interface (multiple open file descriptors -> multiple "struct file"'s). That makes the gup_test interface problematic to use concurrently: two users, such as concurrent selftest runs, can interfere with the same longterm pin state. While this has not been observed as a problem so far in practice, let's just handle it cleanly. There could be a way to trigger selftest failures by e.g., running the cow.c and gup_longerm.c selftests concurrently, but we usually run them sequentially. Let's add a "Fixes" tag to be safe, but not need to CC stable. Link: https://lore.kernel.org/20260810-gup_test_data-v1-1-fb1d41be5bb4@kernel.org Fixes: c77369b437f9 ("mm/gup_test: start/stop/read functionality for PIN LONGTERM test") Signed-off-by: David Hildenbrand (Arm) Reported-by: yunhui cui Closes: https://lore.kernel.org/r/20260608025043.88087-1-cuiyunhui@bytedance.com Tested-by: Yunhui Cui Tested-by: Lance Yang Cc: Jason Gunthorpe Cc: John Hubbard Cc: Peter Xu Cc: Yang Li Signed-off-by: Andrew Morton --- mm/gup_test.c | 94 +++++++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 62 insertions(+), 32 deletions(-) diff --git a/mm/gup_test.c b/mm/gup_test.c index eb4c9cda16ed..44c1cdfb9c37 100644 --- a/mm/gup_test.c +++ b/mm/gup_test.c @@ -8,6 +8,12 @@ #include #include "gup_test.h" +struct gup_test_data { + struct mutex longterm_mutex; + struct page **longterm_pages; + unsigned long longterm_nr_pages; +}; + static void put_back_pages(unsigned int cmd, struct page **pages, unsigned long nr_pages, unsigned int gup_test_flags) { @@ -208,23 +214,20 @@ free_pages: return ret; } -static DEFINE_MUTEX(pin_longterm_test_mutex); -static struct page **pin_longterm_test_pages; -static unsigned long pin_longterm_test_nr_pages; - -static inline void pin_longterm_test_stop(void) +static inline void pin_longterm_test_stop(struct gup_test_data *data) { - if (pin_longterm_test_pages) { - if (pin_longterm_test_nr_pages) - unpin_user_pages(pin_longterm_test_pages, - pin_longterm_test_nr_pages); - kvfree(pin_longterm_test_pages); - pin_longterm_test_pages = NULL; - pin_longterm_test_nr_pages = 0; + if (data->longterm_pages) { + if (data->longterm_nr_pages) + unpin_user_pages(data->longterm_pages, + data->longterm_nr_pages); + kvfree(data->longterm_pages); + data->longterm_pages = NULL; + data->longterm_nr_pages = 0; } } -static inline int pin_longterm_test_start(unsigned long arg) +static inline int pin_longterm_test_start(struct gup_test_data *data, + unsigned long arg) { long nr_pages, cur_pages, addr, remaining_pages; int gup_flags = FOLL_LONGTERM; @@ -233,7 +236,7 @@ static inline int pin_longterm_test_start(unsigned long arg) int ret = 0; bool fast; - if (pin_longterm_test_pages) + if (data->longterm_pages) return -EINVAL; if (copy_from_user(&args, (void __user *)arg, sizeof(args))) @@ -263,12 +266,12 @@ static inline int pin_longterm_test_start(unsigned long arg) return -EINTR; } - pin_longterm_test_pages = pages; - pin_longterm_test_nr_pages = 0; + data->longterm_pages = pages; + data->longterm_nr_pages = 0; - while (nr_pages - pin_longterm_test_nr_pages) { - remaining_pages = nr_pages - pin_longterm_test_nr_pages; - addr = args.addr + pin_longterm_test_nr_pages * PAGE_SIZE; + while (nr_pages - data->longterm_nr_pages) { + remaining_pages = nr_pages - data->longterm_nr_pages; + addr = args.addr + data->longterm_nr_pages * PAGE_SIZE; if (fast) cur_pages = pin_user_pages_fast(addr, remaining_pages, @@ -277,11 +280,11 @@ static inline int pin_longterm_test_start(unsigned long arg) cur_pages = pin_user_pages(addr, remaining_pages, gup_flags, pages); if (cur_pages < 0) { - pin_longterm_test_stop(); + pin_longterm_test_stop(data); ret = cur_pages; break; } - pin_longterm_test_nr_pages += cur_pages; + data->longterm_nr_pages += cur_pages; pages += cur_pages; } @@ -290,19 +293,20 @@ static inline int pin_longterm_test_start(unsigned long arg) return ret; } -static inline int pin_longterm_test_read(unsigned long arg) +static inline int pin_longterm_test_read(struct gup_test_data *data, + unsigned long arg) { __u64 user_addr; unsigned long i; - if (!pin_longterm_test_pages) + if (!data->longterm_pages) return -EINVAL; if (copy_from_user(&user_addr, (void __user *)arg, sizeof(user_addr))) return -EFAULT; - for (i = 0; i < pin_longterm_test_nr_pages; i++) { - void *addr = kmap_local_page(pin_longterm_test_pages[i]); + for (i = 0; i < data->longterm_nr_pages; i++) { + void *addr = kmap_local_page(data->longterm_pages[i]); unsigned long ret; ret = copy_to_user((void __user *)(unsigned long)user_addr, addr, @@ -318,25 +322,26 @@ static inline int pin_longterm_test_read(unsigned long arg) static long pin_longterm_test_ioctl(struct file *filep, unsigned int cmd, unsigned long arg) { + struct gup_test_data *data = filep->private_data; int ret = -EINVAL; - if (mutex_lock_killable(&pin_longterm_test_mutex)) + if (mutex_lock_killable(&data->longterm_mutex)) return -EINTR; switch (cmd) { case PIN_LONGTERM_TEST_START: - ret = pin_longterm_test_start(arg); + ret = pin_longterm_test_start(data, arg); break; case PIN_LONGTERM_TEST_STOP: - pin_longterm_test_stop(); + pin_longterm_test_stop(data); ret = 0; break; case PIN_LONGTERM_TEST_READ: - ret = pin_longterm_test_read(arg); + ret = pin_longterm_test_read(data, arg); break; } - mutex_unlock(&pin_longterm_test_mutex); + mutex_unlock(&data->longterm_mutex); return ret; } @@ -375,15 +380,40 @@ static long gup_test_ioctl(struct file *filep, unsigned int cmd, return 0; } +static int gup_test_open(struct inode *inode, struct file *file) +{ + struct gup_test_data *data; + int ret; + + data = kzalloc_obj(*data); + if (!data) + return -ENOMEM; + + ret = nonseekable_open(inode, file); + if (ret) { + kfree(data); + return ret; + } + + mutex_init(&data->longterm_mutex); + file->private_data = data; + return 0; +} + static int gup_test_release(struct inode *inode, struct file *file) { - pin_longterm_test_stop(); + struct gup_test_data *data = file->private_data; + + pin_longterm_test_stop(data); + mutex_destroy(&data->longterm_mutex); + kfree(data); + file->private_data = NULL; return 0; } static const struct file_operations gup_test_fops = { - .open = nonseekable_open, + .open = gup_test_open, .unlocked_ioctl = gup_test_ioctl, .compat_ioctl = compat_ptr_ioctl, .release = gup_test_release, -- cgit v1.2.3 From fb496eb062306f0a447f6059012205e10ab7b900 Mon Sep 17 00:00:00 2001 From: Catalin Marinas Date: Mon, 13 Jul 2026 04:48:04 -0700 Subject: mm: kmemleak: confirm suspected leaks with a second scan Patch series "mm: kmemleak: reduce transient false positives by confirming leaks". This series combines two kmemleak enhancements that were originally submitted separately but both required rebasing after commit 79c37ae3733e9 ("mm/kmemleak: fix checksum computation for per-cpu objects"). The first feature introduces a second scan to confirm suspected leaks: https://lore.kernel.org/all/20260709173347.689607-1-catalin.marinas@arm.com/ The second feature adds a module parameter controlling the minimum number of consecutive unreferenced scans before a leak is reported, as discussed in: https://lore.kernel.org/all/20260626-kmemleak_twice-v1-0-ab28f7cc0971@debian.org/ Changes from v1: Now that commit 79c37ae3733e9 is upstream, the selftest includes an additional priming phase scan as requested by Catalin. Additionally, I've factored out the leak-detection conditional into a helper function to be more digestible for the reader's eye. This 4-patch series resolves all outstanding kmemleak issues I've been tracking. This patch (of 4): The kmemleak marking phase is not atomic. While the object graph is traversed, the kernel can modify pointers, free objects or allocate new ones. If a reference to an object is moved from one location to another, kmemleak scanning may miss it. We have explicit annotations like kmemleak_transient_leak() but identifying and maintaining them is not trivial. Given that such transient leaks are short-lived, rather than just reporting such objects as leaks, do another scan to confirm the suspected objects. If no new leaks are found during the first scan, skip the confirmation one. Link: https://lore.kernel.org/20260713-catalin_pto-v1-0-5b93b1131089@debian.org Link: https://lore.kernel.org/20260713-catalin_pto-v1-1-5b93b1131089@debian.org Signed-off-by: Catalin Marinas Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Breno Leitao Cc: David Hildenbrand Cc: Jonathan Corbet Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Geert Uytterhoeven Signed-off-by: Andrew Morton --- mm/kmemleak.c | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 54 insertions(+), 3 deletions(-) diff --git a/mm/kmemleak.c b/mm/kmemleak.c index 0a6045c857d6..95bd8ccd3c5b 100644 --- a/mm/kmemleak.c +++ b/mm/kmemleak.c @@ -175,6 +175,8 @@ struct kmemleak_object { #define OBJECT_PHYS (1 << 4) /* flag set for per-CPU pointers */ #define OBJECT_PERCPU (1 << 5) +/* flag set on an object left unreferenced by the full scan, pending confirmation */ +#define OBJECT_SUSPECT (1 << 6) /* set when __remove_object() called */ #define DELSTATE_REMOVED (1 << 0) @@ -235,6 +237,8 @@ static unsigned long jiffies_min_age; static unsigned long jiffies_last_scan; /* delay between automatic memory scannings */ static unsigned long jiffies_scan_wait; +/* number of objects flagged OBJECT_SUSPECT during the current scan */ +static int nr_suspects; /* enables or disables the task stacks scanning */ static int kmemleak_stack_scan = 1; /* protects the memory scanning, parameters and debug/kmemleak file access */ @@ -1440,6 +1444,11 @@ static void update_refs(struct kmemleak_object *object) */ object->count++; if (color_gray(object)) { + /* referenced after all, no longer a suspect */ + if (object->flags & OBJECT_SUSPECT) { + object->flags &= ~OBJECT_SUSPECT; + nr_suspects--; + } /* put_object() called when removing from gray_list */ WARN_ON(!get_object(object)); list_add_tail(&object->gray_list, &gray_list); @@ -1844,16 +1853,16 @@ static void dedup_flush(struct xarray *dedup) * kernel's standard allocators. This function must be called with the * scan_mutex held. */ -static void kmemleak_scan(void) +static int __kmemleak_scan(bool full) { struct kmemleak_object *object; struct zone *zone; int __maybe_unused i; - struct xarray dedup; - int new_leaks = 0; int stop = 0; jiffies_last_scan = jiffies; + if (full) + nr_suspects = 0; /* prepare the kmemleak_object's */ rcu_read_lock(); @@ -1883,6 +1892,8 @@ static void kmemleak_scan(void) /* reset the reference count (whiten the object) */ object->count = 0; + if (full) + object->flags &= ~OBJECT_SUSPECT; if (color_gray(object) && get_object(object)) list_add_tail(&object->gray_list, &gray_list); @@ -1950,6 +1961,10 @@ static void kmemleak_scan(void) scan_gray: scan_gray_list(); + /* a confirmation scan does not look for modified objects */ + if (!full) + return nr_suspects; + /* * Check for new or unreferenced objects modified since the previous * scan and color them gray until the next scan. @@ -1972,6 +1987,11 @@ scan_gray: /* color it gray temporarily */ object->count = object->min_count; list_add_tail(&object->gray_list, &gray_list); + } else if (unreferenced_object(object) && + !(object->flags & OBJECT_REPORTED)) { + /* flag the objects left unreferenced by this scan */ + object->flags |= OBJECT_SUSPECT; + nr_suspects++; } raw_spin_unlock_irq(&object->lock); } @@ -1982,12 +2002,42 @@ scan_gray: */ scan_gray_list(); + return nr_suspects; +} + +/* + * Scan the memory and report the unreferenced objects as leaks. Must be + * called with the scan_mutex held. + */ +static void kmemleak_scan(void) +{ + struct kmemleak_object *object; + struct xarray dedup; + int new_leaks = 0; + + /* + * Full scan. Objects left unreferenced are flagged OBJECT_SUSPECT and + * counted in the return value; nothing to confirm or report otherwise. + */ + if (!__kmemleak_scan(true)) + return; + /* * If scanning was stopped do not report any new unreferenced objects. */ if (scan_should_stop()) return; + /* + * A live object whose only reference is moved by, for example, a + * concurrent RCU update can be missed for one scan and reported as a + * transient false positive. Scan again and only report the objects + * left unreferenced (still flagged OBJECT_SUSPECT) by both scans. + */ + __kmemleak_scan(false); + if (scan_should_stop()) + return; + /* * Scanning result reporting. When verbose printing is enabled, dedupe * by stackdepot trace_handle so each unique backtrace is logged once @@ -2015,6 +2065,7 @@ scan_gray: trace_handle = 0; dedup_print = false; if (unreferenced_object(object) && + (object->flags & OBJECT_SUSPECT) && !(object->flags & OBJECT_REPORTED)) { object->flags |= OBJECT_REPORTED; if (kmemleak_verbose) { -- cgit v1.2.3 From e776db8e710165c2bc47566b62edcf36ff46bbdd Mon Sep 17 00:00:00 2001 From: Breno Leitao Date: Mon, 13 Jul 2026 04:48:05 -0700 Subject: mm: kmemleak: report leaks only after N consecutive unreferenced scans kmemleak reports an object the first scan it is found unreferenced. Its mark phase runs without stopping the rest of the kernel and without a write barrier, so a live object whose only reference is briefly invisible during a concurrent RCU update -- e.g. a VMA moved between maple tree nodes, or a page-cache xa_node -- can be seen as unreferenced for that one scan. Because an object is flagged as reported only once, such a transient race turns into a permanent false positive. Track how many consecutive scans each object has been seen unreferenced and only report it once that reaches min_unref_scans, a new module parameter. It defaults to 1, leaving the behaviour unchanged; setting it higher (e.g. 2) still reports a genuine leak, one scan later, while an object referenced again before the threshold restarts its run and is never reported. min_unref_scans can be set at boot with kmemleak.min_unref_scans= or at run-time via /sys/module/kmemleak/parameters/min_unref_scans. Link: https://lore.kernel.org/20260713-catalin_pto-v1-2-5b93b1131089@debian.org Signed-off-by: Breno Leitao Reviewed-by: Catalin Marinas Cc: David Hildenbrand Cc: Geert Uytterhoeven Cc: Jonathan Corbet Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- Documentation/dev-tools/kmemleak.rst | 8 ++++++++ mm/kmemleak.c | 13 ++++++++++++- 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/Documentation/dev-tools/kmemleak.rst b/Documentation/dev-tools/kmemleak.rst index 7d784e03f3f9..a8a83bc69ceb 100644 --- a/Documentation/dev-tools/kmemleak.rst +++ b/Documentation/dev-tools/kmemleak.rst @@ -198,6 +198,14 @@ systems, because of pointers temporarily stored in CPU registers or stacks. Kmemleak defines MSECS_MIN_AGE (defaulting to 1000) representing the minimum age of an object to be reported as a memory leak. +The ``min_unref_scans`` module parameter (default 1) requires an object to +be seen unreferenced in that many consecutive scans before it is reported. +Keeping it at 1 preserves the historical behaviour; higher values filter +the transient false positives described above, at the cost of delaying +genuine reports by up to that many scans. It can be set at boot with +``kmemleak.min_unref_scans=`` or at run-time via +``/sys/module/kmemleak/parameters/min_unref_scans``. + Limitations and Drawbacks ------------------------- diff --git a/mm/kmemleak.c b/mm/kmemleak.c index 95bd8ccd3c5b..7afd08ed8546 100644 --- a/mm/kmemleak.c +++ b/mm/kmemleak.c @@ -151,6 +151,8 @@ struct kmemleak_object { int min_count; /* the total number of pointers found pointing to this object */ int count; + /* consecutive scans the object has been seen unreferenced */ + unsigned int unref_scans; /* checksum for detecting modified objects */ u32 checksum; depot_stack_handle_t trace_handle; @@ -234,6 +236,9 @@ static unsigned long max_percpu_addr; static struct task_struct *scan_thread; /* used to avoid reporting of recently allocated objects */ static unsigned long jiffies_min_age; +/* consecutive scans an object must stay unreferenced before reporting */ +static unsigned int min_unref_scans = 1; +module_param(min_unref_scans, uint, 0644); static unsigned long jiffies_last_scan; /* delay between automatic memory scannings */ static unsigned long jiffies_scan_wait; @@ -692,6 +697,7 @@ static struct kmemleak_object *__alloc_object(gfp_t gfp) object->excess_ref = 0; object->count = 0; /* white color initially */ object->checksum = ~0; + object->unref_scans = 0; object->del_state = 0; /* task information */ @@ -1890,6 +1896,9 @@ static int __kmemleak_scan(bool full) __paint_it(object, KMEMLEAK_BLACK); } + /* referenced last scan: restart the unreferenced run */ + if (!color_white(object)) + object->unref_scans = 0; /* reset the reference count (whiten the object) */ object->count = 0; if (full) @@ -2064,9 +2073,11 @@ static void kmemleak_scan(void) raw_spin_lock_irq(&object->lock); trace_handle = 0; dedup_print = false; + if (unreferenced_object(object) && (object->flags & OBJECT_SUSPECT) && - !(object->flags & OBJECT_REPORTED)) { + !(object->flags & OBJECT_REPORTED) && + ++object->unref_scans >= min_unref_scans) { object->flags |= OBJECT_REPORTED; if (kmemleak_verbose) { trace_handle = object->trace_handle; -- cgit v1.2.3 From 70a964bafe5541d3d319399d814526b26690cddc Mon Sep 17 00:00:00 2001 From: Breno Leitao Date: Mon, 13 Jul 2026 04:48:06 -0700 Subject: mm: kmemleak: factor leak confirmation into a helper The reporting loop in kmemleak_scan() decided whether to tag an object as a reported leak with a four-term compound condition whose last operand also had a side effect (++object->unref_scans). Mixing the candidate tests with the counter update made the check hard to read. Move the state transition into confirm_leak(): it returns true when a still-unreferenced suspect crosses min_unref_scans consecutive scans and is newly flagged OBJECT_REPORTED, leaving only the reporting bookkeeping in the caller. No functional change. Link: https://lore.kernel.org/20260713-catalin_pto-v1-3-5b93b1131089@debian.org Signed-off-by: Breno Leitao Reviewed-by: Catalin Marinas Cc: David Hildenbrand Cc: Geert Uytterhoeven Cc: Jonathan Corbet Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- mm/kmemleak.c | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/mm/kmemleak.c b/mm/kmemleak.c index 7afd08ed8546..f63dfacee7ca 100644 --- a/mm/kmemleak.c +++ b/mm/kmemleak.c @@ -2014,6 +2014,26 @@ scan_gray: return nr_suspects; } +/* + * Promote a suspected object to a reported leak once it has stayed + * unreferenced for min_unref_scans consecutive scans. Called with + * object->lock held; returns true when the object is newly reported. + */ +static bool confirm_leak(struct kmemleak_object *object) +{ + if (!unreferenced_object(object) || + !(object->flags & OBJECT_SUSPECT) || + (object->flags & OBJECT_REPORTED)) + return false; + + object->unref_scans += 1; + if (object->unref_scans < min_unref_scans) + return false; + + object->flags |= OBJECT_REPORTED; + return true; +} + /* * Scan the memory and report the unreferenced objects as leaks. Must be * called with the scan_mutex held. @@ -2074,11 +2094,7 @@ static void kmemleak_scan(void) trace_handle = 0; dedup_print = false; - if (unreferenced_object(object) && - (object->flags & OBJECT_SUSPECT) && - !(object->flags & OBJECT_REPORTED) && - ++object->unref_scans >= min_unref_scans) { - object->flags |= OBJECT_REPORTED; + if (confirm_leak(object)) { if (kmemleak_verbose) { trace_handle = object->trace_handle; dedup_print = true; -- cgit v1.2.3 From 8f07855f579ae8b06a90703cd8a4d02849728890 Mon Sep 17 00:00:00 2001 From: Breno Leitao Date: Mon, 13 Jul 2026 04:48:07 -0700 Subject: selftests: mm: test kmemleak's N-consecutive-scan leak confirmation Add a functional test for the min_unref_scans kmemleak module parameter. Using samples/kmemleak's helper module it checks that min_unref_scans=1 reports an orphan on the first scan, min_unref_scans=2 reports nothing on the first scan but does on the second, and that the parameter reads back what was written. It counts only the helper module's own orphans (matched by their [kmemleak_test] backtrace, with the module kept loaded so the symbols resolve) so unrelated leaks already present on the system do not perturb the result. The test skips when run as non-root, without CONFIG_DEBUG_KMEMLEAK / CONFIG_SAMPLE_KMEMLEAK, on a kernel without the parameter, or when the helper yields no detectable orphan. Link: https://lore.kernel.org/20260713-catalin_pto-v1-4-5b93b1131089@debian.org Signed-off-by: Breno Leitao Reviewed-by: Catalin Marinas Cc: David Hildenbrand Cc: Geert Uytterhoeven Cc: Jonathan Corbet Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- tools/testing/selftests/mm/Makefile | 1 + .../testing/selftests/mm/ksft_kmemleak_confirm.sh | 132 +++++++++++++++++++++ 2 files changed, 133 insertions(+) create mode 100755 tools/testing/selftests/mm/ksft_kmemleak_confirm.sh diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile index 0f31d850707d..2d5366196e30 100644 --- a/tools/testing/selftests/mm/Makefile +++ b/tools/testing/selftests/mm/Makefile @@ -149,6 +149,7 @@ TEST_PROGS += ksft_gup_test.sh TEST_PROGS += ksft_hmm.sh TEST_PROGS += ksft_hugetlb.sh TEST_PROGS += ksft_hugevm.sh +TEST_PROGS += ksft_kmemleak_confirm.sh TEST_PROGS += ksft_kmemleak_dedup.sh TEST_PROGS += ksft_ksm.sh TEST_PROGS += ksft_ksm_numa.sh diff --git a/tools/testing/selftests/mm/ksft_kmemleak_confirm.sh b/tools/testing/selftests/mm/ksft_kmemleak_confirm.sh new file mode 100755 index 000000000000..3a8576e835c8 --- /dev/null +++ b/tools/testing/selftests/mm/ksft_kmemleak_confirm.sh @@ -0,0 +1,132 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Functional test for kmemleak's N-consecutive-scan leak confirmation +# (the min_unref_scans module parameter). +# +# kmemleak only reports an object once it has stayed unreferenced for +# min_unref_scans consecutive scans. The default of 1 reports on the first +# scan (historical behaviour); higher values filter transient false +# positives where a live object's only reference is briefly invisible to a +# single scan (e.g. an RCU tree update in flight while the scan runs). The +# test loads samples/kmemleak's helper module to create orphan allocations +# and, counting only those orphans (matched by their [kmemleak_test] +# backtrace so unrelated leaks already present on the system are ignored), +# checks that: +# - a freshly allocated object is greyed on its first scan (its checksum +# settles then), so nothing can be reported before that priming scan; +# each case below primes once first, +# - with the default threshold (min_unref_scans=1) one scan after priming +# reports the orphans, +# - raising the threshold to 2 needs two scans after priming: one is not +# enough, the second reports, +# - the parameter reads back what was written. +# +# The "one post-prime scan is not enough at min_unref_scans=2" check is the +# core regression test: raising min_unref_scans must push the report +# strictly later. Like ksft_kmemleak_dedup.sh, if the module yields no +# detectable orphan at all in the running environment the test skips rather +# than failing. +# +# Author: Breno Leitao + +# KTAP output helpers (ktap_skip_all, ktap_exit_fail_msg, ktap_test_pass, ...). +DIR="$(dirname "$(readlink -f "$0")")" +# shellcheck source=../kselftest/ktap_helpers.sh +source "${DIR}"/../kselftest/ktap_helpers.sh + +KMEMLEAK=/sys/kernel/debug/kmemleak +PARAM=/sys/module/kmemleak/parameters/min_unref_scans +MODULE=kmemleak-test +AGE=6 # seconds; must exceed kmemleak's 5s minimum object age + +ktap_print_header + +[ "$(id -u)" -eq 0 ] || { ktap_skip_all "must run as root"; exit "$KSFT_SKIP"; } +[ -r "$KMEMLEAK" ] || + { ktap_skip_all "no kmemleak debugfs (CONFIG_DEBUG_KMEMLEAK)"; exit "$KSFT_SKIP"; } +[ -w "$PARAM" ] || + { ktap_skip_all "min_unref_scans module parameter not present"; exit "$KSFT_SKIP"; } +modinfo "$MODULE" >/dev/null 2>&1 || + { ktap_skip_all "$MODULE not built (CONFIG_SAMPLE_KMEMLEAK)"; exit "$KSFT_SKIP"; } + +# kmemleak can be present but disabled at runtime (kmemleak=off boot arg, +# or it self-disabled after an internal error); a "scan" then returns +# EPERM. Probe once and skip if so. +echo scan > "$KMEMLEAK" 2>/dev/null || + { ktap_skip_all "kmemleak is disabled (check dmesg or kmemleak= boot arg)"; exit "$KSFT_SKIP"; } + +prev=$(cat "$PARAM") +# shellcheck disable=SC2317 # invoked indirectly via trap +cleanup() { + echo "$prev" > "$PARAM" 2>/dev/null # restore the parameter + echo scan=on > "$KMEMLEAK" 2>/dev/null # re-enable auto scan + rmmod "$MODULE" 2>/dev/null + echo clear > "$KMEMLEAK" 2>/dev/null +} +trap cleanup EXIT + +# Stop the automatic scan thread: only our manual scans should advance an +# object's consecutive-unreferenced run. An auto scan landing between two +# manual scans would change the result and make the test flaky. +echo scan=off > "$KMEMLEAK" 2>/dev/null + +# Create a fresh, aged set of orphan objects from the helper module's init +# path (its kmalloc/vmalloc/percpu allocations are dropped right away). +# Pre-existing reported leaks are greyed first ("clear") so only our +# orphans are counted. The module is left loaded on purpose: once it is +# unloaded its symbols are gone, so the orphan backtraces no longer resolve +# to [kmemleak_test] and could not be matched below. +gen_orphans() { + rmmod "$MODULE" 2>/dev/null + echo clear > "$KMEMLEAK" + modprobe "$MODULE" || + { ktap_skip_all "failed to load $MODULE"; exit "$KSFT_SKIP"; } + sleep "$AGE" +} + +scan() { echo scan > "$KMEMLEAK"; } + +# Number of helper-module orphans currently reported by kmemleak. Matching +# the module's own backtrace ([kmemleak_test]) keeps the count immune to +# unrelated leaks on the running system. kmemleak only lists an object here +# once it has been reported, so this reflects the confirmation gating. +count_orphans() { + c=$(grep -c '\[kmemleak_test\]' "$KMEMLEAK" 2>/dev/null) + echo "${c:-0}" +} + +# 0) the parameter reads back what was written. +echo 3 > "$PARAM" +[ "$(cat "$PARAM")" = "3" ] || ktap_exit_fail_msg "min_unref_scans did not read back as 3" + +# Priming scan: kmemleak greys a freshly allocated object on its first scan +# (its checksum settles then), so nothing can be reported until a second +# scan. Every case below runs this priming scan before counting. +prime() { scan; } + +# 1) min_unref_scans=1 (default): one scan after priming reports the +# orphans. This also establishes that the helper produces detectable +# orphans here. +echo 1 > "$PARAM" +gen_orphans +prime +scan +first=$(count_orphans) +[ "$first" -gt 0 ] || + { ktap_skip_all "$MODULE produced no detectable orphans (cannot test min_unref_scans)"; exit "$KSFT_SKIP"; } + +# 2) min_unref_scans=2: after priming, one scan is not enough (still +# gated), the second reports. The gated-scan-zero check is the core +# regression. +echo 2 > "$PARAM" +gen_orphans +prime +scan; s1=$(count_orphans) +scan; s2=$(count_orphans) +[ "$s1" -eq 0 ] || ktap_exit_fail_msg "min_unref_scans=2: $s1 orphan(s) after 1 post-prime scan (must be 0)" +[ "$s2" -gt 0 ] || ktap_exit_fail_msg "min_unref_scans=2: no report after 2 post-prime scans (false negative)" + +ktap_set_plan 1 +ktap_test_pass "min_unref_scans=1 reported $first orphan(s) one scan after priming; =2 held them one scan longer ($s1 after one scan, $s2 after two); param read-back ok" +ktap_finished -- cgit v1.2.3 From dd0dcfe8ef21fb73dbc48af5c630ef2a6f4ab27c Mon Sep 17 00:00:00 2001 From: Breno Leitao Date: Fri, 31 Jul 2026 03:13:04 -0700 Subject: mm: kmemleak: default min_unref_scans to 2 for verbose auto-scan Patch series "mm: kmemleak: default min_unref_scans to 2 for verbose kernels", v2. When CONFIG_DEBUG_KMEMLEAK_VERBOSE is set, which means the host is in auto scan mode, set min_unref_scans to 2, avoiding false positives. CONFIG_DEBUG_KMEMLEAK_VERBOSE depends on CONFIG_DEBUG_KMEMLEAK_AUTO_SCAN, so a kernel built with it already runs the scan thread periodically and the user has asked for detailed leak reports. The confirming second scan comes for free there, so default min_unref_scans to 2 in that case and keep it at 1 everywhere else. CONFIG_DEBUG_KMEMLEAK_VERBOSE defaults to n, so nothing changes for kernels that do not opt in. The other two patches bring the documentation and the selftest comments in line with the new conditional default. PS: A similar patch (v1 of this patchset) is applied to Meta's kernel, in real production hosts. This patch (of 3): min_unref_scans defers reporting an object as leaked until it has stayed unreferenced for that many consecutive scans, filtering out objects that are only transiently unreferenced during a scan. It defaults to 1, which reports on the first unreferenced scan. CONFIG_DEBUG_KMEMLEAK_VERBOSE depends on CONFIG_DEBUG_KMEMLEAK_AUTO_SCAN, so a kernel built with it runs the scan thread periodically and the user has opted into detailed leak reporting. A second confirming scan then happens on its own. Default min_unref_scans to 2 there to suppress transient false positives, and keep it at 1 otherwise, where a manually triggered scan is expected to report immediately. The value stays writable through the module parameter. CONFIG_DEBUG_KMEMLEAK_VERBOSE defaults to n, so this does not change the default for kernels that do not opt in. Link: https://lore.kernel.org/20260731-kmemleak_hardened-v2-0-7b9689ac77cb@debian.org Link: https://lore.kernel.org/20260731-kmemleak_hardened-v2-1-7b9689ac77cb@debian.org Signed-off-by: Breno Leitao Acked-by: Catalin Marinas Cc: David Hildenbrand Cc: Jonathan Corbet Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- mm/kmemleak.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/kmemleak.c b/mm/kmemleak.c index f63dfacee7ca..8fa409a4f9fb 100644 --- a/mm/kmemleak.c +++ b/mm/kmemleak.c @@ -237,7 +237,8 @@ static struct task_struct *scan_thread; /* used to avoid reporting of recently allocated objects */ static unsigned long jiffies_min_age; /* consecutive scans an object must stay unreferenced before reporting */ -static unsigned int min_unref_scans = 1; +static unsigned int min_unref_scans = + IS_ENABLED(CONFIG_DEBUG_KMEMLEAK_VERBOSE) ? 2 : 1; module_param(min_unref_scans, uint, 0644); static unsigned long jiffies_last_scan; /* delay between automatic memory scannings */ -- cgit v1.2.3 From 09dde5e9bac0429fa565d3a9fcb0e80d0f668f19 Mon Sep 17 00:00:00 2001 From: Breno Leitao Date: Fri, 31 Jul 2026 03:13:05 -0700 Subject: Documentation: kmemleak: document the conditional min_unref_scans default min_unref_scans now defaults to 2 when CONFIG_DEBUG_KMEMLEAK_AUTO_SCAN and CONFIG_DEBUG_KMEMLEAK_VERBOSE are both enabled, but the documentation still states that the default is unconditionally 1. Link: https://lore.kernel.org/20260731-kmemleak_hardened-v2-2-7b9689ac77cb@debian.org Signed-off-by: Breno Leitao Acked-by: Catalin Marinas Cc: David Hildenbrand Cc: Jonathan Corbet Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- Documentation/dev-tools/kmemleak.rst | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/Documentation/dev-tools/kmemleak.rst b/Documentation/dev-tools/kmemleak.rst index a8a83bc69ceb..d1b690b17169 100644 --- a/Documentation/dev-tools/kmemleak.rst +++ b/Documentation/dev-tools/kmemleak.rst @@ -198,11 +198,13 @@ systems, because of pointers temporarily stored in CPU registers or stacks. Kmemleak defines MSECS_MIN_AGE (defaulting to 1000) representing the minimum age of an object to be reported as a memory leak. -The ``min_unref_scans`` module parameter (default 1) requires an object to -be seen unreferenced in that many consecutive scans before it is reported. -Keeping it at 1 preserves the historical behaviour; higher values filter -the transient false positives described above, at the cost of delaying -genuine reports by up to that many scans. It can be set at boot with +The ``min_unref_scans`` module parameter requires an object to be seen +unreferenced in that many consecutive scans before it is reported. It +defaults to 2 when CONFIG_DEBUG_KMEMLEAK_VERBOSE is enabled, where the +periodic scan thread confirms a leak on its own, and to 1 otherwise. A +value of 1 preserves the historical behaviour; higher values filter the +transient false positives described above, at the cost of delaying genuine +reports by up to that many scans. It can be set at boot with ``kmemleak.min_unref_scans=`` or at run-time via ``/sys/module/kmemleak/parameters/min_unref_scans``. -- cgit v1.2.3 From 972195eb9b4c9a70f04d5677cff799c2524786d8 Mon Sep 17 00:00:00 2001 From: Breno Leitao Date: Fri, 31 Jul 2026 03:13:06 -0700 Subject: selftests/mm: kmemleak: drop stale min_unref_scans default from comments The test writes min_unref_scans explicitly for every case, so its comments describing 1 as the default are both unnecessary and, since the default is now conditional, wrong. Refer to the threshold values directly. No functional change. Link: https://lore.kernel.org/20260731-kmemleak_hardened-v2-3-7b9689ac77cb@debian.org Signed-off-by: Breno Leitao Acked-by: Catalin Marinas Cc: David Hildenbrand Cc: Jonathan Corbet Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- tools/testing/selftests/mm/ksft_kmemleak_confirm.sh | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/tools/testing/selftests/mm/ksft_kmemleak_confirm.sh b/tools/testing/selftests/mm/ksft_kmemleak_confirm.sh index 3a8576e835c8..72ded5e6794c 100755 --- a/tools/testing/selftests/mm/ksft_kmemleak_confirm.sh +++ b/tools/testing/selftests/mm/ksft_kmemleak_confirm.sh @@ -5,7 +5,7 @@ # (the min_unref_scans module parameter). # # kmemleak only reports an object once it has stayed unreferenced for -# min_unref_scans consecutive scans. The default of 1 reports on the first +# min_unref_scans consecutive scans. A threshold of 1 reports on the first # scan (historical behaviour); higher values filter transient false # positives where a live object's only reference is briefly invisible to a # single scan (e.g. an RCU tree update in flight while the scan runs). The @@ -16,8 +16,7 @@ # - a freshly allocated object is greyed on its first scan (its checksum # settles then), so nothing can be reported before that priming scan; # each case below primes once first, -# - with the default threshold (min_unref_scans=1) one scan after priming -# reports the orphans, +# - at min_unref_scans=1 one scan after priming reports the orphans, # - raising the threshold to 2 needs two scans after priming: one is not # enough, the second reports, # - the parameter reads back what was written. @@ -105,9 +104,8 @@ echo 3 > "$PARAM" # scan. Every case below runs this priming scan before counting. prime() { scan; } -# 1) min_unref_scans=1 (default): one scan after priming reports the -# orphans. This also establishes that the helper produces detectable -# orphans here. +# 1) min_unref_scans=1: one scan after priming reports the orphans. This +# also establishes that the helper produces detectable orphans here. echo 1 > "$PARAM" gen_orphans prime -- cgit v1.2.3 From cdd719b3f2b88437aa5a6ca8ee2bea6f7015c35b Mon Sep 17 00:00:00 2001 From: Mark Sercombe Date: Thu, 13 Aug 2026 20:38:47 +0200 Subject: maple_tree: fix comment typo Fix a spelling mistake n a code comment. This is a comment only change with no functional impact. Link: https://lore.kernel.org/20260813183847.474357-1-sercombe.joel.mark@gmail.com Signed-off-by: Mark Sercombe Reviewed-by: Andrew Morton Reviewed-by: Liam R. Howlett (Oracle) Cc: Alice Ryhl Cc: Andrew Ballance Signed-off-by: Andrew Morton --- lib/maple_tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index b48bc4064ad2..a0542b491bc2 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -3590,7 +3590,7 @@ set_content: /** * mas_prealloc_calc() - Calculate number of nodes needed for a - * given store oepration + * given store operation * @wr_mas: The maple write state * @entry: The entry to store into the tree * -- cgit v1.2.3 From a8b5875741d416703e19ad8eeac6fce8a12bd6e4 Mon Sep 17 00:00:00 2001 From: David Carlier Date: Mon, 10 Aug 2026 21:22:37 +0100 Subject: zram: fix slot lock bit position on big-endian 64-bit The slot lock is a bit operation on the whole __lock word, which flags and ac_time alias as two u32s. On little-endian the lock bit lands in the position ZRAM_ENTRY_LOCK reserves in flags, so the aliasing works out. On 64-bit big-endian it lands in ac_time instead: with ZRAM_TRACK_ENTRY_ACTIME enabled, storing the access time from mark_slot_accessed() or slot_free() wipes out the held lock bit, letting another CPU take the same slot lock; an access time value with that bit set makes the slot look locked forever. Shift the lock bit into the flags half of the word on big-endian 64-bit. Link: https://lore.kernel.org/20260810202241.2436603-1-devnexen@gmail.com Fixes: 2e8ff2f51dde ("zram: use u32 for entry ac_time tracking") Signed-off-by: David Carlier Reviewed-by: Sergey Senozhatsky Cc: Minchan Kim Cc: Signed-off-by: Andrew Morton --- drivers/block/zram/zram_drv.c | 6 +++--- drivers/block/zram/zram_drv.h | 14 ++++++++++++++ 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index d09fdca49cbd..a9b3bb1d3bef 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -74,7 +74,7 @@ static __must_check bool slot_trylock(struct zram *zram, unsigned long index) { unsigned long *lock = &zram->table[index].__lock; - if (!test_and_set_bit_lock(ZRAM_ENTRY_LOCK, lock)) { + if (!test_and_set_bit_lock(ZRAM_ENTRY_LOCK_BIT, lock)) { mutex_acquire(&zram->table_lock_map, 0, 1, _RET_IP_); lock_acquired(&zram->table_lock_map, _RET_IP_); return true; @@ -88,7 +88,7 @@ static void slot_lock(struct zram *zram, unsigned long index) unsigned long *lock = &zram->table[index].__lock; mutex_acquire(&zram->table_lock_map, 0, 0, _RET_IP_); - wait_on_bit_lock(lock, ZRAM_ENTRY_LOCK, TASK_UNINTERRUPTIBLE); + wait_on_bit_lock(lock, ZRAM_ENTRY_LOCK_BIT, TASK_UNINTERRUPTIBLE); lock_acquired(&zram->table_lock_map, _RET_IP_); } @@ -97,7 +97,7 @@ static void slot_unlock(struct zram *zram, unsigned long index) unsigned long *lock = &zram->table[index].__lock; mutex_release(&zram->table_lock_map, _RET_IP_); - clear_and_wake_up_bit(ZRAM_ENTRY_LOCK, lock); + clear_and_wake_up_bit(ZRAM_ENTRY_LOCK_BIT, lock); } static inline bool init_done(struct zram *zram) diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 4fddc582f3b8..7a55d751417e 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -15,6 +15,7 @@ #ifndef _ZRAM_DRV_H_ #define _ZRAM_DRV_H_ +#include #include #include @@ -57,6 +58,19 @@ enum zram_pageflags { __NR_ZRAM_PAGEFLAGS, }; +/* + * The slot lock is a bit-wait lock on the whole __lock word, while + * flags and ac_time alias that word as two u32s. The lock bit must + * land in the slot that ZRAM_ENTRY_LOCK reserves in attr.flags; on + * 64-bit big-endian the flags word maps to the upper half of __lock, + * so the bit position has to be shifted up. + */ +#if defined(CONFIG_64BIT) && defined(__BIG_ENDIAN) +#define ZRAM_ENTRY_LOCK_BIT (ZRAM_ENTRY_LOCK + 32) +#else +#define ZRAM_ENTRY_LOCK_BIT ZRAM_ENTRY_LOCK +#endif + /* * Allocated for each disk page. We use bit-lock (ZRAM_ENTRY_LOCK bit * of flags) to save memory. There can be plenty of entries and standard -- cgit v1.2.3 From 1dea8e081ec3833e5150250b9fa3afd0dc8768db Mon Sep 17 00:00:00 2001 From: Jann Horn Date: Mon, 10 Aug 2026 20:10:01 +0200 Subject: mm/page-writeback: document folio_mark_dirty() locking more explicitly We have had bugs where set_page_dirty() was used on a page from GUP without appropriate locking, leading to UAF, in: - KVM, see https://lore.kernel.org/r/20260810-x86-kvm-setpagedirty-v1-1-85f180892d4f@google.com - i915, see commit 0d4bbe3d407f ("drm/i915/userptr: Try to acquire the page lock around set_page_dirty()"). - VMCI, see commit 5a16c535409f ("VMCI: Use set_page_dirty_lock() when unregistering guest memory") - kpc2000 staging driver, see commit b6d13bd9f2c1 ("staging: kpc2000: kpc_dma: Convert set_page_dirty() --> set_page_dirty_lock()") I think set_page_dirty() and folio_mark_dirty() need more explicit documentation on how they should be used with pages from GUP; so add a comment on top of set_page_dirty() and make the comment above folio_mark_dirty() more explicit. Link: https://lore.kernel.org/20260810-set-page-dirty-warnings-v2-1-1bd40fadfacd@google.com Signed-off-by: Jann Horn Reviewed-by: Jan Kara Reviewed-by: Christoph Hellwig Cc: Matthew Wilcox (Oracle) Signed-off-by: Andrew Morton --- mm/folio-compat.c | 1 + mm/page-writeback.c | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/mm/folio-compat.c b/mm/folio-compat.c index a02179a0bded..6212fdd6761a 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -41,6 +41,7 @@ void set_page_writeback(struct page *page) } EXPORT_SYMBOL(set_page_writeback); +/* Read the comment above folio_mark_dirty() regarding required locks! */ bool set_page_dirty(struct page *page) { return folio_mark_dirty(page_folio(page)); diff --git a/mm/page-writeback.c b/mm/page-writeback.c index 47495be68598..3a03e04b640c 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -2763,6 +2763,11 @@ EXPORT_SYMBOL(folio_redirty_for_writepage); * in this folio. Truncation will block on the page table lock as it * unmaps pages before removing the folio from its mapping. * + * .. DANGER:: + * Do not use this on a folio obtained from a function like + * get_user_pages_fast() without holding appropriate locks; you might want to + * use set_page_dirty_lock() or folio_mark_dirty_lock() instead. + * * Return: True if the folio was newly dirtied, false if it was already dirty. */ bool folio_mark_dirty(struct folio *folio) -- cgit v1.2.3 From f7bf5cd5b5f2b13fe2361860880c4e214c08b440 Mon Sep 17 00:00:00 2001 From: Longlong Xia Date: Sun, 9 Aug 2026 19:55:18 +0800 Subject: zsmalloc: account for handle size in class lookup zs_lookup_class_index() lets zram recompression decide whether a newly compressed object would use a smaller size class. It currently classifies the payload size directly, while zs_malloc() adds ZS_HANDLE_SIZE before selecting the class. This makes lookup disagree with allocation near size-class boundaries. With 4 KiB pages, CONFIG_ZSMALLOC_CHAIN_SIZE=8, and 64-bit handles, a 1025-to-1024-byte recompression appears to move from class 64 to class 62 although both allocations use class 64. Conversely, a 1049-to-1025-byte recompression appears to stay in class 64 although the allocations move from class 65 to class 64. As a result, zram can accept replacements with no allocation benefit or reject ones that would save memory, potentially marking the object incompressible. Factor size-class selection into lookup_size_class(), account for the handle there, and use the helper for both lookup and allocation. Link: https://lore.kernel.org/20260809115518.3791787-1-xialonglong2025@163.com Fixes: 7c2af309abd2 ("zram: add size class equals check into recompression") Assisted-by: Codex:gpt-5.6-sol Signed-off-by: Longlong Xia Reviewed-by: Sergey Senozhatsky Cc: Minchan Kim Cc: Signed-off-by: Andrew Morton --- mm/zsmalloc.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 8204b76f7830..825022a7a328 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -552,6 +552,11 @@ static int get_size_class_index(int size) return min_t(int, ZS_SIZE_CLASSES - 1, idx); } +static struct size_class *lookup_size_class(struct zs_pool *pool, size_t size) +{ + return pool->size_class[get_size_class_index(size + ZS_HANDLE_SIZE)]; +} + static inline void class_stat_add(struct size_class *class, int type, unsigned long cnt) { @@ -1117,7 +1122,7 @@ unsigned int zs_lookup_class_index(struct zs_pool *pool, unsigned int size) { struct size_class *class; - class = pool->size_class[get_size_class_index(size)]; + class = lookup_size_class(pool, size); return class->index; } @@ -1407,9 +1412,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp, if (!handle) return (unsigned long)ERR_PTR(-ENOMEM); - /* extra space in chunk to keep the handle */ - size += ZS_HANDLE_SIZE; - class = pool->size_class[get_size_class_index(size)]; + class = lookup_size_class(pool, size); /* class->lock effectively protects the zpage migration */ spin_lock(&class->lock); -- cgit v1.2.3 From 3c5194eb2e644b8360a368a1637bb2851be0a9c3 Mon Sep 17 00:00:00 2001 From: Christoph Hellwig Date: Thu, 23 Jul 2026 07:46:04 +0200 Subject: mm/swap: revert to single-folio writes for synchronous swap devices Patch series "swap_ops updates", v2. This series is a follow on to the swap ops series now in mm-unstable. The first patch reintroduces direct folio writes for synchronous swap files, the other two remove the double indirect for file system based swap. This patch (of 3): Kairui Song reported that zram benefits from submitting each folio directly instead of batching up I/O because the classic LRU scanning benefits from clearing the folio writeback bit in the scan loop. Accommodate that by kicking off writes for synchronous devices for each iteration. Link: https://lore.kernel.org/20260723054622.3460249-1-hch@lst.de Link: https://lore.kernel.org/20260723054622.3460249-2-hch@lst.de Signed-off-by: Christoph Hellwig Acked-by: Usama Arif Acked-by: Chris Li Cc: Baoquan He Cc: Kairui Song Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Steve French Signed-off-by: Andrew Morton --- mm/page_io.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/page_io.c b/mm/page_io.c index e4fa7ffffe8b..c984a4023a65 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -358,7 +358,16 @@ static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw) } bvec_set_folio(&sio->bvecs[sio->nr_bvecs], folio, folio_size(folio), 0); sio->len += folio_size(folio); - if (++sio->nr_bvecs == ARRAY_SIZE(sio->bvecs)) { + + /* + * Write out the iocb if we filled it, or if the device is synchronous. + * + * The latter is to work around expectations in the classic LRU code + * which make synchronous clearing of the folio writeback flag in the + * reclaim path beneficial. + */ + if (++sio->nr_bvecs == ARRAY_SIZE(sio->bvecs) || + (rw == WRITE && (sis->flags & SWP_SYNCHRONOUS_IO))) { if (rw == WRITE) swap_write_submit(ctx); else -- cgit v1.2.3 From 52d85ca90c7db41f8cfc72e5fe7dc62c2f983032 Mon Sep 17 00:00:00 2001 From: Christoph Hellwig Date: Thu, 23 Jul 2026 07:46:05 +0200 Subject: mm/swap: add a new swap_ops.h header to allow for pluggable swap ops Add a new header to declare the swap_iocb, swap_ops and swap_ctx to allow for swap_ops implementations outside of mm/page_io.c. This will be used to remove the double indirection for file system-based swap. There is no functional change, just a move of the declarations. Note that there already is a swapops.h header, which is totally unrelated to struct swap_ops. The close naming is a bit unfortunate, but I could not think of a better name for this header. Link: https://lore.kernel.org/20260723054622.3460249-3-hch@lst.de Signed-off-by: Christoph Hellwig Acked-by: Chris Li Cc: Baoquan He Cc: Kairui Song Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Steve French Cc: Usama Arif Signed-off-by: Andrew Morton --- MAINTAINERS | 1 + include/linux/swap_ops.h | 39 +++++++++++++++++++++++++++++++++++++++ mm/madvise.c | 1 + mm/page_io.c | 10 +--------- mm/shmem.c | 1 + mm/swap.h | 23 +---------------------- mm/swap_state.c | 1 + mm/vmscan.c | 1 + mm/zswap.c | 2 +- 9 files changed, 47 insertions(+), 32 deletions(-) create mode 100644 include/linux/swap_ops.h diff --git a/MAINTAINERS b/MAINTAINERS index 4899b81bd839..604285d848e6 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17242,6 +17242,7 @@ S: Maintained F: Documentation/ABI/testing/sysfs-kernel-mm-swap F: Documentation/mm/swap-table.rst F: include/linux/swap.h +F: include/linux/swap_ops.h F: include/linux/swapfile.h F: include/linux/swapops.h F: mm/page_io.c diff --git a/include/linux/swap_ops.h b/include/linux/swap_ops.h new file mode 100644 index 000000000000..e92b4f532604 --- /dev/null +++ b/include/linux/swap_ops.h @@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _MM_SWAP_OPS_H +#define _MM_SWAP_OPS_H + +#include /* for SWAP_CLUSTER_MAX */ + +struct swap_iocb { + union { + struct kiocb iocb; + struct bio bio; + }; + struct bio_vec bvecs[SWAP_CLUSTER_MAX]; + int nr_bvecs; + int len; +}; + +struct swap_io_ctx { + struct swap_iocb *sio; + struct swap_info_struct *sis; +}; + +/* + * SWAP_OPS_F_REQUIRE_NOFS: + * When set, all reclaim operations must operated as GFS_NOFS and not + * just GFP_NOIO, as GFP_NOIO allocations could recourse into the + * file system backing this swap file. + */ +#define SWAP_OPS_F_REQUIRE_NOFS (1U << 0) + +struct swap_ops { + unsigned int flags; + + bool (*can_merge)(struct folio *folio, struct folio *prev_folio, + size_t prev_folio_size, int rw); + void (*submit_write)(struct swap_io_ctx *ctx); + void (*submit_read)(struct swap_io_ctx *ctx); +}; + +#endif /* _MM_SWAP_OPS_H */ diff --git a/mm/madvise.c b/mm/madvise.c index 07a21ca31bad..c179938097bf 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -32,6 +32,7 @@ #include #include #include +#include #include diff --git a/mm/page_io.c b/mm/page_io.c index c984a4023a65..e741e67d6592 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -25,6 +25,7 @@ #include #include #include +#include #include "swap.h" #include "swap_table.h" @@ -300,15 +301,6 @@ static bool folio_blkg_can_merge(struct folio *folio, struct folio *prev_folio) #define bio_associate_blkg_from_page(bio, folio) do { } while (0) #endif /* CONFIG_MEMCG && CONFIG_BLK_CGROUP */ -struct swap_iocb { - union { - struct kiocb iocb; - struct bio bio; - }; - struct bio_vec bvecs[SWAP_CLUSTER_MAX]; - int nr_bvecs; - int len; -}; static mempool_t *sio_pool; int sio_pool_init(void) diff --git a/mm/shmem.c b/mm/shmem.c index 2e4dacdcce11..599665a3d6e7 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -41,6 +41,7 @@ #include #include #include +#include #include "swap.h" static struct vfsmount *shm_mnt __ro_after_init; diff --git a/mm/swap.h b/mm/swap.h index 48379b2ab202..90a551a88df6 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -10,6 +10,7 @@ struct mempolicy; struct swap_iocb; struct swap_memcg_table; +struct swap_io_ctx; #if defined(MAX_POSSIBLE_PHYSMEM_BITS) #define SWAP_CACHE_PFN_BITS (MAX_POSSIBLE_PHYSMEM_BITS - PAGE_SHIFT) @@ -91,28 +92,6 @@ static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg) return READ_ONCE(vm_swappiness); } -struct swap_io_ctx { - struct swap_iocb *sio; - struct swap_info_struct *sis; -}; - -/* - * SWAP_OPS_F_REQUIRE_NOFS: - * When set, all reclaim operations must operated as GFS_NOFS and not - * just GFP_NOIO, as GFP_NOIO allocations could recourse into the - * file system backing this swap file. - */ -#define SWAP_OPS_F_REQUIRE_NOFS (1U << 0) - -struct swap_ops { - unsigned int flags; - - bool (*can_merge)(struct folio *folio, struct folio *prev_folio, - size_t prev_folio_size, int rw); - void (*submit_write)(struct swap_io_ctx *ctx); - void (*submit_read)(struct swap_io_ctx *ctx); -}; - #ifdef CONFIG_SWAP #include /* for swp_offset */ #include /* for bio_end_io_t */ diff --git a/mm/swap_state.c b/mm/swap_state.c index 5be825911e64..b76eb3d876fd 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -23,6 +23,7 @@ #include #include #include +#include #include "internal.h" #include "swap_table.h" #include "swap.h" diff --git a/mm/vmscan.c b/mm/vmscan.c index 3194da7dcc79..be6bd26e8c57 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -58,6 +58,7 @@ #include #include #include +#include #include #include diff --git a/mm/zswap.c b/mm/zswap.c index 9f777a48b106..37f34e406c8e 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -20,7 +20,7 @@ #include #include #include -#include +#include #include #include #include -- cgit v1.2.3 From 22779ae8175aad7c04827db44e934e53bf2bd2d4 Mon Sep 17 00:00:00 2001 From: Christoph Hellwig Date: Thu, 23 Jul 2026 07:46:06 +0200 Subject: mm/swap: move swap_ops into file systems for file system-based swap Currently swap to and from file systems goes through two indirect calls between the swap ops and the swap_rw method. Reduce this by directly providing the swap_ops from the file system. For this refactor swap_fs_submit into a swap_fs_prepare_rw helper that initializes the iov_iter on the callers stack so that file systems can call it directly, and use that to initialize file system specific ops in the NFS and SMB clients, which then get passed to swap_fs_activate. Link: https://lore.kernel.org/20260723054622.3460249-4-hch@lst.de Signed-off-by: Christoph Hellwig Acked-by: Chris Li Cc: Baoquan He Cc: Kairui Song Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Steve French Cc: Usama Arif Signed-off-by: Andrew Morton --- Documentation/filesystems/locking.rst | 9 ++--- Documentation/filesystems/vfs.rst | 8 ++--- fs/nfs/direct.c | 20 ----------- fs/nfs/file.c | 42 ++++++++++++++++++++--- fs/smb/client/file.c | 63 ++++++++++++++++++++++------------- include/linux/fs.h | 1 - include/linux/nfs_fs.h | 1 - include/linux/swap.h | 6 ---- include/linux/swap_ops.h | 5 +++ mm/page_io.c | 34 ++++--------------- 10 files changed, 93 insertions(+), 96 deletions(-) diff --git a/Documentation/filesystems/locking.rst b/Documentation/filesystems/locking.rst index 1a50d41a39a1..f58a8d7d5897 100644 --- a/Documentation/filesystems/locking.rst +++ b/Documentation/filesystems/locking.rst @@ -266,7 +266,6 @@ prototypes:: int (*error_remove_folio)(struct address_space *, struct folio *); int (*swap_activate)(struct swap_info_struct *sis, struct file *f, sector_t *span) int (*swap_deactivate)(struct file *); - int (*swap_rw)(struct kiocb *iocb, struct iov_iter *iter); locking rules: All except dirty_folio and free_folio may block @@ -291,7 +290,6 @@ is_partially_uptodate: yes error_remove_folio: yes swap_activate: no swap_deactivate: no -swap_rw: yes, unlocks ====================== ======================== ========= =============== ->write_begin(), ->write_end() and ->read_folio() may be called from @@ -355,15 +353,12 @@ should perform any validation and preparation necessary to ensure that writes can be performed with minimal memory allocation. It should call add_swap_extent(), or the helper iomap_swapfile_activate(), and return the number of extents added. If IO should be submitted through -->swap_rw(), it should call swap_fs_activate, otherwise IO will be submitted -directly to the block device ``sis->bdev``. +the file system it should call swap_fs_activate, otherwise IO will be +submitted directly to the block device ``sis->bdev``. ->swap_deactivate() will be called in the sys_swapoff() path after ->swap_activate() returned success. -->swap_rw will be called for swap IO if swap_fs_activate was called by -->swap_activate(). - file_lock_operations ==================== diff --git a/Documentation/filesystems/vfs.rst b/Documentation/filesystems/vfs.rst index e7677423a20f..c437a342d4f3 100644 --- a/Documentation/filesystems/vfs.rst +++ b/Documentation/filesystems/vfs.rst @@ -776,7 +776,6 @@ cache in your filesystem. The following members are defined: int (*error_remove_folio)(struct mapping *mapping, struct folio *); int (*swap_activate)(struct swap_info_struct *sis, struct file *f, sector_t *span) int (*swap_deactivate)(struct file *); - int (*swap_rw)(struct kiocb *iocb, struct iov_iter *iter); }; ``read_folio`` @@ -977,16 +976,13 @@ cache in your filesystem. The following members are defined: can be performed with minimal memory allocation. It should call add_swap_extent(), or the helper iomap_swapfile_activate(), and return the number of extents added. If IO should be submitted - through ->swap_rw(), it should call swap_fs_activate, otherwise IO will - be submitted directly to the block device ``sis->bdev``. + through the file system it should call swap_fs_activate, otherwise IO + will be submitted directly to the block device ``sis->bdev``. ``swap_deactivate`` Called during swapoff on files where swap_activate was successful. -``swap_rw`` - Called to read or write swap pages when swap_fs_activate was called. - The File Object =============== diff --git a/fs/nfs/direct.c b/fs/nfs/direct.c index e626c72495e6..ccafdc1ce64d 100644 --- a/fs/nfs/direct.c +++ b/fs/nfs/direct.c @@ -145,26 +145,6 @@ static void nfs_direct_file_adjust_size_locked(struct inode *inode, } } -/** - * nfs_swap_rw - NFS address space operation for swap I/O - * @iocb: target I/O control block - * @iter: I/O buffer - * - * Perform IO to the swap-file. This is much like direct IO. - */ -int nfs_swap_rw(struct kiocb *iocb, struct iov_iter *iter) -{ - ssize_t ret; - - if (iov_iter_rw(iter) == READ) - ret = nfs_file_direct_read(iocb, iter, true); - else - ret = nfs_file_direct_write(iocb, iter, true); - if (ret < 0) - return ret; - return 0; -} - static void nfs_direct_release_pages(struct page **pages, unsigned int npages) { unsigned int i; diff --git a/fs/nfs/file.c b/fs/nfs/file.c index 851d93a09988..e1bdd10b35f1 100644 --- a/fs/nfs/file.c +++ b/fs/nfs/file.c @@ -29,9 +29,8 @@ #include #include #include -#include #include - +#include #include #include @@ -575,6 +574,38 @@ static int nfs_launder_folio(struct folio *folio) return ret; } +#ifdef CONFIG_SWAP +static void nfs_swap_submit_write(struct swap_io_ctx *ctx) +{ + struct swap_iocb *sio = ctx->sio; + struct iov_iter iter; + int ret; + + swap_fs_prepare_rw(ctx, WRITE, &iter); + ret = nfs_file_direct_write(&sio->iocb, &iter, true); + if (ret != -EIOCBQUEUED) + sio->iocb.ki_complete(&sio->iocb, ret); +} + +static void nfs_swap_submit_read(struct swap_io_ctx *ctx) +{ + struct swap_iocb *sio = ctx->sio; + struct iov_iter iter; + int ret; + + swap_fs_prepare_rw(ctx, READ, &iter); + ret = nfs_file_direct_read(&sio->iocb, &iter, true); + if (ret != -EIOCBQUEUED) + sio->iocb.ki_complete(&sio->iocb, ret); +} + +static const struct swap_ops nfs_swap_ops = { + .flags = SWAP_OPS_F_REQUIRE_NOFS, + .submit_write = nfs_swap_submit_write, + .submit_read = nfs_swap_submit_read, + .can_merge = swap_fs_can_merge, +}; + static int nfs_swap_activate(struct swap_info_struct *sis, struct file *file, sector_t *span) { @@ -597,7 +628,7 @@ static int nfs_swap_activate(struct swap_info_struct *sis, struct file *file, ret = rpc_clnt_swap_activate(clnt); if (ret) return ret; - ret = swap_fs_activate(sis); + ret = swap_fs_activate(sis, &nfs_swap_ops); if (ret < 0) { rpc_clnt_swap_deactivate(clnt); return ret; @@ -620,6 +651,10 @@ static void nfs_swap_deactivate(struct file *file) if (cl->rpc_ops->disable_swap) cl->rpc_ops->disable_swap(file_inode(file)); } +#else +#define nfs_swap_activate NULL +#define nfs_swap_deactivate NULL +#endif /* CONFIG_SWAP */ const struct address_space_operations nfs_file_aops = { .read_folio = nfs_read_folio, @@ -636,7 +671,6 @@ const struct address_space_operations nfs_file_aops = { .error_remove_folio = generic_error_remove_folio, .swap_activate = nfs_swap_activate, .swap_deactivate = nfs_swap_deactivate, - .swap_rw = nfs_swap_rw, }; /* diff --git a/fs/smb/client/file.c b/fs/smb/client/file.c index 7f2924ce2881..ead69232ac1c 100644 --- a/fs/smb/client/file.c +++ b/fs/smb/client/file.c @@ -20,7 +20,7 @@ #include #include #include -#include +#include #include #include #include "cifsfs.h" @@ -3410,6 +3410,38 @@ out: cifs_done_oplock_break(cinode); } +#ifdef CONFIG_SWAP +static void cifs_swap_submit_write(struct swap_io_ctx *ctx) +{ + struct swap_iocb *sio = ctx->sio; + struct iov_iter iter; + int ret; + + swap_fs_prepare_rw(ctx, WRITE, &iter); + ret = netfs_unbuffered_write_iter_locked(&sio->iocb, &iter, NULL); + if (ret != -EIOCBQUEUED) + sio->iocb.ki_complete(&sio->iocb, ret); +} + +static void cifs_swap_submit_read(struct swap_io_ctx *ctx) +{ + struct swap_iocb *sio = ctx->sio; + struct iov_iter iter; + int ret; + + swap_fs_prepare_rw(ctx, READ, &iter); + ret = netfs_unbuffered_read_iter_locked(&sio->iocb, &iter); + if (ret != -EIOCBQUEUED) + sio->iocb.ki_complete(&sio->iocb, ret); +} + +static const struct swap_ops cifs_swap_ops = { + .flags = SWAP_OPS_F_REQUIRE_NOFS, + .submit_write = cifs_swap_submit_write, + .submit_read = cifs_swap_submit_read, + .can_merge = swap_fs_can_merge, +}; + static int cifs_swap_activate(struct swap_info_struct *sis, struct file *swap_file, sector_t *span) { @@ -3420,7 +3452,7 @@ static int cifs_swap_activate(struct swap_info_struct *sis, cifs_dbg(FYI, "swap activate\n"); - if (!swap_file->f_mapping->a_ops->swap_rw) + if (swap_file->f_mapping->a_ops != &cifs_addr_ops) /* Cannot support swap */ return -EINVAL; @@ -3451,7 +3483,7 @@ static int cifs_swap_activate(struct swap_info_struct *sis, * but we could add call to grab a byte range lock to prevent others * from reading or writing the file */ - return swap_fs_activate(sis); + return swap_fs_activate(sis, &cifs_swap_ops); } static void cifs_swap_deactivate(struct file *file) @@ -3467,26 +3499,10 @@ static void cifs_swap_deactivate(struct file *file) /* do we need to unpin (or unlock) the file */ } - -/** - * cifs_swap_rw - SMB3 address space operation for swap I/O - * @iocb: target I/O control block - * @iter: I/O buffer - * - * Perform IO to the swap-file. This is much like direct IO. - */ -static int cifs_swap_rw(struct kiocb *iocb, struct iov_iter *iter) -{ - ssize_t ret; - - if (iov_iter_rw(iter) == READ) - ret = netfs_unbuffered_read_iter_locked(iocb, iter); - else - ret = netfs_unbuffered_write_iter_locked(iocb, iter, NULL); - if (ret < 0) - return ret; - return 0; -} +#else +#define cifs_swap_activate NULL +#define cifs_swap_deactivate NULL +#endif /* CONFIG_SWAP */ const struct address_space_operations cifs_addr_ops = { .read_folio = netfs_read_folio, @@ -3503,7 +3519,6 @@ const struct address_space_operations cifs_addr_ops = { */ .swap_activate = cifs_swap_activate, .swap_deactivate = cifs_swap_deactivate, - .swap_rw = cifs_swap_rw, }; /* diff --git a/include/linux/fs.h b/include/linux/fs.h index 50ce731a2b78..87b5e9957c00 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -438,7 +438,6 @@ struct address_space_operations { int (*swap_activate)(struct swap_info_struct *sis, struct file *file, sector_t *span); void (*swap_deactivate)(struct file *file); - int (*swap_rw)(struct kiocb *iocb, struct iov_iter *iter); }; extern const struct address_space_operations empty_aops; diff --git a/include/linux/nfs_fs.h b/include/linux/nfs_fs.h index ec17e602c979..764056498eba 100644 --- a/include/linux/nfs_fs.h +++ b/include/linux/nfs_fs.h @@ -548,7 +548,6 @@ static inline const struct cred *nfs_file_cred(struct file *file) /* * linux/fs/nfs/direct.c */ -int nfs_swap_rw(struct kiocb *iocb, struct iov_iter *iter); ssize_t nfs_file_direct_read(struct kiocb *iocb, struct iov_iter *iter, bool swap); ssize_t nfs_file_direct_write(struct kiocb *iocb, diff --git a/include/linux/swap.h b/include/linux/swap.h index 8dd68733c955..5658a1634b85 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -341,8 +341,6 @@ extern void __meminit kswapd_run(int nid); extern void __meminit kswapd_stop(int nid); #ifdef CONFIG_SWAP - -int swap_fs_activate(struct swap_info_struct *sis); int add_swap_extent(struct swap_info_struct *sis, unsigned long start_page, unsigned long nr_pages, sector_t start_block); int generic_swapfile_activate(struct swap_info_struct *, struct file *, @@ -468,10 +466,6 @@ static inline bool folio_free_swap(struct folio *folio) return false; } -static inline int swap_fs_activate(struct swap_info_struct *sis) -{ - return -EINVAL; -} static inline int add_swap_extent(struct swap_info_struct *sis, unsigned long start_page, unsigned long nr_pages, sector_t start_block) diff --git a/include/linux/swap_ops.h b/include/linux/swap_ops.h index e92b4f532604..57ac6c703f68 100644 --- a/include/linux/swap_ops.h +++ b/include/linux/swap_ops.h @@ -36,4 +36,9 @@ struct swap_ops { void (*submit_read)(struct swap_io_ctx *ctx); }; +void swap_fs_prepare_rw(struct swap_io_ctx *ctx, int rw, struct iov_iter *iter); +bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio, + size_t prev_folio_size, int rw); +int swap_fs_activate(struct swap_info_struct *sis, const struct swap_ops *ops); + #endif /* _MM_SWAP_OPS_H */ diff --git a/mm/page_io.c b/mm/page_io.c index e741e67d6592..88962571cb93 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -650,11 +650,9 @@ const struct swap_ops swap_bdev_ops = { .can_merge = swap_bdev_can_merge, }; -static void swap_fs_submit(struct swap_io_ctx *ctx, int rw) +void swap_fs_prepare_rw(struct swap_io_ctx *ctx, int rw, struct iov_iter *iter) { struct swap_iocb *sio = ctx->sio; - struct iov_iter iter; - int ret; init_sync_kiocb(&sio->iocb, ctx->sis->swap_file); sio->iocb.ki_pos = swap_dev_pos(bvec_folio(&sio->bvecs[0])->swap); @@ -663,40 +661,22 @@ static void swap_fs_submit(struct swap_io_ctx *ctx, int rw) else sio->iocb.ki_complete = swap_fs_read_complete; - iov_iter_bvec(&iter, rw == WRITE ? ITER_SOURCE : ITER_DEST, + iov_iter_bvec(iter, rw == WRITE ? ITER_SOURCE : ITER_DEST, sio->bvecs, sio->nr_bvecs, sio->len); - ret = sio->iocb.ki_filp->f_mapping->a_ops->swap_rw(&sio->iocb, &iter); - if (ret != -EIOCBQUEUED) - sio->iocb.ki_complete(&sio->iocb, ret); } +EXPORT_SYMBOL_GPL(swap_fs_prepare_rw); -static void swap_fs_submit_write(struct swap_io_ctx *ctx) -{ - swap_fs_submit(ctx, WRITE); -} - -static void swap_fs_submit_read(struct swap_io_ctx *ctx) -{ - swap_fs_submit(ctx, READ); -} - -static bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio, +bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio, size_t prev_folio_size, int rw) { return swap_dev_pos(folio->swap) == swap_dev_pos(prev_folio->swap) + prev_folio_size; } +EXPORT_SYMBOL_GPL(swap_fs_can_merge); -static const struct swap_ops swap_fs_ops = { - .flags = SWAP_OPS_F_REQUIRE_NOFS, - .submit_write = swap_fs_submit_write, - .submit_read = swap_fs_submit_read, - .can_merge = swap_fs_can_merge, -}; - -int swap_fs_activate(struct swap_info_struct *sis) +int swap_fs_activate(struct swap_info_struct *sis, const struct swap_ops *ops) { - sis->ops = &swap_fs_ops; + sis->ops = ops; return add_swap_extent(sis, 0, sis->max, 0); } EXPORT_SYMBOL_GPL(swap_fs_activate); -- cgit v1.2.3 From bd1ad3cf07d11fbf203ac362222807113321dfbb Mon Sep 17 00:00:00 2001 From: Hui Su Date: Tue, 11 Aug 2026 15:33:32 +0800 Subject: kasan: fix quarantine_size accounting during cache removal quarantine_size tracks the total number of bytes stored in global_quarantine[]. It is incremented when per-CPU quarantine objects are moved into the global quarantine and decremented when a global batch is evicted by kasan_quarantine_reduce(). kasan_quarantine_remove_cache() also removes objects from the global quarantine. qlist_move_cache() rebuilds the source batch and updates its .bytes field, but quarantine_size is not adjusted accordingly. As a result, quarantine_size remains over-counted by the size of the removed objects. The stale accounting accumulates across cache removals. Once the inflated value exceeds quarantine_max_size, kasan_quarantine_reduce() can evict a batch even though the actual number of bytes in global_quarantine[] is still below quarantine_max_size, shortening the quarantine window. Fix the accounting by recording each batch's size before qlist_move_cache() and subtracting the number of bytes actually removed from quarantine_size while holding quarantine_lock. A KUnit reproducer used during testing observed the over-count grow by 4698864 bytes after one kasan_quarantine_remove_cache() call with the fix reverted. With this change applied, the over-count did not grow. Link: https://lore.kernel.org/20260811073332.1351893-1-sh_def@163.com Fixes: 64abdcb24351 ("kasan: eliminate long stalls during quarantine reduction") Signed-off-by: Hui Su Reported-by: Sashiko Closes: https://sashiko.dev/#/patchset/20260808031459.3032812-1-sh_def%40163.com Reviewed-by: Andrey Ryabinin Cc: Alexander Potapenko Cc: Andrey Konovalov Cc: Dmitry Vyukov Cc: Vincenzo Frascino Signed-off-by: Andrew Morton --- mm/kasan/quarantine.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/kasan/quarantine.c b/mm/kasan/quarantine.c index 16f4e67beee8..c9944fdf48ca 100644 --- a/mm/kasan/quarantine.c +++ b/mm/kasan/quarantine.c @@ -370,9 +370,14 @@ void kasan_quarantine_remove_cache(struct kmem_cache *cache) raw_spin_lock_irqsave(&quarantine_lock, flags); for (i = 0; i < QUARANTINE_BATCHES; i++) { + size_t old_bytes; + if (qlist_empty(&global_quarantine[i])) continue; + old_bytes = global_quarantine[i].bytes; qlist_move_cache(&global_quarantine[i], &to_free, cache); + WRITE_ONCE(quarantine_size, quarantine_size - + (old_bytes - global_quarantine[i].bytes)); /* Scanning whole quarantine can take a while. */ raw_spin_unlock_irqrestore(&quarantine_lock, flags); cond_resched(); -- cgit v1.2.3 From 6f615890b84820c2e223bd14238319f0415eae88 Mon Sep 17 00:00:00 2001 From: Wilson Felipe Pereira Date: Tue, 11 Aug 2026 05:14:11 +0000 Subject: selftests/cgroup: test_zswap: skip test_no_kmem_bypass if debugfs is unavailable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit test_no_kmem_bypass() needs to read /sys/kernel/debug/zswap/stored_pages via get_zswap_stored_pages() to verify that compressed pages are charged to the memcg. When running in an environment where debugfs is not mounted or CONFIG_DEBUG_FS is disabled, get_zswap_stored_pages() fails, causing the loop to terminate early and report a false negative (KSFT_FAIL). Selftests should not fail if debugfs is unavailable, and it should print a message when it is skipped. While I'm here, also add a warning message if the test is being skipped due to totalram size and make the check for totalram more readable. [akpm@linux-foundation.org: clarify debugfs-unavailable error message] Link: https://lore.kernel.org/20260812050848.848882-1-wfelipe@google.com Link: https://lore.kernel.org/20260811051434.3805648-1-wfelipe@google.com Signed-off-by: Wilson Felipe Pereira Reviewed-by: Anshuman Khandual Reviewed-by: SJ Park Cc: Chengming Zhou Cc: Johannes Weiner Cc: Michal Koutný Cc: Nhat Pham Cc: Shuah Khan Cc: Tejun Heo Signed-off-by: Andrew Morton --- tools/testing/selftests/cgroup/lib/include/cgroup_util.h | 1 + tools/testing/selftests/cgroup/test_zswap.c | 13 +++++++++++-- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/cgroup/lib/include/cgroup_util.h b/tools/testing/selftests/cgroup/lib/include/cgroup_util.h index febc1723d090..c0f07226b222 100644 --- a/tools/testing/selftests/cgroup/lib/include/cgroup_util.h +++ b/tools/testing/selftests/cgroup/lib/include/cgroup_util.h @@ -7,6 +7,7 @@ #endif #define MB(x) (x << 20) +#define GB(x) ((unsigned long long)(x) << 30) #define USEC_PER_SEC 1000000L #define NSEC_PER_SEC 1000000000L diff --git a/tools/testing/selftests/cgroup/test_zswap.c b/tools/testing/selftests/cgroup/test_zswap.c index 49b36ee79160..f7b4c4370db6 100644 --- a/tools/testing/selftests/cgroup/test_zswap.c +++ b/tools/testing/selftests/cgroup/test_zswap.c @@ -20,6 +20,7 @@ static int page_size; #define PATH_ZSWAP "/sys/module/zswap" #define PATH_ZSWAP_ENABLED "/sys/module/zswap/parameters/enabled" +#define PATH_ZSWAP_STORED_PAGES "/sys/kernel/debug/zswap/stored_pages" static int read_int(const char *path, size_t *value) { @@ -55,7 +56,7 @@ static int read_min_free_kb(size_t *value) static int get_zswap_stored_pages(size_t *value) { - return read_int("/sys/kernel/debug/zswap/stored_pages", value); + return read_int(PATH_ZSWAP_STORED_PAGES, value); } static long get_cg_wb_count(const char *cg) @@ -570,8 +571,16 @@ static int test_no_kmem_bypass(const char *root) /* Read sys info and compute test values accordingly */ if (sysinfo(&sys_info) != 0) return KSFT_FAIL; - if (sys_info.totalram > 5000000000) + if (sys_info.totalram > GB(4)) { + ksft_print_msg( + "requires less than 4GB total ram, sys_info.totalram: %.1fGB\n", + (double)sys_info.totalram / GB(1)); return KSFT_SKIP; + } + if (access(PATH_ZSWAP_STORED_PAGES, R_OK)) { + ksft_print_msg("debugfs not mounted at /sys/kernel/debug\n"); + return KSFT_SKIP; + } values = mmap(0, sizeof(struct no_kmem_bypass_child_args), PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0); if (values == MAP_FAILED) -- cgit v1.2.3 From 508537753b5ca06c1de9658239648510c9a999cc Mon Sep 17 00:00:00 2001 From: Hui Zhu Date: Wed, 12 Aug 2026 14:59:33 +0800 Subject: mm/mglru: fix young counter undercount for large folios lru_gen_look_around() feeds its local 'young' counter into suitable_to_scan(), which decides whether the current PMD is added to the bloom filter and checked again on the next aging round. The folio triggering the look-around is processed at function entry: test_and_clear_young_ptes_notify() clears the accessed bits of the nr PTEs it maps, and the function bails out if none of them is young. The loop that follows therefore never recounts this folio, since its accessed bits are already cleared. Every other young folio the loop finds is accounted as a batch (young += nr), where nr is the number of consecutive PTEs it maps. The triggering folio, however, still contributes a fixed young = 1 regardless of its size -- a leftover from before PTE batching. A large triggering folio is thus accounted inconsistently with the rest of the window. Initialize young to nr so the triggering folio is accounted the same way as any other young folio batch in the loop. Note this is a deliberate overestimate, not a measured value. The test-and-clear helper only reports whether any of the nr PTEs is young, not how many were accessed, so the true number of accessed PTEs in a large folio is unknown and can be smaller than nr. Counting the full batch is intentional: the mm core tracks accessed/dirty state per folio, not per page, so a per-page count is neither obtainable nor meaningful. The only consumer is suitable_to_scan(), and the bloom filter it feeds tolerates error. Overestimating is also the safe direction: at worst a PMD that saw little access is rescanned, whereas underestimating could skip rescanning a PMD whose folios are still hot and reclaim them incorrectly. (nr here is the PTE batch size, not necessarily folio_nr_pages().) Link: https://lore.kernel.org/20260813061019.49806-1-hui.zhu@linux.dev Link: https://lore.kernel.org/20260812065933.103627-1-hui.zhu@linux.dev Fixes: 56e5b60b2114 ("mm: support batched checking of the young flag for MGLRU") Signed-off-by: Hui Zhu Reviewed-by: Baolin Wang Reviewed-by: Barry Song Cc: Axel Rasmussen Cc: David Hildenbrand Cc: Johannes Weiner Cc: Kairui Song Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Shakeel Butt Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index be6bd26e8c57..6c35f7e21465 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4261,7 +4261,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr) unsigned long end; struct lru_gen_mm_walk *walk; struct folio *last = NULL; - int young = 1; + int young = nr; pte_t *pte = pvmw->pte; unsigned long addr = pvmw->address; struct vm_area_struct *vma = pvmw->vma; -- cgit v1.2.3 From 3372b6631b52b3442a1e3fe379bacc433bd7eec8 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Wed, 12 Aug 2026 08:43:31 +0100 Subject: MAINTAINERS: add drivers/char/mem.c to mm misc, memory mapping sections This file is a 'special' driver that implements /dev/zero and /dev/mem among other things. As such it makes sense for mm to be cc'd on mails and to have some say in how things are changed there, so add it to the mm misc section. Uniquely, it provides the 'old way' of obtaining an anonymous mapping - MAP_PRIVATE of /dev/zero - so is directly tied to memory mapping, therefore also add it to the memory mapping section. scripts/get_maintainer.pl copes perfectly fine with files in multiple sections so everything should work correctly. Link: https://lore.kernel.org/20260812-add-drivers-mem-to-mm-maintainers-v1-1-6218b861f4c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Acked-by: Mike Rapoport (Microsoft) Acked-by: SJ Park Reviewed-by: Anshuman Khandual Cc: Arnd Bergmann Cc: Jann Horn Cc: Liam R. Howlett Cc: Michal Hocko Cc: Mike Rapoport Cc: Pedro Falcato Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- MAINTAINERS | 2 ++ 1 file changed, 2 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 604285d848e6..0f513b42bc18 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17115,6 +17115,7 @@ F: Documentation/ABI/testing/sysfs-kernel-mm-memory-tiers F: Documentation/ABI/testing/sysfs-kernel-mm-numa F: Documentation/admin-guide/mm/ F: Documentation/mm/ +F: drivers/char/mem.c F: include/linux/cma.h F: include/linux/dmapool.h F: include/linux/ioremap.h @@ -17319,6 +17320,7 @@ L: linux-mm@kvack.org S: Maintained W: http://www.linux-mm.org T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm +F: drivers/char/mem.c F: include/trace/events/mmap.h F: fs/proc/task_mmu.c F: fs/proc/task_nommu.c -- cgit v1.2.3 From b86a7d03ea5368a0943afe7e0648ce3edf83eef8 Mon Sep 17 00:00:00 2001 From: "Nico Pache (Red Hat)" Date: Tue, 11 Aug 2026 06:48:33 -0600 Subject: mm/khugepaged: refactor per-scan state clearing into collapse_control_init_scan() Patch series "mm/khugepaged: several cleanups", v4. The following changes stem from a number of reviews during my khugepaged mTHP support series [1]. Some of these are minor code cleanups, issues or reviews that we decided to deferred to a followup series, or in the case of the more major patch of the series, changes [2] Lance Yang attempted while my series was in-flight and we decided to wait till later to try. The first 3 patches introduce helper functions to increase code reuse and readability. This includes a per-scan state clearing function, extracting the young page check into a helper, and a count_collapse_event() function to reduce a repetative pattern used across mTHP collapse. The 4th patch was the byproduct of me throwing Claude at all the comments in khugepaged verifying and looking for any outdated info. The 5th patch is based on Lance Yang's commit series [2] trying to extract the PTE state checking into a helper function. This required a bit of rewriting due to differences after mTHP collapse was introduced. I also took into account the changes requested during his patches review cycle. The remaining 2 patches were review points during my mTHP series that we agreed can be deferred to a later series. Thank you to those whos reviews and work I leveraged to achieve these cleanups. This patch (of 6): Extract the repeated clearing of node_load, alloc_nmask, and mthp_present_ptes into a helper to reduce duplication in collapse_scan_pmd() and collapse_scan_file(). Althought file scans do not current use the bitmap, they will in the future, and clearing it now is harmless. Link: https://lore.kernel.org/20260811-khugepaged_pte_refactor-v4-0-ddac39d61c4a@linux.dev Link: https://lore.kernel.org/20260811-khugepaged_pte_refactor-v4-1-ddac39d61c4a@linux.dev Link: https://lore.kernel.org/all/20260605161422.213817-1-npache@redhat.com/ [1] Link: https://lore.kernel.org/all/20251008043748.45554-1-lance.yang@linux.dev/ [2] Signed-off-by: Nico Pache (Red Hat) Reviewed-by: Baolin Wang Acked-by: Usama Arif Acked-by: David Hildenbrand (Arm) Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Zi Yan Reviewed-by: Pedro Falcato Reviewed-by: Lance Yang Cc: Barry Song Cc: Dev Jain Cc: Jonathan Corbet Cc: Liam R. Howlett Cc: Michal Hocko Cc: Mike Rapoport Cc: Ryan Roberts Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- mm/khugepaged.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index b237f6e7662a..1e26ea97381a 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -629,6 +629,13 @@ void __khugepaged_exit(struct mm_struct *mm) } } +static void collapse_control_init_scan(struct collapse_control *cc) +{ + memset(cc->node_load, 0, sizeof(cc->node_load)); + nodes_clear(cc->alloc_nmask); + bitmap_zero(cc->mthp_present_ptes, MAX_PTRS_PER_PTE); +} + static void release_pte_folio(struct folio *folio) { node_stat_mod_folio(folio, @@ -1617,9 +1624,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, goto out; } - bitmap_zero(cc->mthp_present_ptes, MAX_PTRS_PER_PTE); - memset(cc->node_load, 0, sizeof(cc->node_load)); - nodes_clear(cc->alloc_nmask); + collapse_control_init_scan(cc); enabled_orders = collapse_possible_orders(vma, vma->vm_flags, tva_flags); @@ -2691,8 +2696,7 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm, present = 0; swap = 0; - memset(cc->node_load, 0, sizeof(cc->node_load)); - nodes_clear(cc->alloc_nmask); + collapse_control_init_scan(cc); rcu_read_lock(); xas_for_each(&xas, folio, start + HPAGE_PMD_NR - 1) { if (xas_retry(&xas, folio)) -- cgit v1.2.3 From e8122742cfb4e28f3a499c09c08f968a692647d4 Mon Sep 17 00:00:00 2001 From: "Nico Pache (Red Hat)" Date: Tue, 11 Aug 2026 06:48:34 -0600 Subject: mm/khugepaged: extract reference check into folio_pte_referenced() helper This change deduplicates the "is this PTE/folio referenced enough to be considered for a collapse" condition that was repeated in both __collapse_huge_page_isolate() and collapse_scan_pmd(), extracting it into a single inline helper function. Also move the comment and use it as the function header. While we are at it, updated the comment to clarify that a young pte is a recently accessed one. [nico.pache@linux.dev: drop the trivial helper kerneldoc and inline marker per review] Link: https://lore.kernel.org/9038f552-926b-4c4c-b023-69271f45e3d5@linux.dev Link: https://lore.kernel.org/20260811-khugepaged_pte_refactor-v4-2-ddac39d61c4a@linux.dev Signed-off-by: Nico Pache (Red Hat) Acked-by: Usama Arif Acked-by: David Hildenbrand (Arm) Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Reviewed-by: Lance Yang Cc: Barry Song Cc: Dev Jain Cc: Jonathan Corbet Cc: Liam R. Howlett Cc: Lorenzo Stoakes (ARM) Cc: Michal Hocko Cc: Mike Rapoport Cc: Ryan Roberts Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Pedro Falcato Signed-off-by: Andrew Morton --- mm/khugepaged.c | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 1e26ea97381a..34654d1c1259 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -672,6 +672,16 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte, } } +static bool folio_pte_referenced(struct folio *folio, + struct vm_area_struct *vma, unsigned long addr, pte_t pteval) +{ + /* The folio was referenced previously ... */ + if (folio_test_young(folio) || folio_test_referenced(folio)) + return true; + /* ... or the PTE mapping was recently used */ + return pte_young(pteval) || mmu_notifier_test_young(vma->vm_mm, addr); +} + static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma, unsigned long start_addr, pte_t *pte, struct collapse_control *cc, unsigned int order, struct list_head *compound_pagelist) @@ -810,14 +820,8 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma, if (folio_test_large(folio)) list_add_tail(&folio->lru, compound_pagelist); next: - /* - * If collapse was initiated by khugepaged, check that there is - * enough young pte to justify collapsing the page - */ if (cc->is_khugepaged && - (pte_young(pteval) || folio_test_young(folio) || - folio_test_referenced(folio) || - mmu_notifier_test_young(vma->vm_mm, addr))) + folio_pte_referenced(folio, vma, addr, pteval)) referenced++; } @@ -1766,14 +1770,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, goto out_unmap; } - /* - * If collapse was initiated by khugepaged, check that there is - * enough young pte to justify collapsing the page - */ if (cc->is_khugepaged && - (pte_young(pteval) || folio_test_young(folio) || - folio_test_referenced(folio) || - mmu_notifier_test_young(vma->vm_mm, addr))) + folio_pte_referenced(folio, vma, addr, pteval)) referenced++; } if (cc->is_khugepaged && -- cgit v1.2.3 From 948ec48e568649bc639088c605515afbc1bf9e18 Mon Sep 17 00:00:00 2001 From: "Nico Pache (Red Hat)" Date: Tue, 11 Aug 2026 06:48:35 -0600 Subject: mm/khugepaged: introduce a count_collapse_event() helper Provide a simple helper function to help reduce a often used, and duplicate pattern across the khugepaged code. When collapsing to a PMD we need to record a vm_event and the mTHP_stat event. When doing mTHP collapse we only update the mTHP stat. Link: https://lore.kernel.org/20260811-khugepaged_pte_refactor-v4-3-ddac39d61c4a@linux.dev Signed-off-by: Nico Pache (Red Hat) Reviewed-by: Baolin Wang Acked-by: David Hildenbrand (Arm) Acked-by: Usama Arif Reviewed-by: Zi Yan Reviewed-by: Pedro Falcato Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Lance Yang Cc: Barry Song Cc: Dev Jain Cc: Jonathan Corbet Cc: Liam R. Howlett Cc: Michal Hocko Cc: Mike Rapoport Cc: Ryan Roberts Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- mm/khugepaged.c | 42 +++++++++++++++++++++--------------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 34654d1c1259..6d7206e5f4d2 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -682,6 +682,14 @@ static bool folio_pte_referenced(struct folio *folio, return pte_young(pteval) || mmu_notifier_test_young(vma->vm_mm, addr); } +static void count_collapse_event(unsigned int order, enum vm_event_item vm_event, + enum mthp_stat_item mthp_event) +{ + if (is_pmd_order(order)) + count_vm_event(vm_event); + count_mthp_stat(order, mthp_event); +} + static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma, unsigned long start_addr, pte_t *pte, struct collapse_control *cc, unsigned int order, struct list_head *compound_pagelist) @@ -702,9 +710,8 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma, if (pte_none_or_zero(pteval)) { if (++none_or_zero > max_ptes_none) { result = SCAN_EXCEED_NONE_PTE; - if (is_pmd_order(order)) - count_vm_event(THP_SCAN_EXCEED_NONE_PTE); - count_mthp_stat(order, MTHP_STAT_COLLAPSE_EXCEED_NONE); + count_collapse_event(order, THP_SCAN_EXCEED_NONE_PTE, + MTHP_STAT_COLLAPSE_EXCEED_NONE); goto out; } continue; @@ -746,9 +753,8 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma, */ if (++shared > max_ptes_shared) { result = SCAN_EXCEED_SHARED_PTE; - if (is_pmd_order(order)) - count_vm_event(THP_SCAN_EXCEED_SHARED_PTE); - count_mthp_stat(order, MTHP_STAT_COLLAPSE_EXCEED_SHARED); + count_collapse_event(order, THP_SCAN_EXCEED_SHARED_PTE, + MTHP_STAT_COLLAPSE_EXCEED_SHARED); goto out; } } @@ -1258,15 +1264,12 @@ static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_stru folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask); if (!folio) { *foliop = NULL; - if (is_pmd_order(order)) - count_vm_event(THP_COLLAPSE_ALLOC_FAILED); - count_mthp_stat(order, MTHP_STAT_COLLAPSE_ALLOC_FAILED); + count_collapse_event(order, THP_COLLAPSE_ALLOC_FAILED, + MTHP_STAT_COLLAPSE_ALLOC_FAILED); return SCAN_ALLOC_HUGE_PAGE_FAIL; } - if (is_pmd_order(order)) - count_vm_event(THP_COLLAPSE_ALLOC); - count_mthp_stat(order, MTHP_STAT_COLLAPSE_ALLOC); + count_collapse_event(order, THP_COLLAPSE_ALLOC, MTHP_STAT_COLLAPSE_ALLOC); if (unlikely(mem_cgroup_charge(folio, mm, gfp))) { folio_put(folio); @@ -1656,9 +1659,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, if (pte_none_or_zero(pteval)) { if (++none_or_zero > max_ptes_none) { result = SCAN_EXCEED_NONE_PTE; - count_vm_event(THP_SCAN_EXCEED_NONE_PTE); - count_mthp_stat(HPAGE_PMD_ORDER, - MTHP_STAT_COLLAPSE_EXCEED_NONE); + count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_NONE_PTE, + MTHP_STAT_COLLAPSE_EXCEED_NONE); goto out_unmap; } continue; @@ -1666,9 +1668,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, if (!pte_present(pteval)) { if (++unmapped > max_ptes_swap) { result = SCAN_EXCEED_SWAP_PTE; - count_vm_event(THP_SCAN_EXCEED_SWAP_PTE); - count_mthp_stat(HPAGE_PMD_ORDER, - MTHP_STAT_COLLAPSE_EXCEED_SWAP); + count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_SWAP_PTE, + MTHP_STAT_COLLAPSE_EXCEED_SWAP); goto out_unmap; } /* @@ -1725,9 +1726,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, if (folio_maybe_mapped_shared(folio)) { if (++shared > max_ptes_shared) { result = SCAN_EXCEED_SHARED_PTE; - count_vm_event(THP_SCAN_EXCEED_SHARED_PTE); - count_mthp_stat(HPAGE_PMD_ORDER, - MTHP_STAT_COLLAPSE_EXCEED_SHARED); + count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_SHARED_PTE, + MTHP_STAT_COLLAPSE_EXCEED_SHARED); goto out_unmap; } } -- cgit v1.2.3 From cc044178edc9d7b5cd291ef0e2daf060379e6447 Mon Sep 17 00:00:00 2001 From: "Nico Pache (Red Hat)" Date: Tue, 11 Aug 2026 06:48:36 -0600 Subject: mm/khugepaged: fix outdated comments Fix comment in collapse_scan_pmd() that still described the old folio_mapcount() > folio_ref_count() check and a "512" false-positive scenario. The code now uses folio_expected_ref_count() != folio_ref_count() which doesn't suffer from the same limitation. Fix comment in collapse_huge_page() that referenced ptep_clear_flush, when the code actually uses pmdp_collapse_flush. Fix comment in __collapse_huge_page_swapin() that referenced the old function name khugepaged_scan_pmd, now collapse_scan_pmd. Also clean up some simple typos and stale terminology (mmap_sem -> mmap_lock, PG_lock -> folio lock, page -> folio, grammar). We also clarify a comment regarding where the max_ptes_none check is deferred to in mthp_collapse() from the original collapse_scan_pmd check. Update all comments that references a function to include parentheses. [nico.pache@linux.dev: fix outdated comments] Link: https://lore.kernel.org/1c96e2f3-802f-472b-81e6-4af17a721a3c@linux.dev Link: https://lore.kernel.org/20260811-khugepaged_pte_refactor-v4-4-ddac39d61c4a@linux.dev Signed-off-by: Nico Pache (Red Hat) Acked-by: Usama Arif Assisted-by: Cursor(claude-sonnet-4):4.6 Acked-by: David Hildenbrand (Arm) Reviewed-by: Zi Yan Acked-by: Pedro Falcato Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Lance Yang Cc: Baolin Wang Cc: Barry Song Cc: Dev Jain Cc: Jonathan Corbet Cc: Liam R. Howlett Cc: Michal Hocko Cc: Mike Rapoport Cc: Ryan Roberts Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- mm/khugepaged.c | 44 +++++++++++++++++++++----------------------- 1 file changed, 21 insertions(+), 23 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 6d7206e5f4d2..30f17c7494fa 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -620,7 +620,7 @@ void __khugepaged_exit(struct mm_struct *mm) /* * This is required to serialize against * collapse_test_exit() (which is guaranteed to run - * under mmap sem read mode). Stop here (after we return all + * under mmap_lock read mode). Stop here (after we return all * pagetables will be destroyed) until khugepaged has finished * working on the pagetables under the mmap_lock. */ @@ -782,8 +782,8 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma, /* * We can do it before folio_isolate_lru because the - * folio can't be freed from under us. NOTE: PG_lock - * is needed to serialize against split_huge_page + * folio can't be freed from under us. NOTE: folio lock + * is needed to serialize against split_huge_page() * when invoked from the VM. */ if (!folio_trylock(folio)) { @@ -809,7 +809,7 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma, } /* - * Isolate the page to avoid collapsing an hugepage + * Isolate the folio to avoid collapsing a hugepage * currently in use by the VM. */ if (!folio_isolate_lru(folio)) { @@ -921,7 +921,7 @@ static void __collapse_huge_page_copy_failed(pte_t *pte, * Re-establish the PMD to point to the original page table * entry. Restoring PMD needs to be done prior to releasing * pages. Since pages are still isolated and locked here, - * acquiring anon_vma_lock_write is unnecessary. + * acquiring anon_vma_lock_write() is unnecessary. */ pmd_ptl = pmd_lock(vma->vm_mm, pmd); pmd_populate(vma->vm_mm, pmd, pmd_pgtable(orig_pmd)); @@ -1095,9 +1095,9 @@ static enum scan_result hugepage_vma_revalidate(struct mm_struct *mm, unsigned l return SCAN_VMA_CHECK; /* * Anon VMA expected, the address may be unmapped then - * remapped to file after khugepaged reaquired the mmap_lock. + * remapped to file after khugepaged reacquired the mmap_lock. * - * thp_vma_allowable_orders may return true for qualified file + * thp_vma_allowable_orders() may return true for qualified file * vmas. */ if (expect_anon && (!(*vmap)->anon_vma || !vma_is_anonymous(*vmap))) @@ -1153,7 +1153,7 @@ static enum scan_result check_pmd_still_valid(struct mm_struct *mm, /* * Bring missing pages in from swap, to complete THP collapse. - * Only done if khugepaged_scan_pmd believes it is worthwhile. + * Only done if collapse_scan_pmd() believes it is worthwhile. * * For mTHP orders the function bails on the first swap entry, because * faulting pages back in during collapse could re-populate PTEs that @@ -1221,7 +1221,7 @@ static enum scan_result __collapse_huge_page_swapin(struct mm_struct *mm, pte = NULL; /* - * do_swap_page returns VM_FAULT_RETRY with released mmap_lock. + * do_swap_page() returns VM_FAULT_RETRY with released mmap_lock. * Note we treat VM_FAULT_RETRY as VM_FAULT_ERROR here because * we do not retry here and swap entry will remain in pagetable * resulting in later failure. @@ -1285,7 +1285,7 @@ static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_stru } /* - * collapse_huge_page expects the mmap_lock to be unlocked before entering and + * collapse_huge_page() expects the mmap_lock to be unlocked before entering and * will always return with the lock unlocked, to avoid holding the mmap_lock * while allocating a THP, as that could trigger direct reclaim/compaction. * Note that the VMA must be rechecked after grabbing the mmap_lock again. @@ -1332,7 +1332,7 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s if (unmapped) { /* - * __collapse_huge_page_swapin will return with mmap_lock + * __collapse_huge_page_swapin() will return with mmap_lock * released when it fails. So we jump out_nolock directly in * that case. Continuing to collapse causes inconsistency. */ @@ -1345,8 +1345,8 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s mmap_read_unlock(mm); /* * Prevent all access to pagetables with the exception of - * gup_fast later handled by the ptep_clear_flush and the VM - * handled by the anon_vma lock + PG_lock. + * gup_fast later handled by the pmdp_collapse_flush() and the VM + * handled by the anon_vma lock + folio lock. * * UFFDIO_MOVE is prevented to race as well thanks to the * mmap_lock. @@ -1403,9 +1403,9 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s spin_lock(pmd_ptl); VM_WARN_ON_ONCE(!pmd_none(*pmd)); /* - * We can only use set_pmd_at when establishing + * We can only use set_pmd_at() when establishing * hugepmds and never for establishing regular pmds that - * points to regular pagetables. Use pmd_populate for that + * points to regular pagetables. Use pmd_populate() for that */ pmd_populate(mm, pmd, pmd_pgtable(_pmd)); spin_unlock(pmd_ptl); @@ -1637,7 +1637,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, /* * If PMD is the only enabled order, enforce max_ptes_none, otherwise - * scan all pages to populate the bitmap for mTHP collapse. + * scan all pages to populate the bitmap for mTHP collapse. The bitmap + * is then checked again in mthp_collapse() for each attempted order. */ if (enabled_orders != BIT(HPAGE_PMD_ORDER)) max_ptes_none = KHUGEPAGED_MAX_PTES_LIMIT; @@ -1758,12 +1759,9 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, /* * Check if the page has any GUP (or other external) pins. * - * Here the check may be racy: - * it may see folio_mapcount() > folio_ref_count(). - * But such case is ephemeral we could always retry collapse - * later. However it may report false positive if the page - * has excessive GUP pins (i.e. 512). Anyway the same check - * will be done again later the risk seems low. + * Here the check is racy, but such cases are ephemeral and + * we can always retry collapse later. Anyway the same + * check will be done again later, so the risk seems to be low. */ if (folio_expected_ref_count(folio) != folio_ref_count(folio)) { result = SCAN_PAGE_COUNT; @@ -1784,7 +1782,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, out_unmap: pte_unmap_unlock(pte, ptl); if (result == SCAN_SUCCEED) { - /* collapse_huge_page expects the lock to be dropped before calling */ + /* collapse_huge_page() expects the lock to be dropped before calling */ mmap_read_unlock(mm); result = mthp_collapse(mm, start_addr, referenced, unmapped, cc, enabled_orders); -- cgit v1.2.3 From 27490db7ec048175522368ac0aa2e99249e5c48d Mon Sep 17 00:00:00 2001 From: "Nico Pache (Red Hat)" Date: Tue, 11 Aug 2026 06:48:38 -0600 Subject: mm/khugepaged: unmap pte before releasing vma write lock We are currently dropping the anon_vma write lock before unmapping the PTE. Although this is safe, due to us still holding the mmap_write_lock, its safer and less confusing to switch the order of these two operations. Link: https://lore.kernel.org/20260811-khugepaged_pte_refactor-v4-6-ddac39d61c4a@linux.dev Signed-off-by: Nico Pache (Red Hat) Suggested-by: David Hildenbrand Acked-by: David Hildenbrand (Arm) Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Acked-by: Pedro Falcato Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Lance Yang Cc: Barry Song Cc: Dev Jain Cc: Jonathan Corbet Cc: Liam R. Howlett Cc: Michal Hocko Cc: Mike Rapoport Cc: Ryan Roberts Cc: Suren Baghdasaryan Cc: Usama Arif Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- mm/khugepaged.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 30f17c7494fa..11ff98d55c76 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1463,10 +1463,10 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s result = SCAN_SUCCEED; out_up_write: - if (anon_vma_locked) - anon_vma_unlock_write(vma->anon_vma); if (pte) pte_unmap(pte); + if (anon_vma_locked) + anon_vma_unlock_write(vma->anon_vma); mmap_write_unlock(mm); out_nolock: if (folio) -- cgit v1.2.3 From 2a0be246e342e239ef91d28e2658b6bf508cc065 Mon Sep 17 00:00:00 2001 From: "Nico Pache (Red Hat)" Date: Tue, 11 Aug 2026 06:48:39 -0600 Subject: mm: Documentation: clarify where the mTHP stats live The note about khugepaged counters references /proc/vmstat for the PMD case, but never mentions where the mTHPs stats can be found (i.e.: /sys/kernel/mm/transparent_hugepage/hugepages-kB/stats/) Add a small addition to this section for clarity. Also fix a missing period while we are at it. Link: https://lore.kernel.org/20260811-khugepaged_pte_refactor-v4-7-ddac39d61c4a@linux.dev Signed-off-by: Nico Pache (Red Hat) Reviewed-by: Baolin Wang Suggested-by: Lorenzo Stoakes Acked-by: David Hildenbrand (Arm) Reviewed-by: Zi Yan Acked-by: Pedro Falcato Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Lance Yang Cc: Barry Song Cc: Dev Jain Cc: Jonathan Corbet Cc: Liam R. Howlett Cc: Michal Hocko Cc: Mike Rapoport Cc: Ryan Roberts Cc: Suren Baghdasaryan Cc: Usama Arif Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- Documentation/admin-guide/mm/transhuge.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst index 16f37135ed80..b187d618452f 100644 --- a/Documentation/admin-guide/mm/transhuge.rst +++ b/Documentation/admin-guide/mm/transhuge.rst @@ -224,7 +224,7 @@ khugepaged will be automatically started when any THP size is enabled (either of the per-size anon control or the top-level control are set to "always" or "madvise"), and it'll be automatically shutdown when all THP sizes are disabled (when both the per-size anon control and the -top-level control are "never") +top-level control are "never"). process THP controls -------------------- @@ -301,7 +301,9 @@ being replaced by a PMD mapping, or (2) physical pages replaced by one hugepage of various sizes (PMD-sized or mTHP). Each may happen independently, or together, depending on the type of memory and the failures that occur. As such, this value should be interpreted roughly as a sign of progress, -and counters in /proc/vmstat consulted for more accurate accounting):: +and counters in /proc/vmstat consulted for more accurate accounting. +Per-order mTHP collapse statistics are also available under +/sys/kernel/mm/transparent_hugepage/hugepages-kB/stats/):: /sys/kernel/mm/transparent_hugepage/khugepaged/pages_collapsed -- cgit v1.2.3 From 73b5d07990a0e6ea9cdf2c07fa8fbc865d398c1d Mon Sep 17 00:00:00 2001 From: Song Hu Date: Wed, 12 Aug 2026 15:57:39 +0800 Subject: Docs/mm: fix outdated "radix tree" in page_migration Steps 7 and 9 of the migration description still say "radix tree", unlike steps 5 and 11 which already use "i_pages lock". The page cache moved to the XArray at mapping->i_pages long ago. Use "page cache tree" for the two remaining references. Link: https://lore.kernel.org/20260812075739.325441-1-husong@kylinos.cn Signed-off-by: Song Hu Cc: David Hildenbrand Cc: Jonathan Corbet Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Randy Dunlap Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Matthew Wilcox Cc: Jan Kara Signed-off-by: Andrew Morton --- Documentation/mm/page_migration.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Documentation/mm/page_migration.rst b/Documentation/mm/page_migration.rst index 34602b254aa6..5b8d50308db1 100644 --- a/Documentation/mm/page_migration.rst +++ b/Documentation/mm/page_migration.rst @@ -110,13 +110,13 @@ Steps: 6. The refcount of the page is examined and we back out if references remain. Otherwise, we know that we are the only one referencing this page. -7. The radix tree is checked and if it does not contain the pointer to this - page then we back out because someone else modified the radix tree. +7. The page cache tree is checked and if it does not contain the pointer to this + page then we back out because someone else modified the page cache tree. 8. The new page is prepped with some settings from the old page so that accesses to the new page will discover a page with the correct settings. -9. The radix tree is changed to point to the new page. +9. The page cache tree is changed to point to the new page. 10. The reference count of the old page is dropped because the address space reference is gone. A reference to the new page is established because -- cgit v1.2.3 From f7e698e326b239a91ea15844817551921209e826 Mon Sep 17 00:00:00 2001 From: Kairui Song Date: Wed, 12 Aug 2026 20:22:39 +0800 Subject: mm/mglru: fix and remove redundant unevictable folio handling sort_folio() has a shortcut for moving folios that are no longer evictable but are still sitting on a generation list. However, this shortcut is buggy. It does not follow the PG_lru usage convention, and it has a more serious issue. Unevictable folios are not threaded on lists[LRU_UNEVICTABLE], so that folio->lru can be reused to hold folio->mlock_count (see the comment in lruvec_init()). Hence lruvec_add_folio() skips the list_add() for them, and every other place that turns a folio unevictable initialises mlock_count explicitly: lru_add() sets it to 0, __mlock_folio() and __mlock_new_folio() set it to !!folio_test_mlocked(folio). sort_folio() sets nothing, and the lru_gen_del_folio() right above it may have already poisoned folio->lru via list_del(), so mlock_count ends up aliasing LIST_POISON2, which reads as 0x122, i.e. 290. The result is user visible. On munlock, __munlock_folio() decrements that bogus count, finds it still non-zero and bails out before clearing PG_mlocked, so the folio remains unevictable and the Mlocked accounting stays inflated until the folio is freed. The shortcut also touches the LRU flags in the wrong order. It calls lru_gen_del_folio() while PG_lru is still set, so a concurrent folio_test_clear_lru() (e.g. compaction, folio_isolate_lru()) can succeed on a folio that has already been taken off the generation list, which may lead to unexpected behavior. So fix it by isolating them as common folios and letting the generic shrink path cull them. This matches the classical LRU behavior, and there should be no visible effect on the generic eviction or isolation behavior. There is no performance concern either, such a folio goes through this once, and then it is off the generation lists for good. Link: https://lore.kernel.org/20260812-mglru-mlock-fix-v2-1-a3fec5853c08@tencent.com Fixes: ac35a4902374 ("mm: multi-gen LRU: minimal implementation") Signed-off-by: Kairui Song Reviewed-by: Barry Song Reviewed-by: Baolin Wang Cc: Axel Rasmussen Cc: Brian Geffon Cc: David Hildenbrand Cc: Jan Alexander Steffens (heftig) Cc: Johannes Weiner Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Oleksandr Natalenko Cc: Shakeel Butt Cc: Steven Barrett Cc: Suleiman Souhlal Cc: Wei Xu Cc: Yuanchu Xie Cc: Yu Zhao Cc: Signed-off-by: Andrew Morton --- mm/vmscan.c | 19 +++++-------------- 1 file changed, 5 insertions(+), 14 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 6c35f7e21465..c1404a59523d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4649,7 +4649,6 @@ void lru_gen_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent, static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_control *sc, int tier_idx) { - bool success; int gen = folio_lru_gen(folio); int type = folio_is_file_lru(folio); int zone = folio_zonenum(folio); @@ -4661,15 +4660,9 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c VM_WARN_ON_ONCE_FOLIO(gen >= MAX_NR_GENS, folio); - /* unevictable */ - if (!folio_evictable(folio)) { - success = lru_gen_del_folio(lruvec, folio, true); - VM_WARN_ON_ONCE_FOLIO(!success, folio); - folio_set_unevictable(folio); - lruvec_add_folio(lruvec, folio); - __count_vm_events(UNEVICTABLE_PGCULLED, delta); - return true; - } + /* unevictable: let it through and the generic path will cull it */ + if (!folio_evictable(folio)) + return false; /* promoted */ if (gen != lru_gen_from_seq(lrugen->min_seq[type])) { @@ -4922,11 +4915,9 @@ retry: list_for_each_entry_safe_reverse(folio, next, &list, lru) { DEFINE_MIN_SEQ(lruvec); - if (!folio_evictable(folio)) { - list_del(&folio->lru); - folio_putback_lru(folio); + /* move_folios_to_lru() culls unevictable folios via folio_putback_lru() */ + if (!folio_evictable(folio)) continue; - } /* retry folios that may have missed folio_rotate_reclaimable() */ if (!skip_retry && !folio_test_active(folio) && !folio_mapped(folio) && -- cgit v1.2.3 From f525001b3309a0decdcdcdaeceafd7ab9dc927fe Mon Sep 17 00:00:00 2001 From: Tejun Heo Date: Wed, 12 Aug 2026 09:47:43 -1000 Subject: percpu: drop CONFIG_DEBUG_FORCE_WEAK_PER_CPU alpha requires percpu variables in modules to be defined as weak so that the compiler generates GOT based external references for them. This puts two extra restrictions on percpu variable definitions. The symbol must be globally unique even when static and a static percpu variable can't be defined inside a function. DEBUG_FORCE_WEAK_PER_CPU exists to give generic code build coverage for these restrictions without building for alpha. MEM_ALLOC_PROFILING defines a static percpu counter at each allocation call site and thus can't be built with weak percpu definitions, so it depends on !DEBUG_FORCE_WEAK_PER_CPU. As allmodconfig enables DEBUG_FORCE_WEAK_PER_CPU, this knocks MEM_ALLOC_PROFILING out of allmodconfig build coverage. allmodconfig coverage for MEM_ALLOC_PROFILING is worth more than build coverage for restrictions which only matter to alpha module builds. Drop DEBUG_FORCE_WEAK_PER_CPU. Restriction violations will now show up only on alpha builds. Link: https://lore.kernel.org/178656406317.2437052.7257990869957704195@slm.duckdns.org Signed-off-by: Tejun Heo Reported-by: Andrew Morton Reviewed-by: Suren Baghdasaryan Acked-by: Gabriele Monaco [include/rv/da_monitor.h] Cc: Dennis Zhou Cc: Kent Overstreet Cc: Steven Rostedt Signed-off-by: Andrew Morton --- include/linux/percpu-defs.h | 7 +------ include/rv/da_monitor.h | 2 +- lib/Kconfig.debug | 15 --------------- mm/Kconfig.debug | 1 - 4 files changed, 2 insertions(+), 23 deletions(-) diff --git a/include/linux/percpu-defs.h b/include/linux/percpu-defs.h index 2cba7cc2b01f..dbe3267a0a13 100644 --- a/include/linux/percpu-defs.h +++ b/include/linux/percpu-defs.h @@ -65,13 +65,8 @@ * * Archs which need weak percpu definitions should set * CONFIG_ARCH_MODULE_NEEDS_WEAK_PER_CPU when necessary. - * - * To ensure that the generic code observes the above two - * restrictions, if CONFIG_DEBUG_FORCE_WEAK_PER_CPU is set weak - * definition is used for all cases. */ -#if (defined(CONFIG_ARCH_MODULE_NEEDS_WEAK_PER_CPU) && defined(MODULE)) || \ - defined(CONFIG_DEBUG_FORCE_WEAK_PER_CPU) +#if defined(CONFIG_ARCH_MODULE_NEEDS_WEAK_PER_CPU) && defined(MODULE) /* * __pcpu_scope_* dummy variable is used to enforce scope. It * receives the static modifier when it's used in front of diff --git a/include/rv/da_monitor.h b/include/rv/da_monitor.h index 34b8fba9ecd4..6b641697106b 100644 --- a/include/rv/da_monitor.h +++ b/include/rv/da_monitor.h @@ -24,7 +24,7 @@ /* * Per-cpu variables require a unique name although static in some - * configurations (e.g. CONFIG_DEBUG_FORCE_WEAK_PER_CPU or alpha modules). + * configurations (e.g. alpha modules). */ #define DA_MON_NAME CONCATENATE(da_mon_, MONITOR_NAME) diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index b82515cde538..00921b1676e8 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -613,21 +613,6 @@ config BUILTIN_MODULE_RANGES It also records an anchor symbol to determine the load address of the section. -config DEBUG_FORCE_WEAK_PER_CPU - bool "Force weak per-cpu definitions" - depends on DEBUG_KERNEL - help - s390 and alpha require percpu variables in modules to be - defined weak to work around addressing range issue which - puts the following two restrictions on percpu variable - definitions. - - 1. percpu symbols must be unique whether static or not - 2. percpu variables can't be defined inside a function - - To ensure that generic code follows the above rules, this - option forces all percpu variables to be defined as weak. - config WARN_CONTEXT_ANALYSIS bool "Compiler context-analysis warnings" depends on CC_IS_CLANG && CLANG_VERSION >= 230000 diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index 5737a504efbb..15dca19dd07d 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -326,7 +326,6 @@ config MEM_ALLOC_PROFILING default n depends on MMU depends on PROC_FS - depends on !DEBUG_FORCE_WEAK_PER_CPU select CODE_TAGGING select PAGE_EXTENSION select SLAB_OBJ_EXT -- cgit v1.2.3 From 0d0878fd7c9b49b8eff95c4426a2121c5f8f31cc Mon Sep 17 00:00:00 2001 From: Qiang Liu Date: Wed, 12 Aug 2026 17:28:56 +0800 Subject: lib/test_hmm: fix garbage pfn and wrong direction in devmem fault debug Move pr_debug() inside the `if (dpage)` block to avoid printing garbage pfn for NULL dpage, and correct the direction label from "sys to dev" to "dev to sys". Link: https://lore.kernel.org/20260812092856.55296-1-liuqiangneo@163.com Signed-off-by: Qiang Liu Assisted-by: Qoder:Qwen-3.8-MAX-Preview Cc: Jason Gunthorpe Cc: Leon Romanovsky Signed-off-by: Andrew Morton --- lib/test_hmm.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/lib/test_hmm.c b/lib/test_hmm.c index 6205fb313bd0..7c4d10eae6fe 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -1151,10 +1151,9 @@ static vm_fault_t dmirror_devmem_fault_alloc_and_copy(struct migrate_vma *args, if (!dpage && !order) return VM_FAULT_OOM; - pr_debug("migrating from sys to dev pfn src: 0x%lx pfn dst: 0x%lx\n", - page_to_pfn(spage), page_to_pfn(dpage)); - if (dpage) { + pr_debug("migrating from dev to sys pfn src: 0x%lx pfn dst: 0x%lx\n", + page_to_pfn(spage), page_to_pfn(dpage)); lock_page(dpage); *dst |= migrate_pfn(page_to_pfn(dpage)); } -- cgit v1.2.3 From a1b114b4cec1263e693cd6e7aa5c8321295143c6 Mon Sep 17 00:00:00 2001 From: Xie Yuanbin Date: Thu, 13 Aug 2026 21:49:16 +0800 Subject: mm/Kconfig: make MEMORY_FAILURE select MIGRATION For embedded devices, lacking support for NUMA, memory hotplug/hotremove, CMA and huge pages is a quite common scenario. In this scenario, the demand for contiguous physical memory allocation is very low. To reduce the kernel image size, some devices disable the compaction. However, their SoCs do support DDR ECC, meaning that memory-failure may be needed. Migration is very useful for soft_offline_page() in memory-failure, which may be triggered by correctable memory errors. Most anonymous and file-mapped faulty pages can be migrated to other healthy pages. Currently, MEMORY_FAILURE does not explicitly select MIGRATION. When COMPACTION, MEMORY_HOTREMOVE, NUMA_MIGRATION and CMA are all disabled, MEMORY_FAILURE can be enabled, but MIGRATION cannot be selected. Make MEMORY_FAILURE select MIGRATION to handle this situation. Link: https://lore.kernel.org/20260813134916.292733-1-xieyuanbin1@huawei.com Signed-off-by: Xie Yuanbin Suggested-by: Mike Rapoport Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Mike Rapoport (Microsoft) Acked-by: Zi Yan Acked-by: David Hildenbrand (Arm) Acked-by: Miaohe Lin Cc: Alistair Popple Cc: "Borislav Petkov (AMD)" Cc: Byungchul Park Cc: David Hildenbrand Cc: Gregory Price Cc: "Huang, Ying" Cc: Joshua Hahn Cc: Liam R. Howlett Cc: liaohua Cc: "Luck, Tony" Cc: Matthew Brost Cc: Michal Hocko Cc: Naoya Horiguchi Cc: Rakie Kim Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Yuanbin Xie Signed-off-by: Andrew Morton --- mm/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/Kconfig b/mm/Kconfig index 8a24c130d008..604c58199acb 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -767,6 +767,7 @@ config MEMORY_FAILURE depends on ARCH_SUPPORTS_MEMORY_FAILURE bool "Enable recovery from hardware memory errors" select INTERVAL_TREE + select MIGRATION help Enables code to recover from some memory failures on systems with MCA recovery. This allows a system to continue running -- cgit v1.2.3 From 556147fc27b5d9c3c731ae4c5599457831102735 Mon Sep 17 00:00:00 2001 From: Andrew Morton Date: Mon, 10 Aug 2026 13:16:37 -0700 Subject: mm/hmm.c:hmm_do_fault(): suppress sparse warning mm/hmm.c:673 hmm_do_fault() error: we previously assumed 'hmm_vma_walk->locked' could be null (see line 654) Stanislav says this can't happen. Waste a few cycles to make the warning go away. [akpm@linux-foundation.org: WARN_ON_ONCE() if the handler didn't set ->locked, per Stanislav] Link: https://lore.kernel.org/anu1N-DOnQwxO1kF@skinsburskii Fixes: 121170831228 ("mm/hmm: add hmm_range_fault_unlocked_timeout() for mmap lock-drop support") Reported-by: kernel test robot Closes: https://lore.kernel.org/202608101053.PhnVUM4u-lkp@intel.com Cc: Stanislav Kinsburskii Cc: David Hildenbrand Signed-off-by: Andrew Morton --- mm/hmm.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/hmm.c b/mm/hmm.c index 2b05c53b82dc..2f1e98c6b644 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -670,7 +670,10 @@ static int hmm_do_fault(struct mm_struct *mm, ret = handle_mm_fault(vma, addr, fault_flags, NULL); if (ret & (VM_FAULT_COMPLETED | VM_FAULT_RETRY)) { - *hmm_vma_walk->locked = false; + if (hmm_vma_walk->locked) /* needed by sparse */ + *hmm_vma_walk->locked = false; + else + WARN_ON_ONCE(1); /* broken fault handler */ return HMM_FAULT_UNLOCKED; } -- cgit v1.2.3 From f52b3b89faba20cb347f4b908f649fa6351ff10d Mon Sep 17 00:00:00 2001 From: Eric Kim Date: Fri, 14 Aug 2026 15:30:51 +0900 Subject: mm/rmap: synchronize lock and unlock target in anon_vma_clone Currently, in anon_vma_clone(), src vma's anon_vma is assigned to active_anon_vma and is used when unlocking anon_vma after linking new AVCs. However, the anon_vma is locked using src->anon_vma, instead of active_anon_vma, making the lock and unlock target inconsistent. Use active_anon_vma for both locking and unlocking. Link: https://lore.kernel.org/OS7PR01MB139142FE16EC63B892559D40496DA2@OS7PR01MB13914.jpnprd01.prod.outlook.com Signed-off-by: Eric Kim Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Lance Yang Cc: David Hildenbrand Cc: Harry Yoo Cc: Jann Horn Cc: Liam R. Howlett Cc: Rik van Riel Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- mm/rmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/rmap.c b/mm/rmap.c index 14f2f9b07572..d1819fd69938 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -350,7 +350,7 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src, * Now link the anon_vma's back to the newly inserted AVCs. * Note that all anon_vma's share the same root. */ - anon_vma_lock_write(src->anon_vma); + anon_vma_lock_write(active_anon_vma); list_for_each_entry_reverse(avc, &dst->anon_vma_chain, same_vma) { struct anon_vma *anon_vma = avc->anon_vma; -- cgit v1.2.3 From f2b1cb39d5ccab090d8353788f186f7e7a1fffd4 Mon Sep 17 00:00:00 2001 From: Nathan Chancellor Date: Thu, 13 Aug 2026 20:12:55 -0700 Subject: arch_numa: avoid false positive fortify warning in setup_node_to_cpumask_map() When building ARCH=riscv using clang with CONFIG_FORTIFY_SOURCE and CONFIG_UBSAN_BOUNDS enabled, CONFIG_NR_CPUS > 64, and the default value of 2 for CONFIG_NODES_SHIFT, there is a compiletime warning from the fortify routines. In file included from mm/arch_numa.c:11: In file included from include/linux/acpi.h:14: In file included from include/linux/resource_ext.h:11: In file included from include/linux/slab.h:17: In file included from include/linux/gfp.h:7: In file included from include/linux/mmzone.h:8: In file included from include/linux/spinlock.h:60: In file included from include/linux/interrupt_rc.h:17: In file included from include/linux/smp.h:13: In file included from include/linux/cpumask.h:11: In file included from include/linux/bitmap.h:13: In file included from include/linux/string.h:383: include/linux/fortify-string.h:430:4: warning: call to '__write_overflow_field' declared with 'warning' attribute: detected write beyond size of field (1st parameter); maybe use struct_group()? [-Wattribue-warning] 430 | __write_overflow_field(p_size_field, size); | ^ include/linux/fortify-string.h:430:4: note: called by function 'fortify_memset_chk(unsigned long, unsigned long, unsigned long)' include/linux/bitmap.h:248:3: note: inlined by function 'setup_node_to_cpumask_map' 248 | memset(dst, 0, len); | ^ include/linux/fortify-string.h:462:25: note: expanded from macro 'memset' 462 | #define memset(p, c, s) __fortify_memset_chk(p, c, s, \ | ^ include/linux/fortify-string.h:453:2: note: expanded from macro '__fortify_memset_chk' 453 | fortify_memset_chk(__fortify_size, p_size, p_size_field), \ | ^ include/linux/fortify-string.h:430:4: note: use '-gline-directives-only' (implied by '-g1') or higher for more accurate inlining chain locations 430 | __write_overflow_field(p_size_field, size); | ^ 1 warning generated. In this configuration, MAX_NUMNODES is 4. clang unrolls the for loop in setup_node_to_cpumask_map() past this, which triggers the fortify check when accessing node_to_cpumask_map on the theoretical fifth loop iteration because it would be an out of bounds write. Make it clear to clang that nr_node_ids is bounded by MAX_NUMNODES due to the logic in setup_nr_node_ids() by early returning in setup_node_to_cpumask_map() should that condition be violated. Link: https://lore.kernel.org/20260813-arch_numa-avoid-fortify-warning-v2-1-093ad97a78df@kernel.org Signed-off-by: Nathan Chancellor Closes: https://github.com/ClangBuiltLinux/linux/issues/2174 Reviewed-by: Mike Rapoport (Microsoft) Cc: Kees Cook Cc: Bill Wendling Cc: Justin Stitt Cc: Nathan Chancellor Cc: Nick Desaulniers Cc: Signed-off-by: Andrew Morton --- mm/arch_numa.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/mm/arch_numa.c b/mm/arch_numa.c index 442ea239bba7..459fa60a5621 100644 --- a/mm/arch_numa.c +++ b/mm/arch_numa.c @@ -105,6 +105,18 @@ static void __init setup_node_to_cpumask_map(void) if (nr_node_ids == MAX_NUMNODES) setup_nr_node_ids(); + /* + * This check should never be true but it makes it clear to compilers + * that node_to_cpumask_map is bound by nr_node_ids, avoiding false + * positive fortify warnings when accessing node_to_cpumask_map in the + * for loop below. + */ + if (unlikely(nr_node_ids > MAX_NUMNODES)) { + pr_err("nr_node_ids (%u) is larger than MAX_NUMNODES (%u)\n", + nr_node_ids, MAX_NUMNODES); + return; + } + /* allocate and clear the mapping */ for (node = 0; node < nr_node_ids; node++) { alloc_bootmem_cpumask_var(&node_to_cpumask_map[node]); -- cgit v1.2.3 From c1afbd5de131f5e3c4fc7559acf055f8d9d86868 Mon Sep 17 00:00:00 2001 From: Usama Arif Date: Mon, 17 Aug 2026 03:38:35 -0700 Subject: mm/memcontrol: avoid false sharing between vmstats and events Moving v1 userspace eventfd handling into memcontrol-v1.c shrank struct vmpressure from 112 to 24 bytes when CONFIG_MEMCG_V1 is disabled. This moved memory_events_local[MEMCG_SWAP_FAIL] and the hot vmstats_percpu pointer onto the same cacheline. The stress-ng mremap stressor exercises MADV_PAGEOUT with swap disabled, generating about 20 million MEMCG_SWAP_FAIL updates per 60-second run on a 176-CPU test system. Those writes bounce the line while memcg statistics paths load vmstats_percpu. Move cgwb_list into the existing alignment gap and cacheline-align vmstats_percpu. This separates the pointer from the event counters without increasing the size of struct mem_cgroup in the tested configuration. The blamed commit reduced median mremap throughput by 4.38% on the test system with one socket. The patched kernel brings the performance to within 0.5% of the parent which is within the observed boot-to-boot spread (up to 1.2%). Link: https://lore.kernel.org/20260817103835.2937733-1-usama.arif@linux.dev Fixes: ea928e9e18da ("mm/vmpressure: move v1 userspace eventfd code into memcontrol-v1.c") Signed-off-by: Usama Arif Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202608131743.c6a7dda4-lkp@intel.com Tested-by: kernel test robot Link: http://lore.kernel.org/aoAABX59IzUXz/Rv@ly-workstation Acked-by: Shakeel Butt Acked-by: Michal Hocko Cc: David Hildenbrand Cc: Johannes Weiner Cc: Muchun Song Cc: Roman Gushchin Cc: Yi Lai Signed-off-by: Andrew Morton --- include/linux/memcontrol.h | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index e78bc98ab229..215e2e87f42b 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -268,10 +268,15 @@ struct mem_cgroup { #endif int kmemcg_id; - struct memcg_vmstats_percpu __percpu *vmstats_percpu; - #ifdef CONFIG_CGROUP_WRITEBACK struct list_head cgwb_list; +#endif + + /* Keep the hot per-CPU stats pointer away from memory event counters. */ + struct memcg_vmstats_percpu __percpu *vmstats_percpu + ____cacheline_aligned_in_smp; + +#ifdef CONFIG_CGROUP_WRITEBACK struct wb_domain cgwb_domain; struct memcg_cgwb_frn cgwb_frn[MEMCG_CGWB_FRN_CNT]; #endif -- cgit v1.2.3 From dd14e6cd33927fff38c78ce55c436bc0959ace27 Mon Sep 17 00:00:00 2001 From: Hongfu Li Date: Mon, 17 Aug 2026 16:06:16 +0800 Subject: selftests/mm: drop redundant open() in mprotect_tests() Remove duplicate open() for local pagemap_fd in mprotect_tests() that shadows the global pagemap_fd already opened in main(). The local fd is never used in the function. Link: https://lore.kernel.org/20260817080616.52946-1-hongfu.li@linux.dev Signed-off-by: Hongfu Li Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Muhammad Usama Anjum Reviewed-by: SJ Park Acked-by: David Hildenbrand (Arm) Reviewed-by: Anshuman Khandual Cc: Liam R. Howlett Cc: Michal Hocko Cc: Mike Rapoport Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- tools/testing/selftests/mm/pagemap_ioctl.c | 6 ------ 1 file changed, 6 deletions(-) diff --git a/tools/testing/selftests/mm/pagemap_ioctl.c b/tools/testing/selftests/mm/pagemap_ioctl.c index cfd1987339c1..eadc7159ca5b 100644 --- a/tools/testing/selftests/mm/pagemap_ioctl.c +++ b/tools/testing/selftests/mm/pagemap_ioctl.c @@ -1332,12 +1332,6 @@ int mprotect_tests(void) int ret; char *mem, *mem2; struct page_region vec; - int pagemap_fd = open("/proc/self/pagemap", O_RDONLY); - - if (pagemap_fd < 0) { - fprintf(stderr, "open() failed\n"); - exit(1); - } /* 1. Map two pages */ mem = mmap(0, 2 * page_size, PROT_READ|PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); -- cgit v1.2.3 From 9add2cc22de6c56881eabd9f1bd6c85bf98157d0 Mon Sep 17 00:00:00 2001 From: Hui Su Date: Mon, 17 Aug 2026 14:08:46 +0800 Subject: mm/migrate_device: fix cache flush when replacing huge zero PMD migrate_vma_insert_huge_pmd_page() calls flush_cache_page() before replacing an existing huge zero PMD. However, the third argument to flush_cache_page() is a PFN, while addr + HPAGE_PMD_SIZE is an end virtual address. More importantly, the mapping being invalidated is PMD-sized rather than PAGE_SIZE-sized. Flush the whole PMD range with flush_cache_range(), matching other huge PMD invalidation paths. There is no userspace-visible effect today. The architectures that currently enable ARCH_ENABLE_THP_MIGRATION use no-op implementations of flush_cache_page()/flush_cache_range(). 32-bit ARM has non-trivial implementations, but does not enable ARCH_ENABLE_THP_MIGRATION. So this appears to be a latent API misuse rather than a currently observable bug, and I don't think a stable backport is necessary. Link: https://lore.kernel.org/20260817060845.377800-2-sh_def@163.com Fixes: a30b48bf1b24 ("mm/migrate_device: implement THP migration of zone device pages") Signed-off-by: Hui Su Reviewed-by: Balbir Singh Reviewed-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Alistair Popple Cc: Byungchul Park Cc: Gregory Price Cc: "Huang, Ying" Cc: Joshua Hahn Cc: Matthew Brost Cc: Rakie Kim Signed-off-by: Andrew Morton --- mm/migrate_device.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 9a346162c688..762c5cee8fec 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -882,7 +882,7 @@ static int migrate_vma_insert_huge_pmd_page(struct migrate_vma *migrate, if (flush) { pte_free(vma->vm_mm, pgtable); - flush_cache_page(vma, addr, addr + HPAGE_PMD_SIZE); + flush_cache_range(vma, addr, addr + HPAGE_PMD_SIZE); pmdp_invalidate(vma, addr, pmdp); } else { pgtable_trans_huge_deposit(vma->vm_mm, pmdp, pgtable); -- cgit v1.2.3 From dd1638dfbb1c48f13cfb4f4f4e55e6570e25384b Mon Sep 17 00:00:00 2001 From: "Kiryl Shutsemau (Meta)" Date: Tue, 18 Aug 2026 12:50:26 +0100 Subject: mm: include swap.h in swapops.h swapops.h uses MAX_SWAPFILES_SHIFT, SWP_MIGRATION_READ and SWP_PTE_MARKER, all of which swap.h defines, but does not include swap.h. It compiles only where the translation unit pulled swap.h in first. leafops.h includes swapops.h on the line above swap.h, so a file whose include list reaches leafops.h before swap.h gets: In file included from include/linux/leafops.h:11: include/linux/swapops.h:88:21: error: use of undeclared identifier 'MAX_SWAPFILES_SHIFT' A header that uses a definition has to include the header that provides it. Link: https://lore.kernel.org/20260818115026.656406-1-kirill@shutemov.name Signed-off-by: Kiryl Shutsemau (Meta) Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-kbuild-all/202608181757.mza9RRj7-lkp@intel.com/ Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Barry Song Cc: Baoquan He Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Signed-off-by: Andrew Morton --- include/linux/swapops.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/linux/swapops.h b/include/linux/swapops.h index 1f3ff3b93e16..e7d0d529f3e0 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -5,6 +5,7 @@ #include #include #include +#include #ifdef CONFIG_MMU -- cgit v1.2.3 From 48863da10ba1ffe3889fc5945d3292b4e4bf1c60 Mon Sep 17 00:00:00 2001 From: Song Hu Date: Tue, 18 Aug 2026 21:01:35 +0800 Subject: mm: memcg: release the css reference when a stock slot empties consume_stock() can drive a stock slot's nr_pages to zero while its cached[] pointer stays set, so the slot keeps pinning the css reference that refill_stock() took. The offlining drain only flushes slots with cached pages, so the reference is never released unless the slot happens to be displaced by an unrelated charge or by CPU hotplug, and the memcg lingers in the dying state - up to NR_MEMCG_STOCK (7) of them per CPU under container churn. Keeping the slot populated past the last page only saves a css_get()/css_put() pair on the next charge of the same memcg, and costs more than that: the offlining drain has to know about empty slots, and refill_stock() cannot reuse them either, so a charge under a different memcg evicts a live batch through the drain_idx rotation instead. Drop the reference in consume_stock() when the slot empties. Empty slots stop existing, so is_memcg_drain_needed() and the drain path stay as they are, and refill_stock() reuses emptied slots directly. The cost is one refcount pair per emptied slot, at most once per MEMCG_CHARGE_BATCH pages. Link: https://lore.kernel.org/20260818130135.154315-1-husong@kylinos.cn Fixes: d1a05b6973c7 ("memcg: do not try to drain per-cpu caches without pages") Signed-off-by: Song Hu Acked-by: Michal Hocko Acked-by: Shakeel Butt Reviewed-by: Joshua Hahn Cc: Audra Mitchell Cc: Johannes Weiner Cc: Matthew Wilcox (Oracle) Cc: Muchun Song Cc: Roman Gushchin Cc: Nico Pache Signed-off-by: Andrew Morton --- mm/memcontrol.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1d3339520809..11b85f4b6828 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2140,7 +2140,12 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) stock_pages = READ_ONCE(stock->nr_pages[i]); if (stock_pages >= nr_pages) { - WRITE_ONCE(stock->nr_pages[i], stock_pages - nr_pages); + stock_pages -= nr_pages; + WRITE_ONCE(stock->nr_pages[i], stock_pages); + if (!stock_pages) { + css_put(&memcg->css); + WRITE_ONCE(stock->cached[i], NULL); + } ret = true; } break; -- cgit v1.2.3 From c7a4e939f87cc75a9a664485b10c0bf7db632156 Mon Sep 17 00:00:00 2001 From: Anshuman Date: Tue, 18 Aug 2026 19:02:06 +0530 Subject: selftests/mm: fix unchecked ftruncate return value in soft-dirty test test_mprotect() calls ftruncate() to resize the backing file before mmap()'ing it, but never checks the return value. If ftruncate() fails, the file may remain shorter than the requested mapping size. The subsequent mmap() with MAP_SHARED can still succeed in this case, but the very next line writes directly into the mapped memory (*map = 1), which can trigger SIGBUS if the mapping extends beyond the actual file size. Check the return value and fail cleanly with ksft_exit_fail_msg() if ftruncate() fails, matching the error-handling style already used for the mmap() call immediately below it. Link: https://lore.kernel.org/20260818133206.39503-1-anshumantewari123@gmail.com Signed-off-by: Anshuman Reviewed-by: Andrew Morton Reviewed-by: Sarthak Sharma Cc: David Hildenbrand Cc: Shuah Khan Signed-off-by: Andrew Morton --- tools/testing/selftests/mm/soft-dirty.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/mm/soft-dirty.c b/tools/testing/selftests/mm/soft-dirty.c index e198facf78bb..5f278913c4d7 100644 --- a/tools/testing/selftests/mm/soft-dirty.c +++ b/tools/testing/selftests/mm/soft-dirty.c @@ -152,7 +152,8 @@ static void test_mprotect(int pagemap_fd, int pagesize, bool anon) return; } unlink(fname); - ftruncate(test_fd, pagesize); + if (ftruncate(test_fd, pagesize) != 0) + ksft_exit_fail_msg("ftruncate failed\n"); map = mmap(NULL, pagesize, PROT_READ|PROT_WRITE, MAP_SHARED, test_fd, 0); if (map == MAP_FAILED) -- cgit v1.2.3 From f9dc428249ed962a70acf301f01eae8578449161 Mon Sep 17 00:00:00 2001 From: Breno Leitao Date: Tue, 18 Aug 2026 02:03:40 -0700 Subject: mm, swap: ratelimit bad swap entry reports A corrupt page table hands the same bogus entry to get_swap_device() on every access to the mapping, and every rejection is logged. One machine logged 6185620 copies of the same line in a few hours. swap_dup_entry_direct() prints the same message from the fork path, once per call: the WARN_ON_ONCE() guarding it warns once, the pr_err() inside does not. Rate limit all three prints. Link: https://lore.kernel.org/20260818-swap_part_one-v1-1-a4fc58119fc0@debian.org Fixes: 23b230ba8ac3 ("mm/swap: print bad swap offset entry in get_swap_device") Signed-off-by: Breno Leitao Reviewed-by: Barry Song Reviewed-by: Nhat Pham Acked-by: Kairui Song Acked-by: David Hildenbrand (Arm) Cc: Baoquan He Cc: Chris Li Cc: Kemeng Shi Cc: Miaohe Lin Cc: Oscar Salvador Cc: Signed-off-by: Andrew Morton --- mm/swapfile.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index dacef34a3ed7..53bf01d5f7f1 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1899,11 +1899,11 @@ struct swap_info_struct *get_swap_device(swp_entry_t entry) return si; bad_nofile: - pr_err("%s: %s%08lx\n", __func__, Bad_file, entry.val); + pr_err_ratelimited("%s: %s%08lx\n", __func__, Bad_file, entry.val); out: return NULL; put_out: - pr_err("%s: %s%08lx\n", __func__, Bad_offset, entry.val); + pr_err_ratelimited("%s: %s%08lx\n", __func__, Bad_offset, entry.val); percpu_ref_put(&si->users); return NULL; } @@ -3883,7 +3883,7 @@ int swap_dup_entry_direct(swp_entry_t entry) si = swap_entry_to_info(entry); if (WARN_ON_ONCE(!si)) { - pr_err("%s%08lx\n", Bad_file, entry.val); + pr_err_ratelimited("%s%08lx\n", Bad_file, entry.val); return -EINVAL; } -- cgit v1.2.3 From d16e52a9ba9ed5060f97ed3191017a21b5fc25a2 Mon Sep 17 00:00:00 2001 From: Anshuman Date: Wed, 19 Aug 2026 17:44:26 +0530 Subject: selftests/mm: check stat() return value in khugepaged get_finfo() get_finfo() calls stat() to get metadata about the target directory, but never checks the return value. On failure, stat() returns -1 and leaves path_stat unmodified, so path_stat.st_mode may contain uninitialized stack data. The code then checks S_ISDIR(path_stat.st_mode) against this potentially garbage value. This can produce a misleading "Not a directory" error when the real problem is a nonexistent or inaccessible path, or, in the worst case, the check could pass by chance on garbage data and let the function continue using an invalid path_stat for the rest of its logic. Check the return value and fail with a clear error message if stat() fails, matching the error-handling style already used for statfs() and read_file() later in the same function. Link: https://lore.kernel.org/20260819121426.49500-1-anshumantewari123@gmail.com Signed-off-by: Anshuman Reviewed-by: Andrew Morton Reviewed-by: SJ Park Reviewed-by: Sarthak Sharma Acked-by: David Hildenbrand (Arm) Cc: Lorenzo Stoakes Cc: Shuah Khan Signed-off-by: Andrew Morton --- tools/testing/selftests/mm/khugepaged.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/mm/khugepaged.c b/tools/testing/selftests/mm/khugepaged.c index d3a53673e1f9..1d2d6bd72fd2 100644 --- a/tools/testing/selftests/mm/khugepaged.c +++ b/tools/testing/selftests/mm/khugepaged.c @@ -124,7 +124,8 @@ static void get_finfo(const char *dir) char *str, *end; finfo.dir = dir; - stat(finfo.dir, &path_stat); + if (stat(finfo.dir, &path_stat)) + ksft_exit_fail_perror("stat()"); if (!S_ISDIR(path_stat.st_mode)) ksft_exit_fail_msg("%s: Not a directory (%s)\n", __func__, finfo.dir); if (snprintf(finfo.path, sizeof(finfo.path), "%s/" TEST_FILE, -- cgit v1.2.3 From 9e32ec53b1ec2ab28b29c82a95a65bf3d3a5d32c Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:09 -0400 Subject: maple_tree: add rcu locking check when LOCKDEP is enabled Patch series "maple_tree: lock checking and clean ups", v3. In this series: 1. Try to detect lock issues A number of syzbot reports are incorrectly pointing to the mm exit as a source of the locking error. The first three patches attempt to help users detect errors in their locking - but they still have to use LOCKDEP. I guess it's still down to hope and prayers. 2. Documentation fixes The documentation was lacking clarity, there are updates to try and help the users, especially around the erase() cases. 3. Two benign issues The cyclic allocator may have a race, although no in-kernel user can hit it. The erase functions may cause allocation issues if used with the incorrect locking type, but none are present in-tree. 4. The erase gfp uses mas_erase() and mtree_erase() do not take a gfp argument. To improve reliability of the erase, the first attempt to allocate will be GFP_NOWAIT, followed by a retry (if necessary of GFP_KERNEL | GFP_NOFAIL. This will ensure the data is gone. I've updated the documentation to make it more clear as well. mas_store() is not addressed in the same way, but may need to be updated at a later date, but that may require changing callers so it is out of scope here. Beyond these goals there are some test fixes, some general speed-up patches targeting extra work and cycles, and dropping dead code. This patch (of 19): When CONFIG_LOCKDEP and CONFIG_RCU_STRICT_GRACE_PERIOD is enabled, check for rcu locking issues by recording the grace period in the maple state and checking the rcu window is still valid whenever the maple state is reused with a state that is not MA_START or MA_PAUSED. Link: https://lore.kernel.org/20260821192627.4085470-1-liam@infradead.org Link: https://lore.kernel.org/20260821192627.4085470-2-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- include/linux/maple_tree.h | 3 +++ lib/maple_tree.c | 50 +++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 52 insertions(+), 1 deletion(-) diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h index 1b3014377105..1acf932fcd33 100644 --- a/include/linux/maple_tree.h +++ b/include/linux/maple_tree.h @@ -484,6 +484,9 @@ struct ma_state { unsigned char mas_flags; unsigned char end; /* The end of the node */ enum store_type store_type; /* The type of store needed for this operation */ +#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) + unsigned long rcu_gp; +#endif }; struct ma_wr_state { diff --git a/lib/maple_tree.c b/lib/maple_tree.c index a0542b491bc2..6d805521bedd 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1148,6 +1148,42 @@ static inline void mas_free(struct ma_state *mas, struct maple_enode *used) ma_free_rcu(mte_to_node(used)); } +void mas_lock_check(struct ma_state *mas) +{ + +#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) + if (!mas_is_active(mas)) + return; + + if (!mt_locked(mas->tree)) { + if (mt_in_rcu(mas->tree)) + WARN_ON_ONCE(poll_state_synchronize_rcu(mas->rcu_gp)); + } +#endif + +} + +void mas_init_lock_check(struct ma_state *mas) +{ +#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) + if (!mt_locked(mas->tree)) { + if (mt_in_rcu(mas->tree)) + mas->rcu_gp = get_state_synchronize_rcu(); + } +#endif + +} + +static void mas_may_init_lock_check(struct ma_state *mas) +{ +#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) + if (mas_is_start(mas) || mas_is_paused(mas)) + mas_init_lock_check(mas); + else + mas_lock_check(mas); +#endif +} + /* * mas_start() - Sets up maple state for operations. * @mas: The maple state. @@ -1166,6 +1202,7 @@ static inline struct maple_enode *mas_start(struct ma_state *mas) if (likely(mas_is_start(mas))) { struct maple_enode *root; + mas_init_lock_check(mas); mas->min = 0; mas->max = ULONG_MAX; @@ -4355,6 +4392,7 @@ void *mas_walk(struct ma_state *mas) { void *entry; + mas_may_init_lock_check(mas); if (!mas_is_active(mas) && !mas_is_start(mas)) mas->status = ma_start; retry: @@ -4992,6 +5030,7 @@ static void mas_may_activate(struct ma_state *mas) mas->status = ma_start; } else { mas->status = ma_active; + mas_lock_check(mas); } } @@ -5069,6 +5108,7 @@ void *mas_next(struct ma_state *mas, unsigned long max) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_next_setup(mas, max, &entry)) return entry; @@ -5092,6 +5132,7 @@ void *mas_next_range(struct ma_state *mas, unsigned long max) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_next_setup(mas, max, &entry)) return entry; @@ -5200,6 +5241,7 @@ void *mas_prev(struct ma_state *mas, unsigned long min) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_prev_setup(mas, min, &entry)) return entry; @@ -5223,6 +5265,7 @@ void *mas_prev_range(struct ma_state *mas, unsigned long min) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_prev_setup(mas, min, &entry)) return entry; @@ -5269,6 +5312,7 @@ EXPORT_SYMBOL_GPL(mt_prev); */ void mas_pause(struct ma_state *mas) { + mas_lock_check(mas); mas->status = ma_pause; mas->node = NULL; } @@ -5377,6 +5421,7 @@ void *mas_find(struct ma_state *mas, unsigned long max) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_find_setup(mas, max, &entry)) return entry; @@ -5404,6 +5449,7 @@ void *mas_find_range(struct ma_state *mas, unsigned long max) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_find_setup(mas, max, &entry)) return entry; @@ -5516,6 +5562,7 @@ void *mas_find_rev(struct ma_state *mas, unsigned long min) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_find_rev_setup(mas, min, &entry)) return entry; @@ -5542,6 +5589,7 @@ void *mas_find_range_rev(struct ma_state *mas, unsigned long min) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_find_rev_setup(mas, min, &entry)) return entry; @@ -5618,7 +5666,7 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp) if (!mas->sheaf && !mas->alloc) return false; - mas->status = ma_start; + mas_reset(mas); return true; } -- cgit v1.2.3 From 8f2109843137da8068f19a120d79e58ef3838429 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:10 -0400 Subject: locking/lockdep: add sequence counter to held_lock Add an 8 bit small sequence counter to the held_lock struct to detect if the lock as been dropped and reacquired. This is useful when a data structure depends on a constant locking context, but is not able to detect locking and unlocking of the lock through its own API. Since the __lock_unpin_lock() will no longer detect underflow by casting the unsigned int to a signed int, update the casting code to use a temp variable for calculations using a signed int. Link: https://lore.kernel.org/20260821192627.4085470-3-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Suggested-by: Peter Zijlstra Cc: Ingo Molnar Cc: Will Deacon Cc: Boqun Feng Cc: Waiman Long Link: https://lore.kernel.org/all/h3tpnj5kzcrxms5picmimtkpg4aypcpip5wbd6bt2rpdj5k7eb@nhtzs3lefrkq/ Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Jason Gunthorpe Cc: Joe Perches Cc: Rik van Riel Signed-off-by: Andrew Morton --- include/linux/lockdep.h | 3 +++ include/linux/lockdep_types.h | 3 ++- include/linux/sched.h | 1 + kernel/locking/lockdep.c | 58 +++++++++++++++++++++++++++++++++++++------ 4 files changed, 57 insertions(+), 8 deletions(-) diff --git a/include/linux/lockdep.h b/include/linux/lockdep.h index 621566345406..a6451ecbbe9a 100644 --- a/include/linux/lockdep.h +++ b/include/linux/lockdep.h @@ -273,6 +273,9 @@ extern struct pin_cookie lock_pin_lock(struct lockdep_map *lock); extern void lock_repin_lock(struct lockdep_map *lock, struct pin_cookie); extern void lock_unpin_lock(struct lockdep_map *lock, struct pin_cookie); +extern u32 lock_sequence(struct lockdep_map *lock); +#define lockdep_sequence(lock) lock_sequence(&(lock)->dep_map) + #define lockdep_depth(tsk) (debug_locks ? (tsk)->lockdep_depth : 0) #define lockdep_assert(cond) \ diff --git a/include/linux/lockdep_types.h b/include/linux/lockdep_types.h index eae115a26488..55c4b152fedf 100644 --- a/include/linux/lockdep_types.h +++ b/include/linux/lockdep_types.h @@ -253,7 +253,8 @@ struct held_lock { unsigned int hardirqs_off:1; unsigned int sync:1; unsigned int references:11; /* 32 bits */ - unsigned int pin_count; + unsigned int pin_count:24; + unsigned int seq_count:8; }; #else /* !CONFIG_LOCKDEP */ diff --git a/include/linux/sched.h b/include/linux/sched.h index 373bcc0598d1..14d5ce8dd613 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1288,6 +1288,7 @@ struct task_struct { u64 curr_chain_key; int lockdep_depth; unsigned int lockdep_recursion; + unsigned int lockdep_seq; struct held_lock held_locks[MAX_LOCK_DEPTH]; #endif diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c index 2d4c5bab5af8..a69567bdd791 100644 --- a/kernel/locking/lockdep.c +++ b/kernel/locking/lockdep.c @@ -5077,7 +5077,7 @@ static int __lock_is_held(const struct lockdep_map *lock, int read); static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass, int trylock, int read, int check, int hardirqs_off, struct lockdep_map *nest_lock, unsigned long ip, - int references, int pin_count, int sync) + int references, int pin_count, int sync, int seq) { struct task_struct *curr = current; struct lock_class *class = NULL; @@ -5183,6 +5183,7 @@ static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass, hlock->holdtime_stamp = lockstat_clock(); #endif hlock->pin_count = pin_count; + hlock->seq_count = seq; if (check_wait_context(curr, hlock)) return 0; @@ -5388,7 +5389,7 @@ static int reacquire_held_locks(struct task_struct *curr, unsigned int depth, hlock->read, hlock->check, hlock->hardirqs_off, hlock->nest_lock, hlock->acquire_ip, - hlock->references, hlock->pin_count, 0)) { + hlock->references, hlock->pin_count, 0, hlock->seq_count)) { case 0: return 1; case 1: @@ -5669,14 +5670,17 @@ static void __lock_unpin_lock(struct lockdep_map *lock, struct pin_cookie cookie struct held_lock *hlock = curr->held_locks + i; if (match_held_lock(hlock, lock)) { + int pin_count; + if (WARN(!hlock->pin_count, "unpinning an unpinned lock\n")) return; - hlock->pin_count -= cookie.val; + pin_count = hlock->pin_count - cookie.val; - if (WARN((int)hlock->pin_count < 0, "pin count corrupted\n")) - hlock->pin_count = 0; + if (WARN(pin_count < 0, "pin count corrupted\n")) + pin_count = 0; + hlock->pin_count = pin_count; return; } } @@ -5684,6 +5688,24 @@ static void __lock_unpin_lock(struct lockdep_map *lock, struct pin_cookie cookie WARN(1, "unpinning an unheld lock\n"); } +static u32 __lock_sequence(struct lockdep_map *lock) +{ + struct task_struct *curr = current; + int i; + + if (unlikely(!debug_locks)) + return ~0; + + for (i = 0; i < curr->lockdep_depth; i++) { + struct held_lock *hlock = curr->held_locks + i; + + if (match_held_lock(hlock, lock)) + return hlock->seq_count; + } + + return ~0; +} + /* * Check whether we follow the irq-flags state precisely: */ @@ -5866,7 +5888,8 @@ void lock_acquire(struct lockdep_map *lock, unsigned int subclass, lockdep_recursion_inc(); __lock_acquire(lock, subclass, trylock, read, check, - irqs_disabled_flags(flags), nest_lock, ip, 0, 0, 0); + irqs_disabled_flags(flags), nest_lock, ip, 0, 0, 0, + ++current->lockdep_seq); lockdep_recursion_finish(); raw_local_irq_restore(flags); } @@ -5914,7 +5937,8 @@ void lock_sync(struct lockdep_map *lock, unsigned subclass, int read, lockdep_recursion_inc(); __lock_acquire(lock, subclass, 0, read, check, - irqs_disabled_flags(flags), nest_lock, ip, 0, 0, 1); + irqs_disabled_flags(flags), nest_lock, ip, 0, 0, 1, + ++current->lockdep_seq); check_chain_key(current); lockdep_recursion_finish(); raw_local_irq_restore(flags); @@ -6000,6 +6024,26 @@ void lock_unpin_lock(struct lockdep_map *lock, struct pin_cookie cookie) } EXPORT_SYMBOL_GPL(lock_unpin_lock); +u32 lock_sequence(struct lockdep_map *lock) +{ + unsigned long flags; + u32 seq = ~0; + + if (unlikely(!lockdep_enabled())) + return seq; + + raw_local_irq_save(flags); + check_flags(flags); + + lockdep_recursion_inc(); + seq = __lock_sequence(lock); + lockdep_recursion_finish(); + raw_local_irq_restore(flags); + + return seq; +} +EXPORT_SYMBOL_GPL(lock_sequence); + #ifdef CONFIG_LOCK_STAT static void print_lock_contention_bug(struct task_struct *curr, struct lockdep_map *lock, -- cgit v1.2.3 From 19e269917dc416f932474686bf1fcf3e91a740bd Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:11 -0400 Subject: maple_tree: add write lock checking with lockdep sequence numbers Use the lockdep sequence numbers to ensure the write lock is not dropped between write operations. The lockdep sequence is recorded on any walk that starts from the top of the tree and re-checked prior to any operation using an active node. When lockdep detects an issue, it sets debug_locks to 0 disabling further reports. __lock_sequnece() will return u32 ~0 when debug_locks is zero, and the real sequnece count cannot return such a high value as it is less than 32bits. By always updating the sequence number, regardless of lock state and by ignoring ~0 value in the sequence number will avoid ever printing a WARN_ON when lockdep sets debug_locks to 0. Link: https://lore.kernel.org/20260821192627.4085470-4-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Breno Leitao Tested-by: Breno Leitao Cc: Boqun Feng Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- include/linux/maple_tree.h | 7 +++-- lib/maple_tree.c | 64 +++++++++++++++++++++++++++++++++++++--------- 2 files changed, 57 insertions(+), 14 deletions(-) diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h index 1acf932fcd33..d63ac92208d0 100644 --- a/include/linux/maple_tree.h +++ b/include/linux/maple_tree.h @@ -484,9 +484,12 @@ struct ma_state { unsigned char mas_flags; unsigned char end; /* The end of the node */ enum store_type store_type; /* The type of store needed for this operation */ -#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) +#ifdef CONFIG_LOCKDEP + u32 ld_seq; +#ifdef CONFIG_RCU_STRICT_GRACE_PERIOD unsigned long rcu_gp; -#endif +#endif /* CONFIG_RCU_STRICT_GRACE_PERIOD */ +#endif /* CONFIG_LOCKDEP */ }; struct ma_wr_state { diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 6d805521bedd..d8c826e1ca0c 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1148,40 +1148,77 @@ static inline void mas_free(struct ma_state *mas, struct maple_enode *used) ma_free_rcu(mte_to_node(used)); } -void mas_lock_check(struct ma_state *mas) + +#ifdef CONFIG_LOCKDEP +static struct lockdep_map *mas_lockdep_map(struct ma_state *mas) { + struct maple_tree *mt = mas->tree; + + if (mt_external_lock(mt)) + return mt->ma_external_lock; + + return &(mt->ma_lock).dep_map; +} + +#endif + +static void mas_lock_check(struct ma_state *mas) +{ +#ifdef CONFIG_LOCKDEP + struct lockdep_map *map; + u32 seq; -#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) if (!mas_is_active(mas)) return; +#ifdef CONFIG_RCU_STRICT_GRACE_PERIOD if (!mt_locked(mas->tree)) { if (mt_in_rcu(mas->tree)) WARN_ON_ONCE(poll_state_synchronize_rcu(mas->rcu_gp)); } -#endif +#endif /* CONFIG_RCU_STRICT_GRACE_PERIOD */ + + map = mas_lockdep_map(mas); + if (!map) + return; + + seq = lock_sequence(map); + if (seq != UINT_MAX && mas->ld_seq != UINT_MAX) + WARN_ON_ONCE(mas->ld_seq != seq); +#endif /* CONFIG_LOCKDEP */ } -void mas_init_lock_check(struct ma_state *mas) +static void mas_init_lock_check(struct ma_state *mas) { -#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) +#ifdef CONFIG_LOCKDEP + struct lockdep_map *map; +#ifdef CONFIG_RCU_STRICT_GRACE_PERIOD if (!mt_locked(mas->tree)) { if (mt_in_rcu(mas->tree)) mas->rcu_gp = get_state_synchronize_rcu(); + return; } -#endif +#endif /* CONFIG_RCU_STRICT_GRACE_PERIOD */ + + map = mas_lockdep_map(mas); + if (map) /* Update regardless of lock state */ + mas->ld_seq = lock_sequence(map); +#endif /* CONFIG_LOCKDEP */ } static void mas_may_init_lock_check(struct ma_state *mas) { -#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) - if (mas_is_start(mas) || mas_is_paused(mas)) +#ifdef CONFIG_LOCKDEP +#ifdef CONFIG_RCU_STRICT_GRACE_PERIOD + if (mas_is_start(mas) || mas_is_paused(mas)) { mas_init_lock_check(mas); - else - mas_lock_check(mas); -#endif + return; + } +#endif /* CONFIG_RCU_STRICT_GRACE_PERIOD */ + mas_lock_check(mas); +#endif /* CONFIG_LOCKDEP */ } /* @@ -4864,6 +4901,7 @@ void *mas_store(struct ma_state *mas, void *entry) { MA_WR_STATE(wr_mas, mas, entry); + mas_may_init_lock_check(mas); trace_ma_write(TP_FCT, mas, 0, entry); #ifdef CONFIG_DEBUG_MAPLE_TREE if (MAS_WARN_ON(mas, mas->index > mas->last)) @@ -4922,6 +4960,7 @@ int mas_store_gfp(struct ma_state *mas, void *entry, gfp_t gfp) MA_WR_STATE(wr_mas, mas, entry); int ret = 0; + mas_may_init_lock_check(mas); retry: mas_wr_preallocate(&wr_mas, entry); if (unlikely(mas_nomem(mas, gfp))) { @@ -4952,6 +4991,7 @@ void mas_store_prealloc(struct ma_state *mas, void *entry) { MA_WR_STATE(wr_mas, mas, entry); + mas_lock_check(mas); if (mas->store_type == wr_store_root) { mas_wr_prealloc_setup(&wr_mas); goto store; @@ -4984,6 +5024,7 @@ int mas_preallocate(struct ma_state *mas, void *entry, gfp_t gfp) { MA_WR_STATE(wr_mas, mas, entry); + mas_may_init_lock_check(mas); mas_wr_prealloc_setup(&wr_mas); mas->store_type = mas_wr_store_type(&wr_mas); mas_prealloc_calc(&wr_mas, entry); @@ -5469,7 +5510,6 @@ EXPORT_SYMBOL_GPL(mas_find_range); static bool mas_find_rev_setup(struct ma_state *mas, unsigned long min, void **entry) { - switch (mas->status) { case ma_active: goto active; -- cgit v1.2.3 From 5e0b9b71bcf405a0390ea9efc853bd07186c65a0 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:12 -0400 Subject: maple_tree: documentation fix Don't include the word flag in the quotes with the actual flag. Link: https://lore.kernel.org/20260821192627.4085470-5-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- Documentation/core-api/maple_tree.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Documentation/core-api/maple_tree.rst b/Documentation/core-api/maple_tree.rst index ccdd1615cf97..34964ec88d17 100644 --- a/Documentation/core-api/maple_tree.rst +++ b/Documentation/core-api/maple_tree.rst @@ -211,7 +211,7 @@ Advanced Locking The maple tree uses a spinlock by default, but external locks can be used for tree updates as well. To use an external lock, the tree must be initialized -with the ``MT_FLAGS_LOCK_EXTERN flag``, this is usually done with the +with the ``MT_FLAGS_LOCK_EXTERN`` flag, this is usually done with the MTREE_INIT_EXT() #define, which takes an external lock as an argument. Functions and structures -- cgit v1.2.3 From acac9108a6a1a897f66060cafd98c452af1442c1 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:13 -0400 Subject: maple_tree: drop dead code from mas_extend_spanning_null() mas_extend_spanning_null() had a clause if the end of the range being written (mas->last) is the same as the end of the existing range it is overwriting (wr_mas->r_max), action will be taken. This code path is not possible because the only calling function increments mas->last (unless it's ULONG_MAX) to walk to one beyond the write and then resets the value back to the initial value. In the case of mas->last == ULONG_MAX, then the second part of the statement will always be false - mas->last cannot be less than the node max. This code never executed and is flawed anyways (the arguments are incorrectly ordered), so removing it is the safest action. Since the code never executes, it is not fixing any issue so Fixes tag is not given. Link: https://lore.kernel.org/20260821192627.4085470-6-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 7 ------- 1 file changed, 7 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index d8c826e1ca0c..3509e293c84f 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -2997,13 +2997,6 @@ static inline void mas_extend_spanning_null(struct ma_wr_state *l_wr_mas, if (r_mas->last < r_wr_mas->r_max) r_mas->last = r_wr_mas->r_max; r_mas->offset++; - } else if ((r_mas->last == r_wr_mas->r_max) && - (r_mas->last < r_mas->max) && - !mas_slot_locked(r_mas, r_wr_mas->slots, r_mas->offset + 1)) { - r_mas->last = mas_safe_pivot(r_mas, r_wr_mas->pivots, - r_wr_mas->type, r_mas->offset + 1); - r_mas->offset++; - r_wr_mas->r_max = r_mas->last; } } -- cgit v1.2.3 From 3526e09d8cab0aea3f3737dbeb613ccb52660359 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:14 -0400 Subject: maple_tree: drop MAPLE_ALLOC_SLOTS MAPLE_ALLOC_SLOTS is no longer used, so remove it. Link: https://lore.kernel.org/20260821192627.4085470-7-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- include/linux/maple_tree.h | 2 -- 1 file changed, 2 deletions(-) diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h index d63ac92208d0..14ca9ac775d9 100644 --- a/include/linux/maple_tree.h +++ b/include/linux/maple_tree.h @@ -28,13 +28,11 @@ #define MAPLE_NODE_SLOTS 31 /* 256 bytes including ->parent */ #define MAPLE_RANGE64_SLOTS 16 /* 256 bytes */ #define MAPLE_ARANGE64_SLOTS 10 /* 240 bytes */ -#define MAPLE_ALLOC_SLOTS (MAPLE_NODE_SLOTS - 1) #else /* 32bit sizes */ #define MAPLE_NODE_SLOTS 63 /* 256 bytes including ->parent */ #define MAPLE_RANGE64_SLOTS 32 /* 256 bytes */ #define MAPLE_ARANGE64_SLOTS 21 /* 240 bytes */ -#define MAPLE_ALLOC_SLOTS (MAPLE_NODE_SLOTS - 2) #endif /* defined(CONFIG_64BIT) || defined(BUILD_VDSO32_64) */ #define MAPLE_NODE_MASK 255UL -- cgit v1.2.3 From 7d1e34352727cf1073eccbacd13fac075defc7a4 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:15 -0400 Subject: maple_tree: clarify comments on mas_nomem() When an allocation completely fails, the return is false. If the allocation succeeds or partially succeeds, return true to indicate a retry of the operation. Note that since the lock may have been dropped, the operation is retried from the start - including potentially allocating more memory. Link: https://lore.kernel.org/20260821192627.4085470-8-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 3509e293c84f..baaaa128594c 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -5677,10 +5677,11 @@ EXPORT_SYMBOL_GPL(mas_erase); /** * mas_nomem() - Check if there was an error allocating and do the allocation - * if necessary If there are allocations, then free them. + * if necessary. + * * @mas: The maple state * @gfp: The GFP_FLAGS to use for allocations - * Return: true on allocation, false otherwise. + * Return: False on no memory. True otherwise (partial success as well) */ bool mas_nomem(struct ma_state *mas, gfp_t gfp) __must_hold(mas->tree->ma_lock) @@ -5696,6 +5697,10 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp) mas_alloc_nodes(mas, gfp); } + /* + * Return false on zero forward progress. Partial allocations are kept + * so the retry path will attempt to get the rest. + */ if (!mas->sheaf && !mas->alloc) return false; -- cgit v1.2.3 From 4bd59d5bc2c8f3e7e1bd62f3a5029471a6b531bb Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:16 -0400 Subject: maple_tree: use prefetched value in mas_wr_store_type() The slot contents exist in wr_mas->content, which has less overhead than reading the slot again. Link: https://lore.kernel.org/20260821192627.4085470-9-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index baaaa128594c..d5fa85bc7aab 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -3280,7 +3280,7 @@ static inline void mas_wr_slot_store(struct ma_wr_state *wr_mas) void __rcu **slots = wr_mas->slots; bool gap = false; - gap |= !mt_slot_locked(mas->tree, slots, offset); + gap |= !wr_mas->content; gap |= !mt_slot_locked(mas->tree, slots, offset + 1); if (wr_mas->offset_end - offset == 1) { -- cgit v1.2.3 From 88f87f881240da3f09541d8232255376778b8f1c Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:17 -0400 Subject: maple_tree: optimise mas_wr_node_store() when not in rcu mode Clearing the entire node on the stack is unnecessary since most of the node will be overwritten anyways. Just clear what isn't used after the data is in place. Benchmarking shows a speedup of 0.67% on a height 4 tree with 2048 entries. Link: https://lore.kernel.org/20260821192627.4085470-10-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index d5fa85bc7aab..56812db8b0ef 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -3187,7 +3187,7 @@ static void mas_wr_spanning_store(struct ma_wr_state *wr_mas) static inline void mas_wr_node_store(struct ma_wr_state *wr_mas) { unsigned char dst_offset, offset_end; - unsigned char copy_size, node_pivots; + unsigned char copy_size, node_pivots, node_slots; struct maple_node reuse, *newnode; unsigned long *dst_pivots; void __rcu **dst_slots; @@ -3200,6 +3200,7 @@ static inline void mas_wr_node_store(struct ma_wr_state *wr_mas) in_rcu = mt_in_rcu(mas->tree); offset_end = wr_mas->offset_end; node_pivots = mt_pivots[wr_mas->type]; + node_slots = mt_slots[wr_mas->type]; /* Assume last adds an entry */ new_end = mas->end + 1 - offset_end + mas->offset; if (mas->last == wr_mas->end_piv) { @@ -3211,7 +3212,6 @@ static inline void mas_wr_node_store(struct ma_wr_state *wr_mas) if (in_rcu) { newnode = mas_pop_node(mas); } else { - memset(&reuse, 0, sizeof(struct maple_node)); newnode = &reuse; } @@ -3255,7 +3255,21 @@ static inline void mas_wr_node_store(struct ma_wr_state *wr_mas) dst_pivots[new_end] = mas->max; done: - mas_leaf_set_meta(newnode, maple_leaf_64, new_end); + if (!in_rcu && new_end + 2 < node_slots) { + unsigned char clear_from = new_end + 1; + + /* + * Note that the last slot is never cleared, since the metadata + * will be stored there or it has a value. + */ + memset(dst_slots + clear_from, 0, + sizeof(void __rcu *) * (node_slots - clear_from)); + if (clear_from < node_pivots) + memset(dst_pivots + clear_from, 0, + sizeof(unsigned long) * (node_pivots - clear_from)); + } + + mas_leaf_set_meta(newnode, wr_mas->type, new_end); if (in_rcu) { struct maple_enode *old_enode = mas->node; -- cgit v1.2.3 From f0a3892cd726909b0e9c30ec9b5b05f1f63a5fbf Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:18 -0400 Subject: maple_tree: micro optimisation of mas_wr_store_type() Use three new local booleans instead of reading other structures. This has shown an increase of 0.62% on a 2048 entry tree of height 4. Link: https://lore.kernel.org/20260821192627.4085470-11-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 56812db8b0ef..e2c780a64c9c 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -3737,6 +3737,9 @@ static inline enum store_type mas_wr_store_type(struct ma_wr_state *wr_mas) { struct ma_state *mas = wr_mas->mas; unsigned char new_end; + bool appending; + bool one_slot; + bool in_rcu; if (unlikely(mas_is_none(mas) || mas_is_ptr(mas))) return wr_store_root; @@ -3756,21 +3759,30 @@ static inline enum store_type mas_wr_store_type(struct ma_wr_state *wr_mas) return wr_new_root; new_end = mas_wr_new_end(wr_mas); + in_rcu = mt_in_rcu(mas->tree); + appending = mas->offset == mas->end; + one_slot = wr_mas->offset_end - mas->offset == 1; + /* Potential spanning rebalance collapsing a node */ if (new_end < mt_min_slots[wr_mas->type]) { if (!mte_is_root(mas->node)) return wr_rebalance; + if (!in_rcu) { + if (appending) + return wr_append; + else if (mas->end == new_end && one_slot) + return wr_slot_store; + } return wr_node_store; } if (new_end >= mt_slots[wr_mas->type]) return wr_split_store; - if (!mt_in_rcu(mas->tree) && (mas->offset == mas->end)) + if (!in_rcu && appending) return wr_append; - if ((new_end == mas->end) && (!mt_in_rcu(mas->tree) || - (wr_mas->offset_end - mas->offset == 1))) + if (new_end == mas->end && (!in_rcu || one_slot)) return wr_slot_store; return wr_node_store; -- cgit v1.2.3 From cf1f9bae5dac2082db374a0acfe41e55aaf909e2 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:19 -0400 Subject: maple_tree: add bulk parent set helper Instead of calculating the parent pointer each time for a child, cache the majority of the parent pointer and only change the slot per child. Drop the mas_set_parent() function since the last user has been removed. Testing on a tree containing 2048 entries of height 4 had an increased gain of 3.51% on nodes tracking gaps. Link: https://lore.kernel.org/20260821192627.4085470-12-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 97 ++++++++++++++++++++++++-------------------------------- 1 file changed, 42 insertions(+), 55 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index e2c780a64c9c..c968e25bea0a 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -450,46 +450,6 @@ enum maple_type mas_parent_type(struct ma_state *mas, struct maple_enode *enode) return 0; } -/* - * mas_set_parent() - Set the parent node and encode the slot - * @mas: The maple state - * @enode: The encoded maple node. - * @parent: The encoded maple node that is the parent of @enode. - * @slot: The slot that @enode resides in @parent. - * - * Slot number is encoded in the enode->parent bit 3-6 or 2-6, depending on the - * parent type. - */ -static inline -void mas_set_parent(struct ma_state *mas, struct maple_enode *enode, - const struct maple_enode *parent, unsigned char slot) -{ - unsigned long val = (unsigned long)parent; - unsigned long shift; - unsigned long type; - enum maple_type p_type = mte_node_type(parent); - - MAS_BUG_ON(mas, p_type == maple_dense); - MAS_BUG_ON(mas, p_type == maple_leaf_64); - - switch (p_type) { - case maple_range_64: - case maple_arange_64: - shift = MAPLE_PARENT_SLOT_SHIFT; - type = MAPLE_PARENT_RANGE64; - break; - default: - case maple_dense: - case maple_leaf_64: - shift = type = 0; - break; - } - - val &= ~MAPLE_NODE_MASK; /* Clear all node metadata in parent */ - val |= (slot << shift) | type; - mte_to_node(enode)->parent = ma_parent_ptr(val); -} - /* * mte_parent_slot() - get the parent slot of @enode. * @enode: The encoded maple node. @@ -871,6 +831,42 @@ static inline void ma_set_meta_gap(struct maple_node *mn, enum maple_type mt, meta->gap = offset; } +/* + * mas_set_parent_slots() - Bulk operation to set many slot parent pointers + * @mas: The maple state + * @parent: The encoded maple node that is the parent of @enode. + * @slot: The slot that of the @enode. + * @start_slot: The offset into @slot + * @count: The number of slots to set (eg: exclusive) + */ +static inline +void mas_set_parent_slots(struct ma_state *mas, struct maple_enode *parent, + void __rcu **slots, unsigned char start_slot, unsigned char count) +{ + unsigned long val; + unsigned long shift; + unsigned long type; + enum maple_type p_type = mte_node_type(parent); + unsigned char i; + + MAS_BUG_ON(mas, p_type != maple_range_64 && + p_type != maple_arange_64); + + shift = MAPLE_PARENT_SLOT_SHIFT; + type = MAPLE_PARENT_RANGE64; + + val = (unsigned long)parent; + val &= ~MAPLE_NODE_MASK; + + for (i = 0; i < count; i++) { + unsigned long pval = val | ((start_slot + i) << shift) | type; + struct maple_enode *child; + + child = mt_slot_locked(mas->tree, slots, i); + mte_to_node(child)->parent = ma_parent_ptr(pval); + } +} + /* * mat_add() - Add a @dead_enode to the ma_topiary of a list of dead nodes. * @mat: the ma_topiary, a linked list of dead nodes. @@ -1609,14 +1605,10 @@ static inline void mas_adopt_children(struct ma_state *mas, struct maple_node *node = mte_to_node(parent); void __rcu **slots = ma_slots(node, type); unsigned long *pivots = ma_pivots(node, type); - struct maple_enode *child; - unsigned char offset; + unsigned char end; - offset = ma_data_end(node, type, pivots, mas->max); - do { - child = mas_slot_locked(mas, slots, offset); - mas_set_parent(mas, child, parent, offset); - } while (offset--); + end = ma_data_end(node, type, pivots, mas->max); + mas_set_parent_slots(mas, parent, slots, 0, end + 1); } /* @@ -1998,15 +1990,10 @@ unsigned long node_copy(struct ma_state *mas, struct maple_node *src, s_slots = ma_slots(src, s_mt) + start; s_pivots = ma_pivots(src, s_mt) + start; memcpy(d_slots, s_slots, size * sizeof(void __rcu *)); - if (!ma_is_leaf(d_mt) && s_mt == maple_copy) { - struct maple_enode *edst = mt_mk_node(dst, d_mt); - - for (int i = 0; i < size; i++) - mas_set_parent(mas, - mt_slot_locked(mas->tree, d_slots, i), - edst, d_start + i); - } + if (!ma_is_leaf(d_mt) && s_mt == maple_copy) + mas_set_parent_slots(mas, mt_mk_node(dst, d_mt), + d_slots, d_start, size); d_gaps = ma_gaps(dst, d_mt); if (d_gaps) { -- cgit v1.2.3 From 35f1342e5b893a740eff2ef9ab337bfaa63ab76d Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:20 -0400 Subject: maple_tree: catch race in mas_alloc_cyclic() If mas_alloc_cyclic() is called during a low memory situation, it is possible the lock may be dropped so reclaim can occur. There is a window where some other task may allocate the same id and cause the mas_insert() to fail with -EEXIST. In this scenario the function will return -EEXIST, which is not expected. Modifying the retry on mas_nomem() to re-search for a slot means that any race with other writes will not matter as the lock will be held between finding the index and writing the index. Moving the flag logic avoids cases where the flag is modified on drop lock/reacquire or when the write fails after clearing the flag. No existing users are exposed to this issue. Link: https://lore.kernel.org/20260821192627.4085470-13-liam@infradead.org Fixes: 9b6713cc7522 ("maple_tree: Add mtree_alloc_cyclic()") Signed-off-by: Liam R. Howlett (Oracle) Reported-by: Chris Mason Reviewed-by: Chuck Lever Cc: Boqun Feng Cc: Breno Leitao Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 43 ++++++++++++++++++++++++------------------- 1 file changed, 24 insertions(+), 19 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index c968e25bea0a..190f480d6850 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -3868,35 +3868,40 @@ int mas_alloc_cyclic(struct ma_state *mas, unsigned long *startp, void *entry, unsigned long range_lo, unsigned long range_hi, unsigned long *next, gfp_t gfp) { - unsigned long min = range_lo; - int ret = 0; - - range_lo = max(min, *next); - ret = mas_empty_area(mas, range_lo, range_hi, 1); - if ((mas->tree->ma_flags & MT_FLAGS_ALLOC_WRAPPED) && ret == 0) { - mas->tree->ma_flags &= ~MT_FLAGS_ALLOC_WRAPPED; - ret = 1; - } - if (ret < 0 && range_lo > min) { - mas_reset(mas); - ret = mas_empty_area(mas, min, range_hi, 1); - if (ret == 0) - ret = 1; - } - if (ret < 0) - return ret; + int ret; + unsigned long min; + min = range_lo; do { + range_lo = max(min, *next); + ret = mas_empty_area(mas, range_lo, range_hi, 1); + if (ret < 0 && range_lo > min) { + mas_reset(mas); + ret = mas_empty_area(mas, min, range_hi, 1); + if (ret == 0) + ret = 1; + } + if (ret < 0) + goto out; + mas_insert(mas, entry); } while (mas_nomem(mas, gfp)); - if (mas_is_err(mas)) - return xa_err(mas->node); + if (mas_is_err(mas)) { + ret = xa_err(mas->node); + goto out; + } + + if ((mas->tree->ma_flags & MT_FLAGS_ALLOC_WRAPPED) && ret == 0) { + mas->tree->ma_flags &= ~MT_FLAGS_ALLOC_WRAPPED; + ret = 1; + } *startp = mas->index; *next = *startp + 1; if (*next == 0) mas->tree->ma_flags |= MT_FLAGS_ALLOC_WRAPPED; +out: mas_destroy(mas); return ret; } -- cgit v1.2.3 From 3f06ef1f34a7ee3b77ee03c13a0c50db8fd3c536 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:21 -0400 Subject: maple_tree: document that erase may use GFP_KERNEL for allocations State that the mas_erase() and mtree_erase() functions may use GFP_KERNEL on allocation retry. Don't just depend on people reading the documentation by adding a check that will warn of the use. Link: https://lore.kernel.org/20260821192627.4085470-14-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Reviewed-by: Rik van Riel Cc: Jason Gunthorpe Cc: Boqun Feng Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Joe Perches Cc: Peter Zijlstra Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 190f480d6850..440863bdea26 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -5658,6 +5658,10 @@ EXPORT_SYMBOL_GPL(mas_find_range_rev); * Searches for @mas->index, sets @mas->index and @mas->last to the range and * erases that range. * + * Note that erase requires allocations and will use GFP_KERNEL to do so if + * necessary. If the allocation fails, the internal lock will be dropped to + * retry. + * * Return: the entry that was erased or %NULL, @mas->index and @mas->last are updated. */ void *mas_erase(struct ma_state *mas) @@ -5666,13 +5670,21 @@ void *mas_erase(struct ma_state *mas) unsigned long index = mas->index; MA_WR_STATE(wr_mas, mas, NULL); + /* + * In low memory situations, the allocation is retried with the gfp flag + * GFP_KERNEL. The internal spinlock is dropped in mas_nomem(), however + * the external lock is not dropped. + */ + if (mt_external_lock(mas->tree)) + might_alloc(GFP_KERNEL); + if (!mas_is_active(mas) || !mas_is_start(mas)) mas->status = ma_start; write_retry: entry = mas_state_walk(mas); if (!entry) - return NULL; + goto out; /* Must reset to ensure spanning writes of last slot are detected */ mas_reset(mas); @@ -5683,8 +5695,10 @@ write_retry: goto write_retry; } - if (mas_is_err(mas)) + if (mas_is_err(mas)) { + entry = NULL; goto out; + } mas_wr_store_entry(&wr_mas); out: @@ -6012,6 +6026,10 @@ EXPORT_SYMBOL(mtree_alloc_rrange); * Erasing is the same as a walk to an entry then a store of a NULL to that * ENTIRE range. In fact, it is implemented as such using the advanced API. * + * Note that erase requires allocations and will use GFP_KERNEL to do so if + * necessary. If the allocation fails, the internal lock will be dropped to + * retry. + * * Return: The entry stored at the @index or %NULL */ void *mtree_erase(struct maple_tree *mt, unsigned long index) @@ -6021,6 +6039,7 @@ void *mtree_erase(struct maple_tree *mt, unsigned long index) MA_STATE(mas, mt, index, index); trace_ma_op(TP_FCT, &mas); + might_alloc(GFP_KERNEL); mtree_lock(mt); entry = mas_erase(&mas); mtree_unlock(mt); -- cgit v1.2.3 From f1681380b5f928e147954d87d875f57a25df189c Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:22 -0400 Subject: maple_tree: avoid mas_erase() and mtree_erase() failures Failures to remove entries using the two APIs to erase the entries may result in allocation failures. The failures may go unnoticed and an unexpected entry may remain. Instead, fall back to retrying with GFP_KERNEL | __GFP_NOFAIL so that the entry will be removed. Link: https://lore.kernel.org/20260821192627.4085470-15-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Rik van Riel Cc: Jason Gunthorpe Cc: Boqun Feng Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Joe Perches Cc: Peter Zijlstra Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- include/linux/maple_tree.h | 2 ++ lib/maple_tree.c | 68 ++++++++++++++++++++++++++++++++++------------ 2 files changed, 52 insertions(+), 18 deletions(-) diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h index 14ca9ac775d9..173602e87c14 100644 --- a/include/linux/maple_tree.h +++ b/include/linux/maple_tree.h @@ -570,6 +570,8 @@ int mas_alloc_cyclic(struct ma_state *mas, unsigned long *startp, unsigned long *next, gfp_t gfp); bool mas_nomem(struct ma_state *mas, gfp_t gfp); +bool mas_nomem_nofail(struct ma_state *mas, unsigned long index, + unsigned long last); void mas_pause(struct ma_state *mas); void maple_tree_init(void); void mas_destroy(struct ma_state *mas); diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 440863bdea26..d47d4304f781 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -5658,9 +5658,10 @@ EXPORT_SYMBOL_GPL(mas_find_range_rev); * Searches for @mas->index, sets @mas->index and @mas->last to the range and * erases that range. * - * Note that erase requires allocations and will use GFP_KERNEL to do so if - * necessary. If the allocation fails, the internal lock will be dropped to - * retry. + * Note that erase requires allocations and will use GFP_KERNEL | __GFP_NOFAIL + * to do so if necessary. If the allocation fails, the internal lock will be + * dropped to retry. An externally locked tree must be protected by a lock that + * allows blocking for this API. * * Return: the entry that was erased or %NULL, @mas->index and @mas->last are updated. */ @@ -5672,8 +5673,8 @@ void *mas_erase(struct ma_state *mas) /* * In low memory situations, the allocation is retried with the gfp flag - * GFP_KERNEL. The internal spinlock is dropped in mas_nomem(), however - * the external lock is not dropped. + * GFP_KERNEL | __GFP_NOFAIL. The internal spinlock is dropped in + * mas_nomem_nofail(), however the external lock is not dropped. */ if (mt_external_lock(mas->tree)) might_alloc(GFP_KERNEL); @@ -5689,16 +5690,8 @@ write_retry: /* Must reset to ensure spanning writes of last slot are detected */ mas_reset(mas); mas_wr_preallocate(&wr_mas, NULL); - if (mas_nomem(mas, GFP_KERNEL)) { - /* in case the range of entry changed when unlocked */ - mas->index = mas->last = index; + if (mas_nomem_nofail(mas, index, index)) goto write_retry; - } - - if (mas_is_err(mas)) { - entry = NULL; - goto out; - } mas_wr_store_entry(&wr_mas); out: @@ -5721,6 +5714,10 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp) if (likely(mas->node != MA_ERROR(-ENOMEM))) return false; + /* Allocations can fail, don't do this. */ + WARN_ON_ONCE(!gfpflags_allow_blocking(gfp) && + mt_external_lock(mas->tree)); + if (gfpflags_allow_blocking(gfp) && !mt_external_lock(mas->tree)) { mtree_unlock(mas->tree); mas_alloc_nodes(mas, gfp); @@ -5731,7 +5728,9 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp) /* * Return false on zero forward progress. Partial allocations are kept - * so the retry path will attempt to get the rest. + * so the retry path will attempt to get the rest. The failure should + * not happen as we try our best to reclaim. The user would need an + * external lock with a non-blocking gfp in a low memory situation. */ if (!mas->sheaf && !mas->alloc) return false; @@ -5740,6 +5739,39 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp) return true; } +/** + * mas_nomem_nofail() - Retry allocations with __GFP_NOFAIL, if the maple state + * has stored the -ENOMEM error. + * @mas: The maple state + * @index: The start of the range for the @mas reset + * @last: The end of the range for the @mas reset + * + * Return: false if @mas isn't in an -ENOMEM state. True if the allocation + * happens, the state is reset. The internal lock will be dropped and external + * locks must allow blocking. + */ +bool mas_nomem_nofail(struct ma_state *mas, unsigned long index, + unsigned long last) + __must_hold(mas->tree->ma_lock) +{ + gfp_t gfp; + + if (likely(mas->node != MA_ERROR(-ENOMEM))) + return false; + + gfp = GFP_KERNEL | __GFP_NOFAIL; + if (!mt_external_lock(mas->tree)) { + mtree_unlock(mas->tree); + mas_alloc_nodes(mas, gfp); + mtree_lock(mas->tree); + } else { + mas_alloc_nodes(mas, gfp); + } + + mas_set_range(mas, index, last); + return true; +} + void __init maple_tree_init(void) { struct kmem_cache_args args = { @@ -6026,9 +6058,9 @@ EXPORT_SYMBOL(mtree_alloc_rrange); * Erasing is the same as a walk to an entry then a store of a NULL to that * ENTIRE range. In fact, it is implemented as such using the advanced API. * - * Note that erase requires allocations and will use GFP_KERNEL to do so if - * necessary. If the allocation fails, the internal lock will be dropped to - * retry. + * Note that erase requires allocations and will use GFP_KERNEL | __GFP_NOFAIL + * to do so if necessary. If the allocation fails, the internal lock will be + * dropped to retry. * * Return: The entry stored at the @index or %NULL */ -- cgit v1.2.3 From ee2487d9ba6ccf1b10c55fbae2cb0526c7b775e3 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:23 -0400 Subject: maple_tree: document erase and allocations better During a discussion on the maple tree erase process and GFP flags, Jason suggested there be an amendment to the documentation to clarify the situation on allocations within the tree. The added text is an attempt to better explain that the tree may allocate, even when erasing, and provide some guidance on how to work around such issues. [akpm@linux-foundation.org: tweak mtree_erase() description, per Jason] Link: https://lore.kernel.org/all/20260617180419.GA231643@ziepe.ca/ Link: https://lore.kernel.org/20260821192627.4085470-16-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Suggested-by: Jason Gunthorpe Cc: Rik van Riel Cc: Boqun Feng Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Joe Perches Cc: Peter Zijlstra Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- Documentation/core-api/maple_tree.rst | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/Documentation/core-api/maple_tree.rst b/Documentation/core-api/maple_tree.rst index 34964ec88d17..12bccfb6aac1 100644 --- a/Documentation/core-api/maple_tree.rst +++ b/Documentation/core-api/maple_tree.rst @@ -17,7 +17,8 @@ supports iterating over a range of entries and going to the previous or next entry in a cache-efficient manner. The tree can also be put into an RCU-safe mode of operation which allows reading and writing concurrently. Writers must synchronize on a lock, which can be the default spinlock, or the user can set -the lock to an external lock of a different type. +the lock to an external lock of a different type. Note that external locks may +interfere with allocations in a low memory situation. The Maple Tree maintains a small memory footprint and was designed to use modern processor cache efficiently. The majority of the users will be able to @@ -42,6 +43,15 @@ successful store operation within a given code segment when allocating cannot be done. Allocations of nodes are relatively small at around 256 bytes. +Since the maple tree uses internal nodes that are allocated and has rules on +data density, erasing an entry may cause allocations to occur. That is, +erasing an entry may consume memory. Users must take care to ensure that they +do not violate the larger system constraints on when and how memory is +allocated. Most situations are fine to allocate, but the pre-allocation +support is provided as a mechanism to avoid trickier situations. There is also +the possibility of using special entries and clean up the tree later, in +extreme circumstances. + .. _maple-tree-normal-api: Normal API @@ -63,7 +73,10 @@ success or an error code otherwise. mtree_store_range() works in the same way but takes a range. mtree_load() is used to retrieve the entry stored at a given index. You can use mtree_erase() to erase an entire range by only knowing one value within that range, or mtree_store() call with an entry of -NULL may be used to partially erase a range or many ranges at once. +NULL may be used to partially erase a range or many ranges at once. Note that +mtree_erase() may use GFP_KERNEL | __GFP_NOFAIL for allocations and cannot +fail. mtree_erase() can sleep, so it must not be called from an atomic +context. If you want to only store a new entry to a range (or index) if that range is currently ``NULL``, you can use mtree_insert_range() or mtree_insert() which @@ -163,7 +176,10 @@ You can use mas_erase() to erase an entire range by setting index and last of the maple state to the desired range to erase. This will erase the first range that is found in that range, set the maple state index and last as the range that was erased and return the entry that existed -at that location. +at that location. Note that mas_erase() may allocate with the GFP_KERNEL +__GFP_NOFAIL and cannot fail, but may sleep. If this is not okay, consider +using mas_store_gfp() and pass it a ``NULL``, +after setting up the correct range by walking to the entry. You can walk each entry within a range by using mas_for_each(). If you want to walk each element of the tree then ``0`` and ``ULONG_MAX`` may be used as -- cgit v1.2.3 From 18d4f8e6e6ce9b2ebd1c733777babec67356acd3 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:24 -0400 Subject: maple_tree: change two GFP flags in tests The GFP flags in two tests are obviously incorrect. Make the tests correctly run by updating the GFP flags. Link: https://lore.kernel.org/all/d9cbb89faa5bdb71d451781d214a51ce8923a83e.camel@perches.com/ Link: https://lore.kernel.org/20260821192627.4085470-17-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Reported-by: Joe Perches Cc: Boqun Feng Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- tools/testing/radix-tree/maple.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c index 0607913a3022..d967e76a3c06 100644 --- a/tools/testing/radix-tree/maple.c +++ b/tools/testing/radix-tree/maple.c @@ -35234,7 +35234,7 @@ static noinline void __init check_prealloc(struct maple_tree *mt) mt_set_non_kernel(1); /* Spanning store */ mas_set_range(&mas, 1, 100); - MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL & GFP_NOWAIT) == 0); + MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_NOWAIT) == 0); allocated = mas_allocated(&mas); height = mas_mt_height(&mas); MT_BUG_ON(mt, allocated != 0); @@ -35257,7 +35257,7 @@ static noinline void __init check_prealloc(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != 0); mas_set_range(&mas, 0, 200); mt_set_non_kernel(1); - MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL & GFP_NOWAIT) == 0); + MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_NOWAIT) == 0); allocated = mas_allocated(&mas); height = mas_mt_height(&mas); MT_BUG_ON(mt, allocated != 0); -- cgit v1.2.3 From 00f67814a14e614b749ebe54076ef1e3e6454f2b Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:25 -0400 Subject: maple_tree: fix argument name in header The mas_prev_range() function takes a min and not a max. Link: https://lore.kernel.org/20260821192627.4085470-18-liam@infradead.org Fixes: 6b9e93e01020 ("maple_tree: add mas_prev_range() and mas_find_range_rev interface") Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- include/linux/maple_tree.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h index 173602e87c14..e595ae5cd0ee 100644 --- a/include/linux/maple_tree.h +++ b/include/linux/maple_tree.h @@ -577,7 +577,7 @@ void maple_tree_init(void); void mas_destroy(struct ma_state *mas); void *mas_prev(struct ma_state *mas, unsigned long min); -void *mas_prev_range(struct ma_state *mas, unsigned long max); +void *mas_prev_range(struct ma_state *mas, unsigned long min); void *mas_next(struct ma_state *mas, unsigned long max); void *mas_next_range(struct ma_state *mas, unsigned long max); -- cgit v1.2.3 From 4e4fe9d7271edf83ee4475e92b96aa89bd2ccd7e Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:26 -0400 Subject: maple_tree: avoid extra gap calculation Prior to ending the ascension loop of larger operations like split, rebalance, and spanning store the gap in the node had been calculated. Once the node is inserted into the tree, the gap is recalculated in mas_update_gap(). This can be avoided by creating a helper for mas_update_gap() that accepts the known gap value, which reduces the operations required for gap updating path. Link: https://lore.kernel.org/20260821192627.4085470-19-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index d47d4304f781..b43e2ce129b3 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1566,14 +1566,26 @@ ascend: goto ascend; } +static __always_inline void mas_update_gap_known(struct ma_state *mas, + unsigned long gap) +{ + unsigned char pslot; + unsigned long p_gap; + + pslot = mte_parent_slot(mas->node); + p_gap = ma_gaps(mte_parent(mas->node), + mas_parent_type(mas, mas->node))[pslot]; + + if (p_gap != gap) + mas_parent_gap(mas, pslot, gap); +} + /* * mas_update_gap() - Update a nodes gaps and propagate up if necessary. * @mas: the maple state. */ static inline void mas_update_gap(struct ma_state *mas) { - unsigned char pslot; - unsigned long p_gap; unsigned long max_gap; if (!mt_is_alloc(mas->tree)) @@ -1583,13 +1595,7 @@ static inline void mas_update_gap(struct ma_state *mas) return; max_gap = mas_max_gap(mas); - - pslot = mte_parent_slot(mas->node); - p_gap = ma_gaps(mte_parent(mas->node), - mas_parent_type(mas, mas->node))[pslot]; - - if (p_gap != max_gap) - mas_parent_gap(mas, pslot, max_gap); + mas_update_gap_known(mas, max_gap); } /* @@ -2137,8 +2143,8 @@ static inline void mas_wmb_replace(struct ma_state *mas, struct maple_copy *cp) mas->node = mt_slot_locked(mas->tree, cp->slot, 0); /* Insert the new data in the tree */ mas_topiary_replace(mas, old_enode, cp->height); - if (!mte_is_leaf(mas->node)) - mas_update_gap(mas); + if (mt_is_alloc(mas->tree) && !mte_is_root(mas->node)) + mas_update_gap_known(mas, cp->gap[0]); mtree_range_walk(mas); } -- cgit v1.2.3 From d17c749d32b2d6d16981ac003ff1cae85860e819 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:27 -0400 Subject: maple_tree: add helper mas_make_walkable() A check in mas_walk() was incorrect and caused inefficient use of the maple state. The same issue existed in mas_erase(), but was left unfixed. Making a helper function is the obvious answer. Link: https://lore.kernel.org/20260821192627.4085470-20-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- lib/maple_tree.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index b43e2ce129b3..1aba6cced713 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -261,6 +261,12 @@ static inline bool mas_is_underflow(struct ma_state *mas) return mas->status == ma_underflow; } +static inline void mas_make_walkable(struct ma_state *mas) +{ + if (!mas_is_active(mas) && !mas_is_start(mas)) + mas->status = ma_start; +} + static __always_inline struct maple_node *mte_to_node( const struct maple_enode *entry) { @@ -4447,8 +4453,7 @@ void *mas_walk(struct ma_state *mas) void *entry; mas_may_init_lock_check(mas); - if (!mas_is_active(mas) && !mas_is_start(mas)) - mas->status = ma_start; + mas_make_walkable(mas); retry: entry = mas_state_walk(mas); if (mas_is_start(mas)) { @@ -5685,9 +5690,7 @@ void *mas_erase(struct ma_state *mas) if (mt_external_lock(mas->tree)) might_alloc(GFP_KERNEL); - if (!mas_is_active(mas) || !mas_is_start(mas)) - mas->status = ma_start; - + mas_make_walkable(mas); write_retry: entry = mas_state_walk(mas); if (!entry) -- cgit v1.2.3 From 5d3fe91b70e7b71174d2a29eb9731a5601e7df0c Mon Sep 17 00:00:00 2001 From: Enlin Mu Date: Fri, 21 Aug 2026 14:40:57 +0800 Subject: mm/vmscan: fix comment logic in balance_pgdat In balance_pgdat(), when the low watermark is met, processes sleeping on pfmemalloc_wait are woken up because they are able to safely make forward progress. However, the comment incorrectly states "they should not be able", which contradicts the actual code behavior. Fix this typo to accurately reflect the logic. Link: https://lore.kernel.org/20260821064057.4081-1-enlin.mu@linux.dev Signed-off-by: Enlin Mu Signed-off-by: Enlin Mu Reviewed-by: Barry Song Acked-by: Johannes Weiner Acked-by: Shakeel Butt Cc: Axel Rasmussen Cc: David Hildenbrand Cc: Kairui Song Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index c1404a59523d..b569eeca590d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -7276,7 +7276,7 @@ restart: /* * If the low watermark is met there is no need for processes - * to be throttled on pfmemalloc_wait as they should not be + * to be throttled on pfmemalloc_wait as they should now be * able to safely make forward progress. Wake them */ if (waitqueue_active(&pgdat->pfmemalloc_wait) && -- cgit v1.2.3 From 0e0ac326c511d514817cc7b6d7741afd59098ce2 Mon Sep 17 00:00:00 2001 From: Shakeel Butt Date: Fri, 21 Aug 2026 19:47:07 -0700 Subject: memcg: move LRU size accounting on reparenting instead of copying it When a memory cgroup is offlined its LRU folios are reparented to the parent. lruvec_reparent_lru() splices the child's lists into the parent's and credits the parent with the child's per-zone lru_zone_size[], but never clears the child's copy, so the size is copied rather than moved. lru_gen_reparent_memcg() does the same for MGLRU. The parent is left correct, credited with exactly the folios it took over. The stale value sits on the child and nothing will correct it: folio->memcg_data now resolves to the parent, so every later update_lru_size() for those folios goes there. Dying cgroups are not freed immediately and mem_cgroup_iter() still walks them, so shrink_lruvec() keeps being called on them. get_scan_count() reads the phantom counter through lruvec_lru_size() and the scan loop then grinds through nr[] in SWAP_CLUSTER_MAX steps against an empty list, for as long as the dead cgroup lives. Under MGLRU the MGLRU scanner runs instead, but count_shadow_nodes() sums all of NR_LRU_LISTS through lruvec_lru_size() and over-budgets the shadow node limit just the same. On one 251 GiB host a sweep of every mz->lru_zone_size[] found 380 counters describing folios on no list at all: 124777314 pages, 476 GiB, 1.89x the machine's RAM, across 57 cgroups. All were on memcgs with CSS_DYING set and CSS_ONLINE clear, and parent/child pairs reported byte-identical sizes. LRU_UNEVICTABLE needs its size moved too. Its list is deliberately not spliced because lruvec_init() poisons the head - the unevictable LRU is imaginary and folios are never threaded on it - but the size is kept by lruvec_add_folio()/lruvec_del_folio() and those folios account to the parent from here on. This depends on commit bf4ade7dbd76 ("memcg: keep folio's objcg same as its node") and must not be backported ahead of it. Without that invariant a folio's objcg can belong to another node, so a folio already spliced onto the parent's list can still resolve to the child's lruvec until the objcg's node is reparented in a later iteration of memcg_reparent_objcgs(); clearing the child's counter early then lets lruvec_del_folio() underflow it and trip the WARN_ONCE()/VM_BUG_ON() in mem_cgroup_update_lru_size(). Link: https://lore.kernel.org/20260822024707.77192-1-shakeel.butt@linux.dev Fixes: 07a6e9a2c199 ("mm: vmscan: prepare for reparenting traditional LRU folios") Fixes: f304652609ea ("mm: vmscan: prepare for reparenting MGLRU folios") Signed-off-by: Shakeel Butt Acked-by: Michal Hocko Cc: Johannes Weiner Cc: Roman Gushchin Cc: Muchun Song Cc: # After: bf4ade7dbd76: memcg: keep folio's objcg same as its node Signed-off-by: Andrew Morton --- mm/folio.c | 9 +++++++++ mm/vmscan.c | 5 +++++ 2 files changed, 14 insertions(+) diff --git a/mm/folio.c b/mm/folio.c index 59c477120b9a..c02dcea9c03c 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -1130,7 +1130,16 @@ static void lruvec_reparent_lru(struct lruvec *child_lruvec, for_each_managed_zone_pgdat(zone, NODE_DATA(nid), zid, MAX_NR_ZONES - 1) { unsigned long size = mem_cgroup_get_zone_lru_size(child_lruvec, lru, zid); + if (!size) + continue; + + /* + * The folios are accounted to the parent from now on, so the + * size has to be moved, not just copied. Leaving it behind + * makes the dying child describe folios it no longer owns. + */ mem_cgroup_update_lru_size(parent_lruvec, lru, zid, size); + mem_cgroup_update_lru_size(child_lruvec, lru, zid, -(long)size); } } diff --git a/mm/vmscan.c b/mm/vmscan.c index b569eeca590d..73a81b4a3e16 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4635,7 +4635,12 @@ void lru_gen_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent, for_each_managed_zone_pgdat(zone, NODE_DATA(nid), zid, MAX_NR_ZONES - 1) { unsigned long size = mem_cgroup_get_zone_lru_size(child_lruvec, lru, zid); + if (!size) + continue; + + /* Move the accounting, do not duplicate it. */ mem_cgroup_update_lru_size(parent_lruvec, lru, zid, size); + mem_cgroup_update_lru_size(child_lruvec, lru, zid, -(long)size); } } } -- cgit v1.2.3 From 0685630fdccb62dcb0e3f44525a40578da5f6dc8 Mon Sep 17 00:00:00 2001 From: Karl Mehltretter Date: Sat, 8 Aug 2026 22:03:12 +0200 Subject: selftests/proc: make proc-maps-race work with READ_IMPLIES_EXEC test_maps_tearing_from_split times out when READ_IMPLIES_EXEC is set. This happens by default on pre-ARMv6 CPUs, which lack no-execute support. split_vma() re-maps the first page with mod_info->prot | PROT_EXEC to make it differ from its neighbours. With READ_IMPLIES_EXEC the original mapping is already executable, so no split occurs and the test hangs waiting for the modifier child. Use PROT_NONE for the split mapping, which always differs from its readable neighbours. Link: https://lore.kernel.org/20260808200312.6326-1-kmehltretter@gmail.com Fixes: beb69e817246 ("selftests/proc: add /proc/pid/maps tearing from vma split test") Assisted-by: Codex:gpt-5.6-terra Signed-off-by: Karl Mehltretter Acked-by: Suren Baghdasaryan Cc: Alexey Dobriyan Cc: Jann Horn Cc: Liam R. Howlett Cc: Shuah Khan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- tools/testing/selftests/proc/proc-maps-race.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/proc/proc-maps-race.c b/tools/testing/selftests/proc/proc-maps-race.c index 1026d8c400e1..415eccb70468 100644 --- a/tools/testing/selftests/proc/proc-maps-race.c +++ b/tools/testing/selftests/proc/proc-maps-race.c @@ -490,7 +490,8 @@ static bool query_addr_at(int maps_fd, void *addr, static inline bool split_vma(FIXTURE_DATA(proc_maps_race) *self) { - return mmap(self->mod_info->addr, self->page_size, self->mod_info->prot | PROT_EXEC, + /* PROT_NONE differs from both readable neighbors. */ + return mmap(self->mod_info->addr, self->page_size, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0) != MAP_FAILED; } -- cgit v1.2.3