From 9b5e4809806cb300cc163b26fa70dfd36e3577b3 Mon Sep 17 00:00:00 2001 From: Ethan Nelson-Moore Date: Wed, 10 Jun 2026 15:09:04 -0700 Subject: maple_tree: remove undocumented CONFIG_MAPLE_RCU_DISABLED macro consults the macro CONFIG_MAPLE_RCU_DISABLED to determine whether to disable the mt_in_rcu() function (by making it always return false). This macro is not reachable via Kconfig, despite its name, and is not documented anywhere. Remove it to avoid polluting the CONFIG_* namespace. Discovered while searching for CONFIG_* symbols referenced in code but not defined in any Kconfig file. Link: https://lore.kernel.org/20260610220905.99860-1-enelsonmoore@gmail.com Signed-off-by: Ethan Nelson-Moore Acked-by: SeongJae Park Reviewed-by: Liam Howlett Reviewed-by: Alice Ryhl Cc: Andrew Ballance Signed-off-by: Andrew Morton --- include/linux/maple_tree.h | 4 ---- 1 file changed, 4 deletions(-) (limited to 'include') diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h index 4a5631906aff..1b3014377105 100644 --- a/include/linux/maple_tree.h +++ b/include/linux/maple_tree.h @@ -11,7 +11,6 @@ #include #include #include -/* #define CONFIG_MAPLE_RCU_DISABLED */ /* * Allocated nodes are mutable until they have been inserted into the tree, @@ -864,9 +863,6 @@ static inline void mt_init(struct maple_tree *mt) static inline bool mt_in_rcu(struct maple_tree *mt) { -#ifdef CONFIG_MAPLE_RCU_DISABLED - return false; -#endif return mt->ma_flags & MT_FLAGS_USE_RCU; } -- cgit v1.2.3 From 0bd14001eb264247d565a5a44a71675df273640d Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:18 +0100 Subject: mm/vma: introduce VMA anon page offset field and add helpers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Patch series "mm/rmap: index MAP_PRIVATE file-backed folios by anonymous pgoff", v5. In memory management we've managed to manufacture a great deal of confusion around the concept of anonymous memory. We have: 1. 'Pure anon' memory - anonymous VMAs whose folios are anonymous and swap-backed (thus for reclaim purposes, treated as anonymous). These are simple enough. 2. shmem - file-backed VMAs, file-backed folios (from rmap perspective) so present in the page cache and mapped by an address_space object, but whose folios are also swap-backed (thus treated as anonymous for reclaim purposes). 3. MAP_PRIVATE-mapped /dev/zero - a strange beast whose VMAs have vma->vm_file set, but which clears vma->vm_ops to satisfy vma_is_anonymous(), resulting in VMAs that were mmap()'d referencing a file, but are in every other sense anonymous, including the folios. 4. Other MAP_PRIVATE-file backed mappings - These possess file-backed VMAs and have file-backed folios until CoW'd, at which point those CoW'd folios are anonymous. This series fixes issue 3. In order for us to traverse VMAs using the reverse mapping, we require two fields - folio->mapping and folio->index. The first tells the rmap code where to look for VMAs, and the second tells it at which offset the folio starts within the referenced object. For anonymous folios, folio->mapping points at an anon_vma object. For file-backed folios, it points at an address_space. And: * For file-backed folios folio->index is simply the page offset of the start of the folio within the file. * For anonymous folios belonging to pure anon mappings, folio->index is equal to the anonymous page offset of the folio. * For anonymous folios belonging to file-backed mappings (i.e. CoW'd folios of a MAP_PRIVATE file-backed mapping), folio->index is equal to the file page offset. This series establishes a new anonymous page offset property of VMAs to allow us to map anonymous folios at their anonymous page offset, consistent with pure anon. The purpose of doing so is to lay the foundations for the scalable CoW work. This is necessary because scalable CoW looks in the maple tree for the VMA located at folio->index << PAGE_SHIFT, before falling back to looking up tracked remaps if necessary. The MAP_PRIVATE file-backed case means that folio indices will very often conflict with one another and this remap tracking becomes substantially more contended, and of course the fast path can never be used. This also makes it possible, in future, to unshare anonymously mapped folios with deep fork hierarchies on remap, eliminating the need for remap tracking in the vast majority of cases. Similar to page offset of pure anonymous VMAs, we update the anonymous page offset of unfaulted file-backed VMAs on remap, but do not once CoW'd (i.e. vma->anon_vma is non-NULL). Overall, there is little impact on mergeability, which remains exactly the same for pure anonymous and shared file-backed mappings, with the only impact being on MAP_PRIVATE-mapped file-backed mappings, which must now match on anonymous page offset as well as file page offset to be merged. To fail to merge like this would require CoW'ing the mapping, then finding another VMA with identical file and compatible page offset to remap next to. This is therefore very much an edge case that should have very little impact (and which scalable CoW may very well address in any case). This patch (of 16): Establish fields in vm_area_struct to store the anonymous page offset of VMAs. Initially, the anonymous page offset of a VMA is vma->vm_start >> PAGE_SHIFT. When a VMA is remapped to new_address its anonymous page offset is either updated to new_address >> PAGE_SHIFT if unfaulted or, if faulted, remains equal to the anonymous page offset it had when first faulted. Currently, anonymous folios belonging to CoW'd MAP_PRIVATE-mapped file-backed VMAs are tracked by their file offsets. By adding anonymous offset as a property of VMAs, we can now track them by their anonymous page offset instead. By tracking this, we provide the means by which to eliminate this inconsistency, and more importantly lay the foundations for future work for the scalable CoW anonymous rmap rework. This patch simply adds the fields and some simple helpers. Subsequent patches will update mm code to make use of these fields correctly. The fields chosen are packed in the VMA such that, for 64-bit kernel builds, no additional space is taken up. The first field is present on cacheline 0 containing key VMA fields, and the second on cacheline 3, which contains file-backed reverse mapping fields. Given the relative time spent accessing reverse mapping fields as well as updating them, there shouldn't be any performance impact here from false sharing. Update the VMA userland tests to account for this change. No callsites are updated yet, so no functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-0-c21581c0c3c8@kernel.org Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-1-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Reviewed-by: Gregory Price (Meta) Reviewed-by: Xu Xin Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- include/linux/mm.h | 59 +++++++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 12 +++++++++ mm/vma.h | 14 ++++++++++ mm/vma_init.c | 1 + tools/testing/vma/include/dup.h | 26 ++++++++++++++++++ 5 files changed, 112 insertions(+) (limited to 'include') diff --git a/include/linux/mm.h b/include/linux/mm.h index 87feaa5a2b78..df78847f5f07 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4393,6 +4393,65 @@ static inline pgoff_t vma_last_pgoff(const struct vm_area_struct *vma) return vma_end_pgoff(vma) - 1; } +/** + * vma_start_anon_pgoff() - Get the anonymous page offset of the start of @vma + * @vma: The VMA whose anonymous page offset is required. + * + * If unfaulted, then this is vma->vm_start >> PAGE_SHIFT, if faulted then the + * anonymous page offset at the time of first fault. + * + * If the VMA is anonymous, this returns the same value as vma_start_pgoff(). + * + * This value is used for tracking MAP_PRIVATE file-backed mappings by their + * anonymous page offset. + * + * Returns: The anonymous page offset of the start of @vma. + */ +static inline pgoff_t vma_start_anon_pgoff(const struct vm_area_struct *vma) +{ + pgoff_t pgoff = 0; + +#ifdef CONFIG_64BIT + pgoff += vma->__vm_anon_pgoff_hi; + pgoff <<= 32; +#endif + pgoff += vma->__vm_anon_pgoff_lo; + return pgoff; +} + +/** + * vma_end_anon_pgoff() - Get the anonymous page offset of the exclusive end of + * @vma. + * @vma: The VMA whose end anonymous page offset is required. + * + * This returns the anonymous exclusive end page offset of @vma, which is useful + * for expressing page offset ranges. + * + * See the description of vma_start_anon_pgoff() for a description of VMA + * anonymous page offsets. + * + * Returns: The exclusive end anonymous page offset of @vma. + */ +static inline pgoff_t vma_end_anon_pgoff(const struct vm_area_struct *vma) +{ + return vma_start_anon_pgoff(vma) + vma_pages(vma); +} + +/** + * vma_last_anon_pgoff() - Get the anonymous page offset of the last page in + * @vma. + * @vma: The VMA whose last anonymous page offset is required. + * + * See the description of vma_start_anon_pgoff() for a description of VMA + * anonymous page offsets. + * + * Returns: The last anonymous page offset of @vma. + */ +static inline pgoff_t vma_last_anon_pgoff(const struct vm_area_struct *vma) +{ + return vma_end_anon_pgoff(vma) - 1; +} + static inline unsigned long vma_desc_size(const struct vm_area_desc *desc) { return desc->end - desc->start; diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 939b5ea8c9e0..ebf0d912be7d 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -967,6 +967,11 @@ struct vm_area_struct { */ unsigned int vm_lock_seq; #endif + /* + * Low 32-bits of anonymous page offset. + * See vma_start_anon_pgoff() comment for details. + */ + unsigned int __vm_anon_pgoff_lo; /* * A file's MAP_PRIVATE vma can be in both i_mmap tree and anon_vma * list, after a COW of one of the file pages. A MAP_SHARED vma @@ -1041,6 +1046,13 @@ struct vm_area_struct { #ifdef CONFIG_DEBUG_LOCK_ALLOC struct lockdep_map vmlock_dep_map; #endif +#endif +#ifdef CONFIG_64BIT + /* + * High 32-bits of anonymous page offset. + * See vma_start_anon_pgoff() comment for details. + */ + unsigned int __vm_anon_pgoff_hi; #endif /* * For areas with an address space and backing store, diff --git a/mm/vma.h b/mm/vma.h index 0bc7d521e976..54ed7c744e3b 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -283,6 +283,20 @@ static inline void vma_set_pgoff(struct vm_area_struct *vma, pgoff_t pgoff) vma->vm_pgoff = pgoff; } +static inline void __vma_set_anon_pgoff(struct vm_area_struct *vma, pgoff_t pgoff) +{ +#ifdef CONFIG_64BIT + vma->__vm_anon_pgoff_hi = pgoff >> 32; +#endif + vma->__vm_anon_pgoff_lo = pgoff & GENMASK(31, 0); +} + +static inline void vma_set_anon_pgoff(struct vm_area_struct *vma, pgoff_t pgoff) +{ + vma_assert_can_modify(vma); + __vma_set_anon_pgoff(vma, pgoff); +} + static inline void vma_add_pgoff(struct vm_area_struct *vma, pgoff_t delta) { vma_assert_can_modify(vma); diff --git a/mm/vma_init.c b/mm/vma_init.c index 715feee283f0..baa7e82f47e3 100644 --- a/mm/vma_init.c +++ b/mm/vma_init.c @@ -51,6 +51,7 @@ static void vm_area_init_from(const struct vm_area_struct *src, dest->vm_end = src->vm_end; dest->anon_vma = src->anon_vma; dest->vm_pgoff = vma_start_pgoff(src); + __vma_set_anon_pgoff(dest, vma_start_anon_pgoff(src)); dest->vm_file = src->vm_file; dest->vm_private_data = src->vm_private_data; vm_flags_init(dest, src->vm_flags); diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index cdeb53bbdd1b..17f94e5de569 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -577,6 +577,7 @@ struct vm_area_struct { */ unsigned int vm_lock_seq; #endif + unsigned int __vm_anon_pgoff_lo; /* * A file's MAP_PRIVATE vma can be in both i_mmap tree and anon_vma @@ -612,6 +613,9 @@ struct vm_area_struct { #ifdef CONFIG_PER_VMA_LOCK /* Unstable RCU readers are allowed to read this. */ refcount_t vm_refcnt; +#endif +#ifdef CONFIG_64BIT + unsigned int __vm_anon_pgoff_hi; #endif /* * For areas with an address space and backing store, @@ -1320,6 +1324,28 @@ static inline pgoff_t vma_end_pgoff(const struct vm_area_struct *vma) return vma_start_pgoff(vma) + vma_pages(vma); } +static inline pgoff_t vma_start_anon_pgoff(const struct vm_area_struct *vma) +{ + pgoff_t pgoff = 0; + +#ifdef CONFIG_64BIT + pgoff += vma->__vm_anon_pgoff_hi; + pgoff <<= 32; +#endif + pgoff += vma->__vm_anon_pgoff_lo; + return pgoff; +} + +static inline pgoff_t vma_end_anon_pgoff(const struct vm_area_struct *vma) +{ + return vma_start_anon_pgoff(vma) + vma_pages(vma); +} + +static inline pgoff_t vma_last_anon_pgoff(const struct vm_area_struct *vma) +{ + return vma_end_anon_pgoff(vma) - 1; +} + static inline int vfs_mmap_prepare(struct file *file, struct vm_area_desc *desc) { return file->f_op->mmap_prepare(desc); -- cgit v1.2.3 From 51943a18ad4bd6ff8baea2da7b8cce2f86f1a959 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:19 +0100 Subject: mm: provide vma_[flags_]is_cow_mapping() and remove is_cow_mapping() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All remaining callers of is_cow_mapping() are invoking it in the form of is_cow_mapping(vma->vm_flags) or an indirected version of this. Therefore, provide a helper - vma_is_cow_mapping() to directly test the VMA. Additionally provide a new helper vma_flags_is_cow_mapping() which performs the check using the new vma_flags_t type, and share this logic between vma_is_cow_mapping() and vma_desc_is_cow_mapping(). With these changes, no callers of is_cow_mapping() remain, so remove it. Also update the userland VMA tests to reflect the change. No functional change intended. [akpm@linux-foundation.org: fix kerneldoc comment typo, per Lorenzo] Link: https://lore.kernel.org/aob1goSSPH6sTN9y@gremlin Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-2-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Gregory Price (Meta) Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- arch/s390/mm/gmap_helpers.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 4 +- drivers/gpu/drm/drm_gem_shmem_helper.c | 2 +- drivers/gpu/drm/panthor/panthor_gem.c | 2 +- drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +- drivers/gpu/drm/xe/xe_device.c | 2 +- fs/proc/task_mmu.c | 2 +- include/linux/mm.h | 71 ++++++++++++++++++++++++++++++--- kernel/events/uprobes.c | 2 +- mm/gup.c | 2 +- mm/huge_memory.c | 8 ++-- mm/hugetlb.c | 2 +- mm/internal.h | 2 +- mm/memory.c | 25 ++++++------ mm/mempolicy.c | 2 +- tools/testing/vma/include/dup.h | 11 +++++ 16 files changed, 105 insertions(+), 36 deletions(-) (limited to 'include') diff --git a/arch/s390/mm/gmap_helpers.c b/arch/s390/mm/gmap_helpers.c index 4bf7c9012feb..cd5fded159c0 100644 --- a/arch/s390/mm/gmap_helpers.c +++ b/arch/s390/mm/gmap_helpers.c @@ -200,7 +200,7 @@ static int find_zeropage_pte_entry(pte_t *pte, unsigned long addr, * currently only works in COW mappings, which is also where * mm_forbids_zeropage() is checked. */ - if (!is_cow_mapping(walk->vma->vm_flags)) + if (!vma_is_cow_mapping(walk->vma)) return -EFAULT; *found_addr = addr; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c index 6a0699746fbc..0c7309080a7a 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c @@ -377,9 +377,9 @@ static int amdgpu_gem_object_mmap(struct drm_gem_object *obj, struct vm_area_str /* Workaround for Thunk bug creating PROT_NONE,MAP_PRIVATE mappings * for debugger access to invisible VRAM. Should have used MAP_SHARED * instead. Clearing VM_MAYWRITE prevents the mapping from ever - * becoming writable and makes is_cow_mapping(vm_flags) false. + * becoming writable and makes vma_is_cow_mapping(vma) false. */ - if (is_cow_mapping(vma->vm_flags) && + if (vma_is_cow_mapping(vma) && !(vma->vm_flags & VM_ACCESS_FLAGS)) vm_flags_clear(vma, VM_MAYWRITE); diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c b/drivers/gpu/drm/drm_gem_shmem_helper.c index 06d019d51d3e..177d0e0b9334 100644 --- a/drivers/gpu/drm/drm_gem_shmem_helper.c +++ b/drivers/gpu/drm/drm_gem_shmem_helper.c @@ -753,7 +753,7 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_struct return ret; } - if (is_cow_mapping(vma->vm_flags)) + if (vma_is_cow_mapping(vma)) return -EINVAL; dma_resv_lock(shmem->base.resv, NULL); diff --git a/drivers/gpu/drm/panthor/panthor_gem.c b/drivers/gpu/drm/panthor/panthor_gem.c index 770556353968..d2eec46f7abe 100644 --- a/drivers/gpu/drm/panthor/panthor_gem.c +++ b/drivers/gpu/drm/panthor/panthor_gem.c @@ -761,7 +761,7 @@ static int panthor_gem_mmap(struct drm_gem_object *obj, struct vm_area_struct *v return ret; } - if (is_cow_mapping(vma->vm_flags)) + if (vma_is_cow_mapping(vma)) return -EINVAL; if (!refcount_inc_not_zero(&bo->cmap.mmap_count)) { diff --git a/drivers/gpu/drm/ttm/ttm_bo_vm.c b/drivers/gpu/drm/ttm/ttm_bo_vm.c index 88babf435ac2..872bf444b1f0 100644 --- a/drivers/gpu/drm/ttm/ttm_bo_vm.c +++ b/drivers/gpu/drm/ttm/ttm_bo_vm.c @@ -489,7 +489,7 @@ static const struct vm_operations_struct ttm_bo_vm_ops = { int ttm_bo_mmap_obj(struct vm_area_struct *vma, struct ttm_buffer_object *bo) { /* Enforce no COW since would have really strange behavior with it. */ - if (is_cow_mapping(vma->vm_flags)) + if (vma_is_cow_mapping(vma)) return -EINVAL; drm_gem_object_get(&bo->base); diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c index 9d119c95a569..de5fdf49d729 100644 --- a/drivers/gpu/drm/xe/xe_device.c +++ b/drivers/gpu/drm/xe/xe_device.c @@ -330,7 +330,7 @@ static int xe_pci_barrier_mmap(struct file *filp, if (vma->vm_end - vma->vm_start > SZ_4K) return -EINVAL; - if (is_cow_mapping(vma->vm_flags)) + if (vma_is_cow_mapping(vma)) return -EINVAL; if (vma->vm_flags & (VM_READ | VM_EXEC)) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 817e3e0f9194..5c54aebe2118 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1693,7 +1693,7 @@ static inline bool pte_is_pinned(struct vm_area_struct *vma, unsigned long addr, if (!pte_write(pte)) return false; - if (!is_cow_mapping(vma->vm_flags)) + if (!vma_is_cow_mapping(vma)) return false; if (likely(!mm_flags_test(MMF_HAS_PINNED, vma->vm_mm))) return false; diff --git a/include/linux/mm.h b/include/linux/mm.h index df78847f5f07..20361b4344ea 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2271,17 +2271,76 @@ void unpin_user_pages(struct page **pages, unsigned long npages); void unpin_user_folio(struct folio *folio, unsigned long npages); void unpin_folios(struct folio **folios, unsigned long nfolios); -static inline bool is_cow_mapping(vm_flags_t flags) +/** + * vma_flags_is_cow_mapping() - Do these VMA flags imply a CoW mapping? + * @flags: The VMA flags to check. + * + * Mappings which could be CoW'd (subject to Copy-On-Write faults) are + * described as CoW mappings. + * + * All mappings backed by anonymous folios (all anonymous mappings and most + * MAP_PRIVATE-file backed ranges) are CoW mappings. + * + * All other mappings (including all MAP_SHARED mappings) are non-CoW. + * + * The criteria are !VMA_SHARED_BIT, VMA_MAYWRITE_BIT. + * + * VMA_MAYWRITE_BIT is checked instead of VMA_WRITE_BIT to account for both + * future mprotect() calls which can render a read-only mapping writable, and + * GUP with FOLL_FORCE (e.g. ptrace) which can CoW a read-only mapping. + * + * - No anonymous mapping can ever clear VMA_MAYWRITE_BIT. + * + * - Writes to anonymous mappings do not immediately result in CoW faults but + * may do so after the process is forked or if a read is followed by a + * write. + * + * - Writes to MAP_PRIVATE file-backed mappings result in CoW faults and may + * do so again after fork. + * + * - MAP_SHARED mappings of a file opened read-only are transformed into + * VMA_MAYSHARE_BIT, !VMA_SHARED_BIT, !VMA_MAYWRITE_BIT mappings, so remain + * non-CoW. + * + * - Drivers may clear VMA_MAYWRITE_BIT but do so at mmap() time and cannot + * mark themselves anonymous. Having cleared this flag it is not valid for + * them to leave the VMA_WRITE_BIT flag set. + * + * As a consequence, the anonymous reverse mapping only tracks CoW mappings. + * + * Returns: true if the flags indicate a CoW mapping, otherwise false. + */ +static inline bool vma_flags_is_cow_mapping(const vma_flags_t *flags) +{ + return vma_flags_test(flags, VMA_MAYWRITE_BIT) && + !vma_flags_test(flags, VMA_SHARED_BIT); +} + +/** + * vma_is_cow_mapping() - Is this VMA a CoW mapping? + * @vma: The VMA to check. + * + * See vma_flags_is_cow_mapping() for details. + * + * Returns: true if the VMA is a CoW mapping, otherwise false. + */ +static inline bool vma_is_cow_mapping(const struct vm_area_struct *vma) { - return (flags & (VM_SHARED | VM_MAYWRITE)) == VM_MAYWRITE; + return vma_flags_is_cow_mapping(&vma->flags); } +/** + * vma_desc_is_cow_mapping() - Is this VMA descriptor a CoW mapping? + * @desc: The VMA descriptor to check. + * + * See vma_flags_is_cow_mapping() for details. + * + * Returns: true if the VMA descriptor describes a CoW mapping, otherwise + * false. + */ static inline bool vma_desc_is_cow_mapping(struct vm_area_desc *desc) { - const vma_flags_t *flags = &desc->vma_flags; - - return vma_flags_test(flags, VMA_MAYWRITE_BIT) && - !vma_flags_test(flags, VMA_SHARED_BIT); + return vma_flags_is_cow_mapping(&desc->vma_flags); } #ifndef CONFIG_MMU diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index ae2f3b9f8d50..eb0d11092fb3 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -513,7 +513,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, uprobe = container_of(auprobe, struct uprobe, arch); - if (WARN_ON_ONCE(!is_cow_mapping(vma->vm_flags))) + if (WARN_ON_ONCE(!vma_is_cow_mapping(vma))) return -EINVAL; /* diff --git a/mm/gup.c b/mm/gup.c index 99902c15703b..8ea3de60e82d 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1236,7 +1236,7 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags) * Anon pages in shared mappings are surprising: now * just reject it. */ - if (!is_cow_mapping(vm_flags)) + if (!vma_is_cow_mapping(vma)) return -EFAULT; } } else if (!(vm_flags & VM_READ)) { diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 644d6905b49c..ff13b57d9d56 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1681,7 +1681,7 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, unsigned long pfn, BUG_ON(!(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP))); BUG_ON((vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)) == (VM_PFNMAP|VM_MIXEDMAP)); - BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags)); + BUG_ON((vma->vm_flags & VM_PFNMAP) && vma_is_cow_mapping(vma)); pfnmap_setup_cachemode_pfn(pfn, &pgprot); @@ -1789,7 +1789,7 @@ vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, unsigned long pfn, BUG_ON(!(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP))); BUG_ON((vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)) == (VM_PFNMAP|VM_MIXEDMAP)); - BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags)); + BUG_ON((vma->vm_flags & VM_PFNMAP) && vma_is_cow_mapping(vma)); pfnmap_setup_cachemode_pfn(pfn, &pgprot); @@ -1931,7 +1931,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, * applied special bit, or we made the PRIVATE mapping be * able to wrongly write to the backend MMIO. */ - VM_WARN_ON_ONCE(is_cow_mapping(src_vma->vm_flags) && pmd_write(pmd)); + VM_WARN_ON_ONCE(vma_is_cow_mapping(src_vma) && pmd_write(pmd)); goto set_pmd; } @@ -2052,7 +2052,7 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, * TODO: once we support anonymous pages, use * folio_try_dup_anon_rmap_*() and split if duplicating fails. */ - if (is_cow_mapping(vma->vm_flags) && pud_write(pud)) { + if (vma_is_cow_mapping(vma) && pud_write(pud)) { pudp_set_wrprotect(src_mm, addr, src_pud); pud = pud_wrprotect(pud); } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 79e5c3b3e850..49bf325325c0 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4898,7 +4898,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, pte_t *src_pte, *dst_pte, entry; struct folio *pte_folio; unsigned long addr; - bool cow = is_cow_mapping(src_vma->vm_flags); + bool cow = vma_is_cow_mapping(src_vma); struct hstate *h = hstate_vma(src_vma); unsigned long sz = huge_page_size(h); unsigned long npages = pages_per_huge_page(h); diff --git a/mm/internal.h b/mm/internal.h index 68db5abd0a4c..a75e7641ef49 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1353,7 +1353,7 @@ static inline bool gup_must_unshare(struct vm_area_struct *vma, * ... because we only care about writable private ("COW") * mappings where we have to break COW early. */ - return is_cow_mapping(vma->vm_flags); + return vma_is_cow_mapping(vma); } /* Paired with a memory barrier in folio_try_share_anon_rmap_*(). */ diff --git a/mm/memory.c b/mm/memory.c index d2f14ba2261c..396d7b9059e6 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -780,7 +780,7 @@ static inline struct page *__vm_normal_page(struct vm_area_struct *vma, /* Only CoW'ed anon folios are "normal". */ if (pfn == index) return NULL; - if (!is_cow_mapping(vma->vm_flags)) + if (!vma_is_cow_mapping(vma)) return NULL; } } @@ -1002,7 +1002,6 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, pte_t *dst_pte, pte_t *src_pte, struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, unsigned long addr, int *rss) { - vm_flags_t vm_flags = dst_vma->vm_flags; pte_t orig_pte = ptep_get(src_pte); softleaf_t entry = softleaf_from_pte(orig_pte); pte_t pte = orig_pte; @@ -1026,7 +1025,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, rss[mm_counter(folio)]++; if (!softleaf_is_migration_read(entry) && - is_cow_mapping(vm_flags)) { + vma_is_cow_mapping(dst_vma)) { /* * COW mappings require pages in both parent and child * to be set to read. A previously exclusive entry is @@ -1067,7 +1066,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, * save and restore device driver state). */ if (softleaf_is_device_private_write(entry) && - is_cow_mapping(vm_flags)) { + vma_is_cow_mapping(dst_vma)) { entry = make_readable_device_private_entry( swp_offset(entry)); pte = swp_entry_to_pte(entry); @@ -1082,7 +1081,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, * exclusive entries currently only support private writable * (ie. COW) mappings. */ - VM_BUG_ON(!is_cow_mapping(src_vma->vm_flags)); + VM_BUG_ON(!vma_is_cow_mapping(src_vma)); if (try_restore_exclusive_pte(src_vma, addr, src_pte, orig_pte)) return -EBUSY; return -ENOENT; @@ -1181,7 +1180,7 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma, } /* If it's a COW mapping, write protect it both processes. */ - if (is_cow_mapping(src_vma->vm_flags) && writable) { + if (vma_is_cow_mapping(src_vma) && writable) { wrprotect_ptes(src_mm, addr, src_pte, nr); pte = pte_wrprotect(pte); } @@ -1602,9 +1601,9 @@ copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) * We need to invalidate the secondary MMU mappings only when * there could be a permission downgrade on the ptes of the * parent mm. And a permission downgrade will only happen if - * is_cow_mapping() returns true. + * vma_is_cow_mapping() returns true. */ - is_cow = is_cow_mapping(src_vma->vm_flags); + is_cow = vma_is_cow_mapping(src_vma); if (is_cow) { mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_PAGE, @@ -2437,7 +2436,7 @@ static bool vm_mixed_zeropage_allowed(struct vm_area_struct *vma) if (mm_forbids_zeropage(vma->vm_mm)) return false; /* zeropages in COW mappings are common and unproblematic. */ - if (is_cow_mapping(vma->vm_flags)) + if (vma_is_cow_mapping(vma)) return true; /* Mappings that do not allow for writable PTEs are unproblematic. */ if (!(vma->vm_flags & (VM_WRITE | VM_MAYWRITE))) @@ -2888,7 +2887,7 @@ vm_fault_t vmf_insert_pfn_prot(struct vm_area_struct *vma, unsigned long addr, BUG_ON(!(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP))); BUG_ON((vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)) == (VM_PFNMAP|VM_MIXEDMAP)); - BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags)); + BUG_ON((vma->vm_flags & VM_PFNMAP) && vma_is_cow_mapping(vma)); BUG_ON((vma->vm_flags & VM_MIXEDMAP) && pfn_valid(pfn)); if (addr < vma->vm_start || addr >= vma->vm_end) @@ -3300,7 +3299,7 @@ static int remap_pfn_range_prepare_vma(struct vm_area_struct *vma, unsigned long size) { const unsigned long end = addr + PAGE_ALIGN(size); - const bool is_cow = is_cow_mapping(vma->vm_flags); + const bool is_cow = vma_is_cow_mapping(vma); int err; err = get_remap_pgoff(is_cow, addr, end, vma->vm_start, vma->vm_end, @@ -6800,7 +6799,7 @@ static vm_fault_t sanitize_fault_flags(struct vm_area_struct *vma, * FAULT_FLAG_UNSHARE only applies to COW mappings. Let's * just treat it like an ordinary read-fault otherwise. */ - if (!is_cow_mapping(vma->vm_flags)) + if (!vma_is_cow_mapping(vma)) *flags &= ~FAULT_FLAG_UNSHARE; } else if (*flags & FAULT_FLAG_WRITE) { /* Write faults on read-only mappings are impossible ... */ @@ -6808,7 +6807,7 @@ static vm_fault_t sanitize_fault_flags(struct vm_area_struct *vma, return VM_FAULT_SIGSEGV; /* ... and FOLL_FORCE only applies to COW mappings. */ if (WARN_ON_ONCE(!(vma->vm_flags & VM_WRITE) && - !is_cow_mapping(vma->vm_flags))) + !vma_is_cow_mapping(vma))) return VM_FAULT_SIGSEGV; } #ifdef CONFIG_PER_VMA_LOCK diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 5720f7f54d94..3498a5651d50 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -844,7 +844,7 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma, return false; /* Also skip shared copy-on-write folios */ - if (is_cow_mapping(vma->vm_flags) && folio_maybe_mapped_shared(folio)) + if (vma_is_cow_mapping(vma) && folio_maybe_mapped_shared(folio)) return false; /* Folios are pinned and can't be migrated */ diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index 17f94e5de569..af2fd3f607b5 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -1162,6 +1162,17 @@ static inline bool vma_is_shared_maywrite(struct vm_area_struct *vma) return is_shared_maywrite(&vma->flags); } +static inline bool vma_flags_is_cow_mapping(const vma_flags_t *flags) +{ + return vma_flags_test(flags, VMA_MAYWRITE_BIT) && + !vma_flags_test(flags, VMA_SHARED_BIT); +} + +static inline bool vma_is_cow_mapping(const struct vm_area_struct *vma) +{ + return vma_flags_is_cow_mapping(&vma->flags); +} + static inline struct vm_area_struct *vma_next(struct vma_iterator *vmi) { /* -- cgit v1.2.3 From 7e6543d1f939eaaca008c13395e52cbd07605cb0 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:20 +0100 Subject: mm: introduce linear_anon_page_index() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This function provides the anonymous equivalent of linear_page_index(), instead offsetting based on the anonymous page offset of the VMA. It is valid only for anonymous or MAP_PRIVATE file-backed mappings, in other words CoW mappings. For pure anon VMAs, this will be equal to linear_page_index(). Assert that both of these invariants are true in linear_anon_page_index() and implement the algorithm in __linear_anon_page_index(). Note that MAP_PRIVATE-/dev/zero mappings will satisfy vma_is_anonymous() but not fulfill this invariant, so when asserting this we check vma->vm_file to account for this. We do not update callsites yet, so no functional change intended. Also const-ify vma_is_anonymous() to make it compatible with the const-ified linear_anon_page_index(). While we're here, update linear_page_index() to be more succinct. VMA userland tests are also updated accordingly. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-3-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Reviewed-by: Gregory Price (Meta) Acked-by: David Hildenbrand (Arm) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- include/linux/mm.h | 2 +- include/linux/pagemap.h | 40 +++++++++++++++++++++++++++++++++++++--- tools/testing/vma/include/dup.h | 25 ++++++++++++++++++++++++- 3 files changed, 62 insertions(+), 5 deletions(-) (limited to 'include') diff --git a/include/linux/mm.h b/include/linux/mm.h index 20361b4344ea..dd09c438fa23 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1556,7 +1556,7 @@ static inline void vma_desc_set_anonymous(struct vm_area_desc *desc) desc->vm_ops = NULL; } -static inline bool vma_is_anonymous(struct vm_area_struct *vma) +static inline bool vma_is_anonymous(const struct vm_area_struct *vma) { return !vma->vm_ops; } diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index c6fc783aaee5..0adfa6605653 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -1094,10 +1094,44 @@ static inline pgoff_t linear_page_delta(const struct vm_area_struct *vma, static inline pgoff_t linear_page_index(const struct vm_area_struct *vma, const unsigned long address) { - pgoff_t pgoff; + return linear_page_delta(vma, address) + vma_start_pgoff(vma); +} + +static inline pgoff_t __linear_anon_page_index(const struct vm_area_struct *vma, + const unsigned long address) +{ + return linear_page_delta(vma, address) + vma_start_anon_pgoff(vma); +} + +/** + * linear_anon_page_index() - Determine the absolute anonymous page offset of + * @address within @vma. + * @vma: An anonymous or MAP_PRIVATE file-backed VMA in which @address resides. + * @address: The address whose absolute page offset is required. + * + * This returns the anonymous page offset of @address, which is the page offset + * the address possessed at the time the VMA was first faulted. + * + * For anonymous mappings, this returns the same value as linear_page_index(). + * + * For MAP_PRIVATE file-backed mappings, this returns the anonymous page offset + * of @address, which is the page offset the address possessed at the time the + * VMA was first faulted. + * + * It is not valid to call this function for shared file-backed mappings. + * + * Returns: The absolute anonymous page offset of @address within @vma. + */ +static inline pgoff_t linear_anon_page_index(const struct vm_area_struct *vma, + const unsigned long address) +{ + const pgoff_t pgoff = __linear_anon_page_index(vma, address); + + VM_WARN_ON_ONCE(!vma_is_cow_mapping(vma)); + /* Account for MAP_PRIVATE-/dev/zero which is only semi-anonymous. */ + if (vma_is_anonymous(vma) && !vma->vm_file) + VM_WARN_ON_ONCE(pgoff != linear_page_index(vma, address)); - pgoff = linear_page_delta(vma, address); - pgoff += vma_start_pgoff(vma); return pgoff; } diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index af2fd3f607b5..4655aecffaf3 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -1428,7 +1428,7 @@ static inline void vma_iter_set(struct vma_iterator *vmi, unsigned long addr) mas_set(&vmi->mas, addr); } -static inline bool vma_is_anonymous(struct vm_area_struct *vma) +static inline bool vma_is_anonymous(const struct vm_area_struct *vma) { return !vma->vm_ops; } @@ -1621,3 +1621,26 @@ static inline pgprot_t vma_get_page_prot(const struct vm_area_struct *vma) { return vma_flags_to_page_prot(vma->flags); } + +static inline pgoff_t __linear_anon_page_index(const struct vm_area_struct *vma, + const unsigned long address) +{ + pgoff_t pgoff; + + pgoff = linear_page_delta(vma, address); + pgoff += vma_start_anon_pgoff(vma); + return pgoff; +} + +static inline pgoff_t linear_anon_page_index(const struct vm_area_struct *vma, + const unsigned long address) +{ + const pgoff_t pgoff = __linear_anon_page_index(vma, address); + + VM_WARN_ON_ONCE(!vma_is_cow_mapping(vma)); + /* Account for MAP_PRIVATE-/dev/zero which is only semi-anonymous. */ + if (vma_is_anonymous(vma) && !vma->vm_file) + VM_WARN_ON_ONCE(pgoff != linear_page_index(vma, address)); + + return pgoff; +} -- cgit v1.2.3 From 50c5f35a64aad82e3fd22e26f2b9c9c7395fa206 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:27 +0100 Subject: mm/rmap: track whether the page VMA mapped pgoff is anonymous MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update the page_vma_mapped_walk structure to track whether the pgoff being tracked is an anonymous pgoff or not and update the comments to reflect this. This is necessary in order to determine the correct VMA page offset in vma_address_end() when pvmw->nr_pages > 1. Also document that pvmw->pgoff is meaningless for pvmw->nr_pages == 1 and for KSM. Do not set this field where pgoff is not specified. This is laying the groundwork for eventually using anonymous page offsets as the index for all anonymous folios. No functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-10-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Gregory Price (Meta) Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- include/linux/rmap.h | 4 +++- mm/rmap.c | 2 ++ 2 files changed, 5 insertions(+), 1 deletion(-) (limited to 'include') diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 8dc0871e5f00..0574537a355c 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -864,13 +864,14 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr, struct page_vma_mapped_walk { unsigned long pfn; unsigned long nr_pages; - pgoff_t pgoff; + pgoff_t pgoff; /* Only meaningful if nr_pages > 1 and not a KSM walk */ struct vm_area_struct *vma; unsigned long address; pmd_t *pmd; pte_t *pte; spinlock_t *ptl; unsigned int flags; + bool pgoff_is_anon : 1; }; #define DEFINE_FOLIO_VMA_WALK(name, _folio, _vma, _address, _flags) \ @@ -881,6 +882,7 @@ struct page_vma_mapped_walk { .vma = _vma, \ .address = _address, \ .flags = _flags, \ + .pgoff_is_anon = folio_test_anon(_folio), \ } static inline void page_vma_mapped_walk_done(struct page_vma_mapped_walk *pvmw) diff --git a/mm/rmap.c b/mm/rmap.c index bf618e4678d3..1b23ac709f45 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1240,6 +1240,7 @@ static bool mapping_wrprotect_range_one(struct folio *folio, .vma = vma, .address = address, .flags = PVMW_SYNC, + .pgoff_is_anon = false, }; state->cleaned += page_vma_mkclean_one(&pvmw); @@ -1317,6 +1318,7 @@ int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, .pgoff = pgoff, .vma = vma, .flags = PVMW_SYNC, + .pgoff_is_anon = false, }; if (invalid_mkclean_vma(vma, NULL)) -- cgit v1.2.3 From 8e658ecc3be21e43573e6639af2d1c536bbfe67d Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:29 +0100 Subject: mm/huge_memory: update remove_migration_pmd() to accept a folio MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This function does not need to accept a page and requiring it to is unnecessary and misleading. make_[writable, readable]_device_private_entry() must be passed a PMD-aligned PFN as they immediately used to obtain a softleaf PMD entry and the same argument applies to folio_add_[anon, file]_rmap_pmd(). While we are here, update a VM_BUG_ON() to a VM_WARN_ON_ONCE(). No functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-12-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Gregory Price (Meta) Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- include/linux/swapops.h | 6 +++--- mm/huge_memory.c | 16 +++++++--------- mm/migrate.c | 2 +- 3 files changed, 11 insertions(+), 13 deletions(-) (limited to 'include') diff --git a/include/linux/swapops.h b/include/linux/swapops.h index c956bc445ee0..1f3ff3b93e16 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -325,8 +325,8 @@ struct page_vma_mapped_walk; extern int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, struct page *page); -extern void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, - struct page *new); +void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, + struct folio *folio); extern void pmd_migration_entry_wait(struct mm_struct *mm, pmd_t *pmd); @@ -346,7 +346,7 @@ static inline int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, } static inline void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, - struct page *new) + struct folio *folio) { BUILD_BUG(); } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index ff13b57d9d56..2822190daf2b 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -5077,9 +5077,8 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, return 0; } -void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) +void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct folio *folio) { - struct folio *folio = page_folio(new); struct vm_area_struct *vma = pvmw->vma; struct mm_struct *mm = vma->vm_mm; unsigned long address = pvmw->address; @@ -5115,11 +5114,9 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) swp_entry_t entry; if (pmd_write(pmde)) - entry = make_writable_device_private_entry( - page_to_pfn(new)); + entry = make_writable_device_private_entry(folio_pfn(folio)); else - entry = make_readable_device_private_entry( - page_to_pfn(new)); + entry = make_readable_device_private_entry(folio_pfn(folio)); pmde = softleaf_to_pmd(entry); if (pmd_swp_soft_dirty(*pvmw->pmd)) @@ -5134,11 +5131,12 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) if (!softleaf_is_migration_read(entry)) rmap_flags |= RMAP_EXCLUSIVE; - folio_add_anon_rmap_pmd(folio, new, vma, haddr, rmap_flags); + folio_add_anon_rmap_pmd(folio, &folio->page, vma, haddr, rmap_flags); } else { - folio_add_file_rmap_pmd(folio, new, vma); + folio_add_file_rmap_pmd(folio, &folio->page, vma); } - VM_BUG_ON(pmd_write(pmde) && folio_test_anon(folio) && !PageAnonExclusive(new)); + VM_WARN_ON_ONCE(pmd_write(pmde) && folio_test_anon(folio) && + !PageAnonExclusive(&folio->page)); set_pmd_at(mm, haddr, pvmw->pmd, pmde); /* No need to invalidate - it was non-present before */ diff --git a/mm/migrate.c b/mm/migrate.c index 8aaafcea7bc1..9e32af3fe303 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -372,7 +372,7 @@ static bool remove_migration_pte(struct folio *folio, if (!pvmw.pte) { VM_BUG_ON_FOLIO(folio_test_hugetlb(folio) || !folio_test_pmd_mappable(folio), folio); - remove_migration_pmd(&pvmw, new); + remove_migration_pmd(&pvmw, folio); continue; } #endif -- cgit v1.2.3 From 3bf07ce8058bcab03ea1cb9cd11ec1d6b2ee5af0 Mon Sep 17 00:00:00 2001 From: Baolin Wang Date: Mon, 20 Jul 2026 19:12:00 +0800 Subject: mm: vmscan: convert folio_referenced() to use vma_flags_t Patch series "promote mapped executable folios after first usage for MGLRU", v4. Now MGLRU's protection of mapped executable file folios is less reliable. Follow the classical LRU's logic, promoting mapped executable file folios after their first usage to give executable code a better chance to stay in memory and improve workload performance (See patch 2 for more details). This patch (of 3): Replace use of the legacy vm_flags_t flags with vma_flags_t values for folio_referenced() and related logic. This is also a preparation for the following changes. No functional changes. Link: https://lore.kernel.org/cover.1784509721.git.baolin.wang@linux.alibaba.com Link: https://lore.kernel.org/2bd39e16ec19e3e3c4716aa9a1a25775c26cac57.1784509721.git.baolin.wang@linux.alibaba.com Signed-off-by: Baolin Wang Acked-by: David Hildenbrand (Arm) Acked-by: Johannes Weiner Reviewed-by: Barry Song Reviewed-by: Kairui Song Reviewed-by: Axel Rasmussen Cc: Harry Yoo Cc: Jann Horn Cc: Lance Yang Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Rik van Riel Cc: Shakeel Butt Cc: Vlastimil Babka Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton --- include/linux/rmap.h | 7 +++---- mm/rmap.c | 19 +++++++++++-------- mm/vmscan.c | 14 +++++++------- 3 files changed, 21 insertions(+), 19 deletions(-) (limited to 'include') diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 0574537a355c..0b332770abee 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -843,7 +843,7 @@ static inline int folio_try_share_anon_rmap_pmd(struct folio *folio, * Called from mm/vmscan.c to handle paging out */ int folio_referenced(struct folio *, int is_locked, - struct mem_cgroup *memcg, vm_flags_t *vm_flags); + struct mem_cgroup *memcg, vma_flags_t *vma_flags); void try_to_migrate(struct folio *folio, enum ttu_flags flags); void try_to_unmap(struct folio *, enum ttu_flags flags); @@ -977,10 +977,9 @@ struct anon_vma *folio_lock_anon_vma_read(const struct folio *folio, #define anon_vma_prepare(vma) (0) static inline int folio_referenced(struct folio *folio, int is_locked, - struct mem_cgroup *memcg, - vm_flags_t *vm_flags) + struct mem_cgroup *memcg, vma_flags_t *vma_flags) { - *vm_flags = 0; + vma_flags_clear_all(vma_flags); return 0; } diff --git a/mm/rmap.c b/mm/rmap.c index 34ceeb600111..14f2f9b07572 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -908,7 +908,7 @@ out: struct folio_referenced_arg { int mapcount; int referenced; - vm_flags_t vm_flags; + vma_flags_t vma_flags; struct mem_cgroup *memcg; }; @@ -927,7 +927,7 @@ static bool folio_referenced_one(struct folio *folio, address = pvmw.address; nr = 1; - if (vma->vm_flags & VM_LOCKED) { + if (vma_test(vma, VMA_LOCKED_BIT)) { ptes++; pra->mapcount--; @@ -948,7 +948,7 @@ static bool folio_referenced_one(struct folio *folio, /* Restore the mlock which got missed */ mlock_vma_folio(folio, vma); page_vma_mapped_walk_done(&pvmw); - pra->vm_flags |= VM_LOCKED; + vma_flags_set(&pra->vma_flags, VMA_LOCKED_BIT); return false; /* To break the loop */ } @@ -1016,8 +1016,11 @@ static bool folio_referenced_one(struct folio *folio, referenced++; if (referenced) { + vma_flags_t vma_flags = vma->flags; + pra->referenced++; - pra->vm_flags |= vma->vm_flags & ~VM_LOCKED; + vma_flags_clear(&vma_flags, VMA_LOCKED_BIT); + vma_flags_set_mask(&pra->vma_flags, vma_flags); } if (!pra->mapcount) @@ -1055,7 +1058,7 @@ static bool invalid_folio_referenced_vma(struct vm_area_struct *vma, void *arg) * @folio: The folio to test. * @is_locked: Caller holds lock on the folio. * @memcg: target memory cgroup - * @vm_flags: A combination of all the vma->vm_flags which referenced the folio. + * @vma_flags: A combination of all the vma->flags which referenced the folio. * * Quick test_and_clear_referenced for all mappings of a folio, * @@ -1063,7 +1066,7 @@ static bool invalid_folio_referenced_vma(struct vm_area_struct *vma, void *arg) * the function bailed out due to rmap lock contention. */ int folio_referenced(struct folio *folio, int is_locked, - struct mem_cgroup *memcg, vm_flags_t *vm_flags) + struct mem_cgroup *memcg, vma_flags_t *vma_flags) { bool we_locked = false; struct folio_referenced_arg pra = { @@ -1079,7 +1082,7 @@ int folio_referenced(struct folio *folio, int is_locked, }; VM_WARN_ON_ONCE_FOLIO(folio_is_zone_device(folio), folio); - *vm_flags = 0; + vma_flags_clear_all(vma_flags); if (!pra.mapcount) return 0; @@ -1093,7 +1096,7 @@ int folio_referenced(struct folio *folio, int is_locked, } rmap_walk(folio, &rwc); - *vm_flags = pra.vm_flags; + vma_flags_set_mask(vma_flags, pra.vma_flags); if (we_locked) folio_unlock(folio); diff --git a/mm/vmscan.c b/mm/vmscan.c index 8bd0bea62767..206213e56ec8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -861,16 +861,16 @@ static enum folio_references folio_check_references(struct folio *folio, struct scan_control *sc) { int referenced_ptes, referenced_folio; - vm_flags_t vm_flags; + vma_flags_t vma_flags; referenced_ptes = folio_referenced(folio, 1, sc->target_mem_cgroup, - &vm_flags); + &vma_flags); /* * The supposedly reclaimable folio was found to be in a VM_LOCKED vma. * Let the folio, now marked Mlocked, be moved to the unevictable list. */ - if (vm_flags & VM_LOCKED) + if (vma_flags_test(&vma_flags, VMA_LOCKED_BIT)) return FOLIOREF_ACTIVATE; /* @@ -914,7 +914,7 @@ static enum folio_references folio_check_references(struct folio *folio, /* * Activate file-backed executable folios after first usage. */ - if ((vm_flags & VM_EXEC) && folio_is_file_lru(folio)) + if (vma_flags_test(&vma_flags, VMA_EXEC_BIT) && folio_is_file_lru(folio)) return FOLIOREF_ACTIVATE; return FOLIOREF_KEEP; @@ -2065,7 +2065,7 @@ static void shrink_active_list(unsigned long nr_to_scan, { unsigned long nr_taken; unsigned long nr_scanned; - vm_flags_t vm_flags; + vma_flags_t vma_flags; LIST_HEAD(l_hold); /* The folios which were snipped off */ LIST_HEAD(l_active); LIST_HEAD(l_inactive); @@ -2109,7 +2109,7 @@ static void shrink_active_list(unsigned long nr_to_scan, /* Referenced or rmap lock contention: rotate */ if (folio_referenced(folio, 0, sc->target_mem_cgroup, - &vm_flags) != 0) { + &vma_flags) != 0) { /* * Identify referenced, file-backed active folios and * give them one more trip around the active list. So @@ -2119,7 +2119,7 @@ static void shrink_active_list(unsigned long nr_to_scan, * IO, plus JVM can create lots of anon VM_EXEC folios, * so we ignore them here. */ - if ((vm_flags & VM_EXEC) && folio_is_file_lru(folio)) { + if (vma_flags_test(&vma_flags, VMA_EXEC_BIT) && folio_is_file_lru(folio)) { nr_rotated += folio_nr_pages(folio); list_add(&folio->lru, &l_active); continue; -- cgit v1.2.3 From 1b7c8fe294a6bf913832e998f519029cad47dbcb Mon Sep 17 00:00:00 2001 From: Ridong Chen Date: Thu, 23 Jul 2026 11:24:33 +0800 Subject: memcg: move mem_cgroup_swappiness and vm_swappiness to mm/swap.h Patch series "mm: vmscan: fix node reclaim ignoring swappiness parameter", v4. The per-node proactive reclaim interface (/sys/devices/system/node/nodeX/reclaim) accepts a swappiness parameter, but it is silently ignored when CONFIG_MEMCG is disabled. The root cause is that sc_swappiness() has separate implementations for CONFIG_MEMCG and !CONFIG_MEMCG, and the latter never checks proactive_swappiness. Patch 1 moves mem_cgroup_swappiness() and vm_swappiness out of the public include/linux/swap.h into the mm-private mm/swap.h, and makes the helper handle both CONFIG_MEMCG and !CONFIG_MEMCG in a single inline function. This is a prerequisite for unifying sc_swappiness(). Patch 2 consolidates sc_swappiness() into a single definition that works regardless of CONFIG_MEMCG, fixing the node reclaim swappiness bug. This patch (of 2): The per-memcg swappiness knob is v1-only; v2 always uses global vm_swappiness and ignores the per-cgroup field. Both mem_cgroup_swappiness() and vm_swappiness are only used within mm/ (memcontrol.c, memcontrol-v1.c, vmscan.c), so move them out of the public include/linux/swap.h into the mm-private mm/swap.h. This keeps unrelated declarations out of include/linux/swap.h. Guard memcg->swappiness with CONFIG_MEMCG_V1 as well, so v2-only kernels drop the unused field. No functional change for v1; v2-only kernels drop the unused field. Link: https://lore.kernel.org/20260723032434.2016749-1-ridong.chen@linux.dev Link: https://lore.kernel.org/20260723032434.2016749-2-ridong.chen@linux.dev Signed-off-by: Ridong Chen Acked-by: Johannes Weiner Reviewed-by: Barry Song Reviewed-by: Song Hu Acked-by: Shakeel Butt Cc: Axel Rasmussen Cc: Baoquan He Cc: Chris Li Cc: David Hildenbrand Cc: Davidlohr Bueso Cc: Kairui Song Cc: Kemeng Shi Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Muchun Song Cc: Nhat Pham Cc: Roman Gushchin Cc: Wei Xu Cc: Yuanchu Xie Cc: Qi Zheng Signed-off-by: Andrew Morton --- include/linux/memcontrol.h | 4 ++-- include/linux/swap.h | 19 ------------------- mm/memcontrol.c | 4 ++-- mm/swap.h | 14 ++++++++++++++ 4 files changed, 18 insertions(+), 23 deletions(-) (limited to 'include') diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 8170bb8066a2..f619e24fc3bb 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -239,8 +239,6 @@ struct mem_cgroup { */ bool oom_group; - int swappiness; - /* memory.events and memory.events.local */ struct cgroup_file events_file; struct cgroup_file events_local_file; @@ -318,6 +316,8 @@ struct mem_cgroup { /* List of events which userspace want to receive */ struct list_head event_list; spinlock_t event_list_lock; + + int swappiness; #endif /* CONFIG_MEMCG_V1 */ struct mem_cgroup_per_node *nodeinfo[]; diff --git a/include/linux/swap.h b/include/linux/swap.h index 696ed01709c2..330a420fd6de 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -309,7 +309,6 @@ static inline bool lru_cache_disabled(void) } extern unsigned long shrink_all_memory(unsigned long nr_pages); -extern int vm_swappiness; long remove_mapping(struct address_space *mapping, struct folio *folio); #if defined(CONFIG_SYSFS) && defined(CONFIG_NUMA) @@ -468,25 +467,7 @@ static inline int add_swap_extent(struct swap_info_struct *sis, } #endif /* CONFIG_SWAP */ #ifdef CONFIG_MEMCG -static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg) -{ - /* Cgroup2 doesn't have per-cgroup swappiness */ - if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) - return READ_ONCE(vm_swappiness); - - /* root ? */ - if (mem_cgroup_disabled() || mem_cgroup_is_root(memcg)) - return READ_ONCE(vm_swappiness); - - return READ_ONCE(memcg->swappiness); -} - void lru_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent, int nid); -#else -static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg) -{ - return READ_ONCE(vm_swappiness); -} #endif #if defined(CONFIG_SWAP) && defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 69b37f63a307..ec92cb40156d 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -63,6 +63,7 @@ #include #include #include "internal.h" +#include "swap.h" #include "swap_table.h" #include #include @@ -4176,11 +4177,10 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) #endif page_counter_set_high(&memcg->swap, PAGE_COUNTER_MAX); if (parent) { - WRITE_ONCE(memcg->swappiness, mem_cgroup_swappiness(parent)); - page_counter_init(&memcg->memory, &parent->memory, memcg_on_dfl); page_counter_init(&memcg->swap, &parent->swap, false); #ifdef CONFIG_MEMCG_V1 + WRITE_ONCE(memcg->swappiness, mem_cgroup_swappiness(parent)); memcg->memory.track_failcnt = !memcg_on_dfl; memcg->memsw.track_failcnt = !memcg_on_dfl; WRITE_ONCE(memcg->oom_kill_disable, READ_ONCE(parent->oom_kill_disable)); diff --git a/mm/swap.h b/mm/swap.h index b51ad3071a73..2ccf8cf7f6c1 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -4,6 +4,8 @@ #include /* for atomic_long_t */ #include /* for PAGE_SHIFT */ +#include /* for mem_cgroup_swappiness() */ + struct mempolicy; struct swap_iocb; struct swap_memcg_table; @@ -76,6 +78,18 @@ enum swap_cluster_flags { CLUSTER_FLAG_MAX, }; +extern int vm_swappiness; + +static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg) +{ +#ifdef CONFIG_MEMCG_V1 + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys) && + !mem_cgroup_disabled() && !mem_cgroup_is_root(memcg)) + return READ_ONCE(memcg->swappiness); +#endif + return READ_ONCE(vm_swappiness); +} + #ifdef CONFIG_SWAP #include /* for swp_offset */ #include /* for bio_end_io_t */ -- cgit v1.2.3 From a69797fb36452865252f10c8ac9ef6781d07e3d7 Mon Sep 17 00:00:00 2001 From: Usama Arif Date: Mon, 27 Jul 2026 09:23:23 -0700 Subject: mm/vmstat, mm/memcontrol: add _monotonic vmstat readers Patch series "mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost", v5. The anon/file scan balance heuristic in get_scan_count() is fed by two scalars in struct lruvec (anon_cost, file_cost) that every reclaim producer updates under lruvec->lru_lock. The cost-recording work itself is trivial, but it both contends for and contributes to contention on lru_lock - which is often a contention point on memory-pressured workloads. Specifically: - shrink_inactive_list() re-acquires lru_lock at function exit just to call lru_note_cost_unlock_irq(). - shrink_active_list() does the same after rotation accounting. - workingset_refault() takes folio_lruvec_lock_irq() purely to record the refault cost. - prepare_scan_control() snapshots anon_cost/file_cost under lru_lock. - lru_note_cost_unlock_irq() itself walks parent_lruvec() and re-acquires lru_lock on every ancestor, multiplying the cost of every update by memcg-hierarchy depth. This series removes those producer-side acquisitions entirely. The rotation inputs become per-LRU PGROTATE_{ANON,FILE} vmstat counters. NR_VMSCAN_WRITE already captures reclaim-driven pageout at writeout(); charge it through lruvec_stat_mod_folio() so it is available per lruvec and aggregated through the memcg hierarchy. Reclaim does not submit filesystem folios for writeback from this path, so pageout contributes only to anon cost. WORKINGSET_RESTORE_* already captures the refault input. PGROTATE_* are also useful independently of scan balancing. They are cumulative base-page events, not unique-page counts. Classic inactive reclaim records scan work that does not produce immediate reclaim or demotion, while active reclaim records referenced executable file folios retained on the active list. MGLRU records initially isolated pages that remain unreclaimed after its retry passes. Read alongside pgscan_* and pgsteal_*, their deltas identify which LRU type is consuming reclaim CPU without producing immediate yield. Unlike the existing pgrotated event, they do not imply a move to the inactive-list tail. prepare_scan_control() reads the raw cost signals without lru_lock: anon = PGROTATE_ANON + (NR_VMSCAN_WRITE + WORKINGSET_RESTORE_ANON) * SWAP_CLUSTER_MAX file = PGROTATE_FILE + WORKINGSET_RESTORE_FILE * SWAP_CLUSTER_MAX It folds the deltas into a per-lruvec accumulator. A dedicated per-lruvec cost_lock, not touched by isolate_lru_folios(), move_folios_to_lru(), or folio_add_lru(), serialises the accumulator RMW and the lrusize/4 halving check. Hierarchy aggregation is implicit in rstat propagation, so the parent_lruvec() walk and the lru_reparent_memcg() cost-splice both disappear. Moving accumulation and decay to the reclaim side also improves the cost model across reclaim gaps. With producer-side decay, events that happen while reclaim is idle still age each other before reclaim ever samples the costs. If a workload refaults a large anon set and then a smaller file set before reclaim runs again, the later file activity can age the earlier anon activity out of the cost model. The new scheme observes the whole between-reclaim delta and decays anon and file proportionally, so the scan-balance history better represents what happened since the last reclaim pass. Trade-offs: - Cost reads see rstat-aggregated values that can lag until periodic / reader-triggered flushing. - Per-lruvec footprint grows by 4 unsigned longs + a spinlock (a struct lru_cost { count, last_rotated, last_io } per side), which is a small cost. - NR_VMSCAN_WRITE now also updates the folio's lruvec/memcg stat, adding memcg stat accounting to the reclaim writeout path while preserving the existing node-level total. == Numbers == Tested on a 176-core, 256 GB host. The benchmark drives sustained swap-out/refault inside a tight memcg using vm-scalability/usemem: usemem -n 16 --prealloc --prefault --random $((256*1024*1024)) run inside a two-level memcg with memory.max=512M on the leaf (4 GB anon working set has to fit in 512 MB -> continuous shrink_inactive_list + workingset_refault). A 16 GB swap file is used. Measurement is a 30 s `perf lock record -a` window over otherwise-idle hardware. Workload rates are identical on both kernels (the bench drives the same memory pressure): baseline patched delta pgscan_direct / s 172,662 171,817 ~0% pgsteal_direct / s 67,162 66,306 ~0% workingset_refault_anon / s 40,696 39,830 ~0% perf lock contention (total wait per 30 s window): Lock Name Before After % change shrink_lruvec+0x770 722.84 ms 0 -100% (eliminated) (= lru_note_cost_unlock_irq) workingset_refault+0x167 385.26 ms 0 -100% (eliminated) (= lru_note_cost_refault) shrink_node+0x4ad 689.43 ms 26.95 ms -96% shrink_active_list 208.34 ms 15.97 ms -92% lru_add_drain_cpu+0x34 1.96 s 917.71 ms -53% Total LRU lock wait ~4.23 s ~1.66 s -61% The two specific contention sites the patch removes (shrink_lruvec+0x770 = lru_note_cost_unlock_irq; workingset_refault+0x167 = lru_note_cost_refault) are completely absent from the patched perf-lock-contention output. Secondary reductions in shrink_node, shrink_active_list, lru_add_drain_cpu and pgrefill/pgactivate look like knock-on effects from removing the cost-recording overhead and the parent_lruvec walk. The remaining ~1.66 s of LRU lock wait on the patched kernel is dominated by the per-CPU pagevec drain (lru_add_drain_cpu) and the main reclaim path in shrink_lruvec. The numbers above can be reproduced using the script in [1]. This patch (of 3): lruvec_page_state(), node_page_state(), and global_node_page_state() all clamp negative reads to zero on CONFIG_SMP so that a transient per-CPU delta skew presents as zero pages rather than as a garbage unsigned value. This is the right behaviour for non-monotonic page-count readers. It is however incorrect for callers that snapshot a monotonically- incremented event counter and compute a delta from two samples. Once the underlying signed long wraps past LONG_MAX, the clamped read drops to zero while the previously-recorded snapshot still holds the pre-wrap value; the unsigned subtraction then underflows into a ~2^31 spurious delta for 32-bit architecture and corrupts the caller's accumulator. Add non-clamping siblings that return the underlying state value cast to unsigned long: global_node_page_state_monotonic() node_page_state_monotonic() lruvec_page_state_monotonic() With both samples read via the _monotonic variant, unsigned modular subtraction stays correct across a signed-long wraparound as long as the true growth between two samples fits in unsigned long (< 2^32 on 32-bit, < 2^64 on 64-bit); the 32-bit bound is the practically-reachable one that motivates this helper. The variants are only safe for monotonically-incremented counters. Non-monotonic page-count readers must keep using the existing clamped helpers so transient negative reads still present as zero. This is a prerequisite for a later patch which replaces the producer-side anon_cost/file_cost accumulators with a read-side accumulator in prepare_scan_control() that samples monotonic per-LRU vmstat counters (PGROTATE_*, NR_VMSCAN_WRITE, WORKINGSET_RESTORE_*) via lruvec_page_state_monotonic() and folds their unsigned modular deltas into lruvec->cost[].count. Link: https://lore.kernel.org/20260727162550.2032-1-usama.arif@linux.dev Link: https://lore.kernel.org/20260727162550.2032-2-usama.arif@linux.dev Link: https://gist.github.com/uarif1/a4eb33a86c5b2d7bbc55b42f0956e884 [1] Signed-off-by: Usama Arif Acked-by: Johannes Weiner Acked-by: Shakeel Butt Acked-by: Vlastimil Babka (SUSE) Cc: Axel Rasmussen Cc: Baoquan He Cc: Chris Li Cc: David Hildenbrand Cc: David Rientjes Cc: Kairui Song Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Nhat Pham Cc: Roman Gushchin Cc: Suren Baghdasaryan Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton --- include/linux/memcontrol.h | 8 ++++++++ include/linux/vmstat.h | 16 ++++++++++++++++ mm/memcontrol.c | 36 ++++++++++++++++++++++++++++++++++++ mm/vmstat.c | 11 +++++++++++ 4 files changed, 71 insertions(+) (limited to 'include') diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index f619e24fc3bb..e78bc98ab229 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -947,6 +947,8 @@ unsigned long memcg_page_state_output(struct mem_cgroup *memcg, int item); bool memcg_stat_item_valid(int idx); bool memcg_vm_event_item_valid(enum vm_event_item idx); unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx); +unsigned long lruvec_page_state_monotonic(struct lruvec *lruvec, + enum node_stat_item idx); unsigned long lruvec_page_state_local(struct lruvec *lruvec, enum node_stat_item idx); @@ -1399,6 +1401,12 @@ static inline unsigned long lruvec_page_state(struct lruvec *lruvec, return node_page_state(lruvec_pgdat(lruvec), idx); } +static inline unsigned long lruvec_page_state_monotonic(struct lruvec *lruvec, + enum node_stat_item idx) +{ + return node_page_state_monotonic(lruvec_pgdat(lruvec), idx); +} + static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, enum node_stat_item idx) { diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index 3c9c266cf782..fb8c76289e02 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -194,6 +194,19 @@ unsigned long global_node_page_state_pages(enum node_stat_item item) return x; } +/* + * Non-clamping variant of global_node_page_state() intended for callers that + * snapshot a monotonically-incremented counter and subtract two samples. + * Returns the raw wrapping value so that unsigned modular subtraction stays + * correct across a signed-long overflow (a real hazard on 32-bit) that the + * clamp in global_node_page_state() would otherwise turn into a huge spurious + * delta. Do NOT use for non-monotonic page-count reads. + */ +static inline unsigned long global_node_page_state_monotonic(enum node_stat_item item) +{ + return (unsigned long)atomic_long_read(&vm_node_stat[item]); +} + static inline unsigned long global_node_page_state(enum node_stat_item item) { VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); @@ -259,11 +272,14 @@ extern unsigned long node_page_state(struct pglist_data *pgdat, enum node_stat_item item); extern unsigned long node_page_state_pages(struct pglist_data *pgdat, enum node_stat_item item); +extern unsigned long node_page_state_monotonic(struct pglist_data *pgdat, + enum node_stat_item item); extern void fold_vm_numa_events(void); #else #define sum_zone_node_page_state(node, item) global_zone_page_state(item) #define node_page_state(node, item) global_node_page_state(item) #define node_page_state_pages(node, item) global_node_page_state_pages(item) +#define node_page_state_monotonic(node, item) global_node_page_state_monotonic(item) static inline void fold_vm_numa_events(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ec92cb40156d..d804f8d07581 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -508,6 +508,42 @@ unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx) return x; } +/** + * lruvec_page_state_monotonic - non-clamping lruvec stat read for delta sampling + * @lruvec: the LRU vector to read from + * @idx: the node_stat_item to read + * + * Returns the raw state[idx] value cast to unsigned long, skipping the + * clamp-negative-to-zero step in lruvec_page_state(). Intended for callers + * that snapshot a monotonically-incremented counter and subtract two + * samples: unsigned modular arithmetic then yields the correct delta across + * a signed-long wraparound (a real hazard on 32-bit) that the clamp would + * otherwise turn into a huge spurious delta. + * + * Do NOT use for non-monotonic page-count reads where a transient negative + * reading from per-CPU delta skew must present as zero. + * + * XXX: This helper (and its node/global peers) exists because some + * monotonically-incremented event counters are stored in + * enum node_stat_item. + */ +unsigned long lruvec_page_state_monotonic(struct lruvec *lruvec, + enum node_stat_item idx) +{ + struct mem_cgroup_per_node *pn; + int i; + + if (mem_cgroup_disabled()) + return node_page_state_monotonic(lruvec_pgdat(lruvec), idx); + + i = memcg_stats_index(idx); + if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx)) + return 0; + + pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); + return (unsigned long)READ_ONCE(pn->lruvec_stats->state[i]); +} + unsigned long lruvec_page_state_local(struct lruvec *lruvec, enum node_stat_item idx) { diff --git a/mm/vmstat.c b/mm/vmstat.c index 3b5cb1031f72..507118474c03 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1025,6 +1025,17 @@ unsigned long node_page_state(struct pglist_data *pgdat, return node_page_state_pages(pgdat, item); } + +/* + * Non-clamping variant of node_page_state() intended for callers that + * snapshot a monotonically-incremented counter and subtract two samples. + * See global_node_page_state_monotonic() for the rationale. + */ +unsigned long node_page_state_monotonic(struct pglist_data *pgdat, + enum node_stat_item item) +{ + return (unsigned long)atomic_long_read(&pgdat->vm_stat[item]); +} #endif /* -- cgit v1.2.3 From 1b089def0fb8833ec4b331b61908f29d6b491ccb Mon Sep 17 00:00:00 2001 From: Usama Arif Date: Mon, 27 Jul 2026 09:23:24 -0700 Subject: mm/vmscan: add pgrotate_anon and pgrotate_file vmstat counters Reclaim can spend substantial work on an LRU type without immediately reclaiming or demoting a corresponding amount of memory. Record this work in PGROTATE_ANON and PGROTATE_FILE. For classic LRU reclaim: - Inactive-list reclaim adds nr_scanned - nr_reclaimed to the corresponding anon/file counter when isolation succeeds. - Active-list reclaim adds referenced executable file folios that are retained on the active list to PGROTATE_FILE. Active anon reclaim does not contribute this component. For MGLRU, add the number of initially isolated pages that remain unreclaimed after both the initial and retry passes to the counter for the selected anon/file type. These counters are distinct from the existing pgrotated vm event. pgrotated records an actual move to the inactive-list tail, primarily after reclaim-marked writeback completes or failed invalidation leaves a folio for accelerated reclaim. PGROTATE_ANON and PGROTATE_FILE measure reclaim cost and do not imply that a folio moved to an LRU tail. A subsequent patch will consume these counters for anon/file scan balancing. Link: https://lore.kernel.org/20260727162550.2032-3-usama.arif@linux.dev Signed-off-by: Usama Arif Acked-by: Shakeel Butt Acked-by: Johannes Weiner Reviewed-by: Vlastimil Babka (SUSE) Cc: Axel Rasmussen Cc: Baoquan He Cc: Chris Li Cc: David Hildenbrand Cc: David Rientjes Cc: Kairui Song Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Nhat Pham Cc: Roman Gushchin Cc: Suren Baghdasaryan Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton --- include/linux/mmzone.h | 2 ++ mm/memcontrol.c | 2 ++ mm/vmscan.c | 14 +++++++++++++- mm/vmstat.c | 2 ++ 4 files changed, 19 insertions(+), 1 deletion(-) (limited to 'include') diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 0507193b3ae3..1cc5ea506b7c 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -323,6 +323,8 @@ enum node_stat_item { PGSCAN_PROACTIVE, PGSCAN_ANON, PGSCAN_FILE, + PGROTATE_ANON, + PGROTATE_FILE, PGREFILL, #ifdef CONFIG_HUGETLB_PAGE NR_HUGETLB, diff --git a/mm/memcontrol.c b/mm/memcontrol.c index d804f8d07581..011bb3b0346b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -425,6 +425,8 @@ static const unsigned int memcg_node_stat_items[] = { PGSCAN_PROACTIVE, PGSCAN_ANON, PGSCAN_FILE, + PGROTATE_ANON, + PGROTATE_FILE, PGREFILL, #ifdef CONFIG_HUGETLB_PAGE NR_HUGETLB, diff --git a/mm/vmscan.c b/mm/vmscan.c index 5bad065a38f1..fa2c5cf577af 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2038,6 +2038,9 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, item = PGSTEAL_KSWAPD + reclaimer_offset(sc); mod_lruvec_state(lruvec, item, nr_reclaimed); mod_lruvec_state(lruvec, PGSTEAL_ANON + file, nr_reclaimed); + if (nr_scanned > nr_reclaimed) + mod_lruvec_state(lruvec, PGROTATE_ANON + file, + nr_scanned - nr_reclaimed); lruvec_lock_irq(lruvec); lru_note_cost_unlock_irq(lruvec, file, stat.nr_pageout, @@ -2147,6 +2150,8 @@ static void shrink_active_list(unsigned long nr_to_scan, count_vm_events(PGDEACTIVATE, nr_deactivate); count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate); mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); + if (nr_rotated) + mod_lruvec_state(lruvec, PGROTATE_ANON + file, nr_rotated); lruvec_lock_irq(lruvec); lru_note_cost_unlock_irq(lruvec, file, 0, nr_rotated); @@ -4828,7 +4833,8 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, struct reclaim_stat stat; struct lru_gen_mm_walk *walk; int scanned, reclaimed; - int isolated = 0, type, type_scanned; + int isolated = 0, nr_isolated = 0, type, type_scanned; + unsigned long total_reclaimed = 0; bool skip_retry = false; struct mem_cgroup *memcg = lruvec_memcg(lruvec); struct pglist_data *pgdat = lruvec_pgdat(lruvec); @@ -4840,6 +4846,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, scanned = isolate_folios(nr_to_scan, lruvec, sc, swappiness, &list, &isolated, &type, &type_scanned); + nr_isolated = isolated; /* Scanning may have emptied the oldest gen, flush it */ if (scanned) @@ -4852,6 +4859,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, retry: reclaimed = shrink_folio_list(&list, pgdat, sc, &stat, false, memcg); sc->nr_reclaimed += reclaimed; + total_reclaimed += reclaimed; /* Retry pass is only meant for clean folios without new isolation */ if (isolated) handle_reclaim_writeback(isolated, pgdat, sc, &stat); @@ -4903,6 +4911,10 @@ retry: goto retry; } + if (nr_isolated > total_reclaimed) + mod_lruvec_state(lruvec, PGROTATE_ANON + type, + nr_isolated - total_reclaimed); + return scanned; } diff --git a/mm/vmstat.c b/mm/vmstat.c index 507118474c03..7d6e61a01f51 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1301,6 +1301,8 @@ const char * const vmstat_text[] = { [I(PGSCAN_PROACTIVE)] = "pgscan_proactive", [I(PGSCAN_ANON)] = "pgscan_anon", [I(PGSCAN_FILE)] = "pgscan_file", + [I(PGROTATE_ANON)] = "pgrotate_anon", + [I(PGROTATE_FILE)] = "pgrotate_file", [I(PGREFILL)] = "pgrefill", #ifdef CONFIG_HUGETLB_PAGE [I(NR_HUGETLB)] = "nr_hugetlb", -- cgit v1.2.3 From 7b9f4e5f81013bcb0d16bf54b349ee323ce0ac01 Mon Sep 17 00:00:00 2001 From: Usama Arif Date: Mon, 27 Jul 2026 09:23:25 -0700 Subject: mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost The anon/file scan balance in get_scan_count() is driven by two scalars in struct lruvec, anon_cost and file_cost, accumulated by every reclaim producer under lruvec->lru_lock. The acquisition sites for cost work specifically are: - shrink_inactive_list() re-takes lru_lock at function exit purely to call lru_note_cost_unlock_irq() with (nr_pageout, nr_scanned - nr_reclaimed). One acquisition per inactive shrink. - shrink_active_list() does the same with (0, nr_rotated). One acquisition per active shrink. - workingset_refault() takes the lock via folio_lruvec_lock_irq() purely to record the refault cost. One acquisition per refault. - prepare_scan_control() takes lru_lock just to snapshot the two scalars into sc->{anon,file}_cost. - lru_note_cost_unlock_irq() itself walks parent_lruvec and re-acquires lru_lock on each ancestor to propagate the update, adding O(memcg-depth) acquisitions per producer call. This hurts because lru_lock is already a heavy contention point on memory-heavy workloads: every isolate_lru_folios(), move_folios_to_lru() and folio_add_lru() takes it. The cost work itself is trivial (two scalar bumps and one comparison), but it contends with and causes contention for actual LRU manipulation. The parent_lruvec() walk also multiplies cost-update overhead by memcg hierarchy depth. The balance formula for anon and file, respectively, is this: cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated Instead of recording cost and running averaging logic directly when these events occur, snapshot running vmstat counters once per reclaim cycle and derive the balance from event deltas since the last run. Use PGROTATE_* from the preceding patch for the rotation input. WORKINGSET_RESTORE_* and NR_VMSCAN_WRITE provide the remaining event counters. Charge NR_VMSCAN_WRITE through lruvec stats so all inputs can be sampled per lruvec and aggregated through the memcg hierarchy. This is overall cheaper and has fewer lock acquisition sites. Moving accumulation and decay to the reclaim side also improves the cost model across reclaim gaps. With producer-side decay, events that happen while reclaim is idle still age each other before reclaim ever samples the costs. If a workload refaults a large anon set and then a smaller file set before reclaim runs again, the later file activity can age the earlier anon activity out of the cost model. The new scheme observes the whole between-reclaim delta and decays anon and file proportionally, so the scan-balance history better represents what happened since the last reclaim pass. A dedicated per-lruvec spinlock, cost_lock, serialises the delta extraction, the cost->count update and the halving loop against concurrent reclaimers in the same memcg+node. NR_VMSCAN_WRITE is accounted at writeout(), so reclaim_stat.nr_pageout is no longer needed and is removed. memcg-v1's memory.stat anon_cost/file_cost is now sourced from cost[].count instead of the removed lruvec anon_cost/file_cost fields. The reported values only refresh when prepare_scan_control() runs and are bounded at ~lrusize/4 by the halving loop; the scan-balance signal they express is unchanged. Under pure MGLRU the scan-balance signal itself is not consumed (both prepare_scan_control() and get_scan_count() are short-circuited on the MGLRU paths, and MGLRU's own type/tier selection comes from read_ctrl_pos() on lrugen->{avg_refaulted,avg_total,refaulted,evicted}, not from anon_cost/file_cost). NR_VMSCAN_WRITE naturally covers writeout from either reclaim implementation. The preceding patch also bumps PGROTATE_{ANON,FILE} from evict_folios(), so rotation-driven reclaim work is accounted consistently across both implementations. Link: https://lore.kernel.org/20260727162550.2032-4-usama.arif@linux.dev Signed-off-by: Usama Arif Acked-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Vlastimil Babka (SUSE) Cc: Axel Rasmussen Cc: Baoquan He Cc: Chris Li Cc: David Hildenbrand Cc: David Rientjes Cc: Kairui Song Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Nhat Pham Cc: Roman Gushchin Cc: Suren Baghdasaryan Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton --- include/linux/mmzone.h | 13 +++++++-- include/linux/vmstat.h | 1 - mm/folio.c | 69 ------------------------------------------- mm/internal.h | 3 -- mm/memcontrol-v1.c | 4 +-- mm/memcontrol.c | 1 + mm/mmzone.c | 1 + mm/vmscan.c | 79 +++++++++++++++++++++++++++++++++++++++++--------- mm/workingset.c | 5 ---- 9 files changed, 81 insertions(+), 95 deletions(-) (limited to 'include') diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 1cc5ea506b7c..158c1fba2393 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -757,6 +757,12 @@ void lru_gen_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent, #endif /* CONFIG_LRU_GEN */ +struct lru_cost { + unsigned long count; + unsigned long last_rotated; + unsigned long last_io; +}; + struct lruvec { struct list_head lists[NR_LRU_LISTS]; /* per lruvec lru_lock for memcg */ @@ -765,9 +771,12 @@ struct lruvec { * These track the cost of reclaiming one LRU - file or anon - * over the other. As the observed cost of reclaiming one LRU * increases, the reclaim scan balance tips toward the other. + * Updated and decayed at prepare_scan_control() time; cost_lock + * serialises that update. */ - unsigned long anon_cost; - unsigned long file_cost; + struct lru_cost cost[ANON_AND_FILE]; + /* Protects cost[]. */ + spinlock_t cost_lock; /* Non-resident age, driven by LRU movement */ atomic_long_t nonresident_age; /* Refaults at the time of last reclaim cycle */ diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index fb8c76289e02..5b31d8e7ae40 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -20,7 +20,6 @@ struct reclaim_stat { unsigned nr_congested; unsigned nr_writeback; unsigned nr_immediate; - unsigned nr_pageout; unsigned nr_activate[ANON_AND_FILE]; unsigned nr_ref_keep; unsigned nr_unmap_fail; diff --git a/mm/folio.c b/mm/folio.c index d2937600cf72..a9e328c3f21b 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -265,73 +265,6 @@ void folio_rotate_reclaimable(struct folio *folio) folio_batch_add_and_move(folio, lru_move_tail); } -void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file, - unsigned int nr_io, unsigned int nr_rotated) - __releases(lruvec->lru_lock) - __releases(rcu) -{ - unsigned long cost; - - /* - * Reflect the relative cost of incurring IO and spending CPU - * time on rotations. This doesn't attempt to make a precise - * comparison, it just says: if reloads are about comparable - * between the LRU lists, or rotations are overwhelmingly - * different between them, adjust scan balance for CPU work. - */ - cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated; - if (!cost) { - spin_unlock_irq(&lruvec->lru_lock); - rcu_read_unlock(); - return; - } - - for (;;) { - unsigned long lrusize; - - /* Record cost event */ - if (file) - lruvec->file_cost += cost; - else - lruvec->anon_cost += cost; - - /* - * Decay previous events - * - * Because workloads change over time (and to avoid - * overflow) we keep these statistics as a floating - * average, which ends up weighing recent refaults - * more than old ones. - */ - lrusize = lruvec_page_state(lruvec, NR_INACTIVE_ANON) + - lruvec_page_state(lruvec, NR_ACTIVE_ANON) + - lruvec_page_state(lruvec, NR_INACTIVE_FILE) + - lruvec_page_state(lruvec, NR_ACTIVE_FILE); - - if (lruvec->file_cost + lruvec->anon_cost > lrusize / 4) { - lruvec->file_cost /= 2; - lruvec->anon_cost /= 2; - } - - spin_unlock_irq(&lruvec->lru_lock); - lruvec = parent_lruvec(lruvec); - if (!lruvec) { - rcu_read_unlock(); - break; - } - spin_lock_irq(&lruvec->lru_lock); - } -} - -void lru_note_cost_refault(struct folio *folio) -{ - struct lruvec *lruvec; - - lruvec = folio_lruvec_lock_irq(folio); - lru_note_cost_unlock_irq(lruvec, folio_is_file_lru(folio), - folio_nr_pages(folio), 0); -} - static void lru_activate(struct lruvec *lruvec, struct folio *folio) { long nr_pages = folio_nr_pages(folio); @@ -1162,8 +1095,6 @@ void lru_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent, int child_lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid)); parent_lruvec = mem_cgroup_lruvec(parent, NODE_DATA(nid)); - parent_lruvec->anon_cost += child_lruvec->anon_cost; - parent_lruvec->file_cost += child_lruvec->file_cost; for_each_lru(lru) lruvec_reparent_lru(child_lruvec, parent_lruvec, lru, nid); diff --git a/mm/internal.h b/mm/internal.h index 16750b130ec4..07f60ca0b201 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -41,9 +41,6 @@ void workingset_refault(struct folio *folio, void *shadow); void workingset_activation(struct folio *folio); /* mm/folio.c */ -void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file, - unsigned int nr_io, unsigned int nr_rotated); -void lru_note_cost_refault(struct folio *folio); void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma); static inline bool folio_may_be_lru_cached(struct folio *folio) diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c index 2dc599484d00..835fc8e51184 100644 --- a/mm/memcontrol-v1.c +++ b/mm/memcontrol-v1.c @@ -2287,8 +2287,8 @@ void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) for_each_online_pgdat(pgdat) { mz = memcg->nodeinfo[pgdat->node_id]; - anon_cost += mz->lruvec.anon_cost; - file_cost += mz->lruvec.file_cost; + anon_cost += mz->lruvec.cost[WORKINGSET_ANON].count; + file_cost += mz->lruvec.cost[WORKINGSET_FILE].count; } seq_buf_printf(s, "anon_cost %lu\n", anon_cost); seq_buf_printf(s, "file_cost %lu\n", file_cost); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 011bb3b0346b..65057b59b097 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -399,6 +399,7 @@ static const unsigned int memcg_node_stat_items[] = { NR_SHMEM_THPS, NR_FILE_THPS, NR_ANON_THPS, + NR_VMSCAN_WRITE, NR_VMALLOC, NR_KERNEL_STACK_KB, NR_PAGETABLE, diff --git a/mm/mmzone.c b/mm/mmzone.c index 59dc3f2076a6..9cc9ef588580 100644 --- a/mm/mmzone.c +++ b/mm/mmzone.c @@ -79,6 +79,7 @@ void lruvec_init(struct lruvec *lruvec) memset(lruvec, 0, sizeof(struct lruvec)); spin_lock_init(&lruvec->lru_lock); + spin_lock_init(&lruvec->cost_lock); zswap_lruvec_state_init(lruvec); for_each_lru(lru) diff --git a/mm/vmscan.c b/mm/vmscan.c index fa2c5cf577af..3a6701143620 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -670,7 +670,7 @@ static pageout_t pageout(struct folio *folio, struct address_space *mapping, folio_clear_reclaim(folio); trace_mm_vmscan_write_folio(folio); - node_stat_add_folio(folio, NR_VMSCAN_WRITE); + lruvec_stat_mod_folio(folio, NR_VMSCAN_WRITE, folio_nr_pages(folio)); return PAGE_SUCCESS; } @@ -1413,8 +1413,6 @@ retry: sc->nr_scanned -= (nr_pages - 1); nr_pages = 1; } - stat->nr_pageout += nr_pages; - if (folio_test_writeback(folio)) goto keep; if (folio_test_dirty(folio)) @@ -2042,9 +2040,6 @@ static unsigned long shrink_inactive_list(unsigned long nr_to_scan, mod_lruvec_state(lruvec, PGROTATE_ANON + file, nr_scanned - nr_reclaimed); - lruvec_lock_irq(lruvec); - lru_note_cost_unlock_irq(lruvec, file, stat.nr_pageout, - nr_scanned - nr_reclaimed); handle_reclaim_writeback(nr_taken, pgdat, sc, &stat); trace_mm_vmscan_lru_shrink_inactive(pgdat->node_id, nr_scanned, nr_reclaimed, &stat, sc->priority, file); @@ -2153,8 +2148,6 @@ static void shrink_active_list(unsigned long nr_to_scan, if (nr_rotated) mod_lruvec_state(lruvec, PGROTATE_ANON + file, nr_rotated); - lruvec_lock_irq(lruvec); - lru_note_cost_unlock_irq(lruvec, file, 0, nr_rotated); trace_mm_vmscan_lru_shrink_active(pgdat->node_id, nr_taken, nr_activate, nr_deactivate, nr_rotated, sc->priority, file); } @@ -2287,8 +2280,10 @@ enum scan_balance { static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) { - unsigned long file; + struct lru_cost *anon_cost, *file_cost; struct lruvec *target_lruvec; + unsigned long lrusize; + unsigned long file; if (lru_gen_enabled() && !lru_gen_switching()) return; @@ -2304,11 +2299,69 @@ static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) /* * Determine the scan balance between anon and file LRUs. + * + * The cost model is based on rotations, refaults and + * reclaim-driven writes (anon only) on each side. + * + * These event counters are monotonic, so each reclaim cycle + * the delta since the last scan is extracted and incorporated + * into a decaying average. This ensures currency, as workloads + * change over time, and avoids overflow in the calculations. + * + * Use lruvec_page_state_monotonic() so unsigned subtraction + * yields the correct delta across a signed-long wraparound of + * the underlying counter (a real hazard on 32-bit that the + * clamp in lruvec_page_state() would otherwise turn into a huge + * spurious delta). */ - spin_lock_irq(&target_lruvec->lru_lock); - sc->anon_cost = target_lruvec->anon_cost; - sc->file_cost = target_lruvec->file_cost; - spin_unlock_irq(&target_lruvec->lru_lock); + spin_lock(&target_lruvec->cost_lock); + + for (int f = 0; f <= 1; f++) { + struct lru_cost *cost = &target_lruvec->cost[f]; + unsigned long rotated, io, nr_rotated, nr_io; + + rotated = lruvec_page_state_monotonic(target_lruvec, + PGROTATE_ANON + f); + io = lruvec_page_state_monotonic(target_lruvec, + WORKINGSET_RESTORE_BASE + f); + if (f == WORKINGSET_ANON) + io += lruvec_page_state_monotonic(target_lruvec, + NR_VMSCAN_WRITE); + + nr_rotated = rotated - cost->last_rotated; + nr_io = io - cost->last_io; + + /* + * Reflect the relative cost of incurring IO and spending + * CPU time on rotations. This doesn't attempt to make a + * precise comparison, it just says: if reloads are about + * comparable between the LRU lists, or rotations are + * overwhelmingly different between them, adjust scan + * balance for CPU work. + */ + cost->count += nr_io * SWAP_CLUSTER_MAX + nr_rotated; + + cost->last_rotated = rotated; + cost->last_io = io; + } + + anon_cost = &target_lruvec->cost[WORKINGSET_ANON]; + file_cost = &target_lruvec->cost[WORKINGSET_FILE]; + + lrusize = lruvec_page_state(target_lruvec, NR_INACTIVE_ANON) + + lruvec_page_state(target_lruvec, NR_ACTIVE_ANON) + + lruvec_page_state(target_lruvec, NR_INACTIVE_FILE) + + lruvec_page_state(target_lruvec, NR_ACTIVE_FILE); + + while (anon_cost->count + file_cost->count > lrusize / 4) { + anon_cost->count /= 2; + file_cost->count /= 2; + } + + sc->anon_cost = anon_cost->count; + sc->file_cost = file_cost->count; + + spin_unlock(&target_lruvec->cost_lock); /* * Target desirable inactive:active list ratios for the anon diff --git a/mm/workingset.c b/mm/workingset.c index f351798e723a..7ac2b88c80ae 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -584,11 +584,6 @@ void workingset_refault(struct folio *folio, void *shadow) /* Folio was active prior to eviction */ if (workingset) { folio_set_workingset(folio); - /* - * XXX: Move to folio_add_lru() when it supports new vs - * putback - */ - lru_note_cost_refault(folio); mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + file, nr); } out: -- cgit v1.2.3 From fc6415a384f026f48b414a48588c0970c060679f Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Thu, 6 Aug 2026 20:09:06 +0200 Subject: mm/gup: factor out LRU cache draining for folio into lru_cache_drain_for_folio() KVM with guest_memfd wants to remove any folio references due to LRU caches, as it really must only allow to convert folios from shared to private when there are no unexpected folio references (e.g., from GUP references). So, to drive the refcount down, it needs a way to flush the LRU caches. Let's factor out what we have in lru_cache_drain_for_folio(). Document it, and also mention that concurrent folio (un)mapping might, in theory, miss detecting LRU cache references. Keep obtaining the expected refcount twice to minimize the possibility. For the current and future user that should work, and we don't really have a better alternative: we could detect if the mapcount changed, but it would still be racy and add more complexity with questionable benefit. Maybe there is a chance to avoid the draining entirely in the future, by avoiding extra references from the LRU cache: Hugh thinks there might be a way. But for the time being, this handling is unfortunately required. Make folio_may_be_lru_cached() accept a const pointer so lru_cache_drain_for_folio() can accept a const pointer as well. Link: https://lore.kernel.org/20260806-lru_cache_drain_for_folio-v1-1-c6287d295e99@kernel.org Signed-off-by: David Hildenbrand (Arm) Reviewed-by: Fuad Tabba Cc: Ackerley Tng Cc: Baoquan He Cc: Barry Song Cc: Chris Li Cc: Jason Gunthorpe Cc: John Hubbard Cc: Kairui Song Cc: Kemeng Shi Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Nhat Pham Cc: Peter Xu Cc: Sean Christopherson Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- include/linux/swap.h | 8 ++++++++ mm/folio.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ mm/gup.c | 15 ++------------- mm/internal.h | 2 +- 4 files changed, 57 insertions(+), 14 deletions(-) (limited to 'include') diff --git a/include/linux/swap.h b/include/linux/swap.h index 330a420fd6de..b4b1c0a84c8b 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -298,6 +298,14 @@ void folio_add_lru(struct folio *folio); void folio_mark_accessed(struct folio *folio); void lru_add_drain_all(void); +enum lru_cache_drained { + LRU_CACHE_NOT_DRAINED, + LRU_CACHE_DRAINED, + LRU_CACHE_DRAINED_ALL, +}; +void lru_cache_drain_for_folio(const struct folio *folio, + unsigned int extra_refs, enum lru_cache_drained *drained); + /* linux/mm/folio-compat.c */ void mark_page_accessed(struct page *page); diff --git a/mm/folio.c b/mm/folio.c index a9e328c3f21b..59c477120b9a 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -881,6 +881,52 @@ void lru_add_drain_all(void) } #endif /* CONFIG_SMP */ +/** + * lru_cache_drain_for_folio() - drain LRU caches if the caches might hold + * folio references + * @folio: The folio. + * @extra_refs: Extra folio references held by the caller. + * @drained: Drain status for batch folio processing. + * + * Drain LRU caches if the caches might hold folio references. Start + * with a local LRU cache drain, to then drain LRU caches on all CPUs if + * local draining was insufficient. + * + * This function detects LRU cache references by comparing the folio refcount + * with the sum of the expected folio refcount + extra references held by the + * caller. Note that we cannot rely on PG_lru to reliably detect all LRU + * cache references, and there are rare scenarios (concurrent folio (un)mapping) + * where this function might miss detecting LRU cache references. + * + * If @drained is not NULL, the function will avoid re-draining LRU caches + * when processing multiple folios in a row. In that case, the variable + * @drained points at must be initialized to LRU_CACHE_NOT_DRAINED before + * the first invocation by the caller. + */ +void lru_cache_drain_for_folio(const struct folio *folio, + unsigned int extra_refs, enum lru_cache_drained *drained) +{ + if (!folio_may_be_lru_cached(folio)) + return; + + if (!drained || *drained == LRU_CACHE_NOT_DRAINED) { + if (folio_ref_count(folio) == + folio_expected_ref_count(folio) + extra_refs) + return; + lru_add_drain(); + if (drained) + *drained = LRU_CACHE_DRAINED; + } + if (!drained || *drained == LRU_CACHE_DRAINED) { + if (folio_ref_count(folio) == + folio_expected_ref_count(folio) + extra_refs) + return; + lru_add_drain_all(); + if (drained) + *drained = LRU_CACHE_DRAINED_ALL; + } +} + atomic_t lru_disable_count = ATOMIC_INIT(0); /* diff --git a/mm/gup.c b/mm/gup.c index bce275c7dbb6..eb898ea1ee22 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2266,9 +2266,9 @@ static unsigned long collect_longterm_unpinnable_folios( struct list_head *movable_folio_list, struct pages_or_folios *pofs) { + enum lru_cache_drained drained = LRU_CACHE_NOT_DRAINED; unsigned long collected = 0; struct folio *folio; - int drained = 0; long i = 0; for (folio = pofs_get_folio(pofs, i); folio; @@ -2293,18 +2293,7 @@ static unsigned long collect_longterm_unpinnable_folios( * but also to remove any other folio references from LRU * caches. */ - if (drained == 0 && folio_may_be_lru_cached(folio) && - folio_ref_count(folio) != - folio_expected_ref_count(folio) + pin_refs) { - lru_add_drain(); - drained = 1; - } - if (drained == 1 && folio_may_be_lru_cached(folio) && - folio_ref_count(folio) != - folio_expected_ref_count(folio) + pin_refs) { - lru_add_drain_all(); - drained = 2; - } + lru_cache_drain_for_folio(folio, pin_refs, &drained); if (!folio_isolate_lru(folio)) continue; diff --git a/mm/internal.h b/mm/internal.h index 07f60ca0b201..38b1165212c9 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -43,7 +43,7 @@ void workingset_activation(struct folio *folio); /* mm/folio.c */ void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma); -static inline bool folio_may_be_lru_cached(struct folio *folio) +static inline bool folio_may_be_lru_cached(const struct folio *folio) { /* * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting. -- cgit v1.2.3 From 272b0d84b17f72f6396254dbaa6264f2f74a7997 Mon Sep 17 00:00:00 2001 From: Artem Lytkin Date: Sat, 1 Aug 2026 14:49:15 +0300 Subject: mm/vmalloc: make vm_struct.nr_pages an unsigned long vm_struct::nr_pages is an unsigned int, and the file keeps deriving byte counts from it as nr_pages << PAGE_SHIFT. A shift is evaluated in the type of its promoted left operand, so those are 32-bit arithmetic and wrap at 4 GiB of bytes, which is 2^20 pages. Every site depends on a cast being remembered; vmap() has one, two recent commits did not. vread_iter() then computes a size of zero for a 4 GiB VM_ALLOC area and /proc/kcore returns it as zeros while reporting a successful read, which drgn, crash or gdb cannot tell from real memory, and the vrealloc() grow-in-place check declines a request that would have fit. Widen the field so the class of bug goes away instead of one site at a time. Everything feeding or consuming it widens too: vm_area_alloc_pages() and its accumulators, nr_small_pages, new_nr_pages and old_nr_pages, the index range of vm_area_free_pages(), and three page indexes that were plain int. Five casts go. Two prints needed fixing as well, %u in vmalloc_dump_obj() and %d for the unsigned field in vmalloc_info_show(). No bug report behind this, I found it reading the code. The 4 GiB wrap needs only a machine with over 4 GiB of memory. Neither larger threshold is a practical concern: 2^32 pages, where the field itself truncates, is 16 TiB and beyond what hardware can populate, and 2^31, where the plain int indexes break, is 8 TiB and larger than anything in the tree asks for. The int *nr cursor in the mapping path is unchanged and is separate work. Users outside mm/vmalloc.c need no change either. Those handing the count to a narrower parameter cannot drive it near 2^31, and kho_preserve_vmalloc() stores it into a 32-bit ABI field that still receives the same low bits; above 2^32 pages the truncation just moves out of vm_struct into that store. sizeof(struct vm_struct) on x86-64 stays 72 bytes with CONFIG_HAVE_ARCH_HUGE_VMALLOC=n and goes from 72 to 80 with it enabled, both inside the kmalloc-96 bucket it already comes from. Link: https://lore.kernel.org/20260801114915.115224-1-iprintercanon@gmail.com Fixes: 0bca23804632 ("mm/vmalloc: use physical page count in vread_iter() for VM_ALLOC areas") Fixes: d57ac904ffdc ("mm/vmalloc: use physical page count for vrealloc() grow-in-place check") Signed-off-by: Artem Lytkin Suggested-by: Andrew Morton Reviewed-by: Uladzislau Rezki (Sony) Assisted-by: Claude:claude-fable-5 Cc: Matthew Wilcox (Oracle) Cc: Cc: Signed-off-by: Andrew Morton --- include/linux/vmalloc.h | 2 +- mm/vmalloc.c | 58 ++++++++++++++++++++++++------------------------- 2 files changed, 29 insertions(+), 31 deletions(-) (limited to 'include') diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index e4d8d0a9f30f..aed121d729b0 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -62,7 +62,7 @@ struct vm_struct { #ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC unsigned int page_order; #endif - unsigned int nr_pages; + unsigned long nr_pages; phys_addr_t phys_addr; const void *caller; unsigned long requested_size; diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 5c0c2d0d6ae7..72d7f0d81c05 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3361,7 +3361,7 @@ struct vm_struct *remove_vm_area(const void *addr) static inline void set_area_direct_map(const struct vm_struct *area, int (*set_direct_map)(struct page *page)) { - int i; + unsigned long i; /* HUGE_VMALLOC passes small pages to set_direct_map */ for (i = 0; i < area->nr_pages; i++) @@ -3377,7 +3377,7 @@ static void vm_reset_perms(struct vm_struct *area) unsigned long start = ULONG_MAX, end = 0; unsigned int page_order = vm_area_page_order(area); int flush_dmap = 0; - int i; + unsigned long i; /* * Find the start and end range of the direct mappings to make sure that @@ -3450,10 +3450,10 @@ void vfree_atomic(const void *addr) * Caller is responsible for unmapping (vunmap_range) and KASAN * poisoning before calling this. */ -static void vm_area_free_pages(struct vm_struct *vm, unsigned int start_idx, - unsigned int end_idx) +static void vm_area_free_pages(struct vm_struct *vm, unsigned long start_idx, + unsigned long end_idx) { - unsigned int i; + unsigned long i; if (!(vm->flags & VM_MAP_PUT_PAGES)) { for (i = start_idx; i < end_idx; i++) @@ -3665,12 +3665,12 @@ static inline gfp_t vmalloc_gfp_adjust(gfp_t flags, const bool large) return flags; } -static inline unsigned int +static inline unsigned long vm_area_alloc_pages(gfp_t gfp, int nid, - unsigned int order, unsigned int nr_pages, struct page **pages) + unsigned int order, unsigned long nr_pages, struct page **pages) { - unsigned int nr_allocated = 0; - unsigned int nr_remaining = nr_pages; + unsigned long nr_allocated = 0; + unsigned long nr_remaining = nr_pages; unsigned int max_attempt_order = MAX_PAGE_ORDER; struct page *page; int i; @@ -3718,7 +3718,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid, if (!order) { while (nr_allocated < nr_pages) { unsigned int nr, nr_pages_request; - int i; + unsigned long i; /* * A maximum allowed request is hard-coded and is 100 @@ -3726,7 +3726,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid, * long preemption off scenario in the bulk-allocator * so the range is [1:100]. */ - nr_pages_request = min(100U, nr_pages - nr_allocated); + nr_pages_request = min(100UL, nr_pages - nr_allocated); /* memory allocation should consider mempolicy, we can't * wrongly use nearest node when nid == NUMA_NO_NODE, @@ -3872,12 +3872,12 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, unsigned long addr = (unsigned long)area->addr; unsigned long size = get_vm_area_size(area); unsigned long array_size; - unsigned int nr_small_pages = size >> PAGE_SHIFT; + unsigned long nr_small_pages = size >> PAGE_SHIFT; unsigned int page_order; unsigned int flags; int ret; - array_size = (unsigned long)nr_small_pages * sizeof(struct page *); + array_size = nr_small_pages * sizeof(struct page *); /* __GFP_NOFAIL and "noblock" flags are mutually exclusive. */ if (!gfpflags_allow_blocking(gfp_mask)) @@ -4375,7 +4375,7 @@ void *vrealloc_node_align_noprof(const void *p, size_t size, unsigned long align } if (size <= old_size) { - unsigned int new_nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT; + unsigned long new_nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT; /* Zero out "freed" memory, potentially for future realloc. */ if (want_init_on_free() || want_init_on_alloc(flags)) @@ -4404,7 +4404,7 @@ void *vrealloc_node_align_noprof(const void *p, size_t size, unsigned long align !(vm->flags & (VM_FLUSH_RESET_PERMS | VM_USERMAP)) && gfp_has_io_fs(flags)) { unsigned long addr = (unsigned long)kasan_reset_tag(p); - unsigned int old_nr_pages = vm->nr_pages; + unsigned long old_nr_pages = vm->nr_pages; /* * Use the node lock to synchronize with concurrent @@ -4417,16 +4417,13 @@ void *vrealloc_node_align_noprof(const void *p, size_t size, unsigned long align spin_unlock(&vn->busy.lock); /* Notify kmemleak of the reduced allocation size before unmapping. */ - kmemleak_free_part( - (void *)addr + ((unsigned long)new_nr_pages - << PAGE_SHIFT), - (unsigned long)(old_nr_pages - new_nr_pages) - << PAGE_SHIFT); + kmemleak_free_part((void *)addr + + (new_nr_pages << PAGE_SHIFT), + (old_nr_pages - new_nr_pages) + << PAGE_SHIFT); - vunmap_range(addr + ((unsigned long)new_nr_pages - << PAGE_SHIFT), - addr + ((unsigned long)old_nr_pages - << PAGE_SHIFT)); + vunmap_range(addr + (new_nr_pages << PAGE_SHIFT), + addr + (old_nr_pages << PAGE_SHIFT)); vm_area_free_pages(vm, new_nr_pages, old_nr_pages); } @@ -5250,7 +5247,7 @@ bool vmalloc_dump_obj(void *object) struct vmap_area *va; struct vmap_node *vn; unsigned long addr; - unsigned int nr_pages; + unsigned long nr_pages; addr = PAGE_ALIGN((unsigned long) object); vn = addr_to_node(addr); @@ -5270,7 +5267,7 @@ bool vmalloc_dump_obj(void *object) nr_pages = vm->nr_pages; spin_unlock(&vn->busy.lock); - pr_cont(" %u-page vmalloc region starting at %#lx allocated at %pS\n", + pr_cont(" %lu-page vmalloc region starting at %#lx allocated at %pS\n", nr_pages, addr, caller); return true; @@ -5288,16 +5285,17 @@ bool vmalloc_dump_obj(void *object) static void show_numa_info(struct seq_file *m, struct vm_struct *v, unsigned int *counters) { - unsigned int nr; unsigned int step = 1U << vm_area_page_order(v); + unsigned long i; + unsigned int nr; if (!counters) return; memset(counters, 0, nr_node_ids * sizeof(unsigned int)); - for (nr = 0; nr < v->nr_pages; nr += step) - counters[page_to_nid(v->pages[nr])] += step; + for (i = 0; i < v->nr_pages; i += step) + counters[page_to_nid(v->pages[i])] += step; for_each_node_state(nr, N_HIGH_MEMORY) if (counters[nr]) seq_printf(m, " N%u=%u", nr, counters[nr]); @@ -5355,7 +5353,7 @@ static int vmalloc_info_show(struct seq_file *m, void *p) seq_printf(m, " %pS", v->caller); if (v->nr_pages) - seq_printf(m, " pages=%d", v->nr_pages); + seq_printf(m, " pages=%lu", v->nr_pages); if (v->phys_addr) seq_printf(m, " phys=%pa", &v->phys_addr); -- cgit v1.2.3 From 0ddb8bb85b98ff59f4643b7e4500e45f650dddcf Mon Sep 17 00:00:00 2001 From: Sang-Heon Jeon Date: Wed, 5 Aug 2026 00:11:42 +0900 Subject: mm/page_ext: remove pgdat_page_ext_init() pgdat_page_ext_init() sets pgdat->node_page_ext to NULL only on FLATMEM. FLATMEM depends on !NUMA, so the pgdat is always the zero-initialized contig_page_data and the store has no effect. So remove the call site, the unused function and its declaration. No functional change. Link: https://lore.kernel.org/20260804151145.3419768-3-ekffu200098@gmail.com Signed-off-by: Sang-Heon Jeon Acked-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Johannes Weiner Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- include/linux/page_ext.h | 5 ----- mm/mm_init.c | 1 - mm/page_ext.c | 9 --------- 3 files changed, 15 deletions(-) (limited to 'include') diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h index f23d4b218da0..79c53ec45dfa 100644 --- a/include/linux/page_ext.h +++ b/include/linux/page_ext.h @@ -55,7 +55,6 @@ struct page_ext { extern bool early_page_ext; extern unsigned long page_ext_size; -extern void pgdat_page_ext_init(struct pglist_data *pgdat); static inline bool early_page_ext_enabled(void) { @@ -202,10 +201,6 @@ static inline bool early_page_ext_enabled(void) return false; } -static inline void pgdat_page_ext_init(struct pglist_data *pgdat) -{ -} - static inline void page_ext_init(void) { } diff --git a/mm/mm_init.c b/mm/mm_init.c index 711f821f7b3c..e9c4204b73ad 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1394,7 +1394,6 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat) for (i = 0; i < NR_VMSCAN_THROTTLE; i++) init_waitqueue_head(&pgdat->reclaim_wait[i]); - pgdat_page_ext_init(pgdat); lruvec_init(&pgdat->__lruvec); } diff --git a/mm/page_ext.c b/mm/page_ext.c index e2e92bd27ebd..b679a8c1f7d7 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -164,11 +164,6 @@ void __init page_ext_init_flatmem_late(void) invoke_init_callbacks(); } -void __meminit pgdat_page_ext_init(struct pglist_data *pgdat) -{ - pgdat->node_page_ext = NULL; -} - static struct page_ext *lookup_page_ext(const struct page *page) { unsigned long pfn = page_to_pfn(page); @@ -494,10 +489,6 @@ oom: panic("Out of memory"); } -void __meminit pgdat_page_ext_init(struct pglist_data *pgdat) -{ -} - #endif /** -- cgit v1.2.3 From e73aeb8a411e5327da9c0746b2f93924ab081113 Mon Sep 17 00:00:00 2001 From: Muchun Song Date: Wed, 5 Aug 2026 10:25:36 +0800 Subject: mm/sparse: keep mem_section_usage_size() internal mem_section_usage_size() is only needed by sparsemem implementation code after commit ae751d567baa ("mm/bootmem_info: stop marking mem_section_usage as MIX_SECTION_INFO"), so keeping the declaration in mmzone.h now exposes the helper to all mmzone.h users for no reason. Move the helper to sparse.h so sparse.c and sparse-vmemmap.c can share it through the internal header. While doing so, calculate the allocation size with struct_size_t(), which ties the expression to the pageblock_flags trailing array instead of open-coding the struct header plus bitmap size. Link: https://lore.kernel.org/20260805022536.1206575-1-songmuchun@bytedance.com Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- include/linux/mmzone.h | 1 - mm/sparse.c | 10 ---------- mm/sparse.h | 6 ++++++ 3 files changed, 6 insertions(+), 11 deletions(-) (limited to 'include') diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 158c1fba2393..94f9c3ff5416 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2069,7 +2069,6 @@ static inline struct mem_section *__nr_to_section(unsigned long nr) #endif return &mem_section[root][nr & SECTION_ROOT_MASK]; } -extern size_t mem_section_usage_size(void); /* * We use the lower bits of the mem_map pointer to store a little bit of diff --git a/mm/sparse.c b/mm/sparse.c index 67fa192d4289..7c15406e77f5 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -213,16 +213,6 @@ static void __init memblocks_present(void) memory_present(nid, start, end); } -static unsigned long usemap_size(void) -{ - return BITS_TO_LONGS(SECTION_BLOCKFLAGS_BITS) * sizeof(unsigned long); -} - -size_t mem_section_usage_size(void) -{ - return sizeof(struct mem_section_usage) + usemap_size(); -} - #ifdef CONFIG_SPARSEMEM_VMEMMAP unsigned long __init section_map_size(void) { diff --git a/mm/sparse.h b/mm/sparse.h index 95aa031213f2..3b744667a7e6 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -47,6 +47,12 @@ static inline void __section_mark_present(struct mem_section *ms, ms->section_mem_map |= SECTION_MARKED_PRESENT; } + +static inline size_t mem_section_usage_size(void) +{ + return struct_size_t(struct mem_section_usage, pageblock_flags, + BITS_TO_LONGS(SECTION_BLOCKFLAGS_BITS)); +} #else static inline void sparse_init(void) {} #endif /* CONFIG_SPARSEMEM */ -- cgit v1.2.3 From 20d4d490bc0ca5fbfa3f99e1e8e7d398b7971ccb Mon Sep 17 00:00:00 2001 From: Henry Elderman Date: Fri, 7 Aug 2026 11:19:58 +0200 Subject: mm/execmem: fix fallback_end description in kernel-doc The kernel-doc for struct execmem_range incorrectly describes @fallback_end as "start". Correct it to "end". Link: https://lore.kernel.org/20260807091958.4735-1-henry.elderman.edu+linux@gmail.com Signed-off-by: Henry Elderman Reviewed-by: Mike Rapoport (Microsoft) Signed-off-by: Andrew Morton --- include/linux/execmem.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'include') diff --git a/include/linux/execmem.h b/include/linux/execmem.h index 7de229134e30..1bd34925d1aa 100644 --- a/include/linux/execmem.h +++ b/include/linux/execmem.h @@ -89,7 +89,7 @@ static inline int execmem_restore_rox(void *ptr, size_t size) { return 0; } * @end: address space end (inclusive) * @fallback_start: start of the secondary address space range for fallback * allocations on architectures that require it - * @fallback_end: start of the secondary address space (inclusive) + * @fallback_end: end of the secondary address space (inclusive) * @pgprot: permissions for memory in this address space * @alignment: alignment required for text allocations * @flags: options for memory allocations for this range -- cgit v1.2.3 From 28b13c3c4c6592da8dd59a7b06655e39665ee624 Mon Sep 17 00:00:00 2001 From: Bart Van Assche Date: Fri, 7 Aug 2026 10:13:10 -0700 Subject: mm: make VM_FAULT_RESULT_TRACE compatible with sparse Fix the following sparse warnings that appear while building f2fs: ./include/trace/events/f2fs.h:1469:1: warning: incorrect type in initializer (different base types) ./include/trace/events/f2fs.h:1469:1: expected unsigned long mask ./include/trace/events/f2fs.h:1469:1: got restricted vm_fault_t Link: https://lore.kernel.org/e56c9e2aead04f79192c3110de80d846e41e3791.1786122711.git.bvanassche@acm.org Signed-off-by: Bart Van Assche Acked-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Reviewed-by: Anshuman Khandual Signed-off-by: Andrew Morton --- include/linux/mm_types.h | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) (limited to 'include') diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index ebf0d912be7d..6d815f6440c9 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1718,20 +1718,20 @@ enum vm_fault_reason { VM_FAULT_SIGSEGV | VM_FAULT_HWPOISON | \ VM_FAULT_HWPOISON_LARGE | VM_FAULT_FALLBACK) -#define VM_FAULT_RESULT_TRACE \ - { VM_FAULT_OOM, "OOM" }, \ - { VM_FAULT_SIGBUS, "SIGBUS" }, \ - { VM_FAULT_MAJOR, "MAJOR" }, \ - { VM_FAULT_HWPOISON, "HWPOISON" }, \ - { VM_FAULT_HWPOISON_LARGE, "HWPOISON_LARGE" }, \ - { VM_FAULT_SIGSEGV, "SIGSEGV" }, \ - { VM_FAULT_NOPAGE, "NOPAGE" }, \ - { VM_FAULT_LOCKED, "LOCKED" }, \ - { VM_FAULT_RETRY, "RETRY" }, \ - { VM_FAULT_FALLBACK, "FALLBACK" }, \ - { VM_FAULT_DONE_COW, "DONE_COW" }, \ - { VM_FAULT_NEEDDSYNC, "NEEDDSYNC" }, \ - { VM_FAULT_COMPLETED, "COMPLETED" } +#define VM_FAULT_RESULT_TRACE \ + { (__force u32)VM_FAULT_OOM, "OOM" }, \ + { (__force u32)VM_FAULT_SIGBUS, "SIGBUS" }, \ + { (__force u32)VM_FAULT_MAJOR, "MAJOR" }, \ + { (__force u32)VM_FAULT_HWPOISON, "HWPOISON" }, \ + { (__force u32)VM_FAULT_HWPOISON_LARGE, "HWPOISON_LARGE" }, \ + { (__force u32)VM_FAULT_SIGSEGV, "SIGSEGV" }, \ + { (__force u32)VM_FAULT_NOPAGE, "NOPAGE" }, \ + { (__force u32)VM_FAULT_LOCKED, "LOCKED" }, \ + { (__force u32)VM_FAULT_RETRY, "RETRY" }, \ + { (__force u32)VM_FAULT_FALLBACK, "FALLBACK" }, \ + { (__force u32)VM_FAULT_DONE_COW, "DONE_COW" }, \ + { (__force u32)VM_FAULT_NEEDDSYNC, "NEEDDSYNC" }, \ + { (__force u32)VM_FAULT_COMPLETED, "COMPLETED" } struct vm_special_mapping { const char *name; /* The name, e.g. "[vdso]". */ -- cgit v1.2.3 From 1d581ab2348cdbb6d4d0a467382926b68e374ec9 Mon Sep 17 00:00:00 2001 From: Suren Baghdasaryan Date: Wed, 8 Jul 2026 18:01:23 +0000 Subject: alloc_tag: add ioctl to /proc/allocinfo Patch series "alloc_tag: introduce IOCTL-based filtering for MAP", v8. Currently, memory allocation profiling data is primarily exposed through /proc/allocinfo. While useful for manual inspection, this text-based interface poses challenges for production monitoring and large-scale analysis: 1. Userspace must parse large amounts of text to extract specific fields. 2. To find specific tags, userspace must read the entire dataset, requiring many context switches and high data copying. 3. The kernel currently aggregates per-CPU counters for every allocation size, even those the user intends to filter out immediately. This series introduces a new IOCTL-based binary interface for allocinfo that supports kernel-side filtering. By allowing the user to specify a filter mask, we significantly reduce the work performed in-kernel and the amount of data transferred to userspace. The IOCTL mechanism was chosen for allocinfo to address the per-CPU counter aggregation bottleneck. A traditional read() operation must report the total allocation count and sizes for every code tag in the system. Doing so requires iterating across all CPUs to sum their per-CPU counters for thousands of tags, which introduces substantial runtime overhead. The IOCTL interface allows userspace to push selective filtering criteria directly into the kernel before the per-CPU counter aggregation. The kernel aggregates per-CPU counters only for a small subset of tags that match the filter. This results in significant performance improvement. Beyond fast filtered retrieval, the IOCTL foundation allows introducing a context capture mechanism in the future to capture the context for specific allocations. Performance measurements were conducted on an Intel Xeon Platinum 8481C (224 CPUs) with caches dropped before each run. The IOCTL mechanism shows a ~20x performance improvement for filtered queries. The kernel avoids the expensive per-CPU counter aggregation (alloc_tag_read) for any tags that fail the initial string or location filters. Scenario 1: Specific File Filtering (arch/x86/events/rapl.c) 1. Traditional (cat /proc/allocinfo | grep): 22ms (sys) 2. IOCTL Interface: 1ms (sys) Scenario 2: Compound Filtering (Filename + Size) 1. Traditional: (cat ... | grep | awk): 21ms (sys) 2. IOCTL Interface: 1ms (sys) Scenario 3: Size-Based Filtering (min_size = 1MB) 1. Traditional: (cat ... | awk): 21ms (sys) 2. IOCTL Interface: 14ms (sys) This patch (of 6): Add the following ioctl commands for /proc/allocinfo file: ALLOCINFO_IOC_CONTENT_ID - gets content identifier which can be used to check whether the file content has changed specifically due to module load/unload. Every time a module is loaded / unloaded, the returned value will be different. By comparing the identifier value at the beginning and at the end of the content retrieval operation, users can validate retrieved information for consistency. ALLOCINFO_IOC_GET_AT - gets the record at the specified position. This is the position of a record in /proc/allocinfo. ALLOCINFO_IOC_GET_NEXT - gets the record next to the last retrieved one. If no records were previously retrieved, returns the first record. Note, function file and module names often have the same prefixes, therefore when filtering for them, we compare the last 64 characters to minimize the chances of name collisions. [akpm@linux-foundation.org: include compat.h, per Suren] Closes: https://lore.kernel.org/oe-kbuild-all/202607091820.qbjlGhKK-lkp@intel.com/ Link: https://lore.kernel.org/cover.1783532853.git.abhishekbapat@google.com Link: https://lore.kernel.org/15596de2607ef13e7c77c6d74763f4ae992ec475.1783532853.git.abhishekbapat@google.com Signed-off-by: Suren Baghdasaryan Signed-off-by: Abhishek Bapat Acked-by: Hao Ge Cc: Jonathan Corbet Cc: Kent Overstreet Cc: Sourav Panda Signed-off-by: Andrew Morton --- Documentation/mm/allocation-profiling.rst | 5 + Documentation/userspace-api/ioctl/ioctl-number.rst | 2 + MAINTAINERS | 1 + include/linux/codetag.h | 2 + include/uapi/linux/alloc_tag.h | 65 ++++++ lib/codetag.c | 18 ++ mm/alloc_tag.c | 239 ++++++++++++++++++++- 7 files changed, 330 insertions(+), 2 deletions(-) create mode 100644 include/uapi/linux/alloc_tag.h (limited to 'include') diff --git a/Documentation/mm/allocation-profiling.rst b/Documentation/mm/allocation-profiling.rst index e928aa3e4e1e..b2ebcef8af6f 100644 --- a/Documentation/mm/allocation-profiling.rst +++ b/Documentation/mm/allocation-profiling.rst @@ -57,6 +57,11 @@ sysctl: Runtime info: /proc/allocinfo + Profiling data can be retrieved either by reading `/proc/allocinfo` directly as + text or programmatically via `ioctl()` calls defined in ``. + The ioctl interface supports structured binary data extraction as well as filtering + by module name, function, file, line number, accuracy, or allocation size limits. + Example output:: root@moria-kvm:~# sort -g /proc/allocinfo|tail|numfmt --to=iec diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst index 3f0ef1e27eb0..2fc53093752d 100644 --- a/Documentation/userspace-api/ioctl/ioctl-number.rst +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst @@ -346,6 +346,8 @@ Code Seq# Include File Comments 0xA5 20-2F linux/surface_aggregator/dtx.h Microsoft Surface DTX driver +0xA6 00-0F uapi/linux/alloc_tag.h Memory allocation profiling + 0xAA 00-3F linux/uapi/linux/userfaultfd.h 0xAB 00-1F linux/nbd.h 0xAC 00-1F linux/raw.h diff --git a/MAINTAINERS b/MAINTAINERS index 06271e742d32..557e5fd32073 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -16940,6 +16940,7 @@ S: Maintained F: Documentation/mm/allocation-profiling.rst F: include/linux/alloc_tag.h F: include/linux/pgalloc_tag.h +F: include/uapi/linux/alloc_tag.h F: mm/alloc_tag.c MEMORY MANAGEMENT - BALLOON diff --git a/include/linux/codetag.h b/include/linux/codetag.h index ddae7484ca45..a25a085c2df1 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -77,6 +77,8 @@ struct codetag_iterator { void codetag_lock_module_list(struct codetag_type *cttype); bool codetag_trylock_module_list(struct codetag_type *cttype); void codetag_unlock_module_list(struct codetag_type *cttype); +unsigned long codetag_get_content_id(struct codetag_type *cttype); +unsigned int codetag_get_count(struct codetag_type *cttype); struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype); struct codetag *codetag_next_ct(struct codetag_iterator *iter); diff --git a/include/uapi/linux/alloc_tag.h b/include/uapi/linux/alloc_tag.h new file mode 100644 index 000000000000..ee6a023cbaf4 --- /dev/null +++ b/include/uapi/linux/alloc_tag.h @@ -0,0 +1,65 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* + * alloc_tag IOCTL API definition + * + * Copyright (C) 2026 Google, LLC. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef _UAPI_ALLOC_TAG_H +#define _UAPI_ALLOC_TAG_H + +#include + +/* + * Function, file and module names often have the same prefixes, therefore + * when filtering by these criteria, we compare the last 64 characters to + * minimize the chances of name collisions + */ +#define ALLOCINFO_STR_SIZE 64 + +struct allocinfo_content_id { + __u64 id; +}; + +struct allocinfo_tag { + /* Longer names are trimmed */ + char modname[ALLOCINFO_STR_SIZE]; + char function[ALLOCINFO_STR_SIZE]; + char filename[ALLOCINFO_STR_SIZE]; + __u64 lineno; +}; + +/* The alignment ensures 32-bit compatible interfaces are not broken */ +struct allocinfo_counter { + __u64 bytes; + __u64 calls; + __u8 accurate; +} __attribute__((aligned(8))); + +struct allocinfo_tag_data { + struct allocinfo_tag tag; + struct allocinfo_counter counter; +}; + +struct allocinfo_get_at { + __u64 pos; /* input */ + struct allocinfo_tag_data data; +}; + +#define _ALLOCINFO_IOC_CONTENT_ID 0 +#define _ALLOCINFO_IOC_GET_AT 1 +#define _ALLOCINFO_IOC_GET_NEXT 2 + +#define ALLOCINFO_IOC_BASE 0xA6 +#define ALLOCINFO_IOC_CONTENT_ID _IOR(ALLOCINFO_IOC_BASE, _ALLOCINFO_IOC_CONTENT_ID, \ + struct allocinfo_content_id) +#define ALLOCINFO_IOC_GET_AT _IOWR(ALLOCINFO_IOC_BASE, _ALLOCINFO_IOC_GET_AT, \ + struct allocinfo_get_at) +#define ALLOCINFO_IOC_GET_NEXT _IOR(ALLOCINFO_IOC_BASE, _ALLOCINFO_IOC_GET_NEXT, \ + struct allocinfo_tag_data) + +#endif /* _UAPI_ALLOC_TAG_H */ diff --git a/lib/codetag.c b/lib/codetag.c index 4001a7ea6675..a9cda4c962a3 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -19,6 +19,8 @@ struct codetag_type { struct codetag_type_desc desc; /* generates unique sequence number for module load */ unsigned long next_mod_seq; + /* bumped on every module load and unload */ + unsigned long content_id; }; struct codetag_range { @@ -50,6 +52,20 @@ void codetag_unlock_module_list(struct codetag_type *cttype) up_read(&cttype->mod_lock); } +unsigned long codetag_get_content_id(struct codetag_type *cttype) +{ + lockdep_assert_held(&cttype->mod_lock); + + return cttype->content_id; +} + +unsigned int codetag_get_count(struct codetag_type *cttype) +{ + lockdep_assert_held(&cttype->mod_lock); + + return cttype->count; +} + struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype) { struct codetag_iterator iter = { @@ -204,6 +220,7 @@ static int codetag_module_init(struct codetag_type *cttype, struct module *mod) down_write(&cttype->mod_lock); cmod->mod_seq = ++cttype->next_mod_seq; + ++cttype->content_id; mod_id = idr_alloc(&cttype->mod_idr, cmod, 0, 0, GFP_KERNEL); if (mod_id >= 0) { if (cttype->desc.module_load) { @@ -368,6 +385,7 @@ void codetag_unload_module(struct module *mod) cttype->count -= range_size(cttype, &cmod->range); idr_remove(&cttype->mod_idr, mod_id); kfree(cmod); + ++cttype->content_id; } up_write(&cttype->mod_lock); if (found && cttype->desc.free_section_mem) diff --git a/mm/alloc_tag.c b/mm/alloc_tag.c index b60ee89704cc..b2ac166880ac 100644 --- a/mm/alloc_tag.c +++ b/mm/alloc_tag.c @@ -5,6 +5,8 @@ #include #include #include +#include +#include #include #include #include @@ -14,6 +16,7 @@ #include #include #include +#include #include "internal.h" #include "page_alloc.h" @@ -59,6 +62,10 @@ struct allocinfo_private { struct codetag_iterator iter; struct codetag_iterator reported_iter; bool print_header; + /* ioctl uses a separate iterator not to interfere with reads */ + struct codetag_iterator ioctl_iter; + bool positioned; /* seq_open_private() sets to 0 */ + struct mutex ioctl_lock; }; static void *allocinfo_start(struct seq_file *m, loff_t *pos) @@ -142,6 +149,235 @@ static const struct seq_operations allocinfo_seq_op = { .show = allocinfo_show, }; +/* + * Initializes seq_file operations and allocates private state when opening + * the /proc/allocinfo procfs entry. + */ +static int allocinfo_open(struct inode *inode, struct file *file) +{ + int ret; + + ret = seq_open_private(file, &allocinfo_seq_op, + sizeof(struct allocinfo_private)); + if (!ret) { + struct seq_file *m = file->private_data; + struct allocinfo_private *priv = m->private; + + mutex_init(&priv->ioctl_lock); + } + return ret; +} + +/* + * Cleans up the seq_file state and frees up the private state allocated in + * allocinfo_open() when closing the /proc/allocinfo file descriptor. + */ +static int allocinfo_release(struct inode *inode, struct file *file) +{ + struct seq_file *m = file->private_data; + struct allocinfo_private *priv = m->private; + + mutex_destroy(&priv->ioctl_lock); + return seq_release_private(inode, file); +} + +/* + * Returns a pointer to the suffix of a string so that its length fits within + * ALLOCINFO_STR_SIZE, preserving the trailing characters. + * Function, file and module names often have the same prefixes, therefore + * when filtering by these criteria, we compare the last 64 characters to + * minimize the chances of name collisions + */ +static const char *allocinfo_str(const char *str) +{ + size_t len = strlen(str); + + /* Keep an extra space for the trailing NULL. */ + if (len >= ALLOCINFO_STR_SIZE) + str += (len - ALLOCINFO_STR_SIZE) + 1; + return str; +} + +/* Copy a string and trim from the beginning if it's too long */ +static void allocinfo_copy_str(char *dest, const char *src) +{ + strscpy_pad(dest, allocinfo_str(src), ALLOCINFO_STR_SIZE); +} + +/* + * Populates the UAPI allocinfo_tag_data structure with active runtime + * profiling counters extracted from the given kernel codetag. + */ +static void allocinfo_to_params(struct codetag *ct, + struct allocinfo_tag_data *data) +{ + struct alloc_tag *tag = ct_to_alloc_tag(ct); + struct alloc_tag_counters counter = alloc_tag_read(tag); + + if (ct->modname) + allocinfo_copy_str(data->tag.modname, ct->modname); + else + data->tag.modname[0] = '\0'; + allocinfo_copy_str(data->tag.function, ct->function); + allocinfo_copy_str(data->tag.filename, ct->filename); + data->tag.lineno = ct->lineno; + data->counter.bytes = counter.bytes; + data->counter.calls = counter.calls; + data->counter.accurate = !alloc_tag_is_inaccurate(tag); +} + +/* + * Retrieves the unique content ID representing the current allocation tag module + * layout, allowing userspace to detect if modules were loaded / unloaded. + */ +static int allocinfo_ioctl_get_content_id(struct seq_file *m, void __user *arg) +{ + struct allocinfo_content_id params; + + codetag_lock_module_list(alloc_tag_cttype); + params.id = codetag_get_content_id(alloc_tag_cttype); + codetag_unlock_module_list(alloc_tag_cttype); + if (copy_to_user(arg, ¶ms, sizeof(params))) + return -EFAULT; + + return 0; +} + +/* + * Seeks the ioctl iterator to the specified 0-indexed tag position, reads its + * profiling data and returns it to userspace. + */ +static int allocinfo_ioctl_get_at(struct seq_file *m, void __user *arg) +{ + struct allocinfo_private *priv; + struct codetag *ct; + __u64 pos; + struct allocinfo_get_at params = {0}; + + if (copy_from_user(¶ms, arg, sizeof(params))) + return -EFAULT; + + priv = m->private; + pos = params.pos; + + mutex_lock(&priv->ioctl_lock); + codetag_lock_module_list(alloc_tag_cttype); + + if (pos >= codetag_get_count(alloc_tag_cttype)) { + codetag_unlock_module_list(alloc_tag_cttype); + mutex_unlock(&priv->ioctl_lock); + return -ENOENT; + } + + /* Find the codetag */ + priv->ioctl_iter = codetag_get_ct_iter(alloc_tag_cttype); + ct = codetag_next_ct(&priv->ioctl_iter); + while (ct && pos--) + ct = codetag_next_ct(&priv->ioctl_iter); + if (ct) { + allocinfo_to_params(ct, ¶ms.data); + priv->positioned = true; + } + + codetag_unlock_module_list(alloc_tag_cttype); + mutex_unlock(&priv->ioctl_lock); + + if (!ct) + return -ENOENT; + + if (copy_to_user(arg, ¶ms, sizeof(params))) + return -EFAULT; + + return 0; +} + +/* + * Advances the ioctl iterator to the next allocation tag in the sequence and + * returns its profiling data to userspace. + */ +static int allocinfo_ioctl_get_next(struct seq_file *m, void __user *arg) +{ + struct allocinfo_private *priv; + struct codetag *ct; + struct allocinfo_tag_data params; + int ret = 0; + + memset(¶ms, 0, sizeof(params)); + priv = m->private; + + mutex_lock(&priv->ioctl_lock); + codetag_lock_module_list(alloc_tag_cttype); + + if (!priv->positioned) { + priv->ioctl_iter = codetag_get_ct_iter(alloc_tag_cttype); + priv->positioned = true; + } + + ct = codetag_next_ct(&priv->ioctl_iter); + if (ct) + allocinfo_to_params(ct, ¶ms); + + if (!ct) { + priv->positioned = false; + ret = -ENOENT; + } + codetag_unlock_module_list(alloc_tag_cttype); + mutex_unlock(&priv->ioctl_lock); + + if (ret == 0) { + if (copy_to_user(arg, ¶ms, sizeof(params))) + return -EFAULT; + } + return ret; +} + +/* + * Entry point ioctl function for /proc/allocinfo routing requests to fetch the + * layout content ID, seek to a specific tag, or read sequential tags. + */ +static long allocinfo_ioctl(struct file *file, unsigned int cmd, + unsigned long __arg) +{ + void __user *arg = (void __user *)__arg; + int ret; + + switch (cmd) { + case ALLOCINFO_IOC_CONTENT_ID: + ret = allocinfo_ioctl_get_content_id(file->private_data, arg); + break; + case ALLOCINFO_IOC_GET_AT: + ret = allocinfo_ioctl_get_at(file->private_data, arg); + break; + case ALLOCINFO_IOC_GET_NEXT: + ret = allocinfo_ioctl_get_next(file->private_data, arg); + break; + default: + ret = -ENOIOCTLCMD; + break; + } + + return ret; +} + +#ifdef CONFIG_COMPAT +static long allocinfo_compat_ioctl(struct file *file, unsigned int cmd, + unsigned long arg) +{ + return allocinfo_ioctl(file, cmd, (unsigned long)compat_ptr(arg)); +} +#endif + +static const struct proc_ops allocinfo_proc_ops = { + .proc_open = allocinfo_open, + .proc_read_iter = seq_read_iter, + .proc_lseek = seq_lseek, + .proc_release = allocinfo_release, + .proc_ioctl = allocinfo_ioctl, +#ifdef CONFIG_COMPAT + .proc_compat_ioctl = allocinfo_compat_ioctl, +#endif +}; + size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sleep) { struct codetag_iterator iter; @@ -999,8 +1235,7 @@ static int __init alloc_tag_init(void) return 0; } - if (!proc_create_seq_private(ALLOCINFO_FILE_NAME, 0400, NULL, &allocinfo_seq_op, - sizeof(struct allocinfo_private), NULL)) { + if (!proc_create(ALLOCINFO_FILE_NAME, 0400, NULL, &allocinfo_proc_ops)) { pr_err("Failed to create %s file\n", ALLOCINFO_FILE_NAME); shutdown_mem_profiling(false); return -ENOMEM; -- cgit v1.2.3 From 5732a4e4c18acac152768df9ce48e057232edbf4 Mon Sep 17 00:00:00 2001 From: Abhishek Bapat Date: Wed, 8 Jul 2026 18:01:24 +0000 Subject: alloc_tag: add ioctl filters to /proc/allocinfo Extend the capability of the IOCTL mechanism to filter allocations based on tag's module name, function name, file name and line number. Link: https://lore.kernel.org/6a6100c0c58cb2911f39126b9fe177a8c17db16f.1783532853.git.abhishekbapat@google.com Signed-off-by: Abhishek Bapat Acked-by: Hao Ge Acked-by: Suren Baghdasaryan Cc: Jonathan Corbet Cc: Kent Overstreet Cc: Sourav Panda Signed-off-by: Andrew Morton --- include/uapi/linux/alloc_tag.h | 26 +++++++++++++++- mm/alloc_tag.c | 68 +++++++++++++++++++++++++++++++++++++++--- 2 files changed, 89 insertions(+), 5 deletions(-) (limited to 'include') diff --git a/include/uapi/linux/alloc_tag.h b/include/uapi/linux/alloc_tag.h index ee6a023cbaf4..13e9b5916bf5 100644 --- a/include/uapi/linux/alloc_tag.h +++ b/include/uapi/linux/alloc_tag.h @@ -45,8 +45,32 @@ struct allocinfo_tag_data { struct allocinfo_counter counter; }; +enum { + ALLOCINFO_FILTER_MODNAME, + ALLOCINFO_FILTER_FUNCTION, + ALLOCINFO_FILTER_FILENAME, + ALLOCINFO_FILTER_LINENO, + __ALLOCINFO_FILTER_LAST = ALLOCINFO_FILTER_LINENO +}; + +#define ALLOCINFO_FILTER_MASK_MODNAME (1 << ALLOCINFO_FILTER_MODNAME) +#define ALLOCINFO_FILTER_MASK_FUNCTION (1 << ALLOCINFO_FILTER_FUNCTION) +#define ALLOCINFO_FILTER_MASK_FILENAME (1 << ALLOCINFO_FILTER_FILENAME) +#define ALLOCINFO_FILTER_MASK_LINENO (1 << ALLOCINFO_FILTER_LINENO) + +#define ALLOCINFO_FILTER_MASKS \ + ((1 << (__ALLOCINFO_FILTER_LAST + 1)) - 1) + +struct allocinfo_filter { + __u64 mask; /* bitmask of the filter fields used */ + struct allocinfo_tag fields; +}; + struct allocinfo_get_at { - __u64 pos; /* input */ + /* inputs */ + __u64 pos; + struct allocinfo_filter filter; + /* output */ struct allocinfo_tag_data data; }; diff --git a/mm/alloc_tag.c b/mm/alloc_tag.c index b2ac166880ac..d7ed0034c49f 100644 --- a/mm/alloc_tag.c +++ b/mm/alloc_tag.c @@ -62,6 +62,7 @@ struct allocinfo_private { struct codetag_iterator iter; struct codetag_iterator reported_iter; bool print_header; + struct allocinfo_filter filter; /* ioctl uses a separate iterator not to interfere with reads */ struct codetag_iterator ioctl_iter; bool positioned; /* seq_open_private() sets to 0 */ @@ -204,6 +205,12 @@ static void allocinfo_copy_str(char *dest, const char *src) strscpy_pad(dest, allocinfo_str(src), ALLOCINFO_STR_SIZE); } +/* Compare two strings and only consider the trimmed suffix if s1 is too long */ +static int allocinfo_cmp_str(const char *str, const char *template) +{ + return strncmp(allocinfo_str(str), template, ALLOCINFO_STR_SIZE); +} + /* * Populates the UAPI allocinfo_tag_data structure with active runtime * profiling counters extracted from the given kernel codetag. @@ -243,6 +250,40 @@ static int allocinfo_ioctl_get_content_id(struct seq_file *m, void __user *arg) return 0; } +/* + * Verifies whether a given codetag satisfies the active filtering criteria by + * matching its characteristics against the specified filter. + */ +static bool matches_filter(struct codetag *ct, struct allocinfo_filter *filter) +{ + if (!filter || !filter->mask) + return true; + + if (filter->mask & ALLOCINFO_FILTER_MASK_MODNAME) { + /* user wants to filter by modname but ct->modname is NULL */ + if (!ct->modname) { + /* validate if user was attempting to filter for built-in allocations */ + if (filter->fields.modname[0] != '\0') + return false; + } else if (allocinfo_cmp_str(ct->modname, filter->fields.modname)) + return false; + } + + if ((filter->mask & ALLOCINFO_FILTER_MASK_FUNCTION) && + ct->function && allocinfo_cmp_str(ct->function, filter->fields.function)) + return false; + + if ((filter->mask & ALLOCINFO_FILTER_MASK_FILENAME) && + ct->filename && allocinfo_cmp_str(ct->filename, filter->fields.filename)) + return false; + + if ((filter->mask & ALLOCINFO_FILTER_MASK_LINENO) && + ct->lineno != filter->fields.lineno) + return false; + + return true; +} + /* * Seeks the ioctl iterator to the specified 0-indexed tag position, reads its * profiling data and returns it to userspace. @@ -251,29 +292,46 @@ static int allocinfo_ioctl_get_at(struct seq_file *m, void __user *arg) { struct allocinfo_private *priv; struct codetag *ct; - __u64 pos; struct allocinfo_get_at params = {0}; + __u64 skip_count; if (copy_from_user(¶ms, arg, sizeof(params))) return -EFAULT; + if (params.filter.mask & ~ALLOCINFO_FILTER_MASKS) + return -EINVAL; + priv = m->private; - pos = params.pos; mutex_lock(&priv->ioctl_lock); codetag_lock_module_list(alloc_tag_cttype); - if (pos >= codetag_get_count(alloc_tag_cttype)) { + if (params.pos >= codetag_get_count(alloc_tag_cttype)) { codetag_unlock_module_list(alloc_tag_cttype); mutex_unlock(&priv->ioctl_lock); return -ENOENT; } + skip_count = params.pos; + + if (params.filter.mask) + priv->filter = params.filter; + else + priv->filter.mask = 0; + /* Find the codetag */ priv->ioctl_iter = codetag_get_ct_iter(alloc_tag_cttype); ct = codetag_next_ct(&priv->ioctl_iter); - while (ct && pos--) + + while (ct) { + if (matches_filter(ct, &priv->filter)) { + if (skip_count == 0) + break; + skip_count--; + } ct = codetag_next_ct(&priv->ioctl_iter); + } + if (ct) { allocinfo_to_params(ct, ¶ms.data); priv->positioned = true; @@ -314,6 +372,8 @@ static int allocinfo_ioctl_get_next(struct seq_file *m, void __user *arg) } ct = codetag_next_ct(&priv->ioctl_iter); + while (ct && !matches_filter(ct, &priv->filter)) + ct = codetag_next_ct(&priv->ioctl_iter); if (ct) allocinfo_to_params(ct, ¶ms); -- cgit v1.2.3 From 6f6769ea88f89116e8d66a4ac77056e2e3b377c5 Mon Sep 17 00:00:00 2001 From: Abhishek Bapat Date: Wed, 8 Jul 2026 18:01:25 +0000 Subject: alloc_tag: add size-based filtering to ioctl Extend the allocinfo filtering mechanism to allow users to filter tags based on the total number of bytes allocated [min_size, max_size]. The size range is inclusive. Filtering by size involves retrieving allocinfo per-CPU counters, which is an expensive operation. Hence, the performance of size-based filtering will be worse than other filters. Link: https://lore.kernel.org/0a7653b70ae0d64e967fbea0e933bc35f8ac656e.1783532853.git.abhishekbapat@google.com Signed-off-by: Abhishek Bapat Acked-by: Hao Ge Acked-by: Suren Baghdasaryan Cc: Jonathan Corbet Cc: Kent Overstreet Cc: Sourav Panda Signed-off-by: Andrew Morton --- include/uapi/linux/alloc_tag.h | 8 +++++- mm/alloc_tag.c | 64 +++++++++++++++++++++++++++++++++--------- 2 files changed, 58 insertions(+), 14 deletions(-) (limited to 'include') diff --git a/include/uapi/linux/alloc_tag.h b/include/uapi/linux/alloc_tag.h index 13e9b5916bf5..0de5fc180790 100644 --- a/include/uapi/linux/alloc_tag.h +++ b/include/uapi/linux/alloc_tag.h @@ -50,13 +50,17 @@ enum { ALLOCINFO_FILTER_FUNCTION, ALLOCINFO_FILTER_FILENAME, ALLOCINFO_FILTER_LINENO, - __ALLOCINFO_FILTER_LAST = ALLOCINFO_FILTER_LINENO + ALLOCINFO_FILTER_MIN_SIZE, + ALLOCINFO_FILTER_MAX_SIZE, + __ALLOCINFO_FILTER_LAST = ALLOCINFO_FILTER_MAX_SIZE }; #define ALLOCINFO_FILTER_MASK_MODNAME (1 << ALLOCINFO_FILTER_MODNAME) #define ALLOCINFO_FILTER_MASK_FUNCTION (1 << ALLOCINFO_FILTER_FUNCTION) #define ALLOCINFO_FILTER_MASK_FILENAME (1 << ALLOCINFO_FILTER_FILENAME) #define ALLOCINFO_FILTER_MASK_LINENO (1 << ALLOCINFO_FILTER_LINENO) +#define ALLOCINFO_FILTER_MASK_MIN_SIZE (1 << ALLOCINFO_FILTER_MIN_SIZE) +#define ALLOCINFO_FILTER_MASK_MAX_SIZE (1 << ALLOCINFO_FILTER_MAX_SIZE) #define ALLOCINFO_FILTER_MASKS \ ((1 << (__ALLOCINFO_FILTER_LAST + 1)) - 1) @@ -64,6 +68,8 @@ enum { struct allocinfo_filter { __u64 mask; /* bitmask of the filter fields used */ struct allocinfo_tag fields; + __u64 min_size; + __u64 max_size; }; struct allocinfo_get_at { diff --git a/mm/alloc_tag.c b/mm/alloc_tag.c index d7ed0034c49f..04b640d74bcc 100644 --- a/mm/alloc_tag.c +++ b/mm/alloc_tag.c @@ -211,16 +211,20 @@ static int allocinfo_cmp_str(const char *str, const char *template) return strncmp(allocinfo_str(str), template, ALLOCINFO_STR_SIZE); } +/* Fetch the per-CPU counters */ +static inline struct alloc_tag_counters allocinfo_prefetch_counters(struct codetag *ct) +{ + return alloc_tag_read(ct_to_alloc_tag(ct)); +} + /* * Populates the UAPI allocinfo_tag_data structure with active runtime * profiling counters extracted from the given kernel codetag. */ static void allocinfo_to_params(struct codetag *ct, - struct allocinfo_tag_data *data) + struct allocinfo_tag_data *data, + struct alloc_tag_counters *counters) { - struct alloc_tag *tag = ct_to_alloc_tag(ct); - struct alloc_tag_counters counter = alloc_tag_read(tag); - if (ct->modname) allocinfo_copy_str(data->tag.modname, ct->modname); else @@ -228,9 +232,9 @@ static void allocinfo_to_params(struct codetag *ct, allocinfo_copy_str(data->tag.function, ct->function); allocinfo_copy_str(data->tag.filename, ct->filename); data->tag.lineno = ct->lineno; - data->counter.bytes = counter.bytes; - data->counter.calls = counter.calls; - data->counter.accurate = !alloc_tag_is_inaccurate(tag); + data->counter.bytes = counters->bytes; + data->counter.calls = counters->calls; + data->counter.accurate = !alloc_tag_is_inaccurate(ct_to_alloc_tag(ct)); } /* @@ -254,7 +258,9 @@ static int allocinfo_ioctl_get_content_id(struct seq_file *m, void __user *arg) * Verifies whether a given codetag satisfies the active filtering criteria by * matching its characteristics against the specified filter. */ -static bool matches_filter(struct codetag *ct, struct allocinfo_filter *filter) +static bool matches_filter(struct codetag *ct, struct allocinfo_filter *filter, + struct alloc_tag_counters *counters, + bool *fetched_counters) { if (!filter || !filter->mask) return true; @@ -281,6 +287,19 @@ static bool matches_filter(struct codetag *ct, struct allocinfo_filter *filter) ct->lineno != filter->fields.lineno) return false; + if (filter->mask & (ALLOCINFO_FILTER_MASK_MIN_SIZE | ALLOCINFO_FILTER_MASK_MAX_SIZE)) { + if (!*fetched_counters) { + *counters = allocinfo_prefetch_counters(ct); + *fetched_counters = true; + } + if ((filter->mask & ALLOCINFO_FILTER_MASK_MIN_SIZE) && + counters->bytes < filter->min_size) + return false; + if ((filter->mask & ALLOCINFO_FILTER_MASK_MAX_SIZE) && + counters->bytes > filter->max_size) + return false; + } + return true; } @@ -294,6 +313,8 @@ static int allocinfo_ioctl_get_at(struct seq_file *m, void __user *arg) struct codetag *ct; struct allocinfo_get_at params = {0}; __u64 skip_count; + struct alloc_tag_counters counters; + bool fetched_counters; if (copy_from_user(¶ms, arg, sizeof(params))) return -EFAULT; @@ -301,6 +322,11 @@ static int allocinfo_ioctl_get_at(struct seq_file *m, void __user *arg) if (params.filter.mask & ~ALLOCINFO_FILTER_MASKS) return -EINVAL; + if ((params.filter.mask & ALLOCINFO_FILTER_MASK_MIN_SIZE) && + (params.filter.mask & ALLOCINFO_FILTER_MASK_MAX_SIZE) && + params.filter.min_size > params.filter.max_size) + return -EINVAL; + priv = m->private; mutex_lock(&priv->ioctl_lock); @@ -324,7 +350,8 @@ static int allocinfo_ioctl_get_at(struct seq_file *m, void __user *arg) ct = codetag_next_ct(&priv->ioctl_iter); while (ct) { - if (matches_filter(ct, &priv->filter)) { + fetched_counters = false; + if (matches_filter(ct, &priv->filter, &counters, &fetched_counters)) { if (skip_count == 0) break; skip_count--; @@ -333,7 +360,9 @@ static int allocinfo_ioctl_get_at(struct seq_file *m, void __user *arg) } if (ct) { - allocinfo_to_params(ct, ¶ms.data); + if (!fetched_counters) + counters = allocinfo_prefetch_counters(ct); + allocinfo_to_params(ct, ¶ms.data, &counters); priv->positioned = true; } @@ -359,6 +388,8 @@ static int allocinfo_ioctl_get_next(struct seq_file *m, void __user *arg) struct codetag *ct; struct allocinfo_tag_data params; int ret = 0; + struct alloc_tag_counters counters; + bool fetched_counters; memset(¶ms, 0, sizeof(params)); priv = m->private; @@ -372,11 +403,18 @@ static int allocinfo_ioctl_get_next(struct seq_file *m, void __user *arg) } ct = codetag_next_ct(&priv->ioctl_iter); - while (ct && !matches_filter(ct, &priv->filter)) + while (ct) { + fetched_counters = false; + if (matches_filter(ct, &priv->filter, &counters, &fetched_counters)) + break; ct = codetag_next_ct(&priv->ioctl_iter); - if (ct) - allocinfo_to_params(ct, ¶ms); + } + if (ct) { + if (!fetched_counters) + counters = allocinfo_prefetch_counters(ct); + allocinfo_to_params(ct, ¶ms, &counters); + } if (!ct) { priv->positioned = false; ret = -ENOENT; -- cgit v1.2.3 From 33588e0b81df2972922f2591767a4e3c16813ab9 Mon Sep 17 00:00:00 2001 From: Abhishek Bapat Date: Wed, 8 Jul 2026 18:01:26 +0000 Subject: alloc_tag: add accuracy based filtering to ioctl Extend the allocinfo filtering mechanism to allow users to filter tags based on their accuracy. [abhishekbapat@google.com: move `inaccurate` filtering criteria from `struct allocinfo_tag` to `struct allocinfo_filter`] Link: https://lore.kernel.org/e4e49ec4a5960292aeeb9e196526c18dc95228a2.1785867739.git.abhishekbapat@google.com Link: https://lore.kernel.org/396a5e4bc3b2990223ab355f2cd3ceb6aa15499e.1783532853.git.abhishekbapat@google.com Signed-off-by: Abhishek Bapat Acked-by: Hao Ge Acked-by: Suren Baghdasaryan Cc: Jonathan Corbet Cc: Kent Overstreet Cc: Sourav Panda Signed-off-by: Andrew Morton --- include/uapi/linux/alloc_tag.h | 4 ++++ mm/alloc_tag.c | 8 ++++++++ 2 files changed, 12 insertions(+) (limited to 'include') diff --git a/include/uapi/linux/alloc_tag.h b/include/uapi/linux/alloc_tag.h index 0de5fc180790..e3ad94444864 100644 --- a/include/uapi/linux/alloc_tag.h +++ b/include/uapi/linux/alloc_tag.h @@ -50,6 +50,7 @@ enum { ALLOCINFO_FILTER_FUNCTION, ALLOCINFO_FILTER_FILENAME, ALLOCINFO_FILTER_LINENO, + ALLOCINFO_FILTER_INACCURATE, ALLOCINFO_FILTER_MIN_SIZE, ALLOCINFO_FILTER_MAX_SIZE, __ALLOCINFO_FILTER_LAST = ALLOCINFO_FILTER_MAX_SIZE @@ -59,6 +60,7 @@ enum { #define ALLOCINFO_FILTER_MASK_FUNCTION (1 << ALLOCINFO_FILTER_FUNCTION) #define ALLOCINFO_FILTER_MASK_FILENAME (1 << ALLOCINFO_FILTER_FILENAME) #define ALLOCINFO_FILTER_MASK_LINENO (1 << ALLOCINFO_FILTER_LINENO) +#define ALLOCINFO_FILTER_MASK_INACCURATE (1 << ALLOCINFO_FILTER_INACCURATE) #define ALLOCINFO_FILTER_MASK_MIN_SIZE (1 << ALLOCINFO_FILTER_MIN_SIZE) #define ALLOCINFO_FILTER_MASK_MAX_SIZE (1 << ALLOCINFO_FILTER_MAX_SIZE) @@ -70,6 +72,8 @@ struct allocinfo_filter { struct allocinfo_tag fields; __u64 min_size; __u64 max_size; + /* filter criteria only; see allocinfo_counter.accurate for actual accuracy */ + __u64 inaccurate; }; struct allocinfo_get_at { diff --git a/mm/alloc_tag.c b/mm/alloc_tag.c index 04b640d74bcc..b33410310477 100644 --- a/mm/alloc_tag.c +++ b/mm/alloc_tag.c @@ -262,6 +262,8 @@ static bool matches_filter(struct codetag *ct, struct allocinfo_filter *filter, struct alloc_tag_counters *counters, bool *fetched_counters) { + bool inaccurate; + if (!filter || !filter->mask) return true; @@ -287,6 +289,12 @@ static bool matches_filter(struct codetag *ct, struct allocinfo_filter *filter, ct->lineno != filter->fields.lineno) return false; + if (filter->mask & ALLOCINFO_FILTER_MASK_INACCURATE) { + inaccurate = !!(ct->flags & CODETAG_FLAG_INACCURATE); + if (inaccurate != !!(filter->inaccurate)) + return false; + } + if (filter->mask & (ALLOCINFO_FILTER_MASK_MIN_SIZE | ALLOCINFO_FILTER_MASK_MAX_SIZE)) { if (!*fetched_counters) { *counters = allocinfo_prefetch_counters(ct); -- cgit v1.2.3 From a8efc69a65fbef61bed0923c0de3425d74817c1c Mon Sep 17 00:00:00 2001 From: Christoph Hellwig Date: Mon, 13 Jul 2026 11:33:38 +0200 Subject: shmem: provide a shmem_write_folio wrapper Patch series "better block swap batching and a different take on swap_ops v5". This series makes use of the swap_iocb for block as well so that it doesn't do inefficient single-bio I/O, and then rebases the swap_ops from Baoquan on top of the now very different method structure. When running doing kernels builds, which is a workload that doesn't really do much THP anonymous memory it still gets 2x clustering for writeout and 1.2x for reading back swap in. The overall times do not actually change, though. This patch (of 7): Provide a wrapper for the shmem abuses in drm to prepare for swap I/O refactoring by keeping swap_iocb handling entirely contained in mm/. Link: https://lore.kernel.org/20260713093350.2154226-1-hch@lst.de Link: https://lore.kernel.org/20260713093350.2154226-2-hch@lst.de Signed-off-by: Christoph Hellwig Reviewed-by: Baoquan He Reviewed-by: Nhat Pham Reviewed-by: Baolin Wang Acked-by: Chris Li Reviewed-by: Kairui Song Cc: Kemeng Shi Cc: Barry Song Cc: Youngjun Park Signed-off-by: Andrew Morton --- drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 2 +- drivers/gpu/drm/ttm/ttm_backup.c | 2 +- include/linux/shmem_fs.h | 5 +---- mm/shmem.c | 7 ++++++- mm/swap.h | 4 ++++ 5 files changed, 13 insertions(+), 7 deletions(-) (limited to 'include') diff --git a/drivers/gpu/drm/i915/gem/i915_gem_shmem.c b/drivers/gpu/drm/i915/gem/i915_gem_shmem.c index 06543ae60706..ef9440166295 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_shmem.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_shmem.c @@ -325,7 +325,7 @@ void __shmem_writeback(size_t size, struct address_space *mapping) if (folio_mapped(folio)) folio_redirty_for_writepage(&wbc, folio); else - error = shmem_writeout(folio, NULL, NULL); + error = shmem_write_folio(folio); } } diff --git a/drivers/gpu/drm/ttm/ttm_backup.c b/drivers/gpu/drm/ttm/ttm_backup.c index 3c067aadc52d..0c2d53a13b2a 100644 --- a/drivers/gpu/drm/ttm/ttm_backup.c +++ b/drivers/gpu/drm/ttm/ttm_backup.c @@ -160,7 +160,7 @@ ttm_backup_backup_folio(struct file *backup, struct folio *folio, if (writeback && !folio_mapped(to_folio) && folio_clear_dirty_for_io(to_folio)) { folio_set_reclaim(to_folio); - ret = shmem_writeout(to_folio, NULL, NULL); + ret = shmem_write_folio(to_folio); if (!folio_test_writeback(to_folio)) folio_clear_reclaim(to_folio); if (ret == AOP_WRITEPAGE_ACTIVATE) diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h index e729b9b0e38d..5663dff53186 100644 --- a/include/linux/shmem_fs.h +++ b/include/linux/shmem_fs.h @@ -12,8 +12,6 @@ #include #include -struct swap_iocb; - /* inode in-kernel data */ #ifdef CONFIG_TMPFS_QUOTA @@ -123,8 +121,7 @@ static inline bool shmem_mapping(const struct address_space *mapping) void shmem_unlock_mapping(struct address_space *mapping); struct page *shmem_read_mapping_page_gfp(struct address_space *mapping, pgoff_t index, gfp_t gfp_mask); -int shmem_writeout(struct folio *folio, struct swap_iocb **plug, - struct list_head *folio_list); +int shmem_write_folio(struct folio *folio); void shmem_truncate_range(struct inode *inode, loff_t start, uoff_t end); int shmem_unuse(unsigned int type); diff --git a/mm/shmem.c b/mm/shmem.c index 8ea776e52823..d245e01416e9 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1751,7 +1751,12 @@ redirty: folio_mark_dirty(folio); return AOP_WRITEPAGE_ACTIVATE; /* Return with folio locked */ } -EXPORT_SYMBOL_GPL(shmem_writeout); + +int shmem_write_folio(struct folio *folio) +{ + return shmem_writeout(folio, NULL, NULL); +} +EXPORT_SYMBOL_GPL(shmem_write_folio); #if defined(CONFIG_NUMA) && defined(CONFIG_TMPFS) static void shmem_show_mpol(struct seq_file *seq, struct mempolicy *mpol) diff --git a/mm/swap.h b/mm/swap.h index 4e4c291bbfde..276b7975a9dc 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -486,4 +486,8 @@ static inline unsigned int folio_swap_flags(struct folio *folio) } #endif /* CONFIG_SWAP */ + +int shmem_writeout(struct folio *folio, struct swap_iocb **plug, + struct list_head *folio_list); + #endif /* _MM_SWAP_H */ -- cgit v1.2.3 From 563597895e65113b4dafea6cacd1df23b0cf6660 Mon Sep 17 00:00:00 2001 From: Christoph Hellwig Date: Mon, 13 Jul 2026 11:33:42 +0200 Subject: mm/swap: use swap_ops to register swap device's methods This simplifies codes and makes logic clearer. And also makes later any new swap device type being added easier to handle. Currently there are two types of swap devices: fs and bdev. [hch@lst.de: updated for the new submit and can_merge abstraction] Link: https://lore.kernel.org/20260713093350.2154226-6-hch@lst.de Signed-off-by: Baoquan He Signed-off-by: Christoph Hellwig Suggested-by: Chris Li Reviewed-by: Nhat Pham Cc: Baolin Wang Cc: Barry Song Cc: Kairui Song Cc: Kemeng Shi Cc: Youngjun Park Signed-off-by: Andrew Morton --- include/linux/swap.h | 1 + mm/page_io.c | 68 +++++++++++++++++++++++++++++++++------------------- mm/swap.h | 10 ++++++++ mm/swapfile.c | 4 ++++ 4 files changed, 58 insertions(+), 25 deletions(-) (limited to 'include') diff --git a/include/linux/swap.h b/include/linux/swap.h index b4b1c0a84c8b..5979b1427368 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -276,6 +276,7 @@ struct swap_info_struct { struct work_struct reclaim_work; /* reclaim worker */ struct list_head discard_clusters; /* discard clusters list */ struct plist_node avail_list; /* entry in swap_avail_head */ + const struct swap_ops *ops; }; static inline swp_entry_t page_swap_entry(struct page *page) diff --git a/mm/page_io.c b/mm/page_io.c index 56f21e49572e..c36b44ffe947 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -334,21 +334,7 @@ static bool swap_can_merge(struct swap_io_ctx *ctx, struct folio *folio, if (ctx->sis != sis) return false; - - if (sis->flags & SWP_FS_OPS) { - if (swap_dev_pos(folio->swap) != - swap_dev_pos(prev_folio->swap) + prev_folio_size) - return false; - } else { - if (swap_folio_sector(folio) != - swap_folio_sector(prev_folio) + - (prev_folio_size >> SECTOR_SHIFT)) - return false; - if (rw == WRITE && !folio_blkg_can_merge(folio, prev_folio)) - return false; - } - - return true; + return sis->ops->can_merge(folio, prev_folio, prev_folio_size, rw); } static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw) @@ -646,6 +632,23 @@ static void swap_bdev_submit_read(struct swap_io_ctx *ctx) } } +static bool swap_bdev_can_merge(struct folio *folio, struct folio *prev_folio, + size_t prev_folio_size, int rw) +{ + if (swap_folio_sector(folio) != + swap_folio_sector(prev_folio) + (prev_folio_size >> SECTOR_SHIFT)) + return false; + if (rw == WRITE && !folio_blkg_can_merge(folio, prev_folio)) + return false; + return true; +} + +const struct swap_ops swap_bdev_ops = { + .submit_write = swap_bdev_submit_write, + .submit_read = swap_bdev_submit_read, + .can_merge = swap_bdev_can_merge, +}; + static void swap_fs_submit(struct swap_io_ctx *ctx, int rw) { struct swap_iocb *sio = ctx->sio; @@ -666,15 +669,34 @@ static void swap_fs_submit(struct swap_io_ctx *ctx, int rw) sio->iocb.ki_complete(&sio->iocb, ret); } +static void swap_fs_submit_write(struct swap_io_ctx *ctx) +{ + swap_fs_submit(ctx, WRITE); +} + +static void swap_fs_submit_read(struct swap_io_ctx *ctx) +{ + swap_fs_submit(ctx, READ); +} + +static bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio, + size_t prev_folio_size, int rw) +{ + return swap_dev_pos(folio->swap) == + swap_dev_pos(prev_folio->swap) + prev_folio_size; +} + +const struct swap_ops swap_fs_ops = { + .submit_write = swap_fs_submit_write, + .submit_read = swap_fs_submit_read, + .can_merge = swap_fs_can_merge, +}; + void swap_write_submit(struct swap_io_ctx *ctx) { if (!ctx->sio) return; - - if (ctx->sis->flags & SWP_FS_OPS) - swap_fs_submit(ctx, WRITE); - else - swap_bdev_submit_write(ctx); + ctx->sis->ops->submit_write(ctx); ctx->sio = NULL; ctx->sis = NULL; } @@ -683,11 +705,7 @@ void swap_read_submit(struct swap_io_ctx *ctx) { if (!ctx->sio) return; - - if (ctx->sis->flags & SWP_FS_OPS) - swap_fs_submit(ctx, READ); - else - swap_bdev_submit_read(ctx); + ctx->sis->ops->submit_read(ctx); ctx->sio = NULL; ctx->sis = NULL; } diff --git a/mm/swap.h b/mm/swap.h index 86b2a241b734..ffc36695d4ac 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -96,6 +96,13 @@ struct swap_io_ctx { struct swap_info_struct *sis; }; +struct swap_ops { + bool (*can_merge)(struct folio *folio, struct folio *prev_folio, + size_t prev_folio_size, int rw); + void (*submit_write)(struct swap_io_ctx *ctx); + void (*submit_read)(struct swap_io_ctx *ctx); +}; + #ifdef CONFIG_SWAP #include /* for swp_offset */ #include /* for bio_end_io_t */ @@ -483,6 +490,9 @@ static inline unsigned int folio_swap_flags(struct folio *folio) #endif /* CONFIG_SWAP */ +extern const struct swap_ops swap_bdev_ops; +extern const struct swap_ops swap_fs_ops; + int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio, struct list_head *folio_list); diff --git a/mm/swapfile.c b/mm/swapfile.c index be75c995c49b..ad623dae483b 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -2963,6 +2963,8 @@ static int setup_swap_extents(struct swap_info_struct *sis, if (ret) return ret; + sis->ops = &swap_bdev_ops; + if (S_ISBLK(inode->i_mode)) { ret = add_swap_extent(sis, 0, sis->max, 0); *span = sis->pages; @@ -2973,6 +2975,8 @@ static int setup_swap_extents(struct swap_info_struct *sis, ret = mapping->a_ops->swap_activate(sis, swap_file, span); if (ret < 0) return ret; + if (sis->flags & SWP_FS_OPS) + sis->ops = &swap_fs_ops; sis->flags |= SWP_ACTIVATED; return ret; } -- cgit v1.2.3 From 0df74c11587941b35596d1e8990dcab06bdbfeb5 Mon Sep 17 00:00:00 2001 From: Christoph Hellwig Date: Mon, 13 Jul 2026 11:33:43 +0200 Subject: mm/swap: remove SWP_FS_OPS Provide a swap_fs_activate helper that directly sets up swap_fs_ops, and a flag in struct swap_ops to indicate of NOFS swapping is allowed. Link: https://lore.kernel.org/20260713093350.2154226-7-hch@lst.de Signed-off-by: Christoph Hellwig Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Youngjun Park Signed-off-by: Andrew Morton --- Documentation/filesystems/locking.rst | 5 +++-- Documentation/filesystems/vfs.rst | 4 ++-- fs/nfs/file.c | 4 +--- fs/smb/client/file.c | 4 +--- include/linux/swap.h | 6 +++++- mm/page_io.c | 10 +++++++++- mm/swap.h | 22 ++++++++++------------ mm/swapfile.c | 2 -- mm/vmscan.c | 15 +++++++-------- 9 files changed, 38 insertions(+), 34 deletions(-) (limited to 'include') diff --git a/Documentation/filesystems/locking.rst b/Documentation/filesystems/locking.rst index 08d01bc62c31..1a50d41a39a1 100644 --- a/Documentation/filesystems/locking.rst +++ b/Documentation/filesystems/locking.rst @@ -355,13 +355,14 @@ should perform any validation and preparation necessary to ensure that writes can be performed with minimal memory allocation. It should call add_swap_extent(), or the helper iomap_swapfile_activate(), and return the number of extents added. If IO should be submitted through -->swap_rw(), it should set SWP_FS_OPS, otherwise IO will be submitted +->swap_rw(), it should call swap_fs_activate, otherwise IO will be submitted directly to the block device ``sis->bdev``. ->swap_deactivate() will be called in the sys_swapoff() path after ->swap_activate() returned success. -->swap_rw will be called for swap IO if SWP_FS_OPS was set by ->swap_activate(). +->swap_rw will be called for swap IO if swap_fs_activate was called by +->swap_activate(). file_lock_operations ==================== diff --git a/Documentation/filesystems/vfs.rst b/Documentation/filesystems/vfs.rst index 7c753148af88..e7677423a20f 100644 --- a/Documentation/filesystems/vfs.rst +++ b/Documentation/filesystems/vfs.rst @@ -977,7 +977,7 @@ cache in your filesystem. The following members are defined: can be performed with minimal memory allocation. It should call add_swap_extent(), or the helper iomap_swapfile_activate(), and return the number of extents added. If IO should be submitted - through ->swap_rw(), it should set SWP_FS_OPS, otherwise IO will + through ->swap_rw(), it should call swap_fs_activate, otherwise IO will be submitted directly to the block device ``sis->bdev``. ``swap_deactivate`` @@ -985,7 +985,7 @@ cache in your filesystem. The following members are defined: successful. ``swap_rw`` - Called to read or write swap pages when SWP_FS_OPS is set. + Called to read or write swap pages when swap_fs_activate was called. The File Object =============== diff --git a/fs/nfs/file.c b/fs/nfs/file.c index a0d8f1c1cf10..851d93a09988 100644 --- a/fs/nfs/file.c +++ b/fs/nfs/file.c @@ -597,7 +597,7 @@ static int nfs_swap_activate(struct swap_info_struct *sis, struct file *file, ret = rpc_clnt_swap_activate(clnt); if (ret) return ret; - ret = add_swap_extent(sis, 0, sis->max, 0); + ret = swap_fs_activate(sis); if (ret < 0) { rpc_clnt_swap_deactivate(clnt); return ret; @@ -607,8 +607,6 @@ static int nfs_swap_activate(struct swap_info_struct *sis, struct file *file, if (cl->rpc_ops->enable_swap) cl->rpc_ops->enable_swap(inode); - - sis->flags |= SWP_FS_OPS; return ret; } diff --git a/fs/smb/client/file.c b/fs/smb/client/file.c index b279a44be729..7f2924ce2881 100644 --- a/fs/smb/client/file.c +++ b/fs/smb/client/file.c @@ -3451,9 +3451,7 @@ static int cifs_swap_activate(struct swap_info_struct *sis, * but we could add call to grab a byte range lock to prevent others * from reading or writing the file */ - - sis->flags |= SWP_FS_OPS; - return add_swap_extent(sis, 0, sis->max, 0); + return swap_fs_activate(sis); } static void cifs_swap_deactivate(struct file *file) diff --git a/include/linux/swap.h b/include/linux/swap.h index 5979b1427368..8dd68733c955 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -202,7 +202,6 @@ enum { SWP_SOLIDSTATE = (1 << 4), /* blkdev seeks are cheap */ SWP_BLKDEV = (1 << 6), /* its a block device */ SWP_ACTIVATED = (1 << 7), /* set after swap_activate success */ - SWP_FS_OPS = (1 << 8), /* swapfile operations go through fs */ SWP_AREA_DISCARD = (1 << 9), /* single-time swap area discards */ SWP_PAGE_DISCARD = (1 << 10), /* freed swap page-cluster discards */ SWP_STABLE_WRITES = (1 << 11), /* no overwrite PG_writeback pages */ @@ -343,6 +342,7 @@ extern void __meminit kswapd_stop(int nid); #ifdef CONFIG_SWAP +int swap_fs_activate(struct swap_info_struct *sis); int add_swap_extent(struct swap_info_struct *sis, unsigned long start_page, unsigned long nr_pages, sector_t start_block); int generic_swapfile_activate(struct swap_info_struct *, struct file *, @@ -468,6 +468,10 @@ static inline bool folio_free_swap(struct folio *folio) return false; } +static inline int swap_fs_activate(struct swap_info_struct *sis) +{ + return -EINVAL; +} static inline int add_swap_extent(struct swap_info_struct *sis, unsigned long start_page, unsigned long nr_pages, sector_t start_block) diff --git a/mm/page_io.c b/mm/page_io.c index c36b44ffe947..cea438b66bce 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -686,12 +686,20 @@ static bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio, swap_dev_pos(prev_folio->swap) + prev_folio_size; } -const struct swap_ops swap_fs_ops = { +static const struct swap_ops swap_fs_ops = { + .flags = SWAP_OPS_F_REQUIRE_NOFS, .submit_write = swap_fs_submit_write, .submit_read = swap_fs_submit_read, .can_merge = swap_fs_can_merge, }; +int swap_fs_activate(struct swap_info_struct *sis) +{ + sis->ops = &swap_fs_ops; + return add_swap_extent(sis, 0, sis->max, 0); +} +EXPORT_SYMBOL_GPL(swap_fs_activate); + void swap_write_submit(struct swap_io_ctx *ctx) { if (!ctx->sio) diff --git a/mm/swap.h b/mm/swap.h index ffc36695d4ac..1a78578fd067 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -96,7 +96,17 @@ struct swap_io_ctx { struct swap_info_struct *sis; }; +/* + * SWAP_OPS_F_REQUIRE_NOFS: + * When set, all reclaim operations must operated as GFS_NOFS and not + * just GFP_NOIO, as GFP_NOIO allocations could recourse into the + * file system backing this swap file. + */ +#define SWAP_OPS_F_REQUIRE_NOFS (1U << 0) + struct swap_ops { + unsigned int flags; + bool (*can_merge)(struct folio *folio, struct folio *prev_folio, size_t prev_folio_size, int rw); void (*submit_write)(struct swap_io_ctx *ctx); @@ -347,11 +357,6 @@ struct folio *swapin_sync(swp_entry_t entry, gfp_t flag, unsigned long orders, void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma, unsigned long addr); -static inline unsigned int folio_swap_flags(struct folio *folio) -{ - return __swap_entry_to_info(folio->swap)->flags; -} - #else /* CONFIG_SWAP */ static inline struct swap_cluster_info *swap_cluster_lock( struct swap_info_struct *si, pgoff_t offset, bool irq) @@ -482,16 +487,9 @@ static inline void __swap_cache_replace_folio(struct swap_cluster_info *ci, struct folio *old, struct folio *new) { } - -static inline unsigned int folio_swap_flags(struct folio *folio) -{ - return 0; -} - #endif /* CONFIG_SWAP */ extern const struct swap_ops swap_bdev_ops; -extern const struct swap_ops swap_fs_ops; int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio, struct list_head *folio_list); diff --git a/mm/swapfile.c b/mm/swapfile.c index ad623dae483b..dacef34a3ed7 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -2975,8 +2975,6 @@ static int setup_swap_extents(struct swap_info_struct *sis, ret = mapping->a_ops->swap_activate(sis, swap_file, span); if (ret < 0) return ret; - if (sis->flags & SWP_FS_OPS) - sis->ops = &swap_fs_ops; sis->flags |= SWP_ACTIVATED; return ret; } diff --git a/mm/vmscan.c b/mm/vmscan.c index 4742297693fe..3194da7dcc79 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1040,16 +1040,15 @@ static bool may_enter_fs(struct folio *folio, gfp_t gfp_mask) { if (gfp_mask & __GFP_FS) return true; - if (!folio_test_swapcache(folio) || !(gfp_mask & __GFP_IO)) - return false; /* - * We can "enter_fs" for swap-cache with only __GFP_IO - * providing this isn't SWP_FS_OPS. - * ->flags can be updated non-atomically, - * but that will never affect SWP_FS_OPS, so the data_race - * is safe. + * We can "enter_fs" for swap-cache with only __GFP_IO unless backed by + * a swapfile that requires GFP_NOFS I/O. */ - return !data_race(folio_swap_flags(folio) & SWP_FS_OPS); + if (folio_test_swapcache(folio) && (gfp_mask & __GFP_IO) && + !(__swap_entry_to_info(folio->swap)->ops->flags & + SWAP_OPS_F_REQUIRE_NOFS)) + return true; + return false; } /* -- cgit v1.2.3 From c01e6df60e7be422ee5ce5e2a76d43fb05fab4c2 Mon Sep 17 00:00:00 2001 From: Christoph Hellwig Date: Mon, 13 Jul 2026 11:33:44 +0200 Subject: mm/vmstat: add NRSWP{IN,OUT} counters Count how many swap I/Os we cause. Due to batching this can be different than the current counter number of pages written/read, and tracking this information is useful to see how efficient the batching is. The counters are added at the end of enum vm_event_item and the vmstat_text array under the assumption that the order of fields in /proc/vmstat is an ABI. If that is not the case, they could be grouped with the other swap counters. Link: https://lore.kernel.org/20260713093350.2154226-8-hch@lst.de Signed-off-by: Christoph Hellwig Reviewed-by: Nhat Pham Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Youngjun Park Signed-off-by: Andrew Morton --- include/linux/vm_event_item.h | 4 ++++ mm/page_io.c | 2 ++ mm/vmstat.c | 6 +++++- 3 files changed, 11 insertions(+), 1 deletion(-) (limited to 'include') diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 03fe95f5a020..2628ccda076a 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -175,6 +175,10 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, KSTACK_REST, #endif #endif /* CONFIG_DEBUG_STACK_USAGE */ +#ifdef CONFIG_SWAP + NRSWPIN, + NRSWPOUT, +#endif /* CONFIG_SWAP */ NR_VM_EVENT_ITEMS }; diff --git a/mm/page_io.c b/mm/page_io.c index cea438b66bce..e4fa7ffffe8b 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -704,6 +704,7 @@ void swap_write_submit(struct swap_io_ctx *ctx) { if (!ctx->sio) return; + count_vm_events(NRSWPOUT, 1); ctx->sis->ops->submit_write(ctx); ctx->sio = NULL; ctx->sis = NULL; @@ -713,6 +714,7 @@ void swap_read_submit(struct swap_io_ctx *ctx) { if (!ctx->sio) return; + count_vm_events(NRSWPIN, 1); ctx->sis->ops->submit_read(ctx); ctx->sio = NULL; ctx->sis = NULL; diff --git a/mm/vmstat.c b/mm/vmstat.c index 7d6e61a01f51..cb57714539fb 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1502,7 +1502,11 @@ const char * const vmstat_text[] = { #if THREAD_SIZE > 65536 [I(KSTACK_REST)] = "kstack_rest", #endif -#endif +#endif /* CONFIG_DEBUG_STACK_USAGE */ +#ifdef CONFIG_SWAP + [I(NRSWPIN)] = "nrswpin", + [I(NRSWPOUT)] = "nrswpout", +#endif /* CONFIG_SWAP */ #undef I #endif /* CONFIG_VM_EVENT_COUNTERS */ }; -- cgit v1.2.3 From 52d85ca90c7db41f8cfc72e5fe7dc62c2f983032 Mon Sep 17 00:00:00 2001 From: Christoph Hellwig Date: Thu, 23 Jul 2026 07:46:05 +0200 Subject: mm/swap: add a new swap_ops.h header to allow for pluggable swap ops Add a new header to declare the swap_iocb, swap_ops and swap_ctx to allow for swap_ops implementations outside of mm/page_io.c. This will be used to remove the double indirection for file system-based swap. There is no functional change, just a move of the declarations. Note that there already is a swapops.h header, which is totally unrelated to struct swap_ops. The close naming is a bit unfortunate, but I could not think of a better name for this header. Link: https://lore.kernel.org/20260723054622.3460249-3-hch@lst.de Signed-off-by: Christoph Hellwig Acked-by: Chris Li Cc: Baoquan He Cc: Kairui Song Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Steve French Cc: Usama Arif Signed-off-by: Andrew Morton --- MAINTAINERS | 1 + include/linux/swap_ops.h | 39 +++++++++++++++++++++++++++++++++++++++ mm/madvise.c | 1 + mm/page_io.c | 10 +--------- mm/shmem.c | 1 + mm/swap.h | 23 +---------------------- mm/swap_state.c | 1 + mm/vmscan.c | 1 + mm/zswap.c | 2 +- 9 files changed, 47 insertions(+), 32 deletions(-) create mode 100644 include/linux/swap_ops.h (limited to 'include') diff --git a/MAINTAINERS b/MAINTAINERS index 4899b81bd839..604285d848e6 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17242,6 +17242,7 @@ S: Maintained F: Documentation/ABI/testing/sysfs-kernel-mm-swap F: Documentation/mm/swap-table.rst F: include/linux/swap.h +F: include/linux/swap_ops.h F: include/linux/swapfile.h F: include/linux/swapops.h F: mm/page_io.c diff --git a/include/linux/swap_ops.h b/include/linux/swap_ops.h new file mode 100644 index 000000000000..e92b4f532604 --- /dev/null +++ b/include/linux/swap_ops.h @@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _MM_SWAP_OPS_H +#define _MM_SWAP_OPS_H + +#include /* for SWAP_CLUSTER_MAX */ + +struct swap_iocb { + union { + struct kiocb iocb; + struct bio bio; + }; + struct bio_vec bvecs[SWAP_CLUSTER_MAX]; + int nr_bvecs; + int len; +}; + +struct swap_io_ctx { + struct swap_iocb *sio; + struct swap_info_struct *sis; +}; + +/* + * SWAP_OPS_F_REQUIRE_NOFS: + * When set, all reclaim operations must operated as GFS_NOFS and not + * just GFP_NOIO, as GFP_NOIO allocations could recourse into the + * file system backing this swap file. + */ +#define SWAP_OPS_F_REQUIRE_NOFS (1U << 0) + +struct swap_ops { + unsigned int flags; + + bool (*can_merge)(struct folio *folio, struct folio *prev_folio, + size_t prev_folio_size, int rw); + void (*submit_write)(struct swap_io_ctx *ctx); + void (*submit_read)(struct swap_io_ctx *ctx); +}; + +#endif /* _MM_SWAP_OPS_H */ diff --git a/mm/madvise.c b/mm/madvise.c index 07a21ca31bad..c179938097bf 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -32,6 +32,7 @@ #include #include #include +#include #include diff --git a/mm/page_io.c b/mm/page_io.c index c984a4023a65..e741e67d6592 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -25,6 +25,7 @@ #include #include #include +#include #include "swap.h" #include "swap_table.h" @@ -300,15 +301,6 @@ static bool folio_blkg_can_merge(struct folio *folio, struct folio *prev_folio) #define bio_associate_blkg_from_page(bio, folio) do { } while (0) #endif /* CONFIG_MEMCG && CONFIG_BLK_CGROUP */ -struct swap_iocb { - union { - struct kiocb iocb; - struct bio bio; - }; - struct bio_vec bvecs[SWAP_CLUSTER_MAX]; - int nr_bvecs; - int len; -}; static mempool_t *sio_pool; int sio_pool_init(void) diff --git a/mm/shmem.c b/mm/shmem.c index 2e4dacdcce11..599665a3d6e7 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -41,6 +41,7 @@ #include #include #include +#include #include "swap.h" static struct vfsmount *shm_mnt __ro_after_init; diff --git a/mm/swap.h b/mm/swap.h index 48379b2ab202..90a551a88df6 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -10,6 +10,7 @@ struct mempolicy; struct swap_iocb; struct swap_memcg_table; +struct swap_io_ctx; #if defined(MAX_POSSIBLE_PHYSMEM_BITS) #define SWAP_CACHE_PFN_BITS (MAX_POSSIBLE_PHYSMEM_BITS - PAGE_SHIFT) @@ -91,28 +92,6 @@ static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg) return READ_ONCE(vm_swappiness); } -struct swap_io_ctx { - struct swap_iocb *sio; - struct swap_info_struct *sis; -}; - -/* - * SWAP_OPS_F_REQUIRE_NOFS: - * When set, all reclaim operations must operated as GFS_NOFS and not - * just GFP_NOIO, as GFP_NOIO allocations could recourse into the - * file system backing this swap file. - */ -#define SWAP_OPS_F_REQUIRE_NOFS (1U << 0) - -struct swap_ops { - unsigned int flags; - - bool (*can_merge)(struct folio *folio, struct folio *prev_folio, - size_t prev_folio_size, int rw); - void (*submit_write)(struct swap_io_ctx *ctx); - void (*submit_read)(struct swap_io_ctx *ctx); -}; - #ifdef CONFIG_SWAP #include /* for swp_offset */ #include /* for bio_end_io_t */ diff --git a/mm/swap_state.c b/mm/swap_state.c index 5be825911e64..b76eb3d876fd 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -23,6 +23,7 @@ #include #include #include +#include #include "internal.h" #include "swap_table.h" #include "swap.h" diff --git a/mm/vmscan.c b/mm/vmscan.c index 3194da7dcc79..be6bd26e8c57 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -58,6 +58,7 @@ #include #include #include +#include #include #include diff --git a/mm/zswap.c b/mm/zswap.c index 9f777a48b106..37f34e406c8e 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -20,7 +20,7 @@ #include #include #include -#include +#include #include #include #include -- cgit v1.2.3 From 22779ae8175aad7c04827db44e934e53bf2bd2d4 Mon Sep 17 00:00:00 2001 From: Christoph Hellwig Date: Thu, 23 Jul 2026 07:46:06 +0200 Subject: mm/swap: move swap_ops into file systems for file system-based swap Currently swap to and from file systems goes through two indirect calls between the swap ops and the swap_rw method. Reduce this by directly providing the swap_ops from the file system. For this refactor swap_fs_submit into a swap_fs_prepare_rw helper that initializes the iov_iter on the callers stack so that file systems can call it directly, and use that to initialize file system specific ops in the NFS and SMB clients, which then get passed to swap_fs_activate. Link: https://lore.kernel.org/20260723054622.3460249-4-hch@lst.de Signed-off-by: Christoph Hellwig Acked-by: Chris Li Cc: Baoquan He Cc: Kairui Song Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Steve French Cc: Usama Arif Signed-off-by: Andrew Morton --- Documentation/filesystems/locking.rst | 9 ++--- Documentation/filesystems/vfs.rst | 8 ++--- fs/nfs/direct.c | 20 ----------- fs/nfs/file.c | 42 ++++++++++++++++++++--- fs/smb/client/file.c | 63 ++++++++++++++++++++++------------- include/linux/fs.h | 1 - include/linux/nfs_fs.h | 1 - include/linux/swap.h | 6 ---- include/linux/swap_ops.h | 5 +++ mm/page_io.c | 34 ++++--------------- 10 files changed, 93 insertions(+), 96 deletions(-) (limited to 'include') diff --git a/Documentation/filesystems/locking.rst b/Documentation/filesystems/locking.rst index 1a50d41a39a1..f58a8d7d5897 100644 --- a/Documentation/filesystems/locking.rst +++ b/Documentation/filesystems/locking.rst @@ -266,7 +266,6 @@ prototypes:: int (*error_remove_folio)(struct address_space *, struct folio *); int (*swap_activate)(struct swap_info_struct *sis, struct file *f, sector_t *span) int (*swap_deactivate)(struct file *); - int (*swap_rw)(struct kiocb *iocb, struct iov_iter *iter); locking rules: All except dirty_folio and free_folio may block @@ -291,7 +290,6 @@ is_partially_uptodate: yes error_remove_folio: yes swap_activate: no swap_deactivate: no -swap_rw: yes, unlocks ====================== ======================== ========= =============== ->write_begin(), ->write_end() and ->read_folio() may be called from @@ -355,15 +353,12 @@ should perform any validation and preparation necessary to ensure that writes can be performed with minimal memory allocation. It should call add_swap_extent(), or the helper iomap_swapfile_activate(), and return the number of extents added. If IO should be submitted through -->swap_rw(), it should call swap_fs_activate, otherwise IO will be submitted -directly to the block device ``sis->bdev``. +the file system it should call swap_fs_activate, otherwise IO will be +submitted directly to the block device ``sis->bdev``. ->swap_deactivate() will be called in the sys_swapoff() path after ->swap_activate() returned success. -->swap_rw will be called for swap IO if swap_fs_activate was called by -->swap_activate(). - file_lock_operations ==================== diff --git a/Documentation/filesystems/vfs.rst b/Documentation/filesystems/vfs.rst index e7677423a20f..c437a342d4f3 100644 --- a/Documentation/filesystems/vfs.rst +++ b/Documentation/filesystems/vfs.rst @@ -776,7 +776,6 @@ cache in your filesystem. The following members are defined: int (*error_remove_folio)(struct mapping *mapping, struct folio *); int (*swap_activate)(struct swap_info_struct *sis, struct file *f, sector_t *span) int (*swap_deactivate)(struct file *); - int (*swap_rw)(struct kiocb *iocb, struct iov_iter *iter); }; ``read_folio`` @@ -977,16 +976,13 @@ cache in your filesystem. The following members are defined: can be performed with minimal memory allocation. It should call add_swap_extent(), or the helper iomap_swapfile_activate(), and return the number of extents added. If IO should be submitted - through ->swap_rw(), it should call swap_fs_activate, otherwise IO will - be submitted directly to the block device ``sis->bdev``. + through the file system it should call swap_fs_activate, otherwise IO + will be submitted directly to the block device ``sis->bdev``. ``swap_deactivate`` Called during swapoff on files where swap_activate was successful. -``swap_rw`` - Called to read or write swap pages when swap_fs_activate was called. - The File Object =============== diff --git a/fs/nfs/direct.c b/fs/nfs/direct.c index e626c72495e6..ccafdc1ce64d 100644 --- a/fs/nfs/direct.c +++ b/fs/nfs/direct.c @@ -145,26 +145,6 @@ static void nfs_direct_file_adjust_size_locked(struct inode *inode, } } -/** - * nfs_swap_rw - NFS address space operation for swap I/O - * @iocb: target I/O control block - * @iter: I/O buffer - * - * Perform IO to the swap-file. This is much like direct IO. - */ -int nfs_swap_rw(struct kiocb *iocb, struct iov_iter *iter) -{ - ssize_t ret; - - if (iov_iter_rw(iter) == READ) - ret = nfs_file_direct_read(iocb, iter, true); - else - ret = nfs_file_direct_write(iocb, iter, true); - if (ret < 0) - return ret; - return 0; -} - static void nfs_direct_release_pages(struct page **pages, unsigned int npages) { unsigned int i; diff --git a/fs/nfs/file.c b/fs/nfs/file.c index 851d93a09988..e1bdd10b35f1 100644 --- a/fs/nfs/file.c +++ b/fs/nfs/file.c @@ -29,9 +29,8 @@ #include #include #include -#include #include - +#include #include #include @@ -575,6 +574,38 @@ static int nfs_launder_folio(struct folio *folio) return ret; } +#ifdef CONFIG_SWAP +static void nfs_swap_submit_write(struct swap_io_ctx *ctx) +{ + struct swap_iocb *sio = ctx->sio; + struct iov_iter iter; + int ret; + + swap_fs_prepare_rw(ctx, WRITE, &iter); + ret = nfs_file_direct_write(&sio->iocb, &iter, true); + if (ret != -EIOCBQUEUED) + sio->iocb.ki_complete(&sio->iocb, ret); +} + +static void nfs_swap_submit_read(struct swap_io_ctx *ctx) +{ + struct swap_iocb *sio = ctx->sio; + struct iov_iter iter; + int ret; + + swap_fs_prepare_rw(ctx, READ, &iter); + ret = nfs_file_direct_read(&sio->iocb, &iter, true); + if (ret != -EIOCBQUEUED) + sio->iocb.ki_complete(&sio->iocb, ret); +} + +static const struct swap_ops nfs_swap_ops = { + .flags = SWAP_OPS_F_REQUIRE_NOFS, + .submit_write = nfs_swap_submit_write, + .submit_read = nfs_swap_submit_read, + .can_merge = swap_fs_can_merge, +}; + static int nfs_swap_activate(struct swap_info_struct *sis, struct file *file, sector_t *span) { @@ -597,7 +628,7 @@ static int nfs_swap_activate(struct swap_info_struct *sis, struct file *file, ret = rpc_clnt_swap_activate(clnt); if (ret) return ret; - ret = swap_fs_activate(sis); + ret = swap_fs_activate(sis, &nfs_swap_ops); if (ret < 0) { rpc_clnt_swap_deactivate(clnt); return ret; @@ -620,6 +651,10 @@ static void nfs_swap_deactivate(struct file *file) if (cl->rpc_ops->disable_swap) cl->rpc_ops->disable_swap(file_inode(file)); } +#else +#define nfs_swap_activate NULL +#define nfs_swap_deactivate NULL +#endif /* CONFIG_SWAP */ const struct address_space_operations nfs_file_aops = { .read_folio = nfs_read_folio, @@ -636,7 +671,6 @@ const struct address_space_operations nfs_file_aops = { .error_remove_folio = generic_error_remove_folio, .swap_activate = nfs_swap_activate, .swap_deactivate = nfs_swap_deactivate, - .swap_rw = nfs_swap_rw, }; /* diff --git a/fs/smb/client/file.c b/fs/smb/client/file.c index 7f2924ce2881..ead69232ac1c 100644 --- a/fs/smb/client/file.c +++ b/fs/smb/client/file.c @@ -20,7 +20,7 @@ #include #include #include -#include +#include #include #include #include "cifsfs.h" @@ -3410,6 +3410,38 @@ out: cifs_done_oplock_break(cinode); } +#ifdef CONFIG_SWAP +static void cifs_swap_submit_write(struct swap_io_ctx *ctx) +{ + struct swap_iocb *sio = ctx->sio; + struct iov_iter iter; + int ret; + + swap_fs_prepare_rw(ctx, WRITE, &iter); + ret = netfs_unbuffered_write_iter_locked(&sio->iocb, &iter, NULL); + if (ret != -EIOCBQUEUED) + sio->iocb.ki_complete(&sio->iocb, ret); +} + +static void cifs_swap_submit_read(struct swap_io_ctx *ctx) +{ + struct swap_iocb *sio = ctx->sio; + struct iov_iter iter; + int ret; + + swap_fs_prepare_rw(ctx, READ, &iter); + ret = netfs_unbuffered_read_iter_locked(&sio->iocb, &iter); + if (ret != -EIOCBQUEUED) + sio->iocb.ki_complete(&sio->iocb, ret); +} + +static const struct swap_ops cifs_swap_ops = { + .flags = SWAP_OPS_F_REQUIRE_NOFS, + .submit_write = cifs_swap_submit_write, + .submit_read = cifs_swap_submit_read, + .can_merge = swap_fs_can_merge, +}; + static int cifs_swap_activate(struct swap_info_struct *sis, struct file *swap_file, sector_t *span) { @@ -3420,7 +3452,7 @@ static int cifs_swap_activate(struct swap_info_struct *sis, cifs_dbg(FYI, "swap activate\n"); - if (!swap_file->f_mapping->a_ops->swap_rw) + if (swap_file->f_mapping->a_ops != &cifs_addr_ops) /* Cannot support swap */ return -EINVAL; @@ -3451,7 +3483,7 @@ static int cifs_swap_activate(struct swap_info_struct *sis, * but we could add call to grab a byte range lock to prevent others * from reading or writing the file */ - return swap_fs_activate(sis); + return swap_fs_activate(sis, &cifs_swap_ops); } static void cifs_swap_deactivate(struct file *file) @@ -3467,26 +3499,10 @@ static void cifs_swap_deactivate(struct file *file) /* do we need to unpin (or unlock) the file */ } - -/** - * cifs_swap_rw - SMB3 address space operation for swap I/O - * @iocb: target I/O control block - * @iter: I/O buffer - * - * Perform IO to the swap-file. This is much like direct IO. - */ -static int cifs_swap_rw(struct kiocb *iocb, struct iov_iter *iter) -{ - ssize_t ret; - - if (iov_iter_rw(iter) == READ) - ret = netfs_unbuffered_read_iter_locked(iocb, iter); - else - ret = netfs_unbuffered_write_iter_locked(iocb, iter, NULL); - if (ret < 0) - return ret; - return 0; -} +#else +#define cifs_swap_activate NULL +#define cifs_swap_deactivate NULL +#endif /* CONFIG_SWAP */ const struct address_space_operations cifs_addr_ops = { .read_folio = netfs_read_folio, @@ -3503,7 +3519,6 @@ const struct address_space_operations cifs_addr_ops = { */ .swap_activate = cifs_swap_activate, .swap_deactivate = cifs_swap_deactivate, - .swap_rw = cifs_swap_rw, }; /* diff --git a/include/linux/fs.h b/include/linux/fs.h index 50ce731a2b78..87b5e9957c00 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -438,7 +438,6 @@ struct address_space_operations { int (*swap_activate)(struct swap_info_struct *sis, struct file *file, sector_t *span); void (*swap_deactivate)(struct file *file); - int (*swap_rw)(struct kiocb *iocb, struct iov_iter *iter); }; extern const struct address_space_operations empty_aops; diff --git a/include/linux/nfs_fs.h b/include/linux/nfs_fs.h index ec17e602c979..764056498eba 100644 --- a/include/linux/nfs_fs.h +++ b/include/linux/nfs_fs.h @@ -548,7 +548,6 @@ static inline const struct cred *nfs_file_cred(struct file *file) /* * linux/fs/nfs/direct.c */ -int nfs_swap_rw(struct kiocb *iocb, struct iov_iter *iter); ssize_t nfs_file_direct_read(struct kiocb *iocb, struct iov_iter *iter, bool swap); ssize_t nfs_file_direct_write(struct kiocb *iocb, diff --git a/include/linux/swap.h b/include/linux/swap.h index 8dd68733c955..5658a1634b85 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -341,8 +341,6 @@ extern void __meminit kswapd_run(int nid); extern void __meminit kswapd_stop(int nid); #ifdef CONFIG_SWAP - -int swap_fs_activate(struct swap_info_struct *sis); int add_swap_extent(struct swap_info_struct *sis, unsigned long start_page, unsigned long nr_pages, sector_t start_block); int generic_swapfile_activate(struct swap_info_struct *, struct file *, @@ -468,10 +466,6 @@ static inline bool folio_free_swap(struct folio *folio) return false; } -static inline int swap_fs_activate(struct swap_info_struct *sis) -{ - return -EINVAL; -} static inline int add_swap_extent(struct swap_info_struct *sis, unsigned long start_page, unsigned long nr_pages, sector_t start_block) diff --git a/include/linux/swap_ops.h b/include/linux/swap_ops.h index e92b4f532604..57ac6c703f68 100644 --- a/include/linux/swap_ops.h +++ b/include/linux/swap_ops.h @@ -36,4 +36,9 @@ struct swap_ops { void (*submit_read)(struct swap_io_ctx *ctx); }; +void swap_fs_prepare_rw(struct swap_io_ctx *ctx, int rw, struct iov_iter *iter); +bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio, + size_t prev_folio_size, int rw); +int swap_fs_activate(struct swap_info_struct *sis, const struct swap_ops *ops); + #endif /* _MM_SWAP_OPS_H */ diff --git a/mm/page_io.c b/mm/page_io.c index e741e67d6592..88962571cb93 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -650,11 +650,9 @@ const struct swap_ops swap_bdev_ops = { .can_merge = swap_bdev_can_merge, }; -static void swap_fs_submit(struct swap_io_ctx *ctx, int rw) +void swap_fs_prepare_rw(struct swap_io_ctx *ctx, int rw, struct iov_iter *iter) { struct swap_iocb *sio = ctx->sio; - struct iov_iter iter; - int ret; init_sync_kiocb(&sio->iocb, ctx->sis->swap_file); sio->iocb.ki_pos = swap_dev_pos(bvec_folio(&sio->bvecs[0])->swap); @@ -663,40 +661,22 @@ static void swap_fs_submit(struct swap_io_ctx *ctx, int rw) else sio->iocb.ki_complete = swap_fs_read_complete; - iov_iter_bvec(&iter, rw == WRITE ? ITER_SOURCE : ITER_DEST, + iov_iter_bvec(iter, rw == WRITE ? ITER_SOURCE : ITER_DEST, sio->bvecs, sio->nr_bvecs, sio->len); - ret = sio->iocb.ki_filp->f_mapping->a_ops->swap_rw(&sio->iocb, &iter); - if (ret != -EIOCBQUEUED) - sio->iocb.ki_complete(&sio->iocb, ret); } +EXPORT_SYMBOL_GPL(swap_fs_prepare_rw); -static void swap_fs_submit_write(struct swap_io_ctx *ctx) -{ - swap_fs_submit(ctx, WRITE); -} - -static void swap_fs_submit_read(struct swap_io_ctx *ctx) -{ - swap_fs_submit(ctx, READ); -} - -static bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio, +bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio, size_t prev_folio_size, int rw) { return swap_dev_pos(folio->swap) == swap_dev_pos(prev_folio->swap) + prev_folio_size; } +EXPORT_SYMBOL_GPL(swap_fs_can_merge); -static const struct swap_ops swap_fs_ops = { - .flags = SWAP_OPS_F_REQUIRE_NOFS, - .submit_write = swap_fs_submit_write, - .submit_read = swap_fs_submit_read, - .can_merge = swap_fs_can_merge, -}; - -int swap_fs_activate(struct swap_info_struct *sis) +int swap_fs_activate(struct swap_info_struct *sis, const struct swap_ops *ops) { - sis->ops = &swap_fs_ops; + sis->ops = ops; return add_swap_extent(sis, 0, sis->max, 0); } EXPORT_SYMBOL_GPL(swap_fs_activate); -- cgit v1.2.3 From f525001b3309a0decdcdcdaeceafd7ab9dc927fe Mon Sep 17 00:00:00 2001 From: Tejun Heo Date: Wed, 12 Aug 2026 09:47:43 -1000 Subject: percpu: drop CONFIG_DEBUG_FORCE_WEAK_PER_CPU alpha requires percpu variables in modules to be defined as weak so that the compiler generates GOT based external references for them. This puts two extra restrictions on percpu variable definitions. The symbol must be globally unique even when static and a static percpu variable can't be defined inside a function. DEBUG_FORCE_WEAK_PER_CPU exists to give generic code build coverage for these restrictions without building for alpha. MEM_ALLOC_PROFILING defines a static percpu counter at each allocation call site and thus can't be built with weak percpu definitions, so it depends on !DEBUG_FORCE_WEAK_PER_CPU. As allmodconfig enables DEBUG_FORCE_WEAK_PER_CPU, this knocks MEM_ALLOC_PROFILING out of allmodconfig build coverage. allmodconfig coverage for MEM_ALLOC_PROFILING is worth more than build coverage for restrictions which only matter to alpha module builds. Drop DEBUG_FORCE_WEAK_PER_CPU. Restriction violations will now show up only on alpha builds. Link: https://lore.kernel.org/178656406317.2437052.7257990869957704195@slm.duckdns.org Signed-off-by: Tejun Heo Reported-by: Andrew Morton Reviewed-by: Suren Baghdasaryan Acked-by: Gabriele Monaco [include/rv/da_monitor.h] Cc: Dennis Zhou Cc: Kent Overstreet Cc: Steven Rostedt Signed-off-by: Andrew Morton --- include/linux/percpu-defs.h | 7 +------ include/rv/da_monitor.h | 2 +- lib/Kconfig.debug | 15 --------------- mm/Kconfig.debug | 1 - 4 files changed, 2 insertions(+), 23 deletions(-) (limited to 'include') diff --git a/include/linux/percpu-defs.h b/include/linux/percpu-defs.h index 2cba7cc2b01f..dbe3267a0a13 100644 --- a/include/linux/percpu-defs.h +++ b/include/linux/percpu-defs.h @@ -65,13 +65,8 @@ * * Archs which need weak percpu definitions should set * CONFIG_ARCH_MODULE_NEEDS_WEAK_PER_CPU when necessary. - * - * To ensure that the generic code observes the above two - * restrictions, if CONFIG_DEBUG_FORCE_WEAK_PER_CPU is set weak - * definition is used for all cases. */ -#if (defined(CONFIG_ARCH_MODULE_NEEDS_WEAK_PER_CPU) && defined(MODULE)) || \ - defined(CONFIG_DEBUG_FORCE_WEAK_PER_CPU) +#if defined(CONFIG_ARCH_MODULE_NEEDS_WEAK_PER_CPU) && defined(MODULE) /* * __pcpu_scope_* dummy variable is used to enforce scope. It * receives the static modifier when it's used in front of diff --git a/include/rv/da_monitor.h b/include/rv/da_monitor.h index 34b8fba9ecd4..6b641697106b 100644 --- a/include/rv/da_monitor.h +++ b/include/rv/da_monitor.h @@ -24,7 +24,7 @@ /* * Per-cpu variables require a unique name although static in some - * configurations (e.g. CONFIG_DEBUG_FORCE_WEAK_PER_CPU or alpha modules). + * configurations (e.g. alpha modules). */ #define DA_MON_NAME CONCATENATE(da_mon_, MONITOR_NAME) diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index b82515cde538..00921b1676e8 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -613,21 +613,6 @@ config BUILTIN_MODULE_RANGES It also records an anchor symbol to determine the load address of the section. -config DEBUG_FORCE_WEAK_PER_CPU - bool "Force weak per-cpu definitions" - depends on DEBUG_KERNEL - help - s390 and alpha require percpu variables in modules to be - defined weak to work around addressing range issue which - puts the following two restrictions on percpu variable - definitions. - - 1. percpu symbols must be unique whether static or not - 2. percpu variables can't be defined inside a function - - To ensure that generic code follows the above rules, this - option forces all percpu variables to be defined as weak. - config WARN_CONTEXT_ANALYSIS bool "Compiler context-analysis warnings" depends on CC_IS_CLANG && CLANG_VERSION >= 230000 diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index 5737a504efbb..15dca19dd07d 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -326,7 +326,6 @@ config MEM_ALLOC_PROFILING default n depends on MMU depends on PROC_FS - depends on !DEBUG_FORCE_WEAK_PER_CPU select CODE_TAGGING select PAGE_EXTENSION select SLAB_OBJ_EXT -- cgit v1.2.3 From c1afbd5de131f5e3c4fc7559acf055f8d9d86868 Mon Sep 17 00:00:00 2001 From: Usama Arif Date: Mon, 17 Aug 2026 03:38:35 -0700 Subject: mm/memcontrol: avoid false sharing between vmstats and events Moving v1 userspace eventfd handling into memcontrol-v1.c shrank struct vmpressure from 112 to 24 bytes when CONFIG_MEMCG_V1 is disabled. This moved memory_events_local[MEMCG_SWAP_FAIL] and the hot vmstats_percpu pointer onto the same cacheline. The stress-ng mremap stressor exercises MADV_PAGEOUT with swap disabled, generating about 20 million MEMCG_SWAP_FAIL updates per 60-second run on a 176-CPU test system. Those writes bounce the line while memcg statistics paths load vmstats_percpu. Move cgwb_list into the existing alignment gap and cacheline-align vmstats_percpu. This separates the pointer from the event counters without increasing the size of struct mem_cgroup in the tested configuration. The blamed commit reduced median mremap throughput by 4.38% on the test system with one socket. The patched kernel brings the performance to within 0.5% of the parent which is within the observed boot-to-boot spread (up to 1.2%). Link: https://lore.kernel.org/20260817103835.2937733-1-usama.arif@linux.dev Fixes: ea928e9e18da ("mm/vmpressure: move v1 userspace eventfd code into memcontrol-v1.c") Signed-off-by: Usama Arif Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202608131743.c6a7dda4-lkp@intel.com Tested-by: kernel test robot Link: http://lore.kernel.org/aoAABX59IzUXz/Rv@ly-workstation Acked-by: Shakeel Butt Acked-by: Michal Hocko Cc: David Hildenbrand Cc: Johannes Weiner Cc: Muchun Song Cc: Roman Gushchin Cc: Yi Lai Signed-off-by: Andrew Morton --- include/linux/memcontrol.h | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) (limited to 'include') diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index e78bc98ab229..215e2e87f42b 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -268,10 +268,15 @@ struct mem_cgroup { #endif int kmemcg_id; - struct memcg_vmstats_percpu __percpu *vmstats_percpu; - #ifdef CONFIG_CGROUP_WRITEBACK struct list_head cgwb_list; +#endif + + /* Keep the hot per-CPU stats pointer away from memory event counters. */ + struct memcg_vmstats_percpu __percpu *vmstats_percpu + ____cacheline_aligned_in_smp; + +#ifdef CONFIG_CGROUP_WRITEBACK struct wb_domain cgwb_domain; struct memcg_cgwb_frn cgwb_frn[MEMCG_CGWB_FRN_CNT]; #endif -- cgit v1.2.3 From dd1638dfbb1c48f13cfb4f4f4e55e6570e25384b Mon Sep 17 00:00:00 2001 From: "Kiryl Shutsemau (Meta)" Date: Tue, 18 Aug 2026 12:50:26 +0100 Subject: mm: include swap.h in swapops.h swapops.h uses MAX_SWAPFILES_SHIFT, SWP_MIGRATION_READ and SWP_PTE_MARKER, all of which swap.h defines, but does not include swap.h. It compiles only where the translation unit pulled swap.h in first. leafops.h includes swapops.h on the line above swap.h, so a file whose include list reaches leafops.h before swap.h gets: In file included from include/linux/leafops.h:11: include/linux/swapops.h:88:21: error: use of undeclared identifier 'MAX_SWAPFILES_SHIFT' A header that uses a definition has to include the header that provides it. Link: https://lore.kernel.org/20260818115026.656406-1-kirill@shutemov.name Signed-off-by: Kiryl Shutsemau (Meta) Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-kbuild-all/202608181757.mza9RRj7-lkp@intel.com/ Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Barry Song Cc: Baoquan He Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Signed-off-by: Andrew Morton --- include/linux/swapops.h | 1 + 1 file changed, 1 insertion(+) (limited to 'include') diff --git a/include/linux/swapops.h b/include/linux/swapops.h index 1f3ff3b93e16..e7d0d529f3e0 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -5,6 +5,7 @@ #include #include #include +#include #ifdef CONFIG_MMU -- cgit v1.2.3 From 9e32ec53b1ec2ab28b29c82a95a65bf3d3a5d32c Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:09 -0400 Subject: maple_tree: add rcu locking check when LOCKDEP is enabled Patch series "maple_tree: lock checking and clean ups", v3. In this series: 1. Try to detect lock issues A number of syzbot reports are incorrectly pointing to the mm exit as a source of the locking error. The first three patches attempt to help users detect errors in their locking - but they still have to use LOCKDEP. I guess it's still down to hope and prayers. 2. Documentation fixes The documentation was lacking clarity, there are updates to try and help the users, especially around the erase() cases. 3. Two benign issues The cyclic allocator may have a race, although no in-kernel user can hit it. The erase functions may cause allocation issues if used with the incorrect locking type, but none are present in-tree. 4. The erase gfp uses mas_erase() and mtree_erase() do not take a gfp argument. To improve reliability of the erase, the first attempt to allocate will be GFP_NOWAIT, followed by a retry (if necessary of GFP_KERNEL | GFP_NOFAIL. This will ensure the data is gone. I've updated the documentation to make it more clear as well. mas_store() is not addressed in the same way, but may need to be updated at a later date, but that may require changing callers so it is out of scope here. Beyond these goals there are some test fixes, some general speed-up patches targeting extra work and cycles, and dropping dead code. This patch (of 19): When CONFIG_LOCKDEP and CONFIG_RCU_STRICT_GRACE_PERIOD is enabled, check for rcu locking issues by recording the grace period in the maple state and checking the rcu window is still valid whenever the maple state is reused with a state that is not MA_START or MA_PAUSED. Link: https://lore.kernel.org/20260821192627.4085470-1-liam@infradead.org Link: https://lore.kernel.org/20260821192627.4085470-2-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- include/linux/maple_tree.h | 3 +++ lib/maple_tree.c | 50 +++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 52 insertions(+), 1 deletion(-) (limited to 'include') diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h index 1b3014377105..1acf932fcd33 100644 --- a/include/linux/maple_tree.h +++ b/include/linux/maple_tree.h @@ -484,6 +484,9 @@ struct ma_state { unsigned char mas_flags; unsigned char end; /* The end of the node */ enum store_type store_type; /* The type of store needed for this operation */ +#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) + unsigned long rcu_gp; +#endif }; struct ma_wr_state { diff --git a/lib/maple_tree.c b/lib/maple_tree.c index a0542b491bc2..6d805521bedd 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1148,6 +1148,42 @@ static inline void mas_free(struct ma_state *mas, struct maple_enode *used) ma_free_rcu(mte_to_node(used)); } +void mas_lock_check(struct ma_state *mas) +{ + +#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) + if (!mas_is_active(mas)) + return; + + if (!mt_locked(mas->tree)) { + if (mt_in_rcu(mas->tree)) + WARN_ON_ONCE(poll_state_synchronize_rcu(mas->rcu_gp)); + } +#endif + +} + +void mas_init_lock_check(struct ma_state *mas) +{ +#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) + if (!mt_locked(mas->tree)) { + if (mt_in_rcu(mas->tree)) + mas->rcu_gp = get_state_synchronize_rcu(); + } +#endif + +} + +static void mas_may_init_lock_check(struct ma_state *mas) +{ +#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) + if (mas_is_start(mas) || mas_is_paused(mas)) + mas_init_lock_check(mas); + else + mas_lock_check(mas); +#endif +} + /* * mas_start() - Sets up maple state for operations. * @mas: The maple state. @@ -1166,6 +1202,7 @@ static inline struct maple_enode *mas_start(struct ma_state *mas) if (likely(mas_is_start(mas))) { struct maple_enode *root; + mas_init_lock_check(mas); mas->min = 0; mas->max = ULONG_MAX; @@ -4355,6 +4392,7 @@ void *mas_walk(struct ma_state *mas) { void *entry; + mas_may_init_lock_check(mas); if (!mas_is_active(mas) && !mas_is_start(mas)) mas->status = ma_start; retry: @@ -4992,6 +5030,7 @@ static void mas_may_activate(struct ma_state *mas) mas->status = ma_start; } else { mas->status = ma_active; + mas_lock_check(mas); } } @@ -5069,6 +5108,7 @@ void *mas_next(struct ma_state *mas, unsigned long max) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_next_setup(mas, max, &entry)) return entry; @@ -5092,6 +5132,7 @@ void *mas_next_range(struct ma_state *mas, unsigned long max) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_next_setup(mas, max, &entry)) return entry; @@ -5200,6 +5241,7 @@ void *mas_prev(struct ma_state *mas, unsigned long min) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_prev_setup(mas, min, &entry)) return entry; @@ -5223,6 +5265,7 @@ void *mas_prev_range(struct ma_state *mas, unsigned long min) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_prev_setup(mas, min, &entry)) return entry; @@ -5269,6 +5312,7 @@ EXPORT_SYMBOL_GPL(mt_prev); */ void mas_pause(struct ma_state *mas) { + mas_lock_check(mas); mas->status = ma_pause; mas->node = NULL; } @@ -5377,6 +5421,7 @@ void *mas_find(struct ma_state *mas, unsigned long max) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_find_setup(mas, max, &entry)) return entry; @@ -5404,6 +5449,7 @@ void *mas_find_range(struct ma_state *mas, unsigned long max) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_find_setup(mas, max, &entry)) return entry; @@ -5516,6 +5562,7 @@ void *mas_find_rev(struct ma_state *mas, unsigned long min) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_find_rev_setup(mas, min, &entry)) return entry; @@ -5542,6 +5589,7 @@ void *mas_find_range_rev(struct ma_state *mas, unsigned long min) { void *entry = NULL; + mas_may_init_lock_check(mas); if (mas_find_rev_setup(mas, min, &entry)) return entry; @@ -5618,7 +5666,7 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp) if (!mas->sheaf && !mas->alloc) return false; - mas->status = ma_start; + mas_reset(mas); return true; } -- cgit v1.2.3 From 8f2109843137da8068f19a120d79e58ef3838429 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:10 -0400 Subject: locking/lockdep: add sequence counter to held_lock Add an 8 bit small sequence counter to the held_lock struct to detect if the lock as been dropped and reacquired. This is useful when a data structure depends on a constant locking context, but is not able to detect locking and unlocking of the lock through its own API. Since the __lock_unpin_lock() will no longer detect underflow by casting the unsigned int to a signed int, update the casting code to use a temp variable for calculations using a signed int. Link: https://lore.kernel.org/20260821192627.4085470-3-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Suggested-by: Peter Zijlstra Cc: Ingo Molnar Cc: Will Deacon Cc: Boqun Feng Cc: Waiman Long Link: https://lore.kernel.org/all/h3tpnj5kzcrxms5picmimtkpg4aypcpip5wbd6bt2rpdj5k7eb@nhtzs3lefrkq/ Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Jason Gunthorpe Cc: Joe Perches Cc: Rik van Riel Signed-off-by: Andrew Morton --- include/linux/lockdep.h | 3 +++ include/linux/lockdep_types.h | 3 ++- include/linux/sched.h | 1 + kernel/locking/lockdep.c | 58 +++++++++++++++++++++++++++++++++++++------ 4 files changed, 57 insertions(+), 8 deletions(-) (limited to 'include') diff --git a/include/linux/lockdep.h b/include/linux/lockdep.h index 621566345406..a6451ecbbe9a 100644 --- a/include/linux/lockdep.h +++ b/include/linux/lockdep.h @@ -273,6 +273,9 @@ extern struct pin_cookie lock_pin_lock(struct lockdep_map *lock); extern void lock_repin_lock(struct lockdep_map *lock, struct pin_cookie); extern void lock_unpin_lock(struct lockdep_map *lock, struct pin_cookie); +extern u32 lock_sequence(struct lockdep_map *lock); +#define lockdep_sequence(lock) lock_sequence(&(lock)->dep_map) + #define lockdep_depth(tsk) (debug_locks ? (tsk)->lockdep_depth : 0) #define lockdep_assert(cond) \ diff --git a/include/linux/lockdep_types.h b/include/linux/lockdep_types.h index eae115a26488..55c4b152fedf 100644 --- a/include/linux/lockdep_types.h +++ b/include/linux/lockdep_types.h @@ -253,7 +253,8 @@ struct held_lock { unsigned int hardirqs_off:1; unsigned int sync:1; unsigned int references:11; /* 32 bits */ - unsigned int pin_count; + unsigned int pin_count:24; + unsigned int seq_count:8; }; #else /* !CONFIG_LOCKDEP */ diff --git a/include/linux/sched.h b/include/linux/sched.h index 373bcc0598d1..14d5ce8dd613 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1288,6 +1288,7 @@ struct task_struct { u64 curr_chain_key; int lockdep_depth; unsigned int lockdep_recursion; + unsigned int lockdep_seq; struct held_lock held_locks[MAX_LOCK_DEPTH]; #endif diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c index 2d4c5bab5af8..a69567bdd791 100644 --- a/kernel/locking/lockdep.c +++ b/kernel/locking/lockdep.c @@ -5077,7 +5077,7 @@ static int __lock_is_held(const struct lockdep_map *lock, int read); static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass, int trylock, int read, int check, int hardirqs_off, struct lockdep_map *nest_lock, unsigned long ip, - int references, int pin_count, int sync) + int references, int pin_count, int sync, int seq) { struct task_struct *curr = current; struct lock_class *class = NULL; @@ -5183,6 +5183,7 @@ static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass, hlock->holdtime_stamp = lockstat_clock(); #endif hlock->pin_count = pin_count; + hlock->seq_count = seq; if (check_wait_context(curr, hlock)) return 0; @@ -5388,7 +5389,7 @@ static int reacquire_held_locks(struct task_struct *curr, unsigned int depth, hlock->read, hlock->check, hlock->hardirqs_off, hlock->nest_lock, hlock->acquire_ip, - hlock->references, hlock->pin_count, 0)) { + hlock->references, hlock->pin_count, 0, hlock->seq_count)) { case 0: return 1; case 1: @@ -5669,14 +5670,17 @@ static void __lock_unpin_lock(struct lockdep_map *lock, struct pin_cookie cookie struct held_lock *hlock = curr->held_locks + i; if (match_held_lock(hlock, lock)) { + int pin_count; + if (WARN(!hlock->pin_count, "unpinning an unpinned lock\n")) return; - hlock->pin_count -= cookie.val; + pin_count = hlock->pin_count - cookie.val; - if (WARN((int)hlock->pin_count < 0, "pin count corrupted\n")) - hlock->pin_count = 0; + if (WARN(pin_count < 0, "pin count corrupted\n")) + pin_count = 0; + hlock->pin_count = pin_count; return; } } @@ -5684,6 +5688,24 @@ static void __lock_unpin_lock(struct lockdep_map *lock, struct pin_cookie cookie WARN(1, "unpinning an unheld lock\n"); } +static u32 __lock_sequence(struct lockdep_map *lock) +{ + struct task_struct *curr = current; + int i; + + if (unlikely(!debug_locks)) + return ~0; + + for (i = 0; i < curr->lockdep_depth; i++) { + struct held_lock *hlock = curr->held_locks + i; + + if (match_held_lock(hlock, lock)) + return hlock->seq_count; + } + + return ~0; +} + /* * Check whether we follow the irq-flags state precisely: */ @@ -5866,7 +5888,8 @@ void lock_acquire(struct lockdep_map *lock, unsigned int subclass, lockdep_recursion_inc(); __lock_acquire(lock, subclass, trylock, read, check, - irqs_disabled_flags(flags), nest_lock, ip, 0, 0, 0); + irqs_disabled_flags(flags), nest_lock, ip, 0, 0, 0, + ++current->lockdep_seq); lockdep_recursion_finish(); raw_local_irq_restore(flags); } @@ -5914,7 +5937,8 @@ void lock_sync(struct lockdep_map *lock, unsigned subclass, int read, lockdep_recursion_inc(); __lock_acquire(lock, subclass, 0, read, check, - irqs_disabled_flags(flags), nest_lock, ip, 0, 0, 1); + irqs_disabled_flags(flags), nest_lock, ip, 0, 0, 1, + ++current->lockdep_seq); check_chain_key(current); lockdep_recursion_finish(); raw_local_irq_restore(flags); @@ -6000,6 +6024,26 @@ void lock_unpin_lock(struct lockdep_map *lock, struct pin_cookie cookie) } EXPORT_SYMBOL_GPL(lock_unpin_lock); +u32 lock_sequence(struct lockdep_map *lock) +{ + unsigned long flags; + u32 seq = ~0; + + if (unlikely(!lockdep_enabled())) + return seq; + + raw_local_irq_save(flags); + check_flags(flags); + + lockdep_recursion_inc(); + seq = __lock_sequence(lock); + lockdep_recursion_finish(); + raw_local_irq_restore(flags); + + return seq; +} +EXPORT_SYMBOL_GPL(lock_sequence); + #ifdef CONFIG_LOCK_STAT static void print_lock_contention_bug(struct task_struct *curr, struct lockdep_map *lock, -- cgit v1.2.3 From 19e269917dc416f932474686bf1fcf3e91a740bd Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:11 -0400 Subject: maple_tree: add write lock checking with lockdep sequence numbers Use the lockdep sequence numbers to ensure the write lock is not dropped between write operations. The lockdep sequence is recorded on any walk that starts from the top of the tree and re-checked prior to any operation using an active node. When lockdep detects an issue, it sets debug_locks to 0 disabling further reports. __lock_sequnece() will return u32 ~0 when debug_locks is zero, and the real sequnece count cannot return such a high value as it is less than 32bits. By always updating the sequence number, regardless of lock state and by ignoring ~0 value in the sequence number will avoid ever printing a WARN_ON when lockdep sets debug_locks to 0. Link: https://lore.kernel.org/20260821192627.4085470-4-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Breno Leitao Tested-by: Breno Leitao Cc: Boqun Feng Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- include/linux/maple_tree.h | 7 +++-- lib/maple_tree.c | 64 +++++++++++++++++++++++++++++++++++++--------- 2 files changed, 57 insertions(+), 14 deletions(-) (limited to 'include') diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h index 1acf932fcd33..d63ac92208d0 100644 --- a/include/linux/maple_tree.h +++ b/include/linux/maple_tree.h @@ -484,9 +484,12 @@ struct ma_state { unsigned char mas_flags; unsigned char end; /* The end of the node */ enum store_type store_type; /* The type of store needed for this operation */ -#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) +#ifdef CONFIG_LOCKDEP + u32 ld_seq; +#ifdef CONFIG_RCU_STRICT_GRACE_PERIOD unsigned long rcu_gp; -#endif +#endif /* CONFIG_RCU_STRICT_GRACE_PERIOD */ +#endif /* CONFIG_LOCKDEP */ }; struct ma_wr_state { diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 6d805521bedd..d8c826e1ca0c 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1148,40 +1148,77 @@ static inline void mas_free(struct ma_state *mas, struct maple_enode *used) ma_free_rcu(mte_to_node(used)); } -void mas_lock_check(struct ma_state *mas) + +#ifdef CONFIG_LOCKDEP +static struct lockdep_map *mas_lockdep_map(struct ma_state *mas) { + struct maple_tree *mt = mas->tree; + + if (mt_external_lock(mt)) + return mt->ma_external_lock; + + return &(mt->ma_lock).dep_map; +} + +#endif + +static void mas_lock_check(struct ma_state *mas) +{ +#ifdef CONFIG_LOCKDEP + struct lockdep_map *map; + u32 seq; -#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) if (!mas_is_active(mas)) return; +#ifdef CONFIG_RCU_STRICT_GRACE_PERIOD if (!mt_locked(mas->tree)) { if (mt_in_rcu(mas->tree)) WARN_ON_ONCE(poll_state_synchronize_rcu(mas->rcu_gp)); } -#endif +#endif /* CONFIG_RCU_STRICT_GRACE_PERIOD */ + + map = mas_lockdep_map(mas); + if (!map) + return; + + seq = lock_sequence(map); + if (seq != UINT_MAX && mas->ld_seq != UINT_MAX) + WARN_ON_ONCE(mas->ld_seq != seq); +#endif /* CONFIG_LOCKDEP */ } -void mas_init_lock_check(struct ma_state *mas) +static void mas_init_lock_check(struct ma_state *mas) { -#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) +#ifdef CONFIG_LOCKDEP + struct lockdep_map *map; +#ifdef CONFIG_RCU_STRICT_GRACE_PERIOD if (!mt_locked(mas->tree)) { if (mt_in_rcu(mas->tree)) mas->rcu_gp = get_state_synchronize_rcu(); + return; } -#endif +#endif /* CONFIG_RCU_STRICT_GRACE_PERIOD */ + + map = mas_lockdep_map(mas); + if (map) /* Update regardless of lock state */ + mas->ld_seq = lock_sequence(map); +#endif /* CONFIG_LOCKDEP */ } static void mas_may_init_lock_check(struct ma_state *mas) { -#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) - if (mas_is_start(mas) || mas_is_paused(mas)) +#ifdef CONFIG_LOCKDEP +#ifdef CONFIG_RCU_STRICT_GRACE_PERIOD + if (mas_is_start(mas) || mas_is_paused(mas)) { mas_init_lock_check(mas); - else - mas_lock_check(mas); -#endif + return; + } +#endif /* CONFIG_RCU_STRICT_GRACE_PERIOD */ + mas_lock_check(mas); +#endif /* CONFIG_LOCKDEP */ } /* @@ -4864,6 +4901,7 @@ void *mas_store(struct ma_state *mas, void *entry) { MA_WR_STATE(wr_mas, mas, entry); + mas_may_init_lock_check(mas); trace_ma_write(TP_FCT, mas, 0, entry); #ifdef CONFIG_DEBUG_MAPLE_TREE if (MAS_WARN_ON(mas, mas->index > mas->last)) @@ -4922,6 +4960,7 @@ int mas_store_gfp(struct ma_state *mas, void *entry, gfp_t gfp) MA_WR_STATE(wr_mas, mas, entry); int ret = 0; + mas_may_init_lock_check(mas); retry: mas_wr_preallocate(&wr_mas, entry); if (unlikely(mas_nomem(mas, gfp))) { @@ -4952,6 +4991,7 @@ void mas_store_prealloc(struct ma_state *mas, void *entry) { MA_WR_STATE(wr_mas, mas, entry); + mas_lock_check(mas); if (mas->store_type == wr_store_root) { mas_wr_prealloc_setup(&wr_mas); goto store; @@ -4984,6 +5024,7 @@ int mas_preallocate(struct ma_state *mas, void *entry, gfp_t gfp) { MA_WR_STATE(wr_mas, mas, entry); + mas_may_init_lock_check(mas); mas_wr_prealloc_setup(&wr_mas); mas->store_type = mas_wr_store_type(&wr_mas); mas_prealloc_calc(&wr_mas, entry); @@ -5469,7 +5510,6 @@ EXPORT_SYMBOL_GPL(mas_find_range); static bool mas_find_rev_setup(struct ma_state *mas, unsigned long min, void **entry) { - switch (mas->status) { case ma_active: goto active; -- cgit v1.2.3 From 3526e09d8cab0aea3f3737dbeb613ccb52660359 Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:14 -0400 Subject: maple_tree: drop MAPLE_ALLOC_SLOTS MAPLE_ALLOC_SLOTS is no longer used, so remove it. Link: https://lore.kernel.org/20260821192627.4085470-7-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- include/linux/maple_tree.h | 2 -- 1 file changed, 2 deletions(-) (limited to 'include') diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h index d63ac92208d0..14ca9ac775d9 100644 --- a/include/linux/maple_tree.h +++ b/include/linux/maple_tree.h @@ -28,13 +28,11 @@ #define MAPLE_NODE_SLOTS 31 /* 256 bytes including ->parent */ #define MAPLE_RANGE64_SLOTS 16 /* 256 bytes */ #define MAPLE_ARANGE64_SLOTS 10 /* 240 bytes */ -#define MAPLE_ALLOC_SLOTS (MAPLE_NODE_SLOTS - 1) #else /* 32bit sizes */ #define MAPLE_NODE_SLOTS 63 /* 256 bytes including ->parent */ #define MAPLE_RANGE64_SLOTS 32 /* 256 bytes */ #define MAPLE_ARANGE64_SLOTS 21 /* 240 bytes */ -#define MAPLE_ALLOC_SLOTS (MAPLE_NODE_SLOTS - 2) #endif /* defined(CONFIG_64BIT) || defined(BUILD_VDSO32_64) */ #define MAPLE_NODE_MASK 255UL -- cgit v1.2.3 From f1681380b5f928e147954d87d875f57a25df189c Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:22 -0400 Subject: maple_tree: avoid mas_erase() and mtree_erase() failures Failures to remove entries using the two APIs to erase the entries may result in allocation failures. The failures may go unnoticed and an unexpected entry may remain. Instead, fall back to retrying with GFP_KERNEL | __GFP_NOFAIL so that the entry will be removed. Link: https://lore.kernel.org/20260821192627.4085470-15-liam@infradead.org Signed-off-by: Liam R. Howlett (Oracle) Cc: Rik van Riel Cc: Jason Gunthorpe Cc: Boqun Feng Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Joe Perches Cc: Peter Zijlstra Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- include/linux/maple_tree.h | 2 ++ lib/maple_tree.c | 68 ++++++++++++++++++++++++++++++++++------------ 2 files changed, 52 insertions(+), 18 deletions(-) (limited to 'include') diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h index 14ca9ac775d9..173602e87c14 100644 --- a/include/linux/maple_tree.h +++ b/include/linux/maple_tree.h @@ -570,6 +570,8 @@ int mas_alloc_cyclic(struct ma_state *mas, unsigned long *startp, unsigned long *next, gfp_t gfp); bool mas_nomem(struct ma_state *mas, gfp_t gfp); +bool mas_nomem_nofail(struct ma_state *mas, unsigned long index, + unsigned long last); void mas_pause(struct ma_state *mas); void maple_tree_init(void); void mas_destroy(struct ma_state *mas); diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 440863bdea26..d47d4304f781 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -5658,9 +5658,10 @@ EXPORT_SYMBOL_GPL(mas_find_range_rev); * Searches for @mas->index, sets @mas->index and @mas->last to the range and * erases that range. * - * Note that erase requires allocations and will use GFP_KERNEL to do so if - * necessary. If the allocation fails, the internal lock will be dropped to - * retry. + * Note that erase requires allocations and will use GFP_KERNEL | __GFP_NOFAIL + * to do so if necessary. If the allocation fails, the internal lock will be + * dropped to retry. An externally locked tree must be protected by a lock that + * allows blocking for this API. * * Return: the entry that was erased or %NULL, @mas->index and @mas->last are updated. */ @@ -5672,8 +5673,8 @@ void *mas_erase(struct ma_state *mas) /* * In low memory situations, the allocation is retried with the gfp flag - * GFP_KERNEL. The internal spinlock is dropped in mas_nomem(), however - * the external lock is not dropped. + * GFP_KERNEL | __GFP_NOFAIL. The internal spinlock is dropped in + * mas_nomem_nofail(), however the external lock is not dropped. */ if (mt_external_lock(mas->tree)) might_alloc(GFP_KERNEL); @@ -5689,16 +5690,8 @@ write_retry: /* Must reset to ensure spanning writes of last slot are detected */ mas_reset(mas); mas_wr_preallocate(&wr_mas, NULL); - if (mas_nomem(mas, GFP_KERNEL)) { - /* in case the range of entry changed when unlocked */ - mas->index = mas->last = index; + if (mas_nomem_nofail(mas, index, index)) goto write_retry; - } - - if (mas_is_err(mas)) { - entry = NULL; - goto out; - } mas_wr_store_entry(&wr_mas); out: @@ -5721,6 +5714,10 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp) if (likely(mas->node != MA_ERROR(-ENOMEM))) return false; + /* Allocations can fail, don't do this. */ + WARN_ON_ONCE(!gfpflags_allow_blocking(gfp) && + mt_external_lock(mas->tree)); + if (gfpflags_allow_blocking(gfp) && !mt_external_lock(mas->tree)) { mtree_unlock(mas->tree); mas_alloc_nodes(mas, gfp); @@ -5731,7 +5728,9 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp) /* * Return false on zero forward progress. Partial allocations are kept - * so the retry path will attempt to get the rest. + * so the retry path will attempt to get the rest. The failure should + * not happen as we try our best to reclaim. The user would need an + * external lock with a non-blocking gfp in a low memory situation. */ if (!mas->sheaf && !mas->alloc) return false; @@ -5740,6 +5739,39 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp) return true; } +/** + * mas_nomem_nofail() - Retry allocations with __GFP_NOFAIL, if the maple state + * has stored the -ENOMEM error. + * @mas: The maple state + * @index: The start of the range for the @mas reset + * @last: The end of the range for the @mas reset + * + * Return: false if @mas isn't in an -ENOMEM state. True if the allocation + * happens, the state is reset. The internal lock will be dropped and external + * locks must allow blocking. + */ +bool mas_nomem_nofail(struct ma_state *mas, unsigned long index, + unsigned long last) + __must_hold(mas->tree->ma_lock) +{ + gfp_t gfp; + + if (likely(mas->node != MA_ERROR(-ENOMEM))) + return false; + + gfp = GFP_KERNEL | __GFP_NOFAIL; + if (!mt_external_lock(mas->tree)) { + mtree_unlock(mas->tree); + mas_alloc_nodes(mas, gfp); + mtree_lock(mas->tree); + } else { + mas_alloc_nodes(mas, gfp); + } + + mas_set_range(mas, index, last); + return true; +} + void __init maple_tree_init(void) { struct kmem_cache_args args = { @@ -6026,9 +6058,9 @@ EXPORT_SYMBOL(mtree_alloc_rrange); * Erasing is the same as a walk to an entry then a store of a NULL to that * ENTIRE range. In fact, it is implemented as such using the advanced API. * - * Note that erase requires allocations and will use GFP_KERNEL to do so if - * necessary. If the allocation fails, the internal lock will be dropped to - * retry. + * Note that erase requires allocations and will use GFP_KERNEL | __GFP_NOFAIL + * to do so if necessary. If the allocation fails, the internal lock will be + * dropped to retry. * * Return: The entry stored at the @index or %NULL */ -- cgit v1.2.3 From 00f67814a14e614b749ebe54076ef1e3e6454f2b Mon Sep 17 00:00:00 2001 From: "Liam R. Howlett (Oracle)" Date: Fri, 21 Aug 2026 15:26:25 -0400 Subject: maple_tree: fix argument name in header The mas_prev_range() function takes a min and not a max. Link: https://lore.kernel.org/20260821192627.4085470-18-liam@infradead.org Fixes: 6b9e93e01020 ("maple_tree: add mas_prev_range() and mas_find_range_rev interface") Signed-off-by: Liam R. Howlett (Oracle) Cc: Boqun Feng Cc: Breno Leitao Cc: Chris Mason Cc: Chuck Lever Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Joe Perches Cc: Peter Zijlstra Cc: Rik van Riel Cc: Waiman Long Cc: Will Deacon Signed-off-by: Andrew Morton --- include/linux/maple_tree.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'include') diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h index 173602e87c14..e595ae5cd0ee 100644 --- a/include/linux/maple_tree.h +++ b/include/linux/maple_tree.h @@ -577,7 +577,7 @@ void maple_tree_init(void); void mas_destroy(struct ma_state *mas); void *mas_prev(struct ma_state *mas, unsigned long min); -void *mas_prev_range(struct ma_state *mas, unsigned long max); +void *mas_prev_range(struct ma_state *mas, unsigned long min); void *mas_next(struct ma_state *mas, unsigned long max); void *mas_next_range(struct ma_state *mas, unsigned long max); -- cgit v1.2.3