kernel/git/stable/linux.git/mm/memremap.c, branch linux-7.2.y

mm: decrement MTHP_STAT_NR_ANON in free_zone_device_folio()

2026-07-29T00:37:31+00:00

Patch series "mm: fix PMD level mTHP accounting bugs", v2. While running selftests I noticed the PMD level per-mTHP stats (nr_anon) remained elevated after each run. After further investigation I noticed this accounting error occurs for both the migration.private_anon_htlb_test and the HMM tests. In the HMM case this is due to folio_add_new_anon_rmap() incrementing the mTHP stats, but never containing a corresponding decrement in free_zone_device_folio(). We solve this by making sure to decrement the counter when freeing device memory. In the migration case, we are incrementing this counter without first checking whether this folio is a hugetlb folio, which relies on a separate accounting system. We solve this by adding the proper hugetlb check before incrementing this counter. With these changes in place, the two tests no longer cause elevated PMD level accounting issues. This patch (of 2): When a zone device folio is mapped as anonymous, folio_add_new_anon_rmap() increments MTHP_STAT_NR_ANON. The corresponding decrement lives in __free_pages_prepare() in page_alloc.c, but zone device folios are freed via free_zone_device_folio() which never calls __free_pages_prepare(). This causes nr_anon to remain permanently elevated after zone device folios are freed. Add the missing mod_mthp_stat() decrement to free_zone_device_folio() so that the counter is properly balanced. Link: https://lore.kernel.org/20260717064502.1980173-1-npache@redhat.com Link: https://lore.kernel.org/20260717064502.1980173-2-npache@redhat.com Fixes: 5d65c8d758f2 ("mm: count the number of anonymous THPs per size") Co-developed-by: David Hildenbrand Signed-off-by: David Hildenbrand Signed-off-by: Nico Pache Reviewed-by: Zi Yan Cc: Alistair Popple Cc: Barry Song Cc: Byungchul Park Cc: Gregory Price Cc: "Huang, Ying" Cc: Joshua Hahn Cc: Lorenzo Stoakes Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Oscar Salvador Cc: Rakie Kim Cc: Baolin Wang Signed-off-by: Andrew Morton

mm/sparse-vmemmap: pass @pgmap argument to memory deactivation paths

2026-05-29T04:04:54+00:00

Currently, the memory hot-remove call chain -- arch_remove_memory(), __remove_pages(), sparse_remove_section() and section_deactivate() -- does not carry the struct dev_pagemap pointer. This prevents the lower levels from knowing whether the section was originally populated with vmemmap optimizations (e.g., DAX with vmemmap optimization enabled). Without this information, we cannot call vmemmap_can_optimize() to determine if the vmemmap pages were optimized. As a result, the vmemmap page accounting during teardown will mistakenly assume a non-optimized allocation, leading to incorrect memmap statistics. To lay the groundwork for fixing the vmemmap page accounting, we need to pass the @pgmap pointer down to the deactivation location. Plumb the @pgmap argument through the APIs of arch_remove_memory(), __remove_pages() and sparse_remove_section(), mirroring the corresponding *_activate() paths. Link: https://lore.kernel.org/20260428081855.1249045-4-songmuchun@bytedance.com Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Reviewed-by: Oscar Salvador Acked-by: David Hildenbrand (Arm) Acked-by: Liam R. Howlett Cc: "Aneesh Kumar K.V" Cc: Joao Martins Cc: Lorenzo Stoakes Cc: Madhavan Srinivasan Cc: Michael Ellerman Cc: Michal Hocko Cc: Nicholas Piggin Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton

mm/zone_device: do not touch device folio after calling ->folio_free()

2026-04-19T06:24:27+00:00

The contents of a device folio can immediately change after calling ->folio_free(), as the folio may be reallocated by a driver with a different order. Instead of touching the folio again to extract the pgmap, use the local stack variable when calling percpu_ref_put_many(). Link: https://lore.kernel.org/20260410230346.4009855-1-matthew.brost@intel.com Fixes: d245f9b4ab80 ("mm/zone_device: support large zone device private folios") Signed-off-by: Matthew Brost Reviewed-by: Balbir Singh Reviewed-by: Vishal Moola Reviewed-by: Alistair Popple Cc: David Hildenbrand Cc: Oscar Salvador Cc: Signed-off-by: Andrew Morton

mm/zone_device: reinitialize large zone device private folios

2026-01-27T03:03:48+00:00

Reinitialize metadata for large zone device private folios in zone_device_page_init prior to creating a higher-order zone device private folio. This step is necessary when the folio's order changes dynamically between zone_device_page_init calls to avoid building a corrupt folio. As part of the metadata reinitialization, the dev_pagemap must be passed in from the caller because the pgmap stored in the folio page may have been overwritten with a compound head. Without this fix, individual pages could have invalid pgmap fields and flags (with PG_locked being notably problematic) due to prior different order allocations, which can, and will, result in kernel crashes. Link: https://lkml.kernel.org/r/20260116111325.1736137-2-francois.dugast@intel.com Fixes: d245f9b4ab80 ("mm/zone_device: support large zone device private folios") Signed-off-by: Matthew Brost Signed-off-by: Francois Dugast Acked-by: Felix Kuehling Reviewed-by: Balbir Singh Acked-by: Vlastimil Babka Cc: Zi Yan Cc: Alistair Popple Cc: Madhavan Srinivasan Cc: Nicholas Piggin Cc: Michael Ellerman Cc: "Christophe Leroy (CS GROUP)" Cc: Alex Deucher Cc: "Christian König" Cc: David Airlie Cc: Simona Vetter Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: Lyude Paul Cc: Danilo Krummrich Cc: David Hildenbrand Cc: Oscar Salvador Cc: Andrew Morton Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Michal Hocko Signed-off-by: Andrew Morton

mm/memremap: fix spurious large folio warning for FS-DAX

2025-12-23T19:23:16+00:00

This patch addresses a warning that I discovered while working on famfs, which is an fs-dax file system that virtually always does PMD faults (next famfs patch series coming after the holidays). However, XFS also does PMD faults in fs-dax mode, and it also triggers the warning. It takes some effort to get XFS to do a PMD fault, but instructions to reproduce it are below. The VM_WARN_ON_ONCE(folio_test_large(folio)) check in free_zone_device_folio() incorrectly triggers for MEMORY_DEVICE_FS_DAX when PMD (2MB) mappings are used. FS-DAX legitimately creates large file-backed folios when handling PMD faults. This is a core feature of FS-DAX that provides significant performance benefits by mapping 2MB regions directly to persistent memory. When these mappings are unmapped, the large folios are freed through free_zone_device_folio(), which triggers the spurious warning. The warning was introduced by commit that added support for large zone device private folios. However, that commit did not account for FS-DAX file-backed folios, which have always supported large (PMD-sized) mappings. The check distinguishes between anonymous folios (which clear AnonExclusive flags for each sub-page) and file-backed folios. For file-backed folios, it assumes large folios are unexpected - but this assumption is incorrect for FS-DAX. The fix is to exempt MEMORY_DEVICE_FS_DAX from the large folio warning, allowing FS-DAX to continue using PMD mappings without triggering false warnings. Link: https://lkml.kernel.org/r/20251219123717.39330-1-john@groves.net Fixes: d245f9b4ab80 ("mm/zone_device: support large zone device private folios") Signed-off-by: John Groves Acked-by: David Hildenbrand (Red Hat) Reviewed-by: Dan Williams Tested-by: Alison Schofield Cc: Alistair Popple Cc: Balbir Singh Cc: "Darrick J. Wong" Cc: Gregory Price Cc: Oscar Salvador Signed-off-by: Andrew Morton

mm/zone_device: rename page_free callback to folio_free

2025-11-24T23:08:47+00:00

Change page_free to folio_free to make the folio support for zone device-private more consistent. The PCI P2PDMA callback has also been updated and changed to folio_free() as a result. For drivers that do not support folios (yet), the folio is converted back into page via &folio->page and the page is used as is, in the current callback implementation. Link: https://lkml.kernel.org/r/20251001065707.920170-3-balbirs@nvidia.com Signed-off-by: Balbir Singh Cc: David Hildenbrand Cc: Zi Yan Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Oscar Salvador Cc: Lorenzo Stoakes Cc: Baolin Wang Cc: "Liam R. Howlett" Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lyude Paul Cc: Danilo Krummrich Cc: David Airlie Cc: Simona Vetter Cc: Ralph Campbell Cc: Mika Penttilä Cc: Matthew Brost Cc: Francois Dugast Cc: Madhavan Srinivasan Cc: Christophe Leroy Cc: Felix Kuehling Cc: Alex Deucher Cc: "Christian König" Signed-off-by: Andrew Morton

mm/zone_device: support large zone device private folios

2025-11-24T23:08:47+00:00

Patch series "mm: support device-private THP", v7. This patch series introduces support for Transparent Huge Page (THP) migration in zone device-private memory. The implementation enables efficient migration of large folios between system memory and device-private memory Background Current zone device-private memory implementation only supports PAGE_SIZE granularity, leading to: - Increased TLB pressure - Inefficient migration between CPU and device memory This series extends the existing zone device-private infrastructure to support THP, leading to: - Reduced page table overhead - Improved memory bandwidth utilization - Seamless fallback to base pages when needed In my local testing (using lib/test_hmm) and a throughput test, the series shows a 350% improvement in data transfer throughput and a 80% improvement in latency These patches build on the earlier posts by Ralph Campbell [1] Two new flags are added in vma_migration to select and mark compound pages. migrate_vma_setup(), migrate_vma_pages() and migrate_vma_finalize() support migration of these pages when MIGRATE_VMA_SELECT_COMPOUND is passed in as arguments. The series also adds zone device awareness to (m)THP pages along with fault handling of large zone device private pages. page vma walk and the rmap code is also zone device aware. Support has also been added for folios that might need to be split in the middle of migration (when the src and dst do not agree on MIGRATE_PFN_COMPOUND), that occurs when src side of the migration can migrate large pages, but the destination has not been able to allocate large pages. The code supported and used folio_split() when migrating THP pages, this is used when MIGRATE_VMA_SELECT_COMPOUND is not passed as an argument to migrate_vma_setup(). The test infrastructure lib/test_hmm.c has been enhanced to support THP migration. A new ioctl to emulate failure of large page allocations has been added to test the folio split code path. hmm-tests.c has new test cases for huge page migration and to test the folio split path. A new throughput test has been added as well. The nouveau dmem code has been enhanced to use the new THP migration capability. mTHP support: The patches hard code, HPAGE_PMD_NR in a few places, but the code has been kept generic to support various order sizes. With additional refactoring of the code support of different order sizes should be possible. The future plan is to post enhancements to support mTHP with a rough design as follows: 1. Add the notion of allowable thp orders to the HMM based test driver 2. For non PMD based THP paths in migrate_device.c, check to see if a suitable order is found and supported by the driver 3. Iterate across orders to check the highest supported order for migration 4. Migrate and finalize The mTHP patches can be built on top of this series, the key design elements that need to be worked out are infrastructure and driver support for multiple ordered pages and their migration. HMM support for large folios was added in 10b9feee2d0d ("mm/hmm: populate PFNs from PMD swap entry"). This patch (of 16) Add routines to support allocation of large order zone device folios and helper functions for zone device folios, to check if a folio is device private and helpers for setting zone device data. When large folios are used, the existing page_free() callback in pgmap is called when the folio is freed, this is true for both PAGE_SIZE and higher order pages. Zone device private large folios do not support deferred split and scan like normal THP folios. Link: https://lkml.kernel.org/r/20251001065707.920170-1-balbirs@nvidia.com Link: https://lkml.kernel.org/r/20251001065707.920170-2-balbirs@nvidia.com Link: https://lore.kernel.org/linux-mm/20201106005147.20113-1-rcampbell@nvidia.com/ [1] Signed-off-by: Balbir Singh Cc: David Hildenbrand Cc: Zi Yan Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Oscar Salvador Cc: Lorenzo Stoakes Cc: Baolin Wang Cc: "Liam R. Howlett" Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lyude Paul Cc: Danilo Krummrich Cc: David Airlie Cc: Simona Vetter Cc: Ralph Campbell Cc: Mika Penttilä Cc: Matthew Brost Cc: Francois Dugast Cc: Madhavan Srinivasan Cc: Christophe Leroy Cc: Felix Kuehling Cc: Alex Deucher Cc: "Christian König" Signed-off-by: Andrew Morton

mm/memremap: remove unused get_dev_pagemap() parameter

2025-09-21T21:22:21+00:00

GUP no longer uses get_dev_pagemap(). As it was the only user of the get_dev_pagemap() pgmap caching feature it can be removed. Link: https://lkml.kernel.org/r/20250903225926.34702-2-apopple@nvidia.com Signed-off-by: Alistair Popple Acked-by: David Hildenbrand Reviewed-by: Jason Gunthorpe Reviewed-by: Dan Williams Cc: John Hubbard Cc: Oscar Salvador Cc: Peter Xu Signed-off-by: Andrew Morton

mm/memremap: reject unreasonable folio/compound page sizes in memremap_pages()

2025-09-21T21:22:02+00:00

Let's reject unreasonable folio sizes early, where we can still fail. We'll add sanity checks to prepare_compound_head/prepare_compound_page next. Is there a way to configure a system such that unreasonable folio sizes would be possible? It would already be rather questionable. If so, we'd probably want to bail out earlier, where we can avoid a WARN and just report a proper error message that indicates where something went wrong such that we messed up. Link: https://lkml.kernel.org/r/20250901150359.867252-8-david@redhat.com Signed-off-by: David Hildenbrand Acked-by: SeongJae Park Reviewed-by: Lorenzo Stoakes Reviewed-by: Liam R. Howlett Signed-off-by: Andrew Morton

mm/page_alloc: add support for initializing pageblock as isolated

2025-07-13T23:38:17+00:00

MIGRATE_ISOLATE is a standalone bit, so a pageblock cannot be initialized to just MIGRATE_ISOLATE. Add init_pageblock_migratetype() to enable initialize a pageblock with a migratetype and isolated. Link: https://lkml.kernel.org/r/20250617021115.2331563-4-ziy@nvidia.com Signed-off-by: Zi Yan Reviewed-by: Vlastimil Babka Acked-by: David Hildenbrand Cc: Baolin Wang Cc: Brendan Jackman Cc: Johannes Weiner Cc: Kirill A. Shuemov Cc: Mel Gorman Cc: Michal Hocko Cc: Oscar Salvador Cc: Richard Chang Cc: Suren Baghdasaryan Signed-off-by: Andrew Morton