kernel/git/next/linux-next.git/mm/page_alloc.c, branch master

mm: fix ASSERT_EXCLUSIVE_BITS by passing memdesc_flags_t by pointer

2026-07-05T23:23:34+00:00

KCSAN reports a data race between page_to_nid()/folio_pgdat() reading page->flags and folio_trylock()/folio_lock() concurrently doing test_and_set_bit_lock(PG_locked, ...) on the same word, e.g.: BUG: KCSAN: data-race in __lruvec_stat_mod_folio / shmem_get_folio_gfp The race is benign: nid/zone bits are set once at page init and never overlap with PG_locked. However, ASSERT_EXCLUSIVE_BITS() inside memdesc_nid/zonenum() was checking a by-value copy of the flags word, not the live page->flags, so it failed to annotate the real access. Change memdesc_nid(), memdesc_zonenum(), memdesc_section(), and memdesc_is_zone_device() to take a const memdesc_flags_t * and update all callers to pass &page->flags / &folio->flags, so ASSERT_EXCLUSIVE_BITS() operates on the actual shared word. Guard the ASSERT_EXCLUSIVE_BITS() calls in memdesc_zonenum() and memdesc_section() under ZONES_WIDTH != 0 / SECTIONS_WIDTH != 0 to avoid a zero-mask check on configs where the corresponding field is absent. Under CONFIG_NUMA=n, stub out page_to_nid() and folio_nid() as plain "return 0" instead of reading page->flags when NODES_MASK is 0 and the check can never fire. Link: https://lore.kernel.org/20260630070810.470763-1-hui.zhu@linux.dev Signed-off-by: Hui Zhu Co-developed-by: David Hildenbrand (Arm) Signed-off-by: David Hildenbrand (Arm) Cc: Axel Rasmussen Cc: Barry Song Cc: Kairui Song Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton

mm/page_alloc: use existing highatomic reserves on the buddy fastpath

2026-07-05T23:23:26+00:00

ALLOC_HIGHATOMIC currently provides both access to MIGRATE_HIGHATOMIC free pages and permission to create new highatomic pageblock reserves. This makes it unsuitable for the fastpath. However, the fastpath can reach rmqueue_buddy() while MIGRATE_HIGHATOMIC reserves have free pages available. In this situation, the allocation can fall back to other migratetypes without trying those reserves first. Allow high-priority non-blocking allocations to use existing MIGRATE_HIGHATOMIC reserves on the buddy fastpath without growing them. First tighten the criteria for reserving pageblocks so that growth may only occur in the slowpath. Then allow fastpath usage by enabling ALLOC_HIGHATOMIC when the GFP mask describes a non-blocking high-priority allocation. This logic has been factored out from gfp_to_alloc_flags() to a new function gfp_to_alloc_flags_nonblocking(). A UDP receive workload was run with free MIGRATE_HIGHATOMIC pageblocks available in the target zone. Before this patch, the workload did not consume these blocks. With this patch, eligible order-1 allocations reaching the buddy path consumed existing MIGRATE_HIGHATOMIC pageblocks, with no highatomic misses observed. The workload did not grow highatomic reserves and NAPI page-frag allocations remained healthy with no failures or order-0 fallbacks. Link: https://lore.kernel.org/20260623004600.113347-1-jp.kobryn@linux.dev Signed-off-by: JP Kobryn Reviewed-by: Vlastimil Babka (SUSE) Acked-by: Johannes Weiner Reviewed-by: Shakeel Butt Cc: Brendan Jackman Cc: David Hildenbrand Cc: Frank van der Linden Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Zi Yan Signed-off-by: Andrew Morton

mm: constify oom_control, scan_control, and alloc_context nodemask

2026-07-05T23:23:11+00:00

The nodemasks in these structures may come from a variety of sources, including tasks and cpusets - and should never be modified by any code when being passed around inside another context. Link: https://lore.kernel.org/20260609002919.3967782-1-gourry@gourry.net Signed-off-by: Gregory Price Reviewed-by: Lorenzo Stoakes Acked-by: David Hildenbrand (Arm) Reviewed-by: Barry Song Acked-by: Vlastimil Babka (SUSE) Tested-by: SeongJae Park Acked-by: SeongJae Park Acked-by: Waiman Long Acked-by: Zi Yan Cc: Axel Rasmussen Cc: Baoquan He Cc: Brendan Jackman Cc: Chris Li Cc: David Rientjes Cc: Johannes Weiner Cc: Kairui Song Cc: Kemeng Shi Cc: Liam R. Howlett Cc: Michal Hocko Cc: Michal Koutný Cc: Mike Rapoport Cc: Nhat Pham Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Tejun Heo Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton

mm/page_alloc: drop flag-conversion "optimisation"

2026-07-05T23:22:59+00:00

This code uses flag equivalences to try to optimise conversion from GFP_ to ALLOC_ but there's no clear reason to believe it makes things faster. Even if it gets rid of conditional branches, it just trades them for a data dependency. CPUs are pretty good at conditional branches. But, in my GCC x86 build it doesn't look like there are any branches anyway, the compiler found some conditional instruction tricks. (Caveat: This was extracted & annotated by Gemini AI, I did not actually read the disasm myself) Old code: ae50: 8b 04 24 mov (%rsp),%eax # Load gfp_mask ... ae5d: 41 89 c4 mov %eax,%r12d ae64: 41 81 e4 20 08 00 00 and $0x820,%r12d # Mask both flags at once ... ae6f: 44 89 e1 mov %r12d,%ecx ae77: 83 c9 40 or $0x40,%ecx # OR with ALLOC_CPUSET (0x40) ae7a: 89 4c 24 60 mov %ecx,0x60(%rsp) # Store to alloc_flags New code: For __GFP_HIGH ( 0x20 ): It uses the Carry Flag (via sbb ) to conditionally add 0x20 to the base 0x40 ( ALLOC_CPUSET ) flag: ae63: 83 e0 20 and $0x20,%eax # Test __GFP_HIGH ... ae6a: 83 f8 01 cmp $0x1,%eax # Set carry flag if 0 ae6f: 45 19 e4 sbb %r12d,%r12d # %r12d = (gfp & 0x20) ? 0 : -1 ae80: 41 83 e4 e0 and $0xffffffe0,%r12d # %r12d = (gfp & 0x20) ? 0 : -32 ae87: 41 83 c4 60 add $0x60,%r12d # %r12d = (gfp & 0x20) ? 0x60 : 0x40 For __GFP_KSWAPD_RECLAIM ( 0x800 ): It uses a conditional move ( cmov ) later in the function to set the ALLOC_KSWAPD ( 0x800 ) bit: ae72: 25 00 08 00 00 and $0x800,%eax # Test __GFP_KSWAPD_RECLAIM ae77: 89 44 24 30 mov %eax,0x30(%rsp) # Store result ... af2c: 80 cf 08 or $0x8,%bh # Set ALLOC_KSWAPD (0x800) in temp reg af2f: 45 85 c9 test %r9d,%r9d # Check if __GFP_KSWAPD_RECLAIM was set af32: 0f 44 d8 cmove %eax,%ebx # If not, revert to flags without it Testing with a modified version[0] of lib/free_pages_test.c (adding printks with timing)... Old results from a Sapphire Rapids consumer CPU: [ 67.157118] page_alloc_test: Testing with GFP_KERNEL [ 67.157122] page_alloc_test: Starting 1,000,000 allocations... [ 70.704446] page_alloc_test: Completed. Time: 3543002 us (Avg: 3543.00 ns per alloc+free loop) [ 70.704456] page_alloc_test: Testing with GFP_KERNEL | __GFP_COMP [ 70.704460] page_alloc_test: Starting 1,000,000 allocations... [ 70.944672] page_alloc_test: Completed. Time: 239980 us (Avg: 239.98 ns per alloc+free loop) [ 70.944675] page_alloc_test: Test completed New results: [ 70.079015] page_alloc_test: Testing with GFP_KERNEL [ 70.079020] page_alloc_test: Starting 1,000,000 allocations... [ 73.669396] page_alloc_test: Completed. Time: 3586954 us (Avg: 3586.95 ns per alloc+free loop) [ 73.669402] page_alloc_test: Testing with GFP_KERNEL | __GFP_COMP [ 73.669405] page_alloc_test: Starting 1,000,000 allocations... [ 73.905084] page_alloc_test: Completed. Time: 235496 us (Avg: 235.49 ns per alloc+free loop) [ 73.905086] page_alloc_test: Test completed Seems like a wash. So, drop the flag value coupling here and let the compiler and CPU do their job. Superscalar CPUs are pretty neat after all. (Used AI for the disasm but the rest is all manual). Link: https://lore.kernel.org/20260629-gfp-pessimisation-v2-1-311ece6a8637@google.com Link: https://lore.kernel.org/20260615-gfp-pessimisation-v2-1-65f1319e6818@google.com Link: https://github.com/bjackman/aethelred/blob/2ccdc84ef087c2a631914f58e106e99e19bd3b98/page-alloc-test/page-alloc-test.c [1] Signed-off-by: Brendan Jackman Reviewed-by: Zi Yan Reviewed-by: Vlastimil Babka (SUSE) Reviewed-by: Gregory Price Acked-by: Johannes Weiner Acked-by: Harry Yoo (Oracle) Signed-off-by: Andrew Morton

mm/page_alloc: don't build vm_numa_stat_key if CONFIG_NUMA=n

2026-07-05T23:22:51+00:00

vm_numa_stat_key is only exported if CONFIG_NUMA is set, so avoid the following warning by guarding it in an #ifdef on CONFIG_NUMA: mm/page_alloc.c:165:1: warning: symbol 'vm_numa_stat_key' was not declared. Should it be static? Link: https://lore.kernel.org/20260618100614.1321950-1-ben.dooks@codethink.co.uk Signed-off-by: Ben Dooks Acked-by: Johannes Weiner Reviewed-by: Zi Yan Reviewed-by: SeongJae Park Cc: Brendan Jackman Cc: Michal Hocko Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton

mm/page_alloc: only update NUMA min ratios on sysctl write

2026-06-21T18:31:29+00:00

The sysctl handlers for min_unmapped_ratio and min_slab_ratio invoke setup_min_unmapped_ratio() and setup_min_slab_ratio() unconditionally after proc_dointvec_minmax(), even for read operations. These setup functions first zero all per-NUMA node thresholds (min_unmapped_pages and min_slab_pages) before recalculating them. Reading /proc sysctl entries therefore temporarily resets node reclaim thresholds to zero, which may disturb the behavior of __node_reclaim() and node_reclaim() during the recomputation. Fix this by only calling the setup functions when the sysctl is actually written (write == 1), matching the behavior of existing sysctl handlers like min_free_kbytes and watermark_scale_factor. This only affects systems with CONFIG_NUMA. Link: https://lore.kernel.org/tencent_5891052AF9A4C2D490A62F478D446F74AB09@qq.com Signed-off-by: Jianlin Shi Cc: Brendan Jackman Cc: Johannes Weiner Cc: Michal Hocko Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Zi Yan Signed-off-by: Andrew Morton

mm/alloc_tag: replace fixed-size early PFN array with dynamic linked list

2026-06-21T18:31:28+00:00

Pages allocated before page_ext is available have their codetag left uninitialized. Track these early PFNs and clear their codetag in clear_early_alloc_pfn_tag_refs() to avoid "alloc_tag was not set" warnings when they are freed later. Currently a fixed-size array of 8192 entries is used, with a warning if the limit is exceeded. However, the number of early allocations depends on the number of CPUs and can be larger than 8192. Replace the fixed-size array with a dynamically allocated linked list of pfn_pool structs. Each node is allocated via alloc_page() and mapped to a pfn_pool containing a next pointer, an atomic slot counter, and a PFN array that fills the remainder of the page. The tracking pages themselves are allocated via alloc_page(), which would trigger __pgalloc_tag_add() -> alloc_tag_add_early_pfn() and recurse indefinitely. Introduce __GFP_NO_CODETAG (reuses the %__GFP_NO_OBJ_EXT bit) and pass gfp_flags through pgalloc_tag_add() so that the early path can skip recording allocations that carry this flag. Link: https://lore.kernel.org/20260604024008.46592-1-hao.ge@linux.dev Signed-off-by: Hao Ge Suggested-by: Suren Baghdasaryan Acked-by: Suren Baghdasaryan Cc: Kent Overstreet Signed-off-by: Andrew Morton

mm/page_alloc: fix deferred compaction accounting

2026-06-09T01:21:26+00:00

COMPACT_DEFERRED means compaction did not start because past failures caused the zone to be deferred. try_to_compact_pages() returns the maximum result seen while walking the zonelist, so a final COMPACT_DEFERRED result means no later zone reported that compaction actually ran. __alloc_pages_direct_compact() skips COMPACTSTALL and COMPACTFAIL accounting when try_to_compact_pages() returns COMPACT_SKIPPED, but not when it returns COMPACT_DEFERRED. A deferred-only direct compaction attempt can therefore look like a stall, and then a failure if the allocation still cannot be satisfied. Treat COMPACT_DEFERRED like COMPACT_SKIPPED in this accounting path. If a later zone runs compaction and returns a result above COMPACT_DEFERRED, or compact_zone_order() reports COMPACT_SUCCESS for a captured page, the final result is not COMPACT_DEFERRED and the existing accounting still runs. Link: https://lore.kernel.org/tencent_368AF1F3821E46232637BE16D65C45CF3308@qq.com Fixes: 06dac2f467fe ("mm: compaction: update the COMPACT[STALL|FAIL] events properly") Signed-off-by: fujunjie Reviewed-by: Vlastimil Babka (SUSE) Cc: Brendan Jackman Cc: Johannes Weiner Cc: Michal Hocko Cc: Suren Baghdasaryan Cc: Zi Yan Signed-off-by: Andrew Morton

mm/compaction: respect cpusets when checking retry suitability

2026-06-09T01:21:25+00:00

should_compact_retry() handles COMPACT_SKIPPED by asking compaction_zonelist_suitable() whether reclaim can make a later compaction attempt worthwhile. That answer is used for the current allocation, so it should follow the same zone eligibility rules as the allocation itself. When cpusets are enabled, allocator slowpath decisions are marked with ALLOC_CPUSET. The allocation path, direct compaction and reclaim retry all skip zones rejected by __cpuset_zone_allowed(). compaction_zonelist_suitable() does not apply that filter. It only walks ac->zonelist/ac->nodemask, so it can return true because a zone that is not usable for the current allocation would pass __compaction_suitable(). That does not let the allocation use the disallowed zone. Later allocation and direct compaction paths still apply cpuset filtering. However, it can make should_compact_retry() retry based on memory that this allocation cannot use. Pass gfp_mask down and apply the same ALLOC_CPUSET check in compaction_zonelist_suitable(). This keeps the retry decision aligned with the zones that the allocation is allowed to use. A temporary debugfs probe was also used to call the old and new compaction_zonelist_suitable() predicates in the same two-node NUMA guest. The task was restricted to mems=0 while ac->nodemask covered nodes 0-1. After putting pressure on node0, node0 failed __compaction_suitable() for order-10 and node1 passed it, but node1 was rejected by __cpuset_zone_allowed(). In that state the old predicate returned true and the patched predicate returned false. Link: https://lore.kernel.org/tencent_F59F2BA2CC5779308E10DF54593C736D3E0A@qq.com Fixes: 435b3894e742 ("mm:page_alloc: fix the NULL ac->nodemask in __alloc_pages_slowpath()") Signed-off-by: fujunjie Reviewed-by: Vlastimil Babka (SUSE) Cc: Brendan Jackman Cc: Johannes Weiner Cc: Michal Hocko Cc: Suren Baghdasaryan Cc: Zi Yan Signed-off-by: Andrew Morton

mm/page_alloc: remove VM_BUG_ON()s from pindex helpers

2026-06-04T21:45:03+00:00

Vlastimil pointed out that the VM_BUG_ON()s have fallen out of favour, so remove them. Link: https://lore.kernel.org/20260526-page_alloc-unmapped-prep-v2-1-412f4d486115@google.com Signed-off-by: Brendan Jackman Suggested-by: Vlastimil Babka (SUSE) Link: https://lore.kernel.org/all/4074a816-9e75-45a6-8141-25459bcc106b@kernel.org/ Reviewed-by: Vlastimil Babka (SUSE) Cc: Johannes Weiner Cc: Michal Hocko Cc: Suren Baghdasaryan Cc: Zi Yan Signed-off-by: Andrew Morton