kernel/git/stable/linux.git/mm/folio.c, branch master

memcg: move LRU size accounting on reparenting instead of copying it

2026-08-25T01:43:33+00:00

When a memory cgroup is offlined its LRU folios are reparented to the parent. lruvec_reparent_lru() splices the child's lists into the parent's and credits the parent with the child's per-zone lru_zone_size[], but never clears the child's copy, so the size is copied rather than moved. lru_gen_reparent_memcg() does the same for MGLRU. The parent is left correct, credited with exactly the folios it took over. The stale value sits on the child and nothing will correct it: folio->memcg_data now resolves to the parent, so every later update_lru_size() for those folios goes there. Dying cgroups are not freed immediately and mem_cgroup_iter() still walks them, so shrink_lruvec() keeps being called on them. get_scan_count() reads the phantom counter through lruvec_lru_size() and the scan loop then grinds through nr[] in SWAP_CLUSTER_MAX steps against an empty list, for as long as the dead cgroup lives. Under MGLRU the MGLRU scanner runs instead, but count_shadow_nodes() sums all of NR_LRU_LISTS through lruvec_lru_size() and over-budgets the shadow node limit just the same. On one 251 GiB host a sweep of every mz->lru_zone_size[] found 380 counters describing folios on no list at all: 124777314 pages, 476 GiB, 1.89x the machine's RAM, across 57 cgroups. All were on memcgs with CSS_DYING set and CSS_ONLINE clear, and parent/child pairs reported byte-identical sizes. LRU_UNEVICTABLE needs its size moved too. Its list is deliberately not spliced because lruvec_init() poisons the head - the unevictable LRU is imaginary and folios are never threaded on it - but the size is kept by lruvec_add_folio()/lruvec_del_folio() and those folios account to the parent from here on. This depends on commit bf4ade7dbd76 ("memcg: keep folio's objcg same as its node") and must not be backported ahead of it. Without that invariant a folio's objcg can belong to another node, so a folio already spliced onto the parent's list can still resolve to the child's lruvec until the objcg's node is reparented in a later iteration of memcg_reparent_objcgs(); clearing the child's counter early then lets lruvec_del_folio() underflow it and trip the WARN_ONCE()/VM_BUG_ON() in mem_cgroup_update_lru_size(). Link: https://lore.kernel.org/20260822024707.77192-1-shakeel.butt@linux.dev Fixes: 07a6e9a2c199 ("mm: vmscan: prepare for reparenting traditional LRU folios") Fixes: f304652609ea ("mm: vmscan: prepare for reparenting MGLRU folios") Signed-off-by: Shakeel Butt Acked-by: Michal Hocko Cc: Johannes Weiner Cc: Roman Gushchin Cc: Muchun Song Cc: # After: bf4ade7dbd76: memcg: keep folio's objcg same as its node Signed-off-by: Andrew Morton

mm/gup: factor out LRU cache draining for folio into lru_cache_drain_for_folio()

2026-08-25T01:43:00+00:00

KVM with guest_memfd wants to remove any folio references due to LRU caches, as it really must only allow to convert folios from shared to private when there are no unexpected folio references (e.g., from GUP references). So, to drive the refcount down, it needs a way to flush the LRU caches. Let's factor out what we have in lru_cache_drain_for_folio(). Document it, and also mention that concurrent folio (un)mapping might, in theory, miss detecting LRU cache references. Keep obtaining the expected refcount twice to minimize the possibility. For the current and future user that should work, and we don't really have a better alternative: we could detect if the mapcount changed, but it would still be racy and add more complexity with questionable benefit. Maybe there is a chance to avoid the draining entirely in the future, by avoiding extra references from the LRU cache: Hugh thinks there might be a way. But for the time being, this handling is unfortunately required. Make folio_may_be_lru_cached() accept a const pointer so lru_cache_drain_for_folio() can accept a const pointer as well. Link: https://lore.kernel.org/20260806-lru_cache_drain_for_folio-v1-1-c6287d295e99@kernel.org Signed-off-by: David Hildenbrand (Arm) Reviewed-by: Fuad Tabba Cc: Ackerley Tng Cc: Baoquan He Cc: Barry Song Cc: Chris Li Cc: Jason Gunthorpe Cc: John Hubbard Cc: Kairui Song Cc: Kemeng Shi Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Nhat Pham Cc: Peter Xu Cc: Sean Christopherson Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton

mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost

2026-08-25T01:42:56+00:00

The anon/file scan balance in get_scan_count() is driven by two scalars in struct lruvec, anon_cost and file_cost, accumulated by every reclaim producer under lruvec->lru_lock. The acquisition sites for cost work specifically are: - shrink_inactive_list() re-takes lru_lock at function exit purely to call lru_note_cost_unlock_irq() with (nr_pageout, nr_scanned - nr_reclaimed). One acquisition per inactive shrink. - shrink_active_list() does the same with (0, nr_rotated). One acquisition per active shrink. - workingset_refault() takes the lock via folio_lruvec_lock_irq() purely to record the refault cost. One acquisition per refault. - prepare_scan_control() takes lru_lock just to snapshot the two scalars into sc->{anon,file}_cost. - lru_note_cost_unlock_irq() itself walks parent_lruvec and re-acquires lru_lock on each ancestor to propagate the update, adding O(memcg-depth) acquisitions per producer call. This hurts because lru_lock is already a heavy contention point on memory-heavy workloads: every isolate_lru_folios(), move_folios_to_lru() and folio_add_lru() takes it. The cost work itself is trivial (two scalar bumps and one comparison), but it contends with and causes contention for actual LRU manipulation. The parent_lruvec() walk also multiplies cost-update overhead by memcg hierarchy depth. The balance formula for anon and file, respectively, is this: cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated Instead of recording cost and running averaging logic directly when these events occur, snapshot running vmstat counters once per reclaim cycle and derive the balance from event deltas since the last run. Use PGROTATE_* from the preceding patch for the rotation input. WORKINGSET_RESTORE_* and NR_VMSCAN_WRITE provide the remaining event counters. Charge NR_VMSCAN_WRITE through lruvec stats so all inputs can be sampled per lruvec and aggregated through the memcg hierarchy. This is overall cheaper and has fewer lock acquisition sites. Moving accumulation and decay to the reclaim side also improves the cost model across reclaim gaps. With producer-side decay, events that happen while reclaim is idle still age each other before reclaim ever samples the costs. If a workload refaults a large anon set and then a smaller file set before reclaim runs again, the later file activity can age the earlier anon activity out of the cost model. The new scheme observes the whole between-reclaim delta and decays anon and file proportionally, so the scan-balance history better represents what happened since the last reclaim pass. A dedicated per-lruvec spinlock, cost_lock, serialises the delta extraction, the cost->count update and the halving loop against concurrent reclaimers in the same memcg+node. NR_VMSCAN_WRITE is accounted at writeout(), so reclaim_stat.nr_pageout is no longer needed and is removed. memcg-v1's memory.stat anon_cost/file_cost is now sourced from cost[].count instead of the removed lruvec anon_cost/file_cost fields. The reported values only refresh when prepare_scan_control() runs and are bounded at ~lrusize/4 by the halving loop; the scan-balance signal they express is unchanged. Under pure MGLRU the scan-balance signal itself is not consumed (both prepare_scan_control() and get_scan_count() are short-circuited on the MGLRU paths, and MGLRU's own type/tier selection comes from read_ctrl_pos() on lrugen->{avg_refaulted,avg_total,refaulted,evicted}, not from anon_cost/file_cost). NR_VMSCAN_WRITE naturally covers writeout from either reclaim implementation. The preceding patch also bumps PGROTATE_{ANON,FILE} from evict_folios(), so rotation-driven reclaim work is accounted consistently across both implementations. Link: https://lore.kernel.org/20260727162550.2032-4-usama.arif@linux.dev Signed-off-by: Usama Arif Acked-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Vlastimil Babka (SUSE) Cc: Axel Rasmussen Cc: Baoquan He Cc: Chris Li Cc: David Hildenbrand Cc: David Rientjes Cc: Kairui Song Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Nhat Pham Cc: Roman Gushchin Cc: Suren Baghdasaryan Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton

mm: rename swap.c to folio.c

2026-08-05T02:18:35+00:00

Rename mm/swap.c to mm/folio.c so the filename better matches the code's main responsibility. This keeps the implementation split from swap-specific code without changing the published LRU helper interfaces. Update MAINTAINERS and the remaining mm/swap.c documentation references after the rename. [wujianyue000@gmail.com: MAINTAINERS: move mm/folio.c to MM CORE] Link: https://lore.kernel.org/20260804111109.393127-1-wujianyue000@gmail.com Link: https://lore.kernel.org/20260708-ch-swap-series-plus-folio-lru-cleanup-v9-2-2bc72b4f8730@gmail.com Signed-off-by: Jianyue Wu Suggested-by: Baoquan He Suggested-by: David Hildenbrand Suggested-by: Matthew Wilcox Reviewed-by: Lorenzo Stoakes Acked-by: Johannes Weiner Acked-by: David Hildenbrand (Arm) Cc: Axel Rasmussen Cc: Baolin Wang Cc: Barry Song Cc: Chris Li Cc: Hugh Dickins Cc: Jonathan Corbet Cc: Kairui Song Cc: Kemeng Shi Cc: Liam R. Howlett Cc: Michal Hocko Cc: Michal Hocko Cc: Mike Rapoport Cc: Nhat Pham Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton