| Age | Commit message (Collapse) | Author |
|---|
|
Patch series "zram: lockmap tweaks".
This patch (of 2):
The zram object contains an array zram_table_entry. Each one has a `lock'
variable and each has a matching struct lockdep_map. This mimics a struct
mutex.
It uses always the same key for all lockdep_map instances. This makes it
look like the same lock to lockdep. Therefore it could be reduced to have
one lockdep_map per struct zram.
Use only one struct lockdep_map per struct zram.
Link: https://lore.kernel.org/20260714141300.3945672-1-bigeasy@linutronix.de
Link: https://lore.kernel.org/20260714141300.3945672-2-bigeasy@linutronix.de
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Reviewed-by: Sergey Senozhatsky <senozhatsky@chromium.org>
Tested-by: Sergey Senozhatsky <senozhatsky@chromium.org>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Minchan Kim <minchan@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The gup_longterm test prints a literal "n" when PIN_LONGTERM_TEST_START
fails with EINVAL because the string is missing the newline escape
sequence.
Print a newline instead.
Link: https://lore.kernel.org/23557F4CB8CF36FF+20260724074603.1479243-1-zhaozhengzhuo@uniontech.com
Signed-off-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
Reviewed-by: Sarthak Sharma <sarthak.sharma@arm.com>
Reviewed-by: Dev Jain <dev.jain@arm.com>
Acked-by: David Hildenbrand (arm) <david@kernel.org>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Shuah Khan <shuah@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
As we deployed defrag_mode into Meta production, pressure spikes and
excessive swapping were observed on some workloads. Tracing confirmed
that this is unmovable/reclaimable requests spinning in the allocator and
direct reclaim, causing excessive amounts of swap.
The initial plan for defrag_mode was to rely on kswapd/kcompactd to
produce blocks, and if those are overwhelmed under high pressure, let the
allocator fall back (__rmqueue_steal()) after its retry loops. However,
that retrying results in more reclaim on some of these workloads than we'd
hoped, sometimes excessively so, spurred on by the !costly order
conditions in should_reclaim_retry().
The storms are dependent on the request type. Reclaim will inevitably
make room in existing movable blocks, since that's where the LRU pages
live. So if movable requests retry on reclaim, they make progress.
When non-movable requests spin in reclaim that isn't productive. They
cannot use the individually freed pages, and the process is unlikely to
accidentally free whole blocks to meet the ALLOC_NOFRAGMENT bar. They
spin and overreclaim excessively, which tanks performance and triggers
userspace guards like swap exhaustion or pressure based OOM.
To fix this, send non-movable requests, regardless of order, into
pageblock reclaim/compaction. This way, they help move things along to
meet the ALLOC_NOFRAGMENT bar. After this patch, the reclaim storms and
excess OOM rates are no longer observed in production.
The longer-term plan is still to have all requests, including the movable
ones, help make blocks to spread the cost of defragmenting more evenly and
fairly; combined with proper watermarking to reduce allocation latencies
in the common case. However, doing this naively unearths scaling and
concurrency limitations in compaction that need to be addressed first.
Promoting just non-movables for now is the minimally viable bug fix for
the above issue.
[brendan.jackman@linux.dev: fix try_to_compact_pages() kerneldoc]
Link: https://lore.kernel.org/DK7NM9RPUJOD.11PNJJ5N2OBED@linux.dev
Link: https://lore.kernel.org/20260722150006.3848560-5-hannes@cmpxchg.org
Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: "Brendan Jackman" <brendan.jackman@linux.dev>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Gregory Price <gourry@gourry.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The compaction capturing code assumes the allocation request order and
compaction target order are the same. That won't be true once defrag_mode
promotes sub-block allocations to pageblock-order compaction: compaction
targets the larger order, while capture should remain at the original
allocation order.
Move the capture_control to the page allocator and give it its own copies
of what the page freeing path matches against - zone, migratetype and the
allocation order - rather than reaching into compaction's live
compact_control. __alloc_pages_direct_compact() fills in migratetype and
order, and installs and hides current->capture_control around the whole
compaction call; try_to_compact_pages() aims capc->zone at each zone while
it is being compacted. compact_zone_order() no longer deals with capture
at all.
Pass the capture_control through try_to_compact_pages() /
compact_zone_order() in place of the bare struct page **.
No functional change.
Link: https://lore.kernel.org/20260722150006.3848560-4-hannes@cmpxchg.org
Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Co-developed-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Gregory Price <gourry@gourry.net>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
While trying to fix a reclaim storm in defrag_mode, I noticed that
non-movable direct compaction is extremely inefficient.
When searching for space to evacuate, compaction only allows blocks of the
same type as the incoming request. This is to prevent migratetype
pollution, where a small non-movable request frees space in a movable
block and provokes the allocator to fall back and pollute it.
This protection is reasonable on one hand, but the downside is that it
makes non-movable direct compaction nearly useless: if we get the type
annotations right, by definition there aren't any movable pages inside the
non-movable blocks it is allowed to scan.
With defrag_mode, the goal is the production of whole blocks, which are
essentially type neutral: __rmqueue_claim() will convert them wholesale on
alloc. This makes type mixing and pollution a non-issue.
Fix the pollution gates to take the requested order into account, and
allow whole-block requests to scan blocks of other types.
The only exception is CMA blocks. That type is sticky and these blocks
cannot be claimed to other types. Continue to be strict with them, and
allow only explicit ALLOC_CMA requests and kcompactd to evacuate them.
Link: https://lore.kernel.org/20260722150006.3848560-3-hannes@cmpxchg.org
Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Gregory Price <gourry@gourry.net>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm: fix reclaim storms in defrag_mode", v2.
As we deployed vm.defrag_mode=1 in Meta production, some workloads
regressed with recurring pressure spikes and swap storms (which in turn
triggered userspace OOM rules on pressure and swap utilization levels).
Tracing pinned this to non-movable requests spinning and reclaiming
unproductively when kswapd/kcompactd are overwhelmed. Direct reclaim
predominantly frees up pages in movable blocks, but those requests cannot
use that space under defrag_mode rules; and it is unlikely to free up
whole blocks incidentally for __rmqueue_claim() to work.
This series fixes it by making non-movable requests participate in
pageblock production in the allocator slowpath - meaning, they will invoke
direct reclaim and direct compaction with pageblock_order.
That requires some small-ish adjustments up front in the allocator and the
compaction code: three prep patches and the fix last.
The series has been in production against one of the affected workloads
for several weeks and restores the OOM kill rate to !defrag_mode baseline.
This patch (of 4):
A subsequent patch will have some order-0 allocations participate in
compaction under defrag_mode, to stave off extfrag events.
Since this is a sprawling expansion of entry points, and compaction can
enter filesystem paths, add lockdep annotations that catches __GFP_FS
passing errors.
Direct reclaim has had this annotation for a while, and since reclaim and
compaction are usually used in conjunction, this is unlikely to unearth
old bugs. It's more about future proofing and peace of mind.
Link: https://lore.kernel.org/20260722150006.3848560-1-hannes@cmpxchg.org
Link: https://lore.kernel.org/20260722150006.3848560-2-hannes@cmpxchg.org
Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Gregory Price <gourry@gourry.net>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
unlock_or_release_subpool() drops spool->lock before calling
subpool_is_free(). However, subpool_is_free() reads fields that are
updated under spool->lock, including count, used_hpages and rsv_hpages.
Keep the free-state evaluation under the same lock that protects those
fields. The reservation accounting and kfree() calls still happen after
dropping spool->lock.
Link: https://lore.kernel.org/20260721035207.1437935-1-chenyichong@uniontech.com
Signed-off-by: Yichong Chen <chenyichong@uniontech.com>
Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
Reviewed-by: Jane Chu <jane.chu@oracle.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Three function definitions terminate with '};' instead of '}', which is
unnecessary and inconsistent with kernel coding style:
- damon_pa_initcall() in paddr.c
- damon_va_initcall() in vaddr.c
- damos_get_some_mem_psi_total() in core.c
No functional change intended.
Link: https://lore.kernel.org/20260721135333.241106-1-sj@kernel.org
Signed-off-by: Xuewen Wang <wangxuewen@kylinos.cn>
Reviewed-by: SJ Park <sj@kernel.org>
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
DAMOS_MIGRATE_{HOT,COLD} passes a target NUMA node to migrate_pages().
But alloc_migration_target() only treats mtc->nid as a preferred node
unless __GFP_THISNODE is set. Hence target allocation can fall back to
another node, and migrate_pages() can report success without placing the
folio on the requested target node.
Consider a two-node tiered system where node 0 is a fast tier and node 1
is a CPU-less slow tier such as CXL memory, and the user wants to promote
hot regions from node 1 to node 0 with a command like:
sudo damo start --ops vaddr --target_pid ${workload_pid} \
--damos_action migrate_hot 0 \
--damos_access_rate 70% max
Without the __GFP_THISNODE flag, when the memory allocator finds that node
0 is nearly full, it can fall back to node 1 without waking up kswapd.
Then the pages allocated for migrate_pages() are still on node 1, and the
regions that are expected to be promoted to node 0 are only moved to
different physical pages on node 1.
Meanwhile, both the mm_migrate_pages tracepoint and DAMOS's own sz_applied
statistics (reported via the damos_stat_after_apply_interval tracepoint)
show the migrations as successful, which makes the failure practically
invisible and hard to investigate.
Running a demotion-purpose DAMOS scheme alongside the promotion scheme
does not fully avoid this either. If demotion cannot keep up with the
promotion rate, allocation can still fall back to node 1 during promotion,
and the same misleading statistics show up.
Make DAMON's migration target allocation strict by setting __GFP_THISNODE,
so that a failed allocation on the target node is reported as a failure
instead of silently landing on a different node. This is consistent with
alloc_misplaced_dst_folio(), alloc_demote_folio(), and with
do_move_pages_to_node(), which all use __GFP_THISNODE for migrations to an
explicit destination node.
Link: https://lore.kernel.org/20260721135607.251869-1-sj@kernel.org
Signed-off-by: Jiahui Zhang <jiahuitry@outlook.com>
Reviewed-by: SJ Park <sj@kernel.org>
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Honggyu Kim <honggyu.kim@sk.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The kernel-doc comment above enum damos_filter_type states that only the
anon and memcg type filters are handled by damon_operations (and therefore
accounted as 'tried'), and that DAMON_OPS_VADDR and DAMON_OPS_FVADDR do
not support those two filter types.
Neither is accurate anymore. damos_filter_for_ops() routes every filter
type except ADDR and TARGET to the operations layer, and the VADDR and
FVADDR operations (the latter being a copy of the former) handle all of
those types through damos_folio_filter_match() / damos_va_filter_out().
Update the comment to match the code.
Link: https://lore.kernel.org/20260721140011.269802-1-sj@kernel.org
Signed-off-by: Song Hu <husong@kylinos.cn>
Reviewed-by: SJ Park <sj@kernel.org>
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
On arm64, __early_set_fixmap(), __late_set_fixmap() and
__late_clear_fixmap() are all __set_fixmap(). Calling
early_ioremap_reset() changes nothing. So remove the call and the macros.
No functional change.
Link: https://lore.kernel.org/20260708170647.362562-4-ekffu200098@gmail.com
Signed-off-by: Sang-Heon Jeon <ekffu200098@gmail.com>
Acked-by: Will Deacon <will@kernel.org>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Alexandre Ghiti <alex@ghiti.fr>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
__late_set_fixmap() and __late_clear_fixmap() are only used after
early_ioremap_reset() has been called. riscv never calls it because
__set_fixmap() works before and after paging_init(). So remove them.
No functional change.
Link: https://lore.kernel.org/20260708170647.362562-3-ekffu200098@gmail.com
Signed-off-by: Sang-Heon Jeon <ekffu200098@gmail.com>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Alexandre Ghiti <alex@ghiti.fr>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm/early_ioremap: clarify and clean up
early_ioremap_reset()".
__late_set_fixmap() and __late_clear_fixmap() are only used after
early_ioremap_reset() has been called, but the comment above them does not
say anything about that. So arm64, riscv and powerpc, whose
__set_fixmap() works before and after paging_init(), describe the same
situation in three different ways:
calls reset defines the macros
arm64 yes yes
riscv no yes
powerpc no no
Patch 1 documents when early_ioremap_reset() needs to be called and that
only architectures calling it need to define the macros.
Patches 2 and 3 remove the unneeded riscv macros, which are unreachable,
and the arm64 reset call and macros, which change nothing.
No functional change.
This patch (of 3):
__late_set_fixmap() and __late_clear_fixmap() are only used after
early_ioremap_reset() has been called.
arm64, riscv and powerpc all have a __set_fixmap() that works before and
after paging_init(), so they do not need to call early_ioremap_reset() or
define the macros, but they describe the same situation in three different
ways:
calls reset defines the macros
arm64 yes yes
riscv no yes
powerpc no no
The existing comment is vague and allows all three. Replace it with
comments that make it clear when the reset and the macros are needed.
No functional change.
Link: https://lore.kernel.org/20260708170647.362562-1-ekffu200098@gmail.com
Link: https://lore.kernel.org/20260708170647.362562-2-ekffu200098@gmail.com
Signed-off-by: Sang-Heon Jeon <ekffu200098@gmail.com>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Alexandre Ghiti <alex@ghiti.fr>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The pagemap_ioctl selftest allocates memory via calloc() in several places
but does not check the return values. If calloc() fails, the subsequent
code will dereference a NULL pointer and crash.
Additionally, in sanity_tests(), the calloc() failure check incorrectly
uses MAP_FAILED (the mmap() error constant) instead of NULL. Since
calloc() returns NULL on failure, the check never triggers and a failed
allocation goes undetected.
Add NULL checks after each calloc() call, and fix the wrong error constant
in sanity_tests(). Use ksft_exit_fail_msg() consistent with the existing
error handling pattern in the file.
Link: https://lore.kernel.org/20260721063611.342-1-yanlonglong@kylinos.cn
Signed-off-by: longlong yan <yanlonglong@kylinos.cn>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: SJ Park <sj@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Replace the direct comparison with (void *)-1 with the standard MAP_FAILED
macro when checking mmap()
Link: https://lore.kernel.org/20260720063439.522-1-yanlonglong@kylinos.cn
Signed-off-by: longlong yan <yanlonglong@kylinos.cn>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Commit a67fe41e214f ("mm: rmap: support batched unmapping for file large
folios") extended batched unmapping for file folios. That also required
making pte_install_uffd_wp_if_needed() support batching, but that was left
out for the time being. Correctness was maintained by stopping batching
if the VMA the folio belongs to is marked uffd-wp.
Now that cond_install_uffd_wp_ptes() supports batching, call it with the
full batch length and allow folio_unmap_pte_batch() to batch file folios
belonging to uffd-wp VMAs.
For file folios, if the uffd-wp bit is set, unmapping converts present
PTEs into uffd-wp markers. We must ensure that the same PTE range is not
reprocessed by the try_to_unmap_one() loop.
The page_vma_mapped_walk API ensures this: check_pte() only returns true
if any PFN in [pvmw->pfn, pvmw->pfn + nr_pages) is mapped by the PTE.
There is no PFN underlying a uffd-wp marker PTE, so check_pte() returns
false and the walk skips ahead until it reaches a present entry again.
Link: https://lore.kernel.org/20260720065508.2695106-4-dev.jain@arm.com
Signed-off-by: Dev Jain <dev.jain@arm.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Harry Yoo <harry@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Enable batch setting of uffd-wp PTE markers.
The code paths passing nr > 1 to zap_install_uffd_wp_if_needed() produce
that nr through either folio_pte_batch() or swap_pte_batch(), therefore
batching is correct:
1) All PTEs belong to the same type of VMA: anonymous or non-anonymous,
wp-armed or non-wp-armed.
2) All PTEs are either marked with uffd-wp or not marked with uffd-wp;
the same applies to the pte_swp_uffd_any() check.
3) uffd_supports_wp_marker() is independent of the function parameters.
Use set_pte_at() in a loop instead of set_ptes(), because set_ptes()
cannot handle nonpresent to nonpresent conversion for nr_pages > 1.
Rename the helper to cond_install_uffd_wp_ptes().
Link: https://lore.kernel.org/20260720065508.2695106-3-dev.jain@arm.com
Signed-off-by: Dev Jain <dev.jain@arm.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Harry Yoo <harry@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "Batch unmap of uffd-wp file folios", v2.
Currently, batched unmapping is supported if:
1) folio is a file folio, not belonging to uffd-wp VMA
2) folio is anonymous and not swapbacked (lazyfree), not belonging to
uffd-wp VMA
So the cases which are not supported are
1) folio belonging to uffd-wp VMA
2) folio is anonymous and swapbacked
It is easy to see that this adds a lot of cognitive load while reading
try_to_unmap_one - we need to remember throughout whether nr_pages == 1 or
> 1.
The uffd-wp handling in try_to_unmap_one is regarding preserving the
uffd-wp state for file folios via pte_install_uffd_wp_if_needed (for anon
folio, we handle that while constructing the swap pte).
Stop special casing on uffd-wp VMAs by simply adding batching support to
pte_install_uffd_wp_if_needed.
This patch (of 3):
pte_install_uffd_wp_if_needed() has grown too large for mm_inline.h. Move
it to memory.c.
This helper is only used inside mm/, so declare it in mm/internal.h
instead of a public header.
While at it, convert the comment to kerneldoc and rename the local
arguments from pte/pteval to ptep/pte so the pointer and PTE value are
easier to distinguish.
Link: https://lore.kernel.org/20260720065508.2695106-1-dev.jain@arm.com
Link: https://lore.kernel.org/20260720065508.2695106-2-dev.jain@arm.com
Signed-off-by: Dev Jain <dev.jain@arm.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Harry Yoo <harry@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Both callers of hugepage_put_subpool() check whether the subpool pointer
is NULL before calling it. Move the NULL check into
hugepage_put_subpool() so callers can use the helper unconditionally.
This is a follow-up cleanup after using hugepage_put_subpool() from the
hugetlbfs_fill_super() failure path.
Link: https://lore.kernel.org/20260720073841.1389354-1-chenyichong@uniontech.com
Signed-off-by: Yichong Chen <chenyichong@uniontech.com>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
For KPF_IDLE, folio_test_idle() check in set_ps_flags() already handles
all config cases, so remove the duplicate code in stable_page_flags() that
deals with CONFIG_PAGE_IDLE_FLAG for the 64‑bit case.
No functional change is intended.
Link: https://lore.kernel.org/20260720033021.4091944-4-tujinjiang@huawei.com
Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Kefeng Wang <wangkefeng.wang@huawei.com>
Cc: Luiz Capitulino <luizcap@redhat.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Nanyong Sun <sunnanyong@huawei.com>
Cc: Svetly Todorov <svetly.todorov@memverge.com>
Cc: xu xin <xu.xin16@zte.com.cn>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Since commit 304daa8132a9 ("maps4: add /proc/kpageflags interface"),
/proc/kpageflags directly operates on page->flags to determine page
status. Later, commit 177975495914 ("proc: export more page flags in
/proc/kpageflags") started using page helper functions when exposing new
flags, leading to a mix of both approaches.
For tail pages, the original code did not return corresponding status.
commit 0a71649cb724 ("/proc/kpageflags: return KPF_SLAB for slab tail
pages") and commit 832fc1de01ae ("/proc/kpageflags: return KPF_BUDDY for
"tail" buddy pages") made tail slab/buddy pages also return corresponding
status. Then commit dee3d0bef2b0 ("proc: rewrite stable_page_flags()")
made all tail pages return the same status as their head page, except for
hwpoison and mapped flags. It also cached the folio's flags and operate
on the flags directly to avoid concurrency issues if using folio_test_*()
helpers.
Since commit 476d87d6a061 ("fs: stable_page_flags(): use
snapshot_page()"), we can now safely switch to folio_test_*() helpers
instead of directly operating on flags, which is more readable and
consistent with the rest of the kernel. Only convert cfolio-specific
flags (i.e., anon, ksm, swapcache) to folio_test_*() helpers, which
reduces redundant code. Keep others unchanged due to they aren't
folio-specific flags or coverting them doesn't cleanup.
No functional change is intended.
Link: https://lore.kernel.org/20260720033021.4091944-3-tujinjiang@huawei.com
Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Kefeng Wang <wangkefeng.wang@huawei.com>
Cc: Luiz Capitulino <luizcap@redhat.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Nanyong Sun <sunnanyong@huawei.com>
Cc: Svetly Todorov <svetly.todorov@memverge.com>
Cc: xu xin <xu.xin16@zte.com.cn>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "cleanup for stable_page_flags()", v2.
This series cleans up stable_page_flags(), see the details in the commit
messages.
This patch (of 3):
The stable_page_flags() function currently sets page flag bits using "1 <<
KPF_xxx" for various KPF_* definitions. KPF_* may be larger than 32,
which will trigger -Wshift-count-overflow warning. All KPF_* values
currently used in "1 << KPF_xxx" are smaller than 33, so no warning is
triggered with current code.
Replace all occurrences of "1 << KPF_*" with the BIT_ULL() macro to ensure
all shifts are performed on a 64-bit unsigned type. This makes the code
robust, and safe for future extension.
No functional change is intended.
Link: https://lore.kernel.org/20260720033021.4091944-1-tujinjiang@huawei.com
Link: https://lore.kernel.org/20260720033021.4091944-2-tujinjiang@huawei.com
Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Zi Yan <ziy@nvidia.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Kefeng Wang <wangkefeng.wang@huawei.com>
Cc: Luiz Capitulino <luizcap@redhat.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Nanyong Sun <sunnanyong@huawei.com>
Cc: Svetly Todorov <svetly.todorov@memverge.com>
Cc: xu xin <xu.xin16@zte.com.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
deferred_split_queue was removed from struct pglist_data {} in commit
fafaeceb89a5 ("mm: switch deferred split shrinker to list_lru"). Remove
its description in doc as well.
Link: https://lore.kernel.org/20260721111919.22923-1-zenghui.yu@linux.dev
Signed-off-by: Zenghui Yu (Huawei) <zenghui.yu@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_apply_min_nr_regions() repeatedly split each region until its size
becomes small enough to meet the user-defined low limit of the number of
regions. The loop assumes the split operation (damon_split_region_at())
will always succeed and create the new region. But the operation could
silently fail for memory allocation failures, for example.
If such failure happens and the region was the last region, the linked
list-based next region fetching returns invalid pointer. As a result,
invalid memory dereference and corruption could happen. Even if the
corner case is handled, it imposes stress to the allocator by trying split
regions for other targets. Fix the issue by breaking all the loops for
any region split failure.
This means there could be a min_nr_regions violation. It will only rarely
happen since the allocation is arguably too small to fail. Even if it
happens, it is only temporal. damon_apply_min_nr_regions() will be called
again after the aggregation interval.
The user impact of the issue should be minor, since the allocation is
arguably too small to fail. But, it could still theoretically happen, and
the consequence is very bad.
This issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260719155442.88794-1-sj@kernel.org
Link: https://lore.kernel.org/20260717011834.120715-1-sj@kernel.org [1]
Fixes: b1029f29eb1d ("mm/damon/core: split regions for min_nr_regions")
Signed-off-by: SJ Park <sj@kernel.org>
Cc: <stable@vger.kernel.org> # 7.1.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Since commit 42f994b71404 ("mm/damon/core: implement scheme-specific apply
interval"), DAMOS scheme can be applied at any time. At that time,
nr_accesses may not be fully aggregated. But the quota prioritization
score is calculated using the not fully aggregated count. As a result,
the performance of DAMOS could be degraded. Fix by using
damon_nr_accesses_mvsum() instead.
The user impact of the issue is suboptimum DAMOS performance under certain
setups. Nonetheless, the bug was there from the beginning of the setup
availability. In other words, the suboptimum performance is the baseline
of the setup and hence it didn't cause regression. Also the extent of the
suboptimality was not big enough to be found from users and testers.
Still, this is a clear bug that is better to be fixed, and can be easily
fixed.
Link: https://lore.kernel.org/20260719161136.90191-1-sj@kernel.org
Fixes: 42f994b71404 ("mm/damon/core: implement scheme-specific apply interval")
Signed-off-by: SJ Park <sj@kernel.org>
Cc: <stable@vger.kernel.org> # 6.7.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
hugetlbfs_fill_super() allocates a hugepage subpool when size or min_size
mount options are specified. hugepage_new_subpool() may also reserve huge
pages for min_size.
If root dentry creation fails after the subpool is created, the failure
path frees the subpool with kfree(). This bypasses hugepage_put_subpool()
and can leave min_size reservations charged.
Use hugepage_put_subpool() on the failure path, matching the normal
put_super path.
Link: https://lore.kernel.org/20260720021900.1376309-1-chenyichong@uniontech.com
Fixes: 7ca02d0ae586 ("hugetlbfs: accept subpool min_size mount option and setup accordingly")
Signed-off-by: Yichong Chen <chenyichong@uniontech.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_migrate_folio_list()
In damon_migrate_folio_list(), we're redeclaring folio inside the first
while loop, but it just shadows the outer one. Since the second loop uses
the outer folio anyway, the inner declaration is pointless.
Remove it to consistently reuse the same variable throughout the
function and improve readability.
Link: https://lore.kernel.org/20260718002125.637104-1-lienze@kylinos.cn
Signed-off-by: Enze Li <lienze@kylinos.cn>
Reviewed-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_test_commit_target_regions_for() traverses expected values array
after damon_commit_target_regions() call. It assumes
damon_commit_target_regions() made expected number of regions. It might
not. Because the traversal is made based on the region count, it could do
out of bounds access to the expectation value array.
The consequent user impact (out-of-bound access) is quite bad. The
realistic user impact would be limited, though. It would affect only test
run setups.
Fix it by testing if the number of regions was also changed as expected
and exit early for the failure.
The issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260718001442.87129-8-sj@kernel.org
Link: https://lore.kernel.org/20260713144757.39740-1-sj@kernel.org [1]
Fixes: 603f67eb91e0 ("mm/damon/tests/core-kunit: add damon_commit_target_regions() test")
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: <stable@vger.kernel.org> # 6.19.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damos_test_commit_quota_goals_for() traverses damos quota goals after
damos_commit_quota_goals() call. It assumes damos_commit_quota_goals()
made expected numbers of goals. It might not. Because the traversal is
made based on destination struct length, it could do out of bounds access
for source expectation value array.
The consequent user impact (out-of-bound access ) is quite bad. The
realistic user impact would be limited though. It would affect only test
run setups.
Fix it by testing if the number of goals was also changed as expected and
exit early for the failure.
The issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260718001442.87129-7-sj@kernel.org
Link: https://lore.kernel.org/20260713144757.39740-1-sj@kernel.org [1]
Fixes: d9adfb8a28e7 ("mm/damon/tests/core-kunit: add damos_commit_quota_goals() test")
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: <stable@vger.kernel.org> # 6.19.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damos_test_commit_dests_for() traverse damos action destinations after
damos_commit_dests(). It assumes damos_commit_dests() made expected
numbers of destinations for source and destination structures. It might
not. Because the traversal is made based on destination struct length, it
could do out of bounds access for source value expectation.
The consequent user impact (out-of-bound access ) is quite bad. The
realistic user impact would be limited, though. It would affect only test
run setups.
Fix it by exiting early for the number of regions test failure.
The issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260718001442.87129-6-sj@kernel.org
Link: https://lore.kernel.org/20260713144757.39740-1-sj@kernel.org [1]
Fixes: eec573b8dd65 ("mm/damon/tests/core-kunit: add damos_commit_dests() test")
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: <stable@vger.kernel.org> # 6.19.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damos_test_filter_out() test checks if damos_filter_match() of an address
filter splits the region as expected under a given condition. But, the
test continued regardless of the split successes. As a result, the later
part of the test could dereference invalid pointers that returned from
damon_next_region(). Further, it could corrupt memory from
damon_destroy_region().
The consequent user impact (memory corruption) is quite bad. The
realistic user impact would be limited, though. It would affect only test
run setups.
Fix it by exiting early for the number of regions test failure.
The issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260718001442.87129-5-sj@kernel.org
Link: https://lore.kernel.org/20260714142352.100478-1-sj@kernel.org [1]
Fixes: 26713c890875 ("mm/damon/core-test: add a unit test for __damos_filter_out()")
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: <stable@vger.kernel.org> # 6.6.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_do_test_apply_three_regions() iterates regions after
damon_set_regions() call assuming the function would succeed at setting
the number of regions the same to the expected one. It might have failed.
In this case, __nth_region_of() in the iteration could return NULL and
NULL dereference can happen in the test.
The consequent user impact (NULL dereference) is quite bad. The realistic
user impact would be limited, though. It would affect only test run
setups.
Fix it by testing if the number of regions was also changed as expected
and exit early for the failure.
The issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260718001442.87129-4-sj@kernel.org
Link: https://lore.kernel.org/20260713144757.39740-1-sj@kernel.org [1]
Fixes: 17ccae8bb5c9 ("mm/damon: add kunit tests")
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: <stable@vger.kernel.org> # 5.15.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
damon_test_split_at() test next region that is assumed to be created by
damon_split_region_at() invocation. But the split might fail. In this
case, the succeeding test may dereference invalid pointers returned by
damon_next_region().
The invalid pointer may not cause a really bad user impact, because of the
implementation detail. It would only read wrong contents in the belonging
damon_target struct. Depending on the future change of the offset from
the link header to the accessing field, this could also be really
dangerous, though. Still, the realistic user impact would be limited. It
would affect only test run setups.
Fix it by testing if the number of regions was also changed as expected
and exit early for the failure.
The issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260718001442.87129-3-sj@kernel.org
Link: https://lore.kernel.org/20260714142352.100478-1-sj@kernel.org [1]
Fixes: 17ccae8bb5c9 ("mm/damon: add kunit tests")
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: <stable@vger.kernel.org> # 5.15.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm/damon: fix uninitialized DAMOS field and kunit exec
expectation bugs".
Fix a few Sashiko-found unurgent bugs. Patch 1 fixes use of uninitialized
damos->last_applied field. Patches 2-7 fix DAMON kunit tests that do
invalid memory access under test failures.
The bugs are better to be fixed and eventually merged into stable@ kernel.
That said, the fixes are arguably not urgent. Patch 1 only introduces
negligible DAMOS efficiency degradation in occasional cases. Kunit fixes
could introduce quite bad consequences but those are test code that affect
only test run setups.
This patch (of 7):
Multiple DAMON regions could exist across a folio. If they fulfill the
condition to apply a DAMOS scheme, the scheme could be applied multiple
times to the folio. To avoid this, each DAMOS scheme stores the folio
that the scheme was applied to last time in the damos->last_applied field
and skips repeatedly applying the same scheme to the same folio.
The field is being used without initialization, though. Hence, the
mechanism could wrongly skip applying a scheme to a folio at the very
first time of DAMOS run.
The user impact is trivial. DAMON might unexpectedly skip applying DAMOS
action for one folio for the first time per scheme. In the DAMON's
best-effort world, this is never a real problem. No critical consequences
such as kernel panic or memory corruption happen.
It is a clear bug, though, and the fix is straightforward. Fix the issue
by initializing the field in DAMOS scheme creation function,
damon_new_scheme().
The issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260718001442.87129-1-sj@kernel.org
Link: https://lore.kernel.org/20260718001442.87129-2-sj@kernel.org
Link: https://lore.kernel.org/20260714055436.120034-1-sj@kernel.org [1]
Fixes: 94ba17adaba0 ("mm/damon: avoid applying DAMOS action to same entity multiple times")
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: <stable@vger.kernel.org> # 6.15.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
split_folio_to_list_to_order() had no direct callers; its only user was
split_folio_to_order(), which always passed a NULL list. Fold it into
split_folio_to_order() and call split_huge_page_to_list_to_order()
directly.
No functional change.
Link: https://lore.kernel.org/20260716100345.473473-3-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Acked-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <npache@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm/huge_memory: drop dead split helper variants".
Two trivial cleanups in the folio split API, found while auditing the
split helpers for the inode UAF fix [1]:
- can_split_folio() is dead: only the !THP stub remains, no definition
under CONFIG_TRANSPARENT_HUGEPAGE and no callers. Leftover from an
earlier cleanup.
- split_folio_to_list_to_order() has no direct callers; its only user
is split_folio_to_order(), which always passes a NULL list. Fold it.
No functional change.
This patch (of 2):
can_split_folio() has no callers and no definition under
CONFIG_TRANSPARENT_HUGEPAGE; only the !THP stub remains, left behind by an
earlier cleanup. Remove it.
No functional change.
Link: https://lore.kernel.org/20260716100345.473473-1-kirill@shutemov.name
Link: https://lore.kernel.org/20260716100345.473473-2-kirill@shutemov.name
Link: https://lore.kernel.org/linux-mm/aldjhtfVByHDQXe6@thinkstation [1]
Link: https://lore.kernel.org/linux-mm/20260714122344.351895-1-kirill@shutemov.name [2]
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Acked-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <npache@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Use early returns to reduce indentation and improve readability.
Link: https://lore.kernel.org/20260717091347.1144789-7-wangkefeng.wang@huawei.com
Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
walk_hugetlb_range() always passes a non-NULL pte, so remove the dead NULL
check. Replace the per-page iteration loop with memset() for better
performance.
[wangkefeng.wang@huawei.com: fixup improve mincore_hugetlb()]
Link: https://lore.kernel.org/20260720124252.1483984-1-wangkefeng.wang@huawei.com
Link: https://lore.kernel.org/20260717091347.1144789-6-wangkefeng.wang@huawei.com
Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The swap_cache_get_folio() no longer returns shadow entries, so the
xa_is_value() check is unnecessary.
Link: https://lore.kernel.org/20260717091347.1144789-5-wangkefeng.wang@huawei.com
Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Remove ugly casts by using the more natural kmalloc/kfree allocation, also
replace GFP_USER(pointless here) with GFP_KERNEL.
Link: https://lore.kernel.org/20260717091347.1144789-4-wangkefeng.wang@huawei.com
Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
As David pointed out, "it's hard to believe that someone depends on pages
in VM_PFNMAP to *not* be present", so remove the historical behavior that
always reports VM_PFNMAP pages as non-resident.
Adding mincore_pud_range() allows the page table walker to traverse the
full page table hierarchy without splitting a huge PUD, enabling much more
precise identification of resident pages in PFNMAP VMAs.
[wangkefeng.wang@huawei.com: address David's comments]
Link: https://lore.kernel.org/20260720124151.1483820-1-wangkefeng.wang@huawei.com
Link: https://lore.kernel.org/linux-mm/0e619d71-1c3d-4534-8376-2982c7348c31@kernel.org/
Link: https://lore.kernel.org/20260717091347.1144789-3-wangkefeng.wang@huawei.com
Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm: mincore: misc cleanups", v3.
This series cleans up and simplifies the mincore. Most importantly, it
removes the historical special behavior that always reports VM_PFNMAP
pages as non-resident.
This patch (of 6):
Introduce the pud_is_huge() helper function to check whether a PUD is a
huge PUD or a software leaf entry. Note that, unlike PMD, PUD currently
does not support transparent migration entries or device private entries.
However, the function still includes `!pud_none()` branches to accommodate
potential future implementations.
Link: https://lore.kernel.org/20260717091347.1144789-1-wangkefeng.wang@huawei.com
Link: https://lore.kernel.org/20260717091347.1144789-2-wangkefeng.wang@huawei.com
Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
In mremap_dontunmap_partial_mapping_overwrite(), mremap() stores its
returned remapped address into remapped_mapping, while the BUG_ON
incorrectly checks dest_mapping instead.
If mremap() fails, dest_mapping still holds a valid pointer obtained from
the prior mmap() call, making this BUG_ON never trigger at all. Correct
the check to validate the actual mremap() return value stored in
remapped_mapping.
Link: https://lore.kernel.org/20260717070251.73212-1-hongfu.li@linux.dev
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
Reviewed-by: SJ Park <sj@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The previous patch fixed this issue for the traditional LRU. The same
problem exists in MGLRU [1]: when swappiness=max (SWAPPINESS_ANON_ONLY) is
set, reclaim is expected to evict anonymous pages exclusively, but file
pages can still be reclaimed when anonymous pages cannot be reclaimed
(e.g. no swap and no demotion target).
Fix it the same way as the traditional LRU: keep returning
SWAPPINESS_ANON_ONLY in get_swappiness(), and return 0 from
get_nr_to_scan() when SWAPPINESS_ANON_ONLY is set but anon pages cannot be
reclaimed. Since get_nr_to_scan() decides how much MGLRU scans, returning
0 skips the scan entirely and avoids the useless scan work when there is
nothing eligible to reclaim.
The test result:
Before fix:
# cat /sys/kernel/mm/lru_gen/enabled
0x0007
# cat memory.stat
anon 204800
file 67108864
...
pgsteal_proactive 0
pgscan_proactive 0
# echo "64M swappiness=max" > memory.reclaim
# cat memory.stat
anon 208896
file 0
...
pgsteal_proactive 16384
pgscan_proactive 16384
After fix:
# cat memory.stat
anon 188416
file 67215360
kernel 1970176
...
pgsteal_proactive 0
pgscan_proactive 0
# echo "64M swappiness=max" > memory.reclaim
-bash: echo: write error: Resource temporarily unavailable
# cat memory.stat
anon 204800
file 67215360
...
pgsteal_proactive 0
pgscan_proactive 0
Link: https://lore.kernel.org/20260724033435.2573323-5-ridong.chen@linux.dev
Link: https://sashiko.dev/#/patchset/20260717113300.214717-1-ridong.chen@linux.dev [1]
Fixes: 68a1436bde00 ("mm: add swappiness=max arg to memory.reclaim for only anon reclaim")
Signed-off-by: Ridong Chen <chenridong@xiaomi.com>
Reviewed-by: Kairui Song <kasong@tencent.com>
Reviewed-by: Barry Song <baohua@kernel.org>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Zhongkun He <hezhongkun.hzk@bytedance.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Commit 57972c78e678 ("mm/vmscan: make __node_reclaim() more generic")
moved the scan_control construction out to the callers and passed the
struct in by pointer. After that change every use of the gfp mask inside
__node_reclaim() goes through sc->gfp_mask, leaving the gfp_mask parameter
unused. Just remove the dead parameter and update the callers
accordingly. No functional change.
Link: https://lore.kernel.org/20260724033435.2573323-4-ridong.chen@linux.dev
Signed-off-by: Ridong Chen <chenridong@xiaomi.com>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Barry Song <baohua@kernel.org>
Reviewed-by: Qi Zheng <qi.zheng@linux.dev>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Kairui Song <kasong@tencent.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Zhongkun He <hezhongkun.hzk@bytedance.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
It has been observed that per-node proactive reclaim always returns
-EAGAIN when any error occurs. As discussed in the mailing list [1], the
interface should distinguish between cases where no reclaimable memory is
left and where another entity is concurrently using the same interface.
Propagate the real error code, consistent with how memcg proactive reclaim
handles errors.
Link: https://lore.kernel.org/20260724033435.2573323-3-ridong.chen@linux.dev
Link: https://lore.kernel.org/all/20250717235604.2atyx2aobwowpge3@offworld/T/#m3514718be82a31b05726a49da9b61fbfc69a589e [1]
Fixes: b980077899ea ("mm: introduce per-node proactive reclaim interface")
Signed-off-by: Ridong Chen <chenridong@xiaomi.com>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Reviewed-by: Qi Zheng <qi.zheng@linux.dev>
Reviewed-by: Barry Song <baohua@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Kairui Song <kasong@tencent.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Zhongkun He <hezhongkun.hzk@bytedance.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm/vmscan: fix swappiness=max and clean up per-node
proactive reclaim", v4.
Fixes and one cleanup.
Patch 1 fixes "swappiness=max": the anon-only test in get_scan_count() sat
after the "cannot reclaim anon" check, so when no anon was reclaimable the
request fell back to SCAN_FILE and evicted page cache instead.
Patch 2 fixes reclaim_store() collapsing every error into -EAGAIN, so
callers can no longer tell an invalid argument from a busy interface;
propagate the real error code, matching the memcg path.
Patch 3 drops the now-unused gfp_mask parameter from __node_reclaim().
Patch 4 fixes the same "swappiness=max" issue for MGLRU.
This patch (of 4):
As Qi mentioned [1], when swappiness=max (SWAPPINESS_ANON_ONLY) is set,
the reclaim logic is expected to reclaim anonymous pages exclusively.
However, due to the current ordering of checks in get_scan_count(), file
pages may still be evicted if can_reclaim_anon_pages() returns false,
which contradicts the semantics of SWAPPINESS_ANON_ONLY.
Reproducer in a cgroup holding 64M of file cache, with no swap configured:
Before (file cache is wrongly evicted):
# cat memory.stat
anon 196608
file 67178496
pgscan_proactive 0
# echo "64M swappiness=max" > memory.reclaim
# cat memory.stat
anon 208896
file 4096 <- page cache evicted
pgsteal_proactive 16400
pgscan_proactive 16400
After (file cache is left intact):
# cat memory.stat
anon 200704
file 67178496
pgscan_proactive 0
# echo "64M swappiness=max" > memory.reclaim
-bash: echo: write error: Resource temporarily unavailable
# cat memory.stat
anon 208896
file 67178496 <- page cache untouched
pgsteal_proactive 0
pgscan_proactive 0
Fix this by bailing out early when SWAPPINESS_ANON_ONLY is set and no
anonymous pages are reclaimable, before falling back to file reclaim.
Link: https://lore.kernel.org/20260724033435.2573323-1-ridong.chen@linux.dev
Link: https://lore.kernel.org/20260724033435.2573323-2-ridong.chen@linux.dev
Link: https://lore.kernel.org/cgroups/7ddf3eee-5fe2-45f7-8614-c8936a039e04@linux.dev/ [1]
Fixes: 68a1436bde00 ("mm: add swappiness=max arg to memory.reclaim for only anon reclaim")
Signed-off-by: Ridong Chen <chenridong@xiaomi.com>
Suggested-by: Qi Zheng <qi.zheng@linux.dev>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Qi Zheng <qi.zheng@linux.dev>
Reviewed-by: Barry Song <baohua@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Kairui Song <kasong@tencent.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Zhongkun He <hezhongkun.hzk@bytedance.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
mseal.o is built only for 64-bit kernels, so 32-bit kernels fall back to
sys_ni_syscall() and return -ENOSYS rather than -EPERM.
Drop architecture description in mseal.rst, since the arch feature doc has
the latest state of mseal for each architecture.
Fix the CONFIG_MSEAL_SYSTEM_MAPPINGS typo in init/Kconfig.
Link: https://lore.kernel.org/20260715131258.55499-1-leon.hwang@linux.dev
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Acked-by: Lance Yang <lance.yang@linux.dev>
Cc: Alice Ryhl <aliceryhl@google.com>
Cc: Anand Moon <linux.amoon@gmail.com>
Cc: Doug Anderson <dianders@chromium.org>
Cc: Gary Guo <gary@garyguo.net>
Cc: Jann Horn <jannh@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Leon Hwang <leon.hwang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Miguel Ojeda <ojeda@kernel.org>
Cc: Nathan Chancellor <nathan@kernel.org>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
There's no need to abstract do_mseal() any longer so put the system call
implementation in the system call declaration.
The comment around do_mseal() is strangely formatted, overly long and adds
a lot of superfluous information that the code already provides, so boil
it down to the essentials.
Link: https://lore.kernel.org/20260717-mseal-fixups-v2-3-0daa0014b813@kernel.org
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Jan Kara <jack@suse.cz>
Cc: Jann Horn <jannh@google.com>
Cc: Kees Cook <kees@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Commit 44f65d900698 ("binfmt_elf: mseal address zero") unconditionally
provided do_mseal() to any internal kernel caller in order to address a
corner case slated for possible removal.
It also incorrectly attempts to mseal without checking to see whether the
mapping even succeeded.
Restrict the scope to the corner case by providing mseal_mmap_page_zero()
which asserts the MMAP_PAGE_ZERO personality.
Avoid unnecessary checks in the start, end range by abstracting the actual
mseal()'ing to mseal_range() and have mseal_mmap_page_zero() call that
instead.
Also only try to seal the VMA if we mapped the VMA. This isn't strictly
necessary as the operation would error out anyway, but it's useless work
and could be problematic if me make future changes to mseal semantics.
Link: https://lore.kernel.org/20260717-mseal-fixups-v2-2-0daa0014b813@kernel.org
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Jan Kara <jack@suse.cz>
Cc: Jann Horn <jannh@google.com>
Cc: Kees Cook <kees@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
