kernel/git/stable/linux.git/fs/btrfs/extent_io.c, branch master

btrfs: trigger cow fixup via dirty_folio()

2026-07-30T17:28:36+00:00

The problem scenario: If we have a folio mmapped shared and then somebody does a dio read with that folio as the read destination, then it is possible that the dio will see a dirty destination page when it starts (and thus skip dirtying and just GUP pin it) but then while it is doing the read, btrfs finishes writing it back and by the endio, the folio is clean. In that case, the dio read must re-dirty the folio with aops->dirty_folio(): btrfs_check_read_bio() |- __iomap_dio_bio_end_io() from btrfs_bio_end_io() |- bio_check_pages_dirty() |- bio_dirty_fn() |- bio_release_pages(bio, true) |- __bio_release_pages(bio, mark_dirty == true) |- folio_lock() |- folio_mark_dirty() |- aops->dirty_folio() |- folio_unlock() A data block normally moves through writeback as follows: TASK folio_lock write clean -> dirty bit + delalloc folio_unlock WRITEBACK for-each-dirty-folio: folio_lock run_delalloc delalloc consumed -> dirty bit + OE submission dirty bit consumed -> writeback bit + OE folio_unlock ENDIO endio OE bytes accounted OE finish writeback -> clean; destroy OE Three critical invariants that this path maintains are: I1. Any dirty block is covered by delalloc xor an ordered extent I2. Any dirty block covered by an OE will be submitted into that OE I3. Any dirty block already submitted into an OE will not be submitted again into the same OE. These ensure that the block will be written exactly once. It is clear that not reserving delalloc for the re-dirty case violates I1. This situation, even without bs < folio_size, has long required btrfs to fixup such dirty pages during writeback with an asynchronous worker that is allowed to do this expensive work and writeback does not proceed for a folio while it is doing this work. Commit 247e743cbe6e ("Btrfs: Use async helpers to deal with pages that have been improperly dirtied") introduced the COW fixup to catch exactly this class at writeback, way back in 2008. Since then, there have been many advances to prevent most of the causes of such re-dirtying and we thought we could get away with removing the annoying cow-fixup in the hope of simplifying writeback for large folio support. Commit b2a9f217ad3f ("btrfs: remove the COW fixup mechanism") Commit 4927b141877c ("btrfs: remove folio ordered flag and subpage bitmap") Since it turns out this assumption was incorrect, as evidenced by the report and attendant reproducers, we must reintroduce the fixup concept. This is of course critically further complicated by bs < folio_size. In that case, rather than just a folio dirty bit, we have a bitmap for the dirty blocks in the folio. And the (also broken) invariant is: I4. folio dirty IFF at least one block bitmap dirty. The original report of a stall on a misinterpreted empty bitmap is exactly evidence of a violation of I4. It is exactly because of bs < folio_size we don't want to simply revert the removal patches. The original fixup was not properly bs < folio_size aware, which motivated removal in the first place. So we wish to build a bs < folio_size aware fixup. One other important detail from the old design, any normal write that happens after a re-dirty but before a fixup is racing with the cow fixup to do the delalloc reservation, therefore it must cancel the fixup state. If it arrives after the reservation exists, it will be a normal dirty overwrite. This critically informs the design in a pretty clear way. fixup requiring re-dirty has folio granularity, while cancellation has delalloc (block) granularity so while we only ever produce fixup in chunks of folios, we must be able to clear it in blocks. Therefore we must track the blocks needing fixup at block granularity. The obvious way to do this is with a new bitmap in btrfs_folio_state, but it is desirable to avoid that if possible. Unfortunately, I don't think it is possible and the reason is subtle and leans on a sort of extreme reproducer, but I think can be explained relatively succinctly. Consider a folio whose two halves will land in different ordered extents (can be accomplished with tricks using nodatasum) and a dio read is running with it as the shared mmap destination. 1. The front half: a. folio comes clean on a normal write b. dio read completes into the folio marking it fixup. c. a write comes for the previous folio for a range extending into this folio, this is a cancellation of the fixup which reserves space. d. writeback runs on the range *not* overlapping the folio. This half remains dirty but is now covered by an OE and is awaiting writeback running on its range to be submitted and finish the OE. 2. The back half: a. the folio is part of an OE that gets far enough along to clear writeback. b. dio read completes into the folio marking it fixup. After this, the folio's front half is dirty in the "normal" sense, it needs to be submitted to the OE waiting for it. It's a cancelled fixup. Meanwhile, the second half is a true fresh fixup. So at this point if we run writeback on this folio, we genuinely can't know what to do without block level information. If we submit it, we submit unreserved dirty from the back half. If we don't, we will never finish the OE waiting for it. So it's either a corruption or a deadlock. Thus, the full high level design picture: - btrfs_data_dirty_folio(): For out of band non-reserving dirties, mark still-clean blocks inside EOF dirty and set their fixup bits (the event carries no range, so every clean block is suspect). Already-dirty blocks are covered or pending and are left alone. - Writeback: skip fixup blocks and enqueue work for them - writepage_fixup(): for each fixup block do the fixup reservation in a worker, after which the blocks can be written back normally. - Typical reserving write paths cancel fixup state for the ranges they cover with btrfs_folio_cancel_fixup() Link: https://lore.kernel.org/linux-btrfs/20260721191152.101118-1-borntraeger@linux.ibm.com/ Assisted-by: LLM Reviewed-by: Qu Wenruo Signed-off-by: Boris Burkov Signed-off-by: David Sterba

btrfs: fix u32 to s64 type conversion in dirty_metadata_bytes accounting

2026-07-14T05:04:19+00:00

The percpu_counter dirty_metadata_bytes is updated by negating eb->len and passing it to percpu_counter_add_batch(), whose amount parameter is s64. Since commit 84cda1a6087d ("btrfs: cache folio size and shift in extent_buffer"), eb->len is u32. The u32 result of -eb->len, when widened to the s64 parameter, becomes a large positive value instead of the intended negative value. For eb->len == 16384 the counter adds +4294950912 instead of subtracting 16384. The counter therefore grows on every metadata writeback instead of shrinking by the extent buffer size, permanently exceeding BTRFS_DIRTY_METADATA_THRESH and causing __btrfs_btree_balance_dirty() to trigger balance_dirty_pages_ratelimited() unconditionally, adding unnecessary writeback pressure. Cast eb->len to s64 before negation at both call sites so the subtraction is performed in signed 64-bit arithmetic. Reviewed-by: Filipe Manana Fixes: 84cda1a6087d ("btrfs: cache folio size and shift in extent_buffer") Signed-off-by: Dave Chen Signed-off-by: Filipe Manana Signed-off-by: David Sterba

btrfs: switch local indicator variables to bools

2026-06-09T16:22:46+00:00

For all local indicator variables do simple switch to bool, done on all files. Signed-off-by: David Sterba

btrfs: allocate eb-attached btree pages as movable

2026-06-09T16:22:45+00:00

Extent buffer pages allocated by alloc_extent_buffer() are attached to btree_inode->i_mapping (the buffer_tree path), reach the LRU, and are served by the btree_migrate_folio aops in fs/btrfs/disk-io.c. They are migratable in practice once their owning extent buffer hits refs == 1, which happens naturally. The buddy allocator classifies them by GFP, however, and bare GFP_NOFS lands them in MIGRATE_UNMOVABLE pageblocks. The result: every btree_inode page we read in pins an unmovable pageblock from the page-superblock allocator's perspective, even though the page itself can be moved. Have each caller of btrfs_alloc_page_array, btrfs_alloc_folio_array, and alloc_eb_folio_array pass in the full GFP mask directly, instead of having the functions calculate it from boolean flags. The alloc_extent_buffer call site passes GFP_NOFS | __GFP_NOFAIL | __GFP_MOVABLE. All other call sites pass plain GFP_NOFS. Three categories of caller stay on bare GFP_NOFS, deliberately: - alloc_dummy_extent_buffer / btrfs_clone_extent_buffer: the resulting eb is EXTENT_BUFFER_UNMAPPED, folio->mapping stays NULL, the folios never enter LRU, never get migrate_folio aops. Tagging them __GFP_MOVABLE would violate the page allocator's migrability contract and they would defeat compaction in MOVABLE pageblocks where isolate_migratepages_block skips non-LRU non-movable_ops pages outright. - btrfs_alloc_page_array callers in fs/btrfs/raid56.c (stripe pages), fs/btrfs/inode.c (encoded reads), fs/btrfs/ioctl.c (io_uring encoded reads), fs/btrfs/relocation.c (relocation buffers): same contract violation. raid56 stripe_pages additionally persist in the stripe cache (RBIO_CACHE_SIZE=1024) well beyond a single I/O, so they are not transient enough to hand-wave the contract. - btrfs_alloc_folio_array caller in fs/btrfs/scrub.c (stripe folios): same -- stripe->folios[] are private buffers freed via folio_put in release_scrub_stripe. This change targets the dominant fragmentation source observed on the page-superblock series: ~28 GB of btree_inode pages parked across many tainted superpageblocks on a 250 GB test system with btrfs root, preventing 1 GiB hugepage allocation from those regions. With the movable hint, those pages now land in MOVABLE pageblocks where the existing background defragger drains them through the standard PB_has_movable gate, no LRU-sample fallback needed. Assisted-by: Claude:claude-opus-4-6 Signed-off-by: Rik van Riel Signed-off-by: David Sterba

btrfs: replace __free_page with folio_put() in attach_eb_folio_to_filemap()

2026-06-09T16:22:44+00:00

Calling __free_page() on folio_page() happens to work today, but won't always. Besides, it's far simpler to call folio_put(). Reviewed-by: Boris Burkov Reviewed-by: Qu Wenruo Tested-by: Boris Burkov Signed-off-by: Matthew Wilcox (Oracle) Signed-off-by: David Sterba

btrfs: introduce support for huge folios

2026-06-09T10:49:26+00:00

With all the previous preparations, it's finally time to enable the huge folio support. - The max folio size Here we define BTRFS_MAX_FOLIO_SIZE, which is fixed at 2MiB. This will ensure we have a large enough but not too large folio for btrfs. This limit applies to all systems regardless of page size. Then we also define BTRFS_MAX_BLOCKS_PER_FOLIO, which depends on CONFIG_BTRFS_EXPERIMENTAL. If it's an experimental build, BTRFS_MAX_BLOCKS_PER_FOLIO is 512, otherwise it's BITS_PER_LONG. The filemap max order will be calculated using both BTRFS_MAX_FOLIO_SIZE and BTRFS_MAX_BLOCKS_PER_FOLIO. E.g. for 64K page size with 64K fs block size, the limit will be BTRFS_MAX_FOLIO_SIZE (2M), which limits the filemap max order to 5. This will be lower than the old order (6), but folios larger than 2M are rarely any better for IO performance. Meanwhile excessively large folios can cause other problems like stalling the IO pipeline for too long. For 4K page size and 4K fs block size, the limit will be increased to 2M from the old 256K. This new size is constrained by both BTRFS_MAX_FOLIO_SIZE (2M) and BTRFS_MAX_BLOCKS_PER_FOLIO (512 * 4K), allowing x86_64 to achieve huge folio support, and the filemap max order will be 9. - btrfs_bio_ctrl::submit_bitmap This will be enlarged to contain BTRFS_MAX_BLOCKS_PER_FOLIO bits, and this will be on-stack memory. This will increase on-stack memory usage by 56 bytes compared to the baseline (before the first patch in the series). - Local @delalloc_bitmap inside writepage_delalloc() Unfortunately we cannot afford to handle an allocation error here, thus again we use on-stack memory. Thus this will increase on-stack memory usage by 56 bytes again. So unfortunately this means during the delalloc window, the writeback path will have +112 bytes on-stack memory usage, and for other cases the writeback path will have +56 bytes on-stack memory usage. The +56 bytes (btrfs_bio_ctrl::submit_bitmap) can be removed after we have reworked the compression submission, so the current on-stack submit_bitmap is mostly a workaround until then. Signed-off-by: Qu Wenruo Signed-off-by: David Sterba

btrfs: migrate btrfs_bio_ctrl::submit_bitmap to support larger bitmaps

2026-06-09T10:49:26+00:00

[CURRENT LIMIT] Btrfs currently only supports sub-bitmaps (e.g. dirty bitmap) no larger than BITS_PER_LONG. One call site that utilizes this limit is btrfs_bio_ctrl::submit_bitmap, which makes it very simple and straightforward to just grab an unsigned long value and assign it to submit_bitmap. Unfortunately that limit prevents us from supporting huge folios. For 4K page size and block size, a huge folio (order 9) means 512 blocks inside a 2M folio. [ENHANCEMENT] Instead of using a fixed unsigned long value, change btrfs_bio_ctrl::submit_bitmap to an unsigned long pointer. And for cases where an unsigned long can hold the whole bitmap, introduce @submit_bitmap_value, and just point that pointer to that unsigned long. Then update all direct users of bio_ctrl->submit_bitmap to use the pointer version. There are several call sites that get extra changes: - @range_bitmap inside extent_writepage_io() Which is only utilized to truncate the bitmap. Since we do not want to allocate new memory just for such temporary usage, change the original bitmap_set() and bitmap_and() into bitmap_clear() for the ranges outside of the target range. - Getting dirty subpage bitmap inside writepage_delalloc() Since we're passing an unsigned long pointer now, we need to go with different handling (bs == ps, blocks_per_folio <= BITS_PER_LONG, blocks_per_folio > BITS_PER_LONG). Signed-off-by: Qu Wenruo Signed-off-by: David Sterba

btrfs: prepare subpage operations to support more than BITS_PER_LONG sub-bitmaps

2026-06-09T10:49:26+00:00

[CURRENT LIMIT] Btrfs currently only supports sub-bitmaps (e.g. dirty bitmap) no larger than BITS_PER_LONG. That limit allows us to easily grab an unsigned long without the need to properly allocate memory for a larger bitmap. Unfortunately that limit prevents us from supporting huge folios. For 4K page size and block size, a huge folio (order 9) means 512 blocks inside a 2M folio. [ENHANCEMENT] To allow direct bitmap operations without allocating new memory, introduce two different ways to access the subpage bitmaps: - Return an unsigned long value This only happens if blocks_per_folio <= BITS_PER_LONG. We read out the sub-bitmap into an unsigned long, and return the value. This is the old existing method. This involves get_bitmap_value_##name() helper functions. And this time the helper functions are defined as inline functions instead of macros to provide better type checks. - Return a pointer where the sub-bitmap starts This only happens if blocks_per_folio >= BITS_PER_LONG. This is the new method for sub-bitmaps larger than BITS_PER_LONG. Since the sizes of sub-bitmaps are all aligned to BITS_PER_LONG, we can directly access the start word of the sub-bitmap. This involves get_bitmap_pointer_##name() helper functions. Then change the existing sub-bitmaps users to use the new helpers: - Bitmap dumping Switch between get_bitmap_value_##name() and get_bitmap_pointer_##name() depending on the sub-bitmap size. - btrfs_get_subpage_dirty_bitmap() Rename it to btrfs_get_subpage_dirty_bitmap_value() to follow the new value/pointer naming. Since we do not support huge folios yet, there is no pointer version for the dirty bitmap. Furthermore, add the support for block size == page size cases for btrfs_get_subpage_dirty_bitmap_value(), so that the caller no longer needs to check if the folio needs subpage handling. Signed-off-by: Qu Wenruo Signed-off-by: David Sterba

btrfs: remove folio ordered flag and subpage bitmap

2026-06-08T13:53:32+00:00

Btrfs has an internal flag/subpage bitmap called ordered, which is to indicate that a block has corresponding ordered extent covering it. However this requires extra synchronization between the inode ordered tree, and the folio flag/subpage bitmap, not to mention we need to maintain the extra folio flag with subpage bitmap. As a step to align btrfs_folio_state more closely to iomap_folio_state, remove the btrfs specific ordered flag/bitmap. This will also save us 64 bytes for the bitmap of a huge folio. Since we're here, also update the ASCII graph of the bitmap, as there are only 3 sub-bitmaps now, show all sub-bitmaps directly. Signed-off-by: Qu Wenruo Signed-off-by: David Sterba

btrfs: remove folio_test_ordered() usage

2026-06-08T13:53:32+00:00

This involves: - The ASSERT() inside end_bbio_data_write() It's only an ASSERT() and it has never been triggered as far as I know. - btrfs_migrate_folio() Since all folio_test_ordered() usage will be removed, there is no need to copy the folio ordered flag. - The ASSERT() inside btrfs_invalidate_folio() This one has its usefulness as it indeed caught some bugs during development. But that's the last user and will not be worth the folio flag or the subpage bitmap. This will allow btrfs to finally remove the ordered flags. Signed-off-by: Qu Wenruo Signed-off-by: David Sterba