kernel/git/stable/linux.git/fs/btrfs/subpage.h, branch master

btrfs: trigger cow fixup via dirty_folio()

2026-07-30T17:28:36+00:00

The problem scenario: If we have a folio mmapped shared and then somebody does a dio read with that folio as the read destination, then it is possible that the dio will see a dirty destination page when it starts (and thus skip dirtying and just GUP pin it) but then while it is doing the read, btrfs finishes writing it back and by the endio, the folio is clean. In that case, the dio read must re-dirty the folio with aops->dirty_folio(): btrfs_check_read_bio() |- __iomap_dio_bio_end_io() from btrfs_bio_end_io() |- bio_check_pages_dirty() |- bio_dirty_fn() |- bio_release_pages(bio, true) |- __bio_release_pages(bio, mark_dirty == true) |- folio_lock() |- folio_mark_dirty() |- aops->dirty_folio() |- folio_unlock() A data block normally moves through writeback as follows: TASK folio_lock write clean -> dirty bit + delalloc folio_unlock WRITEBACK for-each-dirty-folio: folio_lock run_delalloc delalloc consumed -> dirty bit + OE submission dirty bit consumed -> writeback bit + OE folio_unlock ENDIO endio OE bytes accounted OE finish writeback -> clean; destroy OE Three critical invariants that this path maintains are: I1. Any dirty block is covered by delalloc xor an ordered extent I2. Any dirty block covered by an OE will be submitted into that OE I3. Any dirty block already submitted into an OE will not be submitted again into the same OE. These ensure that the block will be written exactly once. It is clear that not reserving delalloc for the re-dirty case violates I1. This situation, even without bs < folio_size, has long required btrfs to fixup such dirty pages during writeback with an asynchronous worker that is allowed to do this expensive work and writeback does not proceed for a folio while it is doing this work. Commit 247e743cbe6e ("Btrfs: Use async helpers to deal with pages that have been improperly dirtied") introduced the COW fixup to catch exactly this class at writeback, way back in 2008. Since then, there have been many advances to prevent most of the causes of such re-dirtying and we thought we could get away with removing the annoying cow-fixup in the hope of simplifying writeback for large folio support. Commit b2a9f217ad3f ("btrfs: remove the COW fixup mechanism") Commit 4927b141877c ("btrfs: remove folio ordered flag and subpage bitmap") Since it turns out this assumption was incorrect, as evidenced by the report and attendant reproducers, we must reintroduce the fixup concept. This is of course critically further complicated by bs < folio_size. In that case, rather than just a folio dirty bit, we have a bitmap for the dirty blocks in the folio. And the (also broken) invariant is: I4. folio dirty IFF at least one block bitmap dirty. The original report of a stall on a misinterpreted empty bitmap is exactly evidence of a violation of I4. It is exactly because of bs < folio_size we don't want to simply revert the removal patches. The original fixup was not properly bs < folio_size aware, which motivated removal in the first place. So we wish to build a bs < folio_size aware fixup. One other important detail from the old design, any normal write that happens after a re-dirty but before a fixup is racing with the cow fixup to do the delalloc reservation, therefore it must cancel the fixup state. If it arrives after the reservation exists, it will be a normal dirty overwrite. This critically informs the design in a pretty clear way. fixup requiring re-dirty has folio granularity, while cancellation has delalloc (block) granularity so while we only ever produce fixup in chunks of folios, we must be able to clear it in blocks. Therefore we must track the blocks needing fixup at block granularity. The obvious way to do this is with a new bitmap in btrfs_folio_state, but it is desirable to avoid that if possible. Unfortunately, I don't think it is possible and the reason is subtle and leans on a sort of extreme reproducer, but I think can be explained relatively succinctly. Consider a folio whose two halves will land in different ordered extents (can be accomplished with tricks using nodatasum) and a dio read is running with it as the shared mmap destination. 1. The front half: a. folio comes clean on a normal write b. dio read completes into the folio marking it fixup. c. a write comes for the previous folio for a range extending into this folio, this is a cancellation of the fixup which reserves space. d. writeback runs on the range *not* overlapping the folio. This half remains dirty but is now covered by an OE and is awaiting writeback running on its range to be submitted and finish the OE. 2. The back half: a. the folio is part of an OE that gets far enough along to clear writeback. b. dio read completes into the folio marking it fixup. After this, the folio's front half is dirty in the "normal" sense, it needs to be submitted to the OE waiting for it. It's a cancelled fixup. Meanwhile, the second half is a true fresh fixup. So at this point if we run writeback on this folio, we genuinely can't know what to do without block level information. If we submit it, we submit unreserved dirty from the back half. If we don't, we will never finish the OE waiting for it. So it's either a corruption or a deadlock. Thus, the full high level design picture: - btrfs_data_dirty_folio(): For out of band non-reserving dirties, mark still-clean blocks inside EOF dirty and set their fixup bits (the event carries no range, so every clean block is suspect). Already-dirty blocks are covered or pending and are left alone. - Writeback: skip fixup blocks and enqueue work for them - writepage_fixup(): for each fixup block do the fixup reservation in a worker, after which the blocks can be written back normally. - Typical reserving write paths cancel fixup state for the ranges they cover with btrfs_folio_cancel_fixup() Link: https://lore.kernel.org/linux-btrfs/20260721191152.101118-1-borntraeger@linux.ibm.com/ Assisted-by: LLM Reviewed-by: Qu Wenruo Signed-off-by: Boris Burkov Signed-off-by: David Sterba

btrfs: migrate btrfs_bio_ctrl::submit_bitmap to support larger bitmaps

2026-06-09T10:49:26+00:00

[CURRENT LIMIT] Btrfs currently only supports sub-bitmaps (e.g. dirty bitmap) no larger than BITS_PER_LONG. One call site that utilizes this limit is btrfs_bio_ctrl::submit_bitmap, which makes it very simple and straightforward to just grab an unsigned long value and assign it to submit_bitmap. Unfortunately that limit prevents us from supporting huge folios. For 4K page size and block size, a huge folio (order 9) means 512 blocks inside a 2M folio. [ENHANCEMENT] Instead of using a fixed unsigned long value, change btrfs_bio_ctrl::submit_bitmap to an unsigned long pointer. And for cases where an unsigned long can hold the whole bitmap, introduce @submit_bitmap_value, and just point that pointer to that unsigned long. Then update all direct users of bio_ctrl->submit_bitmap to use the pointer version. There are several call sites that get extra changes: - @range_bitmap inside extent_writepage_io() Which is only utilized to truncate the bitmap. Since we do not want to allocate new memory just for such temporary usage, change the original bitmap_set() and bitmap_and() into bitmap_clear() for the ranges outside of the target range. - Getting dirty subpage bitmap inside writepage_delalloc() Since we're passing an unsigned long pointer now, we need to go with different handling (bs == ps, blocks_per_folio <= BITS_PER_LONG, blocks_per_folio > BITS_PER_LONG). Signed-off-by: Qu Wenruo Signed-off-by: David Sterba

btrfs: prepare subpage operations to support more than BITS_PER_LONG sub-bitmaps

2026-06-09T10:49:26+00:00

[CURRENT LIMIT] Btrfs currently only supports sub-bitmaps (e.g. dirty bitmap) no larger than BITS_PER_LONG. That limit allows us to easily grab an unsigned long without the need to properly allocate memory for a larger bitmap. Unfortunately that limit prevents us from supporting huge folios. For 4K page size and block size, a huge folio (order 9) means 512 blocks inside a 2M folio. [ENHANCEMENT] To allow direct bitmap operations without allocating new memory, introduce two different ways to access the subpage bitmaps: - Return an unsigned long value This only happens if blocks_per_folio <= BITS_PER_LONG. We read out the sub-bitmap into an unsigned long, and return the value. This is the old existing method. This involves get_bitmap_value_##name() helper functions. And this time the helper functions are defined as inline functions instead of macros to provide better type checks. - Return a pointer where the sub-bitmap starts This only happens if blocks_per_folio >= BITS_PER_LONG. This is the new method for sub-bitmaps larger than BITS_PER_LONG. Since the sizes of sub-bitmaps are all aligned to BITS_PER_LONG, we can directly access the start word of the sub-bitmap. This involves get_bitmap_pointer_##name() helper functions. Then change the existing sub-bitmaps users to use the new helpers: - Bitmap dumping Switch between get_bitmap_value_##name() and get_bitmap_pointer_##name() depending on the sub-bitmap size. - btrfs_get_subpage_dirty_bitmap() Rename it to btrfs_get_subpage_dirty_bitmap_value() to follow the new value/pointer naming. Since we do not support huge folios yet, there is no pointer version for the dirty bitmap. Furthermore, add the support for block size == page size cases for btrfs_get_subpage_dirty_bitmap_value(), so that the caller no longer needs to check if the folio needs subpage handling. Signed-off-by: Qu Wenruo Signed-off-by: David Sterba

btrfs: remove folio ordered flag and subpage bitmap

2026-06-08T13:53:32+00:00

Btrfs has an internal flag/subpage bitmap called ordered, which is to indicate that a block has corresponding ordered extent covering it. However this requires extra synchronization between the inode ordered tree, and the folio flag/subpage bitmap, not to mention we need to maintain the extra folio flag with subpage bitmap. As a step to align btrfs_folio_state more closely to iomap_folio_state, remove the btrfs specific ordered flag/bitmap. This will also save us 64 bytes for the bitmap of a huge folio. Since we're here, also update the ASCII graph of the bitmap, as there are only 3 sub-bitmaps now, show all sub-bitmaps directly. Signed-off-by: Qu Wenruo Signed-off-by: David Sterba

btrfs: remove locked subpage bitmap

2026-06-08T13:53:32+00:00

Currently there are two members inside btrfs_folio_state that are related to locked bitmap: - locked sub-bitmap inside btrfs_folio_state::bitmaps[] The enum btrfs_bitmap_nr_locked determines the sub-bitmap. - btrfs_folio_state::nr_locked Which records how many blocks are locked inside the folio. The locked sub-bitmap is a btrfs specific per-block tracking mechanism, which is mostly for async-submission, utilized by compressed writes. The sub-bitmap itself is a super set of nr_locked, as it can provide a more reliable tracking. But the sub-bitmap itself can be pretty large for the incoming huge folio, 2M sized folio for 4K page size, meaning 512 bits for one sub-bitmap. Furthermore, in the long run compression will be reworked to get rid of async-submission completely, there is not much need for a full sub-bitmap to track the locked status. This patch removes the locked sub-bitmap and only relies on @nr_locked atomic to do the tracking. This can also save 64 bytes from btrfs_folio_state::bitmaps[] for a huge folio. This will reduce some safety checks, as previously if a block is not locked, btrfs_folio_end_lock()/btrfs_folio_end_lock_bitmap() will find out that, and skip reducing @nr_locked for that block, and avoid under-flow. But this safety net itself shouldn't be necessary in the first place. If we're unlocking a block that is not locked, it's a bug in the logic, and we should catch it, not silently ignoring it. Thus I believe the removal of the extra safety net should not be a problem. Reviewed-by: David Sterba Signed-off-by: Qu Wenruo Signed-off-by: David Sterba

btrfs: remove folio checked subpage bitmap tracking

2026-06-08T13:53:28+00:00

The folio checked flag is only utilized by the COW fixup mechanism inside btrfs. Since the COW fixup is already removed from non-experimental builds, there is no need to keep the checked subpage bitmap. This will saves us some space for large folios, for example for a single 256K sized large folio on 4K page sized systems: Old bitmap size = 6 * (256K / 4K / 8) = 48 bytes New bitmap size = 5 * (256K / 4K / 8) = 40 bytes This will be more obvious when we're going to support huge folios (order = 9). Signed-off-by: Qu Wenruo Reviewed-by: David Sterba Signed-off-by: David Sterba

btrfs: headers cleanup to remove unnecessary local includes

2025-11-24T21:34:52+00:00

[BUG] When I tried to remove btrfs_bio::fs_info and use btrfs_bio::inode to grab the fs_info, the header "btrfs_inode.h" is needed to access the full btrfs_inode structure. Then btrfs will fail to compile. [CAUSE] There is a recursive including chain: "bio.h" -> "btrfs_inode.h" -> "extent_map.h" -> "compression.h" -> "bio.h" That recursive including is causing problems for btrfs. [ENHANCEMENT] To reduce the risk of recursive including: - Remove unnecessary local includes from btrfs headers Either the included header is pulled in by other headers, or is completely unnecessary. - Remove btrfs local includes if the header only requires a pointer In that case let the implementing C file to pull the required header. This is especially important for headers like "btrfs_inode.h" which pulls in a lot of other btrfs headers, thus it's a mine field of recursive including. - Remove unnecessary temporary structure definition Either if we have included the header defining the structure, or completely unused. Now including "btrfs_inode.h" inside "bio.h" is completely fine, although "btrfs_inode.h" still includes "extent_map.h", but that header only includes "fs.h", no more chain back to "bio.h". Signed-off-by: Qu Wenruo Reviewed-by: David Sterba Signed-off-by: David Sterba

btrfs: fix typos in comments and strings

2025-09-23T06:49:16+00:00

Annual typo fixing pass. Strangely codespell found only about 30% of what is in this patch, the rest was done manually using text spellchecker with a custom dictionary of acceptable terms. Reviewed-by: Neal Gompa Signed-off-by: David Sterba

btrfs: enable large data folio support under CONFIG_BTRFS_EXPERIMENTAL

2025-07-21T21:53:30+00:00

With all the preparation patches already merged, it's pretty easy to enable large data folios: - Remove the ASSERT() on folio size in btrfs_end_repair_bio() - Add a helper to properly set the max folio order Currently due to several call sites that are fetching the bitmap content directly into an unsigned long, we can only support BITS_PER_LONG blocks for each bitmap. - Call the helper when reading/creating an inode The support has the following limitations: - No large folios for data reloc inode The relocation code still requires page sized folio. But it's not that hot nor common compared to regular buffered ios. Will be improved in the future. - Requires CONFIG_BTRFS_EXPERIMENTAL - Will require all folio related operations to check if it needs the extra btrfs_subpage structure Now any folio larger than block size will need btrfs_subpage structure handling. Unfortunately I do not have a physical machine for performance test, but if everything goes like XFS/EXT4, it should mostly bring single digits percentage performance improvement in the real world. Although I believe there are still quite some optimizations to be done, let's focus on testing the current large data folio support first. Signed-off-by: Qu Wenruo Reviewed-by: David Sterba Signed-off-by: David Sterba

btrfs: rename btrfs_subpage structure

2025-07-21T21:53:27+00:00

With the incoming large data folios support, the structure name btrfs_subpage is no longer correct, as for we can have multiple blocks inside a large folio, and the block size is still page size. So to follow the schema of iomap, rename btrfs_subpage to btrfs_folio_state, along with involved enums. There are still exported functions with "btrfs_subpage_" prefix, and I believe for metadata the name "subpage" will stay forever as we will never allocate a folio larger than nodesize anyway. The full cleanup of the word "subpage" will happen in much smaller steps in the future. Signed-off-by: Qu Wenruo Reviewed-by: David Sterba Signed-off-by: David Sterba