| Age | Commit message (Collapse) | Author |
|---|
|
The problem scenario:
If we have a folio mmapped shared and then somebody does a dio read with
that folio as the read destination, then it is possible that the dio
will see a dirty destination page when it starts (and thus skip
dirtying and just GUP pin it) but then while it is doing the read, btrfs
finishes writing it back and by the endio, the folio is clean. In that
case, the dio read must re-dirty the folio with aops->dirty_folio():
btrfs_check_read_bio()
|- __iomap_dio_bio_end_io() from btrfs_bio_end_io()
|- bio_check_pages_dirty()
|- bio_dirty_fn()
|- bio_release_pages(bio, true)
|- __bio_release_pages(bio, mark_dirty == true)
|- folio_lock()
|- folio_mark_dirty()
|- aops->dirty_folio()
|- folio_unlock()
A data block normally moves through writeback as follows:
TASK
folio_lock
write clean -> dirty bit + delalloc
folio_unlock
WRITEBACK
for-each-dirty-folio:
folio_lock
run_delalloc delalloc consumed -> dirty bit + OE
submission dirty bit consumed -> writeback bit + OE
folio_unlock
ENDIO
endio OE bytes accounted
OE finish writeback -> clean; destroy OE
Three critical invariants that this path maintains are:
I1. Any dirty block is covered by delalloc xor an ordered extent
I2. Any dirty block covered by an OE will be submitted into that OE
I3. Any dirty block already submitted into an OE will not be submitted
again into the same OE.
These ensure that the block will be written exactly once. It is clear
that not reserving delalloc for the re-dirty case violates I1.
This situation, even without bs < folio_size, has long required btrfs to
fixup such dirty pages during writeback with an asynchronous worker that
is allowed to do this expensive work and writeback does not proceed for
a folio while it is doing this work.
Commit 247e743cbe6e ("Btrfs: Use async helpers to deal with pages that
have been improperly dirtied") introduced the COW fixup to catch exactly
this class at writeback, way back in 2008.
Since then, there have been many advances to prevent most of the causes
of such re-dirtying and we thought we could get away with removing the
annoying cow-fixup in the hope of simplifying writeback for large folio
support.
Commit b2a9f217ad3f ("btrfs: remove the COW fixup mechanism")
Commit 4927b141877c ("btrfs: remove folio ordered flag and subpage bitmap")
Since it turns out this assumption was incorrect, as evidenced by the
report and attendant reproducers, we must reintroduce the fixup concept.
This is of course critically further complicated by bs < folio_size. In
that case, rather than just a folio dirty bit, we have a bitmap for the
dirty blocks in the folio. And the (also broken) invariant is:
I4. folio dirty IFF at least one block bitmap dirty.
The original report of a stall on a misinterpreted empty bitmap is
exactly evidence of a violation of I4.
It is exactly because of bs < folio_size we don't want to simply revert the
removal patches. The original fixup was not properly bs < folio_size
aware, which motivated removal in the first place. So we wish to build a
bs < folio_size aware fixup.
One other important detail from the old design, any normal write that
happens after a re-dirty but before a fixup is racing with the cow fixup
to do the delalloc reservation, therefore it must cancel the fixup state.
If it arrives after the reservation exists, it will be a normal dirty
overwrite. This critically informs the design in a pretty clear way.
fixup requiring re-dirty has folio granularity, while cancellation has
delalloc (block) granularity so while we only ever produce fixup in
chunks of folios, we must be able to clear it in blocks. Therefore we
must track the blocks needing fixup at block granularity.
The obvious way to do this is with a new bitmap in btrfs_folio_state,
but it is desirable to avoid that if possible. Unfortunately, I don't
think it is possible and the reason is subtle and leans on a sort of
extreme reproducer, but I think can be explained relatively succinctly.
Consider a folio whose two halves will land in different ordered extents
(can be accomplished with tricks using nodatasum) and a dio read is
running with it as the shared mmap destination.
1. The front half:
a. folio comes clean on a normal write
b. dio read completes into the folio marking it fixup.
c. a write comes for the previous folio for a range extending into
this folio, this is a cancellation of the fixup which reserves
space.
d. writeback runs on the range *not* overlapping the folio. This half
remains dirty but is now covered by an OE and is awaiting
writeback running on its range to be submitted and finish the OE.
2. The back half:
a. the folio is part of an OE that gets far enough along to clear
writeback.
b. dio read completes into the folio marking it fixup.
After this, the folio's front half is dirty in the "normal" sense, it
needs to be submitted to the OE waiting for it. It's a cancelled fixup.
Meanwhile, the second half is a true fresh fixup. So at this point if we
run writeback on this folio, we genuinely can't know what to do without
block level information. If we submit it, we submit unreserved dirty
from the back half. If we don't, we will never finish the OE waiting for
it. So it's either a corruption or a deadlock.
Thus, the full high level design picture:
- btrfs_data_dirty_folio(): For out of band non-reserving dirties,
mark still-clean blocks inside EOF dirty and set their fixup bits
(the event carries no range, so every clean block is suspect).
Already-dirty blocks are covered or pending and are left alone.
- Writeback: skip fixup blocks and enqueue work for them
- writepage_fixup(): for each fixup block do the fixup reservation in a
worker, after which the blocks can be written back normally.
- Typical reserving write paths cancel fixup state for the ranges they
cover with btrfs_folio_cancel_fixup()
Link: https://lore.kernel.org/linux-btrfs/20260721191152.101118-1-borntraeger@linux.ibm.com/
Assisted-by: LLM
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Boris Burkov <boris@bur.io>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
[BUG]
Mounting with rescue=ibadroots after corrupting the block group tree
root triggers a NULL pointer dereference:
BUG: kernel NULL pointer dereference, address: 0000000000000100
RIP: 0010:btrfs_update_global_block_rsv+0x9d/0x1c0 [btrfs]
Call Trace:
fill_dummy_bgs+0xd4/0x120 [btrfs]
open_ctree+0xc6e/0x1ca0 [btrfs]
btrfs_get_tree+0x50d/0xa40 [btrfs]
The same crash occurs with a corrupted raid stripe tree root, via
btrfs_read_block_groups() instead of fill_dummy_bgs().
[CAUSE]
With rescue=ibadroots, btrfs_read_roots() allows the mount to continue
when either root cannot be read, leaving the corresponding root pointer
NULL while its on-disk feature bit remains set.
btrfs_update_global_block_rsv() then dereferences the missing root based
on the feature bit alone.
[FIX]
Rescue mounts are fully read-only and cannot start transactions, so the
global reserve is never consumed. Under btrfs_is_full_ro(), mark the
reserve as full and return before performing the accounting.
And since we need to check if the fs is mount fully RO, export
fs_is_full_ro() as btrfs_is_full_ro(), and move it to fs.h.
Fixes: 8dbfc14fc736 ("btrfs: account block group tree when calculating global reserve size")
Fixes: 515020900d44 ("btrfs: read raid stripe tree from disk")
Suggested-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Dongjiang Zhu <zhudongjiang@fnnas.com>
[ Squash the fs_is_full_ro() export commit into this one. ]
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
If we get a failure during relocation, before we update all the extent
buffers that have file extent items pointing to extents from the block
group being relocated, we can trigger a user-after-free on the reloc
control structure (fs_info->reloc_control) if we have a concurrent task
that is COWing a subvolume leaf.
This happens like this:
1) Relocation of data block group X starts;
2) Relocation changes its state to UPDATE_DATA_PTRS;
3) A task doing a rename for example, COWs leaf A from a subvolume tree
and ends up at btrfs_reloc_cow_block() and extracts fs_info->reloc_ctl
into a local variable, which then passes to replace_file_extents();
4) The relocation task gets an error and under the label 'out_put_bg' in
btrfs_relocate_block_group() calls free_reloc_control(), which frees
the reloc control structure that the rename task is using;
5) The rename task triggers a use-after-free on the reloc control
structure that was just freed.
Syzbot reported this recently, with the following stack trace:
[ 88.389822][ T5325] BTRFS error (device loop0 state A): Transaction aborted (error -5)
[ 88.389842][ T5325] BTRFS: error (device loop0 state A) in cleanup_transaction:2067: errno=-5 IO failure
[ 88.389864][ T5325] BTRFS info (device loop0 state EA): forced readonly
[ 88.392277][ T5324] BTRFS: error (device loop0 state EA) in btrfs_sync_log:3572: errno=-5 IO failure
[ 88.396630][ T5325] BTRFS info (device loop0 state EA): balance: ended with status: -5
[ 88.400135][ T5346] ==================================================================
[ 88.400148][ T5346] BUG: KASAN: slab-use-after-free in replace_file_extents+0x85f/0x1590
[ 88.400288][ T5346] Read of size 8 at addr ffff888012312010 by task syz.0.0/5346
[ 88.400299][ T5346]
[ 88.400306][ T5346] CPU: 0 UID: 0 PID: 5346 Comm: syz.0.0 Not tainted syzkaller #0 PREEMPT(full)
[ 88.400319][ T5346] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 88.400325][ T5346] Call Trace:
[ 88.400331][ T5346] <TASK>
[ 88.400336][ T5346] dump_stack_lvl+0xe8/0x150
[ 88.400351][ T5346] print_address_description+0x55/0x1e0
[ 88.400364][ T5346] ? replace_file_extents+0x85f/0x1590
[ 88.400378][ T5346] print_report+0x58/0x70
[ 88.400389][ T5346] kasan_report+0x117/0x150
[ 88.400405][ T5346] ? replace_file_extents+0x85f/0x1590
[ 88.400420][ T5346] replace_file_extents+0x85f/0x1590
[ 88.400440][ T5346] ? __pfx_replace_file_extents+0x10/0x10
[ 88.400452][ T5346] ? update_ref_for_cow+0xa71/0x1270
[ 88.400473][ T5346] btrfs_force_cow_block+0xa4d/0x2450
[ 88.400492][ T5346] ? __pfx_btrfs_force_cow_block+0x10/0x10
[ 88.400508][ T5346] ? __pfx_btrfs_get_32+0x10/0x10
[ 88.400523][ T5346] btrfs_cow_block+0x3c4/0xa90
[ 88.400542][ T5346] push_leaf_left+0x2ac/0x4a0
[ 88.400561][ T5346] split_leaf+0xd16/0x12e0
[ 88.400574][ T5346] ? btrfs_bin_search+0x924/0xc70
[ 88.400592][ T5346] ? __pfx_split_leaf+0x10/0x10
[ 88.400602][ T5346] ? leaf_space_used+0x177/0x1e0
[ 88.400618][ T5346] ? btrfs_leaf_free_space+0x14a/0x2f0
[ 88.400634][ T5346] btrfs_search_slot+0x2641/0x2d20
[ 88.400654][ T5346] ? __pfx_btrfs_search_slot+0x10/0x10
[ 88.400669][ T5346] ? rcu_is_watching+0x15/0xb0
[ 88.400681][ T5346] ? trace_kmem_cache_alloc+0x29/0xe0
[ 88.400694][ T5346] btrfs_insert_empty_items+0x9c/0x190
[ 88.400711][ T5346] btrfs_insert_inode_ref+0x229/0xcb0
[ 88.400724][ T5346] ? __pfx_btrfs_insert_inode_ref+0x10/0x10
[ 88.400736][ T5346] ? __pfx_btrfs_qgroup_convert_reserved_meta+0x10/0x10
[ 88.400751][ T5346] ? btrfs_record_root_in_trans+0x124/0x180
[ 88.400767][ T5346] ? start_transaction+0x8a0/0x1820
[ 88.400778][ T5346] ? btrfs_set_inode_index+0x5e/0x100
[ 88.400787][ T5346] btrfs_rename2+0x17bb/0x40d0
[ 88.400800][ T5346] ? check_noncircular+0xda/0x150
[ 88.400814][ T5346] ? add_lock_to_list+0xc7/0x100
[ 88.400828][ T5346] ? __pfx_btrfs_rename2+0x10/0x10
[ 88.400842][ T5346] ? lockdep_hardirqs_on+0x7a/0x110
[ 88.400901][ T5346] ? lock_acquire+0x221/0x350
[ 88.400915][ T5346] ? down_write_nested+0x174/0x210
[ 88.400931][ T5346] ? __pfx_down_write_nested+0x10/0x10
[ 88.400941][ T5346] ? do_raw_spin_unlock+0x4d/0x210
[ 88.400952][ T5346] ? try_break_deleg+0x5b/0x180
[ 88.400963][ T5346] ? __pfx_btrfs_rename2+0x10/0x10
[ 88.400973][ T5346] vfs_rename+0xa96/0xeb0
[ 88.400992][ T5346] ? __pfx_vfs_rename+0x10/0x10
[ 88.401010][ T5346] ovl_fill_super+0x46b7/0x5e20
[ 88.401030][ T5346] ? __pfx_ovl_fill_super+0x10/0x10
[ 88.401042][ T5346] ? xas_create+0x1902/0x1b90
[ 88.401060][ T5346] ? __pfx___mutex_trylock_common+0x10/0x10
[ 88.401076][ T5346] ? trace_contention_end+0x3d/0x140
[ 88.401094][ T5346] ? shrinker_register+0x124/0x230
[ 88.401111][ T5346] ? __mutex_unlock_slowpath+0x1be/0x6f0
[ 88.401127][ T5346] ? shrinker_register+0x61/0x230
[ 88.401143][ T5346] ? __pfx___mutex_lock+0x10/0x10
[ 88.401158][ T5346] ? __pfx___mutex_unlock_slowpath+0x10/0x10
[ 88.401177][ T5346] ? __raw_spin_lock_init+0x45/0x100
[ 88.401196][ T5346] ? sget_fc+0x962/0xa40
[ 88.401208][ T5346] ? __pfx_set_anon_super_fc+0x10/0x10
[ 88.401222][ T5346] ? __pfx_ovl_fill_super+0x10/0x10
[ 88.401241][ T5346] get_tree_nodev+0xbb/0x150
[ 88.401257][ T5346] vfs_get_tree+0x92/0x2a0
[ 88.401272][ T5346] do_new_mount+0x341/0xd30
[ 88.401283][ T5346] ? apparmor_capable+0x126/0x170
[ 88.401301][ T5346] ? __pfx_do_new_mount+0x10/0x10
[ 88.401311][ T5346] ? ns_capable+0x89/0xe0
[ 88.401322][ T5346] ? path_mount+0x690/0x10e0
[ 88.401333][ T5346] ? user_path_at+0xd4/0x160
[ 88.401346][ T5346] __se_sys_mount+0x31d/0x420
[ 88.401358][ T5346] ? __pfx___se_sys_mount+0x10/0x10
[ 88.401370][ T5346] ? __x64_sys_mount+0x20/0xc0
[ 88.401381][ T5346] ? entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 88.401391][ T5346] do_syscall_64+0x15f/0xf80
[ 88.401403][ T5346] ? trace_irq_disable+0x3b/0x140
[ 88.401413][ T5346] ? clear_bhb_loop+0x40/0x90
[ 88.401421][ T5346] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 88.401429][ T5346] RIP: 0033:0x7fa1ff79ce59
[ 88.401436][ T5346] Code: ff c3 66 (...)
[ 88.401443][ T5346] RSP: 002b:00007fa2005affe8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
[ 88.401456][ T5346] RAX: ffffffffffffffda RBX: 00007fa1ffa16180 RCX: 00007fa1ff79ce59
[ 88.401464][ T5346] RDX: 0000200000000100 RSI: 0000200000002240 RDI: 0000000000000000
[ 88.401474][ T5346] RBP: 00007fa1ff832d6f R08: 0000200000000440 R09: 0000000000000000
[ 88.401481][ T5346] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[ 88.401488][ T5346] R13: 00007fa1ffa16218 R14: 00007fa1ffa16180 R15: 00007ffc734fba78
[ 88.401500][ T5346] </TASK>
[ 88.401506][ T5346]
[ 88.401510][ T5346] Allocated by task 5325:
[ 88.401516][ T5346] kasan_save_track+0x3e/0x80
[ 88.401529][ T5346] __kasan_kmalloc+0x93/0xb0
[ 88.401542][ T5346] __kmalloc_cache_noprof+0x31c/0x660
[ 88.401554][ T5346] btrfs_relocate_block_group+0x217/0xc40
[ 88.401568][ T5346] btrfs_relocate_chunk+0x115/0x820
[ 88.401577][ T5346] __btrfs_balance+0x1db0/0x2ae0
[ 88.401587][ T5346] btrfs_balance+0xaf3/0x11b0
[ 88.401596][ T5346] btrfs_ioctl_balance+0x3d3/0x610
[ 88.401612][ T5346] __se_sys_ioctl+0xfc/0x170
[ 88.401626][ T5346] do_syscall_64+0x15f/0xf80
[ 88.401640][ T5346] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 88.401650][ T5346]
[ 88.401653][ T5346] Freed by task 5325:
[ 88.401659][ T5346] kasan_save_track+0x3e/0x80
[ 88.401671][ T5346] kasan_save_free_info+0x46/0x50
[ 88.401680][ T5346] __kasan_slab_free+0x5c/0x80
[ 88.401692][ T5346] kfree+0x1c5/0x640
[ 88.401703][ T5346] btrfs_relocate_block_group+0x95d/0xc40
[ 88.401715][ T5346] btrfs_relocate_chunk+0x115/0x820
[ 88.401724][ T5346] __btrfs_balance+0x1db0/0x2ae0
[ 88.401733][ T5346] btrfs_balance+0xaf3/0x11b0
[ 88.401742][ T5346] btrfs_ioctl_balance+0x3d3/0x610
[ 88.401757][ T5346] __se_sys_ioctl+0xfc/0x170
[ 88.401770][ T5346] do_syscall_64+0x15f/0xf80
[ 88.401785][ T5346] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 88.401795][ T5346]
[ 88.401798][ T5346] The buggy address belongs to the object at ffff888012312000
[ 88.401798][ T5346] which belongs to the cache kmalloc-2k of size 2048
[ 88.401807][ T5346] The buggy address is located 16 bytes inside of
[ 88.401807][ T5346] freed 2048-byte region [ffff888012312000, ffff888012312800)
[ 88.401819][ T5346]
[ 88.401822][ T5346] The buggy address belongs to the physical page:
[ 88.401829][ T5346] page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x12310
[ 88.401840][ T5346] head: order:3 mapcount:0 entire_mapcount:0 nr_pages_mapped:0 pincount:0
[ 88.401849][ T5346] flags: 0xfff00000000040(head|node=0|zone=1|lastcpupid=0x7ff)
[ 88.401860][ T5346] page_type: f5(slab)
[ 88.401871][ T5346] raw: 00fff00000000040 ffff88801ac42000 dead000000000100 dead000000000122
[ 88.401881][ T5346] raw: 0000000000000000 0000000800080008 00000000f5000000 0000000000000000
[ 88.401892][ T5346] head: 00fff00000000040 ffff88801ac42000 dead000000000100 dead000000000122
[ 88.401902][ T5346] head: 0000000000000000 0000000800080008 00000000f5000000 0000000000000000
[ 88.401913][ T5346] head: 00fff00000000003 fffffffffffffe01 00000000ffffffff 00000000ffffffff
[ 88.401923][ T5346] head: ffffffffffffffff 0000000000000000 00000000ffffffff 0000000000000008
[ 88.401929][ T5346] page dumped because: kasan: bad access detected
[ 88.401935][ T5346] page_owner tracks the page as allocated
[ 88.401941][ T5346] page last allocated via order 3, migratetype Unmovable, gfp_mask 0xd20c0(__GFP_IO|__GFP_FS|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_NOMEMALLOC), pid 9, tgid 9 (kworker/0:0), ts 83905464494, free_ts 83674944822
[ 88.401961][ T5346] post_alloc_hook+0x231/0x280
[ 88.401975][ T5346] get_page_from_freelist+0x24ba/0x2540
[ 88.401990][ T5346] __alloc_frozen_pages_noprof+0x18d/0x380
[ 88.402004][ T5346] allocate_slab+0x77/0x660
[ 88.402019][ T5346] refill_objects+0x339/0x3d0
[ 88.402033][ T5346] __pcs_replace_empty_main+0x321/0x720
[ 88.402043][ T5346] __kmalloc_node_track_caller_noprof+0x572/0x7b0
[ 88.402055][ T5346] __alloc_skb+0x2c1/0x7d0
[ 88.402067][ T5346] mld_newpack+0x14c/0xc90
[ 88.402080][ T5346] add_grhead+0x5a/0x2a0
[ 88.402093][ T5346] add_grec+0x1452/0x1740
[ 88.402105][ T5346] mld_ifc_work+0x6e6/0xe70
[ 88.402116][ T5346] process_scheduled_works+0xb5d/0x1860
[ 88.402127][ T5346] worker_thread+0xa53/0xfc0
[ 88.402138][ T5346] kthread+0x389/0x470
[ 88.402150][ T5346] ret_from_fork+0x514/0xb70
[ 88.402161][ T5346] page last free pid 5282 tgid 5282 stack trace:
[ 88.402168][ T5346] __free_frozen_pages+0xbc7/0xd30
[ 88.402180][ T5346] __slab_free+0x274/0x2c0
[ 88.402191][ T5346] qlist_free_all+0x99/0x100
[ 88.402201][ T5346] kasan_quarantine_reduce+0x148/0x160
[ 88.402211][ T5346] __kasan_slab_alloc+0x22/0x80
[ 88.402221][ T5346] __kmalloc_cache_noprof+0x2ba/0x660
[ 88.402231][ T5346] kernfs_fop_open+0x3f0/0xda0
[ 88.402253][ T5346] do_dentry_open+0x785/0x14e0
[ 88.402262][ T5346] vfs_open+0x3b/0x340
[ 88.402270][ T5346] path_openat+0x2e08/0x3860
[ 88.402281][ T5346] do_file_open+0x23e/0x4a0
[ 88.402292][ T5346] do_sys_openat2+0x113/0x200
[ 88.402300][ T5346] __x64_sys_openat+0x138/0x170
[ 88.402309][ T5346] do_syscall_64+0x15f/0xf80
[ 88.402326][ T5346] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 88.402336][ T5346]
[ 88.402339][ T5346] Memory state around the buggy address:
[ 88.402345][ T5346] ffff888012311f00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 88.402352][ T5346] ffff888012311f80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 88.402359][ T5346] >ffff888012312000: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 88.402365][ T5346] ^
[ 88.402370][ T5346] ffff888012312080: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 88.402380][ T5346] ffff888012312100: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 88.402385][ T5346] ==================================================================
Fix this by:
1) Making the reloc control structure ref counted;
2) Make revery place that access fs_info->reloc_ctl outside the relocation
code, which at the moment it's only replace_file_extents() and
btrfs_init_reloc_root(), get a reference count on the structure.
There's also btrfs_update_reloc_root() that is called outside the
relocation code, but this case is safe because it's only called in
the transaction commit path while under the fs_info->reloc_mutex
protection, but nevertheless grab a reference to make the code more
consistent and avoid false alerts from AI reviews;
3) Add a spinlock to protect fs_info->reloc_ctl, since we can not take the
fs_info->reloc_mutex as that would cause a deadlock since that lock is
taken in the transaction commit path. That spinlock is taken before
setting fs_info->reloc_ctl to an allocated structure, setting it to
NULL and reading fs_info->reloc_ctl;
4) Make sure the structure is freed only when its reference count drops to
zero.
Reported-by: syzbot+0eea49bba18051dea35e@syzkaller.appspotmail.com
Link: https://lore.kernel.org/linux-btrfs/6a1df323.bb0696ed.125a22.000a.GAE@google.com/
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
The qgroupid has a specific format, add common format specifier, similar
to what we have for checksums and keys.
Reviewed-by: Boris Burkov <boris@bur.io>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
With all the previous preparations, it's finally time to enable the
huge folio support.
- The max folio size
Here we define BTRFS_MAX_FOLIO_SIZE, which is fixed at 2MiB.
This will ensure we have a large enough but not too large folio for
btrfs. This limit applies to all systems regardless of page size.
Then we also define BTRFS_MAX_BLOCKS_PER_FOLIO, which depends on
CONFIG_BTRFS_EXPERIMENTAL.
If it's an experimental build, BTRFS_MAX_BLOCKS_PER_FOLIO is 512,
otherwise it's BITS_PER_LONG.
The filemap max order will be calculated using both
BTRFS_MAX_FOLIO_SIZE and BTRFS_MAX_BLOCKS_PER_FOLIO.
E.g. for 64K page size with 64K fs block size, the limit will be
BTRFS_MAX_FOLIO_SIZE (2M), which limits the filemap max order to 5.
This will be lower than the old order (6), but folios larger than 2M
are rarely any better for IO performance. Meanwhile excessively large
folios can cause other problems like stalling the IO pipeline for too
long.
For 4K page size and 4K fs block size, the limit will be increased to
2M from the old 256K.
This new size is constrained by both BTRFS_MAX_FOLIO_SIZE (2M) and
BTRFS_MAX_BLOCKS_PER_FOLIO (512 * 4K), allowing x86_64 to achieve huge
folio support, and the filemap max order will be 9.
- btrfs_bio_ctrl::submit_bitmap
This will be enlarged to contain BTRFS_MAX_BLOCKS_PER_FOLIO bits, and
this will be on-stack memory.
This will increase on-stack memory usage by 56 bytes compared to the
baseline (before the first patch in the series).
- Local @delalloc_bitmap inside writepage_delalloc()
Unfortunately we cannot afford to handle an allocation error here, thus
again we use on-stack memory.
Thus this will increase on-stack memory usage by 56 bytes again.
So unfortunately this means during the delalloc window, the writeback path
will have +112 bytes on-stack memory usage, and for other cases the
writeback path will have +56 bytes on-stack memory usage.
The +56 bytes (btrfs_bio_ctrl::submit_bitmap) can be removed
after we have reworked the compression submission, so the current
on-stack submit_bitmap is mostly a workaround until then.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
Btrfs has an internal flag/subpage bitmap called ordered, which is to
indicate that a block has corresponding ordered extent covering it.
However this requires extra synchronization between the inode ordered
tree, and the folio flag/subpage bitmap, not to mention we need to
maintain the extra folio flag with subpage bitmap.
As a step to align btrfs_folio_state more closely to iomap_folio_state,
remove the btrfs specific ordered flag/bitmap.
This will also save us 64 bytes for the bitmap of a huge folio.
Since we're here, also update the ASCII graph of the bitmap, as there
are only 3 sub-bitmaps now, show all sub-bitmaps directly.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
Currently btrfs_writepages() just accumulates as large bio as possible
(within writeback_control constraints) and then submits it. This can
however lead to significant latency in writeback IO submission (I have
observed tens of milliseconds) because the submitted bio easily has over
hundred of megabytes. Consequently this leads to IO pipeline stalls and
reduced throughput.
At the same time beyond certain size submitting so large bio provides
diminishing returns because the bio is split by the block layer
immediately anyway. So compute (estimate of) bio size beyond which we
are unlikely to improve performance and just submit the bio for
writeback once we accumulate that much to keep the IO pipeline busy.
This improves writeback throughput for sequential writes by about 15% on
the test machine I was using.
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Jan Kara <jack@suse.cz>
[ Fix the handling of missing device to avoid NULL pointer dereference. ]
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
Originally 2K block size support was introduced to test subpage (block
size < page size) on x86_64 where the page size is exactly the original
minimal block size.
However that 2K block size support has some problems:
- No 2K nodesize support
This is critical, as there is still no way to exercise the subpage
metadata routine.
- Very easy to test subpage data path now
With the currently experimental large folio support, it's very easy to
test the subpage data folio path already, as when a folio larger than
4K is encountered on x86_64, we will need all the subpage folio states
and bitmaps.
So there is no need to use 2K block size just to verify subpage data
path even on x86_64.
And with the incoming huge folio (2M on x86_64) support, the 2K block
size will easily double the bitmap size, considering the burden to
maintain and the limited extra coverage, I believe it's time to remove
it for the incoming huge folio support.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
The op field always points to the same use_bitmap function, the only
exception is during self tests where we make it temporarily point to a
different function. So just because of this op pointer field we are
increasing the structure size by 8 bytes.
Instead of storing a pointer to a use_bitmap function in struct
btrfs_free_space_ctl, move the pointer to struct btrfs_info, make
insert_into_bitmap() use that pointer if we are running the self tests
and initialize that pointer to the current, default use_bitmap function
(now exported for the tests as btrfs_use_bitmap). This way we reduce
the size of struct btrfs_free_space_ctl from 136 to 128 bytes and can
now fit 32 structures in a 4K page instead of 30. This also avoids the
cost of the indirection of a function pointer call when we are not
running the self tests.
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
[BACKGROUND]
Btrfs has a special mechanism called COW fixup, which detects dirty
pages without an ordered extent (folio ordered flag).
Normally a dirty folio must go through delayed allocation (delalloc)
before it can be submitted, and delalloc will create an ordered extent
for it and mark the range with ordered flag.
However in older kernels, there are bugs related to get_user_pages()
which can lead to some page marked dirty but without notifying the fs to
properly prepare them for writeback.
In that case without an ordered extent btrfs is unable to properly
submit such dirty folios, thus the COW fixup mechanism is introduced,
which do the extra space reservation so that they can be written back
properly.
[MODERN SOLUTIONS]
The MM layer has solved it properly now with the introduction of
pin_user_pages*(), so we're handling cases that are no longer valid.
So commit 7ca3e84980ef ("btrfs: reject out-of-band dirty folios during
writeback") is introduced to change the behavior from going through
COW fixup to rejecting them directly for experimental builds.
So far it works fine, but when errors are injected into the IO path, we
have random failures triggering the new warnings.
It looks like we have error path that cleared the ordered flag but
leaves the folio dirty flag, which later triggers the warning.
[REMOVAL OF COW FIXUP]
Although I hope to fix all those known warnings cases, I just can not
figure out the root cause yet.
But on the other hand, if we remove the ordered and checked flags in the
future, and purely rely on the dirty flags and ordered extent search, we
can get a much cleaner handling.
Considering it's no longer hitting the COW fixup for normal IO paths, I
think it's finally the time to remove the COW fixup completely.
Furthermore, the function name "btrfs_writepage_cow_fixup()" is no
longer meaningful, and since it's pretty small, only a folio flag check
with error message, there is no need to put it as a dedicated helper,
just open code it inside extent_writepage_io().
Signed-off-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
The first transaction that enables squotas is special and a bit tricky.
We have to set BTRFS_FS_QUOTA_ENABLED after the transaction to avoid a
deadlock, so any delayed refs that run before we set the bit are not
squota accounted. For data this is fine, we don't get an owner_ref, so
there is no real harm, it's as if the extent predated squotas. However
for metadata, the tree block will have gen == enable_gen so when we free
it later, we will decrement the squota accounting, which can result in
an underflow. Before it is freed, btrfs check shows errors, as we have
mismatched usage between the node generations/owners and the squota
values.
There are two angles to this fix:
1. For extents that come in delayed_refs that run during the
enable_gen transaction, we must actually set enable_gen to the *next*
transaction. That is the first transaction that we can really
properly account in any way.
2. For extents that come in between the end of our transaction handle
and the time we set the BTRFS_FS_QUOTA_ENABLED bit, we need an
additional bit, BTRFS_FS_SQUOTA_ENABLING which only affects recording
squota deltas, so we do pick up those extents. Otherwise, we would
miss them, even for enable_gen + 1.
Fixes: bd7c1ea3a302 ("btrfs: qgroup: check generation when recording simple quota delta")
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Boris Burkov <boris@bur.io>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
Commit 347b7042fb26 ("Merge patch series "fs: generic file IO error
reporting"") has introduced a common framework for reporting errors to
fsnotify in a standard way.
One of the functions being introduced is fserror_report_shutdown() that,
when combined with the experimental support for shutdown in btrfs, it
means that user-space can also easily detect whenever a btrfs filesystem
has been marked as shutdown.
Signed-off-by: Miquel Sabaté Solà <mssola@mssola.com>
Reviewed-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
There are several functions that take pointer arguments but don't need to
modify the objects they point to, so add the const qualifiers.
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
When trimming unallocated space, btrfs_trim_fs() holds the device_list_mutex
for the entire duration while iterating through all devices. On large
filesystems with significant unallocated space, this operation can take
minutes to hours on large storage systems.
This causes a problem because btrfs_run_dev_stats(), which is called
during transaction commit, also requires device_list_mutex:
btrfs_trim_fs()
mutex_lock(&fs_devices->device_list_mutex)
list_for_each_entry(device, ...)
btrfs_trim_free_extents(device)
mutex_unlock(&fs_devices->device_list_mutex)
commit_transaction()
btrfs_run_dev_stats()
mutex_lock(&fs_devices->device_list_mutex) // blocked!
...
While trim is running, all transaction commits are blocked waiting for
the mutex.
Fix this by refactoring btrfs_trim_free_extents() to process devices in
bounded chunks (up to 2GB per iteration) and release device_list_mutex
between chunks.
Signed-off-by: robbieko <robbieko@synology.com>
Signed-off-by: jinbaohong <jinbaohong@synology.com>
Reviewed-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
There are two unnecessary 4B holes in btrfs_delayed_root;
struct btrfs_delayed_root {
spinlock_t lock; /* 0 4 */
/* XXX 4 bytes hole, try to pack */
struct list_head node_list; /* 8 16 */
struct list_head prepare_list; /* 24 16 */
atomic_t items; /* 40 4 */
atomic_t items_seq; /* 44 4 */
int nodes; /* 48 4 */
/* XXX 4 bytes hole, try to pack */
wait_queue_head_t wait; /* 56 24 */
/* size: 80, cachelines: 2, members: 7 */
/* sum members: 72, holes: 2, sum holes: 8 */
/* last cacheline: 16 bytes */
};
Reordering 'nodes' after 'lock' reduces size by 8B, to 72 on release
config.
Reviewed-by: Boris Burkov <boris@bur.io>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
The fs_info::delayed_root is allocated dynamically but there's only one
instance per filesystem so we can embed it into the fs_info itself.
The two object have the same lifetime and delayed roots are always
present so we don't need to allocate it on demand from slab.
There's still some space left in fs_info until the 4K so there won't be
an spill over to next page on release config (size grows from 3880 to
3952). In case we want to shrink fs_info there are still holes to fill
or we can separate other non-core or optional structures if needed.
Link: https://lore.kernel.org/all/cover.1767979013.git.dsterba@suse.com/
Reviewed-by: Boris Burkov <boris@bur.io>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
Handle the case where we free an extent from a block group that has the
REMAPPED flag set. Because the remap tree is orthogonal to the extent
tree, for data this may be within any number of identity remaps or
actual remaps. If we're freeing a metadata node, this will be wholly
inside one or the other.
btrfs_remove_extent_from_remap_tree() searches the remap tree for the
remaps that cover the range in question, then calls
remove_range_from_remap_tree() for each one, to punch a hole in the
remap and adjust the free-space tree.
For an identity remap, remove_range_from_remap_tree() will adjust the
block group's `identity_remap_count` if this changes. If it reaches
zero we mark the block group as fully remapped.
For an identity remap, remove_range_from_remap_tree() will adjust the
block group's `identity_remap_count` if this changes. If it reaches
zero we mark the block group as fully remapped.
Fully remapped block groups have their chunk stripes removed and their
device extents freed, which makes the disk space available again to the
chunk allocator. This happens asynchronously: in the cleaner thread for
sync discard and nodiscard, and (in a later patch) in the discard worker
for async discard.
Reviewed-by: Boris Burkov <boris@bur.io>
Signed-off-by: Mark Harmstone <mark@harmstone.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
If we encounter a filesystem with the remap-tree incompat flag set,
validate its compatibility with the other flags, and load the remap tree
using the values that have been added to the superblock.
The remap-tree feature depends on the free-space-tree, but no-holes and
block-group-tree have been made dependencies to reduce the testing
matrix. Similarly I'm not aware of any reason why mixed-bg and zoned would be
incompatible with remap-tree, but this is blocked for the time being
until it can be fully tested.
Reviewed-by: Boris Burkov <boris@bur.io>
Signed-off-by: Mark Harmstone <mark@harmstone.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
Add a new METADATA_REMAP chunk type, which is a metadata chunk that holds the
remap tree.
This is needed for bootstrapping purposes: the remap tree can't itself
be remapped, and must be relocated the existing way, by COWing every
leaf. The remap tree can't go in the SYSTEM chunk as space there is
limited, because a copy of the chunk item gets placed in the superblock.
The changes in fs/btrfs/volumes.h are because we're adding a new block
group type bit after the profile bits, and so can no longer rely on the
const_ilog2 trick.
The sizing to 32MB per chunk, matching the SYSTEM chunk, is an estimate
here, we can adjust it later if it proves to be too big or too small.
This works out to be ~500,000 remap items, which for a 4KB block size
covers ~2GB of remapped data in the worst case and ~500TB in the best case.
Reviewed-by: Boris Burkov <boris@bur.io>
Signed-off-by: Mark Harmstone <mark@harmstone.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
There are two tests in btrfs_fs_closing() but checking the
BTRFS_FS_CLOSING_DONE bit is done only in one place
load_extent_tree_free(). As this is an inline we can reduce size of the
generated code. The types can be also changed to bool as this becomes a
simple condition.
text data bss dec hex filename
1674006 146704 15560 1836270 1c04ee pre/btrfs.ko
1673772 146704 15560 1836036 1c0404 post/btrfs.ko
DELTA: -234
Reviewed-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
Instead of surrounding every caller of btrfs_is_shutdown() with unlikely,
move the unlikely into the helper itself, like we do in other places in
btrfs and is common in the kernel outside btrfs too. Also make the fs_info
argument of btrfs_is_shutdown() const.
On a x86_84 box using gcc 14.2.0-19 from Debian, this resulted in a slight
reduction of the module's text size.
Before:
$ size fs/btrfs/btrfs.ko
text data bss dec hex filename
1939044 172568 15592 2127204 207564 fs/btrfs/btrfs.ko
After:
$ size fs/btrfs/btrfs.ko
text data bss dec hex filename
1938876 172568 15592 2127036 2074bc fs/btrfs/btrfs.ko
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
Make btrfs use the library APIs instead of crypto_shash, for all
checksum computations. This has many benefits:
- Allows future checksum types, e.g. XXH3 or CRC64, to be more easily
supported. Only a library API will be needed, not crypto_shash too.
- Eliminates the overhead of the generic crypto layer, including an
indirect call for every function call and other API overhead. A
microbenchmark of btrfs_check_read_bio() with crc32c checksums shows a
speedup from 658 cycles to 608 cycles per 4096-byte block.
- Decreases the stack usage of btrfs by reducing the size of checksum
contexts from 384 bytes to 240 bytes, and by eliminating the need for
some functions to declare a checksum context at all.
- Increases reliability. The library functions always succeed and
return void. In contrast, crypto_shash can fail and return errors.
Also, the library functions are guaranteed to be available when btrfs
is loaded; there's no longer any need to use module softdeps to try to
work around the crypto modules sometimes not being loaded.
- Fixes a bug where blake2b checksums didn't work on kernels booted with
fips=1. Since btrfs checksums are for integrity only, it's fine for
them to use non-FIPS-approved algorithms.
Note that with having to handle 4 algorithms instead of just 1-2, this
commit does result in a slightly positive diffstat. That being said,
this wouldn't have been the case if btrfs had actually checked for
errors from crypto_shash, which technically it should have been doing.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Reviewed-by: Neal Gompa <neal@gompa.dev>
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
[BUG]
There is a bug report where a heavily fuzzed fs is mounted with all
rescue mount options, which leads to the following warnings during
unmount:
BTRFS: Transaction aborted (error -22)
Modules linked in:
CPU: 0 UID: 0 PID: 9758 Comm: repro.out Not tainted
6.19.0-rc5-00002-gb71e635feefc #7 PREEMPT(full)
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
RIP: 0010:find_free_extent_update_loop fs/btrfs/extent-tree.c:4208 [inline]
RIP: 0010:find_free_extent+0x52f0/0x5d20 fs/btrfs/extent-tree.c:4611
Call Trace:
<TASK>
btrfs_reserve_extent+0x2cd/0x790 fs/btrfs/extent-tree.c:4705
btrfs_alloc_tree_block+0x1e1/0x10e0 fs/btrfs/extent-tree.c:5157
btrfs_force_cow_block+0x578/0x2410 fs/btrfs/ctree.c:517
btrfs_cow_block+0x3c4/0xa80 fs/btrfs/ctree.c:708
btrfs_search_slot+0xcad/0x2b50 fs/btrfs/ctree.c:2130
btrfs_truncate_inode_items+0x45d/0x2350 fs/btrfs/inode-item.c:499
btrfs_evict_inode+0x923/0xe70 fs/btrfs/inode.c:5628
evict+0x5f4/0xae0 fs/inode.c:837
__dentry_kill+0x209/0x660 fs/dcache.c:670
finish_dput+0xc9/0x480 fs/dcache.c:879
shrink_dcache_for_umount+0xa0/0x170 fs/dcache.c:1661
generic_shutdown_super+0x67/0x2c0 fs/super.c:621
kill_anon_super+0x3b/0x70 fs/super.c:1289
btrfs_kill_super+0x41/0x50 fs/btrfs/super.c:2127
deactivate_locked_super+0xbc/0x130 fs/super.c:474
cleanup_mnt+0x425/0x4c0 fs/namespace.c:1318
task_work_run+0x1d4/0x260 kernel/task_work.c:233
exit_task_work include/linux/task_work.h:40 [inline]
do_exit+0x694/0x22f0 kernel/exit.c:971
do_group_exit+0x21c/0x2d0 kernel/exit.c:1112
__do_sys_exit_group kernel/exit.c:1123 [inline]
__se_sys_exit_group kernel/exit.c:1121 [inline]
__x64_sys_exit_group+0x3f/0x40 kernel/exit.c:1121
x64_sys_call+0x2210/0x2210 arch/x86/include/generated/asm/syscalls_64.h:232
do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
do_syscall_64+0xe8/0xf80 arch/x86/entry/syscall_64.c:94
entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x44f639
Code: Unable to access opcode bytes at 0x44f60f.
RSP: 002b:00007ffc15c4e088 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
RAX: ffffffffffffffda RBX: 00000000004c32f0 RCX: 000000000044f639
RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000001
RBP: 0000000000000001 R08: ffffffffffffffc0 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000004c32f0
R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000001
</TASK>
Since rescue mount options will mark the full fs read-only, there should
be no new transaction triggered.
But during unmount we will evict all inodes, which can trigger a new
transaction, and triggers warnings on a heavily corrupted fs.
[CAUSE]
Btrfs allows new transaction even on a read-only fs, this is to allow
log replay happen even on read-only mounts, just like what ext4/xfs do.
However with rescue mount options, the fs is fully read-only and cannot
be remounted read-write, thus in that case we should also reject any new
transactions.
[FIX]
If we find the fs has rescue mount options, we should treat the fs as
error, so that no new transaction can be started.
Reported-by: Jiaming Zhang <r772577952@gmail.com>
Link: https://lore.kernel.org/linux-btrfs/CANypQFYw8Nt8stgbhoycFojOoUmt+BoZ-z8WJOZVxcogDdwm=Q@mail.gmail.com/
Reviewed-by: Boris Burkov <boris@bur.io>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
Move the CSUM_FMT* definitions to fs.h where is be the BTRFS_KEY_FMT
and add the prefix for consistency.
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
The reason why end_bbio_compressed_write() queues a work into
compressed_write_workers wq is for end_compressed_writeback() call, as
it will grab all the involved folios and clear the writeback flags,
which may sleep.
However now we always run btrfs_bio::end_io() in task context, there is
no need to queue the work anymore.
Just remove btrfs_fs_info::compressed_write_workers and
compressed_bio::write_end_work.
There is a comment about the works queued into
compressed_write_workers, now change to flush endio wq instead, which is
responsible to handle all data endio functions.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
There's a lot of places where we need to print a key, and it's tiresome
to type the format specifier, typically "(%llu %u %llu)", as well as
passing 3 arguments to a prink family function (key->objectid, key->type,
key->offset).
So add a couple macros for this just like we have for csum values in
btrfs_inode.h (CSUM_FMT and CSUM_FMT_VALUE).
This also ensures that we consistently print a key in the same format,
always as "(%llu %llu %llu)", which is the most common format we use, but
we have a few variations such as "[%llu %llu %llu]" for no good reason.
This patch introduces the macros while the next one makes use of it.
This is to ease backports of future patches, since then we can backport
this patch which is simple and short and then backport those future
patches, as the next patch in the series that makes use of these new
macros is quite large and may have some dependencies.
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
A new fs state EMERGENCY_SHUTDOWN is introduced, which is btrfs'
equivalent of XFS_IOC_GOINGDOWN or EXT4_IOC_SHUTDOWN, after entering
emergency shutdown state, all operations will return errors (-EIO), and
can not be bring back to normal state until unmouont.
The new state will reject the following file operations:
- read_iter()
- write_iter()
- mmap()
- open()
- remap_file_range()
- uring_cmd()
- splice_read()
This requires a small wrapper to do the extra shutdown check, then call
the regular filemap_splice_read() function
This should reject most of the file operations on a shutdown btrfs.
And for the existing dirty folios, extra shutdown checks are introduced
to the following functions:
- run_delalloc_nocow()
- run_delalloc_compressed()
- cow_file_range()
So that dirty ranges will still be properly cleaned without being
submitted.
Finally the shutdown state will also set the fs error, so that no new
transaction will be committed, protecting the metadata from any possible
further corruption.
And when the fs entered shutdown mode for the first time, a critical
level kernel message will show up to indicate the incident.
That message will be important for end users as rejected delalloc ranges
will output error messages, hopefully that shutdown message and the fact
that all fs operations are returning error will prevent end users from
getting too confused about the delalloc error messages.
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Reviewed-by: Anand Jain <asj@kernel.org>
Tested-by: Anand Jain <asj@kernel.org>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
This includes the following preparation for bs > ps cases:
- Always alloc/free the folio directly if bs > ps
This adds a new @fs_info parameter for btrfs_alloc_compr_folio(), thus
affecting all compression algorithms.
For btrfs_free_compr_folio() it needs no parameter for now, as we can
use the folio size to skip the caching part.
For now the change is just to passing a @fs_info into the function,
all the folio size assumption is still based on page size.
- Properly zero the last folio in compress_file_range()
Since the compressed folios can be larger than a page, we need to
properly zero the whole folio.
- Use correct folio size for btrfs_add_compressed_bio_folios()
Instead of page size, use the correct folio size.
- Use correct folio size/shift for btrfs_compress_filemap_get_folio()
As we are not only using simple page sized folios anymore.
- Use correct folio size for btrfs_decompress()
There is an ASSERT() making sure the decompressed range is no larger
than a page, which will be triggered for bs > ps cases.
- Skip readahead for compressed pages
Similar to subpage cases.
- Make btrfs_alloc_folio_array() to accept a new @order parameter
- Add a helper to calculate the minimal folio size
All those changes should not affect the existing bs <= ps handling.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
We can annotate btrfs_is_testing() as unlikely since that's the most
expected scenario and it's desirable for the compiler to optimize for
the case we are not running the self tests. So add the annotation to
btrfs_is_testing() and while at it also make it return bool instead of
int.
Also make two of the existing callers use btrfs_is_testing() directly
instead of storing its result in a local variable.
On x86_64 with Debian's gcc 14.2.0-19 this resulted in a very tiny object
code reduction.
Before this change:
$ size fs/btrfs/btrfs.ko
text data bss dec hex filename
1913263 161567 15592 2090422 1fe5b6 fs/btrfs/btrfs.ko
After this change:
$ size fs/btrfs/btrfs.ko
text data bss dec hex filename
1913257 161567 15592 2090416 1fe5b0 fs/btrfs/btrfs.ko
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
Currently debugging log replay failures can be harder than needed, since
all we do now is abort a transaction, which gives us a line number, a
stack trace and an error code. But that is most of the times not enough
to give some clue about what went wrong. So add a new helper to abort
log replay and provide contextual information:
1) Dump the current leaf of the log tree being processed and print the
slot we are currently at and the key at that slot;
2) Dump the current subvolume tree leaf if we have any;
3) Print the current stage of log replay;
4) Print the id of the subvolume root associated with the log tree we
are currently processing (as we can have multiple);
5) Print some error message to mention what we were trying to do when we
got an error.
Replace all transaction abort calls (btrfs_abort_transaction()) with the
new helper btrfs_abort_log_replay(), which besides dumping all that extra
information, it also aborts the current transaction.
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
Inside btrfs_fs_info we cache several bits shift like sectorsize_bits.
Apply this to max and min folio orders so that every time mapping order
needs to be applied we can skip the calculation.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
Annual typo fixing pass. Strangely codespell found only about 30% of
what is in this patch, the rest was done manually using text
spellchecker with a custom dictionary of acceptable terms.
Reviewed-by: Neal Gompa <neal@gompa.dev>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
This involves:
- Add zstd_alloc_workspace_manager() and zstd_free_workspace_manager()
Those two functions will accept an fs_info pointer, and alloc/free
fs_info->compr_wsm[BTRFS_COMPRESS_ZSTD] pointer.
- Add btrfs_alloc_compress_wsm() and btrfs_free_compress_wsm()
Those are helpers allocating the workspace managers for all
algorithms.
For now only zstd is supported, and the timing is a little unusual,
the btrfs_alloc_compress_wsm() should only be called after the
sectorsize being initialized.
Meanwhile btrfs_free_fs_info_compress() is called in
btrfs_free_fs_info().
- Move the definition of btrfs_compression_type to "fs.h"
The reason is that "compression.h" has already included "fs.h", thus
we can not just include "compression.h" to get the definition of
BTRFS_NR_COMPRESS_TYPES to define fs_info::compr_wsm[].
For now the per-fs zstd workspace manager won't really have any effect,
and all compression is still going through the global workspace manager.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
The ref_tracker infrastructure aids debugging but is not enabled by
default as it has a performance impact. Add mount option 'ref_tracker'
so it can be selectively enabled on a filesystem. Currently it track
references of 'delayed inodes'.
Signed-off-by: Leo Martins <loemra.dev@gmail.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
Remove CONFIG_BTRFS_FS_REF_VERIFY Kconfig and add it as part of
CONFIG_BTRFS_DEBUG. This should not be impactful to the performance
of debug. The struct btrfs_ref takes an additional u64, btrfs_fs_info
takes an additional spinlock_t and rb_root. All of the ref_verify logic
is still protected by a mount option.
Signed-off-by: Leo Martins <loemra.dev@gmail.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
Currently we manually check the block size against 3 different values:
- 4K
- PAGE_SIZE
- MIN_BLOCKSIZE
Those 3 values can match or differ from each other. This makes it
pretty complex to output the supported block sizes.
Considering we're going to add block size > page size support soon, this
can make the support block size sysfs attribute much harder to
implement.
To make it easier, factor out a helper, btrfs_supported_blocksize() to
do a simple check for the block size.
Then utilize it in the two locations:
- btrfs_validate_super()
This is very straightforward
- supported_sectorsizes_show()
Iterate through all valid block sizes, and only output supported ones.
This is to make future full range block sizes support much easier.
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
So far we've been deriving the buffer tree index using the sector size.
But each extent buffer covers multiple sectors. This makes the buffer
tree rather sparse.
For example the typical and quite common configuration uses sector size
of 4KiB and node size of 16KiB. In this case it means the buffer tree is
using up to the maximum of 25% of it's slots. Or in other words at least
75% of the tree slots are wasted as never used.
We can score significant memory savings on the required tree nodes by
indexing the tree using the node size instead. As a result far less
slots are wasted and the tree can now use up to all 100% of it's slots
this way.
Note: This works even with unaligned tree blocks as we can still get
unique index by doing eb->start >> nodesize_shift.
Getting some stats from running fio write test, there is a bit of
variance. The values presented in the table below are medians from 5
test runs. The numbers are:
- # of allocated ebs in the tree
- # of leaf tree nodes
- highest index in the tree (radix tree width)):
ebs / leaves / Index | bare for-next | with fix
---------------------+--------------------+-------------------
post mount | 16 / 11 / 10e5c | 16 / 10 / 4240
post test | 5810 / 891 / 11cfc | 4420 / 252 / 473a
post rm | 574 / 300 / 10ef0 | 540 / 163 / 46e9
In this case (10GiB filesystem) the height of the tree is still 3 levels
but the 4x width reduction is clearly visible as expected. But since the
tree is more dense we can see the 54-72% reduction of leaf nodes. That's
very close to ideal with this test. It means the tree is getting really
dense with this kind of workload.
Also, the fio results show no performance change.
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Daniel Vacek <neelx@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
The file system type is not a very useful holder as it doesn't allow us
to go back to the actual file system instance. Pass the super_block
instead which is useful when passed back to the file system driver.
This matches what is done for all other block device based file systems,
and allows us to remove btrfs_fs_info::bdev_holder completely.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
It's not used anymore after commit 091344508249 ("btrfs: qgroup: use
qgroup_iterator in qgroup_convert_meta()"), so remove it.
Reviewed-by: Boris Burkov <boris@bur.io>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
When debugging/detecting outlier commit stalls, having an indicator that
we are currently in a long commit critical section can be very useful.
Extend the commit_stats sysfs file to also include the current commit
critical section duration.
Since this requires storing the last commit start time, use that rather
than a separate stack variable for storing the finished commit durations
as well.
This also requires slightly moving up the timing of the stats updating
to *inside* the critical section to avoid the transaction T+1 setting
the critical_section_start_time to 0 before transaction T can update its
stats, which would trigger the new ASSERT. This is an improvement in and
of itself, as it makes the stats more accurately represent the true
critical section time. It may be yet better to pull the stats up to where
start_transaction gets unblocked, rather than the next commit, but this
seems like a good enough place as well.
Signed-off-by: Boris Burkov <boris@bur.io>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
We need to add a dedicated block_rsv for tree-log, because the block_rsv
serves for a tree node allocation in btrfs_alloc_tree_block(). Currently,
tree-log tree uses fs_info->empty_block_rsv, which is shared across trees
and points to the normal metadata space_info. Instead, we add a dedicated
block_rsv and that block_rsv can use the dedicated sub-space_info.
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
In order to fully utilize xarray tagging to improve writeback we need to
convert the buffer_radix to a proper xarray. This conversion is
relatively straightforward as the radix code uses the xarray underneath.
Using xarray directly allows for quite a lot less code.
Reviewed-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
The Btrfs documentation states that if the commit value is greater than
300 a warning should be issued. The warning was accidentally lost in the
new mount API update.
Fixes: 6941823cc878 ("btrfs: remove old mount API code")
CC: stable@vger.kernel.org # 6.12+
Reviewed-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Kyoji Ogasawara <sawara04.o@gmail.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
Since I have triggered the ASSERT() on the delayed iput too many times,
now is the time to add some extra debug warnings for delayed iput.
All delayed iputs should be queued after all ordered extents finish
their IO and all involved workqueues are flushed.
Thus after the btrfs_run_delayed_iputs() inside close_ctree(), there
should be no more delayed puts added.
So introduce a new BTRFS_FS_STATE_NO_DELAYED_IPUT, set after the above
mentioned timing. And all btrfs_add_delayed_iput() will check that flag
and give a WARN_ON_ONCE().
Reviewed-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
The zstd and zlib compression types support setting compression levels.
Enhance the defrag interface to specify the levels as well. For zstd the
negative (realtime) levels are also accepted.
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Daniel Vacek <neelx@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
Currently we only support two block sizes, 4K and PAGE_SIZE.
This means on the most common architecture x86_64, we have no way to
test subpage block size. And that's exactly I have an aarch64 machine
dedicated for subpage tests.
But this is still a hurdle for a lot of btrfs developers, and to improve
the test coverage mostly on x86_64, here we enable debug builds to
accept 2K block size.
This involves:
- Introduce a dedicated minimal block size macro
BTRFS_MIN_BLOCKSIZE, which depends on if CONFIG_BTRFS_DEBUG is set.
If so it's 2K, otherwise it's 4K as usual.
- Allow 4K, PAGE_SIZE and BTRFS_MIN_BLOCKSIZE as block size
- Update subpage block size checks to be based on BTRFS_MIN_BLOCKSIZE
- Export the new supported blocksize through sysfs interfaces
As most of the subpage support is already pretty mature, there is no
extra work needed to support the extra 2K block size.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
For the future large folio support, our filemap can have folios with
different sizes, thus we can no longer rely on a fixed blocks_per_page
value.
To prepare for that future, here we do:
- Remove btrfs_fs_info::sectors_per_page
- Introduce a helper, btrfs_blocks_per_folio()
Which uses the folio size to calculate the number of blocks for each
folio.
- Migrate the existing btrfs_fs_info::sectors_per_page to use that
helper
There are some exceptions:
* Metadata nodesize < page size support
In the future, even if we support large folios, we will only
allocate a folio that matches our nodesize.
Thus we won't have a folio covering multiple metadata unless
nodesize < page size.
* Existing subpage bitmap dump
We use a single unsigned long to store the bitmap.
That means until we change the bitmap dumping code, our upper limit
for folio size will only be 256K (4K block size, 64 bit unsigned
long).
* btrfs_is_subpage() check
This will be migrated into a future patch.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
Allow using the fast modes (negative compression levels) of zstd as a
mount option.
As per the results, the compression ratio is (expectedly) lower:
for level in {-15..-1} 1 2 3; \
do printf "level %3d\n" $level; \
mount -o compress=zstd:$level /dev/sdb /mnt/test/; \
grep sdb /proc/mounts; \
cp -r /usr/bin /mnt/test/; sync; compsize /mnt/test/bin; \
cp -r /usr/share/doc /mnt/test/; sync; compsize /mnt/test/doc; \
cp enwik9 /mnt/test/; sync; compsize /mnt/test/enwik9; \
cp linux-6.13.tar /mnt/test/; sync; compsize /mnt/test/linux-6.13.tar; \
rm -r /mnt/test/{bin,doc,enwik9,linux-6.13.tar}; \
umount /mnt/test/; \
done |& tee results | \
awk '/^level/{print}/^TOTAL/{print$3"\t"$2" |"}' | paste - - - - -
266M bin | 45M doc | 953M wiki | 1.4G source
=============================+===============+===============+===============+
level -15 180M 67% | 30M 68% | 694M 72% | 598M 40% |
level -14 180M 67% | 30M 67% | 683M 71% | 581M 39% |
level -13 177M 66% | 29M 66% | 671M 70% | 566M 38% |
level -12 174M 65% | 29M 65% | 658M 69% | 548M 37% |
level -11 174M 65% | 28M 64% | 645M 67% | 530M 35% |
level -10 171M 64% | 28M 62% | 631M 66% | 512M 34% |
level -9 165M 62% | 27M 61% | 615M 64% | 493M 33% |
level -8 161M 60% | 27M 59% | 598M 62% | 475M 32% |
level -7 155M 58% | 26M 58% | 582M 61% | 457M 30% |
level -6 151M 56% | 25M 56% | 565M 59% | 437M 29% |
level -5 145M 54% | 24M 55% | 545M 57% | 417M 28% |
level -4 139M 52% | 23M 52% | 520M 54% | 391M 26% |
level -3 135M 50% | 22M 50% | 495M 51% | 369M 24% |
level -2 127M 47% | 22M 48% | 470M 49% | 349M 23% |
level -1 120M 45% | 21M 47% | 452M 47% | 332M 22% |
level 1 110M 41% | 17M 39% | 362M 38% | 290M 19% |
level 2 106M 40% | 17M 38% | 349M 36% | 288M 19% |
level 3 104M 39% | 16M 37% | 340M 35% | 276M 18% |
The samples represent some data sets that can be commonly found and show
approximate compressibility. The fast levels trade off speed for ratio
and are best suitable for highly compressible data.
As can be seen above, comparing the results to the current default zstd
level 3, the negative levels are roughly 2x worse at -15 and the
ratio increases almost linearly with each level.
Signed-off-by: Daniel Vacek <neelx@suse.com>
[ update changelog ]
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
Track number of read blocks in the whole filesystem. The counter is
initialized when devices are opened. The counter is increased at
btrfs_submit_dev_bio() if the stats tracking is enabled (depends on the
read policy). Stats tracking is disabled by default and is enabled
through fs_devices::collect_fs_stats when required.
The code is not under the EXPERIMENTAL define, as stats can be expanded
to include write counts and other performance counters, with the user
interface independent of its internal use.
This is an in-memory-only feature, not related to the dev error stats.
Signed-off-by: Anand Jain <anand.jain@oracle.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
|
The header clearly states that it does not want to be included directly,
only via linux/spinlock_types.h. Drop this as we can simply use the
spinlock.h which is already included.
Signed-off-by: Wolfram Sang <wsa+renesas@sang-engineering.com>
Reviewed-by: David Sterba <dsterba@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
|
