| Age | Commit message (Collapse) | Author |
|---|
|
The default core-sched ordering runs the longest waiting task first by
comparing p->scx.core_sched_at stamps. The stamp is maintained under two
rules. touch_core_sched() stamps when a task starts waiting for a CPU and
when its slice runs out. If the scheduler implements
ops.core_sched_before(), touch_core_sched_dispatch() re-stamps on every
dispatch.
A comparison can see one stamp taken under each rule, which isn't a
meaningful ordering. The dispatch rule also buys little - it only aligns
bypass-mode comparisons with the local DSQ order. Multiple schedulers make
the mixed comparisons more common.
Wait time is what p->scx.runnable_at already tracks for the stall watchdog.
Delete core_sched_at with both touch functions and compare runnable_at in
the scx_prio_less() fallback.
runnable_at is refreshed only on enqueue and goes stale while a task keeps
occupying its CPU. Instead of re-stamping, order a running task after every
waiting task as it is the most recently serviced.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
scx_prio_less() implements prio_less() semantics - %true means that @a is
the lower priority and should run after @b. ops.core_sched_before() is
documented to return %true when @a should run before @b. scx_prio_less()
returns the op's value as-is, inverting the documented semantics at runtime.
Call the op with the arguments swapped.
scx_qmap followed the wiring instead of the documentation and returned %true
for the younger task, so the two inversions canceled out and it behaved as
intended. Flip its comparison to match. scx_qmap is likely the only current
user in or out of the kernel tree. Any scheduler written the same way needs
the same flip, while schedulers following the documentation are fixed by
this change.
Fixes: 7b0888b7cc19 ("sched_ext: Implement core-sched support")
Cc: stable@vger.kernel.org # v6.12+
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
blk_mq_update_nr_hw_queues() in nbd_start_device() may cause a queue
freeze. The previous commit addressed this for newly created nbd
devices by setting the expected nr_hw_queues in nbd_dev_add(). However,
when reusing an old inactive nbd device, the queue freeze can still
occur if the old nbd->tag_set->nr_hw_queues does not match the new
socket connection count. Inactive nbd devices can originate from two
sources: loading the nbd module with nbds_max, which sets the default
nr_hw_queues to 1, and the netlink method, which sets nr_hw_queues
according to the expected number of socket connections. For the first
case, add a module parameter so the default nr_hw_queues can be
changed. Users who know their expected number of connections can then
prevent queue freezes on pre-created devices via nbds_max.
Before this patchset:
real 0m2.195s
user 0m0.005s
sys 0m0.022s
After this patchset:
real 0m0.090s
user 0m0.004s
sys 0m0.018s
Signed-off-by: Yang Erkun <yangerkun@huawei.com>
Reviewed-by: Yu Kuai <yukuai@fygo.io>
Link: https://patch.msgid.link/20260805122930.57647-9-yangerkun@huawei.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Previous commits has removed the queue freeze in nbd_add_socket and
nbd_set_size during nbd device setup. However, a queue freeze can still
occur when nbd_start_device calls blk_mq_update_nr_hw_queues if the
socket connection count does not match nbd->tag_set->nr_hw_queues.
The nbd_start_device function can be invoked through either the ioctl or
netlink paths. The ioctl path only allows reusing an existing inactivate
nbd device, there is nothing more we can do to prevent the queue freeze
since the old nbd->tag_set->nr_hw_queues may not match the new socket
connection count. Similarly, the netlink path can reuse a preferred
inactivate nbd device, and again, we cannot do more in this scenario.
However, the netlink path can also add a new nbd device using
nbd_dev_add. In this case, we can obtain the new number of socket
connections, and by adding a new argument representing the expected
nr_hw_queues in nbd_dev_add, we can ensure the queue freeze is avoided
for this situation.
Reviewed-by: Yu Kuai <yukuai@fygo.io>
Signed-off-by: Yang Erkun <yangerkun@huawei.com>
Link: https://patch.msgid.link/20260805122930.57647-8-yangerkun@huawei.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The NBD_ATTR_SOCKETS walk is duplicated in nbd_genl_connect (add sockets)
and nbd_genl_reconfigure (reconnect). Factor out a single helper that
walks the list and calls a callback per fd; with a NULL callback it is a
pure counter, used by a later patch to learn nr_hw_queues before the
device exists. Returns the number of fds walked (>= 0) or a negative
errno; a callback >0 will stops early.
Reviewed-by: Yu Kuai <yukuai@fygo.io>
Signed-off-by: Yang Erkun <yangerkun@huawei.com>
Link: https://patch.msgid.link/20260805122930.57647-7-yangerkun@huawei.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Commit 242a49e5c878 ("nbd: freeze the queue for queue limits updates")
added the freeze to keep in-flight commands from seeing
torn queue_limits. But at startup the capacity is still 0
(invalidate_disk cleared it) and the write cache is off (the previous
patch cleared it on disconnect, and nbd_set_size sets it back only after
the commit), so submit_bio_noacct() rejects any bio before it reaches
the driver and no I/O is in flight. Drop the freeze by checking
capacity and write cache state in nbd_set_size.
Reviewed-by: Yu Kuai <yukuai@fygo.io>
Signed-off-by: Yang Erkun <yangerkun@huawei.com>
Link: https://patch.msgid.link/20260805122930.57647-6-yangerkun@huawei.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
nbd_add_socket() kreallocs config->socks, which a concurrent reader in
nbd_handle_cmd() could UAF; commit b98e762e3d71 ("nbd: freeze the queue
while we're adding connections")froze the queue to block that. But the
freeze costs an RCU grace period on every socket added, and setup adds
them one by one.
After the previous patch, nbd_add_socket() is rejected once nbd->pid is
set, so it only runs during setup. There the capacity is 0 and the
write cache is off (cleared on disconnect by the preceding patch, and
re-enabled only later in nbd_set_size), so submit_bio_noacct() rejects
every bio before it reaches the driver -- non-zero-sector ones via
bio_check_eod(), and flush-only ones via the !bdev_write_cache() branch.
No I/O is in flight, so the freeze is unnecessary.
Reviewed-by: Yu Kuai <yukuai@fygo.io>
Signed-off-by: Yang Erkun <yangerkun@huawei.com>
Link: https://patch.msgid.link/20260805122930.57647-5-yangerkun@huawei.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
An inactive nbd device may refuse any I/O operations. The nbd_config_put
function calls invalidate_disk, which sets the device capacity to zero
to reject all read and write I/O. For zero-sector flush I/O requests
from blkdev_issue_flush, if the write cache is disabled, the zero-sector
flush I/O immediately returns 0 in submit_bio_noacct. However, since
nbd_config_put does not clear the write cache state, an inactive nbd
device might still have the write cache enabled. In this situation,
zero-sector flush I/O will return -EIO because there is no active socket.
Additionally, BLK_FEAT_FUA and BLK_FEAT_ROTATIONAL flags may also remain
stale, resetting all of them ensures consistent behavior.
The limits update uses queue_limits_commit_update() (the non-freezing
variant) because config_refs == 0 here means every fd is closed and recv
threads have drained, so no in-flight I/O can read q->limits concurrently.
Reviewed-by: Yu Kuai <yukuai@fygo.io>
Signed-off-by: Yang Erkun <yangerkun@huawei.com>
Link: https://patch.msgid.link/20260805122930.57647-4-yangerkun@huawei.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
We cannot add a socket to an already running nbd device, the reconfigure
for netlink can only active an inactive socket. But for ioctl path, we can
call NBD_SET_SOCK after NBD_DO_IT, reject this using nbd->pid which has
been setted when NBD_DO_IT. Besides, it is the root cause for commit
b98e762e3d71 ("nbd: freeze the queue while we're adding connections").
Reviewed-by: Yu Kuai <yukuai@fygo.io>
Signed-off-by: Yang Erkun <yangerkun@huawei.com>
Link: https://patch.msgid.link/20260805122930.57647-3-yangerkun@huawei.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The second conditional checking nsock->fallback_index validity is the
logical inverse of the first, so drop it and let execution fall through
naturally. Consolidate the two identical dev_err_ratelimited() + return
paths into a single no_fallback label to reduce duplication.
Reviewed-by: Yu Kuai <yukuai@fygo.io>
Signed-off-by: Long Li <leo.lilong@huawei.com>
Link: https://patch.msgid.link/20260805122930.57647-2-yangerkun@huawei.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The conversion to explicit netlink serialization dropped the
exclude_sensitive parameter from net_conf_to_skb(), so each caller has
to sanitize by hand. Two dump paths were missed:
drbd_nl_get_connections_dumpit() and the volume-less connection branch
of get_one_status(). Neither op carries GENL_ADMIN_PERM, so any
unprivileged local user could read the CRAM-HMAC secret.
Add a net_conf_to_skb_sanitized() wrapper and route all three callers
through it.
Fixes: 8098eeb693c4 ("drbd: replace genl_magic with explicit netlink serialization")
Reported-by: Vivek Parikh <vivek.parikh@breachx.ai>
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
Link: https://patch.msgid.link/20260814151617.73752-1-christoph.boehmwalder@linbit.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Commit 05c3e88488ed ("srcu: Queue sdp->work when the delay timer is
successfully deleted") added a check in cleanup_srcu_struct() if the
call to srcu_barrier() has been made before calling it, which
revealed a missing call to srcu_barrier() before calling
cleanup_srcu_struct(set->srcu). Fix this.
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Link: https://patch.msgid.link/20260812060510.3220294-1-m.szyprowski@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The EOPNOTSUPP stubs for the sub-cap kfuncs live in ext.c under #ifndef
CONFIG_EXT_SUB_SCHED while the real definitions live in sub.c. Move the
stubs into sub.c so all sub kfunc definitions live in one file. Pure code
move, no functional change.
Signed-off-by: Tejun Heo <tj@kernel.org>
|
|
If IORING_URING_CMD_MULTISHOT is set, io_uring_cmd() treats any
non-negative return from ->uring_cmd() as the driver having taken
ownership of the request and returns IOU_ISSUE_SKIP_COMPLETE. But
nothing guarantees that the driver did so, and any handler that just
completes the command inline and returns 0 or a positive result then
leaves the request orphaned, leaking the io_kiocb, the async data,
and the file reference.
The special case isn't needed. ublk returns -EIOCBQUEUED for the
multishot fetch command, which is passed through as-is, and the poll
driven socket timestamp command returns -EAGAIN. Kill it, a multishot
handler that wants to hang on to the request must return -EIOCBQUEUED
or -EAGAIN like any other command.
Fixes: 620a50c92700 ("io_uring: uring_cmd: add multishot support")
Cc: stable@vger.kernel.org
Reported-by: syzbot+a4ccdd7ebf452e4d4701@syzkaller.appspotmail.com
Closes: https://lore.kernel.org/all/6a7a0194.b50370da.49fe0.0056.GAE@google.com/
Link: https://lore.kernel.org/all/20260811115125.1831170-1-vasilisalmpanis@gmail.com/
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_mem_alloc_compound() allocates get_order(size) pages, which rounds a
region size that is not a power of two up to the next order. The pages
past the region are part of the same allocation and cannot be used for
anything else, but io_create_region() accounts reg->size >> PAGE_SHIFT,
so they are never charged against RLIMIT_MEMLOCK. For a ring with 4096
SQ entries and the default CQ size the region is 37 pages while the
allocation is 64.
Account the tail pages together with the region, and fall back to the
exact sized bulk allocation when they do not fit the limit, so a user
close to their limit still gets the region rather than an error.
io_free_region() gives the same amount back, the compound case being the
one that set IO_REGION_F_SINGLE_REF.
Counting how many 4096 entry rings an unprivileged user can create under
a given RLIMIT_MEMLOCK, before and after:
limit (pages) before after
256 2 2
300 2 2
350 3 2
400 3 3
512 5 4
Five rings under a 512 page limit really pin 640 pages.
Fixes: dfbbfbf19187 ("io_uring: introduce concept of memory regions")
Link: https://lore.kernel.org/all/87ik5ncj8d.fsf@mailhost.krisman.be/
Signed-off-by: Ali Ahmet Memis <ali@iusegentoo.com>
Link: https://patch.msgid.link/20260806180044.275543-1-ali@iusegentoo.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The ioctl handlers only test REMOVE_PENDING before entering
mtip_hw_ioctl(). Removal can set that bit immediately afterwards and free
dd->port in mtip_hw_exit() while an ioctl still dereferences it. An already
open block device can reach the handlers while del_gendisk() is in
progress.
Serialize both native and compat ioctls with removal. Set REMOVE_PENDING
before taking the mutex so new callers fail after an in-flight ioctl has
drained, and hold the mutex until the port has been torn down.
Fixes: 88523a61558a ("block: Add driver for Micron RealSSD pcie flash cards")
Signed-off-by: Hongyan Xu <getshell@seu.edu.cn>
Link: https://patch.msgid.link/20260806060441.676-1-getshell@seu.edu.cn
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
__ublk_shmem_remove_ranges() removes matching maple tree ranges in
batches, but first stores each range into a temporary xarray so that the
pages can be unpinned after dropping the maple tree lock.
That temporary xarray is filled under the maple tree lock with
xa_store(..., GFP_ATOMIC). If the store fails before mas_erase(), the
current range is left in the tree and the helper returns false. The
outer ublk_shmem_remove_ranges() loop then immediately retries the same
range. While the atomic allocation keeps failing, the teardown path has
no forward progress.
The issue can be reproduced with radix_tree_node failslab injection after
a SHMEM_ZC buffer has already been registered:
# Kernel config:
# CONFIG_BLK_DEV_UBLK=y
# CONFIG_DEBUG_FS=y
# CONFIG_FAULT_INJECTION=y
# CONFIG_FAULT_INJECTION_DEBUG_FS=y
# CONFIG_FAILSLAB=y
echo 10 > /proc/sys/vm/nr_hugepages
mkdir -p /tmp/htlb
mount -t hugetlbfs none /tmp/htlb
fallocate -l 4M /tmp/htlb/ublk_buf
dev_id=$(kublk add -t null --shmem_zc \
--htlb /tmp/htlb/ublk_buf |
awk -F '[ :]' '/dev id/ {print $3}')
echo 1 > /sys/kernel/slab/radix_tree_node/failslab
echo Y > /sys/kernel/debug/failslab/cache-filter
echo Y > /sys/kernel/debug/failslab/ignore-gfp-wait
echo 1 > /sys/kernel/debug/failslab/interval
echo -1 > /sys/kernel/debug/failslab/times
echo 100 > /sys/kernel/debug/failslab/probability
kublk del -n "$dev_id"
On the unfixed kernel the delete command was still running after 3
seconds. Disabling failslab made it return. The fault-injection stack
showed:
should_failslab
kmem_cache_alloc_lru_noprof
__xas_nomem
__xa_store
xa_store
__ublk_shmem_remove_ranges
ublk_cdev_rel
ublk_ctrl_del_dev
Remove the allocation from the teardown loop. Keep the existing batch
limit, but collect {base_pfn, nr_pages} pairs in a fixed-size stack array.
Once a matching range is found, the range is erased from the maple tree
before dropping the lock, so each successful scan makes progress without
depending on any GFP_ATOMIC allocation.
With the same failslab settings, the fixed kernel completed
"kublk del -n $dev_id" successfully in about 45 ms.
Fixes: 309e02dccf64 ("ublk: avoid unpinning pages under maple tree spinlock")
Signed-off-by: Yao Sang <sangyao@kylinos.cn>
Reviewed-by: Ming Lei <tom.leiming@gmail.com>
Link: https://patch.msgid.link/20260804125736.2011774-1-sangyao@kylinos.cn
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
null_zone_no() does sect >> ilog2(dev->zone_size_sects). When
zone_size_sects is 0, ilog2(0) returns -1, producing shift exponent -1
which UBSAN reports as shift-out-of-bounds.
UBSAN: shift-out-of-bounds in drivers/block/null_blk/zoned.c:21:14
shift exponent -1 is negative
Call Trace:
null_zone_no drivers/block/null_blk/zoned.c:21 [inline]
null_process_zoned_cmd+0xf76/0xf80 drivers/block/null_blk/zoned.c:728
null_handle_cmd drivers/block/null_blk/main.c:1455 [inline]
null_queue_rq+0x8bc/0xe70 drivers/block/null_blk/main.c:1703
__blk_mq_issue_directly block/blk-mq.c:2694 [inline]
blk_mq_try_issue_directly+0x3f4/0x880 block/blk-mq.c:2754
blk_mq_submit_bio+0x20c0/0x2a40 block/blk-mq.c:3208
submit_bio_noacct_nocheck+0x2f4/0xa40 block/blk-core.c:790
block_read_full_folio+0x7a6/0x810 fs/buffer.c:2463
filemap_read_folio+0x12c/0x3a0 mm/filemap.c:2510
read_part_sector+0xb6/0x2b0 block/partitions/core.c:724
adfspart_check_ICS+0xb1/0x960 block/partitions/acorn.c:357
check_partition block/partitions/core.c:143 [inline]
blk_add_partitions block/partitions/core.c:591 [inline]
bdev_disk_changed+0x851/0x17a0 block/partitions/core.c:695
blkdev_get_whole+0x372/0x510 block/bdev.c:751
add_disk_final block/genhd.c:412 [inline]
add_disk_fwnode+0x24b/0x3a0 block/genhd.c:606
null_add_dev+0x130b/0x1d70 drivers/block/null_blk/main.c:2052
nullb_device_power_store+0x240/0x380 drivers/block/null_blk/main.c:501
configfs_write_iter+0x337/0x430 fs/configfs/file.c:229
Syzkaller triggers this by creating a zoned null_blk device via
configfs. The Call Trace shows configfs_write_iter in configfs/file.c
handling a write to power file, which calls nullb_device_power_store in
main.c, which calls null_add_dev in main.c, which calls add_disk in
genhd.c, which triggers partition scan via bdev_disk_changed in
partitions/core.c.
A zoned null_blk device with zone_size 0 should not be legal. Existing
code tries to reject it via is_power_of_2() check in zoned.c and
!zone_size check in main.c, but syzkaller can still reach
null_zone_no() with zone_size_sects 0 via two paths:
1. Direct 0 via configfs: zone_size attribute store in main.c has
NULLB_DEVICE_ATTR(zone_size, ulong, NULL) with no validation callback,
so echo 0 > zone_size succeeds before power store. If zoned is false
at power store time, the !zone_size check in main.c is skipped, and
later zoned set true leaves zone_size 0.
2. Large value overflow: mb_to_sects() in zoned.c does
(sector_t)mb * SZ_1M >> SECTOR_SHIFT which is mb * 2048. If mb is
1UL << 53 (9PB), mb * 2048 overflows 64-bit to 0. The value is
power-of-two so is_power_of_2() passes, but mb_to_sects() returns 0.
Check for zero zone_size explicitly in null_init_zoned_dev() in
zoned.c, returning -EINVAL with "must be non-zero power-of-two".
Check for zero zone_size_sects after mb_to_sects() conversion,
returning -EINVAL for overflow case. Keep defensive check in
null_zone_no() returning 0 for zero sectors to avoid shift out-of-bounds
even if zero slips through.
This change should be safe because zone_size is set once in
null_init_zoned_dev() under device lock and never changes after, and 0
is never valid for a zoned device. Returning -EINVAL at init time fails
device creation early with clear error, while defensive return 0 in
null_zone_no() makes zoned command fail via offline zone check.
No new locking is introduced.
Reported-by: syzbot+abd6a8dca0f2b7726060@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=abd6a8dca0f2b7726060
Link: https://lore.kernel.org/all/6a75205c.01d0871a.3a0d52.0033.GAE@google.com/
Fixes: 8a3cf049af68 ("null_blk: add zoned block device emulation")
Cc: stable@vger.kernel.org
Assisted-by: Hermes:muse-spark-1.2 syzkaller
Signed-off-by: Rik van Riel <riel@surriel.com>
Reviewed-by: Damien Le Moal <dlemoal@kernel.org>
Link: https://patch.msgid.link/20260808114239.69167f68@fangorn
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
https://git.kernel.org/pub/scm/linux/kernel/git/mdraid/linux into for-7.3/block
Pull MD updates from Yu Kuai:
"This pull request contains:
Bug Fixes:
- Protect RAID5 bitmap batching, stripe-cache limits, and reshape
recovery state; avoid failed-device reshape deadlocks, discard hangs,
and PPL use-after-free. (Chen Cheng, Genjian Zhang, Sajal Gupta)
- Recheck spare changes under array suspension before sync to avoid
racing device removal. (Abd-Alrhman Masalkhi)
- Fix RAID1 atomic-write constraints, serialized-device setup, and
takeover I/O freezes. (Abd-Alrhman Masalkhi, Martin Wilck,
Bruce Johnston)
- Fix RAID10 atomic-write failure handling and reshape pool/bio
lifetime bugs. (Abd-Alrhman Masalkhi, Chen Cheng)
- Fix bitmap error recovery, flush/sync accounting, reclaim safety,
teardown, timer, use-after-free, and empty-range bugs, plus stale
clone I/O accounting. (Chen Cheng, Yu Kuai)
- Reject zero-sector RAID5 reshape chunks and correctly round bitmap
ranges for non-power-of-two stripe widths. (Yu Kuai)
- Prevent PF_MEMALLOC_NOIO state from leaking across tasks.
(Chen Cheng)
- Validate bad-block-log shift bounds and skip discard on unsupported
member devices. (Coly Li, Wale Zhang)
- Prevent RAID10 recovery corruption and large-array resync soft
lockups. (Yunye Zhao)
Improvements:
- Add lockless bitmap reshape support for RAID5 and RAID10, including
exact old/new mapping, cache growth, geometry lifecycle, checkpoint
remapping, and bio splitting. (Yu Kuai)
Cleanups:
- Make RAID1 sequential-read hint accesses explicit to suppress false
KCSAN reports. (Chen Cheng)
- Remove redundant RAID10 barrier handling and align badblock range
types. (Abd-Alrhman Masalkhi, Hiroshi Nishida)"
* tag 'md-7.3-20260809' of https://git.kernel.org/pub/scm/linux/kernel/git/mdraid/linux: (53 commits)
md/raid1: don't set array_frozen in raid1_takeover()
md: skip discard on unsupported member devices
md: add cond_resched() to md_do_sync()'s skip path
md/raid10: fix still_degraded being inverted in raid10_sync_request()
md/raid5: split reshape bios before bitmap accounting
md/raid5: wire llbitmap reshape lifecycle
md/raid5: reject llbitmap reshape when md chunk shrinks
md/raid5: add exact old and new llbitmap mapping helpers
md/raid10: split reshape bios before bitmap accounting
md/raid10: wire llbitmap reshape lifecycle
md/raid10: reject llbitmap reshape when md chunk shrinks
md/md-llbitmap: clamp state-machine walks to tracked bits
md/md-llbitmap: remap checkpointed bits as reshape progresses
md/md-llbitmap: don't skip reshape ranges from bitmap state
md/md-llbitmap: add reshape range mapping helpers
md/md-llbitmap: refuse reshape while llbitmap still needs sync
md/md-llbitmap: finish reshape geometry
md/md-llbitmap: track target reshape geometry fields
md/md-llbitmap: grow the page cache in place for reshape
md/md-llbitmap: allocate page controls independently
...
|
|
Not needed, probably since this file was split out.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Anuj Gupta <anuj20.g@samsung.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Link: https://patch.msgid.link/20260810144603.453283-1-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
XFS already defers some writes to a workqueue when transactions are
needed to process the I/O completion. Disable the block layer bio task
completion in this case to avoid a major performance drop.
Fixes: efbde6f9f449 ("iomap: use BIO_COMPLETE_IN_TASK for dropbehind writeback")
Link: https://lore.kernel.org/all/8124341f-3af2-4a16-897d-38db5ab5a9d4@columbia.edu/
Signed-off-by: Tal Zussman <tz2294@columbia.edu>
Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260810-xfs-dontcache-double-defer-v1-1-aea7484b3e49@columbia.edu
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
All block driver code except loop_set_dio() calls
queue_limits_start_update() before it freezes the request queue. Make
loop_set_dio() follow this convention. This patch fixes the following
lockdep complaint:
======================================================
WARNING: possible circular locking dependency detected
7.2.0-rc5-dbg #11 Not tainted
------------------------------------------------------
losetup/2924 is trying to acquire lock:
ffff88816c76da68 (&q->limits_lock){+.+.}-{4:4}, at: loop_set_dio+0x318/0x720 [loop]
but task is already holding lock:
ffff88816c76d430 (&q->q_usage_counter(io)#24){++++}-{0:0}, at: blk_mq_freeze_queue_nomemsave+0x1a/0x30
which lock already depends on the new lock.
Cc: Keith Busch <kbusch@kernel.org>
Fixes: 6c8dec275ccc ("loop: set dma_alignment from the backing file for direct I/O")
Reported-by: syzbot+cc0de396bac84da51919@syzkaller.appspotmail.com
Closes: https://lore.kernel.org/linux-block/6a7d5368.d5f0ebe7.22d851.0013.GAE@google.com/
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Keith Busch <kbusch@kernel.org>
Link: https://patch.msgid.link/d919f5285d16afbec6c51ecdf201692a484566e5.1786637565.git.bvanassche@acm.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
It's not always possible for the user to predict during registration how
much memory zcrx will need to sustain the traffic. Allow to dynamically
add more areas with a new ctrl code ZCRX_CTRL_ADD_AREA.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/43495ca686713b3f0014c9f47c4c0276d704da4d.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Protect __zcrx_create_area() with pp_lock. It's not needed for now,
nobody can take the lock in parallel, but we'll need it for dynamic area
creation for avoiding races with the device dying.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/a667399971120227a28d54ebabc50e707f717bf3.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Currently, we have only a one area per zcrx instance, and struct
io_zcrx_ifq stores a single pointer. To prepare for adding more areas,
replace it with an array of areas.
We'll be creating them at runtime, and the array is protected by 3
locks: ->pp_lock, ->alloc_lock and ->rq.lock. It takes all of them when
switching arrays, and readers should hold either of them.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/d186adfa0ea26f651804cffef77d477f257eb404.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
freelist_lock, which protects slow path allocations, is currently stored
in struct io_zcrx_area. Once we add support for multiple queues, we'll
need a lock in the zcrx ctx, move it there.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/be7feadff1e210861433f99fe44170aa27ec5b33.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Make sure we unmap areas while closing a queue.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/1c41f349f8bcbcaafb17a9c81d4157aa9d30de93.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Until now unmapping and destroying dmabuf were the same thing. To keep
it consistent with non-dmabuf, split it into two separate helpers. Unmap
destroys mappings and attachements as it should, and release only
putting down the dmabuf fd reference.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/bd8c1fd8dac85ce0cf839b389ecd6050761965ca.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
We might want to create an area without having an instance of struct
io_uring_zcrx_ifq_reg. Extract a helper that doesn't have the ifq
registration structure as an argument but takes the buf length
explicitly.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/76751ee4747abb4893058fcaaa36895dfe24a297.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Add zcrx_area_id_to_token() to deduplicate the way the area token is
calculated out of the area index.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/c7752161065a1f7273ce3de9683804dd105c6d53.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Keep skb_shinfo in a local variable so that it doesn't reload it on
every iteration of the loop.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/1e4c864a5ea639803c866aea70b8d5bf558f189f.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
A preparation patch that limits scopes of a couple variables in
io_zcrx_recv_skb() and rename them, it makes it easier to reason about
the code.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/182c140502e7be534caf10360714a4d78f2ce5f4.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_import_area() doesn't modify its struct io_uring_zcrx_area_reg
argument, add const to enforce that, it'll make later modifications
easier.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/2cd4a3e43875576587c900ebdf8f1fb4af266e4a.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
With large rx piages I often see >10 sequential RQEs referring to the
same niov. Instead of putting them one by one, count such RQEs during
parsing and batch refcounting for the niov.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/71f11d5de47553f187737e0e892ab783b03f53b1.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The RQ tail is updated by the user space. Cache it to reduce cache line
bouncing. Refilling now tries to exhaust the previous batch of rqes, but
since it could be too low, the iterator is allowed to recalculate the
rqes to process once after synching the tail value.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/b42d2ed8b0e697110b646573b5da2a5f8e85f92e.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Add a iterator structure and helper functions for the refill queue
processing to avoid polluting io_zcrx_ring_refill() with extra state
and logic once it's extended in following patches.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/5d2933ea7bcf9c19c44e43afb3638d7afb362b3c.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
RQ head and tail are currently put into the same cache line, which can
cause false sharing problems when refill is run on another CPU. Put them
into separate cache lines.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Reviewed-by: Mina Almasry <almasrymina@google.com>
Link: https://patch.msgid.link/f769116a3f5267f06cb9a9a819d482ddea2e0852.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io_zcrx_ring_refill() caps the loop by mixing the max number of
allocated netmems and the number of available RQEs together, which
caps the number of entries to process the pp cache size. As a result,
when niovs are heavily fragmented, the refilling logic allocates only a
small number of niovs per call on average and sometimes even none.
Keep a separate counter for the number of processed RQ entries, which is
capped by a roughly calculated from the page size value to keep the
cache full. And separately break if it allocates enough niovs.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://patch.msgid.link/143cf439299728759eb0a840c866363fe99293f0.1786108672.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
create_worker_cb() reserves an io-wq worker slot only after its
task-work callback runs. If the callback is canceled before then,
io_worker_cancel_cb() still decrements acct->nr_workers. When an
existing worker retires with its creation callback pending, that
worker has already decremented the same account's worker count.
The resulting undercount permits worker creation beyond the account's
configured limit. On an AST2600 OpenBMC system, an unchanged sensor
daemon reached 4,291 threads with the original kernel. With an
equivalent downstream fix, 25 passive samples under its normal
workload showed 6-9 threads.
Decrement nr_workers only when the canceled callback is not
create_worker_cb(). Continuation callbacks still release their reserved
slot, and both callback types retain the existing running-count,
reference-count, and create-state cleanup.
Fixes: 1d5f5ea7cb7d ("io-wq: remove worker to owner tw dependency")
Cc: stable@vger.kernel.org
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Vishnu Razdan <vrazdan@openai.com>
Reviewed-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260811-vrazdan-io-wq-b4-submit-v1-1-719ced16c921@openai.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
ddir was only ever forwarded to io_compat_msg_copy_hdr, which
never looked at it. Drop it.
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260813004022.3514537-4-krisman@suse.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Similar to commit f4eaf8eda89e ("io_uring/rsrc: Drop io_copy_iov in
favor of iovec API"), avoid the custom copy of a single iovec and just
rely on the iovec api. This lets the compat and native paths share the
buffer-select length lookup
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260813004022.3514537-3-krisman@suse.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Similar to commit f4eaf8eda89e ("io_uring/rsrc: Drop io_copy_iov in
favor of iovec API"), avoid custom copy and just rely on the iovec api.
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://patch.msgid.link/20260813004022.3514537-2-krisman@suse.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Disks created via blk_mq_alloc_disk_for_queue() (e.g. SCSI SD disks)
do not have GD_OWNS_QUEUE set. Currently __blk_mark_disk_dead() only
sets QUEUE_FLAG_DYING when GD_OWNS_QUEUE is set, so for such disks
blk_queue_enter() and __bio_queue_enter() cannot detect the dying
state via blk_queue_dying() and remain blocked waiting for I/O that
will never complete after surprise removal.
blk_mark_disk_dead() is the explicit "surprise removal" API -- the
caller has already decided the disk is dead. Setting QUEUE_FLAG_DYING
unconditionally here is appropriate: any in-flight I/O from other
threads should get -ENODEV immediately from blk_queue_enter()
regardless of GD_OWNS_QUEUE ownership.
For disks that already have GD_OWNS_QUEUE set, __blk_mark_disk_dead()
will set the flag again which is harmless.
Fixes: 6f8191fdf41d ("block: simplify disk shutdown")
Cc: stable@vger.kernel.org
Signed-off-by: Lianqin Hu <hulianqin@vivo.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/PUZPR06MB62247E82E66A3ED46CC3E6C7D2DC2@PUZPR06MB6224.apcprd06.prod.outlook.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Since commit 6c6c1fc09de3 ("modpost: require a MODULE_DESCRIPTION()"),
modpost complains that swim3.ko is missing a module description.
WARNING: modpost: drivers/block/swim3.ko: missing MODULE_DESCRIPTION()
Add one to clear up the warning.
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260811-swim3-module-description-v1-1-28398c5a0e32@kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
Add test_params_01.sh for SET_PARAMS. The test checks valid basic
parameters and several invalid parameter cases.
Also cover zoned parameters, including a non-power-of-2 zone size. This
case must fail in SET_PARAMS instead of being accepted and rejected later
when the device is started.
Signed-off-by: Yao Sang <sangyao@kylinos.cn>
Link: https://patch.msgid.link/20260814023226.354288-4-sangyao@kylinos.cn
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
The normal kublk add command goes through device startup. It does not
tell the shell test whether a bad parameter is rejected by SET_PARAMS or
later by START_DEV.
Add a set_params command. It creates a temporary ublk device, sends
SET_PARAMS with the command line parameters, returns the ioctl result,
and deletes the device before START_DEV.
Signed-off-by: Yao Sang <sangyao@kylinos.cn>
Link: https://patch.msgid.link/20260814023226.354288-3-sangyao@kylinos.cn
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
UBLK_F_ZONED uses params.basic.chunk_sectors as zone size. ublk uses
ilog2(chunk_sectors) to get number of zones, so the value must be power
of 2.
If chunk_sectors is 96 and dev_sectors is 96 * 16, userspace asks for
16 zones. But the shift calculation gets 24 zones.
Block layer rejects such zone size when the disk is started. But
SET_PARAMS has already returned success, which is confusing for
userspace. Reject it in SET_PARAMS with other zoned parameter checks.
Fixes: 29802d7ca33b ("ublk: enable zoned storage support")
Signed-off-by: Yao Sang <sangyao@kylinos.cn>
Link: https://patch.msgid.link/20260814023226.354288-2-sangyao@kylinos.cn
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
blk_cgroup_congested() walks the current task's blkcg ancestor chain on
every readahead decision and, once swap is in use, on every anonymous and
shmem folio allocation. The answer is almost always "no", but finding that
out costs two loads per level on two cold cache lines, plus an out-of-line
kthread_blkcg() and an RCU read-side pair. On a fleet profile of hosts
running containers with 5-10 level hierarchies it costs about as much as
all of mutex_lock(), 99.4% of it under __folio_throttle_swaprate().
Gate the walk on a global count of blkcgs with a non-zero
congestion_count. The counter only moves on the 0 <-> 1 transitions of
each blkcg's congestion_count, so the extra atomic stays in the throttle
arm/disarm paths and never appears in steady state. When something is
throttled the counter is non-zero and the walk runs as before.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://patch.msgid.link/20260814165712.510132-4-usama.arif@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
iocg_kick_delay() turns sufficiently large debt into an explicit
block-cgroup delay with blkcg_set_delay(), setting blkg->use_delay to
-1 and incrementing blkcg->congestion_count. Clearing it again depends
on iocg_kick_delay() running from the period timer, the waitq timer or
the issue path.
ioc_pd_free() removes the iocg from active_iocgs and cancels its waitq
timer, and no further bios can arrive, so once it has run nothing is
left which can reduce the debt and clear the delay. The blkcg stays
marked congested for the rest of its life.
blk_cgroup_congested() then returns true for every task in that cgroup
and its descendants: page_cache_sync_ra() cuts readahead to a single
page, page_cache_async_ra() skips it altogether, and
__folio_throttle_swaprate() takes swap_avail_lock and schedules a
throttle on anonymous folio allocation.
Clear it explicitly, after the list removal and the synchronous
hrtimer_cancel() so that neither timer processing nor an I/O path can
re-arm it. The free callback can also see policy data which was never
attached to a blkg, hence the pd->blkg check.
Fixes: 7caa47151ab2 ("blkcg: implement blk-iocost")
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://patch.msgid.link/20260814165712.510132-3-usama.arif@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
io.latency can throttle a group which has no latency target of its own.
When a sibling misses its target, check_scale_change() scales down its
peers, and a peer that reaches queue depth one gets blkcg_use_delay()
called on it on every further scale-down, even with min_lat_nsec == 0.
iolatency_pd_offline() resets the target through
iolatency_set_min_lat_nsec(), which clears the delay only on a nonzero
to zero transition, so it never clears such a peer. Freeing the policy
data then leaves blkg->use_delay set and blkcg->congestion_count
elevated with nothing left that can drop it.
blk_cgroup_congested() then returns true for every task in that cgroup
and its descendants for as long as the cgroup lives: page_cache_sync_ra()
cuts readahead to a single page, page_cache_async_ra() skips it
altogether, and __folio_throttle_swaprate() takes swap_avail_lock and
schedules a throttle on anonymous folio allocation.
Clear the delay in iolatency_pd_free(). By then bio-held blkg
references have drained, or the queue is frozen for policy
deactivation, so check_scale_change() cannot re-arm it. The free
callback can also see policy data which was never attached to a blkg,
hence the pd->blkg check.
Fixes: d70675121546 ("block: introduce blk-iolatency io controller")
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://patch.msgid.link/20260814165712.510132-2-usama.arif@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
