| Age | Commit message (Collapse) | Author |
|---|
|
into clk-pile
Pull clk patches from Brian Masney:
- New clock controller drivers for the Cix Sky1 audio subsystem (AUDSS),
UltraRISC DP1000, and MediaTek MT8173 MFG_TOP, along with their devicetree
bindings. Si549 support was added to the existing si544 driver.
- New clock and reset support for the Aspeed AST2700 PECI controller and Airoha
EN7523 PCIe PERSTOUT reset lines.
- The clk core gains devm_clk_bulk_get_enable() as the mandatory counterpart to
the existing optional variant, and exports
devm_clk_hw_register_composite_pdata() for modular drivers.
- Tegra gets a proper EMC clock implementation for Tegra114, 48MHz pll_p_out1
support needed for UEFI on Surface2, and the Xilinx clocking-wizard gains PLL
charge pump/lock parameter programming during dynamic reconfiguration.
- Bug fixes for a NULL pointer dereference from uninitialized clk_init_data in
the eswin driver, an IO remap leak in the MediaTek pllfh error path, inverted
gate control for MT8135 devapc_ck, a missing OF node put in tegra124-emc on
registration failure, a prepare reference leak in the palmas driver,
unregistered PLLs on MT6735 probe failure, a missing kasprintf NULL check in
pmc_atom, PCIe warm boot instability in the Airoha EN7523 driver, and a
clocking-wizard clock difference detection bug.
- Various cleanups across tegra, st, and mvebu providers to stop misusing the
consumer clock API, along with other minor fixes and improvements.
Signed-off-by: Brian Masney <bmasney@redhat.com>
* tag 'clk-misc-round-two-for-v7.3' of ssh://github.com/masneyb/linux: (45 commits)
clk: tegra: set up proper EMC clock implementation for Tegra114
clk: clocking-wizard: remove 20kHz restriction
clk: clocking-wizard: optimize clock search
clk: clocking-wizard: fix clock difference detection
clk: mediatek: mt8135: Fix inverted gate control for devapc_ck
clk: clocking-wizard: Program PLL CP/RES and lock parameters on reconfig
clk: tegra: support 48MHz clock for pll_p_out1
clk: aspeed: add AST2700 PECI clock
dt-bindings: clock: ast2700: add PECI clock
clk: mediatek: Add mt8173-mfgtop driver
dt-bindings: clock: mediatek: Add mt8173 mfgtop
clk: tegra: clean-up simple provider misuse of the consumer API
clk: st: clean-up simple provider misuse of the consumer API
clk: mvebu: clean-up simple provider misuse of the consumer API
clk: remove conditional return with no effect
clk: Add devm_clk_bulk_get_enable()
clk: en7523: add support for dedicated PCIe PERSTOUT reset
dt-bindings: clock: airoha: Add additional reset for PCIe PERSTOUT
arm64: dts: cix: sky1: add audss cru
reset: cix: add sky1 audss auxiliary reset driver
...
|
|
Pull clk patches picked up by Brian Masney:
Here's various improvements and fixes for the clk subsystem that was posted prior
to the opening of the last merge window.
- Add spread spectrum clock (SSC) support to the clk framework, including a new
assigned-clock-sscs devicetree property and clk_hw_set_spread_spectrum()
API with KUnit tests (Peng Fan)
- Add SCMI clock OEM extensions for the i.MX95 clock driver and introduce a
common SCMI clock header (Peng Fan)
- Add clock, reset, and devicetree bindings for the ESWIN EIC7700 HSP clock and
reset generator (Xuyang Dong)
- Add clk_determine_rate_noop() helper for clock drivers that can do any rate,
and convert existing open-coded implementations across hisilicon, imx, qcom,
renesas, rp1, samsung, scpi, sprd, mediatek phy, and mediatek pmdomain
drivers (Brian Masney)
- Add kernel-doc documentation for struct clk_core and the core clock flags,
and wire up clk identifiers into the documentation build (Brian Masney)
- Fix clk_divider_bestdiv() returning the minimum rate instead of the maximum
rate for large rate requests, with KUnit tests (Lad Prabhakar)
- Fix Nuvoton MA35D1 PLL frequency calculation including ignored div_u64
return values, incorrect PLL_CTL1_FRAC bit field width, and broken
determine_rate logic (Joey Lu)
- Support unique clock names for multi-socket Tegra platforms (Jon Hunter)
- Allow COMPILE_TEST builds for HiSilicon clock drivers (Rosen Penev)
- Various fixes and cleanups from Akari Tsuyukusa, Alexander A. Klimov,
Brian Masney, David Carlier, David Laight, Min zhang, Myeonghun Pak,
Pavel Löbl, Randy Dunlap, Rob Herring, Rosen Penev, Uwe Kleine-König,
William Theesfeld, Xuyang Dong, and Yu-Chun Lin
Signed-off-by: Brian Masney <bmasney@redhat.com>
[sboyd@kernel.org: Fix allmodconfig modpost failure in scmi]
* tag 'clk-misc-for-v7.3' of ssh://github.com/masneyb/linux: (53 commits)
pmdomain: mediatek: mtk-mfg: use clk_determine_rate_noop()
pmdomain: mediatek: airoha: use clk_determine_rate_noop()
phy: mediatek: phy-mtk-hdmi-mt2701: use clk_determine_rate_noop()
clk: sprd: use clk_determine_rate_noop()
clk: scpi: use clk_determine_rate_noop()
clk: samsung: acpm: use clk_determine_rate_noop()
clk: rp1: use clk_determine_rate_noop()
clk: renesas: rzg2l-cpg: use clk_determine_rate_noop()
clk: qcom: smd-rpm: use clk_determine_rate_noop()
clk: qcom: rpmh: use clk_determine_rate_noop()
clk: qcom: rpm: use clk_determine_rate_noop()
clk: imx: scu: use clk_determine_rate_noop()
clk: hisilicon: hi3660-stub: use clk_determine_rate_noop()
clk: add clk_determine_rate_noop()
clk: imx: scu: drop redundant init.ops variable assignment
clk: test: convert constants to use HZ_PER_MHZ
docs: clk: include some identifiers to keep documentation up to date
clk: add kernel docs for struct clk_core
clk: add kernel docs for the core flags
clk: hisilicon: allow COMPILE_TEST builds
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull failfs filesystem from Christian Brauner:
"Add failfs and expose a FD_FAILFS_ROOT sentinel.
This allows userspace to shed their filesystem state completely. A
process with its root or working directory in failfs must anchor every
path lookup at an explicit file descriptor. Absolute paths, absolute
symlinks and AT_FDCWD-relative lookups simply fail.
Failfs is the counterpart to nullfs. nullfs says adds a permanently
empty, immutable directory whose lookups fail with ENOENT but which
can be opened, read, stat'd and mounted upon. Failfs on the other hand
fails every operation. The root cannot be opened at all. A single
instance is mounted during early boot via kern_mount(), which makes it
logically distinct from every mount namespace.
This is accompanied by a new fchroot() system call which makes
chrooting via a file descriptor a first class concept. It's possible
to chroot into failfs as an unprivileged user provided the task has no
new privileges set"
* tag 'vfs-7.3-rc1.failfs' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs:
Documentation: add failfs documentation
selftests/filesystems: add failfs selftests
arch: hookup fchroot() system call
fs: support FD_FAILFS_ROOT in fchroot()
fs: add fchroot()
fs: support FD_FAILFS_ROOT in fchdir()
fs: add failfs
|
|
When open(2) is called with O_CREAT on a path that already exists as a
symlink, over an NFSv3 mount with a cold dcache, the kernel returns
ENXIO instead of following the symlink to its target.
Reproducer script (MNT is an NFSv3 mount, kernel is 7.1-rc6):
MNT=/mnt/export
ln -sf /tmp/target $MNT/mylink
echo 3 | sudo tee /proc/sys/vm/drop_caches # cold dcache
python3 - <<'EOF'
import os
fd = os.open('/mnt/export/mylink', os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o666)
os.close(fd)
EOF
Expected: success (follow symlink, open target)
Actual: OSError: [Errno 6] No such device or address
The bug does not trigger when the dcache is warm (e.g. after a prior
stat(2)), because lookup_open() then finds a positive dentry and skips
atomic_open entirely, leaving symlink resolution to the VFS.
Root cause:
nfs_atomic_open_v23(), registered as inode->i_op->atomic_open for
NFSv3, handles O_CREAT by sending a CREATE UNCHECKED RPC. As
implemented in nfsd3_create_file() (fs/nfsd/nfs3proc.c) and as required
by RFC 1813 (3.3.8), when the name already exists as a non-regular file
the server returns NFS3_OK with the existing object's file handle rather
than NFS3ERR_EXIST causing nfs_do_create() to return 0 with the
dentry now pointing to a symlink.
The code then unconditionally calls finish_open(), which dispatches
through inode->i_fop->open(). Symlink inodes never have i_fop set — the
VFS initialises it to &no_open_fops because POSIX requires open(2) to
follow symlinks, never open them directly. no_open() returns -ENXIO.
Fix:
After nfs_do_create() succeeds, verify the returned inode is a regular
file before calling finish_open(). If the object is not regular, return
finish_no_open(file, NULL) so the VFS follows the symlink through the
normal open path. NULL is passed because nfs_do_create() instantiates
the inode on the dentry already owned by the caller; passing dentry back
would cause atomic_open() to dput() it a second time.
!S_ISREG() is used rather than S_ISLNK() to cover any other non-regular
types a server might return.
Changes in v2:
- Pass NULL to finish_no_open() per Trond's feedback.
Fixes: 7c6c5249f061 ("NFS: add atomic_open for NFSv3 to handle O_TRUNC correctly.")
Link: https://lore.kernel.org/linux-nfs/20260614122911.3485467-1-michael.nemanov@vastdata.com/ (v1)
Signed-off-by: Michael Nemanov <michael.nemanov@vastdata.com>
Tested-by: Michael Nemanov <michael.nemanov@vastdata.com>
[trond.myklebust@hammerspace.com: use d_is_reg() to catch negative dentries]
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
xs_tls_handshake_sync() gives xs_tls_handshake_done() a reference to the
lower transport before submitting the handshake request. On timeout or
signal, the synchronous waiter drops that reference after calling
tls_handshake_cancel().
handshake_req_cancel() returns false when handshake_complete() has
already marked the request complete. In that case the completion callback
can still be running, so dropping the callback-owned reference in the
waiter can free the lower transport before xs_tls_handshake_done() stores
xprt_err or drops its own reference.
If cancellation loses to completion, wait until xs_tls_handshake_done()
signals handshake_done and let the callback release its reference. This
mirrors the server-side handshake lifetime handling and keeps the timeout
or signal return value unchanged.
Fixes: 75eb6af7acdf ("SUNRPC: Add a TCP-with-TLS RPC transport class")
Cc: stable@vger.kernel.org
Assisted-by: Codex:gpt-5
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Reviewed-by: Chuck Lever <cel@kernel.org>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
When nfs4_add_lease() races with a delegation return, it calls
nfs4_delete_lease() to clean up. Previously, it passed priv,
which can legitimately be NULL. Passing a NULL priv eventually
leads to a NULL pointer dereference in generic_setlease().
Fixes: e93a5e9306a5 ("NFSv4: Add support for application leases underpinned by a delegation")
Signed-off-by: Zhansong Gao <zhsgao@hotmail.com>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
nlmclnt_locks_init_private() installs NLM file lock operations even when
nlmclnt_find_lockowner() fails to allocate a lockowner. nlmclnt_proc()
then returns -ENOMEM, but the VFS still tears down the partially
initialized file_lock and calls locks_release_private().
That invokes nlmclnt_locks_release_private(), which dereferences
fl->fl_u.nfs_fl.owner and crashes because the owner was never installed.
Clear fl_ops before attempting to initialize the NLM private state, and
install the NLM lock operations only after a lockowner has been allocated
successfully.
Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
Cc: stable@vger.kernel.org
Signed-off-by: Shuangpeng Bai <shuangpeng.kernel@gmail.com>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
nfs4_server_common_setup() allocates server->delegation_hash_table
first, but server->destroy - the only path that frees the table via
nfs4_destroy_server() - is not assigned until the very end of the
function. If any intermediate step fails (the is_ds_only_client()
check, nfs4_init_session(), nfs4_get_rootfh(), or nfs_probe_server()),
the function returns with server->destroy still NULL, so the caller's
nfs_free_server() skips the destroy callback and the hash table is
leaked (4 KiB per attempt with the default delegation watermark).
This is trivially reachable from userspace: every failed NFSv4 mount
leaks one allocation. A client that persistently retries a mount that
cannot succeed leaks kernel memory without bound. Observed in
production where a Longhorn backup poller retried mount.nfs4 against
an NFSv3-only server roughly 10 times per second, leaking ~3.4 GiB of
unreclaimable slab (kmalloc-rnd-13-4k) per day; the node accumulated
12 GiB of leaked slab before the source was identified via the
kmem:kmalloc tracepoint (call_site=nfs4_delegation_hash_alloc).
Reproducer:
# server exports NFSv3 only (or export path absent for v4)
while :; do mount -t nfs4 <server>:/missing /mnt; done
# watch SUnreclaim in /proc/meminfo grow 4 KiB per iteration
Free the table on the error paths between the allocation and the
assignment of server->destroy.
Fixes: f5b3108e6a14 ("NFS: use a hash table for delegation lookup")
Cc: stable@vger.kernel.org
Signed-off-by: Nate Prodromou <nate@prodromou.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
A flexfiles storage device is tightly coupled to the MDS only when the
decoded ds_versions[0].tightly_coupled flag is set (RFC 8435, sections
2.3 and 4.1). The client currently ignores that flag and treats every
data server as tightly coupled, which breaks I/O to loosely coupled DSes.
Two things force that assumption on an NFSv4.1+ DS:
1) nfs4_set_ds_client() always sets NFS_CS_PNFS on the new client, so
EXCHANGE_ID is sent with EXCHGID4_FLAG_USE_PNFS_DS.
2) nfs4_init_ds_session() then calls is_ds_client() and returns -ENODEV
if the reply does not carry EXCHGID4_FLAG_USE_PNFS_DS.
A loosely coupled DS is just a normal NFS server and does not act in the
pNFS DS role, so the client must not require it to advertise that role.
Thread the ds_versions[0].tightly_coupled flag from the flexfiles driver
down to the DS connect path. When it is false, skip both the NFS_CS_PNFS
flag and the is_ds_client() check. The file layout driver always passes
true because NFSv4.1 file layout data servers use the pNFS DS role.
Signed-off-by: Jeuk Kim <jeuk20.kim@samsung.com>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
flexfiles accepts NFSv4.0 data servers, but two NFSv4 code paths assume
the data server client has a session. Unlike NFSv4.1+, an NFSv4.0 client
has no session (clp->cl_session is NULL; it uses clp->cl_slot_tbl), so
I/O to a v4.0 flexfiles DS oopses:
- nfs4_init_ds_session() dereferences clp->cl_session->session_state
while seeding the DS lease. It also only seeds cl_lease_time when
NFS4_SESSION_INITING is set; without a session that never happens, so
cl_lease_time stays 0 and nfs4_renew_state() busy-loops, requeuing
every 5 seconds. Seed the lease whenever there is no session and
return before touching session state.
- ff_layout_async_handle_error_v4() dereferences
clp->cl_session->fc_slot_table on every DS I/O error. Fall back to the
v4.0 transport slot table (clp->cl_slot_tbl) when there is no session.
Fixes: a7878ca14008 ("nfs: flexfilelayout: remove v3-only data server limitation")
Signed-off-by: Jeuk Kim <jeuk20.kim@samsung.com>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
NFSv4 open state can outlive the file and dentry that created it. This
was observed in production when NFSv4 state recovery, such as after a
server reboot or lease expiration, raced with unmount.
The race requires recovery to hold an open state reference while the last
open file is closed and the filesystem is unmounted, allowing the
superblock's active reference to drop to zero between
refcount_inc(&state->count) and nfs4_put_open_state():
state manager umount
nfs4_run_state_manager()
nfs4_do_reclaim()
nfs4_reclaim_open_state()
refcount_inc(&state->count)
...
close last file
generic_shutdown_super()
"Busy inodes after unmount"
nfs_free_server()
nfs4_put_open_state()
iput(inode)
evict()
nfs_clear_inode()
nfs_zap_acl_cache()
The "VFS: Busy inodes after unmount" warning is the visible symptom of
that lifetime mismatch: superblock teardown proceeds even though the NFS
open state still pins an inode. After umount has freed the server, the
state manager can then run nfs4_put_open_state() for the last open-state
reference. The resulting iput(inode) can evict an NFS inode with freed
server data, causing crashes at nfs_zap_acl_cache(). This can be
reproduced by delaying the reclaim path before nfs4_put_open_state(),
then closing the last file and unmounting the NFS mount.
Pin the superblock while a state owner is active, and drop the pin when
the owner becomes idle again, so the NFS server stays alive until all
open state associated with the owner has been released.
Assisted-by: Codex:GPT-5
Signed-off-by: Jia Zhu <zhujia.zj@bytedance.com>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
__rpc_clnt_remove_pipedir
Normal client creation goes through rpc_setup_pipedir(), which records
clnt->pipefs_sb, but the mount-event path in __rpc_clnt_handle_event()
calls rpc_setup_pipedir_sb() directly and never refreshes that field.
The umount path also removes the directory without clearing
clnt->pipefs_sb.
After a late pipefs mount or any remount, rpc_clnt_remove_pipedir()
compares the current superblock against a stale pipefs_sb pointer and
skips cleanup, leaving pipefs dentries whose inode private data still
points at a freed rpc_clnt, leading to a potential use-after-free during
subsequent rpc_info_open() or rpc_show_info() calls.
Fix this by properly updating clnt->pipefs_sb upon mount events and
clearing it during unmount or failure paths.
Fixes: bfca5fb4e97c ("SUNRPC: Fix RPC client cleaned up the freed pipefs dentries")
Cc: stable@vger.kernel.org
Reported-by: Yuan Tan <yuantan098@gmail.com>
Reported-by: Xin Liu <dstsmallbird@foxmail.com>
Reviewed-by: Ren Wei <enjou1224z@gmail.com>
Assisted-by: Codex:gpt-5.4
Signed-off-by: Luxiao Xu <rakukuip@gmail.com>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
Extend the memory-reclaim-context test used for LOCALIO reads and writes to
the commit (fsync) path. As with data IO, bouncing every commit through the
dedicated !WQ_MEM_RECLAIM nfslocaliod_workqueue is only required when the
submitting context is a memory-reclaim context: nfs_local_run_commit() calls
vfs_fsync_range(), which may flush the underlying filesystem's own
!WQ_MEM_RECLAIM workqueue, and doing so from a WQ_MEM_RECLAIM worker or a
PF_MEMALLOC task trips check_flush_dependency().
The writeback path does exercise this: nfs_write_inode() (the ->write_inode
super_op) runs under wb_workfn on the WQ_MEM_RECLAIM bdi_wq and reaches
nfs_local_commit() via __nfs_commit_inode(), so that case must keep
deferring. Application-context commits -- fsync (nfs_file_fsync), O_DIRECT
(nfs_direct), and copy/clone (nfs42) -- are not in a reclaim context and now
run the fsync inline via nfs_local_defer_io(), avoiding the per-commit
workqueue hop.
Completion (nfs_commit_release_pages -> nfs_commit_end) then runs
synchronously in the submitting context; higher layers already cope with
this, as __nfs_commit_inode() dispatches the commit async and waits for it
separately via wait_on_commit().
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
nfs_local_commit() is reached only through nfs_initiate_commit(), and every
path that supplies its "how" argument has already cleared FLUSH_SYNC:
__nfs_commit_inode() strips it (how &= ~FLUSH_SYNC) before dispatch and does
its own waiting via wait_on_commit(), while the O_DIRECT path passes how=0.
filelayout issues its DS commit with a NULL localio, so it never enters
nfs_local_commit() at all. The FLUSH_SYNC branch has therefore been dead
since it was introduced with commit 70ba381e1a43 ("nfs: add LOCALIO
support").
Remove the never-taken FLUSH_SYNC branch along with the completion plumbing
it was the sole user of: the struct nfs_local_fsync_ctx::done member, its
initialization, and the complete() call in nfs_local_fsync_work(). With the
branch gone the "how" parameter is unused, so drop it from nfs_local_commit()
and its callers. No functional change.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
Every LOCALIO read and write is currently bounced through the dedicated
!WQ_MEM_RECLAIM nfslocaliod_workqueue. That bounce is only actually
required when the submitting context is a memory-reclaim context: LOCALIO
issues IO directly into a stacked local filesystem (e.g. XFS) which may in
turn flush its own !WQ_MEM_RECLAIM workqueue. Doing that from a
WQ_MEM_RECLAIM worker (most importantly writeback's wb_workfn on bdi_wq) or
an explicit PF_MEMALLOC reclaim task trips check_flush_dependency() and
risks a forward-progress deadlock, which is why commit b9f5dd57f4a5
("nfs/localio: use dedicated workqueues for filesystem read and write")
introduced the intermediate workqueue.
Outside of reclaim context -- ordinary application/task submission such as
O_DIRECT or fsync-driven writeback -- the workqueue hop buys nothing and
merely adds a context switch and scheduling latency per IO while discarding
the NFS client's inherent application-context parallelism.
Add current_is_workqueue_mem_reclaim(), which reports whether %current is a
WQ_MEM_RECLAIM worker using the same predicate check_flush_dependency()
warns on. Use it, together with the PF_MEMALLOC check, in the new
nfs_local_defer_io() helper to decide per-IO whether nfs_local_do_read()
and nfs_local_do_write() must defer to nfslocaliod_workqueue or may issue
the IO inline. Buffered writeback continues to bounce (wb_workfn is a
WQ_MEM_RECLAIM worker); O_DIRECT and app-context submission now run inline.
Running nfs_local_call_write() inline is safe: it already saves and
restores current->flags around the PF_LOCAL_THROTTLE|PF_MEMALLOC_NOIO it
sets and scopes the file opener's creds. The async O_DIRECT completion
path is likewise unaffected: when the underlying filesystem returns
-EIOCBQUEUED, the kiocb ki_complete callback (nfs_local_read_aio_complete /
nfs_local_write_aio_complete) can run in bottom-half context and so must
still defer the pgio completion (nfs_local_pgio_release -> rpc_call_done) to
nfsiod_workqueue via nfs_local_pgio_aio_complete(). That completion hop is
independent of how the IO was submitted, and this change leaves it as-is;
only the submission side stops unconditionally hopping through
nfslocaliod_workqueue.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
The delayed delegation return handling added a separate
delegations_delayed list to keep delegations whose return needs to be
retried later. The delayed list is then spliced back to
delegations_return by nfs_server_clear_delayed_delegations(), which also
causes the state manager to retry the delegation return.
However, nfs_end_delegation_return() still moves delayed delegations to
delegations_return instead of delegations_delayed. As a result, the new
delayed list is never populated, nfs_server_clear_delayed_delegations()
always returns false, and NFS4CLNT_DELEGRETURN is not set again to drive
a retry.
Move delayed delegations to delegations_delayed so that the delayed
return path can splice them back to delegations_return and schedule the
retry as intended.
Fixes: 4039fbedcbcb ("NFS: fix delayed delegation return handling")
Signed-off-by: Guangshuo Li <lgs201920130244@gmail.com>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
nfs_symlink() copies the symlink target into a folio before issuing the
SYMLINK RPC. After a successful reply, it caches that folio in the
instantiated inode mapping and assumes that the dentry now names a
symlink.
If the dentry is instantiated with a non-symlink inode, the raw symlink
target folio can be inserted into the wrong mapping. When that inode is
a directory, reclaim or unmount later calls nfs_readdir_clear_array()
through nfs_dir_aops and interprets the symlink target as a readdir
cache array, which can lead to invalid kfree() calls.
A vmcore from a 4.19-based kernel showed the crash when reclaiming a
directory mapping on unmount:
Stack trace:
nfs_readdir_clear_array+0x4d/0x70 [nfs]
page_cache_free_page.isra.35+0x1a/0x90
delete_from_page_cache_batch+0x1cf/0x2c0
truncate_inode_pages_range+0x24d/0x910
[...]
nfs_evict_inode+0x15/0x30 [nfs]
evict+0x115/0x2b0
dispose_list+0x48/0x60
evict_inodes+0x16c/0x1b0
generic_shutdown_super+0x3f/0x120
nfs_kill_super+0x1b/0x40 [nfs]
deactivate_locked_super+0x3f/0x70
cleanup_mnt+0x3b/0x80
The current code still has the same unchecked cache insertion pattern,
so it may be susceptible to the same failure mode.
Verify that the instantiated inode is a symlink before caching the
target folio. If the type is wrong, drop the suspect dentry and skip
the cache insertion while preserving the successful SYMLINK result.
Co-developed-by: Jackie Liu <liuyun01@kylinos.cn>
Signed-off-by: Jackie Liu <liuyun01@kylinos.cn>
Signed-off-by: ZhangGuoDong <zhangguodong@kylinos.cn>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
nfs_inode_remove_request() obtains the folio for the head request via
nfs_page_to_folio(), which returns NULL when the PG_FOLIO flag is not
set on req->wb_head.
The presence of the "if (likely(folio))" check shows the code already
assumes folio can be NULL. However, folio was dereferenced before that
check:
folio = nfs_page_to_folio(req->wb_head);
mapping = folio->mapping; /* deref */
spin_lock(&mapping->i_private_lock);
if (likely(folio)) { /* too late */
folio->mapping is read (and mapping->i_private_lock is taken, and
folio_end_dropbehind(folio) is called outside the check) before folio
is validated, so a NULL folio would crash before the guard is ever
reached, rendering the check useless.
Move the folio->mapping read, the i_private_lock section and the
folio_end_dropbehind() call inside the "if (likely(folio))" block so
the folio is only dereferenced after it has been confirmed non-NULL.
The behaviour is unchanged when folio is non-NULL.
Signed-off-by: Jiangshan Yi <yijiangshan@kylinos.cn>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
Honor the per-file UNCACHEABLE_FILE_DATA attribute by transparently
opening such regular files with O_DIRECT, so reads and writes bypass the
page cache as the attribute requires, without the application having to
request O_DIRECT itself.
This follows the model the specification describes: the attribute is
"similar in intent to O_DIRECT" and clients "retain flexibility in how
they satisfy the requirements" (draft-ietf-nfsv4-uncacheable-files
Section 4.4, "Relationship to Direct I/O"), and its Implementation
Status (Section 6) describes a prototype Linux client that "treats the
attribute as an indication to use O_DIRECT-like behavior for file
access".
Introduce an NFS_CONTEXT_O_DIRECT open-context flag: nfs4_atomic_open()
sets it when the resolved inode has uncacheable_file_data set (and the
open is not O_APPEND), and the open paths nfs_atomic_open() and
nfs4_file_open() apply O_DIRECT to the file when the flag is set.
The I/O mode is thus selected at open time and is not changed for an
already-open file: a later change to the attribute takes effect on the
next open. The specification permits this -- a client that has already
opened a file MAY continue with its existing caching behavior and apply
the updated attribute to subsequent operations (Section 5).
The delegation interaction in Section 4.3 was considered: it permits read
caching to remain when another NFSv4.2 mechanism, such as a delegation,
already ensures a consistent view of the file. That relaxation is
optional ("may remain appropriate") and read-only -- it does not relax
write-behind suppression (Section 4.1) or the WRITE durability invariant
(Section 4.2). This implementation deliberately does not take it: an
uncacheable file is opened O_DIRECT regardless of any delegation held,
which is compliant (read caching is simply suppressed more aggressively
than the Section 4.3 minimum) and avoids decoupling read vs write caching
behind a single open flag. Relaxing reads under a delegation is left as
a possible future optimization.
Section 6 observes the benefit holds "for applications that issue
well-formed I/O requests". That alignment caveat does not constrain the
Linux NFS client's over-the-wire path: the client readily issues
misaligned I/O using O_DIRECT over SunRPC to the remote NFS server. The
only place a fallback from O_DIRECT to buffered I/O for misaligned I/O
applies is NFS LOCALIO (fs/nfs/localio.c), which detects non-DIO-aligned
I/O and falls back internally; that path is unaffected by this change.
Link: https://datatracker.ietf.org/doc/draft-ietf-nfsv4-uncacheable-files/
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
The UNCACHEABLE_FILE_DATA attribute applies only to regular files
(NF4REG); per draft-ietf-nfsv4-uncacheable-files a server MUST reject a
query of it on any other object type with NFS4ERR_INVAL. The previous
commit decodes and tracks the attribute but does not gate it: the bit
rides in the per-server attribute bitmask (server->attr_bitmask) and in
the generic getattr request bitmap (nfs4_fattr_bitmap), so it would be
requested for non-regular objects too -- e.g. a plain directory GETATTR,
a LOOKUP that resolves to a directory, or a CREATE (which only ever makes
non-regular objects). A strict server would fail those compounds.
Gate the client accordingly:
- Only set NFS_INO_INVALID_UNCACHEABLE_FILE_DATA on regular-file inodes.
In particular, drop it from nfs4_update_changeattr_locked()'s
force-revalidation aggregation: that helper only ever runs on
directory inodes (its callers update a directory's change information
after OPEN-create, REMOVE, RENAME and LINK), so it was setting the
file-only bit on directories.
- Gate the request by object type at the choke point
nfs4_bitmap_copy_adjust(), which clears
FATTR4_WORD2_UNCACHEABLE_FILE_DATA unless the target inode is a
regular file (a NULL inode -- unknown object type -- clears it too).
This already covers GETATTR, SETATTR and LINK; route LOOKUP, LOOKUPP
and CREATE through it as well.
- Type-gate nfs4_bitmask_set(), which translates
NFS_INO_INVALID_UNCACHEABLE_FILE_DATA into a request for attr 87 in
the getattr attached to WRITE, CLOSE and DELEGRETURN. WRITE and
CLOSE only ever pass regular files, but DELEGRETURN passes whatever
object held the delegation -- with directory delegation support that
includes directories -- so request attr 87 there only for S_ISREG
inodes.
The bit is kept in server->attr_bitmask (it is server-supported, and OPEN
still requests it via its regular-file-only open_bitmap), so no bespoke
per-data-file bitmask plumbing is needed. The remaining getattr-bearing
compounds are already safe: ACCESS and LAYOUTCOMMIT use
server->cache_consistency_bitmask (no word2 attributes); READDIR does not
encode the bit; and LOOKUP_ROOT, FSINFO, STATFS and PATHCONF use fixed
bitmaps without it.
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
Recognize the NFSv4.2 per-file UNCACHEABLE_FILE_DATA attribute (attr 87,
draft-ietf-nfsv4-uncacheable-files): decode it via GETATTR, track per-
exported-filesystem support, and record on the inode whether a regular
file's data must not be cached. Acting on the attribute (opening such
files O_DIRECT) is done by a subsequent change.
If the NFSv4 server reports a regular file's UNCACHEABLE_FILE_DATA as
true, it indicates the file's data must not be cached; the client records
this in NFS_I(inode)->uncacheable_file_data for use by the I/O paths.
The UNCACHEABLE_FILE_DATA attribute applies only to regular files
(NF4REG); per the draft a server MUST reject a query of it on any other
object type with NFS4ERR_INVAL. A subsequent commit gates the client
accordingly.
Link: https://datatracker.ietf.org/doc/draft-ietf-nfsv4-uncacheable-files/
Signed-off-by: Tom Haynes <loghyr@hammerspace.com>
[snitzer: adapt Tom's original code focused on metadata for ABE]
Co-developed-by: Mike Snitzer <snitzer@hammerspace.com>
Signed-off-by: Mike Snitzer <snitzer@hammerspace.com>
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
When a layout is recalled or revoked the client cancels its in-flight I/O
so the layout can be returned. The metadata server needs to learn that
this I/O to the storage device did not complete, so that it can reconcile
the affected mirror instance (or, if none remains, take other action).
The cancellation completed with -EAGAIN, which ff_layout_io_track_ds_error()
does not recognise: it fell through the switch and recorded nothing, so no
error was reported to the server.
-EAGAIN is overloaded in the RPC layer, so rather than key the reporting on
it, cancel the I/O with -ECANCELED and map that to NFS4ERR_NXIO in
ff_layout_io_track_ds_error() -- the status the client already reports for
the transport errors that leave an in-flight write incomplete. The
cancelled I/O is then reported to the server via LAYOUTERROR / LAYOUTRETURN.
Unlike a genuine transport error, though, we aborted the I/O ourselves and
have no evidence the device is at fault, so once the error is recorded we
skip marking the device unreachable and forcing a further layout return.
The retry disposition is unchanged from the original -EAGAIN cancellation:
both NFS4ERR_NXIO and -ECANCELED are no-ops in ff_layout_async_handle_error(),
which still resets the I/O to pNFS (or the MDS), so it is re-driven as before.
Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
When the metadata server recalls a layout with clora_changed FALSE, the
layout is not changing and the client may complete its modified writes to
the storage devices before returning the layout (RFC 8881, Section
20.3.3). Only when clora_changed is TRUE -- the server is restriping, or
a storage device has failed -- should the client stop writing to the
storage devices and redirect through the metadata server.
Since commit b739a5bd9d9f ("NFSv4/flexfiles: Cancel I/O if the layout is
recalled or revoked") the client cancels in-flight I/O on every recall,
regardless of clora_changed. For an unchanged recall this abandons
writes whose data may already have reached the storage device; such a
write can then land after the LAYOUTRETURN, which the server sees as a
write without a layout.
Pass the recall's clora_changed value through
pnfs_mark_matching_lsegs_return() and only cancel in-flight I/O when the
layout is actually changing. When it is not, the existing deferred
return path waits for the in-flight writes to drain before sending the
LAYOUTRETURN. Other callers, which are tearing down or returning the
layout for their own reasons, continue to cancel as before.
Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
A CB_LAYOUTRECALL carries the clora_changed flag (RFC 8881, Section
20.3.3), which tells the client whether the server is changing the
layout (and therefore whether the client should flush modified data to
the storage devices before returning, or stop writing to them and go
through the metadata server). The client decodes this into
cbl_layoutchanged, but it is otherwise invisible.
Give nfs4_cb_layoutrecall_file its own event definition and report
clora_changed, so the intent of a recall can be observed in a trace.
Signed-off-by: Benjamin Coddington <bcodding@hammerspace.com>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
bl_parse_concat() and bl_parse_stripe() allocate a child device array and
then parse each child in turn. If parsing a child fails, the failed child is
not counted in nr_children and the parent may be left with a children array
that bl_free_device() will not release when nr_children is zero.
Release the failed child and the already parsed children before returning the
error. Also make bl_free_device() release the child array whenever the
children pointer is set, so that partially initialised concat or stripe
devices are cleaned up correctly.
bl_parse_scsi() can also fail after assigning d->bdev_file and dropping the
file reference. Clear the pointer after fput() so that an outer cleanup path
does not put it again.
Fixes: 5c83746a0cf2 ("pnfs/blocklayout: in-kernel GETDEVICEINFO XDR parsing")
Signed-off-by: ZhangGuoDong <zhangguodong@kylinos.cn>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
nfs4_alloc_client() allocates an NFSv4.0 callback identifier before it
finishes setting up the client. If any later initialization step fails,
the error path frees the nfs_client directly with nfs_free_client(). That
bypasses nfs_put_client(), which is where the callback IDR entry is
removed during normal teardown.
A failed allocation can therefore leave cb_ident_idr pointing at a freed
nfs_client. A later NFSv4.0 callback lookup by cb_ident would find the
stale pointer and take a reference to it.
Make the callback IDR removal helper callable by the allocation failure
path, and remove the callback identifier before freeing the client.
This was found by a local static-analysis checker for publish-before-free
lifetime bugs and confirmed by manual inspection.
Fixes: f4eecd5da342 ("NFS implement v4.0 callback_ident")
Signed-off-by: Ruoyu Wang <ruoyuw560@gmail.com>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
Commit 8236b0ae31c83 ("bdi: wake up concurrent wb_shutdown() callers.")
introduces the clear_and_wake_up_bit() helper as a wrapper for the
common clear -> barrier -> wake up bitops sequence.
The file pnfs.c has several helpers with identical contents. Thus they
are replaced with the more recent clean_and_wake_up_bit() global helper
which describes accurately its effects at the call and still specifies
the cleared bit. This also homogenizes the code with other subsystems.
Since the helpers are no longer used after this, they can be safely
removed.
Suggested-by: Agatha Isabelle Moreira <code@agatha.dev>
Link: https://kernelnewbies.org/Beginner%20Cleanup%20and%20Refactor%20Tasks%20by%20Agatha%20Isabelle%20Moreira#task_007
Fixes: d67ae825a59d ("pnfs/flexfiles: Add the FlexFile Layout Driver")
Signed-off-by: Arnaud Bonnet <abo@medichon.fr>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
Commit 8236b0ae31c83 ("bdi: wake up concurrent wb_shutdown() callers.")
introduces the clear_and_wake_up_bit() helper as a wrapper for the
common clear -> barrier -> wake up bitops sequence.
Use the helper in nfs_clear_invalid_mapping as inode.c already relies
on functions from <linux/wait_bit.h> and to homogenize with other
subsystems.
Suggested-by: Agatha Isabelle Moreira <code@agatha.dev>
Link: https://kernelnewbies.org/Beginner%20Cleanup%20and%20Refactor%20Tasks%20by%20Agatha%20Isabelle%20Moreira#task_007
Fixes: d529ef83c355 ("NFS: fix the handling of NFS_INO_INVALID_DATA flag in nfs_revalidate_mapping")
Signed-off-by: Arnaud Bonnet <abo@medichon.fr>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull efs and freevxfs removal from Christian Brauner:
"This removes the EFS and freevxfs filesystems:
- EFS was the read-only on-disk format SGI used on IRIX before XFS
- freevxfs provided compatibility with various old-school Unix
systems from the 1990s and was fun 25 years ago. Today it mostly
serves as fodder for automated bug checkers. There has been only
one known user and contributor in the last 15 years"
* tag 'vfs-7.3-rc1.efs' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs:
efs: Remove EFS
* tag 'vfs-7.3-rc1.freevxfs' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs:
freevxfs: remove the driver
|
|
The abort_on_kill path in request_wait_answer() calls fuse_abort_conn()
and returns without waiting for FR_FINISHED. If fuse_dev_do_write() is
concurrently processing the same request (FR_LOCKED set), the caller
frees req->args while it is still being accessed, causing a
use-after-free.
Fix this by jumping to the existing wait_event(FR_FINISHED) instead of
returning early. The wait will not hang because fuse_abort_conn()
ensures all requests are ended.
Reported-by: syzbot+d6540a3fa1626e11360d@syzkaller.appspotmail.com
Fixes: 204aa22a686b ("fuse: abort on fatal signal during sync init")
Cc: stable@vger.kernel.org
Assisted-by: Claude:claude-opus-4-6
Signed-off-by: Rochan Avlur <rochan.avlur@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull binfmt updates from Christian Brauner:
"This contains a bunch of work for binfmt_misc. It fixes a bunch of
old bugs, reworks the locking, and then extends the format registry
so a binary type can be matched programmatically and its interpreter
computed per exec instead of being a fixed string recorded at
registration time.
This allows nixos and other to e.g., implement relocatable binaries
meaning the interpreter/dynamic loader can be determined
programatically, say found relative to the binary. The mechanism is
flexible and can support other policies:
- Handler lookup is now an rcu walk. An exec that matches no
binfmt_misc entry should now never write to a shared cacheline
- remove the VERBOSE_STATUS and USE_DEBUG compile time toggles
- convert the entry file to a seq_file which simplifies things quite
a bit and kills a lot of custom logic
- make flags proper enums
- rename struct Node to binfmt_misc_entry
- allow entries to be removed with unlink(2)
- Add the ability to attach bpf programs to binfmt_misc entries so
it's possible to dynamically choose the execution environment such
as the loader or interpreter on a per binary basis.
A handler is an instance of a binfmt_misc_ops struct_ops with a
->match() and a ->load() program. match() decides from the entry
lookup walk whether the handler applies under the same
registration-order. It can read file content as needed not only the
prefetched 256 bytes in bprm->buf.
load() then selects the interpreter and stages it through the new
bpf_binprm_set_interp(), bpf_binprm_set_interp_arg() and
bpf_binprm_set_flags() kfuncs.
Handlers are published in a registry keyed by the registering
task's user namespace and activated through the existing text
interface with a new 'B' type carrying the handler name:
echo ':origin:B::::nix:' > /proc/sys/fs/binfmt_misc/register
The permission and namespacing model is unchanged. Activating a
handler requires the same write access to an instance as any other
registration. A container mounting its own instance escapes the
host's entries exactly as before. The computed interpreter is
opened with open_exec() under the caller's credentials and goes
through full LSM vetting as the next binprm level. A program can
only ever redirect the caller to something the caller could exec
anyway.
- Two dispatch modes are added. So far the chosen interpreter owns
the whole process identity (argv[0], /proc/pid/cmdline,
/proc/self/exe all name interpreter information). So relocatable
find the dynamic linker instead. Also a binary passed to execveat()
as an inaccessible O_CLOEXEC fd cannot run at all and gdb trips
because AT_ENTRY and AT_PHDR do not match the exe file. So PIE
symbols are unrelocated.
This adds transparent dispatch which allows the interpreter to load
the binary through AT_EXECFD and leaves the argument vector exactly
as the caller built it and labels mm->exe_file and comm with the
binary. It also raises the AT_FLAGS_TRANSPARENT_INTERP aux vector
bit. The interpreter keeps control of mapping the binary.
The second mode is loader substitution. This allows a binary to be
executed natively and only the interpreter to be changed.
- Last, interpreters can be bound at registration time. Each
interpreter is opened by its own write with the credentials the
entry file was opened with. The program picks one per exec with
bpf_binprm_select_interp().
Ucounts are used to properly account for pre-opened interpreters
via /proc/sys/user/max_binfmt_misc_interpreters"
* tag 'vfs-7.3-rc1.binfmt' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (63 commits)
binfmt_misc: document the pre-opened interpreter limit
selftests/exec: test the pre-opened interpreter limit
binfmt_misc: correctly account pre-opened interpreters
binfmt_misc: document interpreters bound by a 'B' entry
selftests/exec: test interpreters bound to a 'B' entry
binfmt_misc: let a 'B' entry bind its interpreters
binfmt_misc: carry pre-opened interpreters in struct binfmt_misc_interp
selftests/exec: share the bpf handler preconditions
binfmt_misc: document registering an entry disabled
selftests/exec: test registering an entry disabled
selftests/exec: let binfmt_flag_supported() return a bool
selftests/exec: check that a binfmt_misc instance cannot be pinned
binfmt_misc: let a register string create an entry disabled
binfmt_misc: document loader substitution
selftests/exec: test binfmt_misc loader substitution
binfmt_misc: let a bpf handler request loader substitution
binfmt_misc: add the 'L' loader substitution flag
binfmt_elf_fdpic: consume a stashed PT_INTERP substitute
binfmt_elf: consume a stashed PT_INTERP substitute
exec: carry a PT_INTERP substitute in struct linux_binprm
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull misc ipc and core updates from Christian Brauner:
- reject mq_notify() with a zero signal number
- fix coding style in the exit path
* tag 'ipc-7.3-rc1.misc' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs:
mqueue: reject mq_notify with signo 0
* tag 'kernel-7.3-rc1.misc' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs:
kernel: exit: fix coding style missing spaces
|
|
The dentry_tree_work is not exported, so make it static
to remove the followign sparse warning:
fs/fuse/dir.c:37:21: warning: symbol 'dentry_tree_work' was not declared. Should it be static?
Signed-off-by: Ben Dooks <ben.dooks@codethink.co.uk>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
|
|
Each cgroup_iter_memcg subtest touches 1024 pages and expects the matching
memcg counter to be non-zero. On a host with many CPUs it reads zero
instead:
test_anon:FAIL:final anon mapped val: actual 0 <= expected 0
memcg stats are cached per-cpu and only become visible once the periodic
flusher runs (FLUSH_TIME, 2s), or once pending updates cross
MEMCG_CHARGE_BATCH * num_online_cpus(). That threshold is 512 pages at 8
CPUs but 8192 at 128, so a single pass no longer reaches it and
bpf_mem_cgroup_flush_stats() returns without flushing anything.
Retry the stat generation, sleeping in between, so that a flusher cycle is
always covered. Sleep before dropping the mapping, so that a flusher cycle
landing in the sleep observes the mapped state. nr_anon_mapped and
nr_file_mapped are rmap gauges, and unmapping first would post a matching
negative delta for the flusher to aggregate to a net zero.
test_file asserts on both nr_file_pages and nr_file_mapped, which have
different lifetimes, as page cache pages outlive the mapping. Retry
while either one is still zero.
Fixes: 6bce6ddbe634 ("bpf: selftests: selftests for memcg stat kfuncs")
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260814232017.2839959-1-andrii@kernel.org
|
|
Add documentation for fuse over io-uring usage of buffer pools and
zero-copy.
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
|
|
Implement zero-copy in fuse io-uring to eliminate memory copies between
the application, kernel, and server for read/write operations. The
server can directly access client pages or page cache folios without
copying data through an intermediary buffer. When a fuse request arrives,
the kernel registers the relevant pages into a sparse slot in the
server's io_uring registered buffer table. The server can then operate
on these pages directly using io-uring fixed buffer operations (eg
read_fixed/write_fixed) and the kernel unregisters these pages when the
request completes. Non-page-backed args (eg op out headers) will go
through the payload buffer as normal. The server can specify which open
files should have their reads/writes go through zero-copy, by setting
the FOPEN_IO_URING_ZERO_COPY flag when servicing opens.
This requires CAP_SYS_ADMIN and bufpools. This is gated behind
CAP_SYS_ADMIN because zero-copy allows the server direct access to the
client's underlying pages, rather than operating on an intermediary
buffer that the contents of the client's pages were copied into or on
page cache folios.
The request flow for the zero-copy direct-io write path (client writes
data, server reads it) is as follows:
=======================================================================
| Kernel | FUSE server
| |
| "write(fd, buf, 1MB)" |
| |
| >sys_write() |
| >fuse_file_write_iter() |
| >fuse_send_one() |
| [req->args->in_pages = true] |
| [folios hold client write data] |
| |
| >fuse_uring_copy_to_ring() |
| >copy_header_to_ring(IN_OUT) |
| [memcpy fuse_in_header] |
| >copy_header_to_ring(OP) |
| [memcpy write_in header] |
| |
| >fuse_uring_args_to_ring() |
| >setup_fuse_copy_state() |
| [skip_folio_copy = true] |
| |
| >fuse_uring_set_up_zero_copy() |
| [folio_get for each client folio] |
| [build bio_vec array from folios] |
| >io_buffer_register_bvec() |
| [register pages at
ent->zero_copy_index] |
| [ent->zero_copied = true] |
| |
| >fuse_copy_args() |
| [skip_folio_copy => return 0 |
| for page arg, skip data copy] |
| |
| >copy_header_to_ring(RING_ENT) |
| [memcpy ent_in_out] |
| >io_uring_cmd_done() |
| |
| | [CQE received]
| |
| | [issue io_uring READ at
| | ent->zero_copy_index]
| | [reads directly from
| |client's pages (ZERO_COPY)]
| |
| | [write data to backing
| | store]
| | [submit COMMIT AND FETCH]
| |
| >fuse_uring_commit_fetch() |
| >fuse_uring_commit() |
| >fuse_uring_copy_from_ring() |
| >fuse_uring_req_end() |
| >io_buffer_unregister(ent->zero_copy_index) |
| [unregister pages from index] |
| >fuse_zero_copy_release() |
| [folio_put for each folio] |
| [ent->zero_copied = false] |
| >fuse_request_end() |
| [wake up client] |
The zero-copy read path is analogous.
Some requests may have both page-backed args and non-page-backed args.
For these requests, the page-backed args are zero-copied while the
non-page-backed args are copied to the buffer selected from the buffer
pool:
zero-copy: pages registered via io_buffer_register_bvec()
non-page-backed: copied to payload buffer via fuse_copy_args()
For a request whose payload is zero-copied, the
registration/unregistration path looks like:
register: fuse_uring_set_up_zero_copy()
folio_get() for each folio
io_buffer_register_bvec(ent->zero_copy_index)
unregister: fuse_uring_req_end()
io_buffer_unregister(ent->zero_copy_index)
-> fuse_zero_copy_release() callback
folio_put() for each folio
Please note that on abort for in-flight zero-copied requests that have
been sent to userspace, the registered bvec slot remains occupied and
its folios remain pinned until the io-uring ring is destroyed, at which
point io-uring unregisters all buffers and the fuse_zero_copy_release()
callback drops the folio references. Unregistering at teardown would
require operating on the ring context directly, whose validity is hard
to ascertain; this is deemed not worth the complexity for the abort
race, since everything is freed when the ring is torn down.
The throughput improvement from zero-copy depends on how much of the
per-request latency is spent on data copying vs backing I/O. The gain
comes from eliminating the payload-buffer memcpy, but accessing the
zero-copied pages requires the server to issue the read/write as an
IORING_OP_READ/WRITE_FIXED operation. The benefit is largest when the
mempcy is a meaningful fraction of per-request latency while backing i/o
is still noticable enough that the extra io-uring op's overhead doesn't
dominate.
Benchmarked with passthrough_hp (--nopassthrough, q_depth=8) on a
2-socket Intel Xeon Gold 6138 (40 cores / 80 threads), using fio (sync
engine, bs=1M, O_DIRECT, numjobs=2, 30s run + 10s ramp, 3 runs) where
direct-I/O throughput is against a RAM-backed (tmpfs) source (backing
I/O is not the bottleneck):
baseline registered-buf zero-copy (zc vs base)
direct read ~5.1 GB/s ~5.4 GB/s ~8.9 GB/s (+75%)
direct write ~3.4 GB/s ~4.8 GB/s ~5.1 GB/s (+50%)
Reads end up higher than writes because the backing store reads faster
than it writes (the baseline shows the same read>write gap, and the raw
device does too). On a device-bound NVMe (~2 GB/s reads) the read gain
shrinks to ~10-16% (and no measurable gains for writes), as backing I/O
rather than the eliminated copy dominates latency.
The benefit overall scales with how much of the
per-request latency is the data copy versus backing I/O.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
|
|
Allow servers to use a buffer pool that is also registered through
io-uring. When the server registers a buffer pool with io-uring, the
pages backing the pool are pinned upfront. This eliminates the overhead
of pinning/unpinning user pages and translating virtual addresses per
i/o request. This also allows servers to use the same registered memory
for subsequent backing store I/O (eg read_fixed/write_fixed), keeping
data in the same pinned pages without additional pinning or mapping
overhead required.
To use this, the server needs to set the FUSE_URING_REGISTERED_BUFPOOL
flag when adding a bufpool through the FUSE_IO_URING_CMD_ADD_BUFPOOL
cmd. For every sqe submitted (including the one for adding the bufpool),
it should set sqe->uring_cmd_flags to include IORING_URING_CMD_FIXED,
and pass in the index where the registered bufpool resides to
sqe->buf_index.
Benchmarked with passthrough_hp (--nopassthrough, q_depth=8) on a
2-socket Intel Xeon Gold 6138 (40 cores / 80 threads), using fio (sync
engine, bs=1M, O_DIRECT, numjobs=2, 30s run + 10s ramp, 3 runs) where
direct-I/O throughput is against a RAM-backed (tmpfs) source (backing
I/O is not the bottleneck):
baseline registered buffers
direct read ~5.1 GB/s ~5.4 GB/s (+~5%)
direct write ~3.4 GB/s ~4.8 GB/s (+~45%)
Registered buffers bring up the write path speed up closer to speed of
reads. There isn't much improvement for reads because it is already fast
enough where it's at the copy-bound ceiling (surpassing that requires
doing zero-copy). On a device-bound NVMe though, the differences are
within noise, as backing I/O dominates per-request latency.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
|
|
Right now, ents and buffers are tightly coupled in fuse io-uring where
each entry has its own dedicated payload buffer, requiring N buffers for
N entries where each buffer must be large enough to accomodate the
maximum payload size. This is suboptimal as most request types (lookup,
open, release, getattr, etc) require vastly less bytes than the maximum
payload size and some requests (unlink, rmdir, fsync, flush, etc) do not
require payload buffers at all.
Instead of requiring a 1:1 coupling between ents and payload buffers,
allow the server to pass in a buffer pool (a contiguous chunk of memory)
that the kernel will use as it wishes for servicing ents/requests.
Entries only reserve a "buffer" from the pool while actively processing
a request that requires a payload buffer. This decoupling and letting
the kernel delegate memory from the pool for requests allows the kernel to
optimize memory usage and reduces the memory usage requirements needed
to use fuse-over-io-uring.
A pool is registered per queue with the new
FUSE_IO_URING_CMD_ADD_BUFPOOL command. The server passes the pool's base
address and length in fuse_uring_cmd_req.bufpool.{uaddr,len}.
Internally, the kernel splits the region into buffers of
ring->max_payload_sz bytes each (nr_bufs = pool len / max_payload_sz). A
queue commits to a payload mode on first use: registering an entry that
carries its own payload selects the legacy per-entry mode, while
ADD_BUFPOOL selects pool mode. The two are mutually exclusive, so
ADD_BUFPOOL must be issued before any payload-carrying entries are
registered on that queue. The queue must have been created before the
bufpool is added, through the FUSE_IO_URING_CMD_ADD_QUEUE command.
The kernel tracks free buffers with a bitmap (a set bit marks a free
buffer). On dispatch, a request that needs a payload claims a free
buffer (find_first_bit + clear). A request that needs none claims
nothing. The buffer's byte offset within the pool is reported to the
server in the new fuse_uring_ent_in_out.offset field so that the server
can locate the payload. On completion the buffer is returned to the pool
or reused directly if the next request on that entry also has a payload.
The FUSE_HAS_IO_URING_BUFPOOL flag advertises kernel support to the
server for bufpools.
Buffer pool request flow
~~~~~~~~~~~~~~~~~~~~~~~~
| Kernel | FUSE daemon
| |
| [request arrives] |
| [claim a free pool buffer] |
| >fuse_uring_select_buffer() |
| [copy headers to ring] |
| [copy payload to buffer] |
| [report buffer offset in ent_in_out] |
| >io_uring_cmd_done() |
| | [read headers]
| | [read/write payload at offset]
| | [process request]
| | >io_uring_submit()
| | COMMIT_AND_FETCH
| >fuse_uring_commit_fetch() |
| [copy reply from ring] |
| [return buffer to the pool] |
| >fuse_uring_recycle_buffer() |
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
|
|
fuse-over-io-uring queues are currently created lazily, as a side effect
of the first FUSE_IO_URING_CMD_REGISTER command for a given qid. This
ties queue creation to entry registration.
Add a FUSE_IO_URING_CMD_ADD_QUEUE command so a server can create a queue
explicitly, decoupling queue setup from entry registration. This is
additionally a prerequisite for FUSE_IO_URING_CMD_ADD_BUFPOOL, which
attaches a buffer pool to an existing queue and therefore needs the
queue to have been created first.
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
|
|
Currently, the connection's fuse_ring is created lazily on the first
FUSE_IO_URING_CMD_REGISTER command. A server registers entries from one
thread per queue (one per CPU) and those threads issue their first
REGISTER command concurrently. They then race to create the single
per-connection fuse_ring, which required open-coded handling in
fuse_uring_create() to detect and protect against concurrent creations.
Decouple fuse_ring creation from ent registration and move it to
FUSE_INIT reply processing after a server has negotiated and set
FUSE_OVER_IO_URING. The ring is published before the connection is
marked initialized. fuse_uring_register() no longer creates the ring and
it instead uses the ring set up at init time.
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
|
|
Rename IO_IMU_DEST and IO_IMU_SOURCE to IO_BUF_DEST and IO_BUF_SOURCE
and export it so subsystems may use it.
This is needed by the io_buffer_register_bvec() path for callers who may
need the buffer to be both readable and writable.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Link: https://patch.msgid.link/20260612184840.4058966-5-joannelkoong@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
|
|
Add io_buffer_register_bvec() for registering a bvec array.
This is a preparatory patch for fuse-over-io-uring zero-copy.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Caleb Sander Mateos <csander@purestorage.com>
Link: https://patch.msgid.link/20260612184840.4058966-4-joannelkoong@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
|
|
Split the main initialization logic in io_buffer_register_request() into
a helper function.
This is a preparatory patch for supporting kernel-populated buffers in
fuse io-uring, which will be reusing this logic.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Caleb Sander Mateos <csander@purestorage.com>
Link: https://patch.msgid.link/20260612184840.4058966-3-joannelkoong@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
|
|
Currently, io_buffer_register_bvec() takes in a request. In preparation
for supporting kernel-populated buffers in fuse io-uring (which will
need to register bvecs directly, not through a struct request), rename
this to io_buffer_register_request().
A subsequent patch will commandeer the "io_buffer_register_bvec()"
function name to support registering bvecs directly.
Rename io_buffer_unregister_bvec() to a more generic name,
io_buffer_unregister(), as both io_buffer_register_request() and
io_buffer_register_bvec() callers will use it for unregistration.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Caleb Sander Mateos <csander@purestorage.com>
Link: https://patch.msgid.link/20260612184840.4058966-2-joannelkoong@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
|
|
Add SoC ID for Qualcomm Snapdragon SDM850.
Signed-off-by: David Heidelberg <david@ixit.cz>
Reviewed-by: Konrad Dybcio <konrad.dybcio@oss.qualcomm.com>
Acked-by: Krzysztof Kozlowski <krzysztof.kozlowski@oss.qualcomm.com>
Link: https://lore.kernel.org/r/20260627-sda850-v2-1-44bf46ade42e@ixit.cz
Signed-off-by: Bjorn Andersson <andersson@kernel.org>
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
|
|
The existing link redirects to a generic page. Update the link to the
specification document.
Signed-off-by: Ninad Naik <ninadnaik07@gmail.com>
Link: https://patch.msgid.link/20260321140212.5026-1-ninadnaik07@gmail.com
Signed-off-by: Takashi Iwai <tiwai@suse.de>
|
|
Enable the kptr_xchg_inline functional test on LoongArch64 now that the
BPF JIT can inline bpf_kptr_xchg() with correct memory ordering.
Acked-by: Hengqi Chen <hengqi.chen@gmail.com>
Acked-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Tested-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Chenguang Zhao <zhaochenguang@kylinos.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
|
|
Implement arch_bpf_stack_walk() on top of the ORC unwinder within
the LoongArch BPF JIT backend to provide generic BPF stack walking
capabilities.
This function is required by advanced BPF features, including timed
may_goto timeout tracing and BPF exceptions. It will be invoked in
the BPF core subsystem unwinding paths: bpf_prog_find_from_stack(),
bpf_stream_stage_dump_stack(), and bpf_throw().
Co-developed-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: George Guo <guodongtai@kylinos.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
|
|
Implement arch_bpf_timed_may_goto() support and advertise it through
bpf_jit_supports_timed_may_goto() so the verifier lowers may_goto into
the timed variant: instead of a fixed iteration counter, the loop is
bounded by a wall-clock timeout maintained in a per-loop stack slot.
arch_bpf_timed_may_goto() uses a custom calling convention: the verifier
passes the count/timestamp stack offset in BPF_REG_AX and expects the
updated count back in the same register. The JIT call path therefore can
skip the usual 'BPF_REG_0 = C return value' move for this helper.
Acked-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Tested-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: George Guo <guodongtai@kylinos.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
|
|
Add support for the internal-only BPF_MOV instruction that resolves the
absolute addresses of the per-CPU data from their per-CPU offsets. This
instruction is used only for internal inlining optimizations between the
BPF verifier and the JITs (e.g. inlining bpf_get_smp_processor_id() and
per-CPU map lookups).
LoongArch keeps the per-CPU offset of the current CPU in $r21 register
(a.k.a. __my_cpu_offset), so resolving a per-CPU address only requires
adding $r21 to the source register holding the per-CPU offset. Advertise
the capability via bpf_jit_supports_percpu_insn().
Acked-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Tested-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: George Guo <guodongtai@kylinos.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
|
