diff options
| author | Linus Torvalds <torvalds@linux-foundation.org> | 2026-08-25 07:59:44 -0700 |
|---|---|---|
| committer | Linus Torvalds <torvalds@linux-foundation.org> | 2026-08-25 07:59:44 -0700 |
| commit | 9cebfe6504488198b012e746bc6b313f88b95439 (patch) | |
| tree | 8fc504cea938827aba411494141c266f448c39bd | |
| parent | ce14fe4cd756d2ad75a1f5b53816872b4e69f7cc (diff) | |
| parent | 34b5c4a6e4fb9dbb3f9d87f3b0fb0372105c8302 (diff) | |
| download | linux-9cebfe6504488198b012e746bc6b313f88b95439.tar.gz linux-9cebfe6504488198b012e746bc6b313f88b95439.zip | |
Merge tag 'fuse-update-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse
Pull fuse updates from Miklos Szeredi:
- Improve performance of the io-uring transport by introducing buffer
pools and zero-copy (Joanne)
- Fix lots of bugs (Baokun Li)
- Fix io-uring initialization issues (Joanne, Bernd)
- More prep work for large folios (Joanne)
- Don't limit buffered read to 128k (Jim Harris)
- Fix zeroing of page end (dirtied with mmap) on file size extension
(Jimmy Zuber)
- Improve performance in certain cases with wake_up_sync() when queuing
request (Xuewen Yan)
- Misc fixes and cleanups (Xuewen Yan)
* tag 'fuse-update-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse: (35 commits)
fuse: zero the partial EOF page when extending a file
io_uring: Add missing include for ITER_SOURCE and ITER_DEST
fuse: Fix the condition to enable over-io-uring
fuse: invalidate the correct range after O_APPEND direct write
selftests/fuse: test post-EOF page zeroing when a file is extended
fuse: wake one waiter per freed slot when raising max_background
fuse: use min_not_zero() in fuse_init_server_timeout()
fuse: copy request headers via a stack buffer for io-uring
fuse: give wakeup hints to the scheduler for synchronous requests
fuse: check for NULL root inode in fuse_fill_super_submount
fuse: reject a duplicate fd= mount option
cuse: wait for pending RCU callbacks on module exit
fuse: fix invalidate lock leak on open O_TRUNC DAX failure
fuse: fix invalidate lock leak on setattr writeback failure
fuse: wait for FR_FINISHED on abort_on_kill to prevent use-after-free
fuse: make dentry_tree_work static
docs: fuse: document io-uring buffer pool and zero-copy uapi
fuse: add zero-copy over io-uring
fuse: support registered buffer pools in io-uring
fuse: add io-uring buffer pools
...
27 files changed, 1474 insertions, 236 deletions
diff --git a/Documentation/block/ublk.rst b/Documentation/block/ublk.rst index 0413dcd9ef69..28300fee22bf 100644 --- a/Documentation/block/ublk.rst +++ b/Documentation/block/ublk.rst @@ -382,17 +382,17 @@ Zero copy --------- ublk zero copy relies on io_uring's fixed kernel buffer, which provides -two APIs: `io_buffer_register_bvec()` and `io_buffer_unregister_bvec`. +two APIs: `io_buffer_register_request()` and `io_buffer_unregister`. ublk adds IO command of `UBLK_IO_REGISTER_IO_BUF` to call -`io_buffer_register_bvec()` for ublk server to register client request +`io_buffer_register_request()` for ublk server to register client request buffer into io_uring buffer table, then ublk server can submit io_uring IOs with the registered buffer index. IO command of `UBLK_IO_UNREGISTER_IO_BUF` -calls `io_buffer_unregister_bvec()` to unregister the buffer, which is -guaranteed to be live between calling `io_buffer_register_bvec()` and -`io_buffer_unregister_bvec()`. Any io_uring operation which supports this -kind of kernel buffer will grab one reference of the buffer until the -operation is completed. +calls `io_buffer_unregister()` to unregister the buffer, which is guaranteed +to be live between calling `io_buffer_register_request()` and +`io_buffer_unregister()`. Any io_uring operation which supports this kind of +kernel buffer will grab one reference of the buffer until the operation is +completed. ublk server implementing zero copy or user copy has to be CAP_SYS_ADMIN and be trusted, because it is ublk server's responsibility to make sure IO buffer diff --git a/Documentation/filesystems/fuse/fuse-io-uring.rst b/Documentation/filesystems/fuse/fuse-io-uring.rst index d73dd0dbd238..29f98057500d 100644 --- a/Documentation/filesystems/fuse/fuse-io-uring.rst +++ b/Documentation/filesystems/fuse/fuse-io-uring.rst @@ -11,6 +11,9 @@ and works. For generic details about FUSE see fuse.rst. This document also covers the current interface, which is still in development and might change. +For the userspace protocol, see +Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst. + Limitations =========== As of now not all requests types are supported through io-uring, userspace @@ -95,5 +98,34 @@ Sending requests with CQEs | <fuse_unlink() | | <sys_unlink() | - - +Buffer pools +============ + +Without a buffer pool, every entry needs to pass a dedicated payload buffer +large enough for the maximum payload size. A buffer pool decouples entries +from payload buffers. The server hands the kernel one contiguous buffer pool +of memory and when the kernel sends the server a request, it indicates the +offset into the pool for that request's payload. Internally, the kernel is +able to manage/optimize the buffer pool memory however it likes. + +A server may also register the pool region with io_uring as a fixed buffer. +The backing pages are then pinned once, avoiding per-request pinning and +address translation. This also allows servers to use the same registered +buffers for subsequent backing store I/O through io-uring, keeping data +in the same pinned pages without additional pinning / mapping overhead. + +Zero-copy +========= + +Zero-copy lets the server read from / write to the client's pages (pinned +user pages for direct I/O, or page-cache folios for buffered I/O) without an +intermediary payload copy. This requires CAP_SYS_ADMIN privileges. + +When a fuse request arrives for a file that opted into zero-copy, the kernel +registers the relevant pages (pinned user pages for direct i/o or underlying +page cache folios for buffered i/o) into a sparse slot in the server's +io_uring registered buffer table. The server can then operate on these pages +directly using io-uring fixed buffer operations (eg read_fixed / write_fixed) +and the kernel unregisters these pages when the request completes. +Non-page-backed args (eg op out headers) will go through the payload buffer as +normal. diff --git a/Documentation/filesystems/fuse/index.rst b/Documentation/filesystems/fuse/index.rst index 393a845214da..3dada6c4057a 100644 --- a/Documentation/filesystems/fuse/index.rst +++ b/Documentation/filesystems/fuse/index.rst @@ -12,3 +12,4 @@ FUSE (Filesystem in Userspace) Technical Documentation fuse-io fuse-io-uring fuse-passthrough + uapi/fuse-uapi-io-uring diff --git a/Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst b/Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst new file mode 100644 index 000000000000..8367be7ea29d --- /dev/null +++ b/Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst @@ -0,0 +1,126 @@ +.. SPDX-License-Identifier: GPL-2.0 + +===================================== +FUSE-over-io-uring uapi documentation +===================================== + +Commands +======== + +``enum fuse_uring_cmd``: + +``FUSE_IO_URING_CMD_ADD_QUEUE`` + Create a queue identified by ``fuse_uring_cmd_req.qid``. Queue-wide + options are passed in ``fuse_uring_cmd_req.flags``: + + ``FUSE_URING_ZERO_COPY`` + Enable zero-copy on this queue. Requires ``CAP_SYS_ADMIN`` and a buffer + pool, which is added separately via ``ADD_BUFPOOL`` before registering + entries (see `Zero-copy`_). + +``FUSE_IO_URING_CMD_ADD_BUFPOOL`` + Register the payload buffer pool for an existing queue. The server provides + a single contiguous region in ``fuse_uring_cmd_req.bufpool.uaddr`` / + ``.len``. This command must be issued after ``ADD_QUEUE`` and before + registering any payload-carrying entries on that queue. + ``fuse_uring_cmd_req.flags`` must be 0. Submitting this command with + ``IORING_URING_CMD_FIXED`` marks the pool as registered, which avoids per + i/o pinning/unpinning and mapping overhead (see `Buffer pools`_). + +``FUSE_IO_URING_CMD_REGISTER`` + Register a ring entry (a long-lived SQE that carries the request header + iovec). For a zero-copy queue, ``fuse_uring_cmd_req.ent_zero_copy_buf_index`` + indicates the reserved registered buffer table slot this entry uses for + zero-copy (see `Zero-copy`_). + +``FUSE_IO_URING_CMD_COMMIT_AND_FETCH`` + Commit the reply for a completed request and fetch the next one. The + request is identified by ``fuse_uring_cmd_req.commit_id`` (the value the + kernel reported in ``fuse_uring_ent_in_out.commit_id``). + +Structures +========== + +``struct fuse_uring_cmd_req`` (80-byte SQE command area): + +============================ ================================================== +Field Meaning +============================ ================================================== +``flags`` Command-specific flags (see each command). +``commit_id`` Request id, for ``COMMIT_AND_FETCH``. +``qid`` Queue index. +``bufpool.uaddr`` Pool base address, for ``ADD_BUFPOOL``. +``bufpool.len`` Pool length in bytes, for ``ADD_BUFPOOL``. +``bufpool.reserved`` Must be 0, for ``ADD_BUFPOOL``. +``ent_zero_copy_buf_index`` Per-entry zero-copy slot, for ``REGISTER``. +============================ ================================================== + +``struct fuse_uring_ent_in_out`` (reported by the kernel per request): + +============================ ================================================== +Field Meaning +============================ ================================================== +``flags`` ``FUSE_URING_ENT_ZERO_COPY`` if zero-copied. +``commit_id`` Id to echo back in ``COMMIT_AND_FETCH``. +``payload_sz`` Total payload size in bytes (see `Zero-copy`_). +``offset`` Payload buffer offset within the pool. +============================ ================================================== + +Buffer pools +============ +Setup: + +* Issue ``ADD_QUEUE`` for the qid. +* Issue ``ADD_BUFPOOL`` with ``bufpool.uaddr`` and ``bufpool.len`` pointing + at the region. +* Register entries with ``REGISTER``. + +For every request that has a payload, the kernel reports where the payload +lives in ``struct fuse_uring_ent_in_out`` (part of +``struct fuse_uring_req_header``): + +``offset`` + Byte offset, within the pool region, for this request's payload buffer. + The server adds this to the pool base address to locate the payload. + +``payload_sz`` + Number of payload bytes for this request. + +To use registered buffers, the server registers the pool region with io_uring +and submits ``ADD_BUFPOOL`` with ``IORING_URING_CMD_FIXED`` set in +``sqe->uring_cmd_flags`` and the index of the registered bufpool in +``sqe->buf_index``. Every SQE the server submits afterwards must follow the +same fixed-buffer protocol, carrying ``IORING_URING_CMD_FIXED`` and that same +``sqe->buf_index``. The same registered buffer can be reused for the server's +backing-store I/O as well (e.g. ``IORING_OP_READ_FIXED`` / +``IORING_OP_WRITE_FIXED``). + +Zero-copy +========= +Requirements: + +* The server must be privileged (``CAP_SYS_ADMIN``). +* A zero-copy queue: ``ADD_QUEUE`` with the ``FUSE_URING_ZERO_COPY`` flag set. +* A buffer pool: ``ADD_BUFPOOL``. +* For each entry, ``REGISTER`` with ``ent_zero_copy_buf_index`` set to the + index this entry uses in the server's io_uring registered-buffer table. + This is where the kernel registers the request's pages for the server to + access (it is separate from the payload pool). On a non-zero-copy queue this + field must be 0. + +Zero-copy is selected per open file. The server sets the open-file flag in +the ``FUSE_OPEN`` / ``FUSE_CREATE`` reply: + +``FOPEN_IO_URING_ZERO_COPY`` + Reads/writes on this open file should use zero-copy. + +For a request that is zero-copied, the kernel sets ``FUSE_URING_ENT_ZERO_COPY`` +in ``fuse_uring_ent_in_out.flags`` and places the request's pages at the +entry's ``ent_zero_copy_buf_index``. The server then issues +``IORING_OP_READ_FIXED`` / ``IORING_OP_WRITE_FIXED`` against that index to +transfer the data directly to/from the client's pages. + +For such a request, ``payload_sz`` includes the zero-copied page bytes +(transferred via the registered buffer at ``ent_zero_copy_buf_index``). Any +non-page-backed args (e.g. op headers) are still copied through the pool +payload buffer at ``offset``. diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c index 4d17ed264da1..6c5bec7da97c 100644 --- a/drivers/block/ublk_drv.c +++ b/drivers/block/ublk_drv.c @@ -1699,8 +1699,8 @@ ublk_auto_buf_register(const struct ublk_queue *ubq, struct request *req, { int ret; - ret = io_buffer_register_bvec(cmd, req, ublk_io_release, - io->buf.auto_reg.index, issue_flags); + ret = io_buffer_register_request(cmd, req, ublk_io_release, + io->buf.auto_reg.index, issue_flags); if (ret) { if (io->buf.auto_reg.flags & UBLK_AUTO_BUF_REG_FALLBACK) { ublk_auto_buf_reg_fallback(ubq, req->tag); @@ -1909,7 +1909,7 @@ static noinline void ublk_batch_dispatch_fail(struct ublk_queue *ubq, ublk_io_unlock(io); if (index != -1) - io_buffer_unregister_bvec(data->cmd, index, + io_buffer_unregister(data->cmd, index, data->issue_flags); } @@ -3192,8 +3192,8 @@ static int ublk_register_io_buf(struct io_uring_cmd *cmd, if (!req) return -EINVAL; - ret = io_buffer_register_bvec(cmd, req, ublk_io_release, index, - issue_flags); + ret = io_buffer_register_request(cmd, req, ublk_io_release, index, + issue_flags); if (ret) { ublk_put_req_ref(io, req); return ret; @@ -3224,8 +3224,8 @@ ublk_daemon_register_io_buf(struct io_uring_cmd *cmd, if (!ublk_dev_support_zero_copy(ub) || !blk_rq_has_data(req)) return -EINVAL; - ret = io_buffer_register_bvec(cmd, req, ublk_io_release, index, - issue_flags); + ret = io_buffer_register_request(cmd, req, ublk_io_release, index, + issue_flags); if (ret) return ret; @@ -3240,7 +3240,7 @@ static int ublk_unregister_io_buf(struct io_uring_cmd *cmd, if (!(ub->dev_info.flags & UBLK_F_SUPPORT_ZERO_COPY)) return -EINVAL; - return io_buffer_unregister_bvec(cmd, index, issue_flags); + return io_buffer_unregister(cmd, index, issue_flags); } static int ublk_check_fetch_buf(const struct ublk_device *ub, __u64 buf_addr) @@ -3384,7 +3384,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd, goto out; /* - * io_buffer_unregister_bvec() doesn't access the ubq or io, + * io_buffer_unregister() doesn't access the ubq or io, * so no need to validate the q_id, tag, or task */ if (_IOC_NR(cmd_op) == UBLK_IO_UNREGISTER_IO_BUF) @@ -3456,7 +3456,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd, req = ublk_fill_io_cmd(io, cmd); ublk_apply_io_buf(ub, io, cmd, addr, &auto_buf, &buf_idx); if (buf_idx != UBLK_INVALID_BUF_IDX) - io_buffer_unregister_bvec(cmd, buf_idx, issue_flags); + io_buffer_unregister(cmd, buf_idx, issue_flags); compl = ublk_need_complete_req(ub, io); if (req_op(req) == REQ_OP_ZONE_APPEND) @@ -3801,7 +3801,7 @@ static int ublk_batch_commit_io(struct ublk_queue *ubq, } if (buf_idx != UBLK_INVALID_BUF_IDX) - io_buffer_unregister_bvec(data->cmd, buf_idx, data->issue_flags); + io_buffer_unregister(data->cmd, buf_idx, data->issue_flags); if (req_op(req) == REQ_OP_ZONE_APPEND) req->__sector = ublk_batch_zone_lba(uc, elem); if (compl) diff --git a/fs/fuse/args.h b/fs/fuse/args.h index ecfe51a192af..5173264a1261 100644 --- a/fs/fuse/args.h +++ b/fs/fuse/args.h @@ -42,6 +42,8 @@ struct fuse_args { bool is_pinned:1; bool invalidate_vmap:1; bool abort_on_kill:1; + /* server requested io-uring zero-copy for this op */ + bool zero_copy:1; struct fuse_in_arg in_args[4]; struct fuse_arg out_args[2]; void (*end)(struct fuse_args *args, int error); diff --git a/fs/fuse/cuse.c b/fs/fuse/cuse.c index 3c15b5ba16d7..4079cf8e5974 100644 --- a/fs/fuse/cuse.c +++ b/fs/fuse/cuse.c @@ -530,7 +530,8 @@ static int cuse_channel_open(struct inode *inode, struct file *file) INIT_LIST_HEAD(&cc->list); - cc->fc.chan->initialized = 1; + /* Pairs with smp_load_acquire() readers of fch->initialized */ + smp_store_release(&cc->fc.chan->initialized, 1); rc = cuse_send_init(cc); if (rc) { fuse_dev_put(fud); @@ -653,6 +654,11 @@ static void __exit cuse_exit(void) { misc_deregister(&cuse_miscdev); class_destroy(cuse_class); + /* + * Wait for pending call_rcu() callbacks that call back into + * this module via fc->release (cuse_fc_release). + */ + rcu_barrier(); } module_init(cuse_init); diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c index 5763a7cd3b37..4fec31fc0b84 100644 --- a/fs/fuse/dev.c +++ b/fs/fuse/dev.c @@ -75,17 +75,23 @@ void fuse_chan_set_initialized(struct fuse_chan *fch, struct fuse_chan_param *pa fch->minor = param->minor; fch->max_write = param->max_write; fch->max_pages = param->max_pages; + + if (param->io_uring_enabled) + fuse_uring_conn_init(fch); } - /* Make sure stores before this are seen on another CPU */ - smp_wmb(); - fch->initialized = 1; + /* Pairs with smp_load_acquire() readers of fch->initialized */ + smp_store_release(&fch->initialized, 1); wake_up_all(&fch->blocked_waitq); } static bool fuse_block_alloc(struct fuse_chan *fch, bool for_background) { - return !fch->initialized || (for_background && fch->blocked) || + /* Pairs with smp_store_release() in fuse_chan_set_initialized() */ + if (!smp_load_acquire(&fch->initialized)) + return true; + + return (for_background && fch->blocked) || (fch->io_uring && fch->connected && !fuse_uring_ready(fch)); } @@ -120,9 +126,6 @@ static struct fuse_req *fuse_get_req(struct fuse_chan *fch, bool for_background) goto out; } - /* Matches smp_wmb() in fuse_chan_set_initialized() */ - smp_rmb(); - err = -ENOTCONN; if (!fch->connected) goto out; @@ -210,10 +213,13 @@ EXPORT_SYMBOL_GPL(fuse_req_hash); /* * A new request is available, wake fiq->waitq */ -static void fuse_dev_wake_and_unlock(struct fuse_iqueue *fiq) +static void fuse_dev_wake_and_unlock(struct fuse_iqueue *fiq, bool sync) __releases(fiq->lock) { - wake_up(&fiq->waitq); + if (sync) + wake_up_sync(&fiq->waitq); + else + wake_up(&fiq->waitq); kill_fasync(&fiq->fasync, SIGIO, POLL_IN); spin_unlock(&fiq->lock); } @@ -230,7 +236,7 @@ void fuse_dev_queue_forget(struct fuse_iqueue *fiq, if (fiq->connected) { fiq->forget_list_tail->next = forget; fiq->forget_list_tail = forget; - fuse_dev_wake_and_unlock(fiq); + fuse_dev_wake_and_unlock(fiq, false); } else { kfree(forget); spin_unlock(&fiq->lock); @@ -240,7 +246,8 @@ void fuse_dev_queue_forget(struct fuse_iqueue *fiq, void fuse_dev_queue_interrupt(struct fuse_iqueue *fiq, struct fuse_req *req) { spin_lock(&fiq->lock); - if (list_empty(&req->intr_entry)) { + /* Repeat FR_SENT test after obtaining the lock to prevent race with fuse_resend() */ + if (list_empty(&req->intr_entry) && test_bit(FR_SENT, &req->flags)) { list_add_tail(&req->intr_entry, &fiq->interrupts); /* * Pairs with smp_mb() implied by test_and_set_bit() @@ -251,7 +258,7 @@ void fuse_dev_queue_interrupt(struct fuse_iqueue *fiq, struct fuse_req *req) list_del_init(&req->intr_entry); spin_unlock(&fiq->lock); } else { - fuse_dev_wake_and_unlock(fiq); + fuse_dev_wake_and_unlock(fiq, false); } } else { spin_unlock(&fiq->lock); @@ -281,11 +288,13 @@ EXPORT_SYMBOL_GPL(fuse_request_assign_unique); static void fuse_dev_queue_req(struct fuse_iqueue *fiq, struct fuse_req *req) { + bool sync = test_and_clear_bit(FR_SYNC_WAKEUP, &req->flags); + spin_lock(&fiq->lock); if (fiq->connected) { fuse_request_assign_unique_locked(fiq, req); list_add_tail(&req->list, &fiq->pending); - fuse_dev_wake_and_unlock(fiq); + fuse_dev_wake_and_unlock(fiq, sync); } else { spin_unlock(&fiq->lock); req->out.h.error = -ENOTCONN; @@ -397,7 +406,8 @@ void fuse_chan_max_background_set(struct fuse_chan *fch, unsigned int val) fch->max_background = val; fch->blocked = fch->num_background >= fch->max_background; if (!fch->blocked) - wake_up(&fch->blocked_waitq); + wake_up_nr(&fch->blocked_waitq, + fch->max_background - fch->num_background); spin_unlock(&fch->bg_lock); } @@ -411,11 +421,6 @@ void fuse_chan_set_fc(struct fuse_chan *fch, struct fuse_conn *fc) fch->conn = fc; } -void fuse_chan_io_uring_enable(struct fuse_chan *fch) -{ - fch->io_uring = 1; -} - void fuse_pqueue_init(struct fuse_pqueue *fpq) { spin_lock_init(&fpq->lock); @@ -725,7 +730,7 @@ static void request_wait_answer(struct fuse_req *req) if (req->args->abort_on_kill) { fuse_chan_abort(fch, false); - return; + goto wait_for_finish; } if (test_bit(FR_URING, &req->flags)) @@ -736,6 +741,7 @@ static void request_wait_answer(struct fuse_req *req) return; } +wait_for_finish: /* * Either request is already in userspace, or it was forced. * Wait it out. @@ -752,6 +758,11 @@ static void __fuse_request_send(struct fuse_req *req) /* acquire extra reference, since request is still needed after fuse_request_end() */ __fuse_get_request(req); + /* + * This is a synchronous request: the caller will block waiting for + * the answer. Hint the scheduler via wake_up_sync(). + */ + set_bit(FR_SYNC_WAKEUP, &req->flags); fuse_send_one(fiq, req); request_wait_answer(req); @@ -1249,11 +1260,25 @@ int fuse_copy_folio(struct fuse_copy_state *cs, struct folio **foliop, if (folio) { size = folio_size(folio); - if (zeroing && count < size) - folio_zero_range(folio, 0, size); + if (zeroing && count < size) { + /* + * When the copy is skipped the folio already holds the + * payload, so only the bytes outside [offset, offset + + * count) may be zeroed. + * + * Otherwise, the whole folio is cleared first so that a + * failed copy leaves zeros rather than stale folio + * contents. + */ + if (cs->skip_folio_copy) + folio_zero_segments(folio, 0, offset, + offset + count, size); + else + folio_zero_range(folio, 0, size); + } } - while (count) { + while (!cs->skip_folio_copy && count) { if (cs->write && cs->pipebufs && folio) { /* * Can't control lifetime of pipe buffers, so always @@ -1346,6 +1371,10 @@ int fuse_copy_args(struct fuse_copy_state *cs, unsigned numargs, for (i = 0; !err && i < numargs; i++) { struct fuse_arg *arg = &args[i]; if (i == numargs - 1 && argpages) + /* + * if cs->skip_folio_copy is set, this just does any + * needed zeroing. No copying is involved. + */ err = fuse_copy_folios(cs, arg->size, zeroing); else err = fuse_copy_one(cs, arg->value, arg->size); @@ -1760,7 +1789,7 @@ out: void fuse_chan_resend(struct fuse_chan *fch) { struct fuse_dev *fud; - struct fuse_req *req, *next; + struct fuse_req *req; struct fuse_iqueue *fiq = &fch->iq; LIST_HEAD(to_queue); unsigned int i; @@ -1775,24 +1804,20 @@ void fuse_chan_resend(struct fuse_chan *fch) struct fuse_pqueue *fpq = &fud->pq; spin_lock(&fpq->lock); - for (i = 0; i < FUSE_PQ_HASH_SIZE; i++) - list_splice_tail_init(&fpq->processing[i], &to_queue); + for (i = 0; i < FUSE_PQ_HASH_SIZE; i++) { + struct list_head *this_queue = &fpq->processing[i]; + + list_for_each_entry(req, this_queue, list) + clear_bit(FR_SENT, &req->flags); + list_splice_tail_init(this_queue, &to_queue); + } spin_unlock(&fpq->lock); } spin_unlock(&fch->lock); - list_for_each_entry_safe(req, next, &to_queue, list) { - set_bit(FR_PENDING, &req->flags); - clear_bit(FR_SENT, &req->flags); - /* mark the request as resend request */ - req->in.h.unique |= FUSE_UNIQUE_RESEND; - } - spin_lock(&fiq->lock); if (!fiq->connected) { spin_unlock(&fiq->lock); - list_for_each_entry(req, &to_queue, list) - clear_bit(FR_PENDING, &req->flags); fuse_dev_end_requests(&to_queue); return; } @@ -1801,12 +1826,16 @@ void fuse_chan_resend(struct fuse_chan *fch) * intr_entry on fiq->interrupts after the request is re-queued. */ list_for_each_entry(req, &to_queue, list) { + set_bit(FR_PENDING, &req->flags); + /* mark the request as resend request */ + req->in.h.unique |= FUSE_UNIQUE_RESEND; + if (test_bit(FR_INTERRUPTED, &req->flags)) list_del_init(&req->intr_entry); } /* iq and pq requests are both oldest to newest */ list_splice(&to_queue, &fiq->pending); - fuse_dev_wake_and_unlock(fiq); + fuse_dev_wake_and_unlock(fiq, false); } /* Look up request on processing list by unique ID */ @@ -1888,7 +1917,8 @@ static ssize_t fuse_dev_do_write(struct fuse_dev *fud, * initialized and connected state */ err = -EINVAL; - if (!fch->initialized || !fch->connected) + /* Pairs with smp_store_release() in fuse_chan_set_initialized() */ + if (!smp_load_acquire(&fch->initialized) || !fch->connected) goto copy_finish; /* Don't try to move folios (yet) */ diff --git a/fs/fuse/dev.h b/fs/fuse/dev.h index aed69fd14c41..8d25378c0918 100644 --- a/fs/fuse/dev.h +++ b/fs/fuse/dev.h @@ -22,6 +22,7 @@ struct fuse_chan_param { unsigned int minor; unsigned int max_write; unsigned int max_pages; + bool io_uring_enabled; }; struct fuse_chan *fuse_chan_new(void); @@ -34,7 +35,6 @@ void fuse_chan_max_background_set(struct fuse_chan *fch, unsigned int val); unsigned int fuse_chan_num_waiting(struct fuse_chan *fch); void fuse_chan_set_fc(struct fuse_chan *fch, struct fuse_conn *fc); void fuse_chan_set_initialized(struct fuse_chan *fch, struct fuse_chan_param *param); -void fuse_chan_io_uring_enable(struct fuse_chan *fch); ssize_t fuse_chan_send(struct fuse_chan *fch, struct fuse_args *args); int fuse_chan_send_bg(struct fuse_chan *fch, struct fuse_args *args, gfp_t gfp_flags); int fuse_chan_send_notify_reply(struct fuse_chan *fch, struct fuse_args *args, u64 unique); diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c index 77c8cec43d9c..c6dd420c4034 100644 --- a/fs/fuse/dev_uring.c +++ b/fs/fuse/dev_uring.c @@ -9,6 +9,7 @@ #include "dev_uring_i.h" #include "fuse_trace.h" +#include <linux/bitmap.h> #include <linux/fs.h> #include <linux/io_uring/cmd.h> @@ -21,6 +22,8 @@ MODULE_PARM_DESC(enable_uring, #define FUSE_URING_IOV_HEADERS 0 #define FUSE_URING_IOV_PAYLOAD 1 +#define FUSE_URING_ADD_QUEUE_FLAGS (FUSE_URING_ZERO_COPY) + bool fuse_uring_enabled(void) { return enable_uring; @@ -30,6 +33,11 @@ struct fuse_uring_pdu { struct fuse_ring_ent *ent; }; +struct fuse_zero_copy_bvs { + unsigned int nr_bvs; + struct bio_vec bvs[]; +}; + static const struct fuse_iqueue_ops fuse_io_uring_ops; enum fuse_uring_header_type { @@ -41,6 +49,32 @@ enum fuse_uring_header_type { FUSE_URING_HEADER_RING_ENT, }; +static inline bool bufpool_enabled(struct fuse_ring_queue *queue) +{ + return queue->payload_mode == FUSE_PAYLOAD_BUFPOOL; +} + +static inline bool bufpool_registered(struct fuse_ring_queue *queue) +{ + return queue->bufpool && queue->bufpool->registered; +} + +/* + * For a registered bufpool, every sqe that drives a payload import (REGISTER, + * COMMIT_AND_FETCH) must carry the registered buffer index of the pool. + * This also must be called from the command's issue handler, where cmd->sqe is + * still valid + */ +static inline bool fuse_uring_cmd_index_ok(struct io_uring_cmd *cmd, + struct fuse_ring_queue *queue) +{ + if (!bufpool_registered(queue)) + return true; + + return (cmd->flags & IORING_URING_CMD_FIXED) && + READ_ONCE(cmd->sqe->buf_index) == queue->bufpool->registered_index; +} + static void uring_cmd_set_ring_ent(struct io_uring_cmd *cmd, struct fuse_ring_ent *ring_ent) { @@ -86,8 +120,36 @@ static void fuse_uring_flush_bg(struct fuse_ring_queue *queue) } } +static bool can_zero_copy_req(struct fuse_ring_ent *ent, struct fuse_req *req) +{ + struct fuse_args *args = req->args; + + if (!ent->queue->zero_copy || !args->zero_copy) + return false; + + if (args->opcode != FUSE_READ && args->opcode != FUSE_WRITE) + return false; + + return args->in_pages || args->out_pages; +} + +static void zero_copy_unregister(struct io_uring_cmd *cmd, + struct fuse_ring_ent *ent, + unsigned int issue_flags) +{ + if (ent->zero_copied) { + int err = io_buffer_unregister(cmd, ent->zero_copy_index, + issue_flags); + + if (err) + pr_warn_ratelimited("qid=%d zero-copy unregister failed: %d\n", + ent->queue->qid, err); + ent->zero_copied = false; + } +} + static void fuse_uring_req_end(struct fuse_ring_ent *ent, struct fuse_req *req, - int error) + int error, unsigned int issue_flags) { struct fuse_ring_queue *queue = ent->queue; struct fuse_ring *ring = queue->ring; @@ -107,6 +169,8 @@ static void fuse_uring_req_end(struct fuse_ring_ent *ent, struct fuse_req *req, spin_unlock(&queue->lock); + zero_copy_unregister(ent->cmd, ent, issue_flags); + if (error) req->out.h.error = error; @@ -204,7 +268,7 @@ void fuse_uring_destruct(struct fuse_chan *fch) return; for (qid = 0; qid < ring->nr_queues; qid++) { - struct fuse_ring_queue *queue = ring->queues[qid]; + struct fuse_ring_queue *queue = READ_ONCE(ring->queues[qid]); struct fuse_ring_ent *ent, *next; if (!queue) @@ -222,8 +286,9 @@ void fuse_uring_destruct(struct fuse_chan *fch) } kfree(queue->fpq.processing); + kfree(queue->bufpool); kfree(queue); - ring->queues[qid] = NULL; + WRITE_ONCE(ring->queues[qid], NULL); } kfree(ring->queues); @@ -238,7 +303,6 @@ static struct fuse_ring *fuse_uring_create(struct fuse_chan *fch) { struct fuse_ring *ring; size_t nr_queues = num_possible_cpus(); - struct fuse_ring *res = NULL; size_t max_payload_size; ring = kzalloc_obj(*ring, GFP_KERNEL_ACCOUNT); @@ -258,12 +322,6 @@ static struct fuse_ring *fuse_uring_create(struct fuse_chan *fch) spin_unlock(&fch->lock); goto out_err; } - if (fch->ring) { - /* race, another thread created the ring in the meantime */ - spin_unlock(&fch->lock); - res = fch->ring; - goto out_err; - } init_waitqueue_head(&ring->stop_waitq); @@ -278,11 +336,18 @@ static struct fuse_ring *fuse_uring_create(struct fuse_chan *fch) out_err: kfree(ring->queues); kfree(ring); - return res; + return NULL; +} + +void fuse_uring_conn_init(struct fuse_chan *fch) +{ + if (fuse_uring_create(fch)) + fch->io_uring = 1; } static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring, - int qid) + int qid, bool zero_copy, + bool fail_if_exists) { struct fuse_chan *fch = ring->chan; struct fuse_ring_queue *queue; @@ -290,16 +355,17 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring, queue = kzalloc_obj(*queue, GFP_KERNEL_ACCOUNT); if (!queue) - return NULL; + return ERR_PTR(-ENOMEM); pq = fuse_pqueue_alloc(); if (!pq) { kfree(queue); - return NULL; + return ERR_PTR(-ENOMEM); } queue->qid = qid; queue->ring = ring; spin_lock_init(&queue->lock); + queue->zero_copy = zero_copy; INIT_LIST_HEAD(&queue->ent_avail_queue); INIT_LIST_HEAD(&queue->ent_commit_queue); @@ -316,14 +382,17 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring, if (ring->queues[qid]) { spin_unlock(&fch->lock); kfree(queue->fpq.processing); + kfree(queue->bufpool); kfree(queue); - return ring->queues[qid]; + return fail_if_exists ? ERR_PTR(-EEXIST) : ring->queues[qid]; } /* - * write_once and lock as the caller mostly doesn't take the lock at all + * fch->lock serializes concurrent creators for this qid. + * smp_store_release() are for the lockless readers who must see a + * fully initialized queue after &ring->queues[qid] is set */ - WRITE_ONCE(ring->queues[qid], queue); + smp_store_release(&ring->queues[qid], queue); spin_unlock(&fch->lock); return queue; @@ -434,7 +503,7 @@ static void fuse_uring_log_ent_state(struct fuse_ring *ring) struct fuse_ring_ent *ent; for (qid = 0; qid < ring->nr_queues; qid++) { - struct fuse_ring_queue *queue = ring->queues[qid]; + struct fuse_ring_queue *queue = READ_ONCE(ring->queues[qid]); if (!queue) continue; @@ -643,30 +712,57 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent, return 0; } +static int fuse_uring_import_payload(struct fuse_ring_ent *ent, int dir, + struct iov_iter *iter, + unsigned int issue_flags) +{ + void __user *base = ent->payload.iov_base; + size_t len = ent->payload.iov_len; + int err = 0; + + if (!base) { + memset(iter, 0, sizeof(*iter)); + return 0; + } + + if (bufpool_registered(ent->queue)) + err = io_uring_cmd_import_fixed((u64)(uintptr_t)base, len, dir, + iter, ent->cmd, issue_flags); + else + err = import_ubuf(dir, base, len, iter); + + if (err) + pr_info_ratelimited("fuse: Import of user buffer failed\n"); + + return err; +} + static int setup_fuse_copy_state(struct fuse_copy_state *cs, - struct fuse_ring *ring, struct fuse_req *req, + struct fuse_req *req, struct fuse_ring_ent *ent, int dir, - struct iov_iter *iter) + struct iov_iter *iter, + unsigned int issue_flags) { int err; - err = import_ubuf(dir, ent->payload, ring->max_payload_sz, iter); - if (err) { - pr_info_ratelimited("fuse: Import of user buffer failed\n"); + err = fuse_uring_import_payload(ent, dir, iter, issue_flags); + if (err) return err; - } fuse_copy_init(cs, dir == ITER_DEST, iter); + if (ent->zero_copied) + cs->skip_folio_copy = true; + cs->is_uring = true; cs->req = req; return 0; } -static int fuse_uring_copy_from_ring(struct fuse_ring *ring, - struct fuse_req *req, - struct fuse_ring_ent *ent) +static int fuse_uring_copy_from_ring(struct fuse_req *req, + struct fuse_ring_ent *ent, + unsigned int issue_flags) { struct fuse_copy_state cs; struct fuse_args *args = req->args; @@ -679,7 +775,8 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring, if (err) return err; - err = setup_fuse_copy_state(&cs, ring, req, ent, ITER_SOURCE, &iter); + err = setup_fuse_copy_state(&cs, req, ent, ITER_SOURCE, &iter, + issue_flags); if (err) return err; @@ -688,11 +785,68 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring, return err; } +static void fuse_zero_copy_release(void *priv) +{ + struct fuse_zero_copy_bvs *zc_bvs = priv; + unsigned int i; + + for (i = 0; i < zc_bvs->nr_bvs; i++) + folio_put(page_folio(zc_bvs->bvs[i].bv_page)); + + kvfree(zc_bvs); +} + +static int fuse_uring_set_up_zero_copy(struct fuse_ring_ent *ent, + struct fuse_req *req, + unsigned int issue_flags) +{ + struct fuse_args_pages *ap; + int err, i, ddir = 0; + struct fuse_zero_copy_bvs *zc_bvs; + struct bio_vec *bvs; + + /* out_pages indicates a read, in_pages indicates a write */ + if (req->args->out_pages) + ddir |= IO_BUF_DEST; + if (req->args->in_pages) + ddir |= IO_BUF_SOURCE; + + ap = container_of(req->args, typeof(*ap), args); + + zc_bvs = kvmalloc_flex(*zc_bvs, bvs, ap->num_folios, + GFP_KERNEL_ACCOUNT); + if (!zc_bvs) + return -ENOMEM; + + zc_bvs->nr_bvs = ap->num_folios; + bvs = zc_bvs->bvs; + for (i = 0; i < ap->num_folios; i++) { + bvs[i].bv_page = folio_page(ap->folios[i], 0); + bvs[i].bv_offset = ap->descs[i].offset; + bvs[i].bv_len = ap->descs[i].length; + folio_get(ap->folios[i]); + } + + err = io_buffer_register_bvec(ent->cmd, bvs, ap->num_folios, + fuse_zero_copy_release, zc_bvs, + ddir, ent->zero_copy_index, + issue_flags); + if (err) { + fuse_zero_copy_release(zc_bvs); + return err; + } + + ent->zero_copied = true; + + return 0; +} + /* * Copy data from the req to the ring buffer */ -static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, - struct fuse_ring_ent *ent) +static int fuse_uring_args_to_ring(struct fuse_req *req, + struct fuse_ring_ent *ent, + unsigned int issue_flags) { struct fuse_copy_state cs; struct fuse_args *args = req->args; @@ -705,7 +859,15 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, .commit_id = req->in.h.unique, }; - err = setup_fuse_copy_state(&cs, ring, req, ent, ITER_DEST, &iter); + if (can_zero_copy_req(ent, req)) { + ent_in_out.flags |= FUSE_URING_ENT_ZERO_COPY; + err = fuse_uring_set_up_zero_copy(ent, req, issue_flags); + if (err) + return err; + } + + err = setup_fuse_copy_state(&cs, req, ent, ITER_DEST, &iter, + issue_flags); if (err) return err; @@ -735,15 +897,32 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req, } ent_in_out.payload_sz = cs.ring.copied_sz; + /* + * on a zero-copied write the pages are registered for the server to + * read via a fixed-buffer op rather than copied into the payload + * buffer, so copied_sz does not account for it. The server still needs + * the total inbound size to know how many bytes to read from the + * registered buffer, so add the page arg (always the last in-arg) back + * in + */ + if (cs.skip_folio_copy && args->in_pages) + ent_in_out.payload_sz += + args->in_args[args->in_numargs - 1].size; + + if (bufpool_enabled(ent->queue) && ent->payload.iov_base) + ent_in_out.offset = + (uintptr_t)ent->payload.iov_base - ent->queue->bufpool->base_uaddr; + return copy_header_to_ring(ent, FUSE_URING_HEADER_RING_ENT, &ent_in_out, sizeof(ent_in_out)); } static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, - struct fuse_req *req) + struct fuse_req *req, + unsigned int issue_flags) { struct fuse_ring_queue *queue = ent->queue; - struct fuse_ring *ring = queue->ring; + struct fuse_in_header in_header; int err; err = -EIO; @@ -758,23 +937,124 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, return err; /* copy the request */ - err = fuse_uring_args_to_ring(ring, req, ent); + err = fuse_uring_args_to_ring(req, ent, issue_flags); if (unlikely(err)) { pr_info_ratelimited("Copy to ring failed: %d\n", err); return err; } /* copy fuse_in_header */ - return copy_header_to_ring(ent, FUSE_URING_HEADER_IN_OUT, &req->in.h, - sizeof(req->in.h)); + in_header = req->in.h; + return copy_header_to_ring(ent, FUSE_URING_HEADER_IN_OUT, &in_header, + sizeof(in_header)); +} + +static bool fuse_uring_req_has_copyable_payload(struct fuse_ring_ent *ent, + struct fuse_req *req) +{ + struct fuse_args *args = req->args; + + if (!can_zero_copy_req(ent, req)) + return args->in_numargs > 1 || args->out_numargs; + + /* + * the asymmetry between in_numargs > 2 and out_numargs > 1 is because + * the per-op header is extracted before fuse_copy_args() for inargs but + * not for outargs + */ + if ((args->in_numargs > 1) && (!args->in_pages || args->in_numargs > 2)) + return true; + if (args->out_numargs && (!args->out_pages || args->out_numargs > 1)) + return true; + + return false; +} + +static int fuse_uring_select_buffer(struct fuse_ring_ent *ent) +{ + struct fuse_ring_queue *queue = ent->queue; + struct fuse_bufpool *pool = queue->bufpool; + unsigned int id; + + lockdep_assert_held(&queue->lock); + + id = find_first_bit(pool->free_map, pool->nr_bufs); + if (id >= pool->nr_bufs) + return -ENOBUFS; + + WARN_ON_ONCE(ent->payload.iov_base); + __clear_bit(id, pool->free_map); + + ent->buf_id = id; + ent->payload.iov_base = + (void __user *)(pool->base_uaddr + id * pool->buf_size); + ent->payload.iov_len = pool->buf_size; + + return 0; +} + +static void fuse_uring_recycle_buffer(struct fuse_ring_ent *ent) +{ + struct iovec *ent_payload = &ent->payload; + struct fuse_ring_queue *queue = ent->queue; + struct fuse_bufpool *pool; + + lockdep_assert_held(&queue->lock); + + if (!bufpool_enabled(queue) || !ent_payload->iov_base) + return; + + pool = queue->bufpool; + + /* a buffer should never be recycled twice */ + WARN_ON_ONCE(test_bit(ent->buf_id, pool->free_map)); + __set_bit(ent->buf_id, pool->free_map); + + memset(ent_payload, 0, sizeof(*ent_payload)); + ent->buf_id = 0; +} + +static int fuse_uring_next_req_update_buffer(struct fuse_ring_ent *ent, + struct fuse_req *req) +{ + bool buffer_selected; + bool has_payload; + + if (!bufpool_enabled(ent->queue)) + return 0; + + buffer_selected = !!ent->payload.iov_base; + has_payload = fuse_uring_req_has_copyable_payload(ent, req); + + if (has_payload && !buffer_selected) + return fuse_uring_select_buffer(ent); + + if (!has_payload && buffer_selected) + fuse_uring_recycle_buffer(ent); + + return 0; +} + +static int fuse_uring_prep_buffer(struct fuse_ring_ent *ent, + struct fuse_req *req) +{ + if (!bufpool_enabled(ent->queue)) + return 0; + + /* no payload to copy, can skip selecting a buffer */ + if (!fuse_uring_req_has_copyable_payload(ent, req)) + return 0; + + return fuse_uring_select_buffer(ent); } static int fuse_uring_prepare_send(struct fuse_ring_ent *ent, - struct fuse_req *req) + struct fuse_req *req, + unsigned int issue_flags) { int err; - err = fuse_uring_copy_to_ring(ent, req); + err = fuse_uring_copy_to_ring(ent, req, issue_flags); if (!err) { set_bit(FR_SENT, &req->flags); trace_fuse_request_sent(req); @@ -788,7 +1068,7 @@ static int fuse_uring_prepare_send(struct fuse_ring_ent *ent, ent->state = FRRS_INVALID; spin_unlock(&ent->queue->lock); - fuse_uring_req_end(ent, req, err); + fuse_uring_req_end(ent, req, err, issue_flags); } return err; @@ -856,9 +1136,12 @@ static struct fuse_req *fuse_uring_ent_assign_req(struct fuse_ring_ent *ent) /* get and assign the next entry while it is still holding the lock */ req = list_first_entry_or_null(req_queue, struct fuse_req, list); - if (req) - fuse_uring_add_req_to_ring_ent(ent, req); + if (!req || fuse_uring_next_req_update_buffer(ent, req)) { + fuse_uring_recycle_buffer(ent); + return NULL; + } + fuse_uring_add_req_to_ring_ent(ent, req); return req; } @@ -870,12 +1153,13 @@ static struct fuse_req *fuse_uring_ent_assign_req(struct fuse_ring_ent *ent) static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req, unsigned int issue_flags) { - struct fuse_ring *ring = ent->queue->ring; + struct fuse_out_header out_header; ssize_t err = -EFAULT; - if (copy_header_from_ring(ent, FUSE_URING_HEADER_IN_OUT, &req->out.h, - sizeof(req->out.h))) + if (copy_header_from_ring(ent, FUSE_URING_HEADER_IN_OUT, &out_header, + sizeof(out_header))) goto out; + req->out.h = out_header; err = fuse_uring_out_header_has_err(&req->out.h, req); if (err) { @@ -883,9 +1167,9 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req, goto out; } - err = fuse_uring_copy_from_ring(ring, req, ent); + err = fuse_uring_copy_from_ring(req, ent, issue_flags); out: - fuse_uring_req_end(ent, req, err); + fuse_uring_req_end(ent, req, err, issue_flags); } /* @@ -895,7 +1179,8 @@ out: * Else, there is no next fuse request and this returns false. */ static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent, - struct fuse_ring_queue *queue) + struct fuse_ring_queue *queue, + unsigned int issue_flags) { int err; struct fuse_req *req; @@ -907,7 +1192,7 @@ retry: spin_unlock(&queue->lock); if (req) { - err = fuse_uring_prepare_send(ent, req); + err = fuse_uring_prepare_send(ent, req, issue_flags); if (err) goto retry; } @@ -967,7 +1252,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags, if (qid >= ring->nr_queues) return -EINVAL; - queue = ring->queues[qid]; + queue = READ_ONCE(ring->queues[qid]); if (!queue) return err; fpq = &queue->fpq; @@ -981,6 +1266,11 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags, return err; } + if (!fuse_uring_cmd_index_ok(cmd, queue)) { + spin_unlock(&queue->lock); + return -EINVAL; + } + /* Find a request based on the unique ID of the fuse request * This should get revised, as it needs a hash calculation and list * search. And full struct fuse_pqueue is needed (memory overhead). @@ -1002,8 +1292,14 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags, if (err != 0) { pr_info_ratelimited("qid=%d commit_id %llu state %d", queue->qid, commit_id, ent->state); + fuse_uring_recycle_buffer(ent); spin_unlock(&queue->lock); - fuse_uring_req_end(ent, req, err); + /* + * Unregister any zero copyable pages since ent->cmd is null + * when it hits fuse_uring_req_end() in this path + */ + zero_copy_unregister(cmd, ent, issue_flags); + fuse_uring_req_end(ent, req, err, issue_flags); return err; } @@ -1019,8 +1315,13 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags, * fuse requests would otherwise not get processed - committing * and fetching is done in one step vs legacy fuse, which has separated * read (fetch request) and write (commit result). + * + * If there is no next request or if all buffers are busy (if using a + * bufpool), the cmd is not returned to userspace. The entry is left + * available and the cmd only returns to userspace when there's a + * next request and an available buffer. */ - if (fuse_uring_get_next_fuse_req(ent, queue)) + if (fuse_uring_get_next_fuse_req(ent, queue, issue_flags)) fuse_uring_send(ent, cmd, 0, issue_flags); return 0; } @@ -1035,7 +1336,7 @@ static bool is_ring_ready(struct fuse_ring *ring, int current_qid) if (current_qid == qid) continue; - queue = ring->queues[qid]; + queue = READ_ONCE(ring->queues[qid]); if (!queue) { ready = false; break; @@ -1122,10 +1423,14 @@ static struct fuse_ring_ent * fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, struct fuse_ring_queue *queue) { + const struct fuse_uring_cmd_req *cmd_req = + io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req); struct fuse_ring *ring = queue->ring; struct fuse_ring_ent *ent; struct iovec iov[FUSE_URING_IOV_SEGS]; struct iovec *headers, *payload; + unsigned int zero_copy_index; + int err; err = fuse_uring_get_iovec_from_sqe(cmd->sqe, iov); @@ -1135,6 +1440,10 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, return ERR_PTR(err); } + zero_copy_index = READ_ONCE(cmd_req->ent_zero_copy_buf_index); + if (zero_copy_index && !queue->zero_copy) + return ERR_PTR(-EINVAL); + err = -EINVAL; headers = &iov[FUSE_URING_IOV_HEADERS]; if (headers->iov_len < sizeof(struct fuse_uring_req_header)) { @@ -1143,11 +1452,29 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, } payload = &iov[FUSE_URING_IOV_PAYLOAD]; - if (payload->iov_len < ring->max_payload_sz) { - pr_info_ratelimited("Invalid req payload len %zu\n", - payload->iov_len); - return ERR_PTR(err); + + spin_lock(&queue->lock); + if (bufpool_enabled(queue)) { + if (payload->iov_base || payload->iov_len || + !fuse_uring_cmd_index_ok(cmd, queue)) { + spin_unlock(&queue->lock); + return ERR_PTR(err); + } + } else { + if (payload->iov_len < ring->max_payload_sz) { + spin_unlock(&queue->lock); + pr_info_ratelimited("Invalid req payload len %zu\n", + payload->iov_len); + return ERR_PTR(err); + } + if (queue->zero_copy) { + spin_unlock(&queue->lock); + pr_info_ratelimited("Can only use zero copy with bufpools\n"); + return ERR_PTR(err); + } + queue->payload_mode = FUSE_PAYLOAD_PER_ENT; } + spin_unlock(&queue->lock); err = -ENOMEM; ent = kzalloc_obj(*ent, GFP_KERNEL_ACCOUNT); @@ -1158,7 +1485,9 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, ent->queue = queue; ent->headers = headers->iov_base; - ent->payload = payload->iov_base; + if (queue->payload_mode == FUSE_PAYLOAD_PER_ENT) + ent->payload = *payload; + ent->zero_copy_index = zero_copy_index; atomic_inc(&ring->queue_refs); return ent; @@ -1176,26 +1505,21 @@ static int fuse_uring_register(struct io_uring_cmd *cmd, struct fuse_ring *ring = smp_load_acquire(&fch->ring); struct fuse_ring_queue *queue; struct fuse_ring_ent *ent; - int err; unsigned int qid = READ_ONCE(cmd_req->qid); - err = -ENOMEM; - if (!ring) { - ring = fuse_uring_create(fch); - if (!ring) - return err; - } + if (!ring) + return -EINVAL; if (qid >= ring->nr_queues) { pr_info_ratelimited("fuse: Invalid ring qid %u\n", qid); return -EINVAL; } - queue = ring->queues[qid]; + queue = READ_ONCE(ring->queues[qid]); if (!queue) { - queue = fuse_uring_create_queue(ring, qid); - if (!queue) - return err; + queue = fuse_uring_create_queue(ring, qid, false, false); + if (IS_ERR(queue)) + return PTR_ERR(queue); } /* @@ -1210,6 +1534,110 @@ static int fuse_uring_register(struct io_uring_cmd *cmd, return fuse_uring_do_register(ent, cmd, issue_flags); } +static int fuse_uring_add_queue(struct io_uring_cmd *cmd, struct fuse_chan *fch) +{ + const struct fuse_uring_cmd_req *cmd_req = + io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req); + struct fuse_ring *ring = smp_load_acquire(&fch->ring); + unsigned int qid = READ_ONCE(cmd_req->qid); + uint64_t flags = READ_ONCE(cmd_req->flags); + struct fuse_ring_queue *queue; + bool zero_copy = flags & FUSE_URING_ZERO_COPY; + + if (!ring) + return -EINVAL; + + if (qid >= ring->nr_queues) { + pr_info_ratelimited("fuse: Invalid ring qid %u\n", qid); + return -EINVAL; + } + + if (flags & ~FUSE_URING_ADD_QUEUE_FLAGS) + return -EINVAL; + + if (zero_copy && !capable(CAP_SYS_ADMIN)) + return -EPERM; + + queue = fuse_uring_create_queue(ring, qid, zero_copy, true); + if (IS_ERR(queue)) + return PTR_ERR(queue); + + return 0; +} + +static int fuse_uring_add_bufpool(struct io_uring_cmd *cmd, + struct fuse_chan *fch) +{ + const struct fuse_uring_cmd_req *cmd_req = + io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req); + unsigned int qid = READ_ONCE(cmd_req->qid); + uint64_t flags = READ_ONCE(cmd_req->flags); + /* paired with the smp_store_release() in fuse_uring_create */ + struct fuse_ring *ring = smp_load_acquire(&fch->ring); + struct fuse_ring_queue *queue; + struct fuse_bufpool *pool; + uintptr_t pool_uaddr; + unsigned int pool_len, nr_bufs; + size_t pool_size, buf_size; + bool registered = cmd->flags & IORING_URING_CMD_FIXED; + + if (!ring || qid >= ring->nr_queues || flags) + return -EINVAL; + + /* reserved for future use, must be zero */ + if (READ_ONCE(cmd_req->bufpool.reserved)) + return -EINVAL; + + /* Pairs with smp_store_release() in fuse_uring_create_queue() */ + queue = smp_load_acquire(&ring->queues[qid]); + if (!queue) + return -EINVAL; + + pool_uaddr = READ_ONCE(cmd_req->bufpool.uaddr); + pool_len = READ_ONCE(cmd_req->bufpool.len); + + /* each buffer holds the max payload size */ + buf_size = queue->ring->max_payload_sz; + + nr_bufs = pool_len / buf_size; + if (!nr_bufs) + return -EINVAL; + + pool_size = struct_size(pool, free_map, BITS_TO_LONGS(nr_bufs)); + pool = kzalloc(pool_size, GFP_KERNEL_ACCOUNT); + if (!pool) + return -ENOMEM; + + pool->base_uaddr = pool_uaddr; + pool->buf_size = buf_size; + pool->nr_bufs = nr_bufs; + /* all buffers are free */ + bitmap_set(pool->free_map, 0, nr_bufs); + + /* + * A registered bufpool is reached through an io_uring fixed buffer, so + * the pool is registered iff this command was submitted with + * IORING_URING_CMD_FIXED. The registered buffer index is taken from + * sqe->buf_index. + */ + if (registered) { + pool->registered = true; + pool->registered_index = READ_ONCE(cmd->sqe->buf_index); + } + + spin_lock(&queue->lock); + if (queue->payload_mode != FUSE_PAYLOAD_UNSET) { + spin_unlock(&queue->lock); + kfree(pool); + return -EINVAL; + } + queue->bufpool = pool; + queue->payload_mode = FUSE_PAYLOAD_BUFPOOL; + spin_unlock(&queue->lock); + + return 0; +} + /* * Entry function from io_uring to handle the given passthrough command * (op code IORING_OP_URING_CMD) @@ -1237,23 +1665,30 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags) } fch = fud->chan; - /* Once a connection has io-uring enabled on it, it can't be disabled */ - if (!enable_uring && !fch->io_uring) { - pr_info_ratelimited("fuse-io-uring is disabled\n"); - return -EOPNOTSUPP; - } + /* + * The ring is sized from values negotiated by FUSE_INIT + * + * Pairs with smp_store_release() in fuse_chan_set_initialized() + */ + if (!smp_load_acquire(&fch->initialized)) + return -EAGAIN; if (fch->abort_with_err) return -ECONNABORTED; if (!fch->connected) return -ENOTCONN; - /* - * fuse_uring_register() needs the ring to be initialized, - * we need to know the max payload size - */ - if (!fch->initialized) - return -EAGAIN; + /* Once a connection has io-uring enabled on it, it can't be disabled */ + if (!enable_uring && !fch->io_uring) { + pr_info_ratelimited("fuse-io-uring is disabled by module parameter\n"); + return -EOPNOTSUPP; + } + + if (!fch->io_uring) { + pr_info_ratelimited( + "fuse-io-uring not enabled on this connection\n"); + return -EOPNOTSUPP; + } switch (cmd_op) { case FUSE_IO_URING_CMD_REGISTER: @@ -1274,6 +1709,18 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags) return err; } break; + case FUSE_IO_URING_CMD_ADD_QUEUE: + err = fuse_uring_add_queue(cmd, fch); + if (err) + pr_info_once("FUSE_IO_URING_CMD_ADD_QUEUE failed err=%d\n", + err); + return err; + case FUSE_IO_URING_CMD_ADD_BUFPOOL: + err = fuse_uring_add_bufpool(cmd, fch); + if (err) + pr_info_once("FUSE_IO_URING_ADD_BUFPOOL failed err=%d\n", + err); + return err; default: return -EINVAL; } @@ -1295,9 +1742,10 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw) int err; if (!tw.cancel) { - err = fuse_uring_prepare_send(ent, ent->fuse_req); + err = fuse_uring_prepare_send(ent, ent->fuse_req, issue_flags); if (err) { - if (!fuse_uring_get_next_fuse_req(ent, queue)) + if (!fuse_uring_get_next_fuse_req(ent, queue, + issue_flags)) return; err = 0; } @@ -1307,11 +1755,12 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw) spin_lock(&queue->lock); list_del_init(&ent->list); + fuse_uring_recycle_buffer(ent); spin_unlock(&queue->lock); io_uring_cmd_done(cmd, err, issue_flags); - fuse_uring_req_end(ent, ent->fuse_req, err); + fuse_uring_req_end(ent, ent->fuse_req, err, issue_flags); kfree(ent); if (atomic_dec_and_test(&queue->ring->queue_refs)) wake_up_all(&queue->ring->stop_waitq); @@ -1330,7 +1779,7 @@ static struct fuse_ring_queue *fuse_uring_task_to_queue(struct fuse_ring *ring) ring->nr_queues)) qid = 0; - queue = ring->queues[qid]; + queue = READ_ONCE(ring->queues[qid]); WARN_ONCE(!queue, "Missing queue for qid %d\n", qid); return queue; @@ -1368,15 +1817,16 @@ void fuse_uring_queue_fuse_req(struct fuse_iqueue *fiq, struct fuse_req *req) req->ring_queue = queue; ent = list_first_entry_or_null(&queue->ent_avail_queue, struct fuse_ring_ent, list); - if (ent) - fuse_uring_add_req_to_ring_ent(ent, req); - else - list_add_tail(&req->list, &queue->fuse_req_queue); - spin_unlock(&queue->lock); - if (ent) - fuse_uring_dispatch_ent(ent); + if (!ent || fuse_uring_prep_buffer(ent, req)) { + list_add_tail(&req->list, &queue->fuse_req_queue); + spin_unlock(&queue->lock); + return; + } + fuse_uring_add_req_to_ring_ent(ent, req); + spin_unlock(&queue->lock); + fuse_uring_dispatch_ent(ent); return; err_unlock: @@ -1424,10 +1874,9 @@ bool fuse_uring_queue_bq_req(struct fuse_req *req) */ req = list_first_entry_or_null(&queue->fuse_req_queue, struct fuse_req, list); - if (ent && req) { + if (ent && req && !fuse_uring_prep_buffer(ent, req)) { fuse_uring_add_req_to_ring_ent(ent, req); spin_unlock(&queue->lock); - fuse_uring_dispatch_ent(ent); } else { spin_unlock(&queue->lock); diff --git a/fs/fuse/dev_uring_i.h b/fs/fuse/dev_uring_i.h index 55f8d04e4b0b..263d0f8b9714 100644 --- a/fs/fuse/dev_uring_i.h +++ b/fs/fuse/dev_uring_i.h @@ -7,6 +7,8 @@ #ifndef _FS_FUSE_DEV_URING_I_H #define _FS_FUSE_DEV_URING_I_H +#include <linux/uio.h> + #include "fuse_dev_i.h" #ifdef CONFIG_FUSE_IO_URING @@ -36,11 +38,50 @@ enum fuse_ring_req_state { FRRS_RELEASED, }; +/* how a queue's payload buffers are provided */ +enum fuse_queue_payload_mode { + /* not yet committed (a bufpool may still be added) */ + FUSE_PAYLOAD_UNSET = 0, + /* each entry registers its own payload buffer */ + FUSE_PAYLOAD_PER_ENT, + /* each entry's payload buffer is assigned from a bufpool */ + FUSE_PAYLOAD_BUFPOOL, +}; + +struct fuse_bufpool { + bool registered; + + /* + * io_uring registered buffer table index for this pool, bound at + * ADD_BUFPOOL time. Only valid if the bufpool is registered + */ + u16 registered_index; + + /* starting uaddr of the bufpool */ + uintptr_t base_uaddr; + + /* size of each buffer in the pool */ + size_t buf_size; + + /* total number of buffers in the pool */ + unsigned int nr_bufs; + + /* bitmap tracking which buffers are free */ + unsigned long free_map[]; +}; + /** A fuse ring entry, part of the ring queue */ struct fuse_ring_ent { /* userspace buffer */ struct fuse_uring_req_header __user *headers; - void __user *payload; + struct iovec payload; + + /* buffer id in the pool, if bufpools are used. ignored otherwise */ + unsigned int buf_id; + + /* true if the request's pages are being zero-copied */ + bool zero_copied; + unsigned int zero_copy_index; /* the ring queue that owns the request */ struct fuse_ring_queue *queue; @@ -99,6 +140,14 @@ struct fuse_ring_queue { unsigned int active_background; bool stopped; + + /* how this queue's payload buffers are provided */ + enum fuse_queue_payload_mode payload_mode; + + /* only allocated when payload_mode == FUSE_PAYLOAD_BUFPOOL */ + struct fuse_bufpool *bufpool; + + bool zero_copy; }; /* @@ -135,6 +184,7 @@ struct fuse_ring { bool ready; }; +void fuse_uring_conn_init(struct fuse_chan *fch); void fuse_uring_stop_queues(struct fuse_ring *ring); void fuse_uring_abort_end_requests(struct fuse_ring *ring); int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags); @@ -174,6 +224,10 @@ static inline bool fuse_uring_ready(struct fuse_chan *fch) #else /* CONFIG_FUSE_IO_URING */ +static inline void fuse_uring_conn_init(struct fuse_chan *fch) +{ +} + static inline void fuse_uring_abort(struct fuse_chan *fch) { } diff --git a/fs/fuse/dir.c b/fs/fuse/dir.c index d4e0029810c0..e49b4e874b15 100644 --- a/fs/fuse/dir.c +++ b/fs/fuse/dir.c @@ -34,7 +34,7 @@ struct dentry_bucket { #define FUSE_HASH_BITS 5 #define FUSE_HASH_SIZE (1 << FUSE_HASH_BITS) static struct dentry_bucket dentry_hash[FUSE_HASH_SIZE]; -struct delayed_work dentry_tree_work; +static struct delayed_work dentry_tree_work; /* Minimum invalidation work queue frequency */ #define FUSE_DENTRY_INVAL_FREQ_MIN 5 @@ -96,6 +96,7 @@ static void fuse_advise_use_readdirplus(struct inode *dir) struct fuse_dentry { u64 time; + u64 epoch; union { struct rcu_head rcu; struct rb_node node; @@ -236,6 +237,13 @@ void fuse_dentry_tree_cleanup(void) WARN_ON_ONCE(!RB_EMPTY_ROOT(&dentry_hash[i].tree)); } +void fuse_dentry_set_epoch(struct dentry *dentry, u64 epoch) +{ + struct fuse_dentry *fd = dentry->d_fsdata; + + fd->epoch = epoch; +} + static inline void __fuse_dentry_settime(struct dentry *dentry, u64 time) { ((struct fuse_dentry *) dentry->d_fsdata)->time = time; @@ -387,10 +395,11 @@ static int fuse_dentry_revalidate(struct inode *dir, const struct qstr *name, struct fuse_mount *fm; struct fuse_conn *fc; struct fuse_inode *fi; + struct fuse_dentry *fd = entry->d_fsdata; int ret; fc = get_fuse_conn_super(dir->i_sb); - if (entry->d_time < atomic_read(&fc->epoch)) + if (fd->epoch < atomic_read(&fc->epoch)) goto invalid; inode = d_inode_rcu(entry); @@ -480,10 +489,10 @@ static int fuse_dentry_init(struct dentry *dentry) RB_CLEAR_NODE(&fd->node); dentry->d_fsdata = fd; /* - * Initialising d_time (epoch) to '0' ensures the dentry is invalid + * Initialising epoch to '0' ensures the dentry is invalid * if compared to fc->epoch, which is initialized to '1'. */ - dentry->d_time = 0; + fuse_dentry_set_epoch(dentry, 0); return 0; } @@ -641,7 +650,7 @@ static struct dentry *fuse_lookup(struct inode *dir, struct dentry *entry, goto out_err; entry = newent ? newent : entry; - entry->d_time = epoch; + fuse_dentry_set_epoch(entry, epoch); if (outarg_valid) fuse_change_entry_timeout(entry, &outarg); else @@ -898,7 +907,7 @@ static int fuse_create_open(struct mnt_idmap *idmap, struct inode *dir, } kfree(forget); d_instantiate(entry, inode); - entry->d_time = epoch; + fuse_dentry_set_epoch(entry, epoch); fuse_change_entry_timeout(entry, &outentry); fuse_dir_changed(dir); err = generic_file_open(inode, file); @@ -1028,10 +1037,10 @@ static struct dentry *create_new_entry(struct mnt_idmap *idmap, struct fuse_moun return d; if (d) { - d->d_time = epoch; + fuse_dentry_set_epoch(d, epoch); fuse_change_entry_timeout(d, &outarg); } else { - entry->d_time = epoch; + fuse_dentry_set_epoch(entry, epoch); fuse_change_entry_timeout(entry, &outarg); } fuse_dir_changed(dir); @@ -2169,10 +2178,8 @@ int fuse_do_setattr(struct mnt_idmap *idmap, struct dentry *dentry, filemap_invalidate_lock(mapping); fault_blocked = true; err = fuse_dax_break_layouts(inode, 0, -1); - if (err) { - filemap_invalidate_unlock(mapping); - return err; - } + if (err) + goto unlock; } if (attr->ia_valid & ATTR_OPEN) { @@ -2199,7 +2206,7 @@ int fuse_do_setattr(struct mnt_idmap *idmap, struct dentry *dentry, ATTR_TIMES_SET)) { err = write_inode_now(inode, true); if (err) - return err; + goto unlock; fuse_set_nowrite(inode); fuse_release_nowrite(inode); @@ -2290,6 +2297,9 @@ int fuse_do_setattr(struct mnt_idmap *idmap, struct dentry *dentry, */ if ((is_truncate || !is_wb) && S_ISREG(inode->i_mode) && oldsize != outarg.attr.size) { + if (outarg.attr.size > oldsize) + truncate_pagecache_range(inode, oldsize, + outarg.attr.size - 1); truncate_pagecache(inode, outarg.attr.size); invalidate_inode_pages2(mapping); } @@ -2307,6 +2317,7 @@ error: clear_bit(FUSE_I_SIZE_UNSTABLE, &fi->state); +unlock: if (fault_blocked) filemap_invalidate_unlock(mapping); return err; diff --git a/fs/fuse/file.c b/fs/fuse/file.c index f2c081f09791..8d6135a6108a 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -272,7 +272,7 @@ static int fuse_open(struct inode *inode, struct file *file) filemap_invalidate_lock(inode->i_mapping); err = fuse_dax_break_layouts(inode, 0, -1); if (err) - goto out_inode_unlock; + goto out_unlock; } if (is_wb_truncate || dax_truncate) @@ -296,9 +296,9 @@ static int fuse_open(struct inode *inode, struct file *file) else if (!(ff->open_flags & FOPEN_KEEP_CACHE)) invalidate_inode_pages2(inode->i_mapping); } +out_unlock: if (dax_truncate) filemap_invalidate_unlock(inode->i_mapping); -out_inode_unlock: if (is_wb_truncate || dax_truncate) inode_unlock(inode); @@ -605,6 +605,7 @@ void fuse_read_args_fill(struct fuse_io_args *ia, struct file *file, loff_t pos, args->out_argvar = true; args->out_numargs = 1; args->out_args[0].size = count; + args->zero_copy = ff->open_flags & FOPEN_IO_URING_ZERO_COPY; } static void fuse_release_user_pages(struct fuse_args_pages *ap, ssize_t nres, @@ -1153,6 +1154,7 @@ static void fuse_write_args_fill(struct fuse_io_args *ia, struct fuse_file *ff, args->out_numargs = 1; args->out_args[0].size = sizeof(ia->write.out); args->out_args[0].value = &ia->write.out; + args->zero_copy = ff->open_flags & FOPEN_IO_URING_ZERO_COPY; } static unsigned int fuse_write_flags(struct kiocb *iocb) @@ -1346,9 +1348,13 @@ static ssize_t fuse_perform_write(struct kiocb *iocb, struct iov_iter *ii) struct fuse_conn *fc = get_fuse_conn(inode); struct fuse_inode *fi = get_fuse_inode(inode); loff_t pos = iocb->ki_pos; + loff_t old_size = i_size_read(inode); int err = 0; ssize_t res = 0; + if (pos > old_size) + truncate_pagecache_range(inode, old_size, pos - 1); + if (inode->i_size < pos + iov_iter_count(ii)) set_bit(FUSE_I_SIZE_UNSTABLE, &fi->state); @@ -1787,13 +1793,14 @@ static ssize_t fuse_direct_write_iter(struct kiocb *iocb, struct iov_iter *from) { struct inode *inode = file_inode(iocb->ki_filp); struct address_space *mapping = inode->i_mapping; - loff_t pos = iocb->ki_pos; ssize_t res; bool exclusive; fuse_dio_lock(iocb, from, &exclusive); res = generic_write_checks(iocb, from); if (res > 0) { + loff_t pos = iocb->ki_pos; + task_io_account_write(res); if (!is_sync_kiocb(iocb)) { res = fuse_direct_IO(iocb, from); @@ -1808,7 +1815,7 @@ static ssize_t fuse_direct_write_iter(struct kiocb *iocb, struct iov_iter *from) /* * As in generic_file_direct_write(), invalidate after * write, to invalidate read-ahead cache that may have - * with the write. + * competed with the write. */ invalidate_inode_pages2_range(mapping, pos >> PAGE_SHIFT, @@ -2894,6 +2901,11 @@ static long fuse_file_fallocate(struct file *file, int mode, loff_t offset, /* we could have extended the file */ if (!(mode & FALLOC_FL_KEEP_SIZE)) { + loff_t oldsize = i_size_read(inode); + + if (offset + length > oldsize) + truncate_pagecache_range(inode, oldsize, + offset + length - 1); if (fuse_write_update_attr(inode, offset + length, length)) file_update_time(file); } diff --git a/fs/fuse/fuse_dev_i.h b/fs/fuse/fuse_dev_i.h index 668c8391d61c..4b412a76225f 100644 --- a/fs/fuse/fuse_dev_i.h +++ b/fs/fuse/fuse_dev_i.h @@ -38,6 +38,8 @@ struct fuse_iqueue; * @FR_PRIVATE: request is on private list * @FR_ASYNC: request is asynchronous * @FR_URING: request is handled through fuse-io-uring + * @FR_SYNC_WAKEUP: use synchronous wakeup when queueing this request to + * give the scheduler a hint about the waker task */ enum fuse_req_flag { FR_ISREPLY, @@ -53,6 +55,7 @@ enum fuse_req_flag { FR_PRIVATE, FR_ASYNC, FR_URING, + FR_SYNC_WAKEUP, }; /** @@ -325,6 +328,8 @@ struct fuse_copy_state { bool write:1; bool move_folios:1; bool is_uring:1; + /* set when the payload is zero-copied. folios are filled in place */ + bool skip_folio_copy:1; struct { unsigned int copied_sz; /* copied size into the user buffer */ } ring; diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h index 85f738c53122..c8d4c5f3af7e 100644 --- a/fs/fuse/fuse_i.h +++ b/fs/fuse/fuse_i.h @@ -1054,6 +1054,8 @@ u64 fuse_time_to_jiffies(u64 sec, u32 nsec); void fuse_change_entry_timeout(struct dentry *entry, struct fuse_entry_out *o); +void fuse_dentry_set_epoch(struct dentry *dentry, u64 epoch); + /* * Initialize fuse_conn */ diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index d975073c6029..e9552be3637b 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -791,6 +791,9 @@ static int fuse_opt_fd(struct fs_context *fsc, struct file *file) { struct fuse_fs_context *ctx = fsc->fs_private; + if (ctx->fud) + return invalfc(fsc, "Multiple fd specified"); + if (file->f_op != &fuse_dev_operations) return invalfc(fsc, "fd is not a fuse device"); /* @@ -1272,6 +1275,7 @@ static void process_init_reply(struct fuse_args *args, int error) struct fuse_mount *fm = ia->fm; struct fuse_conn *fc = fm->fc; struct fuse_init_out *arg = &ia->out; + bool io_uring_enabled = false; bool ok = true; if (error || arg->major != FUSE_KERNEL_VERSION) @@ -1402,7 +1406,7 @@ static void process_init_reply(struct fuse_args *args, int error) ok = false; } if (flags & FUSE_OVER_IO_URING && fuse_uring_enabled()) - fuse_chan_io_uring_enable(fc->chan); + io_uring_enabled = true; if (flags & FUSE_REQUEST_TIMEOUT) timeout = arg->request_timeout; @@ -1416,6 +1420,7 @@ static void process_init_reply(struct fuse_args *args, int error) fm->sb->s_bdi->ra_pages = min(fm->sb->s_bdi->ra_pages, ra_pages); + fm->sb->s_bdi->io_pages = fc->max_pages; fc->minor = arg->minor; fc->max_write = arg->minor < 5 ? 4096 : arg->max_write; fc->max_write = max_t(unsigned, 4096, fc->max_write); @@ -1432,6 +1437,7 @@ static void process_init_reply(struct fuse_args *args, int error) .minor = fc->minor, .max_write = fc->max_write, .max_pages = fc->max_pages, + .io_uring_enabled = io_uring_enabled, }; fuse_chan_set_initialized(fc->chan, &cp); } @@ -1474,12 +1480,8 @@ static struct fuse_init_args *fuse_new_init(struct fuse_mount *fm) if (IS_ENABLED(CONFIG_FUSE_PASSTHROUGH)) flags |= FUSE_PASSTHROUGH; - /* - * This is just an information flag for fuse server. No need to check - * the reply - server is either sending IORING_OP_URING_CMD or not. - */ if (fuse_uring_enabled()) - flags |= FUSE_OVER_IO_URING; + flags |= FUSE_OVER_IO_URING | FUSE_HAS_IO_URING_BUFPOOL; ia->in.flags = flags; ia->in.flags2 = flags >> 32; @@ -1639,6 +1641,8 @@ static int fuse_fill_super_submount(struct super_block *sb, fuse_fill_attr_from_inode(&root_attr, parent_fi); root = fuse_iget(sb, parent_fi->nodeid, 0, &root_attr, 0, 0, fuse_get_evict_ctr(fm->fc)); + if (!root) + return -ENOMEM; /* * This inode is just a duplicate, so it is not looked up and * its nlookup should not be incremented. fuse_iget() does diff --git a/fs/fuse/readdir.c b/fs/fuse/readdir.c index 0e1321491747..5ca87151d70d 100644 --- a/fs/fuse/readdir.c +++ b/fs/fuse/readdir.c @@ -260,7 +260,7 @@ retry: } if (fc->readdirplus_auto) set_bit(FUSE_I_INIT_RDPLUS, &get_fuse_inode(inode)->state); - dentry->d_time = epoch; + fuse_dentry_set_epoch(dentry, epoch); fuse_change_entry_timeout(dentry, o); dput(dentry); diff --git a/fs/fuse/req_timeout.c b/fs/fuse/req_timeout.c index 6cc6fc491343..95a1acd7bc08 100644 --- a/fs/fuse/req_timeout.c +++ b/fs/fuse/req_timeout.c @@ -128,18 +128,12 @@ static void set_request_timeout(struct fuse_chan *fch, unsigned int timeout) void fuse_init_server_timeout(struct fuse_chan *fch, unsigned int timeout) { - if (!timeout && !fuse_max_req_timeout && !fuse_default_req_timeout) - return; - if (!timeout) timeout = fuse_default_req_timeout; - if (fuse_max_req_timeout) { - if (timeout) - timeout = min(fuse_max_req_timeout, timeout); - else - timeout = fuse_max_req_timeout; - } + timeout = min_not_zero(timeout, fuse_max_req_timeout); + if (!timeout) + return; timeout = max(FUSE_TIMEOUT_TIMER_FREQ, timeout); diff --git a/include/linux/io_uring/cmd.h b/include/linux/io_uring/cmd.h index 331dcbefe72f..42801f0b6456 100644 --- a/include/linux/io_uring/cmd.h +++ b/include/linux/io_uring/cmd.h @@ -91,6 +91,15 @@ struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd, bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd, struct io_br_sel *sel, unsigned int issue_flags); +int io_buffer_register_request(struct io_uring_cmd *cmd, struct request *rq, + void (*release)(void *), unsigned int index, + unsigned int issue_flags); +int io_buffer_register_bvec(struct io_uring_cmd *cmd, const struct bio_vec *bvs, + unsigned int nr_bvecs, void (*release)(void *), + void *priv, u8 dir, unsigned int index, + unsigned int issue_flags); +int io_buffer_unregister(struct io_uring_cmd *cmd, unsigned int index, + unsigned int issue_flags); #else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, @@ -133,6 +142,29 @@ static inline bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd, { return true; } +static inline int io_buffer_register_request(struct io_uring_cmd *cmd, + struct request *rq, + void (*release)(void *), + unsigned int index, + unsigned int issue_flags) +{ + return -EOPNOTSUPP; +} +static inline int io_buffer_register_bvec(struct io_uring_cmd *cmd, + const struct bio_vec *bvs, + unsigned int nr_bvecs, + void (*release)(void *), void *priv, + u8 dir, unsigned int index, + unsigned int issue_flags) +{ + return -EOPNOTSUPP; +} +static inline int io_buffer_unregister(struct io_uring_cmd *cmd, + unsigned int index, + unsigned int issue_flags) +{ + return -EOPNOTSUPP; +} #endif static inline struct io_uring_cmd *io_uring_cmd_from_tw(struct io_tw_req tw_req) @@ -182,10 +214,4 @@ static inline void io_uring_cmd_done32(struct io_uring_cmd *ioucmd, s32 ret, return __io_uring_cmd_done(ioucmd, ret, res2, issue_flags, true); } -int io_buffer_register_bvec(struct io_uring_cmd *cmd, struct request *rq, - void (*release)(void *), unsigned int index, - unsigned int issue_flags); -int io_buffer_unregister_bvec(struct io_uring_cmd *cmd, unsigned int index, - unsigned int issue_flags); - #endif /* _LINUX_IO_URING_CMD_H */ diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 6b9e9c669bb3..39629ee77b91 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -53,6 +53,11 @@ enum io_uring_cmd_flags { IO_URING_F_COMPAT = (1 << 12), }; +enum { + IO_BUF_DEST = 1 << ITER_DEST, + IO_BUF_SOURCE = 1 << ITER_SOURCE, +}; + struct iou_loop_params; struct io_wq_work_node { diff --git a/include/uapi/linux/fuse.h b/include/uapi/linux/fuse.h index c13e1f9a2f12..7435e09c87fe 100644 --- a/include/uapi/linux/fuse.h +++ b/include/uapi/linux/fuse.h @@ -240,6 +240,14 @@ * - add FUSE_COPY_FILE_RANGE_64 * - add struct fuse_copy_file_range_out * - add FUSE_NOTIFY_PRUNE + * + * 7.46 + * - add FUSE_IO_URING_CMD_ADD_QUEUE + * - add FUSE_HAS_IO_URING_BUFPOOL + * - add fuse_uring_cmd_req bufpool struct + * - add bufpool offset field to fuse_uring_ent_in_out struct + * - add FUSE_URING_ZERO_COPY, FUSE_URING_ENT_ZERO_COPY, and + * FOPEN_IO_URING_ZERO_COPY flag */ #ifndef _LINUX_FUSE_H @@ -275,7 +283,7 @@ #define FUSE_KERNEL_VERSION 7 /** Minor version number of this interface */ -#define FUSE_KERNEL_MINOR_VERSION 45 +#define FUSE_KERNEL_MINOR_VERSION 46 /** The node ID of the root inode */ #define FUSE_ROOT_ID 1 @@ -383,6 +391,12 @@ struct fuse_file_lock { * FOPEN_NOFLUSH: don't flush data cache on close (unless FUSE_WRITEBACK_CACHE) * FOPEN_PARALLEL_DIRECT_WRITES: Allow concurrent direct writes on the same inode * FOPEN_PASSTHROUGH: passthrough read/write io for this open file + * FOPEN_IO_URING_ZERO_COPY: use io-uring zero-copy for reads/writes on this + * open file. Honored only when the serving io-uring + * queue was set up for zero-copy + * (FUSE_URING_ZERO_COPY) and the request carries page + * payload. Otherwise reads/writes fall back to + * copying. */ #define FOPEN_DIRECT_IO (1 << 0) #define FOPEN_KEEP_CACHE (1 << 1) @@ -392,6 +406,7 @@ struct fuse_file_lock { #define FOPEN_NOFLUSH (1 << 5) #define FOPEN_PARALLEL_DIRECT_WRITES (1 << 6) #define FOPEN_PASSTHROUGH (1 << 7) +#define FOPEN_IO_URING_ZERO_COPY (1 << 8) /** * INIT request/reply flags @@ -448,6 +463,7 @@ struct fuse_file_lock { * FUSE_OVER_IO_URING: Indicate that client supports io-uring * FUSE_REQUEST_TIMEOUT: kernel supports timing out requests. * init_out.request_timeout contains the timeout (in secs) + * FUSE_HAS_IO_URING_BUFPOOL: kernel supports io-uring buffer pools */ #define FUSE_ASYNC_READ (1 << 0) #define FUSE_POSIX_LOCKS (1 << 1) @@ -495,6 +511,7 @@ struct fuse_file_lock { #define FUSE_ALLOW_IDMAP (1ULL << 40) #define FUSE_OVER_IO_URING (1ULL << 41) #define FUSE_REQUEST_TIMEOUT (1ULL << 42) +#define FUSE_HAS_IO_URING_BUFPOOL (1ULL << 43) /** * CUSE INIT request/reply flags @@ -1251,6 +1268,13 @@ struct fuse_supp_groups { #define FUSE_URING_IN_OUT_HEADER_SZ 128 #define FUSE_URING_OP_IN_OUT_SZ 128 +/** + * fuse_uring_ent_in_out flags + * + * FUSE_URING_ENT_ZERO_COPY: Set if the ent's payload is zero-copied + */ +#define FUSE_URING_ENT_ZERO_COPY (1 << 0) + /* Used as part of the fuse_uring_req_header */ struct fuse_uring_ent_in_out { uint64_t flags; @@ -1263,7 +1287,9 @@ struct fuse_uring_ent_in_out { /* size of user payload buffer */ uint32_t payload_sz; - uint32_t padding; + + /* Offset into the bufpool, if bufpools are used */ + uint32_t offset; uint64_t reserved; }; @@ -1292,8 +1318,22 @@ enum fuse_uring_cmd { /* commit fuse request result and fetch next request */ FUSE_IO_URING_CMD_COMMIT_AND_FETCH = 2, + + /* add a queue */ + FUSE_IO_URING_CMD_ADD_QUEUE = 3, + + /* add a bufpool to a queue */ + FUSE_IO_URING_CMD_ADD_BUFPOOL = 4, }; +/* + * fuse_uring_cmd_req flags for FUSE_IO_URING_CMD_ADD_QUEUE + * + * FUSE_URING_ZERO_COPY is only supported for queues with bufpools on privileged + * servers + */ +#define FUSE_URING_ZERO_COPY (1 << 0) + /** * In the 80B command area of the SQE. */ @@ -1306,6 +1346,25 @@ struct fuse_uring_cmd_req { /* queue the command is for (queue index) */ uint16_t qid; uint8_t padding[6]; + + union { + struct { + /* base address of bufpool */ + uint64_t uaddr; + uint32_t len; + uint32_t reserved; + } bufpool; + + /* + * Index of this entry's slot in the server's io_uring + * registered buffer table, where the kernel registers the + * request's pages for zero-copy. Set for + * FUSE_IO_URING_CMD_REGISTER cmds only, and only on queues + * created with FUSE_URING_ZERO_COPY. On a non-zero-copy queue + * this must be 0 + */ + uint16_t ent_zero_copy_buf_index; + }; }; #endif /* _LINUX_FUSE_H */ diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 76f049e29aa2..61053421d809 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3236,7 +3236,7 @@ static int __init io_uring_init(void) io_uring_optable_init(); /* imu->dir is u8 */ - BUILD_BUG_ON((IO_IMU_DEST | IO_IMU_SOURCE) > U8_MAX); + BUILD_BUG_ON((IO_BUF_DEST | IO_BUF_SOURCE) > U8_MAX); /* * Allow user copy in the per-command field, which starts after the diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 174f74cbbf60..51b46e624ddd 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -912,7 +912,7 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, imu->release = io_release_ubuf; imu->priv = imu; imu->flags = 0; - imu->dir = IO_IMU_DEST | IO_IMU_SOURCE; + imu->dir = IO_BUF_DEST | IO_BUF_SOURCE; if (coalesced) imu->folio_shift = data.folio_shift; refcount_set(&imu->refs, 1); @@ -1015,71 +1015,124 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, return ret; } -int io_buffer_register_bvec(struct io_uring_cmd *cmd, struct request *rq, - void (*release)(void *), unsigned int index, - unsigned int issue_flags) +static struct io_mapped_ubuf *io_kernel_buffer_init(struct io_ring_ctx *ctx, + unsigned int nr_bvecs, + unsigned int total_bytes, + u8 dir, + void (*release)(void *), + void *priv, + unsigned int index) { - struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx; struct io_rsrc_data *data = &ctx->buf_table; - struct req_iterator rq_iter; struct io_mapped_ubuf *imu; struct io_rsrc_node *node; - struct bio_vec bv; - unsigned int nr_bvecs = 0; - int ret = 0; - io_ring_submit_lock(ctx, issue_flags); - if (index >= data->nr) { - ret = -EINVAL; - goto unlock; - } + if (index >= data->nr) + return ERR_PTR(-EINVAL); index = array_index_nospec(index, data->nr); - if (data->nodes[index]) { - ret = -EBUSY; - goto unlock; - } + if (data->nodes[index]) + return ERR_PTR(-EBUSY); node = io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER); - if (!node) { - ret = -ENOMEM; - goto unlock; - } + if (!node) + return ERR_PTR(-ENOMEM); - /* - * blk_rq_nr_phys_segments() may overestimate the number of bvecs - * but avoids needing to iterate over the bvecs - */ - imu = io_alloc_imu(ctx, blk_rq_nr_phys_segments(rq)); + imu = io_alloc_imu(ctx, nr_bvecs); if (!imu) { io_cache_free(&ctx->node_cache, node); - ret = -ENOMEM; - goto unlock; + return ERR_PTR(-ENOMEM); } imu->ubuf = 0; - imu->len = blk_rq_bytes(rq); + imu->len = total_bytes; imu->folio_shift = PAGE_SHIFT; + imu->nr_bvecs = nr_bvecs; refcount_set(&imu->refs, 1); imu->release = release; - imu->priv = rq; + imu->priv = priv; + imu->dir = dir; imu->flags = IO_REGBUF_F_KBUF; - imu->dir = 1 << rq_data_dir(rq); + node->buf = imu; + data->nodes[index] = node; + + return imu; +} + +int io_buffer_register_request(struct io_uring_cmd *cmd, struct request *rq, + void (*release)(void *), unsigned int index, + unsigned int issue_flags) +{ + struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx; + struct req_iterator rq_iter; + struct io_mapped_ubuf *imu; + struct bio_vec bv; + /* + * blk_rq_nr_phys_segments() may overestimate the number of bvecs + * but avoids needing to iterate over the bvecs + */ + unsigned int nr_bvecs = blk_rq_nr_phys_segments(rq); + unsigned int total_bytes = blk_rq_bytes(rq); + int ret = 0; + + io_ring_submit_lock(ctx, issue_flags); + + imu = io_kernel_buffer_init(ctx, nr_bvecs, total_bytes, + 1 << rq_data_dir(rq), release, rq, index); + if (IS_ERR(imu)) { + ret = PTR_ERR(imu); + goto unlock; + } + + nr_bvecs = 0; rq_for_each_bvec(bv, rq, rq_iter) imu->bvec[nr_bvecs++] = bv; imu->nr_bvecs = nr_bvecs; - node->buf = imu; - data->nodes[index] = node; +unlock: + io_ring_submit_unlock(ctx, issue_flags); + return ret; +} +EXPORT_SYMBOL_GPL(io_buffer_register_request); + +/* + * bvs is copied internally. caller may free it on return. + */ +int io_buffer_register_bvec(struct io_uring_cmd *cmd, const struct bio_vec *bvs, + unsigned int nr_bvecs, void (*release)(void *), + void *priv, u8 dir, unsigned int index, + unsigned int issue_flags) +{ + struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx; + struct io_mapped_ubuf *imu; + struct bio_vec *bvec; + unsigned int i, total_bytes = 0; + int ret = 0; + + for (i = 0; i < nr_bvecs; i++) + total_bytes += bvs[i].bv_len; + + io_ring_submit_lock(ctx, issue_flags); + imu = io_kernel_buffer_init(ctx, nr_bvecs, total_bytes, dir, release, + priv, index); + if (IS_ERR(imu)) { + ret = PTR_ERR(imu); + goto unlock; + } + + bvec = imu->bvec; + for (i = 0; i < nr_bvecs; i++) + bvec[i] = bvs[i]; + unlock: io_ring_submit_unlock(ctx, issue_flags); return ret; } EXPORT_SYMBOL_GPL(io_buffer_register_bvec); -int io_buffer_unregister_bvec(struct io_uring_cmd *cmd, unsigned int index, - unsigned int issue_flags) +int io_buffer_unregister(struct io_uring_cmd *cmd, unsigned int index, + unsigned int issue_flags) { struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx; struct io_rsrc_data *data = &ctx->buf_table; @@ -1109,7 +1162,7 @@ unlock: io_ring_submit_unlock(ctx, issue_flags); return ret; } -EXPORT_SYMBOL_GPL(io_buffer_unregister_bvec); +EXPORT_SYMBOL_GPL(io_buffer_unregister); static int validate_fixed_range(u64 buf_addr, size_t len, const struct io_mapped_ubuf *imu) diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index eacfdb70f203..9ef88383b363 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -26,11 +26,6 @@ struct io_rsrc_node { }; enum { - IO_IMU_DEST = 1 << ITER_DEST, - IO_IMU_SOURCE = 1 << ITER_SOURCE, -}; - -enum { IO_REGBUF_F_KBUF = 1, }; diff --git a/tools/testing/selftests/filesystems/fuse/.gitignore b/tools/testing/selftests/filesystems/fuse/.gitignore index 3e72e742d08e..fb51603fe419 100644 --- a/tools/testing/selftests/filesystems/fuse/.gitignore +++ b/tools/testing/selftests/filesystems/fuse/.gitignore @@ -1,3 +1,4 @@ # SPDX-License-Identifier: GPL-2.0-only fuse_mnt fusectl_test +write_extend_eof_test diff --git a/tools/testing/selftests/filesystems/fuse/Makefile b/tools/testing/selftests/filesystems/fuse/Makefile index f47141484275..95a1ee947ca7 100644 --- a/tools/testing/selftests/filesystems/fuse/Makefile +++ b/tools/testing/selftests/filesystems/fuse/Makefile @@ -3,6 +3,7 @@ CFLAGS += -Wall -O2 -g $(KHDR_INCLUDES) TEST_GEN_PROGS := fusectl_test +TEST_GEN_PROGS += write_extend_eof_test TEST_GEN_FILES := fuse_mnt # fuse_acl_cache_test requires libfuse3; add it only when the library is present. @@ -14,6 +15,8 @@ endif include ../../lib.mk +$(OUTPUT)/write_extend_eof_test: LDLIBS += -lpthread + VAR_CFLAGS := $(shell pkg-config fuse --cflags 2>/dev/null) ifeq ($(VAR_CFLAGS),) VAR_CFLAGS := -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse diff --git a/tools/testing/selftests/filesystems/fuse/write_extend_eof_test.c b/tools/testing/selftests/filesystems/fuse/write_extend_eof_test.c new file mode 100644 index 000000000000..ca6ce6eca382 --- /dev/null +++ b/tools/testing/selftests/filesystems/fuse/write_extend_eof_test.c @@ -0,0 +1,368 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Regression test for the fuse write-extend partial-EOF-page zeroing bug. + * + * A buffered write that extends i_size past a non-page-aligned EOF must zero + * the tail of the old last page. If an application has mmap'd that page and + * stored into the post-EOF region (undefined until the file grows), the + * now-in-bounds tail must read back as zero, not as the stale stored bytes. + * + * The bug is exposed on a non-writeback_cache server that keeps the page cache + * across the write (FOPEN_KEEP_CACHE without FOPEN_DIRECT_IO). This test is a + * raw /dev/fuse server in that mode; the backing data is always zero in the + * hole, so any non-zero byte a read sees is stale page-cache data. + * + * Requires root to mount fuse. + */ +#define _GNU_SOURCE +#include <errno.h> +#include <fcntl.h> +#include <linux/falloc.h> +#include <pthread.h> +#include <stdint.h> +#include <string.h> +#include <unistd.h> +#include <sys/mman.h> +#include <sys/mount.h> +#include <sys/stat.h> +#include <sys/uio.h> +#include <linux/fuse.h> + +#include "../../kselftest_harness.h" + +#define FUSE_ROOT_ID 1 +#define FILE_INO 2 +#define MAX_WRITE (128 * 1024) +#define BACKING_SIZE (4 * 1024 * 1024) +#define POLLUTE 0xee + +/* Server-side state, shared with the responder thread. */ +struct server { + int fd; + unsigned char backing[BACKING_SIZE]; /* authoritative bytes */ + uint64_t size; +}; + +static void reply(int fd, uint64_t unique, int error, void *data, size_t len) +{ + struct fuse_out_header oh = { + .len = sizeof(oh) + (data ? len : 0), + .error = error, + .unique = unique, + }; + struct iovec iov[2] = { { &oh, sizeof(oh) }, { data, len } }; + + /* Errors here are teardown races (device closed on unmount); ignore. */ + if (writev(fd, iov, data ? 2 : 1) < 0) + return; +} + +static void fill_attr(struct fuse_attr *a, uint64_t ino, uint32_t mode, + uint64_t size) +{ + memset(a, 0, sizeof(*a)); + a->ino = ino; + a->mode = mode; + a->nlink = 1; + a->size = size; + a->blksize = sysconf(_SC_PAGESIZE); +} + +static void *server_thread(void *arg) +{ + struct server *s = arg; + static char buf[MAX_WRITE + 4096]; + + for (;;) { + ssize_t n = read(s->fd, buf, sizeof(buf)); + struct fuse_in_header *ih = (void *)buf; + + if (n < 0) { + if (errno == EINTR || errno == EAGAIN) + continue; + return NULL; /* device closed on unmount */ + } + if (n < (ssize_t)sizeof(*ih)) + continue; + + switch (ih->opcode) { + case FUSE_INIT: { + struct fuse_init_in *in = (void *)(ih + 1); + struct fuse_init_out out = {0}; + + /* No FUSE_WRITEBACK_CACHE: the exposed configuration. */ + out.major = FUSE_KERNEL_VERSION; + out.minor = FUSE_KERNEL_MINOR_VERSION; + out.max_readahead = in->max_readahead; + out.max_write = MAX_WRITE; + out.max_background = 16; + out.congestion_threshold = 12; + out.flags = FUSE_MAX_PAGES; + out.max_pages = MAX_WRITE / sysconf(_SC_PAGESIZE); + reply(s->fd, ih->unique, 0, &out, sizeof(out)); + break; + } + case FUSE_GETATTR: { + struct fuse_attr_out out = {0}; + int root = ih->nodeid == FUSE_ROOT_ID; + + out.attr_valid = 3600; + fill_attr(&out.attr, ih->nodeid, + root ? (S_IFDIR | 0755) : (S_IFREG | 0644), + root ? 0 : s->size); + reply(s->fd, ih->unique, 0, &out, sizeof(out)); + break; + } + case FUSE_LOOKUP: { + struct fuse_entry_out out = {0}; + + out.nodeid = FILE_INO; + out.attr_valid = 3600; + out.entry_valid = 3600; + fill_attr(&out.attr, FILE_INO, S_IFREG | 0644, s->size); + reply(s->fd, ih->unique, 0, &out, sizeof(out)); + break; + } + case FUSE_OPEN: + case FUSE_OPENDIR: { + struct fuse_open_out out = {0}; + + /* Keep the cache across the write, but not direct I/O. */ + out.open_flags = FOPEN_KEEP_CACHE; + reply(s->fd, ih->unique, 0, &out, sizeof(out)); + break; + } + case FUSE_READ: { + struct fuse_read_in *in = (void *)(ih + 1); + uint64_t off = in->offset; + uint32_t size = in->size; + + if (off >= BACKING_SIZE) + size = 0; + else if (off + size > BACKING_SIZE) + size = BACKING_SIZE - off; + reply(s->fd, ih->unique, 0, s->backing + off, size); + break; + } + case FUSE_WRITE: { + struct fuse_write_in *in = (void *)(ih + 1); + struct fuse_write_out out = {0}; + uint64_t off = in->offset; + uint32_t size = in->size; + + if (off < BACKING_SIZE) { + uint32_t c = size; + + if (off + c > BACKING_SIZE) + c = BACKING_SIZE - off; + memcpy(s->backing + off, in + 1, c); + if (off + c > s->size) + s->size = off + c; + } + out.size = size; + reply(s->fd, ih->unique, 0, &out, sizeof(out)); + break; + } + case FUSE_SETATTR: { + struct fuse_setattr_in *in = (void *)(ih + 1); + struct fuse_attr_out out = {0}; + + if ((in->valid & FATTR_SIZE) && in->size <= BACKING_SIZE) { + if (in->size > s->size) + memset(s->backing + s->size, 0, + in->size - s->size); + s->size = in->size; + } + out.attr_valid = 3600; + fill_attr(&out.attr, ih->nodeid, S_IFREG | 0644, s->size); + reply(s->fd, ih->unique, 0, &out, sizeof(out)); + break; + } + case FUSE_FALLOCATE: { + struct fuse_fallocate_in *in = (void *)(ih + 1); + uint64_t end = in->offset + in->length; + + /* Only plain (size-extending) fallocate is used here. */ + if (!(in->mode & FALLOC_FL_KEEP_SIZE) && + end <= BACKING_SIZE && end > s->size) { + memset(s->backing + s->size, 0, end - s->size); + s->size = end; + } + reply(s->fd, ih->unique, 0, NULL, 0); + break; + } + case FUSE_FLUSH: + case FUSE_RELEASE: + case FUSE_RELEASEDIR: + case FUSE_FSYNC: + case FUSE_ACCESS: + reply(s->fd, ih->unique, 0, NULL, 0); + break; + case FUSE_FORGET: + break; + default: + reply(s->fd, ih->unique, -EOPNOTSUPP, NULL, 0); + break; + } + } +} + +FIXTURE(fuse) +{ + struct server *srv; + pthread_t thread; + char dir[64]; + long page; /* runtime page size */ + off_t eof; /* mid-page EOF, page-relative */ + int fd; /* open test file */ + char *map; /* mmap of the EOF page */ + int mounted; +}; + +FIXTURE_SETUP(fuse) +{ + char opts[128]; + pthread_t t; + + if (geteuid() != 0) + SKIP(return, "need root to mount fuse"); + + self->page = sysconf(_SC_PAGESIZE); + self->fd = -1; + self->map = MAP_FAILED; + + self->srv = mmap(NULL, sizeof(*self->srv), PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_ANONYMOUS, -1, 0); + ASSERT_NE(MAP_FAILED, self->srv); + + self->srv->fd = open("/dev/fuse", O_RDWR); + ASSERT_GE(self->srv->fd, 0); + + strcpy(self->dir, "/tmp/fuse_weof_XXXXXX"); + ASSERT_NE(NULL, mkdtemp(self->dir)); + + snprintf(opts, sizeof(opts), + "fd=%d,rootmode=40000,user_id=0,group_id=0", + self->srv->fd); + ASSERT_EQ(0, mount("fuse", self->dir, "fuse", 0, opts)); + self->mounted = 1; + + ASSERT_EQ(0, pthread_create(&t, NULL, server_thread, self->srv)); + self->thread = t; +} + +FIXTURE_TEARDOWN(fuse) +{ + if (self->map != MAP_FAILED) + munmap(self->map, self->page); + if (self->fd >= 0) + close(self->fd); + if (self->mounted) + umount2(self->dir, MNT_DETACH); + if (self->srv && self->srv != MAP_FAILED) { + if (self->srv->fd > 0) + close(self->srv->fd); + munmap(self->srv, sizeof(*self->srv)); + } + if (self->dir[0]) + rmdir(self->dir); +} + +/* + * Create the test file with a mid-page EOF and mmap-store POLLUTE into its + * post-EOF tail (a legal store, undefined until the file grows). Leaves the + * file open and the EOF page mapped in the fixture for the caller to extend. + */ +static void pollute_eof_tail(struct __test_metadata *_metadata, + FIXTURE_DATA(fuse) * self) +{ + off_t eof = 2 * self->page + self->page / 4; + char path[128]; + char *buf; + + snprintf(path, sizeof(path), "%s/file", self->dir); + self->fd = open(path, O_RDWR | O_CREAT | O_TRUNC, 0644); + ASSERT_GE(self->fd, 0); + self->eof = eof; + + buf = malloc(eof); + ASSERT_NE(NULL, buf); + memset(buf, 'A', eof); + ASSERT_EQ(eof, pwrite(self->fd, buf, eof, 0)); + free(buf); + + self->map = mmap(NULL, self->page, PROT_READ | PROT_WRITE, MAP_SHARED, + self->fd, eof & ~(self->page - 1)); + ASSERT_NE(MAP_FAILED, self->map); + memset(self->map + (eof & (self->page - 1)), POLLUTE, + self->page - (eof & (self->page - 1))); +} + +/* Assert the old post-EOF tail [eof, end of its page) now reads back as zero. */ +static void assert_tail_zeroed(struct __test_metadata *_metadata, + FIXTURE_DATA(fuse) * self) +{ + off_t base = self->eof & ~(self->page - 1); + char *tail = malloc(self->page); + int i; + + ASSERT_NE(NULL, tail); + ASSERT_EQ(self->page, pread(self->fd, tail, self->page, base)); + for (i = self->eof & (self->page - 1); i < self->page; i++) + ASSERT_EQ(0, tail[i]); + free(tail); +} + +/* Basic: pollute the post-EOF tail, extend past it by a later write. */ +TEST_F(fuse, write_extend) +{ + pollute_eof_tail(_metadata, self); + ASSERT_EQ(4, pwrite(self->fd, "data", 4, 5 * self->page + self->page / 3)); + assert_tail_zeroed(_metadata, self); +} + +/* Extend via ftruncate() rather than a write. */ +TEST_F(fuse, ftruncate_extend) +{ + pollute_eof_tail(_metadata, self); + ASSERT_EQ(0, ftruncate(self->fd, 8 * self->page)); + assert_tail_zeroed(_metadata, self); +} + +/* Extend via fallocate() starting at the old EOF. */ +TEST_F(fuse, fallocate_extend) +{ + pollute_eof_tail(_metadata, self); + ASSERT_EQ(0, fallocate(self->fd, 0, self->eof, 4 * self->page)); + assert_tail_zeroed(_metadata, self); +} + +/* A write landing inside the old EOF page must not clobber its own data. */ +TEST_F(fuse, extend_into_eof_page_preserves_data) +{ + off_t base, wr; + char *buf, *rd; + int i; + + pollute_eof_tail(_metadata, self); + base = self->eof & ~(self->page - 1); + wr = base + 3 * self->page / 4; /* starts in the EOF page */ + + buf = malloc(2 * self->page); + ASSERT_NE(NULL, buf); + memset(buf, 'B', 2 * self->page); + ASSERT_EQ(2 * self->page, pwrite(self->fd, buf, 2 * self->page, wr)); + free(buf); + + rd = malloc(self->page); + ASSERT_NE(NULL, rd); + ASSERT_EQ(self->page, pread(self->fd, rd, self->page, base)); + /* [eof, wr) is hole -> zero; [wr, page) is written data -> 'B'. */ + for (i = self->eof & (self->page - 1); i < wr - base; i++) + ASSERT_EQ(0, rd[i]); + for (i = wr - base; i < self->page; i++) + ASSERT_EQ('B', rd[i]); + free(rd); +} + +TEST_HARNESS_MAIN |