kernel/git/next/linux-next.git/lib/percpu-refcount.c, branch master

percpu_ref: fix documentation of maximum value

2026-07-05T23:23:00+00:00

Tejun changd percpu_ref to use long instead of int back in 2014 but missed updating this bit of the documentation. Also add the documentation to the htmldocs. Link: https://lore.kernel.org/20241211204017.184512-1-willy@infradead.org Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Tejun Heo Signed-off-by: Andrew Morton

treewide: Replace kmalloc with kmalloc_obj for non-scalar types

2026-02-21T09:02:28+00:00

This is the result of running the Coccinelle script from scripts/coccinelle/api/kmalloc_objs.cocci. The script is designed to avoid scalar types (which need careful case-by-case checking), and instead replace kmalloc-family calls that allocate struct or union object instances: Single allocations: kmalloc(sizeof(TYPE), ...) are replaced with: kmalloc_obj(TYPE, ...) Array allocations: kmalloc_array(COUNT, sizeof(TYPE), ...) are replaced with: kmalloc_objs(TYPE, COUNT, ...) Flex array allocations: kmalloc(struct_size(PTR, FAM, COUNT), ...) are replaced with: kmalloc_flex(*PTR, FAM, COUNT, ...) (where TYPE may also be *VAR) The resulting allocations no longer return "void *", instead returning "TYPE *". Signed-off-by: Kees Cook

percpu-refcount: Use call_rcu_hurry() for atomic switch

2022-11-30T21:16:40+00:00

Earlier commits in this series allow battery-powered systems to build their kernels with the default-disabled CONFIG_RCU_LAZY=y Kconfig option. This Kconfig option causes call_rcu() to delay its callbacks in order to batch callbacks. This means that a given RCU grace period covers more callbacks, thus reducing the number of grace periods, in turn reducing the amount of energy consumed, which increases battery lifetime which can be a very good thing. This is not a subtle effect: In some important use cases, the battery lifetime is increased by more than 10%. This CONFIG_RCU_LAZY=y option is available only for CPUs that offload callbacks, for example, CPUs mentioned in the rcu_nocbs kernel boot parameter passed to kernels built with CONFIG_RCU_NOCB_CPU=y. Delaying callbacks is normally not a problem because most callbacks do nothing but free memory. If the system is short on memory, a shrinker will kick all currently queued lazy callbacks out of their laziness, thus freeing their memory in short order. Similarly, the rcu_barrier() function, which blocks until all currently queued callbacks are invoked, will also kick lazy callbacks, thus enabling rcu_barrier() to complete in a timely manner. However, there are some cases where laziness is not a good option. For example, synchronize_rcu() invokes call_rcu(), and blocks until the newly queued callback is invoked. It would not be a good for synchronize_rcu() to block for ten seconds, even on an idle system. Therefore, synchronize_rcu() invokes call_rcu_hurry() instead of call_rcu(). The arrival of a non-lazy call_rcu_hurry() callback on a given CPU kicks any lazy callbacks that might be already queued on that CPU. After all, if there is going to be a grace period, all callbacks might as well get full benefit from it. Yes, this could be done the other way around by creating a call_rcu_lazy(), but earlier experience with this approach and feedback at the 2022 Linux Plumbers Conference shifted the approach to call_rcu() being lazy with call_rcu_hurry() for the few places where laziness is inappropriate. And another call_rcu() instance that cannot be lazy is the one on the percpu refcounter's "per-CPU to atomic switch" code path, which uses RCU when switching to atomic mode. The enqueued callback wakes up waiters waiting in the percpu_ref_switch_waitq. Allowing this callback to be lazy would result in unacceptable slowdowns for users of per-CPU refcounts, such as blk_pre_runtime_suspend(). Therefore, make __percpu_ref_switch_to_atomic() use call_rcu_hurry() in order to revert to the old behavior. [ paulmck: Apply s/call_rcu_flush/call_rcu_hurry/ feedback from Tejun Heo. ] Signed-off-by: Joel Fernandes (Google) Acked-by: Tejun Heo Signed-off-by: Paul E. McKenney Cc: Dennis Zhou Cc: Christoph Lameter Cc:

percpu_ref_init(): clean ->percpu_count_ref on failure

2022-05-18T06:20:17+00:00

That way percpu_ref_exit() is safe after failing percpu_ref_init(). At least one user (cgroup_create()) had a double-free that way; there might be other similar bugs. Easier to fix in percpu_ref_init(), rather than playing whack-a-mole in sloppy users... Usual symptoms look like a messed refcounting in one of subsystems that use percpu allocations (might be percpu-refcount, might be something else). Having refcounts for two different objects share memory is Not Nice(tm)... Reported-by: syzbot+5b1e53987f858500ec00@syzkaller.appspotmail.com Signed-off-by: Al Viro

percpu_ref: Don't opencode percpu_ref_is_dying

2021-05-13T03:27:38+00:00

Signed-off-by: Nikolay Borisov Signed-off-by: Dennis Zhou

percpu_ref: Dump mem_dump_obj() info upon reference-count underflow

2021-01-22T23:24:23+00:00

Reference-count underflow for percpu_ref is detected in the RCU callback percpu_ref_switch_to_atomic_rcu(), and the resulting warning does not print anything allowing easy identification of which percpu_ref use case is underflowing. This is of course not normally a problem when developing a new percpu_ref use case because it is most likely that the problem resides in this new use case. However, when deploying a new kernel to a large set of servers, the underflow might well be a new corner case in any of the old percpu_ref use cases. This commit therefore calls mem_dump_obj() to dump out any additional available information on the underflowing percpu_ref instance. Cc: Ming Lei Cc: Jens Axboe Cc: Joonsoo Kim Reported-by: Andrii Nakryiko Tested-by: Naresh Kamboju Signed-off-by: Paul E. McKenney

percpu_ref: don't refer to ref->data if it isn't allocated

2020-10-09T18:32:06+00:00

We can't check ref->data->confirm_switch directly in __percpu_ref_exit(), since ref->data may not be allocated in one not-initialized refcount. Fixes: 2b0d3d3e4fcf ("percpu_ref: reduce memory footprint of percpu_ref in fast path") Reported-by: syzbot+fd15ff734dace9e16437@syzkaller.appspotmail.com Signed-off-by: Ming Lei Signed-off-by: Jens Axboe

percpu_ref: reduce memory footprint of percpu_ref in fast path

2020-10-06T13:29:36+00:00

'struct percpu_ref' is often embedded into one user structure, and the instance is usually referenced in fast path, however actually only 'percpu_count_ptr' is needed in fast path. So move other fields into one new structure of 'percpu_ref_data', and allocate it dynamically via kzalloc(), then memory footprint of 'percpu_ref' in fast path is reduced a lot and becomes suitable to put into hot cacheline of user structure. Signed-off-by: Ming Lei Tested-by: Veronika Kabatova Reviewed-by: Christoph Hellwig Acked-by: Tejun Heo Cc: Sagi Grimberg Cc: Tejun Heo Cc: Christoph Hellwig Cc: Jens Axboe Cc: Bart Van Assche Signed-off-by: Jens Axboe

lib/percpu-refcount.c: use a more common logging style

2020-06-05T02:06:25+00:00

Remove the trailing newline from the used-once pr_fmt and add it to the single use of pr_ in this code to use a more common logging style. Miscellanea: o Use %lu in the pr_debug format and remove the unnecessary cast Signed-off-by: Joe Perches Signed-off-by: Andrew Morton Cc: Christophe JAILLET Link: http://lkml.kernel.org/r/47372467902a047c03b0fd29aab56e0c38d3f848.camel@perches.com Signed-off-by: Linus Torvalds

percpu_ref: Fix comment regarding percpu_ref_init flags

2020-03-05T21:10:26+00:00

The comment for percpu_ref_init() implies that using PERCPU_REF_ALLOW_REINIT will cause the refcount to start at 0. But this is not true. PERCPU_REF_ALLOW_REINIT starts the count at 1 as if the flags were zero. Add this fact to the kernel doc comment. Signed-off-by: Ira Weiny [Dennis: reworded] Signed-off-by: Dennis Zhou