dlan17/linux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ddb20d1d3aed8f130519c0a29cd5392efcc067b8
linux/kernel
T
History
Waiman Long ddb20d1d3a locking/rwsem: Optimize down_read_trylock() 
Modify __down_read_trylock() to optimize for an unlocked rwsem and make
it generate slightly better code.

Before this patch, down_read_trylock:

   0x0000000000000000 <+0>:     callq  0x5 <down_read_trylock+5>
   0x0000000000000005 <+5>:     jmp    0x18 <down_read_trylock+24>
   0x0000000000000007 <+7>:     lea    0x1(%rdx),%rcx
   0x000000000000000b <+11>:    mov    %rdx,%rax
   0x000000000000000e <+14>:    lock cmpxchg %rcx,(%rdi)
   0x0000000000000013 <+19>:    cmp    %rax,%rdx
   0x0000000000000016 <+22>:    je     0x23 <down_read_trylock+35>
   0x0000000000000018 <+24>:    mov    (%rdi),%rdx
   0x000000000000001b <+27>:    test   %rdx,%rdx
   0x000000000000001e <+30>:    jns    0x7 <down_read_trylock+7>
   0x0000000000000020 <+32>:    xor    %eax,%eax
   0x0000000000000022 <+34>:    retq
   0x0000000000000023 <+35>:    mov    %gs:0x0,%rax
   0x000000000000002c <+44>:    or     $0x3,%rax
   0x0000000000000030 <+48>:    mov    %rax,0x20(%rdi)
   0x0000000000000034 <+52>:    mov    $0x1,%eax
   0x0000000000000039 <+57>:    retq

After patch, down_read_trylock:

   0x0000000000000000 <+0>:	callq  0x5 <down_read_trylock+5>
   0x0000000000000005 <+5>:	xor    %eax,%eax
   0x0000000000000007 <+7>:	lea    0x1(%rax),%rdx
   0x000000000000000b <+11>:	lock cmpxchg %rdx,(%rdi)
   0x0000000000000010 <+16>:	jne    0x29 <down_read_trylock+41>
   0x0000000000000012 <+18>:	mov    %gs:0x0,%rax
   0x000000000000001b <+27>:	or     $0x3,%rax
   0x000000000000001f <+31>:	mov    %rax,0x20(%rdi)
   0x0000000000000023 <+35>:	mov    $0x1,%eax
   0x0000000000000028 <+40>:	retq
   0x0000000000000029 <+41>:	test   %rax,%rax
   0x000000000000002c <+44>:	jns    0x7 <down_read_trylock+7>
   0x000000000000002e <+46>:	xor    %eax,%eax
   0x0000000000000030 <+48>:	retq

By using a rwsem microbenchmark, the down_read_trylock() rate (with a
load of 10 to lengthen the lock critical section) on a x86-64 system
before and after the patch were:

                 Before Patch    After Patch
   # of Threads     rlock           rlock
   ------------     -----           -----
        1           14,496          14,716
        2            8,644           8,453
	4            6,799           6,983
	8            5,664           7,190

On a ARM64 system, the performance results were:

                 Before Patch    After Patch
   # of Threads     rlock           rlock
   ------------     -----           -----
        1           23,676          24,488
        2            7,697           9,502
        4            4,945           3,440
        8            2,641           1,603

For the uncontended case (1 thread), the new down_read_trylock() is a
little bit faster. For the contended cases, the new down_read_trylock()
perform pretty well in x86-64, but performance degrades at high
contention level on ARM64.

Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tim Chen <tim.c.chen@linux.intel.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-c6x-dev@linux-c6x.org
Cc: linux-m68k@lists.linux-m68k.org
Cc: linux-riscv@lists.infradead.org
Cc: linux-um@lists.infradead.org
Cc: linux-xtensa@linux-xtensa.org
Cc: linuxppc-dev@lists.ozlabs.org
Cc: nios2-dev@lists.rocketboards.org
Cc: openrisc@lists.librecores.org
Cc: uclinux-h8-devel@lists.sourceforge.jp
Link: https://lkml.kernel.org/r/20190322143008.21313-4-longman@redhat.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2019-04-03 14:50:52 +02:00
..
bpf
bpf: verifier: propagate liveness on all frames
2019-03-21 19:57:02 -07:00
cgroup
Merge branch 'work.mount' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
2019-03-12 14:08:19 -07:00
configs
kvm_config: add CONFIG_VIRTIO_MENU
2018-10-24 20:55:56 -04:00
debug
kdb: use bool for binary state indicators
2018-12-30 08:31:52 +00:00
dma
memblock: drop memblock_alloc_*_nopanic() variants
2019-03-12 10:04:02 -07:00
events
Merge tag 'perf-core-for-mingo-5.1-20190311' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux into perf/urgent
2019-03-22 22:50:41 +01:00
gcov
kernel/gcov/gcc_3_4.c: use struct_size() in kzalloc()
2019-03-07 18:32:02 -08:00
irq
genirq: Mark expected switch case fall-through
2019-03-23 12:32:01 +01:00
livepatch
Merge branch 'for-5.1/atomic-replace' into for-linus
2019-03-05 15:56:59 +01:00
locking
locking/rwsem: Optimize down_read_trylock()
2019-04-03 14:50:52 +02:00
power
treewide: add checks for the return value of memblock_alloc*()
2019-03-12 10:04:02 -07:00
printk
Merge tag 'fbdev-v5.1' of git://github.com/bzolnier/linux
2019-03-15 14:22:59 -07:00
rcu
Merge branch 'perf-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2019-03-06 07:59:36 -08:00
sched
Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
2019-03-24 11:42:10 -07:00
time
time/jiffies: Make refined_jiffies static
2019-03-22 13:38:26 +01:00
trace
ftrace: Fix warning using plain integer as NULL & spelling corrections
2019-03-26 08:35:36 -04:00
.gitignore
kernel/configs: use .incbin directive to embed config_data.gz
2019-03-07 18:32:02 -08:00
acct.c
…
async.c
async: Add support for queueing on specific NUMA node
2019-01-31 14:20:54 +01:00
audit_fsnotify.c
audit: add syscall information to CONFIG_CHANGE records
2019-01-18 17:53:29 -05:00
audit_tree.c
audit: hand taken context to audit_kill_trees for syscall logging
2019-01-14 18:01:05 -05:00
audit_watch.c
audit: add syscall information to CONFIG_CHANGE records
2019-01-18 17:53:29 -05:00
audit.c
audit: remove audit_context when CONFIG_ AUDIT and not AUDITSYSCALL
2019-02-03 17:49:35 -05:00
audit.h
audit: hide auditsc_get_stamp and audit_serial prototypes
2019-02-07 21:44:27 -05:00
auditfilter.c
audit: mark expected switch fall-through
2019-02-12 20:17:13 -05:00
auditsc.c
audit: remove audit_context when CONFIG_ AUDIT and not AUDITSYSCALL
2019-02-03 17:49:35 -05:00
backtracetest.c
…
bounds.c
kbuild: fix kernel/bounds.c 'W=1' warning
2018-10-31 08:54:14 -07:00
capability.c
LSM: add SafeSetID module that gates setid calls
2019-01-25 11:22:43 -08:00
compat.c
time: make adjtime compat handling available for 32 bit
2019-02-07 00:13:27 +01:00
configs.c
kernel/configs: use .incbin directive to embed config_data.gz
2019-03-07 18:32:02 -08:00
context_tracking.c
…
cpu_pm.c
…
cpu.c
cpu/hotplug: Prevent crash when CPU bringup fails on CONFIG_HOTPLUG_CPU=n
2019-03-28 13:34:58 +01:00
crash_core.c
kexec: export PG_offline to VMCOREINFO
2019-03-05 21:07:14 -08:00
crash_dump.c
…
cred.c
SELinux: Remove cred security blob poisoning
2019-01-08 13:18:44 -08:00
delayacct.c
delayacct: track delays from thrashing cache pages
2018-10-26 16:26:32 -07:00
dma.c
…
elfcore.c
…
exec_domain.c
…
exit.c
Merge branch 'for-5.1' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
2019-03-07 10:11:41 -08:00
extable.c
…
fail_function.c
kernel/fail_function.c: remove meaningless null pointer check before debugfs_remove_recursive
2018-10-31 08:54:12 -07:00
fork.c
Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma
2019-03-09 15:53:03 -08:00
freezer.c
…
futex.c
futex: Ensure that futex address is aligned in handle_futex_death()
2019-03-22 13:05:26 +01:00
groups.c
…
hung_task.c
kernel/hung_task.c: Use continuously blocked time when reporting.
2019-03-07 18:31:59 -08:00
iomem.c
…
irq_work.c
…
jump_label.c
locking/static_key: Fix false positive warnings on concurrent dec/inc
2019-04-03 11:42:33 +02:00
kallsyms.c
bpf: Add module name [bpf] to ksymbols for bpf programs
2019-01-21 17:38:56 -03:00
kcmp.c
…
Kconfig.freezer
…
Kconfig.hz
…
Kconfig.locks
locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all archs
2019-04-03 14:50:52 +02:00
Kconfig.preempt
kconfig: warn no new line at end of file
2018-12-15 17:44:35 +09:00
kcov.c
kcov: convert kcov.refcount to refcount_t
2019-03-07 18:32:02 -08:00
kexec_core.c
mm: convert totalram_pages and totalhigh_pages variables to atomic
2018-12-28 12:11:47 -08:00
kexec_file.c
kexec_file: kexec_walk_memblock() only walks a dedicated region at kdump
2018-12-06 14:38:50 +00:00
kexec_internal.h
…
kexec.c
…
kmod.c
…
kprobes.c
kprobes: Search non-suffixed symbol in blacklist
2019-02-13 08:16:40 +01:00
ksysfs.c
…
kthread.c
Merge branch 'akpm' (patches from Andrew)
2019-03-06 10:31:36 -08:00
latencytop.c
…
Makefile
kernel/configs: use .incbin directive to embed config_data.gz
2019-03-07 18:32:02 -08:00
memremap.c
mm/hmm: fix memremap.h, move dev_page_fault_t callback to hmm
2018-12-28 12:11:52 -08:00
module_signing.c
modsign: use all trusted keys to verify module signature
2018-11-07 14:41:41 +01:00
module-internal.h
…
module.c
dynamic_debug: add static inline stub for ddebug_add_module
2019-03-07 18:32:00 -08:00
notifier.c
…
nsproxy.c
…
padata.c
padata: clean an indentation issue, remove extraneous space
2018-11-16 14:11:04 +08:00
panic.c
kernel/panic.c: taint: fix debugfs_simple_attr.cocci warnings
2019-03-07 18:31:59 -08:00
params.c
…
pid_namespace.c
signal: Use group_send_sig_info to kill all processes in a pid namespace
2018-09-16 16:08:25 +02:00
pid.c
Fix failure path in alloc_pid()
2018-12-28 12:42:30 -08:00
profile.c
mm: remove include/linux/bootmem.h
2018-10-31 08:54:16 -07:00
ptrace.c
ptrace: take into account saved_sigmask in PTRACE{GET,SET}SIGMASK
2019-03-29 10:01:37 -07:00
range.c
…
reboot.c
kernel/reboot.c: export pm_power_off_prepare
2018-09-11 16:13:24 +01:00
relay.c
Merge branch 'work.misc' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
2019-03-12 13:27:20 -07:00
resource.c
Merge tag 'devdax-for-5.1' of git://git.kernel.org/pub/scm/linux/kernel/git/nvdimm/nvdimm
2019-03-16 13:05:32 -07:00
rseq.c
Remove 'type' argument from access_ok() function
2019-01-03 18:57:57 -08:00
seccomp.c
Merge branch 'next-general' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security
2019-03-07 11:44:01 -08:00
signal.c
Merge tag 'pidfd-v5.1-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux
2019-03-16 13:47:14 -07:00
smp.c
cpu/hotplug: Fix "SMT disabled by BIOS" detection for KVM
2019-01-30 19:27:00 +01:00
smpboot.c
…
smpboot.h
…
softirq.c
softirq: Don't skip softirq execution when softirq thread is parking
2019-02-10 21:51:39 +01:00
stackleak.c
stackleak: Mark stackleak_track_stack() as notrace
2018-12-05 19:31:44 -08:00
stacktrace.c
…
stop_machine.c
…
sys_ni.c
Merge tag 'pidfd-v5.1-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux
2019-03-16 13:47:14 -07:00
sys.c
Merge branch 'akpm' (patches from Andrew)
2019-03-07 19:25:37 -08:00
sysctl_binary.c
kernel/sysctl: add panic_print into sysctl
2019-01-04 13:13:47 -08:00
sysctl.c
kernel/sysctl.c: define minmax conv functions in terms of non-minmax versions
2019-03-12 10:04:00 -07:00
task_work.c
…
taskstats.c
…
test_kprobes.c
…
torture.c
Merge branches 'doc.2019.01.26a', 'fixes.2019.01.26a', 'sil.2019.01.26a', 'spdx.2019.02.09a', 'srcu.2019.01.26a' and 'torture.2019.01.26a' into HEAD
2019-02-09 08:47:52 -08:00
tracepoint.c
tracing: Replace synchronize_sched() and call_rcu_sched()
2018-11-27 09:21:41 -08:00
tsacct.c
…
ucount.c
…
uid16.c
…
uid16.h
…
umh.c
umh: add exit routine for UMH process
2019-01-11 18:05:40 -08:00
up.c
smp,cpumask: introduce on_each_cpu_cond_mask
2018-10-09 16:51:11 +02:00
user_namespace.c
userns: also map extents in the reverse map to kernel IDs
2018-11-07 23:51:16 -06:00
user-return-notifier.c
…
user.c
…
utsname_sysctl.c
…
utsname.c
…
watchdog_hld.c
watchdog: Mark watchdog touch functions as notrace
2018-08-30 12:56:40 +02:00
watchdog.c
watchdog: Respect watchdog cpumask on CPU hotplug
2019-03-28 13:32:01 +01:00
workqueue_internal.h
psi: fix aggregation idle shut-off
2019-02-01 15:46:23 -08:00
workqueue.c
workqueue: Only unregister a registered lockdep key
2019-03-21 12:00:18 +01:00