bcc-wqlat - Man Page

wqlat [-h] [-T] [-N] [-W] [-w WQNAME] [interval [count]]

wqlat traces  work's waiting on workqueue, and records the distribution of work's queuing latency (time). This is printed as a histogram  either on Ctrl-C, or after a given interval in seconds.

This tool uses in-kernel eBPF maps for storing timestamps and the histogram, for efficiency.

This tool uses the workqueue:workqueue_queue_work and workqueue:workqueue_execute_start kernel tracepoints, which is a stable tracing mechanism. Please note BPF programs can  attach to tracepoints from Linux 4.7 only, so this tools can only support kernel 4.7 or later version.

Since this uses BPF, only the root user can use this tool.

CONFIG_BPF and bcc.

-h Print usage message.

-T

Include timestamps on output.

-N

Output histogram in nanoseconds.

-W

Print a histogram per workqueue.

-w WQNAME

Trace this workqueue only

interval

Output interval, in seconds.

count

Number of outputs.

Summarize kernel workqueue latency as a histogram:

# wqlat

Print 1 second summaries, 10 times:

# wqlat 1 10

Print 1 second summaries, using nanoseconds as units for the histogram, and

include timestamps on output: # wqlat -NT 1

Print 1 second summaries, 10 times per workqueue:

# wqlat -W 1 10

Print 1 second summaries for workqueue nvmet_tcp_wq:

# wqlat -w nvmet_tcp_wq 1

usecs

Microsecond range

nsecs

Nanosecond range

count

How many works into this range

distribution

An ASCII bar chart to visualize the distribution (count column)

This traces kernel functions and maintains in-kernel timestamps and a histogram, which are asynchronously copied to user-space. This method is very efficient, and the overhead for most workqueue scheduling rates (< 100k) should be  negligible.If you have a higher workqueue scheduling, please test and quantify  the overhead before use.

This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

Linux

Unstable - in development.

Ping Gan

biolatency