Performance Profiling

Collect perf data of a ceph process at runtime¶

There are some cases where the debug logs are not sufficient to investigate issues like high CPU utilization of a Ceph process. In that situation, coredump and perf information of a Ceph process is useful to be collected which can be shared with the Ceph team in an issue.

To collect this information, please follow these steps:

• Edit the rook-ceph-operator deployment and set ROOK_HOSTPATH_REQUIRES_PRIVILEGED to true.
• Wait for the pods to get reinitialized:

1

# watch kubectl -n rook-ceph get pods

• Enter the respective pod of the Ceph process which needs to be investigated. For example:

1

# kubectl -n rook-ceph exec -it deploy/rook-ceph-mon-a -- bash

• Install gdb , perf and git inside the pod. For example:

  1	# dnf install gdb git perf -y

• Capture perf data of the respective Ceph process:

1
2

# perf record -e cycles --call-graph dwarf -p <pid of the process>
# perf report > perf_report_<process/thread>

• Grab the pid of the respective Ceph process to collect its backtrace at multiple time instances, attach gdb to it and share the output gdb.txt:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# gdb -p <pid_of_the_process>

- set pag off
- set log on
- thr a a bt full # This captures the complete backtrace of the process
- backtrace
- Ctrl+C
- backtrace
- Ctrl+C
- backtrace
- Ctrl+C
- backtrace
- set log off
- q (to exit out of gdb)

• Grab the live coredump of the respective process using gcore:

1

# gcore <pid_of_the_process>

• Capture the Wallclock Profiler data for the respective Ceph process and share the output gdbpmp.data generated:

1
2
3

# git clone https://github.com/markhpc/gdbpmp
# cd gdbpmp
# ./gdbpmp.py -p <pid_of_the_process> -n 100 -o gdbpmp.data

• Collect the perf.data, perf_report, backtrace of the process gdb.txt , core file and profiler data gdbpmp.data and upload it to the tracker issue for troubleshooting purposes.