************************************************************************** * The following are additional notes on the cputimes command. * * $Id: cputimes_notes.txt,v 1.1.1.1 2015/09/30 22:01:09 christos Exp $ * * COPYRIGHT: Copyright (c) 2007 Brendan Gregg. ************************************************************************** * How cputimes works cputimes measures time consumed by the kernel, idle therads and processes, by tracking the activity of the schedular. In particular we track on-cpu and off-cpu events for kernel therads, measuring the timestamps at each event. * Why cputimes? If you are interested in how much time processes are consuming, the data given by "prstat" or "prstat -m" is fine. However there is no easy way to see kernel consumed time, which is the idea behind cputimes. * What does it mean? The output shows categories of threads by the sum of time, in nanoseconds. A nanosecond is 10^-9, or 0.000000001 of a second. This program uses nanoseconds as units, but does not have nanosecond accuracy. It would be reasonable to assume that this has microsecond accuracy (10^-6), so in practise ignore the last three digits of the times. The sections reported are, PROCESSES - the sum of all the process time on the CPU. KERNEL - the sum of the time spent in the kernel. IDLE - the time the kernel spent in the idle thread, waiting for some work. If your system isn't doing much, then the idle time will be quite large. If your system is running many applications, then there may be no idle time at all - instead most of the time appearing under processes. * When is there a problem? Expect to see most of the time in processes or idle, depending on how busy your server is. Seeing a considerable amout of time in kernel would definately be interesting. The kernel generally doesn't use much CPU time, usually less than 5%. If it were using more, that may indicate heavy activity from an interrupt thread, or activity caused by DTrace. For example, # cputimes 1 2005 Apr 27 23:49:32, THREADS TIME (ns) IDLE 28351679 KERNEL 436022725 PROCESS 451304688 In this sample the kernel is using a massive amount of the CPUs, around 47%. This sample was taken during heavy network utilisation, the time consumed by the TCP/IP and network driver threads (and DTrace). The "intrstat" command could be used for further analysis of the interrupt threads responsible for servicing the network interface. * Problems with cputimes The way cputimes measures schedular activity turns out to be a lot of work. There are many scheduling events per second where one thread steps onto a CPU and another leaves. It turns out that cputimes itself causes some degree of kernel load. Here we run 1 cputimes, # cputimes 1 2005 May 15 12:00:41, THREADS TIME (ns) KERNEL 12621985 PROCESS 982751579 2005 May 15 12:00:42, THREADS TIME (ns) KERNEL 12267577 PROCESS 983513765 [...] Now a second cputimes is run at the same time, # cputimes 1 2005 May 15 12:02:06, THREADS TIME (ns) KERNEL 17366426 PROCESS 978804165 2005 May 15 12:02:07, THREADS TIME (ns) KERNEL 17614829 PROCESS 978671601 [...] And now a third, # cputimes 1 2005 May 15 12:03:09, THREADS TIME (ns) KERNEL 21303089 PROCESS 974925124 2005 May 15 12:03:10, THREADS TIME (ns) KERNEL 21222992 PROCESS 975152727 [...] Each extra cputimes is consuming an extra 4 to 5 ms of the CPU as kernel time. Around 0.5%. This can be used as an estimate of the kernel load caused by running cputimes, and a similar strategy could be used to measure the kernel load of other DTrace scripts. However the following CPU characteristics must be taken into consideration, # psrinfo -v Status of virtual processor 0 as of: 05/15/2005 12:06:05 on-line since 04/30/2005 13:32:32. The i386 processor operates at 867 MHz, and has an i387 compatible floating point processor. as well as the type of activity that was also running on the system, which cputimes was monitoring (frequency of scheduling events). A system with a slower CPU will use a larger proportion of kernel time to perform the same tasks. Also, a system that is context switching more (switching between different processes) is likely to consume more kernel time as well.