Solaris 10: tips and tricks for system administration - DTrace

Announcing a new website: DTrace Scripts

nospam@example.com (Gleb Reys) — Fri, 09 Nov 2007 14:53:56 +0300

Thanks to CurThread.org, I've recently learned about the new online project which just got started: DTrace Scripts. It seems to be a promising project with a constantly expanding collection of DTrace scripts and tips, and it's great to see yet another project aimed at making DTrace more accessible and better known in Unix communities.

Another good article on DTrace

nospam@example.com (Gleb Reys) — Wed, 08 Feb 2006 16:32:11 +0300

Thanks to Adam Leventhal, who is one of DTrace authors, I've learned about the recent article on DTrace by Bryan Cantrill.

I liked this article a lot, so it's been immediately added to my Useful DTrace resources index.

You can read the article by following this link: Hidden in Plain Sight

Useful resources on DTrace

nospam@example.com (Gleb Reys) — Fri, 27 Jan 2006 13:06:14 +0300

I've finally found time to start yet another good tradition on my blog. From now on, I'll maintain an index of links to all my articles on DTrace, which will also include links to all other resources on DTrace I've found useful.

This link repositary can be found at the following URL: http://solaris.reys.net/english/dtrace

DTrace: playing with mod_auth_pam and Apache 2

nospam@example.com (Gleb Reys) — Wed, 21 Dec 2005 13:29:25 +0300

One of our users has asked me to help with PAM authentication for Apache 2 on one of our remote servers.

It did seem like a rather trivial task - download mod_auth_pam, compile it for Apache 2, make sure everything works. The remote server uses NIS, I could freely log onto it.

And you know what? It didn't work. The module compiled pretty easy, but I couldn't get it to work - error_log was reporting the same lines over and over again (I had to break the line to squeeze it into my blog format):

CODE:

[Tue Dec 20 16:48:44 2005] [error] [client xxx.xxx.xxx.xxx] \
PAM: user 'greys' - not authenticated: No account present for user

Up to this point, everything was done by the user himself with small help from me. At this stage is was decided I'd have a look myself. To make things easier, I also wanted to:

1) Move all the experiments onto my own Solaris 10 box
2) Make it all work for local users first, and only then get NIS working as well

Seeing the same errors on my box, I've decided to take a serious look at PAM. I've created Apache-related lines in /etc/pam.conf and added a debug option:

CODE:

httpd auth required pam_authtok_get.so.1 debug
httpd auth required pam_unix_auth.so.1 debug

To make it more convenient to track all the messages, I've updated /etc/syslog.conf to include the following:

CODE:

auth.debug /var/log/auth_log
user.debug /var/log/user_log

It got easier to see what was happening, at least I knew it was the /etc/pam.conf section I was thinking of. But the error was still there - my user definitely did exist in the system, only it wouldn't be recognized by Apache (and I've broken the lines yet again here):

CODE:

Dec 21 09:47:35 pele httpd[29143]: [ID 634615 user.debug] \
pam_authtok_get:pam_sm_authenticate: flags = 1
Dec 21 09:47:35 pele httpd[29143]: [ID 378613 user.debug] \
pam_dhkeys: user greys not found
Dec 21 09:47:35 pele httpd[29143]: [ID 896952 user.debug] \
pam_unix_auth: entering pam_sm_authenticate()
Dec 21 09:47:35 pele httpd[29143]: [ID 219349 user.debug] \
pam_unix_auth: user greys not found

It was the right time to look at how exactly PAM and Apache 2 interacted when I was giving the username and password trying to access my page. I immediately thought of Brendan Gregg's opensnoop script for DTrace. And it was this script indeed which helped me see the problem:

CODE:

60001 24744 httpd         -1   2 /etc/pam_debug
60001 24744 httpd         11   0 /etc/pam.conf
60001 24744 httpd         11   0 /usr/lib/security/pam_authtok_get.so.1
60001 24744 httpd         11   0 /usr/lib/passwdutil.so.1
60001 24744 httpd         11   0 /usr/lib/libsldap.so.1
60001 24744 httpd         11   0 /usr/lib/security/pam_dhkeys.so.1
60001 24744 httpd         11   0 /usr/lib/security/pam_unix_cred.so.1
60001 24744 httpd         11   0 /lib/libbsm.so.1
60001 24744 httpd         11   0 /lib/libsecdb.so.1
60001 24744 httpd         11   0 /usr/lib/libproject.so.1
60001 24744 httpd         11   0 /lib/libproc.so.1
60001 24744 httpd         11   0 /lib/librtld_db.so.1
60001 24744 httpd         11   0 /lib/libelf.so.1
60001 24744 httpd         11   0 /lib/libctf.so.1
60001 24744 httpd         11   0 /usr/lib/security/pam_unix_auth.so.1
60001 24744 httpd         -1   2 /var/run/syslog_door
60001 24744 httpd         -1  13 /etc/shadow
60001 24744 httpd         11   0 /var/run/name_service_door
60001 24744 httpd         11   0 /etc/passwd
60001 24744 httpd         -1  13 /etc/shadow
60001 24744 httpd         -1   2 /var/run/syslog_door

Apparently, httpd could not open /etc/shadow (-1 means error), obviously becase of permissions for the file. But just to make sure, I've looked for the error code (it's 13 as you can see):

CODE:

bash-3.00# grep 13 /usr/include/sys/errno.h
#define EACCES 13 /* Permission denied */

So it all was suddenly clear, and I've proceeded like this:

CODE:

#groupadd shadows
# chown root:shadows /etc/shadow
# chmod 440 /etc/shadow

All that's left to be done now was to make sure httpd.conf has a Group parameter set to this shadows group, and not nobody.

pS: you might ask why didn't it work for NIS on the remote server in first place. This is because /etc/nsswitch.conf had not this: passwd: files nis, but this: passwd: compat, so NIS maps were consulted, but only based on + and - specifications in /etc/passwd and /etc/shadow files. This means we still needed to be able to read /etc/shadow, even though later PAM would consult NIS.

DTrace: observing PHP, part 2

nospam@example.com (Gleb Reys) — Tue, 13 Dec 2005 02:16:13 +0300

Your next step in observing PHP with DTrace could easily be the following script. It times how much time (in nanoseconds) each Apache process has spent running a particular PHP function.

This script generates a table of Apache PIDs and PHP scripts started within, and later you'll get a table of all the functions with PIDs which ran them and the time spent.

You can download this script here: php-scripts-timer.d

CODE:

#!/usr/sbin/dtrace -s
#pragma D option quiet

BEGIN
{
printf("php scripts timer started\n");
}

:dtrace.so::function-entry
/ copyinstr(arg0) == "" && name[pid,copyinstr(arg1)] != 1 /
{
        printf("%d -> %-30s\n", pid, copyinstr(arg1));
        name[pid,copyinstr(arg1)] = 1;
}

:dtrace.so::function-entry
/ copyinstr(arg0) != "" /
{
        self->ts = timestamp;
}

:dtrace.so::function-return
/ copyinstr(arg0) != "" /
{
        self->elapsed = timestamp - self->ts;
        @time[pid, copyinstr(arg0)] = sum(self->elapsed);
}

END
{
        printf("<------ php scripts timer finished");
        printf("\n%10s %40s %14s\n", "PID", "PHP function", "Time elapsed");
        printa("%10d %40s %14@u\n", @time);
}

A typical result of running this script while opening some PHP pages served by your Apache will look like this:

CODE:

php scripts timer started
223074 -> /export/www/nightly/pma/left.php
223071 -> /export/www/nightly/pma/main.php
223074 -> /export/www/nightly/pma/css/phpmyadmin.css.php
223071 -> /export/www/nightly/pma/css/phpmyadmin.css.php

<------ php scripts timer finished
       PID                             PHP function   Time elapsed
    223074                             ob_get_level          22280
    223074                              get_cfg_var          32678
    223074                    mysql_get_client_info          32705
    223074                                  strrpos          39160
    223074                        PMA_auth_set_user          39178
    223071                     get_magic_quotes_gpc          39478
    223071                             ob_get_level          43673
    223071                               addslashes          54847
    223074                                   strpos          56869
    223071                                 basename          61874
    223071                     PMA_reloadNavigation          63588
    223071                    mysql_get_client_info          64570
    223071                              get_cfg_var          64592
    223071                             ereg_replace          65993
    223074                     get_magic_quotes_gpc          67738
...
    234950                                     main       84254684
    238255                                     main      105016748
    238255                                   substr      153752861
    238255                              mysql_query      192585647
    238255                        PMA_DBI_try_query      195113800
    234950                            PMA_DBI_query      220108343
    234950                              mysql_query      276732073
    234950                        PMA_DBI_try_query      280801228

DTrace: observing PHP

nospam@example.com (Gleb Reys) — Tue, 06 Dec 2005 20:16:23 +0300

I've finally got some spare time to download a PEAR module for DTrace. What a great module! I've found a little problem with it, wrote a letter to Wez Furlong, and he immediately changed the source code to make it work even better. Thanks a lot, Wez! :)

So, here it is - my first PHP Dtrace script: php-scripts-snoop.d

CODE:

#!/usr/sbin/dtrace -s
#pragma D option quiet

:dtrace.so::function-entry
/ copyinstr(arg0) == "" /
{
printf("%d -> %-30s\n", pid, copyinstr(arg1));
}

:dtrace.so::function-return
/ copyinstr(arg0) == "" /
{
printf("%d <- %-30s\n", pid, copyinstr(arg1));
}

So go ahead, download the script, do chmod a+x for it, and after you start it in a terminal of your web-server, you can run PHP scripts from this server, and our DTrace script will be reporting us whenever each PHP script stars or finishes, specifying which Apache process is responsible for this:

CODE:

bash-3.00# ./php-scripts-snoop.d
204934 -> /export/www/pma/index.php
204934 <- /export/www/pma/index.php
204934 -> /export/www/pma/css/phpmyadmin.css.php
204934 <- /export/www/pma/css/phpmyadmin.css.php
204936 -> /export/www/pma/left.php
204934 -> /export/www/pma/queryframe.php
204936 <- /export/www/pma/left.php
204936 -> /export/www/pma/main.php
204934 <- /export/www/pma/queryframe.php
204934 -> /export/www/pma/css/phpmyadmin.css.php
204934 <- /export/www/pma/css/phpmyadmin.css.php
204934 -> /export/www/pma/css/phpmyadmin.css.php
204936 -> /export/www/pma/css/phpmyadmin.css.php
204934 <- /export/www/pma/css/phpmyadmin.css.php
204934 -> /export/www/1.php
204934 <- /export/www/1.php
204936 <- /export/www/pma/css/phpmyadmin.css.php

pS: yes, starting of today I've decided to provide download links for each of my scripts - should be more convenient for you now. This means that sooner or later I'll update all the archive blog entries to have download links for respective scripts.

DTrace: command line parameters in scripting

nospam@example.com (Gleb Reys) — Wed, 18 May 2005 20:05:49 +0400

When you're writing a DTrace script, often you'd like to pass some parameter in command line. Such an easy task might be really hard unless you read a docs.sun.com section about DTrace scripting, particularly a section about comman line parameters macros.

As with sh or bash, you can access command line parameters via $0..$9 macroses. But there's a catch. Values of these macroses should match a certain definition - a number, an identifier or a string. If there isn't a match here, you'll get an error.

Let's start with a simple script.

CODE:

#!/usr/sbin/dtrace -s

#pragma D option quiet

BEGIN {
printf("Your parameter: %d\n", $1);
}

If you pass a number as a command line parameter, there will be no error, and you'll get this message:
Your parameter: 123

But if you pass a string instead of number, here's the error you'll get:

bash-3.00# ./args.d a1b2c3
dtrace: failed to compile script ./args.d: line 9: failed to resolve a1b2c3: Unknown variable name

This happens because DTrace tries to find an identifier "a1b2c3". Why does this happen? Because of the way DTrace tries to recognize the type of a command line parameter. The parameter doesn't look like a number, but to make DTrace accept the value as a string, you have to specify it in quotes. Without them, DTrace accepts we've passed an identifier name. And since there isn't one with such a name, we get an error.

To prove this theory, you simply have to create a variable with the same name in your script:

CODE:

#!/usr/sbin/dtrace -s

#pragma D option quiet

BEGIN {
a1b2c3=1979;
printf("Your parameter: %d\n", $1);
}

There you go - it works just like I said:

Your parameter: 1979

And if your parameter is definitely a string, you have to change your script (the line with printf, so that it has %s for a string, and not %d for an integer, otherwise you'll get another error)...

CODE:

#!/usr/sbin/dtrace -s

#pragma D option quiet

BEGIN {
a1b2c3=1979;
printf("Your parameter: %s\n", $1);
}

... and also put your parameter in quotes, like this:

bash-3.00# ./args.d '"a1b2c3"'
Your parameter: a1b2c3

As you can see, we're getting a string we passed, and not the variable with the same name, which was left in the script intentionally to demonstrate it's not used anymore.

And finally, one more thing. If you want all your command line parameters to be treated as strings by default and even leave the quotes around parameters out, all you have to do is to access the macroses for your parameters with a double $ sign, and then the parameter type definition will be skipped:

CODE:

#!/usr/sbin/dtrace -s

#pragma D option quiet

BEGIN {
a1b2c3=1979;
printf("Your parameter: %s\n", $$1);
}

And here's how you'd access such a script with the same parameter we've used before:

bash-3.00# ./args.d a1b2c3
Your parameter: a1b2c3

That's it for today! Good luck with your DTrace scripts!

Tips and tricks from DTrace authors

nospam@example.com (Gleb Reys) — Tue, 01 Mar 2005 13:38:04 +0300

Yesterday Bryan Cantrill (one of DTrace authors) provided a link to the presentation he along with two other DTrace authors gave recently on advanced DTrace scripting. There are lots of useful tips, tricks and gotchas explained, which might be useful for both newbies and some seasoned DTrace users.

As it's states one the very first slides of the presentation, some of the things presented are quite simple - most of the tips you'd know if you read the documentation carefully, but the tips given are very useful. I've learned lots of new things, so I strongly recommend anyone interested in DTrace to find some time and read it as well. Here is the presentation in a PDF file.

DTrace: pid provider

nospam@example.com (Gleb Reys) — Sat, 26 Feb 2005 23:52:58 +0300

Quite ofter we'd like to know what happens to a particular process on a system, and if we're using DTrace for our investigation, we'd probably use the $target variable (I've already spoked about it in the past) or simply specify the process identifier (pid) in some predicates of our probes, thus pointing to DTrace that we're after only a certain process. $target variable still can and should be used when creating our probes with pid provider.

For DTrace beginners it's quite a common and good practice. But there is a better way: using the pid provider.

As it's explained in the official DTrace guide, pid provider isn't just a provider, but in fact is a whole class of providers. And, as the name suggests, it provides us with probes related to a certain process.

For instance, pid737 provider allows us to trace everything happening within a process with pid of 737. What's really cool about this provide - it's not only easier to scope our interest, but it also gets much easier to see how where certain probes originate from (what libraries provide certain functions used by our process, read on - I'll explain it a bit later).

Such a way of looking at a certain process, in contrast to using predicates like /pid == $target/, is more correct. Instead of enabling all the specified probes by default, and then using predicates for each probe firing for each process on the system only to see whether we're interested in a given process, DTrace won't even look at other processes - and pid737 will make sure that all the probes specified will be enabled and possibly fired only for the specified process.

We already know how to get the list of all the probes known to DTrace. I've explained it a while ago. The thing you need to remember about the pid provider is this: after you've used pid provider in some probes for a process, dtrace -l will start reporting you these probes at the end of all the available probes. They've been dynamically allocated just for you. If the process in question dies or finishes, all the dynamically allocated probes for its pid provider will disappear.

One more thing I really like about pid provider - modules used in probes are in fact libraries used for the execution of the process you're looking at. This gives you a wonderful opportunity of observing only what's been done in your process by functions of a particular library. Very useful.

Right, enough of theory - here's an example. First, we create a simple bit of "Hello, world!" code in C:

CODE:

void main() {
printf("Hello, world!\n");
}

After you compile it, let's get going with out DTrace script:

CODE:

#!/usr/sbin/dtrace -s
#pragma D option quiet

BEGIN {
printf("Target pid: %d\n", $target);
}

pid$target:::entry {
printf("%s:%s:%s:%s\n", probeprov, probemod, probefunc, probename);
}

Save the script, make it executable, and start it, specifying our hello as a command to start:

CODE:

$ ./pid-global.d -c ./hello | more
Target pid: 1059
Hello, world!
pid1059:LM1`ld.so.1:call_array:entry
pid1059:LM1`ld.so.1:call_init:entry
pid1059:LM1`ld.so.1:leave:entry
pid1059:LM1`ld.so.1:fmap_setup:entry
pid1059:LM1`ld.so.1:munmap:entry
pid1059:LM1`ld.so.1:rt_bind_clear:entry
pid1059:LM1`ld.so.1:_rt_bind_clear:entry
pid1059:LM1`ld.so.1:rt_mutex_unlock:entry
pid1059:LM1`ld.so.1:_rt_null:entry
pid1059:LM1`ld.so.1:rt_bind_clear:entry
pid1059:LM1`ld.so.1:_rt_bind_clear:entry
pid1059:libc.so.1:libc_init:entry
pid1059:libc.so.1:__amd64id:entry
pid1059:libc.so.1:atexit:entry
pid1059:libc.so.1:lmalloc:entry
pid1059:libc.so.1:getbucketnum:entry
...

You see this list of probes? I won't give you the full list - it's going to be almost a 1000 lines, but I hope you get the idea. It's also a perfect opportunity to see how easy it's going to be for you to track probes of a particular library.

Back to our example, it's obvious what I'm going to do now: I'd like to get our printf caught with a probe.

CODE:

#!/usr/sbin/dtrace -s
#pragma D option quiet

BEGIN {
printf("Target pid: %d\n", $target);
}

pid$target::printf:entry {
printf("%s:%s:%s:%s\n", probeprov, probemod, probefunc, probename);
printf("Argument: %s\n", copyinstr(arg0));
}

If we start tihs script, here's what you'll get:

CODE:

$ ./pid-printf.d -c ./hello
Target pid: 1201
Hello, world!
pid1201:libc.so.1:printf:entry
Argument: Hello, world!

DTrace open sourced

nospam@example.com (Gleb Reys) — Tue, 25 Jan 2005 19:49:17 +0300

At last, OpenSolaris.Org is up!

As it was expected, it's not possible to download OS sources just yet. Instead, to prove that the whole OpenSolaris idea is serious, Sun revealed DTrace sources.

Bryan Cantrill has posted a very interesting entry in his blog today, I strongly recommend you read it: Solaris 10 Revealed.

DTrace: aggregation functions (aggregations)

nospam@example.com (Gleb Reys) — Wed, 12 Jan 2005 19:26:30 +0300

Quite often it is more important for you to see the whole picture, rather than to know each case when a particular probe fires. For instance, observing a process which consumes a lot of CPU time according to prstat, we ask ourselves: what is this process doing? And often we think about the overall number of system calls executed by this process, and not every particular one of these system calls.

DTrace aggregations are created specifically for answering such questions. Working with sets of numerical data, aggregation functions help us to easily determing average, min and max values of data sets, count numbers and sums of elements in these data sets, and even build frequency distributions.

More information can be found here, but today I'll show only few examples of using aggregations.

DTrace stores the results of aggregation functions in objects called aggregations.

The syntax is:
@name[ keys ] = aggfunc ( args );
here:
name - name of our aggregation
keys - index keys, very similar to those of associative arrays.
aggfunc - aggregation function
args - function arguments, usually it's a set of numerical data

Today I'll use sum aggregation function, which as it's obvious from the name, helps us calculate the sum of a numerical data set,

Only the current value of the aggregation function is stored in the memory, it's calculated off the previous function argument values. Then the function is calculated for the current value of the argument, and this result is applied to the current aggregation value.

For basic analysis you can use as few aggregations keys as you can - for instance, use only PID, without specifying thread ID as another index key.You can even use execname - but keep in mind that there might be more than one process running on your system with the same name. Still, execname-based aggregation will be quite useful to track what's getting most of our system's attention:

Here's our script:

CODE:

#!/usr/sbin/dtrace -s
#pragma D option quiet

syscall:::entry
{
self->ts=timestamp;
}

syscall:::return
{
self->elapsed = timestamp - self->ts;
@time[execname]=sum(self->elapsed);
}

END
{
        printf("\n%20s %20s\n",
"Execname", "Time elapsed");

        printa("%20s %20@u\n", @time);
}

Save it into a file, do chmod a+x on this file and start it. You'll get an output similar to this:

CODE:

     Execname         Time elapsed
        xscreensaver                 5710
       gnome-smproxy                 6886
         wnck-applet                 7735
         gnome-panel                52032
            metacity                54430
    gnome-settings-d                60419
                 vim                66452
       mixer_applet2            670078802
    gnome-netstatus-            819836799
            nautilus            900025830
                Xorg           1074783431
      gnome-terminal           1074816530
         firefox-bin           2000076871
              dtrace        9843796821648

The time in the last column is given in nanoseconds, so don't be scared with such huge numbers :) For timing some actions in your script timestamp proves to be VERY useful. It's a nanoseconds counter which increments from an arbitrary point, so it can be used for relative computations only.

Now you can look at the output and notice how much of unnecessary actions is happening on my laptop with started Gnome ;)

If you want a deeper analysis, you can make our script a bit more complex, and you'll get a powerful tool for determining what process was doing what on our system, and for how long.

CODE:

#!/usr/sbin/dtrace -s
#pragma D option quiet

syscall:::entry
{
self->ts=timestamp;
}

syscall:::return
{
self->elapsed = timestamp - self->ts;
@time[execname,probefunc,pid,tid]=sum(self->elapsed);
}

END
{
printf("\n%20s %20s %8s %7s %20s\n",
"Execname", "Syscall", "Process", "Thread", "Time elapsed");
printa("%20s %20s %8d %7d %20@u\n", @time);
}

If you save this code into syscalls.d and start the script, you'll probably get something similar (I've deleted lots of lines to save up some space):

CODE:

  Execname              Syscall  Process  Thread         Time elapsed
       gnome-terminal                 read     7199       1                12867
                Xorg                 read     6953       1                13769
              dtrace                  brk     7331       1                33647
              dtrace             schedctl     7331       1                39320
      gnome-terminal                write     7199       1                48052
                Xorg               writev     6953       1                57112
      gnome-terminal              pollsys     7199       1               646816
              dtrace             lwp_park     7331       1             44817217
                Xorg              pollsys     6953       1             46936244
              dtrace                ioctl     7331       1        9269224066655

If you leave such a script running for a couple of minutes, you'll get a very long output with the list of every process running on your system all the system calls it's been executing, with the time spent by each thread of each process.

DTrace: $target macro variable

nospam@example.com (Gleb Reys) — Fri, 07 Jan 2005 18:34:22 +0300

Macro variables are meant to make your life MUCH easier when scripting with DTrace. Names of such variables must begin with a $ sign, and DTrace will automatically replace all the macro variables with the appropriate values when parsing your script.

Full list of macro variables can be found here, I will only talk about $target variable this time.

$target specifies the PID of the process you associate with your DTrace script.

DTrace has two useful command line options which can be used not only when running a dtrace command, but also when running DTrace-scripts:

./script.d -c xterm
This command asks DTrace to run a command which follows the -c option, and process the actual script.d only after this. Our macro variable, $target, will containt a PID of the started command we specified - xterm in this example.

./script.d -p 654
This is a similar way of running a DTrace script, only we notify DTrace that we'd like to work with the process with PID of 654. This way of using DTrace is especially useful as you don't have to start or restart an already running process. $target will contain the PID specified in the command line - 654 in this example.

$target may be used like this:

CODE:

#!/usr/sbin/dtrace -s
#pragma D option quiet

syscall:::entry, syscall:::return
/pid == $target/
{
printf("%d/%d: %s:%s\n", pid, tid, probefunc, probename);
}

If you execute this script, you'll probably get an output like this:

CODE:

./script.d -c xterm
26525/1: munmap:entry
26525/1: munmap:return
26525/1: mmap:entry
26525/1: mmap:return
26525/1: setcontext:entry
26525/1: setcontext:return
26525/1: getrlimit:entry
26525/1: getrlimit:return
26525/1: getpid:entry
26525/1: getpid:return
...

Just a little bit later I'll talk about aggregations - yet another one of the wonderful DTrace features, and then I'll be able to give you more interesting and useful examples of DTrace scripting.

More on how DTrace works

nospam@example.com (Gleb Reys) — Thu, 06 Jan 2005 01:12:57 +0300

Just went to BigAdmin: DTrace once again, and bumped into this useful link - and decided to highlight it yet again.

This link leads to a pdf-file with the technical paper of the three guys who created DTrace - Bryan M. Cantrill, Michael W. Shapiro and Adam H. Leventhal, for the last USENIX conference. The title is: Dynamic Instrumentation of Production Systems.
Apart from listing all the advantages of DTrace, the paper also gives a thorough conceptual description of DTrace dynamic tracing mechanisms DTrace - I highly recommend you read it.

DTrace options: flowindent

nospam@example.com (Gleb Reys) — Tue, 14 Dec 2004 19:26:42 +0300

One of the most useful (for me) options in DTrace is flowindent. All it does is indent the entry to each function with "->", and mark the return from this function with "<-". Doesn't seem like much, does it? But just look at the results!

The following example catches all the probes of the fbt provided. By default (that's our case since we're not changing any other options but flowindent), DTrace registers and prints to the standard output all the probes matched in the tracing process.

Here is our script:

CODE:

#!/usr/sbin/dtrace -s

fbt:::
/execname == "dtrace"/
{
}

And this is the output we'll get by running this script:

CODE:

dtrace: script './try5.d' matched 32659 probes
CPU     ID                    FUNCTION:NAME
  1   2027                    xc_sync:entry
  1    819                 xc_do_call:entry
  1   1210         mutex_vector_enter:entry
  1   1238               lock_set_spl:entry
  1   2353                       splx:entry
  1  15484                apic_setspl:entry
  1  15485               apic_setspl:return
  1   2354                      splx:return
  1   1239              lock_set_spl:return
  1   1211        mutex_vector_enter:return
  1    821                  xc_common:entry
  1  15510              apic_send_ipi:entry
  1  15511             apic_send_ipi:return
...

But if we only add the flowindent option to our script (you can also achieve the same result by using -F dtrace command line option), the output is going to appear much more readable and easy to follow:

Our new script:

CODE:

#!/usr/sbin/dtrace -s
#pragma D option flowindent

fbt:::
/execname == "dtrace"/
{
}

And this is the output we'll get:

CODE:

dtrace: script './try5.d' matched 32659 probes
CPU FUNCTION
  1  -> xc_sync
  1    -> xc_do_call
  1      -> mutex_vector_enter
  1        -> lock_set_spl
  1          -> splx
  1            -> apic_setspl
  1            <- apic_setspl
  1          <- splx
  1        <- lock_set_spl
  1      <- mutex_vector_enter
  1      -> xc_common
  1        -> apic_send_ipi
  1        <- apic_send_ipi
...

DTrace: predicates

nospam@example.com (Gleb Reys) — Fri, 10 Dec 2004 19:52:52 +0300

You don't have conditional statements like if..then..else in DTrace, instead of them predicates are used.
Predicates are slash (/) wrapped conditions which are put right after the lines of probe definitions, defining whether DTrace should or should not executre the code of such probes.

Here's an example. The following script will print the list of all system calls, excluding write, for all the processes currently running on our system. Since DTrace tracks events on a thread-level, you can see in this example that some PIDs are the same. This is because these lines were printed by probes fired for different threads of the same process:

CODE:

#!/usr/sbin/dtrace -s
#pragma D option quiet

syscall:::
/probefunc != "write"/
{
printf("probefunc: %s, pid: %d, execname: %s\n", probefunc, pid, execname);
}

When you execute this script, you'll get something similar to this:

CODE:

probefunc: ioctl, pid: 28603, execname: dtrace
probefunc: ioctl, pid: 28603, execname: dtrace
probefunc: ioctl, pid: 28603, execname: dtrace
probefunc: ioctl, pid: 28603, execname: dtrace
probefunc: ioctl, pid: 28603, execname: dtrace
probefunc: sysconfig, pid: 28603, execname: dtrace
probefunc: sysconfig, pid: 28603, execname: dtrace
probefunc: sysconfig, pid: 28603, execname: dtrace
probefunc: sysconfig, pid: 28603, execname: dtrace
probefunc: sigaction, pid: 28603, execname: dtrace
probefunc: sigaction, pid: 28603, execname: dtrace
probefunc: sigaction, pid: 28603, execname: dtrace
probefunc: sigaction, pid: 28603, execname: dtrace
...

As you can see, our script even took a note of all the dtrace syscalls which was used to run the script. To exclude dtrace from our data, we should alter the predicate to include one more condition:

CODE:

/probefunc != "write" && execname != "dtrace"/

this will result in our output including all the processes but dtrace. In this example, we can see a couple of my rxvt terminals and java:

CODE:

probefunc: pollsys, pid: 28411, execname: rxvt
probefunc: ioctl, pid: 28411, execname: rxvt
probefunc: ioctl, pid: 28411, execname: rxvt
probefunc: pollsys, pid: 28411, execname: rxvt
probefunc: pollsys, pid: 28406, execname: rxvt
probefunc: ioctl, pid: 28406, execname: rxvt
probefunc: ioctl, pid: 28406, execname: rxvt
probefunc: pollsys, pid: 28406, execname: rxvt
probefunc: lwp_cond_wait, pid: 27870, execname: java_vm
probefunc: lwp_cond_signal, pid: 27870, execname: java_vm
probefunc: lwp_cond_signal, pid: 27870, execname: java_vm
probefunc: lwp_cond_wait, pid: 27870, execname: java_vm
...

Predicates allow you to use any constants and variables defined earlier in the code, or standard ones, just like the ones shows in my examples. And if you keep in mind, that local variables are unique on thread-level, you can build up pretty complex constructions.

Thread-specific local variables are defined with the help of self indicator and a special -> operator, for instance: self->flag = 1;.

My next example will print only one message for each write syscall of every thread. As you can see in the code, every time we note a thread returning from write syscall, we mark this thread with a local variable, so that we know the thread has been processedt: self->processed = 1;. Next time a write probe fires for this thread, the predicate will take care of it and skip the probe code execution if needed:

CODE:

#!/usr/sbin/dtrace -s
#pragma D option quiet

syscall::write:return
/ self->processed == 0/
{
printf("tid: %d, pid: %d, execname: %s\n", tid, pid, execname);
self->processed = 1;
}

As it's clearly seen, every thread has been registered only once:

CODE:

tid: 1, pid: 21583, execname: soffice.bin
tid: 4, pid: 21750, execname: thunderbird-bin
tid: 4, pid: 27406, execname: mozilla-bin
tid: 1, pid: 27406, execname: mozilla-bin
tid: 1, pid: 21450, execname: icewm
tid: 4, pid: 21583, execname: soffice.bin
tid: 1, pid: 21750, execname: thunderbird-bin
tid: 1, pid: 28636, execname: dtrace

It's impossibly to even try to cover all different ways of using DTrace predicates in only one entry, so I think I'll call it a day. Next time I'll show a few more ways of making your life easier.