Welcome to 2012!

Hi all, happy New Year!

Just thought I’ll take a few minutes to welcome you to the new year and to thank you once again for staying with the Solaris Blog for so long!

My plans for this blog are quite humble in the view of Solaris not being a primary Unix-like OS at work anymore, but I’m still quite curious about a few things and would gladly share knowledge and answer your questions to my best ability.

So far, the following topics appear to be most useful:

  • Anything to do with SSH (stay tuned and take a minute to become RSS reader of my Unix Tutorial blog – I plan on releasing an SSH eBook later this year which many of you will find useful) – yes, it will cover passwordless SSH and will have some of the best X11 forwarding recipes
  • DTrace tips – still my favorite from Solaris 10 times, DTrace is truly amazing. I’m both surprised and glad that it made its way into many other operating systems. If anything, I’ll be sharing occasional tips on DTrace in Mac OSX. I will assume you’ve all seen Brendan Gregg’s excellent DTrace Tools kit!
  • ZFS – this proved to be so useful and revolutionary at the time that ZFS can now be found in a number of appliances and OS distros for NAS storage management (hello ZFS Build, Nexenta and Zena Box!)

I plan to expand sections of the blog covering these topics, but will gladly add anything you may find useful enough to flag in the comments section.

I’m really looking forward to 2012, and hope there will be more than one occasion when you will find my tips useful and readily applicable. Talk soon!


DTrace: mod_auth_pam and apache2

One of our users has asked me to help with PAM authentication for Apache 2 on one of our remote servers.

Note: going with mod_auth_pam was decision made without me, and I was only asked to help get the chosen solution work. This being said, the recommended way to do external authentication for Apache 2 now is to use the mod_authz_external module and pwauth password authenticator.

It did seem like a rather trivial task – download mod_auth_pam, compile it for Apache 2, make sure everything works. The remote server uses NIS, I could freely log onto it.

And you know what? It didn’t work!

[Read more…]

Debugging PHP with DTrace, part 2

Your next step in observing PHP with DTrace could easily be the following script. It times how much time (in nanoseconds) each Apache process has spent running a particular PHP function.

This script generates a table of Apache PIDs and PHP scripts started within, and later you’ll get a table of all the functions with PIDs which ran them and the time spent.

[Read more…]

Passing command line parameters to DTrace scripts

When you’re writing a DTrace script, often you’d like to pass some parameter in command line. Such an easy task might be really hard unless you read a docs.sun.com section about DTrace scripting, particularly a section about command line parameters macros.

As with sh or bash, you can access command line parameters via $0..$9 macroses. But there’s a catch. Values of these macroses should match a certain definition – a number, an identifier or a string. If there isn’t a match here, you’ll get an error.

Let’s start with a simple script. [Read more…]

DTrace: Aggregations

Quite often it is more important for you to see the whole picture, rather than to know each case when a particular probe fires. For instance, observing a process which consumes a lot of CPU time according to prstat, we ask ourselves: what is this process doing? And often we think about the overall number of system calls executed by this process, and not every particular one of these system calls.

DTrace aggregations are created specifically for answering such questions. Working with sets of numerical data, aggregation functions help us to easily determing average, min and max values of data sets, count numbers and sums of elements in these data sets, and even build frequency distributions.

[Read more…]

DTrace: $target macro variable

Macro variables are meant to make your life MUCH easier when scripting with DTrace. Names of such variables must begin with a $ sign, and DTrace will automatically replace all the macro variables with the appropriate values when parsing your script.

Full list of macro variables can be found here, I will only talk about $target variable this time.

$target specifies the PID of the process you associate with your DTrace script.

[Read more…]

DTrace options: flowindent

One of the most useful (for me) options in DTrace is flowindent. All it does is indent the entry to each function with “->”, and mark the return from this function with “<-“. Doesn’t seem like much, does it? But just look at the results!

The following example catches all the probes of the fbt provided. By default (that’s our case since we’re not changing any other options but flowindent), DTrace registers and prints to the standard output all the probes matched in the tracing process.

[Read more…]

DTrace: count system calls for any process

One more variation of using DTrace allows to track which system calls are made from a certain process. Or do the same for lots of processes, if you use execname and not PIDs…

Such a command line will show you all system calls made by Xorg in the last 5 seconds:

solaris# dtrace -n 'syscall:::entry /execname == "Xorg"/ {@[probefunc] = count(); } tick-5sec {printa(@); clear(@);}'
dtrace: description 'syscall:::entry ' matched 227 probes
CPU     ID                    FUNCTION:NAME
0  36588                       :tick-5sec
writev                                                           20
lwp_sigmask                                                      38
setcontext                                                       38
setitimer                                                        75
read                                                            422
pollsys                                                         426

pS: some time later I’ll hopefully write more about what DTrace and D language are

Basic usage of DTrace

I’m slowly getting used to seeing and using DTrace in my everyday work…

For example, here’s a start of the most basic analysis.

My setup: The system is constantly busy with something, and our task is to find what’s responsible for this. How? One of the possible solutions is to simply watch the system for a certain period of time (5 seconds in our case) to see what process makes most system calles… And it’s clearly seen now that the most active consumer of system calls is Java VM which happens to run a very complex graphics applet at this very moment.

[Read more…]

What’s new in Solaris 10?

Here are just some of the revolutionary changes introduced in Solaris 10:

DTrace – dynamic tracing

DTrace allows you to dynamically trace anything and everything in real time. You can observe processes both in userland and kernel space, watch them as closely as you want and this all with virtually no performance impact. This allows DTrace to be used on a production system being absolutely sure it’s not going to considerably slow the system down or crash it.
I think it’s one of the best Solaris 10 innovations – all the administrators and developers will like it.

ZFS file system (zettabyte file system)

[Read more…]