Systemtap is amazing, plain and simple. I’ve only begun to scratch the surface myself, but I can already see its power and have even used it in a few cases to make my life easier.

I found this example in a document somewhere, and I love it. Have you ever had one of those cases where a process was being mysteriously killed off on a server and you couldn’t quite figure out why? Well in comes Systemtap. With just a few lines of code:

if ( sig_name == "SIGKILL ")
printf ("%s was sent to %s ( pid :%d ) by %s uid :%d \n " ,
 sig_name , pid_name , sig_pid , execname ( ) , u i d ( ) )

in a Systemtap script. Execute it and any time a “SIGKILL” is sent to the kernel for any reason (SIGKILL == kill -9), it outputs what was killed ¬†and what pid / process executed it.

For example:

# stap /usr/share/systemtap/tapset/signal.stp
SIGKILL was sent to saslauthd (pid :6202) by AntiCloseWait.s uid :0

In this case “AntiCoseWait.s” was “”, a long-forgotten cronjob.

Simple, Powerful, Flexible. One of my favorite new tools to use.