Systemtap is amazing, plain and simple. I’ve only begun to scratch the surface myself, but I can already see its power and have even used it in a few cases to make my life easier.
I found this example in a document somewhere, and I love it. Have you ever had one of those cases where a process was being mysteriously killed off on a server and you couldn’t quite figure out why? Well in comes Systemtap. With just a few lines of code:
if ( sig_name == "SIGKILL ") printf ("%s was sent to %s ( pid :%d ) by %s uid :%d \n " , sig_name , pid_name , sig_pid , execname ( ) , u i d ( ) ) }
in a Systemtap script. Execute it and any time a “SIGKILL” is sent to the kernel for any reason (SIGKILL == kill -9), it outputs what was killed and what pid / process executed it.
# stap /usr/share/systemtap/tapset/signal.stp SIGKILL was sent to saslauthd (pid :6202) by AntiCloseWait.s uid :0
In this case “AntiCoseWait.s” was “AntiCloseWait.sh”, a long-forgotten cronjob.
Simple, Powerful, Flexible. One of my favorite new tools to use.