SystemTap server and client

Introduction

SystemTap is a great technology, it can help you find the direct source of a problem that would be way harder to debug with other tools. But, one thing that happen to me frequently, is that I don’t want to install debuginfo kernel and compiler on production servers just to run some SystemTap scripts to help me with a problem.

One can always manually compile the SystemTap script on a development server (if it match the kernel version and architecture), manually copy it to the production server and run it there. It works, but it’s a little bit hard to maintain and it start to become a pain if you want to make some modification to your SystemTap script and retest it. For each modification, you must recompile the script on the development server, recopy it to the production server and rerun it.

There is a way to automate some steps in this workflow that is somewhat not well documented for the moment : you can install a SystemTap server on your development server and SystemTap clients on your production servers. With that setup, your client connect to the SystemTap server to compile the script and automatically receive the compiled output and run it. That way, you don’t have to install any compiler and debuginfo on your production servers and you don’t have to manually do the copy/compile/run stuff. Talk about saving some time! ;)

The rest of this tutorial is done and work fine with RHEL and Fedora based distributions. For the moment, everything is done with the “root” user and with firewalls disabled. See the TODO section for more informations.

Configuring the SystemTap server

On the development server that you want to run the SystemTap server, install the following packages (the systemtap packages will automatically fetch the kernel-devel, gcc, headers, etc as dependencies) :

  • systemtap
  • kernel-*-debuginfo

These are the standard packages that you need to run SystemTap scripts on your server. You can test that everything works fine with the following command :

stap -r `uname -r` -e 'probe vfs.read {printf("Read Performed\n"); exit()}'

This command will output “Read Performed” as soon as a read operation is done on your server. If everything went well, congratulation, you have installed SystemTap correctly and you can start to use it on this server.

Now, the next step is to transform this setup into a global SystemTap server that will receive request from clients. To achieve this, install the following packages :

  • systemtap-server

Now, start the SystemTap server and Avahi. Also, make this change effective at boot time :

service stap-server start
service avahi-daemon start
chkconfig stap-server on
chkconfig avahi-daemon on

That’s it, the server part is now done!

Configuring the SystemTap clients

Now, on each server that you want to run SystemTap scripts, install the following packages :

  • systemtap-client

After the installation of the package, be sure that Avahi is running and make it persistent across reboot :

service avahi-daemon start
chkconfig avahi-daemon on

Now, copy the SystemTap server certificate on your clients with the following command :

scp root@your.systemtap.server.com:~stap-server/.systemtap/ssl/server/stap.cert /tmp/stap_server.cert

When this command is done, authorize the clients with the server with the following commands :

stap-authorize-server-cert /tmp/stap_server.cert
rm /tmp/stap_server.cert

The SystemTap client can now exchange securely with the SystemTap server. We can now test the configuration with the following command :

stap-client -e 'probe begin { printf("It works!\n"); exit(); }'

If everything is configured correctly, you should see the message in your console after a small delay. You can now run SystemTap script on your production servers!

TODO

The following was assumed in this documentation and will be corrected or more documented in the future :

  • The SystemTap clients all run scripts with the “root” user, which is not recommended for obvious reason. I will update this tutorial with informations on how to configure users to run SystemTap scripts.
  • The firewalls between the server and the clients must allow the Avahi protocol. For the moment, the firewall was disabled but I plan to document how to configure this correctly with the firewall enabled.
  • I also plan to document on how to not use Avahi at all if you don’t want to advertise your SystemTap server to your network.
  • The kernel version is the same between the clients and the server. I will update this tutorial with informations on how to support multiple kernel at a later time.

More informations

This tutorial is heavily based on the following documentation :

8 thoughts on “SystemTap server and client

  1. Pingback: SystemTap client/server tutorial « Jeff's Blog

  2. I have some doubts :
    1. Whether is it necessary to have the linux kernel of the production server to have the debug option enabled ?? If not, then how it will come to know about the virtual addresses of symbols and functions?
    2. In case of the different kernel versions on the production server and the development server, how the .ko file generated in the development server will run on the production server?
    3. How to configure systemtap to run the scripts for all the users in the stapdev group??

    • Thanks for your feedback, I will try to answer your questions the best I can.

      1 – The production server does not need to have the debug kernel installed because all the symbols are already resolved on the development server, at least, this is the way I understand how SystemTap work.

      2 – You need to have all the kernel version installed on your development server and compile your .ko with the “-r” option to the stap command. You need to pass the correct kernel version that is on your production server.

      3 – I need to update this tutorial to include that informations but time is missing currently! I will try to put a priority on this task in the following weeks. But everything is currently in the official SystemTap documentation if you want to look it up till I update this tutorial.

  3. Can you kindly refer me the link of the documentation you are referring to. I want to ask you one more question I want to analyze the process running in the user-space through system-tap but I am not able to find the patches which are needed for that and any proper documentation regarding this.I need to implement these things very urgently. I would be highly thankful to you if you can provide me some good links for that also.

  4. The official documentation is there : http://sourceware.org/systemtap/documentation.html

    For running script as a user, the documentation is there : http://sourceware.org/systemtap/SystemTap_Beginners_Guide/using-usage.html

    And for the userspace probing, there is some documentation there : http://sourceware.org/systemtap/SystemTap_Beginners_Guide/userspace-probing.html

    Keep in mind that the documentation is for the latest SystemTap version (normally available in Fedora). I assume that you use RHEL 5 and I don’t think userspace probing is active in RHEL 5.

  5. I am using Ubuntu 10.10 having kernel 2.6.35. it would be nice if you can tell me that whether it is available for Ubuntu or not and if not then which would be the best option for me?

  6. Thanks a lot jeff for the help..Actually this documentation is providing help only for the kernel space application but none of them is providing the information related to userspace probing. Thanks a lot jeff for quick replies.
    Now I am also planning to move to fedora and try the userspace probing from there only.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s