This option is recognized by pbench-register-tool and pbench-unregister-tool: it specifies the name of the tool that is to be (un)registered. pbench-list-tools with the --name option list all the groups that contain the named tool.

This option is recognized by the benchmark scripts (see Available benchmark scripts above) which use it as a tag for the directory where the benchmark is going to run. The default value is empty. The run directory for the benchmark is constructed this way:

$pbench_run/${benchmark}_${config}_$date

where $pbench_run and $date are set by the /opt/pbench-agent/base script and $benchmark is set to the obvious value by the benchmark script; e.g. a fio run with config=foo would run in the directory /var/lib/pbench-agent/fio_foo_2014-11-10_15:47:04.

This option is recognized by pbench-start-tools, pbench-stop-tools, pbench-tool-trigger and pbench-postprocess-tools. It specifies the directory where the tools are going to stash their data. The default value is /tmp. Each group then uses it as a prefix for its own stash, which has the form $dir/tools-$group . Part of the stash is the set of cmds to start and stop the tools - they are stored in $dir/tools-$group/cmds. The output of the tool is in $dir/tools-$group/$tool.txt.

This option has to be specified identically for each command when invoking these commands (actually, each of the commands should be invoked with the identical set of all options, not just --dir.)

If you use these tools explicitly (i.e. you don't use one of the benchmark scripts), it is highly recommended that you specify this option explicitly and not rely on the /tmp default. For one, you should make sure that different iterations of your benchmark use a different value for this option, otherwise later results will overwrite earlier ones.

N.B. If you want to run pbench-move-results or pbench-copy-results after the end of the run, your results should be under /var/lib/pbench-agent: pbench-move/copy-results does not know anything about your choice for this option; it only looks in /var/lib/pbench-agent for results to upload. So if you are planning to use pbench-move/copy-results, make sure that the specified directory is a subdirectory of /var/lib/pbench-agent.

Pbench can register tools on remote hosts, start them and stop them remotely and gather up the results from the remote hosts for post-processing. The model is that one has a controller or orchestrator and a bunch of remote hosts that participate in the benchmark run.

The Pbench setup is as follows: pbench-register-tool-set or pbench-register-tool is called on the controller with the --remote option, once for each remote host:

for remote in $remotes ;do
   pbench-register-tool-set --remote=$remote --label=foo --group=$group
done

That has two effects: it adds a stanza for the tool to the appropriate tools-$group directory on the remote host and it also adds a stanza like this to the controller tools-$group directory's file for the remote host:

remote@<host>:<label>

The label is optionally specified with --label and is empty by default.

When pbench-start-tools is called on the controller, it starts the local collection (if any), but it also interprets the above stanzas and starts the appropriate tools on the remote hosts. Similarly for pbench-stop-tools and pbench-postprocess-tools.

TBD

Tool scripts are mostly boilerplate: they need to take a standard set of commands (–install, –start, –stop, –postprocess) and a standard set of options (–iteration, –group, –dir, –interval, –options). Consequently, the easiest thing to do is to take an existing script and modify it slightly to call the tool of your choice. I describe here the case of turbostat.

There are some tools that timestamp each output stanza; there are others that do not. In the former case, make sure to use whatever option the tool requires to include such timestamps (e.g. vmstat -t on RHEL6 or RHEL7 - but strangely not on Fedora 20 - will produce such timestamps).

There are some tools that are included in the default installation - others need to be installed separately. Turbostat is not always installed by default, so the tool script installs the package (which is named differently on RHEL6 and RHEL7) if necessary. In some cases (e.g. the sysstat tools), we provide an RPM in the pbench repo and the tool script makes sure to install that if necessary.

The only other knowledge required is where the tool executable resides (usually /usr/bin/ or /usr/local/bin/ - /usr/bin/turbostat in this case) and the default options to pass to the tool (which can be modified by passing –options to the tool script).

So here are the non-boilerplate portions of the turbostat tool script. The first interesting part is to set tool_bin to point to the binary:

# Defaults
tool=$script_name
tool_bin=/usr/bin/$tool

This only works if the script is named the same as the tool, which is encouraged. If the installed location of your tool is not /usr/bin, then adjust accordingly.

Since turbostat does not provide a timestamp option, we define a datalog script to add timestamps (no need for that for vmstat e.g.) and use that as the tool command:

case "$script_name" in turbostat)
   tool_cmd="$script_path/datalog/$tool-datalog $interval $tool_output_file"
   ;;
esac

The datalog script uses the pbench-log-timestamp pbench utility to timestamp the output. It will then be up to the postprocessing script to tease out the data appropriately.

The last interesting part dispatches on the command - the install is turbostat-specific, but the rest is boilerplate: --start just executes the tool_cmd as defined above and stashes away the pid, so that --stop can kill the command later; --postprocess calls the separate post-processing script (see below):

release=$(awk '{x=$7; split(x, a, "."); print a[1];}' /etc/redhat-release)
case $release in
  6)
     pkg=cpupowerutils
     ;;
  7)
     pkg=kernel-tools
     ;;
  *)
     # better be installed already
      ;;
esac
case "$mode" in
  install)
    if [ ! -e $tool_bin ]; then
      yum install $pkg
      echo $script_name is installed
    else
      echo $script_name is installed
    fi
  start)
    mkdir -p $tool_output_dir
    echo "$tool_cmd" >$tool_cmd_file
    debug_log "$script_name: running $tool_cmd"
    $tool_cmd >>"$tool_output_file" & echo $! >$tool_pid_file
    wait
    ;;
  stop)
     pid=`cat "$tool_pid_file"`
     debug_log "stopping $script_name"
     kill $pid && /bin/rm "$tool_pid_file"
     ;;
  postprocess)
     debug_log "postprocessing $script_name"
     $script_path/postprocess/$script_name-postprocess $tool_output_dir
     ;;
esac

Finally, there is the post-processing tool: the simplest thing to do is nothing. That's currently the case for the turbostat post-processing tool, but ideally it should produce a JSON file with the data points and an HTML file that uses the nv3 library to plot the data graphically in a browser. See the postprocess directory for examples, e.g. iostat postprocessing tool.

TBD

Running

pbench-user-benchmark -- sleep 60

will start whatever data collections are specified in the default tool group, then sleep for 60 seconds. At the end of that period, it will stop the running collections tools and postprocess the collected data. Running pbench-move-results afterwards will move the results to the results server as usual.

Pbench is a set of building blocks, so it allows you to use it in many different ways, but it also makes certain assumptions which if not satisfied, lead to problems.

Let's assume that you want to run a number of iozone experiments, each with different parameters. Your script probably contains a loop, running one experiment each time around. If you can change your script so that it executes one experiment specified by an argument, then the best way is to use the pbench-user-benchmark script:

pbench-register-tool-set for exp in experiment1 experiment2 experiment3 ;do    pbench-user-benchmark --config $exp -- my-script.sh $exp done pbench-move-results

The results are going to end up in directories named /var/lib/pbench-agent/pbench-user-benchmark_$exp_$ts for each experiment (unfortunately, the timestamp will be recalculated at the beginning of each pbench-user-benchmark invocation), before being uploaded to the results server.

Alternatively, you can modify your script so that each experiment is wrapped with start/stop/postprocess-tools and then call pbench-move-results at the end:

pbench-register-tool-set dir=/var/lib/pbench-agent tool_group=default typeset -i iter=1 for exp in experiment1 experiment2 experiment3 ;do    pbench-start-tools --group=$tool_group --dir=$dir --iteration=$iter       pbench-stop-tools --group=$tool_group --dir=$dir --iteration=$iter    pbench-postprocess-tools --group=$tool_group --dir=$dir --iteration=$iter    iter=$iter+1 done pbench-move-results

N.B.You need to invoke the pbench-{start,stop,postprocess}-tools scripts with the same arguments.

See the Installation Instructions.