PBS tools
Copyright 2005-2008, Ohio Supercomputer Center

Contained herein are several utilities that have been developed at OSC
to aid in the administration and management of PBS variants (including
OpenPBS, PBS Pro, and TORQUE).  They have been developed primarily on
Linux clusters, but most of them should be applicable to virtually any
system running PBS.


User tools:
-----------

bin/ja -- Job accounting within a PBS job; similar to "ja" in NQE.

bin/pbsdcp -- Distributed copy command within a PBS job.  This can
optionally include an MPI-based scatter functionality that should
scale better than the default rcp-based approach.

bin/qps -- ps for PBS jobs; relies on the "all" command.

bin/qpeek -- Allows users to look at spooled output of running jobs.

bin/qexec -- Workalike for SGE qlogin/qrsh/qsh

bin/parallel-command-processor -- An MPI program for running an array
of serial jobs in the context of a single parallel job (a.k.a. the poor
man's job arrays).

Admin tools:
------------

sbin/dezombify -- Requeue a "running" PBS job that's not actually
running because it had JOIN JOB failures.  Relies on the "all"
command; requires root privilege.

sbin/qtracejob.pbs-server -- Wrapper around the tracejob command to
give a time-sorted list of log messages from all hosts involved.  Must
run on host where pbs_server resides; requires root privilege.

sbin/reaver -- Find (and optionally clean up) processes on a host
which have not been allocated jobs on that host; requires root
privilege.

sbin/showscript.pbs-server -- Show the script associated with a jobid.
Must run on host where pbs_server resides; requires root privilege.


Reporting tools:
----------------

sbin/jobstats -- Generate an HTML report summarizing PBS job
statistics over a period of months.  Relies on gnuplot; requires root
privilege.

sbin/find-outlyers -- Extract statistical outlyers from the
jobdist.dat file generated by jobstats.


Database/web tools:
-------------------
Note:  these all assume a MySQL DB back end running on the same server
as pbs_server and PHP-enabled httpd.

etc/create-tables.sql -- Create "pbsacct" database, table, and two
user accounts (pbsacct and webapp) used by the other DB tools.  To
initialize, run "mysql -u root -p passwd <create-tables.sql".

sbin/job-db-update -- Parses PBS job accounting records and inserts
them into DB.

sbin/jobscript-to-db -- Inserts one or more job scripts into the job DB.

sbin/spool-jobscripts -- Spools newly submitted jobscripts to /tmp and
then invokes jobscript-to-db on them.

web/jobinfo.php -- Presents information on a single job based from the
job DB.

web/jobs-by-user.php -- Presents information on all jobs by a given
user (over an optional date range) from the job DB.

web/jobs-by-group.php -- Presents information on all jobs by a given
group (over an optional date range) from the job DB.

web/jobs-by-node.php -- Presents information on all jobs on a given
node (over an optional date range) from the job DB.

web/jobstats*.php -- Presents statistics on all jobs (over an optional
date range) from the job DB.

web/software-usage*.php -- Presents information on software
usage on all jobs (over an optional date range) from the job DB.

web/active-users.php -- Presents a list of the most active users over
a date range.

web/active-groups.php -- Presents a list of the most active groups over
a date range.

web/problem-jobs.php -- Searches for potentially problematic jobs over
a date range.  (Note that what constitutes "problematic" is extremely
site specific.)

web/usage-summary.php -- Presents a usage summary over a date range.

web/site-specific.php -- Site specific settings.

web/sql-term.php -- "Terminal" for running arbitrary SQL queries on
job DB.

examples/db-example -- An example of putting job data from multiple PBS
instances into a single job DB.


Suggestions:
------------

Comments and suggestions on this software are welcomed; please send
them to Troy Baer <troy@osc.edu>.