Bug 4411 - Pre WS GRAM Auditing
: Pre WS GRAM Auditing
Status: RESOLVED FIXED
: GRAM
Campaign
: development
: Macintosh All
: P3 normal
: 4.0.5
Assigned To:
:
: Auditing
:
: 4409
  Show dependency treegraph
 
Reported: 2006-05-17 11:35 by
Modified: 2007-02-14 14:35 (History)


Attachments


Note

You need to log in before you can comment on or make changes to this bug.


Description From 2006-05-17 11:35:08
Title: Pre WS GRAM Auditing

Technologies:    Globus Resource Allocation Manager (GRAM)

Definition:

Production grids need a way to produce audit trails for job submissions.  For
Pre WS GRAM, only basic information is written to the gatekeeper's log file
(the submitter's grid DN and mapped local account.  This is not an effective
way to programmatically find or query for audit entries.  Nor is it an
efficient way to store a large number of audit records.

A Pre WS GRAM audit record per job, indexed by a grid job id, can be useful to
provide access to audit or other local job related information.  For example,
it could be used to join to a grid's local accounting database.  Often, these
accounting databases do not have knowledge of the service's grid job id (the
service that provided access to the local compute resource).

This feature will be turned off by default.  A new job manager configuration
option will be added that specifies the directory where the files should be
written.

To accommodate this need, it is proposed that Pre WS GRAM be altered to
generate audit records in a file that can be easily uploaded into a database. 
This campaign must use the same DB schema used in the WS GRAM Auditing
campaign.  A simple java gram audit upload program will read the individual job
audit files and insert the comma separated records into the gram audit table
using the JDBCAppender class used by WS GRAM.

The reason for Pre-WS GRAM to use a different method than WS GRAM, writing
audit files instead of direct DB inserts, is to have a more trusted entity
writing to the database, instead of requiring all Pre-WS GRAM users with the
ability to do so.

The following PostgreSQL database schema will be used:

create table gram_audit_table (
     "job_grid_id" varchar(256) primary key,
     "local_job_id" varchar(512) unique,
     "submission_job_id" varchar(512) not null,
     "subject_name" varchar(256) not null,
     "username" varchar(16) not null,
     "creation_time" timestamp not null,
     "queued_time" timestamp not null,
     "stage_in_gid" varchar(256),
     "stage_out_grid_id" varchar(256),
     "clean_up_grid_id" varchar(256),
     "globus_toolkit_version" varchar(16) not null,
     "resource_manager_type" varchar(16) not null,
     "job_description" text not null,
     "success_flag" boolean not null
     );

The format of the string that is logged to the database appender is a
comma-separated list of double-quoted strings. The items in the list MUST be in
the same order as listed in the above schema. Any double quotes within the
strings MUST be converted to the substring """.  Any value that is allowed
to be null by the schema and is indeed null MUST be indicated with the string
"NULL". Such strings will be converted to actual null values in the database.
Here is a Pre-WS GRAM example:

"https://penny.dyndns.org:30004/7898/1146859550/","123.0","NULL","/DC=org/DC=doegrids/OU=People/CN=Peter
G Lane 364243","lane","Wed May 03 16:06:44 MDT 2006","Wed May 03 16:06:45 MDT
2006","NULL","NULL","NULL","4.0 Community","fork",
"&("rsl_substitution" = ("GLOBUSRUN_GASS_URL"
"https://penny.dyndns.org:30002" ) ) ("stderr" =
$("GLOBUSRUN_GASS_URL") # "/dev/stderr" )
("stdout" = $("GLOBUSRUN_GASS_URL") #
"/dev/stdout" )("executable" = "/bin/date"
)","true" 

Modifications to the GRAM setup packages will set the audit file directory when
requested. This directory must have the sticky-bit set on it to ensure that
users may not modify others' audit record files. The owner of the audit
directory must be the same as the user which will be running the database
upload program.

The pre-ws Job Manager will, at final job termination (FAILED or DONE state)
write a record to a unique file in a directory specified by a configuration
file. 

The audit file upload program will be run periodically and will create audit
records from the user audit files. It must ensure that the files are owned by
the user whose name is in the username field of the audit record before
uploading to the database. Once the record is uploaded, the program will remove
the audit file.

Deliverables:

1) Modified job manager configuration and code to produce audit record files
2) Modified job manager setup script to create audit file directory
2) New gram audit upload program
3) Commit to CVS trunk
4) Commit to CVS 4.0 community branch
5) 4.0.1 Pre-WS GRAM audit update packages

Tasks:

1) Add new job manager configuration option: -audit-directory which is the
directory where audit files will be written.
2) Modify job manager to record timestamps for creation and queued time values.
3) Modify job manager to write audit record before terminating.
4) Add new option to setup scheduler-specific packages to create/set auditing
directory in job manager configs.
4) Write audit upload program

Time Estimate:

4 days
------- Comment #1 From 2006-06-02 16:27:55 -------
The auditing updates for the job manager are committed to community branch and
trunk.

An auditing update script which reads the audit records and passes the valid
ones to the java audit appender program is committed to community branch and
trunk, but the appender is currently not building in trunk due to unrelated
wsgram build issues.

The changes related to this work can be found in CVS using
cvs diff -r campaign_4411_point -r campaign_4411_branch in gram/jobmanager and
wsrf/ws-gram/utils/java

joe
------- Comment #2 From 2006-06-14 10:03:59 -------
Update bundle for 4.0.x branch is located at
http://www.mcs.anl.gov/~bester/bug4411/gram_audit-1.0-src_bundle.tar.gz

Campaign completed.
------- Comment #3 From 2006-09-20 17:33:56 -------
Just for the record, here are the steps I took to configure this stuff. This is
based off of a globus_4_0_community branch installation.

1) Add the following lines to $GLOBUS_LOCATION/log4j.propertie:

# AUDIT
log4j.appender.AUDIT=org.globus.exec.utils.AuditDatabaseAppender
log4j.appender.AUDIT.layout=org.apache.log4j.PatternLayout
log4j.category.org.globus.exec.utils.AuditDatabaseAppender.audit=INFO, AUDIT
log4j.additivity.org.globus.exec.service.exec.StateMachine.audit=false

2) Create an audit record directory that is owned by 'globus' and has the
following permissions: rws-wsrwx

4) Add the following line to $GLOUBS_LOCATION/etc/globus-job-manager.conf:

        -audit-directory <audit record directory>

5)Install the following packages from the community branch (a 4.0.x
installation will need the GPT update bundle attached to this bugzilla entry):

globus_gram_job_manager_auditing
globus_gram_job_manager_auditing_setup_scripts

6) Remove the following code from the gloubs-gram-audit script:

    # don't process entries where the owner != the job user
    #if ($record_entries[3] ne (getpwuid($owner))[0])
    #{
        #next;
    #}

7) Edit $GLOBUS_LOCATION/etc/globus-job-manager-audit.conf for the correct
database connection parameters.

8) Check $GLOBUS_LOCATION/etc/globus-job-manager-audit.conf to make sure it is
only readable by the 'globus' user so that the database password is not
publically readable.

9) Creat a script in $GLOUBS_LOCATION/libexec/ called cron-globus-gram-audit
that runs globus-gram-audit after first settin up the environment needed to run
it. The following is an example from TeraGrid:

----------------------------------------------------------------------
#!/bin/sh

eval "`/software/linux-sles8-ia64/softenv-1.4.2/bin/soft-dec sh add
@teragrid-basic`"
eval "`/software/linux-sles8-ia64/softenv-1.4.2/bin/soft-dec sh add
+java-1.4.2_07-b05-sun-r1`"

export GLOBUS_LOCATION=/home/globus/Audit/globus
. $GLOBUS_LOCATION/etc/globus-user-env.sh

$GLOBUS_LOCATION/libexec/globus-gram-audit
----------------------------------------------------------------------

10) Created the following cron job (crontab -e) for the 'globus' user:

0,15,30,45 * * * * <GLOBUS_LOCATION>/libexec/cron-globus-gram-audit