722 2003-02-17 17:23 gatekeeper reboot could spell doom for globus jobs submitted to Condor 2003-03-10 11:16:31 1 1 1 Unclassified GRAM gt2 Gatekeeper/Jobmanager 1.6 PC All RESOLVED FIXED P2 critical --- 1 tannenba@cs.wisc.edu smartin@mcs.anl.gov adesmet@cs.wisc.edu bester@mcs.anl.gov jfrey@cs.wisc.edu jms@mcs.anl.gov pfc@cs.wisc.edu roy@cs.wisc.edu vwelch@uiuc.edu tannenba@cs.wisc.edu 2003-02-17 17:23:04 Hi - In Globus 2.2.4 (and probably anything based on gram 1.6), there is a nasty bug in the source file: gram/jobmanager/setup/condor/condor.in which ultimately gets installed in: lib/perl/Globus/GRAM/JobManager/condor.pm script The problem is the value used for the variable condor_logfile. It looks like it uses tmpdir which is set to /tmp in Core::Paths, which ultimately tells Condor to place the job log someplace in /tmp/. This is bad!! The Condor job log has the same requirements has the jobmanager state files -- specifically, it : a) must be persistent even across system reboots (and thus /tmp is a bad idea) b) must NOT be on NFS volumes because problems w/ file locks on NFS. Perhaps condor_logfile variable should point to the same subdirectory that jobmanager state files go? Thanks, Todd UW-Madison Condor Team smartin@mcs.anl.gov 2003-02-18 13:09:17 Todd, what about using the GLOBUS_LOCATION env var in the condor perl script? For now that would probably be the best way to go. So instead of this: $self->{condor_logfile} = $Globus::Core::Paths::tmpdir . "/gram_condor_log." . $self->{JobDescription}->uniq_id(); Do this: $self->{condor_logfile} = $ENV{GLOBUS_LOCATION} . "/gram_condor_log." . $self->{JobDescription}->uniq_id(); It is no guarantee that the $GL is not on NFS, but I think it typically isn't. The JM code writes the state file to $GL/tmp/... unless the JM is configured with "-state-file-dir <dir>" A more complete solution would be to set another env var like GLOBUS_JM_STATE_DIR and then the condor script could look for that and fallback to $GL. Maybe better yet would be to change the name of the -state-file-dir argument and ENV var that the JM sets to something that indicates it's use like -local- tmp-dir (env var GLOBUS_LOCAL_TMP_DIR). Then the condor script could use the GLOBUS_LOCAL_TMP_DIR env var to set the location of the condor_logfile. What do you think? -Stu jfrey@cs.wisc.edu 2003-02-18 14:05:01 As I suggested in my duplicate bug report (#721, sorry about that), it seems most obvious to put the condor user log files in the same directory as the job state files. Both represent persistent information about the job. Also, both get locked so they can't be on NFS. I think local-tmp-dir is a bad name, since it implies that files written there don't need to be long-lived. I think job-state-dir or jm-state-dir is better. -- Jaime smartin@mcs.anl.gov 2003-02-18 14:27:34 *** Bug 721 has been marked as a duplicate of this bug. *** tannenba@cs.wisc.edu 2003-02-20 09:05:01 At 01:09 PM 2/18/2003, bugzilla-daemon@mcs.anl.gov wrote: >http://bugzilla.globus.org/bugzilla/show_bug.cgi?id=722 > >smartin@mcs.anl.gov changed: > > What |Removed |Added >---------------------------------------------------------------------------- > Status|NEW |ASSIGNED > > > >------- Additional Comments From smartin@mcs.anl.gov 2003-02-18 13:09 ------- >Todd, > >what about using the GLOBUS_LOCATION env var in the condor perl script? For >now that would probably be the best way to go. > >So instead of this: > > $self->{condor_logfile} = $Globus::Core::Paths::tmpdir > . "/gram_condor_log." > . $self->{JobDescription}->uniq_id(); > >Do this: > > $self->{condor_logfile} = $ENV{GLOBUS_LOCATION} > . "/gram_condor_log." > . $self->{JobDescription}->uniq_id(); I don't think GLOBUS_LOCATION is a good idea, because wouldn't most sites disallow world writes to this directory? >It is no guarantee that the $GL is not on NFS, but I think it typically isn't. >The JM code writes the state file to $GL/tmp/... I think it makes good sense to set condor_logfile to the same place jobmanager state files go ($GL/tmp), since they both have the same requirements: (a) must be local [not on nfs], (b) must be a subdirectory writable by all users in the gridmap file, (c) must be persistent until explicitly removed ( i.e. not be deleted at bootup or by cron or some such ). >Maybe better yet would be to change the name of the -state-file-dir argument >and ENV var that the JM sets to something that indicates it's use like -local- >tmp-dir (env var GLOBUS_LOCAL_TMP_DIR). Then the condor script could use the >GLOBUS_LOCAL_TMP_DIR env var to set the location of the condor_logfile. > >What do you think? Sure, that'd would probably be the cleanest. But I would remove the term "TMP" from the actual name of the environment variable, since that implies to sysadmins it is safe to point it at /tmp, which is definitely *not* safe! Perhaps GLOBUS_SPOOL_DIR ? very best regards from CERN, Todd p.s. Joe Bester is sitting here, and he says that does does _not_ want to say hi to you. Perhaps you guys have worked together too much? ;^) -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Todd Tannenbaum University of Wisconsin-Madison Condor Project Research Department of Computer Sciences tannenba@cs.wisc.edu 1210 W. Dayton St. Rm #3357 http://www.cs.wisc.edu/~tannenba Madison, WI 53706-1685 Phone: (608) 263-7132 FAX: (240) 359-5654 bester@mcs.anl.gov 2003-03-10 11:16:31 Following update packages use the state file directory for condor log files. ftp://ftp.globus.org/pub/gt2/2.2/2.2.4/updates/src/globus_gram_job_manager-3.7.tar.gz ftp://ftp.globus.org/pub/gt2/2.2/2.2.4/updates/src/globus_gram_job_manager_setup_condor-1.2.tar.gz joe