4253 2006-03-06 16:38 unable to monitor job for state changes 2006-03-21 14:14:34 1 1 1 Unclassified GRAM wsrf managed execution job service 4.0.1 Macintosh All RESOLVED FIXED P3 normal 4.0.2 1 smartin@mcs.anl.gov lane@mcs.anl.gov bester@mcs.anl.gov lane@mcs.anl.gov madduri@mcs.anl.gov smartin@mcs.anl.gov smartin@mcs.anl.gov 2006-03-06 16:38:07 Frequently the larger throughput runs are turning up a "unable to monitor job state changes" fault. An example: http://skynet-login.isi.edu/gram-testing/results/8ad818eb-097c-4200-8b70-2ec105808b7a/ throughput-tester.log lane@mcs.anl.gov 2006-03-06 17:23:22 As was discussed on the MUD, it's possible this is simply a problem with the system reusing PIDs too soon. A race condition between exists in this situation between the unregistering of the PID from one job and the registering of the PID again from a newly submitted job. Debug logging should be turned on for the org.globus.exec.service.exec.JobManagerScript class. This allows for the comparison of PIDs. To support the theory, one should look to see if two instances of the adapter returning the same PID occur close to each other. bacon@mcs.anl.gov 2006-03-16 15:25:15 It's happening every day in the tests, as far as I can tell. Is there anything you can do to add monitoring code to figure out what's causing it? If it's a race, it's pretty reliable. lane@mcs.anl.gov 2006-03-16 16:43:16 I suggested a way to see if PIDs were being reused in a short amount of time. Are you asking me to add code to explicitly look for this situtation? I'd rather have some evidence in support of the theory before I risk wasting my time trying to fix it. rynge@isi.edu 2006-03-16 18:17:13 The reproducability of this makes me think it is not a pid problem, but something else. Maybe it will be enough to log what the conflicting job id is to get a clue on what is going on. rynge@isi.edu 2006-03-16 20:03:59 http://skynet-login.isi.edu/gram-testing/ See test 1143, 1144, 1146, and 1147 An interesting note is that I had to make the tests run for longer (2000 seconds vs 1200 seconds) in order to trigger the problem with logging enabled. This is probably because not enough jobs were run, about 800. With the longer test, about 1300 jobs were run. bester@mcs.anl.gov 2006-03-20 09:05:54 Created an attachment (id=891) Patch to globus_fork_starter Attaching a patch to prepend a uuid to the fork pid in the fork starter. This should eliminate job id reuse problems. No changes should be needed for the SEG for this patch. bacon@mcs.anl.gov 2006-03-21 10:00:46 What's the status on this? It looks like the HEAD version of the test is passing now, but the branch is still failing. rynge@isi.edu 2006-03-21 12:18:10 I tested the patch, and it looks good to me: http://skynet-login.isi.edu/gram-testing/ Test 1413-1418 bester@mcs.anl.gov 2006-03-21 14:14:34 I've committed this patch to CVS in trunk and 4.0 branch after Mats confirmed the 4/4/never test succeeds. 891 2006-03-20 09:05 Patch to globus_fork_starter uuid.diff text/plain 6995 bester@mcs.anl.gov PyBnbG9idXMtZm9yay1zdGFydGVyCj8gZ2xvYnVzcnVuLmxvZwo/IHJzbAo/IHVpZC10aW1lc3Rh bXAtaW4tam9iaWQuZGlmZgo/IHV1aWQuZGlmZgpJbmRleDogZm9ya19zdGFydGVyLmMKPT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PQpSQ1MgZmlsZTogL2hvbWUvZ2xvYmRldi9DVlMvZ2xvYnVzLXBhY2thZ2VzL3dzLWdyYW0v am9iX21hbmFnZW1lbnQvZm9yay9jL3NvdXJjZS9mb3JrX3N0YXJ0ZXIuYyx2CnJldHJpZXZpbmcg cmV2aXNpb24gMS42CmRpZmYgLXUgLXIxLjYgZm9ya19zdGFydGVyLmMKLS0tIGZvcmtfc3RhcnRl ci5jCTIwIEFwciAyMDA1IDE1OjQ5OjAxIC0wMDAwCTEuNgorKysgZm9ya19zdGFydGVyLmMJMjAg TWFyIDIwMDYgMTU6MDM6NTEgLTAwMDAKQEAgLTMwLDYgKzMwLDcgQEAKICAgICBzaXplX3QgdGFz a2xlbjsKICAgICBzaXplX3QgdGFza2J1ZmxlbjsKICAgICBzaXplX3QgcGlkY291bnQ7CisgICAg Y2hhciAqIGpvYmlkX3ByZWZpeDsKIAogICAgIGludCBjb3VudDsKIApAQCAtMTI1LDYgKzEyNiw3 IEBACiB2b2lkCiBnbG9idXNfbF9mb3JrX2xvZ19zdGF0ZV9jaGFuZ2UoCiAgICAgcGlkX3QgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgcGlkLAorICAgIGdsb2J1c19mb3JrX3Rhc2tfdCAg ICAgICAgICAgICAgICAgIHRhc2ssCiAgICAgZ2xvYnVzX2dyYW1fcHJvdG9jb2xfam9iX3N0YXRl X3QgICAgam9iX3N0YXRlLAogICAgIGludCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IGV4aXRfY29kZSk7CiAKQEAgLTE0Myw3ICsxNDUsNiBAQAogICAgIGdsb2J1c194aW9faGFuZGxl X3QgICAgICAgICAgICAgICAgIHN0ZGluX2hhbmRsZTsKICAgICBnbG9idXNfZm9ya190YXNrX3Qg ICAgICAgICAgICAgICAgICBuZXdfdGFzazsKICAgICBwaWRfdCAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBwaWQ7Ci0gICAgY2hhciAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg cGlkYnVmWzY0XTsKICAgICBpbnQgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBzdGF0 dXM7CiAgICAgZ2xvYnVzX3hpb19kcml2ZXJfdCAgICAgICAgICAgICAgICAgZmlsZV9kcml2ZXI7 CiAgICAgZ2xvYnVzX3hpb19zdGFja190ICAgICAgICAgICAgICAgICAgZmlsZV9zdGFjazsKQEAg LTI4MCw3ICsyODEsNyBAQAogCiAgICAgcmVzdWx0ID0gZ2xvYnVzX3hpb19yZWdpc3Rlcl9yZWFk KAogICAgICAgICAgICAgc3RkaW5faGFuZGxlLAotICAgICAgICAgICAgbmV3X3Rhc2stPnRhc2ss CisgICAgICAgICAgICAoZ2xvYnVzX2J5dGVfdCAqKSBuZXdfdGFzay0+dGFzaywKICAgICAgICAg ICAgIG5ld190YXNrLT50YXNrYnVmbGVuIC0gMSwKICAgICAgICAgICAgIDEsCiAgICAgICAgICAg ICBOVUxMLApAQCAtMzA3LDE1ICszMDgsMTggQEAKICAgICAgICAgICAgICAgICAvKiBOb25ibG9j a2luZyBwZWVrIGF0IGNoaWxkIHBpZCBsaXN0ICovCiAgICAgICAgICAgICAgICAgcGlkID0gd2Fp dHBpZCgtMSwgJnN0YXR1cywgV05PSEFORyk7CiAKLSAgICAgICAgICAgICAgICBzcHJpbnRmKHBp ZGJ1ZiwgIiVsdSIsICh1bnNpZ25lZCBsb25nKSBwaWQpOwotCiAgICAgICAgICAgICAgICAgaWYg KHBpZCA+IDApCiAgICAgICAgICAgICAgICAgeworICAgICAgICAgICAgICAgICAgICBuZXdfdGFz ayA9IGdsb2J1c19oYXNodGFibGVfcmVtb3ZlKAorICAgICAgICAgICAgICAgICAgICAgICAgJmds b2J1c19sX2ZvcmtfYWN0aXZlX3Rhc2tzLAorICAgICAgICAgICAgICAgICAgICAgICAgKHZvaWQg KikgcGlkKTsKKwogICAgICAgICAgICAgICAgICAgICAvKiBTb21lIHByb2Nlc3MgY29tcGxldGVk ICovCiAgICAgICAgICAgICAgICAgICAgIGlmIChXSUZFWElURUQoc3RhdHVzKSkKICAgICAgICAg ICAgICAgICAgICAgewogICAgICAgICAgICAgICAgICAgICAgICAgZ2xvYnVzX2xfZm9ya19sb2df c3RhdGVfY2hhbmdlKAogICAgICAgICAgICAgICAgICAgICAgICAgICAgIHBpZCwKKyAgICAgICAg ICAgICAgICAgICAgICAgICAgICBuZXdfdGFzaywKICAgICAgICAgICAgICAgICAgICAgICAgICAg ICBHTE9CVVNfR1JBTV9QUk9UT0NPTF9KT0JfU1RBVEVfRE9ORSwKICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBXRVhJVFNUQVRVUyhzdGF0dXMpKTsKICAgICAgICAgICAgICAgICAgICAgfQpA QCAtMzIzLDE0ICszMjcsMTEgQEAKICAgICAgICAgICAgICAgICAgICAgewogICAgICAgICAgICAg ICAgICAgICAgICAgZ2xvYnVzX2xfZm9ya19sb2dfc3RhdGVfY2hhbmdlKAogICAgICAgICAgICAg ICAgICAgICAgICAgICAgIHBpZCwKKyAgICAgICAgICAgICAgICAgICAgICAgICAgICBuZXdfdGFz aywKICAgICAgICAgICAgICAgICAgICAgICAgICAgICBHTE9CVVNfR1JBTV9QUk9UT0NPTF9KT0Jf U1RBVEVfRkFJTEVELAogICAgICAgICAgICAgICAgICAgICAgICAgICAgIDApOwogICAgICAgICAg ICAgICAgICAgICB9CiAKLSAgICAgICAgICAgICAgICAgICAgbmV3X3Rhc2sgPSBnbG9idXNfaGFz aHRhYmxlX3JlbW92ZSgKLSAgICAgICAgICAgICAgICAgICAgICAgICZnbG9idXNfbF9mb3JrX2Fj dGl2ZV90YXNrcywKLSAgICAgICAgICAgICAgICAgICAgICAgICh2b2lkICopIHBpZCk7Ci0KICAg ICAgICAgICAgICAgICAgICAgaWYgKG5ld190YXNrICE9IE5VTEwpCiAgICAgICAgICAgICAgICAg ICAgIHsKICAgICAgICAgICAgICAgICAgICAgICAgIG5ld190YXNrLT5waWRjb3VudC0tOwpAQCAt NDg2LDcgKzQ4Nyw3IEBACiAgICAgICAgICAgICAvKiBSb29tIGluIHRoZSB0YXNrIGJ1ZmZlciB0 byByZWFkIG1vcmUgZGF0YSAqLwogICAgICAgICAgICAgcmVzdWx0ID0gZ2xvYnVzX3hpb19yZWdp c3Rlcl9yZWFkKAogICAgICAgICAgICAgICAgICAgICBoYW5kbGUsCi0gICAgICAgICAgICAgICAg ICAgIHRhc2stPnRhc2sgKyB0YXNrLT50YXNrbGVuLAorICAgICAgICAgICAgICAgICAgICAoZ2xv YnVzX2J5dGVfdCAqKSB0YXNrLT50YXNrICsgdGFzay0+dGFza2xlbiwKICAgICAgICAgICAgICAg ICAgICAgdGFzay0+dGFza2J1ZmxlbiAtIHRhc2stPnRhc2tsZW4sCiAgICAgICAgICAgICAgICAg ICAgIDEsCiAgICAgICAgICAgICAgICAgICAgIE5VTEwsCkBAIC0xMDIyLDYgKzEwMjMsNyBAQAog ICAgIHsKICAgICAgICAgZ2xvYnVzX2xfZm9ya19sb2dfc3RhdGVfY2hhbmdlKAogICAgICAgICAg ICAgICAgIHRhc2stPnBpZHNbaV0sCisgICAgICAgICAgICAgICAgdGFzaywKICAgICAgICAgICAg ICAgICBHTE9CVVNfR1JBTV9QUk9UT0NPTF9KT0JfU1RBVEVfQUNUSVZFLAogICAgICAgICAgICAg ICAgIDApOwogICAgIH0KQEAgLTEwMjksNyArMTAzMSw3IEBACiAKICAgICBmb3IgKGkgPSAwOyBp IDwgdGFzay0+Y291bnQ7IGkrKykKICAgICB7Ci0gICAgICAgIHByaW50ZigiJWx1IiwgKHVuc2ln bmVkIGxvbmcpIHRhc2stPnBpZHNbaV0pOworICAgICAgICBwcmludGYoIiVzOiVsdSIsIHRhc2st PmpvYmlkX3ByZWZpeCwgKHVuc2lnbmVkIGxvbmcpIHRhc2stPnBpZHNbaV0pOwogCiAgICAgICAg IGlmIChpICsgMSA8IHRhc2stPmNvdW50KQogICAgICAgICB7CkBAIC0xMDU3LDYgKzEwNTksNyBA QAogdm9pZAogZ2xvYnVzX2xfZm9ya19sb2dfc3RhdGVfY2hhbmdlKAogICAgIHBpZF90ICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIHBpZCwKKyAgICBnbG9idXNfZm9ya190YXNrX3QgICAg ICAgICAgICAgICAgICB0YXNrLAogICAgIGdsb2J1c19ncmFtX3Byb3RvY29sX2pvYl9zdGF0ZV90 ICAgIGpvYl9zdGF0ZSwKICAgICBpbnQgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBl eGl0X2NvZGUpCiB7CkBAIC0xMDY0LDcgKzEwNjcsNyBAQAogICAgIHN0cnVjdCBmbG9jayAgICAg ICAgICAgICAgICAgICAgICAgIGxvY2s7CiAgICAgaW50ICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgcmM7CiAgICAgY2hhciAqICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYnVm OwotICAgIHNpemVfdCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGJ1ZmxlbjsKKyAgICBp bnQgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBidWZsZW47CiAgICAgdGltZV90ICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgbm93ID0gdGltZShOVUxMKTsKIAogICAgIGxvZ2Zk ID0gb3BlbihnbG9idXNfbF9mb3JrX2xvZ2ZpbGVfcGF0aCwgT19XUk9OTFl8T19BUFBFTkQpOwpA QCAtMTExMywxOSArMTExNiwxNCBAQAogICAgICAgICByZXR1cm47IC8qIEJBRCEgKi8KICAgICB9 CiAKLSAgICBidWZsZW4gPSBnbG9idXNfbGliY19wcmludGZfbGVuZ3RoKAotICAgICAgICAgICAg IjAwMTslbHU7JWx1OyVkOyVkXG4iLAotICAgICAgICAgICAgKHVuc2lnbmVkIGxvbmcpIG5vdywK LSAgICAgICAgICAgICh1bnNpZ25lZCBsb25nKSBwaWQsCi0gICAgICAgICAgICAoaW50KSBqb2Jf c3RhdGUsCi0gICAgICAgICAgICAoaW50KSBleGl0X2NvZGUpOwotICAgIGJ1ZiA9IGdsb2J1c19s aWJjX21hbGxvYyhidWZsZW4rMSk7Ci0gICAgc3ByaW50ZihidWYsCi0gICAgICAgICAgICAiMDAx OyVsdTslbHU7JWQ7JWRcbiIsCisgICAgYnVmID0gZ2xvYnVzX2NvbW1vbl9jcmVhdGVfc3RyaW5n KAorICAgICAgICAgICAgIjAwMTslbHU7JXM6JWx1OyVkOyVkXG4lbiIsCiAgICAgICAgICAgICAo dW5zaWduZWQgbG9uZykgbm93LAorICAgICAgICAgICAgdGFzay0+am9iaWRfcHJlZml4LAogICAg ICAgICAgICAgKHVuc2lnbmVkIGxvbmcpIHBpZCwKICAgICAgICAgICAgIChpbnQpIGpvYl9zdGF0 ZSwKLSAgICAgICAgICAgIChpbnQpIGV4aXRfY29kZSk7CisgICAgICAgICAgICAoaW50KSBleGl0 X2NvZGUsCisgICAgICAgICAgICAmYnVmbGVuKTsKICAgICB3cml0ZShsb2dmZCwgYnVmLCBidWZs ZW4pOwogICAgIGNsb3NlKGxvZ2ZkKTsgLyogd2lsbCByZWxlYXNlIHRoZSBsb2NrICovCiB9CkBA IC0xMjEyLDEyICsxMjEwLDI5IEBACiBnbG9idXNfbF9mb3JrX3Rhc2tfaW5pdCgKICAgICBnbG9i dXNfZm9ya190YXNrX3QgKiAgICAgICAgICAgICAgICB0YXNrKQogeworICAgIGdsb2J1c191dWlk X3QgICAgICAgICAgICAgICAgICAgICAgIHV1aWQ7CisgICAgaW50ICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgcmM7CisKKyAgICByYyA9IGdsb2J1c191dWlkX2NyZWF0ZSgmdXVpZCk7 CisgICAgaWYgKHJjICE9IEdMT0JVU19TVUNDRVNTKQorICAgIHsKKyAgICAgICAgcmV0dXJuIEdM T0JVU19HUkFNX1BST1RPQ09MX0VSUk9SX01BTExPQ19GQUlMRUQ7CisgICAgfQogICAgICgqdGFz aykgPSBnbG9idXNfbGliY19jYWxsb2MoMSwgc2l6ZW9mKGdsb2J1c19sX2ZvcmtfdGFza190KSk7 CiAKICAgICBpZiAoKCp0YXNrKSA9PSBOVUxMKQogICAgIHsKICAgICAgICAgcmV0dXJuIEdMT0JV U19HUkFNX1BST1RPQ09MX0VSUk9SX01BTExPQ19GQUlMRUQ7CiAgICAgfQorCisgICAgKCp0YXNr KS0+am9iaWRfcHJlZml4ID0gZ2xvYnVzX2NvbW1vbl9jcmVhdGVfc3RyaW5nKCIlcyIsIHV1aWQu dGV4dCk7CisKKyAgICBpZiAoKCp0YXNrKS0+am9iaWRfcHJlZml4ID09IE5VTEwpCisgICAgewor ICAgICAgICBmcmVlKCp0YXNrKTsKKyAgICAgICAgKnRhc2sgPSBOVUxMOworICAgICAgICByZXR1 cm4gR0xPQlVTX0dSQU1fUFJPVE9DT0xfRVJST1JfTUFMTE9DX0ZBSUxFRDsKKyAgICB9CiAgICAg KCp0YXNrKS0+dGFza2J1ZmxlbiA9IEdMT0JVU19GT1JLX1RBU0tfQlVGX1NJWkU7CiAKICAgICAo KnRhc2spLT50YXNrID0gZ2xvYnVzX2xpYmNfbWFsbG9jKCgqdGFzayktPnRhc2tidWZsZW4pOwpA QCAtMTI0Nyw3ICsxMjYyLDYgQEAKICAgICAgICAgZ2xvYnVzX2xpYmNfZnJlZSh0YXNrLT50YXNr KTsKICAgICB9CiAKLQogICAgIGlmICh0YXNrLT5lbnZpcm9ubWVudCkKICAgICB7CiAgICAgICAg IGdsb2J1c19saWJjX2ZyZWUodGFzay0+ZW52aXJvbm1lbnQpOwpAQCAtMTI1Nyw2ICsxMjcxLDEw IEBACiAgICAgewogICAgICAgICBnbG9idXNfbGliY19mcmVlKHRhc2stPmFyZ3VtZW50cyk7CiAg ICAgfQorICAgIGlmICh0YXNrLT5qb2JpZF9wcmVmaXgpCisgICAgeworICAgICAgICBmcmVlKHRh c2stPmpvYmlkX3ByZWZpeCk7CisgICAgfQogCiAgICAgZ2xvYnVzX2xpYmNfZnJlZSh0YXNrKTsK IH0KSW5kZXg6IHBrZ2RhdGEvcGtnX2RhdGFfc3JjLmdwdC5pbgo9PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09ClJDUyBmaWxl OiAvaG9tZS9nbG9iZGV2L0NWUy9nbG9idXMtcGFja2FnZXMvd3MtZ3JhbS9qb2JfbWFuYWdlbWVu dC9mb3JrL2Mvc291cmNlL3BrZ2RhdGEvcGtnX2RhdGFfc3JjLmdwdC5pbix2CnJldHJpZXZpbmcg cmV2aXNpb24gMS4zCmRpZmYgLXUgLXIxLjMgcGtnX2RhdGFfc3JjLmdwdC5pbgotLS0gcGtnZGF0 YS9wa2dfZGF0YV9zcmMuZ3B0LmluCTMxIE1hciAyMDA1IDIxOjU3OjU5IC0wMDAwCTEuMworKysg cGtnZGF0YS9wa2dfZGF0YV9zcmMuZ3B0LmluCTIwIE1hciAyMDA2IDE1OjAzOjUxIC0wMDAwCkBA IC0zLDcgKzMsNyBAQAogCiA8Z3B0X3BhY2thZ2VfbWV0YWRhdGEgRm9ybWF0X1ZlcnNpb249IjAu MDIiIE5hbWU9Imdsb2J1c19mb3JrX3N0YXJ0ZXIiID4KIAotICA8QWdpbmdfVmVyc2lvbiBBZ2U9 IjAiIE1ham9yPSIwIiBNaW5vcj0iMSIgLz4KKyAgPEFnaW5nX1ZlcnNpb24gQWdlPSIwIiBNYWpv cj0iMCIgTWlub3I9IjIiIC8+CiAgIDxEZXNjcmlwdGlvbiA+Rm9yayBTdGFydGVyPC9EZXNjcmlw dGlvbj4KICAgPEZ1bmN0aW9uYWxfR3JvdXAgPlJlc291cmNlTWFuYWdlbWVudDwvRnVuY3Rpb25h bF9Hcm91cD4KICAgPFZlcnNpb25fU3RhYmlsaXR5IFJlbGVhc2U9IkV4cGVyaW1lbnRhbCIgRGly dD0iQERJUlRfVElNRVNUQU1QQC1ARElSVF9CUkFOQ0hfSURAIiAvPgo=