Hi,
it is more than 2 hours that a Workunit seems to be "stuck" at 6% completion rate with no checkpoints.
Is it normal or it is gone to an endless loop and to be aborted?
Workunit: 148365_Hs_TCGA-AR_wu-25_1543429200632_2
Application: gene@home PC-IM v1.11 (avx)
OS: Windows 7
Nevermind! after 2.5 hours decided to go ahead and went to 6.5% and correctly wrote the checkpoint
Reply
Quote
valtercProject administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 623
Credit: 34,677,535
RAC: 1
The input file is made by some 'computational chunks', usually any chunks run for more or less the same time, so it was easy to decide how many chunks to put inside a workunit and to forecast the overall computational time.
The TCGA workunits behave in a (somewhat unexpected) different way, i.e. some chunks may take *really* a lot of time more than the others in order to complete.
We don't know exactly why, it could be because the intrinsic 'nature' of the TCGA dataset or there could be some kind of 'artifact' inside it.
Anyway, we sent around a limited number of such workunits, hoping to gather enough feedback to help us solve the problem. (we don't want to distribute workunits that may take days to complete).
So, please, if you have such workunits, let them finish.
Reply
Quote
Thanks for the explanation and quick feedback Valterc.
I will for sure let it finish.
Reply
Quote
Just had one that was stuck on 16+ hours and still at 17% on a relatively fast and previously reliable machine. Aborted it before I came over here to check...
Have another one that's getting close to 75% and 7.6 hours on a very fast machine. That one's still running.
Reply
Quote
>> Have another one that's getting close to 75% and 7.6 hours on a very fast machine. That one's still running.
The one above finished in 9:10.
Now I have one on the same very fast machine that's run for 9:06 hours and is still at 0.5%.
Reply
Quote
I also have to abort my task unfortunately.
My crunching time is usually working day hours (9/10 more or less).
If the application is not able to write a checkpoint at least every 4/5 hours I have to drop it.
Reply
Quote
valtercProject administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 623
Credit: 34,677,535
RAC: 1
Any work-unit is built up from a certain number of small computational pieces, in the TCGA experiments the numbers are 1000 or 200. The checkpoint is written at he end of every chunk. Obviously, if one chunk is one of the 'abnormal' ones, running for hours, there will be no checkpoint for a long period of time, unfortunately.
Anyway, results are coming back, although slower than usual and even the long workunits seems to be validated (and credit given) correctly.
Reply
Quote
My hyper-threaded i7-6700K Windows 10 system has 1 validated TCGA task (148365_Hs_TCGA-AR_wu-124_1543429375131_2, 561.1 credits for 11:59:57 runtime), 1 pending validation (148368_Hs_TCGA-KLF6_wu-102_1543433840285_2, 9:10:54 runtime), 4 running and 5 in its work queue.
The running tasks are:
- 148365_Hs_TCGA-AR_wu-120_1543429368264_3 - 40.5%, running for 13 hours, last checkpoint 9 hours ago
- 148366_Hs_TCGA-BRCA2_wu-75_1543431300189_2 - 36%, running for 6½ hours, last checkpoint 4½ hours ago
- 148364_Hs_TCGA-PTEN_wu-18_1543428819030_2 - 34.5%, running for 4 hours, last checkpoint 1½ hours ago
- 148366_Hs_TCGA-BRCA2_wu-88_1543431326968_2 - 13.5%, running for 4 hours, last checkpoint 3½ hours ago
____________
"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer
Reply
Quote
valtercProject administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 623
Credit: 34,677,535
RAC: 1
The TCGA workunits are problematic. If you happen to have one and find it like 'frozen', i.e. no progress after a long time, feel free to abort it.
Tomorrow, Monday, I will abort them all, server side.
Reply
Quote
3 of those tasks have now completed:
- 148365_Hs_TCGA-AR_wu-120_1543429368264_3 - 24:36:40 runtime, 14 checkpoints, longest interval 19:15:47 for #8, validation pending
- 148366_Hs_TCGA-BRCA2_wu-75_1543431300189_2 - 27:22:26, 16 checkpoints, longest interval 19:03:23 for #6, validation pending
- 148364_Hs_TCGA-PTEN_wu-18_1543428819030_2 - 9:27:25 runtime, 27 checkpoints, longest interval 1:53:40 for #11, valid (436.59 credits)
The other task mentioned in my previous message was pre-empted for 9 hours and is now running again:
- 148366_Hs_TCGA-BRCA2_wu-88_1543431326968_2 - 19%, running for 21½ hours, 4 checkpoints made, #2 took 19:13:40
One of the queued tasks has also completed today:
- 148365_Hs_TCGA-AR_wu-34_1543429216393_2- 12:37:28 runtime, 19 checkpoints, longest interval 4:08:26 for #10, validation inconclusive (my task was run on an Intel i7-6700K, Windows 10 system using v1.11 (sse2); _1 task ran for 28:04:24 on an AMD FX-8320E, Linux 4.4.76-1-default system using v1.10 (fma))
____________
"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer
Reply
Quote
The other task mentioned in my previous message was pre-empted for 9 hours and is now running again:
- 148366_Hs_TCGA-BRCA2_wu-88_1543431326968_2 - 19%, running for 21½ hours, 4 checkpoints made, #2 took 19:13:40
Now completed with 30:52:42 runtime and 16 checkpoints made, validation pending.
____________
"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer
Reply
Quote
valtercProject administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 623
Credit: 34,677,535
RAC: 1
The output of the 'problematic' TCGA workunits were, in some cases, different if calculated on Windows or Linux. There are some validation errors because of this so I wrote down some code in order to give credits even for the TCGA invalids. See this workunit as an example: http://gene.disi.unitn.it/test/workunit.php?wuid=17810593
Reply
Quote
The output of the 'problematic' TCGA workunits were, in some cases, different if calculated on Windows or Linux. There are some validation errors because of this so I wrote down some code in order to give credits even for the TCGA invalids. See this workunit as an example: http://gene.disi.unitn.it/test/workunit.php?wuid=17810593
Very nice and much appreciated.
Reply
Quote
Crunched on my 1950X and waiting for validation:
http://gene.disi.unitn.it/test/workunit.php?wuid=17809940 - CPU time: 251,481.41 seconds :(
http://gene.disi.unitn.it/test/workunit.php?wuid=17810657 - CPU time: 149,729.80 seconds, five people resigned...
Validated:
http://gene.disi.unitn.it/test/workunit.php?wuid=17810577 - CPU time: 396,309.20 :O It's too much even for me.
Reply
Quote
valtercProject administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 623
Credit: 34,677,535
RAC: 1
The last one is probably one of the longest I ever seen (4,704.77 credits...). This one https://gene.disi.unitn.it/test/workunit.php?wuid=17810524 is probably the record until now.
Anyway, there are only a few (TCGA) workunits still around, probably the longest...
Reply
Quote
GPUGRID has two queues - short one and long one. Can you maybe do the same here?
Reply
Quote
valtercProject administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 623
Credit: 34,677,535
RAC: 1
GPUGRID has two queues - short one and long one. Can you maybe do the same here?
The TCGA batch behavior was unexpected. Workunits like those, without checkpoints for a very long time and somewhat unpredictable running time are not for BOINC. We don't have any plan to distribute very long workunits in the future and, for sure, workunits like the TCGA ones.
Just wanted to point up that the TCGAz workunits behave correctly.
Reply
Quote
My i7 has just completed the _8 task from workunit 148368_Hs_TCGA-KLF6_wu-154_1543433935389 with 57:17:44 runtime, 41:07:29 of it being between the 15th and 16th checkpoints.
The over deadline _2 task was completed by a Linux ARMv7 system after running for 320:02:11, and pro rata the long checkpoint would have taken almost 10 days! Aaron deserves a special commendation for sticking with it.
As expected for TCGA tasks, the fact that it's been completed by a Windows and Linux system means that validation is inconclusive. 4 other tasks from the WU have timed out, 3 have been aborted and 2 are in progress.
____________
"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer
Reply
Quote