valtercProject administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 623
Credit: 34,677,535
RAC: 1
The new TCGA workunits (with a modified dataset) that contain the TCGAz string should behave correctly. Please let them run. I started with just a few batches.
Reply
Quote
Thanks for the warning - some of these jobs seem to need considerably more CPU-time. 17 h for 32% in one case. Anyhow, if there's a chance they come to an end I'll let them run.
Reply
Quote
Thanks for the warning - some of these jobs seem to need considerably more CPU-time. 17 h for 32% in one case. Anyhow, if there's a chance they come to an end I'll let them run.
if you can, let them run. If you need to turn off your pc at the end of the day, abort them. If you are in the middle of long checkpoint, and you turn off your pc, you'll come back to the previously checkpoint, losing your time.
One wu stay at 52% for 9h. Total run time 33h
Reply
Quote
Thanks for the warning - some of these jobs seem to need considerably more CPU-time. 17 h for 32% in one case. Anyhow, if there's a chance they come to an end I'll let them run.
if you can, let them run. If you need to turn off your pc at the end of the day, abort them. If you are in the middle of long checkpoint, and you turn off your pc, you'll come back to the previously checkpoint, losing your time.
One wu stay at 52% for 9h. Total run time 33h
I refer to old TCGA wus
Reply
Quote
Thanks for the warning - some of these jobs seem to need considerably more CPU-time. 17 h for 32% in one case. Anyhow, if there's a chance they come to an end I'll let them run.
if you can, let them run. If you need to turn off your pc at the end of the day, abort them. If you are in the middle of long checkpoint, and you turn off your pc, you'll come back to the previously checkpoint, losing your time.
One wu stay at 52% for 9h. Total run time 33h
No problem to let them run - looking good. One job finished after 1 days 3 hours 52 min 36 sec, completed and validated.
There's one job now at 5% after one day, 19 days to go. Let's see. :-)
Reply
Quote
valtercProject administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 623
Credit: 34,677,535
RAC: 1
OK, just to summarize: the TCGA workunits are the problematic ones, really long and with long periods of time (hours) without any checkpoint. Nevertheless we would be very happy if you let them run until completion.
The new TCGAz should behave in the usual way, i.e. you shouldn't worry if you got some.
Reply
Quote
OK, just to summarize: the TCGA workunits are the problematic ones, really long and with long periods of time (hours) without any checkpoint. Nevertheless we would be very happy if you let them run until completion.
The new TCGAz should behave in the usual way, i.e. you shouldn't worry if you got some.
No problem, my CPUs are working 24/7. I have no mercy.
Reply
Quote
Here's one job with no progress but growing time:
gene@home PC-IM 1.11 (avx)
148368_Hs_TCGA-KLF6_wu-22_1543433692665
03.12.2018 11:36:21
87.729 GFLOPs
time running
2d 14:56:04
time to complete
49d 21:49:11
recent progress
5,000%
Since the job seems to be frozen at 5% but rising times to complete, I'll better ask what to do? No problem to let it run for 100 days, if there's hope for a result.
Reply
Quote
valtercProject administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 623
Credit: 34,677,535
RAC: 1
Well, I got this https://gene.disi.unitn.it/test/result.php?resultid=37659353 that spent almost 24 hours without any (visible) progress and eventually got validated. This one https://gene.disi.unitn.it/test/result.php?resultid=37623246 ran for about three days and still needs to be validated. The longest one, by now, is this one https://gene.disi.unitn.it/test/workunit.php?wuid=17810415 that got more than 2300 credits.
However, there is no 'theoretical' computational limit in the algorithm. A task may run forever.
It would be useful to let the tasks run, even for many days, by taking care not to stop BOINC in between (because of the lack of checkpointing in the 'critical' period.
I'm constantly monitoring the situation, adding credits for 'invalid' and 'too late to validate' workunits.
Reply
Quote
...I'm constantly monitoring the situation, adding credits for 'invalid' and 'too late to validate' workunits.
Thanks, that's what I love to read. If someone cares for the result, there's some reason in it. I'll let it run and we'll see what happens. Curiosity is a strong motive.
Reply
Quote
Here's the longest one I've seen yet:
http://gene.disi.unitn.it/test/workunit.php?wuid=17810256
I suspect the 4 "Timed out - no response" instances are still crunching away. My example is on a pretty slow machine. It's at 137 hours and 12%. It looks like these aren't checkpointing often. This WU is showing 117 hours since the last checkpoint (in BoincTasks).
Reply
Quote
Here's the longest one I've seen yet:
http://gene.disi.unitn.it/test/workunit.php?wuid=17810256
I suspect the 4 "Timed out - no response" instances are still crunching away. My example is on a pretty slow machine. It's at 137 hours and 12%. It looks like these aren't checkpointing often. This WU is showing 117 hours since the last checkpoint (in BoincTasks).
Now at 183 hours, and 156 hours since the last checkpoint (which is troubling). Still at 12%. I assume that if the computer reboots or the WU gets interrupted for any reason it will restart. It appears that 6 computers are currently running this WU and no one has finished it. Is it viable?
Reply
Quote
from the few i have looked at, the sse2 and avx clearly have problems but fma are succeeding.
Reply
Quote
Here's the longest one I've seen yet:
http://gene.disi.unitn.it/test/workunit.php?wuid=17810256
I suspect the 4 "Timed out - no response" instances are still crunching away. My example is on a pretty slow machine. It's at 137 hours and 12%. It looks like these aren't checkpointing often. This WU is showing 117 hours since the last checkpoint (in BoincTasks).
Now at 183 hours, and 156 hours since the last checkpoint (which is troubling). Still at 12%. I assume that if the computer reboots or the WU gets interrupted for any reason it will restart. It appears that 6 computers are currently running this WU and no one has finished it. Is it viable?
Now at 282 hours, 238 hours since the last checkpoint. Still at 12%. Now there seems to be 7 computers actively running this WU (counting the 5 that are listed as "Timed out - no response", one of which is mine). Any possibility of adding some checkpoints to these TCGA WUs? A power glitch or any other interruption would wipe out nearly 10 days of work.
Reply
Quote
Now at 183 hours, and 156 hours since the last checkpoint (which is troubling). Still at 12%.
I have seen a few of those. Before they get that far, I abort them. I think you are unnecessarily conscientious; they are duds.
Reply
Quote
valtercProject administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 623
Credit: 34,677,535
RAC: 1
There are about 30 TCGA workunits still around and for sure those are the very long ones. Theoretically those could run forever. The reason is that for a certain, very rare, type of input the algorithm's completely is exponential. We usually manage to adjust the input dataset in order to avoid this but in the current case there were an issue that we were able to fix only after the workunits were distributed. The results are of scientific value, of course, but without a checkpoint inside the critical section of the algorithm this is not the kind of computation to do inside the BOINC framework.
Anyway, I will wait for them for another couple of days then I will abort them server side, I will figure out a way to give credits even in this case.
Reply
Quote
There are about 30 TCGA workunits still around and for sure those are the very long ones. Theoretically those could run forever. The reason is that for a certain, very rare, type of input the algorithm's completely is exponential. We usually manage to adjust the input dataset in order to avoid this but in the current case there were an issue that we were able to fix only after the workunits were distributed. The results are of scientific value, of course, but without a checkpoint inside the critical section of the algorithm this is not the kind of computation to do inside the BOINC framework.
Anyway, I will wait for them for another couple of days then I will abort them server side, I will figure out a way to give credits even in this case.
Thanks for the update and explanation. I have four of these WUs currently running on 3 machines. Thought I'd post a small BoinkTasks screenshot:
From clues gleaned by looking at the WU history I expect a couple of these to finish within the next 2 days. A third one will probably be longer and the currently 306 hour one is totally mysterious as several faster machines are running it and it's never been completed.
Reply
Quote
Three of the above WUs finished as expected. The 4th is STILL running at 12% completion after 377 hours (and 312 hours since the last checkpoint). It looks like 8 machines are still running this one, some longer than I have:
http://gene.disi.unitn.it/test/workunit.php?wuid=17810256
I'm not going to be able to run more of these non checkpointing WUs for a while as I'm doing major rearrangement of machines (long overdue).
Reply
Quote
valtercProject administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 623
Credit: 34,677,535
RAC: 1
Do not worry, if you need to abort them do that. There are about 18 workunits of the TCGA batch still around (of a total of 1240), some of them are probably the most problematic ones.
Reply
Quote
The 4th is STILL running at 12% completion after 377 hours (and 312 hours since the last checkpoint).
All projects have stuck work units; some more than others. If the Progress % is not making any progress after a few hours (24 hours is more than enough time), then it is stuck in a loop.
Reply
Quote