World Gene Expansion Challenge 2021
log in

Advanced search

Message boards : Number crunching : World Gene Expansion Challenge 2021

1 · 2 · Next
Author Message
Profile Buro87 [Lombardia]
Send message
Joined: 23 Nov 16
Posts: 100
Credit: 4,000,541
RAC: 0
Italy
Message 2216 - Posted: 1 Mar 2021, 21:04:13 UTC
Last modified: 1 Mar 2021, 21:47:00 UTC

Boinc.Italy is proud to announce second edition of World Gene Expansion Challenge

Start: 8th March 2021 12:00 UTC
End : 15th March 2021 12:00 UTC


Boincstats: https://www.boincstats.com/stats/challenge/team/chat/1089

Here our Ranking http://boincrankings.eu5.org/wgec21/classifica.php?id=1000
Special thanks to our teammate Luigi.R who made them

Let's crunch togheter :)

Luigi R.
Send message
Joined: 1 Mar 15
Posts: 4
Credit: 2,060,645
RAC: 0
Italy
Message 2218 - Posted: 2 Mar 2021, 9:24:28 UTC

If you can't join this challenge on BOINCstats, let me know here (or by private message) and I will add your team on BOINC Rankings.


P.S. you have to be founder to make a request.

Aurum
Send message
Joined: 18 Jul 18
Posts: 97
Credit: 291,241,518
RAC: 266
United States
Message 2220 - Posted: 2 Mar 2021, 12:25:15 UTC - in response to Message 2218.
Last modified: 2 Mar 2021, 12:26:44 UTC

...

Profile Buro87 [Lombardia]
Send message
Joined: 23 Nov 16
Posts: 100
Credit: 4,000,541
RAC: 0
Italy
Message 2221 - Posted: 2 Mar 2021, 15:31:21 UTC

Just for reference, during last year challenge we've crunched a total of 16.2 Millions creditis

Let's crush it :)

Profile Buro87 [Lombardia]
Send message
Joined: 23 Nov 16
Posts: 100
Credit: 4,000,541
RAC: 0
Italy
Message 2230 - Posted: 6 Mar 2021, 10:30:54 UTC

20 teams joined the battle
Are you readi? Just over two days at the start

Frank [NT]
Send message
Joined: 14 Sep 17
Posts: 2
Credit: 6,063,960
RAC: 58
Germany
Message 2233 - Posted: 6 Mar 2021, 14:17:43 UTC - in response to Message 2216.


Here our Ranking http://boincrankings.eu5.org/wgec21/classifica.php?id=1000


Hello BOINC.Italy, thanks for invitation !
2 days ago i already saw a ranking there, but now i see only "Team rankings
Waiting for stats generation...".

Was this only a testrun, and now it waits until the race starts ?

I've read here about 20 teams, on BoincStats are only 18 teams, "In Science We Trust" and "Wolrdwide Techservices" are not listed there.
Which one is the "official" ranking for the challenge ?

Luigi R.
Send message
Joined: 1 Mar 15
Posts: 4
Credit: 2,060,645
RAC: 0
Italy
Message 2234 - Posted: 6 Mar 2021, 15:07:17 UTC - in response to Message 2233.

Hello BOINC.Italy, thanks for invitation !

Hi Frank, welcome on board!

2 days ago i already saw a ranking there, but now i see only "Team rankings
Waiting for stats generation...".

Was this only a testrun, and now it waits until the race starts ?

Yeah, it was. Technically it's ready to start.
There will be another one or two just to be 100% sure that teams are correctly set up.
Don't worry about it, challenge will officially start on the date of BOINCstats (2021-03-08 12:00:00 UTC).


I've read here about 20 teams, on BoincStats are only 18 teams, "In Science We Trust" and "Wolrdwide Techservices" are not listed there.
Which one is the "official" ranking for the challenge ?

In Science We Trust and Worldwide TechServices asked me to join on BOINC Rankings.
Hopefully we will try to merge them and there will be no problems if team gaps are large enough. Anyway one year ago team points were pretty identical apart from roundoff errors (BS vs BR).
Another issue is that BOINC Rankings does not allow late entrant because it's independent from BOINCstats, so some teams could join the challenge on BOINCstats and not to be present on BOINC Rankings.

Frank [NT]
Send message
Joined: 14 Sep 17
Posts: 2
Credit: 6,063,960
RAC: 58
Germany
Message 2236 - Posted: 7 Mar 2021, 11:55:51 UTC - in response to Message 2234.

Hi Luigi,
thank you for the answer.
How often will your ranking update, every 15 minutes like on BoincStats ?

Btw : 24 hours until start... :-)

Luigi R.
Send message
Joined: 1 Mar 15
Posts: 4
Credit: 2,060,645
RAC: 0
Italy
Message 2237 - Posted: 7 Mar 2021, 14:19:01 UTC - in response to Message 2236.

How often will your ranking update, every 15 minutes like on BoincStats ?

I usually do updates hourly.

Magiceye04
Send message
Joined: 14 Dec 16
Posts: 10
Credit: 3,461,526
RAC: 0
Germany
Message 2238 - Posted: 7 Mar 2021, 21:26:22 UTC

Will there be some WUs ready to be send tomorrow?
At the moment they are like corona vaccine - very rare.

Aurum
Send message
Joined: 18 Jul 18
Posts: 97
Credit: 291,241,518
RAC: 266
United States
Message 2239 - Posted: 7 Mar 2021, 22:28:27 UTC - in response to Message 2238.

Will there be some WUs ready to be send tomorrow?

Just Say No to Bunkering.
They should use the 2 WUs per CPU thread mode and we wouldn't run out.

xii5ku
Send message
Joined: 20 Jul 17
Posts: 14
Credit: 35,946,522
RAC: 0
Message 2240 - Posted: 7 Mar 2021, 22:38:18 UTC - in response to Message 2238.
Last modified: 7 Mar 2021, 22:40:59 UTC

Magiceye04 wrote:
Will there be some WUs ready to be send tomorrow?

Maybe. But only if there are enough people who give up and stop requesting work.

The pace of TN-Grid's work generator is limited.


Aurum wrote:
Just Say No to Bunkering.
They should use the 2 WUs per CPU thread mode and we wouldn't run out.

The limit is 8 tasks in progress per active CPU AFAIK, counting up to 64 CPUs per host.
This is already a low limit.

Bunkering is not the reason for work scarcity.

Bryn Mawr
Send message
Joined: 23 Jun 20
Posts: 43
Credit: 14,256,442
RAC: 7
United Kingdom
Message 2241 - Posted: 7 Mar 2021, 23:10:59 UTC - in response to Message 2240.
Last modified: 7 Mar 2021, 23:12:09 UTC

Magiceye04 wrote:
Will there be some WUs ready to be send tomorrow?

Maybe. But only if there are enough people who give up and stop requesting work.

The pace of TN-Grid's work generator is limited.


Aurum wrote:
Just Say No to Bunkering.
They should use the 2 WUs per CPU thread mode and we wouldn't run out.

The limit is 8 tasks in progress per active CPU AFAIK, counting up to 64 CPUs per host.
This is already a low limit.

Bunkering is not the reason for work scarcity.


I keep a very small cache and all I can say is that since the announcement of this challenge I’ve been getting very few WUs to process and even fewer of the WUs I have processed are moving out of pending - their counterparts must still be in someone’s cache waiting to run.

Profile Buro87 [Lombardia]
Send message
Joined: 23 Nov 16
Posts: 100
Credit: 4,000,541
RAC: 0
Italy
Message 2242 - Posted: 7 Mar 2021, 23:26:21 UTC
Last modified: 7 Mar 2021, 23:44:18 UTC

As xii5ku said, the problem here is the peak of high request of wu, and work generator can't keep the pace.
If i remember correctly work generator can create 294wu (x2 copies =588) every 10 miutes.

An example was the peak on Thursday morning: "Running wus" skyrocketed from 28k to 50k in few hours. "Available wus" dropped to around 5k.
When this peak wave passed away, then itslowly recovered to 15k

we are definitely pushing the project to its limit

Pending wus will drop at the at the start of the challenge

Jim1348
Send message
Joined: 29 Dec 16
Posts: 87
Credit: 21,013,002
RAC: 0
United States
Message 2243 - Posted: 8 Mar 2021, 3:28:13 UTC - in response to Message 2242.

My buffer is down to 9 work units, a bit low for an i7-8700.

But they have plenty from SiDock in Ljubljana, less than 300 km east of Trento.
https://www.sidock.si/sidock/server_status.php
Buy local.

Aurum
Send message
Joined: 18 Jul 18
Posts: 97
Credit: 291,241,518
RAC: 266
United States
Message 2244 - Posted: 8 Mar 2021, 10:38:21 UTC - in response to Message 2240.

Bunkering is not the reason for work scarcity.

If you say so, but then why have my Validation Pending WUs grown to 6600???
Flood gates open at noon UTC today...

Profile Buro87 [Lombardia]
Send message
Joined: 23 Nov 16
Posts: 100
Credit: 4,000,541
RAC: 0
Italy
Message 2245 - Posted: 8 Mar 2021, 10:50:19 UTC - in response to Message 2243.
Last modified: 8 Mar 2021, 11:00:13 UTC

Hi
I just talked to the adminin to improve the job availability situation.

For those with jobs that would go beyond the deadline, not already started,it is preferable to abort them BEFORE deadline.
In this way the wu will only be renamed an resend straight away.

For wu expired, wu generator, need to create a new wu


Another thing: he set for the first time, a "grace period" of 8hours after deadline, within which, THEORETICALLY, wus will be considered valid.
NO GUARANTEES ON THIS. IF YOU WANT, TRY AT YOUR HOWN RISK


Personally, i will try with few wu to give a feedback

Thanks

xii5ku
Send message
Joined: 20 Jul 17
Posts: 14
Credit: 35,946,522
RAC: 0
Message 2248 - Posted: 8 Mar 2021, 20:29:03 UTC
Last modified: 8 Mar 2021, 20:36:37 UTC

xii5ku wrote:
Bunkering is not the reason for work scarcity.

Aurum wrote:
If you say so, but then why have my Validation Pending WUs grown to 6600???

I didn't say that there was no bunkering going on.
I said that the reason for work scarcity is (in other words) that work requests are more frequent than WU generation.

BTW, tasks in progress normally at TN-Grid:
25...35 k during this month, 25...55 k during the past several months,
right before the start of the current challenge: < 80 k,
right now: 53 k

These less than 80 k in progress do not look like a massive bunkering effort to me.

Bunkering would contribute to work scarcity if the bunker builders would issue frequent work requests over a prolonged time, and it would cause work scarcity if the number of tasks in progress would inflate the database so much that the work generator stopped producing.


Buro87 wrote:
For those with jobs that would go beyond the deadline, not already started,it is preferable to abort them BEFORE deadline.
In this way the wu will only be renamed an resend straight away.

For wu expired, wu generator, need to create a new wu

The last part is not correct.
TN-Grid's workunits are configured with
. . . . minimum quorum = 2
. . . . initial replication = 2
. . . . max # of error tasks = 8
. . . . max # of total tasks = 16
. . . . max # of success tasks = 8
which among else means that a workunit fails only if there were 8 tasks ending in an error (which includes timeout, among other error types) or if there were 16 tasks created but still no two correct result reports coming in before the tasks' respective reporting deadline. (This 16 tasks limit can never be reached in practice, and the 8 error tasks limit is extremely unlikely to happen.)
As long as there failed less than 8 tasks, a failed (e.g. timed out) task will merely cause the server to make another task out of the same already existing workunit.

Buro87 wrote:
Another thing: he set for the first time, a "grace period" of 8hours after deadline, within which, THEORETICALLY, wus will be considered valid.

At TN-Grid, there is no difference between the reporting deadline listed at the website (which is what the server should take as deadline) and the reporting deadline listed locally on the hosts (which the clients were told to obey as deadline). However:

When a host "Y" does not report a result at the respective task's reporting deadline, the server replicates another task from the same workunit and assigns this task to another host "Z" at the next opportunity.
If host "Y" then reports a valid result before host "Z" does, host "Y" gets credit for this result despite the passed deadline, and the server notifies host "Z" that his task is no longer needed. This notification will be delivered at the next point in time when host "Z" happens to emit a scheduler request.
If host "Z" receives this notification before it started the now superfluous task, it marks it as 'error/ cancelled by server' and reports it. But if host "Z" already started the task before this notification, it completes the task as if nothing happened, reports the result to the server, and will get credit if the result is valid, even though the corresponding task was not needed anymore for the server to finish the workunit.

(The above assumes that there was already another host "X" reporting a valid result in time, such that either X&Y or X&Z would satisfy the minimum quorum of 2.)

To summarize, there are regular situations in which a result is validated and gets credit even if the reporting deadline was exceeded.

(This is my understanding; somebody correct me if I got something wrong.)

Aurum
Send message
Joined: 18 Jul 18
Posts: 97
Credit: 291,241,518
RAC: 266
United States
Message 2250 - Posted: 8 Mar 2021, 22:07:40 UTC - in response to Message 2248.

xii5ku wrote:
Bunkering is not the reason for work scarcity.

Aurum wrote:
If you say so, but then why have my Validation Pending WUs grown to 6600???

I didn't say that there was no bunkering going on.
I said that the reason for work scarcity is that work requests are more frequent than WU generation.
Yes we all know that TN-GRID could put a faster server to good use.

BTW, tasks in progress normally at TN-Grid:
25...35 k during this month, 25...55 k during the past several months,
right before the start of the current challenge: < 80 k,
right now: 53 k
These less than 80 k in progress do not look like a massive bunkering effort to me.

Bunkering would contribute to work scarcity if the bunker builders would issue frequent work requests over a prolonged time, and it would cause work scarcity if the number of tasks in progress would inflate the database so much that the work generator stopped producing.
I don't bunker but here's what I think happens. First, increase your queue to request as many WUs as you can get. Then run them but don't return them until after the race starts.
With a slow work generator this makes it likely that all WUs will be depleted.
Since this project has quorum 2 this works in my favor. Since I return all my WUs immediately my bunkering wingmen hold the 2nd Q WU until after the race starts and so push my recent credit into race points.
The Science Status page showed the rate dropped from 55/day to 44/day yesterday and now it's back up to 51/day. Since it's a 10-day average the dip was actually bigger. I'd enjoy seeing a long term graph of the work rate.
Before the race I was running TN-GRID as my only CPU project with Resource = 0. It runs really good that way or as a backup project. That's why I'm confidant that switching the server to supply only 2 WUs per CPU thread would work great and mean that one could only bunker about 8 hours of work instead of several days worth.
The good news is that the work is getting done.

Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 616
Credit: 34,514,943
RAC: 395
Italy
Message 2251 - Posted: 8 Mar 2021, 22:32:45 UTC - in response to Message 2250.
Last modified: 8 Mar 2021, 23:34:20 UTC

That's why I'm confidant that switching the server to supply only 2 WUs per CPU thread would work great and mean that one could only bunker about 8 hours of work instead of several days worth.

Theoretically. If you really want to maximize your cache of workunits you could also lie about the number of cores (say that you just have 8 but you can easily tell the server that you have 128) or (although not so easy to implement) you may also start a lot of different clients on the same pc. So, at the end, if you really want, you may bypass any server side limitations.
On the other hand, forcing a small user's cache will create problems to users with intermittent network activity.

1 · 2 · Next
Post to thread

Message boards : Number crunching : World Gene Expansion Challenge 2021


Main page · Your account · Message boards


Copyright © 2024 CNR-TN & UniTN