Application group request (compact stderr)
log in

Advanced search

Message boards : Development : Application group request (compact stderr)

Author Message
Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 320
Credit: 16,276,743
RAC: 4,520
Italy
Message 239 - Posted: 3 Jan 2014, 18:12:29 UTC
Last modified: 3 Jan 2014, 18:17:10 UTC

Please try to make the stderr output more compact. From

#132 Tue Dec 24 02:05:01 2013
Graph created File: complete_gene_net read Standard deviations computed Correlations computed l: 0 1 2 3 4 5 6 7 8 9 # of arcs: 2223
Writing checkpoint's file
to something like
132|13-12-24 02:05:01|G F S|C(9:2223)
CHK
I mean, all the needed info squeezed in few bytes.

Also, do you need to read the complete_gene_net at every iteration? Shouldn't be more efficient to just read it at start or returning from checkpoint and keep it (or a copy of it) in memory?

Profile paolomorettin
Project developer
Project tester
Project scientist
Send message
Joined: 20 Nov 13
Posts: 19
Credit: 13,027
RAC: 0
Message 241 - Posted: 3 Jan 2014, 21:32:00 UTC - in response to Message 239.

Please try to make the stderr output more compact. From
#132 Tue Dec 24 02:05:01 2013
Graph created File: complete_gene_net read Standard deviations computed Correlations computed l: 0 1 2 3 4 5 6 7 8 9 # of arcs: 2223
Writing checkpoint's file
to something like
132|13-12-24 02:05:01|G F S|C(9:2223)
CHK
I mean, all the needed info squeezed in few bytes.


I did not work on that, but I think that the new version we already sent has a more compact stderr.

Also, do you need to read the complete_gene_net at every iteration? Shouldn't be more efficient to just read it at start or returning from checkpoint and keep it (or a copy of it) in memory?


When we developed the first versions of the application, we had no bounds on the complete_gene size, so the best strategy was to read on-the-fly the lines needed.
I think that we should discuss this with professor Blanzieri, reading it once should be faster, but I suspect that working with different species could lead to far bigger input files, compared to the Arabidopsis. Splitting those files may be a solution, though..
____________
Paolo - Application team dev (SSC11)

"If you were plowing a field, which would you rather use: two strong oxen or 1024 chickens?" Seymour Cray

Profile FrancescoAsnicar [SSC11]
Project developer
Project tester
Project scientist
Avatar
Send message
Joined: 14 Nov 13
Posts: 50
Credit: 6,645,797
RAC: 6,199
Italy
Message 243 - Posted: 4 Jan 2014, 8:42:36 UTC - in response to Message 241.
Last modified: 4 Jan 2014, 8:44:47 UTC

Please try to make the stderr output more compact. From
#132 Tue Dec 24 02:05:01 2013
Graph created File: complete_gene_net read Standard deviations computed Correlations computed l: 0 1 2 3 4 5 6 7 8 9 # of arcs: 2223
Writing checkpoint's file
to something like
132|13-12-24 02:05:01|G F S|C(9:2223)
CHK
I mean, all the needed info squeezed in few bytes.


I did not work on that, but I think that the new version we already sent has a more compact stderr.


Yes, the new application version have a more compact stderr with timestamp!

Also, do you need to read the complete_gene_net at every iteration? Shouldn't be more efficient to just read it at start or returning from checkpoint and keep it (or a copy of it) in memory?


When we developed the first versions of the application, we had no bounds on the complete_gene size, so the best strategy was to read on-the-fly the lines needed.
I think that we should discuss this with professor Blanzieri, reading it once should be faster, but I suspect that working with different species could lead to far bigger input files, compared to the Arabidopsis. Splitting those files may be a solution, though..


We cannot read and store all in ram. For arabidopsis, that is a small genome, we calculated that the ram needed will be around 8 GB!

Profile FrancescoAsnicar [SSC11]
Project developer
Project tester
Project scientist
Avatar
Send message
Joined: 14 Nov 13
Posts: 50
Credit: 6,645,797
RAC: 6,199
Italy
Message 252 - Posted: 5 Jan 2014, 14:01:15 UTC - in response to Message 243.

I've computed some wus with the new application... There are still a thousand lines in the stderr with 500 tiles, but I think that the lines printed are necessary for future debug!


Post to thread

Message boards : Development : Application group request (compact stderr)


Main page · Your account · Message boards


Copyright © 2017 CNR-TN & UniTN