Disk space
log in

Advanced search

Message boards : Number crunching : Disk space

Author Message
Profile [VENETO] boboviz
Send message
Joined: 12 Dec 13
Posts: 130
Credit: 908,789
RAC: 1,432
Italy
Message 203 - Posted: 27 Dec 2013, 14:42:43 UTC
Last modified: 27 Dec 2013, 14:43:16 UTC

I've fished and uploaded all wus on my 6 core pc (also due to the beginning of Rita Levi Montalcini Challenge on Rosetta).
Now i have 4,3 Gb of used space by your project (it's a lot of space). Is it normal? Can i clean up the project folder?

marco giglio
Volunteer moderator
Project developer
Send message
Joined: 12 Nov 13
Posts: 20
Credit: 1,708
RAC: 0
Italy
Message 205 - Posted: 27 Dec 2013, 15:15:24 UTC - in response to Message 203.
Last modified: 27 Dec 2013, 15:27:03 UTC

I think you can clean the project folder, there should be no problem.
Given the number of WUs you have elaborated, I think that data is normal, but we'll investigate this matter, in order to reduce the average used disk space.
Thank you for your cooperation

Profile danicampa90
Volunteer moderator
Project developer
Project tester
Project scientist
Send message
Joined: 12 Nov 13
Posts: 28
Credit: 552
RAC: 0
Italy
Message 206 - Posted: 27 Dec 2013, 15:19:16 UTC

Once the upload of the results are completed you can delete everything,
Even if you don't do it, Boinc should automatically take care of deleting things if it's running low on the space reserved to him. Our project take up a lot of space because we work with big files. I agree that we need to try to reduce the file size, maybe by keeping them compressed.

marco giglio
Volunteer moderator
Project developer
Send message
Joined: 12 Nov 13
Posts: 20
Credit: 1,708
RAC: 0
Italy
Message 207 - Posted: 27 Dec 2013, 15:25:44 UTC - in response to Message 206.
Last modified: 27 Dec 2013, 15:27:53 UTC

Also, some of the files are marked as sticky because in theory they are needed by more than a WUs. Problem is that boinc has not an intelligent scheduler with respect to sticky files.
The best behavior would be to send new WUs minimizing the number of files to be downloaded, hence WUs which you can compute using the files you've already downloaded; this mechanism is not provided so far. This implies that you probably are storing many sticky files which you just used once or twice but you're client is keeping them because they are sticky.
Right now we are addressing other, more critical issues, but I think we should think whether mark those files as sticky is really necessary.
It depends on some other factors such as the length we decide for the WUs and the variability in the length

Profile [VENETO] boboviz
Send message
Joined: 12 Dec 13
Posts: 130
Credit: 908,789
RAC: 1,432
Italy
Message 208 - Posted: 27 Dec 2013, 16:13:13 UTC - in response to Message 205.

I think that data is normal


I don't think so. Here some projects i run on my pc and "residual data"
Ralph@home: 250 mb
WCG: 31 mb
Poem: 7 mb

My fear is that, with next download, there are more than 10gb....

Profile [VENETO] boboviz
Send message
Joined: 12 Dec 13
Posts: 130
Credit: 908,789
RAC: 1,432
Italy
Message 209 - Posted: 27 Dec 2013, 16:18:15 UTC - in response to Message 207.

Right now we are addressing other, more critical issues, but I think we should think whether mark those files as sticky is really necessary.


Yeap, this is not a critical problem now, with little work.
But in the future, may be.

Ah, i forget:
Congratulations to everyone for the great work you are doing!!

marco giglio
Volunteer moderator
Project developer
Send message
Joined: 12 Nov 13
Posts: 20
Credit: 1,708
RAC: 0
Italy
Message 210 - Posted: 27 Dec 2013, 16:55:47 UTC
Last modified: 27 Dec 2013, 17:19:37 UTC

In the future we'll address the problem for sure!
However, here's some data.
The disk space is mostly needed to store those sticky files I mentioned before. I took a look in my BOINC folder and I have many of them, many probably obsolete.
I took 3 of them, whose sizes were 241, 70 and 68 MB and using different compression algorithm I achieved compression factors around 38%, which means that when we'll use the compression you'll spare almost 2/3 of the space.
From 4.3GB of used space we'll go to 1.7GB
It is still a lot, but it is something.
Moreover, we'll need to talk to the professor and the preprocessing group in order to understand the size of these files and to determine the WUs length.
We may choose to have a unique WU for each file, which is the worst choice in terms of bandwidth consumption, but is probably the best in terms of disk usage, since the file would not be sticky anymore and would be removed at the end of the WU's computation.

EDIT: I run some more test and it seems the best choice would be to use bzip2.
The compression factor is almost as high as in lzma but the compression time is much lower

Profile paolomorettin
Project developer
Project tester
Project scientist
Send message
Joined: 20 Nov 13
Posts: 19
Credit: 13,027
RAC: 0
Message 211 - Posted: 27 Dec 2013, 20:56:47 UTC - in response to Message 209.

Ah, i forget:
Congratulations to everyone for the great work you are doing!!


Thank you for your valuable advices! ;)
____________
Paolo - Application team dev (SSC11)

"If you were plowing a field, which would you rather use: two strong oxen or 1024 chickens?" Seymour Cray

Profile [VENETO] boboviz
Send message
Joined: 12 Dec 13
Posts: 130
Credit: 908,789
RAC: 1,432
Italy
Message 303 - Posted: 20 Jan 2014, 7:30:51 UTC - in response to Message 205.

I think you can clean the project folder, there should be no problem.


Waiting for new work, i stopped wu request and cleaned the project folder (6gb).
But, after some minutes, boinc client begins to download files (.csv and Others) and NO wus. I killed the download, but it restarted. It has happened several times...

Profile FrancescoAsnicar [SSC11]
Project developer
Project tester
Project scientist
Avatar
Send message
Joined: 14 Nov 13
Posts: 50
Credit: 6,646,731
RAC: 5,480
Italy
Message 304 - Posted: 20 Jan 2014, 13:54:39 UTC - in response to Message 303.

I reset the project in order to test the new application version, as soon as there will be new wus. It didn't happen to me that the client downloaded files without wus.
Did you try to reset the project instead of clean the directory? Could you try to remove the project and re-add.


Post to thread

Message boards : Number crunching : Disk space


Main page · Your account · Message boards


Copyright © 2017 CNR-TN & UniTN