We are almost done with the Ec (Escherichia coli mono) experiment. As many of you have probably noticed we were switching to another organism (Vv, Vitis Vinifera). Complex organisms have a larger genome (our Ec dataset contains ~4000 genes, the Vv one ~28000). The application basically finds one-to-one interactions between all the genes of the dataset, so the output file is bigger on more complex organisms.

This is obviously a problem. A ~30 minutes run will produce a ~6 Mb output file (gzipped). Many people around the world simply don't have enough internet bandwidth to handle this. This is also a problem for us (more hard disk storage used).

The only way to solve this is to rethink the way the output file is built, maybe with some kind of filtering. But this means rewriting the application output code, i.e. it will take time to do this.

In the meanwhile we stopped the Vv experiment and are switching back to the dream5 experiment (see the science section of the forum) which is also based on Ec (smaller output file)

Thanks all for your understanding.

[edit] You may notice some workunits shortage in the meanwhile

What about something similar to what rosetta@home does? Where the end user can pick how long the WU's run and that would make the file size bigger/smaller depending on one's choices

BTW I think running out of work is worse than having to big of an output file.

Maybe you could store data in binary format? Or use more effective compressing algorithm like bz2 or even xz?
____________

Forgive the novice question, but what is the difference between the two Ec experiments? If we've run out of work on e-coli, I'm not sure how you can say there's more e-coli work to do which is not redundant. Are we analyzing it from a different angle? Are the results used differently?

I just don't want to feel like my limited CPU cycles are being kept busy so I'll stick around while we sort out the upload size. But if there's useful work which can be done, then send more of those Ec units my way while the Vv units are sorted!

Thanks!

Escherichia coli str. K-12 is the model organism for Gram negative bacteria. More information can be found on the corresponding NICB genome project webpage and EcoliWiki.

The Escherichia coli compendium in colombos contains expression values for 4321 genes, measured for 4077 condition contrasts. This corresponds to a total of 269 experiments and 5510 samples measured on 73 different platforms. Of these, 254 experiments were retrieved from GEO and 15 experiments were retrieved from ArrayExpress. For additional gene annotations, colombos mainly relies on data from the databases or resources indicated in the table below.

Maybe you could store data in binary format? Or use more effective compressing algorithm like bz2 or even xz?

Maybe you could store data in binary format? Or use more effective compressing algorithm like bz2 or even xz?

There is not a big difference between a binary format and the gzipped text we use. I just picked a random Vv output file, plain text, it is 1139829 lines, each line needs at least 4+1 bytes (indexes instead of gene names), a total of 5699145, which is almost the same as the gzipped version. On the other way gzip is internally supported by boinc. A dramatic change of the output format would also mean a lot of changes in the post-processing. ... But we are thinking about a solution like removing from the output file all the interactions that are present only once.

BTW I think running out of work is worse than having to big of an output file.

BTW I think running out of work is worse than having to big of an output file.

You have to think about the scale of things though.

Let's go with a user with 100 cores to offer(since I have around that many and that's how I did the math since it was easy),and each WU only takes 30 minutes and makes that 6MB output file.

That's 1.2GB to upload every hour. ~29GB every day. ~864GB per month.

For someone with a lot more compute power that could really scale out. It looks like you have ~500 cores,and while I'm sure that's spread out across projects if you only ran this project and those Vv WUs you would end up using terabytes of upload per month....and I'm pretty sure your ISP would be giving you a call at that point.


Not to mention what it would be like to the computer at the other end of that. The server status shows ~13K tasks underway,so imagining that as the core count of all end users,scale that out. That's terabytes of transfer PER DAY.

The worst scenario (boinc-related) is a fast computation with big output file (Vv), the ideal one would be slow and small (Ec-mono, ecm), Ec dream5 is something in between (fast and small).

But there's another point to take into account: science. Scientists would like to study a specific organism, following their own interests.

Now, Ec-mono is almost ended, Ec-dream5 will be distributed shortly (but it wont last long), Vv is really important for us and should be done. In the future we are thinking about Hs (Homo sapiens, us), we don't know yet about computational speed but we know that we have a lot of genes (slightly less than a plant...).

So, we definitely have to solve this 'big output file' issue.....