Workunits length and size
log in

Advanced search

Message boards : Development : Workunits length and size

Author Message
Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 320
Credit: 16,279,861
RAC: 3,910
Italy
Message 22 - Posted: 25 Nov 2013, 11:18:11 UTC

Just some thoughts about size and length of a workunit:

Length (expected cpu time): Ideally a couple of hours on a I7 Ivy Bridge or later (assuming one thread continuously dedicated to a single workunit).

Size (input): In order to minimize bandwidth, get the input data just once (if there is a unique input data file valid for a large set of workunits).

Size (output): Much better have a single large (but not too much...) output file then a lot of small files. Compress it before transfer it back (use the gzip flag) if it is proven to be useful.

Performance: Try to limit I/O (avoid too much stress to the local hd).

Buffer your data and write the largest blocks possible (better not after any iteration if this is too fast). The stdio routines buffer by default, but writing a byte at a time is still a lot slower than writing large blocks (avoid fprintf, prefer fwrite or WriteFile, better not using sprintf like tools for formatting). Also be aware about I/O differences between operating systems. As a rule of thumb do as few writes as possible, accordingly to the checkpoints.


Post to thread

Message boards : Development : Workunits length and size


Main page · Your account · Message boards


Copyright © 2017 CNR-TN & UniTN