Clear Linux - All results Invalid or Inconclusive (fma, avx, sse)
log in

Advanced search

Message boards : Number crunching : Clear Linux - All results Invalid or Inconclusive (fma, avx, sse)

Author Message
TeKNiX
Send message
Joined: 2 Mar 18
Posts: 7
Credit: 6,108,262
RAC: 1
Philippines
Message 1640 - Posted: 29 Nov 2019, 19:15:58 UTC
Last modified: 29 Nov 2019, 19:44:22 UTC

Any one running Clear Linux?

http://gene.disi.unitn.it/test/show_host_detail.php?hostid=55060

Computer isn't overclock,
temps are good
ram is good, psu is good

Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 616
Credit: 34,514,943
RAC: 395
Italy
Message 1644 - Posted: 2 Dec 2019, 11:23:06 UTC - in response to Message 1640.
Last modified: 2 Dec 2019, 11:34:35 UTC

Hi TeKNiX, I just started to check this issue by looking at this workunit http://gene.disi.unitn.it/test/workunit.php?wuid=21463187. At this moment two files were uploaded to the server and they don't match. By looking at them I found this

33354 Dec 2 12:11 160481_Hs_T193199-ZCCHC7_wu-15_1575214475199_0_0 10195 Dec 2 12:11 160481_Hs_T193199-ZCCHC7_wu-15_1575214475199_0_0.gz 33354 Dec 2 12:11 160481_Hs_T193199-ZCCHC7_wu-15_1575214475199_1_0 9617 Dec 2 12:11 160481_Hs_T193199-ZCCHC7_wu-15_1575214475199_1_0.gz

The content (uncompressed) of the two files is the same so your computer did the computation correctly. The real problem is that the gzipped files are different (the gzipped file is the one transferred from your host to the server and also bitwise checked, skipping the gzip header, by the validator).

For some (yet unknown to me) reason your computer is gzipping the output files in a unusual way

TeKNiX
Send message
Joined: 2 Mar 18
Posts: 7
Credit: 6,108,262
RAC: 1
Philippines
Message 1645 - Posted: 2 Dec 2019, 16:11:59 UTC - in response to Message 1644.
Last modified: 2 Dec 2019, 16:18:07 UTC

Hello Valterc,

So, I chased down gzip. It seem ClearLinux is using ParallelGzip
If this is the default behavior of Clear Linux,
Is there a way for us to fix this?

======Clear Linux==========
$ gzip -V
pigz 2.4

$ which gzip
/usr/bin/gzip

$ ls -l /usr/bin/gzip
lrwxrwxrwx 1 root root 4 Oct 17 22:27 /usr/bin/gzip -> pigz

======End of Clear Linux=======


=====Ubuntu 18.04=======
$ gzip -V
gzip 1.6
Copyright (C) 2007, 2010, 2011 Free Software Foundation, Inc.
Copyright (C) 1993 Jean-loup Gailly.
This is free software. You may redistribute copies of it under the terms of
the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.
=====End of Ubuntu======

P.S
*I turned off Boinc on this machine, since clearly there is an Issue/bug that needs to be solve
*Reinstalling gzip still installs pigz -https://clearlinux.org/software/bundle/gzip

TeKNiX
Send message
Joined: 2 Mar 18
Posts: 7
Credit: 6,108,262
RAC: 1
Philippines
Message 1646 - Posted: 2 Dec 2019, 17:45:42 UTC - in response to Message 1645.

Ok, I installed an external gzip and replace the "clearlinux symlinked gzip"

$ gzip -V
gzip 1.9
Copyright (C) 2017 Free Software Foundation, Inc.
Copyright (C) 1993 Jean-loup Gailly.
This is free software. You may redistribute copies of it under the terms of
the GNU General Public License <https://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.


And for a little testing, all default. Its clearly is different

-rwxr-xr-x 1 xxxx 50680 Dec 2 12:37 gzip.gzip
-rwxr-xr-x 1 xxxx 51087 Dec 2 12:43 gzip.pigz

Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 616
Credit: 34,514,943
RAC: 395
Italy
Message 1647 - Posted: 2 Dec 2019, 18:17:36 UTC - in response to Message 1646.
Last modified: 2 Dec 2019, 18:17:51 UTC

by looking at the BOINC client source code I found this

int FILE_INFO::gzip() { char buf[BUFSIZE]; char inpath[MAXPATHLEN], outpath[MAXPATHLEN]; get_pathname(this, inpath, sizeof(inpath)); safe_strcpy(outpath, inpath); safe_strcat(outpath, ".gz"); FILE* in = boinc_fopen(inpath, "rb"); if (!in) return ERR_FOPEN; gzFile out = gzopen(outpath, "wb"); while (1) { int n = (int)fread(buf, 1, BUFSIZE, in); if (n <= 0) break; int m = gzwrite(out, buf, n); if (m != n) { fclose(in); gzclose(out); return ERR_WRITE; } } fclose(in); gzclose(out);


I may guess that the client is using a shared library for gzipping, like libz.so (should check the client with the ldd command)

TeKNiX
Send message
Joined: 2 Mar 18
Posts: 7
Credit: 6,108,262
RAC: 1
Philippines
Message 1648 - Posted: 2 Dec 2019, 20:04:22 UTC - in response to Message 1647.

Hello Valtrec,

Gzip for Centos and Ubuntu different results, Size wise.

====Centos 8======
50680 Dec 2 14:46 test.gzip
50619 Dec 2 14:46 test.pigz

$ file test.*
test.gzip: gzip compressed data, was "test", last modified: Mon Dec 2 19:46:29 2019, from Unix, original size 111912
test.pigz: gzip compressed data, was "test", last modified: Mon Dec 2 19:46:35 2019, from Unix, original size 111912

===End of Centos====

====Ubuntu====
50687 Dec 2 14:55 test_ubuntu.gzip
50619 Dec 2 14:54 test_ubuntu.pigz

test_ubuntu.gzip: gzip compressed data, ....
test_ubuntu.pigz: gzip compressed data, ....
====End of Ubuntu====


Added as extra

=====Clear Linux===

$ file gzip.*
gzip.gzip: gzip compressed data, was "gzip", last modified: Mon Dec 2 17:37:55 2019, from Unix, original size modulo 2^32 111912
gzip.pigz: gzip compressed data, was "gzip", from Unix, original size modulo 2^32 111912
=====End of Clear====

Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 616
Credit: 34,514,943
RAC: 395
Italy
Message 1649 - Posted: 3 Dec 2019, 10:01:39 UTC - in response to Message 1648.
Last modified: 3 Dec 2019, 16:01:21 UTC

gzip stores (by default) the file name inside its archive, the test_ubuntu.gzip file is exactly 7 bytes larger than test.gzip. Try with same file name length (and content)

TeKNiX
Send message
Joined: 2 Mar 18
Posts: 7
Credit: 6,108,262
RAC: 1
Philippines
Message 1650 - Posted: 3 Dec 2019, 18:56:02 UTC - in response to Message 1649.

I got it working, I have to replaced the libz.so file. (Hackish way)

My first valid results.

http://gene.disi.unitn.it/test/workunit.php?wuid=21474260


Wow, did we just discovered a bad bug? I'll see, if I can reproduce it again, using a different machine.

Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 616
Credit: 34,514,943
RAC: 395
Italy
Message 1651 - Posted: 4 Dec 2019, 10:10:17 UTC - in response to Message 1650.
Last modified: 4 Dec 2019, 10:10:46 UTC

I don't know if it is (technically) a bug, the fact is that the output of pigz (or the 'new' libz) is not, bitwise, the same as gzip, although it is 'compatible' (correctly uncompressed). The side effect here is that the validator checks the results by comparing two output files (just skipping the gzip header), thus failing in this case. I will post this issue to the boinc_projects mailing list, asking for BOINC developers comments.

TeKNiX
Send message
Joined: 2 Mar 18
Posts: 7
Credit: 6,108,262
RAC: 1
Philippines
Message 1652 - Posted: 7 Dec 2019, 16:12:54 UTC - in response to Message 1651.

Hello Valtrec,

So I was able to spin up a new Clear Linux on a different Machine

http://gene.disi.unitn.it/test/show_host_detail.php?hostid=55927

Same deal, Clear Linux default install has issues, I'll let it run for a day more. If is producing invalid results, I'll try to apply the workaround on libz.so again to see if this will fix the issue.

TeKNiX
Send message
Joined: 2 Mar 18
Posts: 7
Credit: 6,108,262
RAC: 1
Philippines
Message 1653 - Posted: 7 Dec 2019, 22:34:26 UTC - in response to Message 1652.
Last modified: 7 Dec 2019, 22:36:49 UTC

all errors , so I applied the workaround

Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 616
Credit: 34,514,943
RAC: 395
Italy
Message 1654 - Posted: 9 Dec 2019, 10:47:15 UTC - in response to Message 1653.

all errors , so I applied the workaround

Yep, I sent a mail to the boinc developers about this, no answers yet.


Post to thread

Message boards : Number crunching : Clear Linux - All results Invalid or Inconclusive (fma, avx, sse)


Main page · Your account · Message boards


Copyright © 2024 CNR-TN & UniTN