validation issues in 0.02
log in

Advanced search

Message boards : Development : validation issues in 0.02

1 · 2 · Next
Author Message
Profile paolomorettin
Project developer
Project tester
Project scientist
Send message
Joined: 20 Nov 13
Posts: 19
Credit: 13,027
RAC: 0
Message 163 - Posted: 24 Dec 2013, 13:09:49 UTC

Hi everyone, we are trying to get rid of the validation issues we're having with version 0.02. Is it possible to see the source code of the validator?
____________
Paolo - Application team dev (SSC11)

"If you were plowing a field, which would you rather use: two strong oxen or 1024 chickens?" Seymour Cray

Profile danicampa90
Volunteer moderator
Project developer
Project tester
Project scientist
Send message
Joined: 12 Nov 13
Posts: 28
Credit: 552
RAC: 0
Italy
Message 164 - Posted: 24 Dec 2013, 13:35:49 UTC

I noticed that too, and I was going to post a new thread right now.
I don't have the source code of the validator, nor I know how it works.
I examined two results that were different (of WU 23056), and I noticed
- files are gzipped (this is expected)
- files, when unzipped, present differences in newlines.
- when correcting the newlines there were only a few lines of differences, so the workunits should have been accepted. (WU 23056)

There is also another issue, for which I will create a separate topic....

chau
Send message
Joined: 12 Nov 13
Posts: 15
Credit: 229
RAC: 0
Italy
Message 167 - Posted: 24 Dec 2013, 14:14:04 UTC - in response to Message 164.

OK. I will check it. Thanks

chau
Send message
Joined: 12 Nov 13
Posts: 15
Credit: 229
RAC: 0
Italy
Message 170 - Posted: 24 Dec 2013, 15:06:43 UTC - in response to Message 164.

Hi,

The current validator is checking files by md5. That's why when you receive two files with different check_sum, they will mark as different. I'm improving it with removing \r (Carriage-Return) before comparing it. Thanks

chau
Send message
Joined: 12 Nov 13
Posts: 15
Credit: 229
RAC: 0
Italy
Message 171 - Posted: 24 Dec 2013, 15:24:16 UTC - in response to Message 170.

By the way, Paolo, when you output results to files, can you please take a look at the format of output? I mean, can you set the mode to "binary" instead of "text"? In windows, the new line is '\r\n' and in unix, the new line is '\n'. Thanks

marco giglio
Volunteer moderator
Project developer
Send message
Joined: 12 Nov 13
Posts: 20
Credit: 1,708
RAC: 0
Italy
Message 172 - Posted: 24 Dec 2013, 15:35:11 UTC - in response to Message 170.

Hi,

The current validator is checking files by md5. That's why when you receive two files with different check_sum, they will mark as different. I'm improving it with removing \r (Carriage-Return) before comparing it. Thanks

Wait a sec: the md5sum validator is the one provided by boinc or is it something you have written?
Please mind that it seems there are still some slight difference among the WU computed by linux and the ones computed by windows, hence we should use something as the method discussed in one of the meeting with the professors. An md5um, instead, would keep failing

Profile paolomorettin
Project developer
Project tester
Project scientist
Send message
Joined: 20 Nov 13
Posts: 19
Credit: 13,027
RAC: 0
Message 175 - Posted: 24 Dec 2013, 17:44:03 UTC - in response to Message 171.

Hi,

The current validator is checking files by md5. That's why when you receive two files with different check_sum, they will mark as different. I'm improving it with removing \r (Carriage-Return) before comparing it. Thanks

Wait a sec: the md5sum validator is the one provided by boinc or is it something you have written?
Please mind that it seems there are still some slight difference among the WU computed by linux and the ones computed by windows, hence we should use something as the method discussed in one of the meeting with the professors. An md5um, instead, would keep failing


I agree, comparing output files by their md5s is even less informative than comparing them bitwise.

By the way, Paolo, when you output results to files, can you please take a look at the format of output? I mean, can you set the mode to "binary" instead of "text"? In windows, the new line is '\r\n' and in unix, the new line is '\n'. Thanks


I'm not sure I understood. What should we change in the syntax?
____________
Paolo - Application team dev (SSC11)

"If you were plowing a field, which would you rather use: two strong oxen or 1024 chickens?" Seymour Cray

chau
Send message
Joined: 12 Nov 13
Posts: 15
Credit: 229
RAC: 0
Italy
Message 176 - Posted: 24 Dec 2013, 17:47:08 UTC

Hi,

After checking files from WU 23056 and removed the character CR(\r), I still see the difference between two files( result_id 36654 and 36655) even though they are running on the CPU Intel core-i( result_id 36786, after removing the CR \r, they have a big different size with the rest files). Can you please check it again? Thanks

chau
Send message
Joined: 12 Nov 13
Posts: 15
Credit: 229
RAC: 0
Italy
Message 177 - Posted: 24 Dec 2013, 17:48:40 UTC - in response to Message 175.

I mean, when you write to a file, do you set the mode to write files as binary mode or text mode?

Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 320
Credit: 16,288,722
RAC: 4,233
Italy
Message 178 - Posted: 24 Dec 2013, 18:13:47 UTC - in response to Message 177.
Last modified: 24 Dec 2013, 18:18:57 UTC

it would probably wise to stop the validator until we fix this problem. There is nothing that piss off users more then not receiving credit for the work done... If we don't fix this soon it would also be better to stop the work generation.

I also got a bunch of workunits that were waiting for a wingman return and were marked as invalid because the workunit itself was 'cancelled'.

Profile danicampa90
Volunteer moderator
Project developer
Project tester
Project scientist
Send message
Joined: 12 Nov 13
Posts: 28
Credit: 552
RAC: 0
Italy
Message 179 - Posted: 25 Dec 2013, 10:49:19 UTC

@Chau: The difference was expected, and we spent meetings discussing this. If I recall correctly, in the meetings we have said that the differences (if small) would need to be "merged down", and the workunits accepted nevertheless. If the difference is "high" the workunits need to be rejected (probably someone was cheating, or an error occurred)
The small differences are OK, and totally expected (due to errors in floating point computations).
I have a big question: Why did you compare the results by md5? I don't remember we spoke about md5 in the meetings. If we were to accept workunits only when equal we would have used the standard bitwise validator. Why did you implemented again the standard bitwise validator using md5?

@Paolo: He was asking if you could in the windows executables end the lines only with \n and not by \r\n. The conversion \n -> \r\n is done automatically by windows unless you open the file in "binary mode". The way to open a file in binary mode changes depending on the API you use.
I think that this should simplify the work the validator has to do.


Merry Christmas to all.

Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 320
Credit: 16,288,722
RAC: 4,233
Italy
Message 180 - Posted: 25 Dec 2013, 12:15:57 UTC - in response to Message 179.
Last modified: 25 Dec 2013, 12:25:03 UTC

we could also try to tell boinc to send workunits couples only to computers with similar architectures and see what happens...

also, did you try to use the *same* erf() function regardless the architecture?

chau
Send message
Joined: 12 Nov 13
Posts: 15
Credit: 229
RAC: 0
Italy
Message 181 - Posted: 25 Dec 2013, 12:42:32 UTC - in response to Message 179.

Hi Daniel,

The results before I test them are the same, that's why I used md5. I will improve it but the problem is how much difference between WUs would probably be marked as unaccepted or accepted. How much difference you will consider as small difference and high difference? What kind of difference will be considered as different or similar. What is the case if we marked them in a wrong way?. I sent an email to Paolo to ask more detail about that. Thanks and Merry Christmas.

Profile NadirS [SSC11]
Project developer
Project tester
Project scientist
Send message
Joined: 20 Nov 13
Posts: 13
Credit: 133,573
RAC: 0
Italy
Message 182 - Posted: 25 Dec 2013, 13:16:37 UTC - in response to Message 181.
Last modified: 25 Dec 2013, 13:29:02 UTC

If we have some edge inserted or deleted, it could be a problem of the erf function. Please check if the problem is on newlines or on edges. In the second case,could someone take these invalid wu from the server (with input and output) and give us the files so that we can run on our machine and test?

Profile paolomorettin
Project developer
Project tester
Project scientist
Send message
Joined: 20 Nov 13
Posts: 19
Credit: 13,027
RAC: 0
Message 183 - Posted: 25 Dec 2013, 14:19:49 UTC - in response to Message 182.

Chau already sent to me the output files. I will investigate the differences as soon as possible.
____________
Paolo - Application team dev (SSC11)

"If you were plowing a field, which would you rather use: two strong oxen or 1024 chickens?" Seymour Cray

Trung
Send message
Joined: 28 Nov 13
Posts: 7
Credit: 15
RAC: 0
Italy
Message 184 - Posted: 25 Dec 2013, 15:22:45 UTC

For detail, take Wu 23056 as an example. We have three output results.


    1. Expansion_At2_work-1387833585.xml_pn19926_0_0 runs on GenuineIntel Intel(R) Core(TM) i7-2630QM CPU @ 2.00GHz [Family 6 Model 42 Stepping 7] with x86_64-pc-linux-gnu platform.

    2. Expansion_At2_work-1387833585.xml_pn19926_1_0 runs on GenuineIntel Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz [Family 6 Model 42 Stepping 7] with windows_x86_64 platform.

    3. Expansion_At2_work-1387833585.xml_pn19926_2_0 runs on AuthenticAMD AMD FX(tm)-6300 Six-Core Processor [Family 21 Model 2 Stepping 0] with windows_x86_64 platform



This first and the second run on same Intel core i7, different platform (Windows & Linux) and two results have small differences. The first one have 3 new edges: 252411_at,255604_at, 253467_at,262159_at, 253467_at,262159_at. While the second one have 1 new edge: 252002_at,253467_at

If we consider two results are all correct. Please give us a threshold of difference to judge. For example: If total of new edges < 1% total of edges, we will give both result credit.

The third is totally different. The size is much bigger (38 Mb vs 27 Mb), and have 661 lines in format "#... list_gene" compare to 500 lines of the first and the second.

Thanks and Merry Christmas.


chau
Send message
Joined: 12 Nov 13
Posts: 15
Credit: 229
RAC: 0
Italy
Message 185 - Posted: 25 Dec 2013, 17:57:40 UTC - in response to Message 183.
Last modified: 25 Dec 2013, 18:23:43 UTC

Hi Paolo, Daniel

Thanks for this. Moreover, when you try to compress the file, can you distinguish name of original file and name of compressed file as different names.

For example: name of compressed file should be "name_c" and name of original file should be "name". Actually, this is more convenient for me to process files after unzipping it(avoid coincidence in name of files). Moreover, after unzipping files, I will delete the compressed file. So it is easier to distinguish between two kinds of files.

I think I solved the problem. So you don't need to change names of files.

Profile paolomorettin
Project developer
Project tester
Project scientist
Send message
Joined: 20 Nov 13
Posts: 19
Credit: 13,027
RAC: 0
Message 186 - Posted: 25 Dec 2013, 18:05:36 UTC - in response to Message 184.

For detail, take Wu 23056 as an example. We have three output results.


    1. Expansion_At2_work-1387833585.xml_pn19926_0_0 runs on GenuineIntel Intel(R) Core(TM) i7-2630QM CPU @ 2.00GHz [Family 6 Model 42 Stepping 7] with x86_64-pc-linux-gnu platform.

    2. Expansion_At2_work-1387833585.xml_pn19926_1_0 runs on GenuineIntel Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz [Family 6 Model 42 Stepping 7] with windows_x86_64 platform.

    3. Expansion_At2_work-1387833585.xml_pn19926_2_0 runs on AuthenticAMD AMD FX(tm)-6300 Six-Core Processor [Family 21 Model 2 Stepping 0] with windows_x86_64 platform



This first and the second run on same Intel core i7, different platform (Windows & Linux) and two results have small differences. The first one have 3 new edges: 252411_at,255604_at, 253467_at,262159_at, 253467_at,262159_at. While the second one have 1 new edge: 252002_at,253467_at

If we consider two results are all correct. Please give us a threshold of difference to judge. For example: If total of new edges < 1% total of edges, we will give both result credit.

The third is totally different. The size is much bigger (38 Mb vs 27 Mb), and have 661 lines in format "#... list_gene" compare to 500 lines of the first and the second.

Thanks and Merry Christmas.




IMHO the best idea is to develop a parametric validator and ask to Professor Blanzieri and to the post-processing team which threshold you should put in.

The third result is insanely wrong and surely caused by an application bug.

____________
Paolo - Application team dev (SSC11)

"If you were plowing a field, which would you rather use: two strong oxen or 1024 chickens?" Seymour Cray

marco giglio
Volunteer moderator
Project developer
Send message
Joined: 12 Nov 13
Posts: 20
Credit: 1,708
RAC: 0
Italy
Message 188 - Posted: 26 Dec 2013, 18:05:17 UTC - in response to Message 186.

As Paolo stated, you should (and actually you should already had) contact the Professors and post-processing group in order to decide:
- up to what difference two file can be considered as both correct
- whether we should keep all the lines contained in both files or whether we should keep only the intersection of the two
Please, let us know your progresses.

Profile danicampa90
Volunteer moderator
Project developer
Project tester
Project scientist
Send message
Joined: 12 Nov 13
Posts: 28
Credit: 552
RAC: 0
Italy
Message 191 - Posted: 26 Dec 2013, 21:08:11 UTC
Last modified: 26 Dec 2013, 21:09:08 UTC

I will try to stop only the validator of the project, so that maybe the workunits won't be sent multiple times on validation errors.
I think that when we have a good validator we can start it up again, and hopefully it will assign credits also for old workunits.
Is it a good idea, or is it better to stop the entire project?
I will wait your answer before doing anything, valterc.

1 · 2 · Next
Post to thread

Message boards : Development : validation issues in 0.02


Main page · Your account · Message boards


Copyright © 2017 CNR-TN & UniTN