Things to do (or to check) for server and application before massive testing
log in

Advanced search

Message boards : Development : Things to do (or to check) for server and application before massive testing

Author Message
Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 320
Credit: 16,284,006
RAC: 4,205
Italy
Message 25 - Posted: 28 Nov 2013, 10:42:16 UTC

in random order (no implicit priority)

application:
- optimization (check if extended instruction sets, SSEx, AVX, etc. will help)
- provide more platforms (both x64 and x32)
- provide a more verbose stderr output
- do statistics for choosing the best compromise between running time, output size and server load
- test checkpoints and progress bar

server side:
- embed info into wu name (something like gene_datasource_npi_otherinfo_index)
- do redundancy (now it seems that the result is self-validated)
- find a backup host (ask Enrico)
- limit somehow the number of distributed wu (no too big client caches now) say max 2 per host per core (just an idea)

names:
- the 'test' in the url should be simply 'boinc'
- give meaningful names to specialized daemons, i.e. sample_work_generator should be gene_work_generator
- application name should be something like "Gene Network ..." (no more testing application)

others:
- completely remove the test application from the list (and any associated credit)

credits:
- everything will be zapped just before the 'go slightly public' phase... so don't crunch just for credits now....

I will add things or modify this post when there will be the need

Profile danicampa90
Volunteer moderator
Project developer
Project tester
Project scientist
Send message
Joined: 12 Nov 13
Posts: 28
Credit: 552
RAC: 0
Italy
Message 26 - Posted: 29 Nov 2013, 17:42:15 UTC
Last modified: 5 Dec 2013, 10:34:57 UTC

This post will be updated as we complete things, and serves as a list on what needs to be done by the server group.
I will update this post as we go. Please subscribe to this topic to receive updates!

A checklist specifically for the server group:
- {DONE} [Daniele] Provide more redundancy
- - With the new workunits that are generated the redundancy factor will be 2.
- {DONE} [Daniele] Delete old WU, restart work generator, generate new ones with replication factor of 2.
- {DONE} [Daniele] Improve the reliability of the task generator.
- {DONE} [Daniele] Make a script that takes the files, and registers it with the workunit.
- {DONE} [Daniele] Embed more information on the WU names.
- {DONE} [Daniele] Create a script that automatically run the script of the preprocessing group and register all the generation directives in the task generator.
- {DONE} [Daniele] Give a proportional time estimation to client for the workunit. Note: currently the flops estimation is incorrect, but at least is somehow proprtional to the amount of work in a workunit.
- [Trung] Rename the project
- - URL: gene.disi.unitn.it/boinc/
- - Name of the project: Gene
- [Trung] Rename the application, and delete the old ones
- [Chau][Trung] Test the validator with live data (the work generator will distribute data with replication factor of 2 in a few hours)
- [Chau] Test the differences between the results obtained on Intel and AMD
- {DONE} [Daniele] Rename the work generator executable
- [Chau] Rename the validator executable
- [Claudio] Check with the post-processing group what they want as input (do a new topic on that and send email)
- [Claudio][Chau] Check how boinc handles exporting data, and what needs to be modified in order to export data in their format (assimilator?)
- [??] Limit the numbers of assigned workunits.

The names near the task are who (I think) wants to do the task.

>> IF YOU WANT TO DO SOMETHING ELSE WRITE ME AN EMAIL. <<

Profile FrancescoAsnicar [SSC11]
Project developer
Project tester
Project scientist
Avatar
Send message
Joined: 14 Nov 13
Posts: 50
Credit: 6,652,949
RAC: 5,354
Italy
Message 27 - Posted: 30 Nov 2013, 17:03:19 UTC

Now the Linux 32 bit is working! :)
I tested it on Ubuntu 12.04 with kernel 3.2.0-56

Profile FrancescoAsnicar [SSC11]
Project developer
Project tester
Project scientist
Avatar
Send message
Joined: 14 Nov 13
Posts: 50
Credit: 6,652,949
RAC: 5,354
Italy
Message 28 - Posted: 2 Dec 2013, 16:47:25 UTC - in response to Message 25.

in random order (no implicit priority)

application:
- test checkpoints and progress bar
I will add things or modify this post when there will be the need

Both tested and working!

Profile danicampa90
Volunteer moderator
Project developer
Project tester
Project scientist
Send message
Joined: 12 Nov 13
Posts: 28
Credit: 552
RAC: 0
Italy
Message 30 - Posted: 5 Dec 2013, 10:58:22 UTC

These thing are done and tested to work:

Application:
- provide more platforms (both x64 and x32)
- - we have mac, linux x86 x64 and win x32 (runs also on win64)
- test checkpoints and progress bar
- - they work
- - time estimation is all messed up, but at least it is proportional to the work in the workunit. boinc will auto-adapt (hopefully) to the wrong estimation.

Server side:
- embed info into wu name (something like gene_datasource_npi_otherinfo_index)
- - ok: contains the name of an xml file containing all the info.
- do redundancy (now it seems that the result is self-validated)
- - Redundancy of 2 is enabled.
- NOTE: now I inserted in the queue the analysis of all the network that the preprocessing gave us.

Names:
- give meaningful names to specialized daemons, i.e. sample_work_generator should be gene_work_generator
- - Renamed to gene_work_generator. Also the validators, assimilators etc have names in the form gene_xx

Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 320
Credit: 16,284,006
RAC: 4,205
Italy
Message 31 - Posted: 5 Dec 2013, 11:42:54 UTC - in response to Message 30.
Last modified: 5 Dec 2013, 11:43:20 UTC

About the validator

Now that we have redundancy, remember that Boinc also provides "homogeneous redundancy" (sending workunits only to computers of the same platform—e.g.: Win XP SP2 only.) This could solve problems of slightly different results coming from different architectures.

Trung
Send message
Joined: 28 Nov 13
Posts: 7
Credit: 15
RAC: 0
Italy
Message 33 - Posted: 5 Dec 2013, 19:27:56 UTC

[About Rename the Project]

I find a way to change a project's name and project's URL while maintaining the forum content and user's credit. Users need to remove old project from boinc-client and add the new one. Tested on Boinc Virtual Machine.

For example, we want to change the name from 'test' to 'gene'. The URL becomes 'http://gene.disi.unitn.it/gene/'

Stop the project and take the website down

>> ./bin/stop >> su -c 'apache2ctl -k stop'


Create backup and change the name of project folder
>> cp -Rp test/ gene/ >> sudo chown www-data:boincadm gene/log_debian6/scheduler.log >> sudo chown www-data:boincadm gene/log-debian6/file_upload_handler.log


Create new database named 'gene' and copy data from 'test'. (Keep 'test' as a backup)
>> mysql -u root -p >> create database gene; >> grant all on `gene`.* to 'boincadm'@'localhost'; >> exit >> mysqldump -u boincadm --single-transaction test | mysql -u boincadm gene


Create backup and change /etc/apache2/httpd.conf (or virtual host)
>> su >> cp httpd.conf httpd.conf.old Replace all Alias and Directory for the project. Alias /test /home/boincadm/projects/test/html/user -> Alias /gene /home/boincadm/projects/gene/html/user ....


Change the config.xml
Replace all URL and directory in config.xml with the new one in <upload_dir>, <cgi_url>, <upload_url>, <download_url>, <log_dir>, <app_dir>, <download_dir>, <master_url> Change <db_name> from 'test' to 'gene' Change <long_name> (project name) from 'test' to 'gene'


If we have project skin http://boinc.berkeley.edu/trac/wiki/ProjectSkin. We also need change project.xml to correct the URL for downloaded logo or image.

Replace the scheduler URL in 'html/users/schedulers.txt'. Its prefix should match the URL in <cgi_url> in config.xml

Correct the path in 'bin/boinc_path_config.py' to make python script run.

Correct the path in crontab (crontab -e)

Find all missing link in folder gene and fix. For example:
stop -> /home/boincadm/projects/test/bin/start becomes stop -> /home/boincadm/projects/gene/bin/start


Correct the path to .htpasswd in html/ops/.htaccess

Find and replace all incorrect URLs in database
>> mysql -u boincadm -p >> use gene; >> UPDATE result SET xml_doc_in = REPLACE (xml_doc_in,"http://192.168.56.102/test_cgi/","http://192.168.56.102/gene_cgi/") >> UPDATE result SET xml_doc_out = REPLACE (xml_doc_out,"http://192.168.56.102/test_cgi/","http://192.168.56.102/gene_cgi/") >> UPDATE workunit SET xml_doc = REPLACE(xml_doc,"http://192.168.56.102/test/","http://192.168.56.102/gene/") >> UPDATE app_version SET xml_doc = REPLACE(xml_doc, "http://192.168.56.102/test/", "http://192.168.56.102/gene/");


Delete everything in html/cache, since cached pages may have links to the old URL
>> rm -rf html/cache/*


Run apache and project
>> su -c 'apache2ctl -k start' >> ./bin/start

Profile FrancescoAsnicar [SSC11]
Project developer
Project tester
Project scientist
Avatar
Send message
Joined: 14 Nov 13
Posts: 50
Credit: 6,652,949
RAC: 5,354
Italy
Message 34 - Posted: 6 Dec 2013, 9:05:30 UTC

About the score compute by the client, we implement an md5 summary. So, now the file

"score.txt"
looks like
"1.234567e9 4da1733d0560c8f8cc9606b5e585711d"
.
We will explain to the server how to compute the md5 summary in order to prevent tampering.

Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 320
Credit: 16,284,006
RAC: 4,205
Italy
Message 35 - Posted: 6 Dec 2013, 11:50:57 UTC - in response to Message 33.
Last modified: 6 Dec 2013, 11:53:17 UTC

For example, we want to change the name from 'test' to 'gene'. The URL becomes 'http://gene.disi.unitn.it/gene/'

'gene' will be simply 'boinc'. Sometime in the future we will move the whole system from the virtual machine to a 'real' one and the public url will be something like 'tn-grid.tn.cnr.it/boinc'. ('tn-grid' is just a tentative name, maybe just 'grid', I don't know...).

BTW this procedure is not really simple, but probably it's the only way to do this...

When we will be ready to do this we have to coordinate. Some things must obviously done by root.

Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 320
Credit: 16,284,006
RAC: 4,205
Italy
Message 36 - Posted: 6 Dec 2013, 11:51:55 UTC - in response to Message 34.

About the score compute by the client, we implement an md5 summary. So, now the file

Ok. That's fine.

Trung
Send message
Joined: 28 Nov 13
Posts: 7
Credit: 15
RAC: 0
Italy
Message 37 - Posted: 9 Dec 2013, 3:22:31 UTC

About:

limit somehow the number of distributed wu (no too big client caches now) say max 2 per host per core (just an idea)


I think, we can add a option in 'config.xml' with N is number of task per host per core (logical core):

<max_wus_in_progress> N </max_wus_in_progress>


Tested on boinc client of Ubuntu and Windows XP.

Trung
Send message
Joined: 28 Nov 13
Posts: 7
Credit: 15
RAC: 0
Italy
Message 41 - Posted: 11 Dec 2013, 12:09:06 UTC

I want to ask about the way we provide credits for "Anonymous platform". http://boinc.berkeley.edu/wiki/Anonymous_platform. Because they will not use our application.

Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 320
Credit: 16,284,006
RAC: 4,205
Italy
Message 42 - Posted: 11 Dec 2013, 17:02:01 UTC - in response to Message 41.
Last modified: 11 Dec 2013, 17:03:22 UTC

I want to ask about the way we provide credits for "Anonymous platform". http://boinc.berkeley.edu/wiki/Anonymous_platform. Because they will not use our application.

The "Anonymous platform" is a mechanism that will enable 'skilled' users to change workunit parameters and build their own executables.

Through the anonymous platform file, or app_info.xml file, you tell BOINC which downloaded or self-built applications and libraries it should use and to ignore anything that it is/was sent by the project.

This is very useful to use when the project does not have their own application for your platform OS. You can then build your own (if their source code is available) and use that instead of waiting for the project to come up with the goods.

They can build up their (enhanced or optimized or even fake) application but its output has also to pass our validation scheme in order to be considered good.

We will not provide source code (at least for now) so using the anonymous platform is pointless (and risky). I'm quite sure that there is a way to simply disable it server-side.

Most boinc users will use the more simple and appropriate app_config.xml mechanism for fine tuning the application on their computers.

Profile FrancescoAsnicar [SSC11]
Project developer
Project tester
Project scientist
Avatar
Send message
Joined: 14 Nov 13
Posts: 50
Credit: 6,652,949
RAC: 5,354
Italy
Message 49 - Posted: 13 Dec 2013, 20:37:20 UTC

I was wondering: the URL to use in the "pre-opening by invitation" will be

http://gene.disi.unitn.it/test
or will be changed in something like
http://gene.disi.unitn.it/geneathome
?

Wouldn't be better, since the idea is that TN-Grid groups all the BOINC projects, to have something like
http://tngrid.disi.unitn.it
and then specialise the URL with the project of interest, which will be
http://tngrid.disi.unitn.it/geneathome
instead of the current URL?

Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 320
Credit: 16,284,006
RAC: 4,205
Italy
Message 50 - Posted: 14 Dec 2013, 11:03:35 UTC - in response to Message 49.
Last modified: 14 Dec 2013, 11:43:49 UTC

Now, being actually a test phase, with invitation codes, the name it's somewhat appropriate:
http://gene.disi.unitn.it/test

When we will go public, after a worldwide announcement (with limited hardware, ie the vm) the name will change to:
http://gene.disi.unitn.it/boinc

When we will move to the final domain (with more powerful hardware)
http://grid.tn.cnr.it/boinc

(or something similar, the host name has not been decided yet)

the idea is that the platform (grid) will host (hopefully) various projects and the url should be the same. The user will select which project to run by selecting the wanted application, something like yoyo, wcg or ibercivis.

Profile FrancescoAsnicar [SSC11]
Project developer
Project tester
Project scientist
Avatar
Send message
Joined: 14 Nov 13
Posts: 50
Credit: 6,652,949
RAC: 5,354
Italy
Message 51 - Posted: 14 Dec 2013, 16:22:33 UTC - in response to Message 50.

Ok, thanks


Post to thread

Message boards : Development : Things to do (or to check) for server and application before massive testing


Main page · Your account · Message boards


Copyright © 2017 CNR-TN & UniTN