valtercProject administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 623
Credit: 34,677,535
RAC: 1
Got a couple of hosts getting this error: -1073741795 (0xffffffffc000001d) Unknown error number.
The first host is a Xeon E5-2650 running MS Windows Server 2008 "R2" Enterprise x64 (app is v0.10 (avx))
The second one is Xeon X5680 running MS Windows 10 Pro X64 (anonymous platform)
Any hints about this kind of error and about how to fix it?
Reply
Quote
Recently I investigated similar case here: http://gene.disi.unitn.it/test/forum_thread.php?id=135&postid=817#817. Someone tried to run AVX app on non-AVX CPU. When I googled for this error code (truncated to 32-bit, 0xc000001d) I found pages where people also had this problem when they tried to run some SSE apps on non-SSE CPU.
Edit: I found that it is possible to disable AVX support in Windows by executing command "bcdedit /set xsavedisable 1". Maybe first person did this for some reason (overheating?).
2nd CPU (Xeon X5680) supports up to SSE 4.2, so maybe he/she tried to run AVX or FMA app on it.
____________
Reply
Quote
Hi!
I got these crashes with the standard app too.
It seems that there are different "versions" of AVX (not AVX2) or some AVX-capable processors don't support the full used instruction set. This problem affects the new optimized AVX (not AVX2) app too, both crashed here on and i5 3570, which should be able to run AVX. The SSE2 app runs great on this machine.
Reply
Quote
Hi!
I got these crashes with the standard app too.
It seems that there are different "versions" of AVX (not AVX2) or some AVX-capable processors don't support the full used instruction set. This problem affects the new optimized AVX (not AVX2) app too, both crashed here on and i5 3570, which should be able to run AVX. The SSE2 app runs great on this machine.
Unfortunately your computers are hidden, so I cannot check details. Please send me a link to some AVX WU which crashed for you.
____________
Reply
Quote
http://gene.disi.unitn.it/test/result.php?resultid=6502952
http://gene.disi.unitn.it/test/result.php?resultid=6502909
There you have two of them. The standard app switched randomly between SSE2 and AVX and all the AVX tasks failed, blocking the work unit slot due to a windows error report.
I tried to use the new AVX optimized app on this machine too, which failed the same way. The manually installed new SSE2 optimized app runs without problems.
Reply
Quote
http://gene.disi.unitn.it/test/result.php?resultid=6502952
http://gene.disi.unitn.it/test/result.php?resultid=6502909
There you have two of them. The standard app switched randomly between SSE2 and AVX and all the AVX tasks failed, blocking the work unit slot due to a windows error report.
I tried to use the new AVX optimized app on this machine too, which failed the same way. The manually installed new SSE2 optimized app runs without problems.
Thanks. Your CPU is an Intel Ivy Bridge, so it should have working AVX. I checked these WUs. They worked for some time before crashing, so looks that they were able to execute AVX for some time. Apps on CPUs without AVX usually crashes within few seconds.
I have noticed one thing: second WU worked for over 11 hours before it finally crashed, what is strange. Do you have similar problems with apps from other projects? I suspect that your CPU may be overheating or you have some other hardware issue, e.g. with memory. Please try to stress-test your PC, here is list of some software do do this: https://www.raymond.cc/blog/test-system-stability-by-putting-heavy-load-on-system-resources/. And here are memory testers: http://www.howtogeek.com/260813/how-to-test-your-computers-ram-for-problems/
____________
Reply
Quote
This machine is not constantly monitored because of its location. I have to drive there when there is such a problem or to do majer changes (no remote disk access).
When the science application fails (immediately after the WU is started) windows throw up an error report, which has to be acknowledged by the user. If nobody does this the WU will halt until someone does, blocking the working slot and leaving the CPU idle.
EDIT:
The machine is rockstable, even on Primegrid LLR tests (which are really hard for the CPU) and has produced 107 good and 59 pending WUs since I installed the SSE2 app manually.
Reply
Quote
Well, I am puzzled. It should work for you, bot for some reason it crashes. I checked compilation options and they should be fine, according to various pages enabled instruction sets should be supported by your CPU.
I have created app specifically tailored for IvyBridge CPUs (compiled with -march=ivybridge -mtune=ivybridge). Please try it and let me know if it works or still crashes.
https://bitbucket.org/sirzooro/pc-boinc/downloads/TN-Grid.windows-x86-64-ivybridge-v1.1.zip
____________
Reply
Quote
Sorry to tell you... but it crashes exactly like the previous AVX version.
Reply
Quote
Sorry to tell you... but it crashes exactly like the previous AVX version.
Strange. I suspect that my app uses some rarely used AVX instruction, which is not recognized by your CPU because of some bug in its microcode so it reports error "illegal instruction". Other projects apparently does not use it, so they work fine. Please try updating microcode in your CPU. This update should work for you: https://support.microsoft.com/pl-pl/help/3064209/june-2015-intel-cpu-microcode-update-for-windows. It can be also done from Linux: https://askubuntu.com/questions/545925/how-to-update-intel-microcode-properly/546056. You can also try to update BIOS, CPU microcode updates may be distributed this way too.
____________
Reply
Quote
Strange. I suspect that my app uses some rarely used AVX instruction, which is not recognized by your CPU
Looking at that host's specs,he's on Win7 but it doesn't declare SP1,which is required for AVX support.
Reply
Quote
Strange. I suspect that my app uses some rarely used AVX instruction, which is not recognized by your CPU
Looking at that host's specs,he's on Win7 but it doesn't declare SP1,which is required for AVX support.
You are right, I missed this detail. SP1 is required for AVX.
____________
Reply
Quote
That might be. I'll try that out, just to confirm. Some of my machines are used for nothing else than boinc and therefore not updated to prevent problems with the updates (unplanned restarts, machines not coming up again).
Can the standard application get a check against this before it uses AVX? This will make it more stable.
Reply
Quote
That might be. I'll try that out, just to confirm. Some of my machines are used for nothing else than boinc and therefore not updated to prevent problems with the updates (unplanned restarts, machines not coming up again).
This is very bad from security perspective - new security holes are found every month, and computers connected to Internet and constantly scanned for these holes in attempt to turn them into zombies connected to some botnet. There is even malware which scans local network and tries to infect computers there. Without antivirus and firewall which will block all incoming traffic such computer will sooner or later be infected.
Can the standard application get a check against this before it uses AVX? This will make it more stable.
Yes, it is possible. App could check this and print some user-friendly error. However it still will exit with fail status, unsupported instruction set usually is not something what user may be able to fix without upgrading CPU. I will include such check when I will be releasing new app.
____________
Reply
Quote
All of these computers are behind firewalls and can't be reached by unrequested incoming connections. They are no http servers or something like that. This is why I have to drive to them when I have to make changes. I think this is safe enough.
As the standard application chooses the processor-bounded module by itself, a check function could switch the application to SSE even on AVX-capable processors. This way the workunit will not crash. Of course, this can't be done for manually installed applications (which is not necessary because the user should know what he does when playing with optimized applications).
I updated the machine to SP1 but now I am getting errors that the workunits are committed to another platforms, maybe due to the newest standard application change.
I'll be leaving TN Grid for now, maybe it is more stable and not changed constantly in a couple of months. If I have a single machine only to maintain, then this might be tolerable, but I haven't. I don't like projects which are modified frequently because of that.
Anyway, thanks for your support.
Reply
Quote
... now I am getting errors that the workunits are committed to another platforms, maybe due to the newest standard application change.
I think that particular problem has now been resolved - refer to valterc's response to the thread at http://gene.disi.unitn.it/test/forum_thread.php?id=155.
I was getting that problem myself but have just now been able to download new WUs again so, if you're inclined, it might be worth your while to try agin with the latest official version (v0.11) of the application and see what success you have.
Reply
Quote
All of these computers are behind firewalls and can't be reached by unrequested incoming connections. They are no http servers or something like that. This is why I have to drive to them when I have to make changes. I think this is safe enough.
Windows by default opens few ports for file and printer sharing, please check that you block them too.
As the standard application chooses the processor-bounded module by itself, a check function could switch the application to SSE even on AVX-capable processors. This way the workunit will not crash. Of course, this can't be done for manually installed applications (which is not necessary because the user should know what he does when playing with optimized applications).
BOINC Client sends list of CPU capabilities to server, and server uses it to select app version which will be used. Additionally it can try to compute few WUs using every version supported by given CPU to find which one is the fastest one for it. This for sure could be improved a bit, to check if Win 7 has SP1 installed and do not sent AVX if it does not have it.
It is also possible to create app which will contains all 3 code versions, and
will check CPU capabilities during start to select appropriate one. However creation of such app is more difficult, also performance tests of different versions would be more complicated. So for me simple sanity check in app that required instruction set is available is more reasonable.
I updated the machine to SP1 but now I am getting errors that the workunits are committed to another platforms, maybe due to the newest standard application change.
This is resolved now, you can try again.
I'll be leaving TN Grid for now, maybe it is more stable and not changed constantly in a couple of months. If I have a single machine only to maintain, then this might be tolerable, but I haven't. I don't like projects which are modified frequently because of that.
Anyway, thanks for your support.
You can stick to official app versions, BOINC Client will take care of them. Usually this works fine, except rare situation like this missing SP1.
____________
Reply
Quote
valtercProject administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 623
Credit: 34,677,535
RAC: 1
I could define a minimum os version for avx apps. This will for sure solve this problem. Will check this tomorrow.
Reply
Quote
Windows by default opens few ports for file and printer sharing, please check that you block them too.
That all ends up in the router's firewall.
Reply
Quote
valtercProject administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 623
Credit: 34,677,535
RAC: 1
I could define a minimum os version for avx apps. This will for sure solve this problem. Will check this tomorrow.
Well. It's not so easy (I should make another platform/plan_class etc...) This is something that should be checked by the Boinc client... I don't know why it may report unsupported cpu features.
Does someone know how the handle this at Asteroids@home? (the only project I know that uses "explicit" avx apps)
Reply
Quote