Storage problem
log in

Advanced search

Message boards : News : Storage problem

1 · 2 · Next
Author Message
Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 558
Credit: 31,320,606
RAC: 14,727
Italy
Message 2384 - Posted: 14 Oct 2021, 11:35:54 UTC
Last modified: 14 Oct 2021, 14:26:09 UTC

The University's storage system (that we heavily use) is having big performance issues thus everything is painfully slow. I stopped the work generator, which actually was not working because of this. No new tasks will be generated until the assimilator will be able to move the finished results to the proper place. There is nothing else to do but wait until the problem will be solved, no ETA, unfortunately.

Profile adrianxw
Send message
Joined: 22 Dec 16
Posts: 28
Credit: 8,089,436
RAC: 1,947
Denmark
Message 2385 - Posted: 14 Oct 2021, 14:23:16 UTC - in response to Message 2384.

Most crunchers attatch several projects to a computing node, so if one, or more, temporarily, stops producing work, their machines happily move onto other projects. Nice of you to let us know though.

It would be interesting to know what kind of systems, (processors, RAM, disk capacity, communications etc.), are used to run a project.
____________
Wave upon wave of demented avengers march cheerfully out of obscurity into the dream.

Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 558
Credit: 31,320,606
RAC: 14,727
Italy
Message 2386 - Posted: 15 Oct 2021, 15:22:21 UTC - in response to Message 2385.

Well, I really don't know what's behind the University storage system (hardware, configuration, etc.), we just use it :)
BTW, it seems that the system is slowly recovering, when the numbers in the project status page will be close to zero I will try to start the work generator again and see if it will work as usual.

Gunnar Hjern
Send message
Joined: 29 Mar 17
Posts: 14
Credit: 13,550,117
RAC: 318
Sweden
Message 2387 - Posted: 15 Oct 2021, 17:28:49 UTC - in response to Message 2386.

Good news!!!
Really looking forward to that, as my computers are starting to get very hungry now! :-)

Btw., I can see that there where no stats exported to BoincStats today. Is that issue also related to the failing storage system?

Aurum
Send message
Joined: 18 Jul 18
Posts: 91
Credit: 288,118,799
RAC: 14,850
United States
Message 2388 - Posted: 16 Oct 2021, 13:46:02 UTC

Can we donate our GRC so you can buy your own storage node?

Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 558
Credit: 31,320,606
RAC: 14,727
Italy
Message 2391 - Posted: 18 Oct 2021, 9:08:12 UTC - in response to Message 2387.

Good news!!!
Really looking forward to that, as my computers are starting to get very hungry now! :-)

Btw., I can see that there where no stats exported to BoincStats today. Is that issue also related to the failing storage system?

I had also to stop all the BOINC's server cron activities...

Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 558
Credit: 31,320,606
RAC: 14,727
Italy
Message 2394 - Posted: 25 Oct 2021, 22:25:04 UTC - in response to Message 2391.

We are still facing the "storage" problem. Everything, sometimes, is painfully slow (I temporarily disabled the work generator)

Profile Keith Myers
Send message
Joined: 26 Jun 20
Posts: 58
Credit: 12,663,635
RAC: 22,859
United States
Message 2405 - Posted: 10 Nov 2021, 1:14:09 UTC - in response to Message 2394.

Any recent news from the university on their storage problem?

I've never been out of work on this project till now.

Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 558
Credit: 31,320,606
RAC: 14,727
Italy
Message 2406 - Posted: 10 Nov 2021, 18:20:36 UTC - in response to Message 2405.
Last modified: 16 Nov 2021, 16:43:05 UTC

Any recent news from the university on their storage problem?

I've never been out of work on this project till now.

They are working on it, but it seems that they don't know the origin of the problem. A new storage system has been ordered but delayed due to legal issues....

I will wait until the end of this week, then I will try to switch from the NFS UniTN storage to a local one (not that big but reliable). It will unfortunately take some time to do that, every file operation affecting that storage is painfully slow...

In the meantime you may see the work generator working, from time to time

[EDIT] It seems that people of UniTN were able to fix the storage problem. Let's hope it lasts.

kotenok2000
Send message
Joined: 18 Feb 20
Posts: 3
Credit: 125,416
RAC: 10
Russia
Message 2414 - Posted: 24 Nov 2021, 22:28:12 UTC - in response to Message 2406.

I can't get any workunits.

Falconet
Send message
Joined: 21 Dec 16
Posts: 75
Credit: 1,788,793
RAC: 5,058
Portugal
Message 2415 - Posted: 24 Nov 2021, 23:04:53 UTC - in response to Message 2414.
Last modified: 24 Nov 2021, 23:05:00 UTC

Work generator is down and there are no tasks ready to send.

Nothing to do but wait.
____________

Profile Keith Myers
Send message
Joined: 26 Jun 20
Posts: 58
Credit: 12,663,635
RAC: 22,859
United States
Message 2416 - Posted: 25 Nov 2021, 0:35:26 UTC

Seems to be storage server issues again. Keep getting backed off on uploads.

kotenok2000
Send message
Joined: 18 Feb 20
Posts: 3
Credit: 125,416
RAC: 10
Russia
Message 2418 - Posted: 25 Nov 2021, 0:38:52 UTC - in response to Message 2417.

And doubleposting problems

Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 558
Credit: 31,320,606
RAC: 14,727
Italy
Message 2419 - Posted: 25 Nov 2021, 10:36:29 UTC - in response to Message 2417.

Seems to be storage server issues again. Keep getting backed off on uploads.

Yes, again... Unfortunately I don't know any other detail, yet...

Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 558
Credit: 31,320,606
RAC: 14,727
Italy
Message 2557 - Posted: 26 Feb 2022, 0:25:45 UTC - in response to Message 2419.
Last modified: 26 Feb 2022, 0:26:26 UTC

The "storage" problem is here again.... I stopped the work generator to lighten the load of the system, I will check again the server in a few (a dozen) hours.

Greg_BE
Send message
Joined: 22 Feb 22
Posts: 5
Credit: 202,350
RAC: 1,001
Belgium
Message 2558 - Posted: 26 Feb 2022, 9:34:46 UTC

" A new storage system has been ordered but delayed due to legal issues...."

Legal issues over a computer storage system?
Is this thing from China or a classified system?
I have never heard of "legal problems" over something so simple unless the supplier is in legal trouble.

Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 558
Credit: 31,320,606
RAC: 14,727
Italy
Message 2559 - Posted: 26 Feb 2022, 12:06:51 UTC - in response to Message 2558.

While buying expensive things we have to issue a government tender, open to any company in the EU. Someone won the contract, an excluded one did not agree and started a legal issue, so everything was frozen. Unfortunately this kind of legal issue is very slow to resolve here in Italy.

ChinookFoehn
Send message
Joined: 14 Feb 21
Posts: 5
Credit: 1,269,153
RAC: 2,255
Message 2560 - Posted: 26 Feb 2022, 14:56:50 UTC - in response to Message 2559.

I'm curious...
Does this mean the project will shut down until the court process is finished or...
Might it be possible to "rent" the equipment until the court process is completed?

Profile valterc
Project administrator
Project tester
Send message
Joined: 30 Oct 13
Posts: 558
Credit: 31,320,606
RAC: 14,727
Italy
Message 2561 - Posted: 26 Feb 2022, 15:20:04 UTC - in response to Message 2560.
Last modified: 26 Feb 2022, 15:27:41 UTC

The court process is related to a new storage system. The old one, that we currently use, has (and had) some serious problems, simply speaking it's painfully slow, sometimes. There are two possible scenarios:
- The IT crew will fix the problem in a few hours or days (as happened in the past) and I will restart everything.
- The problem is not fixable. I already managed to find another disk, although not so big. In this case I have to wait until the current queue becomes almost empty (you may look at the numbers in the project status), transfer everything to that disk and start again.

[EDIT} now it "seems" that everything is working at full speed, will check again in a few hours.

ChinookFoehn
Send message
Joined: 14 Feb 21
Posts: 5
Credit: 1,269,153
RAC: 2,255
Message 2562 - Posted: 26 Feb 2022, 19:06:29 UTC - in response to Message 2561.

Understand the problem.

Same sort of problem that happens in the United States of America where a loser than sues the American Government, to stop the winner, and thereby making everything more expensive.

My thoughts on renting...
Could you not "rent" the new system, from the company who won the bid, and pay a monthly rental fee, which would then be deducted from the purchase price - once you win the court case?

Of course, if the company who sued wins... It would mean a more expensive system, in the end but, it seems to me, unless major "bribes" (political donations) are paid, those who lose a bid, also lose in the courts.

1 · 2 · Next
Post to thread

Message boards : News : Storage problem


Main page · Your account · Message boards


Copyright © 2022 CNR-TN & UniTN