Problems and Technical Issues with Rosetta@home

Message boards : Number crunching : Problems and Technical Issues with Rosetta@home

To post messages, you must log in.

Previous · 1 . . . 292 · 293 · 294 · 295 · 296 · 297 · 298 . . . 313 · Next

AuthorMessage
Bill Swisher

Send message
Joined: 10 Jun 13
Posts: 43
Credit: 35,474,078
RAC: 33,162
Message 109833 - Posted: 9 Oct 2024, 21:32:36 UTC - in response to Message 109831.  

With the weather turning colder recently I could do with some tasks from somewhere to warm the place up a little


I understand this...it's currently 40F (lets call it 4.5F) outside, and I too generate a lot of heat with the computers. Although it's not the most cost effective heating method I consider it my little contribution to the greater good. WCG is at least accepting results now.
ID: 109833 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sid Celery

Send message
Joined: 11 Feb 08
Posts: 2166
Credit: 41,629,484
RAC: 5,494
Message 109834 - Posted: 9 Oct 2024, 22:04:27 UTC - in response to Message 109833.  
Last modified: 9 Oct 2024, 22:05:13 UTC

With the weather turning colder recently I could do with some tasks from somewhere to warm the place up a little

I understand this... it's currently 40F (lets call it 4.5F) outside, and I too generate a lot of heat with the computers. Although it's not the most cost effective heating method I consider it my little contribution to the greater good. WCG is at least accepting results now.

I just remembered - you're in Alaska. It's 11C here right now, but going down as low as 1C overnight in the next couple of days. I need work!
And yes, my WCG results have been accepted too, now that you mention it. No new tasks just yet though.
ID: 109834 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile robertmiles

Send message
Joined: 16 Jun 08
Posts: 1235
Credit: 14,341,506
RAC: 292
Message 109836 - Posted: 9 Oct 2024, 23:22:33 UTC
Last modified: 9 Oct 2024, 23:26:48 UTC

The SiDock@home project has a rather steady supply of tasks. You might want to look there to see if you like those tasks, and if so, check how much they warm up your computer.

The weather report here in Alabama has recently gone as low as 51 F.
ID: 109836 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sid Celery

Send message
Joined: 11 Feb 08
Posts: 2166
Credit: 41,629,484
RAC: 5,494
Message 109838 - Posted: 10 Oct 2024, 0:38:55 UTC - in response to Message 109836.  

The SiDock@home project has a rather steady supply of tasks. You might want to look there to see if you like those tasks, and if so, check how much they warm up your computer.

The weather report here in Alabama has recently gone as low as 51 F

I've known the name of that project for a while, but for some reason I've only just checked it out.
Seems ok, so I've joined and grabbed tasks already.
Let's see how it goes. Ta.
ID: 109838 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1743
Credit: 18,534,891
RAC: 3,108
Message 109844 - Posted: 12 Oct 2024, 3:42:54 UTC
Last modified: 12 Oct 2024, 4:01:52 UTC

There is a new batch of Beta work out.
Just be warned- you need roughly 2.2GB of RAM per Task (although it looks like it will drop down after a while to 1-2GB).
My system that picked up some work a while ago is using 70% of available RAM. The one that only just picked up some work, is using 98% of available RAM (both systems 6c/12t with 32GB RAM).


Edit- the longer running Tasks have reduced their need for RAM even further, now down to around 1GB per Task.
Grant
Darwin NT
ID: 109844 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Bill Swisher

Send message
Joined: 10 Jun 13
Posts: 43
Credit: 35,474,078
RAC: 33,162
Message 109845 - Posted: 12 Oct 2024, 4:38:49 UTC - in response to Message 109844.  

Yep, they got me again. 2 computers, each running 32 threads, each with 32GB of memory, swapping to beat the dickens. Reboot, suspend all the processes, abort all the beta jobs, stop getting Rosetta work for a while and let those things filter out, and resume processing. I'll ask for more early next week. Not looking good to allow these computers to run Rosetta for 5 months when I'm 4,000 miles away.
ID: 109845 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sid Celery

Send message
Joined: 11 Feb 08
Posts: 2166
Credit: 41,629,484
RAC: 5,494
Message 109847 - Posted: 12 Oct 2024, 10:54:21 UTC - in response to Message 109844.  

There is a new batch of Beta work out.
Just be warned- you need roughly 2.2GB of RAM per Task (although it looks like it will drop down after a while to 1-2GB).
My system that picked up some work a while ago is using 70% of available RAM. The one that only just picked up some work, is using 98% of available RAM (both systems 6c/12t with 32GB RAM).

Edit- the longer running Tasks have reduced their need for RAM even further, now down to around 1GB per Task.

You made me look.
All less than 400Mb here for some reason
I'm more annoyed I now have tasks for 2 other projects getting in the way of grabbing more from here.
I can't win
ID: 109847 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Klimax

Send message
Joined: 27 Apr 07
Posts: 44
Credit: 2,801,226
RAC: 45
Message 109850 - Posted: 12 Oct 2024, 14:50:47 UTC

All of my tasks are using after several hours about 2GBs. Also only three days of deadline? Will be tight.
ID: 109850 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1743
Credit: 18,534,891
RAC: 3,108
Message 109851 - Posted: 12 Oct 2024, 21:43:35 UTC - in response to Message 109850.  
Last modified: 12 Oct 2024, 21:50:26 UTC

Also only three days of deadline?
Which has been the case for years now, and which is why you don't return a large amount of the work you download- you miss the deadline almost 50% of the time.

Your initial Estimated completion times are set by the project at 8 hours, but your actual Target CPU time appears to be 12 hours. So even with a small cache, you would miss deadlines. You need no cache, or at the very most a very small cache to avoid missing deadlines (1 day or less).
Running more than one project there is no need for a cache at all. If one project doesn't have work, the other will make up that gap, till the first project has work again. No missed deadlines, no not getting work from one project because the other has filled the cache with work.

Ideally
           Store at least 0.02 days of work
Store up to an additional 0.01 days of work

But if you really feel the need for a cache
           Store at least 0.35 days of work
Store up to an additional 0.01 days of work
would be plenty.


If all of your projects have server issues/lots of periods with no work, then "Store at least xx days of work could be set to 1 day (or 1.5 days if you go with the default Target CPU time of 8 hours).
Grant
Darwin NT
ID: 109851 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1743
Credit: 18,534,891
RAC: 3,108
Message 109852 - Posted: 12 Oct 2024, 21:56:06 UTC - in response to Message 109847.  
Last modified: 12 Oct 2024, 21:56:41 UTC

You made me look.
All less than 400Mb here for some reason
After about half an hour mine end up down around 700-800MB. But when they first start after 5min or so they're still up around 2GB+ before dropping down again.
I'd check what application is actually running those other Tasks- probably your other projects, or they're resends and not one of the latest Beta batch.
Grant
Darwin NT
ID: 109852 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jean-David Beyer

Send message
Joined: 2 Nov 05
Posts: 199
Credit: 6,668,482
RAC: 3,812
Message 109853 - Posted: 13 Oct 2024, 0:11:58 UTC - in response to Message 109844.  

There is a new batch of Beta work out.
Just be warned- you need roughly 2.2GB of RAM per Task (although it looks like it will drop down after a while to 1-2GB)
.

My latest of these are taking 2.3G to 2.5G each on my Linux machine. I allow 4 Rosetta tasks to run at a time. IIRC, they take 8 to 9 hours of wall clock time to run.

Computer 5910575
Computer information

CPU type 	GenuineIntel
Intel(R) Xeon(R) W-2245 CPU @ 3.90GHz [Family 6 Model 85 Stepping 7]
Number of processors 	16
Coprocessors 	---
Operating System 	Linux Red Hat Enterprise Linux
Red Hat Enterprise Linux 8.10 (Ootpa) [4.18.0-553.22.1.el8_10.x86_64|libc 2.28]
BOINC version 	7.20.2
Memory 	128085.97 MB
Cache 	16896 KB
Swap space 	15992 MB
Total disk space 	488.04 GB
Free Disk Space 	479.37 GB

ID: 109853 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sid Celery

Send message
Joined: 11 Feb 08
Posts: 2166
Credit: 41,629,484
RAC: 5,494
Message 109855 - Posted: 13 Oct 2024, 3:25:51 UTC - in response to Message 109852.  

You made me look.
All less than 400Mb here for some reason
After about half an hour mine end up down around 700-800MB. But when they first start after 5min or so they're still up around 2GB+ before dropping down again.
I'd check what application is actually running those other Tasks - probably your other projects, or they're resends and not one of the latest Beta batch.

When I wrote that I was on my 6-core machine.
More odd is now I've returned to my 8C/16T machine I found it was running only 8 tasks plus 2 waiting for memory and only 1 WCG running - 7 cores idle,
Checking task manager all Rosetta tasks were again only using 3-400Mb each and I had a lot of RAM spare too - what happened to the other cores, I don't know.
I suspended all Rosetta tasks, letting SiDock get its turn - 16 tasks very low RAM - then when they finished, priority went back to Rosetta and all 16 threads started running tasks again.
Some funny business going on somewhere...

At least I created a little space to download more of the remaining few Rosetta tasks available. Not many left now.
ID: 109855 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sid Celery

Send message
Joined: 11 Feb 08
Posts: 2166
Credit: 41,629,484
RAC: 5,494
Message 109857 - Posted: 13 Oct 2024, 8:35:03 UTC - in response to Message 109855.  
Last modified: 13 Oct 2024, 8:52:22 UTC

You made me look.
All less than 400Mb here for some reason
After about half an hour mine end up down around 700-800MB. But when they first start after 5min or so they're still up around 2GB+ before dropping down again.
I'd check what application is actually running those other Tasks - probably your other projects, or they're resends and not one of the latest Beta batch.

When I wrote that I was on my 6-core machine.
More odd is now I've returned to my 8C/16T machine I found it was running only 8 tasks plus 2 waiting for memory and only 1 WCG running - 7 cores idle,
Checking task manager all Rosetta tasks were again only using 3-400Mb each and I had a lot of RAM spare too - what happened to the other cores, I don't know.
I suspended all Rosetta tasks, letting SiDock get its turn - 16 tasks very low RAM - then when they finished, priority went back to Rosetta and all 16 threads started running tasks again.
Some funny business going on somewhere...

At least I created a little space to download more of the remaining few Rosetta tasks available. Not many left now.

Got up to find only 6 Rosetta tasks running, plus 4 waiting for memory and 6 threads idle, while RAM is at 65% used and 5.5Gb free
5 of the tasks are using 310-440Mb, only one using 2.122Gb
This is very odd

Edit: This is getting even weirder.
Having set NNT for all projects, I suspended all the <non-running> Rosetta tasks first and immediately the 6 idle threads started running 6 SiDock tasks. Those Rosetta tasks waiting for memory were still waiting for memory.
Then I suspended all the running & waiting for memory Rosetta tasks and all 16 threads are now running SiDock tasks.
My intention, as before, was to free up my non-Rosetta offline cache to allow more Rosetta into the cache, then return to Rosetta tasks, restarting them one at a time which seemed to allow all 16 threads to run Rosetta the last time.
Why suspending unstarted tasks allowed the idle threads to be utilised, I have no idea. Never seen that before,
ID: 109857 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1743
Credit: 18,534,891
RAC: 3,108
Message 109858 - Posted: 13 Oct 2024, 9:07:18 UTC - in response to Message 109857.  

Got up to find only 6 Rosetta tasks running, plus 4 waiting for memory and 6 cores idle, while RAM is at 65% used and 5.5Gb free
5 of the tasks are using 310-440Mb, only one using 2.122Gb
This is very odd
Very, very odd.
Most of my Tasks are now using around 2GB of RAM, even after running for a few hours.

I'd suggest checking your "When and how BOINC uses your computer" preferences.

These are mine- the most likely to be causing issues- the Memory preferences. Is "Leave non-GPU tasks in memory while suspended" selected? And low "Use at most preferences" would also cause issues.

Computing
Usage limits	
                                  Use at most 100 % of the CPUs
                                  Use at most 100 % of CPU time

When to suspend	
          Suspend when computer is on battery No
              Suspend when computer is in use No
Suspend GPU computing when computer is in use No
  'In use' means mouse/keyboard input in last 3 minutes
 Suspend when no mouse/keyboard input in last --- minutes
    Suspend when non-BOINC CPU usage is above --- %
                         Compute only between ---

Other	
                               Store at least 0.35 days of work
                    Store up to an additional 0.01 days of work
                   Switch between tasks every 60 minutes
    Request tasks to checkpoint at most every 60 seconds

Disk
                             Use no more than 30 GB
                               Leave at least 2 GB free
                             Use no more than 60 % of total

Memory
         When computer is in use, use at most 95 %
     When computer is not in use, use at most 98 %
Leave non-GPU tasks in memory while suspended No
                  Page/swap file: use at most 75 %

Grant
Darwin NT
ID: 109858 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Klimax

Send message
Joined: 27 Apr 07
Posts: 44
Credit: 2,801,226
RAC: 45
Message 109859 - Posted: 13 Oct 2024, 9:53:29 UTC - in response to Message 109851.  
Last modified: 13 Oct 2024, 9:55:38 UTC

Also only three days of deadline?
Which has been the case for years now, and which is why you don't return a large amount of the work you download- you miss the deadline almost 50% of the time.

Your initial Estimated completion times are set by the project at 8 hours, but your actual Target CPU time appears to be 12 hours. So even with a small cache, you would miss deadlines. You need no cache, or at the very most a very small cache to avoid missing deadlines (1 day or less).
Running more than one project there is no need for a cache at all. If one project doesn't have work, the other will make up that gap, till the first project has work again. No missed deadlines, no not getting work from one project because the other has filled the cache with work.

Ideally
           Store at least 0.02 days of work
Store up to an additional 0.01 days of work

But if you really feel the need for a cache
           Store at least 0.35 days of work
Store up to an additional 0.01 days of work
would be plenty.


If all of your projects have server issues/lots of periods with no work, then "Store at least xx days of work could be set to 1 day (or 1.5 days if you go with the default Target CPU time of 8 hours).

While on surface or first pass you'd be correct, things are bit more complex (Side note: My question about deadlines shows how long ago I fully paid attention to project).

First, variable memory consumption. Often several tasks are waiting for others to finish or to drop currently allocated memory (Seems that nlike others, I got tasks that keep all 2GB allocated even way later).

Second, BOINC or Rosetta has very weird accounting of remaining time to completion. (Lots of task are now around 50% mark yet estimated time to completion is still 12 hours, while waiting tasks have 8 hours. (And yes, my target runtime is 12 hours)

All those canceled were in any case from beginning of month were computer was configured for all 20 cores to be used and ran out of memory very fast making lost of tasks waiting for memory and thus lots of cancellations. I have since then changed configuration to use only 10 cores, so same situation shouldn't occur.

BTW: My cache is configured for 1+1. As long as computer is running 24h there soul be a day of reserve.

ETA: Looks like some tasks finally dropped in memory usage to 1GB and one to 700MB.
ID: 109859 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1743
Credit: 18,534,891
RAC: 3,108
Message 109871 - Posted: 14 Oct 2024, 7:31:42 UTC - in response to Message 109859.  
Last modified: 14 Oct 2024, 7:32:16 UTC

BTW: My cache is configured for 1+1. As long as computer is running 24h there soul be a day of reserve.
Setting it that way may not give you what you might expect it to.
If you want 2 days worth, then set it to 2+ 0.01.

Those additional days are just that- additional days. They will only be added on when the cache gets low enough to reach the "Store at least value" and it needs to be topped up. Then it will also top up the additional day, which will then run down again until the "Store at least value" is reached again.


With it set to 1+1 you will get one day's worth, plus another day's worth, but then the cache will run down to just under 1 day's worth, then it will refill the 1 day & then re-fill the second additional day.
With it set to 2+ 0.01 as it returns a Task, it will download another to keep the cache at the 2 days level.
Grant
Darwin NT
ID: 109871 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
tgbauer

Send message
Joined: 5 Jan 06
Posts: 11
Credit: 103,500,003
RAC: 18,884
Message 109875 - Posted: 15 Oct 2024, 1:26:27 UTC

Looks like Application "Rosetta Beta 6.06" tasks are using 2.5GB of RAM each! That becomes a bit inefficient when have 128 cores in a computer and 128GB RAM (only 46/128 cores used). Ones before that and "Rosetta 4.20" are consuming less than 0.5GB (and all 128 cores used).
Is it possible to limit the RAM usage per task, so can consume all cores again?
ID: 109875 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1743
Credit: 18,534,891
RAC: 3,108
Message 109876 - Posted: 15 Oct 2024, 5:46:03 UTC - in response to Message 109875.  

Is it possible to limit the RAM usage per task, so can consume all cores again?
No.
As mentioned in the previous posts, the high RAM usage is generally only for the first 30min or so. After that, it drops down to 1GB or less (although there can be some Tasks where it goes up to 2GB per Task for a while later on, before dropping down again- my current Tasks after 4 hours are using around 800MB each).
Grant
Darwin NT
ID: 109876 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Bill Swisher

Send message
Joined: 10 Jun 13
Posts: 43
Credit: 35,474,078
RAC: 33,162
Message 109877 - Posted: 15 Oct 2024, 15:01:46 UTC - in response to Message 109876.  
Last modified: 15 Oct 2024, 15:02:31 UTC

[No. As mentioned in the previous posts, the high RAM usage is generally only for the first 30min or so. After that, it drops down to 1GB or less (although there can be some Tasks where it goes up to 2GB per Task for a while later on, before dropping down again- my current Tasks after 4 hours are using around 800MB each).


Then I have no choice but to NOT run Rosetta beta and I don't see an option to turn the beta work off. If I could limit the number (say 4) of them running it would be possible. So I guess I'll leave Rosetta turned off. As I said, 16c/32t and 32GB of memory (I pretty much build all the computers now days with 32GB of memory) and I'm physically away from the computers for 5 months out of the year. Well it was fun while it lasted.
ID: 109877 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
PMH_UK

Send message
Joined: 9 Aug 08
Posts: 16
Credit: 1,243,749
RAC: 0
Message 109878 - Posted: 15 Oct 2024, 15:23:59 UTC - in response to Message 109877.  

You can limit the number running using app_config.xml file in Rosetta's project directory.
Create/amend then from menu select Read config files.
Paul.
ID: 109878 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Previous · 1 . . . 292 · 293 · 294 · 295 · 296 · 297 · 298 . . . 313 · Next

Message boards : Number crunching : Problems and Technical Issues with Rosetta@home



©2025 University of Washington
https://www.bakerlab.org