Problems and Technical Issues with Rosetta@home

Message boards : Number crunching : Problems and Technical Issues with Rosetta@home

To post messages, you must log in.

Previous · 1 . . . 139 · 140 · 141 · 142 · 143 · 144 · 145 . . . 317 · Next

AuthorMessage
Jean-David Beyer

Send message
Joined: 2 Nov 05
Posts: 204
Credit: 6,927,641
RAC: 9,107
Message 103623 - Posted: 30 Nov 2021, 19:14:35 UTC - in response to Message 103597.  

project? I have never seen that before.
Most of us tried to use max_concurrent and then got buried in tons of tasks we could never complete by their deadlines.


Project_max_current will limit the total number of work units running for all projects.

But either one of them can cause the problem of excessive downloads.


Well, I use only the <project_max_concurrent> not <max_concurrent>,
and then only in the app_config.xml file in the

/var/lib/boinc/projects/boinc.bakerlab.org_rosetta directory.

[/var/lib/boinc/projects/boinc.bakerlab.org_rosetta]$ cat app_config.xml 
<app_config>
   <project_max_concurrent>3</project_max_concurrent>
</app_config>


I use similar app_config.xml files in the project directories for my other projects as well (with different limits).
localhost:jeandavid8[/var/lib/boinc/projects]$ ls -l

 16384 Nov 30 06:20 boinc.bakerlab.org_rosetta
 12288 Nov 29 16:53 climateprediction.net
 24576 Nov 29 23:13 universeathome.pl_universe
 40960 Nov 30 13:15 www.worldcommunitygrid.org

I am running
boinc-client-7.16.11-3.el8.x86_64
that is the most up-to-date one for this machine and OS. The OS is up-to date as well.
Computer 5910575

CPU type 	GenuineIntel
Intel(R) Xeon(R) W-2245 CPU @ 3.90GHz [Family 6 Model 85 Stepping 7]
Number of processors 	16
Operating System 	Linux Red Hat Enterprise Linux
Red Hat Enterprise Linux 8.5 (Ootpa) [4.18.0-348.2.1.el8_5.x86_64|libc 2.28 (GNU libc)]
BOINC version 	7.16.11
Memory 	         63902.16 MB
Cache 	            16896 KB
Swap space 	    15992 MB
Total disk space   117.21 GB
Free Disk Space     92.03 GB

ID: 103623 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Falconet

Send message
Joined: 9 Mar 09
Posts: 354
Credit: 1,282,599
RAC: 86
Message 103624 - Posted: 30 Nov 2021, 19:20:00 UTC

Pythons are back and I received 5 on my 8 GB laptop which previously received none.
ID: 103624 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Greg_BE
Avatar

Send message
Joined: 30 May 06
Posts: 5691
Credit: 5,859,226
RAC: 0
Message 103626 - Posted: 30 Nov 2021, 19:43:39 UTC - in response to Message 103620.  

Is there a dummies page with a simple explantion to set this up?

With BOINC, it only gets worse.
https://boinc.berkeley.edu/wiki/Client_configuration#Options


I've seen something else elsewhere. I'll look that up this weekend.
ID: 103626 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Greg_BE
Avatar

Send message
Joined: 30 May 06
Posts: 5691
Credit: 5,859,226
RAC: 0
Message 103627 - Posted: 30 Nov 2021, 19:45:55 UTC - in response to Message 103623.  

project? I have never seen that before.
Most of us tried to use max_concurrent and then got buried in tons of tasks we could never complete by their deadlines.


Project_max_current will limit the total number of work units running for all projects.

But either one of them can cause the problem of excessive downloads.


Well, I use only the <project_max_concurrent> not <max_concurrent>,
and then only in the app_config.xml file in the

/var/lib/boinc/projects/boinc.bakerlab.org_rosetta directory.

[/var/lib/boinc/projects/boinc.bakerlab.org_rosetta]$ cat app_config.xml 
<app_config>
   <project_max_concurrent>3</project_max_concurrent>
</app_config>


I use similar app_config.xml files in the project directories for my other projects as well (with different limits).
localhost:jeandavid8[/var/lib/boinc/projects]$ ls -l

 16384 Nov 30 06:20 boinc.bakerlab.org_rosetta
 12288 Nov 29 16:53 climateprediction.net
 24576 Nov 29 23:13 universeathome.pl_universe
 40960 Nov 30 13:15 www.worldcommunitygrid.org

I am running
boinc-client-7.16.11-3.el8.x86_64
that is the most up-to-date one for this machine and OS. The OS is up-to date as well.
Computer 5910575

CPU type 	GenuineIntel
Intel(R) Xeon(R) W-2245 CPU @ 3.90GHz [Family 6 Model 85 Stepping 7]
Number of processors 	16
Operating System 	Linux Red Hat Enterprise Linux
Red Hat Enterprise Linux 8.5 (Ootpa) [4.18.0-348.2.1.el8_5.x86_64|libc 2.28 (GNU libc)]
BOINC version 	7.16.11
Memory 	         63902.16 MB
Cache 	            16896 KB
Swap space 	    15992 MB
Total disk space   117.21 GB
Free Disk Space     92.03 GB



So? How's that working out on Python? That might be the solution to limit them.
ID: 103627 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Falconet

Send message
Joined: 9 Mar 09
Posts: 354
Credit: 1,282,599
RAC: 86
Message 103628 - Posted: 30 Nov 2021, 19:59:39 UTC - in response to Message 103624.  

Pythons are back and I received 5 on my 8 GB laptop which previously received none.


Looks like I can run 2 Pythons plus 3 MCM tasks. So 3 threads are idle.
ID: 103628 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
.clair.

Send message
Joined: 2 Jan 07
Posts: 274
Credit: 26,399,595
RAC: 0
Message 103629 - Posted: 30 Nov 2021, 20:03:20 UTC
Last modified: 30 Nov 2021, 20:15:04 UTC

PYTHON MINI ARE IN
names like . aagb-AGLY......
`task` - `properties` gives
working set size 2.79GB
virtual memory size 99MB
progressing @ 12.6% per hour
that's a lot better :)

edit
Hellow Falconet, we were typing at the same time :)
another edit
got some cosmology@hum docker tasks to finish off then find out how many Python mini fits into 32GB
yet another idiot, oops, I ment edit
front page news - Total queued jobs: 2,197,433
ID: 103629 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
.clair.

Send message
Joined: 2 Jan 07
Posts: 274
Credit: 26,399,595
RAC: 0
Message 103631 - Posted: 30 Nov 2021, 22:01:16 UTC
Last modified: 30 Nov 2021, 22:04:28 UTC

Had to reboot it coz it cant handle 15 , changed `use cpu` settings and got 11 running so far
but its thrashing the disk when it starts them
only 20GB memory in use
100GB disk space in use by rosetta
ID: 103631 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jean-David Beyer

Send message
Joined: 2 Nov 05
Posts: 204
Credit: 6,927,641
RAC: 9,107
Message 103632 - Posted: 30 Nov 2021, 22:52:16 UTC - in response to Message 103627.  

Well, I use only the <project_max_concurrent> not <max_concurrent>,
and then only in the app_config.xml file in the

/var/lib/boinc/projects/boinc.bakerlab.org_rosetta directory.

[/var/lib/boinc/projects/boinc.bakerlab.org_rosetta]$ cat app_config.xml
<app_config>
<project_max_concurrent>3</project_max_concurrent>
</app_config>


So? How's that working out on Python? That might be the solution to limit them.

I have no idea.

Mon 29 Nov 2021 01:31:22 AM EST | Rosetta@home | Message from server: VirtualBox is not installed

I do no have VirtualBox, so I cannot run them.
ID: 103632 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Greg_BE
Avatar

Send message
Joined: 30 May 06
Posts: 5691
Credit: 5,859,226
RAC: 0
Message 103633 - Posted: 30 Nov 2021, 23:18:50 UTC - in response to Message 103632.  
Last modified: 30 Nov 2021, 23:20:32 UTC

Well, I use only the <project_max_concurrent> not <max_concurrent>,
and then only in the app_config.xml file in the

/var/lib/boinc/projects/boinc.bakerlab.org_rosetta directory.

[/var/lib/boinc/projects/boinc.bakerlab.org_rosetta]$ cat app_config.xml
<app_config>
<project_max_concurrent>3</project_max_concurrent>
</app_config>


So? How's that working out on Python? That might be the solution to limit them.

I have no idea.

Mon 29 Nov 2021 01:31:22 AM EST | Rosetta@home | Message from server: VirtualBox is not installed

I do no have VirtualBox, so I cannot run them.




hmm...ok...well maybe after they load up the project again I will try that.
ID: 103633 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
.clair.

Send message
Joined: 2 Jan 07
Posts: 274
Credit: 26,399,595
RAC: 0
Message 103634 - Posted: 1 Dec 2021, 0:58:23 UTC
Last modified: 1 Dec 2021, 0:59:52 UTC

Well I gave the confuzer a while to get its act together with 11 wu, then tried to increase it to twelve cpu`s
it don`t want to play, BM put it on hold with ` waiting for memory` [20GB in actual use]
So that's my lot with a 16 cpu opteron and 32GB mem, using about 75< 80% cpu
a lot better than the 4 wu it ran with big pythons.

Now then what can I meddle with next . . . .
How about going to `vbox64_mt` and setting the default cpu count to two and then half the default run time
that would offset the memory use and use more cpu`s , hmm ;)
ID: 103634 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile robertmiles

Send message
Joined: 16 Jun 08
Posts: 1235
Credit: 14,372,156
RAC: 692
Message 103635 - Posted: 1 Dec 2021, 1:03:17 UTC - in response to Message 103634.  
Last modified: 1 Dec 2021, 1:08:29 UTC

Well I gave the confuzer a while to get its act together with 11, then tried to increase it to twelve cpu`s
it don`t want to play, BM put it on hold with ` waiting for memory` [20GB in actual use]
So that's my lot with 16 cpu opteron and 32GB mem, using about 75< 80% cpu
a lot better than the 4 wu it ran with big pythons.

Now then what can I meddle with next . . . .
How about going to `vbox64_mt` and setting the default cpu count to two and then half the default run time
that would offset the memory use and use more cpu`s , hmm ;)

You'd better check if Oracle provides vbox64_mt and whether the Python tasks are able to use it before doing much with that.

The Python tasks now only reserve 2.79GB of memory each, at least for Windows 10, so the project staff HAS found a way to control the amount of memory reserved.
ID: 103635 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1761
Credit: 18,534,891
RAC: 214
Message 103638 - Posted: 1 Dec 2021, 6:46:39 UTC - in response to Message 103635.  

The Python tasks now only reserve 2.79GB of memory each, at least for Windows 10, so the project staff HAS found a way to control the amount of memory reserved.
And if they bring it down to 1.5GB or so then everything should be OK.

Would be nice if we were to get some more Rosetta 4.20 Tasks as well.
Grant
Darwin NT
ID: 103638 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jonathan

Send message
Joined: 4 Oct 17
Posts: 43
Credit: 1,337,472
RAC: 0
Message 103639 - Posted: 1 Dec 2021, 10:42:31 UTC - in response to Message 103638.  

The created VMs still have the same Hard Disk and RAM set up as before. 8Gb HD and 6Gb RAM.
All tasks I had running had to be aborted. They weren't using any cpu cycles.

Is anyone successfully completing one?
ID: 103639 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Falconet

Send message
Joined: 9 Mar 09
Posts: 354
Credit: 1,282,599
RAC: 86
Message 103640 - Posted: 1 Dec 2021, 11:29:44 UTC - in response to Message 103639.  
Last modified: 1 Dec 2021, 11:31:32 UTC

The created VMs still have the same Hard Disk and RAM set up as before. 8Gb HD and 6Gb RAM.
All tasks I had running had to be aborted. They weren't using any cpu cycles.

Is anyone successfully completing one?



Mine are saying 2.79 GB of RAM and using barely over 56 MB each.
I've completed 4 so far on my laptop.


I see you got a few done. OThers were aborted. If they weren't using CPU cycles, it's possible they were some of those that hang for hours and hours.
ID: 103640 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jonathan

Send message
Joined: 4 Oct 17
Posts: 43
Credit: 1,337,472
RAC: 0
Message 103641 - Posted: 1 Dec 2021, 11:47:52 UTC - in response to Message 103640.  

What Boinc reports and what the VM is created with is two different things. I get about the same results as you looking at the Boinc task properties.
All the successful tasks were the previous Python versions. Started with "boinc_cages_IL_"

I haven't got a single, newer one to run yet. they all hang and when I call up the VM's screen the last line is
"Intel MKL FATAL ERROR: Error on loading function mkl_lapack_ps_mc3_dsytrf_l_small."

If you call up Virtual box and look at a VM, is you last line like that on the monitor? You just go to Machine - Detach GUI to close it afterwards.
ID: 103641 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Falconet

Send message
Joined: 9 Mar 09
Posts: 354
Credit: 1,282,599
RAC: 86
Message 103642 - Posted: 1 Dec 2021, 12:30:59 UTC - in response to Message 103641.  

I see VB reports 6144 MB of base memory.
No, I don't have that line when I open the VM on either of the 2 Pythons that seem to be running fine.
ID: 103642 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile dcdc

Send message
Joined: 3 Nov 05
Posts: 1832
Credit: 119,951,714
RAC: 4,006
Message 103643 - Posted: 1 Dec 2021, 14:22:48 UTC

I've got one here that's saying 6GB, but also one that has 10200MB! And 20GB of disk space...
ID: 103643 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Falconet

Send message
Joined: 9 Mar 09
Posts: 354
Credit: 1,282,599
RAC: 86
Message 103644 - Posted: 1 Dec 2021, 14:30:10 UTC - in response to Message 103643.  

I'm still running 2 Pythons and Rosetta is using 24 GB of my SSD.
ID: 103644 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
.clair.

Send message
Joined: 2 Jan 07
Posts: 274
Credit: 26,399,595
RAC: 0
Message 103645 - Posted: 1 Dec 2021, 17:51:04 UTC - in response to Message 103635.  

snip...
Now then what can I meddle with next . . . .
How about going to `vbox64_mt` and setting the default cpu count to two and then half the default run time
that would offset the memory use and use more cpu`s , hmm ;)

You'd better check if Oracle provides vbox64_mt and whether the Python tasks are able to use it before doing much with that.
The Python tasks now only reserve 2.79GB of memory each, at least for Windows 10, so the project staff HAS found a way to control the amount of memory reserved.

cosmology@home has been using `vbox64_mt` for a few years on its vbox work

I see there are still a lot of big Python _1 tasks in the pipeline, resends that will take a while to clean up, that still demand big memory, and stop the python mini`s running
ID: 103645 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile robertmiles

Send message
Joined: 16 Jun 08
Posts: 1235
Credit: 14,372,156
RAC: 692
Message 103646 - Posted: 1 Dec 2021, 22:17:52 UTC - in response to Message 103645.  

snip...
Now then what can I meddle with next . . . .
How about going to `vbox64_mt` and setting the default cpu count to two and then half the default run time
that would offset the memory use and use more cpu`s , hmm ;)

You'd better check if Oracle provides vbox64_mt and whether the Python tasks are able to use it before doing much with that.
The Python tasks now only reserve 2.79GB of memory each, at least for Windows 10, so the project staff HAS found a way to control the amount of memory reserved.

cosmology@home has been using `vbox64_mt` for a few years on its vbox work

I see there are still a lot of big Python _1 tasks in the pipeline, resends that will take a while to clean up, that still demand big memory, and stop the python mini`s running

Good. The other problem is whether the Rosetta python tasks are written to know how to use it.
ID: 103646 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Previous · 1 . . . 139 · 140 · 141 · 142 · 143 · 144 · 145 . . . 317 · Next

Message boards : Number crunching : Problems and Technical Issues with Rosetta@home



©2025 University of Washington
https://www.bakerlab.org