Message boards : Number crunching : Rosetta needs 6675.72 MB RAM: is the restriction really needed?
Previous · 1 · 2 · 3 · 4 · 5 · 6 · Next
Author | Message |
---|---|
Sid Celery Send message Joined: 11 Feb 08 Posts: 2121 Credit: 41,179,074 RAC: 11,480 |
Sid Celery said: I'm not entirely sure I'm any the wiser, but thanks for the link anyway. What I'm trying to do is make Rosetta available to more than just those who've gone for the 8Gb option So that, in the end, it's only people who've gone for the bottom of the range - the 2Gb - who can't contribute. If I rank that RAM-availability of 4Gb machines I found, I can speculate 4060Mb - can run with current RAM demands - probably an extreme tweaker of settings 3972Mb - ditto 3700Mb - can't run with current demand for 3814.7Mb RAM even after a little effort following advice here 3653Mb - ditto 3460Mb - can't run with current demand for 3814.7Mb RAM, probably a default user with a default machine and with default Boinc settings, never been to these forums to see what can be changed to help If I assume people report similar numbers within that max/min range, I can say A 5% reduction in RAM req't = 3624Mb, meaning 4 out of 5 of the above 4Gb machines can run Rosetta instead of 2 out of 5 A 10% reduction in RAM req't = 3434Mb, meaning even default machines with default settings and non-tweakers (probably the vast majority) can run Rosetta The impression I'm being given is that a further 5% reduction is doable but "it's nearly impossible for us to know RAM requirements for all jobs since they can be so different from the various researchers" so going further may cause different kinds of problems at the project itself - eg crashes and other computation errors at the margins |
Sid Celery Send message Joined: 11 Feb 08 Posts: 2121 Credit: 41,179,074 RAC: 11,480 |
And going back to the proxy I'm using for downloadability - In Progress tasks Pre increase in RAM & Disk req'ts - 550k IP Extreme RAM & Disk demands - 318k IP Current In Progress - 407k - 26% below max, 28% above min - continuing improvement |
mikey Send message Joined: 5 Jan 06 Posts: 1895 Credit: 9,154,260 RAC: 4,107 |
Sid Celery said: Maybe the Developers could introduce a choice of which task to run then with the lower memory machines just not being able to contribute at the upper end of the ram requirements but still being able to do so at the lower end. Oh and you are very welcome about the Raspberry Pi's, they are fun, expensive and seem to work great. |
Sid Celery Send message Joined: 11 Feb 08 Posts: 2121 Credit: 41,179,074 RAC: 11,480 |
Sid Celery said: I think you meant to write "inexpensive" - lol You make a good point about choosing to run high or low resource tasks, but it makes me think of something I wrote earlier I've got various World Community Grid tasks here and noted what they require, if it's of use to anyone If different settings can be applied to different sub-projects within WCGrid, maybe this is analogous to the various task-types issued to run on Rosetta. I make the assumption each researcher knows the resource req'ts for each batch of work they issue - presumably they run a few before releasing them so they know they're doing what they want. So, even if they can't provide one all-encompassing figure for all tasks, and rather than changing it for every batch, maybe they can just decide if their tasks fit into one of a few categories equivalent to the capacity of hosts of varying size. That is, a PC set up at default, running Boinc also set at default, using the estimates I've observed For 2Gb RAM hosts, set limit at 1717Mb (the old limit) will run on 100% of hosts For 4Gb RAM hosts, set limit at 3434Mb will run on 75% of hosts For 6Gb RAM hosts, set limit at 5150Mb will run on ??% of hosts (do such machines exist any more? I don't know) For 8Gb RAM hosts, set limit at 6675Mb will run on 55% of hosts Let me have another think... |
Grant (SSSF) Send message Joined: 28 Mar 20 Posts: 1677 Credit: 17,755,516 RAC: 22,864 |
For 6Gb RAM hosts, set limit at 5150Mb will run on ??% of hosts (do such machines exist any more? I don't know)I've never come across a system sold with 6GB of RAM, but anything that can support 8GB or more will support 6GB, however it's a poor choice from a performance/memory bandwidth point of view. 2, 4, 6, 8 16, 32, 64, 128, 256 etc allows matched pairs to be be used, and allows 1 pair, 2 pairs, 4 pairs, 8 pairs etc to be used to give maximum memory bandwidth with that maximum amount of memory (some CPUs can support 6 pairs, to support larger amounts of RAM, but bandwidth suffers). The RAM requirement for a Task should reflect the actual requirements for that Task. I've had Tasks that use less than 200MB, and those that have used over 4GB. When the Work Unit is created, the RAM needed to process any Tasks sent out from it needs to be an accurate value of the Task's requirements. The present problem is a Disk & RAM setting for all Tasks, regardless of how much they actually need- and that is well in excess of what they actually require (my understanding is that value was actually changed back to a lower value- however it will take time for Tasks with the problematical larger values to be cleared out- unless the project runs a script to amend the values in question). Grant Darwin NT |
mikey Send message Joined: 5 Jan 06 Posts: 1895 Credit: 9,154,260 RAC: 4,107 |
You are correct!! You make a good point about choosing to run high or low resource tasks, but it makes me think of something I wrote earlier Exactly and great idea!! Then they could make the tasks available as they can, sometimes there won't be any 2gb and 4gb tasks but there could be 6gb and 8gb tasks available, or vice versa or any combination of the above as the research progresses. |
PorkyPies Send message Joined: 6 Apr 20 Posts: 45 Credit: 1,650,779 RAC: 0 |
What is the exact free RAM figure you guys have available to Boinc? I would think 3.2Gb would be the ideal. They take up to 800MB when running so 4 x 800 = 3.2Gb. They don’t all use that, but I based it on the largest ones going through at the moment. MarksRpiCluster |
Sid Celery Send message Joined: 11 Feb 08 Posts: 2121 Credit: 41,179,074 RAC: 11,480 |
For 6Gb RAM hosts, set limit at 5150Mb will run on ??% of hosts (do such machines exist any more? I don't know)I've never come across a system sold with 6GB of RAM, but anything that can support 8GB or more will support 6GB, however it's a poor choice from a performance/memory bandwidth point of view. You're making me think I'm imagining it, but I'm sure at one time there were PCs with 3 RAM slots that took 3 sticks giving 3Gb or 6Gb RAM, though admittedly not for a very long time The RAM requirement for a Task should reflect the actual requirements for that Task. I agree with what you're saying - I think I may've written that part badly to give the wrong impression. The problem is, the project seems inclined to demand a single (huge) amount of resources to cover all their eventualities (present and future) for all tasks, blind to the knock-on effects to Users/Hosts and their ability to contribute at all. For those with the resources, great. For those without, tough. Find somewhere else to contribute. Even when the loss of hosts was as much as 45% Not only is that harsh, but counter-productive, not only for the loss of capacity, but for the loss of goodwill. That's why this particular topic exists and also why I'm trying to broach a compromise that serves everyone as best I can |
Sid Celery Send message Joined: 11 Feb 08 Posts: 2121 Credit: 41,179,074 RAC: 11,480 |
What is the exact free RAM figure you guys have available to Boinc? Thanks, that sounds generally right. We may not get all the way there (likely not from the impression I get) but every bit towards it helps |
Sid Celery Send message Joined: 11 Feb 08 Posts: 2121 Credit: 41,179,074 RAC: 11,480 |
Let me have another think... I've asked the question and I didn't get a no. What kind of a yes I get is out of my hands. Let's see. Every bit helps more people contribute to the extent they can. |
Sid Celery Send message Joined: 11 Feb 08 Posts: 2121 Credit: 41,179,074 RAC: 11,480 |
And going back to the proxy I'm using for downloadability - In Progress tasks Current In Progress - 423,900 - 23% below max, 33% above min |
mikey Send message Joined: 5 Jan 06 Posts: 1895 Credit: 9,154,260 RAC: 4,107 |
And going back to the proxy I'm using for downloadability - In Progress tasks That's very good!! As an aside I am now running 3 units at a time on my 8gb RPi and they are all validating, not much but more than a few days ago. Thank you PorkyPies and a couple of magazines and the internet for getting me up and running and using zram to make things even faster. |
Sid Celery Send message Joined: 11 Feb 08 Posts: 2121 Credit: 41,179,074 RAC: 11,480 |
And going back to the proxy I'm using for downloadability - In Progress tasks Most of the the crashes have stopped (pre-helical ones still working their way through), all the failed uploads have stopped and we can largely get a reasonable supply of tasks without being forced to jump through hoops, with the promise of a little more to come. Plus, finally, a little bit of feedback that lets us know something really is happening rather than just hoping it is. And the forums have quietened down as a result. We'll have lost some hosts, inevitably, but we kind of know why and it's for a positive reason. It's been a successful week, which is just as well as I'm away for work again in a few more hours. I know I've been one of the worst to complain about forum whining in the past, but recent events had the potential for losing a lot of goodwill and reputational damage if it continued that 40% of tasks/hosts were going to be discarded without a word. Crashes, non-availability, lack of response, no fault-resolution - all together is a perfect storm, so I'm glad I took some time for the benefit of lots of people. And current In Progress is now 431k, so still creeping up even before further changes come through |
Kissagogo27 Send message Joined: 31 Mar 20 Posts: 86 Credit: 2,907,406 RAC: 2,505 |
some new messages from the server till the 30th of April ... Before :
Now :
no worries because > 01-May-2021 15:07:28 [Rosetta@home] Not requesting tasks: don't need (job cache full) |
Sid Celery Send message Joined: 11 Feb 08 Posts: 2121 Credit: 41,179,074 RAC: 11,480 |
some new messages from the server till the 30th of April ... Thanks for this information. I expected at least a 5% reduction in RAM demands and hoped for 10% and it looks like they've delivered a 12.5% reduction. In terms of what I suggested in order to make 4Gb RAM machines viable again at Rosetta, they've delivered. Separately, I've proposed a way of going back to the pre-April RAM & Disk req'ts on <some> tasks so that 2Gb machines can participate again, but because it requires the researchers to do something they haven't had to do before I haven't had any feedback on that yet. Several people here have quite rightly reported that the maximum resources that tasks call on while they're actually running never reaches 1Gb, let alone 2Gb so it ought to be quite possible. To be honest I'm tiptoeing my way toward something they don't want to do, but they're throwing away so much resource I'm still hoping to convince them of the plain rationality of it. |
Dirk Broer Send message Joined: 16 Nov 05 Posts: 22 Credit: 3,332,536 RAC: 1,059 |
Rosetta@home: Notice from server Rosetta needs 3814.70 MB RAM but only 1879.68 MB is available for use. 5/1/2021 7:29:03 PM jetson-nano2GB |
JLDun Send message Joined: 31 May 08 Posts: 7 Credit: 68,063 RAC: 447 |
Host 6057423 Mon May 03 12:09:16 CDT 2021|Rosetta@home|Message from server: Rosetta needs 1386.40MB more disk space. You currently have 2428.29 MB available and it needs 3814.70 MB. It's already working on a task that's almost finished (24 hour run time selected), so there are files already present.... |
PorkyPies Send message Joined: 6 Apr 20 Posts: 45 Credit: 1,650,779 RAC: 0 |
Host 6057423 Its complaining about disk space. the host is running Android. Is it a phone by any chance? Can you free up some space (delete something). If its a phone I don't expect there is much you can do to get more space. Maybe once the current task finishes it will free up some space. MarksRpiCluster |
Grant (SSSF) Send message Joined: 28 Mar 20 Posts: 1677 Credit: 17,755,516 RAC: 22,864 |
Or increase the amount BOINC can use. It won't actually need it, but it thinks it does...Host 6057423 My 6 core/12 thread system with an 8 hour cache and has several older versions of Rosetta & Mini Rosetta on it has never used more than 2.5GB. Grant Darwin NT |
Mad_Max Send message Joined: 31 Dec 09 Posts: 209 Credit: 25,891,138 RAC: 10,695 |
Not sure about Linux, but on Windows BOINC count only real RAM and it does not matter how many swap space you allocate. It wont help with this abnormal R@H RAM requirements. I still (for more than month already) see errors like 05-May-2021 07:48:46 [Rosetta@home] Scheduler request completed: got 0 new tasksOn computers with 8 GB of RAM + 8 GB of swap space. And after such task finally downloaded they usually use less < 1 GB of RAM per task. And computer run up to 4-8 R@H tasks simultaneously without any problems. But last month usually can not get any because server thinks that there is not enough RAM for just one task and refuse to send any work. Pure stupidity. |
Message boards :
Number crunching :
Rosetta needs 6675.72 MB RAM: is the restriction really needed?
©2024 University of Washington
https://www.bakerlab.org