Sid Celery said:

As the thread title says, RAM req'ts shot up to 6675.72Mb and we've managed to reduce this to 3814.7Mb on the majority of tasks coming down now, but is this enough to allow 4Gb machines to run Rosetta tasks?
This question applies to all hosts, but only for people with 4Gb RAM machines - PC or Raspberry Pi
(Admission: I don't even know what a Raspberry Pi is, but I'm running with this anyway)

Here's a link to a search about Raspberry Pi's so you can at least know what they are. Essentially they are Arm cpu's as opposed to AMD or Intel and the current version comes with 4 cpu cores and 4 or 8gb of onboard ram. They are SLOW crunchers only running at @1.6ghz and require some tweaking to make them do exactly what you want them to do for crunching here at Rosetta but they can crunch Einstein and Universe as soon as Boinc is loaded up and running on a 16gb micro SD card. Once setup they are pretty much like any other pc people have except the cost is around $120US to be up and running with the Pi, the case, the heat sinks and all the cables except a standard usb keyboard and mouse. They even have wifi built in but there is also a 1gb plug for an internet cable if you want it that way. If you use VNC they can even be run remotely.

https://www.bing.com/search?q=raspberry+pi+4+model+b&qs=HS&pq=raspberry+pi&sk=HS1&sc=8-12&cvid=1B78618E723941D4AB49FB63380AFFC9&FORM=QBLH&sp=2

If you need more info the User PorkyPies is VERY experienced and who unbeknownst to him has helped me a lot in getting mine up and running.

I'm not entirely sure I'm any the wiser, but thanks for the link anyway.

What I'm trying to do is make Rosetta available to more than just those who've gone for the 8Gb option
So that, in the end, it's only people who've gone for the bottom of the range - the 2Gb - who can't contribute.

If I rank that RAM-availability of 4Gb machines I found, I can speculate

4060Mb - can run with current RAM demands - probably an extreme tweaker of settings
3972Mb - ditto
3700Mb - can't run with current demand for 3814.7Mb RAM even after a little effort following advice here
3653Mb - ditto
3460Mb - can't run with current demand for 3814.7Mb RAM, probably a default user with a default machine and with default Boinc settings, never been to these forums to see what can be changed to help

If I assume people report similar numbers within that max/min range, I can say
A 5% reduction in RAM req't = 3624Mb, meaning 4 out of 5 of the above 4Gb machines can run Rosetta instead of 2 out of 5
A 10% reduction in RAM req't = 3434Mb, meaning even default machines with default settings and non-tweakers (probably the vast majority) can run Rosetta

The impression I'm being given is that a further 5% reduction is doable but "it's nearly impossible for us to know RAM requirements for all jobs since they can be so different from the various researchers" so going further may cause different kinds of problems at the project itself - eg crashes and other computation errors at the margins

---

And going back to the proxy I'm using for downloadability - In Progress tasks
Pre increase in RAM & Disk req'ts - 550k IP
Extreme RAM & Disk demands - 318k IP
Current In Progress - 407k - 26% below max, 28% above min - continuing improvement

---

Sid Celery said:
The impression I'm being given is that a further 5% reduction is doable but "it's nearly impossible for us to know RAM requirements for all jobs since they can be so different from the various researchers" so going further may cause different kinds of problems at the project itself - eg crashes and other computation errors at the margins

Maybe the Developers could introduce a choice of which task to run then with the lower memory machines just not being able to contribute at the upper end of the ram requirements but still being able to do so at the lower end.

Oh and you are very welcome about the Raspberry Pi's, they are fun, expensive and seem to work great.

Sid Celery said:
The impression I'm being given is that a further 5% reduction is doable but "it's nearly impossible for us to know RAM requirements for all jobs since they can be so different from the various researchers" so going further may cause different kinds of problems at the project itself - eg crashes and other computation errors at the margins

Maybe the Developers could introduce a choice of which task to run then with the lower memory machines just not being able to contribute at the upper end of the ram requirements but still being able to do so at the lower end.

Oh and you are very welcome about the Raspberry Pi's, they are fun, expensive and seem to work great.

I think you meant to write "inexpensive" - lol

You make a good point about choosing to run high or low resource tasks, but it makes me think of something I wrote earlier

I've got various World Community Grid tasks here and noted what they require, if it's of use to anyone

Open Pandemics (CV19) - RAM 240Mb, Disk - 96Mb
Mapping Cancer Markers - RAM 400Mb, Disk - 500Mb
Microbiome Immunity Project - RAM 650Mb, Disk - 250Mb
Africa Rainfall Project - RAM 1024Mb, Disk - 1536Mb

If different settings can be applied to different sub-projects within WCGrid, maybe this is analogous to the various task-types issued to run on Rosetta.
I make the assumption each researcher knows the resource req'ts for each batch of work they issue - presumably they run a few before releasing them so they know they're doing what they want.
So, even if they can't provide one all-encompassing figure for all tasks, and rather than changing it for every batch, maybe they can just decide if their tasks fit into one of a few categories equivalent to the capacity of hosts of varying size.

That is, a PC set up at default, running Boinc also set at default, using the estimates I've observed

For 2Gb RAM hosts, set limit at 1717Mb (the old limit) will run on 100% of hosts
For 4Gb RAM hosts, set limit at 3434Mb will run on 75% of hosts
For 6Gb RAM hosts, set limit at 5150Mb will run on ??% of hosts (do such machines exist any more? I don't know)
For 8Gb RAM hosts, set limit at 6675Mb will run on 55% of hosts

Let me have another think...

---

For 6Gb RAM hosts, set limit at 5150Mb will run on ??% of hosts (do such machines exist any more? I don't know)

I've never come across a system sold with 6GB of RAM, but anything that can support 8GB or more will support 6GB, however it's a poor choice from a performance/memory bandwidth point of view.
2, 4, 6, 8 16, 32, 64, 128, 256 etc allows matched pairs to be be used, and allows 1 pair, 2 pairs, 4 pairs, 8 pairs etc to be used to give maximum memory bandwidth with that maximum amount of memory (some CPUs can support 6 pairs, to support larger amounts of RAM, but bandwidth suffers).



The RAM requirement for a Task should reflect the actual requirements for that Task.
I've had Tasks that use less than 200MB, and those that have used over 4GB. When the Work Unit is created, the RAM needed to process any Tasks sent out from it needs to be an accurate value of the Task's requirements.

The present problem is a Disk & RAM setting for all Tasks, regardless of how much they actually need- and that is well in excess of what they actually require (my understanding is that value was actually changed back to a lower value- however it will take time for Tasks with the problematical larger values to be cleared out- unless the project runs a script to amend the values in question).

---

Grant
Darwin NT

Oh and you are very welcome about the Raspberry Pi's, they are fun, expensive and seem to work great.

I think you meant to write "inexpensive" - lol

You are correct!!

You make a good point about choosing to run high or low resource tasks, but it makes me think of something I wrote earlier

I've got various World Community Grid tasks here and noted what they require, if it's of use to anyone

Open Pandemics (CV19) - RAM 240Mb, Disk - 96Mb
Mapping Cancer Markers - RAM 400Mb, Disk - 500Mb
Microbiome Immunity Project - RAM 650Mb, Disk - 250Mb
Africa Rainfall Project - RAM 1024Mb, Disk - 1536Mb

If different settings can be applied to different sub-projects within WCGrid, maybe this is analogous to the various task-types issued to run on Rosetta.
I make the assumption each researcher knows the resource req'ts for each batch of work they issue - presumably they run a few before releasing them so they know they're doing what they want.
So, even if they can't provide one all-encompassing figure for all tasks, and rather than changing it for every batch, maybe they can just decide if their tasks fit into one of a few categories equivalent to the capacity of hosts of varying size.

That is, a PC set up at default, running Boinc also set at default, using the estimates I've observed

For 2Gb RAM hosts, set limit at 1717Mb (the old limit) will run on 100% of hosts
For 4Gb RAM hosts, set limit at 3434Mb will run on 75% of hosts
For 6Gb RAM hosts, set limit at 5150Mb will run on ??% of hosts (do such machines exist any more? I don't know)
For 8Gb RAM hosts, set limit at 6675Mb will run on 55% of hosts

Let me have another think...

Exactly and great idea!! Then they could make the tasks available as they can, sometimes there won't be any 2gb and 4gb tasks but there could be 6gb and 8gb tasks available, or vice versa or any combination of the above as the research progresses.

What is the exact free RAM figure you guys have available to Boinc?

TIA

I would think 3.2Gb would be the ideal. They take up to 800MB when running so 4 x 800 = 3.2Gb. They don’t all use that, but I based it on the largest ones going through at the moment.

---

MarksRpiCluster

For 6Gb RAM hosts, set limit at 5150Mb will run on ??% of hosts (do such machines exist any more? I don't know)

I've never come across a system sold with 6GB of RAM, but anything that can support 8GB or more will support 6GB, however it's a poor choice from a performance/memory bandwidth point of view.
2, 4, 6, 8 16, 32, 64, 128, 256 etc allows matched pairs to be be used, and allows 1 pair, 2 pairs, 4 pairs, 8 pairs etc to be used to give maximum memory bandwidth with that maximum amount of memory (some CPUs can support 6 pairs, to support larger amounts of RAM, but bandwidth suffers).

You're making me think I'm imagining it, but I'm sure at one time there were PCs with 3 RAM slots that took 3 sticks giving 3Gb or 6Gb RAM, though admittedly not for a very long time

The RAM requirement for a Task should reflect the actual requirements for that Task.
I've had Tasks that use less than 200MB, and those that have used over 4GB. When the Work Unit is created, the RAM needed to process any Tasks sent out from it needs to be an accurate value of the Task's requirements.

The present problem is a Disk & RAM setting for all Tasks, regardless of how much they actually need- and that is well in excess of what they actually require (my understanding is that value was actually changed back to a lower value- however it will take time for Tasks with the problematical larger values to be cleared out- unless the project runs a script to amend the values in question).

I agree with what you're saying - I think I may've written that part badly to give the wrong impression.

The problem is, the project seems inclined to demand a single (huge) amount of resources to cover all their eventualities (present and future) for all tasks, blind to the knock-on effects to Users/Hosts and their ability to contribute at all.
For those with the resources, great. For those without, tough. Find somewhere else to contribute. Even when the loss of hosts was as much as 45%
Not only is that harsh, but counter-productive, not only for the loss of capacity, but for the loss of goodwill.

That's why this particular topic exists and also why I'm trying to broach a compromise that serves everyone as best I can

---

What is the exact free RAM figure you guys have available to Boinc?

TIA

I would think 3.2Gb would be the ideal. They take up to 800MB when running so 4 x 800 = 3.2Gb. They don’t all use that, but I based it on the largest ones going through at the moment.

Thanks, that sounds generally right.
We may not get all the way there (likely not from the impression I get) but every bit towards it helps

---

Let me have another think...

Exactly and great idea!! Then they could make the tasks available as they can, sometimes there won't be any 2gb and 4gb tasks but there could be 6gb and 8gb tasks available, or vice versa or any combination of the above as the research progresses.

I've asked the question and I didn't get a no.
What kind of a yes I get is out of my hands. Let's see.
Every bit helps more people contribute to the extent they can.

---

And going back to the proxy I'm using for downloadability - In Progress tasks
Pre increase in RAM & Disk req'ts - 550k IP
Extreme RAM & Disk demands - 318k IP - 42% down
Current In Progress - 407k - 26% below max, 28% above min - continuing improvement

Current In Progress - 423,900 - 23% below max, 33% above min

---

And going back to the proxy I'm using for downloadability - In Progress tasks
Pre increase in RAM & Disk req'ts - 550k IP
Extreme RAM & Disk demands - 318k IP - 42% down
Current In Progress - 407k - 26% below max, 28% above min - continuing improvement

Current In Progress - 423,900 - 23% below max, 33% above min

That's very good!!

As an aside I am now running 3 units at a time on my 8gb RPi and they are all validating, not much but more than a few days ago.
Thank you PorkyPies and a couple of magazines and the internet for getting me up and running and using zram to make things even faster.

And going back to the proxy I'm using for downloadability - In Progress tasks
Pre increase in RAM & Disk req'ts - 550k IP
Extreme RAM & Disk demands - 318k IP - 42% down
Current In Progress - 407k - 26% below max, 28% above min - continuing improvement

Current In Progress - 423,900 - 23% below max, 33% above min

That's very good!!

As an aside I am now running 3 units at a time on my 8gb RPi and they are all validating, not much but more than a few days ago.
Thank you PorkyPies and a couple of magazines and the internet for getting me up and running and using zram to make things even faster.

Most of the the crashes have stopped (pre-helical ones still working their way through), all the failed uploads have stopped and we can largely get a reasonable supply of tasks without being forced to jump through hoops, with the promise of a little more to come. Plus, finally, a little bit of feedback that lets us know something really is happening rather than just hoping it is. And the forums have quietened down as a result.
We'll have lost some hosts, inevitably, but we kind of know why and it's for a positive reason.
It's been a successful week, which is just as well as I'm away for work again in a few more hours.

I know I've been one of the worst to complain about forum whining in the past, but recent events had the potential for losing a lot of goodwill and reputational damage if it continued that 40% of tasks/hosts were going to be discarded without a word. Crashes, non-availability, lack of response, no fault-resolution - all together is a perfect storm, so I'm glad I took some time for the benefit of lots of people.

And current In Progress is now 431k, so still creeping up even before further changes come through

---

some new messages from the server till the 30th of April ...

Before :

29-Apr-2021 09:16:56 [Rosetta@home] Scheduler request completed: got 0 new tasks
29-Apr-2021 09:16:56 [Rosetta@home] No tasks sent
29-Apr-2021 09:16:56 [Rosetta@home] Rosetta needs 6675.72 MB RAM but only 2012.49 MB is available for use.
29-Apr-2021 09:16:56 [Rosetta@home] Rosetta needs 3814.70 MB RAM but only 2012.49 MB is available for use.

Now :

30-Apr-2021 09:07:57 [Rosetta@home] Requesting new tasks for CPU
30-Apr-2021 09:08:00 [Rosetta@home] Scheduler request completed: got 0 new tasks
30-Apr-2021 09:08:00 [Rosetta@home] No tasks sent
30-Apr-2021 09:08:00 [Rosetta@home] Rosetta needs 6675.72 MB RAM but only 2012.49 MB is available for use.
30-Apr-2021 09:08:00 [Rosetta@home] Rosetta needs 3814.70 MB RAM but only 2012.49 MB is available for use.
30-Apr-2021 09:08:00 [Rosetta@home] Rosetta needs 3337.86 MB RAM but only 2012.49 MB is available for use.

no worries because > 01-May-2021 15:07:28 [Rosetta@home] Not requesting tasks: don't need (job cache full)

some new messages from the server till the 30th of April ...

Before :

29-Apr-2021 09:16:56 [Rosetta@home] Scheduler request completed: got 0 new tasks
29-Apr-2021 09:16:56 [Rosetta@home] No tasks sent
29-Apr-2021 09:16:56 [Rosetta@home] Rosetta needs 6675.72 MB RAM but only 2012.49 MB is available for use.
29-Apr-2021 09:16:56 [Rosetta@home] Rosetta needs 3814.70 MB RAM but only 2012.49 MB is available for use.

Now :

30-Apr-2021 09:07:57 [Rosetta@home] Requesting new tasks for CPU
30-Apr-2021 09:08:00 [Rosetta@home] Scheduler request completed: got 0 new tasks
30-Apr-2021 09:08:00 [Rosetta@home] No tasks sent
30-Apr-2021 09:08:00 [Rosetta@home] Rosetta needs 6675.72 MB RAM but only 2012.49 MB is available for use.
30-Apr-2021 09:08:00 [Rosetta@home] Rosetta needs 3814.70 MB RAM but only 2012.49 MB is available for use.
30-Apr-2021 09:08:00 [Rosetta@home] Rosetta needs 3337.86 MB RAM but only 2012.49 MB is available for use.

no worries because > 01-May-2021 15:07:28 [Rosetta@home] Not requesting tasks: don't need (job cache full)

Thanks for this information.
I expected at least a 5% reduction in RAM demands and hoped for 10% and it looks like they've delivered a 12.5% reduction.
In terms of what I suggested in order to make 4Gb RAM machines viable again at Rosetta, they've delivered.

Separately, I've proposed a way of going back to the pre-April RAM & Disk req'ts on <some> tasks so that 2Gb machines can participate again, but because it requires the researchers to do something they haven't had to do before I haven't had any feedback on that yet. Several people here have quite rightly reported that the maximum resources that tasks call on while they're actually running never reaches 1Gb, let alone 2Gb so it ought to be quite possible.

To be honest I'm tiptoeing my way toward something they don't want to do, but they're throwing away so much resource I'm still hoping to convince them of the plain rationality of it.

---

Rosetta@home: Notice from server
Rosetta needs 3814.70 MB RAM but only 1879.68 MB is available for use.
5/1/2021 7:29:03 PM jetson-nano2GB

---

Host 6057423

Mon May 03 12:09:16 CDT 2021|Rosetta@home|Message from server: Rosetta needs 1386.40MB more disk space. You currently have 2428.29 MB available and it needs 3814.70 MB.
Mon May 03 12:09:16 CDT 2021|Rosetta@home|No tasks sent

It's already working on a task that's almost finished (24 hour run time selected), so there are files already present....

---

Host 6057423

Mon May 03 12:09:16 CDT 2021|Rosetta@home|Message from server: Rosetta needs 1386.40MB more disk space. You currently have 2428.29 MB available and it needs 3814.70 MB.
Mon May 03 12:09:16 CDT 2021|Rosetta@home|No tasks sent

It's already working on a task that's almost finished (24 hour run time selected), so there are files already present....

Its complaining about disk space. the host is running Android. Is it a phone by any chance? Can you free up some space (delete something). If its a phone I don't expect there is much you can do to get more space. Maybe once the current task finishes it will free up some space.

---

MarksRpiCluster

Host 6057423

Mon May 03 12:09:16 CDT 2021|Rosetta@home|Message from server: Rosetta needs 1386.40MB more disk space. You currently have 2428.29 MB available and it needs 3814.70 MB.
Mon May 03 12:09:16 CDT 2021|Rosetta@home|No tasks sent

It's already working on a task that's almost finished (24 hour run time selected), so there are files already present....

Its complaining about disk space. the host is running Android. Is it a phone by any chance? Can you free up some space (delete something). If its a phone I don't expect there is much you can do to get more space. Maybe once the current task finishes it will free up some space.

Or increase the amount BOINC can use. It won't actually need it, but it thinks it does...
My 6 core/12 thread system with an 8 hour cache and has several older versions of Rosetta & Mini Rosetta on it has never used more than 2.5GB.

---

Grant
Darwin NT

Indeed Windows and most Linux flavors allocate a swap file the same size as physical memory. For some reason, probably historical, the Raspberry Pi foundation allocate 100MB to this day. Increasing it should allow it to download tasks although I haven't tried it.

Not sure about Linux, but on Windows BOINC count only real RAM and it does not matter how many swap space you allocate. It wont help with this abnormal R@H RAM requirements.
I still (for more than month already) see errors like

05-May-2021 07:48:46 [Rosetta@home] Scheduler request completed: got 0 new tasks
05-May-2021 07:48:46 [Rosetta@home] No tasks sent
05-May-2021 07:48:46 [Rosetta@home] Rosetta needs 6675.72 MB RAM but only 6117.47 MB is available for use.

On computers with 8 GB of RAM + 8 GB of swap space.
And after such task finally downloaded they usually use less < 1 GB of RAM per task. And computer run up to 4-8 R@H tasks simultaneously without any problems.
But last month usually can not get any because server thinks that there is not enough RAM for just one task and refuse to send any work.

Pure stupidity.