As a newer user, I've never seen it under 15k in the last few days - here it is at just above 8k

I know it looks like that ought to be the tasks ready to send (because it's called 'tasks ready to send') but it's actually a pile of crap. The whole server status page is a mess tbh

Scroll down to where it says "Tasks by application". That seems to be the best reflection of where we are. Don't ask why - no-one knows.

Also note the page isn't live. I thought it updated every 15 or 30 minutes, but a time on the page indicates it hasn't been refreshed for nearly 3hrs.

Yeah, you thought this project was the height of technology and sophistication? You'll soon have that idea knocked out of you.

We wait...

---

As a newer user, I've never seen it under 15k in the last few days - here it is at just above 8k

I know it looks like that ought to be the tasks ready to send (because it's called 'tasks ready to send') but it's actually a pile of crap. The whole server status page is a mess tbh

Scroll down to where it says "Tasks by application". That seems to be the best reflection of where we are. Don't ask why - no-one knows.

Also note the page isn't live. I thought it updated every 15 or 30 minutes, but a time on the page indicates it hasn't been refreshed for nearly 3hrs.

Yeah, you thought this project was the height of technology and sophistication? You'll soon have that idea knocked out of you.

We wait...

well then. That's fun.

Well, I'm glad I have the 5 tasks I do. Thanks for pointing that out - a complete noob moment there, on my part.

I was wondering when they would hit the wall. The growth has been phenomenal. They were doing 150,000 GigaFLOPS before the virus hit, and 570,318 GigaFLOPS now.
Maybe they are adding more capacity and had to take the servers down?

I have enough work for a couple of days, but if they would communicate the status to us, it would help, especially for new users who might gravitate elsewhere.
It is a good (actually great) project, and in fact they invented the field they are in, but their public communication skills are less than some other projects I have seen.

---

I was wondering when they would hit the wall. The growth has been phenomenal. They were doing 150,000 GigaFLOPS before the virus hit, and 570,318 GigaFLOPS now.
Maybe they are adding more capacity and had to take the servers down?

I have enough work for a couple of days, but if they would communicate the status to us, it would help, especially for new users who might gravitate elsewhere.
It is a good (actually great) project, and in fact they invented the field they are in, but their public communication skills are less than some other projects I have seen.

I'll be out of work in approx. 8 hours, we'll see what happens.

This growth is really impressive - nearly quadrupling the computer power they've got access to... wowza.

I was wondering when they would hit the wall. The growth has been phenomenal. They were doing 150,000 GigaFLOPS before the virus hit, and 570,318 GigaFLOPS now.
Maybe they are adding more capacity and had to take the servers down?

This seems like as good a time as any to build out more capacity.

Not only has the coronavirus spiked volunteer interest with our R@h project, but it is also spiking a lot of interest towards R@h within the lab so the communication should definitely get better. A lot has happened in a short amount of time.

1. As mentioned before we spiked ~4x or more in computing in the last week. So far our infrastructure has been handling this okay but we will definitely look into expansion and optimization as we expect many more volunteers to join in the near future. Also, our bandwidth has maxed out on a few occasions but has been okay since.

2. A number of outside groups have approached us to help.

a. The ARM developer community came together and helped port Rosetta for aarch64. The application is being tested on Ralph@home currently and will be released soon along with an announcement.
b. We are in communication with Microsoft to help us integrate and support Rosetta BOINC builds into the Rosetta Commons build and testing infrastructure and possibly with other developments.
c. We are in communication with Charity Engine to possibly use their infrastructure to run GPU machine learning based jobs and to help integrate such capability to R@h.

3. We have plenty of COVID-19 jobs but there may be a pause from time to time to give us a chance to prepare them based on previous results and new targets/methods etc.
4. We hope to update the Rosetta apps that are currently being tested on Ralph soon and plan to do another update in the near future, so 2 application updates are lined up for the near future.
5. With these updates, we hope to recruit more researchers within the lab to use this resource, but in the immediate future, priority will of course be towards COVID-19, specifically for the design of scaffolds and binders to COVID-19 targets as possible therapeutics.

Thanks Everyone!

---

Not only has the coronavirus spiked volunteer interest with our R@h project, but it is also spiking a lot of interest towards R@h within the lab so the communication should definitely get better. A lot has happened in a short amount of time.

1. As mentioned before we spiked ~4x or more in computing in the last week. So far our infrastructure has been handling this okay but we will definitely look into expansion and optimization as we expect many more volunteers to join in the near future. Also, our bandwidth has maxed out on a few occasions but has been okay since.

2. A number of outside groups have approached us to help.

a. The ARM developer community came together and helped port Rosetta for aarch64. The application is being tested on Ralph@home currently and will be released soon along with an announcement.
b. We are in communication with Microsoft to help us integrate and support Rosetta BOINC builds into the Rosetta Commons build and testing infrastructure and possibly with other developments.
c. We are in communication with Charity Engine to possibly use their infrastructure to run GPU machine learning based jobs and to help integrate such capability to R@h.

3. We have plenty of COVID-19 jobs but there may be a pause from time to time to give us a chance to prepare them based on previous results and new targets/methods etc.
4. We hope to update the Rosetta apps that are currently being tested on Ralph soon and plan to do another update in the near future, so 2 application updates are lined up for the near future.
5. With these updates, we hope to recruit more researchers within the lab to use this resource, but in the immediate future, priority will of course be towards COVID-19, specifically for the design of scaffolds and binders to COVID-19 targets as possible therapeutics.

Thanks Everyone!

Thanks for updating us. That's fantastic, and I think it's fair to say that we all look forward to the good that researchers can do with this data. Keep up the good work.

If the well is running dry of new tasks, but millions are shown to be currently out in the wild, I think this would indicate that many users have quite large queues. (also evident from some comments I've seen stating folks have a couple days worth of work) With such high compute capacity at the moment, I wonder if the admins would consider limiting, somewhat, the amount of work one client can queue, and that might actually result in getting all work units completed a couple days sooner.
It might be interesting to see how all the active clients for the past few days would distributed across a chart showing them based upon how often they contact the servers, to get a better idea of how many might actually be operating in a semi-offline batch mode, or if they're all working the the default just-in-time configuration, where there's little point in queuing a large backlog of work.

If the well is running dry of new tasks, but millions are shown to be currently out in the wild, I think this would indicate that many users have quite large queues. (also evident from some comments I've seen stating folks have a couple days worth of work) With such high compute capacity at the moment, I wonder if the admins would consider limiting, somewhat, the amount of work one client can queue, and that might actually result in getting all work units completed a couple days sooner.
It might be interesting to see how all the active clients for the past few days would distributed across a chart showing them based upon how often they contact the servers, to get a better idea of how many might actually be operating in a semi-offline batch mode, or if they're all working the the default just-in-time configuration, where there's little point in queuing a large backlog of work.

We'll have to wait and see. A very interesting thing to consider, though. Sparked my curiousity now, as well!

Currently, I have 14 machines with 80+ cores and 14+ GPUs. When I switched over to run one of them to Rosetta, from cancer research (WCG), I did NOT receive anything related to covid19. Whenever I would receive ALL WUs for covid19, I will switch over... Till then I will stick with cancer research. Let me know when covid19 is the priority here. I am NOT going to download the fold@home app. If it is not thru BOINC, I do not run it.
James

There has been a discussion that Folding@home is likely to produce a BOINC version soon, Not ready yet, though.

Currently, I have 14 machines with 80+ cores and 14+ GPUs. When I switched over to run one of them to Rosetta, from cancer research (WCG), I did NOT receive anything related to covid19. Whenever I would receive ALL WUs for covid19, I will switch over... Till then I will stick with cancer research. Let me know when covid19 is the priority here. I am NOT going to download the fold@home app. If it is not thru BOINC, I do not run it.
James

There has been a discussion that Folding@home is likely to produce a BOINC version soon, Not ready yet, though.

Even if so, that would still take ages to finish, as they appear to not have even started. And, considering how hard it was to attempt to get Rosetta to run on GPUs - which it still can't, despite their best efforts - it would be months away at best.

We are a not for profit academic institution and not a technology company. R@h is based on BOINC which is also a project run by an academic institution.

As an academic research lab our priority is in protein design and structure prediction research. You hopefully can judge our success by the peer reviewed publications and impact from new promising technologies based on our research. Such information is regularly updated on our News.

The BOINC server status page is open source. We are not running the latest version of BOINC however.

The tasks ready to send are indeed the tasks ready to send by our feeder daemons. We try to keep this buffered to at least 15,000 or so whereas the actual queue that researchers submit to is not part of BOINC and is not shown to users other than the "Total queued jobs" on the R@h homepage. When this "tasks ready to send" buffer starts to reduce as it has been mentioned by the attentive participant, that means our internal queue is also low. This may happen from time to time as mentioned in my last post. These stats do not get updated frequently. The server status should be updated hourly so there may be a slowdown gathering the data as you note. The homepage status should be updated every 4 hours. Hope this explains things.

---

Very good point. We can consider reducing the "delay_bound" for important time critical jobs.

---

The tasks ready to send are indeed the tasks ready to send by our feeder daemons. We try to keep this buffered to at least 15,000 or so whereas the actual queue that researchers submit to is not part of BOINC and is not shown to users other than the "Total queued jobs" on the R@h homepage. When this "tasks ready to send" buffer starts to reduce as it has been mentioned by the attentive participant, that means our internal queue is also low. This may happen from time to time as mentioned in my last post. These stats do not get updated frequently. The server status should be updated hourly so there may be a slowdown gathering the data as you note. The homepage status should be updated every 4 hours. Hope this explains things.

Some projects do not even utilize the stats page - so I'm glad that you grant us access to what numbers you do, for our convenience.

Thanks for clarifying this for us, and for your patience as you deal with a huge influx of processing power.

Not only has the coronavirus spiked volunteer interest with our R@h project, but it is also spiking a lot of interest towards R@h within the lab so the communication should definitely get better. A lot has happened in a short amount of time.

1. As mentioned before we spiked ~4x or more in computing in the last week. So far our infrastructure has been handling this okay but we will definitely look into expansion and optimization as we expect many more volunteers to join in the near future. Also, our bandwidth has maxed out on a few occasions but has been okay since.

2. A number of outside groups have approached us to help.

a. The ARM developer community came together and helped port Rosetta for aarch64. The application is being tested on Ralph@home currently and will be released soon along with an announcement.
b. We are in communication with Microsoft to help us integrate and support Rosetta BOINC builds into the Rosetta Commons build and testing infrastructure and possibly with other developments.
c. We are in communication with Charity Engine to possibly use their infrastructure to run GPU machine learning based jobs and to help integrate such capability to R@h.

3. We have plenty of COVID-19 jobs but there may be a pause from time to time to give us a chance to prepare them based on previous results and new targets/methods etc.
4. We hope to update the Rosetta apps that are currently being tested on Ralph soon and plan to do another update in the near future, so 2 application updates are lined up for the near future.
5. With these updates, we hope to recruit more researchers within the lab to use this resource, but in the immediate future, priority will of course be towards COVID-19, specifically for the design of scaffolds and binders to COVID-19 targets as possible therapeutics.

Thanks Everyone!

Please post this on the homepage! Very informative.

Thanks for the update!

---

I'm not getting new tasks since yesterday.
Is it something problem on my machine?
Or... does it mean that R@H is getting exploded with new participants?
If the latter is the case, I'm very happy to see my malfunctioning client.

I'm not getting new tasks since yesterday.
Is it something problem on my machine?
Or... does it mean that R@H is getting exploded with new participants?
If the latter is the case, I'm very happy to see my malfunctioning client.

Nothing wrong with your machine. Everyone is in the same boat. Their processing capabilities have quadrupled over the last week.

It is good to see such interest in this project that we can actually deplete the queue. My WU buffers are only set for 2 hours (instead of days) so when I run out of WUs, I just have another project that will pick up the slack until there is work available. I just set the priority of this project 100 times greater than the secondary project, and my computers will always stay running, but with this project as my main priority.
Please keep us posted on if and when GPU tasks become available. I have about 14 GPUs that would love to run this project as well.

Thanks,
James

---

I was surprised when I saw a comment from someone who had queue for a couple of days. I joined the project about 10 days ago and I usually got 2-3 extra tasks besides the 12 running tasks. My tasks take 7-8 hours to complete so I never got a large queue. Wonder why there's such a big difference.

Also since yesterday I didn't receive any new tasks and I'm now running the last 5 tasks.

Not only has the coronavirus spiked volunteer interest with our R@h project, but it is also spiking a lot of interest towards R@h within the lab so the communication should definitely get better.

?? IPD is not interested a lot in Rosetta@Home?

---