Excessive workunit fetch

Message boards : Number crunching : Excessive workunit fetch

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
skydivingnerd

Send message
Joined: 28 Jan 13
Posts: 5
Credit: 87,995,889
RAC: 384
Message 103107 - Posted: 5 Nov 2021, 1:44:21 UTC

Rosetta is fetching excessive workunits that are being aborted when they hit their Deadline time. I have my BOINC client set for only 1 day of work plus 0.1 days of additional work. However, I currently have 580ish Rosetta tasks. Work fetch appears to be broken.

Client in question:
https://boinc.bakerlab.org/rosetta/results.php?hostid=5773406&offset=0&show_names=0&state=6&appid=


11/4/2021 9:34:34 PM |  | [work_fetch] Request work fetch: Backoff ended for Rosetta@home
11/4/2021 9:34:38 PM |  | choose_project(): 1636076078.512051
11/4/2021 9:34:38 PM |  | [work_fetch] ------- start work fetch state -------
11/4/2021 9:34:38 PM |  | [work_fetch] target work buffer: 77760.00 + 8640.00 sec
11/4/2021 9:34:38 PM |  | [work_fetch] --- project states ---
11/4/2021 9:34:38 PM | Rosetta@home | [work_fetch] REC 14584.134 prio -2.285 can request work
11/4/2021 9:34:38 PM |  | [work_fetch] --- state for CPU ---
11/4/2021 9:34:38 PM |  | [work_fetch] shortfall 123996.59 nidle 0.00 saturated 65604.44 busy 56046.79
11/4/2021 9:34:38 PM | Rosetta@home | [work_fetch] share 1.000
11/4/2021 9:34:38 PM |  | [work_fetch] --- state for NVIDIA GPU ---
11/4/2021 9:34:38 PM |  | [work_fetch] shortfall 86400.00 nidle 1.00 saturated 0.00 busy 0.00
11/4/2021 9:34:38 PM | Rosetta@home | [work_fetch] share 0.000 no applications
11/4/2021 9:34:38 PM |  | [work_fetch] ------- end work fetch state -------
11/4/2021 9:34:38 PM | Rosetta@home | choose_project: scanning
11/4/2021 9:34:38 PM | Rosetta@home | can fetch CPU
11/4/2021 9:34:38 PM | Rosetta@home | CPU needs work - buffer low
11/4/2021 9:34:38 PM | Rosetta@home | checking CPU
11/4/2021 9:34:38 PM | Rosetta@home | [work_fetch] using MC shortfall 82536.591114 instead of shortfall 123996.592757
11/4/2021 9:34:38 PM | Rosetta@home | [work_fetch] set_request() for CPU: ninst 16 nused_total 596.00 nidle_now 0.00 fetch share 1.00 req_inst 0.00 req_secs 82536.59
11/4/2021 9:34:38 PM | Rosetta@home | CPU set_request: 82536.591114
11/4/2021 9:34:38 PM | Rosetta@home | checking NVIDIA GPU
11/4/2021 9:34:38 PM | Rosetta@home | NVIDIA GPU can't fetch: no applications
11/4/2021 9:34:38 PM | Rosetta@home | [work_fetch] request: CPU (82536.59 sec, 0.00 inst) NVIDIA GPU (0.00 sec, 0.00 inst)
11/4/2021 9:34:38 PM | Rosetta@home | Sending scheduler request: To fetch work.
11/4/2021 9:34:38 PM | Rosetta@home | Requesting new tasks for CPU
11/4/2021 9:34:40 PM | Rosetta@home | Scheduler request completed: got 3 new tasks
11/4/2021 9:34:40 PM | Rosetta@home | Project requested delay of 31 seconds
11/4/2021 9:34:40 PM |  | [work_fetch] Request work fetch: RPC complete
11/4/2021 9:34:42 PM | Rosetta@home | Started download of degrader_site_3mup_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_5od4py9r.zip
11/4/2021 9:34:42 PM | Rosetta@home | Started download of degrader_site_3mup_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_5od4py9r.flags
11/4/2021 9:34:43 PM | Rosetta@home | Finished download of degrader_site_3mup_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_5od4py9r.zip
11/4/2021 9:34:43 PM | Rosetta@home | Finished download of degrader_site_3mup_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_5od4py9r.flags
11/4/2021 9:34:43 PM | Rosetta@home | Started download of flags_rb_11_04_146922_143262__t000__3_C1_robetta
11/4/2021 9:34:43 PM | Rosetta@home | Started download of input_rb_11_04_146922_143262__t000__3_C1_robetta.zip
11/4/2021 9:34:44 PM | Rosetta@home | Finished download of flags_rb_11_04_146922_143262__t000__3_C1_robetta
11/4/2021 9:34:44 PM | Rosetta@home | Finished download of input_rb_11_04_146922_143262__t000__3_C1_robetta.zip
11/4/2021 9:34:44 PM | Rosetta@home | Started download of 5nvx_graft_buwei_xac_SAVE_ALL_OUT_IGNORE_THE_REST_4yg7pe5t.zip
11/4/2021 9:34:44 PM | Rosetta@home | Started download of 5nvx_graft_buwei_xac_SAVE_ALL_OUT_IGNORE_THE_REST_4yg7pe5t.flags
11/4/2021 9:34:46 PM | Rosetta@home | Finished download of 5nvx_graft_buwei_xac_SAVE_ALL_OUT_IGNORE_THE_REST_4yg7pe5t.zip
11/4/2021 9:34:46 PM | Rosetta@home | Finished download of 5nvx_graft_buwei_xac_SAVE_ALL_OUT_IGNORE_THE_REST_4yg7pe5t.flags
11/4/2021 9:34:46 PM |  | choose_project(): 1636076086.051674
11/4/2021 9:34:46 PM |  | [work_fetch] ------- start work fetch state -------
11/4/2021 9:34:46 PM |  | [work_fetch] target work buffer: 77760.00 + 8640.00 sec
11/4/2021 9:34:46 PM |  | [work_fetch] --- project states ---
11/4/2021 9:34:46 PM | Rosetta@home | [work_fetch] REC 14584.143 prio -16.563 can't request work: scheduler RPC backoff (25.76 sec)
11/4/2021 9:34:46 PM |  | [work_fetch] --- state for CPU ---
11/4/2021 9:34:46 PM |  | [work_fetch] shortfall 124201.94 nidle 0.00 saturated 65629.74 busy 56029.26
11/4/2021 9:34:46 PM | Rosetta@home | [work_fetch] share 0.000
11/4/2021 9:34:46 PM |  | [work_fetch] --- state for NVIDIA GPU ---
11/4/2021 9:34:46 PM |  | [work_fetch] shortfall 86400.00 nidle 1.00 saturated 0.00 busy 0.00
11/4/2021 9:34:46 PM | Rosetta@home | [work_fetch] share 0.000 no applications
11/4/2021 9:34:46 PM |  | [work_fetch] ------- end work fetch state -------
11/4/2021 9:34:46 PM | Rosetta@home | choose_project: scanning
11/4/2021 9:34:46 PM | Rosetta@home | skip: scheduler RPC backoff
11/4/2021 9:34:46 PM |  | [work_fetch] No project chosen for work fetch
11/4/2021 9:35:12 PM |  | [work_fetch] Request work fetch: Backoff ended for Rosetta@home
11/4/2021 9:35:16 PM |  | choose_project(): 1636076116.201881
11/4/2021 9:35:16 PM |  | [work_fetch] ------- start work fetch state -------
11/4/2021 9:35:16 PM |  | [work_fetch] target work buffer: 77760.00 + 8640.00 sec
11/4/2021 9:35:16 PM |  | [work_fetch] --- project states ---
11/4/2021 9:35:16 PM | Rosetta@home | [work_fetch] REC 14584.143 prio -2.287 can request work
11/4/2021 9:35:16 PM |  | [work_fetch] --- state for CPU ---
11/4/2021 9:35:16 PM |  | [work_fetch] shortfall 124658.22 nidle 0.00 saturated 65492.43 busy 55998.45
11/4/2021 9:35:16 PM | Rosetta@home | [work_fetch] share 1.000
11/4/2021 9:35:16 PM |  | [work_fetch] --- state for NVIDIA GPU ---
11/4/2021 9:35:16 PM |  | [work_fetch] shortfall 86400.00 nidle 1.00 saturated 0.00 busy 0.00
11/4/2021 9:35:16 PM | Rosetta@home | [work_fetch] share 0.000 no applications
11/4/2021 9:35:16 PM |  | [work_fetch] ------- end work fetch state -------
11/4/2021 9:35:16 PM | Rosetta@home | choose_project: scanning
11/4/2021 9:35:16 PM | Rosetta@home | can fetch CPU
11/4/2021 9:35:16 PM | Rosetta@home | CPU needs work - buffer low
11/4/2021 9:35:16 PM | Rosetta@home | checking CPU
11/4/2021 9:35:16 PM | Rosetta@home | [work_fetch] using MC shortfall 82970.159435 instead of shortfall 124658.222652
11/4/2021 9:35:16 PM | Rosetta@home | [work_fetch] set_request() for CPU: ninst 16 nused_total 599.00 nidle_now 0.00 fetch share 1.00 req_inst 0.00 req_secs 82970.16
11/4/2021 9:35:16 PM | Rosetta@home | CPU set_request: 82970.159435
11/4/2021 9:35:16 PM | Rosetta@home | checking NVIDIA GPU
11/4/2021 9:35:16 PM | Rosetta@home | NVIDIA GPU can't fetch: no applications
11/4/2021 9:35:16 PM | Rosetta@home | [work_fetch] request: CPU (82970.16 sec, 0.00 inst) NVIDIA GPU (0.00 sec, 0.00 inst)
11/4/2021 9:35:16 PM | Rosetta@home | Sending scheduler request: To fetch work.
11/4/2021 9:35:16 PM | Rosetta@home | Requesting new tasks for CPU
11/4/2021 9:35:18 PM | Rosetta@home | Scheduler request completed: got 3 new tasks
11/4/2021 9:35:18 PM | Rosetta@home | Project requested delay of 31 seconds
11/4/2021 9:35:18 PM |  | [work_fetch] Request work fetch: RPC complete
11/4/2021 9:35:20 PM | Rosetta@home | Started download of degrader_site_5fqd_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_4fa1lw1p.zip
11/4/2021 9:35:20 PM | Rosetta@home | Started download of degrader_site_5fqd_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_4fa1lw1p.flags
11/4/2021 9:35:21 PM | Rosetta@home | Finished download of degrader_site_5fqd_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_4fa1lw1p.flags
11/4/2021 9:35:21 PM | Rosetta@home | Started download of degrader_site_6ud7_1_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_4qd3qv3y.zip
11/4/2021 9:35:22 PM | Rosetta@home | Finished download of degrader_site_5fqd_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_4fa1lw1p.zip
11/4/2021 9:35:22 PM | Rosetta@home | Finished download of degrader_site_6ud7_1_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_4qd3qv3y.zip
11/4/2021 9:35:22 PM | Rosetta@home | Started download of degrader_site_6ud7_1_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_4qd3qv3y.flags
11/4/2021 9:35:22 PM | Rosetta@home | Started download of degrader_site_6ud7_0_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_9tf7nh2p.zip
11/4/2021 9:35:23 PM | Rosetta@home | Finished download of degrader_site_6ud7_1_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_4qd3qv3y.flags
11/4/2021 9:35:23 PM | Rosetta@home | Finished download of degrader_site_6ud7_0_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_9tf7nh2p.zip
11/4/2021 9:35:23 PM | Rosetta@home | Started download of degrader_site_6ud7_0_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_9tf7nh2p.flags
11/4/2021 9:35:23 PM |  | choose_project(): 1636076123.741503
11/4/2021 9:35:23 PM |  | [work_fetch] ------- start work fetch state -------
11/4/2021 9:35:23 PM |  | [work_fetch] target work buffer: 77760.00 + 8640.00 sec
11/4/2021 9:35:23 PM |  | [work_fetch] --- project states ---
11/4/2021 9:35:23 PM | Rosetta@home | [work_fetch] REC 14584.153 prio -16.564 can't request work: scheduler RPC backoff (25.75 sec)
11/4/2021 9:35:23 PM |  | [work_fetch] --- state for CPU ---
11/4/2021 9:35:23 PM |  | [work_fetch] shortfall 124759.37 nidle 0.00 saturated 65516.88 busy 56035.66
11/4/2021 9:35:23 PM | Rosetta@home | [work_fetch] share 0.000
11/4/2021 9:35:23 PM |  | [work_fetch] --- state for NVIDIA GPU ---
11/4/2021 9:35:23 PM |  | [work_fetch] shortfall 86400.00 nidle 1.00 saturated 0.00 busy 0.00
11/4/2021 9:35:23 PM | Rosetta@home | [work_fetch] share 0.000 no applications
11/4/2021 9:35:23 PM |  | [work_fetch] ------- end work fetch state -------
11/4/2021 9:35:23 PM | Rosetta@home | choose_project: scanning
11/4/2021 9:35:23 PM | Rosetta@home | skip: scheduler RPC backoff
11/4/2021 9:35:23 PM |  | [work_fetch] No project chosen for work fetch
11/4/2021 9:35:24 PM | Rosetta@home | Finished download of degrader_site_6ud7_0_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_9tf7nh2p.flags
11/4/2021 9:35:49 PM |  | [work_fetch] Request work fetch: Backoff ended for Rosetta@home
11/4/2021 9:35:53 PM |  | choose_project(): 1636076153.953737
11/4/2021 9:35:53 PM |  | [work_fetch] ------- start work fetch state -------
11/4/2021 9:35:53 PM |  | [work_fetch] target work buffer: 77760.00 + 8640.00 sec
11/4/2021 9:35:53 PM |  | [work_fetch] --- project states ---
11/4/2021 9:35:53 PM | Rosetta@home | [work_fetch] REC 14584.153 prio -2.288 can request work
11/4/2021 9:35:53 PM |  | [work_fetch] --- state for CPU ---
11/4/2021 9:35:53 PM |  | [work_fetch] shortfall 125245.36 nidle 0.00 saturated 65465.14 busy 55998.93
11/4/2021 9:35:53 PM | Rosetta@home | [work_fetch] share 1.000
11/4/2021 9:35:53 PM |  | [work_fetch] --- state for NVIDIA GPU ---
11/4/2021 9:35:53 PM |  | [work_fetch] shortfall 86400.00 nidle 1.00 saturated 0.00 busy 0.00
11/4/2021 9:35:53 PM | Rosetta@home | [work_fetch] share 0.000 no applications
11/4/2021 9:35:53 PM |  | [work_fetch] ------- end work fetch state -------
11/4/2021 9:35:53 PM | Rosetta@home | choose_project: scanning
11/4/2021 9:35:53 PM | Rosetta@home | can fetch CPU
11/4/2021 9:35:53 PM | Rosetta@home | CPU needs work - buffer low
11/4/2021 9:35:53 PM | Rosetta@home | checking CPU
11/4/2021 9:35:53 PM | Rosetta@home | [work_fetch] using MC shortfall 83424.534840 instead of shortfall 125245.361612
11/4/2021 9:35:53 PM | Rosetta@home | [work_fetch] set_request() for CPU: ninst 16 nused_total 602.00 nidle_now 0.00 fetch share 1.00 req_inst 0.00 req_secs 83424.53
11/4/2021 9:35:53 PM | Rosetta@home | CPU set_request: 83424.534840
11/4/2021 9:35:53 PM | Rosetta@home | checking NVIDIA GPU
11/4/2021 9:35:53 PM | Rosetta@home | NVIDIA GPU can't fetch: no applications
11/4/2021 9:35:53 PM | Rosetta@home | [work_fetch] request: CPU (83424.53 sec, 0.00 inst) NVIDIA GPU (0.00 sec, 0.00 inst)
11/4/2021 9:35:54 PM | Rosetta@home | Sending scheduler request: To fetch work.
11/4/2021 9:35:54 PM | Rosetta@home | Requesting new tasks for CPU
11/4/2021 9:35:55 PM | Rosetta@home | Scheduler request completed: got 3 new tasks
11/4/2021 9:35:55 PM | Rosetta@home | Project requested delay of 31 seconds
11/4/2021 9:35:55 PM |  | [work_fetch] Request work fetch: RPC complete
11/4/2021 9:35:57 PM | Rosetta@home | Started download of degrader_site_5fnu_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_5ca7or4w.zip
11/4/2021 9:35:57 PM | Rosetta@home | Started download of degrader_site_5fnu_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_5ca7or4w.flags
11/4/2021 9:35:59 PM | Rosetta@home | Finished download of degrader_site_5fnu_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_5ca7or4w.zip
11/4/2021 9:35:59 PM | Rosetta@home | Finished download of degrader_site_5fnu_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_5ca7or4w.flags
11/4/2021 9:35:59 PM | Rosetta@home | Started download of flags_rb_11_04_146922_143262__t000__2_C1_robetta
11/4/2021 9:35:59 PM | Rosetta@home | Started download of input_rb_11_04_146922_143262__t000__2_C1_robetta.zip
11/4/2021 9:36:00 PM | Rosetta@home | Finished download of flags_rb_11_04_146922_143262__t000__2_C1_robetta
11/4/2021 9:36:00 PM | Rosetta@home | Started download of degrader_site_6ud7_1_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_3ox4nd7i.zip
11/4/2021 9:36:00 PM |  | choose_project(): 1636076160.417528
11/4/2021 9:36:00 PM |  | [work_fetch] ------- start work fetch state -------
11/4/2021 9:36:00 PM |  | [work_fetch] target work buffer: 77760.00 + 8640.00 sec
11/4/2021 9:36:00 PM |  | [work_fetch] --- project states ---
11/4/2021 9:36:00 PM | Rosetta@home | [work_fetch] REC 14584.163 prio -16.349 can't request work: scheduler RPC backoff (25.82 sec)
11/4/2021 9:36:00 PM |  | [work_fetch] --- state for CPU ---
11/4/2021 9:36:00 PM |  | [work_fetch] shortfall 125158.34 nidle 0.00 saturated 65411.47 busy 55969.15
11/4/2021 9:36:00 PM | Rosetta@home | [work_fetch] share 0.000
11/4/2021 9:36:00 PM |  | [work_fetch] --- state for NVIDIA GPU ---
11/4/2021 9:36:00 PM |  | [work_fetch] shortfall 86400.00 nidle 1.00 saturated 0.00 busy 0.00
11/4/2021 9:36:00 PM | Rosetta@home | [work_fetch] share 0.000 no applications
11/4/2021 9:36:00 PM |  | [work_fetch] ------- end work fetch state -------
11/4/2021 9:36:00 PM | Rosetta@home | choose_project: scanning
11/4/2021 9:36:00 PM | Rosetta@home | skip: scheduler RPC backoff
11/4/2021 9:36:00 PM |  | [work_fetch] No project chosen for work fetch
11/4/2021 9:36:01 PM | Rosetta@home | Finished download of input_rb_11_04_146922_143262__t000__2_C1_robetta.zip
11/4/2021 9:36:01 PM | Rosetta@home | Finished download of degrader_site_6ud7_1_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_3ox4nd7i.zip
11/4/2021 9:36:01 PM | Rosetta@home | Started download of degrader_site_6ud7_1_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_3ox4nd7i.flags
11/4/2021 9:36:03 PM | Rosetta@home | Finished download of degrader_site_6ud7_1_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_3ox4nd7i.flags
11/4/2021 9:36:26 PM |  | [work_fetch] Request work fetch: Backoff ended for Rosetta@home
11/4/2021 9:36:30 PM |  | choose_project(): 1636076190.696714
11/4/2021 9:36:30 PM |  | [work_fetch] ------- start work fetch state -------
11/4/2021 9:36:30 PM |  | [work_fetch] target work buffer: 77760.00 + 8640.00 sec
11/4/2021 9:36:30 PM |  | [work_fetch] --- project states ---
11/4/2021 9:36:30 PM | Rosetta@home | [work_fetch] REC 14584.164 prio -2.290 can request work
11/4/2021 9:36:30 PM |  | [work_fetch] --- state for CPU ---
11/4/2021 9:36:30 PM |  | [work_fetch] shortfall 125498.41 nidle 0.00 saturated 65367.40 busy 55938.84
11/4/2021 9:36:30 PM | Rosetta@home | [work_fetch] share 1.000
11/4/2021 9:36:30 PM |  | [work_fetch] --- state for NVIDIA GPU ---
11/4/2021 9:36:30 PM |  | [work_fetch] shortfall 86400.00 nidle 1.00 saturated 0.00 busy 0.00
11/4/2021 9:36:30 PM | Rosetta@home | [work_fetch] share 0.000 no applications
11/4/2021 9:36:30 PM |  | [work_fetch] ------- end work fetch state -------
11/4/2021 9:36:30 PM | Rosetta@home | choose_project: scanning
11/4/2021 9:36:30 PM | Rosetta@home | can fetch CPU
11/4/2021 9:36:30 PM | Rosetta@home | CPU needs work - buffer low
11/4/2021 9:36:30 PM | Rosetta@home | checking CPU
11/4/2021 9:36:30 PM | Rosetta@home | [work_fetch] using MC shortfall 83549.395553 instead of shortfall 125498.410038
11/4/2021 9:36:30 PM | Rosetta@home | [work_fetch] set_request() for CPU: ninst 16 nused_total 605.00 nidle_now 0.00 fetch share 1.00 req_inst 0.00 req_secs 83549.40
11/4/2021 9:36:30 PM | Rosetta@home | CPU set_request: 83549.395553
11/4/2021 9:36:30 PM | Rosetta@home | checking NVIDIA GPU
11/4/2021 9:36:30 PM | Rosetta@home | NVIDIA GPU can't fetch: no applications
11/4/2021 9:36:30 PM | Rosetta@home | [work_fetch] request: CPU (83549.40 sec, 0.00 inst) NVIDIA GPU (0.00 sec, 0.00 inst)
11/4/2021 9:36:30 PM | Rosetta@home | Sending scheduler request: To fetch work.
11/4/2021 9:36:30 PM | Rosetta@home | Requesting new tasks for CPU
11/4/2021 9:36:32 PM | Rosetta@home | Scheduler request completed: got 3 new tasks
11/4/2021 9:36:32 PM | Rosetta@home | Project requested delay of 31 seconds
11/4/2021 9:36:32 PM |  | [work_fetch] Request work fetch: RPC complete
11/4/2021 9:36:35 PM | Rosetta@home | Started download of degrader_site_5fqd_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_6lq4wx4a.zip
11/4/2021 9:36:35 PM | Rosetta@home | Started download of degrader_site_5fqd_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_6lq4wx4a.flags
11/4/2021 9:36:36 PM | Rosetta@home | Finished download of degrader_site_5fqd_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_6lq4wx4a.zip
11/4/2021 9:36:36 PM | Rosetta@home | Finished download of degrader_site_5fqd_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_6lq4wx4a.flags
11/4/2021 9:36:36 PM | Rosetta@home | Started download of 5nvx_graft_buwei_xad_SAVE_ALL_OUT_IGNORE_THE_REST_5vh8ii2p.zip
11/4/2021 9:36:36 PM | Rosetta@home | Started download of 5nvx_graft_buwei_xad_SAVE_ALL_OUT_IGNORE_THE_REST_5vh8ii2p.flags
11/4/2021 9:36:37 PM | Rosetta@home | Finished download of 5nvx_graft_buwei_xad_SAVE_ALL_OUT_IGNORE_THE_REST_5vh8ii2p.zip
11/4/2021 9:36:37 PM | Rosetta@home | Finished download of 5nvx_graft_buwei_xad_SAVE_ALL_OUT_IGNORE_THE_REST_5vh8ii2p.flags
11/4/2021 9:36:37 PM | Rosetta@home | Started download of 5nvx_graft_buwei_xab_SAVE_ALL_OUT_IGNORE_THE_REST_2ao5rl6j.zip
11/4/2021 9:36:37 PM | Rosetta@home | Started download of 5nvx_graft_buwei_xab_SAVE_ALL_OUT_IGNORE_THE_REST_2ao5rl6j.flags
11/4/2021 9:36:38 PM | Rosetta@home | Finished download of 5nvx_graft_buwei_xab_SAVE_ALL_OUT_IGNORE_THE_REST_2ao5rl6j.zip
11/4/2021 9:36:38 PM | Rosetta@home | Finished download of 5nvx_graft_buwei_xab_SAVE_ALL_OUT_IGNORE_THE_REST_2ao5rl6j.flags
11/4/2021 9:36:38 PM |  | choose_project(): 1636076198.227594
11/4/2021 9:36:38 PM |  | [work_fetch] ------- start work fetch state -------
11/4/2021 9:36:38 PM |  | [work_fetch] target work buffer: 77760.00 + 8640.00 sec
11/4/2021 9:36:38 PM |  | [work_fetch] --- project states ---
11/4/2021 9:36:38 PM | Rosetta@home | [work_fetch] REC 14584.173 prio -16.567 can't request work: scheduler RPC backoff (25.76 sec)
11/4/2021 9:36:38 PM |  | [work_fetch] --- state for CPU ---
11/4/2021 9:36:38 PM |  | [work_fetch] shortfall 125725.80 nidle 0.00 saturated 65318.64 busy 55921.64
11/4/2021 9:36:38 PM | Rosetta@home | [work_fetch] share 0.000
11/4/2021 9:36:38 PM |  | [work_fetch] --- state for NVIDIA GPU ---
11/4/2021 9:36:38 PM |  | [work_fetch] shortfall 86400.00 nidle 1.00 saturated 0.00 busy 0.00
11/4/2021 9:36:38 PM | Rosetta@home | [work_fetch] share 0.000 no applications
11/4/2021 9:36:38 PM |  | [work_fetch] ------- end work fetch state -------
11/4/2021 9:36:38 PM | Rosetta@home | choose_project: scanning
11/4/2021 9:36:38 PM | Rosetta@home | skip: scheduler RPC backoff
11/4/2021 9:36:38 PM |  | [work_fetch] No project chosen for work fetch
11/4/2021 9:37:04 PM |  | [work_fetch] Request work fetch: Backoff ended for Rosetta@home
11/4/2021 9:37:08 PM |  | choose_project(): 1636076228.364588
11/4/2021 9:37:08 PM |  | [work_fetch] ------- start work fetch state -------
11/4/2021 9:37:08 PM |  | [work_fetch] target work buffer: 77760.00 + 8640.00 sec
11/4/2021 9:37:08 PM |  | [work_fetch] --- project states ---
11/4/2021 9:37:08 PM | Rosetta@home | [work_fetch] REC 14584.173 prio -2.291 can request work
11/4/2021 9:37:08 PM |  | [work_fetch] --- state for CPU ---
11/4/2021 9:37:08 PM |  | [work_fetch] shortfall 126099.59 nidle 0.00 saturated 65267.32 busy 55894.76
11/4/2021 9:37:08 PM | Rosetta@home | [work_fetch] share 1.000
11/4/2021 9:37:08 PM |  | [work_fetch] --- state for NVIDIA GPU ---
11/4/2021 9:37:08 PM |  | [work_fetch] shortfall 86400.00 nidle 1.00 saturated 0.00 busy 0.00
11/4/2021 9:37:08 PM | Rosetta@home | [work_fetch] share 0.000 no applications
11/4/2021 9:37:08 PM |  | [work_fetch] ------- end work fetch state -------
11/4/2021 9:37:08 PM | Rosetta@home | choose_project: scanning
11/4/2021 9:37:08 PM | Rosetta@home | can fetch CPU
11/4/2021 9:37:08 PM | Rosetta@home | CPU needs work - buffer low
11/4/2021 9:37:08 PM | Rosetta@home | checking CPU
11/4/2021 9:37:08 PM | Rosetta@home | [work_fetch] using MC shortfall 83946.905035 instead of shortfall 126099.594440
11/4/2021 9:37:08 PM | Rosetta@home | [work_fetch] set_request() for CPU: ninst 16 nused_total 608.00 nidle_now 0.00 fetch share 1.00 req_inst 0.00 req_secs 83946.91
11/4/2021 9:37:08 PM | Rosetta@home | CPU set_request: 83946.905035
11/4/2021 9:37:08 PM | Rosetta@home | checking NVIDIA GPU
11/4/2021 9:37:08 PM | Rosetta@home | NVIDIA GPU can't fetch: no applications
11/4/2021 9:37:08 PM | Rosetta@home | [work_fetch] request: CPU (83946.91 sec, 0.00 inst) NVIDIA GPU (0.00 sec, 0.00 inst)
11/4/2021 9:37:08 PM | Rosetta@home | Sending scheduler request: To fetch work.
11/4/2021 9:37:08 PM | Rosetta@home | Requesting new tasks for CPU
11/4/2021 9:37:09 PM | Rosetta@home | Scheduler request completed: got 3 new tasks
11/4/2021 9:37:09 PM | Rosetta@home | Project requested delay of 31 seconds
11/4/2021 9:37:09 PM |  | [work_fetch] Request work fetch: RPC complete
11/4/2021 9:37:11 PM | Rosetta@home | Started download of flags_rb_11_04_146922_143262__t000__4_C1_robetta
11/4/2021 9:37:11 PM | Rosetta@home | Started download of input_rb_11_04_146922_143262__t000__4_C1_robetta.zip
11/4/2021 9:37:12 PM | Rosetta@home | Finished download of flags_rb_11_04_146922_143262__t000__4_C1_robetta
11/4/2021 9:37:12 PM | Rosetta@home | Finished download of input_rb_11_04_146922_143262__t000__4_C1_robetta.zip
11/4/2021 9:37:12 PM | Rosetta@home | Started download of degrader_site_5fnu_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_2hz8em0i.zip
11/4/2021 9:37:12 PM | Rosetta@home | Started download of degrader_site_5fnu_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_2hz8em0i.flags
11/4/2021 9:37:13 PM | Rosetta@home | Finished download of degrader_site_5fnu_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_2hz8em0i.zip
11/4/2021 9:37:13 PM | Rosetta@home | Finished download of degrader_site_5fnu_3h_graft_bcov_1_SAVE_ALL_OUT_IGNORE_THE_REST_2hz8em0i.flags
11/4/2021 9:37:14 PM |  | choose_project(): 1636076234.913389
11/4/2021 9:37:14 PM |  | [work_fetch] ------- start work fetch state -------
11/4/2021 9:37:14 PM |  | [work_fetch] target work buffer: 77760.00 + 8640.00 sec
11/4/2021 9:37:14 PM |  | [work_fetch] --- project states ---
11/4/2021 9:37:14 PM | Rosetta@home | [work_fetch] REC 14584.182 prio -16.351 can't request work: scheduler RPC backoff (25.74 sec)
11/4/2021 9:37:14 PM |  | [work_fetch] --- state for CPU ---
11/4/2021 9:37:14 PM |  | [work_fetch] shortfall 126308.31 nidle 0.00 saturated 65291.55 busy 55888.47
11/4/2021 9:37:14 PM | Rosetta@home | [work_fetch] share 0.000
11/4/2021 9:37:14 PM |  | [work_fetch] --- state for NVIDIA GPU ---
11/4/2021 9:37:14 PM |  | [work_fetch] shortfall 86400.00 nidle 1.00 saturated 0.00 busy 0.00
11/4/2021 9:37:14 PM | Rosetta@home | [work_fetch] share 0.000 no applications
11/4/2021 9:37:14 PM |  | [work_fetch] ------- end work fetch state -------
11/4/2021 9:37:14 PM | Rosetta@home | choose_project: scanning
11/4/2021 9:37:14 PM | Rosetta@home | skip: scheduler RPC backoff
11/4/2021 9:37:14 PM |  | [work_fetch] No project chosen for work fetch
11/4/2021 9:37:40 PM |  | [work_fetch] Request work fetch: Backoff ended for Rosetta@home
11/4/2021 9:37:44 PM |  | choose_project(): 1636076264.965370
11/4/2021 9:37:44 PM |  | [work_fetch] ------- start work fetch state -------
11/4/2021 9:37:44 PM |  | [work_fetch] target work buffer: 77760.00 + 8640.00 sec
11/4/2021 9:37:44 PM |  | [work_fetch] --- project states ---
11/4/2021 9:37:44 PM | Rosetta@home | [work_fetch] REC 14584.182 prio -2.293 can request work
11/4/2021 9:37:44 PM |  | [work_fetch] --- state for CPU ---
11/4/2021 9:37:44 PM |  | [work_fetch] shortfall 126671.93 nidle 0.00 saturated 65235.29 busy 55860.76
11/4/2021 9:37:44 PM | Rosetta@home | [work_fetch] share 1.000
11/4/2021 9:37:44 PM |  | [work_fetch] --- state for NVIDIA GPU ---
11/4/2021 9:37:44 PM |  | [work_fetch] shortfall 86400.00 nidle 1.00 saturated 0.00 busy 0.00
11/4/2021 9:37:44 PM | Rosetta@home | [work_fetch] share 0.000 no applications
11/4/2021 9:37:44 PM |  | [work_fetch] ------- end work fetch state -------
11/4/2021 9:37:44 PM | Rosetta@home | choose_project: scanning
11/4/2021 9:37:44 PM | Rosetta@home | can fetch CPU
11/4/2021 9:37:44 PM | Rosetta@home | CPU needs work - buffer low
11/4/2021 9:37:44 PM | Rosetta@home | checking CPU
11/4/2021 9:37:44 PM | Rosetta@home | [work_fetch] using MC shortfall 84386.263874 instead of shortfall 126671.929353
11/4/2021 9:37:44 PM | Rosetta@home | [work_fetch] set_request() for CPU: ninst 16 nused_total 611.00 nidle_now 0.00 fetch share 1.00 req_inst 0.00 req_secs 84386.26
11/4/2021 9:37:44 PM | Rosetta@home | CPU set_request: 84386.263874
11/4/2021 9:37:44 PM | Rosetta@home | checking NVIDIA GPU
11/4/2021 9:37:44 PM | Rosetta@home | NVIDIA GPU can't fetch: no applications
11/4/2021 9:37:44 PM | Rosetta@home | [work_fetch] request: CPU (84386.26 sec, 0.00 inst) NVIDIA GPU (0.00 sec, 0.00 inst)
11/4/2021 9:37:45 PM | Rosetta@home | Sending scheduler request: To fetch work.
11/4/2021 9:37:45 PM | Rosetta@home | Requesting new tasks for CPU
11/4/2021 9:37:47 PM | Rosetta@home | Scheduler request completed: got 3 new tasks
11/4/2021 9:37:47 PM | Rosetta@home | Project requested delay of 31 seconds
11/4/2021 9:37:47 PM |  | [work_fetch] Request work fetch: RPC complete
11/4/2021 9:37:49 PM | Rosetta@home | Started download of 5nvx_graft_buwei_xaf_SAVE_ALL_OUT_IGNORE_THE_REST_0in9vt6s.zip
11/4/2021 9:37:49 PM | Rosetta@home | Started download of 5nvx_graft_buwei_xaf_SAVE_ALL_OUT_IGNORE_THE_REST_0in9vt6s.flags
11/4/2021 9:37:50 PM | Rosetta@home | Finished download of 5nvx_graft_buwei_xaf_SAVE_ALL_OUT_IGNORE_THE_REST_0in9vt6s.zip
11/4/2021 9:37:50 PM | Rosetta@home | Finished download of 5nvx_graft_buwei_xaf_SAVE_ALL_OUT_IGNORE_THE_REST_0in9vt6s.flags
11/4/2021 9:37:50 PM | Rosetta@home | Started download of 5nvx_graft_buwei_xaa_SAVE_ALL_OUT_IGNORE_THE_REST_5ha5ul8d.zip
11/4/2021 9:37:50 PM | Rosetta@home | Started download of 5nvx_graft_buwei_xaa_SAVE_ALL_OUT_IGNORE_THE_REST_5ha5ul8d.flags
11/4/2021 9:37:51 PM | Rosetta@home | Finished download of 5nvx_graft_buwei_xaa_SAVE_ALL_OUT_IGNORE_THE_REST_5ha5ul8d.zip
11/4/2021 9:37:51 PM | Rosetta@home | Finished download of 5nvx_graft_buwei_xaa_SAVE_ALL_OUT_IGNORE_THE_REST_5ha5ul8d.flags
11/4/2021 9:37:51 PM | Rosetta@home | Started download of 5nvx_graft_buwei_xac_SAVE_ALL_OUT_IGNORE_THE_REST_6vk0ii7h.zip
11/4/2021 9:37:51 PM | Rosetta@home | Started download of 5nvx_graft_buwei_xac_SAVE_ALL_OUT_IGNORE_THE_REST_6vk0ii7h.flags
ID: 103107 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1681
Credit: 17,854,150
RAC: 20,118
Message 103108 - Posted: 5 Nov 2021, 2:12:22 UTC - in response to Message 103107.  
Last modified: 5 Nov 2021, 2:12:55 UTC

Rosetta is fetching excessive workunits that are being aborted when they hit their Deadline time. I have my BOINC client set for only 1 day of work plus 0.1 days of additional work. However, I currently have 580ish Rosetta tasks. Work fetch appears to be broken.
Do you have a app_config file with max_concurrent in it? If so- that's the cause of your problem.
It has been an issue for some time, and there is still no fix for when it does occur, nor any idea of why it occurs on some projects & not others. The only fix at present is removing the max_concurrent setting & re-reading the configuration file.

I'd suggest posting to the BOINC forums to let them know it is still an issue. That may or may not help in having it fixed, someday, maybe.
Grant
Darwin NT
ID: 103108 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
skydivingnerd

Send message
Joined: 28 Jan 13
Posts: 5
Credit: 87,995,889
RAC: 384
Message 103118 - Posted: 5 Nov 2021, 20:58:28 UTC - in response to Message 103108.  

Thanks for the heads up Grant. Yes, I do have an app_config with <max_concurrent> settings. I'll comment them out and see how things go.

<app_config>

  <app>
    <name>rosetta</name>
    <max_concurrent>14</max_concurrent>
    <fraction_done_exact/>
  </app>
  <app_version>
    <app_name>rosetta</app_name>
<!--
    <plan_class>####</plan_class>
-->
    <avg_ncpus>1.0</avg_ncpus>
    <cmdline>--nthreads 1</cmdline>
  </app_version>

  <app>
    <name>rosetta_python_projects</name>
    <max_concurrent>2</max_concurrent>
    <fraction_done_exact/>
  </app>
  <app_version>
    <app_name>rosetta_python_projects</app_name>
    <plan_class>vbox64</plan_class>
    <avg_ncpus>1.0</avg_ncpus>
    <cmdline>--nthreads 1 --memory_size_mb 6144</cmdline>
  </app_version>

</app_config>
ID: 103118 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [AF>Le_Pommier] Jerome_C2005

Send message
Joined: 22 Aug 06
Posts: 42
Credit: 1,258,039
RAC: 0
Message 103170 - Posted: 11 Nov 2021, 16:12:56 UTC

Hi,

I have the exact same problem : I was forced to limit the python tasks to 1 at a time using app_config (recommended in another thread on this forum) because I would get many problems with those tasks on my iMac (with lots of resources : core i9 + 40 Gb RAM, but they stall / don't stop to crunch after almost 2 days and go over the deadline, ... ) and now it downloaded almost 1000 tasks with 2 days deadline (regular rosetta, not python where it only got 9 waiting to crunch).

So this is "not rosetta's fault" ? I never got such a problem on any other project and it is not the first time I limit number of tasks (I used to do it on LCH back in the days when they didn't have much options in project setup).

And rosetta offers no setup whatsoever with project selection.

I feel much more that *it is* "rosetta's fault" actually.
ID: 103170 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
MJH333

Send message
Joined: 29 Jan 21
Posts: 18
Credit: 6,290,467
RAC: 13,756
Message 103172 - Posted: 11 Nov 2021, 21:22:20 UTC - in response to Message 103170.  

I never got such a problem on any other project

I’ve had this problem several times on World Community Grid. I’ve stopped using max_concurrent on WCG because of this.
ID: 103172 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mikey
Avatar

Send message
Joined: 5 Jan 06
Posts: 1895
Credit: 9,169,305
RAC: 3,400
Message 103191 - Posted: 12 Nov 2021, 11:53:29 UTC - in response to Message 103170.  

Hi,

I have the exact same problem : I was forced to limit the python tasks to 1 at a time using app_config (recommended in another thread on this forum) because I would get many problems with those tasks on my iMac (with lots of resources : core i9 + 40 Gb RAM, but they stall / don't stop to crunch after almost 2 days and go over the deadline, ... ) and now it downloaded almost 1000 tasks with 2 days deadline (regular rosetta, not python where it only got 9 waiting to crunch).

So this is "not rosetta's fault" ? I never got such a problem on any other project and it is not the first time I limit number of tasks (I used to do it on LCH back in the days when they didn't have much options in project setup).

And rosetta offers no setup whatsoever with project selection.

I feel much more that *it is* "rosetta's fault" actually.


Go back thru your logs if you still have them and see how much work Boinc requested from Rosetta and compare it to how much you actually received, it sounds like Boinc asked for ALOT more than it should have and Rosetta responded. Boinc Projects don't decide on how much work to give you they respond to a request by the Boinc Manager and then try to give you as much as you ask for. You running an app_config file to limit the number of tasks you run at one time does not currently fit into the calculations when Boinc asks for work, it uses the idea that every single one of the cpu cores you allow Boinc to use will be crunching the tasks the project sends you, THEN the app_config files takes over and does what you tell it to do.
ID: 103191 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [AF>Le_Pommier] Jerome_C2005

Send message
Joined: 22 Aug 06
Posts: 42
Credit: 1,258,039
RAC: 0
Message 103198 - Posted: 12 Nov 2021, 18:12:20 UTC - in response to Message 103191.  
Last modified: 12 Nov 2021, 18:13:46 UTC

I don't understand what you mean with "how much work Boinc requested", boinc just asks "new tasks", it never asks for any given amount of work... ??

Ven 12 nov 17:56:56 2021 | Rosetta@home | Sending scheduler request: To fetch work.
Ven 12 nov 17:56:56 2021 | Rosetta@home | Reporting 1 completed tasks
Ven 12 nov 17:56:56 2021 | Rosetta@home | Requesting new tasks for CPU
Ven 12 nov 17:57:04 2021 | Rosetta@home | Scheduler request completed: got 1 new tasks
Ven 12 nov 17:57:04 2021 | Rosetta@home | Project requested delay of 31 seconds
Ven 12 nov 17:57:06 2021 | Rosetta@home | Started download of aaap-ACPhenC_pp-mHPS-PHE-AMACBEN3_8.gz
Ven 12 nov 17:57:13 2021 | Rosetta@home | Finished download of aaap-ACPhenC_pp-mHPS-PHE-AMACBEN3_8.gz
Ven 12 nov 18:02:31 2021 | Rosetta@home | 1001 UT CPU (992 non démarées, 9 en cours, 0 terminées)
Or do you mean "how often it does it" ?
Ven 12 nov 18:27:10 2021 | Rosetta@home | Sending scheduler request: Requested by project.
Ven 12 nov 18:27:10 2021 | Rosetta@home | Not requesting tasks: too many runnable tasks
Ven 12 nov 18:27:19 2021 | Rosetta@home | Scheduler request completed
Ven 12 nov 18:27:19 2021 | Rosetta@home | Project requested delay of 31 seconds
Ven 12 nov 18:57:22 2021 | Rosetta@home | Sending scheduler request: Requested by project.
Ven 12 nov 18:57:22 2021 | Rosetta@home | Not requesting tasks: too many runnable tasks
Ven 12 nov 18:57:28 2021 | Rosetta@home | Scheduler request completed
Ven 12 nov 18:57:28 2021 | Rosetta@home | Project requested delay of 31 seconds

I (and you) can see there was a 30mn delay between last 2 requests from boinc, but last one didn't even succeed.
If I look up in the history I confirm it is every 30mn.

All these tasks will start to reach the deadline between Saturday and Monday, so I guess my boinc (and / or the rosetta server) is going to cancel them massively ? so next step is that rosetta is going to send new ones massively again ?? at some point the server should start to blacklist my machine I suppose ?

So there is no solution to that problem, since rosetta doesn't propose any setup option on the website (to block the python tasks) and above all since rosetta python tasks are causing so many problems that I can't run many of them at the same time ?

Even with my run limitation I see the currently running task has now *more than 2 days of execution*, it is now a few hours away from the deadline, the completion % is moving up extremely slowly (+ 0,001% after a long time, it is now 99,951%, earlier in the afternoon I remember it was 99,950%).

So probably the only real solution is to stop rosetta :(
ID: 103198 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [AF>Le_Pommier] Jerome_C2005

Send message
Joined: 22 Aug 06
Posts: 42
Credit: 1,258,039
RAC: 0
Message 103199 - Posted: 12 Nov 2021, 18:15:06 UTC - in response to Message 103172.  

I never got such a problem on any other project

I’ve had this problem several times on World Community Grid. I’ve stopped using max_concurrent on WCG because of this.

You are not lucky, I've been using this in many occasions in the past.
ID: 103199 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Bryn Mawr

Send message
Joined: 26 Dec 18
Posts: 393
Credit: 12,110,248
RAC: 4,952
Message 103205 - Posted: 12 Nov 2021, 20:28:46 UTC - in response to Message 103198.  

I don't understand what you mean with "how much work Boinc requested", boinc just asks "new tasks", it never asks for any given amount of work... ??


If you turn on sched_ops_debug and work_fetch_debug, do an update and then switch them off again quick (they produce a lot of output) then you’ll see that the client specifies exactly how many seconds work it wants for CPU and for GPU with an idea of the number of tasks it expects that to be.
ID: 103205 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [AF>Le_Pommier] Jerome_C2005

Send message
Joined: 22 Aug 06
Posts: 42
Credit: 1,258,039
RAC: 0
Message 103254 - Posted: 14 Nov 2021, 11:17:03 UTC
Last modified: 14 Nov 2021, 11:17:40 UTC

Dim 14 nov 12:11:56 2021 |  | [work_fetch] Request work fetch: Core client configuration
Dim 14 nov 12:12:00 2021 |  | [work_fetch] Request work fetch: application exited
Dim 14 nov 12:12:00 2021 |  | choose_project(): 1636888320.744567
Dim 14 nov 12:12:00 2021 |  | [work_fetch] ------- start work fetch state -------
Dim 14 nov 12:12:00 2021 |  | [work_fetch] target work buffer: 43200.00 + 86400.00 sec
Dim 14 nov 12:12:00 2021 |  | [work_fetch] --- project states ---
Dim 14 nov 12:12:00 2021 | Rosetta@home | [work_fetch] REC 3867.255 prio -0.841 can't request work: too many runnable tasks
Dim 14 nov 12:12:00 2021 |  | [work_fetch] --- state for CPU ---
Dim 14 nov 12:12:00 2021 |  | [work_fetch] shortfall 1058742.41 nidle 7.00 saturated 0.00 busy 0.00
Dim 14 nov 12:12:00 2021 | Rosetta@home | [work_fetch] share 0.000
Dim 14 nov 12:12:00 2021 |  | [work_fetch] --- state for AMD/ATI GPU ---
Dim 14 nov 12:12:00 2021 |  | [work_fetch] shortfall 1503.29 nidle 0.00 saturated 128096.71 busy 0.00
Dim 14 nov 12:12:00 2021 | Rosetta@home | [work_fetch] share 0.000 no applications
Dim 14 nov 12:12:00 2021 |  | [work_fetch] ------- end work fetch state -------
Dim 14 nov 12:12:00 2021 | Rosetta@home | skip: too many runnable tasks
Dim 14 nov 12:12:00 2021 |  | [work_fetch] No project chosen for work fetch
Dim 14 nov 12:12:05 2021 | Rosetta@home | update requested by user
Dim 14 nov 12:12:05 2021 |  | [work_fetch] Request work fetch: project updated by user
Dim 14 nov 12:12:05 2021 | Rosetta@home | [sched_op] sched RPC pending: Requested by user
Dim 14 nov 12:12:05 2021 | Rosetta@home | piggyback_work_request()
Dim 14 nov 12:12:05 2021 |  | [work_fetch] ------- start work fetch state -------
Dim 14 nov 12:12:05 2021 |  | [work_fetch] target work buffer: 43200.00 + 86400.00 sec
Dim 14 nov 12:12:05 2021 |  | [work_fetch] --- project states ---
Dim 14 nov 12:12:05 2021 | Rosetta@home | [work_fetch] REC 3867.255 prio -0.841 can't request work: too many runnable tasks
Dim 14 nov 12:12:05 2021 |  | [work_fetch] --- state for CPU ---
Dim 14 nov 12:12:05 2021 |  | [work_fetch] shortfall 1058747.99 nidle 7.00 saturated 0.00 busy 0.00
Dim 14 nov 12:12:05 2021 | Rosetta@home | [work_fetch] share 0.000
Dim 14 nov 12:12:05 2021 |  | [work_fetch] --- state for AMD/ATI GPU ---
Dim 14 nov 12:12:05 2021 |  | [work_fetch] shortfall 1507.87 nidle 0.00 saturated 128092.13 busy 0.00
Dim 14 nov 12:12:05 2021 | Rosetta@home | [work_fetch] share 0.000 no applications
Dim 14 nov 12:12:05 2021 |  | [work_fetch] ------- end work fetch state -------
Dim 14 nov 12:12:05 2021 | Rosetta@home | [sched_op] Starting scheduler request
Dim 14 nov 12:12:05 2021 | Rosetta@home | [work_fetch] request: CPU (0.00 sec, 0.00 inst) AMD/ATI GPU (0.00 sec, 0.00 inst)
Dim 14 nov 12:12:07 2021 | Rosetta@home | Sending scheduler request: Requested by user.
Dim 14 nov 12:12:07 2021 | Rosetta@home | Not requesting tasks: too many runnable tasks
Dim 14 nov 12:12:07 2021 | Rosetta@home | [sched_op] CPU work request: 0.00 seconds; 0.00 devices
Dim 14 nov 12:12:07 2021 | Rosetta@home | [sched_op] AMD/ATI GPU work request: 0.00 seconds; 0.00 devices
Dim 14 nov 12:12:14 2021 | Rosetta@home | Scheduler request completed
Dim 14 nov 12:12:14 2021 | Rosetta@home | [sched_op] Server version 707
Dim 14 nov 12:12:14 2021 | Rosetta@home | Project requested delay of 31 seconds
Dim 14 nov 12:12:14 2021 | Rosetta@home | [sched_op] Deferring communication for 00:00:31
Dim 14 nov 12:12:14 2021 | Rosetta@home | [sched_op] Reason: requested by project
Dim 14 nov 12:12:14 2021 |  | [work_fetch] Request work fetch: RPC complete
Dim 14 nov 12:12:19 2021 |  | choose_project(): 1636888339.635691
Dim 14 nov 12:12:19 2021 |  | [work_fetch] ------- start work fetch state -------
Dim 14 nov 12:12:19 2021 |  | [work_fetch] target work buffer: 43200.00 + 86400.00 sec
Dim 14 nov 12:12:19 2021 |  | [work_fetch] --- project states ---
Dim 14 nov 12:12:19 2021 | Rosetta@home | [work_fetch] REC 3867.255 prio -0.802 can't request work: too many runnable tasks (25.94 sec)
Dim 14 nov 12:12:19 2021 |  | [work_fetch] --- state for CPU ---
Dim 14 nov 12:12:19 2021 |  | [work_fetch] shortfall 1058762.19 nidle 7.00 saturated 0.00 busy 0.00
Dim 14 nov 12:12:19 2021 | Rosetta@home | [work_fetch] share 0.000
Dim 14 nov 12:12:19 2021 |  | [work_fetch] --- state for AMD/ATI GPU ---
Dim 14 nov 12:12:19 2021 |  | [work_fetch] shortfall 1521.56 nidle 0.00 saturated 128078.44 busy 0.00
Dim 14 nov 12:12:19 2021 | Rosetta@home | [work_fetch] share 0.000 no applications
Dim 14 nov 12:12:19 2021 |  | [work_fetch] ------- end work fetch state -------
Dim 14 nov 12:12:19 2021 | Rosetta@home | skip: too many runnable tasks
Dim 14 nov 12:12:19 2021 |  | [work_fetch] No project chosen for work fetch
Dim 14 nov 12:12:36 2021 |  | [work_fetch] Request work fetch: application exited
Dim 14 nov 12:12:39 2021 |  | choose_project(): 1636888359.985780
Dim 14 nov 12:12:39 2021 |  | [work_fetch] ------- start work fetch state -------
Dim 14 nov 12:12:39 2021 |  | [work_fetch] target work buffer: 43200.00 + 86400.00 sec
Dim 14 nov 12:12:39 2021 |  | [work_fetch] --- project states ---
Dim 14 nov 12:12:39 2021 | Rosetta@home | [work_fetch] REC 3867.423 prio -0.802 can't request work: too many runnable tasks (5.59 sec)
Dim 14 nov 12:12:39 2021 |  | [work_fetch] --- state for CPU ---
Dim 14 nov 12:12:39 2021 |  | [work_fetch] shortfall 1058781.68 nidle 7.00 saturated 0.00 busy 0.00
Dim 14 nov 12:12:39 2021 | Rosetta@home | [work_fetch] share 0.000
Dim 14 nov 12:12:39 2021 |  | [work_fetch] --- state for AMD/ATI GPU ---
Dim 14 nov 12:12:39 2021 |  | [work_fetch] shortfall 1544.87 nidle 0.00 saturated 128055.13 busy 0.00
Dim 14 nov 12:12:39 2021 | Rosetta@home | [work_fetch] share 0.000 no applications
Dim 14 nov 12:12:39 2021 |  | [work_fetch] ------- end work fetch state -------
Dim 14 nov 12:12:39 2021 | Rosetta@home | skip: too many runnable tasks
Dim 14 nov 12:12:39 2021 |  | [work_fetch] No project chosen for work fetch
Dim 14 nov 12:12:46 2021 |  | [work_fetch] Request work fetch: Backoff ended for Rosetta@home
Dim 14 nov 12:12:50 2021 |  | choose_project(): 1636888370.128871
Dim 14 nov 12:12:50 2021 |  | [work_fetch] ------- start work fetch state -------
Dim 14 nov 12:12:50 2021 |  | [work_fetch] target work buffer: 43200.00 + 86400.00 sec
Dim 14 nov 12:12:50 2021 |  | [work_fetch] --- project states ---
Dim 14 nov 12:12:50 2021 | Rosetta@home | [work_fetch] REC 3867.423 prio -0.841 can't request work: too many runnable tasks
Dim 14 nov 12:12:50 2021 |  | [work_fetch] --- state for CPU ---
Dim 14 nov 12:12:50 2021 |  | [work_fetch] shortfall 1058792.27 nidle 7.00 saturated 0.00 busy 0.00
Dim 14 nov 12:12:50 2021 | Rosetta@home | [work_fetch] share 0.000
Dim 14 nov 12:12:50 2021 |  | [work_fetch] --- state for AMD/ATI GPU ---
Dim 14 nov 12:12:50 2021 |  | [work_fetch] shortfall 1555.47 nidle 0.00 saturated 128044.53 busy 0.00
Dim 14 nov 12:12:50 2021 | Rosetta@home | [work_fetch] share 0.000 no applications
Dim 14 nov 12:12:50 2021 |  | [work_fetch] ------- end work fetch state -------
Dim 14 nov 12:12:50 2021 | Rosetta@home | skip: too many runnable tasks
Dim 14 nov 12:12:50 2021 |  | [work_fetch] No project chosen for work fetch
Dim 14 nov 12:12:59 2021 |  | Re-reading cc_config.xml
Dim 14 nov 12:12:59 2021 |  | Config: don't compute while Diablo III is running
Dim 14 nov 12:12:59 2021 |  | Config: don't use GPUs while Diablo III is running
Dim 14 nov 12:12:59 2021 |  | log flags: file_xfer, sched_ops, task
Dim 14 nov 12:12:59 2021 | Rosetta@home | Found app_config.xml


I understand this is because there are now too many, so it doesn't help ?

I would have to wait for one task to finish and be there "just at this moment" to monitor ?
Almost impossible for me... last 2 days I didn't check at all and I realize the bloody python task has now 4 days of processing stuck at 99,999% with 2 days over the deadline.......
ID: 103254 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [AF>Le_Pommier] Jerome_C2005

Send message
Joined: 22 Aug 06
Posts: 42
Credit: 1,258,039
RAC: 0
Message 103255 - Posted: 14 Nov 2021, 11:22:58 UTC - in response to Message 103254.  
Last modified: 14 Nov 2021, 11:28:10 UTC

I decided to cancel that 4 days running task (probably never going to complete)

Dim 14 nov 12:18:37 2021 |  | [work_fetch] Request work fetch: Core client configuration
Dim 14 nov 12:18:40 2021 |  | choose_project(): 1636888720.569206
Dim 14 nov 12:18:40 2021 |  | [work_fetch] ------- start work fetch state -------
Dim 14 nov 12:18:40 2021 |  | [work_fetch] target work buffer: 43200.00 + 86400.00 sec
Dim 14 nov 12:18:40 2021 |  | [work_fetch] --- project states ---
Dim 14 nov 12:18:40 2021 | Rosetta@home | [work_fetch] REC 3869.097 prio -0.841 can't request work: too many runnable tasks
Dim 14 nov 12:18:40 2021 |  | [work_fetch] --- state for CPU ---
Dim 14 nov 12:18:40 2021 |  | [work_fetch] shortfall 1059135.02 nidle 7.00 saturated 0.00 busy 0.00
Dim 14 nov 12:18:40 2021 | Rosetta@home | [work_fetch] share 0.000
Dim 14 nov 12:18:40 2021 |  | [work_fetch] --- state for AMD/ATI GPU ---
Dim 14 nov 12:18:40 2021 |  | [work_fetch] shortfall 1889.00 nidle 0.00 saturated 127711.00 busy 0.00
Dim 14 nov 12:18:40 2021 | Rosetta@home | [work_fetch] share 0.000 no applications
Dim 14 nov 12:18:40 2021 |  | [work_fetch] ------- end work fetch state -------
Dim 14 nov 12:18:40 2021 | Rosetta@home | skip: too many runnable tasks
Dim 14 nov 12:18:40 2021 |  | [work_fetch] No project chosen for work fetch
Dim 14 nov 12:18:45 2021 | Rosetta@home | task boinc_cages_IL_2727241_2142_0 aborted by user
Dim 14 nov 12:18:45 2021 | Rosetta@home | [sched_op] Deferring communication for 00:01:38

Dim 14 nov 12:18:45 2021 | Rosetta@home | [sched_op] Reason: Unrecoverable error for task boinc_cages_IL_2727241_2142_0
Dim 14 nov 12:18:45 2021 | | [work_fetch] Request work fetch: result aborted by user
Dim 14 nov 12:18:45 2021 |  | choose_project(): 1636888725.591459
Dim 14 nov 12:18:45 2021 |  | [work_fetch] ------- start work fetch state -------
Dim 14 nov 12:18:45 2021 |  | [work_fetch] target work buffer: 43200.00 + 86400.00 sec
Dim 14 nov 12:18:45 2021 |  | [work_fetch] --- project states ---
Dim 14 nov 12:18:45 2021 | Rosetta@home | [work_fetch] REC 3869.131 prio -0.841 can't request work: scheduler RPC backoff (97.99 sec)
Dim 14 nov 12:18:45 2021 |  | [work_fetch] --- state for CPU ---
Dim 14 nov 12:18:45 2021 |  | [work_fetch] shortfall 1048535.58 nidle 7.00 saturated 0.00 busy 0.00
Dim 14 nov 12:18:45 2021 | Rosetta@home | [work_fetch] share 0.000
Dim 14 nov 12:18:45 2021 |  | [work_fetch] --- state for AMD/ATI GPU ---
Dim 14 nov 12:18:45 2021 |  | [work_fetch] shortfall 1893.78 nidle 0.00 saturated 127706.22 busy 0.00
Dim 14 nov 12:18:45 2021 | Rosetta@home | [work_fetch] share 0.000 no applications
Dim 14 nov 12:18:45 2021 |  | [work_fetch] ------- end work fetch state -------
Dim 14 nov 12:18:45 2021 | Rosetta@home | skip: scheduler RPC backoff
Dim 14 nov 12:18:45 2021 |  | [work_fetch] No project chosen for work fetch
Dim 14 nov 12:18:53 2021 |  | [work_fetch] Request work fetch: application exited
Dim 14 nov 12:18:53 2021 | Rosetta@home | Computation for task boinc_cages_IL_2727241_2142_0 finished
Dim 14 nov 12:18:55 2021 |  | choose_project(): 1636888735.875360
Dim 14 nov 12:18:55 2021 |  | [work_fetch] ------- start work fetch state -------
Dim 14 nov 12:18:55 2021 |  | [work_fetch] target work buffer: 43200.00 + 86400.00 sec
Dim 14 nov 12:18:55 2021 |  | [work_fetch] --- project states ---
Dim 14 nov 12:18:55 2021 | Rosetta@home | [work_fetch] REC 3869.176 prio -0.802 can't request work: scheduler RPC backoff (87.70 sec)
Dim 14 nov 12:18:55 2021 |  | [work_fetch] --- state for CPU ---
Dim 14 nov 12:18:55 2021 |  | [work_fetch] shortfall 1048547.57 nidle 7.00 saturated 0.00 busy 0.00
Dim 14 nov 12:18:55 2021 | Rosetta@home | [work_fetch] share 0.000
Dim 14 nov 12:18:55 2021 |  | [work_fetch] --- state for AMD/ATI GPU ---
Dim 14 nov 12:18:55 2021 |  | [work_fetch] shortfall 1903.34 nidle 0.00 saturated 127696.66 busy 0.00
Dim 14 nov 12:18:55 2021 | Rosetta@home | [work_fetch] share 0.000 no applications
Dim 14 nov 12:18:55 2021 |  | [work_fetch] ------- end work fetch state -------
Dim 14 nov 12:18:55 2021 | Rosetta@home | skip: scheduler RPC backoff
Dim 14 nov 12:18:55 2021 |  | [work_fetch] No project chosen for work fetch
Dim 14 nov 12:19:56 2021 |  | choose_project(): 1636888796.737827
Dim 14 nov 12:19:56 2021 |  | [work_fetch] ------- start work fetch state -------
Dim 14 nov 12:19:56 2021 |  | [work_fetch] target work buffer: 43200.00 + 86400.00 sec
Dim 14 nov 12:19:56 2021 |  | [work_fetch] --- project states ---
Dim 14 nov 12:19:56 2021 | Rosetta@home | [work_fetch] REC 3869.459 prio -0.802 can't request work: scheduler RPC backoff (26.84 sec)
Dim 14 nov 12:19:56 2021 |  | [work_fetch] --- state for CPU ---
Dim 14 nov 12:19:56 2021 |  | [work_fetch] shortfall 1048619.42 nidle 7.00 saturated 0.00 busy 0.00
Dim 14 nov 12:19:56 2021 | Rosetta@home | [work_fetch] share 0.000
Dim 14 nov 12:19:56 2021 |  | [work_fetch] --- state for AMD/ATI GPU ---
Dim 14 nov 12:19:56 2021 |  | [work_fetch] shortfall 1960.34 nidle 0.00 saturated 127639.66 busy 0.00
Dim 14 nov 12:19:56 2021 | Rosetta@home | [work_fetch] share 0.000 no applications
Dim 14 nov 12:19:56 2021 |  | [work_fetch] ------- end work fetch state -------
Dim 14 nov 12:19:56 2021 | Rosetta@home | skip: scheduler RPC backoff
Dim 14 nov 12:19:56 2021 |  | [work_fetch] No project chosen for work fetch
Dim 14 nov 12:20:24 2021 |  | [work_fetch] Request work fetch: Backoff ended for Rosetta@home
Dim 14 nov 12:20:27 2021 | Rosetta@home | piggyback_work_request()
Dim 14 nov 12:20:27 2021 |  | [work_fetch] ------- start work fetch state -------
Dim 14 nov 12:20:27 2021 |  | [work_fetch] target work buffer: 43200.00 + 86400.00 sec
Dim 14 nov 12:20:27 2021 |  | [work_fetch] --- project states ---
Dim 14 nov 12:20:27 2021 | Rosetta@home | [work_fetch] REC 3869.459 prio -0.841 can request work
Dim 14 nov 12:20:27 2021 |  | [work_fetch] --- state for CPU ---
Dim 14 nov 12:20:27 2021 |  | [work_fetch] shortfall 1048655.18 nidle 7.00 saturated 0.00 busy 0.00
Dim 14 nov 12:20:27 2021 | Rosetta@home | [work_fetch] share 1.000
Dim 14 nov 12:20:27 2021 |  | [work_fetch] --- state for AMD/ATI GPU ---
Dim 14 nov 12:20:27 2021 |  | [work_fetch] shortfall 1989.37 nidle 0.00 saturated 127610.63 busy 0.00
Dim 14 nov 12:20:27 2021 | Rosetta@home | [work_fetch] share 0.000 no applications
Dim 14 nov 12:20:27 2021 |  | [work_fetch] ------- end work fetch state -------
Dim 14 nov 12:20:27 2021 | Rosetta@home | piggyback: resource CPU
Dim 14 nov 12:20:27 2021 | Rosetta@home | piggyback: collatz can't fetch work
Dim 14 nov 12:20:27 2021 | Rosetta@home | piggyback: Cosmology@Home can't fetch work
Dim 14 nov 12:20:27 2021 | Rosetta@home | piggyback: GPUGRID can't fetch work
Dim 14 nov 12:20:27 2021 | Rosetta@home | piggyback: iThena can't fetch work
Dim 14 nov 12:20:27 2021 | Rosetta@home | piggyback: Kryptos@Home can't fetch work
Dim 14 nov 12:20:27 2021 | Rosetta@home | piggyback: LHC@home can't fetch work
Dim 14 nov 12:20:27 2021 | Rosetta@home | piggyback: lhcathome-dev can't fetch work
Dim 14 nov 12:20:27 2021 | Rosetta@home | piggyback: Moo! Wrapper can't fetch work
Dim 14 nov 12:20:27 2021 | Rosetta@home | piggyback: PrimeGrid can't fetch work
Dim 14 nov 12:20:27 2021 | Rosetta@home | piggyback: QuChemPedIA@home can't fetch work
Dim 14 nov 12:20:27 2021 | Rosetta@home | piggyback: ralph@home can't fetch work
Dim 14 nov 12:20:27 2021 | Rosetta@home | piggyback: SRBase can't fetch work
Dim 14 nov 12:20:27 2021 | Rosetta@home | piggyback: TN-Grid Platform can't fetch work
Dim 14 nov 12:20:27 2021 | Rosetta@home | piggyback: yoyo@home can't fetch work
Dim 14 nov 12:20:27 2021 | Rosetta@home | piggyback: WUProp@Home can't fetch work
Dim 14 nov 12:20:27 2021 | Rosetta@home | [work_fetch] using MC shortfall 119018.758792 instead of shortfall 1048655.175950
Dim 14 nov 12:20:27 2021 | Rosetta@home | [work_fetch] set_request() for CPU: ninst 9 nused_total 1000.00 nidle_now 7.00 fetch share 1.00 req_inst 0.00 req_secs 119018.76
Dim 14 nov 12:20:27 2021 | Rosetta@home | piggyback: resource AMD/ATI GPU
Dim 14 nov 12:20:27 2021 | Rosetta@home | piggyback: can't fetch AMD/ATI GPU: no applications
Dim 14 nov 12:20:27 2021 | Rosetta@home | [sched_op] Starting scheduler request
Dim 14 nov 12:20:27 2021 | Rosetta@home | [work_fetch] request: CPU (119018.76 sec, 0.00 inst) AMD/ATI GPU (0.00 sec, 0.00 inst)
Dim 14 nov 12:20:28 2021 | Rosetta@home | Sending scheduler request: To report completed tasks.
Dim 14 nov 12:20:28 2021 | Rosetta@home | Reporting 1 completed tasks
Dim 14 nov 12:20:28 2021 | Rosetta@home | Requesting new tasks for CPU
Dim 14 nov 12:20:28 2021 | Rosetta@home | [sched_op] CPU work request: 119018.76 seconds; 0.00 devices
Dim 14 nov 12:20:28 2021 | Rosetta@home | [sched_op] AMD/ATI GPU work request: 0.00 seconds; 0.00 devices
Dim 14 nov 12:20:36 2021 | Rosetta@home | Scheduler request completed: got 1 new tasks
Dim 14 nov 12:20:36 2021 | Rosetta@home | [sched_op] Server version 707
Dim 14 nov 12:20:36 2021 | Rosetta@home | Project requested delay of 31 seconds
Dim 14 nov 12:20:36 2021 | Rosetta@home | [sched_op] estimated total CPU task duration: 29571 seconds
Dim 14 nov 12:20:36 2021 | Rosetta@home | [sched_op] estimated total AMD/ATI GPU task duration: 0 seconds
Dim 14 nov 12:20:36 2021 | Rosetta@home | [sched_op] handle_scheduler_reply(): got ack for task boinc_cages_IL_2727241_2142_0
Dim 14 nov 12:20:36 2021 | Rosetta@home | [sched_op] Deferring communication for 00:00:31
Dim 14 nov 12:20:36 2021 | Rosetta@home | [sched_op] Reason: requested by project
Dim 14 nov 12:20:36 2021 |  | [work_fetch] Request work fetch: RPC complete
Dim 14 nov 12:20:38 2021 | Rosetta@home | Started download of aaam-FPR-mALA_pp-SAR-AMACBEN2_pp_14.gz
Dim 14 nov 12:20:41 2021 |  | choose_project(): 1636888841.122885
Dim 14 nov 12:20:41 2021 |  | [work_fetch] ------- start work fetch state -------
Dim 14 nov 12:20:41 2021 |  | [work_fetch] target work buffer: 43200.00 + 86400.00 sec
Dim 14 nov 12:20:41 2021 |  | [work_fetch] --- project states ---
Dim 14 nov 12:20:41 2021 | Rosetta@home | [work_fetch] REC 3869.459 prio -0.803 can't request work: too many runnable tasks (25.88 sec)
Dim 14 nov 12:20:41 2021 |  | [work_fetch] --- state for CPU ---
Dim 14 nov 12:20:41 2021 |  | [work_fetch] shortfall 1048671.92 nidle 7.00 saturated 0.00 busy 0.00
Dim 14 nov 12:20:41 2021 | Rosetta@home | [work_fetch] share 0.000
Dim 14 nov 12:20:41 2021 |  | [work_fetch] --- state for AMD/ATI GPU ---
Dim 14 nov 12:20:41 2021 |  | [work_fetch] shortfall 2002.86 nidle 0.00 saturated 127597.14 busy 0.00
Dim 14 nov 12:20:41 2021 | Rosetta@home | [work_fetch] share 0.000 no applications
Dim 14 nov 12:20:41 2021 |  | [work_fetch] ------- end work fetch state -------
Dim 14 nov 12:20:41 2021 | Rosetta@home | skip: too many runnable tasks
Dim 14 nov 12:20:41 2021 |  | [work_fetch] No project chosen for work fetch
Dim 14 nov 12:20:45 2021 | Rosetta@home | Finished download of aaam-FPR-mALA_pp-SAR-AMACBEN2_pp_14.gz
Dim 14 nov 12:21:07 2021 |  | [work_fetch] Request work fetch: Backoff ended for Rosetta@home
Dim 14 nov 12:21:11 2021 |  | choose_project(): 1636888871.599030
Dim 14 nov 12:21:11 2021 |  | [work_fetch] ------- start work fetch state -------
Dim 14 nov 12:21:11 2021 |  | [work_fetch] target work buffer: 43200.00 + 86400.00 sec
Dim 14 nov 12:21:11 2021 |  | [work_fetch] --- project states ---
Dim 14 nov 12:21:11 2021 | Rosetta@home | [work_fetch] REC 3869.693 prio -0.841 can't request work: too many runnable tasks
Dim 14 nov 12:21:11 2021 |  | [work_fetch] --- state for CPU ---
Dim 14 nov 12:21:11 2021 |  | [work_fetch] shortfall 1048707.53 nidle 7.00 saturated 0.00 busy 0.00
Dim 14 nov 12:21:11 2021 | Rosetta@home | [work_fetch] share 0.000
Dim 14 nov 12:21:11 2021 |  | [work_fetch] --- state for AMD/ATI GPU ---
Dim 14 nov 12:21:11 2021 |  | [work_fetch] shortfall 2030.80 nidle 0.00 saturated 127569.20 busy 0.00
Dim 14 nov 12:21:11 2021 | Rosetta@home | [work_fetch] share 0.000 no applications
Dim 14 nov 12:21:11 2021 |  | [work_fetch] ------- end work fetch state -------
Dim 14 nov 12:21:11 2021 | Rosetta@home | skip: too many runnable tasks
Dim 14 nov 12:21:11 2021 |  | [work_fetch] No project chosen for work fetch


I hope in all this you can see useful information.
ID: 103255 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [AF>Le_Pommier] Jerome_C2005

Send message
Joined: 22 Aug 06
Posts: 42
Credit: 1,258,039
RAC: 0
Message 103258 - Posted: 14 Nov 2021, 14:38:39 UTC - in response to Message 103255.  

Oh well, I realize that there were a *second* python task that has already calculated 15 hours and was suspended hiding in the bushes, and now that I cancelled big sister it started to run exactly the same way, it has now ran 16 hours with a slooooowly progression %, and is is 2 days overdue !!!

That's too many problem with one project and 0 support - apart from you good people, thank you for this, but the admin on here doesn't give a damn.

I'm done with rosetta for now, we'll see in the future if they propose options that make sense and listen to their community.
ID: 103258 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 0
Message 103267 - Posted: 14 Nov 2021, 17:30:56 UTC - in response to Message 103258.  

There are certainly too many problems, but you don't need to waste so much time on them.
https://boinc.bakerlab.org/rosetta/forum_thread.php?id=6893&postid=103232#103232
ID: 103267 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [AF>Le_Pommier] Jerome_C2005

Send message
Joined: 22 Aug 06
Posts: 42
Credit: 1,258,039
RAC: 0
Message 103276 - Posted: 14 Nov 2021, 18:31:27 UTC

I'm not sure if you mean "don't bother too much and change project for now" or "there are useful tips in that other thread" ? I had a (very) quick look and it's long and complicated, not even sure if the problems they mention are similar ?

In any case it'll have to wait, for now I have dropped all those many tasks I had (running or not) and I'm focusing on other projects for the moment.

Thanks !
ID: 103276 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mikey
Avatar

Send message
Joined: 5 Jan 06
Posts: 1895
Credit: 9,169,305
RAC: 3,400
Message 103325 - Posted: 16 Nov 2021, 0:11:39 UTC - in response to Message 103276.  

I'm not sure if you mean "don't bother too much and change project for now" or "there are useful tips in that other thread" ? I had a (very) quick look and it's long and complicated, not even sure if the problems they mention are similar ?

In any case it'll have to wait, for now I have dropped all those many tasks I had (running or not) and I'm focusing on other projects for the moment.

Thanks !


Since each task here at Rosetta takes 8gb of ram and you only have 40gb of ram in that pc you need to limit the cache and extra cache settings on this pc and limit the total number of task you run at one time to no more than 4 at one time and depending on what else you do with that pc fewer than that. It seems to me with 20 cpu's in that pc and a cache size of even 1 day could mean 120 tasks if it used all the cpu cores at once. Of course you can't use 20 cores at once as your pc would be swapping memory like crazy and destroying your hard drives and pushing your memory to it's limit in the process.

One key thing, to me you said. it that it was only advancing at 0.01% even after 48 hours of crunching, that means you are trying to run to many tasks at the same time, again at 8gb of ram per task you can only 4 at a time max because the pc itself uses some overhead with Windows using more than most Linux versions.
ID: 103325 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [AF>Le_Pommier] Jerome_C2005

Send message
Joined: 22 Aug 06
Posts: 42
Credit: 1,258,039
RAC: 0
Message 103338 - Posted: 16 Nov 2021, 11:15:56 UTC - in response to Message 103325.  
Last modified: 16 Nov 2021, 11:16:08 UTC

On macOS those python VB tasks take much much less than 8 Gb (I would 13 Gb free when it was running) + as I explained, I had limited them and only 1 was running at a time (the rest was regular rosetta).
ID: 103338 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Falconet

Send message
Joined: 9 Mar 09
Posts: 353
Credit: 1,227,479
RAC: 1,013
Message 103339 - Posted: 16 Nov 2021, 11:27:46 UTC - in response to Message 103338.  

They reserve almost 8 GB of memory but they don't actually use anywhere near that. I haven't seen one using more than 500ish MB of RAM on Windows 10.
On my Ryzen 1400, Rosetta@home sends me 3 Python and 2 run at the same time plus 6 MCM tasks at WCG.
ID: 103339 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mikey
Avatar

Send message
Joined: 5 Jan 06
Posts: 1895
Credit: 9,169,305
RAC: 3,400
Message 103340 - Posted: 16 Nov 2021, 12:18:46 UTC - in response to Message 103338.  

On macOS those python VB tasks take much much less than 8 Gb (I would 13 Gb free when it was running) + as I explained, I had limited them and only 1 was running at a time (the rest was regular rosetta).


The regular Rosetta tasks also reserve 8gb of ram per task, so if you are trying to run 20 tasks total, or near that and all of them are Rosetta tasks your machine doesn't have enough memory. Yes I know Mac's use less memory but I don't remember how much, you can check by clicking on a running task in the Boinc Manager and then in the left hand panel click on Properties and it will tell you how much the Virtual Memory Size is and the Working set size is, you care about the one that shows the most memory.
ID: 103340 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [AF>Le_Pommier] Jerome_C2005

Send message
Joined: 22 Aug 06
Posts: 42
Credit: 1,258,039
RAC: 0
Message 103343 - Posted: 16 Nov 2021, 12:46:49 UTC

Well whatever the amount of memory really used or "reserved" or whatever, what is sure is that I had over 13 Gb of free memory available at that time.

Also I don't run 20 tasks on that machine : I have a linux VM with 10 thread for other boinc projects + I only let 9 threads available for boinc in macOS, so it was 8 regular rosetta + 1 python running in parallel, but now if you "officially" need 80 Gb of RAM for such a setup AND rosetta doesn't propose any setup in the preferences to limit this in any way, this is just a nonsense.
ID: 103343 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Falconet

Send message
Joined: 9 Mar 09
Posts: 353
Credit: 1,227,479
RAC: 1,013
Message 103345 - Posted: 16 Nov 2021, 13:59:46 UTC - in response to Message 103340.  

On macOS those python VB tasks take much much less than 8 Gb (I would 13 Gb free when it was running) + as I explained, I had limited them and only 1 was running at a time (the rest was regular rosetta).


The regular Rosetta tasks also reserve 8gb of ram per task, so if you are trying to run 20 tasks total, or near that and all of them are Rosetta tasks your machine doesn't have enough memory. Yes I know Mac's use less memory but I don't remember how much, you can check by clicking on a running task in the Boinc Manager and then in the left hand panel click on Properties and it will tell you how much the Virtual Memory Size is and the Working set size is, you care about the one that shows the most memory.



That can't be true otherwise my laptop with 7.2 GB of available RAM wouldn't be able to run 8 Rosetta tasks.
It wouldn't run or even receive Rosetta at all just like it doesn't run or receive Rosetta Python.
ID: 103345 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
1 · 2 · Next

Message boards : Number crunching : Excessive workunit fetch



©2024 University of Washington
https://www.bakerlab.org