Message boards : Number crunching : Multiattach mode disk images
Author | Message |
---|---|
[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,602,405 RAC: 8,832 |
I hope Rosetta@Home will soon implement this new function of Boinc's wrapper. This would give a great advantage to the project!! |
Grant (SSSF) Send message Joined: 28 Mar 20 Posts: 1679 Credit: 17,817,112 RAC: 22,737 |
Multiattach mode images can be attached to more than one virtual machine at the same time, even if these machines are running simultaneously. For each virtual machine to which such an image is attached, a differencing image is created. As a result, data that is written to such a virtual disk by one machine is not seen by the other machines to which the image is attached. Each machine creates its own write history of the multiattach image.Too much to hope for. It would certainly end the massive disk activity needed for Python Tasks by not having to write whole the whole disk image every time a new Task starts, but just the data that has changed from the previous image. ie the the ID number for the new VM, and the new Task data files. All the other files are unchanged (the VM itself & the Python executables & support files). Grant Darwin NT |
[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,602,405 RAC: 8,832 |
Too much to hope for. All the other files are unchanged (the VM itself & the Python executables & support files). I have over 42 gb of space used by Rosetta and everytime a python wu starts it's a lot of disk usage (computers without SSD....a nightmare). A simple update to boinc infrasctucture solve two problems: space and performances. |
dcdc Send message Joined: 3 Nov 05 Posts: 1831 Credit: 119,603,398 RAC: 11,044 |
This would make a huge difference to throughput, and presumably to the impact on users and to drive wear too. Any idea what is required to implement it? A yet-to-be released version of the BOINC client, or just an updated wrapper file? And what do the project need to do to implement this? |
computezrmle Send message Joined: 9 Dec 11 Posts: 63 Credit: 9,680,103 RAC: 0 |
The project team would have to provide a slightly modified app_version including: - the new vboxwrapper; Either a self compiled version from the github source or an officially released version (don't know if/when BOINC will provide it independend of a full release). - a new job template without the copy flag (to ensure the client does not copy the vdi to the worker slot) Everything else is done by standard functions VirtualBox has built in for many years. Additional measures Although it's not a must to change the vdi image (AIMNet_vm_v2.vdi) I would suggest to do some cleanups since it still wastes 253 MB in /usr/lib/modules/4.19.0-13-amd64 from an older kernel which is not in use any more and 514 MB in /var/cache/apt/archives which is only required during installation. After the cleanup unused space needs to be overwritten with zeros and the new vdi needs to be compacted with "vboxmanage ... --compact". That way the size shrinks from 6.9 GB to 4.2 GB. I also suggest to switch the VM's virtual disk adapter to sata/IntelAHCI (via vbox_job.xml) and to get rid of forcing a VM snapshot every 10 min. See: https://boinc.berkeley.edu/trac/wiki/VboxApps |
[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,602,405 RAC: 8,832 |
The project team would have to provide a slightly modified app_version including: Your explanation is very clear |
[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,602,405 RAC: 8,832 |
Any news??? |
computezrmle Send message Joined: 9 Dec 11 Posts: 63 Credit: 9,680,103 RAC: 0 |
Required steps 1. A premade vboxwrapper executable needs to be created. See: https://github.com/BOINC/boinc/issues/4649 Or The Rosetta team distributes a self compiled vboxwrapper (I doubt they will). 2. The Rosetta team will have to create a new app version based on that vboxwrapper |
[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,602,405 RAC: 8,832 |
Required steps Seems not so difficult... |
[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,602,405 RAC: 8,832 |
Seems not so difficult... Indeed....they did nothing.... :-( |
computezrmle Send message Joined: 9 Dec 11 Posts: 63 Credit: 9,680,103 RAC: 0 |
The github issue regarding Step 1 has recently been added to the next BOINC client milestone. That's a good sign. Step 2 is a Rosetta task, so :-(( |
[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,602,405 RAC: 8,832 |
The github issue regarding Step 1 has recently been added to the next BOINC client milestone. Great!! Maybe other projects can use this feature... |
rjs5 Send message Joined: 22 Nov 10 Posts: 273 Credit: 23,033,636 RAC: 7,361 |
The github issue regarding Step 1 has recently been added to the next BOINC client milestone. You can also download other premade vbox images of the Rosetta environment. https://www.osboxes.org/virtualbox-images/ |
computezrmle Send message Joined: 9 Dec 11 Posts: 63 Credit: 9,680,103 RAC: 0 |
You can also download other premade vbox images of the Rosetta environment. VM images from that website have nothing to do with Rosetta. They just offer plain (Linux) VMs without the required scientific environment. |
[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,602,405 RAC: 8,832 |
The github issue regarding Step 1 has recently been added to the next BOINC client milestone. Yes, is passed from "Backlog Prioritized" to "In Progress". |
computezrmle Send message Joined: 9 Dec 11 Posts: 63 Credit: 9,680,103 RAC: 0 |
BOINC recently published vboxwrapper 26204 which introduces the code to deal with differencing images. Downloads: https://boinc.berkeley.edu/dl/vboxwrapper_26204_windows_x86_64.zip https://boinc.berkeley.edu/dl/vboxwrapper_26204_x86_64-apple-darwin.zip https://boinc.berkeley.edu/dl/vboxwrapper_26204_x86_64-pc-linux-gnu It's now up to the Rosetta developers to create new app versions that make use of it. Hint: https://github.com/BOINC/boinc/issues/4602 |
[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,602,405 RAC: 8,832 |
BOINC recently published vboxwrapper 26204 which introduces the code to deal with differencing images. Great news!!! |
Jim1348 Send message Joined: 19 Jan 06 Posts: 881 Credit: 52,257,545 RAC: 0 |
That would solve a lot of problems at once. Thanks for your efforts. |
[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,602,405 RAC: 8,832 |
Atlas@Home introduced yesterday the new wrapper as beta test |
[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,602,405 RAC: 8,832 |
Atlas@Home introduced yesterday the new wrapper as beta test Now also CMS@Home and Test4Theory have new versions with the new wrapper (in test on boinc lhc-dev site) |
Message boards :
Number crunching :
Multiattach mode disk images
©2024 University of Washington
https://www.bakerlab.org