Multiattach mode disk images

Message boards : Number crunching : Multiattach mode disk images

To post messages, you must log in.

1 · 2 · 3 · Next

AuthorMessage
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1994
Credit: 9,551,716
RAC: 6,403
Message 104952 - Posted: 18 Feb 2022, 21:24:47 UTC

I hope Rosetta@Home will soon implement this new function of Boinc's wrapper.
This would give a great advantage to the project!!
ID: 104952 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Grant (SSSF)

Send message
Joined: 28 Mar 20
Posts: 1675
Credit: 17,740,150
RAC: 22,892
Message 104954 - Posted: 18 Feb 2022, 21:45:55 UTC

Multiattach mode images can be attached to more than one virtual machine at the same time, even if these machines are running simultaneously. For each virtual machine to which such an image is attached, a differencing image is created. As a result, data that is written to such a virtual disk by one machine is not seen by the other machines to which the image is attached. Each machine creates its own write history of the multiattach image.

Technically, a multiattach image behaves identically to an immutable image except the differencing image is not reset every time the machine starts.

This mode is useful for sharing files which are almost never written, for instance picture galleries, where every guest changes only a small amount of data and the majority of the disk content remains unchanged. The modified blocks are stored in differencing images which remain relatively small and the shared content is stored only once at the host.
Too much to hope for.
It would certainly end the massive disk activity needed for Python Tasks by not having to write whole the whole disk image every time a new Task starts, but just the data that has changed from the previous image.
ie the the ID number for the new VM, and the new Task data files. All the other files are unchanged (the VM itself & the Python executables & support files).
Grant
Darwin NT
ID: 104954 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1994
Credit: 9,551,716
RAC: 6,403
Message 104956 - Posted: 18 Feb 2022, 22:10:40 UTC - in response to Message 104954.  

Too much to hope for. All the other files are unchanged (the VM itself & the Python executables & support files).


I have over 42 gb of space used by Rosetta and everytime a python wu starts it's a lot of disk usage (computers without SSD....a nightmare).
A simple update to boinc infrasctucture solve two problems: space and performances.
ID: 104956 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile dcdc

Send message
Joined: 3 Nov 05
Posts: 1831
Credit: 119,526,853
RAC: 6,993
Message 104999 - Posted: 19 Feb 2022, 13:58:04 UTC

This would make a huge difference to throughput, and presumably to the impact on users and to drive wear too.

Any idea what is required to implement it? A yet-to-be released version of the BOINC client, or just an updated wrapper file? And what do the project need to do to implement this?
ID: 104999 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
computezrmle

Send message
Joined: 9 Dec 11
Posts: 63
Credit: 9,680,103
RAC: 0
Message 105000 - Posted: 19 Feb 2022, 14:43:39 UTC - in response to Message 104999.  

The project team would have to provide a slightly modified app_version including:
- the new vboxwrapper; Either a self compiled version from the github source or an officially released version (don't know if/when BOINC will provide it independend of a full release).
- a new job template without the copy flag (to ensure the client does not copy the vdi to the worker slot)

Everything else is done by standard functions VirtualBox has built in for many years.



Additional measures

Although it's not a must to change the vdi image (AIMNet_vm_v2.vdi) I would suggest to do some cleanups since it still wastes 253 MB in /usr/lib/modules/4.19.0-13-amd64 from an older kernel which is not in use any more and 514 MB in /var/cache/apt/archives which is only required during installation.

After the cleanup unused space needs to be overwritten with zeros and the new vdi needs to be compacted with "vboxmanage ... --compact".
That way the size shrinks from 6.9 GB to 4.2 GB.


I also suggest to switch the VM's virtual disk adapter to sata/IntelAHCI (via vbox_job.xml) and to get rid of forcing a VM snapshot every 10 min.
See:
https://boinc.berkeley.edu/trac/wiki/VboxApps
ID: 105000 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1994
Credit: 9,551,716
RAC: 6,403
Message 105047 - Posted: 20 Feb 2022, 9:53:17 UTC - in response to Message 105000.  

The project team would have to provide a slightly modified app_version including:
- the new vboxwrapper; Either a self compiled version from the github source or an officially released version (don't know if/when BOINC will provide it independend of a full release).
- a new job template without the copy flag (to ensure the client does not copy the vdi to the worker slot)


Your explanation is very clear
ID: 105047 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1994
Credit: 9,551,716
RAC: 6,403
Message 105350 - Posted: 7 Mar 2022, 13:23:47 UTC

Any news???
ID: 105350 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
computezrmle

Send message
Joined: 9 Dec 11
Posts: 63
Credit: 9,680,103
RAC: 0
Message 105351 - Posted: 7 Mar 2022, 13:50:02 UTC - in response to Message 105350.  

Required steps

1.
A premade vboxwrapper executable needs to be created.
See: https://github.com/BOINC/boinc/issues/4649

Or
The Rosetta team distributes a self compiled vboxwrapper (I doubt they will).


2.
The Rosetta team will have to create a new app version based on that vboxwrapper
ID: 105351 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1994
Credit: 9,551,716
RAC: 6,403
Message 105353 - Posted: 7 Mar 2022, 21:25:20 UTC - in response to Message 105351.  

Required steps
1.
A premade vboxwrapper executable needs to be created.
See: https://github.com/BOINC/boinc/issues/4649

2.
The Rosetta team will have to create a new app version based on that vboxwrapper


Seems not so difficult...
ID: 105353 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1994
Credit: 9,551,716
RAC: 6,403
Message 105728 - Posted: 28 Mar 2022, 8:14:06 UTC - in response to Message 105353.  

Seems not so difficult...


Indeed....they did nothing.... :-(
ID: 105728 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
computezrmle

Send message
Joined: 9 Dec 11
Posts: 63
Credit: 9,680,103
RAC: 0
Message 105729 - Posted: 28 Mar 2022, 8:45:32 UTC - in response to Message 105728.  

The github issue regarding Step 1 has recently been added to the next BOINC client milestone.
That's a good sign.


Step 2 is a Rosetta task, so :-((
ID: 105729 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1994
Credit: 9,551,716
RAC: 6,403
Message 105736 - Posted: 28 Mar 2022, 13:10:43 UTC - in response to Message 105729.  
Last modified: 28 Mar 2022, 13:13:08 UTC

The github issue regarding Step 1 has recently been added to the next BOINC client milestone.
That's a good sign.


Great!! Maybe other projects can use this feature...
ID: 105736 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
rjs5

Send message
Joined: 22 Nov 10
Posts: 273
Credit: 22,998,797
RAC: 6,223
Message 105746 - Posted: 28 Mar 2022, 19:31:53 UTC - in response to Message 105736.  

The github issue regarding Step 1 has recently been added to the next BOINC client milestone.
That's a good sign.


Great!! Maybe other projects can use this feature...


You can also download other premade vbox images of the Rosetta environment.
https://www.osboxes.org/virtualbox-images/
ID: 105746 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
computezrmle

Send message
Joined: 9 Dec 11
Posts: 63
Credit: 9,680,103
RAC: 0
Message 105747 - Posted: 28 Mar 2022, 19:48:11 UTC - in response to Message 105746.  

You can also download other premade vbox images of the Rosetta environment.
https://www.osboxes.org/virtualbox-images/

VM images from that website have nothing to do with Rosetta.
They just offer plain (Linux) VMs without the required scientific environment.
ID: 105747 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1994
Credit: 9,551,716
RAC: 6,403
Message 105937 - Posted: 14 Apr 2022, 16:02:57 UTC - in response to Message 105729.  

The github issue regarding Step 1 has recently been added to the next BOINC client milestone.
That's a good sign.


Yes, is passed from "Backlog Prioritized" to "In Progress".
ID: 105937 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
computezrmle

Send message
Joined: 9 Dec 11
Posts: 63
Credit: 9,680,103
RAC: 0
Message 106376 - Posted: 10 Jun 2022, 17:36:38 UTC

BOINC recently published vboxwrapper 26204 which introduces the code to deal with differencing images.
Downloads:
https://boinc.berkeley.edu/dl/vboxwrapper_26204_windows_x86_64.zip
https://boinc.berkeley.edu/dl/vboxwrapper_26204_x86_64-apple-darwin.zip
https://boinc.berkeley.edu/dl/vboxwrapper_26204_x86_64-pc-linux-gnu


It's now up to the Rosetta developers to create new app versions that make use of it.


Hint:
https://github.com/BOINC/boinc/issues/4602
ID: 106376 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1994
Credit: 9,551,716
RAC: 6,403
Message 106383 - Posted: 11 Jun 2022, 14:00:12 UTC - in response to Message 106376.  

BOINC recently published vboxwrapper 26204 which introduces the code to deal with differencing images.

It's now up to the Rosetta developers to create new app versions that make use of it.
Hint:
https://github.com/BOINC/boinc/issues/4602


Great news!!!
ID: 106383 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Jim1348

Send message
Joined: 19 Jan 06
Posts: 881
Credit: 52,257,545
RAC: 0
Message 106384 - Posted: 11 Jun 2022, 17:07:49 UTC - in response to Message 106376.  

That would solve a lot of problems at once. Thanks for your efforts.
ID: 106384 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1994
Credit: 9,551,716
RAC: 6,403
Message 106387 - Posted: 14 Jun 2022, 7:29:51 UTC

Atlas@Home introduced yesterday the new wrapper as beta test
ID: 106387 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile [VENETO] boboviz

Send message
Joined: 1 Dec 05
Posts: 1994
Credit: 9,551,716
RAC: 6,403
Message 106389 - Posted: 14 Jun 2022, 15:31:47 UTC - in response to Message 106387.  
Last modified: 14 Jun 2022, 15:31:59 UTC

Atlas@Home introduced yesterday the new wrapper as beta test


Now also CMS@Home and Test4Theory have new versions with the new wrapper (in test on boinc lhc-dev site)
ID: 106389 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
1 · 2 · 3 · Next

Message boards : Number crunching : Multiattach mode disk images



©2024 University of Washington
https://www.bakerlab.org