Save ~50% of bandwidth :) Switching from gzip to PPMD

Author	Message
Piotr Skrodzewicz Send message Joined: 21 Oct 06 Posts: 19 Credit: 278,430 RAC: 0	Message 37906 - Posted: 17 Mar 2007, 11:48:14 UTC Last modified: 17 Mar 2007, 12:20:39 UTC ID: 37906 · Rating: 2 · rate: / Reply Quote

Christoph Send message Joined: 10 Dec 05 Posts: 57 Credit: 1,512,386 RAC: 0	Message 37912 - Posted: 17 Mar 2007, 14:51:25 UTC Last modified: 17 Mar 2007, 15:16:05 UTC Normally LZMA has better compression than PPMD. PPMD is good for text files, LZMA for binaries. I think there's only a LZMA SDK, no PPMD SDK. You can find it here. Anyway, that big file bbdep02.May.sortlib is only donwloaded once, so those 2~3 MB won't really matter. ;) bbdep02.May.7z - 3599799 bytes (3.5mb/sec compression speed - Core Duo 2Ghz, PPMD 16mb dictionary size). bbdep02.May.rk - 3030160 bytes (8 min compression/10 sec decompression - P3 1.5 MHz). - YEAH, RK even beats PPMD !!! ID: 37912 · Rating: 0 · rate: / Reply Quote

FluffyChicken Send message Joined: 1 Nov 05 Posts: 1260 Credit: 369,635 RAC: 0	Message 37924 - Posted: 17 Mar 2007, 17:59:10 UTC Rosetta know about 7-ZIP since it's what brought up shortly after they started. Will the reading from the file effect rosetta's performance (since that is done more often than the single download of it. The real files to be attacked with decent compression (they can even use ZIP since that is in the boinc source) 1 to 4MB data files that come down with for each target. Team mauisun.org ID: 37924 · Rating: 0 · rate: / Reply Quote

Piotr Skrodzewicz Send message Joined: 21 Oct 06 Posts: 19 Credit: 278,430 RAC: 0	Message 37977 - Posted: 18 Mar 2007, 20:49:59 UTC - in response to Message 37912. Last modified: 18 Mar 2007, 21:00:53 UTC Normally LZMA has better compression than PPMD. PPMD is good for text files, LZMA for binaries. I think there's only a LZMA SDK, no PPMD SDK. You can find it here. Anyway, that big file bbdep02.May.sortlib is only donwloaded once, so those 2~3 MB won't really matter. ;) bbdep02.May.7z - 3599799 bytes (3.5mb/sec compression speed - Core Duo 2Ghz, PPMD 16mb dictionary size). bbdep02.May.rk - 3030160 bytes (8 min compression/10 sec decompression - P3 1.5 MHz). - YEAH, RK even beats PPMD !!! 1. All project files (exept exes) are in fact text files, so compression format change will still give ~50% gain. 2. There is no PPMD SDK AFAIK, but how about implementing PPMD w/o SDK ? 3. If PPMD is too big problem, try LZMA. 4. RK is AFAIK closed source. EDIT: bbdep02.May.sortlib.bz2 - 3688826 bytes - but bzip2 is slow... bbdep02.May.sortlib.7z - 2949796 bytes !!! - 7-ZIP LZMA 16mb dictionary. Forget RK, forget Winrar, forget PPMD - use 7-zip LZMA !!! Both Winrar and 7-zip use LZMA, so I thought that 7-zip will not be noticeably better than Winrar... I was wrong. 7-ZIP LZMA destroys all competition. So, don't wait, switch to LZMA ASAP !!! ID: 37977 · Rating: 1 · rate: / Reply Quote

netwraith Send message Joined: 3 Sep 06 Posts: 80 Credit: 13,483,227 RAC: 0	Message 37982 - Posted: 18 Mar 2007, 22:07:02 UTC -- Just remember.. it should be something that all Rosetta O.S. supports have in the O.S. by default... or be something that gets downloaded with the Rosetta binary... Downloading would create a bunch of headaches with supporting multiple version of GLIBC on linux... Not downloading would require linuces to download and install additional software in order to run Rosetta (would likely have chilling effect on the number of Linux crunchers)... Would probably be best to leave well enough alone.... *Looking for a team ??? Join BoincSynergy!!* ID: 37982 · Rating: 0 · rate: / Reply Quote

Piotr Skrodzewicz Send message Joined: 21 Oct 06 Posts: 19 Credit: 278,430 RAC: 0	Message 38012 - Posted: 19 Mar 2007, 14:57:17 UTC - in response to Message 37982. -- Just remember.. it should be something that all Rosetta O.S. supports have in the O.S. by default... or be something that gets downloaded with the Rosetta binary... Downloading would create a bunch of headaches with supporting multiple version of GLIBC on linux... Not downloading would require linuces to download and install additional software in order to run Rosetta (would likely have chilling effect on the number of Linux crunchers)... Would probably be best to leave well enough alone.... IMO, you are wrong. Just implement LZMA using http://www.7-zip.org/sdk.html into rosetta executable. EXE will handle compression. No additional software needed. Renderfarm@Home uses LZMA... ID: 38012 · Rating: 0 · rate: / Reply Quote