Proxy to reduce downloads

Message boards : Number crunching : Proxy to reduce downloads

To post messages, you must log in.

AuthorMessage
Profile dcdc

Send message
Joined: 3 Nov 05
Posts: 1832
Credit: 119,821,902
RAC: 15,180
Message 48755 - Posted: 17 Nov 2007, 19:58:14 UTC
Last modified: 17 Nov 2007, 20:02:21 UTC

Hi all

i'm resurrecting an old project with some old redundant hardware. My plan is to have 4-8 minimal PCs in a box with each PSU shared between two PCs. They'll all be running on compactflash and running Damn Small Linux except the 'server' which is running XP Pro (i don't know linux well enough to set up ICS or something similar, and i've got XP down to 75MB RAM).

I'll be running these comps in place of an electric heater, so there's no CO² issues!

I think this will have to connect to the internet using dial-up so i want to reduce the downloads as much as possible. I think I'll be running 24 or 48hr work units, but need to minimise downloads as much as possible. I put a request in on the BOINC dev boards for a folder that all big downloads are copied to which can then be checked before a download is initiated - that'd save a lot of bandwidth, both for users with lots of hosts and of course on the project's side.

Anyway, until that's implemented, the question is: Can I set up a proxy in XP that'll store the big rosetta files to stop repeat downloads?

Any suggestions welcomed!

cheers
Danny
ID: 48755 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
hugothehermit

Send message
Joined: 26 Sep 05
Posts: 238
Credit: 314,893
RAC: 0
Message 48756 - Posted: 17 Nov 2007, 21:05:38 UTC

I'm not sure under XP, but you could always use SmoothWall it's a firewall, proxy, DHCP server etc...

Just run your computers on dynamic IPs, run SmoothWall as a DHCP server with Web proxy settings as transparent and with the cache XX size set as enabled.

I run SmoothWall 2.0, the new one is 3.0 and it might be a little different, but the documentation was good with 2.0 so I would assume that 3.0 would be just a good.

One thing to remember is that when you boot from the SmoothWall setup disk it will erase the HDD completely then repartition it to it's heart content.

ID: 48756 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile dcdc

Send message
Joined: 3 Nov 05
Posts: 1832
Credit: 119,821,902
RAC: 15,180
Message 48757 - Posted: 17 Nov 2007, 21:54:24 UTC - in response to Message 48756.  
Last modified: 17 Nov 2007, 21:55:16 UTC

I'm not sure under XP, but you could always use SmoothWall it's a firewall, proxy, DHCP server etc...

Just run your computers on dynamic IPs, run SmoothWall as a DHCP server with Web proxy settings as transparent and with the cache XX size set as enabled.

I run SmoothWall 2.0, the new one is 3.0 and it might be a little different, but the documentation was good with 2.0 so I would assume that 3.0 would be just a good.

One thing to remember is that when you boot from the SmoothWall setup disk it will erase the HDD completely then repartition it to it's heart content.


ah... looks interesting! As this is gonna have to be on dial-up i guess one of the key factors is whether i can get a modem working with it. I've got a choice of three here but i couldn't find drivers for two of them for XP! I'll investigate - cheers hugo.

one question - how much RAM does the version you're running use? i've got 256MB in this machine - will that be enough?
ID: 48757 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile Feet1st
Avatar

Send message
Joined: 30 Dec 05
Posts: 1755
Credit: 4,690,520
RAC: 0
Message 48758 - Posted: 17 Nov 2007, 22:24:39 UTC

dcdc, in addition to the ideas here, I've been working on a cacheing proxy. Still making changes and burning it in, so it's not yet ready for prime time, but it should give you exactly what you are looking for. Shoot me an EMail and we can compare notes.

I can't recall if you joined the Yahoo group or not. I'm hoping to work with members of the group to further test my code. But you can use this EMail addr:

HelpOutRosetta-owner at NOyahooSPgroAMups.comERS
(remove NOSPAMERS from the domain name).
Add this signature to your EMail:
Running Microsoft's "System Idle Process" will never help cure cancer, AIDS nor Alzheimer's. But running Rosetta@home just might!
https://boinc.bakerlab.org/rosetta/
ID: 48758 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile dcdc

Send message
Joined: 3 Nov 05
Posts: 1832
Credit: 119,821,902
RAC: 15,180
Message 48760 - Posted: 17 Nov 2007, 23:15:19 UTC

sent ;)
ID: 48760 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
hugothehermit

Send message
Joined: 26 Sep 05
Posts: 238
Credit: 314,893
RAC: 0
Message 48761 - Posted: 18 Nov 2007, 3:04:19 UTC - in response to Message 48757.  
Last modified: 18 Nov 2007, 3:46:39 UTC


ah... looks interesting! As this is gonna have to be on dial-up i guess one of the key factors is whether i can get a modem working with it. I've got a choice of three here but i couldn't find drivers for two of them for XP! I'll investigate - cheers hugo.

one question - how much RAM does the version you're running use? i've got 256MB in this machine - will that be enough?

My SmoothWall machine
http://stats.kwsn.net/host.php?proj=all&hostid[]=5356007

P3 500MHz
128MB RAM
IDE ~450MB HDD (this is too small, but I had it in the back of draw so to speak)
Generic old NIC (was auto detected) hooked to 10/100 4 port hub.
1.44 floppy
IDE CD-ROM

It's hooked to a 128k ISDN modem through Com1 D9 pin 115200 baud rate.

External modems work without drivers just esc strings mine is

+++ATZ!T0=45!T1=20!T4=1

what that works out to my modem is
+++ esc
AT attention modem
Z initialise
!T0=45 add B channel if more 45kbs (dial second telephone line and add it to the internet)
!T1=20 drop B channel if less than 20kbs ( hang up second telephone line)
!T4=1 enable DVO (Data Voice Override) (second telephone line automatically hangs up to allow people to ring out and in)

I should also add that I use this all the time, once the machine is setup it will give internet access to anything I've put on it 98SE, XP, Vista, Linux.

I used to run ICS on windows then found something called SOHO something or was that something SOHO, doesn't matter SmoothWall it's better than sliced bread if you ask me.



Edit: because the url tags don't like ...hostid[]=535... in it.
Edit: for the modem stuff.
Edit: more added
Edit: spellnig loots off it :)
ID: 48761 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
mdettweiler
Avatar

Send message
Joined: 15 Oct 06
Posts: 33
Credit: 2,509
RAC: 0
Message 48763 - Posted: 18 Nov 2007, 4:30:13 UTC - in response to Message 48757.  

I'm not sure under XP, but you could always use SmoothWall it's a firewall, proxy, DHCP server etc...

Just run your computers on dynamic IPs, run SmoothWall as a DHCP server with Web proxy settings as transparent and with the cache XX size set as enabled.

I run SmoothWall 2.0, the new one is 3.0 and it might be a little different, but the documentation was good with 2.0 so I would assume that 3.0 would be just a good.

One thing to remember is that when you boot from the SmoothWall setup disk it will erase the HDD completely then repartition it to it's heart content.


ah... looks interesting! As this is gonna have to be on dial-up i guess one of the key factors is whether i can get a modem working with it. I've got a choice of three here but i couldn't find drivers for two of them for XP! I'll investigate - cheers hugo.

one question - how much RAM does the version you're running use? i've got 256MB in this machine - will that be enough?

I've checked out Smoothwall before (actually installed it in Virtual PC), and it seems that it's supposed to work very well with dial-up modems. Also, it doesn't need much RAM--I think the minimum is 32MB.
ID: 48763 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Klimax

Send message
Joined: 27 Apr 07
Posts: 44
Credit: 2,800,788
RAC: 184
Message 48764 - Posted: 18 Nov 2007, 7:19:43 UTC

It is interesting how people from different places can have same idea.

What I would suggest is to try network boot,so you can dump compactflash in favour of one bigger HDD in server.That is the only part,which was showstopper for me...(Unable to figure out how and no time to do experiments)
ID: 48764 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile dcdc

Send message
Joined: 3 Nov 05
Posts: 1832
Credit: 119,821,902
RAC: 15,180
Message 48767 - Posted: 18 Nov 2007, 12:15:06 UTC - in response to Message 48764.  

It is interesting how people from different places can have same idea.

What I would suggest is to try network boot,so you can dump compactflash in favour of one bigger HDD in server.That is the only part,which was showstopper for me...(Unable to figure out how and no time to do experiments)

netbooting would be good - BitSpit on my team (XPC) netboots his crunchers - but the hardware is all different so i'd have to configure it for each machine, and if there's any network interruptions it tends to need resetting. Also, i don't think the HD is quick enough to serve 8 machines as it's a 4300RPM drive from an old laptop.

I am running out of CF-IDE adapters though so if i've got any lan cards with a boot rom on them (i believe that's pretty much all that's needed?) then I might have a look at getting a couple of them netbooting so there's not too much load on the HD. If i've got two identical motherboards i'll have a look at it...
ID: 48767 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote

Message boards : Number crunching : Proxy to reduce downloads



©2024 University of Washington
https://www.bakerlab.org