Latency over exceptionally wide area network...guesses?

[obob]

This isn't technical support because it isn't a problem, it's more of a curiosity.

I'm wondering what kind of connection this network is going to be backboned with, and what kinds of latency I should expect:

to give some backround on the network we're talking about, it covers 2 or 3 cities, with about 20 miles between them, and I've never noticed exceptionally slow connections when trying to access stuff at that distance, although i've never ping'd anything to get latency responses.

So on the presumption that this is just one big VPN connecting multiple facilities, what kind of latency is it likely going to be kicking back?

Also, at what point does grid computing start to hate you for latency, I know that GigE on shorter leads (under 100ft) can maintain around half-ms latency, but what kind of latency should I be considering on this VPN? 5ms? 10ms? higher?

And yes, before anyone asks, I'm looking at this network as a potential canidate for a very large scale grid computing system, which, if ever completed would be comprised of something around 24,000 CPU's (mostly Pentium 4 HT's (3GHZ+) and Athlon64's (3000+ or higher)) now while this may seem impressive, poor interconnect latency would kill the entire thing, because it'd spend so much time waiting on the network (while systems of similar or lesser size are usually backboned on something like Myrinet or InfiniBand, which provide exceptionally low latency and exceptionally high bandwidth).

SO!
If anyone has some guesses on where the performance will actually sit based on the higher latency of the VPN, and how well a VPN will handle traffic best described as "very chatty, but not much to say" (lots of packets but very small) that'd be helpful, also, anyone know of MOSIX having a top end for nodes (or of a mainstream (including Gentoo) having a top end for connected nodes?)

Also, 24,000 CPU's sounds a lot higher now that I throw it out there, so my estimation may be a bit high (give or take maybe 5000, and yes that's huge, but I can't verify exact #'s as I haven't seen every single system on this network).

Also, anyone know how much it would cost, just guesstimate, to rent processing time on a system that big? (it'd be in the 100k GFlop range assuming latency doesn't rape it)

[Orethrius]

The only advice I can offer is to take the WorldCommunityGrid approach.  That is, assign work to individual processors, let those LANs duke out the results, then have them batch-submit back to a central server.  Attempting full-duplex on a latency like that would likely rape the entire project.

[Pathos]

I have no idea about the logistics but ...

If latency could be an issue you may just have to build that assumption into the application by making requests/sending unrequested data in anticipation for an underflow of data while the node is still processing. Basically a bandwidth/latency tradeoff.

If you are worried about latency then I assume the application must be waiting on a packet.

Its a bad idea having a machine on a distributed system idle, waiting for a return packet as it reduces the efficiency of the system. If its a 10% hit then thats the processing power of 2400 computers wasted.

Is it impossible for the application to not rely on latency ?

[Lead Head]

im going to take a stab in the dark and say the average latency will be about 3-5ms, i have gotten 20ms pings on CSS from rhode island to a newjersey/new york server

[obob]

im going to take a stab in the dark and say the average latency will be about 3-5ms, i have gotten 20ms pings on CSS from rhode island to a newjersey/new york server

That does me nothing, you're talking about game servers which are probably running on 100mbit pipes, I'm talking about a VPN which is probably running on a 5mbit connection on hardware that can date back to the mid to late 90's.

As far as letting each thread onto a single CPU, I'm wanting to use either MOSIX or OpenMosix, which while an SSI Kernel, does do exactly that, it doesn't allow single threads to be shared and broken apart at run-time.

Although I'm thinking another way to get the performance up where I want it, is to keep each network that ISNT on the VPN (it's a collection of LAN's joined by (guessing at this) VPN to be a single WAN, if it's not a VPN, I have no clue what it is)

But keep each LAN as a single cluster, which OpenMosix can do, and then cluster together those clusters, it's not going to change much in the dynamics of the system though, because it's going to assign threads in exactly the same way, however it might restrict them ideally to a single cluster (say CPU's 1 through 199 are on one LAN, and 200 through 250 are on a second one, restrict threads ideally to 1-199 and 200-250, instead of threads going to 1 and 201 for the same application, still haven't found any way to actually contact the OpenMosix dev team and ask...and MOSIX is closed source )

[Jack2000]

Is this place like a Uni campus or some Military thing
couse in both cases you can seek out the "nerds" or the "man" they'll
be glad to help you out