IPC's, Granularity and Barriers

OK, by now you should be getting the hang of things. A beowulf is a parallel supercomputer built out of COTS nodes interconnected by a COTS network of some kind. One can build a beowulf to speed up a piece of parallelized code (in the classic Amdahlian sense) so it finishes faster. One can build a beowulf to be able to do a task at all by assembling more resources than one can either afford any other way or than are currently available in a system at any price. One can build a beowulf to speed up a code in an exotic way (by providing a faster extended virtual memory space, for example).

In the previous chapter we discussed all sorts of ways the basic bottlenecks between the CPU and memory subsystems (within a node, by assumption) can affect program speed, trying to provide a semi-quantitative understanding so that you can at least do the back of the envelope calculations required to compare the cost-benefit of various alternative ways of accomplishing a task. In this chapter we'll focus on the sine qua non bottleneck of beowulfery, the network.

There is so much to learn about networking and how it relates to serious beowulfery that it is hard to know just how much to put into an introductory book like this. To invert the point, there is such a wide range of ignorance about networking out there that I could easily be speaking to someone who doesn't know Appletalk from Ethernet, has never heard of the ISO or OSI, for whom TCP and IP are a mystery, and who thinks that a router is a device for cutting interesting curves in a piece of wood.

If this is you: Sorry, chum, you won't learn about these things here, or at least you won't learn much (certainly not enough to assemble a functional linux network). What can I say - there are whole books that focus on just setting up and running a network, and I cannot compress all that into a chapter and have time to say anything at all about networking in the fairly strict context of beowulfery.

So, even though a network is key to a beowulf, I'm going to assume that in fact you do know what the following are:

• NIC (Network Interface Card), typically a PCI (regular, fast, or gigabit) ethernet adaptor or PCI Myrinet adaptor, although there are now some exotic alternatives with more on the horizon. This little pup plugs into the PCI bus of your node and is connected via RJ45 cables to an appropriate
• Hub. This is something that distributes a signal in the transmit wires of one connection to the receive wires of all the other connections. There are all sorts of technical things associated with hubs (like the number of hubs you can put in between hosts). In general, hubs are ``bad'' in beowulf design unless one's needs are modest and likely to stay that way. If one can afford it, one would usually do better with a
• Switch. This is something that establishes a virtual pairwise connection between hosts plugged into the switch. Where the hub allows only one host at a time to talk in one direction, a switch allows all the hosts to talk at the same time in both directions. In principle - be sure to check the ``bisection bandwidth'' of your switch to ensure that it can indeed handle full-duplex on all lines at once as a cheap switch might not.
• Ethernet. The most common (and cheapest) networking hardware protocol. Ethernet comes with an amazing and complex set of rules and standards, such as how long a cable can be used to connect a host to a hub or switch, how many hubs can exist between hosts, what an adaptor has to do in the event of a collision (what a collision is in the first place), in addition to certain baseline latencies and bandwidths and packet sizes. Can't know too much about ethernet, no sirree-bob!
• Myrinet. The most common and probably premier gigabit network. It's about ten times faster than 100BT ethernet, and costs about twenty times as much per node (or even a bit more).
• IP. ``Internet Protocol''. Yes, it is the fundamental protocol upon which the internet is founded. Associated with IP is the ``internet address'' (or IP address) of a host (adaptor), a packet structure (headers and so forth), and above all a routing and filtering mechanism.
• TCP. ``Transmission Control Protocol''. This lives ``on top of'' IP and regulates things like the reliable delivery of packets across an uncertain network. Most common services (like mail, ftp, telnet, http and so forth) are defined in terms of standard port addresses within TCP. TCP is a bit smarter and more reliable than ``raw'' IP, and hence is a tiny bit slower.
• Sockets. A ``socket'' is an abstraction for a network connection. One opens up a socket on a host and a remote service connects to that socket. Information flows between the sockets. Sockets can be read or written to like files (of course, in Unix and hence linux, everything is a file, right?).
• UDP. ``Unix Datagram Protocol''. This is a kind of socket and hence network connection. It is the ``raw IP'' connection I referred to earlier. A TCP socket is more reliable but has certain limitations and costs. UDP is commonly used to provide local services within a local area network LAN), TCP across a wide area network or WAN. Except that isn't necessarily true anymore, as reliability is almost always more important than raw speed. NFS is probably the most famous UDP service.
• Router. Something that routes packets, usually IP packets (actually a router can often route lots of kinds of packets but we don't care about any other kinds).
• Gateway. A router that typically lives between an ``inner'' network (LAN) and an ``outer'' network (ultimately the rest of the Internet). It lets in the good packets and lets out the bad packets. A true beowulf typically has a gateway node that is also usually a ``head'' node from which it is controlled and may also be a server node if the internal nodes require (e.g.) NFS services. The gateway node can keep spurious external traffic off the private internal network of the nodes.

As you can see, I am omitting all sorts of useful and important things. You won't learn about netmasks, broadcasts, how to configure a NIC, or any of that from me. However, I will direct you to the /usr/doc/HOWTO directory (in most linux distributions) that has explicit step by step instructions for setting up all sorts of things including the network. Don't forget about Linux Headquarters (http://www.linuxhq.com/) either, which has links to all the HOWTOs and other documentation. There are a bunch of key learning documents in my own personal website including http://www.phy.duke.edu/$\sim$rgb/security/local.guide, which is ``the'' classic 1988 Rutgers white paper by Charles Hedrick describing all sorts of networking concepts. Finally, there are a whole bunch of useful URL's on the Brahma website (http://www.phy.duke.edu/brahma) which might be of interest to the neophyte.

SO, from here on I'm going to assume that you can design and set up a simple ethernet-based IP subnet without having your hand held. We'll still address some of this sort of thing in the next chapter, but for now we'll focus on the technical details (especially things like latency, effect of packet size on bandwidth, problems, solutions) and not on truly introductory things.