From newt@scyld.com Tue Nov 14 14:11:05 2000
Date: Tue, 14 Nov 2000 13:14:17 -0500 (EST)
From: Daniel Ridge <newt@scyld.com>
To: Franz Marini <marini@pcmenelao.mi.infn.it>
Cc: Walter B. Ligon III <walt@parl.ces.clemson.edu>, beowulf@beowulf.org
Subject: Re: Scyld and pvfs ... 


Franz,

On Tue, 14 Nov 2000, Franz Marini wrote:

> 4) (quite a long list of questions ...) we're using the CPMD
> (Carr-Parrinello Molecur Dynamics) program on our current beowulf, but we
> have some performance problems w/ mpich 1.2.1 (w/ or without mpd). It
> seems like it uses LOTS of all2all comms. Has someone tried it w/ Scyld ?

Scyld Beowulf isn't tied to MPICH especially. Our changes are actually
quite slight and can be easily applied to just about any other message
passing system.

Many parallel systems will work just fine if you tell them to use 'bpsh'
instead of 'rsh'. bpsh will use the copy of the binary installed on the
frontend.

Other systems require a bit more help. In particular, systems which expect
to put a daemon on each node will need help -- as the remote systems
probably don't have a copy of the target binary to execute.

In these cases (for example PVM), you can do a couple of things. Adopt a
new build target (Paul Springer's BEOLIN PVM arch target, for example) or
replace the fork()/exec("foo") pair in the daemon with something more like

nodenum=bproc_currnode(); bproc_rfork(-1); bproc_execmove(nodenum,"foo");

We don't recommend doing the latter, but it can be useful to get an
existing system up and running -- thereby giving you some time to think
about how to fix the software better.

Regards,
	Dan Ridge
	Scyld Computing Corporation 


_______________________________________________
Beowulf mailing list
Beowulf@beowulf.org
http://www.beowulf.org/mailman/listinfo/beowulf

From rbw@networkcs.com Wed Nov 22 10:29:35 2000
Date: Tue, 21 Nov 2000 15:47:08 -0600 (CST)
From: Richard Walsh <rbw@networkcs.com>
To: beowulf@beowulf.org
Subject: "beoserv" daemon not reacting to slave rarp requests ...


All,

I am trying to bring up the slave nodes from the 
head node using Beowulf-2 software from Scyld. I am
following the documentation carefully. 

"beoserv" (listening from eth0 on port 2223) and bpmaster 
seem to come up without trouble. The config file is correct
(eth0 is the internal address 10.0.0.17, eth1 is a second
internal network for file transfers only, and eth2 is external)
Range (10.0.0.1 10.0.0.16) and netmask are specified correctly.

The slave nodes boot from floppy and begin r-arping.
This is clear from the lights on the interfaces and
a tcpdump on the head node. At this point, the config 
file in /etc/beowulf/config should be updated ... but this 
never happens ... the slave nodes just r-rarp forever. 

I am assuming that bpmaster and beoserv take care of all 
that is required to assign the IP address out of the specified 
range in the config file, but perhaps I am wrong ... does 
some other service have to be running?

Suggestions appreciated ...

Richard Walsh

#---------------------------------------------------
#
# Richard Walsh
# Project Manager
# netASPx, Inc.
# 1200 Washington Ave. So.
# Minneapolis, MN 55415
# VOX:    612-337-3467
# FAX:    612-337-3400
# EMAIL:  rbw@networkcs.com, richard.walsh@netaspx.com
#
#---------------------------------------------------
# "What you can do, or dream you can, begin it;
#  Boldness has genius, power, and magic in it."
#                                       -Goethe
#---------------------------------------------------


_______________________________________________
Beowulf mailing list
Beowulf@beowulf.org
http://www.beowulf.org/mailman/listinfo/beowulf

From newt@scyld.com Wed Nov 22 10:29:48 2000
Date: Tue, 21 Nov 2000 18:11:39 -0500 (EST)
From: Daniel Ridge <newt@scyld.com>
To: Richard Walsh <rbw@networkcs.com>
Cc: beowulf@beowulf.org
Subject: Re: "beoserv" daemon not reacting to slave rarp requests ...


Richard,

On Tue, 21 Nov 2000, Richard Walsh wrote:
> "beoserv" (listening from eth0 on port 2223) and bpmaster 
> seem to come up without trouble. The config file is correct
> (eth0 is the internal address 10.0.0.17, eth1 is a second
> internal network for file transfers only, and eth2 is external)
> Range (10.0.0.1 10.0.0.16) and netmask are specified correctly.

> The slave nodes boot from floppy and begin r-arping.
> This is clear from the lights on the interfaces and
> a tcpdump on the head node. At this point, the config 
> file in /etc/beowulf/config should be updated ... but this 
> never happens ... the slave nodes just r-rarp forever. 

These nodes are actually going into '/var/beowulf/unknown_addresses'.

These are moved into '/etc/beowulf/config' via the 'beosetup' tool
or manually. (see comments in this file for details...)

Beyond that, the only suggestion I can make is that you might want
to have the frontend before your IP range instead of after. This
may make it slightly less traumatic to expand the size of your cluster
later.

> I am assuming that bpmaster and beoserv take care of all 
> that is required to assign the IP address out of the specified 
> range in the config file, but perhaps I am wrong ... does 
> some other service have to be running?

Run 'beosetup'. We don't presume -- entirely automatically -- that
you want a given box in your cluster. The beosetup tool can be 
configured to automatically add new nodes.

Good luck,
	Dan Ridge
	Scyld Computing Corporation


_______________________________________________
Beowulf mailing list
Beowulf@beowulf.org
http://www.beowulf.org/mailman/listinfo/beowulf

From shew0469@cs.uidaho.edu Fri Dec  8 20:37:22 2000
Date: Fri, 8 Dec 2000 16:36:24 -0800 (PST)
From: Andrew Shewmaker <shew0469@cs.uidaho.edu>
To: Robert Sand <rsand@d.umn.edu>
Cc: beowulf@beowulf.org
Subject: Re: Problems Creating/Overwriting partition tables on the slave
    nodes.

Robert,

Your situation reminded me of what my classmates and I experienced.  I 
wasn't present for the fix, but I asked one of the others (Wade) if 
we would describe it to me.

Wade wrote:

The problem we were having is that the old systems that we wanted to use as
nodes had a pre-existing Linux install. When we did the partition table
copying step, I seem to remember we got an error message saying the
partition tables on the nodes couldn't be written.

Brad used another Linux CD he had to boot up the node machines in rescue
(because we didn't know the root password on the existing Linux installs),
and used dd to zero out the first 512 bytes of the disk (destroying whatever
evil in the existing partition tables was breaking Scyld's attempts to write
a new partition table to those machines.)

I honestly don't remember where we found the tip to that. I thought it was
in the Scyld documentation, but now I can't find it.

Anyway, after that, we rebooted the nodes with Scyld boot floppies, and did
everything by the book. It worked exactly the way it was supposed to.

Wade
----

If bpsh works, then you might be able to zero out all of the nodes at once.

I hope this helps,

Andrew Shewmaker

On Fri, 8 Dec 2000, Robert Sand wrote:

All,

I've been having problems setting up the IDE drives on my slave nodes.  First of
all I am using the software installed from the Distribution CD from Scyld.  The
drives on the slaves are all the same type of drive and currently have RedHat
Linux installed on them.  What I want to do is instead of booting from the
floppy drive is to boot from the IDE drive.

When I do a /usr/lib/beoboot/bin/beofdisk -q the partition table in
/etc/beowulf/fdisk looks good so I want to write that partion table to the disks
using /usr/lib/beoboot/bin/beofdisk -w.  This runs just fine and it tells me I
have to reboot the slave nodes for the new table to be saved.  I reboot the
slaves and they come up just fine but I can't do the beoboot-install because it
has problems writing to the disk.  Doing a bpsh 0 /sbin/sfdisk -l /dev/hda it
shows me that the partition table has not even changed: it is still the old
Redhat installation.

Something is wrong here and the Help I've received from Richard Walsh is great
but it also brings up other problems.  He states that I should copy everything
in /lib to the slaves using bpcp.  Every single time I do this I get error
messages saying invalid option or file not found.  Then I do a bpsh 0 ls -la
/lib and it only shows me the one library that is there after booting.  What is
up here?  Does bpcp really work or does it only work when you boot from the hard
drive?

These are really bad problems and I do need to get this thing running again so
that our faculity can use it.  If anyone has had this problem I'd appreciate it
if I could find out how you got around it.  If anyone at Scyld can help me that
would be great too.  I was told by someone at Scyld that I should get the
distribution CD when I was trying to install this from the RPMs.  Now I have and
I don't get answers to these questions from them.  I know this is free software
but I am perplexed at why they wouldn't like to know there may be a problem
here.


-- 
Robert Sand.
mailto:rsand@d.umn.edu                             US Mail
University of Minnesota Duluth                     10 University Dr.
Information Technology Systems and Services        MWAH 176
144 MWAH                                           Duluth, MN 55812
Phone 218-726-6122        fax 218-726-7674

"Walk behind me I may not lead, Walk in front of me I may not follow,
 Walk beside me and we walk together"  UTE Tribal proverb.

_______________________________________________
Beowulf mailing list
Beowulf@beowulf.org
http://www.beowulf.org/mailman/listinfo/beowulf


_______________________________________________
Beowulf mailing list
Beowulf@beowulf.org
http://www.beowulf.org/mailman/listinfo/beowulf

From newt@scyld.com Fri Dec 15 14:08:20 2000
Date: Fri, 15 Dec 2000 13:15:27 -0500 (EST)
From: Daniel Ridge <newt@scyld.com>
To: David Leunen <leunen.d@fsagx.ac.be>
Cc: beowulf@beowulf.org, annmarie@scyld.com
Subject: Re: Scyld and fstab for Diskless slaves


On Fri, 15 Dec 2000, David Leunen wrote:

> But I have another prob: How to configure /etc/beowulf/fstab correctly
> for diskless nodes? I didn't see anything about diskless in the doc.

For reference, the details of the slave fstab file are in the Beowulf
install guide bundled with our distribution. To summarize:

Configuring ramdisks for Scyld Beowulf is straightforward.
Comment or remove the '/' and 'swap' entries from /etc/beowulf/fstab.
Create a line which looks like:

/dev/ram3	/	ext2	fs_size=65536	0 0

The magic bit is that we have added a mount option -- fs_size -- that
tells us how large to make the disk.

This relates to another question asked recently in this forum:
	Why does our default kernel command line specify a ramdisk_size
	of 128M?

This should be pretty safe on boxes that don't even have 128M -- but
you will get into trouble if you try to use the large ramdisks. If
you have troubles installing Scyld Beowulf -- this isn't the problem.

Also, while we set up a beefy ramdisk during boot -- this space is
reclaimed when the node finishes booting if you aren't doing a ramdisk
root.

This, in turn, relates to another freqeuntly asked question:
	Why do my nodes show up as unavailable?

The reason is usually that the default fstab (/etc/beowulf/fstab) does
not relate to the partitioning on your disks. Whatever the reason,
the details of the node boot do not appear on the console. This record
can be found in /var/log/beowulf/node.<nodenum> where <nodenum> is the
node number that you seek.

Our upcoming contains a lot of effort directed towards reducing the size
of ramdisk. This is to make the system work on smaller memory machines
and to make diskless operation more palatable on medium memory machines. I
have run our upcoming version on VMware instances with as little as 16M
with a beowulf ram footprint of about 2M.

Because we can now do this, I expect that the next version of our product
will ship with ramdisk root as the default.

Regards,
	Dan Ridge
	Scyld Computing Corporation




_______________________________________________
Beowulf mailing list
Beowulf@beowulf.org
http://www.beowulf.org/mailman/listinfo/beowulf

From hendriks-beowulf@scyld.com Tue Dec 19 15:36:02 2000
Date: Tue, 19 Dec 2000 11:44:28 -0500
From: Erik Arjan Hendriks <hendriks-beowulf@scyld.com>
To: Chip Coldwell <coldwell@frank.harvard.edu>
Cc: Daniel Ridge <newt@scyld.com>, David Leunen <leunen.d@fsagx.ac.be>,
     beowulf@beowulf.org
Subject: Re: Scyld and fstab for Diskless slaves

On Tue, Dec 19, 2000 at 09:35:36AM -0500, Chip Coldwell wrote:
> I know how to compile a kernel to NFS mount a root filesystem, and I
> have set up diskless clusters using this method; I'm very curious how
> the ramdisk root filesystem works.  Does the node tftp down a
> compressed filesystem image and load it into the ramdisk analogously
> to the root floppies of yore, except coming over tftp instead of from
> a floppy?  Does this require custom kernel modifications, or is it a
> functionality available in stock kernels?

The floppy is really just a generic netboot.  It contains its own
kernel and ramdisk.  It downloads another kernel and another ramdisk.
The download works via TCP because TFTP is flaky under load.  TCP will
fall down too at some point but it will do so later than TFTP and it
was easy to implement for a first cut.

The ramdisk you get from the front end isn't the one you end up
running on.  That one contains enough stuff to get the network off the
ground and start the BProc slave daemon.  Then scripts on the front
end then setup a /dev/ramX device in the same way they would setup a
normal hard disk partition.  This is why it's possible to have
collisions in the fstab with /dev/ramX devices.  Most of the stuff in
the ramdisk downloaded with the kernel is eventually discarded leaving
a megabyte or so of stuff to run BProc and whatever the size of your
ram disk root is.

The tricky part here is that root file system that your processes see
is actually a chroot()'ed environment.  That way nobody has to worry
about switching root file systems (which is tricky if not impossible),
NFS root or anything like that.  (See the last two steps in the
/usr/lib/beoboot/bin/node_up.)

- Erik
-- 
Erik Arjan Hendriks          Printed On 100 Percent Recycled Electrons
erik@hendriks.cx                   Contents may settle during shipment

_______________________________________________
Beowulf mailing list
Beowulf@beowulf.org
http://www.beowulf.org/mailman/listinfo/beowulf

From newt@scyld.com Tue Dec 19 15:37:22 2000
Date: Tue, 19 Dec 2000 12:44:30 -0500 (EST)
From: Daniel Ridge <newt@scyld.com>
To: Chip Coldwell <coldwell@frank.harvard.edu>
Cc: Erik Arjan Hendriks <hendriks-beowulf@scyld.com>,
     David Leunen <leunen.d@fsagx.ac.be>, beowulf@beowulf.org
Subject: Re: Scyld and fstab for Diskless slaves



On Tue, 19 Dec 2000, Chip Coldwell wrote:

> Let me see if I understand you correctly: each node initially boots
> from floppy and then goes through a sort of second-stage boot that
> sets up the ramdisk, etc., is that right?  So a local floppy disk is
> required on every node?
> 
> Have you ever tried doing this with PXE?  Or does the PXE reliance on
> tftp (even multicast tftp) make it undesirable?

We can do this with PXE. Erik Hendriks wrote 2 kernel monte so that we
would not need to rely on Intel to solve our cluster booting problem.
Given the complexity of the PXE spec -- it's amazing that it works at all.

In addition, PXE is of no use if you want to boot a Myrinet-only cluster
or a Gigabit Ethernet-only cluster without PXE BIOSes for these adapters.

If you want to use PXE, I have had good luck with HPA's syslinux PXE
loader. Our beoboot tool can be gently coaxed into spitting out
seperate kernel and ramdisk images for this purpose.

Our upcoming beoboot can be explicity invoked to create these images for
you. Users of the preview release can hack beoboot to not delete the
interim initrd image (comment out line 316) to get this file for use with
PXE.

Regards,
	Dan Ridge
	Scyld Computing Corporation



_______________________________________________
Beowulf mailing list
Beowulf@beowulf.org
http://www.beowulf.org/mailman/listinfo/beowulf

From newt@scyld.com Fri May 11 17:08:22 2001
Date: Fri, 11 May 2001 15:08:43 -0400 (EDT)
From: Daniel Ridge <newt@scyld.com>
To: Greg Kurtzer <gmkurtzer@lbl.gov>
Cc: beowulf@beowulf.org
Subject: Re: SCYLD-- mounting boot.img in loopback


> Has anyone been able to sucessfully mount the /var/beowulf/boot.img using
> loopback? What kind of filesystem are they using?

boot.img is not a filesystem image. It is a special format used by monte
that contains a kernel, initrd, and command line arguments for the phase 2
kernel.

If you want to generate custom boot images, I suggest you use:

`beoboot -2 -i -o /tmp/foo`

which will generate separate kernel,initrd images to
/tmp/foo,/tmp/foo.initrd. You may then play with these to your hearts
content. This is what, for instance, you would do if you wanted images for
use with PXE.

If you merely want to alter the contents of the boot.img without needing
split images, I suggest that you investigate the options to the 'beoboot'
tool provided with Scyld. This may provide you with what you seek.

Regards,
	Dan Ridge	
	Scyld Computing Corporation


_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

From edwards@icantbelieveimdoingthis.com Wed Jun 27 11:22:23 2001
Date: Wed, 27 Jun 2001 08:35:42 -0600
From: Art Edwards <edwards@icantbelieveimdoingthis.com>
To: Jarrod Smith <jsmith@structbio.vanderbilt.edu>, beowulf@beowulf.org
Subject: Re: Scyld: disabling mpi processes on master

On Tue, Jun 26, 2001 at 06:03:40PM -0500, Jarrod Smith wrote:
> I don't want to use the master as a compute node for MPI runs (by design,
> it has slower processors than the rest of the cluster).  I've read some
> things in the archive which hint at the fact that it is possible, but
> nothing I've tried works.  There doesn't appear to be a -nolocal
> option (despite posts from Scyld that it is in the mpirun
> man page), and using bpsh to spawn the job doesn't work, because
> there is no /var on the slaves, and the root process seems to want
> to want to write some temp files there.
> 
> So...can it be done?  If so, exactly how?
Thanks to Keith Underwood, I now can run jobs off of the head node using the following command.

NO_INLINE_MPIRUN=true bpsh n application -p4pg file

The p4p4 switch indicates that you will use a p4pg file to indicate which
nodes you will use for the calculation. I attach, below, a sample p4pg file.
application is the executable. n is the node on which the p4pg file resides.

I am making this post because I have stated before that I was unable to 
open files on slave nodes unless the head node was part of the run. It turns
out that, at 4:30 in the morning, I had made a dumb mistake in the call 
above so that I couldn't open the files. I have since been successful in
opening files in the /tmp directory of the slave nodes. 

Here is the sample file:

0 0 /home/edwardsa/QUEST/lcao2p46a/lcao_nebgga.x
1 1 /home/edwardsa/QUEST/lcao2p46a/lcao_nebgga.x
2 1 /home/edwardsa/QUEST/lcao2p46a/lcao_nebgga.x
3 1 /home/edwardsa/QUEST/lcao2p46a/lcao_nebgga.x
4 1 /home/edwardsa/QUEST/lcao2p46a/lcao_nebgga.x

Note that the application here has been the same as the application in the
call above.

I hope this helps.

Art Edwards
> 
> TIA
> 
> -- 
> Jarrod A. Smith
> Research Asst. Professor, Biochemistry
> Asst. Director, Center for Structural Biology
> Computation and Molecular Graphics
> Vanderbilt University
> 
> jsmith@structbio.vanderbilt.edu
> 
> 
> _______________________________________________
> Beowulf mailing list, Beowulf@beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
