From Marcin.Kolbuszewski@nrc.ca Fri Jan  5 14:47:56 2001
Date: Fri, 5 Jan 2001 14:03:47 -0500 
From: "Kolbuszewski, Marcin" <Marcin.Kolbuszewski@nrc.ca>
To: 'Robert G. Brown' <rgb@phy.duke.edu>,
     Jared Hodge <jared_hodge@iat.utexas.edu>
Cc: Krishna Prasad <sita_krish@hotmail.com>
Subject: RE: Help needed please !!!

    [ The following text is in the "iso-8859-1" character set. ]
    [ Your display is set for the "US-ASCII" character set.  ]
    [ Some characters may be displayed incorrectly. ]

I've been playing with my 5 node MOSIX cluster along the lines
desribed below and have private $0.02 to add: 

My nodes have 256MB memory and 128MB swap. 4 of them are invisible to the
actual users, who
get their account on the front end machine. They are encouraged to start 
their jobs by putting them in the background and MOSIX is migrating them as
it sees fit

This very quickly lead to problems with insufficient amount of memory on the
access node: users would submit more jobs that would not fit in available
memory and some
would be killed. The quick fix was easy: I installed SCSI disk on the access
node and created
1GB of swap there and removed swap from compute nodes: When the user submits
his 8 100MB jobs one 
after the other, they do not have time to migrate to other nodes, but they
are swapped. 
A few seconds later MOSIX finds they are in swap and moves them away. 

MOSIX is aware of the amount of memory on the system and will not try to put
two 130MB 
jobs on a 256MB machine. So what happens is that if I have 10 100MB jobs,
they are even 
distributed on 5 nodes, 2 each.
When I put 10 150MB jobs I have 4 jobs on "swapless" nodes and six on the
main one (more or less)
It seems (have not tested thoroughly) that when I start a few more small
jobs (a few Megs)
they migrate promptly to the nodes with available memory.

Marcin

----------------------------------------------------------------------------
--
High Performance Computing Group             Coordination Office    
Institute for Information Technology         C3.ca Association    
National Research Council of Canada

Rm 286, M-50, 1500 Montreal Road             tel 613-998-7749
Ottawa, Canada                               fax 613-998-5400
K1A 0R6                                      e-mail
Marcin.Kolbuszewski@nrc.ca



-----Original Message-----
From: Robert G. Brown [mailto:rgb@phy.duke.edu]
Sent: Friday, January 05, 2001 12:45 PM
To: Jared Hodge
Cc: Krishna Prasad; beowulf@beowulf.org
Subject: Re: Help needed please !!!


On Fri, 5 Jan 2001, Jared Hodge wrote:

> I have a dumb idea, or at least a dumb suggestion.  You say your problem
> involves a lot of iterations that you want to execute in parallel.  If
> you don't need to complete one iteration before beginning the next, you
> could very easily write a shell script to farm out the iterations to
> different processors and combine the results.  This could be your
> parallel "executable".  If this is the case (the "Embarrassingly
> Parallel" that RGB was talking about), then you could use a very
> advanced timing routing, "date" or something like that.  This probably
> isn't the case, though, since I doubt you would be asking for help if it
> were.  The problem is that if it isn't, you have to speed up every
> iteration of the loop through parallelism, something that may not be
> easy if the number of iterations is very large since you'll lose some
> time just waiting on communication because of network latency.  So if at
> all possible, go with Embarrassingly parallel.  Keep it simple.  Hey I
> told you this was a dumb idea, that's why they call it embarrassingly
> parallel.

You are absolutely correct (and this isn't dumb at all, if the code will
work that way), except that an even easier approach is to install MOSIX
and just run N copies all at once as if your local system had N CPUs.
Well, I don't know if it is easier or not -- a simple process
distribution shell script is pretty easy, and installing and configuring
MOSIX isn't quite so simple for a novice.  But it makes for a better,
more userproof cluster when you're done and is much cooler besides.

MOSIX is more or less designed to manage job distribution and load
balancing for embarrassingly parallel applications that don't do
anything too complicated with I/O and have a large run time:startup time
ratio.

   rgb

>
> Krishna Prasad wrote:
> >
> > Hello anyone,
> >
> >           I am doing my Btech Final year project in clustering.
> > I have gone through various projects given in beowulf site and Still I
> > coudn't make out the basic idea of how to split a job into parallely
> > executably form.  I plan to do my project on linux.
> >            What I plan to do is to split a process involving lot of
> > iterations into parallelly executable form and execute in the different
> > nodes of the cluster.  Please someone guide in simple terms as to how to
> > split the process or where to find the information about it.  Can it be
> > completed within 2 months time as it is my limit. Which language is
suitable
> > for this?  What program or job can be used as input for this?
> >
> >           Also tell me how could I show the time taken for execution
when a
> > process is executed serially and in a cluster.
> >           What we have is 3 nodes with lan connection in our college lab
> > each of having a  pentium processor with linux loaded in them.
> >
> > Please reply soon as my time is limited.
> >
> > Thanks in advance.
> > My email addresses are: sita_krish@hotmail.com and
> > sita_krish@rediffmail.com.
> > krishna prasad
> >
> >
_________________________________________________________________________
> > Get Your Private, Free E-mail from MSN Hotmail at
http://www.hotmail.com.
> >
> > _______________________________________________
> > Beowulf mailing list
> > Beowulf@beowulf.org
> > http://www.beowulf.org/mailman/listinfo/beowulf
>
>

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb@phy.duke.edu




_______________________________________________
Beowulf mailing list
Beowulf@beowulf.org
http://www.beowulf.org/mailman/listinfo/beowulf
