From patrick@myri.com Wed May  7 14:46:40 2003
Date: 07 May 2003 14:02:33 -0400
From: Patrick Geoffray <patrick@myri.com>
To: Tim Wilcox <tim@linux-force.com>
Cc: Beowulf cluster mailing list <beowulf@beowulf.org>
Subject: Re: Questions about Myricom

Hi Tim,

On Tue, 2003-05-06 at 14:06, Tim Wilcox wrote:
> I have been unable to find answers to these questions that came to me 
I recommend you to sent all of your questions to help@myri.com, they can
either reply directly or dispatch to someone available who can.

> 1) Does MPICH-GM support the MPI Ordering Rules for the calculation of
> collectives?

MPICH does. MPICH-GM collectives are MPICH collectives.
 
> 2) How much memory will GM-2.0 be able to register?

GM-(1,2) can register 5/6 of the physical memory of the machine. It's
not an hard limit but it's a safeguard to prevent someone to lock all of
the pages of the system and prevent swapping.

> Is there a limit to 
> the size of an MPICH-GM message, other than the amount of memory GM can 
> register on the target node?

4 GB max message size (length on 32 bits). Actually, the amount of
memory GM can register has no impact on the maximum message size in
MPICH-GM as the registration is pipelined with the communications
(chunck of 1 MB).

> 3) What are the limitations to the number of messages that can be
> simultaneously in transit? 

At the GM level, the number of simultaneous sends is the number of GM
send tokens: 28 on 4K pages systems (IA32), 61 on 8K pages systems
(solaris), a lot but don't remember exactly on 16K pages systems (IA64,
AMD64).
There are more GM receive tokens: 247 on IA32, more but don't remember
exactly on other systems.
You can get more send or receive tokens by recompiling the firmware but
I don't think it will change anything at the MPI level.
 
> 4) Does MPICH-GM use optimized collectives provided by GM?

GM does not provide collectives, MX will.

> 5) How does the amount of buffer space needed by MPICH-GM scale with the
> number of processors?

What do you mean, processors in a node or in the whole cluster ? 
There is no buffering depending on the number of processors in the
cluster.
The shared memory queues for intra-nodes communications only (if not
disabled) depend on the number of local processes in a node, so if you
scale the number of processes with the number of processors, these
buffers will depend on the number of processors. These queues are 1 MB
long and there are N*(N-1) queues per node where N is the number of
local processes in the same MPI job.

> 6) What performance issues are there with unaligned data transfers?

None, DMAs are byte-aligned.


The corresponding answers regarding MX and MPICH-MX are very different
(collectives in the NIC, no memory registration, 1000s simultaneous
messages in transit, etc), but MX is not available/finished yet.

Hope it helps.

Patrick
-- 

Patrick Geoffray
Myricom, Inc.
http://www.myri.com

_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
