   \input texinfo

Linux SMP FAQ
*************

     David Mentré, `David.Mentre@irisa.fr'
     v0.27, 05 may 1998

   This FAQ review main issues (and I hope solutions) related to SMP
configuration under Linux.

Introduction
************

   Linux can work on SMP (Symetric Multi-Processors) machines. SMP
support has started with the 2.0 family and has been improved in the
2.1 (future 2.2) saga.

   FAQ maintained by David Mentré (David.Mentre@irisa.fr). The latest
edition of this FAQ can be found at
http://www.irisa.fr/prive/mentre/smp-faq/.

   If you want to contribute to this FAQ, I would prefer a diff against
the SGML version
(http://www.irisa.fr/prive/mentre/smp-faq/smp-faq.sgml) of this
document, but any remarks (in plain text) will be greatly appreciated.
If you send me an email about this FAQ, please include a tag like
`[Linux SMP FAQ]'. It helps me automatically sort mails (and you will
have a faster reply `;)').

   This FAQ is an improvement of a first draft made by `Chris Pirih'.

   All information contained in this FAQ is provided "as is." All
warranties, expressed, implied or statutory, concerning the accuracy of
the information of the suitability for any particular use are hereby
specifically disclaimed. While every effort has been taken to ensure the
accuracy of the information contained in this FAQ, the authors assume(s)
no responsibility for errors or omissions, or for damages resulting from
the use of the information contained herein.

What's new ?
************

`v.27, 05 may 1998'
        *  New info for the Adaptec and TekRam drivers

        *  Micronics W6-LI motherboard works under SMP

Questions related to any architectures
**************************************

Kernel side
===========

  1.  `Does Linux support multi-threading?  If I start two or more
     processes, will they be distributed among the available CPUs?' Yes.

  2.  `What kind of architectures are supported in SMP?'
    `From `Alan Cox':'
          SMP is supported in 2.0 on the hypersparc (SS20, etc.)
          systems and Intel 486, Pentium or higher machines which are
          Intel MP1.1/1.4 compliant. `Richard Jelinek' adds: right now,
          systems have been tested up to 4 CPUs and the MP standard
          (and so Linux) theoretically allows up to 16 CPUs.

          SMP support for UltraSparc, SparcServer, Alpha and PowerPC
          machines is in progress in 2.1.x.

    `From `Ralf Bächle':'
          MIPS, m68k and ARM does not support SMP; the latter two
          probly won't ever.

          That is, I'm going to hack on MIPS-SMP as soon as I get a SMP
          box ...

  3.  `How do I make a Linux SMP kernel?' Uncomment the `SMP=1' line in
     the main Makefile (`/usr/src/linux/Makefile').

     AND

     enable "RTC support" (from `Robert G. Brown'). Note that inserting
     RTC support actually doesn't afaik prevent drift, but according to
     a discussion [lsqb  ]Robert G. Brown] remember from a year ago or
     so it can prevent lockup when the clock is read at boot time. A
     note from `Richard Jelinek' says also that activating the Enhanced
     RTC is necessary to get the second CPU working (identified) on
     some original Intel Mainboards.

     AND

     do NOT enable APM!  APM and SMP are not compatible, and your
     system will almost certainly (or at least probably `;)') crash
     under boot if APM is enabled (`Jakob Oestergaard'). `Alan Cox'
     confirms this : 2.1.x turns APM off for SMP boxes.  Basically APM
     is undefined in the presence of SMP systems, and anything could
     occur.

     You must rebuild all your kernel and kernel modules when changing
     to and from SMP mode. Remember to `make modules' and `make
     modules[lowbar]install' (from `Alan Cox').

  4.  `How do I make a Linux `non'-SMP kernel?' `Comment' the `SMP=1'
     line in the Makefile (and not set SMP to 0).

     You must rebuild all your kernel and kernel modules when changing
     to and from SMP mode. Remember to `make modules' and `make
     modules[lowbar]install'.

  5.  `How can I tell if it worked?'  cat /proc/cpuinfo

     Typical output (dual PentiumII):
          processor       : 0
          cpu             : 686
          model           : 3
          vendor_id       : GenuineIntel
          stepping        : 3
          fdiv_bug        : no
          hlt_bug         : no
          fpu             : yes
          fpu_exception   : yes
          cpuid           : yes
          wp              : yes
          flags           : fpu vme de pse tsc msr pae mce cx8 apic 11 mtrr pge mca cmov mmx
          bogomips        : 267.06

          processor       : 1
          cpu             : 686
          model           : 3
          vendor_id       : GenuineIntel
          stepping        : 3
          fdiv_bug        : no
          hlt_bug         : no
          fpu             : yes
          fpu_exception   : yes
          cpuid           : yes
          wp              : yes
          flags           : fpu vme de pse tsc msr pae mce cx8 apic 11 mtrr pge mca cmov mmx
          bogomips        : 267.06

  6.  `What is the status of converting the kernel toward finer grained
     locking and multithreading?' 2.1.x has signal handling, interrupts
     and some I/O stuff fine grain locked.  The rest is gradually
     migrating. All the scheduling is SMP safe

  7.  `Does Linux SMP support processor affinity?' No and Yes.  There
     is no way to force a process onto specific CPU's but the linux
     scheduler has a processor bias for each process, which tends to
     keep processes tied to a specific CPU.

User side
=========

  1.  `Do I really need SMP?' If you have to ask, you probably don't.
     `:)'

  2.  `How does one display mutiple cpu performance?' Thanks to `Samuel
     S. Chessman', here is some useful utilities:
    `Character based:'
          http://www.cs.inf.ethz.ch/~rauch/procps.html

          Basically, it's   procps v1.12.2   (top, ps, et. al.)  and
          some patches to support SMP.

    `Graphic:'
          xosview-1.5.1 supports SMP. And kernels above 2.1.85
          (included) have the /proc/stat/cpuX entry.

          The official homepage for xosview is:
          http://lore.ece.utexas.edu/~bgrayson/xosview.html

          The various forissier's kernel patches are at:
          http://www-isia.cma.fr/~forissie/smp_kernel_patch/

  3.  `How can I program to use two (or more CPUs) ?' Use a
     kernel-thread library. A good library, the pthread library made by
     Xavier Leroy (http://pauillac.inria.fr/~xleroy/linuxthreads/).

     LinuxThread is now integrated with glibc2 (aka libc6).

     From `Jakob Oestergaard': Also consider using MPI. It's the
     industry standard message passing interface. It doesn't give you
     shared memory like threads, but it allows you to use your program
     in a cluster too.

  4.  `What has changed in the threads packages, linuxthread, etc.'
     Glibc is the big change. glibc is threadsafe and includes
     linuxthreads Posix.4 threads by default. Real time signals are
     also in glibc so POSIX AIO should also be in glibc2.1 (I hope).

  5.  `How can I enable more than 1 process for my kernel compile?' use:
                  # make [modules|zImage|bzImages] MAKE="make -jX"
                  where X=max number of processes.
                  WARNING: This won't work for "make dep".

     With a 2.1.x like kernel, see also the file
     `/usr/src/linux/Documentation/smp' for specific instruction.

     BTW, since running multiple compilers allows a machine with
     sufficient memory to use use the otherwise wasted CPU time during
     I/O caused delays `make MAKE="make -j 2" -j 2' actually even helps
     on uniprocessor boxes (from `Ralf Bächle').

  6.  `Why the time given by the `time' command is false ?' (from `Joel
     Marchand') In the 2.0 series, the result given by the `time'
     command is false. The sum user+system is right *but* the spreading
     between user and system time is false.

     This bug in corrected in 2.1 series.

  7.  `How will my application perform under SMP?' Look at SMP
     Performance of Linux
     (http://www.interlog.com/~mackin/linux-smp.html) which gives useful
     hints how to bench a specific machine (from a post made by `Cameron
     MacKinnon').

  8.  `Where can I found more information about parallel programming?'
     Look at the Linux Parallel Processing HOWTO
     (http://yara.ecn.purdue.edu/~pplinux/PPHOWTO/pphowto.html)

     Lots of useful information can be found at Parallel Processing
     using Linux (http://yara.ecn.purdue.edu/~pplinux/)

Intel architecture specific questions
*************************************

Why it doesn't work on my machine?
==================================

  1.  `Can I use my Cyrix/AMD/non-Intel CPU in SMP?' `Short answer:' no.

     `Long answer:' Intel claims ownership to the APIC SMP scheme, and
     unless a company licenses it from Intel they may not use it. There
     are currently no companies that have done so.  (This of course can
     change in the future) FYI - Both Cyrix and AMD support the
     non-proprietary OpenPIC SMP standard but currently there are no
     motherboards that use it.

  2.  `Why doesn't my old Compaq work?' Put it into MP1.1/1.4 compliant
     mode.

  3.  `Why doesnt my ALR work?' From `Robert Hyatt' : ALR Revolution
     quad-6 seems quite safe, while some older revolution quad machines
     without P6 processors seem "iffy"...

  4.  `Why does SMP go so slowly?' or `Why does one CPU show a very low
     bogomips value while the first one is normal?' From `Alan Cox': If
     one of your CPU's is reporting a very low bogomips value the cache
     is not enabled on it. Your vendor probably provides a buggy BIOS.
     Get the patch to work around this or better yet send it back and
     buy a board from a competent supplier.

  5.  `I've heard IBM machines have problems'

     Some IBM machines have the MP1.4 bios block in the EBDA, allowed
     but not supported by <2.1.80. Please update to the right kernel.

     There is an old 486SLC based IBM SMP box. Linux/SMP requires
     hardware FPU support.

  6.  `Is there any advantage of Intel MP 1.4 over 1.1 specification?'
     Nope (according to Alan `:)' ), 1.4 is just a stricker specs of
     1.1.

  7.  `Why does the clock drift so rapidly when I run linux SMP?'

     This is known problem with IRQ handling and long kernel locks in
     the 2.0 series kernels.  Consider upgrading to a later 2.1 kernel
     (not garenteed to work).

     From `Jakob Oestergaard': Or, consider running xntpd. That should
     keep your clock right on time.  (I think that I've heard that
     enabling RTC in the kernel also fixes the clock drift. It works
     for me! but I'm not sure whether that's general or I'm just being
     lucky)

  8.  `Why are my CPU's numbered 0 and 2 instead of 0 and 1 (or some
     other odd numbering)?' The CPU number is assigned by the MB
     manufacturer and doesn't mean anything.  Ignore it.

  9.  `My SMP system is locking up all the time.  Black screen, nothing
     in the logs.  Help!' If you're running a 2.0 kernel, consider
     upgrading to later 2.0.32+ kernels or apply Leonard Zubkoff's
     deadlock patch.  If you still have deadlocks, apply Ingo Molnar's
     deadlock detection patch and post the results (against your
     System.map) to linux-smp or linux-kernel.  You might also consider
     running a 2.1 kernel.

Possible causes of crash
========================

   You'll find in this section some `possible' reasons for a crash of
an SMP machine (credits are due to `Jakob Oestergaard' for this part).
As far as I (david) know, theses problems are Intel specific.

   *  `Cooling problems' From `Ralf Bächle': [lsqb  ]Related to case
     size and fans] It's important that the air is flowing.  It of
     course can't where cables etc. are preventing this like in too
     small cases.  On the other side I've seen oversized cases causing
     big problems.  There are some tower cases on the market that
     actually are worse for cooling than desktops.  In short, the right
     thing is thinking about aerodynamics in the case.  Extra cases for
     hot peripherals are usefull as well.

   *  `Bad memory' Don't buy too cheap RAM and don't use mixed RAM
     modules on a motherboard that is picky about it.

     Especially Tyan motherboards are known to be picky about RAM speed.

   *  `Bad combination of different stepping CPUs' Check
     `/proc/cpuinfo' to see that your CPUs are same stepping.

   *  `You are running 2.0.33 aren't you ?' If you run 2.0.31 or 2.1.xx
     you can't be sure that SMP is stable. 2.0.33 is the right kernel
     for a production system. 2.1.xx kernels perform better, but they
     are development releases and should NOT be considered stable!

   *  `If your system is unstable, then DON'T overclock it!' ...and
     even if it is stable, DON'T overclock.

     From `Ralf Bächle': Overclocking causes very subtile problems.  I
     have a nice example, one of my overclocked old machines
     misscomputes a couple of pixels of a 640 x 400 fractal.  The
     problem is only visible when comparing them using tools. So better
     say `never, nuncas, jamais, niemals' overclock.

   *  `2.0.x kernel and fast ethernet' (from `Robert G. Brown') 2.0.X
     kernels on high performance fast ethernet systems have significant
     (and known) problems with a race/deadlock condition in the
     networking interrupt handler.

     The solution is to get the latest 100BT development drivers from
     CESDIS (ones that define SMPCHECK).

   *  `A bug in the 440FX chipset' (from `Emil Briggs') If you had a
     system using the 440FX chipset then your problem with the lockups
     was possibly due to a documented errata in the chipset.  Here is a
     reference

     References: Intel 440FX PCIset 82441FX (PMC) and 82442FX (DBX)
     Specification Update.  pg. 13

     http://www.intel.com/design/pcisets/specupdt/297654.htm

     The problem can be fixed with a BIOS workaround (Or a kernel
     patch) and in fact David Wragg wrote a patch that's included with
     Richard Gooch's MTTR patch. For more information and a fix look
     here:

     http://nemo.physics.ncsu.edu/~briggs/vfix.html

   *  `DONT run emm386.exe before booting linux SMP' From `Mark
     Duguid', dumb rule [num   ]1 with W6LI motherboards. `;)'

   Some hardware is also known to cause problems. This includes:
   *  `Adaptec SCSI controllers' Latest news (05 may 1998):

     The latest version of this driver (5.0.13 at this writting) should
     be SMP safe (both in 2.0 and 2.1 kernels). Doug had verified it
     through code review and intensive tests (on a dual PII system).

     This said, some machines still have problems. Doug has kindly
     given some explanations:
        *  Doug: The simple fact of the matter is that this version of
          the aic7xxx driver places a larger demand on DMA transfers
          than the old driver did, especially for for smaller DMA
          transactions. So, the point should be made that this version
          of the driver *is* more demanding on your system than either
          the 4.1 or 4.1.1 driver, and that certain marginal systems
          may get hit by this problem (such as my own P133 system is).

        *  In 2.1 kernels, they are still problems (with IRQ) with the
          kernel (2.1.99) and they are not related to this specific
          driver. People are working on this.

   *  `3Com 3c905 cards' Some work, some don't. Try disabling
     busmastering if your system is unstable.

Motherboard specific information
================================

   Some more specific information can be found with the survey of SMP
motherboards (http://styx.phy.vanderbilt.edu/smp/mainboards.html)

Motherboards with known problems
--------------------------------

   *  Gigabyte Solution: BIOS upgrade

   *  SuperMicro Solution: BIOS upgrade

   *  EPoX KP6-LS (`Christopher Allen Wing', 16 march 1998) It appears
     to have the same BIOS related BogoMIPS problem as other
     motherboards. (one CPU only gives about 3 BogoMIPS, the other
     gives the full amount) All 2.0.x kernels lock up soon after
     booting, late 2.1.x kernels run slowly but don't seem to lock up.
     There is no BIOS upgrade available (yet). I wrote the manufacturer
     but have not received a reply.

   *  Tyan Tyan motherboards are known to be picky about RAM speed
     (`Jakob Oestergaard').

     From `Doug Ledford' about the onboard aic-7895 SCSI controller
     (for which he wrote the driver): "BTW, make sure you have at least
     BIOS version 1.16 on that Tyan motherboard.  The 1.15 and below
     BIOS versions have a bug relating to IRQ allocation for the 7895
     SCSI controller" (submitted by `Szakacsits Szabolcs').

     But, as motherboard problems are more close to grayshade than
     black and white `;)', `Richard Jelinek' (from S.u.S.E.) tells that
     they sell/sold several Tyan Titan Pro (Dual-PPro) Motherboards.
     They work perfectly with SMP.

   *  GA686DLX (`Andrew Crane') Same BIOS related BogoMIPS problem as
     other motherboards.

     Solution from Alan Cox: Congratulations, send the bill for your
     hair damage to the supplier. You have yet another SMP box with
     faulty bios. There is a patch for 2.0.x on www.uk.linux.org and
     there are people working on generic MTRR handling for 2.1.x

     From `Claus-Justus Heine': I'm able to boot above MB with 2.1.90 +
     Ingo's apic + Richard's mtrr patches.

   *  MS-6114 More details for this motherboard at
     http://www.msi.com.tw/product/6114/6114.htm
     (http://www.msi.com.tw/product/6114/6114.htm)

     Solution: BIOS upgrade

     Somebody experienced solid hangs (nothing in the log files) under
     constant load of about 5 running processes within less than 12
     hours with AMI BIOS v1.1.  v1.4b3 runs without problems.

Motherboards with no known problems
-----------------------------------

   *  AIR P6NDP and P6NDI (`Leonard N. Zubkoff') My primary production
     machine is based on an AIR P6NDP and one of my test machines uses
     a P6NDI.  Both seem to be fine motherboards in my experience.  The
     P6NDI BIOS is a little conservative in its programming of the
     Natoma chipset for 50ns EDO, but a minor tweek to one register in
     rc.local took care of that.

   *  AIR 54CDP (`Chris Mauritz') You can also list the following
     motherboard as working with no problems:

     AIR 54CDP motherboard / EISA/PCI / onboard aic7870 / dual P120 /
     Redhat 5.0 (2.0.32 and 2.0.33 kernels)

     However, `Jon Lewis' adds the following comments: Has Chris tried
     putting multiple PCI cards in these motherboards...say a SCSI card
     and a network card?  Due to BIOS brain damage, the 54CDP insists
     on having all PCI devices share a single IRQ.  AIR lead me on for
     2 weeks and then said "there have been / will be no updates for
     your board".  This means I can't have Tulip network cards and PCI
     SCSI without hacking the drives (which seems to work, though I was
     cautioned not to do it).

   *  HP XU 6/200 (`Jean-Francois Micouleau') Works with 2.0 and 2.1
     kernels. Some problems under high network load with 2.0.x kernel.
     Works under 2.1.78 with Ingo Molnar IO-APIC patch.

   *  Elitegroup P6FX2-A (`Benedikt Heinen') Had this mainboard running
     with ONE PPro on it for several months, and since about a year,
     it's running without problems with TWO PPro 200MHz. The only
     crashes this machine ever experienced were before Leonard
     Zubkoff's deadlock-patches for Linux 2.0.30... `;)'

     Elitegroup P6FX2-A / ISA/PCI / Dual PPro200 / Debian "hamm"

   *  QDI P6I440LX/DP "Legend IV" (`Tony Kocurko') The QDI P6I440LX/DP
     "Legend IV" dual Pentium II motherboard now boots properly with
     Linux-SMP. As of 0326 hours on 15 April, there is a necessary BIOS
     upgrade to version 1.6 at the Canadian site (http://www.qdi.ca).
     Previously, the Legend IV would bring up the first processor
     normally, but the `/proc/cpuinfo' file would show the second
     processor barely alive. The BIOS update seems to have corrected
     the problem.

   *  ASUS P2L97-DS ASUS P2L97-DS works great under Linux-SMP (from
     `Richard Jelinek').

   *  Micronics W6-LI From `Mark Garlanger': 2.0.33 is very stable on
     the board, and the 2.1.xx kernels are improving.  People have also
     been able to run other OSes including Windows NT(yuck...),
     FreeBSD-SMP, Rhapsody.

Useful pointers
***************

Various
=======

   *  Parallel Processing using Linux
     (http://yara.ecn.purdue.edu/~pplinux/)

   *  Linux Parallel Processing HOWTO
     (http://yara.ecn.purdue.edu/~pplinux/PPHOWTO/pphowto.html)

   *  `(outdated)' Linux SMP home page
     (http://www.uk.linux.org/SMP/title.html)

   *  linux-smp mailing list To `subscribe', send `subscribe linux-smp'
     in the message body at majordomo@vger.rutgers.edu

     To `unsubscribe', send `unsubscribe linux-smp' in the message body
     at majordomo@vger.rutgers.edu

     Linux SMP archives (http://www.linuxhq.com/lnxlists/linux-smp/)

   *  pthread library made by Xavier Leroy
     (http://pauillac.inria.fr/~xleroy/linuxthreads/)

   *  Survey of SMP motherboards
     (http://styx.phy.vanderbilt.edu/smp/mainboards.html)

   *  procps (http://www.cs.inf.ethz.ch/~rauch/procps.html)

   *  xosview (http://lore.ece.utexas.edu/~bgrayson/xosview.html)

   *  Pentium Pro Optimized BLAS and FFTs for Intel Linux
     (http://www.cs.utk.edu/~ghenry/distrib/)

   *  SMP Performance of Linux
     (http://www.interlog.com/~mackin/linux-smp.html)

   *  Multithreaded programs on linux
     (http://www.e.kth.se/~e94_bek/mthread.html)

SMP specific patches
====================

   *  Forissier kernel patches
     (http://www-isia.cma.fr/~forissie/smp_kernel_patch/)

   *  Patch for a bug in the 440FX chipset
     (http://nemo.physics.ncsu.edu/~briggs/vfix.html)

   *  MTRR patch (latest version: 1.9)
     (http://www.atnf.csiro.au/~rgooch/kernel-patches.html)

Glossary
********

   *  `SMP' Symetric Multi-Processors

   *  `APIC' Advanced Programmable Interrupt Controler

   *  `thread' A thread is a processor activity in a process. The same
     process can have multiple threads. Those threads share the process
     address space and can therefore share data.

   *  `pthread' Posix thread, threads defined by the Posix standard.

List of contributors
********************

   Many thanks to those who help me to maintain this FAQ.

   *  Emil Briggs

   *  Robert G. Brown

   *  Samuel S. Chessman

   *  Alan Cox

   *  Andrew Crane

   *  Mark Duguid

   *  Jocelyne Erhel

   *  Byron Faber

   *  Mark Garlanger

   *  Claus-Justus Heine

   *  Benedikt Heinen

   *  Robert Hyatt

   *  Richard Jelinek

   *  Tony Kocurko

   *  Doug Ledford

   *  Cameron MacKinnon

   *  Joel Marchand

   *  Chris Mauritz

   *  Jean-Francois Micouleau

   *  Jakob Oestergaard

   *  Jean-Michel Rouet

   *  Ralf Bächle

   *  Sumit Roy

   *  Szakacsits Szabolcs

   *  El Warren

   *  Christopher Allen Wing

   *  Leonard N. Zubkoff