I bought my first Arduino to help answer the question, "What can open source hardware teach me about running concurrent processes on my own computer?" The second was purchased to indulge my quest for knowledge of parallelism at a similarly low level, with Symmetric Multiprocessing (SMP). The third - a Mini - was just because I'm hooked. This is a post about the first and second. I'm only telling you about the Mini because of how great I think these little things are!
Threading: it's a technique to allow two logical processes to run concurrently, using shared physical computational resources. A program is usually logically separated into code and data. Code is executed, while data is read/written during execution (although executing data is not usually "done", it's completely possible). In a single-threaded program there is one call stack (representing the current method) and a bunch of special memory locations called registers, one of the registers "points" to the next instruction that will be executed. In a dual-threaded program, there must be two call stacks (one for each method currently being executed) and two sets of special memory locations (in one, the instruction pointer (IP) points to one part of the code; in the other it (usually) points somewhere different in the same code). However, in a single-processor system only one of these sets of memory locations can ever be stored in the physical registers at any given time, because there is only one set of physical registers. The other must be saved somewhere nearby, waiting in the wings. Something has to stop the current thread X, save its register values, restore Y's register values and start Y... e.g. the context switch.
SMP: two (or more) physical processors share the same bus (and thus access to all other hardware). With one thread, one processor would sit idle. With two threads, both processors get to do something (and unlike our single-processor example above, both sets of register values can now be physically stored in processor registers). With just two threads, we don't have to switch contexts and both processes can keep busy the whole time. Well, not all the time. While processor A is performing I/O operations against device C (reading/writing values in memory,) the bus route is busy and processor B must wait for access to C. Immediately, a memory bottleneck. Luckily, not all I/O needs to go over the bus every time: each processor has its own little local cache.
In the case of the Arduino, I don't have any fancy switching hardware to blur the distinction between buses and networks, so my bus is a set of parallel copper wires (some wires for addresses and some for data) connecting two Arduino devices to four EEPROM memory devices. Data written to an EEPROM by one processor can be read from the same EEPROM by another processor, and vice versa.
If both the Arduino devices are masters and all the EEPROM devices are slaves, how do we arbitrate access to the bus? How indeed. Something needs to send a clock signal. Something needs to stop both processors from trying to talk at the same time.