MATmarks User's Manual

Last Updated: Oct. 25 1998

1 Introduction and Disclaimers

1.1 What is MATmarks?

MATmarks is an environment that allows users to run multiple MATLAB® programs in parallel using the shared memory programming style. The environment extends the MATLAB® language with several primitives to enable shared variables and process synchronization.

1.2 Legalese

Read this license agreement before you start downloading/installing the software.

2 Installing MATmarks

This section is not yet complete. We need to figure out how to distribute source code as well as precompiled binaries for people who only have Treadmarks® binary licenses.

2.1 System Requirements

To run MATmarks, you need to have:

2.2 Performing the Installation

Get and unzip the MATmarks package and follow the directions therein. Depending on whether you download source code or binaries, the instructions may be different. In either case you will end up with a directory full of .mex files. If you don't know what .mex files are, check the MATLAB® documentation.

Adjust your MATLAB path - the MATLAB executable must be able to see the new .mex files that came with the MATmarks package. Check the MATLAB® documentation to see how to adjust the MATLABPATH environment variable, or just type

setenv MATLABPATH .:/home/padua2/galmasi/matmarks

Now you are done.

3 Running MATmarks

This section will help you start your first MATmarks session.

3.1 Preliminaries

  • If you haven't yet done so, this would be a great time to set your MATLABPATH environment variable to wherever you have compiled (or unzipped) the MATmarks .mex files.

    Example: setenv MATLABPATH .:/home/padua2/galmasi/matmarks

  • If you plan to always run MATmarks on the same set of machines, you could save yourself some typing by preparing a file called .Tmkrc in your home directory. This file should list all the hosts you plan to run MATmarks on in the correct order, separated by newlines. For more on this check the documentation of the TreadMarks® package.

  • You may have to edit the tkmarks script file to adjust the location of the Tk executable (usually called wish). If you don't know what this is all about, don't worry - we will come to back to it later.

    3.2 Running MATmarks Manually

    This is intended just for testing that MATmarks works. Start two instances of MATLAB® on the first two machines specified in your .Tmkrc file. Into the first window type
    This command will not return immediately, but will hang and dump out the text of another "MMK_Init" command, similar to what you just typed but with more arguments.

    Cut-and-paste this second MMK_Init command into the window of the second interpreter. Now both interpreters should present you with a prompt.

    If you got this far, MATmarks is now up and running. Type MMK_Barrier into one of the windows; it should hang the interpreter until you type MMK_Barrier into the second window, too. You are now set to run any MATmarks code you can dream up on two processors. The process is similar on any number of processors where n>1. You can run MATmarks on a single processor, but it doesn't make much sense, except for testing purposes, do to so.

    3.3 Running MATmarks with the help of tkmarks

    Here is a picture of what the tkmarks GUI frontend looks like. To see the full picture, click on the thumbnail.

    To start up tkmarks, you need to have Tcl/Tk installed on your system. Go to the MATmarks installation directory and type

    wish tkmarks 2
    to get two processes up and running. Alternatively, if you wish to specify the hosts yourself rather than let MATmarks figure them out based on your .Tmkrc file, type
    wish tkmarks host1 host2 ... 

    The window may become unwieldy if you use very many processors. We are working on fixing this problem.

    In the tkmarks application there is a scrollable window for each of the member processes; if you type into a member process window, the text will be sent to that process only. You can use the per-process text entry widgets (just below the text windows) for the same purpose.

    To talk to all processes at once, type into the all-process text entry (as shown on the figure above). The text you type in will be sent when you hit .

    The Init button takes care of sending the necessary MMK_Init commands to all processes. When this is done, the Init button will gray out to prevent you from accidentally sending the same set of commands again.

    The initial directory text widget specifies the current directory where all the MATLAB processes when Init is done.

    The Exit button kills all the MATLABtkmarks processes, but doesn't exit from tkmarks to let you see the Treadmarks® statistics. Use your window manager to kill tkmarks.

    4 MATmarks Commands

    This section contains a brief description of each MATmarks command. MATmarks is implemented as a set of .mex files, one for each command listed in this section.

    4.1 Process Initialization and Finalization

    4.1.1 Starting up MATmarks: MMK_Init

    Syntax: MMK_Init (nprocs)or
    MMK_Init ('host1 host2 ...')

    Initializes the MATmarks system.

    If the argument is a number, then MATmarks reads nprocs hostnames from the $HOME/.Tmkrc file and runs the processes on those hosts. If the $HOME/.Tmkrc does not exist, the command aborts with an error.

    If the argument to MMK_Init specifies a list of hosts, then $HOME/.Tmkrc is ignored; instead, MATmarks expects to start the processes on the hosts as listed in the argument.

    The startup process is done either manually, by the user typing MMK_Init, or automatically through the GUI .

    The manual startup process is as follows: the user emits the startup command on the first of the MATLAB® processes. The interpreter writes back to the console the proper command line to start up the second process, then hangs waiting to establish communication. If the startup sequence is done manually, the user is required to copy the proper command into the command line of the second MATLAB® process. This procedure is repeated until all the member processes have been initialized. Then, and only then, the first process returns to the command prompt.

    The automatic startup process is similar, except that the GUI takes care of the proper distribution of the MMK_Init commands.

    4.1.2 Leaving MATmarks: MMK_Exit

    Syntax: MMK_Exit [(exitcode)]

    Quits the MATmarks system. The exitcode parameter is optional.

    MMK_Exit kills not only the MATmarks subsystem, but quits the interpreter(s) as well. The standard MATLAB® exit command works as well and performs the same function.

    Note that MMK_Exit involves the passage of a barrier. Thus, if only one of e the member processes calls MMK_Exit, this process will hang. The barrier is necessary because, if one of the processes were to leave the cluster before the others finish their work, the TreadMarks® system would collapse.

    4.2 Shared variables

    4.2.1 Declaring shared variables: MMK_Share

    Syntax: MMK_Share('varname', initial_value)

    This command declares varname to be a variable shared by all processors. The initial_value parameter declares both the size and the initial value for the variable.

    A shared variable must not be allowed to grow beyond the size declared by initial_value (i.e., a shared variable can not grow in size during the execution of the program). Growing a shared variable during execution results in a runtime error which leaves all running instances of MATLAB® in an unpredictable state.

    All processors should see the same value for a shared variable. However, because of the lazy release consistency protocol that is used in TreadMarks®, processors may see different values of the variable, until a synchronization point is reached. Such uses of shared variables is considered to be a data race, and it is not considered good deterministic programming.

    4.2.2 Removing shared variables: MMK_Clear

    Syntax: MMK_Clear

    MMK_Clear removes all MATLAB® variables, including those from the shared memory. It performs pretty much the same function as the MATLAB® clear command, except that it is safe to use on shared variables.

    MMK_Clear involves the passage of a barrier. This is necessary to ensure that all member processes forget their shared variables at the same time.

    More work needs to be done on this, to make MMK_Clear discriminative (i.e. to make it free only shared variables) and to make the MATLAB® clear command safe to use.

    4.3 Synchronization

    4.3.1 Barriers

    Syntax: MMK_Barrier

    A barrier provides a global synchronization point across all processors. Processors arriving at a barrier will wait for all processors to arrive to the barrier before proceeding further. Barriers will also make the shared memory consistent in all running processes, communicating all changes that occured since the last synchronization point.

    Note that since a barrier involves communication between all processors, and our processors are workstations, barriers could be very expensive.

    Also note that MATmarks barriers, as opposed to TreadMarks® barriers, have no arguments. We don't consider this a restriction, as good shared-memory programs don't require barrier numbers anyway.

    4.3.2 Locks

    Syntax: MMK_Lock(lock_id)

    Syntax: MMK_Unlock(lock_id)

    Locks define critical sections in the code. Only one processor can be in a critical section at any moment. Locks are used to protect changes to shared variables.

    There is only a finite number of locks available. Use low numbers (i.e. less than 1000) as lock arguments.

    4.4 Miscellaneous MATmarks Commands

    4.4.1 MMK_proc_id

    Syntax: MMK_proc_id

    Returns this processor's number. Each processor running in a cluster of MATmarks processes is guaranteed to have a unique number, anywhere from 0 to MMK_nprocs-1.

    4.4.2 MMK_nprocs

    Syntax: MMK_nprocs

    Returns the number of processors involved in running the program. This number is guaranteed to be the same on all processes in a cluster.

    4.4.3 MMK_addr

    Syntax: MMK_addr varname

    This command is not strictly related to shared memory programming; it is mostly a debugging tool. MMK_addr prints out the MATLAB® address of a variable.

    The MATLAB® representation of variables, described in the documentation, is a complex structure called mxArray (or Matrix in versions of MATLAB® prior to 5.0).

    MMK_addr prints three things: the address of the allocated mxArray; the address of the real part of a "dense" (i.e. not "sparse") array; and the address of the imaginary part of the same. An all-real array will have a NULL imaginary part.

    4.4.4 MMK_Debug

    Syntax: MMK_Debug debuglevel

    This command sets the debug level of MATmarks. Unless you want to dig into the innards of MATmarks, MMK_Debug will probably not do anything useful for you.

    5 Restrictions on MATLAB® when used with MATmarks

    Yes, unfortunately there are some. Ideally we would have liked MATmarks to be a transparent subsystem of MATLAB®. However, due to several factors including the fact that MATLAB® internals are proprietary and hard to get at, we had to make do with several no-no's when writing MATmarks programs.

    1. MATmarks is only tested with "dense" arrays. There is nothing in the world why it shouldn't work with sparse arrays, but we have not tested that it actually works.

      The only thing we know for sure is that MATmarks will not work if you try to share a variable that is dense in one member of the cluster and sparse in the other. The most likely result of this setup is a core dump.

    2. Never copy a shared array with a simple assignment. The following MATLAB® code shows the problem:
      localA = sharedA; % WRONG!!!!
      sharedB = localB; % this too is wrong
      This code, when used with MATmarks, will result in unspecified behavior - most likely a crash. Instead of a direct copy, use this:
      localA(:,:) = sharedA(:,:); % slower, but correct
      sharedB(:,:) = localB(:,:);
    3. Never use clear. Use MMK_Clear instead. This restriction is due to the fact that MATmarks has its own symbol table which keeps track of shared variables only. The regular clear command will clear the MATLAB® symbol table, but not the MATmarks symbol table, leaving the system in an inconsistent state.

    4. Never try to "grow" a shared array. I.e., if you have declared an array foo to be of size 4*4, don't try to assign foo(4, 5); assignments outside array bounds cause a reallocation in MATLAB®, which would cause the process attempting this to free up shared memory without telling anyone else about it. Again, the likely result of such an action is a crash.

    6 Examples of MATmarks Code

    6.1 Sum Of First 100 Naturals

    The following example program assumes that it is run on all processors (remember that MATmarks is a MIMD programming environment, meaning that all programs do not have to execute the same code; but in this example they do). Also, it is assumed that all processes have been initialized and connected with MMK_Init.

    The program adds up the sum of 1 to 100 into the shared variable sum. You will *not* notice performance increase in this case, because the use of locks prohibits effective parallelism and because there is a lot of communication between the processes (in fact, the value of sum is ping-ponged between the processes until everybody gets dizzy).

    MMK_Share('sum', 0);
    for i=1+MMK_proc_id:MMK_nprocs:10,
      fprintf (1, 'processor %d adding %d to sum\n', MMK_proc_id, i);
      MMK_Lock (1);
      sum = sum + i;

    6.2 Distributed Matrix Multiplication

    This is a more "serious" example program, which will show real speedups executed on more than one processor. The basic assumptions (SPMD execution, MATmarks already initialized) are the same.

    The program performs a "banded" matrix multiplication. The "bands" are column-by-column, to increase the grain size of parallelism (bear in mind that MATLAB® arrays are in column-first format).

    Note how all shared assignments/operations make heavy use of the ":" operator, in order to comply with the restrictions we imposed on the language (see section 5).

    MMK_Share('n', 1000);
    MMK_Share('a', zeros(n,n));
    MMK_Share('b', zeros(n,n));
    MMK_Share('c', zeros(n,n));
    % ------------- assign variables ------------
    if (MMK_proc_id == 0),
      a(:,:) = rand(n,n);
      locala = a(:,:);
      b(:,:) = rand(n,n);
      localb = b(:,:);
    % ------------ calculate bounds -------------
    lbound = ceil (n/MMK_nprocs*MMK_proc_id);
    if (mod(n*MMK_proc_id, MMK_nprocs)==0), lbound = lbound + 1; end
    ubound = floor (n/MMK_nprocs*(MMK_proc_id+1));
    % ------------ perform multiplication -------
    t1= cputime;
    c(:, lbound:ubound) = a(:,:)*b(:,lbound:ubound);
    t2= cputime;
    % ------------- check result ----------------
    if (MMK_proc_id == 0),
      fprintf (1, 'Error: %f\n', sum(sum (locala*localb - c)));
    TIME=t2 - t1

    7 Bibliography

  • Cristiana Amza, Alan L. Cox, Sandhya Dwardakas, Pete Keleher, Honghui Lu, Ramakrishnan Rahamony, Weimin Yu and Willy Zwaenepoel: Treadmarks: Shared Memory Computing on Networks of Workstations, in IEEE Computer, Vol. 29, No. 2, February 1996.
  • Vijay Menon, Anne E. Trefethen: MultiMATLAB: Integrating MATLAB with High-Performance Parallel Computing
  • MATLAB Home Page:
  • Tcl Home Page:
  • Polaris Research Group:
  • more to come ...