\appendix
\section{Benchmark Code}
\label{appendix: sim}

\begin{figure}[tbh]
{\small
\begin{verbatim}
global
  integer  work  /* the work parameter */
  integer  sum_latency, latencyN, list_place
  diffracting_tree counter
  real avg_latency[], total_latency
 
per processor code {
    local
      integer latency, start, end, i, j, randw
    forever
       start = current_time()
       i = fetch_and_increment(counter)
       end = current_time()
       atomically {
         latency = end - start
         sum_latency = sum_latency + latency
         latencyN = latencyN + 1
         if latencyN  == 1000 then
            avg_latency[list_place] = 
                      sum_latency / latencyN
            list_place = list_place + 1
            sum_latency = 0
            latencyN = 0
         endif
       }
       randw = random(work)
       repeat randw times
         /* nothing */
       endrepeat
       if i > MAXINDEX then quit
    endfor
}

when all processors are done do {
  local 
     integer i
  for i = 2 to list_place-1 do
     total_latency = total_latency + avg_latency[i]
  endfor
  total_latency = total_latency / ( list_place - 2 )
}
\end{verbatim}
}
\caption{Code for Measuring Fetch\&Increment Latency}
\label{figure: benchcode}
\end{figure}






The code in Figure~\ref{figure: benchcode}
measures the average number of cycles it takes the counter to 
deliver an index. In each simulation a counter was accessed 
between 10,000 and 20,000 times, and the time to deliver an index was 
measured for each access. The average latency was measured after each 
1000 indices delivered, the average of these times is the latency of 
the counter. In order to take into account start-up times we ignored 
the latency of the first 2,000 indices delivered. The pseudo-code in 
Figure~\ref{figure: benchcode} illustrates how the measurements were 
performed. In the code, {\tt current\_time} gives the number of 
cycles since some agreed global event, {\tt fetch\_and\_increment} is the 
diffracting tree, {\tt MAXINDEX} is the last index to be delivered 
and {\tt total\_latency} is estimate of the counter's latency. 
By monitoring the different values in the {\tt avg\_latency} array for 
different values of {\tt i}, we can make sure that the simulation has 
reached a steady state.
