- Layout of processors:
- A snapshot of the simulation (every dot is a particle):
Simulation Techniques
- Numerical Integration of the Equation Of Motion
We used Gear's 5th order predictor-corrector method to numerically integrate the classical equation of motion. The interaction potential is Lennard Jones type pairwise potential with smooth cutoff. This illustrative example shows a 2D fluid flow over an obstacle, driven by the RPM method. - The Neighbor List
A neighbor list is maintained for each particle to speed up force calculation. We do not want to recalculate the list every time a particle moves, so the range of the neighbor list should be slightly larger than the potential cutoff. Once a particle moves certain distance away from where last time its neighbor list was calculated, it needs to be updated (called a flash). There is a compromise between the frequency of updating and the range of the neighbor list, the best range depends on the particle density, potential cutoff and other parameters of the simulation. In general, when the potential cutoff is 1, the neighbor list range should be around 1.05.In the graph (not to scale), only particle 1 is interacting with the center particle, but 1,2,3,4,7 are all in the neighbor list of the center particle.
- The Bin List
The simulation cell is divided into many bins and each bin maintains a list of all the particles inside. If there is no bin list, then the maintainance of neighbor lists will be O(N^2) operations. By keeping the bin list, neighbor list calculation is restricted to particles in the nearby 9 bins (or 27 bins in 3D). It's easy to see that the bin size could be as small as the neighbor list range, but in order to divide the entire cell into an integer number of bins, the bin size is usually a bit larger.
- Reflecting Particle Method (RPM) to Drive Fluid Flow
We want to simulate fluid flow as close to nature as possible, which means near the region of interest, there should be no artificial force or perturbations of any kind. The fluid under investigation should only be pushed by other fluids. Yet, heat generated by viscous action must be removed from the simulation cell. Another requirement is that the simulation should fall into steady state readily, and it should be economical.
We devised a method which satisfies all these criteria using ``Maxwell's daemons'', in actuality a membrane at x=0: it is such that if a fluid atom crosses x=0 from left to right, it passes through without hindrance. But if it crosses x=0 from right to left, then it could be elasticly reflected with probabilty p. The membrane is otherwise transparent in the sense that particles on two sides can interact. It is easy to see that particle number and energy is conserved, thus making steady flow dynamically attractive. The heat is removed by constant entropy/information input from the computer. Here is more about the method.
- Reflecting Particle Method (RPM) to Drive Fluid Flow
Programming Techniques
- Shared Memory and Semaphore
- Optimization of C program
- Make small subroutines inline-able.
- Localize global variables.
- Minimize the number of local variables and their active ranges.
- Use #define or const whenever possible.
- Make iteration number of loops known to the compiler.
- (Unsuccessful) Increase cache hit by rearranging data storage.
Performance
- Performance on Xolas (benchmarked by a
well equilibrated system of 6000 particles for 1000 timesteps).
Data presented are the best of several runs (because the cluster is
pretty heavily loaded at the time). You can look at the raw data here. Speed is scaled to 1 for single
processor because the total flop rate is very hard to calculate.
- Performance on a single processor Pentium 166
(benchmarked by a well equilibrated system of 6000 particles for 1000
timesteps):
- Performance comparison between different machines:
Machine Type Execution time(s) Performance Compilation options Pentium166 192.21 1(Unit) gcc -O3 -ffast-math -finline-functions -fforce-mem -fforce-addr DEC AlphaStation200 4/233 127.31 1.510 cc -O5 -om -ifo -migrate -inline speed -fp_reorder -non_shared DEC 3000 104.35 1.842 *same as above* SGI Indy 237.46 0.809 cc -mips2 -O3 -kpicopt SGI Indigo 2 158.4 1.213 *same as above* SPARC Station 4 355.31 0.541 cc -xO4 -fsimple -dalign -fast SPARC Station 5 454.67 0.423 *same as above* SPARC Classic 1010.77 0.190 *same as above* Xolas(1 processor) 84.24 2.282 cc -xO5 -xdepend -xarch=v8plus -xchip=ultra -fsimple -dalign
Simulation Results
- This is the flow field of fluids flowing over a vertical wall
(enlarged):
It is impressive to us that it only took about 10 hours to simulate 6000 particles for 1,000,000 timesteps on a heavily loaded xolas0, which usually takes us a week!
- This is the flow field of fluids flowing over a square
block (enlarged):
Further Improvement
- Dynamical Load Balancing
The Code of Simulation
Every simulation run is controlled by a configuration file (which is a header file of the Source). Below is a sample configuration file and the simulation output:The configuration
//======================================================
// User Specifications
//======================================================
#define _APPLICATION "Test"
#define _2DIM
#define _NPROC 8
#define _LCFG
#define _LCFG_READ "config2.dat"
//#define _RPM 1
#define _NP 6000
#define _NSTEP 1000
#define _NDUMP 100
#define _NRESCALE 0
#define _NBL 80
#define _DT 0.001
#define _RC 2.5
#define _NNL 50
#define _TR 1.2
#define _CUBEX 150.0
#define _CUBEY 50.0
#define _PROP_MESH "mesh.out"
#define _PROP_MESH_VX
#define _PROP_MESH_VY
#define _MESHX (53*3)
#define _MESHY 53
#define _PROP_POSITION "pos.out"
This is the output of the simulation
======================================================
Atomic Simulation Library/(C/C++) (Version 1.21) : Test
Initializing...
CFG: Loaded from config2.dat
Basic parameters: particle number : 6000
Density : 0.800000, Temperature: 1.200000
Cube : 150.000000 X 50.000000
NEIB: Bin/DList: 53x17 bins, average 9.170667/50 neighbors
MDFrame: Using Gear's 5-order corr-pred method
PROP:Pos(pos.out)BIN Mesh(mesh.out,159x53)[Density Vx Vy]
Spawning children (8)...
Child 1:0--6
Child 2:7--13
Child 3:14--20
Child 4:21--27
Child 5:28--34
Child 6:35--40
Child 7:41--46
Child 0:47--52
Start running...
100 1.199459 4.563456 -0.69159692
200 1.188688 4.620878 -0.69169823
300 1.182552 4.625586 -0.69169808
400 1.201941 4.515919 -0.69169757
500 1.199589 4.524973 -0.69169805
600 1.191410 4.581781 -0.69169879
700 1.194896 4.564951 -0.69169861
800 1.199836 4.533135 -0.69169878
900 1.186185 4.594204 -0.69170014
1000 1.190483 4.544438 -0.69170024
Finished! Exiting...
CFG: Wrote to config.dat
====================================================== You can
look at the source code
here. For general comments and comparison with the first example of this project, please return to the main page. Last Modified : Tue May 20 01:41:19 1997