denghuilu/deepmd-kit: optimized version of deepmd-kit for CPC

In order to reproduce the results of the paper "86 PFLOPS Deep Potential Molecular Dynamics simulation of 100 million atoms with ab initio accuracy", please follow the next steps: The following modules must be loaded for the compilation and running of the optimized deepmd-kit code.

cuda/10.1.168
gcc/4.8.5
spectrum-mpi/10.3.1.2-20200121
ibm-wml-ce/1.6.2-2

Install the DeePMD-kit's C++ interface

Download all the dependent packages according to the DOI of the artifacts.

cub: https://doi.org/10.5281/zenodo.3907387
cutlass: https://doi.org/10.5281/zenodo.3960757
lammps: https://doi.org/10.5281/zenodo.3907391

Check the download packages:

ls
cub-vx.x.x.zip cutlass-vx.x.x.zip deepmd-kit-vx.x.x.zip lammps-vx.x.x.zip

Where x.x.x denotes the package version, and it may be different among the packages. Then unzip all the packages, and rename them as cub, cutlass, deepmd-kit, and lammps accordingly.

ls
cub  cutlass  deepmd-kit  lammps

For convenience, you may want to record the location of source to a variable, saying deepmd_source_dir by

cd deepmd-kit
deepmd_source_dir=`pwd`

Now, copy the submodules cub and cutlass into the deepmd-kit

cd ../cub
cp -r * $deepmd_source_dir/source/op/cuda/cub
cd ../cutlass
cp -r * $deepmd_source_dir/source/op/cuda/cutlass

Now go to the source code directory of DeePMD-kit and make a build place.

cd $deepmd_source_dir/source
mkdir build
cd build

I assume you want to install DeePMD-kit into path \$deepmd_root. then execute cmake

cmake -DTENSORFLOW_ROOT=$tensorflow_root -DCMAKE_INSTALL_PREFIX=$deepmd_root ..

where the variable tensorflow_root stores the location where the tensorflow's C++ interface is installed. The DeePMD-kit will automatically detect if a CUDA tool-kit is available on your machine and build the GPU support accordingly, then

make
make install

If everything works fine, you will have the following libraries installed in \$deepmd_root/lib

ls $deepmd_root/lib
libdeepmd.so  libdeepmd_op.so  libdeepmd_op_cuda.so

Install LAMMPS's DeePMD-kit module

DeePMD-kit provide module for running MD simulation with LAMMPS. Now make the DeePMD-kit module for LAMMPS.

cd $deepmd_source_dir/source/build
make lammps

DeePMD-kit will generate a module called USER-DEEPMD in the build directory. The source code of LAMMPS is stored in directory, for example $lammps. Now go into the LAMMPS code, and copy the DeePMD-kit module like this

cd $lammps/src/
cp -r $deepmd_source_dir/source/build/USER-DEEPMD ./

Now build LAMMPS

make yes-kspace
make yes-user-deepmd
make mpi -j4

The option -j4 means using 4 processes in parallel. You may want to use a different number according to your hardware. If everything works fine, you will end up with an executable lmp_mpi.

Reproduce the results

For example, Go to the $deepmd_source_dir/test/1_water: Change the system size by change the following line in the file 'water.in': replicate 64 32 32 then change the number of nodes (GPUs/CPUs) and run it.

The following is an example job script:

#!/bin/bash
# Begin LSF Directives
#BSUB -P projectname
#BSUB -W 0:10
#BSUB -nnodes 80
#BSUB -J deepmd
#BSUB -o deepmd.%J
#BSUB -e deepmd.%J
#BSUB -alloc_flags gpudefault

cd $LS_SUBCWD
echo $LS_SUBCWD
date
module list
module load cuda/10.1.168
export OMP_NUM_THREADS=1
export TF_CPP_MIN_LOG_LEVEL=3
jsrun --nrs 480  --tasks_per_rs 1 --cpu_per_rs 7 --gpu_per_rs 1 --rs_per_host 6 $lammps/src/lmp_mpi < water.in