wangkuiyi-patch-2
Yancey1989
7 years ago
commit
13e7194ebd
@ -0,0 +1,33 @@
|
||||
if(NOT WITH_GPU)
|
||||
return()
|
||||
endif()
|
||||
|
||||
set(TENSORRT_ROOT "/usr" CACHE PATH "TENSORRT ROOT")
|
||||
find_path(TENSORRT_INCLUDE_DIR NvInfer.h
|
||||
PATHS ${TENSORRT_ROOT} ${TENSORRT_ROOT}/include
|
||||
$ENV{TENSORRT_ROOT} $ENV{TENSORRT_ROOT}/include
|
||||
NO_DEFAULT_PATH
|
||||
)
|
||||
|
||||
find_library(TENSORRT_LIBRARY NAMES libnvinfer.so libnvinfer.a
|
||||
PATHS ${TENSORRT_ROOT} ${TENSORRT_ROOT}/lib
|
||||
$ENV{TENSORRT_ROOT} $ENV{TENSORRT_ROOT}/lib
|
||||
NO_DEFAULT_PATH
|
||||
DOC "Path to TensorRT library.")
|
||||
|
||||
if(TENSORRT_INCLUDE_DIR AND TENSORRT_LIBRARY)
|
||||
set(TENSORRT_FOUND ON)
|
||||
else()
|
||||
set(TENSORRT_FOUND OFF)
|
||||
endif()
|
||||
|
||||
if(TENSORRT_FOUND)
|
||||
file(READ ${TENSORRT_INCLUDE_DIR}/NvInfer.h TENSORRT_VERSION_FILE_CONTENTS)
|
||||
string(REGEX MATCH "define NV_TENSORRT_MAJOR +([0-9]+)" TENSORRT_MAJOR_VERSION
|
||||
"${TENSORRT_VERSION_FILE_CONTENTS}")
|
||||
string(REGEX REPLACE "define NV_TENSORRT_MAJOR +([0-9]+)" "\\1"
|
||||
TENSORRT_MAJOR_VERSION "${TENSORRT_MAJOR_VERSION}")
|
||||
|
||||
message(STATUS "Current TensorRT header is ${TENSORRT_INCLUDE_DIR}/NvInfer.h. "
|
||||
"Current TensorRT version is v${TENSORRT_MAJOR_VERSION}. ")
|
||||
endif()
|
||||
@ -0,0 +1,57 @@
|
||||
## Distributed training overview doc
|
||||
|
||||
Currently Paddle Fluid use parameter server architecture to support distributed training.
|
||||
|
||||
For synchronous and asynchronous training, the differences are mostly in the logic of parameter server. Now we have already support synchronous training.
|
||||
|
||||
### Synchronous training
|
||||
|
||||
The training process of synchronous training is:
|
||||
|
||||
|
||||
|
||||
1. Pserver
|
||||
1. set `barrier_condition_` to 0 and waits for trainers to send gradient.
|
||||
1. Trainer
|
||||
1. Trainer read minibatch of data, run forward-backward with local parameter copy and get the gradients for parameters.
|
||||
1. Trainer use split op to split all the gradient into blocks. The split method is determined at compile time.
|
||||
1. Trainer use send_op to send all the split gradients to corresponding parameter server.
|
||||
1. After trainer send all the gradients, it will send a `BATCH_BARRIER_MESSAGE` to all pservers.
|
||||
1. Trainer call GetVariable to pserver and wait for `barrier_condition_` on pserver to be 1.
|
||||
1. Pserver
|
||||
1. Pserver will count the number of `BATCH_BARRIER_MESSAGE`.
|
||||
1. When the count of `BATCH_BARRIER_MESSAGE` is equal to the number of Trainer. Pserver thinks it received all gradient from all trainers.
|
||||
1. Pserver will run the optimization block to optimize the parameters.
|
||||
1. After optimization, pserver set `barrier_condition_` to 1.
|
||||
1. Pserver wait for `FETCH_BARRIER_MESSAGE`.
|
||||
1. Trainer.
|
||||
1. The trainer uses GetVariable to get all the parameters from pserver.
|
||||
1. Trainer sends a `FETCH_BARRIER_MESSAGE` to each pserver.
|
||||
1. Pserver.
|
||||
1. when the number of `FETCH_BARRIER_MESSAGE` reach the number of all trainers. Pserver think all the parameters have been got. it will go back to 1. to set `barrier_condition_` to 0.
|
||||
|
||||
### Asynchronous training
|
||||
In the above process. There are two barriers for all trainers to synchronize with each other. In asynchronous training, these two barriers are not needed. The trainer can just send gradients to pserver and then get parameters back.
|
||||
|
||||
The training process of asynchronous training can be:
|
||||
|
||||
|
||||
|
||||
1. Pserver:
|
||||
1. Each parameter has a queue to receive its gradient from trainers.
|
||||
1. Each parameter has a thread to read data from the queue and run optimize block, using the gradient to optimize the parameter.
|
||||
1. Using an independent thread to handle RPC call `GetVariable` for trainers to get parameters back.(Maybe here we should use a thread pool to speed up fetching the parameters.)
|
||||
|
||||
1. Trainer:
|
||||
1. Trainer read a batch of data. Run forward and backward with local parameter copy and get the gradients for parameters.
|
||||
1. Trainer split all gradients to blocks and then send these gradient blocks to pservers(pserver will put them into the queue).
|
||||
2. Trainer gets all parameters back from pserver.
|
||||
|
||||
### Note:
|
||||
There are also some conditions that need to consider. For exmaple:
|
||||
|
||||
1. If trainer needs to wait for the pserver to apply it's gradient and then get back the parameters back.
|
||||
1. If we need a lock between parameter update and parameter fetch.
|
||||
1. If one parameter must be on one server, or it can also be split and send to multiple parameter servers.
|
||||
|
||||
The above architecture of asynchronous training can support different mode, we can have a detailed test in the future for these problems.
|
||||
@ -0,0 +1,58 @@
|
||||
# Design Doc: Asynchronous Update With Distributed Training
|
||||
|
||||
## Background
|
||||
|
||||
For the typical synchronous distributed training, some significant steps are as follows:
|
||||
|
||||
1. A Trainer will compute the gradients and SEND them to the Parameter Server(PServer) nodes.
|
||||
1. After the PServer node received gradients came from all the Trainers, It will aggregate the
|
||||
gradient variables for the same parameter into one gradient variable and then apply the aggregated
|
||||
gradient to the respective parameter, finally using an optimize algorithms(SGD, Monument...)
|
||||
to update the parameters.
|
||||
1. The Trainer would wait for the PServers finished the optimize stage, and GET the parameters from PServer,
|
||||
so all the Trainers would get the same parameters.
|
||||
|
||||
In the synchronously distributed training, there should be a `Barrier` to synchronise the
|
||||
parameters after the optimizing stage. The performance of a distributed training job would
|
||||
depend on the slowest node if there were hundreds or thousands of training nodes in a
|
||||
Job, the performance of synchronously distributed training might be very poor because of
|
||||
the slow node. So this design doc would introduce an approach to implement
|
||||
*asynchronously* distributed training in PaddlePaddle Fluid.
|
||||
|
||||
## Design
|
||||
|
||||
<img src="./src/async_update.png" width="600"/>
|
||||
|
||||
As the figure above, we describe a global view of asynchronously update process and use
|
||||
the parameter `w1` as an example to introduce the steps:
|
||||
1. For each gradient variables, they may distribute on different GPU card and aggregate
|
||||
them while they are all calculated.
|
||||
1. Split the gradient variable into multiple blocks according to the number of PServer
|
||||
instances and then send them.
|
||||
1. PServer would run an `Optimize Block` using a specified optimize algorithm to update
|
||||
the specified parameter.
|
||||
1. The trainer will fetch latest parameter from PServer before running forward Op which depends
|
||||
on the specified parameter.
|
||||
1. Broadcast the received variable into multiple GPU cards and continue to run the next
|
||||
mini-batch.
|
||||
|
||||
### Trainer
|
||||
|
||||
- For the multiple devices distributed training, we need to aggregate the gradient
|
||||
variables which placed on different devices firstly and then schedule a `SendVars` Operator to
|
||||
send the gradient variables to the multiple PServer instances.
|
||||
- Schedule `FetchVars` operator to fetch the latest parameter from PServer before running
|
||||
the forward ops.
|
||||
- There could be a large number of gradient variables to be sent, so we need to use another
|
||||
thread pool(IO Threadpool) whose a number of the schedulable threads is larger than the
|
||||
computing thread pool to avoid competitive the thread resources with computing.
|
||||
|
||||
### Parameter Server
|
||||
|
||||
<img src="./src/async_pserver.png" width="750"/>
|
||||
|
||||
- There should be multiple trainer instances want to optimize the same parameter at
|
||||
the same time, to avoid the racing, we need one `BlockingQueue` for each gradient
|
||||
variable to process them one by one.
|
||||
- We need a `Map` structure to map a gradient variable name to the `OptimizeBlock` which
|
||||
can optimize the respective parameter.
|
||||
@ -0,0 +1,46 @@
|
||||
# MPI-enabled PaddlePaddle Design doc
|
||||
|
||||
# Background
|
||||
When we do distribute multi GPU training, the communication overhead between servers become the major bottleneck, because of the following reasons:
|
||||
1. Must copy at least once from GPU to CPU memory so that the data can be ready to transfer. And for the pserver side, copy data from CPU to GPU introduce more overhead.
|
||||
2. GPU->CPU data transfer is 10 times slower than data transfer between GPUs or between PCIe devices.
|
||||
3. TCP connections can not make full use of RDMA 100Gb devices.
|
||||
|
||||
We will use OpenMPI API to PaddlePaddle, which can bring two benefits to PaddlePaddle:
|
||||
1. Enable RDMA with PaddlePaddle, which bring high-performance low latency networks.
|
||||
2. Enable GPUDriect with PaddlePaddle, which bring the highest throughput and lowest latency GPU read and write.
|
||||
|
||||
# Change list
|
||||
* Compile args: Need add compile args to enable MPI support.
|
||||
* Execute args: Need add execute args to assign when and how to use MPI operations.
|
||||
* New ops: Need new op ```mpi_send_op``` and ```mpi_listenandserve_op``` to support MPI send and receive.
|
||||
* Transpiler optimized: Which can add ```mpi_send_op``` and ```mpi_listenandserve_op``` to the running graph.
|
||||
* MPI utils package: Need MPI utils package as the low-level API supported.
|
||||
|
||||
## Compile args
|
||||
Because MPI or CUDA need hardware supported, so we will add compile args to enable MPI support and control compiling.Add ```WITH_MPI``` compile args to control MPI to use or not. If the ```WITH_MPI``` is ```ON```, compile system will find openMPI codes in configuration. We should prepare openMPI environment before compiling.
|
||||
|
||||
## Execute args
|
||||
Launch the script using the ```mpirun``` launcher, For example: ```mpirun -np 3 -hosts node1,node2,node3 python train.py```. By doing this, We can number the actors (trainer/pserver/master) with o .. (n-1). The node's number is the Rank of the calling process in a group of comm (integer), The MPI processes identify each other using a Rank ID. We have to create a mapping between PaddlePaddle's nodes and their Rank ID so that we can communicate with the correct destinations when using MPI operations.
|
||||
|
||||
## New ops
|
||||
We won't replace all the gRPC requests to MPI requests, the standard gRPC library is used for all administrative operations and the MPI API will be used to transfer tensor or selectRows to Pservers. The base of this idea, we create two new operators to handle requests and receives, the two operators are ```mpi_send_op``` and ```mpi_listenandserve_op```. They are a little similar to [send_op](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/operators/send_op.cc) and [listen_and_serv_op](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/operators/listen_and_serv_op.cc), also, We will build a new module to package MPI send and receive process.
|
||||
|
||||
### mpi_send_op
|
||||
Very similar with ```send_op```, we will replace gRPC code which used to send gradient with ```mpi_module```, at the same time, we will wrap it with ```framework::Async```.
|
||||
|
||||
### mpi_listenandserve_op
|
||||
Very similar with ```listen_and_serv_op```, we will replace gRPC code which used to receive gradient with ```mpi_module```, at the same time, we will wrap it with ```framework::Async```.
|
||||
|
||||
## Transpiler optimized
|
||||
**We can get env ```OMPI_COMM_WORLD_SIZE``` and ```OMPI_COMM_WORLD_RANK``` to distinguish use MPI or not, If we use openMPI, the variable in env must exist.**
|
||||
if confirm to use MPI, we will modify ```send_op``` to ```mpi_send_op``` in distribute_transpiler, and modify ```listenandserve_op``` to ```mpi_listenandserve_op``` also.
|
||||
|
||||
## MPI utils package
|
||||
In this package, We will write openMPI low-level API to use MPI.
|
||||
The API included in this package are:
|
||||
* MPI send and receive module, We will build a new module to package MPI send and receive process. MPI send and receive are different to gRPC, the MPI [recvice](https://www.open-mpi.org/doc/v1.8/man3/MPI_Irecv.3.php) must know receive buffer size and receive buffer element. For this reason, We have to make communications twice, the first one is to send metadata about gradient through gRPC, the second one is the real communication through MPI which send gradient data to mpi_listenandserve_op.
|
||||
The detailed flow is below:
|
||||
|
||||
* MPI global configurations, which store the Rank ID and the mapping in global variables, for example:
|
||||
gRPC client : MPI nodes :``` 127.0.0.1:32004 : 3 ```
|
||||
|
After Width: | Height: | Size: 180 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 166 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 180 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 104 KiB |
|
After Width: | Height: | Size: 184 KiB |
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in new issue