Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into quick_help
tangwei12
7 years ago
commit
e05f3eaf69
@ -1,67 +0,0 @@
|
||||
# Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
if(NOT WITH_GPU)
|
||||
return()
|
||||
endif()
|
||||
|
||||
include(ExternalProject)
|
||||
|
||||
set(NCCL_SOURCE_DIR ${THIRD_PARTY_PATH}/nccl)
|
||||
|
||||
include_directories(${NCCL_SOURCE_DIR}/src/extern_nccl/src)
|
||||
|
||||
if(WITH_DSO)
|
||||
# If we use DSO, we do not build nccl, just download the dependencies
|
||||
set(NCCL_BUILD_COMMAND "")
|
||||
set(NCCL_INSTALL_COMMAND "")
|
||||
set(NCCL_INSTALL_DIR "")
|
||||
else()
|
||||
# otherwise, we build nccl and link it.
|
||||
set(NCCL_INSTALL_DIR ${THIRD_PARTY_PATH}/install/nccl)
|
||||
# Note: cuda 8.0 is needed to make nccl
|
||||
# When cuda is not installed on the system directory, need to set CUDA_HOME to your cuda root
|
||||
set(NCCL_BUILD_COMMAND "make -j 8")
|
||||
set(NCCL_INSTALL_COMMAND "make install PREFIX=${NCCL_INSTALL_DIR}")
|
||||
endif()
|
||||
|
||||
ExternalProject_Add(
|
||||
extern_nccl
|
||||
${EXTERNAL_PROJECT_LOG_ARGS}
|
||||
GIT_REPOSITORY "https://github.com/NVIDIA/nccl.git"
|
||||
GIT_TAG "v1.3.4-1"
|
||||
PREFIX "${NCCL_SOURCE_DIR}"
|
||||
UPDATE_COMMAND ""
|
||||
CONFIGURE_COMMAND ""
|
||||
BUILD_COMMAND "${NCCL_BUILD_COMMAND}"
|
||||
INSTALL_COMMAND "${NCCL_INSTALL_COMMAND}"
|
||||
INSTALL_DIR "${NCCL_INSTALL_DIR}"
|
||||
TEST_COMMAND ""
|
||||
)
|
||||
|
||||
if(WITH_DSO)
|
||||
if(${CMAKE_VERSION} VERSION_LESS "3.3.0")
|
||||
set(dummyfile ${CMAKE_CURRENT_BINARY_DIR}/lib_nccl_dummy.c)
|
||||
file(WRITE ${dummyfile} "const char * dummy_nccl = \"${dummyfile}\";")
|
||||
add_library(nccl STATIC ${dummyfile})
|
||||
else()
|
||||
add_library(nccl INTERFACE)
|
||||
endif()
|
||||
else()
|
||||
add_library(nccl STATIC IMPORTED GLOBAL)
|
||||
set_property(TARGET nccl PROPERTY IMPORTED_LOCATION
|
||||
${NCCL_INSTALL_DIR}/lib/libnccl_static.a)
|
||||
endif()
|
||||
|
||||
add_dependencies(nccl extern_nccl)
|
||||
@ -0,0 +1,57 @@
|
||||
## Distributed training overview doc
|
||||
|
||||
Currently Paddle Fluid use parameter server architecture to support distributed training.
|
||||
|
||||
For synchronous and asynchronous training, the differences are mostly in the logic of parameter server. Now we have already support synchronous training.
|
||||
|
||||
### Synchronous training
|
||||
|
||||
The training process of synchronous training is:
|
||||
|
||||
|
||||
|
||||
1. Pserver
|
||||
1. set `barrier_condition_` to 0 and waits for trainers to send gradient.
|
||||
1. Trainer
|
||||
1. Trainer read minibatch of data, run forward-backward with local parameter copy and get the gradients for parameters.
|
||||
1. Trainer use split op to split all the gradient into blocks. The split method is determined at compile time.
|
||||
1. Trainer use send_op to send all the split gradients to corresponding parameter server.
|
||||
1. After trainer send all the gradients, it will send a `BATCH_BARRIER_MESSAGE` to all pservers.
|
||||
1. Trainer call GetVariable to pserver and wait for `barrier_condition_` on pserver to be 1.
|
||||
1. Pserver
|
||||
1. Pserver will count the number of `BATCH_BARRIER_MESSAGE`.
|
||||
1. When the count of `BATCH_BARRIER_MESSAGE` is equal to the number of Trainer. Pserver thinks it received all gradient from all trainers.
|
||||
1. Pserver will run the optimization block to optimize the parameters.
|
||||
1. After optimization, pserver set `barrier_condition_` to 1.
|
||||
1. Pserver wait for `FETCH_BARRIER_MESSAGE`.
|
||||
1. Trainer.
|
||||
1. The trainer uses GetVariable to get all the parameters from pserver.
|
||||
1. Trainer sends a `FETCH_BARRIER_MESSAGE` to each pserver.
|
||||
1. Pserver.
|
||||
1. when the number of `FETCH_BARRIER_MESSAGE` reach the number of all trainers. Pserver think all the parameters have been got. it will go back to 1. to set `barrier_condition_` to 0.
|
||||
|
||||
### Asynchronous training
|
||||
In the above process. There are two barriers for all trainers to synchronize with each other. In asynchronous training, these two barriers are not needed. The trainer can just send gradients to pserver and then get parameters back.
|
||||
|
||||
The training process of asynchronous training can be:
|
||||
|
||||
|
||||
|
||||
1. Pserver:
|
||||
1. Each parameter has a queue to receive its gradient from trainers.
|
||||
1. Each parameter has a thread to read data from the queue and run optimize block, using the gradient to optimize the parameter.
|
||||
1. Using an independent thread to handle RPC call `GetVariable` for trainers to get parameters back.(Maybe here we should use a thread pool to speed up fetching the parameters.)
|
||||
|
||||
1. Trainer:
|
||||
1. Trainer read a batch of data. Run forward and backward with local parameter copy and get the gradients for parameters.
|
||||
1. Trainer split all gradients to blocks and then send these gradient blocks to pservers(pserver will put them into the queue).
|
||||
2. Trainer gets all parameters back from pserver.
|
||||
|
||||
### Note:
|
||||
There are also some conditions that need to consider. For exmaple:
|
||||
|
||||
1. If trainer needs to wait for the pserver to apply it's gradient and then get back the parameters back.
|
||||
1. If we need a lock between parameter update and parameter fetch.
|
||||
1. If one parameter must be on one server, or it can also be split and send to multiple parameter servers.
|
||||
|
||||
The above architecture of asynchronous training can support different mode, we can have a detailed test in the future for these problems.
|
||||
@ -0,0 +1,58 @@
|
||||
# Design Doc: Asynchronous Update With Distributed Training
|
||||
|
||||
## Background
|
||||
|
||||
For the typical synchronous distributed training, some significant steps are as follows:
|
||||
|
||||
1. A Trainer will compute the gradients and SEND them to the Parameter Server(PServer) nodes.
|
||||
1. After the PServer node received gradients came from all the Trainers, It will aggregate the
|
||||
gradient variables for the same parameter into one gradient variable and then apply the aggregated
|
||||
gradient to the respective parameter, finally using an optimize algorithms(SGD, Monument...)
|
||||
to update the parameters.
|
||||
1. The Trainer would wait for the PServers finished the optimize stage, and GET the parameters from PServer,
|
||||
so all the Trainers would get the same parameters.
|
||||
|
||||
In the synchronously distributed training, there should be a `Barrier` to synchronise the
|
||||
parameters after the optimizing stage. The performance of a distributed training job would
|
||||
depend on the slowest node if there were hundreds or thousands of training nodes in a
|
||||
Job, the performance of synchronously distributed training might be very poor because of
|
||||
the slow node. So this design doc would introduce an approach to implement
|
||||
*asynchronously* distributed training in PaddlePaddle Fluid.
|
||||
|
||||
## Design
|
||||
|
||||
<img src="./src/async_update.png" width="600"/>
|
||||
|
||||
As the figure above, we describe a global view of asynchronously update process and use
|
||||
the parameter `w1` as an example to introduce the steps:
|
||||
1. For each gradient variables, they may distribute on different GPU card and aggregate
|
||||
them while they are all calculated.
|
||||
1. Split the gradient variable into multiple blocks according to the number of PServer
|
||||
instances and then send them.
|
||||
1. PServer would run an `Optimize Block` using a specified optimize algorithm to update
|
||||
the specified parameter.
|
||||
1. The trainer will fetch latest parameter from PServer before running forward Op which depends
|
||||
on the specified parameter.
|
||||
1. Broadcast the received variable into multiple GPU cards and continue to run the next
|
||||
mini-batch.
|
||||
|
||||
### Trainer
|
||||
|
||||
- For the multiple devices distributed training, we need to aggregate the gradient
|
||||
variables which placed on different devices firstly and then schedule a `SendVars` Operator to
|
||||
send the gradient variables to the multiple PServer instances.
|
||||
- Schedule `FetchVars` operator to fetch the latest parameter from PServer before running
|
||||
the forward ops.
|
||||
- There could be a large number of gradient variables to be sent, so we need to use another
|
||||
thread pool(IO Threadpool) whose a number of the schedulable threads is larger than the
|
||||
computing thread pool to avoid competitive the thread resources with computing.
|
||||
|
||||
### Parameter Server
|
||||
|
||||
<img src="./src/async_pserver.png" width="750"/>
|
||||
|
||||
- There should be multiple trainer instances want to optimize the same parameter at
|
||||
the same time, to avoid the racing, we need one `BlockingQueue` for each gradient
|
||||
variable to process them one by one.
|
||||
- We need a `Map` structure to map a gradient variable name to the `OptimizeBlock` which
|
||||
can optimize the respective parameter.
|
||||
|
After Width: | Height: | Size: 180 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 166 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 180 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 184 KiB |
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in new issue