Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into add_parallel_executor_tests
JiayiFeng
7 years ago
commit
93276fd146
@ -0,0 +1,57 @@
|
||||
## Distributed training overview doc
|
||||
|
||||
Currently Paddle Fluid use parameter server architecture to support distributed training.
|
||||
|
||||
For synchronous and asynchronous training, the differences are mostly in the logic of parameter server. Now we have already support synchronous training.
|
||||
|
||||
### Synchronous training
|
||||
|
||||
The training process of synchronous training is:
|
||||
|
||||
|
||||
|
||||
1. Pserver
|
||||
1. set `barrier_condition_` to 0 and waits for trainers to send gradient.
|
||||
1. Trainer
|
||||
1. Trainer read minibatch of data, run forward-backward with local parameter copy and get the gradients for parameters.
|
||||
1. Trainer use split op to split all the gradient into blocks. The split method is determined at compile time.
|
||||
1. Trainer use send_op to send all the split gradients to corresponding parameter server.
|
||||
1. After trainer send all the gradients, it will send a `BATCH_BARRIER_MESSAGE` to all pservers.
|
||||
1. Trainer call GetVariable to pserver and wait for `barrier_condition_` on pserver to be 1.
|
||||
1. Pserver
|
||||
1. Pserver will count the number of `BATCH_BARRIER_MESSAGE`.
|
||||
1. When the count of `BATCH_BARRIER_MESSAGE` is equal to the number of Trainer. Pserver thinks it received all gradient from all trainers.
|
||||
1. Pserver will run the optimization block to optimize the parameters.
|
||||
1. After optimization, pserver set `barrier_condition_` to 1.
|
||||
1. Pserver wait for `FETCH_BARRIER_MESSAGE`.
|
||||
1. Trainer.
|
||||
1. The trainer uses GetVariable to get all the parameters from pserver.
|
||||
1. Trainer sends a `FETCH_BARRIER_MESSAGE` to each pserver.
|
||||
1. Pserver.
|
||||
1. when the number of `FETCH_BARRIER_MESSAGE` reach the number of all trainers. Pserver think all the parameters have been got. it will go back to 1. to set `barrier_condition_` to 0.
|
||||
|
||||
### Asynchronous training
|
||||
In the above process. There are two barriers for all trainers to synchronize with each other. In asynchronous training, these two barriers are not needed. The trainer can just send gradients to pserver and then get parameters back.
|
||||
|
||||
The training process of asynchronous training can be:
|
||||
|
||||
|
||||
|
||||
1. Pserver:
|
||||
1. Each parameter has a queue to receive its gradient from trainers.
|
||||
1. Each parameter has a thread to read data from the queue and run optimize block, using the gradient to optimize the parameter.
|
||||
1. Using an independent thread to handle RPC call `GetVariable` for trainers to get parameters back.(Maybe here we should use a thread pool to speed up fetching the parameters.)
|
||||
|
||||
1. Trainer:
|
||||
1. Trainer read a batch of data. Run forward and backward with local parameter copy and get the gradients for parameters.
|
||||
1. Trainer split all gradients to blocks and then send these gradient blocks to pservers(pserver will put them into the queue).
|
||||
2. Trainer gets all parameters back from pserver.
|
||||
|
||||
### Note:
|
||||
There are also some conditions that need to consider. For exmaple:
|
||||
|
||||
1. If trainer needs to wait for the pserver to apply it's gradient and then get back the parameters back.
|
||||
1. If we need a lock between parameter update and parameter fetch.
|
||||
1. If one parameter must be on one server, or it can also be split and send to multiple parameter servers.
|
||||
|
||||
The above architecture of asynchronous training can support different mode, we can have a detailed test in the future for these problems.
|
||||
@ -0,0 +1,58 @@
|
||||
# Design Doc: Asynchronous Update With Distributed Training
|
||||
|
||||
## Background
|
||||
|
||||
For the typical synchronous distributed training, some significant steps are as follows:
|
||||
|
||||
1. A Trainer will compute the gradients and SEND them to the Parameter Server(PServer) nodes.
|
||||
1. After the PServer node received gradients came from all the Trainers, It will aggregate the
|
||||
gradient variables for the same parameter into one gradient variable and then apply the aggregated
|
||||
gradient to the respective parameter, finally using an optimize algorithms(SGD, Monument...)
|
||||
to update the parameters.
|
||||
1. The Trainer would wait for the PServers finished the optimize stage, and GET the parameters from PServer,
|
||||
so all the Trainers would get the same parameters.
|
||||
|
||||
In the synchronously distributed training, there should be a `Barrier` to synchronise the
|
||||
parameters after the optimizing stage. The performance of a distributed training job would
|
||||
depend on the slowest node if there were hundreds or thousands of training nodes in a
|
||||
Job, the performance of synchronously distributed training might be very poor because of
|
||||
the slow node. So this design doc would introduce an approach to implement
|
||||
*asynchronously* distributed training in PaddlePaddle Fluid.
|
||||
|
||||
## Design
|
||||
|
||||
<img src="./src/async_update.png" width="600"/>
|
||||
|
||||
As the figure above, we describe a global view of asynchronously update process and use
|
||||
the parameter `w1` as an example to introduce the steps:
|
||||
1. For each gradient variables, they may distribute on different GPU card and aggregate
|
||||
them while they are all calculated.
|
||||
1. Split the gradient variable into multiple blocks according to the number of PServer
|
||||
instances and then send them.
|
||||
1. PServer would run an `Optimize Block` using a specified optimize algorithm to update
|
||||
the specified parameter.
|
||||
1. The trainer will fetch latest parameter from PServer before running forward Op which depends
|
||||
on the specified parameter.
|
||||
1. Broadcast the received variable into multiple GPU cards and continue to run the next
|
||||
mini-batch.
|
||||
|
||||
### Trainer
|
||||
|
||||
- For the multiple devices distributed training, we need to aggregate the gradient
|
||||
variables which placed on different devices firstly and then schedule a `SendVars` Operator to
|
||||
send the gradient variables to the multiple PServer instances.
|
||||
- Schedule `FetchVars` operator to fetch the latest parameter from PServer before running
|
||||
the forward ops.
|
||||
- There could be a large number of gradient variables to be sent, so we need to use another
|
||||
thread pool(IO Threadpool) whose a number of the schedulable threads is larger than the
|
||||
computing thread pool to avoid competitive the thread resources with computing.
|
||||
|
||||
### Parameter Server
|
||||
|
||||
<img src="./src/async_pserver.png" width="750"/>
|
||||
|
||||
- There should be multiple trainer instances want to optimize the same parameter at
|
||||
the same time, to avoid the racing, we need one `BlockingQueue` for each gradient
|
||||
variable to process them one by one.
|
||||
- We need a `Map` structure to map a gradient variable name to the `OptimizeBlock` which
|
||||
can optimize the respective parameter.
|
||||
|
After Width: | Height: | Size: 180 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 166 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 180 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 184 KiB |
@ -0,0 +1,53 @@
|
||||
if(NOT DEFINED SPHINX_THEME)
|
||||
set(SPHINX_THEME default)
|
||||
endif()
|
||||
|
||||
if(NOT DEFINED SPHINX_THEME_DIR)
|
||||
set(SPHINX_THEME_DIR)
|
||||
endif()
|
||||
|
||||
# configured documentation tools and intermediate build results
|
||||
set(BINARY_BUILD_DIR_EN "${CMAKE_CURRENT_BINARY_DIR}/en/_build")
|
||||
|
||||
# Sphinx cache with pickled ReST documents
|
||||
set(SPHINX_CACHE_DIR_EN "${CMAKE_CURRENT_BINARY_DIR}/en/_doctrees")
|
||||
|
||||
# HTML output director
|
||||
set(SPHINX_HTML_DIR_EN "${CMAKE_CURRENT_BINARY_DIR}/en/html")
|
||||
|
||||
configure_file(
|
||||
"${CMAKE_CURRENT_SOURCE_DIR}/../templates/conf.py.en.in"
|
||||
"${BINARY_BUILD_DIR_EN}/conf.py"
|
||||
@ONLY)
|
||||
|
||||
sphinx_add_target(paddle_mobile_docs
|
||||
html
|
||||
${BINARY_BUILD_DIR_EN}
|
||||
${SPHINX_CACHE_DIR_EN}
|
||||
${CMAKE_CURRENT_SOURCE_DIR}
|
||||
${SPHINX_HTML_DIR_EN})
|
||||
|
||||
add_dependencies(paddle_mobile_docs gen_proto_py paddle_python)
|
||||
|
||||
# configured documentation tools and intermediate build results
|
||||
set(BINARY_BUILD_DIR_CN "${CMAKE_CURRENT_BINARY_DIR}/cn/_build")
|
||||
|
||||
# Sphinx cache with pickled ReST documents
|
||||
set(SPHINX_CACHE_DIR_CN "${CMAKE_CURRENT_BINARY_DIR}/cn/_doctrees")
|
||||
|
||||
# HTML output director
|
||||
set(SPHINX_HTML_DIR_CN "${CMAKE_CURRENT_BINARY_DIR}/cn/html")
|
||||
|
||||
configure_file(
|
||||
"${CMAKE_CURRENT_SOURCE_DIR}/../templates/conf.py.cn.in"
|
||||
"${BINARY_BUILD_DIR_CN}/conf.py"
|
||||
@ONLY)
|
||||
|
||||
sphinx_add_target(paddle_mobile_docs_cn
|
||||
html
|
||||
${BINARY_BUILD_DIR_CN}
|
||||
${SPHINX_CACHE_DIR_CN}
|
||||
${CMAKE_CURRENT_SOURCE_DIR}
|
||||
${SPHINX_HTML_DIR_CN})
|
||||
|
||||
add_dependencies(paddle_mobile_docs_cn gen_proto_py paddle_python)
|
||||
@ -0,0 +1,9 @@
|
||||
移动端
|
||||
=====
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
cross_compiling_for_android_cn.md
|
||||
cross_compiling_for_ios_cn.md
|
||||
cross_compiling_for_raspberry_cn.md
|
||||
@ -0,0 +1,9 @@
|
||||
Mobile
|
||||
======
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
cross_compiling_for_android_en.md
|
||||
cross_compiling_for_ios_en.md
|
||||
cross_compiling_for_raspberry_en.md
|
||||
@ -0,0 +1 @@
|
||||
nv_test(test_tensorrt SRCS test_tensorrt.cc DEPS dynload_cuda device_context dynamic_loader)
|
||||
@ -0,0 +1,155 @@
|
||||
/* Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License. */
|
||||
|
||||
#include <glog/logging.h>
|
||||
#include <gtest/gtest.h>
|
||||
#include "NvInfer.h"
|
||||
#include "cuda.h"
|
||||
#include "cuda_runtime_api.h"
|
||||
#include "paddle/fluid/platform/dynload/tensorrt.h"
|
||||
|
||||
namespace dy = paddle::platform::dynload;
|
||||
|
||||
class Logger : public nvinfer1::ILogger {
|
||||
public:
|
||||
void log(nvinfer1::ILogger::Severity severity, const char* msg) override {
|
||||
switch (severity) {
|
||||
case Severity::kINFO:
|
||||
LOG(INFO) << msg;
|
||||
break;
|
||||
case Severity::kWARNING:
|
||||
LOG(WARNING) << msg;
|
||||
break;
|
||||
case Severity::kINTERNAL_ERROR:
|
||||
case Severity::kERROR:
|
||||
LOG(ERROR) << msg;
|
||||
break;
|
||||
default:
|
||||
break;
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
class ScopedWeights {
|
||||
public:
|
||||
ScopedWeights(float value) : value_(value) {
|
||||
w.type = nvinfer1::DataType::kFLOAT;
|
||||
w.values = &value_;
|
||||
w.count = 1;
|
||||
}
|
||||
const nvinfer1::Weights& get() { return w; }
|
||||
|
||||
private:
|
||||
float value_;
|
||||
nvinfer1::Weights w;
|
||||
};
|
||||
|
||||
// The following two API are implemented in TensorRT's header file, cannot load
|
||||
// from the dynamic library. So create our own implementation and directly
|
||||
// trigger the method from the dynamic library.
|
||||
nvinfer1::IBuilder* createInferBuilder(nvinfer1::ILogger& logger) {
|
||||
return static_cast<nvinfer1::IBuilder*>(
|
||||
dy::createInferBuilder_INTERNAL(&logger, NV_TENSORRT_VERSION));
|
||||
}
|
||||
nvinfer1::IRuntime* createInferRuntime(nvinfer1::ILogger& logger) {
|
||||
return static_cast<nvinfer1::IRuntime*>(
|
||||
dy::createInferRuntime_INTERNAL(&logger, NV_TENSORRT_VERSION));
|
||||
}
|
||||
|
||||
const char* kInputTensor = "input";
|
||||
const char* kOutputTensor = "output";
|
||||
|
||||
// Creates a network to compute y = 2x + 3
|
||||
nvinfer1::IHostMemory* CreateNetwork() {
|
||||
Logger logger;
|
||||
// Create the engine.
|
||||
nvinfer1::IBuilder* builder = createInferBuilder(logger);
|
||||
ScopedWeights weights(2.);
|
||||
ScopedWeights bias(3.);
|
||||
|
||||
nvinfer1::INetworkDefinition* network = builder->createNetwork();
|
||||
// Add the input
|
||||
auto input = network->addInput(kInputTensor, nvinfer1::DataType::kFLOAT,
|
||||
nvinfer1::DimsCHW{1, 1, 1});
|
||||
EXPECT_NE(input, nullptr);
|
||||
// Add the hidden layer.
|
||||
auto layer = network->addFullyConnected(*input, 1, weights.get(), bias.get());
|
||||
EXPECT_NE(layer, nullptr);
|
||||
// Mark the output.
|
||||
auto output = layer->getOutput(0);
|
||||
output->setName(kOutputTensor);
|
||||
network->markOutput(*output);
|
||||
// Build the engine.
|
||||
builder->setMaxBatchSize(1);
|
||||
builder->setMaxWorkspaceSize(1 << 10);
|
||||
auto engine = builder->buildCudaEngine(*network);
|
||||
EXPECT_NE(engine, nullptr);
|
||||
// Serialize the engine to create a model, then close.
|
||||
nvinfer1::IHostMemory* model = engine->serialize();
|
||||
network->destroy();
|
||||
engine->destroy();
|
||||
builder->destroy();
|
||||
return model;
|
||||
}
|
||||
|
||||
void Execute(nvinfer1::IExecutionContext& context, const float* input,
|
||||
float* output) {
|
||||
const nvinfer1::ICudaEngine& engine = context.getEngine();
|
||||
// Two binds, input and output
|
||||
ASSERT_EQ(engine.getNbBindings(), 2);
|
||||
const int input_index = engine.getBindingIndex(kInputTensor);
|
||||
const int output_index = engine.getBindingIndex(kOutputTensor);
|
||||
// Create GPU buffers and a stream
|
||||
void* buffers[2];
|
||||
ASSERT_EQ(0, cudaMalloc(&buffers[input_index], sizeof(float)));
|
||||
ASSERT_EQ(0, cudaMalloc(&buffers[output_index], sizeof(float)));
|
||||
cudaStream_t stream;
|
||||
ASSERT_EQ(0, cudaStreamCreate(&stream));
|
||||
// Copy the input to the GPU, execute the network, and copy the output back.
|
||||
ASSERT_EQ(0, cudaMemcpyAsync(buffers[input_index], input, sizeof(float),
|
||||
cudaMemcpyHostToDevice, stream));
|
||||
context.enqueue(1, buffers, stream, nullptr);
|
||||
ASSERT_EQ(0, cudaMemcpyAsync(output, buffers[output_index], sizeof(float),
|
||||
cudaMemcpyDeviceToHost, stream));
|
||||
cudaStreamSynchronize(stream);
|
||||
|
||||
// Release the stream and the buffers
|
||||
cudaStreamDestroy(stream);
|
||||
ASSERT_EQ(0, cudaFree(buffers[input_index]));
|
||||
ASSERT_EQ(0, cudaFree(buffers[output_index]));
|
||||
}
|
||||
|
||||
TEST(TensorrtTest, BasicFunction) {
|
||||
// Create the network serialized model.
|
||||
nvinfer1::IHostMemory* model = CreateNetwork();
|
||||
|
||||
// Use the model to create an engine and an execution context.
|
||||
Logger logger;
|
||||
nvinfer1::IRuntime* runtime = createInferRuntime(logger);
|
||||
nvinfer1::ICudaEngine* engine =
|
||||
runtime->deserializeCudaEngine(model->data(), model->size(), nullptr);
|
||||
model->destroy();
|
||||
nvinfer1::IExecutionContext* context = engine->createExecutionContext();
|
||||
|
||||
// Execute the network.
|
||||
float input = 1234;
|
||||
float output;
|
||||
Execute(*context, &input, &output);
|
||||
EXPECT_EQ(output, input * 2 + 3);
|
||||
|
||||
// Destroy the engine.
|
||||
context->destroy();
|
||||
engine->destroy();
|
||||
runtime->destroy();
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in new issue