tensor-tang
8 years ago
commit
59ccb01a00
|
Before Width: | Height: | Size: 58 KiB After Width: | Height: | Size: 56 KiB |
|
Before Width: | Height: | Size: 50 KiB After Width: | Height: | Size: 49 KiB |
|
Before Width: | Height: | Size: 32 KiB After Width: | Height: | Size: 30 KiB |
@ -0,0 +1,105 @@
|
||||
## Optimizer Design
|
||||
|
||||
### The Problem
|
||||
|
||||
A PaddlePaddle program, or a block, is a sequence of operators operating variables. A training program needs to do three kinds of works:
|
||||
|
||||
1. the forward pass, which computes intermediate results and the cost(s),
|
||||
1. the backward pass, which derives gradients from intermediate results and costs, and
|
||||
1. the optimization pass, which update model parameters to optimize the cost(s).
|
||||
|
||||
These works rely on three kinds of operators:
|
||||
|
||||
1. forward operators,
|
||||
1. gradient operators, and
|
||||
1. optimization operators.
|
||||
|
||||
It's true that users should be able to create all these operators manually by calling some low-level API, but it would be much more convenient if they could only describe the forward pass and let PaddlePaddle create the backward and optimization operators automatically.
|
||||
|
||||
In this design, we propose a high-level API that automatically derives the optimisation pass and operators from the forward pass.
|
||||
|
||||
|
||||
### High-level Python API to describe the training process
|
||||
|
||||
1. User write code to describe the network:
|
||||
|
||||
```python
|
||||
images = layer.data("images")
|
||||
labels = layer.data("labels")
|
||||
w1 = pd.var("w1")
|
||||
b1 = pd.var("b1")
|
||||
hidden = layer.fc(images, w=w1, b=b1)
|
||||
cost = layer.mse(hidden, labels)
|
||||
```
|
||||
|
||||
The above code snippet will create forward operators in [Block](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/block.md).
|
||||
|
||||
|
||||
2. Users create a certain kind of Optimizer with some argument.
|
||||
|
||||
```python
|
||||
optimizer = AdagradOptimizer(learing_rate=0.001)
|
||||
```
|
||||
|
||||
3. Users use the optimizer to `minimize` a certain `cost` through updating parameters in parameter_list.
|
||||
|
||||
```python
|
||||
opt_op_list = optimizer.minimize(cost, parameter_list=[w1, b1])
|
||||
```
|
||||
The above code snippet will create gradient and optimization operators in Block. The return value of `minimize()` is list of optimization operators that will be run by session.
|
||||
|
||||
4. Users use Session/Executor to run this opt_op_list as target to do training.
|
||||
|
||||
```python
|
||||
sess.run(target= opt_op_list, ...)
|
||||
```
|
||||
|
||||
#### Optimizer Python interface:
|
||||
|
||||
```python
|
||||
class Optimizer(object):
|
||||
"""Optimizer Base class.
|
||||
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
pass
|
||||
|
||||
def create_backward_pass(self, loss, parameter_list=None):
|
||||
"""
|
||||
create and add gradient Operators in BlockDesc to Compute gradients of `loss`
|
||||
for parameters in parameter_list
|
||||
|
||||
Args:
|
||||
loss: an variable generated by cost function.
|
||||
parameter_list: parameters that need to compute gradient and update to optimize the lost.
|
||||
|
||||
Returns:
|
||||
list of (parameters, gradients) pair.
|
||||
"""
|
||||
return None
|
||||
|
||||
def create_optimization_pass(self, parameters_and_grads):
|
||||
"""Add optimization operators to update gradients to variables.
|
||||
|
||||
Args:
|
||||
parameters_and_grads: a list of (variable, gradient) pair to update.
|
||||
|
||||
Returns:
|
||||
optmization_op_list: a list of optimization operator that will update parameter using gradient.
|
||||
"""
|
||||
return None
|
||||
|
||||
def minimize(self, loss, parameter_list):
|
||||
"""Add operations to minimize `loss` by updating `parameter_list`.
|
||||
|
||||
This method combines interface `create_backward_pass()` and
|
||||
`create_optimization_pass()` into one.
|
||||
"""
|
||||
params_grads = self.create_backward_pass(loss, parameter_list)
|
||||
update_ops = self.create_optimization_pass(params_grads)
|
||||
return update_ops
|
||||
|
||||
```
|
||||
|
||||
Users can inherit the Optimizer above to create their own Optimizer with some special logic, such as AdagradOptimizer.
|
||||
@ -0,0 +1,74 @@
|
||||
# Design Doc: Selected Rows
|
||||
|
||||
`SelectedRows` is a kind of sparse tensor data type, which is designed to support `embedding` operators. The gradient of embedding table is a sparse tensor. Only a few rows are non-zero values in that tensor. It is straightforward to represent the sparse tensor by the following sparse tensor data structure:
|
||||
|
||||
```cpp
|
||||
class SelectedRows {
|
||||
private:
|
||||
vector<int> rows_;
|
||||
Tensor value_;
|
||||
int height_;
|
||||
};
|
||||
```
|
||||
|
||||
The field `height_` shows the first dimension of `SelectedRows`. The `rows` are the indices of which rows of `SelectedRows` are non-zeros. The `value_` field is an N-dim tensor and shape is `[rows.size() /* NUM_ROWS */, ...]`, which supplies values for each row. The dimension of `SelectedRows` satisfies `[height_] + value_.shape[1:]`.
|
||||
|
||||
Suppose that a SelectedRows-typed variable `x` has many rows, but only two of them have values -- row 73 is `[1, 2]` and row 84 is `[3, 4]`, the `SelectedRows` representation would be:
|
||||
|
||||
```
|
||||
x = SelectedRow {
|
||||
rows = [73, 84],
|
||||
value = [[1, 2], [3,4]]
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## SelectedRows in Protobuf
|
||||
|
||||
`SelectedRows` is a kind of `Variable`. `VarDesc` in protobuf should describe the `SelectedRows` information. Only the tensor dimension of a `SelectedRows` will be described in compile-time since the `rows_` and `value_` are related to training data.
|
||||
So we use `TensorDesc` to unify `data_type` and `dims`. A LodTensorDesc contains a `TensorDesc` and `lod_level`. The description of `SelectedRows` is a Tensor description.
|
||||
|
||||
```proto
|
||||
message TensorDesc {
|
||||
required DataType data_type = 1;
|
||||
repeated int64 dims = 2; // [UNK, 640, 480] is saved as [-1, 640, 480]
|
||||
}
|
||||
|
||||
message LodTensorDesc {
|
||||
required TensorDesc tensor = 1;
|
||||
optional int lod_level = 2;
|
||||
}
|
||||
|
||||
message VarDesc {
|
||||
required string name = 1;
|
||||
enum VarType {
|
||||
LOD_TENSOR = 0;
|
||||
SELECTED_ROWS = 1;
|
||||
}
|
||||
required VarType type = 2;
|
||||
optional LodTensorDesc lod_desc = 3;
|
||||
optional TensorDesc selected_rows_desc = 4;
|
||||
optional bool persistable = 5 [ default = false ];
|
||||
}
|
||||
```
|
||||
|
||||
## InferShape for Selected Rows
|
||||
|
||||
Just like `LoD` information, `InferShape` method will inference output tensor type as well. The operator should decide whether its output is a `SelectedRows` or `Dense` tensor.
|
||||
|
||||
For example, the gradient operator of `TableLookup` will always generate `SelectedRows`. Its `InferShape` method should be like following
|
||||
|
||||
```cpp
|
||||
void TableLookupGrad::InferShape(context) {
|
||||
...
|
||||
context.SetDataType("Embedding.Grad", kSelectedRows);
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## Sparse Operators
|
||||
|
||||
There are several operators should be written to support `SelectedRows`. They are:
|
||||
|
||||
1. Operators which generates `SelectedRows` gradient. e.g. Gradient of `TableLookupOp`.
|
||||
2. Optimize operators which support `SelectedRows` gradient. e.g. `SGD` or `AdaGrad` for `SelectedRows`. However, there should be only one `SGD` operator. `OpWithKernel::Run` should select a suitable kernel for both `dense` tensor or `SelectedRows`.
|
||||
@ -0,0 +1,163 @@
|
||||
/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License. */
|
||||
|
||||
#include "paddle/framework/executor.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <iostream>
|
||||
#include <memory>
|
||||
#include <set>
|
||||
#include <vector>
|
||||
|
||||
#include "paddle/framework/lod_tensor.h"
|
||||
#include "paddle/framework/op_registry.h"
|
||||
#include "paddle/framework/scope.h"
|
||||
|
||||
namespace paddle {
|
||||
namespace framework {
|
||||
|
||||
const std::string kFeedOpType = "feed";
|
||||
const std::string kFetchOpType = "fetch";
|
||||
|
||||
Executor::Executor(const std::vector<platform::Place>& places) {
|
||||
PADDLE_ENFORCE_GT(places.size(), 0);
|
||||
device_contexts_.resize(places.size());
|
||||
for (size_t i = 0; i < places.size(); i++) {
|
||||
if (platform::is_cpu_place(places[i])) {
|
||||
device_contexts_[i] = new platform::CPUDeviceContext(
|
||||
boost::get<platform::CPUPlace>(places[i]));
|
||||
} else if (platform::is_gpu_place(places[i])) {
|
||||
#ifdef PADDLE_WITH_CUDA
|
||||
device_contexts_[i] = new platform::CUDADeviceContext(
|
||||
boost::get<platform::GPUPlace>(places[i]));
|
||||
#else
|
||||
PADDLE_THROW(
|
||||
"'GPUPlace' is not supported, Please re-compile with WITH_GPU "
|
||||
"option");
|
||||
#endif
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Executor::~Executor() {
|
||||
for (auto& device_context : device_contexts_) {
|
||||
delete device_context;
|
||||
}
|
||||
}
|
||||
|
||||
void Executor::Run(const ProgramDesc& pdesc, Scope* scope, int block_id) {
|
||||
// TODO(tonyyang-svail):
|
||||
// - only runs on the first device (i.e. no interdevice communication)
|
||||
// - will change to use multiple blocks for RNN op and Cond Op
|
||||
PADDLE_ENFORCE_GT(pdesc.blocks_size(), block_id);
|
||||
auto& block = pdesc.blocks(block_id);
|
||||
auto& device = device_contexts_[0];
|
||||
|
||||
// Instantiate all the vars in the global scope
|
||||
for (auto& var : block.vars()) {
|
||||
scope->NewVar(var.name());
|
||||
}
|
||||
|
||||
Scope& local_scope = scope->NewScope();
|
||||
|
||||
std::vector<bool> should_run = Prune(pdesc, block_id);
|
||||
PADDLE_ENFORCE_EQ(should_run.size(), static_cast<size_t>(block.ops_size()));
|
||||
for (size_t i = 0; i < should_run.size(); ++i) {
|
||||
if (should_run[i]) {
|
||||
for (auto& var : block.ops(i).outputs()) {
|
||||
for (auto& argu : var.arguments()) {
|
||||
if (local_scope.FindVar(argu) == nullptr) {
|
||||
local_scope.NewVar(argu);
|
||||
}
|
||||
}
|
||||
}
|
||||
auto op = paddle::framework::OpRegistry::CreateOp(block.ops(i));
|
||||
op->Run(local_scope, *device);
|
||||
}
|
||||
}
|
||||
|
||||
// TODO(tonyyang-svail):
|
||||
// - Destroy local_scope
|
||||
}
|
||||
|
||||
std::vector<bool> Prune(const ProgramDesc& pdesc, int block_id) {
|
||||
// TODO(tonyyang-svail):
|
||||
// - will change to use multiple blocks for RNN op and Cond Op
|
||||
|
||||
auto& block = pdesc.blocks(block_id);
|
||||
auto& ops = block.ops();
|
||||
|
||||
bool expect_feed = true;
|
||||
for (auto& op_desc : ops) {
|
||||
PADDLE_ENFORCE(op_desc.type() != kFeedOpType || expect_feed,
|
||||
"All FeedOps are at the beginning of the ProgramDesc");
|
||||
expect_feed = (op_desc.type() == kFeedOpType);
|
||||
}
|
||||
|
||||
bool expect_fetch = true;
|
||||
for (auto op_iter = ops.rbegin(); op_iter != ops.rend(); ++op_iter) {
|
||||
auto& op_desc = *op_iter;
|
||||
PADDLE_ENFORCE(op_desc.type() != kFetchOpType || expect_fetch,
|
||||
"All FetchOps must at the end of the ProgramDesc");
|
||||
expect_fetch = (op_desc.type() == kFetchOpType);
|
||||
}
|
||||
|
||||
std::set<std::string> dependent_vars;
|
||||
std::vector<bool> should_run;
|
||||
for (auto op_iter = ops.rbegin(); op_iter != ops.rend(); ++op_iter) {
|
||||
auto& op_desc = *op_iter;
|
||||
|
||||
bool found_dependent_vars = false;
|
||||
for (auto& var : op_desc.outputs()) {
|
||||
for (auto& argu : var.arguments()) {
|
||||
if (dependent_vars.count(argu) != 0) {
|
||||
found_dependent_vars = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (op_desc.type() == kFetchOpType || found_dependent_vars) {
|
||||
// erase its output to the dependency graph
|
||||
for (auto& var : op_desc.outputs()) {
|
||||
for (auto& argu : var.arguments()) {
|
||||
dependent_vars.erase(argu);
|
||||
}
|
||||
}
|
||||
|
||||
// insert its input to the dependency graph
|
||||
for (auto& var : op_desc.inputs()) {
|
||||
for (auto& argu : var.arguments()) {
|
||||
dependent_vars.insert(argu);
|
||||
}
|
||||
}
|
||||
|
||||
should_run.push_back(true);
|
||||
} else {
|
||||
should_run.push_back(false);
|
||||
}
|
||||
}
|
||||
|
||||
// TODO(tonyyang-svail):
|
||||
// - check this after integration of Init
|
||||
// PADDLE_ENFORCE(dependent_vars.empty());
|
||||
|
||||
// since we are traversing the ProgramDesc in reverse order
|
||||
// we reverse the should_run vector
|
||||
std::reverse(should_run.begin(), should_run.end());
|
||||
|
||||
return should_run;
|
||||
}
|
||||
|
||||
} // namespace framework
|
||||
} // namespace paddle
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in new issue