superjom
8 years ago
commit
b818e64720
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,106 @@
|
||||
# Design Doc: Operation Graph Based Parameter Server
|
||||
|
||||
## Abstract
|
||||
|
||||
We propose an approach to implement the parameter server. In this
|
||||
approach, there is no fundamental difference between the trainer and
|
||||
the parameter server: they both run subgraphs, but subgraphs of
|
||||
different purposes.
|
||||
|
||||
## Background
|
||||
|
||||
The previous implementations of the parameter server does not run a
|
||||
subgraph. parameter initialization, optimizer computation, network
|
||||
communication and checkpointing are implemented twice on both the
|
||||
trainer and the parameter server.
|
||||
|
||||
It would be great if we can write code once and use them on both the
|
||||
trainer and the parameter server: reduces code duplication and
|
||||
improves extensibility. Given that after the current refactor, we are
|
||||
representing everything as a computing graph on the
|
||||
trainer. Representing everything as a computing graph on the parameter
|
||||
server becomes a natural extension.
|
||||
|
||||
## Design
|
||||
|
||||
### Graph Converter
|
||||
|
||||
The *graph converter* converts the user-defined operation (OP) graph
|
||||
into subgraphs to be scheduled on different nodes with the following
|
||||
steps:
|
||||
|
||||
1. OP placement: the OPs will be placed on different nodes according
|
||||
to heuristic that minimizes estimated total computation
|
||||
time. Currently we will use a simple heuristic that puts parameter
|
||||
varable on parameter server workers and everything else on trainer
|
||||
workers.
|
||||
|
||||
1. Add communication OPs to enable the communication between nodes.
|
||||
|
||||
We will need these OPs: *Send*, *Recv*, *Enqueue*, *Dequeue*.
|
||||
|
||||
Below is an example of converting the user defined graph to the
|
||||
subgraphs for the trainer and the parameter server:
|
||||
|
||||
<img src="src/local-graph.png" width="300"/>
|
||||
|
||||
After converting:
|
||||
|
||||
<img src="src/dist-graph.png" width="700"/>
|
||||
|
||||
1. The parameter variable W and it's optimizer subgraph are placed on the parameter server.
|
||||
1. Operators are added to the subgraphs.
|
||||
- *Send* sends data to the connected *Recv* operator. The
|
||||
scheduler on the receive node will only schedule *Recv* operator
|
||||
to run when the *Send* operator has ran (the *Send* OP will mark
|
||||
the *Recv* OP runnable automatically).
|
||||
- *Enueue* enqueues the input variable, it can block until space
|
||||
become available in the queue.
|
||||
- *Dequeue* outputs configurable numbers of tensors from the
|
||||
queue. It will block until the queue have the required number of
|
||||
tensors.
|
||||
|
||||
|
||||
### Benefits
|
||||
|
||||
- Model parallelism become easier to implement: it's an extension to
|
||||
the trainer - parameter server approach. we already have the
|
||||
communication OPs, but need to extend the graph converter's
|
||||
placement functionality.
|
||||
|
||||
- User-defined optimizer is easier to add - user can now express it as
|
||||
a subgraph.
|
||||
|
||||
- No more duplication logic inside the trainer and the parameter
|
||||
server mentioned in the background section.
|
||||
|
||||
### Challenges
|
||||
|
||||
- It might be hard for the graph converter to cut a general graph
|
||||
(without any hint for which subgraph is the optimizer). We may need
|
||||
to label which subgraph inside the OP graph is the optimizer.
|
||||
|
||||
- It's important to balance the parameter shards of on multiple
|
||||
parameter server. If a single parameter is very big (some
|
||||
word-embedding, fully connected, softmax layer), we need to
|
||||
automatically partition the single parameter onto different
|
||||
parameter servers when possible (only element-wise optimizer depends
|
||||
on the parameter variable).
|
||||
|
||||
### Discussion
|
||||
|
||||
- In the "Aync SGD" figure, the "W" variable on the parameter server
|
||||
could be read and wrote concurrently, what is our locking strategy?
|
||||
E.g., each variable have a lock cpp method to be invoked by every
|
||||
OP, or, have a lock OP.
|
||||
|
||||
- Can the Enqueue OP be implemented under our current tensor design
|
||||
(puts the input tensor into the queue tensor)?
|
||||
|
||||
- *Dequeue* OP will have variable numbers of output (depends on the
|
||||
`min_count` attribute), does our current design support it? (similar
|
||||
question for the *Add* OP)
|
||||
|
||||
|
||||
### References:
|
||||
[1] [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45166.pdf)
|
||||
@ -0,0 +1,56 @@
|
||||
digraph G {
|
||||
|
||||
rnn [label="1-th level RNN" shape=box]
|
||||
|
||||
subgraph cluster0 {
|
||||
label = "time step 0"
|
||||
|
||||
sent0 [label="sentence"]
|
||||
sent1 [label="sentence"]
|
||||
|
||||
rnn1 [label="2-th level RNN" shape=box]
|
||||
|
||||
sent0 -> rnn1
|
||||
sent1 -> rnn1
|
||||
}
|
||||
|
||||
subgraph cluster1 {
|
||||
label = "time step 1"
|
||||
|
||||
sent2 [label="sentence"]
|
||||
sent3 [label="sentence"]
|
||||
|
||||
rnn2 [label="2-th level RNN" shape=box]
|
||||
|
||||
sent2 -> rnn2
|
||||
sent3 -> rnn2
|
||||
}
|
||||
|
||||
subgraph cluster2 {
|
||||
label = "time step 2"
|
||||
|
||||
sent4 [label="sentence"]
|
||||
sent5 [label="sentence"]
|
||||
|
||||
rnn3 [label="2-th level RNN" shape=box]
|
||||
|
||||
sent4 -> rnn3
|
||||
sent5 -> rnn3
|
||||
}
|
||||
|
||||
|
||||
para0 [label="paragraph info 0"]
|
||||
para1 [label="paragraph info 1"]
|
||||
para2 [label="paragraph info 2"]
|
||||
|
||||
rnn1 -> para0
|
||||
rnn2 -> para1
|
||||
rnn3 -> para2
|
||||
|
||||
para0 -> rnn
|
||||
para1 -> rnn
|
||||
para2 -> rnn
|
||||
|
||||
chapter [label="chapter info"]
|
||||
rnn -> chapter
|
||||
}
|
||||
|
After Width: | Height: | Size: 51 KiB |
@ -0,0 +1,87 @@
|
||||
digraph G {
|
||||
label = "simple RNN implementation"
|
||||
|
||||
ranksep=2;
|
||||
|
||||
//graph [nodesep=1, ranksep=1];
|
||||
|
||||
node[nodesep=1]
|
||||
|
||||
subgraph cluster0 {
|
||||
label = "global scope"
|
||||
rankdir = TB
|
||||
W
|
||||
boot_memory
|
||||
input
|
||||
output
|
||||
}
|
||||
|
||||
subgraph cluster1 {
|
||||
label = "step-scope 0"
|
||||
rankdir = TB
|
||||
memory0[label="memory"]
|
||||
prememory0[label="pre-memory"]
|
||||
step_input0[label="step input"]
|
||||
step_output0[label="step output"]
|
||||
}
|
||||
|
||||
subgraph cluster2 {
|
||||
label = "step-scope 1"
|
||||
rankdir = TB
|
||||
memory1[label="memory"]
|
||||
prememory1[label="pre-memory"]
|
||||
step_input1[label="step input"]
|
||||
step_output1[label="step output"]
|
||||
}
|
||||
|
||||
subgraph cluster3 {
|
||||
label = "step-scope 2"
|
||||
rankdir = TB
|
||||
memory2[label="memory"]
|
||||
prememory2[label="pre-memory"]
|
||||
step_input2[label="step input"]
|
||||
step_output2[label="step output"]
|
||||
}
|
||||
|
||||
stepnet [shape=box]
|
||||
stepnet0 [shape=box, style=dashed]
|
||||
stepnet1 [shape=box, style=dashed]
|
||||
stepnet2 [shape=box, style=dashed]
|
||||
|
||||
|
||||
edge[color=blue]
|
||||
boot_memory -> prememory0 [label="init" color="blue"]
|
||||
memory0 -> prememory1 [label="copy/reference" color="blue"]
|
||||
memory1 -> prememory2 [label="copy/reference" color="blue"]
|
||||
|
||||
edge[color=black]
|
||||
W -> stepnet0[constraint=false, style=dashed]
|
||||
W -> stepnet1[constraint=false, style=dashed]
|
||||
W -> stepnet2[constraint=false, style=dashed]
|
||||
|
||||
memory0 -> stepnet0[style=dashed]
|
||||
prememory0 -> stepnet0 -> step_output0[style=dashed]
|
||||
|
||||
memory1 -> stepnet1[style=dashed]
|
||||
prememory1 -> stepnet1 -> step_output1[style=dashed]
|
||||
|
||||
memory2 -> stepnet2[style=dashed]
|
||||
prememory2 -> stepnet2 -> step_output2[style=dashed]
|
||||
|
||||
input -> step_input0
|
||||
input -> step_input1
|
||||
input -> step_input2
|
||||
|
||||
step_input0 -> stepnet0 [style=dashed]
|
||||
step_input1 -> stepnet1[style=dashed]
|
||||
step_input2 -> stepnet2[style=dashed]
|
||||
|
||||
step_output0 -> output
|
||||
step_output1 -> output
|
||||
step_output2 -> output
|
||||
|
||||
stepnet0 -> stepnet[style=dashed]
|
||||
stepnet1 -> stepnet[style=dashed]
|
||||
stepnet2 -> stepnet[style=dashed]
|
||||
|
||||
}
|
||||
|
After Width: | Height: | Size: 43 KiB |
|
After Width: | Height: | Size: 181 KiB |
@ -0,0 +1,75 @@
|
||||
digraph G {
|
||||
chapter [label="chapter"]
|
||||
|
||||
subgraph cluster0 {
|
||||
label = "paragraph 0"
|
||||
|
||||
top_rnn0[label="top rnn step 0" shape=box]
|
||||
|
||||
p0 [label="paragraph 0"]
|
||||
p1 [label="paragraph 1"]
|
||||
}
|
||||
|
||||
subgraph cluster1{
|
||||
label = "paragraph 1"
|
||||
|
||||
top_rnn1[label="top rnn step 1" shape=box]
|
||||
|
||||
p2 [label="paragraph 0"]
|
||||
p3 [label="paragraph 1"]
|
||||
}
|
||||
|
||||
subgraph cluster_p0 {
|
||||
label = "sentence 0"
|
||||
|
||||
low_rnn0 [label="low rnn step 0" shape=box]
|
||||
s00 [label="sentence 0"]
|
||||
s01 [label="sentence 1"]
|
||||
|
||||
low_rnn0 -> s00
|
||||
low_rnn0 -> s01
|
||||
}
|
||||
|
||||
subgraph cluster_p1 {
|
||||
label = "sentence 1"
|
||||
low_rnn1 [label="low rnn step 1" shape=box]
|
||||
s10 [label="sentence 0"]
|
||||
s11 [label="sentence 1"]
|
||||
low_rnn1 -> s10
|
||||
low_rnn1 -> s11
|
||||
}
|
||||
|
||||
subgraph cluster_p2 {
|
||||
label = "sentence 1"
|
||||
low_rnn2 [label="low rnn step 0" shape=box]
|
||||
s20 [label="sentence 0"]
|
||||
s21 [label="sentence 1"]
|
||||
low_rnn2 -> s20
|
||||
low_rnn2 -> s21
|
||||
}
|
||||
|
||||
subgraph cluster_p3 {
|
||||
label = "sentence 1"
|
||||
low_rnn3 [label="low rnn step 1" shape=box]
|
||||
s30 [label="sentence 0"]
|
||||
s31 [label="sentence 1"]
|
||||
low_rnn3 -> s30
|
||||
low_rnn3 -> s31
|
||||
}
|
||||
|
||||
|
||||
chapter -> top_rnn0
|
||||
chapter -> top_rnn1
|
||||
|
||||
top_rnn0 -> p0
|
||||
top_rnn0 -> p1
|
||||
top_rnn1 -> p2
|
||||
top_rnn1 -> p3
|
||||
|
||||
|
||||
p0 -> low_rnn0
|
||||
p1 -> low_rnn1
|
||||
p2 -> low_rnn2
|
||||
p3 -> low_rnn3
|
||||
|
||||
}
|
||||
|
After Width: | Height: | Size: 67 KiB |
@ -0,0 +1,153 @@
|
||||
# RNNOp design
|
||||
|
||||
This document is about an RNN operator which requires that instances in a mini-batch have the same length. We will have a more flexible RNN operator.
|
||||
|
||||
## RNN Algorithm Implementation
|
||||
|
||||
<p aligh="center">
|
||||
<img src="./images/rnn.jpg"/>
|
||||
</p>
|
||||
|
||||
The above diagram shows an RNN unrolled into a full network.
|
||||
|
||||
There are several important concepts:
|
||||
|
||||
- *step-net*: the sub-graph to run at each step,
|
||||
- *memory*, $h_t$, the state of the current step,
|
||||
- *ex-memory*, $h_{t-1}$, the state of the previous step,
|
||||
- *initial memory value*, the ex-memory of the first step.
|
||||
|
||||
### Step-scope
|
||||
|
||||
There could be local variables defined in step-nets. PaddlePaddle runtime realizes these variables in *step-scopes* -- scopes created for each step.
|
||||
|
||||
<p aligh="center">
|
||||
<img src="./images/rnn.png"/><br/>
|
||||
Figure 2 the RNN's data flow
|
||||
</p>
|
||||
|
||||
Please be aware that all steps run the same step-net. Each step
|
||||
|
||||
1. creates the step-scope,
|
||||
2. realizes local variables, including step-outputs, in the step-scope, and
|
||||
3. runs the step-net, which could use these variables.
|
||||
|
||||
The RNN operator will compose its output from step outputs in step scopes.
|
||||
|
||||
### Memory and Ex-memory
|
||||
|
||||
Let's give more details about memory and ex-memory via a simply example:
|
||||
|
||||
$$
|
||||
h_t = U h_{t-1} + W x_t
|
||||
$$,
|
||||
|
||||
where $h_t$ and $h_{t-1}$ are the memory and ex-memory of step $t$'s respectively.
|
||||
|
||||
In the implementation, we can make an ex-memory variable either "refers to" the memory variable of the previous step,
|
||||
or copy the value of the previous memory value to the current ex-memory variable.
|
||||
|
||||
### Usage in Python
|
||||
|
||||
For more information on Block, please refer to the [design doc](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/block.md).
|
||||
|
||||
We can define an RNN's step-net using Block:
|
||||
|
||||
```python
|
||||
import paddle as pd
|
||||
|
||||
X = some_op() # x is some operator's output, and is a LoDTensor
|
||||
a = some_op()
|
||||
|
||||
# declare parameters
|
||||
W = pd.Variable(shape=[20, 30])
|
||||
U = pd.Variable(shape=[20, 30])
|
||||
|
||||
rnn = pd.create_rnn_op(output_num=1)
|
||||
with rnn.stepnet():
|
||||
x = rnn.add_input(X)
|
||||
# declare a memory (rnn's step)
|
||||
h = rnn.add_memory(init=a)
|
||||
# h.pre_state() means previous memory of rnn
|
||||
new_state = pd.add_two( pd.matmul(W, x) + pd.matmul(U, h.pre_state()))
|
||||
# update current memory
|
||||
h.update(new_state)
|
||||
# indicate that h variables in all step scopes should be merged
|
||||
rnn.add_outputs(h)
|
||||
|
||||
out = rnn()
|
||||
```
|
||||
|
||||
Python API functions in above example:
|
||||
|
||||
- `rnn.add_input` indicates the parameter is a variable that will be segmented into step-inputs.
|
||||
- `rnn.add_memory` creates a variable used as the memory.
|
||||
- `rnn.add_outputs` mark the variables that will be concatenated across steps into the RNN output.
|
||||
|
||||
### Nested RNN and LoDTensor
|
||||
|
||||
An RNN whose step-net includes other RNN operators is known as an *nested RNN*.
|
||||
|
||||
For example, we could have a 2-level RNN, where the top level corresponds to paragraphs, and the lower level corresponds to sentences.
|
||||
|
||||
The following figure illustrates the feeding of text into the lower level, one sentence each step, and the feeding of step outputs to the top level. The final top level output is about the whole text.
|
||||
|
||||
<p aligh="center">
|
||||
<img src="./images/2_level_rnn.png"/>
|
||||
</p>
|
||||
|
||||
```python
|
||||
import paddle as pd
|
||||
|
||||
W = pd.Variable(shape=[20, 30])
|
||||
U = pd.Variable(shape=[20, 30])
|
||||
|
||||
W0 = pd.Variable(shape=[20, 30])
|
||||
U0 = pd.Variable(shape=[20, 30])
|
||||
|
||||
# a is output of some op
|
||||
a = some_op()
|
||||
|
||||
# chapter_data is a set of 128-dim word vectors
|
||||
# the first level of LoD is sentence
|
||||
# the second level of LoD is chapter
|
||||
chapter_data = pd.Variable(shape=[None, 128], type=pd.lod_tensor, level=2)
|
||||
|
||||
def lower_level_rnn(paragraph):
|
||||
'''
|
||||
x: the input
|
||||
'''
|
||||
rnn = pd.create_rnn_op(output_num=1)
|
||||
with rnn.stepnet():
|
||||
sentence = rnn.add_input(paragraph, level=0)
|
||||
h = rnn.add_memory(shape=[20, 30])
|
||||
h.update(
|
||||
pd.matmul(W, sentence) + pd.matmul(U, h.pre_state()))
|
||||
# get the last state as sentence's info
|
||||
rnn.add_outputs(h)
|
||||
return rnn
|
||||
|
||||
top_level_rnn = pd.create_rnn_op(output_num=1)
|
||||
with top_level_rnn.stepnet():
|
||||
paragraph_data = rnn.add_input(chapter_data, level=1)
|
||||
low_rnn = lower_level_rnn(paragraph_data)
|
||||
paragraph_out = low_rnn()
|
||||
|
||||
h = rnn.add_memory(init=a)
|
||||
h.update(
|
||||
pd.matmul(W0, paragraph_data) + pd.matmul(U0, h.pre_state()))
|
||||
top_level_rnn.add_outputs(h)
|
||||
|
||||
# just output the last step
|
||||
chapter_out = top_level_rnn(output_all_steps=False)
|
||||
```
|
||||
|
||||
in above example, the construction of the `top_level_rnn` calls `lower_level_rnn`. The input is a LoD Tensor. The top level RNN segments input text data into paragraphs, and the lower level RNN segments each paragraph into sentences.
|
||||
|
||||
By default, the `RNNOp` will concatenate the outputs from all the time steps,
|
||||
if the `output_all_steps` set to False, it will only output the final time step.
|
||||
|
||||
|
||||
<p align="center">
|
||||
<img src="images/rnn_2level_data.png"/>
|
||||
</p>
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 222 KiB |
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in new issue