Xinghai Sun
8 years ago
commit
0532662902
|
Before Width: | Height: | Size: 54 KiB After Width: | Height: | Size: 58 KiB |
|
Before Width: | Height: | Size: 46 KiB After Width: | Height: | Size: 50 KiB |
|
Before Width: | Height: | Size: 28 KiB After Width: | Height: | Size: 32 KiB |
@ -0,0 +1,106 @@
|
||||
# Design Doc: Operation Graph Based Parameter Server
|
||||
|
||||
## Abstract
|
||||
|
||||
We propose an approach to implement the parameter server. In this
|
||||
approach, there is no fundamental difference between the trainer and
|
||||
the parameter server: they both run subgraphs, but subgraphs of
|
||||
different purposes.
|
||||
|
||||
## Background
|
||||
|
||||
The previous implementations of the parameter server does not run a
|
||||
subgraph. parameter initialization, optimizer computation, network
|
||||
communication and checkpointing are implemented twice on both the
|
||||
trainer and the parameter server.
|
||||
|
||||
It would be great if we can write code once and use them on both the
|
||||
trainer and the parameter server: reduces code duplication and
|
||||
improves extensibility. Given that after the current refactor, we are
|
||||
representing everything as a computing graph on the
|
||||
trainer. Representing everything as a computing graph on the parameter
|
||||
server becomes a natural extension.
|
||||
|
||||
## Design
|
||||
|
||||
### Graph Converter
|
||||
|
||||
The *graph converter* converts the user-defined operation (OP) graph
|
||||
into subgraphs to be scheduled on different nodes with the following
|
||||
steps:
|
||||
|
||||
1. OP placement: the OPs will be placed on different nodes according
|
||||
to heuristic that minimizes estimated total computation
|
||||
time. Currently we will use a simple heuristic that puts parameter
|
||||
varable on parameter server workers and everything else on trainer
|
||||
workers.
|
||||
|
||||
1. Add communication OPs to enable the communication between nodes.
|
||||
|
||||
We will need these OPs: *Send*, *Recv*, *Enqueue*, *Dequeue*.
|
||||
|
||||
Below is an example of converting the user defined graph to the
|
||||
subgraphs for the trainer and the parameter server:
|
||||
|
||||
<img src="src/local-graph.png" width="300"/>
|
||||
|
||||
After converting:
|
||||
|
||||
<img src="src/dist-graph.png" width="700"/>
|
||||
|
||||
1. The parameter variable W and it's optimizer subgraph are placed on the parameter server.
|
||||
1. Operators are added to the subgraphs.
|
||||
- *Send* sends data to the connected *Recv* operator. The
|
||||
scheduler on the receive node will only schedule *Recv* operator
|
||||
to run when the *Send* operator has ran (the *Send* OP will mark
|
||||
the *Recv* OP runnable automatically).
|
||||
- *Enueue* enqueues the input variable, it can block until space
|
||||
become available in the queue.
|
||||
- *Dequeue* outputs configurable numbers of tensors from the
|
||||
queue. It will block until the queue have the required number of
|
||||
tensors.
|
||||
|
||||
|
||||
### Benefits
|
||||
|
||||
- Model parallelism become easier to implement: it's an extension to
|
||||
the trainer - parameter server approach. we already have the
|
||||
communication OPs, but need to extend the graph converter's
|
||||
placement functionality.
|
||||
|
||||
- User-defined optimizer is easier to add - user can now express it as
|
||||
a subgraph.
|
||||
|
||||
- No more duplication logic inside the trainer and the parameter
|
||||
server mentioned in the background section.
|
||||
|
||||
### Challenges
|
||||
|
||||
- It might be hard for the graph converter to cut a general graph
|
||||
(without any hint for which subgraph is the optimizer). We may need
|
||||
to label which subgraph inside the OP graph is the optimizer.
|
||||
|
||||
- It's important to balance the parameter shards of on multiple
|
||||
parameter server. If a single parameter is very big (some
|
||||
word-embedding, fully connected, softmax layer), we need to
|
||||
automatically partition the single parameter onto different
|
||||
parameter servers when possible (only element-wise optimizer depends
|
||||
on the parameter variable).
|
||||
|
||||
### Discussion
|
||||
|
||||
- In the "Aync SGD" figure, the "W" variable on the parameter server
|
||||
could be read and wrote concurrently, what is our locking strategy?
|
||||
E.g., each variable have a lock cpp method to be invoked by every
|
||||
OP, or, have a lock OP.
|
||||
|
||||
- Can the Enqueue OP be implemented under our current tensor design
|
||||
(puts the input tensor into the queue tensor)?
|
||||
|
||||
- *Dequeue* OP will have variable numbers of output (depends on the
|
||||
`min_count` attribute), does our current design support it? (similar
|
||||
question for the *Add* OP)
|
||||
|
||||
|
||||
### References:
|
||||
[1] [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45166.pdf)
|
||||
@ -0,0 +1,56 @@
|
||||
digraph G {
|
||||
|
||||
rnn [label="1-th level RNN" shape=box]
|
||||
|
||||
subgraph cluster0 {
|
||||
label = "time step 0"
|
||||
|
||||
sent0 [label="sentence"]
|
||||
sent1 [label="sentence"]
|
||||
|
||||
rnn1 [label="2-th level RNN" shape=box]
|
||||
|
||||
sent0 -> rnn1
|
||||
sent1 -> rnn1
|
||||
}
|
||||
|
||||
subgraph cluster1 {
|
||||
label = "time step 1"
|
||||
|
||||
sent2 [label="sentence"]
|
||||
sent3 [label="sentence"]
|
||||
|
||||
rnn2 [label="2-th level RNN" shape=box]
|
||||
|
||||
sent2 -> rnn2
|
||||
sent3 -> rnn2
|
||||
}
|
||||
|
||||
subgraph cluster2 {
|
||||
label = "time step 2"
|
||||
|
||||
sent4 [label="sentence"]
|
||||
sent5 [label="sentence"]
|
||||
|
||||
rnn3 [label="2-th level RNN" shape=box]
|
||||
|
||||
sent4 -> rnn3
|
||||
sent5 -> rnn3
|
||||
}
|
||||
|
||||
|
||||
para0 [label="paragraph info 0"]
|
||||
para1 [label="paragraph info 1"]
|
||||
para2 [label="paragraph info 2"]
|
||||
|
||||
rnn1 -> para0
|
||||
rnn2 -> para1
|
||||
rnn3 -> para2
|
||||
|
||||
para0 -> rnn
|
||||
para1 -> rnn
|
||||
para2 -> rnn
|
||||
|
||||
chapter [label="chapter info"]
|
||||
rnn -> chapter
|
||||
}
|
||||
|
After Width: | Height: | Size: 51 KiB |
@ -0,0 +1,87 @@
|
||||
digraph G {
|
||||
label = "simple RNN implementation"
|
||||
|
||||
ranksep=2;
|
||||
|
||||
//graph [nodesep=1, ranksep=1];
|
||||
|
||||
node[nodesep=1]
|
||||
|
||||
subgraph cluster0 {
|
||||
label = "global scope"
|
||||
rankdir = TB
|
||||
W
|
||||
boot_memory
|
||||
input
|
||||
output
|
||||
}
|
||||
|
||||
subgraph cluster1 {
|
||||
label = "step-scope 0"
|
||||
rankdir = TB
|
||||
memory0[label="memory"]
|
||||
prememory0[label="pre-memory"]
|
||||
step_input0[label="step input"]
|
||||
step_output0[label="step output"]
|
||||
}
|
||||
|
||||
subgraph cluster2 {
|
||||
label = "step-scope 1"
|
||||
rankdir = TB
|
||||
memory1[label="memory"]
|
||||
prememory1[label="pre-memory"]
|
||||
step_input1[label="step input"]
|
||||
step_output1[label="step output"]
|
||||
}
|
||||
|
||||
subgraph cluster3 {
|
||||
label = "step-scope 2"
|
||||
rankdir = TB
|
||||
memory2[label="memory"]
|
||||
prememory2[label="pre-memory"]
|
||||
step_input2[label="step input"]
|
||||
step_output2[label="step output"]
|
||||
}
|
||||
|
||||
stepnet [shape=box]
|
||||
stepnet0 [shape=box, style=dashed]
|
||||
stepnet1 [shape=box, style=dashed]
|
||||
stepnet2 [shape=box, style=dashed]
|
||||
|
||||
|
||||
edge[color=blue]
|
||||
boot_memory -> prememory0 [label="init" color="blue"]
|
||||
memory0 -> prememory1 [label="copy/reference" color="blue"]
|
||||
memory1 -> prememory2 [label="copy/reference" color="blue"]
|
||||
|
||||
edge[color=black]
|
||||
W -> stepnet0[constraint=false, style=dashed]
|
||||
W -> stepnet1[constraint=false, style=dashed]
|
||||
W -> stepnet2[constraint=false, style=dashed]
|
||||
|
||||
memory0 -> stepnet0[style=dashed]
|
||||
prememory0 -> stepnet0 -> step_output0[style=dashed]
|
||||
|
||||
memory1 -> stepnet1[style=dashed]
|
||||
prememory1 -> stepnet1 -> step_output1[style=dashed]
|
||||
|
||||
memory2 -> stepnet2[style=dashed]
|
||||
prememory2 -> stepnet2 -> step_output2[style=dashed]
|
||||
|
||||
input -> step_input0
|
||||
input -> step_input1
|
||||
input -> step_input2
|
||||
|
||||
step_input0 -> stepnet0 [style=dashed]
|
||||
step_input1 -> stepnet1[style=dashed]
|
||||
step_input2 -> stepnet2[style=dashed]
|
||||
|
||||
step_output0 -> output
|
||||
step_output1 -> output
|
||||
step_output2 -> output
|
||||
|
||||
stepnet0 -> stepnet[style=dashed]
|
||||
stepnet1 -> stepnet[style=dashed]
|
||||
stepnet2 -> stepnet[style=dashed]
|
||||
|
||||
}
|
||||
|
After Width: | Height: | Size: 43 KiB |
|
After Width: | Height: | Size: 181 KiB |
@ -0,0 +1,75 @@
|
||||
digraph G {
|
||||
chapter [label="chapter"]
|
||||
|
||||
subgraph cluster0 {
|
||||
label = "paragraph 0"
|
||||
|
||||
top_rnn0[label="top rnn step 0" shape=box]
|
||||
|
||||
p0 [label="paragraph 0"]
|
||||
p1 [label="paragraph 1"]
|
||||
}
|
||||
|
||||
subgraph cluster1{
|
||||
label = "paragraph 1"
|
||||
|
||||
top_rnn1[label="top rnn step 1" shape=box]
|
||||
|
||||
p2 [label="paragraph 0"]
|
||||
p3 [label="paragraph 1"]
|
||||
}
|
||||
|
||||
subgraph cluster_p0 {
|
||||
label = "sentence 0"
|
||||
|
||||
low_rnn0 [label="low rnn step 0" shape=box]
|
||||
s00 [label="sentence 0"]
|
||||
s01 [label="sentence 1"]
|
||||
|
||||
low_rnn0 -> s00
|
||||
low_rnn0 -> s01
|
||||
}
|
||||
|
||||
subgraph cluster_p1 {
|
||||
label = "sentence 1"
|
||||
low_rnn1 [label="low rnn step 1" shape=box]
|
||||
s10 [label="sentence 0"]
|
||||
s11 [label="sentence 1"]
|
||||
low_rnn1 -> s10
|
||||
low_rnn1 -> s11
|
||||
}
|
||||
|
||||
subgraph cluster_p2 {
|
||||
label = "sentence 1"
|
||||
low_rnn2 [label="low rnn step 0" shape=box]
|
||||
s20 [label="sentence 0"]
|
||||
s21 [label="sentence 1"]
|
||||
low_rnn2 -> s20
|
||||
low_rnn2 -> s21
|
||||
}
|
||||
|
||||
subgraph cluster_p3 {
|
||||
label = "sentence 1"
|
||||
low_rnn3 [label="low rnn step 1" shape=box]
|
||||
s30 [label="sentence 0"]
|
||||
s31 [label="sentence 1"]
|
||||
low_rnn3 -> s30
|
||||
low_rnn3 -> s31
|
||||
}
|
||||
|
||||
|
||||
chapter -> top_rnn0
|
||||
chapter -> top_rnn1
|
||||
|
||||
top_rnn0 -> p0
|
||||
top_rnn0 -> p1
|
||||
top_rnn1 -> p2
|
||||
top_rnn1 -> p3
|
||||
|
||||
|
||||
p0 -> low_rnn0
|
||||
p1 -> low_rnn1
|
||||
p2 -> low_rnn2
|
||||
p3 -> low_rnn3
|
||||
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in new issue