m53297601
/
Paddle
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

fix by comments

Browse Source
refactor_docs
gongweibao 8 years ago
parent 327727d462
commit 60f69c2b0e
8 changed files with 88 additions and 120 deletions
Show Stats Download Patch File Download Diff File

6
doc/design/file_manager/README.md
Unescape View File

@ -3,7 +3,7 @@
在本文档中，我们设计说明了名为FileManager系统，方便用户管理存放到PaddlePaddle Cloud上的文件。
主要功能包括：
- 提供常用的命令行文件管理命令管理文件
- 提供常用的命令行管理命令管理文件和目录
- 支持的命令在[Here](./pfs/pfs.md)
- 支持大文件的断点上传、下载
@ -27,7 +27,7 @@
- 用Golang写，可以跨平台执行
- 双向验证
PFSClient需要和Ingress之间做双向验证<sup>[tls](#tls)</sup>，所有用户需要首先在`cloud.paddlepaddle.org`上注册一下，申请用户空间，并且把系统生成的Key、CRT、CA下载到本地，然后才能使用PFSClient。
PFSClient需要和Ingress之间做双向验证<sup>[tls](#tls)</sup>，所以用户需要首先在`cloud.paddlepaddle.org`上注册一下，申请用户空间，并且把系统生成的Key、CRT、CA下载到本地，然后才能使用PFSClient。
### Ingress
- 功能：
@ -43,7 +43,7 @@ FileServer是一个用GoRPC写的HTTPServer，提供[RESTful API](./RESTAPI.md)
## 文件传输优化
### 分块文件传输
用户文件可能是比较大的，上传到Cloud或者下载到本地的时间可能比较长，而且在传输的过程中也可能出现网络不稳定的情况。为了应对以上的问题，我们提出了Chunk的概念，一个Chunk由所在的文件偏移、数据、数据长度及校验值组成。文件数据内容的上传和下载都是都过Chunk的操作来实现的。由于Chunk比较小（默认256K），完成一个传输动作完成的时间也比较短，不容易出错。PFSClient在传输完毕最后一个Chunk的时候检查desttination文件的MD5值是否和source文件一致。
用户文件可能是比较大的，上传到Cloud或者下载到本地的时间可能比较长，而且在传输的过程中也可能出现网络不稳定的情况。为了应对以上的问题，我们提出了Chunk的概念，一个Chunk由所在的文件偏移、数据、数据长度及校验值组成。文件数据内容的上传和下载都是都过Chunk的操作来实现的。由于Chunk比较小（默认256K），完成一个传输动作完成的时间也比较短，不容易出错。PFSClient在传输完毕最后一个Chunk的时候检查destination文件的MD5值是否和source文件一致。
一个典型的Chunk如下所示：

46
doc/design/file_manager/pfs/cp.md
Unescape View File

@ -1,41 +1,45 @@
# Name
cp - copy files and directories
cp - copy files
# Synopsis
` cp [OPTION]...
<LocalPath> <PFSPath> or <PFSPath> <LocalPath> or <PFSPath> <PFSPath>`
```
cp [-r] [-f | -n] [-v] [--preserve--links] <LocalPath> <PFSPath>
cp [-r] [-f | -n] [-v] [--preserve--links] <LocalPath> ... <PFSPath>
cp [-r] [-f | -n] [-v] [--preserve--links] <PFSPath> <LocalPath>
cp [-r] [-f | -n] [-v] [--preserve--links] <PFSPath> ... <LocalPath>
cp [-r] [-f | -n] [-v] [--preserve--links] <PFSPath> <PFSPath>
cp [-r] [-f | -n] [-v] [--preserve--links] <PFSPath> ... <PFSPath>
```
# Description
```
The following options are available:
-r
Copy directories recursively
-f
Do not prompt for confirmation before overwriting the destination path. (The -f option overrides previous -n options.)
-n
Do not overwrite an existing file. (The -n option overrides previous -f options.)
-v
Cause cp to be verbose, showing files after they are copied.
```
--preserve--links
Reserve links when copy links
-R, -r, --recursive
Copy directories recursively
```
# Examples
- The following command copies a single file to pfs
```
paddle pfs cp ./text1.txt /pfs/mydir/text1.txt
```
Output
```
upload ./text1.txt to /pfs/mydir/text1.txt
paddle pfs cp ./text1.txt /pfs/$DATACENTER/home/$USER/text1.txt
```
- The following command copies pfs file to a local file
```
paddle pfs cp /pfs/mydir/text1.txt ./text1.txt
```
Output
```
download /pfs/mydir/text1.txt to ./text1.txt
paddle pfs cp /pfs/$DATACENTER/home/$USER/text1.txt ./text1.txt
```

33
doc/design/file_manager/pfs/ls.md
Unescape View File

@ -1,42 +1,27 @@
# Name
ls - list directory contents or a file attributes
ls - list directory(ies)'s contents or file(s)'s attributes
# Synopsis
` ls [OPTION]... <PFSPath>`
`ls [-r] <PFSPath> ...`
# Description
```
-R, -r, --recursive
Copy directories recursively
--page-size (integer)
The number of results to return in each response to a list operation. The default value is 1000 (the maximum allowed). Using a lower value may help if operation time out.
```
The following options are available:
-r
List directory(ies) recursively
```
# Examples
- The following command lists a single file
```
paddle pfs ls /pfs/mydir/text1.txt
```
Output
```
2017-05-5 17:04:30 text1.txt
paddle pfs ls /pfs/$DATACENTER/home/$USER/text1.txt
```
- The following command lists directory contents
```
paddle pfs ls /pfs/mydir
```
Output
```
2017-05-5 17:04:30 text1.txt
2017-05-5 17:04:30 text2.txt
...
paddle pfs ls / /pfs/$DATACENTER/home/$USER/folder
```

12
doc/design/file_manager/pfs/mkdir.md
Unescape View File

@ -2,18 +2,12 @@
mkdir - mkdir directory(ies)
# Synopsis
`mkdir [OPTION]... <PFSPath> ...`
`mkdir <PFSPath> ...`
# Description
Create the pfs directory(ies), if they do not already exist. And create intermediate directories as required
Create the pfs directory(ies), if it(they) does(do) not already exist. And create intermediate directories as required.
# Examples
```
paddle pfs mkdir /pfs/mydir1
```
Output
```
make directory /pfs/mydir1
paddle pfs mkdir /pfs/$DATACENTER/home/$USER/folder
```

30
doc/design/file_manager/pfs/mv.md
Unescape View File

@ -1,28 +1,34 @@
# Name
mv - move (rename) files or directories
mv - move (rename) files
# Synopsis
If destination already exist, please [rm](rm.md) it first.
```
mv [OPTION]...
<LocalPath> <PFSPath> or <PFSPath> <LocalPath> or <PFSPath> <PFSPath>
mv [-f | -n] [-v] <LocalPath> <PFSPath>
mv [-f | -n] [-v] <LocalPath> ... <PFSPath>
mv [-f | -n] [-v] <PFSPath> <LocalPath>
mv [-f | -n] [-v] <PFSPath> ... <LocalPath>
mv [-f | -n] [-v] <PFSPath> <PFSPath>
mv [-f | -n] [-v] <PFSPath> ... <PFSPath>
```
# Description
```
```
The following options are available:
# Examples
- The following command move a single file to pfs
-f
Do not prompt for confirmation before overwriting the destination path. (The -f option overrides previous -n options.)
```
paddle pfs mv ./text1.txt /pfs/mydirectory/test1.txt
-n
Do not overwrite an existing file. (The -n option overrides previous -f options.)
-v
Cause mv to be verbose, showing files after they are moved.
```
Output
# Examples
- The following command moves a single file to pfs
```
move ./text1.txt /pfs/mydirectory/test1.txt
paddle pfs mv ./text1.txt /pfs/$DATACENTER/home/$USER/text1.txt
```

6
doc/design/file_manager/pfs/pfs.md
Unescape View File

@ -1,7 +1,7 @@
# PFS Client
## Description
The `pfs` command is a Command Line Interface to manage your files on Paddle Cloud
The `pfs` command is a Command Line Interface to manage your files on PaddlePaddle Cloud
## Synopsis
```
@ -28,10 +28,10 @@ paddle [options] pfs <subcommand> [parameters]
## Path Arguments
When using a command, we need to specify path arguments. There are two path argument type: `localpath` and `pfspath`.
A `pfspath` begin with `/pfs`, eg: `/pfs/mydir/text1.txt`.
A `pfspath` begin with `/pfs`, eg: `/pfs/$DATACENTER/home/$USER/folder`.
## order of Path Arguments
Commonly, if there are two path arguments. The first is the source, and the second is the destination.
Commonly, if there are two path arguments, the first is the source, and the second is the destination.
## Subcommonds
- [rm](rm.md)

35
doc/design/file_manager/pfs/rm.md
Unescape View File

@ -3,43 +3,30 @@ rm - remove files or directories
# Synopsis
```
rm [OPTION]... <PFSPath>...
rm [-r] [-v] <PFSPath> ...
```
# Description
```
-r, -R, --recursive
remove directories and their contents recursively
The following options are available:
--page-size (integer)
The number of results to return in each response to a list operation. The default value is 1000 (the maximum allowed). Using a lower value may help if an operation times out
-r
remove directories and their contents recursively
-v
Cause rm to be verbose, showing files after they are removed.
```
# Examples
- The following command deletes a single files
```
paddle pfs rm /pfs/mydirectory/test1.txt
```
Output
```
delete /pfs/mydirectory/test1.txt
```
- The following command deletes a directory recursively.
- The following command deletes a single file:
```
paddle pfs rm -r /pfs/mydirectory
paddle pfs rm /pfs/$DATACENTER/home/$USER/test1.txt
```
Output
- The following command deletes a directory recursively:
```
delete /pfs/mydirectory/1.txt
delete /pfs/mydirectory/2.txt
...
paddle pfs rm -r /pfs/$DATACENTER/home/$USER/folder
```

40
doc/design/file_manager/pfs/sync.md
Unescape View File

@ -1,42 +1,34 @@
# Name
sync - sync directories. Recursively copies new and updated files from the source directory to the destination
sync - sync directories. Recursively copies new and updated files from the source directory to the destination.
# Synopsis
` sync [OPTIONS]...
<LocalPath> <PFSPath> or <PFSPath> <LocalPath> or <PFSPath> <PFSPath>`
```
sync [--preserve--links] [-v] <LocalPath> <PFSPath>
sync [--preserve--links] [-v] <PFSPath> <LocalPath>
sync [--preserve--links] [-v] <PFSPath> <PFSPath>`
```
# Description
```
The following options are available:
--preserve--links
Reserve links when copy links
Reserve links when copy links.
-v
Cause sync to be verbose, showing files after their's synchronization is complete.
```
# Examples
- The following command sync locally directory to pfs
```
paddle pfs sync ./dir1 /pfs/mydir1
```
Output
```
upload ./dir1/text1.txt to /pfs/mydir1/text2.txt
upload ./dir1/text2.txt to /pfs/mydir1/text2.txt
...
```
- The following command sync pfs directory to local
- The following command sync locally directory to pfs.
```
paddle pfs sync /pfs/mydir1 .
paddle pfs sync ./dir1 /pfs/$DATACENTER/home/$USER/mydir1
```
Output
- The following command sync pfs directory to local.
```
download /pfs/mydir1/text1.txt to ./text1.txt
download /pfs/mydir1/text2.txt to ./text2.txt
...
paddle pfs sync /pfs/$DATACENTER/home/$USER/mydir1 .
```

Write Preview
Loading…
Cancel
Save