m53297601
/
Paddle
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

modify commands

Browse Source
refactor_docs
gongweibao 8 years ago
parent ee454f9380
commit 177d35afa1
9 changed files with 66 additions and 113 deletions
Show Stats Download Patch File Download Diff File

4
doc/design/file_mananger/README.md
Unescape View File

@ -2,7 +2,7 @@
## Objetive
在本文档中，我们设计说明了用户上传、下载、管理自己在PaddlePaddle Cloud上的文件所涉及到的模块和流程
<image src=./src/filemanager.png width=600>
<image src=./src/filemanager.png width=8900>
## Module
### PFS Client
@ -25,7 +25,7 @@
- 在kubernets中运行
- [RESTAPI](./RESTAPI.md)接口
## 流程
## 解释
### 为什么有chunk的抽象：
用户文件可能是比较大的，上传到Cloud或者下载到本地的时间可能比较长，而且在传输的过程中也可能出现网络不稳定的情况。为了应对以上的问题，我们提出了chunk的概念，一个chunk由所在的文件偏移、数据、数据长度及校验值组成。文件数据内容的上传和下载都是都过chunk的操作来实现的。由于chunk比较小（默认256K），完成一个传输动作的transaction的时间也比较短，不容易出错。

2
doc/design/file_mananger/RESTAPI.md
Unescape View File

@ -1,4 +1,4 @@
# REST API
# REST API Interface
- file
```

18
doc/design/file_mananger/pfs/cp.md
Unescape View File

@ -1,19 +1,13 @@
# Name
pfs_cp - copy files and directories
cp - copy files and directories
# Synopsis
` pfs_cp
` cp [OPTION]...
<LocalPath> <PFSPath> or <PFSPath> <LocalPath> or <PFSPath> <PFSPath>`
# Description
```
--only-show-errors (boolean)
Only errors and warnings are displayed. All other output is suppressed.
--page-size (integer)
The number of results to return in each response to a list operation. The default value is 1000 (the maximum allowed). Using a lower value may help if an operation times out.
--preserve--links
Reserve links when copy files
@ -22,10 +16,10 @@ pfs_cp - copy files and directories
```
# Examples
- The following command cp a single file to pfs
- The following command copies a single file to pfs
```
pfs_cp ./text1.txt pfs://mydir/text1.txt
paddle pfs cp ./text1.txt pfs://mydir/text1.txt
```
Output
@ -34,10 +28,10 @@ Output
upload ./text1.txt to pfs://mydir/text1.txt
```
- The following command cp pfs file to a local file
- The following command copies pfs file to a local file
```
pfs_cp pfs://mydir/text1.txt ./text1.txt
paddle pfs cp pfs://mydir/text1.txt ./text1.txt
```
Output

33
doc/design/file_mananger/pfs/ls.md
Unescape View File

@ -1,49 +1,42 @@
# Name
pfs_ls - list directory contents o
ls - list directory contents or a file attributes
# Synopsis
` pfs_ls [OPTION]... <PFSPath>`
` ls [OPTION]... <PFSPath>`
# Description
```
-h, --help
Display this help and exit
--version
Output version information and exit
--only-show-errors (boolean)
Only errors and warnings are displayed. All other output is suppressed.
--page-size (integer)
The number of results to return in each response to a list operation. The default value is 1000 (the maximum allowed). Using a lower value may help if an operation times out.
-R, -r, --recursive
Copy directories recursively
--page-size (integer)
The number of results to return in each response to a list operation. The default value is 1000 (the maximum allowed). Using a lower value may help if operation time out.
```
# Examples
- The following command cp a single file to pfs
- The following command lists a single file
```
pfs_cp ./text1.txt pfs://mydir/text1.txt
paddle pfs ls pfs://mydir/text1.txt
```
Output
```
upload ./text1.txt to pfs://mydir/text1.txt
2017-05-5 17:04:30 text1.txt
```
- The following command cp pfs file to a local file
- The following command lists directory contents
```
pfs_cp pfs://mydir/text1.txt ./text1.txt
paddle pfs ls pfs://mydir
```
Output
```
download pfs://mydir/text1.txt to ./text1.txt
2017-05-5 17:04:30 text1.txt
2017-05-5 17:04:30 text2.txt
...
```

22
doc/design/file_mananger/pfs/mkdir.md
Unescape View File

@ -1,29 +1,15 @@
# Name
pdf_mkdir - mkdir directory(ies)
mkdir - mkdir directory(ies)
# Synopsis
`pfs_mkdir [OPTION]... PFS_DIRECTORY...`
`mkdir [OPTION]... <PFSPath> ...`
# Description
Create the pfs directory(ies), if they do not already exist.
```
-h, --help
display this help and exit
--version
output version information and exit
--only-show-errors (boolean)
Only errors and warnings are displayed. All other output is suppressed.
--page-size (integer)
The number of results to return in each response to a list operation. The default value is 1000 (the maximum allowed). Using a lower value may help if an operation times out.
```
Create the pfs directory(ies), if they do not already exist. And create intermediate directories as required
# Examples
```
pfs_mkdir pfs://mydir1
paddle pfs mkdir pfs://mydir1
```
Output

18
doc/design/file_mananger/pfs/mv.md
Unescape View File

@ -3,7 +3,8 @@ mv - move (rename) files or directories
# Synopsis
If destination already exist, please rm it first.
If destination already exist, please [rm](rm.md) it first.
```
mv [OPTION]...
<LocalPath> <PFSPath> or <PFSPath> <LocalPath> or <PFSPath> <PFSPath>
@ -11,28 +12,17 @@ mv [OPTION]...
# Description
```
-h, --help
display this help and exit
-v, --version
output version information and exit
--only-show-errors (boolean)
Only errors and warnings are displayed. All other output is suppressed.
--page-size (integer)
The number of results to return in each response to a list operation. The default value is 1000 (the maximum allowed). Using a lower value may help if an operation times out.
```
# Examples
- The following command move a single file to pfs
```
mv ./text1.txt pfs://mydirectory/test1.txt
paddle pfs mv ./text1.txt pfs://mydirectory/test1.txt
```
Output
```
move text1.txt pfs://mydirectory/test1.txt
move ./text1.txt pfs://mydirectory/test1.txt
```

18
doc/design/file_mananger/pfs/pfs.md
Unescape View File

@ -12,20 +12,26 @@ paddle [options] pfs <subcommand> [parameters]
```
--profile (string)
Use a specific profile from your credential file.
--help pfs
Display more information about pfs
--help (string)
Display more information about command
--version
Output version information and exit
--debug
Show detailed debugging log
--only-show-errors (boolean)
Only errors and warnings are displayed. All other output is suppressed.
```
## Path Arguments
When using a commnd, we need to specify path arguments. There are two path argument type: `localpath` and `pfspath`.
A pfspath begin with `pfs://`, eg: `pfs://mydir/text1.txt`.
When using a command, we need to specify path arguments. There are two path argument type: `localpath` and `pfspath`.
A `pfspath` begin with `pfs://`, eg: `pfs://mydir/text1.txt`.
## Path Arguments‘s order
Commonly, there maybe two path arguments. The first is source, and the second is destination.
## order of Path Arguments
Commonly, if there are two path arguments. The first is the source, and the second is the destination.
## Subcommonds
- [rm](rm.md)

21
doc/design/file_mananger/pfs/rm.md
Unescape View File

@ -2,30 +2,25 @@
rm - remove files or directories
# Synopsis
`rm [OPTION]... PFS_DIRECTORY...`
```
rm [OPTION]... <PFSPath>...
```
# Description
Create the directory, if it does not already exits
```
-r, -R, --recursive
remove directories and their contents recursively
-v, --version
output version information and exit
--only-show-errors (boolean)
Only errors and warnings are displayed. All other output is suppressed.
--page-size (integer)
The number of results to return in each response to a list operation. The default value is 1000 (the maximum allowed). Using a lower value may help if an operation times out.
The number of results to return in each response to a list operation. The default value is 1000 (the maximum allowed). Using a lower value may help if an operation times out
```
# Examples
- The following command delete a single file
- The following command deletes a single files
```
pfs_rm pfs://mydirectory/test1.txt
paddle pfs rm pfs://mydirectory/test1.txt
```
Output
@ -35,10 +30,10 @@ delete pfs://mydirectory/test1.txt
```
- The following command delete a directory recursively.
- The following command deletes a directory recursively.
```
pfs_rm -r pfs://mydirectory
paddle pfs rm -r pfs://mydirectory
```
Output

39
doc/design/file_mananger/pfs/sync.md
Unescape View File

@ -1,53 +1,42 @@
# Name
pfs_cp - copy files and directories
sync - sync directories. Recursively copies new and updated files from the source directory to the destination
# Synopsis
` pfs_cp
` sync [OPTIONS]...
<LocalPath> <PFSPath> or <PFSPath> <LocalPath> or <PFSPath> <PFSPath>`
# Description
```
-h, --help
Display this help and exit
--version
Output version information and exit
--only-show-errors (boolean)
Only errors and warnings are displayed. All other output is suppressed.
--page-size (integer)
The number of results to return in each response to a list operation. The default value is 1000 (the maximum allowed). Using a lower value may help if an operation times out.
--preserve--links
Reserve links when copy files
-R, -r, --recursive
Copy directories recursively
-l, --links
copy symlinks as symlinks
```
# Examples
- The following command cp a single file to pfs
- The following command sync locally directory to pfs
```
pfs_cp ./text1.txt pfs://mydir/text1.txt
sync ./dir1 pfs://mydir1
```
Output
```
upload ./text1.txt to pfs://mydir/text1.txt
upload ./dir1/text1.txt to pfs://mydir1/text2.txt
upload ./dir1/text2.txt to pfs://mydir1/text2.txt
...
```
- The following command cp pfs file to a local file
- The following command sync pfs directory to local
```
pfs_cp pfs://mydir/text1.txt ./text1.txt
sync pfs://mydir1 .
```
Output
```
download pfs://mydir/text1.txt to ./text1.txt
download pfs://mydir1/text1.txt to ./text1.txt
download pfs://mydir1/text2.txt to ./text2.txt
...
```

Write Preview
Loading…
Cancel
Save