Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into HEAD
Yu Yang
7 years ago
commit
37792e546b
@ -1,124 +0,0 @@
|
||||
# Build using Docker
|
||||
|
||||
## What Developers Need
|
||||
|
||||
To contribute to PaddlePaddle, you need
|
||||
|
||||
1. A computer -- Linux, BSD, Windows, MacOS, and
|
||||
1. Docker.
|
||||
|
||||
Nothing else. Not even Python and GCC, because you can install all build tools into a Docker image. We run all the tools by running this image.
|
||||
|
||||
## General Process
|
||||
|
||||
1. Retrieve source code.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/paddlepaddle/paddle
|
||||
```
|
||||
|
||||
2. Install build tools into a Docker image.
|
||||
|
||||
```bash
|
||||
cd paddle; docker build -t paddle:dev .
|
||||
```
|
||||
|
||||
Please be aware of the `.` at the end of the command, which refers to the [`./Dockerfile` file](https://github.com/PaddlePaddle/Paddle/blob/develop/Dockerfile). `docker build` follows instructions in this file to create a Docker image named `paddle:dev`, and installs building tools into it.
|
||||
|
||||
3. Build from source.
|
||||
|
||||
This following command starts a Docker container that executes the Docker image `paddle:dev`, mapping the current directory to `/paddle/` in the container, and runs the default entry-point [`build.sh`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/scripts/docker/build.sh) as specified in the Dockefile. `build.sh` invokes `cmake` and `make` to build PaddlePaddle source code, which had been mapped to `/paddle`, and writes outputs to `/paddle/build`, which maps to `build` in the current source directory on the computer.
|
||||
|
||||
```bash
|
||||
docker run -v $PWD:/paddle paddle:dev
|
||||
```
|
||||
|
||||
Above command builds a CUDA-enabled version. If we want to build a CPU-only version, we can type
|
||||
|
||||
```bash
|
||||
docker run -e WITH_GPU=OFF -v $PWD:/paddle paddle:dev
|
||||
```
|
||||
|
||||
4. Run unit tests.
|
||||
|
||||
To run all unit tests using the first GPU of a node:
|
||||
|
||||
```bash
|
||||
NV_GPU=0 nvidia-docker run -v $PWD:/paddle paddle:dev bash -c "cd /paddle/build; ctest"
|
||||
```
|
||||
|
||||
If we used `WITH_GPU=OFF` at build time, it generates only CPU-based unit tests, and we don't need nvidia-docker to run them. We can just run
|
||||
|
||||
```bash
|
||||
docker run -v $PWD:/paddle paddle:dev bash -c "cd /paddle/build; ctest"
|
||||
```
|
||||
|
||||
Sometimes we want to run a specific unit test, say `memory_test`, we can run
|
||||
|
||||
```bash
|
||||
nvidia-docker run -v $PWD:/paddle paddle:dev bash -c "cd /paddle/build; ctest -V -R memory_test"
|
||||
```
|
||||
|
||||
5. Clean Build.
|
||||
|
||||
Sometimes, we might want to clean all thirt-party dependents and built binaries. To do so, just
|
||||
|
||||
```bash
|
||||
rm -rf build
|
||||
```
|
||||
|
||||
## Docker, Or Not?
|
||||
|
||||
- What is Docker?
|
||||
|
||||
If you haven't heard of it, consider it something like Python's virtualenv.
|
||||
|
||||
- Docker or virtual machine?
|
||||
|
||||
Some people compare Docker with VMs, but Docker doesn't virtualize any hardware nor running a guest OS, which means there is no compromise on the performance.
|
||||
|
||||
- Why Docker?
|
||||
|
||||
Using a Docker image of build tools standardizes the building environment, which makes it easier for others to reproduce your problems and to help.
|
||||
|
||||
Also, some build tools don't run on Windows or Mac or BSD, but Docker runs almost everywhere, so developers can use whatever computer they want.
|
||||
|
||||
- Can I choose not to use Docker?
|
||||
|
||||
Sure, you don't have to install build tools into a Docker image; instead, you can install them in your local computer. This document exists because Docker would make the development way easier.
|
||||
|
||||
- How difficult is it to learn Docker?
|
||||
|
||||
It takes you ten minutes to read [an introductory article](https://docs.docker.com/get-started) and saves you more than one hour to install all required build tools, configure them, especially when new versions of PaddlePaddle require some new tools. Not even to mention the time saved when other people trying to reproduce the issue you have.
|
||||
|
||||
- Can I use my favorite IDE?
|
||||
|
||||
Yes, of course. The source code resides on your local computer, and you can edit it using whatever editor you like.
|
||||
|
||||
Many PaddlePaddle developers are using Emacs. They add the following few lines into their `~/.emacs` configure file:
|
||||
|
||||
```emacs
|
||||
(global-set-key "\C-cc" 'compile)
|
||||
(setq compile-command
|
||||
"docker run --rm -it -v $(git rev-parse --show-toplevel):/paddle paddle:dev")
|
||||
```
|
||||
|
||||
so they could type `Ctrl-C` and `c` to build PaddlePaddle from source.
|
||||
|
||||
- Does Docker do parallel building?
|
||||
|
||||
Our building Docker image runs a [Bash script](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/scripts/docker/build.sh), which calls `make -j$(nproc)` to starts as many processes as the number of your CPU cores.
|
||||
|
||||
## Some Gotchas
|
||||
|
||||
- Docker requires sudo
|
||||
|
||||
An owner of a computer has the administrative privilege, a.k.a., sudo, and Docker requires this privilege to work properly. If you use a shared computer for development, please ask the administrator to install and configure Docker. We will do our best to support rkt, another container technology that doesn't require sudo.
|
||||
|
||||
- Docker on Windows/MacOS builds slowly
|
||||
|
||||
On Windows and MacOS, Docker containers run in a Linux VM. You might want to give this VM some more memory and CPUs so to make the building efficient. Please refer to [this issue](https://github.com/PaddlePaddle/Paddle/issues/627) for details.
|
||||
|
||||
- Not enough disk space
|
||||
|
||||
Examples in this article uses option `--rm` with the `docker run` command. This option ensures that stopped containers do not exist on hard disks. We can use `docker ps -a` to list all containers, including stopped. Sometimes `docker build` generates some intermediate dangling images, which also take disk space. To clean them, please refer to [this article](https://zaiste.net/posts/removing_docker_containers/).
|
||||
@ -0,0 +1,184 @@
|
||||
/* Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License. */
|
||||
|
||||
#include "paddle/fluid/operators/detection_map_op.h"
|
||||
|
||||
namespace paddle {
|
||||
namespace operators {
|
||||
|
||||
using Tensor = framework::Tensor;
|
||||
|
||||
class DetectionMAPOp : public framework::OperatorWithKernel {
|
||||
public:
|
||||
using framework::OperatorWithKernel::OperatorWithKernel;
|
||||
|
||||
void InferShape(framework::InferShapeContext* ctx) const override {
|
||||
PADDLE_ENFORCE(ctx->HasInput("DetectRes"),
|
||||
"Input(DetectRes) of DetectionMAPOp should not be null.");
|
||||
PADDLE_ENFORCE(ctx->HasInput("Label"),
|
||||
"Input(Label) of DetectionMAPOp should not be null.");
|
||||
PADDLE_ENFORCE(
|
||||
ctx->HasOutput("AccumPosCount"),
|
||||
"Output(AccumPosCount) of DetectionMAPOp should not be null.");
|
||||
PADDLE_ENFORCE(
|
||||
ctx->HasOutput("AccumTruePos"),
|
||||
"Output(AccumTruePos) of DetectionMAPOp should not be null.");
|
||||
PADDLE_ENFORCE(
|
||||
ctx->HasOutput("AccumFalsePos"),
|
||||
"Output(AccumFalsePos) of DetectionMAPOp should not be null.");
|
||||
PADDLE_ENFORCE(ctx->HasOutput("MAP"),
|
||||
"Output(MAP) of DetectionMAPOp should not be null.");
|
||||
|
||||
auto det_dims = ctx->GetInputDim("DetectRes");
|
||||
PADDLE_ENFORCE_EQ(det_dims.size(), 2UL,
|
||||
"The rank of Input(DetectRes) must be 2, "
|
||||
"the shape is [N, 6].");
|
||||
PADDLE_ENFORCE_EQ(det_dims[1], 6UL,
|
||||
"The shape is of Input(DetectRes) [N, 6].");
|
||||
auto label_dims = ctx->GetInputDim("Label");
|
||||
PADDLE_ENFORCE_EQ(label_dims.size(), 2UL,
|
||||
"The rank of Input(Label) must be 2, "
|
||||
"the shape is [N, 6].");
|
||||
PADDLE_ENFORCE_EQ(label_dims[1], 6UL,
|
||||
"The shape is of Input(Label) [N, 6].");
|
||||
|
||||
if (ctx->HasInput("PosCount")) {
|
||||
PADDLE_ENFORCE(ctx->HasInput("TruePos"),
|
||||
"Input(TruePos) of DetectionMAPOp should not be null when "
|
||||
"Input(TruePos) is not null.");
|
||||
PADDLE_ENFORCE(
|
||||
ctx->HasInput("FalsePos"),
|
||||
"Input(FalsePos) of DetectionMAPOp should not be null when "
|
||||
"Input(FalsePos) is not null.");
|
||||
}
|
||||
|
||||
ctx->SetOutputDim("MAP", framework::make_ddim({1}));
|
||||
}
|
||||
|
||||
protected:
|
||||
framework::OpKernelType GetExpectedKernelType(
|
||||
const framework::ExecutionContext& ctx) const override {
|
||||
return framework::OpKernelType(
|
||||
framework::ToDataType(
|
||||
ctx.Input<framework::Tensor>("DetectRes")->type()),
|
||||
ctx.device_context());
|
||||
}
|
||||
};
|
||||
|
||||
class DetectionMAPOpMaker : public framework::OpProtoAndCheckerMaker {
|
||||
public:
|
||||
DetectionMAPOpMaker(OpProto* proto, OpAttrChecker* op_checker)
|
||||
: OpProtoAndCheckerMaker(proto, op_checker) {
|
||||
AddInput("DetectRes",
|
||||
"(LoDTensor) A 2-D LoDTensor with shape [M, 6] represents the "
|
||||
"detections. Each row has 6 values: "
|
||||
"[label, confidence, xmin, ymin, xmax, ymax], M is the total "
|
||||
"number of detect results in this mini-batch. For each instance, "
|
||||
"the offsets in first dimension are called LoD, the number of "
|
||||
"offset is N + 1, if LoD[i + 1] - LoD[i] == 0, means there is "
|
||||
"no detected data.");
|
||||
AddInput("Label",
|
||||
"(LoDTensor) A 2-D LoDTensor with shape[N, 6] represents the"
|
||||
"Labeled ground-truth data. Each row has 6 values: "
|
||||
"[label, is_difficult, xmin, ymin, xmax, ymax], N is the total "
|
||||
"number of ground-truth data in this mini-batch. For each "
|
||||
"instance, the offsets in first dimension are called LoD, "
|
||||
"the number of offset is N + 1, if LoD[i + 1] - LoD[i] == 0, "
|
||||
"means there is no ground-truth data.");
|
||||
AddInput("PosCount",
|
||||
"(Tensor) A tensor with shape [Ncls, 1], store the "
|
||||
"input positive example count of each class, Ncls is the count of "
|
||||
"input classification. "
|
||||
"This input is used to pass the AccumPosCount generated by the "
|
||||
"previous mini-batch when the multi mini-batches cumulative "
|
||||
"calculation carried out. "
|
||||
"When the input(PosCount) is empty, the cumulative "
|
||||
"calculation is not carried out, and only the results of the "
|
||||
"current mini-batch are calculated.")
|
||||
.AsDispensable();
|
||||
AddInput("TruePos",
|
||||
"(LoDTensor) A 2-D LoDTensor with shape [Ntp, 2], store the "
|
||||
"input true positive example of each class."
|
||||
"This input is used to pass the AccumTruePos generated by the "
|
||||
"previous mini-batch when the multi mini-batches cumulative "
|
||||
"calculation carried out. ")
|
||||
.AsDispensable();
|
||||
AddInput("FalsePos",
|
||||
"(LoDTensor) A 2-D LoDTensor with shape [Nfp, 2], store the "
|
||||
"input false positive example of each class."
|
||||
"This input is used to pass the AccumFalsePos generated by the "
|
||||
"previous mini-batch when the multi mini-batches cumulative "
|
||||
"calculation carried out. ")
|
||||
.AsDispensable();
|
||||
AddOutput("AccumPosCount",
|
||||
"(Tensor) A tensor with shape [Ncls, 1], store the "
|
||||
"positive example count of each class. It combines the input "
|
||||
"input(PosCount) and the positive example count computed from "
|
||||
"input(Detection) and input(Label).");
|
||||
AddOutput("AccumTruePos",
|
||||
"(LoDTensor) A LoDTensor with shape [Ntp', 2], store the "
|
||||
"true positive example of each class. It combines the "
|
||||
"input(TruePos) and the true positive examples computed from "
|
||||
"input(Detection) and input(Label).");
|
||||
AddOutput("AccumFalsePos",
|
||||
"(LoDTensor) A LoDTensor with shape [Nfp', 2], store the "
|
||||
"false positive example of each class. It combines the "
|
||||
"input(FalsePos) and the false positive examples computed from "
|
||||
"input(Detection) and input(Label).");
|
||||
AddOutput("MAP",
|
||||
"(Tensor) A tensor with shape [1], store the mAP evaluate "
|
||||
"result of the detection.");
|
||||
|
||||
AddAttr<float>(
|
||||
"overlap_threshold",
|
||||
"(float) "
|
||||
"The lower bound jaccard overlap threshold of detection output and "
|
||||
"ground-truth data.")
|
||||
.SetDefault(.3f);
|
||||
AddAttr<bool>("evaluate_difficult",
|
||||
"(bool, default true) "
|
||||
"Switch to control whether the difficult data is evaluated.")
|
||||
.SetDefault(true);
|
||||
AddAttr<std::string>("ap_type",
|
||||
"(string, default 'integral') "
|
||||
"The AP algorithm type, 'integral' or '11point'.")
|
||||
.SetDefault("integral")
|
||||
.InEnum({"integral", "11point"})
|
||||
.AddCustomChecker([](const std::string& ap_type) {
|
||||
PADDLE_ENFORCE_NE(GetAPType(ap_type), APType::kNone,
|
||||
"The ap_type should be 'integral' or '11point.");
|
||||
});
|
||||
AddComment(R"DOC(
|
||||
Detection mAP evaluate operator.
|
||||
The general steps are as follows. First, calculate the true positive and
|
||||
false positive according to the input of detection and labels, then
|
||||
calculate the mAP evaluate value.
|
||||
Supporting '11 point' and 'integral' mAP algorithm. Please get more information
|
||||
from the following articles:
|
||||
https://sanchom.wordpress.com/tag/average-precision/
|
||||
https://arxiv.org/abs/1512.02325
|
||||
|
||||
)DOC");
|
||||
}
|
||||
};
|
||||
|
||||
} // namespace operators
|
||||
} // namespace paddle
|
||||
|
||||
namespace ops = paddle::operators;
|
||||
REGISTER_OP_WITHOUT_GRADIENT(detection_map, ops::DetectionMAPOp,
|
||||
ops::DetectionMAPOpMaker);
|
||||
REGISTER_OP_CPU_KERNEL(
|
||||
detection_map, ops::DetectionMAPOpKernel<paddle::platform::CPUPlace, float>,
|
||||
ops::DetectionMAPOpKernel<paddle::platform::CPUPlace, double>);
|
||||
File diff suppressed because it is too large
Load Diff
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in new issue