From 746517cc4abafe8b795ba0f0ff9d1cf49996acf8 Mon Sep 17 00:00:00 2001 From: Liu Yiqun Date: Thu, 28 Sep 2017 09:55:37 +0800 Subject: [PATCH 01/63] Update the documentation of how to cross-compile for Android. --- .../cross_compiling_for_android_cn.md | 111 +++++++++++++++--- 1 file changed, 97 insertions(+), 14 deletions(-) diff --git a/doc/howto/cross_compiling/cross_compiling_for_android_cn.md b/doc/howto/cross_compiling/cross_compiling_for_android_cn.md index 90dc84718c..0e67c12c57 100644 --- a/doc/howto/cross_compiling/cross_compiling_for_android_cn.md +++ b/doc/howto/cross_compiling/cross_compiling_for_android_cn.md @@ -1,9 +1,56 @@ # 构建Android平台上的PaddlePaddle库 -用户可通过交叉编译的方式，在用户熟悉的开发平台（Linux，Mac OS X和Windows）上编译Android平台上适用的PaddlePaddle库。 +用户可通过如下两种方式，交叉编译Android平台上适用的PaddlePaddle库： +- 基于Docker容器的编译方式 +- 基于Linux交叉编译环境的编译方式 + +## 基于Docker容器的编译方式 +Docker能在所有主要操作系统（包括Linux，Mac OS X和Windows）上运行，因此，使用基于Docker容器的编译方式，用户可在自己熟悉的开发平台上编译Android平台上适用的PaddlePaddle库。 + +### 构建PaddlePaddle的Android开发镜像 +我们把PaddlePaddle的交叉编译环境打包成一个镜像，称为开发镜像，里面涵盖了交叉编译Android版PaddlePaddle库需要的所有编译工具。 + +```bash +$ git clone https://github.com/PaddlePaddle/Paddle.git +$ cd Paddle +$ docker build -t username/paddle-android:dev . -f Dockerfile.android +``` + +### 编译PaddlePaddle C-API库 +构建好开发镜像后，即可使用开发镜像来编译Android版PaddlePaddle C-API库。 +Android的Docker开发镜像向用户提供两个可配置的参数： + +| Argument | Optional Values | Default | +|-----------------|-------------------------|---------| +|`ANDROID_ABI` |`armeabi-v7a, arm64-v8a` | `armeabi-v7a` | +|`ANDROID_API` |`armeabi-v7a(>15), arm64-v8a(>20)` | `21` | + +- 编译`armeabi-v7a`，`Android API 21`的PaddlePaddle库 +```bash +$ docker run -it --rm -v $PWD:/paddle -e "ANDROID_ABI=armeabi-v7a" -e "ANDROID_API=21" username/paddle-android:dev +``` +或 +```bash +$ docker run -it --rm -v $PWD:/paddle username/paddle-android:dev +``` + +如需编译Android API低于21的Paddle库，请参考本文档的**准备交叉编译环境**章节。 + +- 编译`arm64-v8a`，`Android API 21`的PaddlePaddle库 +```bash +$ docker run -it --rm -v $PWD:/paddle -e "ANDROID_ABI=arm64-v8a" -e "ANDROID_API=21" username/paddle-android:dev +``` +或 +```bash +$ docker run -it --rm -v $PWD:/paddle -e "ANDROID_ABI=arm64-v8a" username/paddle-android:dev +``` + +执行上述`docker run`命令时，容器默认执行[paddle/scripts/docker/build_android.sh](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/scripts/docker/build_android.sh)脚本。该脚本中记录了交叉编译Android版PaddlePaddle库常用的CMake配置，并且会根据`ANDROID_ABI`和`ANDROID_API`自动构建独立工具链、进行编译和安装。由于arm64架构要求Android API不小于21。因此当`ANDROID_ABI=arm64-v8a`，`ANDROID_API<21`时，Docker容器中将默认使用`Android API 21`的编译工具链。用户可以参考下文**配置交叉编译参数**章节，根据个人的需求修改定制Docker容器所执行的脚本。编译安装结束之后，PaddlePaddle的C-API库将被安装到`$PWD/install_android`目录，所依赖的第三方库同时也被安装到`$PWD/install_android/third_party`目录。 + +## 基于Linux交叉编译环境的编译方式 本文档将以Linux x86-64平台为例，介绍交叉编译Android平台上适用的PaddlePaddle库的方法和步骤。 -## 准备交叉编译环境 +### 准备交叉编译环境 从源码交叉编译PaddlePaddle，用户需要提前准备好交叉编译环境。Android平台上使用的C/C++交叉编译工具链为[Android NDK](https://developer.android.com/ndk/downloads/index.html?hl=zh-cn)，用户可自行前往下载预编译好的版本，也可通过以下命令获取： @@ -13,18 +60,29 @@ unzip -q android-ndk-r14b-linux-x86_64.zip ``` Android NDK中包含了所有Android API级别、所有架构（arm/arm64/x86/mips）需要用到的编译工具和系统库。用户可根据自己的编译目标架构、所需支持的最低Android API级别，构建[独立工具链](https://developer.android.google.cn/ndk/guides/standalone_toolchain.html?hl=zh-cn)。 -比如： +- 构建`armeabi-v7a`、 `Android API 21`的独立工具链： + +```bash +your/path/to/android-ndk-r14b-linux-x86_64/build/tools/make-standalone-toolchain.sh \ + --arch=arm --platform=android-21 --install-dir=your/path/to/arm_standalone_toolchain +``` + +此命令将在`your/path/to/arm_standalone_toolchain`目录生成一套独立编译工具链，面向架构为32位ARM架构，支持的最小的Android API级别为21，支持编译器`arm-linux-androideabi-gcc (GCC) 4.9`和`clang 3.8`。 + +注意：**PaddlePaddle要求使用的编译工具链所支持的Andoid API级别不小于16**。但由于PaddlePaddle所依赖的第三方库`glog`不支持低于21的Android API，所以在编译Android API低于21的Paddle C-API库时，需要将[cmake/external/glog.cmake](https://github.com/PaddlePaddle/Paddle/blob/develop/cmake/external/glog.cmake#L33)中的`GIT_REPOSITORY`临时修改为`https://github.com/Xreki/glog.git`。 + +- 构建`arm64-v8a`、 `Android API 21`的独立工具链： ```bash your/path/to/android-ndk-r14b-linux-x86_64/build/tools/make-standalone-toolchain.sh \ - --arch=arm --platform=android-21 --install-dir=your/path/to/my_standalone_toolchain + --arch=arm64 --platform=android-21 --install-dir=your/path/to/arm64_standalone_toolchain ``` -此命令将在your/path/to/my_standalone_toolchain目录生成一套编译工具链，面向架构为32位ARM架构，支持的最小的Android API级别为21，使用的编译器为arm-linux-androideabi-gcc (GCC) 4.9。 +此命令将在`your/path/to/arm64_standalone_toolchain`目录生成一套独立编译工具链，面向架构为64位ARM64架构，支持的最小Android API级别为21，支持编译器`arm-linux-androideabi-gcc (GCC) 4.9`和`clang 3.8`。 -注意：**PaddlePaddle要求使用的编译工具链所支持的Andoid API级别不小于21**。 +注意：**arm64架构要求Android API不小于21**。 -## 配置交叉编译参数 +### 配置交叉编译参数 CMake系统对交叉编译提供了支持[cmake-toolchains](https://cmake.org/cmake/help/v3.0/manual/cmake-toolchains.7.html#cross-compiling)。为了简化cmake配置，PaddlePaddle为交叉编译提供了工具链配置文档[cmake/cross_compiling/android.cmake](https://github.com/PaddlePaddle/Paddle/blob/develop/cmake/cross_compiling/android.cmake)，以提供一些默认的编译器和编译参数相关配置。注意，从CMake 3.7版本开始，CMake官方对Android平台的交叉编译提供了通用的支持。PaddlePaddle若检测到用户使用的CMake版本不低于3.7时，将会将用户传进来的配置参数传递CMake系统，交由CMake系统本身来处理。有关参数配置的详细说明见[cmake-toolchains](https://cmake.org/cmake/help/v3.7/manual/cmake-toolchains.7.html#cross-compiling)。 @@ -36,32 +94,57 @@ CMake系统对交叉编译提供了支持[cmake-toolchains](https://cmake.org/cm Android平台可选配置参数： - `ANDROID_STANDALONE_TOOLCHAIN`，独立工具链所在的绝对路径，或者相对于构建目录的相对路径。PaddlePaddle的CMake系统将根据该值自动推导和设置需要使用的交叉编译器、sysroot、以及Android API级别；否则，用户需要在cmake时手动设置这些值。无默认值。 -- `ANDROID_ABI`，目标架构ABI。目前只支持`armeabi-v7a`，默认值为`armeabi-v7a`。 +- `ANDROID_TOOLCHAIN`，目标工具链。可设置`gcc/clang`，默认值为`clang`。 + - CMake 3.7以上，将会始终使用`clang`工具链；CMake 3.7以下，可设置`ANDROID_TOOLCHAIN=gcc`以使用`gcc`工具链。 + - Android官方提供的`clang`编译器要求系统支持`GLIBC 2.15`以上。 +- `ANDROID_ABI`，目标架构ABI。目前支持`armeabi-v7a`和`arm64-v8a`，默认值为`armeabi-v7a`。 - `ANDROID_NATIVE_API_LEVEL`，工具链的Android API级别。若没有显式设置，PaddlePaddle将根据`ANDROID_STANDALONE_TOOLCHAIN`的值自动推导得到。 -- `ANROID_ARM_MODE`，是否使用ARM模式。可设置`ON/OFF`，默认值为`ON`。 -- `ANDROID_ARM_NEON`，是否使用NEON指令。目前必须设置成`ON`，默认值为`ON`。 +- `ANROID_ARM_MODE`，是否使用ARM模式。 + - `ANDROID_ABI=armeabi-v7a`时，可设置`ON/OFF`，默认值为`ON`； + - `ANDROID_ABI=arm64-v8a`时，不需要设置。 +- `ANDROID_ARM_NEON`，是否使用NEON指令。 + - `ANDROID_ABI=armeabi-v7a`时，可设置`ON/OFF`，默认值为`ON`； + - `ANDROID_ABI=arm64-v8a`时，不需要设置。 其他配置参数： +- `USE_EIGEN_FOR_BLAS`，是否使用Eigen库进行矩阵计算。可设置`ON/OFF`，默认值为`OFF`。 - `HOST_C/CXX_COMPILER`，宿主机的C/C++编译器。在编译宿主机版protoc可执行文件和目标机版OpenBLAS库时需要用到。默认设置成环境变量`CC`的值；若环境变量`CC`没有设置，则设置成`cc`编译器。 -一种常用的cmake配置如下： +常用的cmake配置如下： ```bash cmake -DCMAKE_SYSTEM_NAME=Android \ - -DANDROID_STANDALONE_TOOLCHAIN=your/path/to/my_standalone_toolchain \ + -DANDROID_STANDALONE_TOOLCHAIN=your/path/to/arm_standalone_toolchain \ -DANDROID_ABI=armeabi-v7a \ -DANDROID_ARM_NEON=ON \ -DANDROID_ARM_MODE=ON \ + -DUSE_EIGEN_FOR_BLAS=ON \ -DCMAKE_INSTALL_PREFIX=your/path/to/install \ -DWITH_C_API=ON \ -DWITH_SWIG_PY=OFF \ .. ``` +``` +cmake -DCMAKE_SYSTEM_NAME=Android \ + -DANDROID_STANDALONE_TOOLCHAIN=your/path/to/arm64_standalone_toolchain \ + -DANDROID_ABI=arm64-v8a \ + -DUSE_EIGEN_FOR_BLAS=OFF \ + -DCMAKE_INSTALL_PREFIX=your/path/to/install \ + -DWITH_C_API=ON \ + -DWITH_SWIG_PY=OFF \ + .. +``` + 用户还可根据自己的需求设置其他编译参数。比如希望最小化生成的库的大小，可以设置`CMAKE_BUILD_TYPE`为`MinSizeRel`；若希望最快的执行速度，则可设置`CMAKE_BUILD_TYPE`为`Release`。亦可以通过手动设置`CMAKE_C/CXX_FLAGS_MINSIZEREL/RELEASE`来影响PaddlePaddle的编译过程。 -## 编译和安装 +**性能TIPS**，为了达到最快的计算速度，在CMake参数配置上，有以下建议： +- 设置`CMAKE_BUILD_TYPE`为`Release` +- 使用`clang`编译工具链 +- `armeabi-v7a`时，设置`USE_EIGEN_BLAS=ON`，使用Eigen进行矩阵计算；`arm64-v8a`时，设置`USE_EIGEN_FOR_BLAS=OFF`，使用OpenBLAS进行矩阵计算 + +### 编译和安装 CMake配置完成后，执行以下命令，PaddlePaddle将自动下载和编译所有第三方依赖库、编译和安装PaddlePaddle预测库。 @@ -72,4 +155,4 @@ make install 注意：如果你曾经在源码目录下编译过其他平台的PaddlePaddle库，请先使用`rm -rf`命令删除`third_party`目录和`build`目录，以确保所有的第三方依赖库和PaddlePaddle代码都是针对新的CMake配置重新编译的。 -执行完安装命令后，`your/path/to/install`目录中会包含`include`和`lib`目录，其中`include`中包含C-API的头文件，`lib`中包含一个Android版本的库。自此，PaddlePaddle的已经安装完成，用户可将`your/path/to/install`目录下的生成文件用于深度学习相关Android App中，调用方法见C-API文档。 +执行完安装命令后，`your/path/to/install`目录中会包含`include`、`lib`和`third_party`目录，其中`include`中包含C-API的头文件，`lib`中包含若干个不同Android ABI的PaddlePaddle库，`third_party`中包含所依赖的所有第三方库。自此，PaddlePaddle的已经安装完成，用户可将`your/path/to/install`目录下的生成文件用于深度学习相关Android App中，调用方法见C-API文档。 From d28b3094dd75bce8df079fce5d1fa2f33654ba56 Mon Sep 17 00:00:00 2001 From: sidgoyal78 Date: Mon, 2 Oct 2017 20:52:31 -0700 Subject: [PATCH 02/63] Add momentum operator --- paddle/operators/momentum_op.cc | 89 +++++++++++++++++++ paddle/operators/momentum_op.cu | 20 +++++ paddle/operators/momentum_op.h | 53 +++++++++++ .../v2/framework/tests/test_momentum_op.py | 35 ++++++++ 4 files changed, 197 insertions(+) create mode 100644 paddle/operators/momentum_op.cc create mode 100644 paddle/operators/momentum_op.cu create mode 100644 paddle/operators/momentum_op.h create mode 100644 python/paddle/v2/framework/tests/test_momentum_op.py diff --git a/paddle/operators/momentum_op.cc b/paddle/operators/momentum_op.cc new file mode 100644 index 0000000000..2c6ffd618a --- /dev/null +++ b/paddle/operators/momentum_op.cc @@ -0,0 +1,89 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. */ + +#include "paddle/operators/momentum_op.h" + +namespace paddle { +namespace operators { + +class MomentumOp : public framework::OperatorWithKernel { + public: + using framework::OperatorWithKernel::OperatorWithKernel; + + protected: + void InferShape(framework::InferShapeContextBase *ctx) const override { + PADDLE_ENFORCE(ctx->HasInput("Param"), + "Input(param) of Momentum should not be null."); + PADDLE_ENFORCE(ctx->HasInput("Grad"), + "Input(grad) of Momentum should not be null."); + PADDLE_ENFORCE(ctx->HasInput("Velocity"), + "Input(velocity) of Momentum should not be null."); + PADDLE_ENFORCE(ctx->HasInput("LearningRate"), + "Input(LearningRate) of Momentum should not be null."); + + PADDLE_ENFORCE(ctx->HasOutput("ParamOut"), + "Output(ParamOut) of Momentum should not be null."); + PADDLE_ENFORCE(ctx->HasOutput("VelocityOut"), + "Output(VelocityOut) of Momentum should not be null."); + + auto param_dim = ctx->GetInputDim("Param"); + PADDLE_ENFORCE_EQ( + param_dim, ctx->GetInputDim("Grad"), + "Param and Grad input of MomentumOp should have the same dimension."); + PADDLE_ENFORCE_EQ( + param_dim, ctx->GetInputDim("Velocity"), + "Param and Velocity of MomentumOp should have the same dimension."); + PADDLE_ENFORCE_EQ(framework::product(ctx->GetInputDim("LearningRate")), 1, + "Learning_rate should be a scalar"); + + ctx->SetOutputDim("ParamOut", param_dim); + ctx->SetOutputDim("VelocityOut", param_dim); + } +}; + +class MomentumOpMaker : public framework::OpProtoAndCheckerMaker { + public: + MomentumOpMaker(framework::OpProto *proto, + framework::OpAttrChecker *op_checker) + : OpProtoAndCheckerMaker(proto, op_checker) { + AddInput("Param", "Input parameter"); + AddInput("Grad", "Input gradient"); + AddInput("Velocity", "Input velocity"); + AddInput("LearningRate", "Input learning rate"); + + AddOutput("ParamOut", "Output parameter"); + AddOutput("VelocityOut", "Output velocity"); + + AddAttr("mu", "Momentum coefficient"); + AddComment(R"DOC( + +Momentum Algorithm (momentum). + +velocity_out = mu * velocity - learning_rate * grad +param_out = param + velocity_out + +Ref: Sutskever, Ilya, et al. "On the importance of initialization + and momentum in deep learning." ICML 2013; + http://jmlr.org/proceedings/papers/v28/sutskever13.pdf + +)DOC"); + } +}; +} // namespace operators +} // namespace paddle + +namespace ops = paddle::operators; +REGISTER_OP_WITHOUT_GRADIENT(momentum, ops::MomentumOp, ops::MomentumOpMaker); +REGISTER_OP_CPU_KERNEL( + momentum, ops::MomentumOpKernel); diff --git a/paddle/operators/momentum_op.cu b/paddle/operators/momentum_op.cu new file mode 100644 index 0000000000..efc24e795e --- /dev/null +++ b/paddle/operators/momentum_op.cu @@ -0,0 +1,20 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#define EIGEN_USE_GPU +#include "paddle/operators/momentum_op.h" + +namespace ops = paddle::operators; +REGISTER_OP_GPU_KERNEL( + momentum, ops::MomentumOpKernel); diff --git a/paddle/operators/momentum_op.h b/paddle/operators/momentum_op.h new file mode 100644 index 0000000000..60ff2b7590 --- /dev/null +++ b/paddle/operators/momentum_op.h @@ -0,0 +1,53 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. */ + +#pragma once +#include "paddle/framework/eigen.h" +#include "paddle/framework/op_registry.h" + +namespace paddle { +namespace operators { + +using Tensor = framework::Tensor; +template +using EigenVector = framework::EigenVector; + +template +class MomentumOpKernel : public framework::OpKernel { + public: + void Compute(const framework::ExecutionContext& ctx) const override { + auto param_out = ctx.Output("ParamOut"); + auto velocity_out = ctx.Output("VelocityOut"); + + param_out->mutable_data(ctx.GetPlace()); + velocity_out->mutable_data(ctx.GetPlace()); + + float mu = ctx.Attr("mu"); + + auto p = EigenVector::Flatten(*ctx.Input("Param")); + auto g = EigenVector::Flatten(*ctx.Input("Grad")); + auto v = EigenVector::Flatten(*ctx.Input("Velocity")); + float lr = ctx.Input("LearningRate")->data()[0]; + auto p_out = EigenVector::Flatten(*param_out); + auto v_out = EigenVector::Flatten(*velocity_out); + auto place = ctx.GetEigenDevice(); + + v_out.device(place) = mu * v - lr * g; + p_out.device(place) = p + v_out; + } +}; + +} // namespace operators +} // namespace paddle diff --git a/python/paddle/v2/framework/tests/test_momentum_op.py b/python/paddle/v2/framework/tests/test_momentum_op.py new file mode 100644 index 0000000000..cb455bdc9f --- /dev/null +++ b/python/paddle/v2/framework/tests/test_momentum_op.py @@ -0,0 +1,35 @@ +import unittest +import numpy as np +from op_test import OpTest + + +class TestMomentumOp(OpTest): + def setUp(self): + self.op_type = "momentum" + + param = np.random.random((123, 321)).astype("float32") + grad = np.random.random((123, 321)).astype("float32") + velocity = np.zeros((123, 321)).astype("float32") + learning_rate = np.array([0.001]).astype("float32") + mu = 0.0001 + + self.inputs = { + 'Param': param, + 'Grad': grad, + 'Velocity': velocity, + 'LearningRate': learning_rate + } + + self.attrs = {'mu': mu} + + velocity_out = mu * velocity - learning_rate * grad + param_out = param + velocity_out + + self.outputs = {'ParamOut': param_out, 'VelocityOut': velocity_out} + + def test_check_output(self): + self.check_output() + + +if __name__ == "__main__": + unittest.main() From c10da26cf5626c6b34ab31b864fc49ea9c6e725c Mon Sep 17 00:00:00 2001 From: sidgoyal78 Date: Thu, 5 Oct 2017 14:54:33 -0700 Subject: [PATCH 03/63] Modify implementation --- paddle/operators/momentum_op.cc | 35 +++++++++++-------- paddle/operators/momentum_op.h | 12 +++---- .../v2/framework/tests/test_momentum_op.py | 4 +-- 3 files changed, 28 insertions(+), 23 deletions(-) diff --git a/paddle/operators/momentum_op.cc b/paddle/operators/momentum_op.cc index 2c6ffd618a..efa0b59992 100644 --- a/paddle/operators/momentum_op.cc +++ b/paddle/operators/momentum_op.cc @@ -57,25 +57,30 @@ class MomentumOpMaker : public framework::OpProtoAndCheckerMaker { MomentumOpMaker(framework::OpProto *proto, framework::OpAttrChecker *op_checker) : OpProtoAndCheckerMaker(proto, op_checker) { - AddInput("Param", "Input parameter"); - AddInput("Grad", "Input gradient"); - AddInput("Velocity", "Input velocity"); - AddInput("LearningRate", "Input learning rate"); - - AddOutput("ParamOut", "Output parameter"); - AddOutput("VelocityOut", "Output velocity"); - - AddAttr("mu", "Momentum coefficient"); + AddInput("Param", + "(Tensor, default Tensor) " + "Input parameter that has to be updated"); + AddInput("Grad", + "(Tensor, default Tensor) " + "Input gradient of the parameter"); + AddInput("Velocity", + "(Tensor, default Tensor) " + "Input velocity (corresponding to the parameter) " + "that has to be updated"); + AddInput("LearningRate", + "(Tensor, default Tensor) " + "Input learning rate"); + + AddOutput("ParamOut", "(Tensor) Output updated parameter"); + AddOutput("VelocityOut", "(Tensor) Output updated velocity"); + + AddAttr("mu", "(float) Momentum coefficient"); AddComment(R"DOC( Momentum Algorithm (momentum). -velocity_out = mu * velocity - learning_rate * grad -param_out = param + velocity_out - -Ref: Sutskever, Ilya, et al. "On the importance of initialization - and momentum in deep learning." ICML 2013; - http://jmlr.org/proceedings/papers/v28/sutskever13.pdf +velocity = mu * velocity + gradient +param = param - learning_rate * velocity )DOC"); } diff --git a/paddle/operators/momentum_op.h b/paddle/operators/momentum_op.h index 60ff2b7590..fa3788a8ab 100644 --- a/paddle/operators/momentum_op.h +++ b/paddle/operators/momentum_op.h @@ -36,16 +36,16 @@ class MomentumOpKernel : public framework::OpKernel { float mu = ctx.Attr("mu"); - auto p = EigenVector::Flatten(*ctx.Input("Param")); - auto g = EigenVector::Flatten(*ctx.Input("Grad")); - auto v = EigenVector::Flatten(*ctx.Input("Velocity")); - float lr = ctx.Input("LearningRate")->data()[0]; + auto param = EigenVector::Flatten(*ctx.Input("Param")); + auto grad = EigenVector::Flatten(*ctx.Input("Grad")); + auto velocity = EigenVector::Flatten(*ctx.Input("Velocity")); + float learning_rate = ctx.Input("LearningRate")->data()[0]; auto p_out = EigenVector::Flatten(*param_out); auto v_out = EigenVector::Flatten(*velocity_out); auto place = ctx.GetEigenDevice(); - v_out.device(place) = mu * v - lr * g; - p_out.device(place) = p + v_out; + v_out.device(place) = velocity * mu + grad; + p_out.device(place) = param - learning_rate * v_out; } }; diff --git a/python/paddle/v2/framework/tests/test_momentum_op.py b/python/paddle/v2/framework/tests/test_momentum_op.py index cb455bdc9f..d3353ff6e4 100644 --- a/python/paddle/v2/framework/tests/test_momentum_op.py +++ b/python/paddle/v2/framework/tests/test_momentum_op.py @@ -22,8 +22,8 @@ class TestMomentumOp(OpTest): self.attrs = {'mu': mu} - velocity_out = mu * velocity - learning_rate * grad - param_out = param + velocity_out + velocity_out = mu * velocity + grad + param_out = param - learning_rate * velocity_out self.outputs = {'ParamOut': param_out, 'VelocityOut': velocity_out} From db77937ea4985dfc6404cc120457d21774fcd3ed Mon Sep 17 00:00:00 2001 From: sidgoyal78 Date: Thu, 5 Oct 2017 16:11:45 -0700 Subject: [PATCH 04/63] Fix learning_rate usage for momentum --- paddle/operators/momentum_op.h | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/paddle/operators/momentum_op.h b/paddle/operators/momentum_op.h index fa3788a8ab..f7a724f048 100644 --- a/paddle/operators/momentum_op.h +++ b/paddle/operators/momentum_op.h @@ -19,33 +19,35 @@ limitations under the License. */ namespace paddle { namespace operators { -using Tensor = framework::Tensor; -template -using EigenVector = framework::EigenVector; - template class MomentumOpKernel : public framework::OpKernel { public: void Compute(const framework::ExecutionContext& ctx) const override { - auto param_out = ctx.Output("ParamOut"); - auto velocity_out = ctx.Output("VelocityOut"); + auto param_out = ctx.Output("ParamOut"); + auto velocity_out = ctx.Output("VelocityOut"); + auto param = ctx.Input("Param"); + auto velocity = ctx.Input("Velocity"); + auto grad = ctx.Input("Grad"); + auto learning_rate = ctx.Input("LearningRate"); param_out->mutable_data(ctx.GetPlace()); velocity_out->mutable_data(ctx.GetPlace()); float mu = ctx.Attr("mu"); - auto param = EigenVector::Flatten(*ctx.Input("Param")); - auto grad = EigenVector::Flatten(*ctx.Input("Grad")); - auto velocity = EigenVector::Flatten(*ctx.Input("Velocity")); - float learning_rate = ctx.Input("LearningRate")->data()[0]; - auto p_out = EigenVector::Flatten(*param_out); - auto v_out = EigenVector::Flatten(*velocity_out); + auto p_out = framework::EigenVector::Flatten(*param_out); + auto v_out = framework::EigenVector::Flatten(*velocity_out); + + auto p = framework::EigenVector::Flatten(*param); + auto v = framework::EigenVector::Flatten(*velocity); + auto g = framework::EigenVector::Flatten(*grad); + auto lr = framework::EigenVector::Flatten(*learning_rate); + auto place = ctx.GetEigenDevice(); - v_out.device(place) = velocity * mu + grad; - p_out.device(place) = param - learning_rate * v_out; + Eigen::DSizes grad_dsize(grad->numel()); + v_out.device(place) = v * mu + g; + p_out.device(place) = p - lr.broadcast(grad_dsize) * v_out; } }; From d34aadb3f6c92ef0b7909cfaa7e006b1e9ff7561 Mon Sep 17 00:00:00 2001 From: "Yang Yang(Tony)" Date: Mon, 9 Oct 2017 14:17:13 -0700 Subject: [PATCH 05/63] Create executor.md --- doc/design/executor.md | 95 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 95 insertions(+) create mode 100644 doc/design/executor.md diff --git a/doc/design/executor.md b/doc/design/executor.md new file mode 100644 index 0000000000..bf7f05552c --- /dev/null +++ b/doc/design/executor.md @@ -0,0 +1,95 @@ +# Executor Desgin Doc + +## Overview + +`Executor` evaluates a `ProgramDesc`. Essentially, it instantializes Variables and Operators, then run all the operators + +```c++ +void Executor::Run(const ProgramDesc& pdesc, Scope* scope) { + auto& block = pdesc.blocks(0); + auto& device = device_contexts_[0]; + + // Instantiate all the vars in the global scope + for (auto& var : block.vars()) { + scope->NewVar(var.name()); + } + + // Decide which operator should be run + std::vector should_run = Preprocess(pdesc); + + // Run the block + Scope& local_scope = scope->NewScope(); + for (size_t i = 0; i < should_run.size(); ++i) { + if (should_run[i]) { + for (auto var : block.ops(i).outputs()) { + for (auto argu : var.arguments()) { + // Create variable in the local_scope + if (local_scope.FindVar(argu) == nullptr) { + local_scope.NewVar(argu); + } + } + } + auto op = paddle::framework::OpRegistry::CreateOp(block.ops(i)); + op->Run(local_scope, *device); + } + } +} +``` + +## Tasks + +As shown above, it is not hard to simply evaluate the graph. The real problem +is how do we actually construct the `ProgramDesc`. There are several different +situations that we need to consider. + +### 1. Init @tony @qijun + +##### Problem: + +Not sure which block to put init ops. Same concerns applys to `Load Model`. + +##### Solution: In seperate Blocks + +All `initop` and `parameter` goes to `block[0]`. Actual run starts from `block[1]`. + +When user writes `a = Parameter(Variable, init)`, a init op is inserted into +`block[0]`, and a `NOP` is inserted into `block[1]` to substitute init op. + +- Pro: + - Init Op can be run multiple times. + - Compatiable with current `Executor::Preprocessing` + - Still only one `ProgramDesc` +- Con: + - Let others know! + +### 2. IO + +#### 2.1 FeedOp and FetchOp + +Design Doc: https://github.com/PaddlePaddle/Paddle/pull/4599 + +FeedOp and FetchOp in distributed environment: +https://github.com/PaddlePaddle/Paddle/issues/4613 + +#### 2.2 ReaderOp and WriterOp + +### 3. Backward @jiayi + +Executor test case is a good place to test `backward` module, even though executor +is not necessarily depends on `backward`. Currently exposed issue: + +- Fill One: https://github.com/PaddlePaddle/Paddle/issues/4627 +- Attribute map: https://github.com/PaddlePaddle/Paddle/issues/4642 + +### 4. Optimizer @longfei + +Executor test case is a good place to test `optimizer `module, even though executor +is not necessarily depends on `optimizer `. + +### 5. RNN @chunwei + +To be discussed. + +- How to deal with multiple blocks +- How to deal with LoDTensor + From 4545a058bdcdb95f0b4a1116b40f325765644bf1 Mon Sep 17 00:00:00 2001 From: ranqiu Date: Tue, 10 Oct 2017 16:34:11 +0800 Subject: [PATCH 06/63] add dot-product attention --- doc/api/v2/config/networks.rst | 5 ++ .../paddle/trainer_config_helpers/networks.py | 85 ++++++++++++++++++- 2 files changed, 88 insertions(+), 2 deletions(-) diff --git a/doc/api/v2/config/networks.rst b/doc/api/v2/config/networks.rst index 6e813ab1a8..048379cf01 100644 --- a/doc/api/v2/config/networks.rst +++ b/doc/api/v2/config/networks.rst @@ -125,3 +125,8 @@ simple_attention :members: simple_attention :noindex: +dot_product_attention +--------------------- +.. automodule:: paddle.v2.networks + :members: dot_product_attention + :noindex: diff --git a/python/paddle/trainer_config_helpers/networks.py b/python/paddle/trainer_config_helpers/networks.py index 93e8ac173e..0ecbacb7bb 100644 --- a/python/paddle/trainer_config_helpers/networks.py +++ b/python/paddle/trainer_config_helpers/networks.py @@ -26,8 +26,9 @@ __all__ = [ 'sequence_conv_pool', 'simple_lstm', "simple_img_conv_pool", "img_conv_bn_pool", 'lstmemory_group', 'lstmemory_unit', 'small_vgg', 'img_conv_group', 'vgg_16_network', 'gru_unit', 'gru_group', 'simple_gru', - 'simple_attention', 'simple_gru2', 'bidirectional_gru', 'text_conv_pool', - 'bidirectional_lstm', 'inputs', 'outputs' + 'simple_attention', 'dot_product_attention', 'simple_gru2', + 'bidirectional_gru', 'text_conv_pool', 'bidirectional_lstm', 'inputs', + 'outputs' ] ###################################################### @@ -1361,6 +1362,7 @@ def simple_attention(encoded_sequence, compute attention weight. :type transform_param_attr: ParameterAttribute :return: a context vector + :rtype: LayerOutput """ assert encoded_proj.size == decoder_state.size proj_size = encoded_proj.size @@ -1396,6 +1398,85 @@ def simple_attention(encoded_sequence, input=scaled, pooling_type=SumPooling(), name="%s_pooling" % name) +@wrap_name_default() +def dot_product_attention(encoded_sequence, + attending_sequence, + transformed_state, + softmax_param_attr=None, + name=None): + """ + Calculate and return a context vector with dot-product attention mechanism. + Size of the context vector equals to size of the attending_sequence. + + .. math:: + + a(s_{i-1},h_{j}) & = s_{i-1}^\mathrm{T} h_{j} + + e_{i,j} & = a(s_{i-1}, h_{j}) + + a_{i,j} & = \\frac{exp(e_{i,j})}{\\sum_{k=1}^{T_x}{exp(e_{i,k})}} + + c_{i} & = \\sum_{j=1}^{T_{x}}a_{i,j}z_{j} + + where :math:`h_{j}` is the jth element of encoded_sequence, + :math:`z_{j}` is the jth element of attending_sequence, + :math:`s_{i-1}` is transformed_state + + The example usage is: + + .. code-block:: python + + context = dot_product_attention(encoded_sequence=enc_seq, + attending_sequence=att_seq, + transformed_state=state,) + + :param name: name of the dot-product attention model. + :type name: basestring + :param softmax_param_attr: parameter attribute of sequence softmax + that is used to produce attention weight. + :type softmax_param_attr: ParameterAttribute + :param encoded_sequence: output of the encoder + :type encoded_sequence: LayerOutput + :param attending_sequence: attention weight is computed by a feed forward neural + network which has two inputs : decoder's transformed + hidden state of previous time step and encoder's output. + attending_sequence is the sequence to be attended. + :type attending_sequence: LayerOutput + :param transformed_state: transformed hidden state of decoder in previous time step, + its size should equal to encoded_sequence's. Here we do the + transformation outside dot_product_attention for flexibility + consideration. + :type transformed_state: LayerOutput + :return: a context vector + :rtype: LayerOutput + """ + assert transformed_state.size == encoded_sequence.size + + expanded = expand_layer( + input=transformed_state, + expanded_as=encoded_sequence, + name='%s_expand' % name) + + m = linear_comb_layer( + weights=expanded, vectors=encoded_sequence, name='%s_dot-product') + + attention_weight = fc_layer( + input=m, + size=1, + act=SequenceSoftmaxActivation(), + param_attr=softmax_param_attr, + name="%s_softmax" % name, + bias_attr=False) + + scaled = scaling_layer( + weight=attention_weight, + input=attending_sequence, + name='%s_scaling' % name) + + return pooling_layer( + input=scaled, pooling_type=SumPooling(), name="%s_pooling" % name) + + def inputs(layers, *args): """ Declare the inputs of network. The order of input should be as same as From 3b879598c49b9caa0a2e1b2ad8bec103a15a63ef Mon Sep 17 00:00:00 2001 From: Yang Yang Date: Wed, 11 Oct 2017 01:23:34 +0000 Subject: [PATCH 07/63] update executor design doc --- doc/design/executor.md | 77 ++++++++++++++---------------------------- 1 file changed, 25 insertions(+), 52 deletions(-) diff --git a/doc/design/executor.md b/doc/design/executor.md index bf7f05552c..7376ecaef0 100644 --- a/doc/design/executor.md +++ b/doc/design/executor.md @@ -15,7 +15,7 @@ void Executor::Run(const ProgramDesc& pdesc, Scope* scope) { } // Decide which operator should be run - std::vector should_run = Preprocess(pdesc); + std::vector should_run = Prune(pdesc); // Run the block Scope& local_scope = scope->NewScope(); @@ -23,7 +23,7 @@ void Executor::Run(const ProgramDesc& pdesc, Scope* scope) { if (should_run[i]) { for (auto var : block.ops(i).outputs()) { for (auto argu : var.arguments()) { - // Create variable in the local_scope + // Create temp variable in the local_scope if (local_scope.FindVar(argu) == nullptr) { local_scope.NewVar(argu); } @@ -36,60 +36,33 @@ void Executor::Run(const ProgramDesc& pdesc, Scope* scope) { } ``` -## Tasks +## Challenge -As shown above, it is not hard to simply evaluate the graph. The real problem -is how do we actually construct the `ProgramDesc`. There are several different -situations that we need to consider. +It is not hard to simply evaluate a graph. However, it is hard to determine which op should be run. Consider the following different situations. -### 1. Init @tony @qijun - -##### Problem: - -Not sure which block to put init ops. Same concerns applys to `Load Model`. - -##### Solution: In seperate Blocks - -All `initop` and `parameter` goes to `block[0]`. Actual run starts from `block[1]`. - -When user writes `a = Parameter(Variable, init)`, a init op is inserted into -`block[0]`, and a `NOP` is inserted into `block[1]` to substitute init op. - -- Pro: - - Init Op can be run multiple times. - - Compatiable with current `Executor::Preprocessing` - - Still only one `ProgramDesc` -- Con: - - Let others know! - -### 2. IO - -#### 2.1 FeedOp and FetchOp - -Design Doc: https://github.com/PaddlePaddle/Paddle/pull/4599 - -FeedOp and FetchOp in distributed environment: -https://github.com/PaddlePaddle/Paddle/issues/4613 - -#### 2.2 ReaderOp and WriterOp - -### 3. Backward @jiayi - -Executor test case is a good place to test `backward` module, even though executor -is not necessarily depends on `backward`. Currently exposed issue: - -- Fill One: https://github.com/PaddlePaddle/Paddle/issues/4627 -- Attribute map: https://github.com/PaddlePaddle/Paddle/issues/4642 - -### 4. Optimizer @longfei +```python +# Case 1: run foward pass. +cost_np = executor.run(target=cost) +# Case 2: run backward passing. +opts_np, _ = executor.run(target=[cost, opt]) +# Case 3: run checkpointing +_ = executor.run(target=checkpoint) +``` -Executor test case is a good place to test `optimizer `module, even though executor -is not necessarily depends on `optimizer `. +We want to support the evaluation of both variables and operators. -### 5. RNN @chunwei +## Solution -To be discussed. +To support evaluation of operators, we add `is_target` field in the `OpDesc`. -- How to deal with multiple blocks -- How to deal with LoDTensor +```c++ +message OpDesc { + required string type = 3; + repeated Var inputs = 1; + repeated Var outputs = 2; + repeated Attr attrs = 4; + required bool is_target = 5 [ default = false ]; // true if the op is target +}; +``` +To support evaluation of variables, we add [fetch_op](https://github.com/PaddlePaddle/Paddle/pull/4599). For each variable in the `target`, we insert a `fetch_op` into the `ProgramDesc`. (Also, a user may want to overwrite a variable, so we also added [feed_op](https://github.com/PaddlePaddle/Paddle/pull/4599). ) From a6fbfed2c18b72ad08cc91180fbd8e090f223a61 Mon Sep 17 00:00:00 2001 From: "Yang Yang(Tony)" Date: Wed, 11 Oct 2017 10:37:28 -0700 Subject: [PATCH 08/63] Update executor.md --- doc/design/executor.md | 54 +++++++----------------------------------- 1 file changed, 9 insertions(+), 45 deletions(-) diff --git a/doc/design/executor.md b/doc/design/executor.md index 7376ecaef0..049ddb6a59 100644 --- a/doc/design/executor.md +++ b/doc/design/executor.md @@ -5,8 +5,8 @@ `Executor` evaluates a `ProgramDesc`. Essentially, it instantializes Variables and Operators, then run all the operators ```c++ -void Executor::Run(const ProgramDesc& pdesc, Scope* scope) { - auto& block = pdesc.blocks(0); +void Executor::Run(const ProgramDesc& pdesc, Scope* scope, int block_id) { + auto& block = pdesc.blocks(block_id); auto& device = device_contexts_[0]; // Instantiate all the vars in the global scope @@ -14,55 +14,19 @@ void Executor::Run(const ProgramDesc& pdesc, Scope* scope) { scope->NewVar(var.name()); } - // Decide which operator should be run - std::vector should_run = Prune(pdesc); - // Run the block Scope& local_scope = scope->NewScope(); for (size_t i = 0; i < should_run.size(); ++i) { - if (should_run[i]) { - for (auto var : block.ops(i).outputs()) { - for (auto argu : var.arguments()) { - // Create temp variable in the local_scope - if (local_scope.FindVar(argu) == nullptr) { - local_scope.NewVar(argu); - } + for (auto var : block.ops(i).outputs()) { + for (auto argu : var.arguments()) { + // Create temp variable in the local_scope + if (local_scope.FindVar(argu) == nullptr) { + local_scope.NewVar(argu); } } - auto op = paddle::framework::OpRegistry::CreateOp(block.ops(i)); - op->Run(local_scope, *device); } + auto op = paddle::framework::OpRegistry::CreateOp(block.ops(i)); + op->Run(local_scope, *device); } } ``` - -## Challenge - -It is not hard to simply evaluate a graph. However, it is hard to determine which op should be run. Consider the following different situations. - -```python -# Case 1: run foward pass. -cost_np = executor.run(target=cost) -# Case 2: run backward passing. -opts_np, _ = executor.run(target=[cost, opt]) -# Case 3: run checkpointing -_ = executor.run(target=checkpoint) -``` - -We want to support the evaluation of both variables and operators. - -## Solution - -To support evaluation of operators, we add `is_target` field in the `OpDesc`. - -```c++ -message OpDesc { - required string type = 3; - repeated Var inputs = 1; - repeated Var outputs = 2; - repeated Attr attrs = 4; - required bool is_target = 5 [ default = false ]; // true if the op is target -}; -``` - -To support evaluation of variables, we add [fetch_op](https://github.com/PaddlePaddle/Paddle/pull/4599). For each variable in the `target`, we insert a `fetch_op` into the `ProgramDesc`. (Also, a user may want to overwrite a variable, so we also added [feed_op](https://github.com/PaddlePaddle/Paddle/pull/4599). ) From 11c6dc6798d5e12f3ea7e0d4eea996e47ca8bb7d Mon Sep 17 00:00:00 2001 From: "Yang Yang(Tony)" Date: Wed, 11 Oct 2017 11:00:32 -0700 Subject: [PATCH 09/63] Update executor.md --- doc/design/executor.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/design/executor.md b/doc/design/executor.md index 049ddb6a59..30b76b3385 100644 --- a/doc/design/executor.md +++ b/doc/design/executor.md @@ -16,16 +16,16 @@ void Executor::Run(const ProgramDesc& pdesc, Scope* scope, int block_id) { // Run the block Scope& local_scope = scope->NewScope(); - for (size_t i = 0; i < should_run.size(); ++i) { - for (auto var : block.ops(i).outputs()) { - for (auto argu : var.arguments()) { + for (auto& op_desc : block.ops()) { + for (auto& var : op_desc) { + for (auto& argu : var.arguments()) { // Create temp variable in the local_scope if (local_scope.FindVar(argu) == nullptr) { local_scope.NewVar(argu); } } } - auto op = paddle::framework::OpRegistry::CreateOp(block.ops(i)); + auto op = paddle::framework::OpRegistry::CreateOp(op_desc); op->Run(local_scope, *device); } } From 9d3d82f9799b64a410da66c17c0fbd3cff6a25da Mon Sep 17 00:00:00 2001 From: "Yang Yang(Tony)" Date: Wed, 11 Oct 2017 11:04:29 -0700 Subject: [PATCH 10/63] Update executor.md --- doc/design/executor.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design/executor.md b/doc/design/executor.md index 30b76b3385..777f451b11 100644 --- a/doc/design/executor.md +++ b/doc/design/executor.md @@ -17,7 +17,7 @@ void Executor::Run(const ProgramDesc& pdesc, Scope* scope, int block_id) { // Run the block Scope& local_scope = scope->NewScope(); for (auto& op_desc : block.ops()) { - for (auto& var : op_desc) { + for (auto& var : op_desc.outputs()) { for (auto& argu : var.arguments()) { // Create temp variable in the local_scope if (local_scope.FindVar(argu) == nullptr) { From 1c9e461520a3e7dc9c8753b29a2c39dda9ce1600 Mon Sep 17 00:00:00 2001 From: "Yang Yang(Tony)" Date: Wed, 11 Oct 2017 11:48:39 -0700 Subject: [PATCH 11/63] add detailed overview --- doc/design/executor.md | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/doc/design/executor.md b/doc/design/executor.md index 777f451b11..ab2d6c3558 100644 --- a/doc/design/executor.md +++ b/doc/design/executor.md @@ -1,8 +1,26 @@ -# Executor Desgin Doc +# Executor Design Doc + +## Motivation + +We use executor to do the runtime evaluation of a `ProgramDesc`. ## Overview -`Executor` evaluates a `ProgramDesc`. Essentially, it instantializes Variables and Operators, then run all the operators +An executor takes a `ProgramDesc`, a `block_id` and a `Scope`. The `ProgramDesc` is a list of blocks and each block contains the protobuf definition of all the parameters and operators. The `block_id` specifies the entrance block. And the `Scope` is the container of all the variable instance, which is persistent throughout different runs. + +### What does executor do? + +It evaluates all the operators in the `block_id`th block of a `ProgramDesc`. + +### What does executor NOT do? + +It does not do runtime optimization, meaning intelligently parse the dependency of each op a choose which one to be run and in which order they should be run. + +It does not do graph partitioning, meaning dividing the `ProgramDesc` into several small pieces and executing them on different devices. + +## Implementation + +`Executor` evaluates a `ProgramDesc`. Essentially, it instantiates Variables and Operators, then run all the operators ```c++ void Executor::Run(const ProgramDesc& pdesc, Scope* scope, int block_id) { From 1b1cb44f13242f2e315b6f648679cf936eb999a2 Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Fri, 13 Oct 2017 14:05:28 -0700 Subject: [PATCH 12/63] Complete infer_var_type --- paddle/framework/CMakeLists.txt | 3 + paddle/framework/details/op_registry.h | 19 +++- paddle/framework/op_desc.cc | 14 +++ paddle/framework/op_desc.h | 2 + paddle/framework/op_info.h | 2 +- paddle/framework/type_defs.h | 9 ++ paddle/framework/var_type_inference.h | 29 ++++++ paddle/framework/var_type_inference_test.cc | 103 ++++++++++++++++++++ 8 files changed, 178 insertions(+), 3 deletions(-) create mode 100644 paddle/framework/var_type_inference.h create mode 100644 paddle/framework/var_type_inference_test.cc diff --git a/paddle/framework/CMakeLists.txt b/paddle/framework/CMakeLists.txt index 14947b6f21..2c61ae40a5 100644 --- a/paddle/framework/CMakeLists.txt +++ b/paddle/framework/CMakeLists.txt @@ -53,3 +53,6 @@ endif() cc_library(tensor_array SRCS tensor_array.cc DEPS lod_tensor) cc_test(tensor_array_test SRCS tensor_array_test.cc DEPS tensor_array place) + +cc_test(var_type_inference_test SRCS var_type_inference_test.cc DEPS op_registry + proto_desc) diff --git a/paddle/framework/details/op_registry.h b/paddle/framework/details/op_registry.h index ca8584b78a..71ee143018 100644 --- a/paddle/framework/details/op_registry.h +++ b/paddle/framework/details/op_registry.h @@ -18,6 +18,7 @@ #include "paddle/framework/op_info.h" #include "paddle/framework/op_proto_maker.h" #include "paddle/framework/operator.h" +#include "paddle/framework/var_type_inference.h" namespace paddle { namespace framework { @@ -26,7 +27,8 @@ namespace details { enum OpInfoFillType { kOperator = 0, kOpProtoAndCheckerMaker = 1, - kGradOpDescMaker = 2 + kGradOpDescMaker = 2, + kVarTypeInference = 3 }; template @@ -38,7 +40,9 @@ struct OpInfoFillTypeID { ? kOpProtoAndCheckerMaker : (std::is_base_of::value ? kGradOpDescMaker - : static_cast(-1))); + : (std::is_base_of::value + ? kVarTypeInference + : static_cast(-1)))); } }; @@ -105,6 +109,17 @@ struct OpInfoFiller { }; } }; + +template +struct OpInfoFiller { + void operator()(const char* op_type, OpInfo* info) const { + info->infer_var_type_ = [](const OpDescBind& fwd_op, BlockDescBind* block) { + T inference; + inference(fwd_op, block); + }; + } +}; + } // namespace details } // namespace framework diff --git a/paddle/framework/op_desc.cc b/paddle/framework/op_desc.cc index a5d515bbca..09a544fb9e 100644 --- a/paddle/framework/op_desc.cc +++ b/paddle/framework/op_desc.cc @@ -236,5 +236,19 @@ void OpDescBind::InferShape(const BlockDescBind &block) const { it->second(&ctx); } +void OpDescBind::InferVarType(BlockDescBind *block) const { + auto &info = OpInfoMap::Instance().Get(this->Type()); + if (info.infer_var_type_) { + info.infer_var_type_(*this, block); + } else { + // all output type is LoDTensor by default + for (auto &out_pair : this->outputs_) { + for (auto &out_var_name : out_pair.second) { + block->Var(out_var_name)->SetType(VarDesc::LOD_TENSOR); + } + } + } +} + } // namespace framework } // namespace paddle diff --git a/paddle/framework/op_desc.h b/paddle/framework/op_desc.h index 90155fadea..d05ee0875d 100644 --- a/paddle/framework/op_desc.h +++ b/paddle/framework/op_desc.h @@ -104,6 +104,8 @@ class OpDescBind { void InferShape(const BlockDescBind &block) const; + void InferVarType(BlockDescBind *block) const; + private: template static std::vector MapKeys(const MapType &map) { diff --git a/paddle/framework/op_info.h b/paddle/framework/op_info.h index c504f69e30..e926180780 100644 --- a/paddle/framework/op_info.h +++ b/paddle/framework/op_info.h @@ -19,7 +19,6 @@ #include #include "paddle/framework/attribute.h" -#include "paddle/framework/op_desc.h" #include "paddle/framework/type_defs.h" #include "paddle/platform/macros.h" @@ -31,6 +30,7 @@ struct OpInfo { GradOpMakerFN grad_op_maker_; OpProto* proto_{nullptr}; OpAttrChecker* checker_{nullptr}; + InferVarTypeFN infer_var_type_; bool HasOpProtoAndChecker() const { return proto_ != nullptr && checker_ != nullptr; diff --git a/paddle/framework/type_defs.h b/paddle/framework/type_defs.h index 7e1b79c97b..a4e8253bfd 100644 --- a/paddle/framework/type_defs.h +++ b/paddle/framework/type_defs.h @@ -16,12 +16,18 @@ #include #include #include +#include +#include +#include +#include #include "paddle/platform/variant.h" namespace paddle { namespace framework { class OperatorBase; class OpDescBind; +class BlockDescBind; +class BlockDesc; using VariableNameMap = std::map>; // The order should be as same as framework.proto @@ -39,5 +45,8 @@ using OpCreator = std::function>( const OpDescBind&, const std::unordered_set& /*no_grad_set*/)>; +using InferVarTypeFN = std::function; + } // namespace framework } // namespace paddle diff --git a/paddle/framework/var_type_inference.h b/paddle/framework/var_type_inference.h new file mode 100644 index 0000000000..32abbeb334 --- /dev/null +++ b/paddle/framework/var_type_inference.h @@ -0,0 +1,29 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#pragma once +#include "paddle/framework/type_defs.h" + +namespace paddle { +namespace framework { + +class VarTypeInference { + public: + virtual ~VarTypeInference() {} + virtual void operator()(const OpDescBind& op_desc, + BlockDescBind* block) const = 0; +}; + +} // namespace framework +} // namespace paddle diff --git a/paddle/framework/var_type_inference_test.cc b/paddle/framework/var_type_inference_test.cc new file mode 100644 index 0000000000..e3f4893f1a --- /dev/null +++ b/paddle/framework/var_type_inference_test.cc @@ -0,0 +1,103 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include "paddle/framework/var_type_inference.h" +#include "gtest/gtest.h" +#include "paddle/framework/op_registry.h" +#include "paddle/framework/operator.h" +#include "paddle/framework/program_desc.h" + +namespace paddle { +namespace framework { + +class SumOpMaker : public OpProtoAndCheckerMaker { + public: + SumOpMaker(OpProto *proto, OpAttrChecker *op_checker) + : OpProtoAndCheckerMaker(proto, op_checker) { + AddInput("X", "").AsDuplicable(); + AddOutput("Out", ""); + AddComment(""); + } +}; + +class SumOpVarTypeInference : public VarTypeInference { + public: + void operator()(const OpDescBind &op_desc, + BlockDescBind *block) const override { + auto default_var_type = VarDesc::LOD_TENSOR; + for (auto &in_var_name : op_desc.Input("X")) { + auto in_var_type = block->Var(in_var_name)->GetType(); + if (in_var_type != default_var_type) { + default_var_type = in_var_type; + break; + } + } + auto out_var_name = op_desc.Output("Out").front(); + block->Var(out_var_name)->SetType(default_var_type); + } +}; +} // namespace framework +} // namespace paddle + +REGISTER_OPERATOR(sum, paddle::framework::NOP, paddle::framework::SumOpMaker, + paddle::framework::SumOpVarTypeInference); +REGISTER_OPERATOR(sum_without_infer_var_type, paddle::framework::NOP, + paddle::framework::SumOpMaker); + +namespace paddle { +namespace framework { + +TEST(InferVarType, sum_op) { + auto &prog = ProgramDescBind::Instance(&GetProgramDesc()); + auto *op = prog.Block(0)->AppendOp(); + op->SetType("sum"); + op->SetInput("X", {"test_a", "test_b", "test_c"}); + op->SetOutput("Out", {"test_out"}); + + prog.Block(0)->NewVar("test_a")->SetType(VarDesc_VarType_LOD_TENSOR); + prog.Block(0)->NewVar("test_b")->SetType(VarDesc_VarType_LOD_TENSOR); + prog.Block(0)->NewVar("test_c")->SetType(VarDesc_VarType_LOD_TENSOR); + prog.Block(0)->NewVar("test_out"); + + op->InferVarType(prog.Block(0)); + + ASSERT_EQ(VarDesc_VarType_LOD_TENSOR, + prog.Block(0)->Var("test_out")->GetType()); + + prog.Block(0)->Var("test_b")->SetType(VarDesc_VarType_SELECTED_ROWS); + op->InferVarType(prog.Block(0)); + ASSERT_EQ(VarDesc_VarType_SELECTED_ROWS, + prog.Block(0)->Var("test_out")->GetType()); +} + +TEST(InferVarType, sum_op_without_infer_var_type) { + auto &prog = ProgramDescBind::Instance(&GetProgramDesc()); + auto *op = prog.Block(0)->AppendOp(); + op->SetType("sum_without_infer_var_type"); + op->SetInput("X", {"test2_a", "test2_b", "test2_c"}); + op->SetOutput("Out", {"test2_out"}); + + prog.Block(0)->NewVar("test2_a")->SetType(VarDesc_VarType_LOD_TENSOR); + prog.Block(0)->NewVar("test2_b")->SetType(VarDesc_VarType_SELECTED_ROWS); + prog.Block(0)->NewVar("test2_c")->SetType(VarDesc_VarType_LOD_TENSOR); + prog.Block(0)->NewVar("test2_out"); + + op->InferVarType(prog.Block(0)); + + ASSERT_EQ(VarDesc_VarType_LOD_TENSOR, + prog.Block(0)->Var("test2_out")->GetType()); +} + +} // namespace framework +} // namespace paddle \ No newline at end of file From 5be10872f95c18434bce79c0111717efa1994029 Mon Sep 17 00:00:00 2001 From: qijun Date: Fri, 13 Oct 2017 14:59:20 -0700 Subject: [PATCH 13/63] add selected_rows add cpu functor --- paddle/operators/cross_entropy_op.cu | 3 +- paddle/operators/cross_entropy_op.h | 3 +- paddle/operators/math/math_function.cc | 60 +++++++++++++++++++++ paddle/operators/math/math_function.h | 22 ++++++-- paddle/operators/math/math_function_test.cc | 60 +++++++++++++++++++-- 5 files changed, 137 insertions(+), 11 deletions(-) diff --git a/paddle/operators/cross_entropy_op.cu b/paddle/operators/cross_entropy_op.cu index 5e2024e0ea..07b0388b60 100644 --- a/paddle/operators/cross_entropy_op.cu +++ b/paddle/operators/cross_entropy_op.cu @@ -91,7 +91,8 @@ class CrossEntropyGradientOpCUDAKernel : public framework::OpKernel { .stream()>>>(dx_data, dy_data, x_data, label_data, batch_size, class_num); } else { - math::SetConstant(ctx.device_context(), dx, 0); + math::SetConstant functor; + functor(ctx.device_context(), dx, 0); auto* label_data = label->data(); grid = (batch_size + block - 1) / block; CrossEntropyGradientKernel<<< diff --git a/paddle/operators/cross_entropy_op.h b/paddle/operators/cross_entropy_op.h index d2d321aa7e..19c276d23f 100644 --- a/paddle/operators/cross_entropy_op.h +++ b/paddle/operators/cross_entropy_op.h @@ -70,7 +70,8 @@ class CrossEntropyGradientOpKernel : public framework::OpKernel { const T* x_data = x->data(); const int* label_data = label->data(); - math::SetConstant(ctx.device_context(), dx, 0); + math::SetConstant functor; + functor(ctx.device_context(), dx, 0); for (int i = 0; i < batch_size; ++i) { PADDLE_ASSERT(label_data[i] >= 0 || label_data[i] < class_num); diff --git a/paddle/operators/math/math_function.cc b/paddle/operators/math/math_function.cc index ba653afa2c..75a705b346 100644 --- a/paddle/operators/math/math_function.cc +++ b/paddle/operators/math/math_function.cc @@ -13,6 +13,7 @@ See the License for the specific language governing permissions and limitations under the License. */ #include "paddle/operators/math/math_function.h" +#include namespace paddle { namespace operators { @@ -130,6 +131,65 @@ void matmul( matrix_b.data(), beta, matrix_out->data()); } +template struct SetConstant; + +namespace detail { +size_t FindPos(const std::vector& rows, int64_t value) { + for (size_t i = 0; i < rows.size(); i++) { + if (rows[i] == value) { + return i; + } + } + return 0; +} +} // namespace detail + +template +struct SelectedRowsAdd { + void operator()(const platform::DeviceContext& context, + const framework::SelectedRows& input1, + const framework::SelectedRows& input2, + framework::SelectedRows* output) { + auto in1_height = input1.height(); + PADDLE_ENFORCE_EQ(in1_height, input2.height()); + PADDLE_ENFORCE_EQ(in1_height, output->height()); + + auto& in1_rows = input1.rows(); + auto& in2_rows = input2.rows(); + auto& out_rows = output->rows(); + + auto* out_value = output->mutable_value(); + auto& in1_value = input1.value(); + auto& in2_value = input2.value(); + + auto in1_row_numel = in1_value.numel() / in1_rows.size(); + PADDLE_ENFORCE_EQ(in1_row_numel, in2_value.numel() / in2_rows.size()); + PADDLE_ENFORCE_EQ(in1_row_numel, out_value->numel() / out_rows.size()); + + SetConstant functor; + functor(context, out_value, 0.0); + auto* out_data = out_value->data(); + + auto* in1_data = in1_value.data(); + for (size_t i = 0; i < in1_rows.size(); i++) { + auto row = detail::FindPos(out_rows, in1_rows[i]); + for (size_t j = 0; j < in1_row_numel; j++) { + out_data[row * in1_row_numel + j] += in1_data[i * in1_row_numel + j]; + } + } + + auto* in2_data = in2_value.data(); + for (size_t i = 0; i < in2_rows.size(); i++) { + auto row = detail::FindPos(out_rows, in2_rows[i]); + for (size_t j = 0; j < in1_row_numel; j++) { + out_data[row * in1_row_numel + j] += in2_data[i * in1_row_numel + j]; + } + } + } +}; + +template struct SelectedRowsAdd; + } // namespace math } // namespace operators } // namespace paddle diff --git a/paddle/operators/math/math_function.h b/paddle/operators/math/math_function.h index 473eff4d19..f298f34bab 100644 --- a/paddle/operators/math/math_function.h +++ b/paddle/operators/math/math_function.h @@ -53,6 +53,7 @@ int LAPACKE_dgetri(int matrix_layout, int n, double* a, int lda, #include #include "paddle/framework/eigen.h" +#include "paddle/framework/selected_rows.h" #include "paddle/framework/tensor.h" #include "paddle/platform/device_context.h" #include "paddle/platform/enforce.h" @@ -86,11 +87,22 @@ void matmul(const platform::DeviceContext& context, framework::Tensor* matrix_out, T beta); template -void SetConstant(const platform::DeviceContext& context, - framework::Tensor* tensor, T num) { - auto t = framework::EigenVector::Flatten(*tensor); - t.device(*context.GetEigenDevice()) = t.constant(static_cast(num)); -} +struct SetConstant { + void operator()(const platform::DeviceContext& context, + framework::Tensor* tensor, T num) { + auto t = framework::EigenVector::Flatten(*tensor); + t.device(*context.GetEigenDevice()) = + t.constant(static_cast(num)); + } +}; + +template +struct SelectedRowsAdd { + void operator()(const platform::DeviceContext& context, + const framework::SelectedRows& input1, + const framework::SelectedRows& input2, + framework::SelectedRows* output); +}; } // namespace math } // namespace operators diff --git a/paddle/operators/math/math_function_test.cc b/paddle/operators/math/math_function_test.cc index c87d200c3a..43760bc601 100644 --- a/paddle/operators/math/math_function_test.cc +++ b/paddle/operators/math/math_function_test.cc @@ -1,4 +1,5 @@ #include "paddle/operators/math/math_function.h" +#include "glog/logging.h" #include "gtest/gtest.h" #ifdef PADDLE_WITH_CUDA @@ -253,18 +254,69 @@ TEST(math_function, zero) { auto* cpu_place = new paddle::platform::CPUPlace(); float* t = tensor.mutable_data({2, 2}, *cpu_place); paddle::platform::CPUDeviceContext context(*cpu_place); - paddle::operators::math::SetConstant( - context, &tensor, 0); + paddle::operators::math::SetConstant + functor; + functor(context, &tensor, 0); EXPECT_EQ(t[0], 0); EXPECT_EQ(t[1], 0); EXPECT_EQ(t[2], 0); EXPECT_EQ(t[3], 0); - paddle::operators::math::SetConstant( - context, &tensor, 1); + functor(context, &tensor, 1); EXPECT_EQ(t[0], 1); EXPECT_EQ(t[1], 1); EXPECT_EQ(t[2], 1); EXPECT_EQ(t[3], 1); } + +TEST(math_function, selected_rows_add) { + using namespace paddle::framework; + using namespace paddle::platform; + using namespace paddle::operators::math; + + CPUPlace cpu_place; + CPUDeviceContext ctx(cpu_place); + SetConstant functor; + int64_t height = 10; + int64_t row_numel = 10; + + std::vector rows1{0, 4, 7}; + std::unique_ptr selected_rows1{new SelectedRows(rows1, height)}; + auto* in1_value = selected_rows1->mutable_value(); + in1_value->mutable_data( + make_ddim({static_cast(rows1.size()), row_numel}), cpu_place); + functor(ctx, in1_value, 2.0); + + std::vector rows2{0, 5, 7, 9}; + std::unique_ptr selected_rows2{new SelectedRows(rows2, height)}; + auto* in2_value = selected_rows2->mutable_value(); + in2_value->mutable_data( + make_ddim({static_cast(rows2.size()), row_numel}), cpu_place); + functor(ctx, in2_value, 1.0); + + std::unique_ptr output{new SelectedRows()}; + output->set_height(height); + std::vector out_rows = {0, 4, 5, 7, 9}; + output->set_rows(out_rows); + + auto* out_value = output->mutable_value(); + out_value->mutable_data(make_ddim({5, 10}), cpu_place); + + SelectedRowsAdd add_functor; + add_functor(ctx, *selected_rows1, *selected_rows2, output.get()); + + auto* data = output->value().data(); + // out_rows[0] = 0 + EXPECT_EQ(data[0 * row_numel + 0], 3.0); + EXPECT_EQ(data[0 * row_numel + 8], 3.0); + // out_rows[1] = 4 + EXPECT_EQ(data[1 * row_numel + 1], 2.0); + // out_rows[2] = 5 + EXPECT_EQ(data[2 * row_numel + 6], 1.0); + // out_rows[3] = 7 + EXPECT_EQ(data[3 * row_numel + 3], 3.0); + EXPECT_EQ(data[3 * row_numel + 8], 3.0); + // out_rows[4] = 9 + EXPECT_EQ(data[4 * row_numel + 4], 1.0); +} From 7b1834330537594b2dc9267d9249f9d84fef391b Mon Sep 17 00:00:00 2001 From: qijun Date: Fri, 13 Oct 2017 15:02:52 -0700 Subject: [PATCH 14/63] remove unused header file --- paddle/operators/math/math_function.cc | 1 - 1 file changed, 1 deletion(-) diff --git a/paddle/operators/math/math_function.cc b/paddle/operators/math/math_function.cc index 75a705b346..306612b65f 100644 --- a/paddle/operators/math/math_function.cc +++ b/paddle/operators/math/math_function.cc @@ -13,7 +13,6 @@ See the License for the specific language governing permissions and limitations under the License. */ #include "paddle/operators/math/math_function.h" -#include namespace paddle { namespace operators { From 931572e2109d25c825d1d55b17358020c4d4c317 Mon Sep 17 00:00:00 2001 From: qijun Date: Fri, 13 Oct 2017 16:25:22 -0700 Subject: [PATCH 15/63] SelectedRowsAddTensor method --- paddle/framework/selected_rows.h | 3 + paddle/operators/math/math_function.cc | 68 ++++++++++++++---- paddle/operators/math/math_function.h | 9 +++ paddle/operators/math/math_function_test.cc | 79 +++++++++++++++------ 4 files changed, 124 insertions(+), 35 deletions(-) diff --git a/paddle/framework/selected_rows.h b/paddle/framework/selected_rows.h index f9f563051e..ddc6dec194 100644 --- a/paddle/framework/selected_rows.h +++ b/paddle/framework/selected_rows.h @@ -45,6 +45,9 @@ class SelectedRows { } private: + // Notice: rows can be duplicate. We can have {0, 4, 7, 0, 5, 7, 9} here. + // SelectedRows are simplely concated when adding together. Until a + // SelectedRows add a Tensor, will the duplicate rows be handled. std::vector rows_; std::unique_ptr value_{nullptr}; int64_t height_; diff --git a/paddle/operators/math/math_function.cc b/paddle/operators/math/math_function.cc index 306612b65f..ed49a0a549 100644 --- a/paddle/operators/math/math_function.cc +++ b/paddle/operators/math/math_function.cc @@ -13,6 +13,8 @@ See the License for the specific language governing permissions and limitations under the License. */ #include "paddle/operators/math/math_function.h" +#include "paddle/framework/eigen.h" +#include "paddle/memory/memcpy.h" namespace paddle { namespace operators { @@ -151,11 +153,17 @@ struct SelectedRowsAdd { framework::SelectedRows* output) { auto in1_height = input1.height(); PADDLE_ENFORCE_EQ(in1_height, input2.height()); - PADDLE_ENFORCE_EQ(in1_height, output->height()); + output->set_height(in1_height); auto& in1_rows = input1.rows(); auto& in2_rows = input2.rows(); - auto& out_rows = output->rows(); + std::vector out_rows; + out_rows.reserve(in1_rows.size() + in2_rows.size()); + + // concat rows + out_rows.insert(out_rows.end(), in1_rows.begin(), in1_rows.end()); + out_rows.insert(out_rows.end(), in2_rows.begin(), in2_rows.end()); + output->set_rows(out_rows); auto* out_value = output->mutable_value(); auto& in1_value = input1.value(); @@ -165,29 +173,59 @@ struct SelectedRowsAdd { PADDLE_ENFORCE_EQ(in1_row_numel, in2_value.numel() / in2_rows.size()); PADDLE_ENFORCE_EQ(in1_row_numel, out_value->numel() / out_rows.size()); - SetConstant functor; - functor(context, out_value, 0.0); auto* out_data = out_value->data(); auto* in1_data = in1_value.data(); - for (size_t i = 0; i < in1_rows.size(); i++) { - auto row = detail::FindPos(out_rows, in1_rows[i]); - for (size_t j = 0; j < in1_row_numel; j++) { - out_data[row * in1_row_numel + j] += in1_data[i * in1_row_numel + j]; - } - } + memory::Copy(platform::CPUPlace(), out_data, platform::CPUPlace(), in1_data, + in1_value.numel() * sizeof(T)); auto* in2_data = in2_value.data(); - for (size_t i = 0; i < in2_rows.size(); i++) { - auto row = detail::FindPos(out_rows, in2_rows[i]); - for (size_t j = 0; j < in1_row_numel; j++) { - out_data[row * in1_row_numel + j] += in2_data[i * in1_row_numel + j]; + memory::Copy(platform::CPUPlace(), out_data + in1_value.numel(), + platform::CPUPlace(), in2_data, in2_value.numel() * sizeof(T)); + } +}; + +template struct SelectedRowsAdd; + +template +struct SelectedRowsAddTensor { + void operator()(const platform::DeviceContext& context, + const framework::SelectedRows& input1, + const framework::Tensor& input2, framework::Tensor* output) { + auto in1_height = input1.height(); + auto in2_dims = input2.dims(); + auto out_dims = output->dims(); + PADDLE_ENFORCE_EQ(in1_height, in2_dims[0]); + PADDLE_ENFORCE_EQ(in1_height, out_dims[0]); + + auto& in1_value = input1.value(); + auto& in1_rows = input1.rows(); + + int64_t in1_row_numel = in1_value.numel() / in1_rows.size(); + PADDLE_ENFORCE_EQ(in1_row_numel, input2.numel() / in1_height); + PADDLE_ENFORCE_EQ(in1_row_numel, output->numel() / in1_height); + + SetConstant functor; + functor(context, output, 0.0); + + auto* in1_data = in1_value.data(); + auto* out_data = output->data(); + + for (size_t i = 0; i < in1_rows.size(); i++) { + for (int64_t j = 0; j < in1_row_numel; j++) { + out_data[in1_rows[i] * in1_row_numel + j] += + in1_data[i * in1_row_numel + j]; } } + + auto out_eigen = framework::EigenVector::Flatten(*output); + auto in2_eigen = framework::EigenVector::Flatten(input2); + out_eigen.device(*context.GetEigenDevice()) = + out_eigen + in2_eigen; } }; -template struct SelectedRowsAdd; +template struct SelectedRowsAddTensor; } // namespace math } // namespace operators diff --git a/paddle/operators/math/math_function.h b/paddle/operators/math/math_function.h index f298f34bab..0d0d4cdd73 100644 --- a/paddle/operators/math/math_function.h +++ b/paddle/operators/math/math_function.h @@ -96,6 +96,8 @@ struct SetConstant { } }; +// SelectedRows + SelectedRows will simplely concat value and rows. +// The real computation happens in dealing with LoDTensor. template struct SelectedRowsAdd { void operator()(const platform::DeviceContext& context, @@ -104,6 +106,13 @@ struct SelectedRowsAdd { framework::SelectedRows* output); }; +template +struct SelectedRowsAddTensor { + void operator()(const platform::DeviceContext& context, + const framework::SelectedRows& input1, + const framework::Tensor& input2, framework::Tensor* output); +}; + } // namespace math } // namespace operators } // namespace paddle diff --git a/paddle/operators/math/math_function_test.cc b/paddle/operators/math/math_function_test.cc index 43760bc601..e3186171d1 100644 --- a/paddle/operators/math/math_function_test.cc +++ b/paddle/operators/math/math_function_test.cc @@ -286,37 +286,76 @@ TEST(math_function, selected_rows_add) { auto* in1_value = selected_rows1->mutable_value(); in1_value->mutable_data( make_ddim({static_cast(rows1.size()), row_numel}), cpu_place); - functor(ctx, in1_value, 2.0); + functor(ctx, in1_value, 1.0); std::vector rows2{0, 5, 7, 9}; std::unique_ptr selected_rows2{new SelectedRows(rows2, height)}; auto* in2_value = selected_rows2->mutable_value(); in2_value->mutable_data( make_ddim({static_cast(rows2.size()), row_numel}), cpu_place); - functor(ctx, in2_value, 1.0); + functor(ctx, in2_value, 2.0); std::unique_ptr output{new SelectedRows()}; - output->set_height(height); - std::vector out_rows = {0, 4, 5, 7, 9}; - output->set_rows(out_rows); - auto* out_value = output->mutable_value(); - out_value->mutable_data(make_ddim({5, 10}), cpu_place); + + // simplely concat two SelectedRows + out_value->mutable_data(make_ddim({7, 10}), cpu_place); SelectedRowsAdd add_functor; add_functor(ctx, *selected_rows1, *selected_rows2, output.get()); - auto* data = output->value().data(); - // out_rows[0] = 0 - EXPECT_EQ(data[0 * row_numel + 0], 3.0); - EXPECT_EQ(data[0 * row_numel + 8], 3.0); - // out_rows[1] = 4 - EXPECT_EQ(data[1 * row_numel + 1], 2.0); - // out_rows[2] = 5 - EXPECT_EQ(data[2 * row_numel + 6], 1.0); - // out_rows[3] = 7 - EXPECT_EQ(data[3 * row_numel + 3], 3.0); - EXPECT_EQ(data[3 * row_numel + 8], 3.0); - // out_rows[4] = 9 - EXPECT_EQ(data[4 * row_numel + 4], 1.0); + auto out_height = output->height(); + EXPECT_EQ(out_height, height); + + auto& out_rows = output->rows(); + + // input1 rows + EXPECT_EQ(out_rows[0], 0); + EXPECT_EQ(out_rows[1], 4); + EXPECT_EQ(out_rows[2], 7); + // input2 rows + EXPECT_EQ(out_rows[3], 0); + EXPECT_EQ(out_rows[4], 5); + EXPECT_EQ(out_rows[5], 7); + EXPECT_EQ(out_rows[6], 9); + + auto* out_data = output->value().data(); + // input1 value + EXPECT_EQ(out_data[0 * row_numel + 0], 1.0); + EXPECT_EQ(out_data[0 * row_numel + 8], 1.0); + EXPECT_EQ(out_data[1 * row_numel + 1], 1.0); + EXPECT_EQ(out_data[2 * row_numel + 6], 1.0); + // input2 value + EXPECT_EQ(out_data[3 * row_numel + 3], 2.0); + EXPECT_EQ(out_data[3 * row_numel + 8], 2.0); + EXPECT_EQ(out_data[4 * row_numel + 4], 2.0); + EXPECT_EQ(out_data[5 * row_numel + 7], 2.0); + EXPECT_EQ(out_data[6 * row_numel + 9], 2.0); + + std::unique_ptr tensor1{new Tensor()}; + tensor1->mutable_data(make_ddim({height, row_numel}), cpu_place); + SetConstant constant_functor; + constant_functor(ctx, tensor1.get(), 3.0); + + std::unique_ptr tensor2{new Tensor()}; + tensor2->mutable_data(make_ddim({height, row_numel}), cpu_place); + + SelectedRowsAddTensor add_tensor_functor; + add_tensor_functor(ctx, *output, *tensor1, tensor2.get()); + + auto* tensor2_data = tensor2->data(); + // row0: 1.0 + 2.0 + 3.0 + EXPECT_EQ(tensor2_data[0 * row_numel + 0], 6.0); + // row1: 3.0 + EXPECT_EQ(tensor2_data[1 * row_numel + 1], 3.0); + // row4 : 1.0 + 3.0 + EXPECT_EQ(tensor2_data[4 * row_numel + 6], 4.0); + // row5: 2.0 + 3.0 + EXPECT_EQ(tensor2_data[5 * row_numel + 7], 5.0); + // row6: 3.0 + EXPECT_EQ(tensor2_data[6 * row_numel + 1], 3.0); + // row7: 1.0 + 2.0 + 3.0 + EXPECT_EQ(tensor2_data[7 * row_numel + 3], 6.0); + // row9: 2.0 + 3.0 + EXPECT_EQ(tensor2_data[9 * row_numel + 6], 5.0); } From 4741266d6fcac37c678e7815a2b93994fcddaec7 Mon Sep 17 00:00:00 2001 From: qijun Date: Fri, 13 Oct 2017 16:27:39 -0700 Subject: [PATCH 16/63] remove unused method --- paddle/operators/math/math_function.cc | 11 ----------- paddle/operators/math/math_function_test.cc | 1 - 2 files changed, 12 deletions(-) diff --git a/paddle/operators/math/math_function.cc b/paddle/operators/math/math_function.cc index ed49a0a549..ddb904aa46 100644 --- a/paddle/operators/math/math_function.cc +++ b/paddle/operators/math/math_function.cc @@ -134,17 +134,6 @@ void matmul( template struct SetConstant; -namespace detail { -size_t FindPos(const std::vector& rows, int64_t value) { - for (size_t i = 0; i < rows.size(); i++) { - if (rows[i] == value) { - return i; - } - } - return 0; -} -} // namespace detail - template struct SelectedRowsAdd { void operator()(const platform::DeviceContext& context, diff --git a/paddle/operators/math/math_function_test.cc b/paddle/operators/math/math_function_test.cc index e3186171d1..fe0d1981fa 100644 --- a/paddle/operators/math/math_function_test.cc +++ b/paddle/operators/math/math_function_test.cc @@ -1,5 +1,4 @@ #include "paddle/operators/math/math_function.h" -#include "glog/logging.h" #include "gtest/gtest.h" #ifdef PADDLE_WITH_CUDA From 29819ba7646d5a44d927347d49ad6d6ab36039c9 Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Sat, 14 Oct 2017 14:29:44 -0700 Subject: [PATCH 17/63] Fix unittest --- paddle/framework/var_type_inference_test.cc | 37 +++++++++++---------- 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/paddle/framework/var_type_inference_test.cc b/paddle/framework/var_type_inference_test.cc index e3f4893f1a..97b8c64748 100644 --- a/paddle/framework/var_type_inference_test.cc +++ b/paddle/framework/var_type_inference_test.cc @@ -35,14 +35,17 @@ class SumOpVarTypeInference : public VarTypeInference { public: void operator()(const OpDescBind &op_desc, BlockDescBind *block) const override { - auto default_var_type = VarDesc::LOD_TENSOR; - for (auto &in_var_name : op_desc.Input("X")) { - auto in_var_type = block->Var(in_var_name)->GetType(); - if (in_var_type != default_var_type) { - default_var_type = in_var_type; - break; - } + auto &inputs = op_desc.Input("X"); + auto default_var_type = VarDesc::SELECTED_ROWS; + + bool any_input_is_lod_tensor = std::any_of( + inputs.begin(), inputs.end(), [block](const std::string &name) { + return block->Var(name)->GetType() == VarDesc::LOD_TENSOR; + }); + if (any_input_is_lod_tensor) { + default_var_type = VarDesc::LOD_TENSOR; } + auto out_var_name = op_desc.Output("Out").front(); block->Var(out_var_name)->SetType(default_var_type); } @@ -65,20 +68,18 @@ TEST(InferVarType, sum_op) { op->SetInput("X", {"test_a", "test_b", "test_c"}); op->SetOutput("Out", {"test_out"}); - prog.Block(0)->NewVar("test_a")->SetType(VarDesc_VarType_LOD_TENSOR); - prog.Block(0)->NewVar("test_b")->SetType(VarDesc_VarType_LOD_TENSOR); - prog.Block(0)->NewVar("test_c")->SetType(VarDesc_VarType_LOD_TENSOR); + prog.Block(0)->NewVar("test_a")->SetType(VarDesc::SELECTED_ROWS); + prog.Block(0)->NewVar("test_b")->SetType(VarDesc::SELECTED_ROWS); + prog.Block(0)->NewVar("test_c")->SetType(VarDesc::SELECTED_ROWS); prog.Block(0)->NewVar("test_out"); op->InferVarType(prog.Block(0)); - ASSERT_EQ(VarDesc_VarType_LOD_TENSOR, - prog.Block(0)->Var("test_out")->GetType()); + ASSERT_EQ(VarDesc::SELECTED_ROWS, prog.Block(0)->Var("test_out")->GetType()); - prog.Block(0)->Var("test_b")->SetType(VarDesc_VarType_SELECTED_ROWS); + prog.Block(0)->Var("test_b")->SetType(VarDesc::LOD_TENSOR); op->InferVarType(prog.Block(0)); - ASSERT_EQ(VarDesc_VarType_SELECTED_ROWS, - prog.Block(0)->Var("test_out")->GetType()); + ASSERT_EQ(VarDesc::LOD_TENSOR, prog.Block(0)->Var("test_out")->GetType()); } TEST(InferVarType, sum_op_without_infer_var_type) { @@ -88,9 +89,9 @@ TEST(InferVarType, sum_op_without_infer_var_type) { op->SetInput("X", {"test2_a", "test2_b", "test2_c"}); op->SetOutput("Out", {"test2_out"}); - prog.Block(0)->NewVar("test2_a")->SetType(VarDesc_VarType_LOD_TENSOR); - prog.Block(0)->NewVar("test2_b")->SetType(VarDesc_VarType_SELECTED_ROWS); - prog.Block(0)->NewVar("test2_c")->SetType(VarDesc_VarType_LOD_TENSOR); + prog.Block(0)->NewVar("test2_a")->SetType(VarDesc::SELECTED_ROWS); + prog.Block(0)->NewVar("test2_b")->SetType(VarDesc::SELECTED_ROWS); + prog.Block(0)->NewVar("test2_c")->SetType(VarDesc::SELECTED_ROWS); prog.Block(0)->NewVar("test2_out"); op->InferVarType(prog.Block(0)); From f59a7c1d36d2e930c48e118ad90f35a541e00223 Mon Sep 17 00:00:00 2001 From: qijun Date: Sat, 14 Oct 2017 14:57:07 -0700 Subject: [PATCH 18/63] add gpu functor for SelectedRows --- paddle/framework/lod_tensor.h | 3 - paddle/framework/selected_rows.h | 7 +- paddle/framework/selected_rows_test.cc | 2 +- paddle/framework/type_defs.h | 6 +- paddle/operators/math/CMakeLists.txt | 2 +- paddle/operators/math/math_function.cc | 17 +- paddle/operators/math/math_function.cu | 107 ++++++++ paddle/operators/math/math_function_test.cc | 179 ------------- paddle/operators/math/math_function_test.cu | 277 ++++++++++++++++++++ 9 files changed, 405 insertions(+), 195 deletions(-) create mode 100644 paddle/operators/math/math_function_test.cu diff --git a/paddle/framework/lod_tensor.h b/paddle/framework/lod_tensor.h index 4db36ee766..ee040a9144 100644 --- a/paddle/framework/lod_tensor.h +++ b/paddle/framework/lod_tensor.h @@ -1,11 +1,8 @@ /* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. - Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 - Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. diff --git a/paddle/framework/selected_rows.h b/paddle/framework/selected_rows.h index ddc6dec194..cd90781371 100644 --- a/paddle/framework/selected_rows.h +++ b/paddle/framework/selected_rows.h @@ -10,6 +10,7 @@ See the License for the specific language governing permissions and limitations under the License. */ #pragma once +#include "paddle/framework/lod_tensor.h" #include "paddle/framework/tensor.h" namespace paddle { @@ -34,9 +35,9 @@ class SelectedRows { void set_height(int64_t height) { height_ = height; } - const std::vector& rows() const { return rows_; } + const Vector& rows() const { return rows_; } - void set_rows(const std::vector& rows) { rows_ = rows; } + void set_rows(const Vector& rows) { rows_ = rows; } DDim GetCompleteDims() const { std::vector dims = vectorize(value_->dims()); @@ -48,7 +49,7 @@ class SelectedRows { // Notice: rows can be duplicate. We can have {0, 4, 7, 0, 5, 7, 9} here. // SelectedRows are simplely concated when adding together. Until a // SelectedRows add a Tensor, will the duplicate rows be handled. - std::vector rows_; + Vector rows_; std::unique_ptr value_{nullptr}; int64_t height_; }; diff --git a/paddle/framework/selected_rows_test.cc b/paddle/framework/selected_rows_test.cc index 4ee13a65d7..055b867600 100644 --- a/paddle/framework/selected_rows_test.cc +++ b/paddle/framework/selected_rows_test.cc @@ -18,7 +18,7 @@ namespace framework { class SelectedRowsTester : public ::testing::Test { public: virtual void SetUp() override { - std::vector rows{0, 4, 7}; + Vector rows{0, 4, 7}; int64_t height = 10; int64_t row_numel = 100; selected_rows_.reset(new SelectedRows(rows, height)); diff --git a/paddle/framework/type_defs.h b/paddle/framework/type_defs.h index 7e1b79c97b..0c0a72de31 100644 --- a/paddle/framework/type_defs.h +++ b/paddle/framework/type_defs.h @@ -1,11 +1,8 @@ /* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. - Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 - Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. @@ -37,7 +34,8 @@ using OpCreator = std::function; using GradOpMakerFN = std::function>( - const OpDescBind&, const std::unordered_set& /*no_grad_set*/)>; + const OpDescBind&, const std::unordered_set& /*no_grad_set*/, + std::unordered_map* /*grad_to_var*/)>; } // namespace framework } // namespace paddle diff --git a/paddle/operators/math/CMakeLists.txt b/paddle/operators/math/CMakeLists.txt index 1a2f623ce7..a7f275bae5 100644 --- a/paddle/operators/math/CMakeLists.txt +++ b/paddle/operators/math/CMakeLists.txt @@ -1,6 +1,6 @@ if(WITH_GPU) nv_library(math_function SRCS math_function.cc math_function.cu im2col.cc im2col.cu DEPS cblas device_context operator) - nv_test(math_function_test SRCS math_function_test.cc DEPS math_function tensor) + nv_test(math_function_gpu_test SRCS math_function_test.cu DEPS math_function tensor) nv_library(softmax SRCS softmax.cc softmax.cu DEPS operator) nv_library(cross_entropy SRCS cross_entropy.cc cross_entropy.cu DEPS operator) nv_library(pooling SRCS pooling.cc pooling.cu DEPS device_context) diff --git a/paddle/operators/math/math_function.cc b/paddle/operators/math/math_function.cc index ddb904aa46..a1faafb7c4 100644 --- a/paddle/operators/math/math_function.cc +++ b/paddle/operators/math/math_function.cc @@ -162,15 +162,24 @@ struct SelectedRowsAdd { PADDLE_ENFORCE_EQ(in1_row_numel, in2_value.numel() / in2_rows.size()); PADDLE_ENFORCE_EQ(in1_row_numel, out_value->numel() / out_rows.size()); - auto* out_data = out_value->data(); + auto in1_place = input1.place(); + PADDLE_ENFORCE(platform::is_cpu_place(in1_place)); + auto in2_place = input2.place(); + PADDLE_ENFORCE(platform::is_cpu_place(in2_place)); + auto out_place = context.GetPlace(); + PADDLE_ENFORCE(platform::is_cpu_place(out_place)); + auto* out_data = out_value->data(); auto* in1_data = in1_value.data(); - memory::Copy(platform::CPUPlace(), out_data, platform::CPUPlace(), in1_data, + memory::Copy(boost::get(out_place), out_data, + boost::get(in1_place), in1_data, in1_value.numel() * sizeof(T)); auto* in2_data = in2_value.data(); - memory::Copy(platform::CPUPlace(), out_data + in1_value.numel(), - platform::CPUPlace(), in2_data, in2_value.numel() * sizeof(T)); + memory::Copy(boost::get(out_place), + out_data + in1_value.numel(), + boost::get(in2_place), in2_data, + in2_value.numel() * sizeof(T)); } }; diff --git a/paddle/operators/math/math_function.cu b/paddle/operators/math/math_function.cu index 649f1f352c..26bf0ec2f1 100644 --- a/paddle/operators/math/math_function.cu +++ b/paddle/operators/math/math_function.cu @@ -155,6 +155,113 @@ void matmul( matrix_b.data(), beta, matrix_out->data()); } +template +struct SelectedRowsAdd { + void operator()(const platform::DeviceContext& context, + const framework::SelectedRows& input1, + const framework::SelectedRows& input2, + framework::SelectedRows* output) { + auto in1_height = input1.height(); + PADDLE_ENFORCE_EQ(in1_height, input2.height()); + output->set_height(in1_height); + + auto& in1_rows = input1.rows(); + auto& in2_rows = input2.rows(); + std::vector out_rows; + out_rows.reserve(in1_rows.size() + in2_rows.size()); + + // concat rows + out_rows.insert(out_rows.end(), in1_rows.begin(), in1_rows.end()); + out_rows.insert(out_rows.end(), in2_rows.begin(), in2_rows.end()); + output->set_rows(out_rows); + + auto* out_value = output->mutable_value(); + auto& in1_value = input1.value(); + auto& in2_value = input2.value(); + + auto in1_row_numel = in1_value.numel() / in1_rows.size(); + PADDLE_ENFORCE_EQ(in1_row_numel, in2_value.numel() / in2_rows.size()); + PADDLE_ENFORCE_EQ(in1_row_numel, out_value->numel() / out_rows.size()); + + auto* out_data = out_value->data(); + auto* in1_data = in1_value.data(); + + auto in1_place = input1.place(); + PADDLE_ENFORCE(platform::is_gpu_place(in1_place)); + auto in2_place = input2.place(); + PADDLE_ENFORCE(platform::is_gpu_place(in2_place)); + auto out_place = context.GetPlace(); + PADDLE_ENFORCE(platform::is_gpu_place(out_place)) + + memory::Copy( + boost::get(out_place), out_data, + boost::get(in1_place), in1_data, + in1_value.numel() * sizeof(T), + reinterpret_cast(context).stream()); + + auto* in2_data = in2_value.data(); + memory::Copy( + boost::get(out_place), out_data + in1_value.numel(), + boost::get(in2_place), in2_data, + in2_value.numel() * sizeof(T), + reinterpret_cast(context).stream()); + } +}; + +template struct SelectedRowsAdd; + +namespace { +template +__global__ void SelectedRowsAddTensorKernel(T* selected_rows, int64_t* rows, + T* tensor_in, T* tensor_out, + const int64_t row_numel) { + const ty = blockIdx.y; + int tid = threadIdx.x; + + selected_rows += ty * row_numel; + tensor_in += rows[ty] * row_numel; + tensor_out += rows[ty] * row_numel; + + for (int index = tid; index < row_numel; index += block_size) { + tensor_out[index] = tensor_in[index] + selected_rows[index]; + } +} +} + +template +struct SelectedRowsAddTensor { + void operator()(const platform::DeviceContext& context, + const framework::SelectedRows& input1, + const framework::Tensor& input2, framework::Tensor* output) { + auto in1_height = input1.height(); + auto in2_dims = input2.dims(); + auto out_dims = output->dims(); + PADDLE_ENFORCE_EQ(in1_height, in2_dims[0]); + PADDLE_ENFORCE_EQ(in1_height, out_dims[0]); + + auto& in1_value = input1.value(); + auto& in1_rows = input1.rows(); + + int64_t in1_row_numel = in1_value.numel() / in1_rows.size(); + PADDLE_ENFORCE_EQ(in1_row_numel, input2.numel() / in1_height); + PADDLE_ENFORCE_EQ(in1_row_numel, output->numel() / in1_height); + + auto* in1_data = in1_value.data(); + auto* in2_data = input2.data(); + auto* out_data = output->data(); + + const int block_size = 256; + dim3 threads(block_size, 1); + dim3 grid(1, in1_height); + SelectedRowsAddTensorKernel<<< + grid, threads, 0, + reinterpret_cast(ctx).stream()>>>( + in1_data, in1_rows.data(), in2_data, out_data, in1_row_numel); + } +}; + +template struct SelectedRowsAddTensor; + } // namespace math } // namespace operators } // namespace paddle diff --git a/paddle/operators/math/math_function_test.cc b/paddle/operators/math/math_function_test.cc index fe0d1981fa..33c561f6c6 100644 --- a/paddle/operators/math/math_function_test.cc +++ b/paddle/operators/math/math_function_test.cc @@ -1,185 +1,6 @@ #include "paddle/operators/math/math_function.h" #include "gtest/gtest.h" -#ifdef PADDLE_WITH_CUDA -TEST(math_function, notrans_mul_trans) { - paddle::framework::Tensor input1; - paddle::framework::Tensor input1_gpu; - paddle::framework::Tensor input2_gpu; - paddle::framework::Tensor out_gpu; - paddle::framework::Tensor out; - - auto* cpu_place = new paddle::platform::CPUPlace(); - float* input1_ptr = input1.mutable_data({2, 3}, *cpu_place); - float arr[6] = {0, 1, 2, 3, 4, 5}; - memcpy(input1_ptr, arr, 6 * sizeof(float)); - - auto* gpu_place = new paddle::platform::GPUPlace(0); - paddle::platform::CUDADeviceContext context(*gpu_place); - - input1_gpu.CopyFrom(input1, *gpu_place, context); - input2_gpu.CopyFrom(input1, *gpu_place, context); - - out_gpu.mutable_data({2, 2}, *gpu_place); - - paddle::operators::math::matmul( - context, input1_gpu, false, input2_gpu, true, 1, &out_gpu, 0); - - out.CopyFrom(out_gpu, *cpu_place, context); - - float* out_ptr = out.data(); - context.Wait(); - EXPECT_EQ(out_ptr[0], 5); - EXPECT_EQ(out_ptr[1], 14); - EXPECT_EQ(out_ptr[2], 14); - EXPECT_EQ(out_ptr[3], 50); - delete gpu_place; -} - -TEST(math_function, trans_mul_notrans) { - paddle::framework::Tensor input1; - paddle::framework::Tensor input1_gpu; - paddle::framework::Tensor input2_gpu; - paddle::framework::Tensor out_gpu; - paddle::framework::Tensor out; - - auto* cpu_place = new paddle::platform::CPUPlace(); - float* input1_ptr = input1.mutable_data({2, 3}, *cpu_place); - float arr[6] = {0, 1, 2, 3, 4, 5}; - memcpy(input1_ptr, arr, 6 * sizeof(float)); - - auto* gpu_place = new paddle::platform::GPUPlace(0); - paddle::platform::CUDADeviceContext context(*gpu_place); - - input1_gpu.CopyFrom(input1, *gpu_place, context); - input2_gpu.CopyFrom(input1, *gpu_place, context); - - out_gpu.mutable_data({3, 3}, *gpu_place); - - paddle::operators::math::matmul( - context, input1_gpu, true, input2_gpu, false, 1, &out_gpu, 0); - - out.CopyFrom(out_gpu, *cpu_place, context); - - float* out_ptr = out.data(); - context.Wait(); - EXPECT_EQ(out_ptr[0], 9); - EXPECT_EQ(out_ptr[1], 12); - EXPECT_EQ(out_ptr[2], 15); - EXPECT_EQ(out_ptr[3], 12); - EXPECT_EQ(out_ptr[4], 17); - EXPECT_EQ(out_ptr[5], 22); - EXPECT_EQ(out_ptr[6], 15); - EXPECT_EQ(out_ptr[7], 22); - EXPECT_EQ(out_ptr[8], 29); - delete gpu_place; -} - -TEST(math_function, gemm_notrans_cublas) { - paddle::framework::Tensor input1; - paddle::framework::Tensor input2; - paddle::framework::Tensor input3; - paddle::framework::Tensor input1_gpu; - paddle::framework::Tensor input2_gpu; - paddle::framework::Tensor input3_gpu; - - int m = 2; - int n = 3; - int k = 3; - auto* cpu_place = new paddle::platform::CPUPlace(); - float* input1_ptr = input1.mutable_data({2, 3}, *cpu_place); - float arr1[6] = {0, 1, 2, 3, 4, 5}; - memcpy(input1_ptr, arr1, 6 * sizeof(float)); - float* input2_ptr = input2.mutable_data({3, 4}, *cpu_place); - float arr2[12] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}; - memcpy(input2_ptr, arr2, 12 * sizeof(float)); - float* input3_ptr = input3.mutable_data({2, 4}, *cpu_place); - float arr3[8] = {0, 1, 2, 3, 4, 5, 6, 7}; - memcpy(input3_ptr, arr3, 8 * sizeof(float)); - - auto* gpu_place = new paddle::platform::GPUPlace(0); - paddle::platform::CUDADeviceContext context(*gpu_place); - - input1_gpu.CopyFrom(input1, *gpu_place, context); - input2_gpu.CopyFrom(input2, *gpu_place, context); - input3_gpu.CopyFrom(input3, *gpu_place, context); - float* a = input1_gpu.data(); - float* b = input2_gpu.data(); - float* c = input3_gpu.mutable_data(*gpu_place); - - paddle::operators::math::gemm( - context, false, false, m, n, k, 1, a, 3, b + 1, 4, 1, c + 1, 4); - - input3.CopyFrom(input3_gpu, *cpu_place, context); - - // numpy code: - // a = np.arange(6).reshape(2, 3) - // b = np.arange(12).reshape(3, 4)[:, 1:] - // c = np.arange(8).reshape(2, 4)[:, 1:] - // out = np.arange(8).reshape(2, 4) - // out[:, 1:] = np.dot(a, b) + c - context.Wait(); - EXPECT_EQ(input3_ptr[0], 0); - EXPECT_EQ(input3_ptr[1], 24); - EXPECT_EQ(input3_ptr[2], 28); - EXPECT_EQ(input3_ptr[3], 32); - EXPECT_EQ(input3_ptr[4], 4); - EXPECT_EQ(input3_ptr[5], 73); - EXPECT_EQ(input3_ptr[6], 86); - EXPECT_EQ(input3_ptr[7], 99); - delete gpu_place; -} - -TEST(math_function, gemm_trans_cublas) { - paddle::framework::Tensor input1; - paddle::framework::Tensor input2; - paddle::framework::Tensor input3; - paddle::framework::Tensor input1_gpu; - paddle::framework::Tensor input2_gpu; - paddle::framework::Tensor input3_gpu; - - int m = 2; - int n = 3; - int k = 3; - auto* cpu_place = new paddle::platform::CPUPlace(); - float* input1_ptr = input1.mutable_data({2, 3}, *cpu_place); - float arr1[6] = {0, 1, 2, 3, 4, 5}; - memcpy(input1_ptr, arr1, 6 * sizeof(float)); - float* input2_ptr = input2.mutable_data({4, 3}, *cpu_place); - float arr2[12] = {0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11}; - memcpy(input2_ptr, arr2, 12 * sizeof(float)); - float* input3_ptr = input3.mutable_data({2, 4}, *cpu_place); - float arr3[8] = {0, 1, 2, 3, 4, 5, 6, 7}; - memcpy(input3_ptr, arr3, 8 * sizeof(float)); - - auto* gpu_place = new paddle::platform::GPUPlace(0); - paddle::platform::CUDADeviceContext context(*gpu_place); - - input1_gpu.CopyFrom(input1, *gpu_place, context); - input2_gpu.CopyFrom(input2, *gpu_place, context); - input3_gpu.CopyFrom(input3, *gpu_place, context); - float* a = input1_gpu.data(); - float* b = input2_gpu.data(); - float* c = input3_gpu.mutable_data(*gpu_place); - - paddle::operators::math::gemm( - context, false, true, m, n, k, 1, a, 3, b + 3, 3, 1, c + 1, 4); - - input3.CopyFrom(input3_gpu, *cpu_place, context); - context.Wait(); - - EXPECT_EQ(input3_ptr[0], 0); - EXPECT_EQ(input3_ptr[1], 24); - EXPECT_EQ(input3_ptr[2], 28); - EXPECT_EQ(input3_ptr[3], 32); - EXPECT_EQ(input3_ptr[4], 4); - EXPECT_EQ(input3_ptr[5], 73); - EXPECT_EQ(input3_ptr[6], 86); - EXPECT_EQ(input3_ptr[7], 99); - delete gpu_place; -} -#endif - TEST(math_function, gemm_notrans_cblas) { paddle::framework::Tensor input1; paddle::framework::Tensor input2; diff --git a/paddle/operators/math/math_function_test.cu b/paddle/operators/math/math_function_test.cu new file mode 100644 index 0000000000..e691078bb6 --- /dev/null +++ b/paddle/operators/math/math_function_test.cu @@ -0,0 +1,277 @@ +#include "gtest/gtest.h" +#include "paddle/operators/math/math_function.h" + +TEST(math_function, notrans_mul_trans) { + paddle::framework::Tensor input1; + paddle::framework::Tensor input1_gpu; + paddle::framework::Tensor input2_gpu; + paddle::framework::Tensor out_gpu; + paddle::framework::Tensor out; + + auto* cpu_place = new paddle::platform::CPUPlace(); + float* input1_ptr = input1.mutable_data({2, 3}, *cpu_place); + float arr[6] = {0, 1, 2, 3, 4, 5}; + memcpy(input1_ptr, arr, 6 * sizeof(float)); + + auto* gpu_place = new paddle::platform::GPUPlace(0); + paddle::platform::CUDADeviceContext context(*gpu_place); + + input1_gpu.CopyFrom(input1, *gpu_place, context); + input2_gpu.CopyFrom(input1, *gpu_place, context); + + out_gpu.mutable_data({2, 2}, *gpu_place); + + paddle::operators::math::matmul( + context, input1_gpu, false, input2_gpu, true, 1, &out_gpu, 0); + + out.CopyFrom(out_gpu, *cpu_place, context); + + float* out_ptr = out.data(); + context.Wait(); + EXPECT_EQ(out_ptr[0], 5); + EXPECT_EQ(out_ptr[1], 14); + EXPECT_EQ(out_ptr[2], 14); + EXPECT_EQ(out_ptr[3], 50); + delete gpu_place; +} + +TEST(math_function, trans_mul_notrans) { + paddle::framework::Tensor input1; + paddle::framework::Tensor input1_gpu; + paddle::framework::Tensor input2_gpu; + paddle::framework::Tensor out_gpu; + paddle::framework::Tensor out; + + auto* cpu_place = new paddle::platform::CPUPlace(); + float* input1_ptr = input1.mutable_data({2, 3}, *cpu_place); + float arr[6] = {0, 1, 2, 3, 4, 5}; + memcpy(input1_ptr, arr, 6 * sizeof(float)); + + auto* gpu_place = new paddle::platform::GPUPlace(0); + paddle::platform::CUDADeviceContext context(*gpu_place); + + input1_gpu.CopyFrom(input1, *gpu_place, context); + input2_gpu.CopyFrom(input1, *gpu_place, context); + + out_gpu.mutable_data({3, 3}, *gpu_place); + + paddle::operators::math::matmul( + context, input1_gpu, true, input2_gpu, false, 1, &out_gpu, 0); + + out.CopyFrom(out_gpu, *cpu_place, context); + + float* out_ptr = out.data(); + context.Wait(); + EXPECT_EQ(out_ptr[0], 9); + EXPECT_EQ(out_ptr[1], 12); + EXPECT_EQ(out_ptr[2], 15); + EXPECT_EQ(out_ptr[3], 12); + EXPECT_EQ(out_ptr[4], 17); + EXPECT_EQ(out_ptr[5], 22); + EXPECT_EQ(out_ptr[6], 15); + EXPECT_EQ(out_ptr[7], 22); + EXPECT_EQ(out_ptr[8], 29); + delete gpu_place; +} + +TEST(math_function, gemm_notrans_cublas) { + paddle::framework::Tensor input1; + paddle::framework::Tensor input2; + paddle::framework::Tensor input3; + paddle::framework::Tensor input1_gpu; + paddle::framework::Tensor input2_gpu; + paddle::framework::Tensor input3_gpu; + + int m = 2; + int n = 3; + int k = 3; + auto* cpu_place = new paddle::platform::CPUPlace(); + float* input1_ptr = input1.mutable_data({2, 3}, *cpu_place); + float arr1[6] = {0, 1, 2, 3, 4, 5}; + memcpy(input1_ptr, arr1, 6 * sizeof(float)); + float* input2_ptr = input2.mutable_data({3, 4}, *cpu_place); + float arr2[12] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}; + memcpy(input2_ptr, arr2, 12 * sizeof(float)); + float* input3_ptr = input3.mutable_data({2, 4}, *cpu_place); + float arr3[8] = {0, 1, 2, 3, 4, 5, 6, 7}; + memcpy(input3_ptr, arr3, 8 * sizeof(float)); + + auto* gpu_place = new paddle::platform::GPUPlace(0); + paddle::platform::CUDADeviceContext context(*gpu_place); + + input1_gpu.CopyFrom(input1, *gpu_place, context); + input2_gpu.CopyFrom(input2, *gpu_place, context); + input3_gpu.CopyFrom(input3, *gpu_place, context); + float* a = input1_gpu.data(); + float* b = input2_gpu.data(); + float* c = input3_gpu.mutable_data(*gpu_place); + + paddle::operators::math::gemm( + context, false, false, m, n, k, 1, a, 3, b + 1, 4, 1, c + 1, 4); + + input3.CopyFrom(input3_gpu, *cpu_place, context); + + // numpy code: + // a = np.arange(6).reshape(2, 3) + // b = np.arange(12).reshape(3, 4)[:, 1:] + // c = np.arange(8).reshape(2, 4)[:, 1:] + // out = np.arange(8).reshape(2, 4) + // out[:, 1:] = np.dot(a, b) + c + context.Wait(); + EXPECT_EQ(input3_ptr[0], 0); + EXPECT_EQ(input3_ptr[1], 24); + EXPECT_EQ(input3_ptr[2], 28); + EXPECT_EQ(input3_ptr[3], 32); + EXPECT_EQ(input3_ptr[4], 4); + EXPECT_EQ(input3_ptr[5], 73); + EXPECT_EQ(input3_ptr[6], 86); + EXPECT_EQ(input3_ptr[7], 99); + delete gpu_place; +} + +TEST(math_function, gemm_trans_cublas) { + paddle::framework::Tensor input1; + paddle::framework::Tensor input2; + paddle::framework::Tensor input3; + paddle::framework::Tensor input1_gpu; + paddle::framework::Tensor input2_gpu; + paddle::framework::Tensor input3_gpu; + + int m = 2; + int n = 3; + int k = 3; + auto* cpu_place = new paddle::platform::CPUPlace(); + float* input1_ptr = input1.mutable_data({2, 3}, *cpu_place); + float arr1[6] = {0, 1, 2, 3, 4, 5}; + memcpy(input1_ptr, arr1, 6 * sizeof(float)); + float* input2_ptr = input2.mutable_data({4, 3}, *cpu_place); + float arr2[12] = {0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11}; + memcpy(input2_ptr, arr2, 12 * sizeof(float)); + float* input3_ptr = input3.mutable_data({2, 4}, *cpu_place); + float arr3[8] = {0, 1, 2, 3, 4, 5, 6, 7}; + memcpy(input3_ptr, arr3, 8 * sizeof(float)); + + auto* gpu_place = new paddle::platform::GPUPlace(0); + paddle::platform::CUDADeviceContext context(*gpu_place); + + input1_gpu.CopyFrom(input1, *gpu_place, context); + input2_gpu.CopyFrom(input2, *gpu_place, context); + input3_gpu.CopyFrom(input3, *gpu_place, context); + float* a = input1_gpu.data(); + float* b = input2_gpu.data(); + float* c = input3_gpu.mutable_data(*gpu_place); + + paddle::operators::math::gemm( + context, false, true, m, n, k, 1, a, 3, b + 3, 3, 1, c + 1, 4); + + input3.CopyFrom(input3_gpu, *cpu_place, context); + context.Wait(); + + EXPECT_EQ(input3_ptr[0], 0); + EXPECT_EQ(input3_ptr[1], 24); + EXPECT_EQ(input3_ptr[2], 28); + EXPECT_EQ(input3_ptr[3], 32); + EXPECT_EQ(input3_ptr[4], 4); + EXPECT_EQ(input3_ptr[5], 73); + EXPECT_EQ(input3_ptr[6], 86); + EXPECT_EQ(input3_ptr[7], 99); + delete gpu_place; +} + +TEST(math_function, selected_rows_add) { + using namespace paddle::framework; + using namespace paddle::platform; + using namespace paddle::operators::math; + + CPUPlace gpu_place(0); + CUDADeviceContext ctx(gpu_place); + SetConstant functor; + int64_t height = 10; + int64_t row_numel = 10; + + Vector rows1{0, 4, 7}; + std::unique_ptr selected_rows1{new SelectedRows(rows1, height)}; + auto* in1_value = selected_rows1->mutable_value(); + in1_value->mutable_data( + make_ddim({static_cast(rows1.size()), row_numel}), gpu_place); + functor(ctx, in1_value, 1.0); + + Vector rows2{0, 5, 7, 9}; + std::unique_ptr selected_rows2{new SelectedRows(rows2, height)}; + auto* in2_value = selected_rows2->mutable_value(); + in2_value->mutable_data( + make_ddim({static_cast(rows2.size()), row_numel}), gpu_place); + functor(ctx, in2_value, 2.0); + + std::unique_ptr output{new SelectedRows()}; + auto* out_value = output->mutable_value(); + + // simplely concat two SelectedRows + out_value->mutable_data(make_ddim({7, 10}), gpu_place); + + SelectedRowsAdd add_functor; + add_functor(ctx, *selected_rows1, *selected_rows2, output.get()); + + auto out_height = output->height(); + EXPECT_EQ(out_height, height); + + auto& out_rows = output->rows(); + + // input1 rows + EXPECT_EQ(out_rows[0], 0); + EXPECT_EQ(out_rows[1], 4); + EXPECT_EQ(out_rows[2], 7); + // input2 rows + EXPECT_EQ(out_rows[3], 0); + EXPECT_EQ(out_rows[4], 5); + EXPECT_EQ(out_rows[5], 7); + EXPECT_EQ(out_rows[6], 9); + + Tensor out_cpu; + out_cpu.CopyFrom(*out_value, platform::CPUPlace(), ctx); + ctx.Wait(); + + auto* out_cpu_data = out_cpu.data(); + // input1 value + EXPECT_EQ(out_cpu_data[0 * row_numel + 0], 1.0); + EXPECT_EQ(out_cpu_data[0 * row_numel + 8], 1.0); + EXPECT_EQ(out_cpu_data[1 * row_numel + 1], 1.0); + EXPECT_EQ(out_cpu_data[2 * row_numel + 6], 1.0); + // input2 value + EXPECT_EQ(out_cpu_data[3 * row_numel + 3], 2.0); + EXPECT_EQ(out_cpu_data[3 * row_numel + 8], 2.0); + EXPECT_EQ(out_cpu_data[4 * row_numel + 4], 2.0); + EXPECT_EQ(out_cpu_data[5 * row_numel + 7], 2.0); + EXPECT_EQ(out_cpu_data[6 * row_numel + 9], 2.0); + + std::unique_ptr tensor1{new Tensor()}; + tensor1->mutable_data(make_ddim({height, row_numel}), gpu_place); + SetConstant constant_functor; + constant_functor(ctx, tensor1.get(), 3.0); + + std::unique_ptr tensor2{new Tensor()}; + tensor2->mutable_data(make_ddim({height, row_numel}), gpu_place); + + SelectedRowsAddTensor add_tensor_functor; + add_tensor_functor(ctx, *output, *tensor1, tensor2.get()); + + Tensor tensor2_cpu; + tensor2_cpu.CopyFrom(*tensor2, platform::CPUPlace(), ctx); + ctx.Wait(); + + auto* tensor2_cpu_data = tensor2_cpu->data(); + // row0: 1.0 + 2.0 + 3.0 + EXPECT_EQ(tensor2_cpu_data[0 * row_numel + 0], 6.0); + // row1: 3.0 + EXPECT_EQ(tensor2_cpu_data[1 * row_numel + 1], 3.0); + // row4 : 1.0 + 3.0 + EXPECT_EQ(tensor2_cpu_data[4 * row_numel + 6], 4.0); + // row5: 2.0 + 3.0 + EXPECT_EQ(tensor2_cpu_data[5 * row_numel + 7], 5.0); + // row6: 3.0 + EXPECT_EQ(tensor2_cpu_data[6 * row_numel + 1], 3.0); + // row7: 1.0 + 2.0 + 3.0 + EXPECT_EQ(tensor2_cpu_data[7 * row_numel + 3], 6.0); + // row9: 2.0 + 3.0 + EXPECT_EQ(tensor2_cpu_data[9 * row_numel + 6], 5.0); +} From 7ef568e8935768e7ba9ebd4b82bc42b1fcf6ade0 Mon Sep 17 00:00:00 2001 From: qijun Date: Sat, 14 Oct 2017 23:13:04 +0000 Subject: [PATCH 19/63] fix gpu unittest error --- paddle/operators/math/CMakeLists.txt | 4 +-- paddle/operators/math/math_function.cu | 40 ++++++++++++++------- paddle/operators/math/math_function_test.cu | 13 +++---- 3 files changed, 36 insertions(+), 21 deletions(-) diff --git a/paddle/operators/math/CMakeLists.txt b/paddle/operators/math/CMakeLists.txt index a7f275bae5..77a3603eb6 100644 --- a/paddle/operators/math/CMakeLists.txt +++ b/paddle/operators/math/CMakeLists.txt @@ -1,18 +1,18 @@ if(WITH_GPU) nv_library(math_function SRCS math_function.cc math_function.cu im2col.cc im2col.cu DEPS cblas device_context operator) - nv_test(math_function_gpu_test SRCS math_function_test.cu DEPS math_function tensor) + nv_test(math_function_gpu_test SRCS math_function_test.cu DEPS math_function tensor selected_rows) nv_library(softmax SRCS softmax.cc softmax.cu DEPS operator) nv_library(cross_entropy SRCS cross_entropy.cc cross_entropy.cu DEPS operator) nv_library(pooling SRCS pooling.cc pooling.cu DEPS device_context) nv_library(vol2col SRCS vol2col.cc vol2col.cu DEPS device_context) else() cc_library(math_function SRCS math_function.cc im2col.cc DEPS cblas device_context operator) - cc_test(math_function_test SRCS math_function_test.cc DEPS math_function tensor) cc_library(softmax SRCS softmax.cc DEPS operator) cc_library(cross_entropy SRCS cross_entropy.cc DEPS operator) cc_library(pooling SRCS pooling.cc DEPS device_context) cc_library(vol2col SRCS vol2col.cc DEPS device_context) endif() +cc_test(math_function_test SRCS math_function_test.cc DEPS math_function tensor selected_rows) cc_test(im2col_test SRCS im2col_test.cc DEPS math_function tensor) cc_test(vol2col_test SRCS vol2col_test.cc DEPS vol2col tensor) diff --git a/paddle/operators/math/math_function.cu b/paddle/operators/math/math_function.cu index 26bf0ec2f1..d31b223b2c 100644 --- a/paddle/operators/math/math_function.cu +++ b/paddle/operators/math/math_function.cu @@ -13,6 +13,7 @@ See the License for the specific language governing permissions and limitations under the License. */ #include "paddle/operators/math/math_function.h" +#include "paddle/platform/cuda_helper.h" namespace paddle { namespace operators { @@ -191,7 +192,7 @@ struct SelectedRowsAdd { auto in2_place = input2.place(); PADDLE_ENFORCE(platform::is_gpu_place(in2_place)); auto out_place = context.GetPlace(); - PADDLE_ENFORCE(platform::is_gpu_place(out_place)) + PADDLE_ENFORCE(platform::is_gpu_place(out_place)); memory::Copy( boost::get(out_place), out_data, @@ -211,22 +212,26 @@ struct SelectedRowsAdd { template struct SelectedRowsAdd; namespace { -template -__global__ void SelectedRowsAddTensorKernel(T* selected_rows, int64_t* rows, - T* tensor_in, T* tensor_out, - const int64_t row_numel) { - const ty = blockIdx.y; +template +__global__ void SelectedRowsAddTensorKernel(const T* selected_rows, + const int64_t* rows, + T* tensor_out, + int64_t row_numel, + int block_size) { + const int ty = blockIdx.y; int tid = threadIdx.x; selected_rows += ty * row_numel; - tensor_in += rows[ty] * row_numel; tensor_out += rows[ty] * row_numel; for (int index = tid; index < row_numel; index += block_size) { - tensor_out[index] = tensor_in[index] + selected_rows[index]; + // Since index in rows of SelectedRows can be duplicate, we can not use + // tensor_out[index] += selected_rows[index]; Instead, we have to use + // AtomicAdd to avoid concurrent write error. + paddle::platform::CudaAtomicAdd(&tensor_out[index], selected_rows[index]); } } -} +} // namespace template struct SelectedRowsAddTensor { @@ -250,13 +255,22 @@ struct SelectedRowsAddTensor { auto* in2_data = input2.data(); auto* out_data = output->data(); - const int block_size = 256; + SetConstant functor; + functor(context, output, 0.0); + + int block_size = 256; dim3 threads(block_size, 1); dim3 grid(1, in1_height); - SelectedRowsAddTensorKernel<<< + SelectedRowsAddTensorKernel<<< grid, threads, 0, - reinterpret_cast(ctx).stream()>>>( - in1_data, in1_rows.data(), in2_data, out_data, in1_row_numel); + reinterpret_cast(context).stream() + >>>(in1_data, in1_rows.data(), + out_data, in1_row_numel, block_size); + + auto out_eigen = framework::EigenVector::Flatten(*output); + auto in2_eigen = framework::EigenVector::Flatten(input2); + out_eigen.device(*context.GetEigenDevice()) = + out_eigen + in2_eigen; } }; diff --git a/paddle/operators/math/math_function_test.cu b/paddle/operators/math/math_function_test.cu index e691078bb6..1acc5f66a6 100644 --- a/paddle/operators/math/math_function_test.cu +++ b/paddle/operators/math/math_function_test.cu @@ -183,20 +183,21 @@ TEST(math_function, selected_rows_add) { using namespace paddle::platform; using namespace paddle::operators::math; - CPUPlace gpu_place(0); + GPUPlace gpu_place(0); + CPUPlace cpu_place; CUDADeviceContext ctx(gpu_place); SetConstant functor; int64_t height = 10; int64_t row_numel = 10; - Vector rows1{0, 4, 7}; + std::vector rows1{0, 4, 7}; std::unique_ptr selected_rows1{new SelectedRows(rows1, height)}; auto* in1_value = selected_rows1->mutable_value(); in1_value->mutable_data( make_ddim({static_cast(rows1.size()), row_numel}), gpu_place); functor(ctx, in1_value, 1.0); - Vector rows2{0, 5, 7, 9}; + std::vector rows2{0, 5, 7, 9}; std::unique_ptr selected_rows2{new SelectedRows(rows2, height)}; auto* in2_value = selected_rows2->mutable_value(); in2_value->mutable_data( @@ -228,7 +229,7 @@ TEST(math_function, selected_rows_add) { EXPECT_EQ(out_rows[6], 9); Tensor out_cpu; - out_cpu.CopyFrom(*out_value, platform::CPUPlace(), ctx); + out_cpu.CopyFrom(*out_value, cpu_place, ctx); ctx.Wait(); auto* out_cpu_data = out_cpu.data(); @@ -256,10 +257,10 @@ TEST(math_function, selected_rows_add) { add_tensor_functor(ctx, *output, *tensor1, tensor2.get()); Tensor tensor2_cpu; - tensor2_cpu.CopyFrom(*tensor2, platform::CPUPlace(), ctx); + tensor2_cpu.CopyFrom(*tensor2, cpu_place, ctx); ctx.Wait(); - auto* tensor2_cpu_data = tensor2_cpu->data(); + auto* tensor2_cpu_data = tensor2_cpu.data(); // row0: 1.0 + 2.0 + 3.0 EXPECT_EQ(tensor2_cpu_data[0 * row_numel + 0], 6.0); // row1: 3.0 From df2d1769fd530fa3a57b92e50819d341768a7e80 Mon Sep 17 00:00:00 2001 From: qijun Date: Sat, 14 Oct 2017 16:21:26 -0700 Subject: [PATCH 20/63] fix code style --- paddle/framework/lod_tensor.h | 3 +++ paddle/framework/type_defs.h | 3 +++ paddle/operators/math/math_function.cu | 16 +++++++--------- 3 files changed, 13 insertions(+), 9 deletions(-) diff --git a/paddle/framework/lod_tensor.h b/paddle/framework/lod_tensor.h index ee040a9144..4db36ee766 100644 --- a/paddle/framework/lod_tensor.h +++ b/paddle/framework/lod_tensor.h @@ -1,8 +1,11 @@ /* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. diff --git a/paddle/framework/type_defs.h b/paddle/framework/type_defs.h index 0c0a72de31..0d1564a751 100644 --- a/paddle/framework/type_defs.h +++ b/paddle/framework/type_defs.h @@ -1,8 +1,11 @@ /* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. diff --git a/paddle/operators/math/math_function.cu b/paddle/operators/math/math_function.cu index d31b223b2c..fc16d1b0a7 100644 --- a/paddle/operators/math/math_function.cu +++ b/paddle/operators/math/math_function.cu @@ -214,10 +214,8 @@ template struct SelectedRowsAdd; namespace { template __global__ void SelectedRowsAddTensorKernel(const T* selected_rows, - const int64_t* rows, - T* tensor_out, - int64_t row_numel, - int block_size) { + const int64_t* rows, T* tensor_out, + int64_t row_numel, int block_size) { const int ty = blockIdx.y; int tid = threadIdx.x; @@ -261,11 +259,11 @@ struct SelectedRowsAddTensor { int block_size = 256; dim3 threads(block_size, 1); dim3 grid(1, in1_height); - SelectedRowsAddTensorKernel<<< - grid, threads, 0, - reinterpret_cast(context).stream() - >>>(in1_data, in1_rows.data(), - out_data, in1_row_numel, block_size); + SelectedRowsAddTensorKernel< + T><<(context) + .stream()>>>(in1_data, in1_rows.data(), out_data, + in1_row_numel, block_size); auto out_eigen = framework::EigenVector::Flatten(*output); auto in2_eigen = framework::EigenVector::Flatten(input2); From 89758adb83cea198ee6d2d31bc8e1d9bad5e827e Mon Sep 17 00:00:00 2001 From: qijun Date: Sat, 14 Oct 2017 16:21:26 -0700 Subject: [PATCH 21/63] fix code style --- paddle/framework/lod_tensor.h | 3 +++ paddle/framework/selected_rows_test.cc | 2 +- paddle/framework/type_defs.h | 3 +++ paddle/operators/math/math_function.cu | 16 +++++++--------- 4 files changed, 14 insertions(+), 10 deletions(-) diff --git a/paddle/framework/lod_tensor.h b/paddle/framework/lod_tensor.h index ee040a9144..4db36ee766 100644 --- a/paddle/framework/lod_tensor.h +++ b/paddle/framework/lod_tensor.h @@ -1,8 +1,11 @@ /* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. diff --git a/paddle/framework/selected_rows_test.cc b/paddle/framework/selected_rows_test.cc index 055b867600..4ee13a65d7 100644 --- a/paddle/framework/selected_rows_test.cc +++ b/paddle/framework/selected_rows_test.cc @@ -18,7 +18,7 @@ namespace framework { class SelectedRowsTester : public ::testing::Test { public: virtual void SetUp() override { - Vector rows{0, 4, 7}; + std::vector rows{0, 4, 7}; int64_t height = 10; int64_t row_numel = 100; selected_rows_.reset(new SelectedRows(rows, height)); diff --git a/paddle/framework/type_defs.h b/paddle/framework/type_defs.h index 0c0a72de31..0d1564a751 100644 --- a/paddle/framework/type_defs.h +++ b/paddle/framework/type_defs.h @@ -1,8 +1,11 @@ /* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. diff --git a/paddle/operators/math/math_function.cu b/paddle/operators/math/math_function.cu index d31b223b2c..fc16d1b0a7 100644 --- a/paddle/operators/math/math_function.cu +++ b/paddle/operators/math/math_function.cu @@ -214,10 +214,8 @@ template struct SelectedRowsAdd; namespace { template __global__ void SelectedRowsAddTensorKernel(const T* selected_rows, - const int64_t* rows, - T* tensor_out, - int64_t row_numel, - int block_size) { + const int64_t* rows, T* tensor_out, + int64_t row_numel, int block_size) { const int ty = blockIdx.y; int tid = threadIdx.x; @@ -261,11 +259,11 @@ struct SelectedRowsAddTensor { int block_size = 256; dim3 threads(block_size, 1); dim3 grid(1, in1_height); - SelectedRowsAddTensorKernel<<< - grid, threads, 0, - reinterpret_cast(context).stream() - >>>(in1_data, in1_rows.data(), - out_data, in1_row_numel, block_size); + SelectedRowsAddTensorKernel< + T><<(context) + .stream()>>>(in1_data, in1_rows.data(), out_data, + in1_row_numel, block_size); auto out_eigen = framework::EigenVector::Flatten(*output); auto in2_eigen = framework::EigenVector::Flatten(input2); From 7c0facd1958faa2e48d5764b3274459124e51c53 Mon Sep 17 00:00:00 2001 From: qijun Date: Sun, 15 Oct 2017 22:01:51 -0700 Subject: [PATCH 22/63] init --- paddle/framework/scope.h | 62 +++++++++++++++++++ paddle/pybind/pybind.cc | 6 ++ .../framework/tests/test_feed_fetch_method.py | 50 +++++++++++++++ 3 files changed, 118 insertions(+) create mode 100644 python/paddle/v2/framework/tests/test_feed_fetch_method.py diff --git a/paddle/framework/scope.h b/paddle/framework/scope.h index a7fce3514b..7376245b53 100644 --- a/paddle/framework/scope.h +++ b/paddle/framework/scope.h @@ -18,8 +18,10 @@ limitations under the License. */ #include #include +#include "paddle/framework/lod_tensor.h" #include "paddle/framework/variable.h" #include "paddle/platform/macros.h" +#include "paddle/platform/place.h" namespace paddle { namespace framework { @@ -75,5 +77,65 @@ class Scope { framework::Scope& GetGlobalScope(); +// template +// void SetFeedVariable(const std::vector& input, const Lod& lod, +// const std::vector& dims, +// const std::string& var_name, size_t index) { +// Variable* g_feed_value = GetGlobalScope().Var("var_name"); +// // feed variable holds vector +// auto& feed_inputs = +// *(g_feed_value->GetMutable< +// std::vector>()); +// if (index >= feed_inputs.size()) { +// feed_inputs.resize(index); +// } +// // copy tensor +// T* dst = feed_inputs[index].mutable_data(make_ddim(dims), +// platform::CPUPlace()); +// memcpy(dst, inputs[i].data(), inputs[i].size() * sizeof(T)); +// // copy lod +// feed_inputs[index].set_lod(lod); +// } + +template +void SetFeedVariable(const LoDTensor& input, const std::string& var_name, + size_t index) { + std::cout << "into SetFeedVariable" << std::endl; + std::cout << var_name << std::endl; + std::cout << index << std::endl; + Variable* g_feed_value = GetGlobalScope().Var(var_name); + auto& feed_inputs = + *(g_feed_value->GetMutable>()); + if (index >= feed_inputs.size()) { + feed_inputs.resize(index + 1); + } + // shared data with input tensor + feed_inputs[index].ShareDataWith(input); + // set lod + feed_inputs[index].set_lod(input.lod()); +} + +// template +// std::vector GetFetchVariable(const std::string& var_name, size_t index) { +// Variable* g_fetch_value = GetGlobalScope().Var(var_name); +// auto& fetch_outputs = +// *(g_fetch_value->GetMutable< +// std::vector>()); +// std::vector result; +// result.resize(fetch_outputs[index].numel()); +// memcpy(result.data(), fetch_outputs[i].data(), +// fetch_outputs[i].numel() * sizeof(T)); +// } + +template +LoDTensor& GetFetchVariable(const std::string& var_name, size_t index) { + Variable* g_fetch_value = GetGlobalScope().Var(var_name); + auto& fetch_outputs = + *(g_fetch_value->GetMutable>()); + std::cout << "into GetFetchVariable" << std::endl; + PADDLE_ENFORCE_LT(index, fetch_outputs.size()); + return fetch_outputs[index]; +} + } // namespace framework } // namespace paddle diff --git a/paddle/pybind/pybind.cc b/paddle/pybind/pybind.cc index b143cb9f59..a4d14c9303 100644 --- a/paddle/pybind/pybind.cc +++ b/paddle/pybind/pybind.cc @@ -394,6 +394,12 @@ All parameter, weight, gradient are variables in Paddle. m.def("unique_integer", UniqueIntegerGenerator); m.def("is_compile_gpu", IsCompileGPU); + m.def("set_feed_variable", SetFeedVariable); + // m.def("set_feed_variable", SetFeedVariable); + // m.def("set_feed_variable", SetFeedVariable); + m.def("get_fetch_variable", GetFetchVariable); + // m.def("get_fetch_variable", GetFetchVariable); + // m.def("get_fetch_variable", GetFetchVariable); BindProgramDesc(m); BindBlockDesc(m); diff --git a/python/paddle/v2/framework/tests/test_feed_fetch_method.py b/python/paddle/v2/framework/tests/test_feed_fetch_method.py new file mode 100644 index 0000000000..cd8eeee68f --- /dev/null +++ b/python/paddle/v2/framework/tests/test_feed_fetch_method.py @@ -0,0 +1,50 @@ +import paddle.v2.framework.core as core +import unittest +import numpy as np + +# class TestFeedFetch(unittest.TestCase): +# def test_feed_fetch(self): +# place = core.CPUPlace() +# input_tensor = core.LoDTensor([[0, 2, 4]]) +# input_tensor.set_dims([4, 4, 6]) +# input_tensor.alloc_int(place) +# input_array = np.array(input_tensor) +# input_array[0, 0, 0] = 3 +# input_array[3, 3, 5] = 10 +# input_tensor.set(input_array, place) + +# core.set_feed_variable(input_tensor, "feed", 0) + +# output_tensor = core.get_fetch_variable("feed", 0) +# print type(output_tensor) + +# output_lod = output_tensor.lod() +# print type(output_lod) +# print output_lod[0] +# print output_lod[0][0] +# print output_lod[0][1] +# print output_lod[0][2] +# # self.assertEqual(0, output_lod[0][0]) +# # self.assertEqual(0, output_lod[0][0]) +# # self.assertEqual(2, output_lod[0][1]) +# # self.assertEqual(4, output_lod[0][2]) + +# # output_array = np.array(output_tensor) +# # self.assertEqual(3, output_array[0, 0, 0]) +# # self.assertEqual(10, output_array[3, 3, 5]); + + +class TestFeedFetch(unittest.TestCase): + def test_feed_fetch(self): + place = core.CPUPlace() + input_tensor = core.LoDTensor([[0, 2, 4]]) + input_tensor.set_dims([4, 4, 6]) + input_tensor.alloc_float(place) + input_array = np.array(input_tensor) + input_array[0, 0, 0] = 3 + input_array[3, 3, 5] = 10 + input_tensor.set(input_array, place) + + +if __name__ == "__main__": + unittest.main() From dee7f8132518b38327c7c1493190dbc7224c9177 Mon Sep 17 00:00:00 2001 From: tensor-tang Date: Mon, 16 Oct 2017 13:54:08 +0800 Subject: [PATCH 23/63] add finish work of mkldnn --- paddle/gserver/gradientmachines/NeuralNetwork.cpp | 15 +++++++++++++++ paddle/gserver/gradientmachines/NeuralNetwork.h | 3 +++ 2 files changed, 18 insertions(+) diff --git a/paddle/gserver/gradientmachines/NeuralNetwork.cpp b/paddle/gserver/gradientmachines/NeuralNetwork.cpp index dcf0acb5a2..dbadc352a4 100644 --- a/paddle/gserver/gradientmachines/NeuralNetwork.cpp +++ b/paddle/gserver/gradientmachines/NeuralNetwork.cpp @@ -21,6 +21,10 @@ limitations under the License. */ #include "paddle/utils/Logging.h" #include "paddle/utils/Stat.h" +#ifdef PADDLE_USE_MKLDNN +#include "paddle/gserver/layers/MKLDNNLayer.h" +#endif + #ifndef PADDLE_MOBILE_INFERENCE #include "MultiNetwork.h" #include "RecurrentGradientMachine.h" @@ -300,6 +304,17 @@ void NeuralNetwork::backward(const UpdateCallback& callback) { } } +void NeuralNetwork::finish() { +#ifdef PADDLE_USE_MKLDNN + FOR_EACH_R(layer, layers_) { + MKLDNNLayerPtr dnnLayer = std::dynamic_pointer_cast(*layer); + if (dnnLayer) { + dnnLayer->convertWeightsToPaddle(); + } + } +#endif +} + Argument NeuralNetwork::getLayerOutput(const std::string& layerName) { return getLayer(layerName)->getOutput(); } diff --git a/paddle/gserver/gradientmachines/NeuralNetwork.h b/paddle/gserver/gradientmachines/NeuralNetwork.h index 56a1ec7846..16971883b4 100644 --- a/paddle/gserver/gradientmachines/NeuralNetwork.h +++ b/paddle/gserver/gradientmachines/NeuralNetwork.h @@ -134,6 +134,9 @@ public: const std::string& getName() const { return subModelName_; } + /// some finish work, like convert the weight format of MKLDNNLayers + void finish() override; + protected: /** * The constructor of NeuralNetwork. From ab5dc9fe185215760ee351e36d772ad245e1eb3d Mon Sep 17 00:00:00 2001 From: qijun Date: Sun, 15 Oct 2017 22:45:52 -0700 Subject: [PATCH 24/63] remove SelectedRows functors to selected_rows_functor.h --- paddle/operators/math/CMakeLists.txt | 8 +- paddle/operators/math/math_function.cc | 93 ------------ paddle/operators/math/math_function.cu | 119 +-------------- paddle/operators/math/math_function.h | 18 --- paddle/operators/math/math_function_test.cc | 90 ----------- paddle/operators/math/math_function_test.cu | 99 ------------ .../operators/math/selected_rows_functor.cc | 114 ++++++++++++++ .../operators/math/selected_rows_functor.cu | 142 ++++++++++++++++++ paddle/operators/math/selected_rows_functor.h | 41 +++++ .../math/selected_rows_functor_test.cc | 106 +++++++++++++ .../math/selected_rows_functor_test.cu | 115 ++++++++++++++ 11 files changed, 525 insertions(+), 420 deletions(-) create mode 100644 paddle/operators/math/selected_rows_functor.cc create mode 100644 paddle/operators/math/selected_rows_functor.cu create mode 100644 paddle/operators/math/selected_rows_functor.h create mode 100644 paddle/operators/math/selected_rows_functor_test.cc create mode 100644 paddle/operators/math/selected_rows_functor_test.cu diff --git a/paddle/operators/math/CMakeLists.txt b/paddle/operators/math/CMakeLists.txt index 77a3603eb6..72ce858504 100644 --- a/paddle/operators/math/CMakeLists.txt +++ b/paddle/operators/math/CMakeLists.txt @@ -1,18 +1,22 @@ if(WITH_GPU) nv_library(math_function SRCS math_function.cc math_function.cu im2col.cc im2col.cu DEPS cblas device_context operator) - nv_test(math_function_gpu_test SRCS math_function_test.cu DEPS math_function tensor selected_rows) + nv_test(math_function_gpu_test SRCS math_function_test.cu DEPS math_function tensor) + nv_library(selected_rows_functor SRCS selected_rows_functor.cc selected_rows_functor.cu DEPS selected_rows math_function) + nv_test(selected_rows_functor_gpu_test SRCS selected_rows_functor_test.cu DEPS selected_rows_functor) nv_library(softmax SRCS softmax.cc softmax.cu DEPS operator) nv_library(cross_entropy SRCS cross_entropy.cc cross_entropy.cu DEPS operator) nv_library(pooling SRCS pooling.cc pooling.cu DEPS device_context) nv_library(vol2col SRCS vol2col.cc vol2col.cu DEPS device_context) else() cc_library(math_function SRCS math_function.cc im2col.cc DEPS cblas device_context operator) + cc_library(selected_rows_functor SRCS selected_rows_functor.cc DEPS selected_rows math_function) cc_library(softmax SRCS softmax.cc DEPS operator) cc_library(cross_entropy SRCS cross_entropy.cc DEPS operator) cc_library(pooling SRCS pooling.cc DEPS device_context) cc_library(vol2col SRCS vol2col.cc DEPS device_context) endif() -cc_test(math_function_test SRCS math_function_test.cc DEPS math_function tensor selected_rows) +cc_test(math_function_test SRCS math_function_test.cc DEPS math_function tensor) +cc_test(selected_rows_functor_test SRCS selected_rows_functor_test.cc DEPS selected_rows_functor) cc_test(im2col_test SRCS im2col_test.cc DEPS math_function tensor) cc_test(vol2col_test SRCS vol2col_test.cc DEPS vol2col tensor) diff --git a/paddle/operators/math/math_function.cc b/paddle/operators/math/math_function.cc index a1faafb7c4..77a1e22b41 100644 --- a/paddle/operators/math/math_function.cc +++ b/paddle/operators/math/math_function.cc @@ -13,8 +13,6 @@ See the License for the specific language governing permissions and limitations under the License. */ #include "paddle/operators/math/math_function.h" -#include "paddle/framework/eigen.h" -#include "paddle/memory/memcpy.h" namespace paddle { namespace operators { @@ -134,97 +132,6 @@ void matmul( template struct SetConstant; -template -struct SelectedRowsAdd { - void operator()(const platform::DeviceContext& context, - const framework::SelectedRows& input1, - const framework::SelectedRows& input2, - framework::SelectedRows* output) { - auto in1_height = input1.height(); - PADDLE_ENFORCE_EQ(in1_height, input2.height()); - output->set_height(in1_height); - - auto& in1_rows = input1.rows(); - auto& in2_rows = input2.rows(); - std::vector out_rows; - out_rows.reserve(in1_rows.size() + in2_rows.size()); - - // concat rows - out_rows.insert(out_rows.end(), in1_rows.begin(), in1_rows.end()); - out_rows.insert(out_rows.end(), in2_rows.begin(), in2_rows.end()); - output->set_rows(out_rows); - - auto* out_value = output->mutable_value(); - auto& in1_value = input1.value(); - auto& in2_value = input2.value(); - - auto in1_row_numel = in1_value.numel() / in1_rows.size(); - PADDLE_ENFORCE_EQ(in1_row_numel, in2_value.numel() / in2_rows.size()); - PADDLE_ENFORCE_EQ(in1_row_numel, out_value->numel() / out_rows.size()); - - auto in1_place = input1.place(); - PADDLE_ENFORCE(platform::is_cpu_place(in1_place)); - auto in2_place = input2.place(); - PADDLE_ENFORCE(platform::is_cpu_place(in2_place)); - auto out_place = context.GetPlace(); - PADDLE_ENFORCE(platform::is_cpu_place(out_place)); - - auto* out_data = out_value->data(); - auto* in1_data = in1_value.data(); - memory::Copy(boost::get(out_place), out_data, - boost::get(in1_place), in1_data, - in1_value.numel() * sizeof(T)); - - auto* in2_data = in2_value.data(); - memory::Copy(boost::get(out_place), - out_data + in1_value.numel(), - boost::get(in2_place), in2_data, - in2_value.numel() * sizeof(T)); - } -}; - -template struct SelectedRowsAdd; - -template -struct SelectedRowsAddTensor { - void operator()(const platform::DeviceContext& context, - const framework::SelectedRows& input1, - const framework::Tensor& input2, framework::Tensor* output) { - auto in1_height = input1.height(); - auto in2_dims = input2.dims(); - auto out_dims = output->dims(); - PADDLE_ENFORCE_EQ(in1_height, in2_dims[0]); - PADDLE_ENFORCE_EQ(in1_height, out_dims[0]); - - auto& in1_value = input1.value(); - auto& in1_rows = input1.rows(); - - int64_t in1_row_numel = in1_value.numel() / in1_rows.size(); - PADDLE_ENFORCE_EQ(in1_row_numel, input2.numel() / in1_height); - PADDLE_ENFORCE_EQ(in1_row_numel, output->numel() / in1_height); - - SetConstant functor; - functor(context, output, 0.0); - - auto* in1_data = in1_value.data(); - auto* out_data = output->data(); - - for (size_t i = 0; i < in1_rows.size(); i++) { - for (int64_t j = 0; j < in1_row_numel; j++) { - out_data[in1_rows[i] * in1_row_numel + j] += - in1_data[i * in1_row_numel + j]; - } - } - - auto out_eigen = framework::EigenVector::Flatten(*output); - auto in2_eigen = framework::EigenVector::Flatten(input2); - out_eigen.device(*context.GetEigenDevice()) = - out_eigen + in2_eigen; - } -}; - -template struct SelectedRowsAddTensor; - } // namespace math } // namespace operators } // namespace paddle diff --git a/paddle/operators/math/math_function.cu b/paddle/operators/math/math_function.cu index fc16d1b0a7..7fbc03acf2 100644 --- a/paddle/operators/math/math_function.cu +++ b/paddle/operators/math/math_function.cu @@ -13,7 +13,6 @@ See the License for the specific language governing permissions and limitations under the License. */ #include "paddle/operators/math/math_function.h" -#include "paddle/platform/cuda_helper.h" namespace paddle { namespace operators { @@ -156,123 +155,7 @@ void matmul( matrix_b.data(), beta, matrix_out->data()); } -template -struct SelectedRowsAdd { - void operator()(const platform::DeviceContext& context, - const framework::SelectedRows& input1, - const framework::SelectedRows& input2, - framework::SelectedRows* output) { - auto in1_height = input1.height(); - PADDLE_ENFORCE_EQ(in1_height, input2.height()); - output->set_height(in1_height); - - auto& in1_rows = input1.rows(); - auto& in2_rows = input2.rows(); - std::vector out_rows; - out_rows.reserve(in1_rows.size() + in2_rows.size()); - - // concat rows - out_rows.insert(out_rows.end(), in1_rows.begin(), in1_rows.end()); - out_rows.insert(out_rows.end(), in2_rows.begin(), in2_rows.end()); - output->set_rows(out_rows); - - auto* out_value = output->mutable_value(); - auto& in1_value = input1.value(); - auto& in2_value = input2.value(); - - auto in1_row_numel = in1_value.numel() / in1_rows.size(); - PADDLE_ENFORCE_EQ(in1_row_numel, in2_value.numel() / in2_rows.size()); - PADDLE_ENFORCE_EQ(in1_row_numel, out_value->numel() / out_rows.size()); - - auto* out_data = out_value->data(); - auto* in1_data = in1_value.data(); - - auto in1_place = input1.place(); - PADDLE_ENFORCE(platform::is_gpu_place(in1_place)); - auto in2_place = input2.place(); - PADDLE_ENFORCE(platform::is_gpu_place(in2_place)); - auto out_place = context.GetPlace(); - PADDLE_ENFORCE(platform::is_gpu_place(out_place)); - - memory::Copy( - boost::get(out_place), out_data, - boost::get(in1_place), in1_data, - in1_value.numel() * sizeof(T), - reinterpret_cast(context).stream()); - - auto* in2_data = in2_value.data(); - memory::Copy( - boost::get(out_place), out_data + in1_value.numel(), - boost::get(in2_place), in2_data, - in2_value.numel() * sizeof(T), - reinterpret_cast(context).stream()); - } -}; - -template struct SelectedRowsAdd; - -namespace { -template -__global__ void SelectedRowsAddTensorKernel(const T* selected_rows, - const int64_t* rows, T* tensor_out, - int64_t row_numel, int block_size) { - const int ty = blockIdx.y; - int tid = threadIdx.x; - - selected_rows += ty * row_numel; - tensor_out += rows[ty] * row_numel; - - for (int index = tid; index < row_numel; index += block_size) { - // Since index in rows of SelectedRows can be duplicate, we can not use - // tensor_out[index] += selected_rows[index]; Instead, we have to use - // AtomicAdd to avoid concurrent write error. - paddle::platform::CudaAtomicAdd(&tensor_out[index], selected_rows[index]); - } -} -} // namespace - -template -struct SelectedRowsAddTensor { - void operator()(const platform::DeviceContext& context, - const framework::SelectedRows& input1, - const framework::Tensor& input2, framework::Tensor* output) { - auto in1_height = input1.height(); - auto in2_dims = input2.dims(); - auto out_dims = output->dims(); - PADDLE_ENFORCE_EQ(in1_height, in2_dims[0]); - PADDLE_ENFORCE_EQ(in1_height, out_dims[0]); - - auto& in1_value = input1.value(); - auto& in1_rows = input1.rows(); - - int64_t in1_row_numel = in1_value.numel() / in1_rows.size(); - PADDLE_ENFORCE_EQ(in1_row_numel, input2.numel() / in1_height); - PADDLE_ENFORCE_EQ(in1_row_numel, output->numel() / in1_height); - - auto* in1_data = in1_value.data(); - auto* in2_data = input2.data(); - auto* out_data = output->data(); - - SetConstant functor; - functor(context, output, 0.0); - - int block_size = 256; - dim3 threads(block_size, 1); - dim3 grid(1, in1_height); - SelectedRowsAddTensorKernel< - T><<(context) - .stream()>>>(in1_data, in1_rows.data(), out_data, - in1_row_numel, block_size); - - auto out_eigen = framework::EigenVector::Flatten(*output); - auto in2_eigen = framework::EigenVector::Flatten(input2); - out_eigen.device(*context.GetEigenDevice()) = - out_eigen + in2_eigen; - } -}; - -template struct SelectedRowsAddTensor; +template struct SetConstant; } // namespace math } // namespace operators diff --git a/paddle/operators/math/math_function.h b/paddle/operators/math/math_function.h index 0d0d4cdd73..6f92d83aab 100644 --- a/paddle/operators/math/math_function.h +++ b/paddle/operators/math/math_function.h @@ -53,7 +53,6 @@ int LAPACKE_dgetri(int matrix_layout, int n, double* a, int lda, #include #include "paddle/framework/eigen.h" -#include "paddle/framework/selected_rows.h" #include "paddle/framework/tensor.h" #include "paddle/platform/device_context.h" #include "paddle/platform/enforce.h" @@ -96,23 +95,6 @@ struct SetConstant { } }; -// SelectedRows + SelectedRows will simplely concat value and rows. -// The real computation happens in dealing with LoDTensor. -template -struct SelectedRowsAdd { - void operator()(const platform::DeviceContext& context, - const framework::SelectedRows& input1, - const framework::SelectedRows& input2, - framework::SelectedRows* output); -}; - -template -struct SelectedRowsAddTensor { - void operator()(const platform::DeviceContext& context, - const framework::SelectedRows& input1, - const framework::Tensor& input2, framework::Tensor* output); -}; - } // namespace math } // namespace operators } // namespace paddle diff --git a/paddle/operators/math/math_function_test.cc b/paddle/operators/math/math_function_test.cc index 33c561f6c6..3b9f92e7ae 100644 --- a/paddle/operators/math/math_function_test.cc +++ b/paddle/operators/math/math_function_test.cc @@ -89,93 +89,3 @@ TEST(math_function, zero) { EXPECT_EQ(t[2], 1); EXPECT_EQ(t[3], 1); } - -TEST(math_function, selected_rows_add) { - using namespace paddle::framework; - using namespace paddle::platform; - using namespace paddle::operators::math; - - CPUPlace cpu_place; - CPUDeviceContext ctx(cpu_place); - SetConstant functor; - int64_t height = 10; - int64_t row_numel = 10; - - std::vector rows1{0, 4, 7}; - std::unique_ptr selected_rows1{new SelectedRows(rows1, height)}; - auto* in1_value = selected_rows1->mutable_value(); - in1_value->mutable_data( - make_ddim({static_cast(rows1.size()), row_numel}), cpu_place); - functor(ctx, in1_value, 1.0); - - std::vector rows2{0, 5, 7, 9}; - std::unique_ptr selected_rows2{new SelectedRows(rows2, height)}; - auto* in2_value = selected_rows2->mutable_value(); - in2_value->mutable_data( - make_ddim({static_cast(rows2.size()), row_numel}), cpu_place); - functor(ctx, in2_value, 2.0); - - std::unique_ptr output{new SelectedRows()}; - auto* out_value = output->mutable_value(); - - // simplely concat two SelectedRows - out_value->mutable_data(make_ddim({7, 10}), cpu_place); - - SelectedRowsAdd add_functor; - add_functor(ctx, *selected_rows1, *selected_rows2, output.get()); - - auto out_height = output->height(); - EXPECT_EQ(out_height, height); - - auto& out_rows = output->rows(); - - // input1 rows - EXPECT_EQ(out_rows[0], 0); - EXPECT_EQ(out_rows[1], 4); - EXPECT_EQ(out_rows[2], 7); - // input2 rows - EXPECT_EQ(out_rows[3], 0); - EXPECT_EQ(out_rows[4], 5); - EXPECT_EQ(out_rows[5], 7); - EXPECT_EQ(out_rows[6], 9); - - auto* out_data = output->value().data(); - // input1 value - EXPECT_EQ(out_data[0 * row_numel + 0], 1.0); - EXPECT_EQ(out_data[0 * row_numel + 8], 1.0); - EXPECT_EQ(out_data[1 * row_numel + 1], 1.0); - EXPECT_EQ(out_data[2 * row_numel + 6], 1.0); - // input2 value - EXPECT_EQ(out_data[3 * row_numel + 3], 2.0); - EXPECT_EQ(out_data[3 * row_numel + 8], 2.0); - EXPECT_EQ(out_data[4 * row_numel + 4], 2.0); - EXPECT_EQ(out_data[5 * row_numel + 7], 2.0); - EXPECT_EQ(out_data[6 * row_numel + 9], 2.0); - - std::unique_ptr tensor1{new Tensor()}; - tensor1->mutable_data(make_ddim({height, row_numel}), cpu_place); - SetConstant constant_functor; - constant_functor(ctx, tensor1.get(), 3.0); - - std::unique_ptr tensor2{new Tensor()}; - tensor2->mutable_data(make_ddim({height, row_numel}), cpu_place); - - SelectedRowsAddTensor add_tensor_functor; - add_tensor_functor(ctx, *output, *tensor1, tensor2.get()); - - auto* tensor2_data = tensor2->data(); - // row0: 1.0 + 2.0 + 3.0 - EXPECT_EQ(tensor2_data[0 * row_numel + 0], 6.0); - // row1: 3.0 - EXPECT_EQ(tensor2_data[1 * row_numel + 1], 3.0); - // row4 : 1.0 + 3.0 - EXPECT_EQ(tensor2_data[4 * row_numel + 6], 4.0); - // row5: 2.0 + 3.0 - EXPECT_EQ(tensor2_data[5 * row_numel + 7], 5.0); - // row6: 3.0 - EXPECT_EQ(tensor2_data[6 * row_numel + 1], 3.0); - // row7: 1.0 + 2.0 + 3.0 - EXPECT_EQ(tensor2_data[7 * row_numel + 3], 6.0); - // row9: 2.0 + 3.0 - EXPECT_EQ(tensor2_data[9 * row_numel + 6], 5.0); -} diff --git a/paddle/operators/math/math_function_test.cu b/paddle/operators/math/math_function_test.cu index 1acc5f66a6..14359d835b 100644 --- a/paddle/operators/math/math_function_test.cu +++ b/paddle/operators/math/math_function_test.cu @@ -177,102 +177,3 @@ TEST(math_function, gemm_trans_cublas) { EXPECT_EQ(input3_ptr[7], 99); delete gpu_place; } - -TEST(math_function, selected_rows_add) { - using namespace paddle::framework; - using namespace paddle::platform; - using namespace paddle::operators::math; - - GPUPlace gpu_place(0); - CPUPlace cpu_place; - CUDADeviceContext ctx(gpu_place); - SetConstant functor; - int64_t height = 10; - int64_t row_numel = 10; - - std::vector rows1{0, 4, 7}; - std::unique_ptr selected_rows1{new SelectedRows(rows1, height)}; - auto* in1_value = selected_rows1->mutable_value(); - in1_value->mutable_data( - make_ddim({static_cast(rows1.size()), row_numel}), gpu_place); - functor(ctx, in1_value, 1.0); - - std::vector rows2{0, 5, 7, 9}; - std::unique_ptr selected_rows2{new SelectedRows(rows2, height)}; - auto* in2_value = selected_rows2->mutable_value(); - in2_value->mutable_data( - make_ddim({static_cast(rows2.size()), row_numel}), gpu_place); - functor(ctx, in2_value, 2.0); - - std::unique_ptr output{new SelectedRows()}; - auto* out_value = output->mutable_value(); - - // simplely concat two SelectedRows - out_value->mutable_data(make_ddim({7, 10}), gpu_place); - - SelectedRowsAdd add_functor; - add_functor(ctx, *selected_rows1, *selected_rows2, output.get()); - - auto out_height = output->height(); - EXPECT_EQ(out_height, height); - - auto& out_rows = output->rows(); - - // input1 rows - EXPECT_EQ(out_rows[0], 0); - EXPECT_EQ(out_rows[1], 4); - EXPECT_EQ(out_rows[2], 7); - // input2 rows - EXPECT_EQ(out_rows[3], 0); - EXPECT_EQ(out_rows[4], 5); - EXPECT_EQ(out_rows[5], 7); - EXPECT_EQ(out_rows[6], 9); - - Tensor out_cpu; - out_cpu.CopyFrom(*out_value, cpu_place, ctx); - ctx.Wait(); - - auto* out_cpu_data = out_cpu.data(); - // input1 value - EXPECT_EQ(out_cpu_data[0 * row_numel + 0], 1.0); - EXPECT_EQ(out_cpu_data[0 * row_numel + 8], 1.0); - EXPECT_EQ(out_cpu_data[1 * row_numel + 1], 1.0); - EXPECT_EQ(out_cpu_data[2 * row_numel + 6], 1.0); - // input2 value - EXPECT_EQ(out_cpu_data[3 * row_numel + 3], 2.0); - EXPECT_EQ(out_cpu_data[3 * row_numel + 8], 2.0); - EXPECT_EQ(out_cpu_data[4 * row_numel + 4], 2.0); - EXPECT_EQ(out_cpu_data[5 * row_numel + 7], 2.0); - EXPECT_EQ(out_cpu_data[6 * row_numel + 9], 2.0); - - std::unique_ptr tensor1{new Tensor()}; - tensor1->mutable_data(make_ddim({height, row_numel}), gpu_place); - SetConstant constant_functor; - constant_functor(ctx, tensor1.get(), 3.0); - - std::unique_ptr tensor2{new Tensor()}; - tensor2->mutable_data(make_ddim({height, row_numel}), gpu_place); - - SelectedRowsAddTensor add_tensor_functor; - add_tensor_functor(ctx, *output, *tensor1, tensor2.get()); - - Tensor tensor2_cpu; - tensor2_cpu.CopyFrom(*tensor2, cpu_place, ctx); - ctx.Wait(); - - auto* tensor2_cpu_data = tensor2_cpu.data(); - // row0: 1.0 + 2.0 + 3.0 - EXPECT_EQ(tensor2_cpu_data[0 * row_numel + 0], 6.0); - // row1: 3.0 - EXPECT_EQ(tensor2_cpu_data[1 * row_numel + 1], 3.0); - // row4 : 1.0 + 3.0 - EXPECT_EQ(tensor2_cpu_data[4 * row_numel + 6], 4.0); - // row5: 2.0 + 3.0 - EXPECT_EQ(tensor2_cpu_data[5 * row_numel + 7], 5.0); - // row6: 3.0 - EXPECT_EQ(tensor2_cpu_data[6 * row_numel + 1], 3.0); - // row7: 1.0 + 2.0 + 3.0 - EXPECT_EQ(tensor2_cpu_data[7 * row_numel + 3], 6.0); - // row9: 2.0 + 3.0 - EXPECT_EQ(tensor2_cpu_data[9 * row_numel + 6], 5.0); -} diff --git a/paddle/operators/math/selected_rows_functor.cc b/paddle/operators/math/selected_rows_functor.cc new file mode 100644 index 0000000000..f2305ea169 --- /dev/null +++ b/paddle/operators/math/selected_rows_functor.cc @@ -0,0 +1,114 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. */ + +#include "paddle/operators/math/selected_rows_functor.h" +#include "paddle/operators/math/math_function.h" + +namespace paddle { +namespace operators { +namespace math { +template +struct SelectedRowsAdd { + void operator()(const platform::DeviceContext& context, + const framework::SelectedRows& input1, + const framework::SelectedRows& input2, + framework::SelectedRows* output) { + auto in1_height = input1.height(); + PADDLE_ENFORCE_EQ(in1_height, input2.height()); + output->set_height(in1_height); + + auto& in1_rows = input1.rows(); + auto& in2_rows = input2.rows(); + std::vector out_rows; + out_rows.reserve(in1_rows.size() + in2_rows.size()); + + // concat rows + out_rows.insert(out_rows.end(), in1_rows.begin(), in1_rows.end()); + out_rows.insert(out_rows.end(), in2_rows.begin(), in2_rows.end()); + output->set_rows(out_rows); + + auto* out_value = output->mutable_value(); + auto& in1_value = input1.value(); + auto& in2_value = input2.value(); + + auto in1_row_numel = in1_value.numel() / in1_rows.size(); + PADDLE_ENFORCE_EQ(in1_row_numel, in2_value.numel() / in2_rows.size()); + PADDLE_ENFORCE_EQ(in1_row_numel, out_value->numel() / out_rows.size()); + + auto in1_place = input1.place(); + PADDLE_ENFORCE(platform::is_cpu_place(in1_place)); + auto in2_place = input2.place(); + PADDLE_ENFORCE(platform::is_cpu_place(in2_place)); + auto out_place = context.GetPlace(); + PADDLE_ENFORCE(platform::is_cpu_place(out_place)); + + auto* out_data = out_value->data(); + auto* in1_data = in1_value.data(); + memory::Copy(boost::get(out_place), out_data, + boost::get(in1_place), in1_data, + in1_value.numel() * sizeof(T)); + + auto* in2_data = in2_value.data(); + memory::Copy(boost::get(out_place), + out_data + in1_value.numel(), + boost::get(in2_place), in2_data, + in2_value.numel() * sizeof(T)); + } +}; + +template struct SelectedRowsAdd; + +template +struct SelectedRowsAddTensor { + void operator()(const platform::DeviceContext& context, + const framework::SelectedRows& input1, + const framework::Tensor& input2, framework::Tensor* output) { + auto in1_height = input1.height(); + auto in2_dims = input2.dims(); + auto out_dims = output->dims(); + PADDLE_ENFORCE_EQ(in1_height, in2_dims[0]); + PADDLE_ENFORCE_EQ(in1_height, out_dims[0]); + + auto& in1_value = input1.value(); + auto& in1_rows = input1.rows(); + + int64_t in1_row_numel = in1_value.numel() / in1_rows.size(); + PADDLE_ENFORCE_EQ(in1_row_numel, input2.numel() / in1_height); + PADDLE_ENFORCE_EQ(in1_row_numel, output->numel() / in1_height); + + SetConstant functor; + functor(context, output, 0.0); + + auto* in1_data = in1_value.data(); + auto* out_data = output->data(); + + for (size_t i = 0; i < in1_rows.size(); i++) { + for (int64_t j = 0; j < in1_row_numel; j++) { + out_data[in1_rows[i] * in1_row_numel + j] += + in1_data[i * in1_row_numel + j]; + } + } + + auto out_eigen = framework::EigenVector::Flatten(*output); + auto in2_eigen = framework::EigenVector::Flatten(input2); + out_eigen.device(*context.GetEigenDevice()) = + out_eigen + in2_eigen; + } +}; + +template struct SelectedRowsAddTensor; + +} // namespace math +} // namespace operators +} // namespace paddle diff --git a/paddle/operators/math/selected_rows_functor.cu b/paddle/operators/math/selected_rows_functor.cu new file mode 100644 index 0000000000..a406bef39a --- /dev/null +++ b/paddle/operators/math/selected_rows_functor.cu @@ -0,0 +1,142 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. */ + +#include "paddle/operators/math/math_function.h" +#include "paddle/operators/math/selected_rows_functor.h" +#include "paddle/platform/cuda_helper.h" + +namespace paddle { +namespace operators { +namespace math { +template +struct SelectedRowsAdd { + void operator()(const platform::DeviceContext& context, + const framework::SelectedRows& input1, + const framework::SelectedRows& input2, + framework::SelectedRows* output) { + auto in1_height = input1.height(); + PADDLE_ENFORCE_EQ(in1_height, input2.height()); + output->set_height(in1_height); + + auto& in1_rows = input1.rows(); + auto& in2_rows = input2.rows(); + std::vector out_rows; + out_rows.reserve(in1_rows.size() + in2_rows.size()); + + // concat rows + out_rows.insert(out_rows.end(), in1_rows.begin(), in1_rows.end()); + out_rows.insert(out_rows.end(), in2_rows.begin(), in2_rows.end()); + output->set_rows(out_rows); + + auto* out_value = output->mutable_value(); + auto& in1_value = input1.value(); + auto& in2_value = input2.value(); + + auto in1_row_numel = in1_value.numel() / in1_rows.size(); + PADDLE_ENFORCE_EQ(in1_row_numel, in2_value.numel() / in2_rows.size()); + PADDLE_ENFORCE_EQ(in1_row_numel, out_value->numel() / out_rows.size()); + + auto* out_data = out_value->data(); + auto* in1_data = in1_value.data(); + + auto in1_place = input1.place(); + PADDLE_ENFORCE(platform::is_gpu_place(in1_place)); + auto in2_place = input2.place(); + PADDLE_ENFORCE(platform::is_gpu_place(in2_place)); + auto out_place = context.GetPlace(); + PADDLE_ENFORCE(platform::is_gpu_place(out_place)); + + memory::Copy( + boost::get(out_place), out_data, + boost::get(in1_place), in1_data, + in1_value.numel() * sizeof(T), + reinterpret_cast(context).stream()); + + auto* in2_data = in2_value.data(); + memory::Copy( + boost::get(out_place), out_data + in1_value.numel(), + boost::get(in2_place), in2_data, + in2_value.numel() * sizeof(T), + reinterpret_cast(context).stream()); + } +}; + +template struct SelectedRowsAdd; + +namespace { +template +__global__ void SelectedRowsAddTensorKernel(const T* selected_rows, + const int64_t* rows, T* tensor_out, + int64_t row_numel, int block_size) { + const int ty = blockIdx.y; + int tid = threadIdx.x; + + selected_rows += ty * row_numel; + tensor_out += rows[ty] * row_numel; + + for (int index = tid; index < row_numel; index += block_size) { + // Since index in rows of SelectedRows can be duplicate, we can not use + // tensor_out[index] += selected_rows[index]; Instead, we have to use + // AtomicAdd to avoid concurrent write error. + paddle::platform::CudaAtomicAdd(&tensor_out[index], selected_rows[index]); + } +} +} // namespace + +template +struct SelectedRowsAddTensor { + void operator()(const platform::DeviceContext& context, + const framework::SelectedRows& input1, + const framework::Tensor& input2, framework::Tensor* output) { + auto in1_height = input1.height(); + auto in2_dims = input2.dims(); + auto out_dims = output->dims(); + PADDLE_ENFORCE_EQ(in1_height, in2_dims[0]); + PADDLE_ENFORCE_EQ(in1_height, out_dims[0]); + + auto& in1_value = input1.value(); + auto& in1_rows = input1.rows(); + + int64_t in1_row_numel = in1_value.numel() / in1_rows.size(); + PADDLE_ENFORCE_EQ(in1_row_numel, input2.numel() / in1_height); + PADDLE_ENFORCE_EQ(in1_row_numel, output->numel() / in1_height); + + auto* in1_data = in1_value.data(); + auto* in2_data = input2.data(); + auto* out_data = output->data(); + + SetConstant functor; + functor(context, output, 0.0); + + int block_size = 256; + dim3 threads(block_size, 1); + dim3 grid(1, in1_height); + SelectedRowsAddTensorKernel< + T><<(context) + .stream()>>>(in1_data, in1_rows.data(), out_data, + in1_row_numel, block_size); + + auto out_eigen = framework::EigenVector::Flatten(*output); + auto in2_eigen = framework::EigenVector::Flatten(input2); + out_eigen.device(*context.GetEigenDevice()) = + out_eigen + in2_eigen; + } +}; + +template struct SelectedRowsAddTensor; + +} // namespace math +} // namespace operators +} // namespace paddle diff --git a/paddle/operators/math/selected_rows_functor.h b/paddle/operators/math/selected_rows_functor.h new file mode 100644 index 0000000000..53ab240ca6 --- /dev/null +++ b/paddle/operators/math/selected_rows_functor.h @@ -0,0 +1,41 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. */ +#pragma once +#include "paddle/framework/selected_rows.h" +#include "paddle/platform/device_context.h" + +namespace paddle { +namespace operators { +namespace math { + +// SelectedRows + SelectedRows will simplely concat value and rows. +// The real computation happens in dealing with LoDTensor. +template +struct SelectedRowsAdd { + void operator()(const platform::DeviceContext& context, + const framework::SelectedRows& input1, + const framework::SelectedRows& input2, + framework::SelectedRows* output); +}; + +template +struct SelectedRowsAddTensor { + void operator()(const platform::DeviceContext& context, + const framework::SelectedRows& input1, + const framework::Tensor& input2, framework::Tensor* output); +}; + +} // namespace math +} // namespace operators +} // namespace paddle diff --git a/paddle/operators/math/selected_rows_functor_test.cc b/paddle/operators/math/selected_rows_functor_test.cc new file mode 100644 index 0000000000..4f7760cb71 --- /dev/null +++ b/paddle/operators/math/selected_rows_functor_test.cc @@ -0,0 +1,106 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. */ + +#include "paddle/operators/math/selected_rows_functor.h" +#include "gtest/gtest.h" +#include "paddle/operators/math/math_function.h" + +TEST(selected_rows_functor, cpu_add) { + using namespace paddle::framework; + using namespace paddle::platform; + using namespace paddle::operators::math; + + CPUPlace cpu_place; + CPUDeviceContext ctx(cpu_place); + SetConstant functor; + int64_t height = 10; + int64_t row_numel = 10; + + std::vector rows1{0, 4, 7}; + std::unique_ptr selected_rows1{new SelectedRows(rows1, height)}; + auto* in1_value = selected_rows1->mutable_value(); + in1_value->mutable_data( + make_ddim({static_cast(rows1.size()), row_numel}), cpu_place); + functor(ctx, in1_value, 1.0); + + std::vector rows2{0, 5, 7, 9}; + std::unique_ptr selected_rows2{new SelectedRows(rows2, height)}; + auto* in2_value = selected_rows2->mutable_value(); + in2_value->mutable_data( + make_ddim({static_cast(rows2.size()), row_numel}), cpu_place); + functor(ctx, in2_value, 2.0); + + std::unique_ptr output{new SelectedRows()}; + auto* out_value = output->mutable_value(); + + // simplely concat two SelectedRows + out_value->mutable_data(make_ddim({7, 10}), cpu_place); + + SelectedRowsAdd add_functor; + add_functor(ctx, *selected_rows1, *selected_rows2, output.get()); + + auto out_height = output->height(); + EXPECT_EQ(out_height, height); + + auto& out_rows = output->rows(); + + // input1 rows + EXPECT_EQ(out_rows[0], 0); + EXPECT_EQ(out_rows[1], 4); + EXPECT_EQ(out_rows[2], 7); + // input2 rows + EXPECT_EQ(out_rows[3], 0); + EXPECT_EQ(out_rows[4], 5); + EXPECT_EQ(out_rows[5], 7); + EXPECT_EQ(out_rows[6], 9); + + auto* out_data = output->value().data(); + // input1 value + EXPECT_EQ(out_data[0 * row_numel + 0], 1.0); + EXPECT_EQ(out_data[0 * row_numel + 8], 1.0); + EXPECT_EQ(out_data[1 * row_numel + 1], 1.0); + EXPECT_EQ(out_data[2 * row_numel + 6], 1.0); + // input2 value + EXPECT_EQ(out_data[3 * row_numel + 3], 2.0); + EXPECT_EQ(out_data[3 * row_numel + 8], 2.0); + EXPECT_EQ(out_data[4 * row_numel + 4], 2.0); + EXPECT_EQ(out_data[5 * row_numel + 7], 2.0); + EXPECT_EQ(out_data[6 * row_numel + 9], 2.0); + + std::unique_ptr tensor1{new Tensor()}; + tensor1->mutable_data(make_ddim({height, row_numel}), cpu_place); + functor(ctx, tensor1.get(), 3.0); + + std::unique_ptr tensor2{new Tensor()}; + tensor2->mutable_data(make_ddim({height, row_numel}), cpu_place); + + SelectedRowsAddTensor add_tensor_functor; + add_tensor_functor(ctx, *output, *tensor1, tensor2.get()); + + auto* tensor2_data = tensor2->data(); + // row0: 1.0 + 2.0 + 3.0 + EXPECT_EQ(tensor2_data[0 * row_numel + 0], 6.0); + // row1: 3.0 + EXPECT_EQ(tensor2_data[1 * row_numel + 1], 3.0); + // row4 : 1.0 + 3.0 + EXPECT_EQ(tensor2_data[4 * row_numel + 6], 4.0); + // row5: 2.0 + 3.0 + EXPECT_EQ(tensor2_data[5 * row_numel + 7], 5.0); + // row6: 3.0 + EXPECT_EQ(tensor2_data[6 * row_numel + 1], 3.0); + // row7: 1.0 + 2.0 + 3.0 + EXPECT_EQ(tensor2_data[7 * row_numel + 3], 6.0); + // row9: 2.0 + 3.0 + EXPECT_EQ(tensor2_data[9 * row_numel + 6], 5.0); +} diff --git a/paddle/operators/math/selected_rows_functor_test.cu b/paddle/operators/math/selected_rows_functor_test.cu new file mode 100644 index 0000000000..8a9f25b982 --- /dev/null +++ b/paddle/operators/math/selected_rows_functor_test.cu @@ -0,0 +1,115 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. */ + +#include "gtest/gtest.h" +#include "paddle/operators/math/math_function.h" +#include "paddle/operators/math/selected_rows_functor.h" + +TEST(selected_rows_functor, gpu_add) { + using namespace paddle::framework; + using namespace paddle::platform; + using namespace paddle::operators::math; + + GPUPlace gpu_place(0); + CPUPlace cpu_place; + CUDADeviceContext ctx(gpu_place); + SetConstant functor; + int64_t height = 10; + int64_t row_numel = 10; + + std::vector rows1{0, 4, 7}; + std::unique_ptr selected_rows1{new SelectedRows(rows1, height)}; + auto* in1_value = selected_rows1->mutable_value(); + in1_value->mutable_data( + make_ddim({static_cast(rows1.size()), row_numel}), gpu_place); + functor(ctx, in1_value, 1.0); + + std::vector rows2{0, 5, 7, 9}; + std::unique_ptr selected_rows2{new SelectedRows(rows2, height)}; + auto* in2_value = selected_rows2->mutable_value(); + in2_value->mutable_data( + make_ddim({static_cast(rows2.size()), row_numel}), gpu_place); + functor(ctx, in2_value, 2.0); + + std::unique_ptr output{new SelectedRows()}; + auto* out_value = output->mutable_value(); + + // simplely concat two SelectedRows + out_value->mutable_data(make_ddim({7, 10}), gpu_place); + + SelectedRowsAdd add_functor; + add_functor(ctx, *selected_rows1, *selected_rows2, output.get()); + + auto out_height = output->height(); + EXPECT_EQ(out_height, height); + + auto& out_rows = output->rows(); + + // input1 rows + EXPECT_EQ(out_rows[0], 0); + EXPECT_EQ(out_rows[1], 4); + EXPECT_EQ(out_rows[2], 7); + // input2 rows + EXPECT_EQ(out_rows[3], 0); + EXPECT_EQ(out_rows[4], 5); + EXPECT_EQ(out_rows[5], 7); + EXPECT_EQ(out_rows[6], 9); + + Tensor out_cpu; + out_cpu.CopyFrom(*out_value, cpu_place, ctx); + ctx.Wait(); + + auto* out_cpu_data = out_cpu.data(); + // input1 value + EXPECT_EQ(out_cpu_data[0 * row_numel + 0], 1.0); + EXPECT_EQ(out_cpu_data[0 * row_numel + 8], 1.0); + EXPECT_EQ(out_cpu_data[1 * row_numel + 1], 1.0); + EXPECT_EQ(out_cpu_data[2 * row_numel + 6], 1.0); + // input2 value + EXPECT_EQ(out_cpu_data[3 * row_numel + 3], 2.0); + EXPECT_EQ(out_cpu_data[3 * row_numel + 8], 2.0); + EXPECT_EQ(out_cpu_data[4 * row_numel + 4], 2.0); + EXPECT_EQ(out_cpu_data[5 * row_numel + 7], 2.0); + EXPECT_EQ(out_cpu_data[6 * row_numel + 9], 2.0); + + std::unique_ptr tensor1{new Tensor()}; + tensor1->mutable_data(make_ddim({height, row_numel}), gpu_place); + functor(ctx, tensor1.get(), 3.0); + + std::unique_ptr tensor2{new Tensor()}; + tensor2->mutable_data(make_ddim({height, row_numel}), gpu_place); + + SelectedRowsAddTensor add_tensor_functor; + add_tensor_functor(ctx, *output, *tensor1, tensor2.get()); + + Tensor tensor2_cpu; + tensor2_cpu.CopyFrom(*tensor2, cpu_place, ctx); + ctx.Wait(); + + auto* tensor2_cpu_data = tensor2_cpu.data(); + // row0: 1.0 + 2.0 + 3.0 + EXPECT_EQ(tensor2_cpu_data[0 * row_numel + 0], 6.0); + // row1: 3.0 + EXPECT_EQ(tensor2_cpu_data[1 * row_numel + 1], 3.0); + // row4 : 1.0 + 3.0 + EXPECT_EQ(tensor2_cpu_data[4 * row_numel + 6], 4.0); + // row5: 2.0 + 3.0 + EXPECT_EQ(tensor2_cpu_data[5 * row_numel + 7], 5.0); + // row6: 3.0 + EXPECT_EQ(tensor2_cpu_data[6 * row_numel + 1], 3.0); + // row7: 1.0 + 2.0 + 3.0 + EXPECT_EQ(tensor2_cpu_data[7 * row_numel + 3], 6.0); + // row9: 2.0 + 3.0 + EXPECT_EQ(tensor2_cpu_data[9 * row_numel + 6], 5.0); +} From 0d7b1fd4feed8d5249c6904f10126c7d58319503 Mon Sep 17 00:00:00 2001 From: qijun Date: Mon, 16 Oct 2017 08:28:03 +0000 Subject: [PATCH 25/63] fix bug in SelectedRowsAddTensorKernel --- paddle/operators/math/selected_rows_functor.cu | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/paddle/operators/math/selected_rows_functor.cu b/paddle/operators/math/selected_rows_functor.cu index a406bef39a..ea149ebbc1 100644 --- a/paddle/operators/math/selected_rows_functor.cu +++ b/paddle/operators/math/selected_rows_functor.cu @@ -89,7 +89,7 @@ __global__ void SelectedRowsAddTensorKernel(const T* selected_rows, // Since index in rows of SelectedRows can be duplicate, we can not use // tensor_out[index] += selected_rows[index]; Instead, we have to use // AtomicAdd to avoid concurrent write error. - paddle::platform::CudaAtomicAdd(&tensor_out[index], selected_rows[index]); + paddle::platform::CudaAtomicAdd(tensor_out + index, selected_rows[index]); } } } // namespace @@ -121,7 +121,7 @@ struct SelectedRowsAddTensor { int block_size = 256; dim3 threads(block_size, 1); - dim3 grid(1, in1_height); + dim3 grid(1, in1_rows.size()); SelectedRowsAddTensorKernel< T><<(context) From 1199aa6876fc8c6759e3e2be7bb89a597d88359a Mon Sep 17 00:00:00 2001 From: tensor-tang Date: Mon, 16 Oct 2017 17:37:39 +0800 Subject: [PATCH 26/63] fix bug: clear grad and always share data when output has cpu and add activation in unit tests --- paddle/gserver/layers/MKLDNNConvLayer.cpp | 8 +++++--- paddle/gserver/layers/MKLDNNLayer.h | 19 ++++++++++++++++++- paddle/gserver/layers/MKLDNNPoolLayer.cpp | 8 +++++--- paddle/gserver/tests/test_MKLDNN.cpp | 3 +++ 4 files changed, 31 insertions(+), 7 deletions(-) diff --git a/paddle/gserver/layers/MKLDNNConvLayer.cpp b/paddle/gserver/layers/MKLDNNConvLayer.cpp index 8b67a1ef4f..26810a6483 100644 --- a/paddle/gserver/layers/MKLDNNConvLayer.cpp +++ b/paddle/gserver/layers/MKLDNNConvLayer.cpp @@ -313,6 +313,7 @@ void MKLDNNConvLayer::resetOutValue( cvtOutVal_ = MKLDNNMatrix::createReorder(out, cpuOutVal_); CHECK(cvtOutVal_) << "should not be empty"; } else { + cpuOut->setData(output_.value->getData()); cpuOutVal_ = out; } // when output is cpu device, change the mkldnn output value and make them @@ -456,17 +457,18 @@ void MKLDNNConvLayer::resetOutGrad( MKLDNNLayer::resetOutGrad(out, outVal_->getPrimitiveDesc()); } else { const MatrixPtr& cpuOut = getOutput(CPU_DEVICE).grad; + // always share the same grad data of CPU output + // then the activation can get the right grad from output_.grad + output_.grad->setData(cpuOut->getData()); // same PrimitiveDesc with cpuInVal_ CHECK(cpuOutVal_); cpuOutGrad_ = MKLDNNMatrix::create(cpuOut, cpuOutVal_->getPrimitiveDesc()); // create reorder if primitive desc does not match if (cpuOutGrad_->getPrimitiveDesc() != outVal_->getPrimitiveDesc()) { - out = MKLDNNMatrix::create(output_.grad, outVal_->getPrimitiveDesc()); + out = MKLDNNMatrix::create(nullptr, outVal_->getPrimitiveDesc()); cvtOutGrad_ = MKLDNNMatrix::createReorder(cpuOutGrad_, out); CHECK(cvtOutGrad_); } else { - // share the same data of CPU output - output_.grad->setData(cpuOut->getData()); out = cpuOutGrad_; } } diff --git a/paddle/gserver/layers/MKLDNNLayer.h b/paddle/gserver/layers/MKLDNNLayer.h index 5f9923da76..4e2753eba2 100644 --- a/paddle/gserver/layers/MKLDNNLayer.h +++ b/paddle/gserver/layers/MKLDNNLayer.h @@ -46,6 +46,9 @@ protected: // backward also need reset after reset forward handle bool needResetBwd_; + // is output only mkldnn + bool outputOnlyMKLDNN_; + // mkldnn engine, stream and primivtives mkldnn::engine engine_; std::shared_ptr stream_; @@ -141,6 +144,9 @@ public: updateInputData(); } + if (!outputOnlyMKLDNN_) { + clearGrads(); + } stream_->submit(pipelineFwd_); } @@ -389,7 +395,8 @@ protected: CHECK_EQ(outputOtherDevice_[i].deviceId, CPU_DEVICE) << "Only support other device is CPU yet"; } - return outputOtherDevice_.size() == 0; + outputOnlyMKLDNN_ = outputOtherDevice_.size() == 0; + return outputOnlyMKLDNN_; } /** @@ -398,6 +405,16 @@ protected: void setDevice(int id) { deviceId_ = id; } private: + /** + * clear all grad + */ + void clearGrads() { + output_.grad->zeroMem(); + for (size_t i = 0; i < outputOtherDevice_.size(); i++) { + outputOtherDevice_[i].grad->zeroMem(); + } + } + /** * Set deviceId of the params used in this layer. */ diff --git a/paddle/gserver/layers/MKLDNNPoolLayer.cpp b/paddle/gserver/layers/MKLDNNPoolLayer.cpp index 5606aae80c..0e53e2d1b7 100644 --- a/paddle/gserver/layers/MKLDNNPoolLayer.cpp +++ b/paddle/gserver/layers/MKLDNNPoolLayer.cpp @@ -146,6 +146,7 @@ void MKLDNNPoolLayer::resetOutValue(MKLDNNMatrixPtr& out) { cvtOutVal_ = MKLDNNMatrix::createReorder(out, cpuOutVal_); CHECK(cvtOutVal_) << "should not be emptry"; } else { + cpuOut->setData(output_.value->getData()); cpuOutVal_ = out; } output_.value = std::dynamic_pointer_cast(cpuOutVal_); @@ -213,15 +214,16 @@ void MKLDNNPoolLayer::resetOutGrad(MKLDNNMatrixPtr& out) { MKLDNNLayer::resetOutGrad(out, outVal_->getPrimitiveDesc()); } else { const MatrixPtr& cpuOut = getOutput(CPU_DEVICE).grad; + // always share the same grad data of CPU output + // then the activation can get the right grad from output_.grad + output_.grad->setData(cpuOut->getData()); cpuOutGrad_ = MKLDNNMatrix::create( cpuOut, memory::dims{bs_, oc_, oh_, ow_}, format::nchw, engine_); if (cpuOutGrad_->getPrimitiveDesc() != outVal_->getPrimitiveDesc()) { - out = MKLDNNMatrix::create(output_.grad, outVal_->getPrimitiveDesc()); + out = MKLDNNMatrix::create(nullptr, outVal_->getPrimitiveDesc()); cvtOutGrad_ = MKLDNNMatrix::createReorder(cpuOutGrad_, out); CHECK(cvtOutGrad_) << "should not be emptry"; } else { - // share the same data of CPU output - output_.grad->setData(cpuOut->getData()); out = cpuOutGrad_; } } diff --git a/paddle/gserver/tests/test_MKLDNN.cpp b/paddle/gserver/tests/test_MKLDNN.cpp index a70b2f17f4..03515e9469 100644 --- a/paddle/gserver/tests/test_MKLDNN.cpp +++ b/paddle/gserver/tests/test_MKLDNN.cpp @@ -46,6 +46,7 @@ struct testFcDesc { static void getMKLDNNFcConfig(TestConfig& cfg, const testFcDesc& pm) { cfg.layerConfig.set_type("mkldnn_fc"); + cfg.layerConfig.set_active_type("sigmoid"); cfg.layerConfig.set_size(pm.oc); cfg.inputDefs.push_back( {INPUT_DATA, @@ -86,6 +87,7 @@ struct testConvDesc { static void getMKLDNNConvConfig(TestConfig& cfg, const testConvDesc& pm) { cfg.layerConfig.set_type("mkldnn_conv"); + cfg.layerConfig.set_active_type("relu"); cfg.layerConfig.set_num_filters(pm.oc); cfg.layerConfig.set_size(pm.oc * pm.oh * pm.ow); cfg.layerConfig.set_shared_biases(true); @@ -158,6 +160,7 @@ struct testPoolDesc { static void getMKLDNNPoolConfig(TestConfig& cfg, const testPoolDesc& pm) { cfg.layerConfig.set_type("mkldnn_pool"); + cfg.layerConfig.set_active_type("relu"); cfg.layerConfig.set_size(pm.ic * pm.oh * pm.ow); cfg.inputDefs.push_back( {INPUT_DATA, From 5993497cb746ccd5e6d99b1605e5c062484b63dd Mon Sep 17 00:00:00 2001 From: qijun Date: Mon, 16 Oct 2017 10:04:51 -0700 Subject: [PATCH 27/63] fix merge conflict --- paddle/operators/sequence_pool_op.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/paddle/operators/sequence_pool_op.h b/paddle/operators/sequence_pool_op.h index ce68204d41..a5569d1aac 100644 --- a/paddle/operators/sequence_pool_op.h +++ b/paddle/operators/sequence_pool_op.h @@ -111,7 +111,8 @@ class SequencePoolGradKernel : public framework::OpKernel { in_g->mutable_data(context.GetPlace()); if (strategy == LAST || strategy == FIRST) { // set X@Grad be zero at first when strategy is LAST/FIRST - math::SetConstant(context.device_context(), in_g, 0); + math::SetConstant functor; + functor(context.device_context(), in_g, 0); } auto place = context.GetEigenDevice(); for (int i = 0; i < static_cast(lod.size()) - 1; ++i) { From c87e060c187b29cfcf1272ddbaf741915ffcdee4 Mon Sep 17 00:00:00 2001 From: qijun Date: Mon, 16 Oct 2017 10:40:42 -0700 Subject: [PATCH 28/63] export feed/fetch method to Python --- paddle/framework/feed_fetch_method.h | 50 ++++++++++++++++ paddle/framework/scope.h | 60 ------------------- paddle/pybind/pybind.cc | 16 +++-- .../framework/tests/test_feed_fetch_method.py | 50 +++++----------- 4 files changed, 72 insertions(+), 104 deletions(-) create mode 100644 paddle/framework/feed_fetch_method.h diff --git a/paddle/framework/feed_fetch_method.h b/paddle/framework/feed_fetch_method.h new file mode 100644 index 0000000000..be96dc3267 --- /dev/null +++ b/paddle/framework/feed_fetch_method.h @@ -0,0 +1,50 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. */ + +#pragma once +#include "paddle/framework/scope.h" +#include "paddle/framework/variable.h" + +namespace paddle { +namespace framework { + +template +void SetFeedVariable(const LoDTensor& input, const std::string& var_name, + size_t index) { + // If var_name Variable is not found in GlobalScope, a new variable will + // be created. + Variable* g_feed_value = GetGlobalScope().Var(var_name); + auto& feed_inputs = + *(g_feed_value->GetMutable>()); + if (index >= feed_inputs.size()) { + feed_inputs.resize(index + 1); + } + // shared data with input tensor + feed_inputs[index].ShareDataWith(input); + // set lod + feed_inputs[index].set_lod(input.lod()); +} + +LoDTensor& GetFetchVariable(const std::string& var_name, size_t index) { + // If var_name Variable is not found in GlobalScope, a new variable will + // be created. + Variable* g_fetch_value = GetGlobalScope().Var(var_name); + auto& fetch_outputs = + *(g_fetch_value->GetMutable>()); + PADDLE_ENFORCE_LT(index, fetch_outputs.size()); + return fetch_outputs[index]; +} + +} // namespace framework +} // namespace paddle diff --git a/paddle/framework/scope.h b/paddle/framework/scope.h index 7376245b53..78eca28009 100644 --- a/paddle/framework/scope.h +++ b/paddle/framework/scope.h @@ -77,65 +77,5 @@ class Scope { framework::Scope& GetGlobalScope(); -// template -// void SetFeedVariable(const std::vector& input, const Lod& lod, -// const std::vector& dims, -// const std::string& var_name, size_t index) { -// Variable* g_feed_value = GetGlobalScope().Var("var_name"); -// // feed variable holds vector -// auto& feed_inputs = -// *(g_feed_value->GetMutable< -// std::vector>()); -// if (index >= feed_inputs.size()) { -// feed_inputs.resize(index); -// } -// // copy tensor -// T* dst = feed_inputs[index].mutable_data(make_ddim(dims), -// platform::CPUPlace()); -// memcpy(dst, inputs[i].data(), inputs[i].size() * sizeof(T)); -// // copy lod -// feed_inputs[index].set_lod(lod); -// } - -template -void SetFeedVariable(const LoDTensor& input, const std::string& var_name, - size_t index) { - std::cout << "into SetFeedVariable" << std::endl; - std::cout << var_name << std::endl; - std::cout << index << std::endl; - Variable* g_feed_value = GetGlobalScope().Var(var_name); - auto& feed_inputs = - *(g_feed_value->GetMutable>()); - if (index >= feed_inputs.size()) { - feed_inputs.resize(index + 1); - } - // shared data with input tensor - feed_inputs[index].ShareDataWith(input); - // set lod - feed_inputs[index].set_lod(input.lod()); -} - -// template -// std::vector GetFetchVariable(const std::string& var_name, size_t index) { -// Variable* g_fetch_value = GetGlobalScope().Var(var_name); -// auto& fetch_outputs = -// *(g_fetch_value->GetMutable< -// std::vector>()); -// std::vector result; -// result.resize(fetch_outputs[index].numel()); -// memcpy(result.data(), fetch_outputs[i].data(), -// fetch_outputs[i].numel() * sizeof(T)); -// } - -template -LoDTensor& GetFetchVariable(const std::string& var_name, size_t index) { - Variable* g_fetch_value = GetGlobalScope().Var(var_name); - auto& fetch_outputs = - *(g_fetch_value->GetMutable>()); - std::cout << "into GetFetchVariable" << std::endl; - PADDLE_ENFORCE_LT(index, fetch_outputs.size()); - return fetch_outputs[index]; -} - } // namespace framework } // namespace paddle diff --git a/paddle/pybind/pybind.cc b/paddle/pybind/pybind.cc index 1c8d2cfd61..983f19b30d 100644 --- a/paddle/pybind/pybind.cc +++ b/paddle/pybind/pybind.cc @@ -12,10 +12,10 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ -#include "paddle/pybind/protobuf.h" - +#include "paddle/pybind/pybind.h" #include "paddle/framework/backward.h" #include "paddle/framework/executor.h" +#include "paddle/framework/feed_fetch_method.h" #include "paddle/framework/lod_tensor.h" #include "paddle/framework/tensor_array.h" #include "paddle/operators/cond_op.h" @@ -25,7 +25,7 @@ limitations under the License. */ #include "paddle/platform/enforce.h" #include "paddle/platform/place.h" #include "paddle/pybind/exception.h" -#include "paddle/pybind/pybind.h" +#include "paddle/pybind/protobuf.h" #include "paddle/pybind/tensor_py.h" #include "paddle/string/to_string.h" @@ -403,12 +403,10 @@ All parameter, weight, gradient are variables in Paddle. m.def("unique_integer", UniqueIntegerGenerator); m.def("is_compile_gpu", IsCompileGPU); - m.def("set_feed_variable", SetFeedVariable); - // m.def("set_feed_variable", SetFeedVariable); - // m.def("set_feed_variable", SetFeedVariable); - m.def("get_fetch_variable", GetFetchVariable); - // m.def("get_fetch_variable", GetFetchVariable); - // m.def("get_fetch_variable", GetFetchVariable); + m.def("set_feed_variable_float", framework::SetFeedVariable); + m.def("set_feed_variable_double", framework::SetFeedVariable); + m.def("set_feed_variable_int", framework::SetFeedVariable); + m.def("get_fetch_variable", framework::GetFetchVariable); BindProgramDesc(m); BindBlockDesc(m); diff --git a/python/paddle/v2/framework/tests/test_feed_fetch_method.py b/python/paddle/v2/framework/tests/test_feed_fetch_method.py index cd8eeee68f..47eedddcb6 100644 --- a/python/paddle/v2/framework/tests/test_feed_fetch_method.py +++ b/python/paddle/v2/framework/tests/test_feed_fetch_method.py @@ -2,49 +2,29 @@ import paddle.v2.framework.core as core import unittest import numpy as np -# class TestFeedFetch(unittest.TestCase): -# def test_feed_fetch(self): -# place = core.CPUPlace() -# input_tensor = core.LoDTensor([[0, 2, 4]]) -# input_tensor.set_dims([4, 4, 6]) -# input_tensor.alloc_int(place) -# input_array = np.array(input_tensor) -# input_array[0, 0, 0] = 3 -# input_array[3, 3, 5] = 10 -# input_tensor.set(input_array, place) - -# core.set_feed_variable(input_tensor, "feed", 0) - -# output_tensor = core.get_fetch_variable("feed", 0) -# print type(output_tensor) - -# output_lod = output_tensor.lod() -# print type(output_lod) -# print output_lod[0] -# print output_lod[0][0] -# print output_lod[0][1] -# print output_lod[0][2] -# # self.assertEqual(0, output_lod[0][0]) -# # self.assertEqual(0, output_lod[0][0]) -# # self.assertEqual(2, output_lod[0][1]) -# # self.assertEqual(4, output_lod[0][2]) - -# # output_array = np.array(output_tensor) -# # self.assertEqual(3, output_array[0, 0, 0]) -# # self.assertEqual(10, output_array[3, 3, 5]); - class TestFeedFetch(unittest.TestCase): def test_feed_fetch(self): place = core.CPUPlace() - input_tensor = core.LoDTensor([[0, 2, 4]]) - input_tensor.set_dims([4, 4, 6]) - input_tensor.alloc_float(place) - input_array = np.array(input_tensor) + input_array = np.ones((4, 4, 6)).astype("float32") input_array[0, 0, 0] = 3 input_array[3, 3, 5] = 10 + input_tensor = core.LoDTensor([[0, 2, 4]]) input_tensor.set(input_array, place) + core.set_feed_variable_float(input_tensor, "feed", 0) + + output_tensor = core.get_fetch_variable("feed", 0) + + output_lod = output_tensor.lod() + self.assertEqual(0, output_lod[0][0]) + self.assertEqual(2, output_lod[0][1]) + self.assertEqual(4, output_lod[0][2]) + + output_array = np.array(output_tensor) + self.assertEqual(3, output_array[0, 0, 0]) + self.assertEqual(10, output_array[3, 3, 5]) + if __name__ == "__main__": unittest.main() From 701c90f6f74f2907baaa8d580e64b51b2db3fba0 Mon Sep 17 00:00:00 2001 From: qijun Date: Mon, 16 Oct 2017 10:47:08 -0700 Subject: [PATCH 29/63] remove unused header file --- paddle/framework/scope.h | 2 -- 1 file changed, 2 deletions(-) diff --git a/paddle/framework/scope.h b/paddle/framework/scope.h index 78eca28009..a7fce3514b 100644 --- a/paddle/framework/scope.h +++ b/paddle/framework/scope.h @@ -18,10 +18,8 @@ limitations under the License. */ #include #include -#include "paddle/framework/lod_tensor.h" #include "paddle/framework/variable.h" #include "paddle/platform/macros.h" -#include "paddle/platform/place.h" namespace paddle { namespace framework { From 5fe3f8f630cf4459a30a399d63f27db7b474ca8e Mon Sep 17 00:00:00 2001 From: qijun Date: Mon, 16 Oct 2017 10:50:20 -0700 Subject: [PATCH 30/63] refine code --- paddle/framework/feed_fetch_method.h | 6 +++--- paddle/pybind/pybind.cc | 5 +++-- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/paddle/framework/feed_fetch_method.h b/paddle/framework/feed_fetch_method.h index be96dc3267..826d180bfc 100644 --- a/paddle/framework/feed_fetch_method.h +++ b/paddle/framework/feed_fetch_method.h @@ -37,9 +37,9 @@ void SetFeedVariable(const LoDTensor& input, const std::string& var_name, } LoDTensor& GetFetchVariable(const std::string& var_name, size_t index) { - // If var_name Variable is not found in GlobalScope, a new variable will - // be created. - Variable* g_fetch_value = GetGlobalScope().Var(var_name); + // Since we want to fetch LodTensor from a variable, the variable must + // be created alreadly. + Variable* g_fetch_value = GetGlobalScope().FindVar(var_name); auto& fetch_outputs = *(g_fetch_value->GetMutable>()); PADDLE_ENFORCE_LT(index, fetch_outputs.size()); diff --git a/paddle/pybind/pybind.cc b/paddle/pybind/pybind.cc index 983f19b30d..008a9441db 100644 --- a/paddle/pybind/pybind.cc +++ b/paddle/pybind/pybind.cc @@ -12,7 +12,8 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ -#include "paddle/pybind/pybind.h" +#include "paddle/pybind/protobuf.h" + #include "paddle/framework/backward.h" #include "paddle/framework/executor.h" #include "paddle/framework/feed_fetch_method.h" @@ -25,7 +26,7 @@ limitations under the License. */ #include "paddle/platform/enforce.h" #include "paddle/platform/place.h" #include "paddle/pybind/exception.h" -#include "paddle/pybind/protobuf.h" +#include "paddle/pybind/pybind.h" #include "paddle/pybind/tensor_py.h" #include "paddle/string/to_string.h" From 3e427441910bbccddf0222d4bd05ecfdea9302e7 Mon Sep 17 00:00:00 2001 From: "Yang Yang(Tony)" Date: Mon, 16 Oct 2017 11:15:10 -0700 Subject: [PATCH 31/63] update based on review --- doc/design/executor.md | 29 +---------------------------- 1 file changed, 1 insertion(+), 28 deletions(-) diff --git a/doc/design/executor.md b/doc/design/executor.md index ab2d6c3558..b5fb6c5c3c 100644 --- a/doc/design/executor.md +++ b/doc/design/executor.md @@ -20,31 +20,4 @@ It does not do graph partitioning, meaning dividing the `ProgramDesc` into sever ## Implementation -`Executor` evaluates a `ProgramDesc`. Essentially, it instantiates Variables and Operators, then run all the operators - -```c++ -void Executor::Run(const ProgramDesc& pdesc, Scope* scope, int block_id) { - auto& block = pdesc.blocks(block_id); - auto& device = device_contexts_[0]; - - // Instantiate all the vars in the global scope - for (auto& var : block.vars()) { - scope->NewVar(var.name()); - } - - // Run the block - Scope& local_scope = scope->NewScope(); - for (auto& op_desc : block.ops()) { - for (auto& var : op_desc.outputs()) { - for (auto& argu : var.arguments()) { - // Create temp variable in the local_scope - if (local_scope.FindVar(argu) == nullptr) { - local_scope.NewVar(argu); - } - } - } - auto op = paddle::framework::OpRegistry::CreateOp(op_desc); - op->Run(local_scope, *device); - } -} -``` +`Executor` evaluates a `ProgramDesc`. Essentially, it instantiates Variables and Operators, then run all the operators in sequence. [[code]](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/executor.cc) From 219f46ae16fa3f3e74f28bcbf1b7f815b9b5ac92 Mon Sep 17 00:00:00 2001 From: qijun Date: Mon, 16 Oct 2017 11:56:15 -0700 Subject: [PATCH 32/63] export SelectedRows to Python --- paddle/pybind/pybind.cc | 17 +++++++++ .../v2/framework/tests/test_selected_rows.py | 37 +++++++++++++++++++ 2 files changed, 54 insertions(+) create mode 100644 python/paddle/v2/framework/tests/test_selected_rows.py diff --git a/paddle/pybind/pybind.cc b/paddle/pybind/pybind.cc index afc80b25b1..23e76011c9 100644 --- a/paddle/pybind/pybind.cc +++ b/paddle/pybind/pybind.cc @@ -17,6 +17,7 @@ limitations under the License. */ #include "paddle/framework/backward.h" #include "paddle/framework/executor.h" #include "paddle/framework/lod_tensor.h" +#include "paddle/framework/selected_rows.h" #include "paddle/framework/tensor_array.h" #include "paddle/operators/cond_op.h" #include "paddle/operators/dynamic_recurrent_op.h" @@ -138,6 +139,22 @@ PYBIND11_PLUGIN(core) { #endif }); + py::class_(m, "SelectedRows") + .def("__init__", + [](SelectedRows &instance) { new (&instance) SelectedRows(); }) + .def("__init__", + [](SelectedRows &instance, const std::vector rows, + const int64_t &height) { + new (&instance) SelectedRows(rows, height); + }) + .def("get_tensor", + [](SelectedRows &self) { return self.mutable_value(); }, + py::return_value_policy::reference) + .def("set_height", &SelectedRows::set_height) + .def("height", &SelectedRows::height) + .def("set_rows", &SelectedRows::set_rows) + .def("rows", &SelectedRows::rows, py::return_value_policy::reference); + py::class_(m, "Variable", R"DOC(Variable Class. All parameter, weight, gradient are variables in Paddle. diff --git a/python/paddle/v2/framework/tests/test_selected_rows.py b/python/paddle/v2/framework/tests/test_selected_rows.py new file mode 100644 index 0000000000..661e818179 --- /dev/null +++ b/python/paddle/v2/framework/tests/test_selected_rows.py @@ -0,0 +1,37 @@ +import paddle.v2.framework.core as core +import unittest +import numpy as np + + +class TestSelectedRows(unittest.TestCase): + def test_selected_rows(self): + place = core.CPUPlace() + height = 10 + rows = [0, 4, 7] + row_numel = 10 + selcted_rows = core.SelectedRows(rows, row_numel) + np_array = np.ones((len(rows), height)).astype("float32") + np_array[0, 0] = 2.0 + np_array[2, 8] = 4.0 + tensor = selcted_rows.get_tensor() + tensor.set(np_array, place) + + # compare rows + self.assertEqual(0, selcted_rows.rows()[0]) + self.assertEqual(4, selcted_rows.rows()[1]) + self.assertEqual(7, selcted_rows.rows()[2]) + + # compare height + self.assertEqual(10, selcted_rows.height()) + + # compare tensor + self.assertAlmostEqual(2.0, + selcted_rows.get_tensor().get_float_element(0)) + self.assertAlmostEqual(1.0, + selcted_rows.get_tensor().get_float_element(1)) + self.assertAlmostEqual( + 4.0, selcted_rows.get_tensor().get_float_element(2 * row_numel + 8)) + + +if __name__ == "__main__": + unittest.main() From eb27c735cb95accf5abdf97fb35b8e7c39519856 Mon Sep 17 00:00:00 2001 From: qijun Date: Mon, 16 Oct 2017 13:35:28 -0700 Subject: [PATCH 33/63] remove unused C++ class OpRegistrar --- paddle/framework/op_registry.h | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/paddle/framework/op_registry.h b/paddle/framework/op_registry.h index 226e8ddcd4..0bda87dfa1 100644 --- a/paddle/framework/op_registry.h +++ b/paddle/framework/op_registry.h @@ -82,16 +82,6 @@ class OpRegistry { static std::unique_ptr CreateOp(const OpDescBind& op_desc); }; -template -class OpRegistrar : public Registrar { - public: - explicit OpRegistrar(const char* op_type) { OpRegistrar(op_type, ""); } - OpRegistrar(const char* op_type, const char* grad_op_type) { - OpRegistry::RegisterOp(op_type, - grad_op_type); - } -}; - template struct OpKernelRegistrarFunctor; From 186d1655650c8e07615609c5f299ffc15512ad74 Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Mon, 16 Oct 2017 14:53:05 -0700 Subject: [PATCH 34/63] AttributeChecker Better error log and speicalize bool (#4840) * AttributeChecker Better error log and speicalize bool Since lots of types can be cast to bool * add FIXME comment --- paddle/framework/attribute.h | 56 ++++++++++++++++++++++++++++++++++-- 1 file changed, 54 insertions(+), 2 deletions(-) diff --git a/paddle/framework/attribute.h b/paddle/framework/attribute.h index d13530e340..8a7a949346 100644 --- a/paddle/framework/attribute.h +++ b/paddle/framework/attribute.h @@ -120,6 +120,57 @@ class EnumInContainer { std::unordered_set container_; }; +template +struct ExtractAttribute { + explicit ExtractAttribute(const std::string& attr_name) + : attr_name_(attr_name) {} + + T* operator()(Attribute& attr) const { + T* attr_value = nullptr; + try { + attr_value = &boost::get(attr); + } catch (boost::bad_get& bad_get) { + PADDLE_THROW("Cannot get attribute %s by type %s, its type is %s", + attr_name_, typeid(T).name(), attr.type().name()); + } + return attr_value; + } + + const std::string& attr_name_; +}; + +// special handle bool +// FIXME(yuyang18): Currently we cast bool into int in python binding. It is +// hard to change the logic there. In another way, we should correct handle +// if the user set `some_flag=1`. +// +// FIX ME anytime if there is a better solution. +template <> +struct ExtractAttribute { + explicit ExtractAttribute(const std::string& attr_name) + : attr_name_(attr_name) {} + + bool* operator()(Attribute& attr) const { + if (attr.type() == typeid(int)) { // NOLINT + int val = boost::get(attr); + attr = static_cast(val); + } else if (attr.type() == typeid(float)) { // NOLINT + float val = boost::get(attr); + attr = static_cast(val); + } + bool* attr_value = nullptr; + try { + attr_value = &boost::get(attr); + } catch (boost::bad_get& bad_get) { + PADDLE_THROW("Cannot get attribute %s by type bool, its type is %s", + attr_name_, attr.type().name()); + } + return attr_value; + } + + const std::string& attr_name_; +}; + // check whether a certain attribute fit its limits // an attribute can have more than one limits template @@ -171,9 +222,10 @@ class TypedAttrChecker { attr_map[attr_name_] = val; } Attribute& attr = attr_map.at(attr_name_); - T& attr_value = boost::get(attr); + ExtractAttribute extract_attr(attr_name_); + T* attr_value = extract_attr(attr); for (const auto& checker : value_checkers_) { - checker(attr_value); + checker(*attr_value); } } From 790b9ce4c5532d5e10e2911581b9358554e5de29 Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Mon, 16 Oct 2017 15:01:05 -0700 Subject: [PATCH 35/63] Update design doc for Python Layer (#4698) * Update design doc for Python Layer * Update document --- doc/design/python_api.md | 114 ++++++++++++++++++++++++++++++--------- 1 file changed, 89 insertions(+), 25 deletions(-) diff --git a/doc/design/python_api.md b/doc/design/python_api.md index 56ae1d925a..cb5fdc765b 100644 --- a/doc/design/python_api.md +++ b/doc/design/python_api.md @@ -179,40 +179,104 @@ init_attr={ `optimize_op_attrs` is not in the `VarDesc` message, but kept in the Python instance, as it will be used in the Python space when creating the optimize operator's `OpDesc`, and will be in the `OpDesc` message. -## Layer Functions +## Layer Function -A layer is a Python function that creates some operators and variables. Layers simplify the work of application programmers. +A layer is a Python function that creates some operators and variables. Layers simplify the work of application programmers. -### Data Layer +Layer functions take `Variable` and configuration parameters as its input and return the output variable(s). + +For example, `FullyConnected` take one or more variable as its input. The input could be input data or another layer's output. There are many configuration options for a `FullyConnected` layer, such as layer size, activation, parameter names, initialization strategies of parameters, and so on. The `FullyConnected` layer will return an output variable. + + +### Necessity for reusing code between layer functions + +There are a lot of code that can be reused. Such as + +* Give the default value of configuration. e.g., default initialize strategy for parameters is uniform random with `min = -1.0`, `max = 1.0`. and default initialize strategy for bias is to fill zero. +* Append the activation operator. +* Create a temporary variable. +* Create parameter. +* Generate a unique name. +* Add a bias. +* ... + +A mechanism to reuse code between layer functions is necessary. It will be around [150 lines of code](https://github.com/PaddlePaddle/Paddle/pull/4724/files#diff-823b27e07e93914ada859232ae23f846R12) if we write a `FullyConnected` layer without any helper functions. + + + +### Comparision between global functions and helper class + +The `FullyConnected` layer will be as follow when we provide global functions: ```python -def data_layer(name, type, column_name): - block = the_current_program.glolal_block() - var = block.create_global_var( - name=name, - shape=[None] + type.dims(), - dtype=type.dtype) - block.prepend_operator(block, - type="Feed", - inputs = None, - outputs = [var], - {column_name: column_name}) - return var +def fc_layer(input, size, param_attr=None, bias_attr=None, act=None, name=None): + if name is None: + name = unique_name("fc") + input = multiple_input(input) + param_attr = default_param_attr(param_attr) + param_attr = multiple_param_attr(param_attr, len(input)) + + # mul + mul_results = [] + for ipt, attr in zip(input, param_attr): + shape = ipt.shape[1:] + [size] + w = g_program.global_block().create_parameter(shape, ipt.dtype, name, attr) + tmp = create_tmp_var(name) + g_program.current_block().append_op("mul", {ipt, w}, {tmp}) + mul_results.append(tmp) + + # add sum + ... + # add bias + ... + # add activation + ... + return out ``` -The input to the feed operator is a special variable in the global scope, which is the output of [Python readers](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/reader/README.md). +We can provide many helpers functions for layer developers. However, there are several disadvantages for global helper functions: + +1. We need a namespace for these methods, then layer developers can quickly figure out what method they can use. +2. Global functions will force layer developers to pass its parameter time by time. + +So we provide a helper class, `LayerHelper`, to share code between layer functions. The `FullyConnected` Layer will be as follow. + +```python +def fc_layer(input, size, param_attr=None, bias_attr=None, act=None, name=None): + helper = LayerHelper(locals()) # pass all parameter to LayerHelper + + mul_results = [] + for ipt, param in helper.iter_multiple_input_and_param(): + w = helper.create_parameter(shape=ipt.shape[1:] + [size], dtype = ipt.dtype) + tmp = helper.create_tmp_variable() + helper.append_op('mul', {ipt, w}, {tmp}) + mul_results.append(tmp) + + pre_bias = helper.add_sum(mul_results) + pre_activation = helper.add_bias(pre_bias) + return helper.add_activation(pre_activation) +``` + +We not only use the fewer lines of code to write `fc_layer` but also make the code clearer to understand. At the same time, layer developers can figure out what function they can invoke by typing `helper.` in a python editor. + + +### Implementation of layer helper -### FC Layer +We just keep all parameters of a layer function as a dictionary in layer helper as a private data member. Every method of layer helper will look up the dictionary after it is invoked. In that way, we can implement a layer helper for all layer functions even some layer does not contain some operator. For example, The `activation` is used by the FullyConnected layer or convolution layers, but a cross-entropy layer does not use it. The example code of `add_activation` are: ```python -def fc_layer(input, size, ...): - block = program.current_block() - w = block.create_parameter(...) - b = block.create_parameter(...) - out = block.create_var() - op = block.append_operator("FC", X=input, W=w, b=b, out=out) - out.writer = op - return out +class LayerHelper(object): + def __init__(self, **kwargs): # kwargs is short for `keyword arguments` + self.kwargs = kwargs + + def add_activation(self, input_var): + act = self.kwargs.get("act", None) # default value is None + if act is None: # do nothing if no act + return input_var + + tmp = self.create_tmp_var(self) + self.append_op(type=act, input=input_var, output=tmp) + return tmp ``` ## Optimizer From 75d0c7901530248afbcad7fc79c12241c1e4f00d Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Mon, 16 Oct 2017 15:01:39 -0700 Subject: [PATCH 36/63] Change Name convention of operator attributes (#4807) * Change dataType to data_type Follow PEP8 * Change name_convention to fit PEP8 --- paddle/framework/backward.cc | 2 +- paddle/operators/cross_entropy_op.cc | 22 ++++++++-------- paddle/operators/cross_entropy_op.cu | 4 +-- paddle/operators/cross_entropy_op.h | 4 +-- paddle/operators/fill_constant_op.cc | 4 +-- paddle/operators/name_convention.md | 2 +- paddle/operators/pool_op.cc | 26 +++++++++---------- paddle/operators/pool_op.h | 8 +++--- paddle/operators/pool_with_index_op.cc | 18 ++++++------- paddle/operators/pool_with_index_op.h | 4 +-- .../softmax_with_cross_entropy_op.cc | 14 +++++----- .../softmax_with_cross_entropy_op.cu | 4 +-- .../operators/softmax_with_cross_entropy_op.h | 4 +-- .../framework/tests/test_cross_entropy_op.py | 4 +-- .../v2/framework/tests/test_pool2d_op.py | 4 +-- .../v2/framework/tests/test_pool3d_op.py | 4 +-- .../v2/framework/tests/test_pool_max_op.py | 2 +- .../test_softmax_with_cross_entropy_op.py | 2 +- 18 files changed, 66 insertions(+), 66 deletions(-) diff --git a/paddle/framework/backward.cc b/paddle/framework/backward.cc index e3d7dacd7f..c78e056071 100644 --- a/paddle/framework/backward.cc +++ b/paddle/framework/backward.cc @@ -433,7 +433,7 @@ ParamGradInfoMap AppendBackward( new OpDescBind("fill_constant", {}, {{"Out", {fill_one_op_out}}}, {{"shape", std::vector{1}}, {"value", static_cast(1.0)}, - {"dataType", framework::DataType::FP32}})); + {"data_type", framework::DataType::FP32}})); all_ops.push_back(std::move(fill_one_op)); size_t forward_op_num = all_ops.size(); size_t forward_block_num = program_desc.Size(); diff --git a/paddle/operators/cross_entropy_op.cc b/paddle/operators/cross_entropy_op.cc index 708e80e96a..923ae5be6c 100644 --- a/paddle/operators/cross_entropy_op.cc +++ b/paddle/operators/cross_entropy_op.cc @@ -34,13 +34,13 @@ class CrossEntropyOp : public framework::OperatorWithKernel { PADDLE_ENFORCE_EQ(x_dims[0], label_dims[0], "The 1st dimension of Input(X) and Input(Label) should " "be equal."); - if (ctx->Attrs().Get("softLabel")) { + if (ctx->Attrs().Get("soft_label")) { PADDLE_ENFORCE_EQ(x_dims[1], label_dims[1], - "If Attr(softLabel) == true, the 2nd dimension of " + "If Attr(soft_label) == true, the 2nd dimension of " "Input(X) and Input(Label) should be equal."); } else { PADDLE_ENFORCE_EQ(label_dims[1], 1, - "If Attr(softLabel) == false, the 2nd dimension of " + "If Attr(soft_label) == false, the 2nd dimension of " "Input(Label) should be 1."); } @@ -82,13 +82,13 @@ class CrossEntropyGradientOp : public framework::OperatorWithKernel { "be equal."); PADDLE_ENFORCE_EQ(dy_dims[1], 1, "The 2nd dimension of Input(Y@Grad) should be 1."); - if (ctx->Attrs().Get("softLabel")) { + if (ctx->Attrs().Get("soft_label")) { PADDLE_ENFORCE_EQ(x_dims[1], label_dims[1], - "When Attr(softLabel) == true, the 2nd dimension of " + "When Attr(soft_label) == true, the 2nd dimension of " "Input(X) and Input(Label) should be equal."); } else { PADDLE_ENFORCE_EQ(label_dims[1], 1, - "When Attr(softLabel) == false, the 2nd dimension of " + "When Attr(soft_label) == false, the 2nd dimension of " "Input(Label) should be 1."); } ctx->SetOutputDim(framework::GradVarName("X"), x_dims); @@ -115,15 +115,15 @@ class CrossEntropyOpMaker : public framework::OpProtoAndCheckerMaker { "Label", "(Tensor, default Tensor), the ground truth which is " "a 2-D tensor. " - "When softLabel is set to false, `Label` is a Tensor with shape " + "When soft_label is set to false, `Label` is a Tensor with shape " "[N x 1]. " - "When softLabel is set to true, `Label` is a Tensor " + "When soft_label is set to true, `Label` is a Tensor " "with shape [N x K]."); AddOutput("Y", "(Tensor, default Tensor), a 2-D tensor " "with shape [N x 1]. The cross entropy loss."); AddAttr( - "softLabel", + "soft_label", "(bool, default false), a flag to indicate whether to interpretate " "the given labels as soft labels.") .SetDefault(false); @@ -133,12 +133,12 @@ CrossEntropy Operator. It supports both standard cross-entropy and soft-label cross-entropy loss computation. 1) One-hot cross-entropy: - softLabel = false, Label[i, 0] indicates the class index for sample i: + soft_label = false, Label[i, 0] indicates the class index for sample i: Y[i] = -log(X[i, Label[i]]) 2) Soft-label cross-entropy: - softLabel = true, Label[i, j] indicates the soft label of class j + soft_label = true, Label[i, j] indicates the soft label of class j for sample i: Y[i] = \sum_j{-Label[i, j] * log(X[i, j])} diff --git a/paddle/operators/cross_entropy_op.cu b/paddle/operators/cross_entropy_op.cu index 07b0388b60..c492dddb09 100644 --- a/paddle/operators/cross_entropy_op.cu +++ b/paddle/operators/cross_entropy_op.cu @@ -56,7 +56,7 @@ class CrossEntropyOpCUDAKernel : public framework::OpKernel { y->mutable_data(ctx.GetPlace()); math::CrossEntropyFunctor()( - ctx.device_context(), y, x, label, ctx.Attr("softLabel")); + ctx.device_context(), y, x, label, ctx.Attr("soft_label")); } }; @@ -83,7 +83,7 @@ class CrossEntropyGradientOpCUDAKernel : public framework::OpKernel { int block = 512; int grid = (batch_size * class_num + block - 1) / block; - if (ctx.Attr("softLabel")) { + if (ctx.Attr("soft_label")) { auto* label_data = label->data(); SoftCrossEntropyGradientKernel<<< grid, block, 0, reinterpret_cast( diff --git a/paddle/operators/cross_entropy_op.h b/paddle/operators/cross_entropy_op.h index 19c276d23f..42f282103b 100644 --- a/paddle/operators/cross_entropy_op.h +++ b/paddle/operators/cross_entropy_op.h @@ -38,7 +38,7 @@ class CrossEntropyOpKernel : public framework::OpKernel { y->mutable_data(ctx.GetPlace()); math::CrossEntropyFunctor()( - ctx.device_context(), y, x, labels, ctx.Attr("softLabel")); + ctx.device_context(), y, x, labels, ctx.Attr("soft_label")); } }; @@ -55,7 +55,7 @@ class CrossEntropyGradientOpKernel : public framework::OpKernel { T* dx_data = dx->mutable_data(ctx.GetPlace()); int class_num = x->dims()[1]; - if (ctx.Attr("softLabel")) { + if (ctx.Attr("soft_label")) { auto x_mat = EigenMatrix::From(*x); auto dy_mat = EigenMatrix::From(*dy); auto lbl_mat = EigenMatrix::From(*label); diff --git a/paddle/operators/fill_constant_op.cc b/paddle/operators/fill_constant_op.cc index 65d03d5fa4..a56832e202 100644 --- a/paddle/operators/fill_constant_op.cc +++ b/paddle/operators/fill_constant_op.cc @@ -35,7 +35,7 @@ class FillConstantOp : public framework::OperatorWithKernel { framework::DataType IndicateDataType( const framework::ExecutionContext &ctx) const override { - return static_cast(ctx.Attr("dataType")); + return static_cast(ctx.Attr("data_type")); } }; @@ -44,7 +44,7 @@ class FillConstantOpMaker : public framework::OpProtoAndCheckerMaker { FillConstantOpMaker(framework::OpProto *proto, framework::OpAttrChecker *op_checker) : framework::OpProtoAndCheckerMaker(proto, op_checker) { - AddAttr("dataType", + AddAttr("data_type", "(int, default 5 (FP32)) " "Output data type") .SetDefault(framework::DataType::FP32); diff --git a/paddle/operators/name_convention.md b/paddle/operators/name_convention.md index 379385dc5d..5a21690795 100644 --- a/paddle/operators/name_convention.md +++ b/paddle/operators/name_convention.md @@ -11,7 +11,7 @@ When defining an operator in Paddle, a corresponding [OpProtoMaker](https://gith - If an operator's Input/Output are tensors in math, not match to any meaningful words, input name should starts from `X`. e.g. `X`, `Y`, and output name should starts from `Out`. e.g. `Out`. This rule intends making operators which have few inputs/outputs unified. - Attribute. - - Attribute name follows the **camelCase**. e.g. `x`, `y`, `axis`, `rowwiseMatrix`. Also, attribute name prefers to meaningful English words. + - Attribute name follows the **snake_case**. e.g. `x`, `y`, `axis`, `rowwise_matrix`. Also, attribute name prefers to meaningful English words. - Comments. - Input/Output/Attr comment follow the format of **(type,default value) usage**, corresponding to which type it can be and how it will be used in the operator. e.g. Attribute in Accumulator`"gamma" `,`(float, default 1.0) Accumulation multiplier`. diff --git a/paddle/operators/pool_op.cc b/paddle/operators/pool_op.cc index c6d9aae133..a326839c0f 100644 --- a/paddle/operators/pool_op.cc +++ b/paddle/operators/pool_op.cc @@ -29,7 +29,7 @@ void PoolOp::InferShape(framework::InferShapeContext *ctx) const { auto in_x_dims = ctx->GetInputDim("X"); - std::string pooling_type = ctx->Attrs().Get("poolingType"); + std::string pooling_type = ctx->Attrs().Get("pooling_type"); std::vector ksize = ctx->Attrs().Get>("ksize"); std::vector strides = ctx->Attrs().Get>("strides"); std::vector paddings = ctx->Attrs().Get>("paddings"); @@ -37,7 +37,7 @@ void PoolOp::InferShape(framework::InferShapeContext *ctx) const { PADDLE_ENFORCE(in_x_dims.size() == 4 || in_x_dims.size() == 5, "Pooling intput should be 4-D or 5-D tensor."); - if (ctx->Attrs().Get("globalPooling")) { + if (ctx->Attrs().Get("global_pooling")) { ksize.resize(static_cast(in_x_dims.size()) - 2); for (size_t i = 0; i < ksize.size(); ++i) ksize[i] = static_cast(in_x_dims[i + 2]); @@ -80,23 +80,23 @@ Pool2dOpMaker::Pool2dOpMaker(framework::OpProto *proto, "the number of channels, H and W is the height and " "width of feature."); - AddAttr("poolingType", - "PoolingType of pooling operator." + AddAttr("pooling_type", + "Pooling_type of pooling operator." "Str constant equal to 'max' or 'avg'.") .InEnum({"max", "avg"}); AddAttr>( "ksize", "The pooling window size(height, width) of pooling operator." - "If globalPooling = true, ksize is ignored and need not be " + "If global_pooling = true, ksize is ignored and need not be " "specified."); // TODO(Chengduo): Add checker. (Currently, // TypedAttrChecker don't support vector type.) AddAttr( - "globalPooling", - "Whether to use the globalPooling." + "global_pooling", + "Whether to use the global_pooling." "Bool constant equal to false or true." "Default false." - "If globalPooling = true, ksize is ignored and need not be specified.") + "If global_pooling = true, ksize is ignored and need not be specified.") .SetDefault(false); AddAttr>("strides", "The strides(height, width) of pooling window." @@ -146,7 +146,7 @@ Pool3dOpMaker::Pool3dOpMaker(framework::OpProto *proto, "the number of channels, D, H and W is the depth, height and " "width of feature."); - AddAttr("poolingType", + AddAttr("pooling_type", "PoolingType of pooling operator." "Str constant equal to 'max' or 'avg'.") .InEnum({"max", "avg"}); @@ -154,15 +154,15 @@ Pool3dOpMaker::Pool3dOpMaker(framework::OpProto *proto, AddAttr>( "ksize", "The pooling window size(depth, height, width) of pooling operator." - "If globalPooling = true, ksize is ignored and need not be " + "If global_pooling = true, ksize is ignored and need not be " "specified."); // TODO(Chengduo): Add checker. (Currently, // TypedAttrChecker don't support vector type.) AddAttr( - "globalPooling", - "Whether to use the globalPooling." + "global_pooling", + "Whether to use the global_pooling." "Bool constant equal to false or true." "Default false." - "If globalPooling = true, ksize is ignored and need not be specified.") + "If global_pooling = true, ksize is ignored and need not be specified.") .SetDefault(false); AddAttr>("strides", "Strides(depth, height, width) of pooling operator." diff --git a/paddle/operators/pool_op.h b/paddle/operators/pool_op.h index e5016d573d..f6169e05b3 100644 --- a/paddle/operators/pool_op.h +++ b/paddle/operators/pool_op.h @@ -59,11 +59,11 @@ class PoolKernel : public framework::OpKernel { const Tensor* in_x = context.Input("X"); Tensor* out = context.Output("Out"); - std::string pooling_type = context.Attr("poolingType"); + std::string pooling_type = context.Attr("pooling_type"); std::vector ksize = context.Attr>("ksize"); std::vector strides = context.Attr>("strides"); std::vector paddings = context.Attr>("paddings"); - if (context.Attr("globalPooling")) { + if (context.Attr("global_pooling")) { for (size_t i = 0; i < ksize.size(); ++i) { ksize[i] = static_cast(in_x->dims()[i + 2]); } @@ -119,12 +119,12 @@ class PoolGradKernel : public framework::OpKernel { context.Input(framework::GradVarName("Out")); Tensor* in_x_grad = context.Output(framework::GradVarName("X")); - std::string pooling_type = context.Attr("poolingType"); + std::string pooling_type = context.Attr("pooling_type"); std::vector ksize = context.Attr>("ksize"); std::vector strides = context.Attr>("strides"); std::vector paddings = context.Attr>("paddings"); - if (context.Attr("globalPooling")) { + if (context.Attr("global_pooling")) { for (size_t i = 0; i < ksize.size(); ++i) ksize[i] = static_cast(in_x->dims()[i + 2]); } diff --git a/paddle/operators/pool_with_index_op.cc b/paddle/operators/pool_with_index_op.cc index 005ee88693..8eee20c2a9 100644 --- a/paddle/operators/pool_with_index_op.cc +++ b/paddle/operators/pool_with_index_op.cc @@ -45,7 +45,7 @@ class MaxPoolWithIndexOp : public framework::OperatorWithKernel { PADDLE_ENFORCE(in_x_dims.size() == 4 || in_x_dims.size() == 5, "Pooling intput should be 4-D or 5-D tensor."); - if (ctx->Attrs().Get("globalPooling")) { + if (ctx->Attrs().Get("global_pooling")) { ksize.resize(static_cast(in_x_dims.size()) - 2); for (size_t i = 0; i < ksize.size(); ++i) ksize[i] = static_cast(in_x_dims[i + 2]); @@ -108,15 +108,15 @@ class MaxPool2dWithIndexOpMaker : public framework::OpProtoAndCheckerMaker { AddAttr>( "ksize", "The pooling window size(height, width) of pooling operator." - "If globalPooling = true, ksize is ignored and need not be " + "If global_pooling = true, ksize is ignored and need not be " "specified."); // TODO(Chengduo): Add checker. (Currently, // TypedAttrChecker don't support vector type.) AddAttr( - "globalPooling", - "Whether to use the globalPooling." + "global_pooling", + "Whether to use the global_pooling." "Bool constant equal to false or true." "Default false." - "If globalPooling = true, ksize is ignored and need not be specified.") + "If global_pooling = true, ksize is ignored and need not be specified.") .SetDefault(false); AddAttr>("strides", "The strides(height, width) of pooling window." @@ -179,15 +179,15 @@ class MaxPool3dWithIndexOpMaker : public framework::OpProtoAndCheckerMaker { AddAttr>( "ksize", "The pooling window size(depth, height, width) of pooling operator." - "If globalPooling = true, ksize is ignored and need not be " + "If global_pooling = true, ksize is ignored and need not be " "specified."); // TODO(Chengduo): Add checker. (Currently, // TypedAttrChecker don't support vector type.) AddAttr( - "globalPooling", - "Whether to use the globalPooling." + "global_pooling", + "Whether to use the global_pooling." "Bool constant equal to false or true." "Default false." - "If globalPooling = true, ksize is ignored and need not be specified.") + "If global_pooling = true, ksize is ignored and need not be specified.") .SetDefault(false); AddAttr>( "strides", diff --git a/paddle/operators/pool_with_index_op.h b/paddle/operators/pool_with_index_op.h index 01b961ca82..455c453efc 100644 --- a/paddle/operators/pool_with_index_op.h +++ b/paddle/operators/pool_with_index_op.h @@ -35,7 +35,7 @@ class MaxPoolWithIndexKernel : public framework::OpKernel { std::vector ksize = context.Attr>("ksize"); std::vector strides = context.Attr>("strides"); std::vector paddings = context.Attr>("paddings"); - if (context.Attr("globalPooling")) { + if (context.Attr("global_pooling")) { for (size_t i = 0; i < ksize.size(); ++i) { ksize[i] = static_cast(in_x->dims()[i + 2]); } @@ -70,7 +70,7 @@ class MaxPoolWithIndexGradKernel : public framework::OpKernel { std::vector ksize = context.Attr>("ksize"); std::vector strides = context.Attr>("strides"); std::vector paddings = context.Attr>("paddings"); - if (context.Attr("globalPooling")) { + if (context.Attr("global_pooling")) { for (size_t i = 0; i < ksize.size(); ++i) { ksize[i] = static_cast(in_x_grad->dims()[i + 2]); } diff --git a/paddle/operators/softmax_with_cross_entropy_op.cc b/paddle/operators/softmax_with_cross_entropy_op.cc index 5431a1657c..9121609f10 100644 --- a/paddle/operators/softmax_with_cross_entropy_op.cc +++ b/paddle/operators/softmax_with_cross_entropy_op.cc @@ -46,7 +46,7 @@ class SoftmaxWithCrossEntropyOpMaker "(Tensor, default: Tensor), A 2-D tensor. The cross " "entropy loss with shape [N x 1]."); AddAttr( - "softLabel", + "soft_label", "(bool, default: false), A flag to indicate whether to interpretate " "the given labels as soft labels.") .SetDefault(false); @@ -100,13 +100,13 @@ class SoftmaxWithCrossEntropyOp : public framework::OperatorWithKernel { PADDLE_ENFORCE_EQ(labels_dims.size(), 2UL, "The labels should be a 2-D tensor."); - if (ctx->Attrs().Get("softLabel")) { + if (ctx->Attrs().Get("soft_label")) { PADDLE_ENFORCE_EQ(logits_dims[1], labels_dims[1], - "If Attr(softLabel) == true, the 2nd dimension of " + "If Attr(soft_label) == true, the 2nd dimension of " "Input(X) and Input(Label) should be equal."); } else { PADDLE_ENFORCE_EQ(labels_dims[1], 1UL, - "If Attr(softLabel) == false, the 2nd dimension of " + "If Attr(soft_label) == false, the 2nd dimension of " "Input(Label) should be 1."); } @@ -142,13 +142,13 @@ class SoftmaxWithCrossEntropyOpGrad : public framework::OperatorWithKernel { PADDLE_ENFORCE_EQ(labels_dims.size(), 2UL, "The labels should be a 2-D tensor."); - if (ctx->Attrs().Get("softLabel")) { + if (ctx->Attrs().Get("soft_label")) { PADDLE_ENFORCE_EQ(softmax_dims[1], labels_dims[1], - "When Attr(softLabel) == true, the 2nd dimension of " + "When Attr(soft_label) == true, the 2nd dimension of " "Input(X) and Input(Label) should be equal."); } else { PADDLE_ENFORCE_EQ(labels_dims[1], 1UL, - "When Attr(softLabel) == false, the 2nd dimension of " + "When Attr(soft_label) == false, the 2nd dimension of " "Input(Label) should be 1."); } diff --git a/paddle/operators/softmax_with_cross_entropy_op.cu b/paddle/operators/softmax_with_cross_entropy_op.cu index 2bc53ecf87..d03a1a7658 100644 --- a/paddle/operators/softmax_with_cross_entropy_op.cu +++ b/paddle/operators/softmax_with_cross_entropy_op.cu @@ -70,7 +70,7 @@ class SoftmaxWithCrossEntropyCUDAKernel : public framework::OpKernel { logits, softmax); math::CrossEntropyFunctor()( context.device_context(), loss, softmax, labels, - context.Attr("softLabel")); + context.Attr("soft_label")); } }; @@ -93,7 +93,7 @@ class SoftmaxWithCrossEntropyGradCUDAKernel : public framework::OpKernel { int block = 512; int grid = (batch_size * class_num + block - 1) / block; - if (context.Attr("softLabel")) { + if (context.Attr("soft_label")) { const T* label_data = labels->data(); SoftCrossEntropyGradientKernel<<< grid, block, 0, reinterpret_cast( diff --git a/paddle/operators/softmax_with_cross_entropy_op.h b/paddle/operators/softmax_with_cross_entropy_op.h index cffd422f18..66d7bc1569 100644 --- a/paddle/operators/softmax_with_cross_entropy_op.h +++ b/paddle/operators/softmax_with_cross_entropy_op.h @@ -44,7 +44,7 @@ class SoftmaxWithCrossEntropyKernel : public framework::OpKernel { logits, softmax); math::CrossEntropyFunctor()( context.device_context(), loss, softmax, labels, - context.Attr("softLabel")); + context.Attr("soft_label")); } }; @@ -60,7 +60,7 @@ class SoftmaxWithCrossEntropyGradKernel : public framework::OpKernel { logit_grad->ShareDataWith(*context.Input("Softmax")); const int class_num = logit_grad->dims()[1]; - if (context.Attr("softLabel")) { + if (context.Attr("soft_label")) { auto out_grad_mat = EigenMatrix::From(*out_grad); auto logit_grad_mat = EigenMatrix::From(*logit_grad); auto lbl_mat = EigenMatrix::From(*labels); diff --git a/python/paddle/v2/framework/tests/test_cross_entropy_op.py b/python/paddle/v2/framework/tests/test_cross_entropy_op.py index 4ea14da7fd..919b6c3f67 100644 --- a/python/paddle/v2/framework/tests/test_cross_entropy_op.py +++ b/python/paddle/v2/framework/tests/test_cross_entropy_op.py @@ -49,7 +49,7 @@ class TestCrossEntropyOp2(OpTest): self.inputs = {"X": X, "Label": label} self.outputs = {"Y": cross_entropy} - self.attrs = {"softLabel": True} + self.attrs = {"soft_label": True} def test_check_output(self): self.check_output() @@ -82,7 +82,7 @@ class TestCrossEntropyOp3(OpTest): self.inputs = {"X": X, "Label": label.astype(np.float32)} self.outputs = {"Y": cross_entropy} - self.attrs = {"softLabel": True} + self.attrs = {"soft_label": True} def test_check_output(self): self.check_output() diff --git a/python/paddle/v2/framework/tests/test_pool2d_op.py b/python/paddle/v2/framework/tests/test_pool2d_op.py index 2941fda81b..3fcd8941d4 100644 --- a/python/paddle/v2/framework/tests/test_pool2d_op.py +++ b/python/paddle/v2/framework/tests/test_pool2d_op.py @@ -56,8 +56,8 @@ class TestPool2d_Op(OpTest): 'strides': self.strides, 'paddings': self.paddings, 'ksize': self.ksize, - 'poolingType': self.pool_type, - 'globalPooling': self.global_pool, + 'pooling_type': self.pool_type, + 'global_pooling': self.global_pool, } self.outputs = {'Out': output} diff --git a/python/paddle/v2/framework/tests/test_pool3d_op.py b/python/paddle/v2/framework/tests/test_pool3d_op.py index 8792b492e3..f4e938041f 100644 --- a/python/paddle/v2/framework/tests/test_pool3d_op.py +++ b/python/paddle/v2/framework/tests/test_pool3d_op.py @@ -64,8 +64,8 @@ class TestPool3d_Op(OpTest): 'strides': self.strides, 'paddings': self.paddings, 'ksize': self.ksize, - 'poolingType': self.pool_type, - 'globalPooling': self.global_pool, + 'pooling_type': self.pool_type, + 'global_pooling': self.global_pool, } self.outputs = {'Out': output} diff --git a/python/paddle/v2/framework/tests/test_pool_max_op.py b/python/paddle/v2/framework/tests/test_pool_max_op.py index f0f8aa6089..b78f9bba05 100644 --- a/python/paddle/v2/framework/tests/test_pool_max_op.py +++ b/python/paddle/v2/framework/tests/test_pool_max_op.py @@ -86,7 +86,7 @@ class TestMaxPoolWithIndex_Op(OpTest): 'strides': self.strides, 'paddings': self.paddings, 'ksize': self.ksize, - 'globalPooling': self.global_pool, + 'global_pooling': self.global_pool, } self.inputs = {'X': input} diff --git a/python/paddle/v2/framework/tests/test_softmax_with_cross_entropy_op.py b/python/paddle/v2/framework/tests/test_softmax_with_cross_entropy_op.py index 377d07fb59..05ba954c0b 100644 --- a/python/paddle/v2/framework/tests/test_softmax_with_cross_entropy_op.py +++ b/python/paddle/v2/framework/tests/test_softmax_with_cross_entropy_op.py @@ -57,7 +57,7 @@ class TestSoftmaxWithCrossEntropyOp2(OpTest): self.inputs = {"Logits": logits, "Label": labels} self.outputs = {"Softmax": softmax, "Loss": cross_entropy} - self.attrs = {"softLabel": True} + self.attrs = {"soft_label": True} def test_check_output(self): self.check_output() From c65bdd95656fa4abee93d02429b8694787667f0e Mon Sep 17 00:00:00 2001 From: qijun Date: Mon, 16 Oct 2017 15:49:44 -0700 Subject: [PATCH 37/63] fix SelectedRows rows() method gpu runtime error --- paddle/pybind/pybind.cc | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/paddle/pybind/pybind.cc b/paddle/pybind/pybind.cc index 23e76011c9..f313eda964 100644 --- a/paddle/pybind/pybind.cc +++ b/paddle/pybind/pybind.cc @@ -153,7 +153,17 @@ PYBIND11_PLUGIN(core) { .def("set_height", &SelectedRows::set_height) .def("height", &SelectedRows::height) .def("set_rows", &SelectedRows::set_rows) - .def("rows", &SelectedRows::rows, py::return_value_policy::reference); + .def("rows", [](SelectedRows &self) { +#ifndef PADDLE_WITH_CUDA + return self.rows(); +#else + auto rows = self.rows(); + std::vector new_rows; + new_rows.reserve(rows.size()); + std::copy(rows.begin(), rows.end(), std::back_inserter(new_rows)); + return new_rows; +#endif + }); py::class_(m, "Variable", R"DOC(Variable Class. From 8e52b34a0c2f14175724571b89814277cbf28d59 Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Mon, 16 Oct 2017 16:08:01 -0700 Subject: [PATCH 38/63] Implement FC layer with helper (#4726) * Implement FC layer with helper * Update LayerHelper * Add debug string for Python ProtoBuf and Rename `Sync` to `Flush` * Add check of ProtoBuf initialization * Layer wrapper for FC * Fix unittest * Fix CI * Add code generator * AttributeChecker Better error log and speicalize bool Since lots of types can be cast to bool * Complete mlp, fit_a_line --- python/paddle/v2/framework/framework.py | 14 +- python/paddle/v2/framework/layer_helper.py | 160 ++++++++++++++++++ python/paddle/v2/framework/layers.py | 143 ++++++++++++++++ .../paddle/v2/framework/tests/test_layers.py | 43 +++++ .../v2/framework/tests/test_operator_desc.py | 2 +- 5 files changed, 354 insertions(+), 8 deletions(-) create mode 100644 python/paddle/v2/framework/layer_helper.py create mode 100644 python/paddle/v2/framework/layers.py create mode 100644 python/paddle/v2/framework/tests/test_layers.py diff --git a/python/paddle/v2/framework/framework.py b/python/paddle/v2/framework/framework.py index d649e69d58..a17f988bf4 100644 --- a/python/paddle/v2/framework/framework.py +++ b/python/paddle/v2/framework/framework.py @@ -153,7 +153,8 @@ class OpProtoHolder(object): self.op_proto_map[proto.type] = proto def get_op_proto(self, type): - assert type in self.op_proto_map, "Operator \"%s\" has not been registered." % type + if type not in self.op_proto_map: + raise ValueError("Operator \"%s\" has not been registered." % type) return self.op_proto_map[type] @@ -374,10 +375,10 @@ class Program(object): cls._instance = cls() return cls._instance - def __init__(self): - assert not hasattr(self.__class__, - '_instance'), 'Do not call constructor directly!' - self.desc = core.ProgramDesc.instance() + def __init__(self, desc=None): + if desc is None: + desc = core.ProgramDesc.instance() + self.desc = desc self.blocks = [Block(self, 0)] self.current_block_idx = 0 @@ -428,7 +429,6 @@ class Parameter(Variable): if each < 0: raise ValueError("Parameter shape should not be related with " "batch-size") - Variable.__init__(self, block, shape=shape, dtype=dtype, **kwargs) self.trainable = kwargs.get('trainable', True) self.init_attr = kwargs.get('initialize_attr', { @@ -441,7 +441,7 @@ class Parameter(Variable): self._append_initialize_ops_() def _append_initialize_ops_(self): - attr = copy.deepcopy(self.init_attr) + attr = self.init_attr op_type = attr.pop('type', None) block = self.block assert isinstance(block, Block) diff --git a/python/paddle/v2/framework/layer_helper.py b/python/paddle/v2/framework/layer_helper.py new file mode 100644 index 0000000000..26d3e04310 --- /dev/null +++ b/python/paddle/v2/framework/layer_helper.py @@ -0,0 +1,160 @@ +from paddle.v2.framework.framework import Variable, OpProtoHolder, g_program +import paddle.v2.framework.core as core +import copy +import itertools + + +def unique_name(prefix): + uid = core.unique_integer() # unique during whole process. + return "_".join([prefix, str(uid)]) + + +class LayerHelper(object): + def __init__(self, layer_type, **kwargs): + self.kwargs = kwargs + self.layer_type = layer_type + name = self.kwargs.get('name', None) + if name is None: + self.kwargs['name'] = unique_name(self.layer_type) + + @property + def name(self): + return self.kwargs['name'] + + @property + def program(self): + prog = self.kwargs.get('program', None) + if prog is None: + return g_program + else: + return prog + + def append_op(self, *args, **kwargs): + return self.program.current_block().append_op(*args, **kwargs) + + def multiple_input(self, input_param_name='input'): + inputs = self.kwargs.get(input_param_name, []) + type_error = TypeError( + "Input of {0} layer should be Variable or sequence of Variable". + format(self.layer_type)) + if isinstance(inputs, Variable): + inputs = [inputs] + elif not isinstance(inputs, list) and not isinstance(inputs, tuple): + raise type_error + else: + for each in inputs: + if not isinstance(each, Variable): + raise type_error + return inputs + + def input(self, input_param_name='input'): + inputs = self.multiple_input(input_param_name) + if len(inputs) != 1: + raise "{0} layer only takes one input".format(self.layer_type) + return inputs[0] + + @property + def param_attr(self): + default = { + 'name': None, + 'init_attr': { + 'type': 'uniform_random', + 'min': -1.0, + 'max': 1.0 + } + } + actual = self.kwargs.get('param_attr', None) + return actual if actual is not None else default + + def bias_attr(self, size, dtype): + bias_attr = self.kwargs.get('bias_attr', False) + if bias_attr is None or bias_attr: + bias_attr = { + 'name': None, + 'init_attr': { + 'type': 'fill_constant', + 'value': 0.0, + 'shape': [size], + 'dataType': dtype + } + } + return bias_attr + + def multiple_param_attr(self, length): + param_attr = self.param_attr + if isinstance(param_attr, dict): + param_attr = [param_attr] + + if len(param_attr) != 1 and len(param_attr) != length: + raise ValueError("parameter number mismatch") + elif len(param_attr) == 1 and length != 1: + tmp = [None] * length + for i in xrange(length): + tmp[i] = copy.deepcopy(param_attr[0]) + param_attr = tmp + return param_attr + + def iter_inputs_and_params(self, input_param_name='input'): + inputs = self.multiple_input(input_param_name) + param_attrs = self.multiple_param_attr(len(inputs)) + for ipt, param_attr in itertools.izip(inputs, param_attrs): + yield ipt, param_attr + + def input_dtype(self, input_param_name='input'): + inputs = self.multiple_input(input_param_name) + dtype = None + for each in inputs: + if dtype is None: + dtype = each.data_type + elif dtype != each.data_type: + raise ValueError("Data Type mismatch") + return dtype + + def create_parameter(self, attr, shape, dtype, suffix='w'): + if attr['name'] is None: + attr['name'] = unique_name(".".join([self.name, suffix])) + return self.program.global_block().create_parameter( + name=attr['name'], + dtype=dtype, + shape=shape, + initialize_attr=attr['init_attr']) + + def create_tmp_variable(self, dtype): + return self.program.current_block().create_var( + name=unique_name(".".join([self.name, 'tmp'])), dtype=dtype) + + def create_global_variable(self, *args, **kwargs): + return self.program.global_block().create_var(*args, **kwargs) + + def append_bias_op(self, input_var): + bias_attr = self.bias_attr( + self.kwargs['size'], dtype=input_var.data_type) + if not bias_attr: + return input_var + b = self.create_parameter( + attr=bias_attr, + shape=[self.kwargs['size']], + dtype=input_var.data_type, + suffix='b') + tmp = self.create_tmp_variable(dtype=input_var.data_type) + self.append_op( + type='elementwise_add', + inputs={'X': [input_var], + 'Y': [b]}, + outputs={'Out': [tmp]}) + return tmp + + def append_activation(self, input_var): + act = self.kwargs.get('act', None) + if act is None: + return input_var + if isinstance(act, basestring): + act = {'type': act} + tmp = self.create_tmp_variable(dtype=input_var.data_type) + act_type = act.pop('type') + self.append_op( + type=act_type, + inputs={"X": [input_var]}, + outputs={"Y": [tmp]}, + attrs=act) + return tmp diff --git a/python/paddle/v2/framework/layers.py b/python/paddle/v2/framework/layers.py new file mode 100644 index 0000000000..44b587b116 --- /dev/null +++ b/python/paddle/v2/framework/layers.py @@ -0,0 +1,143 @@ +from paddle.v2.framework.layer_helper import LayerHelper +import paddle.v2.framework.core as core +from paddle.v2.framework.framework import OpProtoHolder, Variable +import re + +__all__ = ['fc_layer', 'data_layer', 'cross_entropy'] + + +def fc_layer(input, + size, + param_attr=None, + bias_attr=True, + name=None, + act=None, + num_flatten_dims=1, + program=None): + # create helper + helper = LayerHelper('fc', **locals()) + + dtype = helper.input_dtype() + + # mul + mul_results = [] + for input_var, param_attr in helper.iter_inputs_and_params(): + input_shape = input_var.shape + param_shape = list(input_shape[num_flatten_dims:]) + [size] + w = helper.create_parameter( + attr=param_attr, shape=param_shape, dtype=dtype) + tmp = helper.create_tmp_variable(dtype) + helper.append_op( + type="mul", + inputs={ + "X": input_var, + "Y": w, + }, + outputs={"Out": tmp}, + attrs={'x_num_col_dims': num_flatten_dims}) + mul_results.append(tmp) + + # sum + if len(mul_results) == 1: + pre_bias = mul_results[0] + else: + pre_bias = helper.create_tmp_variable(dtype) + helper.append_op( + type="sum", inputs={"X": mul_results}, outputs={"Out": pre_bias}) + # add bias + pre_activation = helper.append_bias_op(pre_bias) + # add activation + return helper.append_activation(pre_activation) + + +def data_layer(name, + shape, + data_type='float32', + type=core.VarDesc.VarType.LOD_TENSOR, + program=None): + helper = LayerHelper('data', **locals()) + shape = [-1] + shape # append batch size as -1 + return helper.create_global_variable( + name=name, shape=shape, dtype=data_type, type=type) + + +def _convert_(name): + s1 = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', name) + return re.sub('([a-z0-9])([A-Z])', r'\1_\2', s1).lower() + + +def _create_op_func_(op_type): + op_proto = OpProtoHolder.instance().get_op_proto(op_type) + if len(op_proto.outputs) != 1: + raise ValueError( + "Only one output operator can be automatically generated") + + if op_proto.outputs[0].duplicable: + raise ValueError( + "Only not duplicable op can be automatically generated") + + o_name = op_proto.outputs[0].name + + def func(**kwargs): + helper = LayerHelper(op_type, **kwargs) + inputs = dict() + dtype = None + for ipt in op_proto.inputs: + name = _convert_(ipt.name) + val = kwargs.pop(name, []) + if not isinstance(val, list) and not isinstance(val, tuple): + val = [val] + for each in val: + if not isinstance(each, Variable): + raise ValueError("input of {0} must be variable".format( + op_type)) + + if dtype is None: + dtype = each.data_type + elif dtype != each.data_type: + raise ValueError( + "operator {0} must input same dtype".format(op_type)) + inputs[ipt.name] = val + + out = helper.create_tmp_variable(dtype=dtype) + helper.append_op( + type=op_type, inputs=inputs, outputs={o_name: [out]}, attrs=kwargs) + return out + + func.__name__ = op_type + globals()[op_type] = func + global __all__ + __all__.append(op_type) + + +_create_op_func_('mean') + + +def cross_entropy(input, label, **kwargs): + helper = LayerHelper('cross_entropy', **kwargs) + out = helper.create_tmp_variable(dtype=input.data_type) + helper.append_op( + type='cross_entropy', + inputs={'X': [input], + 'Label': [label]}, + outputs={'Y': [out]}, + attrs=kwargs) + return out + + +def square_error_cost(input, label, **kwargs): + helper = LayerHelper('square_error_cost', **kwargs) + minus_out = helper.create_tmp_variable(dtype=input.data_type) + helper.append_op( + type='elementwise_sub', + inputs={'X': [input], + 'Y': [label]}, + outputs={'Out': [minus_out]}) + + square_out = helper.create_tmp_variable(dtype=input.data_type) + helper.append_op( + type='pow', + inputs={'X': [minus_out]}, + outputs={'Y': [square_out]}, + attrs={'factor': 2.0}) + return square_out diff --git a/python/paddle/v2/framework/tests/test_layers.py b/python/paddle/v2/framework/tests/test_layers.py new file mode 100644 index 0000000000..1ef2591cca --- /dev/null +++ b/python/paddle/v2/framework/tests/test_layers.py @@ -0,0 +1,43 @@ +from paddle.v2.framework.layers import fc_layer, data_layer, cross_entropy, mean, square_error_cost +from paddle.v2.framework.framework import Program, g_program +import paddle.v2.framework.core as core +import unittest + + +class TestBook(unittest.TestCase): + def test_fit_a_line(self): + pd = core.ProgramDesc.__create_program_desc__() + program = Program(desc=pd) + x = data_layer( + name='x', shape=[13], data_type='float32', program=program) + y_predict = fc_layer(input=x, size=1, act=None, program=program) + + y = data_layer( + name='y', shape=[1], data_type='float32', program=program) + cost = square_error_cost(input=y_predict, label=y, program=program) + + avg_cost = mean(x=cost, program=program) + self.assertIsNotNone(avg_cost) + print str(program) + + def test_recognize_digits_mlp(self): + pd = core.ProgramDesc.__create_program_desc__() + program = Program(desc=pd) + + # Change g_program, so the rest layers use `g_program` + images = data_layer( + name='pixel', shape=[784], data_type='float32', program=program) + label = data_layer( + name='label', shape=[1], data_type='int32', program=program) + hidden1 = fc_layer(input=images, size=128, act='relu', program=program) + hidden2 = fc_layer(input=hidden1, size=64, act='relu', program=program) + predict = fc_layer( + input=hidden2, size=10, act='softmax', program=program) + cost = cross_entropy(input=predict, label=label, program=program) + avg_cost = mean(x=cost, program=program) + self.assertIsNotNone(avg_cost) + print str(program) + + +if __name__ == '__main__': + unittest.main() diff --git a/python/paddle/v2/framework/tests/test_operator_desc.py b/python/paddle/v2/framework/tests/test_operator_desc.py index dfe39c98f7..af4e980b8e 100644 --- a/python/paddle/v2/framework/tests/test_operator_desc.py +++ b/python/paddle/v2/framework/tests/test_operator_desc.py @@ -16,7 +16,7 @@ class TestOperator(unittest.TestCase): try: block.append_op(type="no_such_op") self.assertFail() - except AssertionError as a_err: + except ValueError as a_err: self.assertEqual(a_err.message, "Operator \"no_such_op\" has not been registered.") From b10cd435547290d6417b00688c190bc98169d1df Mon Sep 17 00:00:00 2001 From: Qiao Longfei Date: Mon, 16 Oct 2017 16:35:55 -0700 Subject: [PATCH 39/63] rm cpp executor_test, rewrite in python later (#4849) * rm cpp executor_test, rewrite in python later * remove executor_test code in CMakeList.txt --- paddle/framework/CMakeLists.txt | 7 - paddle/framework/executor_test.cc | 348 ------------------------------ 2 files changed, 355 deletions(-) delete mode 100644 paddle/framework/executor_test.cc diff --git a/paddle/framework/CMakeLists.txt b/paddle/framework/CMakeLists.txt index c8d9dac21d..6585fb7b5d 100644 --- a/paddle/framework/CMakeLists.txt +++ b/paddle/framework/CMakeLists.txt @@ -43,13 +43,6 @@ cc_library(backward SRCS backward.cc DEPS net_op) cc_test(backward_test SRCS backward_test.cc DEPS backward recurrent_op device_context) cc_library(executor SRCS executor.cc DEPS op_registry device_context scope framework_proto backward) -set(EXECUTOR_TEST_OP elementwise_add_op gaussian_random_op feed_op fetch_op - mul_op sum_op squared_l2_distance_op fill_constant_op sgd_op mean_op) -if(WITH_GPU) - nv_test(executor_test SRCS executor_test.cc DEPS executor ${EXECUTOR_TEST_OP}) -else() - cc_test(executor_test SRCS executor_test.cc DEPS executor ${EXECUTOR_TEST_OP}) -endif() cc_library(tensor_array SRCS tensor_array.cc DEPS lod_tensor) cc_test(tensor_array_test SRCS tensor_array_test.cc DEPS tensor_array place) diff --git a/paddle/framework/executor_test.cc b/paddle/framework/executor_test.cc deleted file mode 100644 index e08d31e361..0000000000 --- a/paddle/framework/executor_test.cc +++ /dev/null @@ -1,348 +0,0 @@ -/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. */ - -#include "paddle/framework/executor.h" - -#include -#include - -#include "gflags/gflags.h" -#include "gtest/gtest.h" -#include "paddle/framework/attribute.h" -#include "paddle/framework/backward.h" -#include "paddle/framework/block_desc.h" -#include "paddle/framework/op_desc.h" -#include "paddle/framework/op_registry.h" -#include "paddle/framework/operator.h" - -USE_OP(elementwise_add); -USE_OP(gaussian_random); -USE_NO_KERNEL_OP(feed); -USE_NO_KERNEL_OP(fetch); -USE_OP(mul); -USE_OP(sum); -USE_OP(squared_l2_distance); -USE_OP(fill_constant); -USE_OP(mean); -USE_OP(sgd); - -constexpr auto kFeedValueName = "feed_value"; -constexpr auto kFetchValueName = "fetch_value"; - -using namespace paddle::platform; -using namespace paddle::framework; - -void AddOp(const std::string& type, const VariableNameMap& inputs, - const VariableNameMap& outputs, AttributeMap attrs, - paddle::framework::BlockDescBind* block) { - // insert output - for (auto kv : outputs) { - for (auto v : kv.second) { - // <<<<<<< HEAD - // auto var = block->Var(v); - // var->SetType(VarDesc::LOD_TENSOR); - // var->SetDataType(paddle::framework::DataType::FP32); - // ======= - if (!block->HasVar(v)) { - auto var = block->Var(v); - var->SetDataType(paddle::framework::DataType::FP32); - } - // >>>>>>> origin/develop - } - } - - // insert op - auto op = block->AppendOp(); - op->SetType(type); - for (auto& kv : inputs) { - op->SetInput(kv.first, kv.second); - } - for (auto& kv : outputs) { - op->SetOutput(kv.first, kv.second); - } - op->SetAttrMap(attrs); - op->CheckAttrs(); -} - -// Tensors in feed value variable will only be in CPUPlace -// So we can memcpy the data from vector to feed_value -template -void SetFeedVariable(const std::vector>& inputs, - const std::vector>& dims) { - Variable* g_feed_value = GetGlobalScope().FindVar(kFeedValueName); - auto& feed_inputs = - *(g_feed_value->GetMutable>()); - size_t size = inputs.size(); - feed_inputs.resize(size); - for (size_t i = 0; i < size; i++) { - T* dst = feed_inputs[i].mutable_data(make_ddim(dims[i]), CPUPlace()); - memcpy(dst, inputs[i].data(), inputs[i].size() * sizeof(T)); - } -} - -// Tensors in fetch value variable will only be in CPUPlace -// So we can memcpy the data from fetch_value to vector -template -std::vector> GetFetchVariable() { - Variable* g_fetch_value = GetGlobalScope().FindVar(kFetchValueName); - auto& fetch_outputs = - *(g_fetch_value->GetMutable>()); - - size_t size = fetch_outputs.size(); - std::vector> result; - result.reserve(size); - for (size_t i = 0; i < size; i++) { - std::vector tmp; - tmp.resize(fetch_outputs[i].numel()); - memcpy(tmp.data(), fetch_outputs[i].data(), - fetch_outputs[i].numel() * sizeof(T)); - result.push_back(tmp); - } - - return result; -} - -class ExecutorTesterRandom : public ::testing::Test { - public: - virtual void SetUp() override { - int input_dim = 3, batch_size = 2, embed_dim = 5; - - auto temp_init_root_block = init_pdesc_.add_blocks(); - temp_init_root_block->set_idx(0); - temp_init_root_block->set_parent_idx(-1); - paddle::framework::ProgramDescBind& init_program = - paddle::framework::ProgramDescBind::Instance(&init_pdesc_); - paddle::framework::BlockDescBind* init_root_block = init_program.Block(0); - - AddOp("gaussian_random", {}, {{"Out", {"w1"}}}, - {{"dims", std::vector{input_dim, embed_dim}}}, init_root_block); - AddOp("gaussian_random", {}, {{"Out", {"w2"}}}, - {{"dims", std::vector{embed_dim, input_dim}}}, init_root_block); - AddOp("fetch", {{"Input", {"w1"}}}, {{"Out", {kFetchValueName}}}, - {{"col", 0}}, init_root_block); - AddOp("fetch", {{"Input", {"w2"}}}, {{"Out", {kFetchValueName}}}, - {{"col", 1}}, init_root_block); - - // flush - init_program.Proto(); - - // run block - auto temp_root_block = pdesc_.add_blocks(); - temp_root_block->set_idx(0); - temp_root_block->set_parent_idx(-1); - paddle::framework::ProgramDescBind& program = - paddle::framework::ProgramDescBind::Instance(&pdesc_); - paddle::framework::BlockDescBind* root_block = program.Block(0); - - // feed data - inputs_.push_back({1.0, 1.0, 1.0, 1.0, 1.0, 1.0}); - dims_.push_back({batch_size, input_dim}); - AddOp("feed", {{"Input", {kFeedValueName}}}, {{"Out", {"a"}}}, - {{"dims", std::vector{batch_size, input_dim}}, {"col", 0}}, - root_block); - - // forward - AddOp("mul", {{"X", {"a"}}, {"Y", {"w1"}}}, {{"Out", {"b"}}}, {}, - root_block); - AddOp("mul", {{"X", {"b"}}, {"Y", {"w2"}}}, {{"Out", {"a_out"}}}, {}, - root_block); - AddOp("squared_l2_distance", {{"X", {"a"}}, {"Y", {"a_out"}}}, - {{"Out", {"l2_distance"}}, {"sub_result", {"l2_distance_sub"}}}, {}, - root_block); - AddOp("mean", {{"X", {"l2_distance"}}}, {{"Out", {"mean_out"}}}, {}, - root_block); - - // backward - auto target = VarDescBind("mean_out"); - AppendBackward(program, target, {}); - - // update - AddOp("fill_constant", {}, {{"Out", {"learning_rate"}}}, - {{"shape", std::vector{1}}, {"value", float(0.001)}}, - root_block); - AddOp("sgd", {{"Param", {"w1"}}, - {"LearningRate", {"learning_rate"}}, - {"Grad", {"w1@GRAD"}}}, - {{"ParamOut", {"w1"}}}, {}, root_block); - AddOp("sgd", {{"Param", {"w2"}}, - {"LearningRate", {"learning_rate"}}, - {"Grad", {"w2@GRAD"}}}, - {{"ParamOut", {"w2"}}}, {}, root_block); - - AddOp("fetch", {{"Input", {"w1"}}}, {{"Out", {kFetchValueName}}}, - {{"col", 0}}, root_block); - AddOp("fetch", {{"Input", {"w2"}}}, {{"Out", {kFetchValueName}}}, - {{"col", 1}}, root_block); - AddOp("fetch", {{"Input", {"l2_distance"}}}, {{"Out", {kFetchValueName}}}, - {{"col", 0}}, root_block); - - // flush - program.Proto(); - } - - protected: - ProgramDesc init_pdesc_; - ProgramDesc pdesc_; - std::vector> inputs_; - std::vector> dims_; -}; - -class ExecutorTesterFeedAndFetch : public ::testing::Test { - public: - virtual void SetUp() override { - auto temp_root_block = pdesc_.add_blocks(); - temp_root_block->set_idx(0); - temp_root_block->set_parent_idx(-1); - - // wrap to BlockDescBind - paddle::framework::ProgramDescBind& program = - paddle::framework::ProgramDescBind::Instance(&pdesc_); - paddle::framework::BlockDescBind* root_block = program.Block(0); - - std::vector dim{6}; - - AddOp("feed", {{"Input", {kFeedValueName}}}, {{"Out", {"a"}}}, - {{"dims", dim}, {"col", 0}}, root_block); - AddOp("feed", {{"Input", {kFeedValueName}}}, {{"Out", {"b"}}}, - {{"dims", dim}, {"col", 1}}, root_block); - AddOp("fetch", {{"Input", {"a"}}}, {{"Out", {kFetchValueName}}}, - {{"col", 0}}, root_block); - AddOp("fetch", {{"Input", {"b"}}}, {{"Out", {kFetchValueName}}}, - {{"col", 1}}, root_block); - - // flush - program.Proto(); - - std::vector vec1 = {1.0, 2.0, 3.0, 4.0, 5.0, 6.0}; - std::vector vec2 = {4.0, 5.0, 6.0, 7.0, 8.0, 9.0}; - inputs_.push_back(vec1); - inputs_.push_back(vec2); - dims_.push_back({static_cast(vec1.size())}); - dims_.push_back({static_cast(vec2.size())}); - } - - protected: - ProgramDesc pdesc_; - std::vector> inputs_; - std::vector> dims_; -}; - -#ifndef PADDLE_WITH_CUDA -TEST_F(ExecutorTesterRandom, CPU) { - std::vector places; - CPUPlace cpu_place; - places.push_back(cpu_place); - - // We have a global Scope and BuddyAllocator, and we must ensure - // global BuddyAllocator is initialized before global Scope. Thus, - // global Scope will deconstruct before BuddyAllocator. Otherwise, - // "pointer being freed was not allocated" error will appear. - paddle::memory::Used(cpu_place); - - std::unique_ptr executor(new Executor(places)); - executor->Run(init_pdesc_, &GetGlobalScope(), 0); - SetFeedVariable(inputs_, dims_); - executor->Run(pdesc_, &GetGlobalScope(), 0); - std::vector> result = GetFetchVariable(); -} - -TEST_F(ExecutorTesterFeedAndFetch, CPU) { - std::vector places; - CPUPlace cpu_place; - places.emplace_back(cpu_place); - - // We have a global Scope and BuddyAllocator, and we must ensure - // global BuddyAllocator is initialized before global Scope. Thus, - // global Scope will deconstruct before BuddyAllocator. Otherwise, - // "pointer being freed was not allocated" error will appear. - paddle::memory::Used(cpu_place); - - std::unique_ptr executor(new Executor(places)); - - for (int batch_id = 0; batch_id < 3; batch_id++) { - SetFeedVariable(inputs_, dims_); - executor->Run(pdesc_, &GetGlobalScope(), 0); - std::vector> result = GetFetchVariable(); - ASSERT_EQ(result.size(), inputs_.size()); - for (size_t i = 0; i < result.size(); ++i) { - ASSERT_EQ(result[i].size(), inputs_[i].size()); - for (size_t j = 0; j < result[i].size(); ++j) { - ASSERT_EQ(result[i][j], inputs_[i][j]); - } - } - } -} -#else -TEST_F(ExecutorTesterRandom, GPU) { - std::vector places; - GPUPlace gpu_place(0); - places.push_back(gpu_place); - - // We have a global Scope and BuddyAllocator, and we must ensure - // global BuddyAllocator is initialized before global Scope. Thus, - // global Scope will deconstruct before BuddyAllocator. Otherwise, - // "pointer being freed was not allocated" error will appear. - // If paddle is compiled with GPU, both CPU and GPU BuddyAllocator - // need to be used at first. - paddle::memory::Used(CPUPlace()); - paddle::memory::Used(gpu_place); - - std::unique_ptr executor(new Executor(places)); - - executor->Run(init_pdesc_, &GetGlobalScope(), 0); - for (int batch_id = 0; batch_id < 3; batch_id++) { - SetFeedVariable(inputs_, dims_); - executor->Run(pdesc_, &GetGlobalScope(), 0); - } -} - -TEST_F(ExecutorTesterFeedAndFetch, GPU) { - std::vector places; - GPUPlace gpu_place(0); - places.push_back(gpu_place); - // We have a global Scope and BuddyAllocator, and we must ensure - // global BuddyAllocator is initialized before global Scope. Thus, - // global Scope will deconstruct before BuddyAllocator. Otherwise, - // "pointer being freed was not allocated" error will appear. - // If paddle is compiled with GPU, both CPU and GPU BuddyAllocator - // need to be used at first. - paddle::memory::Used(CPUPlace()); - paddle::memory::Used(gpu_place); - - std::unique_ptr executor(new Executor(places)); - - for (int batch_id = 0; batch_id < 3; batch_id++) { - SetFeedVariable(inputs_, dims_); - executor->Run(pdesc_, &GetGlobalScope(), 0); - std::vector> result = GetFetchVariable(); - PADDLE_ENFORCE_EQ(result.size(), inputs_.size()); - for (size_t i = 0; i < result.size(); ++i) { - PADDLE_ENFORCE_EQ(result[i].size(), inputs_[i].size()); - for (size_t j = 0; j < result[i].size(); ++j) { - PADDLE_ENFORCE_EQ(result[i][j], inputs_[i][j]); - } - } - } -} - -DECLARE_double(fraction_of_gpu_memory_to_use); - -int main(int argc, char** argv) { - testing::InitGoogleTest(&argc, argv); - // Use less GPU memory for unittest. - FLAGS_fraction_of_gpu_memory_to_use = 0.25; - return RUN_ALL_TESTS(); -} - -#endif From f06637002d0b1de8f6750bb40e15e1a88b7b33ae Mon Sep 17 00:00:00 2001 From: Yang Yang Date: Tue, 17 Oct 2017 01:07:08 +0000 Subject: [PATCH 40/63] simplify executor; pass compile --- paddle/framework/executor.cc | 94 ++++-------------------------------- paddle/framework/executor.h | 11 ----- 2 files changed, 9 insertions(+), 96 deletions(-) diff --git a/paddle/framework/executor.cc b/paddle/framework/executor.cc index 8e82e28bac..b3b85b5865 100644 --- a/paddle/framework/executor.cc +++ b/paddle/framework/executor.cc @@ -64,99 +64,23 @@ void Executor::Run(const ProgramDesc& pdesc, Scope* scope, int block_id) { auto& block = pdesc.blocks(block_id); auto& device = device_contexts_[0]; - // Instantiate all the vars in the global scope - for (auto& var : block.vars()) { - scope->Var(var.name()); - } - Scope& local_scope = scope->NewScope(); - std::vector should_run = Prune(pdesc, block_id); - PADDLE_ENFORCE_EQ(should_run.size(), static_cast(block.ops_size())); - for (size_t i = 0; i < should_run.size(); ++i) { - if (should_run[i]) { - for (auto& var : block.ops(i).outputs()) { - for (auto& argu : var.arguments()) { - if (local_scope.FindVar(argu) == nullptr) { - local_scope.Var(argu); - } - } - } - auto op = paddle::framework::OpRegistry::CreateOp(block.ops(i)); - op->Run(local_scope, *device); + for (auto& var : block.vars()) { + if (var.persistable()) { + scope->Var(var.name()); + } else { + local_scope.Var(var.name()); } } - // TODO(tonyyang-svail): - // - Destroy local_scope -} - -std::vector Prune(const ProgramDesc& pdesc, int block_id) { - // TODO(tonyyang-svail): - // - will change to use multiple blocks for RNN op and Cond Op - - auto& block = pdesc.blocks(block_id); - auto& ops = block.ops(); - - bool expect_feed = true; - for (auto& op_desc : ops) { - PADDLE_ENFORCE(op_desc.type() != kFeedOpType || expect_feed, - "All FeedOps are at the beginning of the ProgramDesc"); - expect_feed = (op_desc.type() == kFeedOpType); - } - - bool expect_fetch = true; - for (auto op_iter = ops.rbegin(); op_iter != ops.rend(); ++op_iter) { - auto& op_desc = *op_iter; - PADDLE_ENFORCE(op_desc.type() != kFetchOpType || expect_fetch, - "All FetchOps must at the end of the ProgramDesc"); - expect_fetch = (op_desc.type() == kFetchOpType); - } - - std::set dependent_vars; - std::vector should_run; - for (auto op_iter = ops.rbegin(); op_iter != ops.rend(); ++op_iter) { - auto& op_desc = *op_iter; - - bool found_dependent_vars = false; - for (auto& var : op_desc.outputs()) { - for (auto& argu : var.arguments()) { - if (dependent_vars.count(argu) != 0) { - found_dependent_vars = true; - } - } - } - - if (op_desc.type() == kFetchOpType || found_dependent_vars) { - // erase its output to the dependency graph - for (auto& var : op_desc.outputs()) { - for (auto& argu : var.arguments()) { - dependent_vars.erase(argu); - } - } - - // insert its input to the dependency graph - for (auto& var : op_desc.inputs()) { - for (auto& argu : var.arguments()) { - dependent_vars.insert(argu); - } - } - - should_run.push_back(true); - } else { - should_run.push_back(false); - } + for (auto& op_desc : block.ops()) { + auto op = paddle::framework::OpRegistry::CreateOp(op_desc); + op->Run(local_scope, *device); } // TODO(tonyyang-svail): - // - check this after integration of Init - // PADDLE_ENFORCE(dependent_vars.empty()); - - // since we are traversing the ProgramDesc in reverse order - // we reverse the should_run vector - std::reverse(should_run.begin(), should_run.end()); - - return should_run; + // - Destroy local_scope } } // namespace framework diff --git a/paddle/framework/executor.h b/paddle/framework/executor.h index 4e3bc2c0a5..793ee954e2 100644 --- a/paddle/framework/executor.h +++ b/paddle/framework/executor.h @@ -40,16 +40,5 @@ class Executor { std::vector device_contexts_; }; -/* @Brief - * Pruning the graph - * - * @param - * ProgramDesc - * - * @return - * vector Same size as ops. Indicates whether an op should be run. - */ -std::vector Prune(const ProgramDesc& pdesc, int block_id); - } // namespace framework } // namespace paddle From fafc54d6c08b06a8b86217118eace48a9a5a122f Mon Sep 17 00:00:00 2001 From: Qiao Longfei Date: Mon, 16 Oct 2017 19:00:22 -0700 Subject: [PATCH 41/63] fix var_type_inference_test (#4854) --- paddle/framework/var_type_inference_test.cc | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/paddle/framework/var_type_inference_test.cc b/paddle/framework/var_type_inference_test.cc index 97b8c64748..87399208e9 100644 --- a/paddle/framework/var_type_inference_test.cc +++ b/paddle/framework/var_type_inference_test.cc @@ -68,10 +68,10 @@ TEST(InferVarType, sum_op) { op->SetInput("X", {"test_a", "test_b", "test_c"}); op->SetOutput("Out", {"test_out"}); - prog.Block(0)->NewVar("test_a")->SetType(VarDesc::SELECTED_ROWS); - prog.Block(0)->NewVar("test_b")->SetType(VarDesc::SELECTED_ROWS); - prog.Block(0)->NewVar("test_c")->SetType(VarDesc::SELECTED_ROWS); - prog.Block(0)->NewVar("test_out"); + prog.Block(0)->Var("test_a")->SetType(VarDesc::SELECTED_ROWS); + prog.Block(0)->Var("test_b")->SetType(VarDesc::SELECTED_ROWS); + prog.Block(0)->Var("test_c")->SetType(VarDesc::SELECTED_ROWS); + prog.Block(0)->Var("test_out"); op->InferVarType(prog.Block(0)); @@ -89,10 +89,10 @@ TEST(InferVarType, sum_op_without_infer_var_type) { op->SetInput("X", {"test2_a", "test2_b", "test2_c"}); op->SetOutput("Out", {"test2_out"}); - prog.Block(0)->NewVar("test2_a")->SetType(VarDesc::SELECTED_ROWS); - prog.Block(0)->NewVar("test2_b")->SetType(VarDesc::SELECTED_ROWS); - prog.Block(0)->NewVar("test2_c")->SetType(VarDesc::SELECTED_ROWS); - prog.Block(0)->NewVar("test2_out"); + prog.Block(0)->Var("test2_a")->SetType(VarDesc::SELECTED_ROWS); + prog.Block(0)->Var("test2_b")->SetType(VarDesc::SELECTED_ROWS); + prog.Block(0)->Var("test2_c")->SetType(VarDesc::SELECTED_ROWS); + prog.Block(0)->Var("test2_out"); op->InferVarType(prog.Block(0)); From 73a8b78a720e2998634fc597efb78cc8fe53b2d7 Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Mon, 16 Oct 2017 19:11:42 -0700 Subject: [PATCH 42/63] Correct OpWithKernel's infershape (#4847) They are public now --- paddle/operators/accuracy_op.cc | 1 - paddle/operators/activation_op.cc | 2 -- paddle/operators/adadelta_op.cc | 1 - paddle/operators/adagrad_op.cc | 1 - paddle/operators/adam_op.cc | 1 - paddle/operators/adamax_op.cc | 1 - paddle/operators/clip_op.cc | 2 -- paddle/operators/concat_op.cc | 2 -- paddle/operators/conv2d_op.h | 2 -- paddle/operators/conv_shift_op.cc | 2 -- paddle/operators/cos_sim_op.cc | 2 -- paddle/operators/crop_op.cc | 2 -- paddle/operators/cross_entropy_op.cc | 4 ++-- paddle/operators/decayed_adagrad_op.cc | 1 - paddle/operators/dropout_op.cc | 2 -- paddle/operators/elementwise_op.h | 2 -- paddle/operators/fill_constant_op.cc | 2 +- paddle/operators/fill_zeros_like_op.cc | 1 - paddle/operators/gather_op.cc | 4 ++-- paddle/operators/gaussian_random_op.cc | 2 +- paddle/operators/gru_unit_op.cc | 2 -- paddle/operators/lookup_table_op.cc | 4 ++-- paddle/operators/lstm_unit_op.cc | 2 -- paddle/operators/margin_rank_loss_op.cc | 2 -- paddle/operators/mean_op.cc | 2 -- paddle/operators/minus_op.cc | 1 - paddle/operators/modified_huber_loss_op.cc | 2 -- paddle/operators/mul_op.cc | 2 -- paddle/operators/multiplex_op.cc | 4 ++-- paddle/operators/pad_op.cc | 2 -- paddle/operators/pool_op.h | 2 -- paddle/operators/pool_with_index_op.cc | 2 -- paddle/operators/prelu_op.cc | 2 -- paddle/operators/rank_loss_op.cc | 2 -- paddle/operators/reduce_op.cc | 2 -- paddle/operators/reshape_op.cc | 2 -- paddle/operators/rmsprop_op.cc | 1 - paddle/operators/scale_op.cc | 2 -- paddle/operators/scatter_op.cc | 4 ++-- paddle/operators/sequence_concat_op.cc | 2 -- paddle/operators/sequence_pool_op.cc | 2 -- paddle/operators/sequence_softmax_op.cc | 2 -- paddle/operators/sgd_op.cc | 1 - paddle/operators/sigmoid_cross_entropy_with_logits_op.cc | 2 -- paddle/operators/smooth_l1_loss_op.cc | 2 -- paddle/operators/softmax_op.cc | 2 -- paddle/operators/softmax_with_cross_entropy_op.cc | 4 ++-- paddle/operators/split_op.cc | 1 - paddle/operators/squared_l2_distance_op.cc | 2 -- paddle/operators/sum_op.cc | 1 - paddle/operators/top_k_op.cc | 1 - paddle/operators/transpose_op.cc | 2 -- paddle/operators/uniform_random_op.cc | 2 +- 53 files changed, 15 insertions(+), 90 deletions(-) diff --git a/paddle/operators/accuracy_op.cc b/paddle/operators/accuracy_op.cc index c5fb113e0f..037bb49abc 100644 --- a/paddle/operators/accuracy_op.cc +++ b/paddle/operators/accuracy_op.cc @@ -21,7 +21,6 @@ class AccuracyOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("Inference"), "Input(Inference) of AccuracyOp should not be null."); diff --git a/paddle/operators/activation_op.cc b/paddle/operators/activation_op.cc index 84c3775b4f..ee4f9b0ef2 100644 --- a/paddle/operators/activation_op.cc +++ b/paddle/operators/activation_op.cc @@ -21,7 +21,6 @@ class ActivationOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { ctx->SetOutputDim("Y", ctx->GetInputDim("X")); ctx->ShareLoD("X", /*->*/ "Y"); @@ -32,7 +31,6 @@ class ActivationOpGrad : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { ctx->SetOutputDim(framework::GradVarName("X"), ctx->GetInputDim("Y")); } diff --git a/paddle/operators/adadelta_op.cc b/paddle/operators/adadelta_op.cc index cf1bca1658..24e419b532 100644 --- a/paddle/operators/adadelta_op.cc +++ b/paddle/operators/adadelta_op.cc @@ -21,7 +21,6 @@ class AdadeltaOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("Param"), "Input(Param) of AdadeltaOp should not be null."); diff --git a/paddle/operators/adagrad_op.cc b/paddle/operators/adagrad_op.cc index a17747efb7..bc081f87dc 100644 --- a/paddle/operators/adagrad_op.cc +++ b/paddle/operators/adagrad_op.cc @@ -21,7 +21,6 @@ class AdagradOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("Param"), "Input(Param) of AdagradOp should not be null."); diff --git a/paddle/operators/adam_op.cc b/paddle/operators/adam_op.cc index 293b37b775..e3db70ea12 100644 --- a/paddle/operators/adam_op.cc +++ b/paddle/operators/adam_op.cc @@ -21,7 +21,6 @@ class AdamOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("Param"), "Input(Param) of AdamOp should not be null."); diff --git a/paddle/operators/adamax_op.cc b/paddle/operators/adamax_op.cc index 5cf727742c..e848333ef8 100644 --- a/paddle/operators/adamax_op.cc +++ b/paddle/operators/adamax_op.cc @@ -21,7 +21,6 @@ class AdamaxOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("Param"), "Input(Param) of AdamaxOp should not be null."); diff --git a/paddle/operators/clip_op.cc b/paddle/operators/clip_op.cc index 3e9b0d82ba..2d029394dd 100644 --- a/paddle/operators/clip_op.cc +++ b/paddle/operators/clip_op.cc @@ -21,7 +21,6 @@ class ClipOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) of ClipOp should not be null."); @@ -60,7 +59,6 @@ class ClipOpGrad : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) should not be null"); PADDLE_ENFORCE(ctx->HasInput(framework::GradVarName("Out")), diff --git a/paddle/operators/concat_op.cc b/paddle/operators/concat_op.cc index 235c4449ac..e11e51b458 100644 --- a/paddle/operators/concat_op.cc +++ b/paddle/operators/concat_op.cc @@ -23,7 +23,6 @@ class ConcatOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE_GE(ctx->Inputs("X").size(), 1UL, "Inputs(X) of ConcatOp should be empty.") @@ -82,7 +81,6 @@ class ConcatOpGrad : public framework::OperatorWithKernel { const framework::AttributeMap &attrs) : OperatorWithKernel(type, inputs, outputs, attrs) {} - protected: void InferShape(framework::InferShapeContext *ctx) const override { ctx->SetOutputsDim(framework::GradVarName("X"), ctx->GetInputsDim("X")); } diff --git a/paddle/operators/conv2d_op.h b/paddle/operators/conv2d_op.h index 7ebdbe81cb..bd1734879e 100644 --- a/paddle/operators/conv2d_op.h +++ b/paddle/operators/conv2d_op.h @@ -44,7 +44,6 @@ class Conv2DOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override; }; @@ -52,7 +51,6 @@ class Conv2DOpGrad : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override; }; diff --git a/paddle/operators/conv_shift_op.cc b/paddle/operators/conv_shift_op.cc index e1e321ed5f..6156a2d6af 100644 --- a/paddle/operators/conv_shift_op.cc +++ b/paddle/operators/conv_shift_op.cc @@ -27,7 +27,6 @@ class ConvShiftOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) should be not null."); PADDLE_ENFORCE(ctx->HasInput("Y"), "Input(Y) should be not null."); @@ -54,7 +53,6 @@ class ConvShiftGradOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) should be not null."); PADDLE_ENFORCE(ctx->HasInput("Y"), "Input(Y) should be not null."); diff --git a/paddle/operators/cos_sim_op.cc b/paddle/operators/cos_sim_op.cc index 2b4c4b9c45..55f69fb03a 100644 --- a/paddle/operators/cos_sim_op.cc +++ b/paddle/operators/cos_sim_op.cc @@ -23,7 +23,6 @@ class CosSimOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { // notnull check PADDLE_ENFORCE(ctx->HasInput("X"), @@ -97,7 +96,6 @@ class CosSimOpGrad : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { // notnull check PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) must not be null."); diff --git a/paddle/operators/crop_op.cc b/paddle/operators/crop_op.cc index a1424993cc..a994d91676 100644 --- a/paddle/operators/crop_op.cc +++ b/paddle/operators/crop_op.cc @@ -24,7 +24,6 @@ class CropOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) of CropOp should not be null."); @@ -114,7 +113,6 @@ class CropOpGrad : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) should not be null"); PADDLE_ENFORCE(ctx->HasInput(framework::GradVarName("Out")), diff --git a/paddle/operators/cross_entropy_op.cc b/paddle/operators/cross_entropy_op.cc index 923ae5be6c..a865991db3 100644 --- a/paddle/operators/cross_entropy_op.cc +++ b/paddle/operators/cross_entropy_op.cc @@ -21,7 +21,6 @@ class CrossEntropyOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) should be not null."); PADDLE_ENFORCE(ctx->HasInput("Label"), "Input(Label) should be not null."); @@ -48,6 +47,7 @@ class CrossEntropyOp : public framework::OperatorWithKernel { ctx->ShareLoD("X", /*->*/ "Y"); } + protected: // CrossEntropy's data type just determined by "X" framework::DataType IndicateDataType( const framework::ExecutionContext& ctx) const override { @@ -59,7 +59,6 @@ class CrossEntropyGradientOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) should be not null."); PADDLE_ENFORCE(ctx->HasInput("Label"), "Input(Label) should be not null."); @@ -94,6 +93,7 @@ class CrossEntropyGradientOp : public framework::OperatorWithKernel { ctx->SetOutputDim(framework::GradVarName("X"), x_dims); } + protected: // CrossEntropy's data type just determined by "X" framework::DataType IndicateDataType( const framework::ExecutionContext& ctx) const override { diff --git a/paddle/operators/decayed_adagrad_op.cc b/paddle/operators/decayed_adagrad_op.cc index 7f583f18c8..17b394aa07 100644 --- a/paddle/operators/decayed_adagrad_op.cc +++ b/paddle/operators/decayed_adagrad_op.cc @@ -21,7 +21,6 @@ class DecayedAdagradOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("Param"), "Input(Param) of DecayedAdagradOp should not be null."); diff --git a/paddle/operators/dropout_op.cc b/paddle/operators/dropout_op.cc index 708ccfa0bf..29858c9083 100644 --- a/paddle/operators/dropout_op.cc +++ b/paddle/operators/dropout_op.cc @@ -23,7 +23,6 @@ class DropoutOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) must not be null."); PADDLE_ENFORCE_GE(ctx->Attrs().Get("dropout_prob"), 0); @@ -69,7 +68,6 @@ class DropoutOpGrad : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE_EQ(ctx->Attrs().Get("is_training"), 1, "GradOp is only callable when is_training is true"); diff --git a/paddle/operators/elementwise_op.h b/paddle/operators/elementwise_op.h index 66f1910a47..fce4b24a22 100644 --- a/paddle/operators/elementwise_op.h +++ b/paddle/operators/elementwise_op.h @@ -23,7 +23,6 @@ class ElementwiseOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: using Tensor = framework::Tensor; void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), @@ -105,7 +104,6 @@ class ElementwiseOpGrad : public framework::OperatorWithKernel { using framework::OperatorWithKernel::OperatorWithKernel; using Tensor = framework::Tensor; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) should not be null"); PADDLE_ENFORCE(ctx->HasInput("Y"), "Input(Y) should not be null"); diff --git a/paddle/operators/fill_constant_op.cc b/paddle/operators/fill_constant_op.cc index a56832e202..0438d4d085 100644 --- a/paddle/operators/fill_constant_op.cc +++ b/paddle/operators/fill_constant_op.cc @@ -21,7 +21,6 @@ class FillConstantOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasOutput("Out"), "Output(Out) of FillConstantOp should not be null."); @@ -33,6 +32,7 @@ class FillConstantOp : public framework::OperatorWithKernel { ctx->SetOutputDim("Out", dims); } + protected: framework::DataType IndicateDataType( const framework::ExecutionContext &ctx) const override { return static_cast(ctx.Attr("data_type")); diff --git a/paddle/operators/fill_zeros_like_op.cc b/paddle/operators/fill_zeros_like_op.cc index 4c70b9a36b..ed529ac40a 100644 --- a/paddle/operators/fill_zeros_like_op.cc +++ b/paddle/operators/fill_zeros_like_op.cc @@ -21,7 +21,6 @@ class FillZerosLikeOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) of FillZerosLikeOp should not be null."); diff --git a/paddle/operators/gather_op.cc b/paddle/operators/gather_op.cc index fb99c6c016..f6c7f472da 100644 --- a/paddle/operators/gather_op.cc +++ b/paddle/operators/gather_op.cc @@ -22,7 +22,6 @@ class GatherOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) of GatherOp should not be null."); @@ -40,6 +39,7 @@ class GatherOp : public framework::OperatorWithKernel { ctx->SetOutputDim("Out", output_dims); } + protected: framework::DataType IndicateDataType( const framework::ExecutionContext& ctx) const override { return framework::ToDataType(ctx.Input("X")->type()); @@ -50,11 +50,11 @@ class GatherGradOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { ctx->SetOutputDim(framework::GradVarName("X"), ctx->GetInputDim("X")); } + protected: framework::DataType IndicateDataType( const framework::ExecutionContext& ctx) const override { return framework::ToDataType(ctx.Input("X")->type()); diff --git a/paddle/operators/gaussian_random_op.cc b/paddle/operators/gaussian_random_op.cc index ca7fb38505..f59f497d9f 100644 --- a/paddle/operators/gaussian_random_op.cc +++ b/paddle/operators/gaussian_random_op.cc @@ -42,7 +42,6 @@ class GaussianRandomOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasOutput("Out"), "Output(Out) of GaussianRandomOp should not be null."); @@ -57,6 +56,7 @@ class GaussianRandomOp : public framework::OperatorWithKernel { ctx->SetOutputDim("Out", framework::make_ddim(temp)); } + protected: framework::DataType IndicateDataType( const framework::ExecutionContext& ctx) const override { return static_cast(Attr("data_type")); diff --git a/paddle/operators/gru_unit_op.cc b/paddle/operators/gru_unit_op.cc index 24f84597cd..72dd841c85 100644 --- a/paddle/operators/gru_unit_op.cc +++ b/paddle/operators/gru_unit_op.cc @@ -23,7 +23,6 @@ class GRUUnitOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("Input"), "Input(%s) of GRUUnitOp should not be null.", "Input"); @@ -131,7 +130,6 @@ class GRUUnitGradOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("Input"), "Input(%s) of GRUUnitGradOp should not be null.", "Input"); diff --git a/paddle/operators/lookup_table_op.cc b/paddle/operators/lookup_table_op.cc index 3f8d4ab857..b88cd14d78 100644 --- a/paddle/operators/lookup_table_op.cc +++ b/paddle/operators/lookup_table_op.cc @@ -21,7 +21,6 @@ class LookupTableOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("W"), "Input(W) of LookupTableOp should not be null."); @@ -37,6 +36,7 @@ class LookupTableOp : public framework::OperatorWithKernel { ctx->ShareLoD("Ids", /*->*/ "Out"); } + protected: framework::DataType IndicateDataType( const framework::ExecutionContext& ctx) const override { return framework::ToDataType(ctx.Input("W")->type()); @@ -69,12 +69,12 @@ class LookupTableOpGrad : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { auto table_dims = ctx->GetInputDim("W"); ctx->SetOutputDim(framework::GradVarName("W"), table_dims); } + protected: framework::DataType IndicateDataType( const framework::ExecutionContext& ctx) const override { return framework::ToDataType(ctx.Input("W")->type()); diff --git a/paddle/operators/lstm_unit_op.cc b/paddle/operators/lstm_unit_op.cc index 13a45ec246..5d63017208 100644 --- a/paddle/operators/lstm_unit_op.cc +++ b/paddle/operators/lstm_unit_op.cc @@ -21,7 +21,6 @@ class LstmUnitOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) of LSTM should not be null."); PADDLE_ENFORCE(ctx->HasInput("C_prev"), @@ -76,7 +75,6 @@ class LstmUnitGradOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput(framework::GradVarName("C")), "Input(C@GRAD) should not be null"); diff --git a/paddle/operators/margin_rank_loss_op.cc b/paddle/operators/margin_rank_loss_op.cc index 5be61dfec3..638a99addc 100644 --- a/paddle/operators/margin_rank_loss_op.cc +++ b/paddle/operators/margin_rank_loss_op.cc @@ -21,7 +21,6 @@ class MarginRankLossOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { // input check PADDLE_ENFORCE(ctx->HasInput("Label"), "Input(Label) shouldn't be null."); @@ -94,7 +93,6 @@ class MarginRankLossGradOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("Label"), "Input(Label) shouldn't be null."); PADDLE_ENFORCE(ctx->HasInput("X1"), "Input(X1) shouldn't be null."); diff --git a/paddle/operators/mean_op.cc b/paddle/operators/mean_op.cc index 441543049f..9556fdf731 100644 --- a/paddle/operators/mean_op.cc +++ b/paddle/operators/mean_op.cc @@ -21,7 +21,6 @@ class MeanOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) of MeanOp should not be null."); @@ -46,7 +45,6 @@ class MeanGradOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { ctx->SetOutputDim(framework::GradVarName("X"), ctx->GetInputDim("X")); } diff --git a/paddle/operators/minus_op.cc b/paddle/operators/minus_op.cc index d7fd2f901b..f7943e99ac 100644 --- a/paddle/operators/minus_op.cc +++ b/paddle/operators/minus_op.cc @@ -25,7 +25,6 @@ class MinusOp : public framework::OperatorWithKernel { const framework::AttributeMap &attrs) : OperatorWithKernel(type, inputs, outputs, attrs) {} - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) of MinusOp should not be null."); diff --git a/paddle/operators/modified_huber_loss_op.cc b/paddle/operators/modified_huber_loss_op.cc index 6522327fdc..7b9e952895 100644 --- a/paddle/operators/modified_huber_loss_op.cc +++ b/paddle/operators/modified_huber_loss_op.cc @@ -21,7 +21,6 @@ class ModifiedHuberLossOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "X must be initialized."); PADDLE_ENFORCE(ctx->HasInput("Y"), "Y must be initialized."); @@ -73,7 +72,6 @@ class ModifiedHuberLossGradOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "X must be initialized."); PADDLE_ENFORCE(ctx->HasInput("Y"), "Y must be initialized."); diff --git a/paddle/operators/mul_op.cc b/paddle/operators/mul_op.cc index ec0683d887..943f81e949 100644 --- a/paddle/operators/mul_op.cc +++ b/paddle/operators/mul_op.cc @@ -23,7 +23,6 @@ class MulOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) of MulOp should not be null."); PADDLE_ENFORCE(ctx->HasInput("Y"), "Input(Y) of MulOp should not be null."); @@ -96,7 +95,6 @@ class MulOpGrad : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) should not be null"); PADDLE_ENFORCE(ctx->HasInput("Y"), "Input(Y) should not be null"); diff --git a/paddle/operators/multiplex_op.cc b/paddle/operators/multiplex_op.cc index 051051b051..4d86769026 100644 --- a/paddle/operators/multiplex_op.cc +++ b/paddle/operators/multiplex_op.cc @@ -23,7 +23,6 @@ class MultiplexOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("Ids"), "Input(Ids) shouldn't be null."); PADDLE_ENFORCE(!ctx->Inputs("X").empty(), @@ -51,6 +50,7 @@ class MultiplexOp : public framework::OperatorWithKernel { ctx->SetOutputDim("Out", in_dim); } + protected: framework::DataType IndicateDataType( const framework::ExecutionContext& ctx) const override { return framework::ToDataType(ctx.MultiInput("X")[0]->type()); @@ -89,7 +89,6 @@ class MultiplexGradOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(!ctx->Inputs("X").empty(), "Input(X) should not be null."); PADDLE_ENFORCE(!ctx->Outputs(framework::GradVarName("X")).empty(), @@ -105,6 +104,7 @@ class MultiplexGradOp : public framework::OperatorWithKernel { ctx->SetOutputsDim(framework::GradVarName("X"), d_ins); } + protected: framework::DataType IndicateDataType( const framework::ExecutionContext& ctx) const override { return framework::ToDataType(ctx.MultiInput("X")[0]->type()); diff --git a/paddle/operators/pad_op.cc b/paddle/operators/pad_op.cc index 2f26ada85e..73a0b8baff 100644 --- a/paddle/operators/pad_op.cc +++ b/paddle/operators/pad_op.cc @@ -23,7 +23,6 @@ class PadOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) of PadOp should not be null."); PADDLE_ENFORCE(ctx->HasOutput("Out"), @@ -97,7 +96,6 @@ class PadOpGrad : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) should not be null"); PADDLE_ENFORCE(ctx->HasInput(framework::GradVarName("Out")), diff --git a/paddle/operators/pool_op.h b/paddle/operators/pool_op.h index f6169e05b3..ada9565019 100644 --- a/paddle/operators/pool_op.h +++ b/paddle/operators/pool_op.h @@ -28,7 +28,6 @@ class PoolOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override; }; @@ -36,7 +35,6 @@ class PoolOpGrad : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override; }; diff --git a/paddle/operators/pool_with_index_op.cc b/paddle/operators/pool_with_index_op.cc index 8eee20c2a9..29d0322a27 100644 --- a/paddle/operators/pool_with_index_op.cc +++ b/paddle/operators/pool_with_index_op.cc @@ -27,7 +27,6 @@ class MaxPoolWithIndexOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "X(Input) of Pooling should not be null."); @@ -72,7 +71,6 @@ class MaxPoolWithIndexOpGrad : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("Mask"), "Input(Mask) must not be null."); PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) must not be null."); diff --git a/paddle/operators/prelu_op.cc b/paddle/operators/prelu_op.cc index 166fe26824..eef2e34eaa 100644 --- a/paddle/operators/prelu_op.cc +++ b/paddle/operators/prelu_op.cc @@ -25,7 +25,6 @@ class PReluOp : public framework::OperatorWithKernel { const framework::AttributeMap &attrs) : OperatorWithKernel(type, inputs, outputs, attrs) {} - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) should not be null"); PADDLE_ENFORCE(ctx->HasInput("Alpha"), "Input(Alpha) should not be null"); @@ -62,7 +61,6 @@ class PReluGradOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) must not be null."); PADDLE_ENFORCE(ctx->HasInput(framework::GradVarName("Out")), diff --git a/paddle/operators/rank_loss_op.cc b/paddle/operators/rank_loss_op.cc index e0abbc4db1..17ef2b1d01 100644 --- a/paddle/operators/rank_loss_op.cc +++ b/paddle/operators/rank_loss_op.cc @@ -24,7 +24,6 @@ class RankLossOp : public framework::OperatorWithKernel { const framework::AttributeMap &attrs) : OperatorWithKernel(type, inputs, outputs, attrs) {} - protected: void InferShape(framework::InferShapeContext *ctx) const override { // input check PADDLE_ENFORCE(ctx->HasInput("Label"), "Input(Label) shouldn't be null"); @@ -89,7 +88,6 @@ class RankLossGradOp : public framework::OperatorWithKernel { const framework::AttributeMap &attrs) : OperatorWithKernel(type, inputs, outputs, attrs) {} - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("Label"), "Input(Label) shouldn't be null."); PADDLE_ENFORCE(ctx->HasInput("Left"), "Input(Left) shouldn't be null."); diff --git a/paddle/operators/reduce_op.cc b/paddle/operators/reduce_op.cc index 005f88b57c..5e878353ce 100644 --- a/paddle/operators/reduce_op.cc +++ b/paddle/operators/reduce_op.cc @@ -23,7 +23,6 @@ class ReduceOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) of ReduceOp should not be null."); @@ -57,7 +56,6 @@ class ReduceGradOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) should not be null."); PADDLE_ENFORCE(ctx->HasInput(framework::GradVarName("Out")), diff --git a/paddle/operators/reshape_op.cc b/paddle/operators/reshape_op.cc index 3cd54930a0..a8eb8d45ee 100644 --- a/paddle/operators/reshape_op.cc +++ b/paddle/operators/reshape_op.cc @@ -25,7 +25,6 @@ class ReshapeOp : public framework::OperatorWithKernel { const framework::AttributeMap &attrs) : OperatorWithKernel(type, inputs, outputs, attrs) {} - protected: void InferShape(framework::InferShapeContext *ctx) const override { // input check PADDLE_ENFORCE(ctx->HasInput("X"), @@ -93,7 +92,6 @@ class ReshapeGradOp : public framework::OperatorWithKernel { const framework::AttributeMap &attrs) : OperatorWithKernel(type, inputs, outputs, attrs) {} - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) shouldn't be null."); PADDLE_ENFORCE(ctx->HasInput(framework::GradVarName("Out")), diff --git a/paddle/operators/rmsprop_op.cc b/paddle/operators/rmsprop_op.cc index ada6f2bc3c..fd5567a365 100644 --- a/paddle/operators/rmsprop_op.cc +++ b/paddle/operators/rmsprop_op.cc @@ -21,7 +21,6 @@ class RmspropOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("Param"), "Input(Param) of RmspropOp should not be null."); diff --git a/paddle/operators/scale_op.cc b/paddle/operators/scale_op.cc index ac297da6b7..7f1a21bea7 100644 --- a/paddle/operators/scale_op.cc +++ b/paddle/operators/scale_op.cc @@ -25,7 +25,6 @@ class ScaleOp : public framework::OperatorWithKernel { const framework::AttributeMap &attrs) : OperatorWithKernel(type, inputs, outputs, attrs) {} - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) of ScaleOp should not be null."); @@ -56,7 +55,6 @@ class ScaleGradMaker : public framework::SingleGradOpDescMaker { public: using framework::SingleGradOpDescMaker::SingleGradOpDescMaker; - protected: std::unique_ptr Apply() const override { auto *grad_op = new framework::OpDescBind(); grad_op->SetType("scale"); diff --git a/paddle/operators/scatter_op.cc b/paddle/operators/scatter_op.cc index fbea01a8db..62e6c70b45 100644 --- a/paddle/operators/scatter_op.cc +++ b/paddle/operators/scatter_op.cc @@ -22,7 +22,6 @@ class ScatterOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("Ref"), "Input(Ref) of ScatterOp should not be null."); @@ -49,6 +48,7 @@ class ScatterOp : public framework::OperatorWithKernel { ctx->SetOutputDim("Out", ref_dims); } + protected: framework::DataType IndicateDataType( const framework::ExecutionContext& ctx) const override { return framework::ToDataType(ctx.Input("Ref")->type()); @@ -59,13 +59,13 @@ class ScatterGradOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { ctx->SetOutputDim(framework::GradVarName("Updates"), ctx->GetInputDim("Updates")); ctx->SetOutputDim(framework::GradVarName("Ref"), ctx->GetInputDim("Ref")); } + protected: framework::DataType IndicateDataType( const framework::ExecutionContext& ctx) const override { return framework::ToDataType(ctx.Input("Ref")->type()); diff --git a/paddle/operators/sequence_concat_op.cc b/paddle/operators/sequence_concat_op.cc index 287fb1942e..1fce96cdfe 100644 --- a/paddle/operators/sequence_concat_op.cc +++ b/paddle/operators/sequence_concat_op.cc @@ -21,7 +21,6 @@ class SequenceConcatOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInputs("X"), "Inputs(X) of SequenceConcatOp should not be null."); @@ -105,7 +104,6 @@ class SequenceConcatGradOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput(framework::GradVarName("Out")), "The gradient of Out should not be null."); diff --git a/paddle/operators/sequence_pool_op.cc b/paddle/operators/sequence_pool_op.cc index 8dc4a59ba8..e3f5d509a8 100644 --- a/paddle/operators/sequence_pool_op.cc +++ b/paddle/operators/sequence_pool_op.cc @@ -21,7 +21,6 @@ class SequencePoolOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) of SequencePoolOp should not be null."); @@ -72,7 +71,6 @@ class SequencePoolGradOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput(framework::GradVarName("Out")), "Gradient of Out should not be null."); diff --git a/paddle/operators/sequence_softmax_op.cc b/paddle/operators/sequence_softmax_op.cc index ea217ba459..c891ab1fdc 100644 --- a/paddle/operators/sequence_softmax_op.cc +++ b/paddle/operators/sequence_softmax_op.cc @@ -21,7 +21,6 @@ class SequenceSoftmaxOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) of SequenceSoftmaxOp should not be null."); @@ -66,7 +65,6 @@ class SequenceSoftmaxGradOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("Out"), "Input(Out) of SequenceSoftmaxGradOp should not be null."); diff --git a/paddle/operators/sgd_op.cc b/paddle/operators/sgd_op.cc index 2a6a162a02..0f78eeab9b 100644 --- a/paddle/operators/sgd_op.cc +++ b/paddle/operators/sgd_op.cc @@ -21,7 +21,6 @@ class SGDOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("Param"), "Input(Param) of SGDOp should not be null."); diff --git a/paddle/operators/sigmoid_cross_entropy_with_logits_op.cc b/paddle/operators/sigmoid_cross_entropy_with_logits_op.cc index b6653e1cc7..e781c8db20 100644 --- a/paddle/operators/sigmoid_cross_entropy_with_logits_op.cc +++ b/paddle/operators/sigmoid_cross_entropy_with_logits_op.cc @@ -23,7 +23,6 @@ class SigmoidCrossEntropyWithLogitsOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) should be not null."); PADDLE_ENFORCE(ctx->HasInput("Labels"), @@ -52,7 +51,6 @@ class SigmoidCrossEntropyWithLogitsGradOp public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) should be not null."); PADDLE_ENFORCE(ctx->HasInput("Labels"), diff --git a/paddle/operators/smooth_l1_loss_op.cc b/paddle/operators/smooth_l1_loss_op.cc index 91391dc945..a4f0f37764 100644 --- a/paddle/operators/smooth_l1_loss_op.cc +++ b/paddle/operators/smooth_l1_loss_op.cc @@ -21,7 +21,6 @@ class SmoothL1LossOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "X must be initialized."); PADDLE_ENFORCE(ctx->HasInput("Y"), "Y must be initialized."); @@ -93,7 +92,6 @@ class SmoothL1LossGradOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { auto in_dims = ctx->GetInputDim("X"); auto out_dims = ctx->GetInputDim(framework::GradVarName("Out")); diff --git a/paddle/operators/softmax_op.cc b/paddle/operators/softmax_op.cc index 4c131ed44d..00fd0b32a9 100644 --- a/paddle/operators/softmax_op.cc +++ b/paddle/operators/softmax_op.cc @@ -21,7 +21,6 @@ class SoftmaxOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) of SoftmaxOp should not be null."); @@ -68,7 +67,6 @@ class SoftmaxOpGrad : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("Y"), "Input(Y) should be not null."); PADDLE_ENFORCE(ctx->HasInput(framework::GradVarName("Y")), diff --git a/paddle/operators/softmax_with_cross_entropy_op.cc b/paddle/operators/softmax_with_cross_entropy_op.cc index 9121609f10..942fbb42df 100644 --- a/paddle/operators/softmax_with_cross_entropy_op.cc +++ b/paddle/operators/softmax_with_cross_entropy_op.cc @@ -82,7 +82,6 @@ class SoftmaxWithCrossEntropyOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("Logits"), "Input(Logits) should be not null."); @@ -117,6 +116,7 @@ class SoftmaxWithCrossEntropyOp : public framework::OperatorWithKernel { ctx->ShareLoD("Logits", /*->*/ "Loss"); } + protected: framework::DataType IndicateDataType( const framework::ExecutionContext& ctx) const override { return framework::ToDataType(ctx.Input("Logits")->type()); @@ -127,7 +127,6 @@ class SoftmaxWithCrossEntropyOpGrad : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput(framework::GradVarName("Loss")), "Input(Loss@Grad) should not be null."); @@ -156,6 +155,7 @@ class SoftmaxWithCrossEntropyOpGrad : public framework::OperatorWithKernel { ctx->GetInputDim("Softmax")); } + protected: framework::DataType IndicateDataType( const framework::ExecutionContext& ctx) const override { return framework::ToDataType( diff --git a/paddle/operators/split_op.cc b/paddle/operators/split_op.cc index d5dd4df2a2..4a6c50f797 100644 --- a/paddle/operators/split_op.cc +++ b/paddle/operators/split_op.cc @@ -23,7 +23,6 @@ class SplitOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) of SplitOp should not be null."); diff --git a/paddle/operators/squared_l2_distance_op.cc b/paddle/operators/squared_l2_distance_op.cc index cce4e527c3..e360c19b47 100644 --- a/paddle/operators/squared_l2_distance_op.cc +++ b/paddle/operators/squared_l2_distance_op.cc @@ -21,7 +21,6 @@ class SquaredL2DistanceOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) of SquaredL2DistanceOp should not be null."); @@ -85,7 +84,6 @@ class SquaredL2DistanceGradOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput(framework::GradVarName("Out")), "Gradient of Out should not be null"); diff --git a/paddle/operators/sum_op.cc b/paddle/operators/sum_op.cc index 573487b835..5214a8413e 100644 --- a/paddle/operators/sum_op.cc +++ b/paddle/operators/sum_op.cc @@ -21,7 +21,6 @@ class SumOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInputs("X"), "Inputs(X) should not be null"); auto x_dims = ctx->GetInputsDim("X"); diff --git a/paddle/operators/top_k_op.cc b/paddle/operators/top_k_op.cc index c954819912..d5c2c91a5f 100644 --- a/paddle/operators/top_k_op.cc +++ b/paddle/operators/top_k_op.cc @@ -21,7 +21,6 @@ class TopkOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) of TopkOp should not be null."); diff --git a/paddle/operators/transpose_op.cc b/paddle/operators/transpose_op.cc index 1101bbe3ef..d785e57c83 100644 --- a/paddle/operators/transpose_op.cc +++ b/paddle/operators/transpose_op.cc @@ -23,7 +23,6 @@ class TransposeOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) should not be null"); PADDLE_ENFORCE(ctx->HasOutput("Out"), "Output(Out) should not be null"); @@ -92,7 +91,6 @@ class TransposeOpGrad : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("X"), "Input(X) should not be null"); PADDLE_ENFORCE(ctx->HasInput(framework::GradVarName("Out")), diff --git a/paddle/operators/uniform_random_op.cc b/paddle/operators/uniform_random_op.cc index 75928f1ec8..612bdd70db 100644 --- a/paddle/operators/uniform_random_op.cc +++ b/paddle/operators/uniform_random_op.cc @@ -46,7 +46,6 @@ class UniformRandomOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - protected: void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasOutput("Out"), "Output(Out) of UniformRandomOp should not be null."); @@ -63,6 +62,7 @@ class UniformRandomOp : public framework::OperatorWithKernel { ctx->SetOutputDim("Out", framework::make_ddim(temp)); } + protected: framework::DataType IndicateDataType( const framework::ExecutionContext& ctx) const override { return static_cast(Attr("data_type")); From f43b1a90d8d0dccb54b4243f33df87660f7679d6 Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Mon, 16 Oct 2017 19:32:10 -0700 Subject: [PATCH 43/63] Design Doc: infer_var_type (#4795) --- doc/design/infer_var_type.md | 78 ++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 doc/design/infer_var_type.md diff --git a/doc/design/infer_var_type.md b/doc/design/infer_var_type.md new file mode 100644 index 0000000000..d9d5397bec --- /dev/null +++ b/doc/design/infer_var_type.md @@ -0,0 +1,78 @@ +# Design Doc: InferVarType + +## The Problem Posed + +The variable in our design can hold variant types. Such as `LoDTensor` and `SelectedRows`. An operator should be able to inference the variable types of its output. + +For example, a `lookup table` operator takes two `LoDTensor`; one is a float tensor as the embedding table, the other is an int tensor as word ID. The gradient operator of `lookup table` will generate a `SelectedRows` as its output. A `sum` operator can take both `LoDTensor` and `SelectedRows` as its inputs and will generate a `LoDTensor` if any of its inputs is `LoDTensor`, otherwise, the `sum` operator will generate `SelectedRows` as its output. + +The variable type will be constant at runtime. Every variable's type can either be set by the user (input data and parameter) or be inferred by the operator in compile time. + +## Proposed Solution + +The `InferVarType` is a compile-time function which is registered to each operator. The inferface of that function is: + + +```c++ +using InferVarTypeFN = std::function< + void (const OpDescBind& /*op_desc*/, BlockDescBind* /*block*/)>; +``` + +It takes an operator description as its input and will write the output variable type and store them in block description. + +The `InferVarTypeFN` will be registered in `OpInfo`, to replace `infer_var_type_` field. The `OpInfo` should be + +```cpp +struct OpInfo { + InferVarTypeFN infer_var_type_; + ... +}; +``` + +The default `InferVarType` will set output type as `LoDTensor`. It can be done by `GetInferVarType()`. + +```cpp +void DefaultInferVarType(const OpDescBind& op_desc, BlockDescBind* block) { + // set the output type of variable as `LoDTensor`. + // ... +} + +struct OpInfo { + InferVarTypeFN infer_var_type_; + InferVarTypeFN GetInferVarType() const { + if (infer_var_type_) { + return infer_var_type_; + } else { + return DefaultInferVarType; + } + } +}; +``` + +## Register InferVarType + +We provide a thin base class for registering an `InferVarTypeFN`. To use a base class will ease the implementation of registry since we can detect the registry entry is an `InferVarTypeFN` or not. + +```cpp +class VarTypeInferer { +public: + virtual void operator()(const OpDescBind& op_desc, BlockDescBind* block) const = 0; +} +``` + +Operator developers can write the specialize `VarTypeInferer` as follow. + +```cpp +class SpecialVarTypeInferer : public VarTypeInferer { +public: + virtual void operator()(const OpDescBind& op_desc, BlockDescBind* block) const { + // .. own logic + } +} +``` + +Then user can register the `InferVarType` just like `GradOpDescMaker` and `OpInfoMaker`. + +``` +REGISTER_OPERATOR(some_op, OpType, SpecialVarTypeInferer, ...); +``` From 064c3695ec9b6da32829d765256e8fed048c49ed Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Mon, 16 Oct 2017 20:58:31 -0700 Subject: [PATCH 44/63] Make global scope not thread-safe (#4850) * Make global scope not thread-safe 1. It is no need to make global scope thread-safe, since it will be invoked in Python main thread. 2. Do not free the global scope when C++ exit. Let the OS free memories, otherwise, we need to handle the destroy dependencies. See https://google.github.io/styleguide/cppguide.html#Static_and_Global_Variables * Revert "FIX: Release CPU/GPU memory via deleter" This reverts commit 8f80f5bc794d8900f9d57b51eea167f4dde2903c. --- paddle/framework/scope.cc | 14 ++++------- paddle/memory/memory.cc | 51 ++++++++++++--------------------------- 2 files changed, 20 insertions(+), 45 deletions(-) diff --git a/paddle/framework/scope.cc b/paddle/framework/scope.cc index 8f8a53eec8..5bf5e91f25 100644 --- a/paddle/framework/scope.cc +++ b/paddle/framework/scope.cc @@ -65,16 +65,12 @@ void Scope::DropKids() { kids_.clear(); } -std::once_flag feed_variable_flag; - framework::Scope& GetGlobalScope() { - static std::unique_ptr g_scope{nullptr}; - std::call_once(feed_variable_flag, [&]() { - g_scope.reset(new framework::Scope()); - g_scope->Var("feed_value"); - g_scope->Var("fetch_value"); - }); - return *(g_scope.get()); + static framework::Scope* g_scope = nullptr; + if (g_scope == nullptr) { + g_scope = new framework::Scope(); + } + return *g_scope; } } // namespace framework diff --git a/paddle/memory/memory.cc b/paddle/memory/memory.cc index 5087c02385..8e561528f0 100644 --- a/paddle/memory/memory.cc +++ b/paddle/memory/memory.cc @@ -14,11 +14,6 @@ limitations under the License. */ #include "paddle/memory/memory.h" -#include // for transform -#include // for memcpy -#include // for unique_ptr -#include // for call_once - #include "glog/logging.h" #include "paddle/memory/detail/buddy_allocator.h" @@ -32,19 +27,14 @@ namespace memory { using BuddyAllocator = detail::BuddyAllocator; -std::once_flag cpu_allocator_flag; -std::once_flag gpu_allocator_flag; - BuddyAllocator* GetCPUBuddyAllocator() { - static std::unique_ptr a{nullptr}; - - std::call_once(cpu_allocator_flag, [&]() { - a.reset(new BuddyAllocator(new detail::CPUAllocator, - platform::CpuMinChunkSize(), - platform::CpuMaxChunkSize())); - }); - - return a.get(); + static detail::BuddyAllocator* a = nullptr; + if (a == nullptr) { + a = new detail::BuddyAllocator(new detail::CPUAllocator, + platform::CpuMinChunkSize(), + platform::CpuMaxChunkSize()); + } + return a; } template <> @@ -65,35 +55,24 @@ size_t Used(platform::CPUPlace place) { #ifdef PADDLE_WITH_CUDA BuddyAllocator* GetGPUBuddyAllocator(int gpu_id) { - using BuddyAllocVec = std::vector; - static std::unique_ptr as{ - new BuddyAllocVec, [](BuddyAllocVec* p) { - std::for_each(p->begin(), p->end(), - [](BuddyAllocator* p) { delete p; }); - }}; - - // GPU buddy allocators - auto& allocators = *as.get(); - - // GPU buddy allocator initialization - std::call_once(gpu_allocator_flag, [&]() { + static BuddyAllocator** as = NULL; + if (as == NULL) { int gpu_num = platform::GetCUDADeviceCount(); - allocators.reserve(gpu_num); + as = new BuddyAllocator*[gpu_num]; for (int gpu = 0; gpu < gpu_num; gpu++) { platform::SetDeviceId(gpu); - allocators.emplace_back(new BuddyAllocator(new detail::GPUAllocator, - platform::GpuMinChunkSize(), - platform::GpuMaxChunkSize())); + as[gpu] = new BuddyAllocator(new detail::GPUAllocator, + platform::GpuMinChunkSize(), + platform::GpuMaxChunkSize()); } VLOG(3) << "\n\nNOTE: each GPU device use " << FLAGS_fraction_of_gpu_memory_to_use * 100 << "% of GPU memory.\n" << "You can set environment variable '" << platform::kEnvFractionGpuMemoryToUse << "' to change the fraction of GPU usage.\n\n"; - }); - + } platform::SetDeviceId(gpu_id); - return allocators[gpu_id]; + return as[gpu_id]; } template <> From 2dc6cea65ca55eb0c3e4af385cf61685782fb68c Mon Sep 17 00:00:00 2001 From: tensor-tang Date: Mon, 16 Oct 2017 22:54:10 +0800 Subject: [PATCH 45/63] add unit tests for mkldnn branches, and fix typo --- paddle/gserver/tests/CMakeLists.txt | 5 +- paddle/gserver/tests/MKLDNNTester.cpp | 141 ++++++++++++++++++ paddle/gserver/tests/MKLDNNTester.h | 38 ++++- .../mkldnn_branches_conv_conv_addto.conf | 45 ++++++ .../mkldnn_branches_conv_conv_addto_32c.conf | 45 ++++++ .../mkldnn_branches_conv_conv_concat.conf | 45 ++++++ .../mkldnn_branches_conv_conv_concat_32c.conf | 45 ++++++ paddle/gserver/tests/test_MKLDNN.cpp | 17 ++- 8 files changed, 369 insertions(+), 12 deletions(-) create mode 100644 paddle/gserver/tests/mkldnn_branches_conv_conv_addto.conf create mode 100644 paddle/gserver/tests/mkldnn_branches_conv_conv_addto_32c.conf create mode 100644 paddle/gserver/tests/mkldnn_branches_conv_conv_concat.conf create mode 100644 paddle/gserver/tests/mkldnn_branches_conv_conv_concat_32c.conf diff --git a/paddle/gserver/tests/CMakeLists.txt b/paddle/gserver/tests/CMakeLists.txt index fcee19415c..329536afaf 100644 --- a/paddle/gserver/tests/CMakeLists.txt +++ b/paddle/gserver/tests/CMakeLists.txt @@ -26,7 +26,10 @@ if(WITH_MKLDNN) test_MKLDNN.cpp MKLDNNTester.cpp LayerGradUtil.cpp) - add_test(NAME test_MKLDNN COMMAND test_MKLDNN) + add_test(NAME test_MKLDNN + COMMAND .set_python_path.sh -d ${PADDLE_SOURCE_DIR}/python + ${CMAKE_CURRENT_BINARY_DIR}/test_MKLDNN + WORKING_DIRECTORY ${PADDLE_SOURCE_DIR}/paddle) endif() ################ test_CRFLayerGrad #################### diff --git a/paddle/gserver/tests/MKLDNNTester.cpp b/paddle/gserver/tests/MKLDNNTester.cpp index eaebdd671c..3bf6a9e176 100644 --- a/paddle/gserver/tests/MKLDNNTester.cpp +++ b/paddle/gserver/tests/MKLDNNTester.cpp @@ -15,6 +15,7 @@ limitations under the License. */ #include "MKLDNNTester.h" #include "paddle/gserver/layers/MKLDNNBase.h" #include "paddle/gserver/layers/MKLDNNLayer.h" +#include "paddle/trainer/Trainer.h" namespace paddle { @@ -315,6 +316,7 @@ void MKLDNNTester::runOnce() { auto& value = para->getBuf(PARAMETER_VALUE); real lr = 1e-3; value->add(*grad, lr); + grad->zeroMem(); }; randomTopDiffs(); dnnLayer_->backward(updateCallback); @@ -411,4 +413,143 @@ void MKLDNNTester::run(const TestConfig& dnn, } } +void MKLDNNTester::initArgument(DataIn& data, + const std::string& configPath, + const size_t iter) { + TrainerConfigHelper config(configPath); + size_t batchSize = config.getOptConfig().batch_size(); + data.inArgs.resize(iter); + data.outGrads.resize(iter); + data.paraValues.clear(); + for (const auto& layer_name : config.getModelConfig().input_layer_names()) { + auto layer_config = std::find_if(config.getModelConfig().layers().begin(), + config.getModelConfig().layers().end(), + [=](const LayerConfig& layer_config) { + return layer_config.name() == layer_name; + }); + CHECK(layer_config != config.getModelConfig().layers().end()); + + size_t layerSize = layer_config->size(); + for (size_t i = 0; i < iter; ++i) { + Argument arg; + arg.value = Matrix::create(batchSize, layerSize, false, false); + arg.grad = Matrix::create(batchSize, layerSize, false, false); + arg.value->randomizeUniform(); + arg.value->add(-0.5); + arg.value->sigmoid(*arg.value); + arg.grad->zeroMem(); + arg.ids = VectorT::create(batchSize, false); + arg.ids->rand(layerSize); + generateSequenceStartPositions(batchSize, arg.sequenceStartPositions); + data.inArgs[i].push_back(arg); + } + } + + for (const auto& layer_name : config.getModelConfig().output_layer_names()) { + auto layer_config = std::find_if(config.getModelConfig().layers().begin(), + config.getModelConfig().layers().end(), + [=](const LayerConfig& layer_config) { + return layer_config.name() == layer_name; + }); + CHECK(layer_config != config.getModelConfig().layers().end()); + + size_t layerSize = layer_config->size(); + for (size_t i = 0; i < iter; ++i) { + MatrixPtr grad = Matrix::create(batchSize, layerSize, false, false); + grad->randomizeUniform(); + data.outGrads[i].push_back(grad); + } + } + + for (const auto& para_config : config.getModelConfig().parameters()) { + VectorPtr value = Vector::create(para_config.size(), false); + value->randnorm(0, 2); + data.paraValues.push_back(value); + } +} + +void MKLDNNTester::getOutResult(const std::string& configPath, + DataIn& in, + DataOut& out, + bool use_mkldnn, + size_t iter) { + FLAGS_use_gpu = false; + FLAGS_use_mkldnn = use_mkldnn; + *ThreadLocalRand::getSeed() = 1; + srand(1); + + Trainer trainer; + auto config = std::make_shared(configPath); + trainer.init(config, false); + auto gradientMachine = trainer.getGradientMachine(); + std::vector parameters = gradientMachine->getParameters(); + for (size_t i = 0; i < in.paraValues.size(); i++) { + parameters[i]->getBuf(PARAMETER_VALUE)->copyFrom(*in.paraValues[i]); + } + UpdateCallback simpleUpdate = [](Parameter* para) { + auto& grad = para->getBuf(PARAMETER_GRADIENT); + auto& value = para->getBuf(PARAMETER_VALUE); + real lr = 1e-2; + value->add(*grad, lr); + grad->zeroMem(); + }; + + vector outArgs; + gradientMachine->start(); + out.outValues.clear(); + out.paraValues.clear(); + for (size_t i = 0; i < iter; ++i) { + VLOG(MKLDNN_TESTS) << "runing iteration " << i; + gradientMachine->forward(in.inArgs[i], &outArgs, PASS_TRAIN); + // save forward result + for (size_t k = 0; k < outArgs.size(); k++) { + MatrixPtr value = Matrix::create(outArgs[k].value->getHeight(), + outArgs[k].value->getWidth(), + false, + false); + value->copyFrom(*outArgs[k].value); + out.outValues.push_back(value); + } + + // random backward input + for (size_t k = 0; k < outArgs.size(); k++) { + outArgs[k].grad->copyFrom(*in.outGrads[i][k]); + } + gradientMachine->backward(simpleUpdate); + } + gradientMachine->finish(); + + // save param value + for (size_t i = 0; i < in.paraValues.size(); i++) { + VectorPtr val = Vector::create( + parameters[i]->getBuf(PARAMETER_VALUE)->getSize(), false); + val->copyFrom(*parameters[i]->getBuf(PARAMETER_VALUE)); + out.paraValues.push_back(val); + } +} + +void MKLDNNTester::compareResult(DataOut& ref, DataOut& dnn, float eps) { + CHECK_EQ(ref.outValues.size(), dnn.outValues.size()); + CHECK_EQ(ref.paraValues.size(), dnn.paraValues.size()); + for (size_t i = 0; i < ref.outValues.size(); i++) { + EXPECT_LE(fabs(compareMatrix(ref.outValues[i], dnn.outValues[i])), eps); + } + for (size_t i = 0; i < ref.paraValues.size(); i++) { + EXPECT_LE(fabs(compareVector(ref.paraValues[i], dnn.paraValues[i])), eps); + } +} + +void MKLDNNTester::runBranchesTest(const std::string& configPath, + size_t iter, + float eps) { + DataIn in; + initArgument(in, configPath, iter); + + DataOut outCpu, outDnn; + getOutResult(configPath, in, outCpu, false, iter); + getOutResult(configPath, in, outDnn, true, iter); + + compareResult(outCpu, outDnn, eps); +} + } // namespace paddle diff --git a/paddle/gserver/tests/MKLDNNTester.h b/paddle/gserver/tests/MKLDNNTester.h index 171d176ee7..51abfcb67e 100644 --- a/paddle/gserver/tests/MKLDNNTester.h +++ b/paddle/gserver/tests/MKLDNNTester.h @@ -33,6 +33,17 @@ class MKLDNNTester { NUM = 2, // Number of total }; + struct DataIn { + std::vector> inArgs; + std::vector> outGrads; + std::vector paraValues; + }; + + struct DataOut { + std::vector outValues; + std::vector paraValues; + }; + protected: std::vector configs_; vector layerNames_; @@ -74,7 +85,17 @@ public: float epsilon = 1e-4, bool log = false, int level = MKLDNN_ALL); - void setLogLevel(int lvl) { lvl_ = lvl; } + static void runBranchesTest(const std::string& configPath, + size_t iter = 3, + float eps = 1e-4); + static void initArgument(DataIn& data, + const std::string& configPath, + size_t iter = 3); + static void getOutResult(const std::string& configPath, + DataIn& in, + DataOut& out, + bool use_mkldnn, + size_t iter = 3); private: void reset(const TestConfig& dnn, const TestConfig& ref, size_t batchSize); @@ -101,8 +122,9 @@ private: void saveWgt(const vector& from, vector& to); void restoreWgt(const vector& from, vector& to); - double compareMatrix(const MatrixPtr& m1, const MatrixPtr& m2); - double compareVector(const VectorPtr& v1, const VectorPtr& v2); + static double compareMatrix(const MatrixPtr& m1, const MatrixPtr& m2); + static double compareVector(const VectorPtr& v1, const VectorPtr& v2); + static void compareResult(DataOut& ref, DataOut& dnn, float eps = 1e-4); /** * Get delta percent @@ -111,11 +133,11 @@ private: * else return sum(abs(a-b)) / sum(abs(b)) * The return value should be smaller than eps when passing. */ - double getDelta(const real* d1, - const real* d2, - size_t len, - const float failRate = 1e-3, - const float thres = 0.1); + static double getDelta(const real* d1, + const real* d2, + size_t len, + const float failRate = 1e-3, + const float thres = 0.1); }; } // namespace paddle diff --git a/paddle/gserver/tests/mkldnn_branches_conv_conv_addto.conf b/paddle/gserver/tests/mkldnn_branches_conv_conv_addto.conf new file mode 100644 index 0000000000..59791ecc67 --- /dev/null +++ b/paddle/gserver/tests/mkldnn_branches_conv_conv_addto.conf @@ -0,0 +1,45 @@ +# Copyright (c) 2017 PaddlePaddle Authors. All Rights Reserved +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from paddle.trainer_config_helpers import * + +settings(batch_size=2) + +data = data_layer(name ="input", size=3*4*4) + +conv = img_conv_layer(input=data, + num_channels=1, + filter_size=3, + num_filters=2, + padding=1, + shared_biases=True, + act=ReluActivation()) + +b1 = img_conv_layer(input=conv, + filter_size=1, + num_filters=2, + padding=0, + shared_biases=True, + act=ReluActivation()) + +b2 = img_conv_layer(input=conv, + filter_size=3, + num_filters=2, + padding=1, + shared_biases=True, + act=ReluActivation()) + +concat = addto_layer(input=[b1, b2]) + +outputs(concat) diff --git a/paddle/gserver/tests/mkldnn_branches_conv_conv_addto_32c.conf b/paddle/gserver/tests/mkldnn_branches_conv_conv_addto_32c.conf new file mode 100644 index 0000000000..f6d30f12e8 --- /dev/null +++ b/paddle/gserver/tests/mkldnn_branches_conv_conv_addto_32c.conf @@ -0,0 +1,45 @@ +# Copyright (c) 2017 PaddlePaddle Authors. All Rights Reserved +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from paddle.trainer_config_helpers import * + +settings(batch_size=16) + +data = data_layer(name ="input", size=32*16*16) + +conv = img_conv_layer(input=data, + num_channels=32, + filter_size=3, + num_filters=32, + padding=1, + shared_biases=True, + act=ReluActivation()) + +b1 = img_conv_layer(input=conv, + filter_size=1, + num_filters=32, + padding=0, + shared_biases=True, + act=ReluActivation()) + +b2 = img_conv_layer(input=conv, + filter_size=3, + num_filters=32, + padding=1, + shared_biases=True, + act=ReluActivation()) + +concat = addto_layer(input=[b1, b2]) + +outputs(concat) diff --git a/paddle/gserver/tests/mkldnn_branches_conv_conv_concat.conf b/paddle/gserver/tests/mkldnn_branches_conv_conv_concat.conf new file mode 100644 index 0000000000..1178c6f5b2 --- /dev/null +++ b/paddle/gserver/tests/mkldnn_branches_conv_conv_concat.conf @@ -0,0 +1,45 @@ +# Copyright (c) 2017 PaddlePaddle Authors. All Rights Reserved +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from paddle.trainer_config_helpers import * + +settings(batch_size=2) + +data = data_layer(name ="input", size=3*4*4) + +conv = img_conv_layer(input=data, + num_channels=1, + filter_size=3, + num_filters=2, + padding=1, + shared_biases=True, + act=ReluActivation()) + +b1 = img_conv_layer(input=conv, + filter_size=1, + num_filters=2, + padding=0, + shared_biases=True, + act=ReluActivation()) + +b2 = img_conv_layer(input=conv, + filter_size=3, + num_filters=2, + padding=1, + shared_biases=True, + act=ReluActivation()) + +concat = concat_layer(input=[b1, b2]) + +outputs(concat) diff --git a/paddle/gserver/tests/mkldnn_branches_conv_conv_concat_32c.conf b/paddle/gserver/tests/mkldnn_branches_conv_conv_concat_32c.conf new file mode 100644 index 0000000000..8a624ac838 --- /dev/null +++ b/paddle/gserver/tests/mkldnn_branches_conv_conv_concat_32c.conf @@ -0,0 +1,45 @@ +# Copyright (c) 2017 PaddlePaddle Authors. All Rights Reserved +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from paddle.trainer_config_helpers import * + +settings(batch_size=16) + +data = data_layer(name ="input", size=32*16*16) + +conv = img_conv_layer(input=data, + num_channels=32, + filter_size=3, + num_filters=32, + padding=1, + shared_biases=True, + act=ReluActivation()) + +b1 = img_conv_layer(input=conv, + filter_size=1, + num_filters=32, + padding=0, + shared_biases=True, + act=ReluActivation()) + +b2 = img_conv_layer(input=conv, + filter_size=3, + num_filters=32, + padding=1, + shared_biases=True, + act=ReluActivation()) + +concat = concat_layer(input=[b1, b2]) + +outputs(concat) diff --git a/paddle/gserver/tests/test_MKLDNN.cpp b/paddle/gserver/tests/test_MKLDNN.cpp index 03515e9469..d455b738c2 100644 --- a/paddle/gserver/tests/test_MKLDNN.cpp +++ b/paddle/gserver/tests/test_MKLDNN.cpp @@ -13,6 +13,7 @@ See the License for the specific language governing permissions and limitations under the License. */ #include +#include #include #include #include "MKLDNNTester.h" @@ -40,13 +41,13 @@ DECLARE_bool(use_mkldnn); struct testFcDesc { int bs; int ic; - int oc; int ih, iw; // oh == ow == 1 + int oc; }; static void getMKLDNNFcConfig(TestConfig& cfg, const testFcDesc& pm) { cfg.layerConfig.set_type("mkldnn_fc"); - cfg.layerConfig.set_active_type("sigmoid"); + cfg.layerConfig.set_active_type("relu"); cfg.layerConfig.set_size(pm.oc); cfg.inputDefs.push_back( {INPUT_DATA, @@ -247,13 +248,23 @@ TEST(MKLDNNActivation, Activations) { } } -// TODO(TJ): add branch test +TEST(MKLDNNLayer, branches) { + std::vector cases = {"conv_conv_concat", + "conv_conv_concat_32c", + "conv_conv_addto", + "conv_conv_addto_32c"}; + for (auto name : cases) { + std::string config = "./gserver/tests/mkldnn_branches_" + name + ".conf"; + MKLDNNTester::runBranchesTest(config); + } +} int main(int argc, char** argv) { testing::InitGoogleTest(&argc, argv); FLAGS_use_gpu = false; FLAGS_use_mkldnn = true; initMain(argc, argv); + initPython(argc, argv); FLAGS_thread_local_rand_use_global_seed = true; srand(1); return RUN_ALL_TESTS(); From 78320194aa25c0ced76a3a758413dedfd24e9550 Mon Sep 17 00:00:00 2001 From: ranqiu Date: Tue, 17 Oct 2017 17:15:06 +0800 Subject: [PATCH 46/63] refine dot-product attention according to the comments --- .../paddle/trainer_config_helpers/networks.py | 41 ++++++++++--------- 1 file changed, 22 insertions(+), 19 deletions(-) diff --git a/python/paddle/trainer_config_helpers/networks.py b/python/paddle/trainer_config_helpers/networks.py index 0ecbacb7bb..120c9d11a5 100644 --- a/python/paddle/trainer_config_helpers/networks.py +++ b/python/paddle/trainer_config_helpers/networks.py @@ -1400,13 +1400,13 @@ def simple_attention(encoded_sequence, @wrap_name_default() def dot_product_attention(encoded_sequence, - attending_sequence, + attended_sequence, transformed_state, softmax_param_attr=None, name=None): """ Calculate and return a context vector with dot-product attention mechanism. - Size of the context vector equals to size of the attending_sequence. + The dimension of the context vector equals to that of the attended_sequence. .. math:: @@ -1419,35 +1419,38 @@ def dot_product_attention(encoded_sequence, c_{i} & = \\sum_{j=1}^{T_{x}}a_{i,j}z_{j} where :math:`h_{j}` is the jth element of encoded_sequence, - :math:`z_{j}` is the jth element of attending_sequence, - :math:`s_{i-1}` is transformed_state + :math:`z_{j}` is the jth element of attended_sequence, + :math:`s_{i-1}` is transformed_state. The example usage is: .. code-block:: python context = dot_product_attention(encoded_sequence=enc_seq, - attending_sequence=att_seq, + attended_sequence=att_seq, transformed_state=state,) - :param name: name of the dot-product attention model. + :param name: A prefix attached to the name of each layer that defined inside + the dot_product_attention. :type name: basestring - :param softmax_param_attr: parameter attribute of sequence softmax + :param softmax_param_attr: The parameter attribute of sequence softmax that is used to produce attention weight. :type softmax_param_attr: ParameterAttribute - :param encoded_sequence: output of the encoder + :param encoded_sequence: The output hidden vectors of the encoder. :type encoded_sequence: LayerOutput - :param attending_sequence: attention weight is computed by a feed forward neural - network which has two inputs : decoder's transformed - hidden state of previous time step and encoder's output. - attending_sequence is the sequence to be attended. - :type attending_sequence: LayerOutput - :param transformed_state: transformed hidden state of decoder in previous time step, - its size should equal to encoded_sequence's. Here we do the - transformation outside dot_product_attention for flexibility - consideration. + :param attended_sequence: The attention weight is computed by a feed forward neural + network which has two inputs : decoder's transformed hidden + state of previous time step and encoder's output. + attended_sequence is the sequence to be attended. + :type attended_sequence: LayerOutput + :param transformed_state: The transformed hidden state of decoder in previous time step. + Since the dot-product operation will be performed on it and the + encoded_sequence, their dimensions must be equal. For flexibility, + we suppose transformations of the decoder's hidden state have been + done outside dot_product_attention and no more will be performed + inside. Then users can use either the original or transformed one. :type transformed_state: LayerOutput - :return: a context vector + :return: The context vector. :rtype: LayerOutput """ assert transformed_state.size == encoded_sequence.size @@ -1470,7 +1473,7 @@ def dot_product_attention(encoded_sequence, scaled = scaling_layer( weight=attention_weight, - input=attending_sequence, + input=attended_sequence, name='%s_scaling' % name) return pooling_layer( From 60b84856e7402306b79a867aa321530f85c54856 Mon Sep 17 00:00:00 2001 From: tensor-tang Date: Tue, 17 Oct 2017 17:48:51 +0800 Subject: [PATCH 47/63] refine the conf files, combine in one file --- ...dto_32c.conf => mkldnn_branches_conv.conf} | 39 ++++++++++------ .../mkldnn_branches_conv_conv_addto.conf | 45 ------------------- .../mkldnn_branches_conv_conv_concat.conf | 45 ------------------- .../mkldnn_branches_conv_conv_concat_32c.conf | 45 ------------------- paddle/gserver/tests/test_MKLDNN.cpp | 13 +++--- 5 files changed, 33 insertions(+), 154 deletions(-) rename paddle/gserver/tests/{mkldnn_branches_conv_conv_addto_32c.conf => mkldnn_branches_conv.conf} (62%) delete mode 100644 paddle/gserver/tests/mkldnn_branches_conv_conv_addto.conf delete mode 100644 paddle/gserver/tests/mkldnn_branches_conv_conv_concat.conf delete mode 100644 paddle/gserver/tests/mkldnn_branches_conv_conv_concat_32c.conf diff --git a/paddle/gserver/tests/mkldnn_branches_conv_conv_addto_32c.conf b/paddle/gserver/tests/mkldnn_branches_conv.conf similarity index 62% rename from paddle/gserver/tests/mkldnn_branches_conv_conv_addto_32c.conf rename to paddle/gserver/tests/mkldnn_branches_conv.conf index f6d30f12e8..2628509db4 100644 --- a/paddle/gserver/tests/mkldnn_branches_conv_conv_addto_32c.conf +++ b/paddle/gserver/tests/mkldnn_branches_conv.conf @@ -15,31 +15,42 @@ from paddle.trainer_config_helpers import * settings(batch_size=16) +channels = get_config_arg("channels", int, 2) -data = data_layer(name ="input", size=32*16*16) +def two_conv(input, group_name): + out1 = img_conv_layer(input=input, + name=group_name+'_conv1', + filter_size=1, + num_filters=channels, + padding=0, + shared_biases=True, + act=ReluActivation()) -conv = img_conv_layer(input=data, - num_channels=32, + out2 = img_conv_layer(input=input, + name=group_name+'_conv2', filter_size=3, - num_filters=32, + num_filters=channels, padding=1, shared_biases=True, act=ReluActivation()) + return out1, out2 -b1 = img_conv_layer(input=conv, - filter_size=1, - num_filters=32, - padding=0, - shared_biases=True, - act=ReluActivation()) +data = data_layer(name ="input", size=channels*16*16) -b2 = img_conv_layer(input=conv, +conv = img_conv_layer(input=data, + num_channels=channels, filter_size=3, - num_filters=32, + num_filters=channels, padding=1, shared_biases=True, act=ReluActivation()) -concat = addto_layer(input=[b1, b2]) +a1, a2 = two_conv(input=conv, group_name='a') + +concat = concat_layer(input=[a1, a2]) + +b1, b2 = two_conv(input=conv, group_name='b') + +addto = addto_layer(input=[b1, b2]) -outputs(concat) +outputs([concat, addto]) diff --git a/paddle/gserver/tests/mkldnn_branches_conv_conv_addto.conf b/paddle/gserver/tests/mkldnn_branches_conv_conv_addto.conf deleted file mode 100644 index 59791ecc67..0000000000 --- a/paddle/gserver/tests/mkldnn_branches_conv_conv_addto.conf +++ /dev/null @@ -1,45 +0,0 @@ -# Copyright (c) 2017 PaddlePaddle Authors. All Rights Reserved -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -from paddle.trainer_config_helpers import * - -settings(batch_size=2) - -data = data_layer(name ="input", size=3*4*4) - -conv = img_conv_layer(input=data, - num_channels=1, - filter_size=3, - num_filters=2, - padding=1, - shared_biases=True, - act=ReluActivation()) - -b1 = img_conv_layer(input=conv, - filter_size=1, - num_filters=2, - padding=0, - shared_biases=True, - act=ReluActivation()) - -b2 = img_conv_layer(input=conv, - filter_size=3, - num_filters=2, - padding=1, - shared_biases=True, - act=ReluActivation()) - -concat = addto_layer(input=[b1, b2]) - -outputs(concat) diff --git a/paddle/gserver/tests/mkldnn_branches_conv_conv_concat.conf b/paddle/gserver/tests/mkldnn_branches_conv_conv_concat.conf deleted file mode 100644 index 1178c6f5b2..0000000000 --- a/paddle/gserver/tests/mkldnn_branches_conv_conv_concat.conf +++ /dev/null @@ -1,45 +0,0 @@ -# Copyright (c) 2017 PaddlePaddle Authors. All Rights Reserved -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -from paddle.trainer_config_helpers import * - -settings(batch_size=2) - -data = data_layer(name ="input", size=3*4*4) - -conv = img_conv_layer(input=data, - num_channels=1, - filter_size=3, - num_filters=2, - padding=1, - shared_biases=True, - act=ReluActivation()) - -b1 = img_conv_layer(input=conv, - filter_size=1, - num_filters=2, - padding=0, - shared_biases=True, - act=ReluActivation()) - -b2 = img_conv_layer(input=conv, - filter_size=3, - num_filters=2, - padding=1, - shared_biases=True, - act=ReluActivation()) - -concat = concat_layer(input=[b1, b2]) - -outputs(concat) diff --git a/paddle/gserver/tests/mkldnn_branches_conv_conv_concat_32c.conf b/paddle/gserver/tests/mkldnn_branches_conv_conv_concat_32c.conf deleted file mode 100644 index 8a624ac838..0000000000 --- a/paddle/gserver/tests/mkldnn_branches_conv_conv_concat_32c.conf +++ /dev/null @@ -1,45 +0,0 @@ -# Copyright (c) 2017 PaddlePaddle Authors. All Rights Reserved -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -from paddle.trainer_config_helpers import * - -settings(batch_size=16) - -data = data_layer(name ="input", size=32*16*16) - -conv = img_conv_layer(input=data, - num_channels=32, - filter_size=3, - num_filters=32, - padding=1, - shared_biases=True, - act=ReluActivation()) - -b1 = img_conv_layer(input=conv, - filter_size=1, - num_filters=32, - padding=0, - shared_biases=True, - act=ReluActivation()) - -b2 = img_conv_layer(input=conv, - filter_size=3, - num_filters=32, - padding=1, - shared_biases=True, - act=ReluActivation()) - -concat = concat_layer(input=[b1, b2]) - -outputs(concat) diff --git a/paddle/gserver/tests/test_MKLDNN.cpp b/paddle/gserver/tests/test_MKLDNN.cpp index d455b738c2..3571fbb9e3 100644 --- a/paddle/gserver/tests/test_MKLDNN.cpp +++ b/paddle/gserver/tests/test_MKLDNN.cpp @@ -248,14 +248,17 @@ TEST(MKLDNNActivation, Activations) { } } +DECLARE_string(config_args); TEST(MKLDNNLayer, branches) { - std::vector cases = {"conv_conv_concat", - "conv_conv_concat_32c", - "conv_conv_addto", - "conv_conv_addto_32c"}; + std::vector cases = {"conv"}; for (auto name : cases) { std::string config = "./gserver/tests/mkldnn_branches_" + name + ".conf"; - MKLDNNTester::runBranchesTest(config); + for (auto channels : {2, 32}) { + std::ostringstream oss; + oss << "channels=" << channels; + FLAGS_config_args = oss.str(); + MKLDNNTester::runBranchesTest(config); + } } } From df0946ebe211870451b92b1829b71cf4f590a82e Mon Sep 17 00:00:00 2001 From: Qiao Longfei Date: Tue, 17 Oct 2017 11:05:24 -0700 Subject: [PATCH 48/63] Impl optimizer (#4734) * init parameter base class * optimize the Comments of optimizer * basic implimentation of optimizer * add test_optimizer * add no_grad_set to interface * update optimizer.py * python code can run * fix some problem * add sync_with_cpp to Python Program and Block * sync vars and ops in block from cpp * optimize code and add some comment * add more check for sync * update optimizer with return value of Backward * rm unused code * infer shape when create gradient vairiable * update test_optimizer * update test_program.py * update backward test * follow comment --- paddle/framework/backward.cc | 34 ++++- paddle/framework/backward_test.cc | 81 +++++++++--- paddle/operators/mul_op.cc | 8 +- paddle/pybind/protobuf.cc | 5 + python/paddle/v2/framework/framework.py | 17 ++- python/paddle/v2/framework/optimizer.py | 124 ++++++++++++++++++ .../v2/framework/tests/test_optimizer.py | 31 +++++ .../paddle/v2/framework/tests/test_program.py | 70 +++++----- 8 files changed, 302 insertions(+), 68 deletions(-) create mode 100644 python/paddle/v2/framework/optimizer.py create mode 100644 python/paddle/v2/framework/tests/test_optimizer.py diff --git a/paddle/framework/backward.cc b/paddle/framework/backward.cc index c78e056071..ac80879c54 100644 --- a/paddle/framework/backward.cc +++ b/paddle/framework/backward.cc @@ -281,12 +281,16 @@ static void CreateGradVarInBlock( auto ops = block_desc->AllOps(); for (size_t op_index = grad_op_start_index; op_index < ops.size(); ++op_index) { + bool need_infer_shape = false; ForEachVarName(ops[op_index]->Outputs(), [&](const std::string& grad_var_name) { if (block_desc->HasVar(grad_var_name)) { return false; } - block_desc->Var(grad_var_name); + need_infer_shape = true; + auto var = block_desc->Var(grad_var_name); + // FIXME(qiao) infer the datatype + var->SetDataType(framework::DataType::FP32); auto it = param_name_map.find(grad_var_name); if (it == param_name_map.end()) { return false; @@ -298,6 +302,9 @@ static void CreateGradVarInBlock( grad_record.op_idx_ = static_cast(op_index); return false; /* not break */ }); + if (need_infer_shape) { + ops[op_index]->InferShape(*block_desc); + } } } @@ -428,10 +435,16 @@ ParamGradInfoMap AppendBackward( auto& all_ops = root_block->ops_; // insert fill one op for target + // TODO(qiao) add some check to the target. std::string fill_one_op_out = GradVarName(target.Name()); + std::vector target_shape_desc = target.Shape(); + std::vector target_shape; + std::transform(target_shape_desc.begin(), target_shape_desc.end(), + std::back_inserter(target_shape), + [](int64_t dim) { return static_cast(dim); }); std::unique_ptr fill_one_op( new OpDescBind("fill_constant", {}, {{"Out", {fill_one_op_out}}}, - {{"shape", std::vector{1}}, + {{"shape", target_shape}, {"value", static_cast(1.0)}, {"data_type", framework::DataType::FP32}})); all_ops.push_back(std::move(fill_one_op)); @@ -443,13 +456,22 @@ ParamGradInfoMap AppendBackward( auto backward_op_descs = MakeBlockBackward(program_desc, root_block_idx, &no_grad_var_names, &grad_to_var); - std::unordered_map retv; - - // Create Variable for (auto& ptr : backward_op_descs) { all_ops.push_back(std::move(ptr)); } - root_block->Var(fill_one_op_out); + // Create Variable + + // Create target gradient variable + std::unordered_map retv; + + auto var = root_block->Var(fill_one_op_out); + // FIXME(qiao) infer the data type + var->SetDataType(framework::DataType::FP32); + var->SetShape(target.Shape()); + auto& target_grad = retv[target.Name()]; + target_grad.name_ = fill_one_op_out; + target_grad.block_idx_ = root_block_idx; + target_grad.op_idx_ = static_cast(forward_op_num); // create grad_var for all blocks in this program CreateGradVarInBlock(forward_op_num, grad_to_var, root_block, &retv); diff --git a/paddle/framework/backward_test.cc b/paddle/framework/backward_test.cc index 5302afcafb..0c35a157bc 100644 --- a/paddle/framework/backward_test.cc +++ b/paddle/framework/backward_test.cc @@ -26,6 +26,20 @@ namespace framework { using DeviceContext = platform::DeviceContext; +class NoneOp : public framework::OperatorWithKernel { + public: + using framework::OperatorWithKernel::OperatorWithKernel; + + protected: + void InferShape(framework::InferShapeContext *ctx) const override {} +}; + +template +class NoneKernel : public framework::OpKernel { + public: + void Compute(const framework::ExecutionContext &context) const override {} +}; + class RowWiseAddOpMaker : public OpProtoAndCheckerMaker { public: RowWiseAddOpMaker(OpProto *proto, OpAttrChecker *op_checker) @@ -215,19 +229,51 @@ class MinusOpMaker : public OpProtoAndCheckerMaker { namespace f = paddle::framework; namespace ops = paddle::operators; using EnforceNotMet = paddle::platform::EnforceNotMet; -REGISTER_OPERATOR(rowwise_add, f::NOP, f::RowWiseAddOpMaker, +// rowwise_add +REGISTER_OPERATOR(rowwise_add, f::NoneOp, f::RowWiseAddOpMaker, f::RowWiseAddGradMaker); -REGISTER_OPERATOR(rowwise_add_grad, f::NOP); -REGISTER_OP(mul, f::NOP, f::MulOpMaker, mul_grad, f::NOP); -REGISTER_OP(sigmoid, f::NOP, f::SigmoidOpMaker, sigmoid_grad, f::NOP); -REGISTER_OP_WITHOUT_GRADIENT(nograd, f::NOP, f::NoGradOpMaker); -REGISTER_OP_WITHOUT_GRADIENT(fill_zeros_like, f::NOP, f::FillZeroOpMaker); -REGISTER_OP(sum, f::NOP, f::SumOpMaker, sum_grad, f::NOP); +REGISTER_OP_CPU_KERNEL(rowwise_add, + f::NoneKernel); +REGISTER_OPERATOR(rowwise_add_grad, f::NoneOp); +REGISTER_OP_CPU_KERNEL(rowwise_add_grad, + f::NoneKernel); +// mul +REGISTER_OP(mul, f::NoneOp, f::MulOpMaker, mul_grad, f::NoneOp); +REGISTER_OP_CPU_KERNEL(mul, f::NoneKernel); +REGISTER_OP_CPU_KERNEL(mul_grad, + f::NoneKernel); +// sigmoid +REGISTER_OP(sigmoid, f::NoneOp, f::SigmoidOpMaker, sigmoid_grad, f::NoneOp); +REGISTER_OP_CPU_KERNEL(sigmoid, + f::NoneKernel); +REGISTER_OP_WITHOUT_GRADIENT(nograd, f::NoneOp, f::NoGradOpMaker); +// fill_zeros_like +REGISTER_OP_WITHOUT_GRADIENT(fill_zeros_like, f::NoneOp, f::FillZeroOpMaker); +REGISTER_OP_CPU_KERNEL(fill_zeros_like, + f::NoneKernel); +// sum +REGISTER_OP(sum, f::NoneOp, f::SumOpMaker, sum_grad, f::NoneOp); +REGISTER_OP_CPU_KERNEL(sum, f::NoneKernel); +REGISTER_OP_CPU_KERNEL(sum_grad, + f::NoneKernel); +// fc REGISTER_OP_WITHOUT_GRADIENT(fc, f::FcOp, f::FcOpMaker); -REGISTER_OP(many_output_op, f::NOP, f::ManyOutputOpMaker, many_output_op_grad, - f::NOP); -REGISTER_OP(mult_in_out, f::NOP, f::MultInOutOpMaker, mult_in_out_grad, f::NOP); -REGISTER_OPERATOR(minus, f::NOP, f::MinusOpMaker, f::MinusGradOpDescMaker); +// many_output_op +REGISTER_OP(many_output_op, f::NoneOp, f::ManyOutputOpMaker, + many_output_op_grad, f::NoneOp); +// mult_in_out +REGISTER_OP(mult_in_out, f::NoneOp, f::MultInOutOpMaker, mult_in_out_grad, + f::NoneOp); +REGISTER_OP_CPU_KERNEL(mult_in_out, + f::NoneKernel); +REGISTER_OP_CPU_KERNEL(mult_in_out_grad, + f::NoneKernel); +// minus +REGISTER_OPERATOR(minus, f::NoneOp, f::MinusOpMaker, f::MinusGradOpDescMaker); +REGISTER_OP_CPU_KERNEL(minus, f::NoneKernel); +// scale +REGISTER_OPERATOR(scale, f::NoneOp); +REGISTER_OP_CPU_KERNEL(scale, f::NoneKernel); TEST(Backward, simple_op_not_need_grad) { auto fwd = f::OpRegistry::CreateOp( @@ -463,6 +509,7 @@ TEST(Backward, simple_single_op) { f::ProgramDesc *program_desc = GetNewProgramDesc(); f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); f::BlockDescBind *block = program.Block(0); + f::OpDescBind *op = block->AppendOp(); op->SetType("rowwise_add"); op->SetInput("X", {"x"}); @@ -487,7 +534,7 @@ TEST(Backward, simple_single_op) { EXPECT_EQ(grad_op->Output(f::GradVarName("b")), std::vector({f::GradVarName("b")})); - EXPECT_EQ(var_to_grad.size(), 2UL); + EXPECT_EQ(var_to_grad.size(), 3UL); EXPECT_EQ(var_to_grad.at("b"), f::GradVarInfo(f::GradVarName("b"), 0, 2)); EXPECT_EQ(var_to_grad.at("x"), f::GradVarInfo(f::GradVarName("x"), 0, 2)); @@ -588,7 +635,7 @@ TEST(Backward, simple_mult_op) { EXPECT_EQ(grad_op3->Output(f::GradVarName("b")), std::vector({f::GradVarName("b3")})); - EXPECT_EQ(var_to_grad.size(), 6UL); + EXPECT_EQ(var_to_grad.size(), 7UL); EXPECT_EQ(var_to_grad.at("x1"), f::GradVarInfo(f::GradVarName("x1"), 0, 6)); EXPECT_EQ(var_to_grad.at("b1"), f::GradVarInfo(f::GradVarName("b1"), 0, 6)); EXPECT_EQ(var_to_grad.at("out1"), @@ -666,7 +713,7 @@ TEST(Backward, intermedia_var_no_grad) { std::vector({f::GradVarName("out1")})); EXPECT_EQ(grad_op4->Output(f::GradVarName("Y")), std::vector()); - EXPECT_EQ(var_to_grad.size(), 3UL); + EXPECT_EQ(var_to_grad.size(), 4UL); EXPECT_EQ(var_to_grad.at("x1"), f::GradVarInfo(f::GradVarName("x1"), 0, 6)); EXPECT_EQ(var_to_grad.at("b1"), f::GradVarInfo(f::GradVarName("b1"), 0, 6)); EXPECT_EQ(var_to_grad.at("out1"), @@ -744,7 +791,7 @@ TEST(Backward, var_no_grad) { EXPECT_EQ(grad_op1->Output(f::GradVarName("H")), std::vector({f::GradVarName("h1")})); - EXPECT_EQ(var_to_grad.size(), 3UL); + EXPECT_EQ(var_to_grad.size(), 4UL); EXPECT_EQ(var_to_grad.at("y1"), f::GradVarInfo(f::GradVarName("y1"), 0, 3)); EXPECT_EQ(var_to_grad.at("x1"), f::GradVarInfo(f::GradVarName("x1"), 0, 5)); EXPECT_EQ(var_to_grad.at("h1"), f::GradVarInfo(f::GradVarName("h1"), 0, 5)); @@ -830,7 +877,7 @@ TEST(Backward, shared_var) { EXPECT_EQ(grad_op1->Output(f::GradVarName("b")), std::vector({f::GradVarName("b1")})); - EXPECT_EQ(var_to_grad.size(), 5UL); + EXPECT_EQ(var_to_grad.size(), 6UL); EXPECT_EQ(var_to_grad.at("b3"), f::GradVarInfo(f::GradVarName("b3"), 0, 4)); EXPECT_EQ(var_to_grad.at("y2"), f::GradVarInfo(f::GradVarName("y2"), 0, 5)); EXPECT_EQ(var_to_grad.at("out1"), @@ -863,7 +910,7 @@ TEST(Backward, half_backward) { auto ops = block->AllOps(); ASSERT_EQ(3UL, ops.size()); - EXPECT_EQ(var_to_grad.size(), 1UL); + EXPECT_EQ(var_to_grad.size(), 2UL); EXPECT_EQ(var_to_grad.at("a"), f::GradVarInfo(f::GradVarName("a"), 0, forward_len + 1)); } diff --git a/paddle/operators/mul_op.cc b/paddle/operators/mul_op.cc index 943f81e949..065800f250 100644 --- a/paddle/operators/mul_op.cc +++ b/paddle/operators/mul_op.cc @@ -104,10 +104,10 @@ class MulOpGrad : public framework::OperatorWithKernel { auto y_dims = ctx->GetInputDim("Y"); auto out_dims = ctx->GetInputDim(framework::GradVarName("Out")); - auto x_mat_dims = - framework::flatten_to_2d(x_dims, Attr("x_num_col_dims")); - auto y_mat_dims = - framework::flatten_to_2d(y_dims, Attr("y_num_col_dims")); + auto x_mat_dims = framework::flatten_to_2d( + x_dims, ctx->Attrs().Get("x_num_col_dims")); + auto y_mat_dims = framework::flatten_to_2d( + y_dims, ctx->Attrs().Get("y_num_col_dims")); PADDLE_ENFORCE_EQ( x_mat_dims[0], out_dims[0], diff --git a/paddle/pybind/protobuf.cc b/paddle/pybind/protobuf.cc index b360b05d16..82aae72ba9 100644 --- a/paddle/pybind/protobuf.cc +++ b/paddle/pybind/protobuf.cc @@ -163,6 +163,11 @@ void BindBlockDesc(py::module &m) { return self.Var(name); }, py::return_value_policy::reference) + .def("has_var", + [](BlockDescBind &self, py::bytes byte_name) { + std::string name = byte_name; + return self.HasVar(name); + }) .def("find_var", [](BlockDescBind &self, py::bytes byte_name) { std::string name = byte_name; diff --git a/python/paddle/v2/framework/framework.py b/python/paddle/v2/framework/framework.py index a17f988bf4..3fb6efe42a 100644 --- a/python/paddle/v2/framework/framework.py +++ b/python/paddle/v2/framework/framework.py @@ -306,6 +306,14 @@ class Block(object): def idx(self): return self.desc.id + def var(self, name): + if name not in self.vars: + raise ValueError("var %s not in this block" % name) + return self.vars[name] + + def all_parameters(self): + return {v for k, v in self.vars.iteritems() if isinstance(v, Parameter)} + def create_var(self, *args, **kwargs): return Variable(self, *args, **kwargs) @@ -314,7 +322,8 @@ class Block(object): def create_parameter(self, *args, **kwargs): global_block = self.program.global_block() - return Parameter(global_block, *args, **kwargs) + param = Parameter(global_block, *args, **kwargs) + return param def append_op(self, *args, **kwargs): op_desc = self.desc.append_op() @@ -392,10 +401,16 @@ class Program(object): def global_block(self): return self.blocks[0] + def block(self, index): + return self.blocks[index] + def current_block(self): return self.blocks[self.current_block_idx] def append_backward(self, target, no_grad_set): + """ + return map(param_name -> (grad_name, block_index, op_index)) + """ assert isinstance(target, Variable) param_to_grad_info = self.desc.append_backward(target.desc, no_grad_set) self.sync_with_cpp() diff --git a/python/paddle/v2/framework/optimizer.py b/python/paddle/v2/framework/optimizer.py new file mode 100644 index 0000000000..e356a7aadb --- /dev/null +++ b/python/paddle/v2/framework/optimizer.py @@ -0,0 +1,124 @@ +import paddle.v2.framework.framework as framework + +__all__ = ['SGDOptimizer'] + + +class Optimizer(object): + """Optimizer Base class. + + Define the common interface of an optimizer. + User should not use this class directly, but need to use one of it's implementation. + """ + + def __init__(self): + pass + + def _append_optimize_op(self, block, param_and_grad): + """ append optimize operator to block and return all the added optimize_op + """ + raise NotImplementedError() + + def create_backward_pass(self, loss, parameter_list=None, no_grad_set=None): + """ + create and add gradient Operators in BlockDesc to Compute gradients of `loss` + for parameters in parameter_list + + Args: + loss: an variable generated by cost function. + no_grad_set: variable that should not create gradient + parameter_list: parameters that need to compute gradient and update to optimize the lost. + + Returns: + list of (parameters, gradients) pair. + """ + assert isinstance(loss, framework.Variable) + param_grad_map = loss.block.program.append_backward(loss, no_grad_set or + set()) + if parameter_list is not None: + parameters = parameter_list + else: + params = loss.block.program.global_block().all_parameters() + parameters = [param.name for param in params] + params_and_grads = [] + for param in parameters: + if param not in param_grad_map: + raise Exception("param %s is not in map" % param) + grad_info = param_grad_map[param] + grad_block = loss.block.program.block(grad_info[1]) + if not grad_block.has_var(grad_info[0]): + raise Exception("grad block[%d] did not have grad var %s" % + grad_info[1], grad_info[0]) + param_var = loss.block.var(param) + grad_var = grad_block.var(grad_info[0]) + if loss.block.has_var(grad_info[0]): + params_and_grads.append((param_var, grad_var)) + else: + params_and_grads.append((param_var, None)) + return params_and_grads + + def create_optimization_pass(self, parameters_and_grads, loss): + """Add optimization operators to update gradients to variables. + + Args: + loss: the target that this optimization is for. + parameters_and_grads: a list of (variable, gradient) pair to update. + + Returns: + optmization_op_list: a list of optimization operator that will update parameter using gradient. + """ + optimize_ops = [] + for param_and_grad in parameters_and_grads: + if param_and_grad[1] is not None: + optimize_op = self._append_optimize_op(loss.block, + param_and_grad) + optimize_ops.append(optimize_op) + return optimize_ops + + def minimize(self, loss, parameter_list=None, no_grad_set=None): + """Add operations to minimize `loss` by updating `parameter_list`. + + This method combines interface `create_backward_pass()` and + `create_optimization_pass()` into one. + """ + params_grads = self.create_backward_pass(loss, parameter_list, + no_grad_set or set()) + optimize_ops = self.create_optimization_pass(params_grads, loss) + return optimize_ops + + +class SGDOptimizer(Optimizer): + """ Simple SGD optimizer without any state. + """ + + def __init__(self, learning_rate): + assert learning_rate is not None + super(Optimizer, self).__init__() + self.type = "sgd" + self._learning_rate = learning_rate + + def _append_optimize_op(self, block, param_and_grad): + assert isinstance(block, framework.Block) + lr_shape = [1] + # create a var for learning_rate + lr = block.create_var(dtype="float32", shape=lr_shape, lod_level=0) + + # create an op to init the learning_rate + init_op = block.append_op( + type="fill_constant", + outputs={"Out": lr}, + attrs={"shape": lr_shape, + "value": self._learning_rate}) + + # create the optimize op + sgd_op = block.append_op( + type=self.type, + inputs={ + "Param": param_and_grad[0], + "Grad": param_and_grad[1], + "LearningRate": lr + }, + outputs={"ParamOut": param_and_grad[0]}, + attrs={"shape": [1], + "value": self._learning_rate}) + + return sgd_op diff --git a/python/paddle/v2/framework/tests/test_optimizer.py b/python/paddle/v2/framework/tests/test_optimizer.py new file mode 100644 index 0000000000..3d6fa70737 --- /dev/null +++ b/python/paddle/v2/framework/tests/test_optimizer.py @@ -0,0 +1,31 @@ +import unittest + +import paddle.v2.framework.framework as framework +import paddle.v2.framework.optimizer as optimizer + + +class TestOptimizer(unittest.TestCase): + def test_sgd_optimizer(self): + program = framework.g_program + block = program.global_block() + mul_x = block.create_parameter( + dtype="float32", shape=[5, 10], lod_level=0, name="mul.x") + mul_y = block.create_var( + dtype="float32", shape=[10, 8], lod_level=0, name="mul.y") + mul_out = block.create_var( + dtype="float32", shape=[5, 8], lod_level=0, name="mul.out") + mul_op = block.append_op( + type="mul", + inputs={"X": mul_x, + "Y": mul_y}, + outputs={"Out": mul_out}, + attrs={"x_num_col_dims": 1}) + sgd_optimizer = optimizer.SGDOptimizer(learning_rate=0.01) + opts = sgd_optimizer.minimize(mul_out) + self.assertEqual(len(opts), 1) + sgd_op = opts[0] + self.assertEqual(sgd_op.type, "sgd") + + +if __name__ == '__main__': + unittest.main() diff --git a/python/paddle/v2/framework/tests/test_program.py b/python/paddle/v2/framework/tests/test_program.py index d06f86c09f..c98dc3492b 100644 --- a/python/paddle/v2/framework/tests/test_program.py +++ b/python/paddle/v2/framework/tests/test_program.py @@ -34,49 +34,11 @@ class TestProgram(unittest.TestCase): self.assertEqual(1, b.idx) self.assertEqual(0, b.parent_idx) - def test_desc_append_backward(self): - prog = core.ProgramDesc.__create_program_desc__() - self.assertIsNotNone(prog) - block = prog.block(0) - self.assertIsNotNone(block) - - mul_op_desc = block.append_op() - mul_op_desc.set_type("mul") - mul_op_desc.set_input("X", ["x1"]) - mul_op_desc.set_input("Y", ["y1"]) - mul_op_desc.set_output("Out", ["out1"]) - - sum_op_desc = block.append_op() - sum_op_desc.set_type("elementwise_add") - sum_op_desc.set_input("X", ["out1"]) - sum_op_desc.set_input("Y", ["b1"]) - sum_op_desc.set_output("Out", ["out2"]) - - target = block.var("out2") - - expect_ops = [ - "mul", "elementwise_add", "fill_constant", "elementwise_add_grad", - "mul_grad" - ] - - def grad_name(name): - return name + "@GRAD" - - actual_ops = [] - param_to_grad = prog.append_backward(target, set()) - for var_name in ("x1", "y1", "out1", "b1"): - self.assertEqual(param_to_grad[var_name][0], grad_name(var_name)) - self.assertEqual(param_to_grad[var_name][1], 0) - - for op in block.all_ops(): - actual_ops.append(op.type()) - self.assertEqual(actual_ops, expect_ops) - def test_append_backward(self): prog = Program.instance() block = prog.global_block() - mul_x = block.create_parameter( + mul_x = block.create_var( dtype="float32", shape=[5, 10], lod_level=0, name="mul.x") mul_y = block.create_var( dtype="float32", shape=[10, 8], lod_level=0, name="mul.y") @@ -88,7 +50,35 @@ class TestProgram(unittest.TestCase): "Y": mul_y}, outputs={"Out": [mul_out]}, attrs={"x_num_col_dims": 1}) - param_to_grad = prog.append_backward(mul_out, set()) + + add_y = block.create_var( + dtype="float32", shape=[5, 8], lod_level=0, name="add.y") + add_out = block.create_var( + dtype="float32", shape=[5, 8], lod_level=0, name="add.out") + add_op = block.append_op( + type="elementwise_add", + inputs={"X": mul_out, + "Y": add_y}, + outputs={"Out": add_out}, + attrs={"x_num_col_dims": 1}) + + param_to_grad = prog.append_backward(add_out, set()) + + def grad_name(name): + return name + "@GRAD" + + for var_name in ("mul.x", "mul.y", "mul.out", "add.y", "add.out"): + self.assertEqual(param_to_grad[var_name][0], grad_name(var_name)) + self.assertEqual(param_to_grad[var_name][1], 0) + + expect_ops = [ + "mul", "elementwise_add", "fill_constant", "elementwise_add_grad", + "mul_grad" + ] + actual_ops = [] + for op in block.ops: + actual_ops.append(op.type) + self.assertEqual(actual_ops, expect_ops) if __name__ == '__main__': From 46f7854e5c911c0e742ac3abed2c6c9007347bcf Mon Sep 17 00:00:00 2001 From: Qiao Longfei Date: Tue, 17 Oct 2017 12:48:14 -0700 Subject: [PATCH 49/63] fix mac compile (#4870) --- paddle/gserver/gradientmachines/NeuralNetwork.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/paddle/gserver/gradientmachines/NeuralNetwork.h b/paddle/gserver/gradientmachines/NeuralNetwork.h index 16971883b4..6888380290 100644 --- a/paddle/gserver/gradientmachines/NeuralNetwork.h +++ b/paddle/gserver/gradientmachines/NeuralNetwork.h @@ -135,7 +135,7 @@ public: const std::string& getName() const { return subModelName_; } /// some finish work, like convert the weight format of MKLDNNLayers - void finish() override; + void finish(); protected: /** From e23c1fee66b80c8faf1c01840ddc10ff64e0187e Mon Sep 17 00:00:00 2001 From: Abhinav Arora Date: Tue, 17 Oct 2017 16:41:18 -0700 Subject: [PATCH 50/63] Correcting some grammatical mistakes in SelectedRows Document (#4872) --- doc/design/selected_rows.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/design/selected_rows.md b/doc/design/selected_rows.md index 9e6f3b20cb..1a98839a95 100644 --- a/doc/design/selected_rows.md +++ b/doc/design/selected_rows.md @@ -1,6 +1,6 @@ # Design Doc: Selected Rows -`SelectedRows` is a kind of sparse tensor data type, which is designed to support `embedding` operators. The gradient of embedding table is a sparse tensor. Only a few rows are non-zero values in that tensor. It is straightforward to represent the sparse tensor by the following sparse tensor data structure: +`SelectedRows` is a type of sparse tensor data type, which is designed to support `embedding` operators. The gradient of embedding table is a sparse tensor. Only a few rows are non-zero values in this tensor. It is straight-forward to represent a sparse tensor by the following sparse tensor data structure: ```cpp class SelectedRows { @@ -11,7 +11,7 @@ class SelectedRows { }; ``` -The field `height_` shows the first dimension of `SelectedRows`. The `rows` are the indices of which rows of `SelectedRows` are non-zeros. The `value_` field is an N-dim tensor and shape is `[rows.size() /* NUM_ROWS */, ...]`, which supplies values for each row. The dimension of `SelectedRows` satisfies `[height_] + value_.shape[1:]`. +The field `height_` is the first dimension of `SelectedRows`. The `rows` are the indices of the non-zero rows of `SelectedRows`. The `value_` field is an N-dim tensor of shape `[rows.size() /* NUM_ROWS */, ...]`, which supplies values for each row. The dimension of `SelectedRows` satisfies `[height_] + value_.shape[1:]`. Suppose that a SelectedRows-typed variable `x` has many rows, but only two of them have values -- row 73 is `[1, 2]` and row 84 is `[3, 4]`, the `SelectedRows` representation would be: @@ -25,7 +25,7 @@ x = SelectedRow { ## SelectedRows in Protobuf -`SelectedRows` is a kind of `Variable`. `VarDesc` in protobuf should describe the `SelectedRows` information. Only the tensor dimension of a `SelectedRows` will be described in compile-time since the `rows_` and `value_` are related to training data. +`SelectedRows` is a type of `Variable`. `VarDesc` in protobuf should describe the `SelectedRows` information. Only the tensor dimension of a `SelectedRows` will be described in compile-time because the `rows_` and `value_` are dependent on the training data. So we use `TensorDesc` to unify `data_type` and `dims`. A LodTensorDesc contains a `TensorDesc` and `lod_level`. The description of `SelectedRows` is a Tensor description. ```proto @@ -54,7 +54,7 @@ message VarDesc { ## InferShape for Selected Rows -Just like `LoD` information, `InferShape` method will inference output tensor type as well. The operator should decide whether its output is a `SelectedRows` or `Dense` tensor. +Just like `LoD` information, `InferShape` method will infer the output tensor type as well. The operator should decide whether its output is a `SelectedRows` or `Dense` tensor. For example, the gradient operator of `TableLookup` will always generate `SelectedRows`. Its `InferShape` method should be like following @@ -68,7 +68,7 @@ void TableLookupGrad::InferShape(context) { ## Sparse Operators -There are several operators should be written to support `SelectedRows`. They are: +There are several operators that need to be written to support `SelectedRows`. These are: -1. Operators which generates `SelectedRows` gradient. e.g. Gradient of `TableLookupOp`. +1. Operators which generate `SelectedRows` gradient. e.g. Gradient of `TableLookupOp`. 2. Optimize operators which support `SelectedRows` gradient. e.g. `SGD` or `AdaGrad` for `SelectedRows`. However, there should be only one `SGD` operator. `OpWithKernel::Run` should select a suitable kernel for both `dense` tensor or `SelectedRows`. From 5d67677c607b0b6517f3da77e55b5447fb769b1e Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Tue, 17 Oct 2017 17:49:54 -0700 Subject: [PATCH 51/63] Remove private data members in OpRegister (#4871) --- paddle/framework/op_registry.h | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/paddle/framework/op_registry.h b/paddle/framework/op_registry.h index 0bda87dfa1..dfca46b789 100644 --- a/paddle/framework/op_registry.h +++ b/paddle/framework/op_registry.h @@ -45,18 +45,15 @@ class Registrar { template struct OperatorRegistrar : public Registrar { - explicit OperatorRegistrar(const char* op_type) : op_type(op_type) { + explicit OperatorRegistrar(const char* op_type) { PADDLE_ENFORCE(!OpInfoMap::Instance().Has(op_type), "'%s' is registered more than once.", op_type); static_assert(sizeof...(ARGS) != 0, "OperatorRegistrar should be invoked at least by OpClass"); + OpInfo info; details::OperatorRegistrarRecursive<0, false, ARGS...>(op_type, &info); OpInfoMap::Instance().Insert(op_type, info); } - - const char* op_type; - - OpInfo info; }; class OpRegistry { From 164898277c3f274e6d57c4baceb00fe62a8b769c Mon Sep 17 00:00:00 2001 From: Markus Kliegl Date: Tue, 17 Oct 2017 19:29:09 -0700 Subject: [PATCH 52/63] MatMul operator (#4856) * initial matmul operator Similar to np.matmul, but also has transpose_X and transpose_Y flags, and only supports tensors from rank 1 to 3 inclusive. For GPU, uses cublas?gemmStridedBatched. For CPU, uses cblas_?gemm_batch if available via MKL; otherwise a simple serial implementation that loops over the batch dimension is employed for now. --- paddle/operators/math/math_function.cc | 81 +++++++ paddle/operators/math/math_function.cu | 48 ++++ paddle/operators/math/math_function.h | 10 +- paddle/operators/math/matmul.h | 124 ++++++++++ paddle/operators/matmul_op.cc | 208 ++++++++++++++++ paddle/operators/matmul_op.cu | 21 ++ paddle/operators/matmul_op.h | 228 ++++++++++++++++++ paddle/platform/dynload/cublas.h | 4 + .../v2/framework/tests/test_matmul_op.py | 119 +++++++++ 9 files changed, 842 insertions(+), 1 deletion(-) create mode 100644 paddle/operators/math/matmul.h create mode 100644 paddle/operators/matmul_op.cc create mode 100644 paddle/operators/matmul_op.cu create mode 100644 paddle/operators/matmul_op.h create mode 100644 python/paddle/v2/framework/tests/test_matmul_op.py diff --git a/paddle/operators/math/math_function.cc b/paddle/operators/math/math_function.cc index 77a1e22b41..aad1357598 100644 --- a/paddle/operators/math/math_function.cc +++ b/paddle/operators/math/math_function.cc @@ -130,6 +130,87 @@ void matmul( matrix_b.data(), beta, matrix_out->data()); } +#ifdef PADDLE_USE_MKLML +// Use cblas_{s,d}gemm_batched if available: Run with 1 group of size batchSize. +template <> +void batched_gemm( + const platform::DeviceContext& context, const CBLAS_TRANSPOSE transA, + const CBLAS_TRANSPOSE transB, const int M, const int N, const int K, + const float alpha, const float* A, const float* B, const float beta, + float* C, const int batchCount, const int strideA, const int strideB) { + int lda = (transA == CblasNoTrans) ? K : M; + int ldb = (transB == CblasNoTrans) ? N : K; + int ldc = N; + auto a_array = std::vector(batchCount); + auto b_array = std::vector(batchCount); + auto c_array = std::vector(batchCount); + for (int k = 0; k < batchCount; ++k) { + a_array[k] = &A[k * strideA]; + b_array[k] = &B[k * strideB]; + c_array[k] = &C[k * M * N]; + } + cblas_sgemm_batch(CblasRowMajor, &transA, &transB, &M, &N, &K, &alpha, + a_array.data(), &lda, b_array.data(), &ldb, &beta, + c_array.data(), &ldc, 1 /* group_count */, &batchCount); +} + +template <> +void batched_gemm( + const platform::DeviceContext& context, const CBLAS_TRANSPOSE transA, + const CBLAS_TRANSPOSE transB, const int M, const int N, const int K, + const double alpha, const double* A, const double* B, const double beta, + double* C, const int batchCount, const int strideA, const int strideB) { + int lda = (transA == CblasNoTrans) ? K : M; + int ldb = (transB == CblasNoTrans) ? N : K; + int ldc = N; + auto a_array = std::vector(batchCount); + auto b_array = std::vector(batchCount); + auto c_array = std::vector(batchCount); + for (int k = 0; k < batchCount; ++k) { + a_array[k] = &A[k * strideA]; + b_array[k] = &B[k * strideB]; + c_array[k] = &C[k * M * N]; + } + cblas_dgemm_batch(CblasRowMajor, &transA, &transB, &M, &N, &K, &alpha, + a_array.data(), &lda, b_array.data(), &ldb, &beta, + c_array.data(), &ldc, 1 /* group_count */, &batchCount); +} +#else +// The below is a naive but correct serial implementation that just loops +// over the batch dimension. This is a fallback for when the batched gemm +// functions of Intel MKL are not available. In the future, this computation +// should be parallelized. +template <> +void batched_gemm( + const platform::DeviceContext& context, const CBLAS_TRANSPOSE transA, + const CBLAS_TRANSPOSE transB, const int M, const int N, const int K, + const float alpha, const float* A, const float* B, const float beta, + float* C, const int batchCount, const int strideA, const int strideB) { + for (int k = 0; k < batchCount; ++k) { + const float* Ak = &A[k * strideA]; + const float* Bk = &B[k * strideB]; + float* Ck = &C[k * M * N]; + gemm(context, transA, transB, M, N, K, alpha, Ak, + Bk, beta, Ck); + } +} + +template <> +void batched_gemm( + const platform::DeviceContext& context, const CBLAS_TRANSPOSE transA, + const CBLAS_TRANSPOSE transB, const int M, const int N, const int K, + const double alpha, const double* A, const double* B, const double beta, + double* C, const int batchCount, const int strideA, const int strideB) { + for (int k = 0; k < batchCount; ++k) { + const double* Ak = &A[k * strideA]; + const double* Bk = &B[k * strideB]; + double* Ck = &C[k * M * N]; + gemm(context, transA, transB, M, N, K, alpha, + Ak, Bk, beta, Ck); + } +} +#endif + template struct SetConstant; } // namespace math diff --git a/paddle/operators/math/math_function.cu b/paddle/operators/math/math_function.cu index 7fbc03acf2..5583683c6e 100644 --- a/paddle/operators/math/math_function.cu +++ b/paddle/operators/math/math_function.cu @@ -155,6 +155,54 @@ void matmul( matrix_b.data(), beta, matrix_out->data()); } +template <> +void batched_gemm( + const platform::DeviceContext& context, const CBLAS_TRANSPOSE transA, + const CBLAS_TRANSPOSE transB, const int M, const int N, const int K, + const float alpha, const float* A, const float* B, const float beta, + float* C, const int batchCount, const int strideA, const int strideB) { + // Note that cublas follows fortran order, so the order is different from + // the cblas convention. + int lda = (transA == CblasNoTrans) ? K : M; + int ldb = (transB == CblasNoTrans) ? N : K; + int ldc = N; + cublasOperation_t cuTransA = + (transA == CblasNoTrans) ? CUBLAS_OP_N : CUBLAS_OP_T; + cublasOperation_t cuTransB = + (transB == CblasNoTrans) ? CUBLAS_OP_N : CUBLAS_OP_T; + const int strideC = M * N; + + PADDLE_ENFORCE(platform::dynload::cublasSgemmStridedBatched( + reinterpret_cast(context) + .cublas_handle(), + cuTransB, cuTransA, N, M, K, &alpha, B, ldb, strideB, A, lda, strideA, + &beta, C, ldc, strideC, batchCount)); +} + +template <> +void batched_gemm( + const platform::DeviceContext& context, const CBLAS_TRANSPOSE transA, + const CBLAS_TRANSPOSE transB, const int M, const int N, const int K, + const double alpha, const double* A, const double* B, const double beta, + double* C, const int batchCount, const int strideA, const int strideB) { + // Note that cublas follows fortran order, so the order is different from + // the cblas convention. + int lda = (transA == CblasNoTrans) ? K : M; + int ldb = (transB == CblasNoTrans) ? N : K; + int ldc = N; + cublasOperation_t cuTransA = + (transA == CblasNoTrans) ? CUBLAS_OP_N : CUBLAS_OP_T; + cublasOperation_t cuTransB = + (transB == CblasNoTrans) ? CUBLAS_OP_N : CUBLAS_OP_T; + const int strideC = M * N; + + PADDLE_ENFORCE(platform::dynload::cublasDgemmStridedBatched( + reinterpret_cast(context) + .cublas_handle(), + cuTransB, cuTransA, N, M, K, &alpha, B, ldb, strideB, A, lda, strideA, + &beta, C, ldc, strideC, batchCount)); +} + template struct SetConstant; } // namespace math diff --git a/paddle/operators/math/math_function.h b/paddle/operators/math/math_function.h index 6f92d83aab..9777ebfd15 100644 --- a/paddle/operators/math/math_function.h +++ b/paddle/operators/math/math_function.h @@ -63,7 +63,7 @@ namespace math { // Support continuous memory now // If transA = N, and transB = N -// Then matrixA: M * K, matrixB: K * N matrixC : M * N +// Then matrixA: M * K, matrixB: K * N, matrixC : M * N // For more detailed info, please refer to // http://www.netlib.org/lapack/explore-html/d4/de2/sgemm_8f.html template @@ -85,6 +85,14 @@ void matmul(const platform::DeviceContext& context, const framework::Tensor& matrix_b, bool trans_b, T alpha, framework::Tensor* matrix_out, T beta); +// Batched gemm +template +void batched_gemm(const platform::DeviceContext& context, + const CBLAS_TRANSPOSE transA, const CBLAS_TRANSPOSE transB, + const int M, const int N, const int K, const T alpha, + const T* A, const T* B, const T beta, T* C, + const int batchCount, const int strideA, const int strideB); + template struct SetConstant { void operator()(const platform::DeviceContext& context, diff --git a/paddle/operators/math/matmul.h b/paddle/operators/math/matmul.h new file mode 100644 index 0000000000..6ba9a0ba9a --- /dev/null +++ b/paddle/operators/math/matmul.h @@ -0,0 +1,124 @@ +/* Copyright (c) 2017 PaddlePaddle Authors. All Rights Reserve. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. */ + +#pragma once +#include "paddle/operators/math/math_function.h" + +namespace paddle { +namespace operators { +namespace math { + +// Implements the logic of numpy matmul: +// https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.matmul.html +// +// but allowing also for a, b to be transposed +// +// Both a & b can be 1- to 3-dimensional. Higher rank tensors are not supported +// yet. +template +class MatMulFunctor { + public: + void operator()(const platform::DeviceContext& context, + const framework::Tensor& a, bool trans_a, + const framework::Tensor& b, bool trans_b, T alpha, + framework::Tensor* out, T beta) { + auto dim_a = a.dims(); + auto dim_b = b.dims(); + + PADDLE_ENFORCE(a.place() == b.place() && b.place() == out->place(), + "Tensors must all be in the same place."); + PADDLE_ENFORCE_GE(dim_a.size(), 1, + "Input tensor a must be at least 1-dimensional."); + PADDLE_ENFORCE_GE(dim_b.size(), 1, + "Input tensor b must be at least 1-dimensional."); + PADDLE_ENFORCE_LE(dim_a.size(), 3, + "Input tensor a must be at most 3-dimensional."); + PADDLE_ENFORCE_LE(dim_b.size(), 3, + "Input tensor b must be at most 3-dimensional."); + + int M = 0, N = 0, kA = 0, kB = 0, batchCountA = 0, batchCountB = 0, + strideA = 0, strideB = 0; + + switch (dim_a.size()) { + case 1: + // similar to np.matmul: + // prepend dimension 1 (no transpose) or append dimension 1 (transpose) + M = trans_a ? dim_a[0] : 1; + kA = trans_a ? 1 : dim_a[0]; + break; + case 2: + M = trans_a ? dim_a[1] : dim_a[0]; + kA = trans_a ? dim_a[0] : dim_a[1]; + break; + case 3: + batchCountA = dim_a[0]; + M = trans_a ? dim_a[2] : dim_a[1]; + kA = trans_a ? dim_a[1] : dim_a[2]; + strideA = M * kA; + break; + default: + assert(false); + } + + switch (dim_b.size()) { + case 1: + // similar to np.matmul: + // append dimension 1 (no transpose) or prepend dimension 1 (transpose) + kB = trans_b ? 1 : dim_b[0]; + N = trans_b ? dim_b[0] : 1; + break; + case 2: + kB = trans_b ? dim_b[1] : dim_b[0]; + N = trans_b ? dim_b[0] : dim_b[1]; + break; + case 3: + batchCountB = dim_b[0]; + kB = trans_b ? dim_b[2] : dim_b[1]; + N = trans_b ? dim_b[1] : dim_b[2]; + strideB = kB * N; + break; + default: + assert(false); + } + + PADDLE_ENFORCE_EQ( + kA, kB, + "First matrix's width must be equal with second matrix's height."); + if (batchCountA && batchCountB) { + PADDLE_ENFORCE_EQ( + batchCountA, batchCountB, + "When input tensors a and b are both batched, they must have the " + "same batch dimension."); + } + int batchCount = std::max(batchCountA, batchCountB); + + CBLAS_TRANSPOSE transA = (trans_a == false) ? CblasNoTrans : CblasTrans; + CBLAS_TRANSPOSE transB = (trans_b == false) ? CblasNoTrans : CblasTrans; + + if (!batchCount) { + // regular matrix multiplication + gemm(context, transA, transB, M, N, kA, alpha, a.data(), + b.data(), beta, out->data()); + } else { + // batched matrix multiplication + batched_gemm(context, transA, transB, M, N, kA, alpha, + a.data(), b.data(), beta, out->data(), + batchCount, strideA, strideB); + } + } +}; + +} // namespace math +} // namespace operators +} // namespace paddle diff --git a/paddle/operators/matmul_op.cc b/paddle/operators/matmul_op.cc new file mode 100644 index 0000000000..5ecbee3b41 --- /dev/null +++ b/paddle/operators/matmul_op.cc @@ -0,0 +1,208 @@ +/* Copyright (c) 2017 PaddlePaddle Authors. All Rights Reserve. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. */ + +#include "paddle/operators/matmul_op.h" + +namespace paddle { +namespace operators { + +using framework::Tensor; + +class MatMulOp : public framework::OperatorWithKernel { + public: + using framework::OperatorWithKernel::OperatorWithKernel; + + protected: + void InferShape(framework::InferShapeContext* context) const override { + PADDLE_ENFORCE(context->HasInput("X"), + "Input(X) of MatMulOp should not be null."); + PADDLE_ENFORCE(context->HasInput("Y"), + "Input(Y) of MatMulOp should not be null."); + PADDLE_ENFORCE(context->HasOutput("Out"), + "Output(Out) of MatMulOp should not be null."); + + auto dim_x = context->GetInputDim("X"); + auto dim_y = context->GetInputDim("Y"); + bool transpose_x = context->Attrs().Get("transpose_X"); + bool transpose_y = context->Attrs().Get("transpose_Y"); + + PADDLE_ENFORCE_GE(dim_x.size(), 1, + "Input tensor X must be at least 1-dimensional."); + PADDLE_ENFORCE_GE(dim_y.size(), 1, + "Input tensor Y must be at least 1-dimensional."); + PADDLE_ENFORCE_LE(dim_x.size(), 3, + "Input tensor X must be at most 3-dimensional."); + PADDLE_ENFORCE_LE(dim_y.size(), 3, + "Input tensor Y must be at most 3-dimensional."); + + int M = 0, N = 0, KX = 0, KY = 0, batchCountX = 0, batchCountY = 0; + bool remove_initial_dim = false, remove_final_dim = false; + + switch (dim_x.size()) { + case 1: + if (transpose_x) { + M = dim_x[0]; + KX = 1; + } else { + M = 1; + KX = dim_x[0]; + remove_initial_dim = true; + } + break; + case 2: + M = transpose_x ? dim_x[1] : dim_x[0]; + KX = transpose_x ? dim_x[0] : dim_x[1]; + break; + case 3: + batchCountX = dim_x[0]; + M = transpose_x ? dim_x[2] : dim_x[1]; + KX = transpose_x ? dim_x[1] : dim_x[2]; + break; + default: + assert(false); + } + + switch (dim_y.size()) { + case 1: + if (transpose_y) { + N = dim_y[0]; + KY = 1; + } else { + N = 1; + KY = dim_y[0]; + remove_final_dim = true; + } + break; + case 2: + KY = transpose_y ? dim_y[1] : dim_y[0]; + N = transpose_y ? dim_y[0] : dim_y[1]; + break; + case 3: + batchCountY = dim_y[0]; + KY = transpose_y ? dim_y[2] : dim_y[1]; + N = transpose_y ? dim_y[1] : dim_y[2]; + break; + default: + assert(false); + } + + PADDLE_ENFORCE_EQ( + KX, KY, + "First matrix's width must be equal with second matrix's height."); + if (batchCountX && batchCountY) { + PADDLE_ENFORCE_EQ( + batchCountX, batchCountY, + "When Input(X) and Input(Y) are both three dimensional, they " + "must have the same batch dimension."); + } + int batchCount = std::max(batchCountX, batchCountY); + + std::vector dim_out; + if (batchCount) { + dim_out.push_back(batchCount); + } + if (!remove_initial_dim) { + dim_out.push_back(M); + } + if (!remove_final_dim) { + dim_out.push_back(N); + } + if (dim_out.size() == 0) { + // We don't support 0-dimensional Tensors (scalars), so instead + // treat the output as a Tensor of shape (1, ) in this case. + dim_out.push_back(1); + } + context->SetOutputDim("Out", framework::make_ddim(dim_out)); + context->ShareLoD("X", /*->*/ "Out"); + } +}; + +class MatMulOpMaker : public framework::OpProtoAndCheckerMaker { + public: + MatMulOpMaker(framework::OpProto* proto, framework::OpAttrChecker* op_checker) + : OpProtoAndCheckerMaker(proto, op_checker) { + AddInput("X", "The first input of MatMul op"); + AddInput("Y", "The second input of MatMul op"); + AddOutput("Out", "The output of MatMul op"); + AddAttr("transpose_X", + R"DOC(If true, use the transpose of `X`. + )DOC") + .SetDefault(false); + AddAttr("transpose_Y", + R"DOC(If true, use the transpose of `Y`. + )DOC") + .SetDefault(false); + AddComment(R"DOC( +The MatMul operator is used to perform (batched) matrix multiplication +over the last two dimensions of the input tensors `X` and `Y`. + +If a transpose flag is specified, the last two dimensions of the +tensor are transposed. If the tensor is rank-1 of shape [D], then +for `X` it is treated as [1, D] in nontransposed form and as [D, 1] +in transposed form, whereas for `Y` it is the opposite: It is treated +as [D, 1] in nontransposed form and as [1, D] in transposed form. + +Examples without transpose: +- X: [K], Y: [K] => Out: [1] +- X: [K], Y: [K, N] => Out: [N] +- X: [B, M, K], Y: [K] => Out: [B, M] +- X: [M, K], Y: [B, K, N] => Out: [B, M, N] +- X: [B, M, K], Y: [B, K, N] => Out: [B, M, N] + +The behavior is designed to be similar to the `numpy.matmul` function. +The differences are: +- Currently only rank 1 to rank 3 input tensors are supported. +- We add `transpose_X` and `transpose_Y` flags. + +Both the input `X` and `Y` can carry the LoD (Level of Details) information, +or not. But the output only shares the LoD with input `X`. +)DOC"); + } +}; + +class MatMulOpGrad : public framework::OperatorWithKernel { + public: + using framework::OperatorWithKernel::OperatorWithKernel; + + protected: + void InferShape(framework::InferShapeContext* context) const override { + PADDLE_ENFORCE(context->HasInput("X"), "Input(X) should not be null"); + PADDLE_ENFORCE(context->HasInput("Y"), "Input(Y) should not be null"); + PADDLE_ENFORCE(context->HasInput(framework::GradVarName("Out")), + "Input(Out@GRAD) should not be null"); + auto x_dims = context->GetInputDim("X"); + auto y_dims = context->GetInputDim("Y"); + + auto x_grad_name = framework::GradVarName("X"); + auto y_grad_name = framework::GradVarName("Y"); + + if (context->HasOutput(x_grad_name)) { + context->SetOutputDim(x_grad_name, x_dims); + } + if (context->HasOutput(y_grad_name)) { + context->SetOutputDim(y_grad_name, y_dims); + } + } +}; + +} // namespace operators +} // namespace paddle + +namespace ops = paddle::operators; +REGISTER_OP(matmul, ops::MatMulOp, ops::MatMulOpMaker, matmul_grad, + ops::MatMulOpGrad); +REGISTER_OP_CPU_KERNEL(matmul, + ops::MatMulKernel); +REGISTER_OP_CPU_KERNEL( + matmul_grad, ops::MatMulGradKernel); diff --git a/paddle/operators/matmul_op.cu b/paddle/operators/matmul_op.cu new file mode 100644 index 0000000000..b7e66382f0 --- /dev/null +++ b/paddle/operators/matmul_op.cu @@ -0,0 +1,21 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include "paddle/operators/matmul_op.h" + +namespace ops = paddle::operators; +REGISTER_OP_GPU_KERNEL(matmul, + ops::MatMulKernel); +REGISTER_OP_GPU_KERNEL( + matmul_grad, ops::MatMulGradKernel); diff --git a/paddle/operators/matmul_op.h b/paddle/operators/matmul_op.h new file mode 100644 index 0000000000..8ae54e1eec --- /dev/null +++ b/paddle/operators/matmul_op.h @@ -0,0 +1,228 @@ +/* Copyright (c) 2017 PaddlePaddle Authors. All Rights Reserve. + + Licensed under the Apache License, Version 2.0 (the "License"); + You may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#pragma once + +#include "paddle/framework/op_registry.h" +#include "paddle/operators/math/matmul.h" +#include "paddle/operators/transpose_op.h" + +namespace paddle { +namespace operators { +namespace matmul_detail { + +using Tensor = framework::Tensor; +using DDim = framework::DDim; +using framework::make_ddim; +using framework::vectorize; + +template +class MatMulKernel : public framework::OpKernel { + public: + void Compute(const framework::ExecutionContext& context) const override { + const Tensor& x = *context.Input("X"); + const Tensor& y = *context.Input("Y"); + Tensor* out = context.Output("Out"); + out->mutable_data(context.GetPlace()); + bool transpose_x = context.Attr("transpose_X"); + bool transpose_y = context.Attr("transpose_Y"); + + math::MatMulFunctor()(context.device_context(), x, transpose_x, y, + transpose_y, T(1), out, T(0)); + } +}; + +template +inline Tensor Reshape(const Tensor& input, const DDim& dims) { + Tensor output; + output.ShareDataWith(input); + output.Resize(dims); + return output; +} + +// Reshape a rank-3 tensor from P x M x N to (P * M) x N. +// Identity op if the tensor is not of rank 3. +template +Tensor CombineBatchAndM(const Tensor& input) { + Tensor output; + output.ShareDataWith(input); + auto in_dims = input.dims(); + if (in_dims.size() == 3) { + std::vector out_dims = {in_dims[0] * in_dims[1], in_dims[2]}; + output.Resize(make_ddim(out_dims)); + } + return output; +} + +// Reshape a rank-3 tensor from P x M x N to M x (P * N). +// (Warning: This requires transposing data and writes into new memory.) +// Identity op if the tensor is not of rank 3. +template +Tensor CombineBatchAndN(const framework::ExecutionContext& context, + const Tensor& input) { + Tensor output; + auto in_dims = input.dims(); + if (in_dims.size() == 3) { + output.Resize(in_dims); + output.mutable_data(context.GetPlace()); + EigenTranspose(context, input, output, {1, 0, 2}); + std::vector out_dims = {in_dims[1], in_dims[0] * in_dims[2]}; + output.Resize(make_ddim(out_dims)); + } else { + output.ShareDataWith(input); + } + return output; +} + +// Using dimensional constraints on matrix multiplication, it is +// straight-forward to check the following table for when X and Y +// are both matrices. +// +// transpose_X | False | True | False | True +// transpose_Y | False | False | True | True +// -----------+----------+----------+----------+----------- +// dX = | dOut Y^T | Y dOut^T | dOut Y | Y^T dOut^T +// dY = | X^T dOut | X dOut | dOut^T X | dOut^T X^T +// +// When X is a vector of size K, we treat it instead as a matrix of shape +// (1, K). Similarly, when Y is a vector of size K, we treat it instead as +// a matrix of shape (K, 1). +// +// When X and Y are both 3-dimensional tensors, then the first dimension +// the batch dimension can be ignored and the exact same formulas apply +// as for two matrices. +// +// Finally, when, e.g., X is a 3-dimensional tensor but Y is a matrix, we end +// up with formulas like +// +// dY_{ij} = \sum_{p, m} X_{pmi} dOut_{pmj} +// +// To handle this sort of scenario, we reshape X : P x M x K, dOut: P x M x N +// to X: (P * M) x K, dOut: (P * M) x N. +template +class MatMulGradKernel : public framework::OpKernel { + public: + void Compute(const framework::ExecutionContext& context) const override { + const Tensor& x = *context.Input("X"); + const Tensor& y = *context.Input("Y"); + const Tensor& dout = *context.Input(framework::GradVarName("Out")); + Tensor* dx = context.Output(framework::GradVarName("X")); + Tensor* dy = context.Output(framework::GradVarName("Y")); + bool transpose_x = context.Attr("transpose_X"); + bool transpose_y = context.Attr("transpose_Y"); + + std::vector x_dims = vectorize(x.dims()); + std::vector y_dims = vectorize(y.dims()); + + // If X is a vector, reshape it to a matrix. + if (x_dims.size() == 1) { + x_dims.insert(x_dims.begin(), 1); + } + + // If Y is a vector, reshape it to a matrix. + if (y_dims.size() == 1) { + y_dims.push_back(1); + } + + // Fix the dOut dimensions. + int M = 0, N = 0, batchCountX = 0, batchCountY = 0; + + switch (x_dims.size()) { + case 2: + M = transpose_x ? x_dims[1] : x_dims[0]; + break; + case 3: + batchCountX = x_dims[0]; + M = transpose_x ? x_dims[2] : x_dims[1]; + break; + default: + assert(false); + } + + switch (y_dims.size()) { + case 2: + N = transpose_y ? y_dims[0] : y_dims[1]; + break; + case 3: + batchCountY = y_dims[0]; + N = transpose_y ? y_dims[1] : y_dims[2]; + break; + default: + assert(false); + } + if (batchCountX && batchCountY) { + PADDLE_ENFORCE_EQ( + batchCountX, batchCountY, + "When Input(X) and Input(Y) are both three dimensional, they " + "must have the same batch dimension."); + } + int batchCount = std::max(batchCountX, batchCountY); + std::vector dout_dims = {M, N}; + if (batchCount) { + dout_dims.insert(dout_dims.begin(), batchCount); + } + Tensor X = Reshape(x, make_ddim(x_dims)); + Tensor Y = Reshape(y, make_ddim(y_dims)); + Tensor dOut = Reshape(dout, make_ddim(dout_dims)); + + if (dx) { + dx->mutable_data(context.GetPlace()); + const Tensor& dOut_for_dX = + (x_dims.size() == 2 && y_dims.size() == 3) + ? CombineBatchAndN(context, dOut) + : dOut; + if (x_dims.size() == 2 && y_dims.size() == 3) { + Y = transpose_y ? CombineBatchAndM(Y) + : CombineBatchAndN(context, Y); + } + if (transpose_x) { + math::MatMulFunctor()(context.device_context(), Y, + transpose_y, dOut_for_dX, transpose_x, + T(1), dx, T(0)); + } else { + math::MatMulFunctor()(context.device_context(), dOut_for_dX, + transpose_x, Y, !transpose_y, T(1), dx, + T(0)); + } + } + + if (dy) { + dy->mutable_data(context.GetPlace()); + const Tensor& dOut_for_dY = (y_dims.size() == 2 && x_dims.size() == 3) + ? CombineBatchAndM(dOut) + : dOut; + if (y_dims.size() == 2 && x_dims.size() == 3) { + X = transpose_x ? CombineBatchAndN(context, X) + : CombineBatchAndM(X); + dOut = CombineBatchAndM(dOut); + } + if (transpose_y) { + math::MatMulFunctor()(context.device_context(), dOut_for_dY, + transpose_y, X, transpose_x, T(1), dy, + T(0)); + } else { + math::MatMulFunctor()(context.device_context(), X, + !transpose_x, dOut_for_dY, transpose_y, + T(1), dy, T(0)); + } + } + } +}; +} // namespace matmul_detail + +using matmul_detail::MatMulKernel; +using matmul_detail::MatMulGradKernel; + +} // namespace operators +} // namespace paddle diff --git a/paddle/platform/dynload/cublas.h b/paddle/platform/dynload/cublas.h index 9d8343c0b5..6b64539b0a 100644 --- a/paddle/platform/dynload/cublas.h +++ b/paddle/platform/dynload/cublas.h @@ -77,6 +77,10 @@ extern void *cublas_dso_handle; __macro(cublasDgemmBatched); \ __macro(cublasCgemmBatched); \ __macro(cublasZgemmBatched); \ + __macro(cublasSgemmStridedBatched); \ + __macro(cublasDgemmStridedBatched); \ + __macro(cublasCgemmStridedBatched); \ + __macro(cublasZgemmStridedBatched); \ __macro(cublasSgetrfBatched); \ __macro(cublasSgetriBatched); \ __macro(cublasDgetrfBatched); \ diff --git a/python/paddle/v2/framework/tests/test_matmul_op.py b/python/paddle/v2/framework/tests/test_matmul_op.py new file mode 100644 index 0000000000..d51572c8ab --- /dev/null +++ b/python/paddle/v2/framework/tests/test_matmul_op.py @@ -0,0 +1,119 @@ +import unittest +import numpy as np +from op_test import OpTest + + +def generate_compatible_shapes(dim_X, dim_Y, transpose_X, transpose_Y): + BATCH_SIZE = 2 + M = 3 + N = 4 + K = 5 + if (dim_X == 1 and transpose_X) or (dim_Y == 1 and transpose_Y): + K = 1 + if dim_X == 1: + if transpose_X: + shape_X = [M] + else: + shape_X = [K] + if dim_Y == 1: + if transpose_Y: + shape_Y = [N] + else: + shape_Y = [K] + if dim_X >= 2: + if transpose_X: + shape_X = [K, M] + else: + shape_X = [M, K] + if dim_X == 3: + shape_X = [BATCH_SIZE] + shape_X + if dim_Y >= 2: + if transpose_Y: + shape_Y = [N, K] + else: + shape_Y = [K, N] + if dim_Y == 3: + shape_Y = [BATCH_SIZE] + shape_Y + return shape_X, shape_Y + + +def reference_matmul(X, Y, transpose_X=False, transpose_Y=False): + """Reference forward implementation using np.matmul.""" + # np.matmul does not support the transpose flags, so we manually + # transpose X and Y appropriately. + if transpose_X: + if X.ndim == 1: + X = X.reshape((X.size, 1)) + elif X.ndim == 2: + X = X.T + elif X.ndim == 3: + X = np.transpose(X, (0, 2, 1)) + else: + raise ValueError('X must have between 1 and 3 dimensions') + if transpose_Y: + if Y.ndim == 1: + Y = Y.reshape((1, Y.size)) + elif Y.ndim == 2: + Y = Y.T + elif Y.ndim == 3: + Y = np.transpose(Y, (0, 2, 1)) + else: + raise ValueError('Y must have between 1 and 3 dimensions') + Out = np.matmul(X, Y) + if not Out.shape: + # We do not support 0-dimensional Tensors (scalars). So where + # np.matmul outputs a scalar, we must convert to a Tensor of + # shape (1, ) instead. + # Everywhere else, we are compatible with np.matmul. + Out = np.array([Out], dtype="float32") + return Out + + +class Generator(object): + def setUp(self): + self.op_type = "matmul" + X = np.random.random(self.shape_X).astype("float32") + Y = np.random.random(self.shape_Y).astype("float32") + Out = reference_matmul(X, Y, self.transpose_X, self.transpose_Y) + self.inputs = {'X': X, 'Y': Y} + self.attrs = { + 'transpose_X': self.transpose_X, + 'transpose_Y': self.transpose_Y + } + self.outputs = {'Out': Out} + + def test_check_output(self): + self.check_output(atol=1e-2) + + def test_check_grad_normal(self): + self.check_grad(['X', 'Y'], 'Out', max_relative_error=0.5) + + def test_check_grad_ignore_x(self): + self.check_grad( + ['Y'], 'Out', max_relative_error=0.5, no_grad_set=set("X")) + + def test_check_grad_ignore_y(self): + self.check_grad( + ['X'], 'Out', max_relative_error=0.5, no_grad_set=set('Y')) + + +# Generate test cases for all possibilities +for dim_X in [1, 2, 3]: + for dim_Y in [1, 2, 3]: + for transpose_X in [False, True]: + for transpose_Y in [False, True]: + test_name = ( + 'TestMatMulOp_dimX_{}_dim_Y_{}_transX_{}_transY_{}'.format( + dim_X, dim_Y, transpose_X, transpose_Y)) + shape_X, shape_Y = generate_compatible_shapes( + dim_X, dim_Y, transpose_X, transpose_Y) + test_class = type(test_name, (Generator, OpTest), { + 'shape_X': shape_X, + 'shape_Y': shape_Y, + 'transpose_X': transpose_X, + 'transpose_Y': transpose_Y, + }) + globals()[test_name] = test_class + +if __name__ == "__main__": + unittest.main() From 521514da361059dcad0b918f27081fdedccf6720 Mon Sep 17 00:00:00 2001 From: QI JUN Date: Tue, 17 Oct 2017 20:01:23 -0700 Subject: [PATCH 53/63] fix conflict (#4883) --- paddle/operators/momentum_op.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/paddle/operators/momentum_op.cc b/paddle/operators/momentum_op.cc index efa0b59992..9be4d15a43 100644 --- a/paddle/operators/momentum_op.cc +++ b/paddle/operators/momentum_op.cc @@ -22,7 +22,7 @@ class MomentumOp : public framework::OperatorWithKernel { using framework::OperatorWithKernel::OperatorWithKernel; protected: - void InferShape(framework::InferShapeContextBase *ctx) const override { + void InferShape(framework::InferShapeContext *ctx) const override { PADDLE_ENFORCE(ctx->HasInput("Param"), "Input(param) of Momentum should not be null."); PADDLE_ENFORCE(ctx->HasInput("Grad"), From 8938a9b0accdf577e2c8ec103d9556227b36c875 Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Tue, 17 Oct 2017 20:18:03 -0700 Subject: [PATCH 54/63] Correct implement BlockDesc destructor (#4882) --- paddle/framework/block_desc.cc | 20 ++++++++++++++++++-- paddle/framework/block_desc.h | 9 +++++++++ 2 files changed, 27 insertions(+), 2 deletions(-) diff --git a/paddle/framework/block_desc.cc b/paddle/framework/block_desc.cc index 47b75228cd..ba970254e5 100644 --- a/paddle/framework/block_desc.cc +++ b/paddle/framework/block_desc.cc @@ -72,13 +72,13 @@ std::vector BlockDescBind::AllOps() const { void BlockDescBind::Flush() { if (need_update_) { auto &op_field = *this->desc_->mutable_ops(); - op_field.Clear(); + this->ClearPBOps(); op_field.Reserve(static_cast(ops_.size())); for (auto &op_desc : ops_) { op_field.AddAllocated(op_desc->Proto()); } auto &var_field = *this->desc_->mutable_vars(); - var_field.Clear(); + this->ClearPBVars(); var_field.Reserve(static_cast(vars_.size())); for (auto &var_desc : vars_) { var_field.AddAllocated(var_desc.second->Proto()); @@ -99,5 +99,21 @@ BlockDesc *BlockDescBind::Proto() { return desc_; } +void BlockDescBind::ClearPBOps() { + auto ops = this->desc_->mutable_ops(); + while (!ops->empty()) { + // we do not own the OpDesc, so release the ownership. + ops->ReleaseLast(); + } +} + +void BlockDescBind::ClearPBVars() { + auto vars = this->desc_->mutable_vars(); + while (!vars->empty()) { + // we do not own the VarDesc, so release the ownership. + vars->ReleaseLast(); + } +} + } // namespace framework } // namespace paddle diff --git a/paddle/framework/block_desc.h b/paddle/framework/block_desc.h index 9fb88f9632..dd7b1228be 100644 --- a/paddle/framework/block_desc.h +++ b/paddle/framework/block_desc.h @@ -36,6 +36,11 @@ class BlockDescBind { BlockDescBind(ProgramDescBind *prog, BlockDesc *desc) : prog_(prog), desc_(desc), need_update_(false) {} + ~BlockDescBind() { + this->ClearPBVars(); + this->ClearPBOps(); + } + int32_t ID() const { return desc_->idx(); } int32_t Parent() const { return desc_->parent_idx(); } @@ -60,6 +65,10 @@ class BlockDescBind { BlockDesc *Proto(); + private: + void ClearPBOps(); + void ClearPBVars(); + // FIXME(yuyang18): backward will access private data of BlockDesc. // Mark it public temporary. We can fix it later. public: From cd099c72627790104db0feaa7028fd551d5fc5c0 Mon Sep 17 00:00:00 2001 From: Liu Yiqun Date: Wed, 18 Oct 2017 14:02:20 +0800 Subject: [PATCH 55/63] Remove the words related to the support of Android 16. --- .../cross_compiling_for_android_cn.md | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-) diff --git a/doc/howto/cross_compiling/cross_compiling_for_android_cn.md b/doc/howto/cross_compiling/cross_compiling_for_android_cn.md index 0e67c12c57..1fc58c37cc 100644 --- a/doc/howto/cross_compiling/cross_compiling_for_android_cn.md +++ b/doc/howto/cross_compiling/cross_compiling_for_android_cn.md @@ -23,27 +23,17 @@ Android的Docker开发镜像向用户提供两个可配置的参数： | Argument | Optional Values | Default | |-----------------|-------------------------|---------| |`ANDROID_ABI` |`armeabi-v7a, arm64-v8a` | `armeabi-v7a` | -|`ANDROID_API` |`armeabi-v7a(>15), arm64-v8a(>20)` | `21` | +|`ANDROID_API` |`>= 21` | `21` | - 编译`armeabi-v7a`，`Android API 21`的PaddlePaddle库 ```bash $ docker run -it --rm -v $PWD:/paddle -e "ANDROID_ABI=armeabi-v7a" -e "ANDROID_API=21" username/paddle-android:dev ``` -或 -```bash -$ docker run -it --rm -v $PWD:/paddle username/paddle-android:dev -``` - -如需编译Android API低于21的Paddle库，请参考本文档的**准备交叉编译环境**章节。 - 编译`arm64-v8a`，`Android API 21`的PaddlePaddle库 ```bash $ docker run -it --rm -v $PWD:/paddle -e "ANDROID_ABI=arm64-v8a" -e "ANDROID_API=21" username/paddle-android:dev ``` -或 -```bash -$ docker run -it --rm -v $PWD:/paddle -e "ANDROID_ABI=arm64-v8a" username/paddle-android:dev -``` 执行上述`docker run`命令时，容器默认执行[paddle/scripts/docker/build_android.sh](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/scripts/docker/build_android.sh)脚本。该脚本中记录了交叉编译Android版PaddlePaddle库常用的CMake配置，并且会根据`ANDROID_ABI`和`ANDROID_API`自动构建独立工具链、进行编译和安装。由于arm64架构要求Android API不小于21。因此当`ANDROID_ABI=arm64-v8a`，`ANDROID_API<21`时，Docker容器中将默认使用`Android API 21`的编译工具链。用户可以参考下文**配置交叉编译参数**章节，根据个人的需求修改定制Docker容器所执行的脚本。编译安装结束之后，PaddlePaddle的C-API库将被安装到`$PWD/install_android`目录，所依赖的第三方库同时也被安装到`$PWD/install_android/third_party`目录。 @@ -70,8 +60,6 @@ your/path/to/android-ndk-r14b-linux-x86_64/build/tools/make-standalone-toolchain 此命令将在`your/path/to/arm_standalone_toolchain`目录生成一套独立编译工具链，面向架构为32位ARM架构，支持的最小的Android API级别为21，支持编译器`arm-linux-androideabi-gcc (GCC) 4.9`和`clang 3.8`。 -注意：**PaddlePaddle要求使用的编译工具链所支持的Andoid API级别不小于16**。但由于PaddlePaddle所依赖的第三方库`glog`不支持低于21的Android API，所以在编译Android API低于21的Paddle C-API库时，需要将[cmake/external/glog.cmake](https://github.com/PaddlePaddle/Paddle/blob/develop/cmake/external/glog.cmake#L33)中的`GIT_REPOSITORY`临时修改为`https://github.com/Xreki/glog.git`。 - - 构建`arm64-v8a`、 `Android API 21`的独立工具链： ```bash your/path/to/android-ndk-r14b-linux-x86_64/build/tools/make-standalone-toolchain.sh \ @@ -80,7 +68,7 @@ your/path/to/android-ndk-r14b-linux-x86_64/build/tools/make-standalone-toolchain 此命令将在`your/path/to/arm64_standalone_toolchain`目录生成一套独立编译工具链，面向架构为64位ARM64架构，支持的最小Android API级别为21，支持编译器`arm-linux-androideabi-gcc (GCC) 4.9`和`clang 3.8`。 -注意：**arm64架构要求Android API不小于21**。 +注意：**PaddlePaddle要求使用的编译工具链所支持的Android API级别不小于21**。 ### 配置交叉编译参数 From 40f3e0c19421b30e510ad3f55eeb652504179831 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=AD=A6=E6=AF=85?= Date: Wed, 18 Oct 2017 19:09:45 +0800 Subject: [PATCH 56/63] fix_fault_tolerant_dist_lock (#4888) --- go/pserver/client/client.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/go/pserver/client/client.go b/go/pserver/client/client.go index 20d91e7703..e5187ce3df 100644 --- a/go/pserver/client/client.go +++ b/go/pserver/client/client.go @@ -137,7 +137,7 @@ func (c *Client) FinishInitParams() error { return err } } - return nil + return c.sel.Done() } // SendGrads sends gradients to parameter servers for updating From efd009a063d089922098a1c766686fd1c3667043 Mon Sep 17 00:00:00 2001 From: fengjiayi Date: Wed, 18 Oct 2017 12:37:44 -0700 Subject: [PATCH 57/63] implementation of simple conv2d layer (#4868) * Implement FC layer with helper * Update LayerHelper * Add debug string for Python ProtoBuf and Rename `Sync` to `Flush` * Add check of ProtoBuf initialization * Layer wrapper for FC * Fix unittest * Fix CI * Add code generator * AttributeChecker Better error log and speicalize bool Since lots of types can be cast to bool * Complete mlp, fit_a_line * Implementation of simple conv_2d layer * Fix bugs * Remove debug code --- python/paddle/v2/framework/framework.py | 2 +- python/paddle/v2/framework/layer_helper.py | 18 ++++--- python/paddle/v2/framework/layers.py | 48 ++++++++++++++++++- .../paddle/v2/framework/tests/test_layers.py | 12 ++++- 4 files changed, 67 insertions(+), 13 deletions(-) diff --git a/python/paddle/v2/framework/framework.py b/python/paddle/v2/framework/framework.py index 3fb6efe42a..e16bc72447 100644 --- a/python/paddle/v2/framework/framework.py +++ b/python/paddle/v2/framework/framework.py @@ -232,7 +232,7 @@ class Operator(object): if attrs is not None: for attr in proto.attrs: attr_name = attr.name - if not attr_name in attrs: + if (not attr_name in attrs) or (attrs[attr_name] is None): continue if not isinstance(attrs[attr_name], Block): self.desc.set_attr(attr_name, attrs[attr_name]) diff --git a/python/paddle/v2/framework/layer_helper.py b/python/paddle/v2/framework/layer_helper.py index 26d3e04310..6615bdcd3b 100644 --- a/python/paddle/v2/framework/layer_helper.py +++ b/python/paddle/v2/framework/layer_helper.py @@ -66,15 +66,15 @@ class LayerHelper(object): actual = self.kwargs.get('param_attr', None) return actual if actual is not None else default - def bias_attr(self, size, dtype): - bias_attr = self.kwargs.get('bias_attr', False) - if bias_attr is None or bias_attr: + def bias_attr(self, shape, dtype): + bias_attr = self.kwargs.get('bias_attr', None) + if bias_attr is True: bias_attr = { 'name': None, 'init_attr': { 'type': 'fill_constant', 'value': 0.0, - 'shape': [size], + 'shape': shape, 'dataType': dtype } } @@ -127,15 +127,13 @@ class LayerHelper(object): return self.program.global_block().create_var(*args, **kwargs) def append_bias_op(self, input_var): - bias_attr = self.bias_attr( - self.kwargs['size'], dtype=input_var.data_type) + size = list(input_var.shape[1:]) + bias_attr = self.bias_attr(size, dtype=input_var.data_type) if not bias_attr: return input_var + b = self.create_parameter( - attr=bias_attr, - shape=[self.kwargs['size']], - dtype=input_var.data_type, - suffix='b') + attr=bias_attr, shape=size, dtype=input_var.data_type, suffix='b') tmp = self.create_tmp_variable(dtype=input_var.data_type) self.append_op( type='elementwise_add', diff --git a/python/paddle/v2/framework/layers.py b/python/paddle/v2/framework/layers.py index 44b587b116..1821da197e 100644 --- a/python/paddle/v2/framework/layers.py +++ b/python/paddle/v2/framework/layers.py @@ -3,7 +3,7 @@ import paddle.v2.framework.core as core from paddle.v2.framework.framework import OpProtoHolder, Variable import re -__all__ = ['fc_layer', 'data_layer', 'cross_entropy'] +__all__ = ['fc_layer', 'data_layer', 'cross_entropy', 'conv2d_layer'] def fc_layer(input, @@ -24,6 +24,7 @@ def fc_layer(input, for input_var, param_attr in helper.iter_inputs_and_params(): input_shape = input_var.shape param_shape = list(input_shape[num_flatten_dims:]) + [size] + w = helper.create_parameter( attr=param_attr, shape=param_shape, dtype=dtype) tmp = helper.create_tmp_variable(dtype) @@ -111,6 +112,7 @@ def _create_op_func_(op_type): _create_op_func_('mean') +_create_op_func_('pool2d') def cross_entropy(input, label, **kwargs): @@ -141,3 +143,47 @@ def square_error_cost(input, label, **kwargs): outputs={'Y': [square_out]}, attrs={'factor': 2.0}) return square_out + + +def conv2d_layer(input, + num_filters, + name=None, + filter_size=[1, 1], + act=None, + groups=None, + stride=[1, 1], + padding=None, + bias_attr=None, + param_attr=None, + program=None): + helper = LayerHelper('conv2d', **locals()) + dtype = helper.input_dtype() + + num_channels = input.shape[1] + if groups is None: + num_filter_channels = num_channels + else: + if num_channels % groups is not 0: + raise ValueError("num_channels must be divisible by groups.") + num_filter_channels = num_channels / groups + + input_shape = input.shape + filter_shape = [num_filters, num_filter_channels] + filter_size + filter = helper.create_parameter( + attr=helper.param_attr, shape=filter_shape, dtype=dtype) + pre_bias = helper.create_tmp_variable(dtype) + + helper.append_op( + type='conv2d', + inputs={ + 'Input': input, + 'Filter': filter, + }, + outputs={"Output": pre_bias}, + attrs={'strides': stride, + 'paddings': padding, + 'groups': groups}) + + pre_act = helper.append_bias_op(pre_bias) + + return helper.append_activation(pre_act) diff --git a/python/paddle/v2/framework/tests/test_layers.py b/python/paddle/v2/framework/tests/test_layers.py index 1ef2591cca..ce20371cfb 100644 --- a/python/paddle/v2/framework/tests/test_layers.py +++ b/python/paddle/v2/framework/tests/test_layers.py @@ -1,4 +1,4 @@ -from paddle.v2.framework.layers import fc_layer, data_layer, cross_entropy, mean, square_error_cost +from paddle.v2.framework.layers import fc_layer, data_layer, cross_entropy, mean, square_error_cost, conv2d_layer from paddle.v2.framework.framework import Program, g_program import paddle.v2.framework.core as core import unittest @@ -38,6 +38,16 @@ class TestBook(unittest.TestCase): self.assertIsNotNone(avg_cost) print str(program) + def test_simple_conv2d(self): + pd = core.ProgramDesc.__create_program_desc__() + program = Program(desc=pd) + images = data_layer( + name='pixel', shape=[3, 48, 48], data_type='int32', program=program) + conv2d_layer( + input=images, num_filters=3, filter_size=[4, 4], program=program) + + print str(program) + if __name__ == '__main__': unittest.main() From e747623e8639ee43a8dd2b33d04f6110a1182de3 Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Wed, 18 Oct 2017 13:14:41 -0700 Subject: [PATCH 58/63] Change ProgramDesc not a global variable (#4879) * Change ProgramDesc not a global variable * Polish code style * Correct implement BlockDesc destructor * Unify program as parameter name --- paddle/framework/attribute.cc | 18 +++------- paddle/framework/attribute.h | 5 +-- paddle/framework/backward_test.cc | 31 ++++------------- paddle/framework/executor.cc | 3 +- paddle/framework/op_registry.cc | 5 +-- paddle/framework/op_registry.h | 3 +- paddle/framework/op_registry_test.cc | 12 +++---- paddle/framework/operator_test.cc | 6 ++-- paddle/framework/program_desc.cc | 33 +++++-------------- paddle/framework/program_desc.h | 7 ++-- paddle/framework/var_type_inference_test.cc | 4 +-- paddle/operators/dynamic_recurrent_op_test.cc | 2 +- paddle/pybind/protobuf.cc | 16 +-------- paddle/pybind/pybind.cc | 9 ++--- python/paddle/v2/framework/framework.py | 6 ++-- .../v2/framework/tests/test_infer_shape.py | 4 +-- .../paddle/v2/framework/tests/test_layers.py | 6 ++-- .../v2/framework/tests/test_protobuf_descs.py | 18 +++++----- 18 files changed, 62 insertions(+), 126 deletions(-) diff --git a/paddle/framework/attribute.cc b/paddle/framework/attribute.cc index d6a2975aaa..29fe352ca4 100644 --- a/paddle/framework/attribute.cc +++ b/paddle/framework/attribute.cc @@ -19,19 +19,7 @@ limitations under the License. */ namespace paddle { namespace framework { -static ProgramDesc* g_program_desc = nullptr; - -ProgramDesc& GetProgramDesc() { - if (g_program_desc == nullptr) { - g_program_desc = new ProgramDesc(); - auto root_block = g_program_desc->mutable_blocks()->Add(); - root_block->set_idx(0); - root_block->set_parent_idx(-1); - } - return *g_program_desc; -} - -Attribute GetAttrValue(const OpDesc::Attr& attr_desc) { +Attribute GetAttrValue(const OpDesc::Attr& attr_desc, ProgramDesc* program) { switch (attr_desc.type()) { case framework::AttrType::BOOLEAN: { return attr_desc.b(); @@ -74,7 +62,9 @@ Attribute GetAttrValue(const OpDesc::Attr& attr_desc) { return val; } case framework::AttrType::BLOCK: { - return GetProgramDesc().mutable_blocks(attr_desc.block_idx()); + PADDLE_ENFORCE(program != nullptr, + "Need to specify ProgramDesc when get a block attr"); + return program->mutable_blocks(attr_desc.block_idx()); } } PADDLE_ENFORCE(false, "Unknown OpDesc::AttrDesc::type !"); diff --git a/paddle/framework/attribute.h b/paddle/framework/attribute.h index 8a7a949346..9744662b8f 100644 --- a/paddle/framework/attribute.h +++ b/paddle/framework/attribute.h @@ -26,16 +26,13 @@ limitations under the License. */ namespace paddle { namespace framework { - -ProgramDesc& GetProgramDesc(); - template inline AttrType AttrTypeID() { Attribute tmp = T(); return static_cast(tmp.which() - 1); } -Attribute GetAttrValue(const OpDesc::Attr& attr_desc); +Attribute GetAttrValue(const OpDesc::Attr& attr_desc, ProgramDesc* desc); class AttrReader { public: diff --git a/paddle/framework/backward_test.cc b/paddle/framework/backward_test.cc index 0c35a157bc..10301f7e39 100644 --- a/paddle/framework/backward_test.cc +++ b/paddle/framework/backward_test.cc @@ -495,19 +495,8 @@ TEST(Backward, linear_net_intermediate_variable_has_no_grad) { EXPECT_EQ(bwd_net->ops_[2]->Outputs(all).size(), 0UL); } -// =================================== // - -f::ProgramDesc *GetNewProgramDesc() { - auto *program_desc = new f::ProgramDesc(); - auto *root_block = program_desc->add_blocks(); - root_block->set_idx(0); - root_block->set_parent_idx(-1); - return program_desc; -} - TEST(Backward, simple_single_op) { - f::ProgramDesc *program_desc = GetNewProgramDesc(); - f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::ProgramDescBind program; f::BlockDescBind *block = program.Block(0); f::OpDescBind *op = block->AppendOp(); @@ -543,8 +532,7 @@ TEST(Backward, simple_single_op) { } TEST(Backward, default_attribute) { - f::ProgramDesc *program_desc = GetNewProgramDesc(); - f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::ProgramDescBind program; f::BlockDescBind *block = program.Block(0); f::OpDescBind *op = block->AppendOp(); op->SetType("mul"); @@ -570,8 +558,7 @@ TEST(Backward, default_attribute) { } TEST(Backward, simple_mult_op) { - f::ProgramDesc *program_desc = GetNewProgramDesc(); - f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::ProgramDescBind program; f::BlockDescBind *block = program.Block(0); f::OpDescBind *op1 = block->AppendOp(); op1->SetType("rowwise_add"); @@ -654,8 +641,7 @@ TEST(Backward, simple_mult_op) { } TEST(Backward, intermedia_var_no_grad) { - f::ProgramDesc *program_desc = GetNewProgramDesc(); - f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::ProgramDescBind program; f::BlockDescBind *block = program.Block(0); f::OpDescBind *op1 = block->AppendOp(); op1->SetType("rowwise_add"); @@ -725,8 +711,7 @@ TEST(Backward, intermedia_var_no_grad) { } TEST(Backward, var_no_grad) { - f::ProgramDesc *program_desc = GetNewProgramDesc(); - f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::ProgramDescBind program; f::BlockDescBind *block = program.Block(0); f::OpDescBind *op1 = block->AppendOp(); op1->SetType("mult_in_out"); @@ -802,8 +787,7 @@ TEST(Backward, var_no_grad) { } TEST(Backward, shared_var) { - f::ProgramDesc *program_desc = GetNewProgramDesc(); - f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::ProgramDescBind program; f::BlockDescBind *block = program.Block(0); f::OpDescBind *op1 = block->AppendOp(); op1->SetType("rowwise_add"); @@ -893,8 +877,7 @@ TEST(Backward, shared_var) { } TEST(Backward, half_backward) { - f::ProgramDesc *program_desc = GetNewProgramDesc(); - f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::ProgramDescBind program; f::BlockDescBind *block = program.Block(0); auto *op1 = block->AppendOp(); op1->SetType("minus"); diff --git a/paddle/framework/executor.cc b/paddle/framework/executor.cc index b3b85b5865..00caa6e1d5 100644 --- a/paddle/framework/executor.cc +++ b/paddle/framework/executor.cc @@ -75,7 +75,8 @@ void Executor::Run(const ProgramDesc& pdesc, Scope* scope, int block_id) { } for (auto& op_desc : block.ops()) { - auto op = paddle::framework::OpRegistry::CreateOp(op_desc); + auto op = paddle::framework::OpRegistry::CreateOp( + op_desc, const_cast(&pdesc)); op->Run(local_scope, *device); } diff --git a/paddle/framework/op_registry.cc b/paddle/framework/op_registry.cc index 504afbd5db..c2f2438edf 100644 --- a/paddle/framework/op_registry.cc +++ b/paddle/framework/op_registry.cc @@ -43,12 +43,13 @@ static VariableNameMap ConvertOpDescVarsToVarNameMap( return ret_val; } -std::unique_ptr OpRegistry::CreateOp(const OpDesc& op_desc) { +std::unique_ptr OpRegistry::CreateOp(const OpDesc& op_desc, + ProgramDesc* program) { VariableNameMap inputs = ConvertOpDescVarsToVarNameMap(op_desc.inputs()); VariableNameMap outputs = ConvertOpDescVarsToVarNameMap(op_desc.outputs()); AttributeMap attrs; for (auto& attr : op_desc.attrs()) { - attrs[attr.name()] = GetAttrValue(attr); + attrs[attr.name()] = GetAttrValue(attr, program); } return CreateOp(op_desc.type(), inputs, outputs, attrs); diff --git a/paddle/framework/op_registry.h b/paddle/framework/op_registry.h index dfca46b789..d25b4abccb 100644 --- a/paddle/framework/op_registry.h +++ b/paddle/framework/op_registry.h @@ -74,7 +74,8 @@ class OpRegistry { const VariableNameMap& outputs, AttributeMap attrs); - static std::unique_ptr CreateOp(const OpDesc& op_desc); + static std::unique_ptr CreateOp(const OpDesc& op_desc, + ProgramDesc* program); static std::unique_ptr CreateOp(const OpDescBind& op_desc); }; diff --git a/paddle/framework/op_registry_test.cc b/paddle/framework/op_registry_test.cc index b860fe6cac..6289125d7c 100644 --- a/paddle/framework/op_registry_test.cc +++ b/paddle/framework/op_registry_test.cc @@ -74,7 +74,7 @@ TEST(OpRegistry, CreateOp) { attr->set_type(paddle::framework::AttrType::FLOAT); attr->set_f(scale); - auto op = paddle::framework::OpRegistry::CreateOp(op_desc); + auto op = paddle::framework::OpRegistry::CreateOp(op_desc, nullptr); paddle::framework::Scope scope; paddle::platform::CPUDeviceContext dev_ctx; op->Run(scope, dev_ctx); @@ -95,7 +95,7 @@ TEST(OpRegistry, IllegalAttr) { bool caught = false; try { - paddle::framework::OpRegistry::CreateOp(op_desc); + paddle::framework::OpRegistry::CreateOp(op_desc, nullptr); } catch (paddle::platform::EnforceNotMet err) { caught = true; std::string msg = "larger_than check fail"; @@ -115,7 +115,7 @@ TEST(OpRegistry, DefaultValue) { ASSERT_TRUE(op_desc.IsInitialized()); - auto op = paddle::framework::OpRegistry::CreateOp(op_desc); + auto op = paddle::framework::OpRegistry::CreateOp(op_desc, nullptr); paddle::framework::Scope scope; paddle::platform::CPUDeviceContext dev_ctx; op->Run(scope, dev_ctx); @@ -131,7 +131,7 @@ TEST(OpRegistry, CustomChecker) { // attr 'test_attr' is not set bool caught = false; try { - paddle::framework::OpRegistry::CreateOp(op_desc); + paddle::framework::OpRegistry::CreateOp(op_desc, nullptr); } catch (paddle::platform::EnforceNotMet err) { caught = true; std::string msg = "Attribute 'test_attr' is required!"; @@ -149,7 +149,7 @@ TEST(OpRegistry, CustomChecker) { attr->set_i(3); caught = false; try { - paddle::framework::OpRegistry::CreateOp(op_desc); + paddle::framework::OpRegistry::CreateOp(op_desc, nullptr); } catch (paddle::platform::EnforceNotMet err) { caught = true; std::string msg = "'test_attr' must be even!"; @@ -166,7 +166,7 @@ TEST(OpRegistry, CustomChecker) { attr->set_name("test_attr"); attr->set_type(paddle::framework::AttrType::INT); attr->set_i(4); - auto op = paddle::framework::OpRegistry::CreateOp(op_desc); + auto op = paddle::framework::OpRegistry::CreateOp(op_desc, nullptr); paddle::platform::CPUDeviceContext dev_ctx; paddle::framework::Scope scope; op->Run(scope, dev_ctx); diff --git a/paddle/framework/operator_test.cc b/paddle/framework/operator_test.cc index d7890ac8d0..c358f1a2b6 100644 --- a/paddle/framework/operator_test.cc +++ b/paddle/framework/operator_test.cc @@ -83,7 +83,7 @@ TEST(OperatorBase, all) { paddle::platform::CPUDeviceContext device_context; paddle::framework::Scope scope; - auto op = paddle::framework::OpRegistry::CreateOp(op_desc); + auto op = paddle::framework::OpRegistry::CreateOp(op_desc, nullptr); scope.Var("OUT1"); ASSERT_EQ(paddle::framework::op_run_num, 0); op->Run(scope, device_context); @@ -208,7 +208,7 @@ TEST(OpKernel, all) { paddle::platform::CPUDeviceContext cpu_device_context; paddle::framework::Scope scope; - auto op = paddle::framework::OpRegistry::CreateOp(op_desc); + auto op = paddle::framework::OpRegistry::CreateOp(op_desc, nullptr); ASSERT_EQ(paddle::framework::cpu_kernel_run_num, 0); op->Run(scope, cpu_device_context); ASSERT_EQ(paddle::framework::cpu_kernel_run_num, 1); @@ -244,7 +244,7 @@ TEST(OpKernel, multi_inputs) { scope.Var("y0")->GetMutable(); scope.Var("y1")->GetMutable(); - auto op = paddle::framework::OpRegistry::CreateOp(op_desc); + auto op = paddle::framework::OpRegistry::CreateOp(op_desc, nullptr); op->Run(scope, cpu_device_context); } diff --git a/paddle/framework/program_desc.cc b/paddle/framework/program_desc.cc index fcb7292884..df846f115a 100644 --- a/paddle/framework/program_desc.cc +++ b/paddle/framework/program_desc.cc @@ -18,27 +18,10 @@ limitations under the License. */ namespace paddle { namespace framework { -using ProgDescMap = - std::unordered_map>; -static ProgDescMap *g_bind_map = nullptr; - -ProgramDescBind &ProgramDescBind::Instance(ProgramDesc *prog) { - if (g_bind_map == nullptr) { - g_bind_map = new ProgDescMap(); - } - auto &map = *g_bind_map; - auto &ptr = map[prog]; - - if (ptr == nullptr) { - ptr.reset(new ProgramDescBind(prog)); - } - return *ptr; -} - BlockDescBind *ProgramDescBind::AppendBlock(const BlockDescBind &parent) { - auto *b = prog_->add_blocks(); + auto *b = prog_.add_blocks(); b->set_parent_idx(parent.ID()); - b->set_idx(prog_->blocks_size() - 1); + b->set_idx(prog_.blocks_size() - 1); blocks_.emplace_back(new BlockDescBind(this, b)); return blocks_.back().get(); } @@ -47,14 +30,14 @@ ProgramDesc *ProgramDescBind::Proto() { for (auto &block : blocks_) { block->Flush(); } - return prog_; + return &prog_; } -ProgramDescBind::ProgramDescBind(ProgramDesc *prog) { - prog_ = prog; - for (auto &block : *prog->mutable_blocks()) { - blocks_.emplace_back(new BlockDescBind(this, &block)); - } +ProgramDescBind::ProgramDescBind() { + auto *block = prog_.mutable_blocks()->Add(); + block->set_idx(0); + block->set_parent_idx(-1); + blocks_.emplace_back(new BlockDescBind(this, block)); } } // namespace framework } // namespace paddle diff --git a/paddle/framework/program_desc.h b/paddle/framework/program_desc.h index f29b1c54e7..514b62654d 100644 --- a/paddle/framework/program_desc.h +++ b/paddle/framework/program_desc.h @@ -26,7 +26,7 @@ class BlockDescBind; class ProgramDescBind { public: - static ProgramDescBind &Instance(ProgramDesc *prog); + ProgramDescBind(); BlockDescBind *AppendBlock(const BlockDescBind &parent); @@ -37,10 +37,7 @@ class ProgramDescBind { ProgramDesc *Proto(); private: - explicit ProgramDescBind(ProgramDesc *prog); - - // Not owned - ProgramDesc *prog_; + ProgramDesc prog_; std::vector> blocks_; diff --git a/paddle/framework/var_type_inference_test.cc b/paddle/framework/var_type_inference_test.cc index 87399208e9..918de1fd05 100644 --- a/paddle/framework/var_type_inference_test.cc +++ b/paddle/framework/var_type_inference_test.cc @@ -62,7 +62,7 @@ namespace paddle { namespace framework { TEST(InferVarType, sum_op) { - auto &prog = ProgramDescBind::Instance(&GetProgramDesc()); + ProgramDescBind prog; auto *op = prog.Block(0)->AppendOp(); op->SetType("sum"); op->SetInput("X", {"test_a", "test_b", "test_c"}); @@ -83,7 +83,7 @@ TEST(InferVarType, sum_op) { } TEST(InferVarType, sum_op_without_infer_var_type) { - auto &prog = ProgramDescBind::Instance(&GetProgramDesc()); + ProgramDescBind prog; auto *op = prog.Block(0)->AppendOp(); op->SetType("sum_without_infer_var_type"); op->SetInput("X", {"test2_a", "test2_b", "test2_c"}); diff --git a/paddle/operators/dynamic_recurrent_op_test.cc b/paddle/operators/dynamic_recurrent_op_test.cc index 83a5ba36d9..36f405568d 100644 --- a/paddle/operators/dynamic_recurrent_op_test.cc +++ b/paddle/operators/dynamic_recurrent_op_test.cc @@ -51,7 +51,7 @@ class DynamicRecurrentOpTestHelper : public ::testing::Test { CreateGlobalVariables(); auto op_desc = CreateOpDesc(); - op = paddle::framework::OpRegistry::CreateOp(op_desc); + op = paddle::framework::OpRegistry::CreateOp(op_desc, nullptr); dop = dynamic_cast(op.get()); InitCacheManually(); InitStepNet(); diff --git a/paddle/pybind/protobuf.cc b/paddle/pybind/protobuf.cc index 82aae72ba9..fbdd673295 100644 --- a/paddle/pybind/protobuf.cc +++ b/paddle/pybind/protobuf.cc @@ -100,21 +100,7 @@ using namespace paddle::framework; // NOLINT // Bind Methods void BindProgramDesc(py::module &m) { py::class_(m, "ProgramDesc", "") - .def_static("instance", - []() -> ProgramDescBind * { - return &ProgramDescBind::Instance(&GetProgramDesc()); - }, - py::return_value_policy::reference) - .def_static("__create_program_desc__", - []() -> ProgramDescBind * { - // Only used for unit-test - auto *prog_desc = new ProgramDesc; - auto *block = prog_desc->mutable_blocks()->Add(); - block->set_idx(0); - block->set_parent_idx(-1); - return &ProgramDescBind::Instance(prog_desc); - }, - py::return_value_policy::reference) + .def(py::init<>()) .def("append_block", &ProgramDescBind::AppendBlock, py::return_value_policy::reference) .def("append_backward", diff --git a/paddle/pybind/pybind.cc b/paddle/pybind/pybind.cc index fcae92ad99..9eb1bf4a16 100644 --- a/paddle/pybind/pybind.cc +++ b/paddle/pybind/pybind.cc @@ -17,6 +17,7 @@ limitations under the License. */ #include "paddle/framework/backward.h" #include "paddle/framework/executor.h" #include "paddle/framework/feed_fetch_method.h" +#include "paddle/framework/framework.pb.h" #include "paddle/framework/lod_tensor.h" #include "paddle/framework/selected_rows.h" #include "paddle/framework/tensor_array.h" @@ -259,7 +260,7 @@ All parameter, weight, gradient are variables in Paddle. PADDLE_ENFORCE(desc.IsInitialized(), "User OpDesc is not initialized, reason %s", desc.InitializationErrorString()); - return OpRegistry::CreateOp(desc); + return OpRegistry::CreateOp(desc, nullptr); }) .def("backward", [](const OperatorBase &forwardOp, @@ -363,7 +364,7 @@ All parameter, weight, gradient are variables in Paddle. PADDLE_ENFORCE(desc.IsInitialized(), "User OpDesc is not initialized, reason %s", desc.InitializationErrorString()); - auto rnn_op = OpRegistry::CreateOp(desc); + auto rnn_op = OpRegistry::CreateOp(desc, nullptr); return static_cast(rnn_op.release()); }) .def("set_stepnet", [](operators::RecurrentOp &self, @@ -381,7 +382,7 @@ All parameter, weight, gradient are variables in Paddle. PADDLE_ENFORCE(desc.IsInitialized(), "User OpDesc is not initialized, reason %s", desc.InitializationErrorString()); - auto rnn_op = OpRegistry::CreateOp(desc); + auto rnn_op = OpRegistry::CreateOp(desc, nullptr); return static_cast( rnn_op.release()); }) @@ -408,7 +409,7 @@ All parameter, weight, gradient are variables in Paddle. PADDLE_ENFORCE(desc.IsInitialized(), "User OpDesc is not initialized, reason %s", desc.InitializationErrorString()); - auto cond_op = OpRegistry::CreateOp(desc); + auto cond_op = OpRegistry::CreateOp(desc, nullptr); return static_cast(cond_op.release()); }) .def("set_truenet", diff --git a/python/paddle/v2/framework/framework.py b/python/paddle/v2/framework/framework.py index e16bc72447..93e2218eab 100644 --- a/python/paddle/v2/framework/framework.py +++ b/python/paddle/v2/framework/framework.py @@ -384,10 +384,8 @@ class Program(object): cls._instance = cls() return cls._instance - def __init__(self, desc=None): - if desc is None: - desc = core.ProgramDesc.instance() - self.desc = desc + def __init__(self): + self.desc = core.ProgramDesc() self.blocks = [Block(self, 0)] self.current_block_idx = 0 diff --git a/python/paddle/v2/framework/tests/test_infer_shape.py b/python/paddle/v2/framework/tests/test_infer_shape.py index 19bb45acef..5cfb9e6687 100644 --- a/python/paddle/v2/framework/tests/test_infer_shape.py +++ b/python/paddle/v2/framework/tests/test_infer_shape.py @@ -5,7 +5,7 @@ import paddle.v2.framework.core as core class TestInferShape(unittest.TestCase): def test_sum_op(self): - prog = core.ProgramDesc.__create_program_desc__() + prog = core.ProgramDesc() self.assertIsNotNone(prog) block = prog.block(0) self.assertIsNotNone(block) @@ -33,7 +33,7 @@ class TestInferShape(unittest.TestCase): self.assertEqual(out.shape(), shape) def test_mul_op(self): - prog = core.ProgramDesc.__create_program_desc__() + prog = core.ProgramDesc() self.assertIsNotNone(prog) block = prog.block(0) self.assertIsNotNone(block) diff --git a/python/paddle/v2/framework/tests/test_layers.py b/python/paddle/v2/framework/tests/test_layers.py index ce20371cfb..2ffadf7371 100644 --- a/python/paddle/v2/framework/tests/test_layers.py +++ b/python/paddle/v2/framework/tests/test_layers.py @@ -6,8 +6,7 @@ import unittest class TestBook(unittest.TestCase): def test_fit_a_line(self): - pd = core.ProgramDesc.__create_program_desc__() - program = Program(desc=pd) + program = Program() x = data_layer( name='x', shape=[13], data_type='float32', program=program) y_predict = fc_layer(input=x, size=1, act=None, program=program) @@ -21,8 +20,7 @@ class TestBook(unittest.TestCase): print str(program) def test_recognize_digits_mlp(self): - pd = core.ProgramDesc.__create_program_desc__() - program = Program(desc=pd) + program = Program() # Change g_program, so the rest layers use `g_program` images = data_layer( diff --git a/python/paddle/v2/framework/tests/test_protobuf_descs.py b/python/paddle/v2/framework/tests/test_protobuf_descs.py index c775b1a398..6ed8edf91c 100644 --- a/python/paddle/v2/framework/tests/test_protobuf_descs.py +++ b/python/paddle/v2/framework/tests/test_protobuf_descs.py @@ -4,7 +4,7 @@ import paddle.v2.framework.core as core class TestOpDesc(unittest.TestCase): def test_op_desc(self): - prog = core.ProgramDesc.__create_program_desc__() + prog = core.ProgramDesc() self.assertIsNotNone(prog) block = prog.block(0) self.assertIsNotNone(block) @@ -64,16 +64,16 @@ class TestOpDesc(unittest.TestCase): class TestProgramDesc(unittest.TestCase): def test_instance(self): - program_desc = core.ProgramDesc.__create_program_desc__() + program_desc = core.ProgramDesc() self.assertIsNotNone(program_desc) del program_desc - program_desc = core.ProgramDesc.instance() + program_desc = core.ProgramDesc() self.assertIsNotNone(program_desc) self.assertIsNotNone(program_desc.block(0)) del program_desc def test_append_block(self): - prog_desc = core.ProgramDesc.__create_program_desc__() + prog_desc = core.ProgramDesc() self.assertIsNotNone(prog_desc) block_root = prog_desc.block(0) self.assertIsNotNone(block_root) @@ -91,7 +91,7 @@ class TestProgramDesc(unittest.TestCase): class TestVarDesc(unittest.TestCase): def test_shape(self): - program_desc = core.ProgramDesc.__create_program_desc__() + program_desc = core.ProgramDesc() block = program_desc.block(0) var = block.var('my_var') var.set_type(core.VarDesc.VarType.SELECTED_ROWS) @@ -102,7 +102,7 @@ class TestVarDesc(unittest.TestCase): self.assertEqual(core.VarDesc.VarType.SELECTED_ROWS, var.type()) def test_data_type(self): - program_desc = core.ProgramDesc.__create_program_desc__() + program_desc = core.ProgramDesc() block = program_desc.block(0) var = block.var('my_var') var.set_type(core.VarDesc.VarType.LOD_TENSOR) @@ -113,7 +113,7 @@ class TestVarDesc(unittest.TestCase): class TestBlockDesc(unittest.TestCase): def test_add_var(self): - prog = core.ProgramDesc.__create_program_desc__() + prog = core.ProgramDesc() self.assertIsNotNone(prog) block = prog.block(0) self.assertIsNotNone(block) @@ -121,12 +121,12 @@ class TestBlockDesc(unittest.TestCase): var2 = block.var("var2") var3 = block.var("var3") all_vars = block.all_vars() - self.assertEqual(set(all_vars), set([var1, var2, var3])) + self.assertEqual(set(all_vars), {var1, var2, var3}) var2_re = block.find_var("var2") self.assertEqual(var2_re, var2) def test_add_op(self): - prog = core.ProgramDesc.__create_program_desc__() + prog = core.ProgramDesc() self.assertIsNotNone(prog) block = prog.block(0) self.assertIsNotNone(block) From f4a21e387ffb0a864b8bb9822716fe64aacddaee Mon Sep 17 00:00:00 2001 From: Abhinav Arora Date: Wed, 18 Oct 2017 14:06:30 -0700 Subject: [PATCH 59/63] Design Doc for Regularization (#4869) * Add initail design doc for regularization * Updating image links * Commiting the images for the equations * Adding computation graph images * Adding section on computation graph --- doc/design/images/feed_forward.png | Bin 0 -> 32247 bytes .../images/feed_forward_regularized.png | Bin 0 -> 46036 bytes doc/design/images/l1_regularization.png | Bin 0 -> 1157 bytes doc/design/images/l2_regularization.png | Bin 0 -> 989 bytes doc/design/images/loss_equation.png | Bin 0 -> 1589 bytes doc/design/regularization.md | 103 ++++++++++++++++++ 6 files changed, 103 insertions(+) create mode 100644 doc/design/images/feed_forward.png create mode 100644 doc/design/images/feed_forward_regularized.png create mode 100644 doc/design/images/l1_regularization.png create mode 100644 doc/design/images/l2_regularization.png create mode 100644 doc/design/images/loss_equation.png create mode 100644 doc/design/regularization.md diff --git a/doc/design/images/feed_forward.png b/doc/design/images/feed_forward.png new file mode 100644 index 0000000000000000000000000000000000000000..d312371a04c26aa6cd196e0bd1f51becb425180b GIT binary patch literal 32247 zcmeFYWn7ir*EI^bC6z`+kl1viNW-RE*mS3el!PGNAt@y(9ZGjMNJ>aZcY}0yowe`t zzn|wh?>Xn=`|b7Thy7#2b*-3d&N0UrD_BuM>M`bXOe7?v$1>95%1B5kX-G)OQRt}P zCsde|Ye+~GNHXFgs?Pd5DHzU#5?A*IwA_>!Mv*G$^SM}<(4Y{szso52FSiDX6vboO zA3${RvVuy3SSfOop_$&x22`j%v1%eBoexArv~<~g?Rq2MiBfDFcr_Qj;5YhOS8rPH z)_f|^V4PHJJuHg&tI}7PkvVN#*6`;spmm5fN-MGBQ)KX=!OIht6?9zns*|QXC~ykvdIR zuf3l{dPhM%5rL;mIuiAT)1o|n{yITJN$KPD!BYFS$@}+RDNge)e@czS`Y7n>=_5;8 z?zHXf?fW(cQrrZ%xXwDMq{1Inym??|CJu%xNhribhb)Hi3<_rQc$edewHFFYvckr- z$JL3sp#6IPl{$WMmBVBIzqv|T;l-)sZ47l z>$}d{@2yc3a*kWFB*QF{Qc|p}#r#UGk5+qLfrq8A#&7RXkZn*YiF+?|gzzQFhdoCz z!UKP(#F09wn1XJYC&a)_|G96+JmNA)(f4q1ofmQPZNNXr>gh(CBhp{`KqEGXZ}>K|lVXIAwQK z*)NxffkF{e{moBd_RR&IhcvtgtjA6F7h;a(e&g7eabL&;>)po-wcsuSWY@ba?5J9! zv9gE64tfKqGcm&ApTXA2OA#ggAVJ^noDAH*v0 z*?)il4I5wj<#kZ(?)^vw&sDV9mTMli}dTc#)3DT)l%Y?#mQD2W6gFkMp@#rURsx z%b{%fp6gh+4|CZ>Z<>CC2%m2Bn-ci1lS%Ov;)|c^5Ih(Xf*BkZrZydumc}L_A)&0V z&q_>8tR{3mZkRmzLcqC{S-(*ecDYKhIbBt-pi3`&E7|$wj0zm%7MMw>FSYO?F*OfN z62*ubxALG+-aTLLaB>g>nb%=!G=DwWbt|vjs2$aOs{F?h2j{*VTN8D>p5wPre2MbHLjl^H5&P0Cw z3XRuy&AYi?xL5atAYc9FUXWH+R*t=J->ch*@soP0YwJS_#+;~myH!vEF}X+r+RQJRicrv`;cN7qD&+Nb6Yk;SEoK1OSle$>4Kg=yLYF(HK0y7Td ziY~L~blaC>0dZ~lbA#hAMRluXZ*DrneIyXSt%)$-*;v^4= z_vXvgJqXjrx7f5vQa}MomkfeC7|@Gy=-L0~L_Wp*Y${xvFa?8b5vJvr`qhH$6BVt% zClk!P#DYlq$P233Xp%)2Ck{H#J9Ef7c6CrYb3Cr&SSp_Sq=SO9tYR-M&07B#N+Yh~ zhx*tZKj+{iIEZ*hu9fYO`0tO$tqv!Dm3Tk>jf6^Ehy$W>%abqPeRYJ z1svti%OlMXI7wNu{!<*UaM5MGi?u2wUC;MexEI_jn02b(TnaC>50@ASX_dbVe)Nnz zJ<(yyhYIpd=zP|`;AlA@OUI^3tta$)9xnhxhJ>K9_`S0876-a!hzZ684eT}r&iJt3 z$U#3}sNsCxEtVcX>^WyXYTrWRLG8dd4yoXcf^00dMnxBmFQmN6T8=pvl{-AIKQ9rc zW&cHeE`s7o07i7B=ZZw!*V`M@uo(3 zeK!&HJit-OJplPt0R?gjMuMdG)d(vl+>8DtoDlaw&fWJxhn}|fG={XStSn&!b=+80 z0gg?}ZS~3Kh*VFD$F*bKR(?SYldXaKp#&&!ioz~q9x3#4Zr4*+r`tc=&i7&zjNO6&QMeeqf=aEKHi z-P=3wA^m4lB*C7-VPOyp7(T(bW0TjxMJ&_)1glRC+{-#TJ9|KcF|air_?9=qtY>)` zSXk7`ySlsk-1eL3z)^8klm4C|7I+o=^qQy4fN)-|h+D17Cuc6pC?Oyxj4*9!sb5>V zl^!LVFr6?}(t*3LiXFM7gHEBr3#1?A7SCJlByC+?<`jOXIFK_&gR^vMtjjIN7!k?W zz@@Cx^ZxdnNnd&Asf0A0Rpv$I!C#JVo8ZuI<0gush~TovBKMCk)u?>gsx%-|U>L;@l%si&FiR3-RKznV;QnWYMW6Q>n$<-~=Z zyGyp)a#n-kb{&~)Qsc6M$_<9Qv}HYREP_%W46oE93a&66jKianRBiWZm*y!7$D%0Z zejVAkrLWAqy@DxkwVVpa3g?SXd~<02Cp<%o8W-XdgsF%*Na6Y=LYx3uuaoZV)3Pw0 zz>mq5sbW3;&op~M`%@J;<^3?@u16-sf{~4B?nlA8c0Dw*TAJljUks^}Gi?@{n~wHD zNQ{(nW*33c*_z~0<1(P&;epXbfzeq~qSS}LnJWxj)?$wiyUCf*Z57c2!?NLN#W&p8 zP!9l6dc&lARwcY$bcKCrJiZqeGRsIyM{4N1{J`Yw*a#tFqNb(IR=;V~9}%VEK*?5i zmqg+#phorsD9c;f`T1-T^E@je?`)05^4L~d&2Y)U;pbDk!-u4fiv zo~6kraTQ&;P`G@wBYS&xxsg`lN=7z*xYCV2yRz{yBATPU&Fb&qgmd^odyu*3L zAr~#~6J*&X^A)^~>`nP}g+pRllk2lxl|9F9WGdafJfalNzQ@#TaRRfS)A~H!8%tMFRC=f7U*UWGxl`H^=<|uIbG}d!~Dh@1fUa{ zJRR~2bZ5kD_##d4s1;L?I_ab!7(>TBFNIO4572xpP-5DmS9~FlC;>u1SASw5fpYSn zSN??y} z;F6Pj?X>zl2!2e$^*a5NJ&~58A_a&d76v9JxmWQfm0iqWZomH!RQf^s!P5) zkZj%jbjM9cm$SPTZ(v;!mok}*BXpw(8Ystl-Qo(X1x+q?dN3vtqZKh~Lmlv`dRLo^ z`74AJyL&Z?O6m^4PvVbJl~kk-5&sSNQ+I4fyZH*Gz*}5BBtvCdrNJLc{xO*vezREE z(a~WA!ZNhx_z!>FY!bJPzPhsVAARxW&~t1)af+f!=W$=DxTR;`p?F~ z!a|?Wr)zX)q7% z!DQZw<`;2`Wuq{Q(cCQ1Nl)+E)wu@!(pSODd6|s>ul1}MBGkA_h<9}k_JViCEiy4O zT9d=votygm`)iK}_~C`r;*>lscUP7@zkjo&@jLAuPEJlfts5m{$>Kh7RCW&-h@^@1 zP5~HwI*)tmNd*B<-;(X4H*okMBO{}Qin@B7nWLZD_43M!8~~|~Ya1II&H|}40=n;F zQ#k@t2-DX8qG^kuyhnsyTx$9Y>U13B0do(Ya>5rN?zhmRC8E3^Vld-I`9NPOh`$dG zy2^OY09BEIkcBlcPqHSy0TIlPh(X8*IBT!<;)f{k(V2V*@jeDie-0ugh?0cbX^Pvj z=obd@NPMR+d{Xub1gjw8*)q{4o1)&!Do?*K`$L%J7cuaNS)hlB>M_B;9^twC`oofS z2)cE`G|~tEW3?Pn4m|H@p%{sYauTc5u7d6JPYWX6gs|x5%+SSvF!Wv=T9ZTo8l4OE4DnOskODWU;X|4+dv~K zXkucrP&>CbD86csH}&>6?T7__kT#gT(#+|)6;Ef5%XL6ply@qm91sHDSyB=DW8Y;6 z27kosKOgxa7r$CDz$XWi@pqYMTF_}j%gV}>m6Rw!Gi7(MDALo{7rozntpEd)?}g}>UyK1GbOT3KZzIjivlC>9fcU7Er0cJ`lyic z(Swj|Sg_Eaa!C1t9)&u$LIIyWyp<=g{cy0%xZAZWjO1xjYO1HGJZroh*M~_8fapHE zU+>l&jlhL^?ryJtl$I*?_Vv-QvMPXW_^ws)-VQ*r-nxxcrlYWj6x_lnC!1piW)C8Zo7qYfmzYvL}U9Sc%1U<~4i6?XVTVZvxRBX!?Z zaunzW6FHfRvU3Q0k7G6Tqmq+JB|hnY>?Hd0L++{hL~%;8nGY|a3NqNg{kJo>i~?TpGEpPXVa!rOu{Qd@QMCu+l&JFvBmpGwg;}#y*t+Rdz3#A! zBJgH)o~4b^>TX+`>;-qe0lqgfmi^g7Nx|*S3jpSt@weXmw~%j zWRfwoQt>ZcL5FON(b3ff8_tvn#j^pF0%PM>3*Q|3Hh+xrPn*ibg07WNR1$9M!lajh z*e~#d=7>4o|MEtD@D4PHV*(im#q+l9kN29~&Ij^xbDgDYLbPx}gtQ}F?rtnj9rx#(Z2RI^Yyizw z#IK2mkqZV*-0jN%b|A4_$BYo54kWo<(EI-^g$vzQW+YlV!$?nd+4mW3rc7IF!T6 zDk?NDUn&89m=4fl@n65U7E>WTyhJyUUHW+GujO!hD^S3>% zcNXR=&7%tqTAE*RbJv)i@6DA32M61LGT1^5wfv18e5Mr&fL_&A=(yxdWKg_+gA_V2 z7N8>9H~`LPQrn7Fh2cd(DjCXN^tC zMjd7=Fwq|k5nWOkKI(w#o(O-zPrCJ6Li0~o@EK^q639fl!(YTkA6Pu2o%R1TzzdK9 z8Jxtk_059EYjy$1TL2=++1S{;%s}Ff1)we13J|8w7TD==574lGfsXoPto0jR-B|)q z%O>CH6AR<@%l@bI;c?{+j4(*jnD(GKbV_uK5N=GcA+-3Loww~8K`;1 z!2#E=wKeka7b~|AeU?Kh7C`#@_wU=WH9I{z=zW!ij+xU1x~5Y{U<95e)F%iJkQ|KD z^$wN~97^c0iK{+L{fWt$@epa-JQs93Q+)wzn;*n}!AHvLxbw`~$O!6VMma?F^5sh> zFzp^th-<8-1>DX)_`5N&GBETI;&ot{Kz+19jM~{odb?n}bJ(4BSfWdF&zybd-;^3g z!X3BZc`bjLGxi724XfXy74UBeL`tJrn=h7qLD%qjDfMN+Dedm#ufjHg?Pag%=(<17 zIoRS)Gc}!#>wJxk{oL;FA@Q!NNxOUeFUZR(@I>ZN$S3N>Pkb705?^ofaTae>48oYu zU3{bW)a7{&!-rl18mi=vztFO0^O{}Vl~$E}WfJ6udT4IRwz1*;@8^L;=T28sl0RWX z`c1AP-uf)i?HiBq%S>>@O+TG->wxB5({7>8-qa6p)=cVD=>pa-w?=XzSc^0(-iMt4 zT!U8uNbLevlPkt|WBF>nGZCY$#{lJY^?0y0U88g_=%lm~13^e9jzwo2>F4rpDsFPK z*Zp0>MD6b#w}zVdU94pAuEqWf2!_9(aQBxlUrrkwcV#6^H0=V>Sy)-ei}nzFd>sxM z-B5eoIWsS>76sqfk8>`o>DuF?qvDRSjC#)Shf&_a=sYO{R#Q_`yOLqgO`ru|I8E03 zzrAufTHR%hFK8;`Ec?WSXo%mI46mC}uor?sjKs0(C4T8@i!QZY64h$bZn090!cRM> zUS6jfd+^$@)qB~bFK+4WmCNu@*$3lKc+%3+&gD+~^X;Gw%G@*5Q&(3X2AO;#E-nuC zOs{ER{0dOko1wT~a1=ZW@7AMeIyl+)y)aTC&qhE|Y`OO85v2j9xewRO~~VKZYTEG<0+}vsIQFnM61k0hnlk zSTmL8D#{>j$cu9YYLlha-~h0GeL|!2mI?*L*xlQya+8$1tBDrIqTc@g7!f~IH5aT? zvwTL_l23=4=kfaOrnA81YV?}W{bfWlAhumq)ruZDe|*ZO&x~N429@CVcemW+e4qY` z&M_JLP~(dH^ISuKTH;sU2ARoEj&lGC8wckF-zSO6SB4F6&^Y%fiJ;I@Mqwx)P|c> z4*10%x*gX$vvtR?nwdG#i4e_a7~*lQKUlOYHJeNa$xIfi6BkcgcFOx=JZ6LRGv;-l zDO4z#B{k4JKikn3z9gJCXIU+bXirn);q)uWJ zv;jx04{&~|zy*IWDKK~b%nEV-Azf=x;c~n#B_}5rdt3B|+7Jx|@V>58{QUgech~zR zo?wv?+qpZnRG{(oqGDxVasxB70c03&2x6F$#V-AG)Qbqo?o>L8(ZDIt#1aDcG-yT5n*vzU$Net9iV%TC^Yp{PX@Psoc&fv zOL{kaS3~x=$YoJlhZdW>%4`^g{E?neWL4EhM7z8@`Q~sLbNeI6mBaS9)o{RLl9-vE zN(RkC7eLi_>V%vlVUxd2tLDl}9Za%m*3x%bBVu4?R*gH0s=W_<`L7qimWC}gkHsi< z1pp~ovi{4&z{i#ZX%`QUCJelYBD%8(=d$HVl{9q99h?n&GMCk;rTdF+Aq6wZbo&BO z-lndO{$@p}GhspT(9TQYBr-&47nebJRI6p(WCd~LPX#FYzj2B1V3P@G&V$v;C!b(` z+Hyb(!#gWl|NPYXY`12ii)@czI*bcsW%E72*hp&D*_b!L0N6ZT`lT!_tbwvLdz=@b zoqwrhF_s^@ILeQWzUe@Z;$SFg64{iX^wuaCJ94YVymxrLD z)xQ&7idm3VUV#?PRvC8xD~c+taJHsW@4sZSbS59$u1o(Vww9anN2EN8=6+|D{Ab}y z?LSQ4Cc_2I%b7XwgMx$e=vi6A0kw=S<1RY%vkwCgvn*#0o`)&yF!RN6s%f3!s_6AN6^sNSE4(AT1x#C`PvjKH#&35>|rNjK31xh=-ae(9H%{Nj7-QIyc z1(hF7sl6i)Nzv5K&JN(dwf!9WZ6;6%EJZheM^)fz zjntrq*1uhTELFe!&S;&=dp|#?`hl;HGi)q->D>)r8j5d>(4#KLGz=JCaz|N&z|lpS zzgcR$qIg?UyNr1Wpi}Gy=sgOS9hS3S-f-`z;c(@17v$y4Sh~W-V)4dcAJAGo|Mao7 z=(^a9#yOV`x%+V{KT&d*3=!C-@u|mn=2_G5Tmj>{4cC%@Oonqw48KNqHG`nHHl1lRt+DraoWxv2je{>0b7|9OfXJgx!1x%3U59S0&o3wC=VA_Erl$QRF z9}bNEzyjKV51F?tY0CZ(o5uia+Fg?L$LQGkeBeJpxQ}I{a3uLdSHmX%qZ{5wl)#9C1x_n(jbqRPt5?tVCvwWwm8&qp5K{G;AzlY{ z+<94qs(ilY+4~3Z2Kxas1nv}w^p?Q@IEVaTlCb#pt z&Qw1`j|zM?%ODYAGSU}R=i@pyYIjZRb9+D%tcRLK0lF_|^0P-O`mMjeP|QByy)1sO zJeE3pHtAaByR5?Y6;#qk6y871K^x7b*dF)j1)nN!l>Tuw!GrwqCh255Df%ZJ^X zJD`WK?a_1k!#o6mxXD@TuSFlU ze1W`QtqpVX>`?jlD4jVyeE#c^-f_aBqHQM#9^o1wq3)#(P1JS~V)?R)JX@YVYs!fG{~*|`OJw^7f` zU-#wADHQ;~BlxrBx0eyjbGXk-NB~xzQ!J;zXAeL^8Neao`p_B;hvm&a@}*(k*>?p6 zF?Nl!f^OE?4nR{ZYdxiNSJ=g(9}yrGb+#NEY`7b4D1Fa*5O{)3`{)09txh%p40jp&sg8vfY_3GY%A+yAEzgLiZ~-3B43E zerqrdiq%89xv0{KV*pVPfzrSQJqk@t*L;GJ@I^Rhb#*li(G68DQ~p%cTDCPx;Ms1Rw*@8ovm4Rr@o1-qMo1kz%V$qeOj_ZQT|ddm9Rst>RHA*Jr9 zm?ZlN2s8#=@H|z#pZ5vUt>XyPo4PaCaDD(>YPq2%NwKkNm9D2-bw7Umz?rU(o~rc* zNd@)G7o14pcZl}NhE_iDQ}Oe20(FXceSoCpU9Mi_%;0UqAJgNw-3M)77WJ)N$-Sd5 zSI0*e0_0E~3qOCu95COt8MydcB=g$SViQ|A)#>h1y+bMBd0U$~}C51o0@DIE}(b8oxFRKv0fq>$$|~TqUYb zY=4=}nVOPW=asPR1Mjw#V2V+6_pn+&X%LE%Z{T2J4N{@Mu%aNp>SZWIA@%&E;Z75U zS}Veejt$vWmJ_li`c0)d*A{qMrW20kdKn1OG>klr)~A#pC@jol*%yoT*%=@*?#Oob zad7B;CDH&C!<%~$BA6<4bKu8p(A?O&f^iM(ESbIlli2A?NOX!Bp><)(vOGv9vw zHftTwjPmy^G|0iy2kGGvx3x=%WUc!Ri%~Ffaylpgi)VDsn{Qud^VOI}YHk2fi~{C@ zrzN^|<&(RP#^oCc^JdS@7(r8-y@mZ8SVIW@0%|Az)7I$bDo|g3>4NmlG<%mM`6BcdyzT=M{KO+2qnhFaIP9--0_#v9=%bkkB80VCO z;`b&dJuEe|%D+c~U2W}$LHfbswweF3R^#(1hwziZ(a*nBxCG?n_s)RT69a)&M9T60 z4S-kj@={A6@xKBd$`JFB>`vgS5mZuFKg$hLO&m~s{>~LK>V4>QX%u9!nr za7eKLz<~YyH^?$3r(0v??|Q!af+me|eC3dto`uB&yKK*oH3sJ^62_=Rx~Z${l@&OW z6CiojLpeA&*1+z6K{$(wqnJyN$^f9&YVq_iv9`7z1`jj_R@H@p!9gz>@$vFdX<<0n z7S})Tw5SdBQM43t>|2Mkq)IIQd=K9EBgYoD352#+0+|Qd2>T>ZyQ#Uj)PV~AJvUbx zG`I8zo~`YdQ>JM|UMIs+v4E-nG+JVCFBsof6%+WVe`cz(zZz1{OBzlF*pB+@($Z2r zpaz9k*(~VuIqoEcG|li7`tQC{$Wi%^=z-gXu@nhxWgWYYOi}Emzfl7NN^{(RNVErZ zt0hwzb93_<9CFxAb@z%$@<3Sik6!^22y(Ea!5cMFcoJi=d2Il=PW1rmaj&edK3-l~ z`ML@b)oTuW<;+q&kO1Mia5sV{I&TU{E3T9L%Z5o%W) zRO!H%vb#N2Fp&*{3Gr6Ob$if>?SR7weyXwW3BEo&~T9lHoa|M+VcSIJizbOHU;uF?qXmspNE8;Oe;A`^2nXy+tgB!*&i`kt4pF} z)3xHnZZ-s=eiasm@Z>yb>+D|wrW<1H-a-J?#k|QGw@-*zXpp_oZ56Honb8KEKpcg& zTErL$sYpVq{~pD~WNaV+2l|0k( z-im@pOb=ZVe-{^&OP(JEeWfN2ZdT}JS0tiFfixlMZJ%h|egbwwenH}OB;*lIBoxYu zmxmi=4E-Z5jvXrfHK-#bNJTv4QI+>+aBl^u2lm=PJ?Q(_`b2HWfC3NtWri0#p{Yj; zcC~p%CGeMdYSM6#Na~=#;yv5{oLv}=F9C3X7n98RVf5=t#@985Z zR}kz0xQoeL5GJfK5T`k0gj%Q=lYZbW#^Uez&RwUo#HK=W=~{VToQkz3BZ1=lub(9K z>;LqV?15wAAk~X!Ero;x0A{&I6 zKcas=DS6}lKIP1GH?NiAgq;EjWc+()Dg}(>wHbR!X6Eu4Xrd8QQ2Ld83 z=e>9ZCMM?Uc1cUz?CdO}eIknh2=EZp)pq6)q~{h`DC%ia@An}C_7&%mw>vAzT-9*i zoUk>^>OWP>6`w~o-CKaqpvZI}S!+Sow}LD8SwFaY!dDMy!8y>H2L|nM>Yxpg&ES>J z3A}Q{`Iws26v%8tzVRucA<~|&zm-Z+FZSXSU&{RpP7BqUot+)@1fvhJKzNAz#$mem zsZQ6FM2A2w!V$$W6%0e?pJ8zBt|W)+wtQ5O3U3%P&|QjvfOM<)7jCX3ZO(R zH8pjuv9U1@nUJS@eBatMEtFqkRR?d%x+A^9{F~BCusKee#i3ta_G~l`JQmsV2#!B0 zpbaY2(DCNnikjEC=npR0z9HkaTNQfWpMbZaLBNu1e)4yC4OAr8-j%~ia~l!v!1m!R zr7X#bCr_SK0~|Y5cD6es2S`d4IREZzbHKyC22E|ULeBT>Y}vPOGe-+F;Nj##tf2ZX zJ2rH#*&$3Z0ONW_k$c@JRezXTp0Bnd@r(mgs=fu+mu%bwQqLvcWqoc~P#noJCnLsG z4s5gH8@#1yKlf?*?!LN`JdW`lyHor!`Al-?=(v&WtO7K6l}$HCeGbo{>#WP|I6^Ui z;N{=B-TD93=ovx1s&^1IGRgoPh|GA2fx)+=B%MkCD<*--uLjsm4e+C?>eb%)`DU7v zeQRp?@Oto8;x}RhD8IUSUm!92RY)#@5sa!IH|o2)-;@C@ntIx$M{if6b|3VY+Mu+? zy*to-nR)TQqMu1%jPLpNi#?OI#AJPKtp`|d-vMjOLl+kp6Hsi4@m%!9u>*diN2?Rc zAOHHUr)Uz_xlz~Rdn$?=1jwi+wF%4v$K*cIqb9bD1$4U6-qF~uwU^IM?^ zz=Zz~eZIjG^xm&PSY?GiVUd-Uk@>do*pL(Siw%#P*$!2FOdy2x{k=u?qQH5H5!9Xh zLXPpo-6(Bj5)}Xu*iLsmxGTfE`8_&@MdeNAho=>jHT*Hw6}JL87mrMwpIv`=1D0TA zZEX!F>;-?dU4O#MGN8nAR(Daz3fV4tze$5Ny#-v;OMt53F|o0NjL=vSLX%Uew!4gS zK6a|znx(*y{RdQNU(U*0)4xh3bpK31k z$o*736W2MlBi4B?@=nrRbN%&viiYmT&o_Y4Q7Z!O0xE1zb<0ygrBcW1+R40=PDocz z`fY9_o@J4$;+|-zY%Xs$S-KPoV&`^!FfDDo!W15@wK0(Sr0(-eCA6WixIHoj#6Ps~i&e7GY>+WMVP_zL_Jh`*W`l zb}Ilafq19#XJmv5lSVmq#zetwCX)rk zM}%=}TYR+zS-^U%Fu*If5R(?)ab^)QZ)#M9Q@+P(BXaSqu|5yb7^|S=za*aOeG)A= z-W|Wn+1X^{$lp?yRK<8WNO4br+);Sne0`^z!(o+3s z#-;@$GfDVh`QW3KPUFpQQ}Kx@Lj|{X{3^s21!>_sfBUZn*&R6j(e5f`(h>ySs=t6p zsr~+)T16a}ZgimM04FwOX>*P%)GPM_HsmG@(MZj(fBrG9;nOBGDDMUk!M#ATu&lBJ zrmmw&{$6^7 z`6grex-}9HPcDJ(nJCcTMN>1wn6|jiV^+(!9u`PC825B}>M&oE6-* z$8*y*2-EHc9F310oOnq0!DXB-qt4ruJLtcLgypc>frSm`xWOTu*iKFHS=~@iu*g9d zzc3caVUTo7t7iLPElsG|E@}f=LxXT19W}v*(XZkn!4hX($fGBt* z;QGk>=x!tv2|NO$1zN zf)fdlN^dQ+v@EZ6z=jhyo3Hl>1pJz^e&y`(O2Bq7KGzUJPY}$$!Yq~fdG>hh;9s)J z>EyF6zvCM?Be0hbK4V*>qox*^0=mo+Agvhia$%WjfMg&l0u|pHICT8*IK}UKAKxl& z+7vg*RsKPw%aLZBM=2xI>~fl@z+!jS(sO^i?mklAtbNtVE4(Q+7(7biO zJ5Iarpn0Lc5pZT0xizdR@!o2gz)f%;(Bl?s8KllK2-7HT1;So7w*PSm9pzcy89*8N zd^Gn!ed_^+w5PT(D%!fw>yZ(k0C-Tf78X@Le?LwO10{sv+qZ8B!!9H8Ch(c$_l~$~ zQ!+-7@GJZux2SVpQTUM`1!b8ZRYfqoAfneD>D}X9#j|DPe~7&vlQRhfZL{Sz%+`^ zbFM_cmhe}rT}aiZ`n0KN{gnA%FTnY{%G=u%AaFatLd4J~BMJscoTb4n->Q7W@##Qt z!Aq2ett;-Eia!7yW^HG%i<{%Y0@2G+2YIUb9~QsJXE}?BXzsWe6WnP%$e+P-mOg#} z`dn-dUX%zft;}Pcld~BcxrOE+Wja|>xW7^xe-qbXaO@BJK|u`$oOw99PRwUsL2!X= zE!-J>1ejGN*=Z>(4i_$*az^q-{YG|cP0RtI6EZ0=tLSxPj>G(r1Jzy7IEr_-h?sKF z{>I2xuzs3JQwi_niNI9Ntesp+efQHD+gpA9L5~nPC#gT%3KAvo3#QADH20pIpKo>T z0>M&!s=>GCt>obppwYR@D=I21SOYI+2854+B}N)=ReJsmDUXio&@D)@F}W=k;dmj1 z_42x396VmiA6ayLbg%f(qNPUDd%m|-pTl3{XXDR6G_psW>&hGKp$}n1hadjP>Un0L6%|R`@UakzxhcI zaMNpd5-%v9gmDQA2^1#HA>@DL`UfGc8a`) zGOuqBy~m&6xKyGErLx#v(uS&{E%}T~OEhKUg6b`RU}Ifqq?pd81>o+sb^h4T?-P)& z+D*XBs3lJt_KpqMr!A`xIX?R|x$6E$hK|8Y!bNV9L^*!qjV_U?!CPj^$5PSPk{tX9F~|&Z;9>sAbn__cu9= z9ba26zmT#$Nf?<=B>y9TuunZ~!^bcx!OdF)46w=_iM~BgtS!H~mA&@hUG6pxnb-F4 zfPG76PED;*Goyg(NvJ$WCIW{2oWP;0?N<7n%XU%NgkOrjpk_

`E*o11uHRpCa7ndF@-pdZH2O73WuQYT+T-i5lHsIzjjyTbVNZk5 z-B%>$B@Mwv<#nL3Q?)P2V>&6{%Stv~$X$Y*RKoXDGwBz&hX{X9`~hwVc0#K45KcYW#qD~5C3NAimbNnS&_jaYF>Hf3@&_5O)~|YtvC))X1;U77zBq=U>lK9tBx|%5a@C(BEFBc z;f>bjpX|ga+?yBhEdd;9_qQ%OZpNZkv6Q48z$+GG;IuQTbeWJR=b*ST?I+{2n)1pd zMBwJ2UAE3UC~RyJEsOTX0jU&E+fj%XI{AT5Pds~+Faw0pe7*bS(Q(;6$riUr_NG5|iqTruh`oYDcl>xuqlf@7595>_0Vu3uI)bJ@(-V~=!SP|&Ho zJ$rIL6w*z3hSUN;Y;OqB*LpoMk}fQvKtJ5Ks&mH?LW*S)XwWBaurbK(XTMN~ z&g>$qgxup}}-wPmZQOKfrn@khTu2S8LW zpjA+M3--3fwY*=tU)V~#D+Vt6E$vctH<(n&FCfY)xz)C&j)x4PK7LL*UuG;-U*Au( zcNw~i!_%3+xBJuE-!{TOP|S+IO%jxR-b)ZRd3%^R+@$wg1r=VunPI~MjpB{BN&$yK zLVtl*UELwKwcm~RcAHU!dHCCzNBZ1M!7bH%eD{pO+0XUR*6UU~dp^x-N9O_YDx2QmJ?T{RLdsH0IurG&d$!4z-f->hI0e(V9Am23fmLsbgDW796HT6dl#vny5AwB~POA+jBqW3wwp{1OVXE%rDSc>68Tsi(mT`Iz z-hIEwE>5aJ%^6V0$5q;dd0Ez(UwokN6Heohdd&i(WM9hvjNjGu&Z;e4cvdJmPJN6o zZlWyk5fP;k0f+8So~9r5QQ$I9KD`^b`)!JcN%nu=C#MV=+4DKVr17c?TJ$=lVo<0g zGp~BdiKasz_ULR%I%L7ns`xOO;c7xhXH+oF8N-P9;Y!K+GsHyCP7Z= z6XWHNCW(ick2{I>6fLVBqeh=3nP1TtMt7~oetY*}R6^zjw}mFkUnY3zf`@Jr=KodN zS4LGCZEF)7*mOxNu_={O2?6O238foRM3hF97LbwWbpnp?zrEoN$e$&Kz26ubn;S`~iemUm~L4F}~v-tRV2*N2Yc0SwM>-1bge z$%Co8)~_rz^FDin{V7ZDaMfFu^vr&Z{v!1Z7;tyq3v(Iu|OBXR9O9k@q3}+*3Ee> zbG?F_Ml6%G_Lsh-Xz8Rl45WDW)%-56WMRv~MX6QkpfoFkzbaU8pVl~Y_lp7gl3(wn zdr%H#D7mrVw>Rv(UuLJL0T!;sd7*;g1)-EK4A7l}HoF1%F}&r9J>;A^{jqr68aEY@|E*kW|@*}uazZjJx-hP-lx9BOiecQFn2Ah*nKPT*yKy^?3zCKQ7eS8mosA3<85a`9UvSN+w(rJXml zsl(l~T2nLfPrTlg53PK&&L-6*K_i622P|%9rz6}&0yf5{egtm$On07LNzoGItxa6@ zq4=7#y#ugqe6mx^0~FRo$B*7y&t9 zd{@DYsmrHx4GgTJSmoaBx_+Q6tXDA1LXh#sHq)P_ti10Q*%NUH^ozFej@RGCIgKTc zvdhO9S;?kj1L9ZX05X#&H;KALxH%d^r;;4qb`QFw{N5R|PtgDn zXw5PID6`iTy*YZXh)|Am@M@2=6Yumb&(67(A1a|r`F2gRIb5RYtI0j$pr~`F{`hcS zT5p!hOYR$I5E+TS0P23FS6W)x-Zw>mVtN89r`NYTYvmqyhI4xRHM+1$1&m6P)-OlS zVL9~Y^fCe0#VCC|d@;{Qe@`;*K9KE`TmkaQly6l^enoy?`&Be{28SgXdl{otf+&aK z*qeu?BP?G$ehCoQlmFqc{<C>FRP+Y0EJcq6ws@_vD(>&CxGvs3!4>*~P@FyBw<g(E{p>;tUH1@qUAaap30Ao$l|MP?ZJdNM$YLS~?#^xLSBT%fK!&3+>hVjk z|9VM)SartpDO@96F z2t5|-dF>F?m{JFvug?Hf$+P#kMm~?Ru-~y~KaIg$vkamLpkWqVc||Jf=oF6{;=4<| zO<+ZyUb7i>ITy3)3ZC9&upHUR2=7jYQheGFJ6)2Ds+dJ~FK!v7@21Vz#ZZl=GZB+wE1c%J2aJ-KfK z{|M}vpZ&OhAw_FNa8aRI)(kD-HYJR?HqA=?XMUyL-o1iVsvHumtH!wo5+C}WgbGq* zE}Bh_#y`I+#8`eB#jHFc8W@npY!zfc<3;rCMjivvHB#Kb{f50COvw30fUmS6B-`(? z&hI@z((ONUDK;M`gMiTGU!&zbl!@+{d^ebs{(&E?ee(G>Q@L~&d2FO;e8xH2eeE?L z%gMj4%=~K)rYbxah!$lS z(51oezTxS~)6UF?66E?menwzLYm0z=igh<;Y>clxk>^=Vkv%Q9(3=_J62@qLT~yDB zAtJ%)9m9|vy8=cI6YQcMjU>w+vd()%dVqq8zm_$?W?n$9n>pph3!NRVMD+0O`>}9F zsR%6J6u_tMr}er8RUbZ+(!EhJusvY300#vAwo?RbaOPp&G0K%v!%^UapWe}bA;6t% zEuON05_4*@d(8u$aURSiqC-ezx)w0I{E1)~_usK}U zIhJ6aUSYI#%$_LUD`Ok%@9{Kz{)x)H^Ks4F9i7Oh{*Kh|g1-xSFbJJXh;HO`?gSok1qMrVl)&v-vQie5dK*_u|g20oyIGLd?VT;n06E;l(XeCuri9XC%(z{%X<&LUn>%c z${BTWJI*;zJ7E3LQGN5SlUKp6u4LY#?{(9I(DvJ#VJ!B&Vk`ER&MDz&5SFxfPzMIM z4Fm)Txbh~b;#@#Wg;nAp?DOwW94oKn_O(Z-Rp*SYzmd~S5kxbI2;U>K)ePy|$gOi6 zOAI+hs#6!cy90;1C8LoKLvf$H_=TjqYT8X`QWxEZjYSu++1-@}KJ$(8BwiZ?CV!01 z&Sc!~;{_|?Zd=FB$bDo=bnx~ZzhVZabTwe*)Ro+cYdKo7n65mqbB42UF0!kmug%KK z*rDZ}BY|}uw8u!xw<&Mt&r=fwIt9G1V0wDN9Au-+%I5{OyR#qI){20XmB)ad!Iv7h zr8$UpbpyGq0x)dB-0$O!Z@iUUd75`>fN8^NDYEAJ4)B9jAVklbPQPkOkjqsdN?NYY zCk5#b>9_M@A8m339jVx3)3UO z3#*ThiyL(ObGT=#KTZ&Klq?@Y#>dOcYkEyvdv!jw5x@A7Q8b%R&ecKM_QabnkwK(9 z69#KN$u6Xl+KbtTbNNNfq#9Z|v>W6w&mI>@vK4v?IoI|8STjaj9ct?^T%4W+dE>C+ zz#IiBdpYlJ*qZCtiV-%X3feq{&1n94+U)aE$`SNz0PWB$uWcylf9j+zTx1wHi%oFH z`BBy7wWV-*vxds{r0v1hJ4uk!Pw&8=P3ctA&`9p9bex(yr}Kv6-tlvx!nPi@*5pzU zy%aF7cCjuQTVZCyh^5Egsm=zMY*bjzb$R&qNzV#S@(O0-);TggbUz09%{CFN@uf)4 zN7;Qwk?+n+mDp_Lmv&um;r?|LTxCC)|7wWYf443#>Y<#>qE0d(r)3WM9A)u9?&fZ5Yh$K~Zlee< zpP7x@(>9`XND~=bMGjfh9dx?2dCF-z}=DuT%b2#F>h=+M>%Bm{pd`z1_X#Q9iMSb%=71$rdA$kQAS%}>_ z#Bv3)aTwEu2U@@!{|X$U=dipvh%3sUgpR*TNbs~!Qc^mEOfQp_qXzssoPeM+hkw@p z2XtnjDo@xOk6T~)b1t>dY>ZpHKLHGU_U@I+1|Tv9(TKBf7JtcS!i9@ZqOAf*a1)AR zDM(JJtaD4za=MN(35<1Zcp?(9aN1>$fVU^`!;FZV%Pqh?T`}Vy8o6EGt5xgyCOdly zB$iz4AtS^CymXzy_v>x+f$vq+)i+AfQzJyg#CQ+MivAaw1)rA`l42+D4FmgsgKyBS zxbDMdSM4Xa=^6tQlL@!joFqsdJ}0m_wfS_ADhR6QXwtdi30ei~fIvD1wnRh~){obm zrBDA&C(2G;q_MVl;xw6Z6tFwYObiW&;o$Z5r_oAB0buZ5m0T(7i=o373ArgKW{%mt zcS-=?j^tDLM?h_+i%=Mlg#=T0yZ^7KO_7;5_jI7@3q)Oh&2|X-1zhugK);;!lIm8w zdUd=4P5|An2-JPB|1Z^B%z84}MpM3G8FjR)Vpc)lf}I5b?nP*5Xf)K+)pgq2+o>TH zl=mkv5;%LI;o&@R^6m7gwUt@I_i&dV;Her455hiIV%|;UM+&8z0ea}_=eh*dYtWM` zMD^r{oNiW9T?^AA3)>v?Wfo`oE_bs@Ib-b20aDudF`Vw$cfy~v@1GbbfE0bcz$7Bj z!*!0J6UQfXZ;6Y7bmQ~#O}`C^9OjvWG+Mi)KG{faGnSd8LDKBpAJrJC=;8)z$Pn9? zmaddxj=aVob%w}bm3=VRnowdRz7*`2JRg0rWs@g%!&EwscR>r_pV3UD@Xt-h`<;3- zIts=>w%9oU&Wk5-*tb@SQREMg{)dKfB!1g^W{Tg+kLVGvn2qse$y*)=qS+tDgOd7g z;4}L#md)toGPjdJuT=u5_PzP-+qcKza58)Rj%941Z1{8qz%9Omij5>Q6Vn0avoB8Sdhu(vD4`3(p?QbvZPK74;+ z+miApS}zjROF_t*gahD;2Z|`V%g4TYgw%*4HvIg&3JM1{X}nKSyjKyaD|Squ;v5@s z<>NPq@HQ4ne*toQp7|IcnR=b2dvoa0N!MjjkSL6SB}0@m-6=ermt}xxh)^^2E$-@ zgLO?9P>XJ$%FMOOsXV1gKXe8l^4qL60CZG(cwD;0fmM{Pu5ey)E+M+7fT1*An20xp?j?d-Rh*QKFWG|Lczz_Q@O8JTt^GOl)JLm+;Nx`$jza$(^`;T_T z!ds2Jf*(W!=O$-H@KSlH{@=Qe0i&hNhpzMlO&rz@Xkx*WzLH4;m?>mi4wBQ$vX zmqo=>6Fq>LS?CfiptE$mh$5Mx3ut8*Q}Ae`#Q!)YKc`$B3$`3hP_w z7A3uhr6r(h#6jVUfPihf7*!iPpq$R}D`k;LId(Eo4u0}yu4jHv;d8gZFB$MD{Am6| z58%J_Ue(jwTRq<)~t=)}*6j3Y3zK5{B^dI#aQ(XWSY<6#;krdJ@1u57XMXT&|cOy|k!>91u_nL^mity#ob;fxOclZeD zL29w@PBU*R(kB@Rl0AkTOC%L_7ZJFAb~63`t(5rZGRl+-NeKy~oZ!g?0FcfHM+fwqLBHw>v`qjotMlMEF%LW=DwN+qC1GR33sZ`w$U2G&Gi61H07iH zttr=PQ*}iSZjSKX3xB7bE_6#on}bbeW?YdK~Y$UErTvNiuf_fKJoA!rXis@pnWy3x2MF2)^Lkn#9ORDjWm~$nxLG z8q?qUuOz>;-NV8au zAyA+_kU~Y2M29!dA$Ti{9HbBD;!GUGWJ8;&#la~x1MTH_(WM&fiN05wHP7GI|I64! zvpI!?@+KkiEtvlVyj44?jSj(gmQ6TFi1i(|6ftl}g|h$tRxfW308%dj*=qFbA1tO* z-8mwiaEPxILGRB(N*PTTP#A$QQjm@Xjqhx}+Stf2S{x9e&?>vrYt_Q!_&%DbC@W!l zAqiB`dCh8cm7UO8PNwsjT||AmqC!Nq{7bQ1&)y2L{k{j+l~E*Rh^I&dI}cL z@AWold{_UehTOoGl0x5!zzmG@ChTVW3?}eQo1MCSk^j?l4Oj?Ap=n{`^Uyjx^4E~g!70Y7) z(kV$9@sBxofni()T`8Fm@(eQ}pMPR;Fd$PZ`0mi?3L#1^4fmJ+F`PWdez)gvmjJfW zUF#ywZI1-b8^_$+DFFMb)nnGSBltO9EqJwMVx8o}t+=j?Gf0n7+=l6V1HWrgh1C$b z8X*w`6O;2uvjg3eCu$#F4*FDjyY@fJeen`VDgU+E_E5y&FKJ^INmT3B@`&p=0O)nx z8qp!)fF5P9`P4_}yNrzla}Ur9|KOTmUTtGJU|%(I zT`vlI7Hoj#-QrbK!rB9BV;evg7zUlxnktBH#211p^a#iv>9OvE%3!e-Y-!Xd9ClWk z;@ao`F0_0^7%$;SDK9gm@f@IokC9D^ou_Bb*F~>i)yiO}Z|!d`#HC-j-9Zlu5V7+q zUYaFhU^fx~Vtvir-1klBq1~(C9_g|WkN(Xam?a%C7+WV7dWQX%XEx#LY5@c){?Wgv`E!N^l4UOu$EU(73WL|=JgdWDDT;7{t-$dNH}L3PAC~b`^#>wl;mz@IlXs zSRuP+a}tDW9-5m^##V_7^}LHeS+gpQo?uf7hHT12#nB{7Qw_&uAXdtiSq++a+76eD z>pb0h3Z#7xtB_EBy%}=$-x7)=insvd44|!?b(IH8gBsP0f+G5ad~dS=5Bmk!?dSd} z{|?~TfOoH z>#S~$bAU_-<-&_|I)C*Mm=WddJ@DpE3FfjjkB)t9VU5sC#>AZB%+~?&-k4NSAmU+5 zK_rb7QB%&J&d5H*)2suT($6vHbYd>C1^|?ePH@`0M+K2Rg=qUS3#aRV~a~Zqf^goWzOn;quekVK#J0hwn=H(0M&Jg|Cnj z%Rf8-jUc|2u}bnm)W`5z_cg08okzzj6U273-%~!VH3-z%QHB}P(n$1Y7*Re|BD${K z!fO;W>?zyAdf8PJC3~t_@kXh$IwyXA0LGA_%Rzy^7d`WwR+of3LpE22Qag_l?*ZP* z^^%XcW5fc9$AJsB?XQ1Z|5%X_oKo$gTm%&G#-7%qKzE14hJ`*8C`hHl{VbkCKr?g< zuf9g6fQ5~sxS-^>YI`;J=bdLLz|2R1l9>6;qTjg!IFL3KDeTGi6`zfN@q!%!ffoQ` z{|ug#PyZmU246FobbxhqckmF-s_p7lvji9xB65R=3@=OWK!(5?1ibXAYbI?#oc^hv zfx*Kklt@vY_N_qz5-nN!)0H6J0Y;{dH z@Kv|_rYJ2k))vtBzN8HP zKU!@fKrWwI2a1P@V8)t*#0EnV>nCf4WCMz!`o2UOoREiMjj@hA8Gw7ZzNrNcin_kB`2KA8ZIw*&Ij8&LvG)8Aj_p{Zf>5p`RU0ES~@y)AW>}v*EulCd2Rk&J6Pz;zIkk( zO?@U4sR)*TAR&bmwNmF3icn2fcJ`aIKoe(9lJI!vHVx{{$Ym;U$HUv7jO#s~wnfdHqL%_?!3HGqB(hg_ ztb4fF|7D-6=1*Ul#?Vf`o5EJ=Hs1~$t{+u)OC8t+gxX!Om)N9(4wYdQ!EUy(b2uF& zBZt}?^n%%33vqX^_!NY{{IHEEmkP2SedAbt3{z%(f2qjC>=*#eh`k?oxm@txY@ zzf}fq))4mTH!?tMZw8q;>87^79nW@`2Cx8x(q9W&!-n`1F3kR)4>) zQI%8rMG44%#Uvy!8r9s-9k4m_p4T%l)3Y!z*wMnQXI{ij&phcRXP9oYTv^ztahe{y zd2WpaArMqESNSTqP?`$4jJmuvjE4TwfpY2cAe`g&*>_uK%H86#*2Qc_fhX;5eCg#olmr-3w zV(j~74~iU+I|o3MQt17ytRqBJVgQL7WfV3a*`2fu8>|w3{xr#usXs_u;sQ!x$-3wu zm%TS=9unxwUn-}zR<^;)3iF;VI+Cm`oHu|e|468kYV-8T&@)2~Bs@=t%UI!gE^(iQ z%)s}GF^AGFu+q})&)DSiO-A>(*o*A;Y1n;)EFm^>*%~k~nfU71va>X#bsW!!u{K}e zIYPRGLM0#Tn_$Ug5~;sC-k)(0BT$57!vb=j#=u~d^q;f9Mri`WH1XLzfCGMp zlLQVPp7zCAkVblY&r_cTt^f!QJyd>=J{^Tj0n0*I>*J%n8-7HYXssa-p%31A?cqbH>gc$XxcbC2SoNjO*UO_@&R7S;N+L^;|IdS= z+i=j+8*sLuHT$f7Y_7CSlp(1~5 z)xnogwZSy2as)PQ8ol!pE8k_{dz0Wb1H5@z=)Ly>GP|3;kw4jR z=xBOOR?TYedssnsWMNxfrKXf$0j|9hfX#k1BIHf2CBX&?6)soTA$GbBFbde>KZl&W zima^c2b^_Js$?IZ-ESdzz#24eDR}T{*cOf0%^>JYI_p~$*mS$4p@)heEN89b`xP4S z0OeT(=v1EkSK2=3&H+#0cHzv7)si(Pc_B&EIpb=wc=1eE4!TJ4Ly{ zR0s~4-hweY(gJNe!#EwHVVciQV3+;T;C(3NN|4ygpvLpF;E%Yz{wWzooOL7Dz=pM0p*sn zr^$|)Ech?Ysih~vh|BPR)ixna>BCHCG_pWQm3jpvOXA^lE-6{a_PWc)D{bzs0>R|= zmjXpV92w$=m1Lr0W_I-3J8y!}XcSo>hkP)^4}BQSit*k-kd#UiBuDKC61@8$n{OVS zn8*VH=;21+!3#n7f&oavAA6X=s*s^m)X?a>Znh-#LrIL{WZ7!t1a`~BCN|+4TDmJK zwZ?nX5LTq$=I5KTW0-K3zx14VJgZZz-mU0QH_-IFX;b>YU#6s_ym-bTLg-CdTc9-#6 zUGT^B;GrZiJ?0Jl=P+<^uNt-a!~yYS=-sREb!ot*)2JDWcma%{;;r=1Rnl!yRPT(nK>-?kV4vuD)<$k;f@my=&6d64CM8X{9NG3LJ_GcNK3@ zmZtuoiDpLL&xNLt(#WCmQD`*tZ!JyPeieUoe<*1XnL~}>MR83$@t?{(mYFruItc0T zb@4x`R24>M_kK{U66FOY85>C+#kq*kB&ulm-+)q;I+}y>%k+rW&AY0~NPn@T(5Xr> zPc$*p35`UQ3_KWDgv4br)8~FONcN6^6PrKb#nDf5dPX&XLgjs}l`>qvk?F z18lf>6dKX2mWih4k=YfhxPKq9(ih|*zEev6$%*p>ybhOI!Uj?`Y#Jz_Y?Xw`KO7oX zE%f(!_^_r;eHKm64bKag97E|H+S5cf+nC_803}=*}QXTNt1U&)IaKAD5%gP{I7AO>51;&1w4(o%m-`Fox z+>k3&dy>YfA$f#oZ zyCQGLLxz>d{m1;gx~#1KpJc2yH-C7Pbvg`;7L7imV&hr(8i}nO913G%VpPmiZfoJ> zT+_=cMa(f9lp#-~c$v~kHBEK3tXme#O*QfM$s7(`{i5h6jk?D1s3l zaDvfX2tIF*ndZY*4&cV_H&J|j8EHf=G(v|*xq&kDYGd!TDpn*(a^1%U@;&&y*D2&_Wj@hy>TZFp2hGKaarmZy@I2aHo=~ zo;xYW3O;YmXuXJbH(*U~dV#e#LF}Kx1`n9Stnn^3prHsrmSqORY ztM?-&v~UU-q5phqV(5vHVL|+|nT4@3DkS(C#{oFOXMvh6kCSyc)jnd`-29>;DraaRy$|OY@=*S(2V)1^gC5UwxfG9uR-lm>xCw6cl2r>DQPnz;PK78WCmwXjxxo#^0b%ErnClxi zJ73>=381pB17x-hl&!v6d7*k}p9~{oj*fu2f8q7*G%1Quad#2&)ZcYG=KX@H5bnH# zD;BCjVJ!=epJ`_yBY}t>L9QY^IJJCE9m`%d^br!-pkNX(M2HKjZQxlErMx6c1?vn( z`qmflxN@2)66f69+@8OBHL|)vT#_v5rQaIGA^`V%1VeN$k558RQNoWcWOe=lveown zOR;Hh%Mc%hvLZ{NBTn5eaQ8{k!JPw?3S$+d{Q1EFz69}0W9iZzv+}#KB#c5T0CY&` zK^Ttfz%5mUt9BLUP(7jX!-U9;E`Y(Ud~IzhMH$NDUU5pwJ`)5j+5!9s5jx*sl#m|+ z{D5;7#^!ktdMcocf`lYHa*E!(2v=7qj)mT5_ftCm(HEK2$6yoxQ?{QFC=@~-|a0TK#bilO?RFh36C<6 z36%M<9?hZI7(G35eYfA{GLTV6R;iv#ZR85jKsH7T+&Ld$SLcJY>|FtIhX`Qhn2Xj> zoWT}yraNUL2gSSxbH1s@`I7)g1Fvh*MG4NSBgbB^U02^8`?cLIeU{}kB^->lrpTtU|lVTfd6gM7>(tb`>K18Bo(FQXyLty>|ve2A}_@8u3)AkWX7@k zJ6CO`=@?hE2~&E*N~DqX#R8Glqq|>J4tXdHu-w?aN4tX-zg`9WQlwhSGy@IY6~v*{ zgA3U&%p&=mjdPJe`*O2Q=F7mWN5_8#sWjkPgnQ8`(<2zlE65V4r-Q$&Zz1VX#<_^s z1EDHnMv=QB2Mf*a@#SSX)`4Q>RpcxUkR7&iw;G(X!aES9y&D!s&Amu0>$BzcyN2Y2 zL*x*uE=IViPuTz?kBrLb2PwD8g0GX@U{gJ+dgM-Ep2Cu;0uVL;SI>kZPPXsZ< zg5X(h4@R03m^2Dv3q8L?A;EsliUn~*qR>^-{Rz?g>W!gTYO9+=il6Qv_KF*(j^^2~ z_&5dZ(+lv0U2=FpDPrHup*5a$a+~D->csAs{HP?V?M_TRKNr}^QJF?>Y%M(Pfb2cG zQ|yRO9S5&eldC?4OdV?MnCVwl0j`L&LcNSAv?x2w z6i#<>9h5OLE&<>(NcL$OcOP?6=pa>Expd<_GIzbe%k0?MGGG@GDbUCerkL}wA0aEF z19~aJ=(1;|%zGS4=suhG9e?*=J*Kc{681-2COZLq`H_ z`%U+h85x~75KCzkoIX-&6!Ygos(PB24`~|oG7QgK7!jIcoI7A$K^Nrb)P_P9iV+$D yHAn(yBhIJhg^uU3M8>ZgFKZsDF(J+|d0ZNUQy34n09@#frK+f@P$*{^@c#gkQmLl^ literal 0 HcmV?d00001 diff --git a/doc/design/images/feed_forward_regularized.png b/doc/design/images/feed_forward_regularized.png new file mode 100644 index 0000000000000000000000000000000000000000..677e99bfd9f8e72ed9fe4b27127af2ced202f447 GIT binary patch literal 46036 zcmeFY^;?x)*DegWB&AE~Mj9lgr8}h?q#Fe35@b;VN=PH!Ai3ynQ55NJ>F(Nd-Ou}c z&-?EE9s4iX_i_Kg;acmu<~8QH#yHP&oHIgAMHUmC1RVhZ0aIR1N&^7_sR983aTygE z{7dCCVHyMkDg=3{SFgQ{cQequ2(-@6RRx8p$t7xNA(mw7$gibvQCuj=#B*eBg^<_> zn=^FNrn=iCmJ+|lEEkYw8azT&)dQLT|3FrV7uAbs^I&mcU|E4@hTr&1Jvyw=3 zIJiiWHW@E0Bb71)2{E(1Vg@*?H1Mbp@u-6PP}N{=0dWXu&vQIRQh8svEQ#widzL>T zfy^r$4Hduo*QHpI2rBwWueonpRY9hm)ca~%XVSvZz$zD0+C|iA3X#gqyOv9qxjh_m?9Pv;z%;IOP_7>y?AG; zzKQ&6@dOp=kLYlWAh&qs1eL#dUrrgof5Rz=h^~_xi!&>Wbd8HDSF6#o&5ywPPXOeo z1i%tIk5|hc9A+0W4zg$nroA0rD0XTrgdUE?X#g%z`fZ|_75 zV)n9#s3_l!sc#n3uZV_+?k`h$?ayCk{7{r$3*u6%A=IV=Pu8n$nI$J~eG&8_=s1*3g%|HI zHmG~~(&zT_$jHsj?Y4Oo(padP#qPfOV}9?gpep|>_i3rmc-bzzTOHq~=|)k5*N2n&kM__{)|;+2(qcC^HY&1bgRc61u(B1n@D9s;%b;5# z4X%pRu9GLIY{SHIm*x!~`}FBkJ*BW0XD9kTB4!NT9P&)Pqk%fOS_Wx-j!#>7*$F`f zwd8P!Jk>6QhDM*sxLfo`*=Q&PY{aSSYTup3kMx&E3=VB3%9)&I66-DAClv0@^AjiG zhfxsj@DPMB`{zZ~N_ePZ!BCJ&lZV&>_B_Ohm-QPP8=j4aT^J+dhY0#N3%&>P+S)|^ zMvqMU6I z8(m6`lRXGd-3F{$=a*!M8rT93I~ z$oOI)EjXNcAc@yo=n-Qf&kbL4$%&(YloS+ygG+~li1dL7Bp&V;+_K+)!KX2z5)N5R z;I&CL|1u?qS0vfN@$340)_Hh6*@V4e-ZPEW5T@q9mu4cMQ)$cw^6L1vBFCV|q|xSN z-N1J>y#U(P#}n|)A91y;>{1Hq;{D#CM~()O2iJ*gGWK34eD>lE)X;l9@p#6*Qr^L# z%tYDi{%X_m@95VTAUjh8Tyxy^5|}jCmcuEve=r{`kS~*vsuEN#pdMYQi-YT-Bc~WW z5kW+s9Obscq{}8oY?rq8U%tNHsl0J?ov(LHbC{}nh0|AP{WDX+#HszxQ?JT2*zbJW zx=!q5LOxQ*IrP#u3IxCI5~0R-nOc0AgyKh87$k%qC(g!E38t;3C2we${=1^9OM;1= zJ!xo2VQ^re=&rF)#=`m(ic9UjMaEKcEIq1At%w72wU(U)I7Aav#Y1-t4H5IPwI z!BRRzF*FFPJ}F*B$qmA-4z;NDk957+JWZ9C(tM6HbbGT6IspY6{$N@8ChY-k$AgTM zw)gUk8FWJL_6eHj_ca@y8~bIc4QU zQUB|K!21)*by=KB6aTYBI!4Ax^hw4nKaJ@;j~_2L~t#tQFHhRWvGc9csP6}gB{?aIydgn{#*$^9|0?RZJz?n0}< z@3FBRRGrKEF0G}HLi>la{dED>OvbI4Vu8+FAe)Ft)jSzrfGE&*XViTP;zw$NYCCrk zww0)goW0)rz;+2Ue>8EvH@klFQ!JT6#3!A~P#s=y))yb{8%)w_ggw5jjL-5c9kBhM zYqN_W0=+gs{?JlyGHkY(&xL-?_CNY~SLdM)hj3$Y`~kl^1^{sHla>Oy+$&UeJKa zTtQ4P&OkPlV9O>+Abq9Ej`DK+zCt3utniB@lBxRzrc$>0@-x9ohYIzcC`NT_8x&Kx{e66=7}o22yn=CkU@}$D!<}Xq+)8hfj5xFtTvU;aXXPt zx>QH%yQvY9D4I?n_9;TeYJco-JgqDN7lzLJzKI+$Z`V|cpb<@));bBCji*W0C*y^(}3z&k1! zJwBo*5R`2{SyAWh>NXL>zfiOWe}D0qG2}^4>b`a6*sQ<)=muYh$>irDJbPWE0%mqH&DZF$s>cH6TrwlX{l9)7lm(c?1`&_!^-Anrc z3Jn;Eo9@(eJe!0dvz}6pRjFUw81x$vNbqNH3F1^zR1`v#*V9WeM+;*KIxp=fb#9@4 zkBq5td@%J*lD&EN#k%eW=+iPzZC6(9$0Y$}&lP=37W!gH*FPz;t8Zub@}gUn1~A{F3N5|pQR)2S(TSTE1ua63vNj^|r%d{1%vDRS|r5t7D$^5khYs;RC_ ze=KEX`rA##g?if&K6azGmyif) zxjUadJ|5(DKso5Jd8fvWRC_uO z4%H*cu)xwLb-YuYHuMR-+i3s>vQFxq7mfcs9XSB`RjyhpNV83icIsNUOU5hR|J+cS zOUZ!_&5Pa?^+@5@wnzFGY9F+}42HpKgkMfJeKJ4ORdf-GmQDFR#$fZ)Z}!+J zK)Aiy%*qdZxR12&NPp>`p!#sXc6Zo~z5X-szD9#+?g`E#60~3FP45mWZZn8~bY){XMp*nw+^NA)nPqrij56KcAyN zc;ScS_3zhummdkFL!40M^z`&TefjeIr`Utv9`qHg?R>@_kCb$F>G|=%Wl{_57#>M zAMAG~%IyR9RgI;d2jK*47Z=)9hFe~=-R$F&sUR$)^CLMHm0ic=yPOI7eaE2!ogJa` z=mj39KyVdQCsO2icX%>rOx+mYXi8J+(aL}$I&OQ*RMYlJE4Syg?&m- zXMOhUnZi?c6a28!%DgW>O+Z(=$Fm5)M(gZuftpj~8b{4DUYjaB^}=#^7ifJ$?<4x5(ho&jcfwc_YjU8>E?%iBZ2rckP?Zbq1c{F`1FNwuMgSxwk@fM?XEG&_%l+yXqe zP06vrfS85G<2dscPD1`gX|8BKTeke0(g=9&>edgY{@6$j1&=3>963T4%n7z06W>M~ zKLfI~BN%Xd$^P=?#7!gsfNxWT-m&`|fcLB14 zCnP*Yt8XbjtBbXPsQRlEo@n5{G+j9|8^crq6?ExQ&qloX7myYoo(Ysj zsUW$K7X9x$@{4?rL|MSqdh(~_WSvHr<*rm13}*e80w=@$ol3nstoge<^3Ahf$j zW;S-B*`S`(dqH7mbtchB_( zTTWaRM9t^-8$CK4ZzRL9c*;lh;}B1=$|scGD$){HeTqGK2lK?hn&0r6Kzy zBh7I!)PdJJdD;#Sst>=d(CNgkKv*ene_|_5@HM2G46DBiq?E}-i5APdEKz@R-?QyR zw;9_)c^R1xwRYnQK)Rr3XIBA|L>>&Ldwg;>$oU0t)-yG;{Zf?$|^4K2N zk8K$;(KPlb0jbCf-0TA-4c{mw_~U-m+!&9OIQTO@KK?<~v_nIN$Aq3%cR02)A5e$3 z0G;S)ooaB-Dl=-<)-Ka^J&odMg94(iI5a$*MF0|rJ%RWW0@`2spu*)X`OlWm0R1Ul z{t${u6P%6qwrX|{J2@!{s~O-I`|6sSRWrSDp-#MJn_oFYS0|hOfLT?(Z8XG=98mR{ z>^=tK)-6rP*x_wg?{_OWORLAV@RiSYMf&W>H!H@doVu1N?Coag<8zJHn4wbWFjFfp zdb7(t-RNF;3mUrH-%)BNX68?TU&66eZ7q-OSV2s>+Yr<5I(wa$7jy1#Vz}b61et0o zuBuYA0{jus4%W2Uy*3jmE4Mgse;NhwAkqM_Vbg>1toGY-nhr-4$y$i)BcX`zxv=d* zi?{6=TNzF%=;|$N8yn0aXE{;qpNw05fDJw%|(*<{$sBOQ70jOV?h?yVy zw@mfG1|^Z}d$LBN&N0&{60_=30hOs@%u)Asme&fJUiF)ZIj_~27jFh|p7GeIb=XXl z>Ctku-&PBQXLXx(Y#0RlM;KMc#|9LJzUXJ~DPsuY3m;WP!?z3~37+(^Wb7nK64)}c z@DG@@hzAg$k5Sl8R?L6VG4}4ovM)+CHG^x3WrmF!3`$?JQ3r|G^u0Jn`^jWKlHsDR zzjWUis50w@H!cs>s;X^c-@UpDKv7uX^=-1>JS7GqCoYC0jHxP?#5yPr>L(`8QwMak zOGzp=Hb=c=cB2a81}CGMjghoT zPEE!2B_Sb5&p1sxe$6+(YuA4Lnuq4KqiG{h{xI>DY`xOSlHhsMNNpLO6O15bA4?%p zF-~02;iXdFOP^{lyB~jMHaLsmhJ)7z1>E)G_ zqW}E)1K?~5%#c0!LCE-R;Pnxq;DagLD7SPh-D-dlC>3v@ClzFXG0JFl_V` z7{IlHFqCAY8nWOT={GxtSpg+}pJclp{RKou$4_y2(k=lKsB<~fb~a&9m!t#ZOn+&6Lts!=f!k-LKG>S`7t(7O|XGUjvCOWc3HKX0J(;Bi?Rc=}7W8fb~z zEiEl8{x|2nfZVjI>*`*XLCg3N8Tq&r;Fxewye4gfamhuX4RCRN9mfHd9*fqsPl`{> zY|B-jCQV|J@;yH?84p#{!xld5P$?X{VmygO^%X~=rX5v~x_4@r?YDaKhA#RkSI+df zjtLup#MGr#0i8fvNSrkGRZe0v@O%pjFClyK4jQhvg4A*M1S&`%3UG5b{kWi)s7m{b z?GJ1Gi+(kp+(dllS~@zie+tJ*1ut9r`uZlOt2}xMbqLB$+9?X{k3!j;5w31*XoFT$ z4_MgQDR|9$pucx_cNswSi57UzW>CN6wYfT+#HOgCtZcOF)@XI;wF$*m{KE&jkzIHu zIY==qzBI?5fZ#c7N9bNYi5);RXmH~77P@B)d`5Kxt(g$wZt^@HyVs>Wan< zlId38cYD;2L48sk&ykMOZZfPeKw?Xtj1Q7VeKWmx{`fr#wn_?@g$z2M?lTcHi%I{B zd7WyXvu(WO;V_G1x0nU1`k#WJj+ApJ?MCpTrIS%oKD2{=)t_Hj$Rg6R`ca6Eiwidi zd+gL}r6)qe*qB{a^eXT72GAbiAXm%Ma-C9oO?K>UC?-83IU11fcT;L>YfFhajMFFD z0|Utwu~gM}CM%QGb7jDwfUwmQz~i>i4=BEWCMA|h&NPD%qLrw)xLnWw{>nY@1I1wV z*N+ULcSBhZx7M9~5o(#xe1800-M+q0*rJ!>O`g(I^6}04O{H$jU11OG92{9i@BCR~ zAx~76-U3AJV`a4=ub9m47!s#trHop~3%J-SKo4R<89z#;*6eG}!}j=45Kk(I+z%~T z%Y12r}5mgHr1_&cu#F=rnSNJ zV5Xu8y-(ggf6^q-2)w;jM{ZTnvO4X9Dbid3ym*J?s>tco;ByNv$C+ANonlu5pn3#i zbp58v3DU!-!JumSnelF+`0a87tfCSghmje^`ZS0iolC@sm9IV6MLzE9n(Haq;KiLF z#79)d9K`XMZN~k6bW89li-|lJC0b|-;0&Gz3-h%W{T~%Z1RM|wEig%Vs9B zvB-*0mw{BIp8TSfu-a;bq_bFm`zr)QQ!*6NgyHq5e=#kiqZq{6K&M=PoB#z)S7do9 zow7U49N{G*!H*Xx~YOG_)i93`Lkm~dDdM0;TH+0lbQ4gs+xLnWV=jEY7O1IfZi!~W17g~Ke`dsUcNk^W*p*L2zg|CW=O2g4=e_GX9 zFt7LiT;mJ>>r---!Wv&w`a@>CLm>Spa=(W}%TxIRX}?Ch!Z0Odqch~&+bs>fW^J*C zZ*pily56oiNp{{Od=9#UC0eBn5d$<9SdR^)k%IB%l$8A5E_DRwy1TpQI^mmZYil3P z)Y%g!B_}W93%oLxNAiHvWSO`iM6{*R;H|O3#j@*=g?EeopAr)C?0FEiV4`P#G$Z4L zy$-X@f5pVc>XbEqMgR5wFA{*dFD#H%VhDl{oK#ePxKYTMc0>#?JXXxM2C26JY8PFK ztE6RBM!(E=l?=he#Ds)X!g#jMr2ZXw1Q5lS*m4~~5dB)4qiL{cpZj%=Kz04(loTdV zG~LS+HDVb`y9kq1$v7HqejOnH{H!71x5&fIThZLojoBJ zd<$1se~h=jAd*AEr5go>=3SvuiW@Z+X2b+UC$Ctuu$O%JJa;CPfyhcf-@LQQUVyGs*I8U#d_38HuXVrpa1%Hbdw2k#mWGPx z)f=PhJfdp<8&6a%0QAC-X?xs7faZk-YMW=1c2AcaRW;R#rFeH%)dsRhTt4ZFDRkq*jPhOVmFGq&s&Ec6_ zSgbc6v}vkr4*mR@4E{2p@ZIGyLR&teW^IJZMuo1LfQsS^+Nj~trX>v`3y#e~%N5JZ zmoN3+&3ou%?2E%P-G97P5MBkKYlp$G{qEf0YL)WgIRJ_IEBLW>Wjb9OBabnN2CTFE z3qos7_?PeDxjPpU5@LVXnsgmiT$JdPl$8-k5fv2`sYRBRlzi&xk?QgPb0A-;W1)wI29co}w5lN= z{`oUjE7!WHR~&Y}J+6pFF8GFISrA}_USbioW%M1lQtk4Rd8c=pxJ(#Nm?~+4P2dU6 z5EsOS_Gjsd3$Iyj!ug^PPviXtt7FPk(KtZv!VYi!VSInG7WqPs0`mGhs-Mu}z?z&F z+lIl6DKFipP&pWV60kBBZT&fvDs}t-DM~`se|rJ8X7cn6LFj$FffkrwR3zsrE8fNR z3s&$ow2~!1KVN~O?CP478%(gguHd3%@17@%^t!rL≻!9U8;XLdRCNnLgTQW9yiF2qn@ndCnD^@{icj`3PH>^P2MEvL9{!@SS5~ zax%-lAZR?M8P!yr-v)65w&Y4&dVlsuKcmGKIaSNJ6}Wx$^=~a&j(ZOj_ZUp1#_wSP zjhA6SCLZ-W!p+8aZLOAk z=q`B>-x>Y-o~j9fPjjm-7~(P7pNPM(4S$ip?Ex4pyaOhgnEEi$;{nUD`Ze%o$grjMPYr^9xL!6>%aZT9VM7_l*quCoZ5 zjzigN{zzk>k3=r+jH!wl8T`~S@uxQe?uD@n02@)({awKeT~jW4ziVOx7g~}y6d8kq zP(!&(5GYijy4mw#Ek0*<^bq1(6O+I@52JU_xl6>wHA9F~pyFc%${3z|Gl?IQ+ph;H zqe`3N!34$Vcq7eO0rb?wl_lbUVYY^Wp+pw@#|7FjWAIW`dmWkEPL!n(5EBnC-kh9)v}d z<)#3lMm{h}{RLp<@jgmb{mPhVP9l%b$EvmATK1G zLDDka&9w)%J1n&5!D)oLOEBV1A11y)F)Je?SYWNfqRb6&$Tt4Gb)vi6PD!UT@t$`lMzT47zHcTCiiZeRAg3cA8 ztatf#QLTi-kTh~NGRFMjda|8Bar@!^LTnue$9W6jUyZWLvoMyKu$xCJC#6}Yz5Wk( z7dw|fX|UNU7Hm!~{v&vLp`8v43{1hSe^RXfArGzVEQkiiinu!S#Ww%Q#DbbOdsP7-UUV_(- zXey{C9s(?k3v}IJb#}kyW$vi8 zLH8~DhrAzIXdW*OOZo@u>b?N2uM~D54(s|IIY3!rNid2 zhYG0{!VU%pRKOJ*g?Pya_KbRs@Oagl*w7^yZyPJ;=|iz>4^0`Np*NM?xA;=&C?fFI zjGJgT7vp!I19t~Tm&MuMtPY>Ulv@9%VSx&wFH^`{NMWU~kV8hO5V8L|%w7y`MxdZ+ zzdepRE}X_I69y#j^$M)S2?^Y*^H;3swM`!M_Btj$0#f#>V`oqgCnkpeGP%n`)`PHP zKOnR_#74`aykyd%<~7h$2v|x|P0{wo3S;q}iN~y>FKY98mNLJHHxa2taC$V=5f)0k z1_!=Z%_P#|^JQm%jI)?OMQhu(A->he&BQ?Cn0`0sDx0x?w8m<$4?9^o7RW(Rh9H-hP>zVujxEss`Q{^sZsBb0KW+_KnV$>7|n}I(7FgD>HuZ+ zxDI}VV%n$`R(`Gcr+B_4-)Z+6tTypIiYc`WDM(9|rI1JX`(Ws$kHP+8DZg2(RD0a1 zp}82EaE0;tN?u$M;N_EG$dmt_t||F7br+!bSiZJkB!OMkDR(z+O&vbA|LBwR!|2_- z7F>J}OioT#<&mVElEFI+7Y90t37`UlC6v&`tQUV;px3UKhi28>|D7^st!FNfp&M*BzD|w_u{8C#%wbo zy5BNYrX^Wt0NUOpA|Nm%m3#2UgrgL!xFB!T|FGL)Y=~b-_j~j6-RXj@-ku2jL>GZ6 zxsXs4S1<`IWoBVHay0C-8(Xt?XM9Qh~! z^jnQVGmt_$1B&ppgD+^cmYV)X-&~Qt+SNPp2K?2$T#g~-SHV$7Kx-?vPIvtRMrC+N6e^!xXE-LK-7{Y5GOVuMBGT&7A1oEH9GkoAGj3_f16O=dJ{&lU z3c4(H`~q!Tow!thg{9PMZ>A1oczF2W+bLCzyBR&0@oCSMMMIGOBO~!vuqa3f_bReC z_Qa_p<47ZXBOC8V9$i>msRxB7BM?0uQb~vCj_V>oHtX)@))Yf2wqS^fol^%C;@%Hs z|FAlo-a`#M0WfT1VSJ4CCA?!=fQYbgjA{)N2yo#EL5o1U(X5O9gp-M~{{)H=VKgcS zMf3*4`{hdLa?QwdM@u0&swWT?Wfhf$CcE*H0;sRu+39R28mdQl-F^A^&G3C#GSFwd zyleRIP?I{!$%Q=AYaD0!d_g`ZR^tp=9WHgkwPQ(LV>4UnAUBA#tf71!&{umLJb;_Z zIRq+9&I&PBMPm4G%p0=XV5i?6(E8*lCxYm1a?}8F>G|ioNHU<74}7y8p~Jxm2J)pW zP)gHxV?o+antp88Gscy>#SXlbl=3%T9=irTV-SoHbnR`WZ+?h9+~#Hl-e2W_F^2D> zZ5}Y?vB&vuO2Rg?{i)no6wngLQK-dhv54a9l6I(s-k<`|@!=^lUkI50#U&-xmjT=| z2|O@gJ!^;*IzF%gsj(;6<-?G~0E-k>5XO5?TB4b^gY?M+j`K}t<6uUEdw=$+QNH8T zf8-JLj~aCfsbYbmbBX^FN7l$oPowTQ|C7L?xItcfzw?L3;p6v$Cd?&?%cQPF!p-Pa zn<=%-sxbU-OC*QDHN(`eQbJuhiUk8s5zeK*pnaO;+ML(dA-*2Ax9aYlGeiY73PQIua+X{Vlwe$Cx%sCGproA3x>^L^eGXTLQ z5vY$7OxVR3^J=@4uWw{?(mSSsHX{8qQ$!Vj1LIVmuB0Id(bQ&Eps|~~JE8sam3BD!1KK?U zSX>e)E==r?kB*Ks^z@iq5CvS8g8TsbsEl8( z0drlAtaq3WQT@>U?3j7Tny&c(J9sKIG&G!rh2z)2B*EBXJ&)Gp!TPT z_UH7%$v6l!U)KI5pe^ zWI`>=(Rvp`6U9-Q=L1Y!U5|=zX=M_utgZRCrfaNqNtcSN0d5!oQpZOQgL>ANyu2RY zhSyU&S5s+4S=DC|&?k$?eLUy7Acu#(u-e+UQ$Pze{+y9PZnB6}D!XWAW>jmVzJ7ak zV)5zoXPqWPc3qQz-?kJR=@x$frG)yu9 zH9*y$y2M`W%`yVMl}N~h_7oU!gj~QLDBs#*5CH^t4pQ@^=wilO`?GAk; z7t*%0vMMn4JL>(`>U%N4YmV9m^ZUQd0yc5tgVe#ckXX`9Vd0^&1+#{CbM0OpttQvm zjStNOzw{KC-{p7UVX&wG#ej~NR||~6@=8h+BBP?{#l%cZJ%TOIz%Yf$8)Q=H$;pYM zaw?Da?AF_gwGYvwSWguh0ie5{ai~{J<980QR99DT0I)Ox$d5)qbTl?CoE6 zfzCT6=jGA8Dq;&Q13eo*c{pE+EZ820Pq#DNAJmizPq1{a> zw7@kF_y6{&o2AK7)k0hXF1A`Y$^VQ1+^*SaP^vQ7|6>p};SNZhp|UdIkxEL+4h|0E z&O0-N#Kg%anwmBJTW_6+HiDU|6fX=Vh7XHyfBnvh#C;aRi|`Ym3mVEEyIbQ z?HVJPkU*l#euP+IOHNgdx(Yw|us>A=lnE@PVwx3#cxGHC!uSmu4Jw*J8azfCEL8IpkevUtcTLkDj;zj~7Of6xq#in#sPuXVgD-JQsB;-g?y?_~y%qUFG< zpt)Uq@#1e#J0!T(rL4vpcwao~(3cmt22LKqN0zSfi~y%zahYPsbv%cdetk(t;62 zVZ&_9(}cw@`Fjq9>!@Em$)EYt!G3^)3`vfZlW(CPlI#VgOxbjI(Yoq@#UctoH>YPJ zK5vz*tgLQRtTUOw;UaG3Nia=jHVL@OUt3?-VmE|UF@W2p0CZrPR`AH7h#vKeKe#T= zKi8eG2iLXfPp}uP^(%kfqz$Y8AO5N@@@0HHRJy4DyTdBK)57G5e23f4KwJWE%!(lY zHEK$mzu}k}g+#>`=OcbSkGih$Wfwn@e_6X;GVFw0Y^Y?38g2Z@Xa@B=b6!EY=rF-_Y1yzy!I`M%W8`<;yR zgDHy6s3XVE%hAD@^$&iQ{Wf#pM$wReBriMX;Jl3O1JF_MN0%^!1w!}r@v)A@OyC=zvI4H<|FOI@d6e3NdD^9(o4)?cM=%NC2Yp?Lrh2-koAP&o?3f@rlN>pE z!(YLR#3u(RZJ&-m#oFsL#t9POuiMYycNQ6Mavd%Xj*RTNEr$_rbGAks_4ppNG6D@* z`_N-AD>E}dQ%`R?5s;d)q3_?7T7fIq0)P-U&^((kB|ujH-yp$cISmE@f%4A4-)_sV z9{VhI3)p`2>$0L1wqmAX0`>$IwyIK%KLT^{KL3|nl8&Vrt%w^TiwvMZBW-YXvK0@a z$Q6T#fyF(~o2lsN=um*ic$xumrl+N)CGhZOb%(pLz=@o}YRln|fC4mht-Fa()2K$s z1x%5cjYj6!l0SQkYL{kt3>IJvUjU)0XvG#nXV_BpbG@<50oM%QY$VD_5nG>7))4zkAd0ASSbevOWBV;VxYU~2;s@b*pv9KaapzHAz-Jf zvHt!0cZG3_m#aP)1$1kbJDru8v>VIBkZ8Sr{rYs~SFWt{uX58>VuGab^dqjPxnbJg z14jWkzpe<5d5ttQXf`w6W$w<@DQ^9ZBxI=y)&=M~AIQ4(j1jf90ch> z53HOyYBXAnS`)R50fw9RBh2VJ4_;v}F1ElC!L#q*zkg0*H=0>%kR}|IB7dXRHMe}h6sg+j}TyKeV$1wk_h(MLL3U%Os-m}aFf9@;Ej~9>>e)Oy~ZiR5w95ilHq`Y*u zdS9L{=usjJjww}xA$KD%3W0gLaXhu;hXG(&S_1~GWZ*!P17bODliKvuStknR0PmzB z?bTH~il00<%0+DK^$HC8bq!R_%)ZV6o(Vr(FA24>;Hz%vN^`9-&LlYYTw=X=bb zD{WOX)&wsc|Edm1FOyISkaLj!PfG*+-0Pa^>N%$d-+wt8woe)IuwP=hW5`b`-NI9x zj+-W1Lb9oV8X1=wS^F0dKy-1xwJ~VH>`o+MK&fo5HtR-i1aV9>>1;Lq{)G!Zf@3l# z+M2e`N`%X@FGG=6`1vm;_fWx9tB^*`?6FLUc)&K3ln_DIao9D>*NDm5LbG+j3y*5w z0oi2|h%h;OYmZe;Yn{`)VA``gJ^wwiTVo(QARI+DZy}pha_hCliLNR2(X5Q{Zh>IG zhyF}IY>ndt*8G(k-yFfLo;?D&*rN~SU z8dmu+W6#MTV^-NMd0H1--kI?P)5ep5aXaZR%J zAj{kP3Q1d40A>mt3;ha;ioVc+2BKXKaEc6hJQVxo0`?Q0zyNy%xyjRp#Q>JT+f~Z< zcoonW2rQ-BJJ;Y{WsXN#!f$rpR9XMx%nL{kbNdZ&2_`%MFINewbsh+kEPLbHry(JY3@S{pjJ5FtM z-&~0a3?JLtbsECrS@;WL#X7)}XY^tg9;-N&aB&RmtyJwW7Ab~X!@F;~B}9{f4>yZ$ z{p9cdoMWdoe83_v(bZWdHV}&eywnBL&oW;1?3=XcWk;6+2BL50W8&xWm~pjju$XlW zg6hi26h~#>ZQ{ws%G`M`aCppL;95(;+4}M2EkMW!TVG$l z(9IT_VGli}w&v2NKKPhI;nqU=^gICPH*Yy+SeWX?d^7*8=mRhx$pS?@VN`cY>2dq( z%R*5gcI3>DmZIxl7JMXoS^a9c#JB-WnNm2-LfjJG-JK55#-j4scHVVN`h7K=veF;R z6mZGdY52ipDB1X1IC-9%p5F2VoWlFG^qIhMYapH$by!1NL*uuORK0G|1GWEv9#m7e z$vg*MW%^nvL*FWQe?l|S=yAE*6nltxWGGyLxyQgf{;350&xcSJKZ65M=mpomlIdE* zZLBz5I)<*&dk5c`Fjk&bAOj3i8Y6QEM#cjGK5o5hQ)6|*FSk{l-xQp5&CrB>Fc0I^ zu=@X`=9!;DW=;}4rQIW*|CX9_p!_oJ$RJFGCfxoUz=JK}QiAO_e>fVXn$D+^cR(K| zR;UM$9n_qc=kFcWnmq{(?!Nh@QtSW{P@MbFqSnFrElrgP2@QXBFrbIqCjqK>xt^ez z(2c&bx@tc>rtd{97fUY0;6fhMK@JQyj-ow*7?` z3!`6O$49x{3Pa%UJ+{sLO0Yb*SO~;F`-_9chvt4isHvG-w<_A_YapqF7@5lvQ5D*_PVi%A-G9o<=PjB~!pMp%({Rp43=O&wXBj3i3 zdpsKcuE^2)n)I)s?@{AsX6ah*+zYqJL%|C|<*9uXfaYu+VflR@v3gryV-E8Gql#ci z-=xapPd~H4>rrW|E7%Mwv;DFZgebIHLsGHzsE_o87bgu`Bwium*9sR^XF1qPlr|Wr zCmx%6gguwh(;b7KY=fVe!>xm#t~FH=$5~yxZ0=rG+kQoWn7&Zwxu-AvoR+iM6T?8%qLc1xejM2MaezH;si3FyY;=PB`R`Gz@p077gWR!UOcZP1V4 z0)SV*%P+nxW0Q-eD*OMKR7&56SxGI=fMe!5;616{4(vLr%evnHuqlORBVPDQxL8Q- z$QM<2ZmKI0Zn|;!GsO1%IgHQH;{z-+PHgSgvAs|?HghP*nOtJYbbjgUX{kt3WB40Y`!C_Ns-aw_o@OBwj@&!Byk3nbY1I!j;F!C z^{ExWe25&Q>BDCFJq_^@@}E9LtUSZkiAEjx*;XZ8fzSqE>GSqINm#-d`%=dp_wm{- zMjJ-6)cT68LY^;ePZykaF1xncQCtiee4mEH>IB@03S6KDKG>Bz&!iDaR4-u~-smxX z@$qk**m}FNtP6~#Y38dX9FtFlRdp%c?m5*cop6||viVLW!cp+cMD-~J{=AQQ_L)Ih zV?y{h=#fRjbEaQqCFID331Yq|9mZkY+#%SvfDz!3i?`3wXuzkwR*H*{F7)pgu)FFD zjx6!*F|syy3Yjzuo|9ICp_^S`*d|%m^OA71KkdltOL2&fUEE|pL%@sycHYmav`kli zog|F94kq;Nb!Jt5Bg8{j<{`X$#W%0YPk^^kk4hmPC-n&~wcxw`vRl##pQm{)+xn9B z`ah3sb}a`deNOc^MKN?#U6)&U&w*?hqDqYO$%Qv4)=+XVtEIQMH{|y64H9lq6MDyr zo$0cV2f!oAU^ZCnnuX!4c6Te{1O=8fwTj;Xc3nrob})h#E#Kqv;~M7nnu~vXexL&p z2KsmGY%9c}d(@lus^g?10NqrkyYBD3AuX@6wb6NTyLdQ_qfK-6hTM7mdd67fBxQ-3 zIQ|ClBO$0Qx)Cbm0N?h@(VY1)!IO8+sib2IVJ zp`uUH(;)c8utVGSk&TN;GDlsT=In#o6Xsy9Vn;|*;R>86^ZY_7KH7_<)>DsEjEbM~9 z)2R3p9{RnA!_?s4o-2$1t7R4a zMPA2weGCWKZ0PmUW#HY}WZ*ykC|dRQEDt`xHW{J{U1Ht97(E#-S*NBghILKLyX=hs zw--{LkA&Br_==4-K{_sQwV$5-{VCTrAhQ%;+QjP!?Irvw^VC$iROEus*$(aT$(+tb z(WRZ&C>93aMpyeYG#S}T;dX%1hmv2=jrU5C+F^=?yW&b{&#wk08^8!(A$0d^DUX^= ztxfChE`KP9pxBwSUXGMmLj01H15 z+I|4oAU@5xZMw3hgpy?lX#kKf79%Jw4BKl_TtMT~Pk6)T~fu!1|z zGzJx>?)_u1P}q>yPA9l^pL^9|&b%S_2eScagHhvZT z^lzb9tBv&E0MioQa$)wN$D%pIowJFD2(z zk0_&Kpkq;aeW==-$c9~Ulul+bVWubV*tfp@@FxOC-#IxXgS#UP(m223OyV~G9&doa zARX3NF;XGj{NE?1=nGTZWMfG3ThzUVF({Q=UUrm})~%E{qEeLvwU@mgG-0B-tdyq< zi37|ETPlW<#$%gk{?=yS|orcL_4a+WQ~xCnS`NZ3OqML`8DYiyYC;>3cv%hc5!- zGOgDrZ$d40d!Gz(TcZ6$Sv+ujv-Hn<4`aMT*foFeG4WZ83Sl`iAe`pe@BhW#dqzdI zwC%biCFh(GC1(&pvP1<*N)wwbIY*J$t$*$wmM1a|d$zjyEV z+hdP2&X4os{Me2)Rv4>$&gz;~HLITIzOSb{XepA@PFLT>3`Yw0vSm%p$;7+$-R~t< zzRgbqVjJpOsQcR+VX}Ll?TILGQ7fxYs4)G#=D*NwxjfBnHgSp(t6zz~K(BY2f2j1( z!9gLN*afF(JEPSK7$tGR^RUN1 zzRs}3yfwKL78Y$+Ja;*>Fofkde>>v9?i!#KQ`T5|jFSC4L`!fzA^5&mzp$7w+L_;k zTRVW!#H=!q1YQUJp03NwDs zPFX4uDJVJ^y6@_~Gf7p1Eh>~P8GgR`nIj3oH5FlU?9M!$HKh9X#Ao4!?JsBDD+f$qXC@HCSp_Zj^c%W-pGh+E0R_a!x>W@RcT$yLrYcf z>rU9}v<;{0KkAYvA4B9m#9;4vbA#UyG2uN0(c$PjDbWl9E*v)Ce2T(ljq#oj4&7-9 zS=;5RBX75USoS$hgO|H4LNI^lJ_L@~ZPp0z|5ZKWL0Pf$g3wC^X8lUt_E}jq(42Es zpzfr@q~FyOLg3V&UHXa2W$?eRMq6-%B(6BS(AJZ7`2XT^V0!1|(wPh6>i2YmhkImH(|7@ zaPXo4C)3!c6!Dpr%!Cg5N=kz{`{%15fz$H9Kc*H)L#U;@f5q8%FANmr=SFCGKCmahlGWT=Qq> zHo=W{r=rik8qk~mZOaYi`uowB+Cbusi(~US*7HU7`e$Wb?qDC|F``5G&?>tmYE|sw ziyt$|^mSmdqKzLkc9qkl^ZJ>F-TaIsBNZ)`@oM_7L7J)eg$&z$*J7p*Ge>;7`J-+p(b!*2S`!HN8IF=>A7_u>HUA zhX>bNkIM?kmsFc3P_8{jf9moY%vK1zr0+{N8;Pl#@6)Ha{}UZy#YC))Dgdolio8ms zbV4QS=7*GGo1oH$lLF~uER!rIk(a^DvrN6c@*_k?-vX&8E_2sn- z2?~em9%ABS>v$-+6ukG-Ua%NLc89**cvuxo7*Dq6HINdQ;H&%`WgZ(m9LpAam^=fz zWMd3u)6K80uBuCfCyf31W!p}Fs02BuoROXq#*jpyXh^>kOCSp(HJ&tqYLVjUdZ}5* zl3&t4UO@(D=HFp6>grnMqN7jYdY8LZc261i&;PAj?zE2`rWM_)znyCg+zMSlWm+d? zWysNIByie~{rS`B-$DLM6QDapp1oK^kWx;Yi`VDEnF1k^lD6a0sN z=^5a&8gpa!QSES{eI z^-9WhOzWww0d{TY*@!0vWKX_#t^>357?E%+%OdXBd+>@@%-(Dt2?#j(WC|ihiAKA0 z7p5`Pc(_$j>-_A$vcGkQtT7Ng!#R$=f<5QY%Ih@A>Uth2rtzfx!^6trnyIJ3?^d0D z*ky)m!4V5zoPWk0{XCoiYMlD(i^In7{7S`&{QuGdu6F*sh+R~Zo~M|U8;SEyox(&T zHrke|)A-(f-Q`q(=Piw8tBScUSJv#*sf64-3i@6ii8+exC|drs`}Nl=q)fht(Dzfw zY-phxEjd;gqkU@^an3wO!1K$OB?mZ1;UVfQ-5aLH*yjbGvB#~Co>H$X#1bF;E-gzz zm4YB8U+opKq<=oW0Y8&*rfCEjI$X6Nn+@gBuk(64kiBU3yOA@xb)8P*3iT)%ys>+0)QFiA+YqXqu4H zCtteBrv=YuU37Jc=P+fpCDV;)g|uQ3`6do;$07j{61i3DWZXNGO9b&TYMZ#zcrG!~{sX zlSaSJ%HWQFUW(ZG{nzVEhG8;LkO+s+;wz#gEV*Zt`0%yg{Rf00{69dX@(i}8j1Y!jFl$xAy`Bi&5F!9kND^gl%r^rqaz4T7l?l`rdxB$v>eMm(U~0Ndm> z#QGmn$5?T4g&c?63c>%>J7Rknn(|724u_^L*k0`noq~{dXVpJrnr+uc4p;pqrlRfs z+Sa5-9Gu{|fjFBKjY4}SOik!jsND^J`20~smailDL7YgdNjX1=nm&=AS+SM`GA0Z3 z1{xY{TAxwzLkSDHxaLTM1xm^Hlp+&p=Z{CA+lX>EoEa?1ZhC^g^NfrP$B1eQ9&8sJ z5E^O!`rF;J(Igv8KD~G)`Ia{g$R|&n$%_wRm`tl+grHRG6H^!acS99g-F(AzDmmEM zHM)uB_B))ykKF5;ia(>x`_A)w*HYuZDX9f}-%>!|=t+A5<=3T>ha=5CLQ zuVTf{Q>bP{3Q!QJUj})|Onib%ZB?XK6~uJ_i|JyMxuz@c25;wql_o3kZpsTr5buKS}91^z0)fv}lKO^?SLD0)wd7$JIBRZfDVsa!)orwmrOmgJM_Lm}UzwaN*gi+JVvd z+O*hTj?K!n-E(iYdb)4Q7%UrZzH;5<>!&q?yotaH4!oLjWC>BJmCqx_ady9GPuk|= zta_=RJyOglD)4`R#7q>vJzTCo(qMhh4oTFWy)RzCYCMtEJnF%)S^%&jL{KbV@g-nP z$oll@-H#uID(M)cPaa`7oRmIaWcrVc9k$S)8Gmw9!#K ziFvoxHV=i^+9-Veb~w)G^a`+)#i{C4xtFw;>;$4&2cNS$Phta9&7{?3Ky5JL>tRxX!l35JkstnU|SmKs24wm zsBo&4>w~~G3BlmR)iDr)B)(Rcgb)`J74#RYMy!Z=9j%J7xvpIR##}t`wa@F8mN?S` z`6`mr<1tbb?t>z0HYI&ds)%dG^Lx2?&rvrDj`cJcX z)H(hWDE?6+Lq@Le|CSRD803F<3x+4_Var;YQ#Eo#Bz!V0(wn%=VvG~|-En;&-s^$7 zJ^59&Yt4Bww+u(;@^hBwyiJsI@6&UlZy<>nPR8_xqO9GUB^qQVbm1d@ z8Qf$QQbAEN4gW{2D1D&iW1~%wcC`SSeF)%JL;=TaKtW#K`ZMA$PoQ#gD@+p-KV{>X z{|M_IA!p5HF+{7<2JGZX$z>!XP$=He<-jJIPnD4w@iFJjYkpI9s2_}MjhlYY^V$!D z9rG1qs5H|uGnX*7WtGGn=4upxIN#lOuEyd!u&PEI^3LE9Uvd1{k21N@yorxydK(8Z z_-B4zvKoLp1%&JZ(JaQXDX?NDkG}(^@y&0vPLm!5+^t0b03hb{=Dg?8qen>$Qof?t zTLGoTfIXrNM2dqzbNE4Zc1D}|&)b$IsV6}0U<%BH!>cMuKor_gvA5KsQ^jEdnxO>b zBL-`(O<6<<#qy7&?PK|^6~^n{iU0ht`1?k;f$Hbw}_tw~(fu3dw zfS26!8^-%Q&;HMHQ7n}YDJfg##LKn7*^fOkwhVVM_b@UtItFlzbzr<&X6NSi)s~<3 z;d|iiDnB76i1@5;op~LVhLlWPCGgG9oC1G#@)1+)f-5>Isu`)H zssr+sA3&MlvptfToQ_5Jd(A73mS{Q1<#iybq@yy3*0yfXB0C)*E%}YZixH~lo_Vbtg>3x0?|s7Mq|fQ2BL9G1G|{Sb zyZldKk|$+naTw4fXoGdsgEGO%T~|esCoet1ojie;AUxj(=s}Bt%1aw0B+mgipa6uC z=5@AbXJ<<;FE49=Iadb?h5DyVZ&ORm105+w z*2;LPywl?god~R**jKKw$=2{T+!hvo4d_Tk)4FR-glkcm0nvaO5XNWR{&&XwF?T8; zH*ReJa|Rbs{-!u|9b7H2SiH{apcDw$z{oQN5;6iHKX@$L%J>)-YbTa-Rj{i7o$y{O z`$8P4`8a#-wFBGgsL;OzVA_bN+ufs={}O=N-CNPF9TgdDTp0j5px{0a$5UcKt7~}C zLlH-Cfttn~XYz#qj%VhAYtb=GK1~ zp=qEr7CA$nuT4A$$}S!-gWi818j6ty6w%~F!J!ZQYs8nLe0-HiIE1i^i_7SkWG8pN zRi72LGz> z2A$&q_t8h94F9+9qwpY6=*T`(1;2ZmPUk!dT#G_yxOsIoMqR?V|jU4zu z9ny66f1COLQE#TDp!fY-3-Et?_WxJov|~#qR!%;w!JcC#Ic0#Fh2?|x>xWM|qD(A1 zuzPwU#90)gJ|tSq@OAXxktRTx1=Kol*Y7oLj%f9 z*LSz-?+K}yRejFMY*vgOJ(=!9OXV>^o0?+2zKFJ;WB&7J&~>lyW5EmG*1x5B%$BbI z*jlO{gZZpN$S1CZrU}q4Q@g7(83|du?{2PD(v=vKi|EC@+hEkPFHKgY5_}F4%y(x7 zbGv1~lSR&vuXE~2V5i1_ssH_c%rFZo$x#9@`sL@?j#$rYEeHH0e$lZ(z!)iLFpDK| zRnXsdp}baY{v&){*nUs*u~p!+#iqisZ$GA0FMe6iX{HC*L7`?Rr-x{EiY`Q-EO}YSdIY|?0;;O(FUjX)r zAJWn|!E5ss1i}I~#2>P=`M@438EC;2kpid>K;GFQCM~@(4Q%U;0I{b`{C%o6@EdRG zH)7IT}c$mPQ;;dmw57xylV$r7p0>=b^ zn6{v2l?f3BaWG4;HBEZ|jJ*;B)4l_U4wAsF)Bz;OqKyk}1+A@rUl3mjz>8>ZcG9W= zCGM`Ee(Kl8sn4~Ehu^V6lkuZ1?321Q9v0L#tjYg+A3d_nir|FW{R7?9dqCorZd4yy zo000Q{jC}FMYn8DFyL$bH{%xJV97Fe$}AA>K}NErrCC8-qHu82TSMOeM+?Klq?sCv z;p=FM@7a}2->}L}X?-6Y?UXXl&Hd@(>*zT>W|2NYn$#u>qOia zI+&39Z=eAHHm#(~ORpCyDv2No+YgLw%s{=A4D7=?h_hWb0?hUO8P$K;+lETOK57mK z!*4p;h7D|E(w>xa%r?FL5zMb`5EV`>hW<9N!jRaM={})7D@lYbns8i1KL+uOG52;^ z`K^eaMLTUK&`SEJ01vf8BD%2VT=r|OcKZd0$g7k(nTF~jq0B5IB09unl!opJr~jKz z(2P8fe4U*6yfYB}4~R_+f(_>C2Ipb>bp^8t(0Tq0qjeO0*3*q9?zjlg13CV`d+uV# zk+$iO`y8xAMa?y@>?NR`5)Y%~fIw_*n}*avNp{lxzx4d^2Lb{=zLY5ICx47?t9kXI zq?Ft){ezi#A*Ynl010U?IVIbbuy9S^@1%X^8UuOHsGdFk(TLhYHWH1pijIiqd0THc znScB!o7O%J?oM{g#28Xv)h5&C@_vRk@}b% z35PPP75C@LF-zY(0v*U~&=_TR?YW6Gu+`+}$N>M^CVfKc@bsqjRMu2=bK3H}@|x|` zy8(X%{C`ZsVB&~u0YPu|SDF%5U0r}^Vn#}p0U?-Owg!yrW*1;0kC0&>1&EX|d|Flz zXDv4m424_(RlNt0x0yVHK$2~wU)&JHbVC}ylYR>-1U+ zuA}V)r7}->oRz=|lRk#hxUeL<395qlA=<6`Fuz zFGjd~fEVq&?sQ(iSteh+?1|a4mq4S^q=pnkMbNo_n%O~c!8R867C6^-a(D`JBV?E`ela)$ zbq(PSPy~P2-=!kY0g!i4+(j6GU8$eDJu4RnOibWSy$5*m7m6%ZgVa_T31?txsJ*RR zfQ8s;h$yGhxHUQpo7+xxU;g!76nA}uqZA{vSKSwe4Mkb{drr&m!HHA!^4Uu$9fPDV z6(O3UDT=DRDI_sG_u#o|@7dg(b#k`xx>f+$$jlu_FL3d^B55cp-XKR&N4vj7iN;w2 zzaOHKtiZI6;V5z3L&|8-;qUihpP>rC!l1yvx*V+IjvwKkyNg2T4AvJskoq_x2zNvW z^3Bm@fM%1cg&%dMgxD2aKp=>>mCnM~EG%nH(R_++N(|2=8LC{p*SadM55C@73|2{t z(>9_g;t!NE<;To9Fzrj)zAMC)W4nz) zr(dMxROz1L@ZpMgfosB9+0yxDD&thFhGcGV`S9K}jC{Tx5$}wDupHs7Qd1K{n;}r* z9NWD|O&eVpP5JOVFS>iG*ysA)iE56Mu~W*iRA7V-w$4RGlcLdAtb(ECpKnaN%M=vhtOeM z+nH}aI9_WUP8*ayS)dn3@!rCt`RRvmuqIZE`OU_#4l5$I9^;nU3WX9%wD`GkR0=nS zF7hUv;2n?c-Iw{HMc3qapnidu$+ASh08=MR%MRzyk?c5Z*s*#5Z?f}Q_=lYc@FX)< zkG(tr9{IbuRnNMemm+~Ka4o%2+EO|MLc}9Qx+*yffv?kFinH{AiDi-wrlR(s#Tfo- zw)I%?Dkv=M_+<3%CP?a>%6U7}pSupd6fOr(Ym;E$HmJkOQsMm~CNJfl;VUR($0(-_ zN!kRcuQrPDm1Ltf*gagB_gRV5`yV>4qi4I}U^4*4qv8shsj1ahRuvX$JEu|mJw2?q z;ri8DyBw_ojDbof>*ziR%TJv^Z7{Dq&l#uG8NWR^U%d4<>JOD3=g>roUK~4e8Oi63W9b2Et{-CX%cjK#;_liDJ-@`b#8wD8VE^tP3X+-R0&on4Z!x z*d0B|-1v6yZQ<{W`PyQgR!__rNSDJBb$M^^r+3uUp9_O^11m8J?S-vjU$3shoNVV^ zX+JpQXH-`Z1v-;h6H`)N+&FdIP^5qOu)crMu;3`?>nl;DmPs?>qE%W6oMKFTe1xMC zCozqgKxIznvsZl&B)i$rXN}6geRXwl$JH@7O=_qZ$;1Ut4&UQYal`u=Jka69oyo4| zr*83`m?1k90inApe5fMJSgjXXn=~dLo!U`TIj0$;9U$M7xsG6{{gPhs;X<*DB|pMy zD_t&}fd z!)9O^*no-&M#)&tVEt2MSk&*JD;V?BRPS2vl1Sk|h@IZiZ>gJ$4ufpp zGx9Nb_1BO63f;m|%NdiUlwmyjAvybg0ij$1o13V|QhXNgsNxP#2_@g?nZJW`X=&mk z&g4d-$Wv23TE)1oyf3ltIk=6DIMsg+SMH zo9|#WL^XG~goih?y}eTmNmH}f#{^Hhw_Y#}0mWdJX96&Y?;+Tvyww{GQa7g)x&WIe zQq^IA`w-WCZV{KNV*?UTpSR6_T+mpu)feeks4w|}C~oc~O|o))z>v}*X}Tqa_;)EE0T z+{WhHyRJf_@6(Gnw8Sr5_r&85OTY_FN9873<|)Q6)+*pP4FN=vaf9IR_fzRhL2?aZ zb`CNDYrvn>2Oi@wuto_yf<$_@1L|akXHQoG+9wY`KtMKd{_%n>7m%os{J3#ovV`?j z`(C;7m!@PUA%&kY@vCTWQQxj4*{|tqrWV*tzziHw-%B&-w$oisWTP1w^%)ZN5A#a| zi?lqp2_l!iizzUvMM0CsQkkYu=l@TM*&T%>FwVu z_@4nTl{$spZ5}ONn21>yk~g(}0s!#2Kx$&NHZZ2`nG`I8sa0ttsZlk5fHs^iGP>O4 z#`Z8mOq9L6#Ywrr*|53Jex*0tA5BfPtEOfo)%0FrTo?)XA#^7WW1Ud1yPIE>QE+&J zbK81U6shWLc$KKJzOEBI_s^Sqxh5x<8%Ejl3e?o!EOM=-|2i~Zi!lFfRX(QzPOxs_ zAEkx23OV*@a!76sGHSG%cWGStobP>X=s8{@us**PEGc+MvL+?lT$G~yx0&MB6 z9>IQ(Ea4rrb7;cb|3r!Ozm1Gh_9e)(NDgGM?k_%z4R<48;5Ldpbq8J0y|BPN%Cf3DR*fyjagj>=C z3GTU9KjaEO7YRB}a&s-~A*r`R?_=0~js;=RLO#M7PV?zdW+E}Pmtsd_a)gFX*Z**D zKZ9dyeLi;UEn?Sb?Hwp6w*h1!vYBEYt%;XFoj*RiJ?mv=mM3yLjs5s5YX9iyMc5j$ z#8oRALa`*BlR}YF)cqF`!@HlvPPk-aAQ*ASHKL|SG@t&8Tod63YI}{kDM#-9nwW2_ zWU(FOGWN%I|F!z|?fWlO;BlS?KjGV-C+G7!C2kGp2{uHI;ugDUUA-}YzWrn8A`L34 zFmUZX-Da{ov&BP;Vd>f;zCD*cq3!aZ7akZ0+cLO}e_?Fu|CVdu1BE{UWLMyynKwnX z%SnvI5eOD3e*sWI?|s1a!brIUWT&`hOS{L#cs+aCdBTFoTSLlI} zPGD*(Rb^^{U8h2fB7xUeU3&FxCI09iGuD02HzC}03>{!>>keE@quc_Tvw0^B)ecbM zZ(sGY20MScl7JDz5?*_n=QeTNGhMaKLqh`(vy`xAT{0=&MCzgwo$8xtNiKU#uYulU z*r+t7WApbYCJiI(VrPrfC~Q)G9+8d~Hzu$LEMN9FGbi$RFE5565S}hu* zXJpz-Nh&IC^Q@1?m6BJs+GC870N2by8w~rmn7XcbiKOCSC`uLRfj-Tu}WLCgWBm;0iHR6MO<7wXnRJr zoh~ldhYzOY&u0=G@VbOk+HhMs5Ukj@jrvBzQ5PCt!?qAtC&yMR_K>N=i6tIjFJ5M+ zFfJ^l-d~a=eJ=&?AdV>?oNTRoZ|wi{=vRI(_QPLKI#XLb{fQSfER- zKWKa=Nh0FnR_+G*IwtE0Wdv@2gDgK~(q%b-yA+W_~L#8o*+r#n{TTS<%gaT8X{1aVep{jcG+3eZd}Nba#~Yli4yV zEI~!G6&=;9wE%{HpvocWR`#y#zN=7~R3`XAvO%A!ZAf3e_gc~FXkoWr>thUjt*&6{ zSTWvtGl6~Jgb~1Scd0bE;=7@^dsN^gpbG25xLl8Nl!0JZYU@64GW4Z#yKrm{MLq3O zdtT*jp1I;*`z;~E-j5=RMF&ooJWld~a8akN8C!C3 zF_<4pv}pzVqu;VJxyqW#yPOlXKX_o%Mhpe(r8hNs#28{!h(W=cPLOVI`v^FCOdtjc z*ufDLqgWI|^e{j4%~f9v?wTv~%5`4k$O{&#{d1x55>N2rP$|agTgmF_?`7@Ou2LgO zPz#oZH^Q9Q=xyF`df;odYJ!*k*F4Tc0v*OySyO&T&tL~^cqDGWh&tKB=H^gHRW949 z@*xfCj9gsvCPlueCfDsJPI!^$2Q9sRtyip_7}Twnqgm@6;RDUI$R9h^w(>4kq8U5M z%eOf7jTX!#2@njB?%U-mZLmN{0;+c;SY_sYj_l}8q=G&ra^rH9m+BaLB+nq4fq~P> z@0)A6BZnm8SCbC&k_CJAiqY-4$0#1bA<|^#TIriapT9*1ng~uYl)x;Rf>=_cNxTf! zJlOIs?N=CP8}Kg&jbUJ)>!F7Fl@8%~&$OnLK)A+X;oQ}K4EhuKH zT1dy!2KGGKL#ao+9tWDGQ0TR{H>yBFxyqZJ+AQcX@}s+r?`*Ll`t~-Vg;YiHmOpTA z{gzwT@q7Zy-YR&YUsWi>??*S!69+PpU=_x3g+B!l=t_^|?j*f3010bzL_00C$X(%h zUHj+oBh(c;E8zAW(hY8;V($?h><49(bd^z&s+cl-6AoqUAFe{kog?5WE#W^m5I)@p zCocGv;>0U|hO!BPKbggN=K8~%m&?RJfQ<$rued`{Iu8r?me2EINDzI4cX(FU?q2CT zl-{*f>tp@1cjex(Rl*2*&ZMPzZ|}kUTIJr|=xhbx0G@?)o8=myDe#n6dmiJ*K{CH6 zs%W7uKeEobU5{RwSS=N1$S`3R)EDa3+e@F>Z=}4gffU}E)Nd?Y1ft70R%}-G(I#j1 za%j1jp*K-$hiZPeBZDw$gt6q~q?%{$@M(Q-?Ow;}4t{-VV8OjiXHbEv*Mr6FjG=^s4p7KI_h?+xOerhVfPhk z_Xo9;r9`E1ob`R!gMu}wX8mXRO}5nr??$9xvd1gc=KS95bBatNUw1AT=0q!|7y~X1 zoQ>&bj`W~c@qI7#hCV<#ZVzX+t`EzfqxmL}5-t3tU!PklxrC?O-p+N`pRTY#zuwVn z#xbwbpydu^SfHCk=^$zaT~Tmr-(>0!EG(;OcaBoKz&w0kcWRSc*1Z|2DW`Rq*R4G9 z4dwE?p!(%E&AnPXT9*C-u3Qzu)ipbd_hKt;3p&EgOcvOmlb4< zFHb;ug*fj-GK_o>xyA*RF{IM&*#m_J-mt6B zxwqN04eMh-e=kk|-xCDB`wzK}?Cxx87?-y3zv(_n#OB)Q#$A2?ZS_qzx-Mwm>?%>6 zJ3-=I4GS7IN#D%+06OK~$2tn>`V0pY4p|pJ@6QcT<{3o@QZn*HJ z_lL@JmF&sl%_I@+I7|~Y?DuhZRk5^=RfwyvjAhQ^ang?1?=10(1hVt8ivfY7Nfa(t zD_$)r^|~5gfAc;*n(b80H!-&&7Eg8Y;WWMt-SeG$L#e2b9rRt#=vkq5J`(*)wU`23 z5ur8)e zZ9n<&?I8#(SRF1lo%o{cW{5Zb3FX3{BC9C8N8h{McA+m-<%^&73ETa**^&l(57lTQ zTHG{AGpR=Zs}MuiNS0al1NNOgGXqEJ;Dq(Wra=e0U%QnCmG};&(Us2iJZ^NDyBdE5 zO`a}PE&MzqtMxRUAKlJwlrQCh?azdlZO}4=Yj<3%V0w*xQ+q1rS>}QB2`%$v13#rZ z&EJb#q77gSr@vU=nhVuzkJ=ry5MCb1*=3OxIklQDwf?@9K^^(g)?(4L>0!}iA*upL z!>38bJ?vn-IcTj#h_^8ZeA{VcKu}x~os7xpCMB3cDGj(hLX$&c6l z;{)Gick%8S>UB?@KP-f|?{Jp3XU;6QN6t;tuh70x;Kd{>w92xHbiPKxywSlVsFN?* z_jonGGnCHZ2MxGHeXd+{>-Ps7L@%92vh$LsjTbn-Ck~TUd#<_}z!A%-eKFOJd%hMx z6Lu^nstQN5{K%?WwZ5Ib=Eqd7#-R>(ZMzyf-6@?$Cz{ES{WR^+qOs~MeY6&G9ZM*4 zYO#eL-B$ef%j?Wz5r-%sfg8u=m#R<`A8C35|J2};+4gTOK&@z{5(Db(;a!wff4E+I zn2=&&lSJsZVxoAlz&v7O@zmeMC9PR} zXuyf9x3u+PQymoW=$H8o8X2GN3xQI*z$=UHR8U;ZSdB00VYZ2@v(mX(s5ClrKxWv` zzse=C{D%GWCE|VM-%l;=4hCG5Pj9B!nXk{d9KY0E;%dQbSekJ(;*UC&v?-2``%+ly z>zvQXGdHnyaNL_kkG7t3arSaXXVQEKf8k}mS!xocxAnTw3D%C5&U?STJ2`O5#*c?S zZM!(T%>CoJ?$bMuUFCMEdg|nJHqR_)izmz{!2;&aj*wUGh=a=qhD4oJNw9e z*9HWR9=s2l67flvY}l>DRD56OFTK|tt}F$cwHxTDWw`rxS?OMH>cc$*S9nWnx%J24 z>w886v3X3!lqNne+~)K7PHc#~0(j{ukYn-s)U%>>JsLY45*p380;m%&xt9}$IUqzf z+w!*ZV#SdwW@SLM`fuYttMjy;m4zQjKD?BH_s^YhfZ(YXkFJL6sX?Q@^by|8Pao?% ziG)ENKQcTr$3nS@s&YVSc62H9qLyB+Sx(|@@1AOyouBg@Agwr8jlGllU_HFReUH@; z^-{ma{a6Y9=jY?{io$It+{cJ6*O*Q0xBAg41`%jY)L~9{pXhg|zV6VAYqW?NZ~W*^ zHcRBCs;g+bX|VXx7&@vzNnT7)*hp+Bb}6D?e)R-KNVA=3)#iD-NxA#1v_{PLp+>T7 z_TtvCi}jtq`@NWYh|SkeGa-s_*9u?VpNA-nVtT(AzOK-({6sJOOu2`W#mS)BX+zv@`>Z(OkyY ze>|M^D$xGMVCMcRlHtO3GV?8beh0x(jq3W5qA`-oy(Z?SkU%gS;1VW z`Q}l@vw)W?|B(VPQd@s)hOS3Q3zL2P>!aXN!r6TNAA!FGC*9r+{7mOqUNwtR#qN66 zvYIX3hgdy*ed+)OGP@5b23~Y*L-ATj1iI(JVkE-i-h~@`G6+`Zf)w}ik%2>8Yhp8a zyPp)Rrk}e5(dpAJqm0L&IzvnMMU<;MzrL#L{4+&s+&9sY7>X7sdI0|_cBZm6pwsW$ zHo^!PYF>ibLE{twWND`cwdynzL5kUW-xPUN;Pc%{6sg|JROHjSZ2GfCUR}zsR^y!O zD@_Mpvzj-^chzm)rj&g(B6MRipn^TgZ`R zrPG9@8o&gxf9CIh`yHi0Jb>huH(Z5(X|XZ@CG z?_m{2IaaQGT>gCAqt1kzoh8RUsU@eG+-#1@T#uMa?<5Bx=ZZEe|1+l zv56qE$`k?M2EXDCS2Ugh;dYbb_1<5~+t0&4PL=B9gIYA8cHxQU$~3Y<8PKx&d3t)D z;}bC6CafhpxD;+iX?KH#H5B!$=rI!pPx@a|u%}@jYdku-w_B~j%^yN$yj#n^I58@1 zkfIl5fT+Jxq%on#F+~XdP@0~}o>sdWumk7MC=(OPC>8xJ_NaGW;h56wWbfK4oh|s? z8sxY#>32#_z2~|{uavXHrT63>k-^f1o3XDMzi>eP#7;ktx@3?S90ecN68<|Y!?3_Mwc?UQQ6UlYH5_f_ ztB9uoUe`;mC*|kE=@Y7b>%*9+2pJ(MzL{V`hQ(4}>yk18p0nGOx68*2H0H~R2e9ga zmEA>%dUw6pE;%@DgCpI8aytbk2@JI(F9~tbmp`lh45{^2!Q$Q{#!%_$1t`?$2k;F7 z{yY?1KLd5ivq^vk&)Y)rcfUM}!{hT3VTlroQb&*r{!Gjijsm*|L7c~k@E^>BGYuAP z-?nr7uZNY7pXcg{h1u)9q*e@XFZX2uJQsa&jz_;6h{}c_bV7bCa}3*^E-vjcStdRW zRmM@g3l6zu)qh|_Jk1%5`CV5+9X0dVKP{=6{H)LacvabtGkdX9-S8BKa`m#^MA~F- z$-Tcp>FyAUki=cmy3FbJSI)-Ue#WC%YodjD5IL|jHZzwkD-!B|)0%f?)`?|wpFBW} z&@OZJMlr-=mf$xxGVWI@N1^(aC{P*eKQf!O+9x?6vzaTBpYj@4=uK}=51nLuFS4+m z`HJ{qs}M2Doy!aC8`VhJ*t}q~*{2zIig0dTWenmueolBChY98`#9T&fpcYvM>AF;P&A$^ZWN zv&3dO2^BI<7g$dhHNVC2<%<`{&C*G`vF>PFS~Ljd2^jJ6oGr4|ISLuj5?)+UAgF{a zs+L%Vls+c+g;a1iOwDAAZEBh%MRoKCNVS+kDl?GKCk3(u0Z^>9=X{1c&%#>5Ia#Q?q;c&)}HQR^K2FjKW^@T90ItYVf~76x>%L(E&>hJ&5%S@OJ!l zRnj%GvGD^z;%fUj!8%dbJhq1qg9Qv~!NIB5{!B{AX|ZOpnWM>~(6^5oj0(LIq5rN` z7;y8xI#zN@f!_?Km z%QtF0K%Lz@1;|k!`Vc`_?sNEkltBT=5*IXbEanRYcjM&5<*+}2olXF)T5Sl{BNz0# zIL~B&tf!}UZTxHaF_Q_Im?de54HI}77R?xagyDL{tb4H|AzraMu6{NUDkAA3p zPE3uUo}2T+;D?g1JR+MM@G3+Kv8xybc}a61w;K-luwy zl4CibC^y>u_(EES{4H?=!J4;L=MyyN*RM#zMC`)!gYQ516p2ga#|#Yx4TA0dF9lIw zCsm=BI{e$}qW{ok2*zHPI3<26iC(WX+R&Eg*;If3umWjk|H(Gze{L(aNYfMuyNTvcr*;n&8$uLkF#tg7* zl@f8Q#LC-08hYx1val`LZk0Z;!yXnWJ%aqRv+g(xRAsLZ_w_61rCd4quO3p(v0_4! zQE5g2t1LY_J`t+jUeS(hf1K7kDV&eI3L_PU5#@%oPL`JIkWMc)DnEH0TFcQ{U2elx zs%nzTobZFaPRPe_3VhW$}fuxrL?k*_M9RuRj`8+2+2(qx&JdOLEi(0m-hn` zxTySK59+CggD}ge`aG7z-U#^}Rd(d5-5@SmAEG-^kgrL@s^>Lv_xih1B}qy6OW!(8 z0_x8r{`e>YdvZQ9LNkHn6I?!~w^y!oBg4JIff3r+I!cb-_{tb1@;QEB(JceoNQOI5 zKw-L3$bODz98LV~0hC+6uHO#d;x7myi*M9TsL6pvc&y+BX{E|DD3lHqByX#>2Rfoy zFyhw%x2FqW7!G3lluI3f(#;u2+dfuR)i7H2G=rk8=jDbi=f$U|?psqXdi|w;nrOrH zQnLxwU%pJ9R-{ybp$NJ6_U86Kj-_d^-LgTsd_2c0OumnYK8%Yq#?D9h;_B+Np)Quw zj&({Z6&aYv8om-W=H}IXkEu+|MhEe#X1(utC`>+MWt9){m$Fsp zDhLt`g^SXuDR)nwy0@aOz6e-C_Z9!xE6lEvt0>}NJf5KZ(| zk3T3lV1%j?y$4)Vt(w^E+8p{EMjU3;0reo8YZT3SZx=r?@t42p`Y|C%oVxOBLG|Ar*&DP^uFHn4HYEndh8zY*L{}{hBe#TQYL8+B* zewItcj`;W)LKp>Gr-T2?2P}A0Cs-+7Y=G#B=n>$D3Inwajm|t8ZOq(Qw-~6q_!k^P zra!+f9svf+Jxo;j%`fZb?**i@r0A_K_G$54Z6DHMQ1G2wMk7GnM)9NbX|a@i=N!f( zXl9!fO(uZwW7mPSQ^)Ye@ghiG-8HSI*;Twam@1`ID+`-*2B~%3YO{MF*_{lYp>Z#O zMN4Lb(j|DX{qfjEy5$aO2JO$Z7iE7Eh{W$OsR~JQnymGYqDJ491r~UXOCHbjG@XD7ZenX)Z0!WHo#m=-`d=q8XX<2G}O}idF&;j05s*{0Qz__ zRRY=F28iQ;^vukgr!ZlZ>87{yQ?&2O^|yN3{ai}RDyBB5-j~`qfF)v-LCQXcKq+8H z4kJTdrhDqju7EfShJ~TblrU$GN{7kBHWqCsdy_vjgkx0%wes z4ubA|rlk!3{>KJiz)Seh40JxPkp#XWhI|egswd37akP<<_nA2?Yp4BNPc(T6~J%ci&c?#p)m<$yT0>3a;*HZNrhAkHII+7N8 zhIakyc4^?j3I~G!sj6!+tchWZhN^Esyt|7YW1aUoVkl`hU|<;dB{3ltI+f`&D_C`D1(k^hFx7FD7hl zdBH7VI(6^C+Zr&~9uh|C?Bo2?dK%Dr&Dx~22+(>K{f~Uc??HVPQ(%$!wUpirS%>(5 z7&7RNU}SfcHDc#WOJq0eGux8ph13T~eF5q#&gqTG@d zO*mL4G!L>~Z)uM<{0fY!BBFTbat^NYcTqBrNsd(Nlv@=ye%PsjHVPyAeHi+7SRZ3i zor!6j^LXd>-V7e{Rw(h#Q@o&eoEZZ-Bef%SE_mt`xsmi3V|owKv|nb~_qAw?)6*=HTDkUc6} zxXYd)gh~>Toq0uEk}WHQjH6-XmywbAe@^3{*S)-6p8I;9=lgj+>v^8f`||ey={>1FpdjUSzV4M#1*q_A$7IIB0+PYMZEe|E|moE9?qG8DoX# z7%igk-*!;*kp;BB`v^x6nd8q&&iKF>h|gc1W<@;uw_Os(aP<86F+}3O?WV9KdgAkw zJcxV$wjYBfF%tX#z>J9ew_O32#7umCLKt!9pLTjnSQ0B}PnqOe1hV~?-Mp!=7_B9c zus<5OWpkjCY8yz>2%=J9+3g@wtH02ISJI*qLQDdzK3cT~Qls}b_b&E>^Qw*ce=JxFzDI7%$32I_#zI(9K9 zu*c*M&RVE7 zYP7=c4v0G-fcjnMLC(QfnhhXKWEK2H0EtAEcIvy3`m?q28X%$$bfj?}2)L=ka5*nXW zNGv!{y5qscb1W$8ya$3qdx4dn1Cf)o^WI2HYOz%-lluBEQZRmcTcB$QpUbASa=$Rh za@=qQi9LpMUU(@*g1v@7E6?X z7)r=Wvl;t7oG1}MVGy-%#9j5;w{D`~SeKkrls@oUp)pvi|H7rKGIU;>6x zs|YG^tod60782A+fn_IZxvAq!Es4B!8(3dO3i{y)FdqoKSkhMWI4XqJlt8Qj9yOq= zqf_Lg0@jk>n3Ono3Meoj^y+4J0aerKi&6t@Ufd80+E|PPiQnu&=+jicd4ja}LczVO zpqv3a0L{dDF2Aw7q}~r4*C)PN%1Z!DvoT@=O8T8LWsUdh8I5a`87STdwB>`P5LZcT zZP6U=ssa?`sG<7lTideiTj>Qrp|kiMA3(&h7~TgD^3TWs#dx$Cicw#oRJzc(b3!o| zc@)!4q)94Z8_|Vg55P8ElsJt>-5O+5a`BO=d@KUS2x?nA2bfeID3c2CYEqBRvJo|a zH?U(gY%UuDz(UADwd2fVksj8>m-c1^!-Q>30aZ$$O9FqH^0wX-UZ)uWSy;1{Ihc@AGDZOWQwfQGR1^>uLlVl&SjZrh8KOjy z3&k9R_(ED?I;06)r*5Zs`_2`+^za*=#Ks(rulIZWbt8yBwCMAR;x`Z;l7!%g3WMrx z`~(Dr)G!P5D^pBNK)SQnl{Lt!xDOPP3)<>9T!qy)ZpR z33#%M(pvW>Gk7l%EF;o<)Y=-vf4!hqbY%e5M=gjGRnvQ{LZmaj7)+?Nyu7|n7k9yx z_n937D29{_<~6b+nHR)Bq}d|Fij|Ev1mx(OM&8GwNQ>(^eQGI`9;)jQ`x&}tGsi^a z`TGFh^A}N5%q;~|j8->%3+UK8rj-2*Fsun>rWCXW)~|Y!wKp}jUqzU#osH%3mC&=P zRMbW?YP_pJ+ewLQBwWEO=(F?CQQ6xY(X&N2X0m^~6R!zEG;3Nk_R=0EN|BO&phS~n z$}rD+W2^7I9gZ!@lZypB0zVJ9fGq6F`#hp_q#587p#2=`?PN6l!wLguMaX-NjlGK; z^_cevbTMv|h>dIoXwn&0r7!G!LZmvnu*G2#*HNaO znfhtWWCK>py7DmWQ1hV9Bc``UYq?eHo5Hcuz7D)A#0`3jmOU~BB}X}`GiW|dR3gKS zfbR%4k{iN+8^Ei7ZEKsY=@fyJwoPv@l6za^4)Oi%PN$CAv=5vjq5WD<18^-i7kjL#{vWKi^CwOA=i1q*x_)6 zWR{6(^-+yN4|IhL%|*3A3@FjJSdPc6Q@{mng{&UdW55m|MqPnkHu$7_%<-(OdEbMd z9WNHe1-BL*U#^=9}lqucE*v*>d4Ufea z4FF4oOT>)P<7e^(_nw$Zyhq5x#msFf1w|S>c$#^=kgSzxsH{Die7)ye0)T2k(Zc9_ z(wlxDNFe`Nofh5(F3t={`k&RML?qCio$wyy#*)2{r?e-IK=z4-GwLO`ic(K$UAcmbGfxoFvxhg(@ z;Epf01RST8foZotJHI-EHF`4hV#_vwJ*W67Yt(em4<9o|xg;ts-)5d38R2Hu zxuC)jBMq}%Rk_x~p1^r3>BJHZ`n-?6QT36`c>|gk=A*6d@D!=JFftk_9p?&p`Q#37 z82mz_#=8kGd%J7GQP8$Eb#3V&Ws@fm2$S7Kt#hUXgs9vw8+Kl3h6RvnpaGwqb!37B zMed(qV!A%|W1(oe+R1tM#?2)ObC_oIRlCcytk)bl!b!5?Bi#6EOwEbr;@2ls8?XB6 z97&_!n4^+s+p&n_XViTpUnd_|s`Ne{+S`F=!K@ajRj2-f2p#P_01aZ)v*0}?_V%lb z@iP42STIu55Jrq=n@a?zTzFrMTz%D&5K2NIzmI4De|>V@2UPN|+pFSy#8MB|*p3&A zV`Z8kJ3r2C$~2=@(U;1KxG6`AjYoWm>gwpDn4#`os$6&7K4E)RTQH535GR2Fm-Pn+ zTPp%Vb2KxX6IdEfOj=s;R#@K|2%Twf7D<4rnO9+zL^K__-!d$88=5s8cBhbvD&hHO z8v%k9hkl6k@4mqZppe*Bty^>o0M&ecxrFgs#+>wslQ4`3+05EFWRr&PY*5Rb-3(kR z!++@=W-1RSCz%eT14n+qd3>pCi2LsYv~i8f8;HusN~j5BE~P3c&6{RYrO1(}=!qX} zkpOz?eIBWBg=P2-?=1)lgzXNS;~yQE-pB*knEnE4?}799RH4T#p#==&dblXJ1$Tn3 z-t`M){m+;-J*JZypmULf_P~^l78mjU=@oj9Nw%-VgxssbCVtz*ptkfGbNP<};J$o% zzqVf1JivI`D`%s4QUUvO{nc20<#hq?H#GR(QzRY}taiaOkd+@pc()3u{(YGLB0x7T2ua`|dEm?1Uvf`T+AiMBmIolf@!c}k88g%}@POoR zT@q5~JnX{$dy(PYxG1id5VlT({{-cl>r)H9OOYG8pSs@D1rt^Eh+l8#B{6vl(Yt4r z@#V25w)voTJNp6G%=2~(Kwa41Aw5{7b8CDc5GkPyZrB{}`LFN=LE6Zh&gkM^?n&em zfUiwtHsA+@V=P6KIjo)DY1!`%H%pXv${b8AkLUaxF*VaA{b7+HIXig@@`)4G;t-Y~V?L;z)B2jR4NQ`74379<5 zXEB#2s`g#g?VE#Pz31xQJd->1t-6=a46*^Tp`BeC5_l!&J9;gOT){RWzup7<#$~JD zsQTH3$Kuu&3XLXmVx&-kF|kM!tCMqcQs?})CXrHBN@BKZKjBKd->x#W_tkmTm!>pk z{0FbTW{I@02AO)7DoB`pnai2+*7XOrI35)TNR4-m&U^8DvXP^qV%{E}(N7<4>QnYGm)|sewzh>n?dNkF(P87+TkMt<{GtNulwO znK7rb=T+X9@7Y?Zs!fM_HgiW`>jMaB$+;j(bMvqeg@8Yth+Scj?elm~;q~8CqSmw~ zz?evRan)Qp_#UP{*9LjJ{HZe|*<+xEF`n7qe~Pkkopxo?AIZ%Vv9oF2o!5D_flI*+ zsYvK5g6(cXnd_O$g13SUzT0;sUv&e8UNctCtY1QZ>6sk^SK}UslNnN28BiPkZkhRE z$d1J<&dY9Q=A2t7HLJx4(#mq7cx7`!=&Lq^zJ#phSUGHyf{Gd|te;%%L?;bPp!?}> ziF#ANG<^3rKgD3EDHYtEL(c}vs|*CqTe}l>2DCFM@;e*|5}{W53(>k^m*Y(0>c8el z1h3VY4)R}AGs`KLk(#+h>X6H7PG!C`Jm^*H^W3g)q9uLKK@i%ICWNYYhhA3cWXuMj z!C3E%E-lp%yUuVVK;cL+*chp8IwdH0Z%ta<&A8^1;q6Pfr>)8QOHBY*Tm-tl$0}9n zFmWbn;pYQ9l>U{AuweSH%l9a)(&lDMB|5M4^YNY8-?}~N+2E+>m4?-W&V>Ntz1SK) z+U_6maR~(=o;3GqRyGr#bZM6JEV&j>SVoe>@!^+M&wZCr`kDQpVmJ_#^vg})!w31u z5$qf%-FoX-xr4QoGevQ#rpzLYldK~h&9SqL zrJZLw?I)Q5FcTqyD4UIq0rX!3)S`_qbprsIo(LnAW6bS_Dc-eZxn)+Ctfo94TnAf& z-o6{!-23h_RGPc{gZi^XP}HN@bK+JGPCbwtH>J8o?X-Np9t7wibG)+mkw z%m69-GUlL+i*D%snxNVnPcvZ@NBT_DZs~K?y-G*1g3N;24Duf55JwRA(LAVs`|*qU z25`19f|a6j*jdf-%U1N``ET~M95`|e#Fv{O^GJ5)pXLL{SP-~>URxz1e?$#x&LF${ zFj)Fp=n6Eo8?o7yj{In9CsAy32ia3@BQ*+XLKIFwBE`xwm|guI^MS%mN(JIlpgI(5 z^wYduNF*QlB1cm7?f3hMODWot$DMbJ);84MitS$$OIzT_tN3N93^Tsn+ts9l3S6)l zF@72=wLGUw(7UcT(DhtCw9M;+WStAO*GtAqy9*kX)N>!#Zdz}Hl*t(rw{0>56B5`z zV>Y_FS~K+@(H&|erV>E z>xp71Z(*;zRIghP1()+a&|P>axae#Roj-x9;Gq&iJ;bRX{%R)m2+GX&I_LcwhbibT zOrCJdc-&(-PVv_#;x`n=>R6y^nH`zH3E}CZ1JPd&mN*~1C$D)JYdCqG-}kyyvi@{T z;aWI13;xmdh=!zFs5knkec+t`GDNsIb=(swQ?lsS`xUS19JnlO!AR+z=@orY{OoV6>mTd)hSKzbvIHvJspWQCV%gN$-JY%+j%>oLxiOzhepEEq z4ZbFOa*UVu+qh#v3C&ZX`eJ*NFQvt)L;v`M5g5x++2s3Kly4q=Gkts;AV=}s{*a0$ z*B4Z2(sha>=I`G<^0EPXAC!qtAK&;MCexHkr~^pZnT5i#X1U2F5;};O7_!IWY1Wy5 z{u}3VBkKd#4SUPi{I-_Yle&TE!3Q|=){@NrA?-)4+$)+l7d|}4ZI4(BoO0cK<$Zjm z_mujR3yycf4~i~5?vf8ukac17CPV^7g`c})9s0cPG<4s5qQ3s%q+e-z=mv0kjx1P> z*0*nCNNcAD(8eE(KwqcaKdkNxYJVXq)jKahK)moZ?EAacNh0W+={(X5_lc;T6ok0l zgp39H=Yv6hE2E#A)-yFjn1=6ll7&AE&UoY3?$|tseW)Vs=&O&IuX4nnXR|DmvY`Ty z75vgf9qEU!N!%mT0?Xq4RYF9^lBm?d9(Gw{L*kD=JVOPe;mm#nuwj=+?Rc{Y##)1; zK5wGBB9}hC7+Km+n> z6QR`Hb~SEA(ZwZdwA#EyQ9ju46}UC|&2~|kK+h)^0syQ_@PbAwFw~zdQGo&RunKLu z?!2xYkpDnXh=t@D9F#ARB8)#5ki9AFJ)|$hhB;f|KjlJcF)_Q-#))ZX%8I<&GyZ@3 zE`$=Ord>4qv(>;m0!N|OoQ$?L-9RmIoWQfw8jF9mm2s=~`)RVc5l&CY*}Z<11d!BZ zWq%8Qy`5z=&szP_*|#m}kNP9gyM8l622%B~oyLPDJG|nkk|u5{^fYa6P9mwLWt)+8 zy<|CIDrC{%?`V5nD@aQ-q<>v@7M?u8%HKI4jq7`%ha+L)&@1UXb5f z!3r6M?3-I|P|L=IU@Sb%5=$~*K<)0c2Q{Wo9;s81Q8X{F0_A_2cW4XEaKKYODbl;}w zd=}yE;|`Vv4L;B1-4Z$MDUGR2E{HgfzN5NxUQfb?#3DL{Le!N*B*PO9Fw3$lYIJ^P zexj0@$CNgcxKsXk;~siAvb#0o;701#z9!}d43spoLa${j7EM=P2`GGdOBw9^GEQS` zoFhZl_4P`}BP$L$$l_uB(#Ni!XF%}MWjv>cR9a)0x_&{@lD1u-?$ z@Vke|d5H@QD;xf`Z2TvN;s{;rl{ARGsp6pRO5c!IK?F+`VT96GE(gyK7%|MMj^!&| zT$Gd4`}jXzpgJ=M5mIc=xg{Ufyen-i%41fnjo$JE`v`~7!!tSJqOxrnquCK|8%j&A z|JQ(ec&-7Q91qCPbMc3=*)s$*YJPEue5`rz_9{lMHV-L?DrvJY|Ep?aAdvON4;GC8 z;2ZO5(+WzN2BAxy3=jY{_q@5$J!Rj5C`Y>}7H<9-0LoC5j0p?^g-n*Ky1hRpa1=>c z95ArU!^v5tTPrv?a4^}XAAjRu{%BTk{LiO6_8==Wru(UZcF0q6GqL;luJkFTs@)qm zDrF_Q*|{i0C`Z+upMjf)KjLWW9?G`JX3^mI*LQxw>L1#qVaAF>LlQu=3C*dGn8qt_4C^(cb!BdNMEWja zT!y@7KHp<_l#IEzwfsg4{{fLec@akzo;Ut`OHSOC>r!T3Vmz%RMtSHwOjR3o>|9s6 zIjplY-FAMP%F+^ln-rhG(ed#mOLTShDjbuANn!1IO%#Er{}%yzXfs9T8P1cf$vu~T zs2twqyof)Z!=QfIc=44(9`O>R7vin^?PKZ=ua8Q;+78&J=VJv^lU*h-I zNzlqQhF$Z3+*5FmfhsHGX7cYQj~3FUzB^5qQ*MLZ-qz)$u#Rk0MQ^yygwq+;_=r^j zJj!Zp?6!dw;tM2zM3y!(NEY-dT)O%HKJi|#FE5@y-*&qpFWuFFckvkKYtUnB^y)G@J{Yt5czTG+_&nt( zAJ+85UxDPON5QTg!@Ps2oOH&NKaJf~R(8%vy{#!_oi{@lrjA+~DH_d>{d^VRUe$G} zlGEU_d&uQ+brfnfW^Em-84Zal?0QEge**g|7}~Tp{fZOVtlh&cwNG9XQ6G9Py}x{^ z_3|71PQtNLvPxF`SBw$+M`f}RAcE*dQ|mtpP#w;H3GYss%Yymx{JC>RTEvK)oKjN} z1J literal 0 HcmV?d00001 diff --git a/doc/design/images/l1_regularization.png b/doc/design/images/l1_regularization.png new file mode 100644 index 0000000000000000000000000000000000000000..e1b9c7a44f94dc027598a98da93ddb8133190972 GIT binary patch literal 1157 zcmeAS@N?(olHy`uVBq!ia0vp^JAqh(1xPSB@w`k2QY`6?zK#qG8~eHcB(ehe3dtTp zz6=aiY77hwEes65fIJ>L8=O=iFto34NS4sQC8zgol zZL&q_W0k+f-{!`DR$|?;@O;v7%l{?+?iUxb6zBy1C_lK~h<&gAiZ4q}I=9=Ww|C$B z(z)}O{!#0mxZ`X*r!O43Xw?D_hwvgcZnEQm{_3^8 zhsuuzp3b>BJ4`C;HZe`x{=39}X4vA`!`&H1&iM^4_8hGSGw)x`^IczF^Oskzq-f@g zvX4(9mQ*Q~2Ujlb{TyPy!%Wck!lzlcWq&W*$CWr88)?-|J0a9T$QQfnrF0kyVKW-cQ5QFZ_(%SKIZd`&t;Njbd;IBmqF;N^uXzJ zl-^Cy=c;i3Z8URtX|(-agGE(3<;B)l-u7Ht{UT%I&ilJ&`-ZEWc+&S};)`WBW_TR) zI(SBL)$uItOYgT`*yzDm?el+DPk3pdYNXnwWhT$b)S{?BioJpb$ZHDR?|+q6ek`!1gBe<%65)YoH=@UI-- z(}A&JkAB5Z?{VC?~8j6=l)#A9zPQvH14cg+9uCt|)UO_QmvAUQh^kM zk%6JPuAzahp?Qd*k(H5!m4S(_fvJ^&!QvgCPorqa%}>cptHiD0(o^qppavz74F$zk z9+^R@#ZLL9c`2EB=}!3-42H(W6-JiYMnG(4XjEdnGaRVe9IDzUwJbGsk#1QCO1w%bkL$378RboIRT%kq;7vz^X z=jY@X=^8NL)`>+XxH2~>KL==Ri0?Tkpn--M1{$In7?he`nv+snEgnpd2ep9j=@ UKlKVQmoqSUy85}Sb4q9e06^~6>i_@% literal 0 HcmV?d00001 diff --git a/doc/design/images/l2_regularization.png b/doc/design/images/l2_regularization.png new file mode 100644 index 0000000000000000000000000000000000000000..d5c2fcbc2ccae75ad083162e5a2dceb0210be298 GIT binary patch literal 989 zcmeAS@N?(olHy`uVBq!ia0vp^RY0uG0wfs11@y9k6id3JuOkD)#(wTUiL5}rLb6AY zFHoHt14Ba#1H&(%P{RubhEf9thF1v;3|2E37{m+a>rPdj=S$Y3w=^mS!_$R)@l&6x7ZV+&9dzo(01h{y4_Q|)=zksHMv-tGIFs>q;;7T_puq7aBJ%baLVf@sTyCX+9wCvCnbMwna{IL7}haC>*@R z8anY-o_M3Qxy84`Kl!Kjo;#^2_b~F@zTNZhecxLxFY72WF;TkewF=W((RnTl8Xwu# zwr2##2Ay|bR_fGoW~ROJX{VW=zc%MxzL>e-@`sD{cmDj6zOy#7db7GRb1QF`{Zx&D0x-jV@gGQj>Ev*9!uWR;bt}%Mnds%8~ z6~|h2jj~zKwA;5i8~=)*#;aYysGjatpL+ayc?~3k0Kkqi6n)eKYMz*j4Ld^>5V?yx)#z*shgRkDV32eO~6=Q`sLb>o4@a zu{~3_`Q8)VU5`~h&cAoDOr5EOYr4_BsxMEIr$4*a8~1DBH$CYE#{(y>yL5v2uI8Ja z-^GHq8>+puv~~NMrg7!tjSC-bTld_s@rF_UrLQ*vB0rntcsAtOA9}H6YPQ<~_7<^<Y-?eQ=H?$UF9LeQ+{b) zN@iZVQ+@@5g++z2p|+8!p|+W!;j^rZvw@1up^AM{%TjX~98>a>Qr+_NN^}kN46GC! zi&D#i6Z497{gZMs3rkZKf>Lu*6N^(74D^f)6Sy)39qU#@G2=t}`7MbA6+@$;*pk#>eIVYfj44$rjF6*2UngC+| BgDC(2 literal 0 HcmV?d00001 diff --git a/doc/design/images/loss_equation.png b/doc/design/images/loss_equation.png new file mode 100644 index 0000000000000000000000000000000000000000..14212ec8d36c803de96bde8a9a4b5591bd20434e GIT binary patch literal 1589 zcmZ{kdpOg39LIl!94k~pQPPD|*xGMx%tAIcTQV$N6tOb1usdz2(5VqBbxJadP9fuw zOV1hV6qY%pJj*SYM`Q~*b2&s8&hMP3f6jTH^T+r5{(QdQ_xttv{`F1uqk0*_EMNct z7*fcdGyquU1J&Dfp$(WE8k0bx%f$QO0pQVArCmq&6PdR;At51 zN@SCI_vKmi;`qFQAg=V0GjrHxDXzaUD>fdTYEt2IP6o0@u~xV;s4sfH{y%$VK=?V zsaQ|xD0ZLrSmCQj6qwna*UflTbE3Th?ST(pM+^QrsR%b%U;$X4s3-jvVZ z+1yqBTt;DG1;}PrZL+rqEWnPXlc_q5T*Is-xJtC7z%O>0I-$FL8hdX0X(=W6+-=ka zx@fsVeo?x9$rY@)G>J=7DH;&mMQK zYmR793m(!B9=55un>i$WUvT@nOQQ2s_nJ0a_j&5?$VoE(_K<8kr82l=uV2;q#}=t) zB?iiiGw!8`tu{}#MkZ{z(JTuPjj3uv zci(UG`OwPTWqK0UX`yJgNEN6w`7+d1*l6FX*)6#_AHX09_rl6MRpLq7V}HA1&rQiz zlo&G6`O0PGA!^Ve^xBw|R|9TpF0yk=et02H7c43pwmhE^d6E~HI^-EFvAtJg)Uw4q zK2}*&koj;t)Y{P2Te&G^9M_jqG+)gNKDHxAF3e4j9KmJB`^u`*(E+(Gd54B^ffgOj zAKQi6!9ef+($N>Tl)gO{Fjmd!$)htxUTY#9Vzq;$Pe!rJYnuA|Z@YOBhO7=(9-DDx zKYfw>341ExO3TGRMUL@BH)v?yuCfu?Nc>b#22rc%u75IbRi)py)Vg5&F-|d9#d>}N z`iX3W#3RBmx{!f8$!9ms$a6|y;h!lZN_9zfz?}$Rz1-B
+ +The parameter `alpha` is a hyperparameter that weights the relative contribution of the norm penalty term, `omega`, relative to the standard objective function `J`. + +The most commonly used norm penalties are the L2 norm penalty and the L1 norm penalty. These are given as follows: + +##### L2 Regularization: +
+ +##### L1 Regularization +
+ +A much more detailed mathematical background of reguilarization can be found [here](http://www.deeplearningbook.org/contents/regularization.html). + + +## How to do Regularization in PaddlePaddle + +On surveying existing frameworks like Tensorflow, PyTorch, Caffe, etc, it can be seen that there are 2 common approaches of doing regularization: + +1. Making regularization a part of the optimizer using an attribute like `weight_decay` that is used to control the scale of the L2 Penalty. This approach is used in PyTorch as follows: + ```python + opt = torch.optim.SGD(params, lr=0.2, weight_decay=0.2) + ``` + At every optimization step, this code will add the gradient of the L2 Norm of the params to the gradient of the params with respect to the loss function. This can seen in the following code snippet: + ```python + if weight_decay != 0: + d_p.add_(weight_decay, p.data) + ``` + This is a very restyrictive way of doing regularization and does not give the users enough flexibility. + + **Advantages**: + - It is easy to implement for us. + - Faster execution of backward. However, it can be done manually by advanced users too. + + **Disadvantages**: + - Not flexible for other regularizations such as L1/L0 regularization. + - Does not allow for different regularization coefficient for different parameters. For example, in most models, ony the weight matrices are regularized and the bias vectors are unregularized. + - Tightly coupled optimizer and regularization implementation. + + +2. Adding regularization ops to the graph through Python API. This approach is used by Tensorflow and Caffe. Using this approach, we manually add regularization ops to the graph and then add the regularization loss to the final loss function before sending them to the optimizer. + + **Advantages**: + - Allows for greater flexibility to the users of Paddle. Using this approach, the users can put different regularization to different parameters and also choose parameters that are not a part of regularization. + - Makes it easy for the users to customize and extend the framework. + + **Disadvantages**: + - Implementation requires comprehensive design and time. + +## Proposal for Regularization in PaddlePaddle + +### Low-Level implementation + +In the new design, we propose to create new operations for regularization. For now, we can add 2 ops thgat correspond to the most frequently used regularizations: +- L2_regularization_op +- L1_regularization_op + +These ops can be like any other ops with their own CPU/GPU implementations either using Eigen or separate Cpu and GPU kernels. As the initial implementation, we can implement their kernels using Eigen following the abstraction pattern implemented for [Activation Ops](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/operators/accuracy_op.h). This abstraction pattern can make it very easy to implement new regularization schemes. other than L1 and L2 norm penalties. + +The idea of building ops for regularization is in sync with the refactored Paddle philosophy of using operators to represent any computation unit. The way these ops will be added to the computation graph, will be decided by the [layer functions](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/python_api.md#layer-function) in Python API. + +### Computation Graph + +Below is an example of a really simple feed forward neural network. + +
+ +The Python API will modify this computation graph to add regularization operators. The modified computation graph will look as follows: + +
+    +### Python API implementation for Regularization + +Using the low level ops, `L2_regularization_op` and `L1_regularization_op`, any user can add regularization to their computation graphs. However, this will require a lot of lines of code and we should design Python APIs that support regularization. An example of such an API can be seen in [Keras](https://keras.io/regularizers/). As per the PaddlePaddle [Python API design](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/python_api.md), the layer functions are responsible for creating operators, operator parameters and variables. Since regularization is a property of parameters, it makes sense to create these in the layer functions. + +#### Creation of Regularization ops +There are two possibilities for creating the regularization ops: +1. We create these ops immediately while building the computation graph. +2. We add these ops in a lazy manner, just before the backward, similar to the way the optimization ops are added. + +The proposal is to add these ops in a lazy manner just before the backward pass. + +#### Storage of Regularization attributes + +Since we want to create the regularization ops in a lazy manner, the regularization attributes (type of regularization and weight of regularization penalty) can be stored as attributes of the [`Parameter`](https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/v2/framework/framework.py#L421) class. This is because regularization is a property of the parameters and storing regularization properties with Parameters also allows for shared parameters. + +#### High-level API + +In PaddlePaddle Python API, users will primarily rely on [layer functions](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/python_api.md#layer-function) to create neural network layers. Hence, we lso need to provide regularization functionality in layer functions. The design of these APIs can be postponed for later right now. A good reference for these APIs can be found in [Keras](https://keras.io/regularizers/) and also by looking at Tensorflow in [`tf.contrib.layers`](https://www.tensorflow.org/api_guides/python/contrib.layers). + + + + + + From af215a1a532137686a696cb1c5da5a8797ac51ca Mon Sep 17 00:00:00 2001 From: fengjiayi Date: Wed, 18 Oct 2017 14:17:39 -0700 Subject: [PATCH 60/63] Design doc: Batch Normalization Operator (#3748) * Add design doc of batch_norm_op * Move batch_norm_op.png to operator/images * Refine batch_norm_op design doc --- paddle/operators/batch_norm_op.md | 134 ++++++++++++++++++ paddle/operators/images/batch_norm_fork.dot | 25 ++++ paddle/operators/images/batch_norm_fork.png | Bin 0 -> 23873 bytes .../operators/images/batch_norm_op_kernel.png | Bin 0 -> 165209 bytes 4 files changed, 159 insertions(+) create mode 100644 paddle/operators/batch_norm_op.md create mode 100644 paddle/operators/images/batch_norm_fork.dot create mode 100644 paddle/operators/images/batch_norm_fork.png create mode 100644 paddle/operators/images/batch_norm_op_kernel.png diff --git a/paddle/operators/batch_norm_op.md b/paddle/operators/batch_norm_op.md new file mode 100644 index 0000000000..80948adf2b --- /dev/null +++ b/paddle/operators/batch_norm_op.md @@ -0,0 +1,134 @@ +# Batch Normalization + +## What is batch normalization + +Batch normalization is a frequently-used method in deep network training. It adjusts the mean and variance of a layer's output, and make the data distribution easier for next layer's training. + +The principle of batch normalization can be summarized into a simple function: + +``` +y = (x - E[x]) / STD[x]) * scale + bias +``` + +`x` is a batch of output data of a certain layer. `E[x]` and `STD[x]` is the mean and standard deviation of `x`, respectively。 `scale` and `bias` are two trainable parameters. The training of batch normalization layer equals to the learning of best values of `scale` and `bias`. + +In our design, we use a single operator(`batch_norm_op`) to implement the whole batch normalization in C++, and wrap it as a layer in Python. + +## Differences with normal operators + +`batch_norm_op` is a single operator. However, there are a few differences between `BatchNormOp` and normal operators, which we shall take into consideration in our design. + +1. `batch_norm_op` shall behave differently in training and inferencing. For example, during inferencing, there is no batch data and it's impossible to compute `E[x]` and `STD[x]`, so we have to use an `estimated_mean` and an `estimated_variance` instead of them. These require our framework to be able to inform operators current running type (training/inferencing), then operators can switch their behaviors. + +2. `batch_norm_op` shall have the ability to maintain `estimated_mean` and `estimated_variance` across mini-batch. In each mini-batch, `estimated_mean` is iterated by the following equations: + +``` +if batch_id == 0 + estimated_mean = E[x] +else + estimated_mean = estimated_mean * momentum + (1.0 - momentum_) * E[x] +``` + +The iterating of `estimated_variance` is similar. `momentum` is an attribute, which controls estimated_mean updating speed. + +## Implementation + +Batch normalization is designed as a single operator is C++, and then wrapped as a layer in Python. + +### C++ + +As most C++ operators do, `batch_norm_op` is defined by inputs, outputs, attributes and compute kernels. + +#### Inputs + +- `x`: The inputs data, which is generated by the previous layer. +- `estimated_mean`: The estimated mean of all previous data batches. It is updated in each forward propagation and will be used in inferencing to take the role of `E[x]`. +- `estimated_var`: The estimated standard deviation of all previous data batches. It is updated in each forward propagation and will be used in inferencing to take the role of `STD[x]`. +- `scale`: trainable parameter 'scale' +- `bias`: trainable parameter 'bias' + +#### Outputs + +- `y`: The output data. +- `batch_mean`: The mean value of batch data. +- `batch_var`: The standard deviation value of batch data. +- `saved_mean`: Updated `estimated_mean` with current batch data. It's supposed to share the memory with input `estimated_mean`. +- `saved_var`: Updated `estimated_var` with current batch data. It's supposed to share the memory with input `estimated_var`. + +#### Attributes + +- `is_infer`: *bool*. If true, run `batch_norm_op` in inferencing mode. +- `use_global_est`: *bool*. If true, use `saved_mean` and `saved_var` instead of `E[x]` and `STD[x]` in trainning. +- `epsilon`: *float*. The epsilon value to avoid division by zero. +- `momentum`: *float*. Factor used in `estimated_mean` and `estimated_var` updating. The usage is shown above. + +#### Kernels + +The following graph showes the training computational process of `batch_norm_op`: + + + +cudnn provides APIs to finish the whole series of computation, we can use them in our GPU kernel. + +### Python + +`batch_norm_op` is warpped as a layer in Python: + +```python +def batch_norm_layer(net, + input, + output, + scale, + bias, + use_global_est = False, + epsilon = 1e-6, + momentum = 0.99): + mean_cache = scope.new_var(name = 'estimated_mean', trainable = False) + var_cache = scop.new_var(name = 'estimated_var', trainable = False) + batch_mean = scope.new_var(name = 'batch_mean') + batch_var = scope.new_var(name = 'batch_var') + batch_norm_op = Operator('batch_norm_op', + x = input, + estimated_mean = mean_cache, + estimated_mean = var_cache, + scale = scale, + bias = bias, + y = output, + batch_mean = batch_mean, + batch_var = batch_var, + saved_mean = mean_cache, + saved_var = var_cache, + is_infer = False, + use_global_est = use_global_est, + epsilon = epsilon, + momentum = momentum) + net.append_op(batch_norm_op) + return output +``` + +Because Python API has not been finally decided, the code above can be regarded as pseudo code. There are a few key points we shall note: + +1. `estimated_mean` and `estimated_var` are assigned the same variables with `saved_mean` and `saved_var` respectively. So they share same the memories. The output mean and variance values(`saved_mean` and `saved_var`) of a certain batch will be the inputs(`estimated_mean` and `estimated_var`) of the next batch. + +2. `is_infer` decided whether `batch_norm_op` will run in training mode or inferencing mode. However, a network may contains both training and inferencing parts. And user may switch `batch_norm_op`'s running mode in Python `for` loop like this: + +```python +for pass_id in range(PASS_NUM): + # ... + net.train() # run training model + if pass_id % 100 == 0: + net.infer(test_image) # run inferencing model + # ... +``` + +`is_infer` is an attribute. Once an operator is created, its attributes can not be changed. It suggests us that we shall maintain two `batch_norm_op` in the model, one's `is_infer` is `True`(we call it `infer_batch_norm_op`) and the other one's is `False`(we call it `train_batch_norm_op`). They share all parameters and variables, but be placed in two different branches. That is to say, if a network contains a `batch_norm_op`, it will fork into two branches, one go through `train_batch_norm_op` and the other one go through `infer_batch_norm_op`: + +

+ +
+ +Just like what is shown in the above graph, the net forks before `batch_norm_op` and will never merge again. All the operators after `batch_norm_op` will duplicate. + +When the net runs in training mode, the end of the left branch will be set as the running target, so the dependency tracking process will ignore right branch automatically. When the net runs in inferencing mode, the process is reversed. + +How to set a target is related to Python API design, so I will leave it here waiting for more discussions. diff --git a/paddle/operators/images/batch_norm_fork.dot b/paddle/operators/images/batch_norm_fork.dot new file mode 100644 index 0000000000..4bc47713cb --- /dev/null +++ b/paddle/operators/images/batch_norm_fork.dot @@ -0,0 +1,25 @@ +digraph ImageBatchNormForkGragh { + subgraph cluster_before { + Prev [label="...", shape=plaintext]; + Rnn [label="rnn_op", shape=box]; + BatchNorm [label="batch_norm_op", shape=box]; + Fc [label="fc_op", shape=box]; + After [label="...", shape=plaintext]; + Prev -> Rnn -> BatchNorm -> Fc -> After; + label="original"; + } + + subgraph cluster_after { + Prev2 [label="...", shape=plaintext]; + Rnn2 [label="rnn_op", shape=box]; + BatchNorm2_1 [label="train_batch_norm_op", shape=box]; + BatchNorm2_2 [label="infer_batch_norm_op", shape=box]; + Fc2_1 [label="fc_op", shape=box]; + Fc2_2 [label="fc_op", shape=box]; + After2_1 [label="...", shape=plaintext]; + After2_2 [label="...", shape=plaintext]; + Prev2 -> Rnn2 -> BatchNorm2_1 -> Fc2_1 -> After2_1; + Rnn2 -> BatchNorm2_2 ->Fc2_2 ->After2_2 + label="forked"; + } +} diff --git a/paddle/operators/images/batch_norm_fork.png b/paddle/operators/images/batch_norm_fork.png new file mode 100644 index 0000000000000000000000000000000000000000..aded62bce5bc268b7a3ef4dc96c89fe21d6ea955 GIT binary patch literal 23873 zcmeFZby!wg^fd|yNQo#2ilj;-2olmDBGN4&B@I&2(xE7d0@4lANOy;zBHj5CDjhE< zyma09o&)Fpp8MRt@Ao~={r)%}MfYB7uf5isV~#QAeygk~eSv_I00RT#g3O}_su&np zrtsf6JY4vR)D6q0@IOpPRq6W}1>ICj7#QLhG7luwUtl69&ySJ~js5u>KP7SL3eMLJ zRs!-H%yb0)5=7W?$?#*H2Q2PJg2O z7Yp0;N+UkT+4Y7bbj*a3x0P{Eug`vg>0k108XKPZpE>y5!6S(X_(aZp?%(n7)ax)L zw7=P5VVq6@hQMoYRwj(VJe@PEzHdt2Z+=XS)5oAAf*}pFaoW#@yuvB2{P=G?2FB@} zNO1jX^a+Zee!Ru!8vozH|FHg~&%3PT+Ftf^FNv(a;MV5N zP>vJ|oPb*wrs{ORkd?4!N}-=Kn;o-WMq|;9MK6%gbm@kt^nWja4(5vMME>@4x$)8o zNg^_W{!pJjED7^VSdMGG+P6-Zhc1*uTsexP+4bywq@W+)RNVaIfA6FQuBZuYT_4f+ z$e(Z;P{3IzahQ`8_#Jq2HZ4Sy^~IF+UC-TLZ#D{jj^w_F&?eh(f2a2qe}&UO{2)P#Mx-VsH)5Ti~zP{-zUWSA6 zFHp-;bol#?qGQr^`u2!{k5?ZXYB4ROJtr%P_2h6tEU5eDzP z@YKtE7rKqvo)o^0G^l$1X|O==MczSlBfoKZZN}9fzQ1bdgZ2=qgvZhZsqr$uH;s{J_2m0mVe73_*d0Cz ziJ0w1qEgFa2sInqB^&Tw$uu>JaLQY0Kh;L5{3hhAqZ2#cO8}m%VEO4wJlG(bTx8mDJz-7o$lapvq#GAWM$^nERWrku-pDpmXr^v)q4F-RDoV? zyIH(VRLpGUw0HPIl7}K{ZCKv}Y5Ha*q+SfwE9vn2EmQl(TnBRK4ivx()o%XVwbr!N3f`we1Cr}wKYV?w)$JaY>LE7 zla>6+isT@Xs0^zmE@TjAL($91Q6s`@cbfg4x zDE!CQ^Us&_+f`Yu;{3kyCe7!uNp=f;$0Xh<&bwk$A5mnPjH}+8ugp!@neyo2I?rRi zGwJHOJ?-PuN)@{LY|3pRQE}$`2WnDZ0se{SFH2i#JQgq3AS@N)@3ac@J+GK>=peg1 zz}K=jLwm9qgIgAipOmn_wIVc+F^^iGo_3py)TDCEScyUG^U)44Pj_1pmvzs_+u6R+ zbDebV{`H2|cC*)1&wFp~?P24=noukD)9$-H!ZX&RMPI6C!{idSkq9Bi6nXhH=fMga z#MhlFPiB(?HU3RRw7s_T@HxzdXajG1^4m{lV;Bj7@f}vpk8CNAbP|n1ObZ*2YmD{D zi&|---Hs3U6f12fsT}Q>@(dcMh==FmDFUe+{g&&t$E~)nnyvMZ&BO{NyJ0WK1a*G( zWPI*K$z$f9U`Rtg&x**B1t z(-l9`q(c;m?z(F#ZNl(n&?{C$k%$excd%NDyyM6a7t=ja?O5Zq6t?T#BeHXEUd6Z6tcy}e{$g;AN{kER zTT8t5&I|zwZ6lALl6BZTyZy%E4jb<5a>VT@{~qBhMG{XpxCKa>^5NyKW=B zGb!S2`^DEf@Afz~jvG&HBkYuvqey;9CSc26inua$V=`g#cO2?NMxyJK_LDM;k6KcW zQcMV&%EXwT(x`Grg|#%lFE7s{)&)c2J2g@c`i_rwXF{Y5xRlnGjgq~#FOlxP74?3u zw(luer;_&2#mV8ATF8YAIp3@>9_lkzVrsP3JvO&^Q(1xSao%&LUz}KY>M`Oh!KSjA zT|8xm$0x_@C-16Tl2fxMPwMuUa~%%?bWf3k;dbp;C(oC=^i_4QuXv1Hs8zgZ2P$_#9hANPy|-rjJ62Lw2^;=N2lzD zl_4CZ=hP=vk+^(O?C>d~wkFP9Wv`7XdGz>V`2`(?HeuZIm#_;)BKojRSg1TDU_L@H zpHJrrY_gw?oBa6x;ksI=%Y?o3fnF;`9FKv`tz6gkP8&~zSWoSGgVps#@;FU}0jr|j z+UcY22I1#x)eG+|l^ze65J_VA5nN)PEAA8yE~ZGArgm0;m*gQ4mv11o)8;>QTTThv z44tcfHbRJ((ebxDq09VLSI>~lGukkeZATSbLDKee^VnLShy-b=<1xg~d0~_$Kd6h% z>ytZ!73I`Bw~tN1zlpVLNTx4k3;U^3cXS5HZ}(-mY`b^yj@+VCK>Sc%&$x+^?yXXfj2yQrnbwuE zwZkV-N@7RqG#;y5LWKvUO8XWPv|JYb8;eeB!nXQ{QW$=Vgch|LB8rv^1Gcc`2iAOf zmh4x5+zg2*Hf|kHez@P`b3DV*JheAstUcY}WfLtt-q6W&SlN0VS+nr-<;4mcr>z(J z5XljHsH$|NzOyKDc34D_k$dhORwTz$#X+g8s(P;+l1v|NbJ+gQiUMsbq(Pt^<_#4##>z>g{`F@Z%Pa9Z|^l6tgS4@&(Cw{7(EfC zBXRaqt@F~ByVOanrp|rjHnvwbDUlEsR=3$LwA~#wztmAAV5IHp&hB`6k#@ha_%V3!^4Ke~HllS649{HN)Dg_uoCDk2Ae z_D0=zGHSnOY*{zBTmt2f^Ftz>tFXE|_(lO^gR2|3MpZ{2~*WV;x+uBRliTcgC zUn)nryCf0*>I@e4_D(O@Je}{1P1SowL@TN*kGMEy8m%`w<*|MRiD*R~uW$CZ(VT+D z)EiBS+{sq5LW&7ImSKAD8l>8crINp#{=Huy!SgKkrgt~#2ub3l=>I{@BwPS?0yVCF z`v-#pD>g40ReU=&v-FD6K; zBbd{r+&k}rPKW$U21ATP%fmzWj=^SzmeSVIGC@ z*sJJ66QN9(JVVl&dD%bw3f47j3$-ZB*I)iC+CpJ#N<1rNKP&6#&~+c~Rot6rRjmY^ zNSvImlGDA6;fJpKu06f_{&X+nVWW3@I$y`>alypqhRWL1*YM8iaiJqghg*h@vlE^k z50W2HeEx4T3C6%LKu?ez0VRTo-F)jg;GMR2SJ;}>Z?*f2OW_UX2bL2p$xbSu@%BVTv`@1Pj}B5yFROHTGu+3!m9{|BIy& z3P2!Oqu8&WtwZ)@2*V{mNzWEanGZr!Nnh#HvzL(~3sz3BMW*+P&GVrq3F=CS7s7EJ0e8wuwg#&wn*X{L0s< zjDm76i)e-r9WmZC{r(nYdbGbaRH9#O6p&6EFX*z$3>W3{-9ASmp27p?EW^Gtwgc)V z)4^;NUd54mkDZ8LYMJu%%yO50!A*7 z0Gs5Ggl}uKXv%QJ+|MsBGbCQ(@_vxLmN~Fkb_;eG9iAVcpm|;SL_uehpG|)K5YZ9_ zxG}Nt$k0Sqot}gvpJo47Tpi~Xvj%~j86LU;OBvXS;v`rvaW90uzHr@4ZxD(A)6N(! z?JPqeie3O`w`m*WN(_Z!)P?JOCJFkjOFaLK#q9U+17%w%z2q$fPMSV5@zpzY**A)* zuvs5NE^Yd}KHb>JV>8Y(bJ;mTQ4)O?{mFi~e98VY-up>%^O^T0lUQwXEPF_sghh|b zKdt`x)sg`)bSUa`i_?D>ERGZ=ms#|sewSLNJ_IB_bnouDOX4YXa76s8>Xh=%SH|g! zjTpvGb)y>89nd@=H{s%B(WRRY2VZ;{1%SDPsI;4pz4j6VivfL%u9Z%U`nOaudPLptAsspB1_KA#t* z&YXc-%U4_g+kJbLW!&D`p`Y%9badMd&xcJ0n5h^(URIrYw4}ETJ}#S6KSV!vJA-oW zWM4$LMvPTV12ZeTlzhE=B z*W$yv!UQialXc`}A`R-^pU506Ep`Z2q>B^PrK?3P#4h(jICB5q!N8n`6ywQM;D))T z6GOH8Y}=3-(u8zVz0mL{$@5E)soV&@s9wre_UmliBfT36ESVmfC{fPWfGJqJMX_t& z*ym)Ix%eFlJrAvSk^&@w%d}SF&SLtT$%c9FXNBuS;m&T)0mD*&)$o+|mJRvsa(x$#DIu2&g<(i%ZUXBt_BrJ(zxdHxIt9G?-v*{qse zG5xux-)f&-Ko_b52dMIr+JCcZfGhSCJF(6N^_ze=QvD$Y>n!X1(Sxn(YD9jP;PGb3 z>6klgzLK5I%6A>EXdo@cI2-i;Wht^(4lduQ^L5*tZv!5RwRW{6XabUC$;t6SN0P@H zOONp8d-PFoM=jEa2yX<|9c?4HJ$IjvLEgM%x&@0o7u0VN)dA%(H*Fz0n>P! zijP3AjJc7L&hHrlLfQa^=SPUxu|>yoU@v8XXT9v`1N9xZ@pmGD?NQ@7;5EaUl3v^g zCjBNrt#qDegUnpifR5haxS{d$2eBX#i_s0j;h}W6` z4r|GIVUf`UG+lhOHPQieHA|1!Nga>Xu>4x%iT8C~xQh(mw5Q|25%7XhrX5i-03mHs zGQaeWms@?ar}F3>?D08TF_3zh1tn|N>kKm*r*X@yHaJV{Xp+bixHX4S!#8b_n))ng zzSFv5%#7Q#o$UJI5!Cu6PD`rS9~EeqephsaXXyailzD4;sG|c!NcS>V2++rpXTT7#+Q zzF)WdhkG!d1Y*eg+_#q(MYZ^BCzz3l%IG(=-dY9K^OQQlVkdh&dmO<_B-uomq&#Ga zuZgLb7aN&xk&fP*Z)aamEzl}%=F;&sUVASv|Ey;ZCQVJMB_MW$3a6gRQld479LOi; z_@=`MnNvH%esqcY@}6i@mcy~l}7>AK2M^7>;KUFOyxTP=^fXogxtag!S!-J zR|L}AX6-A&ty7QnfIrz-v7xn4H((D4_yk}G$Ett&6RI*&3pk{!HF|q3eBhrdYMJ)h z(bE~8-F#L3)T#4Z-T<1a<>Fxor))qg8hpt(gye&Fq9pKMNs9MyxflEm1C&j_J>|ZV zkBsp?(Ef9ngzQ=6`**Td<3`^yu8&n5p7A|1bqYXj+++6A+U#;G%T;mc0dnKIqtSwj_+P-ZC5 zSc|QLAh2CXbMb@a4CC$@N?ERj1a7Z# z?6mnTdc{$w&+NCp9h7a-AZ*d&=Nz*2BM>kwqWAqBwT|YK(9@1E& zJHFxDbOOZ0P!S87o=Tlmw&1o?`yqDx_oByE*7+9rSr2ith)0F+{$kn&oZcU>zir=G zwj2Pp`E8ptv(U?`-+muY#Xxez`#Y*&{!c=} z16X289PYbMJ#{$_ml@66nUW6u>Y2?gKpzoIZn9Yx#af=9279x3D_V6@Ri@(<$9m`L>}2b=xI>SW=#9+`wh2!VApxZ>FkCc@>~*t_eWq0C;K<;G{j))~Kq6p@ zw}XjcCp)nRB@eg~uL>)GCLV?L9Dqx_x{6BnITlzz846{Zu=aq2ngNtT$-!FfKPbhq9<&B> z`HUhm?DkvAToT%`p4x!t>3+)m{Pcz<4Dw2Q@-d33ujc|K&z}f`7_%v0W=+;6s`+}+ z8n5YGd!!*Njj06Pb9fC+ZQ68jZD?{6T4YTl0aEQ_o$R)X%?92K4lV5w)v>Mn`!Gi%_c2P7wT05U zVQ+o|lC*Y14R|RI*l5Ut28a_L z>-A_+AHiSBW{%7qE<;Pxmi@QJtV_F0pu|q~VxsXm{HsCw2jA~1dg@gU+PpCJ*G*tb zxN-C0`%pC8H0z9^KTqwX%tNrI#!`MXF{Ej58^CE8sp36}vdvMD-Fve#KHkRq1)G~t zNl*DQkn!SvwBEa4TwIGS`fkKRR7S0im113arf=%iCb1qv?byjxVmm3=7ft*7uvxN; zfGGbKyffL)Hp5pivF|hdfg##_G0$0_=mKk)=d?BCV^Z!%Hxe^LC9j^+1crmv_}9sv8*l6WG!tT*Xjk|(9?o%cJI>1kaZs~J zhyg%MEh^_)fVdMBG^skS5mcDiH&|g+X-9)q%WucJB-9Kj3Q0LB__J-V`5-^&K6>y>|X=J|P|M7_ZR?Vfk3y;ujARR#hkHGN< zd>!Krt;c`fh>d#xpyaou`pe7e^LMYlb7KUPfw>+Eyd{OUOAwtN;aOZfLIeEwG$P~2 z*2?wnTbE~ce=7J+ccA85Cw>7PKxmMtd;|8v+os0jEzaxjmqLP8?6ijqy67thH9FQ! zqn<<@wzG&Y6weKskHMzZ@nb8fy5B9bqg%aN(y=-{T3~_q`2BPTkpzSCt1xsy|LUVW zI?}TG;gd3sPQ!uvetoWLCODV!3)3|AZoT}pMcX}bSSE~OZ89vpuR1>gZk09;yg-{= zz`SDPL&-d4Mb%I~v%}q@$rQot1ON6gKjtz#&kyDzV2%;q*5I(%=#{+K2JQ-O#ST0l z9JsHKSKFi*=99#IbGdHFDfC$TI`8-6-@_R!Il=qUZtxdYeQ(A0^1rF!nj^Zj#P_pn z*n3DU{72kvDIY zXgzXj{}X;R{L5GmHh0Yb8 zlMPLV`+6(Nqt%*Q*uX+ik3npc@>uZs6+p3i@jcVS4f+Tym@#v1!2JvWL>Yd6;LMQx zVEi*~RX8SOqGMAYxft?(#Xyw@7PZr2P>mrvEF~|@D%n{Y!_OakzB|zh5egg6c4JUY z30<2J#wEY);B10d3WrF-3|6(1@4W<6^_Q`fg#W(s>QJ37NDy$$L=#qA!58~3|J>nD z(=dose?`n0 z440|^-p;{BjY5Zyg2rR_zGDqS+Y!W>EpK(?NAHiCp)-878S-tx^=p`qH9MJ=~57DdGfLSy$KCjBc z*oWH;2A$7bx#*IBD4E-oKtlT+o&m`pX>pHVBi&L~vXp^(E@P%>ex{uzD?xSO7E#bg z@rK6eJ<$Bd_-(JlF-j{p*J!>WE_ven?7k00ArOlK(WkWHAI+a*=E;#sA|CC=AtF&F zbglNQY3=Bvfa+tw)Fx2MYn$^Jkwq0vksLc7WnFdkt;W_4tpdLVOy(^shG$LRcKXX0NqY6z6snG z!gRmOn+-tBZ2)&U_7h);=fG~0y&cEU2n8eJrFW(b>Q%}B)E+>!g8UnnWlE? zmEiU{_T1Y{K4AwtL=hS_bDR7i@gbC0`*h{eNxl4$joMM4+fnWk7;-Mvhs)ki8HAXZ zLEaT+aB_T9x^0twMDq{X`A12dR%uzUWW4;uWA^~c?p2w`>!HG%op(28!Lf@RVv;`TbhV9M-bCp&Tm#Wuu&#h>TqB98WxVbL!tXz zVuMcf=1%~bop%iluf3-dP#uQ*+AvZ_Eg?Ifk=}*f{qAxvX|9WpvYLUh;2OAoDqN*R z!L*aQVV?N{p9<3Awn_yqa!IB#T4=3Gatefhl!~Wx3fiugy7Vr-*mZrXA87AqcUOvl zn$@`84g3*vA{uV6jGL!S&1JJJy?uP$VF!sA!L{I3XenwZ>E%$dt)7dD5c<4iRnSQA zyjx%){o=e3mK5aPD+-zu@5h6srq^V>U*#k_#~{9ER3*fMh*W}aF=H*#XGZX2!fmTK zQ*=rgSvD{sahQJx^GG`%a7Ed7q0en(lwVhZ=wU-RY&;dcsaytOq|fPF&po5Bff3R% z`-`e?0I)TQ`2p{vEqZde`JV7OX9Oy^ydg2iE8-)on)m^m$MN369NJgmr(IC+9;He9 z2ruG3WFoS6IP-lF;v#4cwwNNFuHs{2A70u$K3GTg++yHgbB#`C(!KDbOkn|V6!Ln^ zIb5P2ZqZ*8lNwSo*U8}J&q7p zUsi*}Leiu1_?K`B#|ntcbY6DxPfsKVgj-o-qbfN<1m>^00lzf&mMIBM=Qb6@>uar4 zY(bE116Js2{FbRy`_0fjeji!gq_I--4*qFRwcomFiF!a8uM(iGk0{!uMy%UU%8KgE zqgy1WI|pcU7{5tBZI58tK-bq-#%(GmO?p!#-U=DrHR}xQc{i^`ip)KTE*;i{PboY*AEJ|60Qa&2k zw!LzikKL}LMTQ?ad~yg2Lqf}Jlmz-6dqi4iW*wT&mvw3ycxj*v7&3wL>WJYo_SAOv zQd9B4|I{De@F+FWH~cqKit7abmEwI%sq1W4B5^Ey9>tZ7SoHhg zpj5&djPwiAk?$$_teN)*Sko5?^MLmfjU)Q~l+r98=TL>lZT6DD&V-{}gTBdC1wX68`p334OPQ&JCum+7ZP5jps$wB1 z^=`ZI!|g=j(3tdF=;3Hy{KN#QzYbkPF=)5l=iUQGbT0YiFe|}rK6V3+IR!V{z>S~U zPydJGSY#nGNdg8&*9VHie;-ss5jPO@ZHM4W3}k=#B;o4pav$)D&H{k0m5;e20g4Bc z0GMZ`&iOQ6J_t4_nT_WlX-`4RG~wX5&5SNzTn|{?Maw{UyYHnfnsOP>;e)XE7+s)* zcXt;Sh-*=mC;HQi_gi&`FHtG?%1@Z2wekfI$82qKz+9OZ=4r*aTt zKB{!e&~~EAUd3(hF6n`ysI0sifuAz2X^oNd^593%y4>`oCzlAlOj0`-=|e-B&_DSw`@6;vOQO-lF-y=fSdeD{G+gmoO%v*Z zBluw5#j@Ym`p?fVE?M>>)Uec?ccGer3fko(#8Wx6L)fD{8+ROc9Wp=^Q2M$+%@u#G`P^on?8OKHs;FbY z*p6Va0&R5bxEHs)@ z5lUEeUl3TJ_6Dj;2V3g?MJG#^h^MRl!00nyYR9k;;0r z4V<#=;(73l&j+{H7%(!n&%C6CowjhR@F2M0NyEY|&I96wbYZ?!`S2K74#C)S5osqI zGO+5o#m3muYM}Bs6cb7UMfOX5UJUtgx43>_r1-Xinr9;?@A;!eM!PU#o!H{iR zdDN96Mud%?%hSd77_IWud|0dqQvwcCtu{w3Yrb<>c-aMVO^l;p;!iYOYly3iTXD%o zsAtynyEzH7zU{j z(o98A$KgRmdm$<4CsBa4)Qec9e&a3B<$i(2)@j=+sMN6a`QZJUdClZuMJs*EK;p&hWE?d05=sxeE$m%lvc1FMw4Ad?jjvOrZRrG z*wK47ces9$ta3YA+iA_AqP=V#syb0x|csNS8L1EMlpq z3NZojF-fc3+IdUtWK)cf*`8?_srK?7e{8M!aOZ_~r>g4ZvX*_+k}>BErx+UDpfQ@Q zufLreKU*d`3k?va&!%I3u)H33HJyok^4JOC625l6?Z9717%}ZBjKZ#m*$AOa4cW(( zgUkC?78ljp{anpfJ72^KQa7w-hF-5Qs-sNVXRUVXr1 zi@513>P57|IC9&K+UeTA6Tf4UQu^`1DeSo-OlFR1RSvbEYYQ45P zdel1+t4K8PeQy8Wz{D4OO_svORm~eq2zSw~xC@%Kqj!`fzfQa@KPjmCP2h3d%g7uD zi(lax-nv)3J_{%3TSMB)Z`+=K7ylj;gB@>x7qPQ)R0FHK?~n_xt@5X2Ub?%QEvZ83 z73KZuNRcqL4g3LN{SwSqxV&v`x*|+I`P*Y6*Ph)pFO+%7>iET|w6Ag8MG8TZ!L(WR zmAF4Vs34aykv}=YyJ^6!Q==hqNOy7}Jm9vmsA)$;Y*JC@4D|t@VW#D8Zeh3U9gj7~ zyA=`5F|EB7Q(NiD4fXTJzirR2xkU5`$7cSmF<27)g!Jr88;y|4ep@@8{{GQ^>Wo~u zl4V7~8{E3M{v9^gV?h}~} z$igwZS*?nb(>zz6jNt!ar>$EPiYt$HRAt$wzAd>Sz_REi9i6;>tcHJ}nC{PA?=bXo z70y86mPMhW&ra?1*!Q*l*BO>dW^b}iqCJ}jNmFbwGWG87o0#qum=56PWZFC^4-eP> zXx0BWAamAPF_|AxHEUTPrLf*^DQQ{R5VjeNSnyQnn$h;iTK$|`P{NsC#gypa{$jEa zC7yE-=|X%xY7mMNgr0j>9^|2nU$GaDoKbp;g;&`nV-CW7aJ?MQk7i}FJzpM_JnS#LR-SO zCY16WP`oAV(#)P8cqoPXn^R!m(9d6r%5QMM<<)P`WP0F5$az%!d_(ZzQK`ornW^vG z`Qf6!9X#kcy3FFH3x5TyX)dkgxz?gM${SQZA4xG}v8ZJRHoU#!)Q^0uwx8x` zvzF3I6)**+zerQ^i`^3w6`Oa<|GZk@?W`1NZ-HSdcO!$(~=a@$g$f>llHr+w8^4_tqj!lUt=%%xT4BGYY%0GmJfK3a&uwch z&eDQyUO@uQaL4ffUbPy_{8FAMeU3+A&EROYLhkv=a*<~k?#H+?>Y(_tJa>vVuR#Yu z_2b)D+UgTm+^v+;W>B{8JUlcP6qG1?E&GKh(_P}iI#%XC`KdOrB$rgX#p32}e@r-- z zc9Jy{Mp!gmJssW}1A_6m!xiu6O|cPjwJ!AbF-zD_I*_<2HDXI8lg@)1E897GO=Kks z^E01pSXw_^H}?ux`oY<-o4QDCu^CmuA!@8=SzS&O-tl>0r6hcZg+Jted*No0!O_kO zQ%WSF;LhN!WcQ&Zl%^*1poj=86&6{H2zwoDU$Lpp!3?*Jt|3CcW*+d4O^`0=dGcL# zU1&T}8W5hI%f_}=Cq#wlm4sR6=|0ovimCxO!UcB85)VBORHt9ux&9d$);w{kv`}n? zZD%|nAYR^ptAdbdpu5H8q9Jj^wgLw-Ny4CdIK5CQp3TiXT!GKNvxjNS(5%%Nb;Z{4 zo85^e1&*M6`0>ydH+0Y4CM^?)D|FHP)FOY*y*0rXSyp{~_dIUv#9YbdQtmjTZ&{&L ze6FC7{78o$t)qX%VGh!cE}TR!hncu1xln z>DGK;YIM$>B#UXi;gaCO@Cgx69b{IU+P74WxaQd0R!to6ViDmve)QTh0XYHAz3#@7 zBfDC$#p67d0q+dcfi>qc-SNa={8VbkuwpxhC+)TJPaRw~R!iSB_?h6qDFX?n>pFbG7GH zT(vd~CV!#Ck&KgE0~gn728PL~EE;dlFHa_xulkg)j^hMw*Bckqb?7I}V$HI~1^ z`7-8ndfC*+kbO&$x0y&0O?FnCCht%g0pu#bl6R1XYq`(QF0%HmjU_QNJCn{1%$v`uU311;`aN~X^o=$y_(&38Ek&$OmbDpAZpVa zWtJL}RupHBd7+)L)&KZeG?i^7Rjk=y<*Iz+)7Xb2_jEhfTufDFuoDj98treRB?AqHBRxf&v`+bZ1tRP3W zjk~MxSlw7*$x?)wZq7;`PX;A^+7t!`)jIlH0IclM_f~~6^xtH!31OQC*c`Y#LsktS z>q-O1B)Q~RMkJ<-?pD6ANn0!J7L@Dk$-lLR{mRJHSU*s8*i<0rcM(mdjC32`SfT90 zSl%?%VeMg@y?flm2Z}5s^z8@460spdvKc1%cAAVQ_x(p&?IVAGcfS#;;CsXxPdr2N zgPml~>rD!~V?8&cl3$V7=PpOy7$QGq{33&Z&w|YIudq|=)?d{mm*Cl)BSVWk|JWq| z08usnHYE1UL*Uns7T3Oi)IH^;Fre!m+car6=Gxg6%pZW5vZQJ|W6wzbKe~nA@%9|r zW(bYYo9a)^f-9SV!|$l^51v6cQ;sQiJRxMS8^o$nI7ob4pxfp>!_U2Uo}&n?i{jDO zER?sK&Rq;VzDXad@N`7awTo|j$9`p`FdCc)dks|NzTyekXw%|;2s9jgBg7_;xUE-H zwyXNxNpa_^7;rl=1%{}{RuJ}5PaYi;79AbfQ4L-;>q(*>Ow|g$l&z5)wdiKN*!$_l zS9xeW<;qoleExL7QqdhAUG8avXT05A;O$~Gm~+v3ESlWwL-#Y8)vG)^eI7|?Hd_qD0d=9CF@;#v8idfM3{ z-U(I{qsO1}h)?JA0NvIVxAidRjC?eMA^-ol3@NreLDcoCg-^<5fes2+{o}kcJho0q z!6AQbs>a#M5?W8&?=;Ynm||9(Fptofa%TP4$?&eo=o^8In5z@iN9Zpd*gU~~APvi! zQWFQ>t@o6?pTB1~uTP1X?n3)NUc2gp-OJ8!m+TR`InL-Nu%MYj^*KcD13K^9V@EZp z_f5f*yQX>kP#roMFf}Gl`xwsT#^1M^wrR7*P4xW9t_nDqbblLyE-PGOD(0?u{vLtb zq_5EK=#+iS{`9vODwc=4JRMbAm)hjA7|;fUiW9T};c|Kth~w#SinJ-J?PnfdyK5#t zYz;qk1-+k;P|d`r23p3_XCBuV^!WLsF+1hPa=&TZ$5|nTKND^3zf$&MNigw;qOEK?BN~I2BV!F`(!YP*h@=?zN zOw(-1`U{Dk-_nwuuj2d3k~}>qeD%bPTJHA0eGaI9`y5*3>UB2=AH%a~+0~Yz`ES%{ zp;%6O%OgB>f4D!xAot4R8}SXC52A+G_-zwtgXnv>A*Mv)8@Mt~=8P7%t7EcP!LYtVb@nNmqr()wh2Kfd<@}~zE9g)fvcL=LoTLqmK z>3}RCa8zN@xr7z#vzZmx*(}=Ok3(nz_JmI?X6UvD?SI=Ih-5k*To1T2_+Y7{X*(Ga zffTD$VNm3P=jhGbW{kHs>Hsc>0$;5fZG8tC>%s>$bwX7YiLvq#{p+7Uj80lW=ghn} z`+))Sn)~u#jw$DP9I&Dwd~DS!P9`2k-jj+6?khO?x9@>nrssOiNoXCu`oxUfqy<$g z{x`9DtE`h|RT>v&J0Uf)c{6CkUN1e8HYDUQxW>DP*HP>aob)G_ptw+3CS?`ZcG=d) zaS2sJI+W55Z!=*SbO?+#5U0Ho_M1i7-W8f@Km6a#w9p#BzRde*?<@q8LkKL+e)#yT zc_04#@gmk>^gmikS$vknFoXp#tWzo6=U7Igz)+pvHB82L}C28kydq+O|c zTo$pjOaOu^JNeKL*|SVAh|UE1FC%ErG66Uj$b+7lFq~zAWtfvnp+tMgX(lL!FZ~2b z@s?aZo0Bul2^~?+v@k8a!4Kb_VAov+jxit1rXnRKZI|I71td5RYxkF!cKFJD`Axzc z0*m7|q5TYnqyOR;jvJVaX3^H^Pm*uiA?l72qng-H1_;#RvuN@GR zbDU%hc!?OU?HhSv>y0P+newqWK+*_>6;c32o?K`FV&Y|?5s+2b(cKJ(VB+D15Vr)q zXi3#siWM*Z8ueA2cenC@Z+JwUyC{6;61MGVk;MP(nwTorsdmVqa`-`yZrmAszx!JJ zEd&=iY#F{QOer=$c<6NCvq+1HW+JZEkCTii+$j#iN2smyz> z?~f=X2j9>{P#d4#N;?B>=*shiR+0*8f8vn}e6%+Z-50^_wqb&{7tMj# zmDe7q1iiDq$i`POCK@VO=$40IYN2;C_ewyL8h3nC!Q!aK(oOA7vp80oHJGcZ6fAc9 z0^Oj1ecnhMM59PnW~N=#K~NmHnl*$XzdQ@W^fGIYbkEa*91S!oTfhCDtdgCRb_9F$-Hmz`FIFrjki9A*NaV zontF_B7U!e@YIb%p&J^u^W)=9YDccQm|7>7s^kUYLQxi+kUv0yX9s|1f&1nnFPdQ7 zUej_AMK_;xCX0!=DR=kmlV^=puTI``HL6b?x2;85Q&5dx@e{2Wj5xGVFKj?N6FqoQ zAePpaZczA%!;3#>XhzRP#{8{e6ZBLB2VafrdeV(S=t9b`t=-^ReML^pwQAvljd9d%`?VjLq=)zXxkKhD6Tg zma9s@mf4rcD`5H3TR~7%4OGxb>hiO`g9u&N1heTTTJ%^MH^w5t;t&kLWADYm=c9mgw2EVOz@C=WWR+&F)=vlW(kB`peVLsM;+;IMa>>gtJm~s9?n?J!A^clvVfYfV!zMOK_lAmI~ z9R-p=QrJHfoyV(TA&XM;o(i!YrzXtv>GCZU*s=^w+Xk#M8#z3fUhh&D&MAy{O08Qj z+37E@Ge$2@*u%L9s$dy7=GqcY_X@L2HXYB(a4`>0xaWGy>eZi8w*oyU)^BXvQk->x z*N!*}EeYpi7B+*Ax57FnxmtOaT{|Z^Tb&-nheeKkT4k%8&!=7hXW8@4n`+WooKhQz zEHYF1?zGU2IK2??xUV(8MvA$u%sM7uABuXzR_j}f03YE>Oz^YZzXxt>e-K&Y?XR4~ zS!%vQoX0{gGO;>ab(mIG1RN`|v?I>l5?` zArQ8=J>Snfr**0NuO47UN&=cL^pi& z@BV;XTv@BxWh@mySRwast9&oQU&L`O@REL6aXinZBNK%d^8@*a zhSF`iQ29NuRWz*bDu)Gp#K&T|{PckH(ycM|2;XC#@MAWq-xDeX-=1!OS>${5Ex9pp zU(BJ2qLT@ta$3sESdnz_TJXy|1^4Nj8fM;n7IV8Cu{X0USFlg6upiIUz#b@n3GWkR zRy?+V8&dauy_hZ!nGni|nfB$d$-XbqC?PGO6Cm6d`H)E1-g$foRzd_b6sLhYWmrb z#xUenO^)2xXC*C|w}TY*R8-G^OtJX4qwTrnp+&n#EqQF z60RFcoUNHi?li3R1JP1atl6^CZuH~Cy_H)!XMRkgy5oMHe&L>T|G&ra|Fh?OpYOiU z_xpK&p7-nBRd*NzsOMZ~hYf`s9{RC3zDeQ?1G?-k2rF&=aCLE!A?n?w&LU9p+1=F1 zND?kEFu-GXPaxF^uW7ZIP#q*oS(j>KJ8M#j?x>DBS!8k04FRA)bsh+^+`%{Caqwn2 z<2p8iPJd!?e>9+Q(kn;kga|Xt#~L3f$ocI8Bh1%24*`$kCPZWi#?l6~GWdL-1QCVd zs#lNlH~4`!V(p>VSRPRGhLxp#4*M2ir@DkXLf5DuD0K=s$Y-GR0+?wVpz4g)x)MEM zxnLR_NAm`p^Fbhzs`FL}mj>`{s{|7CE>OBBmzvD+KjWayV-%FUhmXFD-gBa!#|186 z_1}G4zaSU!sxX(dn@*hR2w;k`)Gd3(zw|a#9C6Mvl)c#T2aOAIMZ*;mQA`=r$bi_x zp9)zww*4hK>FLi1^>hNwZ3deAk_THr#F|=6n~f4-awT{pRd9Y3`&TQspqe|F0dB6L z(1DQLFNJfzQ@v!3oej7RBFb_er=7(Zo4(&@sh>!LH_|8~o?sY!C%Cv;BoDh{X#7Is z=MhbcH`Z?{^njSQTc>MG)&%kYgb0sXQP>Z0TfJ++hvNsFz~u*d!?s-FaoE0NrL0G! z@w_C%03Be)=#W!O!tdv`v5!$G^LC*RxbDacFZc2%(rVZ8Y?~bf_H?8Io^Ei3pJM*h z`m%69OeC*U(C_yRH?}{uE=Y{bpN7irHUI~?Ku=s3_l%iqKkI$er}g#X6B4U(tKaJU zm?#&9$ZBA7Qa2U-xK_4jS4~5>=kZk987ye~kTR+090F9Fr7qZO5jR6|K^<|TJ9!Sy z!FxteOm`=Fra}6-><*6Rp#cbGIoq`tkNp_IS%bkQ)@orsUIhO>op zhCS-VBxbN+>5(VejdiR!@QUVA@7Hg5JFt zHZwiI@=8GC$Z1wCd|=pXG=e3o6}8qp$Omh<5mWN<7Z&lJQyQ&ekUsE%#jh$3sDroY ztl677&ZnfO5Xd%_SSaA#6)0yXp1wOCQt4uOy)hykwS9{3Yh!{ko=Sgu8zD`{lS-io zx=*J>HC^LsqwP)_yUt*1-5NJUj`jl~Pvb$jQ`HXhx}D$$ISui%2pkKb6=p_LF literal 0 HcmV?d00001 diff --git a/paddle/operators/images/batch_norm_op_kernel.png b/paddle/operators/images/batch_norm_op_kernel.png new file mode 100644 index 0000000000000000000000000000000000000000..a99ce81ff3bf42880ebbd6a1297de3bf038e09b2 GIT binary patch literal 165209 zcmeEOha;A4-@Z#l8cLc(gp5jLl!S!bB(jo(kYtvS85)XGMzWHOWbZ8_g(zgNv=A~v zBKtee-e>(0U+?=o@0)R7*L9x1aU93*I4>{N6UWxjZ=K`hMvHC* zzEaC{Bm)1l!cgg$EM<}W=V{UN0DNWTC56*h@Efh7c0l64!W@Lvo5ve z2OVl>b%Ux*GTWEo^8fyczZ1gv@89tM6fEgPm;CoD3U>ehh5uiVzgA%MUbuv#T9NZq z%a>QT8z<&j4m)MV9uMKTTV8f!?dSM|YD@pKR8PW7OQl7uK4_mjd2)PwTg#e*R&Ef{f$Kz7j?Ne-IZIrcCCutld-We@B8E^TZ*On4XNrMMiKkL!UCYYK z9v*hKKmYbo&F`vU!Dy$+p^&&A;fJnjWSO2kc;LVrW1+2!;^N{7ih+EKjZ_vIi zFE1Zq@t?VDW208=GTjt?j3Hdyu6jiLok`}mcOg47+mkdCwc=GGBv(@?Dz^+a{O2a> zO1lcL{+by~9c)QHQ~l&{VN1%{u%6w=BO@cd@7>EX&(ui{6*Tg^zx|L&>$RWXl+PC0 z*Q9)MUB7-k!(-)vlVz`7ozHCNq;dxgc+4khCCVHc)ypW0AyxK^AuIoL6P#fKjn5OW{`lx0!*=cWk>SopDeF*`Z?AR=PpW zW}3a#fp-1x*?ZrGsK0#qvf7_(70uGp4LjTZQ+uR^3SDP&=r{7kjE-I^nj3#Ber@je z(F-#EF(;$t2f80fJ+c*OGS1fN{r)}SZIA#zu50faB%r@nN=nMn!J)q++uX+1_V}Zp zlM@q7`B(akMZ+{RO_VJ^6(2qsar5Rf5sNm}Lsv&0a`sjQ9P6tM55S-L3*8kyW=(N^ zoEJZrdTcvnzxxJm%ZOL}Kg;Let|B#0?un`vA?eDx)yzL;`yqP;|M71%=DP(qGczaR zVv=d8sj1c(w?g;oy@x-vuI2ts&kr%C4NLi^#2Lsmi|O&RBvsA?J- z8d0A&P$bq~SN_k%IHT{}p+jV-_>)BFedE{d7j)2J^r@ktp)r0l@3N8IK!Y5202TR( zwQKZT!+fv!+_`h&j^i;gJq)5q9R>f#;o+LdG}=i!TU%k1NTINr0*5ifC`tUOm@Kdx z+_Hu8WXnr#2{ENdyH8@_16(|m_&Ij&?CtA&9&Pf)d}Ul*bYh5)&_J*|t=&u&@Z9uVyMN77bHw&p59>(C{o^_(iT&7vJ4-4y2@6 zb3#>n`?QDa(W6I)C25M-?#w*jCStbb%X5^V6)W6#eN$LDDshkHpxV*fYXhr{8v^-H zX`~sv{8J*BHf}sWbfNkoHQ%OJaeO94za-_&8-@J*e0=0-T*IGpBd=PIRD_X>%Zt7G z#g9!DE8VVKxuQPuAwNKOgP6Fu4mPB5y!Wl$@6mHLj}QJ#Z>e4W&x4JAok29;8^-sg zTF#HqhPk-9a#73gZa!7y^efAN&-d3tz!9-ZZ9ro;y}+`M_}eHs&Gj(NW*d-?qD z?g!h%ZR0pEj(&dWG@RYa`0I4ZwIZ9|>Su=K-dm**zAYCbTpz{#_))TGHBl2LmYQ+? zEw_q_ildX$_g}60i*=_rAIU4mh(?9zbRr)Swwm;l|;e^rRN#4HB zTx(J$`OX$(X(ef&y9#XeN%P0YA}(sgwVC)a+JDQYdls*R#D})pP?5o%vuxtF3W9=y zLLtW!PDXpyf6Th5zHzUvHgXF$h)Ueho_PVA!b`1UvnVKe{5Uf+GeIMM{QZJb&|Z<8 z?JO)!NvAV8XOUK~+KL^2a^IjS-N4>_yrKLrTG>yDi9EjMjRbEjcbbunU{<^Y) z;E^LoW~TchxTuGnCRD$a-D2yIQh6dFe);Rod3TX85hQu{t=VGTe47E(LH`(3o&@vO z6u<|kgr7CO>^F3Eb#2GK%Ap)yHfu|Z^xm|`Z0^N)HUIav^ytaq&gYC=if=4L-YAGaUoQFV zwa!AuJHO;(B4^RmN9PAa*6G^!HjjQSzsuan{{)-GENY?95T{m})=rfj6wi zs5Y zmaSp1Aqm!zt)$kV^cDz5`NxZ&_s1LuT&s#!ceygm>T3971)t+PRfg#b{xefQQiaY} zOQbCZ?0rj1S(Hs%$;-jVSFLT-{Pyiz?SkQRYuO}|;^KHhj&s+9A8MuBf+BAU=w)S_r%yZecY%$G^;8NP;=P}ub?=W9)4J|;>Z{$Z9K7$y{OnZgSa&7cpv&TP zejIX&Gyvlg%g&s5{8Le!o0}{DfOY=;AW|we#yL}Y<8E!=zgM1a^=jc3KUeJo{ShOn zJI`boQ*l3u*XCzzE?x31H>|JlVI~bBhKcylSct``~m_RX;Q^@?|zr(inyKL!T}&0hpdCG($3;o;T7Hd@ z^QU6h)qt>n{n@bw_w=aB*q?LwQ;Agl7_*pF91;SUrKzVXfQ$NEB|@vKz%JXWtAH~# z)A~~}d61Cf3$;=s+f4I+8FaoMtxB`x znrm~azJ7D!sZ(QL%Qcg<1(n0aN#O_BIP5&x^k~nSp+_H-sy%3E?+s&31@}=T1W`BE zyUqU!N}o^rrTGIWxPA!i}UgEDWctx>M}m_`0?W``(d5o_RM2I!Y-eO zul=kMOHhrx&VAx>9O?q^m2dB;lVQN7*XG75D+>fR@@r{;Yy^r~f83Su5^$Q6TK)K7 zrcRDU)l*G1I>J5}SRX+|nThv!-!>v`D2^PCO8L{u&^%bkPJoV>N`u^H$gS@kY zL*95_T`HJJNR7!x0X7wlKBUcGhu_O4DfTLPpM6z+hXC1@oll$Ms#P6F$A zk?TnMsd6E54BSDk~i(qiTOSMslYB#Fth14+qF0ZgXa-!S~}BqHCX_|^x^N{ zXa4G-XdfNfxcNBO*~tnU+g&P8nh+eERFoiUj;|_OJ8popYt_#AS&0Qp{37dN zC(iadVS)K4Vj`0BO)+~f`)8+-ef>OlU#7$gw~#GNMk_ZQWyVHV}# z;E?$>*cwz);^zIS4^)Ubcd)N->OGa6^7$m4D7)MT6?Ua5#xjO0ZaFb%K)UI!U?Hfl0#(;GA zo$m)(8NOdxS!s^qPd0Kb%YhwK6#kisMwMSxyCNjA*4}1d6U})W$Y1&2NAUn=y;ymF z`!)zfAFs_0UvN6JRmfNsQp%IJ^TJ^@W-Un}8OBvnkddIc+@fVGZ9JT+x-@1P8ylO^ zEKsZ0ecBZ(5-i#?h>ikivKzz!)P;Pdq9bv!r>7@o>XVy3ctike*6Opiwpq205AFmT zyjvcH2Qo){qfX8bW!7n==pHOw_o&9C5xqh)Qf4`b4XT$6I&%OT%=w$4B!k%8=1zhX z4D1|C18(Fr{};pTuuRlS4n++J|0cETv8ZLJ{CyT`+)C+;9f0&FpDAw4FD%6J{rI>C zi0-Njwhu~19wQ?o5}c{4$oW*|1Gd&!!cQwid|>Fe{W9g$hsKty)`u=?$g5jY_{{1 z6&`CUa_Uuixg5wo7PE{ds6Ih?zCjVuFUWssWlHOzJW^<5LUrGhV>O4ZZEMyk&(FffZA-zk63mvesnt$^9b7YA?ryC)RN z=Ox7k1nNR$MYjwGkgHa`~$)RSl6xC-Prm zTVg5LvK%muGMC~*Rq#ks_yIxQl$ZC0eEbID(G)}70Jyq*`SN(MXI`g>WUWMF6=wG8ds)5Z#GXZxPe?(m%h| zr~^Mnv|CzrWKq#+yDJ~4jHw22x{g=v(#f?7=e$^PZ|lv_zdu29=r>=l+%gM>{3JIx!v3h%Xoy3ET)L9#fh6MtY6E>Zal0u|{mZa0Y0|UGQ z0s@5*U>s`3l=!v#MZe^0rx|z@*bb1i0qWSQ)O2_i#62Qhc{B0)f1UaXK8ckHMj0!f z`Od#)yZA#|w!iO7;tELm2`2YEhp{MVN4B$*mU8m)e4?US_4V}?=UKLI?;RQ0i)IZm zg|xM5lzuQaKHLTA=6ZU1Xd*o9JfFAT_3>#2=F2i~-AnKh3IP48RnD=wAwcP%!&DMy zAme(jzcddXJUC`RaY&)V-M1Gx=K^&ca~&(+j1RvM`r-i!b^YSPloXn90<;T>{=aW_ zNBgZ?w=UK_IV?~4`g^R~-s4%%x6hxSqADb!4vtpAS;>8`OCdk4PO z3J6BPEV;ofx5=UQ;YSbn^-R3Z98qp~5)xEYR1_1g2(@KbeoPn2 z*7bqTpmINdi0_8N=m<>f8YW>!?WrVIN_;J)( zRB&Q#(TETQW>6gH>F881T^gmC-OR6b96V4HUz^@eJ9>P*#lPi1jUA92AjEuOZhEFS zNFQ-H%sG4T(4oQXR=of;$rnXMn#npM0Q$or;irJs?z2f*-DF$3bg3<*YR=iveWUs2$_-gyXPg&KzlBit(dDNyp)VgleneZEy>j>ySe7YniBw3f z6y12R9`5hmh$^jQor}OopoM%4Lhi=L(*D(&6y&~s{W|fALCUo3=1TMm2Y**I=I0>m ziD<$^vFrKzbzHY@x+z{AkN-YY#GJ$+1o~Yk*^oHH^(p|Rf+Ikyf9<_ zKi-D+7!RN`{`03g3UF_Ke=y&V+t?;mRbTLsSkmTNQ}tDvo@3d+1I={gT1S}upmpEO z@baH=$_~_mRh#$rqcIeKYl9Xf2M2HIDt0~N=jTUi3>NT1R~Jcb&;`_(D(%{YOJB;! zj0a_lmwS|rWE#qNsL*-u1}PAFnSy&{!v({AmjXi~hAk{?I% zASDHGD+?RHS%-M|S-Mjhz!(+Rj{qo$Yg{wiP1`e`eRTeDxHSJY#UvYnXA|6<19YsRqU+Ft$CnOf&l@|0dq6ZMt2cSdv8+#@DWg0p!uKVo}1l zofZ%^3EannOBwAbPw2_MJjtB>Grmb7!L5NUiJ6!iSScvjNUa!V%OZV*A zz=KyzfJXR{>J`kl4%+lQefp>Pm7YLn2M7~GQt^QzdU$xyGcc%({p10JM}OaCZEX!v zCg6M=Wq$GA-W#+GE$Goh_j_x?P7#@|9ZCuTpCxVio6uisg-!Zj7Z18CM9GZGR~T|ce@8$xi= z=XG0{Tg^yOxo5CJ&w-R~{M1GkmL$+%GU6f8hI?O6f5*p16Butj-K8s6_QxvOLw*OV zzo#nw^7?XUqCC6ZFL0pHK|T|=>A3@X<^QYG*Z(2*u6uguS&*I@@_bsJ^Urb&yhr=Y za{6Vq=WKxnGWOcg6f5{5nvwv1>aA6NJ|h2ZO^tGGZLRmcdl;cSG&C~uIsYA*VELkM zJ>+)n++R_<8n092JLZiB|VvQOAV-GTq`^PNuEfNu9in~-)P}uip z=b`mhLGMH7Sk3riALe`{d)Kf@ei*TPA1rw1!{1&2+`z)Z!ZrzqO9iJBetoM6oB4WI z0AW5MTk`3X%TJ$zt3PVpmyF7y58kqwnYf7F!o$O(fH{I12r<<1m>!mjy?ghP?)~D$ z3ys+0Tai_7J3G%H%Dye3jLM>?FU$-Iba(nf_-#ULJ$aiy>Z9;yks6ALdXAQ)Q(zk} zYSH!-KO&K44aK&%yE__uZkxV8QDwjH@Mty@F^OG^=&TCZm3{Sx-ga@@HzTiV-1yJr zB%v@yLqSP!82c&_tk8%NXLI3I>pgle+ER)D!U?6oHV0W1ySt9}P}i^=P=)@LZaxLw z3BUuFdgXp3;VNVT!kuvo!X3xC15h`Tp?$=5PIZHUiP-k5?m3h5E~N+zIduPp`(RO( zpO=hkgT`qdLNpL+y-e*B&=Zi_wP7)pKUmtF zC*ANCy8onhP~oDb0-26DIJDL5zPcFZ2bhYHEd;xqGYNYk(m7=A6A~Ivw@O7#HS5et zCeI91@w4;~+#FrHYVFz<$QdGT3odfUj_nf1a89GZ_98C7Mc;`J0L=S|5rUjBK3tqx zUr+hFE>~4$bL2N3Q%biem$=Uzd~jI})-gvv_7 z3e_#gdE|TJ^9Lw_{1+}o{FgrsmO zJu;mh?`2qX&IE!sx-8KH6d&@Olen4>=<_yT7aD_uWJ=| zZNdsryM#M{)KZa0gr%(rZBHfzZG~BO;+USEEPaLMK|J)&hQ^ z4x+&mIP-h-3$5o;0B_A)tFvUDPee-)?J^qrm}s$`oA&C)d2QH5)HUj4RggdgZZ8^1 z_y*A?P%>xZt+Eq(Q8IL^iv-mGU1OaC9(#YZ5FNy5u+@DDP0#Rf4b30EDb1H{SWfS= zO+*oMCq>&NJhT(Eil!XPGmrp>Zskc47QpBPsYe}n(^wCSr=_MmWj8EkmM42%fjr$5ZPm`JkzL*3BS z02~rX2{@e@s~EI*UN@E%ls9L+O-He7A)b7=;q??6Y9cO4_%G0}qpycB7^ivYp{pqv z%{=ZII*wV{DWH}nOl-h8Vhea1p-B<&yhl4tI!x43al#~$vxtnS@DadUbz(|?Dy#z6 zT2=`jH#axH&bKLDV2!Pq**n9);xm-D`NLuqbu+7s}b!Si*{kt_d8sF zf4L2fh=yjVz)is3MBpdyJ9qYVY!kCe!rky=V8hhE1eOGo+Q+=OK;c$^k-=EBRxp1r zjxYIi^z;pwZKPnbMx9yY+HmXvYq$mf{unz(UpQ=w#q=7*Bk_agSmRWiT*8 zH#q^gQQwwsG~A+AREHLYYinZrn^1@}rrDY>H!#J;bn-4mqU=B7=|S;o29Ji;rh>ii zK0@1Y=k8{K??3CLTBGFdGHu>`;p%RDWJNAJ3O_F`?d{E9B_Zt&_C>h00E-dQG(TH+ z9rcvR-`_tL9kSl^XZc1S&mjW-^hEQP-Gx&e;o;%UvC7OORY@p^7Sv6BI>x%+J8|<3$4?qVHXWxnnFLphjf;PSlF>B?h;V$L}ED z2w!aA^kI|IgMq@n5~=yCg4vo#!$c9lI8!fp79f&~fDz~d82nW-Zcl~e1f3}!k+z@C zeX|Cj57CBFi)X4wI*<7G3=G5rNhSce3e0KrLwgK`+DgX60P2NCFV4}CC6cbb``|$< zlxJb94+pxBY7Bg&jR$ff=2gp(2NIz^sm+Pr@oN!c%q8A2)tDB;CjrKfXXCH6d&!TdiZh>;0>jA zi~ibLsEyYarY~Sx82(k_mcHA(rghP;M{s~}QPBl30$;I2m10lCR?N%g$@~w%WT>O7 ztLxWD;SVz+G3(}yu5y#2S-Mo%fE5qeQsFwx|SB zhFpa5wfk)IPImSq*zabD5q9%4cF|qU&6?5jKIGH&`?-Gm+P^L@j|UXiDYQ?n*TSGH zStnZ&8fC1s%DJ95eh_+t$EKA)o>7On%M2a9k%0*mulH>N>aq7w0i(f-RUaE)f+j|0 zFxbsNTWiM>OiJ|t1fCzDj2sqNXHk}E`?++W8(dqK(pfCkK1%=tV z`_DRX?nHIO>VF3%$#S+%2a=ij7G~X0?##$=1x_`e?T56`ku5 zmwc*%f-jmb1X9Z^iE-G}_A_PQ=i z_k9Or4h{Rl$%|gmf<4MMc)9d+vQ~80t5-)6Y$A<3OXJ^T9SI8qyw(L*mUqV8i9jbL zD`1j&@N9F>0<-4^?&E2RRCEy$>yHwTI{u8yc7*6+?vLbc0a5c21D7p^+16MtA4ac>9TJUsx-;wZ4q`YnE#e6%p_t^FMIik2TWQ-ZGbfUP&RL6>Eh0| zo5r~)h-BO0wTh~L#C~)I@f(mT_z>X9_EZ2ySA;fnq#NBn>x@knah|jQj|=jkiCIdh z0cv%Y+v2t8J&@zTEz;)Z=P~tulrs2}h%>Oo**BUZL@>J^d@E4$Ff()ijX%Z2ebE%s z89|1&!yO{g$xt;(kq53i=Hc&)r-m!1A4U}aaznum=r4%?9`ZZ!7re9qgg`%`ls9@8Y2bC)y4)Fws^SLO8(J!`7RTwha4RD$y$+ zU+BjWpUj)DzrBC|zD~Z)Gt4$lTz0nwb@W%Zut@Sm1ppLot7&LL8z9^D8kt}#)^x)aT3izQ*#{q@njwfpTcek1z}GT|R1;W*B-Z=Whc{Mi8qK42L?6sz=} zi}1q{5CjyW{+MaqrcK=X`uf|YuIc&{4+D{=A%Qc|Dm+ISJv|4h#c6gp*Y4}7GS8dv@3VFs3QS0oc z9ejM7q)VRU?9hJiKkmfRqtNufhhP0jWqJ(^?nA$6w}QvQ4cg`9WG;;OSKq-|2zZkS zu^C9-KSr;}=^{q#ctXf6ATUl9NmtRjT5wUKRIGjSh8U1xii2XO@Bi@7Rjcg>F1fe% z_Vqv+Rzr=1!oy2V&@U;zHvj8XvG3ZLqb7>(&jo=ZP+zx3azg_oENupWe5J@?8GsZF zAO)ecyfP(pm(0Ndu~m0`-g*eOD)2pv&1L{jg@7xvaRV!DF|9;NX@c31bKqlBCmBj& zkZgRs@qZKEq4LVgw{>-Odz+f#PVNLaz005yaUC|ZH-0;_;qd%pOG?qrk3vZ%V>Twb zDzFH35_mqU>Ii#+cj$kzI5<0-BX%klun!_OJqq+UcC~=}?`$HoI~1$Yf;pDJ-KW*C z{J!lj0u5=7KTTZWAYfWCZ~{||u_^h2`9l%s+!r~3Ix)Uc10?g&kAlIgACY8#WV?he z6!1S@Xk5m|#>SuXmZxU|Le<{djG~|i`BoEH!H4%aEP29c0yV*cMcOF|y zRfFGxRg0hnlvwc0uxTrTQ;kaOaDk04Mi|g!}6n3GB2psq3$Okl{jUQK3eGA# z5fL1qw<=v8RfRpCSTz6C0R(eq*z(nNaFI8|=y7gtUL#JG9p;65(gzw|09cx$c~r>m zCq*1&rQ!UZ;PjO$Jqu!I7Pgv^u0e7}Mp-2u5z|rPRsvLd@r42d%K_g3H4oz~qsp}+ zW=(JCL|`6M1)RDW>hA@ECBfG4o-wPxVwH5!1`ZKyDgEOT+EE1AP8Cy(VHstNhj}Ao zFT}wWg5?SjyrPaMC6o&=K#gVY0ybAfM%iB&4Od1v{w6*a;mHBzh^hws^3iS4mHx&t zLKN1U5nB}?EUzLTG*pobwJg20;b;79zmzS9dKz4|qXxdAkS<1BbOYtwY=a7Ji5$ci z?py(lK~PT?o$&b*niar~OsuRaWYz%UE8mH!R{Z|ahd`Xsh^6m9F8(98LLu?^r7Y4p zJVN9?M-SNMb_repWiYjeg@uR8|Lk;(5}Y1SiWe80jTR9$4SAQld5+T`_+N)k#EoPsQbrwyYjI%y`_|S0vFgE!C z7dN*&oKO2?l;IVF7{hMybG3tmgDGYq#0OXZ?Z=P0FqTsFg=0AgH2JuAX^*Ag@rTHs z0mc(1jrz+;g#*>BBP(9;%>Q`gKo#J{kc=|=^0)b7CXHejN9agZ+IBl&Ho^bHv_qo& z<%Y4TDKQyk$MK$e^pK+`)gnR8fh_bYu7NKTNJM`_7T$`cRDCI<5`6t;sG1-kA*l)e za2ygqA$dhQ^j95K71@b7X0t7`nLiqrAlgd;1i1=<2k@n%kcJfh+zJHUOu8pDjIDnV z!_`l&$1c0?B8I6VXtq%D{5eU`KwnTxH!SC=-QR+fDyVR;=){QYnv|X3-G7dD`94_& zm;lRO#X_fbNpgMkG4)`k$!h*oBDLWR#Y5X4cP~Nae~QXP{AzzH6SM-zw-!v?fkdLE z-@JD>wcy%(!C8mpc|YKe)6~`u=8VV82j69qqxKfhjvOMVJ^XEu%*SeJ#3D!?<4qqOY)*8QHU>hzHD8__>}cEJZ5i^e|)^!e;SsS#9FEf`CE7K5R$ z9fRNt0Pu|G1U=lqVHk-R45fl&!4~lJk1f;PmK~dfGOdM*%}ESIgjL>hJV9(LCe@)* zZpDPqBJW^VB=XDHts;WXeZ)RK-4`ERSXhY8VdOF}{ygGZ#%$8616M~Lo76qA9CAZ5 z!_@rwzIa`r(cz_EF`y?lC*IW!j{!%vDkbP7YeB$`8FUJ^I3DJHdkd(RNW09U2ITjm0Lqgg>zq3;@!iw5}_6tVQ#VS-xr$PGrET zhm$(UZlAKx9z^TIQ6okF43nA!sFqmx9d?5tWcUUTb$o2B9>su%o|8&g4*306W&tw! zgA_gh@5e(<2q`z)b4DX%j~z1ymD9l?g8Qt86Hp9zz+dqDCtFj_{^~GKJv^F08idKa z;7BY}qISBWS!+sof&H-AGP8gTkd5sJZDfHa)qysBCSToVFvk-YT?5ft(J4(>@yRuWa}`c>cF+HPblIGW5uh4Y?l~ zs{s{6jyM$1?Krz;-DflFB%}8!Gz_(80uJ71IY>&P_xMs?-MloY6IMeE+r>5BO}k@Q zvo*p)^kTzJ)y7r*HDaHHwfZsTBXScDJ?mRcIUwnCL0Lz)CK@d88gV}%wDt_>xnIP} zI>CoJEE&Ld&?aWR8JX6iEgP|b8~>OqG*%jx?+zr0Z$en<@2b?x^T+M=Jehbk7JdXgg? z^fYrvp~vCJFU%cLg_qj0t3VUxwxz3hbn!QA8skA8+O zhEr$_a_1W25lRHjB2ds339Kgy-8nsJU`SPTRW2$JuejTSM8t*eNTPKSsEfQME=5k# zNr;e*0>L++_mm9h37f0$!}2x5`}rJ4(h>l72nIqUBPK**N5SohS{Z$P?d2u&9vZzG zv>G2*6ni}Nwt&!^5O0BPDh4%h63v$AW{X%j&2&SY*;qP5fGnc0vDv8dp0a{5#1Z+p zDg+SXwnuJ9pabe|?+>75H;~banRLp?R$)`$^=8C4gd;ph;&?amY9>5{@@_Gt5-Lo3 zIny7Y=4*0~P3j|^2uvU5_(rgVmIqh+RPW z@^RDxv7-&daC+u>F?VOAr>A>aIRmiS z&%${IIOC7u`X4v}u1J9J*|@4Xx1laa-FMS$w4<%>9x^e;Gx_)EM&0b{xpPOeWozMX zrTgV0IwfaYI_!G5&9@Kgof;C)*qvQ_f^pL=?PF1Dj2j~hLw@{tkUBjP9?DVh?a+@t zOgm;RV?&A~ez=tE|2bb={3BPx8>8vc^77@7^)K1jG!&?AkbcRn_6B4&-F84t^vCe9 z@~JotK_vHuXB)5B*z7x3w#ueLd+HB(cuvj+YD4-{1X$Gs3Y~4%Bm0GV+{AK zSh-$J>m2O8>w9bt;#4oRA@n}`ILlux1$O3>S8mC8V2md#E1Nd@I5CkQqKMav7ZSf` zW-_k)VyR58R+Pb%a!7_Y*WSSh9T z_3ME^BR%4AUUDlSPPvn%=k42x8dB`CXU?0L7~-(ekm4#3_0?>Wk5|NccVM+Co=rDg{`8Z=|?;cr1YzMfpb5OHiJ>K)dPF(EWT?$gM)FT?SER2}N5+CnLdC_6i8a zmT%v_QQTj@mVJES^7R$Bm@Y$i0wheQ`0MVigxID1IeXvv<*{Y65ANMtO)*IB!ywW3 z(W8nHBMXarh{=u9U5H(+C5n54Ky_Hl7m~}?J(T4*IqT4I=en|N@2XDpwk4FAnVGiy zE6+Q89%&R~z-EXvv{(SJDg*E{HMpa7Cd+hv=UQ^Z08))lHO_6?vlNI<9 z9`JtwbJH{l2Ze?{7KPQ2#C%_0AL&s~!FqcnC-0$rMrRnEoQyrf&bP-0#vsLJMp*+e z5Q`a1s`++BdrexlLin>~)mR2u$`f&!=J@4;pbVw>YXijhr;M9`{&^**djbsIcCoX2 zY*bSPN?U?3uE!DR)hq|rQPN;yn9Ng)2f1RElU&@z$%%GA%O}Op+s3qRU8#&J{jH@$ zpbw;@7xim}n@huqUO+c*9fCR9BfboCm|KX6C7=15>6G>`Mp?2-T}x8Sw*aQt9jTI# zf+kVc-oBNhFLDWeNtowPmf&B)EvOXQvjOm%)l=!#a0Du9>d>KzZSwN+65gk?d*@CX zw}t6lVG#!|bFA^ELpk1rQoN!xm$S3PO=^wzCNCc!IRk?ol;T6xAaJItvR^S~yU>U| zlXG$sUqgQm6npIROPWoambwFYC0ZVN4z1~Q&jsB0Q8hKjy}EgnvuDp5pFjVFNKq@< zG5BJdE`q5?bdR5cfsql-s#UA9>%x3}-H}Xzb#XX#lErxBC3p4{Nmp;2KzvnMNegq) zo0^(EHW5ut8e~E@IyUwaDJ@*A-v?d9FFbq=rN1eD1Ht7^>>8H~3JRcG98&UN1;=-i z*^r!VT*auhkPcZzW0THN1x`i`hdj@2{0R)yje(%>zJHeMgoXwafPVWF@fkC`7`L2-T z#1E};c6O#n+uMu69eCn~OfYUCUgJXdf%}@ee6@(WyUQX7-5);yef`b5cg8=txw(l+ z$*+_DiVw_)+iK zt_hrJRJb9Nbpmts>V>P|+FY?o)aC|^-d8h=-g6CR(%Vk`C!M_E|40jX28Jt-5rC-i$v>m#a2{jnsTHnyX*wWG>C&^DoqXeXXDl8qd zW<$+zRAmedPY?a1KdOq-4^SJm@%OgnVS0EAfauWBP#UlYM{`g^W8)akB%MR;0Hcxs zr8pVmt&a7>%;+Xz4j(drj)L}NCMKGD9o|lD?(Xefe)#ZVinO-&7Qk*ftV#9t<*R%H z0%$1x4bN5qxnEQjeuv0AUH1h>kc5Z5oPP&ULc#Cf4!>t-X+Eo%{VB_GN8vJwf^JCp zEvg9@?DHe_GF!K9rK6)W_=&}fJ+gkik&#hOI6o6Ty|k%UUtL5uPKgZ_6~J2~%)_zY zpp+u&N(}BfMzV!C^qOH1pus1HPOr3koN@bhA2_NqKlv$ej*gDm%h?pig`m8L3tD#x$Afko^#aLLv^QhvL+q-&bU`2JzaKJtSuJ0ql{s z&6D#5$1PJ&T7;6z>FgmAL6Y`}zUX3&e^lW}+l4MOAudR{mX}{jpu? z8`i;Fj%mNh8t)Zh;N4*#A?_*OT^~gbCj3eCwCrp-G-^~L`7Q3hF%vVDu3lexS%FH8 zpD5`DLr8i90~gJ^ckfJl@7zOmHy8Ll)V^OlSgH)dC6XfjfsM}o}8{O3q+%RI7v)k1BLB6;D>s? z+m{x5DBTBfUM4ZHONFOxnE{%#f&$!b79CkD35XC#!0D%S#PP zkvYl{SG)^5$`T6R=P{r}`_~fsa-b{RgD;QomsnM=jm4+p@EVoieI8m5J!fGdsE3=J zGUs&?)1GY)>HR}Oyoe)a=F@t3BgdU~eEm=xh>wfUP&(+Zz)81ejT}T1L9oVU%a$<> zc&o3+H`|5nQq~K6DcG2DMQsuW#>?P?K5cF~00vL}Wg^|v(=Vq_DJv_NqtIxKvy(G# zq`cXblv$!SDW8PCn?1{V&X1+n@f{#tx~`XyY5eK~yb-zZP9UDv&bDLN-8R@!TYF+q z?}`qa1%wEuRUQsbPKF={e7QJ@nt8GJE^%dJop>@Lt9xxGhC5FgJvz+G>-8%jP=Ql} z}P9htMRQ_ z(?dx`r4k2@wPR(0zOW6H(s5pZrQ7-W2Pr%F_`ElEprG*0^5yH^hdM$-0lrFXsG*um zMw`#d=R!ZBxIcNqx(TN(dM1IY0v%B;|A3r*7|FDCbYyG}dj5P@@!YtQ&*@1#OB(-U zplAy z#}^aYh}&%odIDS3OPr0@y63s!(fEhLs{*L)7ghi6N&DpGH!hFz%v#>ytlXDpcC^1Lkci=)YQ2sxo@Pl*0+Oh=| zrVL$+AD4!;=n^;@Wsa(J?AS3KUJkCL1(hXt#>Lb#)NkicYM6nMXnE=7D~@uUO<}d#wPMfS=UD`_xJa|1WPQX6ff(ju-f36zma}< z;tu-JgYIynEuj!Urdk#CjHz9kSNDO*sI73DbZ~aTqoh%NgBZl(PG@pc$T@^On_haL zNYA{y17twVH@_)S2y}>4a7RABr0LzecWcbZtZEft?~O!w)2&_WiQq!R23i!vY~sLy z1EpnUO95M6!W;);ptZNI1SL-NFqSYUl{+*M>UxrkiI&!;K&M@QHCHuR@`zB%Y(Z@( zm%1y~U>q0?hs!=u(F&mHbX02kl~FOzo_)d8fl+PWcTC9OM6DV3Aa3iUQhdjyJadasbE=(mi_%!5(Wc!Nq!OJYdK)}WxEV>e`~ zyD&aKh_~2GcJR-sa0gt*xa9+GHXmgF3jM9GMG^b=?!6ySWz0fB$+UrHk;NhV6(arl z)Nr_a>)roiuyiN+_w4Z*#6UK^awp{ycB2=d?Inx}m!UGAUl1cS?rAL>amv36uBjwCcGpnDyoZ_ zLx0c|5SPW-+>4E#qIvY>0P~4cr*dBppP5cYM14T!mck$q3ZJ9n^S;i^3k=RL(67yE z4yc3(-2#005~Ie816&CrZ^;X_T-8eLR3CJ6eNVa-%xajc5U$-$M@J$R+K9fmK44`rhkVL`GUe z70e-#&z~RjQ(NEPp1BJyGCGRFi4!IptLS)tEOSv%phLnoebmbf4Tz7&Nl}S3NWfQ5 zm@dLem2~qwVyr&zvXRbujn0B)TUl=EUnrK-(euS!6i7h)5thCM0rtAVR)L`*R}C(eO)O zzFdOyww0*;8hoq|J<}L9;G@VkNdj>p)q|6pn*)P8esLMnY%J$6lniXB)i{Ic!ZJv@ zu8EV$@G`JH641YoF%WT%!;eY04?IPqEp=$5hef-hn=wD^r1}`|KJD%t~8}rYqs4!qoR0`^u2L7~#nYm<4FigbU6R-Bd zxGwvRWo&FL5wVbc^cd6yhlVD682LudeTv4-9p?)yYi-?((IWqNE-_nx#mL~dvL{8j zpVcHeKzHQ(2|j-mfLi2QJ{IsZrjOrn)K`W7%x>l^4R18TjA=mnhNr&tvro6Mu;^Ok zQF*wymc#TGEPRoX>?``$yHJr;P_~lZ{S2dHV4yYgD^_GU(u;aM zK%AIs21Z2a_qZ&H--7nBNAN>TjK%!d|Mk#d0I-Sv`q5v%AQQ}-k%}1QbO4;pHhJ9G z(9o83oV(F`QW=m+7H3=9L|Hd)KE|xOoKHEAN_D(;1z^gwDQ-PEYfPSnTp#Rza%$vL zNnI=|n&J%sn=(6=4I7TqsCf_=VwgwmhX`G6RoA;w7!Yuam=y~+=w+M%o`a*M4N}Hl zy%%@z!VyEnBy>r>W}_UlW_svz7!Q;~JW0c`$eTBB0^zYj;palVB$y8+__3txHuwzW zWo4Jm&CNYcP1Wxi!_lO`^0wYaJI$rB%E5$ZSeog}$M-*)|F)^`D`u6bg z<9lw4^P4G8qodV1^N$Nhj2fh6&M#U2+Di8}Ga7&Dgx^q*$aZ0STvb0PV zG1{GU{N-g93k!=Z!+c)K$^4H15feGhk8l1MhJPW_qcv(+TvYVYikRzx+#e?Pc3wGs zu{CA;3I^8IaN*m2Z(0dK-zl}y{6n4~IIwR##E6 z0`4|4L{i0>0;h9J-bb?Y=t)acR^58oJ8mH~l>w&YGkm{MQ*$kk$y>AQh;o-MeZ7uf}l0~6bFG^E?mMZn!Y_xIDqYj_jU zu{|E)I^nrjW@Ka}82yp5@bsLX-V^y3C5(UR%keDLm&nn8OPK4aEf!~l24qA9WM_+z z&lwrHPxP~cf{}B@2^t?R8%0e&4i8U9KM2dvp+;pzVd@ zel%M_*!AnP%F=UkOgincOt5($VLr2b|Ni~2(7OcTpO`98)jwT>(>iRBLeSZMkG^6M zv}oHGcsCAAMdij`WK7bU=Dvg|s7^E#4A#a^l%ytoiy4w`hMJvMScs$d%ZUQ{Q%=YG zx^np)FRvqph8$si2kvKA@XpD%!-Oc|w!k8wU;G+#^u)Q0l}g{#(x@)T6_}A8U}B!Z z(2dzzDHi?WC52ldQf`vOKCV{w+jZ@M-TK8Wz~^8^((>{v3LlIn9OPg{R2>1*#!KaB zUO4@tM-ftYUKjFraatb=ICKpsu>-Z#e#o4zxrh*d-`aXRJbXKbe)mDuqajmMN)xCw z(ilHwo7}@8i8RxC4)`9I0VG#KSCTDyfj3sN+uGUr1qI!P48txi9=cp#p&X0ChVkR) z!NFO*2hmC2vhWIm{{H^bmX=LGnPtf9_T_;U6%|p3E^k}6jsj7S$cEbKOW_iF4FnOH zn79fr2J9agNN;(J{T-UZYy9rPu$pGChdP4?P2;JM-QupDZ^MOCyj12;x=Sp#!In@p zjDS{r6*z}-$|4VQNCHS&;O#iy!xL!zGKgNR`@&(Yh^-Iya-)sma5LUV=Q6|`vcEG| z;MT_MJ6g11C>`l|e-5fKTJciM9FuOTeYw^STGfwu@T>d{$H3^^^c6~IeUJ!J_Sec3 z09yEqrO$ZKAyG07)J;u!R!H|a=KP)Ws! zm)x$du34zH(4=n5M3w@_j==={-jkgJ!;d?6?ntAuf*HQPW`#Q{IKO7~>YK70=aPSu z?u~36`c%S(F!Av6T4_FQWM)PWngLw!6+?ifm6wO6$P_lzD5H#W6WCBIn=MI^iGDdK ziB!z<=RpJAikyt(;y9FbuK6hx{i8HUgqBbs-t!w5?;J3`aG@LZkq~8s2+1-Wt_aGp z-1D^gWh{ybIb!Rp_zats=RCESpI!IB81zl0{=vb-x#N3-jKK3LzilevFiRYN>+}o@Lv&^L22^V@Nu*sv0t~Al(0X1qBT#Q?D+w5X-3AbV^e39i9^D!SCP8-oCvn zqs-CU({mGvL-RQ`D%y3+--1pq4u+oE8~E18$43w^+=J4w8RT+we0(Vd!TJ=?SZ9AR zPCXCxT!PjP$RPWvpsKcZEf$-$RNwfy^2UOMsHCLby66c6JI~|^g{%f6e~oiqlu`E0 z=>tfEYty}(d6?*Pwu6R}_=h#^2H4svQC=o`0P160tGNOdBUO1u@otto--46&QNDcr zx)d)}>3GAZZDPWWp|q+=BsX*jbo?9H50lRpiaHln8w;78d$#;$7aJ*78^ickR$c~K z}0R?Gg56|Np$)p^r_o&`Yc5U0qCa$Q+jpJa??oxsXK70fJ zzY!pn1mZ(-YFN78h31h5)jI%)5On{PLsCQS&CRtKKAHZpN5yG#_IqqyHp<9Tpt~%B zn6J5aCr-TQCGA%&uU`j0f9@N#OEF6TKu`4*UzkYLzIn45%@7ScK`0U-P;Elr(9dBU zrdi)Q@%7s`jYEevVoW7|te_TCCDM$DH54uu5#~Q3Dv*;r;H=lsgtLd>MoA*z*1tR1 z{@6mnDMq$W(bYxu3s7&0BMMbt@iOQ&_h~E}hzSY6Bd~Y3%gM2$o4J9iW;4UTBIikL z`cC_-Zx?OPo_%xs5OEgA8WQi!^mzhH)!G-Il$0cYX-){9ZFCkE7I7cZ#6Xm?@%AX8 zE*Grf0Xz=u1(k=E%;0FqkII=#)J7h$>DddpjMhdrOa`0FLuEFDj%!kE|hbUvK6FvqzGkbMIQ1tQb7 zZCe71*gFd?_7T%J6uSkUL{;n#Te*nE-Ax|6#3}naH96@-&$ws*{{8Qa3@Zm{#oDjD zettbU*fa@M#ZAe1KNwcYeMaxjeV(Zse+ztICN?(e+oOzl3K&~=)&5JXVk0Lt^xfzb zSPz+-n!DCzV`fc&ZJ;lNXbRSqId>OBo_1kw@Y5$=j4OGYa(Qc_+Q!VQ{S*+pZ>+5N znIAts44xezlK93*%m+88WZM{z`J@^c59jHpDK=ld8y zjo!dO!n2k{Dgb4CI-L$4U7I$1dE6>k3ZR z$!j%Xer(O8ZPMS@N1&pGzI1X-|1A*33d+*6D+*gpqJg}~eFmr9Fv1bw5%dIffhq%~ zFakg~0p$y&)4KvLpt_`)rVFv%Aa4+-N#|xcW}Ub~G=xBHaSzj@)5Fh1*dCOn@3gh? zfF!ZFtjzDnx2|%Mrj8DsKMa0&!FHRP!pxTFM~OBMLM_AF6x7?vRtUf-OBvA&j;x|F ziNvHg9mGt;mWBOhu1v?QVLd_FlKMZ#kB!-Ey7zl16Qf>P#3CUhIPbg!eT|8k`6`;s!&yeZe}3i}yt2Ue zBjO%LzYzA&`1|_ z*kBYfr_f|KzkR#w0)MLqFY;9o0P0~-rR$L!I$?BSXxf=Z(tJf0>&blZ;2LnKJ7?Q2 zT)1FAcOvGn5GvtAfJfzTYv2SB3U0$uu$3b*Qhm|wuYKHp7QZA*c^^E^iosC#gSd@b z)(Rbk>j@BH^UCPOiX=7f?}3)uI-vaIM= zaY1cE04R3da2S)>{_o{N|M$)Yk{Fkx%>m&5Og1>71sGKLVsIBJSnecXUBifc6piQ) zgSqxP4B?{_g1g;XUyoL!G( z_w2D{yJlo$WODoXojF6J970>+?%`o8a;R-WJ{^@q%CQSIj4^osuTxd{!C9rxx1_`e z0(b{KPQ<-KP|55kCTCaHRu`<%W<9{93m;}93?nr6>|_nhk)Kf4w>ORZu8%_W29Xtp zUbZB&yDGX5kzj@~@cp8$@o?^L%wj`8wh=rq+6&TQQQ)xFqHEzQIw+5C(17bY2r}lASn?P z6Oi$Th`xg&4P&_Qoqd@_MO%05*m1!34+AWu4(sZ&d=6qadfF3`+?qi)#y$g2bJ53O z4epka>FIk+&%jV{C%^jEty?pMByUolw&wh%qU*x{eP2460F_+m>^k$x(Fw^}_ zMNs37H&y_homp6D!bg1!rUsDgJA5h;k&%uIu<$Zns>0>9MV+7FdR+Jb?fs-AZ5~?% zV-6Q`KJiPIALrlXNtqueEcPeja^>su1yJzI{(VVUarBiRzfb&H zl)h%Co3t0Lg#EVe`1eGs@Fk#{NH{~}&!ZCIPrVI+lzKqh`>Fa0503FGxW&}fZ`Q$$ zL#}B;&gnoO-+4R4U5VbpEUM%ogDk9xnNe?$e-Tg9w^&R`>0U?@?jjM^Fkv8gXCxD; z+eZN#B{I*=kFH+zW z)Kj@YD1t_TJRH{1A)8WXLuq6SRM|k36<}OFrVYCDFF@Vm?!Rox87?a;n-x*aTQLRo z4;{YtKGfPx5|f=E1cAt|D`Epc71y0O%GZca6`!j;)&6upmWqx`H3Qs7%5v! zsgdI{aF^J)CGr{SS)L;$l(>P%>w$gdL&F+rXUsa!y~i5|D*33VCLM|}*S~oF+zPiZU%s66NJjWm02ns4 zA3mHAAJ7F9*p&UMhrJ^2(&CH&^bmmr?JrL{iVh?!$z`MMev~YfX!ef1vIk>di+LVJ zjFx{>cQ*@(z|;`u!04eEI+|8w{#f1Qhh^n`@hSR+)n%YmL*F*`5YP0c7gvog z;K;C=#$$eo9n{wEV##AJ%^SA{c0XLYVt$Oh*p0uC^&k4CiGhGR43EgG(~S2!r1ZzA z^u3nm>_JrDg#HXws^RkjjJkk&Heq@=G%*nlI(|Bk{|ug+s88hz9~iehH2vZ^)Fr$& zBF@8O7*UF64R+W^9bd3IHi(Oh6K|2kALuk#UFFnRftsn(>)RHLM~TsM+(*RwKDk{{ zhfPkDeIFmUfvj^84DebH%Pf;mVp8=a&5&SaPtU+o$Dq-sr2uIzMpjL?nK3=ibEweL zDVP+770YwS7g{D={dsE6()MIr2AcgSt>JxRS zctbE19!1u^(_%-CONI<27`cZ%>8tz}z*mic#5({*fhR|UYMT8`;->drS!Miwb5!z| zqE)OsCRlg-$vw@3irTt^su!ermhN4em;xQDph}}-Annx;chy1Aibil{J{fK~kqDmyzluQ_~ z2*d?l)juI5B^9ZEQT0DO3I z>S=d(cfU-N2l>~j`I%vj^1DDtcgxC3=JQ>5G0lFjroeXg!;F9kovi65G=6e?OiUCZ z>H^6Z!ZmxrN-(z}QZHeR(ytO$Lhha&U0MEpZ$EWKZ+_E_z{%&HUV_>ke&QZ9n0W7`DDH6HxtSSNOtpyVp}>IyysCB7&KSb6=@*M(0H~i}!SkrzG>NLcMejaCf%Vb! zY)_$r4#JZT6IFDyoLpQ2N=kg5Pc365h3Q!?tT5784`_RsCiQHU3rH=YGm6?*Qd}H} zGCevbW)|TTC{WQU^3okT4lb^iU4L=dz5DN(iCV^Kf0jFvzEi!GjXI?)lv7ve$M^4S zJTBmJX}s9({ve_*TY59~r<(Q}f#oHNhev;z@x5r78LoYM4`;)^yh_~O(ecdhlClR| z%h{f9^_>YEbQ7b5nP~OWMbRG3|2~mS&Yf53cz`dm^_C#?*8vcLV-reGBzJz`JFm#! z#6KDXS(AhP&S|0t#G7mcy>NR%Tjz^QUY%4sQ1zLca`5rpD>>6-YGk_8TeQbS05jzN z-{=|29G*tpx_d{~Oq;16E!d z=Fa2_G`AY*27#mU$SjXjVVw$4mIH**29T77CnpPL<1#~jm>*xQ>+4%w{lJlxITN=n zmg5dxW3eX7PJ^7>t7T4|8-Otq)fY1hi^uxf6>|%VGp|a35{<%Wy(8KC3W%d}tH&rp z`^>A8PpFctPfoqM*`;LL7QgZbRm)kYxHOcJ5`B=4(m`&^&B_G3F^BG~T^9sis6LbV z=yOlePc=zhKR8xghu4YeEIl>V#*G^P=^oHY)OU71?sEr`gAO&l$g@3+*EgU7A6%iI zHj2~cLpbuZadO)9U*HQAYyeqcv!w+w?HR7KjoCM!V)nB}$M`=#`tONdB6wqRtC*B@ zN~R*4q37=zh`)2ER$W=!MO^_+g%$yd$!=Si9yELYb1+<;Lh~Ryua{Cl^%q=fy+lwF zHPf@{1OXDlm55ixpBMSBBOc+Q<;|N{z=Qi*fp$~~ zKyXo_>?1V5h7ol<8WhtO|NZ_0k-|bk#qTRXoY@>6O1ogIRsZn6|M7p{n7)39Fqu3i z>3iY?cgnl?@DKIguBT7``fBa-9$!gRLx0*>I6U?QBA9ux7(Oj3?<_T@BuwM&92_=~ z2)za%pqYIK{9l{d&n*O?@IrLU)hVv1xbgV$9^gPxiHR6NY&xH|J3NONZBO9@zoY*6 zq~rN9d!rk8y(XeM+j4d!P##T%pr6vdSL!+4Znoc(VB|g)jfZ#D7D(+>;RrxXQ&-=M zJOF&1Uvi7LDIKRr%U=Zu19*8T(iR;?+fgBfGcRWkzRE&B7u=HT3itZc)6Z+U-)e!Dggk&bK6V6ByIn z=l@$u3b@(P4N(P(lX&kdYpJVm0Cbka)>aKIMuFv1=I`IXC&J6=n1mOgyOi=LPsT*< zE02jQ6&$Dh^n&(<^1+i1lT$&L=L1s%vNu0s?mK?GuC0HMyRbPEkB~|m~<D*VEO^MEa1^$pK$)0;{I%!pW-HKPLF_&`=7{wA3Jn~Ig!V@rf;!D7 z)zd_wf2PcdLc)qce*Z7G5%~6&#~mT5?i{Ugf1j(d%xZKF6VaK$AFJA*3AZszzbl5W@Gp2rm69yud=+|w zGriB)X;Ktr&d{vPQ1UyGe~jjzEM`3>!dL?eKGC7UzpmdOLP83 zj7jUp5$U~ReI_9W#k4~fj;r$=ckaFp;2M@!&g?9Gn8RKQaxQ_7U<@AxQbt)>+0{RG z+!c|Rf$&nrr2@OyjG*xfh`0yv==aNfNbb~D)qY{P?SOzblNkB=BMDFoQ&SY%eeii0 zw|ZCKaRbP?b&%`0!P;Q?+1k}*KWDpufa-apb3d1r(kK`Kg&v7O|yjxj;~1vZ)ev1 zBTmj8^j%+2S;54I6o_i$6v>o=Jqy%LArTQu5*{eExP?X972oP;BUKW=`?JezhmT5q z6e%yIyv{Hz)hNi;4svvSyhL9aG|>g%YN3mikrN~)CE1@hZV7%=x;Ma3zSeAF6Rnv1 z#kuN}Cyk5(Yu0J5UZbF==U&lqAlbkTu0Z}OT(fho5e|7x=(thu(4Qy`^Lu`KRZ!rPe-74R>LJQla^B+D5R_FxlXK%8MLJ5X|8q za@>d5{eHQOK(WBS~73C1!+9*NxVvb{mpcm?P?#LWIgw<`ItM1twbEqpwW>CXar7R z&IX!qi>ZonyF7Qr8+Q*}bANFan7XwQ4x+FKVULPHmdzW$zVLii_zSji4P z^{Ne6r)W!{+`5f*^2d%HORCRz9@ZVDyl7~s`r{o+M}z0(ImcpRt#ZlJ4tZv(=cji) z(<-Z|aO_UDoNB18B>_3xKoYf@wghC&PcSZcn^5R+LK$-dtn%f1z@h6tf98Y5Pc2Z8 zbRgrneqqu*R3yA;`|hd1vhG4dO81N3V%lbBkv9hy`d?A|oE)1dw0D?`G_~9mbZ4Oh zwKwcDwvuK5!nqt0IfahWrW@G$t3gBS)Il>zi^hPNTVcZs_UF#2rn6_d2_^h8PP)#cBNasZN23)?q_p#4h0O(pLC)O-uEC=F@F?Y=AEnH>m7hne-S`LB zgj7=K7maSXgkK4~;GKx%KE8dq2(t!<33JBT;SOSkB1ppV92kNXz zWpGb8Wt3?_+cUOe&XI6Xj>8bl)3RhQQa)!evqJ3dkbIrVblNaP6hR-xwOY>ipEQRMEHU z@~hMuv#-jyY=7sqD#Atkg!~?pja`%foRSQ|h^FrKYX%Y`a5yYD!cRTLe2fC~K@+KA z%_qPKfN4-cZT~l+boZp;sjx3y(T5kUGD;mN($iIS=!SmK`&`Z|a*t1<{XKnjc&AZz zuvrrVdibMk#&qa0OS>Ke9<;d0URyD^LFDL$++07YDKX>4t(QVjW$ZX)R2lZ) z(){pa@MCJTS65b$@es@5;<>{WhNM;J<4%SfS0)>quoj=9=0BfvtZ;GgV@0G2P5v+k z`Ai1jxl(7~uAi!W6QElsyrf+(!`mRoI$q?;kN4lCVn5ZS7OgR})4aaFx$h%K;hN{S zE!MQb_4;wycmNR3)1+7z+faCl#%6dB^e%z|zXag@G1+%o%lzI{3(L@JuAwIWrC$$T znR+Q+VBl2dZGMx*_8|A_wWT{LM#t}4OSugXQ&afx@xA%`QCH3Pn*PG{uQ1)LTGifr zohf-5nVB>E$~{$`V_)Y?1(;d#Kb_JQVvaKi1kNy(qd5*~oNc%8pHP46Cub-%ZfvMo z-y$T0AkAL*H8zR1@+lDnlJMS%qM8H^$aP>6+^kVzenZ8UoSZXtZSy~tqGu-Z6!3%p z9wsg5;Z!%@b1Vysh>T(Y42dMcOoCpbWC_tSzMbWXGbO|Vsi}vxDYo!!*m;DCTb6v; znxEfh(wuIeX)&^UZEr6d3I3vyco>kPrh(B{9G{N(l?dOwC8Z?2#)^j)FE}ebL)?_i zD*toNKV#gDxjZFF70yj?Ojf^^VGHkwptARcFNro2%$s3LdWL8%mOjY z;!q6Cy01j?4p(^5?QiR2hA0HCl`GL*#*?C&d zJy+fIyUWB<)5@QWiSN_YWc3qp>Y{S$D;}Tr;w`qPx3PcqP}ZWHYU15kTcd;HubLW9 zVH;P5GM85>*PqB1Uv!J(rXfd3oP+Zmq3a}(0E3KM8TK{_F{ho}Ki_-T<$X4)6k`-& zj@4}>#sq>OWFt!Kg(rp_ z7L|ozJ;kw4G5Psl@jF!`xO?IsF@OK@gMeskei6!41VtwsnFidsS#Z*GCm*(Dp6E_V zOracYoc%_BUs*EetDR$0sz{c*ac`rCpYBkq|HwkyV8GT-m316HeP*xLMCvt~9k`sb z#b9{YZ~V9vrR1*+*bMGxuH(*jqjhw8;7uxL;RXoOXh7BdGB$?U5S$0MZ0@Cg8hQ}wASRF}C`o|u zIJcO@#mv5cH0w44Y1WPPO_#Pp4?qmENY!ubuzTW_pOf&|Pejaut>p;1&|dZjJ<#!5 z*yTRK3S!f&w{N4VKS-{a~Whg-!&rj-(!b6b`{&*WnL!gE>BG-y9Y`we` zt)k$)BER0=pHxKJ5!cn4w;i3g1Bdw*I0I*m$mf0tZsNg9xQ)D@=XGRq_B8OPp$Cyp zTBcX;-sL4=Hk7-hKvb!xp%<6g&0YI{QJ;+@=q*QK(ij)8ZTP!!b?voO+`#^7zK-q( zai*rh*XK<4&{xF99bUZR?&DL}e(Vud_41>|roYQ>)gxbjsPE7MzF^(2yLfNAyV`jO2$){a(Py5y%&$tF%$ zud8vL=&oA)g^Tj-cOc(QKv0FpLHT1oI z%-y?QP`GsWuur8+NcPAx0egoH8aL+HFj*$d zEpqHjLxRumKf8R?#OBA-|8l0MQ)@q6UyYQa|2p1m<~-366hJ|~m5urs%VsZXfl0N*&-JGB1 z3e>nk=Pzfs=}zJ%Ugg&+13y}C56KU`lT_upi`DN^$&{oY?|-EW3+J+Kq9gzH6FHJY z6X$w%Y29~$=8@&_`fbXMf&Qy>o@6>|Yr8MC6l>P1Bt~9|WK*%9Hr>@%sN^B zI<)owrnhqkZ*fu>8yk~UVW6kOMv-xxs6N2$D7|Fs6mbXgnvk=3pFyj0QBc^aGja{qCNC5 zb54|Nc>KwYl#fld4q&$-mzIFsHc6 ze2R4E-kSwk6YPApv|RjtZR67J?qXZNrQ-L_gLxUK@5#MKfMp%pzi;ybdQs5bQ{NoT zIc2M;-g*KmVRL5dT?yU(dN08bw*kQKx&}Y&jQ^)lX9C4|2JRM*FJaRVOjbH;oze(; zJR&kF5Xb*CkQcbYKU> zP3)NeL9MM|fAuJrpGzNOUnNuD^Ms=>P1Wl2^_gNiXa_i+K^;@=D2WoRaXdkQaU_UZ(7#+~(4ir`Xrl(+ zcXmcihFx)>|M!-@o4^E+;A8!rMP@ABNDWF zg#Um}R&?x?;FFd#T;yzA$)T@tLcwb!aD}@Vq@$SaT5^~yb3mG zzQ{pXKbh!2WleLy9A0$=YnSjb zL1WP!n3UcQL;*hWBvFfU32@Eh)R>N61>K}^xeq{VC72ludNiXLoUR!eWoXjE8pqDy zUrBE#UtV7B2Y*Ns=3gU|lM>gW)v#nO6?j2De|zGdYj`#-wV|49yDZH|SD@C|2T#qY zAnlDvzu)1xv}gL95lFTC%avU!>VE1F#PkE;R($>*-56w;cNt#qb8vHO$Mlc_rdzgP zr?KKU(Hs<-nmq2ac0jrIaMsm$NtW{G=V`5Kj`_cPcm0|7Xuj90{%5kvK8G!MurB;S z(iTy!jBP6KERW@}N}OP|9{v%jGBT_yySo*5~H2@W;&l*mRrsz%jy61j^yq1E6AcY&1+EE!qD@Y5jz9x!)=t68Mi%J zDJU!~I4W_6EJWpZWpw#i@%F@Ihl$V0m-fn46`Ai5H4&Y?Ci`=1a$@45^!X~^m+dMq zpG!%HJ#(k1uOHc(*C4ov7%#2Lr+oL(jWfien(TL?NUuFA%P~mdh86OLy?+TczGFAv7BiUvDW_o9Yu4(F&G-1KqJLh@7^72?aTB6 znPwDJ8IsePGo^4$n4k9)Q2x5gGl}{ETG==l$2QmcQIg}r!4dU0RY{5&jVM6h`M zP90O~S*TC2HtED@pW>ytOa?t}(A7?3pVdSYaMNf8SHVkbtNWD2qs!$IlbmT3Uuowl zzK`1BK1K%$s(Z?bC(-_&uPn=0S6n#jqw>73B_>(>;)f^Qepe2gTkYDl2NtV=4;}rU z8$Xmu{c%!|M>Zg!uUyi2>fLLuzTaycuUz{9uduj%`<|QB-%`Kf+^4`NXX1KJdyGw5 zp0sUb7z+&h<)3MdBS~hHS0szve(w8h+at_lI$_r`&p!zIUi`in3?sO75?JK33IPAR z^s=g*Hu7;67;hx#rHF~H_cPllf7qsn0ME$f*KJhz7bUswm6yvArk##El&}1i!RRMG zNc-Xa1D7&9y3eDqL-x_YP1Lp2&V(=1hMB+fXP``J8{H~0c++Gb#G@#1dn19W7jBVe z8}R}+BLZ%USg}V zMnj|X{MyF2?df=(F`pV*?(^v+99fY~RifNivH0y~SWpn9>_^ldK3(?~_*AwH(!}1< zudCB(1lxLqW#z77){^CPQ(kbeD24gSXHGO4_P-at>Rl?17oA;uxK)Tf>dM*+hn8m< zT5;SrHf|A(^!ASHvEfq;jM#jOk6zXV8zxKq7tX!;hI@+v%3^s~udALTvBGCdsDjVT zb`-qv`>M_#$ujB$54e6X>p?!6GGg;v5YzE^cxQp{IXy7aekRt5_4`qPl{h?tWHKEI zh7K+A|E6q$#stMjW@jYqsA;evi!1SbGmO@ zXF#l~Nh1?+2<{r70;zjfB2`TkWV1(J40h z^vkqp3;)0`WSqTD-(P+?fC&2cyq~+t+q(vjGmQ8o6i+NUI^`7nn(zJeXSH^xh=_2d zuCDlZ)dB`lFB&1W35;Y8)3pZ zL3FQ%-PeTX34%64tXA>|{Otqry`4>0E`kT)+E=r@+z|*sQ#<+Mzr&u?h>{@(p4Vxg z7^Yxy0oIlD?(fa$7CgMXC}SsiF|iUn^yvzBE#TnmBry#-mu&fO>s(Z6A1%qdS-okpsHp!~(b|6Egw9HM`2|1k;`LQ2 z7s(FP^y*2{t?iNgFYhX}w!eEf+qO3dF(-Df$nT$plQVthSysa;>!K} zP~WHg5^?JG@~UG5$~yxz(ZT-SvgN zg-X8JLX}VbmHd-*s9)U_w;vx-F4}*=WQ|`&RJmW@gcaK?mrew%oHewXc#if=a$7tp$`E%%Yn4#eGBkLO+Qnfv- z195)^E#$^(O><-a6k0Snp6}{1GKvnl-dHmbFR2q~LgR_8JeV9F{$x8mF`?5`$zqnt zgIsJs_&l@|lgax~0Oabr(?<2e_c;txj~v~*#Kw{xtIrq92`eV}2P~XShRMbmV%zJ} zp?~rkL1UcrZ5S_2G(qiohc;`l4j+fecOTEQ9bVH{l7SjnL40+axhsYh)P_CQ7hAQ%5X?<$xpHuHfjM zieI`Rt*|g6CWZxqglG!gpxOI-9k;+G%J8U(lwk%HMtx~8@0V5`X=C7~(v5Fu^47aw z(BM&EPEl7kt9IDKeo*m($%eFT$|hIu{`_qG?$AN9{lwaug*LL&1FG%NoAD}SP-(mh z`N+v7#+51|!93kP+GJ2(!0el0q?V=AQN(Zgbi-#1ab`2E=^8tzPG=LpDHc$*N1ApiQa3@-PrTDSY|_t6MtD^uk$sV0cx<@ zXx}&7+2>}3{qJ9{?cnpcaKQmh{Pe*4lHsv2t$PfY3uvQG5UZSV>U+Pn09JR7qG-M9 zK#j8YbldZiBG5BnArM99k^1x>GN!qcSQY`CVWVi$U9?Vu{Xx}HSpPxLz5#xvS*};Z zZv#eO@pVjM4&eF!o*uMT3OWGWU+_gY-po9AKHSdt*VoO2z{z(sdJ75muBYKLBE4jh z8453j(`U}KeXrC}szo84YxsZ|X+A)`aKL9qwdx$m5AecN-TBi$9lJWZP_oP`Q3e7X z_JY?EvF3qm`|OyYU~S{h>atJEo?}jyRZMH}P0z6-IU&vHbw7(U-{3tnHNS5|ISbG1#+{Ce41kKykAhX;-ZZP4LSs#=_{ ztHT(uJ@w_nptv{B-eR;k#;_Zo-{=LD3f8B$UYl!ggsk=;*a|(RN;u2);PBen1TPZ9 zOhlkD!h#IUWe*TdY15V(36&(y;NQoujoAMCR-)M8YkvCNIl>2)LKJ%!!18!t*Agu) z?+k$?%6Ta)D%%=(`}URqSoFeK0~M{9poGLNV4~9SZGd&N%sX({9wTp}zZIaI4}n8Y zBq{}zY^GABB_)JFmbpi$`UG5U;j2r;&(W2$FoHu3VW8rV(JxR9LNhyI4{EI=ifs^E z4$;i*MNtwAEX5u6j`Jm$(v+Y(;nIyV3Rn@dAC_~4G(T?ctfU1w%#DuC=mW139X&^V z1dl;B!!5pip$hHEr+-pl3DK21%BX!Xbz$HW}5 zJ%jn-$S7s0gV?NO2%Q*L2J`Z=fIMgRk&qaXIQhh6Sy3Po zXD-?=5<5?!i|**^imqF_9#DpFe6O|^#whO}y#XV?(^sYW$fk65R#r`PV1#!=TI^Z4 zmmI*Oi?3Uv0SMrX$FOI*>oT!R9<-R|+W$3l%-at5sKUx-5fPM>)zAvToiL9Yw_&ac zL8-x(shkAtj0wfbY#j<^CA?8#Gy)eO11Z8w6w2N!`usMqLI8QdwB)&8HX`R~d;3?V z7n8(59^+KHEQ1&svwUnM{~pYeQ8>h-&K@%T8fJ{8c+C~08~Z7?Z35r@tzhyVH0r+aWjxDwrj zK)468H_wp<*OJkR39_2Lx}PR|1c){Xo(3df2sa5(Hm9L$f4B*;l@c@7%q=c(9@&N= zTx`%kLT;2jtCZDUPy_Qy!!hxy!A`DVs0mOu1%V>xOeG87`$JcgHNy#e=GU)p0c&%j zlBKxCx1Sw$Jj4nyD>;0o*Kgj;{`wVDr+Xc4DAE+;@-$rkB9N|bqF}O^0?R%Ny-;6&KNB}M`C3XaZs}=AGUDRm z6mzsKEK<-UVUZ0TtqZfdkoJZY&1k4ph?mt9iHeRmKXx|Nb^JdOI7jiWkFD2z`TF%B z3<2!jgBOD_zD@!yHLqp}w<;BLX{n#u{~pj-543m~ zyXIX>0X*Ui3-CyOmH0YsEu^r{L>qh97@$_PvuQBH(jQv9#-2wfQ!suOGEY?r@Q2;e z5&UrlSQ;RQn-+mN%u9%DAtXL*7?=6^#iba`?e=0x-n?BeeBM0aCQGcQLi996r?!NH zeEg}Wee}0s%z|%WaZ!UbQkvaHv?0QP;mroSS!M}4YD6;peAeaTSa+gu z`L`$>BcxT;LnEVWcg`9@O#cvXm4r<~&~0-N3uh=}4uT&#j@lY~<|pjoqR~@u+?eK2 zs2yZ%U5gt^pkYKDN`Jde$f*4WhZA%PqDa~V4hkeYn&O|kQq&ls0Y)_d;Do|n-2kst z*vpt~kZXe3kIn<;AP^%7`ZPW~&0)VNz^NdVi&sAJFE^Tu}Kam zCck|Z+ySDXfIC=EQmE$2)93I3!jWsmq#(;`0Hb??m=qEc!VtZ93Jw(y&?`f4kkj^Z zXEsg=NVBUk?S%Cq4#hz%ndoW5=tK=VwBpjzm|r{u#TKuNIPsW4SQ~`{*S%q~+(g5d(5;!eux@#%|v%bT}wkwX0lF550yZG9H@LZoJ5d zUrmjT*U?lTR$4Zh!x=nfYny(g?jClp!fIR<#_Ql#*|o)QhV4Jb#RAvO|X;!AOGpPm1nvFov&jZH0n^l?cE(h41_pLBHE#MaiT0$fRysT%hr z4~4eF(|I!}nytexZ^ZQa17?PShH%R~umyW~*AW&%$O7}87=mGHr3Uc~Jj`2-POo5k zj$OuWXp&@GuM!RjTyr5(N8i5E}_Yb9~B9bBJ|b0mDHRafdfiJ+xggNMEpjo6_arkPvE^A5m-v zYu^<+FAq6uy2ImJE?D{vW0;GBfwak0O6KsFMePq(r`6wX_Yt+ zGB>!@O*rea@fPOhyRlFg%=Y;2O^d*&U*A=Tfx+zTpDBxZPdH71okpO}R6^lW?A+X~ z#OgDY(k1P-Btpfr@7#)M(G?g`;?XrC6w1b@BqvKlF!1-!)XktEwQTNbwD{%cKDXkK zu)HY3_GLCe>tg#wf)`IQ_X#PtfgX`#T~n>t>+*?X3hFB)zrdj(8z8C*7bX)g6FcP< zu-qxroJ}x9D4wYh=s7h1zp%5QjdUdhh=r}2ThX5Go;PnIP*&}9TK@Wm2lgNYH4hav z9I7>1H~f=oHcP*~uXGdT9GDGmglPVrdd-V z*vOT`evI%6fBT85104d4#mG^=tcTIFVC~4rw{HnUkts+3hff?7FzP^$ z=#xpTmvF|~bA}GbaCDrwq(p?RD!{6F^e#650Es;P48Bt?IQ&>70Sm$1@DSzF`yX)h z0RfO;SL11!Iyz(0KLQm>!-o&a0ATYzvP+oX!BOdkNgh{ae@Gi&Pe3x{!6U^H)4{%{ z#%K6+qj3JhWRN#>yY(uNq=j7#3s3ii{E}CbSBB3VaGp?GD_sa#1+06#Ig&# zN)2mk6^&b7pn~GnU{-Js-%>U9o^bG@9Pxakh+|q%AhULOAG-cG zZ}eLgt_&v#L%i!PR2m=8P8jAP#N&W>6&&BYW!nMK?+-&3b>Le-ymW{;9vJ2JJeh>& z9YynfrTy64Z3p~V+IM>r%H>+On$>KN;U{<&u*6Ql4st(FUo6%|f2j#=4IXedUbb}3 zzkXC&7zquM#!E9pL6?t1KlK#tXPVtPI8Jk-uHer4j#JzY^2b=CjUZ}|n}vmi2Y23r zl^+FosV%Q1mD|AL`vvEDZ94sF4|z!{;M!9ZG5ON%hrN0LZii;hOpmvbA?L-j$F8%PhZ zgbYO!-HBlmmyAGl2d??!(o$pV0}@7=n*hX&U|$$6hf8K=A#NjPA;c0tX?K1C3j%Tm z8`5L{=>qZnQs=YmQRd=a(j!Wu`djPe2>dq#4E_BDSrv>e5$@d{z=43>J$U^3<;!cB z8qECuZaiOt)cqK)P=K05Dd`|)y9Gsovt`P=uC7yP?umjN4n@t|!|cM^V#~1n7CzZE ziHPzI5zYMupyoV6c`S!aH&v7w+M^AcToB`jWG>ClRclxVl zeRV68vod_|A?pKzLU_^)3=G^=^wv?*UeQ1^cnX6LgeCR2jslu_cTno~qPir`9Y#tM zQg<+Zv&Zr==7?n`VtX$VFySCOVXqa=N;p=mOb4tROMOHW2Iv<~|L))@B{>TWRwIXqdR|M`ihlJs2qZ7-s$}TK?1DpWU!+MU0 zgsLhfgrF?US5R>T;d!;G{u_8VcL2abmbQf%2V_S==7SJ0powMq{0B)8Z>|*Qgz6)) zaRjm3p=A$hZ#y`U5*`18yGPU2L*Qq#8FgkX3MPQXGRveD=OnzQC3MRBh##v+Emy^9b8K-Gq=@;4I`6RD=aLBY_1 z2j#(=lIhceV|br?u_^;!`KXCWkC~Dgwg6YcuMZzcw|j7ScD&OPc7L_>_x_4tzR{Cwc$C5N2=?^MXexIbjB~71fo#XA174Eu`h}N6};EP|s9E zl_*A-!(#q73&YF#5`6y>=kA^GR1o1xTgWjX!|L=Z%2?pk6Jr2T!1`QE!y+2m^6#Aw zHlhy6z{KsYlCRw7tW;J(B~9fktJife_%Bf^ zl%Wl7(EIoA$0?0=^@1%wa=}99HxF;`S7>~ZD033*+wb2a+&EyrWt65GB(di{>N7Op zBje-EXftKr9k;M3g>5i=YrfR7v_Sh5AdL?K{^xzcc1P6Rco;@c+F=lR_rRrSd`!Z& zI>L7ke`~6W3O0D*vERhi&ZXRrv8nMFOFx8TMbZc=O`l7bW-;Ep(=&Atmk}j+Aeu7V z+dE#SnVV2b5#bph#OTWs5weJ_$$t|19#cL$PZ3AA>ty}Do!5!KhRZ~i>dn7@aP;jg zHXzzy_ZKXg(CKq>bGwvX0c4NF2=kMxI9Fg=W-BMKv@s+k(y@_qt|e7#F=jCe{y;Yx zbqc$nbkNku7?I(aDSfJ86NpoBLzFV&&rmp__9qPgC+sn%(@Rw*x*Q=c+BFyf5QhMZ z#|S%U1phe7BcYMRIYcK%yztKXX~#2vJ0Kn$hes1Qf<{tp;T#MqUm$EFw$YcIpdds( zKt-cbcog|Z;8PM_VxYI<@}slSDVO+HL~!XK3*&Ft16T=j5D&w@@?pqvui%>>W+#ME z8XPJo3YJJPEMvgP3lShjC16cY;Xc9+kuuSQdO$o8;SKo44!=a^%sag;2cjeZm^y&< zk0SD6pBFbL=MeCRC_$~R$%W9{x4)v%N4-OdoEQ-pyhW9zc3bEbHy%kzNxXQ%hLCVr z!&KxYhF73yWBm=xti*ZVL~ydm~~4ZUZm00RZ^mlmfwX1S8bb=6~^+5w44mD=NMrC8xT=MnW6m zROQD~k#)^mjBw?BX3`oxq3=i80ngCaGW&Q3UtA)1lIBokAg}b8C!j4s$sCEG4}Fy9 z5K`x@TVL>@xySNw8C#-u+&H)&?@U#U;yt!7qi34y{Kg0D5YxFuhZ-|*SMYnjpr@q* zi_u+5>O9368%ORx@aVijHqFbS+)m7eaFE7oS;T&#I2qZ~v5k*qQff=T7<%157?kOXa66sDIfAS-A)$+9W!!PL z)d%sZdXiG%WkuB1c$dd7WK=K;u14X?k`cvU!U!ta=A9vcCYV95IR-el=V0mt+c085 zDR+<+zBy5jJ9lNa)7|FW>%agkEE5ABOKd4U%^t#d{4=mz!lD+pV9n_{0?sLW`?!K{ zctG~(F1e4tQ%%a+jb*!_S|&NSWx<2#F>JpeW6a-^q6MEjydFHSL=)ew@IGcl$+#0{ z$SBrMpFJzNvGQLeKK}!MhQM~w>DEfMy3b_0ElamA`wZw0Jvy=Q3Mcw;C?J+;#Mz+{c+CwqLzj=a*Q`_J3!*$>3hZ&o%1#N#Y5j5 zQ9n6)??>Fe-3as-&@n6o zY6zQZg$~Vlg4~McT*jzcf?&R zS`_O4$Om^e{35^J|HIaIhjZP(eOD?;5sDNk84204RVaH`8bV}K*|I{VD6+~PWmSq$ z_G(xWrHl~C%t%r;&+A=X_kBFa^IZS@j^n2i_sf?`)b03Z>VCht zxVT6v9$ISPYoSm4<48AyzlUoW!^cHD-QKW^objO>J`Nfe3j^IWL&P2(KcvHPyYu^C zf8mKNbhyrSN?Lx}eCSZpaRGzUZE^+Q@d+Bxj$y<*IU`oDS%WEC^=RS%jI_AXBB7NN zWe--D0_zGorg}5XXGl&?X4%7TNKH$d4t(j`NFB*cfbbE$E|=v9;59<%inCI$U*+gn z;Lh-^H9#Q%0G>8Xff+^?tjv%A-s2E1i3mm&Wn-9i1xs8P^Iq}~B(-dj|Kfv_AeEs_KL8elPko1gL=+Y2l~qVm1Dr)3R;TnUc#pWR1Iuvg(%D6%RAPpou8} zK)*Nh9=nO}r<-XYgvA<#v)r$-8W&+&gFDL(8jYBW;b!fp>~8J{6J&B;UfIMRO)j=* z@Y}+WvOolJduP8@8ARaEo~5F|Wqv~Ai7uJ2e4yY6txC0T2RW4S$$nxFah@2QcEptA zD}Yqa+S=BmM_gKSV7m4%u%`?x6%jY7cM#}fkx5Q*!MfMd8txi0CvgvAa^ z97xl9yJJNNNC5}zhcQ9;$eOqBJb!eD5tCD=wu#>a+~O17{S|9~5b_`a2tNt$ck@2Z z?Ee)0q1RD4AUm`U90a|YlqLzL#DuX9{oxLBEMocr?$RFID_bc9Z^o(o zCv!$HB1WHt9|5OQ!V?hb>=O#TNZMwgny8d`46=6; z>fdvAGDH8NV#rI7KJW1O&vy*lv_@nK96SiKvTZjFbN1@gO*~du^dR&Ig^r?w@CIbg zs|j}QHaqBe8&TTnmg@-#8QX2Nx7W_x8YfBNV5GHfms}9pMAPVZ0j|BpMMp(LqnTZX z!wRnYW`v>rVq+QBz`YAF|23wNVKdNg-5QmofkP0bKsu-}=VRh7fl~d5dU*YH+aV${KLvMpQeMKk1by$JhGK3~o}XL1c9Kq5);S2eRHD-lt`piM_|^Dz@s6y1;Nv!IP~a&q{KcThvnZQiP^ z(^E-l_c@Wy%HAFcH_qffdwAnWSka8Wb2Z(kUr^10$ zKQ^B$r}C4WZQ7oGjWuK`bo;@MCC`bX+|DN9zZ!H?1+F({88<;(y9*BofwWM9EPBxW zgG=ClLie~F{u!O3S24{t~9a7*P5N~Bb97~M~r zA*Ntt#Gs5@+x?5BQzrVXcL~Q3xZB1i2qILVI|vUsW4efk22$>7n0a4KiL`^w@h9GSQARja|A1qB*zv5H}Z!3y%C}g_;J>1n$Wsis^q^;863!wAP z(0>rdqRrk686`77Kah08L5ZgS%3zJpjXOTI-R^!#fv+nQru=9JH#o;_BD*lkk_WQD z|LLs0a#$#pUHTP>29b1B2#2atSB3->#PM=%#0)kX+P-U82PW3mG1-+L9zZTW>DZo= z?UAp$Jiq;dvvVpMMrfQJLFp2ZjXU7K+Ejlzp&Xz-(fh<|naR#YpgF#P&-X##F(duG z)fhK^=-0Ok3lb%FZ}npJqOU#X)zDmB&0Rb4l!p^j?mPk40~vet2EAaHtHb3R=l5a9 zlS>&t9*&9&SHU5u9q98!1O>G%H+t|`*rKFN4BCI1EgeZYRpx#_rTSKEXj(dDI5+0aru@Fz zrbxVt5Wisz4T(!*pso6k^4_Iefy3k$rqKe|Q4byLz*rv~MQUA|N0fY1Q|X5woxwe{ zX1%8qL!|0!)Z6sxQIpft+?aoXMFlIt{JTMjU0%F;bp^MsE?6@~?u)}JH~2!|0^&k5 zPz@{?U3o7+n-pm9Z#)+h7AE%W74U^YaZ#?3PC=3E?N#8->b=Aq%zI3B;z;%Nec`YbO!=(ak3e5msT&zI-kvQtB$J+e?+I0Bd= ze-0%Oxt^bmW7Fut(?VjU-#SeLNbp{&Z}O#x6gczU=KboH$Nu+}nx_ho)|-lh87PV# zB+~$rkD&DJzw$rAjk9oA6E0Ql4xh@@Ni)*dE~B8-fVld{uV0)f7;6Xl(2(KqR*tOy zX}x#L)VI{r;WdeDnu*791kx`;7pOCU$y*lA z`#QpiMC6Xz$h>J&(mo+37M8fat2(%MoL!1gQ$5ISjpRSY2#Op~P`vPohYO2e=D+U} zKj!E@-r3W;>Va&>6a_AE@=;JClj;!nL1Gcpwg37AGaei|BvZQYZT$AXu+&Fib+)w= zBrOGd6*UPdal-mBT;Z6T~lTN2+9#wNm+oZA;qhQgPbE2Xm)=Tih z-g#cWtJZ)!89?){j{<*8dSOmX)P94Uoc0rzIGNc7u9MWEXNW=$?DwWkn+~A$B~i@) zGgO|JyZ_{pN7xy(G+;$|e@VG4N_}`tU0I1lz0A`*(rW8=>0ZY6*}#ut?x1!Bp5w=r zD^af_4pH&`LO25D0gsYzgmF3mzmC9GkE`aUl&r%Idd{%BANzjW^&CRxUF&G>fw`O*|LkIDqe#Omkt1JNO^gAl>?p(Al}rn+m^Pr>i7v_a6@;tJmiS+ zX}8gvrrTFzT9Bh720zXH@&^L2z=^&O`YR5P-=lq4BV(?bL5vrMw0lP~A4g7?iF!6t zty@QYIs}XZ5k!P$lD_|uNhuGA3d_W#66-z#cL~5}vZSa!Me?1XN4=PX%jHu1i9-6( z(VZSHZTX0}zRSv?NDm{?OEla#^Unb>A#G`R2drXIXxp=K!{mLxV6ZjM_Q7hANS^!Z z*?hW+E-5vWiIpHGt6N*|f@r*ep3h;Re)gRmreH(ueU8Zf&;V-22g`bAavUh|VLO?> zcK`kv*_r$}t9(P9Cy@*YLzUuyVgmnDg~s??_w^0G zl=>jDA?iq3>i11eg0;e?;3=`aR{$QrRq=Xy3ut0gLm+{JKa9-G8|m}$M+)qX7V@|q z5HP94$HS@iqUx4*)|nXRTHr!p2pEt}i(fu`rz-mXf6va$5VsAX7V%q%%?9-;p?a}6 zRhnGTW{_U~msGBgeBOClA))L`&vb(|*M(*fZ3WW(@nqlPXtRP=F6TKfPAhaIsfst~ ztrpEGkg`(gubml1|3Uv@#Hey;s*;k@I}AJlha@B>){kmz^B(P<@D7~M!WcBghzBpl z>anqJu*ho%S6w?Pm0eBqj=CN4z4=+Cdhfvx?u=ij{hAe0-AU zAVxyIcy1Rhl0g-5!lb{Bvvg_63j>wc(jMDakm8H?(Tnj~2fzeK_$-ygDgQ?&>l)hH z@)2n=o>Ve;e-?=gE3kh6R5Rr90~|x+Z}ZK0;KiDuPUd^}O(xBt2<+WW0u*q(0J1j6 zBdevcUWHGl{fSacNeP-ARc$@rRXxM_Ozdy;9_W*KC1^v|Gwr*Xo^Fxj7jxjjKyOda zZ7J5Eo2z+wIWry{S|f4cfL{!{{i`ToHyJ%LJ<`D@KObzAI-Lwu$mSiC6K&oi`3M2_xV2HDJb^$JvhMX#?PNx|GD~A z)r)&d+K;!LE{);p%0bhKMx?}R{todGB0ySF<-e7b@({i(@X>@!lf)zmw3MrrMYC() zy`$^;Q1{JyFgQ$BN^~A!`b8r1y+P)0R}%F-|=|Kt1j_blJtHGV?~w=5a`X-h(4XMX!Wy4@pa?7l-gwTzs+E z^dtnGD9{3EUf0y@dG{Qso7R-y*+A_F5!BZb60{4lCm}Y8_Mu$(y}hsQ*Rad+ti8;c zaq*E)RnI|kc<-;h>I&8z>P{~1p#LeV3G0r7g+%Z}IswF4sdW6M&1ST@$%l{6knHc|Ess=;^p?t}>1p(+VXqzioTUlA{#Q)l+(iCxSm6J+Y@!~53J2JBKua{s9>@CId;3xHzMvY`A-!2z7{erFO| ztv{x!h+#pOZK)7yIpgm<$m9z=FNrj107PK=M&x$jeC%tz0NA;&k+R}%pU&Lunw5sSSH zp3crDt@~2L@IJ2V@4~uBxzh4DGImJ?`H%JWJ9qBXC<+%al7sp^1pgL@J4uqs2~oVZ z1)*fNRF>s5FuDEFQl z4jXBdj}&n9_?=>%KRM9w1{F29uou8_pysc+@!v!Z_SbH1ZUnbZh7bf?4TwIfPbF+^ z5%@g7s=_B(Zvcc~eI$LtHRk(QlzC7diGqhlGo^>PHPov9=(7*Up8g(r zL_}ElECQ<@G)E(fyKAzNlHjqp3YLH+qh@id4V7<|}!pY-aG07=J)dVwcT5`8vmURNJND+Nd(0WuBffS|uH zm>X()cOQ?w<IQ!Qj;2$3aewl!(3$1Kvxgc^X7LTwckXlOTK#%;v+`q`gNsnjsrGn(;qhuWh?uO( zJe6+d(4J3Q7a5L9h8>FQq4%lOXpzbE|ZE898BpGTkXpK>(*P0V`sWLwI$hsuH;!J!b^rfbM~^Z?pPY#?~Nxf9jCI*0xW3jgIByh}*$XCD0+h^FXwnxOGIBoctE(^&!41+C)!JRGMx}3x#po z|6R0l9l)GG*HHoggT^HXy4R5BA;oe+*nGEgoOsUu`Zkjkz<(;q#g43mUdRgJk0kX8 ztax|~oNC5xf~j#IabB73CXazg8|^+tK-#EV?(Kygul3w^xaZFAc&5kNTdf#pLuYMV&qdVVhExs@K(DperTC3iP@-O0I{8>&<~cTgl10aJJruCWxmsG-FTp^R-4sd-T?JcKtcsFHZLB)meGp3=XE>bgWE< z!+GUQ!}g(hpWoxsPwFObeVw7C+tMk$MeI%8d*L_guVu9l>a$1+<`=&FkYn=1`_4u+ z$Dn~78JE}fRy;0v<8aQrUtDyv&d#bY3rhuW@-pj_26uK(k9-WGkco5bfBhwH(EG`J z(WTg~w7UZ7HpFaX1~pP2E*0F~f*6X|uK@-Z|1IgBJh@GvY=#|N3-Lsf`xvjG>to}c zJvys=e3qxE2j;(POb^Yt4Ry9}7Qa!$M$8g8km=aiv>+bD!1%DQl%>{@Cr;(I_moZ_ zh@Dhifhk-dOs}BTL3?ro3e;Z*Q&SMzYkq(HC9Q&+L|^TFvm4hJs`cAA!ak3u8fG@E zHLZ#Yns0q_PT2qrO0m1{{tG<^G|em4?FsckRDT0e(B$Qb68w_dRZ^=A2i=D4;(Uv% zo^~|QJoq(|Y^h^V%ow;u&(HgOPlf#QnNW7~{llqh*`@&}uH3qH_5Kl!fFk#wJ@Kwd zK0dk4{oq*%tfSVt$p&L(F>zppK>30&a88}ie;-Vr*2Wz&QKaz)i2+Oo{_T3hWm_1h zzZb3hb{uU{3JZ*7Rq1qHf!kUv`Y%YIN8sxPcGTXUAY+%|=B>L6)TtVlM%yxieZeMThcLOy0ea?c6Q{AD`IQS;^ zd`!Hqy?wS7;@qFCUW`8ZZ0Fu@Ax7svcWxFN>!7@~ExI?mnEC77n3!E(mx!1ho|rwS z3k1=D_9KMg*!aI^Ar9k_362Jcm2ym8u0<<$PGct()xhaX)Anw^PqY;sjo+m>-UEO? z1>O#*N3WvlCg$8~qmj>r3^C$ju`4jUtXGj!cQqVtJ*jukyg>Xgh-3`T9@*+!`a6bs z%LAJ=?Cf{+s<}d^jy=yTZ1c$qW)b=t+meazPW%se^thmVLE;`Sd*y&;ffdlRJL5AJ;t%>mcPV^MU2QCk)8uPaaku-Ol}Z{b*gPjz z;vaK^$J8#vnvI44v zXzU^er{FGcfLDU0Jj1GS$Hb(tl7{T13hu-`q3j2MY@w#6G=_^KSnq?VT@3k1LJjMj z9v@JArMnl7ly%?qGxnhw}Fi#7aQj9fYK2`iCDeP zcGe;HL2GTT?KS)Nina7~PovhyDheMecDcQ~I=*~*gi3Aflb{v-)Uc^N(_W|fi&j?V zRnM~oQXE}F@iOsQ1i}%x3*QY;Qth{W@~*HFVm5DGez*4<$41#Bz}=jKHqHND6z}k^ z-;QH_6sDU}di7mr z7I6QyH_v{fi_qfxU4d3&y0IdFCTFyk3a zS;aI|5?lixbt3d?do3mtRWLvg0(-#iB(?+7o^iQFiB)<|$MaMaI#FHvB`lrRH?DLf z_MKyGZMm1-%e9uBGx?44zpqA%XfUNyP}X0U3PNQQiZk3Kx80z`O$bjypGN7=qsd8v z6hf!i#SRePA(VhoKSrupc==A6?Yn3G#=J<;pj}M$)xmg4N8~GwIDGrSL*!x5w4h@M zW}T`H*G{K7U1C}EGVA2IqSzp_i=$ zo*TqW0W`N?LpZL>J{?^x37x}!q~EgTwLh(#rH0^L0sjC>&v||~kJ#61j+{3fyDsH= z?RAjP{6d?M+rp1-IV}~TtC}6pquFyBV)+eUMT=U|!?#d{W)8L3k50!%o`Aob^M?PDw$51e%v>biX7Jfe0+%unL|CsLC|NPd^#sH+(p5&CX7v z=o6|W92i2|*SK7rbv)D3#TePeKYsKKTI`!!j_iHiT$)jPMK=`d_-ua1Fl?1LEb3y- z3V__?K*dI+kN7B=yw54} zMeIiBL1>k68=+_@PY1S(+Uf0&pqRIb0dv1Jbo3fx%(wXm#wfyToQ!A8f_x-=J_bg{ zfvrj!LPF-IM2v!i1lAdsh3`{nn3$^oe}`kM=i6%?-|+b*i{?-#c zmEfWD@Dn6_2DQjzDh?sKlX;jq?aQ4g8o$e-Puow?p7i{%pVebmyN^DtHYoR&1tf~v z`fG2J3d2kEl9ljb^lxpeR9Mxj<74D_kzYy1`vJA>PZ_5s9n8BHiXKT({nfk z;Y@D*e*j90ezh_7&XyFKv{n=!dUPDL=R&Et0 z80vgxK^BQQg_9+~%Ovy%5`tUAZX9fJwQe@nc>Fd2%4~J#aQC`kHrN6$@x;7t@I3;dS zvbF9mOln_J@A7Tz0P%z3A{PAx9_1?(GLdS4jOpk?=Sg}K@$^A2ijB3EmX>zJk6C8s z_uEZAt-lO7f>e~1D{vVD&(Q&5L^iVfh~nxa5pf&rI>0~-NU}1!&u^v-<+BC)ib3SJ z@Hgb)q_hT4CQ2lE?jdhtSCXR!D6#EFLJP@CwlA3{Vf873&BHU+`whL z97w5OwYIhv8CTu%=h0K&KYjGk2j)Y4t~d^Xph*HWOb2>H0#1QCOSv@h@T6r}jh)MVts(b>Qfu9V2M?UaJEH;B@%l`LU_>OmJPoLaFzjF(ASAQKgsvFl zvgI8y`}a3KejBf05^feJd~STHZT%Lw$eJsMpS9Mr-&TTNAL8PDC?NVFTP1L0N?O|9 z#J_iZ^alvEhM)k%A1nF!R(h(WPvqxuO!i{mdJ7N`3N-lWvt^dVMj^mTLFmsMm;_mT-TcqfHvk_3Q6J zX{UJj@Z`5gS$pOGec2|ga`>;ih|&`hkX~Qt7ra4cnd5@#QL$aldMAuHE-DBKO|1!xBlj93-3T0a6$J$k%#*+ilEdyZt!p1k zd-G;$xM#q;Wx=NPn6Q%IZqDsu)~^7vyG_<|BRfGFIBK?BHZ@a{xrT(-Cw13#R4X5G zsiWhR)j`L*T-R}a*-M83c`*|MvA6G-pKPysWk5g9pe0jT{Jo6&0DPDQZa?lh?D4T7 zL$L5Lpkl=lWkNdCweeEX_tG5tcgmbr(g?1vt#xXCs81Elw%=#|{d+#@bQIu(^0^G{ zNDj!9|GqRsfy>x7kVL?ANj0AXXXXegQdend>3#T+OtFyjmW%ybgE^UF55qpJV=Ezn z7@R=^omFlh7<1aNA?7}cIeWQuj3sG%X2S%`AJ0?bunwlkpz6MtHki0SzD5572Yc9X z2HE8AGv0NvNjszrPdF{f7_{*ayAVFzk7h0TcaKl)|InkcREO{T2B{Fx^-#iSfV`jf zBI7gO%!d?`FPJp1pYx@R*m>kGH*MybjY@%0or~Xn?p$jI<^rt?hJ_}BP9##by3jut z_jL+2b@ddenaOk{>{mLV(dcY4sW2KF5)K1C$KIQ_@^9_24n!5sQ zIBW(nXx4e5qT74B+{_AxzRO#9lf1n+_4=6Ll7M|u8^r*Q{EM!x`EnV3W+JP8zY+Ly zgL^0YDu=;iWUcRt*SI0kCnB<)@ANuk?OhP8k2Y1=iUcJh;3Z)jz*0!saAyrGNdiK2 z^KBrC5WA~mt39R$D-k8$FdB0bWs8S8RUKblq(7%xMLYB15OGv%q9!j^gW121S<&d? z(SGtC`MbGkYW@{#);}uZ_}dhCyW%uFQwBJ_5tHvl`qOA$;E3}6v3U~=7(`mflNl8I`Jca z_N*eyP4s<`HK8o)9;MeQabe9PcG=M-@S)e%# z`j7niny-9svPg!3T*Q3xI`oT=$LN35@ow6@diH1RJ+4RVGZk4rY(HNDF{mEQUX*Yc zrg)og?H50v(C#+cx4JpqKQWP>XEF$12-h!d1$NfNuV2nLD?WSBa%l*g?)LZV8BUMn zqYG7kTl=Jif<1VwYNOi#^UUv4801qGlE3HXcHqJY($bn7h&?Zi;-jA{NE(<2fJhju zZXi~c3}PAan_Y0sINm5@^!d`8ysLNW#|9IVHCfxryPYncEN*e?&8Y#v2=5m=KmRRg zdfu0LdHg$wUY#X=Er^#;+~gvJ4DSDPhJs(KUv57i6_i0o*OZ-1E+cFYV!Hd+-2uLH zT}V3q1o>hW=Kdjae#r`v%ArQ&$)o+kti(pRWl2Vi>FVcZH>|ey^sRn#CwlEFb{s{h zH#kJv%GqRO>Zv|l+jxinCR}Y~gcO$LKjyklgV7m2N~o)d^_yUYz_Y>BpJJ$|l}8-& zCRRJCSOA2YCpmsfJFZRepxI~W*Luc(x;tMqju2IUV{ggHk8!#8UX23)9E&-g4llPeSE zR9L_9#WpiBk#V~41@+DjCR%{nYS|+8`qhD4^Rx`KTT#--l{BWqh0pDjXd87|?4ii* zs5*oaBo+E@AcF6aZUh&3^J9MNPemVZ>g9LfbOIi*@RmE# z(;+v*3bkwW{R|%A_fFQT8dpPiAN_z=3a0fvss_-6SMY%#VspJ&LBn$S*kg5_=I!S* zGE@}%sc{3Hs+6#@dsIy9c9E9vU$T7Jl&Bc&|v zA}EleaU<^icbF~{DV8>DxKGZMk~D-UguX{s9vJ(yX}ir~eXprdvNTDGKiVu5g;kIT z4Khgxx65P9w2@fCXi6KCkbV0KdzcWNNk7-AXzt?|Y&~V;FS9t!< z!j$x2+xXRuCXBw%io$Q(cT-z{r( z2D42i$o33VwPs=byP(JOz}<@8pF+O69lQ4;)P*jYf zTEWilVDv8fjR^O7C0npj?7%(#zU+JShf3j-VS?CnmjKb3-7H}gzs3~)FRU?UJ9r!4%f z1kAF*2a@L0<2N5DS<}DE-mrM>nKM}S_J&dNMoJp&`9xVIP&!3V%s#a&N*YF9=Mq}+ z1@G=NH+_m?XGM0$7zwIHoS16Fo`Ys)DW;pnI5`dkG2*vaA2oQUir>OM%@c2T!%Q4a zV30t}u>x=JJBFR#p-d(R3B&>DH0}+YpU!vv>>hx%69L8qF;<7ceBsvbl+f!7XqWj=eD)2 zMm`PUT5@PjuREJohKDC>Y?p9axA&On_AJqRr~A^tx-{Vo!I40u%HSz*j+W1^jS&?J zr$3p0P0~?6bB{w1cpTz^B^no?hi({j^O$xo=W7|G=~3SRwrdj?YQm2I+%MD`j*nJ0 z7q#0aGq(o=ZfyXfbNT%sVwI{Zzwh1OVoc;R52Zw_Llu?gG$g_~9?-P7lxHkfr6P z51I1jjZbj*p9Rw6x$vVJcCT#QKM(aEob}RnkuI5{&orV{3W{m{+WVT^kn>UDhIYpk zg#n_YjL&YbkIg@Pi;v0_Uk5&%@ZCpfa1d7OY00K`b{;{gK{VfB7$B<^0h>AE2VGK( zn27XlZVkV?M#ube9x4T?4@xmHwH3U(nUOon@<|h+JO*W+k|DCcw?Z}h)ZW+d?WeO)abLrwdBt1y7ovfvt>uh(@Q0eG| z*YJ~EB-p7iU;FF*5IKjqdr*z3>x2hUucmy2ONApy8$DM&AAiFrM|;j16vo^}j4*o) zMq4C&!Ab+Oe3MpwLA$WFk7Tpr^0e(TXEm_?6sCd8 zr-Nl|vlT}JgMdk2hj0}`fZqQSobk6&jB*tpb?i4in{1CJo6MKu0@rzNH+fX zu+p_VTB&~xHQz=H$h1UZMa48SlqJNK_3uSG(2}nQISOQkXsc1s7oTFPe{R>5LdC%u z{X_%l#|PQV$<>7)>MXeE5AmiDDoLO;EYd@52Jp?6&8$keRA;+wpT2VK=GoW4JTRwp zEy=e9;zqbl+u-bXZX?IAmrHi|Osp)Kdk7jm>}p2;Rb}(RahhE7=nkBk>ksfvjOI2H zlq2-c9n7#bL{M6wLpT*S%0?Jl2%mwKcH+5hH13xhH_{l|e|*(`5H_rwc8{y@X!4;= zjB6oVeR0*C(ttvI1EWD^M~V|5w^A5kBJ`mL(1>h@0}CXABSNO#BjK(wYCI zN7f{CbqYxN1z5S3WK~1p7i{!lGn2*oE#)(-c6e=W$vRGc48};y&H^Fw9q~Ns6Xm;w z0%L+%X2*UN`1qipx^8yxTN{T%`A|zk-is%?YkrUVvPdrd+@!0gN7e$t?a=~bEWNIK zt)lnRtOc$-;_1VwZw7TQ%fl{%=)ey*L#6a}d)gB^=iR5AQL) z;BqWX5hD>7!Uz$9;3c4)5al?s-k8F#WK~-<#@MV?yBp(dOh;F%J9XEn&s)g(Vf#6UiJc|C?r`n`^cRod zzEEPF_VWXo`NMEIv%y@ChTkl=eW1V@V;fKiL$BBJ$;a*ywnPo@IX0gIz1+x?anSgo4 z;D+6p{(uBC{CXDD#$0Q+?Zn zCU1nYrnsSR`P}*hw^hd;`<*U~;GpY?JzrMY+#HS~3;o+`uyVBtvc;y9@nL5JV_KeE zJ^NZ+N3YO9HLZppNDUDpLnsAQvUhyk-b))rdJZRxJWb@@=qHOfWpMqRh=6Qhocrjb z7a2On`3lqHXt`9j0$jX@r3fPO7O;RHPl|;2jmqK(xDdcdRHt|+gug|%n~;-p+p=A_ z*_|73f8M%n;`)Al`?}wmiM)>A!uz7tDE3Djqh<#`bkfMC>uXF*1m=fTIDNOeW6T8& z&McNgQv{YB{;O4`r>BSHXbs{`25Y$!IV2>a;*J>>eGOuC6D~fyg}2wj&~f6UMpq!4oXA z&qRFmBZhMr=~f0e9aL3RP@{IFLh4g@#)_n(I@WJ}g=r1Ev`ekK}+ z9%<9ot=I7n@C?o1MD@8e3JyqFQnl<`7v8i4CsPOm-#$q^_!Qr9@|oux%U?>pFi3EnU1$OVTz_UY~;6|GERVa7w+ zTV)KW!aRNs-QToz-P^Z<4k_)S7Rl%o2)Y0S05Jr8xU*el#sSgoKd%)ds^b3l8O$dD zaRMs}HHmJDG99+9TaPbgCbi|96Xt%xKOV{Z1y~9PU4bO~B%TDsjM-!!nB<(>;bPXc z8!+IsvY{c9_UPk>W`UjTGBRTi1QAK~9ftm?$rOvG!5r$Nm!6=3R(dN?+} z72_fhY7=>ctO2^htkNG<+mTDl1Sk#hMh&JRvgU?@yvHgke0%!L*6)&$joeAsgBJLY za>jec7Y89+S0~|t9GmeiUn=4`=Hz^J;zEYusoKqjMC0_2aDn71W+la1l%XqE3c6&r z=M}jLNvKlt(9u1OBgeq;$ERX-oaC1`0O<~4oIyDS$2C3N$709^nqv)-^i=!u5#4b_ z@di-TLIf5K2_Y>i2+$Wr#!n1Ilr$_F4|Nr#br^`3imFHf+mdp+Jkr|G{j$wK?Pau- z^Inv)GWaLt`sCo|?*6@aiaHj{0ezs(UvW%dnHc;YpM93SPMYHrm?Ss~#wz!Ab6*38ppk{zONq|H(?ZQ}oLscz{r24zFi*sc2|6?n`ypd`e!a&Kx z#wI2cYLq-csJVI zYu-R=x0W@Iqg(T=&xx=8Jxq~Rnzs7J3Zmvk2kE)^s}%q~$v-E7KzIz{TIx*aEi4QQ zHkA-``4pKjfm7EJ!N>62BgDA~c%7?&(C$nEU9 zXYZT0wdAh*3NZF`#y_}qJ|9}7;CH?m8YIsCIEZ1qt&2kW70^E4P zY%>u&f@ar;*McM~neYqa`9qukOB{gS3vr6K3DU@hRJMU%5^_5(O2t8iR!%XaJR7ipAviH%G`wjh0g`JHOU z)P$h|tL|f^=AWi?eEa`0yJR*!&dkTuT>IWSIY@eKdw5i2Fz}e&68*)Us4HHgFu?YT zJHqA_0lHZahi5gd=VzIT>HVxF>00}@?^Wtc`%5;yfpUU`t&mIkZG6XEg*U%&nZp|N zS60mtcm{>L2}!=@Wq^YJ+6%*BJoY&q%k%Q$P`hAXV02OqSPXj?cC?op&==*i7NemL z3lERXcRKs8Aaosng`h{o^_}dHn!6uj+^eW(Nxyk@BrmT@*i-9|!sV_m(o;h#6MgKh zBv8|ngfP&=lP{&8PSgU#_AIM;l7epR6S z%4Gj88J5n@2#(sx-mhP=-AzbE4QLF?e{G0=@ErP8hd#b>y1qQ2;5Pi{ZgeS*2SS8+ z7ZsRwuiw0xV(|2A3#E%q=#QU`vFxTthIYbAPNM>O%!%IUgpG`RNNzV7I~gVPGE5LQ z=my-}D~UT2nUDN!P?ixq`-z6dZ zEYW-J20+w{$TKB@>nIm%?ljyxXI0!4xBc+pLpSILOyAv84Ks5`1xQRskREFPQ%;=7 zN%=}yX1xV_UwnJX{F7mN0rq^7sEUi71Ic2LVqBwS^K6<+3JY)8Hg?zO#n+J72Q+^R zC6+tkUG@h@80nrKzL4#n8^y~s+C{A;8)@%aiaBW@j;jbHhJc8r9Je$`9O1&hiDtJK zI)YQ@OMeBJ0uC`ia+kwzfm`5P0L)QhvH+T6FhSY-qU<`L-ceQ+?sCj)+PVBJVl$dk zmr+5(IB`4S_?PeBK?XqZb!bme5bQ{TTb87IQSNZN3^q}>#O)%uMP;C0P%Q+Zff8^Wp1?Qp+l>PEe$?KnfcLN;JR0dM2e|t;tNo?+xxRlq&F4b zSqHLn2GhTByCOpnW3INNFSz}catAY&oSgaNi+##lMGm8Bj{lF0z?DTyTMeDfT+;jZ zNeYw(PgkklxnfxSm)cTX3d22{6LY2%VT&0Cbicn-9&dFj9zo0f9=X9`VPWK>BOdJ@ zuxOcIAN2O0XM$B3-mJaHpP;+9Ab9{rUcWxFjq@U>8#b@6X~%k|ZDD{xAw-EX56LyY zSy@@ml^FB#%;x3CM@?CY&T7>W|7iit&jmSqFVKNWc^fZc7$DSXCxHc^UmuT5 z$c`XrYO{SVFPnz34*RQJ|0?s$7ioHOGFMuW)?F(@L8g5mmI#t5OddeSw`7 zn?23R+&GvAU-tV*{w8U+z{svaipWFyN6=b4ki21TMkN-@9)#&3&ZqC)5w>UhlClMX zLm0Bl&2PX!+AxkcYM{x{Ly;;r&k`c4_Crv2u*y%!dQ>zsU}hMi%6{8kaL_%ZWd9Vw z%GhK-OHK$)`3A3 z0s?v;^ozw;uYH)Il6kzeh{tl(P4v?tm5nXY$5|db0JN@xI|HdrL)FPZ zP6B7*`{zKzIfY+!L&(%-yybJLdEc#uwafE~1iz1MAOH}VUk$Dm^3;8fo5j3-MV_!J z@n$V@5*m?4>s{D@OO>Fv&>5pdm)6v24&$6?Idz+VJKxC^B2Aq=^5Llp$D>QjUT%v> zWvY+cl;+4g0ZBC?8|Fxihx6(ZO{N4@~?L9KG>{Pvxa zv*XoFT-^To`)xbdX~~!b9>FN2uvZsL7KGIkY?qc;C03-Wo@-O!Mep!+ zX?uhu+iwO>O&>?M*2hMKEScXv9#1w=C~%^J8QJN{MH~&>X!XdndnTs49z`K|MCb}r zPcQn1Gom^ptQxi=EbXduatDle#>xA*R(~2ZFKI>0K@NwaB1k}%#%eZ&#kq;EAx+O8 zGi%&ENbn?reIC#HEW5aX*4IAx`=1Vt?xH?^nJMrq(j5XppAx`&f^Mzn)3b8<-@Qtf zoo$=N+&y0AS*hMObH_c4bao<>B_S>Fy%8+g{e9*a#Ul#MX=y)$2vUOp00sDRB{4;Z z&_;M%!8#wDdRCy|4UGBhra^E)2yQBY4+P6NY@m!j|0}qKCMW~kP{Dd&1oc~4n-&g2=|!~ zh!kIb+-kZJ-4qhtjy+i#NJ+&KBMrfRt&>O+i7ca~-?l!0w%gF8d1^R~Avh?AXPY}t zR~44y09;u%6gs~2TVpP2xYohJ;RR$uq<*wWa35e>90l@oP+vd7f$Pt=rx`@$i5U|y zSXyViejUa9p0Fj_u&8y3o@ecsKh7ztuFtcfq@;a8tY-z1gABF3$RU1NovOt@jO%MhKt3T4=A4zoNHt6f?^ zeLS_MrfL+F0u`Exi(a{wUx`+|FYgHS`&v|2dOw-e_36mh7Ovn4a}=fJvW-CmR+7@kbHPwt6kFGt3ema_6X zU@Kh0!eIdQPO<49T9s$Hl3Vk@$A{V0Tl#8}q13IBCBm4BS2whiSFy-io3mKlMyE=| z>6oA}6SA+*hNUZ>$g8G?9e<{rgSv^?@qLa7#c20=yrJM4R_P=!5AbPiMK>)a1;TVd zLwDCW+;_RfA%1x9|i%T|6-LVtiz|x zh_y;j@Vm9Wl1{4WxB0E}cP4c=-Ta&x?xO zM!Ywo8eO|iSfT-f|3%e(E}p22iIitxpbGcD$aqSBOJS?e^9yc&`e!MhUw`{&l>rg^ zxae8${y@pu;E-jl9nayU@SN?xTe2|uMq*Jz+B~)aB$7x3@1Jk~^fW3)IFRBn1`gd1 z{kCnYXnI8kD$y7;r^D`|If7Qb(&M{tt_a_WRbcm2!umC$1bs=BO8_`N1JKv}Eb>DI z5^zO=blKtc!O58S59tz*@&v<%h&ddsbrnX-%q4kuOA4{C3S|0o>>D}e05Dw?xxNmz zI%BCjoB7uV&{R^;nhm{-M_N1%`-^yW#p}i>$mZ-HVuIVA1Rp}|!+^PZIATNwe8FOD z6Z4+;oI;Gnx8J7wP78{RYbCB_V-p!!Vsq-FzqqG0EB5tw(`xmL&2^E>!=uw^Bb$M4 zy-kz{{D-Y{n{+6sd7OL}&@YfqEp~YktvLz!gEo$wW~WcHAXA7NYy(O`a^mAGN1y)^ zcrk`9_F`rmt|7O+&?9@$e*zd!#~Q}!7@Agl#g4?!^4XH5AJuGy-iuO@azF_~50Mdy zPLQ4?AP*2S8(M}=%t=GeDEv^H`A<#_{47L14S;sTY0v?7wBklb9P2y=_<4FY zJk7O$SYs|VjPE>X#gfT7Uc3k*l|Tc86>w$s#H62HhkE+mu@30Qfz-PGFOf~W!+$J( zqt9A(BPyHA4De_PHQQE4teHve~LQr|PPh zt}?)biE^G$VUs%#Ay___>d}u7bgTSQ90cItcW7`*SE9<3M1&{0Qs&9!2n3xVGE@P# z6jDCLk!pAk{oCH)O)3_6^Thv*I|0nF%THO&OLF;!r%&&JrH`x_Sb8zcE}SUXI#RXmpVIPSQcPu!v=%(iQ!YV-?qG313DNe*q zkVs-7tDr`T8UJ31ouY<^;|@VF`BaeKA=hfgs5>8^Gz`0-3KKEp8}>Kn*00aL*S=3} zdeA&(^yqYZrqYVhZrWcXu}_K-Tvii0rxeb29^a3gCMzHe4u^zK+PkT?Hf{I9r)NFm zkoML^>Io}(TOg4|)UxUF%#0gYg@%mo^NP>w@~l9778D#Xh$^fCV;`o8(uY02_(CNg z&Eixzv)X=3?Q8NWsPSRK>jAsZ!OuUY6(SAeo|epySMp`c$mouS>Liyuc_a!PH}ZN- zQiDd9xq)QLOf@O!KRPxxux4Ru8v}`cA=Xda{q+cZWtQ`FtKuAh$~M|UeZI};*U(DG zfv651o}RIdp`qGkGNYf)6?xt0q;1LCU*7E^edSsVo%8ZjB6z_-b|9jY0pDfkhVHY- zdoP?n(%SV6)hULIr zq!Odz0C%o-)_v;T5t7i0OisMofPfvBl&RnB^OOLts zj5sg`Kgeb|gm`hRv8u#E5FX$iJ_|o$ZMHz`kODMedBz)r z3$bUzO(uZ$w;xi4+JdApmt0NfnTatYpNWGrUnpnBrJI%J-g6Fpq+YdZN{#!(XIzBC zA0ILOAF93sn(McJ-z=+=5*k)ym6ReWq>`P?$P5XQjO>zV${r1?tPoNtdzKNAks^`o zLblBRb@%=K{^$Ij^PKaXr&D}B@ArMb#`U^h*M&+K(_zxdS8q5BXewvlJLY%y9tElm z=fny-%90h<|DkhY=)AEI6vd>$#4#6k(q$?}^ykO7G?I!Iu!mAHwk7aSgLF)V8h1mq8%7G_9fixLIMJXd!$ z2zKyPS`Ubd$Th<4^q~1o`_3)CzV@pIPb4lJi0ZOeSKqQ*>>VtP9`#}zR{Qpf*m;`1 z2HW@hzaKa2FLl9ek~ngr)?E2hjKuJBc=&IW>~-)r33CiP^O%YcjDMeH71He65VH*n z(F!nGA=MvXrM`wIY>3-{)UP=C>6|~$ix>d6N=MZE(pRe%3(w!*Z&}xrX(UZ{Ww3vo z(k&Yzd*$z>C7Cyas{;-|X}Wy!xwuX&QuQ=shC$aGmGv30<*`Dbq@uD|>2w%52MZxx zUNT5m|0EyM7S{F5-xjL8!RTc1IMLd^fpWcu2IZBVQcj7H%O}KD8jK>72ZWtW`3=-* zX!Lvqh3|=9+3>Ar%bPm^Ju7POxRjyjpiH;Z*wLR-jhjB)-!6~R zJ`80hw5A93I>tBhW)W8`(E2PuWFcV%0hn#23n}lswe_RSJrN-x8-YtANjfdg+KoQ`7!>){!Eq?QaW|B>!TmG$9z@VS zL(qkQ{c`PV#=1uZpteu0ck}O?vf~5$C?E-sna~N)8$irzVbn1Ng0>ZdTv}*JLVIFX zHb}|1^YOs~2GLdBw=Xk0GH>3@JQ?%k$ymOoi+bM`lptg+k+7Sfqkt=syvi68&D2#` zIoXfWHj~$@Ep7ip`K1p?o-g660R;PPVc{lr_UBw}x^L|Lbu+kmD3euh>@5i{hp1}E zFkiM3T}U=1HNfL8|0VCmvgZBgR(_+ov~JJb0fj5IIdN%eX=1_hV>~G_k$L0BavUFw z?tfyluyI-Xbh_}-Vu#gfc@yHMVp#O%)hkE{ZIvrz{{w?C$Zmar+(|y@JOzyNm{F|$ zmHE}0dJ`%rKmo%=^u}c4xS$3P_YPnZ#0&||ETq|^U8s?aisoW7kDA#uJo#n7x}Gaj zT2V+t&b22r!qti)L@T_x%R1*gt#(?X%Jgn+!L?hCL+HK|amjuD&r^wR2>#>MC3+&S z&Hcra;s2b}>jUoskfVO|^u+TU>89(h+qUh=ZubpH#y^l2QD1AN%Ur?S!%C(U`z4SZ z_yi9kxi`*kZEnL+N1uK16q~WKT0yb+@+X_ApAKc26+GRlcU(d*R zKWfET#&yiF;O9dSYMj7H7+kSOF3uU1Qt4EuH;V^43(5`$_8uEEHe=XfO(er_-lWFR z4W7TEEu96s`1l+U958EBf@C%Vw<%PcA-Ty{Ir!w^!=@Z-tv&k)ST`I?P(Q?wKurdC z3t^y$+AI|I6q_s|xHX-gj}kY@<0&|9!>-{aUCva{*!zHj-|)NAE+y94$> z=C3idY8TIm_nb$454$+ii{4x_+|g_qh7u9sgc`a7&C8cCGO$*T<^pO=1eeiAAAOzC z+@kplIprDJ6Bu`CWg1GsgyWUP-=9B!!a7n6oq<8>U(B42IDG|iBM3(h;nwVyzY_w;XimB_?6lQ$6AUaaO)$+{FGSyg>JLBXNN$KJ@~&H7hE zuYZ;zcSd74@#=E7j2+PvCP5G2WHH|eGqYucW-0<7SPg9~k5nCpihah;kCD^~ z0}voVFdY%^ate6x0BbW=nEoC*e7J5wa59>hIDd?;MFrE$Cm~~b{yce;7{#{DUYQ&$ zqb*LcdmWos1;JydTX)rW!EFPgUm(CXzcf3}VHzU9ziSuF9jlS<(5SmzpmGHAn$!0a z6I!zH*JNe8Vj)+vP;ZGaL%a?WaJ!o`)oZSf4iZb(mfz_qx|l@R$rfRLyVv=q zmTpWInWlH<*_i{=MV(;vPz)k*YB)krCVDyh7i;Uco;=xxd_nA@8f6xD|6)7DN3)X7 z+uC`#^8JH?w4biy7na69?&vBlbai2k#SpMM@RwhWFrEW$12*hyFsdO-3M+eaH2(~z zh`EZM)gR95fB(Qz%)iFeq!NQFfWE(f|3;@@_W85KAlvR;3Q;~@!z|8YOdleLS%RSM zb?3!b*Uj>Gte_hy6s(gsQgSs{u9br7;TMCro_Zt$Y+jOkWrQ#Khx^ls407`4D z@#r*@p}!n6!{{~pLNhs8d02gK;<8>^xaO{11ungBx{ZI&goZ`jyRRk8e=<+V?BT~a zy)?HgT@qq8+JT~VAnLTj#W%OH1RSR4h*IegvlApr>_6HU4fig@v*01ShYAIn}Zxfc`^<)Yc=0cx8T#PBq5G8(%wp*s5_>cc7`Fze>c ziNs8r*xi0hM(j<%z7DedGp4sSoAB{7P=!DH?lslG^oHKa$*F&AEE9Z`7bv;YBtPtc zFbvuvZDgys{G)6lx~dO2j~W=|JDyx)&?kgEqys*vHu{tXhfYB5&qqEb+-6{4&;&5; z@s|@JFNypz9G-~ORO?%dJ1{iX#h%b>*rt_IwPN4v)n&>ZJH%xqC#K^2)Ti2X&VfkXrE)il!n@3 zxZ1a?D7Ce4d?e58)Tty+_=S@Mmy~!ujJfx3vRcc`2~p$i>v#A{ey{l~TZ(XpUAOeE znaeoQ=qg~KqgXHqJa{vx9iR-7>l?Jv%t0(9@deEB)G-4VGU9MU21k!e#*6r^{Fqe} z0SA~xBmcj?Mk)K328{7}D+PrDE#DaFn+fI%(p=&TRjO8dK2c2U42KI9Uk0>3$k-iF`<66Vuexjs2GT48WaAiIM60vKI&nw5=81qh z%E?+39c~bJcpNwQA&O9^|Dc-Jf)+W4SBPy_v1b$!K*w;#xidE+f)&S?IJ3a;yDGk< z6CH>zbZx*`5rNR2>YNiC$Kd-yM^9f4UD97{7YEgiT$0)?Eoh28hCg0r72*|nS>u8f z0N?s`uzy{li>1ByK&P&v_uaUg*LPVht*#ioAkNyW ze2%v{#8>je99CDhp9{cru&wm^Dw9Y)?75zUUl$RoBAf`9Z;{KF$C*m0fjnmrmET`U zHLQVLvY9w{Eg6H*`>p%>RjU0`JARSo_wV29nw$4`d1e^qzryNN%nK&o_B;3QYoU$I z-!Cb7`b|mHZhcKPI8+*yUh6XL?gl{6hVXzGTpDBMA4S3oK8XIt6PcmW(J#o2#8kaG zV~$@roOs56n7H@pktO0sT}z92rTKaQV(6%TyfC4YeA6zw*UT!Vb)ezZL^{UG_4GY;CN=nMPm$N!=eF#4@(tDCa z5E7|hlIEQo?$mn|SUMA_`PFORBuZ5qtS=0QdckQ$2Lupw zoZ6Y|77=S&vGkDg9lBu7-D|Ogt_N~cq>C1NB39gl@dBJPOh|ko(J>J8E0OLw^ro#K zc);f_X@AjK4-i@PrTog3RaLd|vGnrt@`gW;1Gh{EP?dfvUvNTJN=l!WmR88hYS4MS zy{=Aq9hZzD9{*`wCbK`WF7~{V_U)-)OBmprrP!XdzHlKKJBV>EqnR5xz49fTIJz#v z#jN$_+JBE1QP$vo4PF=f#|O-^E6+RCA3TUySa#)5#+g7ipualrvgY5ZQe*kBhzQLvfnQ_an0Y1d1?i<- z^X^*sUX-tS_wLt8E4Tx;f-?{?CZ3J@WCR)9eb*9nI5j{kP;)hV_wAb@PU9zTk-57? zLj3$Emw`a0K`q7lvn6Ip2TjX-UKA=mw?ev01UmlQC|D`xr4%F`S-H6Oirg0yM7+?! z#6@||8%f!mb?es0zq(g}5=7j-eX3y21T{+kuU}7Wg_Z9W@k>bP=I7@poy)?v$s!X1 ztIxm4txGx0Wr;Q-vtk4S?qq%Gnh+-M=9bcsd2=xvbf z=h3&isiBsm@6b;ZI$T7i<1w(8+puM-?d4TOGBg7tndBG(h=fu?C|0)U;DpYfc_845uzFA#&0v#Hp}9|z0*wajwmYe{7KKLl@ z-l`><#*rA;?@F6?o;!Cg7g@Xg`b6wela(UB^mNU^#t&t849C&TreQk8paF$BvFOBP z9}1)UiW(z@;}+chZnrSwOa(F{D2Hq{-1xV&r{{X!kGN~0|DNT}f6vl!J(q4>g}mPW zL*nXZg^Z*lVHG5Z#`1u`yX`x6v>|lz4_vPvf~2b+l7ZqT;?kXp_*J^XH9uCUr(!cX ztv#~f!oeLPxyK-}bN%k!;dKF{ViADSgq@tG4s=4<(zAf?^iyq3%}_L-nztr!0HKgR zra9E3y%(<{!JwjCSfB z#jk2Jp01!xs;@U*iuTLM&@3t0&(3@Lko#n!2lOE~#}j1wQ1!+q63b5JU%q7T50*bt ziR;n^(S>v)wogy&j_lVIvbF7!IAZ4eFRpGkPYlh^nVx1S3_kSW4s z1eXiSFaG$~wo_;nNGD9x)36xXyPW0b4gT=_;-c*o(h??;M+i44{oSt#b3qf%Ok?kf zZudpIhtg~YIl;#^x_73;RW!=}l36=9=HHxT@~a==L>^5qC>U)6`UVF3K`}$^-!DEP zMQn!9#~s2|lYmb_AhET9ZajT(&A*))P)wjarTPR0OHzd007ZU&xkiVTC3oc6lWuNr zC9<$T2gS%AeSLhCEK()c(3(bKyV-gQxX!=r-KM8EX#}a!_WeZY>%-;mD8gxeMJDB+ zUkFjMc$fi-VA_5t82grXu_-A6#BlBXdj+a#Cg>ue2D^=;X+4F%tOPMFRI39YioXcN zfeW{H*6{S{u56s?{dBzT_X7iom^lQ)K#gDnnh0N)rd8UYJ46NS$8q)}>=!k--m_5gRH-cf7I$z0mm3z6 zUEYpa#3T&WSE7ic|Vvtyi5jHE(;*KI9f@ISGf-L;_4G2HtSPDRn zNZ~)oiv3eeFHy7$DXFcNW2+umT*}uHTrWj<&_KJg`+jLtTYB)w*`Y_@uWggC^NkSU zPgUxphiYLJcr-|=L}CzlsyVxoT1ZIGNScZbS*flMF5leCvB-lKglMTF484TX-}0Ba zw_f^h)>Xed_Um(j#d#Ecyla(6|JBdIQGU7f*j<)_Fs0WQilVmKn2O1EyDQupg zRq(a+x-4w>g4tKsV(_rC@GI*8+ey6)gOg-wY+-*y6w4+T*0;td8Xmv2jj}JA8jL!| zOa=Msr6p&b#;c>gl+?zNQA8d925A5q1SmbQ6T^BQ16ho*xPTy0h?@x zzg_n3q_60t9~OLi>7smkV z(_|F+K~K7#VQ%b3p|s&!ikV-)jSI%j{(Dm*hk2^fJ6|4qbJNy7vygF+_8cAj6MR-7 zG1CzfOa!`QGmnE zO$-4s>}<&`GExL@JS;q%W@zk|@#_ogaH1PmlKL#V%4@rIUhi*Bi?H|l5D~`BIANlj z^*J{|{9v2TYbWQWMw7eXS7U?l+v+0CHJrb{&>yhw%*&_}-$bMwF{;W2f!OXlI7d7> zJSQTzB1i&#$dE8X&ku}JNT53@YZRpf6snJ^bs_*7fa54w3%eJ=r1|-uLHaLEZ(M4B z^c&Ji&cDBeU^C4w7N{X4gkGuL-%Z4Aag3XD?$6B5UI3}X;GFry2-eRqRqR1)(NRRU z$C1|quABWns~E94Gh|!F2~L-v@7JM(k-GYu9cZ@&>b1@L{HlCG+-s)EG`tbH$E;y% z@VGP2ewff7M@9H{)519R(XLct*Zli?@ww2)`^(`z)-v5)a(z_>?d1iWVLW42k=U44?L@lzxUBR# zezr?~L{D}^{jI<*tTa1}eRGN{f%ZGm!7c@mLxK*eM|1w`ylR8*8Mz4o2@#BkzEVIBGyC3`sn zj78x`>~L%$i*z?qg^e;zN@MZrLV|<2DOvOfx|P)6cTE>?yFP=qI-d;no~tAb`rl1hO@h+d}8 zYjK)2%9>%!jby{gj>`>cVj-lJM$GxSXlknIv9oW~L*e2hKg1e_4XU6hF*+XON2x+^ z0K31XJktnr7)6Wr1u2AWj#q5kf_ruzRj~nYpqn>Tmf{g65m&C@R1xLTxw#b73d5dG zBD7rvaH|?NscF%QYT~my=;!7Rv(uGjW>(rCKbl?(JbaYo*@4Rdt8rQg%2EP0jwIqv zx*5Xz>Mby4`ycIujIIV8N!Y`M%^u^`f}Sf;aJFp$afOD4#&m|OZTGtmZ~_D%zXhWn zV%$lAIZ7@564`y-<@iI(f+cA)G2Mo8VDkX!BgxjWfwmy*Qq|5KF)j1yL-c;Ls7y> z%NH@Z_6kc&bVH>A;p#@z%B1CLZPA}eP6-Jp? zP?+Y&oHaIU2U{z>jTz>rx?*AR`S5QD8ZNoO7&_`6Lg z+%XO(X3svNN?)t1Z)36}?{#GJp0q_SFnwGi%GW(T{DAop0)ZzP-J?l@MB?~M(8HCJ z4wt6AvH#Y@e-M*{!XF2BtA?_mKf|;D*e1w5hq=6`H-etFkZr0GB|kS-FctM7>X*G2 z42*cD14x+Pe_ONX$5RnNWmMZFeY5uwZFgkX=Ge~|yZ=|d>HkKXk6Do;*2yVg+x?}$ zMFiDS)6u8S)+oB>zEshmJ9!_pHh9~kRk{fKf;WYB)wrX9kAlgBAIO2vV|r3(x36ZWN1(& zKHAc&gMoo0)>n0(jKQG=r#U1s|B{&3yaSmgsLi1TfHnC^s-4E?l$9BfqhFYmu8dcj zZUXp7R>f!pwxKR!2GX9<@eH=eI7Su2H5>oT-ohTw#QET5$B^Xjs(q%9(({bhowMKCPg3QG;92z|P$tn&&04Y>L(BDHPG;)e+b)cbOHtTe3SpWxC z&6pyY<>`v~2tZcv1S7%clzLee8)RQ5%hM850enUZc+&HtvhZp)IeHES0t-$F&}ynO z}NRGIq0>_E?;g$5lQKA^(u&*XG0&q`U+XlhX-vV$0*us z?qHw-=}i6`8?!9PY&p;@(MDr-(RJGt_+sK{2IvFa_Tc&QgTB>PB0}WyFhJ%P`#NNO zKLbNoDveiX_lY8pcI0Jb&OFG?rBUkpYOW1BE?CkPV7@A01q9_%`zmdz>uFb~U$IAF zvhH^qa$_|<3?V-T95>*KqX6HFQ*#Ox(C39e#n8DwB!#FdcHS>@M+_e4)0BVIfqbmC zOW3{be8qoR;RH}`Zwj&-x_*6RV)CV{@J}RKz{nyL4%nk}Qf5$CoGnKy*jHO{4=>}z z#7w)G{r1hG^{#fE`)xk)$_dl<=is=K9n+7oM;YaiCBGDVN}@In?YR?W5TvPRnvTU+ zbVXkjJN+-EgjZm*4wDROm{Jqr1RxWEc*x;^B0m8dI8ZHy92;iyTI!|R9E_zR=3Cy?PZ(f~K~%Po@~J z4lo@$RE9DzS7d3WFcZ=C|3ALrvCCiN>@HtMnxG`COJW55Je>C<01CSbs1kyxxs~F+ zO4LBio^$Z-;5#I5aWJXp<_4&Gs#C1VBdGyQo@!Dm1EhRS zJlLOvErV4HB?m`25QBhpSYZZ0u~|sd59e%hw)3)zLzRk@+aGRtKBH}>^TFxw9UtGj z=p{9TS_%H&cjo8aC~(PqF!xb#+^5r5K3;%wg=PEp0J!*qHo7n1=Uq5$8%e`{y)OI{ z|3MCUf54$Wn)jeU6AT$RjaL>*;K?ar2iNghx&1dEg$F&Y0JCamewBYST%$_I<7XU} z;Kp+zSy@Tx$4%to(8rVT0MyV3<~!IQPkQMqy(cI#DOo|dVCNLQv5{Aro#To}6hM~@x>VPsHVHd^LEQ;%K0+uN0xgVf;xp({RjNf)XL97q7O>hzv`Z*#(u?eryS%`5{ zV7f#k(lHlMHg)_0*a9lJz2CZ900@8&yNQLJ=hayRMDLyS`+^LW&MShK6^s_`K7 z9Eh4|KH-EQG)7aQWp@TUUR!M-VJMkcLt2ZZ3V3;RSIp{0`y zzoQbA%e{_F32T+%5RCo>eUMM)limclXK%8vcwG?vF>ag%bp) zvKG{MWRC$Qc>j_-|1SC4>(RT}?##csr>}AEO@2S-3T{R8z9*7NJAoO{E*>7rCL``x zF*9Eph129*dwP13nt`aJ09ptx)bX#82P|%PcNX3Byo@_5cRC+7}x(}>u5Y@_PaGATujHaKL=obhVLJgyF&nC zuU)G4UMPb{X3?SFUnp5c>IlLn#_sq|fi6S{BshGe(#Tfr(rez}#*5&4!{BeXNy&B$ zbScnxyuk0&Q5llT8{)#$1x8fI$)bGncY9J<`ZUBF~e zA`7r}PdA}Oc1LkY0pHooNM@d@(xb(1=MSW&q9zXDk~OAEutNWT3KapRQN(@Z<|&wU zAte$q$xk47?n$k!tUMn2cslseBMuZW74`K@K8Q48;CRyq)U++`03Q=%zqKMFBDksR z1X71vPP-_n>FOdUg|_lliIay5#uAkLDzGFoy)%{Gbl$;8Gc1CYhE@p;k38MYf5SG_ zeclVs=%OfyTyE*o>wnTTmu##{ei*C=J~59p z97gDx;x6qwPlIWBNZFkOD8>?XD5;3>5JXh+XcRUqdzv@{2giLqOSQ8ZjWi_&rg94> z^q!g!Wh0z^3Lx8j;6&xpoIIj6^3!nJwrx;B7diGx3&o*g%T^Ceav?Erh6?7zQoiV9 ziG>chAfOWLQG?=LfIrths)3KRDk?^uX<*&Rp>RcILgEu?8vu5pTxpAnv3_`%wZ6Wr zn``TVf5bsE2hue3<}Md{SCTJ0K$YFJ7iXWarSWB{3AGH>1~03CU(E@)Xgot{j&E-b z78<5GA!S3^pfzYfadbVG43iwnZd~}VGCDa|o{I0^W09ps3&k;>1Vb4_;9KN>?fk z>}nD%gQxQTd}35S$ZyDG705#r*2k>qz}PACp`^}PP6y#mBVBh7eSqRmL^lut?jwm( zpz@qa(e$m+z^VwLAVXW=6_==E1`66gKs?Ot?bX4PMe}zny9Xrks=KE4nJTv&T%!ok zNi?Z&^@$RYyF0OUT0Ue2?EvTDXXz!gzp|RrqyANxH~Fl30V3EzFTg+(Rv!nky#0{o zWYE##ZOhjAbEq6hEsy*Y27Qeb>vw-4uZ;6VDJCgNr0oqtQ{XfmNg;f~!Qo|Os|6|= zOsDu$YbjJQ+)qyfbhYOiNY3>4n#es+vGIV&M$&^kN@Dqrme6PH!h=>FV?Q?=a3t^n z@YwnAkB?;J%sY20&sLta7x$RHi0jeV*ys@Y_(DB8mPZfELix#|cluV~!Uz~1UlpW1 z;9q*OA2}{{F&y=jS7z{9Ed~IN0C3S!!kBKt0P6c{2;L+nEujc7s0e5>A?Hnt0nlDH z=h7UqReEU~ato7< z&HdbAOsO0I3(*H~A3ppT?YDIcPS3lF3VEu8L3jg_B?v?k3cwMVOKhPJXd#%|l5h4&=)E@>bb`JJ5%U`%`i${;{3(rdwu6vK_Q7ptXZIE_$r?Cx+J8 z<=adFcy*9bisPaNv|DnPTfmPIBbf0$({FFD7-*K>zt0G=BFkp#3ACLgC)#6XyPhqoy^LgEH;qwE~z zjk2(30$KRTaTJOF7s*FtL9KQEd|AV~v2UOFcdg7P{CC0yNDrl#2Z0gjw=(%4!r}*q z#j^(^lO6H**Q5hrf{K0OKI%O zOUT$5I{9Qi&e8sUpnmxeSOqoL-qx^QXpC*ZTydX-#D!1KPU7jj6Wj8waf8f)Vtby= zh&H>J+#0PrgX{qJ@usn<6pqYDXmHUc*i9{aM=NSXb?VgFqO*G5YA7(p9475{`MFu9 zp58x$=9WwpG?)S)^Ln{xVhshEmQ+`t!uu9c2JYH2)=Nxs{*^D(5Q91d?1B>U=JvNe zvA36jPZ-$N@}9hxX#G=|=@hE#;4Oau+Kt^T$+W*T1pp1tIM@In< z(<#)8?7<`QzfdhD*@P)NCFsDQ3OYccH1Fi9C#)BV3GC>F!#qs2DAOv?QNo$G7rZES zr=J1Fj7)x_M%+>yC5!82Rt^7sd4NJtXDe@R4h4panrVDsN&a2+@yFAlZ?v}`CqyICg# z{~6p12qbLxUYP!rd_6q0%l$Lf({rkk)e@iugYz6wp!OVIEug7I#P=NoCzXuG*K5lb?<>E6)`F6h+woE+ukyG2)C+a|H^_}S6;{1NJH1M{kr^iiK?0v)Bm`hlOu!f=l$5;u7-n4o7nhl?I zE(#lHN4*eSJWeJK9RElyIj9?1n3?0%RCHJ>QCyZBFc)N_1;~ntvKeT@6zm)vNvX3t znY4hX69*IuqJ^JI^n;X6!m72WAhsgqGB7!}-vMyof~1VIc+f zWGqM9)Na@&aVfa(_7P!ta6pm5wk7n=rFl?&FMfT+PXSGfC5!-BHb0$%Tpkga$4-&g z;^-T)28tX_hLH>#xM0yHW`2t)nYl3=AUi^*qO$UNL`>_zeNL@dld)QV)}Z3?!T*1d zcaeh}HSp_~3hDp~49>JO40L?Y_mHDHx!iSG8q-Y7a1=HC+~|qyijWY!59(5LD=UFx zwX~Wh(@;ao)}@#0y9RiN5cVrwj-*H)KZGdg0K7cSy*IFEKjV4eGY z5LYAi!z|e`*Q2$!cUYAHNnJ+fk89g`o)aF#6w%G?4-D&xr4IgX78m2dPim9v>vrve zr}6)O>D~WsZ&c)hy;Ocvlr&K#2Jmg$Qo6cQh!f0~7_K~(FX(!AkkhUsr~Li(ezq|CDH5LiTH$cH`9pvrp%}+12 zLAh&v0TiqaShK!?B;yP5C!)_c&@ndz69x3d6mPMRg91J*!e}T7I6sjMCFLd@&5%yS z188O4Sq$?0I^W>W=lj{jB2`YxA0aC(4p*-HK=J>!uuvTH@#V8Gb8UZEcYe@+?QF;I zk3eOpJge)^yqDm(U_mgE{_4l@u+XtuekhEj9yNYtao2D&b1yAmjv%xq)TE~D_@Lxx z2u>TX8jBz5FMMn86Z~i2;tR0X#(c=fM%U4tYB;Y)F2@|gwDDL*hl+<~ZXa5XcgiM)jyK}9YN|bMED${f3Mh6T;FkB#L zCy`@-a>>JyN{QE=lCbpT<1hE42aW!}3F5(j_c^>(APjmDzPZ2w`J)uQPxaxhVb)$s zUj0juI!3P`2cZOwitpo1-P8sRTRv|0G63RVirimIbYWwKURksm^*rRhDM{o@+F``G z5w!B$eNZMKvg<72hP);vUYPFft6bh6vPipjZCG?P6U0#-2x2WI(=!OSnSe&fn@!*o z0O%kk2at$-8L2mfdm>ZK&!ImQhWv$LI9lE#1Tx2-v{(RbsWZKyVNV^a!_`}5ay3z> z*J$6d!dh@5N=dR}^X2Q;joY`s;A%e}$km=33RHPv=4Q6J6rumLcI~RXgf4=!`<>-aSr-ajy^UtQ& zT!jP_S~@gmtE(_hVHL;%K$=O0wDno4!E^m-24)jrpVg+w^N&VQ(%Ee z1_5K;rNmA7>{|yOL?DN|{lplA>SOc`gIv?D!}E)4II$x1fAOva0F+Xt8vlpO(5wuu z)`{;@3gOHcH)@ZB-9lsJp4rt52#s`Ox+9!DSP8=hAVtskcq8O#9u8ngm|qDZ$GU-| zdH9_mUm#h5YQrlks$O_plP=^ta&Hcw59k^u9Zmz9sKX6%gV4=E^o{#iEfv6bb97u_ ze=uO$hNd+1=b|)u&?YT8RCw?H$&x1^TCk^<%s=Q9IxvGj=a=YG}53CKLSbsg8{ zQ7>CkZO}poSdEOoyJS?83<(sgy@EI zITtsgf~JW_3E1(@9taYMb=7DM>@>!qps@ryQ8piJd*CJP5-CBVNAodyh49+A^TT2H zHhWfv?X@~ELu^MMBC7?!jyOS-CEy&vy$x*NzPLIypa%SPjT~HN(8-&{-d)9|{+eUG z34>Q&%*=yi7VdyhL&15(tqesU)R9OTiVkx>mQSz(z%*IB1py4$88udUiLaFfbv-YH z)3)ILCn5hsjp7#O9Lf1g+|pQ;8XPu%Sj0PX?d-@tHvI%l_(||L2kaNV=UgQEG%?I9 zG1^j6Rjr?0{?oc>!|OVJJ0&quQR74)fpmaZ^&v@;-PRz%huZ=y#npM|FD}Z6}Fm@fQ~ z{Q@=FE;h2g58&5**DUS^ksIxB#&iKXqm<5f8}UM(fho{JKRr>^xZ}_p;we3Gd4j% zIu#Wa1mh~);UJ8lku?bxu^ausmoHy-FD|Ca7Vg---9pY$2wys5q#LjDm9wnI9V;pC zg#&~gR_qZ05+7R8$pex{M>785?9e5ui)_=e@%Xt8bI91A*8Pn*r}#pV$@i`VE-X2n z&1GQN(u%C~KP(gXz<+H8(iamc51m9qZr-(AN2L493c!s997~H4)ECHx&w7V zVdDez6eKi4R&_MGEbKrY zGc`5M>v|MtE?xiry@3mx4{nHxgwJm6G;JZ=sK{j2E*uCl8NiFJ0G|{&nrK{r#`6v} z!2Z$8DeAgjU3LpdhM0?n;-iJ`HIV^far*EW>iy5NGqDC>_((Lqe}4;a#q83hdTi@eCgvO($RM}vI$Tz2%d z!0B6$#LK^JhXMBEga0L2rDQ@Vy|3tfEFO=;K(jF&?LE9E4{Js4fiN1-6{v!LRrM@I z-TdPL)7W{B$6s1u?nR_+G48@ru^AX+0Qw){I8oh-`4Ya6_yey@L%Dd?jWLAydZ-s( zykKjqg3g}l!dGmr5+`Kjqo+7$S<`>>(Y0aaXI&XUW6+B=Gdag`r>x^92>TP!_SHs< zr%=LXhN&CTS#_VZD2R(Y>KRGPxaB1lx`Fc&EU&GN75n|d+Ry*gU6*uub-tnfkUq@n z^l+XdFMJ38=OQc^zQ11rK;5xojLH8OFfp;z`7gT19Mw{rO_m0 zs{5xUi2mGpJUMg}n&<`$3q`e`Yfzg1tS_zph?*D+aFD|H*Sv?y@Dn1{*ZEgBpuU4u z-YQH)i9E=B*H(xelDnLndyptZ02*~3YC4wgw)Bcq_F6rrz!YNp_9=z=h&m2ln@2v$ zok>NR#IiZWuDgL;=eG`&n3kf}{f~2eTD6t`pxqaWyu1Qswf+wud>AIS1MP*3!Ii;} zJJ43n1GbqlFOL-U&nlrZH=kd2b-4Q9f}zzvf~!BYX-;?=if?| zZ`8aSL>*fJFRZJd5aaOz0g% z8Rs$M$*tNoXz0kB3#BuB5D;f8pvGW4?4(f?P2vGCl|lQK)9F$fkm2+}66*Q!gS=!U z1MlIyz3774IT0d_vchISPOC*Kd?aCKkZHUD0YngQFS6(=9##%b8wUMK=d~nkK#7JL z!}SY}En1)%@gb$7N(EF>fh{;S3`dDb2{se=E=hS|A%4^|8^z;Uef?VQzVoD*T6PE- zMSTjj-(IXx6FUzatD@E`Cj*t#Z^a(vImCXC^=8A9^0JS5bRk{H_x?EII3YN~y;w%# zzdOIzZtB8N*S2uq;g<3it$&+_vH#Epw$O#XGZzfYBo7{Z_x(F(PwrwfTIP#Ei0L`^ z?`GB2)Z78ay$_`G18lP30pxqk2!o}DDWbWh)nN&jnPtM5@o^da=zu4_MO`GcyI!+cEk=JU?@T3qPMfJsc0E4vW4JUsNxMGXe3=q z+drc$ga|GO#o5oC^`Ad)N!pAK&DwY`x?5cDW=!u6*ePiOFfWJj9Ihc@;o)A(Ke|168|Am!Rve>N;@<3QbM?qBvgnNv?%x1hzvSp`pRKTf94!*?~+_z^mIi~f;Htf)aBr$LaC*!t!*a1 zEt<(%*089g)qM9|HC}1M(rYmw6*xeClIZ+l;RW+|pOw9DHQ_4!(n|{7*6Cj?$|Q{$ zLU!QugG;+dKs`Dhe;2&Mp;Lvm-iu&6KmdN@PSYCZe**>g*nf4w9hta8zhT#e9R$K8 zRI%BLK|5qe-3G?Lm3TivZGj_-MP)jG2}Hja4OqBlG~zwxWm_dKJzxIVa(^9w&62C9 zhJX>{+^p$A?2{XN+k*oEJ4Kp!=_}70Go1WtgpewmIHTHsz8WJR4uB-4Hb3C~2s!~O zsy3Em*z~6~I=7a%!$3U(t;k#FVHTiRRHm3#6I*TU@?rM-0jtD0-CD}l8NMrxaPvBO zW$DU%af0H;mwE^C&YqXFxpas2nD*}HU(Vm%`ry-&YQUbYH}7VPY<=+T7W=t7>$aIz zJ{O$b>Ur|y=99;g6s5*y=c@Z9T+jR%;-8XfJlgoZYwY*8`i76cKU&F{V4^^NBknm$ z%4$rpxc3T25O2;`P_V>M`g={fW@=Ssk`I7U%^SjPnJA1hF*yRL7>iD$mk0j9u`{1Gr|$bd}(D%Z%8dicwVD@1}v)a$)g5NLCVlawu&yIc%{6*=@enzF9Yz59Y-X0q2=DN_p62Z?*M?*y!zxgCL zGhF}==C0pA-rqEjGK%z_lQg|(lLe(W6g>brfM_|R3p#}^69_1QNSLtm`rQi1@S5r& zKQ+KzQL?t!l!u1NnnhL`v#^y)v*p|Gsc!UeL9e8L6hnl8uU?>#sBYOvmW2sD0t5M) z4Ym+QxsQ~|V5cEJW@{#^M zAPd|?RwTVLCg&e88+KcmPA!Vtr!FRSBR^jXvFCkN6^+0A4U7+Q{(JF(p=DA$!wM-J zpn@poNwz_{XaZCguAFp4ePNDW0E8cc%QWHhQebCLl<^7f3nhBbr@P&D(+#!1zP8Lb zdFsh$DJhqsrW^RUKcI;S4hi%ILW+V`t{S^=7Xu^(S7@y%27U6a;q=qvT2Md%zNvk#Ga|Jtf4 zb<+`VhS3hP#lyKiWe+(p$JBYZKcS(zai(vE-V1FLmLB*D=>#`3prEJ*!X$+B^U3-J zrV~VL9aj`Hc>N-`eV8fm!E6>7xSX_9mS$?-T3dAUVFEkJhDr>T>D2CBe`u-9V-GI1 zNbLXlJG2X%1<=&tB4st75*i(1!=kE|TUiDT4M)fd|LsVXR_M;Q2QfGr()1{kC>-#| z`1?ZF!R71TMi*0GU;pr^XWlUxtk_%5MWLSfIXb%M^hIJFLr&84uvV+hK>+Fm_`vL$ z`c%xfDg*^BEiD8VAT%Gu|1MI4Pc}R$j20eBW@;m zYi>m=7~aixVC|`8kJ*`-gATo$`j`Ns@<6?Q{f+1|_`UdnQs~~f8L!?rGD!3MCuGFr zLizq%Gc20dvY?82!gn!7P<-2@VX!M~Km;IfQXCp0_>m`d+bz+ z(8yOt_`kMu8RH;D5FQ*282sglFo9P8MRW5oo4KOugwD1BD9%-KKR!99glww20XZS53osPY8j|HLNMvn|Eo7d=Qu-T0FousLR3r74)fZY?i+2!Am_gvDB;RTxh=!pA% zc!y>GPEI_H*Zu|cMR>DZ`}qaEIQW)KW%z$&dGqh+n{K9A7z0w0U5m~)r;S5OFe`ys zd8}1kHUtTgov0ij8-><|Z>#)zqmPXqGj9_hcG9tL}f4Km) z&}Ug^Xb(kAX+X1gz%czCps&&7YP^V_0g5%tx;?k=1_fCG5Po#^l1JxE0U9kJ&1cu^ zN3g}HV)!d0{RxP%P<7*Fb93r4KzDGN{4lpFfSwb59^L>x7b0FCbF^=?CFa_=L&w)! zKGrAc0MHzM2NBmM)U~S@U+3kC{^=J|(v18a76)F2*Dvpfa>4q>2rKCD4?*z~>Sla` zf`Y4ue3zHR1GcSI=6_+t$jSLkp4RsO9TSrU-Vg<-GLVX%D{A^wd1vFx%Ut2R={;xV zu3ZV(L=G1B+Km#jrL;8A2A88DQDw*`bYe}}tS~x-(3b3caV{xAu zVGZTm2jT?*AOULys8@gbv=N;4zuq_*T3w|;b6!A!{M3j!5)U!EBqU|9H2z}ZtdHT} zZh43l@Zf_Rg~cokwL9q^4cJ1*Fq!-5NYJ_CSpCcfE(7-EJD?uj1&x5aKtkOK{PHPa zs>^`D8WFYuw@SYA&Sjqi_HEX|w564I(#=3p-Ni`%Jf|>%IAej)W5B=lQ@2;XY zscUt|7!?&sA5PGkxMH^nUled=mX;A?2UNlZVL!1K_1URVK0YOY&o}WR&CnP)A12EL zfV+=B)gZh*0^&%8Cn_gB!{J%8hzT_@g(XdXMuuHpy9;E5Q4|fJMkB*9=#~7l>%@5B zIAA?O9^I}kEYgEfsw^njys}`03gA9oX+vq3+97D z5z6R(?ttB~zT;H*=QIJOe zBnFZAWxc=y(Q!LrFtha9&9Y%DyY!aL3};KWuVG?_LtJ9{a7zmQy8&%7s?cOx9_PIM z)}eW7=k&447R)$D`!BWit@V4U#Lb+jboJADK$B$=k)#P@Q#)p}_Y8!EH z_#RstIMds_B!E#uDWe|0<q&_(3t6jIQBnNeFk0~s*U zy&eFlV`AoB_@qN8nsB9LrzR(PA=}ckbzp1^0<0#bCocf^slM7xqz?dJ z;W@Ba01HVN*>lw0%xr8(TP@{N!cHP#fD+48;A{;3zUYR?q8br*-o24Oxh>PBA{n3< z&!sxE{X1=%T`e7*oDwi4AOEIT+IUT`)Hqnt_G#O7%4OTPL$7J|kKq8~BtRnjC?<#m zNYw(ivlQ|-h77mnZ(&)&Da8?1r{O%H3^Ex@z&cQ*ls1rYrJ7cy15#4KhR%5sEpiz>h6NYsko z%@VicNW#ajZOq45ug;eOg(ND9LoOrXzhwwZ9P%pa7zJ5ICF6;5+))Iv$X@JN8zMJo zOd_RmD0~5Q$UW7l8QTCkiQv+?`A(K@^j+whP$n#Y4%UUL*JZl+NzEZ{@26JAPAh`X zEl#v*@(@`-&EQ6wI95@u>b&%VGfjTNNpHXA_u#aWD%7h zGTp?JAXy$WkTp4GT#&;Ry1iCga5JIWqY}4Rh+fUTJ5eOBa1#7*UHsf^w~6zZ%{C%< zAqDY+5Ab*F@pewmk0=x=hCow7dOtWX189NlP|cODf+=)-dZeB&|LY%}XnF+~cI0 za#AQ+jsc!Yemc4!P&LG%!l|g!Iz7Iq!y~bJIXKEc&3EaxauNyj*#5`9MJ)jSQ~-c0 zHA{gznzFc_w~5(qNn}ws=4QS~+#OI*;TJcfi-pw4l)`(7%eVHHG#$>$FrXC`efA2; zSdLq9RNqIClXA5`Jd_o)tsg902wyVwR6Yl_5$mJV#+{I*}O_Acn1ecYguuSi63E zSx3iX^d9#(OLJ z5Jnpe+@1_n>8kVksY!i6Lk*xD6c+K+u@ntKLf*uv1MCihnlefGS|t55iTjKAf)PYW z<(qd9-;a)7Qg~SH++FtQF>yqJ5Y*1Xj0YGQBV-T8(LZ`4X7zD9HVK~VWpSdS*$HP% zeCAr0C9JUgA$_HW;7fF2%nsF6Rm9^H;Lu+bh!3{(-j|4(O4Mt$2oMR&p4;c1ngw8} zWOsaP{TzWz~Ba$2% zm=0pLDyWmT8F9%RdFQ#dFvf9Fqwb&B+AJ*DU5<_5qQhRZ@j%x#kgD(E(~-yU_N~R2 zh^f+^w{HL;++19q?im}475TB}-66wZE*u4-MS%oJ+(T!_x}KzaUo4|*DR6Aya-n6V#F{8c1znUjfwS6Bs|B#rL@=Kre0 zg-zZZ>r`gZ?2F?LGY!ItSG;RWHt=Hn^Scn{s!ew}di3a2#>-SZ61=VbXve`{4dJ_s z2=IZ257ciJG;r9zf4{AI7e7Bi1SoF)5g(T?jeY0mBlqX(^t(tPUhWccU}>IsA}MeyW^pmL<8kso*NsH=3V2jWBXo z@mbGK;c?(|Mx~V*vk3zs3dAEMvpdxoae+z^+XGvPfimhSoS0VWx%&YD7Wn;>Z;KG3 z)^uYcKnB3!%KWGS6YzwIq5)cin$lW?LSX4N)dxQG2eX!!mjl1qicvk1f(j1ax)afW zx*-e&OD+!!Pvhy6qR)&5nNVdXA<|#AGe;lH$f~T zDw-(bHDF?o0wFsu@8fcSQU2s<&y^*ohhoRHBMFp2qrmN_h9*Vr)%nR1jKm&2c3wg! zw%^$hq+CGow}0<$#+X<&sv5Tnwl~g^5tlLi$Cg_PY7^_vWG}dX4&S98)e!}N%iV3P z7eI|5(>(n0CQM5XOeLB$o}V;Ey+?46H!gPnkFGO;>alItekGwnp;Qz}WJo14Mn#5@ zQkkPtQj#GVqGXCFLuG6tR5DAZq-1KMq6kSSDkL%$4d3tbJntU9^?labd$0Yz)c=3q z*L4oZah%6F@caTVuK~dkue%TIBr3yx`(+W}fd7=K%O;~Zm8m^q8lb;Drm&@nXuNDy z#pGb-XL1(wCzsO3<<*MSYvL45>d?%hqJ7l( zH-F5E-gpd}t-8yg4~}x87zaBOYZlB)=W_sVeDTR~G zedcxczoK<%V%S~47I6Rt*74_0dJp})OL)PsXtlfI*zLDR-oGog>DGmZ2%tk4TLJf; zERfbkGX10I{Sy6QKUG~w`Jzo*bQXPnx9(Eixk9?GMNE~p~#IUV% z&i(~*a`KtGsegCSaj%YL%(Wi?pGpxGL%?I}Dr~76zinIPD{0|aWRE~bK%XP8HqLc8 z_@v6TVv2R7KK=lLB>v(%sJzc(E<0?yg8Nvzg-(h2ahCH{ ztyeiR5RZ`d-7hg*kXlMdy{21&ZK4_jd1R<~K5F&i$lpKC1X!uM4ru9Yyzp{=yE*

?3STk+%#h za37rPRnmDQu4vnqg|szaiI2@}Lw9HAP2-P?h44v-X01u*%(5EiD7O)U^cyJUmeoVt z9voT|bDW#Ob^$yfo(z5Qk&08`4#?f+Uuh}+Uh8ZTnYY2`O(ldC^4UbiGfUp#N)c23StDn#;q-GuiozvV`(+4gLf z+AyUJ4Xv9uH0y$=J_MwmjYJ3y2wC42waUwUW|n-aBv9Z-4!N*Y)AidS8ebCf=BnQIDs4C6c?rQ;5R5D)qz2 zg*z-Ce=^ryY}-d!$o~k_0BHMNmTEsAumK9>IC+=7rnyHBA3DS^ZR1th$OHQG`i(rE z_V8@GWM^^L7A+U7S9WL#cdo?G0APK738K1SZGqY z4JbHYa3OWH1UQfp7$cR*R@PV02#T)rqXscc7!1**11LVndL~}?nCH_o*$j3GoJKTr z_H$5A5He*f!$M+AG~rZ0G{A&}|3AT(7$G5)arj^iccdvBmuPRq-gPj$+qX9Z2I^QE z&GK*3zbl=_0Uj~1_bn>8Z}Xai+3JEC$Y}rg#XSsn^dQ+pNjhis{Moat&ttEfUw*Ha*rIiYlQw zS$w$*4njcb9!fGp5Oudtc!pylJ|!U|#PXerFN?8u21PE@y2)>`qnTXO&5>f8x~BeV zt@n=O+KAY4h<3 z?C(^*wcKe$#1O9t{i|=zj4xZ&WAI>mhb3~uewQzQQ_8EA!wMDQom{s4o2{r!W=Um;BA z&Bu@Z&(D$uFYL!J6H8|h^liexpuGfA8d0}cw}rX7hpcDj#E~VvaUB)FJGQzJk4(E_b1TAFuG!C7;=+$1Fw#WMRkaGKM z2p=6)aMkf4Y8m`Pbj^8>Ja06QnzHU?iP5e(3YB>o->VBwv|Ca#jiuM&V1O=!KXg@(Yow2On)sbmn5oi1vZmzDZq3`z$7vaEMyN7~=7<+60Jz$l zE2A&ETyAf5FL*FH(VGh*}3CgQJ{y7sFSSC+BPSt&k!ksOo9 z+dwGySYJhC{>uERlp(K07@wajN3SnnbWYMea0lRnw^dbHjDE{#&_=DdJ`g6QI~76ag*65ascplg+qZwQ-```abVt`H z-Bm_KvBEILHo4_U&TtIx&oZ*2au6;gGyas@jTm{T$O$^-krl9^7Tv`WYo@ zzdmgmwCL!{MbA>E55Bb|ZTlnDo36#zj^a&+SPeCU;6dxTY-~Q12P8)@uT+q-7i#{; zSq}jj!-B=yCX%tJfrS~`^B+^Ln!Ok~-D_P?9C{S|!%=N!mVh*#ywc7^);pREt zyJ+6mN5>(_k7@<&kPMXSYyi}#w96op(sHN9fhNW%Atx3%JNJ-{-^eC1g)J8&%$($@ zgeqW_Zr#1Rl}h;LrP_xpP1c=QwKyf9`<�oe_6`SRr5mp%arf%)^S*QntK9p0UX& z=jTQjlD9^WjCHg6H(0tVi;6YG37rf&gvX3a8||MlX3S>SrAzfC(HWKK1diz;83x@% z?bI_HM+B)8VqCA(gZDf?F999FK7PmN!SPe0`$>io8JHp0dUAH|Oeq~;!-OVMuLodW z-Fg>C?2%+De*=TQs}){TZm;ln)~;dO)QEuQ)DtsK_p%H-LFeTG(DoRxSZCMxZ}$h1 z-VR7u#@bB8R3KD>bzye3CIGr&9Fuo`~x&^Q^$Lk$+)-kgXS6n2#ukRl) zcLvt%Ng2OjVZ+kXL55C zq)(tWSybEo`JigVrUjN>A>J6B}Oy1vbe@!J*eWCfF zwkN<9E%{V5%KJ@!{rF+^?RV_@%U&f+PYN>tM+9t<=3+VaSH=0CnYq{>LLYpNs{-XsLV;^tkB#qd49}I9Wvk!#{dm0~@6wkgqsU*f~ijGIg zkZIHYRt2sd-#pXdq#q+q*8PM$HroG`K8S`ro&`ukTtz1gF~Wb~#sQ+|Z1uXLV&6F* zUSux3sAP?79u*y^?1JxKX`!bu6VN_D_?$6{Mrg#DL&`EtAPv+o>~z~r=t*MyfKt}& z`*3XG!xEGF&rA8<_4EUV5X=JY^_Hw!Iu+)OX_f^HKLC}zy52%ewwyRo;x+xbz@d&w zww10uO5sJrDJAU77w_CZ(*vz_E_qp!)pokWhSe*~4CIVXPFt$}u|6zX(3$=3A=0Eh z&h%DMY*V~DGGYi(ZKMq_s*o0;3&LP;)kdL=$_>nV;I z(A-5Yj>O{=7uRX0c;rJENKjKi^D@#22lkENi`qx2SH3-7k0T-wUXmD61m zfe)1~boC(c(a<8c2g6sEDCF^LM*elu||Y7`b>-*j?(5wu*|2*iGmWBO2vp znw4(tmaSa)yJSVC1P+OI;caK{_hL?cKc*31>fTL8MoD3k3I1D>SB!8<41Wt{*_^X( z#4|e|zSeq03(F`|}Ch zWapT^M_oOS4Cu2QJjjrqNLjWu@+NsTL#+VU#&?seo^?1kZ%fLm%T0M@OFs#*m=N(H zxP+VHA}Ww=9(G;&K~j=cLN98LY}njwl*Ph!f^XVu`Ik^v6DkhZ)4%wOINI)gvsV+S zSOQ59P*euaF1frt8cT|+*%Ql=)%iqFV8nsPB5HQ_^4f)=C4i)VWiur&!X0)acOsYG z1FaYB0h>qbMk93)X3@w(c5@!y)YK@rcYj5-<>KsY4z>wGzG<3oa#GS;7?}rOQ$9Ys z*pt0`VEcSmnsEA-_tAeo$`${ms#Z};>&Zp3xG2g0v3~kWfC(AJZXsYAeeK4l%}8JS zb<8VW8Z>^XPH0c@l!b#Sv+9;}Ppi6~tS01sDzFONQxObj|K4$|G(jJ!W;stb6zGi^a}> z&RFKaYL1%sR-S4NxT>CXgdaf$=no0`;U2>d`vwMIM=R^vij(!e0ao`hNub<{}==(`P8i8?fU(E zzUcYr3?HsKsNbe3dU-adFOPAVXX@YFcw<~-y7PRpeL}{o_I|y9SkyEc89=fcXV54}D=Uk3l#eBOR%t}Z*fDZ-#_3Hz zyV4~~J26WuK&ghFNmPIQV`OB0$~yPJoP?kd*+l8;coMi+IxpN$n7s5E zX-1h_M|~y0Yp@6ab2!8c*yjm92o?jH$QZ2-a}3kL0#N71wp|oYG-T>Q>zW^eoB>Mp zqyuLO-368@Qv=)dQRYhlL~LR*T6o($d}uX4dw!yAFF#1O&z-yAdx_Jd zgk&l_(27t*4jnZ&zZj{6C~*un6kJcK# z=n>|ACVs#B`u&PNsrOgLo7A?)611P=3AkA#Z^@c1}U-tG91YeqK}dMwT{_WAiECB=Hr12 zQr58*E+tEVX$25;F=g?~YXi|72!W4C;w0KG?@U%Qtwi%6agtGJ$(M3Z!J)uma{Hse%(|n)7KGjmW z2ZR5S3~R~#RdzZJ#!ul4Q-B5Abm@Ml6o2KB3Va8TJQrSuQonN~1mq)|gu@#W zu`?*~vz$|0lNN8b@6M^24tUvLE|RS)gLK+I?Vc$-!Nt;tiNns!IE_)GdHjYwI+jUx z;X0OtIjldm15jcaEHe#gcIeg$*exq|^R5h}DZ~O`U4&A~NK?bpZ}@%_({Axpz`)51 zJ^cN@WVG?)f2v2g@fYe=pK{52@bAf00i6%TJ)1wm2?gy=7X!CkE#XKH5%-SMN5Z5c z#P?!WO#jFpW49>>jhK;WpXlf^s66NU5T9f57hT4+${h!F>`|I>In|6ssC!#f3}ZzV z2>u$!iX0UGd&P*cvhr3q*W-_Dz!YHYg#$vx=ya_e0Y!^1>$SJ8U8$WBupNw}0GtAF z476JYT~eMw^uZZL$%_>C^@~$5J(ztVPbwoIkc=wU4b#Uy2mN9e02yrmNo##Jq4A9dPc9$;qLwZ2-8l=^4rp0#a_mIxn7U%@Z2%4P4 z8WtWtt&g&|Z-vb7bF*o_?>lUr*rDxXF?1~ItP7sS`aPDclQS@?nzK)JtJS7Wo4o5& zshK@Be`sspD(Q;d*aP+Y8_RM#ekf*zP5noMubJ7sQl6Zl=x>J#o^s(^W5Y_|(9jP9 zp$RZop80u#m6a7D)ufTR*l6lnr@%wqtuhSes*7kMgn`d%J((IZZ(*y4)$EriY!eyR zR)>51j(&zPA2-q;{Aze|^ll*HoVB=7EPY_&E4n)J(epB)#~qX_!ZQ^1+Pf5 zmJ}o&M?Q?bQ9yl&&ESp(&&~H(xDcI@Uk0*I2?1`SoH=Hcw1dCpn_)?WIyw7nmrt)0 z(wk%|^ESm47!~$WHuM{2fA*Bhq;}r>)4X0}`PXtLWTP%v4&4?*&u7wk`1+q`OvW=W z8ixoOeCTy$<#uEm0Izz6=`FNzb#H?N=?)hEH9=KPxSp_*HF#3o0vLZ`^+gG_*jT>H z2~E!feIx^#+xgx-J4-9;BTdF%%T8R}5P#H-Pd^wXKWhDA)tgUF#c6JDu6eB|sGLn& zA8jVux*gmnC%w6)BfoTCbbpZp>CAVH&}V#~K=6%-$mT8t(AW)7p$y# zVa(;en3UA9b<7=6k}%ya0dSrNvu_a#!6|&0@OqJQ58Mmp2{~t>9MTPekoUEa?@o&r zZMRK)@FaE3$EUmC1*b_pdy+Ql;ql{_6;Iz)N_qYG#?ba1I`Bv&UbIQ@7=Bia_Wp&m zj|`x}9O!aCs}@(344U-ie+L~6@*7`*sH8R4O=eT&B$S)=jK0Y`7#HSJstTR(#wbzp zbI=69sh7WSWV6@{#ct?^F5f;?+da^qUv%yO24mM&>VCAIb4IP`OrnZKTkpW>d3hm? zC0;DhIVbb>%BLlB(eNDmXEWBEwvkDpaHXW$j$I_v(^^I=X!q=)`Q~=(hJ&i<&69Rm zn~ppF$O%vxp;OY}GwYW!wSN!F#e=yAnvIiB7Be!eZJ^e_>enX<-tcix+3lcCWBAtE zU~fBF*<<`tdiFY4f47NykMw7DQB9KB#YG7!h4fGT>;a`25@Rd*)Jco4Wsk(o4+RGKglLa){-88e#V+`mUI$Rs=v3_MvzAq@lv1>YF!j z_EFn(t1Wnh_W79kY+lECG_lVEz$}v-|F@kWk|x z-S$UBu$b(i$doi>;&XQ8=k_0qJ$3gla2Y45V$)4y^pv>tJK>177`zIa(ym_Gw+&io z^!!2fCG%=d9_fDe?OUghGoD?wzOi|9|BDxJq%+sMo!!;%&w%!wdS@-DFd3A6*h=Vn z1^84;`KrIYOzc1>0}p+nVsxJI`^|8aqMCAo?-g3i#>&GMus7hJd;_}a<6!0ca<%JZ#C zL)uAi_E`a)#I@M0V|gedq9axV&`0AklqG~;vDm_G_OYLs)SiZjG_;Y$pkm5Lf^pfM z93pT839+MwLnk#Sb4=-p!w;UEeqlJcO?&t5PKox1O%tyVU-C-6G-Zcu);O^tk4Gd_ zK8f}nyL8D}zu?KjG*~JA!+u}r@Ffr!2J`3VxcQYJvF_P%HB4!5CC(epO;zPY=?eV-!Q#~7$O{T{a8IF;C+lZ{+iDV*%fS* zq?m`I+`)Wh$NJgo>jtWdL1J)unWMlg^hCFA)y(};_6`?ZLJn)Nayezh6!TC=hk?3Q zn|;6CdfZW}Rjk+CX|Yn91kth`LWkjsY)Tg%q+{Lq`z+bQZ}wGfYnjdA;r}VsJNzxx ztM6xIn8&#z}~I58tHBl9IU_h7mR&_ck)>2<2)YAUBs2Z0|HZqSAc4tc_qiZNT(r{tM`}eOa+ssP|+L_hs`?y9LsdB#aF&kM4Ad!U- zI1-}JC6=VGudK|!?9axJ3!bWt1H2z7jd{6m5do?-$-P(gGTnzXe+AbhwyxTVf|-H{ z#3y#fX;D{z?qSLy9cL@FsX$@KEFLk93tc_qXy;i~x6NBIj1vkTw$t>+Mj^IK^Iwm< zV+Nv<*+|6CZPyyIClP^FhU8LB@LG9&B_C2&!+}?j?^66huQMVrGQ7TN8QB*v; zK18dyK(X(2*dyQD_Im?nUs-ZEwMi)4J~!H=3?_N1sE;3)@%+;3>UP~Xob+ECJwGfw zc9Hd2n@M~7qYpjE9g5Xy z@eNG@3RudenOYnEP){-bC5_(Skj1}WC#Zgbxnd*bU9uL7$yITyZ!_UzcTw3xH&$z0cFddI9<$eDh!y^dUBw(9wSOMoWTD|7Ap zU9MU8Sf_2fM=8l(ug=t*&fPaqU!lj+obd85K=7S|R;=ZlE8IUK=E83wbl6ByS)&5NJJCT?a~HBEsD=rzl!mI^$v3@oZN-ZX zi)!Z7j1q^t)BZ{0=(~kECp{t~%H2IZKY~p?zcgn}O@-?r6ZN!4+ka2@@PFgI9sK|d zt9H60CQyC$`8^fHf;I8Zg|3;qwr7K=?b)!&?0HFm6H3&JCQTK0qPI`zrvKb>sJN##kOl^~nrBEjP zm+DiA(mJOpByii}vB9j(Y-u+1Jn(PO_ErkiSul~DD!=^K=XqkAs@PKm+9cLADNh+K zh9^beEc|I#f?<+U&1L@Pla?#{nK~BBpOIb8=C2BETGFLl5!q-Wen#t!nDg-iUD>D7 z)2l-2*{+lE^z4f^?wd7wVOsz+Oh#$9@#@tzzoRxt`);N_684`umSYm_y+6MkPJ^*` zlq_$0*bD{$^JeDMpQm^H*xm zoe!q+W7CmiE1N4zW~0RKxqVZI3Pm(+6v0x-J@yNef~esuypx99ee&w&iy{8+cE0(= zzw58Tl4}yY)SN|E2rwA>513}y5QH|13Tq6I{!s-<6RiXX!=lB*{C^)w``zeU;GSI8 zZiU^sLNc9Lt#FG1qJdvMF1Dh)vJ=jygn=EK8}@XhBE}_0Br@I+o#@bITVsv3X^u?R z-`KbBoCyb)rF9Tkj@Uq;j!!s-L~j+0bb@lNH2{-3TKq&0*wLj*BaYI#Ly>RtX&I#DZ$OdPyoq|FfIZtdGxZN<$LP`0A z!q}L7Clow>{~a5U>M4X;n6EOl$}Yq^N6oi*o7(l;f5ob{qz${&@2DKqAMkI-;vhPf z%`KPKRBjqry1sj6+4Fm8{m1R^bN1Oz?clp%&koM&=y~lx%HT=Uo{aN)Qc`s8a>C;N z9r{^}k?x@LGKS3syC!Ie~3DKCB32f{e!5eZ^O%u zG!B59EdO{uUFIm_C5;h26TNN^5YuQhu`;t;iWOW@H43-dzz)8$jE7o^MFirO(XRE| z*_s{TwN)Gq2})KS9UZZ#_?z0xWu;Naj_;b>#s1{ccI$VGDL|wX5<*DN(EkZavwk6m zbMx|mTm!j35^hh&8T6Oit+B}}GQ>G(kLe8eufs>sn?TauaWX^(dk@CNsoP|SEJs8O zMj~R6%*1ks#|$@rdwyO~b8|~W_o`y+-0+GpI`~c^IAJ%&k(xP8uVVdmNZt5vov@Rj zWj6PpSgpmDoSUZK2w7Ro}8Fc+j>dwWV9$+;M)YXLT~G=T3Hf@#{{+#U0Ln zXd%<;d#xh-`4VmKFHJpvGd`PQD7%;JkR4fNV&@)`|7%#VNf0Oo?YYsn!T^2aPCz0l z5gWshkxbESeb*0*h)CwaCH7*lDCSFBuT*IL>YB63Z2f#I(#tGyDDj!m(bsR!Y;W67 z*QftS3!rj;^ZT>t4?yc|hCZFxo}MA~$J4out5XK)=xl|lOEh%b5XFA`v48iifTcZ4 z_6U~^Z&P}0#TFP`Z}Nu_Oi~={Rl03%`jb&o@+|aHKtNkrsix68Ou|G)WQi`wo_G4n zpS8QxTiGRyX>PpW(q{1D6MrAoxTpO#1vP3K-2C1V;orIHuQ*#~a=WLi4Z6y_tli0M z;xgkTr~7DIPouuX?E7T(3Ro*5x}EcbS!QM%`xe2C_aa{;91GeJ9UYxGw%0Ad{-PZZ z#R|FI)}tHlen%K(I;6~|Q=g)Jax5EUfln(5ru>9qM8BPf=caGC*;mY%--BIKSsb=q zXv2U=BD=9=Vjm~ww4vLlV1qQnK(Qh$AChPKK9fH5j39_Bcm~4I3tVV`ZjsOTlNZ0e zj~-c&pr|+hj7@k}PgZ%HVCo+36(B29xUzR^6*)9tZE@12Ew$-0Pg+`7+?7A+5+2m( zCBiBi1MZB(hh~@0ED{vXY1hDZC9vljEzEvpd)DoPUJ;{r-!Rya;Ac+2S6bF(8;7at z*~Q~uUq8#)iLn>6nA6pKCqjQ(MMp5^nrUs?WWnT>z1kobL$xnR>S2H+#F`r<$B4%5 zHL~A)%Qvm$Imv`f-b-h(kBolG(5(-I^Uzmo?K|dWB!M+? zHXvle?L)CZx3;hXxZz3&4%Vu)bFbQKt3Fa8CyP-T9#e--o$gFFxXHaTT_tN`^7@5w z)l2oKXC&v&wK|p66ETR`UdNAlTU~AG>UyI3{uT$7j#8u6sU18vmo>L{F93B^jA$Kc zQIUwuU6luGe0jYiYq)+WRn0NZpU$ zJ`@UqPAau3m1(2Ud@E#N5cPD1}+cBB{0|4y-u=PO@J}Vv!+s%9js15FIUQ zXwm5=>$%Tgg}8_}G<^AMQS0>T-Fq8|=Ir~gJ9Rv2+3A{hYh3Jib@$*A`$mZ)}Wb#{*VoG#KbT@D^+?7zdMqnB1`*V#i5RJi{dO?3j@8hiWD zVJ=eS7%SLO#?Si}>D0;l`uaWsYA-C8Y$bCjrNr{H2)SitWy*^MT|eSkcEEIIWuP%# z)@d*!%rWC4))iWt4t&2x_ef@l`f;1^#=blfHr-7rO$96Q9G&YJl+Eq4itUZeqcguI zAdyKSy2Rq|YoVE=uqzE5*i+1w^ew7jtG!q>V`JBCC6Et8IRL9|PR_ch%{#gjWy6-> z*Y6g}8yhcc-*reIK5G#YFre0|F5XDR6lwupr7l z%7KX)HI#XaP5N9((q^6(cIKpE-9dCt@S~DP+0H`Ma}dNJ(eSz+SL5odb-BsXRV2$i z3}zm>3~x2ilj43611l23WFALS8V^~Q-T`++jK^^7*0bv#kp2BaOv0dS-eR5ye^%@_@X<9>#}w(bv$AH${K4rDZIXotUc>Hk-@qzc{z`2R zl0=)HgvMgqq1RdVHyvgz_xsw$`ma+Ma|{ff;7tMEg;`NbD2cW2_Rk#VQzg0<+ZEoH zGiG!~!LW#&C?R-lup99XR3;*Cl8hoBmEp%;CFtFSh-Ax@Q!j$LNFJ2Gp+K>aY?^HQow0- zGYy$ir%o>2GPX^i3SYbK_Kr?ASM@C>Y)NUm(!v49=qz8k zC0uvcpEa&0MqFU_{ZNm2R-Hl784R~Zt*+A zCkrbiS>xHWXA8kjU_?debCw2Tcv)CCRFn&(+jz62idSWZ>46b^bn4OQF;;cYL~BGv zZnXy^k@c7nox5Fu9@7>5#}Zj%0?;nVG&19Ms*8fNbBA)HSimqCuIWIh#gi;I%{Utp z8y{ipTrwmu!lK()8yh7!Be?Z#RKF1uPWClc8L@iswuq?c9^!ivjs`6pWPUY1ILi)` zjXWWhr*W5bQPo9hHGf0HLKR(AajFGZv+?VgGIF)B5+=1pagKr+RuY_!gxD6ws28r^ zS9X5W;H9Qpn4+LJHdxmPTA#xw|WW!gp89 zN^`@|R$kr6A}}=57FD=7mZZe&oSe6~a0pXhY~pGPuZx{o!UI1(-Isz(}S$i+=F35FZ5(71n4?L1EZu+Oe&&-!;gueKG^nsC2e+X%h{#ub&_ibB%(S zui>E}il;;h6j5tfeg=(X4ApM%6l$rlvDjfa| z63ezN^_iyiUV%S&9%36{>uD7`glRif@&!(y1U2AzW`llKS1!RyLm=7lnPMG>sOn^U z1_+BrAo4d1d8oq#lC84vp*%{> zz!Hj+_ijwo7YsXi^PtkiC7zzSJg6$Lyq(|V!6@5VTU-Av$WiDM6mH?Yb}uyjo|9n# zvwy;KtMCfkoRXT_$+sphU;%=GJ#$7+g0q(iu6z0AxkfM8cV~7H#Ap6jq4!OLg)e&; zu(X?XkunEAf_BIn2$EMgTZI(Wyoc6e@_=mIJb-^QGiSEOkl^I~)j6WqL+DJ8T`_Yeo#M=f%P zqLN3}@qK6rL<6EBwwc=83r$$(P*q)>O<%Ffpv?B$FrvOUlNlSa_i~SA-~#O zSvxSbX~?r@V7af$kBfHaqD0C&D~p)ELx*0~&^?ZcNYM7}ZNUC!u80W*=J;EhwcZ46 zasdE>VEbSqCHu3ag>pbk63s1AAGoOYQ3;p(Hz;ibVA+tvp6P*Qeuo15#E)GU)DI=t}KBIz68AJx6kON?{+Gc>J9le;aJG9 zR(aY<%a|0A8C6uq5qvkbD8Az4S>E%_5fUJR>&mSWu!Se6-^3-Ojo%0!L|h_MEktm8|MKNa7H8bU zYt3+5Btvfk=A9muu;k6{K)~QTKC$OH>LN;UBu@|1c8h131A+D1{jSR-?g#f1Q333RN2s6=h3G9hLJdQ`9#Ua01j5(s*L_^t%-nVVvywq%$&IcYJ? z-n(bdRvSf&!dK_iOxN-~h4UY7$ijOl-ixNhjcPVN|A@)Wrr*C}<0)gq0eA-V@0&jb z@(_c}H&0sTPW=E3!Ahf03W$@#th1T>TZVZkalz2*55y(TJW)HtxATGypidGT-5IJS z94E##4$ClYX_t{?DAp;#T>>>dGvWFq_PKv$HYvfSzqb1 zvZ3a_g65)@rrKP@*O64ckyonk%lpO%75c%0s!)E$UR~K83!--iqo3C6#?Yc4C8)7R z<4Lj?*?CIDV9R^iW5r53F6shGp$BEoGOeVgZ>9R~H8r&uzxl?E-u4xtYkIU2$AY-N zk8K6K+`q%;-Jj)~Zi(!wqYK7%!9BG1WLodrqV+Zz3hTGz*@Xw|s_+3CQ@7o&F!(CiiIhFG7dZNR1BZf!s78VvJ59re;NMJIIJ@MDi z;DE%G6(NE_98qb>8owLJM_qtx8q%^0Y{Aa`J#xvXSuQ8DA3eI}x}O`lXCe2Eb<9z8 z7$i&QZn5#X2V+-xa;N8wEFUerHabrbSN5C${g6;q?};Fh;VPC#wQ%{wolY28>QO6$ z!`@4#`&ugc6bkv%X4;*_b{2g4Lb=?LSDL_M=(I2KDO9RD2$Y6TZp7}F!Yd;9A){mg zk(J8C7yzrTg8yexjr)gs;hc$5RY=gM6~3Hsm0<*tL_un-O2m>UD2={zl26L&hb`FIArfkb>;@LtGFG5YFekSdS7=$YjUK=cuHl`M`0^J;Zy_YzhR3 z&vnXdPak9B{j_$>AupP<`PZJs{SDF(cN%SITnji=TwLre(^8%A<=sIRa(NbSj?;_Q zUC^5vI}e5m^CcH#+e~E-{ncjGpCL@>s_L8F*>=BIZ!a%nQQvi*D4yqMLDQg>z+!Wt zS0v+qR6MyCmBSm?aQWi^=!E1!f-mxwBLoBcfT0TK}FMf7tJFd_yD-nti0EY^0sD&XWW;WamG-lhebswo8v?_;&HG+^L9ze z$X{*t$F!f9ax1i_&E)e$?eug z$>Ag15L%0&0tR@!KAh{R#qb5i)IVZXSN1p&!CF z{yX8W(r<4?3T|(ksA%c{p-Pf#E*H-iQ4I9LruFYu%<11QD>I$&>$2!LXn44XiBdbB zgUWV-K*+kNJP*K-nm*9iC&sX+y1F_6bBB;F@?!pZ|H{f1Y#y{fVM2G(3$2Ng;j7eP zyue)b#$v}i7o45E6mk!aJwHFfCNz`wcq|hCiIXS0{Qh2zd8-l6HCmzcM%NqX_xEqJ z{@3mP=l^-GhiqGEhk3%yh*@5UzagvC)}hKIao)?#4dxl1n%Lq@jaPJL_MizT6B&pg ze)a?2k>aYoVHkj+pIgaTNru8(y{)Z1Kh?ktEP$7K>-qEH{0rI;At@ER)M9%N?0Wv@ z&n^PKr7DP@I+lqdVL~ii$WY6io@fTP1NwKrv~(2zROC-=1+ZIV_%WOCL(w!QLU;|bu;|5V;9k)`qD#2$ z|L4k1X%XT8fr8mL{s0us_C#AD3ZjP*0T+nFg`z=1OjjLHSv=>8{5C}v$R8SNE$Sn& z`4AbHc)sYusE?%7jC0f$_j6Fw)>d9jx#%k?veTi!BLU6oTx@cNRSEspU15<5PCkLN zTj%4yq2ZmXSa3ig(cB{U@~Bcm{&J`p$zuh4smTP_B_uuqDrT%6=l z*n9kL4R*tg*NjkGC zA(%Y?TJt5l{K)8V>GJBDIb0nv7jIP}#&EC+6S9dqpR6GxMnr(|ysMf~2>=Ktq&{k0 z@_~KIm)OhdjoKfRlieP3aaLBtxUe?$+te6`(O*H3{ZKfP$%bUE##ICv~aI&K`Ry@$2yp?uQtLx+O6fgEF_Jt}Hx z^-`l7l|I$%ijD6);BSL%AOV3W|!AOpjJVLk+*e!Hxn? zvbnfmoWIlEPxDgOhLQ|O{&^ZDbQxKs7%Wt??S8=H4nD&4`q<0?EKq>n-%J{<8r!pT zIU|U%ry;$Cu`vQ$}-G9^J^rh<%v#U$@9&UFeV5z)$P*%^$|2J6g`_lEwVwLZoy*}OUtg+-}{x}PZ ziol~z9N=g%Y=t`w;x4tq!OTyj8_bv z*sXDcA4S!(tIHgDia3o@UAa--vU~e`;QBIbtQceAs)Y8Gwmmaz;OP3-t#CSs)wfy@ z3TmIvZSVdf;q>&=_qi|I)6-kzp6?$BHpkR7PolXX@GU?PHL)-v`ytsO_>Y6ZtP2~d zQA~XrUoi+J8l?H_Yc8v%$bb;=tC9fZG=`p{joeF9kHWq?9{lGOgJ)@M$+!h2qQ>qO zmZW5TjRNi`ah;&&c>gHwt@@{8NBt)i%{XnjmP^*90=I94_rq;5e3tTd?vxkd3;{qp)Zx#HLg!kQ=y#Vjik`u*C^ zXRp3mz-- zQXjIpdnYQ+_YJ**?j<$N&YTY7!85%NRv?7@^H1jWCY3t#88LdRY&&>1%Q5373s>v- z&z^N-CR$rB;vfqRAf?>9wdWvJ#J(#bsbrl|s?r~^QfM~0dv~po;VYjZF8AeuM$agY z>!w1}$MlkLG~}hZlI{)+s=qeO{dvdF|4se6WaZ=tBpQ<0mLCH7F_~J*JHYAd*Mj7} zFK^1ZWZq71juVZ(Z$_VII@)eEykkcHe#FYPl0fI-jD?WF-5qzhz!w>>=jb0Q*PpnN zqi)0yoXK_NBSDSVMJ+}sg6gq?H{D$cIfv)5@CZl2^WxYTDcTb{+kdBT50nq5L?-m# zCt|RN>hX57`F#jff35Jp-cwc7l%psq;3zW&xIHp5^4_YgVyR1+|1VA6dT*AoWv--n z1SAQfWwl9iwe(l22wVZPC%PTu{XvrCNTN%A3`EFQHV{nl zvAQ63%j^wDgyIG^Nm$QP@--EF1NA>aKg)ss5a@V+LIs_m*l2<)HLXbE_80?pfYV+O zgGO`RB5GFTL)c8|anJPk(%)yhKKWCde!ka|ioh#$LZou$wsl92{1e-B9p?f1Z}vw! z|15P^kRh&W2)c;h9d_tYeSxQdPx!~ySYaDC)ZOONbfF}wgr3?^HNMqGG2YD}CvR#S zM`8AIgB|VvqXkIYUQ}c++IGsE*_gJ0?5(ddHt;T9;@eo3Oj+hlA{49XKw$<{%A_8j z5IcL&#Hrgx`#V+Ju9ZBvN7?s4?F!A*J30@Rak##r31eNcf1UoMVh}DqoTi+naTK#H5(X&T73h_?XgdKer3Q2gbIhLqwhv1yKkXeJnt5W;TigRhVtv!<#(f)u=S=h?%!m5XLw zl6g?EX2xAxXkhUa@KqDjo?IeWtv|nFeOI~+z4H5qbLG@Nvuwh9X=~SQwuKwid(3yA zEh>7qhr-K%N10le(;o*j*v(GBcHVKh6Xwk6LO_a_tLHuktnG1!(Q1nIfz&s1lHFt5 zbq-B`$Mq9aNsQo|eyXpF1YLOZ;lobwSU|}WykcQ(khp49PF`MbiZ&<<@V59wf1w|(6^0j`klJHc!OYt`Dye|_7h)q6^g0!7k*`%n%ZBdk z*aboCO@JY9G`;yqiz-A^%wd7w!e&G#tdnnEFPvN{q=i{3qgxDS{wfH&n*IwfXq+o1 zBewD-V;Uz)(a(ylo9fhscZ-V;!raZC`s>_QY4^v{=NvX27~ot!-e#sxXeJ5BoS&Wb z;6cUd^J&^zT3R%2p^};-%v^Qs&tZf+=CG0ALqb&#i5U4P>Cz>tn;x7|+Ta)*0e@IK z{&_svfaOx#z=bIuGO@WQc#OI%w*hr|Z>-*UO@*3!0iP;>UjlyDVqnpg*TdO9i5X9C zdyAM9GO5@xI&osw5gGGJ*?1LZ$>Jv||CU?w38 zvS~4fulcOO6@CMu-|}PRm!V>V1@)AOfHgH|-whqtEGQy^+`1Fx-6T*Yz!tl|I1(Kd zbrN8VF_r}JD+bIdDl+cNruq1$AADB9YVj)=+r!;HtXi$|s{hCV18UNn(}|SU{4jDu z9E?l;fPFm{^o0f>^NQ;WKCyzfz|{XNGck0Mtf}uZ{O2NF-2~mwehh@($CT+d$k3EO zzu*;yvr?=3tDvuuTwoL@Tj)XfnbRd^S6GW7cEB+v)yFU?(tWjO^Rmp>?4U?x>&);C zmjXtvd4n0nha?}s$HHYlVW@KOcCz~8GiJPAC2oKX1;`@Jx|%?ADTWxqy354fPnI_ zu((O)OY|ykD?9{}s`Kkh>0%H9WU%`4%WUc|(D`1&hc_6sX!3~X`@T~_)y7Mfr8OBG zJZ92>ajhw26!oMl>el_*z2#;AfBU?oQxuTO9y2=qJ zzWjiisPB42;HucE%5&D)ReasE9@7{Ab!R>+EM`^c=B@cVAtn)b+%7HEq%{}7N8Edh zZqvBk^Z?XDN(>9i%dcr z%(P|67dU-Q;;s=-FGiJZ^ehg^&4-xV)pe}>z5^#a`cCZmp)(rz+g##P?K*b2w4AEUZq4*c=_;l+W+VYtO47yi%^}rS70^>e7KDVT?!O*AAv*+fs)RGtfEvU15QZ|H| zWH=d$VO4VBL*yM+^I&UJz8C4&i<$W%qcolbP?-{+JICFBS%-^S#Ayye#`jUa;ChAL z_V5?n|NfaDH8nM}4;pkAvnbS&Xb6llAJ3$kx1Bc5+j7pF2oH~}vY*x1W)_Z)4hvf_ zC#Scu@snhSDDFPKTCwO=30+apaifzPr<<34)vtTW4AEC?UMgnI7i9smr_Tll0~-f1 zMK88yXKt8xdGYiQPtQ&UCYbIM>M&ZSqdB_(zH{ZOBIf!$@FT3()~ggN&% z$K?j|6u2+)@i!r$y8u*^TYX{u$IgR2WlWpf8hb*0+Z@~4dwRt{@_rlc1RBjLNJ+2; zOz`z7S7^7X+o+Jqj|R+^YWnRV(9b$0pbj1DKTWMgqzh*sJxC4H0otQRMRPTgd$OtL zZ1r0|QXfnBhOwjdD8AsrgBkKk2sg-H_VzFX^f&Lx8P~Zz$W9ob&rW_BmEojCW6cjL zjQ+u6H4f}f!nlV>@l3CG^rSrqO|%PJC_2dglR)C%NxxWDCSzbQY^~C%8G)rY-ozao zjC!d@&z@UQK?>y*c|z#E^yiKE)uwYNmrr^74#;fUWat(mx?G(-LcjE1e{>Z8ur^~C zd}?}Qog5lHH)`Mmg`0jg`@0v)tghL-Qw7v#=KMzxwla`@grr$%e#0re#J(!=hJLmA zS!n(GbNjx14PDzlh)fWDT=teT)204?jJCuq(cGW%E&6yI7YA!+bjmu@C?KUkuIn z(Ai0>Hu%q%|2}L2#N3`tZM=*91{E3uN}oYjdI%r(TzL zc{wDbtGtqugt<9%Ai)2RkDj*Ux-nXlNe|tF;v#nJb|Y$2)FEHq+;4ByYm(Z*gJ*2a zMbWE`YMWp5@1Kgl<9gKc>^_YYA{WFH;tu5cwxZ2X^?hB{_RX8upU;oC!zhoNXD{w2{oQJ=x*@blKs#>rtHXdw9w0Ia!qkgy$ zHT1#~b@}e?XUs5M*6sU;IG(!{aHTG45sW6p+W~Uk`)A9Dl`=1`ZeRb)sX(jTJ6wF` zfLk+q{f|4~dy%tN$O<83C01}DK|6Q8rDICw7cAA3SCe@D+;CMt`EF;#FTj4#iOCyb z35v!^xNQB|lH#YHn)>>_f_rhEnu9W*xuMhlI+=A+1J&e(vo?v)_NbX8orMKqS+7!I z$uo&j>&#|!_G=y<((SDH`CKceNjrex2*KIu)1IFdKm7D^F98|axJ`le;Y*%uH|KRW zNB!^T-^8gc3At90e7wKvfySBt>+Gj$2|fWgkL7gofOvF#bMju7zrB^&=JDg#rE8oY z>!42P1aG8;o0-7Q!FyD>7gx!NvJE4A*-j||4cY{H$Mb9h1GDzw2vaD`qj-F@8p|{ z`hNLB8HhQm=(>9Z^;qD{3EDzl+OF6=o}m-`0mDHfE1GY?N$8s|TG%=xqu^|$NNE<` zf}2PDkE43ghU(ajfxTIC{|{YX0*>|CeT_oqBvFz`M1>M%2$4!iW zLpQ&5IOjtXPJuQ0eyJAbPbQ77AXVOl+LHf9h+XD14v-xA(SyeQgMInL@gK7C6$gux z{>Mkj%KB!wJ%XTt3Z%j+1qo6(Hu2$Uu-P7PA4T&Vq?B>(L6TRZlK%Yyv1z+ej1jVn zjDrZi2;`wD>rJw_x`x?weQ3a8TntZPZycY{%%WX-Yy7t2Hnl0d| zj=fXRpVgyZAtl9zG#`ZUQ}OYVAoc;}#+Le6?iZt#TUM5q{BW#p2#P1Z;=h07?}1y4 z>X8an8l8v;3|R7Ho(f0a4n2>LMAfc!-1e zKtaBW&G)3BU;`Qo0HAHax!IACQ!jGoF032J&R)yEQc&J*2T-+ut>t2Mw_|b~-NE+l z#qA#6j7&_k=S}Kr^+RCyJVN1rhMLDR=wV1aPV8Ao8~~M*c!DSPdhR(<%B0sK>_kof zD|SAAf_w>x8#8=%;hK{IRNLmj!o8h7*zWID?VV3_*xS1h@iB&2@-E*@h$RmVBNEE; zp%rr_Fx$hvTcWo=uX^YHtN8fl%Xb!o^m~C$vhhFSE|HKHQ27jz6PmMVj7)%V@?tlO z+R#)9eU`(Aw`9+H``{IQ2~^Lp@jHR~UsP1o8A@V+S?70i&L19>cXiEqecMc@egihd zt{?)I|M~ND4=Qg6VyN|vmk68HIe>uh1(szekfeY-a(nS1M}(FC1k1wMO-&~Yu$Gmj z99}wg{-~-i$VzYBi$4>@t0>eyw6qkU_5|Yvj)GNpGq|%C3Ntw>z<2jlCJyLm#vsJP=jg}FePOtHq#+B zWSQ_hn#Y?JS#6X7XpSIT5hENLnmd><*e$q4gW-83$K18>8>b|mIL~gje?J#yv^uY$ zZAZSuD3;IB!T>2AY3JwHAfOl8%503!YkPqLK$4s=aSb$l4?ndBoT-|(Z$A#_6bAa~ zgyg*7eQ;ocvGG`__&b46wL)9`3ciFIBgE(ek^(amN>TDz+Fo-lmWSA^F`tKaavSmz zx?db%M99Yf8Y4N{bnbE;IFOQh?A+wG+O4D8tUU7mpM^%(AYvH#1i&0y57eSJSw}MV z_=G(;Ab-gE+4B<)O(0S92`yf{mtXE;&I#5EfxQM|K=zNp^d!NX@YcBO;~|a*XfYR*zoz zqg8(`9aWm;PE&aRY_g)ZCSzd`De(ng>UrM3w{PXfn_jWPY|r&m{BD{68mTQyQW=yZ zu-&V}CcZ4+@QHn?ZO#~zkv6g8g}-vd@*0SA@CLYmXkLYVCh=5IkZ&y$TIN1AOu@T( zciT(Wi!#jn7qW|X#UeK^LZihSvv)w|9V_)@cCrWvKpgS*zITN|%KcZA1FcD-#=t*+ zny0VT)xD0ZLO*aLOwn*71puq8mFyb+%!5{$vj$}o2N*~@A5sb=5tzAxSW%aCptd}4 z<}qI`^CIqf1{z}zOV7&<=jQJ}s&lA@fg=_`QPBCcFvb}#dCHX2kzue02I&1t8cgY7 zUpTv=zq}5BATm~YBthV~x9{Rq*mRpcToI=#LWHH(U)50*@eiOpSc{G%u`tkt%Iqu@ zLX42vaa=;TVbE>w`;)upJ2F^IO_9ho6{eVbBa!pM{6aCz zqQmDitE;PL^?#a=A~j(HexxyPGH=XivT{lIuRtAjmFh6uG@Mg_N|~c^30CFyGNCkw z{ItQ^#Ek|IlNNn3NVy`hxtVS5+_{Y{Eo5K~SZ=}g2b}x&%crgEDGD&0E$2 zTOC~kmQPW$IURxk47S7w2`;Xn!l6$Gs@+F4x>3Q6RFAwKx#RNPLAknbq)5amZ`sOI zYq_7cr6inx8{xz2yHYRvVnV`9WyA2VX1niNm%NRTejp|ltaUp7^uf40${ZK07u=lu z)Vy$Ja=nQ0j-3yhEUiDLe{{&MOaGab^eE|a&*!0ZHK*|D<)4StTF!4w*mQFm0=;^S z5;GIy@X&WJA{@R)9RfgwR5+!ug{Qy%k2GduQ+i*5pCqj*Y9k$bv!`)mPOf)F99QPg zX;n#FiHQcOSI=DVjLFPY9q#Z`Pr+btfcejWp~Kc!vr*rs&oOP-ebPj;=YnmoOD$pZ zFE(Eqh6yGt{My*hA3g-glCNU~Tyiphz1?5WHW%Z7&VlUh2qIbZPbCWr3muH}mi6l6 z5HNLZl#SYsHXZ3>mM#qiKfN1t(sy#B*DIcQDFYMU$yyWTqBGcCdEWcuDj%i}r#91{^5jfr zl(SrLz#M&?sl-aFu?XA%F(W7u!lp5CUu@dmDhOKN*rw3yrKMZR2C^$%7W#4WD2A`{ zTKE6hh=n6)o&IYZ2z9II>6M#>_{RdDL3Yd_h@=Reg#KXxd*>w}>&-@!&UvN@daFnJ ze~3<=bYge_&v#3v>!MIocd5yo%h2ozfcfxufpQ1b(lI+%eoJfOMhEdg#Mng|an`|$ zn?woSjh&jG;`2f8A!7LxP9ty?Bc}@ zH*Pju3&iWYq%M->b~g=|y@4*f45d8qZ4L%91Vp{4-WP!`CGWxKDjdTqH9u}6^2?j$ zFm!cyKP@SNbFmK&sgVzl&}dR}UiqF+Uv-72XBOA_@t1!pd+4j3YHx!#pfNf40;4RO zr53_7vEkzOh5~^Y54Sbzy!(IB!~I=Tm6j55ME&!bciHI?Gt)lKa2-0hY`|tq+GNkd z?2I+R;{5z9;YQ8iC_sSrJ4M+iDg0(>rkaubH2*i?abw>{P8r z$gYH4J_DU?`;#AAPQcU+Twm-ik_*-ztloi`*NIU?j)+irniCiWRR!bLAFI3$L05x} zOt5Dva;F!l%s?mA1nP`=2{JRP`*oKxyI9;bl5%Zs-WUI7fllz$&76vf#h|`0&z6?< zXHjnoZI6hUNlE}y(^+V)!|m>qq=?ro)FBZS9)Zh+4&y;98jQsqfBSOPZcfN5RFI31 zK^b@eKE{mszPV3MO%@TUeSCf2#&GG#qo8v}ps7Lh2tbO^14z%{%jhw6{r1T0Xy+^A z#t0{yi{d)$DtJh-1bqJZ;mm+WC8AQEzM2(es{nhL&V`2dco$V1*$=8tQJu`;zI-bq9?`u)S3=p5W-n!C;fQznGq@hQl{#l9UH%UuN zJ42KAQN6zx?Ay<2JEal*4rw}}9p}Dx5VX9SmE%{V`51lGAD^oO2M0X}mPc^TyLl9) zO!JzPr;fEoLlb0!73ApXNNFOm;!!wYV$C_opr#H#u`FeuwRAc^v%Wglfp>4|Jw3pC zTBCGm>GkNl+xaGX>%MrA8|sLFvh4G%yPh4X$25+@UHM#iIEMy0hZvQq6r&XF_)IFM z6Go{kW=~ED8hd4iClKhkt4$9tgrB|YU`KwfQ~C17O*y@t_N|eaLI%R$_bT+#~C!UnWI7%*@iYUGn=#&@4sF;>22OdKLfcf9(L zR3`f+Wq#Z2wLvcyTH8hHFS@;N_LoUxsN}4Wix0O2QCL(lJ`PUZJ zu@|V*xDW~q{mbNY&w8D^PiL|bId>3VuB{#d;im-Xl`2Yx$8gf<6S5-8|FGqUDa9_EXBN*LID8y|uzADawHCl(X*?ZNr zU)zdH012A48DO(r!TWXmeEZ=P_?v=83nwUDJe_(qYDay6{T8M&H>8GOG4SR@TN}kj zxPMBF=}axE96Cqt+Q(d9cw z`i+w^zNP0y8(B(>>O#?=m2;6fzhECNt0!{DV#KcFoQ~=A8t#r3IbwrRM<6SS%E|^J z=|-FS=i|V4pdcCVILHs77)Bya78WiAIT=aQYllD;m_1TtqSF-YsoCo4BUYtuXu8F8 zzOjlvz&9RG<$BNn8#ka@k5kkQWMKf0?9oiGZ+n+Tiq7Y-&ET5lUX0*Dk21zW3gkDO z$6D-OhHh>ZH;p^OlDF|sTX;M~|H0N^ajC1WQzQM)cbBOIqW^$3LqkuGX*95om~}uo zNZsqw@^5bSO1|IjmaAdmOsDe=9F8eRu^~FUN@UR?{SjHd6wo%&!e>u5L->P$q1)>5 zyAL>y+0O(#QF=SJml5Pa0r!Ka2x*Iu^$>a7^=l>O4MC|qQ;BJ5Ekomjm*ajAqOspy6*1%4kA1g znSftKAu>UY8q%bx57h^GfC4~5ky?PqSqk#F2&xN@&H~B|;n~`FIQtiv_|y6)Xk+bQ zx*@5( z8CGG;S3RGBV!!~Z)pZ=FzmGk3Ey!g=jZNKXM0gV#Zy(`CEe@;d2==4%lTQGLT_@oB zar(($(G0PCptBRhnh+Jz8UdA#@mf94C`2;=qxr4T0Ahg4gM}pqP|yIzi%%EY&5Tub7~`m@KemwB&~mhC%0Xyn{x&8Z0eg_=BMs%6?jX1c}mKOgEqr)b&}B@@31KSVab~ zb~`ibht5K{lfHdC`=9DdgkW$yNmfC>vhL!77-W^GroPqXMutJlF+=f0mC>a5zIED0 zj*g2dyufG(eBBuWKG)u>4KH(Tu6_TwB3oT`=-wIJV!?4%o11HtaZMrkOD^U;YQl}FQX3VL>-J6L2NPfSgvO~5ZSOFLfdD^5QWGgB)Q0)As+tjJ_`XwqqE2qr^^R6|22~C7& z`k4!Q6(IZ&_lkJ^$lNi#L0~u*)>tuddrFR152PP7NDQXhy1*Z?#0?D*Rm|LpIQ;Dq zp!lm5t$dRvxWOkCqP8WF~csaPog#IunZvP5BFZ`aYQpMUVL2C z@51~T_n&47iL1DT=0{$k{2T$1l44{HsN5YJ^`+HC+<%=2-RUd6$!y3y*+nPVXu))w z@sAhR&O|K?#&t2SJ^}h9`uNlGWs{yiGPo5|?IrR-*l@XEr=G0@WkvE@(; zq>@DW1Y69Is@Uk7)6+ZR)mJz#Ss%@Pd;2>i#YVRfe#W0p?3(i+Y9oGZP2I(eypzTr zS8>Kio@HW8YnkTSsXcG!tjbDsDDdn%uJd03`y)0k4hrqLXzs2DF_gn=S4n98&uZ4> zp+y$QRqoR*IDM=6eGC4}unbI`N)=*qG%-;PnKyC-ra#Iqro9z8g=?A3u%}3u0NP$E ztmII}V66)+)r{9W7s=K0p2MZqoEELzzaK0IX*IcX(2e49$hW_*@sP*66mvNTu_s0O zc|H~Q63<*9fj@JQ(XBc@dKrp5w8MzW=9T@0K=y<_tG8YdX+;(#D1bo|) zw|49)n}2y{01d)jRAx*R6+xM9-1NkoCvQ%m#;GZQ^h_Ll=t@yoJ%Xhl5-C!CgQXJp z->5@y79x?kIr1**mg9U(#rDOiw+b>$JZ|P+3gC_;fd7u-h_)Y@SwR+PfDj-wdpw6Y zh6;-$K2ZIxL30+RYzu%C?qspTvpzv{L?u|FywuVH0%G|dz`U5~Bc7gWD7FQwcew1p zj+-+fDIGxIJlCq`!M~e1bVtH`-*p~xsnpnBvEMC3d^n{=uB0C(C zlQ!fWgL_p>hvbD2K28NN=d7Z+K3C_D)#tGzyAZ1jv6;&N+heCmfRZq>bw!+=%NB4H zt^ga2aIVw=3`IydnE<2wLuxLGH;h-#JxjG1rPNb!vQhdDw!-T{cuy$25%l!R^gT;) zV>Eq~8qa`EYr6_^ek?};YXCGb5C_a<6e%QI0V0## z<3N-@9DhspRi1#;dk;9>;ox~gcgkTSgDRDdYXr;A*W2wa8?{_imS?!%SWvkt4qFxL zj=q$7*jKeVAg$KKF9hT*UFiB_1KPgBefnvGj-wYpfi9jV3ZX~27?=>YQg(MIQhb_U zhg4x7-X)HLt;c*Xc_v{&zV{kAg8Y*nFr*CaJuHz}$Z3!ql%iUT0bi#W}=68aNrYV#W@HR;DP8 zU)^X>01q)xm8qxJg0{aWi??q7rURgzpyP4}2Y%DHKp5bm`lvA~$g$!yHx)^?Nt7=vZvxKMEt3@3js^!lYa~_{j(qXns@X%wgqw`zH8t z=W;F0IWGnL6+5ZptDY@@5%{|zDQE(Q5T{#lYr4lzesIJ1B;TT7ij9717UIbdLCjnL zs=I+b7k%Ar1e4~Z$G^6H%j_<@+R1&2=@5vCQAd7l9scz&I~AAQq;?1@4r&{~W4euF z<=7MJ_n{M2cWK?koDhAr=`;M!px#1LYs9myPAKXy_tk3$ZW(FRiy+0)2SN=c&Q)Y> zKI(~>NxWeWCJCgw_zCR%5f&=0(=Q@uV|a{xBGYN1rDy4Ff*$AhEs(@1;dT)EUEZj_ zT;W5tsHCJHSl(TzR#Lr2hAW0(0#OT3HPzwXCX8V%6TrYpy0#zcuCS>{-|Gb|+{h z7w;jz(HWTlC{J#uM<+8JMVnmCM(#(7h7}V7(5}E%mkPvLZq2-unL>!yHjIEKhj)dY z{f%Sw`16_`bgINzIZgZbU+vTi~o{c(^{JY@Kb^>8H zBqGA3Hv~=LCo8u+kW zkYX=-UYZsc@f69852_)!WQo93NM=x{rG+^#GF+Y&l)RsxW40-b#yJk`OEzNN!)Hhy zubsV40N$ynRkTp&AWqu{on7SgaXlzjuC<-c=<4$a8>MJY84p$YR*E(Sd?2 zj(~r-VDlt19s17wt?EakUqy;&Y#b!(ikqbw;4h#A;|L;xBbOvmxXp%T?;SeKP|!E1xnK~RiW1vF|Y8<=rysOJjC{69r-098J{I!{R#Gaf7Q;AqmOHJw$-ldbykv$`T_hg_9W@~j2h zEm6tA=tPO!CCIi2=ONgcTb5|*hDfq%=`LC%cO*Rq$agGL!&%+_Xo9}OZPQPImBxT` z)9u@%R+N<)0p05T4cNt*nX;K$3DYI)W`UH8{LvV^R zapkvfo&?tzbeR9Gnb=KQjn1GV|pYeulgfBa>z#GA*1D6UQZV#j4yaK}zY3 z0Sb9TMsx;-Y+&7?j;L(kb_n?m%|-yJ*WN+T01JW2UZ6i5b=8m+q6V0$oTN9lUi_U2 zk4Jy&1LDh7h-Fo~pUbaY?e}89>F!-`l?My9P26;o#l`x@)|ZGwyUFGeM<$X1yrMOj zc8Y{w#A~LiqwROowtA5$&$nm>iu2EQv8gQY!t0>bg1Q@d+|nu|4UCX{3JAUko&j1t zx8a{O4x$R6ZA3eY_d9xW*KzQZn+GiiklIfeCrC*`Ps`yOCT;9nWu&ER$QXvy5Iu?c z)QP#g`1;-xVNf4Dd4h6Ni}lj77o(RThfC0(frA_rtC)AliYx%8K!s9S(8o_WB)Ayz z&ML?=fCF3r9{Fh5_ybvkykXQ$9vclSSsqRau;R;)KbonE1H;sVSK8=kO9|+QLi730 zp=`%Q`t4ZRuITRyF5bPSQU6oJ3a}d9yRph-EO4z0m9n^tn`6I}D>0M+MGgExQYyZp zK1BZ|*#JFPfmY)m@PFtkQXgLAoPkwDu7r5N0^{+Rw*}c9I+7qY_qD<`=pxEnpXzYw z35;odmKgo zrQ?rM31wxHbG7g2O+**&>6E>1t7vtA-a%ChO&dg}4ejrS-ql7QTeZrgI=V^>uCs$} zuV7t+Vm5ZC&ibxYtJ6C5B-15i1{DwksqdRAJS`|tPM(y4(K!q-Kr6m%74>v(sL6*9 zW2;2!3#QJG0KB+Jj&-PzQXuq4F(7QJ%_+oUQKadi8XG$?@OrS(Y{0-p{Kh7^6i(p_ zK_>-p2~3%sp;i`1fRa*JkP2D9B-lABl9Fgd`Z&tWV{>*Dm}(vMIb$fmhX~Q};Bs zUgO@nD;CQ;awf!p|LnX3r{VFKIL*!IY>|}$E|C#dEKA8ZMpm}GAZYYj47a?=aFGRX zYd0D`CP3~Mz)5vbQF1!>@Gmc20+)K zm?@NskC-xA-94$@Z}pVh#o_NcAhjLy%w?V zAfo2c0Ez|sk%6ZN-Phdj{mlhE?^;Cbw=kpQ(#bz!KjP(lDk9>;o#@obQDvj|Gs#y~=Q6ra+9P z3kQDM-vHC`GM-0^sJcr_vaULF+&rJ&1sFeR{hw~4aj^#s0M1#eV?++jDiS#|apCs7 z4+R~iiR^ed{a>dBEO{bduBwInjoJ>AYabD;jH;AE#~Pft#l^+FWlr+)@>(j*E3I9d zANRbsfC*pBJHX(IRYog8hwg%_N>kdLcEpnA@uicv2F8%HfhB^0PU9fJYC+m^{^GRS zM4U$7vg>g!=lo5H(4JFuK8VqH2S3Q~dna=)oncLHZz`{d4YKQ5^j1(af{rv>kQJ)qBvmS{elHVf09DJYWjHflikJyXpEOzauy$Fv^$ z!HN%C+(dBf(eg_{)k-Kbu)q!2#f0O4Xu)r)jlwWeCI7`W`*#QA9Ag2-RA4($js!HA zL}6oSu%lx`*nZHy8^7E&Z`7Q({k0`Nmp}NVn!hbJ8Fcf0(1c@1l`bj*Jdjub7(_Y( zJ-Qn}1^ovQD9~)6@#>hw^ff$shf~7VyAmS+rm6datmRCa!Zx%5a5PN30-*zq8VC0A zZvM5C@W+osLTmFrbNW#n${{fQN&CI_r$7UVIfWc3jUx`d-FFKeRkRyrGyPqgxYn#c zq2<@$D`fU;YVXI=JHdMYfdUH&ZBGkfp9q-SseHxtwNP9(% zII~etspYUp#=*~1D3LsHoa4ZuEslH}HW4PheBn#jSh!E5r6EN-u$`)}v3o(UwDH~Z z*H~d*a#a&?7`HmWFKI%@i`61z#7HMQdE_u4$B#NWI3YjPYEwPBUA|#?6{#7D%aN+a zRNopq^L}hE0owryY$mW38e~vaRn@_vaNr%w!tE?tx`0RIEd{NE*$t4||AmK};Iz4G zncI+?8~FzL>u5sFJ0&$Y4$@<$3M4!-kh?~E400{4ojfBME*ro?8A(P+Tq%do@+c7> zKs%a4&j!8n%o%xrY)eT3I@u$%1gY-tCpq*HP$SyO4L$;<4Y475Vjx`*?cSJI<&efM zu`h3#Z*R-ZbThaUTZ(-O2ou(iGjOU}UX`UIaEM z^WpU#9F?%2n}(5wYwcg^E_IuGA{D~l2o#!3=fFn6h5YmQEU^k^Ax5tR>I+?7EKWE; z0Pi2?HnpNnDU&|Q(6Aw#k5Qce^ncPqTJ`qU&&uj&2b?oTR59e8`gO`+G<8SQ-T|zY zonfUSkb+z>Ok*~K2Kv}ycM3m{6Ix6sE(7PH%ZBl#b`#KtAmQpj3CIS@#G!y6mx5kk z#-YD@fr$)%lR&DkcKUW58MvYw-oSglG(7*_zR6$JFH^A99zK8QG@^ru%Yd(t?a8zm z=$5)7;n#Wyw<^1)IT}dRm3~%jWTaLHC?JRaxfXY{S<%3BZrhTGeAmyIarIe*la7z! z1{sRL9}4`Dyz>|&ZqIGqBs3>(j1JI|6mQ5RLi{{m`*U|>+2i5*N~9JbV5W~Tv~c?{COBP zAOS2vMg!oP6D2QB`B|u`!6N_4KHr$dn`^$YCb+hEB;Yg5LHS3;3IK5k+_o(lUjE_N zI~Wr;e6s<8_;%RnN%3L*{h3Wb3t}7qq0dRm0_g|YfaXJlTg#VoiqYsNQX_K?=y+Q* z4&4qA8+{zHObVDs45(5MV7M#d=_7DO@^2oEnA!GvVcNQNEzfezv`WM1Lg?)NRCNl- zJdO6DLB&t6+E;CNT)h4gLuaR)ml|4ybS3N)Si;YCmZ^Mapy(r$)^PmfO*_;XF*_1H zdot?Dog#deIp#NLHh~VDHhXMOOUwNyZxMW+c%ICDD0&&zT@>-%%N8SmAU=#Qcp}Q|OD537LOtov$lY-n^TO0I#86Lw_ zntz$d+KmYhcVx_7#QWOT!7Zt^wgVJ0FjEOl(g@mCl$!ZQ> zCwAijDjsmz3WV%3bCHYed!MO9987@V1zkVkDPRY)jfpv+5v|8G3xP*t%fvrA)dIEh z?9mpcpzS%MD}QI#V@1Hhj@jaKbANrcf3v5fa^2+5>VC8=Zte|bFvTrZqyZG*-P-i9 zsDcsh(&}Q?&&tc=f#Xq`f-a9dh-T;oXo46A-p&FdYUQ@V+cgxR{8`eC0IWax>1XQD zk)Ar2$2nuDq0u0=q04(Vs#T?bv^9pUVTJ|#0m`ykaNqo$fjVAy7b*c+&FhRTXippeuM?kV*H zg9LA)wGe{QB`6TYZd9WsjF8K86kwqPytQNz~l3?_d(t95zZ{5NAMF)VP-9zhFY(F_*Yc-*g z^EtQ>L9+!HDvbK6|3wk>7_bP^ShcFqp+^MxE^zt@2tgN4a@;L=^2Gl2Rx6*^4K?q_ zJ7~NIe%YjT7MkTWzF6J(F?J(n5w0~A_#bIkZ? zVNG<|uZ-kwPFLvZ-+aLh_`;MV8VCafd?0NAVHL8`InvUf(gyC{y_@Y6*Mr7tad|oG zycye@hS>qwYFc>X9HB=gJq@ze{mDL=s)Fb;av#)(8nX} z6!ccSH>#9qoC~J>IJYeVQlhyx0&f#N5C(#AQ4IzbM@2gnPOpFGHTEv0#IBDMNPFse z@_ipk7%~E8AXYUrvQ#?2<_c!?AkNqna`wKAo~=A1PTy|~FEX6CsAeM3DnM&RQ&V8& z?jx|&;sSkAcDf|>++%Rao*}b0Y9>1bK5Fz8qG}oTQWKAx#{p1ev1vYdb4 z*{|KHLlo%nSF9}kV8tW(nakN9*0Ic4oi0(C{5R{PvPqA7-jaXe8g8IP6rmwSP9cEi#(RiM|fJzB|0xJu<@PElCV&`(>cFo52S|T&W zYR8BsOoKSjhi1|V~Dz;N7k*z?NYZwf5)m?! zALs7Gp1gww;PaHjE?`btusrFHw@thPG2N+g+BlNA>7(fPZ<5I*nMV_`>Et}M9^(Ez z!|cqZSkUg%Y=YC@u1y2X0wx`i1H(V!)wMn9ihSR!?Ohj1PZ^J@;H>buOOrAVLEQ!; zl9_i8KZ-q0;}|d|q8vyFkuR{CadR!rISSP|pOf?7SZS?gKfjPD`oUrmc;5{*7EjME zznHXa+1Gn7&t_l~?gKMnO6Nh5jCC7zMpEXCepBokx*hTs;u1oMh#r89AtN}kBG3mD zwv2O%nxj`a1`>)i7;MwoAlTWc?^6)am_0l!i#1-n8*6;t!RN9)aKS3L0dphup|p31 zcJrnkw6e7Hv>O@kJf+-ov4c#BNBu;AjAQ(?cH`jxVajSZmEHwqs(d-?^>z8jnE^jj z@JYNRhqo0mDsBK*(tp#}5#eSuZH-bB4imrI0sZ~Qlg1w0)U2%G8IY2ehR2vLN3_>d zdvh>|eG8x&^n-R`QPa#u^bts;4u)B*PDk4{qEloth3>SaN9P|@v&-0F^X85^)P%@q z5!vAo^J1LQSTq1`c%=&B^)!`H){>5z7&OZ?HN}6jqx!>82*)t3`xt#%qtfRhE`=Rrd}p zgq9ANet&yXz>oQY=>0Sm1xkXgiU8+;S_-_!|E-`43_95qg_pC#6TNPrTt)W7x@189 zdoyZqx~ITkcGt^acxFC@P)N!s9`qu(!~p>2klvW&E4Cvzl50ms-m?pU&pSpGtkYx= zAwdKKvIdz$J`kgQP!f!bHk4f1i;gndmoFy}Ht~|QT5epgD@KKX%O2?+*nV%`v=;O( z-$0`f6T0RhuDT*^&EVo+LYU)B!|Y<-`5YR2pX=@EcVYC6SB}U~XUtnW;EH2QL#I=g zn|}W$GM)IFe5&H2Iq<9Fm2t&mmD3`lEK>i}wq2XPG7Dg6*N97ohg*Vi@|N?aIP5E2 zfzlUT{Rn5skdTlG8pz{Mot2p-+XG+wU zD44%lWBq!2j{^RJkE|a!WYUhJnrG?&8(JVBj2zf2(v}Gq{X)$3DJ11(@<73%FO!BH z_I2*Un&%dVIKlVclCek(!q3Yi?#-a!oQ>{6Ffmuiq8`B`3d*<>PwS_;n*9+M6H}lnGPS;an1_JWz zd|#u;o$R_kDNYmYRba`ypzlHv*_p0i6iw%>+6dQ zLVk?A$>{D#C~+ET`u>#Z#15te!PsY4t7SJfoS}mQ+BIyU{J0wv^+tBs+V--3>FeHKdXAc@Kz}A2B?1bVdXMz7Qv>|{ z6Oxixh~xhK$oIoPKTJ3RmXWD@`}MFP(D3bk^}R`9l+b0b=dX^LaC+wS;RPOB6B75V zbN#hdYdqQP@AvE$O6xH_8R*B%am8+D(Mn|&>G|_ND575BGI=db5CHfz{vf>1K4h#MfuP9NYfT*IQ@Z#S`Cs-`xLP9DQ$3Nry$cOOKt= z*y=s99JNNBi=UU>z!N?>v!fPb!sr7+ZoA6Bz<}^2Bp&f-5eLbF16TZveMA@nve-D% z^N9;zwJohN+QDUP(z%Z&9l`!)p474E5mk=`f}A6L;#GH@Bu(66MkIkyj_h1J0cc^Cd3>?J{oQOH zT>}H^!@Mj!52(uRk4WdY2~+03xRAq+m8o%GK!!8hcb@5vZTEqCWe<0SMweweLv+4% zKV~$Pltvyo)Z6>c&M(l|IN#1oNKMh&Xj!dMz>lx*LII~TKz+ah?cL`uLWcp417ih9 zBdFxK`*EKJYyw{@yaNu~9s^GdpNN@)%1bi!`X06Vem{YgrWunw^3UD;&|N;ftk!*h zZ%>-MB~qVd5|xViLXfO=opRcz8y(Wq!I)h5um;(<_>mZmc|UZg zaBbLIEwg4VdwJ$mK0Z^wp}Tjd=tgr}6rOk(uD~HH|IkbQo;CllU$?MJ&XJxY!-u1e zD4xHNduC~#zULIvFpM`>wBs(Hw5fz{?D%mehFQp?3q2M~9ISx(^Ln%pK9%~>oC!%K zrRhhJ3*Q-f+_U$5G*4fj!Inwaudn|8G;FD#`q0z+Wn+oakDs@<@I-QhGWQE5kXM=NMf3D_gcJFU=wz%-qAZH zYzzE5W|U4g^_5XnoTCYc+yYv8id?btdpy-dUhj)txUR??UP_UM+vBz-by1*}oy%t}4alzPkq;W?HkRibB+wTGc9*KgI zRGsU7l^sq$wXGYv;BMZcQr}k=q~5#t`N!R9bpT8Kr z%W{sM?xjY1)1!rB?9FyUnYdX zGw3K#(*RBI3ee7r1r)6GT-nk!W@p|hykI;Qm<}A=Ul^x8GX2AchNeLCu4a3=YuX&H zz2+!II}U1VBZySQ>=d?U0%$bTy1!ZJ(@E$WI&bbAx?wqjS4@uFo?_OQX;jFL9b7qJ zmAL)T)Mcn2vUT-uXelu-;w~Aw2Bxv5tedPSFNX{pboIXhD_THc8?3 z7GBd0bEdM7c%2!|LXT)2v?`3#$IMiCShcjEz?1V>cF&e;;8*UoK;Lqg&p2&`xK|vx z@ZuW4*gQsXpg&g`Y}&lJ^8S9~VN9Q4um?2S{dwoeTYc??8@6m&1DidPjJANBq~fWb z;(16hUSQ{m39M?%Tc3~TL&44_aFYo^#cN};0Qvz7wA64&!KUgBA9$&3d<{L>4@VE( z!cV3t?R!}aO1`eR}2(_Vd47b0QXkmH9E&`Z_i&E zMXo3QM2PJ7-8J)geody*Olzh!`pF@(#BKd}>uwp78L&e5kG##QW_lT~pUei1N5#?l zm5yv_ak}Q9Z9lgRSIqsKtOaqY^@THIp)5Go!VF0^r;y_58#kmS^dba>W8!$!j;@no z^9ytT$)0@GJb`vaeXzWua2ZG~Pw(B9&1BfO?_k#+=vFU6WO?J}O-AC~3o6=N8VWQ- zn)LxmizP~b2ve3w-N_xVEp#d!Hg4Ot7Gf>bHBE8hwqlrq(WU*ho=0FOCn#L45qw#l z_n}5cEQ_@(ogU5|pD=Im{+@N{A}J3nhWjn`{rCSWIz2rf43RHhQ`vnq2eU_dMyxe; z>j&QV^cKi^Hs*};G!2(`fCUkPP)GxhL7Xp-29i@OuxTrsu?a7?LZqj1bIWw%zf^9p zNNKm%Q<^*Y^Wyf|YFT>ORm7VIP$GPkN3HkhJKK!XVaz$XYq`O|fI$QO8`2eU%~)b@ zHoor?&%V2B#RI_93M-qKYRC^Dh!A*lRZ~<@Y0AZg-mhnvta77Rj;>+cH;9k$ z@2p7ND9O&j0pHXE%^Vq@WzxQ@?X4Az^2*Xn_Jrsmwiu=hTD8tEbyd5H=P(fQ2~uV! z;0!8qyuZL;Z~&~gVcRzL(Jsk`4ZNT=I@D$|j zHKW-muC6X}6bkD(fvTF0j&~-EL1*GR2WtVoUIO0?j5$bFjzU7Mos|I6fBNSu=`#?_ zZPRjB(Hpt6s3x`?Q+y`qFaJ3esu~6&Xu#yeaL10X4KuCieS@T*^d=ha5{Pxz|M&*f ze$x?j5u?#kf}D-7@2Y=5V^>LI4kK{PtP;7Kk}~ zd=|65xn2VO3f&Y*K~5$*3xVo2_m1}}422(pg$BuY5r9R~D+{W7dNc<}%le(4I&Z#r zj8@}Q$=rHl{B%27IkEXb-llr>zl^s21=*`52(EA&v9~`PLmD`As>2Lk50o=jK*kAFaNeO>is5 z**d#+xi$x_1`97iiKc;hc7;Tzpx?yWHg6Jk0Q#o8ckhNBIa|(js|XlvGU6mYbo8HH zax)=;`WNTVpQm{eXo2h#8DO1WPE)`;#hHEV=UyANRHfoyjVS5&BL`c z74-=^$!<8Wgs<53P54X}0Vmk_dUx+BCPf7ROovJxNu_a-La>n^v-b*^rp#NgKpc%T zP$%I@R8lOUxNizHxNK}4KVDYczgbE#%>h14s~3#9pn%Ta1KThLSAZ(;Cmy9{9L+jL z^coVM0mFMW^YhDi9v>f{-$ki-;w&1%i1SPgsRSP9_Vji1dC;=Q>jV1&Y%j8V;XXJ1 z_+B|c_C^{8Y;aWn?K8OUia~ODZMEOSk>1sUsx`EIq;@gI=)cYi&WaN-~O4SXYixViBBsh-u3(zVr;2pF`6NA~vu9i9ehBV)E>5 z54qic9zQaWx9-Kz$EAdL>VRFMQ~zvasD8u&4F9{Jl80GsmLf4H<0!N?wpKpN&$KQ2HBHqt+T@giu@ z&u|c@wa0}QmzACM*Z`WHyD~Vr5iYZCZDAa*)wG$tb(L?B*M(>ix6c_1$uU^@|+64Lw{M(?; z@#Cg`h|Y!rFyj;PechlMMSxfE^;{F3Z^+XuF>?phi|cxW2K{gg!@$4Y?;&E;DvPww zVZHKi@oF&groKA4mPbn{u;HMv!o9PWBZU%%LM#>JzkLi;OpIDj>BtGMG0g zA4FRbZ>Fod$}O$6pMcrL5^8E$n;$>G^X`B~&1dy2w-e~s3p*yk!MqkDZwKLZP5h}W z-rwOZitYbM0+#pW$a*qz&CoENHP~A;cH>Y)o(+dmnWR<@IHot6m<&GZmW;4xB2FR1 zB(b11ZV+;GMkNu<87Iw|tZcxN}(-I|6JU0w{s_OMh4 zlyOVb`mVwz=QQV}i4BSI_)PM13ch+VS3=kYSRX2PL+b*PDJ zqmT2+hV=Jj6@x%WD%L?5&kyCv@eyyW?@hU9A@;^P07N$Teg`KD2sL3dXDbS*l>pIr z{b-zuyO>QJ*=DLv^6H&%TF7d2BfsKQl)||m;D${^XE~DK{>snP4aAnE0%RF!qiY|@ zBa$rt^m8VB8u4$iHDodM=+q{g9jcy@NoRNTgC@C{2yi?idN{(MU{~CPMMZx_=S}P#`abqJqH-m)eGKM3!SkZaA$#{*x4z65al&x|XBsOK-yxUlEpy2t z$5GN&o#4}(4Mq=kJGoB5E<6JuGx44Scph0x z0w22%4Fq*VYAUs0L2OFOIc&4lx1wNvRJOdw1TIM{;6VV}n$&frn%l6V~YZJgoOMeLP!TWX6(F&c+RXyq`}vJgKS%vNHanpuZTv>eK%?R>ncN~Wo5}4^Uj^cpnevD=V!fn>d8w( z1C1X7Ftw&3!pZEFG*m{k853V4@!po0p5E_NQhXsav>SelWcf<-4jAZqCG9$~q%p#E zHA;qkXgJa0 z@9aTCk(lGISLyUOybY6r@iimfE_5Ry;RRS}MuGxnHLO<(b}X^2<%DVl}i z1ZY*d^C((%vOOn~TK7yo0P5&VOTbjLX1 z;S99~sS;44K#vVP%P02Kv37Ps0c&D^j7m?#+c;+bxy1d?o;{=D0Tg0b=B>ki1oHP! z9U$@J^XM)!y2-2=V^D1VJ9p4grm6U0oxvfc1V%w{W?VbcTj@b zrngIgT!9+{ci2*J)YZf$0=ai3d1S4c5sX*q_31@?u-*h zAoOPbqyGcq7{c93HSpcv9cVPzxJzv6R1nkg+9LmaZS^Nm1)LV7l$bn=VLgg;U{zL% zPKa5*^O`kBYfakTz8c*QWq&RaJKOA`Lm=VP5JL7R9y~4sS?f=OX6U%;+K@cu~$Rab`13(lBDv2}ziePy>jT^}L z>Kp?`zvuB}r1_$LD}KMWS1xFmUd2g8W$ZX4w>m4~j1c$&5A6J&ufL=;TGb`K|JOAp z_|xMLA9jIsOCyNT_OV1Mh?K8n8t|LaaPu67Et#O9k0OSjxc*&@iKg@a|DXb&##1)` zol3o$|L?0de0=}^2noWflZFQJyIMernmP?G{G&(nPrcJxL@A-p(tc$XNTY9k>-H()>&|ooU!Dn4@V=b`e=4~ zqx#QZqRFfZEk#)}PiunI0MTSY&-%;Mksl7t}#rddyA08^IW2VX8Zd$VO#OU#dt)|2h>5vjjF zU;LFDRKl%jpzp$ckijZup%n}KEK@lb%(VSsUPoi&+%_i0(Z=skz(2TGuu=Slf3js% znX}Q_(}Mq0by(vtgho zzd#^=ooudBE6!Qi5mD3+9zOHOS4w_Gj~bK+e}Dfr)O=tegQqJIwD88ADKZ(HfTUjP z``zE)_=Jxr_1OJ?gngrGqs^Na?=GH9zY84}oP|z#BuGs?9yABtqu00dOUg%qv*4`r ze|%y5{}Bv;ou(FH)(af?z>%hCpyLNm6uMyg`%22C)`1m_=q3AKp9VLUeDm-~E3{B^ zOcvqc5XfT|7>&p6^Z$sD2jM6nPwv{Jg^X5@7J-2_w%dDEL3tJ?#!E_k{Xg$g(NoEL z(rKJ}gy6M<##EprTnBjvyfi^pS~Us>^?!~59Q0UV91g+}F&WSzxA*uZ>f9(rGff+= zV6YZpARRq8J2=$}3P1;d4oJ>=>hnvZE>f|{xqs}KS4aqTFv%T~W5BXuh(CxTU5ZFK^3 zQAvN_XRuLj()8&F|Atc*?D&7}efdAt`}h4Rg%G8KB56P5Hqu#^4obwu<&;8tc?X}l( zWWm7QqC>XndYS)`9h^eyitu0M0=f?<*?(COzWhTka~7r!kY>W)X6ne1BX1uIX1tCw zQ1R82mez4m^7zg|7UEn{OZ<8MY8YNWhe;!FjKQlfQ4=4&_A{-`TmLsF0G0y) z18U=o?j2k-QgLt4Dk`L6D-it2lFB{ij8oLRs_ikkyog`Rd)IZ3`IO z9D|PRte~@+3lhi&xRE*bzUN^@eNDmP(Z+K1Du1qWS){5hcY3uOKY{_#o6-5#Xw`pt zew@{Qr7_U{+CL+eH5O2^Q@G*py)BCbTMjoRGlo|MI<~A%)G&=2-smwa&$9&c3^a>| zTwW+x=OUmcYn7LmSNjk2(R>1{Lxi0yI0>MgP4!#}D%&NuD(Q2i*1_~l_>U50u&aaK z{W-1qnM@*;#(jJ?WQl~fwz`p}z>FETz}z)&44y;LwO)gKS)=_ zO?sv!_jSvVvr+SZE@nk5eF#jjcMf)4^?wx#<79Jek1YaF_F^`yt|&VTj|mv^OB-l9 z40>N+&SRh|9-itbDx*W*XZ1|ho`itPuf)nGOi%H$@b*6$&(V0G^I!*Js@bk4b1x<} z=CIk0Nd0%MR;yr!<<)pbCe7pd@e);O`E zVVG{w-^%Ccc`y>)U+atAgtrtLY0Ds91ejham|r>YnD&0?K|dB?EfD=Fe^2)`)!q0!qVkpV;{gs|HSo#iwbcCbdzI_Tzw3Dcs;L)?? z(A%3_=-i5)t zf$;KS1w&Sd1JJ=OFZ#u&4srn?^2HPpAzgc@wF&>%iE(A*(C?ngGqXeEe9Ktvis7WQtcKNO+HpO!XRT@-S3 zleAa|g4rQRXq9;bilCN|R!lul5#syKXYz>z@PmcFeK0d9_x@X9;M(+Q`p&v6Tt>yq zV~_V2GY))haA)NOy~_?Am~tjxc&f9&juEqJ1O4Ep{Y!DcK0-YYQK#vG>6QUtYFk;- zYmJm-w25de8y&*q;o-R$Z(NB#tcO!T>&5j~?=8Odx;2 z-x!22qFe&df>!ii(2SuvH5+!C-AnGML3@Cp8j^&;`A9~vD7|b;@rdY8xdi&^WIu`x zrV7MllHcz-SD)EUCb#VP+rLfd*R)|Q_&Fj!pVe_nKI8USRzDsjV9ZmyF<_&i4U>a!CVGprOyC7xCs?wy%Z?S3uX7bYbJRF$D@FF4}ogv^HI|^qYh-g-kq2` zrg`@m%9~ZUrD+J0Ta5I$rE)Q-A3##y9Vw2<9&b~{F?5dKE!GyEDw@7xT`#k zlwU7qzallrkv3rVK(Vo$n<_W_xpM1&7L-GYO7r*pn8!-EOeIk)n4nEKPpt~OP%EYo z!#NR^mF5f_$OUB85nln9R)!zfo7jVR#mtw5YPG|wp*Dqw9vY&_xj2)Hv{P(abOWp^rrHwVo;7>ky?wrGNnh# zxBYec6%94DJNDlAnL*&w^dex=0?@XWEn7AjxD8-b)EM4B_9d^5Uc_a7gW=c0#g0`| zxFeZsz;87I4oT!QFkDx4JuAc{d_~|%JQ{%3ArJj9MH2E2W?$WwkR!e{(xbaLUUG+60#Qt-1E9Z zC*4BFO0NSmczY|q4s`ktoIQ8WsJ0tj;xFk}=BcXQw4aa(@r>1L*4RzsbR)_q49Sfl z-uRw{g;(#H7O1PTch-iz@$B!KOy7lFr&fg)&s@B(^>?nwnwP<$tq79ac~((TQATpi z;Y44S>T6tDd*!pvWaHIND&w~Ru2(l5yfqTa0nWMv@c@7ur-hcs{Xf&Kaue2b{9d*0 z);sK_)J$zA-%Y??c!KEj(azVI+!9{_*)sn`oR!!cVECXDLK9`R_kWw_n>0Q z;IjO+e8q|h5H_LNYlya@Xa;8hguwp3P0d#0wI}*@&S$^CP4vau7Quo^=YyJd5PnZ> zy2%SHOV%msX&|MHc)wa@JaKs!D(up5D1Pr$uE<`H2Ctp*=^Nah%3w_M2#)VIP=h9Q zB$Ryo019UK3Y)KM8_W48`7)<&q_QE-R-tH$F_cl`3|@8s)yr<}MvF!=%|FAY)YA1K z^?RJ`oxVUFKy0JOYm;Pe6r9$-#n}NzI{7YzDQum1+&+GLHfrDxAEx%!jsl7he>S(u zH4i94sZrimm$^hFHPe`9dot!3G>FY^`AR=Z0AZx!#qm?fKQV3jPPuPUFZx&sX=&j) zlSwP+9C&78R+bhPU3b3rOII?1qoeuZ9LN54VBq-4DSE-i$Tkr8EYh|Z)VqGZMoDGB zS*!g3Au<m35^#LQ7u-gFBHt3_o0rU|7ai!V9<2ZiEMYy;}6f;hoc0%a5*_S=p3kH z9uk%lRm;RiPFtFrW>lpLj8h>TZw}Tq-Pr>od_e77hGM9dI+tJfogY}wS`2} z!~lrN{RG)P3R&i_8BR4Y!HhQM4WPa8XQN;D_^;yVKeFX7W3l@*u@1)4ZfAhZkB9Q$ z^9DJl=9Z55SKZ%T{hzz~_f#qOWkFO<4Xl>W=sNW0wnc5+pK||x1WvLapH_nHb z>dFn(i6;+9A}2&LGcZGM70yI5S+VZv!bmO&F+)dx3kJkP9(tl$qJM=KrF5@e7!qlY z9GfTUa`7RMFl4f#kOgYB>R-2#6rAPTt6J@?T)j7u9scCc#;h(wo$(19zX1Zz^B{0W z=Fg;6uGWAl#$^!FIr^rpp`n2b#=);rGd)2qP#bm?Y|6?(x0^dla4LZ8Fb~}+8-(@O zV;xaRJXaP9iCA)E7J;+U; zj1$BUp;uN?^2_`%@>wPX2#_%lvl9LCk5J$GY`(RHbp42SF30=;X4V4dq~w^G{Z1+O zf{~Jwr!P6(RnFmT#5=j9R1NA0z%HVqz`OcMNCc2BZC|vQ0Bd}aXC(oUGm!U+PfV;B z5sbNp>>XGrQl$-mLSs?DS9?}%M_DM@@fJ;NYN;aaB56rvgXx>tr%8B>6&v2FhsV1# zL~zM=4(SPY`~hwvIJwQ5X(ez(oHXR54M56h+)WuZ=8a#20)bUD@uwH+t)*BX2ho4% z@9k=DuC0~j+(nodRvlZLcq*nQ%pr17@A_7NdsCya`EB0taENZVEHR}YCh_n)xp)iu zQ22r3l!?hJbZAM8X!<3z?oP0Srr8V5$rZ16e$gaBuAVxCD|z zxd=YVKow9Jk@YB=S*uM<7NPz^)Q!gp31Hb4Jd7Y8yk zdq5}~N&xEE@n7zLDs2B`J)c37ndZbdvWpi>oL?Tc>Up~1LFvA3QN`MmVIOZ7%de6N zdN%7!_~u~aaK%+ws;8%>2^`(7KD(m%*o<|7S&HYJQubE6rr+BbwcWm^xuU1-o=f`q zsy44|8G~#H6Hk4M#YeLizw!4RcoUewq+G0UV9}@%bJ4PPfPMv%HH@^Jht5{P;N38H z;uYwkVte}J$z|O4%1aB*XJqJF@YLjsQRinYpGDrj0jH$O6s;Pe96V&3p=T zz_3XZ4FOefFvjSngH{JE3TUv)lY%DN>}Vv3xx{+@2&~_N*+_AXUL`B!3!bXL!26zSZ%mbL5dEa&u^te8EnVJ7#&VpomgbDQPu_5Qz!ESk^32VnHFI9 zjIClq-=r*QES+|^9H2RQ!`SmOfS|CTUxPslEi(gz7Oh3NAc=~kH>lpny zyPs$af*_?`e;;KFX1sF|E?0meH_?W(8w>SS#>auDNL>O<`l~|z{R`wbGma?81vsVb z1DV0G+@{|rY0dLTv`*WvPH;AjCe2MyzHHH73)$%S17%hm7C8X|5Urd4;cfQrW-HY5 z1z%IMS7DVy_b_AwF@N2`^(qB}G1oeqGlXMm1})VdstwV9k%9a(vKdjoSTsa}M;~ze zWK%S8Zs!Ib%8yH25x@$|L4DZM7##v;NzIf7=&Tum;DTOa9auk#vwW#%MR~;$jT#c3 z>g_$Di3xWiwcmrUF1njPWjSpJ53|uy+80Pr4>S)1m1k7E{XXK=Y5?(_<);F+m0sR% z$@yYnj)nS5mT)7ln+)1U(UeC~xI4mAh|4m0j9f|IhKF8ikg*s}+aIh#o{Lwlv$a|C zJsaE(Y6P-_Yh3JR;$Q(+0r6`_YNjr(oiE!2?f7>Opym1T@uG?pL23)c)v{<@I7May zw4h1^JmueeU;&+h()%8s7!)r0nY;Rs+RY{D zOBysKYfS8$m@ICh2;$s8nspA@WXQ3?!ot$4>y!y~z}C#PVBf5CpDZ>0K*yl5Akr@a zfiSv7;?5qTJAA7d-PaRxCYE)aEX)CS>pt?P9Z0ajORGO{R0CcNIM&=ycQPMDklO-* zEcgUieDt>bL?YJ)vo=~&-XA$&d^by-6%ofG-9-}HjSezE*SXuvAUp;o_-*4nzH(!h z?@Y|!FTDa$iXG1t)5%b)Eu0OaBOs(al_|;OEEOfUwisH~57?pCH%*2oQ%KoB+9Lh!RgEs46LQLB_XP z=Tx~M0Eswta+TW+dM1);s%Lck^9flhiF>*zIstc*hLTv%a3BO9d*aL)np(?Ut?!EJ zlkNkfj2yvlY58lUpo-2m`t@vy#{`=(E)Uh1jtJ+93o%*9x+Cny>?&+rRIh}H@g(c{VS3= zgG3Tu+JkSi9v_o3N;Zw3$q6-P0!Jp!3%7K+7las02qZ9Mgc@h`oz?C{=)~QWnLz`EkMF(p00imSH&cirsYzomre*~B)b7Ax4XPYv#-%2v zPywO04{ocJwmM`R$(Y>5C1p@KKkOj(GK-gtCIexaPk7aU_f%})>goaqiOmR>q{_g0 zJMcG_1wkK5TXHeoDy+F=l5e{S%oR?w_@M|n2WAjxKDRlfUQ_0CqzM>vfz0W70t&p^ z9bmkJWerE}H^N}PV0ySNk8=YTpcSq_ArJRyv?m++9Lm?z$P5m-3uyJWk`}AHbrKn?5aR62k9~K5$~W{su4}#%p6#_1oIpyLP=6VQrO1V2D)I8=oIjsl23I*6<3ZCl zv#+D-#83aqeOwrggVF$$i@JZY*I7`m%}K$ZT@6ufu+ItXBvB-63CB%SVIOygl0ur+ z!O$>umH0VtTx1cF*F*>fg7;%o5gk90#iI|D%LF;>}_H$OoTG z3ghT_kPJ_`Z$Ck86)Ef!2Iu;PUH@3r2#lqIm!yS(J~E1t1d;*a;QQA01asHHf6VMlZd}LFHv)snFtPR3c6;PMFeED(!nl~huoQ9u|3S;uZ zOja4mnfRUAZL89Zowk$PlT*Y!qLsF834rf6jLg;e_p-{rdU~HU&6u! z)Ur?XEZ@5C>CqeJL88;KoU^*54;bd)741PO$Jo;pf0ksJLJy8Qod7Redd-abPoILe zs)vcoDIN@~EtUss$pKhq$fVlP)B;x1c^f`F4H9J(KpP(=dPy1D&0LSMfV_pj$aAbm}$4 zUf5Oe24F=4*|y^-$j0g3fHWrg)gW*cqWXRTOYAkjk3?umxMYsqzi9ld5%)cJvG;K1 zW+`Vc$k8$~Kwl;maUo4QKai^I89@3z9Uz}$H>T#5#|m%{BlHDr<+g)zAIQQg?`SF~ zXndqz2u?UM8Kp`DT59~+CWqV#r&tbh0mSk?oz-(uf&l14@A)hMH!M)+#`x({c*3U@ z)=Y)11mq~w01T0-2j;L)Ys-Q&j1W5}05De?l(b<~CNARp>gCH%K&9<*rsz1VuV0~b zFMoOrON1TiE9~ba1K{}-3O5T&j<|f&+{!dNe&irBxJm#RG*^SC$d_(n?LQiFcQEg| zZs~v__#D=bHhASEl);VpP{5`#qVqX&>;?8L{N1cd_wBs8H}7ZQOv2SG_)Voai~%cjmK zn1=l_^`LNCc6Hq`1Ua|^;alzP?aLIKpvIfg(m|a|uzNT)LgR~-4G=NY2cW^1Jt)mUIyJoq&&eiPzXn)OCS`2 z?=+M2_hH)7{jt{Hd%_$CA^nuB_FEG!yQS@Jbq5`F3E-A+^ytjS6J8v zT%WLGWE5ONuu9DxH9e;+8S@wHlw#_PlnhUwfmt3X7-Er~g*>zezk=s`;p4|k*3=He zJ%3+4CI}c9fMjx8pdbJjMZDCR9LqJ2Q<+F5pnej9k!C`f*r*6fl}lcHC{=PtfE>UI zD`s&bJvOTW?;qXs5;JUhRbw+tN%Icmx&f8Luuu!~PfYVc(* zIHilq9+mcQP46l34C5=9Re2j-KQ#3P#|U=yR+G=BmbW?sG!eA(mg?7vft*^Sje)8| z?My5JEmY3 z2*)&$pYFQ66F=>%y~9UnB0PdwA=6}YpBH%Nm06NX*%L2$7qP!kp+LLxR7AvbU>Fv# znxJ|SLI~uXgu&}^P+hneNa?s^x8M6~T3avfT6=SRGnto=u1){6j>>oe(L_Np zCi6I|Xar1_y99czn^Nc31&(70P@n~_s`~omz?BX6W}$^O%m(EHD*S0z3cbPPAO;bR zX9{|=(Gs|_E_D?enN}!+OXH45@Ap?+w>%4eAOq5$7S6exh-qr}{250Xz_9cnKor%Yh!aSYFXJ z1cd=w74Ypcfaqh4*EM%Hw+xJt0-lb=%Ana|Fpapzup8415Svgg=-Z9{f|Xm-kBjsv z?)UnzQzR3v8s7>|n?+<@wPxnR9Mfz+UqWpVlR*}&HlEqtk8?LL9eqLZHPP4e zCd_k_requiHC;2b;C~$VV(*cdyP`b773$mH*3aj#fXvE%{TMxNy6)zw=wYfktsh5= z#^h?{$Ve!n4WolP?43<>)b&xiD@yc1jOzG_>W5JbB+PNr=VLuk+XaJ;0ZF8`m^4A2 zqiy>>Lh6%VZ8ER;+%cE@CkYGNv~EeV<}7o^0!R7#pyT4)<)WTFJ{-`GV&+AhX63$y zMn+rlzF-=)6Re#{ZvD7?TSPAgwAJ?^5HGQl6r9+Aq^I^LYh{PW#eDg6c2n`>&?7;r zwwL$1J5iLB{~~gALL{9kSWbAzpn;}Uj`joza!(IN3q!wTfde**_vMRU2xsIg2Ds0D z`6)U0kw?9NZ^sDu*+F+GDiO(_iA4xkVOP#i&J?IwAbfgvSC4%=NnJp5_gxn77Z$Wx z#QDnlI|^7W^_XXl-&_z@;Bo`+KDP>36uu7NNQjPueF`Nl0LJxFaEK5snM5GAOJcd~ zt^D%^i+#G~H@J_6xZC=$A1M#69mI{sZH_e`83}8U7obF8C`bOxfs1F)iq#xFcI?<& zzkZA+BcT+4}45m5fVe<2fwH;RH-}_8Ts|k%RaSb3#=M$Y!w1Uc|mKa8nyY% zuH}oQ3yM`3kJE8>cHZL+0K=r}6_Nb~xIv_iL4+nRHJVc=@;3Toe?wDWzdJCqn(2gd`#_QPk?cB%OTQS@hae#{Y4qB>O} zc=OCfKX=;Pz_iW`$DVmeDoEWucbXL};a|3NVAsXo6-8lM_nqr(JZFzFqut!=u3WWw zGYh?y1j7R*@z)Qr9`;k4)v7FmcuyolW*sC3P0#oDC!10vdKAx991DHWfUWxuXi{o8 zd%eqti!fmlhTekl_4SQ+Ol7{(T1>7~sBy2bw}T4%@lzvsW|n%I&zlFGUc85uD;>IP z-HYT+-(B~}k;>uWwu*U*c1EtR7TRcn-Y*eQ;gGO>&j2v!BX2Exb9ScGFVx|z`zQaT zD?^v))C``|M7`_l|Dow{)0=BEA%=L1$JA8UU*L?==;;?2d0dp4+FE4v5dixP_Kt$J0*6o@bU>e6NWcYn~J1_3m9XLST-nh*vdqzg~clzr^d3JSJRzB!V{HW$MsmmLp z%P{wA*Ah|Caceb-3O}zcdEZlU);mbY4oAAiMQTAN^PI>&-0!CkRG9|a77lRHG9w=? z4`!;-Ksi9zbf>N+CCw6;_n zlM(`sd)K=&_}AAdT3WX!ZY)iR{wU?g>)s%GUYKl>>b_HR_Db+>9M(|RU{E~+Oy0e? zi_khuC}(wjmS9l8MWqqj;VHy@fal0gU_ShbQ{KGED&+trOJ{kjgMc>TfXbF-C%RRx z(~FXHQY}a+Os^GexG2Uq2P*qQA(k;;L!_XuFyppw!^M`)?2t1lT!>!$O1yoCbuKR( z8HskAM2!_}smw-y@3(7YAo*B@e9|1`Xe0nz!4;YVEhS``48l7SjG~}lzqzdD3z|5R z-u@TP&x9NYpNPW5DC-Tzj6-4}2}M8Nie&xH-Myau=$dwjDuccef{Do5h3&&weubvy z8*YC4)&J?)X4~jpU-PViaJ57Y=#*bUdnn`*FcagXh5 zPFXGQ{=!PwSSe=pby$*jFp3qydnw5INEpH-drlRAa6$1-TTCGC8SvfHtc97VyDp;@ z-_3QGo#Wg%pY^2BN_0noBsm{#`r_*nz299~OFG;y;Ww64o0R$8nRBkY`e{o~m-2$j zKLnM6?+!Hh3yxN@vwPIu6pbS39$x4VjRljpxY@MGh`;Doy#bQ&LOjmr=v6(!MO5B# z5}O|7GmGY%q4yoTUDYDW50is+-;wY)H}<#e9}bd4jjx#fivYmMO?v~PbPfIcZ#Ac} z*L=yV?eF*`H3R#_;1Z17%;>5BE8gXXExoa=)8)x?{vgHAMSb!CV|jI+p9!&AxJkFM zJAZLwOSsFo&UHj3)kp4h z$S|O38I&qM<^g8#%&<5w2+djc-uKwz-I)uv+?R5YPcVcbv)o*aUajWgMnQGZ*J0Ke zT6qBbZ)>2K!WP-(OFMy{(+3x<}u!757Z8Fb|81 ztS?CkO)`z=0|(3wwI_HiH?b24wWtr#R8$-tfK=N3@r13+9-jHA7GRnZ*y&#}UW}Pj z5)W15=+yT1V)ynH&&~?He7}X&oS};{vzp}5G(5^@c;ZFT^%E|yoWKf;2200We3StF zvHM@ah>t}xP z;Z;W@dWsa-TcK|Qm1yGKHa4cF4)*OQSEVV@n{Mad9Fwwi@80NbKAj;Eu-JKXYZJ`s zf0ae`2fTO{QeUr^Jy2>iW&Zp*{HrnBBJ?8fX_27f&UGV@^eJR&L4P1=d2}sq;@-zPLy9Q=z#IHbOZsA2CZoR^eZ~IcS_Cw9G+WnB(`DbU9jR4HT4NDzeI83Iy zRc%ifX^M_Qivg6>SnP{}{^)*q(06?}q|BI$V!)yF${Gi+ubnmUzOg9?D_&{1RPRk% zw9$DP+2Bh&?00Iw6FGZ&dvuk~JaQdN|UC z@giFCa=;uQr(N7{FuiY%-Z&!joN6y!sb*UUP<>+WZ0&B|^>+mnH>KMyb+Au}dj3l2 z`o2q%5^tmCj7rIfd2-qbX%oWj5xhy%CGfTaTR~HMto{ZNd~cVovcpvyeydLMgeC2{Y137l(`xilQyIov@AJT{QPjBp;c5S`L zec9uKPob!vpaUSZs77*>ij7 zd>gxmTwF`JxN{=Zq8Us4{IKf`PUzhjeJ^#~QGwGQX+bNRLzHok>|S81OTSx!u*e`VpMZ5Ag|N#2zh*%dt)*X2NUOcazw*^q_lSj3 zlf2}Ze%kUBPbGkR2h$cOE4;UGFG10V|^h?YJKj{ zhn>3CFsVJZqVC^s%ipYhWTf9Q90Z3AA#LJ2pPBb2j70IkF%{53v#<+c6t!IIsDnFz zMuE_Evj3xL7+@5j@ilT69y`As(VQ*CH}BD@Wnr2cAnHC4rL!mHL7^dbgmJA`L_L1%Nsc?kdXBZ zvC6W~Kavf&eY@BxQoZ2r2$YRf48uh*2h(0?RK;{I8Bv8cEr=BUdj_(4e#9ceop_y5 zX^ytwB;Pshoa38ymF*Jum>X9Kq7b@wFFK$!NKEvr8>vk8LUnccWmGS{WPgp4Tw8u@ zBM1?b;qn*&tc-^D@Nexny&~47nVRi*oFFWNmKgf+N*R|c&)S061f+>809sA^z#Hj4 zwfHhL2}AS@(W^K-abC$_QpWoIFomMZ5m4Dvwa@H`go+nf(|h;$1E3_rwWzEv>YHF1 zF9W1R9-Wg=?ArhP`yu|AcOQrmNw*H8BabALuzN2B$#0=v9Gdjt74QXMGT{hWAz&$6 z4(b`w_GPZQkgzP|7Y{N6wtcHYEif)fG|A0Mmnj>bSHn=Q-P_RP}) z>Pu6Y!9_C8$cj!AEfZ_r=eujGLwj{BJK7 z6`E)7WC8eo#J+!A&VMZXrfp&C?~lhqI(r}=30TMJU%y)x23q`ko#&UQC0nK|VcBm1 z;gbP1kaOs2mjYcp zT5jT{wSqune!+GoW?O+Fg`YjkqXV#ys0?67-r!IfSWiLz3ixiRz>bMV2vJOp+(M#X zVeEWnk!mXrHFpR@k<~}w8Sa(f(4j9Z%JH3N{l30dMOsNJX#ibK&3Xn2MG-qrR5U~>}SF0eDjVyUFB2*Z!TFcj8a3;3I; zn8=Ww_YLkx)@6J+Y}oyhP%ylH&~ldM7D9P}MC#3NJDeEVElK+wd6^KO8o=zsFg25K zWZg68Dhk5E!XcMkNQ1T_+Q1V~CNjG)>Vu;KX~BmhyGH`YSpZvg>LkJ9T$Fd@WAKQ^ zlIIv0ku!&0l)~NNixU5NZ9`Fy4M_7hjn{#n_o?c^e`&}fQ&L@77SyM39`IJDTJc+z8P7U=qU1Y$B{M$L(KE! z%F2Mo1B7IY77t!k(^8T?{6P-GFfO%0omFQGknsK8DwmI>bqH{Erl*>`oSY4W8tE?K ztgPy@0F8ZWEmy;858ZCpNrSH*k1S4#IGQ)X+n}WiCf~dk(W?N;vBW z5cVqpp#ea>RejI-$jIji`+KnFG5`H(QN!}HN-xnuEKiAHaxOAa(IcBzIG8pxzlb z{@QE%7?}M}M4MucYt03ao=~4CqW^r z+g*APw|fo!2UYJ5XRz3FXUl%7uV>ce3G=cD_%NkKCzGb+F~5?;d^C)~;Oa0~yvQtO zaBzzCwmk$eP=DeCW>vwHB4$2mKMp|C+5)nrgm-?Sa;r{D=Sb8Gp47SPL<1 z?q6LL)|cx~JYYEFDj*PY^zTd#>sQ7W{?!4YpNtFrzsvah#QyJTuoemDf&72-G};D+ Z-Tb=L#MkK`L;wcEK!4TZ6kVJB{|{cHab^Gj literal 0 HcmV?d00001 From a204fefe16814d9ef5fcad4071daf79207d5dc36 Mon Sep 17 00:00:00 2001 From: fengjiayi Date: Wed, 18 Oct 2017 14:45:32 -0700 Subject: [PATCH 61/63] Fix several bugs in compile time backward and Protobuf desc (#4894) * Implement FC layer with helper * Update LayerHelper * Add debug string for Python ProtoBuf and Rename `Sync` to `Flush` * Add check of ProtoBuf initialization * Layer wrapper for FC * Fix unittest * Fix CI * Add code generator * AttributeChecker Better error log and speicalize bool Since lots of types can be cast to bool * Complete mlp, fit_a_line * Implementation of simple conv_2d layer * Fix bugs * Correct implement BlockDesc destructor * Fix bugs * Fix unit test error * Follow comments --- paddle/framework/backward.cc | 14 ++++++-------- paddle/framework/block_desc.cc | 11 ++++++++++- paddle/framework/block_desc.h | 10 +++++++--- paddle/pybind/protobuf.cc | 4 ++-- python/paddle/v2/framework/framework.py | 5 ++++- python/paddle/v2/framework/tests/test_layers.py | 13 ++++++++++++- .../v2/framework/tests/test_protobuf_descs.py | 4 +++- 7 files changed, 44 insertions(+), 17 deletions(-) diff --git a/paddle/framework/backward.cc b/paddle/framework/backward.cc index ac80879c54..fb552fe344 100644 --- a/paddle/framework/backward.cc +++ b/paddle/framework/backward.cc @@ -309,8 +309,7 @@ static void CreateGradVarInBlock( } std::vector> MakeOpGrad( - const std::unique_ptr& op_desc, - std::unordered_set* no_grad_vars, + const OpDescBind* op_desc, std::unordered_set* no_grad_vars, std::unordered_map* grad_to_var) { std::vector> grad_op_descs; // All input gradients of forwarding operator do not need to calculate. @@ -357,7 +356,7 @@ std::vector> MakeBlockBackward( std::unordered_set* no_grad_vars, std::unordered_map* grad_to_var) { BlockDescBind* cur_block = program_desc.Block(block_idx); - std::deque>& op_descs = cur_block->ops_; + std::vector op_descs = cur_block->AllOps(); std::unordered_map> dup_out_ops; size_t grad_desc_idx = 0; std::vector> backward_descs; @@ -375,7 +374,7 @@ std::vector> MakeBlockBackward( program_desc, step_block_idx, no_grad_vars, grad_to_var); BlockDescBind* backward_block = program_desc.AppendBlock(*cur_block); for (auto& ptr : backward_block_op_descs) { - backward_block->ops_.push_back(std::move(ptr)); + backward_block->AppendAllocatedOp(std::move(ptr)); } op_grads[0]->SetBlockAttr("step_block", *backward_block); } @@ -432,7 +431,6 @@ ParamGradInfoMap AppendBackward( const int root_block_idx = 0; auto root_block = program_desc.Block(root_block_idx); - auto& all_ops = root_block->ops_; // insert fill one op for target // TODO(qiao) add some check to the target. @@ -447,8 +445,8 @@ ParamGradInfoMap AppendBackward( {{"shape", target_shape}, {"value", static_cast(1.0)}, {"data_type", framework::DataType::FP32}})); - all_ops.push_back(std::move(fill_one_op)); - size_t forward_op_num = all_ops.size(); + root_block->AppendAllocatedOp(std::move(fill_one_op)); + size_t forward_op_num = root_block->OpSize(); size_t forward_block_num = program_desc.Size(); // Insert backward operators @@ -457,7 +455,7 @@ ParamGradInfoMap AppendBackward( &no_grad_var_names, &grad_to_var); for (auto& ptr : backward_op_descs) { - all_ops.push_back(std::move(ptr)); + root_block->AppendAllocatedOp(std::move(ptr)); } // Create Variable diff --git a/paddle/framework/block_desc.cc b/paddle/framework/block_desc.cc index ba970254e5..92ac302e46 100644 --- a/paddle/framework/block_desc.cc +++ b/paddle/framework/block_desc.cc @@ -19,11 +19,11 @@ namespace paddle { namespace framework { VarDescBind *BlockDescBind::Var(const std::string &name) { - need_update_ = true; auto it = vars_.find(name); if (it != vars_.end()) { return it->second.get(); } + need_update_ = true; auto *var = new VarDescBind(name); vars_[name].reset(var); return var; @@ -55,6 +55,11 @@ OpDescBind *BlockDescBind::AppendOp() { return ops_.back().get(); } +void BlockDescBind::AppendAllocatedOp(std::unique_ptr &&op_desc) { + need_update_ = true; + ops_.emplace_back(std::move(op_desc)); +} + OpDescBind *BlockDescBind::PrependOp() { need_update_ = true; ops_.emplace_front(new OpDescBind()); @@ -70,6 +75,10 @@ std::vector BlockDescBind::AllOps() const { } void BlockDescBind::Flush() { + for (auto &op_desc : ops_) { + op_desc->Flush(); + } + if (need_update_) { auto &op_field = *this->desc_->mutable_ops(); this->ClearPBOps(); diff --git a/paddle/framework/block_desc.h b/paddle/framework/block_desc.h index dd7b1228be..5e1f10c1ae 100644 --- a/paddle/framework/block_desc.h +++ b/paddle/framework/block_desc.h @@ -57,10 +57,16 @@ class BlockDescBind { OpDescBind *AppendOp(); + void AppendAllocatedOp(std::unique_ptr &&op_desc); + OpDescBind *PrependOp(); std::vector AllOps() const; + size_t OpSize() const { return ops_.size(); } + + OpDescBind *Op(int idx) { return ops_.at(idx).get(); } + void Flush(); BlockDesc *Proto(); @@ -69,9 +75,7 @@ class BlockDescBind { void ClearPBOps(); void ClearPBVars(); - // FIXME(yuyang18): backward will access private data of BlockDesc. - // Mark it public temporary. We can fix it later. - public: + private: ProgramDescBind *prog_; // not_own BlockDesc *desc_; // not_own bool need_update_; diff --git a/paddle/pybind/protobuf.cc b/paddle/pybind/protobuf.cc index fbdd673295..d9647717d2 100644 --- a/paddle/pybind/protobuf.cc +++ b/paddle/pybind/protobuf.cc @@ -162,8 +162,8 @@ void BindBlockDesc(py::module &m) { py::return_value_policy::reference) .def("all_vars", &BlockDescBind::AllVars, py::return_value_policy::reference) - .def("all_ops", &BlockDescBind::AllOps, - py::return_value_policy::reference) + .def("op_size", &BlockDescBind::OpSize) + .def("op", &BlockDescBind::Op, py::return_value_policy::reference) .def("serialize_to_string", [](BlockDescBind &block_desc) -> py::bytes { const BlockDesc *desc = block_desc.Proto(); PADDLE_ENFORCE(desc->IsInitialized(), diff --git a/python/paddle/v2/framework/framework.py b/python/paddle/v2/framework/framework.py index 93e2218eab..5a8ded46ea 100644 --- a/python/paddle/v2/framework/framework.py +++ b/python/paddle/v2/framework/framework.py @@ -344,7 +344,10 @@ class Block(object): self.create_var(name=var.name(), desc=var, type=var.type()) # sync operators from cpp - ops_in_cpp = self.desc.all_ops() + ops_in_cpp = [] + for op_idx in range(0, self.desc.op_size()): + ops_in_cpp.append(self.desc.op(op_idx)) + first_op_in_python = self.ops[0].desc last_op_in_python = self.ops[len(self.ops) - 1].desc start_index = None diff --git a/python/paddle/v2/framework/tests/test_layers.py b/python/paddle/v2/framework/tests/test_layers.py index 2ffadf7371..2d8c2e5518 100644 --- a/python/paddle/v2/framework/tests/test_layers.py +++ b/python/paddle/v2/framework/tests/test_layers.py @@ -17,6 +17,7 @@ class TestBook(unittest.TestCase): avg_cost = mean(x=cost, program=program) self.assertIsNotNone(avg_cost) + program.append_backward(avg_cost, set()) print str(program) def test_recognize_digits_mlp(self): @@ -34,7 +35,17 @@ class TestBook(unittest.TestCase): cost = cross_entropy(input=predict, label=label, program=program) avg_cost = mean(x=cost, program=program) self.assertIsNotNone(avg_cost) - print str(program) + # print str(program) + + def test_simple_conv2d(self): + pd = core.ProgramDesc.__create_program_desc__() + program = Program(desc=pd) + images = data_layer( + name='pixel', shape=[3, 48, 48], data_type='int32', program=program) + conv2d_layer( + input=images, num_filters=3, filter_size=[4, 4], program=program) + + # print str(program) def test_simple_conv2d(self): pd = core.ProgramDesc.__create_program_desc__() diff --git a/python/paddle/v2/framework/tests/test_protobuf_descs.py b/python/paddle/v2/framework/tests/test_protobuf_descs.py index 6ed8edf91c..2fd3d5d165 100644 --- a/python/paddle/v2/framework/tests/test_protobuf_descs.py +++ b/python/paddle/v2/framework/tests/test_protobuf_descs.py @@ -133,7 +133,9 @@ class TestBlockDesc(unittest.TestCase): op1 = block.append_op() op2 = block.append_op() op0 = block.prepend_op() - all_ops = block.all_ops() + all_ops = [] + for idx in xrange(0, block.op_size()): + all_ops.append(block.op(idx)) self.assertEqual(all_ops, [op0, op1, op2]) From c10b8e808fc88d96ce0b4f864014bd461098de87 Mon Sep 17 00:00:00 2001 From: kavyasrinet Date: Wed, 18 Oct 2017 16:21:16 -0700 Subject: [PATCH 62/63] Adding Proximal Gradient Descent (#4848) * Adding Proximal Gradient Descent * Fixing review comments --- paddle/operators/proximal_gd_op.cc | 93 +++++++++++++++++++ paddle/operators/proximal_gd_op.cu | 19 ++++ paddle/operators/proximal_gd_op.h | 64 +++++++++++++ .../v2/framework/tests/test_proximal_gd_op.py | 33 +++++++ 4 files changed, 209 insertions(+) create mode 100644 paddle/operators/proximal_gd_op.cc create mode 100644 paddle/operators/proximal_gd_op.cu create mode 100644 paddle/operators/proximal_gd_op.h create mode 100644 python/paddle/v2/framework/tests/test_proximal_gd_op.py diff --git a/paddle/operators/proximal_gd_op.cc b/paddle/operators/proximal_gd_op.cc new file mode 100644 index 0000000000..e4b014b9f5 --- /dev/null +++ b/paddle/operators/proximal_gd_op.cc @@ -0,0 +1,93 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. */ + +#include "paddle/operators/proximal_gd_op.h" + +namespace paddle { +namespace operators { + +class ProximalGDOp : public framework::OperatorWithKernel { + public: + using framework::OperatorWithKernel::OperatorWithKernel; + + protected: + void InferShape(framework::InferShapeContext *ctx) const override { + PADDLE_ENFORCE(ctx->HasInput("Param"), + "Input(Param) of ProximalGDOp should not be null."); + PADDLE_ENFORCE(ctx->HasInput("Grad"), + "Input(Grad) of ProximalGDOp should not be null."); + PADDLE_ENFORCE(ctx->HasInput("LearningRate"), + "Input(LearningRate) of ProximalGDOp should not be null."); + + PADDLE_ENFORCE(ctx->HasOutput("ParamOut"), + "Output(ParamOut) of ProximalGDOp should not be null."); + + auto param_dim = ctx->GetInputDim("Param"); + PADDLE_ENFORCE_EQ(param_dim, ctx->GetInputDim("Grad"), + "Two input of ProximalGD Op's dimension must be same."); + + auto lr_dim = ctx->GetInputDim("LearningRate"); + PADDLE_ENFORCE_EQ(framework::product(lr_dim), 1, + "Learning Rate should be a scalar."); + + ctx->SetOutputDim("ParamOut", param_dim); + } +}; + +class ProximalGDOpMaker : public framework::OpProtoAndCheckerMaker { + public: + ProximalGDOpMaker(framework::OpProto *proto, + framework::OpAttrChecker *op_checker) + : OpProtoAndCheckerMaker(proto, op_checker) { + AddInput("Param", + "(Tensor, default Tensor) " + "Input parameter value that has to be updated."); + AddInput("Grad", + "(Tensor, default Tensor) " + "Input gradient of the parameter."); + AddInput("LearningRate", + "(Tensor, default Tensor) " + "The learning rate should be a tensor of size 1."); + + AddOutput("ParamOut", "(Tensor) Output updated parameter value."); + + AddAttr("l1", + "(float, default 0.0) " + "L1 regularization strength.") + .SetDefault(0.0f); + AddAttr("l2", + "(float, default 0.0)" + "L2 regularization strength.") + .SetDefault(0.0f); + AddComment(R"DOC( + +Optimizer that implements the proximal gradient descent algorithm. + +prox_param = param - learning_rate * grad +param = sign(prox_param) / (1 + learning_rate * l2) * + max { |prox_param| - learning_rate * l1 , 0 } + +The paper that proposed Proximal Gradient Descent: +(http://papers.nips.cc/paper/3793-efficient-learning-using-forward-backward-splitting.pdf) +)DOC"); + } +}; +} // namespace operators +} // namespace paddle + +namespace ops = paddle::operators; +REGISTER_OP_WITHOUT_GRADIENT(proximal_gd, ops::ProximalGDOp, + ops::ProximalGDOpMaker); +REGISTER_OP_CPU_KERNEL( + proximal_gd, ops::ProximalGDOpKernel); diff --git a/paddle/operators/proximal_gd_op.cu b/paddle/operators/proximal_gd_op.cu new file mode 100644 index 0000000000..26f4ebaa0f --- /dev/null +++ b/paddle/operators/proximal_gd_op.cu @@ -0,0 +1,19 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + +Licensed under the Apache License, Version 2.0 (the "License"); +You may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software distributed +under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR +CONDITIONS OF ANY KIND, either express or implied. See the License for the +specific language governing permissions and limitations under the License. */ + +#define EIGEN_USE_GPU +#include "paddle/operators/proximal_gd_op.h" + +namespace ops = paddle::operators; +REGISTER_OP_GPU_KERNEL( + proximal_gd, ops::ProximalGDOpKernel); diff --git a/paddle/operators/proximal_gd_op.h b/paddle/operators/proximal_gd_op.h new file mode 100644 index 0000000000..bebda02041 --- /dev/null +++ b/paddle/operators/proximal_gd_op.h @@ -0,0 +1,64 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. */ + +#pragma once +#include "paddle/framework/eigen.h" +#include "paddle/framework/op_registry.h" + +namespace paddle { +namespace operators { + +using Tensor = framework::Tensor; +template +using EigenVector = framework::EigenVector; + +template +class ProximalGDOpKernel : public framework::OpKernel { + public: + void Compute(const framework::ExecutionContext& ctx) const override { + auto* param_out = ctx.Output("ParamOut"); + + param_out->mutable_data(ctx.GetPlace()); + + auto grad = ctx.Input("Grad"); + + auto l1 = static_cast(ctx.Attr("l1")); + auto l2 = static_cast(ctx.Attr("l2")); + + auto p = EigenVector::Flatten(*ctx.Input("Param")); + auto g = EigenVector::Flatten(*grad); + auto lr = EigenVector::Flatten(*ctx.Input("LearningRate")); + + auto p_out = EigenVector::Flatten(*param_out); + auto place = ctx.GetEigenDevice(); + + Eigen::DSizes grad_dsize(grad->numel()); + + auto prox_param = p - lr.broadcast(grad_dsize) * g; + if (l1 > 0) { + p_out.device(place) = + prox_param.sign() * + (((prox_param.abs() - (lr * l1).broadcast(grad_dsize)) + .cwiseMax(T(0.0))) / + (1.0 + (lr * l2).broadcast(grad_dsize))); + } else { + p_out.device(place) = + prox_param / (1.0 + (lr * l2).broadcast(grad_dsize)); + } + } +}; + +} // namespace operators +} // namespace paddle diff --git a/python/paddle/v2/framework/tests/test_proximal_gd_op.py b/python/paddle/v2/framework/tests/test_proximal_gd_op.py new file mode 100644 index 0000000000..9ca79ce6b3 --- /dev/null +++ b/python/paddle/v2/framework/tests/test_proximal_gd_op.py @@ -0,0 +1,33 @@ +import unittest +import numpy as np +from op_test import OpTest + + +class TestProximalGDOp(OpTest): + def setUp(self): + self.op_type = "proximal_gd" + w = np.random.random((102, 105)).astype("float32") + g = np.random.random((102, 105)).astype("float32") + lr = np.array([0.1]).astype("float32") + l1 = 0.1 + l2 = 0.2 + + self.inputs = {'Param': w, 'Grad': g, 'LearningRate': lr} + self.attrs = {'l1': l1, 'l2': l2} + prox_param = w - lr * g + param_out = 0.0 + if l1 > 0.0: + x = np.abs(prox_param) - lr * l1 + x[x < 0] = 0 + param_out = np.sign(prox_param) * (x / (1.0 + lr * l2)) + else: + param_out = prox_param / (1.0 + lr * l2) + + self.outputs = {'ParamOut': param_out} + + def test_check_output(self): + self.check_output() + + +if __name__ == "__main__": + unittest.main() From c93596d35b621959d28f16ffba7689a79bd9b068 Mon Sep 17 00:00:00 2001 From: fengjiayi Date: Wed, 18 Oct 2017 17:21:32 -0700 Subject: [PATCH 63/63] unify layer names (#4913) --- python/paddle/v2/framework/layers.py | 50 +++++++++---------- .../paddle/v2/framework/tests/test_layers.py | 38 +++++++------- 2 files changed, 45 insertions(+), 43 deletions(-) diff --git a/python/paddle/v2/framework/layers.py b/python/paddle/v2/framework/layers.py index 1821da197e..c7397716c4 100644 --- a/python/paddle/v2/framework/layers.py +++ b/python/paddle/v2/framework/layers.py @@ -3,17 +3,17 @@ import paddle.v2.framework.core as core from paddle.v2.framework.framework import OpProtoHolder, Variable import re -__all__ = ['fc_layer', 'data_layer', 'cross_entropy', 'conv2d_layer'] +__all__ = ['fc', 'data', 'cross_entropy', 'conv2d'] -def fc_layer(input, - size, - param_attr=None, - bias_attr=True, - name=None, - act=None, - num_flatten_dims=1, - program=None): +def fc(input, + size, + param_attr=None, + bias_attr=True, + name=None, + act=None, + num_flatten_dims=1, + program=None): # create helper helper = LayerHelper('fc', **locals()) @@ -51,11 +51,11 @@ def fc_layer(input, return helper.append_activation(pre_activation) -def data_layer(name, - shape, - data_type='float32', - type=core.VarDesc.VarType.LOD_TENSOR, - program=None): +def data(name, + shape, + data_type='float32', + type=core.VarDesc.VarType.LOD_TENSOR, + program=None): helper = LayerHelper('data', **locals()) shape = [-1] + shape # append batch size as -1 return helper.create_global_variable( @@ -145,17 +145,17 @@ def square_error_cost(input, label, **kwargs): return square_out -def conv2d_layer(input, - num_filters, - name=None, - filter_size=[1, 1], - act=None, - groups=None, - stride=[1, 1], - padding=None, - bias_attr=None, - param_attr=None, - program=None): +def conv2d(input, + num_filters, + name=None, + filter_size=[1, 1], + act=None, + groups=None, + stride=[1, 1], + padding=None, + bias_attr=None, + param_attr=None, + program=None): helper = LayerHelper('conv2d', **locals()) dtype = helper.input_dtype() diff --git a/python/paddle/v2/framework/tests/test_layers.py b/python/paddle/v2/framework/tests/test_layers.py index 2d8c2e5518..dbbb653538 100644 --- a/python/paddle/v2/framework/tests/test_layers.py +++ b/python/paddle/v2/framework/tests/test_layers.py @@ -1,4 +1,4 @@ -from paddle.v2.framework.layers import fc_layer, data_layer, cross_entropy, mean, square_error_cost, conv2d_layer +import paddle.v2.framework.layers as layers from paddle.v2.framework.framework import Program, g_program import paddle.v2.framework.core as core import unittest @@ -7,15 +7,16 @@ import unittest class TestBook(unittest.TestCase): def test_fit_a_line(self): program = Program() - x = data_layer( + x = layers.data( name='x', shape=[13], data_type='float32', program=program) - y_predict = fc_layer(input=x, size=1, act=None, program=program) + y_predict = layers.fc(input=x, size=1, act=None, program=program) - y = data_layer( + y = layers.data( name='y', shape=[1], data_type='float32', program=program) - cost = square_error_cost(input=y_predict, label=y, program=program) + cost = layers.square_error_cost( + input=y_predict, label=y, program=program) - avg_cost = mean(x=cost, program=program) + avg_cost = layers.mean(x=cost, program=program) self.assertIsNotNone(avg_cost) program.append_backward(avg_cost, set()) print str(program) @@ -24,16 +25,18 @@ class TestBook(unittest.TestCase): program = Program() # Change g_program, so the rest layers use `g_program` - images = data_layer( + images = layers.data( name='pixel', shape=[784], data_type='float32', program=program) - label = data_layer( + label = layers.data( name='label', shape=[1], data_type='int32', program=program) - hidden1 = fc_layer(input=images, size=128, act='relu', program=program) - hidden2 = fc_layer(input=hidden1, size=64, act='relu', program=program) - predict = fc_layer( - input=hidden2, size=10, act='softmax', program=program) - cost = cross_entropy(input=predict, label=label, program=program) - avg_cost = mean(x=cost, program=program) + hidden1 = layers.fc(input=images, size=128, act='relu', program=program) + hidden2 = layers.fc(input=hidden1, size=64, act='relu', program=program) + predict = layers.fc(input=hidden2, + size=10, + act='softmax', + program=program) + cost = layers.cross_entropy(input=predict, label=label, program=program) + avg_cost = layers.mean(x=cost, program=program) self.assertIsNotNone(avg_cost) # print str(program) @@ -48,11 +51,10 @@ class TestBook(unittest.TestCase): # print str(program) def test_simple_conv2d(self): - pd = core.ProgramDesc.__create_program_desc__() - program = Program(desc=pd) - images = data_layer( + program = Program() + images = layers.data( name='pixel', shape=[3, 48, 48], data_type='int32', program=program) - conv2d_layer( + layers.conv2d( input=images, num_filters=3, filter_size=[4, 4], program=program) print str(program)