From 746517cc4abafe8b795ba0f0ff9d1cf49996acf8 Mon Sep 17 00:00:00 2001 From: Liu Yiqun Date: Thu, 28 Sep 2017 09:55:37 +0800 Subject: [PATCH 01/76] Update the documentation of how to cross-compile for Android. --- .../cross_compiling_for_android_cn.md | 111 +++++++++++++++--- 1 file changed, 97 insertions(+), 14 deletions(-) diff --git a/doc/howto/cross_compiling/cross_compiling_for_android_cn.md b/doc/howto/cross_compiling/cross_compiling_for_android_cn.md index 90dc84718c..0e67c12c57 100644 --- a/doc/howto/cross_compiling/cross_compiling_for_android_cn.md +++ b/doc/howto/cross_compiling/cross_compiling_for_android_cn.md @@ -1,9 +1,56 @@ # 构建Android平台上的PaddlePaddle库 -用户可通过交叉编译的方式，在用户熟悉的开发平台（Linux，Mac OS X和Windows）上编译Android平台上适用的PaddlePaddle库。 +用户可通过如下两种方式，交叉编译Android平台上适用的PaddlePaddle库： +- 基于Docker容器的编译方式 +- 基于Linux交叉编译环境的编译方式 + +## 基于Docker容器的编译方式 +Docker能在所有主要操作系统（包括Linux，Mac OS X和Windows）上运行，因此，使用基于Docker容器的编译方式，用户可在自己熟悉的开发平台上编译Android平台上适用的PaddlePaddle库。 + +### 构建PaddlePaddle的Android开发镜像 +我们把PaddlePaddle的交叉编译环境打包成一个镜像，称为开发镜像，里面涵盖了交叉编译Android版PaddlePaddle库需要的所有编译工具。 + +```bash +$ git clone https://github.com/PaddlePaddle/Paddle.git +$ cd Paddle +$ docker build -t username/paddle-android:dev . -f Dockerfile.android +``` + +### 编译PaddlePaddle C-API库 +构建好开发镜像后，即可使用开发镜像来编译Android版PaddlePaddle C-API库。 +Android的Docker开发镜像向用户提供两个可配置的参数： + +| Argument | Optional Values | Default | +|-----------------|-------------------------|---------| +|`ANDROID_ABI` |`armeabi-v7a, arm64-v8a` | `armeabi-v7a` | +|`ANDROID_API` |`armeabi-v7a(>15), arm64-v8a(>20)` | `21` | + +- 编译`armeabi-v7a`，`Android API 21`的PaddlePaddle库 +```bash +$ docker run -it --rm -v $PWD:/paddle -e "ANDROID_ABI=armeabi-v7a" -e "ANDROID_API=21" username/paddle-android:dev +``` +或 +```bash +$ docker run -it --rm -v $PWD:/paddle username/paddle-android:dev +``` + +如需编译Android API低于21的Paddle库，请参考本文档的**准备交叉编译环境**章节。 + +- 编译`arm64-v8a`，`Android API 21`的PaddlePaddle库 +```bash +$ docker run -it --rm -v $PWD:/paddle -e "ANDROID_ABI=arm64-v8a" -e "ANDROID_API=21" username/paddle-android:dev +``` +或 +```bash +$ docker run -it --rm -v $PWD:/paddle -e "ANDROID_ABI=arm64-v8a" username/paddle-android:dev +``` + +执行上述`docker run`命令时，容器默认执行[paddle/scripts/docker/build_android.sh](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/scripts/docker/build_android.sh)脚本。该脚本中记录了交叉编译Android版PaddlePaddle库常用的CMake配置，并且会根据`ANDROID_ABI`和`ANDROID_API`自动构建独立工具链、进行编译和安装。由于arm64架构要求Android API不小于21。因此当`ANDROID_ABI=arm64-v8a`，`ANDROID_API<21`时，Docker容器中将默认使用`Android API 21`的编译工具链。用户可以参考下文**配置交叉编译参数**章节，根据个人的需求修改定制Docker容器所执行的脚本。编译安装结束之后，PaddlePaddle的C-API库将被安装到`$PWD/install_android`目录，所依赖的第三方库同时也被安装到`$PWD/install_android/third_party`目录。 + +## 基于Linux交叉编译环境的编译方式 本文档将以Linux x86-64平台为例，介绍交叉编译Android平台上适用的PaddlePaddle库的方法和步骤。 -## 准备交叉编译环境 +### 准备交叉编译环境 从源码交叉编译PaddlePaddle，用户需要提前准备好交叉编译环境。Android平台上使用的C/C++交叉编译工具链为[Android NDK](https://developer.android.com/ndk/downloads/index.html?hl=zh-cn)，用户可自行前往下载预编译好的版本，也可通过以下命令获取： @@ -13,18 +60,29 @@ unzip -q android-ndk-r14b-linux-x86_64.zip ``` Android NDK中包含了所有Android API级别、所有架构（arm/arm64/x86/mips）需要用到的编译工具和系统库。用户可根据自己的编译目标架构、所需支持的最低Android API级别，构建[独立工具链](https://developer.android.google.cn/ndk/guides/standalone_toolchain.html?hl=zh-cn)。 -比如： +- 构建`armeabi-v7a`、 `Android API 21`的独立工具链： + +```bash +your/path/to/android-ndk-r14b-linux-x86_64/build/tools/make-standalone-toolchain.sh \ + --arch=arm --platform=android-21 --install-dir=your/path/to/arm_standalone_toolchain +``` + +此命令将在`your/path/to/arm_standalone_toolchain`目录生成一套独立编译工具链，面向架构为32位ARM架构，支持的最小的Android API级别为21，支持编译器`arm-linux-androideabi-gcc (GCC) 4.9`和`clang 3.8`。 + +注意：**PaddlePaddle要求使用的编译工具链所支持的Andoid API级别不小于16**。但由于PaddlePaddle所依赖的第三方库`glog`不支持低于21的Android API，所以在编译Android API低于21的Paddle C-API库时，需要将[cmake/external/glog.cmake](https://github.com/PaddlePaddle/Paddle/blob/develop/cmake/external/glog.cmake#L33)中的`GIT_REPOSITORY`临时修改为`https://github.com/Xreki/glog.git`。 + +- 构建`arm64-v8a`、 `Android API 21`的独立工具链： ```bash your/path/to/android-ndk-r14b-linux-x86_64/build/tools/make-standalone-toolchain.sh \ - --arch=arm --platform=android-21 --install-dir=your/path/to/my_standalone_toolchain + --arch=arm64 --platform=android-21 --install-dir=your/path/to/arm64_standalone_toolchain ``` -此命令将在your/path/to/my_standalone_toolchain目录生成一套编译工具链，面向架构为32位ARM架构，支持的最小的Android API级别为21，使用的编译器为arm-linux-androideabi-gcc (GCC) 4.9。 +此命令将在`your/path/to/arm64_standalone_toolchain`目录生成一套独立编译工具链，面向架构为64位ARM64架构，支持的最小Android API级别为21，支持编译器`arm-linux-androideabi-gcc (GCC) 4.9`和`clang 3.8`。 -注意：**PaddlePaddle要求使用的编译工具链所支持的Andoid API级别不小于21**。 +注意：**arm64架构要求Android API不小于21**。 -## 配置交叉编译参数 +### 配置交叉编译参数 CMake系统对交叉编译提供了支持[cmake-toolchains](https://cmake.org/cmake/help/v3.0/manual/cmake-toolchains.7.html#cross-compiling)。为了简化cmake配置，PaddlePaddle为交叉编译提供了工具链配置文档[cmake/cross_compiling/android.cmake](https://github.com/PaddlePaddle/Paddle/blob/develop/cmake/cross_compiling/android.cmake)，以提供一些默认的编译器和编译参数相关配置。注意，从CMake 3.7版本开始，CMake官方对Android平台的交叉编译提供了通用的支持。PaddlePaddle若检测到用户使用的CMake版本不低于3.7时，将会将用户传进来的配置参数传递CMake系统，交由CMake系统本身来处理。有关参数配置的详细说明见[cmake-toolchains](https://cmake.org/cmake/help/v3.7/manual/cmake-toolchains.7.html#cross-compiling)。 @@ -36,32 +94,57 @@ CMake系统对交叉编译提供了支持[cmake-toolchains](https://cmake.org/cm Android平台可选配置参数： - `ANDROID_STANDALONE_TOOLCHAIN`，独立工具链所在的绝对路径，或者相对于构建目录的相对路径。PaddlePaddle的CMake系统将根据该值自动推导和设置需要使用的交叉编译器、sysroot、以及Android API级别；否则，用户需要在cmake时手动设置这些值。无默认值。 -- `ANDROID_ABI`，目标架构ABI。目前只支持`armeabi-v7a`，默认值为`armeabi-v7a`。 +- `ANDROID_TOOLCHAIN`，目标工具链。可设置`gcc/clang`，默认值为`clang`。 + - CMake 3.7以上，将会始终使用`clang`工具链；CMake 3.7以下，可设置`ANDROID_TOOLCHAIN=gcc`以使用`gcc`工具链。 + - Android官方提供的`clang`编译器要求系统支持`GLIBC 2.15`以上。 +- `ANDROID_ABI`，目标架构ABI。目前支持`armeabi-v7a`和`arm64-v8a`，默认值为`armeabi-v7a`。 - `ANDROID_NATIVE_API_LEVEL`，工具链的Android API级别。若没有显式设置，PaddlePaddle将根据`ANDROID_STANDALONE_TOOLCHAIN`的值自动推导得到。 -- `ANROID_ARM_MODE`，是否使用ARM模式。可设置`ON/OFF`，默认值为`ON`。 -- `ANDROID_ARM_NEON`，是否使用NEON指令。目前必须设置成`ON`，默认值为`ON`。 +- `ANROID_ARM_MODE`，是否使用ARM模式。 + - `ANDROID_ABI=armeabi-v7a`时，可设置`ON/OFF`，默认值为`ON`； + - `ANDROID_ABI=arm64-v8a`时，不需要设置。 +- `ANDROID_ARM_NEON`，是否使用NEON指令。 + - `ANDROID_ABI=armeabi-v7a`时，可设置`ON/OFF`，默认值为`ON`； + - `ANDROID_ABI=arm64-v8a`时，不需要设置。 其他配置参数： +- `USE_EIGEN_FOR_BLAS`，是否使用Eigen库进行矩阵计算。可设置`ON/OFF`，默认值为`OFF`。 - `HOST_C/CXX_COMPILER`，宿主机的C/C++编译器。在编译宿主机版protoc可执行文件和目标机版OpenBLAS库时需要用到。默认设置成环境变量`CC`的值；若环境变量`CC`没有设置，则设置成`cc`编译器。 -一种常用的cmake配置如下： +常用的cmake配置如下： ```bash cmake -DCMAKE_SYSTEM_NAME=Android \ - -DANDROID_STANDALONE_TOOLCHAIN=your/path/to/my_standalone_toolchain \ + -DANDROID_STANDALONE_TOOLCHAIN=your/path/to/arm_standalone_toolchain \ -DANDROID_ABI=armeabi-v7a \ -DANDROID_ARM_NEON=ON \ -DANDROID_ARM_MODE=ON \ + -DUSE_EIGEN_FOR_BLAS=ON \ -DCMAKE_INSTALL_PREFIX=your/path/to/install \ -DWITH_C_API=ON \ -DWITH_SWIG_PY=OFF \ .. ``` +``` +cmake -DCMAKE_SYSTEM_NAME=Android \ + -DANDROID_STANDALONE_TOOLCHAIN=your/path/to/arm64_standalone_toolchain \ + -DANDROID_ABI=arm64-v8a \ + -DUSE_EIGEN_FOR_BLAS=OFF \ + -DCMAKE_INSTALL_PREFIX=your/path/to/install \ + -DWITH_C_API=ON \ + -DWITH_SWIG_PY=OFF \ + .. +``` + 用户还可根据自己的需求设置其他编译参数。比如希望最小化生成的库的大小，可以设置`CMAKE_BUILD_TYPE`为`MinSizeRel`；若希望最快的执行速度，则可设置`CMAKE_BUILD_TYPE`为`Release`。亦可以通过手动设置`CMAKE_C/CXX_FLAGS_MINSIZEREL/RELEASE`来影响PaddlePaddle的编译过程。 -## 编译和安装 +**性能TIPS**，为了达到最快的计算速度，在CMake参数配置上，有以下建议： +- 设置`CMAKE_BUILD_TYPE`为`Release` +- 使用`clang`编译工具链 +- `armeabi-v7a`时，设置`USE_EIGEN_BLAS=ON`，使用Eigen进行矩阵计算；`arm64-v8a`时，设置`USE_EIGEN_FOR_BLAS=OFF`，使用OpenBLAS进行矩阵计算 + +### 编译和安装 CMake配置完成后，执行以下命令，PaddlePaddle将自动下载和编译所有第三方依赖库、编译和安装PaddlePaddle预测库。 @@ -72,4 +155,4 @@ make install 注意：如果你曾经在源码目录下编译过其他平台的PaddlePaddle库，请先使用`rm -rf`命令删除`third_party`目录和`build`目录，以确保所有的第三方依赖库和PaddlePaddle代码都是针对新的CMake配置重新编译的。 -执行完安装命令后，`your/path/to/install`目录中会包含`include`和`lib`目录，其中`include`中包含C-API的头文件，`lib`中包含一个Android版本的库。自此，PaddlePaddle的已经安装完成，用户可将`your/path/to/install`目录下的生成文件用于深度学习相关Android App中，调用方法见C-API文档。 +执行完安装命令后，`your/path/to/install`目录中会包含`include`、`lib`和`third_party`目录，其中`include`中包含C-API的头文件，`lib`中包含若干个不同Android ABI的PaddlePaddle库，`third_party`中包含所依赖的所有第三方库。自此，PaddlePaddle的已经安装完成，用户可将`your/path/to/install`目录下的生成文件用于深度学习相关Android App中，调用方法见C-API文档。 From c4d3fef15757c3811108db4f975e344d63108959 Mon Sep 17 00:00:00 2001 From: "Yang Yang(Tony)" Date: Thu, 28 Sep 2017 12:07:33 -0700 Subject: [PATCH 02/76] update doc: no need to modify pybind_file `paddle/operators/CMakeLists.txt` will automatically generate the bind. --- doc/design/refactorization.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/doc/design/refactorization.md b/doc/design/refactorization.md index ad801ca421..ffcc069ccd 100644 --- a/doc/design/refactorization.md +++ b/doc/design/refactorization.md @@ -177,8 +177,6 @@ REGISTER_OP(op_type, op_class, op_maker_class, grad_op_type, grad_op_class) REGISTER_OP_WITHOUT_GRADIENT(op_type, op_class, op_maker_class) ``` -### `USE` Macros -make sure the registration process is executed and linked. --- # Register Process @@ -188,7 +186,7 @@ make sure the registration process is executed and linked. 1. call maker class to complete `proto` and `checker` 2. with the completed `proto` and `checker`, build a new key-value pair in the `OpInfoMap` -4. Invoke `USE` macro in where the Op is used to make sure it is linked. + --- # Backward Module (1/2) From a53191f12a41593b9f7e35e6c039fe76a350e2f7 Mon Sep 17 00:00:00 2001 From: guosheng Date: Fri, 29 Sep 2017 11:21:15 +0800 Subject: [PATCH 03/76] Add norm_op --- paddle/operators/reduce_op.cc | 63 +++++++++++++++++++ .../v2/framework/tests/test_reduce_op.py | 28 +++++++++ 2 files changed, 91 insertions(+) diff --git a/paddle/operators/reduce_op.cc b/paddle/operators/reduce_op.cc index 3ef443d1c7..e4791e6c07 100644 --- a/paddle/operators/reduce_op.cc +++ b/paddle/operators/reduce_op.cc @@ -13,6 +13,7 @@ limitations under the License. */ #include "paddle/operators/reduce_op.h" +#include "paddle/operators/net_op.h" namespace paddle { namespace operators { @@ -161,6 +162,66 @@ class ReduceMinOpMaker : public ReduceOpMaker { } }; +class NormOp : public NetOp { + public: + NormOp(const std::string &type, const framework::VariableNameMap &inputs, + const framework::VariableNameMap &outputs, + const framework::AttributeMap &attrs) + : NetOp(type, inputs, outputs, attrs) { + PADDLE_ENFORCE_NE(Input("X"), framework::kEmptyVarName, + "Input(X) of NormOp should not be null."); + PADDLE_ENFORCE_NE(Output("AbsOut"), framework::kEmptyVarName, + "Output(AbsOut) of NormOp should not be null."); + PADDLE_ENFORCE_NE(Output("PowOut"), framework::kEmptyVarName, + "Output(PowOut) of NormOp should not be null."); + PADDLE_ENFORCE_NE(Output("SumOut"), framework::kEmptyVarName, + "Output(SumOut) of NormOp should not be null."); + PADDLE_ENFORCE_NE(Output("Out"), framework::kEmptyVarName, + "Output(Out) of NormOp should not be null."); + auto dim = Attr("dim"); + auto keep_dim = Attr("keep_dim"); + auto p = Attr("p"); + PADDLE_ENFORCE_GT(p, 0, "Order of the norm should be positive."); + AppendOp(framework::OpRegistry::CreateOp("abs", {{"X", {Input("X")}}}, + {{"Y", {Output("AbsOut")}}}, {})); + AppendOp(framework::OpRegistry::CreateOp("pow", {{"X", {Output("AbsOut")}}}, + {{"Y", {Output("PowOut")}}}, + {{"factor", p}})); + framework::AttributeMap sum_attr; + sum_attr["dim"] = dim; + sum_attr["keep_dim"] = keep_dim; + AppendOp(framework::OpRegistry::CreateOp( + "reduce_sum", {{"X", {Output("PowOut")}}}, + {{"Out", {Output("SumOut")}}}, sum_attr)); + AppendOp(framework::OpRegistry::CreateOp( + "pow", {{"X", {Output("SumOut")}}}, {{"Y", {Output("Out")}}}, + {{"factor", static_cast(1. / p)}})); + CompleteAddOp(false); + } +}; + +class NormOpMaker : public ReduceOpMaker { + public: + NormOpMaker(framework::OpProto *proto, framework::OpAttrChecker *op_checker) + : ReduceOpMaker(proto, op_checker) { + AddOutput("AbsOut", + "(Tensor) The intermediate output of Norm operator, " + "saving the absolute value of the input tensor X.") + .AsIntermediate(); + AddOutput("PowOut", + "(Tensor) The intermediate output of Norm operator, " + "saving the p-th power of the output tensor AbsOut.") + .AsIntermediate(); + AddOutput("SumOut", + "(Tensor) the intermediate output of Norm operator, " + "saving the sum of PowOut reduced on the given dimension.") + .AsIntermediate(); + AddAttr("p", "(float, default 2) The order of Norm.").SetDefault(2); + SetComment("Norm", "vector p-norm"); + AddComment(comment_); + } +}; + } // namespace operators } // namespace paddle @@ -201,3 +262,5 @@ REGISTER_OP_CPU_KERNEL( REGISTER_OP_CPU_KERNEL(reduce_min_grad, ops::ReduceGradKernel); + +REGISTER_OP_WITHOUT_GRADIENT(norm, ops::NormOp, ops::NormOpMaker); diff --git a/python/paddle/v2/framework/tests/test_reduce_op.py b/python/paddle/v2/framework/tests/test_reduce_op.py index 70359d60cb..0fec31c2e2 100644 --- a/python/paddle/v2/framework/tests/test_reduce_op.py +++ b/python/paddle/v2/framework/tests/test_reduce_op.py @@ -85,5 +85,33 @@ class Test1DReduce(OpTest): self.check_grad(['X'], 'Out') +class TestNorm(OpTest): + def setUp(self): + # use x away from 0 to avoid errors of numerical gradient when gradient near 0 + x = np.random.random((5, 6, 10)).astype("float32") + 0.2 + p = 2 + dim = 1 + keep_dim = False + abs_out = np.absolute(x) + pow_out = np.power(x, p) + sum_out = np.sum(pow_out, axis=dim, keepdims=keep_dim) + out = np.power(sum_out, 1. / p) + self.op_type = "norm" + self.inputs = {'X': x} + self.attrs = {"p": p, "dim": dim, "keep_dim": keep_dim} + self.outputs = { + "AbsOut": abs_out, + "PowOut": pow_out, + "SumOut": sum_out, + "Out": out + } + + def test_check_output(self): + self.check_output() + + def test_check_grad(self): + self.check_grad(['X'], 'Out', max_relative_error=0.01) + + if __name__ == '__main__': unittest.main() From a31ff363fdb2bb02317ed72be8768dd1d5f0d2fe Mon Sep 17 00:00:00 2001 From: Yang Yang Date: Wed, 11 Oct 2017 23:18:08 +0000 Subject: [PATCH 04/76] prune pass dummy test --- paddle/framework/CMakeLists.txt | 3 + paddle/framework/framework.proto | 1 + paddle/framework/prune.cc | 107 +++++++++++++++++ paddle/framework/prune.h | 26 ++++ paddle/framework/prune_test.cc | 200 +++++++++++++++++++++++++++++++ 5 files changed, 337 insertions(+) create mode 100644 paddle/framework/prune.cc create mode 100644 paddle/framework/prune.h create mode 100644 paddle/framework/prune_test.cc diff --git a/paddle/framework/CMakeLists.txt b/paddle/framework/CMakeLists.txt index 6b34c3bbcf..d9c84f3c0a 100644 --- a/paddle/framework/CMakeLists.txt +++ b/paddle/framework/CMakeLists.txt @@ -49,5 +49,8 @@ cc_library(executor SRCS executor.cc DEPS op_registry device_context scope frame # cc_test(executor_test SRCS executor_test.cc DEPS executor) #endif() +cc_library(prune SRCS prune.cc) +cc_test(prune_test SRCS prune_test.cc DEPS prune recurrent_op device_context) + cc_library(tensor_array SRCS tensor_array.cc DEPS lod_tensor) cc_test(tensor_array_test SRCS tensor_array_test.cc DEPS tensor_array place) diff --git a/paddle/framework/framework.proto b/paddle/framework/framework.proto index b7a63f9ba1..7739c17215 100644 --- a/paddle/framework/framework.proto +++ b/paddle/framework/framework.proto @@ -55,6 +55,7 @@ message OpDesc { repeated Var inputs = 1; repeated Var outputs = 2; repeated Attr attrs = 4; + required bool is_target = 5 [ default = false ]; }; // OpProto describes a C++ framework::OperatorBase derived class. diff --git a/paddle/framework/prune.cc b/paddle/framework/prune.cc new file mode 100644 index 0000000000..ddb9ed7ae0 --- /dev/null +++ b/paddle/framework/prune.cc @@ -0,0 +1,107 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. */ + +#include "paddle/framework/prune.h" + +#include +#include +#include +#include + +#include + +namespace paddle { +namespace framework { + +const std::string kFeedOpType = "feed"; +const std::string kFetchOpType = "fetch"; + +bool HasDependentVar(const OpDesc& op_desc, + const std::set& dependent_vars) { + for (auto& var : op_desc.outputs()) { + for (auto& argu : var.arguments()) { + if (dependent_vars.count(argu) != 0) { + return true; + } + } + } + return false; +} + +void Prune(const ProgramDesc& input, ProgramDesc& output, int id) { + // TODO(tonyyang-svail): + // - will change to use multiple blocks for RNN op and Cond Op + + auto& block = input.blocks(0); + auto& ops = block.ops(); + + bool expect_feed = true; + for (auto& op_desc : ops) { + PADDLE_ENFORCE(op_desc.type() != kFeedOpType || expect_feed, + "All FeedOps are at the beginning of the ProgramDesc"); + expect_feed = (op_desc.type() == kFeedOpType); + } + + bool expect_fetch = true; + for (auto op_iter = ops.rbegin(); op_iter != ops.rend(); ++op_iter) { + auto& op_desc = *op_iter; + PADDLE_ENFORCE(op_desc.type() != kFetchOpType || expect_fetch, + "All FetchOps must at the end of the ProgramDesc"); + expect_fetch = (op_desc.type() == kFetchOpType); + } + + std::set dependent_vars; + std::vector should_run; + for (auto op_iter = ops.rbegin(); op_iter != ops.rend(); ++op_iter) { + auto& op_desc = *op_iter; + + if (op_desc.is_target() || HasDependentVar(op_desc, dependent_vars)) { + // erase its output to the dependency graph + for (auto& var : op_desc.outputs()) { + for (auto& argu : var.arguments()) { + dependent_vars.erase(argu); + } + } + + // insert its input to the dependency graph + for (auto& var : op_desc.inputs()) { + for (auto& argu : var.arguments()) { + dependent_vars.insert(argu); + } + } + + should_run.push_back(true); + } else { + should_run.push_back(false); + } + } + + // since we are traversing the ProgramDesc in reverse order + // we reverse the should_run vector + std::reverse(should_run.begin(), should_run.end()); + + output = input; + auto* op_field = output.mutable_blocks(id)->mutable_ops(); + op_field->Clear(); + for (size_t i = 0; i < should_run.size(); ++i) { + if (should_run[i]) { + *op_field->Add() = input.blocks(id).ops(i); + } + } + + // return should_run; +} + +} // namespace framework +} // namespace paddle diff --git a/paddle/framework/prune.h b/paddle/framework/prune.h new file mode 100644 index 0000000000..3e1d58f61f --- /dev/null +++ b/paddle/framework/prune.h @@ -0,0 +1,26 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. */ + +#pragma once + +#include "paddle/framework/framework.pb.h" +#include "paddle/platform/enforce.h" + +namespace paddle { +namespace framework { + +void Prune(const ProgramDesc& input, ProgramDesc& output, int id); + +} // namespace framework +} // namespace paddle diff --git a/paddle/framework/prune_test.cc b/paddle/framework/prune_test.cc new file mode 100644 index 0000000000..b66db94528 --- /dev/null +++ b/paddle/framework/prune_test.cc @@ -0,0 +1,200 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include "paddle/framework/prune.h" + +#include +#include "paddle/framework/attribute.h" +#include "paddle/framework/block_desc.h" +#include "paddle/framework/op_desc.h" +#include "paddle/framework/op_registry.h" +#include "paddle/framework/operator.h" +#include "paddle/framework/program_desc.h" +#include "paddle/operators/net_op.h" + +namespace paddle { +namespace framework { + +using DeviceContext = platform::DeviceContext; + +class RowWiseAddOpMaker : public OpProtoAndCheckerMaker { + public: + RowWiseAddOpMaker(OpProto *proto, OpAttrChecker *op_checker) + : OpProtoAndCheckerMaker(proto, op_checker) { + AddInput("X", "Input X of Add"); + AddInput("b", "Bias of Add"); + AddOutput("Out", "Out of Add"); + AddComment("Add Op"); + } +}; + +class RowWiseAddGradMaker : public SingleGradOpDescMaker { + public: + using SingleGradOpDescMaker::SingleGradOpDescMaker; + + protected: + std::unique_ptr Apply() const override { + auto grad_op = new OpDescBind(); + grad_op->SetInput(GradVarName("Out"), OutputGrad("Out")); + grad_op->SetOutput(GradVarName("X"), InputGrad("X")); + grad_op->SetOutput(GradVarName("b"), InputGrad("b")); + grad_op->SetType("rowwise_add_grad"); + return std::unique_ptr(grad_op); + } +}; + +class MulOpMaker : public OpProtoAndCheckerMaker { + public: + MulOpMaker(OpProto *proto, OpAttrChecker *op_checker) + : OpProtoAndCheckerMaker(proto, op_checker) { + AddInput("X", "A"); + AddInput("Y", "B"); + AddOutput("Out", "Out"); + AddAttr("x_num_col_dims", "").SetDefault(1).EqualGreaterThan(1); + AddAttr("y_num_col_dims", "").SetDefault(1).EqualGreaterThan(1); + AddComment("Mul"); + } +}; + +class SigmoidOpMaker : public OpProtoAndCheckerMaker { + public: + SigmoidOpMaker(OpProto *proto, OpAttrChecker *op_checker) + : OpProtoAndCheckerMaker(proto, op_checker) { + AddInput("X", "X"); + AddOutput("Out", "Y"); + AddComment("Sigmoid"); + } +}; + +class NoGradOpMaker : public OpProtoAndCheckerMaker { + public: + NoGradOpMaker(OpProto *proto, OpAttrChecker *op_checker) + : OpProtoAndCheckerMaker(proto, op_checker) { + AddInput("X", "X input"); + AddOutput("Out", "Y output"); + AddComment("NoGradOp, same input output. no Grad"); + } +}; + +class ManyOutputOpMaker : public OpProtoAndCheckerMaker { + public: + ManyOutputOpMaker(OpProto *proto, OpAttrChecker *op_checker) + : OpProtoAndCheckerMaker(proto, op_checker) { + AddInput("x", "x"); + AddOutput("y", "y"); + AddOutput("z", "z"); + AddComment(""); + } +}; + +class FillZeroOpMaker : public OpProtoAndCheckerMaker { + public: + FillZeroOpMaker(OpProto *proto, OpAttrChecker *op_checker) + : OpProtoAndCheckerMaker(proto, op_checker) { + AddInput("X", "x"); + AddOutput("Y", "out"); + AddComment(""); + } +}; + +class SumOpMaker : public framework::OpProtoAndCheckerMaker { + public: + SumOpMaker(framework::OpProto *proto, framework::OpAttrChecker *op_checker) + : OpProtoAndCheckerMaker(proto, op_checker) { + AddInput("X", "the input tensors of sum operator.").AsDuplicable(); + AddOutput("Out", "the output tensor of sum operator."); + AddComment(""); + } +}; + +class MultInOutOpMaker : public OpProtoAndCheckerMaker { + public: + MultInOutOpMaker(OpProto *proto, OpAttrChecker *op_checker) + : OpProtoAndCheckerMaker(proto, op_checker) { + AddInput("X", "x"); + AddInput("H", "h"); + AddOutput("Y", "y"); + AddOutput("Z", "z"); + AddComment(""); + } +}; + +} // namespace framework +} // namespace paddle + +namespace f = paddle::framework; +namespace ops = paddle::operators; +using EnforceNotMet = paddle::platform::EnforceNotMet; +REGISTER_OPERATOR(rowwise_add, f::NOP, f::RowWiseAddOpMaker, + f::RowWiseAddGradMaker); +REGISTER_OPERATOR(rowwise_add_grad, f::NOP); +REGISTER_OP(mul, f::NOP, f::MulOpMaker, mul_grad, f::NOP); +REGISTER_OP(sigmoid, f::NOP, f::SigmoidOpMaker, sigmoid_grad, f::NOP); +REGISTER_OP_WITHOUT_GRADIENT(nograd, f::NOP, f::NoGradOpMaker); +REGISTER_OP_WITHOUT_GRADIENT(fill_zeros_like, f::NOP, f::FillZeroOpMaker); +REGISTER_OP(sum, f::NOP, f::SumOpMaker, sum_grad, f::NOP); +REGISTER_OP(many_output_op, f::NOP, f::ManyOutputOpMaker, many_output_op_grad, + f::NOP); +REGISTER_OP(mult_in_out, f::NOP, f::MultInOutOpMaker, mult_in_out_grad, f::NOP); + +void AddOp(const std::string &type, const f::VariableNameMap &inputs, + const f::VariableNameMap &outputs, f::AttributeMap attrs, + paddle::framework::BlockDescBind *block) { + // insert output + for (auto kv : outputs) { + for (auto v : kv.second) { + auto var = block->NewVar(v); + var->SetDataType(paddle::framework::DataType::FP32); + } + } + + // insert op + auto op = block->AppendOp(); + op->SetType(type); + for (auto &kv : inputs) { + op->SetInput(kv.first, kv.second); + } + for (auto &kv : outputs) { + op->SetOutput(kv.first, kv.second); + } + op->SetAttrMap(attrs); +} + +f::ProgramDesc *GetNewProgramDesc() { + auto *program_desc = new f::ProgramDesc(); + auto *root_block = program_desc->add_blocks(); + root_block->set_idx(0); + root_block->set_parent_idx(-1); + return program_desc; +} + +TEST(Prune, one_operator) { + f::ProgramDesc *program_desc = GetNewProgramDesc(); + f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::BlockDescBind *block = program.Block(0); + + AddOp("mul", {{"X", {"a"}}, {"Y", {"w1"}}}, {{"Out", {"b"}}}, {}, block); + + f::ProgramDesc *pdesc = program.Proto(); + f::ProgramDesc pruned; + + Prune(*pdesc, pruned, 0); + PADDLE_ENFORCE_EQ(pruned.blocks(0).ops_size(), 0); + + pdesc->mutable_blocks(0)->mutable_ops(0)->set_is_target(true); + Prune(*pdesc, pruned, 0); + PADDLE_ENFORCE_EQ(pruned.blocks(0).ops_size(), 1); +} + +TEST(Prune, simple_optimize) {} From fd72e9c7516af791e25ebc50004f297784b87051 Mon Sep 17 00:00:00 2001 From: Yang Yang Date: Thu, 12 Oct 2017 00:57:58 +0000 Subject: [PATCH 05/76] pass multiple unit test --- paddle/framework/prune.cc | 9 +- paddle/framework/prune_test.cc | 175 +++++++++++++-------------------- 2 files changed, 70 insertions(+), 114 deletions(-) diff --git a/paddle/framework/prune.cc b/paddle/framework/prune.cc index ddb9ed7ae0..284541f199 100644 --- a/paddle/framework/prune.cc +++ b/paddle/framework/prune.cc @@ -43,7 +43,7 @@ void Prune(const ProgramDesc& input, ProgramDesc& output, int id) { // TODO(tonyyang-svail): // - will change to use multiple blocks for RNN op and Cond Op - auto& block = input.blocks(0); + auto& block = input.blocks(id); auto& ops = block.ops(); bool expect_feed = true; @@ -67,13 +67,6 @@ void Prune(const ProgramDesc& input, ProgramDesc& output, int id) { auto& op_desc = *op_iter; if (op_desc.is_target() || HasDependentVar(op_desc, dependent_vars)) { - // erase its output to the dependency graph - for (auto& var : op_desc.outputs()) { - for (auto& argu : var.arguments()) { - dependent_vars.erase(argu); - } - } - // insert its input to the dependency graph for (auto& var : op_desc.inputs()) { for (auto& argu : var.arguments()) { diff --git a/paddle/framework/prune_test.cc b/paddle/framework/prune_test.cc index b66db94528..ab08b851d3 100644 --- a/paddle/framework/prune_test.cc +++ b/paddle/framework/prune_test.cc @@ -28,105 +28,24 @@ namespace framework { using DeviceContext = platform::DeviceContext; -class RowWiseAddOpMaker : public OpProtoAndCheckerMaker { +class OneOneOpMaker : public OpProtoAndCheckerMaker { public: - RowWiseAddOpMaker(OpProto *proto, OpAttrChecker *op_checker) + OneOneOpMaker(OpProto *proto, OpAttrChecker *op_checker) : OpProtoAndCheckerMaker(proto, op_checker) { - AddInput("X", "Input X of Add"); - AddInput("b", "Bias of Add"); - AddOutput("Out", "Out of Add"); - AddComment("Add Op"); + AddInput("input", "input"); + AddOutput("output", "output"); + AddComment("Op has one input and one output"); } }; -class RowWiseAddGradMaker : public SingleGradOpDescMaker { +class TwoOneOpMaker : public OpProtoAndCheckerMaker { public: - using SingleGradOpDescMaker::SingleGradOpDescMaker; - - protected: - std::unique_ptr Apply() const override { - auto grad_op = new OpDescBind(); - grad_op->SetInput(GradVarName("Out"), OutputGrad("Out")); - grad_op->SetOutput(GradVarName("X"), InputGrad("X")); - grad_op->SetOutput(GradVarName("b"), InputGrad("b")); - grad_op->SetType("rowwise_add_grad"); - return std::unique_ptr(grad_op); - } -}; - -class MulOpMaker : public OpProtoAndCheckerMaker { - public: - MulOpMaker(OpProto *proto, OpAttrChecker *op_checker) - : OpProtoAndCheckerMaker(proto, op_checker) { - AddInput("X", "A"); - AddInput("Y", "B"); - AddOutput("Out", "Out"); - AddAttr("x_num_col_dims", "").SetDefault(1).EqualGreaterThan(1); - AddAttr("y_num_col_dims", "").SetDefault(1).EqualGreaterThan(1); - AddComment("Mul"); - } -}; - -class SigmoidOpMaker : public OpProtoAndCheckerMaker { - public: - SigmoidOpMaker(OpProto *proto, OpAttrChecker *op_checker) + TwoOneOpMaker(OpProto *proto, OpAttrChecker *op_checker) : OpProtoAndCheckerMaker(proto, op_checker) { - AddInput("X", "X"); - AddOutput("Out", "Y"); - AddComment("Sigmoid"); - } -}; - -class NoGradOpMaker : public OpProtoAndCheckerMaker { - public: - NoGradOpMaker(OpProto *proto, OpAttrChecker *op_checker) - : OpProtoAndCheckerMaker(proto, op_checker) { - AddInput("X", "X input"); - AddOutput("Out", "Y output"); - AddComment("NoGradOp, same input output. no Grad"); - } -}; - -class ManyOutputOpMaker : public OpProtoAndCheckerMaker { - public: - ManyOutputOpMaker(OpProto *proto, OpAttrChecker *op_checker) - : OpProtoAndCheckerMaker(proto, op_checker) { - AddInput("x", "x"); - AddOutput("y", "y"); - AddOutput("z", "z"); - AddComment(""); - } -}; - -class FillZeroOpMaker : public OpProtoAndCheckerMaker { - public: - FillZeroOpMaker(OpProto *proto, OpAttrChecker *op_checker) - : OpProtoAndCheckerMaker(proto, op_checker) { - AddInput("X", "x"); - AddOutput("Y", "out"); - AddComment(""); - } -}; - -class SumOpMaker : public framework::OpProtoAndCheckerMaker { - public: - SumOpMaker(framework::OpProto *proto, framework::OpAttrChecker *op_checker) - : OpProtoAndCheckerMaker(proto, op_checker) { - AddInput("X", "the input tensors of sum operator.").AsDuplicable(); - AddOutput("Out", "the output tensor of sum operator."); - AddComment(""); - } -}; - -class MultInOutOpMaker : public OpProtoAndCheckerMaker { - public: - MultInOutOpMaker(OpProto *proto, OpAttrChecker *op_checker) - : OpProtoAndCheckerMaker(proto, op_checker) { - AddInput("X", "x"); - AddInput("H", "h"); - AddOutput("Y", "y"); - AddOutput("Z", "z"); - AddComment(""); + AddInput("input_1", "input_1"); + AddInput("input_2", "input_2"); + AddOutput("output", "output"); + AddComment("Op has two inputs and one output"); } }; @@ -135,18 +54,8 @@ class MultInOutOpMaker : public OpProtoAndCheckerMaker { namespace f = paddle::framework; namespace ops = paddle::operators; -using EnforceNotMet = paddle::platform::EnforceNotMet; -REGISTER_OPERATOR(rowwise_add, f::NOP, f::RowWiseAddOpMaker, - f::RowWiseAddGradMaker); -REGISTER_OPERATOR(rowwise_add_grad, f::NOP); -REGISTER_OP(mul, f::NOP, f::MulOpMaker, mul_grad, f::NOP); -REGISTER_OP(sigmoid, f::NOP, f::SigmoidOpMaker, sigmoid_grad, f::NOP); -REGISTER_OP_WITHOUT_GRADIENT(nograd, f::NOP, f::NoGradOpMaker); -REGISTER_OP_WITHOUT_GRADIENT(fill_zeros_like, f::NOP, f::FillZeroOpMaker); -REGISTER_OP(sum, f::NOP, f::SumOpMaker, sum_grad, f::NOP); -REGISTER_OP(many_output_op, f::NOP, f::ManyOutputOpMaker, many_output_op_grad, - f::NOP); -REGISTER_OP(mult_in_out, f::NOP, f::MultInOutOpMaker, mult_in_out_grad, f::NOP); +REGISTER_OP_WITHOUT_GRADIENT(one_one, f::NOP, f::OneOneOpMaker); +REGISTER_OP_WITHOUT_GRADIENT(two_one, f::NOP, f::TwoOneOpMaker); void AddOp(const std::string &type, const f::VariableNameMap &inputs, const f::VariableNameMap &outputs, f::AttributeMap attrs, @@ -184,7 +93,7 @@ TEST(Prune, one_operator) { f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); f::BlockDescBind *block = program.Block(0); - AddOp("mul", {{"X", {"a"}}, {"Y", {"w1"}}}, {{"Out", {"b"}}}, {}, block); + AddOp("one_one", {{"input", {"a"}}}, {{"output", {"b"}}}, {}, block); f::ProgramDesc *pdesc = program.Proto(); f::ProgramDesc pruned; @@ -197,4 +106,58 @@ TEST(Prune, one_operator) { PADDLE_ENFORCE_EQ(pruned.blocks(0).ops_size(), 1); } -TEST(Prune, simple_optimize) {} +TEST(Prune, forward) { + f::ProgramDesc *program_desc = GetNewProgramDesc(); + f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::BlockDescBind *block = program.Block(0); + + AddOp("one_one", {{"input", {"a"}}}, {{"output", {"b"}}}, {}, block); + AddOp("one_one", {{"input", {"b"}}}, {{"output", {"c"}}}, {}, block); + AddOp("one_one", {{"input", {"c"}}}, {{"output", {"d"}}}, {}, block); + AddOp("one_one", {{"input", {"d"}}}, {{"output", {"e"}}}, {}, block); + + f::ProgramDesc *pdesc = program.Proto(); + + for (int i = 0; i < pdesc->blocks(0).ops_size(); ++i) { + f::ProgramDesc pruned; + pdesc->mutable_blocks(0)->mutable_ops(i)->set_is_target(true); + Prune(*pdesc, pruned, 0); + PADDLE_ENFORCE_EQ(pruned.blocks(0).ops_size(), i + 1); + } +} + +TEST(Prune, multi_input_op) { + f::ProgramDesc *program_desc = GetNewProgramDesc(); + f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::BlockDescBind *block = program.Block(0); + + AddOp("one_one", {{"input", {"a0"}}}, {{"output", {"b0"}}}, {}, block); + AddOp("one_one", {{"input", {"a1"}}}, {{"output", {"b1"}}}, {}, block); + AddOp("one_one", {{"input", {"a2"}}}, {{"output", {"b2"}}}, {}, block); + AddOp("three_one", {{"input", {"b0", "b1", "b2"}}}, {{"output", {"c"}}}, {}, + block); + + f::ProgramDesc *pdesc = program.Proto(); + pdesc->mutable_blocks(0)->mutable_ops(3)->set_is_target(true); + + f::ProgramDesc pruned; + Prune(*pdesc, pruned, 0); + PADDLE_ENFORCE_EQ(pruned.blocks(0).ops_size(), 4); +} + +TEST(Prune, multi_output_op) { + f::ProgramDesc *program_desc = GetNewProgramDesc(); + f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::BlockDescBind *block = program.Block(0); + + AddOp("one_two", {{"input", {"a"}}}, {{"output", {"b", "c"}}}, {}, block); + AddOp("one_one", {{"input", {"b"}}}, {{"output", {"b1"}}}, {}, block); + AddOp("one_one", {{"input", {"c"}}}, {{"output", {"c1"}}}, {}, block); + + f::ProgramDesc *pdesc = program.Proto(); + pdesc->mutable_blocks(0)->mutable_ops(2)->set_is_target(true); + + f::ProgramDesc pruned; + Prune(*pdesc, pruned, 0); + PADDLE_ENFORCE_EQ(pruned.blocks(0).ops_size(), 2); +} From fc96463b25c1f0bf9d48541bdfb2d0f0cf3e082b Mon Sep 17 00:00:00 2001 From: Yang Yang Date: Thu, 12 Oct 2017 01:15:37 +0000 Subject: [PATCH 06/76] pass multiple target --- paddle/framework/prune_test.cc | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/paddle/framework/prune_test.cc b/paddle/framework/prune_test.cc index ab08b851d3..790fa16924 100644 --- a/paddle/framework/prune_test.cc +++ b/paddle/framework/prune_test.cc @@ -161,3 +161,21 @@ TEST(Prune, multi_output_op) { Prune(*pdesc, pruned, 0); PADDLE_ENFORCE_EQ(pruned.blocks(0).ops_size(), 2); } + +TEST(Prune, multi_target) { + f::ProgramDesc *program_desc = GetNewProgramDesc(); + f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::BlockDescBind *block = program.Block(0); + + AddOp("one_two", {{"input", {"a"}}}, {{"output", {"b", "c"}}}, {}, block); + AddOp("one_one", {{"input", {"b"}}}, {{"output", {"b1"}}}, {}, block); + AddOp("one_one", {{"input", {"c"}}}, {{"output", {"c1"}}}, {}, block); + + f::ProgramDesc *pdesc = program.Proto(); + pdesc->mutable_blocks(0)->mutable_ops(1)->set_is_target(true); + pdesc->mutable_blocks(0)->mutable_ops(2)->set_is_target(true); + + f::ProgramDesc pruned; + Prune(*pdesc, pruned, 0); + PADDLE_ENFORCE_EQ(pruned.blocks(0).ops_size(), 3); +} From 58b8a1ae4c9854ed04483f14c6f93dc0d74b9fcf Mon Sep 17 00:00:00 2001 From: Yang Yang Date: Thu, 12 Oct 2017 02:31:51 +0000 Subject: [PATCH 07/76] prune link fail --- paddle/framework/CMakeLists.txt | 2 +- paddle/framework/prune_test.cc | 2 -- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/paddle/framework/CMakeLists.txt b/paddle/framework/CMakeLists.txt index d9c84f3c0a..1ba23a2c3f 100644 --- a/paddle/framework/CMakeLists.txt +++ b/paddle/framework/CMakeLists.txt @@ -50,7 +50,7 @@ cc_library(executor SRCS executor.cc DEPS op_registry device_context scope frame #endif() cc_library(prune SRCS prune.cc) -cc_test(prune_test SRCS prune_test.cc DEPS prune recurrent_op device_context) +cc_test(prune_test SRCS prune_test.cc DEPS op_info prune recurrent_op device_context) cc_library(tensor_array SRCS tensor_array.cc DEPS lod_tensor) cc_test(tensor_array_test SRCS tensor_array_test.cc DEPS tensor_array place) diff --git a/paddle/framework/prune_test.cc b/paddle/framework/prune_test.cc index 790fa16924..c351c12d22 100644 --- a/paddle/framework/prune_test.cc +++ b/paddle/framework/prune_test.cc @@ -54,8 +54,6 @@ class TwoOneOpMaker : public OpProtoAndCheckerMaker { namespace f = paddle::framework; namespace ops = paddle::operators; -REGISTER_OP_WITHOUT_GRADIENT(one_one, f::NOP, f::OneOneOpMaker); -REGISTER_OP_WITHOUT_GRADIENT(two_one, f::NOP, f::TwoOneOpMaker); void AddOp(const std::string &type, const f::VariableNameMap &inputs, const f::VariableNameMap &outputs, f::AttributeMap attrs, From 7c48335b7cfe257c30b6ccc7991151d441859175 Mon Sep 17 00:00:00 2001 From: Yang Yang Date: Mon, 16 Oct 2017 17:44:57 +0000 Subject: [PATCH 08/76] merge fix linking --- paddle/framework/prune_test.cc | 38 +++++----------------------------- 1 file changed, 5 insertions(+), 33 deletions(-) diff --git a/paddle/framework/prune_test.cc b/paddle/framework/prune_test.cc index c351c12d22..dc066facb2 100644 --- a/paddle/framework/prune_test.cc +++ b/paddle/framework/prune_test.cc @@ -14,43 +14,15 @@ #include "paddle/framework/prune.h" -#include #include "paddle/framework/attribute.h" -#include "paddle/framework/block_desc.h" -#include "paddle/framework/op_desc.h" -#include "paddle/framework/op_registry.h" #include "paddle/framework/operator.h" -#include "paddle/framework/program_desc.h" #include "paddle/operators/net_op.h" -namespace paddle { -namespace framework { - -using DeviceContext = platform::DeviceContext; - -class OneOneOpMaker : public OpProtoAndCheckerMaker { - public: - OneOneOpMaker(OpProto *proto, OpAttrChecker *op_checker) - : OpProtoAndCheckerMaker(proto, op_checker) { - AddInput("input", "input"); - AddOutput("output", "output"); - AddComment("Op has one input and one output"); - } -}; - -class TwoOneOpMaker : public OpProtoAndCheckerMaker { - public: - TwoOneOpMaker(OpProto *proto, OpAttrChecker *op_checker) - : OpProtoAndCheckerMaker(proto, op_checker) { - AddInput("input_1", "input_1"); - AddInput("input_2", "input_2"); - AddOutput("output", "output"); - AddComment("Op has two inputs and one output"); - } -}; +#include "paddle/framework/block_desc.h" +#include "paddle/framework/op_desc.h" +#include "paddle/framework/program_desc.h" -} // namespace framework -} // namespace paddle +#include namespace f = paddle::framework; namespace ops = paddle::operators; @@ -61,7 +33,7 @@ void AddOp(const std::string &type, const f::VariableNameMap &inputs, // insert output for (auto kv : outputs) { for (auto v : kv.second) { - auto var = block->NewVar(v); + auto var = block->Var(v); var->SetDataType(paddle::framework::DataType::FP32); } } From a64a6f527b5c170b726c205cb6548b19171d5810 Mon Sep 17 00:00:00 2001 From: Yang Yang Date: Mon, 16 Oct 2017 18:17:25 +0000 Subject: [PATCH 09/76] id to block_id --- paddle/framework/prune.cc | 8 ++++---- paddle/framework/prune.h | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/paddle/framework/prune.cc b/paddle/framework/prune.cc index 284541f199..c9a1d7d5cf 100644 --- a/paddle/framework/prune.cc +++ b/paddle/framework/prune.cc @@ -39,11 +39,11 @@ bool HasDependentVar(const OpDesc& op_desc, return false; } -void Prune(const ProgramDesc& input, ProgramDesc& output, int id) { +void Prune(const ProgramDesc& input, ProgramDesc& output, int block_id) { // TODO(tonyyang-svail): // - will change to use multiple blocks for RNN op and Cond Op - auto& block = input.blocks(id); + auto& block = input.blocks(block_id); auto& ops = block.ops(); bool expect_feed = true; @@ -85,11 +85,11 @@ void Prune(const ProgramDesc& input, ProgramDesc& output, int id) { std::reverse(should_run.begin(), should_run.end()); output = input; - auto* op_field = output.mutable_blocks(id)->mutable_ops(); + auto* op_field = output.mutable_blocks(block_id)->mutable_ops(); op_field->Clear(); for (size_t i = 0; i < should_run.size(); ++i) { if (should_run[i]) { - *op_field->Add() = input.blocks(id).ops(i); + *op_field->Add() = input.blocks(block_id).ops(i); } } diff --git a/paddle/framework/prune.h b/paddle/framework/prune.h index 3e1d58f61f..1c74d3b763 100644 --- a/paddle/framework/prune.h +++ b/paddle/framework/prune.h @@ -20,7 +20,7 @@ limitations under the License. */ namespace paddle { namespace framework { -void Prune(const ProgramDesc& input, ProgramDesc& output, int id); +void Prune(const ProgramDesc& input, ProgramDesc& output, int block_id); } // namespace framework } // namespace paddle From 865c2c8ed870a35369c2914d7723f6359d6e8c49 Mon Sep 17 00:00:00 2001 From: Yang Yang Date: Mon, 16 Oct 2017 19:38:39 +0000 Subject: [PATCH 10/76] add compile DEPS --- paddle/framework/CMakeLists.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/paddle/framework/CMakeLists.txt b/paddle/framework/CMakeLists.txt index 00a9802ef8..9d039a54d6 100644 --- a/paddle/framework/CMakeLists.txt +++ b/paddle/framework/CMakeLists.txt @@ -51,7 +51,7 @@ else() cc_test(executor_test SRCS executor_test.cc DEPS executor ${EXECUTOR_TEST_OP}) endif() -cc_library(prune SRCS prune.cc) +cc_library(prune SRCS prune.cc DEPS framework_proto) cc_test(prune_test SRCS prune_test.cc DEPS op_info prune recurrent_op device_context) cc_library(tensor_array SRCS tensor_array.cc DEPS lod_tensor) From e0cee58c844ff7fdabdad9fe0a0e25341384bfdf Mon Sep 17 00:00:00 2001 From: Yang Yang Date: Tue, 17 Oct 2017 02:48:35 +0000 Subject: [PATCH 11/76] modify protobuf --- paddle/framework/framework.proto | 2 +- paddle/framework/prune.cc | 9 ++++++++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/paddle/framework/framework.proto b/paddle/framework/framework.proto index fd4c0440eb..008fb45fb7 100644 --- a/paddle/framework/framework.proto +++ b/paddle/framework/framework.proto @@ -55,7 +55,7 @@ message OpDesc { repeated Var inputs = 1; repeated Var outputs = 2; repeated Attr attrs = 4; - required bool is_target = 5 [ default = false ]; + optional bool is_target = 5 [ default = false ]; }; // OpProto describes a C++ framework::OperatorBase derived class. diff --git a/paddle/framework/prune.cc b/paddle/framework/prune.cc index c9a1d7d5cf..b08e0116b7 100644 --- a/paddle/framework/prune.cc +++ b/paddle/framework/prune.cc @@ -39,6 +39,13 @@ bool HasDependentVar(const OpDesc& op_desc, return false; } +bool IsTarget(const OpDesc& op_desc) { + if (op_desc.has_is_target()) { + return op_desc.is_target(); + } + return false; +} + void Prune(const ProgramDesc& input, ProgramDesc& output, int block_id) { // TODO(tonyyang-svail): // - will change to use multiple blocks for RNN op and Cond Op @@ -66,7 +73,7 @@ void Prune(const ProgramDesc& input, ProgramDesc& output, int block_id) { for (auto op_iter = ops.rbegin(); op_iter != ops.rend(); ++op_iter) { auto& op_desc = *op_iter; - if (op_desc.is_target() || HasDependentVar(op_desc, dependent_vars)) { + if (IsTarget(op_desc) || HasDependentVar(op_desc, dependent_vars)) { // insert its input to the dependency graph for (auto& var : op_desc.inputs()) { for (auto& argu : var.arguments()) { From bdca4b37c434b26b2c6ae300899a1c562a82e133 Mon Sep 17 00:00:00 2001 From: Yang Yang Date: Tue, 17 Oct 2017 02:58:08 +0000 Subject: [PATCH 12/76] change api based on design doc --- paddle/framework/prune.cc | 6 ++++-- paddle/framework/prune.h | 2 +- paddle/framework/prune_test.cc | 12 ++++++------ 3 files changed, 11 insertions(+), 9 deletions(-) diff --git a/paddle/framework/prune.cc b/paddle/framework/prune.cc index b08e0116b7..9583369292 100644 --- a/paddle/framework/prune.cc +++ b/paddle/framework/prune.cc @@ -46,7 +46,7 @@ bool IsTarget(const OpDesc& op_desc) { return false; } -void Prune(const ProgramDesc& input, ProgramDesc& output, int block_id) { +void prune_impl(const ProgramDesc& input, ProgramDesc& output, int block_id) { // TODO(tonyyang-svail): // - will change to use multiple blocks for RNN op and Cond Op @@ -99,8 +99,10 @@ void Prune(const ProgramDesc& input, ProgramDesc& output, int block_id) { *op_field->Add() = input.blocks(block_id).ops(i); } } +} - // return should_run; +void Prune(const ProgramDesc& input, ProgramDesc& output) { + prune_impl(input, output, 0); } } // namespace framework diff --git a/paddle/framework/prune.h b/paddle/framework/prune.h index 1c74d3b763..9414ac64f9 100644 --- a/paddle/framework/prune.h +++ b/paddle/framework/prune.h @@ -20,7 +20,7 @@ limitations under the License. */ namespace paddle { namespace framework { -void Prune(const ProgramDesc& input, ProgramDesc& output, int block_id); +void Prune(const ProgramDesc& input, ProgramDesc& output); } // namespace framework } // namespace paddle diff --git a/paddle/framework/prune_test.cc b/paddle/framework/prune_test.cc index dc066facb2..a8faf1891e 100644 --- a/paddle/framework/prune_test.cc +++ b/paddle/framework/prune_test.cc @@ -68,11 +68,11 @@ TEST(Prune, one_operator) { f::ProgramDesc *pdesc = program.Proto(); f::ProgramDesc pruned; - Prune(*pdesc, pruned, 0); + Prune(*pdesc, pruned); PADDLE_ENFORCE_EQ(pruned.blocks(0).ops_size(), 0); pdesc->mutable_blocks(0)->mutable_ops(0)->set_is_target(true); - Prune(*pdesc, pruned, 0); + Prune(*pdesc, pruned); PADDLE_ENFORCE_EQ(pruned.blocks(0).ops_size(), 1); } @@ -91,7 +91,7 @@ TEST(Prune, forward) { for (int i = 0; i < pdesc->blocks(0).ops_size(); ++i) { f::ProgramDesc pruned; pdesc->mutable_blocks(0)->mutable_ops(i)->set_is_target(true); - Prune(*pdesc, pruned, 0); + Prune(*pdesc, pruned); PADDLE_ENFORCE_EQ(pruned.blocks(0).ops_size(), i + 1); } } @@ -111,7 +111,7 @@ TEST(Prune, multi_input_op) { pdesc->mutable_blocks(0)->mutable_ops(3)->set_is_target(true); f::ProgramDesc pruned; - Prune(*pdesc, pruned, 0); + Prune(*pdesc, pruned); PADDLE_ENFORCE_EQ(pruned.blocks(0).ops_size(), 4); } @@ -128,7 +128,7 @@ TEST(Prune, multi_output_op) { pdesc->mutable_blocks(0)->mutable_ops(2)->set_is_target(true); f::ProgramDesc pruned; - Prune(*pdesc, pruned, 0); + Prune(*pdesc, pruned); PADDLE_ENFORCE_EQ(pruned.blocks(0).ops_size(), 2); } @@ -146,6 +146,6 @@ TEST(Prune, multi_target) { pdesc->mutable_blocks(0)->mutable_ops(2)->set_is_target(true); f::ProgramDesc pruned; - Prune(*pdesc, pruned, 0); + Prune(*pdesc, pruned); PADDLE_ENFORCE_EQ(pruned.blocks(0).ops_size(), 3); } From c7ebe0e134d4c9a22bc10b14d0752b7c640e2197 Mon Sep 17 00:00:00 2001 From: "Yang Yang(Tony)" Date: Tue, 17 Oct 2017 10:34:11 -0700 Subject: [PATCH 13/76] Update refactorization.md --- doc/design/refactorization.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/doc/design/refactorization.md b/doc/design/refactorization.md index bf24022504..f93d6155e1 100644 --- a/doc/design/refactorization.md +++ b/doc/design/refactorization.md @@ -185,9 +185,6 @@ REGISTER_OP_WITHOUT_GRADIENT(op_type, op_class, op_maker_class) 1. Call maker class to complete `proto` and `checker` 2. Using the completed `proto` and `checker`, it will add a new key-value pair to the `OpInfoMap` - -4. Invoke the `USE` macro in which the Op is used to make sure that it is linked. - --- # Backward Module (1/2) ### Create Backward Operator From 182ce51c6d73d98420aa91d998a328503eac538d Mon Sep 17 00:00:00 2001 From: qijun Date: Tue, 17 Oct 2017 14:48:40 -0700 Subject: [PATCH 14/76] add sparse kernel of sgd operator --- paddle/operators/sgd_op.cc | 40 ++++++++++++++++++++++--- paddle/operators/sgd_op.cu | 60 ++++++++++++++++++++++++++++++++++++++ paddle/operators/sgd_op.h | 47 ++++++++++++++++++++--------- 3 files changed, 130 insertions(+), 17 deletions(-) diff --git a/paddle/operators/sgd_op.cc b/paddle/operators/sgd_op.cc index 0f78eeab9b..e26a1c7893 100644 --- a/paddle/operators/sgd_op.cc +++ b/paddle/operators/sgd_op.cc @@ -21,7 +21,7 @@ class SGDOp : public framework::OperatorWithKernel { public: using framework::OperatorWithKernel::OperatorWithKernel; - void InferShape(framework::InferShapeContext *ctx) const override { + void InferShape(framework::InferShapeContext* ctx) const override { PADDLE_ENFORCE(ctx->HasInput("Param"), "Input(Param) of SGDOp should not be null."); PADDLE_ENFORCE(ctx->HasInput("Grad"), @@ -35,15 +35,15 @@ class SGDOp : public framework::OperatorWithKernel { PADDLE_ENFORCE_EQ(framework::product(lr_dims), 1, "Learning rate should have 1 element"); auto param_dim = ctx->GetInputDim("Param"); - PADDLE_ENFORCE_EQ(param_dim, ctx->GetInputDim("Grad"), - "Two input of SGD Op's dimension must be same."); + // TODO(qijun): check dimensions of Param and Grad at complie + // and run time. ctx->SetOutputDim("ParamOut", param_dim); } }; class SGDOpMaker : public framework::OpProtoAndCheckerMaker { public: - SGDOpMaker(framework::OpProto *proto, framework::OpAttrChecker *op_checker) + SGDOpMaker(framework::OpProto* proto, framework::OpAttrChecker* op_checker) : OpProtoAndCheckerMaker(proto, op_checker) { AddInput("Param", "Input parameter"); AddInput("LearningRate", "Learning rate of SGD"); @@ -58,6 +58,38 @@ param_out = param - learning_rate * grad; )DOC"); } }; + +template +struct SparseSGDFunctor { + void operator()(const platform::DeviceContext& ctx, + const framework::SelectedRows& input, + const framework::Tensor& learning_rate, + framework::Tensor* output) { + auto in_height = input.height(); + auto out_dims = output->dims(); + PADDLE_ENFORCE_EQ(in_height, out_dims[0]); + + auto& in_value = input.value(); + auto& in_rows = input.rows(); + + int64_t in_row_numel = in_value.numel() / in_rows.size(); + PADDLE_ENFORCE_EQ(in_row_numel, output->numel() / in_height); + + auto* in_data = in_value.data(); + auto* out_data = output->data(); + auto* lr = learning_rate.data(); + + for (size_t i = 0; i < in_rows.size(); i++) { + for (int64_t j = 0; j < in_row_numel; j++) { + out_data[in_rows[i] * in_row_numel + j] -= + lr[0] * in_data[i * in_row_numel + j]; + } + } + } +}; + +template struct SparseSGDFunctor; + } // namespace operators } // namespace paddle diff --git a/paddle/operators/sgd_op.cu b/paddle/operators/sgd_op.cu index f5ba6d3c29..5c28314141 100644 --- a/paddle/operators/sgd_op.cu +++ b/paddle/operators/sgd_op.cu @@ -14,6 +14,66 @@ #define EIGEN_USE_GPU #include "paddle/operators/sgd_op.h" +#include "paddle/platform/cuda_helper.h" + +namespace paddle { +namespace operators { + +namespace { +template +__global__ void SparseSGDFunctorKernel(const T* selected_rows, + const int64_t* rows, + const T* learning_rate, T* tensor_out, + int64_t row_numel, int block_size) { + const int ty = blockIdx.y; + int tid = threadIdx.x; + + selected_rows += ty * row_numel; + tensor_out += rows[ty] * row_numel; + + for (int index = tid; index < row_numel; index += block_size) { + // Since index in rows of SelectedRows can be duplicate, we have to use + // Atomic Operation to avoid concurrent write error. + paddle::platform::CudaAtomicSub(tensor_out + index, + learning_rate[0] * selected_rows[index]); + } +} +} // namespace + +template +struct SparseSGDFunctor { + void operator()(const platform::DeviceContext& ctx, + const framework::SelectedRows& input, + const framework::Tensor& learning_rate, + framework::Tensor* output) { + auto in_height = input.height(); + auto out_dims = output->dims(); + PADDLE_ENFORCE_EQ(in_height, out_dims[0]); + + auto& in_value = input.value(); + auto& in_rows = input.rows(); + + int64_t in_row_numel = in_value.numel() / in_rows.size(); + PADDLE_ENFORCE_EQ(in_row_numel, output->numel() / in_height); + + auto* in_data = in_value.data(); + auto* out_data = output->data(); + + int block_size = 256; + dim3 threads(block_size, 1); + dim3 grid(1, in_rows.size()); + SparseSGDFunctorKernel< + T><<(context) + .stream()>>>(in_data, in_rows.data(), learning_rate.data(), + out_data, in_row_numel, block_size); + } +}; + +template struct SparseSGDFunctor; + +} // namespace operators +} // namespace paddle namespace ops = paddle::operators; REGISTER_OP_GPU_KERNEL(sgd, diff --git a/paddle/operators/sgd_op.h b/paddle/operators/sgd_op.h index 26f4012f25..a872d7f749 100644 --- a/paddle/operators/sgd_op.h +++ b/paddle/operators/sgd_op.h @@ -15,31 +15,52 @@ limitations under the License. */ #pragma once #include "paddle/framework/eigen.h" #include "paddle/framework/op_registry.h" +#include "paddle/framework/selected_rows.h" namespace paddle { namespace operators { +template +struct SparseSGDFunctor { + void operator()(const platform::DeviceContext& ctx, + const framework::SelectedRows& input, + const framework::Tensor& learning_rate, + framework::Tensor* output); +}; + template class SGDOpKernel : public framework::OpKernel { public: void Compute(const framework::ExecutionContext& ctx) const override { - auto param = ctx.Input("Param"); - auto grad = ctx.Input("Grad"); - auto param_out = ctx.Output("ParamOut"); - auto learning_rate = ctx.Input("LearningRate"); + auto* param = ctx.Input("Param"); + auto* param_out = ctx.Output("ParamOut"); + auto* learning_rate = ctx.Input("LearningRate"); - param_out->mutable_data(ctx.GetPlace()); + auto* grad_var = ctx.InputVar("Grad"); + if (grad_var->IsType()) { + param_out->mutable_data(ctx.GetPlace()); + auto* grad = ctx.Input("Grad"); - auto p = framework::EigenVector::Flatten(*param); - auto g = framework::EigenVector::Flatten(*grad); - auto o = framework::EigenVector::Flatten(*param_out); - auto lr = framework::EigenVector::Flatten(*learning_rate); - auto place = ctx.GetEigenDevice(); + auto p = framework::EigenVector::Flatten(*param); + auto g = framework::EigenVector::Flatten(*grad); + auto o = framework::EigenVector::Flatten(*param_out); + auto lr = framework::EigenVector::Flatten(*learning_rate); + auto place = ctx.GetEigenDevice(); - Eigen::DSizes grad_dsize(grad->numel()); - o.device(place) = p - lr.broadcast(grad_dsize) * g; + Eigen::DSizes grad_dsize(grad->numel()); + o.device(place) = p - lr.broadcast(grad_dsize) * g; + } else if (grad_var->IsType()) { + // TODO(qijun): In Sparse SGD operator, in-place update is enforced. + // This manual optimization brings difficulty to track data dependency. + // It's better to find a more elegant solution. + PADDLE_ENFORCE_EQ(param, param_out); + auto* grad = ctx.Input("Grad"); + SparseSGDFunctor functor; + functor(ctx.device_context(), *grad, *learning_rate, param_out); + } else { + PADDLE_THROW("Unsupported Variable Type of Grad"); + } } }; - } // namespace operators } // namespace paddle From ab8cc401e61dd49d393a72903a427ea6fa14bec7 Mon Sep 17 00:00:00 2001 From: qijun Date: Tue, 17 Oct 2017 16:05:05 -0700 Subject: [PATCH 15/76] add sparse sgd operator unittest --- paddle/operators/sgd_op.h | 3 +- paddle/pybind/pybind.cc | 5 ++ .../v2/framework/tests/test_selected_rows.py | 23 +++---- .../paddle/v2/framework/tests/test_sgd_op.py | 60 +++++++++++++++++++ 4 files changed, 79 insertions(+), 12 deletions(-) diff --git a/paddle/operators/sgd_op.h b/paddle/operators/sgd_op.h index a872d7f749..8c28d5e66b 100644 --- a/paddle/operators/sgd_op.h +++ b/paddle/operators/sgd_op.h @@ -37,7 +37,8 @@ class SGDOpKernel : public framework::OpKernel { auto* learning_rate = ctx.Input("LearningRate"); auto* grad_var = ctx.InputVar("Grad"); - if (grad_var->IsType()) { + // Actually, all tensors are LoDTensor except SelectedRows. + if (grad_var->IsType()) { param_out->mutable_data(ctx.GetPlace()); auto* grad = ctx.Input("Grad"); diff --git a/paddle/pybind/pybind.cc b/paddle/pybind/pybind.cc index fcae92ad99..65e265b614 100644 --- a/paddle/pybind/pybind.cc +++ b/paddle/pybind/pybind.cc @@ -186,6 +186,11 @@ All parameter, weight, gradient are variables in Paddle. return self.GetMutable(); }, py::return_value_policy::reference) + .def("get_selected_rows", + [](Variable &self) -> SelectedRows * { + return self.GetMutable(); + }, + py::return_value_policy::reference) .def("get_net", [](Variable &self) -> operators::NetOp * { return self.GetMutable(); diff --git a/python/paddle/v2/framework/tests/test_selected_rows.py b/python/paddle/v2/framework/tests/test_selected_rows.py index 661e818179..e8a930cb08 100644 --- a/python/paddle/v2/framework/tests/test_selected_rows.py +++ b/python/paddle/v2/framework/tests/test_selected_rows.py @@ -8,29 +8,30 @@ class TestSelectedRows(unittest.TestCase): place = core.CPUPlace() height = 10 rows = [0, 4, 7] - row_numel = 10 - selcted_rows = core.SelectedRows(rows, row_numel) - np_array = np.ones((len(rows), height)).astype("float32") + row_numel = 12 + selected_rows = core.SelectedRows(rows, height) + np_array = np.ones((len(rows), row_numel)).astype("float32") np_array[0, 0] = 2.0 np_array[2, 8] = 4.0 - tensor = selcted_rows.get_tensor() + tensor = selected_rows.get_tensor() tensor.set(np_array, place) # compare rows - self.assertEqual(0, selcted_rows.rows()[0]) - self.assertEqual(4, selcted_rows.rows()[1]) - self.assertEqual(7, selcted_rows.rows()[2]) + self.assertEqual(0, selected_rows.rows()[0]) + self.assertEqual(4, selected_rows.rows()[1]) + self.assertEqual(7, selected_rows.rows()[2]) # compare height - self.assertEqual(10, selcted_rows.height()) + self.assertEqual(10, selected_rows.height()) # compare tensor self.assertAlmostEqual(2.0, - selcted_rows.get_tensor().get_float_element(0)) + selected_rows.get_tensor().get_float_element(0)) self.assertAlmostEqual(1.0, - selcted_rows.get_tensor().get_float_element(1)) + selected_rows.get_tensor().get_float_element(1)) self.assertAlmostEqual( - 4.0, selcted_rows.get_tensor().get_float_element(2 * row_numel + 8)) + 4.0, + selected_rows.get_tensor().get_float_element(2 * row_numel + 8)) if __name__ == "__main__": diff --git a/python/paddle/v2/framework/tests/test_sgd_op.py b/python/paddle/v2/framework/tests/test_sgd_op.py index 2dd881e5e1..c7d6a3b345 100644 --- a/python/paddle/v2/framework/tests/test_sgd_op.py +++ b/python/paddle/v2/framework/tests/test_sgd_op.py @@ -1,5 +1,7 @@ import unittest import numpy as np +import paddle.v2.framework.core as core +from paddle.v2.framework.op import Operator from op_test import OpTest @@ -17,5 +19,63 @@ class TestSGDOp(OpTest): self.check_output() +class TestSparseSGDOp(unittest.TestCase): + def test_sparse_sgd(self): + scope = core.Scope() + + # create and initialize Grad Variable + place = core.CPUPlace() + height = 10 + rows = [0, 4, 7] + row_numel = 12 + + grad_selected_rows = scope.var('Grad').get_selected_rows() + grad_selected_rows.set_height(height) + grad_selected_rows.set_rows(rows) + np_array = np.ones((len(rows), row_numel)).astype("float32") + np_array[0, 0] = 2.0 + np_array[2, 8] = 4.0 + grad_tensor = grad_selected_rows.get_tensor() + grad_tensor.set(np_array, place) + + # create and initialize Param Variable + param = scope.var('Param').get_tensor() + param_array = np.full((height, row_numel), 5.0).astype("float32") + param.set(param_array, place) + + # create and initialize LeraningRate Variable + lr = scope.var('LearningRate').get_tensor() + lr_array = np.full((1), 2.0).astype("float32") + lr.set(lr_array, place) + + # create and run sgd operator + sgd_op = Operator( + "sgd", + Param='Param', + Grad='Grad', + ParamOut='Param', + LearningRate='LearningRate') + ctx = core.DeviceContext.create(place) + sgd_op.run(scope, ctx) + + # get and compare result + result_array = np.array(param) + + # rows[0] = 0, 5.0 - 2.0 * 2.0 + self.assertAlmostEqual(1.0, result_array[rows[0], 0]) + # rows[0] = 0, 5.0 - 2.0 * 1.0 + self.assertAlmostEqual(3.0, result_array[rows[0], 2]) + # 5.0 - 2.0 * 0.0 + self.assertAlmostEqual(5.0, result_array[1, 0]) + # rows[1] = 4, 5.0 - 2.0 * 1.0 + self.assertAlmostEqual(3.0, result_array[rows[1], 10]) + # 5.0 - 2.0 * 0.0 + self.assertAlmostEqual(5.0, result_array[5, 8]) + # rows[2] = 7, 5.0 - 2.0 * 1.0 + self.assertAlmostEqual(3.0, result_array[rows[2], 1]) + # rows[2] = 7, 5.0 - 2.0 * 4.0 + self.assertAlmostEqual(-3.0, result_array[rows[2], 8]) + + if __name__ == "__main__": unittest.main() From f9681459b2075e8067e6bda45a62967fc4baec62 Mon Sep 17 00:00:00 2001 From: qijun Date: Tue, 17 Oct 2017 16:33:52 -0700 Subject: [PATCH 16/76] fix gpu build error --- paddle/operators/sgd_op.cc | 2 +- paddle/operators/sgd_op.cu | 6 +++--- paddle/operators/sgd_op.h | 2 +- paddle/pybind/pybind.cc | 10 +++++++++- python/paddle/v2/framework/tests/test_sgd_op.py | 11 +++++++++-- 5 files changed, 23 insertions(+), 8 deletions(-) diff --git a/paddle/operators/sgd_op.cc b/paddle/operators/sgd_op.cc index e26a1c7893..2acb96d1b4 100644 --- a/paddle/operators/sgd_op.cc +++ b/paddle/operators/sgd_op.cc @@ -61,7 +61,7 @@ param_out = param - learning_rate * grad; template struct SparseSGDFunctor { - void operator()(const platform::DeviceContext& ctx, + void operator()(const platform::DeviceContext& context, const framework::SelectedRows& input, const framework::Tensor& learning_rate, framework::Tensor* output) { diff --git a/paddle/operators/sgd_op.cu b/paddle/operators/sgd_op.cu index 5c28314141..106f9b746b 100644 --- a/paddle/operators/sgd_op.cu +++ b/paddle/operators/sgd_op.cu @@ -34,15 +34,15 @@ __global__ void SparseSGDFunctorKernel(const T* selected_rows, for (int index = tid; index < row_numel; index += block_size) { // Since index in rows of SelectedRows can be duplicate, we have to use // Atomic Operation to avoid concurrent write error. - paddle::platform::CudaAtomicSub(tensor_out + index, - learning_rate[0] * selected_rows[index]); + paddle::platform::CudaAtomicAdd( + tensor_out + index, -1.0 * learning_rate[0] * selected_rows[index]); } } } // namespace template struct SparseSGDFunctor { - void operator()(const platform::DeviceContext& ctx, + void operator()(const platform::DeviceContext& context, const framework::SelectedRows& input, const framework::Tensor& learning_rate, framework::Tensor* output) { diff --git a/paddle/operators/sgd_op.h b/paddle/operators/sgd_op.h index 8c28d5e66b..78b595fc6c 100644 --- a/paddle/operators/sgd_op.h +++ b/paddle/operators/sgd_op.h @@ -22,7 +22,7 @@ namespace operators { template struct SparseSGDFunctor { - void operator()(const platform::DeviceContext& ctx, + void operator()(const platform::DeviceContext& context, const framework::SelectedRows& input, const framework::Tensor& learning_rate, framework::Tensor* output); diff --git a/paddle/pybind/pybind.cc b/paddle/pybind/pybind.cc index 65e265b614..80854fb0c5 100644 --- a/paddle/pybind/pybind.cc +++ b/paddle/pybind/pybind.cc @@ -153,7 +153,15 @@ PYBIND11_PLUGIN(core) { py::return_value_policy::reference) .def("set_height", &SelectedRows::set_height) .def("height", &SelectedRows::height) - .def("set_rows", &SelectedRows::set_rows) + .def("set_rows", + [](SelectedRows &self, std::vector rows) { +#ifndef PADDLE_WITH_CUDA + self.set_rows(rows); +#else + Vector new_rows(rows); + self.set_rows(new_rows); +#endif + }) .def("rows", [](SelectedRows &self) { #ifndef PADDLE_WITH_CUDA return self.rows(); diff --git a/python/paddle/v2/framework/tests/test_sgd_op.py b/python/paddle/v2/framework/tests/test_sgd_op.py index c7d6a3b345..01262bba4d 100644 --- a/python/paddle/v2/framework/tests/test_sgd_op.py +++ b/python/paddle/v2/framework/tests/test_sgd_op.py @@ -20,11 +20,10 @@ class TestSGDOp(OpTest): class TestSparseSGDOp(unittest.TestCase): - def test_sparse_sgd(self): + def check_with_place(self, place): scope = core.Scope() # create and initialize Grad Variable - place = core.CPUPlace() height = 10 rows = [0, 4, 7] row_numel = 12 @@ -35,6 +34,7 @@ class TestSparseSGDOp(unittest.TestCase): np_array = np.ones((len(rows), row_numel)).astype("float32") np_array[0, 0] = 2.0 np_array[2, 8] = 4.0 + grad_tensor = grad_selected_rows.get_tensor() grad_tensor.set(np_array, place) @@ -76,6 +76,13 @@ class TestSparseSGDOp(unittest.TestCase): # rows[2] = 7, 5.0 - 2.0 * 4.0 self.assertAlmostEqual(-3.0, result_array[rows[2], 8]) + def test_sparse_sgd(self): + places = [core.CPUPlace()] + if core.is_compile_gpu(): + places.append(core.GPUPlace(0)) + for place in places: + self.check_with_place(place) + if __name__ == "__main__": unittest.main() From cd099c72627790104db0feaa7028fd551d5fc5c0 Mon Sep 17 00:00:00 2001 From: Liu Yiqun Date: Wed, 18 Oct 2017 14:02:20 +0800 Subject: [PATCH 17/76] Remove the words related to the support of Android 16. --- .../cross_compiling_for_android_cn.md | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-) diff --git a/doc/howto/cross_compiling/cross_compiling_for_android_cn.md b/doc/howto/cross_compiling/cross_compiling_for_android_cn.md index 0e67c12c57..1fc58c37cc 100644 --- a/doc/howto/cross_compiling/cross_compiling_for_android_cn.md +++ b/doc/howto/cross_compiling/cross_compiling_for_android_cn.md @@ -23,27 +23,17 @@ Android的Docker开发镜像向用户提供两个可配置的参数： | Argument | Optional Values | Default | |-----------------|-------------------------|---------| |`ANDROID_ABI` |`armeabi-v7a, arm64-v8a` | `armeabi-v7a` | -|`ANDROID_API` |`armeabi-v7a(>15), arm64-v8a(>20)` | `21` | +|`ANDROID_API` |`>= 21` | `21` | - 编译`armeabi-v7a`，`Android API 21`的PaddlePaddle库 ```bash $ docker run -it --rm -v $PWD:/paddle -e "ANDROID_ABI=armeabi-v7a" -e "ANDROID_API=21" username/paddle-android:dev ``` -或 -```bash -$ docker run -it --rm -v $PWD:/paddle username/paddle-android:dev -``` - -如需编译Android API低于21的Paddle库，请参考本文档的**准备交叉编译环境**章节。 - 编译`arm64-v8a`，`Android API 21`的PaddlePaddle库 ```bash $ docker run -it --rm -v $PWD:/paddle -e "ANDROID_ABI=arm64-v8a" -e "ANDROID_API=21" username/paddle-android:dev ``` -或 -```bash -$ docker run -it --rm -v $PWD:/paddle -e "ANDROID_ABI=arm64-v8a" username/paddle-android:dev -``` 执行上述`docker run`命令时，容器默认执行[paddle/scripts/docker/build_android.sh](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/scripts/docker/build_android.sh)脚本。该脚本中记录了交叉编译Android版PaddlePaddle库常用的CMake配置，并且会根据`ANDROID_ABI`和`ANDROID_API`自动构建独立工具链、进行编译和安装。由于arm64架构要求Android API不小于21。因此当`ANDROID_ABI=arm64-v8a`，`ANDROID_API<21`时，Docker容器中将默认使用`Android API 21`的编译工具链。用户可以参考下文**配置交叉编译参数**章节，根据个人的需求修改定制Docker容器所执行的脚本。编译安装结束之后，PaddlePaddle的C-API库将被安装到`$PWD/install_android`目录，所依赖的第三方库同时也被安装到`$PWD/install_android/third_party`目录。 @@ -70,8 +60,6 @@ your/path/to/android-ndk-r14b-linux-x86_64/build/tools/make-standalone-toolchain 此命令将在`your/path/to/arm_standalone_toolchain`目录生成一套独立编译工具链，面向架构为32位ARM架构，支持的最小的Android API级别为21，支持编译器`arm-linux-androideabi-gcc (GCC) 4.9`和`clang 3.8`。 -注意：**PaddlePaddle要求使用的编译工具链所支持的Andoid API级别不小于16**。但由于PaddlePaddle所依赖的第三方库`glog`不支持低于21的Android API，所以在编译Android API低于21的Paddle C-API库时，需要将[cmake/external/glog.cmake](https://github.com/PaddlePaddle/Paddle/blob/develop/cmake/external/glog.cmake#L33)中的`GIT_REPOSITORY`临时修改为`https://github.com/Xreki/glog.git`。 - - 构建`arm64-v8a`、 `Android API 21`的独立工具链： ```bash your/path/to/android-ndk-r14b-linux-x86_64/build/tools/make-standalone-toolchain.sh \ @@ -80,7 +68,7 @@ your/path/to/android-ndk-r14b-linux-x86_64/build/tools/make-standalone-toolchain 此命令将在`your/path/to/arm64_standalone_toolchain`目录生成一套独立编译工具链，面向架构为64位ARM64架构，支持的最小Android API级别为21，支持编译器`arm-linux-androideabi-gcc (GCC) 4.9`和`clang 3.8`。 -注意：**arm64架构要求Android API不小于21**。 +注意：**PaddlePaddle要求使用的编译工具链所支持的Android API级别不小于21**。 ### 配置交叉编译参数 From 40f3e0c19421b30e510ad3f55eeb652504179831 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=AD=A6=E6=AF=85?= Date: Wed, 18 Oct 2017 19:09:45 +0800 Subject: [PATCH 18/76] fix_fault_tolerant_dist_lock (#4888) --- go/pserver/client/client.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/go/pserver/client/client.go b/go/pserver/client/client.go index 20d91e7703..e5187ce3df 100644 --- a/go/pserver/client/client.go +++ b/go/pserver/client/client.go @@ -137,7 +137,7 @@ func (c *Client) FinishInitParams() error { return err } } - return nil + return c.sel.Done() } // SendGrads sends gradients to parameter servers for updating From efd009a063d089922098a1c766686fd1c3667043 Mon Sep 17 00:00:00 2001 From: fengjiayi Date: Wed, 18 Oct 2017 12:37:44 -0700 Subject: [PATCH 19/76] implementation of simple conv2d layer (#4868) * Implement FC layer with helper * Update LayerHelper * Add debug string for Python ProtoBuf and Rename `Sync` to `Flush` * Add check of ProtoBuf initialization * Layer wrapper for FC * Fix unittest * Fix CI * Add code generator * AttributeChecker Better error log and speicalize bool Since lots of types can be cast to bool * Complete mlp, fit_a_line * Implementation of simple conv_2d layer * Fix bugs * Remove debug code --- python/paddle/v2/framework/framework.py | 2 +- python/paddle/v2/framework/layer_helper.py | 18 ++++--- python/paddle/v2/framework/layers.py | 48 ++++++++++++++++++- .../paddle/v2/framework/tests/test_layers.py | 12 ++++- 4 files changed, 67 insertions(+), 13 deletions(-) diff --git a/python/paddle/v2/framework/framework.py b/python/paddle/v2/framework/framework.py index 3fb6efe42a..e16bc72447 100644 --- a/python/paddle/v2/framework/framework.py +++ b/python/paddle/v2/framework/framework.py @@ -232,7 +232,7 @@ class Operator(object): if attrs is not None: for attr in proto.attrs: attr_name = attr.name - if not attr_name in attrs: + if (not attr_name in attrs) or (attrs[attr_name] is None): continue if not isinstance(attrs[attr_name], Block): self.desc.set_attr(attr_name, attrs[attr_name]) diff --git a/python/paddle/v2/framework/layer_helper.py b/python/paddle/v2/framework/layer_helper.py index 26d3e04310..6615bdcd3b 100644 --- a/python/paddle/v2/framework/layer_helper.py +++ b/python/paddle/v2/framework/layer_helper.py @@ -66,15 +66,15 @@ class LayerHelper(object): actual = self.kwargs.get('param_attr', None) return actual if actual is not None else default - def bias_attr(self, size, dtype): - bias_attr = self.kwargs.get('bias_attr', False) - if bias_attr is None or bias_attr: + def bias_attr(self, shape, dtype): + bias_attr = self.kwargs.get('bias_attr', None) + if bias_attr is True: bias_attr = { 'name': None, 'init_attr': { 'type': 'fill_constant', 'value': 0.0, - 'shape': [size], + 'shape': shape, 'dataType': dtype } } @@ -127,15 +127,13 @@ class LayerHelper(object): return self.program.global_block().create_var(*args, **kwargs) def append_bias_op(self, input_var): - bias_attr = self.bias_attr( - self.kwargs['size'], dtype=input_var.data_type) + size = list(input_var.shape[1:]) + bias_attr = self.bias_attr(size, dtype=input_var.data_type) if not bias_attr: return input_var + b = self.create_parameter( - attr=bias_attr, - shape=[self.kwargs['size']], - dtype=input_var.data_type, - suffix='b') + attr=bias_attr, shape=size, dtype=input_var.data_type, suffix='b') tmp = self.create_tmp_variable(dtype=input_var.data_type) self.append_op( type='elementwise_add', diff --git a/python/paddle/v2/framework/layers.py b/python/paddle/v2/framework/layers.py index 44b587b116..1821da197e 100644 --- a/python/paddle/v2/framework/layers.py +++ b/python/paddle/v2/framework/layers.py @@ -3,7 +3,7 @@ import paddle.v2.framework.core as core from paddle.v2.framework.framework import OpProtoHolder, Variable import re -__all__ = ['fc_layer', 'data_layer', 'cross_entropy'] +__all__ = ['fc_layer', 'data_layer', 'cross_entropy', 'conv2d_layer'] def fc_layer(input, @@ -24,6 +24,7 @@ def fc_layer(input, for input_var, param_attr in helper.iter_inputs_and_params(): input_shape = input_var.shape param_shape = list(input_shape[num_flatten_dims:]) + [size] + w = helper.create_parameter( attr=param_attr, shape=param_shape, dtype=dtype) tmp = helper.create_tmp_variable(dtype) @@ -111,6 +112,7 @@ def _create_op_func_(op_type): _create_op_func_('mean') +_create_op_func_('pool2d') def cross_entropy(input, label, **kwargs): @@ -141,3 +143,47 @@ def square_error_cost(input, label, **kwargs): outputs={'Y': [square_out]}, attrs={'factor': 2.0}) return square_out + + +def conv2d_layer(input, + num_filters, + name=None, + filter_size=[1, 1], + act=None, + groups=None, + stride=[1, 1], + padding=None, + bias_attr=None, + param_attr=None, + program=None): + helper = LayerHelper('conv2d', **locals()) + dtype = helper.input_dtype() + + num_channels = input.shape[1] + if groups is None: + num_filter_channels = num_channels + else: + if num_channels % groups is not 0: + raise ValueError("num_channels must be divisible by groups.") + num_filter_channels = num_channels / groups + + input_shape = input.shape + filter_shape = [num_filters, num_filter_channels] + filter_size + filter = helper.create_parameter( + attr=helper.param_attr, shape=filter_shape, dtype=dtype) + pre_bias = helper.create_tmp_variable(dtype) + + helper.append_op( + type='conv2d', + inputs={ + 'Input': input, + 'Filter': filter, + }, + outputs={"Output": pre_bias}, + attrs={'strides': stride, + 'paddings': padding, + 'groups': groups}) + + pre_act = helper.append_bias_op(pre_bias) + + return helper.append_activation(pre_act) diff --git a/python/paddle/v2/framework/tests/test_layers.py b/python/paddle/v2/framework/tests/test_layers.py index 1ef2591cca..ce20371cfb 100644 --- a/python/paddle/v2/framework/tests/test_layers.py +++ b/python/paddle/v2/framework/tests/test_layers.py @@ -1,4 +1,4 @@ -from paddle.v2.framework.layers import fc_layer, data_layer, cross_entropy, mean, square_error_cost +from paddle.v2.framework.layers import fc_layer, data_layer, cross_entropy, mean, square_error_cost, conv2d_layer from paddle.v2.framework.framework import Program, g_program import paddle.v2.framework.core as core import unittest @@ -38,6 +38,16 @@ class TestBook(unittest.TestCase): self.assertIsNotNone(avg_cost) print str(program) + def test_simple_conv2d(self): + pd = core.ProgramDesc.__create_program_desc__() + program = Program(desc=pd) + images = data_layer( + name='pixel', shape=[3, 48, 48], data_type='int32', program=program) + conv2d_layer( + input=images, num_filters=3, filter_size=[4, 4], program=program) + + print str(program) + if __name__ == '__main__': unittest.main() From e747623e8639ee43a8dd2b33d04f6110a1182de3 Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Wed, 18 Oct 2017 13:14:41 -0700 Subject: [PATCH 20/76] Change ProgramDesc not a global variable (#4879) * Change ProgramDesc not a global variable * Polish code style * Correct implement BlockDesc destructor * Unify program as parameter name --- paddle/framework/attribute.cc | 18 +++------- paddle/framework/attribute.h | 5 +-- paddle/framework/backward_test.cc | 31 ++++------------- paddle/framework/executor.cc | 3 +- paddle/framework/op_registry.cc | 5 +-- paddle/framework/op_registry.h | 3 +- paddle/framework/op_registry_test.cc | 12 +++---- paddle/framework/operator_test.cc | 6 ++-- paddle/framework/program_desc.cc | 33 +++++-------------- paddle/framework/program_desc.h | 7 ++-- paddle/framework/var_type_inference_test.cc | 4 +-- paddle/operators/dynamic_recurrent_op_test.cc | 2 +- paddle/pybind/protobuf.cc | 16 +-------- paddle/pybind/pybind.cc | 9 ++--- python/paddle/v2/framework/framework.py | 6 ++-- .../v2/framework/tests/test_infer_shape.py | 4 +-- .../paddle/v2/framework/tests/test_layers.py | 6 ++-- .../v2/framework/tests/test_protobuf_descs.py | 18 +++++----- 18 files changed, 62 insertions(+), 126 deletions(-) diff --git a/paddle/framework/attribute.cc b/paddle/framework/attribute.cc index d6a2975aaa..29fe352ca4 100644 --- a/paddle/framework/attribute.cc +++ b/paddle/framework/attribute.cc @@ -19,19 +19,7 @@ limitations under the License. */ namespace paddle { namespace framework { -static ProgramDesc* g_program_desc = nullptr; - -ProgramDesc& GetProgramDesc() { - if (g_program_desc == nullptr) { - g_program_desc = new ProgramDesc(); - auto root_block = g_program_desc->mutable_blocks()->Add(); - root_block->set_idx(0); - root_block->set_parent_idx(-1); - } - return *g_program_desc; -} - -Attribute GetAttrValue(const OpDesc::Attr& attr_desc) { +Attribute GetAttrValue(const OpDesc::Attr& attr_desc, ProgramDesc* program) { switch (attr_desc.type()) { case framework::AttrType::BOOLEAN: { return attr_desc.b(); @@ -74,7 +62,9 @@ Attribute GetAttrValue(const OpDesc::Attr& attr_desc) { return val; } case framework::AttrType::BLOCK: { - return GetProgramDesc().mutable_blocks(attr_desc.block_idx()); + PADDLE_ENFORCE(program != nullptr, + "Need to specify ProgramDesc when get a block attr"); + return program->mutable_blocks(attr_desc.block_idx()); } } PADDLE_ENFORCE(false, "Unknown OpDesc::AttrDesc::type !"); diff --git a/paddle/framework/attribute.h b/paddle/framework/attribute.h index 8a7a949346..9744662b8f 100644 --- a/paddle/framework/attribute.h +++ b/paddle/framework/attribute.h @@ -26,16 +26,13 @@ limitations under the License. */ namespace paddle { namespace framework { - -ProgramDesc& GetProgramDesc(); - template inline AttrType AttrTypeID() { Attribute tmp = T(); return static_cast(tmp.which() - 1); } -Attribute GetAttrValue(const OpDesc::Attr& attr_desc); +Attribute GetAttrValue(const OpDesc::Attr& attr_desc, ProgramDesc* desc); class AttrReader { public: diff --git a/paddle/framework/backward_test.cc b/paddle/framework/backward_test.cc index 0c35a157bc..10301f7e39 100644 --- a/paddle/framework/backward_test.cc +++ b/paddle/framework/backward_test.cc @@ -495,19 +495,8 @@ TEST(Backward, linear_net_intermediate_variable_has_no_grad) { EXPECT_EQ(bwd_net->ops_[2]->Outputs(all).size(), 0UL); } -// =================================== // - -f::ProgramDesc *GetNewProgramDesc() { - auto *program_desc = new f::ProgramDesc(); - auto *root_block = program_desc->add_blocks(); - root_block->set_idx(0); - root_block->set_parent_idx(-1); - return program_desc; -} - TEST(Backward, simple_single_op) { - f::ProgramDesc *program_desc = GetNewProgramDesc(); - f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::ProgramDescBind program; f::BlockDescBind *block = program.Block(0); f::OpDescBind *op = block->AppendOp(); @@ -543,8 +532,7 @@ TEST(Backward, simple_single_op) { } TEST(Backward, default_attribute) { - f::ProgramDesc *program_desc = GetNewProgramDesc(); - f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::ProgramDescBind program; f::BlockDescBind *block = program.Block(0); f::OpDescBind *op = block->AppendOp(); op->SetType("mul"); @@ -570,8 +558,7 @@ TEST(Backward, default_attribute) { } TEST(Backward, simple_mult_op) { - f::ProgramDesc *program_desc = GetNewProgramDesc(); - f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::ProgramDescBind program; f::BlockDescBind *block = program.Block(0); f::OpDescBind *op1 = block->AppendOp(); op1->SetType("rowwise_add"); @@ -654,8 +641,7 @@ TEST(Backward, simple_mult_op) { } TEST(Backward, intermedia_var_no_grad) { - f::ProgramDesc *program_desc = GetNewProgramDesc(); - f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::ProgramDescBind program; f::BlockDescBind *block = program.Block(0); f::OpDescBind *op1 = block->AppendOp(); op1->SetType("rowwise_add"); @@ -725,8 +711,7 @@ TEST(Backward, intermedia_var_no_grad) { } TEST(Backward, var_no_grad) { - f::ProgramDesc *program_desc = GetNewProgramDesc(); - f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::ProgramDescBind program; f::BlockDescBind *block = program.Block(0); f::OpDescBind *op1 = block->AppendOp(); op1->SetType("mult_in_out"); @@ -802,8 +787,7 @@ TEST(Backward, var_no_grad) { } TEST(Backward, shared_var) { - f::ProgramDesc *program_desc = GetNewProgramDesc(); - f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::ProgramDescBind program; f::BlockDescBind *block = program.Block(0); f::OpDescBind *op1 = block->AppendOp(); op1->SetType("rowwise_add"); @@ -893,8 +877,7 @@ TEST(Backward, shared_var) { } TEST(Backward, half_backward) { - f::ProgramDesc *program_desc = GetNewProgramDesc(); - f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::ProgramDescBind program; f::BlockDescBind *block = program.Block(0); auto *op1 = block->AppendOp(); op1->SetType("minus"); diff --git a/paddle/framework/executor.cc b/paddle/framework/executor.cc index b3b85b5865..00caa6e1d5 100644 --- a/paddle/framework/executor.cc +++ b/paddle/framework/executor.cc @@ -75,7 +75,8 @@ void Executor::Run(const ProgramDesc& pdesc, Scope* scope, int block_id) { } for (auto& op_desc : block.ops()) { - auto op = paddle::framework::OpRegistry::CreateOp(op_desc); + auto op = paddle::framework::OpRegistry::CreateOp( + op_desc, const_cast(&pdesc)); op->Run(local_scope, *device); } diff --git a/paddle/framework/op_registry.cc b/paddle/framework/op_registry.cc index 504afbd5db..c2f2438edf 100644 --- a/paddle/framework/op_registry.cc +++ b/paddle/framework/op_registry.cc @@ -43,12 +43,13 @@ static VariableNameMap ConvertOpDescVarsToVarNameMap( return ret_val; } -std::unique_ptr OpRegistry::CreateOp(const OpDesc& op_desc) { +std::unique_ptr OpRegistry::CreateOp(const OpDesc& op_desc, + ProgramDesc* program) { VariableNameMap inputs = ConvertOpDescVarsToVarNameMap(op_desc.inputs()); VariableNameMap outputs = ConvertOpDescVarsToVarNameMap(op_desc.outputs()); AttributeMap attrs; for (auto& attr : op_desc.attrs()) { - attrs[attr.name()] = GetAttrValue(attr); + attrs[attr.name()] = GetAttrValue(attr, program); } return CreateOp(op_desc.type(), inputs, outputs, attrs); diff --git a/paddle/framework/op_registry.h b/paddle/framework/op_registry.h index dfca46b789..d25b4abccb 100644 --- a/paddle/framework/op_registry.h +++ b/paddle/framework/op_registry.h @@ -74,7 +74,8 @@ class OpRegistry { const VariableNameMap& outputs, AttributeMap attrs); - static std::unique_ptr CreateOp(const OpDesc& op_desc); + static std::unique_ptr CreateOp(const OpDesc& op_desc, + ProgramDesc* program); static std::unique_ptr CreateOp(const OpDescBind& op_desc); }; diff --git a/paddle/framework/op_registry_test.cc b/paddle/framework/op_registry_test.cc index b860fe6cac..6289125d7c 100644 --- a/paddle/framework/op_registry_test.cc +++ b/paddle/framework/op_registry_test.cc @@ -74,7 +74,7 @@ TEST(OpRegistry, CreateOp) { attr->set_type(paddle::framework::AttrType::FLOAT); attr->set_f(scale); - auto op = paddle::framework::OpRegistry::CreateOp(op_desc); + auto op = paddle::framework::OpRegistry::CreateOp(op_desc, nullptr); paddle::framework::Scope scope; paddle::platform::CPUDeviceContext dev_ctx; op->Run(scope, dev_ctx); @@ -95,7 +95,7 @@ TEST(OpRegistry, IllegalAttr) { bool caught = false; try { - paddle::framework::OpRegistry::CreateOp(op_desc); + paddle::framework::OpRegistry::CreateOp(op_desc, nullptr); } catch (paddle::platform::EnforceNotMet err) { caught = true; std::string msg = "larger_than check fail"; @@ -115,7 +115,7 @@ TEST(OpRegistry, DefaultValue) { ASSERT_TRUE(op_desc.IsInitialized()); - auto op = paddle::framework::OpRegistry::CreateOp(op_desc); + auto op = paddle::framework::OpRegistry::CreateOp(op_desc, nullptr); paddle::framework::Scope scope; paddle::platform::CPUDeviceContext dev_ctx; op->Run(scope, dev_ctx); @@ -131,7 +131,7 @@ TEST(OpRegistry, CustomChecker) { // attr 'test_attr' is not set bool caught = false; try { - paddle::framework::OpRegistry::CreateOp(op_desc); + paddle::framework::OpRegistry::CreateOp(op_desc, nullptr); } catch (paddle::platform::EnforceNotMet err) { caught = true; std::string msg = "Attribute 'test_attr' is required!"; @@ -149,7 +149,7 @@ TEST(OpRegistry, CustomChecker) { attr->set_i(3); caught = false; try { - paddle::framework::OpRegistry::CreateOp(op_desc); + paddle::framework::OpRegistry::CreateOp(op_desc, nullptr); } catch (paddle::platform::EnforceNotMet err) { caught = true; std::string msg = "'test_attr' must be even!"; @@ -166,7 +166,7 @@ TEST(OpRegistry, CustomChecker) { attr->set_name("test_attr"); attr->set_type(paddle::framework::AttrType::INT); attr->set_i(4); - auto op = paddle::framework::OpRegistry::CreateOp(op_desc); + auto op = paddle::framework::OpRegistry::CreateOp(op_desc, nullptr); paddle::platform::CPUDeviceContext dev_ctx; paddle::framework::Scope scope; op->Run(scope, dev_ctx); diff --git a/paddle/framework/operator_test.cc b/paddle/framework/operator_test.cc index d7890ac8d0..c358f1a2b6 100644 --- a/paddle/framework/operator_test.cc +++ b/paddle/framework/operator_test.cc @@ -83,7 +83,7 @@ TEST(OperatorBase, all) { paddle::platform::CPUDeviceContext device_context; paddle::framework::Scope scope; - auto op = paddle::framework::OpRegistry::CreateOp(op_desc); + auto op = paddle::framework::OpRegistry::CreateOp(op_desc, nullptr); scope.Var("OUT1"); ASSERT_EQ(paddle::framework::op_run_num, 0); op->Run(scope, device_context); @@ -208,7 +208,7 @@ TEST(OpKernel, all) { paddle::platform::CPUDeviceContext cpu_device_context; paddle::framework::Scope scope; - auto op = paddle::framework::OpRegistry::CreateOp(op_desc); + auto op = paddle::framework::OpRegistry::CreateOp(op_desc, nullptr); ASSERT_EQ(paddle::framework::cpu_kernel_run_num, 0); op->Run(scope, cpu_device_context); ASSERT_EQ(paddle::framework::cpu_kernel_run_num, 1); @@ -244,7 +244,7 @@ TEST(OpKernel, multi_inputs) { scope.Var("y0")->GetMutable(); scope.Var("y1")->GetMutable(); - auto op = paddle::framework::OpRegistry::CreateOp(op_desc); + auto op = paddle::framework::OpRegistry::CreateOp(op_desc, nullptr); op->Run(scope, cpu_device_context); } diff --git a/paddle/framework/program_desc.cc b/paddle/framework/program_desc.cc index fcb7292884..df846f115a 100644 --- a/paddle/framework/program_desc.cc +++ b/paddle/framework/program_desc.cc @@ -18,27 +18,10 @@ limitations under the License. */ namespace paddle { namespace framework { -using ProgDescMap = - std::unordered_map>; -static ProgDescMap *g_bind_map = nullptr; - -ProgramDescBind &ProgramDescBind::Instance(ProgramDesc *prog) { - if (g_bind_map == nullptr) { - g_bind_map = new ProgDescMap(); - } - auto &map = *g_bind_map; - auto &ptr = map[prog]; - - if (ptr == nullptr) { - ptr.reset(new ProgramDescBind(prog)); - } - return *ptr; -} - BlockDescBind *ProgramDescBind::AppendBlock(const BlockDescBind &parent) { - auto *b = prog_->add_blocks(); + auto *b = prog_.add_blocks(); b->set_parent_idx(parent.ID()); - b->set_idx(prog_->blocks_size() - 1); + b->set_idx(prog_.blocks_size() - 1); blocks_.emplace_back(new BlockDescBind(this, b)); return blocks_.back().get(); } @@ -47,14 +30,14 @@ ProgramDesc *ProgramDescBind::Proto() { for (auto &block : blocks_) { block->Flush(); } - return prog_; + return &prog_; } -ProgramDescBind::ProgramDescBind(ProgramDesc *prog) { - prog_ = prog; - for (auto &block : *prog->mutable_blocks()) { - blocks_.emplace_back(new BlockDescBind(this, &block)); - } +ProgramDescBind::ProgramDescBind() { + auto *block = prog_.mutable_blocks()->Add(); + block->set_idx(0); + block->set_parent_idx(-1); + blocks_.emplace_back(new BlockDescBind(this, block)); } } // namespace framework } // namespace paddle diff --git a/paddle/framework/program_desc.h b/paddle/framework/program_desc.h index f29b1c54e7..514b62654d 100644 --- a/paddle/framework/program_desc.h +++ b/paddle/framework/program_desc.h @@ -26,7 +26,7 @@ class BlockDescBind; class ProgramDescBind { public: - static ProgramDescBind &Instance(ProgramDesc *prog); + ProgramDescBind(); BlockDescBind *AppendBlock(const BlockDescBind &parent); @@ -37,10 +37,7 @@ class ProgramDescBind { ProgramDesc *Proto(); private: - explicit ProgramDescBind(ProgramDesc *prog); - - // Not owned - ProgramDesc *prog_; + ProgramDesc prog_; std::vector> blocks_; diff --git a/paddle/framework/var_type_inference_test.cc b/paddle/framework/var_type_inference_test.cc index 87399208e9..918de1fd05 100644 --- a/paddle/framework/var_type_inference_test.cc +++ b/paddle/framework/var_type_inference_test.cc @@ -62,7 +62,7 @@ namespace paddle { namespace framework { TEST(InferVarType, sum_op) { - auto &prog = ProgramDescBind::Instance(&GetProgramDesc()); + ProgramDescBind prog; auto *op = prog.Block(0)->AppendOp(); op->SetType("sum"); op->SetInput("X", {"test_a", "test_b", "test_c"}); @@ -83,7 +83,7 @@ TEST(InferVarType, sum_op) { } TEST(InferVarType, sum_op_without_infer_var_type) { - auto &prog = ProgramDescBind::Instance(&GetProgramDesc()); + ProgramDescBind prog; auto *op = prog.Block(0)->AppendOp(); op->SetType("sum_without_infer_var_type"); op->SetInput("X", {"test2_a", "test2_b", "test2_c"}); diff --git a/paddle/operators/dynamic_recurrent_op_test.cc b/paddle/operators/dynamic_recurrent_op_test.cc index 83a5ba36d9..36f405568d 100644 --- a/paddle/operators/dynamic_recurrent_op_test.cc +++ b/paddle/operators/dynamic_recurrent_op_test.cc @@ -51,7 +51,7 @@ class DynamicRecurrentOpTestHelper : public ::testing::Test { CreateGlobalVariables(); auto op_desc = CreateOpDesc(); - op = paddle::framework::OpRegistry::CreateOp(op_desc); + op = paddle::framework::OpRegistry::CreateOp(op_desc, nullptr); dop = dynamic_cast(op.get()); InitCacheManually(); InitStepNet(); diff --git a/paddle/pybind/protobuf.cc b/paddle/pybind/protobuf.cc index 82aae72ba9..fbdd673295 100644 --- a/paddle/pybind/protobuf.cc +++ b/paddle/pybind/protobuf.cc @@ -100,21 +100,7 @@ using namespace paddle::framework; // NOLINT // Bind Methods void BindProgramDesc(py::module &m) { py::class_(m, "ProgramDesc", "") - .def_static("instance", - []() -> ProgramDescBind * { - return &ProgramDescBind::Instance(&GetProgramDesc()); - }, - py::return_value_policy::reference) - .def_static("__create_program_desc__", - []() -> ProgramDescBind * { - // Only used for unit-test - auto *prog_desc = new ProgramDesc; - auto *block = prog_desc->mutable_blocks()->Add(); - block->set_idx(0); - block->set_parent_idx(-1); - return &ProgramDescBind::Instance(prog_desc); - }, - py::return_value_policy::reference) + .def(py::init<>()) .def("append_block", &ProgramDescBind::AppendBlock, py::return_value_policy::reference) .def("append_backward", diff --git a/paddle/pybind/pybind.cc b/paddle/pybind/pybind.cc index fcae92ad99..9eb1bf4a16 100644 --- a/paddle/pybind/pybind.cc +++ b/paddle/pybind/pybind.cc @@ -17,6 +17,7 @@ limitations under the License. */ #include "paddle/framework/backward.h" #include "paddle/framework/executor.h" #include "paddle/framework/feed_fetch_method.h" +#include "paddle/framework/framework.pb.h" #include "paddle/framework/lod_tensor.h" #include "paddle/framework/selected_rows.h" #include "paddle/framework/tensor_array.h" @@ -259,7 +260,7 @@ All parameter, weight, gradient are variables in Paddle. PADDLE_ENFORCE(desc.IsInitialized(), "User OpDesc is not initialized, reason %s", desc.InitializationErrorString()); - return OpRegistry::CreateOp(desc); + return OpRegistry::CreateOp(desc, nullptr); }) .def("backward", [](const OperatorBase &forwardOp, @@ -363,7 +364,7 @@ All parameter, weight, gradient are variables in Paddle. PADDLE_ENFORCE(desc.IsInitialized(), "User OpDesc is not initialized, reason %s", desc.InitializationErrorString()); - auto rnn_op = OpRegistry::CreateOp(desc); + auto rnn_op = OpRegistry::CreateOp(desc, nullptr); return static_cast(rnn_op.release()); }) .def("set_stepnet", [](operators::RecurrentOp &self, @@ -381,7 +382,7 @@ All parameter, weight, gradient are variables in Paddle. PADDLE_ENFORCE(desc.IsInitialized(), "User OpDesc is not initialized, reason %s", desc.InitializationErrorString()); - auto rnn_op = OpRegistry::CreateOp(desc); + auto rnn_op = OpRegistry::CreateOp(desc, nullptr); return static_cast( rnn_op.release()); }) @@ -408,7 +409,7 @@ All parameter, weight, gradient are variables in Paddle. PADDLE_ENFORCE(desc.IsInitialized(), "User OpDesc is not initialized, reason %s", desc.InitializationErrorString()); - auto cond_op = OpRegistry::CreateOp(desc); + auto cond_op = OpRegistry::CreateOp(desc, nullptr); return static_cast(cond_op.release()); }) .def("set_truenet", diff --git a/python/paddle/v2/framework/framework.py b/python/paddle/v2/framework/framework.py index e16bc72447..93e2218eab 100644 --- a/python/paddle/v2/framework/framework.py +++ b/python/paddle/v2/framework/framework.py @@ -384,10 +384,8 @@ class Program(object): cls._instance = cls() return cls._instance - def __init__(self, desc=None): - if desc is None: - desc = core.ProgramDesc.instance() - self.desc = desc + def __init__(self): + self.desc = core.ProgramDesc() self.blocks = [Block(self, 0)] self.current_block_idx = 0 diff --git a/python/paddle/v2/framework/tests/test_infer_shape.py b/python/paddle/v2/framework/tests/test_infer_shape.py index 19bb45acef..5cfb9e6687 100644 --- a/python/paddle/v2/framework/tests/test_infer_shape.py +++ b/python/paddle/v2/framework/tests/test_infer_shape.py @@ -5,7 +5,7 @@ import paddle.v2.framework.core as core class TestInferShape(unittest.TestCase): def test_sum_op(self): - prog = core.ProgramDesc.__create_program_desc__() + prog = core.ProgramDesc() self.assertIsNotNone(prog) block = prog.block(0) self.assertIsNotNone(block) @@ -33,7 +33,7 @@ class TestInferShape(unittest.TestCase): self.assertEqual(out.shape(), shape) def test_mul_op(self): - prog = core.ProgramDesc.__create_program_desc__() + prog = core.ProgramDesc() self.assertIsNotNone(prog) block = prog.block(0) self.assertIsNotNone(block) diff --git a/python/paddle/v2/framework/tests/test_layers.py b/python/paddle/v2/framework/tests/test_layers.py index ce20371cfb..2ffadf7371 100644 --- a/python/paddle/v2/framework/tests/test_layers.py +++ b/python/paddle/v2/framework/tests/test_layers.py @@ -6,8 +6,7 @@ import unittest class TestBook(unittest.TestCase): def test_fit_a_line(self): - pd = core.ProgramDesc.__create_program_desc__() - program = Program(desc=pd) + program = Program() x = data_layer( name='x', shape=[13], data_type='float32', program=program) y_predict = fc_layer(input=x, size=1, act=None, program=program) @@ -21,8 +20,7 @@ class TestBook(unittest.TestCase): print str(program) def test_recognize_digits_mlp(self): - pd = core.ProgramDesc.__create_program_desc__() - program = Program(desc=pd) + program = Program() # Change g_program, so the rest layers use `g_program` images = data_layer( diff --git a/python/paddle/v2/framework/tests/test_protobuf_descs.py b/python/paddle/v2/framework/tests/test_protobuf_descs.py index c775b1a398..6ed8edf91c 100644 --- a/python/paddle/v2/framework/tests/test_protobuf_descs.py +++ b/python/paddle/v2/framework/tests/test_protobuf_descs.py @@ -4,7 +4,7 @@ import paddle.v2.framework.core as core class TestOpDesc(unittest.TestCase): def test_op_desc(self): - prog = core.ProgramDesc.__create_program_desc__() + prog = core.ProgramDesc() self.assertIsNotNone(prog) block = prog.block(0) self.assertIsNotNone(block) @@ -64,16 +64,16 @@ class TestOpDesc(unittest.TestCase): class TestProgramDesc(unittest.TestCase): def test_instance(self): - program_desc = core.ProgramDesc.__create_program_desc__() + program_desc = core.ProgramDesc() self.assertIsNotNone(program_desc) del program_desc - program_desc = core.ProgramDesc.instance() + program_desc = core.ProgramDesc() self.assertIsNotNone(program_desc) self.assertIsNotNone(program_desc.block(0)) del program_desc def test_append_block(self): - prog_desc = core.ProgramDesc.__create_program_desc__() + prog_desc = core.ProgramDesc() self.assertIsNotNone(prog_desc) block_root = prog_desc.block(0) self.assertIsNotNone(block_root) @@ -91,7 +91,7 @@ class TestProgramDesc(unittest.TestCase): class TestVarDesc(unittest.TestCase): def test_shape(self): - program_desc = core.ProgramDesc.__create_program_desc__() + program_desc = core.ProgramDesc() block = program_desc.block(0) var = block.var('my_var') var.set_type(core.VarDesc.VarType.SELECTED_ROWS) @@ -102,7 +102,7 @@ class TestVarDesc(unittest.TestCase): self.assertEqual(core.VarDesc.VarType.SELECTED_ROWS, var.type()) def test_data_type(self): - program_desc = core.ProgramDesc.__create_program_desc__() + program_desc = core.ProgramDesc() block = program_desc.block(0) var = block.var('my_var') var.set_type(core.VarDesc.VarType.LOD_TENSOR) @@ -113,7 +113,7 @@ class TestVarDesc(unittest.TestCase): class TestBlockDesc(unittest.TestCase): def test_add_var(self): - prog = core.ProgramDesc.__create_program_desc__() + prog = core.ProgramDesc() self.assertIsNotNone(prog) block = prog.block(0) self.assertIsNotNone(block) @@ -121,12 +121,12 @@ class TestBlockDesc(unittest.TestCase): var2 = block.var("var2") var3 = block.var("var3") all_vars = block.all_vars() - self.assertEqual(set(all_vars), set([var1, var2, var3])) + self.assertEqual(set(all_vars), {var1, var2, var3}) var2_re = block.find_var("var2") self.assertEqual(var2_re, var2) def test_add_op(self): - prog = core.ProgramDesc.__create_program_desc__() + prog = core.ProgramDesc() self.assertIsNotNone(prog) block = prog.block(0) self.assertIsNotNone(block) From f4a21e387ffb0a864b8bb9822716fe64aacddaee Mon Sep 17 00:00:00 2001 From: Abhinav Arora Date: Wed, 18 Oct 2017 14:06:30 -0700 Subject: [PATCH 21/76] Design Doc for Regularization (#4869) * Add initail design doc for regularization * Updating image links * Commiting the images for the equations * Adding computation graph images * Adding section on computation graph --- doc/design/images/feed_forward.png | Bin 0 -> 32247 bytes .../images/feed_forward_regularized.png | Bin 0 -> 46036 bytes doc/design/images/l1_regularization.png | Bin 0 -> 1157 bytes doc/design/images/l2_regularization.png | Bin 0 -> 989 bytes doc/design/images/loss_equation.png | Bin 0 -> 1589 bytes doc/design/regularization.md | 103 ++++++++++++++++++ 6 files changed, 103 insertions(+) create mode 100644 doc/design/images/feed_forward.png create mode 100644 doc/design/images/feed_forward_regularized.png create mode 100644 doc/design/images/l1_regularization.png create mode 100644 doc/design/images/l2_regularization.png create mode 100644 doc/design/images/loss_equation.png create mode 100644 doc/design/regularization.md diff --git a/doc/design/images/feed_forward.png b/doc/design/images/feed_forward.png new file mode 100644 index 0000000000000000000000000000000000000000..d312371a04c26aa6cd196e0bd1f51becb425180b GIT binary patch literal 32247 zcmeFYWn7ir*EI^bC6z`+kl1viNW-RE*mS3el!PGNAt@y(9ZGjMNJ>aZcY}0yowe`t zzn|wh?>Xn=`|b7Thy7#2b*-3d&N0UrD_BuM>M`bXOe7?v$1>95%1B5kX-G)OQRt}P zCsde|Ye+~GNHXFgs?Pd5DHzU#5?A*IwA_>!Mv*G$^SM}<(4Y{szso52FSiDX6vboO zA3${RvVuy3SSfOop_$&x22`j%v1%eBoexArv~<~g?Rq2MiBfDFcr_Qj;5YhOS8rPH z)_f|^V4PHJJuHg&tI}7PkvVN#*6`;spmm5fN-MGBQ)KX=!OIht6?9zns*|QXC~ykvdIR zuf3l{dPhM%5rL;mIuiAT)1o|n{yITJN$KPD!BYFS$@}+RDNge)e@czS`Y7n>=_5;8 z?zHXf?fW(cQrrZ%xXwDMq{1Inym??|CJu%xNhribhb)Hi3<_rQc$edewHFFYvckr- z$JL3sp#6IPl{$WMmBVBIzqv|T;l-)sZ47l z>$}d{@2yc3a*kWFB*QF{Qc|p}#r#UGk5+qLfrq8A#&7RXkZn*YiF+?|gzzQFhdoCz z!UKP(#F09wn1XJYC&a)_|G96+JmNA)(f4q1ofmQPZNNXr>gh(CBhp{`KqEGXZ}>K|lVXIAwQK z*)NxffkF{e{moBd_RR&IhcvtgtjA6F7h;a(e&g7eabL&;>)po-wcsuSWY@ba?5J9! zv9gE64tfKqGcm&ApTXA2OA#ggAVJ^noDAH*v0 z*?)il4I5wj<#kZ(?)^vw&sDV9mTMli}dTc#)3DT)l%Y?#mQD2W6gFkMp@#rURsx z%b{%fp6gh+4|CZ>Z<>CC2%m2Bn-ci1lS%Ov;)|c^5Ih(Xf*BkZrZydumc}L_A)&0V z&q_>8tR{3mZkRmzLcqC{S-(*ecDYKhIbBt-pi3`&E7|$wj0zm%7MMw>FSYO?F*OfN z62*ubxALG+-aTLLaB>g>nb%=!G=DwWbt|vjs2$aOs{F?h2j{*VTN8D>p5wPre2MbHLjl^H5&P0Cw z3XRuy&AYi?xL5atAYc9FUXWH+R*t=J->ch*@soP0YwJS_#+;~myH!vEF}X+r+RQJRicrv`;cN7qD&+Nb6Yk;SEoK1OSle$>4Kg=yLYF(HK0y7Td ziY~L~blaC>0dZ~lbA#hAMRluXZ*DrneIyXSt%)$-*;v^4= z_vXvgJqXjrx7f5vQa}MomkfeC7|@Gy=-L0~L_Wp*Y${xvFa?8b5vJvr`qhH$6BVt% zClk!P#DYlq$P233Xp%)2Ck{H#J9Ef7c6CrYb3Cr&SSp_Sq=SO9tYR-M&07B#N+Yh~ zhx*tZKj+{iIEZ*hu9fYO`0tO$tqv!Dm3Tk>jf6^Ehy$W>%abqPeRYJ z1svti%OlMXI7wNu{!<*UaM5MGi?u2wUC;MexEI_jn02b(TnaC>50@ASX_dbVe)Nnz zJ<(yyhYIpd=zP|`;AlA@OUI^3tta$)9xnhxhJ>K9_`S0876-a!hzZ684eT}r&iJt3 z$U#3}sNsCxEtVcX>^WyXYTrWRLG8dd4yoXcf^00dMnxBmFQmN6T8=pvl{-AIKQ9rc zW&cHeE`s7o07i7B=ZZw!*V`M@uo(3 zeK!&HJit-OJplPt0R?gjMuMdG)d(vl+>8DtoDlaw&fWJxhn}|fG={XStSn&!b=+80 z0gg?}ZS~3Kh*VFD$F*bKR(?SYldXaKp#&&!ioz~q9x3#4Zr4*+r`tc=&i7&zjNO6&QMeeqf=aEKHi z-P=3wA^m4lB*C7-VPOyp7(T(bW0TjxMJ&_)1glRC+{-#TJ9|KcF|air_?9=qtY>)` zSXk7`ySlsk-1eL3z)^8klm4C|7I+o=^qQy4fN)-|h+D17Cuc6pC?Oyxj4*9!sb5>V zl^!LVFr6?}(t*3LiXFM7gHEBr3#1?A7SCJlByC+?<`jOXIFK_&gR^vMtjjIN7!k?W zz@@Cx^ZxdnNnd&Asf0A0Rpv$I!C#JVo8ZuI<0gush~TovBKMCk)u?>gsx%-|U>L;@l%si&FiR3-RKznV;QnWYMW6Q>n$<-~=Z zyGyp)a#n-kb{&~)Qsc6M$_<9Qv}HYREP_%W46oE93a&66jKianRBiWZm*y!7$D%0Z zejVAkrLWAqy@DxkwVVpa3g?SXd~<02Cp<%o8W-XdgsF%*Na6Y=LYx3uuaoZV)3Pw0 zz>mq5sbW3;&op~M`%@J;<^3?@u16-sf{~4B?nlA8c0Dw*TAJljUks^}Gi?@{n~wHD zNQ{(nW*33c*_z~0<1(P&;epXbfzeq~qSS}LnJWxj)?$wiyUCf*Z57c2!?NLN#W&p8 zP!9l6dc&lARwcY$bcKCrJiZqeGRsIyM{4N1{J`Yw*a#tFqNb(IR=;V~9}%VEK*?5i zmqg+#phorsD9c;f`T1-T^E@je?`)05^4L~d&2Y)U;pbDk!-u4fiv zo~6kraTQ&;P`G@wBYS&xxsg`lN=7z*xYCV2yRz{yBATPU&Fb&qgmd^odyu*3L zAr~#~6J*&X^A)^~>`nP}g+pRllk2lxl|9F9WGdafJfalNzQ@#TaRRfS)A~H!8%tMFRC=f7U*UWGxl`H^=<|uIbG}d!~Dh@1fUa{ zJRR~2bZ5kD_##d4s1;L?I_ab!7(>TBFNIO4572xpP-5DmS9~FlC;>u1SASw5fpYSn zSN??y} z;F6Pj?X>zl2!2e$^*a5NJ&~58A_a&d76v9JxmWQfm0iqWZomH!RQf^s!P5) zkZj%jbjM9cm$SPTZ(v;!mok}*BXpw(8Ystl-Qo(X1x+q?dN3vtqZKh~Lmlv`dRLo^ z`74AJyL&Z?O6m^4PvVbJl~kk-5&sSNQ+I4fyZH*Gz*}5BBtvCdrNJLc{xO*vezREE z(a~WA!ZNhx_z!>FY!bJPzPhsVAARxW&~t1)af+f!=W$=DxTR;`p?F~ z!a|?Wr)zX)q7% z!DQZw<`;2`Wuq{Q(cCQ1Nl)+E)wu@!(pSODd6|s>ul1}MBGkA_h<9}k_JViCEiy4O zT9d=votygm`)iK}_~C`r;*>lscUP7@zkjo&@jLAuPEJlfts5m{$>Kh7RCW&-h@^@1 zP5~HwI*)tmNd*B<-;(X4H*okMBO{}Qin@B7nWLZD_43M!8~~|~Ya1II&H|}40=n;F zQ#k@t2-DX8qG^kuyhnsyTx$9Y>U13B0do(Ya>5rN?zhmRC8E3^Vld-I`9NPOh`$dG zy2^OY09BEIkcBlcPqHSy0TIlPh(X8*IBT!<;)f{k(V2V*@jeDie-0ugh?0cbX^Pvj z=obd@NPMR+d{Xub1gjw8*)q{4o1)&!Do?*K`$L%J7cuaNS)hlB>M_B;9^twC`oofS z2)cE`G|~tEW3?Pn4m|H@p%{sYauTc5u7d6JPYWX6gs|x5%+SSvF!Wv=T9ZTo8l4OE4DnOskODWU;X|4+dv~K zXkucrP&>CbD86csH}&>6?T7__kT#gT(#+|)6;Ef5%XL6ply@qm91sHDSyB=DW8Y;6 z27kosKOgxa7r$CDz$XWi@pqYMTF_}j%gV}>m6Rw!Gi7(MDALo{7rozntpEd)?}g}>UyK1GbOT3KZzIjivlC>9fcU7Er0cJ`lyic z(Swj|Sg_Eaa!C1t9)&u$LIIyWyp<=g{cy0%xZAZWjO1xjYO1HGJZroh*M~_8fapHE zU+>l&jlhL^?ryJtl$I*?_Vv-QvMPXW_^ws)-VQ*r-nxxcrlYWj6x_lnC!1piW)C8Zo7qYfmzYvL}U9Sc%1U<~4i6?XVTVZvxRBX!?Z zaunzW6FHfRvU3Q0k7G6Tqmq+JB|hnY>?Hd0L++{hL~%;8nGY|a3NqNg{kJo>i~?TpGEpPXVa!rOu{Qd@QMCu+l&JFvBmpGwg;}#y*t+Rdz3#A! zBJgH)o~4b^>TX+`>;-qe0lqgfmi^g7Nx|*S3jpSt@weXmw~%j zWRfwoQt>ZcL5FON(b3ff8_tvn#j^pF0%PM>3*Q|3Hh+xrPn*ibg07WNR1$9M!lajh z*e~#d=7>4o|MEtD@D4PHV*(im#q+l9kN29~&Ij^xbDgDYLbPx}gtQ}F?rtnj9rx#(Z2RI^Yyizw z#IK2mkqZV*-0jN%b|A4_$BYo54kWo<(EI-^g$vzQW+YlV!$?nd+4mW3rc7IF!T6 zDk?NDUn&89m=4fl@n65U7E>WTyhJyUUHW+GujO!hD^S3>% zcNXR=&7%tqTAE*RbJv)i@6DA32M61LGT1^5wfv18e5Mr&fL_&A=(yxdWKg_+gA_V2 z7N8>9H~`LPQrn7Fh2cd(DjCXN^tC zMjd7=Fwq|k5nWOkKI(w#o(O-zPrCJ6Li0~o@EK^q639fl!(YTkA6Pu2o%R1TzzdK9 z8Jxtk_059EYjy$1TL2=++1S{;%s}Ff1)we13J|8w7TD==574lGfsXoPto0jR-B|)q z%O>CH6AR<@%l@bI;c?{+j4(*jnD(GKbV_uK5N=GcA+-3Loww~8K`;1 z!2#E=wKeka7b~|AeU?Kh7C`#@_wU=WH9I{z=zW!ij+xU1x~5Y{U<95e)F%iJkQ|KD z^$wN~97^c0iK{+L{fWt$@epa-JQs93Q+)wzn;*n}!AHvLxbw`~$O!6VMma?F^5sh> zFzp^th-<8-1>DX)_`5N&GBETI;&ot{Kz+19jM~{odb?n}bJ(4BSfWdF&zybd-;^3g z!X3BZc`bjLGxi724XfXy74UBeL`tJrn=h7qLD%qjDfMN+Dedm#ufjHg?Pag%=(<17 zIoRS)Gc}!#>wJxk{oL;FA@Q!NNxOUeFUZR(@I>ZN$S3N>Pkb705?^ofaTae>48oYu zU3{bW)a7{&!-rl18mi=vztFO0^O{}Vl~$E}WfJ6udT4IRwz1*;@8^L;=T28sl0RWX z`c1AP-uf)i?HiBq%S>>@O+TG->wxB5({7>8-qa6p)=cVD=>pa-w?=XzSc^0(-iMt4 zT!U8uNbLevlPkt|WBF>nGZCY$#{lJY^?0y0U88g_=%lm~13^e9jzwo2>F4rpDsFPK z*Zp0>MD6b#w}zVdU94pAuEqWf2!_9(aQBxlUrrkwcV#6^H0=V>Sy)-ei}nzFd>sxM z-B5eoIWsS>76sqfk8>`o>DuF?qvDRSjC#)Shf&_a=sYO{R#Q_`yOLqgO`ru|I8E03 zzrAufTHR%hFK8;`Ec?WSXo%mI46mC}uor?sjKs0(C4T8@i!QZY64h$bZn090!cRM> zUS6jfd+^$@)qB~bFK+4WmCNu@*$3lKc+%3+&gD+~^X;Gw%G@*5Q&(3X2AO;#E-nuC zOs{ER{0dOko1wT~a1=ZW@7AMeIyl+)y)aTC&qhE|Y`OO85v2j9xewRO~~VKZYTEG<0+}vsIQFnM61k0hnlk zSTmL8D#{>j$cu9YYLlha-~h0GeL|!2mI?*L*xlQya+8$1tBDrIqTc@g7!f~IH5aT? zvwTL_l23=4=kfaOrnA81YV?}W{bfWlAhumq)ruZDe|*ZO&x~N429@CVcemW+e4qY` z&M_JLP~(dH^ISuKTH;sU2ARoEj&lGC8wckF-zSO6SB4F6&^Y%fiJ;I@Mqwx)P|c> z4*10%x*gX$vvtR?nwdG#i4e_a7~*lQKUlOYHJeNa$xIfi6BkcgcFOx=JZ6LRGv;-l zDO4z#B{k4JKikn3z9gJCXIU+bXirn);q)uWJ zv;jx04{&~|zy*IWDKK~b%nEV-Azf=x;c~n#B_}5rdt3B|+7Jx|@V>58{QUgech~zR zo?wv?+qpZnRG{(oqGDxVasxB70c03&2x6F$#V-AG)Qbqo?o>L8(ZDIt#1aDcG-yT5n*vzU$Net9iV%TC^Yp{PX@Psoc&fv zOL{kaS3~x=$YoJlhZdW>%4`^g{E?neWL4EhM7z8@`Q~sLbNeI6mBaS9)o{RLl9-vE zN(RkC7eLi_>V%vlVUxd2tLDl}9Za%m*3x%bBVu4?R*gH0s=W_<`L7qimWC}gkHsi< z1pp~ovi{4&z{i#ZX%`QUCJelYBD%8(=d$HVl{9q99h?n&GMCk;rTdF+Aq6wZbo&BO z-lndO{$@p}GhspT(9TQYBr-&47nebJRI6p(WCd~LPX#FYzj2B1V3P@G&V$v;C!b(` z+Hyb(!#gWl|NPYXY`12ii)@czI*bcsW%E72*hp&D*_b!L0N6ZT`lT!_tbwvLdz=@b zoqwrhF_s^@ILeQWzUe@Z;$SFg64{iX^wuaCJ94YVymxrLD z)xQ&7idm3VUV#?PRvC8xD~c+taJHsW@4sZSbS59$u1o(Vww9anN2EN8=6+|D{Ab}y z?LSQ4Cc_2I%b7XwgMx$e=vi6A0kw=S<1RY%vkwCgvn*#0o`)&yF!RN6s%f3!s_6AN6^sNSE4(AT1x#C`PvjKH#&35>|rNjK31xh=-ae(9H%{Nj7-QIyc z1(hF7sl6i)Nzv5K&JN(dwf!9WZ6;6%EJZheM^)fz zjntrq*1uhTELFe!&S;&=dp|#?`hl;HGi)q->D>)r8j5d>(4#KLGz=JCaz|N&z|lpS zzgcR$qIg?UyNr1Wpi}Gy=sgOS9hS3S-f-`z;c(@17v$y4Sh~W-V)4dcAJAGo|Mao7 z=(^a9#yOV`x%+V{KT&d*3=!C-@u|mn=2_G5Tmj>{4cC%@Oonqw48KNqHG`nHHl1lRt+DraoWxv2je{>0b7|9OfXJgx!1x%3U59S0&o3wC=VA_Erl$QRF z9}bNEzyjKV51F?tY0CZ(o5uia+Fg?L$LQGkeBeJpxQ}I{a3uLdSHmX%qZ{5wl)#9C1x_n(jbqRPt5?tVCvwWwm8&qp5K{G;AzlY{ z+<94qs(ilY+4~3Z2Kxas1nv}w^p?Q@IEVaTlCb#pt z&Qw1`j|zM?%ODYAGSU}R=i@pyYIjZRb9+D%tcRLK0lF_|^0P-O`mMjeP|QByy)1sO zJeE3pHtAaByR5?Y6;#qk6y871K^x7b*dF)j1)nN!l>Tuw!GrwqCh255Df%ZJ^X zJD`WK?a_1k!#o6mxXD@TuSFlU ze1W`QtqpVX>`?jlD4jVyeE#c^-f_aBqHQM#9^o1wq3)#(P1JS~V)?R)JX@YVYs!fG{~*|`OJw^7f` zU-#wADHQ;~BlxrBx0eyjbGXk-NB~xzQ!J;zXAeL^8Neao`p_B;hvm&a@}*(k*>?p6 zF?Nl!f^OE?4nR{ZYdxiNSJ=g(9}yrGb+#NEY`7b4D1Fa*5O{)3`{)09txh%p40jp&sg8vfY_3GY%A+yAEzgLiZ~-3B43E zerqrdiq%89xv0{KV*pVPfzrSQJqk@t*L;GJ@I^Rhb#*li(G68DQ~p%cTDCPx;Ms1Rw*@8ovm4Rr@o1-qMo1kz%V$qeOj_ZQT|ddm9Rst>RHA*Jr9 zm?ZlN2s8#=@H|z#pZ5vUt>XyPo4PaCaDD(>YPq2%NwKkNm9D2-bw7Umz?rU(o~rc* zNd@)G7o14pcZl}NhE_iDQ}Oe20(FXceSoCpU9Mi_%;0UqAJgNw-3M)77WJ)N$-Sd5 zSI0*e0_0E~3qOCu95COt8MydcB=g$SViQ|A)#>h1y+bMBd0U$~}C51o0@DIE}(b8oxFRKv0fq>$$|~TqUYb zY=4=}nVOPW=asPR1Mjw#V2V+6_pn+&X%LE%Z{T2J4N{@Mu%aNp>SZWIA@%&E;Z75U zS}Veejt$vWmJ_li`c0)d*A{qMrW20kdKn1OG>klr)~A#pC@jol*%yoT*%=@*?#Oob zad7B;CDH&C!<%~$BA6<4bKu8p(A?O&f^iM(ESbIlli2A?NOX!Bp><)(vOGv9vw zHftTwjPmy^G|0iy2kGGvx3x=%WUc!Ri%~Ffaylpgi)VDsn{Qud^VOI}YHk2fi~{C@ zrzN^|<&(RP#^oCc^JdS@7(r8-y@mZ8SVIW@0%|Az)7I$bDo|g3>4NmlG<%mM`6BcdyzT=M{KO+2qnhFaIP9--0_#v9=%bkkB80VCO z;`b&dJuEe|%D+c~U2W}$LHfbswweF3R^#(1hwziZ(a*nBxCG?n_s)RT69a)&M9T60 z4S-kj@={A6@xKBd$`JFB>`vgS5mZuFKg$hLO&m~s{>~LK>V4>QX%u9!nr za7eKLz<~YyH^?$3r(0v??|Q!af+me|eC3dto`uB&yKK*oH3sJ^62_=Rx~Z${l@&OW z6CiojLpeA&*1+z6K{$(wqnJyN$^f9&YVq_iv9`7z1`jj_R@H@p!9gz>@$vFdX<<0n z7S})Tw5SdBQM43t>|2Mkq)IIQd=K9EBgYoD352#+0+|Qd2>T>ZyQ#Uj)PV~AJvUbx zG`I8zo~`YdQ>JM|UMIs+v4E-nG+JVCFBsof6%+WVe`cz(zZz1{OBzlF*pB+@($Z2r zpaz9k*(~VuIqoEcG|li7`tQC{$Wi%^=z-gXu@nhxWgWYYOi}Emzfl7NN^{(RNVErZ zt0hwzb93_<9CFxAb@z%$@<3Sik6!^22y(Ea!5cMFcoJi=d2Il=PW1rmaj&edK3-l~ z`ML@b)oTuW<;+q&kO1Mia5sV{I&TU{E3T9L%Z5o%W) zRO!H%vb#N2Fp&*{3Gr6Ob$if>?SR7weyXwW3BEo&~T9lHoa|M+VcSIJizbOHU;uF?qXmspNE8;Oe;A`^2nXy+tgB!*&i`kt4pF} z)3xHnZZ-s=eiasm@Z>yb>+D|wrW<1H-a-J?#k|QGw@-*zXpp_oZ56Honb8KEKpcg& zTErL$sYpVq{~pD~WNaV+2l|0k( z-im@pOb=ZVe-{^&OP(JEeWfN2ZdT}JS0tiFfixlMZJ%h|egbwwenH}OB;*lIBoxYu zmxmi=4E-Z5jvXrfHK-#bNJTv4QI+>+aBl^u2lm=PJ?Q(_`b2HWfC3NtWri0#p{Yj; zcC~p%CGeMdYSM6#Na~=#;yv5{oLv}=F9C3X7n98RVf5=t#@985Z zR}kz0xQoeL5GJfK5T`k0gj%Q=lYZbW#^Uez&RwUo#HK=W=~{VToQkz3BZ1=lub(9K z>;LqV?15wAAk~X!Ero;x0A{&I6 zKcas=DS6}lKIP1GH?NiAgq;EjWc+()Dg}(>wHbR!X6Eu4Xrd8QQ2Ld83 z=e>9ZCMM?Uc1cUz?CdO}eIknh2=EZp)pq6)q~{h`DC%ia@An}C_7&%mw>vAzT-9*i zoUk>^>OWP>6`w~o-CKaqpvZI}S!+Sow}LD8SwFaY!dDMy!8y>H2L|nM>Yxpg&ES>J z3A}Q{`Iws26v%8tzVRucA<~|&zm-Z+FZSXSU&{RpP7BqUot+)@1fvhJKzNAz#$mem zsZQ6FM2A2w!V$$W6%0e?pJ8zBt|W)+wtQ5O3U3%P&|QjvfOM<)7jCX3ZO(R zH8pjuv9U1@nUJS@eBatMEtFqkRR?d%x+A^9{F~BCusKee#i3ta_G~l`JQmsV2#!B0 zpbaY2(DCNnikjEC=npR0z9HkaTNQfWpMbZaLBNu1e)4yC4OAr8-j%~ia~l!v!1m!R zr7X#bCr_SK0~|Y5cD6es2S`d4IREZzbHKyC22E|ULeBT>Y}vPOGe-+F;Nj##tf2ZX zJ2rH#*&$3Z0ONW_k$c@JRezXTp0Bnd@r(mgs=fu+mu%bwQqLvcWqoc~P#noJCnLsG z4s5gH8@#1yKlf?*?!LN`JdW`lyHor!`Al-?=(v&WtO7K6l}$HCeGbo{>#WP|I6^Ui z;N{=B-TD93=ovx1s&^1IGRgoPh|GA2fx)+=B%MkCD<*--uLjsm4e+C?>eb%)`DU7v zeQRp?@Oto8;x}RhD8IUSUm!92RY)#@5sa!IH|o2)-;@C@ntIx$M{if6b|3VY+Mu+? zy*to-nR)TQqMu1%jPLpNi#?OI#AJPKtp`|d-vMjOLl+kp6Hsi4@m%!9u>*diN2?Rc zAOHHUr)Uz_xlz~Rdn$?=1jwi+wF%4v$K*cIqb9bD1$4U6-qF~uwU^IM?^ zz=Zz~eZIjG^xm&PSY?GiVUd-Uk@>do*pL(Siw%#P*$!2FOdy2x{k=u?qQH5H5!9Xh zLXPpo-6(Bj5)}Xu*iLsmxGTfE`8_&@MdeNAho=>jHT*Hw6}JL87mrMwpIv`=1D0TA zZEX!F>;-?dU4O#MGN8nAR(Daz3fV4tze$5Ny#-v;OMt53F|o0NjL=vSLX%Uew!4gS zK6a|znx(*y{RdQNU(U*0)4xh3bpK31k z$o*736W2MlBi4B?@=nrRbN%&viiYmT&o_Y4Q7Z!O0xE1zb<0ygrBcW1+R40=PDocz z`fY9_o@J4$;+|-zY%Xs$S-KPoV&`^!FfDDo!W15@wK0(Sr0(-eCA6WixIHoj#6Ps~i&e7GY>+WMVP_zL_Jh`*W`l zb}Ilafq19#XJmv5lSVmq#zetwCX)rk zM}%=}TYR+zS-^U%Fu*If5R(?)ab^)QZ)#M9Q@+P(BXaSqu|5yb7^|S=za*aOeG)A= z-W|Wn+1X^{$lp?yRK<8WNO4br+);Sne0`^z!(o+3s z#-;@$GfDVh`QW3KPUFpQQ}Kx@Lj|{X{3^s21!>_sfBUZn*&R6j(e5f`(h>ySs=t6p zsr~+)T16a}ZgimM04FwOX>*P%)GPM_HsmG@(MZj(fBrG9;nOBGDDMUk!M#ATu&lBJ zrmmw&{$6^7 z`6grex-}9HPcDJ(nJCcTMN>1wn6|jiV^+(!9u`PC825B}>M&oE6-* z$8*y*2-EHc9F310oOnq0!DXB-qt4ruJLtcLgypc>frSm`xWOTu*iKFHS=~@iu*g9d zzc3caVUTo7t7iLPElsG|E@}f=LxXT19W}v*(XZkn!4hX($fGBt* z;QGk>=x!tv2|NO$1zN zf)fdlN^dQ+v@EZ6z=jhyo3Hl>1pJz^e&y`(O2Bq7KGzUJPY}$$!Yq~fdG>hh;9s)J z>EyF6zvCM?Be0hbK4V*>qox*^0=mo+Agvhia$%WjfMg&l0u|pHICT8*IK}UKAKxl& z+7vg*RsKPw%aLZBM=2xI>~fl@z+!jS(sO^i?mklAtbNtVE4(Q+7(7biO zJ5Iarpn0Lc5pZT0xizdR@!o2gz)f%;(Bl?s8KllK2-7HT1;So7w*PSm9pzcy89*8N zd^Gn!ed_^+w5PT(D%!fw>yZ(k0C-Tf78X@Le?LwO10{sv+qZ8B!!9H8Ch(c$_l~$~ zQ!+-7@GJZux2SVpQTUM`1!b8ZRYfqoAfneD>D}X9#j|DPe~7&vlQRhfZL{Sz%+`^ zbFM_cmhe}rT}aiZ`n0KN{gnA%FTnY{%G=u%AaFatLd4J~BMJscoTb4n->Q7W@##Qt z!Aq2ett;-Eia!7yW^HG%i<{%Y0@2G+2YIUb9~QsJXE}?BXzsWe6WnP%$e+P-mOg#} z`dn-dUX%zft;}Pcld~BcxrOE+Wja|>xW7^xe-qbXaO@BJK|u`$oOw99PRwUsL2!X= zE!-J>1ejGN*=Z>(4i_$*az^q-{YG|cP0RtI6EZ0=tLSxPj>G(r1Jzy7IEr_-h?sKF z{>I2xuzs3JQwi_niNI9Ntesp+efQHD+gpA9L5~nPC#gT%3KAvo3#QADH20pIpKo>T z0>M&!s=>GCt>obppwYR@D=I21SOYI+2854+B}N)=ReJsmDUXio&@D)@F}W=k;dmj1 z_42x396VmiA6ayLbg%f(qNPUDd%m|-pTl3{XXDR6G_psW>&hGKp$}n1hadjP>Un0L6%|R`@UakzxhcI zaMNpd5-%v9gmDQA2^1#HA>@DL`UfGc8a`) zGOuqBy~m&6xKyGErLx#v(uS&{E%}T~OEhKUg6b`RU}Ifqq?pd81>o+sb^h4T?-P)& z+D*XBs3lJt_KpqMr!A`xIX?R|x$6E$hK|8Y!bNV9L^*!qjV_U?!CPj^$5PSPk{tX9F~|&Z;9>sAbn__cu9= z9ba26zmT#$Nf?<=B>y9TuunZ~!^bcx!OdF)46w=_iM~BgtS!H~mA&@hUG6pxnb-F4 zfPG76PED;*Goyg(NvJ$WCIW{2oWP;0?N<7n%XU%NgkOrjpk_

`E*o11uHRpCa7ndF@-pdZH2O73WuQYT+T-i5lHsIzjjyTbVNZk5 z-B%>$B@Mwv<#nL3Q?)P2V>&6{%Stv~$X$Y*RKoXDGwBz&hX{X9`~hwVc0#K45KcYW#qD~5C3NAimbNnS&_jaYF>Hf3@&_5O)~|YtvC))X1;U77zBq=U>lK9tBx|%5a@C(BEFBc z;f>bjpX|ga+?yBhEdd;9_qQ%OZpNZkv6Q48z$+GG;IuQTbeWJR=b*ST?I+{2n)1pd zMBwJ2UAE3UC~RyJEsOTX0jU&E+fj%XI{AT5Pds~+Faw0pe7*bS(Q(;6$riUr_NG5|iqTruh`oYDcl>xuqlf@7595>_0Vu3uI)bJ@(-V~=!SP|&Ho zJ$rIL6w*z3hSUN;Y;OqB*LpoMk}fQvKtJ5Ks&mH?LW*S)XwWBaurbK(XTMN~ z&g>$qgxup}}-wPmZQOKfrn@khTu2S8LW zpjA+M3--3fwY*=tU)V~#D+Vt6E$vctH<(n&FCfY)xz)C&j)x4PK7LL*UuG;-U*Au( zcNw~i!_%3+xBJuE-!{TOP|S+IO%jxR-b)ZRd3%^R+@$wg1r=VunPI~MjpB{BN&$yK zLVtl*UELwKwcm~RcAHU!dHCCzNBZ1M!7bH%eD{pO+0XUR*6UU~dp^x-N9O_YDx2QmJ?T{RLdsH0IurG&d$!4z-f->hI0e(V9Am23fmLsbgDW796HT6dl#vny5AwB~POA+jBqW3wwp{1OVXE%rDSc>68Tsi(mT`Iz z-hIEwE>5aJ%^6V0$5q;dd0Ez(UwokN6Heohdd&i(WM9hvjNjGu&Z;e4cvdJmPJN6o zZlWyk5fP;k0f+8So~9r5QQ$I9KD`^b`)!JcN%nu=C#MV=+4DKVr17c?TJ$=lVo<0g zGp~BdiKasz_ULR%I%L7ns`xOO;c7xhXH+oF8N-P9;Y!K+GsHyCP7Z= z6XWHNCW(ick2{I>6fLVBqeh=3nP1TtMt7~oetY*}R6^zjw}mFkUnY3zf`@Jr=KodN zS4LGCZEF)7*mOxNu_={O2?6O238foRM3hF97LbwWbpnp?zrEoN$e$&Kz26ubn;S`~iemUm~L4F}~v-tRV2*N2Yc0SwM>-1bge z$%Co8)~_rz^FDin{V7ZDaMfFu^vr&Z{v!1Z7;tyq3v(Iu|OBXR9O9k@q3}+*3Ee> zbG?F_Ml6%G_Lsh-Xz8Rl45WDW)%-56WMRv~MX6QkpfoFkzbaU8pVl~Y_lp7gl3(wn zdr%H#D7mrVw>Rv(UuLJL0T!;sd7*;g1)-EK4A7l}HoF1%F}&r9J>;A^{jqr68aEY@|E*kW|@*}uazZjJx-hP-lx9BOiecQFn2Ah*nKPT*yKy^?3zCKQ7eS8mosA3<85a`9UvSN+w(rJXml zsl(l~T2nLfPrTlg53PK&&L-6*K_i622P|%9rz6}&0yf5{egtm$On07LNzoGItxa6@ zq4=7#y#ugqe6mx^0~FRo$B*7y&t9 zd{@DYsmrHx4GgTJSmoaBx_+Q6tXDA1LXh#sHq)P_ti10Q*%NUH^ozFej@RGCIgKTc zvdhO9S;?kj1L9ZX05X#&H;KALxH%d^r;;4qb`QFw{N5R|PtgDn zXw5PID6`iTy*YZXh)|Am@M@2=6Yumb&(67(A1a|r`F2gRIb5RYtI0j$pr~`F{`hcS zT5p!hOYR$I5E+TS0P23FS6W)x-Zw>mVtN89r`NYTYvmqyhI4xRHM+1$1&m6P)-OlS zVL9~Y^fCe0#VCC|d@;{Qe@`;*K9KE`TmkaQly6l^enoy?`&Be{28SgXdl{otf+&aK z*qeu?BP?G$ehCoQlmFqc{<C>FRP+Y0EJcq6ws@_vD(>&CxGvs3!4>*~P@FyBw<g(E{p>;tUH1@qUAaap30Ao$l|MP?ZJdNM$YLS~?#^xLSBT%fK!&3+>hVjk z|9VM)SartpDO@96F z2t5|-dF>F?m{JFvug?Hf$+P#kMm~?Ru-~y~KaIg$vkamLpkWqVc||Jf=oF6{;=4<| zO<+ZyUb7i>ITy3)3ZC9&upHUR2=7jYQheGFJ6)2Ds+dJ~FK!v7@21Vz#ZZl=GZB+wE1c%J2aJ-KfK z{|M}vpZ&OhAw_FNa8aRI)(kD-HYJR?HqA=?XMUyL-o1iVsvHumtH!wo5+C}WgbGq* zE}Bh_#y`I+#8`eB#jHFc8W@npY!zfc<3;rCMjivvHB#Kb{f50COvw30fUmS6B-`(? z&hI@z((ONUDK;M`gMiTGU!&zbl!@+{d^ebs{(&E?ee(G>Q@L~&d2FO;e8xH2eeE?L z%gMj4%=~K)rYbxah!$lS z(51oezTxS~)6UF?66E?menwzLYm0z=igh<;Y>clxk>^=Vkv%Q9(3=_J62@qLT~yDB zAtJ%)9m9|vy8=cI6YQcMjU>w+vd()%dVqq8zm_$?W?n$9n>pph3!NRVMD+0O`>}9F zsR%6J6u_tMr}er8RUbZ+(!EhJusvY300#vAwo?RbaOPp&G0K%v!%^UapWe}bA;6t% zEuON05_4*@d(8u$aURSiqC-ezx)w0I{E1)~_usK}U zIhJ6aUSYI#%$_LUD`Ok%@9{Kz{)x)H^Ks4F9i7Oh{*Kh|g1-xSFbJJXh;HO`?gSok1qMrVl)&v-vQie5dK*_u|g20oyIGLd?VT;n06E;l(XeCuri9XC%(z{%X<&LUn>%c z${BTWJI*;zJ7E3LQGN5SlUKp6u4LY#?{(9I(DvJ#VJ!B&Vk`ER&MDz&5SFxfPzMIM z4Fm)Txbh~b;#@#Wg;nAp?DOwW94oKn_O(Z-Rp*SYzmd~S5kxbI2;U>K)ePy|$gOi6 zOAI+hs#6!cy90;1C8LoKLvf$H_=TjqYT8X`QWxEZjYSu++1-@}KJ$(8BwiZ?CV!01 z&Sc!~;{_|?Zd=FB$bDo=bnx~ZzhVZabTwe*)Ro+cYdKo7n65mqbB42UF0!kmug%KK z*rDZ}BY|}uw8u!xw<&Mt&r=fwIt9G1V0wDN9Au-+%I5{OyR#qI){20XmB)ad!Iv7h zr8$UpbpyGq0x)dB-0$O!Z@iUUd75`>fN8^NDYEAJ4)B9jAVklbPQPkOkjqsdN?NYY zCk5#b>9_M@A8m339jVx3)3UO z3#*ThiyL(ObGT=#KTZ&Klq?@Y#>dOcYkEyvdv!jw5x@A7Q8b%R&ecKM_QabnkwK(9 z69#KN$u6Xl+KbtTbNNNfq#9Z|v>W6w&mI>@vK4v?IoI|8STjaj9ct?^T%4W+dE>C+ zz#IiBdpYlJ*qZCtiV-%X3feq{&1n94+U)aE$`SNz0PWB$uWcylf9j+zTx1wHi%oFH z`BBy7wWV-*vxds{r0v1hJ4uk!Pw&8=P3ctA&`9p9bex(yr}Kv6-tlvx!nPi@*5pzU zy%aF7cCjuQTVZCyh^5Egsm=zMY*bjzb$R&qNzV#S@(O0-);TggbUz09%{CFN@uf)4 zN7;Qwk?+n+mDp_Lmv&um;r?|LTxCC)|7wWYf443#>Y<#>qE0d(r)3WM9A)u9?&fZ5Yh$K~Zlee< zpP7x@(>9`XND~=bMGjfh9dx?2dCF-z}=DuT%b2#F>h=+M>%Bm{pd`z1_X#Q9iMSb%=71$rdA$kQAS%}>_ z#Bv3)aTwEu2U@@!{|X$U=dipvh%3sUgpR*TNbs~!Qc^mEOfQp_qXzssoPeM+hkw@p z2XtnjDo@xOk6T~)b1t>dY>ZpHKLHGU_U@I+1|Tv9(TKBf7JtcS!i9@ZqOAf*a1)AR zDM(JJtaD4za=MN(35<1Zcp?(9aN1>$fVU^`!;FZV%Pqh?T`}Vy8o6EGt5xgyCOdly zB$iz4AtS^CymXzy_v>x+f$vq+)i+AfQzJyg#CQ+MivAaw1)rA`l42+D4FmgsgKyBS zxbDMdSM4Xa=^6tQlL@!joFqsdJ}0m_wfS_ADhR6QXwtdi30ei~fIvD1wnRh~){obm zrBDA&C(2G;q_MVl;xw6Z6tFwYObiW&;o$Z5r_oAB0buZ5m0T(7i=o373ArgKW{%mt zcS-=?j^tDLM?h_+i%=Mlg#=T0yZ^7KO_7;5_jI7@3q)Oh&2|X-1zhugK);;!lIm8w zdUd=4P5|An2-JPB|1Z^B%z84}MpM3G8FjR)Vpc)lf}I5b?nP*5Xf)K+)pgq2+o>TH zl=mkv5;%LI;o&@R^6m7gwUt@I_i&dV;Her455hiIV%|;UM+&8z0ea}_=eh*dYtWM` zMD^r{oNiW9T?^AA3)>v?Wfo`oE_bs@Ib-b20aDudF`Vw$cfy~v@1GbbfE0bcz$7Bj z!*!0J6UQfXZ;6Y7bmQ~#O}`C^9OjvWG+Mi)KG{faGnSd8LDKBpAJrJC=;8)z$Pn9? zmaddxj=aVob%w}bm3=VRnowdRz7*`2JRg0rWs@g%!&EwscR>r_pV3UD@Xt-h`<;3- zIts=>w%9oU&Wk5-*tb@SQREMg{)dKfB!1g^W{Tg+kLVGvn2qse$y*)=qS+tDgOd7g z;4}L#md)toGPjdJuT=u5_PzP-+qcKza58)Rj%941Z1{8qz%9Omij5>Q6Vn0avoB8Sdhu(vD4`3(p?QbvZPK74;+ z+miApS}zjROF_t*gahD;2Z|`V%g4TYgw%*4HvIg&3JM1{X}nKSyjKyaD|Squ;v5@s z<>NPq@HQ4ne*toQp7|IcnR=b2dvoa0N!MjjkSL6SB}0@m-6=ermt}xxh)^^2E$-@ zgLO?9P>XJ$%FMOOsXV1gKXe8l^4qL60CZG(cwD;0fmM{Pu5ey)E+M+7fT1*An20xp?j?d-Rh*QKFWG|Lczz_Q@O8JTt^GOl)JLm+;Nx`$jza$(^`;T_T z!ds2Jf*(W!=O$-H@KSlH{@=Qe0i&hNhpzMlO&rz@Xkx*WzLH4;m?>mi4wBQ$vX zmqo=>6Fq>LS?CfiptE$mh$5Mx3ut8*Q}Ae`#Q!)YKc`$B3$`3hP_w z7A3uhr6r(h#6jVUfPihf7*!iPpq$R}D`k;LId(Eo4u0}yu4jHv;d8gZFB$MD{Am6| z58%J_Ue(jwTRq<)~t=)}*6j3Y3zK5{B^dI#aQ(XWSY<6#;krdJ@1u57XMXT&|cOy|k!>91u_nL^mity#ob;fxOclZeD zL29w@PBU*R(kB@Rl0AkTOC%L_7ZJFAb~63`t(5rZGRl+-NeKy~oZ!g?0FcfHM+fwqLBHw>v`qjotMlMEF%LW=DwN+qC1GR33sZ`w$U2G&Gi61H07iH zttr=PQ*}iSZjSKX3xB7bE_6#on}bbeW?YdK~Y$UErTvNiuf_fKJoA!rXis@pnWy3x2MF2)^Lkn#9ORDjWm~$nxLG z8q?qUuOz>;-NV8au zAyA+_kU~Y2M29!dA$Ti{9HbBD;!GUGWJ8;&#la~x1MTH_(WM&fiN05wHP7GI|I64! zvpI!?@+KkiEtvlVyj44?jSj(gmQ6TFi1i(|6ftl}g|h$tRxfW308%dj*=qFbA1tO* z-8mwiaEPxILGRB(N*PTTP#A$QQjm@Xjqhx}+Stf2S{x9e&?>vrYt_Q!_&%DbC@W!l zAqiB`dCh8cm7UO8PNwsjT||AmqC!Nq{7bQ1&)y2L{k{j+l~E*Rh^I&dI}cL z@AWold{_UehTOoGl0x5!zzmG@ChTVW3?}eQo1MCSk^j?l4Oj?Ap=n{`^Uyjx^4E~g!70Y7) z(kV$9@sBxofni()T`8Fm@(eQ}pMPR;Fd$PZ`0mi?3L#1^4fmJ+F`PWdez)gvmjJfW zUF#ywZI1-b8^_$+DFFMb)nnGSBltO9EqJwMVx8o}t+=j?Gf0n7+=l6V1HWrgh1C$b z8X*w`6O;2uvjg3eCu$#F4*FDjyY@fJeen`VDgU+E_E5y&FKJ^INmT3B@`&p=0O)nx z8qp!)fF5P9`P4_}yNrzla}Ur9|KOTmUTtGJU|%(I zT`vlI7Hoj#-QrbK!rB9BV;evg7zUlxnktBH#211p^a#iv>9OvE%3!e-Y-!Xd9ClWk z;@ao`F0_0^7%$;SDK9gm@f@IokC9D^ou_Bb*F~>i)yiO}Z|!d`#HC-j-9Zlu5V7+q zUYaFhU^fx~Vtvir-1klBq1~(C9_g|WkN(Xam?a%C7+WV7dWQX%XEx#LY5@c){?Wgv`E!N^l4UOu$EU(73WL|=JgdWDDT;7{t-$dNH}L3PAC~b`^#>wl;mz@IlXs zSRuP+a}tDW9-5m^##V_7^}LHeS+gpQo?uf7hHT12#nB{7Qw_&uAXdtiSq++a+76eD z>pb0h3Z#7xtB_EBy%}=$-x7)=insvd44|!?b(IH8gBsP0f+G5ad~dS=5Bmk!?dSd} z{|?~TfOoH z>#S~$bAU_-<-&_|I)C*Mm=WddJ@DpE3FfjjkB)t9VU5sC#>AZB%+~?&-k4NSAmU+5 zK_rb7QB%&J&d5H*)2suT($6vHbYd>C1^|?ePH@`0M+K2Rg=qUS3#aRV~a~Zqf^goWzOn;quekVK#J0hwn=H(0M&Jg|Cnj z%Rf8-jUc|2u}bnm)W`5z_cg08okzzj6U273-%~!VH3-z%QHB}P(n$1Y7*Re|BD${K z!fO;W>?zyAdf8PJC3~t_@kXh$IwyXA0LGA_%Rzy^7d`WwR+of3LpE22Qag_l?*ZP* z^^%XcW5fc9$AJsB?XQ1Z|5%X_oKo$gTm%&G#-7%qKzE14hJ`*8C`hHl{VbkCKr?g< zuf9g6fQ5~sxS-^>YI`;J=bdLLz|2R1l9>6;qTjg!IFL3KDeTGi6`zfN@q!%!ffoQ` z{|ug#PyZmU246FobbxhqckmF-s_p7lvji9xB65R=3@=OWK!(5?1ibXAYbI?#oc^hv zfx*Kklt@vY_N_qz5-nN!)0H6J0Y;{dH z@Kv|_rYJ2k))vtBzN8HP zKU!@fKrWwI2a1P@V8)t*#0EnV>nCf4WCMz!`o2UOoREiMjj@hA8Gw7ZzNrNcin_kB`2KA8ZIw*&Ij8&LvG)8Aj_p{Zf>5p`RU0ES~@y)AW>}v*EulCd2Rk&J6Pz;zIkk( zO?@U4sR)*TAR&bmwNmF3icn2fcJ`aIKoe(9lJI!vHVx{{$Ym;U$HUv7jO#s~wnfdHqL%_?!3HGqB(hg_ ztb4fF|7D-6=1*Ul#?Vf`o5EJ=Hs1~$t{+u)OC8t+gxX!Om)N9(4wYdQ!EUy(b2uF& zBZt}?^n%%33vqX^_!NY{{IHEEmkP2SedAbt3{z%(f2qjC>=*#eh`k?oxm@txY@ zzf}fq))4mTH!?tMZw8q;>87^79nW@`2Cx8x(q9W&!-n`1F3kR)4>) zQI%8rMG44%#Uvy!8r9s-9k4m_p4T%l)3Y!z*wMnQXI{ij&phcRXP9oYTv^ztahe{y zd2WpaArMqESNSTqP?`$4jJmuvjE4TwfpY2cAe`g&*>_uK%H86#*2Qc_fhX;5eCg#olmr-3w zV(j~74~iU+I|o3MQt17ytRqBJVgQL7WfV3a*`2fu8>|w3{xr#usXs_u;sQ!x$-3wu zm%TS=9unxwUn-}zR<^;)3iF;VI+Cm`oHu|e|468kYV-8T&@)2~Bs@=t%UI!gE^(iQ z%)s}GF^AGFu+q})&)DSiO-A>(*o*A;Y1n;)EFm^>*%~k~nfU71va>X#bsW!!u{K}e zIYPRGLM0#Tn_$Ug5~;sC-k)(0BT$57!vb=j#=u~d^q;f9Mri`WH1XLzfCGMp zlLQVPp7zCAkVblY&r_cTt^f!QJyd>=J{^Tj0n0*I>*J%n8-7HYXssa-p%31A?cqbH>gc$XxcbC2SoNjO*UO_@&R7S;N+L^;|IdS= z+i=j+8*sLuHT$f7Y_7CSlp(1~5 z)xnogwZSy2as)PQ8ol!pE8k_{dz0Wb1H5@z=)Ly>GP|3;kw4jR z=xBOOR?TYedssnsWMNxfrKXf$0j|9hfX#k1BIHf2CBX&?6)soTA$GbBFbde>KZl&W zima^c2b^_Js$?IZ-ESdzz#24eDR}T{*cOf0%^>JYI_p~$*mS$4p@)heEN89b`xP4S z0OeT(=v1EkSK2=3&H+#0cHzv7)si(Pc_B&EIpb=wc=1eE4!TJ4Ly{ zR0s~4-hweY(gJNe!#EwHVVciQV3+;T;C(3NN|4ygpvLpF;E%Yz{wWzooOL7Dz=pM0p*sn zr^$|)Ech?Ysih~vh|BPR)ixna>BCHCG_pWQm3jpvOXA^lE-6{a_PWc)D{bzs0>R|= zmjXpV92w$=m1Lr0W_I-3J8y!}XcSo>hkP)^4}BQSit*k-kd#UiBuDKC61@8$n{OVS zn8*VH=;21+!3#n7f&oavAA6X=s*s^m)X?a>Znh-#LrIL{WZ7!t1a`~BCN|+4TDmJK zwZ?nX5LTq$=I5KTW0-K3zx14VJgZZz-mU0QH_-IFX;b>YU#6s_ym-bTLg-CdTc9-#6 zUGT^B;GrZiJ?0Jl=P+<^uNt-a!~yYS=-sREb!ot*)2JDWcma%{;;r=1Rnl!yRPT(nK>-?kV4vuD)<$k;f@my=&6d64CM8X{9NG3LJ_GcNK3@ zmZtuoiDpLL&xNLt(#WCmQD`*tZ!JyPeieUoe<*1XnL~}>MR83$@t?{(mYFruItc0T zb@4x`R24>M_kK{U66FOY85>C+#kq*kB&ulm-+)q;I+}y>%k+rW&AY0~NPn@T(5Xr> zPc$*p35`UQ3_KWDgv4br)8~FONcN6^6PrKb#nDf5dPX&XLgjs}l`>qvk?F z18lf>6dKX2mWih4k=YfhxPKq9(ih|*zEev6$%*p>ybhOI!Uj?`Y#Jz_Y?Xw`KO7oX zE%f(!_^_r;eHKm64bKag97E|H+S5cf+nC_803}=*}QXTNt1U&)IaKAD5%gP{I7AO>51;&1w4(o%m-`Fox z+>k3&dy>YfA$f#oZ zyCQGLLxz>d{m1;gx~#1KpJc2yH-C7Pbvg`;7L7imV&hr(8i}nO913G%VpPmiZfoJ> zT+_=cMa(f9lp#-~c$v~kHBEK3tXme#O*QfM$s7(`{i5h6jk?D1s3l zaDvfX2tIF*ndZY*4&cV_H&J|j8EHf=G(v|*xq&kDYGd!TDpn*(a^1%U@;&&y*D2&_Wj@hy>TZFp2hGKaarmZy@I2aHo=~ zo;xYW3O;YmXuXJbH(*U~dV#e#LF}Kx1`n9Stnn^3prHsrmSqORY ztM?-&v~UU-q5phqV(5vHVL|+|nT4@3DkS(C#{oFOXMvh6kCSyc)jnd`-29>;DraaRy$|OY@=*S(2V)1^gC5UwxfG9uR-lm>xCw6cl2r>DQPnz;PK78WCmwXjxxo#^0b%ErnClxi zJ73>=381pB17x-hl&!v6d7*k}p9~{oj*fu2f8q7*G%1Quad#2&)ZcYG=KX@H5bnH# zD;BCjVJ!=epJ`_yBY}t>L9QY^IJJCE9m`%d^br!-pkNX(M2HKjZQxlErMx6c1?vn( z`qmflxN@2)66f69+@8OBHL|)vT#_v5rQaIGA^`V%1VeN$k558RQNoWcWOe=lveown zOR;Hh%Mc%hvLZ{NBTn5eaQ8{k!JPw?3S$+d{Q1EFz69}0W9iZzv+}#KB#c5T0CY&` zK^Ttfz%5mUt9BLUP(7jX!-U9;E`Y(Ud~IzhMH$NDUU5pwJ`)5j+5!9s5jx*sl#m|+ z{D5;7#^!ktdMcocf`lYHa*E!(2v=7qj)mT5_ftCm(HEK2$6yoxQ?{QFC=@~-|a0TK#bilO?RFh36C<6 z36%M<9?hZI7(G35eYfA{GLTV6R;iv#ZR85jKsH7T+&Ld$SLcJY>|FtIhX`Qhn2Xj> zoWT}yraNUL2gSSxbH1s@`I7)g1Fvh*MG4NSBgbB^U02^8`?cLIeU{}kB^->lrpTtU|lVTfd6gM7>(tb`>K18Bo(FQXyLty>|ve2A}_@8u3)AkWX7@k zJ6CO`=@?hE2~&E*N~DqX#R8Glqq|>J4tXdHu-w?aN4tX-zg`9WQlwhSGy@IY6~v*{ zgA3U&%p&=mjdPJe`*O2Q=F7mWN5_8#sWjkPgnQ8`(<2zlE65V4r-Q$&Zz1VX#<_^s z1EDHnMv=QB2Mf*a@#SSX)`4Q>RpcxUkR7&iw;G(X!aES9y&D!s&Amu0>$BzcyN2Y2 zL*x*uE=IViPuTz?kBrLb2PwD8g0GX@U{gJ+dgM-Ep2Cu;0uVL;SI>kZPPXsZ< zg5X(h4@R03m^2Dv3q8L?A;EsliUn~*qR>^-{Rz?g>W!gTYO9+=il6Qv_KF*(j^^2~ z_&5dZ(+lv0U2=FpDPrHup*5a$a+~D->csAs{HP?V?M_TRKNr}^QJF?>Y%M(Pfb2cG zQ|yRO9S5&eldC?4OdV?MnCVwl0j`L&LcNSAv?x2w z6i#<>9h5OLE&<>(NcL$OcOP?6=pa>Expd<_GIzbe%k0?MGGG@GDbUCerkL}wA0aEF z19~aJ=(1;|%zGS4=suhG9e?*=J*Kc{681-2COZLq`H_ z`%U+h85x~75KCzkoIX-&6!Ygos(PB24`~|oG7QgK7!jIcoI7A$K^Nrb)P_P9iV+$D yHAn(yBhIJhg^uU3M8>ZgFKZsDF(J+|d0ZNUQy34n09@#frK+f@P$*{^@c#gkQmLl^ literal 0 HcmV?d00001 diff --git a/doc/design/images/feed_forward_regularized.png b/doc/design/images/feed_forward_regularized.png new file mode 100644 index 0000000000000000000000000000000000000000..677e99bfd9f8e72ed9fe4b27127af2ced202f447 GIT binary patch literal 46036 zcmeFY^;?x)*DegWB&AE~Mj9lgr8}h?q#Fe35@b;VN=PH!Ai3ynQ55NJ>F(Nd-Ou}c z&-?EE9s4iX_i_Kg;acmu<~8QH#yHP&oHIgAMHUmC1RVhZ0aIR1N&^7_sR983aTygE z{7dCCVHyMkDg=3{SFgQ{cQequ2(-@6RRx8p$t7xNA(mw7$gibvQCuj=#B*eBg^<_> zn=^FNrn=iCmJ+|lEEkYw8azT&)dQLT|3FrV7uAbs^I&mcU|E4@hTr&1Jvyw=3 zIJiiWHW@E0Bb71)2{E(1Vg@*?H1Mbp@u-6PP}N{=0dWXu&vQIRQh8svEQ#widzL>T zfy^r$4Hduo*QHpI2rBwWueonpRY9hm)ca~%XVSvZz$zD0+C|iA3X#gqyOv9qxjh_m?9Pv;z%;IOP_7>y?AG; zzKQ&6@dOp=kLYlWAh&qs1eL#dUrrgof5Rz=h^~_xi!&>Wbd8HDSF6#o&5ywPPXOeo z1i%tIk5|hc9A+0W4zg$nroA0rD0XTrgdUE?X#g%z`fZ|_75 zV)n9#s3_l!sc#n3uZV_+?k`h$?ayCk{7{r$3*u6%A=IV=Pu8n$nI$J~eG&8_=s1*3g%|HI zHmG~~(&zT_$jHsj?Y4Oo(padP#qPfOV}9?gpep|>_i3rmc-bzzTOHq~=|)k5*N2n&kM__{)|;+2(qcC^HY&1bgRc61u(B1n@D9s;%b;5# z4X%pRu9GLIY{SHIm*x!~`}FBkJ*BW0XD9kTB4!NT9P&)Pqk%fOS_Wx-j!#>7*$F`f zwd8P!Jk>6QhDM*sxLfo`*=Q&PY{aSSYTup3kMx&E3=VB3%9)&I66-DAClv0@^AjiG zhfxsj@DPMB`{zZ~N_ePZ!BCJ&lZV&>_B_Ohm-QPP8=j4aT^J+dhY0#N3%&>P+S)|^ zMvqMU6I z8(m6`lRXGd-3F{$=a*!M8rT93I~ z$oOI)EjXNcAc@yo=n-Qf&kbL4$%&(YloS+ygG+~li1dL7Bp&V;+_K+)!KX2z5)N5R z;I&CL|1u?qS0vfN@$340)_Hh6*@V4e-ZPEW5T@q9mu4cMQ)$cw^6L1vBFCV|q|xSN z-N1J>y#U(P#}n|)A91y;>{1Hq;{D#CM~()O2iJ*gGWK34eD>lE)X;l9@p#6*Qr^L# z%tYDi{%X_m@95VTAUjh8Tyxy^5|}jCmcuEve=r{`kS~*vsuEN#pdMYQi-YT-Bc~WW z5kW+s9Obscq{}8oY?rq8U%tNHsl0J?ov(LHbC{}nh0|AP{WDX+#HszxQ?JT2*zbJW zx=!q5LOxQ*IrP#u3IxCI5~0R-nOc0AgyKh87$k%qC(g!E38t;3C2we${=1^9OM;1= zJ!xo2VQ^re=&rF)#=`m(ic9UjMaEKcEIq1At%w72wU(U)I7Aav#Y1-t4H5IPwI z!BRRzF*FFPJ}F*B$qmA-4z;NDk957+JWZ9C(tM6HbbGT6IspY6{$N@8ChY-k$AgTM zw)gUk8FWJL_6eHj_ca@y8~bIc4QU zQUB|K!21)*by=KB6aTYBI!4Ax^hw4nKaJ@;j~_2L~t#tQFHhRWvGc9csP6}gB{?aIydgn{#*$^9|0?RZJz?n0}< z@3FBRRGrKEF0G}HLi>la{dED>OvbI4Vu8+FAe)Ft)jSzrfGE&*XViTP;zw$NYCCrk zww0)goW0)rz;+2Ue>8EvH@klFQ!JT6#3!A~P#s=y))yb{8%)w_ggw5jjL-5c9kBhM zYqN_W0=+gs{?JlyGHkY(&xL-?_CNY~SLdM)hj3$Y`~kl^1^{sHla>Oy+$&UeJKa zTtQ4P&OkPlV9O>+Abq9Ej`DK+zCt3utniB@lBxRzrc$>0@-x9ohYIzcC`NT_8x&Kx{e66=7}o22yn=CkU@}$D!<}Xq+)8hfj5xFtTvU;aXXPt zx>QH%yQvY9D4I?n_9;TeYJco-JgqDN7lzLJzKI+$Z`V|cpb<@));bBCji*W0C*y^(}3z&k1! zJwBo*5R`2{SyAWh>NXL>zfiOWe}D0qG2}^4>b`a6*sQ<)=muYh$>irDJbPWE0%mqH&DZF$s>cH6TrwlX{l9)7lm(c?1`&_!^-Anrc z3Jn;Eo9@(eJe!0dvz}6pRjFUw81x$vNbqNH3F1^zR1`v#*V9WeM+;*KIxp=fb#9@4 zkBq5td@%J*lD&EN#k%eW=+iPzZC6(9$0Y$}&lP=37W!gH*FPz;t8Zub@}gUn1~A{F3N5|pQR)2S(TSTE1ua63vNj^|r%d{1%vDRS|r5t7D$^5khYs;RC_ ze=KEX`rA##g?if&K6azGmyif) zxjUadJ|5(DKso5Jd8fvWRC_uO z4%H*cu)xwLb-YuYHuMR-+i3s>vQFxq7mfcs9XSB`RjyhpNV83icIsNUOU5hR|J+cS zOUZ!_&5Pa?^+@5@wnzFGY9F+}42HpKgkMfJeKJ4ORdf-GmQDFR#$fZ)Z}!+J zK)Aiy%*qdZxR12&NPp>`p!#sXc6Zo~z5X-szD9#+?g`E#60~3FP45mWZZn8~bY){XMp*nw+^NA)nPqrij56KcAyN zc;ScS_3zhummdkFL!40M^z`&TefjeIr`Utv9`qHg?R>@_kCb$F>G|=%Wl{_57#>M zAMAG~%IyR9RgI;d2jK*47Z=)9hFe~=-R$F&sUR$)^CLMHm0ic=yPOI7eaE2!ogJa` z=mj39KyVdQCsO2icX%>rOx+mYXi8J+(aL}$I&OQ*RMYlJE4Syg?&m- zXMOhUnZi?c6a28!%DgW>O+Z(=$Fm5)M(gZuftpj~8b{4DUYjaB^}=#^7ifJ$?<4x5(ho&jcfwc_YjU8>E?%iBZ2rckP?Zbq1c{F`1FNwuMgSxwk@fM?XEG&_%l+yXqe zP06vrfS85G<2dscPD1`gX|8BKTeke0(g=9&>edgY{@6$j1&=3>963T4%n7z06W>M~ zKLfI~BN%Xd$^P=?#7!gsfNxWT-m&`|fcLB14 zCnP*Yt8XbjtBbXPsQRlEo@n5{G+j9|8^crq6?ExQ&qloX7myYoo(Ysj zsUW$K7X9x$@{4?rL|MSqdh(~_WSvHr<*rm13}*e80w=@$ol3nstoge<^3Ahf$j zW;S-B*`S`(dqH7mbtchB_( zTTWaRM9t^-8$CK4ZzRL9c*;lh;}B1=$|scGD$){HeTqGK2lK?hn&0r6Kzy zBh7I!)PdJJdD;#Sst>=d(CNgkKv*ene_|_5@HM2G46DBiq?E}-i5APdEKz@R-?QyR zw;9_)c^R1xwRYnQK)Rr3XIBA|L>>&Ldwg;>$oU0t)-yG;{Zf?$|^4K2N zk8K$;(KPlb0jbCf-0TA-4c{mw_~U-m+!&9OIQTO@KK?<~v_nIN$Aq3%cR02)A5e$3 z0G;S)ooaB-Dl=-<)-Ka^J&odMg94(iI5a$*MF0|rJ%RWW0@`2spu*)X`OlWm0R1Ul z{t${u6P%6qwrX|{J2@!{s~O-I`|6sSRWrSDp-#MJn_oFYS0|hOfLT?(Z8XG=98mR{ z>^=tK)-6rP*x_wg?{_OWORLAV@RiSYMf&W>H!H@doVu1N?Coag<8zJHn4wbWFjFfp zdb7(t-RNF;3mUrH-%)BNX68?TU&66eZ7q-OSV2s>+Yr<5I(wa$7jy1#Vz}b61et0o zuBuYA0{jus4%W2Uy*3jmE4Mgse;NhwAkqM_Vbg>1toGY-nhr-4$y$i)BcX`zxv=d* zi?{6=TNzF%=;|$N8yn0aXE{;qpNw05fDJw%|(*<{$sBOQ70jOV?h?yVy zw@mfG1|^Z}d$LBN&N0&{60_=30hOs@%u)Asme&fJUiF)ZIj_~27jFh|p7GeIb=XXl z>Ctku-&PBQXLXx(Y#0RlM;KMc#|9LJzUXJ~DPsuY3m;WP!?z3~37+(^Wb7nK64)}c z@DG@@hzAg$k5Sl8R?L6VG4}4ovM)+CHG^x3WrmF!3`$?JQ3r|G^u0Jn`^jWKlHsDR zzjWUis50w@H!cs>s;X^c-@UpDKv7uX^=-1>JS7GqCoYC0jHxP?#5yPr>L(`8QwMak zOGzp=Hb=c=cB2a81}CGMjghoT zPEE!2B_Sb5&p1sxe$6+(YuA4Lnuq4KqiG{h{xI>DY`xOSlHhsMNNpLO6O15bA4?%p zF-~02;iXdFOP^{lyB~jMHaLsmhJ)7z1>E)G_ zqW}E)1K?~5%#c0!LCE-R;Pnxq;DagLD7SPh-D-dlC>3v@ClzFXG0JFl_V` z7{IlHFqCAY8nWOT={GxtSpg+}pJclp{RKou$4_y2(k=lKsB<~fb~a&9m!t#ZOn+&6Lts!=f!k-LKG>S`7t(7O|XGUjvCOWc3HKX0J(;Bi?Rc=}7W8fb~z zEiEl8{x|2nfZVjI>*`*XLCg3N8Tq&r;Fxewye4gfamhuX4RCRN9mfHd9*fqsPl`{> zY|B-jCQV|J@;yH?84p#{!xld5P$?X{VmygO^%X~=rX5v~x_4@r?YDaKhA#RkSI+df zjtLup#MGr#0i8fvNSrkGRZe0v@O%pjFClyK4jQhvg4A*M1S&`%3UG5b{kWi)s7m{b z?GJ1Gi+(kp+(dllS~@zie+tJ*1ut9r`uZlOt2}xMbqLB$+9?X{k3!j;5w31*XoFT$ z4_MgQDR|9$pucx_cNswSi57UzW>CN6wYfT+#HOgCtZcOF)@XI;wF$*m{KE&jkzIHu zIY==qzBI?5fZ#c7N9bNYi5);RXmH~77P@B)d`5Kxt(g$wZt^@HyVs>Wan< zlId38cYD;2L48sk&ykMOZZfPeKw?Xtj1Q7VeKWmx{`fr#wn_?@g$z2M?lTcHi%I{B zd7WyXvu(WO;V_G1x0nU1`k#WJj+ApJ?MCpTrIS%oKD2{=)t_Hj$Rg6R`ca6Eiwidi zd+gL}r6)qe*qB{a^eXT72GAbiAXm%Ma-C9oO?K>UC?-83IU11fcT;L>YfFhajMFFD z0|Utwu~gM}CM%QGb7jDwfUwmQz~i>i4=BEWCMA|h&NPD%qLrw)xLnWw{>nY@1I1wV z*N+ULcSBhZx7M9~5o(#xe1800-M+q0*rJ!>O`g(I^6}04O{H$jU11OG92{9i@BCR~ zAx~76-U3AJV`a4=ub9m47!s#trHop~3%J-SKo4R<89z#;*6eG}!}j=45Kk(I+z%~T z%Y12r}5mgHr1_&cu#F=rnSNJ zV5Xu8y-(ggf6^q-2)w;jM{ZTnvO4X9Dbid3ym*J?s>tco;ByNv$C+ANonlu5pn3#i zbp58v3DU!-!JumSnelF+`0a87tfCSghmje^`ZS0iolC@sm9IV6MLzE9n(Haq;KiLF z#79)d9K`XMZN~k6bW89li-|lJC0b|-;0&Gz3-h%W{T~%Z1RM|wEig%Vs9B zvB-*0mw{BIp8TSfu-a;bq_bFm`zr)QQ!*6NgyHq5e=#kiqZq{6K&M=PoB#z)S7do9 zow7U49N{G*!H*Xx~YOG_)i93`Lkm~dDdM0;TH+0lbQ4gs+xLnWV=jEY7O1IfZi!~W17g~Ke`dsUcNk^W*p*L2zg|CW=O2g4=e_GX9 zFt7LiT;mJ>>r---!Wv&w`a@>CLm>Spa=(W}%TxIRX}?Ch!Z0Odqch~&+bs>fW^J*C zZ*pily56oiNp{{Od=9#UC0eBn5d$<9SdR^)k%IB%l$8A5E_DRwy1TpQI^mmZYil3P z)Y%g!B_}W93%oLxNAiHvWSO`iM6{*R;H|O3#j@*=g?EeopAr)C?0FEiV4`P#G$Z4L zy$-X@f5pVc>XbEqMgR5wFA{*dFD#H%VhDl{oK#ePxKYTMc0>#?JXXxM2C26JY8PFK ztE6RBM!(E=l?=he#Ds)X!g#jMr2ZXw1Q5lS*m4~~5dB)4qiL{cpZj%=Kz04(loTdV zG~LS+HDVb`y9kq1$v7HqejOnH{H!71x5&fIThZLojoBJ zd<$1se~h=jAd*AEr5go>=3SvuiW@Z+X2b+UC$Ctuu$O%JJa;CPfyhcf-@LQQUVyGs*I8U#d_38HuXVrpa1%Hbdw2k#mWGPx z)f=PhJfdp<8&6a%0QAC-X?xs7faZk-YMW=1c2AcaRW;R#rFeH%)dsRhTt4ZFDRkq*jPhOVmFGq&s&Ec6_ zSgbc6v}vkr4*mR@4E{2p@ZIGyLR&teW^IJZMuo1LfQsS^+Nj~trX>v`3y#e~%N5JZ zmoN3+&3ou%?2E%P-G97P5MBkKYlp$G{qEf0YL)WgIRJ_IEBLW>Wjb9OBabnN2CTFE z3qos7_?PeDxjPpU5@LVXnsgmiT$JdPl$8-k5fv2`sYRBRlzi&xk?QgPb0A-;W1)wI29co}w5lN= z{`oUjE7!WHR~&Y}J+6pFF8GFISrA}_USbioW%M1lQtk4Rd8c=pxJ(#Nm?~+4P2dU6 z5EsOS_Gjsd3$Iyj!ug^PPviXtt7FPk(KtZv!VYi!VSInG7WqPs0`mGhs-Mu}z?z&F z+lIl6DKFipP&pWV60kBBZT&fvDs}t-DM~`se|rJ8X7cn6LFj$FffkrwR3zsrE8fNR z3s&$ow2~!1KVN~O?CP478%(gguHd3%@17@%^t!rL≻!9U8;XLdRCNnLgTQW9yiF2qn@ndCnD^@{icj`3PH>^P2MEvL9{!@SS5~ zax%-lAZR?M8P!yr-v)65w&Y4&dVlsuKcmGKIaSNJ6}Wx$^=~a&j(ZOj_ZUp1#_wSP zjhA6SCLZ-W!p+8aZLOAk z=q`B>-x>Y-o~j9fPjjm-7~(P7pNPM(4S$ip?Ex4pyaOhgnEEi$;{nUD`Ze%o$grjMPYr^9xL!6>%aZT9VM7_l*quCoZ5 zjzigN{zzk>k3=r+jH!wl8T`~S@uxQe?uD@n02@)({awKeT~jW4ziVOx7g~}y6d8kq zP(!&(5GYijy4mw#Ek0*<^bq1(6O+I@52JU_xl6>wHA9F~pyFc%${3z|Gl?IQ+ph;H zqe`3N!34$Vcq7eO0rb?wl_lbUVYY^Wp+pw@#|7FjWAIW`dmWkEPL!n(5EBnC-kh9)v}d z<)#3lMm{h}{RLp<@jgmb{mPhVP9l%b$EvmATK1G zLDDka&9w)%J1n&5!D)oLOEBV1A11y)F)Je?SYWNfqRb6&$Tt4Gb)vi6PD!UT@t$`lMzT47zHcTCiiZeRAg3cA8 ztatf#QLTi-kTh~NGRFMjda|8Bar@!^LTnue$9W6jUyZWLvoMyKu$xCJC#6}Yz5Wk( z7dw|fX|UNU7Hm!~{v&vLp`8v43{1hSe^RXfArGzVEQkiiinu!S#Ww%Q#DbbOdsP7-UUV_(- zXey{C9s(?k3v}IJb#}kyW$vi8 zLH8~DhrAzIXdW*OOZo@u>b?N2uM~D54(s|IIY3!rNid2 zhYG0{!VU%pRKOJ*g?Pya_KbRs@Oagl*w7^yZyPJ;=|iz>4^0`Np*NM?xA;=&C?fFI zjGJgT7vp!I19t~Tm&MuMtPY>Ulv@9%VSx&wFH^`{NMWU~kV8hO5V8L|%w7y`MxdZ+ zzdepRE}X_I69y#j^$M)S2?^Y*^H;3swM`!M_Btj$0#f#>V`oqgCnkpeGP%n`)`PHP zKOnR_#74`aykyd%<~7h$2v|x|P0{wo3S;q}iN~y>FKY98mNLJHHxa2taC$V=5f)0k z1_!=Z%_P#|^JQm%jI)?OMQhu(A->he&BQ?Cn0`0sDx0x?w8m<$4?9^o7RW(Rh9H-hP>zVujxEss`Q{^sZsBb0KW+_KnV$>7|n}I(7FgD>HuZ+ zxDI}VV%n$`R(`Gcr+B_4-)Z+6tTypIiYc`WDM(9|rI1JX`(Ws$kHP+8DZg2(RD0a1 zp}82EaE0;tN?u$M;N_EG$dmt_t||F7br+!bSiZJkB!OMkDR(z+O&vbA|LBwR!|2_- z7F>J}OioT#<&mVElEFI+7Y90t37`UlC6v&`tQUV;px3UKhi28>|D7^st!FNfp&M*BzD|w_u{8C#%wbo zy5BNYrX^Wt0NUOpA|Nm%m3#2UgrgL!xFB!T|FGL)Y=~b-_j~j6-RXj@-ku2jL>GZ6 zxsXs4S1<`IWoBVHay0C-8(Xt?XM9Qh~! z^jnQVGmt_$1B&ppgD+^cmYV)X-&~Qt+SNPp2K?2$T#g~-SHV$7Kx-?vPIvtRMrC+N6e^!xXE-LK-7{Y5GOVuMBGT&7A1oEH9GkoAGj3_f16O=dJ{&lU z3c4(H`~q!Tow!thg{9PMZ>A1oczF2W+bLCzyBR&0@oCSMMMIGOBO~!vuqa3f_bReC z_Qa_p<47ZXBOC8V9$i>msRxB7BM?0uQb~vCj_V>oHtX)@))Yf2wqS^fol^%C;@%Hs z|FAlo-a`#M0WfT1VSJ4CCA?!=fQYbgjA{)N2yo#EL5o1U(X5O9gp-M~{{)H=VKgcS zMf3*4`{hdLa?QwdM@u0&swWT?Wfhf$CcE*H0;sRu+39R28mdQl-F^A^&G3C#GSFwd zyleRIP?I{!$%Q=AYaD0!d_g`ZR^tp=9WHgkwPQ(LV>4UnAUBA#tf71!&{umLJb;_Z zIRq+9&I&PBMPm4G%p0=XV5i?6(E8*lCxYm1a?}8F>G|ioNHU<74}7y8p~Jxm2J)pW zP)gHxV?o+antp88Gscy>#SXlbl=3%T9=irTV-SoHbnR`WZ+?h9+~#Hl-e2W_F^2D> zZ5}Y?vB&vuO2Rg?{i)no6wngLQK-dhv54a9l6I(s-k<`|@!=^lUkI50#U&-xmjT=| z2|O@gJ!^;*IzF%gsj(;6<-?G~0E-k>5XO5?TB4b^gY?M+j`K}t<6uUEdw=$+QNH8T zf8-JLj~aCfsbYbmbBX^FN7l$oPowTQ|C7L?xItcfzw?L3;p6v$Cd?&?%cQPF!p-Pa zn<=%-sxbU-OC*QDHN(`eQbJuhiUk8s5zeK*pnaO;+ML(dA-*2Ax9aYlGeiY73PQIua+X{Vlwe$Cx%sCGproA3x>^L^eGXTLQ z5vY$7OxVR3^J=@4uWw{?(mSSsHX{8qQ$!Vj1LIVmuB0Id(bQ&Eps|~~JE8sam3BD!1KK?U zSX>e)E==r?kB*Ks^z@iq5CvS8g8TsbsEl8( z0drlAtaq3WQT@>U?3j7Tny&c(J9sKIG&G!rh2z)2B*EBXJ&)Gp!TPT z_UH7%$v6l!U)KI5pe^ zWI`>=(Rvp`6U9-Q=L1Y!U5|=zX=M_utgZRCrfaNqNtcSN0d5!oQpZOQgL>ANyu2RY zhSyU&S5s+4S=DC|&?k$?eLUy7Acu#(u-e+UQ$Pze{+y9PZnB6}D!XWAW>jmVzJ7ak zV)5zoXPqWPc3qQz-?kJR=@x$frG)yu9 zH9*y$y2M`W%`yVMl}N~h_7oU!gj~QLDBs#*5CH^t4pQ@^=wilO`?GAk; z7t*%0vMMn4JL>(`>U%N4YmV9m^ZUQd0yc5tgVe#ckXX`9Vd0^&1+#{CbM0OpttQvm zjStNOzw{KC-{p7UVX&wG#ej~NR||~6@=8h+BBP?{#l%cZJ%TOIz%Yf$8)Q=H$;pYM zaw?Da?AF_gwGYvwSWguh0ie5{ai~{J<980QR99DT0I)Ox$d5)qbTl?CoE6 zfzCT6=jGA8Dq;&Q13eo*c{pE+EZ820Pq#DNAJmizPq1{a> zw7@kF_y6{&o2AK7)k0hXF1A`Y$^VQ1+^*SaP^vQ7|6>p};SNZhp|UdIkxEL+4h|0E z&O0-N#Kg%anwmBJTW_6+HiDU|6fX=Vh7XHyfBnvh#C;aRi|`Ym3mVEEyIbQ z?HVJPkU*l#euP+IOHNgdx(Yw|us>A=lnE@PVwx3#cxGHC!uSmu4Jw*J8azfCEL8IpkevUtcTLkDj;zj~7Of6xq#in#sPuXVgD-JQsB;-g?y?_~y%qUFG< zpt)Uq@#1e#J0!T(rL4vpcwao~(3cmt22LKqN0zSfi~y%zahYPsbv%cdetk(t;62 zVZ&_9(}cw@`Fjq9>!@Em$)EYt!G3^)3`vfZlW(CPlI#VgOxbjI(Yoq@#UctoH>YPJ zK5vz*tgLQRtTUOw;UaG3Nia=jHVL@OUt3?-VmE|UF@W2p0CZrPR`AH7h#vKeKe#T= zKi8eG2iLXfPp}uP^(%kfqz$Y8AO5N@@@0HHRJy4DyTdBK)57G5e23f4KwJWE%!(lY zHEK$mzu}k}g+#>`=OcbSkGih$Wfwn@e_6X;GVFw0Y^Y?38g2Z@Xa@B=b6!EY=rF-_Y1yzy!I`M%W8`<;yR zgDHy6s3XVE%hAD@^$&iQ{Wf#pM$wReBriMX;Jl3O1JF_MN0%^!1w!}r@v)A@OyC=zvI4H<|FOI@d6e3NdD^9(o4)?cM=%NC2Yp?Lrh2-koAP&o?3f@rlN>pE z!(YLR#3u(RZJ&-m#oFsL#t9POuiMYycNQ6Mavd%Xj*RTNEr$_rbGAks_4ppNG6D@* z`_N-AD>E}dQ%`R?5s;d)q3_?7T7fIq0)P-U&^((kB|ujH-yp$cISmE@f%4A4-)_sV z9{VhI3)p`2>$0L1wqmAX0`>$IwyIK%KLT^{KL3|nl8&Vrt%w^TiwvMZBW-YXvK0@a z$Q6T#fyF(~o2lsN=um*ic$xumrl+N)CGhZOb%(pLz=@o}YRln|fC4mht-Fa()2K$s z1x%5cjYj6!l0SQkYL{kt3>IJvUjU)0XvG#nXV_BpbG@<50oM%QY$VD_5nG>7))4zkAd0ASSbevOWBV;VxYU~2;s@b*pv9KaapzHAz-Jf zvHt!0cZG3_m#aP)1$1kbJDru8v>VIBkZ8Sr{rYs~SFWt{uX58>VuGab^dqjPxnbJg z14jWkzpe<5d5ttQXf`w6W$w<@DQ^9ZBxI=y)&=M~AIQ4(j1jf90ch> z53HOyYBXAnS`)R50fw9RBh2VJ4_;v}F1ElC!L#q*zkg0*H=0>%kR}|IB7dXRHMe}h6sg+j}TyKeV$1wk_h(MLL3U%Os-m}aFf9@;Ej~9>>e)Oy~ZiR5w95ilHq`Y*u zdS9L{=usjJjww}xA$KD%3W0gLaXhu;hXG(&S_1~GWZ*!P17bODliKvuStknR0PmzB z?bTH~il00<%0+DK^$HC8bq!R_%)ZV6o(Vr(FA24>;Hz%vN^`9-&LlYYTw=X=bb zD{WOX)&wsc|Edm1FOyISkaLj!PfG*+-0Pa^>N%$d-+wt8woe)IuwP=hW5`b`-NI9x zj+-W1Lb9oV8X1=wS^F0dKy-1xwJ~VH>`o+MK&fo5HtR-i1aV9>>1;Lq{)G!Zf@3l# z+M2e`N`%X@FGG=6`1vm;_fWx9tB^*`?6FLUc)&K3ln_DIao9D>*NDm5LbG+j3y*5w z0oi2|h%h;OYmZe;Yn{`)VA``gJ^wwiTVo(QARI+DZy}pha_hCliLNR2(X5Q{Zh>IG zhyF}IY>ndt*8G(k-yFfLo;?D&*rN~SU z8dmu+W6#MTV^-NMd0H1--kI?P)5ep5aXaZR%J zAj{kP3Q1d40A>mt3;ha;ioVc+2BKXKaEc6hJQVxo0`?Q0zyNy%xyjRp#Q>JT+f~Z< zcoonW2rQ-BJJ;Y{WsXN#!f$rpR9XMx%nL{kbNdZ&2_`%MFINewbsh+kEPLbHry(JY3@S{pjJ5FtM z-&~0a3?JLtbsECrS@;WL#X7)}XY^tg9;-N&aB&RmtyJwW7Ab~X!@F;~B}9{f4>yZ$ z{p9cdoMWdoe83_v(bZWdHV}&eywnBL&oW;1?3=XcWk;6+2BL50W8&xWm~pjju$XlW zg6hi26h~#>ZQ{ws%G`M`aCppL;95(;+4}M2EkMW!TVG$l z(9IT_VGli}w&v2NKKPhI;nqU=^gICPH*Yy+SeWX?d^7*8=mRhx$pS?@VN`cY>2dq( z%R*5gcI3>DmZIxl7JMXoS^a9c#JB-WnNm2-LfjJG-JK55#-j4scHVVN`h7K=veF;R z6mZGdY52ipDB1X1IC-9%p5F2VoWlFG^qIhMYapH$by!1NL*uuORK0G|1GWEv9#m7e z$vg*MW%^nvL*FWQe?l|S=yAE*6nltxWGGyLxyQgf{;350&xcSJKZ65M=mpomlIdE* zZLBz5I)<*&dk5c`Fjk&bAOj3i8Y6QEM#cjGK5o5hQ)6|*FSk{l-xQp5&CrB>Fc0I^ zu=@X`=9!;DW=;}4rQIW*|CX9_p!_oJ$RJFGCfxoUz=JK}QiAO_e>fVXn$D+^cR(K| zR;UM$9n_qc=kFcWnmq{(?!Nh@QtSW{P@MbFqSnFrElrgP2@QXBFrbIqCjqK>xt^ez z(2c&bx@tc>rtd{97fUY0;6fhMK@JQyj-ow*7?` z3!`6O$49x{3Pa%UJ+{sLO0Yb*SO~;F`-_9chvt4isHvG-w<_A_YapqF7@5lvQ5D*_PVi%A-G9o<=PjB~!pMp%({Rp43=O&wXBj3i3 zdpsKcuE^2)n)I)s?@{AsX6ah*+zYqJL%|C|<*9uXfaYu+VflR@v3gryV-E8Gql#ci z-=xapPd~H4>rrW|E7%Mwv;DFZgebIHLsGHzsE_o87bgu`Bwium*9sR^XF1qPlr|Wr zCmx%6gguwh(;b7KY=fVe!>xm#t~FH=$5~yxZ0=rG+kQoWn7&Zwxu-AvoR+iM6T?8%qLc1xejM2MaezH;si3FyY;=PB`R`Gz@p077gWR!UOcZP1V4 z0)SV*%P+nxW0Q-eD*OMKR7&56SxGI=fMe!5;616{4(vLr%evnHuqlORBVPDQxL8Q- z$QM<2ZmKI0Zn|;!GsO1%IgHQH;{z-+PHgSgvAs|?HghP*nOtJYbbjgUX{kt3WB40Y`!C_Ns-aw_o@OBwj@&!Byk3nbY1I!j;F!C z^{ExWe25&Q>BDCFJq_^@@}E9LtUSZkiAEjx*;XZ8fzSqE>GSqINm#-d`%=dp_wm{- zMjJ-6)cT68LY^;ePZykaF1xncQCtiee4mEH>IB@03S6KDKG>Bz&!iDaR4-u~-smxX z@$qk**m}FNtP6~#Y38dX9FtFlRdp%c?m5*cop6||viVLW!cp+cMD-~J{=AQQ_L)Ih zV?y{h=#fRjbEaQqCFID331Yq|9mZkY+#%SvfDz!3i?`3wXuzkwR*H*{F7)pgu)FFD zjx6!*F|syy3Yjzuo|9ICp_^S`*d|%m^OA71KkdltOL2&fUEE|pL%@sycHYmav`kli zog|F94kq;Nb!Jt5Bg8{j<{`X$#W%0YPk^^kk4hmPC-n&~wcxw`vRl##pQm{)+xn9B z`ah3sb}a`deNOc^MKN?#U6)&U&w*?hqDqYO$%Qv4)=+XVtEIQMH{|y64H9lq6MDyr zo$0cV2f!oAU^ZCnnuX!4c6Te{1O=8fwTj;Xc3nrob})h#E#Kqv;~M7nnu~vXexL&p z2KsmGY%9c}d(@lus^g?10NqrkyYBD3AuX@6wb6NTyLdQ_qfK-6hTM7mdd67fBxQ-3 zIQ|ClBO$0Qx)Cbm0N?h@(VY1)!IO8+sib2IVJ zp`uUH(;)c8utVGSk&TN;GDlsT=In#o6Xsy9Vn;|*;R>86^ZY_7KH7_<)>DsEjEbM~9 z)2R3p9{RnA!_?s4o-2$1t7R4a zMPA2weGCWKZ0PmUW#HY}WZ*ykC|dRQEDt`xHW{J{U1Ht97(E#-S*NBghILKLyX=hs zw--{LkA&Br_==4-K{_sQwV$5-{VCTrAhQ%;+QjP!?Irvw^VC$iROEus*$(aT$(+tb z(WRZ&C>93aMpyeYG#S}T;dX%1hmv2=jrU5C+F^=?yW&b{&#wk08^8!(A$0d^DUX^= ztxfChE`KP9pxBwSUXGMmLj01H15 z+I|4oAU@5xZMw3hgpy?lX#kKf79%Jw4BKl_TtMT~Pk6)T~fu!1|z zGzJx>?)_u1P}q>yPA9l^pL^9|&b%S_2eScagHhvZT z^lzb9tBv&E0MioQa$)wN$D%pIowJFD2(z zk0_&Kpkq;aeW==-$c9~Ulul+bVWubV*tfp@@FxOC-#IxXgS#UP(m223OyV~G9&doa zARX3NF;XGj{NE?1=nGTZWMfG3ThzUVF({Q=UUrm})~%E{qEeLvwU@mgG-0B-tdyq< zi37|ETPlW<#$%gk{?=yS|orcL_4a+WQ~xCnS`NZ3OqML`8DYiyYC;>3cv%hc5!- zGOgDrZ$d40d!Gz(TcZ6$Sv+ujv-Hn<4`aMT*foFeG4WZ83Sl`iAe`pe@BhW#dqzdI zwC%biCFh(GC1(&pvP1<*N)wwbIY*J$t$*$wmM1a|d$zjyEV z+hdP2&X4os{Me2)Rv4>$&gz;~HLITIzOSb{XepA@PFLT>3`Yw0vSm%p$;7+$-R~t< zzRgbqVjJpOsQcR+VX}Ll?TILGQ7fxYs4)G#=D*NwxjfBnHgSp(t6zz~K(BY2f2j1( z!9gLN*afF(JEPSK7$tGR^RUN1 zzRs}3yfwKL78Y$+Ja;*>Fofkde>>v9?i!#KQ`T5|jFSC4L`!fzA^5&mzp$7w+L_;k zTRVW!#H=!q1YQUJp03NwDs zPFX4uDJVJ^y6@_~Gf7p1Eh>~P8GgR`nIj3oH5FlU?9M!$HKh9X#Ao4!?JsBDD+f$qXC@HCSp_Zj^c%W-pGh+E0R_a!x>W@RcT$yLrYcf z>rU9}v<;{0KkAYvA4B9m#9;4vbA#UyG2uN0(c$PjDbWl9E*v)Ce2T(ljq#oj4&7-9 zS=;5RBX75USoS$hgO|H4LNI^lJ_L@~ZPp0z|5ZKWL0Pf$g3wC^X8lUt_E}jq(42Es zpzfr@q~FyOLg3V&UHXa2W$?eRMq6-%B(6BS(AJZ7`2XT^V0!1|(wPh6>i2YmhkImH(|7@ zaPXo4C)3!c6!Dpr%!Cg5N=kz{`{%15fz$H9Kc*H)L#U;@f5q8%FANmr=SFCGKCmahlGWT=Qq> zHo=W{r=rik8qk~mZOaYi`uowB+Cbusi(~US*7HU7`e$Wb?qDC|F``5G&?>tmYE|sw ziyt$|^mSmdqKzLkc9qkl^ZJ>F-TaIsBNZ)`@oM_7L7J)eg$&z$*J7p*Ge>;7`J-+p(b!*2S`!HN8IF=>A7_u>HUA zhX>bNkIM?kmsFc3P_8{jf9moY%vK1zr0+{N8;Pl#@6)Ha{}UZy#YC))Dgdolio8ms zbV4QS=7*GGo1oH$lLF~uER!rIk(a^DvrN6c@*_k?-vX&8E_2sn- z2?~em9%ABS>v$-+6ukG-Ua%NLc89**cvuxo7*Dq6HINdQ;H&%`WgZ(m9LpAam^=fz zWMd3u)6K80uBuCfCyf31W!p}Fs02BuoROXq#*jpyXh^>kOCSp(HJ&tqYLVjUdZ}5* zl3&t4UO@(D=HFp6>grnMqN7jYdY8LZc261i&;PAj?zE2`rWM_)znyCg+zMSlWm+d? zWysNIByie~{rS`B-$DLM6QDapp1oK^kWx;Yi`VDEnF1k^lD6a0sN z=^5a&8gpa!QSES{eI z^-9WhOzWww0d{TY*@!0vWKX_#t^>357?E%+%OdXBd+>@@%-(Dt2?#j(WC|ihiAKA0 z7p5`Pc(_$j>-_A$vcGkQtT7Ng!#R$=f<5QY%Ih@A>Uth2rtzfx!^6trnyIJ3?^d0D z*ky)m!4V5zoPWk0{XCoiYMlD(i^In7{7S`&{QuGdu6F*sh+R~Zo~M|U8;SEyox(&T zHrke|)A-(f-Q`q(=Piw8tBScUSJv#*sf64-3i@6ii8+exC|drs`}Nl=q)fht(Dzfw zY-phxEjd;gqkU@^an3wO!1K$OB?mZ1;UVfQ-5aLH*yjbGvB#~Co>H$X#1bF;E-gzz zm4YB8U+opKq<=oW0Y8&*rfCEjI$X6Nn+@gBuk(64kiBU3yOA@xb)8P*3iT)%ys>+0)QFiA+YqXqu4H zCtteBrv=YuU37Jc=P+fpCDV;)g|uQ3`6do;$07j{61i3DWZXNGO9b&TYMZ#zcrG!~{sX zlSaSJ%HWQFUW(ZG{nzVEhG8;LkO+s+;wz#gEV*Zt`0%yg{Rf00{69dX@(i}8j1Y!jFl$xAy`Bi&5F!9kND^gl%r^rqaz4T7l?l`rdxB$v>eMm(U~0Ndm> z#QGmn$5?T4g&c?63c>%>J7Rknn(|724u_^L*k0`noq~{dXVpJrnr+uc4p;pqrlRfs z+Sa5-9Gu{|fjFBKjY4}SOik!jsND^J`20~smailDL7YgdNjX1=nm&=AS+SM`GA0Z3 z1{xY{TAxwzLkSDHxaLTM1xm^Hlp+&p=Z{CA+lX>EoEa?1ZhC^g^NfrP$B1eQ9&8sJ z5E^O!`rF;J(Igv8KD~G)`Ia{g$R|&n$%_wRm`tl+grHRG6H^!acS99g-F(AzDmmEM zHM)uB_B))ykKF5;ia(>x`_A)w*HYuZDX9f}-%>!|=t+A5<=3T>ha=5CLQ zuVTf{Q>bP{3Q!QJUj})|Onib%ZB?XK6~uJ_i|JyMxuz@c25;wql_o3kZpsTr5buKS}91^z0)fv}lKO^?SLD0)wd7$JIBRZfDVsa!)orwmrOmgJM_Lm}UzwaN*gi+JVvd z+O*hTj?K!n-E(iYdb)4Q7%UrZzH;5<>!&q?yotaH4!oLjWC>BJmCqx_ady9GPuk|= zta_=RJyOglD)4`R#7q>vJzTCo(qMhh4oTFWy)RzCYCMtEJnF%)S^%&jL{KbV@g-nP z$oll@-H#uID(M)cPaa`7oRmIaWcrVc9k$S)8Gmw9!#K ziFvoxHV=i^+9-Veb~w)G^a`+)#i{C4xtFw;>;$4&2cNS$Phta9&7{?3Ky5JL>tRxX!l35JkstnU|SmKs24wm zsBo&4>w~~G3BlmR)iDr)B)(Rcgb)`J74#RYMy!Z=9j%J7xvpIR##}t`wa@F8mN?S` z`6`mr<1tbb?t>z0HYI&ds)%dG^Lx2?&rvrDj`cJcX z)H(hWDE?6+Lq@Le|CSRD803F<3x+4_Var;YQ#Eo#Bz!V0(wn%=VvG~|-En;&-s^$7 zJ^59&Yt4Bww+u(;@^hBwyiJsI@6&UlZy<>nPR8_xqO9GUB^qQVbm1d@ z8Qf$QQbAEN4gW{2D1D&iW1~%wcC`SSeF)%JL;=TaKtW#K`ZMA$PoQ#gD@+p-KV{>X z{|M_IA!p5HF+{7<2JGZX$z>!XP$=He<-jJIPnD4w@iFJjYkpI9s2_}MjhlYY^V$!D z9rG1qs5H|uGnX*7WtGGn=4upxIN#lOuEyd!u&PEI^3LE9Uvd1{k21N@yorxydK(8Z z_-B4zvKoLp1%&JZ(JaQXDX?NDkG}(^@y&0vPLm!5+^t0b03hb{=Dg?8qen>$Qof?t zTLGoTfIXrNM2dqzbNE4Zc1D}|&)b$IsV6}0U<%BH!>cMuKor_gvA5KsQ^jEdnxO>b zBL-`(O<6<<#qy7&?PK|^6~^n{iU0ht`1?k;f$Hbw}_tw~(fu3dw zfS26!8^-%Q&;HMHQ7n}YDJfg##LKn7*^fOkwhVVM_b@UtItFlzbzr<&X6NSi)s~<3 z;d|iiDnB76i1@5;op~LVhLlWPCGgG9oC1G#@)1+)f-5>Isu`)H zssr+sA3&MlvptfToQ_5Jd(A73mS{Q1<#iybq@yy3*0yfXB0C)*E%}YZixH~lo_Vbtg>3x0?|s7Mq|fQ2BL9G1G|{Sb zyZldKk|$+naTw4fXoGdsgEGO%T~|esCoet1ojie;AUxj(=s}Bt%1aw0B+mgipa6uC z=5@AbXJ<<;FE49=Iadb?h5DyVZ&ORm105+w z*2;LPywl?god~R**jKKw$=2{T+!hvo4d_Tk)4FR-glkcm0nvaO5XNWR{&&XwF?T8; zH*ReJa|Rbs{-!u|9b7H2SiH{apcDw$z{oQN5;6iHKX@$L%J>)-YbTa-Rj{i7o$y{O z`$8P4`8a#-wFBGgsL;OzVA_bN+ufs={}O=N-CNPF9TgdDTp0j5px{0a$5UcKt7~}C zLlH-Cfttn~XYz#qj%VhAYtb=GK1~ zp=qEr7CA$nuT4A$$}S!-gWi818j6ty6w%~F!J!ZQYs8nLe0-HiIE1i^i_7SkWG8pN zRi72LGz> z2A$&q_t8h94F9+9qwpY6=*T`(1;2ZmPUk!dT#G_yxOsIoMqR?V|jU4zu z9ny66f1COLQE#TDp!fY-3-Et?_WxJov|~#qR!%;w!JcC#Ic0#Fh2?|x>xWM|qD(A1 zuzPwU#90)gJ|tSq@OAXxktRTx1=Kol*Y7oLj%f9 z*LSz-?+K}yRejFMY*vgOJ(=!9OXV>^o0?+2zKFJ;WB&7J&~>lyW5EmG*1x5B%$BbI z*jlO{gZZpN$S1CZrU}q4Q@g7(83|du?{2PD(v=vKi|EC@+hEkPFHKgY5_}F4%y(x7 zbGv1~lSR&vuXE~2V5i1_ssH_c%rFZo$x#9@`sL@?j#$rYEeHH0e$lZ(z!)iLFpDK| zRnXsdp}baY{v&){*nUs*u~p!+#iqisZ$GA0FMe6iX{HC*L7`?Rr-x{EiY`Q-EO}YSdIY|?0;;O(FUjX)r zAJWn|!E5ss1i}I~#2>P=`M@438EC;2kpid>K;GFQCM~@(4Q%U;0I{b`{C%o6@EdRG zH)7IT}c$mPQ;;dmw57xylV$r7p0>=b^ zn6{v2l?f3BaWG4;HBEZ|jJ*;B)4l_U4wAsF)Bz;OqKyk}1+A@rUl3mjz>8>ZcG9W= zCGM`Ee(Kl8sn4~Ehu^V6lkuZ1?321Q9v0L#tjYg+A3d_nir|FW{R7?9dqCorZd4yy zo000Q{jC}FMYn8DFyL$bH{%xJV97Fe$}AA>K}NErrCC8-qHu82TSMOeM+?Klq?sCv z;p=FM@7a}2->}L}X?-6Y?UXXl&Hd@(>*zT>W|2NYn$#u>qOia zI+&39Z=eAHHm#(~ORpCyDv2No+YgLw%s{=A4D7=?h_hWb0?hUO8P$K;+lETOK57mK z!*4p;h7D|E(w>xa%r?FL5zMb`5EV`>hW<9N!jRaM={})7D@lYbns8i1KL+uOG52;^ z`K^eaMLTUK&`SEJ01vf8BD%2VT=r|OcKZd0$g7k(nTF~jq0B5IB09unl!opJr~jKz z(2P8fe4U*6yfYB}4~R_+f(_>C2Ipb>bp^8t(0Tq0qjeO0*3*q9?zjlg13CV`d+uV# zk+$iO`y8xAMa?y@>?NR`5)Y%~fIw_*n}*avNp{lxzx4d^2Lb{=zLY5ICx47?t9kXI zq?Ft){ezi#A*Ynl010U?IVIbbuy9S^@1%X^8UuOHsGdFk(TLhYHWH1pijIiqd0THc znScB!o7O%J?oM{g#28Xv)h5&C@_vRk@}b% z35PPP75C@LF-zY(0v*U~&=_TR?YW6Gu+`+}$N>M^CVfKc@bsqjRMu2=bK3H}@|x|` zy8(X%{C`ZsVB&~u0YPu|SDF%5U0r}^Vn#}p0U?-Owg!yrW*1;0kC0&>1&EX|d|Flz zXDv4m424_(RlNt0x0yVHK$2~wU)&JHbVC}ylYR>-1U+ zuA}V)r7}->oRz=|lRk#hxUeL<395qlA=<6`Fuz zFGjd~fEVq&?sQ(iSteh+?1|a4mq4S^q=pnkMbNo_n%O~c!8R867C6^-a(D`JBV?E`ela)$ zbq(PSPy~P2-=!kY0g!i4+(j6GU8$eDJu4RnOibWSy$5*m7m6%ZgVa_T31?txsJ*RR zfQ8s;h$yGhxHUQpo7+xxU;g!76nA}uqZA{vSKSwe4Mkb{drr&m!HHA!^4Uu$9fPDV z6(O3UDT=DRDI_sG_u#o|@7dg(b#k`xx>f+$$jlu_FL3d^B55cp-XKR&N4vj7iN;w2 zzaOHKtiZI6;V5z3L&|8-;qUihpP>rC!l1yvx*V+IjvwKkyNg2T4AvJskoq_x2zNvW z^3Bm@fM%1cg&%dMgxD2aKp=>>mCnM~EG%nH(R_++N(|2=8LC{p*SadM55C@73|2{t z(>9_g;t!NE<;To9Fzrj)zAMC)W4nz) zr(dMxROz1L@ZpMgfosB9+0yxDD&thFhGcGV`S9K}jC{Tx5$}wDupHs7Qd1K{n;}r* z9NWD|O&eVpP5JOVFS>iG*ysA)iE56Mu~W*iRA7V-w$4RGlcLdAtb(ECpKnaN%M=vhtOeM z+nH}aI9_WUP8*ayS)dn3@!rCt`RRvmuqIZE`OU_#4l5$I9^;nU3WX9%wD`GkR0=nS zF7hUv;2n?c-Iw{HMc3qapnidu$+ASh08=MR%MRzyk?c5Z*s*#5Z?f}Q_=lYc@FX)< zkG(tr9{IbuRnNMemm+~Ka4o%2+EO|MLc}9Qx+*yffv?kFinH{AiDi-wrlR(s#Tfo- zw)I%?Dkv=M_+<3%CP?a>%6U7}pSupd6fOr(Ym;E$HmJkOQsMm~CNJfl;VUR($0(-_ zN!kRcuQrPDm1Ltf*gagB_gRV5`yV>4qi4I}U^4*4qv8shsj1ahRuvX$JEu|mJw2?q z;ri8DyBw_ojDbof>*ziR%TJv^Z7{Dq&l#uG8NWR^U%d4<>JOD3=g>roUK~4e8Oi63W9b2Et{-CX%cjK#;_liDJ-@`b#8wD8VE^tP3X+-R0&on4Z!x z*d0B|-1v6yZQ<{W`PyQgR!__rNSDJBb$M^^r+3uUp9_O^11m8J?S-vjU$3shoNVV^ zX+JpQXH-`Z1v-;h6H`)N+&FdIP^5qOu)crMu;3`?>nl;DmPs?>qE%W6oMKFTe1xMC zCozqgKxIznvsZl&B)i$rXN}6geRXwl$JH@7O=_qZ$;1Ut4&UQYal`u=Jka69oyo4| zr*83`m?1k90inApe5fMJSgjXXn=~dLo!U`TIj0$;9U$M7xsG6{{gPhs;X<*DB|pMy zD_t&}fd z!)9O^*no-&M#)&tVEt2MSk&*JD;V?BRPS2vl1Sk|h@IZiZ>gJ$4ufpp zGx9Nb_1BO63f;m|%NdiUlwmyjAvybg0ij$1o13V|QhXNgsNxP#2_@g?nZJW`X=&mk z&g4d-$Wv23TE)1oyf3ltIk=6DIMsg+SMH zo9|#WL^XG~goih?y}eTmNmH}f#{^Hhw_Y#}0mWdJX96&Y?;+Tvyww{GQa7g)x&WIe zQq^IA`w-WCZV{KNV*?UTpSR6_T+mpu)feeks4w|}C~oc~O|o))z>v}*X}Tqa_;)EE0T z+{WhHyRJf_@6(Gnw8Sr5_r&85OTY_FN9873<|)Q6)+*pP4FN=vaf9IR_fzRhL2?aZ zb`CNDYrvn>2Oi@wuto_yf<$_@1L|akXHQoG+9wY`KtMKd{_%n>7m%os{J3#ovV`?j z`(C;7m!@PUA%&kY@vCTWQQxj4*{|tqrWV*tzziHw-%B&-w$oisWTP1w^%)ZN5A#a| zi?lqp2_l!iizzUvMM0CsQkkYu=l@TM*&T%>FwVu z_@4nTl{$spZ5}ONn21>yk~g(}0s!#2Kx$&NHZZ2`nG`I8sa0ttsZlk5fHs^iGP>O4 z#`Z8mOq9L6#Ywrr*|53Jex*0tA5BfPtEOfo)%0FrTo?)XA#^7WW1Ud1yPIE>QE+&J zbK81U6shWLc$KKJzOEBI_s^Sqxh5x<8%Ejl3e?o!EOM=-|2i~Zi!lFfRX(QzPOxs_ zAEkx23OV*@a!76sGHSG%cWGStobP>X=s8{@us**PEGc+MvL+?lT$G~yx0&MB6 z9>IQ(Ea4rrb7;cb|3r!Ozm1Gh_9e)(NDgGM?k_%z4R<48;5Ldpbq8J0y|BPN%Cf3DR*fyjagj>=C z3GTU9KjaEO7YRB}a&s-~A*r`R?_=0~js;=RLO#M7PV?zdW+E}Pmtsd_a)gFX*Z**D zKZ9dyeLi;UEn?Sb?Hwp6w*h1!vYBEYt%;XFoj*RiJ?mv=mM3yLjs5s5YX9iyMc5j$ z#8oRALa`*BlR}YF)cqF`!@HlvPPk-aAQ*ASHKL|SG@t&8Tod63YI}{kDM#-9nwW2_ zWU(FOGWN%I|F!z|?fWlO;BlS?KjGV-C+G7!C2kGp2{uHI;ugDUUA-}YzWrn8A`L34 zFmUZX-Da{ov&BP;Vd>f;zCD*cq3!aZ7akZ0+cLO}e_?Fu|CVdu1BE{UWLMyynKwnX z%SnvI5eOD3e*sWI?|s1a!brIUWT&`hOS{L#cs+aCdBTFoTSLlI} zPGD*(Rb^^{U8h2fB7xUeU3&FxCI09iGuD02HzC}03>{!>>keE@quc_Tvw0^B)ecbM zZ(sGY20MScl7JDz5?*_n=QeTNGhMaKLqh`(vy`xAT{0=&MCzgwo$8xtNiKU#uYulU z*r+t7WApbYCJiI(VrPrfC~Q)G9+8d~Hzu$LEMN9FGbi$RFE5565S}hu* zXJpz-Nh&IC^Q@1?m6BJs+GC870N2by8w~rmn7XcbiKOCSC`uLRfj-Tu}WLCgWBm;0iHR6MO<7wXnRJr zoh~ldhYzOY&u0=G@VbOk+HhMs5Ukj@jrvBzQ5PCt!?qAtC&yMR_K>N=i6tIjFJ5M+ zFfJ^l-d~a=eJ=&?AdV>?oNTRoZ|wi{=vRI(_QPLKI#XLb{fQSfER- zKWKa=Nh0FnR_+G*IwtE0Wdv@2gDgK~(q%b-yA+W_~L#8o*+r#n{TTS<%gaT8X{1aVep{jcG+3eZd}Nba#~Yli4yV zEI~!G6&=;9wE%{HpvocWR`#y#zN=7~R3`XAvO%A!ZAf3e_gc~FXkoWr>thUjt*&6{ zSTWvtGl6~Jgb~1Scd0bE;=7@^dsN^gpbG25xLl8Nl!0JZYU@64GW4Z#yKrm{MLq3O zdtT*jp1I;*`z;~E-j5=RMF&ooJWld~a8akN8C!C3 zF_<4pv}pzVqu;VJxyqW#yPOlXKX_o%Mhpe(r8hNs#28{!h(W=cPLOVI`v^FCOdtjc z*ufDLqgWI|^e{j4%~f9v?wTv~%5`4k$O{&#{d1x55>N2rP$|agTgmF_?`7@Ou2LgO zPz#oZH^Q9Q=xyF`df;odYJ!*k*F4Tc0v*OySyO&T&tL~^cqDGWh&tKB=H^gHRW949 z@*xfCj9gsvCPlueCfDsJPI!^$2Q9sRtyip_7}Twnqgm@6;RDUI$R9h^w(>4kq8U5M z%eOf7jTX!#2@njB?%U-mZLmN{0;+c;SY_sYj_l}8q=G&ra^rH9m+BaLB+nq4fq~P> z@0)A6BZnm8SCbC&k_CJAiqY-4$0#1bA<|^#TIriapT9*1ng~uYl)x;Rf>=_cNxTf! zJlOIs?N=CP8}Kg&jbUJ)>!F7Fl@8%~&$OnLK)A+X;oQ}K4EhuKH zT1dy!2KGGKL#ao+9tWDGQ0TR{H>yBFxyqZJ+AQcX@}s+r?`*Ll`t~-Vg;YiHmOpTA z{gzwT@q7Zy-YR&YUsWi>??*S!69+PpU=_x3g+B!l=t_^|?j*f3010bzL_00C$X(%h zUHj+oBh(c;E8zAW(hY8;V($?h><49(bd^z&s+cl-6AoqUAFe{kog?5WE#W^m5I)@p zCocGv;>0U|hO!BPKbggN=K8~%m&?RJfQ<$rued`{Iu8r?me2EINDzI4cX(FU?q2CT zl-{*f>tp@1cjex(Rl*2*&ZMPzZ|}kUTIJr|=xhbx0G@?)o8=myDe#n6dmiJ*K{CH6 zs%W7uKeEobU5{RwSS=N1$S`3R)EDa3+e@F>Z=}4gffU}E)Nd?Y1ft70R%}-G(I#j1 za%j1jp*K-$hiZPeBZDw$gt6q~q?%{$@M(Q-?Ow;}4t{-VV8OjiXHbEv*Mr6FjG=^s4p7KI_h?+xOerhVfPhk z_Xo9;r9`E1ob`R!gMu}wX8mXRO}5nr??$9xvd1gc=KS95bBatNUw1AT=0q!|7y~X1 zoQ>&bj`W~c@qI7#hCV<#ZVzX+t`EzfqxmL}5-t3tU!PklxrC?O-p+N`pRTY#zuwVn z#xbwbpydu^SfHCk=^$zaT~Tmr-(>0!EG(;OcaBoKz&w0kcWRSc*1Z|2DW`Rq*R4G9 z4dwE?p!(%E&AnPXT9*C-u3Qzu)ipbd_hKt;3p&EgOcvOmlb4< zFHb;ug*fj-GK_o>xyA*RF{IM&*#m_J-mt6B zxwqN04eMh-e=kk|-xCDB`wzK}?Cxx87?-y3zv(_n#OB)Q#$A2?ZS_qzx-Mwm>?%>6 zJ3-=I4GS7IN#D%+06OK~$2tn>`V0pY4p|pJ@6QcT<{3o@QZn*HJ z_lL@JmF&sl%_I@+I7|~Y?DuhZRk5^=RfwyvjAhQ^ang?1?=10(1hVt8ivfY7Nfa(t zD_$)r^|~5gfAc;*n(b80H!-&&7Eg8Y;WWMt-SeG$L#e2b9rRt#=vkq5J`(*)wU`23 z5ur8)e zZ9n<&?I8#(SRF1lo%o{cW{5Zb3FX3{BC9C8N8h{McA+m-<%^&73ETa**^&l(57lTQ zTHG{AGpR=Zs}MuiNS0al1NNOgGXqEJ;Dq(Wra=e0U%QnCmG};&(Us2iJZ^NDyBdE5 zO`a}PE&MzqtMxRUAKlJwlrQCh?azdlZO}4=Yj<3%V0w*xQ+q1rS>}QB2`%$v13#rZ z&EJb#q77gSr@vU=nhVuzkJ=ry5MCb1*=3OxIklQDwf?@9K^^(g)?(4L>0!}iA*upL z!>38bJ?vn-IcTj#h_^8ZeA{VcKu}x~os7xpCMB3cDGj(hLX$&c6l z;{)Gick%8S>UB?@KP-f|?{Jp3XU;6QN6t;tuh70x;Kd{>w92xHbiPKxywSlVsFN?* z_jonGGnCHZ2MxGHeXd+{>-Ps7L@%92vh$LsjTbn-Ck~TUd#<_}z!A%-eKFOJd%hMx z6Lu^nstQN5{K%?WwZ5Ib=Eqd7#-R>(ZMzyf-6@?$Cz{ES{WR^+qOs~MeY6&G9ZM*4 zYO#eL-B$ef%j?Wz5r-%sfg8u=m#R<`A8C35|J2};+4gTOK&@z{5(Db(;a!wff4E+I zn2=&&lSJsZVxoAlz&v7O@zmeMC9PR} zXuyf9x3u+PQymoW=$H8o8X2GN3xQI*z$=UHR8U;ZSdB00VYZ2@v(mX(s5ClrKxWv` zzse=C{D%GWCE|VM-%l;=4hCG5Pj9B!nXk{d9KY0E;%dQbSekJ(;*UC&v?-2``%+ly z>zvQXGdHnyaNL_kkG7t3arSaXXVQEKf8k}mS!xocxAnTw3D%C5&U?STJ2`O5#*c?S zZM!(T%>CoJ?$bMuUFCMEdg|nJHqR_)izmz{!2;&aj*wUGh=a=qhD4oJNw9e z*9HWR9=s2l67flvY}l>DRD56OFTK|tt}F$cwHxTDWw`rxS?OMH>cc$*S9nWnx%J24 z>w886v3X3!lqNne+~)K7PHc#~0(j{ukYn-s)U%>>JsLY45*p380;m%&xt9}$IUqzf z+w!*ZV#SdwW@SLM`fuYttMjy;m4zQjKD?BH_s^YhfZ(YXkFJL6sX?Q@^by|8Pao?% ziG)ENKQcTr$3nS@s&YVSc62H9qLyB+Sx(|@@1AOyouBg@Agwr8jlGllU_HFReUH@; z^-{ma{a6Y9=jY?{io$It+{cJ6*O*Q0xBAg41`%jY)L~9{pXhg|zV6VAYqW?NZ~W*^ zHcRBCs;g+bX|VXx7&@vzNnT7)*hp+Bb}6D?e)R-KNVA=3)#iD-NxA#1v_{PLp+>T7 z_TtvCi}jtq`@NWYh|SkeGa-s_*9u?VpNA-nVtT(AzOK-({6sJOOu2`W#mS)BX+zv@`>Z(OkyY ze>|M^D$xGMVCMcRlHtO3GV?8beh0x(jq3W5qA`-oy(Z?SkU%gS;1VW z`Q}l@vw)W?|B(VPQd@s)hOS3Q3zL2P>!aXN!r6TNAA!FGC*9r+{7mOqUNwtR#qN66 zvYIX3hgdy*ed+)OGP@5b23~Y*L-ATj1iI(JVkE-i-h~@`G6+`Zf)w}ik%2>8Yhp8a zyPp)Rrk}e5(dpAJqm0L&IzvnMMU<;MzrL#L{4+&s+&9sY7>X7sdI0|_cBZm6pwsW$ zHo^!PYF>ibLE{twWND`cwdynzL5kUW-xPUN;Pc%{6sg|JROHjSZ2GfCUR}zsR^y!O zD@_Mpvzj-^chzm)rj&g(B6MRipn^TgZ`R zrPG9@8o&gxf9CIh`yHi0Jb>huH(Z5(X|XZ@CG z?_m{2IaaQGT>gCAqt1kzoh8RUsU@eG+-#1@T#uMa?<5Bx=ZZEe|1+l zv56qE$`k?M2EXDCS2Ugh;dYbb_1<5~+t0&4PL=B9gIYA8cHxQU$~3Y<8PKx&d3t)D z;}bC6CafhpxD;+iX?KH#H5B!$=rI!pPx@a|u%}@jYdku-w_B~j%^yN$yj#n^I58@1 zkfIl5fT+Jxq%on#F+~XdP@0~}o>sdWumk7MC=(OPC>8xJ_NaGW;h56wWbfK4oh|s? z8sxY#>32#_z2~|{uavXHrT63>k-^f1o3XDMzi>eP#7;ktx@3?S90ecN68<|Y!?3_Mwc?UQQ6UlYH5_f_ ztB9uoUe`;mC*|kE=@Y7b>%*9+2pJ(MzL{V`hQ(4}>yk18p0nGOx68*2H0H~R2e9ga zmEA>%dUw6pE;%@DgCpI8aytbk2@JI(F9~tbmp`lh45{^2!Q$Q{#!%_$1t`?$2k;F7 z{yY?1KLd5ivq^vk&)Y)rcfUM}!{hT3VTlroQb&*r{!Gjijsm*|L7c~k@E^>BGYuAP z-?nr7uZNY7pXcg{h1u)9q*e@XFZX2uJQsa&jz_;6h{}c_bV7bCa}3*^E-vjcStdRW zRmM@g3l6zu)qh|_Jk1%5`CV5+9X0dVKP{=6{H)LacvabtGkdX9-S8BKa`m#^MA~F- z$-Tcp>FyAUki=cmy3FbJSI)-Ue#WC%YodjD5IL|jHZzwkD-!B|)0%f?)`?|wpFBW} z&@OZJMlr-=mf$xxGVWI@N1^(aC{P*eKQf!O+9x?6vzaTBpYj@4=uK}=51nLuFS4+m z`HJ{qs}M2Doy!aC8`VhJ*t}q~*{2zIig0dTWenmueolBChY98`#9T&fpcYvM>AF;P&A$^ZWN zv&3dO2^BI<7g$dhHNVC2<%<`{&C*G`vF>PFS~Ljd2^jJ6oGr4|ISLuj5?)+UAgF{a zs+L%Vls+c+g;a1iOwDAAZEBh%MRoKCNVS+kDl?GKCk3(u0Z^>9=X{1c&%#>5Ia#Q?q;c&)}HQR^K2FjKW^@T90ItYVf~76x>%L(E&>hJ&5%S@OJ!l zRnj%GvGD^z;%fUj!8%dbJhq1qg9Qv~!NIB5{!B{AX|ZOpnWM>~(6^5oj0(LIq5rN` z7;y8xI#zN@f!_?Km z%QtF0K%Lz@1;|k!`Vc`_?sNEkltBT=5*IXbEanRYcjM&5<*+}2olXF)T5Sl{BNz0# zIL~B&tf!}UZTxHaF_Q_Im?de54HI}77R?xagyDL{tb4H|AzraMu6{NUDkAA3p zPE3uUo}2T+;D?g1JR+MM@G3+Kv8xybc}a61w;K-luwy zl4CibC^y>u_(EES{4H?=!J4;L=MyyN*RM#zMC`)!gYQ516p2ga#|#Yx4TA0dF9lIw zCsm=BI{e$}qW{ok2*zHPI3<26iC(WX+R&Eg*;If3umWjk|H(Gze{L(aNYfMuyNTvcr*;n&8$uLkF#tg7* zl@f8Q#LC-08hYx1val`LZk0Z;!yXnWJ%aqRv+g(xRAsLZ_w_61rCd4quO3p(v0_4! zQE5g2t1LY_J`t+jUeS(hf1K7kDV&eI3L_PU5#@%oPL`JIkWMc)DnEH0TFcQ{U2elx zs%nzTobZFaPRPe_3VhW$}fuxrL?k*_M9RuRj`8+2+2(qx&JdOLEi(0m-hn` zxTySK59+CggD}ge`aG7z-U#^}Rd(d5-5@SmAEG-^kgrL@s^>Lv_xih1B}qy6OW!(8 z0_x8r{`e>YdvZQ9LNkHn6I?!~w^y!oBg4JIff3r+I!cb-_{tb1@;QEB(JceoNQOI5 zKw-L3$bODz98LV~0hC+6uHO#d;x7myi*M9TsL6pvc&y+BX{E|DD3lHqByX#>2Rfoy zFyhw%x2FqW7!G3lluI3f(#;u2+dfuR)i7H2G=rk8=jDbi=f$U|?psqXdi|w;nrOrH zQnLxwU%pJ9R-{ybp$NJ6_U86Kj-_d^-LgTsd_2c0OumnYK8%Yq#?D9h;_B+Np)Quw zj&({Z6&aYv8om-W=H}IXkEu+|MhEe#X1(utC`>+MWt9){m$Fsp zDhLt`g^SXuDR)nwy0@aOz6e-C_Z9!xE6lEvt0>}NJf5KZ(| zk3T3lV1%j?y$4)Vt(w^E+8p{EMjU3;0reo8YZT3SZx=r?@t42p`Y|C%oVxOBLG|Ar*&DP^uFHn4HYEndh8zY*L{}{hBe#TQYL8+B* zewItcj`;W)LKp>Gr-T2?2P}A0Cs-+7Y=G#B=n>$D3Inwajm|t8ZOq(Qw-~6q_!k^P zra!+f9svf+Jxo;j%`fZb?**i@r0A_K_G$54Z6DHMQ1G2wMk7GnM)9NbX|a@i=N!f( zXl9!fO(uZwW7mPSQ^)Ye@ghiG-8HSI*;Twam@1`ID+`-*2B~%3YO{MF*_{lYp>Z#O zMN4Lb(j|DX{qfjEy5$aO2JO$Z7iE7Eh{W$OsR~JQnymGYqDJ491r~UXOCHbjG@XD7ZenX)Z0!WHo#m=-`d=q8XX<2G}O}idF&;j05s*{0Qz__ zRRY=F28iQ;^vukgr!ZlZ>87{yQ?&2O^|yN3{ai}RDyBB5-j~`qfF)v-LCQXcKq+8H z4kJTdrhDqju7EfShJ~TblrU$GN{7kBHWqCsdy_vjgkx0%wes z4ubA|rlk!3{>KJiz)Seh40JxPkp#XWhI|egswd37akP<<_nA2?Yp4BNPc(T6~J%ci&c?#p)m<$yT0>3a;*HZNrhAkHII+7N8 zhIakyc4^?j3I~G!sj6!+tchWZhN^Esyt|7YW1aUoVkl`hU|<;dB{3ltI+f`&D_C`D1(k^hFx7FD7hl zdBH7VI(6^C+Zr&~9uh|C?Bo2?dK%Dr&Dx~22+(>K{f~Uc??HVPQ(%$!wUpirS%>(5 z7&7RNU}SfcHDc#WOJq0eGux8ph13T~eF5q#&gqTG@d zO*mL4G!L>~Z)uM<{0fY!BBFTbat^NYcTqBrNsd(Nlv@=ye%PsjHVPyAeHi+7SRZ3i zor!6j^LXd>-V7e{Rw(h#Q@o&eoEZZ-Bef%SE_mt`xsmi3V|owKv|nb~_qAw?)6*=HTDkUc6} zxXYd)gh~>Toq0uEk}WHQjH6-XmywbAe@^3{*S)-6p8I;9=lgj+>v^8f`||ey={>1FpdjUSzV4M#1*q_A$7IIB0+PYMZEe|E|moE9?qG8DoX# z7%igk-*!;*kp;BB`v^x6nd8q&&iKF>h|gc1W<@;uw_Os(aP<86F+}3O?WV9KdgAkw zJcxV$wjYBfF%tX#z>J9ew_O32#7umCLKt!9pLTjnSQ0B}PnqOe1hV~?-Mp!=7_B9c zus<5OWpkjCY8yz>2%=J9+3g@wtH02ISJI*qLQDdzK3cT~Qls}b_b&E>^Qw*ce=JxFzDI7%$32I_#zI(9K9 zu*c*M&RVE7 zYP7=c4v0G-fcjnMLC(QfnhhXKWEK2H0EtAEcIvy3`m?q28X%$$bfj?}2)L=ka5*nXW zNGv!{y5qscb1W$8ya$3qdx4dn1Cf)o^WI2HYOz%-lluBEQZRmcTcB$QpUbASa=$Rh za@=qQi9LpMUU(@*g1v@7E6?X z7)r=Wvl;t7oG1}MVGy-%#9j5;w{D`~SeKkrls@oUp)pvi|H7rKGIU;>6x zs|YG^tod60782A+fn_IZxvAq!Es4B!8(3dO3i{y)FdqoKSkhMWI4XqJlt8Qj9yOq= zqf_Lg0@jk>n3Ono3Meoj^y+4J0aerKi&6t@Ufd80+E|PPiQnu&=+jicd4ja}LczVO zpqv3a0L{dDF2Aw7q}~r4*C)PN%1Z!DvoT@=O8T8LWsUdh8I5a`87STdwB>`P5LZcT zZP6U=ssa?`sG<7lTideiTj>Qrp|kiMA3(&h7~TgD^3TWs#dx$Cicw#oRJzc(b3!o| zc@)!4q)94Z8_|Vg55P8ElsJt>-5O+5a`BO=d@KUS2x?nA2bfeID3c2CYEqBRvJo|a zH?U(gY%UuDz(UADwd2fVksj8>m-c1^!-Q>30aZ$$O9FqH^0wX-UZ)uWSy;1{Ihc@AGDZOWQwfQGR1^>uLlVl&SjZrh8KOjy z3&k9R_(ED?I;06)r*5Zs`_2`+^za*=#Ks(rulIZWbt8yBwCMAR;x`Z;l7!%g3WMrx z`~(Dr)G!P5D^pBNK)SQnl{Lt!xDOPP3)<>9T!qy)ZpR z33#%M(pvW>Gk7l%EF;o<)Y=-vf4!hqbY%e5M=gjGRnvQ{LZmaj7)+?Nyu7|n7k9yx z_n937D29{_<~6b+nHR)Bq}d|Fij|Ev1mx(OM&8GwNQ>(^eQGI`9;)jQ`x&}tGsi^a z`TGFh^A}N5%q;~|j8->%3+UK8rj-2*Fsun>rWCXW)~|Y!wKp}jUqzU#osH%3mC&=P zRMbW?YP_pJ+ewLQBwWEO=(F?CQQ6xY(X&N2X0m^~6R!zEG;3Nk_R=0EN|BO&phS~n z$}rD+W2^7I9gZ!@lZypB0zVJ9fGq6F`#hp_q#587p#2=`?PN6l!wLguMaX-NjlGK; z^_cevbTMv|h>dIoXwn&0r7!G!LZmvnu*G2#*HNaO znfhtWWCK>py7DmWQ1hV9Bc``UYq?eHo5Hcuz7D)A#0`3jmOU~BB}X}`GiW|dR3gKS zfbR%4k{iN+8^Ei7ZEKsY=@fyJwoPv@l6za^4)Oi%PN$CAv=5vjq5WD<18^-i7kjL#{vWKi^CwOA=i1q*x_)6 zWR{6(^-+yN4|IhL%|*3A3@FjJSdPc6Q@{mng{&UdW55m|MqPnkHu$7_%<-(OdEbMd z9WNHe1-BL*U#^=9}lqucE*v*>d4Ufea z4FF4oOT>)P<7e^(_nw$Zyhq5x#msFf1w|S>c$#^=kgSzxsH{Die7)ye0)T2k(Zc9_ z(wlxDNFe`Nofh5(F3t={`k&RML?qCio$wyy#*)2{r?e-IK=z4-GwLO`ic(K$UAcmbGfxoFvxhg(@ z;Epf01RST8foZotJHI-EHF`4hV#_vwJ*W67Yt(em4<9o|xg;ts-)5d38R2Hu zxuC)jBMq}%Rk_x~p1^r3>BJHZ`n-?6QT36`c>|gk=A*6d@D!=JFftk_9p?&p`Q#37 z82mz_#=8kGd%J7GQP8$Eb#3V&Ws@fm2$S7Kt#hUXgs9vw8+Kl3h6RvnpaGwqb!37B zMed(qV!A%|W1(oe+R1tM#?2)ObC_oIRlCcytk)bl!b!5?Bi#6EOwEbr;@2ls8?XB6 z97&_!n4^+s+p&n_XViTpUnd_|s`Ne{+S`F=!K@ajRj2-f2p#P_01aZ)v*0}?_V%lb z@iP42STIu55Jrq=n@a?zTzFrMTz%D&5K2NIzmI4De|>V@2UPN|+pFSy#8MB|*p3&A zV`Z8kJ3r2C$~2=@(U;1KxG6`AjYoWm>gwpDn4#`os$6&7K4E)RTQH535GR2Fm-Pn+ zTPp%Vb2KxX6IdEfOj=s;R#@K|2%Twf7D<4rnO9+zL^K__-!d$88=5s8cBhbvD&hHO z8v%k9hkl6k@4mqZppe*Bty^>o0M&ecxrFgs#+>wslQ4`3+05EFWRr&PY*5Rb-3(kR z!++@=W-1RSCz%eT14n+qd3>pCi2LsYv~i8f8;HusN~j5BE~P3c&6{RYrO1(}=!qX} zkpOz?eIBWBg=P2-?=1)lgzXNS;~yQE-pB*knEnE4?}799RH4T#p#==&dblXJ1$Tn3 z-t`M){m+;-J*JZypmULf_P~^l78mjU=@oj9Nw%-VgxssbCVtz*ptkfGbNP<};J$o% zzqVf1JivI`D`%s4QUUvO{nc20<#hq?H#GR(QzRY}taiaOkd+@pc()3u{(YGLB0x7T2ua`|dEm?1Uvf`T+AiMBmIolf@!c}k88g%}@POoR zT@q5~JnX{$dy(PYxG1id5VlT({{-cl>r)H9OOYG8pSs@D1rt^Eh+l8#B{6vl(Yt4r z@#V25w)voTJNp6G%=2~(Kwa41Aw5{7b8CDc5GkPyZrB{}`LFN=LE6Zh&gkM^?n&em zfUiwtHsA+@V=P6KIjo)DY1!`%H%pXv${b8AkLUaxF*VaA{b7+HIXig@@`)4G;t-Y~V?L;z)B2jR4NQ`74379<5 zXEB#2s`g#g?VE#Pz31xQJd->1t-6=a46*^Tp`BeC5_l!&J9;gOT){RWzup7<#$~JD zsQTH3$Kuu&3XLXmVx&-kF|kM!tCMqcQs?})CXrHBN@BKZKjBKd->x#W_tkmTm!>pk z{0FbTW{I@02AO)7DoB`pnai2+*7XOrI35)TNR4-m&U^8DvXP^qV%{E}(N7<4>QnYGm)|sewzh>n?dNkF(P87+TkMt<{GtNulwO znK7rb=T+X9@7Y?Zs!fM_HgiW`>jMaB$+;j(bMvqeg@8Yth+Scj?elm~;q~8CqSmw~ zz?evRan)Qp_#UP{*9LjJ{HZe|*<+xEF`n7qe~Pkkopxo?AIZ%Vv9oF2o!5D_flI*+ zsYvK5g6(cXnd_O$g13SUzT0;sUv&e8UNctCtY1QZ>6sk^SK}UslNnN28BiPkZkhRE z$d1J<&dY9Q=A2t7HLJx4(#mq7cx7`!=&Lq^zJ#phSUGHyf{Gd|te;%%L?;bPp!?}> ziF#ANG<^3rKgD3EDHYtEL(c}vs|*CqTe}l>2DCFM@;e*|5}{W53(>k^m*Y(0>c8el z1h3VY4)R}AGs`KLk(#+h>X6H7PG!C`Jm^*H^W3g)q9uLKK@i%ICWNYYhhA3cWXuMj z!C3E%E-lp%yUuVVK;cL+*chp8IwdH0Z%ta<&A8^1;q6Pfr>)8QOHBY*Tm-tl$0}9n zFmWbn;pYQ9l>U{AuweSH%l9a)(&lDMB|5M4^YNY8-?}~N+2E+>m4?-W&V>Ntz1SK) z+U_6maR~(=o;3GqRyGr#bZM6JEV&j>SVoe>@!^+M&wZCr`kDQpVmJ_#^vg})!w31u z5$qf%-FoX-xr4QoGevQ#rpzLYldK~h&9SqL zrJZLw?I)Q5FcTqyD4UIq0rX!3)S`_qbprsIo(LnAW6bS_Dc-eZxn)+Ctfo94TnAf& z-o6{!-23h_RGPc{gZi^XP}HN@bK+JGPCbwtH>J8o?X-Np9t7wibG)+mkw z%m69-GUlL+i*D%snxNVnPcvZ@NBT_DZs~K?y-G*1g3N;24Duf55JwRA(LAVs`|*qU z25`19f|a6j*jdf-%U1N``ET~M95`|e#Fv{O^GJ5)pXLL{SP-~>URxz1e?$#x&LF${ zFj)Fp=n6Eo8?o7yj{In9CsAy32ia3@BQ*+XLKIFwBE`xwm|guI^MS%mN(JIlpgI(5 z^wYduNF*QlB1cm7?f3hMODWot$DMbJ);84MitS$$OIzT_tN3N93^Tsn+ts9l3S6)l zF@72=wLGUw(7UcT(DhtCw9M;+WStAO*GtAqy9*kX)N>!#Zdz}Hl*t(rw{0>56B5`z zV>Y_FS~K+@(H&|erV>E z>xp71Z(*;zRIghP1()+a&|P>axae#Roj-x9;Gq&iJ;bRX{%R)m2+GX&I_LcwhbibT zOrCJdc-&(-PVv_#;x`n=>R6y^nH`zH3E}CZ1JPd&mN*~1C$D)JYdCqG-}kyyvi@{T z;aWI13;xmdh=!zFs5knkec+t`GDNsIb=(swQ?lsS`xUS19JnlO!AR+z=@orY{OoV6>mTd)hSKzbvIHvJspWQCV%gN$-JY%+j%>oLxiOzhepEEq z4ZbFOa*UVu+qh#v3C&ZX`eJ*NFQvt)L;v`M5g5x++2s3Kly4q=Gkts;AV=}s{*a0$ z*B4Z2(sha>=I`G<^0EPXAC!qtAK&;MCexHkr~^pZnT5i#X1U2F5;};O7_!IWY1Wy5 z{u}3VBkKd#4SUPi{I-_Yle&TE!3Q|=){@NrA?-)4+$)+l7d|}4ZI4(BoO0cK<$Zjm z_mujR3yycf4~i~5?vf8ukac17CPV^7g`c})9s0cPG<4s5qQ3s%q+e-z=mv0kjx1P> z*0*nCNNcAD(8eE(KwqcaKdkNxYJVXq)jKahK)moZ?EAacNh0W+={(X5_lc;T6ok0l zgp39H=Yv6hE2E#A)-yFjn1=6ll7&AE&UoY3?$|tseW)Vs=&O&IuX4nnXR|DmvY`Ty z75vgf9qEU!N!%mT0?Xq4RYF9^lBm?d9(Gw{L*kD=JVOPe;mm#nuwj=+?Rc{Y##)1; zK5wGBB9}hC7+Km+n> z6QR`Hb~SEA(ZwZdwA#EyQ9ju46}UC|&2~|kK+h)^0syQ_@PbAwFw~zdQGo&RunKLu z?!2xYkpDnXh=t@D9F#ARB8)#5ki9AFJ)|$hhB;f|KjlJcF)_Q-#))ZX%8I<&GyZ@3 zE`$=Ord>4qv(>;m0!N|OoQ$?L-9RmIoWQfw8jF9mm2s=~`)RVc5l&CY*}Z<11d!BZ zWq%8Qy`5z=&szP_*|#m}kNP9gyM8l622%B~oyLPDJG|nkk|u5{^fYa6P9mwLWt)+8 zy<|CIDrC{%?`V5nD@aQ-q<>v@7M?u8%HKI4jq7`%ha+L)&@1UXb5f z!3r6M?3-I|P|L=IU@Sb%5=$~*K<)0c2Q{Wo9;s81Q8X{F0_A_2cW4XEaKKYODbl;}w zd=}yE;|`Vv4L;B1-4Z$MDUGR2E{HgfzN5NxUQfb?#3DL{Le!N*B*PO9Fw3$lYIJ^P zexj0@$CNgcxKsXk;~siAvb#0o;701#z9!}d43spoLa${j7EM=P2`GGdOBw9^GEQS` zoFhZl_4P`}BP$L$$l_uB(#Ni!XF%}MWjv>cR9a)0x_&{@lD1u-?$ z@Vke|d5H@QD;xf`Z2TvN;s{;rl{ARGsp6pRO5c!IK?F+`VT96GE(gyK7%|MMj^!&| zT$Gd4`}jXzpgJ=M5mIc=xg{Ufyen-i%41fnjo$JE`v`~7!!tSJqOxrnquCK|8%j&A z|JQ(ec&-7Q91qCPbMc3=*)s$*YJPEue5`rz_9{lMHV-L?DrvJY|Ep?aAdvON4;GC8 z;2ZO5(+WzN2BAxy3=jY{_q@5$J!Rj5C`Y>}7H<9-0LoC5j0p?^g-n*Ky1hRpa1=>c z95ArU!^v5tTPrv?a4^}XAAjRu{%BTk{LiO6_8==Wru(UZcF0q6GqL;luJkFTs@)qm zDrF_Q*|{i0C`Z+upMjf)KjLWW9?G`JX3^mI*LQxw>L1#qVaAF>LlQu=3C*dGn8qt_4C^(cb!BdNMEWja zT!y@7KHp<_l#IEzwfsg4{{fLec@akzo;Ut`OHSOC>r!T3Vmz%RMtSHwOjR3o>|9s6 zIjplY-FAMP%F+^ln-rhG(ed#mOLTShDjbuANn!1IO%#Er{}%yzXfs9T8P1cf$vu~T zs2twqyof)Z!=QfIc=44(9`O>R7vin^?PKZ=ua8Q;+78&J=VJv^lU*h-I zNzlqQhF$Z3+*5FmfhsHGX7cYQj~3FUzB^5qQ*MLZ-qz)$u#Rk0MQ^yygwq+;_=r^j zJj!Zp?6!dw;tM2zM3y!(NEY-dT)O%HKJi|#FE5@y-*&qpFWuFFckvkKYtUnB^y)G@J{Yt5czTG+_&nt( zAJ+85UxDPON5QTg!@Ps2oOH&NKaJf~R(8%vy{#!_oi{@lrjA+~DH_d>{d^VRUe$G} zlGEU_d&uQ+brfnfW^Em-84Zal?0QEge**g|7}~Tp{fZOVtlh&cwNG9XQ6G9Py}x{^ z_3|71PQtNLvPxF`SBw$+M`f}RAcE*dQ|mtpP#w;H3GYss%Yymx{JC>RTEvK)oKjN} z1J literal 0 HcmV?d00001 diff --git a/doc/design/images/l1_regularization.png b/doc/design/images/l1_regularization.png new file mode 100644 index 0000000000000000000000000000000000000000..e1b9c7a44f94dc027598a98da93ddb8133190972 GIT binary patch literal 1157 zcmeAS@N?(olHy`uVBq!ia0vp^JAqh(1xPSB@w`k2QY`6?zK#qG8~eHcB(ehe3dtTp zz6=aiY77hwEes65fIJ>L8=O=iFto34NS4sQC8zgol zZL&q_W0k+f-{!`DR$|?;@O;v7%l{?+?iUxb6zBy1C_lK~h<&gAiZ4q}I=9=Ww|C$B z(z)}O{!#0mxZ`X*r!O43Xw?D_hwvgcZnEQm{_3^8 zhsuuzp3b>BJ4`C;HZe`x{=39}X4vA`!`&H1&iM^4_8hGSGw)x`^IczF^Oskzq-f@g zvX4(9mQ*Q~2Ujlb{TyPy!%Wck!lzlcWq&W*$CWr88)?-|J0a9T$QQfnrF0kyVKW-cQ5QFZ_(%SKIZd`&t;Njbd;IBmqF;N^uXzJ zl-^Cy=c;i3Z8URtX|(-agGE(3<;B)l-u7Ht{UT%I&ilJ&`-ZEWc+&S};)`WBW_TR) zI(SBL)$uItOYgT`*yzDm?el+DPk3pdYNXnwWhT$b)S{?BioJpb$ZHDR?|+q6ek`!1gBe<%65)YoH=@UI-- z(}A&JkAB5Z?{VC?~8j6=l)#A9zPQvH14cg+9uCt|)UO_QmvAUQh^kM zk%6JPuAzahp?Qd*k(H5!m4S(_fvJ^&!QvgCPorqa%}>cptHiD0(o^qppavz74F$zk z9+^R@#ZLL9c`2EB=}!3-42H(W6-JiYMnG(4XjEdnGaRVe9IDzUwJbGsk#1QCO1w%bkL$378RboIRT%kq;7vz^X z=jY@X=^8NL)`>+XxH2~>KL==Ri0?Tkpn--M1{$In7?he`nv+snEgnpd2ep9j=@ UKlKVQmoqSUy85}Sb4q9e06^~6>i_@% literal 0 HcmV?d00001 diff --git a/doc/design/images/l2_regularization.png b/doc/design/images/l2_regularization.png new file mode 100644 index 0000000000000000000000000000000000000000..d5c2fcbc2ccae75ad083162e5a2dceb0210be298 GIT binary patch literal 989 zcmeAS@N?(olHy`uVBq!ia0vp^RY0uG0wfs11@y9k6id3JuOkD)#(wTUiL5}rLb6AY zFHoHt14Ba#1H&(%P{RubhEf9thF1v;3|2E37{m+a>rPdj=S$Y3w=^mS!_$R)@l&6x7ZV+&9dzo(01h{y4_Q|)=zksHMv-tGIFs>q;;7T_puq7aBJ%baLVf@sTyCX+9wCvCnbMwna{IL7}haC>*@R z8anY-o_M3Qxy84`Kl!Kjo;#^2_b~F@zTNZhecxLxFY72WF;TkewF=W((RnTl8Xwu# zwr2##2Ay|bR_fGoW~ROJX{VW=zc%MxzL>e-@`sD{cmDj6zOy#7db7GRb1QF`{Zx&D0x-jV@gGQj>Ev*9!uWR;bt}%Mnds%8~ z6~|h2jj~zKwA;5i8~=)*#;aYysGjatpL+ayc?~3k0Kkqi6n)eKYMz*j4Ld^>5V?yx)#z*shgRkDV32eO~6=Q`sLb>o4@a zu{~3_`Q8)VU5`~h&cAoDOr5EOYr4_BsxMEIr$4*a8~1DBH$CYE#{(y>yL5v2uI8Ja z-^GHq8>+puv~~NMrg7!tjSC-bTld_s@rF_UrLQ*vB0rntcsAtOA9}H6YPQ<~_7<^<Y-?eQ=H?$UF9LeQ+{b) zN@iZVQ+@@5g++z2p|+8!p|+W!;j^rZvw@1up^AM{%TjX~98>a>Qr+_NN^}kN46GC! zi&D#i6Z497{gZMs3rkZKf>Lu*6N^(74D^f)6Sy)39qU#@G2=t}`7MbA6+@$;*pk#>eIVYfj44$rjF6*2UngC+| BgDC(2 literal 0 HcmV?d00001 diff --git a/doc/design/images/loss_equation.png b/doc/design/images/loss_equation.png new file mode 100644 index 0000000000000000000000000000000000000000..14212ec8d36c803de96bde8a9a4b5591bd20434e GIT binary patch literal 1589 zcmZ{kdpOg39LIl!94k~pQPPD|*xGMx%tAIcTQV$N6tOb1usdz2(5VqBbxJadP9fuw zOV1hV6qY%pJj*SYM`Q~*b2&s8&hMP3f6jTH^T+r5{(QdQ_xttv{`F1uqk0*_EMNct z7*fcdGyquU1J&Dfp$(WE8k0bx%f$QO0pQVArCmq&6PdR;At51 zN@SCI_vKmi;`qFQAg=V0GjrHxDXzaUD>fdTYEt2IP6o0@u~xV;s4sfH{y%$VK=?V zsaQ|xD0ZLrSmCQj6qwna*UflTbE3Th?ST(pM+^QrsR%b%U;$X4s3-jvVZ z+1yqBTt;DG1;}PrZL+rqEWnPXlc_q5T*Is-xJtC7z%O>0I-$FL8hdX0X(=W6+-=ka zx@fsVeo?x9$rY@)G>J=7DH;&mMQK zYmR793m(!B9=55un>i$WUvT@nOQQ2s_nJ0a_j&5?$VoE(_K<8kr82l=uV2;q#}=t) zB?iiiGw!8`tu{}#MkZ{z(JTuPjj3uv zci(UG`OwPTWqK0UX`yJgNEN6w`7+d1*l6FX*)6#_AHX09_rl6MRpLq7V}HA1&rQiz zlo&G6`O0PGA!^Ve^xBw|R|9TpF0yk=et02H7c43pwmhE^d6E~HI^-EFvAtJg)Uw4q zK2}*&koj;t)Y{P2Te&G^9M_jqG+)gNKDHxAF3e4j9KmJB`^u`*(E+(Gd54B^ffgOj zAKQi6!9ef+($N>Tl)gO{Fjmd!$)htxUTY#9Vzq;$Pe!rJYnuA|Z@YOBhO7=(9-DDx zKYfw>341ExO3TGRMUL@BH)v?yuCfu?Nc>b#22rc%u75IbRi)py)Vg5&F-|d9#d>}N z`iX3W#3RBmx{!f8$!9ms$a6|y;h!lZN_9zfz?}$Rz1-B
+ +The parameter `alpha` is a hyperparameter that weights the relative contribution of the norm penalty term, `omega`, relative to the standard objective function `J`. + +The most commonly used norm penalties are the L2 norm penalty and the L1 norm penalty. These are given as follows: + +##### L2 Regularization: +
+ +##### L1 Regularization +
+ +A much more detailed mathematical background of reguilarization can be found [here](http://www.deeplearningbook.org/contents/regularization.html). + + +## How to do Regularization in PaddlePaddle + +On surveying existing frameworks like Tensorflow, PyTorch, Caffe, etc, it can be seen that there are 2 common approaches of doing regularization: + +1. Making regularization a part of the optimizer using an attribute like `weight_decay` that is used to control the scale of the L2 Penalty. This approach is used in PyTorch as follows: + ```python + opt = torch.optim.SGD(params, lr=0.2, weight_decay=0.2) + ``` + At every optimization step, this code will add the gradient of the L2 Norm of the params to the gradient of the params with respect to the loss function. This can seen in the following code snippet: + ```python + if weight_decay != 0: + d_p.add_(weight_decay, p.data) + ``` + This is a very restyrictive way of doing regularization and does not give the users enough flexibility. + + **Advantages**: + - It is easy to implement for us. + - Faster execution of backward. However, it can be done manually by advanced users too. + + **Disadvantages**: + - Not flexible for other regularizations such as L1/L0 regularization. + - Does not allow for different regularization coefficient for different parameters. For example, in most models, ony the weight matrices are regularized and the bias vectors are unregularized. + - Tightly coupled optimizer and regularization implementation. + + +2. Adding regularization ops to the graph through Python API. This approach is used by Tensorflow and Caffe. Using this approach, we manually add regularization ops to the graph and then add the regularization loss to the final loss function before sending them to the optimizer. + + **Advantages**: + - Allows for greater flexibility to the users of Paddle. Using this approach, the users can put different regularization to different parameters and also choose parameters that are not a part of regularization. + - Makes it easy for the users to customize and extend the framework. + + **Disadvantages**: + - Implementation requires comprehensive design and time. + +## Proposal for Regularization in PaddlePaddle + +### Low-Level implementation + +In the new design, we propose to create new operations for regularization. For now, we can add 2 ops thgat correspond to the most frequently used regularizations: +- L2_regularization_op +- L1_regularization_op + +These ops can be like any other ops with their own CPU/GPU implementations either using Eigen or separate Cpu and GPU kernels. As the initial implementation, we can implement their kernels using Eigen following the abstraction pattern implemented for [Activation Ops](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/operators/accuracy_op.h). This abstraction pattern can make it very easy to implement new regularization schemes. other than L1 and L2 norm penalties. + +The idea of building ops for regularization is in sync with the refactored Paddle philosophy of using operators to represent any computation unit. The way these ops will be added to the computation graph, will be decided by the [layer functions](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/python_api.md#layer-function) in Python API. + +### Computation Graph + +Below is an example of a really simple feed forward neural network. + +
+ +The Python API will modify this computation graph to add regularization operators. The modified computation graph will look as follows: + +
+    +### Python API implementation for Regularization + +Using the low level ops, `L2_regularization_op` and `L1_regularization_op`, any user can add regularization to their computation graphs. However, this will require a lot of lines of code and we should design Python APIs that support regularization. An example of such an API can be seen in [Keras](https://keras.io/regularizers/). As per the PaddlePaddle [Python API design](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/python_api.md), the layer functions are responsible for creating operators, operator parameters and variables. Since regularization is a property of parameters, it makes sense to create these in the layer functions. + +#### Creation of Regularization ops +There are two possibilities for creating the regularization ops: +1. We create these ops immediately while building the computation graph. +2. We add these ops in a lazy manner, just before the backward, similar to the way the optimization ops are added. + +The proposal is to add these ops in a lazy manner just before the backward pass. + +#### Storage of Regularization attributes + +Since we want to create the regularization ops in a lazy manner, the regularization attributes (type of regularization and weight of regularization penalty) can be stored as attributes of the [`Parameter`](https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/v2/framework/framework.py#L421) class. This is because regularization is a property of the parameters and storing regularization properties with Parameters also allows for shared parameters. + +#### High-level API + +In PaddlePaddle Python API, users will primarily rely on [layer functions](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/python_api.md#layer-function) to create neural network layers. Hence, we lso need to provide regularization functionality in layer functions. The design of these APIs can be postponed for later right now. A good reference for these APIs can be found in [Keras](https://keras.io/regularizers/) and also by looking at Tensorflow in [`tf.contrib.layers`](https://www.tensorflow.org/api_guides/python/contrib.layers). + + + + + + From af215a1a532137686a696cb1c5da5a8797ac51ca Mon Sep 17 00:00:00 2001 From: fengjiayi Date: Wed, 18 Oct 2017 14:17:39 -0700 Subject: [PATCH 22/76] Design doc: Batch Normalization Operator (#3748) * Add design doc of batch_norm_op * Move batch_norm_op.png to operator/images * Refine batch_norm_op design doc --- paddle/operators/batch_norm_op.md | 134 ++++++++++++++++++ paddle/operators/images/batch_norm_fork.dot | 25 ++++ paddle/operators/images/batch_norm_fork.png | Bin 0 -> 23873 bytes .../operators/images/batch_norm_op_kernel.png | Bin 0 -> 165209 bytes 4 files changed, 159 insertions(+) create mode 100644 paddle/operators/batch_norm_op.md create mode 100644 paddle/operators/images/batch_norm_fork.dot create mode 100644 paddle/operators/images/batch_norm_fork.png create mode 100644 paddle/operators/images/batch_norm_op_kernel.png diff --git a/paddle/operators/batch_norm_op.md b/paddle/operators/batch_norm_op.md new file mode 100644 index 0000000000..80948adf2b --- /dev/null +++ b/paddle/operators/batch_norm_op.md @@ -0,0 +1,134 @@ +# Batch Normalization + +## What is batch normalization + +Batch normalization is a frequently-used method in deep network training. It adjusts the mean and variance of a layer's output, and make the data distribution easier for next layer's training. + +The principle of batch normalization can be summarized into a simple function: + +``` +y = (x - E[x]) / STD[x]) * scale + bias +``` + +`x` is a batch of output data of a certain layer. `E[x]` and `STD[x]` is the mean and standard deviation of `x`, respectively。 `scale` and `bias` are two trainable parameters. The training of batch normalization layer equals to the learning of best values of `scale` and `bias`. + +In our design, we use a single operator(`batch_norm_op`) to implement the whole batch normalization in C++, and wrap it as a layer in Python. + +## Differences with normal operators + +`batch_norm_op` is a single operator. However, there are a few differences between `BatchNormOp` and normal operators, which we shall take into consideration in our design. + +1. `batch_norm_op` shall behave differently in training and inferencing. For example, during inferencing, there is no batch data and it's impossible to compute `E[x]` and `STD[x]`, so we have to use an `estimated_mean` and an `estimated_variance` instead of them. These require our framework to be able to inform operators current running type (training/inferencing), then operators can switch their behaviors. + +2. `batch_norm_op` shall have the ability to maintain `estimated_mean` and `estimated_variance` across mini-batch. In each mini-batch, `estimated_mean` is iterated by the following equations: + +``` +if batch_id == 0 + estimated_mean = E[x] +else + estimated_mean = estimated_mean * momentum + (1.0 - momentum_) * E[x] +``` + +The iterating of `estimated_variance` is similar. `momentum` is an attribute, which controls estimated_mean updating speed. + +## Implementation + +Batch normalization is designed as a single operator is C++, and then wrapped as a layer in Python. + +### C++ + +As most C++ operators do, `batch_norm_op` is defined by inputs, outputs, attributes and compute kernels. + +#### Inputs + +- `x`: The inputs data, which is generated by the previous layer. +- `estimated_mean`: The estimated mean of all previous data batches. It is updated in each forward propagation and will be used in inferencing to take the role of `E[x]`. +- `estimated_var`: The estimated standard deviation of all previous data batches. It is updated in each forward propagation and will be used in inferencing to take the role of `STD[x]`. +- `scale`: trainable parameter 'scale' +- `bias`: trainable parameter 'bias' + +#### Outputs + +- `y`: The output data. +- `batch_mean`: The mean value of batch data. +- `batch_var`: The standard deviation value of batch data. +- `saved_mean`: Updated `estimated_mean` with current batch data. It's supposed to share the memory with input `estimated_mean`. +- `saved_var`: Updated `estimated_var` with current batch data. It's supposed to share the memory with input `estimated_var`. + +#### Attributes + +- `is_infer`: *bool*. If true, run `batch_norm_op` in inferencing mode. +- `use_global_est`: *bool*. If true, use `saved_mean` and `saved_var` instead of `E[x]` and `STD[x]` in trainning. +- `epsilon`: *float*. The epsilon value to avoid division by zero. +- `momentum`: *float*. Factor used in `estimated_mean` and `estimated_var` updating. The usage is shown above. + +#### Kernels + +The following graph showes the training computational process of `batch_norm_op`: + + + +cudnn provides APIs to finish the whole series of computation, we can use them in our GPU kernel. + +### Python + +`batch_norm_op` is warpped as a layer in Python: + +```python +def batch_norm_layer(net, + input, + output, + scale, + bias, + use_global_est = False, + epsilon = 1e-6, + momentum = 0.99): + mean_cache = scope.new_var(name = 'estimated_mean', trainable = False) + var_cache = scop.new_var(name = 'estimated_var', trainable = False) + batch_mean = scope.new_var(name = 'batch_mean') + batch_var = scope.new_var(name = 'batch_var') + batch_norm_op = Operator('batch_norm_op', + x = input, + estimated_mean = mean_cache, + estimated_mean = var_cache, + scale = scale, + bias = bias, + y = output, + batch_mean = batch_mean, + batch_var = batch_var, + saved_mean = mean_cache, + saved_var = var_cache, + is_infer = False, + use_global_est = use_global_est, + epsilon = epsilon, + momentum = momentum) + net.append_op(batch_norm_op) + return output +``` + +Because Python API has not been finally decided, the code above can be regarded as pseudo code. There are a few key points we shall note: + +1. `estimated_mean` and `estimated_var` are assigned the same variables with `saved_mean` and `saved_var` respectively. So they share same the memories. The output mean and variance values(`saved_mean` and `saved_var`) of a certain batch will be the inputs(`estimated_mean` and `estimated_var`) of the next batch. + +2. `is_infer` decided whether `batch_norm_op` will run in training mode or inferencing mode. However, a network may contains both training and inferencing parts. And user may switch `batch_norm_op`'s running mode in Python `for` loop like this: + +```python +for pass_id in range(PASS_NUM): + # ... + net.train() # run training model + if pass_id % 100 == 0: + net.infer(test_image) # run inferencing model + # ... +``` + +`is_infer` is an attribute. Once an operator is created, its attributes can not be changed. It suggests us that we shall maintain two `batch_norm_op` in the model, one's `is_infer` is `True`(we call it `infer_batch_norm_op`) and the other one's is `False`(we call it `train_batch_norm_op`). They share all parameters and variables, but be placed in two different branches. That is to say, if a network contains a `batch_norm_op`, it will fork into two branches, one go through `train_batch_norm_op` and the other one go through `infer_batch_norm_op`: + +

+ +
+ +Just like what is shown in the above graph, the net forks before `batch_norm_op` and will never merge again. All the operators after `batch_norm_op` will duplicate. + +When the net runs in training mode, the end of the left branch will be set as the running target, so the dependency tracking process will ignore right branch automatically. When the net runs in inferencing mode, the process is reversed. + +How to set a target is related to Python API design, so I will leave it here waiting for more discussions. diff --git a/paddle/operators/images/batch_norm_fork.dot b/paddle/operators/images/batch_norm_fork.dot new file mode 100644 index 0000000000..4bc47713cb --- /dev/null +++ b/paddle/operators/images/batch_norm_fork.dot @@ -0,0 +1,25 @@ +digraph ImageBatchNormForkGragh { + subgraph cluster_before { + Prev [label="...", shape=plaintext]; + Rnn [label="rnn_op", shape=box]; + BatchNorm [label="batch_norm_op", shape=box]; + Fc [label="fc_op", shape=box]; + After [label="...", shape=plaintext]; + Prev -> Rnn -> BatchNorm -> Fc -> After; + label="original"; + } + + subgraph cluster_after { + Prev2 [label="...", shape=plaintext]; + Rnn2 [label="rnn_op", shape=box]; + BatchNorm2_1 [label="train_batch_norm_op", shape=box]; + BatchNorm2_2 [label="infer_batch_norm_op", shape=box]; + Fc2_1 [label="fc_op", shape=box]; + Fc2_2 [label="fc_op", shape=box]; + After2_1 [label="...", shape=plaintext]; + After2_2 [label="...", shape=plaintext]; + Prev2 -> Rnn2 -> BatchNorm2_1 -> Fc2_1 -> After2_1; + Rnn2 -> BatchNorm2_2 ->Fc2_2 ->After2_2 + label="forked"; + } +} diff --git a/paddle/operators/images/batch_norm_fork.png b/paddle/operators/images/batch_norm_fork.png new file mode 100644 index 0000000000000000000000000000000000000000..aded62bce5bc268b7a3ef4dc96c89fe21d6ea955 GIT binary patch literal 23873 zcmeFZby!wg^fd|yNQo#2ilj;-2olmDBGN4&B@I&2(xE7d0@4lANOy;zBHj5CDjhE< zyma09o&)Fpp8MRt@Ao~={r)%}MfYB7uf5isV~#QAeygk~eSv_I00RT#g3O}_su&np zrtsf6JY4vR)D6q0@IOpPRq6W}1>ICj7#QLhG7luwUtl69&ySJ~js5u>KP7SL3eMLJ zRs!-H%yb0)5=7W?$?#*H2Q2PJg2O z7Yp0;N+UkT+4Y7bbj*a3x0P{Eug`vg>0k108XKPZpE>y5!6S(X_(aZp?%(n7)ax)L zw7=P5VVq6@hQMoYRwj(VJe@PEzHdt2Z+=XS)5oAAf*}pFaoW#@yuvB2{P=G?2FB@} zNO1jX^a+Zee!Ru!8vozH|FHg~&%3PT+Ftf^FNv(a;MV5N zP>vJ|oPb*wrs{ORkd?4!N}-=Kn;o-WMq|;9MK6%gbm@kt^nWja4(5vMME>@4x$)8o zNg^_W{!pJjED7^VSdMGG+P6-Zhc1*uTsexP+4bywq@W+)RNVaIfA6FQuBZuYT_4f+ z$e(Z;P{3IzahQ`8_#Jq2HZ4Sy^~IF+UC-TLZ#D{jj^w_F&?eh(f2a2qe}&UO{2)P#Mx-VsH)5Ti~zP{-zUWSA6 zFHp-;bol#?qGQr^`u2!{k5?ZXYB4ROJtr%P_2h6tEU5eDzP z@YKtE7rKqvo)o^0G^l$1X|O==MczSlBfoKZZN}9fzQ1bdgZ2=qgvZhZsqr$uH;s{J_2m0mVe73_*d0Cz ziJ0w1qEgFa2sInqB^&Tw$uu>JaLQY0Kh;L5{3hhAqZ2#cO8}m%VEO4wJlG(bTx8mDJz-7o$lapvq#GAWM$^nERWrku-pDpmXr^v)q4F-RDoV? zyIH(VRLpGUw0HPIl7}K{ZCKv}Y5Ha*q+SfwE9vn2EmQl(TnBRK4ivx()o%XVwbr!N3f`we1Cr}wKYV?w)$JaY>LE7 zla>6+isT@Xs0^zmE@TjAL($91Q6s`@cbfg4x zDE!CQ^Us&_+f`Yu;{3kyCe7!uNp=f;$0Xh<&bwk$A5mnPjH}+8ugp!@neyo2I?rRi zGwJHOJ?-PuN)@{LY|3pRQE}$`2WnDZ0se{SFH2i#JQgq3AS@N)@3ac@J+GK>=peg1 zz}K=jLwm9qgIgAipOmn_wIVc+F^^iGo_3py)TDCEScyUG^U)44Pj_1pmvzs_+u6R+ zbDebV{`H2|cC*)1&wFp~?P24=noukD)9$-H!ZX&RMPI6C!{idSkq9Bi6nXhH=fMga z#MhlFPiB(?HU3RRw7s_T@HxzdXajG1^4m{lV;Bj7@f}vpk8CNAbP|n1ObZ*2YmD{D zi&|---Hs3U6f12fsT}Q>@(dcMh==FmDFUe+{g&&t$E~)nnyvMZ&BO{NyJ0WK1a*G( zWPI*K$z$f9U`Rtg&x**B1t z(-l9`q(c;m?z(F#ZNl(n&?{C$k%$excd%NDyyM6a7t=ja?O5Zq6t?T#BeHXEUd6Z6tcy}e{$g;AN{kER zTT8t5&I|zwZ6lALl6BZTyZy%E4jb<5a>VT@{~qBhMG{XpxCKa>^5NyKW=B zGb!S2`^DEf@Afz~jvG&HBkYuvqey;9CSc26inua$V=`g#cO2?NMxyJK_LDM;k6KcW zQcMV&%EXwT(x`Grg|#%lFE7s{)&)c2J2g@c`i_rwXF{Y5xRlnGjgq~#FOlxP74?3u zw(luer;_&2#mV8ATF8YAIp3@>9_lkzVrsP3JvO&^Q(1xSao%&LUz}KY>M`Oh!KSjA zT|8xm$0x_@C-16Tl2fxMPwMuUa~%%?bWf3k;dbp;C(oC=^i_4QuXv1Hs8zgZ2P$_#9hANPy|-rjJ62Lw2^;=N2lzD zl_4CZ=hP=vk+^(O?C>d~wkFP9Wv`7XdGz>V`2`(?HeuZIm#_;)BKojRSg1TDU_L@H zpHJrrY_gw?oBa6x;ksI=%Y?o3fnF;`9FKv`tz6gkP8&~zSWoSGgVps#@;FU}0jr|j z+UcY22I1#x)eG+|l^ze65J_VA5nN)PEAA8yE~ZGArgm0;m*gQ4mv11o)8;>QTTThv z44tcfHbRJ((ebxDq09VLSI>~lGukkeZATSbLDKee^VnLShy-b=<1xg~d0~_$Kd6h% z>ytZ!73I`Bw~tN1zlpVLNTx4k3;U^3cXS5HZ}(-mY`b^yj@+VCK>Sc%&$x+^?yXXfj2yQrnbwuE zwZkV-N@7RqG#;y5LWKvUO8XWPv|JYb8;eeB!nXQ{QW$=Vgch|LB8rv^1Gcc`2iAOf zmh4x5+zg2*Hf|kHez@P`b3DV*JheAstUcY}WfLtt-q6W&SlN0VS+nr-<;4mcr>z(J z5XljHsH$|NzOyKDc34D_k$dhORwTz$#X+g8s(P;+l1v|NbJ+gQiUMsbq(Pt^<_#4##>z>g{`F@Z%Pa9Z|^l6tgS4@&(Cw{7(EfC zBXRaqt@F~ByVOanrp|rjHnvwbDUlEsR=3$LwA~#wztmAAV5IHp&hB`6k#@ha_%V3!^4Ke~HllS649{HN)Dg_uoCDk2Ae z_D0=zGHSnOY*{zBTmt2f^Ftz>tFXE|_(lO^gR2|3MpZ{2~*WV;x+uBRliTcgC zUn)nryCf0*>I@e4_D(O@Je}{1P1SowL@TN*kGMEy8m%`w<*|MRiD*R~uW$CZ(VT+D z)EiBS+{sq5LW&7ImSKAD8l>8crINp#{=Huy!SgKkrgt~#2ub3l=>I{@BwPS?0yVCF z`v-#pD>g40ReU=&v-FD6K; zBbd{r+&k}rPKW$U21ATP%fmzWj=^SzmeSVIGC@ z*sJJ66QN9(JVVl&dD%bw3f47j3$-ZB*I)iC+CpJ#N<1rNKP&6#&~+c~Rot6rRjmY^ zNSvImlGDA6;fJpKu06f_{&X+nVWW3@I$y`>alypqhRWL1*YM8iaiJqghg*h@vlE^k z50W2HeEx4T3C6%LKu?ez0VRTo-F)jg;GMR2SJ;}>Z?*f2OW_UX2bL2p$xbSu@%BVTv`@1Pj}B5yFROHTGu+3!m9{|BIy& z3P2!Oqu8&WtwZ)@2*V{mNzWEanGZr!Nnh#HvzL(~3sz3BMW*+P&GVrq3F=CS7s7EJ0e8wuwg#&wn*X{L0s< zjDm76i)e-r9WmZC{r(nYdbGbaRH9#O6p&6EFX*z$3>W3{-9ASmp27p?EW^Gtwgc)V z)4^;NUd54mkDZ8LYMJu%%yO50!A*7 z0Gs5Ggl}uKXv%QJ+|MsBGbCQ(@_vxLmN~Fkb_;eG9iAVcpm|;SL_uehpG|)K5YZ9_ zxG}Nt$k0Sqot}gvpJo47Tpi~Xvj%~j86LU;OBvXS;v`rvaW90uzHr@4ZxD(A)6N(! z?JPqeie3O`w`m*WN(_Z!)P?JOCJFkjOFaLK#q9U+17%w%z2q$fPMSV5@zpzY**A)* zuvs5NE^Yd}KHb>JV>8Y(bJ;mTQ4)O?{mFi~e98VY-up>%^O^T0lUQwXEPF_sghh|b zKdt`x)sg`)bSUa`i_?D>ERGZ=ms#|sewSLNJ_IB_bnouDOX4YXa76s8>Xh=%SH|g! zjTpvGb)y>89nd@=H{s%B(WRRY2VZ;{1%SDPsI;4pz4j6VivfL%u9Z%U`nOaudPLptAsspB1_KA#t* z&YXc-%U4_g+kJbLW!&D`p`Y%9badMd&xcJ0n5h^(URIrYw4}ETJ}#S6KSV!vJA-oW zWM4$LMvPTV12ZeTlzhE=B z*W$yv!UQialXc`}A`R-^pU506Ep`Z2q>B^PrK?3P#4h(jICB5q!N8n`6ywQM;D))T z6GOH8Y}=3-(u8zVz0mL{$@5E)soV&@s9wre_UmliBfT36ESVmfC{fPWfGJqJMX_t& z*ym)Ix%eFlJrAvSk^&@w%d}SF&SLtT$%c9FXNBuS;m&T)0mD*&)$o+|mJRvsa(x$#DIu2&g<(i%ZUXBt_BrJ(zxdHxIt9G?-v*{qse zG5xux-)f&-Ko_b52dMIr+JCcZfGhSCJF(6N^_ze=QvD$Y>n!X1(Sxn(YD9jP;PGb3 z>6klgzLK5I%6A>EXdo@cI2-i;Wht^(4lduQ^L5*tZv!5RwRW{6XabUC$;t6SN0P@H zOONp8d-PFoM=jEa2yX<|9c?4HJ$IjvLEgM%x&@0o7u0VN)dA%(H*Fz0n>P! zijP3AjJc7L&hHrlLfQa^=SPUxu|>yoU@v8XXT9v`1N9xZ@pmGD?NQ@7;5EaUl3v^g zCjBNrt#qDegUnpifR5haxS{d$2eBX#i_s0j;h}W6` z4r|GIVUf`UG+lhOHPQieHA|1!Nga>Xu>4x%iT8C~xQh(mw5Q|25%7XhrX5i-03mHs zGQaeWms@?ar}F3>?D08TF_3zh1tn|N>kKm*r*X@yHaJV{Xp+bixHX4S!#8b_n))ng zzSFv5%#7Q#o$UJI5!Cu6PD`rS9~EeqephsaXXyailzD4;sG|c!NcS>V2++rpXTT7#+Q zzF)WdhkG!d1Y*eg+_#q(MYZ^BCzz3l%IG(=-dY9K^OQQlVkdh&dmO<_B-uomq&#Ga zuZgLb7aN&xk&fP*Z)aamEzl}%=F;&sUVASv|Ey;ZCQVJMB_MW$3a6gRQld479LOi; z_@=`MnNvH%esqcY@}6i@mcy~l}7>AK2M^7>;KUFOyxTP=^fXogxtag!S!-J zR|L}AX6-A&ty7QnfIrz-v7xn4H((D4_yk}G$Ett&6RI*&3pk{!HF|q3eBhrdYMJ)h z(bE~8-F#L3)T#4Z-T<1a<>Fxor))qg8hpt(gye&Fq9pKMNs9MyxflEm1C&j_J>|ZV zkBsp?(Ef9ngzQ=6`**Td<3`^yu8&n5p7A|1bqYXj+++6A+U#;G%T;mc0dnKIqtSwj_+P-ZC5 zSc|QLAh2CXbMb@a4CC$@N?ERj1a7Z# z?6mnTdc{$w&+NCp9h7a-AZ*d&=Nz*2BM>kwqWAqBwT|YK(9@1E& zJHFxDbOOZ0P!S87o=Tlmw&1o?`yqDx_oByE*7+9rSr2ith)0F+{$kn&oZcU>zir=G zwj2Pp`E8ptv(U?`-+muY#Xxez`#Y*&{!c=} z16X289PYbMJ#{$_ml@66nUW6u>Y2?gKpzoIZn9Yx#af=9279x3D_V6@Ri@(<$9m`L>}2b=xI>SW=#9+`wh2!VApxZ>FkCc@>~*t_eWq0C;K<;G{j))~Kq6p@ zw}XjcCp)nRB@eg~uL>)GCLV?L9Dqx_x{6BnITlzz846{Zu=aq2ngNtT$-!FfKPbhq9<&B> z`HUhm?DkvAToT%`p4x!t>3+)m{Pcz<4Dw2Q@-d33ujc|K&z}f`7_%v0W=+;6s`+}+ z8n5YGd!!*Njj06Pb9fC+ZQ68jZD?{6T4YTl0aEQ_o$R)X%?92K4lV5w)v>Mn`!Gi%_c2P7wT05U zVQ+o|lC*Y14R|RI*l5Ut28a_L z>-A_+AHiSBW{%7qE<;Pxmi@QJtV_F0pu|q~VxsXm{HsCw2jA~1dg@gU+PpCJ*G*tb zxN-C0`%pC8H0z9^KTqwX%tNrI#!`MXF{Ej58^CE8sp36}vdvMD-Fve#KHkRq1)G~t zNl*DQkn!SvwBEa4TwIGS`fkKRR7S0im113arf=%iCb1qv?byjxVmm3=7ft*7uvxN; zfGGbKyffL)Hp5pivF|hdfg##_G0$0_=mKk)=d?BCV^Z!%Hxe^LC9j^+1crmv_}9sv8*l6WG!tT*Xjk|(9?o%cJI>1kaZs~J zhyg%MEh^_)fVdMBG^skS5mcDiH&|g+X-9)q%WucJB-9Kj3Q0LB__J-V`5-^&K6>y>|X=J|P|M7_ZR?Vfk3y;ujARR#hkHGN< zd>!Krt;c`fh>d#xpyaou`pe7e^LMYlb7KUPfw>+Eyd{OUOAwtN;aOZfLIeEwG$P~2 z*2?wnTbE~ce=7J+ccA85Cw>7PKxmMtd;|8v+os0jEzaxjmqLP8?6ijqy67thH9FQ! zqn<<@wzG&Y6weKskHMzZ@nb8fy5B9bqg%aN(y=-{T3~_q`2BPTkpzSCt1xsy|LUVW zI?}TG;gd3sPQ!uvetoWLCODV!3)3|AZoT}pMcX}bSSE~OZ89vpuR1>gZk09;yg-{= zz`SDPL&-d4Mb%I~v%}q@$rQot1ON6gKjtz#&kyDzV2%;q*5I(%=#{+K2JQ-O#ST0l z9JsHKSKFi*=99#IbGdHFDfC$TI`8-6-@_R!Il=qUZtxdYeQ(A0^1rF!nj^Zj#P_pn z*n3DU{72kvDIY zXgzXj{}X;R{L5GmHh0Yb8 zlMPLV`+6(Nqt%*Q*uX+ik3npc@>uZs6+p3i@jcVS4f+Tym@#v1!2JvWL>Yd6;LMQx zVEi*~RX8SOqGMAYxft?(#Xyw@7PZr2P>mrvEF~|@D%n{Y!_OakzB|zh5egg6c4JUY z30<2J#wEY);B10d3WrF-3|6(1@4W<6^_Q`fg#W(s>QJ37NDy$$L=#qA!58~3|J>nD z(=dose?`n0 z440|^-p;{BjY5Zyg2rR_zGDqS+Y!W>EpK(?NAHiCp)-878S-tx^=p`qH9MJ=~57DdGfLSy$KCjBc z*oWH;2A$7bx#*IBD4E-oKtlT+o&m`pX>pHVBi&L~vXp^(E@P%>ex{uzD?xSO7E#bg z@rK6eJ<$Bd_-(JlF-j{p*J!>WE_ven?7k00ArOlK(WkWHAI+a*=E;#sA|CC=AtF&F zbglNQY3=Bvfa+tw)Fx2MYn$^Jkwq0vksLc7WnFdkt;W_4tpdLVOy(^shG$LRcKXX0NqY6z6snG z!gRmOn+-tBZ2)&U_7h);=fG~0y&cEU2n8eJrFW(b>Q%}B)E+>!g8UnnWlE? zmEiU{_T1Y{K4AwtL=hS_bDR7i@gbC0`*h{eNxl4$joMM4+fnWk7;-Mvhs)ki8HAXZ zLEaT+aB_T9x^0twMDq{X`A12dR%uzUWW4;uWA^~c?p2w`>!HG%op(28!Lf@RVv;`TbhV9M-bCp&Tm#Wuu&#h>TqB98WxVbL!tXz zVuMcf=1%~bop%iluf3-dP#uQ*+AvZ_Eg?Ifk=}*f{qAxvX|9WpvYLUh;2OAoDqN*R z!L*aQVV?N{p9<3Awn_yqa!IB#T4=3Gatefhl!~Wx3fiugy7Vr-*mZrXA87AqcUOvl zn$@`84g3*vA{uV6jGL!S&1JJJy?uP$VF!sA!L{I3XenwZ>E%$dt)7dD5c<4iRnSQA zyjx%){o=e3mK5aPD+-zu@5h6srq^V>U*#k_#~{9ER3*fMh*W}aF=H*#XGZX2!fmTK zQ*=rgSvD{sahQJx^GG`%a7Ed7q0en(lwVhZ=wU-RY&;dcsaytOq|fPF&po5Bff3R% z`-`e?0I)TQ`2p{vEqZde`JV7OX9Oy^ydg2iE8-)on)m^m$MN369NJgmr(IC+9;He9 z2ruG3WFoS6IP-lF;v#4cwwNNFuHs{2A70u$K3GTg++yHgbB#`C(!KDbOkn|V6!Ln^ zIb5P2ZqZ*8lNwSo*U8}J&q7p zUsi*}Leiu1_?K`B#|ntcbY6DxPfsKVgj-o-qbfN<1m>^00lzf&mMIBM=Qb6@>uar4 zY(bE116Js2{FbRy`_0fjeji!gq_I--4*qFRwcomFiF!a8uM(iGk0{!uMy%UU%8KgE zqgy1WI|pcU7{5tBZI58tK-bq-#%(GmO?p!#-U=DrHR}xQc{i^`ip)KTE*;i{PboY*AEJ|60Qa&2k zw!LzikKL}LMTQ?ad~yg2Lqf}Jlmz-6dqi4iW*wT&mvw3ycxj*v7&3wL>WJYo_SAOv zQd9B4|I{De@F+FWH~cqKit7abmEwI%sq1W4B5^Ey9>tZ7SoHhg zpj5&djPwiAk?$$_teN)*Sko5?^MLmfjU)Q~l+r98=TL>lZT6DD&V-{}gTBdC1wX68`p334OPQ&JCum+7ZP5jps$wB1 z^=`ZI!|g=j(3tdF=;3Hy{KN#QzYbkPF=)5l=iUQGbT0YiFe|}rK6V3+IR!V{z>S~U zPydJGSY#nGNdg8&*9VHie;-ss5jPO@ZHM4W3}k=#B;o4pav$)D&H{k0m5;e20g4Bc z0GMZ`&iOQ6J_t4_nT_WlX-`4RG~wX5&5SNzTn|{?Maw{UyYHnfnsOP>;e)XE7+s)* zcXt;Sh-*=mC;HQi_gi&`FHtG?%1@Z2wekfI$82qKz+9OZ=4r*aTt zKB{!e&~~EAUd3(hF6n`ysI0sifuAz2X^oNd^593%y4>`oCzlAlOj0`-=|e-B&_DSw`@6;vOQO-lF-y=fSdeD{G+gmoO%v*Z zBluw5#j@Ym`p?fVE?M>>)Uec?ccGer3fko(#8Wx6L)fD{8+ROc9Wp=^Q2M$+%@u#G`P^on?8OKHs;FbY z*p6Va0&R5bxEHs)@ z5lUEeUl3TJ_6Dj;2V3g?MJG#^h^MRl!00nyYR9k;;0r z4V<#=;(73l&j+{H7%(!n&%C6CowjhR@F2M0NyEY|&I96wbYZ?!`S2K74#C)S5osqI zGO+5o#m3muYM}Bs6cb7UMfOX5UJUtgx43>_r1-Xinr9;?@A;!eM!PU#o!H{iR zdDN96Mud%?%hSd77_IWud|0dqQvwcCtu{w3Yrb<>c-aMVO^l;p;!iYOYly3iTXD%o zsAtynyEzH7zU{j z(o98A$KgRmdm$<4CsBa4)Qec9e&a3B<$i(2)@j=+sMN6a`QZJUdClZuMJs*EK;p&hWE?d05=sxeE$m%lvc1FMw4Ad?jjvOrZRrG z*wK47ces9$ta3YA+iA_AqP=V#syb0x|csNS8L1EMlpq z3NZojF-fc3+IdUtWK)cf*`8?_srK?7e{8M!aOZ_~r>g4ZvX*_+k}>BErx+UDpfQ@Q zufLreKU*d`3k?va&!%I3u)H33HJyok^4JOC625l6?Z9717%}ZBjKZ#m*$AOa4cW(( zgUkC?78ljp{anpfJ72^KQa7w-hF-5Qs-sNVXRUVXr1 zi@513>P57|IC9&K+UeTA6Tf4UQu^`1DeSo-OlFR1RSvbEYYQ45P zdel1+t4K8PeQy8Wz{D4OO_svORm~eq2zSw~xC@%Kqj!`fzfQa@KPjmCP2h3d%g7uD zi(lax-nv)3J_{%3TSMB)Z`+=K7ylj;gB@>x7qPQ)R0FHK?~n_xt@5X2Ub?%QEvZ83 z73KZuNRcqL4g3LN{SwSqxV&v`x*|+I`P*Y6*Ph)pFO+%7>iET|w6Ag8MG8TZ!L(WR zmAF4Vs34aykv}=YyJ^6!Q==hqNOy7}Jm9vmsA)$;Y*JC@4D|t@VW#D8Zeh3U9gj7~ zyA=`5F|EB7Q(NiD4fXTJzirR2xkU5`$7cSmF<27)g!Jr88;y|4ep@@8{{GQ^>Wo~u zl4V7~8{E3M{v9^gV?h}~} z$igwZS*?nb(>zz6jNt!ar>$EPiYt$HRAt$wzAd>Sz_REi9i6;>tcHJ}nC{PA?=bXo z70y86mPMhW&ra?1*!Q*l*BO>dW^b}iqCJ}jNmFbwGWG87o0#qum=56PWZFC^4-eP> zXx0BWAamAPF_|AxHEUTPrLf*^DQQ{R5VjeNSnyQnn$h;iTK$|`P{NsC#gypa{$jEa zC7yE-=|X%xY7mMNgr0j>9^|2nU$GaDoKbp;g;&`nV-CW7aJ?MQk7i}FJzpM_JnS#LR-SO zCY16WP`oAV(#)P8cqoPXn^R!m(9d6r%5QMM<<)P`WP0F5$az%!d_(ZzQK`ornW^vG z`Qf6!9X#kcy3FFH3x5TyX)dkgxz?gM${SQZA4xG}v8ZJRHoU#!)Q^0uwx8x` zvzF3I6)**+zerQ^i`^3w6`Oa<|GZk@?W`1NZ-HSdcO!$(~=a@$g$f>llHr+w8^4_tqj!lUt=%%xT4BGYY%0GmJfK3a&uwch z&eDQyUO@uQaL4ffUbPy_{8FAMeU3+A&EROYLhkv=a*<~k?#H+?>Y(_tJa>vVuR#Yu z_2b)D+UgTm+^v+;W>B{8JUlcP6qG1?E&GKh(_P}iI#%XC`KdOrB$rgX#p32}e@r-- z zc9Jy{Mp!gmJssW}1A_6m!xiu6O|cPjwJ!AbF-zD_I*_<2HDXI8lg@)1E897GO=Kks z^E01pSXw_^H}?ux`oY<-o4QDCu^CmuA!@8=SzS&O-tl>0r6hcZg+Jted*No0!O_kO zQ%WSF;LhN!WcQ&Zl%^*1poj=86&6{H2zwoDU$Lpp!3?*Jt|3CcW*+d4O^`0=dGcL# zU1&T}8W5hI%f_}=Cq#wlm4sR6=|0ovimCxO!UcB85)VBORHt9ux&9d$);w{kv`}n? zZD%|nAYR^ptAdbdpu5H8q9Jj^wgLw-Ny4CdIK5CQp3TiXT!GKNvxjNS(5%%Nb;Z{4 zo85^e1&*M6`0>ydH+0Y4CM^?)D|FHP)FOY*y*0rXSyp{~_dIUv#9YbdQtmjTZ&{&L ze6FC7{78o$t)qX%VGh!cE}TR!hncu1xln z>DGK;YIM$>B#UXi;gaCO@Cgx69b{IU+P74WxaQd0R!to6ViDmve)QTh0XYHAz3#@7 zBfDC$#p67d0q+dcfi>qc-SNa={8VbkuwpxhC+)TJPaRw~R!iSB_?h6qDFX?n>pFbG7GH zT(vd~CV!#Ck&KgE0~gn728PL~EE;dlFHa_xulkg)j^hMw*Bckqb?7I}V$HI~1^ z`7-8ndfC*+kbO&$x0y&0O?FnCCht%g0pu#bl6R1XYq`(QF0%HmjU_QNJCn{1%$v`uU311;`aN~X^o=$y_(&38Ek&$OmbDpAZpVa zWtJL}RupHBd7+)L)&KZeG?i^7Rjk=y<*Iz+)7Xb2_jEhfTufDFuoDj98treRB?AqHBRxf&v`+bZ1tRP3W zjk~MxSlw7*$x?)wZq7;`PX;A^+7t!`)jIlH0IclM_f~~6^xtH!31OQC*c`Y#LsktS z>q-O1B)Q~RMkJ<-?pD6ANn0!J7L@Dk$-lLR{mRJHSU*s8*i<0rcM(mdjC32`SfT90 zSl%?%VeMg@y?flm2Z}5s^z8@460spdvKc1%cAAVQ_x(p&?IVAGcfS#;;CsXxPdr2N zgPml~>rD!~V?8&cl3$V7=PpOy7$QGq{33&Z&w|YIudq|=)?d{mm*Cl)BSVWk|JWq| z08usnHYE1UL*Uns7T3Oi)IH^;Fre!m+car6=Gxg6%pZW5vZQJ|W6wzbKe~nA@%9|r zW(bYYo9a)^f-9SV!|$l^51v6cQ;sQiJRxMS8^o$nI7ob4pxfp>!_U2Uo}&n?i{jDO zER?sK&Rq;VzDXad@N`7awTo|j$9`p`FdCc)dks|NzTyekXw%|;2s9jgBg7_;xUE-H zwyXNxNpa_^7;rl=1%{}{RuJ}5PaYi;79AbfQ4L-;>q(*>Ow|g$l&z5)wdiKN*!$_l zS9xeW<;qoleExL7QqdhAUG8avXT05A;O$~Gm~+v3ESlWwL-#Y8)vG)^eI7|?Hd_qD0d=9CF@;#v8idfM3{ z-U(I{qsO1}h)?JA0NvIVxAidRjC?eMA^-ol3@NreLDcoCg-^<5fes2+{o}kcJho0q z!6AQbs>a#M5?W8&?=;Ynm||9(Fptofa%TP4$?&eo=o^8In5z@iN9Zpd*gU~~APvi! zQWFQ>t@o6?pTB1~uTP1X?n3)NUc2gp-OJ8!m+TR`InL-Nu%MYj^*KcD13K^9V@EZp z_f5f*yQX>kP#roMFf}Gl`xwsT#^1M^wrR7*P4xW9t_nDqbblLyE-PGOD(0?u{vLtb zq_5EK=#+iS{`9vODwc=4JRMbAm)hjA7|;fUiW9T};c|Kth~w#SinJ-J?PnfdyK5#t zYz;qk1-+k;P|d`r23p3_XCBuV^!WLsF+1hPa=&TZ$5|nTKND^3zf$&MNigw;qOEK?BN~I2BV!F`(!YP*h@=?zN zOw(-1`U{Dk-_nwuuj2d3k~}>qeD%bPTJHA0eGaI9`y5*3>UB2=AH%a~+0~Yz`ES%{ zp;%6O%OgB>f4D!xAot4R8}SXC52A+G_-zwtgXnv>A*Mv)8@Mt~=8P7%t7EcP!LYtVb@nNmqr()wh2Kfd<@}~zE9g)fvcL=LoTLqmK z>3}RCa8zN@xr7z#vzZmx*(}=Ok3(nz_JmI?X6UvD?SI=Ih-5k*To1T2_+Y7{X*(Ga zffTD$VNm3P=jhGbW{kHs>Hsc>0$;5fZG8tC>%s>$bwX7YiLvq#{p+7Uj80lW=ghn} z`+))Sn)~u#jw$DP9I&Dwd~DS!P9`2k-jj+6?khO?x9@>nrssOiNoXCu`oxUfqy<$g z{x`9DtE`h|RT>v&J0Uf)c{6CkUN1e8HYDUQxW>DP*HP>aob)G_ptw+3CS?`ZcG=d) zaS2sJI+W55Z!=*SbO?+#5U0Ho_M1i7-W8f@Km6a#w9p#BzRde*?<@q8LkKL+e)#yT zc_04#@gmk>^gmikS$vknFoXp#tWzo6=U7Igz)+pvHB82L}C28kydq+O|c zTo$pjOaOu^JNeKL*|SVAh|UE1FC%ErG66Uj$b+7lFq~zAWtfvnp+tMgX(lL!FZ~2b z@s?aZo0Bul2^~?+v@k8a!4Kb_VAov+jxit1rXnRKZI|I71td5RYxkF!cKFJD`Axzc z0*m7|q5TYnqyOR;jvJVaX3^H^Pm*uiA?l72qng-H1_;#RvuN@GR zbDU%hc!?OU?HhSv>y0P+newqWK+*_>6;c32o?K`FV&Y|?5s+2b(cKJ(VB+D15Vr)q zXi3#siWM*Z8ueA2cenC@Z+JwUyC{6;61MGVk;MP(nwTorsdmVqa`-`yZrmAszx!JJ zEd&=iY#F{QOer=$c<6NCvq+1HW+JZEkCTii+$j#iN2smyz> z?~f=X2j9>{P#d4#N;?B>=*shiR+0*8f8vn}e6%+Z-50^_wqb&{7tMj# zmDe7q1iiDq$i`POCK@VO=$40IYN2;C_ewyL8h3nC!Q!aK(oOA7vp80oHJGcZ6fAc9 z0^Oj1ecnhMM59PnW~N=#K~NmHnl*$XzdQ@W^fGIYbkEa*91S!oTfhCDtdgCRb_9F$-Hmz`FIFrjki9A*NaV zontF_B7U!e@YIb%p&J^u^W)=9YDccQm|7>7s^kUYLQxi+kUv0yX9s|1f&1nnFPdQ7 zUej_AMK_;xCX0!=DR=kmlV^=puTI``HL6b?x2;85Q&5dx@e{2Wj5xGVFKj?N6FqoQ zAePpaZczA%!;3#>XhzRP#{8{e6ZBLB2VafrdeV(S=t9b`t=-^ReML^pwQAvljd9d%`?VjLq=)zXxkKhD6Tg zma9s@mf4rcD`5H3TR~7%4OGxb>hiO`g9u&N1heTTTJ%^MH^w5t;t&kLWADYm=c9mgw2EVOz@C=WWR+&F)=vlW(kB`peVLsM;+;IMa>>gtJm~s9?n?J!A^clvVfYfV!zMOK_lAmI~ z9R-p=QrJHfoyV(TA&XM;o(i!YrzXtv>GCZU*s=^w+Xk#M8#z3fUhh&D&MAy{O08Qj z+37E@Ge$2@*u%L9s$dy7=GqcY_X@L2HXYB(a4`>0xaWGy>eZi8w*oyU)^BXvQk->x z*N!*}EeYpi7B+*Ax57FnxmtOaT{|Z^Tb&-nheeKkT4k%8&!=7hXW8@4n`+WooKhQz zEHYF1?zGU2IK2??xUV(8MvA$u%sM7uABuXzR_j}f03YE>Oz^YZzXxt>e-K&Y?XR4~ zS!%vQoX0{gGO;>ab(mIG1RN`|v?I>l5?` zArQ8=J>Snfr**0NuO47UN&=cL^pi& z@BV;XTv@BxWh@mySRwast9&oQU&L`O@REL6aXinZBNK%d^8@*a zhSF`iQ29NuRWz*bDu)Gp#K&T|{PckH(ycM|2;XC#@MAWq-xDeX-=1!OS>${5Ex9pp zU(BJ2qLT@ta$3sESdnz_TJXy|1^4Nj8fM;n7IV8Cu{X0USFlg6upiIUz#b@n3GWkR zRy?+V8&dauy_hZ!nGni|nfB$d$-XbqC?PGO6Cm6d`H)E1-g$foRzd_b6sLhYWmrb z#xUenO^)2xXC*C|w}TY*R8-G^OtJX4qwTrnp+&n#EqQF z60RFcoUNHi?li3R1JP1atl6^CZuH~Cy_H)!XMRkgy5oMHe&L>T|G&ra|Fh?OpYOiU z_xpK&p7-nBRd*NzsOMZ~hYf`s9{RC3zDeQ?1G?-k2rF&=aCLE!A?n?w&LU9p+1=F1 zND?kEFu-GXPaxF^uW7ZIP#q*oS(j>KJ8M#j?x>DBS!8k04FRA)bsh+^+`%{Caqwn2 z<2p8iPJd!?e>9+Q(kn;kga|Xt#~L3f$ocI8Bh1%24*`$kCPZWi#?l6~GWdL-1QCVd zs#lNlH~4`!V(p>VSRPRGhLxp#4*M2ir@DkXLf5DuD0K=s$Y-GR0+?wVpz4g)x)MEM zxnLR_NAm`p^Fbhzs`FL}mj>`{s{|7CE>OBBmzvD+KjWayV-%FUhmXFD-gBa!#|186 z_1}G4zaSU!sxX(dn@*hR2w;k`)Gd3(zw|a#9C6Mvl)c#T2aOAIMZ*;mQA`=r$bi_x zp9)zww*4hK>FLi1^>hNwZ3deAk_THr#F|=6n~f4-awT{pRd9Y3`&TQspqe|F0dB6L z(1DQLFNJfzQ@v!3oej7RBFb_er=7(Zo4(&@sh>!LH_|8~o?sY!C%Cv;BoDh{X#7Is z=MhbcH`Z?{^njSQTc>MG)&%kYgb0sXQP>Z0TfJ++hvNsFz~u*d!?s-FaoE0NrL0G! z@w_C%03Be)=#W!O!tdv`v5!$G^LC*RxbDacFZc2%(rVZ8Y?~bf_H?8Io^Ei3pJM*h z`m%69OeC*U(C_yRH?}{uE=Y{bpN7irHUI~?Ku=s3_l%iqKkI$er}g#X6B4U(tKaJU zm?#&9$ZBA7Qa2U-xK_4jS4~5>=kZk987ye~kTR+090F9Fr7qZO5jR6|K^<|TJ9!Sy z!FxteOm`=Fra}6-><*6Rp#cbGIoq`tkNp_IS%bkQ)@orsUIhO>op zhCS-VBxbN+>5(VejdiR!@QUVA@7Hg5JFt zHZwiI@=8GC$Z1wCd|=pXG=e3o6}8qp$Omh<5mWN<7Z&lJQyQ&ekUsE%#jh$3sDroY ztl677&ZnfO5Xd%_SSaA#6)0yXp1wOCQt4uOy)hykwS9{3Yh!{ko=Sgu8zD`{lS-io zx=*J>HC^LsqwP)_yUt*1-5NJUj`jl~Pvb$jQ`HXhx}D$$ISui%2pkKb6=p_LF literal 0 HcmV?d00001 diff --git a/paddle/operators/images/batch_norm_op_kernel.png b/paddle/operators/images/batch_norm_op_kernel.png new file mode 100644 index 0000000000000000000000000000000000000000..a99ce81ff3bf42880ebbd6a1297de3bf038e09b2 GIT binary patch literal 165209 zcmeEOha;A4-@Z#l8cLc(gp5jLl!S!bB(jo(kYtvS85)XGMzWHOWbZ8_g(zgNv=A~v zBKtee-e>(0U+?=o@0)R7*L9x1aU93*I4>{N6UWxjZ=K`hMvHC* zzEaC{Bm)1l!cgg$EM<}W=V{UN0DNWTC56*h@Efh7c0l64!W@Lvo5ve z2OVl>b%Ux*GTWEo^8fyczZ1gv@89tM6fEgPm;CoD3U>ehh5uiVzgA%MUbuv#T9NZq z%a>QT8z<&j4m)MV9uMKTTV8f!?dSM|YD@pKR8PW7OQl7uK4_mjd2)PwTg#e*R&Ef{f$Kz7j?Ne-IZIrcCCutld-We@B8E^TZ*On4XNrMMiKkL!UCYYK z9v*hKKmYbo&F`vU!Dy$+p^&&A;fJnjWSO2kc;LVrW1+2!;^N{7ih+EKjZ_vIi zFE1Zq@t?VDW208=GTjt?j3Hdyu6jiLok`}mcOg47+mkdCwc=GGBv(@?Dz^+a{O2a> zO1lcL{+by~9c)QHQ~l&{VN1%{u%6w=BO@cd@7>EX&(ui{6*Tg^zx|L&>$RWXl+PC0 z*Q9)MUB7-k!(-)vlVz`7ozHCNq;dxgc+4khCCVHc)ypW0AyxK^AuIoL6P#fKjn5OW{`lx0!*=cWk>SopDeF*`Z?AR=PpW zW}3a#fp-1x*?ZrGsK0#qvf7_(70uGp4LjTZQ+uR^3SDP&=r{7kjE-I^nj3#Ber@je z(F-#EF(;$t2f80fJ+c*OGS1fN{r)}SZIA#zu50faB%r@nN=nMn!J)q++uX+1_V}Zp zlM@q7`B(akMZ+{RO_VJ^6(2qsar5Rf5sNm}Lsv&0a`sjQ9P6tM55S-L3*8kyW=(N^ zoEJZrdTcvnzxxJm%ZOL}Kg;Let|B#0?un`vA?eDx)yzL;`yqP;|M71%=DP(qGczaR zVv=d8sj1c(w?g;oy@x-vuI2ts&kr%C4NLi^#2Lsmi|O&RBvsA?J- z8d0A&P$bq~SN_k%IHT{}p+jV-_>)BFedE{d7j)2J^r@ktp)r0l@3N8IK!Y5202TR( zwQKZT!+fv!+_`h&j^i;gJq)5q9R>f#;o+LdG}=i!TU%k1NTINr0*5ifC`tUOm@Kdx z+_Hu8WXnr#2{ENdyH8@_16(|m_&Ij&?CtA&9&Pf)d}Ul*bYh5)&_J*|t=&u&@Z9uVyMN77bHw&p59>(C{o^_(iT&7vJ4-4y2@6 zb3#>n`?QDa(W6I)C25M-?#w*jCStbb%X5^V6)W6#eN$LDDshkHpxV*fYXhr{8v^-H zX`~sv{8J*BHf}sWbfNkoHQ%OJaeO94za-_&8-@J*e0=0-T*IGpBd=PIRD_X>%Zt7G z#g9!DE8VVKxuQPuAwNKOgP6Fu4mPB5y!Wl$@6mHLj}QJ#Z>e4W&x4JAok29;8^-sg zTF#HqhPk-9a#73gZa!7y^efAN&-d3tz!9-ZZ9ro;y}+`M_}eHs&Gj(NW*d-?qD z?g!h%ZR0pEj(&dWG@RYa`0I4ZwIZ9|>Su=K-dm**zAYCbTpz{#_))TGHBl2LmYQ+? zEw_q_ildX$_g}60i*=_rAIU4mh(?9zbRr)Swwm;l|;e^rRN#4HB zTx(J$`OX$(X(ef&y9#XeN%P0YA}(sgwVC)a+JDQYdls*R#D})pP?5o%vuxtF3W9=y zLLtW!PDXpyf6Th5zHzUvHgXF$h)Ueho_PVA!b`1UvnVKe{5Uf+GeIMM{QZJb&|Z<8 z?JO)!NvAV8XOUK~+KL^2a^IjS-N4>_yrKLrTG>yDi9EjMjRbEjcbbunU{<^Y) z;E^LoW~TchxTuGnCRD$a-D2yIQh6dFe);Rod3TX85hQu{t=VGTe47E(LH`(3o&@vO z6u<|kgr7CO>^F3Eb#2GK%Ap)yHfu|Z^xm|`Z0^N)HUIav^ytaq&gYC=if=4L-YAGaUoQFV zwa!AuJHO;(B4^RmN9PAa*6G^!HjjQSzsuan{{)-GENY?95T{m})=rfj6wi zs5Y zmaSp1Aqm!zt)$kV^cDz5`NxZ&_s1LuT&s#!ceygm>T3971)t+PRfg#b{xefQQiaY} zOQbCZ?0rj1S(Hs%$;-jVSFLT-{Pyiz?SkQRYuO}|;^KHhj&s+9A8MuBf+BAU=w)S_r%yZecY%$G^;8NP;=P}ub?=W9)4J|;>Z{$Z9K7$y{OnZgSa&7cpv&TP zejIX&Gyvlg%g&s5{8Le!o0}{DfOY=;AW|we#yL}Y<8E!=zgM1a^=jc3KUeJo{ShOn zJI`boQ*l3u*XCzzE?x31H>|JlVI~bBhKcylSct``~m_RX;Q^@?|zr(inyKL!T}&0hpdCG($3;o;T7Hd@ z^QU6h)qt>n{n@bw_w=aB*q?LwQ;Agl7_*pF91;SUrKzVXfQ$NEB|@vKz%JXWtAH~# z)A~~}d61Cf3$;=s+f4I+8FaoMtxB`x znrm~azJ7D!sZ(QL%Qcg<1(n0aN#O_BIP5&x^k~nSp+_H-sy%3E?+s&31@}=T1W`BE zyUqU!N}o^rrTGIWxPA!i}UgEDWctx>M}m_`0?W``(d5o_RM2I!Y-eO zul=kMOHhrx&VAx>9O?q^m2dB;lVQN7*XG75D+>fR@@r{;Yy^r~f83Su5^$Q6TK)K7 zrcRDU)l*G1I>J5}SRX+|nThv!-!>v`D2^PCO8L{u&^%bkPJoV>N`u^H$gS@kY zL*95_T`HJJNR7!x0X7wlKBUcGhu_O4DfTLPpM6z+hXC1@oll$Ms#P6F$A zk?TnMsd6E54BSDk~i(qiTOSMslYB#Fth14+qF0ZgXa-!S~}BqHCX_|^x^N{ zXa4G-XdfNfxcNBO*~tnU+g&P8nh+eERFoiUj;|_OJ8popYt_#AS&0Qp{37dN zC(iadVS)K4Vj`0BO)+~f`)8+-ef>OlU#7$gw~#GNMk_ZQWyVHV}# z;E?$>*cwz);^zIS4^)Ubcd)N->OGa6^7$m4D7)MT6?Ua5#xjO0ZaFb%K)UI!U?Hfl0#(;GA zo$m)(8NOdxS!s^qPd0Kb%YhwK6#kisMwMSxyCNjA*4}1d6U})W$Y1&2NAUn=y;ymF z`!)zfAFs_0UvN6JRmfNsQp%IJ^TJ^@W-Un}8OBvnkddIc+@fVGZ9JT+x-@1P8ylO^ zEKsZ0ecBZ(5-i#?h>ikivKzz!)P;Pdq9bv!r>7@o>XVy3ctike*6Opiwpq205AFmT zyjvcH2Qo){qfX8bW!7n==pHOw_o&9C5xqh)Qf4`b4XT$6I&%OT%=w$4B!k%8=1zhX z4D1|C18(Fr{};pTuuRlS4n++J|0cETv8ZLJ{CyT`+)C+;9f0&FpDAw4FD%6J{rI>C zi0-Njwhu~19wQ?o5}c{4$oW*|1Gd&!!cQwid|>Fe{W9g$hsKty)`u=?$g5jY_{{1 z6&`CUa_Uuixg5wo7PE{ds6Ih?zCjVuFUWssWlHOzJW^<5LUrGhV>O4ZZEMyk&(FffZA-zk63mvesnt$^9b7YA?ryC)RN z=Ox7k1nNR$MYjwGkgHa`~$)RSl6xC-Prm zTVg5LvK%muGMC~*Rq#ks_yIxQl$ZC0eEbID(G)}70Jyq*`SN(MXI`g>WUWMF6=wG8ds)5Z#GXZxPe?(m%h| zr~^Mnv|CzrWKq#+yDJ~4jHw22x{g=v(#f?7=e$^PZ|lv_zdu29=r>=l+%gM>{3JIx!v3h%Xoy3ET)L9#fh6MtY6E>Zal0u|{mZa0Y0|UGQ z0s@5*U>s`3l=!v#MZe^0rx|z@*bb1i0qWSQ)O2_i#62Qhc{B0)f1UaXK8ckHMj0!f z`Od#)yZA#|w!iO7;tELm2`2YEhp{MVN4B$*mU8m)e4?US_4V}?=UKLI?;RQ0i)IZm zg|xM5lzuQaKHLTA=6ZU1Xd*o9JfFAT_3>#2=F2i~-AnKh3IP48RnD=wAwcP%!&DMy zAme(jzcddXJUC`RaY&)V-M1Gx=K^&ca~&(+j1RvM`r-i!b^YSPloXn90<;T>{=aW_ zNBgZ?w=UK_IV?~4`g^R~-s4%%x6hxSqADb!4vtpAS;>8`OCdk4PO z3J6BPEV;ofx5=UQ;YSbn^-R3Z98qp~5)xEYR1_1g2(@KbeoPn2 z*7bqTpmINdi0_8N=m<>f8YW>!?WrVIN_;J)( zRB&Q#(TETQW>6gH>F881T^gmC-OR6b96V4HUz^@eJ9>P*#lPi1jUA92AjEuOZhEFS zNFQ-H%sG4T(4oQXR=of;$rnXMn#npM0Q$or;irJs?z2f*-DF$3bg3<*YR=iveWUs2$_-gyXPg&KzlBit(dDNyp)VgleneZEy>j>ySe7YniBw3f z6y12R9`5hmh$^jQor}OopoM%4Lhi=L(*D(&6y&~s{W|fALCUo3=1TMm2Y**I=I0>m ziD<$^vFrKzbzHY@x+z{AkN-YY#GJ$+1o~Yk*^oHH^(p|Rf+Ikyf9<_ zKi-D+7!RN`{`03g3UF_Ke=y&V+t?;mRbTLsSkmTNQ}tDvo@3d+1I={gT1S}upmpEO z@baH=$_~_mRh#$rqcIeKYl9Xf2M2HIDt0~N=jTUi3>NT1R~Jcb&;`_(D(%{YOJB;! zj0a_lmwS|rWE#qNsL*-u1}PAFnSy&{!v({AmjXi~hAk{?I% zASDHGD+?RHS%-M|S-Mjhz!(+Rj{qo$Yg{wiP1`e`eRTeDxHSJY#UvYnXA|6<19YsRqU+Ft$CnOf&l@|0dq6ZMt2cSdv8+#@DWg0p!uKVo}1l zofZ%^3EannOBwAbPw2_MJjtB>Grmb7!L5NUiJ6!iSScvjNUa!V%OZV*A zz=KyzfJXR{>J`kl4%+lQefp>Pm7YLn2M7~GQt^QzdU$xyGcc%({p10JM}OaCZEX!v zCg6M=Wq$GA-W#+GE$Goh_j_x?P7#@|9ZCuTpCxVio6uisg-!Zj7Z18CM9GZGR~T|ce@8$xi= z=XG0{Tg^yOxo5CJ&w-R~{M1GkmL$+%GU6f8hI?O6f5*p16Butj-K8s6_QxvOLw*OV zzo#nw^7?XUqCC6ZFL0pHK|T|=>A3@X<^QYG*Z(2*u6uguS&*I@@_bsJ^Urb&yhr=Y za{6Vq=WKxnGWOcg6f5{5nvwv1>aA6NJ|h2ZO^tGGZLRmcdl;cSG&C~uIsYA*VELkM zJ>+)n++R_<8n092JLZiB|VvQOAV-GTq`^PNuEfNu9in~-)P}uip z=b`mhLGMH7Sk3riALe`{d)Kf@ei*TPA1rw1!{1&2+`z)Z!ZrzqO9iJBetoM6oB4WI z0AW5MTk`3X%TJ$zt3PVpmyF7y58kqwnYf7F!o$O(fH{I12r<<1m>!mjy?ghP?)~D$ z3ys+0Tai_7J3G%H%Dye3jLM>?FU$-Iba(nf_-#ULJ$aiy>Z9;yks6ALdXAQ)Q(zk} zYSH!-KO&K44aK&%yE__uZkxV8QDwjH@Mty@F^OG^=&TCZm3{Sx-ga@@HzTiV-1yJr zB%v@yLqSP!82c&_tk8%NXLI3I>pgle+ER)D!U?6oHV0W1ySt9}P}i^=P=)@LZaxLw z3BUuFdgXp3;VNVT!kuvo!X3xC15h`Tp?$=5PIZHUiP-k5?m3h5E~N+zIduPp`(RO( zpO=hkgT`qdLNpL+y-e*B&=Zi_wP7)pKUmtF zC*ANCy8onhP~oDb0-26DIJDL5zPcFZ2bhYHEd;xqGYNYk(m7=A6A~Ivw@O7#HS5et zCeI91@w4;~+#FrHYVFz<$QdGT3odfUj_nf1a89GZ_98C7Mc;`J0L=S|5rUjBK3tqx zUr+hFE>~4$bL2N3Q%biem$=Uzd~jI})-gvv_7 z3e_#gdE|TJ^9Lw_{1+}o{FgrsmO zJu;mh?`2qX&IE!sx-8KH6d&@Olen4>=<_yT7aD_uWJ=| zZNdsryM#M{)KZa0gr%(rZBHfzZG~BO;+USEEPaLMK|J)&hQ^ z4x+&mIP-h-3$5o;0B_A)tFvUDPee-)?J^qrm}s$`oA&C)d2QH5)HUj4RggdgZZ8^1 z_y*A?P%>xZt+Eq(Q8IL^iv-mGU1OaC9(#YZ5FNy5u+@DDP0#Rf4b30EDb1H{SWfS= zO+*oMCq>&NJhT(Eil!XPGmrp>Zskc47QpBPsYe}n(^wCSr=_MmWj8EkmM42%fjr$5ZPm`JkzL*3BS z02~rX2{@e@s~EI*UN@E%ls9L+O-He7A)b7=;q??6Y9cO4_%G0}qpycB7^ivYp{pqv z%{=ZII*wV{DWH}nOl-h8Vhea1p-B<&yhl4tI!x43al#~$vxtnS@DadUbz(|?Dy#z6 zT2=`jH#axH&bKLDV2!Pq**n9);xm-D`NLuqbu+7s}b!Si*{kt_d8sF zf4L2fh=yjVz)is3MBpdyJ9qYVY!kCe!rky=V8hhE1eOGo+Q+=OK;c$^k-=EBRxp1r zjxYIi^z;pwZKPnbMx9yY+HmXvYq$mf{unz(UpQ=w#q=7*Bk_agSmRWiT*8 zH#q^gQQwwsG~A+AREHLYYinZrn^1@}rrDY>H!#J;bn-4mqU=B7=|S;o29Ji;rh>ii zK0@1Y=k8{K??3CLTBGFdGHu>`;p%RDWJNAJ3O_F`?d{E9B_Zt&_C>h00E-dQG(TH+ z9rcvR-`_tL9kSl^XZc1S&mjW-^hEQP-Gx&e;o;%UvC7OORY@p^7Sv6BI>x%+J8|<3$4?qVHXWxnnFLphjf;PSlF>B?h;V$L}ED z2w!aA^kI|IgMq@n5~=yCg4vo#!$c9lI8!fp79f&~fDz~d82nW-Zcl~e1f3}!k+z@C zeX|Cj57CBFi)X4wI*<7G3=G5rNhSce3e0KrLwgK`+DgX60P2NCFV4}CC6cbb``|$< zlxJb94+pxBY7Bg&jR$ff=2gp(2NIz^sm+Pr@oN!c%q8A2)tDB;CjrKfXXCH6d&!TdiZh>;0>jA zi~ibLsEyYarY~Sx82(k_mcHA(rghP;M{s~}QPBl30$;I2m10lCR?N%g$@~w%WT>O7 ztLxWD;SVz+G3(}yu5y#2S-Mo%fE5qeQsFwx|SB zhFpa5wfk)IPImSq*zabD5q9%4cF|qU&6?5jKIGH&`?-Gm+P^L@j|UXiDYQ?n*TSGH zStnZ&8fC1s%DJ95eh_+t$EKA)o>7On%M2a9k%0*mulH>N>aq7w0i(f-RUaE)f+j|0 zFxbsNTWiM>OiJ|t1fCzDj2sqNXHk}E`?++W8(dqK(pfCkK1%=tV z`_DRX?nHIO>VF3%$#S+%2a=ij7G~X0?##$=1x_`e?T56`ku5 zmwc*%f-jmb1X9Z^iE-G}_A_PQ=i z_k9Or4h{Rl$%|gmf<4MMc)9d+vQ~80t5-)6Y$A<3OXJ^T9SI8qyw(L*mUqV8i9jbL zD`1j&@N9F>0<-4^?&E2RRCEy$>yHwTI{u8yc7*6+?vLbc0a5c21D7p^+16MtA4ac>9TJUsx-;wZ4q`YnE#e6%p_t^FMIik2TWQ-ZGbfUP&RL6>Eh0| zo5r~)h-BO0wTh~L#C~)I@f(mT_z>X9_EZ2ySA;fnq#NBn>x@knah|jQj|=jkiCIdh z0cv%Y+v2t8J&@zTEz;)Z=P~tulrs2}h%>Oo**BUZL@>J^d@E4$Ff()ijX%Z2ebE%s z89|1&!yO{g$xt;(kq53i=Hc&)r-m!1A4U}aaznum=r4%?9`ZZ!7re9qgg`%`ls9@8Y2bC)y4)Fws^SLO8(J!`7RTwha4RD$y$+ zU+BjWpUj)DzrBC|zD~Z)Gt4$lTz0nwb@W%Zut@Sm1ppLot7&LL8z9^D8kt}#)^x)aT3izQ*#{q@njwfpTcek1z}GT|R1;W*B-Z=Whc{Mi8qK42L?6sz=} zi}1q{5CjyW{+MaqrcK=X`uf|YuIc&{4+D{=A%Qc|Dm+ISJv|4h#c6gp*Y4}7GS8dv@3VFs3QS0oc z9ejM7q)VRU?9hJiKkmfRqtNufhhP0jWqJ(^?nA$6w}QvQ4cg`9WG;;OSKq-|2zZkS zu^C9-KSr;}=^{q#ctXf6ATUl9NmtRjT5wUKRIGjSh8U1xii2XO@Bi@7Rjcg>F1fe% z_Vqv+Rzr=1!oy2V&@U;zHvj8XvG3ZLqb7>(&jo=ZP+zx3azg_oENupWe5J@?8GsZF zAO)ecyfP(pm(0Ndu~m0`-g*eOD)2pv&1L{jg@7xvaRV!DF|9;NX@c31bKqlBCmBj& zkZgRs@qZKEq4LVgw{>-Odz+f#PVNLaz005yaUC|ZH-0;_;qd%pOG?qrk3vZ%V>Twb zDzFH35_mqU>Ii#+cj$kzI5<0-BX%klun!_OJqq+UcC~=}?`$HoI~1$Yf;pDJ-KW*C z{J!lj0u5=7KTTZWAYfWCZ~{||u_^h2`9l%s+!r~3Ix)Uc10?g&kAlIgACY8#WV?he z6!1S@Xk5m|#>SuXmZxU|Le<{djG~|i`BoEH!H4%aEP29c0yV*cMcOF|y zRfFGxRg0hnlvwc0uxTrTQ;kaOaDk04Mi|g!}6n3GB2psq3$Okl{jUQK3eGA# z5fL1qw<=v8RfRpCSTz6C0R(eq*z(nNaFI8|=y7gtUL#JG9p;65(gzw|09cx$c~r>m zCq*1&rQ!UZ;PjO$Jqu!I7Pgv^u0e7}Mp-2u5z|rPRsvLd@r42d%K_g3H4oz~qsp}+ zW=(JCL|`6M1)RDW>hA@ECBfG4o-wPxVwH5!1`ZKyDgEOT+EE1AP8Cy(VHstNhj}Ao zFT}wWg5?SjyrPaMC6o&=K#gVY0ybAfM%iB&4Od1v{w6*a;mHBzh^hws^3iS4mHx&t zLKN1U5nB}?EUzLTG*pobwJg20;b;79zmzS9dKz4|qXxdAkS<1BbOYtwY=a7Ji5$ci z?py(lK~PT?o$&b*niar~OsuRaWYz%UE8mH!R{Z|ahd`Xsh^6m9F8(98LLu?^r7Y4p zJVN9?M-SNMb_repWiYjeg@uR8|Lk;(5}Y1SiWe80jTR9$4SAQld5+T`_+N)k#EoPsQbrwyYjI%y`_|S0vFgE!C z7dN*&oKO2?l;IVF7{hMybG3tmgDGYq#0OXZ?Z=P0FqTsFg=0AgH2JuAX^*Ag@rTHs z0mc(1jrz+;g#*>BBP(9;%>Q`gKo#J{kc=|=^0)b7CXHejN9agZ+IBl&Ho^bHv_qo& z<%Y4TDKQyk$MK$e^pK+`)gnR8fh_bYu7NKTNJM`_7T$`cRDCI<5`6t;sG1-kA*l)e za2ygqA$dhQ^j95K71@b7X0t7`nLiqrAlgd;1i1=<2k@n%kcJfh+zJHUOu8pDjIDnV z!_`l&$1c0?B8I6VXtq%D{5eU`KwnTxH!SC=-QR+fDyVR;=){QYnv|X3-G7dD`94_& zm;lRO#X_fbNpgMkG4)`k$!h*oBDLWR#Y5X4cP~Nae~QXP{AzzH6SM-zw-!v?fkdLE z-@JD>wcy%(!C8mpc|YKe)6~`u=8VV82j69qqxKfhjvOMVJ^XEu%*SeJ#3D!?<4qqOY)*8QHU>hzHD8__>}cEJZ5i^e|)^!e;SsS#9FEf`CE7K5R$ z9fRNt0Pu|G1U=lqVHk-R45fl&!4~lJk1f;PmK~dfGOdM*%}ESIgjL>hJV9(LCe@)* zZpDPqBJW^VB=XDHts;WXeZ)RK-4`ERSXhY8VdOF}{ygGZ#%$8616M~Lo76qA9CAZ5 z!_@rwzIa`r(cz_EF`y?lC*IW!j{!%vDkbP7YeB$`8FUJ^I3DJHdkd(RNW09U2ITjm0Lqgg>zq3;@!iw5}_6tVQ#VS-xr$PGrET zhm$(UZlAKx9z^TIQ6okF43nA!sFqmx9d?5tWcUUTb$o2B9>su%o|8&g4*306W&tw! zgA_gh@5e(<2q`z)b4DX%j~z1ymD9l?g8Qt86Hp9zz+dqDCtFj_{^~GKJv^F08idKa z;7BY}qISBWS!+sof&H-AGP8gTkd5sJZDfHa)qysBCSToVFvk-YT?5ft(J4(>@yRuWa}`c>cF+HPblIGW5uh4Y?l~ zs{s{6jyM$1?Krz;-DflFB%}8!Gz_(80uJ71IY>&P_xMs?-MloY6IMeE+r>5BO}k@Q zvo*p)^kTzJ)y7r*HDaHHwfZsTBXScDJ?mRcIUwnCL0Lz)CK@d88gV}%wDt_>xnIP} zI>CoJEE&Ld&?aWR8JX6iEgP|b8~>OqG*%jx?+zr0Z$en<@2b?x^T+M=Jehbk7JdXgg? z^fYrvp~vCJFU%cLg_qj0t3VUxwxz3hbn!QA8skA8+O zhEr$_a_1W25lRHjB2ds339Kgy-8nsJU`SPTRW2$JuejTSM8t*eNTPKSsEfQME=5k# zNr;e*0>L++_mm9h37f0$!}2x5`}rJ4(h>l72nIqUBPK**N5SohS{Z$P?d2u&9vZzG zv>G2*6ni}Nwt&!^5O0BPDh4%h63v$AW{X%j&2&SY*;qP5fGnc0vDv8dp0a{5#1Z+p zDg+SXwnuJ9pabe|?+>75H;~banRLp?R$)`$^=8C4gd;ph;&?amY9>5{@@_Gt5-Lo3 zIny7Y=4*0~P3j|^2uvU5_(rgVmIqh+RPW z@^RDxv7-&daC+u>F?VOAr>A>aIRmiS z&%${IIOC7u`X4v}u1J9J*|@4Xx1laa-FMS$w4<%>9x^e;Gx_)EM&0b{xpPOeWozMX zrTgV0IwfaYI_!G5&9@Kgof;C)*qvQ_f^pL=?PF1Dj2j~hLw@{tkUBjP9?DVh?a+@t zOgm;RV?&A~ez=tE|2bb={3BPx8>8vc^77@7^)K1jG!&?AkbcRn_6B4&-F84t^vCe9 z@~JotK_vHuXB)5B*z7x3w#ueLd+HB(cuvj+YD4-{1X$Gs3Y~4%Bm0GV+{AK zSh-$J>m2O8>w9bt;#4oRA@n}`ILlux1$O3>S8mC8V2md#E1Nd@I5CkQqKMav7ZSf` zW-_k)VyR58R+Pb%a!7_Y*WSSh9T z_3ME^BR%4AUUDlSPPvn%=k42x8dB`CXU?0L7~-(ekm4#3_0?>Wk5|NccVM+Co=rDg{`8Z=|?;cr1YzMfpb5OHiJ>K)dPF(EWT?$gM)FT?SER2}N5+CnLdC_6i8a zmT%v_QQTj@mVJES^7R$Bm@Y$i0wheQ`0MVigxID1IeXvv<*{Y65ANMtO)*IB!ywW3 z(W8nHBMXarh{=u9U5H(+C5n54Ky_Hl7m~}?J(T4*IqT4I=en|N@2XDpwk4FAnVGiy zE6+Q89%&R~z-EXvv{(SJDg*E{HMpa7Cd+hv=UQ^Z08))lHO_6?vlNI<9 z9`JtwbJH{l2Ze?{7KPQ2#C%_0AL&s~!FqcnC-0$rMrRnEoQyrf&bP-0#vsLJMp*+e z5Q`a1s`++BdrexlLin>~)mR2u$`f&!=J@4;pbVw>YXijhr;M9`{&^**djbsIcCoX2 zY*bSPN?U?3uE!DR)hq|rQPN;yn9Ng)2f1RElU&@z$%%GA%O}Op+s3qRU8#&J{jH@$ zpbw;@7xim}n@huqUO+c*9fCR9BfboCm|KX6C7=15>6G>`Mp?2-T}x8Sw*aQt9jTI# zf+kVc-oBNhFLDWeNtowPmf&B)EvOXQvjOm%)l=!#a0Du9>d>KzZSwN+65gk?d*@CX zw}t6lVG#!|bFA^ELpk1rQoN!xm$S3PO=^wzCNCc!IRk?ol;T6xAaJItvR^S~yU>U| zlXG$sUqgQm6npIROPWoambwFYC0ZVN4z1~Q&jsB0Q8hKjy}EgnvuDp5pFjVFNKq@< zG5BJdE`q5?bdR5cfsql-s#UA9>%x3}-H}Xzb#XX#lErxBC3p4{Nmp;2KzvnMNegq) zo0^(EHW5ut8e~E@IyUwaDJ@*A-v?d9FFbq=rN1eD1Ht7^>>8H~3JRcG98&UN1;=-i z*^r!VT*auhkPcZzW0THN1x`i`hdj@2{0R)yje(%>zJHeMgoXwafPVWF@fkC`7`L2-T z#1E};c6O#n+uMu69eCn~OfYUCUgJXdf%}@ee6@(WyUQX7-5);yef`b5cg8=txw(l+ z$*+_DiVw_)+iK zt_hrJRJb9Nbpmts>V>P|+FY?o)aC|^-d8h=-g6CR(%Vk`C!M_E|40jX28Jt-5rC-i$v>m#a2{jnsTHnyX*wWG>C&^DoqXeXXDl8qd zW<$+zRAmedPY?a1KdOq-4^SJm@%OgnVS0EAfauWBP#UlYM{`g^W8)akB%MR;0Hcxs zr8pVmt&a7>%;+Xz4j(drj)L}NCMKGD9o|lD?(Xefe)#ZVinO-&7Qk*ftV#9t<*R%H z0%$1x4bN5qxnEQjeuv0AUH1h>kc5Z5oPP&ULc#Cf4!>t-X+Eo%{VB_GN8vJwf^JCp zEvg9@?DHe_GF!K9rK6)W_=&}fJ+gkik&#hOI6o6Ty|k%UUtL5uPKgZ_6~J2~%)_zY zpp+u&N(}BfMzV!C^qOH1pus1HPOr3koN@bhA2_NqKlv$ej*gDm%h?pig`m8L3tD#x$Afko^#aLLv^QhvL+q-&bU`2JzaKJtSuJ0ql{s z&6D#5$1PJ&T7;6z>FgmAL6Y`}zUX3&e^lW}+l4MOAudR{mX}{jpu? z8`i;Fj%mNh8t)Zh;N4*#A?_*OT^~gbCj3eCwCrp-G-^~L`7Q3hF%vVDu3lexS%FH8 zpD5`DLr8i90~gJ^ckfJl@7zOmHy8Ll)V^OlSgH)dC6XfjfsM}o}8{O3q+%RI7v)k1BLB6;D>s? z+m{x5DBTBfUM4ZHONFOxnE{%#f&$!b79CkD35XC#!0D%S#PP zkvYl{SG)^5$`T6R=P{r}`_~fsa-b{RgD;QomsnM=jm4+p@EVoieI8m5J!fGdsE3=J zGUs&?)1GY)>HR}Oyoe)a=F@t3BgdU~eEm=xh>wfUP&(+Zz)81ejT}T1L9oVU%a$<> zc&o3+H`|5nQq~K6DcG2DMQsuW#>?P?K5cF~00vL}Wg^|v(=Vq_DJv_NqtIxKvy(G# zq`cXblv$!SDW8PCn?1{V&X1+n@f{#tx~`XyY5eK~yb-zZP9UDv&bDLN-8R@!TYF+q z?}`qa1%wEuRUQsbPKF={e7QJ@nt8GJE^%dJop>@Lt9xxGhC5FgJvz+G>-8%jP=Ql} z}P9htMRQ_ z(?dx`r4k2@wPR(0zOW6H(s5pZrQ7-W2Pr%F_`ElEprG*0^5yH^hdM$-0lrFXsG*um zMw`#d=R!ZBxIcNqx(TN(dM1IY0v%B;|A3r*7|FDCbYyG}dj5P@@!YtQ&*@1#OB(-U zplAy z#}^aYh}&%odIDS3OPr0@y63s!(fEhLs{*L)7ghi6N&DpGH!hFz%v#>ytlXDpcC^1Lkci=)YQ2sxo@Pl*0+Oh=| zrVL$+AD4!;=n^;@Wsa(J?AS3KUJkCL1(hXt#>Lb#)NkicYM6nMXnE=7D~@uUO<}d#wPMfS=UD`_xJa|1WPQX6ff(ju-f36zma}< z;tu-JgYIynEuj!Urdk#CjHz9kSNDO*sI73DbZ~aTqoh%NgBZl(PG@pc$T@^On_haL zNYA{y17twVH@_)S2y}>4a7RABr0LzecWcbZtZEft?~O!w)2&_WiQq!R23i!vY~sLy z1EpnUO95M6!W;);ptZNI1SL-NFqSYUl{+*M>UxrkiI&!;K&M@QHCHuR@`zB%Y(Z@( zm%1y~U>q0?hs!=u(F&mHbX02kl~FOzo_)d8fl+PWcTC9OM6DV3Aa3iUQhdjyJadasbE=(mi_%!5(Wc!Nq!OJYdK)}WxEV>e`~ zyD&aKh_~2GcJR-sa0gt*xa9+GHXmgF3jM9GMG^b=?!6ySWz0fB$+UrHk;NhV6(arl z)Nr_a>)roiuyiN+_w4Z*#6UK^awp{ycB2=d?Inx}m!UGAUl1cS?rAL>amv36uBjwCcGpnDyoZ_ zLx0c|5SPW-+>4E#qIvY>0P~4cr*dBppP5cYM14T!mck$q3ZJ9n^S;i^3k=RL(67yE z4yc3(-2#005~Ie816&CrZ^;X_T-8eLR3CJ6eNVa-%xajc5U$-$M@J$R+K9fmK44`rhkVL`GUe z70e-#&z~RjQ(NEPp1BJyGCGRFi4!IptLS)tEOSv%phLnoebmbf4Tz7&Nl}S3NWfQ5 zm@dLem2~qwVyr&zvXRbujn0B)TUl=EUnrK-(euS!6i7h)5thCM0rtAVR)L`*R}C(eO)O zzFdOyww0*;8hoq|J<}L9;G@VkNdj>p)q|6pn*)P8esLMnY%J$6lniXB)i{Ic!ZJv@ zu8EV$@G`JH641YoF%WT%!;eY04?IPqEp=$5hef-hn=wD^r1}`|KJD%t~8}rYqs4!qoR0`^u2L7~#nYm<4FigbU6R-Bd zxGwvRWo&FL5wVbc^cd6yhlVD682LudeTv4-9p?)yYi-?((IWqNE-_nx#mL~dvL{8j zpVcHeKzHQ(2|j-mfLi2QJ{IsZrjOrn)K`W7%x>l^4R18TjA=mnhNr&tvro6Mu;^Ok zQF*wymc#TGEPRoX>?``$yHJr;P_~lZ{S2dHV4yYgD^_GU(u;aM zK%AIs21Z2a_qZ&H--7nBNAN>TjK%!d|Mk#d0I-Sv`q5v%AQQ}-k%}1QbO4;pHhJ9G z(9o83oV(F`QW=m+7H3=9L|Hd)KE|xOoKHEAN_D(;1z^gwDQ-PEYfPSnTp#Rza%$vL zNnI=|n&J%sn=(6=4I7TqsCf_=VwgwmhX`G6RoA;w7!Yuam=y~+=w+M%o`a*M4N}Hl zy%%@z!VyEnBy>r>W}_UlW_svz7!Q;~JW0c`$eTBB0^zYj;palVB$y8+__3txHuwzW zWo4Jm&CNYcP1Wxi!_lO`^0wYaJI$rB%E5$ZSeog}$M-*)|F)^`D`u6bg z<9lw4^P4G8qodV1^N$Nhj2fh6&M#U2+Di8}Ga7&Dgx^q*$aZ0STvb0PV zG1{GU{N-g93k!=Z!+c)K$^4H15feGhk8l1MhJPW_qcv(+TvYVYikRzx+#e?Pc3wGs zu{CA;3I^8IaN*m2Z(0dK-zl}y{6n4~IIwR##E6 z0`4|4L{i0>0;h9J-bb?Y=t)acR^58oJ8mH~l>w&YGkm{MQ*$kk$y>AQh;o-MeZ7uf}l0~6bFG^E?mMZn!Y_xIDqYj_jU zu{|E)I^nrjW@Ka}82yp5@bsLX-V^y3C5(UR%keDLm&nn8OPK4aEf!~l24qA9WM_+z z&lwrHPxP~cf{}B@2^t?R8%0e&4i8U9KM2dvp+;pzVd@ zel%M_*!AnP%F=UkOgincOt5($VLr2b|Ni~2(7OcTpO`98)jwT>(>iRBLeSZMkG^6M zv}oHGcsCAAMdij`WK7bU=Dvg|s7^E#4A#a^l%ytoiy4w`hMJvMScs$d%ZUQ{Q%=YG zx^np)FRvqph8$si2kvKA@XpD%!-Oc|w!k8wU;G+#^u)Q0l}g{#(x@)T6_}A8U}B!Z z(2dzzDHi?WC52ldQf`vOKCV{w+jZ@M-TK8Wz~^8^((>{v3LlIn9OPg{R2>1*#!KaB zUO4@tM-ftYUKjFraatb=ICKpsu>-Z#e#o4zxrh*d-`aXRJbXKbe)mDuqajmMN)xCw z(ilHwo7}@8i8RxC4)`9I0VG#KSCTDyfj3sN+uGUr1qI!P48txi9=cp#p&X0ChVkR) z!NFO*2hmC2vhWIm{{H^bmX=LGnPtf9_T_;U6%|p3E^k}6jsj7S$cEbKOW_iF4FnOH zn79fr2J9agNN;(J{T-UZYy9rPu$pGChdP4?P2;JM-QupDZ^MOCyj12;x=Sp#!In@p zjDS{r6*z}-$|4VQNCHS&;O#iy!xL!zGKgNR`@&(Yh^-Iya-)sma5LUV=Q6|`vcEG| z;MT_MJ6g11C>`l|e-5fKTJciM9FuOTeYw^STGfwu@T>d{$H3^^^c6~IeUJ!J_Sec3 z09yEqrO$ZKAyG07)J;u!R!H|a=KP)Ws! zm)x$du34zH(4=n5M3w@_j==={-jkgJ!;d?6?ntAuf*HQPW`#Q{IKO7~>YK70=aPSu z?u~36`c%S(F!Av6T4_FQWM)PWngLw!6+?ifm6wO6$P_lzD5H#W6WCBIn=MI^iGDdK ziB!z<=RpJAikyt(;y9FbuK6hx{i8HUgqBbs-t!w5?;J3`aG@LZkq~8s2+1-Wt_aGp z-1D^gWh{ybIb!Rp_zats=RCESpI!IB81zl0{=vb-x#N3-jKK3LzilevFiRYN>+}o@Lv&^L22^V@Nu*sv0t~Al(0X1qBT#Q?D+w5X-3AbV^e39i9^D!SCP8-oCvn zqs-CU({mGvL-RQ`D%y3+--1pq4u+oE8~E18$43w^+=J4w8RT+we0(Vd!TJ=?SZ9AR zPCXCxT!PjP$RPWvpsKcZEf$-$RNwfy^2UOMsHCLby66c6JI~|^g{%f6e~oiqlu`E0 z=>tfEYty}(d6?*Pwu6R}_=h#^2H4svQC=o`0P160tGNOdBUO1u@otto--46&QNDcr zx)d)}>3GAZZDPWWp|q+=BsX*jbo?9H50lRpiaHln8w;78d$#;$7aJ*78^ickR$c~K z}0R?Gg56|Np$)p^r_o&`Yc5U0qCa$Q+jpJa??oxsXK70fJ zzY!pn1mZ(-YFN78h31h5)jI%)5On{PLsCQS&CRtKKAHZpN5yG#_IqqyHp<9Tpt~%B zn6J5aCr-TQCGA%&uU`j0f9@N#OEF6TKu`4*UzkYLzIn45%@7ScK`0U-P;Elr(9dBU zrdi)Q@%7s`jYEevVoW7|te_TCCDM$DH54uu5#~Q3Dv*;r;H=lsgtLd>MoA*z*1tR1 z{@6mnDMq$W(bYxu3s7&0BMMbt@iOQ&_h~E}hzSY6Bd~Y3%gM2$o4J9iW;4UTBIikL z`cC_-Zx?OPo_%xs5OEgA8WQi!^mzhH)!G-Il$0cYX-){9ZFCkE7I7cZ#6Xm?@%AX8 zE*Grf0Xz=u1(k=E%;0FqkII=#)J7h$>DddpjMhdrOa`0FLuEFDj%!kE|hbUvK6FvqzGkbMIQ1tQb7 zZCe71*gFd?_7T%J6uSkUL{;n#Te*nE-Ax|6#3}naH96@-&$ws*{{8Qa3@Zm{#oDjD zettbU*fa@M#ZAe1KNwcYeMaxjeV(Zse+ztICN?(e+oOzl3K&~=)&5JXVk0Lt^xfzb zSPz+-n!DCzV`fc&ZJ;lNXbRSqId>OBo_1kw@Y5$=j4OGYa(Qc_+Q!VQ{S*+pZ>+5N znIAts44xezlK93*%m+88WZM{z`J@^c59jHpDK=ld8y zjo!dO!n2k{Dgb4CI-L$4U7I$1dE6>k3ZR z$!j%Xer(O8ZPMS@N1&pGzI1X-|1A*33d+*6D+*gpqJg}~eFmr9Fv1bw5%dIffhq%~ zFakg~0p$y&)4KvLpt_`)rVFv%Aa4+-N#|xcW}Ub~G=xBHaSzj@)5Fh1*dCOn@3gh? zfF!ZFtjzDnx2|%Mrj8DsKMa0&!FHRP!pxTFM~OBMLM_AF6x7?vRtUf-OBvA&j;x|F ziNvHg9mGt;mWBOhu1v?QVLd_FlKMZ#kB!-Ey7zl16Qf>P#3CUhIPbg!eT|8k`6`;s!&yeZe}3i}yt2Ue zBjO%LzYzA&`1|_ z*kBYfr_f|KzkR#w0)MLqFY;9o0P0~-rR$L!I$?BSXxf=Z(tJf0>&blZ;2LnKJ7?Q2 zT)1FAcOvGn5GvtAfJfzTYv2SB3U0$uu$3b*Qhm|wuYKHp7QZA*c^^E^iosC#gSd@b z)(Rbk>j@BH^UCPOiX=7f?}3)uI-vaIM= zaY1cE04R3da2S)>{_o{N|M$)Yk{Fkx%>m&5Og1>71sGKLVsIBJSnecXUBifc6piQ) zgSqxP4B?{_g1g;XUyoL!G( z_w2D{yJlo$WODoXojF6J970>+?%`o8a;R-WJ{^@q%CQSIj4^osuTxd{!C9rxx1_`e z0(b{KPQ<-KP|55kCTCaHRu`<%W<9{93m;}93?nr6>|_nhk)Kf4w>ORZu8%_W29Xtp zUbZB&yDGX5kzj@~@cp8$@o?^L%wj`8wh=rq+6&TQQQ)xFqHEzQIw+5C(17bY2r}lASn?P z6Oi$Th`xg&4P&_Qoqd@_MO%05*m1!34+AWu4(sZ&d=6qadfF3`+?qi)#y$g2bJ53O z4epka>FIk+&%jV{C%^jEty?pMByUolw&wh%qU*x{eP2460F_+m>^k$x(Fw^}_ zMNs37H&y_homp6D!bg1!rUsDgJA5h;k&%uIu<$Zns>0>9MV+7FdR+Jb?fs-AZ5~?% zV-6Q`KJiPIALrlXNtqueEcPeja^>su1yJzI{(VVUarBiRzfb&H zl)h%Co3t0Lg#EVe`1eGs@Fk#{NH{~}&!ZCIPrVI+lzKqh`>Fa0503FGxW&}fZ`Q$$ zL#}B;&gnoO-+4R4U5VbpEUM%ogDk9xnNe?$e-Tg9w^&R`>0U?@?jjM^Fkv8gXCxD; z+eZN#B{I*=kFH+zW z)Kj@YD1t_TJRH{1A)8WXLuq6SRM|k36<}OFrVYCDFF@Vm?!Rox87?a;n-x*aTQLRo z4;{YtKGfPx5|f=E1cAt|D`Epc71y0O%GZca6`!j;)&6upmWqx`H3Qs7%5v! zsgdI{aF^J)CGr{SS)L;$l(>P%>w$gdL&F+rXUsa!y~i5|D*33VCLM|}*S~oF+zPiZU%s66NJjWm02ns4 zA3mHAAJ7F9*p&UMhrJ^2(&CH&^bmmr?JrL{iVh?!$z`MMev~YfX!ef1vIk>di+LVJ zjFx{>cQ*@(z|;`u!04eEI+|8w{#f1Qhh^n`@hSR+)n%YmL*F*`5YP0c7gvog z;K;C=#$$eo9n{wEV##AJ%^SA{c0XLYVt$Oh*p0uC^&k4CiGhGR43EgG(~S2!r1ZzA z^u3nm>_JrDg#HXws^RkjjJkk&Heq@=G%*nlI(|Bk{|ug+s88hz9~iehH2vZ^)Fr$& zBF@8O7*UF64R+W^9bd3IHi(Oh6K|2kALuk#UFFnRftsn(>)RHLM~TsM+(*RwKDk{{ zhfPkDeIFmUfvj^84DebH%Pf;mVp8=a&5&SaPtU+o$Dq-sr2uIzMpjL?nK3=ibEweL zDVP+770YwS7g{D={dsE6()MIr2AcgSt>JxRS zctbE19!1u^(_%-CONI<27`cZ%>8tz}z*mic#5({*fhR|UYMT8`;->drS!Miwb5!z| zqE)OsCRlg-$vw@3irTt^su!ermhN4em;xQDph}}-Annx;chy1Aibil{J{fK~kqDmyzluQ_~ z2*d?l)juI5B^9ZEQT0DO3I z>S=d(cfU-N2l>~j`I%vj^1DDtcgxC3=JQ>5G0lFjroeXg!;F9kovi65G=6e?OiUCZ z>H^6Z!ZmxrN-(z}QZHeR(ytO$Lhha&U0MEpZ$EWKZ+_E_z{%&HUV_>ke&QZ9n0W7`DDH6HxtSSNOtpyVp}>IyysCB7&KSb6=@*M(0H~i}!SkrzG>NLcMejaCf%Vb! zY)_$r4#JZT6IFDyoLpQ2N=kg5Pc365h3Q!?tT5784`_RsCiQHU3rH=YGm6?*Qd}H} zGCevbW)|TTC{WQU^3okT4lb^iU4L=dz5DN(iCV^Kf0jFvzEi!GjXI?)lv7ve$M^4S zJTBmJX}s9({ve_*TY59~r<(Q}f#oHNhev;z@x5r78LoYM4`;)^yh_~O(ecdhlClR| z%h{f9^_>YEbQ7b5nP~OWMbRG3|2~mS&Yf53cz`dm^_C#?*8vcLV-reGBzJz`JFm#! z#6KDXS(AhP&S|0t#G7mcy>NR%Tjz^QUY%4sQ1zLca`5rpD>>6-YGk_8TeQbS05jzN z-{=|29G*tpx_d{~Oq;16E!d z=Fa2_G`AY*27#mU$SjXjVVw$4mIH**29T77CnpPL<1#~jm>*xQ>+4%w{lJlxITN=n zmg5dxW3eX7PJ^7>t7T4|8-Otq)fY1hi^uxf6>|%VGp|a35{<%Wy(8KC3W%d}tH&rp z`^>A8PpFctPfoqM*`;LL7QgZbRm)kYxHOcJ5`B=4(m`&^&B_G3F^BG~T^9sis6LbV z=yOlePc=zhKR8xghu4YeEIl>V#*G^P=^oHY)OU71?sEr`gAO&l$g@3+*EgU7A6%iI zHj2~cLpbuZadO)9U*HQAYyeqcv!w+w?HR7KjoCM!V)nB}$M`=#`tONdB6wqRtC*B@ zN~R*4q37=zh`)2ER$W=!MO^_+g%$yd$!=Si9yELYb1+<;Lh~Ryua{Cl^%q=fy+lwF zHPf@{1OXDlm55ixpBMSBBOc+Q<;|N{z=Qi*fp$~~ zKyXo_>?1V5h7ol<8WhtO|NZ_0k-|bk#qTRXoY@>6O1ogIRsZn6|M7p{n7)39Fqu3i z>3iY?cgnl?@DKIguBT7``fBa-9$!gRLx0*>I6U?QBA9ux7(Oj3?<_T@BuwM&92_=~ z2)za%pqYIK{9l{d&n*O?@IrLU)hVv1xbgV$9^gPxiHR6NY&xH|J3NONZBO9@zoY*6 zq~rN9d!rk8y(XeM+j4d!P##T%pr6vdSL!+4Znoc(VB|g)jfZ#D7D(+>;RrxXQ&-=M zJOF&1Uvi7LDIKRr%U=Zu19*8T(iR;?+fgBfGcRWkzRE&B7u=HT3itZc)6Z+U-)e!Dggk&bK6V6ByIn z=l@$u3b@(P4N(P(lX&kdYpJVm0Cbka)>aKIMuFv1=I`IXC&J6=n1mOgyOi=LPsT*< zE02jQ6&$Dh^n&(<^1+i1lT$&L=L1s%vNu0s?mK?GuC0HMyRbPEkB~|m~<D*VEO^MEa1^$pK$)0;{I%!pW-HKPLF_&`=7{wA3Jn~Ig!V@rf;!D7 z)zd_wf2PcdLc)qce*Z7G5%~6&#~mT5?i{Ugf1j(d%xZKF6VaK$AFJA*3AZszzbl5W@Gp2rm69yud=+|w zGriB)X;Ktr&d{vPQ1UyGe~jjzEM`3>!dL?eKGC7UzpmdOLP83 zj7jUp5$U~ReI_9W#k4~fj;r$=ckaFp;2M@!&g?9Gn8RKQaxQ_7U<@AxQbt)>+0{RG z+!c|Rf$&nrr2@OyjG*xfh`0yv==aNfNbb~D)qY{P?SOzblNkB=BMDFoQ&SY%eeii0 zw|ZCKaRbP?b&%`0!P;Q?+1k}*KWDpufa-apb3d1r(kK`Kg&v7O|yjxj;~1vZ)ev1 zBTmj8^j%+2S;54I6o_i$6v>o=Jqy%LArTQu5*{eExP?X972oP;BUKW=`?JezhmT5q z6e%yIyv{Hz)hNi;4svvSyhL9aG|>g%YN3mikrN~)CE1@hZV7%=x;Ma3zSeAF6Rnv1 z#kuN}Cyk5(Yu0J5UZbF==U&lqAlbkTu0Z}OT(fho5e|7x=(thu(4Qy`^Lu`KRZ!rPe-74R>LJQla^B+D5R_FxlXK%8MLJ5X|8q za@>d5{eHQOK(WBS~73C1!+9*NxVvb{mpcm?P?#LWIgw<`ItM1twbEqpwW>CXar7R z&IX!qi>ZonyF7Qr8+Q*}bANFan7XwQ4x+FKVULPHmdzW$zVLii_zSji4P z^{Ne6r)W!{+`5f*^2d%HORCRz9@ZVDyl7~s`r{o+M}z0(ImcpRt#ZlJ4tZv(=cji) z(<-Z|aO_UDoNB18B>_3xKoYf@wghC&PcSZcn^5R+LK$-dtn%f1z@h6tf98Y5Pc2Z8 zbRgrneqqu*R3yA;`|hd1vhG4dO81N3V%lbBkv9hy`d?A|oE)1dw0D?`G_~9mbZ4Oh zwKwcDwvuK5!nqt0IfahWrW@G$t3gBS)Il>zi^hPNTVcZs_UF#2rn6_d2_^h8PP)#cBNasZN23)?q_p#4h0O(pLC)O-uEC=F@F?Y=AEnH>m7hne-S`LB zgj7=K7maSXgkK4~;GKx%KE8dq2(t!<33JBT;SOSkB1ppV92kNXz zWpGb8Wt3?_+cUOe&XI6Xj>8bl)3RhQQa)!evqJ3dkbIrVblNaP6hR-xwOY>ipEQRMEHU z@~hMuv#-jyY=7sqD#Atkg!~?pja`%foRSQ|h^FrKYX%Y`a5yYD!cRTLe2fC~K@+KA z%_qPKfN4-cZT~l+boZp;sjx3y(T5kUGD;mN($iIS=!SmK`&`Z|a*t1<{XKnjc&AZz zuvrrVdibMk#&qa0OS>Ke9<;d0URyD^LFDL$++07YDKX>4t(QVjW$ZX)R2lZ) z(){pa@MCJTS65b$@es@5;<>{WhNM;J<4%SfS0)>quoj=9=0BfvtZ;GgV@0G2P5v+k z`Ai1jxl(7~uAi!W6QElsyrf+(!`mRoI$q?;kN4lCVn5ZS7OgR})4aaFx$h%K;hN{S zE!MQb_4;wycmNR3)1+7z+faCl#%6dB^e%z|zXag@G1+%o%lzI{3(L@JuAwIWrC$$T znR+Q+VBl2dZGMx*_8|A_wWT{LM#t}4OSugXQ&afx@xA%`QCH3Pn*PG{uQ1)LTGifr zohf-5nVB>E$~{$`V_)Y?1(;d#Kb_JQVvaKi1kNy(qd5*~oNc%8pHP46Cub-%ZfvMo z-y$T0AkAL*H8zR1@+lDnlJMS%qM8H^$aP>6+^kVzenZ8UoSZXtZSy~tqGu-Z6!3%p z9wsg5;Z!%@b1Vysh>T(Y42dMcOoCpbWC_tSzMbWXGbO|Vsi}vxDYo!!*m;DCTb6v; znxEfh(wuIeX)&^UZEr6d3I3vyco>kPrh(B{9G{N(l?dOwC8Z?2#)^j)FE}ebL)?_i zD*toNKV#gDxjZFF70yj?Ojf^^VGHkwptARcFNro2%$s3LdWL8%mOjY z;!q6Cy01j?4p(^5?QiR2hA0HCl`GL*#*?C&d zJy+fIyUWB<)5@QWiSN_YWc3qp>Y{S$D;}Tr;w`qPx3PcqP}ZWHYU15kTcd;HubLW9 zVH;P5GM85>*PqB1Uv!J(rXfd3oP+Zmq3a}(0E3KM8TK{_F{ho}Ki_-T<$X4)6k`-& zj@4}>#sq>OWFt!Kg(rp_ z7L|ozJ;kw4G5Psl@jF!`xO?IsF@OK@gMeskei6!41VtwsnFidsS#Z*GCm*(Dp6E_V zOracYoc%_BUs*EetDR$0sz{c*ac`rCpYBkq|HwkyV8GT-m316HeP*xLMCvt~9k`sb z#b9{YZ~V9vrR1*+*bMGxuH(*jqjhw8;7uxL;RXoOXh7BdGB$?U5S$0MZ0@Cg8hQ}wASRF}C`o|u zIJcO@#mv5cH0w44Y1WPPO_#Pp4?qmENY!ubuzTW_pOf&|Pejaut>p;1&|dZjJ<#!5 z*yTRK3S!f&w{N4VKS-{a~Whg-!&rj-(!b6b`{&*WnL!gE>BG-y9Y`we` zt)k$)BER0=pHxKJ5!cn4w;i3g1Bdw*I0I*m$mf0tZsNg9xQ)D@=XGRq_B8OPp$Cyp zTBcX;-sL4=Hk7-hKvb!xp%<6g&0YI{QJ;+@=q*QK(ij)8ZTP!!b?voO+`#^7zK-q( zai*rh*XK<4&{xF99bUZR?&DL}e(Vud_41>|roYQ>)gxbjsPE7MzF^(2yLfNAyV`jO2$){a(Py5y%&$tF%$ zud8vL=&oA)g^Tj-cOc(QKv0FpLHT1oI z%-y?QP`GsWuur8+NcPAx0egoH8aL+HFj*$d zEpqHjLxRumKf8R?#OBA-|8l0MQ)@q6UyYQa|2p1m<~-366hJ|~m5urs%VsZXfl0N*&-JGB1 z3e>nk=Pzfs=}zJ%Ugg&+13y}C56KU`lT_upi`DN^$&{oY?|-EW3+J+Kq9gzH6FHJY z6X$w%Y29~$=8@&_`fbXMf&Qy>o@6>|Yr8MC6l>P1Bt~9|WK*%9Hr>@%sN^B zI<)owrnhqkZ*fu>8yk~UVW6kOMv-xxs6N2$D7|Fs6mbXgnvk=3pFyj0QBc^aGja{qCNC5 zb54|Nc>KwYl#fld4q&$-mzIFsHc6 ze2R4E-kSwk6YPApv|RjtZR67J?qXZNrQ-L_gLxUK@5#MKfMp%pzi;ybdQs5bQ{NoT zIc2M;-g*KmVRL5dT?yU(dN08bw*kQKx&}Y&jQ^)lX9C4|2JRM*FJaRVOjbH;oze(; zJR&kF5Xb*CkQcbYKU> zP3)NeL9MM|fAuJrpGzNOUnNuD^Ms=>P1Wl2^_gNiXa_i+K^;@=D2WoRaXdkQaU_UZ(7#+~(4ir`Xrl(+ zcXmcihFx)>|M!-@o4^E+;A8!rMP@ABNDWF zg#Um}R&?x?;FFd#T;yzA$)T@tLcwb!aD}@Vq@$SaT5^~yb3mG zzQ{pXKbh!2WleLy9A0$=YnSjb zL1WP!n3UcQL;*hWBvFfU32@Eh)R>N61>K}^xeq{VC72ludNiXLoUR!eWoXjE8pqDy zUrBE#UtV7B2Y*Ns=3gU|lM>gW)v#nO6?j2De|zGdYj`#-wV|49yDZH|SD@C|2T#qY zAnlDvzu)1xv}gL95lFTC%avU!>VE1F#PkE;R($>*-56w;cNt#qb8vHO$Mlc_rdzgP zr?KKU(Hs<-nmq2ac0jrIaMsm$NtW{G=V`5Kj`_cPcm0|7Xuj90{%5kvK8G!MurB;S z(iTy!jBP6KERW@}N}OP|9{v%jGBT_yySo*5~H2@W;&l*mRrsz%jy61j^yq1E6AcY&1+EE!qD@Y5jz9x!)=t68Mi%J zDJU!~I4W_6EJWpZWpw#i@%F@Ihl$V0m-fn46`Ai5H4&Y?Ci`=1a$@45^!X~^m+dMq zpG!%HJ#(k1uOHc(*C4ov7%#2Lr+oL(jWfien(TL?NUuFA%P~mdh86OLy?+TczGFAv7BiUvDW_o9Yu4(F&G-1KqJLh@7^72?aTB6 znPwDJ8IsePGo^4$n4k9)Q2x5gGl}{ETG==l$2QmcQIg}r!4dU0RY{5&jVM6h`M zP90O~S*TC2HtED@pW>ytOa?t}(A7?3pVdSYaMNf8SHVkbtNWD2qs!$IlbmT3Uuowl zzK`1BK1K%$s(Z?bC(-_&uPn=0S6n#jqw>73B_>(>;)f^Qepe2gTkYDl2NtV=4;}rU z8$Xmu{c%!|M>Zg!uUyi2>fLLuzTaycuUz{9uduj%`<|QB-%`Kf+^4`NXX1KJdyGw5 zp0sUb7z+&h<)3MdBS~hHS0szve(w8h+at_lI$_r`&p!zIUi`in3?sO75?JK33IPAR z^s=g*Hu7;67;hx#rHF~H_cPllf7qsn0ME$f*KJhz7bUswm6yvArk##El&}1i!RRMG zNc-Xa1D7&9y3eDqL-x_YP1Lp2&V(=1hMB+fXP``J8{H~0c++Gb#G@#1dn19W7jBVe z8}R}+BLZ%USg}V zMnj|X{MyF2?df=(F`pV*?(^v+99fY~RifNivH0y~SWpn9>_^ldK3(?~_*AwH(!}1< zudCB(1lxLqW#z77){^CPQ(kbeD24gSXHGO4_P-at>Rl?17oA;uxK)Tf>dM*+hn8m< zT5;SrHf|A(^!ASHvEfq;jM#jOk6zXV8zxKq7tX!;hI@+v%3^s~udALTvBGCdsDjVT zb`-qv`>M_#$ujB$54e6X>p?!6GGg;v5YzE^cxQp{IXy7aekRt5_4`qPl{h?tWHKEI zh7K+A|E6q$#stMjW@jYqsA;evi!1SbGmO@ zXF#l~Nh1?+2<{r70;zjfB2`TkWV1(J40h z^vkqp3;)0`WSqTD-(P+?fC&2cyq~+t+q(vjGmQ8o6i+NUI^`7nn(zJeXSH^xh=_2d zuCDlZ)dB`lFB&1W35;Y8)3pZ zL3FQ%-PeTX34%64tXA>|{Otqry`4>0E`kT)+E=r@+z|*sQ#<+Mzr&u?h>{@(p4Vxg z7^Yxy0oIlD?(fa$7CgMXC}SsiF|iUn^yvzBE#TnmBry#-mu&fO>s(Z6A1%qdS-okpsHp!~(b|6Egw9HM`2|1k;`LQ2 z7s(FP^y*2{t?iNgFYhX}w!eEf+qO3dF(-Df$nT$plQVthSysa;>!K} zP~WHg5^?JG@~UG5$~yxz(ZT-SvgN zg-X8JLX}VbmHd-*s9)U_w;vx-F4}*=WQ|`&RJmW@gcaK?mrew%oHewXc#if=a$7tp$`E%%Yn4#eGBkLO+Qnfv- z195)^E#$^(O><-a6k0Snp6}{1GKvnl-dHmbFR2q~LgR_8JeV9F{$x8mF`?5`$zqnt zgIsJs_&l@|lgax~0Oabr(?<2e_c;txj~v~*#Kw{xtIrq92`eV}2P~XShRMbmV%zJ} zp?~rkL1UcrZ5S_2G(qiohc;`l4j+fecOTEQ9bVH{l7SjnL40+axhsYh)P_CQ7hAQ%5X?<$xpHuHfjM zieI`Rt*|g6CWZxqglG!gpxOI-9k;+G%J8U(lwk%HMtx~8@0V5`X=C7~(v5Fu^47aw z(BM&EPEl7kt9IDKeo*m($%eFT$|hIu{`_qG?$AN9{lwaug*LL&1FG%NoAD}SP-(mh z`N+v7#+51|!93kP+GJ2(!0el0q?V=AQN(Zgbi-#1ab`2E=^8tzPG=LpDHc$*N1ApiQa3@-PrTDSY|_t6MtD^uk$sV0cx<@ zXx}&7+2>}3{qJ9{?cnpcaKQmh{Pe*4lHsv2t$PfY3uvQG5UZSV>U+Pn09JR7qG-M9 zK#j8YbldZiBG5BnArM99k^1x>GN!qcSQY`CVWVi$U9?Vu{Xx}HSpPxLz5#xvS*};Z zZv#eO@pVjM4&eF!o*uMT3OWGWU+_gY-po9AKHSdt*VoO2z{z(sdJ75muBYKLBE4jh z8453j(`U}KeXrC}szo84YxsZ|X+A)`aKL9qwdx$m5AecN-TBi$9lJWZP_oP`Q3e7X z_JY?EvF3qm`|OyYU~S{h>atJEo?}jyRZMH}P0z6-IU&vHbw7(U-{3tnHNS5|ISbG1#+{Ce41kKykAhX;-ZZP4LSs#=_{ ztHT(uJ@w_nptv{B-eR;k#;_Zo-{=LD3f8B$UYl!ggsk=;*a|(RN;u2);PBen1TPZ9 zOhlkD!h#IUWe*TdY15V(36&(y;NQoujoAMCR-)M8YkvCNIl>2)LKJ%!!18!t*Agu) z?+k$?%6Ta)D%%=(`}URqSoFeK0~M{9poGLNV4~9SZGd&N%sX({9wTp}zZIaI4}n8Y zBq{}zY^GABB_)JFmbpi$`UG5U;j2r;&(W2$FoHu3VW8rV(JxR9LNhyI4{EI=ifs^E z4$;i*MNtwAEX5u6j`Jm$(v+Y(;nIyV3Rn@dAC_~4G(T?ctfU1w%#DuC=mW139X&^V z1dl;B!!5pip$hHEr+-pl3DK21%BX!Xbz$HW}5 zJ%jn-$S7s0gV?NO2%Q*L2J`Z=fIMgRk&qaXIQhh6Sy3Po zXD-?=5<5?!i|**^imqF_9#DpFe6O|^#whO}y#XV?(^sYW$fk65R#r`PV1#!=TI^Z4 zmmI*Oi?3Uv0SMrX$FOI*>oT!R9<-R|+W$3l%-at5sKUx-5fPM>)zAvToiL9Yw_&ac zL8-x(shkAtj0wfbY#j<^CA?8#Gy)eO11Z8w6w2N!`usMqLI8QdwB)&8HX`R~d;3?V z7n8(59^+KHEQ1&svwUnM{~pYeQ8>h-&K@%T8fJ{8c+C~08~Z7?Z35r@tzhyVH0r+aWjxDwrj zK)468H_wp<*OJkR39_2Lx}PR|1c){Xo(3df2sa5(Hm9L$f4B*;l@c@7%q=c(9@&N= zTx`%kLT;2jtCZDUPy_Qy!!hxy!A`DVs0mOu1%V>xOeG87`$JcgHNy#e=GU)p0c&%j zlBKxCx1Sw$Jj4nyD>;0o*Kgj;{`wVDr+Xc4DAE+;@-$rkB9N|bqF}O^0?R%Ny-;6&KNB}M`C3XaZs}=AGUDRm z6mzsKEK<-UVUZ0TtqZfdkoJZY&1k4ph?mt9iHeRmKXx|Nb^JdOI7jiWkFD2z`TF%B z3<2!jgBOD_zD@!yHLqp}w<;BLX{n#u{~pj-543m~ zyXIX>0X*Ui3-CyOmH0YsEu^r{L>qh97@$_PvuQBH(jQv9#-2wfQ!suOGEY?r@Q2;e z5&UrlSQ;RQn-+mN%u9%DAtXL*7?=6^#iba`?e=0x-n?BeeBM0aCQGcQLi996r?!NH zeEg}Wee}0s%z|%WaZ!UbQkvaHv?0QP;mroSS!M}4YD6;peAeaTSa+gu z`L`$>BcxT;LnEVWcg`9@O#cvXm4r<~&~0-N3uh=}4uT&#j@lY~<|pjoqR~@u+?eK2 zs2yZ%U5gt^pkYKDN`Jde$f*4WhZA%PqDa~V4hkeYn&O|kQq&ls0Y)_d;Do|n-2kst z*vpt~kZXe3kIn<;AP^%7`ZPW~&0)VNz^NdVi&sAJFE^Tu}Kam zCck|Z+ySDXfIC=EQmE$2)93I3!jWsmq#(;`0Hb??m=qEc!VtZ93Jw(y&?`f4kkj^Z zXEsg=NVBUk?S%Cq4#hz%ndoW5=tK=VwBpjzm|r{u#TKuNIPsW4SQ~`{*S%q~+(g5d(5;!eux@#%|v%bT}wkwX0lF550yZG9H@LZoJ5d zUrmjT*U?lTR$4Zh!x=nfYny(g?jClp!fIR<#_Ql#*|o)QhV4Jb#RAvO|X;!AOGpPm1nvFov&jZH0n^l?cE(h41_pLBHE#MaiT0$fRysT%hr z4~4eF(|I!}nytexZ^ZQa17?PShH%R~umyW~*AW&%$O7}87=mGHr3Uc~Jj`2-POo5k zj$OuWXp&@GuM!RjTyr5(N8i5E}_Yb9~B9bBJ|b0mDHRafdfiJ+xggNMEpjo6_arkPvE^A5m-v zYu^<+FAq6uy2ImJE?D{vW0;GBfwak0O6KsFMePq(r`6wX_Yt+ zGB>!@O*rea@fPOhyRlFg%=Y;2O^d*&U*A=Tfx+zTpDBxZPdH71okpO}R6^lW?A+X~ z#OgDY(k1P-Btpfr@7#)M(G?g`;?XrC6w1b@BqvKlF!1-!)XktEwQTNbwD{%cKDXkK zu)HY3_GLCe>tg#wf)`IQ_X#PtfgX`#T~n>t>+*?X3hFB)zrdj(8z8C*7bX)g6FcP< zu-qxroJ}x9D4wYh=s7h1zp%5QjdUdhh=r}2ThX5Go;PnIP*&}9TK@Wm2lgNYH4hav z9I7>1H~f=oHcP*~uXGdT9GDGmglPVrdd-V z*vOT`evI%6fBT85104d4#mG^=tcTIFVC~4rw{HnUkts+3hff?7FzP^$ z=#xpTmvF|~bA}GbaCDrwq(p?RD!{6F^e#650Es;P48Bt?IQ&>70Sm$1@DSzF`yX)h z0RfO;SL11!Iyz(0KLQm>!-o&a0ATYzvP+oX!BOdkNgh{ae@Gi&Pe3x{!6U^H)4{%{ z#%K6+qj3JhWRN#>yY(uNq=j7#3s3ii{E}CbSBB3VaGp?GD_sa#1+06#Ig&# zN)2mk6^&b7pn~GnU{-Js-%>U9o^bG@9Pxakh+|q%AhULOAG-cG zZ}eLgt_&v#L%i!PR2m=8P8jAP#N&W>6&&BYW!nMK?+-&3b>Le-ymW{;9vJ2JJeh>& z9YynfrTy64Z3p~V+IM>r%H>+On$>KN;U{<&u*6Ql4st(FUo6%|f2j#=4IXedUbb}3 zzkXC&7zquM#!E9pL6?t1KlK#tXPVtPI8Jk-uHer4j#JzY^2b=CjUZ}|n}vmi2Y23r zl^+FosV%Q1mD|AL`vvEDZ94sF4|z!{;M!9ZG5ON%hrN0LZii;hOpmvbA?L-j$F8%PhZ zgbYO!-HBlmmyAGl2d??!(o$pV0}@7=n*hX&U|$$6hf8K=A#NjPA;c0tX?K1C3j%Tm z8`5L{=>qZnQs=YmQRd=a(j!Wu`djPe2>dq#4E_BDSrv>e5$@d{z=43>J$U^3<;!cB z8qECuZaiOt)cqK)P=K05Dd`|)y9Gsovt`P=uC7yP?umjN4n@t|!|cM^V#~1n7CzZE ziHPzI5zYMupyoV6c`S!aH&v7w+M^AcToB`jWG>ClRclxVl zeRV68vod_|A?pKzLU_^)3=G^=^wv?*UeQ1^cnX6LgeCR2jslu_cTno~qPir`9Y#tM zQg<+Zv&Zr==7?n`VtX$VFySCOVXqa=N;p=mOb4tROMOHW2Iv<~|L))@B{>TWRwIXqdR|M`ihlJs2qZ7-s$}TK?1DpWU!+MU0 zgsLhfgrF?US5R>T;d!;G{u_8VcL2abmbQf%2V_S==7SJ0powMq{0B)8Z>|*Qgz6)) zaRjm3p=A$hZ#y`U5*`18yGPU2L*Qq#8FgkX3MPQXGRveD=OnzQC3MRBh##v+Emy^9b8K-Gq=@;4I`6RD=aLBY_1 z2j#(=lIhceV|br?u_^;!`KXCWkC~Dgwg6YcuMZzcw|j7ScD&OPc7L_>_x_4tzR{Cwc$C5N2=?^MXexIbjB~71fo#XA174Eu`h}N6};EP|s9E zl_*A-!(#q73&YF#5`6y>=kA^GR1o1xTgWjX!|L=Z%2?pk6Jr2T!1`QE!y+2m^6#Aw zHlhy6z{KsYlCRw7tW;J(B~9fktJife_%Bf^ zl%Wl7(EIoA$0?0=^@1%wa=}99HxF;`S7>~ZD033*+wb2a+&EyrWt65GB(di{>N7Op zBje-EXftKr9k;M3g>5i=YrfR7v_Sh5AdL?K{^xzcc1P6Rco;@c+F=lR_rRrSd`!Z& zI>L7ke`~6W3O0D*vERhi&ZXRrv8nMFOFx8TMbZc=O`l7bW-;Ep(=&Atmk}j+Aeu7V z+dE#SnVV2b5#bph#OTWs5weJ_$$t|19#cL$PZ3AA>ty}Do!5!KhRZ~i>dn7@aP;jg zHXzzy_ZKXg(CKq>bGwvX0c4NF2=kMxI9Fg=W-BMKv@s+k(y@_qt|e7#F=jCe{y;Yx zbqc$nbkNku7?I(aDSfJ86NpoBLzFV&&rmp__9qPgC+sn%(@Rw*x*Q=c+BFyf5QhMZ z#|S%U1phe7BcYMRIYcK%yztKXX~#2vJ0Kn$hes1Qf<{tp;T#MqUm$EFw$YcIpdds( zKt-cbcog|Z;8PM_VxYI<@}slSDVO+HL~!XK3*&Ft16T=j5D&w@@?pqvui%>>W+#ME z8XPJo3YJJPEMvgP3lShjC16cY;Xc9+kuuSQdO$o8;SKo44!=a^%sag;2cjeZm^y&< zk0SD6pBFbL=MeCRC_$~R$%W9{x4)v%N4-OdoEQ-pyhW9zc3bEbHy%kzNxXQ%hLCVr z!&KxYhF73yWBm=xti*ZVL~ydm~~4ZUZm00RZ^mlmfwX1S8bb=6~^+5w44mD=NMrC8xT=MnW6m zROQD~k#)^mjBw?BX3`oxq3=i80ngCaGW&Q3UtA)1lIBokAg}b8C!j4s$sCEG4}Fy9 z5K`x@TVL>@xySNw8C#-u+&H)&?@U#U;yt!7qi34y{Kg0D5YxFuhZ-|*SMYnjpr@q* zi_u+5>O9368%ORx@aVijHqFbS+)m7eaFE7oS;T&#I2qZ~v5k*qQff=T7<%157?kOXa66sDIfAS-A)$+9W!!PL z)d%sZdXiG%WkuB1c$dd7WK=K;u14X?k`cvU!U!ta=A9vcCYV95IR-el=V0mt+c085 zDR+<+zBy5jJ9lNa)7|FW>%agkEE5ABOKd4U%^t#d{4=mz!lD+pV9n_{0?sLW`?!K{ zctG~(F1e4tQ%%a+jb*!_S|&NSWx<2#F>JpeW6a-^q6MEjydFHSL=)ew@IGcl$+#0{ z$SBrMpFJzNvGQLeKK}!MhQM~w>DEfMy3b_0ElamA`wZw0Jvy=Q3Mcw;C?J+;#Mz+{c+CwqLzj=a*Q`_J3!*$>3hZ&o%1#N#Y5j5 zQ9n6)??>Fe-3as-&@n6o zY6zQZg$~Vlg4~McT*jzcf?&R zS`_O4$Om^e{35^J|HIaIhjZP(eOD?;5sDNk84204RVaH`8bV}K*|I{VD6+~PWmSq$ z_G(xWrHl~C%t%r;&+A=X_kBFa^IZS@j^n2i_sf?`)b03Z>VCht zxVT6v9$ISPYoSm4<48AyzlUoW!^cHD-QKW^objO>J`Nfe3j^IWL&P2(KcvHPyYu^C zf8mKNbhyrSN?Lx}eCSZpaRGzUZE^+Q@d+Bxj$y<*IU`oDS%WEC^=RS%jI_AXBB7NN zWe--D0_zGorg}5XXGl&?X4%7TNKH$d4t(j`NFB*cfbbE$E|=v9;59<%inCI$U*+gn z;Lh-^H9#Q%0G>8Xff+^?tjv%A-s2E1i3mm&Wn-9i1xs8P^Iq}~B(-dj|Kfv_AeEs_KL8elPko1gL=+Y2l~qVm1Dr)3R;TnUc#pWR1Iuvg(%D6%RAPpou8} zK)*Nh9=nO}r<-XYgvA<#v)r$-8W&+&gFDL(8jYBW;b!fp>~8J{6J&B;UfIMRO)j=* z@Y}+WvOolJduP8@8ARaEo~5F|Wqv~Ai7uJ2e4yY6txC0T2RW4S$$nxFah@2QcEptA zD}Yqa+S=BmM_gKSV7m4%u%`?x6%jY7cM#}fkx5Q*!MfMd8txi0CvgvAa^ z97xl9yJJNNNC5}zhcQ9;$eOqBJb!eD5tCD=wu#>a+~O17{S|9~5b_`a2tNt$ck@2Z z?Ee)0q1RD4AUm`U90a|YlqLzL#DuX9{oxLBEMocr?$RFID_bc9Z^o(o zCv!$HB1WHt9|5OQ!V?hb>=O#TNZMwgny8d`46=6; z>fdvAGDH8NV#rI7KJW1O&vy*lv_@nK96SiKvTZjFbN1@gO*~du^dR&Ig^r?w@CIbg zs|j}QHaqBe8&TTnmg@-#8QX2Nx7W_x8YfBNV5GHfms}9pMAPVZ0j|BpMMp(LqnTZX z!wRnYW`v>rVq+QBz`YAF|23wNVKdNg-5QmofkP0bKsu-}=VRh7fl~d5dU*YH+aV${KLvMpQeMKk1by$JhGK3~o}XL1c9Kq5);S2eRHD-lt`piM_|^Dz@s6y1;Nv!IP~a&q{KcThvnZQiP^ z(^E-l_c@Wy%HAFcH_qffdwAnWSka8Wb2Z(kUr^10$ zKQ^B$r}C4WZQ7oGjWuK`bo;@MCC`bX+|DN9zZ!H?1+F({88<;(y9*BofwWM9EPBxW zgG=ClLie~F{u!O3S24{t~9a7*P5N~Bb97~M~r zA*Ntt#Gs5@+x?5BQzrVXcL~Q3xZB1i2qILVI|vUsW4efk22$>7n0a4KiL`^w@h9GSQARja|A1qB*zv5H}Z!3y%C}g_;J>1n$Wsis^q^;863!wAP z(0>rdqRrk686`77Kah08L5ZgS%3zJpjXOTI-R^!#fv+nQru=9JH#o;_BD*lkk_WQD z|LLs0a#$#pUHTP>29b1B2#2atSB3->#PM=%#0)kX+P-U82PW3mG1-+L9zZTW>DZo= z?UAp$Jiq;dvvVpMMrfQJLFp2ZjXU7K+Ejlzp&Xz-(fh<|naR#YpgF#P&-X##F(duG z)fhK^=-0Ok3lb%FZ}npJqOU#X)zDmB&0Rb4l!p^j?mPk40~vet2EAaHtHb3R=l5a9 zlS>&t9*&9&SHU5u9q98!1O>G%H+t|`*rKFN4BCI1EgeZYRpx#_rTSKEXj(dDI5+0aru@Fz zrbxVt5Wisz4T(!*pso6k^4_Iefy3k$rqKe|Q4byLz*rv~MQUA|N0fY1Q|X5woxwe{ zX1%8qL!|0!)Z6sxQIpft+?aoXMFlIt{JTMjU0%F;bp^MsE?6@~?u)}JH~2!|0^&k5 zPz@{?U3o7+n-pm9Z#)+h7AE%W74U^YaZ#?3PC=3E?N#8->b=Aq%zI3B;z;%Nec`YbO!=(ak3e5msT&zI-kvQtB$J+e?+I0Bd= ze-0%Oxt^bmW7Fut(?VjU-#SeLNbp{&Z}O#x6gczU=KboH$Nu+}nx_ho)|-lh87PV# zB+~$rkD&DJzw$rAjk9oA6E0Ql4xh@@Ni)*dE~B8-fVld{uV0)f7;6Xl(2(KqR*tOy zX}x#L)VI{r;WdeDnu*791kx`;7pOCU$y*lA z`#QpiMC6Xz$h>J&(mo+37M8fat2(%MoL!1gQ$5ISjpRSY2#Op~P`vPohYO2e=D+U} zKj!E@-r3W;>Va&>6a_AE@=;JClj;!nL1Gcpwg37AGaei|BvZQYZT$AXu+&Fib+)w= zBrOGd6*UPdal-mBT;Z6T~lTN2+9#wNm+oZA;qhQgPbE2Xm)=Tih z-g#cWtJZ)!89?){j{<*8dSOmX)P94Uoc0rzIGNc7u9MWEXNW=$?DwWkn+~A$B~i@) zGgO|JyZ_{pN7xy(G+;$|e@VG4N_}`tU0I1lz0A`*(rW8=>0ZY6*}#ut?x1!Bp5w=r zD^af_4pH&`LO25D0gsYzgmF3mzmC9GkE`aUl&r%Idd{%BANzjW^&CRxUF&G>fw`O*|LkIDqe#Omkt1JNO^gAl>?p(Al}rn+m^Pr>i7v_a6@;tJmiS+ zX}8gvrrTFzT9Bh720zXH@&^L2z=^&O`YR5P-=lq4BV(?bL5vrMw0lP~A4g7?iF!6t zty@QYIs}XZ5k!P$lD_|uNhuGA3d_W#66-z#cL~5}vZSa!Me?1XN4=PX%jHu1i9-6( z(VZSHZTX0}zRSv?NDm{?OEla#^Unb>A#G`R2drXIXxp=K!{mLxV6ZjM_Q7hANS^!Z z*?hW+E-5vWiIpHGt6N*|f@r*ep3h;Re)gRmreH(ueU8Zf&;V-22g`bAavUh|VLO?> zcK`kv*_r$}t9(P9Cy@*YLzUuyVgmnDg~s??_w^0G zl=>jDA?iq3>i11eg0;e?;3=`aR{$QrRq=Xy3ut0gLm+{JKa9-G8|m}$M+)qX7V@|q z5HP94$HS@iqUx4*)|nXRTHr!p2pEt}i(fu`rz-mXf6va$5VsAX7V%q%%?9-;p?a}6 zRhnGTW{_U~msGBgeBOClA))L`&vb(|*M(*fZ3WW(@nqlPXtRP=F6TKfPAhaIsfst~ ztrpEGkg`(gubml1|3Uv@#Hey;s*;k@I}AJlha@B>){kmz^B(P<@D7~M!WcBghzBpl z>anqJu*ho%S6w?Pm0eBqj=CN4z4=+Cdhfvx?u=ij{hAe0-AU zAVxyIcy1Rhl0g-5!lb{Bvvg_63j>wc(jMDakm8H?(Tnj~2fzeK_$-ygDgQ?&>l)hH z@)2n=o>Ve;e-?=gE3kh6R5Rr90~|x+Z}ZK0;KiDuPUd^}O(xBt2<+WW0u*q(0J1j6 zBdevcUWHGl{fSacNeP-ARc$@rRXxM_Ozdy;9_W*KC1^v|Gwr*Xo^Fxj7jxjjKyOda zZ7J5Eo2z+wIWry{S|f4cfL{!{{i`ToHyJ%LJ<`D@KObzAI-Lwu$mSiC6K&oi`3M2_xV2HDJb^$JvhMX#?PNx|GD~A z)r)&d+K;!LE{);p%0bhKMx?}R{todGB0ySF<-e7b@({i(@X>@!lf)zmw3MrrMYC() zy`$^;Q1{JyFgQ$BN^~A!`b8r1y+P)0R}%F-|=|Kt1j_blJtHGV?~w=5a`X-h(4XMX!Wy4@pa?7l-gwTzs+E z^dtnGD9{3EUf0y@dG{Qso7R-y*+A_F5!BZb60{4lCm}Y8_Mu$(y}hsQ*Rad+ti8;c zaq*E)RnI|kc<-;h>I&8z>P{~1p#LeV3G0r7g+%Z}IswF4sdW6M&1ST@$%l{6knHc|Ess=;^p?t}>1p(+VXqzioTUlA{#Q)l+(iCxSm6J+Y@!~53J2JBKua{s9>@CId;3xHzMvY`A-!2z7{erFO| ztv{x!h+#pOZK)7yIpgm<$m9z=FNrj107PK=M&x$jeC%tz0NA;&k+R}%pU&Lunw5sSSH zp3crDt@~2L@IJ2V@4~uBxzh4DGImJ?`H%JWJ9qBXC<+%al7sp^1pgL@J4uqs2~oVZ z1)*fNRF>s5FuDEFQl z4jXBdj}&n9_?=>%KRM9w1{F29uou8_pysc+@!v!Z_SbH1ZUnbZh7bf?4TwIfPbF+^ z5%@g7s=_B(Zvcc~eI$LtHRk(QlzC7diGqhlGo^>PHPov9=(7*Up8g(r zL_}ElECQ<@G)E(fyKAzNlHjqp3YLH+qh@id4V7<|}!pY-aG07=J)dVwcT5`8vmURNJND+Nd(0WuBffS|uH zm>X()cOQ?w<IQ!Qj;2$3aewl!(3$1Kvxgc^X7LTwckXlOTK#%;v+`q`gNsnjsrGn(;qhuWh?uO( zJe6+d(4J3Q7a5L9h8>FQq4%lOXpzbE|ZE898BpGTkXpK>(*P0V`sWLwI$hsuH;!J!b^rfbM~^Z?pPY#?~Nxf9jCI*0xW3jgIByh}*$XCD0+h^FXwnxOGIBoctE(^&!41+C)!JRGMx}3x#po z|6R0l9l)GG*HHoggT^HXy4R5BA;oe+*nGEgoOsUu`Zkjkz<(;q#g43mUdRgJk0kX8 ztax|~oNC5xf~j#IabB73CXazg8|^+tK-#EV?(Kygul3w^xaZFAc&5kNTdf#pLuYMV&qdVVhExs@K(DperTC3iP@-O0I{8>&<~cTgl10aJJruCWxmsG-FTp^R-4sd-T?JcKtcsFHZLB)meGp3=XE>bgWE< z!+GUQ!}g(hpWoxsPwFObeVw7C+tMk$MeI%8d*L_guVu9l>a$1+<`=&FkYn=1`_4u+ z$Dn~78JE}fRy;0v<8aQrUtDyv&d#bY3rhuW@-pj_26uK(k9-WGkco5bfBhwH(EG`J z(WTg~w7UZ7HpFaX1~pP2E*0F~f*6X|uK@-Z|1IgBJh@GvY=#|N3-Lsf`xvjG>to}c zJvys=e3qxE2j;(POb^Yt4Ry9}7Qa!$M$8g8km=aiv>+bD!1%DQl%>{@Cr;(I_moZ_ zh@Dhifhk-dOs}BTL3?ro3e;Z*Q&SMzYkq(HC9Q&+L|^TFvm4hJs`cAA!ak3u8fG@E zHLZ#Yns0q_PT2qrO0m1{{tG<^G|em4?FsckRDT0e(B$Qb68w_dRZ^=A2i=D4;(Uv% zo^~|QJoq(|Y^h^V%ow;u&(HgOPlf#QnNW7~{llqh*`@&}uH3qH_5Kl!fFk#wJ@Kwd zK0dk4{oq*%tfSVt$p&L(F>zppK>30&a88}ie;-Vr*2Wz&QKaz)i2+Oo{_T3hWm_1h zzZb3hb{uU{3JZ*7Rq1qHf!kUv`Y%YIN8sxPcGTXUAY+%|=B>L6)TtVlM%yxieZeMThcLOy0ea?c6Q{AD`IQS;^ zd`!Hqy?wS7;@qFCUW`8ZZ0Fu@Ax7svcWxFN>!7@~ExI?mnEC77n3!E(mx!1ho|rwS z3k1=D_9KMg*!aI^Ar9k_362Jcm2ym8u0<<$PGct()xhaX)Anw^PqY;sjo+m>-UEO? z1>O#*N3WvlCg$8~qmj>r3^C$ju`4jUtXGj!cQqVtJ*jukyg>Xgh-3`T9@*+!`a6bs z%LAJ=?Cf{+s<}d^jy=yTZ1c$qW)b=t+meazPW%se^thmVLE;`Sd*y&;ffdlRJL5AJ;t%>mcPV^MU2QCk)8uPaaku-Ol}Z{b*gPjz z;vaK^$J8#vnvI44v zXzU^er{FGcfLDU0Jj1GS$Hb(tl7{T13hu-`q3j2MY@w#6G=_^KSnq?VT@3k1LJjMj z9v@JArMnl7ly%?qGxnhw}Fi#7aQj9fYK2`iCDeP zcGe;HL2GTT?KS)Nina7~PovhyDheMecDcQ~I=*~*gi3Aflb{v-)Uc^N(_W|fi&j?V zRnM~oQXE}F@iOsQ1i}%x3*QY;Qth{W@~*HFVm5DGez*4<$41#Bz}=jKHqHND6z}k^ z-;QH_6sDU}di7mr z7I6QyH_v{fi_qfxU4d3&y0IdFCTFyk3a zS;aI|5?lixbt3d?do3mtRWLvg0(-#iB(?+7o^iQFiB)<|$MaMaI#FHvB`lrRH?DLf z_MKyGZMm1-%e9uBGx?44zpqA%XfUNyP}X0U3PNQQiZk3Kx80z`O$bjypGN7=qsd8v z6hf!i#SRePA(VhoKSrupc==A6?Yn3G#=J<;pj}M$)xmg4N8~GwIDGrSL*!x5w4h@M zW}T`H*G{K7U1C}EGVA2IqSzp_i=$ zo*TqW0W`N?LpZL>J{?^x37x}!q~EgTwLh(#rH0^L0sjC>&v||~kJ#61j+{3fyDsH= z?RAjP{6d?M+rp1-IV}~TtC}6pquFyBV)+eUMT=U|!?#d{W)8L3k50!%o`Aob^M?PDw$51e%v>biX7Jfe0+%unL|CsLC|NPd^#sH+(p5&CX7v z=o6|W92i2|*SK7rbv)D3#TePeKYsKKTI`!!j_iHiT$)jPMK=`d_-ua1Fl?1LEb3y- z3V__?K*dI+kN7B=yw54} zMeIiBL1>k68=+_@PY1S(+Uf0&pqRIb0dv1Jbo3fx%(wXm#wfyToQ!A8f_x-=J_bg{ zfvrj!LPF-IM2v!i1lAdsh3`{nn3$^oe}`kM=i6%?-|+b*i{?-#c zmEfWD@Dn6_2DQjzDh?sKlX;jq?aQ4g8o$e-Puow?p7i{%pVebmyN^DtHYoR&1tf~v z`fG2J3d2kEl9ljb^lxpeR9Mxj<74D_kzYy1`vJA>PZ_5s9n8BHiXKT({nfk z;Y@D*e*j90ezh_7&XyFKv{n=!dUPDL=R&Et0 z80vgxK^BQQg_9+~%Ovy%5`tUAZX9fJwQe@nc>Fd2%4~J#aQC`kHrN6$@x;7t@I3;dS zvbF9mOln_J@A7Tz0P%z3A{PAx9_1?(GLdS4jOpk?=Sg}K@$^A2ijB3EmX>zJk6C8s z_uEZAt-lO7f>e~1D{vVD&(Q&5L^iVfh~nxa5pf&rI>0~-NU}1!&u^v-<+BC)ib3SJ z@Hgb)q_hT4CQ2lE?jdhtSCXR!D6#EFLJP@CwlA3{Vf873&BHU+`whL z97w5OwYIhv8CTu%=h0K&KYjGk2j)Y4t~d^Xph*HWOb2>H0#1QCOSv@h@T6r}jh)MVts(b>Qfu9V2M?UaJEH;B@%l`LU_>OmJPoLaFzjF(ASAQKgsvFl zvgI8y`}a3KejBf05^feJd~STHZT%Lw$eJsMpS9Mr-&TTNAL8PDC?NVFTP1L0N?O|9 z#J_iZ^alvEhM)k%A1nF!R(h(WPvqxuO!i{mdJ7N`3N-lWvt^dVMj^mTLFmsMm;_mT-TcqfHvk_3Q6J zX{UJj@Z`5gS$pOGec2|ga`>;ih|&`hkX~Qt7ra4cnd5@#QL$aldMAuHE-DBKO|1!xBlj93-3T0a6$J$k%#*+ilEdyZt!p1k zd-G;$xM#q;Wx=NPn6Q%IZqDsu)~^7vyG_<|BRfGFIBK?BHZ@a{xrT(-Cw13#R4X5G zsiWhR)j`L*T-R}a*-M83c`*|MvA6G-pKPysWk5g9pe0jT{Jo6&0DPDQZa?lh?D4T7 zL$L5Lpkl=lWkNdCweeEX_tG5tcgmbr(g?1vt#xXCs81Elw%=#|{d+#@bQIu(^0^G{ zNDj!9|GqRsfy>x7kVL?ANj0AXXXXegQdend>3#T+OtFyjmW%ybgE^UF55qpJV=Ezn z7@R=^omFlh7<1aNA?7}cIeWQuj3sG%X2S%`AJ0?bunwlkpz6MtHki0SzD5572Yc9X z2HE8AGv0NvNjszrPdF{f7_{*ayAVFzk7h0TcaKl)|InkcREO{T2B{Fx^-#iSfV`jf zBI7gO%!d?`FPJp1pYx@R*m>kGH*MybjY@%0or~Xn?p$jI<^rt?hJ_}BP9##by3jut z_jL+2b@ddenaOk{>{mLV(dcY4sW2KF5)K1C$KIQ_@^9_24n!5sQ zIBW(nXx4e5qT74B+{_AxzRO#9lf1n+_4=6Ll7M|u8^r*Q{EM!x`EnV3W+JP8zY+Ly zgL^0YDu=;iWUcRt*SI0kCnB<)@ANuk?OhP8k2Y1=iUcJh;3Z)jz*0!saAyrGNdiK2 z^KBrC5WA~mt39R$D-k8$FdB0bWs8S8RUKblq(7%xMLYB15OGv%q9!j^gW121S<&d? z(SGtC`MbGkYW@{#);}uZ_}dhCyW%uFQwBJ_5tHvl`qOA$;E3}6v3U~=7(`mflNl8I`Jca z_N*eyP4s<`HK8o)9;MeQabe9PcG=M-@S)e%# z`j7niny-9svPg!3T*Q3xI`oT=$LN35@ow6@diH1RJ+4RVGZk4rY(HNDF{mEQUX*Yc zrg)og?H50v(C#+cx4JpqKQWP>XEF$12-h!d1$NfNuV2nLD?WSBa%l*g?)LZV8BUMn zqYG7kTl=Jif<1VwYNOi#^UUv4801qGlE3HXcHqJY($bn7h&?Zi;-jA{NE(<2fJhju zZXi~c3}PAan_Y0sINm5@^!d`8ysLNW#|9IVHCfxryPYncEN*e?&8Y#v2=5m=KmRRg zdfu0LdHg$wUY#X=Er^#;+~gvJ4DSDPhJs(KUv57i6_i0o*OZ-1E+cFYV!Hd+-2uLH zT}V3q1o>hW=Kdjae#r`v%ArQ&$)o+kti(pRWl2Vi>FVcZH>|ey^sRn#CwlEFb{s{h zH#kJv%GqRO>Zv|l+jxinCR}Y~gcO$LKjyklgV7m2N~o)d^_yUYz_Y>BpJJ$|l}8-& zCRRJCSOA2YCpmsfJFZRepxI~W*Luc(x;tMqju2IUV{ggHk8!#8UX23)9E&-g4llPeSE zR9L_9#WpiBk#V~41@+DjCR%{nYS|+8`qhD4^Rx`KTT#--l{BWqh0pDjXd87|?4ii* zs5*oaBo+E@AcF6aZUh&3^J9MNPemVZ>g9LfbOIi*@RmE# z(;+v*3bkwW{R|%A_fFQT8dpPiAN_z=3a0fvss_-6SMY%#VspJ&LBn$S*kg5_=I!S* zGE@}%sc{3Hs+6#@dsIy9c9E9vU$T7Jl&Bc&|v zA}EleaU<^icbF~{DV8>DxKGZMk~D-UguX{s9vJ(yX}ir~eXprdvNTDGKiVu5g;kIT z4Khgxx65P9w2@fCXi6KCkbV0KdzcWNNk7-AXzt?|Y&~V;FS9t!< z!j$x2+xXRuCXBw%io$Q(cT-z{r( z2D42i$o33VwPs=byP(JOz}<@8pF+O69lQ4;)P*jYf zTEWilVDv8fjR^O7C0npj?7%(#zU+JShf3j-VS?CnmjKb3-7H}gzs3~)FRU?UJ9r!4%f z1kAF*2a@L0<2N5DS<}DE-mrM>nKM}S_J&dNMoJp&`9xVIP&!3V%s#a&N*YF9=Mq}+ z1@G=NH+_m?XGM0$7zwIHoS16Fo`Ys)DW;pnI5`dkG2*vaA2oQUir>OM%@c2T!%Q4a zV30t}u>x=JJBFR#p-d(R3B&>DH0}+YpU!vv>>hx%69L8qF;<7ceBsvbl+f!7XqWj=eD)2 zMm`PUT5@PjuREJohKDC>Y?p9axA&On_AJqRr~A^tx-{Vo!I40u%HSz*j+W1^jS&?J zr$3p0P0~?6bB{w1cpTz^B^no?hi({j^O$xo=W7|G=~3SRwrdj?YQm2I+%MD`j*nJ0 z7q#0aGq(o=ZfyXfbNT%sVwI{Zzwh1OVoc;R52Zw_Llu?gG$g_~9?-P7lxHkfr6P z51I1jjZbj*p9Rw6x$vVJcCT#QKM(aEob}RnkuI5{&orV{3W{m{+WVT^kn>UDhIYpk zg#n_YjL&YbkIg@Pi;v0_Uk5&%@ZCpfa1d7OY00K`b{;{gK{VfB7$B<^0h>AE2VGK( zn27XlZVkV?M#ube9x4T?4@xmHwH3U(nUOon@<|h+JO*W+k|DCcw?Z}h)ZW+d?WeO)abLrwdBt1y7ovfvt>uh(@Q0eG| z*YJ~EB-p7iU;FF*5IKjqdr*z3>x2hUucmy2ONApy8$DM&AAiFrM|;j16vo^}j4*o) zMq4C&!Ab+Oe3MpwLA$WFk7Tpr^0e(TXEm_?6sCd8 zr-Nl|vlT}JgMdk2hj0}`fZqQSobk6&jB*tpb?i4in{1CJo6MKu0@rzNH+fX zu+p_VTB&~xHQz=H$h1UZMa48SlqJNK_3uSG(2}nQISOQkXsc1s7oTFPe{R>5LdC%u z{X_%l#|PQV$<>7)>MXeE5AmiDDoLO;EYd@52Jp?6&8$keRA;+wpT2VK=GoW4JTRwp zEy=e9;zqbl+u-bXZX?IAmrHi|Osp)Kdk7jm>}p2;Rb}(RahhE7=nkBk>ksfvjOI2H zlq2-c9n7#bL{M6wLpT*S%0?Jl2%mwKcH+5hH13xhH_{l|e|*(`5H_rwc8{y@X!4;= zjB6oVeR0*C(ttvI1EWD^M~V|5w^A5kBJ`mL(1>h@0}CXABSNO#BjK(wYCI zN7f{CbqYxN1z5S3WK~1p7i{!lGn2*oE#)(-c6e=W$vRGc48};y&H^Fw9q~Ns6Xm;w z0%L+%X2*UN`1qipx^8yxTN{T%`A|zk-is%?YkrUVvPdrd+@!0gN7e$t?a=~bEWNIK zt)lnRtOc$-;_1VwZw7TQ%fl{%=)ey*L#6a}d)gB^=iR5AQL) z;BqWX5hD>7!Uz$9;3c4)5al?s-k8F#WK~-<#@MV?yBp(dOh;F%J9XEn&s)g(Vf#6UiJc|C?r`n`^cRod zzEEPF_VWXo`NMEIv%y@ChTkl=eW1V@V;fKiL$BBJ$;a*ywnPo@IX0gIz1+x?anSgo4 z;D+6p{(uBC{CXDD#$0Q+?Zn zCU1nYrnsSR`P}*hw^hd;`<*U~;GpY?JzrMY+#HS~3;o+`uyVBtvc;y9@nL5JV_KeE zJ^NZ+N3YO9HLZppNDUDpLnsAQvUhyk-b))rdJZRxJWb@@=qHOfWpMqRh=6Qhocrjb z7a2On`3lqHXt`9j0$jX@r3fPO7O;RHPl|;2jmqK(xDdcdRHt|+gug|%n~;-p+p=A_ z*_|73f8M%n;`)Al`?}wmiM)>A!uz7tDE3Djqh<#`bkfMC>uXF*1m=fTIDNOeW6T8& z&McNgQv{YB{;O4`r>BSHXbs{`25Y$!IV2>a;*J>>eGOuC6D~fyg}2wj&~f6UMpq!4oXA z&qRFmBZhMr=~f0e9aL3RP@{IFLh4g@#)_n(I@WJ}g=r1Ev`ekK}+ z9%<9ot=I7n@C?o1MD@8e3JyqFQnl<`7v8i4CsPOm-#$q^_!Qr9@|oux%U?>pFi3EnU1$OVTz_UY~;6|GERVa7w+ zTV)KW!aRNs-QToz-P^Z<4k_)S7Rl%o2)Y0S05Jr8xU*el#sSgoKd%)ds^b3l8O$dD zaRMs}HHmJDG99+9TaPbgCbi|96Xt%xKOV{Z1y~9PU4bO~B%TDsjM-!!nB<(>;bPXc z8!+IsvY{c9_UPk>W`UjTGBRTi1QAK~9ftm?$rOvG!5r$Nm!6=3R(dN?+} z72_fhY7=>ctO2^htkNG<+mTDl1Sk#hMh&JRvgU?@yvHgke0%!L*6)&$joeAsgBJLY za>jec7Y89+S0~|t9GmeiUn=4`=Hz^J;zEYusoKqjMC0_2aDn71W+la1l%XqE3c6&r z=M}jLNvKlt(9u1OBgeq;$ERX-oaC1`0O<~4oIyDS$2C3N$709^nqv)-^i=!u5#4b_ z@di-TLIf5K2_Y>i2+$Wr#!n1Ilr$_F4|Nr#br^`3imFHf+mdp+Jkr|G{j$wK?Pau- z^Inv)GWaLt`sCo|?*6@aiaHj{0ezs(UvW%dnHc;YpM93SPMYHrm?Ss~#wz!Ab6*38ppk{zONq|H(?ZQ}oLscz{r24zFi*sc2|6?n`ypd`e!a&Kx z#wI2cYLq-csJVI zYu-R=x0W@Iqg(T=&xx=8Jxq~Rnzs7J3Zmvk2kE)^s}%q~$v-E7KzIz{TIx*aEi4QQ zHkA-``4pKjfm7EJ!N>62BgDA~c%7?&(C$nEU9 zXYZT0wdAh*3NZF`#y_}qJ|9}7;CH?m8YIsCIEZ1qt&2kW70^E4P zY%>u&f@ar;*McM~neYqa`9qukOB{gS3vr6K3DU@hRJMU%5^_5(O2t8iR!%XaJR7ipAviH%G`wjh0g`JHOU z)P$h|tL|f^=AWi?eEa`0yJR*!&dkTuT>IWSIY@eKdw5i2Fz}e&68*)Us4HHgFu?YT zJHqA_0lHZahi5gd=VzIT>HVxF>00}@?^Wtc`%5;yfpUU`t&mIkZG6XEg*U%&nZp|N zS60mtcm{>L2}!=@Wq^YJ+6%*BJoY&q%k%Q$P`hAXV02OqSPXj?cC?op&==*i7NemL z3lERXcRKs8Aaosng`h{o^_}dHn!6uj+^eW(Nxyk@BrmT@*i-9|!sV_m(o;h#6MgKh zBv8|ngfP&=lP{&8PSgU#_AIM;l7epR6S z%4Gj88J5n@2#(sx-mhP=-AzbE4QLF?e{G0=@ErP8hd#b>y1qQ2;5Pi{ZgeS*2SS8+ z7ZsRwuiw0xV(|2A3#E%q=#QU`vFxTthIYbAPNM>O%!%IUgpG`RNNzV7I~gVPGE5LQ z=my-}D~UT2nUDN!P?ixq`-z6dZ zEYW-J20+w{$TKB@>nIm%?ljyxXI0!4xBc+pLpSILOyAv84Ks5`1xQRskREFPQ%;=7 zN%=}yX1xV_UwnJX{F7mN0rq^7sEUi71Ic2LVqBwS^K6<+3JY)8Hg?zO#n+J72Q+^R zC6+tkUG@h@80nrKzL4#n8^y~s+C{A;8)@%aiaBW@j;jbHhJc8r9Je$`9O1&hiDtJK zI)YQ@OMeBJ0uC`ia+kwzfm`5P0L)QhvH+T6FhSY-qU<`L-ceQ+?sCj)+PVBJVl$dk zmr+5(IB`4S_?PeBK?XqZb!bme5bQ{TTb87IQSNZN3^q}>#O)%uMP;C0P%Q+Zff8^Wp1?Qp+l>PEe$?KnfcLN;JR0dM2e|t;tNo?+xxRlq&F4b zSqHLn2GhTByCOpnW3INNFSz}catAY&oSgaNi+##lMGm8Bj{lF0z?DTyTMeDfT+;jZ zNeYw(PgkklxnfxSm)cTX3d22{6LY2%VT&0Cbicn-9&dFj9zo0f9=X9`VPWK>BOdJ@ zuxOcIAN2O0XM$B3-mJaHpP;+9Ab9{rUcWxFjq@U>8#b@6X~%k|ZDD{xAw-EX56LyY zSy@@ml^FB#%;x3CM@?CY&T7>W|7iit&jmSqFVKNWc^fZc7$DSXCxHc^UmuT5 z$c`XrYO{SVFPnz34*RQJ|0?s$7ioHOGFMuW)?F(@L8g5mmI#t5OddeSw`7 zn?23R+&GvAU-tV*{w8U+z{svaipWFyN6=b4ki21TMkN-@9)#&3&ZqC)5w>UhlClMX zLm0Bl&2PX!+AxkcYM{x{Ly;;r&k`c4_Crv2u*y%!dQ>zsU}hMi%6{8kaL_%ZWd9Vw z%GhK-OHK$)`3A3 z0s?v;^ozw;uYH)Il6kzeh{tl(P4v?tm5nXY$5|db0JN@xI|HdrL)FPZ zP6B7*`{zKzIfY+!L&(%-yybJLdEc#uwafE~1iz1MAOH}VUk$Dm^3;8fo5j3-MV_!J z@n$V@5*m?4>s{D@OO>Fv&>5pdm)6v24&$6?Idz+VJKxC^B2Aq=^5Llp$D>QjUT%v> zWvY+cl;+4g0ZBC?8|Fxihx6(ZO{N4@~?L9KG>{Pvxa zv*XoFT-^To`)xbdX~~!b9>FN2uvZsL7KGIkY?qc;C03-Wo@-O!Mep!+ zX?uhu+iwO>O&>?M*2hMKEScXv9#1w=C~%^J8QJN{MH~&>X!XdndnTs49z`K|MCb}r zPcQn1Gom^ptQxi=EbXduatDle#>xA*R(~2ZFKI>0K@NwaB1k}%#%eZ&#kq;EAx+O8 zGi%&ENbn?reIC#HEW5aX*4IAx`=1Vt?xH?^nJMrq(j5XppAx`&f^Mzn)3b8<-@Qtf zoo$=N+&y0AS*hMObH_c4bao<>B_S>Fy%8+g{e9*a#Ul#MX=y)$2vUOp00sDRB{4;Z z&_;M%!8#wDdRCy|4UGBhra^E)2yQBY4+P6NY@m!j|0}qKCMW~kP{Dd&1oc~4n-&g2=|!~ zh!kIb+-kZJ-4qhtjy+i#NJ+&KBMrfRt&>O+i7ca~-?l!0w%gF8d1^R~Avh?AXPY}t zR~44y09;u%6gs~2TVpP2xYohJ;RR$uq<*wWa35e>90l@oP+vd7f$Pt=rx`@$i5U|y zSXyViejUa9p0Fj_u&8y3o@ecsKh7ztuFtcfq@;a8tY-z1gABF3$RU1NovOt@jO%MhKt3T4=A4zoNHt6f?^ zeLS_MrfL+F0u`Exi(a{wUx`+|FYgHS`&v|2dOw-e_36mh7Ovn4a}=fJvW-CmR+7@kbHPwt6kFGt3ema_6X zU@Kh0!eIdQPO<49T9s$Hl3Vk@$A{V0Tl#8}q13IBCBm4BS2whiSFy-io3mKlMyE=| z>6oA}6SA+*hNUZ>$g8G?9e<{rgSv^?@qLa7#c20=yrJM4R_P=!5AbPiMK>)a1;TVd zLwDCW+;_RfA%1x9|i%T|6-LVtiz|x zh_y;j@Vm9Wl1{4WxB0E}cP4c=-Ta&x?xO zM!Ywo8eO|iSfT-f|3%e(E}p22iIitxpbGcD$aqSBOJS?e^9yc&`e!MhUw`{&l>rg^ zxae8${y@pu;E-jl9nayU@SN?xTe2|uMq*Jz+B~)aB$7x3@1Jk~^fW3)IFRBn1`gd1 z{kCnYXnI8kD$y7;r^D`|If7Qb(&M{tt_a_WRbcm2!umC$1bs=BO8_`N1JKv}Eb>DI z5^zO=blKtc!O58S59tz*@&v<%h&ddsbrnX-%q4kuOA4{C3S|0o>>D}e05Dw?xxNmz zI%BCjoB7uV&{R^;nhm{-M_N1%`-^yW#p}i>$mZ-HVuIVA1Rp}|!+^PZIATNwe8FOD z6Z4+;oI;Gnx8J7wP78{RYbCB_V-p!!Vsq-FzqqG0EB5tw(`xmL&2^E>!=uw^Bb$M4 zy-kz{{D-Y{n{+6sd7OL}&@YfqEp~YktvLz!gEo$wW~WcHAXA7NYy(O`a^mAGN1y)^ zcrk`9_F`rmt|7O+&?9@$e*zd!#~Q}!7@Agl#g4?!^4XH5AJuGy-iuO@azF_~50Mdy zPLQ4?AP*2S8(M}=%t=GeDEv^H`A<#_{47L14S;sTY0v?7wBklb9P2y=_<4FY zJk7O$SYs|VjPE>X#gfT7Uc3k*l|Tc86>w$s#H62HhkE+mu@30Qfz-PGFOf~W!+$J( zqt9A(BPyHA4De_PHQQE4teHve~LQr|PPh zt}?)biE^G$VUs%#Ay___>d}u7bgTSQ90cItcW7`*SE9<3M1&{0Qs&9!2n3xVGE@P# z6jDCLk!pAk{oCH)O)3_6^Thv*I|0nF%THO&OLF;!r%&&JrH`x_Sb8zcE}SUXI#RXmpVIPSQcPu!v=%(iQ!YV-?qG313DNe*q zkVs-7tDr`T8UJ31ouY<^;|@VF`BaeKA=hfgs5>8^Gz`0-3KKEp8}>Kn*00aL*S=3} zdeA&(^yqYZrqYVhZrWcXu}_K-Tvii0rxeb29^a3gCMzHe4u^zK+PkT?Hf{I9r)NFm zkoML^>Io}(TOg4|)UxUF%#0gYg@%mo^NP>w@~l9778D#Xh$^fCV;`o8(uY02_(CNg z&Eixzv)X=3?Q8NWsPSRK>jAsZ!OuUY6(SAeo|epySMp`c$mouS>Liyuc_a!PH}ZN- zQiDd9xq)QLOf@O!KRPxxux4Ru8v}`cA=Xda{q+cZWtQ`FtKuAh$~M|UeZI};*U(DG zfv651o}RIdp`qGkGNYf)6?xt0q;1LCU*7E^edSsVo%8ZjB6z_-b|9jY0pDfkhVHY- zdoP?n(%SV6)hULIr zq!Odz0C%o-)_v;T5t7i0OisMofPfvBl&RnB^OOLts zj5sg`Kgeb|gm`hRv8u#E5FX$iJ_|o$ZMHz`kODMedBz)r z3$bUzO(uZ$w;xi4+JdApmt0NfnTatYpNWGrUnpnBrJI%J-g6Fpq+YdZN{#!(XIzBC zA0ILOAF93sn(McJ-z=+=5*k)ym6ReWq>`P?$P5XQjO>zV${r1?tPoNtdzKNAks^`o zLblBRb@%=K{^$Ij^PKaXr&D}B@ArMb#`U^h*M&+K(_zxdS8q5BXewvlJLY%y9tElm z=fny-%90h<|DkhY=)AEI6vd>$#4#6k(q$?}^ykO7G?I!Iu!mAHwk7aSgLF)V8h1mq8%7G_9fixLIMJXd!$ z2zKyPS`Ubd$Th<4^q~1o`_3)CzV@pIPb4lJi0ZOeSKqQ*>>VtP9`#}zR{Qpf*m;`1 z2HW@hzaKa2FLl9ek~ngr)?E2hjKuJBc=&IW>~-)r33CiP^O%YcjDMeH71He65VH*n z(F!nGA=MvXrM`wIY>3-{)UP=C>6|~$ix>d6N=MZE(pRe%3(w!*Z&}xrX(UZ{Ww3vo z(k&Yzd*$z>C7Cyas{;-|X}Wy!xwuX&QuQ=shC$aGmGv30<*`Dbq@uD|>2w%52MZxx zUNT5m|0EyM7S{F5-xjL8!RTc1IMLd^fpWcu2IZBVQcj7H%O}KD8jK>72ZWtW`3=-* zX!Lvqh3|=9+3>Ar%bPm^Ju7POxRjyjpiH;Z*wLR-jhjB)-!6~R zJ`80hw5A93I>tBhW)W8`(E2PuWFcV%0hn#23n}lswe_RSJrN-x8-YtANjfdg+KoQ`7!>){!Eq?QaW|B>!TmG$9z@VS zL(qkQ{c`PV#=1uZpteu0ck}O?vf~5$C?E-sna~N)8$irzVbn1Ng0>ZdTv}*JLVIFX zHb}|1^YOs~2GLdBw=Xk0GH>3@JQ?%k$ymOoi+bM`lptg+k+7Sfqkt=syvi68&D2#` zIoXfWHj~$@Ep7ip`K1p?o-g660R;PPVc{lr_UBw}x^L|Lbu+kmD3euh>@5i{hp1}E zFkiM3T}U=1HNfL8|0VCmvgZBgR(_+ov~JJb0fj5IIdN%eX=1_hV>~G_k$L0BavUFw z?tfyluyI-Xbh_}-Vu#gfc@yHMVp#O%)hkE{ZIvrz{{w?C$Zmar+(|y@JOzyNm{F|$ zmHE}0dJ`%rKmo%=^u}c4xS$3P_YPnZ#0&||ETq|^U8s?aisoW7kDA#uJo#n7x}Gaj zT2V+t&b22r!qti)L@T_x%R1*gt#(?X%Jgn+!L?hCL+HK|amjuD&r^wR2>#>MC3+&S z&Hcra;s2b}>jUoskfVO|^u+TU>89(h+qUh=ZubpH#y^l2QD1AN%Ur?S!%C(U`z4SZ z_yi9kxi`*kZEnL+N1uK16q~WKT0yb+@+X_ApAKc26+GRlcU(d*R zKWfET#&yiF;O9dSYMj7H7+kSOF3uU1Qt4EuH;V^43(5`$_8uEEHe=XfO(er_-lWFR z4W7TEEu96s`1l+U958EBf@C%Vw<%PcA-Ty{Ir!w^!=@Z-tv&k)ST`I?P(Q?wKurdC z3t^y$+AI|I6q_s|xHX-gj}kY@<0&|9!>-{aUCva{*!zHj-|)NAE+y94$> z=C3idY8TIm_nb$454$+ii{4x_+|g_qh7u9sgc`a7&C8cCGO$*T<^pO=1eeiAAAOzC z+@kplIprDJ6Bu`CWg1GsgyWUP-=9B!!a7n6oq<8>U(B42IDG|iBM3(h;nwVyzY_w;XimB_?6lQ$6AUaaO)$+{FGSyg>JLBXNN$KJ@~&H7hE zuYZ;zcSd74@#=E7j2+PvCP5G2WHH|eGqYucW-0<7SPg9~k5nCpihah;kCD^~ z0}voVFdY%^ate6x0BbW=nEoC*e7J5wa59>hIDd?;MFrE$Cm~~b{yce;7{#{DUYQ&$ zqb*LcdmWos1;JydTX)rW!EFPgUm(CXzcf3}VHzU9ziSuF9jlS<(5SmzpmGHAn$!0a z6I!zH*JNe8Vj)+vP;ZGaL%a?WaJ!o`)oZSf4iZb(mfz_qx|l@R$rfRLyVv=q zmTpWInWlH<*_i{=MV(;vPz)k*YB)krCVDyh7i;Uco;=xxd_nA@8f6xD|6)7DN3)X7 z+uC`#^8JH?w4biy7na69?&vBlbai2k#SpMM@RwhWFrEW$12*hyFsdO-3M+eaH2(~z zh`EZM)gR95fB(Qz%)iFeq!NQFfWE(f|3;@@_W85KAlvR;3Q;~@!z|8YOdleLS%RSM zb?3!b*Uj>Gte_hy6s(gsQgSs{u9br7;TMCro_Zt$Y+jOkWrQ#Khx^ls407`4D z@#r*@p}!n6!{{~pLNhs8d02gK;<8>^xaO{11ungBx{ZI&goZ`jyRRk8e=<+V?BT~a zy)?HgT@qq8+JT~VAnLTj#W%OH1RSR4h*IegvlApr>_6HU4fig@v*01ShYAIn}Zxfc`^<)Yc=0cx8T#PBq5G8(%wp*s5_>cc7`Fze>c ziNs8r*xi0hM(j<%z7DedGp4sSoAB{7P=!DH?lslG^oHKa$*F&AEE9Z`7bv;YBtPtc zFbvuvZDgys{G)6lx~dO2j~W=|JDyx)&?kgEqys*vHu{tXhfYB5&qqEb+-6{4&;&5; z@s|@JFNypz9G-~ORO?%dJ1{iX#h%b>*rt_IwPN4v)n&>ZJH%xqC#K^2)Ti2X&VfkXrE)il!n@3 zxZ1a?D7Ce4d?e58)Tty+_=S@Mmy~!ujJfx3vRcc`2~p$i>v#A{ey{l~TZ(XpUAOeE znaeoQ=qg~KqgXHqJa{vx9iR-7>l?Jv%t0(9@deEB)G-4VGU9MU21k!e#*6r^{Fqe} z0SA~xBmcj?Mk)K328{7}D+PrDE#DaFn+fI%(p=&TRjO8dK2c2U42KI9Uk0>3$k-iF`<66Vuexjs2GT48WaAiIM60vKI&nw5=81qh z%E?+39c~bJcpNwQA&O9^|Dc-Jf)+W4SBPy_v1b$!K*w;#xidE+f)&S?IJ3a;yDGk< z6CH>zbZx*`5rNR2>YNiC$Kd-yM^9f4UD97{7YEgiT$0)?Eoh28hCg0r72*|nS>u8f z0N?s`uzy{li>1ByK&P&v_uaUg*LPVht*#ioAkNyW ze2%v{#8>je99CDhp9{cru&wm^Dw9Y)?75zUUl$RoBAf`9Z;{KF$C*m0fjnmrmET`U zHLQVLvY9w{Eg6H*`>p%>RjU0`JARSo_wV29nw$4`d1e^qzryNN%nK&o_B;3QYoU$I z-!Cb7`b|mHZhcKPI8+*yUh6XL?gl{6hVXzGTpDBMA4S3oK8XIt6PcmW(J#o2#8kaG zV~$@roOs56n7H@pktO0sT}z92rTKaQV(6%TyfC4YeA6zw*UT!Vb)ezZL^{UG_4GY;CN=nMPm$N!=eF#4@(tDCa z5E7|hlIEQo?$mn|SUMA_`PFORBuZ5qtS=0QdckQ$2Lupw zoZ6Y|77=S&vGkDg9lBu7-D|Ogt_N~cq>C1NB39gl@dBJPOh|ko(J>J8E0OLw^ro#K zc);f_X@AjK4-i@PrTog3RaLd|vGnrt@`gW;1Gh{EP?dfvUvNTJN=l!WmR88hYS4MS zy{=Aq9hZzD9{*`wCbK`WF7~{V_U)-)OBmprrP!XdzHlKKJBV>EqnR5xz49fTIJz#v z#jN$_+JBE1QP$vo4PF=f#|O-^E6+RCA3TUySa#)5#+g7ipualrvgY5ZQe*kBhzQLvfnQ_an0Y1d1?i<- z^X^*sUX-tS_wLt8E4Tx;f-?{?CZ3J@WCR)9eb*9nI5j{kP;)hV_wAb@PU9zTk-57? zLj3$Emw`a0K`q7lvn6Ip2TjX-UKA=mw?ev01UmlQC|D`xr4%F`S-H6Oirg0yM7+?! z#6@||8%f!mb?es0zq(g}5=7j-eX3y21T{+kuU}7Wg_Z9W@k>bP=I7@poy)?v$s!X1 ztIxm4txGx0Wr;Q-vtk4S?qq%Gnh+-M=9bcsd2=xvbf z=h3&isiBsm@6b;ZI$T7i<1w(8+puM-?d4TOGBg7tndBG(h=fu?C|0)U;DpYfc_845uzFA#&0v#Hp}9|z0*wajwmYe{7KKLl@ z-l`><#*rA;?@F6?o;!Cg7g@Xg`b6wela(UB^mNU^#t&t849C&TreQk8paF$BvFOBP z9}1)UiW(z@;}+chZnrSwOa(F{D2Hq{-1xV&r{{X!kGN~0|DNT}f6vl!J(q4>g}mPW zL*nXZg^Z*lVHG5Z#`1u`yX`x6v>|lz4_vPvf~2b+l7ZqT;?kXp_*J^XH9uCUr(!cX ztv#~f!oeLPxyK-}bN%k!;dKF{ViADSgq@tG4s=4<(zAf?^iyq3%}_L-nztr!0HKgR zra9E3y%(<{!JwjCSfB z#jk2Jp01!xs;@U*iuTLM&@3t0&(3@Lko#n!2lOE~#}j1wQ1!+q63b5JU%q7T50*bt ziR;n^(S>v)wogy&j_lVIvbF7!IAZ4eFRpGkPYlh^nVx1S3_kSW4s z1eXiSFaG$~wo_;nNGD9x)36xXyPW0b4gT=_;-c*o(h??;M+i44{oSt#b3qf%Ok?kf zZudpIhtg~YIl;#^x_73;RW!=}l36=9=HHxT@~a==L>^5qC>U)6`UVF3K`}$^-!DEP zMQn!9#~s2|lYmb_AhET9ZajT(&A*))P)wjarTPR0OHzd007ZU&xkiVTC3oc6lWuNr zC9<$T2gS%AeSLhCEK()c(3(bKyV-gQxX!=r-KM8EX#}a!_WeZY>%-;mD8gxeMJDB+ zUkFjMc$fi-VA_5t82grXu_-A6#BlBXdj+a#Cg>ue2D^=;X+4F%tOPMFRI39YioXcN zfeW{H*6{S{u56s?{dBzT_X7iom^lQ)K#gDnnh0N)rd8UYJ46NS$8q)}>=!k--m_5gRH-cf7I$z0mm3z6 zUEYpa#3T&WSE7ic|Vvtyi5jHE(;*KI9f@ISGf-L;_4G2HtSPDRn zNZ~)oiv3eeFHy7$DXFcNW2+umT*}uHTrWj<&_KJg`+jLtTYB)w*`Y_@uWggC^NkSU zPgUxphiYLJcr-|=L}CzlsyVxoT1ZIGNScZbS*flMF5leCvB-lKglMTF484TX-}0Ba zw_f^h)>Xed_Um(j#d#Ecyla(6|JBdIQGU7f*j<)_Fs0WQilVmKn2O1EyDQupg zRq(a+x-4w>g4tKsV(_rC@GI*8+ey6)gOg-wY+-*y6w4+T*0;td8Xmv2jj}JA8jL!| zOa=Msr6p&b#;c>gl+?zNQA8d925A5q1SmbQ6T^BQ16ho*xPTy0h?@x zzg_n3q_60t9~OLi>7smkV z(_|F+K~K7#VQ%b3p|s&!ikV-)jSI%j{(Dm*hk2^fJ6|4qbJNy7vygF+_8cAj6MR-7 zG1CzfOa!`QGmnE zO$-4s>}<&`GExL@JS;q%W@zk|@#_ogaH1PmlKL#V%4@rIUhi*Bi?H|l5D~`BIANlj z^*J{|{9v2TYbWQWMw7eXS7U?l+v+0CHJrb{&>yhw%*&_}-$bMwF{;W2f!OXlI7d7> zJSQTzB1i&#$dE8X&ku}JNT53@YZRpf6snJ^bs_*7fa54w3%eJ=r1|-uLHaLEZ(M4B z^c&Ji&cDBeU^C4w7N{X4gkGuL-%Z4Aag3XD?$6B5UI3}X;GFry2-eRqRqR1)(NRRU z$C1|quABWns~E94Gh|!F2~L-v@7JM(k-GYu9cZ@&>b1@L{HlCG+-s)EG`tbH$E;y% z@VGP2ewff7M@9H{)519R(XLct*Zli?@ww2)`^(`z)-v5)a(z_>?d1iWVLW42k=U44?L@lzxUBR# zezr?~L{D}^{jI<*tTa1}eRGN{f%ZGm!7c@mLxK*eM|1w`ylR8*8Mz4o2@#BkzEVIBGyC3`sn zj78x`>~L%$i*z?qg^e;zN@MZrLV|<2DOvOfx|P)6cTE>?yFP=qI-d;no~tAb`rl1hO@h+d}8 zYjK)2%9>%!jby{gj>`>cVj-lJM$GxSXlknIv9oW~L*e2hKg1e_4XU6hF*+XON2x+^ z0K31XJktnr7)6Wr1u2AWj#q5kf_ruzRj~nYpqn>Tmf{g65m&C@R1xLTxw#b73d5dG zBD7rvaH|?NscF%QYT~my=;!7Rv(uGjW>(rCKbl?(JbaYo*@4Rdt8rQg%2EP0jwIqv zx*5Xz>Mby4`ycIujIIV8N!Y`M%^u^`f}Sf;aJFp$afOD4#&m|OZTGtmZ~_D%zXhWn zV%$lAIZ7@564`y-<@iI(f+cA)G2Mo8VDkX!BgxjWfwmy*Qq|5KF)j1yL-c;Ls7y> z%NH@Z_6kc&bVH>A;p#@z%B1CLZPA}eP6-Jp? zP?+Y&oHaIU2U{z>jTz>rx?*AR`S5QD8ZNoO7&_`6Lg z+%XO(X3svNN?)t1Z)36}?{#GJp0q_SFnwGi%GW(T{DAop0)ZzP-J?l@MB?~M(8HCJ z4wt6AvH#Y@e-M*{!XF2BtA?_mKf|;D*e1w5hq=6`H-etFkZr0GB|kS-FctM7>X*G2 z42*cD14x+Pe_ONX$5RnNWmMZFeY5uwZFgkX=Ge~|yZ=|d>HkKXk6Do;*2yVg+x?}$ zMFiDS)6u8S)+oB>zEshmJ9!_pHh9~kRk{fKf;WYB)wrX9kAlgBAIO2vV|r3(x36ZWN1(& zKHAc&gMoo0)>n0(jKQG=r#U1s|B{&3yaSmgsLi1TfHnC^s-4E?l$9BfqhFYmu8dcj zZUXp7R>f!pwxKR!2GX9<@eH=eI7Su2H5>oT-ohTw#QET5$B^Xjs(q%9(({bhowMKCPg3QG;92z|P$tn&&04Y>L(BDHPG;)e+b)cbOHtTe3SpWxC z&6pyY<>`v~2tZcv1S7%clzLee8)RQ5%hM850enUZc+&HtvhZp)IeHES0t-$F&}ynO z}NRGIq0>_E?;g$5lQKA^(u&*XG0&q`U+XlhX-vV$0*us z?qHw-=}i6`8?!9PY&p;@(MDr-(RJGt_+sK{2IvFa_Tc&QgTB>PB0}WyFhJ%P`#NNO zKLbNoDveiX_lY8pcI0Jb&OFG?rBUkpYOW1BE?CkPV7@A01q9_%`zmdz>uFb~U$IAF zvhH^qa$_|<3?V-T95>*KqX6HFQ*#Ox(C39e#n8DwB!#FdcHS>@M+_e4)0BVIfqbmC zOW3{be8qoR;RH}`Zwj&-x_*6RV)CV{@J}RKz{nyL4%nk}Qf5$CoGnKy*jHO{4=>}z z#7w)G{r1hG^{#fE`)xk)$_dl<=is=K9n+7oM;YaiCBGDVN}@In?YR?W5TvPRnvTU+ zbVXkjJN+-EgjZm*4wDROm{Jqr1RxWEc*x;^B0m8dI8ZHy92;iyTI!|R9E_zR=3Cy?PZ(f~K~%Po@~J z4lo@$RE9DzS7d3WFcZ=C|3ALrvCCiN>@HtMnxG`COJW55Je>C<01CSbs1kyxxs~F+ zO4LBio^$Z-;5#I5aWJXp<_4&Gs#C1VBdGyQo@!Dm1EhRS zJlLOvErV4HB?m`25QBhpSYZZ0u~|sd59e%hw)3)zLzRk@+aGRtKBH}>^TFxw9UtGj z=p{9TS_%H&cjo8aC~(PqF!xb#+^5r5K3;%wg=PEp0J!*qHo7n1=Uq5$8%e`{y)OI{ z|3MCUf54$Wn)jeU6AT$RjaL>*;K?ar2iNghx&1dEg$F&Y0JCamewBYST%$_I<7XU} z;Kp+zSy@Tx$4%to(8rVT0MyV3<~!IQPkQMqy(cI#DOo|dVCNLQv5{Aro#To}6hM~@x>VPsHVHd^LEQ;%K0+uN0xgVf;xp({RjNf)XL97q7O>hzv`Z*#(u?eryS%`5{ zV7f#k(lHlMHg)_0*a9lJz2CZ900@8&yNQLJ=hayRMDLyS`+^LW&MShK6^s_`K7 z9Eh4|KH-EQG)7aQWp@TUUR!M-VJMkcLt2ZZ3V3;RSIp{0`y zzoQbA%e{_F32T+%5RCo>eUMM)limclXK%8vcwG?vF>ag%bp) zvKG{MWRC$Qc>j_-|1SC4>(RT}?##csr>}AEO@2S-3T{R8z9*7NJAoO{E*>7rCL``x zF*9Eph129*dwP13nt`aJ09ptx)bX#82P|%PcNX3Byo@_5cRC+7}x(}>u5Y@_PaGATujHaKL=obhVLJgyF&nC zuU)G4UMPb{X3?SFUnp5c>IlLn#_sq|fi6S{BshGe(#Tfr(rez}#*5&4!{BeXNy&B$ zbScnxyuk0&Q5llT8{)#$1x8fI$)bGncY9J<`ZUBF~e zA`7r}PdA}Oc1LkY0pHooNM@d@(xb(1=MSW&q9zXDk~OAEutNWT3KapRQN(@Z<|&wU zAte$q$xk47?n$k!tUMn2cslseBMuZW74`K@K8Q48;CRyq)U++`03Q=%zqKMFBDksR z1X71vPP-_n>FOdUg|_lliIay5#uAkLDzGFoy)%{Gbl$;8Gc1CYhE@p;k38MYf5SG_ zeclVs=%OfyTyE*o>wnTTmu##{ei*C=J~59p z97gDx;x6qwPlIWBNZFkOD8>?XD5;3>5JXh+XcRUqdzv@{2giLqOSQ8ZjWi_&rg94> z^q!g!Wh0z^3Lx8j;6&xpoIIj6^3!nJwrx;B7diGx3&o*g%T^Ceav?Erh6?7zQoiV9 ziG>chAfOWLQG?=LfIrths)3KRDk?^uX<*&Rp>RcILgEu?8vu5pTxpAnv3_`%wZ6Wr zn``TVf5bsE2hue3<}Md{SCTJ0K$YFJ7iXWarSWB{3AGH>1~03CU(E@)Xgot{j&E-b z78<5GA!S3^pfzYfadbVG43iwnZd~}VGCDa|o{I0^W09ps3&k;>1Vb4_;9KN>?fk z>}nD%gQxQTd}35S$ZyDG705#r*2k>qz}PACp`^}PP6y#mBVBh7eSqRmL^lut?jwm( zpz@qa(e$m+z^VwLAVXW=6_==E1`66gKs?Ot?bX4PMe}zny9Xrks=KE4nJTv&T%!ok zNi?Z&^@$RYyF0OUT0Ue2?EvTDXXz!gzp|RrqyANxH~Fl30V3EzFTg+(Rv!nky#0{o zWYE##ZOhjAbEq6hEsy*Y27Qeb>vw-4uZ;6VDJCgNr0oqtQ{XfmNg;f~!Qo|Os|6|= zOsDu$YbjJQ+)qyfbhYOiNY3>4n#es+vGIV&M$&^kN@Dqrme6PH!h=>FV?Q?=a3t^n z@YwnAkB?;J%sY20&sLta7x$RHi0jeV*ys@Y_(DB8mPZfELix#|cluV~!Uz~1UlpW1 z;9q*OA2}{{F&y=jS7z{9Ed~IN0C3S!!kBKt0P6c{2;L+nEujc7s0e5>A?Hnt0nlDH z=h7UqReEU~ato7< z&HdbAOsO0I3(*H~A3ppT?YDIcPS3lF3VEu8L3jg_B?v?k3cwMVOKhPJXd#%|l5h4&=)E@>bb`JJ5%U`%`i${;{3(rdwu6vK_Q7ptXZIE_$r?Cx+J8 z<=adFcy*9bisPaNv|DnPTfmPIBbf0$({FFD7-*K>zt0G=BFkp#3ACLgC)#6XyPhqoy^LgEH;qwE~z zjk2(30$KRTaTJOF7s*FtL9KQEd|AV~v2UOFcdg7P{CC0yNDrl#2Z0gjw=(%4!r}*q z#j^(^lO6H**Q5hrf{K0OKI%O zOUT$5I{9Qi&e8sUpnmxeSOqoL-qx^QXpC*ZTydX-#D!1KPU7jj6Wj8waf8f)Vtby= zh&H>J+#0PrgX{qJ@usn<6pqYDXmHUc*i9{aM=NSXb?VgFqO*G5YA7(p9475{`MFu9 zp58x$=9WwpG?)S)^Ln{xVhshEmQ+`t!uu9c2JYH2)=Nxs{*^D(5Q91d?1B>U=JvNe zvA36jPZ-$N@}9hxX#G=|=@hE#;4Oau+Kt^T$+W*T1pp1tIM@In< z(<#)8?7<`QzfdhD*@P)NCFsDQ3OYccH1Fi9C#)BV3GC>F!#qs2DAOv?QNo$G7rZES zr=J1Fj7)x_M%+>yC5!82Rt^7sd4NJtXDe@R4h4panrVDsN&a2+@yFAlZ?v}`CqyICg# z{~6p12qbLxUYP!rd_6q0%l$Lf({rkk)e@iugYz6wp!OVIEug7I#P=NoCzXuG*K5lb?<>E6)`F6h+woE+ukyG2)C+a|H^_}S6;{1NJH1M{kr^iiK?0v)Bm`hlOu!f=l$5;u7-n4o7nhl?I zE(#lHN4*eSJWeJK9RElyIj9?1n3?0%RCHJ>QCyZBFc)N_1;~ntvKeT@6zm)vNvX3t znY4hX69*IuqJ^JI^n;X6!m72WAhsgqGB7!}-vMyof~1VIc+f zWGqM9)Na@&aVfa(_7P!ta6pm5wk7n=rFl?&FMfT+PXSGfC5!-BHb0$%Tpkga$4-&g z;^-T)28tX_hLH>#xM0yHW`2t)nYl3=AUi^*qO$UNL`>_zeNL@dld)QV)}Z3?!T*1d zcaeh}HSp_~3hDp~49>JO40L?Y_mHDHx!iSG8q-Y7a1=HC+~|qyijWY!59(5LD=UFx zwX~Wh(@;ao)}@#0y9RiN5cVrwj-*H)KZGdg0K7cSy*IFEKjV4eGY z5LYAi!z|e`*Q2$!cUYAHNnJ+fk89g`o)aF#6w%G?4-D&xr4IgX78m2dPim9v>vrve zr}6)O>D~WsZ&c)hy;Ocvlr&K#2Jmg$Qo6cQh!f0~7_K~(FX(!AkkhUsr~Li(ezq|CDH5LiTH$cH`9pvrp%}+12 zLAh&v0TiqaShK!?B;yP5C!)_c&@ndz69x3d6mPMRg91J*!e}T7I6sjMCFLd@&5%yS z188O4Sq$?0I^W>W=lj{jB2`YxA0aC(4p*-HK=J>!uuvTH@#V8Gb8UZEcYe@+?QF;I zk3eOpJge)^yqDm(U_mgE{_4l@u+XtuekhEj9yNYtao2D&b1yAmjv%xq)TE~D_@Lxx z2u>TX8jBz5FMMn86Z~i2;tR0X#(c=fM%U4tYB;Y)F2@|gwDDL*hl+<~ZXa5XcgiM)jyK}9YN|bMED${f3Mh6T;FkB#L zCy`@-a>>JyN{QE=lCbpT<1hE42aW!}3F5(j_c^>(APjmDzPZ2w`J)uQPxaxhVb)$s zUj0juI!3P`2cZOwitpo1-P8sRTRv|0G63RVirimIbYWwKURksm^*rRhDM{o@+F``G z5w!B$eNZMKvg<72hP);vUYPFft6bh6vPipjZCG?P6U0#-2x2WI(=!OSnSe&fn@!*o z0O%kk2at$-8L2mfdm>ZK&!ImQhWv$LI9lE#1Tx2-v{(RbsWZKyVNV^a!_`}5ay3z> z*J$6d!dh@5N=dR}^X2Q;joY`s;A%e}$km=33RHPv=4Q6J6rumLcI~RXgf4=!`<>-aSr-ajy^UtQ& zT!jP_S~@gmtE(_hVHL;%K$=O0wDno4!E^m-24)jrpVg+w^N&VQ(%Ee z1_5K;rNmA7>{|yOL?DN|{lplA>SOc`gIv?D!}E)4II$x1fAOva0F+Xt8vlpO(5wuu z)`{;@3gOHcH)@ZB-9lsJp4rt52#s`Ox+9!DSP8=hAVtskcq8O#9u8ngm|qDZ$GU-| zdH9_mUm#h5YQrlks$O_plP=^ta&Hcw59k^u9Zmz9sKX6%gV4=E^o{#iEfv6bb97u_ ze=uO$hNd+1=b|)u&?YT8RCw?H$&x1^TCk^<%s=Q9IxvGj=a=YG}53CKLSbsg8{ zQ7>CkZO}poSdEOoyJS?83<(sgy@EI zITtsgf~JW_3E1(@9taYMb=7DM>@>!qps@ryQ8piJd*CJP5-CBVNAodyh49+A^TT2H zHhWfv?X@~ELu^MMBC7?!jyOS-CEy&vy$x*NzPLIypa%SPjT~HN(8-&{-d)9|{+eUG z34>Q&%*=yi7VdyhL&15(tqesU)R9OTiVkx>mQSz(z%*IB1py4$88udUiLaFfbv-YH z)3)ILCn5hsjp7#O9Lf1g+|pQ;8XPu%Sj0PX?d-@tHvI%l_(||L2kaNV=UgQEG%?I9 zG1^j6Rjr?0{?oc>!|OVJJ0&quQR74)fpmaZ^&v@;-PRz%huZ=y#npM|FD}Z6}Fm@fQ~ z{Q@=FE;h2g58&5**DUS^ksIxB#&iKXqm<5f8}UM(fho{JKRr>^xZ}_p;we3Gd4j% zIu#Wa1mh~);UJ8lku?bxu^ausmoHy-FD|Ca7Vg---9pY$2wys5q#LjDm9wnI9V;pC zg#&~gR_qZ05+7R8$pex{M>785?9e5ui)_=e@%Xt8bI91A*8Pn*r}#pV$@i`VE-X2n z&1GQN(u%C~KP(gXz<+H8(iamc51m9qZr-(AN2L493c!s997~H4)ECHx&w7V zVdDez6eKi4R&_MGEbKrY zGc`5M>v|MtE?xiry@3mx4{nHxgwJm6G;JZ=sK{j2E*uCl8NiFJ0G|{&nrK{r#`6v} z!2Z$8DeAgjU3LpdhM0?n;-iJ`HIV^far*EW>iy5NGqDC>_((Lqe}4;a#q83hdTi@eCgvO($RM}vI$Tz2%d z!0B6$#LK^JhXMBEga0L2rDQ@Vy|3tfEFO=;K(jF&?LE9E4{Js4fiN1-6{v!LRrM@I z-TdPL)7W{B$6s1u?nR_+G48@ru^AX+0Qw){I8oh-`4Ya6_yey@L%Dd?jWLAydZ-s( zykKjqg3g}l!dGmr5+`Kjqo+7$S<`>>(Y0aaXI&XUW6+B=Gdag`r>x^92>TP!_SHs< zr%=LXhN&CTS#_VZD2R(Y>KRGPxaB1lx`Fc&EU&GN75n|d+Ry*gU6*uub-tnfkUq@n z^l+XdFMJ38=OQc^zQ11rK;5xojLH8OFfp;z`7gT19Mw{rO_m0 zs{5xUi2mGpJUMg}n&<`$3q`e`Yfzg1tS_zph?*D+aFD|H*Sv?y@Dn1{*ZEgBpuU4u z-YQH)i9E=B*H(xelDnLndyptZ02*~3YC4wgw)Bcq_F6rrz!YNp_9=z=h&m2ln@2v$ zok>NR#IiZWuDgL;=eG`&n3kf}{f~2eTD6t`pxqaWyu1Qswf+wud>AIS1MP*3!Ii;} zJJ43n1GbqlFOL-U&nlrZH=kd2b-4Q9f}zzvf~!BYX-;?=if?| zZ`8aSL>*fJFRZJd5aaOz0g% z8Rs$M$*tNoXz0kB3#BuB5D;f8pvGW4?4(f?P2vGCl|lQK)9F$fkm2+}66*Q!gS=!U z1MlIyz3774IT0d_vchISPOC*Kd?aCKkZHUD0YngQFS6(=9##%b8wUMK=d~nkK#7JL z!}SY}En1)%@gb$7N(EF>fh{;S3`dDb2{se=E=hS|A%4^|8^z;Uef?VQzVoD*T6PE- zMSTjj-(IXx6FUzatD@E`Cj*t#Z^a(vImCXC^=8A9^0JS5bRk{H_x?EII3YN~y;w%# zzdOIzZtB8N*S2uq;g<3it$&+_vH#Epw$O#XGZzfYBo7{Z_x(F(PwrwfTIP#Ei0L`^ z?`GB2)Z78ay$_`G18lP30pxqk2!o}DDWbWh)nN&jnPtM5@o^da=zu4_MO`GcyI!+cEk=JU?@T3qPMfJsc0E4vW4JUsNxMGXe3=q z+drc$ga|GO#o5oC^`Ad)N!pAK&DwY`x?5cDW=!u6*ePiOFfWJj9Ihc@;o)A(Ke|168|Am!Rve>N;@<3QbM?qBvgnNv?%x1hzvSp`pRKTf94!*?~+_z^mIi~f;Htf)aBr$LaC*!t!*a1 zEt<(%*089g)qM9|HC}1M(rYmw6*xeClIZ+l;RW+|pOw9DHQ_4!(n|{7*6Cj?$|Q{$ zLU!QugG;+dKs`Dhe;2&Mp;Lvm-iu&6KmdN@PSYCZe**>g*nf4w9hta8zhT#e9R$K8 zRI%BLK|5qe-3G?Lm3TivZGj_-MP)jG2}Hja4OqBlG~zwxWm_dKJzxIVa(^9w&62C9 zhJX>{+^p$A?2{XN+k*oEJ4Kp!=_}70Go1WtgpewmIHTHsz8WJR4uB-4Hb3C~2s!~O zsy3Em*z~6~I=7a%!$3U(t;k#FVHTiRRHm3#6I*TU@?rM-0jtD0-CD}l8NMrxaPvBO zW$DU%af0H;mwE^C&YqXFxpas2nD*}HU(Vm%`ry-&YQUbYH}7VPY<=+T7W=t7>$aIz zJ{O$b>Ur|y=99;g6s5*y=c@Z9T+jR%;-8XfJlgoZYwY*8`i76cKU&F{V4^^NBknm$ z%4$rpxc3T25O2;`P_V>M`g={fW@=Ssk`I7U%^SjPnJA1hF*yRL7>iD$mk0j9u`{1Gr|$bd}(D%Z%8dicwVD@1}v)a$)g5NLCVlawu&yIc%{6*=@enzF9Yz59Y-X0q2=DN_p62Z?*M?*y!zxgCL zGhF}==C0pA-rqEjGK%z_lQg|(lLe(W6g>brfM_|R3p#}^69_1QNSLtm`rQi1@S5r& zKQ+KzQL?t!l!u1NnnhL`v#^y)v*p|Gsc!UeL9e8L6hnl8uU?>#sBYOvmW2sD0t5M) z4Ym+QxsQ~|V5cEJW@{#^M zAPd|?RwTVLCg&e88+KcmPA!Vtr!FRSBR^jXvFCkN6^+0A4U7+Q{(JF(p=DA$!wM-J zpn@poNwz_{XaZCguAFp4ePNDW0E8cc%QWHhQebCLl<^7f3nhBbr@P&D(+#!1zP8Lb zdFsh$DJhqsrW^RUKcI;S4hi%ILW+V`t{S^=7Xu^(S7@y%27U6a;q=qvT2Md%zNvk#Ga|Jtf4 zb<+`VhS3hP#lyKiWe+(p$JBYZKcS(zai(vE-V1FLmLB*D=>#`3prEJ*!X$+B^U3-J zrV~VL9aj`Hc>N-`eV8fm!E6>7xSX_9mS$?-T3dAUVFEkJhDr>T>D2CBe`u-9V-GI1 zNbLXlJG2X%1<=&tB4st75*i(1!=kE|TUiDT4M)fd|LsVXR_M;Q2QfGr()1{kC>-#| z`1?ZF!R71TMi*0GU;pr^XWlUxtk_%5MWLSfIXb%M^hIJFLr&84uvV+hK>+Fm_`vL$ z`c%xfDg*^BEiD8VAT%Gu|1MI4Pc}R$j20eBW@;m zYi>m=7~aixVC|`8kJ*`-gATo$`j`Ns@<6?Q{f+1|_`UdnQs~~f8L!?rGD!3MCuGFr zLizq%Gc20dvY?82!gn!7P<-2@VX!M~Km;IfQXCp0_>m`d+bz+ z(8yOt_`kMu8RH;D5FQ*282sglFo9P8MRW5oo4KOugwD1BD9%-KKR!99glww20XZS53osPY8j|HLNMvn|Eo7d=Qu-T0FousLR3r74)fZY?i+2!Am_gvDB;RTxh=!pA% zc!y>GPEI_H*Zu|cMR>DZ`}qaEIQW)KW%z$&dGqh+n{K9A7z0w0U5m~)r;S5OFe`ys zd8}1kHUtTgov0ij8-><|Z>#)zqmPXqGj9_hcG9tL}f4Km) z&}Ug^Xb(kAX+X1gz%czCps&&7YP^V_0g5%tx;?k=1_fCG5Po#^l1JxE0U9kJ&1cu^ zN3g}HV)!d0{RxP%P<7*Fb93r4KzDGN{4lpFfSwb59^L>x7b0FCbF^=?CFa_=L&w)! zKGrAc0MHzM2NBmM)U~S@U+3kC{^=J|(v18a76)F2*Dvpfa>4q>2rKCD4?*z~>Sla` zf`Y4ue3zHR1GcSI=6_+t$jSLkp4RsO9TSrU-Vg<-GLVX%D{A^wd1vFx%Ut2R={;xV zu3ZV(L=G1B+Km#jrL;8A2A88DQDw*`bYe}}tS~x-(3b3caV{xAu zVGZTm2jT?*AOULys8@gbv=N;4zuq_*T3w|;b6!A!{M3j!5)U!EBqU|9H2z}ZtdHT} zZh43l@Zf_Rg~cokwL9q^4cJ1*Fq!-5NYJ_CSpCcfE(7-EJD?uj1&x5aKtkOK{PHPa zs>^`D8WFYuw@SYA&Sjqi_HEX|w564I(#=3p-Ni`%Jf|>%IAej)W5B=lQ@2;XY zscUt|7!?&sA5PGkxMH^nUled=mX;A?2UNlZVL!1K_1URVK0YOY&o}WR&CnP)A12EL zfV+=B)gZh*0^&%8Cn_gB!{J%8hzT_@g(XdXMuuHpy9;E5Q4|fJMkB*9=#~7l>%@5B zIAA?O9^I}kEYgEfsw^njys}`03gA9oX+vq3+97D z5z6R(?ttB~zT;H*=QIJOe zBnFZAWxc=y(Q!LrFtha9&9Y%DyY!aL3};KWuVG?_LtJ9{a7zmQy8&%7s?cOx9_PIM z)}eW7=k&447R)$D`!BWit@V4U#Lb+jboJADK$B$=k)#P@Q#)p}_Y8!EH z_#RstIMds_B!E#uDWe|0<q&_(3t6jIQBnNeFk0~s*U zy&eFlV`AoB_@qN8nsB9LrzR(PA=}ckbzp1^0<0#bCocf^slM7xqz?dJ z;W@Ba01HVN*>lw0%xr8(TP@{N!cHP#fD+48;A{;3zUYR?q8br*-o24Oxh>PBA{n3< z&!sxE{X1=%T`e7*oDwi4AOEIT+IUT`)Hqnt_G#O7%4OTPL$7J|kKq8~BtRnjC?<#m zNYw(ivlQ|-h77mnZ(&)&Da8?1r{O%H3^Ex@z&cQ*ls1rYrJ7cy15#4KhR%5sEpiz>h6NYsko z%@VicNW#ajZOq45ug;eOg(ND9LoOrXzhwwZ9P%pa7zJ5ICF6;5+))Iv$X@JN8zMJo zOd_RmD0~5Q$UW7l8QTCkiQv+?`A(K@^j+whP$n#Y4%UUL*JZl+NzEZ{@26JAPAh`X zEl#v*@(@`-&EQ6wI95@u>b&%VGfjTNNpHXA_u#aWD%7h zGTp?JAXy$WkTp4GT#&;Ry1iCga5JIWqY}4Rh+fUTJ5eOBa1#7*UHsf^w~6zZ%{C%< zAqDY+5Ab*F@pewmk0=x=hCow7dOtWX189NlP|cODf+=)-dZeB&|LY%}XnF+~cI0 za#AQ+jsc!Yemc4!P&LG%!l|g!Iz7Iq!y~bJIXKEc&3EaxauNyj*#5`9MJ)jSQ~-c0 zHA{gznzFc_w~5(qNn}ws=4QS~+#OI*;TJcfi-pw4l)`(7%eVHHG#$>$FrXC`efA2; zSdLq9RNqIClXA5`Jd_o)tsg902wyVwR6Yl_5$mJV#+{I*}O_Acn1ecYguuSi63E zSx3iX^d9#(OLJ z5Jnpe+@1_n>8kVksY!i6Lk*xD6c+K+u@ntKLf*uv1MCihnlefGS|t55iTjKAf)PYW z<(qd9-;a)7Qg~SH++FtQF>yqJ5Y*1Xj0YGQBV-T8(LZ`4X7zD9HVK~VWpSdS*$HP% zeCAr0C9JUgA$_HW;7fF2%nsF6Rm9^H;Lu+bh!3{(-j|4(O4Mt$2oMR&p4;c1ngw8} zWOsaP{TzWz~Ba$2% zm=0pLDyWmT8F9%RdFQ#dFvf9Fqwb&B+AJ*DU5<_5qQhRZ@j%x#kgD(E(~-yU_N~R2 zh^f+^w{HL;++19q?im}475TB}-66wZE*u4-MS%oJ+(T!_x}KzaUo4|*DR6Aya-n6V#F{8c1znUjfwS6Bs|B#rL@=Kre0 zg-zZZ>r`gZ?2F?LGY!ItSG;RWHt=Hn^Scn{s!ew}di3a2#>-SZ61=VbXve`{4dJ_s z2=IZ257ciJG;r9zf4{AI7e7Bi1SoF)5g(T?jeY0mBlqX(^t(tPUhWccU}>IsA}MeyW^pmL<8kso*NsH=3V2jWBXo z@mbGK;c?(|Mx~V*vk3zs3dAEMvpdxoae+z^+XGvPfimhSoS0VWx%&YD7Wn;>Z;KG3 z)^uYcKnB3!%KWGS6YzwIq5)cin$lW?LSX4N)dxQG2eX!!mjl1qicvk1f(j1ax)afW zx*-e&OD+!!Pvhy6qR)&5nNVdXA<|#AGe;lH$f~T zDw-(bHDF?o0wFsu@8fcSQU2s<&y^*ohhoRHBMFp2qrmN_h9*Vr)%nR1jKm&2c3wg! zw%^$hq+CGow}0<$#+X<&sv5Tnwl~g^5tlLi$Cg_PY7^_vWG}dX4&S98)e!}N%iV3P z7eI|5(>(n0CQM5XOeLB$o}V;Ey+?46H!gPnkFGO;>alItekGwnp;Qz}WJo14Mn#5@ zQkkPtQj#GVqGXCFLuG6tR5DAZq-1KMq6kSSDkL%$4d3tbJntU9^?labd$0Yz)c=3q z*L4oZah%6F@caTVuK~dkue%TIBr3yx`(+W}fd7=K%O;~Zm8m^q8lb;Drm&@nXuNDy z#pGb-XL1(wCzsO3<<*MSYvL45>d?%hqJ7l( zH-F5E-gpd}t-8yg4~}x87zaBOYZlB)=W_sVeDTR~G zedcxczoK<%V%S~47I6Rt*74_0dJp})OL)PsXtlfI*zLDR-oGog>DGmZ2%tk4TLJf; zERfbkGX10I{Sy6QKUG~w`Jzo*bQXPnx9(Eixk9?GMNE~p~#IUV% z&i(~*a`KtGsegCSaj%YL%(Wi?pGpxGL%?I}Dr~76zinIPD{0|aWRE~bK%XP8HqLc8 z_@v6TVv2R7KK=lLB>v(%sJzc(E<0?yg8Nvzg-(h2ahCH{ ztyeiR5RZ`d-7hg*kXlMdy{21&ZK4_jd1R<~K5F&i$lpKC1X!uM4ru9Yyzp{=yE*

?3STk+%#h za37rPRnmDQu4vnqg|szaiI2@}Lw9HAP2-P?h44v-X01u*%(5EiD7O)U^cyJUmeoVt z9voT|bDW#Ob^$yfo(z5Qk&08`4#?f+Uuh}+Uh8ZTnYY2`O(ldC^4UbiGfUp#N)c23StDn#;q-GuiozvV`(+4gLf z+AyUJ4Xv9uH0y$=J_MwmjYJ3y2wC42waUwUW|n-aBv9Z-4!N*Y)AidS8ebCf=BnQIDs4C6c?rQ;5R5D)qz2 zg*z-Ce=^ryY}-d!$o~k_0BHMNmTEsAumK9>IC+=7rnyHBA3DS^ZR1th$OHQG`i(rE z_V8@GWM^^L7A+U7S9WL#cdo?G0APK738K1SZGqY z4JbHYa3OWH1UQfp7$cR*R@PV02#T)rqXscc7!1**11LVndL~}?nCH_o*$j3GoJKTr z_H$5A5He*f!$M+AG~rZ0G{A&}|3AT(7$G5)arj^iccdvBmuPRq-gPj$+qX9Z2I^QE z&GK*3zbl=_0Uj~1_bn>8Z}Xai+3JEC$Y}rg#XSsn^dQ+pNjhis{Moat&ttEfUw*Ha*rIiYlQw zS$w$*4njcb9!fGp5Oudtc!pylJ|!U|#PXerFN?8u21PE@y2)>`qnTXO&5>f8x~BeV zt@n=O+KAY4h<3 z?C(^*wcKe$#1O9t{i|=zj4xZ&WAI>mhb3~uewQzQQ_8EA!wMDQom{s4o2{r!W=Um;BA z&Bu@Z&(D$uFYL!J6H8|h^liexpuGfA8d0}cw}rX7hpcDj#E~VvaUB)FJGQzJk4(E_b1TAFuG!C7;=+$1Fw#WMRkaGKM z2p=6)aMkf4Y8m`Pbj^8>Ja06QnzHU?iP5e(3YB>o->VBwv|Ca#jiuM&V1O=!KXg@(Yow2On)sbmn5oi1vZmzDZq3`z$7vaEMyN7~=7<+60Jz$l zE2A&ETyAf5FL*FH(VGh*}3CgQJ{y7sFSSC+BPSt&k!ksOo9 z+dwGySYJhC{>uERlp(K07@wajN3SnnbWYMea0lRnw^dbHjDE{#&_=DdJ`g6QI~76ag*65ascplg+qZwQ-```abVt`H z-Bm_KvBEILHo4_U&TtIx&oZ*2au6;gGyas@jTm{T$O$^-krl9^7Tv`WYo@ zzdmgmwCL!{MbA>E55Bb|ZTlnDo36#zj^a&+SPeCU;6dxTY-~Q12P8)@uT+q-7i#{; zSq}jj!-B=yCX%tJfrS~`^B+^Ln!Ok~-D_P?9C{S|!%=N!mVh*#ywc7^);pREt zyJ+6mN5>(_k7@<&kPMXSYyi}#w96op(sHN9fhNW%Atx3%JNJ-{-^eC1g)J8&%$($@ zgeqW_Zr#1Rl}h;LrP_xpP1c=QwKyf9`<�oe_6`SRr5mp%arf%)^S*QntK9p0UX& z=jTQjlD9^WjCHg6H(0tVi;6YG37rf&gvX3a8||MlX3S>SrAzfC(HWKK1diz;83x@% z?bI_HM+B)8VqCA(gZDf?F999FK7PmN!SPe0`$>io8JHp0dUAH|Oeq~;!-OVMuLodW z-Fg>C?2%+De*=TQs}){TZm;ln)~;dO)QEuQ)DtsK_p%H-LFeTG(DoRxSZCMxZ}$h1 z-VR7u#@bB8R3KD>bzye3CIGr&9Fuo`~x&^Q^$Lk$+)-kgXS6n2#ukRl) zcLvt%Ng2OjVZ+kXL55C zq)(tWSybEo`JigVrUjN>A>J6B}Oy1vbe@!J*eWCfF zwkN<9E%{V5%KJ@!{rF+^?RV_@%U&f+PYN>tM+9t<=3+VaSH=0CnYq{>LLYpNs{-XsLV;^tkB#qd49}I9Wvk!#{dm0~@6wkgqsU*f~ijGIg zkZIHYRt2sd-#pXdq#q+q*8PM$HroG`K8S`ro&`ukTtz1gF~Wb~#sQ+|Z1uXLV&6F* zUSux3sAP?79u*y^?1JxKX`!bu6VN_D_?$6{Mrg#DL&`EtAPv+o>~z~r=t*MyfKt}& z`*3XG!xEGF&rA8<_4EUV5X=JY^_Hw!Iu+)OX_f^HKLC}zy52%ewwyRo;x+xbz@d&w zww10uO5sJrDJAU77w_CZ(*vz_E_qp!)pokWhSe*~4CIVXPFt$}u|6zX(3$=3A=0Eh z&h%DMY*V~DGGYi(ZKMq_s*o0;3&LP;)kdL=$_>nV;I z(A-5Yj>O{=7uRX0c;rJENKjKi^D@#22lkENi`qx2SH3-7k0T-wUXmD61m zfe)1~boC(c(a<8c2g6sEDCF^LM*elu||Y7`b>-*j?(5wu*|2*iGmWBO2vp znw4(tmaSa)yJSVC1P+OI;caK{_hL?cKc*31>fTL8MoD3k3I1D>SB!8<41Wt{*_^X( z#4|e|zSeq03(F`|}Ch zWapT^M_oOS4Cu2QJjjrqNLjWu@+NsTL#+VU#&?seo^?1kZ%fLm%T0M@OFs#*m=N(H zxP+VHA}Ww=9(G;&K~j=cLN98LY}njwl*Ph!f^XVu`Ik^v6DkhZ)4%wOINI)gvsV+S zSOQ59P*euaF1frt8cT|+*%Ql=)%iqFV8nsPB5HQ_^4f)=C4i)VWiur&!X0)acOsYG z1FaYB0h>qbMk93)X3@w(c5@!y)YK@rcYj5-<>KsY4z>wGzG<3oa#GS;7?}rOQ$9Ys z*pt0`VEcSmnsEA-_tAeo$`${ms#Z};>&Zp3xG2g0v3~kWfC(AJZXsYAeeK4l%}8JS zb<8VW8Z>^XPH0c@l!b#Sv+9;}Ppi6~tS01sDzFONQxObj|K4$|G(jJ!W;stb6zGi^a}> z&RFKaYL1%sR-S4NxT>CXgdaf$=no0`;U2>d`vwMIM=R^vij(!e0ao`hNub<{}==(`P8i8?fU(E zzUcYr3?HsKsNbe3dU-adFOPAVXX@YFcw<~-y7PRpeL}{o_I|y9SkyEc89=fcXV54}D=Uk3l#eBOR%t}Z*fDZ-#_3Hz zyV4~~J26WuK&ghFNmPIQV`OB0$~yPJoP?kd*+l8;coMi+IxpN$n7s5E zX-1h_M|~y0Yp@6ab2!8c*yjm92o?jH$QZ2-a}3kL0#N71wp|oYG-T>Q>zW^eoB>Mp zqyuLO-368@Qv=)dQRYhlL~LR*T6o($d}uX4dw!yAFF#1O&z-yAdx_Jd zgk&l_(27t*4jnZ&zZj{6C~*un6kJcK# z=n>|ACVs#B`u&PNsrOgLo7A?)611P=3AkA#Z^@c1}U-tG91YeqK}dMwT{_WAiECB=Hr12 zQr58*E+tEVX$25;F=g?~YXi|72!W4C;w0KG?@U%Qtwi%6agtGJ$(M3Z!J)uma{Hse%(|n)7KGjmW z2ZR5S3~R~#RdzZJ#!ul4Q-B5Abm@Ml6o2KB3Va8TJQrSuQonN~1mq)|gu@#W zu`?*~vz$|0lNN8b@6M^24tUvLE|RS)gLK+I?Vc$-!Nt;tiNns!IE_)GdHjYwI+jUx z;X0OtIjldm15jcaEHe#gcIeg$*exq|^R5h}DZ~O`U4&A~NK?bpZ}@%_({Axpz`)51 zJ^cN@WVG?)f2v2g@fYe=pK{52@bAf00i6%TJ)1wm2?gy=7X!CkE#XKH5%-SMN5Z5c z#P?!WO#jFpW49>>jhK;WpXlf^s66NU5T9f57hT4+${h!F>`|I>In|6ssC!#f3}ZzV z2>u$!iX0UGd&P*cvhr3q*W-_Dz!YHYg#$vx=ya_e0Y!^1>$SJ8U8$WBupNw}0GtAF z476JYT~eMw^uZZL$%_>C^@~$5J(ztVPbwoIkc=wU4b#Uy2mN9e02yrmNo##Jq4A9dPc9$;qLwZ2-8l=^4rp0#a_mIxn7U%@Z2%4P4 z8WtWtt&g&|Z-vb7bF*o_?>lUr*rDxXF?1~ItP7sS`aPDclQS@?nzK)JtJS7Wo4o5& zshK@Be`sspD(Q;d*aP+Y8_RM#ekf*zP5noMubJ7sQl6Zl=x>J#o^s(^W5Y_|(9jP9 zp$RZop80u#m6a7D)ufTR*l6lnr@%wqtuhSes*7kMgn`d%J((IZZ(*y4)$EriY!eyR zR)>51j(&zPA2-q;{Aze|^ll*HoVB=7EPY_&E4n)J(epB)#~qX_!ZQ^1+Pf5 zmJ}o&M?Q?bQ9yl&&ESp(&&~H(xDcI@Uk0*I2?1`SoH=Hcw1dCpn_)?WIyw7nmrt)0 z(wk%|^ESm47!~$WHuM{2fA*Bhq;}r>)4X0}`PXtLWTP%v4&4?*&u7wk`1+q`OvW=W z8ixoOeCTy$<#uEm0Izz6=`FNzb#H?N=?)hEH9=KPxSp_*HF#3o0vLZ`^+gG_*jT>H z2~E!feIx^#+xgx-J4-9;BTdF%%T8R}5P#H-Pd^wXKWhDA)tgUF#c6JDu6eB|sGLn& zA8jVux*gmnC%w6)BfoTCbbpZp>CAVH&}V#~K=6%-$mT8t(AW)7p$y# zVa(;en3UA9b<7=6k}%ya0dSrNvu_a#!6|&0@OqJQ58Mmp2{~t>9MTPekoUEa?@o&r zZMRK)@FaE3$EUmC1*b_pdy+Ql;ql{_6;Iz)N_qYG#?ba1I`Bv&UbIQ@7=Bia_Wp&m zj|`x}9O!aCs}@(344U-ie+L~6@*7`*sH8R4O=eT&B$S)=jK0Y`7#HSJstTR(#wbzp zbI=69sh7WSWV6@{#ct?^F5f;?+da^qUv%yO24mM&>VCAIb4IP`OrnZKTkpW>d3hm? zC0;DhIVbb>%BLlB(eNDmXEWBEwvkDpaHXW$j$I_v(^^I=X!q=)`Q~=(hJ&i<&69Rm zn~ppF$O%vxp;OY}GwYW!wSN!F#e=yAnvIiB7Be!eZJ^e_>enX<-tcix+3lcCWBAtE zU~fBF*<<`tdiFY4f47NykMw7DQB9KB#YG7!h4fGT>;a`25@Rd*)Jco4Wsk(o4+RGKglLa){-88e#V+`mUI$Rs=v3_MvzAq@lv1>YF!j z_EFn(t1Wnh_W79kY+lECG_lVEz$}v-|F@kWk|x z-S$UBu$b(i$doi>;&XQ8=k_0qJ$3gla2Y45V$)4y^pv>tJK>177`zIa(ym_Gw+&io z^!!2fCG%=d9_fDe?OUghGoD?wzOi|9|BDxJq%+sMo!!;%&w%!wdS@-DFd3A6*h=Vn z1^84;`KrIYOzc1>0}p+nVsxJI`^|8aqMCAo?-g3i#>&GMus7hJd;_}a<6!0ca<%JZ#C zL)uAi_E`a)#I@M0V|gedq9axV&`0AklqG~;vDm_G_OYLs)SiZjG_;Y$pkm5Lf^pfM z93pT839+MwLnk#Sb4=-p!w;UEeqlJcO?&t5PKox1O%tyVU-C-6G-Zcu);O^tk4Gd_ zK8f}nyL8D}zu?KjG*~JA!+u}r@Ffr!2J`3VxcQYJvF_P%HB4!5CC(epO;zPY=?eV-!Q#~7$O{T{a8IF;C+lZ{+iDV*%fS* zq?m`I+`)Wh$NJgo>jtWdL1J)unWMlg^hCFA)y(};_6`?ZLJn)Nayezh6!TC=hk?3Q zn|;6CdfZW}Rjk+CX|Yn91kth`LWkjsY)Tg%q+{Lq`z+bQZ}wGfYnjdA;r}VsJNzxx ztM6xIn8&#z}~I58tHBl9IU_h7mR&_ck)>2<2)YAUBs2Z0|HZqSAc4tc_qiZNT(r{tM`}eOa+ssP|+L_hs`?y9LsdB#aF&kM4Ad!U- zI1-}JC6=VGudK|!?9axJ3!bWt1H2z7jd{6m5do?-$-P(gGTnzXe+AbhwyxTVf|-H{ z#3y#fX;D{z?qSLy9cL@FsX$@KEFLk93tc_qXy;i~x6NBIj1vkTw$t>+Mj^IK^Iwm< zV+Nv<*+|6CZPyyIClP^FhU8LB@LG9&B_C2&!+}?j?^66huQMVrGQ7TN8QB*v; zK18dyK(X(2*dyQD_Im?nUs-ZEwMi)4J~!H=3?_N1sE;3)@%+;3>UP~Xob+ECJwGfw zc9Hd2n@M~7qYpjE9g5Xy z@eNG@3RudenOYnEP){-bC5_(Skj1}WC#Zgbxnd*bU9uL7$yITyZ!_UzcTw3xH&$z0cFddI9<$eDh!y^dUBw(9wSOMoWTD|7Ap zU9MU8Sf_2fM=8l(ug=t*&fPaqU!lj+obd85K=7S|R;=ZlE8IUK=E83wbl6ByS)&5NJJCT?a~HBEsD=rzl!mI^$v3@oZN-ZX zi)!Z7j1q^t)BZ{0=(~kECp{t~%H2IZKY~p?zcgn}O@-?r6ZN!4+ka2@@PFgI9sK|d zt9H60CQyC$`8^fHf;I8Zg|3;qwr7K=?b)!&?0HFm6H3&JCQTK0qPI`zrvKb>sJN##kOl^~nrBEjP zm+DiA(mJOpByii}vB9j(Y-u+1Jn(PO_ErkiSul~DD!=^K=XqkAs@PKm+9cLADNh+K zh9^beEc|I#f?<+U&1L@Pla?#{nK~BBpOIb8=C2BETGFLl5!q-Wen#t!nDg-iUD>D7 z)2l-2*{+lE^z4f^?wd7wVOsz+Oh#$9@#@tzzoRxt`);N_684`umSYm_y+6MkPJ^*` zlq_$0*bD{$^JeDMpQm^H*xm zoe!q+W7CmiE1N4zW~0RKxqVZI3Pm(+6v0x-J@yNef~esuypx99ee&w&iy{8+cE0(= zzw58Tl4}yY)SN|E2rwA>513}y5QH|13Tq6I{!s-<6RiXX!=lB*{C^)w``zeU;GSI8 zZiU^sLNc9Lt#FG1qJdvMF1Dh)vJ=jygn=EK8}@XhBE}_0Br@I+o#@bITVsv3X^u?R z-`KbBoCyb)rF9Tkj@Uq;j!!s-L~j+0bb@lNH2{-3TKq&0*wLj*BaYI#Ly>RtX&I#DZ$OdPyoq|FfIZtdGxZN<$LP`0A z!q}L7Clow>{~a5U>M4X;n6EOl$}Yq^N6oi*o7(l;f5ob{qz${&@2DKqAMkI-;vhPf z%`KPKRBjqry1sj6+4Fm8{m1R^bN1Oz?clp%&koM&=y~lx%HT=Uo{aN)Qc`s8a>C;N z9r{^}k?x@LGKS3syC!Ie~3DKCB32f{e!5eZ^O%u zG!B59EdO{uUFIm_C5;h26TNN^5YuQhu`;t;iWOW@H43-dzz)8$jE7o^MFirO(XRE| z*_s{TwN)Gq2})KS9UZZ#_?z0xWu;Naj_;b>#s1{ccI$VGDL|wX5<*DN(EkZavwk6m zbMx|mTm!j35^hh&8T6Oit+B}}GQ>G(kLe8eufs>sn?TauaWX^(dk@CNsoP|SEJs8O zMj~R6%*1ks#|$@rdwyO~b8|~W_o`y+-0+GpI`~c^IAJ%&k(xP8uVVdmNZt5vov@Rj zWj6PpSgpmDoSUZK2w7Ro}8Fc+j>dwWV9$+;M)YXLT~G=T3Hf@#{{+#U0Ln zXd%<;d#xh-`4VmKFHJpvGd`PQD7%;JkR4fNV&@)`|7%#VNf0Oo?YYsn!T^2aPCz0l z5gWshkxbESeb*0*h)CwaCH7*lDCSFBuT*IL>YB63Z2f#I(#tGyDDj!m(bsR!Y;W67 z*QftS3!rj;^ZT>t4?yc|hCZFxo}MA~$J4out5XK)=xl|lOEh%b5XFA`v48iifTcZ4 z_6U~^Z&P}0#TFP`Z}Nu_Oi~={Rl03%`jb&o@+|aHKtNkrsix68Ou|G)WQi`wo_G4n zpS8QxTiGRyX>PpW(q{1D6MrAoxTpO#1vP3K-2C1V;orIHuQ*#~a=WLi4Z6y_tli0M z;xgkTr~7DIPouuX?E7T(3Ro*5x}EcbS!QM%`xe2C_aa{;91GeJ9UYxGw%0Ad{-PZZ z#R|FI)}tHlen%K(I;6~|Q=g)Jax5EUfln(5ru>9qM8BPf=caGC*;mY%--BIKSsb=q zXv2U=BD=9=Vjm~ww4vLlV1qQnK(Qh$AChPKK9fH5j39_Bcm~4I3tVV`ZjsOTlNZ0e zj~-c&pr|+hj7@k}PgZ%HVCo+36(B29xUzR^6*)9tZE@12Ew$-0Pg+`7+?7A+5+2m( zCBiBi1MZB(hh~@0ED{vXY1hDZC9vljEzEvpd)DoPUJ;{r-!Rya;Ac+2S6bF(8;7at z*~Q~uUq8#)iLn>6nA6pKCqjQ(MMp5^nrUs?WWnT>z1kobL$xnR>S2H+#F`r<$B4%5 zHL~A)%Qvm$Imv`f-b-h(kBolG(5(-I^Uzmo?K|dWB!M+? zHXvle?L)CZx3;hXxZz3&4%Vu)bFbQKt3Fa8CyP-T9#e--o$gFFxXHaTT_tN`^7@5w z)l2oKXC&v&wK|p66ETR`UdNAlTU~AG>UyI3{uT$7j#8u6sU18vmo>L{F93B^jA$Kc zQIUwuU6luGe0jYiYq)+WRn0NZpU$ zJ`@UqPAau3m1(2Ud@E#N5cPD1}+cBB{0|4y-u=PO@J}Vv!+s%9js15FIUQ zXwm5=>$%Tgg}8_}G<^AMQS0>T-Fq8|=Ir~gJ9Rv2+3A{hYh3Jib@$*A`$mZ)}Wb#{*VoG#KbT@D^+?7zdMqnB1`*V#i5RJi{dO?3j@8hiWD zVJ=eS7%SLO#?Si}>D0;l`uaWsYA-C8Y$bCjrNr{H2)SitWy*^MT|eSkcEEIIWuP%# z)@d*!%rWC4))iWt4t&2x_ef@l`f;1^#=blfHr-7rO$96Q9G&YJl+Eq4itUZeqcguI zAdyKSy2Rq|YoVE=uqzE5*i+1w^ew7jtG!q>V`JBCC6Et8IRL9|PR_ch%{#gjWy6-> z*Y6g}8yhcc-*reIK5G#YFre0|F5XDR6lwupr7l z%7KX)HI#XaP5N9((q^6(cIKpE-9dCt@S~DP+0H`Ma}dNJ(eSz+SL5odb-BsXRV2$i z3}zm>3~x2ilj43611l23WFALS8V^~Q-T`++jK^^7*0bv#kp2BaOv0dS-eR5ye^%@_@X<9>#}w(bv$AH${K4rDZIXotUc>Hk-@qzc{z`2R zl0=)HgvMgqq1RdVHyvgz_xsw$`ma+Ma|{ff;7tMEg;`NbD2cW2_Rk#VQzg0<+ZEoH zGiG!~!LW#&C?R-lup99XR3;*Cl8hoBmEp%;CFtFSh-Ax@Q!j$LNFJ2Gp+K>aY?^HQow0- zGYy$ir%o>2GPX^i3SYbK_Kr?ASM@C>Y)NUm(!v49=qz8k zC0uvcpEa&0MqFU_{ZNm2R-Hl784R~Zt*+A zCkrbiS>xHWXA8kjU_?debCw2Tcv)CCRFn&(+jz62idSWZ>46b^bn4OQF;;cYL~BGv zZnXy^k@c7nox5Fu9@7>5#}Zj%0?;nVG&19Ms*8fNbBA)HSimqCuIWIh#gi;I%{Utp z8y{ipTrwmu!lK()8yh7!Be?Z#RKF1uPWClc8L@iswuq?c9^!ivjs`6pWPUY1ILi)` zjXWWhr*W5bQPo9hHGf0HLKR(AajFGZv+?VgGIF)B5+=1pagKr+RuY_!gxD6ws28r^ zS9X5W;H9Qpn4+LJHdxmPTA#xw|WW!gp89 zN^`@|R$kr6A}}=57FD=7mZZe&oSe6~a0pXhY~pGPuZx{o!UI1(-Isz(}S$i+=F35FZ5(71n4?L1EZu+Oe&&-!;gueKG^nsC2e+X%h{#ub&_ibB%(S zui>E}il;;h6j5tfeg=(X4ApM%6l$rlvDjfa| z63ezN^_iyiUV%S&9%36{>uD7`glRif@&!(y1U2AzW`llKS1!RyLm=7lnPMG>sOn^U z1_+BrAo4d1d8oq#lC84vp*%{> zz!Hj+_ijwo7YsXi^PtkiC7zzSJg6$Lyq(|V!6@5VTU-Av$WiDM6mH?Yb}uyjo|9n# zvwy;KtMCfkoRXT_$+sphU;%=GJ#$7+g0q(iu6z0AxkfM8cV~7H#Ap6jq4!OLg)e&; zu(X?XkunEAf_BIn2$EMgTZI(Wyoc6e@_=mIJb-^QGiSEOkl^I~)j6WqL+DJ8T`_Yeo#M=f%P zqLN3}@qK6rL<6EBwwc=83r$$(P*q)>O<%Ffpv?B$FrvOUlNlSa_i~SA-~#O zSvxSbX~?r@V7af$kBfHaqD0C&D~p)ELx*0~&^?ZcNYM7}ZNUC!u80W*=J;EhwcZ46 zasdE>VEbSqCHu3ag>pbk63s1AAGoOYQ3;p(Hz;ibVA+tvp6P*Qeuo15#E)GU)DI=t}KBIz68AJx6kON?{+Gc>J9le;aJG9 zR(aY<%a|0A8C6uq5qvkbD8Az4S>E%_5fUJR>&mSWu!Se6-^3-Ojo%0!L|h_MEktm8|MKNa7H8bU zYt3+5Btvfk=A9muu;k6{K)~QTKC$OH>LN;UBu@|1c8h131A+D1{jSR-?g#f1Q333RN2s6=h3G9hLJdQ`9#Ua01j5(s*L_^t%-nVVvywq%$&IcYJ? z-n(bdRvSf&!dK_iOxN-~h4UY7$ijOl-ixNhjcPVN|A@)Wrr*C}<0)gq0eA-V@0&jb z@(_c}H&0sTPW=E3!Ahf03W$@#th1T>TZVZkalz2*55y(TJW)HtxATGypidGT-5IJS z94E##4$ClYX_t{?DAp;#T>>>dGvWFq_PKv$HYvfSzqb1 zvZ3a_g65)@rrKP@*O64ckyonk%lpO%75c%0s!)E$UR~K83!--iqo3C6#?Yc4C8)7R z<4Lj?*?CIDV9R^iW5r53F6shGp$BEoGOeVgZ>9R~H8r&uzxl?E-u4xtYkIU2$AY-N zk8K6K+`q%;-Jj)~Zi(!wqYK7%!9BG1WLodrqV+Zz3hTGz*@Xw|s_+3CQ@7o&F!(CiiIhFG7dZNR1BZf!s78VvJ59re;NMJIIJ@MDi z;DE%G6(NE_98qb>8owLJM_qtx8q%^0Y{Aa`J#xvXSuQ8DA3eI}x}O`lXCe2Eb<9z8 z7$i&QZn5#X2V+-xa;N8wEFUerHabrbSN5C${g6;q?};Fh;VPC#wQ%{wolY28>QO6$ z!`@4#`&ugc6bkv%X4;*_b{2g4Lb=?LSDL_M=(I2KDO9RD2$Y6TZp7}F!Yd;9A){mg zk(J8C7yzrTg8yexjr)gs;hc$5RY=gM6~3Hsm0<*tL_un-O2m>UD2={zl26L&hb`FIArfkb>;@LtGFG5YFekSdS7=$YjUK=cuHl`M`0^J;Zy_YzhR3 z&vnXdPak9B{j_$>AupP<`PZJs{SDF(cN%SITnji=TwLre(^8%A<=sIRa(NbSj?;_Q zUC^5vI}e5m^CcH#+e~E-{ncjGpCL@>s_L8F*>=BIZ!a%nQQvi*D4yqMLDQg>z+!Wt zS0v+qR6MyCmBSm?aQWi^=!E1!f-mxwBLoBcfT0TK}FMf7tJFd_yD-nti0EY^0sD&XWW;WamG-lhebswo8v?_;&HG+^L9ze z$X{*t$F!f9ax1i_&E)e$?eug z$>Ag15L%0&0tR@!KAh{R#qb5i)IVZXSN1p&!CF z{yX8W(r<4?3T|(ksA%c{p-Pf#E*H-iQ4I9LruFYu%<11QD>I$&>$2!LXn44XiBdbB zgUWV-K*+kNJP*K-nm*9iC&sX+y1F_6bBB;F@?!pZ|H{f1Y#y{fVM2G(3$2Ng;j7eP zyue)b#$v}i7o45E6mk!aJwHFfCNz`wcq|hCiIXS0{Qh2zd8-l6HCmzcM%NqX_xEqJ z{@3mP=l^-GhiqGEhk3%yh*@5UzagvC)}hKIao)?#4dxl1n%Lq@jaPJL_MizT6B&pg ze)a?2k>aYoVHkj+pIgaTNru8(y{)Z1Kh?ktEP$7K>-qEH{0rI;At@ER)M9%N?0Wv@ z&n^PKr7DP@I+lqdVL~ii$WY6io@fTP1NwKrv~(2zROC-=1+ZIV_%WOCL(w!QLU;|bu;|5V;9k)`qD#2$ z|L4k1X%XT8fr8mL{s0us_C#AD3ZjP*0T+nFg`z=1OjjLHSv=>8{5C}v$R8SNE$Sn& z`4AbHc)sYusE?%7jC0f$_j6Fw)>d9jx#%k?veTi!BLU6oTx@cNRSEspU15<5PCkLN zTj%4yq2ZmXSa3ig(cB{U@~Bcm{&J`p$zuh4smTP_B_uuqDrT%6=l z*n9kL4R*tg*NjkGC zA(%Y?TJt5l{K)8V>GJBDIb0nv7jIP}#&EC+6S9dqpR6GxMnr(|ysMf~2>=Ktq&{k0 z@_~KIm)OhdjoKfRlieP3aaLBtxUe?$+te6`(O*H3{ZKfP$%bUE##ICv~aI&K`Ry@$2yp?uQtLx+O6fgEF_Jt}Hx z^-`l7l|I$%ijD6);BSL%AOV3W|!AOpjJVLk+*e!Hxn? zvbnfmoWIlEPxDgOhLQ|O{&^ZDbQxKs7%Wt??S8=H4nD&4`q<0?EKq>n-%J{<8r!pT zIU|U%ry;$Cu`vQ$}-G9^J^rh<%v#U$@9&UFeV5z)$P*%^$|2J6g`_lEwVwLZoy*}OUtg+-}{x}PZ ziol~z9N=g%Y=t`w;x4tq!OTyj8_bv z*sXDcA4S!(tIHgDia3o@UAa--vU~e`;QBIbtQceAs)Y8Gwmmaz;OP3-t#CSs)wfy@ z3TmIvZSVdf;q>&=_qi|I)6-kzp6?$BHpkR7PolXX@GU?PHL)-v`ytsO_>Y6ZtP2~d zQA~XrUoi+J8l?H_Yc8v%$bb;=tC9fZG=`p{joeF9kHWq?9{lGOgJ)@M$+!h2qQ>qO zmZW5TjRNi`ah;&&c>gHwt@@{8NBt)i%{XnjmP^*90=I94_rq;5e3tTd?vxkd3;{qp)Zx#HLg!kQ=y#Vjik`u*C^ zXRp3mz-- zQXjIpdnYQ+_YJ**?j<$N&YTY7!85%NRv?7@^H1jWCY3t#88LdRY&&>1%Q5373s>v- z&z^N-CR$rB;vfqRAf?>9wdWvJ#J(#bsbrl|s?r~^QfM~0dv~po;VYjZF8AeuM$agY z>!w1}$MlkLG~}hZlI{)+s=qeO{dvdF|4se6WaZ=tBpQ<0mLCH7F_~J*JHYAd*Mj7} zFK^1ZWZq71juVZ(Z$_VII@)eEykkcHe#FYPl0fI-jD?WF-5qzhz!w>>=jb0Q*PpnN zqi)0yoXK_NBSDSVMJ+}sg6gq?H{D$cIfv)5@CZl2^WxYTDcTb{+kdBT50nq5L?-m# zCt|RN>hX57`F#jff35Jp-cwc7l%psq;3zW&xIHp5^4_YgVyR1+|1VA6dT*AoWv--n z1SAQfWwl9iwe(l22wVZPC%PTu{XvrCNTN%A3`EFQHV{nl zvAQ63%j^wDgyIG^Nm$QP@--EF1NA>aKg)ss5a@V+LIs_m*l2<)HLXbE_80?pfYV+O zgGO`RB5GFTL)c8|anJPk(%)yhKKWCde!ka|ioh#$LZou$wsl92{1e-B9p?f1Z}vw! z|15P^kRh&W2)c;h9d_tYeSxQdPx!~ySYaDC)ZOONbfF}wgr3?^HNMqGG2YD}CvR#S zM`8AIgB|VvqXkIYUQ}c++IGsE*_gJ0?5(ddHt;T9;@eo3Oj+hlA{49XKw$<{%A_8j z5IcL&#Hrgx`#V+Ju9ZBvN7?s4?F!A*J30@Rak##r31eNcf1UoMVh}DqoTi+naTK#H5(X&T73h_?XgdKer3Q2gbIhLqwhv1yKkXeJnt5W;TigRhVtv!<#(f)u=S=h?%!m5XLw zl6g?EX2xAxXkhUa@KqDjo?IeWtv|nFeOI~+z4H5qbLG@Nvuwh9X=~SQwuKwid(3yA zEh>7qhr-K%N10le(;o*j*v(GBcHVKh6Xwk6LO_a_tLHuktnG1!(Q1nIfz&s1lHFt5 zbq-B`$Mq9aNsQo|eyXpF1YLOZ;lobwSU|}WykcQ(khp49PF`MbiZ&<<@V59wf1w|(6^0j`klJHc!OYt`Dye|_7h)q6^g0!7k*`%n%ZBdk z*aboCO@JY9G`;yqiz-A^%wd7w!e&G#tdnnEFPvN{q=i{3qgxDS{wfH&n*IwfXq+o1 zBewD-V;Uz)(a(ylo9fhscZ-V;!raZC`s>_QY4^v{=NvX27~ot!-e#sxXeJ5BoS&Wb z;6cUd^J&^zT3R%2p^};-%v^Qs&tZf+=CG0ALqb&#i5U4P>Cz>tn;x7|+Ta)*0e@IK z{&_svfaOx#z=bIuGO@WQc#OI%w*hr|Z>-*UO@*3!0iP;>UjlyDVqnpg*TdO9i5X9C zdyAM9GO5@xI&osw5gGGJ*?1LZ$>Jv||CU?w38 zvS~4fulcOO6@CMu-|}PRm!V>V1@)AOfHgH|-whqtEGQy^+`1Fx-6T*Yz!tl|I1(Kd zbrN8VF_r}JD+bIdDl+cNruq1$AADB9YVj)=+r!;HtXi$|s{hCV18UNn(}|SU{4jDu z9E?l;fPFm{^o0f>^NQ;WKCyzfz|{XNGck0Mtf}uZ{O2NF-2~mwehh@($CT+d$k3EO zzu*;yvr?=3tDvuuTwoL@Tj)XfnbRd^S6GW7cEB+v)yFU?(tWjO^Rmp>?4U?x>&);C zmjXtvd4n0nha?}s$HHYlVW@KOcCz~8GiJPAC2oKX1;`@Jx|%?ADTWxqy354fPnI_ zu((O)OY|ykD?9{}s`Kkh>0%H9WU%`4%WUc|(D`1&hc_6sX!3~X`@T~_)y7Mfr8OBG zJZ92>ajhw26!oMl>el_*z2#;AfBU?oQxuTO9y2=qJ zzWjiisPB42;HucE%5&D)ReasE9@7{Ab!R>+EM`^c=B@cVAtn)b+%7HEq%{}7N8Edh zZqvBk^Z?XDN(>9i%dcr z%(P|67dU-Q;;s=-FGiJZ^ehg^&4-xV)pe}>z5^#a`cCZmp)(rz+g##P?K*b2w4AEUZq4*c=_;l+W+VYtO47yi%^}rS70^>e7KDVT?!O*AAv*+fs)RGtfEvU15QZ|H| zWH=d$VO4VBL*yM+^I&UJz8C4&i<$W%qcolbP?-{+JICFBS%-^S#Ayye#`jUa;ChAL z_V5?n|NfaDH8nM}4;pkAvnbS&Xb6llAJ3$kx1Bc5+j7pF2oH~}vY*x1W)_Z)4hvf_ zC#Scu@snhSDDFPKTCwO=30+apaifzPr<<34)vtTW4AEC?UMgnI7i9smr_Tll0~-f1 zMK88yXKt8xdGYiQPtQ&UCYbIM>M&ZSqdB_(zH{ZOBIf!$@FT3()~ggN&% z$K?j|6u2+)@i!r$y8u*^TYX{u$IgR2WlWpf8hb*0+Z@~4dwRt{@_rlc1RBjLNJ+2; zOz`z7S7^7X+o+Jqj|R+^YWnRV(9b$0pbj1DKTWMgqzh*sJxC4H0otQRMRPTgd$OtL zZ1r0|QXfnBhOwjdD8AsrgBkKk2sg-H_VzFX^f&Lx8P~Zz$W9ob&rW_BmEojCW6cjL zjQ+u6H4f}f!nlV>@l3CG^rSrqO|%PJC_2dglR)C%NxxWDCSzbQY^~C%8G)rY-ozao zjC!d@&z@UQK?>y*c|z#E^yiKE)uwYNmrr^74#;fUWat(mx?G(-LcjE1e{>Z8ur^~C zd}?}Qog5lHH)`Mmg`0jg`@0v)tghL-Qw7v#=KMzxwla`@grr$%e#0re#J(!=hJLmA zS!n(GbNjx14PDzlh)fWDT=teT)204?jJCuq(cGW%E&6yI7YA!+bjmu@C?KUkuIn z(Ai0>Hu%q%|2}L2#N3`tZM=*91{E3uN}oYjdI%r(TzL zc{wDbtGtqugt<9%Ai)2RkDj*Ux-nXlNe|tF;v#nJb|Y$2)FEHq+;4ByYm(Z*gJ*2a zMbWE`YMWp5@1Kgl<9gKc>^_YYA{WFH;tu5cwxZ2X^?hB{_RX8upU;oC!zhoNXD{w2{oQJ=x*@blKs#>rtHXdw9w0Ia!qkgy$ zHT1#~b@}e?XUs5M*6sU;IG(!{aHTG45sW6p+W~Uk`)A9Dl`=1`ZeRb)sX(jTJ6wF` zfLk+q{f|4~dy%tN$O<83C01}DK|6Q8rDICw7cAA3SCe@D+;CMt`EF;#FTj4#iOCyb z35v!^xNQB|lH#YHn)>>_f_rhEnu9W*xuMhlI+=A+1J&e(vo?v)_NbX8orMKqS+7!I z$uo&j>&#|!_G=y<((SDH`CKceNjrex2*KIu)1IFdKm7D^F98|axJ`le;Y*%uH|KRW zNB!^T-^8gc3At90e7wKvfySBt>+Gj$2|fWgkL7gofOvF#bMju7zrB^&=JDg#rE8oY z>!42P1aG8;o0-7Q!FyD>7gx!NvJE4A*-j||4cY{H$Mb9h1GDzw2vaD`qj-F@8p|{ z`hNLB8HhQm=(>9Z^;qD{3EDzl+OF6=o}m-`0mDHfE1GY?N$8s|TG%=xqu^|$NNE<` zf}2PDkE43ghU(ajfxTIC{|{YX0*>|CeT_oqBvFz`M1>M%2$4!iW zLpQ&5IOjtXPJuQ0eyJAbPbQ77AXVOl+LHf9h+XD14v-xA(SyeQgMInL@gK7C6$gux z{>Mkj%KB!wJ%XTt3Z%j+1qo6(Hu2$Uu-P7PA4T&Vq?B>(L6TRZlK%Yyv1z+ej1jVn zjDrZi2;`wD>rJw_x`x?weQ3a8TntZPZycY{%%WX-Yy7t2Hnl0d| zj=fXRpVgyZAtl9zG#`ZUQ}OYVAoc;}#+Le6?iZt#TUM5q{BW#p2#P1Z;=h07?}1y4 z>X8an8l8v;3|R7Ho(f0a4n2>LMAfc!-1e zKtaBW&G)3BU;`Qo0HAHax!IACQ!jGoF032J&R)yEQc&J*2T-+ut>t2Mw_|b~-NE+l z#qA#6j7&_k=S}Kr^+RCyJVN1rhMLDR=wV1aPV8Ao8~~M*c!DSPdhR(<%B0sK>_kof zD|SAAf_w>x8#8=%;hK{IRNLmj!o8h7*zWID?VV3_*xS1h@iB&2@-E*@h$RmVBNEE; zp%rr_Fx$hvTcWo=uX^YHtN8fl%Xb!o^m~C$vhhFSE|HKHQ27jz6PmMVj7)%V@?tlO z+R#)9eU`(Aw`9+H``{IQ2~^Lp@jHR~UsP1o8A@V+S?70i&L19>cXiEqecMc@egihd zt{?)I|M~ND4=Qg6VyN|vmk68HIe>uh1(szekfeY-a(nS1M}(FC1k1wMO-&~Yu$Gmj z99}wg{-~-i$VzYBi$4>@t0>eyw6qkU_5|Yvj)GNpGq|%C3Ntw>z<2jlCJyLm#vsJP=jg}FePOtHq#+B zWSQ_hn#Y?JS#6X7XpSIT5hENLnmd><*e$q4gW-83$K18>8>b|mIL~gje?J#yv^uY$ zZAZSuD3;IB!T>2AY3JwHAfOl8%503!YkPqLK$4s=aSb$l4?ndBoT-|(Z$A#_6bAa~ zgyg*7eQ;ocvGG`__&b46wL)9`3ciFIBgE(ek^(amN>TDz+Fo-lmWSA^F`tKaavSmz zx?db%M99Yf8Y4N{bnbE;IFOQh?A+wG+O4D8tUU7mpM^%(AYvH#1i&0y57eSJSw}MV z_=G(;Ab-gE+4B<)O(0S92`yf{mtXE;&I#5EfxQM|K=zNp^d!NX@YcBO;~|a*XfYR*zoz zqg8(`9aWm;PE&aRY_g)ZCSzd`De(ng>UrM3w{PXfn_jWPY|r&m{BD{68mTQyQW=yZ zu-&V}CcZ4+@QHn?ZO#~zkv6g8g}-vd@*0SA@CLYmXkLYVCh=5IkZ&y$TIN1AOu@T( zciT(Wi!#jn7qW|X#UeK^LZihSvv)w|9V_)@cCrWvKpgS*zITN|%KcZA1FcD-#=t*+ zny0VT)xD0ZLO*aLOwn*71puq8mFyb+%!5{$vj$}o2N*~@A5sb=5tzAxSW%aCptd}4 z<}qI`^CIqf1{z}zOV7&<=jQJ}s&lA@fg=_`QPBCcFvb}#dCHX2kzue02I&1t8cgY7 zUpTv=zq}5BATm~YBthV~x9{Rq*mRpcToI=#LWHH(U)50*@eiOpSc{G%u`tkt%Iqu@ zLX42vaa=;TVbE>w`;)upJ2F^IO_9ho6{eVbBa!pM{6aCz zqQmDitE;PL^?#a=A~j(HexxyPGH=XivT{lIuRtAjmFh6uG@Mg_N|~c^30CFyGNCkw z{ItQ^#Ek|IlNNn3NVy`hxtVS5+_{Y{Eo5K~SZ=}g2b}x&%crgEDGD&0E$2 zTOC~kmQPW$IURxk47S7w2`;Xn!l6$Gs@+F4x>3Q6RFAwKx#RNPLAknbq)5amZ`sOI zYq_7cr6inx8{xz2yHYRvVnV`9WyA2VX1niNm%NRTejp|ltaUp7^uf40${ZK07u=lu z)Vy$Ja=nQ0j-3yhEUiDLe{{&MOaGab^eE|a&*!0ZHK*|D<)4StTF!4w*mQFm0=;^S z5;GIy@X&WJA{@R)9RfgwR5+!ug{Qy%k2GduQ+i*5pCqj*Y9k$bv!`)mPOf)F99QPg zX;n#FiHQcOSI=DVjLFPY9q#Z`Pr+btfcejWp~Kc!vr*rs&oOP-ebPj;=YnmoOD$pZ zFE(Eqh6yGt{My*hA3g-glCNU~Tyiphz1?5WHW%Z7&VlUh2qIbZPbCWr3muH}mi6l6 z5HNLZl#SYsHXZ3>mM#qiKfN1t(sy#B*DIcQDFYMU$yyWTqBGcCdEWcuDj%i}r#91{^5jfr zl(SrLz#M&?sl-aFu?XA%F(W7u!lp5CUu@dmDhOKN*rw3yrKMZR2C^$%7W#4WD2A`{ zTKE6hh=n6)o&IYZ2z9II>6M#>_{RdDL3Yd_h@=Reg#KXxd*>w}>&-@!&UvN@daFnJ ze~3<=bYge_&v#3v>!MIocd5yo%h2ozfcfxufpQ1b(lI+%eoJfOMhEdg#Mng|an`|$ zn?woSjh&jG;`2f8A!7LxP9ty?Bc}@ zH*Pju3&iWYq%M->b~g=|y@4*f45d8qZ4L%91Vp{4-WP!`CGWxKDjdTqH9u}6^2?j$ zFm!cyKP@SNbFmK&sgVzl&}dR}UiqF+Uv-72XBOA_@t1!pd+4j3YHx!#pfNf40;4RO zr53_7vEkzOh5~^Y54Sbzy!(IB!~I=Tm6j55ME&!bciHI?Gt)lKa2-0hY`|tq+GNkd z?2I+R;{5z9;YQ8iC_sSrJ4M+iDg0(>rkaubH2*i?abw>{P8r z$gYH4J_DU?`;#AAPQcU+Twm-ik_*-ztloi`*NIU?j)+irniCiWRR!bLAFI3$L05x} zOt5Dva;F!l%s?mA1nP`=2{JRP`*oKxyI9;bl5%Zs-WUI7fllz$&76vf#h|`0&z6?< zXHjnoZI6hUNlE}y(^+V)!|m>qq=?ro)FBZS9)Zh+4&y;98jQsqfBSOPZcfN5RFI31 zK^b@eKE{mszPV3MO%@TUeSCf2#&GG#qo8v}ps7Lh2tbO^14z%{%jhw6{r1T0Xy+^A z#t0{yi{d)$DtJh-1bqJZ;mm+WC8AQEzM2(es{nhL&V`2dco$V1*$=8tQJu`;zI-bq9?`u)S3=p5W-n!C;fQznGq@hQl{#l9UH%UuN zJ42KAQN6zx?Ay<2JEal*4rw}}9p}Dx5VX9SmE%{V`51lGAD^oO2M0X}mPc^TyLl9) zO!JzPr;fEoLlb0!73ApXNNFOm;!!wYV$C_opr#H#u`FeuwRAc^v%Wglfp>4|Jw3pC zTBCGm>GkNl+xaGX>%MrA8|sLFvh4G%yPh4X$25+@UHM#iIEMy0hZvQq6r&XF_)IFM z6Go{kW=~ED8hd4iClKhkt4$9tgrB|YU`KwfQ~C17O*y@t_N|eaLI%R$_bT+#~C!UnWI7%*@iYUGn=#&@4sF;>22OdKLfcf9(L zR3`f+Wq#Z2wLvcyTH8hHFS@;N_LoUxsN}4Wix0O2QCL(lJ`PUZJ zu@|V*xDW~q{mbNY&w8D^PiL|bId>3VuB{#d;im-Xl`2Yx$8gf<6S5-8|FGqUDa9_EXBN*LID8y|uzADawHCl(X*?ZNr zU)zdH012A48DO(r!TWXmeEZ=P_?v=83nwUDJe_(qYDay6{T8M&H>8GOG4SR@TN}kj zxPMBF=}axE96Cqt+Q(d9cw z`i+w^zNP0y8(B(>>O#?=m2;6fzhECNt0!{DV#KcFoQ~=A8t#r3IbwrRM<6SS%E|^J z=|-FS=i|V4pdcCVILHs77)Bya78WiAIT=aQYllD;m_1TtqSF-YsoCo4BUYtuXu8F8 zzOjlvz&9RG<$BNn8#ka@k5kkQWMKf0?9oiGZ+n+Tiq7Y-&ET5lUX0*Dk21zW3gkDO z$6D-OhHh>ZH;p^OlDF|sTX;M~|H0N^ajC1WQzQM)cbBOIqW^$3LqkuGX*95om~}uo zNZsqw@^5bSO1|IjmaAdmOsDe=9F8eRu^~FUN@UR?{SjHd6wo%&!e>u5L->P$q1)>5 zyAL>y+0O(#QF=SJml5Pa0r!Ka2x*Iu^$>a7^=l>O4MC|qQ;BJ5Ekomjm*ajAqOspy6*1%4kA1g znSftKAu>UY8q%bx57h^GfC4~5ky?PqSqk#F2&xN@&H~B|;n~`FIQtiv_|y6)Xk+bQ zx*@5( z8CGG;S3RGBV!!~Z)pZ=FzmGk3Ey!g=jZNKXM0gV#Zy(`CEe@;d2==4%lTQGLT_@oB zar(($(G0PCptBRhnh+Jz8UdA#@mf94C`2;=qxr4T0Ahg4gM}pqP|yIzi%%EY&5Tub7~`m@KemwB&~mhC%0Xyn{x&8Z0eg_=BMs%6?jX1c}mKOgEqr)b&}B@@31KSVab~ zb~`ibht5K{lfHdC`=9DdgkW$yNmfC>vhL!77-W^GroPqXMutJlF+=f0mC>a5zIED0 zj*g2dyufG(eBBuWKG)u>4KH(Tu6_TwB3oT`=-wIJV!?4%o11HtaZMrkOD^U;YQl}FQX3VL>-J6L2NPfSgvO~5ZSOFLfdD^5QWGgB)Q0)As+tjJ_`XwqqE2qr^^R6|22~C7& z`k4!Q6(IZ&_lkJ^$lNi#L0~u*)>tuddrFR152PP7NDQXhy1*Z?#0?D*Rm|LpIQ;Dq zp!lm5t$dRvxWOkCqP8WF~csaPog#IunZvP5BFZ`aYQpMUVL2C z@51~T_n&47iL1DT=0{$k{2T$1l44{HsN5YJ^`+HC+<%=2-RUd6$!y3y*+nPVXu))w z@sAhR&O|K?#&t2SJ^}h9`uNlGWs{yiGPo5|?IrR-*l@XEr=G0@WkvE@(; zq>@DW1Y69Is@Uk7)6+ZR)mJz#Ss%@Pd;2>i#YVRfe#W0p?3(i+Y9oGZP2I(eypzTr zS8>Kio@HW8YnkTSsXcG!tjbDsDDdn%uJd03`y)0k4hrqLXzs2DF_gn=S4n98&uZ4> zp+y$QRqoR*IDM=6eGC4}unbI`N)=*qG%-;PnKyC-ra#Iqro9z8g=?A3u%}3u0NP$E ztmII}V66)+)r{9W7s=K0p2MZqoEELzzaK0IX*IcX(2e49$hW_*@sP*66mvNTu_s0O zc|H~Q63<*9fj@JQ(XBc@dKrp5w8MzW=9T@0K=y<_tG8YdX+;(#D1bo|) zw|49)n}2y{01d)jRAx*R6+xM9-1NkoCvQ%m#;GZQ^h_Ll=t@yoJ%Xhl5-C!CgQXJp z->5@y79x?kIr1**mg9U(#rDOiw+b>$JZ|P+3gC_;fd7u-h_)Y@SwR+PfDj-wdpw6Y zh6;-$K2ZIxL30+RYzu%C?qspTvpzv{L?u|FywuVH0%G|dz`U5~Bc7gWD7FQwcew1p zj+-+fDIGxIJlCq`!M~e1bVtH`-*p~xsnpnBvEMC3d^n{=uB0C(C zlQ!fWgL_p>hvbD2K28NN=d7Z+K3C_D)#tGzyAZ1jv6;&N+heCmfRZq>bw!+=%NB4H zt^ga2aIVw=3`IydnE<2wLuxLGH;h-#JxjG1rPNb!vQhdDw!-T{cuy$25%l!R^gT;) zV>Eq~8qa`EYr6_^ek?};YXCGb5C_a<6e%QI0V0## z<3N-@9DhspRi1#;dk;9>;ox~gcgkTSgDRDdYXr;A*W2wa8?{_imS?!%SWvkt4qFxL zj=q$7*jKeVAg$KKF9hT*UFiB_1KPgBefnvGj-wYpfi9jV3ZX~27?=>YQg(MIQhb_U zhg4x7-X)HLt;c*Xc_v{&zV{kAg8Y*nFr*CaJuHz}$Z3!ql%iUT0bi#W}=68aNrYV#W@HR;DP8 zU)^X>01q)xm8qxJg0{aWi??q7rURgzpyP4}2Y%DHKp5bm`lvA~$g$!yHx)^?Nt7=vZvxKMEt3@3js^!lYa~_{j(qXns@X%wgqw`zH8t z=W;F0IWGnL6+5ZptDY@@5%{|zDQE(Q5T{#lYr4lzesIJ1B;TT7ij9717UIbdLCjnL zs=I+b7k%Ar1e4~Z$G^6H%j_<@+R1&2=@5vCQAd7l9scz&I~AAQq;?1@4r&{~W4euF z<=7MJ_n{M2cWK?koDhAr=`;M!px#1LYs9myPAKXy_tk3$ZW(FRiy+0)2SN=c&Q)Y> zKI(~>NxWeWCJCgw_zCR%5f&=0(=Q@uV|a{xBGYN1rDy4Ff*$AhEs(@1;dT)EUEZj_ zT;W5tsHCJHSl(TzR#Lr2hAW0(0#OT3HPzwXCX8V%6TrYpy0#zcuCS>{-|Gb|+{h z7w;jz(HWTlC{J#uM<+8JMVnmCM(#(7h7}V7(5}E%mkPvLZq2-unL>!yHjIEKhj)dY z{f%Sw`16_`bgINzIZgZbU+vTi~o{c(^{JY@Kb^>8H zBqGA3Hv~=LCo8u+kW zkYX=-UYZsc@f69852_)!WQo93NM=x{rG+^#GF+Y&l)RsxW40-b#yJk`OEzNN!)Hhy zubsV40N$ynRkTp&AWqu{on7SgaXlzjuC<-c=<4$a8>MJY84p$YR*E(Sd?2 zj(~r-VDlt19s17wt?EakUqy;&Y#b!(ikqbw;4h#A;|L;xBbOvmxXp%T?;SeKP|!E1xnK~RiW1vF|Y8<=rysOJjC{69r-098J{I!{R#Gaf7Q;AqmOHJw$-ldbykv$`T_hg_9W@~j2h zEm6tA=tPO!CCIi2=ONgcTb5|*hDfq%=`LC%cO*Rq$agGL!&%+_Xo9}OZPQPImBxT` z)9u@%R+N<)0p05T4cNt*nX;K$3DYI)W`UH8{LvV^R zapkvfo&?tzbeR9Gnb=KQjn1GV|pYeulgfBa>z#GA*1D6UQZV#j4yaK}zY3 z0Sb9TMsx;-Y+&7?j;L(kb_n?m%|-yJ*WN+T01JW2UZ6i5b=8m+q6V0$oTN9lUi_U2 zk4Jy&1LDh7h-Fo~pUbaY?e}89>F!-`l?My9P26;o#l`x@)|ZGwyUFGeM<$X1yrMOj zc8Y{w#A~LiqwROowtA5$&$nm>iu2EQv8gQY!t0>bg1Q@d+|nu|4UCX{3JAUko&j1t zx8a{O4x$R6ZA3eY_d9xW*KzQZn+GiiklIfeCrC*`Ps`yOCT;9nWu&ER$QXvy5Iu?c z)QP#g`1;-xVNf4Dd4h6Ni}lj77o(RThfC0(frA_rtC)AliYx%8K!s9S(8o_WB)Ayz z&ML?=fCF3r9{Fh5_ybvkykXQ$9vclSSsqRau;R;)KbonE1H;sVSK8=kO9|+QLi730 zp=`%Q`t4ZRuITRyF5bPSQU6oJ3a}d9yRph-EO4z0m9n^tn`6I}D>0M+MGgExQYyZp zK1BZ|*#JFPfmY)m@PFtkQXgLAoPkwDu7r5N0^{+Rw*}c9I+7qY_qD<`=pxEnpXzYw z35;odmKgo zrQ?rM31wxHbG7g2O+**&>6E>1t7vtA-a%ChO&dg}4ejrS-ql7QTeZrgI=V^>uCs$} zuV7t+Vm5ZC&ibxYtJ6C5B-15i1{DwksqdRAJS`|tPM(y4(K!q-Kr6m%74>v(sL6*9 zW2;2!3#QJG0KB+Jj&-PzQXuq4F(7QJ%_+oUQKadi8XG$?@OrS(Y{0-p{Kh7^6i(p_ zK_>-p2~3%sp;i`1fRa*JkP2D9B-lABl9Fgd`Z&tWV{>*Dm}(vMIb$fmhX~Q};Bs zUgO@nD;CQ;awf!p|LnX3r{VFKIL*!IY>|}$E|C#dEKA8ZMpm}GAZYYj47a?=aFGRX zYd0D`CP3~Mz)5vbQF1!>@Gmc20+)K zm?@NskC-xA-94$@Z}pVh#o_NcAhjLy%w?V zAfo2c0Ez|sk%6ZN-Phdj{mlhE?^;Cbw=kpQ(#bz!KjP(lDk9>;o#@obQDvj|Gs#y~=Q6ra+9P z3kQDM-vHC`GM-0^sJcr_vaULF+&rJ&1sFeR{hw~4aj^#s0M1#eV?++jDiS#|apCs7 z4+R~iiR^ed{a>dBEO{bduBwInjoJ>AYabD;jH;AE#~Pft#l^+FWlr+)@>(j*E3I9d zANRbsfC*pBJHX(IRYog8hwg%_N>kdLcEpnA@uicv2F8%HfhB^0PU9fJYC+m^{^GRS zM4U$7vg>g!=lo5H(4JFuK8VqH2S3Q~dna=)oncLHZz`{d4YKQ5^j1(af{rv>kQJ)qBvmS{elHVf09DJYWjHflikJyXpEOzauy$Fv^$ z!HN%C+(dBf(eg_{)k-Kbu)q!2#f0O4Xu)r)jlwWeCI7`W`*#QA9Ag2-RA4($js!HA zL}6oSu%lx`*nZHy8^7E&Z`7Q({k0`Nmp}NVn!hbJ8Fcf0(1c@1l`bj*Jdjub7(_Y( zJ-Qn}1^ovQD9~)6@#>hw^ff$shf~7VyAmS+rm6datmRCa!Zx%5a5PN30-*zq8VC0A zZvM5C@W+osLTmFrbNW#n${{fQN&CI_r$7UVIfWc3jUx`d-FFKeRkRyrGyPqgxYn#c zq2<@$D`fU;YVXI=JHdMYfdUH&ZBGkfp9q-SseHxtwNP9(% zII~etspYUp#=*~1D3LsHoa4ZuEslH}HW4PheBn#jSh!E5r6EN-u$`)}v3o(UwDH~Z z*H~d*a#a&?7`HmWFKI%@i`61z#7HMQdE_u4$B#NWI3YjPYEwPBUA|#?6{#7D%aN+a zRNopq^L}hE0owryY$mW38e~vaRn@_vaNr%w!tE?tx`0RIEd{NE*$t4||AmK};Iz4G zncI+?8~FzL>u5sFJ0&$Y4$@<$3M4!-kh?~E400{4ojfBME*ro?8A(P+Tq%do@+c7> zKs%a4&j!8n%o%xrY)eT3I@u$%1gY-tCpq*HP$SyO4L$;<4Y475Vjx`*?cSJI<&efM zu`h3#Z*R-ZbThaUTZ(-O2ou(iGjOU}UX`UIaEM z^WpU#9F?%2n}(5wYwcg^E_IuGA{D~l2o#!3=fFn6h5YmQEU^k^Ax5tR>I+?7EKWE; z0Pi2?HnpNnDU&|Q(6Aw#k5Qce^ncPqTJ`qU&&uj&2b?oTR59e8`gO`+G<8SQ-T|zY zonfUSkb+z>Ok*~K2Kv}ycM3m{6Ix6sE(7PH%ZBl#b`#KtAmQpj3CIS@#G!y6mx5kk z#-YD@fr$)%lR&DkcKUW58MvYw-oSglG(7*_zR6$JFH^A99zK8QG@^ru%Yd(t?a8zm z=$5)7;n#Wyw<^1)IT}dRm3~%jWTaLHC?JRaxfXY{S<%3BZrhTGeAmyIarIe*la7z! z1{sRL9}4`Dyz>|&ZqIGqBs3>(j1JI|6mQ5RLi{{m`*U|>+2i5*N~9JbV5W~Tv~c?{COBP zAOS2vMg!oP6D2QB`B|u`!6N_4KHr$dn`^$YCb+hEB;Yg5LHS3;3IK5k+_o(lUjE_N zI~Wr;e6s<8_;%RnN%3L*{h3Wb3t}7qq0dRm0_g|YfaXJlTg#VoiqYsNQX_K?=y+Q* z4&4qA8+{zHObVDs45(5MV7M#d=_7DO@^2oEnA!GvVcNQNEzfezv`WM1Lg?)NRCNl- zJdO6DLB&t6+E;CNT)h4gLuaR)ml|4ybS3N)Si;YCmZ^Mapy(r$)^PmfO*_;XF*_1H zdot?Dog#deIp#NLHh~VDHhXMOOUwNyZxMW+c%ICDD0&&zT@>-%%N8SmAU=#Qcp}Q|OD537LOtov$lY-n^TO0I#86Lw_ zntz$d+KmYhcVx_7#QWOT!7Zt^wgVJ0FjEOl(g@mCl$!ZQ> zCwAijDjsmz3WV%3bCHYed!MO9987@V1zkVkDPRY)jfpv+5v|8G3xP*t%fvrA)dIEh z?9mpcpzS%MD}QI#V@1Hhj@jaKbANrcf3v5fa^2+5>VC8=Zte|bFvTrZqyZG*-P-i9 zsDcsh(&}Q?&&tc=f#Xq`f-a9dh-T;oXo46A-p&FdYUQ@V+cgxR{8`eC0IWax>1XQD zk)Ar2$2nuDq0u0=q04(Vs#T?bv^9pUVTJ|#0m`ykaNqo$fjVAy7b*c+&FhRTXippeuM?kV*H zg9LA)wGe{QB`6TYZd9WsjF8K86kwqPytQNz~l3?_d(t95zZ{5NAMF)VP-9zhFY(F_*Yc-*g z^EtQ>L9+!HDvbK6|3wk>7_bP^ShcFqp+^MxE^zt@2tgN4a@;L=^2Gl2Rx6*^4K?q_ zJ7~NIe%YjT7MkTWzF6J(F?J(n5w0~A_#bIkZ? zVNG<|uZ-kwPFLvZ-+aLh_`;MV8VCafd?0NAVHL8`InvUf(gyC{y_@Y6*Mr7tad|oG zycye@hS>qwYFc>X9HB=gJq@ze{mDL=s)Fb;av#)(8nX} z6!ccSH>#9qoC~J>IJYeVQlhyx0&f#N5C(#AQ4IzbM@2gnPOpFGHTEv0#IBDMNPFse z@_ipk7%~E8AXYUrvQ#?2<_c!?AkNqna`wKAo~=A1PTy|~FEX6CsAeM3DnM&RQ&V8& z?jx|&;sSkAcDf|>++%Rao*}b0Y9>1bK5Fz8qG}oTQWKAx#{p1ev1vYdb4 z*{|KHLlo%nSF9}kV8tW(nakN9*0Ic4oi0(C{5R{PvPqA7-jaXe8g8IP6rmwSP9cEi#(RiM|fJzB|0xJu<@PElCV&`(>cFo52S|T&W zYR8BsOoKSjhi1|V~Dz;N7k*z?NYZwf5)m?! zALs7Gp1gww;PaHjE?`btusrFHw@thPG2N+g+BlNA>7(fPZ<5I*nMV_`>Et}M9^(Ez z!|cqZSkUg%Y=YC@u1y2X0wx`i1H(V!)wMn9ihSR!?Ohj1PZ^J@;H>buOOrAVLEQ!; zl9_i8KZ-q0;}|d|q8vyFkuR{CadR!rISSP|pOf?7SZS?gKfjPD`oUrmc;5{*7EjME zznHXa+1Gn7&t_l~?gKMnO6Nh5jCC7zMpEXCepBokx*hTs;u1oMh#r89AtN}kBG3mD zwv2O%nxj`a1`>)i7;MwoAlTWc?^6)am_0l!i#1-n8*6;t!RN9)aKS3L0dphup|p31 zcJrnkw6e7Hv>O@kJf+-ov4c#BNBu;AjAQ(?cH`jxVajSZmEHwqs(d-?^>z8jnE^jj z@JYNRhqo0mDsBK*(tp#}5#eSuZH-bB4imrI0sZ~Qlg1w0)U2%G8IY2ehR2vLN3_>d zdvh>|eG8x&^n-R`QPa#u^bts;4u)B*PDk4{qEloth3>SaN9P|@v&-0F^X85^)P%@q z5!vAo^J1LQSTq1`c%=&B^)!`H){>5z7&OZ?HN}6jqx!>82*)t3`xt#%qtfRhE`=Rrd}p zgq9ANet&yXz>oQY=>0Sm1xkXgiU8+;S_-_!|E-`43_95qg_pC#6TNPrTt)W7x@189 zdoyZqx~ITkcGt^acxFC@P)N!s9`qu(!~p>2klvW&E4Cvzl50ms-m?pU&pSpGtkYx= zAwdKKvIdz$J`kgQP!f!bHk4f1i;gndmoFy}Ht~|QT5epgD@KKX%O2?+*nV%`v=;O( z-$0`f6T0RhuDT*^&EVo+LYU)B!|Y<-`5YR2pX=@EcVYC6SB}U~XUtnW;EH2QL#I=g zn|}W$GM)IFe5&H2Iq<9Fm2t&mmD3`lEK>i}wq2XPG7Dg6*N97ohg*Vi@|N?aIP5E2 zfzlUT{Rn5skdTlG8pz{Mot2p-+XG+wU zD44%lWBq!2j{^RJkE|a!WYUhJnrG?&8(JVBj2zf2(v}Gq{X)$3DJ11(@<73%FO!BH z_I2*Un&%dVIKlVclCek(!q3Yi?#-a!oQ>{6Ffmuiq8`B`3d*<>PwS_;n*9+M6H}lnGPS;an1_JWz zd|#u;o$R_kDNYmYRba`ypzlHv*_p0i6iw%>+6dQ zLVk?A$>{D#C~+ET`u>#Z#15te!PsY4t7SJfoS}mQ+BIyU{J0wv^+tBs+V--3>FeHKdXAc@Kz}A2B?1bVdXMz7Qv>|{ z6Oxixh~xhK$oIoPKTJ3RmXWD@`}MFP(D3bk^}R`9l+b0b=dX^LaC+wS;RPOB6B75V zbN#hdYdqQP@AvE$O6xH_8R*B%am8+D(Mn|&>G|_ND575BGI=db5CHfz{vf>1K4h#MfuP9NYfT*IQ@Z#S`Cs-`xLP9DQ$3Nry$cOOKt= z*y=s99JNNBi=UU>z!N?>v!fPb!sr7+ZoA6Bz<}^2Bp&f-5eLbF16TZveMA@nve-D% z^N9;zwJohN+QDUP(z%Z&9l`!)p474E5mk=`f}A6L;#GH@Bu(66MkIkyj_h1J0cc^Cd3>?J{oQOH zT>}H^!@Mj!52(uRk4WdY2~+03xRAq+m8o%GK!!8hcb@5vZTEqCWe<0SMweweLv+4% zKV~$Pltvyo)Z6>c&M(l|IN#1oNKMh&Xj!dMz>lx*LII~TKz+ah?cL`uLWcp417ih9 zBdFxK`*EKJYyw{@yaNu~9s^GdpNN@)%1bi!`X06Vem{YgrWunw^3UD;&|N;ftk!*h zZ%>-MB~qVd5|xViLXfO=opRcz8y(Wq!I)h5um;(<_>mZmc|UZg zaBbLIEwg4VdwJ$mK0Z^wp}Tjd=tgr}6rOk(uD~HH|IkbQo;CllU$?MJ&XJxY!-u1e zD4xHNduC~#zULIvFpM`>wBs(Hw5fz{?D%mehFQp?3q2M~9ISx(^Ln%pK9%~>oC!%K zrRhhJ3*Q-f+_U$5G*4fj!Inwaudn|8G;FD#`q0z+Wn+oakDs@<@I-QhGWQE5kXM=NMf3D_gcJFU=wz%-qAZH zYzzE5W|U4g^_5XnoTCYc+yYv8id?btdpy-dUhj)txUR??UP_UM+vBz-by1*}oy%t}4alzPkq;W?HkRibB+wTGc9*KgI zRGsU7l^sq$wXGYv;BMZcQr}k=q~5#t`N!R9bpT8Kr z%W{sM?xjY1)1!rB?9FyUnYdX zGw3K#(*RBI3ee7r1r)6GT-nk!W@p|hykI;Qm<}A=Ul^x8GX2AchNeLCu4a3=YuX&H zz2+!II}U1VBZySQ>=d?U0%$bTy1!ZJ(@E$WI&bbAx?wqjS4@uFo?_OQX;jFL9b7qJ zmAL)T)Mcn2vUT-uXelu-;w~Aw2Bxv5tedPSFNX{pboIXhD_THc8?3 z7GBd0bEdM7c%2!|LXT)2v?`3#$IMiCShcjEz?1V>cF&e;;8*UoK;Lqg&p2&`xK|vx z@ZuW4*gQsXpg&g`Y}&lJ^8S9~VN9Q4um?2S{dwoeTYc??8@6m&1DidPjJANBq~fWb z;(16hUSQ{m39M?%Tc3~TL&44_aFYo^#cN};0Qvz7wA64&!KUgBA9$&3d<{L>4@VE( z!cV3t?R!}aO1`eR}2(_Vd47b0QXkmH9E&`Z_i&E zMXo3QM2PJ7-8J)geody*Olzh!`pF@(#BKd}>uwp78L&e5kG##QW_lT~pUei1N5#?l zm5yv_ak}Q9Z9lgRSIqsKtOaqY^@THIp)5Go!VF0^r;y_58#kmS^dba>W8!$!j;@no z^9ytT$)0@GJb`vaeXzWua2ZG~Pw(B9&1BfO?_k#+=vFU6WO?J}O-AC~3o6=N8VWQ- zn)LxmizP~b2ve3w-N_xVEp#d!Hg4Ot7Gf>bHBE8hwqlrq(WU*ho=0FOCn#L45qw#l z_n}5cEQ_@(ogU5|pD=Im{+@N{A}J3nhWjn`{rCSWIz2rf43RHhQ`vnq2eU_dMyxe; z>j&QV^cKi^Hs*};G!2(`fCUkPP)GxhL7Xp-29i@OuxTrsu?a7?LZqj1bIWw%zf^9p zNNKm%Q<^*Y^Wyf|YFT>ORm7VIP$GPkN3HkhJKK!XVaz$XYq`O|fI$QO8`2eU%~)b@ zHoor?&%V2B#RI_93M-qKYRC^Dh!A*lRZ~<@Y0AZg-mhnvta77Rj;>+cH;9k$ z@2p7ND9O&j0pHXE%^Vq@WzxQ@?X4Az^2*Xn_Jrsmwiu=hTD8tEbyd5H=P(fQ2~uV! z;0!8qyuZL;Z~&~gVcRzL(Jsk`4ZNT=I@D$|j zHKW-muC6X}6bkD(fvTF0j&~-EL1*GR2WtVoUIO0?j5$bFjzU7Mos|I6fBNSu=`#?_ zZPRjB(Hpt6s3x`?Q+y`qFaJ3esu~6&Xu#yeaL10X4KuCieS@T*^d=ha5{Pxz|M&*f ze$x?j5u?#kf}D-7@2Y=5V^>LI4kK{PtP;7Kk}~ zd=|65xn2VO3f&Y*K~5$*3xVo2_m1}}422(pg$BuY5r9R~D+{W7dNc<}%le(4I&Z#r zj8@}Q$=rHl{B%27IkEXb-llr>zl^s21=*`52(EA&v9~`PLmD`As>2Lk50o=jK*kAFaNeO>is5 z**d#+xi$x_1`97iiKc;hc7;Tzpx?yWHg6Jk0Q#o8ckhNBIa|(js|XlvGU6mYbo8HH zax)=;`WNTVpQm{eXo2h#8DO1WPE)`;#hHEV=UyANRHfoyjVS5&BL`c z74-=^$!<8Wgs<53P54X}0Vmk_dUx+BCPf7ROovJxNu_a-La>n^v-b*^rp#NgKpc%T zP$%I@R8lOUxNizHxNK}4KVDYczgbE#%>h14s~3#9pn%Ta1KThLSAZ(;Cmy9{9L+jL z^coVM0mFMW^YhDi9v>f{-$ki-;w&1%i1SPgsRSP9_Vji1dC;=Q>jV1&Y%j8V;XXJ1 z_+B|c_C^{8Y;aWn?K8OUia~ODZMEOSk>1sUsx`EIq;@gI=)cYi&WaN-~O4SXYixViBBsh-u3(zVr;2pF`6NA~vu9i9ehBV)E>5 z54qic9zQaWx9-Kz$EAdL>VRFMQ~zvasD8u&4F9{Jl80GsmLf4H<0!N?wpKpN&$KQ2HBHqt+T@giu@ z&u|c@wa0}QmzACM*Z`WHyD~Vr5iYZCZDAa*)wG$tb(L?B*M(>ix6c_1$uU^@|+64Lw{M(?; z@#Cg`h|Y!rFyj;PechlMMSxfE^;{F3Z^+XuF>?phi|cxW2K{gg!@$4Y?;&E;DvPww zVZHKi@oF&groKA4mPbn{u;HMv!o9PWBZU%%LM#>JzkLi;OpIDj>BtGMG0g zA4FRbZ>Fod$}O$6pMcrL5^8E$n;$>G^X`B~&1dy2w-e~s3p*yk!MqkDZwKLZP5h}W z-rwOZitYbM0+#pW$a*qz&CoENHP~A;cH>Y)o(+dmnWR<@IHot6m<&GZmW;4xB2FR1 zB(b11ZV+;GMkNu<87Iw|tZcxN}(-I|6JU0w{s_OMh4 zlyOVb`mVwz=QQV}i4BSI_)PM13ch+VS3=kYSRX2PL+b*PDJ zqmT2+hV=Jj6@x%WD%L?5&kyCv@eyyW?@hU9A@;^P07N$Teg`KD2sL3dXDbS*l>pIr z{b-zuyO>QJ*=DLv^6H&%TF7d2BfsKQl)||m;D${^XE~DK{>snP4aAnE0%RF!qiY|@ zBa$rt^m8VB8u4$iHDodM=+q{g9jcy@NoRNTgC@C{2yi?idN{(MU{~CPMMZx_=S}P#`abqJqH-m)eGKM3!SkZaA$#{*x4z65al&x|XBsOK-yxUlEpy2t z$5GN&o#4}(4Mq=kJGoB5E<6JuGx44Scph0x z0w22%4Fq*VYAUs0L2OFOIc&4lx1wNvRJOdw1TIM{;6VV}n$&frn%l6V~YZJgoOMeLP!TWX6(F&c+RXyq`}vJgKS%vNHanpuZTv>eK%?R>ncN~Wo5}4^Uj^cpnevD=V!fn>d8w( z1C1X7Ftw&3!pZEFG*m{k853V4@!po0p5E_NQhXsav>SelWcf<-4jAZqCG9$~q%p#E zHA;qkXgJa0 z@9aTCk(lGISLyUOybY6r@iimfE_5Ry;RRS}MuGxnHLO<(b}X^2<%DVl}i z1ZY*d^C((%vOOn~TK7yo0P5&VOTbjLX1 z;S99~sS;44K#vVP%P02Kv37Ps0c&D^j7m?#+c;+bxy1d?o;{=D0Tg0b=B>ki1oHP! z9U$@J^XM)!y2-2=V^D1VJ9p4grm6U0oxvfc1V%w{W?VbcTj@b zrngIgT!9+{ci2*J)YZf$0=ai3d1S4c5sX*q_31@?u-*h zAoOPbqyGcq7{c93HSpcv9cVPzxJzv6R1nkg+9LmaZS^Nm1)LV7l$bn=VLgg;U{zL% zPKa5*^O`kBYfakTz8c*QWq&RaJKOA`Lm=VP5JL7R9y~4sS?f=OX6U%;+K@cu~$Rab`13(lBDv2}ziePy>jT^}L z>Kp?`zvuB}r1_$LD}KMWS1xFmUd2g8W$ZX4w>m4~j1c$&5A6J&ufL=;TGb`K|JOAp z_|xMLA9jIsOCyNT_OV1Mh?K8n8t|LaaPu67Et#O9k0OSjxc*&@iKg@a|DXb&##1)` zol3o$|L?0de0=}^2noWflZFQJyIMernmP?G{G&(nPrcJxL@A-p(tc$XNTY9k>-H()>&|ooU!Dn4@V=b`e=4~ zqx#QZqRFfZEk#)}PiunI0MTSY&-%;Mksl7t}#rddyA08^IW2VX8Zd$VO#OU#dt)|2h>5vjjF zU;LFDRKl%jpzp$ckijZup%n}KEK@lb%(VSsUPoi&+%_i0(Z=skz(2TGuu=Slf3js% znX}Q_(}Mq0by(vtgho zzd#^=ooudBE6!Qi5mD3+9zOHOS4w_Gj~bK+e}Dfr)O=tegQqJIwD88ADKZ(HfTUjP z``zE)_=Jxr_1OJ?gngrGqs^Na?=GH9zY84}oP|z#BuGs?9yABtqu00dOUg%qv*4`r ze|%y5{}Bv;ou(FH)(af?z>%hCpyLNm6uMyg`%22C)`1m_=q3AKp9VLUeDm-~E3{B^ zOcvqc5XfT|7>&p6^Z$sD2jM6nPwv{Jg^X5@7J-2_w%dDEL3tJ?#!E_k{Xg$g(NoEL z(rKJ}gy6M<##EprTnBjvyfi^pS~Us>^?!~59Q0UV91g+}F&WSzxA*uZ>f9(rGff+= zV6YZpARRq8J2=$}3P1;d4oJ>=>hnvZE>f|{xqs}KS4aqTFv%T~W5BXuh(CxTU5ZFK^3 zQAvN_XRuLj()8&F|Atc*?D&7}efdAt`}h4Rg%G8KB56P5Hqu#^4obwu<&;8tc?X}l( zWWm7QqC>XndYS)`9h^eyitu0M0=f?<*?(COzWhTka~7r!kY>W)X6ne1BX1uIX1tCw zQ1R82mez4m^7zg|7UEn{OZ<8MY8YNWhe;!FjKQlfQ4=4&_A{-`TmLsF0G0y) z18U=o?j2k-QgLt4Dk`L6D-it2lFB{ij8oLRs_ikkyog`Rd)IZ3`IO z9D|PRte~@+3lhi&xRE*bzUN^@eNDmP(Z+K1Du1qWS){5hcY3uOKY{_#o6-5#Xw`pt zew@{Qr7_U{+CL+eH5O2^Q@G*py)BCbTMjoRGlo|MI<~A%)G&=2-smwa&$9&c3^a>| zTwW+x=OUmcYn7LmSNjk2(R>1{Lxi0yI0>MgP4!#}D%&NuD(Q2i*1_~l_>U50u&aaK z{W-1qnM@*;#(jJ?WQl~fwz`p}z>FETz}z)&44y;LwO)gKS)=_ zO?sv!_jSvVvr+SZE@nk5eF#jjcMf)4^?wx#<79Jek1YaF_F^`yt|&VTj|mv^OB-l9 z40>N+&SRh|9-itbDx*W*XZ1|ho`itPuf)nGOi%H$@b*6$&(V0G^I!*Js@bk4b1x<} z=CIk0Nd0%MR;yr!<<)pbCe7pd@e);O`E zVVG{w-^%Ccc`y>)U+atAgtrtLY0Ds91ejham|r>YnD&0?K|dB?EfD=Fe^2)`)!q0!qVkpV;{gs|HSo#iwbcCbdzI_Tzw3Dcs;L)?? z(A%3_=-i5)t zf$;KS1w&Sd1JJ=OFZ#u&4srn?^2HPpAzgc@wF&>%iE(A*(C?ngGqXeEe9Ktvis7WQtcKNO+HpO!XRT@-S3 zleAa|g4rQRXq9;bilCN|R!lul5#syKXYz>z@PmcFeK0d9_x@X9;M(+Q`p&v6Tt>yq zV~_V2GY))haA)NOy~_?Am~tjxc&f9&juEqJ1O4Ep{Y!DcK0-YYQK#vG>6QUtYFk;- zYmJm-w25de8y&*q;o-R$Z(NB#tcO!T>&5j~?=8Odx;2 z-x!22qFe&df>!ii(2SuvH5+!C-AnGML3@Cp8j^&;`A9~vD7|b;@rdY8xdi&^WIu`x zrV7MllHcz-SD)EUCb#VP+rLfd*R)|Q_&Fj!pVe_nKI8USRzDsjV9ZmyF<_&i4U>a!CVGprOyC7xCs?wy%Z?S3uX7bYbJRF$D@FF4}ogv^HI|^qYh-g-kq2` zrg`@m%9~ZUrD+J0Ta5I$rE)Q-A3##y9Vw2<9&b~{F?5dKE!GyEDw@7xT`#k zlwU7qzallrkv3rVK(Vo$n<_W_xpM1&7L-GYO7r*pn8!-EOeIk)n4nEKPpt~OP%EYo z!#NR^mF5f_$OUB85nln9R)!zfo7jVR#mtw5YPG|wp*Dqw9vY&_xj2)Hv{P(abOWp^rrHwVo;7>ky?wrGNnh# zxBYec6%94DJNDlAnL*&w^dex=0?@XWEn7AjxD8-b)EM4B_9d^5Uc_a7gW=c0#g0`| zxFeZsz;87I4oT!QFkDx4JuAc{d_~|%JQ{%3ArJj9MH2E2W?$WwkR!e{(xbaLUUG+60#Qt-1E9Z zC*4BFO0NSmczY|q4s`ktoIQ8WsJ0tj;xFk}=BcXQw4aa(@r>1L*4RzsbR)_q49Sfl z-uRw{g;(#H7O1PTch-iz@$B!KOy7lFr&fg)&s@B(^>?nwnwP<$tq79ac~((TQATpi z;Y44S>T6tDd*!pvWaHIND&w~Ru2(l5yfqTa0nWMv@c@7ur-hcs{Xf&Kaue2b{9d*0 z);sK_)J$zA-%Y??c!KEj(azVI+!9{_*)sn`oR!!cVECXDLK9`R_kWw_n>0Q z;IjO+e8q|h5H_LNYlya@Xa;8hguwp3P0d#0wI}*@&S$^CP4vau7Quo^=YyJd5PnZ> zy2%SHOV%msX&|MHc)wa@JaKs!D(up5D1Pr$uE<`H2Ctp*=^Nah%3w_M2#)VIP=h9Q zB$Ryo019UK3Y)KM8_W48`7)<&q_QE-R-tH$F_cl`3|@8s)yr<}MvF!=%|FAY)YA1K z^?RJ`oxVUFKy0JOYm;Pe6r9$-#n}NzI{7YzDQum1+&+GLHfrDxAEx%!jsl7he>S(u zH4i94sZrimm$^hFHPe`9dot!3G>FY^`AR=Z0AZx!#qm?fKQV3jPPuPUFZx&sX=&j) zlSwP+9C&78R+bhPU3b3rOII?1qoeuZ9LN54VBq-4DSE-i$Tkr8EYh|Z)VqGZMoDGB zS*!g3Au<m35^#LQ7u-gFBHt3_o0rU|7ai!V9<2ZiEMYy;}6f;hoc0%a5*_S=p3kH z9uk%lRm;RiPFtFrW>lpLj8h>TZw}Tq-Pr>od_e77hGM9dI+tJfogY}wS`2} z!~lrN{RG)P3R&i_8BR4Y!HhQM4WPa8XQN;D_^;yVKeFX7W3l@*u@1)4ZfAhZkB9Q$ z^9DJl=9Z55SKZ%T{hzz~_f#qOWkFO<4Xl>W=sNW0wnc5+pK||x1WvLapH_nHb z>dFn(i6;+9A}2&LGcZGM70yI5S+VZv!bmO&F+)dx3kJkP9(tl$qJM=KrF5@e7!qlY z9GfTUa`7RMFl4f#kOgYB>R-2#6rAPTt6J@?T)j7u9scCc#;h(wo$(19zX1Zz^B{0W z=Fg;6uGWAl#$^!FIr^rpp`n2b#=);rGd)2qP#bm?Y|6?(x0^dla4LZ8Fb~}+8-(@O zV;xaRJXaP9iCA)E7J;+U; zj1$BUp;uN?^2_`%@>wPX2#_%lvl9LCk5J$GY`(RHbp42SF30=;X4V4dq~w^G{Z1+O zf{~Jwr!P6(RnFmT#5=j9R1NA0z%HVqz`OcMNCc2BZC|vQ0Bd}aXC(oUGm!U+PfV;B z5sbNp>>XGrQl$-mLSs?DS9?}%M_DM@@fJ;NYN;aaB56rvgXx>tr%8B>6&v2FhsV1# zL~zM=4(SPY`~hwvIJwQ5X(ez(oHXR54M56h+)WuZ=8a#20)bUD@uwH+t)*BX2ho4% z@9k=DuC0~j+(nodRvlZLcq*nQ%pr17@A_7NdsCya`EB0taENZVEHR}YCh_n)xp)iu zQ22r3l!?hJbZAM8X!<3z?oP0Srr8V5$rZ16e$gaBuAVxCD|z zxd=YVKow9Jk@YB=S*uM<7NPz^)Q!gp31Hb4Jd7Y8yk zdq5}~N&xEE@n7zLDs2B`J)c37ndZbdvWpi>oL?Tc>Up~1LFvA3QN`MmVIOZ7%de6N zdN%7!_~u~aaK%+ws;8%>2^`(7KD(m%*o<|7S&HYJQubE6rr+BbwcWm^xuU1-o=f`q zsy44|8G~#H6Hk4M#YeLizw!4RcoUewq+G0UV9}@%bJ4PPfPMv%HH@^Jht5{P;N38H z;uYwkVte}J$z|O4%1aB*XJqJF@YLjsQRinYpGDrj0jH$O6s;Pe96V&3p=T zz_3XZ4FOefFvjSngH{JE3TUv)lY%DN>}Vv3xx{+@2&~_N*+_AXUL`B!3!bXL!26zSZ%mbL5dEa&u^te8EnVJ7#&VpomgbDQPu_5Qz!ESk^32VnHFI9 zjIClq-=r*QES+|^9H2RQ!`SmOfS|CTUxPslEi(gz7Oh3NAc=~kH>lpny zyPs$af*_?`e;;KFX1sF|E?0meH_?W(8w>SS#>auDNL>O<`l~|z{R`wbGma?81vsVb z1DV0G+@{|rY0dLTv`*WvPH;AjCe2MyzHHH73)$%S17%hm7C8X|5Urd4;cfQrW-HY5 z1z%IMS7DVy_b_AwF@N2`^(qB}G1oeqGlXMm1})VdstwV9k%9a(vKdjoSTsa}M;~ze zWK%S8Zs!Ib%8yH25x@$|L4DZM7##v;NzIf7=&Tum;DTOa9auk#vwW#%MR~;$jT#c3 z>g_$Di3xWiwcmrUF1njPWjSpJ53|uy+80Pr4>S)1m1k7E{XXK=Y5?(_<);F+m0sR% z$@yYnj)nS5mT)7ln+)1U(UeC~xI4mAh|4m0j9f|IhKF8ikg*s}+aIh#o{Lwlv$a|C zJsaE(Y6P-_Yh3JR;$Q(+0r6`_YNjr(oiE!2?f7>Opym1T@uG?pL23)c)v{<@I7May zw4h1^JmueeU;&+h()%8s7!)r0nY;Rs+RY{D zOBysKYfS8$m@ICh2;$s8nspA@WXQ3?!ot$4>y!y~z}C#PVBf5CpDZ>0K*yl5Akr@a zfiSv7;?5qTJAA7d-PaRxCYE)aEX)CS>pt?P9Z0ajORGO{R0CcNIM&=ycQPMDklO-* zEcgUieDt>bL?YJ)vo=~&-XA$&d^by-6%ofG-9-}HjSezE*SXuvAUp;o_-*4nzH(!h z?@Y|!FTDa$iXG1t)5%b)Eu0OaBOs(al_|;OEEOfUwisH~57?pCH%*2oQ%KoB+9Lh!RgEs46LQLB_XP z=Tx~M0Eswta+TW+dM1);s%Lck^9flhiF>*zIstc*hLTv%a3BO9d*aL)np(?Ut?!EJ zlkNkfj2yvlY58lUpo-2m`t@vy#{`=(E)Uh1jtJ+93o%*9x+Cny>?&+rRIh}H@g(c{VS3= zgG3Tu+JkSi9v_o3N;Zw3$q6-P0!Jp!3%7K+7las02qZ9Mgc@h`oz?C{=)~QWnLz`EkMF(p00imSH&cirsYzomre*~B)b7Ax4XPYv#-%2v zPywO04{ocJwmM`R$(Y>5C1p@KKkOj(GK-gtCIexaPk7aU_f%})>goaqiOmR>q{_g0 zJMcG_1wkK5TXHeoDy+F=l5e{S%oR?w_@M|n2WAjxKDRlfUQ_0CqzM>vfz0W70t&p^ z9bmkJWerE}H^N}PV0ySNk8=YTpcSq_ArJRyv?m++9Lm?z$P5m-3uyJWk`}AHbrKn?5aR62k9~K5$~W{su4}#%p6#_1oIpyLP=6VQrO1V2D)I8=oIjsl23I*6<3ZCl zv#+D-#83aqeOwrggVF$$i@JZY*I7`m%}K$ZT@6ufu+ItXBvB-63CB%SVIOygl0ur+ z!O$>umH0VtTx1cF*F*>fg7;%o5gk90#iI|D%LF;>}_H$OoTG z3ghT_kPJ_`Z$Ck86)Ef!2Iu;PUH@3r2#lqIm!yS(J~E1t1d;*a;QQA01asHHf6VMlZd}LFHv)snFtPR3c6;PMFeED(!nl~huoQ9u|3S;uZ zOja4mnfRUAZL89Zowk$PlT*Y!qLsF834rf6jLg;e_p-{rdU~HU&6u! z)Ur?XEZ@5C>CqeJL88;KoU^*54;bd)741PO$Jo;pf0ksJLJy8Qod7Redd-abPoILe zs)vcoDIN@~EtUss$pKhq$fVlP)B;x1c^f`F4H9J(KpP(=dPy1D&0LSMfV_pj$aAbm}$4 zUf5Oe24F=4*|y^-$j0g3fHWrg)gW*cqWXRTOYAkjk3?umxMYsqzi9ld5%)cJvG;K1 zW+`Vc$k8$~Kwl;maUo4QKai^I89@3z9Uz}$H>T#5#|m%{BlHDr<+g)zAIQQg?`SF~ zXndqz2u?UM8Kp`DT59~+CWqV#r&tbh0mSk?oz-(uf&l14@A)hMH!M)+#`x({c*3U@ z)=Y)11mq~w01T0-2j;L)Ys-Q&j1W5}05De?l(b<~CNARp>gCH%K&9<*rsz1VuV0~b zFMoOrON1TiE9~ba1K{}-3O5T&j<|f&+{!dNe&irBxJm#RG*^SC$d_(n?LQiFcQEg| zZs~v__#D=bHhASEl);VpP{5`#qVqX&>;?8L{N1cd_wBs8H}7ZQOv2SG_)Voai~%cjmK zn1=l_^`LNCc6Hq`1Ua|^;alzP?aLIKpvIfg(m|a|uzNT)LgR~-4G=NY2cW^1Jt)mUIyJoq&&eiPzXn)OCS`2 z?=+M2_hH)7{jt{Hd%_$CA^nuB_FEG!yQS@Jbq5`F3E-A+^ytjS6J8v zT%WLGWE5ONuu9DxH9e;+8S@wHlw#_PlnhUwfmt3X7-Er~g*>zezk=s`;p4|k*3=He zJ%3+4CI}c9fMjx8pdbJjMZDCR9LqJ2Q<+F5pnej9k!C`f*r*6fl}lcHC{=PtfE>UI zD`s&bJvOTW?;qXs5;JUhRbw+tN%Icmx&f8Luuu!~PfYVc(* zIHilq9+mcQP46l34C5=9Re2j-KQ#3P#|U=yR+G=BmbW?sG!eA(mg?7vft*^Sje)8| z?My5JEmY3 z2*)&$pYFQ66F=>%y~9UnB0PdwA=6}YpBH%Nm06NX*%L2$7qP!kp+LLxR7AvbU>Fv# znxJ|SLI~uXgu&}^P+hneNa?s^x8M6~T3avfT6=SRGnto=u1){6j>>oe(L_Np zCi6I|Xar1_y99czn^Nc31&(70P@n~_s`~omz?BX6W}$^O%m(EHD*S0z3cbPPAO;bR zX9{|=(Gs|_E_D?enN}!+OXH45@Ap?+w>%4eAOq5$7S6exh-qr}{250Xz_9cnKor%Yh!aSYFXJ z1cd=w74Ypcfaqh4*EM%Hw+xJt0-lb=%Ana|Fpapzup8415Svgg=-Z9{f|Xm-kBjsv z?)UnzQzR3v8s7>|n?+<@wPxnR9Mfz+UqWpVlR*}&HlEqtk8?LL9eqLZHPP4e zCd_k_requiHC;2b;C~$VV(*cdyP`b773$mH*3aj#fXvE%{TMxNy6)zw=wYfktsh5= z#^h?{$Ve!n4WolP?43<>)b&xiD@yc1jOzG_>W5JbB+PNr=VLuk+XaJ;0ZF8`m^4A2 zqiy>>Lh6%VZ8ER;+%cE@CkYGNv~EeV<}7o^0!R7#pyT4)<)WTFJ{-`GV&+AhX63$y zMn+rlzF-=)6Re#{ZvD7?TSPAgwAJ?^5HGQl6r9+Aq^I^LYh{PW#eDg6c2n`>&?7;r zwwL$1J5iLB{~~gALL{9kSWbAzpn;}Uj`joza!(IN3q!wTfde**_vMRU2xsIg2Ds0D z`6)U0kw?9NZ^sDu*+F+GDiO(_iA4xkVOP#i&J?IwAbfgvSC4%=NnJp5_gxn77Z$Wx z#QDnlI|^7W^_XXl-&_z@;Bo`+KDP>36uu7NNQjPueF`Nl0LJxFaEK5snM5GAOJcd~ zt^D%^i+#G~H@J_6xZC=$A1M#69mI{sZH_e`83}8U7obF8C`bOxfs1F)iq#xFcI?<& zzkZA+BcT+4}45m5fVe<2fwH;RH-}_8Ts|k%RaSb3#=M$Y!w1Uc|mKa8nyY% zuH}oQ3yM`3kJE8>cHZL+0K=r}6_Nb~xIv_iL4+nRHJVc=@;3Toe?wDWzdJCqn(2gd`#_QPk?cB%OTQS@hae#{Y4qB>O} zc=OCfKX=;Pz_iW`$DVmeDoEWucbXL};a|3NVAsXo6-8lM_nqr(JZFzFqut!=u3WWw zGYh?y1j7R*@z)Qr9`;k4)v7FmcuyolW*sC3P0#oDC!10vdKAx991DHWfUWxuXi{o8 zd%eqti!fmlhTekl_4SQ+Ol7{(T1>7~sBy2bw}T4%@lzvsW|n%I&zlFGUc85uD;>IP z-HYT+-(B~}k;>uWwu*U*c1EtR7TRcn-Y*eQ;gGO>&j2v!BX2Exb9ScGFVx|z`zQaT zD?^v))C``|M7`_l|Dow{)0=BEA%=L1$JA8UU*L?==;;?2d0dp4+FE4v5dixP_Kt$J0*6o@bU>e6NWcYn~J1_3m9XLST-nh*vdqzg~clzr^d3JSJRzB!V{HW$MsmmLp z%P{wA*Ah|Caceb-3O}zcdEZlU);mbY4oAAiMQTAN^PI>&-0!CkRG9|a77lRHG9w=? z4`!;-Ksi9zbf>N+CCw6;_n zlM(`sd)K=&_}AAdT3WX!ZY)iR{wU?g>)s%GUYKl>>b_HR_Db+>9M(|RU{E~+Oy0e? zi_khuC}(wjmS9l8MWqqj;VHy@fal0gU_ShbQ{KGED&+trOJ{kjgMc>TfXbF-C%RRx z(~FXHQY}a+Os^GexG2Uq2P*qQA(k;;L!_XuFyppw!^M`)?2t1lT!>!$O1yoCbuKR( z8HskAM2!_}smw-y@3(7YAo*B@e9|1`Xe0nz!4;YVEhS``48l7SjG~}lzqzdD3z|5R z-u@TP&x9NYpNPW5DC-Tzj6-4}2}M8Nie&xH-Myau=$dwjDuccef{Do5h3&&weubvy z8*YC4)&J?)X4~jpU-PViaJ57Y=#*bUdnn`*FcagXh5 zPFXGQ{=!PwSSe=pby$*jFp3qydnw5INEpH-drlRAa6$1-TTCGC8SvfHtc97VyDp;@ z-_3QGo#Wg%pY^2BN_0noBsm{#`r_*nz299~OFG;y;Ww64o0R$8nRBkY`e{o~m-2$j zKLnM6?+!Hh3yxN@vwPIu6pbS39$x4VjRljpxY@MGh`;Doy#bQ&LOjmr=v6(!MO5B# z5}O|7GmGY%q4yoTUDYDW50is+-;wY)H}<#e9}bd4jjx#fivYmMO?v~PbPfIcZ#Ac} z*L=yV?eF*`H3R#_;1Z17%;>5BE8gXXExoa=)8)x?{vgHAMSb!CV|jI+p9!&AxJkFM zJAZLwOSsFo&UHj3)kp4h z$S|O38I&qM<^g8#%&<5w2+djc-uKwz-I)uv+?R5YPcVcbv)o*aUajWgMnQGZ*J0Ke zT6qBbZ)>2K!WP-(OFMy{(+3x<}u!757Z8Fb|81 ztS?CkO)`z=0|(3wwI_HiH?b24wWtr#R8$-tfK=N3@r13+9-jHA7GRnZ*y&#}UW}Pj z5)W15=+yT1V)ynH&&~?He7}X&oS};{vzp}5G(5^@c;ZFT^%E|yoWKf;2200We3StF zvHM@ah>t}xP z;Z;W@dWsa-TcK|Qm1yGKHa4cF4)*OQSEVV@n{Mad9Fwwi@80NbKAj;Eu-JKXYZJ`s zf0ae`2fTO{QeUr^Jy2>iW&Zp*{HrnBBJ?8fX_27f&UGV@^eJR&L4P1=d2}sq;@-zPLy9Q=z#IHbOZsA2CZoR^eZ~IcS_Cw9G+WnB(`DbU9jR4HT4NDzeI83Iy zRc%ifX^M_Qivg6>SnP{}{^)*q(06?}q|BI$V!)yF${Gi+ubnmUzOg9?D_&{1RPRk% zw9$DP+2Bh&?00Iw6FGZ&dvuk~JaQdN|UC z@giFCa=;uQr(N7{FuiY%-Z&!joN6y!sb*UUP<>+WZ0&B|^>+mnH>KMyb+Au}dj3l2 z`o2q%5^tmCj7rIfd2-qbX%oWj5xhy%CGfTaTR~HMto{ZNd~cVovcpvyeydLMgeC2{Y137l(`xilQyIov@AJT{QPjBp;c5S`L zec9uKPob!vpaUSZs77*>ij7 zd>gxmTwF`JxN{=Zq8Us4{IKf`PUzhjeJ^#~QGwGQX+bNRLzHok>|S81OTSx!u*e`VpMZ5Ag|N#2zh*%dt)*X2NUOcazw*^q_lSj3 zlf2}Ze%kUBPbGkR2h$cOE4;UGFG10V|^h?YJKj{ zhn>3CFsVJZqVC^s%ipYhWTf9Q90Z3AA#LJ2pPBb2j70IkF%{53v#<+c6t!IIsDnFz zMuE_Evj3xL7+@5j@ilT69y`As(VQ*CH}BD@Wnr2cAnHC4rL!mHL7^dbgmJA`L_L1%Nsc?kdXBZ zvC6W~Kavf&eY@BxQoZ2r2$YRf48uh*2h(0?RK;{I8Bv8cEr=BUdj_(4e#9ceop_y5 zX^ytwB;Pshoa38ymF*Jum>X9Kq7b@wFFK$!NKEvr8>vk8LUnccWmGS{WPgp4Tw8u@ zBM1?b;qn*&tc-^D@Nexny&~47nVRi*oFFWNmKgf+N*R|c&)S061f+>809sA^z#Hj4 zwfHhL2}AS@(W^K-abC$_QpWoIFomMZ5m4Dvwa@H`go+nf(|h;$1E3_rwWzEv>YHF1 zF9W1R9-Wg=?ArhP`yu|AcOQrmNw*H8BabALuzN2B$#0=v9Gdjt74QXMGT{hWAz&$6 z4(b`w_GPZQkgzP|7Y{N6wtcHYEif)fG|A0Mmnj>bSHn=Q-P_RP}) z>Pu6Y!9_C8$cj!AEfZ_r=eujGLwj{BJK7 z6`E)7WC8eo#J+!A&VMZXrfp&C?~lhqI(r}=30TMJU%y)x23q`ko#&UQC0nK|VcBm1 z;gbP1kaOs2mjYcp zT5jT{wSqune!+GoW?O+Fg`YjkqXV#ys0?67-r!IfSWiLz3ixiRz>bMV2vJOp+(M#X zVeEWnk!mXrHFpR@k<~}w8Sa(f(4j9Z%JH3N{l30dMOsNJX#ibK&3Xn2MG-qrR5U~>}SF0eDjVyUFB2*Z!TFcj8a3;3I; zn8=Ww_YLkx)@6J+Y}oyhP%ylH&~ldM7D9P}MC#3NJDeEVElK+wd6^KO8o=zsFg25K zWZg68Dhk5E!XcMkNQ1T_+Q1V~CNjG)>Vu;KX~BmhyGH`YSpZvg>LkJ9T$Fd@WAKQ^ zlIIv0ku!&0l)~NNixU5NZ9`Fy4M_7hjn{#n_o?c^e`&}fQ&L@77SyM39`IJDTJc+z8P7U=qU1Y$B{M$L(KE! z%F2Mo1B7IY77t!k(^8T?{6P-GFfO%0omFQGknsK8DwmI>bqH{Erl*>`oSY4W8tE?K ztgPy@0F8ZWEmy;858ZCpNrSH*k1S4#IGQ)X+n}WiCf~dk(W?N;vBW z5cVqpp#ea>RejI-$jIji`+KnFG5`H(QN!}HN-xnuEKiAHaxOAa(IcBzIG8pxzlb z{@QE%7?}M}M4MucYt03ao=~4CqW^r z+g*APw|fo!2UYJ5XRz3FXUl%7uV>ce3G=cD_%NkKCzGb+F~5?;d^C)~;Oa0~yvQtO zaBzzCwmk$eP=DeCW>vwHB4$2mKMp|C+5)nrgm-?Sa;r{D=Sb8Gp47SPL<1 z?q6LL)|cx~JYYEFDj*PY^zTd#>sQ7W{?!4YpNtFrzsvah#QyJTuoemDf&72-G};D+ Z-Tb=L#MkK`L;wcEK!4TZ6kVJB{|{cHab^Gj literal 0 HcmV?d00001 From a204fefe16814d9ef5fcad4071daf79207d5dc36 Mon Sep 17 00:00:00 2001 From: fengjiayi Date: Wed, 18 Oct 2017 14:45:32 -0700 Subject: [PATCH 23/76] Fix several bugs in compile time backward and Protobuf desc (#4894) * Implement FC layer with helper * Update LayerHelper * Add debug string for Python ProtoBuf and Rename `Sync` to `Flush` * Add check of ProtoBuf initialization * Layer wrapper for FC * Fix unittest * Fix CI * Add code generator * AttributeChecker Better error log and speicalize bool Since lots of types can be cast to bool * Complete mlp, fit_a_line * Implementation of simple conv_2d layer * Fix bugs * Correct implement BlockDesc destructor * Fix bugs * Fix unit test error * Follow comments --- paddle/framework/backward.cc | 14 ++++++-------- paddle/framework/block_desc.cc | 11 ++++++++++- paddle/framework/block_desc.h | 10 +++++++--- paddle/pybind/protobuf.cc | 4 ++-- python/paddle/v2/framework/framework.py | 5 ++++- python/paddle/v2/framework/tests/test_layers.py | 13 ++++++++++++- .../v2/framework/tests/test_protobuf_descs.py | 4 +++- 7 files changed, 44 insertions(+), 17 deletions(-) diff --git a/paddle/framework/backward.cc b/paddle/framework/backward.cc index ac80879c54..fb552fe344 100644 --- a/paddle/framework/backward.cc +++ b/paddle/framework/backward.cc @@ -309,8 +309,7 @@ static void CreateGradVarInBlock( } std::vector> MakeOpGrad( - const std::unique_ptr& op_desc, - std::unordered_set* no_grad_vars, + const OpDescBind* op_desc, std::unordered_set* no_grad_vars, std::unordered_map* grad_to_var) { std::vector> grad_op_descs; // All input gradients of forwarding operator do not need to calculate. @@ -357,7 +356,7 @@ std::vector> MakeBlockBackward( std::unordered_set* no_grad_vars, std::unordered_map* grad_to_var) { BlockDescBind* cur_block = program_desc.Block(block_idx); - std::deque>& op_descs = cur_block->ops_; + std::vector op_descs = cur_block->AllOps(); std::unordered_map> dup_out_ops; size_t grad_desc_idx = 0; std::vector> backward_descs; @@ -375,7 +374,7 @@ std::vector> MakeBlockBackward( program_desc, step_block_idx, no_grad_vars, grad_to_var); BlockDescBind* backward_block = program_desc.AppendBlock(*cur_block); for (auto& ptr : backward_block_op_descs) { - backward_block->ops_.push_back(std::move(ptr)); + backward_block->AppendAllocatedOp(std::move(ptr)); } op_grads[0]->SetBlockAttr("step_block", *backward_block); } @@ -432,7 +431,6 @@ ParamGradInfoMap AppendBackward( const int root_block_idx = 0; auto root_block = program_desc.Block(root_block_idx); - auto& all_ops = root_block->ops_; // insert fill one op for target // TODO(qiao) add some check to the target. @@ -447,8 +445,8 @@ ParamGradInfoMap AppendBackward( {{"shape", target_shape}, {"value", static_cast(1.0)}, {"data_type", framework::DataType::FP32}})); - all_ops.push_back(std::move(fill_one_op)); - size_t forward_op_num = all_ops.size(); + root_block->AppendAllocatedOp(std::move(fill_one_op)); + size_t forward_op_num = root_block->OpSize(); size_t forward_block_num = program_desc.Size(); // Insert backward operators @@ -457,7 +455,7 @@ ParamGradInfoMap AppendBackward( &no_grad_var_names, &grad_to_var); for (auto& ptr : backward_op_descs) { - all_ops.push_back(std::move(ptr)); + root_block->AppendAllocatedOp(std::move(ptr)); } // Create Variable diff --git a/paddle/framework/block_desc.cc b/paddle/framework/block_desc.cc index ba970254e5..92ac302e46 100644 --- a/paddle/framework/block_desc.cc +++ b/paddle/framework/block_desc.cc @@ -19,11 +19,11 @@ namespace paddle { namespace framework { VarDescBind *BlockDescBind::Var(const std::string &name) { - need_update_ = true; auto it = vars_.find(name); if (it != vars_.end()) { return it->second.get(); } + need_update_ = true; auto *var = new VarDescBind(name); vars_[name].reset(var); return var; @@ -55,6 +55,11 @@ OpDescBind *BlockDescBind::AppendOp() { return ops_.back().get(); } +void BlockDescBind::AppendAllocatedOp(std::unique_ptr &&op_desc) { + need_update_ = true; + ops_.emplace_back(std::move(op_desc)); +} + OpDescBind *BlockDescBind::PrependOp() { need_update_ = true; ops_.emplace_front(new OpDescBind()); @@ -70,6 +75,10 @@ std::vector BlockDescBind::AllOps() const { } void BlockDescBind::Flush() { + for (auto &op_desc : ops_) { + op_desc->Flush(); + } + if (need_update_) { auto &op_field = *this->desc_->mutable_ops(); this->ClearPBOps(); diff --git a/paddle/framework/block_desc.h b/paddle/framework/block_desc.h index dd7b1228be..5e1f10c1ae 100644 --- a/paddle/framework/block_desc.h +++ b/paddle/framework/block_desc.h @@ -57,10 +57,16 @@ class BlockDescBind { OpDescBind *AppendOp(); + void AppendAllocatedOp(std::unique_ptr &&op_desc); + OpDescBind *PrependOp(); std::vector AllOps() const; + size_t OpSize() const { return ops_.size(); } + + OpDescBind *Op(int idx) { return ops_.at(idx).get(); } + void Flush(); BlockDesc *Proto(); @@ -69,9 +75,7 @@ class BlockDescBind { void ClearPBOps(); void ClearPBVars(); - // FIXME(yuyang18): backward will access private data of BlockDesc. - // Mark it public temporary. We can fix it later. - public: + private: ProgramDescBind *prog_; // not_own BlockDesc *desc_; // not_own bool need_update_; diff --git a/paddle/pybind/protobuf.cc b/paddle/pybind/protobuf.cc index fbdd673295..d9647717d2 100644 --- a/paddle/pybind/protobuf.cc +++ b/paddle/pybind/protobuf.cc @@ -162,8 +162,8 @@ void BindBlockDesc(py::module &m) { py::return_value_policy::reference) .def("all_vars", &BlockDescBind::AllVars, py::return_value_policy::reference) - .def("all_ops", &BlockDescBind::AllOps, - py::return_value_policy::reference) + .def("op_size", &BlockDescBind::OpSize) + .def("op", &BlockDescBind::Op, py::return_value_policy::reference) .def("serialize_to_string", [](BlockDescBind &block_desc) -> py::bytes { const BlockDesc *desc = block_desc.Proto(); PADDLE_ENFORCE(desc->IsInitialized(), diff --git a/python/paddle/v2/framework/framework.py b/python/paddle/v2/framework/framework.py index 93e2218eab..5a8ded46ea 100644 --- a/python/paddle/v2/framework/framework.py +++ b/python/paddle/v2/framework/framework.py @@ -344,7 +344,10 @@ class Block(object): self.create_var(name=var.name(), desc=var, type=var.type()) # sync operators from cpp - ops_in_cpp = self.desc.all_ops() + ops_in_cpp = [] + for op_idx in range(0, self.desc.op_size()): + ops_in_cpp.append(self.desc.op(op_idx)) + first_op_in_python = self.ops[0].desc last_op_in_python = self.ops[len(self.ops) - 1].desc start_index = None diff --git a/python/paddle/v2/framework/tests/test_layers.py b/python/paddle/v2/framework/tests/test_layers.py index 2ffadf7371..2d8c2e5518 100644 --- a/python/paddle/v2/framework/tests/test_layers.py +++ b/python/paddle/v2/framework/tests/test_layers.py @@ -17,6 +17,7 @@ class TestBook(unittest.TestCase): avg_cost = mean(x=cost, program=program) self.assertIsNotNone(avg_cost) + program.append_backward(avg_cost, set()) print str(program) def test_recognize_digits_mlp(self): @@ -34,7 +35,17 @@ class TestBook(unittest.TestCase): cost = cross_entropy(input=predict, label=label, program=program) avg_cost = mean(x=cost, program=program) self.assertIsNotNone(avg_cost) - print str(program) + # print str(program) + + def test_simple_conv2d(self): + pd = core.ProgramDesc.__create_program_desc__() + program = Program(desc=pd) + images = data_layer( + name='pixel', shape=[3, 48, 48], data_type='int32', program=program) + conv2d_layer( + input=images, num_filters=3, filter_size=[4, 4], program=program) + + # print str(program) def test_simple_conv2d(self): pd = core.ProgramDesc.__create_program_desc__() diff --git a/python/paddle/v2/framework/tests/test_protobuf_descs.py b/python/paddle/v2/framework/tests/test_protobuf_descs.py index 6ed8edf91c..2fd3d5d165 100644 --- a/python/paddle/v2/framework/tests/test_protobuf_descs.py +++ b/python/paddle/v2/framework/tests/test_protobuf_descs.py @@ -133,7 +133,9 @@ class TestBlockDesc(unittest.TestCase): op1 = block.append_op() op2 = block.append_op() op0 = block.prepend_op() - all_ops = block.all_ops() + all_ops = [] + for idx in xrange(0, block.op_size()): + all_ops.append(block.op(idx)) self.assertEqual(all_ops, [op0, op1, op2]) From c10b8e808fc88d96ce0b4f864014bd461098de87 Mon Sep 17 00:00:00 2001 From: kavyasrinet Date: Wed, 18 Oct 2017 16:21:16 -0700 Subject: [PATCH 24/76] Adding Proximal Gradient Descent (#4848) * Adding Proximal Gradient Descent * Fixing review comments --- paddle/operators/proximal_gd_op.cc | 93 +++++++++++++++++++ paddle/operators/proximal_gd_op.cu | 19 ++++ paddle/operators/proximal_gd_op.h | 64 +++++++++++++ .../v2/framework/tests/test_proximal_gd_op.py | 33 +++++++ 4 files changed, 209 insertions(+) create mode 100644 paddle/operators/proximal_gd_op.cc create mode 100644 paddle/operators/proximal_gd_op.cu create mode 100644 paddle/operators/proximal_gd_op.h create mode 100644 python/paddle/v2/framework/tests/test_proximal_gd_op.py diff --git a/paddle/operators/proximal_gd_op.cc b/paddle/operators/proximal_gd_op.cc new file mode 100644 index 0000000000..e4b014b9f5 --- /dev/null +++ b/paddle/operators/proximal_gd_op.cc @@ -0,0 +1,93 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. */ + +#include "paddle/operators/proximal_gd_op.h" + +namespace paddle { +namespace operators { + +class ProximalGDOp : public framework::OperatorWithKernel { + public: + using framework::OperatorWithKernel::OperatorWithKernel; + + protected: + void InferShape(framework::InferShapeContext *ctx) const override { + PADDLE_ENFORCE(ctx->HasInput("Param"), + "Input(Param) of ProximalGDOp should not be null."); + PADDLE_ENFORCE(ctx->HasInput("Grad"), + "Input(Grad) of ProximalGDOp should not be null."); + PADDLE_ENFORCE(ctx->HasInput("LearningRate"), + "Input(LearningRate) of ProximalGDOp should not be null."); + + PADDLE_ENFORCE(ctx->HasOutput("ParamOut"), + "Output(ParamOut) of ProximalGDOp should not be null."); + + auto param_dim = ctx->GetInputDim("Param"); + PADDLE_ENFORCE_EQ(param_dim, ctx->GetInputDim("Grad"), + "Two input of ProximalGD Op's dimension must be same."); + + auto lr_dim = ctx->GetInputDim("LearningRate"); + PADDLE_ENFORCE_EQ(framework::product(lr_dim), 1, + "Learning Rate should be a scalar."); + + ctx->SetOutputDim("ParamOut", param_dim); + } +}; + +class ProximalGDOpMaker : public framework::OpProtoAndCheckerMaker { + public: + ProximalGDOpMaker(framework::OpProto *proto, + framework::OpAttrChecker *op_checker) + : OpProtoAndCheckerMaker(proto, op_checker) { + AddInput("Param", + "(Tensor, default Tensor) " + "Input parameter value that has to be updated."); + AddInput("Grad", + "(Tensor, default Tensor) " + "Input gradient of the parameter."); + AddInput("LearningRate", + "(Tensor, default Tensor) " + "The learning rate should be a tensor of size 1."); + + AddOutput("ParamOut", "(Tensor) Output updated parameter value."); + + AddAttr("l1", + "(float, default 0.0) " + "L1 regularization strength.") + .SetDefault(0.0f); + AddAttr("l2", + "(float, default 0.0)" + "L2 regularization strength.") + .SetDefault(0.0f); + AddComment(R"DOC( + +Optimizer that implements the proximal gradient descent algorithm. + +prox_param = param - learning_rate * grad +param = sign(prox_param) / (1 + learning_rate * l2) * + max { |prox_param| - learning_rate * l1 , 0 } + +The paper that proposed Proximal Gradient Descent: +(http://papers.nips.cc/paper/3793-efficient-learning-using-forward-backward-splitting.pdf) +)DOC"); + } +}; +} // namespace operators +} // namespace paddle + +namespace ops = paddle::operators; +REGISTER_OP_WITHOUT_GRADIENT(proximal_gd, ops::ProximalGDOp, + ops::ProximalGDOpMaker); +REGISTER_OP_CPU_KERNEL( + proximal_gd, ops::ProximalGDOpKernel); diff --git a/paddle/operators/proximal_gd_op.cu b/paddle/operators/proximal_gd_op.cu new file mode 100644 index 0000000000..26f4ebaa0f --- /dev/null +++ b/paddle/operators/proximal_gd_op.cu @@ -0,0 +1,19 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + +Licensed under the Apache License, Version 2.0 (the "License"); +You may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software distributed +under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR +CONDITIONS OF ANY KIND, either express or implied. See the License for the +specific language governing permissions and limitations under the License. */ + +#define EIGEN_USE_GPU +#include "paddle/operators/proximal_gd_op.h" + +namespace ops = paddle::operators; +REGISTER_OP_GPU_KERNEL( + proximal_gd, ops::ProximalGDOpKernel); diff --git a/paddle/operators/proximal_gd_op.h b/paddle/operators/proximal_gd_op.h new file mode 100644 index 0000000000..bebda02041 --- /dev/null +++ b/paddle/operators/proximal_gd_op.h @@ -0,0 +1,64 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. */ + +#pragma once +#include "paddle/framework/eigen.h" +#include "paddle/framework/op_registry.h" + +namespace paddle { +namespace operators { + +using Tensor = framework::Tensor; +template +using EigenVector = framework::EigenVector; + +template +class ProximalGDOpKernel : public framework::OpKernel { + public: + void Compute(const framework::ExecutionContext& ctx) const override { + auto* param_out = ctx.Output("ParamOut"); + + param_out->mutable_data(ctx.GetPlace()); + + auto grad = ctx.Input("Grad"); + + auto l1 = static_cast(ctx.Attr("l1")); + auto l2 = static_cast(ctx.Attr("l2")); + + auto p = EigenVector::Flatten(*ctx.Input("Param")); + auto g = EigenVector::Flatten(*grad); + auto lr = EigenVector::Flatten(*ctx.Input("LearningRate")); + + auto p_out = EigenVector::Flatten(*param_out); + auto place = ctx.GetEigenDevice(); + + Eigen::DSizes grad_dsize(grad->numel()); + + auto prox_param = p - lr.broadcast(grad_dsize) * g; + if (l1 > 0) { + p_out.device(place) = + prox_param.sign() * + (((prox_param.abs() - (lr * l1).broadcast(grad_dsize)) + .cwiseMax(T(0.0))) / + (1.0 + (lr * l2).broadcast(grad_dsize))); + } else { + p_out.device(place) = + prox_param / (1.0 + (lr * l2).broadcast(grad_dsize)); + } + } +}; + +} // namespace operators +} // namespace paddle diff --git a/python/paddle/v2/framework/tests/test_proximal_gd_op.py b/python/paddle/v2/framework/tests/test_proximal_gd_op.py new file mode 100644 index 0000000000..9ca79ce6b3 --- /dev/null +++ b/python/paddle/v2/framework/tests/test_proximal_gd_op.py @@ -0,0 +1,33 @@ +import unittest +import numpy as np +from op_test import OpTest + + +class TestProximalGDOp(OpTest): + def setUp(self): + self.op_type = "proximal_gd" + w = np.random.random((102, 105)).astype("float32") + g = np.random.random((102, 105)).astype("float32") + lr = np.array([0.1]).astype("float32") + l1 = 0.1 + l2 = 0.2 + + self.inputs = {'Param': w, 'Grad': g, 'LearningRate': lr} + self.attrs = {'l1': l1, 'l2': l2} + prox_param = w - lr * g + param_out = 0.0 + if l1 > 0.0: + x = np.abs(prox_param) - lr * l1 + x[x < 0] = 0 + param_out = np.sign(prox_param) * (x / (1.0 + lr * l2)) + else: + param_out = prox_param / (1.0 + lr * l2) + + self.outputs = {'ParamOut': param_out} + + def test_check_output(self): + self.check_output() + + +if __name__ == "__main__": + unittest.main() From c93596d35b621959d28f16ffba7689a79bd9b068 Mon Sep 17 00:00:00 2001 From: fengjiayi Date: Wed, 18 Oct 2017 17:21:32 -0700 Subject: [PATCH 25/76] unify layer names (#4913) --- python/paddle/v2/framework/layers.py | 50 +++++++++---------- .../paddle/v2/framework/tests/test_layers.py | 38 +++++++------- 2 files changed, 45 insertions(+), 43 deletions(-) diff --git a/python/paddle/v2/framework/layers.py b/python/paddle/v2/framework/layers.py index 1821da197e..c7397716c4 100644 --- a/python/paddle/v2/framework/layers.py +++ b/python/paddle/v2/framework/layers.py @@ -3,17 +3,17 @@ import paddle.v2.framework.core as core from paddle.v2.framework.framework import OpProtoHolder, Variable import re -__all__ = ['fc_layer', 'data_layer', 'cross_entropy', 'conv2d_layer'] +__all__ = ['fc', 'data', 'cross_entropy', 'conv2d'] -def fc_layer(input, - size, - param_attr=None, - bias_attr=True, - name=None, - act=None, - num_flatten_dims=1, - program=None): +def fc(input, + size, + param_attr=None, + bias_attr=True, + name=None, + act=None, + num_flatten_dims=1, + program=None): # create helper helper = LayerHelper('fc', **locals()) @@ -51,11 +51,11 @@ def fc_layer(input, return helper.append_activation(pre_activation) -def data_layer(name, - shape, - data_type='float32', - type=core.VarDesc.VarType.LOD_TENSOR, - program=None): +def data(name, + shape, + data_type='float32', + type=core.VarDesc.VarType.LOD_TENSOR, + program=None): helper = LayerHelper('data', **locals()) shape = [-1] + shape # append batch size as -1 return helper.create_global_variable( @@ -145,17 +145,17 @@ def square_error_cost(input, label, **kwargs): return square_out -def conv2d_layer(input, - num_filters, - name=None, - filter_size=[1, 1], - act=None, - groups=None, - stride=[1, 1], - padding=None, - bias_attr=None, - param_attr=None, - program=None): +def conv2d(input, + num_filters, + name=None, + filter_size=[1, 1], + act=None, + groups=None, + stride=[1, 1], + padding=None, + bias_attr=None, + param_attr=None, + program=None): helper = LayerHelper('conv2d', **locals()) dtype = helper.input_dtype() diff --git a/python/paddle/v2/framework/tests/test_layers.py b/python/paddle/v2/framework/tests/test_layers.py index 2d8c2e5518..dbbb653538 100644 --- a/python/paddle/v2/framework/tests/test_layers.py +++ b/python/paddle/v2/framework/tests/test_layers.py @@ -1,4 +1,4 @@ -from paddle.v2.framework.layers import fc_layer, data_layer, cross_entropy, mean, square_error_cost, conv2d_layer +import paddle.v2.framework.layers as layers from paddle.v2.framework.framework import Program, g_program import paddle.v2.framework.core as core import unittest @@ -7,15 +7,16 @@ import unittest class TestBook(unittest.TestCase): def test_fit_a_line(self): program = Program() - x = data_layer( + x = layers.data( name='x', shape=[13], data_type='float32', program=program) - y_predict = fc_layer(input=x, size=1, act=None, program=program) + y_predict = layers.fc(input=x, size=1, act=None, program=program) - y = data_layer( + y = layers.data( name='y', shape=[1], data_type='float32', program=program) - cost = square_error_cost(input=y_predict, label=y, program=program) + cost = layers.square_error_cost( + input=y_predict, label=y, program=program) - avg_cost = mean(x=cost, program=program) + avg_cost = layers.mean(x=cost, program=program) self.assertIsNotNone(avg_cost) program.append_backward(avg_cost, set()) print str(program) @@ -24,16 +25,18 @@ class TestBook(unittest.TestCase): program = Program() # Change g_program, so the rest layers use `g_program` - images = data_layer( + images = layers.data( name='pixel', shape=[784], data_type='float32', program=program) - label = data_layer( + label = layers.data( name='label', shape=[1], data_type='int32', program=program) - hidden1 = fc_layer(input=images, size=128, act='relu', program=program) - hidden2 = fc_layer(input=hidden1, size=64, act='relu', program=program) - predict = fc_layer( - input=hidden2, size=10, act='softmax', program=program) - cost = cross_entropy(input=predict, label=label, program=program) - avg_cost = mean(x=cost, program=program) + hidden1 = layers.fc(input=images, size=128, act='relu', program=program) + hidden2 = layers.fc(input=hidden1, size=64, act='relu', program=program) + predict = layers.fc(input=hidden2, + size=10, + act='softmax', + program=program) + cost = layers.cross_entropy(input=predict, label=label, program=program) + avg_cost = layers.mean(x=cost, program=program) self.assertIsNotNone(avg_cost) # print str(program) @@ -48,11 +51,10 @@ class TestBook(unittest.TestCase): # print str(program) def test_simple_conv2d(self): - pd = core.ProgramDesc.__create_program_desc__() - program = Program(desc=pd) - images = data_layer( + program = Program() + images = layers.data( name='pixel', shape=[3, 48, 48], data_type='int32', program=program) - conv2d_layer( + layers.conv2d( input=images, num_filters=3, filter_size=[4, 4], program=program) print str(program) From c5b411c51533c93459661673e797663ed681d8de Mon Sep 17 00:00:00 2001 From: Yang Yang Date: Thu, 19 Oct 2017 00:46:22 +0000 Subject: [PATCH 26/76] make compatible to new programDescBind --- paddle/framework/prune_test.cc | 23 +++++------------------ 1 file changed, 5 insertions(+), 18 deletions(-) diff --git a/paddle/framework/prune_test.cc b/paddle/framework/prune_test.cc index a8faf1891e..3ab4b43d92 100644 --- a/paddle/framework/prune_test.cc +++ b/paddle/framework/prune_test.cc @@ -50,17 +50,8 @@ void AddOp(const std::string &type, const f::VariableNameMap &inputs, op->SetAttrMap(attrs); } -f::ProgramDesc *GetNewProgramDesc() { - auto *program_desc = new f::ProgramDesc(); - auto *root_block = program_desc->add_blocks(); - root_block->set_idx(0); - root_block->set_parent_idx(-1); - return program_desc; -} - TEST(Prune, one_operator) { - f::ProgramDesc *program_desc = GetNewProgramDesc(); - f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::ProgramDescBind program; f::BlockDescBind *block = program.Block(0); AddOp("one_one", {{"input", {"a"}}}, {{"output", {"b"}}}, {}, block); @@ -77,8 +68,7 @@ TEST(Prune, one_operator) { } TEST(Prune, forward) { - f::ProgramDesc *program_desc = GetNewProgramDesc(); - f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::ProgramDescBind program; f::BlockDescBind *block = program.Block(0); AddOp("one_one", {{"input", {"a"}}}, {{"output", {"b"}}}, {}, block); @@ -97,8 +87,7 @@ TEST(Prune, forward) { } TEST(Prune, multi_input_op) { - f::ProgramDesc *program_desc = GetNewProgramDesc(); - f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::ProgramDescBind program; f::BlockDescBind *block = program.Block(0); AddOp("one_one", {{"input", {"a0"}}}, {{"output", {"b0"}}}, {}, block); @@ -116,8 +105,7 @@ TEST(Prune, multi_input_op) { } TEST(Prune, multi_output_op) { - f::ProgramDesc *program_desc = GetNewProgramDesc(); - f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::ProgramDescBind program; f::BlockDescBind *block = program.Block(0); AddOp("one_two", {{"input", {"a"}}}, {{"output", {"b", "c"}}}, {}, block); @@ -133,8 +121,7 @@ TEST(Prune, multi_output_op) { } TEST(Prune, multi_target) { - f::ProgramDesc *program_desc = GetNewProgramDesc(); - f::ProgramDescBind &program = f::ProgramDescBind::Instance(program_desc); + f::ProgramDescBind program; f::BlockDescBind *block = program.Block(0); AddOp("one_two", {{"input", {"a"}}}, {{"output", {"b", "c"}}}, {}, block); From f6e1d959d2f54a8baa183d76c8134f27c60edcba Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Wed, 18 Oct 2017 18:22:31 -0700 Subject: [PATCH 27/76] Expose VarDesc::persistable to Python (#4911) --- paddle/framework/var_desc.h | 4 ++++ paddle/pybind/protobuf.cc | 23 +++++++++++++---------- python/paddle/v2/framework/framework.py | 20 +++++++++++++++++++- 3 files changed, 36 insertions(+), 11 deletions(-) diff --git a/paddle/framework/var_desc.h b/paddle/framework/var_desc.h index 688a46f839..af4c26ca0a 100644 --- a/paddle/framework/var_desc.h +++ b/paddle/framework/var_desc.h @@ -79,6 +79,10 @@ class VarDescBind { void SetType(VarDesc::VarType type) { desc_.set_type(type); } + bool Persistable() const { return desc_.persistable(); } + + void SetPersistable(bool persistable) { desc_.set_persistable(persistable); } + private: const TensorDesc &tensor_desc() const; TensorDesc *mutable_tensor_desc(); diff --git a/paddle/pybind/protobuf.cc b/paddle/pybind/protobuf.cc index d9647717d2..a4fb9b7c07 100644 --- a/paddle/pybind/protobuf.cc +++ b/paddle/pybind/protobuf.cc @@ -202,16 +202,19 @@ void BindVarDsec(py::module &m) { .def("set_lod_level", &VarDescBind::SetLoDLevel) .def("type", &VarDescBind::GetType) .def("set_type", &VarDescBind::SetType) - .def("serialize_to_string", [](VarDescBind &var_desc) -> py::bytes { - const VarDesc *desc = var_desc.Proto(); - PADDLE_ENFORCE(desc->IsInitialized(), - "VarDesc has not been initialized."); - std::string res; - PADDLE_ENFORCE( - desc->SerializeToString(&res), - "Serialize VarDesc Error. This could be a bug of Paddle."); - return res; - }); + .def("serialize_to_string", + [](VarDescBind &var_desc) -> py::bytes { + const VarDesc *desc = var_desc.Proto(); + PADDLE_ENFORCE(desc->IsInitialized(), + "VarDesc has not been initialized."); + std::string res; + PADDLE_ENFORCE( + desc->SerializeToString(&res), + "Serialize VarDesc Error. This could be a bug of Paddle."); + return res; + }) + .def("persistable", &VarDescBind::Persistable) + .def("set_persistable", &VarDescBind::SetPersistable); py::enum_(var_desc, "VarType", "") .value("LOD_TENSOR", VarDesc::LOD_TENSOR) diff --git a/python/paddle/v2/framework/framework.py b/python/paddle/v2/framework/framework.py index 5a8ded46ea..8c63ca9644 100644 --- a/python/paddle/v2/framework/framework.py +++ b/python/paddle/v2/framework/framework.py @@ -15,6 +15,7 @@ class Variable(object): shape=None, dtype=None, lod_level=None, + persistable=False, **kwargs): self.block = block @@ -70,6 +71,17 @@ class Variable(object): "lod_level is {2}. They are not " "matched".format(self.name, self.lod_level, lod_level)) + if persistable is not None: + if is_new_var: + self.desc.set_persistable(persistable) + else: + if persistable != self.persistable: + raise ValueError( + "Variable {0} has been created before." + "The previous persistable is {1}; the new " + "persistable is {2}. They are not matched".format( + self.name, self.persistable, persistable)) + self.block.vars[name] = self self.op = None @@ -80,6 +92,10 @@ class Variable(object): __repr__ = __str__ + @property + def persistable(self): + return self.desc.persistable() + @property def name(self): return self.desc.name() @@ -445,7 +461,9 @@ class Parameter(Variable): if each < 0: raise ValueError("Parameter shape should not be related with " "batch-size") - Variable.__init__(self, block, shape=shape, dtype=dtype, **kwargs) + + Variable.__init__( + self, block, persistable=True, shape=shape, dtype=dtype, **kwargs) self.trainable = kwargs.get('trainable', True) self.init_attr = kwargs.get('initialize_attr', { 'type': 'uniform_random', From e9249d16cb3078e0a1344513d752c9e314ab86f1 Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Wed, 18 Oct 2017 18:28:50 -0700 Subject: [PATCH 28/76] Add glog as dependencies of ops (#4908) * Add glog as dependencies of ops * Use VLOG to logging some information is helpful when we debug Paddle * Fix Unittests --- paddle/framework/CMakeLists.txt | 4 ++-- paddle/framework/op_registry.h | 2 ++ paddle/framework/operator.h | 4 +++- 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/paddle/framework/CMakeLists.txt b/paddle/framework/CMakeLists.txt index 4bc3fdeeea..05ae2daf6a 100644 --- a/paddle/framework/CMakeLists.txt +++ b/paddle/framework/CMakeLists.txt @@ -23,10 +23,10 @@ cc_library(proto_desc SRCS var_desc.cc op_desc.cc block_desc.cc program_desc.cc cc_library(op_proto_maker SRCS op_proto_maker.cc DEPS framework_proto attribute) cc_test(op_proto_maker_test SRCS op_proto_maker_test.cc DEPS op_proto_maker) cc_library(op_info SRCS op_info.cc DEPS attribute framework_proto) -cc_library(operator SRCS operator.cc DEPS op_info device_context tensor scope proto_desc) +cc_library(operator SRCS operator.cc DEPS op_info device_context tensor scope proto_desc glog) cc_test(operator_test SRCS operator_test.cc DEPS operator op_registry) -cc_library(op_registry SRCS op_registry.cc DEPS op_proto_maker op_info operator) +cc_library(op_registry SRCS op_registry.cc DEPS op_proto_maker op_info operator glog) cc_test(op_registry_test SRCS op_registry_test.cc DEPS op_registry) py_proto_compile(framework_py_proto SRCS framework.proto) diff --git a/paddle/framework/op_registry.h b/paddle/framework/op_registry.h index d25b4abccb..ed85c386ec 100644 --- a/paddle/framework/op_registry.h +++ b/paddle/framework/op_registry.h @@ -20,6 +20,8 @@ limitations under the License. */ #include #include #include + +#include "glog/logging.h" // For VLOG() #include "paddle/framework/attribute.h" #include "paddle/framework/details/op_registry.h" #include "paddle/framework/framework.pb.h" diff --git a/paddle/framework/operator.h b/paddle/framework/operator.h index cf15f9933a..12cd307297 100644 --- a/paddle/framework/operator.h +++ b/paddle/framework/operator.h @@ -20,12 +20,13 @@ limitations under the License. */ #include #include -#include "op_info.h" +#include "glog/logging.h" // For VLOG #include "paddle/framework/attribute.h" #include "paddle/framework/block_desc.h" #include "paddle/framework/data_type.h" #include "paddle/framework/framework.pb.h" #include "paddle/framework/lod_tensor.h" +#include "paddle/framework/op_info.h" #include "paddle/framework/scope.h" #include "paddle/framework/shape_inference.h" #include "paddle/framework/tensor.h" @@ -573,6 +574,7 @@ class OperatorWithKernel : public OperatorBase { void Run(const Scope& scope, const platform::DeviceContext& dev_ctx) const final { + VLOG(3) << "Running operator " << this->Type(); RuntimeInferShapeContext infer_shape_ctx(*this, scope); this->InferShape(&infer_shape_ctx); From 3ca3a200ab14454954ba44de3deba5caea229f51 Mon Sep 17 00:00:00 2001 From: "Yang Yang(Tony)" Date: Wed, 18 Oct 2017 19:00:53 -0700 Subject: [PATCH 29/76] Prune Design Doc (#4732) * Create prune.md * modification based on comment * remove insertion * rename id to block_id * Update prune.md * formatting --- doc/design/prune.md | 63 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100644 doc/design/prune.md diff --git a/doc/design/prune.md b/doc/design/prune.md new file mode 100644 index 0000000000..4a5cf10c79 --- /dev/null +++ b/doc/design/prune.md @@ -0,0 +1,63 @@ +# Prune + +## Motivation + +We want to support running inference, training and checkpointing in one `ProgramDesc`. We implement +`void Prune(const ProgramDesc* input, ProgramDesc* output)` function, which takes a `ProgramDesc` +and generate a pruned `ProgramDesc`. + +## Challenge + +Pruning need to support both variables and operators being evaluation targets. Consider the following +different situations. + +```python +# Case 1: run foward pass. +cost_np = session.run(target=cost) +# Case 2: run backward passing. +opts_np, _ = session.run(target=[cost, opt]) +# Case 3: run checkpointing +_ = session.run(target=checkpoint) +``` + +## Solution + +To support evaluation of operators, we add `is_target` field in the `OpDesc`. + +```c++ +message OpDesc { + required string type = 3; + repeated Var inputs = 1; + repeated Var outputs = 2; + repeated Attr attrs = 4; + optional bool is_target = 5 [ default = false ]; +}; +``` + +To support evaluation of variables, we add [fetch_op](https://github.com/PaddlePaddle/Paddle/pull/4599). +For each variable in the `target`, we insert a `fetch_op` into the `ProgramDesc` with `variable` being +`fetch_op`'s input. Then we also set `fetch_op` is a target. + +### Algorithm + +If an operator needs to be run, it must fall into one of the following cases: + +1. It is the target. +2. It is depended by some other ops, meaning its output is some other op's input. + +The first case can be checked by `op_desc.is_traget()` . The second case can be implement as + +```c++ +bool HasDependentVar(const OpDesc& op_desc, const std::set& dependent_vars) { + for (auto& var : op_desc.outputs()) { + for (auto& argu : var.arguments()) { + if (dependent_vars.count(argu) != 0) { + return true; + } + } + } + return false; +} +``` + +Then the whole algorithm can be implemented as the following [code](https://github.com/tonyyang-svail/Paddle/blob/prune_impl/paddle/framework/prune.cc). From 4018754d666bc8045177e295850d941de4b264df Mon Sep 17 00:00:00 2001 From: wanghaoshuang Date: Thu, 19 Oct 2017 10:03:47 +0800 Subject: [PATCH 30/76] fix LoDTensor::lod_element to get last element in level --- paddle/framework/lod_tensor.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/paddle/framework/lod_tensor.h b/paddle/framework/lod_tensor.h index 4db36ee766..3eab91b0d1 100644 --- a/paddle/framework/lod_tensor.h +++ b/paddle/framework/lod_tensor.h @@ -78,7 +78,7 @@ class LoDTensor : public Tensor { */ size_t lod_element(size_t level, size_t elem) const { PADDLE_ENFORCE_LT(level, NumLevels()); - PADDLE_ENFORCE_LT(elem, NumElements(level)); + PADDLE_ENFORCE_LE(elem, NumElements(level)); return (lod_)[level][elem]; } From 47f773ddb21b01e183f258dad1b1b54137b60998 Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Wed, 18 Oct 2017 19:28:23 -0700 Subject: [PATCH 31/76] Copy Constructor for ProgramDesc (#4895) * Implement FC layer with helper * Update LayerHelper * Add debug string for Python ProtoBuf and Rename `Sync` to `Flush` * Add check of ProtoBuf initialization * Layer wrapper for FC * Fix unittest * Fix CI * Add code generator * AttributeChecker Better error log and speicalize bool Since lots of types can be cast to bool * Complete mlp, fit_a_line * Implementation of simple conv_2d layer * Fix bugs * Change ProgramDesc not a global variable * Polish code style * Stash * Correct implement BlockDesc destructor * Correct implement BlockDesc destructor * Unify program as parameter name * Fix bugs * Add unittest * Fix unit test error * Remove unused functions * Add clone for Python Program * Compare OpDescBind directly --- paddle/framework/CMakeLists.txt | 1 + paddle/framework/block_desc.cc | 13 +++ paddle/framework/block_desc.h | 13 +++ paddle/framework/program_desc.cc | 9 ++ paddle/framework/program_desc.h | 4 +- paddle/framework/program_desc_test.cc | 83 +++++++++++++++++++ paddle/pybind/protobuf.cc | 4 + python/paddle/v2/framework/framework.py | 38 ++++++--- .../paddle/v2/framework/tests/test_program.py | 18 ++++ 9 files changed, 168 insertions(+), 15 deletions(-) create mode 100644 paddle/framework/program_desc_test.cc diff --git a/paddle/framework/CMakeLists.txt b/paddle/framework/CMakeLists.txt index 1a6f90c1ef..6e32a1c99b 100644 --- a/paddle/framework/CMakeLists.txt +++ b/paddle/framework/CMakeLists.txt @@ -20,6 +20,7 @@ proto_library(framework_proto SRCS framework.proto) cc_library(attribute SRCS attribute.cc DEPS framework_proto) cc_library(proto_desc SRCS var_desc.cc op_desc.cc block_desc.cc program_desc.cc DEPS attribute ddim op_info) +cc_test(program_desc_test SRCS program_desc_test.cc DEPS proto_desc) cc_library(op_proto_maker SRCS op_proto_maker.cc DEPS framework_proto attribute) cc_test(op_proto_maker_test SRCS op_proto_maker_test.cc DEPS op_proto_maker) cc_library(op_info SRCS op_info.cc DEPS attribute framework_proto) diff --git a/paddle/framework/block_desc.cc b/paddle/framework/block_desc.cc index 92ac302e46..21d4fdaf06 100644 --- a/paddle/framework/block_desc.cc +++ b/paddle/framework/block_desc.cc @@ -107,6 +107,19 @@ BlockDesc *BlockDescBind::Proto() { Flush(); return desc_; } +BlockDescBind::BlockDescBind(const BlockDescBind &other, BlockDesc *desc, + ProgramDescBind *prog) + : prog_(prog), desc_(desc) { + need_update_ = true; + for (auto &op : other.ops_) { + ops_.emplace_back(new OpDescBind(*op)); + } + + for (auto &it : other.vars_) { + auto *var = new VarDescBind(*it.second); + vars_[it.first].reset(var); + } +} void BlockDescBind::ClearPBOps() { auto ops = this->desc_->mutable_ops(); diff --git a/paddle/framework/block_desc.h b/paddle/framework/block_desc.h index 5e1f10c1ae..7d1d33f686 100644 --- a/paddle/framework/block_desc.h +++ b/paddle/framework/block_desc.h @@ -16,8 +16,10 @@ limitations under the License. */ #include #include +#include #include #include + #include "paddle/framework/op_desc.h" #include "paddle/framework/var_desc.h" #include "paddle/platform/macros.h" @@ -36,6 +38,9 @@ class BlockDescBind { BlockDescBind(ProgramDescBind *prog, BlockDesc *desc) : prog_(prog), desc_(desc), need_update_(false) {} + BlockDescBind(const BlockDescBind &other, BlockDesc *desc, + ProgramDescBind *prog); + ~BlockDescBind() { this->ClearPBVars(); this->ClearPBOps(); @@ -51,6 +56,14 @@ class BlockDescBind { bool HasVar(const std::string &var_name) const; + std::set LocalVarNames() const { + std::set var_names; + for (auto &var : vars_) { + var_names.insert(var.first); + } + return var_names; + } + std::vector AllVars() const; BlockDescBind *ParentBlock() const; diff --git a/paddle/framework/program_desc.cc b/paddle/framework/program_desc.cc index df846f115a..e2349cefe0 100644 --- a/paddle/framework/program_desc.cc +++ b/paddle/framework/program_desc.cc @@ -39,5 +39,14 @@ ProgramDescBind::ProgramDescBind() { block->set_parent_idx(-1); blocks_.emplace_back(new BlockDescBind(this, block)); } + +ProgramDescBind::ProgramDescBind(const ProgramDescBind &o) { + prog_ = o.prog_; + + for (int i = 0; i < prog_.blocks_size(); ++i) { + auto *block = prog_.mutable_blocks(i); + blocks_.emplace_back(new BlockDescBind(*o.blocks_[i], block, this)); + } +} } // namespace framework } // namespace paddle diff --git a/paddle/framework/program_desc.h b/paddle/framework/program_desc.h index 514b62654d..20cc1a2325 100644 --- a/paddle/framework/program_desc.h +++ b/paddle/framework/program_desc.h @@ -28,6 +28,8 @@ class ProgramDescBind { public: ProgramDescBind(); + ProgramDescBind(const ProgramDescBind &o); + BlockDescBind *AppendBlock(const BlockDescBind &parent); BlockDescBind *Block(size_t idx) { return blocks_[idx].get(); } @@ -40,8 +42,6 @@ class ProgramDescBind { ProgramDesc prog_; std::vector> blocks_; - - DISABLE_COPY_AND_ASSIGN(ProgramDescBind); }; } // namespace framework } // namespace paddle diff --git a/paddle/framework/program_desc_test.cc b/paddle/framework/program_desc_test.cc new file mode 100644 index 0000000000..32ee275429 --- /dev/null +++ b/paddle/framework/program_desc_test.cc @@ -0,0 +1,83 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include "paddle/framework/program_desc.h" +#include "gtest/gtest.h" +#include "paddle/framework/block_desc.h" + +namespace paddle { +namespace framework { +TEST(ProgramDesc, copy_ctor) { + ProgramDescBind program; + auto* global_block = program.Block(0); + auto* x = global_block->Var("X"); + x->SetType(VarDesc_VarType_LOD_TENSOR); + x->SetLoDLevel(0); + x->SetDataType(FP32); + x->SetShape({1000, 784}); + + auto* y = global_block->Var("Y"); + y->SetType(VarDesc_VarType_LOD_TENSOR); + y->SetLoDLevel(0); + y->SetDataType(FP32); + y->SetShape({784, 100}); + + auto* op = global_block->AppendOp(); + op->SetType("mul"); + op->SetInput("X", {x->Name()}); + op->SetInput("Y", {y->Name()}); + + auto* out = global_block->Var("Out"); + out->SetType(VarDesc_VarType_LOD_TENSOR); + op->SetOutput("Y", {out->Name()}); + + ProgramDescBind program_copy(program); + + auto* global_block_copy = program_copy.Block(0); + ASSERT_NE(global_block, global_block_copy); + + auto assert_same_var = [&](const std::string& name, VarDescBind* var_before) { + ASSERT_TRUE(global_block_copy->HasVar(name)); + auto* copy = global_block_copy->Var(name); + ASSERT_NE(copy, var_before); + ASSERT_EQ(copy->Name(), var_before->Name()); + ASSERT_EQ(copy->GetType(), var_before->GetType()); + ASSERT_EQ(copy->Shape(), var_before->Shape()); + ASSERT_EQ(copy->Proto()->SerializeAsString(), + var_before->Proto()->SerializeAsString()); + }; + + ASSERT_EQ(global_block->LocalVarNames(), global_block_copy->LocalVarNames()); + ASSERT_EQ(3, global_block_copy->LocalVarNames().size()); + assert_same_var("X", x); + assert_same_var("Y", y); + assert_same_var("Out", out); + + for (size_t i = 0; i < global_block->OpSize(); ++i) { + auto op_origin = global_block->Op(i); + auto op_copy = global_block->Op(i); + + ASSERT_EQ(op_origin->Type(), op_copy->Type()); + ASSERT_EQ(op_origin->Inputs(), op_copy->Inputs()); + ASSERT_EQ(op_origin->Outputs(), op_copy->Outputs()); + + ASSERT_EQ(op_copy->Proto()->SerializeAsString(), + op_origin->Proto()->SerializeAsString()); + } + + // Not check block's protostr are same it because the order of vars could be + // different and it is correct. +} +} // namespace framework +} // namespace paddle \ No newline at end of file diff --git a/paddle/pybind/protobuf.cc b/paddle/pybind/protobuf.cc index a4fb9b7c07..58739d888a 100644 --- a/paddle/pybind/protobuf.cc +++ b/paddle/pybind/protobuf.cc @@ -101,6 +101,10 @@ using namespace paddle::framework; // NOLINT void BindProgramDesc(py::module &m) { py::class_(m, "ProgramDesc", "") .def(py::init<>()) + .def("__init__", + [](ProgramDescBind &self, const ProgramDescBind &other) { + new (&self) ProgramDescBind(other); + }) .def("append_block", &ProgramDescBind::AppendBlock, py::return_value_policy::reference) .def("append_backward", diff --git a/python/paddle/v2/framework/framework.py b/python/paddle/v2/framework/framework.py index 8c63ca9644..9c032400a1 100644 --- a/python/paddle/v2/framework/framework.py +++ b/python/paddle/v2/framework/framework.py @@ -364,18 +364,22 @@ class Block(object): for op_idx in range(0, self.desc.op_size()): ops_in_cpp.append(self.desc.op(op_idx)) - first_op_in_python = self.ops[0].desc - last_op_in_python = self.ops[len(self.ops) - 1].desc - start_index = None - end_index = None - for index in range(len(ops_in_cpp)): - if first_op_in_python == ops_in_cpp[index]: - start_index = index - if last_op_in_python == ops_in_cpp[index]: - end_index = index - assert start_index is not None - assert end_index is not None - assert start_index <= end_index + if len(self.ops) != 0: + first_op_in_python = self.ops[0].desc + last_op_in_python = self.ops[len(self.ops) - 1].desc + start_index = None + end_index = None + for index in range(len(ops_in_cpp)): + if first_op_in_python == ops_in_cpp[index]: + start_index = index + if last_op_in_python == ops_in_cpp[index]: + end_index = index + assert start_index is not None + assert end_index is not None + assert start_index <= end_index + else: + start_index = 0 + end_index = -1 # sync ops append to the head of cpp_ops for index in range((start_index - 1 - 1), -1, -1): @@ -413,7 +417,15 @@ class Program(object): proto = framework_pb2.ProgramDesc.FromString(str(protostr)) return proto.__str__() - __repr__ = __str__ + def clone(self): + p = Program() + p.desc = core.ProgramDesc(self.desc) + p.blocks = [Block(p, i) for i in xrange(self.desc.num_blocks())] + p.sync_with_cpp() + return p + + def __repr__(self): + return str(self) def global_block(self): return self.blocks[0] diff --git a/python/paddle/v2/framework/tests/test_program.py b/python/paddle/v2/framework/tests/test_program.py index c98dc3492b..8d8dd46898 100644 --- a/python/paddle/v2/framework/tests/test_program.py +++ b/python/paddle/v2/framework/tests/test_program.py @@ -34,6 +34,24 @@ class TestProgram(unittest.TestCase): self.assertEqual(1, b.idx) self.assertEqual(0, b.parent_idx) + def test_program_clone(self): + prog = Program() + + x = prog.global_block().create_var( + name='X', shape=[1000, 784], dtype='float32') + + y = prog.global_block().create_var( + name='Y', shape=[784, 100], dtype='float32') + out = prog.global_block().create_var(name='Out', dtype='float32') + prog.global_block().append_op( + type="mul", inputs={'X': [x], + 'Y': [y]}, outputs={'Out': [out]}) + + # FIXME(yuyang18): We manual compare the output string, since the order + # of variable could be changed. + print prog + print prog.clone() + def test_append_backward(self): prog = Program.instance() block = prog.global_block() From edb6aba69855b64c28f123f024a0d82422becb32 Mon Sep 17 00:00:00 2001 From: wanghaoshuang Date: Thu, 19 Oct 2017 11:07:35 +0800 Subject: [PATCH 32/76] make lod_element return std::pair --- paddle/framework/lod_tensor.h | 8 ++++---- paddle/framework/lod_tensor_test.cu | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/paddle/framework/lod_tensor.h b/paddle/framework/lod_tensor.h index 3eab91b0d1..3d893baa35 100644 --- a/paddle/framework/lod_tensor.h +++ b/paddle/framework/lod_tensor.h @@ -74,12 +74,12 @@ class LoDTensor : public Tensor { LoD lod() const { return lod_; } /* - * Get a element from LoD. + * Get the start offset and end offset of an element from LoD. */ - size_t lod_element(size_t level, size_t elem) const { + std::pair lod_element(size_t level, size_t elem) const { PADDLE_ENFORCE_LT(level, NumLevels()); - PADDLE_ENFORCE_LE(elem, NumElements(level)); - return (lod_)[level][elem]; + PADDLE_ENFORCE_LT(elem, NumElements(level)); + return std::make_pair((lod_)[level][elem], (lod_)[level][elem + 1]); } /* diff --git a/paddle/framework/lod_tensor_test.cu b/paddle/framework/lod_tensor_test.cu index 647d07536d..25041024cb 100644 --- a/paddle/framework/lod_tensor_test.cu +++ b/paddle/framework/lod_tensor_test.cu @@ -36,8 +36,8 @@ TEST(LoDTensor, LoDInGPU) { lod_tensor.mutable_data(place); lod_tensor.set_lod(src_lod); - CHECK_EQ(lod_tensor.lod_element(0, 2), 4UL); - CHECK_EQ(lod_tensor.lod_element(0, 4), 8UL); + CHECK_EQ(lod_tensor.lod_element(0, 2).first, 4UL); + CHECK_EQ(lod_tensor.lod_element(0, 4).first, 8UL); auto lod = lod_tensor.lod(); From d253df742c1400ee52fc7628671357da4ef3fa40 Mon Sep 17 00:00:00 2001 From: Qiao Longfei Date: Wed, 18 Oct 2017 20:12:55 -0700 Subject: [PATCH 33/76] remove Program.instance (#4915) * remove Program.instance * fix test_program.py --- python/paddle/v2/framework/framework.py | 10 +--------- python/paddle/v2/framework/tests/test_program.py | 2 +- 2 files changed, 2 insertions(+), 10 deletions(-) diff --git a/python/paddle/v2/framework/framework.py b/python/paddle/v2/framework/framework.py index 9c032400a1..a24c78171e 100644 --- a/python/paddle/v2/framework/framework.py +++ b/python/paddle/v2/framework/framework.py @@ -399,14 +399,6 @@ class Block(object): class Program(object): - @classmethod - def instance(cls): - # From https://stackoverflow.com/questions/8212053 - # Making Program as a Singleton class. - if not hasattr(cls, '_instance'): - cls._instance = cls() - return cls._instance - def __init__(self): self.desc = core.ProgramDesc() self.blocks = [Block(self, 0)] @@ -500,4 +492,4 @@ class Parameter(Variable): # program is a global instance. -g_program = Program.instance() +g_program = Program() diff --git a/python/paddle/v2/framework/tests/test_program.py b/python/paddle/v2/framework/tests/test_program.py index 8d8dd46898..c55dd8de72 100644 --- a/python/paddle/v2/framework/tests/test_program.py +++ b/python/paddle/v2/framework/tests/test_program.py @@ -53,7 +53,7 @@ class TestProgram(unittest.TestCase): print prog.clone() def test_append_backward(self): - prog = Program.instance() + prog = Program() block = prog.global_block() mul_x = block.create_var( From c1914543b0eaef98450314a1b56f4f918aa36ce2 Mon Sep 17 00:00:00 2001 From: tensor-tang Date: Thu, 19 Oct 2017 14:34:44 +0800 Subject: [PATCH 34/76] refine mkldnn logic, move reset buffers into MKLDNNLayer --- paddle/gserver/layers/MKLDNNConvLayer.cpp | 233 +++------------- paddle/gserver/layers/MKLDNNConvLayer.h | 66 ----- paddle/gserver/layers/MKLDNNFcLayer.cpp | 101 ++----- paddle/gserver/layers/MKLDNNFcLayer.h | 8 - paddle/gserver/layers/MKLDNNLayer.h | 324 ++++++++++++++++++---- paddle/gserver/layers/MKLDNNPoolLayer.cpp | 103 +------ paddle/gserver/layers/MKLDNNPoolLayer.h | 13 - paddle/math/MKLDNNMatrix.cpp | 2 +- paddle/math/MKLDNNMatrix.h | 14 +- 9 files changed, 358 insertions(+), 506 deletions(-) diff --git a/paddle/gserver/layers/MKLDNNConvLayer.cpp b/paddle/gserver/layers/MKLDNNConvLayer.cpp index 26810a6483..463e6ad0ed 100644 --- a/paddle/gserver/layers/MKLDNNConvLayer.cpp +++ b/paddle/gserver/layers/MKLDNNConvLayer.cpp @@ -116,8 +116,6 @@ void MKLDNNConvLayer::resetFwd(std::vector& pipeline, resetFwdBuffers(fwdPD_, in, wgt, bias, out); resetFwdPipeline(pipeline, fwdPD_, in, wgt, bias, out); - - printValueFormatFlow(); } void MKLDNNConvLayer::resetBwd(std::vector& pipeline, @@ -135,12 +133,6 @@ void MKLDNNConvLayer::resetBwd(std::vector& pipeline, resetBwdBuffers(bwdWgtPD, bwdDataPD, in, wgt, bias, out); resetBwdPipeline(pipeline, bwdWgtPD, bwdDataPD, in, wgt, bias, out); - - printGradFormatFlow(); -} - -void MKLDNNConvLayer::updateInputData() { - cpuInVal_->setData(getInputValue(0, CPU_DEVICE)->getData()); } void MKLDNNConvLayer::updateWeights(const UpdateCallback& callback) { @@ -211,11 +203,18 @@ void MKLDNNConvLayer::resetFwdBuffers( MKLDNNMatrixPtr& bias, MKLDNNMatrixPtr& out) { CHECK(pd); - resetInValue(pd, in); + resetInValue( + in, std::make_shared(pd->src_primitive_desc())); + + resetOutValue(out, pd->dst_primitive_desc()); - resetWgtBiasValue(pd, wgt, bias); + resetWithMatrix(wgt, weight_->getW(), pd->weights_primitive_desc()); - resetOutValue(pd, out); + bias = nullptr; + if (biases_ == nullptr || biases_->getW() == nullptr) { + return; + } + resetWithMatrix(bias, biases_->getW(), pd->bias_primitive_desc()); } void MKLDNNConvLayer::resetFwdPipeline( @@ -225,104 +224,12 @@ void MKLDNNConvLayer::resetFwdPipeline( MKLDNNMatrixPtr& wgt, MKLDNNMatrixPtr& bias, MKLDNNMatrixPtr& out) { - if (cvtInVal_) { - pipeline.push_back(*cvtInVal_); - } - if (bias) { fwd_.reset(new conv_fwd(*pd, *in, *wgt, *bias, *out)); } else { fwd_.reset(new conv_fwd(*pd, *in, *wgt, *out)); } pipeline.push_back(*fwd_); - - if (cvtOutVal_) { - pipeline.push_back(*cvtOutVal_); - } -} - -void MKLDNNConvLayer::resetInValue( - std::shared_ptr& pd, MKLDNNMatrixPtr& in) { - const MatrixPtr& inMat = inputLayers_[0]->getOutputValue(); - in = MKLDNNMatrix::create(inMat, pd->src_primitive_desc()); - - // create buffer and reorder if input value do not match - cpuInVal_ = nullptr; - cvtInVal_ = nullptr; - - MKLDNNMatrixPtr dnnIn = std::dynamic_pointer_cast(inMat); - CHECK_EQ(inputIsOnlyMKLDNN(), dnnIn != nullptr); - if (dnnIn != nullptr && dnnIn->getPrimitiveDesc() == in->getPrimitiveDesc()) { - in = dnnIn; - return; - } - if (dnnIn) { - if (dnnIn->getFormat() == format::nc) { - CHECK(ih_ == 1 && iw_ == 1) << "when input is nc format"; - // create a new one with nchw format and same data - memory::dims inDims = memory::dims{bs_, ic_, 1, 1}; - dnnIn = MKLDNNMatrix::create(inMat, inDims, format::nchw, engine_); - } - if (dnnIn->getPrimitiveDesc() == in->getPrimitiveDesc()) { - in = dnnIn; - return; - } - cpuInVal_ = dnnIn; - in = MKLDNNMatrix::create(nullptr, pd->src_primitive_desc()); - cvtInVal_ = MKLDNNMatrix::createReorder(cpuInVal_, in); - CHECK(cvtInVal_) << "should not be emptry"; - } else { - memory::dims inDims = memory::dims{bs_, ic_, ih_, iw_}; - cpuInVal_ = MKLDNNMatrix::create(inMat, inDims, format::nchw, engine_); - if (cpuInVal_->getPrimitiveDesc() != in->getPrimitiveDesc()) { - // create new mkldnn matrix - in = MKLDNNMatrix::create(nullptr, pd->src_primitive_desc()); - cvtInVal_ = MKLDNNMatrix::createReorder(cpuInVal_, in); - CHECK(cvtInVal_) << "should not be emptry"; - } else { - in = cpuInVal_; - } - } -} - -void MKLDNNConvLayer::resetWgtBiasValue( - std::shared_ptr& pd, - MKLDNNMatrixPtr& wgt, - MKLDNNMatrixPtr& bias) { - wgt = MKLDNNMatrix::create(weight_->getW(), pd->weights_primitive_desc()); - VLOG(MKLDNN_FMTS) << "Weight value format: " << wgt->getFormat(); - - bias = (biases_ && biases_->getW()) - ? MKLDNNMatrix::create(biases_->getW(), pd->bias_primitive_desc()) - : nullptr; -} - -void MKLDNNConvLayer::resetOutValue( - std::shared_ptr& pd, MKLDNNMatrixPtr& out) { - out = MKLDNNMatrix::create(output_.value, pd->dst_primitive_desc()); - - // create reorder if output value has cpu device and pd do not match - cpuOutVal_ = nullptr; - cvtOutVal_ = nullptr; - if (!outputIsOnlyMKLDNN()) { - const MatrixPtr& cpuOut = getOutput(CPU_DEVICE).value; - memory::dims outDims = memory::dims{bs_, oc_, oh_, ow_}; - cpuOutVal_ = MKLDNNMatrix::create(cpuOut, outDims, format::nchw, engine_); - if (cpuOutVal_->getPrimitiveDesc() != pd->dst_primitive_desc()) { - out = MKLDNNMatrix::create(nullptr, pd->dst_primitive_desc()); - cvtOutVal_ = MKLDNNMatrix::createReorder(out, cpuOutVal_); - CHECK(cvtOutVal_) << "should not be empty"; - } else { - cpuOut->setData(output_.value->getData()); - cpuOutVal_ = out; - } - // when output is cpu device, change the mkldnn output value and make them - // share the same data. Then if next layer use inputlayer->getOuputValue() - // to achieve the input value, it will get the right data. - output_.value = std::dynamic_pointer_cast(cpuOutVal_); - return; - } - output_.value = std::dynamic_pointer_cast(out); } void MKLDNNConvLayer::resetBwdWgtPD( @@ -331,8 +238,8 @@ void MKLDNNConvLayer::resetBwdWgtPD( loadConvSettings(wgtDims, biasDims, strides, dilations, padL, padR); // create backward weight using input, output and weight value memory desc - CHECK(inVal_) << "Should have input value"; - CHECK(outVal_) << "Should have output value"; + CHECK(inVal_) << "Should have internal input value"; + CHECK(outVal_) << "Should have internal output value"; CHECK(wgtVal_) << "Should have weight value"; algorithm algo = algorithm::convolution_direct; padding_kind padKind = padding_kind::zero; @@ -372,8 +279,8 @@ void MKLDNNConvLayer::resetBwdDataPD( memory::dims wgtDims, biasDims, strides, dilations, padL, padR; loadConvSettings(wgtDims, biasDims, strides, dilations, padL, padR); - CHECK(inVal_) << "Should have input value"; - CHECK(outVal_) << "Should have output value"; + CHECK(inVal_) << "Should have internal input value"; + CHECK(outVal_) << "Should have internal output value"; // create backward data using input and output value memory desc // but using weight memory desc with any format auto bwdDataDesc = conv_bwdData::desc(algorithm::convolution_direct, @@ -399,12 +306,27 @@ void MKLDNNConvLayer::resetBwdBuffers( MKLDNNMatrixPtr& bias, MKLDNNMatrixPtr& out) { CHECK(wgtPD); - resetOutGrad(wgtPD, out); + resetOutGrad(out, wgtPD->diff_dst_primitive_desc()); - resetWgtBiasGrad(wgtPD, wgt, bias); + resetWithMatrix( + wgt, weight_->getWGrad(), wgtPD->diff_weights_primitive_desc()); + CHECK(wgtVal_ != nullptr && + wgt->getPrimitiveDesc() == wgtVal_->getPrimitiveDesc()) + << "primitive desc of weight grad and value should be equal"; - resetInGrad(dataPD, in); + bias = nullptr; + if (biases_ && biases_->getWGrad()) { + resetWithMatrix( + bias, biases_->getWGrad(), wgtPD->diff_bias_primitive_desc()); + CHECK(bias && biasVal_ && + bias->getPrimitiveDesc() == biasVal_->getPrimitiveDesc()) + << "primitive desc of bias grad should equal the bias value"; + } + if (dataPD == nullptr) { + return; + } + resetInGrad(in, dataPD->diff_src_primitive_desc()); resetWgtValBwdData(dataPD, wgtValBwdData_); } @@ -416,10 +338,7 @@ void MKLDNNConvLayer::resetBwdPipeline( MKLDNNMatrixPtr& wgt, MKLDNNMatrixPtr& bias, MKLDNNMatrixPtr& out) { - if (cvtOutGrad_) { - pipeline.push_back(*cvtOutGrad_); - } - + CHECK(inVal_); // add bwdWgt handle if (bias) { bwdWgt_.reset(new conv_bwdWgt(*wgtPD, *inVal_, *out, *wgt, *bias)); @@ -431,99 +350,13 @@ void MKLDNNConvLayer::resetBwdPipeline( if (dataPD == nullptr) { return; } - if (cvtWgtVal_) { pipeline.push_back(*cvtWgtVal_); } - // add bwdData handle CHECK(wgtValBwdData_) << "Should have weight memory"; bwdData_.reset(new conv_bwdData(*dataPD, *out, *wgtValBwdData_, *in)); pipeline.push_back(*bwdData_); - - if (cvtInGrad_) { - pipeline.push_back(*cvtInGrad_); - } -} - -void MKLDNNConvLayer::resetOutGrad( - std::shared_ptr& wgtPD, MKLDNNMatrixPtr& out) { - cpuOutGrad_ = nullptr; - cvtOutGrad_ = nullptr; - CHECK(outVal_ != nullptr && - outVal_->getPrimitiveDesc() == wgtPD->diff_dst_primitive_desc()) - << "primitive desc of out grad and value should be equal"; - if (outputIsOnlyMKLDNN()) { - MKLDNNLayer::resetOutGrad(out, outVal_->getPrimitiveDesc()); - } else { - const MatrixPtr& cpuOut = getOutput(CPU_DEVICE).grad; - // always share the same grad data of CPU output - // then the activation can get the right grad from output_.grad - output_.grad->setData(cpuOut->getData()); - // same PrimitiveDesc with cpuInVal_ - CHECK(cpuOutVal_); - cpuOutGrad_ = MKLDNNMatrix::create(cpuOut, cpuOutVal_->getPrimitiveDesc()); - // create reorder if primitive desc does not match - if (cpuOutGrad_->getPrimitiveDesc() != outVal_->getPrimitiveDesc()) { - out = MKLDNNMatrix::create(nullptr, outVal_->getPrimitiveDesc()); - cvtOutGrad_ = MKLDNNMatrix::createReorder(cpuOutGrad_, out); - CHECK(cvtOutGrad_); - } else { - out = cpuOutGrad_; - } - } -} - -void MKLDNNConvLayer::resetWgtBiasGrad( - std::shared_ptr& wgtPD, - MKLDNNMatrixPtr& wgt, - MKLDNNMatrixPtr& bias) { - wgt = MKLDNNMatrix::create(weight_->getWGrad(), - wgtPD->diff_weights_primitive_desc()); - CHECK(nullptr != wgtVal_ && - wgt->getPrimitiveDesc() == wgtVal_->getPrimitiveDesc()) - << "primitive desc of weight grad and value should be equal"; - VLOG(MKLDNN_FMTS) << "weight grad format: " << wgt->getFormat(); - - bias = nullptr; - if (biasVal_ == nullptr) { - return; - } - bias = MKLDNNMatrix::create(biases_->getWGrad(), - wgtPD->diff_bias_primitive_desc()); - CHECK(bias->getPrimitiveDesc() == biasVal_->getPrimitiveDesc()) - << "primitive desc of bias grad should equal the bias value"; -} - -void MKLDNNConvLayer::resetInGrad( - std::shared_ptr& dataPD, - MKLDNNMatrixPtr& in) { - in = nullptr; - cpuInGrad_ = nullptr; - cvtInGrad_ = nullptr; - if (dataPD == nullptr) { - return; - } - - if (inputIsOnlyMKLDNN()) { - MKLDNNLayer::resetInGrad(in, dataPD->diff_src_primitive_desc()); - CHECK(nullptr != inVal_ && - in->getPrimitiveDesc() == inVal_->getPrimitiveDesc()) - << "primitive desc of input grad and value should be equal"; - } else { - const MatrixPtr& cpuIn = getInputGrad(0, CPU_DEVICE); - // same PrimitiveDesc with cpuInVal_ - CHECK(cpuInVal_); - cpuInGrad_ = MKLDNNMatrix::create(cpuIn, cpuInVal_->getPrimitiveDesc()); - in = cpuInGrad_; - // create reorder if PrimitiveDesc does not match - if (cpuInGrad_->getPrimitiveDesc() != dataPD->diff_src_primitive_desc()) { - in = MKLDNNMatrix::create(getInputGrad(0, MKLDNN_DEVICE), - dataPD->diff_src_primitive_desc()); - cvtInGrad_ = MKLDNNMatrix::createReorder(in, cpuInGrad_); - CHECK(cvtInGrad_); - } - } } void MKLDNNConvLayer::resetWgtValBwdData( diff --git a/paddle/gserver/layers/MKLDNNConvLayer.h b/paddle/gserver/layers/MKLDNNConvLayer.h index f84f2f737c..1fed0e1c65 100644 --- a/paddle/gserver/layers/MKLDNNConvLayer.h +++ b/paddle/gserver/layers/MKLDNNConvLayer.h @@ -48,17 +48,6 @@ protected: // save forward primitive_desc, which can be used backward std::shared_ptr fwdPD_; - // MKLDNNMatrixPtr which should be created from CPU Device - MKLDNNMatrixPtr cpuInVal_; - MKLDNNMatrixPtr cpuInGrad_; - MKLDNNMatrixPtr cpuOutVal_; - MKLDNNMatrixPtr cpuOutGrad_; - // convert handle between CPU device and MKLDNN device - std::shared_ptr cvtInVal_; - std::shared_ptr cvtInGrad_; - std::shared_ptr cvtOutVal_; - std::shared_ptr cvtOutGrad_; - // whether the weight has been init bool hasInitedWgt_; @@ -94,8 +83,6 @@ public: MKLDNNMatrixPtr& bias, MKLDNNMatrixPtr& out) override; - void updateInputData() override; - void updateWeights(const UpdateCallback& callback) override; void convertWeightsFromPaddle() override; @@ -109,26 +96,6 @@ public: << ", sw: " << sw_ << ", dh: " << dh_ << ", dw: " << dw_; } - void printValueFormatFlow() override { - if (cpuInVal_) { - VLOG(MKLDNN_FMTS) << cpuInVal_->getFormat() << " >>>"; - } - MKLDNNLayer::printValueFormatFlow(); - if (cpuOutVal_) { - VLOG(MKLDNN_FMTS) << " >>> " << cpuOutVal_->getFormat(); - } - } - - void printGradFormatFlow() override { - if (cpuInGrad_) { - VLOG(MKLDNN_FMTS) << cpuInGrad_->getFormat() << " <<<"; - } - MKLDNNLayer::printGradFormatFlow(); - if (cpuOutGrad_) { - VLOG(MKLDNN_FMTS) << " <<< " << cpuOutGrad_->getFormat(); - } - } - protected: /** * load the dims settings of this conv @@ -162,23 +129,6 @@ protected: MKLDNNMatrixPtr& bias, MKLDNNMatrixPtr& out); - /** - * reset MKLDNNMatrix of input value - */ - void resetInValue(std::shared_ptr& pd, - MKLDNNMatrixPtr& in); - /** - * reset MKLDNNMatrix of weight and bias value - */ - void resetWgtBiasValue(std::shared_ptr& pd, - MKLDNNMatrixPtr& wgt, - MKLDNNMatrixPtr& bias); - /** - * reset MKLDNNMatrix of output value - */ - void resetOutValue(std::shared_ptr& pd, - MKLDNNMatrixPtr& out); - /** * reset the backward weight primitive descriptor. */ @@ -207,22 +157,6 @@ protected: MKLDNNMatrixPtr& bias, MKLDNNMatrixPtr& out); - /** - * reset MKLDNNMatrix of output grad - */ - void resetOutGrad(std::shared_ptr& wgtPD, - MKLDNNMatrixPtr& out); - /** - * reset MKLDNNMatrix of weight and bias grad - */ - void resetWgtBiasGrad(std::shared_ptr& wgtPD, - MKLDNNMatrixPtr& wgt, - MKLDNNMatrixPtr& bias); - /** - * reset MKLDNNMatrix of input grad - */ - void resetInGrad(std::shared_ptr& dataPD, - MKLDNNMatrixPtr& in); /** * reset MKLDNNMatrix of weight value for backward data * since the primitive_desc would be different with wgtVal_ diff --git a/paddle/gserver/layers/MKLDNNFcLayer.cpp b/paddle/gserver/layers/MKLDNNFcLayer.cpp index cf19a15568..9f82a3b747 100644 --- a/paddle/gserver/layers/MKLDNNFcLayer.cpp +++ b/paddle/gserver/layers/MKLDNNFcLayer.cpp @@ -62,7 +62,7 @@ void MKLDNNFcLayer::convertWeightsFromPaddle() { CHECK(wgtVal_) << "should have been initialized"; bool hasNoSpatial_ = ih_ == 1 && iw_ == 1; auto targetDim = wgtVal_->getDims(); - auto srcFmt = hasNoSpatial_ ? memory::format::io : memory::format::ihwo; + auto srcFmt = hasNoSpatial_ ? format::io : format::ihwo; wgtVal_->reorderDataFrom(wgtVal_, srcFmt, targetDim); hasInitedWgt_ = true; } @@ -71,7 +71,7 @@ void MKLDNNFcLayer::convertWeightsToPaddle() { CHECK(wgtVal_) << "should have been initialized"; bool hasNoSpatial_ = ih_ == 1 && iw_ == 1; auto targetDim = wgtVal_->getDims(); - auto dstFmt = hasNoSpatial_ ? memory::format::io : memory::format::ihwo; + auto dstFmt = hasNoSpatial_ ? format::io : format::ihwo; wgtVal_->reorderDataTo(wgtVal_, dstFmt, targetDim); } @@ -100,8 +100,6 @@ void MKLDNNFcLayer::resetFwd(std::vector& pipeline, resetFwdPD(fwdPD_, in, wgt, bias, out); resetFwdPipeline(pipeline, fwdPD_, in, wgt, bias, out); - - printValueFormatFlow(); } void MKLDNNFcLayer::resetBwd(std::vector& pipeline, @@ -119,12 +117,6 @@ void MKLDNNFcLayer::resetBwd(std::vector& pipeline, resetBwdDataPD(bwdDataPD, in, out); resetBwdPipeline(pipeline, bwdWgtPD, bwdDataPD, in, wgt, bias, out); - - printGradFormatFlow(); -} - -void MKLDNNFcLayer::updateInputData() { - inVal_->setData(getInputValue(0, CPU_DEVICE)->getData()); } void MKLDNNFcLayer::updateWeights(const UpdateCallback& callback) { @@ -139,51 +131,33 @@ void MKLDNNFcLayer::resetFwdBuffers(MKLDNNMatrixPtr& in, MKLDNNMatrixPtr& bias, MKLDNNMatrixPtr& out) { resetInValue(in); + CHECK(in); + in->downSpatial(); - resetWgtBiasValue(wgt, bias); - - resetOutValue(out); -} + // if (extInVal_) { + // extInVal_->downSpatial(); + // } -void MKLDNNFcLayer::resetInValue(MKLDNNMatrixPtr& in) { - if (inputIsOnlyMKLDNN()) { - const MatrixPtr& dnnIn = getInputValue(0); - in = std::dynamic_pointer_cast(dnnIn); - CHECK(in) << "Input should be MKLDNNMatrix"; - } else { - CHECK_EQ(getPrev(0)->getDeviceId(), CPU_DEVICE) << "Only support CPU yet"; - const MatrixPtr& cpuIn = getInputValue(0, CPU_DEVICE); - in = MKLDNNMatrix::create( - cpuIn, {bs_, ic_, ih_, iw_}, format::nchw, engine_); - } - in->downSpatial(); -} + auto outPD = + MKLDNNMatrix::createPrimitiveDesc({bs_, oc_}, format::nc, engine_); + resetOutValue(out, outPD); -void MKLDNNFcLayer::resetWgtBiasValue(MKLDNNMatrixPtr& wgt, - MKLDNNMatrixPtr& bias) { format wgtFmt = format::oihw; - if (inVal_->getFormat() == format::nChw8c) { + if (in->getFormat() == format::nChw8c) { wgtFmt = format::oIhw8i; - } else if (inVal_->getFormat() == format::nChw16c) { + } else if (in->getFormat() == format::nChw16c) { wgtFmt = format::oIhw16i; } - wgt = MKLDNNMatrix::create( - weight_->getW(), {oc_, ic_, ih_, iw_}, wgtFmt, engine_); + auto wgtPD = + MKLDNNMatrix::createPrimitiveDesc({oc_, ic_, ih_, iw_}, wgtFmt, engine_); + resetWithMatrix(wgt, weight_->getW(), wgtPD); wgt->downSpatial(); - VLOG(MKLDNN_FMTS) << "Weight value format: " << wgt->getFormat(); - - bias = (biases_ && biases_->getW()) - ? MKLDNNMatrix::create(biases_->getW(), {oc_}, format::x, engine_) - : nullptr; -} -void MKLDNNFcLayer::resetOutValue(MKLDNNMatrixPtr& out) { - out = MKLDNNMatrix::create(output_.value, {bs_, oc_}, format::nc, engine_); - if (!outputIsOnlyMKLDNN()) { - // fc cpu output value do not need create convert, just share data - getOutput(CPU_DEVICE).value->setData(out->getData()); + if (biases_ == nullptr || biases_->getW() == nullptr) { + return; } - output_.value = std::dynamic_pointer_cast(out); + auto biasPD = MKLDNNMatrix::createPrimitiveDesc({oc_}, format::x, engine_); + resetWithMatrix(bias, biases_->getW(), biasPD); } void MKLDNNFcLayer::resetFwdPD(std::shared_ptr& pd, @@ -219,7 +193,6 @@ void MKLDNNFcLayer::resetFwdPipeline( } else { fwd_.reset(new fc_fwd(*pd, *in, *wgt, *out)); } - pipeline.push_back(*fwd_); } @@ -227,44 +200,18 @@ void MKLDNNFcLayer::resetBwdBuffers(MKLDNNMatrixPtr& in, MKLDNNMatrixPtr& wgt, MKLDNNMatrixPtr& bias, MKLDNNMatrixPtr& out) { - resetOutGrad(out); - - resetWgtBiasGrad(wgt, bias); - - resetInGrad(in); -} - -void MKLDNNFcLayer::resetOutGrad(MKLDNNMatrixPtr& out) { - CHECK(outVal_); - if (outputIsOnlyMKLDNN()) { - MKLDNNLayer::resetOutGrad(out, outVal_->getPrimitiveDesc()); - } else { - const MatrixPtr& cpuOut = getOutput(CPU_DEVICE).grad; - output_.grad->setData(cpuOut->getData()); - out = MKLDNNMatrix::create(cpuOut, outVal_->getPrimitiveDesc()); - } -} + CHECK(inVal_ && outVal_); + resetOutGrad(out, outVal_->getPrimitiveDesc()); + resetInGrad(in, inVal_->getPrimitiveDesc()); -void MKLDNNFcLayer::resetWgtBiasGrad(MKLDNNMatrixPtr& wgt, - MKLDNNMatrixPtr& bias) { CHECK(wgtVal_); - wgt = MKLDNNMatrix::create(weight_->getWGrad(), wgtVal_->getPrimitiveDesc()); + resetWithMatrix(wgt, weight_->getWGrad(), wgtVal_->getPrimitiveDesc()); bias = nullptr; if (biasVal_ == nullptr) { return; } - bias = - MKLDNNMatrix::create(biases_->getWGrad(), biasVal_->getPrimitiveDesc()); -} - -void MKLDNNFcLayer::resetInGrad(MKLDNNMatrixPtr& in) { - in = nullptr; - if (inputLayers_[0]->getOutput().grad == nullptr) { - return; - } - CHECK(inVal_); - MKLDNNLayer::resetInGrad(in, inVal_->getPrimitiveDesc()); + resetWithMatrix(bias, biases_->getWGrad(), biasVal_->getPrimitiveDesc()); } void MKLDNNFcLayer::resetBwdWgtPD( diff --git a/paddle/gserver/layers/MKLDNNFcLayer.h b/paddle/gserver/layers/MKLDNNFcLayer.h index c76878aafa..ee861763ff 100644 --- a/paddle/gserver/layers/MKLDNNFcLayer.h +++ b/paddle/gserver/layers/MKLDNNFcLayer.h @@ -66,8 +66,6 @@ public: MKLDNNMatrixPtr& bias, MKLDNNMatrixPtr& out) override; - void updateInputData() override; - void updateWeights(const UpdateCallback& callback) override; void convertWeightsFromPaddle() override; @@ -84,9 +82,6 @@ protected: MKLDNNMatrixPtr& wgt, MKLDNNMatrixPtr& bias, MKLDNNMatrixPtr& out); - void resetInValue(MKLDNNMatrixPtr& in); - void resetWgtBiasValue(MKLDNNMatrixPtr& wgt, MKLDNNMatrixPtr& bias); - void resetOutValue(MKLDNNMatrixPtr& out); void resetFwdPD(std::shared_ptr& pd, MKLDNNMatrixPtr in, MKLDNNMatrixPtr wgt, @@ -109,9 +104,6 @@ protected: MKLDNNMatrixPtr& wgt, MKLDNNMatrixPtr& bias, MKLDNNMatrixPtr& out); - void resetOutGrad(MKLDNNMatrixPtr& out); - void resetWgtBiasGrad(MKLDNNMatrixPtr& wgt, MKLDNNMatrixPtr& bias); - void resetInGrad(MKLDNNMatrixPtr& in); void resetBwdWgtPD(std::shared_ptr& pd, MKLDNNMatrixPtr& wgt, MKLDNNMatrixPtr& bias, diff --git a/paddle/gserver/layers/MKLDNNLayer.h b/paddle/gserver/layers/MKLDNNLayer.h index 4e2753eba2..ab59357ad0 100644 --- a/paddle/gserver/layers/MKLDNNLayer.h +++ b/paddle/gserver/layers/MKLDNNLayer.h @@ -58,11 +58,30 @@ protected: std::vector pipelineFwd_; std::vector pipelineBwd_; - // MKLDNNMatrixPtr with internal format + /// value and grad are seperate as internal and external buffers. + /// each MKLDNNLayer must init or reset internal buffer at least, + /// and the external buffer format is always nchw of nc(when h==w==1), + /// which is the same format as paddle. + /// When mixed with cpu device, the output_.value and output_.grad + /// always save the external data. + /// When all layers are all mkldnn layers, they could be internal data. + /// below MKLDNNMatrix buffers are all internal buffers MKLDNNMatrixPtr inVal_; MKLDNNMatrixPtr inGrad_; MKLDNNMatrixPtr outVal_; MKLDNNMatrixPtr outGrad_; + // below are external value and grad + MKLDNNMatrixPtr extInVal_; + MKLDNNMatrixPtr extInGrad_; + MKLDNNMatrixPtr extOutVal_; + MKLDNNMatrixPtr extOutGrad_; + // convert handle between external and internal buffers + std::shared_ptr cvtInVal_; + std::shared_ptr cvtInGrad_; + std::shared_ptr cvtOutVal_; + std::shared_ptr cvtOutGrad_; + + // weight and bias are always internal buffers MKLDNNMatrixPtr wgtVal_; MKLDNNMatrixPtr wgtGrad_; MKLDNNMatrixPtr biasVal_; @@ -91,6 +110,7 @@ public: oh_(0), ow_(0), needResetBwd_(true), + outputOnlyMKLDNN_(false), engine_(mkldnn::engine::cpu, 0), stream_(nullptr), fwd_(nullptr), @@ -128,20 +148,39 @@ public: REGISTER_TIMER_INFO("mkldnn_FwdTimer", getName().c_str()); CHECK(!inputLayers_.empty()); copySeqInfoToOutputs(); - size_t elemenCnt = inputLayers_[0]->getOutput().value->getElementCnt(); + size_t elemenCnt = inputLayers_[0]->getOutputValue()->getElementCnt(); if (inputElemenCnt_ != elemenCnt) { VLOG(MKLDNN_BASE) << getName() << " reset mkldnn forward"; // reset when input total sizes changed, not only the batchsize inputElemenCnt_ = elemenCnt; pipelineFwd_.clear(); reshape(bs_, ic_, ih_, iw_, oc_, oh_, ow_); + // all cpu device output grad or value share output's + shareCPUDevice(); resetFwd(pipelineFwd_, inVal_, wgtVal_, biasVal_, outVal_); + // MKLDNNLayer output value should be MKLDNNMatrix + // so external output value is necessary. + // then external input value is not necessary, + // since input may be mkldnn internal buffer. + CHECK(extOutVal_) << "external output value is necessary"; + output_.value = std::dynamic_pointer_cast(extOutVal_); + CHECK(inVal_ && outVal_) << "internal memories are necessary"; + if (cvtInVal_) { + pipelineFwd_.insert(pipelineFwd_.begin(), *cvtInVal_); + } + if (cvtOutVal_) { + pipelineFwd_.push_back(*cvtOutVal_); + } convertWeightsFromPaddle(); + printValueFormat(); needResetBwd_ = true; } if (inputLayers_[0]->getType() == "data") { - updateInputData(); + // Update input value data when input layer is "data" type, + // since the input value data address might be changed. + CHECK(extInVal_); + extInVal_->setData(getInputValue(0, CPU_DEVICE)->getData()); } if (!outputOnlyMKLDNN_) { @@ -149,8 +188,7 @@ public: } stream_->submit(pipelineFwd_); } - - /* activation */ { + { REGISTER_TIMER_INFO("FwActTimer", getName().c_str()); forwardActivation(); } @@ -163,6 +201,16 @@ public: pipelineMergeGrad_.clear(); mergeGrad_ = nullptr; resetBwd(pipelineBwd_, inGrad_, wgtGrad_, biasGrad_, outGrad_); + // external output grad is not necessary + // since output may be mkldnn internal buffer or merge them directly. + CHECK(outGrad_) << "internal output grad is necessary"; + if (cvtOutGrad_) { + pipelineBwd_.insert(pipelineBwd_.begin(), *cvtOutGrad_); + } + if (cvtInGrad_) { + pipelineBwd_.push_back(*cvtInGrad_); + } + printGradFormat(); needResetBwd_ = false; } @@ -179,7 +227,6 @@ public: REGISTER_TIMER_INFO("mkldnn_bwdTimer", getName().c_str()); stream_->submit(pipelineBwd_); } - { REGISTER_TIMER_INFO("WeightUpdate", getName().c_str()); updateWeights(callback); @@ -195,7 +242,7 @@ public: int& bs, int& ic, int& ih, int& iw, int oc, int& oh, int& ow) = 0; /** - * reset the mkldnn forward primitve and memory + * reset the mkldnn forward primitve and memories * only would be called when input size changes */ virtual void resetFwd(std::vector& pipeline, @@ -205,7 +252,7 @@ public: MKLDNNMatrixPtr& out) = 0; /** - * reset the mkldnn backward primitve and memory for mkldnn fc + * reset the mkldnn backward primitve and memories * only would be called when needed */ virtual void resetBwd(std::vector& pipeline, @@ -214,12 +261,6 @@ public: MKLDNNMatrixPtr& bias, MKLDNNMatrixPtr& out) = 0; - /** - * Update input value data when input layer is "data" type. - * Since the input value data address might be changed. - */ - virtual void updateInputData() {} - /** * Update weights and biases if necessary. */ @@ -272,21 +313,167 @@ protected: } /** - * reset the output grad matrix from primitive desc. - * and reset the merge grad primitive if needed. - * note: when this layer has serval outputs, + * reset MKLDNNMatrix from Matrix and internal primitive desc. + * reset nullptr if matrix or primitive desc is empty + */ + void resetWithMatrix(MKLDNNMatrixPtr& dnn, + const MatrixPtr& mat, + mkldnn::memory::primitive_desc pd) { + dnn = nullptr; + if (mat == nullptr) { + return; + } + dnn = MKLDNNMatrix::create(mat, pd); + } + + /** + * reset input value from input MKLDNNMatrix and internal primitive desc. + * reset both internal and external buffer and create reorder if necessary. + */ + void resetInValue( + MKLDNNMatrixPtr& in, + const std::shared_ptr& intPD = nullptr) { + cvtInVal_ = nullptr; + extInVal_ = nullptr; + in = nullptr; + CHECK_GT(bs_ * ic_ * ih_ * iw_, 0); + auto extPD = MKLDNNMatrix::createPrimitiveDesc( + {bs_, ic_, ih_, iw_}, mkldnn::memory::format::nchw, engine_); + const MatrixPtr& inMat = inputLayers_[0]->getOutputValue(); + in = std::dynamic_pointer_cast(inMat); + CHECK_EQ(inputIsOnlyMKLDNN(), in != nullptr); + if (in == nullptr || in->getFormat() == mkldnn::memory::format::nc) { + in = MKLDNNMatrix::create(inMat, extPD); + } + extInVal_ = isPaddleFormat(in->getFormat()) ? in : nullptr; + if (in->getFormat() == mkldnn::memory::format::nc) { + CHECK(ih_ == 1 && iw_ == 1); + } + if (nullptr == intPD || in->getPrimitiveDesc() == *intPD) { + return; + } + // need create reorder + in = MKLDNNMatrix::create(nullptr, *intPD); + extInVal_ = extInVal_ ? extInVal_ : MKLDNNMatrix::create(inMat, extPD); + cvtInVal_ = MKLDNNMatrix::createReorder(extInVal_, in); + CHECK(cvtInVal_) << "should not be emptry"; + } + + /** + * reset output value from internal primitive desc. + * reset both internal and external buffer and create reorder if necessary. + */ + void resetOutValue(MKLDNNMatrixPtr& out, + mkldnn::memory::primitive_desc intPD) { + cvtOutVal_ = nullptr; + out = MKLDNNMatrix::create(output_.value, intPD); + extOutVal_ = out; + if (outputIsOnlyMKLDNN() || isPaddleFormat(extOutVal_->getFormat())) { + return; + } + // need create reorder + CHECK_GT(bs_ * oc_ * oh_ * ow_, 0); + extOutVal_ = MKLDNNMatrix::create(output_.value, + {bs_, oc_, oh_, ow_}, + mkldnn::memory::format::nchw, + engine_); + out = MKLDNNMatrix::create(nullptr, intPD); + cvtOutVal_ = MKLDNNMatrix::createReorder(out, extOutVal_); + CHECK(cvtOutVal_) << "should not be empty"; + } + + /** + * reset input grad from internal primitive desc. + * reset both internal and external buffer and create reorder if necessary. + */ + void resetInGrad(MKLDNNMatrixPtr& in, mkldnn::memory::primitive_desc intPD) { + cvtInGrad_ = nullptr; + extInGrad_ = nullptr; + in = nullptr; + LayerPtr& input = inputLayers_[0]; + if (input->getOutputGrad() == nullptr) { + // no need input grad + return; + } + CHECK(inputIsOnlyMKLDNN() || input->getOutputMapSize() <= 1) + << "only support input is MKLDNN layer or only have one output layer"; + // when input is a mkldnn branch node, + // this layer will save input grad to a internal buffer, + // and the mkldnn input layer will merge them to actual prev->output_.grad + const MatrixPtr& inMat = + input->getOutputMapSize() <= 1 ? input->getOutputGrad() : nullptr; + in = MKLDNNMatrix::create(inMat, intPD); + Argument& arg = input->getOutput(this->getName()); + arg.grad = std::dynamic_pointer_cast(in); + CHECK(inVal_ != nullptr && inVal_->getPrimitiveDesc() == intPD) + << "should have internal input value and primitive desc must equal"; + if (inputIsOnlyMKLDNN()) { + return; + } + + extInGrad_ = in; + if (isPaddleFormat(extInGrad_->getFormat())) { + return; + } + // need create reorder + CHECK(extInVal_ != nullptr && isPaddleFormat(extInVal_->getFormat())) + << "should have external input value and the format must be nchw(nc)"; + extInGrad_ = MKLDNNMatrix::create(inMat, extInVal_->getPrimitiveDesc()); + CHECK(inVal_ != nullptr && inVal_->getPrimitiveDesc() == intPD) + << "should have internal input value and primitive desc must equal"; + in = MKLDNNMatrix::create(nullptr, intPD); + cvtInGrad_ = MKLDNNMatrix::createReorder(in, extInGrad_); + CHECK(cvtInGrad_); + } + + /** + * reset output grad from internal primitive desc. + * merge grad if necessary. + * reset both internal and external buffer and create reorder if necessary. + * note: about merge grad, when this layer has serval outputs, * it could not be mixed with cpu device, * since it can not get memory desc from cpu device. */ - virtual void resetOutGrad(MKLDNNMatrixPtr& out, - mkldnn::memory::primitive_desc pd) { - CHECK(outputIsOnlyMKLDNN()) << "do not support mixed with other device yet"; + void resetOutGrad(MKLDNNMatrixPtr& out, + mkldnn::memory::primitive_desc intPD) { + cvtOutGrad_ = nullptr; + extOutGrad_ = nullptr; + out = nullptr; + MatrixPtr& outMat = output_.grad; + out = MKLDNNMatrix::create(outMat, intPD); + resetMergeGrad(out); + if (outputIsOnlyMKLDNN()) { + return; + } + CHECK_LE(outputMap_.size(), 1U) << "do not support mixed with cpu device"; + extOutGrad_ = out; + if (isPaddleFormat(extOutGrad_->getFormat())) { + return; + } + // need create reorder + CHECK(extOutVal_ != nullptr && isPaddleFormat(extOutVal_->getFormat())) + << "should have external output value and the format must be nchw(nc)"; + extOutGrad_ = MKLDNNMatrix::create(outMat, extOutVal_->getPrimitiveDesc()); + CHECK(outVal_ != nullptr && outVal_->getPrimitiveDesc() == intPD) + << "should have internal output value and primitive desc must equal"; + out = MKLDNNMatrix::create(nullptr, intPD); + cvtOutGrad_ = MKLDNNMatrix::createReorder(extOutGrad_, out); + CHECK(cvtOutGrad_); + } + + /** + * reset the merge grad primitive if necessary. + * note: do not support the grads are mixed with cpu device, + * since it can not get memory desc from cpu device. + */ + virtual void resetMergeGrad(MKLDNNMatrixPtr& out) { mergeGrad_ = nullptr; pipelineMergeGrad_.clear(); - out = MKLDNNMatrix::create(output_.grad, pd); - if (outputMap_.size() <= 1) { + if (outputMap_.size() <= 1 || !outputIsOnlyMKLDNN()) { + // do not merge when output is not all MKLDNN or only one output return; } + CHECK(out) << "should have reset internal ouput grad"; std::vector scales(outputMap_.size(), 1.0); std::vector srcPDs; std::vector srcs; @@ -309,15 +496,13 @@ protected: for (size_t i = 1; i < srcPDs.size(); ++i) { CHECK(srcPDs[0] == srcPDs[i]); } - tmpOutGrad_ = nullptr; + tmpOutGrad_ = out; tmpCvt_ = nullptr; if (out->getPrimitiveDesc() != srcPDs[0]) { tmpOutGrad_ = MKLDNNMatrix::create(nullptr, srcPDs[0]); tmpCvt_ = MKLDNNMatrix::createReorder(tmpOutGrad_, out); CHECK(tmpCvt_); pipelineMergeGrad_.push_back(*tmpCvt_); - } else { - tmpOutGrad_ = out; } auto sumPD = mkldnn::sum::primitive_desc( @@ -326,21 +511,6 @@ protected: pipelineMergeGrad_.insert(pipelineMergeGrad_.begin(), *mergeGrad_); } - /** - * reset input grad from primitive desc. - * this function is avaiable for input is only mkldnn - * or input do not care cpu device - */ - virtual void resetInGrad(MKLDNNMatrixPtr& in, - mkldnn::memory::primitive_desc pd) { - LayerPtr& input = inputLayers_[0]; - const MatrixPtr& grad = - input->getOutputMapSize() > 1 ? nullptr : input->getOutput().grad; - in = MKLDNNMatrix::create(grad, pd); - Argument& arg = input->getOutput(this->getName()); - arg.grad = std::dynamic_pointer_cast(in); - } - /** * print info about sizes */ @@ -351,22 +521,50 @@ protected: } /** - * Print the mkldnn memory format flow of value + * print the mkldnn memory format of value */ - virtual void printValueFormatFlow() { - if (inVal_ && outVal_) { - VLOG(MKLDNN_FMTS) << inVal_->getFormat() << " >>> " - << outVal_->getFormat(); + virtual void printValueFormat() { + if (extInVal_) { + VLOG(MKLDNN_FMTS) << extInVal_->getFormat() << " >>> "; + } + if (inVal_) { + VLOG(MKLDNN_FMTS) << inVal_->getFormat() << " >>>"; + } + if (outVal_) { + VLOG(MKLDNN_FMTS) << outVal_->getFormat() << " >>> "; + } + if (extOutVal_) { + VLOG(MKLDNN_FMTS) << extOutVal_->getFormat(); + } + if (wgtVal_) { + VLOG(MKLDNN_FMTS) << "Weight value format: " << wgtVal_->getFormat(); + } + if (biasVal_) { + VLOG(MKLDNN_FMTS) << "Bias value format: " << biasVal_->getFormat(); } } /** - * Print the mkldnn memory format flow of grad + * print the mkldnn memory format of grad */ - virtual void printGradFormatFlow() { - if (inGrad_ && outGrad_) { - VLOG(MKLDNN_FMTS) << inGrad_->getFormat() << " <<< " - << outGrad_->getFormat(); + virtual void printGradFormat() { + if (extInGrad_) { + VLOG(MKLDNN_FMTS) << extInGrad_->getFormat() << " <<< "; + } + if (inGrad_) { + VLOG(MKLDNN_FMTS) << inGrad_->getFormat() << " <<<"; + } + if (outGrad_) { + VLOG(MKLDNN_FMTS) << outGrad_->getFormat() << " <<< "; + } + if (extOutGrad_) { + VLOG(MKLDNN_FMTS) << extOutGrad_->getFormat(); + } + if (wgtGrad_) { + VLOG(MKLDNN_FMTS) << "Weight grad format: " << wgtGrad_->getFormat(); + } + if (biasGrad_) { + VLOG(MKLDNN_FMTS) << "Bias grad format: " << biasGrad_->getFormat(); } } @@ -405,6 +603,19 @@ protected: void setDevice(int id) { deviceId_ = id; } private: + /** + * check the format is nchw or nc, + * which is supported by Paddle default memory layout + */ + bool isPaddleFormat(mkldnn::memory::format fmt) { + if (fmt == mkldnn::memory::format::nchw || + fmt == mkldnn::memory::format::nc) { + return true; + } else { + return false; + } + } + /** * clear all grad */ @@ -449,6 +660,19 @@ private: } } + /** + * if have cpu device, share value and grad data with output_ + */ + void shareCPUDevice() { + if (outputIsOnlyMKLDNN()) { + return; + } + for (size_t i = 0; i < outputOtherDevice_.size(); i++) { + outputOtherDevice_[i].value = output_.value; + outputOtherDevice_[i].grad = output_.grad; + } + } + /** * Check the cpu device number of outputOtherDevice_. * should have only one at most. diff --git a/paddle/gserver/layers/MKLDNNPoolLayer.cpp b/paddle/gserver/layers/MKLDNNPoolLayer.cpp index 0e53e2d1b7..6e89260f49 100644 --- a/paddle/gserver/layers/MKLDNNPoolLayer.cpp +++ b/paddle/gserver/layers/MKLDNNPoolLayer.cpp @@ -85,8 +85,6 @@ void MKLDNNPoolLayer::resetFwd(std::vector& pipeline, resetFwdPD(fwdPD_, in, out); resetFwdPipeline(pipeline, fwdPD_, in, out); - - printValueFormatFlow(); } void MKLDNNPoolLayer::resetBwd(std::vector& pipeline, @@ -101,65 +99,22 @@ void MKLDNNPoolLayer::resetBwd(std::vector& pipeline, resetBwdPD(pd, in, out); resetBwdPipeline(pipeline, pd, in, out); - - printGradFormatFlow(); -} - -void MKLDNNPoolLayer::updateInputData() { - inVal_->setData(getInputValue(0, CPU_DEVICE)->getData()); } void MKLDNNPoolLayer::resetFwdBuffers(MKLDNNMatrixPtr& in, MKLDNNMatrixPtr& out) { resetInValue(in); - resetOutValue(out); -} - -void MKLDNNPoolLayer::resetInValue(MKLDNNMatrixPtr& in) { - if (inputIsOnlyMKLDNN()) { - const MatrixPtr& dnnIn = getInputValue(0); - in = std::dynamic_pointer_cast(dnnIn); - CHECK(in) << "Input should be MKLDNNMatrix"; - } else { - CHECK_EQ(getPrev(0)->getDeviceId(), CPU_DEVICE) << "Only support CPU yet"; - const MatrixPtr& cpuIn = getInputValue(0, CPU_DEVICE); - in = MKLDNNMatrix::create( - cpuIn, {bs_, ic_, ih_, iw_}, format::nchw, engine_); - } -} - -void MKLDNNPoolLayer::resetOutValue(MKLDNNMatrixPtr& out) { - CHECK(inVal_) << "Should reset input value first"; memory::dims outDims = memory::dims{bs_, oc_, oh_, ow_}; - out = MKLDNNMatrix::create( - output_.value, outDims, inVal_->getFormat(), engine_); - - // create reorder if output value has cpu device and pd do not match - cpuOutVal_ = nullptr; - cvtOutVal_ = nullptr; - if (!outputIsOnlyMKLDNN()) { - const MatrixPtr& cpuOut = getOutput(CPU_DEVICE).value; - cpuOutVal_ = MKLDNNMatrix::create(cpuOut, outDims, format::nchw, engine_); - if (cpuOutVal_->getPrimitiveDesc() != out->getPrimitiveDesc()) { - out = MKLDNNMatrix::create(nullptr, out->getPrimitiveDesc()); - cvtOutVal_ = MKLDNNMatrix::createReorder(out, cpuOutVal_); - CHECK(cvtOutVal_) << "should not be emptry"; - } else { - cpuOut->setData(output_.value->getData()); - cpuOutVal_ = out; - } - output_.value = std::dynamic_pointer_cast(cpuOutVal_); - return; - } - output_.value = std::dynamic_pointer_cast(outVal_); + CHECK(in); + auto outPD = + MKLDNNMatrix::createPrimitiveDesc(outDims, in->getFormat(), engine_); + resetOutValue(out, outPD); } void MKLDNNPoolLayer::resetFwdPD(std::shared_ptr& pd, MKLDNNMatrixPtr in, MKLDNNMatrixPtr out) { - memory::dims inDims = memory::dims{bs_, ic_, ih_, iw_}; - memory::dims outDims = memory::dims{bs_, oc_, oh_, ow_}; memory::dims kernels = memory::dims{fh_, fw_}; memory::dims strides = memory::dims{sh_, sw_}; memory::dims padL = memory::dims{ph_, pw_}; @@ -194,58 +149,26 @@ void MKLDNNPoolLayer::resetFwdPipeline( ? std::make_shared(pool_fwd(*pd, *in, *out, *workspace_)) : std::make_shared(pool_fwd(*pd, *in, *out)); pipeline.push_back(*fwd_); - - if (cvtOutVal_) { - pipeline.push_back(*cvtOutVal_); - } } void MKLDNNPoolLayer::resetBwdBuffers(MKLDNNMatrixPtr& in, MKLDNNMatrixPtr& out) { - resetOutGrad(out); - - resetInGrad(in); -} -void MKLDNNPoolLayer::resetOutGrad(MKLDNNMatrixPtr& out) { - cpuOutGrad_ = nullptr; - cvtOutGrad_ = nullptr; - CHECK(outVal_); - if (outputIsOnlyMKLDNN()) { - MKLDNNLayer::resetOutGrad(out, outVal_->getPrimitiveDesc()); - } else { - const MatrixPtr& cpuOut = getOutput(CPU_DEVICE).grad; - // always share the same grad data of CPU output - // then the activation can get the right grad from output_.grad - output_.grad->setData(cpuOut->getData()); - cpuOutGrad_ = MKLDNNMatrix::create( - cpuOut, memory::dims{bs_, oc_, oh_, ow_}, format::nchw, engine_); - if (cpuOutGrad_->getPrimitiveDesc() != outVal_->getPrimitiveDesc()) { - out = MKLDNNMatrix::create(nullptr, outVal_->getPrimitiveDesc()); - cvtOutGrad_ = MKLDNNMatrix::createReorder(cpuOutGrad_, out); - CHECK(cvtOutGrad_) << "should not be emptry"; - } else { - out = cpuOutGrad_; - } - } -} - -void MKLDNNPoolLayer::resetInGrad(MKLDNNMatrixPtr& in) { - in = nullptr; - if (inputLayers_[0]->getOutput().grad == nullptr) { - return; - } - CHECK(inVal_); - MKLDNNLayer::resetInGrad(in, inVal_->getPrimitiveDesc()); + CHECK(inVal_ && outVal_); + resetOutGrad(out, outVal_->getPrimitiveDesc()); + resetInGrad(in, inVal_->getPrimitiveDesc()); } void MKLDNNPoolLayer::resetBwdPD(std::shared_ptr& pd, MKLDNNMatrixPtr& in, MKLDNNMatrixPtr& out) { + pd = nullptr; + if (in == nullptr) { + return; + } memory::dims kernels = memory::dims{fh_, fw_}; memory::dims strides = memory::dims{sh_, sw_}; memory::dims padL = memory::dims{ph_, pw_}; memory::dims padR = getPaddingR(); - CHECK(in); CHECK(out); auto bwdDesc = pool_bwd::desc(poolAlgo_, in->getMemoryDesc(), @@ -263,8 +186,8 @@ void MKLDNNPoolLayer::resetBwdPipeline( std::shared_ptr& pd, MKLDNNMatrixPtr& in, MKLDNNMatrixPtr& out) { - if (cvtOutGrad_) { - pipeline.push_back(*cvtOutGrad_); + if (pd == nullptr) { + return; } bwdData_ = diff --git a/paddle/gserver/layers/MKLDNNPoolLayer.h b/paddle/gserver/layers/MKLDNNPoolLayer.h index 891e15a7ef..c5ec87828b 100644 --- a/paddle/gserver/layers/MKLDNNPoolLayer.h +++ b/paddle/gserver/layers/MKLDNNPoolLayer.h @@ -38,13 +38,6 @@ protected: // pooling_avg or pooling_max mkldnn::algorithm poolAlgo_; - // MKLDNNMatrixPtr which should be created from CPU Device - MKLDNNMatrixPtr cpuOutVal_; - MKLDNNMatrixPtr cpuOutGrad_; - // convert handle between CPU device and MKLDNN device - std::shared_ptr cvtOutVal_; - std::shared_ptr cvtOutGrad_; - // save forward primitive_desc, which can be used backward std::shared_ptr fwdPD_; // according to https://github.com/01org/mkl-dnn/blob/master/tests/gtests/ @@ -74,8 +67,6 @@ public: MKLDNNMatrixPtr& bias, MKLDNNMatrixPtr& out) override; - void updateInputData() override; - void printSizeInfo() override { MKLDNNLayer::printSizeInfo(); VLOG(MKLDNN_SIZES) << getName() << ": fh: " << fh_ << ", fw: " << fw_ @@ -90,8 +81,6 @@ protected: * reset pipeline. */ void resetFwdBuffers(MKLDNNMatrixPtr& in, MKLDNNMatrixPtr& out); - void resetInValue(MKLDNNMatrixPtr& in); - void resetOutValue(MKLDNNMatrixPtr& out); void resetFwdPD(std::shared_ptr& pd, MKLDNNMatrixPtr in, MKLDNNMatrixPtr out); @@ -106,8 +95,6 @@ protected: * reset pipeline. */ void resetBwdBuffers(MKLDNNMatrixPtr& in, MKLDNNMatrixPtr& out); - void resetOutGrad(MKLDNNMatrixPtr& out); - void resetInGrad(MKLDNNMatrixPtr& in); void resetBwdPD(std::shared_ptr& pd, MKLDNNMatrixPtr& in, MKLDNNMatrixPtr& out); diff --git a/paddle/math/MKLDNNMatrix.cpp b/paddle/math/MKLDNNMatrix.cpp index 0778bb63b7..c606560473 100644 --- a/paddle/math/MKLDNNMatrix.cpp +++ b/paddle/math/MKLDNNMatrix.cpp @@ -46,7 +46,7 @@ MKLDNNMatrixPtr MKLDNNMatrix::create(MatrixPtr m, memory::format fmt, engine& eg, mkldnn::memory::data_type dtype) { - return create(m, memory::primitive_desc(memory::desc(dims, dtype, fmt), eg)); + return create(m, createPrimitiveDesc(dims, fmt, eg, dtype)); } std::shared_ptr MKLDNNMatrix::createReorder(const MKLDNNMatrixPtr& src, diff --git a/paddle/math/MKLDNNMatrix.h b/paddle/math/MKLDNNMatrix.h index c843115eb9..9e3f29eb57 100644 --- a/paddle/math/MKLDNNMatrix.h +++ b/paddle/math/MKLDNNMatrix.h @@ -52,12 +52,24 @@ public: mkldnn::engine& eg, mkldnn::memory::data_type dtype = mkldnn::memory::data_type::f32); + /** + * Create primitive descriptor. + * default with f32 dtype + */ + static mkldnn::memory::primitive_desc createPrimitiveDesc( + const mkldnn::memory::dims dims, + const mkldnn::memory::format& fmt, + const mkldnn::engine& eg, + const mkldnn::memory::data_type& dtype = mkldnn::memory::data_type::f32) { + return mkldnn::memory::primitive_desc(memory::desc(dims, dtype, fmt), eg); + } + /** * Create Memory descriptor. * default with any format and f32 dtype */ static mkldnn::memory::desc createMemoryDesc( - const mkldnn::memory::dims& dims, + const mkldnn::memory::dims dims, const mkldnn::memory::format& fmt = mkldnn::memory::format::any, const mkldnn::memory::data_type& dtype = mkldnn::memory::data_type::f32) { return mkldnn::memory::desc(dims, dtype, fmt); From 9e38dafa29acc59347a5aee33424be7bb8bcd168 Mon Sep 17 00:00:00 2001 From: tensor-tang Date: Thu, 19 Oct 2017 15:18:34 +0800 Subject: [PATCH 35/76] change MKLDNNMatrix create interface since MatrixPtr is not always required --- .../gserver/activations/MKLDNNActivation.cpp | 6 ++-- paddle/gserver/layers/MKLDNNConvLayer.cpp | 3 +- paddle/gserver/layers/MKLDNNLayer.h | 32 +++++++++---------- paddle/math/MKLDNNMatrix.cpp | 8 ++--- paddle/math/MKLDNNMatrix.h | 5 +-- 5 files changed, 27 insertions(+), 27 deletions(-) diff --git a/paddle/gserver/activations/MKLDNNActivation.cpp b/paddle/gserver/activations/MKLDNNActivation.cpp index 18c5638100..f3ccd68160 100644 --- a/paddle/gserver/activations/MKLDNNActivation.cpp +++ b/paddle/gserver/activations/MKLDNNActivation.cpp @@ -126,7 +126,7 @@ void MKLDNNEltwiseActivation::resetFwd(Argument& act) { copyInVal_ = nullptr; if (act.grad && algo == algorithm::eltwise_tanh) { // tanh need save src input for backward - inVal_ = MKLDNNMatrix::create(nullptr, val_->getPrimitiveDesc()); + inVal_ = MKLDNNMatrix::create(val_->getPrimitiveDesc()); copyInVal_ = std::make_shared(*val_, *inVal_); CHECK(copyInVal_) << "should not be emptry"; pipelineFwd_.push_back(*copyInVal_); @@ -145,7 +145,7 @@ void MKLDNNEltwiseActivation::resetBwd(Argument& act) { algorithm algo = getAlgo(this->getName()); float alpha = getBwdAlpha(); float beta = getBeta(); - grad_ = MKLDNNMatrix::create(act.grad, val_->getPrimitiveDesc()); + grad_ = MKLDNNMatrix::create(val_->getPrimitiveDesc(), act.grad); auto eng = CPUEngine::Instance().getEngine(); auto bwdDesc = eltwise_bwd::desc( algo, grad_->getMemoryDesc(), val_->getMemoryDesc(), alpha, beta); @@ -230,7 +230,7 @@ void MKLDNNActivation::resetFwd(Argument& act) { int ic = cnt_ / bs / ih / iw; CHECK_EQ(cnt_, (size_t)bs * ic * ih * iw); val_ = MKLDNNMatrix::create( - act.value, {bs, ic, ih, iw}, mkldnn::memory::format::nchw, *engine_); + {bs, ic, ih, iw}, mkldnn::memory::format::nchw, *engine_, act.value); CHECK(val_); val_->downSpatial(); } diff --git a/paddle/gserver/layers/MKLDNNConvLayer.cpp b/paddle/gserver/layers/MKLDNNConvLayer.cpp index 463e6ad0ed..3fbfb1ab1f 100644 --- a/paddle/gserver/layers/MKLDNNConvLayer.cpp +++ b/paddle/gserver/layers/MKLDNNConvLayer.cpp @@ -370,8 +370,7 @@ void MKLDNNConvLayer::resetWgtValBwdData( // since the primitive_desc would be different with wgtVal_ CHECK(wgtVal_) << "should have weight value"; if (dataPD->weights_primitive_desc() != wgtVal_->getPrimitiveDesc()) { - wgtValBwdData_ = - MKLDNNMatrix::create(nullptr, dataPD->weights_primitive_desc()); + wgtValBwdData_ = MKLDNNMatrix::create(dataPD->weights_primitive_desc()); cvtWgtVal_ = MKLDNNMatrix::createReorder(wgtVal_, wgtValBwdData_); CHECK(cvtWgtVal_); } else { diff --git a/paddle/gserver/layers/MKLDNNLayer.h b/paddle/gserver/layers/MKLDNNLayer.h index ab59357ad0..80c67529da 100644 --- a/paddle/gserver/layers/MKLDNNLayer.h +++ b/paddle/gserver/layers/MKLDNNLayer.h @@ -323,7 +323,7 @@ protected: if (mat == nullptr) { return; } - dnn = MKLDNNMatrix::create(mat, pd); + dnn = MKLDNNMatrix::create(pd, mat); } /** @@ -343,7 +343,7 @@ protected: in = std::dynamic_pointer_cast(inMat); CHECK_EQ(inputIsOnlyMKLDNN(), in != nullptr); if (in == nullptr || in->getFormat() == mkldnn::memory::format::nc) { - in = MKLDNNMatrix::create(inMat, extPD); + in = MKLDNNMatrix::create(extPD, inMat); } extInVal_ = isPaddleFormat(in->getFormat()) ? in : nullptr; if (in->getFormat() == mkldnn::memory::format::nc) { @@ -353,8 +353,8 @@ protected: return; } // need create reorder - in = MKLDNNMatrix::create(nullptr, *intPD); - extInVal_ = extInVal_ ? extInVal_ : MKLDNNMatrix::create(inMat, extPD); + in = MKLDNNMatrix::create(*intPD); + extInVal_ = extInVal_ ? extInVal_ : MKLDNNMatrix::create(extPD, inMat); cvtInVal_ = MKLDNNMatrix::createReorder(extInVal_, in); CHECK(cvtInVal_) << "should not be emptry"; } @@ -366,18 +366,18 @@ protected: void resetOutValue(MKLDNNMatrixPtr& out, mkldnn::memory::primitive_desc intPD) { cvtOutVal_ = nullptr; - out = MKLDNNMatrix::create(output_.value, intPD); + out = MKLDNNMatrix::create(intPD, output_.value); extOutVal_ = out; if (outputIsOnlyMKLDNN() || isPaddleFormat(extOutVal_->getFormat())) { return; } // need create reorder CHECK_GT(bs_ * oc_ * oh_ * ow_, 0); - extOutVal_ = MKLDNNMatrix::create(output_.value, - {bs_, oc_, oh_, ow_}, + extOutVal_ = MKLDNNMatrix::create(mkldnn::memory::dims{bs_, oc_, oh_, ow_}, mkldnn::memory::format::nchw, - engine_); - out = MKLDNNMatrix::create(nullptr, intPD); + engine_, + output_.value); + out = MKLDNNMatrix::create(intPD); cvtOutVal_ = MKLDNNMatrix::createReorder(out, extOutVal_); CHECK(cvtOutVal_) << "should not be empty"; } @@ -402,7 +402,7 @@ protected: // and the mkldnn input layer will merge them to actual prev->output_.grad const MatrixPtr& inMat = input->getOutputMapSize() <= 1 ? input->getOutputGrad() : nullptr; - in = MKLDNNMatrix::create(inMat, intPD); + in = MKLDNNMatrix::create(intPD, inMat); Argument& arg = input->getOutput(this->getName()); arg.grad = std::dynamic_pointer_cast(in); CHECK(inVal_ != nullptr && inVal_->getPrimitiveDesc() == intPD) @@ -418,10 +418,10 @@ protected: // need create reorder CHECK(extInVal_ != nullptr && isPaddleFormat(extInVal_->getFormat())) << "should have external input value and the format must be nchw(nc)"; - extInGrad_ = MKLDNNMatrix::create(inMat, extInVal_->getPrimitiveDesc()); + extInGrad_ = MKLDNNMatrix::create(extInVal_->getPrimitiveDesc(), inMat); CHECK(inVal_ != nullptr && inVal_->getPrimitiveDesc() == intPD) << "should have internal input value and primitive desc must equal"; - in = MKLDNNMatrix::create(nullptr, intPD); + in = MKLDNNMatrix::create(intPD); cvtInGrad_ = MKLDNNMatrix::createReorder(in, extInGrad_); CHECK(cvtInGrad_); } @@ -440,7 +440,7 @@ protected: extOutGrad_ = nullptr; out = nullptr; MatrixPtr& outMat = output_.grad; - out = MKLDNNMatrix::create(outMat, intPD); + out = MKLDNNMatrix::create(intPD, outMat); resetMergeGrad(out); if (outputIsOnlyMKLDNN()) { return; @@ -453,10 +453,10 @@ protected: // need create reorder CHECK(extOutVal_ != nullptr && isPaddleFormat(extOutVal_->getFormat())) << "should have external output value and the format must be nchw(nc)"; - extOutGrad_ = MKLDNNMatrix::create(outMat, extOutVal_->getPrimitiveDesc()); + extOutGrad_ = MKLDNNMatrix::create(extOutVal_->getPrimitiveDesc(), outMat); CHECK(outVal_ != nullptr && outVal_->getPrimitiveDesc() == intPD) << "should have internal output value and primitive desc must equal"; - out = MKLDNNMatrix::create(nullptr, intPD); + out = MKLDNNMatrix::create(intPD); cvtOutGrad_ = MKLDNNMatrix::createReorder(extOutGrad_, out); CHECK(cvtOutGrad_); } @@ -499,7 +499,7 @@ protected: tmpOutGrad_ = out; tmpCvt_ = nullptr; if (out->getPrimitiveDesc() != srcPDs[0]) { - tmpOutGrad_ = MKLDNNMatrix::create(nullptr, srcPDs[0]); + tmpOutGrad_ = MKLDNNMatrix::create(srcPDs[0]); tmpCvt_ = MKLDNNMatrix::createReorder(tmpOutGrad_, out); CHECK(tmpCvt_); pipelineMergeGrad_.push_back(*tmpCvt_); diff --git a/paddle/math/MKLDNNMatrix.cpp b/paddle/math/MKLDNNMatrix.cpp index c606560473..21a8f73c3e 100644 --- a/paddle/math/MKLDNNMatrix.cpp +++ b/paddle/math/MKLDNNMatrix.cpp @@ -18,7 +18,7 @@ using namespace mkldnn; // NOLINT namespace paddle { -MKLDNNMatrixPtr MKLDNNMatrix::create(MatrixPtr m, memory::primitive_desc pd) { +MKLDNNMatrixPtr MKLDNNMatrix::create(memory::primitive_desc pd, MatrixPtr m) { memory::desc md = pd.desc(); size_t ndims = md.data.ndims; int* dims = md.data.dims; @@ -41,12 +41,12 @@ MKLDNNMatrixPtr MKLDNNMatrix::create(MatrixPtr m, memory::primitive_desc pd) { return std::make_shared(cpuMatrix, pd); } -MKLDNNMatrixPtr MKLDNNMatrix::create(MatrixPtr m, - memory::dims dims, +MKLDNNMatrixPtr MKLDNNMatrix::create(memory::dims dims, memory::format fmt, engine& eg, + MatrixPtr m, mkldnn::memory::data_type dtype) { - return create(m, createPrimitiveDesc(dims, fmt, eg, dtype)); + return create(createPrimitiveDesc(dims, fmt, eg, dtype), m); } std::shared_ptr MKLDNNMatrix::createReorder(const MKLDNNMatrixPtr& src, diff --git a/paddle/math/MKLDNNMatrix.h b/paddle/math/MKLDNNMatrix.h index 9e3f29eb57..fe755d096d 100644 --- a/paddle/math/MKLDNNMatrix.h +++ b/paddle/math/MKLDNNMatrix.h @@ -40,16 +40,17 @@ public: /** * Create MKLDNNMatrix from a MatrixPtr and memory primitive_desc */ - static MKLDNNMatrixPtr create(MatrixPtr m, mkldnn::memory::primitive_desc pd); + static MKLDNNMatrixPtr create(mkldnn::memory::primitive_desc pd, + MatrixPtr m = nullptr); /** * Create MKLDNNMatrix from a MatrixPtr and memory details info */ static MKLDNNMatrixPtr create( - MatrixPtr m, mkldnn::memory::dims dims, mkldnn::memory::format fmt, mkldnn::engine& eg, + MatrixPtr m = nullptr, mkldnn::memory::data_type dtype = mkldnn::memory::data_type::f32); /** From 94e442d4b14c66ba68d8e64c0f51f5bc849437dd Mon Sep 17 00:00:00 2001 From: tensor-tang Date: Thu, 19 Oct 2017 16:32:11 +0800 Subject: [PATCH 36/76] add cpp file of MKLDNNLayer --- paddle/gserver/layers/MKLDNNLayer.cpp | 327 ++++++++++++++++++++++ paddle/gserver/layers/MKLDNNLayer.h | 386 ++++---------------------- 2 files changed, 379 insertions(+), 334 deletions(-) create mode 100644 paddle/gserver/layers/MKLDNNLayer.cpp diff --git a/paddle/gserver/layers/MKLDNNLayer.cpp b/paddle/gserver/layers/MKLDNNLayer.cpp new file mode 100644 index 0000000000..91f0ff5bd3 --- /dev/null +++ b/paddle/gserver/layers/MKLDNNLayer.cpp @@ -0,0 +1,327 @@ +/* Copyright (c) 2017 PaddlePaddle Authors. All Rights Reserve. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. */ + +#include "MKLDNNLayer.h" + +using namespace mkldnn; // NOLINT +typedef memory::format format; + +namespace paddle { + +bool MKLDNNLayer::init(const LayerMap& layerMap, + const ParameterMap& parameterMap) { + CHECK(FLAGS_use_mkldnn) << "MkldnnLayers only support use_mkldnn." + << "Please set WITH_MKLDNN=ON " + << "and set use_mkldnn=True"; + CHECK(!useGpu_) << "Do not support GPU yet"; + + // set device id before Layer::init + setDevice(MKLDNN_DEVICE); + // change param device to MKLDNN device + setParamsDevice(MKLDNN_DEVICE, parameterMap); + if (!Layer::init(layerMap, parameterMap)) { + return false; + } + setOutputMap(); + checkCPUOutputsNumber(); + + stream_.reset(new MKLDNNStream()); + engine_ = CPUEngine::Instance().getEngine(); + return true; +} + +void MKLDNNLayer::forward(PassType passType) { + passType_ = passType; + + { + REGISTER_TIMER_INFO("mkldnn_FwdTimer", getName().c_str()); + CHECK(!inputLayers_.empty()); + copySeqInfoToOutputs(); + size_t elemenCnt = inputLayers_[0]->getOutputValue()->getElementCnt(); + if (inputElemenCnt_ != elemenCnt) { + VLOG(MKLDNN_BASE) << getName() << " reset mkldnn forward"; + // reset when input total sizes changed, not only the batchsize + inputElemenCnt_ = elemenCnt; + pipelineFwd_.clear(); + reshape(bs_, ic_, ih_, iw_, oc_, oh_, ow_); + // all cpu device output grad or value share output's + shareCPUDevice(); + resetFwd(pipelineFwd_, inVal_, wgtVal_, biasVal_, outVal_); + // MKLDNNLayer output value should be MKLDNNMatrix + // so external output value is necessary. + // then external input value is not necessary, + // since input may be mkldnn internal buffer. + CHECK(extOutVal_) << "external output value is necessary"; + output_.value = std::dynamic_pointer_cast(extOutVal_); + CHECK(inVal_ && outVal_) << "internal memories are necessary"; + if (cvtInVal_) { + pipelineFwd_.insert(pipelineFwd_.begin(), *cvtInVal_); + } + if (cvtOutVal_) { + pipelineFwd_.push_back(*cvtOutVal_); + } + convertWeightsFromPaddle(); + printSizeInfo(); + printValueFormat(); + needResetBwd_ = true; + } + + if (inputLayers_[0]->getType() == "data") { + // Update input value data when input layer is "data" type, + // since the input value data address might be changed. + CHECK(extInVal_); + extInVal_->setData(getInputValue(0, CPU_DEVICE)->getData()); + } + + if (!outputOnlyMKLDNN_) { + clearGrads(); + } + stream_->submit(pipelineFwd_); + } + { + REGISTER_TIMER_INFO("FwActTimer", getName().c_str()); + forwardActivation(); + } +} + +void MKLDNNLayer::backward(const UpdateCallback& callback) { + if (needResetBwd_) { + VLOG(MKLDNN_BASE) << getName() << " reset mkldnn backward"; + pipelineBwd_.clear(); + pipelineMergeGrad_.clear(); + mergeGrad_ = nullptr; + resetBwd(pipelineBwd_, inGrad_, wgtGrad_, biasGrad_, outGrad_); + // external output grad is not necessary + // since output may be mkldnn internal buffer or merge them directly. + CHECK(outGrad_) << "internal output grad is necessary"; + if (cvtOutGrad_) { + pipelineBwd_.insert(pipelineBwd_.begin(), *cvtOutGrad_); + } + if (cvtInGrad_) { + pipelineBwd_.push_back(*cvtInGrad_); + } + printGradFormat(); + needResetBwd_ = false; + } + + // merge grad must before backward activation + if (mergeGrad_) { + REGISTER_TIMER_INFO("MergeBpGrad", getName().c_str()); + stream_->submit(pipelineMergeGrad_); + } + { + REGISTER_TIMER_INFO("BpActTimer", getName().c_str()); + backwardActivation(); + } + { + REGISTER_TIMER_INFO("mkldnn_bwdTimer", getName().c_str()); + stream_->submit(pipelineBwd_); + } + { + REGISTER_TIMER_INFO("WeightUpdate", getName().c_str()); + updateWeights(callback); + } +} + +void MKLDNNLayer::reshapeInput(int& batchsize, int& height, int& width) { + const Argument& input = inputLayers_[0]->getOutput(); + batchsize = input.getBatchSize(); + int h = input.getFrameHeight(); + int w = input.getFrameWidth(); + if (h != 0) { + height = h; + } + if (w != 0) { + width = w; + } +} + +void MKLDNNLayer::reshapeOutput(size_t height, size_t width) { + output_.setFrameHeight(height); + output_.setFrameWidth(width); + for (size_t i = 0; i < outputOtherDevice_.size(); i++) { + outputOtherDevice_[i].setFrameHeight(height); + outputOtherDevice_[i].setFrameWidth(width); + } +} + +void MKLDNNLayer::resetWithMatrix(MKLDNNMatrixPtr& dnn, + const MatrixPtr& mat, + memory::primitive_desc pd) { + dnn = nullptr; + if (mat == nullptr) { + return; + } + dnn = MKLDNNMatrix::create(pd, mat); +} + +void MKLDNNLayer::resetInValue( + MKLDNNMatrixPtr& in, const std::shared_ptr& intPD) { + cvtInVal_ = nullptr; + extInVal_ = nullptr; + in = nullptr; + CHECK_GT(bs_ * ic_ * ih_ * iw_, 0); + auto extPD = MKLDNNMatrix::createPrimitiveDesc( + {bs_, ic_, ih_, iw_}, format::nchw, engine_); + const MatrixPtr& inMat = inputLayers_[0]->getOutputValue(); + in = std::dynamic_pointer_cast(inMat); + CHECK_EQ(inputIsOnlyMKLDNN(), in != nullptr); + if (in == nullptr || in->getFormat() == format::nc) { + in = MKLDNNMatrix::create(extPD, inMat); + } + extInVal_ = isPaddleFormat(in->getFormat()) ? in : nullptr; + if (in->getFormat() == format::nc) { + CHECK(ih_ == 1 && iw_ == 1); + } + if (nullptr == intPD || in->getPrimitiveDesc() == *intPD) { + return; + } + // need create reorder + in = MKLDNNMatrix::create(*intPD); + extInVal_ = extInVal_ ? extInVal_ : MKLDNNMatrix::create(extPD, inMat); + cvtInVal_ = MKLDNNMatrix::createReorder(extInVal_, in); + CHECK(cvtInVal_) << "should not be emptry"; +} + +void MKLDNNLayer::resetOutValue(MKLDNNMatrixPtr& out, + memory::primitive_desc intPD) { + cvtOutVal_ = nullptr; + out = MKLDNNMatrix::create(intPD, output_.value); + extOutVal_ = out; + if (outputIsOnlyMKLDNN() || isPaddleFormat(extOutVal_->getFormat())) { + return; + } + // need create reorder + CHECK_GT(bs_ * oc_ * oh_ * ow_, 0); + extOutVal_ = MKLDNNMatrix::create( + memory::dims{bs_, oc_, oh_, ow_}, format::nchw, engine_, output_.value); + out = MKLDNNMatrix::create(intPD); + cvtOutVal_ = MKLDNNMatrix::createReorder(out, extOutVal_); + CHECK(cvtOutVal_) << "should not be empty"; +} + +void MKLDNNLayer::resetInGrad(MKLDNNMatrixPtr& in, + memory::primitive_desc intPD) { + cvtInGrad_ = nullptr; + extInGrad_ = nullptr; + in = nullptr; + LayerPtr& input = inputLayers_[0]; + if (input->getOutputGrad() == nullptr) { + // no need input grad + return; + } + CHECK(inputIsOnlyMKLDNN() || input->getOutputMapSize() <= 1) + << "only support input is MKLDNN layer or only have one output layer"; + // when input is a mkldnn branch node, + // this layer will save input grad to a internal buffer, + // and the mkldnn input layer will merge them to actual prev->output_.grad + const MatrixPtr& inMat = + input->getOutputMapSize() <= 1 ? input->getOutputGrad() : nullptr; + in = MKLDNNMatrix::create(intPD, inMat); + Argument& arg = input->getOutput(this->getName()); + arg.grad = std::dynamic_pointer_cast(in); + CHECK(inVal_ != nullptr && inVal_->getPrimitiveDesc() == intPD) + << "should have internal input value and primitive desc must equal"; + if (inputIsOnlyMKLDNN()) { + return; + } + + extInGrad_ = in; + if (isPaddleFormat(extInGrad_->getFormat())) { + return; + } + // need create reorder + CHECK(extInVal_ != nullptr && isPaddleFormat(extInVal_->getFormat())) + << "should have external input value and the format must be nchw(nc)"; + extInGrad_ = MKLDNNMatrix::create(extInVal_->getPrimitiveDesc(), inMat); + CHECK(inVal_ != nullptr && inVal_->getPrimitiveDesc() == intPD) + << "should have internal input value and primitive desc must equal"; + in = MKLDNNMatrix::create(intPD); + cvtInGrad_ = MKLDNNMatrix::createReorder(in, extInGrad_); + CHECK(cvtInGrad_); +} + +void MKLDNNLayer::resetOutGrad(MKLDNNMatrixPtr& out, + memory::primitive_desc intPD) { + cvtOutGrad_ = nullptr; + extOutGrad_ = nullptr; + out = nullptr; + MatrixPtr& outMat = output_.grad; + out = MKLDNNMatrix::create(intPD, outMat); + resetMergeGrad(out); + if (outputIsOnlyMKLDNN()) { + return; + } + CHECK_LE(outputMap_.size(), 1U) << "do not support mixed with cpu device"; + extOutGrad_ = out; + if (isPaddleFormat(extOutGrad_->getFormat())) { + return; + } + // need create reorder + CHECK(extOutVal_ != nullptr && isPaddleFormat(extOutVal_->getFormat())) + << "should have external output value and the format must be nchw(nc)"; + extOutGrad_ = MKLDNNMatrix::create(extOutVal_->getPrimitiveDesc(), outMat); + CHECK(outVal_ != nullptr && outVal_->getPrimitiveDesc() == intPD) + << "should have internal output value and primitive desc must equal"; + out = MKLDNNMatrix::create(intPD); + cvtOutGrad_ = MKLDNNMatrix::createReorder(extOutGrad_, out); + CHECK(cvtOutGrad_); +} + +void MKLDNNLayer::resetMergeGrad(MKLDNNMatrixPtr& out) { + mergeGrad_ = nullptr; + pipelineMergeGrad_.clear(); + if (outputMap_.size() <= 1 || !outputIsOnlyMKLDNN()) { + // do not merge when output is not all MKLDNN or only one output + return; + } + CHECK(out) << "should have reset internal ouput grad"; + std::vector scales(outputMap_.size(), 1.0); + std::vector srcPDs; + std::vector srcs; + for (auto it = outputMap_.begin(); it != outputMap_.end(); ++it) { + MKLDNNMatrixPtr src = + std::dynamic_pointer_cast(it->second->grad); + VLOG(MKLDNN_BASE) << getName() << " has output grad " << it->first; + CHECK(src) << "should be MKLDNNMatrix"; + auto srcDims = src->getDims(); + auto dstDims = out->getDims(); + CHECK_EQ(srcDims.size(), dstDims.size()); + for (size_t i = 0; i < srcDims.size(); ++i) { + CHECK_EQ(srcDims[i], dstDims[i]); + } + srcPDs.push_back(src->getPrimitiveDesc()); + srcs.push_back(*src); + } + + // TODO(TJ): remove me when mkldnn sum support different formats + for (size_t i = 1; i < srcPDs.size(); ++i) { + CHECK(srcPDs[0] == srcPDs[i]); + } + tmpOutGrad_ = out; + tmpCvt_ = nullptr; + if (out->getPrimitiveDesc() != srcPDs[0]) { + tmpOutGrad_ = MKLDNNMatrix::create(srcPDs[0]); + tmpCvt_ = MKLDNNMatrix::createReorder(tmpOutGrad_, out); + CHECK(tmpCvt_); + pipelineMergeGrad_.push_back(*tmpCvt_); + } + + auto sumPD = + sum::primitive_desc(tmpOutGrad_->getMemoryDesc(), scales, srcPDs); + mergeGrad_.reset(new sum(sumPD, srcs, *tmpOutGrad_)); + pipelineMergeGrad_.insert(pipelineMergeGrad_.begin(), *mergeGrad_); +} + +} // namespace paddle diff --git a/paddle/gserver/layers/MKLDNNLayer.h b/paddle/gserver/layers/MKLDNNLayer.h index 80c67529da..faad434526 100644 --- a/paddle/gserver/layers/MKLDNNLayer.h +++ b/paddle/gserver/layers/MKLDNNLayer.h @@ -119,119 +119,9 @@ public: ~MKLDNNLayer() {} - virtual bool init(const LayerMap& layerMap, - const ParameterMap& parameterMap) { - CHECK(FLAGS_use_mkldnn) << "MkldnnLayers only support use_mkldnn." - << "Please set WITH_MKLDNN=ON " - << "and set use_mkldnn=True"; - CHECK(!useGpu_) << "Do not support GPU yet"; - - // set device id before Layer::init - setDevice(MKLDNN_DEVICE); - // change param device to MKLDNN device - setParamsDevice(MKLDNN_DEVICE, parameterMap); - if (!Layer::init(layerMap, parameterMap)) { - return false; - } - setOutputMap(); - checkCPUOutputsNumber(); - - stream_.reset(new MKLDNNStream()); - engine_ = CPUEngine::Instance().getEngine(); - return true; - } - - void forward(PassType passType) override { - passType_ = passType; - - { - REGISTER_TIMER_INFO("mkldnn_FwdTimer", getName().c_str()); - CHECK(!inputLayers_.empty()); - copySeqInfoToOutputs(); - size_t elemenCnt = inputLayers_[0]->getOutputValue()->getElementCnt(); - if (inputElemenCnt_ != elemenCnt) { - VLOG(MKLDNN_BASE) << getName() << " reset mkldnn forward"; - // reset when input total sizes changed, not only the batchsize - inputElemenCnt_ = elemenCnt; - pipelineFwd_.clear(); - reshape(bs_, ic_, ih_, iw_, oc_, oh_, ow_); - // all cpu device output grad or value share output's - shareCPUDevice(); - resetFwd(pipelineFwd_, inVal_, wgtVal_, biasVal_, outVal_); - // MKLDNNLayer output value should be MKLDNNMatrix - // so external output value is necessary. - // then external input value is not necessary, - // since input may be mkldnn internal buffer. - CHECK(extOutVal_) << "external output value is necessary"; - output_.value = std::dynamic_pointer_cast(extOutVal_); - CHECK(inVal_ && outVal_) << "internal memories are necessary"; - if (cvtInVal_) { - pipelineFwd_.insert(pipelineFwd_.begin(), *cvtInVal_); - } - if (cvtOutVal_) { - pipelineFwd_.push_back(*cvtOutVal_); - } - convertWeightsFromPaddle(); - printValueFormat(); - needResetBwd_ = true; - } - - if (inputLayers_[0]->getType() == "data") { - // Update input value data when input layer is "data" type, - // since the input value data address might be changed. - CHECK(extInVal_); - extInVal_->setData(getInputValue(0, CPU_DEVICE)->getData()); - } - - if (!outputOnlyMKLDNN_) { - clearGrads(); - } - stream_->submit(pipelineFwd_); - } - { - REGISTER_TIMER_INFO("FwActTimer", getName().c_str()); - forwardActivation(); - } - } - - void backward(const UpdateCallback& callback) override { - if (needResetBwd_) { - VLOG(MKLDNN_BASE) << getName() << " reset mkldnn backward"; - pipelineBwd_.clear(); - pipelineMergeGrad_.clear(); - mergeGrad_ = nullptr; - resetBwd(pipelineBwd_, inGrad_, wgtGrad_, biasGrad_, outGrad_); - // external output grad is not necessary - // since output may be mkldnn internal buffer or merge them directly. - CHECK(outGrad_) << "internal output grad is necessary"; - if (cvtOutGrad_) { - pipelineBwd_.insert(pipelineBwd_.begin(), *cvtOutGrad_); - } - if (cvtInGrad_) { - pipelineBwd_.push_back(*cvtInGrad_); - } - printGradFormat(); - needResetBwd_ = false; - } - - // merge grad must before backward activation - if (mergeGrad_) { - REGISTER_TIMER_INFO("MergeBpGrad", getName().c_str()); - stream_->submit(pipelineMergeGrad_); - } - { - REGISTER_TIMER_INFO("BpActTimer", getName().c_str()); - backwardActivation(); - } - { - REGISTER_TIMER_INFO("mkldnn_bwdTimer", getName().c_str()); - stream_->submit(pipelineBwd_); - } - { - REGISTER_TIMER_INFO("WeightUpdate", getName().c_str()); - updateWeights(callback); - } - } + virtual bool init(const LayerMap& layerMap, const ParameterMap& parameterMap); + void forward(PassType passType) override; + void backward(const UpdateCallback& callback) override; /** * reshape the input image sizes @@ -287,30 +177,12 @@ protected: /** * reshape the input image sizes and input batchsize */ - virtual void reshapeInput(int& batchsize, int& height, int& width) { - const Argument& input = inputLayers_[0]->getOutput(); - batchsize = input.getBatchSize(); - int h = input.getFrameHeight(); - int w = input.getFrameWidth(); - if (h != 0) { - height = h; - } - if (w != 0) { - width = w; - } - } + void reshapeInput(int& batchsize, int& height, int& width); /** * reshape output image sizes */ - virtual void reshapeOutput(size_t height, size_t width) { - output_.setFrameHeight(height); - output_.setFrameWidth(width); - for (size_t i = 0; i < outputOtherDevice_.size(); i++) { - outputOtherDevice_[i].setFrameHeight(height); - outputOtherDevice_[i].setFrameWidth(width); - } - } + void reshapeOutput(size_t height, size_t width); /** * reset MKLDNNMatrix from Matrix and internal primitive desc. @@ -318,13 +190,7 @@ protected: */ void resetWithMatrix(MKLDNNMatrixPtr& dnn, const MatrixPtr& mat, - mkldnn::memory::primitive_desc pd) { - dnn = nullptr; - if (mat == nullptr) { - return; - } - dnn = MKLDNNMatrix::create(pd, mat); - } + mkldnn::memory::primitive_desc pd); /** * reset input value from input MKLDNNMatrix and internal primitive desc. @@ -332,99 +198,20 @@ protected: */ void resetInValue( MKLDNNMatrixPtr& in, - const std::shared_ptr& intPD = nullptr) { - cvtInVal_ = nullptr; - extInVal_ = nullptr; - in = nullptr; - CHECK_GT(bs_ * ic_ * ih_ * iw_, 0); - auto extPD = MKLDNNMatrix::createPrimitiveDesc( - {bs_, ic_, ih_, iw_}, mkldnn::memory::format::nchw, engine_); - const MatrixPtr& inMat = inputLayers_[0]->getOutputValue(); - in = std::dynamic_pointer_cast(inMat); - CHECK_EQ(inputIsOnlyMKLDNN(), in != nullptr); - if (in == nullptr || in->getFormat() == mkldnn::memory::format::nc) { - in = MKLDNNMatrix::create(extPD, inMat); - } - extInVal_ = isPaddleFormat(in->getFormat()) ? in : nullptr; - if (in->getFormat() == mkldnn::memory::format::nc) { - CHECK(ih_ == 1 && iw_ == 1); - } - if (nullptr == intPD || in->getPrimitiveDesc() == *intPD) { - return; - } - // need create reorder - in = MKLDNNMatrix::create(*intPD); - extInVal_ = extInVal_ ? extInVal_ : MKLDNNMatrix::create(extPD, inMat); - cvtInVal_ = MKLDNNMatrix::createReorder(extInVal_, in); - CHECK(cvtInVal_) << "should not be emptry"; - } + const std::shared_ptr& intPD = nullptr); /** * reset output value from internal primitive desc. * reset both internal and external buffer and create reorder if necessary. */ void resetOutValue(MKLDNNMatrixPtr& out, - mkldnn::memory::primitive_desc intPD) { - cvtOutVal_ = nullptr; - out = MKLDNNMatrix::create(intPD, output_.value); - extOutVal_ = out; - if (outputIsOnlyMKLDNN() || isPaddleFormat(extOutVal_->getFormat())) { - return; - } - // need create reorder - CHECK_GT(bs_ * oc_ * oh_ * ow_, 0); - extOutVal_ = MKLDNNMatrix::create(mkldnn::memory::dims{bs_, oc_, oh_, ow_}, - mkldnn::memory::format::nchw, - engine_, - output_.value); - out = MKLDNNMatrix::create(intPD); - cvtOutVal_ = MKLDNNMatrix::createReorder(out, extOutVal_); - CHECK(cvtOutVal_) << "should not be empty"; - } + mkldnn::memory::primitive_desc intPD); /** * reset input grad from internal primitive desc. * reset both internal and external buffer and create reorder if necessary. */ - void resetInGrad(MKLDNNMatrixPtr& in, mkldnn::memory::primitive_desc intPD) { - cvtInGrad_ = nullptr; - extInGrad_ = nullptr; - in = nullptr; - LayerPtr& input = inputLayers_[0]; - if (input->getOutputGrad() == nullptr) { - // no need input grad - return; - } - CHECK(inputIsOnlyMKLDNN() || input->getOutputMapSize() <= 1) - << "only support input is MKLDNN layer or only have one output layer"; - // when input is a mkldnn branch node, - // this layer will save input grad to a internal buffer, - // and the mkldnn input layer will merge them to actual prev->output_.grad - const MatrixPtr& inMat = - input->getOutputMapSize() <= 1 ? input->getOutputGrad() : nullptr; - in = MKLDNNMatrix::create(intPD, inMat); - Argument& arg = input->getOutput(this->getName()); - arg.grad = std::dynamic_pointer_cast(in); - CHECK(inVal_ != nullptr && inVal_->getPrimitiveDesc() == intPD) - << "should have internal input value and primitive desc must equal"; - if (inputIsOnlyMKLDNN()) { - return; - } - - extInGrad_ = in; - if (isPaddleFormat(extInGrad_->getFormat())) { - return; - } - // need create reorder - CHECK(extInVal_ != nullptr && isPaddleFormat(extInVal_->getFormat())) - << "should have external input value and the format must be nchw(nc)"; - extInGrad_ = MKLDNNMatrix::create(extInVal_->getPrimitiveDesc(), inMat); - CHECK(inVal_ != nullptr && inVal_->getPrimitiveDesc() == intPD) - << "should have internal input value and primitive desc must equal"; - in = MKLDNNMatrix::create(intPD); - cvtInGrad_ = MKLDNNMatrix::createReorder(in, extInGrad_); - CHECK(cvtInGrad_); - } + void resetInGrad(MKLDNNMatrixPtr& in, mkldnn::memory::primitive_desc intPD); /** * reset output grad from internal primitive desc. @@ -434,81 +221,59 @@ protected: * it could not be mixed with cpu device, * since it can not get memory desc from cpu device. */ - void resetOutGrad(MKLDNNMatrixPtr& out, - mkldnn::memory::primitive_desc intPD) { - cvtOutGrad_ = nullptr; - extOutGrad_ = nullptr; - out = nullptr; - MatrixPtr& outMat = output_.grad; - out = MKLDNNMatrix::create(intPD, outMat); - resetMergeGrad(out); - if (outputIsOnlyMKLDNN()) { - return; - } - CHECK_LE(outputMap_.size(), 1U) << "do not support mixed with cpu device"; - extOutGrad_ = out; - if (isPaddleFormat(extOutGrad_->getFormat())) { - return; - } - // need create reorder - CHECK(extOutVal_ != nullptr && isPaddleFormat(extOutVal_->getFormat())) - << "should have external output value and the format must be nchw(nc)"; - extOutGrad_ = MKLDNNMatrix::create(extOutVal_->getPrimitiveDesc(), outMat); - CHECK(outVal_ != nullptr && outVal_->getPrimitiveDesc() == intPD) - << "should have internal output value and primitive desc must equal"; - out = MKLDNNMatrix::create(intPD); - cvtOutGrad_ = MKLDNNMatrix::createReorder(extOutGrad_, out); - CHECK(cvtOutGrad_); - } + void resetOutGrad(MKLDNNMatrixPtr& out, mkldnn::memory::primitive_desc intPD); /** * reset the merge grad primitive if necessary. * note: do not support the grads are mixed with cpu device, * since it can not get memory desc from cpu device. */ - virtual void resetMergeGrad(MKLDNNMatrixPtr& out) { - mergeGrad_ = nullptr; - pipelineMergeGrad_.clear(); - if (outputMap_.size() <= 1 || !outputIsOnlyMKLDNN()) { - // do not merge when output is not all MKLDNN or only one output - return; - } - CHECK(out) << "should have reset internal ouput grad"; - std::vector scales(outputMap_.size(), 1.0); - std::vector srcPDs; - std::vector srcs; - for (auto it = outputMap_.begin(); it != outputMap_.end(); ++it) { - MKLDNNMatrixPtr src = - std::dynamic_pointer_cast(it->second->grad); - VLOG(MKLDNN_BASE) << getName() << " has output grad " << it->first; - CHECK(src) << "should be MKLDNNMatrix"; - auto srcDims = src->getDims(); - auto dstDims = out->getDims(); - CHECK_EQ(srcDims.size(), dstDims.size()); - for (size_t i = 0; i < srcDims.size(); ++i) { - CHECK_EQ(srcDims[i], dstDims[i]); - } - srcPDs.push_back(src->getPrimitiveDesc()); - srcs.push_back(*src); - } + void resetMergeGrad(MKLDNNMatrixPtr& out); + +protected: + /** + * Set deviceId of this layer. + */ + void setDevice(int id) { deviceId_ = id; } - // TODO(TJ): remove me when mkldnn sum support different formats - for (size_t i = 1; i < srcPDs.size(); ++i) { - CHECK(srcPDs[0] == srcPDs[i]); + /** + * check the format is nchw or nc, + * which is supported by Paddle default memory layout + */ + bool isPaddleFormat(mkldnn::memory::format fmt) { + if (fmt == mkldnn::memory::format::nchw || + fmt == mkldnn::memory::format::nc) { + return true; + } else { + return false; } - tmpOutGrad_ = out; - tmpCvt_ = nullptr; - if (out->getPrimitiveDesc() != srcPDs[0]) { - tmpOutGrad_ = MKLDNNMatrix::create(srcPDs[0]); - tmpCvt_ = MKLDNNMatrix::createReorder(tmpOutGrad_, out); - CHECK(tmpCvt_); - pipelineMergeGrad_.push_back(*tmpCvt_); + } + + /** + * If input only has MKLDNN device. + * Otherwise, only support the previous layer using CPU device. + */ + bool inputIsOnlyMKLDNN(int index = 0) { + int prevDevice = getPrev(index)->getDeviceId(); + if (prevDevice == MKLDNN_DEVICE) { + return true; + } else { + CHECK_EQ(prevDevice, CPU_DEVICE) << "Only support CPU yet"; + return false; } + } - auto sumPD = mkldnn::sum::primitive_desc( - tmpOutGrad_->getMemoryDesc(), scales, srcPDs); - mergeGrad_.reset(new mkldnn::sum(sumPD, srcs, *tmpOutGrad_)); - pipelineMergeGrad_.insert(pipelineMergeGrad_.begin(), *mergeGrad_); + /** + * If output only has MKLDNN device. + * Otherwise, other devices should only using CPU device. + */ + bool outputIsOnlyMKLDNN() { + for (size_t i = 0; i < outputOtherDevice_.size(); i++) { + CHECK_EQ(outputOtherDevice_[i].deviceId, CPU_DEVICE) + << "Only support other device is CPU yet"; + } + outputOnlyMKLDNN_ = outputOtherDevice_.size() == 0; + return outputOnlyMKLDNN_; } /** @@ -568,54 +333,7 @@ protected: } } -protected: - /** - * If input only has MKLDNN device. - * Otherwise, only support the previous layer using CPU device. - */ - bool inputIsOnlyMKLDNN(int index = 0) { - int prevDevice = getPrev(index)->getDeviceId(); - if (prevDevice == MKLDNN_DEVICE) { - return true; - } else { - // do not support GPU yet - CHECK_EQ(prevDevice, CPU_DEVICE) << "Only support CPU yet"; - return false; - } - } - - /** - * If output only has MKLDNN device. - * Otherwise, other devices should only using CPU device. - */ - bool outputIsOnlyMKLDNN() { - for (size_t i = 0; i < outputOtherDevice_.size(); i++) { - CHECK_EQ(outputOtherDevice_[i].deviceId, CPU_DEVICE) - << "Only support other device is CPU yet"; - } - outputOnlyMKLDNN_ = outputOtherDevice_.size() == 0; - return outputOnlyMKLDNN_; - } - - /** - * Set deviceId of this layer. - */ - void setDevice(int id) { deviceId_ = id; } - private: - /** - * check the format is nchw or nc, - * which is supported by Paddle default memory layout - */ - bool isPaddleFormat(mkldnn::memory::format fmt) { - if (fmt == mkldnn::memory::format::nchw || - fmt == mkldnn::memory::format::nc) { - return true; - } else { - return false; - } - } - /** * clear all grad */ From 378dcb1833895d512b4ad20acad5796108e2529a Mon Sep 17 00:00:00 2001 From: hedaoyuan Date: Thu, 19 Oct 2017 19:24:50 +0800 Subject: [PATCH 37/76] Split paddle_capi_whole into paddle_nn_engine and paddle_layers two static libraries. --- paddle/capi/CMakeLists.txt | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/paddle/capi/CMakeLists.txt b/paddle/capi/CMakeLists.txt index 2c458a78c5..f59b1aa3a1 100644 --- a/paddle/capi/CMakeLists.txt +++ b/paddle/capi/CMakeLists.txt @@ -28,8 +28,7 @@ add_style_check_target(paddle_capi ${CAPI_SOURCES} ${CAPI_HEADER} add_dependencies(paddle_capi paddle_proto) -# combine all paddle static libraries together, into libpaddle_capi_whole.a -# user should use PaddleCAPI as -lpaddle_capi_whole +# TODO: paddle_capi_whole will be removed. set(PADDLE_CAPI_INFER_LIBS paddle_utils paddle_parameter @@ -38,10 +37,13 @@ set(PADDLE_CAPI_INFER_LIBS paddle_function paddle_gserver paddle_proto) - cc_library(paddle_capi_whole DEPS paddle_capi ${PADDLE_CAPI_INFER_LIBS}) -# No shared library for iOS +# Link the static library for inference +cc_library(paddle_nn_engine DEPS paddle_capi paddle_utils paddle_parameter paddle_math paddle_cuda paddle_proto) +cc_library(paddle_layers DEPS paddle_function paddle_gserver) + +# Link the shared library for inference if(NOT IOS) set(LINK_FLAGS " -Wl,--retain-symbols-file ${CMAKE_CURRENT_SOURCE_DIR}/export.sym -Wl,--version-script ${CMAKE_CURRENT_SOURCE_DIR}/export.map") # TODO: merge mkl into paddle_capi_shared @@ -55,7 +57,7 @@ endif() install(FILES ${CAPI_HEADERS} DESTINATION include/paddle) install(FILES ${CMAKE_CURRENT_BINARY_DIR}/config.h DESTINATION include/paddle) if(ANDROID) - install(TARGETS paddle_capi_whole paddle_capi_shared + install(TARGETS paddle_nn_engine paddle_layers paddle_capi_shared ARCHIVE DESTINATION lib/${ANDROID_ABI} LIBRARY DESTINATION lib/${ANDROID_ABI}) execute_process( @@ -80,7 +82,7 @@ if(ANDROID) )" ) else(ANDROID) - install(TARGETS paddle_capi_whole ARCHIVE DESTINATION lib) + install(TARGETS paddle_nn_engine paddle_layers ARCHIVE DESTINATION lib) if(NOT IOS) install(TARGETS paddle_capi_shared DESTINATION lib) endif() From 2073fb96cb1645ef9148ef4717a15e49cc57557d Mon Sep 17 00:00:00 2001 From: Yibing Liu Date: Thu, 19 Oct 2017 20:12:31 +0800 Subject: [PATCH 38/76] Enable learning rate annealing of Adam Optimizer --- paddle/parameter/FirstOrderOptimizer.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/paddle/parameter/FirstOrderOptimizer.h b/paddle/parameter/FirstOrderOptimizer.h index 895e8d6a63..f157188a4f 100644 --- a/paddle/parameter/FirstOrderOptimizer.h +++ b/paddle/parameter/FirstOrderOptimizer.h @@ -265,6 +265,10 @@ public: addParameterType(PARAMETER_SECOND_MOMENTUM); } + virtual void startBatch(int64_t numSamplesProcessed) { + learningRate_ = calcLearningRate(numSamplesProcessed, pass_); + } + virtual void finishBatch() { ++step_; } virtual void update(const VectorPtr vecs[], From 63ffe5250a120ff430469b8d000deb2b031c4881 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=AD=A6=E6=AF=85?= Date: Thu, 19 Oct 2017 22:06:02 +0800 Subject: [PATCH 39/76] Cluster train doc for v2 API (#2072) * update cluster train v2 doc * WIP cluster train doc * update * cluster train doc * add TOC for en doc * fix sphix build issue * fix error links * fix link errors * fix image link * polish cluster train docs * update general distributed training document * fix sphinx compile error * fix doc image error --- doc/design/cluster_train/src/trainer.graffle | Bin 5644 -> 6144 bytes doc/howto/usage/cluster/cluster_train_cn.md | 316 ++++++++++++----- doc/howto/usage/cluster/cluster_train_en.md | 327 +++++++++++++----- doc/howto/usage/cluster/src/trainer.png | Bin 0 -> 145107 bytes doc/howto/usage/cluster/src/trainer_cn.png | Bin 0 -> 33865 bytes .../cluster/src/word2vec/api_train_v2.py | 100 ++++++ .../src/word2vec/api_train_v2_cluster.py | 123 +++++++ .../usage/cluster/src/word2vec/prepare.py | 41 +++ .../scripts/cluster_train_v2/fabric/conf.py | 39 +++ .../fabric/docker_cluster/Dockerfile | 11 + .../fabric/docker_cluster/ssh_servers.yaml | 23 ++ paddle/scripts/cluster_train_v2/fabric/run.sh | 14 + .../openmpi/docker_cluster/Dockerfile | 43 +++ .../openmpi/docker_cluster/head.yaml | 25 ++ .../openmpi/docker_cluster/mpi-nodes.yaml | 26 ++ .../openmpi/docker_cluster/ssh/config | 1 + .../openmpi/docker_cluster/ssh/id_rsa.mpi | 27 ++ .../openmpi/docker_cluster/ssh/id_rsa.mpi.pub | 1 + .../openmpi/start_mpi_train.sh | 28 ++ 19 files changed, 955 insertions(+), 190 deletions(-) create mode 100644 doc/howto/usage/cluster/src/trainer.png create mode 100644 doc/howto/usage/cluster/src/trainer_cn.png create mode 100644 doc/howto/usage/cluster/src/word2vec/api_train_v2.py create mode 100644 doc/howto/usage/cluster/src/word2vec/api_train_v2_cluster.py create mode 100644 doc/howto/usage/cluster/src/word2vec/prepare.py create mode 100644 paddle/scripts/cluster_train_v2/fabric/conf.py create mode 100644 paddle/scripts/cluster_train_v2/fabric/docker_cluster/Dockerfile create mode 100644 paddle/scripts/cluster_train_v2/fabric/docker_cluster/ssh_servers.yaml create mode 100644 paddle/scripts/cluster_train_v2/fabric/run.sh create mode 100644 paddle/scripts/cluster_train_v2/openmpi/docker_cluster/Dockerfile create mode 100644 paddle/scripts/cluster_train_v2/openmpi/docker_cluster/head.yaml create mode 100644 paddle/scripts/cluster_train_v2/openmpi/docker_cluster/mpi-nodes.yaml create mode 100644 paddle/scripts/cluster_train_v2/openmpi/docker_cluster/ssh/config create mode 100644 paddle/scripts/cluster_train_v2/openmpi/docker_cluster/ssh/id_rsa.mpi create mode 100644 paddle/scripts/cluster_train_v2/openmpi/docker_cluster/ssh/id_rsa.mpi.pub create mode 100644 paddle/scripts/cluster_train_v2/openmpi/start_mpi_train.sh diff --git a/doc/design/cluster_train/src/trainer.graffle b/doc/design/cluster_train/src/trainer.graffle index 42384a3f059966e22e22f5fa4295cc9ead5cef83..43415ed8cf61a5acfa34f8e56b9577f338dbf254 100644 GIT binary patch literal 6144 zcmZvgWmFVUw}uA?7`kCd=^>;$q*E9`5R{M5%TOp%EBB z(u?2v?)`Pov(`T6-Rt~5=UJOE3J36S0Re}m3$6@6`ulZMjo!xc$C0IGxM&J}Mw8P#=>r1T7qD@EGku?6F*HeaLlEyQhOkf>a_x|%|5ci7+W9XownzMWX!bv&D?^RL)% z(k;*O|L6s)FKhKRtvYsZ`BE}-neI0~ccPnL+HXI(f5TB#|LT>?;$5j(TT4sXoW|E_ zBF97veL4J~N}P?GH$F8pdb!&_N1hl#ea~NeY!=5Z#I88r{oXcSY%}O?`XHUW0)N)M zJUN9rwU;LNaK0TTu1h~~-Imln^@(ofFj%3=L^|nsIqYJqkYhH(*PM+thNeu{h_aJQ zs66^-#`WsJGeS?Lc#1aLUMoXS^HioGGuG>L62gsaf$Ip@!Uj#J?gx;hjA?`uyTiXvI7W zu({ttTk~I7z1J7=|U*3~4 zUZr$AuRlJ#^A%gha-m;@)hZbU1wwmfgr^xchy2hnv93pEjBCoqot7lcaBSpZzi-RO zQ<5^ggw+xUWI>LX^hx%}8t8bYiaG)gN8Ne4Qy=WK{kYjGsew1Rd0f-b{8`@zRV~dW z!KI6{Pn=$X!_0p$T-_ZZ#~c!f;kOs(n1}TKODlG_%x{U1K3*-yWo2+4z73Uh(AnO8 z_8i$)odY^uk1KQ``WksRPwukbPfRu8{I{GBa)m62YlA}H59EjUldRFZxS716y1JLo z9k?C|*>esSaottZIXcO1$to+9{cAI7+OxGXS+{iERGsx=r3FqcFc2WobP5o^c2Sh} zbn340=kH`)AzcYgV)S6+T!<%_^mVD;xLuA~C&Q{!UdviSzB~yzW-~2a7}+7Tp9CA| zjqTjTpWKHuilY5ZZt&kkeiMcAUdz}v-90(*_0;za>`8L}qKk#+tLA4^*dY^ldx*{u zM=~nF?jd@CI|H7dTB4P;*^=h)4_h2$;GFO;{or#&3>WIxQH2z7KH2^mvc2lJRG8u~ z=KCuTSLSu+`EB&EiA8+A_;{(;KdSQDIQDkz<>vyWBq833GJJZ;=$oww?UoF0j5L>FPo}G!6yos)h4)y!_|v;$QCMER7a_kdtfAR{ z?^7DV5xy_I7e9uHB1khiMOQv5(`~*9cFljTS19Jf`NM>C@Q=g6ynocM3rRTwIxqsd zkvzAX8(hAJx3$Q(@BKzz&-47p(eorK5d;%=9OE{f*x8VfXQ{A0VuwYVg`5Vv7Ft_D z1F4|FVMWk}(`GVXw~lHn`6Ff|iYJ8#8Fs0<^!LckCU^&}7X`Ei!r#L9zgskl6$-o< z_f4rpZ7uLW5yAyQfC|WXj=(uR4K5~QBd&ojo}JF+Qe6p~B_rg(Kv?{pZrsl7;f>&y z!~vmcHUCV#X;mJ*zK@cMl0izwFGB8Crpw3jC{lT~Y&v&zw|tm$a`5(Sbx9>_>>R*d zRZc&fK&tSX%pG70uq~>AVN29$WLmmyMAuM{SSa7L3(;um(R2cy0-hEL*~*H0Q-t26 zoglT{-+pbTJu^A27(v4MyFw#U^#s?e+y(0a}7e^AjzU?zyL8 zH&;!g%`FA8X*~HNV-vharhtkOf5V+z&hsCqP}BH5iRO11ykHf-nmY0n5L7Q?s0W(( zzh#oSW8E|DYm-zkfj98I^#YrFS7=T)qfM{ok+ogwKh^0jyg2eW?k`LS8k4Y|t+h2M zu90-T^H5}loo{#U<6fn!-`Q)z_UFe_vvPmj{+Mb|DyD=BL)qYOpb20^8CVg$8Nzz5 zXvd;hp{zVZPHNBz}X|_UV)h)33?LH11Kz(SVvKRQIg8#Q2Y`WW+v`Qp-^LH;iqt z)5xshj-E}?$zbP$^393I$)VHMXw2xx(H~{hKgX9wf2mmB;d;1ymmarlj2PaxII}n# z1p4;H8;M|0^QW>Y@tO%YirUE=7$Kp<(;0Crb%I<%P40EOuGZXm#>R~Vxo*XS01=+_ zqNw35CEyx>@E`@J zJMm1*cj!8ss1qy$cXL7F&3PoUw0w;}&Ox&yl`8Bb3~RSWll2>%lz9|+$<``hrOG#B zZW3n&D7iMYGiBXk-eE~9eMa?>|Rl9uXsJ4wU&Vwszjo&7CO-&E{`7E93o14OA&V72{q>CVT()~ zE(ACUqQiAj^Uj!seOGflyZ3sXWjJ7c5hl;3Hk~r;pnpE*JXfa`{@Yb*PHB!jp1<;Y z2Al2{e(huq_?2H*?D4X-xA2oVF;b@%OX)wG*M5>;z6M_Y{&d53N|+m3Bk9|Qt1F+k zfqBw&Xs%g7<}xwvCRdPi#O~F4?8OF1T1@`o+wXrBN%3+k`!RkSB7={H0(uSOs+~~@ zCd&ikX;%tvwDx!@%w!qcF1ihblXG-#_swhj=f1A&gpUNZ7=0fv_%o?&hZ|)v5i|I1 z{6Z&k=v}t|<>xPbJzhx$S3>xyXdi<%%*r#}#m_S`6ExAZkNb!M4@mx7Y?-9>_rs#% z7-;1^ah@tZ6zi~uA_24x3lM%dtYZAj-{%{oWbxS0b2EG8@nUKI6|qKLj70VxgJxx2 z`ZFV&Z;2h+CT5pzyPM~}oi==QSg2s+&zk-W&O6V~H%o(wZlO-wmarkp?W_;v+s3HL zBXFECAE;+`#-xwbSpgZ>2B)~vhr45&N&%B%12_G)*p!ix(Px{Ps$bt|0i{%bGycxT zx&@^&bimu;VpV@J64ZY)?!W__$ZkQmw!C-5&tHAIk-r40{Q}}cHnlObDdmCH7LP;# z|G11|sP(?Z-cl_CeQ?DQ;ALK29!Nt-^S#ScB zFsSkL39tg}DZZ;({)+*-G4yzVw_MMYA4vfpaAJ`Ho^c6eKl;BSI|OT%raR*TZNIRP z#oC@oX@}w6C>N<0nM2aJYU1W0R;dQkeR?!Ym_1cmE1Y?YMLsK(gottP5Zv3O1-yKkvwtsUOYGx`Xw>dYA+qjQExVb@1S-=j2o1<-M@S)*AQbI z{WG#ZGH0?k(VmprM}F=>$yFU;)GAHOmK}^;@x!s<4asYs zh#PB}d+SeffpQhg*`7Mu$=e#wR@h9ewva#DQ`%Er9&4gB(d<@zq^-{;LoE}bOkeL} ze|n>bu*LZd2MRojk)hzk%9yMu&a{j)FU5DuX<2irM3(o!p3<{`SXRC_H;SFgjPS{- z4z(mh1vJHz*r}4!7ppkOYipY`0@>a&Efk?nb9_18hR zC&Kthb6eD7s8KZN5f_tk>3*uhiOdXmzLFzeGnh^^t#sRLTF`9q^|BnbK?IWgMnZb@D4b3;~LKB3{`92-U7!V7M7 zYe5hBa>1H+w9o7sv=Aaor$(Z-oZ!MoMkS3}cWWxIc0)@w)bn$@Ltmvod`GI!c7GxKUp z_0uf=oXv}V${4mOELAXKPeHf$T)uoiT(1id26H4TbqY{|kiWH0^KP`rBY+INl+(S~ zh4~oObLT;8-+2h9`Dk})p(#LhbuMEa&g7vPWekq7oI&4n@0~gg5&cIm#J{`3G+$Xq z+HaR-P%s=V`&GioEbZjKOQMbiA!^%0wa`z)bH7$LGW+||T=8{fZ?634-Bh_cLT|B9 z#|rg=UiG;>FF68Plj~SgoEs~FE`4Xp^b?aFp_}<)Cwrt}0;;Kj-*bWMfOQAR`Y4+g zVR2@7vefO(0!L|aP__5YD#-!|&6bQqFS#Po)zd(%=(Rv&{mtD;%V>n_ENj~6lu9eJmAmYgA z>fd&*y{eO#dn(yyG+`^X@;dBncn#(zv2x3v>0~y{6?#+bWGRznJRwNm*6@1_6_*5H zoj9j&o1)C6>)WoDIW~)$#M$5u--@Z?)U@nHIQlu5<2dmXG!y^4lm8*G7O-g7p=KWK z6=FKx=VBK5{04A)k7j;9oC(pS}#A;ltu-xwQ+J3 z6Ca_*KW$<95`f*)>?knTVT@;6&Yj8if@?LJNCGG!OgK_qDA4>o%gKI-xEs=DvFr-Bu_ws^T19B6G?P~4Y4HG7r*Lm zFrCA0ky+QIVyFmfmps+0q4f89AWLgKpsvy*%B-iS<611t5qg6n?ve*q4kix@A8C>8DDFmh+SK63GHHgf~Y;IbzEZkM^gEWUtd8l6L+3W zV&q`tq~X280St5J(+T~x6L}|CP=nnkL<23ye2NnQ3g~rTRY(z%DSuN5oie_nG9t)0 zwe-a7jx+l94kW9@4OJWLu9zbDP#Y%NM@y01(xB?z9W7^y)l4?oW$2FCh zo2K?Tp^&yf?N|R-t0|T|?4RgiSH(*ijbKRZ(lpr%_)63PqVF;Cdo_iw(TSegubj}3 zfA;Jf_l3z#OZ8X#PRGtn5ZaL8rK+S+>zJE>ufVk^vy3TX*yMlB8et@YpPHj`=}E5g z;aQ1p(q7az(JZSz#9ZprwATZ3?h!(iSiDYZ?Ag+9v2!fPUN7Mm(0#@u3oEZycjMd*etMa4|Vy#MX{h-o46#o%xC#Yrl zBYF#cjM?h<4kbg4nm+aA=svFogp zD?Dc=s+riFfdZH0e5HE_!+8x?O{%Iyu;$UGp;OQ07=rnI@3+@sl(qYuDxLxX|v3DEA%6_AaEH@Caf7KGMX` zH79BpQ(8D@#DCTQW{tZiQYAFQ4$TY7p#mCag+&o4pb7u!@~GrfZFGz{_jM6{-f{0P zMBtx61KTTkuZuD${e`j0=q)^S4cP%@|F91-_+f20yv=gnG;7L1Y@Y^_P2?2lBCc(% zb&OMo*Z%*mJkO)2&&h7uOGz=>l%BG9;QZ^#d$6eZKdLKXC-pDpK%HP!*X3`c1oiVyrqZTQpr4WHB!s+%X=Kb;1HBd}wGF3y20 z9In+b(`QTe^4Dr@{UCCE8U7cqFO)Z0f=~5#8yYTCUz`5%=WLcx)2*=Jm7s zrD_J4>z#~vWYlsO{t=>2Jeanp$8i;`-;)=h+>5*Jw$PSr%)VY!i2A6mu%vn9yK<}! z;BMmeIZwDb4PgLr-=PivWG%Y)=U9vvL*rZdSQ1On2Q{A%_CwxPC#Itw#nUvGnG>T) zsggyKxMFJ4o_qLPu9Wv~j~5Rg20Pi~T!r2k*5lU`4$mjlz|8AKbf{Y|XZrGoQBZu( zsdH|D?9CCQOnl5Kgp{%9u`qrC%!T!`lid8#%oXQR?=jN^J3$6~({7h{sV`Q@j(;ko z^|&cE+;hIhPx?o(GDo2Vgy~97g^i4o{=T0F ztHh99-EX8|7){5hYB7hIKikW)?1KmrmK9Kb#Q&7R-xxim3Q5^@am~^(cl&Ei5UkPp z8O>S;QSvxf;#7*1ZD3+Jp-M!Zp@mIx&0_uWvINNs~-k1UsS z!E8IwlHVNVYkbs>YKMAA0go&dq0tMS%tHn!RU$@h^sHBb_e&d#h^8!Ank#ePMpVzG zQ4$(CR`Q}RysXBpvNoTJi^-_Xzb-DJSEO6pha^}?jI&ITd-4xA@+4vV{jyiLJ9p*E zg>S64?YAoF+;oE7UoWj2K=0xVByevF=n&`H8$bFFkLGlLWQ&ZTrxF5ql)^Aw@OZNi zW63S;o_`aP%DCUto0B|mz3KNzhR}M;Ocf7az%O^p|NJ>K=<9EjlS+MdUwyUf4GAH( zJ+@7dlTOn@`nU^pG4xe4#TMEB=|fUVj|JNHg2wB0S~UEnhRxVQm4f>}Yiamv@D%H9 evV{*~yagCI?V0MR0-e6yGhXeIgmW)o0R9Ji0@5b{ literal 5644 zcmZA5WmFWvzwq$|1Oy2QLAsX^M7led4ry3A6a-nCB}8&rKpJG}lm_WqQt9qoxDUwORQkX)?1#4vIynyN+0 z9RvTov_0_X>SuX`6*pdhZ;<0|D8pyFxwFrIG`S@+PTFhD@9r;VN(1M`<|Q7t&dbLy z+Jt#Dq}oJ8mjuAAWwTz*X691|&>iT{y>ZKprC8gk1D7x6FS#oc@^7u%0=&Eo++OIF zQ+SeEno31?RYb(!G8xUO_G>w^l#_o?t35Y$+ic*f{Mx#5vo;;w7+7BNJI$Ff@G(8U zt)%p1Yt)&vX}_DWa-BK;D3GhPG&G~bQOUB>L@X|AMd5fliTkS5H;Y*=TBK5Yp43X$ zMwBAb;6kRq8M7sBZ2F`L_Ht&K75MabN(7R*r{hW~Dj>i9)PUK1emyKSWL)@B>hUu0 z;X3eW@vvI4O6C$ft4^?0uk@YL=Ky}6Cc}J8*k@C} zy$_G;_ls*Q;jq+zry7ivN0_<^Sp!9vinv$m;+W-knvj>K zp>h+*`pd<6ZxSGA?s`}8J0?b2RE2%M*bwY;mplxz52jm_H|~~s=@S?dbTH@%4GM_R zdjb0Rme*Wo9Fej4Wu_DQn8N)M_UOIw=E=QrmKZFBQK!})r3HVAE>UrBm& z$?)rH2bl3;?RNHXY2Hqqm(k7b;zil6ns#6LuzcLIzfA`#wTWBn^z^tc zstDP)ZeS{j)I_Na5-z16m1}`ghU(q7dCipfP!LO?aq3i}A`zU)YO$}r9{)Jjh45`j z{)yT*93^Fj6yNhr+Q+Z6Nbvq|DZ-XPwg){4vGOdO1`pz^OF|4qisuLM*p1^2Bxx|O zS20UaA{s2NQ;)f;arw{RN%}vPr-X==qkLU;x-633w7MdM29QsUeK{`DD3F%$oBT)) z+Ivaj6{+$1Amj(CKr6M=`k-RD_ATZH^R9}MpSGC%6$NKelv=A|OM|@lH_~UzdxS0! zsn1t_tU8~s7CwY}QAARMRBFhZyyiV0v>wDS2rpzYL8L1Y1#e@o)9+Iydkya>!!p>n zY-aJDmxVLP`8b2IOS0_2H2em;m!h39Cw0N`z8{Sw((=YV5R~9>afs)Ixh{L`O?Rx{ zD7*y;ML9IjhYP>`LCM<;J59)u^5d$n$L0L;uf#@cnUmU94R9q{U#MZG}58(m&s zgUpkiUT#%h9Bd$lOh7Da(7eujc6wtYE|pBrT!%7|?zDV^9XOf{asWxryx{yhAW8)0 zEBE4XGNUEaT$`}uP)l1&&-Hh)Oaa-ZB*v~d`fgQ|OgI;oXwUWne{Z22g`D6v#u!B# z8v;+X;EvcGU=Oe_jPoN9u2D%}bK8h&dO2btf7`)Jp{hgKg(eOVFC?~yI{#v8yrl@t zA}W+mV$VXw(NR49DaWG%TB(HgnipH?7f0?*43x~bUAAQ>nKwgij$&z#9q96-I9OVS z63$)mKrSJ|>23&P(;NCHCL$J89~)jZ7LfE1*-d#quTyQx!*w3~DOi+fq5HtT6;ts0 z^-tB8W%i0kgSokzNp2UPv~s1^9d+)%^|C8=1rV_qTy^1FX+FO5Tue>f%%u>NGq|^5 zOnzWJHXC?ceDdMm$Cah1WpkpvQI^~wh!usHbtZ*Bq3FIoHazb^Awdg;@${g(2Dox< z#5-^;69f9i+QbRYbw%;Ry{z9+vrejsobUC$!fdoLdB4G=>>2k@`F5jvU#5^fA*x~_ z6#*O%ejMvG`FfgyrTyWqw4U{N`J@AdAjCQ2Z2MJPF;aO-r~GNfk{z+4gQM!@>%yK! z2QY*6sJf}#d#kA%!DZqZr&uPj8rr@!fXKg444bJlzR!EHdeHuMO+XZG6w^bo&epeP zv$$%b5WA5mH`2X}X7xb*$Yr?eDEQ;e@pfd(KWNxi79Pp=>)a-)n~XH=YfxO@NOy zArCJc3$TG1Z1c@}cIbl~u9jXRw?XLzSxBFgT7=r`C8y?>Un3)v$++Zpm)3Xz`0%hj zE63|GX{S$Ee+y$JGlE~A<;aB3@Dc>o0q_C%nteG{(A=HE-@;bGHVdi?+t0W45L1g<4%p+g3qc{ z^m^;W(Iz|;1^-eQMwzBp%6f+B*cfo(8ngMU9*_Nqmgf|KIH-PariqK_i%@WXh2j_w zIH%U|FBaMl*woRYBu~2DxQI3GWGvRS$oYW5gf-bCKKW#E^&O)`n4?jYE3@rFSE*E( zYGb;inhgRkW(~bVlCdLGse)XbtRlX*OHV{};O1xX#gXy>0;6mMqZDX}vhWYSz)$Gj zFhqIhmASkRzPJ~j48fccGDHEY9BYt!ibg2;qgE2!s%Ew_$S1wOP1kjDfO*U8tmAHA z?ydT?Gd~k#*dt|)vt$|#^zm;*o(4W+`oUXqh$-$hXz?;h#zm$aYwD@C3S zLs$!lRIi!d5g!Vw6FhtZW=#;PL4POUGYL;yQ{tW*+$yGY_+q9-r;d6)5ItOQiLk(Q z$kb0wZ|aQRNskZJ`#NYW#hKSSZylIp-j zenHdn&vSVu2lkVr1Z-{nz%n`#esN3Hch-<6dUREYz~3fj zhmL7`+YQ~*r?+|vrriP-&s{TMjmOR9$=p%kS;FPy(=6hr3d1QR+kq5Rb%YUmkyMfA zbv_)AVrEH5r$hBPzEHFfZTeikhOLYB@6o=u{mxTT26f_~3I&@(8j6$dL6!nt!!MFN zW2|FE(+88sr>7HdL8IyrX8j6P;4bm*#J+$umTM@|dA@{OVMSu2+ezj->LzK%ksWF8ZcypEafQKyU!R^9o zP*bfu%M%^GtrELt8Bltqxx|_d&D}OcsjR08NEng4ccHo(C%mt)U9J-+*yE%|wNe41 z_{e^}{i?{bEC=ECuO|PqiTa+@sg1)yWa{yF0k(P9N7hYH&&EJKZx~899{;?eMYi?Y z!mCk{)5EmGl0b=C`wkH1yq!A>Ir~~v25zOo3UUDE>iNxWAs%XqQ`M)!vcR)8sV-DN zGZ!l@!3=kFdSOSjWJ%Y}qWS$8MhW{Se{8+jEnJ_ed4U4e}-X$ zW;0L!_o>{=^>d_7ox~=Ml7wHJ>Le6M$5!1*Cb|TA-qaP_x{|o6?A6RCq{ohVo)t0| zegk56=BJAadOfwT0$mRYva1r)lm(;^i+K*O5fl}wt78)(wqH!-OvxzkbNnRGGg`=k z%Np}>FxNPeB6@0x?C`YTr)9^Esx$OsvE_r=Oa{C7!b5ot*-Y+b=X-0mLdQ7ZI^koy zs+Ho7ZX^;4_MY{vg{#h=Tbhi$+_HzsAKxSPQNXJ-Q;n>l+N5q(i(% zBlHA%29htpTEjBS>MgJzq3LCuGrqs@`40|)4t{qmq%JOKA*WsEhzIB6CXK{rKM9k> zCrbMC1U(Z&a|O6^OxBjX5l9CfyGuhZ-=wS@3)-_Z-*(CTnH`^-S^w5zk8a!^W)<_*U&_PK-k)G zIYsWlsyDtoi_^rii5Zbk4mH8--m&&yd$MlobATPTfr$x$4X{jQDeP}1U3Zaoxd z_EPpl|!x5v=eDFdqnR+bzCQDL#7>hMB2n_se4&i^jh51ua$4puj(#){mfP%h;dimRU6wg_!3M z{ME$pTWL9fgZ{ha*lU+O88|~UpLBA+#k#{UdH^keb}|UtY~buocRjJ5{YCg059esd z86M}T(>{^WKXDn42uLTEwN5!j1Ni!H7N6|Hm!8 z)b73!-%4;e?N2u>)%W=F)7gBior|B+b*6RF8oy??-P#kxs#A~ zF&BcYhg20x15K86zb_-qsyj*w^uXRU05IE-I|`WVK52%ja@*>W(77N5DsK zU&+uW+>d}t^eW5gIW3A0Dcj?aiC3s;5K_;!i1oS86R2Agsm$8efM9+#5|NNC5Fyb+ z8#E`y{AUnXGFRDFx~ho``dUhCCeL2la4Z||M-n$9ivjnje9KOYcJVu(r7QZ}?4CyI z!93j$Fk*gl!6#PSd^BC3drPT@V74SqMt!boc9UOY4A`BImBBIjAr>+AqTqr?_2NNN zqTbw|&CyNoX-@rpXUTSHg_35p-rKQfb~e}Rbr9G$MMeqWy(ab=JgvP+*gW(Qub{8O zn6y^wCV43bP6acBY>p+X>`oJL&{=Wjnp`UiayYWdcrRv)U*K>inUM3O0=K^c)A2Kb ztE9)5cCvv|04Qw}_?Ka$3hgrk9(^Mlr&o2z0*w8SpQe2vaST<+&Ks-)DqwhTHS+YY zm!_}|Ze0s;@cuud?2J&RohA7H3`P8Z4b?Q)_SaCngnr7}eRxt4e+Ei|3LkbK$P^=O zP3NC^A18Od;Rs&&Y>fYH@q>q9exFvFJB>m9r;)nW{F-jQoed_$%FN4^ACWwRJnTGv z13B@O8R5ETHgs7riJ~5hjCsci#zioxr{fZYRO&*{R_7bP@p})nk!#oYsr4B1EPk3| zDIT?UbLAQD(4K4c`Lt;$>R1r;-ogSXDvpV-baXGypU!E#l|ZOSdVi)X?zIu06P zY(7q*)EkkxdkVw9^2iT;Ty>^sTe2n7kPL}b2yaU*3yKg26WB zpyk=N0IFIRK$5#^GI*j=x)JoP>wkbM@jsw?ktv~H{o48`66j?)i%7FKb<<$fwAy~T z!~DNg)vy6QvQQ~V+{BCpEo#;%s?(W;D37!;7qtf;SS+v3MB`_MKU271{}pfoD@zh$ zKea63sgWM|_q*hO`ahuc@k%Kz*|V65RYd~)9v3?M+n90X9dPBS3SJ;8u&%eTiKWg% zbx>s3icicvATczacgX#KIk7Sz+wmNrb1ON(s6T_ou!zMs8O!&=&2F+9suiZIKwz#o z#(#HTwgY!y3_AQ(a62}7%`!A}U3@32pS;yRx;)6Jgya9XrwZjPRmh8rT}g>)=eJdWLm>H(zAb6XlIFQuk6|#s zdD(81ApQ%rQ5wr4DwwdOcFnN5qCl@rD~0C?i#sY}#!{)+AG4rLk_dlZ$2|S`NE4Wd zRbE@=#q5)8M=#oxAVP7I1+6$?xim_yML9fKr+{o7Y*UI+#-X2`_X<5j^TE@ZXTI!K z9@(IN(#tB2?{6kp+zyz3ov=0-MLO(WPEm9AJ`VRz zu5XMAF(^GAe+&fP1poe`#K+Sr + +- 数据分片（Data shard): 用于训练神经网络的数据，被切分成多个部分，每个部分分别给每个trainer使用。 +- 计算节点（Trainer）: 每个trainer启动后读取切分好的一部分数据，开始神经网络的“前馈”和“后馈”计算，并和参数服务器通信。在完成一定量数据的训练后，上传计算得出的梯度（gradients），然后下载优化更新后的神经网络参数（parameters）。 +- 参数服务器（Parameter server）:每个参数服务器只保存整个神经网络所有参数的一部分。参数服务器接收从计算节点上传的梯度，并完成参数优化更新，再将更新后的参数下发到每个计算节点。 + +这样，通过计算节点和参数服务器的分布式协作，可以完成神经网络的SGD方法的训练。PaddlePaddle可以同时支持同步随机梯度下降（SGD）和异步随机梯度下降。 + +在使用同步SGD训练神经网络时，PaddlePaddle使用同步屏障（barrier），使梯度的提交和参数的更新按照顺序方式执行。在异步SGD中，则并不会等待所有trainer提交梯度才更新参数，这样极大地提高了计算的并行性：参数服务器之间不相互依赖，并行地接收梯度和更新参数，参数服务器也不会等待计算节点全部都提交梯度之后才开始下一步，计算节点之间也不会相互依赖，并行地执行模型的训练。可以看出，虽然异步SGD方式会提高参数更新并行度, 但是并不能保证参数同步更新，在任意时间某一台参数服务器上保存的参数可能比另一台要更新，与同步SGD相比，梯度会有噪声。 + +# 环境准备 + +1. 准备您的计算集群。计算集群通常由一组（几台到几千台规模）的Linux服务器组成。服务器之间可以通过局域网（LAN）联通，每台服务器具有集群中唯一的IP地址（或者可被DNS解析的主机名）。集群中的每台计算机通常被成为一个“节点”。 +1. 我们需要在集群的所有节点上安装 PaddlePaddle。 如果要启用GPU，还需要在节点上安装对应的GPU驱动以及CUDA。PaddlePaddle的安装可以参考[build_and_install](https://github.com/PaddlePaddle/Paddle/tree/develop/doc/getstarted/build_and_install)的多种安装方式。我们推荐使用[Docker](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/getstarted/build_and_install/docker_install_cn.rst)安装方式来快速安装PaddlePaddle。 + +安装完成之后，执行下面的命令可以查看已经安装的版本（docker安装方式可以进入docker容器执行：`docker run -it paddlepaddle/paddle:[tag] /bin/bash`）： +```bash +$ paddle version +PaddlePaddle 0.10.0, compiled with + with_avx: ON + with_gpu: OFF + with_double: OFF + with_python: ON + with_rdma: OFF + with_timer: OFF ``` -# 运行分布式训练 +下面以`doc/howto/usage/cluster/src/word2vec`中的代码作为实例，介绍使用PaddlePaddle v2 API完成分布式训练。 -在本文中，我们将阐释如何在集群上运行分布式 Paddle 训练作业。我们将以[推荐系统](https://github.com/baidu/Paddle/tree/develop/demo/recommendation)为例创建分布式的单进程训练。 +# 启动参数说明 +## 启动参数服务器 +执行以下的命令启动一个参数服务器并等待和计算节点的数据交互 +```bash +$ paddle pserver --port=7164 --ports_num=1 --ports_num_for_sparse=1 --num_gradient_servers=1 +``` -在本文中使用的[脚本](https://github.com/baidu/Paddle/tree/develop/paddle/scripts/cluster_train)通过 SSH 运行分布式作业。 它们还可以供那些运行更复杂的集群管理系统（如 MPI 和 [Kubernetes](https://github.com/PaddlePaddle/Paddle/tree/develop/doc/howto/usage/k8s) ）的用户参考。 +如果希望可以在后台运行pserver程序，并保存输出到一个日志文件，可以运行： +```bash +$ stdbuf -oL /usr/bin/nohup paddle pserver --port=7164 --ports_num=1 --ports_num_for_sparse=1 --num_gradient_servers=1 &> pserver.log +``` -## 前提条件 +| 参数 | 是否必选 | 默认值 | 说明 | +| ------------- | ------------- | ------------- | ------------- | +| port | 必选 | 7164 | pserver监听的起始端口，根据ports_num决定
总端口个数，从起始端口监听多个端口用于通信 | +| ports_num | 必选 | 1 | 监听的端口个数 | +| ports_num_for_sparse | 必选 | 1 | 用于稀疏类型参数通信的端口个数 | +| num_gradient_servers | 必选 | 1 | 当前训练任务pserver总数 | + +## 启动计算节点 +执行以下命令启动使用python编写的trainer程序（文件名为任意文件名，如train.py） +```bash +$ python train.py +``` -1. 上述脚本使用 Python 库 [fabric](http://www.fabfile.org/) 来运行 SSH 命令。 我们使用 `pip` 来安装 fabric: +trainer需要和pserver保持网络联通以完成训练。trainer启动需要传入端口、pserver地址等参数使trainer可以正确连接到pserver。这些参数可以通过环境变量（https://zh.wikipedia.org/wiki/环境变量 ）或编写程序时`paddle.init()`中传入参数。如果同时使用`paddle.init()`参数和环境变量，将会优先使用`paddle.init()`中传入的参数。 - ```bash - pip install fabric - ``` +使用环境变量： -2. 我们需要在集群的所有节点上安装 PaddlePaddle。 如果要启用GPU，需要在 `/usr/local/cuda` 中安装 CUDA; 否则 Paddle 将在运行时报错。 +```bash +export PADDLE_INIT_USE_GPU=False +export PADDLE_INIT_TRAINER_COUNT=1 +export PADDLE_INIT_PORT=7164 +export PADDLE_INIT_PORTS_NUM=1 +export PADDLE_INIT_PORTS_NUM_FOR_SPARSE=1 +export PADDLE_INIT_NUM_GRADIENT_SERVERS=1 +export PADDLE_INIT_TRAINER_ID=0 +export PADDLE_INIT_PSERVERS=127.0.0.1 +``` -3. 在 [`cluster_train/conf.py`] 中设置 `ROOT_DIR`， 该 ROOT_DIR 要在所有节点上存在。为了方便起见，我们通常在所有节点上创建一个 Unix 用户 `paddle`，并设置 `ROOT_DIR=/home/paddle`。这样，我们可以将 SSH 公钥写入 `/home/paddle/.ssh/authorized_keys`，以便用户 `paddle` 可以 SSH 到所有节点而不用密码。 +使用参数： -## 准备工作空间 +```python +paddle.init( + use_gpu=False, + trainer_count=1, + port=7164, + ports_num=1, + ports_num_for_sparse=1, + num_gradient_servers=1, + trainer_id=0, + pservers="127.0.0.1") +``` -我们将放置依赖库、配置等文件的目录视为 *工作空间（workspace）*。 +| 参数 | 是否必选 | 默认 | 说明 | +| ------------- | ------------- | ------------- | ------------- | +| use_gpu | 可选 | False | 是否启用GPU训练 | +| trainer_count | 必选 | 1 | 当前训练任务trainer总个数 | +| port | 必选 | 7164 | 连接到pserver的端口 | +| ports_num | 必选 | 1 | 连接到pserver的端口个数 | +| ports_num_for_sparse | 必选 | 1 | 和pserver之间用于稀疏类型参数通信的端口个数 | +| num_gradient_servers | 必选 | 1 | 当前训练任务pserver总数 | +| trainer_id | 必选 | 0 | 每个trainer的唯一ID，从0开始的整数 | +| pservers | 必选 | 127.0.0.1 | 当前训练任务启动的pserver的IP列表，多个IP使用“,”隔开 | -这些 `train/test` 数据应该在启动集群作业之前准备好。 为了满足训练/测试数据放置在工作空间中不同目录的要求，PADDLE 根据在模型配置文件中使用的名为 `train.list/test.list` 的索引文件引用训练/测试数据，所以训练/测试数据也包含 train.list/test.list 两个列表文件。所有本地训练 demo 已经提供了脚本来帮助您创建这两个文件，并且集群作业中的所有节点将在正常情况下处理具有相同逻辑代码的文件。 -通常，你可以使用本地训练中的相同模型文件进行集群训练。请记住，在模型文件的 `setting`函数中设置的 `batch_size` 表示在集群作业**每个**节点中的 batch 大小，而不是使用同步 SGD 的总 batch 大小。 +## 准备数据集 -以下步骤基于 demo 目录中的 [demo/recommendation](https://github.com/PaddlePaddle/Paddle/tree/develop/demo/recommendation)。 +参考样例数据准备脚本[prepare.py](https://github.com/PaddlePaddle/Paddle/tree/develop/doc/howto/usage/cluster/src/word2vec/prepare.py)，准备训练数据和验证数据集，我们使用paddle.dataset.imikolov数据集，并根据分布式训练并发数（trainer节点个数），在`prepare.py`开头部分指定`SPLIT_COUNT`将数据切分成多份。 -你只需完成 demo/recommendation 教程文档到 `Train` 的部分，之后你会得到训练/测试数据和模型配置文件。最后，只需使用 demo/recommendation 作为集群训练的工作空间。 +在线上系统中，通常会使用MapReduce任务的输出结果作为训练结果，这样训练文件的个数会比较多，而且个数并不确定。在trainer中可以使用下面取模的方法为每个trainer分配训练数据文件： -最后，你的工作空间应如下所示： -``` -. -|-- common_utils.py -|-- data -| |-- config.json -| |-- config_generator.py -| |-- meta.bin -| |-- meta_config.json -| |-- meta_generator.py -| |-- ml-1m -| |-- ml_data.sh -| |-- ratings.dat.test -| |-- ratings.dat.train -| |-- split.py -| |-- test.list -| `-- train.list -|-- dataprovider.py -|-- evaluate.sh -|-- prediction.py -|-- preprocess.sh -|-- requirements.txt -|-- run.sh -`-- trainer_config.py +```python +import os +train_list = [] +flist = os.listdir("/train_data/") +for f in flist: + suffix = int(f.split("-")[1]) + if suffix % TRAINER_COUNT == TRAINER_ID: + train_list.append(f) ``` -虽然这些文件并非都需要集群训练，但是也没有必要删除无用的文件。 - -`trainer_config.py` -表示模型配置文件。 -`train.list` 和 `test.list` -文件索引。它存储当前节点所有训练/测试数据的所有相对或绝对文件路径。 +示例程序`prepare.py`会把训练集和测试集分别分割成多个文件（例子中为3个，后缀为`-00000`、`-00001`和`-00002`）: +``` +train.txt +train.txt-00000 +train.txt-00001 +train.txt-00002 +test.txt +test.txt-00000 +test.txt-00001 +test.txt-00002 +``` -`dataprovider.py` -用于读取训练/测试样本。这与本地训练相同。 +在进行分布式训练时，每个trainer进程需要能够读取属于自己的一份数据。在一些分布式系统中，系统会提供一个分布式存储服务，这样保存在分布式存储中的数据可以被集群中的每个节点读取到。如果不使用分布式存储，则需要手动拷贝属于每个trainer节点的训练数据到对应的节点上。 -`data` -数据目录中的所有文件被 train.list/test.list 引用。 +对于不同的训练任务，训练数据格式和训练程序的`reader()`会大不相同，所以开发者需要根据自己训练任务的实际场景完成训练数据的分割和`reader()`的编写。 +## 准备训练程序 -## 准备集群作业配置 +我们会对每个训练任务都会在每个节点上创建一个工作空间（workspace），其中包含了用户的训练程序、程序依赖、挂载或下载的训练数据分片。 -以下选项必须在 cluster_train/conf.py 中认真设置 +最后，工作空间应如下所示： +``` +. +|-- my_lib.py +|-- word_dict.pickle +|-- train.py +|-- train_data_dir/ +| |-- train.txt-00000 +| |-- train.txt-00001 +| |-- train.txt-00002 +`-- test_data_dir/ + |-- test.txt-00000 + |-- test.txt-00001 + `-- test.txt-00002 +``` -`HOSTS` 所有节点运行集群作业的主机名或 IP 。你还可以将用户和 ssh 端口附加到主机名上，例如 root@192.168.100.17:9090。 +- `my_lib.py`：会被`train.py`调用的一些用户定义的库函数，比如PIL库等。 +- `word_dict.pickle`：在`train.py`中会使用到的字典数据文件。 +- `train.py`：训练程序，代码参考[api_train_v2_cluster.py](https://github.com/PaddlePaddle/Paddle/tree/develop/doc/howto/usage/cluster/src/word2vec/prepare.py)。***注意：*** 对于本样例代码，在使用不同的分布式计算平台时，您可能需要修改`train.py`开头的部分（如下），以便获得训练数据的位置和获取环境变量配置： -`ROOT_DIR` 用于放置 JOB 工作空间目录的工作空间 ROOT 目录 + ```python + cluster_train_file = "./train_data_dir/train/train.txt" + cluster_test_file = "./test_data_dir/test/test.txt" + node_id = os.getenv("OMPI_COMM_WORLD_RANK") + if not node_id: + raise EnvironmentError("must provied OMPI_COMM_WORLD_RANK") + ``` -`PADDLE_NIC` 集群通信通道的 NIC(Network Interface Card, 网络接口卡) 接口名称，例如以太网的 eth0，infiniband 的 ib0。 +- `train_data_dir`：包含训练数据的目录，可以是从分布式存储挂载过来的，也可以是在任务启动前下载到本地的。 +- `test_data_dir`：包含测试数据集的目录。 -`PADDLE_PORT` 集群通信通道的端口号 +# 使用分布式计算平台或工具 -`PADDLE_PORTS_NUM` 用于集群通信通道的端口数。 如果集群节点数量少（少于5〜6个节点），建议将其设置为较大，如2〜8，以获得更好的网络性能。 +PaddlePaddle可以使用多种分布式计算平台构建分布式计算任务，包括： +- [Kubernetes](http://kubernetes.io) Google开源的容器集群的调度框架，支持大规模集群生产环境的完整集群方案。 +- [OpenMPI](https://www.open-mpi.org) 成熟的高性能并行计算框架。 +- [Fabric](http://www.fabfile.org) 集群管理工具。可以使用`Fabric`编写集群任务提交和管理脚本。 -`PADDLE_PORTS_NUM_FOR_SPARSE` 用于 sparse remote updater 集群通信信道的端口数。如果使用 sparse remote update，则可以像 `PADDLE_PORTS_NUM` 一样设置。 +对于不同的集群平台，会分别介绍集群作业的启动和停止方法。这些例子都可以在[cluster_train_v2](https://github.com/PaddlePaddle/Paddle/tree/develop/paddle/scripts/cluster_train_v2)找到。 -`LD_LIBRARY_PATH` 为集群作业设置额外的 LD_LIBRARY_PATH。你可以使用它来设置 CUDA 库的路径。 +在使用分布式计算平台进行训练时，任务被调度在集群中时，分布式计算平台通常会通过API或者环境变量提供任务运行需要的参数，比如节点的ID、IP和任务节点个数等。 -默认配置如下： +## 使用Fabric启动集群作业 -```python -HOSTS = [ - "root@192.168.100.17", - "root@192.168.100.18", - ] - -''' -工作空间配置 -''' - -#工作空间根目录 -ROOT_DIR = "/home/paddle" - -''' -网络配置 -''' -#pserver NIC -PADDLE_NIC = "eth0" -#pserver 端口 -PADDLE_PORT = 7164 -#pserver 端口数 -PADDLE_PORTS_NUM = 2 -#pserver sparse ports num -PADDLE_PORTS_NUM_FOR_SPARSE = 2 - -#集群作业中所有进程的环境设置 -LD_LIBRARY_PATH="/usr/local/cuda/lib64:/usr/lib64" -``` +### 准备一个Linux集群 +可以在`paddle/scripts/cluster_train_v2/fabric/docker_cluster`目录下，执行`kubectl -f ssh_servers.yaml`启动一个测试集群，并使用`kubectl get po -o wide`获得这些节点的IP地址。 ### 启动集群作业 -`paddle.py` 提供了自动化脚本来启动不同节点中的所有 PaddlePaddle 集群进程。默认情况下，所有命令行选项可以设置为```paddle.py``` 命令选项并且 `paddle.py` 将透明、自动地将这些选项应用到 PaddlePaddle 底层进程。 + +`paddle.py` 提供了自动化脚本来启动不同节点中的所有 PaddlePaddle 集群进程。默认情况下，所有命令行选项可以设置为 `paddle.py` 命令选项并且 `paddle.py` 将透明、自动地将这些选项应用到 PaddlePaddle 底层进程。 `paddle.py` 为方便作业启动提供了两个独特的命令选项。 -`job_dispatch_package` 设为本地 `workspace` 目录，它将被分发到 conf.py 中设置的所有节点。 它有助于帮助频繁修改和访问工作区文件的用户减少负担，否则频繁的多节点工作空间部署可能会很麻烦。 -`job_workspace` 设为已部署的工作空间目录，`paddle.py` 将跳过分发阶段直接启动所有节点的集群作业。它可以帮助减少分发延迟。 +- `job_dispatch_package` 设为本地 `workspace` 目录，它将被分发到 `conf.py` 中设置的所有节点。它有助于帮助频繁修改和访问工作区文件的用户减少负担，否则频繁的多节点工作空间部署可能会很麻烦。 +- `job_workspace` 设为已部署的工作空间目录，`paddle.py` 将跳过分发阶段直接启动所有节点的集群作业。它可以帮助减少分发延迟。 -`cluster_train/run.sh` 提供了命令样例来运行 `demo/recommendation` 集群工作，只需用你定义的目录修改 `job_dispatch_package` 和 `job_workspace`，然后： +`cluster_train/run.sh` 提供了命令样例来运行 `doc/howto/usage/cluster/src/word2vec` 集群任务，只需用您定义的目录修改 `job_dispatch_package` 和 `job_workspace`，然后： ``` sh run.sh ``` @@ -149,7 +229,7 @@ sh run.sh 提供 pserver 运行日志，有助于诊断分布式错误。 `server.log` -提供 pserver 进程的 stderr 和 stdout。训练失败时可以检查错误日志。 +提供 parameter server 进程的 stderr 和 stdout。训练失败时可以检查错误日志。 `train.log` 提供训练过程的 stderr 和 stdout。训练失败时可以检查错误日志。 @@ -157,3 +237,49 @@ sh run.sh ### 检查模型输出 运行完成后，模型文件将被写入节点 0 的 `output` 目录中。 工作空间中的 `nodefile` 表示当前集群作业的节点 ID。 + +## 在OpenMPI集群中提交训练作业 + +### 准备OpenMPI集群 + +执行下面的命令以启动3个节点的OpenMPI集群和一个"head"节点： + +```bash +paddle/scripts/cluster_train_v2/openmpi/docker_cluster +kubectl create -f head.yaml +kubectl create -f mpi-nodes.yaml +``` + +然后可以从head节点ssh无密码登录到OpenMPI的每个节点上。 + +### 启动集群作业 + +您可以按照下面的步骤在OpenMPI集群中提交paddle训练任务： + +```bash +# 获得head和node节点的IP地址 +kubectl get po -o wide +# 将node节点的IP地址保存到machines文件中 +kubectl get po -o wide | grep nodes | awk '{print $6}' > machines +# 拷贝必要的文件到head节点 +scp -i ssh/id_rsa.mpi.pub machines prepare.py train.py start_mpi_train.sh tutorial@[headIP]:~ +# ssh 登录到head节点 +ssh -i ssh/id_rsa.mpi.pub tutorial@[headIP] +# --------------- 以下操作均在head节点中执行 --------------- +# 准备训练数据 +python prepare.py +# 拷贝训练程序和字典文件到每台MPI节点 +cat machines | xargs -i scp word_dict.pickle train.py start_mpi_train.sh machines {}:/home/tutorial +# 创建日志目录 +mpirun -hostfile machines -n 3 mkdir /home/tutorial/logs +# 拷贝训练数据到各自的节点 +scp train.txt-00000 test.txt-00000 [node1IP]:/home/tutorial +scp train.txt-00001 test.txt-00001 [node2IP]:/home/tutorial +scp train.txt-00002 test.txt-00002 [node3IP]:/home/tutorial +# 启动训练任务 +mpirun -hostfile machines -n 3 /home/tutorial/start_mpi_train.sh +``` + +## 在Kubernetes集群中提交训练作业 + +此部分的使用方法可以参考[here](../k8s/k8s_distributed_cn.md)。 diff --git a/doc/howto/usage/cluster/cluster_train_en.md b/doc/howto/usage/cluster/cluster_train_en.md index c60876721c..1e8b4d54b9 100644 --- a/doc/howto/usage/cluster/cluster_train_en.md +++ b/doc/howto/usage/cluster/cluster_train_en.md @@ -1,129 +1,220 @@ -# Run Distributed Training +# PaddlePaddle Distributed Training + +* [Introduction](#introduction) +* [Preparations](#preparations) +* [Command-line arguments](#command-line-arguments) + * [Starting parameter server](#starting-parameter-server) + * [Starting trainer](#starting-trainer) + * [Prepare Training Dataset](#prepare-training-dataset) + * [Prepare Training program](#prepare-training-program) +* [Use cluster platforms or cluster management tools](#use-cluster-platforms-or-cluster-management-tools) + * [Cluster Training Using Fabric](#cluster-training-using-fabric) + * [Prepare a Linux cluster](#prepare-a-linux-cluster) + * [Launching Cluster Job](#launching-cluster-job) + * [Kill Cluster Job](#kill-cluster-job) + * [Check Cluster Training Result](#check-cluster-training-result) + * [Check Model Output](#check-model-output) + * [Cluster Training Using OpenMPI](#cluster-training-using-openmpi) + * [Prepare an OpenMPI cluster](#prepare-an-openmpi-cluster) + * [Launching Cluster Job](#launching-cluster-job-1) + * [Cluster Training Using Kubernetes](#cluster-training-using-kubernetes) + +# Introduction + +In this article, we'll explain how to run distributed training jobs with PaddlePaddle on different types of clusters. The diagram below shows the main architecture of a distributed trainning job: + + + +- Data shard: training data will be split into multiple partitions, trainers use the partitions of the whole dataset to do the training job. +- Trainer: each trainer reads the data shard, and train the neural network. Then the trainer will upload calculated "gradients" to parameter servers, and wait for parameters to be optimized on the parameter server side. When that finishes, the trainer download optimized parameters and continues its training. +- Parameter server: every parameter server stores part of the whole neural network model data. They will do optimization calculations when gradients are uploaded from trainers, and then send updated parameters to trainers. + +PaddlePaddle can support both synchronize stochastic gradient descent (SGD) and asynchronous SGD. + +When training with synchronize SGD, PaddlePaddle uses an internal "synchronize barrier" which makes gradients update and parameter download in strict order. On the other hand, asynchronous SGD won't wait for all trainers to finish upload at a single step, this will increase the parallelism of distributed training: parameter servers do not depend on each other, they'll do parameter optimization concurrently. Parameter servers will not wait for trainers, so trainers will also do their work concurrently. But asynchronous SGD will introduce more randomness and noises in the gradient. + +# Preparations +1. Prepare your computer cluster. It's normally a bunch of Linux servers connected by LAN. Each server will be assigned a unique IP address. The computers in the cluster can be called "nodes". +2. Install PaddlePaddle on every node. If you are going to take advantage of GPU cards, you'll also need to install proper driver and CUDA libraries. To install PaddlePaddle please read [this build and install](https://github.com/PaddlePaddle/Paddle/tree/develop/doc/getstarted/build_and_install) document. We strongly recommend using [Docker installation](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/getstarted/build_and_install/docker_install_en.rst). + +After installation, you can check the version by typing the below command (run a docker container if using docker: `docker run -it paddlepaddle/paddle:[tag] /bin/bash`): + +```bash +$ paddle version +PaddlePaddle 0.10.0rc, compiled with + with_avx: ON + with_gpu: OFF + with_double: OFF + with_python: ON + with_rdma: OFF + with_timer: OFF +``` -In this article, we explain how to run distributed Paddle training jobs on clusters. We will create the distributed version of the single-process training example, [recommendation](https://github.com/baidu/Paddle/tree/develop/demo/recommendation). +We'll take `doc/howto/usage/cluster/src/word2vec` as an example to introduce distributed training using PaddlePaddle v2 API. -[Scripts](https://github.com/baidu/Paddle/tree/develop/paddle/scripts/cluster_train) used in this article launch distributed jobs via SSH. They also work as a reference for users running more sophisticated cluster management systems like MPI and [Kubernetes](https://github.com/PaddlePaddle/Paddle/tree/develop/doc/howto/usage/k8s). +# Command-line arguments -## Prerequisite +## Starting parameter server -1. Aforementioned scripts use a Python library [fabric](http://www.fabfile.org/) to run SSH commands. We can use `pip` to install fabric: +Type the below command to start a parameter server which will wait for trainers to connect: - ```bash - pip install fabric - ``` +```bash +$ paddle pserver --port=7164 --ports_num=1 --ports_num_for_sparse=1 --num_gradient_servers=1 +``` -1. We need to install PaddlePaddle on all nodes in the cluster. To enable GPUs, we need to install CUDA in `/usr/local/cuda`; otherwise Paddle would report errors at runtime. +If you wish to run parameter servers in background, and save a log file, you can type: +```bash +$ stdbuf -oL /usr/bin/nohup paddle pserver --port=7164 --ports_num=1 --ports_num_for_sparse=1 --num_gradient_servers=1 &> pserver.log +``` -1. Set the `ROOT_DIR` variable in [`cluster_train/conf.py`] on all nodes. For convenience, we often create a Unix user `paddle` on all nodes and set `ROOT_DIR=/home/paddle`. In this way, we can write public SSH keys into `/home/paddle/.ssh/authorized_keys` so that user `paddle` can SSH to all nodes without password. +| param | required | default | description | +| ------------- | ------------- | ------------- | ------------- | +| port | required | 7164 | port which parameter server will listen on. If ports_num greater than 1, parameter server will listen on multiple ports for more network throughput | +| ports_num | required | 1 | total number of ports will listen on | +| ports_num_for_sparse | required | 1 | number of ports which serves sparse parameter update | +| num_gradient_servers | required | 1 | total number of gradient servers | -## Prepare Job Workspace +## Starting trainer +Type the command below to start the trainer(name the file whatever you want, like "train.py") -We refer to the directory where we put dependent libraries, config files, etc., as *workspace*. +```bash +$ python train.py +``` -These `train/test` data should be prepared before launching cluster job. To satisfy the requirement that train/test data are placed in different directory from workspace, PADDLE refers train/test data according to index file named as `train.list/test.list` which are used in model config file. So the train/test data also contains train.list/test.list two list file. All local training demo already provides scripts to help you create these two files, and all nodes in cluster job will handle files with same logical code in normal condition. +Trainers' network need to be connected with parameter servers' network to finish the job. Trainers need to know port and IPs to locate parameter servers. You can pass arguments to trainers through [environment variables](https://en.wikipedia.org/wiki/Environment_variable) or pass to `paddle.init()` function. Arguments passed to the `paddle.init()` function will overwrite environment variables. -Generally, you can use same model file from local training for cluster training. What you should have in mind that, the `batch_size` set in `setting` function in model file means batch size in `each` node of cluster job instead of total batch size if synchronization SGD was used. +Use environment viriables: -Following steps are based on [demo/recommendation](https://github.com/PaddlePaddle/Paddle/tree/develop/demo/recommendation) demo in demo directory. +```bash +export PADDLE_INIT_USE_GPU=False +export PADDLE_INIT_TRAINER_COUNT=1 +export PADDLE_INIT_PORT=7164 +export PADDLE_INIT_PORTS_NUM=1 +export PADDLE_INIT_PORTS_NUM_FOR_SPARSE=1 +export PADDLE_INIT_NUM_GRADIENT_SERVERS=1 +export PADDLE_INIT_TRAINER_ID=0 +export PADDLE_INIT_PSERVERS=127.0.0.1 +python train.py +``` -You just go through demo/recommendation tutorial doc until `Train` section, and at last you will get train/test data and model configuration file. Finaly, just use demo/recommendation as workspace for cluster training. +Pass arguments: -At last your workspace should look like as follow: +```python +paddle.init( + use_gpu=False, + trainer_count=1, + port=7164, + ports_num=1, + ports_num_for_sparse=1, + num_gradient_servers=1, + trainer_id=0, + pservers="127.0.0.1") ``` -. -|-- common_utils.py -|-- data -| |-- config.json -| |-- config_generator.py -| |-- meta.bin -| |-- meta_config.json -| |-- meta_generator.py -| |-- ml-1m -| |-- ml_data.sh -| |-- ratings.dat.test -| |-- ratings.dat.train -| |-- split.py -| |-- test.list -| `-- train.list -|-- dataprovider.py -|-- evaluate.sh -|-- prediction.py -|-- preprocess.sh -|-- requirements.txt -|-- run.sh -`-- trainer_config.py + +| param | required | default | description | +| ------------- | ------------- | ------------- | ------------- | +| use_gpu | optional | False | set to "True" to enable GPU training | +| trainer_count | required | 1 | total count of trainers in the training job | +| port | required | 7164 | port to connect to parameter server | +| ports_num | required | 1 | number of ports for communication | +| ports_num_for_sparse | required | 1 | number of ports for sparse type caculation | +| num_gradient_servers | required | 1 | total number of gradient server | +| trainer_id | required | 0 | ID for every trainer, start from 0 | +| pservers | required | 127.0.0.1 | list of IPs of parameter servers, separated by "," | + +## Prepare Training Dataset + +Here's some example code [prepare.py](https://github.com/PaddlePaddle/Paddle/tree/develop/doc/howto/usage/cluster/src/word2vec/prepare.py), it will download public `imikolov` dataset and split it into multiple files according to job parallelism(trainers count). Modify `SPLIT_COUNT` at the begining of `prepare.py` to change the count of output files. + +In the real world, we often use `MapReduce` job's output as training data, so there will be lots of files. You can use `mod` to assign training file to trainers: + +```python +import os +train_list = [] +flist = os.listdir("/train_data/") +for f in flist: + suffix = int(f.split("-")[1]) + if suffix % TRAINER_COUNT == TRAINER_ID: + train_list.append(f) +``` + +Example code `prepare.py` will split training data and testing data into 3 files with digital suffix like `-00000`, `-00001` and`-00002`: + +``` +train.txt +train.txt-00000 +train.txt-00001 +train.txt-00002 +test.txt +test.txt-00000 +test.txt-00001 +test.txt-00002 ``` -Not all of these files are needed for cluster training, but it's not necessary to remove useless files. -`trainer_config.py` -Indicates the model config file. +When job started, every trainer needs to get it's own part of data. In some distributed systems a storage service will be provided, so the date under that path can be accessed by all the trainer nodes. Without the storage service, you must copy the training data to each trainer node. -`train.list` and `test.list` -File index. It stores all relative or absolute file paths of all train/test data at current node. +Different training jobs may have different data format and `reader()` function, developers may need to write different data prepare scripts and `reader()` functions for their job. -`dataprovider.py` -used to read train/test samples. It's same as local training. +## Prepare Training program -`data` -all files in data directory are refered by train.list/test.list which are refered by data provider. +We'll create a *workspace* directory on each node, storing your training program, dependencies, mounted or downloaded dataset directory. -## Prepare Cluster Job Configuration +Your workspace may looks like: +``` +. +|-- my_lib.py +|-- word_dict.pickle +|-- train.py +|-- train_data_dir/ +| |-- train.txt-00000 +| |-- train.txt-00001 +| |-- train.txt-00002 +`-- test_data_dir/ + |-- test.txt-00000 + |-- test.txt-00001 + `-- test.txt-00002 +``` -The options below must be carefully set in cluster_train/conf.py +- `my_lib.py`: user defined libraries, like PIL libs. This is optional. +- `word_dict.pickle`: dict file for training word embeding. +- `train.py`: training program. Sample code: [api_train_v2_cluster.py](https://github.com/PaddlePaddle/Paddle/tree/develop/doc/howto/usage/cluster/src/word2vec/prepare.py). ***NOTE:*** You may need to modify the head part of `train.py` when using different cluster platform to retrive configuration environment variables: -`HOSTS` all nodes hostname or ip that will run cluster job. You can also append user and ssh port with hostname, such as root@192.168.100.17:9090. + ```python + cluster_train_file = "./train_data_dir/train/train.txt" + cluster_test_file = "./test_data_dir/test/test.txt" + node_id = os.getenv("OMPI_COMM_WORLD_RANK") + if not node_id: + raise EnvironmentError("must provied OMPI_COMM_WORLD_RANK") + ``` -`ROOT_DIR` workspace ROOT directory for placing JOB workspace directory +- `train_data_dir`: containing training data. Mount from storage service or copy trainning data to here. +- `test_data_dir`: containing testing data. -`PADDLE_NIC` the NIC(Network Interface Card) interface name for cluster communication channel, such as eth0 for ethternet, ib0 for infiniband. +# Use cluster platforms or cluster management tools -`PADDLE_PORT` port number for cluster commnunication channel +PaddlePaddle supports running jobs on several platforms including: +- [Kubernetes](http://kubernetes.io) open-source system for automating deployment, scaling, and management of containerized applications from Google. +- [OpenMPI](https://www.open-mpi.org) Mature high performance parallel computing framework. +- [Fabric](http://www.fabfile.org) A cluster management tool. Write scripts to submit jobs or manage the cluster. -`PADDLE_PORTS_NUM` the number of port used for cluster communication channle. if the number of cluster nodes is small(less than 5~6nodes), recommend you set it to larger, such as 2 ~ 8, for better network performance. +We'll introduce cluster job management on these platforms. The examples can be found under [cluster_train_v2](https://github.com/PaddlePaddle/Paddle/tree/develop/paddle/scripts/cluster_train_v2). -`PADDLE_PORTS_NUM_FOR_SPARSE` the number of port used for sparse updater cluster commnunication channel. if sparse remote update is used, set it like `PADDLE_PORTS_NUM` +These cluster platforms provide API or environment variables for training processes, when the job is dispatched to different nodes. Like node ID, IP or total number of nodes etc. -`LD_LIBRARY_PATH` set addtional LD_LIBRARY_PATH for cluster job. You can use it to set CUDA libraries path. +## Cluster Training Using Fabric -Default Configuration as follow: +### Prepare a Linux cluster -```python -HOSTS = [ - "root@192.168.100.17", - "root@192.168.100.18", - ] - -''' -workspace configuration -''' - -#root dir for workspace -ROOT_DIR = "/home/paddle" - -''' -network configuration -''' -#pserver nics -PADDLE_NIC = "eth0" -#pserver port -PADDLE_PORT = 7164 -#pserver ports num -PADDLE_PORTS_NUM = 2 -#pserver sparse ports num -PADDLE_PORTS_NUM_FOR_SPARSE = 2 - -#environments setting for all processes in cluster job -LD_LIBRARY_PATH="/usr/local/cuda/lib64:/usr/lib64" -``` +Run `kubectl -f ssh_servers.yaml` under the directory: `paddle/scripts/cluster_train_v2/fabric/docker_cluster` will launch a demo cluster. Run `kubectl get po -o wide` to get IP addresses of these nodes. ### Launching Cluster Job -`paddle.py` provides automatical scripts to start all PaddlePaddle cluster processes in different nodes. By default, all command line options can set as `paddle.py` command options and `paddle.py` will transparently and automatically set these options to PaddlePaddle lower level processes. +`paddle.py` provides automatical scripts to start all PaddlePaddle cluster processes in different nodes. By default, all command line options can be set as `paddle.py` command options and `paddle.py` will transparently and automatically set these options to PaddlePaddle lower level processes. `paddle.py`provides two distinguished command option for easy job launching. -`job_dispatch_package` set it with local `workspace`directory, it will be dispatched to all nodes set in conf.py. It could be helpful for frequent hacking workspace files, otherwise frequent mulit-nodes workspace deployment could make your crazy. -`job_workspace` set it with already deployed workspace directory, `paddle.py` will skip dispatch stage to directly launch cluster job with all nodes. It could help to reduce heavy +- `job_dispatch_package` set it with local `workspace` directory, it will be dispatched to all nodes which is set in `conf.py`. It could be helpful for frequently manipulating workspace files. otherwise, frequent multi-nodes workspace deployment is very annoying. +- `job_workspace` set it with already deployed workspace directory, `paddle.py` will skip dispatch stage to directly launch cluster job with all nodes. It could help to reduce heavy dispatch latency. `cluster_train/run.sh` provides command line sample to run `demo/recommendation` cluster job, just modify `job_dispatch_package` and `job_workspace` with your defined directory, then: @@ -134,23 +225,69 @@ sh run.sh The cluster Job will start in several seconds. ### Kill Cluster Job -`paddle.py` can capture `Ctrl + C` SIGINT signal to automatically kill all processes launched by it. So just stop `paddle.py` to kill cluster job. You should mannally kill job if program crashed. +`paddle.py` can capture `Ctrl + C` SIGINT signal to automatically kill all processes launched by it. So just stop `paddle.py` to kill cluster job. You should manually kill the job if the program crashed. ### Check Cluster Training Result Check log in $workspace/log for details, each node owns same log structure. `paddle_trainer.INFO` -It provides almost all interal output log for training, same as local training. Check runtime model convergence here. +It provides almost all internal output log for training, same as local training. Check runtime model convergence here. `paddle_pserver2.INFO` -It provides pserver running log, which could help to diagnose distributed error. +It provides parameter server running log, which could help to diagnose distributed error. `server.log` -It provides stderr and stdout of pserver process. Check error log if training crashs. +It provides stderr and stdout of parameter server process. Check error log if training crashes. `train.log` -It provides stderr and stdout of trainer process. Check error log if training crashs. +It provides stderr and stdout of trainer process. Check error log if training crashes. ### Check Model Output -After one pass finished, model files will be writed in `output` directory in node 0. +After one pass finished, model files will be written in `output` directory in node 0. `nodefile` in workspace indicates the node id of current cluster job. + +## Cluster Training Using OpenMPI + +### Prepare an OpenMPI cluster + +Run the following command to start a 3-node MPI cluster and one "head" node. + +```bash +cd paddle/scripts/cluster_train_v2/openmpi/docker_cluster +kubectl create -f head.yaml +kubectl create -f mpi-nodes.yaml +``` + +Then you can log in to every OpenMPI node using ssh without input any passwords. + +### Launching Cluster Job + +Follow the steps to launch a PaddlePaddle training job in OpenMPI cluster:\ + +```bash +# find out node IP addresses +kubectl get po -o wide +# generate a "machines" file containing node IP addresses +kubectl get po -o wide | grep nodes | awk '{print $6}' > machines +# copy necessary files onto "head" node +scp -i ssh/id_rsa.mpi.pub machines prepare.py train.py start_mpi_train.sh tutorial@[headIP]:~ +# login to head node using ssh +ssh -i ssh/id_rsa.mpi.pub tutorial@[headIP] +# --------------- in head node --------------- +# prepare training data +python prepare.py +# copy training data and dict file to MPI nodes +cat machines | xargs -i scp word_dict.pickle train.py start_mpi_train.sh machines {}:/home/tutorial +# creat a directory for storing log files +mpirun -hostfile machines -n 3 mkdir /home/tutorial/logs +# copy training data to every node +scp train.txt-00000 test.txt-00000 [node1IP]:/home/tutorial +scp train.txt-00001 test.txt-00001 [node2IP]:/home/tutorial +scp train.txt-00002 test.txt-00002 [node3IP]:/home/tutorial +# start the job +mpirun -hostfile machines -n 3 /home/tutorial/start_mpi_train.sh +``` + +## Cluster Training Using Kubernetes + +The details can be found [here](../k8s/k8s_cn.md) diff --git a/doc/howto/usage/cluster/src/trainer.png b/doc/howto/usage/cluster/src/trainer.png new file mode 100644 index 0000000000000000000000000000000000000000..6537d3d56589ca9f19a77a50a970e4b5275e6ce0 GIT binary patch literal 145107 zcmeFZbzD_V7d9-Vgea|aNJ~kBlyrA@cXK2KJSZS2f*{@9B@GgS0t!fXiFD^7j>J0$ z@B8t7;{Egc>wEnv9`>F+duGjAYp%K0S|?0ZSq2-E1oOs?8`yHPlIk~Z+>=naOmp2r5PU{lW~ZULZUS)-rqNYWrIK)Q zx1!=<<7VTa5y7OQq7rhqv=&sCl>X~+@S8A=Ed=5!$jK!BZt zlbw^36|`XW@NtHid9yluJow$o-~C8hd04pHxkBt*oT(7~nwh&ig$UEoAYSyZKflKb zv9td3CTEYorUfR*j=01Ah>e5&Uwwl|g%DQ-C0v|b-K{)4K>s2jH5BQgFAkkaLDuxr2uxW{A;=aQ^l5|GmY3_NC%(X9Z?`{q&>1p8n^xzuF72BPRY2 zLHw5T^;IBd5lkWWe=V5^X2l}s?2Q{@H{>LrXnG@U&HA-z4qtZdn|xz6dM_KigRa() zA}=Y!bmz-E0y(yZ993r~=0<)wbOz?7so;}?*{@|n)<6x5*NV)l6cMz?E+&A~U7fAYU>BPRDhN00jd zCimZR`Tz0C$%_HQ5uo2Y=wa_q5#Z_)l=8aP!z`A8YP;dFQUjTW;A-r@MfU0~Q1LZG zEd3xcdsKq3rrnlb>KgZIej6W7I=Fe7!WjIwIiBAg*7U*uFUv&Sj-p1A?zBQr8vb*a zmbq;TD=b`%0`W2`$>&cA#a$<2<$%2nm@ieqtPT>>eD-oKIn?S6`7$xfjq z5(qPezCr(Zb@>bH9$RmBAwS07 zz9Q-cuu1p5tRXs7A>{DSvros3(iC}dukBY>`N#3J3&Z_*GIE)JxgI>zBxaZYn>T6e z{&%+kZN~yrSimks0yF5wM;g=Wj=@Ten5DfRAE6V$`{3_6_#h#d5)X1C4qD;K~noalAeGd$s>@8T3SqM|D4*+E)3+|GfQ)e-7pEH~d=zOWEh9hW~GJ|F*RM zZ{_}5RsKI#OI*U!jpB0sTF0}`_$rds{@hr9d%)i=zz}HZPirzLLU*GApfwKTwJ!4k zm*-~=Vo9ld&4TCPP2_Xq-X&b zFde11H~Kw+zqaepYnip=AA9Y5+$?LNS4y&cIAfFU?w3ikHr zM|8`Y8$B$Ru;uK1#Cxn!#bXSBL$gUfWYoH%1D1Wy0E3G+3W?%p5bOwx5LLue2#O&!ZN*5&2rM zo3wn?EL6Wm=K8$B60c0FIBsqgdh4dOb(6GdzPXjPkxhBiP9xv^GmY#gC$nBA7hL1T zko6%1u-yUMssArerTH-vio>aTw`^zkkzyT}^V9vxMimjT#KzC9h&Wxo6N7yS>PP_m z>-BCCMUK1+^y0^-7lhR69x#g`Zhf^kV4GvgF#Tnk#mi;${o~1;U$J)SRFl^(5a~p- z8@=b&MCOh8y@if@EV@4^vYLkN+p79Sel5kQ%z5++ha1-WS>zICk#Jge_prB?Am1W~ zL@^zMr8f?|fTjB%CkorckJfivuUfKbD~ptzid!y@r_DM-gn>GcJkFblv+MMl_a86T zVGl-kDt?bs?pvCXDI^m?lAFG{V?3IXX;Z9IUYn?{{=UnRnU=YLS*vKGd7l=7d+QEa zLB{jOtvHhg_gr(2Z)%ErNe*QV>v_rID=7iz`(5!yo{7KaIJDLe*0&e59freS8vCy2 zdJd<{%5}`xYry7QH*d>5j5~@9JeRkQF+m~$SUQx^Lt#zCPYSijD2UAaw!(FgR^XaU;$_%Aw1ghKzg}XGkFGJq{RNWje z*ICmhD3bZ)Y%oV3UQx9WNaToxg5O2v@uqBRC|BVrJ9WkLzL8d@3l!0_r5L9>7yzLJ z0b7z>U>>-Muw^*F6($+T|I`);xZ2dwlaTJ#hw%L9e(K)HE);5lcq3mzsN<5Vt37K2H0p?z#S zemI;-Xdk?JswzPnqsXK+fW*l5P#D7}?dh1_1DUs)^EC?8$qwbs+V7B69(ifFH3R4I zhC*2IlG>fU<@9F+-&}!rc;V`+2XE^SzC9#VAIuQ{atdj@Md2+b;Z2~W)~2Z=k3u-h zm2_0^w&;7jb=E~0ca5qilXalWbgj~E@6Ruu2%glkLaXyf|S7_!%CrB!Ds z^d<9oagp-a=j>HpCOo^kIB{-~dsyc6U10U}@O5n!bs*8X5^RaFz9gQlVCUhQ%UfqV zZ%kW_D{+^WGwX{?jc3EH@hKyiwuxx7;*95~Gb6P{ZBs||BlDIKg6tKR^jK@d@__?c zV?_j*YubsIlh$82v2dUHapIIn* z9_0l5Q{g9#v;&e=_mxB&=7gtvpKkei`tXjI=w<5}S#QM#ut(f~^z#kZ_X;a+6IiqY z+u#l2m~13|<`R?j%d?{0kB4uO?MakY?hYWVdZIqcjE2Kgt$GE! zg~Y=zo88Mj)6Xp(G2*D|Kd5%pH;mR7KWOB8k)wl?W6|}N1l8uw;5d~LGWiF`NzQ)i zprQL>XoCdndyui#!ivc5MRgqq%_s-H42l)+d1{gpgmWqF3vb(S=Vk;7yshu61+1#lP@%B=~v4 zwcnXsiLNq2#;$BW%`Zohk(cQvJKLrBA5Q|T4^gl$DZ3o9kIG1v6I4P@G_8*YWm6`j1MnZW$^DwD5HbWo6Gd%G6PH$Rr}!NVd)=Jc@F|c(W|tGUvhK16yN-m)IzPKIA8%9?B%)2%xMdFSc5&GvxEs$v?r z&*jrMJvv|W@;tzC6RnpOu2AVZ{DRU<4?z-@XvPd zKDU{xZ91AhSnPlHGmOExVVS7zBhd%VrKvuq=<3UEA^2MMP9ZE`Y|(f&MKkyHl0=`8 z^^;bHzIoy+PAq3ip0x*IU|ql7ukQ>h)k|mpT3VSAs0z2q^*wiBVbrKAkJh!IZqW7_ z)8#0|G_?3M#@_2ardzQbE4xjCRcq}(MFbwi?9#W+v>VRzti&?NmwU-xh2xObTuS>r zWb#11zK4ZiWqV&t{q| zVW9PGrA1d~(hQEbtb1bOd7FG$Fj;IJ*d`8+hlcl42pOa2S*10gco6~W&&5QC&R;YL zLe?@KX;j{PV|yb=y2ZOw+6XczRT(gAm(wf9kZLK@T!@r}TR2&mwpsM!ZOd-sm!lqS zBo>uN7i2?Xw6o=;f{=o^?1Qb*6p}BPuvZu{<@ii9g$v%{n09UPzdHD9O4S!(qk5%7 z?2n{ugi41d82!U4AMtpVza7dZ6-r zuINO%WbEdOgZ76@>S-f|IF>M;J*6jtMvsF_30-QIXC+a2B#V$tuHfUXGoJReC3Drz zZy8Dla>9HczRGxh3cW?K#+1>A?m0H!{nGu&MIJ?AR)Qc~#~Yy)5crJECLigYvg{!8omThcT?Q*6tVI!LXR z$?JQ?hg>3cXZM%78hV4?4zZ5v&fiu_RU6nl`rcZ&R~d)2K2jJnn>#8Bt5cX~y_gTx zl1^WTYj3q`Py~PPg!h;>5v2|A$wh3A3y18PTbI8&S{6-<4rYq^dbgy#I>zEq^|@V< zkN4m@C-OSpfR4?q!>EEA*~JZ46^~D7_txSSyImpv$D5tmmzxy1QJ;hF85dyAD#mC# zIln3eX!}w&MSkNAkHZ*5fw^s|T&a$jr(ThL_LKJKZ>$+>(Kp{Vc-8VW9ln#HP$Nd` z5fYYnD%LzJJ9`yspU9z7%24zZFJQ`8Z$3*^ zRXJ0}$o_g!;j8VA2XmE$A`=^1Q)>Zd?n&Waq@U~RlU7ohaiaAtUQc#Z(yxZK#w8N@ z3U?Bj3HW>(lBCegX<4Yn)r0y@aa?0xCcD$xh|lYOB_icdue7Lj{P(MNagqm6lDL~Y zQccXg-hsmys%KBA@{kN5ZYrJ!QWUrgxjhVw;b!lDD^lfG^X9!$6GmT?8=nLPQvrXf zfqG;ngiqM6&trdykz`r==S=e@g;j~x%09ZYg^-=fp0eLIijy$icAhyIuM-|zWczDn za+1xGh_3dT{!0H2wxYLqXp(X{UV$kYiEHB~?tT{%M z+-4~pd`OazXGKK7IkrtUyE<^21v~x{90961Upyx2BrD4-X}q#bE63B#yvgpMTdB9L ziFU+6)OTwXyY5cYWRCUp9)s{1($~B!qV=1t^q~rOm(yNmyD&$$MTxpoFNKF;I z@p>1Y{wpoXMO~gfnT|jR2DhQ?F3~LWvkk=t64{&VIQ9a zLf5_WszVR@3McCG{$sqF5(@vO}FO(bE zhYmy(%Gvl*r2yOCh!HoFc#ba%(;6~5AJyLM{KcK4tU%M+n@P@3qV)_9en(wvkj1IP}3e(DTM**rI zJNn#)^*z3!GQu8ElRn-&4u)`}9las=qR0JXobtvfC7lxEaOgYHjZ&%G1R4XDD*X(v zk){3NM;Kd$+FS!3jz^(`J5G}ya+RiE{^Wt%Rwc`OJ0B+)k?fX8@xo9hL^neD718~ypC-s<~OtUSybk*Xe7$c&|~wboJI_J=|_-Bs~QsO|>9NjLcWuahH7J@}EX zPiXVFdZb59hJ5)>!_Ymii-3U`U11(WaD>IYVHak)$t<;13rYVg+~1RNaBgRsT0AvPES*onfX4P zpdR0mhfQo)Yldy2+B315;iV0hRP+c}#tG zTzA0FV6ZV~et{|7wmdFpWBZ+IzWRq=ac-UuRuREjjYrmIw!%KWRU?JJF4fxf`>h9Y zbdt}`R?{5_N+WuvpW&e|y#PUjXZ7EXaG{QP11{8j_w(}{c*8~S=aar^q$|^b`YUkX zUnPF8X?o!dm93E<7RSfzEPN2&h4xkg@~fcQJR^+^92yKM9;bF`w6zxM79kg2p0_0` zU7AA$?3@z5>n5B?o4&BH0z^}o??7gg&OX1Ar3RxsU&`WbPZ>jrMHB8h$*NvQJ{~r? zMZI(AQ#4N=EX6O}Xo~&)2<-9s@g{F92ZlqQ3~?acCr=GN?-Eov2qW~ME=;s zv;G@=SD&Q~hPcqV1IKjtSiJRrbcK)Uy518a%m^{+(%<8zJDlX%Qgb8r%XQ4+;ENfW zs>lMk^zQ2lNoVmMmbFS+C|@E?u^D*ALQdb}Nz6Bt`MY^liFy{)V}6`5@y1u~JmIaz z&AvNY$N4;Nb7&*+P#oc_g(a#e1w@=;-G1mEs{50+)R*S)#k2KjTL$4z^~kMb4EllY z%UlCRv@J6Pr!R&f+6!tb<%HDXHPP|;meAy2V!f!b5W!lFcn6=IgkkS~S1gS#bH$nWr#pzSPG z83JL>`MsQ2zQ~cug#&3uXq?LqEVRNUj|EG^+extC7RB)*c{M{pD|dbKh?01S&N~{P z#$w^Wd2OQw%^}SrwI(K=+&-hpEKE(3H_THcJ~4sH70c;ce@ZV;VRTrF>ApzebYS^R zGL~AzL-gV2P^jpa9%EbV@H#uXYO?%Q;l&-g2r*|Y5E9+qRM^Go7C{Q+V0*S7uPZ$B z!Rqx`$)Ju{>Fi?MBa8Z(6ve#k$s$R#qtRiHS1sy(?$L8By^)F4ye#V!bZ!~qok=`- zR?+7UZGeLqvlwzs257g|ev~=qV9mkS4HA5Vz@e&nV2|iT=IvLVb2W^J?4$XGx?9q% zRkek*G;BDl-5P=HKLNfFEGDiudZk?a!7BpIhyLO$b|>uE-k>wcE!f1G`X|X`O{X|L zd-E>_I@;1;s@Z1Wq=+Ut66i-n|{)QEI9ps0dr_SyLAiAQ^iA0`px~!(_rcJ!sjhs6oKdajSI&E5_cOE@~ym+ z?70mH(eDT5U4_#=qT?M~$Haf5;Lrh}j{ljyj#T6m&H8QNIO;qEW{2%QB+o7QDwZN^ z0dB`8IK=#|@0(q~IND5?+wjzXss|^k4c1IitjpjB+tl)wQZ}>(zsaeQ`KaPEzLp&vOpk-i$BRi*E)@Owz{{qQji3tg1BLszg8JOH)HU z;6^kQ`&}CLJ4v%^JH=5AV%H41uh1R@JFpZyx@_%mtL2!*eqBY(#AF?($Ef!qri%)dw`mgnmH^=*}jh~Kc0U|womKG{TfeGxZ zH}>m;Xk5j?P`?7QuS(_mUi9Lu#uD~U!^EK(TYeY5~Vg4`0uG|yKKKW-eD!n;1J+- zryao$4>0Xf4m)A2FVgxjRC_*-Rd&TY$Y)dlxlxvJ`b$zJqY9}+X%5%8`kCH3@-MUM ziL>Z@^9Wx;(-Dy2aI;-v%#&%C)`rl0KZGtXV-kpLx6<^)u|9lOlNXYcf3~&-mQ<35bo`)YQ$}_kWzNr?HZRXQu*!s`i0AF@_Jf- z8ia#w7F&U!Xm7cX6Kgd+Frb9^vvp#yTDJ6Ae=H$l$?NuyV|pwS4N?>`gL$6obcfh%4KnR3un_&#t{ZS{i#fsEnfO+ zkDLCjdy>F6HH)usJ57yfbbwgLqT4aStH3G_SWb=Ez87^V9S^wy{xjdBwS}|bD!XC& z3vW-AR4I})STgK^)A~>@37J_Qryc$4>GC!R(No@v3(J*|=j2CbO^a8WADZ63TCu-5 z=oq^h2N9e-oQ%0Vm-p(3CVJQlm1D1pA|xrPj2iHh;G5!U_XmueV3Ax5y^6@0rG9G! z!w4_t1J##c&kV;&*Ld(;?D=zsj10V_;0^ZtyD$*?8C-E_c@bDiu?@TZT0>c1^fG&r zx%hq2%Me?ixEF&esu2_Nau=7BT$jm9mCEnOLcR#vtsL?=9gZ2X_ZPNaUHb7cyyjNs zUwHZ8Eq}YyNBTMDk7d;2*d*NiJN1h*M$mPZ4M)>oUwg5k2=d{28d{k1q?Z%6nB%&t zk1x_q+;~NLe%Rv1gegbr8xy|`*9Dw^zu)6f>*oBU9u?2otW&nulw z@3t)dnb@;#=YtpOZC_!eEjC=W*5ZjxAv96$ba$VYdu<&=1dc6}l&;VY?($*?sV0KG zW#iZr=5Gund>LwH*_U5JXt;AFs5{f=A<&o>`>SW zVl%W89=k{OQzT%{qZ^T1b|#q1)EFrOFUZ&Zk*A;a)bgkxqK$sE%xK!s5D7u!u#<(=D(_gXi$?RH=%R?I^?h22vW#0!X*OkI13D z7zuvn3#%GyGWf$YoZ&K>#oB#+kL<2uYU9{VcG&aC`acUVd;erg%RZ7A=c*bN9s1<) zYPQ$1SiIe`Ryd6_P065_zPcrCg>h1ej@NPGhjZgP${l_&e7ae(T-a^sk$YKTexsa&5qJikjhCCuw-UJ% z*}VsmXli|xTRgi>ZEFp}w66snrVUNouTHwIyz<)Q8KWK{T|L(Zob8}tA5~JR{8_NV z$um!Y=zuJSW-Hz+&>NWvfe`h(n+_~5KjK4dDiz|{T$*8V-)7u_9z$uvnZ6G^-0Vgz5~3z5sqlh%3H4s7VM@bJE- z^f2W)UCpz8#2)0 z{~oLU(t5nR(1Ji+8_&35c9&tJfc&PK9;<#e&WBc?^k)ZQY4;1{F!hsL0f`QjBY#!5 zO0ULVtN*j*nO|0Q-V)g4>KrG3meVR>KkNl`d3O}q7slWWWZ&wa(R2#tu_IeXq;FpH zglKkIzYB3rZb}_Z;=G(3a7GFgd@Z=?^33$@<*4_8O@G?M^gJdODL(+mQNU?bbaS*qTthw0<~<%yby@pw;X!OHKugl zVcMfU)6cJP;bFz>w@9*J02fXo`_g;{p=}2~r!2pdolCLUSqFSO``DGQaD$qF3pjvz zQyPAF@aT)Zj=O%Gg!LG=?Gnx5Q%mWVMXK@Ip2I9CDq5b0pKeh)PE=&ha$^LuKRqK* z7KDvx5Grqsm#5P+rGQf*=1^fDTrXF4#v)Rf$l-HLOP$@BfJ$l-)Ekj;!2A%ia6Tei zCUR^`ij@Gd`o8i>eed~7qIM(Nq(dx|KDjj>VaomLno2<7K_48;=w1OaTNggzFvr0G zE;4Sac3xwY9g zuH$V;?L~bad{P3Hm)28kex3eRok6i~_;#uNJ8a~p$|S%rfHx+IIhyVmeaKPz%auvb zn))?2O-rYECtLHcxq*|;hh_ER^BYe!whQyq8u!|zebz4ObQkbg8G&6QI{B$vr&~YdMIYzHU_fhQND_%OsM=Ij) zQn|kK1Dlz1Jp?Oq(S~x@cGd)~s2ZoH1(UIa%k_s?1n#hJWo-_!7TA1wlHosea8#)o zzCgf<`2?iR-6}KBlg-K_vEvLfCYjKxb2c(aPL_2lK2lJeHyvUAk|s2t*(dijc^}|O z%{H~{(5)&Z9wQwj_6P~X3RCDbZ_HTq6(EG}$2l+URCZC&YiOWA?2fk2K*|e@GAlj4 zX#2f}vYKTnAkET{lk^J&a*UchpQ04zy*VfRbad-c&D-55YKithE#|n@))vgR^<~}+ z=?Ie5G{3TBi{V0h-J&_tS~EBLEp|a8@-J*WgA&=20K+-l8JGNWdGSjniTkVLM!<*m zRB9fZchalhQ@7D2VS+gtm7Z2=HYiOQ={q3dLUTax>iDALXD>C}AU|*g({RH-OEmv7 zaTdxGlUV(nEz~B_Gf=BBcd20-x4t(hFz%593#xfz;<~QI6R)rt4ciE`(*A-#J-GId zl_t9jyP;G8f4jl?XR z0v{MG%?7D8?c0ucz>0t6*3FLV)9Hswda1-_m-mru5jlKJZZWs>4Hr4+J)kc&sAB_Z zE2W{R|G;}0YiI<|1}}yJFE&lW(-`wIIKo?cnBj|hv5FF-)n%UtWhjkIUx(fyFFH-d zUPwG3X~~I~qYOA*fcuv22Ejv0cah*c%Y_;a9S+aN0L`CoMh`f{{VkZI{f~(hqjDyV zywgzLiFPK(QIHnEP1HLEf?;jM&6W-pkGy7|#zzdIQe+)(=I*}r?fuBa0Z8C}7ukL& zSPN2+4}qM7nXFxAI9a75YJ^&#<}&2jYTy@cr_po8#9O4WHNV_H$RTWIKa;g|;YTPp z@?QGF^@EyhVB&jjMYC_3&3xyqx4#aG;xy@8xn4GGYWa^Vd?GQz(2ShV?N<@WhAU%Dkn)!?R@iP8wDGb>GD7r{2obtPu~bRa`XG_);RcsA3HRu z?ntvy1Km4O2I!4mc&nUP&|>0K9fdc5w9fNSoU>~yxrCFucT?9^@rFBj^par7kUxFj z7ZwYk*pXgIs~vSh<(MvdgXn7OW;Pj!OsX`tu{nq|wGi4Sd~DJI0ZPVqNAZY}G4e7{+GnXMO91j2Cy@)Ah4EhQ}+~KMEV>A`b)14>EO? zsbyz#4@K{dj^OljV(H~%vePCNVV~Wm?Rp3)?gjJKi%jWSHz*ZZWpiSe8uKyaekZFU4Nb}ABM&9&6$(P6@VKR<^xqUF1+*4U2@MR{Wn4n=xu z6+F=;e5kG2(jvLx>Mx+J&%Qm>C+MF!?kX^wIIai5S9ym#eKU|IxKy1(+(;v)R{>ii z#i-UW*u>39Nw3^~m>tWoB@zmP|CSKtyjN(NSy?PKD zsiqM<@?4L^s+zoFx3qtL)s*`( zHF6L8?nu9K^9bG-mAfLRrZu&|r|i#&fDAD8)44qe$1{ybOfXFpAy|i2D>KZ*2rHs{ zK;hPg`FdBCw8O@@@_CM4W+tC;*IRPfuHg4d+RFRj><*0>cV+HV8f%3ZSv4Q2s9$l^ z;KZhhRVjaHARvCOkP*Xle>*sh}jv3`A7r zs-D|fE%CV4QG`<#Klu z-$s0w4{Z_9g5jp}LxH9s7;gDGM9<<=Nb^wyBhWnc%e3CT-0>6c*)`aC)q%-H8`-R` z;PHe7ir2d^u{&4+&si|?9_3P>y$2M{cVq%kAgU{-ELF)b$JY@4TFik_GpDZOOVh7zpE zRB177nU)k0un(Mw&Yoy*iA9g1q z)XLV7HZm>J51rze2D$vNrI*lZKgASE;cby%H9Fh}p>=_=i!WukHlT5W1h+CTV!m2y zCJv3co)KtT#ZmIq0Ih-6EngrlklUmDC+;Ft?e*Xz2_wq`X2RS;;@?y-*=S*=A7dx(j&WJ zS$g9N}3fOVhv9=a4gBB3|}Ig!jZYT<6pUVr%rTNO^<=xsn9-ah9Vd{qkpY7}?R zJu&!*hXtibF!)j_*EKP?+n%xaGYM9rj|`n}oMM0}-4yS))rK1n77xy`#ZG?LEg&+N zf8q@^ZppFqQ54&k78UB0G&=`$YCj&812l)u10}l%+u<0GBYB^T;E&JN9dy6>k?3Z| zy&Zk%s)LvKZgk*B`AC^3b4s@-Zx~h=H9FOgdlP->Qg{9>S^Zx{1UV8wYXmh5@&AbS z&nGCNj0%a-@?Z56|4e_5x&v;mtL7@B9vF%B%O2o3&bQ3QbRb0Yuig=#d>8>WcNgF^ zX{v2xkehju(viN%)b?NP{z2glBX5qC9;Uc&Pt~N6{Z-HLrzopi0=)QlN`>7$Z-8=z zRued_!Wlx95PFV;nnmBniKW&=Q+?}tCjVf$RV|T0Vl^2^TgGDl_`~i~megxFZ1V&h zntTr?C9ey#s05KOr3lYKD36u(uaYe&3nAmX-0i$cbSfMkD9Cyf{k2tfxYG9vWZEK6 zANL1d9)4v=F98{u0QEFM*70;ht}PA;HCz>>0{`Z)u2axOk>9-ONq10E!Idp<15Tr~ zp@1Gw2V`4wT7AcXxq>`a&v}2ZR78J>oMDgzGc-+LFY|kVOh|Ehvf&Xg zuICDJRUd&89l8}MJCMUl$MNiRFFo+G;iznU&Tn^{9USXwL87J0RP|TXm26p zK5!B?^gtUhonW8>4rwc=P0*d+3KoS6-b`$^LigSXLAC zn`Qp~_X=Mx^PeA7a!@=uiY~K%PC_*Y`LXBLeee&YGrz7TOd&%;{h*%V3)|S9uA6t_ zySZk$yqrBa$p%bsx}0kJPsiOFNK!wY>lZ?VwuNS;Z$LHC4ZRnF3kyO7uT?4QN0^Tq z$v-^6D|8NeHkx}u)>Kis$miH!hp=R-Jj8%w6mZ2|M@B}^(S5Eb%5*bV5cNO~d^&^F zJQx`VCp8|YDQEGA$xX3#`%*h=JLXvfa6LtknE1lUY#=MfG<*rsDDzvmr!qbfj% z$}!JC>|On*7q{|%v5bJ#YoHzg+3A3=aAR7RI>zKs;_)jdXO+1Dg}?|RXQ!zA_mgZa&!j?6_XCM z6W@2^K=z>rpM;uzK<@o51hD9KMRM(K_y6a*h9TBf^Uc9OSLa{ukCq9@UylEd#Glpt z^NC6gX#3xVbbl`Y{sg53q%j@FzpwhAewjhPYLCACs|4+zZ;#3W^P*kgzy0?F{^%Jf zNRUMI%Pj5lKb1ck5f7v#3;o~zzCbKY@@~ezt9Mj7!lR}L`yB_NVQFvsmKrxV8aB9t zV|}T1Y0{ZXUlPyjd+cfH{72VHMWqbp>?W@EZ(}>q5ksoaPxG~Je~Hl(&wf{}0L0My z(}Y3Ch94}yMwxq#gTg+(fOGGv$!5QkY3HWC9eI56sIsVMz_~)`M#kT=g$GI6c1_CpZ9X~O*&VT`d^!_QpQ?0l90*n~2 z5b8Kn?*;~**9+K?4y=6p-4JZjEx9`9*`y+S4Fi>T-8r>bBM2Yx8N`6^DTF*j2oun` zK*2o5QZNv5r^1DTEN6v|8F2bd?0#}pL?Ze>X7~ zdE!qn+x)3eXh;bg6y|LE~LqF$k$2;eUD|kUU!FF zu-@V);HBQEh+Z54O7bmXm>JZj}Y&bQEJN!1Llhvz*|lL3T!jG6w4yaa*3D(=>{OC@o#>TQ>ZhIEu&q>Ocfe9Q zPgNJ@6(;G|xX1t%XTIuOX_`@~=T1Tn>K4`}_)_EH1x?6s$r zjRoFJolZ{pdI8q)Re$uBK3f~eq}I@wjdpH)^5P~q^i>OhwKDkVT?s1zNvsY1@&XK6njaKK9IBPbG95mYOPgVwFM#)zIRF{ z?&oL!{apZ4&*pqqnQ9YkYka?NAxgSd4~)t|(HyAZvTikEe`mk4i*$=D$~N_SqjE!4I(-RDl=TB65QaXS%V_jjqJ7IcD-2Yu_=7Z95*r)iwEj<$4!;0Lji5?6v`uNhkL0Pj z>nCeba|u78xr759*9T5O_Z`1Rq~H)>!`v1-33az)R746_(;~?PIJLnO8OFT{k0kc% z5pm--K_-ODUP=H6-+e1+h#VoCEMS9XV=9vKN!QhiuBb$8hNqyaY2tQinQE5g;FY=h zt8L1w(=MREh&nUCr4FF;*SL-rfh-DMfaT^uQ5K*$@39-t%r2{xgBZm?#uMisidai_ zWt+fb0Tq#{`}nQ;ccmi<Pu(U!#>0u(T|}N*tSA>QRWjgocL0<~vy(9m&G7Gm(~l zb}xaRNBHFOa|V&}4#{8UhLCd45K~3H`4`NXY}<^LQRc2OiEDH80msFabGp122!$9#n9>*z zfEp~s2MF7u!5!dInLX47@wQo(=xXT#*!J{zn!Z}8flKq#u@XH@zeL#-Pjbk>6GWI6 zT;wrryW!QF6!LMs661KH?PbJgrWRhXACZP>n3b-04ocl)@|dVFYuY}(Y9-qR#bLs8 zpG>AfC1NY^pfTnA8P=_Lh&d%+_f3L+V?phq$Le?cz*uH2M{N5Br5J=z51#zEy2@FR za9MwV*s#Rjp-Xy>MaJ#g4dr|VTa++M5^OP@-NO3fXnnXV)nrj%zks)uCnifJxfn&8@bW8eLY*%$T zb@W^(xsYQ9ki|A2d;+$!HH{(bDK&RqKuiNSHjS*P(RZEQ1w<{2&aoM)C;HC&!4Nvo z{tPN4rL1bHqtoCIRy|XFZ7;ROJSb0bbUXXD1~NN7vB1pA zlZzvh+8wb6y=T(sHCA%l9xI&&CAk#36PCOVV~WL#!2|JdqGTjOqBqS z2<^3@Y^zPzI(dlhXywlmCJgj(B)7UPN+;7J9M+$NM&Mu>&7ma;uxzcTJB^!cUL{cL zi?z|lrw$QM1_-DhyDL=%ur(j4N2FnyJ+-;;?j*5&mSUWAnmw<{H|L!l2YUotTh%_< znSH2P@S|`{bhArztgRPr2R0uH1)HvWwqud2BWX#$&zF1XAn(aL%7SwJP8G@27{XrH#cF6R=LZVO z#|)qp7*^MM(U-p4Z#R$ueLSbslh$Yj`6YkXfwO#}BebdM+-}moJ54m54B_WgQ127R zGu%sZMtBq{rXO<`)?<-=bP2FU`zUFk@ZpAH*=g~t2G&WLYA(Q8Kqy6`&Q|P);`o zQIyQi6FW%S<*!Mb^!0_iwKoxnOKc92el@M2(uMFfy%4~}- zGh1JL#*li17+&1-Rkf+`;NY&JYMssS&LX8V5}SAkhqA+S<~W;&qFWUmj*V?EK_cMK zoW>qP2h&uFUL#Yd+UGzr!~jCvI8-jZ}0hi_&0O(@vr7yKBGnFjobVXa4>&SvR}0(3C=^D%rq*F>b2`8Re9 z!$d@0cc>QvH)f+NhrsUhwx+o4S#aYFCkO*T_&SEu?kDCmpXjnuE;xteE5tG{6Y192 z$2JunArK=s^&^q}&-l=ETVK^^5k*io%x^bD+h2LT%bYy`U$}AJFm^si+Tx4i5gc^A6ZC`ml@5bNDe0@B(71{K7(zHI_YCze|QXl*lbEv zHD;5auvSA8B5eYUVjHDN=~3U%)1HTT;{YrG3Zx+sBtg_u2&*-DIteq0^F@2Ib(W1jr(h{_!!z>rLw+!emF6^QsU78U&7$VJ1~ibU|bb@f9kZ zOs)XAO%L0VKL>do9nSPgx&TC-?l0mCe`76ov-BK}D4*=r#o%+#-yg?}_`FmsQ+lHNe4U4{>0-d)+~Ohk=4$xEviqEtucV!1Ex>%?yH0hpaRcGv`(7#0h*4jn z8-hGUl0&OszrH;zKi@RD`jCu4ca)(;nl8U4W-#Gz+wsxokbtbZsU08efi-9|@{V~* zis>T4-x*Riwsyo_c0kyoPk0eT^y-F{ANQrvaG1%QqwUZk_{nZ30NQ`2=>u8;1Bq+1 zix;@F%49iOp?8HZ?KyxhQ}0Y5B@;hfcq`9Xq$diFi(~Jz*jrA;RUGR_f9; zn3p+E^6x|2DcZL<)?)7z1LH?L!evu^VGN}_gGV?oyyZjQ7wVL!%{JK%W+4(7`l3FQ zx_r!BO`Jk@rsEx^54b$D5k4Dn4xuHet-|-#>P>Af&TnZ{K@nYi!zI4)sA-0XNH^hn zD}Rg7tx7C;#5eXpUtxlEU1|qEP(zInB3nUoZoTmIx|2t-H~=<90L!}O@vvVt!8eK* zDZ7b^t^Iwq3)R2uXz8l=}(05GfeTDe@6GKzj?Rl zg{Q_{E8q0FQOY!MU^2vKw)9 z36~dB@Ug$0^mtA8MLWolLyeGsn=co`pdzwSkbSfi>Wvv1&!qQZsSS$al!$PE0OC_a ztH>XF+TN&1%Qd|_nB=wmEw&1b6T99SSgm)03xKC26V7*A0gfP8sY(tvYU~;6OB0^n zIbb;YG!96@*oQh(Mhp41U#|5}-~k{vY}n|?5o9F|D`J5007CpdK-C}hi*JX$D?xGG zBrgldkOg08fX|?On|L?x%XrzxgKvODos=C0h1oU^vTTE3?Eu9)Z9(a?^bR0(fj!&; zz)c|PmE-Vhf z8at)1y|FykvB-ep65Y!76y>Ub>Vqk8+$MKFUb4P01YavbCr_}a)>D#>9QldX#8q5BoX+S124Xu0YA{>m6i_z)HU~QOWroCpf&G znfxdmMU&dER_HGrc;51wtx@)2OJQVqIaAIlAh(4;Cy3PsC3gd}HiijCfZ+Sk(`hH~ z5<{;f=(RJGr$nknCx*>DqVSI2AAev^m^=$EW#hy{PzeVA&5HrOiGdDrgkOplLu$ow zFDyO+-%|p7ye(QUK&!r$q6}bf0#*1m;82TJtJLbs*0ja>cr2ssfEu(&S72-S{Qgl@ z0L8PtPrA^m_FlLTsIeDsf41KXe-uq`PEM z0!nv-ihu}635bXw9SR6acS(v8(xTD|(jcL9h#;UyNWAk}-ur&;{XBbrKffREc*j`7 z0l4B^XUsT`W6m=#{_1wr-y7vIbstyAI?c*!mvCn&z*%3eGl`*?jF8D@r z_<6SLK7qrR=BA5F$%D?5wYu7t1a~$bDBA2}3D1<7@QLpEQuhIvVqcI|ZrSjw+48wb z*fQl>=J7${A=d3EP+|If9V+rVZBVTvq}((qEZzj%^f|Ahq(z9>H49A5jbiY;3m3+> z*pZXZKnV?@Fa$l!0|x6^U$`<7Q=)*xYvBbg5JQwLx$qjymq$>4Hqdk#-W4vRwa<_! zwz2Tp^ZDg?Y7!bUl=75pQbE^{>Rv`|cCG5u0#*m`y5STzyb+7si^Os$F+qyLYY2V= z3ABpDXQEQoN*DXyv?8h3Ukw67{^X;1NIv?n0IB{@KI-`;y$V6)2q^lT1|gwdQCFsm zC?L0eL2WdcO{5}r>$61()A9|{8cw`6(X7&qkKEjr^;H>YUs#6=5PYT-Idb`oQ}Pfz zOSQ7B_Y}(g;N%||0l?Z#0MpFOvY`->DE<@nw_h!YM6vQJ{wbO)a%b_uOq~tgC|!rV zx{Dv{t`TagtB4HC-Xqpg>sJ;RlRbbw5@PYTn#m?gLx)SO2%qy0b^=b_6Bzuag*aq5 z=rIJDV&!)IRz8a4oi|6$mt_}_J+^{YKe|}!pnw|McQfPA{D_7!XF1u2#?N@?S8%U+ zHOYnrJ2?`hTA9&n1r338y6l95SWskx72VAOylKTp)eQw2$yW!(vwz*pSb^+%SfPi} z@Pn?Ueh3uh8uYl4oH9_NsvPY59&+-`R~|~=qVJIlv`islYbyylJpg?g`RVhnuPA8W zULA2BDNW{!eHJYji^QO0_rRI;zrI)nI5=6S0yr~O>yhg7RhE=|jhXc`NUt78b-J>k z+{Y=~ls7q2Hhd?W>_)V~q(UlmLqcA#%v9ddIV~m?ydISogS@|=jkCBuB!f=PKm&YY zkhlU_4F2PczNN+AgJ6!n@ToH0tCGH+IK6p>4a1CO38Es^d1ecYubksKcoFHi5?{5k zy?e87SBALx#}Zcp(rSj=ztA{Pb;T3(sAM%FGvyj6++{Mee10EVq>%ad~hd3zjat^tFyyEY1gx3Q49&XK? z683*y2m4h9Nlli_1}Og;_5X|?2@QbTiu(UP^#8xpi;g3VW{}!yIc|i04YBK979eUn zmYN6zi1yOX$}=X|4%`%h6e{jUT9N<0qGb01L6KK(TG?LZ0b%uu%Lb^|*H*-DeNI~+ z)Rl)_qr(n}E-!`G|9&NAOU1^(Uf9|!9<)dHLLloI$T{MyQFQ;^jTd;IA*Zswe>nV2eywWIYhvxVUC0l5 zXMq*@2A-=C0yfQ5aYKL}?RrvxxUs6V>v5trUj-rqEZPc)`IiBWFveu$5V!)=-uLLN z%u`6BS#C?J~hK5audZVgf1T>Hb{@T2n#Fd##DfMs~CA><9BT-|BE#63R>BZ7#v-tg1g zkFq~LfIQt}#MJ&Fqnug4Xev{Nj*j+OgYZ6mjXfx7`UJh6piR>!_=Z5RcjzFbPI zdQZWmz*~^q??Orrz>~v<9)+eH#>ZBOy$n1chcM*G;->GvZD(~Nhpb7^iXK9Pt%f+s zSLIOebt#UM_<{c7!QPJ;sYM#%tdpmpAaM33oQwgV`j)IG=4+5un>@FYQ<=PKnE%%C zv0||F9F$hAG1Bwi1KomSX{J1d{rh9k69f8Ns$cksSasZzB?5u$A%+K0L5HHW7z8KO zd}iF|&O{02`D0>VDFhfut4r|2h27Ul5~$X*ufCVSWQlWF4`;-y@yrbl*mq>FTA-sH z^n_alE@g}jKIApPA9o?t6IUhd`Pm5B9#znwq*za1RRa`M8;HF&RNzl73Jx72rV*Te zB(v@+;%vsm4M4lRXs8QRAc3@96jiPqK>_)J`;Y2RP%}Ynzxw6%{mt37DNFG~!00W! zK3%8e)Tof+%vLW-xmDukTs#jMgJ3fB?FuR=1+~S9zN9OnWH*2a?9ki$xwI8fb@`VQ z>*JI|z7DC~!zDc{J;bMhf#{eZ0NC>=TR~1Y8DU^J0COD)OnbtsTokv2=@1Af&#dL* ztf+~`prE$&3OffBa}{i)QL7FHLpjf@6pvnD8lq~G)hv>QuZwc%2TGMu7B0+F#SVOR z0=j4kFm`+TXFZU%>U!7!kPdcEPk|Q_nv0*em;2{HIc^*>ZouujK3PW!UjTq*O`*0r z(VQ~C2VPCON8QcREGcqfcY!lJYG4|&RArFHlY0>Y^zG)~>vvbnUM1wt_|vu7k;XjH znOUD|m}nweE1Cl$_bb4&DuOJ;-m zU}KmgQ7Z7o&+7XJLlVT@{LX+%Q2;umCud zlkpDBN#gVpE(dUSd6n}e6Yo?6z12esJKKBe+<5T{!S;ykxWLdo$k)3M?OFfE;^uK7r9sNrYBV5T8+_}M5}dr98zDkxG0pFqE3gB~R+R#|g`QF- z?SfB32~1Nnw+8Ye_VVYS_9rm)^tpK?i0k`9F(c)XzH+7ic=gS~Y~qY$Wjea7Y|50B zylng$I|#}KJ)E@E6f6A!M&DgXw64ECv}soDSogZ2YV^Z?KNZU>_8W+C4>TZjII#@2 zq~MLq5s{cp&ktqPLr?v&%OawOAy0BQUDlW00=IwA_VY&mlFFDLU#qy!hK;6XwUl_a znl;2|V(T9k9e#R)<@{CvEhzlcfY)&BPT(mstA2TRD)=h{lqNw7!>JD@Hy(QHCw=^8 zSh@xz>dyUy)Fz@X%?#O37pN37zt_i{E-}NnSdJvZx@TPN4`qSO@MTV4 zpXW6~3QEA6Or}@PC2#+Lu-*?cV4^^tOV^JkvX$^(y-L}>W`C+~&#pU$UU*P|`!Jv% zH5^IkXcX65Za-7^y68hf=EiO31DN;*HNK&r9K=z5K14lz{oFWwrp|QneWgqdM~@Wm zM)usm3h{5^epGbq7!aD_wRCaeXUWBw36BQM;YDX{p!P49Oeg%_8PPAi_^eK9pUO2X zwXPxI0`Lx}pT11akm2th4U6N997omj=5Bf;JjMx!dlC(oGrcn$#mt67E z(}>FnPOxJI`uOWFlX;Cw`=Sjhi)QZn=9CklXjk7LPnD#Ii$Q?5hykn7gV4Iy~JS(p3#H(64j9?)|*Lu|D;v zN``vkg5VqK!LvM5Z7)(Iae!`6+*TD+I5rQMNb4~Znikw~Jww2YqmJL0NzbT!8Owr8^mH)ChX0OW4m7$gvc>^nbB&tHJ5h|(0H8J=pB|_=D zXRjb60GUdlMlZ&mCWBdtymfv&7-LVffQz0n9A~+4uD}ovmH5=-rtABLJRIwTj!Q)`hW5z75_>`jVy43C}=6=-Yu$hLBH- zWO7lHmdX6tHWr}oznU)5%N5iiJvmPqeRAiD001?JiYTqI)RX9s6NcGz%e7lSpMlDs ze4D`|Fa6Y19uK9535=Lo0*6bC8x~;eE(89)oo4hZpzMt~H~6<57+U%8z9L|fon5Nuv3Q`*oXy?y0Oi3YT^ zC4ssFIo2+!T7Yl2V}F{2bYTP`6_8I>Q?X-WX?LR*5nN^t@N{jZnQ8Yr`Vb;0OAT+5 z+}_2ntmR@|TtkZ3&HAA3seBfsVJoWSC7x{Vnp3m-P?3JeqQgP$AI6xB0NSNn5Rf$d zc~|9@-a(;5^{xaaLwH|?JNNsD5yx{CG*aJWVE3qXX&)lA#To)zFk}f$c*56*{jdOdz+-S`Y}~QK~k4`xUCv%U)y2g z$bYla1P7UD>4Jl2j&_F|f7EVjVdy5`ZKvueYCbv{FDEEKa?&adR&){%E);g*7>7!~ z84$oaKSBadYu^VTTKl-92jP=??VKnO^Kk}6QecCV9Ylo)SS()6-3r3hTi$J# zoIbt04uG!LcWBh+Fo-N@pe#AKtA7!ld;HVABSQNAvs8r1Te)OX;!pPOjr%);7{h~TCCn85Tz0%8;MPKKL+-z~-aC)ZYgssfGxnn$|z?)N`u z^T_g_7e!Fb+>14g_YokB5_2F+sr$$(KULb_)D{0-lx@!w{dqfb_<0bC^m4L&2B;Bk zl%kOGNk;!-Ze%GmlEJ90z9ISbi6jFDLg>;Z*-6hq%?!Vfix9S6xU*$@T+jcnL5;oF!2N4cA=2s@+AVNb^g`FJD}WHcSU0;v0$E0S6Z8|w@H;M8 zH|BZ!tfS7cN$Rh>)sGCU9i}oI93Jz=E$n;0N0X|7BV0o=Fjl*6ZP0pb-&7Si{M^?k z-c)tz(OF>|l4!97)w?~Vnc((6?F_W z`|t$5PuCcU5bSi*gN{4fF!<*<$mgE&TNYbip!>2^&e~dST19i#>o9jvg_Y}|4jVe;xABfL!NciG>_P{Jp^yOUoIy12iin}D{|gSf1g`O zd|bzsMbEhDeQ8i?&M&K`=*NuZ^WD`^KBgwm+-0y?)=j?Ec0rC(czCaW4&(+y2XCtT zh6Q;~O?3pd1Rg#{=Ih#Vu)-) z%+)UvWP-^+8usN*^KYzk*^O1 z?QY_Z(Ba=&bT|d3V7$7z-56lJ0cb4(#CF*vZO2o7Cy_?p;4c6T{S~8LWNJaoU@-v+nHO~M2*dk=V{J-Xla+4=r z{y(Km{ZuqUS6f>nD2Ho+!MZn)rw-VkvFL<)2@){;hZTXkmNeJ5ga8^ft<1o-gP!4) z7VFN@%?|;a2CPa@hJ!hb1~0zr2y8Q_cE$lD(4hQW1fuUbFq1P4R^U_haQTwyjy*mg zbu|usSzz9I=b$Tf)ai)J);LFlWJgFj%zr9L2>f;$#3W@8lnV~QhQ9y&h^k=0{FO^J z0ll~rw}#>R3s45tK|l*<%#y*-wyVLewFkuC=C@(H zdirU3HYsGgDiU_y(y`MB=N)7TO!}>^z5!68iiS6H&GX#_%9`YQJ zUhNo0{O$*ID7062dwjE%k5(HZ!cA^n%vE8`Pg{V<6Vn35*X;#S?OqC-A1F8v!CNCe z2!5Q@cy3X*1rsdM1|;-LL}#ERJcxBy8VP;S1Ay8;E|^0jCngXJ+wPEn1N4Eila=-# z5vQhEK)ui7C-}%zp7F|`!ZiwtWr%0!BX~_;+QRH2p#qd3I68g>w<3Q3=pb7utlX@+ z+h_vb0jZmD=xbS|N}_=pHShw4rGQ5!TKN{4Ul1X7{iufEhH6?+kJ44=M)l`BtDLvt zTFEOuTcc=qm=~Ggi;U}l+ys(7P^;ViHixn@z@vFw3w%tWrU5jHuH8I^KbQNjA+RO{ zzimHl!oWL1hTXLo%mv80l*NT$aBrXcrJZjT?4Ta#Ii~da z;4JcLfQ3#6R5mp4Ez0n9_VCS54TbD37-T$;>8|_7+$Vy1fOk1^z=X#3{dx);C!Q1N33lKg!jfY za*r_17tNfDGFvu#tt2{V48@YPLcZEM>gr~&h#VqujdZyBiv?UMbqrI7E5O;$Jy+Fk$Pm2p7Fyao{qD;XYXFk5o-aeIdGT6~g9SauFTI~)>oH8xHRF17 zNxr%6MoO19t>qfV95~%vdxGoD19R60c-{o->+56|57>8_ioP2yrly}=ODoWO%)G(q*+10LZ0ua61DF1( z693h6#VeDz1GG!un{xdqD$Zl?*7fn*fYdjeeED0!4@AasDR zoRBsjFVPyreSM{rSDkTQeq#?$gh^H~t=jvs6lSE2l?b7Dka+IyZpfFu>{#v03vb*w z`B`!)N#xubVOzo@UGARyAE?E#W+$PcGzzStqeRPU zqHAb(olE_B=RpG@ix}c1Y#g!r1vbVd*#z>gJtqW%F^m>Kj{4E`wKDehx$#=HZ zQ~A~#CSzzb6ElycH<|E#Uf0K4UpIK4^R&&Wn$9zw^0eG^hf>pb#+ABM$+Rn z1v7RT0rNcEn)9M&0J8L;Mm2UR){Ce5oi{+b4td&{Ekg~V-BFhWfu7>LB^=GG9YEMJ5z;A8$L;5;6h=~} zla0F677@SEoK3O_RZzD1_vhu`E(CX{Y7Jbz`*S()7Oxr_Ph&nO2bVr)K1Vs?fk=2z z(Dz$I&vXYK;yovGmsPsx&N5U2t|`2eGNlDC6pP=_i0uUNx}qD!k)&-ScqwDMpGrc0 zBy-U;WaF=&{r&}~b;3%8MT$Mx4iYVuNUB*s(5gDgxq(VNER1|z%I9<&mLF*j$=MH| zs%}vBT0pc8h)a!d(*;IRhgek)^3Kr5Uha{X?HPzR_{?iGqE)MFU1z@Cx^`F=VDJPV z*}qwYpq6!f5T}S7-j)sAgSagm?yvTTweedxN}gT&Z|ZB?-adT#j>#=?c}@nD-xpBx zo)pZ1Sd{4fLJJ|QMxSt=ph|V$Syw{FS9*mRxPq#YYSgq^>Iw?!)JfgByd;$w$z-eRdaLQ6fDYQ_1;rI3` zerAeaM3~an*bH`>(XJ!N!=Jn>>0EA=NNL2)OqYF%PmuMr!}Z-lZ@PJNl}yVyfCOT> zt|z3)Uh@w4E=Im};v;kMxMs32o{$?9#JgVTelC*Nnu9G-&oI_SF0g4SYxjYsD#_JH z42vr+OB!^Hb!4+-mUi#EZ}Iw=IZs9Dg{$IQaZktjk#J&zaY)2nj`oNM$H8owd0fSU z17_@@yKs&WTx_7Ip3_LHTDz=G=K-Q z?ItaN)Np00VF!TnIANSAm&w}AUu)*dv`;{HqxS zXS*U~nUQfz5RP4#cKghNjdFfn2SLE1A7O1+?Cy%~sp7|7B|bk{W7@DUVC}3;aA{Ku zSV50%*#P`Y84z36$$I%P`(9!mTkDKtb0DX8k0ywVUxL0#rnRnD_6BC&KynhMh7L9`2o3j?2`z3hxt1NXhv4KSl{i?or_ToCm|h#vVJ1?_Uly^d~FgJJGXlEZWlY zIPwNi49xH-S?F4*Go{J$v}@9oy6ogjnYr(Y%T6Z+vYpux&s=%^{i2_xEF}ew4r|{C z$5+3n?@B)NOzTi-KgJ@A4G@4W@FRqA3G)jr!LGmpzGp(uz8lv9{yshTYDd&gPGb|O zGSB=(;sW#;l$6I}!YXV1In_rDK6zATdE(s%9lOU&wl3WfETmhDJ@i|7Q?v$LvGzL8 ziRHIANFH-)Mt;-XoD;^Ox=reo8T_f6aE(Uy?gn3|*J*`MTNDqrEb38hHp(QSPoMJU z-?$x+DLKNXkGEuj_ci|2@*Om?(gy-hQxlT4}08`3oDG|rC2^)xPTHB4a1dZ zE)lMWjfZ~ERw$F<)49L5aAsyldG*A%mmX1y4PT>B*~G>@HGh)D^JEO=qhPiv@vaF6H_i#ckh!Auqua7 zZ`pMXP6%@G$eRqUmS#fOA;XpX){vIEP<@2-T-?M7nc8@-JQVI%Bz}=Zbq=-~pQd}< zeTzRSK)P_ZwVm`pY-O-eKqm_;=N;LPsE-m7#2WFO4^!FX=t3tAW%q#fKw^|lfv;&f zJh7@2^EO;x0+uIS*T%)9D9Y9EeA{ZNxmf)Ejn%0DrZVH_a1)FDYJ*0SAWQKe$b>$h z4cwYQ%4-!Th`bBN9&K7;o8OSd7?v`EKH9?5>fJ2JY0afTcayrD4*mt-b;ikO_gmj6 zr=I_$=x{@6CV)|w_E=X{n?3!fZ;ZtJNMI^mxI98CEh{-{^rZx%PUd)TXs%fEhD1%q zooh_N8fB3V?yVNLkDV&+{jzR=j-(9lhAHl=GyZ+U_>ZKNuVwquXHZYSfA5kv^`2Wd zXDoelfn&B}+DY--1jtVO;;zX^R5`lQ1iiiUk_lzyJ0D$H_Qt*;jk9CfCQ6>`TQgZ) zQKaqX*n7ig%Y7hjdKX3-b=j)QWy2CX%xse#3AuK)T2yp`zMs@VA-}~2%;Fv-Y~hro z%eIp~j982RfkeMMkaOSZCNjRo_dwwP%af zlyUFOVGF(2EKDl%^L|8*H3m5}yxExV>@d+HWGm-u=x(=k`*jFSL5_4FlG3yAOY` zrMUq1fo7*u9O#o7))LBd1unn?RB1FW-s%&H8?8_HE%y(9@6=osvF!{v%Ido5h zy}@Qi_HE+TN3W#!*SI!ayfkUnzr6mgm%b#cq2BnYy(C+AHE#OoP2y+Jt+DRa%6 z@BE2M+d+6Isl%^;kV#p#;XTjQ03)K>I)<<~WV+8F77ImJh?)CO1iNet_|a4!_uVM2 zV=X-CfwA{M+0>T8p*AqPMJRExtW9nvs1ijzId zshiZ~H_vKEoru<(Ql_%cULZe-WM6_Mu`A66%YrODo>!I19U=A+yJaNP-K=czOoiBm3O z>}R9m2?VPezHa2#5J_4*;8KSu7R?I-9Q-pgu>p4o!V@x~5ylMZK0rj z?_m1Y_17t9g;xu`zkl|b_XosAR`5GSA?^=iJ=ni~_bHBuPhbkCFqPLUXoi1?zmPWe z?kn`LO`ydC88r=OM5VkZ+sE$i(s9Un7Dy0)ZG3;V3+`S$0i#CMD{X1SdkKT=EP(aVeNAR*0?>V78G zI6tK>`cU&RdqkQHgn@S$#<8gYyHIp6mXjlmg2wI4_SyEYMuptS9%yC&!5wS#+W%hd zue|v_k}5;yDnp+P1OcdG1w_DzohJe*auyE1aIFgvP86_ADRA z&bCZ!gZj*e>Zdc;n{|%nGZ}JsIkiA!)&!;dUO2M8b&#ubX=jAgt3q!&l znB`Y|g*s3wDP=r?(>i&*GghHn*+e3xzB{o7AL1Y$aIlOUV;R9BBzSVpAAQ^=;Rr)< zb~kkS*9ulW^(AjBTGnTC7O(!903>f2o10kA-~wCb5YOd41;)GH8$dTjMSxKA{bP=( z%Pc9p)9xxHm_V0_aN|Wy43-r5ya3#2O;`W|dgX3yXpohbN|ltrw49|lLhAAZ;54=X z=GBgxU$IJ;#_bLzE#9uiLm>#=SuTj{o+)8HQJU#`d_~_!>h@Oy+?4DYJIHHi+aE@4 z%!850-TMfVW8HB*#|(bUdbpsqGckoJD>VH)eVBVm9~8TB0w$`14k8FUs7?z1&4$#vtXMM-zJMe(+4xHHmk#RfaA3 zH4Jbd0j;5g^xr!aIh{z5$E@eyM}aL4<^bA0$h_hbh5Yu1605(R+d!dqOfMG4rG(g`bcd(X(64nV z^7@A)H-M&(N@&A|H(O4mt6yI-61~{6gXhn1F0=p_&i!-qSL?v&wFUqvGKW1#VKTp?ed&R`ccuO+#Ab%6Rz@Qg7}B(5{CRT`3{ zLiir;6};2eV{_5@bZp87275bD1x)+s+T|{cK{$nS1cQ(c{#o??j^Mv6LKa8??jY6R z3J-8y$}{-P#g)xCG@z`1^E0v1(g>nJn*07fjtw8R(|0VYogw?VY{8a#cAD&dqKLf~ z*R|GsC_Uvjt5ST@- zxvuYzruMdY+OyyP>mi&T=R10I;3k@Jp?i+c>Ih%$ybf(o&2UVa(Z(FJUMX}2vA(x$ z=s@06Gj~rubd=PJ=RX*EVR<|z3B}ggXnyEdOwQ-rC1NzS+2y9R^*ywQ$Fa$4bk{_vv6So!mT!4A(&= zpjYyb>%tBjwbUoD+NOR^Za1ZV|+>RX1W<<>b#KO^W|M+8! z-^vs=>aa1D#+}-t#9=((2Y2p_mS1LURyK;oaQ07zi4X4KK;eln;uTwLaFx4BiADh9 z?K?ZXxxr+3V^W*kUA5qy5bG+wgkbG!cS&@H@(3w~`AfT=NYKoFf1VfHM*@E#T4j*) z6|EAQZS34PZT74CaRs}%yEGFEXUQG*Ss*)eWabHXmBgDY+@z+@cHN|e;i`+}3p4IX zqd4w|KjyZ7wrIC1)wT$9pGF44yn$;X!oTsMjun~lXKG-bAT}6PC#0XWWIk~P%Aq&Z z@meUk1S3kB<_FrLc7a_oZ>;4QPht{qubGHQ5U*G32Q)?%R-DKKYC5Jd!Mi@DOj(Vt#uwd1B|8aP{w zq51P~3REc;FRJmx{@Z)}&u9X084&9vbS8rXiS_>DqlXaEAt^l~_W%6f@R|+;iTMy> zkId_Te4ss0p6xdtC7}N6Gg^dTlQ4M9nEuBhAb;#j1E3;>Eg$_~pW&*(oHjQo`!^{5 zXC7?Gke+6M{}(DG|Oh4#|9-l3?Fg%3S{OwEk-&-V@6Q-(F2QS>^7Id)YORjV3Lz#6{y8N4ha%~ zX-T7i<>+Az3p;&oOMLGmrTv7^_tlO+e>yXiaB!x}_e9cIgQ^Tx~ZXs=$ z494(~3>h9|bb(NPQwmXJw&emQEyt{w7@?r1Vb8s6BY2K4g-vU7#?Lx&LYho0d8O#r zVO9TFkORaF`UhX`hi9IER!U4kE)2t**r%a6foh0l776kpe;!%FFf#_9&wjlKs1GUd zqRa2nOAwUhpN{}MBsD;#YEb^SWgS9D8M@zSzW=-}0&vg6Zk9ahB&_-0HkVEiOo!t) zoa+CwIVmczkTtC65{>(Bdyg3QUeOEEp$Grv2`wntzLhPUgUbJ0-hciqh5!QX7j#bk z^N;`h`6S>tz5l-_f=r88>=_<7879I5sGxvwaC;6#rHPqv8xGX2S%1HUf6)p-=H+-K zPkceJ{5$0Od7~lm78u=0L05iNlr)*-_Qx2KlOF1SeRf|8*6X1v&P#XxUbsNc7SUle z<&y8;fB9cOJmr8Dx&A666#10@^uSGl2@mA#-Ttpjb{UrJy^9<%*MG4G1ftmDt^}fP z<*KM7PMsM?ti{q79T<&ypJD1hTSrTi?mAhSkk%% zw`;9=uZ`c{{%Xi;>Lh6XF6876C{=+3Erl2hVxePjnZ$MhZA15dED~P;i0!Odwqgjx zy9)ra!p&*a*KE{0W~7KoNJ76)#EzRQtw8&P3@MLxx|)GY$U3-2LH+gQ+g1f+FgAE7 zkHrr+`g7WVRC@5>L8;AiLMP~;cpbQ-U+Yt;%nB&Vq-&QbGcMdqVNnVVBV~G94=54{ z((1vb0%AM&l;`CX0&E>-G956^41OHRh9IyOn(cAta19eYY=QbPxI_b~VuC3N6%Spy z%~b*X6#ytuzfR`jGW8@Iv!}99$O-Bz0(`gC=VW58K`l^k z0)%lAuQC7Rl5!Mv&qG><9GpNXe;z3@{^&IW2IoA$@6oqs3$#B{xX)+fWZL0mMVY$LNBNKu;dCs&qTS`Y8`IdLT}eW@>-##`T(}#FC(749H_=P|2FZr2 z@Z&_xequ!|MCZjp4a|aTU`crbsQTpg>_CC$st?1*bL!}Q7pN4Sgrx-Nhq*>qU9+rw z2remtPN@e#sl_svTfe?AlgX68#cN)3zR?{>-P@p7QF=5_8$$s5LnZWew#jLREEg@+ zNmfQJ!Q@^|3twUoZ!>;9*C`zh^!UJK%JNkb?3w6yozQ>vY9+L3_||*=5eB2AM0|K_ z>g7SqVDSRMwPbspmOjz#2K~n3I%IV+1Hz*zeUAVIfIGvWVRH1oqq)vIXyTjQ;s+o= z`q&X%g2Qoq5KK&8gAPP(bkE>#FQ#eCaG0ZNCsY%j5!?@=*TU8KMB@6R7O2p+s59Ok zf?;?WSmRbLYN$=NPWKeyd`j2}rE@FiO(Te0SnJHGP`?%N36Z?FS*EhVU$y^-gC zZ1SSNrr9_qIO(q7mNt;q*xx5$BV12Q80B30n8}pvsk|nJzOSupsBeH&Ks;Gf3@7m4 z+v1?h8_Sq~p@+NJn;x~{T%-p|nYf~lk;xPUq%f58YSSVo$2H>rSR$R}dXpDdy)q0qW5Kx>3*fj zf51t=j>{@paBruzEt-~cN_-jm$(f)18sJ4tKv(^v^@{D*eAh~Zhnhxud`+BNA5pKq z^3etV1Cz7FMTCek#!Uj#i+!SZ0a-@$eH2)^a>J6R@3ss*+gD!8WjK%x+zi0wubMM? z?cc<8RZ<*{3(dUVL*3~}L)&za5~n-V^-n)hqF5MvaZQ)9M@E-3^$8d&FfI;PsT%V^ znL4CQ72==;RbrsaRhb)#ef+73g`IM^msxa7i-W2geM~IFl|Dt&j4{FgzO6D^K)ny$ zOQYyV`y7*@d*Zl1i8ZeVbDjuEH=s~2pl}wSl+nP%ZM8U5(Rg`iCF*eO3n*U!8javL zj(7olY;)qNn;@4!w5J-3+$vaWbYx6)W@pjTa|z?;!@4Cs>sdG^SC=SBjAt+|PLK?2 z*r_U0gD1$w@nhZiI?PLz37Qr?2fb-Fn*wbI*CAOZSf}NPmzSPBytg^4A~sy}(nu{B zk82No8+JT-^@aQS9Y+58eUVGmj%2?G-)R6|{c}Sb>hukw~8jF*_R)Fc}A$Nzj-Xvpcw7 z#zUtnF#S@-A-$4IepHPhe86{p4%#13i@~YjeOA|Is2ET@TAm41!g;VeW>pT<_A~cu6c@|3D+pm&+ zhl32G0`V)Q74NGc-f9`Z6?d650dy8hFdFivr|y+re!(y;kIK$3>OMUaR%TK6GX5vO z)MNZW_E=VxNNk)FKCfo~n8j~X79JlXq0Q!FL%B8MsNi*QaY?DsV~$* zHNWSigavWaUfrx143sMNLJ#?^Ow!_~lXv$l7CWw=t%jg{gA!Q>+^1JD>tK)K_MHMx z4AtW~J2|PveY8uQ+_yvP=AwgY0A;s-asRfLrUL~OO#-lTXUGGqw*eCJV!Lp?{cixx zL7|gNG48JGK67bz&n5#k7HT&2{hfvrGGgFxaB>zeC-zD{Yl6GQpctd5NFMooX6%1U z$PNja`Cfp!9H^chwbRmbvTyU>+DN3k>71^62({gM3ie-_31P|$At2hy|A8Y)Yx2^T za6D9+$?j7Lpq7y6(vJ_-eo>~mK6g2R`-9@E;LJD@Mw+L0$mo)K4@!xF!h5N@Me0B2BV zmT~S)g`{>wk>KaUPRVS(?i+Q3DLQzA=W~?6{!e|BodUM%>3LJ2A{@vi*OT^OKY*k$ zOZDNY@b{?_0@A7o{(fg`K3lydC=6PywhdZm_T_DYMVY)cO6JZdJ*OzFyAl1nS5n;A z(DnB-3JBQcAxBsuEl$8tRQqJ;rYBIXvjwQA)p>11nC6)|TO=_9ipWmuUvG^|8uB@f zvY-d!vd?NOx_0!%&8sc29r|;z{cGv}nP!zm-Suro86uB*1)34~T! z3G*k3+dx-~w0s*k%peuSju{slTcH+n9J1|t{3(6@G9F>=_voN4r}E-y5U;cZnmpmAG0vbnFfp` zz!b0t`;qi~OIYF&)_|%WE^(7+XM2Erz_$4@gzedW;_Z@qOIRfER z5SFPq0e^#+&!hs&*sTD1G^^=oW?pEjjkF_y3mM+tYGQCo4}!XTyWX@r^PO?5%>3^2 zompYE(}+$%0R+3zmg*O+FO!*W!6L3~WHe<-WnJ-N z0~*bLhjQTxtxnVr|qkEOnX3582o7rL^tvfwNm7aORjDLfG!QdOEvW3br|f&=FHZH%0y zMZ#s0tUc%6SHSUnch=VgWzX4tNP^uEXVMo(P2Q^Jlwj|ZC4=obu?(BIxe>tJg|@^q zC)-z=gIvx{hlr3o?zcGo&~(z`!8;_*XfZ)f^N7E$Ibz*iFEYE$w!$RKL&Z=C(do2@ z9-a?-B~=Yhx%TFU7T5erlkq0mEABL_mg~CKM9@MEoq=k432M`HuzR$aY%6o{@CVFh z-nMW!S%Xj}MdH@c`jkvV+~VE3EALL!vUNP9=h6t$b_P-$gjj0n($$pLHm^W9b596s z&eF^DHfvezeg^CjR9!Yw*jZx7-gq@>M{FgE+p>|0@FiaSPAm5<- zS_NL$lV_A%?=wHqj@3Aud(I%dsige1_EVyFHp)Gs(rv}8FWmigDlQ0-VX{hxgt~sp zTE7C>9K(s$Po_c;_&{fEkP}$VR@1j6Mv|c`r$OQd7>%@TyPE?YxN|z%;oiGuB3fTR zE*4YZAz`P+MaZDzow<9E;63n|4E<{)5C>n!glNW{KU?s8F9+*&e4nq;N`ZX1L~o4m z9v$rUrpvI!Xqyh6e>Mho5facxa!5@KaPjG6Xf58i;nOx0*ZLZ;IOdH?O=fH(6$HTA z6O+>DELtze%wB~cA>{qVeOAlykT7D0`$8b7O=OcoUl1b@Ad5yM#{Dc1Afg`K9A*CD*~~hAavqtMLs971 zU76!~2sUh~Cl%ki>?O6SsZgHP+f`Ax!vv(-_45G^My)M5qTEda)NC`{>Z`piU|5{su~blqlKT7{cjHCqRycoRi$vmY zrU3qYw*YHGpZVx__H^CX-v%Cm}x)A&&%6EJL}beVWqwP{|c7TMNiL$vGpYdFJ=QhP@~AGBr{kl2mHVNnK=9 zr+nkN%pWPLAq3S+Ik!`=KBLiuAtSPPG(c(cwi-UIhZo@DVd& z(4QiLEal5c^BVPyR?u)08s)((=kX|a03Bd0{G$@lrNsZ11T7rXFfWl7QhA~%E&VU{=dGN8gz?1Q!$ueLqQvN zF2`R{JV0Qm7*M5EAcY)iqvo9=jC`$c0@O{Xh( zYh5|HrKcHR?@&5Lm{fxlxO6{bxvwuO>T|Fw^dO>7ma6(APC*doNtR9O)3X4MoXg9e zoPbI*^8F>P`X){yo$z&Bs3(RhHnC@cpWi^7L$Ti?6?JiDS1*cQqI^gYB$-=DkX5=u zKc#bq=MLOZi6uIWOjje#6)?aSS_yRpBJo?qZMsw#94vqpR|kC#fZ1BEunV=B3LHE# zZP1MeDSQJ^VosJ-$3&;=QH-ut&M|Xj`bp=Lr_TjaKzOxI<5pPc1J3hLHyK%I?`h37^mvZSt@;e$@!d4>uXXHriGgh&DR%(A^&M@3 zPA3SsNLb;f1|cWjysZm*5}3m(D6FNFL~zb2{+_y2%eT zvsRWYwdaZ+hzbsQvd-h#%OOaFW+H+fNjdN5)0_%_W)~h$zy<7)A_fU6uCy|1UGkaj;~SlkIJ5sWE1f?B zowag~ZSGVhAZ_yD(Mm(&52gVsiOzWnr~gcbWKt|pe?B&D0jI8uJ{FJF{VUh|r%Igr z64qIf&B8T+L$a9MPlMobMOimkEow-d>za?cDay5F<|!wTVI*S8SI?|_xDwv(yER$+ zW_E{!C;2=3`d#wt;-D?LLMK~OyX`z!@bJOY&vhP!fZ#yv8(;sX3)M4o^{Ks&4LK@2 zwmVs*u(K>wf?A=OT4<8Uj@98zdK~%h#lA5}nsS9i`Lj`rzy&cp`YS`Fg%DJuCR-FE zGJujS-&mHWllZP(*&B87bjS18eNhftVPtAa5$0#}ypum$(Qy;Dr1yidgs3S(1gPm` z$U5|RY#_YYho(oF^7r{gmAA$rV9zcT4a9%S8lZI+B7?9lNY9j6@GgIHal0m9QOi8b z&~HQJ=J8Uls_jLeAFR3u!t~VbB#r<#XcC9jc#HcE|M_DGs+S`wXH#3+?F`~5pHF9s zSxDWdm+@H^!Zx@r?LMyt9k|*w*aKO=5vZ3kp4@sD43i8OL=-?qF+gctW+vw6)la#E zKh{nLQPZWF2&FvndPh2?8;(WffsoWj&F=#qnEfIDq8?8ulB6eaf*Bi_P{LpsEhkYR z1#8c&OiGf!e*`rBU6w%o56W0YRt}d4qNb0~IkCwaxqSSDd6yH{!Ga4&No zPWE-+>=nboi=pD7;LKH)PCL6?d(QEa@A=+WgwUl?=^QH+naV!PH|RzNv4aL|{(F=k zKDBc~Nlh>!rwRq^FXwmci-+02tjOW#P;ToeA1hh7=kEj;Mxqo434(pzR8Bc2Kztl< z3c!da>HYkAhL6}r%zb(4lvg-e=K#dD!Hd6 zVb*IMC`}FytXKV>k|d%D!Y%-`(%rs(!xO_Gh}MMDHNPOVh7l3&))HfYPA7lU=KLCV0TxN3;o1jMq z3iBbDZF=0HUclN{sLiA+O}3r{I8;)%INcm!AQY@e&Oo}hZOhn!_>ce=B+l=2&Kb40jrY7?&+C5f`+n|cvM;s&EB-H$x}`Jv0m}?8 zPwfTO{+UzVqBa4=AhTpKAdhu9_rOx~J5?0VVwZteBW{Rzmf9ESbZ$U}*lZ>%JD;DrQ6|Ke)lBBsPl%Oo@os?d%v9#vJrr4u3t%j8F>Ua1F6AP|gBEFY$Vl<)ug=*xC z#(bK_6aNluk-Y~zC|P)g#}M_sSblLGvCjnSM^5r{r2ft2;(Dq0V)E|Qu5VBN{W1H2 zl7U@Dn4o8+YX5oUDY{a5(g=DEo{hsBKi9P1;4;6V_CbF5=E+DuR7`T#js5u4*Pl99 zXmH+|3q=<$IgoPp+tC+<#fkCW531$PAW|EUK|T4;?2pYnt0Er2mxk*> zqlPE^87w0oxLhU}279q#<;E8aOn*^Sgq1Ai_1%U$_$d1I1b8z?H!T*Q

>qd&Qx~cKaDi%@2sG0elMko_oO!sN4z=YP36p&C&MLtp2hPVGUQ<9gKN-C z+|9aL)f2OAR52CaNsw5eHJ`

r*zQskxWNV;c#^NX5Ru!a@>#4odlXRLh<y)6e9J1CIs2;Q3~hQ|E=-oo&qw+?*u>{j{@RdMx7RH@V-Q@QGwX%a z1xkCGV0vR4KqgS1_?v&z?+ty9*#6Nt@M?BJK?xEgVWLxUkQ8Sg$Il2)8yv_NPbDhu z`>uvrKWXkSL4fkquH<-&1czQ@-m=1ALOw-t_bK{gdDB8xT*;$drqH zBi$uqV?O>>j$$^4Y+lV`k=~|j3G7ZI2CD9bU>Kart6b5}qMQ!C5{R7UsX)m^s=K05 z@g}~eoAPz?JD|RV+Y~$R{#L8ykg>}~K#ieR01%0V; zyYx37vm4d-gwVs$DF>@%$uti_xaNpa%PqVy&ImE8LN8Ib;cu>3?D5=1;;gMYn?>!? z&Oq@qzs>3ouroL_htG7HRbc%#7FUi!HLv$HxFDw?ZMH zeq~I;>zCYJ#fWB;c0NPzxQYkFpG$YqR87dFuvmVl!>e^(h~f?Wzpuf;QXUM)N|uRK zRd~xEUD~pC-*T5Pzggu_6Y+zkg%n;7Zi=o@@+)`TM`s;2rx-&g8+@qf^Ox-#Hx)$i zM)(VdKqkTmeN>v0?$meGVVlXp1CAt0lUmN#22nk z9RMJLYTDipA)!7$OMPq5dR4I}YAK@!YwOH{zH~nWuM0ri_j&2T)T($Hfo&rlG@n@L z)0YxX9#j9T#(85uMoE^$m9{&4?JfhmCmKwSX@b{$k-=Sat`~=UubYFvMi`e|RLI1P zV=9|PN)oaQ4n@jsl~pG-%iG~R)iko7s_?#CJS+LIWm2I2tD2laN*UlSq9OKcd%p&k z%sv1PbvP-xNTeZXR=dvn{+d33gW>SL<*QGhG2jBJAc@C*jQ_>yA1}R@1w~d1aWM`~ z4U%pJ_7UMo~cmR^Y*|jV;v! zVP|rlUQJqDVs@*F9WIfpxJ34^gwKGexQ0;V#HjQJR_bqbmy+5`UqX%!HoTtsT4Msb zdEAG`UXQWj#Wz0g&IZF%_wz{ue}3HNvMfxSV@7xH-m9HOymGOcc;!AVXU(2@aRyjG)G#}r;MA{Zg~*0Ajs zI%k2fUCB#@O~8|&rZe{8uM%CSJh%}3BzK>CB7I51X9~9{W3P?0Pj{QessM8LYWvZk z(4DAS_p6*%1xT0s|vcHFWNLGcHTfrve-H=nGfTI~ESy{!dgmEA7sQ0dA+Y^}L}I%u)x*2>Rw?=5?D zYWZAEcX=V3F@bm2nR|1SL=UshJw1lz^VZ;tr0J=%B|jSsFquB1#6-+ zo=`EUEyWju{*$r0Ggn~;+}Je&2h^Mx+g$Bf@=N`2E$hzN^t{;&*D5xZ#H=;Ae((?v z_FPRMtf+UUpVmaX9_VH)aPXHxJHhD7rmNrN?ak{;<*W@XAF6YL@|)@=6F_P z(0qqcv5k{*EE$@ZK6ZXIS%QM-<^P;TqbY~i!p4dWgT$$^0wd9w4PytMBAYSqq@AK} z%ccg(Fc?NLf6ZT3@oHHqcWg<01C`S1SI>cI-l%au!bO?Ze`@(Mz3Z+ISYdAhQ9}L_ z{b(!A+mc{;B3Ptg#*|lDIqWar-#_x0l__CO{^!9OR6FYb7!c*}Khu8;7IFbI&|$@Z z2W;;l@M|8wP6uI47Q9vP*#eLV=ArVhI@O~;iGj;yh*5$*m%z;Nc4dX2Q~dng48I#o zo=bRIC2%=g7-gWPEV4iO3K;`&ecTz+jPn|R{=U5S>=^{qMM1NWLs4Qj0={f~oxAQN zrL*_|$wH&@_1uxFp>pFh6^$RPC@>bMH}9+INbOP>@c}tcr+>#J!H@15ph|rD9{BI` z&p}f{K!W0u?fnjCOE;R6$40R<#i3&UU=Tj2Nh!_8Ux6GwIl9AnEM@JScUtytGiRD0 z^nSAQwh*UFnNC4}#H_0WTYnKZCf9%OdjCuoIQ!MFnn2b)ajst8=ht}0F3tSOfrYSd zqy4?^BC+Nk#Q__TuoQ|IHef>JUq4$w;^4fsxf<=-9agNbD|9{>z%3dD)(Ma3*0@+`0QOt<#n>)g@YNa2U zi1SU|kw~NVDN+(xm>V(L7uRSIC;Bvv@*?c$KtlmCeol<42ZOyF$ElNi#jFqoG2VknpMTWJ;%I%<}nb zQn1?el@;an-6MHHMP*Ge9$~+itooJTFEg*zp$mN8y349eohmG)=g-MK@uuhSXwktk z9^RW&O_6bf#595|k|vQeh6oRce@1Z~AE`$jU)KL)03K53Vk?D8v#qOrLnO4*7Kx5^ zYC~C^sI<)Nb}%!SWP9H4W=8*+jx)V&m*OMlL%vdAlO+&Sllf_r9PYubFT2G|qa+1G z`z#VZ*Bex{o_(+T59D;jBjs3#cPw1SUmv8OIJBE{+sDrM=J)Xzhq?@8fek|f@Wupe zeZS5eS0*1d7CBpPw2BB!21fSRgpzVac&*V^e|E?UlYvR zmDWStoe(+CP949{r9#Jo9*PI;@RP1K{~d2UpP_}o4oa{FYgFzktb)NVZ-j#q9cMmx zMia3LXV*V6Xd-v&KzT9nuUBS&``xUS#$U!8Z~d1m!VC-QxQ>q1U}90U-8lvkvjlx1 zEe`*S>51aTJ!sz-Q6BT!7VU6z!TA!hFUNI`Qp&w2spB(b6Oyu_%=m%bcbkq7_O^j9 z11$;jAi;GtUV9gKbV6yM7M`B%UtZ^gaC3=G_mEywuSS$vBqYxlfr20rsk~?tN;Y42 z(St{k?@#7B-UdyCj*YHy^y*bT&jBnS& zBASYfnhn@!iJLy|dk5DG*sMV98L&>`$zk=)cMwm9m;$SZT0UdKFH>|P7%CPq6R&SI z#8`HZ`Y`G>Ih$%QPEKA6!~CA#+kV|joM{es6RKXtg*`V>8vRp98{LQrs;JH!Q)kP7 zYa0V=t-u3?g2u?$(zlWsIW(UyapP}{WsMbce5{SNagm_WaCl-KH{viPFK-gw4dvPP zqvjZq$XRti>-C|H&u(&WigCWo-RM^#mDmPr-BNcuop1rN<3+H6$LqAe?X%D49D}L) z!M$f5qxi=9)i}w{TG1lUVPQ!%KLG8#wpR1Ji4vdbfNP;v0kw`z-)Hc(^R|N-Nvgt2 zTmHL>0qquZN@%CAt!9Np2+8kzSSHB0%Gwhd|2kuTypCQ^h`Y^Rs$JF{8 z0gupL9Z&Y_Z%d?reccan{c&1WYV^pL!)vbUSKFzDC}5!QTu<=*rTF4j(K$-tSjEL& z=95&Z*(3dIjg)@hucdz7xPEOu=*Svedt0RHAo4a71^hYA&2iV_%9ENRAzlS^uN|n( z;`v^GYXRBeatgDkW)8+}bsoH$B)3D;eGWhWeXFqR>4s)%oE3Z>|Jcfw)4kGxHaGqg z6!>tVuHbODEF|8ZXlqiAw_TTCsd-HE){pPPObz#O9ulk(ioep5?N#UFZ{FX_1JL^N z>K0msH>e*r2IP^UL86*-$+H_*bXhgfADiHXhrTL#*2kYONpm6-GpP%w5+?2@7rrwj zj}_^*11GrYDxCZqn7vuwgJo$PR$ce%g&>7R?jo-Dv)}Wc;`Ke4O-fX1_+-|wvtvo! zji{=HS>miS&E@6FTC{&XINB1A9yYv70wIQ*44=hbbOj%uZ!_R9|97c-&xBKtfpjzC zyJM5u6VaBb)r$U6x|_1+L{|Q79Bu(B_(2gR%a%@VZ{jZTnt*fF{^TR^Aof0g-&}}n z1Qz-Kk)z{nNHk0MvpQU-1lK>P+rL1?Je`uL!I^%~o)h`+mtHoxQIj+-IS@^CNOlpq zCl>Y)hGFQmZ3T6I!u}pOXtLVb^e`dA_zP@%6hU3Q;i>7rGq)v8l{w$b91ZMmw4!20 zDi^3Y^0fI9!z6Tks5TQ_%j-k9_s_P>&~xv=iMc<&Qn3b~%uh8Md^Mtas3(9E6ph1d zPl%OYfmqGRkN*{f@CZ;nSrK+-EYffmu3Y;_9fr?rdcTQ-yo^eus%=Kd3fuVB3hp>c z3*Hw2-J$S~`_zusjCAp?^S7o-R6RUn$L+GF0P5W~84CNPvQV>9HB5B8#X4)yQjus) z9#8?l2N2{KCeRPC@Yrli1sfUxR$jg6)H+OIYjU=ynTTd=kY~S<7}M?M??|^nS@60f zzg|_T9+p#l?xKZdTtCP6@avu6mm^9c1@nk{cvtRytoa&Q_(qqDMww~h?YM5R<*KG%8_n7R`#e{Q>z>@EZT%+ke^Sd1Ob=MJ4{OmducyGUOS>cj#8 zyMo-O;MSYQ@5pAPx;{WcAl9Wgy>uQ`$iTh~w?iYWw@D2LH;+x{NU5|(v@pfh97Bvr zrpwm;uZvl9qWu*C_;|XwFB8&o_RVw=-sDz@l-Q7_%vik*A<}iVQ39%bo2bh3y)V#1 z<@h&IBaNZ)F91oFx@rnFi>(-FTJAMVeoMsD@=k$wHzS@me+AU-yJ=sO(RGztU{hY& zEW6zG9OhPlzXa@$BX<-6wbZ$?*Y7+?IS12P6_o@(w4VI8)01!*-UWI3X5qXZlozZQ zrX@~K)#7_)t4-5XDE8iy687X^2J#TOj$GTm)A(}|wA1dlsMo{n-*J^G=1b{7LeaV* z9v6+A%JGXT!t+IEj}9di3NMKm)5xuD2I4^F~BSYvUzskp7IT8V{u2D zzrB8N28X}hVFeh+fT6oBukV@I(UiGC5p$HT#M|i9X%XW<%6tx#_TPwO%I*0gPn;w| zdsi0LM@xby)EO4xf;g;t*2$Y@j=Uvg2zeldOuOv&NC~S~PMq2C#r)IK(6UGW^fJ8)$O1|`lguj@q+HRy4Km|SvCoX$ z=B*t4nEJ}hUct%2bUXajjziC~zHZh3mx~c9BWXUDf1zvZuT(ap=B!cUY>aIep9>St zlWT3n1GZrRyx776U9l+aJYexAN4e~yrtMcfcL?f}k7=GJkldZrLxUq_<{y{>ZxmqR zMfVkEg-hcIz#F&(>9srDD&L|7$?Ucbb;b6bx0&;9T$R0_jehP?jhD?>PuZ=m0h>rS zauIPa9J_d=3-`j9%MgN}P`;%WfuFmkbo`7ZVmLlfdHnrA?lF6d>$XNWCSAfN3RtgT z&J;LHWv3n|+J3Na54SQ$oo!m(DXkE+W!Js-LV}6sH=zI%d%X#1)uPh`ipJq zLyykCdEIF{GLzO_Ru_@l`QdARVm$Om)@sa;`6g_ukwdMyL2KObY%7kn5u?pvtf*V| zvBrx{t4g;0`nDmq(e{v7O@KP$j9q)h^3Aa!NJ4vQ_dJVwbq=Qq z6CBrMG~HEJh*%p(KX{TmeAbGJiAs8$B)y$B9Gy~GQD@~>0rYl+2VPHH-8oau{oDVa z1(51+=wJiH+<-UzNwohWHMS#rzBKsdCFHws9?;57FAJHmYr#yO{JY!p!C(042!~U| z+V@6FeOIQuJ-_b0My$jBkX0QS#-Z($!knX;JvZ!O!IKK|!$q&zg>5Rr+9$Mh-h>Mn zu49vOTGPvmsT%!k@{f=)dcJfsqHc;vgKxKm;osP8QM1iVE1jtyAPocqxP^O@e;fon{Q~YyrmvLrKKr5Y#8qUc zI-lHg^H4@!iqbq@q<`eXLPNB~)kX=p_>nNtQ25i-FM#~num=9^j}U=xzH{oF>4I}n zjkmIX%;eOOtw{=F5Y?D|?Lj$OT#1adOLJ)P?j(1Wmwcg$3?0JodWqCiuUU^M6iUYp z^IQxMVEu{A>QQ=ngjL}g8z$W{G2T|9@^~a%r^b{gkqV1qYgSw`89Y{TduZJeD(rk- zxV12Tzv$Vt+Maz*ti|N9Q+l$(ZQ|*!z8@*xu$2CNJ&ay z#o254pddJ@6b4+`ZZ^GRBwL`l*eM-`2&RIx)No%PF7i9!vKx#S@wI4*t=2o4?_r+3 zQQ|9%*KmK&kFIYwxNnbYbe=l2Rcu(Um$u7#q+0AjZvVZ!r}-=@dF;qR4L8|gYfG#p zfat~+8m2v>^p1-zY^{7~ZVy&rZt;E$U&Srir&`KjeTzj(OmCNUHk;;dD?5MPy^gnP z!LbKOB|eo8#&M+@RlvN7sv5UV<9Y#$MJ)yE8vjzvXve?5e~`~c2@f!s(JCy7ci7{H zImCYs#X_B9s?^VuI)`;Rjvl4QNehGLlCjW%O*-Je9b+8LV&x?F%e@<1bp5 z`MODdWe3gk$*}P+^)!H&(EJ>-TPZX~P0i`hIJVXR5&{V1tH;@4fC+f&OVg9q&RQbh zC*hSy)6uHuHSBi&Ah4fmO)@?-QktDFY@Dh|yvQtlT7ei$*h@~ z;{V4xIG19dr>Ote-u6oeUeYG%JGa!z8C>X(-6|`hMu&QO^x)Usf1%ghSI8YnO^f~v zJNz(xUcfUBrbA}zf@;urO!mQ1BZcC+CKrO0+`Q~c{+dZ`NF~MdH%K?b^)rC|6{QBU2 z7T)E~!wWx&BDZ0^-@J7P;MmB-R%5$R<)?Kw<7LF3y-!=GJP+9hKH@JL*X=u427_|V~A`yMp_X|Hd=juCTAG11-uc}@a59j2C(Z&OKgOtb_-T6I&u}AkvXz#V$9_`DR z$l*kd8o~XYr(`v)aXaeVMH-mhzL{5*`*Pz%;^!6ee7A6K-qHDJps6VEILZy7+sZL5N^nxeJ=b)L(;FUF(&Zs zXXvyGhvMwLGE$_Jkwz{-6>UuTvgXNUsVml?%XZ>-@pM;}OKO6D`ujUUE8D@LMry6{ zo!YwnG+vC}?Ul&U-@mYTD-N^`G1*IhW~&aMob5IQprNH@5XB0J5jz?hwXf!iFU>Yr z9_HrV?b|QJ(B<<*eK8jV*Gk1QQV;Lf{ic{vc#5zV>S^63@mC*qe-|!Zk8V<`GrZV6!@$IwI1 zB*qV=A7!4hO3J-!`fj$xIF9K0*N0$W07QeJ?2p--o;17y!U8Dn+Ll>DNc;;>0&xpv z`mF!4bacV!S6?ZueL1tp0c3^G9<2aR7sjp6d} zRT&tSUP3?)$Bs6%<_thWdwk($2Sckp$>mR8t0+G+FW;gy`0}+$&v~;6cIBmT&`3a+ zhW5wZ4s8W>k01<0cGtcCDPN(>g-@m8TqFq1R4F1n>}K77bC9oh*d$;KNCdvRAwied zRk{^2`;q8KRW@-?6nW7cw;rc#1qus5$f3dD<+HqmM7Z2&e`5cgc^)GlO5qX^C74H!N}u#!P43nh zyu8hDdQd<;=6-J7E?`5ifH%Cc-2+>uj!3R=8@(-XoZHspYpO=jo2If1G7;z}tnNKy zag8A6w*Lz{A<=%j41o}DnDdQ@jD*kZ%wz^7+m%#iZo6GTu-fCu#&B#iQ094&eA#G$a+Z6wLOkW`8{tMK2fr#UI0@RQPC-oeQg3j&X4!mCwX>GSpi$ylw44AMIs{Yj$8aDlN-in!{n!$fNNadG znxWfrmmkud@#{T<1>?ZPU`{HkC7P)4~P|u*H&)&F$9vV){ zx#BwspD>;hu9Uo~j@3P-GU{FV9S$C)7d<6Q4j{-J>Ag1`?}CnF3d@LX5vY}>`$9HW zl$X~lnufgjHdBxqXD~9C9B=DG=qDw`bvvzHEAU_fiI7OatU_yoWMZNi66y5Oi{LUF zyC^-f$RnRKOaqhR{Ft=1G+@@pf&zhyMEIobY4N8qTi-u)v!5-yQ*a&6ijrTuXxHCF z(?H2V-^S~wFpx}U+8tT3ht_CioX6CSBA;1r2yU7v$K%i3qeFKwIF}cK${Vpr-5Df&?+HEBT2C#+nGEM6d%17uQ%KWM8gd0`0>|aT(x9?D&o79|4Y%Jua=Y;s zbAz?SOE?j9ee2W=D(24bL7=_qOUHeOfwK4f=wW|K07`XTqj2Y}G8dLl6x!&nfP9i0 z5sG*O0RaEA#f2D=Q30;~In@318O+q^QH0iz9;owOq?I3tz&SRO|CxkdJx5$K_nEh( zLI$f9;5J9vcmzT|L#NVXHpo%`Wj?FwUh$I&CeASGNxIloom8hv`hKONdn*a`9ygf5 z6n6fcuz0v~UL=cbNoVc93(FXbR90;fKSA=yOQd(z;$2ZIUjlM}-Q7Y4E-yWIlKQb! zO<2UjraA(C-wE0NMNmpFBO4yJsvgXN$~_-MMNl=NCgr1BQ|totRCUfRGe0_h^7!;o z=ONq^`DywZxu}njFR+bw$|rYvbkcA9@!6xj=RloKb=0=5?(cOYQ^Vb1vzep%?!Lg} zsf}U7Mkt1HgcRHp>?w|4&!hi9FXCw-Sfp&n-s)|Z657b6u$dvxx|oH`Ipz?A2=)nE zWF{ejv8s%Ru^XY=K?XZRczi8-f%rcgIq^>U8CBX5cleKAE% z`CtP4BVe|q`+}?9R`^&|9pbXHnFKN>^WX`$bKv5Dl};ky zftq@|CoG8n67L#x=I_UPp~*5?#Sa%My}^!kjkv0Y51u*Po@vy(^ao;%qhu8 zUzG4!ZF_y*6*L3&*HOAwII5yhT+qQt1!TFAZO2YR|-Ww_v-meGYo?* z!e75D=R2Urw~N6liG+q!PnI7O%Kf6b?j~O2iQS}sy;Hko-^>t8j${~wZ0rx2zrYs{ zoghvA+C1VcVcea*75M)H3cdYVF>4R4ISar6R77JYsG#~HYA=GZq&q3^WTeT2$Kj@q z$$`$zTAJzY6B*}ex8LmyoGmg2!!m6+LjLyWZ#zd^&rTCUFmY2QnGL@qUfU1?8!uM= zFC!wDAq^r(Yg_?RV)4m$$MoD)`E0~kXZ9kZj{OSn;q5~UJd&QnI6qkJCO5ULK>cTE zJ77hxskfS1-6pb`kP#PzYTK`exSh{_f;E)xNn2awxIv`eq(m=$YWY+Uw+6%y=!^dv zJ2NsyDd|VvfpNaN{cBz1J1=7VT9=4exOv&*8IF>vDz@u-SDDaaCOVj}9Om^&*phFG z^G@Zq6dwW4Ffxrj4s!D*!$8tA+iOyCz(o!krC50A6T^2!H7G_?t(u z@*yU2V{vjIvv&@2J}yehW;m1P1o`F3FRU!DtEf$luxt|R^lhwSWkf`%CX`WJS*$g6 z2Ei?lDX)7ecr5n`cx<}d$p*(ORy7)lqNy$!uaE%Zd8 ze36>ABd{(KN`yKXt<06r(3)TY&I-*nD<0NRdgEfz2#u-g2{pkd*_J%})pD1l6F8Bo zE$OKEHsYaQC98L<4GD7Vd{3yHlw|ajE!i`YcQs>V1wIR-Hy?mYw!MmI#P={*!F4BF z?!!B>XKDG9B29ol<>?^+t#=ccGMdEx-xO*&vOxq-MJK+#Efet1MZvc=(~CM(A$U_& zMSHnLQrj|8vVj=qX#e6DOxK9V*fkYH+b>QGt42$E?B%QjB=`&qmHusz;`gI;g_(Tf z#M7W0aumnYm{N=W{Uhe*MTJ1)hR?ZWhuY4^sdEZFeBN*cB5Il=`c#7Aer}s2I2UF2 zHJWhV$Z>a|jkdG@*_sk>OAZ!`fUt_OK)ROZn$gS3X#X>=PCEN@G+|Y^`>**(^#$Q-rX8&(+!DlD9kO1S z*VkmPq}k@kN{X)OU_g$adLJm^PN1lbM1Se#DGxBiB7dXM>5tB>^@x#@tfdQwS|y)x z;)PKV7{OdT4`APS?)!q|bUy}Y#cBB3Q^~Lg!N|7LDXuSlPhcvs^A~<9{a@UnKPy4_ z!~R~k>My2z!~dF(jHbT#RQOs$qy5;=`B2*omp?oiKjZWTX0nsE;Zl4fqW)io3qQs4 z%JRln?fQ4htyAQ)hqM{>iUd`JQ^3s=Vt3oX>I#oOY~ znEAG;F|U9#r0W>dVLxbRy&sNPVzJ~MF6XAD@L*3F#h7NjapW~pCN4M}wmq2MKHG7< z(W%3x2$$oTqnym^3HI{Fx!F(U-J%1NV%)D8mAs}(A&$xSoJ!bQ;PmlV#@WBwmq)=3-sE|wQru6EUM zD9?Mv@;V3HhTB2r74O`$P8sL;8)c5DqF&^+(KF4E9V1#v>U%i_v?{klQf1W8Ho)cI z;R2B%%+Chc@pZ4Q0N)@go~nW&i0qw#g<(#-B=V2s$B+|s&NKZ)2v!ew!8(iXw&jnK zm{;>!w!F!W`j+!MAXe&IkcK)j$)KFDV$OAA?*|!+9S=M4!YS>PAEM-a!$%k58K(vn+8hNnP$VDDSqLceT@z>B-w>Me zdrQxRMjw%8xQc;sK;!Q@frJ2NOOo8xt17zZnkr{xYSFs3)lNJTC7-TYGQZOGF07p1 zRgS}Z?xAO#p1iIEyAyNy&@uPJvI!#-5kFLlwm2#CDAE7&c2*->M(RDbwk6?8-YM`7 zi(Y2#(X@%h8NDsor91nvKVs?x7{oqeWImKjmmROX9)q@0!e_4zF+#var}V$<7?vEl zM%)mQ(wgr>*Q=H+b>ziT5SS()rAIxN@BeO9w0!eXDJ{<-EIY=OXuK4$X35BR`-toJ z)5fjVHA;E?Uc-W%^8nipg9A|QOfSl}>f!v(4HGLs*Zn$GySJR;xk?Qjg{R`(cnGxR zhaAs4@#KkOcir-y03K4SIIG$NrL4mQ#r@_HtJW6>UgE$-&!Ai>_*D5XE(!MkGCAt- zGUX`mAq2k8DNw~g#&SQCpk+?7m5&n<^FxBXUjeUg1|b+~rB>zjqHLFcogS{qVuG!b;R3KmrTgSBf`=LkLT< z$(hTsZzfr6s}G~k#Q!BEH^e(EXA+0#15LrsFWRM*Fr4* z`u4tq@y8kwT8#_qLLU0SKR7D`UJ@eaO~XTO92b#Mu|4o1JpwKM>dv7@hu_NEkmsmy zp8BE)7bLm&$x3hf3*O;J7CQ-w=N_0WH6u~hTT}V zwiO+J(NtYq%q+Ddzt$ia?@2j+(Wc*B#usli=7OXDKU0dVh2$oO?MUM3cZO{8C8_Y4 zqE3|sfgf;GEA8`a_Iuq^EW3=k3EqZuK18RDKREFS|6%uRak@7asJYsZ#&H{6Jla)R zPyIM|$k}GKu7~gnFeiwfFxJJ^3_QiJv5T(n1C;E@Q)+cBHfi_25L4e>4J7HA1K?fv zk@^+0q@H_t_;b8gEgv(5mDBAc{(15rhKz2cjo+1G^zLQB z%pewuWXIVR3b*gqsAm7z_~mAMvAf?Nge#tnb~f9!c*kDOQEGQdUA9kLf3tT0F(8{0 z^X}wqb>fFAAVpE&oJ-zF8PDk%Q99 ziwJZvWP6L1*IfuK%*E|RT1Q74n4UXE^37^L>b6C)w96+A8N6!=N(xWJlT5 zBi{H>HSQ!cc{|w;m_zN}6jdka_S2EIgpAF5clPVfeaV(ST=ONP(QhINGlx|_B=;yR zp8tV5;f4sq$R!F+eX|8oha#U`j1YU^J z*08P85vVBCUFP0#ze23*6Nq9}Zc<}JU$U;mT2;O{V)R^O{(D@)0%DLtSAfZp*c!cj z_UxSl4S~kLo9reIH#d8kFg#OONVC9CNeLKAz-)}YS!$&OK9oC?wW`$HuVg!{3qNd0 z9*a$czwH>mqWN4rQ8x83z9}`XsMT-!&!K2Sb|#3A6tci#gvyiuTEaNKZ%G0O4`pgO zho(bC8oG4m+&=PMg{GD^H*iSL3^LHysr}wQu}eNMuVpU{PL`NBo5jU62G27FBf^MdwQ45*cp_q!rR#?MgTiZ+}^ zjNAvfX@Y;9n?;aTlq^!~-#J-o*KtUociap7iXHc3UjQTSuXQ+b?3e~qg%dT^EyI7r ze4*9N<0=dbK=y_-V&|KfSodD_Tj*rv%p%nm5xGKb&nxFof|m&Gllo|++cMcDCnnqu z5so`SnF!Q^d=()oD423=aU%Z(f}`Tt5kECmFP+dXjk!eJ7C@Y*GDM_%I%`)YX)U6q z=JGIml&^mpMnf1jOr6ERg2hitFDK%ehq$DyQlhAUIaE=S50)KXAo%bivEZs-)&3WZ z_!ikb;=LQDDKD$d{;GrFQQ~6PlvVZr66Mt&B)%*z0Y?@GK%lnknDfZ^W~9v`yHv zw*zLjkB`nAX$YJ7heV{E-U{c?K6~|aSOT$J3^{|NFGRQ8D2Z6yzSkVRBIp<$xHTfYSdW)RTO5&G0S2wy`!ao)nm78>K( zCsPP>+0NhbOzD9UdUaiSzC)3-eX5|0zx$HT6eC{Clxa( zz^hD=aXVvr)_l`VgBB#!%~a$95<0}SQ2y5!z?{&oV9<_;M9Q=H~Ge*o`#XF3^78Xgxe(Bq~-2EJm!TdHlBzGQkoB z|D}O)QSPldx3z&kQ0|Q_aj&GGj5esjILMOJ>O!1qF2)vymd$Z`be_AMbA&jTbzDwab~76JtLi2zxg3*)jVVQc^^C8_wU+yf@bINZ3Rv zND;N9A5U}%?`U6zXYrm|Us4wrwa^6L_hBlA8qNdbvq#{KKzLHS`#oi+YR}<60_ZEd=_TGVmMsO3Mf0XPMrUSUZ%*ERSa&`A~$zvi9#aV6paMmASJ7VC-VAA)X@Qg1}nJFu3UUEk2*sT7o z^}$PfU^<}l&dB{9O10GbsjnAft`DWT-3QW~z5#mFbBJhy;!8jN8%7A;JiZnc#>#(0 zSkR*D32Qn};ED2~=&xWIZ6;*i9pXf`2{HAkKz|K#kiJ9L=B<;m79~H#7&>N6fzBE% zN_+t43JOpuruFB2xn&{}|60t~`}RPmCH}OUDMU}ZMe__5+P!jE`A1@4`N;@LTQB<5 z%kvcmygJgKf>_9YK2x)ba0#w<^Szk(7>zI8pDg4ptmlyn9xKF(3~njuRH0UO)5TcXlOPt3k4vt(LSLT1lz6s1enT*sT*axt){hMIIE$6Qi`af!Hf@5+7d&r zatzbmdrKDKN=nT_nPS_OeZjA5*h83yWE3?4n|;N4QOUDf{InVeQrlwa?P>a~z>$f8 z{Y>2U744YutzMnnO;LTDtS^&o>#8e3<< zPcAXrmG8^Hw*`OAgt@1++D#U@PK;wC-+mDmfIXXY*z;`fYC~0y6?r9_+A!YoMMQ6XuwjztYVnQK5O$RY?7Y{=)L_(n2=UF&}U^FT(lc35K}GcRD9Go=(FkS2>Nq$Jd^4w0i@652xLQ{zI(v%1+te?J%M7LHpQsk>zaLY zv#=7Sz~?U--Fh*tql;!U;Y#Ek5LWVMS7SH38qkL2jRP2j@e%q5T-+HH{ZkuzeJqwI z5$KG^q=dUYK)F22IBO$yIC3w`k=iRgm+_&r5LeC=W#^x-%le}wHY@v^VIs2PKqe!f zV?vn_TngWl#W!@lbaor_G^~u#x3zKBNvl)y@?6<>O+@%8w<=6kjnDuUcroEe2etm$ z<}%Dc?f)cv^X#aDP)564=a zvsTVnX{jGjxlYZNEkNkFWcBf)NfbXPcBfJjQ3~;q76O+6iumg^=#7%B4r5d|V9{Iv z=cYwS2^Ji7NC0({VcO#f$RU_uLkU{thcU@H<^K;a~WJSu&1BQ9| z(PNu|(!iO$uaJOGi)ejwV(Ly&U*R;>my%3AXKnr(U&sC0_W7qd=uHpOF-bV+?jc5e z(lmEv4&wAbz`QxJS1|blmFiq-Zmfaw+YlQd+}=`}3;@M>IayyoV*<)}p(i2<44hxT z%zqj=^SWKO8)8~Y2nPr0_>iYsh&Q?3`x;w6b!}Ch+uVY%eVl2)VeefzT1;XtfX4fC z96xKAZO1R(cg_5xTl5(9a(w1m7aA1kc~)J5BlNh&Zy62f%xtminoJkm-Jp^ zq#IvLt~%PGRR_wl_Hk@zu5fDr6L~5gdo}&-<9km5pF@rlwGen~o#S7nDiYCm_+Ye0=1KDn>5km$GsBJWj*+vo%2bCMIlvm%Fp);ZPdW zD<^=F#W~gsDYVD=q<8a%7AlF#IleIH0Lc0Wk59Ug33AyF>aY}iJHx3$k@J&c}Mt2}5!-l~DTGJ}JI z6F>AzSB4~AJ8G#H6CqR)*pNMo{Uu*VH+64xB!jw;usVThw9n{6Fh+q--_lwT!rA!` z2$UO;?=BLDB?-?X-&V<&Vwbb>`Z_EjP;@`4v&Z=ffhiqvs287>T}^2J>7?M#x7Nya zyuuy8JK_~DN_$+)L$6$L{tCo%_$uhzl~p8TYU7{Ijgz#gS%HEEYl0|WK3g$n3$rlQ zdmpQth6#yRI85|enQm|njt+J2GYDGb|AMd=R$ohBgNgNgOUBjBZDO2r`JAN`IQp<3 z!9622_8PP~ALsc3Hn6Pl0e+kM+tv|2fGkqtE?a>?a3fe8aj$>L2V1hN7}Sd$P#}ku zbbj}E9tp(^WX%Ikmat9)&80NfCPao`(l6B{HH!!A==F8^ZA-xR5ZRkv+F@u7fZLO> zNuEh6T{~>|7kPWcWR;JpxgjkzpM~7Bjt|*B!oxavxV`&X5?4Rmi(6Ma7wETu^qUhB zL$^+d?@k!M@;!T5f-8BuI7dcS23A)&O{*!f?D^0fVZzZmX3k!1_uAZ_f+vs%kAjZ6 zVj7dHOy?PD?-cZ0F?-9EuMd}3)NR;~CIL{nTjf@FcwU}+VUP%!&)u5*j!x{sM0Sg^ z=+f?CZ*5rC-gVnM5#N_Prz~dcO6!{#HpG)*e@Rsr5nb5WQOxdgZ>B$nA0=!9)>P^0 z#lV6kgH>p_Dt*p%a2F_k9lo4CtAoPaQfi>fTH?s#C&NG{Rb+17ku=#k*K$Cn_p}j9 z`BFGiY&vSFiVpFMZx-;!GDPgjb>b|&**Xf=&WW6R3*Emys7zD~SpfL(WJ9NxA)d=P zh5q>q#Epz!^rLTnky=tHQBI^+DbLQYPp`{p{rC5>ClLz+zSVk>PQQnbx4H})m(IOB z4>vDuNgAb;RO(O24ZAlysFP%L%}x~fG7CR~-26%8fFW7Fsp&mT6m;6zmvHRPv!a5c zdrs^AO#@syciEuu2slt}AYk9fg<2TO=cvz!Z3})&%Vut>^6lAMY_3JqvLTt#LmoF{ ziDkYP*2H6~^EX5kvGqr@vo}&u1Kptnc}9>Q5kdAKe-<2mY)dX-ur_-CE@NL(bJ|mzm=99#;=;9G?LO z|3*j}_SHIk`K#sGmkZ2y=$|-R%wnOR>z+1hzj(Q2ZG^|rb){V$ zmPbrhXZ4yw;l;e#0E}x;Z$4E1a!^WAvqks*7Bs;JmHDsKxQ>jmho=rr(9+(!!K$vh zFfe4(^_6>ZEo1|X!nrB^6W|RV+DOa*0^^Vui#e$B&8Brb`u>*E(o(d;6kEMEPIy#v zw8SBFR}y>&)!fT0GEX}OzQCxs_xE~n+T(qs4}SILAS3^vYhtIM^H9Ai93IuIX)^PG5C04y2{^nvu#W*=nPk(DSu{TiPxBB==Hb5ky1t@!NTx#v5 z4XbB9Z}E7G>g`^tx8zq}vblLWK25Y9K8^h4DW6ixlqN^*KISy&+qYt6#q76VS8nEC z-xSsWJFbgcd*wojV?QJ)oLoy|^Bcy;`?+K?{a<4d-k44)NTr{|BJr94#P ztK%1uCl)#!oSD-QkW3a|-%A;?Rih`8yN@V;Z!s<5Un$gs{;gbwD0a&|>rnV6tjHL0 z!wx^49&}&1p9}t7=BYAzGk9>w;HF!KIR$iH)bnt@e7j}X>bRpwAr4^|711^v6{|l} z`xJc3Z>xork?Kr|^)-JAp2cG7==CNcr@QE#;9_QsH!JH?SbLp+2*;sqN?@{8L^ah> zm}*wGUOl)P_L`GZ3X~y$kWv;d2F;TuacX>;433Puj7DNQZq&@kA{`ZbfgzvEI>m01lV zP--LADrtFFXJ;2ITRGd=k!&f(TYehf=V%eyTqJ%yIKJb_%n;82qujVhUur&ajhC@d zs>P}W%!09q(Wj{Kilq;S^4Ep*`t`PdKQX)4-1HeI^ID$Jy3xPS`cALKO*ufv9IN5w zI_pbLJUKoePdf$Qsdg9D;=s0NZq6%++2MAd&=m1|yv+0H_`xJCW19Oe1>nAbg|_H571^hz12I6Mx~?+L_H;j(Y$s3cerJnWX3lHcR7- zHqa9Wu1)*0BJ+7JhQoadZ-*xwIZtbo`5VkIMM+GIJ`xfw`?$O@f>~IyD6VeJQ!fRi zNaALU0d$imY}}w}wss`nj`{!idJk}_`~QFZoa4kHaqP{p_a>CGXJr(X8B!W{5+xx! ziXxPdk&!}+N;+mzX;E4_RzoV9D%J1tc7Hzq@Avcley{&^x$f(_@4GnO@7H)fAM2U! z+I=el$DVn7;fSAmvV+r-_fK6_`&I%4Eo*UvY3yD+z={Dou79v41(pDoa9l?>5p}qj zL8PpL4d=9&NoM#yer6t;X-^rZ-7Z-sp}5A?*gXoMqT=w~u^Z9D2E&5C|2{r$uvJd4 zt77*U7{*n@-JuG8B9x&)0&#m#KALmpM2jm~SoSnp+AA&`?R7MdO<#qs2HW&)O*IFC zK8e$Bcln#lBCnzOoK91;2mNf4zQh@$Nbb5tyPO-+jM++;1`;igF!ymXss_Bmt_>oa z6;`Ko_3)JaR+7(0|2%U&eI7aq7UzbVE*@Ih+q`0`Hw{NytB@7ff`Zeg^LD3FbdawFw4w=N@CjmO0$4px zRqWo(;E1{dD}I1;B@5c$LFzwn73&R!HFUmIg@~FH+eSn8?8@CFnWJ;udHFZr6^?&5 zP|%emz2yp-y&#TSbf5OquiX86i5&WPFPj%PQQfz*(s!}r28njhh&+;zNRV6)t5&43 z=g~Sjnnt*IK|k0tgK+%+$jUN8@20YX^zv8kKD+cZQZxFme;{Rj0%c26FodPMV8R-} zSNh>vmEvu$q8I&Se0Brvev(FG{;5f{bxWk06HZ!O2z)9oWE13eF7dBj*D#E&&9_E7 zuVckI%M>n{9PO2*Y7gjhGCpTS9o$%1r~jRFQ7Msx&8(H6V*)RNc{(TK)-POe3~K~O z5N_&#t;WkZt&>Pej{#Lv_v@(%W%~QZPUPdF4I7W5JaS~_IMdU|m1A_Uha}$CnG2u2mwczaJ>9${j zAJsnr{=rgu^}|-P!E#sEjKxK$4La3bR!G(6RnSlo?}R7Va#PF>_}d*I zN_4HT^|>6mU`NGt91%b)PYqH`?BxT9)(zSbzOt(!&lc;h26+M3w+`w={8!u1;H$&g z_0!;SgdaE;>2;so<`T+}zy-*(fXE!spNrl#H8k^b^RmBo%Qyzy?i8Dxcj0|?*=RUs z45-Y*HD22$xG9Uo4d|J2%gXf)P94RzK1%E>{(4CQ6X~S_pkm)xTu3<10a9_DBNZQ{ z$~lvGYY;CriFmol2a(XaWl+0!GHY5Os%Y;>{Vuiw_G)=j6EFBPa+)^G(@?RojTqLR<-}ThY+i zGwi8&r^=L{K;V*DZHzm8TkRR<^HF@=A{!X93$zph?2Qix16}6xEm!eRc-8byM`isC zFg~lF@LlO@^qxI!fBCQMz1lU(#>~-1z|4dzl_u0G>xF|3e#3Hzck^dr7wza#E{#hv zH%p_J9b3ZJ3kwx@-#yy(TlE^15BpY&@YD2g2qtkCKSXm!Eg`V8z|G%L2;D$cG`%C- zn!hoN^r=ip+m|qv+Bqq%Oq$yQ6x_(Mc50v)$0N5vKf_wwF`4Fq z1KSr4J}(A%4PA*PLGN{|@8WiE^lQi3ACXmNbON|;JjO&5K=*yo9Jb-Dbkc=;D#o;D z#e-{}g~km%Kfmb~d(+IqgE79wN<&TyxTeA>vxax`)gR*)YCnl}3Fm8+QVy<6`H4q( z2`N93{AeS085owy5%!ChR1<-vS2>sZ##|mP@Z4;3p>_#T?J*RnZ^ZtMuK7`2{9Md0C8ZOK?D>cJDJOP*~*Kw%}i{tkW-lYz z5r;3?o~(!}Jm%c7x;~1Rf)^5~A?SLD^dVP8v8LDQRyS13ziJ-K?L2JGEa zM2i(;!r(L}h;$y0Ehx*M&f{wL-h(I$DA^NI;Gx}((Tq!g5d;>S#7vjn)4=7}E1Y&A z!m0PwhwlM>dOYE(6szkuMzV==E}R3Jot#3EuWg%ZHcuC5YDC-a4t;xukiWa_v#uP1vttXj37v#gHsp!B3|onJ!Kt6t&9=<{DK`Eo@yXAQ3s_w4;Lup^A+FWzZRK^sFi z-*gi1!J=@TZ|s7IN9Y1wtsC!UCayPgYsyL44t{^ZRapvrh?nhRG;4+V%c5dBouE|WCo8t$i76*%JMnwAf`NAfr-dC3? z9Mj`ozdi!Cu9aIg`DV;LDE7OEMo3m#-aX$(iOUwy2|hsIX)6M)Fy%ueP9J)eEa@0) z0(ySF$n@(gLw=enS@Z>N(rAdSWf%1Ahv8|_B@@YHH?8BVcq(jBeKHJ+{RzhxsHWqp z<`Gg3WiYFh`?gDjwKf4{d9`@a^@T1$!djiUNJsf6P0H%nvZO}#oJ*k&$Re9Mj;}s5 zfQN`@Yq+#zg_VOnQy8l_=)k7(9GpjIsR(SphtB2xQ+P8xN4?22-eMHm+2j{zaGX5g zSMK}6BdU{d)Of=>RTKiak~)2A{Ed+2vmq>KA3tq55jNP@caqR3ja_-BSmSebvE}Th zvX>aVlz#mPw!+S5MF3@Kf;$VbXR%Vzi;w)4my}mk~4Xc;41QEA0gVwo$@85EUNUkGz`MdjVusQH~a32TUIH81~Ei9*?;MBLeog}ei* zE@cT(IrnXsss(H(h)|~fI0T3Q!^2dP?9~;F5UGz8nJ2+#3bu8KAb9HL3t6FnB$!6e z9*fCKLyd?(t@rXtj;s~yzJtbczI5cnJGvt%g{Oz-c{-Db7pzEL;y5~Mz_av7!XVY! z!u|HS;3s+?oKJ7v5l1K-vLlw9QG)LLH|HE~OjN=AyX91u-+J)&`uO7e(^!VMXxve6 zbFjwQeS=jKhPlIK1gpssF|lK-H`FK_a5t=kH_ zT*t~vx*$vB(C)qmABXO5scF@t=12+;H_$o6+`yQf%<>k5=)im@CC<`iZ;FZx0Ra+1 z|E{+Lt=^e7!+*$u;YM|_WdH+0yDm&&p4^BQnb4X?FkLb#f45F&3aJib11wCle*d*9 z>6=_5)JYybfgrx)!UT&Z_Wb_(*!AZr5FY_KXujIpE(2)o&APdV*i6%Xi#FMIB>?AI z$m*2@aAO{Z((vuIwt<7x3vvnHvr)gKIL0*kdn`F~^wFh0hO8T!CMe1>`${Y`;HZ|n z9|LlKb~fZ~9gIA;mAmzLj+Vb-weiRNnKOAl+T~-fepAInmOxH!U1P__ee4@Z1Zd%BhmZ0kVo*h&(>RwT9fr6EEdzpF z`~3RuM{BMD4@HXE+%%iyC{g4kC&j|Y;-GR$NxsgFf31~AVC5vlj)?p{c{0`QW(jqA z7YnU_j?~u_Fl_$Lig`!G8;o{;o4O3C6re-S5rj^-Ukdj4A!lupp_+4qR>legI$DN4 z;1e++4+oo!hkCvjjmakY<>VmICDH*{zyVB1qZeTpZopmIGk}(8tl{F zvs*F2yBO}KP%5mmAL zn}UuXlQdF|BGGZ+&!uX9#}$k28dqjaJ-v9^egWxej0j~4=cS8hC|C7TvRG|qRufRO zK_8z<_y)jspqn7crGceFwFg26Tw1aRls+9dB(T%w(rEsZ=ocywf<{8MWib~b?(m4Q|&sGWT9q(3S9J+chZIk2|k9E(s zeS#zWz+p=7Hax~P&nEqO+g&#ZxvzEI?w)=i?u1SXpp}Qx57szJaU#6vhI^G0;z99p zG?H(f#JZtalEg*{n20+cV_^krx#vILZGjf#BO%uwz?<@E>cCka37&}XTY5mCJ*~@7 z@&p79Ipm(bRqkP|fya{Yl%g^zcO_eT*%BNyBm82;vnTyc5K_m{ZR}$MVwW=fIG2es@r*b@2D}8$CU@;l6kdmG^ zHBftmE_CS^DxD{m!Hbg7hrBA-5Kpx1*lkt>3+gXB-KL!&_-9H{(y z=d<=-dvx%@e=y-p4RTjixtrYX*S889a7=6t+i1q0Za_BnU~qG}ufbiT$$r|aSZ`kJy+xjmpptW z?-_=h1`?V04bdm5>{WNU&MS@D)0HN+oax6ou)4{*4DZs{aG%0hq{m2QJ`j0PKu)0c-=kiF7?7^eP zn~ZD-b;_f)>2|M;Hx)Q+Yc{ z5^uM!ZW$MJmY{$JDR{Z=tz`9`bF2CI8`E(`3MWo9V(H{6PQ#T36~9;ogL~E5HQ7^T zt))~sCDJYJlF?~t3HKk?4G{4J$dqqW+dgZXwJJkBW?VRuv2y@w`MjJAOXK|{{^Piv zo!%_*VGZZ=-s`A#X-vUU~&aM%Oa^H@UW$rRWL>JJy)&hH<5?pGtrvzNXOy2$iU z|Eo)=tc-+hw{kd+8ZVKrIj`lELzN~1W>QbtuUE?MaB}h9B!zqB`o6z^b`8hc(K4=& zIaST8c!`%R_-!xJwo+ON zn(7cMkHAfasOy=sLtQ3BC9#eIB{@PIp8@%~e;yhrOs!ABQJP4dlgA0GTAB8&X=7T8 zlf2p?9CxNU8J)hu4p`)>d4>!j!qd0o^D3NO8SC$>7V#T zCCgi`4a{Fo+uh?T62@UY^M&gZIMMs61yvVC5>Z`tn(^q#SE;I--Jfz`A;2Ek{IobbdwWS6(3DDt$eYh=AUU31M)9YX+shir1%>$cTy^Z5uEj`>@DcsAHfuM%3VBuxJ4kl25GNS~Zt>zc# zmvd#OVV*du;!24x2=DANhW7FC*%vrrKoyA;`NxKB;q1-w0R<+cF5ww#e6@#4|EBU% zh{&yj4oX~o`>sJ3RwRN%QES_nE{(d3=ALU8 zRuST?At(`{wTXII@^gnsdR-|gDI;_F>wMqdJK6bHYdW2ZxH#txD*aw?fTF`VFmrD6 zoy;z00i~7cko_?>vqWX(;o!K4J8a`mz|RbR&A`ZTl<%UL@P`+2Qe{}Qy>E<^{r-&H zZNznd^*Xp&Tl6X_(!db>lo5NV)6lffmc_YkYK1 zHV0GW%C?TysmGJ4k$E(r7$+EnX4`aTKm=H?F#Niv4&VXJBgN~2O9(i}7+v`l-``Yg z8*<4e!Z~-!F`tXVTJV{e6>RBXY=|X(j*y~6Z&eW`O2qPb>#FlOJ`IA0?q$$w&Kptz zv2azdsQfT|ec$!Mk&BOgpIzReTv9XEm9gas@2;zprpEeRS?J_@IC6&np> zpuR#};9>wa8FXZHLeI~dLKs1jMZhx5a^Y0H=67;`fn?mo7uUnbRLn^>5-ujzv<`ds z1;WX4ISHA>CyH=a5#Th2LCT-#Xx}Wy9G~20-9;)jJN*DAQ*3zAZ)>i+Hk+9hwABFW zUV}z~hzmIa=0ZHlC7imLi6ttk^XPz9FLn7Wi%nOaD-{8oM!n~EEWvyF%6U(;`j5D)C4V?qBFeJ# zbq}!!>}1udYgR2gqdU4 z2yC|_$*g`*I|-R+-h#M`Yuxswpf_4zw>7zz%N^tUQpTX~^VoN%*1wU)&}1Lx^)tdLjyG+MY#{!JD2M`{WP$HUNn4 zyxWAD$?_po9(ii+lPJvxd?VCE4JZ#xaBsz09M2OVR5Aah)IrURbkq*^lvM0db+?+F z{->*r-f^-tqo4ygAf!$Zpy^MLwR^~Ji|DI>o~*6E=ZalrwA{q5I8;+-VIOTo|7FE+0P}_EL88hUVLo2?FUNN zM=LvAal21BV%uLmD>uNt$->lDe(cuc)?sCwI zzP=?~R?SLq>Ng*Lvni^ zd!9OnJfzlGw4G|J(d!r{qW}5}mxl{^=bfAfw;o+_nWgg+zT#oCebAJo?@kq^Ef-%B zrP4Yc`MO^UVWK;0X!fsL&_|_LCZ0T;{CU!XM6RQ)NzsBm&PJg^+R=5Pd?Jd|PH&`v zTj$;(-Fg9q)|_Iyy46M_O!9K3`iH}%8I>o4Kl1#}3s#nV15I0D^U*Fnp}JHL)sS+_G>fQEGO z2b==>`WJeg_QJzE-mD<$W)XMCPb?;`LBB79gKv)kE{aA^Oi0=pEYF1^CO<|-qm~T3 z+(j8$G3%7{i&eYoweiazGd%qFiU|0kL2IWxb~(f9Is7M#AoM0?KgI>VE;yB3#|Np- z8emtLP?ykfIY$%02{CTNxbbiQsW`^WQ#PhH3J^YsR{>&aES<+OM{!zbUe`f^OiY#R zIkwu6Zr;%X9Sef3eeM~DpHfnT@5!0={4 zIrW2oVZms9&)cMq-D~Q>}!lpm;?CR;t_lOGx!$vsYGw!H8IxNfAE}@aWMc;+b z3|2*c;H+F(6p6xO`mn&qJ&;xW-wSdGFZ zvAZOj$K494Zj6NL08tkPXWP?*qPms~a#i+^Qe9%2K?M;L8*#^|TJQaQ$LNsM(nodf z>Pxy*+oEL!d0%-#XiCx;qj#hp4G}YV7x(z*+iB~F-rbq>GGG%|2{BBx z(kZbtReqbbAJPbrfB|y#(mhpN$9VldsHC8^GkkEkZar}*BLQm^ef`nf?G`%~5_TUW z)CR{!s$A>WH$@E`Uh>17#1ldD&|F=a-oo)w&{dAT`|4iNBp<1jgc=z`14Hw9hG}jm zsDJDddRqQAvH0-P!fCfb=wql~;X=imNNA8Ob}%`xij%>hz1;9J_Vo!uxg&(o;)4N! zpxK*iZ3UP1Qm4l?4u3#MXcYAR79}ShJ}DqM24!-*LUnf?YSC}*$%i>|NW-KMDzG4IS8R0kET{>d@KqPdkyCl51EW zPRjIEf14a!5rqzSCw|bfs8@<<0_!hq;nVB7gCZ#0JdQ5u&U%@*IK(sU$jP6}`EvRr z!fOO;)%IRFaYD+T@ZTc>NsLU!O@6oXtR$(IN$3F)jvG)a1rT`9nXy|7iFqq_n* zopp2L3932CWGu(Tsm!lZt(i&~obHLt#xl@na*-8OltvL$Irt#>&Sw{P_XE{^NA%FaSRa5Ni`pHMFn&V3(r73 z1GBhAHD6hqFDGjZFxy^u$oZB*k>`uhlH|pEuH{MU#@+A+2jPP;dY#qa>`P zvBdz#XbNQ0chE32!#{(|=mahc9B|nc_)Ze4IW`L)^g!49JSP7Je430LF`fuyYw%SM zDZkwNUCIfzb{lI27DpoC(f>ZYXVms+#57yk!%(rM(Nhra@SJhYEVwLKeCg+_uXoV% z0r&AgE7V+!1&SJH_SHt%BBI0-;X+9-zdc<68NtPe1J!M)>$ULtc91&|%93c$XT104 zg%%kx-Q`e~!e&3Vvq_TgsPWKP_%R43oXvi_5k5C6dHiLqYX$pUA^oF43j@Rdp*7Xk-YU>Y$}eZpkNX22F5&F8y|s5$-|mmf^5(X}i@4DJ`e-b)GU z{vatck@x=D&AyO+tYhDppm%}A? z0)1HW##ZyEmrxO@ble|Wo6@zFAhQEvy}Q;jir;8MVfN=06QT(!9TpU zm`HejuGZ>z&WrGWMs#6mCrkd)VS`ZIuwSa~>o9%Rjh_x*tgg)grOHuC#_fg)IJYC>gMHeFT32Ps>eTmnt6 zrS*)@)kp7q<|@X8a`!H6Sv8h&&&{~&WK`De!h_;^>ADvhj^&wNpwiQ@9NLj)wkDmd z{2c9CUsR3KrX9cPnB&)`PB9V*2e0P^TH@{AG|+?_8u(Eh3h}`n#*X9Vp&uV4BQ=)H zN?oV)JrjLGrVN zmqyoPdzXZbBON-Q^t=p($#zpt#M+2H+Bg1YW=T?(_UETZq=FG{k|K9l?9ySJt7B@# zYjZBCmWSfV(5DjP%eM`xQOW7FEd#l*$SOS$JeDENoDDBg!|+(^R=u`WtbpxIo zSP0}kbS^mvHnLpTuUcYhv@v*7r|&R0VS32)@i1roLr>U5nM*t(iGdyjZ4jXrExNI&6pM|7+)Xu6v7BR6zy)ki)I zC@N5MM82YOnN5}7eB*%1xC(5`=Q;e_W=9X8sXxiKTqa`bq_TGHO48Zz>5wm1X>7~J z+fmHWil zC3GNZ6HagPaSp9t3GNHgj&WXF9A!hSNl)73#5Bz_{_$>CPkxsI61Uc$!G-(N1RY!DU38&?ovgFF`J69- z(>>55@^ZRor2T-R_{_=!!H=-kZEQ_0Fd|=L2E0wr7i^BJ&%4*~(`aN<${9>^!1?_r z+xD*U->2r}q$W7>Yc*1Q{3dPA zx^KJ-0TnpRUp;0Py+7}M&$G4uw#3%er>hmr(KFtrty!Ph#KUCC&tSXnk6vM$=wR+j z(dnBVaZUQDf}NCBW$SuIWo;L=?Z>O**s69>Qro$l9a`t?KFr~e%?yP}6E(~)RBL`( z8EXY__kkFvj6=xfbBew~lPL2l$_u zF8@eLL>vFcef*8`XmO@mw&fL3-4s}-b$Jf51JNnpYTJGBi;@7v>D}Pz*M^2|^-`(9 zKV#<1pBn#7GHlK0e5b%FAyK%0)A-82AK7#4TiDlUqG{e-pz%*&Y8x7fsY8k4$%Ulo z`_Ol~`szcaMqlvgFAS;FSP6C>NzvRud;J@vq;%F$`CdfS;X$5tF2~6{uNyv2XdF*1 zv}vhrLqA%pnagYsS;nC{2T&TS zP+8|#KW&Q6o6cj0EU&!kKJ~9o;urt)$h+FT`IY0SNi7(D!=fD3b?81BwQJ~=qk?4R zgh{^*@jt45_&4d!o_JuY^^My&5gk9#<-JO)BEHL>uU|U#dD{DHLOqapzGtiB)({H@ z06F`b%*;ibD^qGepqdiiDs%fwkjx-0eY6fLzs0JeJ9>(!gd}`Bp*Pt5L&X$BH*r&) z1xOq2=HGb8QvLM-m0ryka5-=I5W{IwhfdV1G}J$cpQb@J?Sf17@p$2Fu^|4uYgc4j z{y-1&-C=WHV(LfYfR4KY{*Gx%AWp{{^oc?_aF33e@t!(cQn$OS>*K&Sp0;kckRK|% zoa+hfO}!=(<8&m+ZsetVzSRd^1V?w!_<*ccD%0oX4rgEYBc!!rxrPYe_~LG=P>LUL zu3=Y`89XQE$ffc-tgp|&1>$nfrxa^_RJ$;?X{I?8?+S#8!M=s?H8oJ_4{ll`F6NsWlgs5TI@N+8} z-_^w=?pjE5W~_QaF(4AW?~nw{#nc>9$LTv=5)VX0%wJ7%Z(43E)No+BndisXx9w;8 zKW$aL2`5e5SA(?uK>+tJR*pzkxp-g1{p9- z&8g>z9zk)uM>MS@LYg)7{jk_Z$yBCCgNiwf;pu-$n2wKERmv&Q9VvWbW9Z`BAiVxOSYcVq_WEF^F-Zwi!~`sR(I&RvO}=}7;&utv?MGVB{VyNsDNLnSpPPrB(^!m18RXP ziYq;y2qO99&Jm~rrZrx`{`g}2?%T27Uqw?>Yp-+%ytB;UO@@ zxGnZtc%g)YxjyB-;ls2^f@>}%wRxAT6c!r+G6g`30{K$#Na89^)`58VQRiZUnFYOw zW-Gv$F`~CNg8gf;qIr{{k_(zb(4ggQbqBuQnQ+@k>37vE7j-udy<4PzV`a9_>&Eg^ zNBI+GFof$pc8>KY$GMp3p)VRY2^%8r@m|5TpG=oGu&0sQHo;I62gAD7!~6RE3uw*` zu-Cax3!bS8ApCWVEkrC%9&E3=FThc>!ibBRq12FQr$7Xy6sGQ}xH-f3C9QHr$IsM) zYmGOLwdO)Am`tU=9|R~|K?*N`Dx|F`pmm@P0rdL-H_HL*X9m zV}P%S_~3qvyT%!o$M;W{M8xc91RM-|>$VDcp(go6G*c%9+2Vn{0|ruCKata~r=fAX zl9*a#(Kv}l@yE#a!-dCQG{ym8G{sK=KUtalCF_Vvj5$Rdu$9wTHw6CBj&_KygWSvX zS=nm;oXrykYxxtJHv*#TxJ%k+6myO9okYoIdB;SVKem?$nh~HyG$gMrvNX=EU>vLF zgl~6?B7?1FlNdVA`9K4Nq!0qPOI^HNy=Cmnd{bp>73=kVgPuT!eE8ty->u&emr-M7 zF1W7yyLy}`XKf);e+N%0Am@pkTI+e^D70W}Xw2Gi&@0w=R<+%DxbdcIY$MDK=T=MT zwf!#r{LJjhOD;hbrgr-7dcBJ}{N{Q0IAr3Wx2N1skI{8IsT<8|I>QMrm5BLqAxo=( zKU%!R@~NjbSA$U^F3$yRk+V<04lXZ{UoD*43egn;nB>ZNf}0m6qLMqRy5<$$G9A{< zCp!I4Z6f>NCuPZyZ)HP0&6JQ3>wZ<5*NQpA7&JFsZkZUJuUHwpoNwsjb`eR3##0bq zbgX%Y2$so*@4=d&IZtSSix3zvUq?k0=LXH82)-A6+|B~7R;d%>&p-KB{G`Gueo&Ip z*axN+cEUX)3sMK!FCCwLAKWBGZWXX};7HGAQ#gG7#GD|JA*M_@DlrB+;ZeEG!qUKi zAmFt4vY{~wPeWyqNwd9AHXYWoP2DoKUOtCmkp%Nlr;y;*xHw3vzCH0Kx8=m>#c$;lu0sW_LK6Y^zV0Ejd^gz3ey(x}NbkhF5q)$s-+Zj=!&)Np57p1(CUCdGI!gT_js zOJ4Bc*Q9Hhuyf8jGias8?Z38BR!B-u&ZV_4MeB$kon%AOc1_mRS}(vJCS9q!=~FUm z8Q)?s7gAFGB=>PsQ&p{y~qHwPmgU#lD@BjTzfV%&rjQ_I;|MQ=T3?jnJ zz6^Tw|Kq#Q|2!2><_Hc@aMegTl3bTg&6CT4@C65rZT4h$=AA|L19h&V4$N`(qrZvl zzW?zB4_Bu6m|w;_+{4U^6A21*It3*5XHy=d#-Qb`;+_;=qwp==V^jb0FM`X|=2?Dn zb;vv|Fl5LiaRZ?IpP=}$`bb<{3 z4-=g|f{`ZPO+9B4P0j(@Y;&R}e~)da-uQFd0~0E)t5S=1ur1xT4m{rxi6L8YSGRvX zrRn^H$sXhNqnP?Sd3x=XGli@SxNetOR>X5Ac>|5-jSTtk;>LYD-bRU7_3!jt^*d@t zJhFdL{mb%qmAQ`>)gL)WNU-K#j5ke1{M4kJqVvB#Gh>!2Iw%qMs`$PFS^xOMl`=;j zHc4<&kCKdn{9+>w{`mmU_*`D-_eYzn>de0$C@n-I&%QkN)J^`cW&h>GkSouBX1ssN z{^-ztuWZqEA5b4yIWYshO^m{O*JV;#6#XLEu3mdQX)8WfuXKCnpPwA;oFM!9(gsT( z!D;RBVu|$yUz@K<{|WJXb~?m~+>?KNtjqVLEX(k0v4pX~!cLDneA8DF?<^8v`65|M z$At%ir2l-w%lZ7?oONy)GAX3q4tCPto207$T8`{s{^X9sUE|BPrt?SNe_M5k?(y@E zCGuiNM`G5SK^c>PypS*SF!gfX02I991Uzc&- z#-4e7k>p6Yu(w}f-Cq~X%P4S5#YBjoeSe#ZcfW=~O^y$7vR$>JmK8=u;p3Ngu-TQ! z+Wn7j%^-s)2`c`p{s*Q7MAi5G0w%A0DAw;V;LXCo7%23P>}v8dr-ifGNz3Eg{FAcw zAD=K>9MyzLNOvs~ApZ9j#eV^Mj$NwvrFeeK9@c{#@eXo>dUHPa>Sx`iJ@3ptnYQP0 zqWnK8d&D;}pXStAX*he^1pjURds4GVR{ipa4ikS*w1}@IP2P(4dnB73KO<~DJ0F-6 zi->`nF4{qiqWnL<(uhX-fia+CtTgZYsar*6ppmOFtiG}?1%p65BdMjC#EH|b?51! z<7q^-kF82r8YfCi0sSnUFf+rPBoOm$&8V4~D0;wTQ2{fwS-907q3oP`m@ri@M(+NuW7N3$S2j^Q z|Hs!qNkLKBpphFOZ;VHeh7N{TdlU*(FdU!31%PTZgr|RGX-kL+rVb(+1v3ubNAAQ= zE>{GZ50~#22-s)Q00?wy4bkNA}pQVX5~qsT!TD=x*N`{WY~h13Lx`GZ=& z<77jlcPmG>6Zl{D`;kKc5sFZ-6A{vZ83N7bf55JGO7DWYb9#TH%|=S*NB>a2?C;X< zL|*KFe06i)(|08anCt2?8 ze;>Hgu?VKw-?{Usl_5g0<|w!tQzt|IO1!1`s|UqBRxvYUcJxXzFK}|3H1Fme+8d|- z4gc@Ot_2hSUh;IpM4deSV&$R@`s*KMiC{530$746BvL%vx7cmIgx@3{Cj;0N!52WC zbU_XvI~3Zv!I%nDY2$DfGU&#)B3ud)Esv2;iLcBaO!9(G<~rz46gwiCjW>w#X`xYo z^a_kJ_Y+175StIyB6@oi!sHrM(+$_Yc_IVST4rbrlYx1Tj3Q;&`W6ZH zYku{`d*SzTBz`a2zkaVMwkLVF4&*6+`@5c=0u#<)NgJtgk>=Wo$(JaULK^H(*7 zz>ABfH$*3%bA_Z9zCX_d#>=axt|SR_BF)%mf*=v0Q&IqSS;a)yZ=- zTosHP6ldLqw#TL7KCo6QhvjjgU%rCg$~a{K{o7SUE+KJw4gM&n>g!{RL%41dV^+hu zWH?>1zl3Pcx2)pbTa|a@5pNntAmMmbFgewa@)a%cZpAb3ZJS-q90koLa?QU@bqo zdZckJYTJV4p9I@>A~3{!r`vr@W$v41lZ$izc7uhbb+lTG0|)#z=wGW6op;~795q(! zL>(Y@(~|c_lL1m>_0c=BA$|{)bjF`wSya~EyLJleebHiecbC!xA&ZH7$08;swXZGa z_iReLoOtnxw5`pSEd2+xi5Z!FbN*5O`%imXA2n80^d29M`+Ns0B2ze-qny4C5y63J zX!=$b8@ras84T`Mbc(XdsW<46EqIPb#8uw%;51GuVBvG|a$=i;c3S*Ed5 zBYs%2dh3N^pJP~1@Kn~CjTawc$fj7sm1VXHG_%mB7Bl-v0 z)XlBpe?)JNbr_wqSx1>9P6MCs(qFkW=AWGZ{F5_nJaG2OmiZ@lXEEoXYxew;%Z-?* z)4b60X+81e42N}GvU`l(AAQG8%P)tBk!z>Kcx~E#AWINgx<_MYB%Ah-B{{~o!mC4n zq~^R#k~zrJkuGEWP;8If#ut_Q_VpZ+32PPnBidupTEO$NU)YD!@2>sez;nFK$V7C( z(zN75MgXK#`eM_r+l;SF@LB20Zz3>WJHu2A>41iD7HPE-OGOXmE++51wrSx5rQVGZ ztnIy*TThq>C~#%-))PxpJWRAs*pot2yYW$DbSskhcZoQ_XpjqLzra@E(pkv6z(3GU zK1AM($vmAtTZ=0LuzS|~-e&|%jDQY@H1-IgukWTW@6TG+)Bnl1yo=WR4hj}6+FQM+ zgqjrXdAgb%E3t?A@1IyJo*lt&lg>-$&{AZ+jEbH5${hNEJOTZ0gpJV@@v6n9m%fnp zW$9?&4xc_w$lI%SM1)t--VjeW|EZgx(kylEpix2j=*R1~j>~#~(hf4+a?HzU>1C0v-34qNs^o@PBDm(3#3gK@sKNI zJGakPyDDsa4O#|)E5_ge$~&=yH#v_5?GNVeB`@E@%3Ii=V!C%^LmXq|hz3J`2YnA; z<6G%h?3W%duDGyPV0!6r95#A~gT?qtoy`_0x8ti<^NT2#hz`WTJfctkzzrT@k| z)VS1GslI9+NPYx{zcIfr(k_U*l6&H!g5TBoXSlp_b*?hKDCRRU<%!&}r{~9s#qDQo zXRWRJsLcf(MtVV(F-p6e=eOY3Vf_KEWNHUTzMWf#lY*`JqZ9Ln5Dk9$7SaAL#exN= z60*1lF_S-xk-XEx6R8rjO5&N%6*27}yR&!|TE9U<)&=$8b7a(NEo8TdIRp{!z~lBO zFHcd&djvk6ZpHcyxpG4#I-Dlzg-r65JHFa2@{GHt_gksfyO{?gv)ciKN@EaU!QfFLFTJ{L@-q5l&GRZq|H@Yk- z%CFb_vitZu-OS_%r|pMI!7zcZD3S0ixd$kTsk0Z3laIcw^+ zaFQ82f64Vfu{Y>ZZF z8LeLPmquMY>>VH15);m$7DAEq`Ss=Pc@!M!c{csjaX(@_Y9#l_s@2vnhRzC*=PNO% zrO8)RY?!lx@@9Is?bG}9R&1SpeJZCbzryq#bSAmWxt&*UcJq^2M*C;U4oPgjfDVMa z%I`}%8=LRP14S6{`G5~qm{9)2bpZyoHf_y@1l4&O~(8)^jlj=JVW!f-q$VWpW)B<0^2HS#Bo&k zj(Iw!r+;M@OGsqtdOPCK>Y_b@V_W{AXmwxa_}$VCnUsZ3iRr%uKh7zZ?+1(IARzry zl%u_NW5(Huds;HmxR=i(;NqQ;WTtIi{AMn_uA{t|q}z!^|v;+|t()X;|R3qIUk{dWPnw z=LHxJruR~>UPHYGXITDuBETV9H+J$H3-mQX_O~W$m%(0DP;E>&iGkz?-U5p;5n)Y~ zbceUTf;ww0{7p{;Qr3V1(sBkX!8r6nnyDEIt8b`wlk19q+5pX6Qu$oQjo?UXJs!&sO>k3VGsu`(6@p|6q@?2t#%$ zN|Lj>48cu@m+qXT?(njvAmy|i)!{Txj#SPPW#j~W668QpEj}fG4x@uj0FvibkMK?> zVC)pf6JMla6eaRx*W_(8eQ|k5^_18jlK&;5%mEiXq1bTA=E)I`#foLK#Oe07t$fHr z$#3N@Mx;`qAXAh*RMPtS!3&&`lBExn^8S1eg5x5RI_7=5h|x4{PS**&ZvGx+(FZMl znsMT^#l)-XaNYHyY3H=gJ{L8my6!-x(tSG_jHsnJT!6gy8>^jd?f|zLL0mS__B=h7 z5HV)KR}k{n!?2(~QYF;y?sJ2y;F2sCK1HiRS-@qVNK79HqBl!ZA<0A#o!+0HxB6#~ z4S^|(RpTYnpTMX@a9v+K+lM(u-oA#?Q)otDend4{&i7ReGLoRFgox={c`sIB!Xe4n zZKg~YLhe92)5O-cSjrkz$uEHFXKTX6sK&8-Z+CXu?S1yg0XINBiu2)N7#$c8zcRf+|{3uaW&Hl>N ze|nHKS37fxW2;N@_-rq8qb;m>Na^0toTl*DDOYS`k37{4|VYce>xVxYnQ#qBIYqNIOPKT_7 z8}e>d7jc~v(i1Cf|m@88W%6^Ixhz`*Q>G4!99QHX2Uzpe6T8lTNfq!P;(m3KpI7h`sa7)^gptVK#7J0>nuB_xSv zQV@1%`lJWq4byx4`;*gqw>%G>ziDGST`to3&0Q|ne@hJ#nCm`>FkrD*)-lFaG5Lo! z@|4s{&CcpB8i;oRdC}6fSvz!fle+rZEas1psp1#iI_?vX#FvK?h(SxK*v)N^>SJf8 z{dtcXb0>-O;1Ypzgrk1S_eR$NhG~?n<)aX1Y7>#hp!Z=gMCLxNbSYKtTGONwDxooR>R5 zul+1E=15jRJ?-ySWSCh?(_8rb&jW9;JxeVJ;0su3{76`JHMQL?=4!rCn6Ci=U~aPI z-G5amD?9q1PkU)By8QNXz62Qqe!tJ%y?jsfL}7i}{eYJR64H{dTGf zy8s+L#r-$KM_aevGhQW-`uvrjJkWA}C-)i}dRY|%G}NaEkj#0kGpm35#s?EDzEL74rQHQX z9b{o^zP}^CUI&E|9Z_{{SVs~Nc)Q%_`9g~YEGhO+cU} z-S@X`SUndj^b+hk0BLG_ulKXh&rOI_oqsUDpY>#T&I_p5R4$yy#bVHi!X9Tk(}Y|#0;4?p#0SMAVSZF zp^ev)9(DD&SRNbmnA5z`Fv(m_Lm+|(II9eHvKKk!L+0*)li%0$^K=R^R?r9YTVUkC zcv&&khBZjRLdHvD9(aVJ*6M#C7+)5`uYO)=*ovqOb1KLgQ|%OW z|K~G{X;=o+(In;x^Sj%yCV48`an9c*5a0`KZ;&f!cPb#pB>214_@AykV5VwPh(r{1 z=TPctHe>c>-r)DM-#|zUBd$Km5}%u}xcsPg7d1rU)#M$;$WOqe1Y#E5lF7|=+?&>6 zK!;>jifgmGn?`R*{-Gf@!yCAkidv7!q3}%Meii#Zx}E$NaT)3sP2G`v7MqstVbA&o zM7RHJ#sdz!9oqfIIFjSeUJ_nz<2JN>-D?GhbCK;LL8Ecc|MlxJA;s7TiM-3#V5K&> zti%`2HKQufnhPJXKJw}bB>d%s`ai!b{0jzX*lW?GU`o`V)%^DlmJfk*^j{ZwEP2o; z@wLG5mTw@^Cub!m(b~W+C8$(E8)xLZx`l& zh(#HvDo6MI%ZYQ)m!py|v#X%axDoiJ;)iJI_X{9=c<9hHIy8~-eQVX996^hQjwOYL zK^TJe`^PpZG#k^4gp?Tn`IrBC5q`A7vW&!)jS%|t`~Q5E@DUQNy_NzL@Be<^zkjcl zu!Z8%Lt@u|ed?cgAv}j3>A@#ui}l}^^yfvm&@j3XKJJT5lmGQk|NSHFc?^mr>yS>u z|Hsu~$-q}#*W7&UxslWG? z6fHk|cko=Y|gRBNOg- zKZvW{H6G1%D&n%Ekw^B-*h(yQGsy@3Yc>A-%}vG-2UACPP-*P}u4Ori0yfsO&pJX6 z7s9VXt9=(Hw53DE2MyDPBZ&GnvjJcsF=v--=0Yug1XR0#4w^fSdxI)hgZ5(OO`MCX zCaYZ=79LU`09XQq?P{a0RxtNDL{0BHT#nD+Iy{C>F<0{4)1MuZJMgYR;^7tz%|*)l z?bNM8S|Q*(%87?E64Wy=^LgPjI$;xi{b1UwACxE)LpP4IA@QvZ#Wn>vuRLes zXg)MD!1%@1{dD_+t9;gNptk{k`L`fB6n2{)AEN!Koj}TW4@|&NLm*C*lFfW4s;Arn z7am;NsoyL)Z5VD-&wNN`SoCROoa`a`zxOk(7DiZp(o=(b^5E8gS=cSjIl@Z;WS)0y zUE)a6*df1dp0H2tLY8D=s^QsMH!R1gENd8ia{dyX$2%{rG8CQk=qpw3@Kv zAmbEYruzpngU*5e+YuFJ4X%8LKF=K($_xFG=Iv2F2d9CTdv`UrkGu20!mS@qsk?>Z0B zqL9!0W6;f!l~2Mn9F3*W zQ;5{?(Wix*`(f`YlrpPj`}WHi9At#Zc)0WVvs;Ww{UsMCjoSZ2R9XyFh>@;H5;=Ur zM(hRZ7Xz#O?4&q<^}_*D_W}l$kB=Zl*1sZb6X87gI^q?--7}qZk?i<*HD&Bc?%Mg# zhMoEz_0k>?dzkT+RkxFR)4qw+cgvpMfeAJZBl;$e*G78qMIa^GENR$@U`glgwOx%< zm{R5__8v8IsH1T;@~lrn30K+G^*@VdKNfIhT!0$#^sqk|rQIxuEzM{gcllwBDTsPj zx6M)xDmS2$U5C4R4>tjI}COFEfwk`a>g z{me)zd{LHJiB$GWrMi?_mRye;uYTL%%=#1136W?Jzog@JQpMJ6?*r-0+;K^0*3ZUh zQ8!DLB1&w7RkXJ5eb*(`6lLqO$!dKo+mT)e^LTSLFAOhv&@ib| zSIoAoA5_ewUUMyK^ygYubh_`PV@|eU#za(lD@4L&ZiQUKMj#-RKV!X<9;-~^3S&`r z?*|@CI3()#`(A2C5nIf)6%(ZJSj&E9zK>wXzQ6iFsy8f+|9G8#e(fVu4>)5dNUWQ-s>^1krIxJe(#Mpe96X0LpwVO?=Hr3A6xw8F`+N})z zB??&uOULQ+y9z^AMAh;UmM$!^m2&5*5_w7Mj0(vNX%3uYXT? zN{(^`C1kD1#*ll;k>R+MGT^$NBv?hGgDKiucSlb@uV^bE=V^QLm-JlZ*(OtH#KExfg6S2X{qv{fwy3l8U`5wV9^c0Wq<_s9$f zM{`kEKVyL>8pJ-*PSPPk3X{47?8H5S6>8@y14v3sjeb>SB5nj94P{7JR8%)zkq0)a zVVNb;)uQMtR1JGTNf{TR(%jvG6Bl4o|2IU(`_MGh!nw^6nM><{aXn(MM3E-(O~)+P zLED?9)G-xiobCfp+<;r`kGiXQ3_JHF$5l@ld@@AzbS<|%&DPyZ$ZFX&U@+wjPuHN` z;S1OL^&IqV;~KZ^S-r}fz;&L@7KDkT*|O@>dzxK0?Y?v?^(LLed3>%)gY)ET?Rrk? zbo*}cP;zLo)}uVG3#8u6vMpvaad`n4DMDf|tDY6U5P#HM%Ehsd=Z{e&j z<`=tcHMg!Eh}gSypFDf+{Icp-7j97!38`}blyVGQ41dY;d~C(UOBi|)94vVfJR7k= zqJ^(oT0`+2EQAX<=-O*{e{QNioW~bi7 z1kTy!+tVJQn6dj@g_&6tTiF*Mxa|m;eBrt5-hvLHwdC#WXl-nr6{p8W-X)_FgUnZU z+qX#L7-H^Vl2dw%i0)f1-zDOuTixaP`oz_fxptq)i@B4dx7wf&VIqVX+!v<8)1=!s zmTq5v4Ip$rhubM>*aw5;65l?1I!r+cvSn6d>B(&B)hp;`_oiIZA(X$6+{ML{`5I&g z(}JQTIMuKDu`wLk%a#?w#~*h6FvJxSI2cF+%D;iHk`>4 zT-WVb=`woP#PP-P@L7*RUl4xxn8`bN94>DYa*m!uW38L%bEd}oK+?K>ol(`|u#7`V zONl4knndgBQ>h~yPSJsdTd8`tn;_MyS)z6+lXk!GL)^e(2Q!$tj-hsYCLqx)>wc8< z*J-?3Z|W%tx11A@+Bk^s)GkF;UpD?>v zNy8Q+pBBg=z5!|F_39Ik4jO}9xxTs6gPIVQSV|{=IK)0oCG2-QK)QCC5$vl?)@BPj z=4yG#Jji}rc-R;fXw@)B**`5TSI5O}B12y3koguQPPSo*F{qyv9Y+vB3%uw@GstxF zsi0F)9=c9sdAg|->HZG$601*0#>;dne&afQfm8}M@&|ut19^To5D~FvwV!Bwh-;fSO$f3 z7fSKCV|`BhsyGs2hKf|U&0NRx%Y-(j{!>&jSS=Tq&m)suin!eZqghLW*k9wOQoeI7 zQg{+1v1S+h>t5HHd=EoNpyN-7dn0yBH zI^&))rBBT=xNhMTeH*YMu;t!_vre`zLb%jqL=aa)2Pw$lFS%hlXL#=cLOY5h8GRFc zl56-kdmrE9XQcz5#hL9dL?@32Os@VupPQ2hBi#8(C!4EjzaVJx9e7r%VceC-zK?~D z<7KAcnCm;0L+S!t=Z?2j&k-{izBb~1UDP6j#CI^d%FjgIUaRWbafXvM_Um(*mJZXG zowFu`f@LsXdVUKKww!Q@_L)j=JIbYL&S&X|vi0tP0sh>4?fxOlC;Zr$}r_gtllDHlW{p; zDV7Fbo3)#Yz3FtFDYsMk!CzYx!cc>x>GpdzWl~sN{hTYA6R}+Fg?E>!gKbobyb0}U zp>a)CnbH38+!@u3$7nDgGM)#{EU*nPduBUrRaeUT z<6C`wrZUZaTHbX!Uuy7l^HHvf=+g^^J2+96>}2ZHY11s3JT>L=Elw|pILR?bHP_rn zg{b@swX1f+1eDhx_&WAaPIticOlR?fP1E$*`XD3{9K{@52=C6*j^E-z7)!d zHEj9uB{E4aoc_}X_r86yk|c4WB}|tBZddI6ZlB9x5$t+7bx=fgsrgWe^0mT{fy+`# zuh@yw9X~0l_F-6YmQcI7*rFrhiQgv&pNV}B;;6%gTO z8a1zW(*AJJt&1Rwy>9!XZ753Z3J7D|Q3 zPjydHr|^z!?c{Ij%ukLfD{tNK95uOp{YPRsH3jik?{69l->Sb~`?R23-M-venM}Ho zBt?wF`E1R7#)%ZS6bC(;z=xA=wJp`aj$_kT+q>4SCjCK&T+7K&;rz25BAkHR1zx(i zLKTT_fhLYPJEauD>eE9XgV%))j@>*@2z;kK6oThM1&`3l+{0?h_kh?r0`|ObmTDGm z!1$%|rs)rp$~bRG6m^hHjSa;m#!05hQxr7Z5rslj^OVUmLBlWc`mg^;i~R!QNp!Km zh&_zx4nTU?)OO5re7`JYrLH+=>wHx z+d%)=(8LP4Ffr`otoQy}Xv;--7)q$b*qpjPHd!z8z5fK;@<&eeA5Hh9>1IT}#XS`a zothQMbnbMw*bI1n$TQQL2&z!cSE5*yN!>#m7HV3jE#P zVnUcYR138`Xj&YdACuWyS1?Odd0ZICp8Yxv5k05Wv=(5KLekqG9+vMz*J2ZFC~SC6 zFz+M*zcS6MPZS{P1L5z(fX*mL@8@VMJOj1zr=gG@Qxf+odTvZ|LU_=PkzeZ=Vs3Vd ze*0XfbKev<5`Q(|7h!V=7NM@W!IWA26S;6_Pm_2PQmcZlH^ttz69uqlHVTJOi=0Af zG~N@6na#O9R+`b2Eh%+&Q1f;|g)HjE(XBflAq5zQRsrW~m2*j5&ODci|IdZ+8rP{& zQ}!#%1%)N$HIK<4o{OjQNxPozU?uxns4-g=Fs5D)OsWVP(D_q0bod{}$P^(+=*jiWhc{q1qFo-v|qX zp0Ro0b`>5K>1@^9ym(2YZh!iecJ*1KFD;vymgIarMIK8~BjD^s(YDAyu{^9kRx{o(d4_~2t zc*Wa&MME1z92S&@jqM4HOFh!z0?pEoE}+YXWO`ns`-1qi(D=o&d-syi=4(#G5E;WK z0K`BTb_5+LAw0LnZ|$gw)87h4@p^sJ#P5_)HFO+xHu@|^uuPt>cK^uN(P?-9;pG{X z(GT4#f=&sNWxD=~1S0bxU{5^NjE-4X&30}Eva1}SjoSD4;++zAKBqVw_1^WIi;o@H zPZ!_F%fgWVtbY%+$gg0L4VYpiqW`*HiRkAs7AXS@$2&A}V`S)|-J4Dg&Yo}9+|}Hi zJ)h943W`?qMRKYnMe2j}s?uit=Yw9MsjYFFXqYSVEYch5iu%Elt|7JXpS4LMJ}5iu znJ8a`UboZd`myVKpJaj45pOYtCp`TXYPPk6k=*nEWpf{~!4JlZ)ARb`2_~N+%fC%5 z(b{@!&b>WLGI8_Y$j_biopR+L>}o>U1sj-pJh)6hz2_c&*|QUz55M5c)!Adg7YV(O zKNk70EXI^jksejanYsk^VnyJOebg_HdCnrG>ECoiA zYsaZH=3D5bfCXOZrnX8YQyf0{F-@89!}b0Tb-YQlY*?9-PzcE30fwH z*~#W{qCB-JJHG3Ka?(X5EM`*4Fzm>zqGyf#Q~PvWe?nByCy$bDVS{o$JWn-RNJjn+ zIEVC^7h=pu3{&>cEBgBW=zZLjorWDi*okBBi~+?&zFvZWbU)9FZ5#2)Q0=Q*U{DK< z#f)NXRwyXVbyCj5F%8{W{ZAir=VX8>1n@E1N8|bj_+x`^47_< zQ-d@&6O|(^Z?n}^-wPV=32B8`M;I-rHsaAvN> zqGHt8xocJ3;YLx@!Q^Q}f!4Ev;L6Kh~d$8jlkT=g773MGt!F*bZ8{!!*9Be& zLsR!C-|h{FXMmg3^<|Oj6#SgVTU`p~o+!4H!h7&8B5f`}Um>i47VKmajt(A4zSCq6 zzh8G;KO|l&FE~Wsf+-D|fvFaH+2+<*_Sqnln6rbyv&ls&W62mGLuN~Hs| zp->v$s(vQ|yLgb9l*CY!WCt3VXd130w@W_06dDTcO3AFm#_Sj-IcPaYAh8rFfHPZ_ z2)|AIa%0_weZ{OK+I5}>ws!yP1E)_5lT$kSx`7B)%{uMrXW1!jSe`Gq$+Jjq@P%r- zr1Gg*BM$L8F55w)jce3=pHUtwL`R_J z)57zKZc_44K5ooTJ`FmX2{;@seaU>0e0PyXCkt+4#(210izx}}xkS+PRq4~-<;Hpz z^Zo7Ve(8ruyB&5?uY&Inlmnbn{k&4fVfM<7?~i1)zA%(qUvQ6wjVCiJKcC1_v>26x z>&YO4L+DkT?UBs)RW-Rqc?Vc)g(fdW5AYrnGq7HWvOWPgVbiE+8f$p`dMh_AUudM= z!AWKFcRl^i*f~k5Lp*$ud9=W=7Y_3M*~AQ;A52z<^Jf|5WBhlZAHSD+m_V@rk2^D8 z=mpDS^H=Yy>MP!q;@X&QDN+*tg|%p=B2+sM!RX;SOwDBC`^u7$+5*+-LuAT!qtMF-}L*!;Q0Ow5qY3%sd|?DnlTt1^+}%H z6~?Yz?r^Ho z?SN*wwkY8-tEx2ulcI~&e&@zt$HNdV-C_Cq5SY%^*@Nx0R{r7^kuNlQ!5kwh_?k%; zrQMQ_z7ENy>~}D%P7-7ZhYRcj1vmH)x;9vBUVRL9d!<(w2D*xGzL#+X<#^c-lG+u8;VieG1u$M6P9$H6 zsdeT5z%LRY4NND#m7^4iCYDAqd6_`BQ zpO7jtfaz1F`K9zk9~9OIr-Z;DYCg9S5r>&5YVHEA)#>r!x_X}r8Ict70Rd)-V))v{ zf8yVneJJGDfx6H0f4}Vo3UJ%^#{~rb`N@C3iPQ_C7K;D=>l+x$!KB2yl1F>=!ep$x zIBC>&a+-fZyDh{R%aO7~f8!1N>LJI`gJ`>wpMPYa<3wVt8U)5({=K*x@o4ED`*mN6 zsQ$gyzyCB72jEdXbTJ*7e?rxHvKVcN&O=I(F&4cFNUizW%4FUVs4?*Wec=cq+7eT5 z#Yn=}^b^hD7+EbIj;!sq%1S6-5AfMaZCd_5;Q&H`(IT*L0}7Jy?WTR0sgy5MiH#a8 z0*6E}${qCc$3ti?3T`OmN6NfruSczMSQ=bvNF(Zc;*AK>R;3ZR-@OomFr<5T@41|K*x6vQ5{&{E7FzRT5aB(Em)hP%z zEC7Yqj}tloVhv?m;BFVo7ie33nEbhEH!HRfMsnruS81bii92@t)WE{cl)is$cTgo6 zi!U8in1W-bUVzVTe0bt;*(=|KCEafkUwq3Qd~|xe*KU6YT%y~_eq*ZAk-<5g4Ggm0 zJEZ#qtFhU!;Vz_p)h@VDo-IQqMwb;?%!fG=4ZeMlqkjR0DL57b>2S-j#m)&4_sNP> zq-xuS{@+Z1P6j0L`4H>@OS z#DK?twJu5IM+sMtK?$%S;T>Gdzz63EPcv%h?JiqoBIzJwOc>A`9Cv8F{y>F)s2WmN zuz@{N?hK23BzbnIQM0P8!j-)?3IJKl^_>&+way-6tCn_}u zf`^X}2X2zxlKAImHny!Ceq*}eRjFvNOc%#B=$ezMn=%VIEmv|sjD6%>ZGhDpbX5|? z#86s;r?R!35A)y8#qtY)J?*|^s+xKJDxXR4+KyAU;AAsZGks%6rVp>!aZ_C!fPSF;^myL(l$zi6k;Kx+XOFq` zB2oTWz1b(FcSm6wi5s33KavK@tl-n{8FpClA3j~vAr_Nsgsi_#S<%6XaSFs$dNv*3 zE+y=359?_7Y$htIV{b)b4~}Md4=Ul@t4!u^V$0EEmH?DEv3!M%>_yUdz{d+eqwwIZ z@*bB!95zBlMaXGM52Dy;H6x{nyRHgeI=uP>J@Ef;9PNRLb$vrCOq$&?K>H`C$cE(# z$AP9%?4@5;_+l#AlM_Ka2oFutj1?hP0+bxf(Q(M*=oL4_HUlKT;T2get*_pC(X%Wt zox5$@^orrHTTa`9z|!Fgyff|X2oQfp2k|QY9k8Ij4Ly-+`SnVdIz#Cl8=PDFhKnvX zCrr5_`hkZ~_K%lytwjr&z{Jk7Ibw$|PD$Ww1EzS16@qTk!NXL+!YP=EgF8?YC1|8| z5a%B7-!Sf(gkUCe@7OPzsV|Bxeg!Bp`3%f1cn^lL7;_4y-6uE{yj3fq5vcFi&F z?z_;{24MJ*{i6INAI@SpCctF+y%IIV2ZoL~aom}(yJW&`sL5DVdC0%=nCfO2pzTcOo{CqB?A*u0DZT(K#no*XV4+n^pRa)?T2z=Xaf}tC0~nmC5XD z;k~g6R~((Zt=|J9U~P6kdEcbdf>JYU&kF+Vh7X)<6sa#LrC%1I%TAa$qN}e7taX^VT#e*Rd}j#s(_m!^9uIzLL^67%tIyAGkMb|`WBpMQ05djH7Cb`Rp zqPoPoB?FH*elYYJs}YN((O{BqD|^G58JS`0a8di^s=si#+N3*Y{e&w4sdD8)^&l6@ zm$@tWh9|=P_F)f9YX4>s;IyYaH@fzr`No0v1Hn;4S@hSRb@(xK!zF+MGVOtrWM`D@ z`evR>Cm(QFGtrgAYnad5J+E+*Cpjx*5%M6z-~*%Xq*CBvyhBX$xW~8XMp+)P((R8# zldtg52-rvVninX?!#D=p)HAIvLUr-&Ybz$a3fTF{u$ksmd=c?2!{(ErlE{31mbIX` z>&l8;Y8X0oV=#@W-S?WquW^;&ckrfA)!{sK_64@PYG5#i&PiIs+N&p9S~5=d#LrMk zi*eTaO8xrbYBsYk2KKspJDDMc$_6ja<%V)TOHSw^vL{Q;XEayiDh)uir71KD4`N0+YE<|b zGN;;gMyYcl=xGF_JJ&-h(C#HXzYC25?_i!o6DLEHb&p3?`=_DzxHmrH1YZ~IBKjSs zm#ZUV>E>BtOm`J}?4Me2hz~OL;^&iJ!LFro}5L z+%1qaFAd)z5-+?h)-tVYO$}3Q#@`XyeFe50O`&C=ZCcH@ zp*oh49FLgtv^}0bR`+Ql7&f+8!5K`bQ{KTo>Prj^RllkC>DDKy=0He_noGGK;zZE# zoUdcN9JDt8{lntR``2PGKDvs^wX!ax8t*;wypZ6-e>@+c`tY`DPd7H>YY2-gChxHb zu{2sv*0njWbVM;3r|n6G$TxM zLy`v$abL-Y;Ygs_w@E6;`gy+_B#0#$l8@{3ZP%a=1mm+XvCRL;%P8R{o;ib*__SA^gr@4w+sE+_@Jb@6Ms#9c$*q45R!*{u|HL3Dxm> z&K`zK0%Ta6q!anBWPqku3L!kAsBoH&uH@I%k{Z*@=fZ?I zmR8=?UH$*(eOUSs~R5T{q6MLfH#n%E1q*>Vu`HEuzw6xBEEh=&U zCkcZ~-AYor5bPg$oz`0Ln0eN~Pm4-w(vxUOQ?xd0z1rNxo2#?1 z9L=1;|8|0Vv5l^t?fa|rYr)4$SD{uJ4NVd;I=@mD1SO#}HeLI#4CZ(3B29-L;e#Ic zcI@0IzC5}zo!;<f?Wnp~Vz&v1`X`*@$ z|24=`Pl?FhvPo_$T=8!U<~X3DN1fii*&jelVs8pSK7co_D zcD!QgAl%=;DcW|Vz^bnUxQn69#u8JEAZ(MR84LMBD-3L=VA419uMwEBNf&SK{-8(Z zXDBs|z~e({0Q4VsVsrklqf?S$rbWVanVwvdaOF(*yO|_BR_Zh@u~YuK@{Kr%Z3YHN zl@4`%gJ#o@CGNUSGlD#R`}c49a${y&!0G>Zx^kD}33AJ$`#x>*H51A}Y4q|V+%J&k z^E33=m+Wy>_{PI{qmtKj>sND_4iGSDq zds`PZVL??@TVMZ}kK6vVJlMx{S4$ne?q%2Z&{13WRke2o5mZYFvhN-~DXAP(CeDP) zhp)8kr#ld0kb>=Oc8=d3&WbA+Zxk=q2OtXa;Ct*>zew>?LZzA*(x)A!PO>*1sLP||a@<5k=F-dy^z=HXq@PgA57>P!){aJnwzy*9mmlr`|yZ{_`fzsKoOsNle2H`IMD z3OZYTP$J#pxsU4^?d}zhDLgtepXn)>$5_%>l%c{Tmo3eUdOl${GQWeHb)isg@DA=> zB=dE<7m=kBFFYR%#m<%(Jde^*^gaafO~uj|pdT`Ed5rqn2kL3N&&hExkJWxix~!?d ztoDrerL6$fYYo~USh%4%JNu8~$dAmyq=wLgFl!i$m8=j;i-BMi!q?ZVE%>!(*AcR2 zF)EXc3Pw_0?q}#xuRdr(6riBoIEVGuMnk9LlL^t+Jt0r`SiwWeU-;t%;Q9`#((H4T zs#lvxfpF~P9()p!AB+a&Tb1MINvcqB`PyHAwHWb8_!w}P#ob?P8;Nsm6&|-wb%m}j z{*bwrqUX8cNfqBQHWHKWH~enOdxkG+O#akU9!AYZyWKJ${~QNWMj zs^^FtXkO?ny=F@}juvtaO;N{Ih4MDAPHS`J>FK?ZTo@ZsueWQpp{#<^4ZQVbl8}QPvU-YOUmB@5}s%Tk@d3CIa#LPPnB!h=_q@35TiV+ zJabh5@PuEE5@n0&PVEgCG)nX&uyXlf36G!>ev0YZxCSwik?V{NQ2M^hQu+zX*Xy-v zMtAIT8iCBcWwqdG!Q&Vi-vVs~HKU48wK5nfBd|+bUu26T_~VYXsHIO8D*338k!$X+ za5(vQEFNCbw4L8aTi_Q*KnV?2f&cKwRY5kyu*C4rYp zuxUCV@73AUXsDJ2nuceJAc6Q+S$R}I*?pZys=HWj{=Et_CSWB zzTNo7)rR=lWDg@w`f)VM^$8NN>DPEls!kmVt1nvHbS$}jr+sBK;E^>(HS`D+z z;WL{kXdd~vbO6Nc3$#&#pD&BwJOGbg{o5?Sh`(VXODBjifx0+BOv_jG^fC8#6&8HF z_+wfsOZK=TKID|5qS&v_XM(a)(knn#sF1PRyT1q>lF_;EAfDbY?E_-O@$^^WC)w?7 ziIV%nIy<+Hj9$w!g8H_IWP2}FQPOK~4*lHuN|%X2S8`K)EOrBh#qRs-b_Z^wzjq=z z0(ujsK4eU)Ra;ylzqTs&O%H+Os;Xf&>pi5l%MZ4YoACc7TXkZw^YEY#NftnN3I$jD zVqm%eZ)?h&S!*0%XV=MEz_g9&X?%vYE?jmGIVI}m|2Y~LeI+e`nKE23Mbh?u#k*qQ z_5X+x-(h`6uke=RS3kI7*?J8d$=M@4FAi6@qZ&spM{DlcC~g5j*ve$Np>UX~QBzMp zN07Jgp^i@#Sxw7+djMKP5&WsgT7z*g(JD!6oi4@*Ldbl6POX#Qr$jpGRJ+TRs+O={3G$^&e-<+&fe_&$axA5TIIhYuE z4~!ckK1ar*DG%kuD=|eA$xk$t8t<~Prk);Nhuj{z#KWl^LsBdWg_#EO0lj)Qd^}o ziXht3xI)a}H^01A{usX3s-lMVr9|ElR>hUKkfQwvle~SeqE{#trm@4o_E)O<>Q^7U zgI?VkzRm8CR@v^%&|$8LYn7g;K$sJUq{!i%4|G{=H$l)JNhMeyeJ-+Sa=0AIV@ zbOz*zKR`1v>i8PHy$EbaIJoyghwu!XR{VO*ZX!>I)E>SCRS+Bg8~E>8K3rbU2G6gt zM??&uDv@h?JBYW^sN=nTR{it+RcpSdY+5zyF))dySg;%I#|&Mk9@}HeC-o9+>O28K-u&?okg_ zUUN777+4QaX+MA|#J9nBaCa@qp=z*H`7is991)@ z%(0I4IIfpoc&%}nq)wbhhc-qUwx`bcot#d=-zT^poZ!Y)+9Wqn;W;iI5~jG}~@$MWnV!>Gn5cp#uUdh=%NB=o-@bdt$$jTcRR zI@9wEGO+yBV#{>H?hJ;2kKUkxF@z>US(m6r!;Ey4A3n!dE=!v#y|O54>hB9Aqc~d~ z2=tMV*DP8^syydi`dbYaM<3I>9Bs^Xcr|`<_c^sNYTJ*cy?T@R7>hLzG)SL)vEs>& z0F@6s{??I5rU0r^#jLCaln95bX4(_Q23~=_OkTo`1OA9mmm^=pc!~vR_{a0^-Ux)y z23sHcV#)h!1F}k<=|H%zN;s5NI&w`0QJBek_5K^d6>=&e7ZKY|LgDr5%jdxH@D2#l zrrqD;!T5s5p6Jl1)@jX7m;gbL&{7Qq#_5P#7TCy-&{Z()s{Y)>V^!ckC%ya|lSI}t zYx=`!!f=*F8*}!?_$-EfY=M)bEKff!Ecs&Nh_=dSAJ{K>rZU6^J3FK+)_?8tH(Z6` zkZA0e^%VW3J9NsBIv+2IBL;Pf#rH)MtMjOg=yo6_+Z>P!{GnNX1Ee;*9I=7<$Dh4J zO`o+VR`PkQh`mZ1p!Ftj*l>zrVZ@4np6f31|I(-Lb#nLQOlkwt_}JXBqInzkCE5=aPE|&lCTJPW~;V)lHGQ&)(m^mrqg)64afy zXZO^mz%B!2Q489@{3k78Fa-*j3q&`>{)a4754C9Qr|$76ngp1L7)$|Etr{w}1u-(W zedsrtdv-Bp+}(|E1IM}REyO=iOd-GuQM7xA822Dh;0cXq&cyXYyPJo@d|@E}0bsHb z>>7%YjhsKA{2y>DVG68d=bH8gYC<3?1+y=QBOzPHCT+{Poj4ag8sre6&VHb|Qx{nX z1838&jfLBvDjkO)$xFMt+lBs6BACZ%BTNdXTSq%wC+4NPZ@k^V7eUjWZ zpQ@Cj#Kq9063U|S`pBokRh1yV3v%rh4q-5bs3|u*%!+4LTuI zRP)F177?V}a4>;gE`jJwnMZH6JHTd4E)Pk=o|p?_4EyWJu|+N7)=nP~YAgG1{JION zw?~ACb=xBl9m{12-ug5T0>6Wc;$Y1>Mmz@gN8n$!1^GU#ZI+DA9LT@Gi-G{gG{O+! zY8#k`Qw>NaN@zlbv#&%exPtv@Vc0gWpUZxP5Oh*vF+3Pbxtz2PGR=GFf0!Ib44?%c z^<6&eVTcy^u(%E90QBdKOm|N}>I}1ng|oI%vdPlsdhuOo5<;MHa)yC%LNH32@`i1N z2{e_!#h1wC5LnYTt5FVmu6aN*Im# zh~{Y;_M5M^gak-ZZF0n>I}V0l(nlr)Euj?ff6+vyyD`zdMJ`S0mc;nAC#rD99>A;$ z?fF$e-SV?kI-;+ZCb*(N@kRj@`iK)CXmb?pCbF95WosmGCsp|N4VYmAguI`ahK|{Z zKy!KyLne-)*~WK=31vc~E_vb|Y=hc=PTeG#@AJDfSpvKH>E`LA`Va(!+yW}JFA^}9 zRXd9?XbpP_Y58P&UF=Rc!`iU+E5S_0)HeLoI34rP2E1>LQ=Cobh$H z39>HPK5YSItNo(6-NWx!XTCy>XN>#-RXj*|d(Fu?z@qxY?9t(B2BeYBfl4n+u>N!R zt139FZ(y^2Z-K!lFu5`aK(7tNOW@M77rz5M)%rCN;q4M53A$CNs$NH7@AkU^+0HS) z$s1k768PDxA0JRFkM+%el8_;1eFP;<7!>3yyl%$bn9`+1w%1%h0luw*dh)X*;mMCe z^QoDdaq>z`X@icWe^o+EN2qeBx@=ePM)2t|Xf^s(vs^qtZ!rE1m#49eQ*`h3T`+xcxdach z91Xd)W_p}hS}26L3eGy%%2JF(z1OqbtDrkC8n$D3w~pIt;Kedl2)=bvd05xU&SvCq3~M)aNcaQg>qLb+Q#8*7v=VQGOTpHpB{VRXd>+%LJT#C2F+(AQ@e^N; z3YtxtO4j+F(dW$oR;^2X={Cqi8$l^SqK1It1P2p5ql8$rglhhaBMZ0q5uD&+8X~2L zaWKa(N+Ls?Wn_fVRx~0kde4TXNlr*5o7snrKqNl7=&|Bca)pdr^3TH^Lp-BG40#l> zRupyP*H&DV0KrJ=T^4n>St>rkBs8TGk~_Rj?NjWg8*^8pV#!*IX(*sSsl>zid_=NO8Y2iH0(=p#L@ z&}8VM|h28X8^KBGpduGK~l-VZii!MdrRPT?a`esRA{8Y&MZ zl1JBPO)*G3)Ave74t}m|Xl#j@mY`KN(G->7W5c`S3pd#(sal5tDS3k%VPooZ$RhF! z5(tH-RCdgoi`(F5AfK-4;0OAhudr#y7P@^2Y38+eR4k{GWljk5Zu&0{E-2gPWxTM? zCR9ua5iE)qpwsHRcqL9MECKV|hC2dXhWn{KVMqwpcyJlG_g;;o{S&uC7;E_;32zr6 zh?%Twge~0r(od}ZB^V~gUdp+z)9*#9dQsvfj_1t4Jmc$$J3-SRoGg%iVmA6s#VgmT)@)3&_#0|?eSZYrn@ zlBBkzSEy2IJ)b1YATa*i2cl8MWGu+p_PM>CYjcpCW=(w6#3q*25v+^R$@P06V?Wjak!q#vMI1Rg7ix0SC)6G{ieV~B+@ z{QX{bqi>`K+c^yzvATJ|gmz$GET0gNu<=OY=-IqeE*5`Z?4fn*b*chfvi6KPQFp*nrJf37BLX z8V|%w>(KVA?I5Fpr8u)(QjP>E)a@}EU_K{qsWk!0a3kYvtQ=Re;3Kw%*F zQ%Nld3I){gJ^Q_KYhCctW>T4kv;7v+=5*b6P}4MjYp0%~!Xac;q%~iO$a(**Thl#7 z7s>@YqCDPNQ!lnG#sbMO1*(hXkHo2%cD$T=aDwblI5@b-6 z+2Hd4R)GYfi(vBOJ8pHsIl}o<<~wkv`2n1Gn+RO&vh>ssB@#s5BEFOx{<22mh(&rr zQ3wsJ%g1db8YnG@nM#@`bL5rl#s>kM9v<6T#Ze^5r#D9zY7u!Oc-Cg96OL1Qg;>>N z1?Z87AcOgAy!h}UWhfJOgjrz|AICt3p?57Yy3=j(@I@~7tWx8-el`JhcM(k2A6B^yB z$F@w+|Gkj@ZJQBrt7>Wg0ZAagteCz`=58Ybo`n?*-SWFfNpK37*0q(-&NjE20mqcA zlSF$O5?wdw4CZ{p;V24W;${?n9`WbHkT0M`z|KO*HEa5v*wb9}IN%DM{0ZDYJ{-;d zU!9Y5@Mz)O?-r!-SXtrEZBohfJjvG#=eX2jhQC6JMVRv(V|gz})BPvSooTDjFfumz zqm$KV)QIm3cvQ9~-J;|fA}aS~c#-3YETQ-v+_GQe>TtnJjFFIT)|7$7^9nIfFAnh| zgJAkK_eKm-Q-(x{DAE`Qt|hzwhr6%->uO!vmam|cN{1lb4N8NgQql;Dlt@U2G$J7_ z-6eu_cc+R-3X+m4p|psgg2X!u_I~y`?&p2~fcN~cKkT#Dcda|-o_pq+nQO=>od%^a zrih9lng~(5S9rln-gK+>dMN+f95%)XuT&UMkwo128%U5LjPlbxEz^)Yz)7*3vyX{n z`R?fWp$pUEHEqcPk&$T^mS=~*WG6H}zpBnw4#yAHcZUUB;~RNTRY8fV6(H&F_MjW6#cm1|nK6a`wYnpHt>t)_ z*)lZJY7OYIr`MpO(k!wZIV7Mh-a<+;@eqFMZ73EqI0(sj^$j~}wWX_I^WnP%r+MsP zFEE3X=Ic_gSUAE>KyqI_e+4r1T#B5X9($7K7I%&D^xxO?L^sw+#}uJ#!mX)OF2+PH z69(hianWK4t9s{IT6F%s+?+P&tb8Sh(CN_Ql^EuFVrLU4sU-gTsM=I&{KTk(kBnJe zuGow1G6(3KovR8BPa)t;S=pv{N2WHL931w!o9XvOD9douF&webFlFi0Y?~X*XcxpT zT6&P9Dppk+U=*j;IKWv=m-NqYJg1`P?)I{JoA7T@`$a0RjE%k-riA8rQL`({2C6vB zasiPM*pCA}XYi}D+TTa%Z%y{GG=dM_6pk#bvTT?8mC0>B^tsVzFx9eC*6jb!kBGqK zmh(z@k3-;`-Xr($aa`h&c9)iEN!RfKrd9yn9gui zxS{91Q^z?#&`~O}xMIJCT5uQK;uP(yfB2Pp=r2eM5OCe*h6UyYG11f{IR)d9XRr}j zr`#7J`|$Efd@(^m#@BCIuIt$JQIs9R;d8^t4_EG9X5;BxbehMb_$wup3eAxz`%Ox6 z4>R7q9;iHwi+o0fmwyrYeGL4*(B_u%4a&cNA0pnpN(aAp{u1@~@A*+3spd9vQF(f( zCHsbs`xksI@qk9O(Y`|*N* zj9tPgCCBLcE&ZI;epV-Z|8F35gw6N9)Z%j~J@=AAS&`LHb?@iP_F<@OnXucT_+5>> zwkt8JyvqealATx??c^V5I{u^5r;7-TR})K7yiOUKPQhsisFs)1^@d%XAA?G0k14FN zR8rL4xXU+En~O<>t-?w^*J-jk>mg-!51^?hwG19o87t^&9}Df!P?9tTDF3uKy|M2!n0!!qxM|CO zjRzDlk8jRny9cBvJEm5L9=uXARWfk+@)e7DlE8!^E5R58T3i)VUHr9U)LPOFMZOb; z({>6GUt~QA=p?(`rH#mIrBQq)QI|8N1D(p~e50`Su3;64@;fY!E)J|ENLJT761_)d ztLdZO5eX?L^LFMaq&6H6f<_e`gmu`KwkYxx_L__|2D+o9PyH69H31z7vk!{Y{M9AQ zeQL1U4yh^i{u=?ulTSXC*~EI2b!0v%=+7C-=#&n_7 zJq76Dpfc~EPU`V{CnKO2{;>ZAsd4dws3BP9E8qh;gw+49#F#5jB&`FJ_Fg8u8hx8C z{u_|vpEyYR9#`GMJ!~jzKzuZ!?_~o}?qG`3cJSppT%4(M;Lq*N#|!|jnhjJ%2$d6`=W_h8;Y2I6Gucj{seGY z`Kidw^N*pDTMY^cG}E8{BWC}+FW()&ealBrj?8_5J?a}bl+5VNc&wHX*IQl%G((f3 zp&1~9uSkVDIQC-tL@pkC%XtWCBoOp9+jc8$0FJILTe#abD+owlT`v@o90Gs=NN`t4 zpY`rb9B#_159{&2LTrtU-%<-Maz1uWKSX%DxY34~7F@qZAyrJ08FyHQ}H?#h^m2??Z&pAQ|tx> zAW3VtAvRSPj~zLx_k*u00AkX&mD2q6ppU~fQhg_gs+(vVp(FYuN)wf>+_l_ED*zD} zUonD@nXzy5Wy~-~5KdNzpm{?!0cqS=KSvD|H6i{7ur@!|nFUA=B7xperDIzKH%en- zdSy)iddtECSe;qG-#>5kC0~AUH!LeN027}Dp`tXsjkJR76+~?CVoqIn%98;f$&{tQ z>_AM_X0a-vl=H%nU@ONs0cA-UTK9os#dt_#1$PXI$b*$Lgm#{2}VviC)GQ5yU#wqy(sgN0qxxQ3;eh%H3x!h@u zp3i6ubBQt#EGZzKGy#x|_10Y9mh;P3yl*J8w!m+2dMh)QL=^W0fIhYKdI-PF56WBs zdZ;zXO3^7N2jdi(MY0dG%}cQSb{dy}ca$rseWcHLh{??iH&);-(0a;II{2?IsmdAI znHGLbjIG8>o}}>yl14{a7*SIHTmvCO!iy&gTU$ZjKkaitd^^fS1R(NM0h^Ru`#pyo zawr#r!%dwlY`4&`FgOlO6_EdRj!G=)ffoI}Y@`sx_Y}%@(jR&%#@{fq4BTQ?*c>NH z)3P80E>qP9?A+;-hYVP-Pfv0Pjy16uy4#YQ4^i{Bgy}v%hNuSEOyHZ{(_`Fr9Q+B1 zX2UY~0dGY}Dl|x31BkVSAck?JAZ=bFJ>}$}(mY0=uEqi^(7OvgH=d&-Tlp3wSn|so zggG$T`04XoAcz^VJUw*nGRzEI>7uk!?{HDy2x!bt?!qt>z#& zA7Q*yLu4r?*!pz3CaPTR`m^Kk^!_Xd8x@g({p zHA2@PCvX>V?YBg+elL@%vG`sQH!DF<14PE37KVR9} zsKtRihd%#w=J@FjW!TPXS@H|ub|*v4BkyTbp}+5xeGtcw=F(uO!D|_ieTQ2Rijjk# zfHpyxQ)E$3AekFlw?M)N-4E5`DQG_#x5>4?iLehNX$p|cy{XSP#8-7E3jb;_8X7{n zQjO&hZ9;kc4HgA>=gFxml!uuUf4aOw4fQgGGE^yocD4hTj(}AMx}_Iu{s4>;;k#89 zm{C^Y^oORb zTR^kA3()ddzslE#9toLp!X6M3n3XzX)YW?TZf5D zKxQA3dD)Z=Atk-B=g^THG9Lu1<85CoJgx_~ZFIck2z*pSu#u6h0RRaS4LmF=Hf%}b zO#^=eOC!H=2d40oWxhWA>dHfXxQBqfxO}pg#q7*Oa1S88g!0lS6RsV@?yQ8*j!5$w zXi5MX0qZk77|F9-ZTpVNtTiTeEgX>u3u1WlHzd|NtRl)8_TG|uoTUEcuU-*!wa;D9 z=EoaoKM$L1!xs#J+?cIGt1(MG5X2iM_#bmMi3Pc_-cagS%z3rOq>|=5dilB< zsXB%eeFP~wkn*Ry}nhaPRJk9r7{U+s2StJc@8oG zxt0k)RvVs2^NdwBT<`sYjJep$1xy!MCm=SZstq@bj!X)f)5`IqCgxy$l`LU!Gq)Oe zd%KLx!(CmX;(tRPi_6aMxJk?zs zvBzf(xvKj35G)F^hK7Jy_}bUT@I_Kq3OWf-JthPI-p1-6t5Tixx6W1GWQjL8*At{a zyI6Y&o8|%ZL{0^QmPops}qi5%8c`*>7BPIx5Z6KVNkz?<%ZcKmwVCQ~?gN+oK z@?hCpBoh!gTq?tNsN^N2ouOCnAS4yCx%A9Fhc$=Hi|*?>-pO0<`3wagsmE#d-C80V zg3^jZ^-npaP05qJ`wKrb9xgsR+-9cu;RZ$!)a=%b44gU$;A~1-&neSwIhX|)4ycp6 znv`u#(-j|BOi%CEp-7PWqx-a#>`y&b3b5<60)_5fA7M`ffl?y5k|SSrdL{DFZRVXX zttG6tqQZx2I=VyFz*_lH7pI`4;rtiK=G~J5LRJ$|p z65V%ePLBpztvIX>;;OQ*-is29Hh#6-sTPFA2Hnp35}t9dUjhvg)X&x!{5pWF!q)ZL zskl!g4-%FBTCl|i!mt0iU=`L<^8AA`auWU5BPeedw>%o9g0v54Su<8PX>5~UrhqaC z%1Y5=c+!JH^J?hAUxQq*?{^HgIigcb^<3hmOTB?89C&wh*=n^o7msq>ZtOmqU}U+v z4V5@`2|}Y^;q?3{<;?#LTwa#kH#db$lJ9JF4L3P>N$lmYeI_Su#+x8y+=uQ}oMrb` z2bq5gjAi?BbRRp0aPJ zQ3Mp~sSE;BJ%r}mHc~BP5aOmN zDD=2AVEJ{@FNIYq>?-7N{*b@fM@CwXcYAIDO{XtM#T5zM{*G1UTVnQ&@L+M#E*|6h za&aHc8H5??W)!_cIm_a+ATY?qUT+CU1`Z^1X)A(=^(2xFQ&NC>|%qDq( z_X5qBiv2}6XQ)1wwxKyC#YBjXQ(P6la-f?xn0d+N{R=PpB|90!Ch?6yH-5O$u`?Lm zH=#OCs8dSCDbUHu(lAr&`*1E!s4H7h$I`l`RU^WBkyl5T7K5}6PEKtOE6XGKvbnhS z5zPV95+ndvB$_(oybhvE^{LTmc@F5rny8f0Y~3lRQRhFzJ$-v=1uOsuZgJB#EyfFk zo38Kb%z2g4l2wAp>!G7;k6dFCuS$@L+ddy+wMM>AS2}oZZ$%`)ek8P-VQ#m}$ZdtQ zCWMczNcCm;W=vI{__15T#(;OPueJ4+Lh8qf^WuY-L0op}!oFu1NOeQvRKC;(0hr1Z z;O4DXaLqs6YBvu1Zq17XsvuJ4vC;8s@ErkPdMR_Kp17{H-W|2EF?NMC4u`F19T7Xm zw=Uj0OS{kgKV$nk5+?Ry$y}E?&{rShxbl`8?WNZDCRg`(y-xy1L3XKEy5t`A2n^C7TzWnrQ z?bu0`YNOs(G<@XUV>^#!9FKGN#W#BS4Mk}rjZ3l=(Lz=v>P*h^=4VJ;ycVn4+RCTIdOE_u zs%RPvE8}?<*$*v@IB`I_?3umw@ZR23QRi@u5k;NGm`p?90bmNX8WtC=CpeettuA@3 zkKGqm9Hw8LQa#$O=hm1`u-yed`I~)ZckcBpDsE8wLXhVaN|+H(r|+r-A&Q9Ru?1n= zp>(8?Y5nl-{A&gs4T&6#hX8=soIXey(wzD_W<#Nd-2ZAZk{kn1-Ka%kgsYZ4j0aBU z$0#^ylkuU@<4so9%C=umiED39O!h){>DiF;;m#Clr8Wql;OTw2QN(-G!cEGV%#YyS zs%E89L#I%r*4;3NbS}ngL-!=3vu&B0jP>_-YisQJ$Gnv@Z^_jeXe%+kePCtXpx139 zckJ3&21*9-{szx^&>uHfc`#GK18odwgHBaATY+EW9PY4pPjX=6BiA(IbKu2DG7dcT zrxJ;o=DykeLF$l;H+NNmOGov^Yo@qTIf)4;1GREv7QdUegd%(00-9_O?&PX{NAA#l zlJT#onMB;@YD1Ub%CKdQf21ujQd|Mx|1Q7C_M~R_iMJxlWMuq<-qVuZ#!o!x%8zQS zQZ1;eyLL-{&4O^Dy5vo%!DCi;KR)Ip&7`$4aC9E2u4Beb@%D0dDbU^^eag8Zpjo*d zb-}id)D+ng3@oj!oJ6>?9QSexHy$wGC{4eA;i=yf&U(#)n=6^iZ$}F>nSaJO+8`N} zQ@`&-79p83i`uzi+3AFq+ezK{=YMwGqIY2o`Qcb?FN-We6qB!_@ehU6&1c0}5~CWE z=;Ja{h3`QTULlSF5X?7)8E>~Cc3MWRPh2DZO6P7<+;n}^vPaC@Q9`M$7^NrEl4sq$ z0T`(D1mX~-^l@_v&3rUuLyZV;zdT~jr~{_+$ngSm$jd8!MD;6kp_lB-04WrJwdIW< z#&dp@B%_LIs3fh8wEUsFLTZNWHKfYyiLUp-j`Q2cE!l{X4OmF@Gtq8wr)rkJC8FE^ z#(1n#DbM$fF~q(Z{j`F)5mk?cG40WF43k8$B3~SF@h80=r1IXsWUEX9 zQjlK#%t6E#!1|Awnw(qleV$B+h?qoA^lPVnzwjYEG#q@L-2s1pU8E7gP^; z;H+1R&Y0R2#Horpk47c3Y#-rDa>*Sb3xL!D{Z}qagHn;H0JXBuvXXIf!wFn8&5Qc( zjrkBPi!Gnu;K24dgPV@j_O&r?AS@~n?$)#1?IF*LLlc~~f6RUaX$g76-=^A4PxSY7 zWF+(y_{w2WY>^K12cK`e2a8cfyW!zMKc3C#w%;lAEd-xZ`DqIjPs~r9h9Je@-a;a| zikzFE)tGfoymrrWC|(l*NIP4m+y^$i!Dqc=1bNF~6Mg_5gIC-^Z#Y%XZS4f@XhS;u zOKkS~H$p5fK&HS8zHYg9#j*VRNY=nWGIbZ`Q;_t(gYVCd_JQrxQZnc1Xe!U8+hpS( z!w}dQA^>7ebBoE7_mM^=V)CVpD{McLGJ+BxLJdLws|jb5DC~mnSF*Qh%AH93B?GMq z26Y1Ecp}S^zu5j#8W7qt#+lGD=SM2JZ0CVBC2}K&Q;w(89k?DQqi6i`!oLjJSoEd~ zq$^F`$bBVou&nxHhzR25bB2j|;Vt`cgOt_JVS6sKx_v&^hLP@2k2w2k1Ly$MK)P)k zG`IoM1XD6MKy=Q7rxHn8^`%JEkUPjy)aK>f>nD-IpOXMHxMSH2_S6Lm0bC_L@uMeZ z>v&y4u-)VzGK}Yxp{FM-osLtXRY+C%PSzc7c}j4q_IAzKS^vgZ)yL|MUZ9?K%`X?! zy-82GpO{{T^vPgb5E{!HujT$c{2MN^34vU-t(%YK*amXUuRK_`r~+iguYX!mm0|$X zwiim1X1_mHrfr1wJZ;kguX`p70Gz*YNu>IHf;%#|*Jil4zdq~QIm8jh>_0wszCH@6 zMubUY_LJxrY+?moXePf=Iye*tG%%i1<>e|y5>*Q0kyh6{Uo=<-`R8Dx92+A6bj_OG zYX)>Z#v7daAN#FUPasN=V(|GB?Oopy8MJ$P;%km^{6P8Cc89|pbBVM1jhtj8$&nt0 z)qXBOO`nOZEiQ!Cx{7~hzGH{|sDNnbtHpdn9C3G5lyL(0l=`~3KFaJiz#ag+IZcfd zL5>37rN`^!9mCfR7ei;g=EDV%E*O^v=NpQYj*MT~`FdAZTJ95m!_MdML5^CbOh)Mt zQ~i78N%T6VNK?~b&AD_So|1mKD)~c+tD$3lV2MjnVNnI5pMbn{mt-TnW;Dt$y`k(& z^Lp3|NEVqbRKo)UVJkF-0v9*ePg7NNz52|#X7X3t9*Bl~;$RurBLpZA>~Ud;D}Z`? zU&iVluL&LQRa%;`ZB>tx#8DV0%eY3&u`Z`_Au__t!;Z>YBX7OZ|twRLNC zUqgYR%`QXqGjPRxH7G3BD@?yQkb`TIe$k8m?5#1VF%&wI8dv1typx)6PeZxsG(t57 zlqirOD{i7R1EezpfCmBh)NfLXM>uOV@34E+SC)mGSG4|8zG4U(h&I=@%|N}WcWgd& zKCoySF_}bfpz5lJ9RJAfQ?6k0Al=GbS`Pj6CU+dIrt2Lxs0j|ukdw?WL(>oxf#=?8 zE#o20m{RnHJ6^8KPjnUbENq%_IVi%rg0?Q>W9Un*l`gG=N^KR5*fUZB^~!=X+kob@ zK*A98BVQ($QdrzaGe9WIaS`aOD!zhKHjqu>c-;;(1IVOGH+T#C7K%eyzs)^j4in zRK@UX!35B(it~HqtzWlgYy1JB7xf#he)q*VcLaYT{Q@7vx1ResVNNZKJ9S2JYAsQ5 zSv@|h|AbjPPL@uE^{2J>%dm8BzIw$mi{P1>F*_`bDwleXT@lO7hf;eC-7SOMM~j_ zQ|#tITs!Tk^z&>1xwg9OVM~P7_0H~#Xk~ULV+%6rPMBSx*=fU-HQkr`vU#}#zQT3O zayhE_7lN>!p|H+4TA3a&4!3F1>w`|u{18n7thCaIBky5PoX~AUg{Pvf2|3o#53Is# z<|X*~O>d|?!>A=@OSHK9kUA0aT;ffZNS=e}64Dt2qy-}#Ze@!*E6)88RsvL#xkfv4 z{Cwp(1-2F|wqHC^>()S52Nys~^ApDOy_5h8kh?Nzf;&JmhY$EAJKw~239LD zvdV>^Qn|uAUlSICOA>7+Ut~pyu+fA08eXc#P5891O=R18=uP}Y!j6iH_oZdIXa}S5 zXJ`-!ta4?AV^O1s9E;#i7SXXVRMZ=yT;5vvoTVj2}X)| zW&s924}WwLi!xlPL)lMf8cdp^=Wn^~Za=^E?V=^OW;E);>7n8zUel1_uLf8D<^qU) z!RYqTJ*cI8fgTp1J(Rx{n5q_cm}F9?Bxeowk}d{R_7IE%@-1Gy(D9|tyT)>K1%iu% zuFqP3jSO9SV|WPVAtvvJ<-F^AwTR4UD~VTzt1UDqe)su^6nJbcbRWL33Fq@zq0dTv zM8Hy(S0adU@HYB|N`xJD4!-+<{!d2;0uSj~E^DVal6A}2`g(8rT*T*EGF@J#V=up2 z7)Sq8=7RV_LoXOraASt%C=c@T6H`6B&0*#ZN}3C+*=W)qAmI1I6}Kk+uX^nzC2UJoouVY#ezO?o|E1jftoAcVN$fbOvUdASBhYozksX@f*Mni!$jsE87oSozAxcwoe zjZ96o!a@mG0*@skJ?kz`dHK4XM_u>Ah6e`xKO4U<1`f)^no9){Vm)_@G35>l-^sOM zZ%4-~Qy(mI|18Z(&{b5q6+eJVvue?*Pbs3z4cFn*>Z3Sf@mD}QA z`dV63+-DIUvp8Gx!?-QWcp4>Ym*hA0s2%OBd{*_t5)~X$H2hct3oJ*$!L9X;C*OgZ zlEm!RBnbMBq-0^IMBBxSl|m3Gl4>+Gn-tShRq zm-8i=cV*e}eShck)!mr%2h3JiWo&OW@1xxBLkT#Ez}#5!s)jw)VCWYq#4gF6 z+16FCp3PQX8ZY~RP;7sFe(`Sf*+yZ{5dj_xLWZPS9i(t;?|foaAltg=WsngZp2&i? zPG$Bu#kjOM9@AY?1N28nP^~t}ey%_d&HbY3}UvRPEj%jeq_F3epoMj z`R5JKXQ^L22Y#j}ch_@C5ZOTe9Xb0QWi$*&w!xu{#|re`+qy(fLWQN_qRJOU&Iw38 zxjG80;RKgf7QE(ebk+nn;s}afXuDOrUH)sMD*$)RFZ(M%>SIG*4c`n1GtO?oJsHi* z@U_#ePVwOEt@`WyN<%!{J`3fTU&IC?9+EJ>)nYxHn0R?hs7cwUZEE|OFx#Q+<+Yrt z^JUTVylni>#{-?KJ_yiK-Waw^Z8r;HIz21kYq@VbP)5&6$W^K^Bk(DhK0n zvZ@ExibLqz^us0lPY|!J0t@K;t$yqh!Y&IfzPaiR`gg^gbs|glY5Qga#B6$p8p|%? zBi&naWg${QA1cl?>tB!?{Z>C5moePIWTaV=R6ipuVf>mCL6VUGP+HC4Bj4I%4aAkL{Ybo;HM6qF%x0fo zYM8Jk?Z*)_?$)9UURc%}foZ?Ty?$h& zFYTPCG3jpLKF^cuSl1+Xe%tXBmaaFv;MELq<=5U6WQe}nXD_j&IG2$f`th$Ch*RD) zupNK(;met?qcHjVug|6qLWFRDq0X9*t3Z9Nvb)~6uFO0S8A66+I(ofPY(WqHcTr)@ z98qiS+q}&+)<;^k7k~28E5|ytJHJO^UE-QDxwu)2H7$5~gu`pEkGn$8DUx{b+VkF@QmI!_B^4n;p;aK0inG0dTV>p+Hh zB}i)9Wa<7`c{ahl1`UHHaEtxb@esj(>XckFJ{bU*G2Ezg^x$_#Gal)u_3_Cj#t@ zRn`9WaHG6WQ&ze|C;2y@)~$F*!xiRxtIN{t(_Q~ku!1@Nct!v7ANmaTJ*;_v7>j=L zfuuN_4bG`+XTM}t8?>9~t%lD$Mt0yop9h)XZXa$uf)*thVc@$S!nWz%&kU$uW`!|{ zk4~o-|LZ4CH;jI)O(1n=P~g*xzuC79WFwx?33KKBJMiJLlEPN;nwxIIQvJuIk#!=V zAw;qU3?OYfZ5v8C5f%XmF2w`6BccQZJAx?Q{_M}zI=~G&L`Xa|Ve##1o6Jl8H9ut%q@M<46Qz$_55gs{ z1Rm2%(5V#2V^vxft?jU37X2Snvq%^W$>H-ZS^xG-(lS7ByI^1 zm-w$$Jb9gODL$wde&tWTe+m_{bwv8`@ppILf7kr;eYho%A>|O9xjMR^~KnyF!0YDz4tqk^$QKh1s zaR32(>>Pw^v3r!&7&%BxtdidL0aF5SZ;~gVtF-J+`in+Y;YJ2H;?Lt{M zAuSAQftp{72@WXwn&JF%E}3|4TV{?984W+Bj$|{;@7P)~DEeh1`FIDMGEaT$iW&)6pQB}k zX|&MTL*_vmz?d!ex#?LbW&BOP(OjO!=yXEh?;Qe;^EnG>8sPwJyWv$N$KxcrU)2t2 z2@TK{xqP@lGmopXi_T`YspVcrc<)P~EW)V{el%CEhwwE*Vcz-^?>_-TyK>lXxuP+&K2(mO=+epJ%vOHcYVx z^nJrnO!Iuwx0QV(F-ifd0BCA?e%E`K)a-VWp7HHtP3>EYbt7HOegD_DyRJk@HdCcD z$L9sIkqzALN`BmRf$mC$G}tUW8L+-+A;lSzx*2O3k1KJ8twhR>6)XsW1}_W$l;5RB zg_9F>mEfc8q?jVVV;UepP=g+ZEMrD7yr7bUODp4fJCHI!`^dKV)_&<(HvJ8Plb`9b z@c_*|TyCNbIR809608m@tye#gJfF&$9Fp~!ezqu0&}YInLKm9up`do`-nR4d#_g^< zIlYLrfe5A&2EZzfNO`O43l5xEC62$=p3c`T+;aMm8j$&0wXhwgpl{<%9WqhG7=?Pj zH5W@lQ@@`(8O}vQ77DK(E^BLv(vzD+Qr8HWw<4h-ndLHMWgWp?l0za=cU9 zfiw$plHPZ;)U#rQG8^%gy4kFd*Qgn_`<^+iegD&o1`A|JzO#fvP;dn;$qhVaITlKW z7P-`qwOK|8DrY=DFnpMEWp&kRE@C0(>i9KFroKpG894E#7?1@U4#_cfYj?OnVWQvY+~rlHDQX9nfde3O zyIH1;QFya#zUf4EuUzxt8Uz!I-6ph-Z6}(oWbRlwpc2YEyM3haKZch+%0Iu_;52LAN5F>-LluiK$(P0 z&{i+u2Bl}t#rV`_Pr(zTx}Eu0yjGHh3N?K#pXm(#O7jJ>r-Y? z69U-?u+mV?=ST7DFKcQpj!~A~Ii5C4y=3RD*b3bj6=LGY+iQlS)pl3!J;f^`AYQIE z`#Q1Cs)mlnt;s3P?s2|^~kawY2{yEq?-TcbgasC*`pROj@qM7H*P~_@%{C8_0AZM zEFZh-DFsr3rVV@{qz5c%D^QoiU=27nGN-r zcE%DY$k5rB>4vXR&W857KSR$^S5h3?uq>65m%imh+zjPdYCvuc>?<@;2Ny@nV8 z8eE2)TN}Jd3|~wvASqWD&tbwws04ujj3W}Q@Lnn$_YVM*BE%{pW6s1U($Rz6m%{%f z^{#3%U(Q-nJP8+6#7@_k57KxClrSuuTfpN`r=jJEn8*IoTs91WPLpnrcr(;2GXW<7c{;9f}9oihqTi6V5u(gX^dogRF}` zHX%)Lu%gh*t~68fE@EQVT8LNVVqS66SAAS0xJWd8Y!^yKqpw78IG5dG5Ffa9y#G?Z zI9&A^t(?ehxl^*Fi%q1_-m8T(QJk*z03B&10tWDzI(~<_4hUz~y$G*wJQ)op!k*L1 zQPl_iB3B2={*lLjR>Q9ya=rGT{oPpJi>wy+c2}>}#stt?_9l8SE{+Ua4bI2 z1HTGr%55*8{AhxV;3Dfcbwp~X;2V!{PfQrNKPWNnmHZhK^(u#ahe)yQ)blk;hE}+A zPVSJ#b|@#=e@wlfq45;%e75GBKUOKY z&9Ix%3@hA%qEoN45Ucg?*C2nB4pe)=XoTz72X6Bqqur~lU!$Zi=8W=P2doj<3e8xJSF z0qcq!o)fd+pOwp)fkm6-+&J9gFyiAHDEU(Hj}V_sbr1udXw1MD^PfgsZ)HvQj6WC6 zDG$ECVNDuYD2nWod6S#gpN}kDH+o-w_tPye9~s-pzSHYz@Qb$-P3C_t6*etg7SdO; znBOtM4%zs*2>$sdqMNWousoq25$TDjegbfOITY$nl-jXoJ5g_u*c~Qmi3LBFnA?X- zufgm&b;tgVgb=_-7#XYDLa(&`+x5N;5@4!c8G({&H%WVj{^xj!?#siwWgeDJ z--BiHD1ITX|K}qqoFK4>>(T{Jgibf6r=DoSe>;01p_V*vxHgz?@SeolH z-kccG|1qZ^4A}2aXY7AAz^oICK5rTQiAOA5^###~|Fvd)4<=gTg6YkSQc;Ef`Namp z;BFyq&3^qU_x5#CSi(mSe*Q~%GGXe(g~H;=B;ava@s_8a=!5?dp8vTdFT=j`YYy8S z--rKkSOy;>t5W_>e6bOp7*+lf`p>NY`in;nS=iHH5`uKunkxUfd_*ktJl5~a%SX-% z#l3TgMm;0qA^)jx;mh9#;eSq0#&tBg&QVOWopchI&~X3P?th7Lz2uF~ot+*2R{pvS z$qE<|ag&$0=$~KI<-&_~Fvy*Bv~rE|h85hklNtDD(kFlUMu2=xLm?FuolLyD=kNvF z3E4JVJK9Lq{~6I=k8+X1HFsZ3x=bsah=hoR9mIj-3C^Eehgi|ToctN$oN80-H>(P7 zm+8O=cPx87Xa8KZE2=aG1(tlJ?)kO_Nr+TUl%~i3xyS27$5QpQyWXX;sZ0zYtloP? z?tOp0%PG-^e!j6tTr7nPS-02W^O=9{ViX}r)pxY>!WO(2st`FBx&Afc&+)T_!Q#3n zTi%@B+#^*~O`1^(K+$s~B9b6@YyAB%;Cakyv*P6ewPO%^;6rl$Qb z(8@OyM!+`oEAamz3NEm1HK`5PBN5>4UEt5^KSz82c{?z#y&QY!BSUy|MvE4ah5z*) zV*`Cu>)2E!hKT2bRrj@jtzXcIP&vQBcPGMjDu7XQ*^pgO`eS0glvu|K>2v<5*hmC& zL18wD{;7~M)L?4ZqiN|k*kM358sj$q5@8B>!XxXccrO#Y!wU60^t=>5YzI~d&i`_jj_)^qSe|f4?XSYHoZ{mP>E(aQm@+9C zWy1#!{*%*F z7+$Fr!3Ft0*Kitc&R+dk_x7iV+GlnD2`=_O?|{D~9T1(B$0;8Bf7MxM;fWndde6{) z1T&?St2@C3{&|rkH*EFl;@Hn=EBJ~IVPgEh7clHtdWN7@${y%OOfBnSsC%#kyHbB{ zhotXy48X823Vu4d{QvfK|CkyYh9w~x0DMp#Lp`>{soYQ;XNglu*}jkwOAtE$WN z(Pm!v`f4B!BWM$~{^jofP`Zc#$})(A4W7ohw(TN!McI$7`+v@2NdunHpX8y^PDDWZ z$}`{W|CEC+l(DG=1%D(<<_k+h>!>Y)Bzb*TNm?2IC^jM z6?q$T-!-a#PW2~nwB6qGhc*d9tv^+U}>XrhXxnb%d?L^Y5xRUU^d~S2k;_zJ!+oct$Qe>4n}+8!XT_2I zQWI05xw3|?uI@{M4D6Y(b#)pwpLZ+Lm)`ZWO83^Z>4QU4_obn}b94_5!Pnlp#5&dNng_IfE6`-X!s#-5gq_DZYiXwAhT+^l?)mv;`c- zvi_$AkYxkufR7pPhA!Ts@ppQm?uogGvBboQc*NVPrZ`Wa@^9_|GutM+_U^d`EjVCH zX{TgPeLj;TeAIc02)lb6udpx6xwPcW`^(++cws$(41PEH*woDiJ5Jr%wgI`#SVt|q zhIVZvrwQ+}$kG|B72=uoXVtWXRBl9tQDx>S8qJk7=;yo!^!k%zP;sUB?Ew=#RRPb> zu}lQb2+!Y+DwOapejvMM>^nGAV2hjB&O^z8d+jr+8`I$|Z-xIj4@J*if%DGi>C@c9 z12(nwVwzuGF?dUe{V``Yd>mW-V0C z$|)i@AZO^(O-`*&CU&6d^mik~QR1lR7zvwJsyJGi7KFK6fVuAFPT>%W+^;{ z&n(Mmu*`)YAM=d`orz8-Pvx2YE9w?J4{0N8G?oO-cWmSy8VB^Vid?J|KGI^LWwb{4 zfr_Wq)~EweS4Xh)Gs$A4#jtTSD_DO7#-Oou7L@b!@ySO`sO(;#zAAPmOD&^JmVu7y z_513NemE{Go6~mYASKRYZQni3(=H}%w?Ua`GhYitBx#pP{IEl6wJGVm6UlvFcRWiM z3Yb`sraP;j!V}=Qv{25_M(l91@m)8<4sN8V{z%OIQbQqV_{2 zJ30Ls3}g&L+GTyWmFI&L9m5R>2z6pN*9Dy-jA=Ux7K-nkzbI(8M?z*v`de1^8D5ff z8g^A2D^D7{Y$R^sQt}vH{1O-MJ0)W}w&?QjMM}e|_5BoA$j8k#UDhHDEvSI6p*xmY zg4cB}^O>_8{@!IV^q_CmH;96%GwozJlBfD_hyB7o!&W!HJuT;oxk*U)E?YQN%=FbPa&PKi&fn4*#W`xqws<@&v#j3QM``V1d52^r&1JX#&rBnM-LB%VQ z09n5cOnBdW2t2OxTQjQ1F9BhtdWd0I?-?dP9piIP#AzkDSxgp=CMaY4!tRr6cNNKp}E)jty8FgwHW89_JV|`;QEjIZ6zk&m|Da&s7 zQ*XV#5jJM>-dmX=+e@LlWJ)3&oTh`F@jP~eVUprvACwKAkrRb)NInsYP*^BX$n*nu zT{c#oCYEo=ds!ZA37mj63gSw;{uFEmj9)ASaO~(9)M>~NVW`?3irpznB)=|q6qF$= zKj#dMjX%!odJ%uPDe3zI$A0xOY%V)5uZb^r{=|jHyk}*f9nKD=k!qEf#GM?bSz&tfCb9;TM~)zCvAsIB>Zl zxeJhJPkk@1s`&bNz)>he?=G?6G#TSGLx9eswf;(sRhB^~w3ip|GEGy4QrHjre)-RXO7HQ0)cvQqIQE%V!HWL0OwTniZ|F5Ad|R=&+TNYtk&*O2 zuOoXRs|Auh1Gd+~^jWS->=J8Qs!_47cOcEO zzMgw}$)8W=mO`UDqs5)6zNh>&?aXN#YV~cjm3{!A&Ao<3^-O2@gY^|RqmaFXij zQu8&JQ%S~g26rc*-Usw5btz50RZwZR56Tps%+MZ?KtmwFhd$4Da(*v9peFRgr)BM6 z>e;7RFskEA;p(Y6)keW9O)JYNO+ZNAs3?)R^~@NSjfrg?vV2^AeV<;V@*KyCV##Zg z)oO&Ro?Tz(MZJKnWHYo;-uam@meH4Coux5elXG??>C_C}a{|}T=e!snvE=A2A2!S8 zK)7oQ5^CtwLn@wyuTRvh0-mD7tXHJd1jWk{zyyQPrkg&-Uamd&oB5FgW6SzEsXH^2 zwr!#qZX)Py4;?iN&r-ShSmDW!##1pzn?Xvz?z_lGS6tTK54j>i)(H$-=q>mG!nlwc zadU!_(kqoLX=&Z<^wYKSF^;fPr03r4(y}29EcD9y?*|NN(z79zwJ@^U_s_TW6TzBx zL94Wr%>k>q(pt;795o(x+2K3KBl4|Uv5C7Sx5?1X^n8=S|Ea*hz{~4Yx;;YH@f}Dp z((`3)OX2s`THul>xvx$L9VsA5_eg}OhEL)n!n4glR6jlSK}%EUEt}7&teUTlt6T&V z*|_zw(mf_`F1d<9U}@JZL3aakt#Db|-JaQb|DMu|ODXIg6~~oYj0i_(;&ze$Bvd5X zoH+LZ3fG68J%LpFASl1?&R_T<>nnm@FW{H&gs8W%#m{2AZ0>f&L=jck{+_IkL58J` z^Lt}NUC>`!Rcmf?eZYg9IM*stNm_t~1GRY#yOc}}s&GL~9|)BK31 zyXJ`vfbe{p1$Wpg?R+O^g^bN3zJsy4)|zgnR1@6^MetEne1gs+B^;0>W#blmb}BmW zfwhG&TOzC4b)WW*oxEvc(dIYYGU!ZPa-5o@*C0<(%Ew%kAbt)5?|KfdnYQ}Jd%jv) zbLN?LkG`jhacs?BOn_OEpxN%oXwmVKYR^fZeRQR=Obh4kf%DnXwn8g`J0BqgftxI3 zTZ1q@rMpa8n9Zfwto`s%@i~%3ViHH0zY2n4zqGxF!hn$n*nm{*U{eC694+CSSXFo1 z)8pJpAw}auJ48wJ<)CDS*P%91&n=}~y*n}h#gCQtXm6R94N}+KkqU+6`jk^8#gd9X zM^gI|bKaW4iu*@%-nR=QsyGqcf$qJBK<}{XOG_~!es?J~@hQhk7)0`z_T#g^IO2w; zx*6GYL6@syc6c5HiY7do6E}z18K7}}GXQd{SCiLsJc#wnNHM^dBTz|I(mZg;CzU`; ze_)0F>qEy;LHYWxwqpJ!$0BNgA%4GqKF^qboifrFe;kpwXFv;4-vRqNLdyevr=A_{ ze-W!reS;M+3N2Sx-e(lC3~Hj_O0f|70JJp-g9{=So-5npw=q(<+QUo9?luBD%j;Bg ziuja>wKF&zcw9~&(HpSD2R_ejv_nP)$L6kwC3dy$Uvnos%6r*5cT}nUb#nbeG(=$z zCnap4S_wL$pFI+XFyvTG(26hXpbaW}w!V$~D5(BU+J~XW5@)e=N-SKk+o?ERjTorw zpjE~iaO_NrP5lm$CC7o&CD|F?^!nSWv$+f5U(V{i17}|isD6mNzleU=9W8rw&3Y2l zr`w8tP71`Kd^OP0Ih>Yr5*_o3k>VNTm;)fMM-gQ;i{V}eVW#qn13IR2bWLzC?O%Sj z+edQ8`bA5%)$DV2mO%>CL(V41z;XXI57*ECY45$msgC>q@sJ2*?p32S?T|i1F9RO+0-@+dz?>m^gf1X-H=ie3)(!Fp#Fh=FuLFRr0p8vO zKK^FY>t;%1gbUNWm0m7lda3l`KcBt#C;&WkFL>b(A7@@8I9-5zo&I#v>&sN8?{qnJ zrN~DV<`c5OMjdfC3ajJ({D@?A(0XI)u2k(GgVx>a*)NatrMiHzTgU=sr_YXfuTT(e zJ>P!*E}tqt70~F|huf`nGtUuETRbIm`_7UHa&;M$pO6JKI^#wbOae$pkVlCo!0O(4WT|UV2#eYdVsSH)C&+IsNd#e^J$P@YPgn)EiBR1WKNE7 z6mZUA=d;@@@ynjN3mg`=eU8@3o!=g-rCf4wDof9?pvVj-I1A{6 z9$BDdvG4D?regBZ1(vsKlesTO+bLX4SsYP(vYaEP6cUx-g8MwiylhtIN03)o#0 zA&$7e5LnMb`1Fk?ai`iq!MAdB`d0AKQ{}#=i=CF5D8|u>7v{K3wN$u;x#sP;vUR^0 zKC$8MxZ~H+qY?65+FIAGi#MB&3y%*rEp4i8A)YzY)|*n$#juMx%DbzbXZ;~2v=Me1 zZFGHRO55TaDvhLH3EDk1ADYwUVhGkUfZ%E1k8aP)U0gQda6jA!&2%@1yMPcrwl3bi zw8#TgFpWcdsCFWB$mumd>D^1nD6q*2B3_L9efIkxewo)9G!Iy~c#9vCON6&(U366BGwf2S&oQ%pb;* zXY$+5pGtgp=J5Jwijab?C5FPTqgBz{*ZNFj$9Yf*;TV-h=(0&04ptJU(Rst{ACGZQ zP*SIuLAkjYrjx5AWMqUWnd2W=is|O-+0Xgv!9LPGGb4{QA~(CE1WxfMlsvk6wcaJ# z#QRhh$8c~u-JwU)e2mT+jou^99aQ_LDHfL#`CYOEC$>=;nF&U(4N1SyoWY3}L|*#} z3%iTpWN8&fM87mTd?(LzqhV=^&VZE$S1*&YdUA)#py(^0PHxs5IYdpYWvCBUh_nc~ zC2f19o2HcT1nh|1PH36X4f+0V)jEq*lvn}x1l^p5 z&WcS53HL(XP)tk|ESP!hCox(={db;L^u-dc?xov2#JTw24l3q&F$$>jZ&Psmh7oYR zm)qE!K3Bx@TyGZiy=Gh|Y4eLAnQ1J#=pfI+O|*R)w4}wIgiq=&jUHF!7ah3KP&tLt zV5Z}?{eW6k!$jdtMZ3iVLOE&aCpvdq8-+}%g%|JZIxY3^MH*zQ`>oquosATTJC2s{ zxSh?VpdNGInj(R21H;!_asG90An^=u+o9o_LI=aii1tz{0g8mHG=!b*u!64b0Zgb| zsb97l&M4qg3XQ*&DCQXNJeFKbUcP+1R4M^r=B?W5R`cf>gRp)kyZoY0rOQY{o#oxP z8hRqV8TdrL7b=flXSf-ClxYY`c z`(G(_F3~n4FMZCVw%fN~L!f+AQ4%qEa>3~MMJ*GVxRQ8u>;&blo$;+jA!e)KgKJ?@ z6{8;kstjdJ6ndegu$q}oyOt%g;H&%RK90=6hI?#O*lXFPl(;JfYyqV5>XR5t! z7PmiYcyL6lB4iX0pEMRR4Gp9#SE+0=SW4}Vz6`M!%xQKdHOf1_pwNnrv-J*fN!e^7Q z-Z&Z31J#hz0Ud8v#n4avtzX`Wcdpor7|$p#j(`665~@D_?+q7Ce6!v`7OP z$WHT~7srY}2(TO%gN_>m$yn;g3sP6RL%2Ah5mRp1l4PU6;xL-|;|{r2R}KxdW31wc zC|`_M^J)nSJ~1b9W4&9}al^HHT!2RJ6Dp()+J0C(>L{D4q;11;(90`(R+u{tUxkvI zXrh;GmLE}5&t%)kT4|7L^tGu_3#y>>MVbtm^=UH$bg_jO=xAd;ad)@Hy=t-jZKrFy zp69$d7>teQf)<9#?d+VcE}>TG__FyJ6J$8+@m<1w!$vi+zE$J1zS1H&kKJ*}EE==p zd|d6r=4GRd#X_I2O}{XBsH04Hv*PI|zbkc&bNP2w_Xl3ioIb}ytBVZYc_+)ejCidW z_26cV)7>}pp zC8e&lM$3sDB-7p}SLUKak?2KWd<=w5EBfgUe8npY(H=$3-Oxk#xAKOV5r`#M>{H_J zebV(cHE~>l)^6fmNSfMJ>i<+;( zvqCL`jVGU|aLjTa3o)4YjZkAqv`x9Lp7Mvu-<2P;mA!0}RO`5QbtW0EOvAr*)Y8Rk zGIpLhkS4;)BD_L?3eW9Gf54Vt`of*d?^b7cp0ll?k_&#bpt`LgcvdGuo9pZ*8|7Rn z&BNuRSqB1-vc_*mT&kc^))lvKT@*uSU1lIB;{VB(kgr6|E2bVYt7k&jsJ7Kc#;=f>2Fizu9kN zCgNk|Tyo5ItVUP9;#;lHYm(txQ?5%AmP$AB&ueP0UeNUan0!i7DT#(o+-Wmr@!gv& z=9!ZHMkv*E&Y;>1-fh_Pd7OS^BhjK8Y7qzhz*9N8{4`lRp6MoaQR2UOpULRE6I73~ zuqd5{B!EE)7v9eAb2V{kM%8l>!SN}21Z<#^E~)DmshZ(^btc#7+WC8Obe`qw-MYk@ zKz}re+=X@3xKx06QSP{uR>hb~Lv>CUT{h#x;%8sSBNRH( z44)I|$yw}RA@B#6Mad-*`-#d*kW%oreJFks*XL%&70on{eK`i>2lL~KVw~L}a~(Wy z2P-L8adaR5kXTP8?JkU}iHOOfm~~HX^4op5Q8#L5TGr+d3Cgjvl}hwSckcFd)vS5k zF7E#{?{b{#_@zLaHysQES#6FbI=52q-rx+C3mv#CJuVt3b28OVa$z&JGo1cuqPUJV@FTMbq2Ky0cuhcRK2CPL|U zwwAV;wJe4So-QrAh0Zl5L7Z?xgG-Wh3TF{sRl;IrMNs~U07o+?j_$Az=nZ^%;KE9q zbphRt%A&ciT#*1LDxvfC&jN-(T!fyjh{WKmOIM(c1;w1ek4H-C;Y&1l-6{y6N?%Wf zD3RZ2X57W(alo1DE!frdRn1A8G#TkB8^$>0UnicshHh1;b|8~*k~)??!ff^JUET2z zwmN$cWRxBtdwfLDf`wTx;gov>hD)Ot8)DJr_-Vzx&-Fgb-vN0k&BtpGS++;QBPZ$x zcOGNIbl`p4a33U1H3Q~hvM*?sa$mKSk$f)r;l>{=$L+) z=e~3E$dL3;4pkAmMs5LfBxxu7tLWq{^x`K=ZV6C{Qqz1QuV>EO$4Z5 z8=&7ow0dtEs5Bs$&)~cr&yYy#8GD_W8Tauh5F38S_wb)UX*c z$%$3-xT~{QMsP6A)n5AgED~|gg$mOACr>hxxu0OKRXcZFJ$*4IIH#Vf}X6V|@7(1XjfDj}>8jU6%@$Utm@0D9=9U{;! zodz{v_N#Y6<)QU(f1T2=KRuH;gjp0_Q;JVvCrIBWiNzM)$uRTERh&8+fw_|-iLMEJ z-!@M1M$&oA#w4YEs;&uIM9sj>kodL^nr~SLBU#Fbxwo^=0y{`T>uVQpTi6C80@Bd9 z^3S+mlyLy6{)qQ<>)w(`?qbXKi}ljy9TbEw^kr6^h$#f2i1ER1m~AoBA}klHj1v2N z#b|OC!uyet(lE@a+JL@-@kiInn{+Y<;igIYvlEO$Z(BH|fp6tkxrO(tX(=2}4Y=?8 zm;@j_$hV3cnA9HDufv$7bmb~c;OzfUpIP7p=**ok78)rFUSTH}ASy!taT6ScegRag z7irH!bSi@PEc$r%1K=bi{_@G7PmHz=mVmEHqhSQqWS`>uk-inCHHZo0;j&cgBs>zb zvYBSK+~3f}`TYC^^e~t2)F&C=2XUH15Eom3{+@=DJK?(i?+)e`QJAa)Yvz7d2-CcS zUW^koeicNTzE&a~>$g5IZ)p8l>im>oO=D$p^DnI@v(X=07#@u+q=<4x@SLSIqUh^b z3H0@LV6z57;5Pex7EuN}>yH_Zr{n@PzZ0X%RDnF%({Lxj1x~A^hnfvcEJongoTt0y zniT@{mcx%^3iUu~=jmp1EZKxR<_6+DcuoEi@X@ej@^M0YeJRZE5ftXrPVVXeeu7uu zZaf?|0kg1vB`QiIG1^`3dQkVLz|-KZZNe7f6P=3Mh|QiEeeBWBV2gI-PXoF_3n-z4 zn~a{k^wkAO>IeJlek)l-K+`gDvmMof*`uDDMsxzGX8i)jH8=N0?l^2BDFXiSx-N^n zk2E;%!j3}Zg6dQM^rEEb5{!{YbYCU&!r6+h0-V|^JK>AsU4(ga24#W61dPi$;)+s+ z7$0EZ2)HJ(-NvU$GaqG#@Tlxk?l3uiUpUr~yAw8C-)v8b3KoylaR-_SSUE#}WJ~GU-vl;D|>VpVwuY4ZwL%lFuzW50N zqRDRC^ot9Z=)MEpmHDRW=OXt)Tk5A_r~p|lrmY~37+8tj1OzZp7wwtEWv#u)uhn}Pzhde# z<5cojT)-{91Ejc;nr{%QAdb~km8cIE1C#>B8Ywa!=-%fqxgS4wbi##x0eoGYEoH@3 z9-Y)n)Vt&<^KY;A4I~+QL%$QThl4RL-S2*$DCz7U^7l)8K^VZ~?x2LMXo3tC)hY1& z0Q5?$$6=R7cThEw#NivsQ>BB-n zLy*>P(lmVE$l?LO(;N%V;Kis%EzBw%ivt)qx9Qf zHTT;cgMAM#4?6o+fjq(457|B-!?5?^aNpd4t17s-+-FJ9^1to-Dwsvl`pL6a^Uyb+%7~yggiJITGi+Zv#fR;uHmy;fg?Yp491PXm$f% zm^_~ZYbZ-SA0|>XXPvp?9x{-U`Rpq#S zJ#0il*TwgyMYAKM54^y=y#4CCu|f?Cpde|^(nMMsniv{3l8Ot+MtFew8$sOFk^n7l z#GL>h@dW)fS#0Ce#)Rs7>k&ij7u{@R--C0m2@K6$J=;2O=Q;J>dq`|h3YGn zKL|Gl)^r6-D-s~R_WUC`_f_(pMNmDX+nqd;b|*k}jL$3;C;)ac#t!i{LdF{7p!;R+ zoAioS!MtlwQuou#SqR-Li(-E34?b8gvJKa(3*W~42J*1|0@#@APqPJWfim2nbXfga z29_*@>6pjFYi~?;#eaoF@tH(JlTDH%7BTKLhe=CBk*4rzd7mBN%{7AQ5na`STbZ-n zA_uc{FES$rNdbSXKp}1aoyVB3WrlDn;@(FALpVT|f{KM8UHEptn_Z`br%4ixnzC(c zp(A_~Ms?rM-*Y`(?=>`ezEtDPMT!W<1_nPMiAES()c>plj6k9d*NaLx8^wM_UO06t zM<;S>(gdE~H2*o?kKQy#ql3;2GSnf1=^ifcX?iZRmkof=yX)#$(xcsPm84YlD}CB7 zeyjEHPpdh^LK7u+!3;QZsm%VopJ?f6D+5E+^TN*0k#S6HlMRn-72XlsK+2Ax6ncI{ zlZDLd{jL1XKgW3%I?U>GPAe}pxL3R9mj=ym9Qp{g7z=SUK{8tt zUd|_elS9Y%CJx#2=!de%o|kCSB)qSCH=vC$NK#xp0Su|-CTR0Ers}LAScbfK9vn=2 zu@H`X5yJrS&Uyp|;^c~n5nf*K7E%^x-7Y*3C_$S3y*8)te3cIhPxMUMeUoz%wGI%X zlXCi)J+OFF`fB9VoEUA4!uO=%C(;Y4>YBFGc9RaW#Q{q;3SgYFlh{ZhaqoWTHQz0I z+~=SW6asnvU1GB)?o{QW)4J0wGT$br*z$YO*)7#1oZU;?fiOkdW9UTX1tEBWK0zn* z0Dw51ckQBOz7&AqXBz5y&m_vA8W34>{BXeB`B8fX{V{A8XO@f*V4|*!JzS_LOiD=vys~$*>qeRiBews;7u>1rp5TU~6NvZ;aL!bhWTiYpLc{Mt)L)t(~Lzv!IK$=97pSQ}M~Q z_XR2>wx0#DgKN6+)OWd7VQbq|Nq)~QI?D3MYEwP5z6v>{+$M>Gy63l~Q_;eOH6xQR z?pE(9buZN|zVCmw*zwtA{k*U_s=e!FIlydUXkXUIHwVr+NAMO0cIKR)Z?ZidJ>tNB zGCJCt%nDFtC<6DL3x1UMX*`d>D9`9n` zwzwto@Uv#$BxcF^;i~a8u57dkncj0k0(AZNk!l%q#cB`oIWUyF@zAEenu)>9n;7jg zms1#!d2vMd7aYApXrWtwZyKHv&nX0~gm%^PZ*hNicVh%MR6?W&s4tr&vDg6ag9^RgIUpz=EKZF89%q(ITH~`)zX0=az zWc4N~giLInUYR$ZZjJsm?=VsytfZNx-l4CNiV_*J{kZ7=`Mt|K;MvnBEX6cwrH9xe zu>mstV|!<-h5zvhZnZI0L&mhIj(h?kqh03mMqM(mG#SwxU5c|89lZn!Upv~yy1sZj zFOlKA#Y$R|3N*q=rUt%Bv}|m^inTnbP8;gqvPzR&PczQnJiERj>g-h3E71%#H3}=f z0Nijp+xOk-S8nW8?r6`E%Vem8{Ql{{b2T3UF^Y0PJ_orag;zoojNI4j7ou!9IA1h^FjTMR!&v7Kw;*ckb2Anu zqe2DfpczO+lC)a4R!ye=_=hY(e%B_WkCX`vqUbjhYub3s*TyTzDP>*-#BU0BlgzgT zT|`o=B4!*bv+bt&QEv;5hRx)S=VwpYY4`jzdOYFNQb^{mzpJsA{$On`eEYAk6bSkb z{<-NLGPL%vtrLOVHJGxE0mpd@lCB>B^}!dru-~Th+|y!6I7BQu+2A@{V4`q1BIvYXfi;Gnze!Q{JXtj>KXf%gi zT2^>mn{yF_yVFxk&CDPCM1>%DtQI0Ns%4|U#?HGLZJ5c<{Z1cw+k7asDJ)Aw$ zZfbwA8JCtGj!A8T+enuA-?!1TMfUIqRA5o7B@N5T%8z{J&v$;<7V-N%U=wG>L4h!2 zz=EltAf%N$$D$<#2~K6IaXzY8!p!M#_MXsd(4wJ_Mo>p(F_GbKVn{i)UB)L~x9)NYzx!%*{`fT%*fJ>>9TkZYdh!libyBEk2Ns&fi3O%LJd5VyF)7Zl+dBh~D zs9&><>vJUOMt{8S;4l&xLV*e-+yZ50Df=o`c@tM>zjNsLca1?m#kwX^@;!gNfQzQI zYGb$7wCwj+cQ@L}zHco5n%*epd9c!PaJc{BFp?bUHI(*AznuY<+Imf;Jyh-+e15!= z*bQ-qT#8hl;roz7PjqfA&S>aKXxf`2n46HldMa>z!?2-9t)Oa@2I|>4#sW^lI}gp) zPy>aUsITt};vC}8SbVZ)g09I6Nk{Rr38Usm-}iwd4m!-Q7AYV-GI09^!Uj>Sd8EuH z8*zG9e@{TxZW9t8f2ndC6@Mep{$USa{M-GmYw3z?pJ&0YX8v->c`F?oAkW(TM68p> zyCMZM6*=Kv<(1mu&=Q4}P}7nkl>SWp0HPc^_OI%ps8n?|&pWuHpL#YHMb}Hzf08l6 z{4!yg!UR4G0;8B-hTn`}vbqc@=y|omst{><|D^iF1iYgJlAHE*p?dd`vdJTrIOxP* z)EPH5E^*vqze46K}YJt2T&KWOXl($WosCXXq-~*JB3q*IBn?dAPZpcn(8p`6m7$!l!st2x? zRTLb-^9+(u)_8LRqiYdTNP7EX2I2aTfuNe+^k3J<2{GongL{rHDrs~{HBXlw$!s1X z3r%vWeSMB&8Z!K6*Wv;SdCLN?OCm}+cZNPs1HKD6rxyl4zWem9!Wx42@Zww6DaP`* zHorkXFJ?VH9z4T5LD?!nxZGKTY#RX!@kX;2*yxJ?*3T2S?uv0n5Mq6k%kh`UyYV2+ zE#K$r2N19hfBW7v-UE>RI{_M(5KZ);m`4x7(+iBfKLEwJt3DCw7y6fUgm-Rat)X5n z6RLU1^0;HqlItn-ClOMA_3N=YOJeY1?3!r#RfC)#PLBUj>ANSdY@!svRnv!hqxQn} z;XKrX$<3c6ew&0X+6jcJl|KsED~i1@^{Uos-PS9?lvW zbxGA`0)K{KGs=Bi;wcQ6V-q(;{SqkTsd_wVt>w_Zq(m67Yw}%}Df4r8!;?mU0C9Ah)IqoR}k;)4bbK`_PL!gl4Iv@GWdskWMOXVGo?8 z`G`1_>^8eoeZVw|S^lntRzwpb^2YjYxO5(@8rf*->j|ehjgQXTK~QIh&I+rwReX;) z^us7TA~2JsC54LDj#qP|iC%imJo8HUDk!lXI~gun^qKsGmV_%~;a0CeGHxW4N8|xXV;ZjE5I3;x54UGQtlf#1lbZ4evBwno~zbL%^NY{udIW^5i@-NX@4am zHaUq2f$k4UaYXdk%*~ug_0I=i3|c9Sry%aKmKxp~m`wPd@iH-;2p;~aNyJo941U3f$=t`aCvbD>z<;F{g1+b) zK7->eglr-~y}5+eJye2=e)(kM3xf~NJ_&QCNK5(im0g^xWjrxCtT&@cEtB{+j-_L+ z{3jD#T>!q>-MCtI>0>+Mp6s{NdHNBR>&TivPG2X}0C~2?a1PRjJE-P;)qe{(=s0B0 zho9T!bTSus!yoyowjk0$?kp1jNedwUABlW7BH)rHORcn|giHV2ccuocUd4Vy5UBu9 zN;Upl87NCfM9v2eUSTrKt(NaxeOc;a`589jPz1^sJet7S+yv@Uo8Dw=rZ?^fbu2s7OF%AV2e}kasWq;@%pTW|2vpoRg!K8Sso>v@ zy=Gs@cn8-D8~cmmz)v*DMqC-3=yIL-rXX1M)(5&5_h^nL+;Fp&_O<+2Yo2nuvJb?_ zbjdo;vx4rPLwaK0U8tHUu`5ph*d2SQ4&aTMC`h1c?)WP$u z{FYw?Sis!gLqiZH*h6~s` z_H5{&VHJwJ{MQ9>`+j*?0YnhsQ;DK)4xJMR)d`MikG(b)fc?iS{UM2cm?!jYbXvxY z@~z46>U^0>)VL)=W}rFk!jj?DF6+t)6ogq@|LyF^ln~Ta`K1?cbf4d=bhdg|qVBmJo`Es=R{~jy<VTo&eN6A+uE=U^aL z6go7mcRf=q066$Ow-KXUz-LplO(lZrS(_TMDhlKpF7UfRLW6K6PB?YF0Xj1|4FQSY zI<@8Fwbw%W>T9vo%{e3|r7d8%cH*Zm4AYQdE7AyWD>hUn0%=pkS_zq4!0>Nl*zX|L z2+_-9ZrKRwfAdg{h*X~IIV3@$zahJhI#fecOA+;JJlhcLv2(;=V)O+VTgExM=j`5I z2a*xIx8N(SMA12f5z@z(u?MH#!@XJ*DMJC)Klc~p0SM_0N`=@G2lGPJ)<;Fd=fMl? zk>tmIZN@|yk1vq-Z$r+}TZ)LJ0;8fA{DBZo!Qa7eW+O7j2NI_p9hnV{)fyt+1?S~^ z#JlUEDZ)YJZ}UU6NmpRnjn~|H;SPk+mHZq??f#qBh5Zc8kGWPlW$Fmbn^GTxfG@;q z93JDeYNu#dsn{iN!)$F9N;5c*zs3wVz>Rmt?y`{j>aVxBQ2nDronPNdiAA8Xhk+ zGKQm-RZv^5eO0{ZG$4if)MyGLO=QEP!eHF23`GGP)>rir>Pj7)u0X&^F1P^G>_uak zmYyLO-HX(0{w#UHPG3aYrV2wZJk5?)^$D6*!NKVXpZ_O1#~fgA4%ZLNqUw+r~VzzBIN@23H1cSGkOtiBrbK1+b59YjVgT)IFMG|l) zf=&fcS7O{hvZpUB_bW9#KSI1G;thVr-W|usz}gtY7S$A~FH;V-Caw?akv`3yqETTK zBJRu^(9$AbRs{me-<`liVw}3j^^#94&sl_f8PZk_ni)M_=lFHb5~G-gkpEr)Kw9qw&(538)#MA>g z6>(w;_p%du{Cq@-k~TUF)5Y6}0kw;U{+ipi%;=FCVu%8~9H?xBLY|1;*Jm3FbT;2R z15}skz5asn$sP&T3B$wtzlIK_0#DFCU2Z2ktu+&v`<1DolHhYH+CTHK(}j!IH{G@m zM0D@Ol$9h>P6@P$B#HdEwinY$L&94oZSj)5vxrQVN{QS&UvknWc5s)kQ3kLW>;8_0 zFse`HSDa0K(napXTgHSK;#A&F{a}s907N@c2{$kA&ilFYlVN@yvCkvmSh9$97sPF5 zMNWH$a-IyT#A}+MRygR4-rsOC0j8a=a3d^PWs635)H~}YM)~f?r4&wVhP~@jQNy=& zsVtXOS86ic%s=va1#WL{&R;^045sT@#D{uysonyWc1P^gr01J$Tk*2SYjv#y982{= zfyS{~Uo%+J+x%XCB^!KxXUp|R*r!7si|pr%)3>0)I&YZHF_wJ&Xni@^jE9qM+YjBt zuArb!m79g~C1+|30itOtS*N||MzTiI(M5P)C<9?kk&Ki@Pz56lPa>(*Jtd@)tWQ)% zMgh?=@b+D$(-gXVbN*aLoB`8dNDak?^|dfeGhG~LjF9erdlSl-`>Q+l6dn_NmUs+| z`GzxZx5(#xLE3s8z>t%~-U6Ki_qZ^ONfUDhN77OOiSh`d zDT|r1h`>=^C!9oG#)TF0T?}luA%EjA#vLuU;wSG6xY3(G1zH@lia~VBQP*$@#}`0TK4^9;*{UcZ@!3OoD=ht=^9;+^p@&S@(luI%8#XYR%}xf6reNlaAQ7@WViOS37sJ$(uM6Op!UO-_Cdb&M+NKz!V5 zQRQ?t9ll|MP28Qz{tPET<~?U=Km43zvO6d5_7_5roBz&441z&F?pB&y*P-Z8dmL~c zBGrEW4J#bF#tlzwD%eI^x`$!NrUO_jfn`u<-?B}oQFGb^qA8rg_Wx>9LqG;@k?@6DcfTgkmUW9 z``#Y6%J`hsmW#&4iTLwngx94J^F!6XOPD=_Xj712B2aNW#ZTNIuWH`S&#JVoYwq-- z-EyP~wUcHq&!XvXrqhf`JJtVYz5nS_NlXIvt_L+&umSIV>8QXyVxT9w@wJ&l6aqyg z@eoSko!HTxh9}Y1+{(;jcV9i5$0nlC_beYQe7y6H*xZ0zac}yipdPu_KGBzROuQuY zr?Yl_4>ZzAHR@bgJ?di_5K$Ym{kF4NLV`LhL_M9OSPB5^ALVu1{)ze=BOZ})Vw2x9 zISf-#4g?i`>2YC=i5gqq*BTyAt%9>@b}?d-60-}-d;mEgJ#Q7hSCKg%4;h-*^)x?)(s0Gli}~*veHK*cILj=~K95;2zoZ+gQY`jN+=NV{;sFUJ)K zswn;K;R*l9j)~(kjbdopT)1i70<3uc?_wWIb$|==RET6P?=8iT$j&9Zu1uO}rQ6(S z>_&d3kz8rHA{Xh-gExMwFITK_*@`&ew)wK(+%=5djQ-qU!z|#Q9CoEJq_Bk!+oxbo zB1s{2Kb#A%6sFUsj+2zF5dB3kdaccSg?;TcZ4O-p9--sxq$rS{cvo_-1nRr!(zj98 zyd0e@M%>!ul38tn8Zx@xmfWl6V`=`$ZqthkR!&jGv|%W!L%aECoVTrp5EyK}Ha^Pd zyLPoDe&`X(y-ew?2m}B55KaT3P-4E2>xm6@(Uj z;gNeufHYU*k7KPBmEwP)`fk45_H+0-Q>8dO7XOJJ5$#aPrilPupRU3kj19b~35*}xa##4Xf!IgdG0Qj#2w8WW0LIU zS}((}0sv_6h_dqLHVTpHtEbPtmWQ%^plVB^_lYc|KK4FHN73g`f1pqPPDi3QiE17I zNwWy;z9`)rEFJW3VmMGn!5-{hWw$7Oq{6WH7CJHhNi4gJ0_m{J1S|#bEeLw$e z12UL1#9hgiSOL~(rQmr&`DfCL-#DdD^aOzdS(px$%VVqnlUDW5Hogu$94P-r)M2im z?3C3(#^&4b^b$#hV06s1VdY~6CAxAgF_vy(P!yCsPF>UqRl3adkr!1De2-ZQd3_}o z>rcR=vYy;OF@QJQi=s6X*yvPWcaiXZwukFJy)P=3EJt;7Wtk!O9VWj%mB=ZpxYKc` zqPV@&1wrS7H%FLAg$PcLGvrJ+3VoF48BOA=C0l0`--Y8O7s$0hoXA=3^B8T*t@itaPziTqp@o9(P}RurfsSYlGj-?`k52D9iZEK3cA)JP6|R zoBd~G*su3DcZ)hV(pzHU4P%m*)*saO3;Z@}6lJ_(^=oiLXYuejqDAW-dSo+O7ooZ` z2E|-PH($YVGvsyXe5ON-(#8}W&0%@YkE2#?W`Bmmp(9m*rgg&-^C;KmVhiJWCoaxS zw%v$=IvoI~vn$aN- zxEvx)c@5t4-?WZSA&rL1Jom!>wH4z}+zkjxg9y(bLjU@M&^kH;p@l4Rk?Y^Jju@1o z#1eb|y6Rt=RY>b70512`g_zKPd%0B5I_kcuHvI3d_Y4`*IvP>u%>H++BQZ!F09YtT z`JWH^pC|hNZs{Jja$MpkK62#DnRl%pMaF^-(egAuJX4Xu4 zu|nT;-#&e0pW0Qm6Q!ywgN%re2mt|sEGH|e4gmp$0s#T(4-W-=Vu1<64*Uh_rY<86 zQ9Dj}2%I1|%YJZ!fIz|o|3E@y<=_Gr%-Lw_y6Y+_@|!t1Fq@b=nOZP=J2(SZLqG_6 z^8>#+Sh$;zdOO%Vy77Apk^gfAKkz&FV-|AKe=c#i6C&4DQYDpeamPliMdp zcN1?WM>mRpzvREZN7BO0%+7|2*g4>s76t+?{}4 zaJ4a$b9A?G1#WgX0k=+={XcjA|F8JJze~l{#sYZjf9_`c&)xs~XaBigkOlnW|651= z+spra3UsqDq9DtE51BAx{Wkj=1cWGroTQkhH{{8hM}e=@67RW=4P*&H>KV%6nWO3@ zj2J#;+L>c`#x`f z-(I@kvc zJmo)!J0=u@mOJKS#VIHVeB=d+ien40DOIE(K!K07P~dzm)ZbP>j1)KuXh;)PrBoC( zj?9Ba!3Pf+aL!YhzpRWV3mgXs@RXWUm@}#TnWTxpF{uPTS2|djxfOYPn22HSI5@8B zt}Sz>y}I8{=ze>BYQJ3fyIZ@O5Xt`?o3QSCrKG6f`*hm6H1s&*Sd`~JE%&$U?&C|o z@5Anw24~NVB{JXp&4_%z+YgNC7~lpHg#!&Fg1t}7Uh(eCBR0~sq02N?(=jNBS`V?Yl3jrSVb=E75LVJWP^q;A3xg^R7k!z zH{w$KXnhd`N8I+XmwkNaeX-E}ihYRLiQoEjh$J9qEMPZH$22q58Uq+2PJ|wA|FeR1 z+o~Ez;6dNU`F~%GWlz37{SEld)cS6Ahq?1)tN7M4RZTk6Bpgo~Xw&-ZZL2=elIDev z3|DzRXARiEYoAu;S3S3F$3OMguXo-3dHPA<)Po=J@Y|?m)%CaE?b3e1+q3`6vC+EA z5K&oiuViugz~U6VzpP;_bM$NQ?C z^L|to^vkkb&*dZ;;`TC@PHiHeZ1&B|{I}XoXdHr!M*em`B$7z<_s5fC%bHol3v3_ge5SVZ z?6Ldpse8fD%kZ`e|FCu4-=`M_XEqUq#A8;?ZImrSnBi^FEP<(I(ez&LVcEdCFf5*T zD-!>vSm#E9l<&5jM4O1ohJ*L9#)rn6m!AxR@4yaz1DKHczLz6zM0%eOOTaUyeLwH% zY5ffcrXW0q<;Q(Efn$bUZM&d2HsYw@)o6QN-?no<@?Ag4EASW3we>=-%TK%*#zy$) z>1}0!(;Y2Z&$IUZ%JV>I6!?VpZX5&WUSRJ37<{j9;5tHeF=yzt7ABxJqR27EIj5;H z_?+P^`?3Kil3|`Kx6=ccqRd~BbABr$@~Ac(ar=|dP0?vvhmH*mqj>I z!1FeV01Mhv!_Z>tZn{3Ru(ndcvD?Ef&9r}UlJu`#pQe!n!OP(fR=IPT$^}@6#!-nL zk;1%|&lf{=&z0;nh-j{TiK}M25p-~%4ez#~MdyygQX~kM$!rHJ~ z(=h6e6k25Or^4~;Ryp{ciD2p~g?t`V$b%qapRFhwe_&LSHqFH(Iqhni-}j<$2IlFR zMo~>F2~J8y%i;H%v;t8{aK-*x94gyW3yHm0aj2_--_4wsDJmxOAsGtvw_pK0Sz>Hy z^ol273C&b&8aTsh?Y>KWo2&DOp%OK-o;Hi;`$?gOT>&-yw-1Shvpk3tn#WimvIWZ< zc)gQ=h%al(#Bs$`5J_*t6bZZwJs!4}JQ_zYt@?+-?SjdO$-mzTr1`@qaV$ zFo(a?{dlzfMOQX!UH&0|!6?A53hFjFlFE(-Dl7>37k1U?2)1m3hd60rG(Y5AyXm+) zy~Vz(HHOY>naE2NDX$}Ub%+LTnvymU{8s40r)63cRNv7ig!&#;lxZ8dNWn?+o+YI; z4ICBwCpt3=TtiO~!`)8Fkc7D{$q{i#h=)}M=i7g0Y{rdQn%glvngt1(tZa+0az2Wh z=WP3L>5xHEO|}YT3zDlLyZw0B>;Ho0D`0AjzdwAlV1%l%C#;z80Uqsz?|{vBLg

D&IYp#aYMgWCxQa%tf*Hgc5N|iG57Q z(3s~$pN~)##^nXBMmcKKyOQ~CW;Lv`QEZM^fAPINU$On*q}oZ}j^#kH)SHQAJkX;T zLEwXAGv|OwvH$l*qtKcQKaOEDWNTdyohQc0LIx9Ur6_wB(a*0#lA^We3YW zm3xTp43?mZRsXTDAJRI#CS_9@Xos5 z{GW}`ki*DQl*rDB#!z-i)0cd&#`+n4(IAaLI%tu~k2d=%!1TeV21(@0L(tHR5YB8$ z1R%(URAB2tB4J#Cm#YH0(eFyAuN**!4e5{tRR@h$t5->(7{dI)`5ODH{iN}?tcKg} zw}fFNNrY6(3Q(0GVOp1jeHdrZ1fyWp6k>7yZ+~4b9|UJZ4oryemP)58DrFgw{G@9tA4B;aB^MUW-zKk&PQ2B^d*(mCgGA)>^-#HBX z_vuQ-V=$NFwFU?HkE=1!Fpj?h3lS{=uK_~!rBry2;;jaA=ySQzk3Sfxn`YI_@Dpve zisbW{%uR{D2P|6OiIaP6hah>PX}#QKc!gUnG}t4Gcn=hy&M0`q4%JyH+(mu(5kn5L<-c ztk*^D7Ttm#SE?Bb03xdmSOo5g$Ft$Ps2-90gu+%S-@>vKi#+(o(+7$FS!E9i4n&}B z3c?a{{lfrPRTC!LS{IzU0l+EbA@_UMSC9Ec9a=BG+fSok>ZKei>#JN)BPG?t0%^XN z2vF7ZA@VMry77oiT8!flqQxpRNEYRrMYhGxy}yu3_35$lFJw}8_2fMVL2BBs(Lz8D ziCSWCz{!K&nG{1~A1pjqGba7|MHlrVPL27}U@dB=VIaVkDZWc;oY{-nqwH+{akzlB zAYNf+0AeCp;H18Pv`((61D4%tcn*&^cl=BtnFYo@44QO@j4ryouNjr9hMyLByr)i7 zEuBp%1VoPTDd^DLf`C5ssQOcRuf$xs|sZ4iBxN)tU$tur zk`pxljpT$RduXcQ_<({yzf`6(SDIh9J`rn4ys}b|-M54+ACwyu_XRqrwGC8EJDo5h zKY)pbIvDO2Bu&ZAZ@xMXqw{9U=0wL~ZzCRSFt(|d-xY>F6+%Yi%%sRCr!$NRU$^9k z*G%y>S+H8q3W3AW8`k!oIFA*nO~?~6UfDO?4UHq>b+|~qUdk85hVrarU_%y^-EE1u z$NEiI%bh2^fxgli#G#B-(BeojdO@pQdJ}FCNJiLet|S(Z|4Wx|9^))!iwLFuoVHaJ zmKU0Pby;ft{zJ6lUo45&-`6!#xRbF>Qbq5k-p^XUL}V56b9*~R>nfl0id)WgU*oZn z6cmfm?G;O+0I9?d#PqJrJ4o%^ALG&<)qer>JQ{S8&h`@*x>LjfzmqE0u+TRNC z2=y$=iH+93Bgj3@0Od6>5Gm{cM?P7wJjLdH3xz)c2CPNBr@-aB&LX;15HWc~T<#y3 zA3?k})`Xvg#oq9x=FAv$+?I%^Hsbwx=Qx9~Cwr?|Betg#ASNs8{dX|U$V;X+%8^-R z)OmLs!s9(`bRnOaI3V2PQ$7~0D#=r2wOXe33d15&Od~>G{=$^4)F2tf7sRsKx%|y? zSA$7X6rDLC{AAyQNd}kPqblw+Ut`4h9@E@ZT2jzrqb)9xg!WzTP?TQPHEke-&63hZ zeQNRFAV~7JWWJag@$nEPwJ_J>aYo|_R$tvQ8ZvZbXXPZTwD%AT{LA{VM2T@JrFqA4iFbf|?M z^Fw9(rGwF40qpQ&Mc4(VWLT&-){{1p@H=}YLI{lENy{Xw&oRZMAIPyKPc#Hf#3C=1 zf@bi7RIWo|LMt;z`1#QxC50q)#Wqr43VsREV|phMRtAnQu|J^e6k#`#n4({Q7%OGh zRZLV#Hqq<&Odx;BC{7a;OB(fyE`L(u8gY`CKPnH_!=hLJ?We+^IHA?=MWS#B$a3U7 zihysfjef(WViGE(!x+qYjD)sjn3d(o`Qu&91yLEQpB5buKgxU;dKX|^i(r-W7hc*2 zA!H`=%%Y>7>j7UTb`wNtb~X*D&mqb=P~Oy!4!nw`{o_d>ERz}WXfYX6tN~In z(3cv@>^-eMGwpZDek*l0P~?%leAdCNsm1&#s+JQvi^10f8yaR&?WD~mjyPd!bdPx3rxrH80)MX;8-pFT7wNEz;q9)8 z^h%Ijf*~l6A+alpMGW*8V)~gm|1Yi+#p`d6r-avSPmrEEP;Y#PnDhne;-?eiNqywu9_FN#s`2!vBjU^U#=$*(I%DKENE!<9?B;U3(09fk z#3`T4HPYImFoIpM&LHPvOL}Vdq#-+pA$f+FCB+Rw5Th)7k0IjzK%4ylT|!Bz%SFi` zLgUC5S)BH+ALO01E;T(CA}d-AVE!pepN)+hPfw-(X-+aI4gDg?U4?M-8}(8ppR^Kc zX;U3Nui5qc5@G`pH_nZKAwM;-s}us2W)h5=B8?`xm|8(-ZlGJ4*#clR=Xli6BkLzVoQA&{|33$K!#!ua6Wzsg9_!R1q?vr%^vd+nnpn zu+L3AN0b9^mhhNTYsdBH50Q@6*H{(g>Q5;sR|t_$o%h z2hk5H`F_Dz3x-hcQ#m#l9(Q?)OYD7cJjPfO_*CWjPHjSPq(Ol{mRDTObkn9=t|>Aw z$@qtQ6gwsKfs9N$rX=-myhhTMML8wJ*n$x8jdu|-t>Xp5RP}b^mi&&mIN=H{h-i`p z>am{K zk@4kPofzXSB=U$$NsBp_(u|%K_Xj6AA;X15FyxSZbNx;R=Kv|A=QJOS8WaQn#oDXj zl4?Swyhs-b;)6`^Koa|$=|*)$=}7#ff%j?9qpd=s$f9?P3-3!>s2SmwD^w;)x6~FH zb_3mJ2T214vN>;Ll=}t4c&e!Wk|6gdW%xjYn?kceNHJF_Uz3ojq-jFS!C!48X}D0r z6nZ>Ah0QTlQEF^(b8BH=xMNskKF-F}{~(!0zOsLzk~3ooKI>&SMmN3<35Ill4AaEM z4UVPwV@m4Z%is3J3^oZy6^S~K`dul40a7Z2g9L%INCovQ3f=~m653qqH)4;XO3`)D zEsP%;DeBTO!W2~p9DTHRupW}g=4a9rzL^@<5FMqIj9z}*aXslB>M7h^QR56{&szS# z#w2JXQ}^?&=?>7vj_{~(xU@w4;Jaf=!K5%1=9jL(e7E*PHl!fYLAc+?_WlCXte_E8 z{_!ciWrsFw4t*75$tG3tPn6uq{ogM8jei^Yzr5 zVr;R$Ys@o3kB;a)KziVoWk$4tJ)Et|@RVl4e=&t81`7Q^HW7;tuU=<-ae_IF@C((z zzPguPf(H~E77{e-jocF=?a7*6tQ)?+C<7A{o{_J#fop9M{6K7W~64qzsL%7 zLtjl<33Ru7?1DZ&`AdC-^XrC(5TlZI^|>^6eV$HEpu1*6DeaB{lw4b6WW^oF@Gs|X z0p`>!3KX%a{-Nxz89*472Md`>+w%W#?$&s~xiucGhSa|edO-{drGNCWoN2=Vfe$@c z;9L*(m!{gklzlgtWbw~k;;j9b@=q57&eMa~wNn43?CZdf`+q+k3lD0G-@FrtVqgTP zp7N=3q{;Y$$2@6ZF7NxLzF-Rr68&s$^M)V#`c|K870Qcr_W4hro34(HI^$(c#J0s4 zpVeXwC=u(dOP4A^C>WXzmtUADZj;d3mvR!{59vd6(&YP2f0d09ztv7hpHT%FJRmVO zCF*(`xH623hDil?Y5P_*gJb({x@sA(7a6ao1*Yr3@%wp!r(dmoT$XO?9sb~a zN~9W5ll{~;CCSqtnnAtTF6kt6O3GIDUJTo`G`#8jyW-bj?mFxMz%iLA%X2~m@$F@pwXS)76CeE=iti6d8VLKu(`2jH9sT}&@x<{83>P%%?aB?!&m7$Ojs1T z9HuA)Z9dGZ{AgP9T#@1w0SH-{apQ`T^yvd2?Nd3rIr%m*4v=#iDniYz@bEb@Eh6C z#7#n`-AAX=QNJ~pxFdt-Jv1+jUz(?XMh39#3<@OWcoD+Z{hv+`%kn?=wRa!ac3O&% zSmn5EQOt!=c;tQB12hhwdYIHdlMn{tNoccE4aK1p{&iiq(R|JrCtC+!YEr^LpUdIzX zrz3D|5$Ks3OEwDQ6gAz-RyaD-#QDhaGnamvH zHPPC{fVQUr=8w8^z4@S4eMW}&R#XFuJ4UvBBSLNPUJf0TU5v={Ch9DKHxyh)_sNnm z`)&g~$U=d!L9%@HFkmm!N*FK}JB7rFR zq?0Ou_ZDL1Um8uVMFu+Aw!bXWX=Z$4On&)+FVLKXs1BHE0+obh6C0<1Oy@En@FnVg zYOUJr5IZ%39zMP=BK8aSn!84D!TE4jyk>;O8fA*&)($c~DBSjHT%Zn$S;bgH5JP{H zQuRU?Q646C`1xug!KMJcklI9E<`*c*8l@`eVG{FJlTv{m59lX6c{zJPmF6U8)<1Bt z7J9Wfp;YP;3mJpuVdp^=x}Gfm8jl|{2ztH%)JzS_ft-vsKm-gT2co7Bcq<~N#V%J| zenBBqKHP zdOt*u+NNoFUSj%-#Z7ztX}u}a0ZdofrXqI+ecZ~PA<(Gz(}kgd)ZLlgK%ce_$w}oZ62u|FEa_ znV_i*_AP}9x9#H81`hb*-&cCWxSD>;o^u_d-YC2OBtMG)4EYgAQ!TnAH$NL8LID`` z40Do9)`z5((9v%b!Ye@xr}xzGrhM{vJqBE|`N2e32-vTr`UtNftTbNI`zP8B@U_Bd zq9#bs9+~k!vg9yyNUj%`mRD!y%P-GCLfpuRDNCvw8zHFF%9Le@)b&esvQ^Fq!q+_6 zDpbTG2@in2O&5nWCO;|{MGbZbq(Gcp@7wHq;cvan5p!2A07h@&$GQXi6<#G(ms06P3es4U+(QiJ)BG1L6>h+U(xJFHK)Sm2gv)mK=?gJ zE*>nOP9Ptmq zG(B#j(Dp?e9Is}A7@uq+8(6r92k~#}Tlyl+1>PYgN3|z?8B;v`ukN`7ch4uN&&8i! zzZgt=exJtWP(f^E5MS>?w1uuUM<|fJii|xX)aJ}m|3mjzALrP*=38h$E&cc4Cof7p zOeIb=?=a|d3YMas$jrouCkexP%@UKYr|;(e8&W8J1kby#|6Dpe~byV}ckl4imXZmZg5jTXq) zQ%6Qzd_al$l{=A_{9w6@D4#BeWmfi+q3DJAX8S952B8vDU`Sd0}t zqG$)_!Fc5ahCq5|dQNA3=Qwdodiu=Bxtw%%KJIk4C5Kc>6hoG^K3ne@j8o+U4IjgIG`4|31!vG5+nv0}@=U;dX19r5)qwCQ1sXHfoB-O#@Q~>=YvAU{7i#R{Vy=^E2d|^*7^vV%x|kw z&-g*f4JR8Kla4+^_NdV|Ps25wGDys0mw&c@YPa-z3O^?4!}C}UT1hK+R2*Z}bq0}} zUh<|g8HBn=IFPJ#c}KSx8w_mP^lP26I_s3Sd%M=?Wkj<+o$176ZZ0EF>@+L9H)%eQ z0QyEu+Hfh(UB459nr^=vjg)+YR5nyW=kzr@TMmIKa1L$e=Qxe?hBj>uI%@<+Flv;y zw7IbS$d51x`XHL}df0dDp}f1;wg7%ZHIo)PXB1ckd=7ezPA0S2&;YpS*{{*)+Jhns zZA32#3@P|!3GCyca+z)eDZ?FO8@mY0JzlUG#uQlICk;b_^UNPCao@Y003+k}F?yPT z*e4R{I$R=NUK0rFYI_x|FT>%epDqS*+Np3z69jf9#d?*(Mll&y%ubWT@Kle6yU7w~ zmu=`cQG!;-aZ;&gwLVF6{*iuc9lC(uFyi1~C)ZqSu_>^js!omT@pds^x6L;ovQL+@cPR}CFEZp87?BQ zui_#$sW1JTqpE{*R&`|`eD{BH*dA3Nj1Q_7r2YEW$Ce7VNTs?i&lLRoG~nr>^x;r7 zn%w_p(5Yy$t<4JHr~83|HUx?jU{k}@CBctu`v7F!&Kl|)D9yj}wSpotI<(=zpGVq6 z=E@3)1d~&NTOSP_R9r>D4Q}x#W#2DxxtUA;4cG+Jcy(XGl7&ik| zv~1qflUHcvUB;IsNLM*+(S3?iN3#EB;A&)vZP7*JiVTlb!56U{<49YEF#c69-nM~lADNDC70j1rF_H%$rP^Miq0 zReijn0@ZfEe!e<(a*-|*i65-J+h1|&o=ZWO#xS<#%L8B}%?`9{vw%uK)fiY0Q<;E`qY~Po zgc>#s7{HL=avCH;0OEa(Wsm-P-U~;AVmQCoH4Xxd zU$uTU#aYMTpzQ%wX%+nf_)Zo-AJ+t&hmi0_O9L5Vg2-zduVL%e&96`HK*s9Tr2{}} zxdRo!HnqEq*jtZxJb=S$W~Z8xI?sEHrTe+jFK%ArA2%N0?GUNPV||#(f7P7OIc0VO z-oUK=+7`grPG0&IuVUb!b5U>*F<|uexB%pJn?QE?`gmLyX9c9i1i(HP&B6LIg_mmp zbY=dhz05PulMJ)?kwQeokp29CAAViT<-t>uVV2H0;FAxB*OZWgiWldtm?;#PziA(4iBIo;t}^+aJ-(+{U1FR(jlO%rEYB)Y^ds%k;ZAlN~XF_+Y zSHMJi0c;1*+b!P}CmRg;up1p8}I*yahF6q6o1u3~Vfw z+;NCd7=zV)n#>|UCE5jN6}V=xKmc=NWv_jhn*1pIc<|K?U{GdMMB=>HfsKqTN`a-z zJ({kR^a@W;%7NAo;93sk(wl>OmM4`dv@D~D*S96h@;)DVl3q&l>-0In&RG^i34tIx z?Eu@9<){Rr(_239jbTU6fN{_%21bBZ5DgRO@<9AX)oNVi%@;8b2+yq#@tN`9$+I2D z^HI#+q$)m=XaI0@#r75OlBIUuF56YTKlY_IG2~6Gu@X7>uDa)&3n%NPuV^4;lmw?D z{IF96EI44xVZ|2<8xwhZ4#OvYBr+2_;pl(ILFjiq1)BUok{xH z2rdEsxH}+Qe!eafsZkT|vSk5v#8jvs*M#*1BQXy@)6G5r$#F}|<({3Jqqua7!Q!m9yqFNL^sYNwaEzIGL%cFbo@fZ^~u z*Pg#w3jx?4iZH0VvVfVrFv>ZjxPvfA0%5y`91fVnx;6}jE(S1ft^kUzvOAwQxuP@+ zkwC57%&XuPELd|sKzA`IznZ5g{q(QqeW5c^0p>`G9L>sRM}%4UN4yCXfUFxL_QC3I zO`R5by-3KUN3PWFjZ92wONFq)2{8D4P`L(PspjqXT_u-G4y|j^>l4CfCevC?IzJW~ zmM4Iz3#2K|0L^%;8>7T6ne7VX-&;Xh)n#k>n5ikr-S=BDU~*C)9-3#El*CLFnd@ee_EesV}d|> zTeR}Vwr6W^*6^;?wNod7t;36A?CKXshg^rsxF8ltcrGraalluNvj>k15M>VWtkd1# zeh?a%K70myb~QY;D;r@ zb=nYyxgx-sN!)?{LO-7kkdOJq8ASUPz~UGtix9;gu&y=1srnLfvj@PVnFqkSEV$q4 zT>rP@ArE~3Fv|9q^Y|ov%O;}#uo6{u@PGi&BQHFFyEm^NbK`6oCqz>(oj41+75?E3 z7htA%l{T#>y3T-q9G&n4wk9BA)JB|#Vw;9>Fok6MST|p5=wMbJoG!U<#&;yV6%iA` z;-#vJP=4>4egh^~>boOgj!LIt=58~6F4=F$A4vP|)iy`s_L-^|v}F4??a%E85%B#8 zYe4rbxsbd1xZ?t2*ExQNSRsorfyyAFHkyOBmmqYD>!b`ARil5Kc0mO|<$R6jm&f}0 z@zA*?UoxPyAd}&GX<7X<;CU}MmEEA%Hj!oo_MY`R|C=gP54*^&Lxf%K512k<07vuy zcn@eg5QOpCEe3VZm&b(ALzc0bxQ`TH|hB9}QynXOB;m^!stwf@&2ZtQQTnb}}!uaSjDzLU* zSqT|S-$ldD!Psl_Jl~)k&7?wWb~rW^*u)wEwo<5C`n>9c9WdU2x~+L-jv8(pz)-+d zc*!qLt_c#Ya#5+Q!$dRY+8H;b(L2&4U~UADMy{O77$bw?iwZi74v7RYhl2i##M7nLiQ}VndZ)N zv}+CLJ2q!xd_1I?($2MST5|v!oyomAl727Pbgtl+`{!v#_0?km-rkBn8s$J_X<0)qd87a|9rEBJ9*k=4LA)@g z&tE$|?$l$a!GtpqTg#tf-s4ctU=ByZNr;zGUxen7`H&&d;2U8X@|>*#WeC^BPUoQW z(oS!tk1rmYU)rMi12v1u`vfO#@}2t-q>eQ09$kS}U7Fkh*uJ&2qr(opzE=ynluJBm zMrHQJ&=4k5xu$tu&)kl@-g6MP*8%JZ&k0J;4)M5Y!$D+w-j-kLDyns}?}DhAzh-m6 zt_d=;vL(w0`%_QN~Bcktnqnb+3g3ro= z7m{~yE*yY+?$DQbmhHWRA-xcC{=C-@9cR6fr%yn+K)u%qDi3jU*=1?Aqw8k7XXZ~p z?!`?op$3TW`|-&et*3|6*7Y+WqS9`O<%Vg#d> zY6#R(aR2c(TMiDsU;k)Rl12=W96;>Mrb1m)w6l88L6!*HUpB9;-?Qu;uEJhEKB~~6 zGlj>?^Cw!AIW@)nc$Ar-uB7Y-*n4jf?B?|uC|0p)+@8$?(i-$a%mu}aG-^A$hkd~S zR?9A+bl{3>pZ+xOv^bDWAN9&O4OXUcy*)-UX9UpcA~zB6^g z)KSCAF>EHWKOe+AqxmbO74abxtYjuLaiZP#T0k);=td&y?GNk(w>Dlv?Xfz!Ifd7w z!`W;g-<<;_Uav@R67s^N8wofS9Prxf+W-7VpNP2jT}R!YtmH*~e>uKJ>Igibg2K#F z4nuOnk=Xf?t=JAS2dceMG{FympRUgf`t|rO1LzC7X2{TGmAm~05DA61`2-AH;ChtW zvUfKtf0$Tki6JQ>1|}W;#uwcSso0)wjiZye*+V|fZO;@qTg^oeP>Z>Sqv+uZ#UGHJ z;oL!G@>xHSDoZbJ0aD&JF~#1UT`(VJX)jaCl2txYiW;9{ z;m46&`5(sK2A1TjJI!PAi={bl;svC%3?3AsfiN+hzfc9wMWJLvAtsQ!pVs~g`U8nU z)9Ob5lfAEg9;kuRViZ7Bc^)wekSJf8Ry$2u#I*>^ENdNZw-N=G_6M|{aqQlc4bB~` zx=oDLNTKsM;zNs`{HkZG1M7Yq-hxx!Yg$aM>V^0 zb)^W=S?X_?IXIJQkDlFpu1jvP53)=g%a2*oMdUw)@~B!d`~)PA8RA8`K*f{r%ITSV zyI-vU01b7r5Oh-UMbJ-gOo!O&4FI21f62x%);_9hpg zGUbnQ2ZgwzWuJqF+fuy%a?9Db>3Uk>TPZdH+h$aov3k!bd~`YuhR&30BDd})fH?So z^wF8%i>H4e=@2oYI~cdlE)QF6@(0RhjKXfNc;y=cScjRowq@CNL*w#^$Hj4kW=$6> zEp%caBNKxOYzxtY`uSe75lFrf_0!KKA*h77Im&=)kz=rN0MVyq$&w;)>!=JS@GOQt zCn~r*I(7g$+3vF`-pycr77++lZ?&V?SZ1uR;pi@Rwy>-PHW-437|5T4tgWH3X;p-0 zygPFT5JZxQxbiD@n3i(b%cN{gao>}QhC!D#HZXIDZY{RQVhw<`v?|iX+Kpz!WTm-p zfQdI3sH{}dn!+J*8xN`@57=*@;2}+K98iTel=zK(_E&}yrZ0Hrl2`FN&Q1rQL$kKT zXO<|5VxvjhM68zv(LicyeXRPLnPYm%Gfvs`wdQ>_TkXZXf!i3Ms13o`6Klx;6p#U& zoSri_?SX+m0tw%DQ(XBotRZFHJD`LLNL*z`s{umxCc|i460vP9{gV?WqWs)E0;IT= z^hNCMG2{A|L2GM6=Izs3Zne=6`(&m9dhelaspcbjXO^L{Q$U0+Jz7aNkB;O5ijd}S zv$QZ&o18NMlqzn_tv>g#pFiQ{PGNLA$P+KfGYq9>FcG-g*9qv~M|4Yd7H#G@9N&K) z0Q(o2wx}RxXOP-g-GJ^bkW|m`_d|ctI6M|RgS7JWBEIf(^U(HFn<=M;URU9MKG;EoX zAZ3G7!tIsirU6On{XU^=H;_8n%1U@H1w*R^L|khHU{O~U=NlvFO*mtDeocvk&#3U>$(rBB%T@h2j#^MC z*XA7=yzR8cHIQ2iz%8p0f4Pa5zs2dkhxw*cREB4nQP|KsrPWp?+cUkmFZ4JEWawjr z>FYGg`Qe-(;-+#QX*>(8lSp1QfMqB{>Wp*)QXwPXi$Niw=q-0)fF907de_Jb4VWB^ zyt4j!SC#AG$ncKswnvg_^m~gBMQyBg6nsCuC+%3;Fo__t+>jW={Col%4o_n4_wJM! zC|`7pFZ|}eNyZLm=gNuW`e^r#mTYr zACo(y^%c825?V_0GYt7|)m~V%1p*RX$4^f?0DcZRUm*`4WP@IyDW%O3Ef2* z%aDuV{RQ}$RkvIg(}>P;8ajs=_jW=2wB9(lm0Pm}q4d=dn%Vcu?VaCFCn<;NM=6^oZ*gb*s~rRaE=L-;(FjLpBbmHbX{ZU z;mZB4%B6*>aafvj{G&QRW9`rkQyo`6#&rW!YBo@vnc=}8=qRr%oKD?ACP~ia0%ie2 zqkW31K#mnN!y|;yd6nE9vZeN=bxU8Mewt{XFe3`6W)7OO6ev|0)pTK8R&$#=D$eHx z95wnY4C!i?5fi=tAhYu85Aj&_Wn`8*BZ|jHZq=7D189stGb=q zW%B;c-T9F~H)bC-TuyY5eis8+Nc;g9QF`yN&gs!*xe1jC?cPP;HyLD!9V_%x!lScO z@lKMBLh#U2W_eoJaw|~+JhEAc=rkUjh$pyR8E-N)Z zpt87`q5zYLXT7O3_5;3}$pYL;DGFov^W{O*^dCJZD$OT|wVPjPZ5L=1j{u2zn1+H{ zk?(%PJkNIQis>Uu!flXY2xj}Hh9n!R+!%!k!c}rf#9KqvR=XEI{Me6H%OC<^k(M^?itrR~~IsTp?R zLxMmta_8dW7kbWGCHM<|h@Y>H&hG<+K1ig!%Y<8C#@b4+ z@NhH5KvjyMo`dPuf61rn(O(Lrb|b>dwOJP zY~qyy05TGdGLc^H+m*;!MWoePJaH%cyakb6D>Sm2ol6cCwLIMN(LVKKwhjI8L9iy# zh<4&D=Q?uUEAPIpL-Sl4uI-N&OIzo>4%$gYWSHl47Ab$Dj%l{p)=qfe?HO;c66jOw zR2fX$XU|!KJpPp0B1|_W$!*DFNrh9VkT)tzKF92DV@?5fA+3Q`Xm@cVJsVO!#UQ}k z`fY5!@As($QR_xC;9~(1%+$_6N}PU*yG&&ZE#X2e)@de71Y)ff9=M|4@i)L!wYId* zzQPX7xeR`(y#XK%MYVN6NJuN|AeP1f#??b_%&j;AZ-Qt750+_)(uo&2{V` zpIvLm)Uu~W{a4G)4+JU2tz!~_7CbZed&h^4$oA?ZpUUC*LJu5O==F!z3xXN0eAptC zV|43Cg}Q-?QjiRWW(koOgn*#*0CKM$$-Cly=a?}nNKKHGKE|`iwy)0`EV_*iR?fU1 zP-*M{so|DZG%!*iTa;B*`ecN8-?nuQKpXdNhPOjVE&`RN*yJbD%K~0K?aKh^^jB15 zC_^*X?A`Fsn?4$3@7Rk*kxxG!IHd{u1HL7|l7g&!L6>rc%l!3N6WK}P_fFbf^50WE!c@Xjb z{b#hkk3;v7hJYOlak0PBa2c2&$^au=Sr?su__zYpcMd?BCUwfQ?*;dv2HZ>dvs*{S ztj+~Jrk@nR29It$yurB?N4yt<&^p`e<(Lqh`Cp}dbySqy8!gSyjYxNcgfvJa-9w{* zfV4 zX7%skRE&nUR~ql}0VpmYLcR~MEiO4afBu?>^Y1C0pw#24Cu8Z$JqFAEh4OI-UE^y1QomNwA2FbcGREsT}u3wYV8fM9BKpU&RU|57D$pZ`9-j zXZ|_hl-M&&~Q4?V{IEuEeppgEd#K&IqWNgobirQb&G@2fRss>Hdx-g1<8+b*_t@&oqi`ok<0N~7_fV|f5i8f51L zmv}{;=i-y=G^+HuVfVbM2(pOw(hi8sIA4D;HpD`Fipdb@;g-l#OPOmoHYH7c$jY9* zbjw_~ITqt8ggh7#9xE%ao{2(6^E&ay1=KGKD5f%VkWKO0mzAlyZ+|N3oO5Cdmq>|i z|2VZTp1OLk3>z@1=?Ge zDJ8cnDQDxB^qM>X^v|X2N#N5P0GA-^TO8QO1rg3$k5SW&c7_xR)#krLyKR(C<1_y9 zo+hD^vOY4XwH|l#Td*p9HElM>B;yMZ5vvDR?Pl|TpqyMT;nKr6=R0U8SR(ZXeH`uzKF{I}lxCC=jcq(5JY$cA~wNE;fxhj=2$p3B$W zlM1WjUpaX&m_-gk7jVu2H7pcbl`sgoEyGJHAeNQ2Fw>}T8QwB|D|oa#x>A2vnJ7zu zH~5_|MbDT#hpa78+on0z% zlUA5bsVN{#`ab89E4K>OM#9xk`z~n>Vx^{zv-pu{WKV8T2U2U*P9npYTBInD!Wt%S z1yF(hodUf!cTm|OGK)UW#Th^y_bE&K94-U?ECl#b3YuCL@7`BV0CN54+Mvr3YM_iWQuXvFv8RdSRR?CD7fX}Z z{)#V|D_joe7gCfWi}IpKrip&QZn!WBX*R_?^+b=RY|1B~*RM1`2PN&}93$l4?>UU< z8Tx7%9$V^>pg|iQ0}IPx?1y_S|Ag)31kCwD#wNG5_5X0^mW1*p zUz&uPqwl6U1D3A?y9J5mz^MY&6(^EGvusTpAe_dCsA95rSOA~|H6BLo0ro{WhZs*f z<3y}t!fM*Tm1j7#hu>K2(11Pr6JR)AFp2*jo%XqE0E+GRpr*JEM$H4LvJSOo!&k-- zaeGYzS-i;i)>oj&m{OJua?a0y3oHJuv_LTH2mJ-O$EAQq`mkouNhfRSh6<;)H8>3x zrDicgUvQyV4vqmzdhiGopgep|BleNx)d-&0>~vRhAhvMW_&p$9q&K+dIdt`h!4=7T zfA=4$|8y9T`jNM80iZ6)|A!3@abl3@9+V?KjjB6lOEOm+2P_Rx zID;t`V3d&8S*|GrqF!#V=wvd|f$2>p%8tsjy9+H1*$>8U`n%StSHMH5*Dcd&j0O5( z6I5IneS?W~Aa7*twd8YK1*C+hDypxY%vKIdB zTE~DLIuw^-hrW`?zj{z6$?^M-_M__6MX<+w9vp_bH(h)J&K*Ggu@T4tb0^A$Z#4(m z3(;#qVhl9VK~TaV;@zQNu>h!Hyh7R;cq^P|7#zA%6A?7YJQZ)DO6A`?sxl@6&AsD> zHP%|C-puR!D!{Bp$L#SPok~txNu+ zuY16#@OmKSBZkEWZeqyFxK5t{xyIOS)dK8u)5H+#@)!VKO@&8!^)dua0M1IV48~_E zcute{mLvjz<5N}#R4srN8YT>Q(zKxYm(89(*XM4*7QGKz9SvWSU85La1s=JlW#<&n zzlML*2%h-{W1)Xu2F4ljWfhUwG1maT2-qP|-35y|mj7~IgcaYkyEPy=ZGs@NU(xKj zQe9uM8>rNPgiH+v+m5^hsrMI9ibMOrR7*t}Uge0dY+XkPONQjkVX>%#5}lMve^?3x zgBERau}$viiB5v)5>8xR3Oiap$0%2*@J(RK6EvTbP*W79cC!HsUq(HjldWXk7l$)U z6q+kUopw8*yx|qiw>0tvGwBn+e(UHEyP$2lq)sPO&}Zd#q;|TuD{EuJ_Px^Sj2Wb* zAx7QR#in0OG#_3e&VVTDo*&q`bc5)^G=5v~*ZLdre8V2#z!HGOQ+Q{g_{cM$&BdDW zYAl&lOYJe^a7A0_9f@9tA>CJi*1V>8P0E^Xp@qem7J=Jg zi%arJ&Z$?y{>yMmC%Ra0D#6$@fI7a`Qn=)>=>Ytlp2kx>Y;A!WZp?R3&7en21F;r| zXaaL4APLb_O@Q)JjA~P<(F%H+B{{tCtPFuYU=1BC1M-&@plo&nf>Yt+*X7UcpB7g- z>3s`Tvw7h7$~CZd{jRa&7~bG(zNsI8PoKzY@1!)Tkol}YdLBgYj>PK{gnEg$QR>AT ztbW!D&FvV4Ia?9~==A-0`vlK;l)xu%tfIogWA= zmQ_K|fXAiPG&!5VAi`Lo zFC=4#H<|8hhRS{!i!hd1A0DC9p%|bS?RC?p1gWxOy+Et%&qCX$0>!>ww@$z@aL0&| zW<45(u6GZE)msIw5ry>-4aH91Ghj=B=fDmXjA5&^@d@HM#-CRh7f7~|5^69M1#5<& zEuGAreUyvLPx*>~q#vj1F;DzT(d*Xp-U&+$NcCJ_t=lhOi|0$b@>TYvLQ2 znlH#aWu~;DJcW>Ll* zx1AaVc_+^kSH#p$vgP$d>;mbAS*5jD>uay^vhFiNOFh~4U|;*}Z04_A{&fQgrCe_{ z0#2?RaFsHF{hK7a=_;<0;|+hQUH~Cmi)FnT8f9678Cs$JfLVCRvYZ+YZ)G$H!Zkfqi*RLjDcxjfxQeV)Y@g`Vw*)JZYSf zdbjENq2ohl5F)Lv5vO#p;SyrsLbT#9u6{$GpLohU| zxU@aMv;$$>_r-*7Pe_r{rHH_SPO)4Y?~l73hR9Awn6GLgvSOAy>vT0L#N48h2REEw zSrYg#JEn#-)v+Cb!Fn!|0t;KwvKb^BL0pnStN@}PHJk<(ZEhGc$NlD(7?)Rs^F4wt zJk+uOyyxNf*+!h9>1ojEns;r`EhvJ$FNo$J8dx z$wR*%@bLQD7)Zd$>DdhU)iD{{;ox&AM@-pSgsniw=){?A^2nm~Zj@6b4M+Tz>y6*) zL_i_f)A%Jj{tL@!S-?gTYs$Zm7-1DgYYIwpdFIW~RtOOlnHSARNKYQZ)F~=jLCC2a z@nhGgvBn5rXEpbXJcVL{8T0|fhpZq5qCH?S*m|75*8o3`?+lA@REEjze{*%)d*p-2 zPiR9b*pgq7G;my;ehBawDGt%z0EVD>!h{_l2l*F{SQwZbpm96a|CW{iV4hlq?8@M? zKcYsU!H$>i+!OmSY&VY>&XCE!YKE-vYZw;JNyS!yXqNo9Bno*TjB+W87TL@Zgh%D8 z9y|!Rb;OJl;kmtleS#9dEyVgd-r*L+S=)c&=K{8dE6W2vNX*W_EXuSg5W8#?d_XVb z>p^ui{|1dZQ*U^M{_kGz$Z2$qUOAceNSnr!A_7jx?|=XiYOZB_0Fyd_Lezj6ewqnj zA`EF#vj#JjxA$vbf)K}PqRf$9;t>K=f`+>!F&vDi_tYH!b)D+Ju8S^+0KPTSqJDs@ zcM1TbKzF*o;yHHxG$}OOC|TSY?YNHG&dmFO2zf0l)Zq&fDX(6_vrD}kBjoHkn>McKJob4R)~pv}HlVzvAWK*o<>-ORzJ2;6^xyJI{oaG!+r zG#FPj^98uyzJmzg?X$3)Eb$J0sH7;=)gDhGTsAQ>MLA!Ws74ed5S#}VO&fWtD!ppZ zqc`@JpjQoYht~c@aDr7F3b3yPgQs10Cz~G&Da8!XW?>awH$WA(G&RMd0M!~owBy{= z-Y+^t) z_6KmJqrq(OuNO;~r{Q{~fc7Wq<=r>k1xm21-(r{io!?FWo8JoNGEX#=XudCIJp6hs zIZ|`?d3ZFs9F)d=K(gUjFvcPDh@INz(HR%2C`!mQOADSS;1H%hCkgN;s3-x~9-!-3 z0Tl1ADHuZ`Jhf8QvW+M=1|RF&ZcmvdID?-r>LOZ@>bm7xry|0a+xQ^PU|hDAD7@V_ zV3;-mv^Nsv)*P)`!-MO%-h;KHYh|s@0O6Ylh~R(B{neHX#;mX> z-f-|aS`@vBg1XTFIcT5wz&cnWakX|~Uc9EF4LLyF=K zEb?PD_}eH2SlyVUW<4hgk2>Js+4WLQF@m@S2v?QY+i6B^QQ{OA{oZ}^{XhMh{-yomgcCLav$B;G(L;XU&GB+?Ks73Y zeR*?sv50A1;~_&*T>BGlx>KOG z!iFvIgc^ghmAYgX7@snGp}ceSPTU!-a+3jy+)Pn9+&enkM4S+&`9fs1KIG~HK|o8tx?HM869rw#P1J@#8%s3+c! zLakqtxHCNcHepRQV}{xqmR0*v%@J+C1(bFWT=w_uYx_{7Xv*wC_W0IWMCT}``zKJ1xKCI=o9#}c zY&?I@HE|{RWR}3x{#|@FvtYTrxy zp7;Tc&e+_bxh-XHI4a%#c9bE68>#{CBspnY(@<|SX{AW2Zg&Sb!)n7GfNB`J-`Bv$ zBeP1FT4O;ie(8IZyM0z!#W zH!Nfi!+$p>DKbj{mvInYUcnZR2hVQIMAsh08kHj@(jrq0VgL~5Y0GqunN+kTAbVG? z{k|u&WTM#;2;-k6z+%skk%Ke98wtbKmw+16lFtBDLL!I?+`MlyIgCH)E7at{&6HQEAWefKg-g-T7eCwhu+bH!ak11&7*xNVV zntO%-Dzc?EV22_DXbP;4>jtnzGJ$3X=B+C+CHCAw=fqnwWR*|*E#N8e#=QinNdNBk zSGNu9Id_kGIBC|q{L4u$5uEeCH)eH$|BdO^z?&!EwiMxOE70M| zRty8!B<8ipO8=9N?9<2JUsyP!r&iDY_93LL)!SWx9ih4Y z*Q?V$U!E<2#f*#?u7%3M3d$L>6v43ebOJPuujyz2EK|-#vit1CAgx%#g>dHCd^ikj z0qb=lZ}K2MoSwAbEWJDGohZO~C6mx$syISy1Q5auJ+>710VQ;}EW-ZhRznZb4g~6< z#oUCS|Fm-WNIbCo1TiW}`LBJAJZ>q>pr%}sVgG-hFsJ6qHOSD{E7sd$1ewQUijCeG06^?hIsdAX*=}~Nzp>rbKE6tz4G~g#L4?lY3;ozvp1=`fN2LAd zB4Drm1n`X;h;9fC8-Lo5?AVX;k$>_8ndQMj!*T^12$z9v0sb4;cv<4JgWY%0C+|T3 zdqzp8&ig|dUE~aaJusF}gvk2LSgtQ0y;IJ!i_GD6$xt0%|LPfn_Fn+seO=OR6UqU{ zeZ-~^@#hdtl4KTl$%eexs(2(4_g#%mL7Or+%2NS=fPVR@^i%H@;*~Jc?iyFzGfGQ` zbm3@L@_n|-S_FI_07yvZ#$Necd}Q)XF98oC6@jBNa{~Z$-vBrerYXOr|KRwN5Vat| z?jA^`_PTA~fKBkKNAR!(K!hON)CVcyXL5_1Wh-eU4I1>t1WmPWBoFLL5kLTv49>AeH(sfBbjCH!KJ4bQs zqo3TZ0phdhuJSH^pFi_C*zNr8zpLB%Qw$y2@~Wc+CxAN8N#0gkFgn|iEK82a$Vie1 z2uDIpM&1&H$Ra}JwY##Y&;kAEkzS~EaoQJ9MGYde6Q39q{2W!l>hpL9lSFEw5^VZk zn%5lyLy3)DG)4v{ebz<10(=48-(fBX0LiOE5tcCC=pGF4KnREYyzSV!bD$1SS-T*P z{QQVj@*DbaBY@|QOq=SiR_@f z-+c$1GA1tpg?CJ#X_TlyMEMAKb`Lu;DaozyvyX*H;! z*w{pHZ~(1^N!aXv9!%T0%%kL*KOcl1gW=#Poyw?(x{j&{ z08a^k(OHSUKXd&y$wAmB)lK9Co-_xnrld&cKL?fp{KN1#5>P;N+Kh~aKT;3RvotFX z%4P4IVAPHe9*vu`w#P&>s$$&$a?K+xWo}uA0-YGty6F$*VV(9~u^=x9T8Ka+lx4#< z+@>c-7xXNp^;Yc_18POy*AYo37EMkjESb-}i9hgDO1nGOzKw ziYkrJs=#5Yy0`(?I#EK1y5{mj9Gy(5~+;|6|&G7-uPg zECl^hLXOLPzLraM2Tpp3IXXF2JRmQ3Z03|>XYcsBys0nk z?Hz(lemcAhVK-3DWlbf=IoW6@tFElrk=_4U?)g|S%-y#oMwVg6(Eb&7xZ9lfl>yaE zDtp0RP|q0=hp;wYy#B;Aql;gHPKSXhK_>GgyQA3}i}DBD!rPhVAwQyC0zpz*Dr$rL z)gGXLd2cb@JEMc$KujyC$(R)6?W-dER;yBok&FC#s$2vMxfh!!fs}WG3-P_)Ef;;N zD6aCXeA%GcRF_5eb2UqA8i=~oCtAs{Dk<)!!&Fk*l7t&kQ5ebK=0}hC4y+0p8fv+y z)ugw|Grv~57B3P*x)N|ZPkTtuJu9T`fAp)Nd(=DY5NUXBB5gOK<*x8G(11%K!sq`e z|AvnjE7ay#xuHNYk3U7R&If3Bj2uNh+#BzjI~xm7Da$%ACDx8M;TA&A3f5yc3$v-5l^Fq2kpw0n`e+(}NypE-{Zm{#zg)+D!x)mzl|tU(cz z!fzx^+TJ8_sIR$QETsWJwcMk>axu@IIIOo=ikL z*e|L6yF8oRF4PPf3-?>{#<(C`662abQ$1dyTEkH2rI5`#AkXJ=LzA-*6q>LgCS`=A zzZ2rj&`;B0XKr_@!M)k7{lYU)pq2t;OV{YW*2PX2K`lHQl+Yq&Q%lpPuoW|Ve9Ihe z0qqZy6Ge-Dzad9RXD*=3Ht>S4`g-r#kI%yiHCUNQ*dnE4VQOif3D)nN+HI@WRIdu| zn<2P%CFM6*-15R2zRtq zRGO;92z&?OwI=EGEH)sxF{ObT%TBO;D`Bd9y}_#??PkQXDT?Iz-rgJVzU=}IMD(2m zaH9X%9x#?WR;@+TBeeSSzgRR?|AV(OD?e}F9z`Jd-X;XZciCtZH}WsDz^N<}3JV#F z!P-?Sj4H*^RX9;+6^catp$_TgDyiDuI&z z=L~To`g!ZBWk#?e9yzajrDx^xK}H*8ScN0z?>N7TZwYcU+Fif2R#PsOjepd?!1{`u z{IV$1lX_;}Ht+g zKctRCeqPk?SZ*QA>*E3{A9Sd_7D>hSu|rfzJw%&X|7Q707Z zn~m({(g=rU20Wzshv5EPy;ltCX;La~!9&=))Dt#|6$)eoHG00{>f@u0PE#@p2;}9g ze}}82vqMAPl)RVGv?!?HpjO5Gj=ouMd~`c_?MR(~uoKYadikU>&Zr5XMy+iAZ4+b9 zlTiPl5fRyOH%6{futpl`1Jmdv8+$#HDX}2z0d!&l(ZO6MVg4#(38FX}{tZqG$U(6b zxee}TOPsUoh+*uW>yc-7ppw-@Q(#QsiKX?$4)`)pZtbv|c z#Z`@)_Jx$}(MgNR4m=#c(B*KmU0QUHQ7yr4rmfHoBHMl5!(~D!GBi z&(-4B*(@uH_cEAt_p?N}U!oJ~2YO@)IK2}iqUqP@PHXHO5s#~ool8fOH1)yNz?*0G{CVD!lHOwq)U8%sYD zN*_E<07VNp4yDN|Vw$-u#ciHi3V#W8em#Y}vGQ)8>;{9X7o7{&3{vqY^5t!SeKH?4 zs+gp3dHEP|d^Gwiqg^Pbh5I7;K{YTRwisdgx;8S!-XP&~(kdvWKy6lEv-Y9d%=lCH z$AzdQ>HW7qv}lgZ!D~!53-8JMao-^~W2wlE{N07#Q*ODZj&%}>eiA~i(%_wELt8i^ggmB*FP$uN`Z@ylq~TBA3YT+K8h74YjFKG~skYKSVV zop9Pm^^BsI#ny$UgrXR+$imCKkZeJVdHOmlGB7uEez&7E*rySm8-UIq_+Mc?R=aJ2oMj zwfXXqOLzqhCb1pqV6>QcF(Bj_tCGH((JKX`j>OFF+-0_W`H z?%5(T$hZ!ls6bESN&9)k-$kVwqwGg3&YUtSTs1^DS-HFKhOoZcvvdKne96>mx zW62-uI?DXiO+s*;ZRq=#N;Rl&Y~+I*sEC<1CxjJj3qFcAo7+1UMvs26iqXJXT$Z>G zwQ(s((_NC2BASG&73jAA0vU;^r5=`SR$|$df;??dom;cok^&wtBN3k&xr@nmR5>_Y9sfZBf1#@38j!cWEH-I&>lGpK7i20y2^& zB=52WpJ2EIgreg&J=k2)6$%hY?(ULHV)DhzSMU`~+3vJCf%jK4^hMQflnCL42ci>R zrLy$B6f0%;PP7NsCo zcS?|OBK4gA0%!Q7@M^mk{j~6tpk*238pn!_zfeFbH{tj=T@i+YrMoCil>&i6G?Olk zI3{b};HW^q*}SF(M8m1xws(fGL%m*&u>1^92x4K!EAM5sq{~$}fB4jK#CO@Lf-6ri zn*p~ju!wS~MG_+s+D{R&`bnUG%!-;piU^SW)2xbVsEc~~4beaF;~HuDSi=a6bE@yf z5Ffq0TeGd1`~)@Eeh(aLmd@?fKCozXM!@U@m6OYY`8_??G{3H+`b%i)(dYQy3nc^? zu*DOe1iDQxvYV93AKK*vV2+{p>B)F#Ph*wbV}JPG%s(TDTrVSe*Q>~&gzVp_s>rt~ zO>qp)vxB+E{uWul@oMxAM^YCPmM;5CTtp7)WiE&c=wdb*URx*vMzu&Q+V-$fBrXlu zj6T_$Pn-W$)l*>X#uPtpXu?0w;IbE6nZ^=AK zuW8?V-V8`=h<7hMDyG1OV}RcNJ+}DmDP8naaG016Hy))1ZUNSf$FXc~a_zKrR+8f! z8Hv-t(H{6D{vro!wV?qSDe!A5HM zne(66XUC6JoqZd6@Sc;;);SENR&klq;NACQy+F>SS;;Fe{HzvJtK<7{vqh6}ieY_y z0JGeBdSPyQcko<1fDdpnF~ealM6eUWUa!2_Fzo>jmA(OJjH#do1rti65Ig)aSBDh3 z@-T;K0^-I|VeojaZRsw^kv##!GTZka=BIfGn91h{U+uDh*~8~LG90sTD(o8tlRwlU z!~rIyP)Q=(k6f9TLoe@tegM?TpCEE>W;W#!hhUo}BJTImCSu*F0QU<(zq|y&rvTvg zkZh_OvO9ZJ&{ie#Jn<7A&R$Z;^0R5x`>Kxo=MJDko#5;}pfjol)yORhg*jNQ83L_2c~yAa83sEY6_x zI_6Ff2P_jYX4c|VGxU#AcN{gtPk0aV+nEeuRuMTsxd+_765GuH?{~mb1%OK76=6!^ zRf&zNyJn^FpvTXT3Y|;)e_MS|%bYV51%>M=74fvFV5{cC#+@ioV-idk2h2|QY>mr9jYjm^S7)_+&#O8V*C z&`*Tkb6gY=XK4|ng|>XO>xw?kwkL01qc!Lve=zO)UE4dc!z7P)p!)9p{&(a-++&N! zT z{yWT;5B8dGrX=Jalgbkrw%Cm+7y0~iA#}k4uU)V%%69%Ob}wKahh02cWjX&AJ7d`5 z2Rx*01ph5|y430}+B@Z0>rj2G_NI*Y6U81_qtD!^PlQ_x{VA-N`83E7tg{wE)I7-m zE~hZSmY1u0`n~g9^{poGaEJ!wBAC$j5klD49WW1bzQzK;NUamKrP%q^7jdOpYM-LL z^OZ5~m(_mHIR$iHcv%+`YehKb4__iefI9h_VG*aK5W8stRA9=rSvVRGESQZW} zalKldUj2Ie%ciE`YVD?n^F=4KuCY>+OT^J=*7}Qxennh}SQ*mQM?O-|IgbyE3l%0L z*yMtKyDVKatV@?i%ZX{|4a!5h;;baZMI7jn#2jlK@80+v?hJ%}oyrOw@`;6yk0`ah4ykzAu)z|m@03Gm zZa9|8n3Ug+M0|U#$##?^&cZ|`xz1v=^|{aOHMioacB+pG)qcrd!|UmKG4SPM(Di1k zd$B?{L*Mh~-e|RPhlJny<^K4GpX6Mutf|?;PJ3ZezAnk!?pMcYNzST~L7)0(rKoQ) zuPl50;DusZ2cHVx-kvcs)I+9f3dHY|$A^7=ru38%vcMX;?QZkdaRj$(>NwD zNz!zmDI_x5Z*^y}|2%e3P`&h^aTpnG^*Aq{j@C|nP|X(o4H`wpZ->kJXn0DR&T4ds z|L0g6`)zhk$c8%rUIuY(Qn#g(|Iy27MF#FqL4}%l7wVLj3ekrvKKga8S=5|binE22Y z_j)qj6`a0}@rZLzKfJOrvCCm9Q%JT6msJKdS?ZK*!R0`q#zluhCAf{QB3Td}|82(f zS@aLXPfs5FD&8K<kqV+r-43hBdXke$eg9L|R$B zxs2Sye2JP!mY7#_8gu{e^RJsZbP3-?-A}j2KRmIlz@~bZsW?}|!h#kPv9I}?A^Uke zBHt67HZMp}whPXBx69X=#21H?!s+Nwz!yFaY<(=JrNBbYfycts%OJwxu+iG5?I?ne z)ETOh!LMM$5xNPQD%*V0tImOSXNPFpr=pO2{?avbP5O1Q?R2)7QMAkDWQG9#r+vP} z37?TYgP@sn^xN2Aj7MJ>Xg>IXafe0?*eX(L!aoMn(<8zOnAW6+g>b!z{c$V#;gdhP zed_c;xA!9vb$4XtCSdq(FP7W#M`La~p8vzF1!c@l=yX5Z1ubpKWo+fRb2Vi$ z;)}G1hbr-vNIhcHsgx&X>uEP1O}Kr!HwPvt3)VL|0oXYIwdjJ8J&8rD(S4y?Wnf_= z@7>?1>ACK6_a}{%^*Bh%tLiaaqIOw-C=Qj~d_(mp<2N({##Y3VJ|x0{#QaA>HeV{e z-`Z`q^F1-Q7UcF&NDxa1nc!<8rwkXnd3VQr$d8c6v?8(4|sl$VGGWY#r8;GUD=2?XKf`Z^X0SFX&9}TA9%} zY<1t=-+9qEbjz9&Y5E!1eg4$t6^s$Up;$@VcLBQ1dzLT-yW<4>_G@7r=%r4$Rx9TEO*;qEZQ=}HrW3g|(zUTNvad^T z+&l+~EYVFCVw9n7W~~_|=_nQHaHQQqCl|$>E*}8Zb2Et|7pgB3&04$lcPg)ywm-Tp zS10ezieaXx$&VJyl2_aBo? zS&w7S6XdU6eVl=VgTGXek<{+5c)!@*F>chr$ zPfe6M)TwT;aMOh~;d!n#TI*$SvES8&``O+4++1;x;+tzRJH@Lvom`98a$V5>xR^;K z2j?n+wlS-yDXvwWY~sEe!SuhzIYZ%bT&TL{H4m2<@~=R|OB5{@vVV#FoWSCOo}6XX zkURZgs^(>n=h1YII32l6bj(X6j%uR~$VA4q+oso-nZ%jIY%ARZg;7tgPHa-iJFB>(2@GPi{Z%`K5V@+CE>G4(q;SYpRsOqpx}H{J;8({ z#sYt0qb$4rhkl5|V@+hF)}xZ{*SF_Rq> /etc/profile && \ +adduser --disabled-password --gecos "" tutorial && \ +echo "tutorial ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers && \ +mkdir /home/tutorial/.ssh/ + +ENV HOME /home/tutorial +ENV NOTVISIBLE "in users profile" + +# ------------------------------------------------------------ +# Set-Up SSH with our Github deploy key +# ------------------------------------------------------------ + +ADD ssh/config /home/tutorial/.ssh/config +ADD ssh/id_rsa.mpi /home/tutorial/.ssh/id_rsa +ADD ssh/id_rsa.mpi.pub /home/tutorial/.ssh/id_rsa.pub +ADD ssh/id_rsa.mpi.pub /home/tutorial/.ssh/authorized_keys + +#--------------------------------------------------------------- +#LD_LIBRARY_PATH +#--------------------------------------------------------------- + +RUN export LD_LIBRARY_PATH=/usr/lib/openmpi/lib/ + +WORKDIR /home/tutorial +EXPOSE 22 +CMD ["/usr/sbin/sshd", "-D"] diff --git a/paddle/scripts/cluster_train_v2/openmpi/docker_cluster/head.yaml b/paddle/scripts/cluster_train_v2/openmpi/docker_cluster/head.yaml new file mode 100644 index 0000000000..34835e5eb8 --- /dev/null +++ b/paddle/scripts/cluster_train_v2/openmpi/docker_cluster/head.yaml @@ -0,0 +1,25 @@ +apiVersion: extensions/v1beta1 +kind: Deployment +metadata: + name: mpi-header + labels: + app: mpi-header +spec: + replicas: 1 + template: + metadata: + labels: + app: mpi-header + spec: + containers: + - image: typhoon1986/paddle-openmpi + name : mpi-header + resources: + limits: + cpu: 500m + memory: 2Gi + requests: + cpu: 500m + memory: 2Gi + ports: + - containerPort: 22 diff --git a/paddle/scripts/cluster_train_v2/openmpi/docker_cluster/mpi-nodes.yaml b/paddle/scripts/cluster_train_v2/openmpi/docker_cluster/mpi-nodes.yaml new file mode 100644 index 0000000000..2fd5cb4d44 --- /dev/null +++ b/paddle/scripts/cluster_train_v2/openmpi/docker_cluster/mpi-nodes.yaml @@ -0,0 +1,26 @@ +apiVersion: extensions/v1beta1 +kind: Deployment +metadata: + name: mpi-nodes + labels: + app: mpi-nodes +spec: + replicas: 3 + template: + metadata: + labels: + app: mpi-nodes + spec: + containers: + - image: typhoon1986/paddle-openmpi + name : mpi-nodes + resources: + limits: + cpu: 500m + memory: 2Gi + requests: + cpu: 500m + memory: 2Gi + ports: + - containerPort: 22 + imagePullPolicy: Always diff --git a/paddle/scripts/cluster_train_v2/openmpi/docker_cluster/ssh/config b/paddle/scripts/cluster_train_v2/openmpi/docker_cluster/ssh/config new file mode 100644 index 0000000000..a9ecad07c3 --- /dev/null +++ b/paddle/scripts/cluster_train_v2/openmpi/docker_cluster/ssh/config @@ -0,0 +1 @@ +StrictHostKeyChecking no diff --git a/paddle/scripts/cluster_train_v2/openmpi/docker_cluster/ssh/id_rsa.mpi b/paddle/scripts/cluster_train_v2/openmpi/docker_cluster/ssh/id_rsa.mpi new file mode 100644 index 0000000000..23768343ed --- /dev/null +++ b/paddle/scripts/cluster_train_v2/openmpi/docker_cluster/ssh/id_rsa.mpi @@ -0,0 +1,27 @@ +-----BEGIN RSA PRIVATE KEY----- +MIIEogIBAAKCAQEA7PWLZmgdJ508dD15T6+xqGDvL9Ehzo9SgsnN6xJ+qpUvvOi4 +1axW0AqR4MnPTg/uuvk+x4tUpuufOW4w22UTGjsdvmIVWa9ujLtcRiN3YPY+SU+Y +O5FfqKg7r/hBn+/GMcSoffwSs7vVgmhBBnp/mJh2O1cOAFZEe98/47mbg3/kHBAk +36NOQktaU3l48B38EhBTnjWfcEGm1HcTRPFxXV5Wiko6ZhKFEuHcTVKng4ROtUqE +mgHyI0aB7TAxg4na0ejItsYWEPWGeDOw6ms/4MwylxNosWzHFPW9p4zgLCLNr+b6 +bDDfYKjXZflAuTQtQhLmJUwD9uuYLAijpSE2fQIDAQABAoIBADgcgRET8Gt0CV/B +OtvKz/f+VEVvcWD3gWNlJDTZIVOFllNWjIZUlA4ZoqenQkbK8Q4nfV1FOht4yjCQ +TlN1oMtiWk297i5Zo4UBzPzy4w774I39oh/g8dT/WXr2/5s+7SDV38xNh6Q2A34o +79T35wUcfUrZ93/O7dKjb/6d8hx2FMha0wVKqY4lmG1lQE3bbx3kakec0PdvU5kO +YHKlpqj3pMR7CpMa+4yL/iXFwWYmnK+uu+zw7JR7PwvH1CzrnvW438wjQ1QmYbSx +mHHOE89X67Lsl5hn81qYWBhpwAlBwi1qscsE0cV9GcFyKqWFqZsj5coM9u3CRfvy +lrWe1OUCgYEA+LBUFEd3Hxs4sFiYElJ8R9SAs1udaqPvAl01hTEijJLfYlMMVs/y +rgNN7j22zjDak2f8QdyMJZX7EZdRmdYcHO0csYOwbYvalzcnwk+U3mxmdD3r4xSo +DSvkJ70fogAqUlcVIg2re6fCmZVJQTvMQYTVEM8zQomJRt/Lb2esSfsCgYEA8+zv +44aToe8uqiDs4w8guRW7LCDkTw4z4IVo9JUibIaPjaAs5bZEBXSB43EEywXCR75H +fML0rU1PVvKh1rqcvZdVzm+XMWVr3asPk0sapaiHaTcmyZvJRDxxqbLFp0zRP1T6 +cCtXNFdHWU4KiuKrUi6cDyOKchpfkSZa4seiT+cCgYB+n4FgBfdQPlMB70oW4irn +g/q32CjxuGCk6oKqu5bkzo+xB6obtavSEFqouIGQwO056tNVUY+GP7Rjg5GH663K +yKw4cl3tmS0Gm43B8TVSfw03mKO3rrfWZQe5eCFYIg9qd26KNT2gK435FzsCXQkm +PxUhhu6JrW/ZR2/U3Iur6wKBgADrWLAb1ryagSuE+j+U1AO+kDkHWrTtkcZ72jxp +v3p3O11GSEUJXdJDcSXhTCpTuDq6/dv7hB6PFwh126RKicKxKlKf2wsFndV1Cpb8 +hnovW2tLGOtTmfuW2rrQAKyzvmolsNfxYd/BoHQ2thV16z1hDZeFA8WQUeHjKh6G +sBbrAoGATdtQlaUxx4izua6k02ihkxx/cRYwDl2N8UDvDBHokS7vJFMX8b8NpsGg +zMElnqSpu/pe/0UG7N2MtPF6uyMcX8AZzzcsRkiMkDvWJzYt8Jpf+Eyd/uryF+Yv +yrXaOEY83tm6x/fny5ZaZmk8lNth7bfWywuTMkZLX3fYpWtIeE4= +-----END RSA PRIVATE KEY----- diff --git a/paddle/scripts/cluster_train_v2/openmpi/docker_cluster/ssh/id_rsa.mpi.pub b/paddle/scripts/cluster_train_v2/openmpi/docker_cluster/ssh/id_rsa.mpi.pub new file mode 100644 index 0000000000..015f2b42e7 --- /dev/null +++ b/paddle/scripts/cluster_train_v2/openmpi/docker_cluster/ssh/id_rsa.mpi.pub @@ -0,0 +1 @@ +ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDs9YtmaB0nnTx0PXlPr7GoYO8v0SHOj1KCyc3rEn6qlS+86LjVrFbQCpHgyc9OD+66+T7Hi1Sm6585bjDbZRMaOx2+YhVZr26Mu1xGI3dg9j5JT5g7kV+oqDuv+EGf78YxxKh9/BKzu9WCaEEGen+YmHY7Vw4AVkR73z/juZuDf+QcECTfo05CS1pTeXjwHfwSEFOeNZ9wQabUdxNE8XFdXlaKSjpmEoUS4dxNUqeDhE61SoSaAfIjRoHtMDGDidrR6Mi2xhYQ9YZ4M7Dqaz/gzDKXE2ixbMcU9b2njOAsIs2v5vpsMN9gqNdl+UC5NC1CEuYlTAP265gsCKOlITZ9 oweidner@peahi diff --git a/paddle/scripts/cluster_train_v2/openmpi/start_mpi_train.sh b/paddle/scripts/cluster_train_v2/openmpi/start_mpi_train.sh new file mode 100644 index 0000000000..c645495448 --- /dev/null +++ b/paddle/scripts/cluster_train_v2/openmpi/start_mpi_train.sh @@ -0,0 +1,28 @@ +#!/bin/bash +# General trainning configurations + +NICS=eth0 +PADDLE_INIT_PORT=7164 +PADDLE_INIT_PORTS_NUM=1 +PADDLE_INIT_PORTS_NUM_FOR_SPARSE=1 +PADDLE_INIT_PSERVERS=$(cat machines | sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/,/g') +PADDLE_INIT_USE_GPU=False + +PADDLE_INIT_NUM_GRADIENT_SERVERS=${OMPI_COMM_WORLD_SIZE} +PADDLE_INIT_TRAINER_ID=${OMPI_COMM_WORLD_RANK} +PADDLE_CLUSTER_TRAIN=True + +env + +# start pserver +stdbuf -oL nohup paddle pserver --port=$PADDLE_INIT_PORT --ports_num=$PADDLE_INIT_PORTS_NUM \ + --ports_num_for_sparse=$PADDLE_INIT_PORTS_NUM_FOR_SPARSE --nics=$NICS \ + --comment=paddle_cluster_pserver \ + --num_gradient_servers=$PADDLE_INIT_NUM_GRADIENT_SERVERS &> logs/pserver.log & + +# start trainer +# NOTE: train.py will use the above environment variables as configuration +python train.py &> logs/train.log + +# kill background pservers when train finishes +ps -ef | grep pserver | awk '{print $2}' | xargs kill From 747b541957a1c8d83d85bf45db7614ae8ae623bf Mon Sep 17 00:00:00 2001 From: hedaoyuan Date: Thu, 19 Oct 2017 22:40:58 +0800 Subject: [PATCH 40/76] Follow comments --- paddle/capi/CMakeLists.txt | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/paddle/capi/CMakeLists.txt b/paddle/capi/CMakeLists.txt index f59b1aa3a1..2198f17378 100644 --- a/paddle/capi/CMakeLists.txt +++ b/paddle/capi/CMakeLists.txt @@ -40,8 +40,8 @@ set(PADDLE_CAPI_INFER_LIBS cc_library(paddle_capi_whole DEPS paddle_capi ${PADDLE_CAPI_INFER_LIBS}) # Link the static library for inference -cc_library(paddle_nn_engine DEPS paddle_capi paddle_utils paddle_parameter paddle_math paddle_cuda paddle_proto) -cc_library(paddle_layers DEPS paddle_function paddle_gserver) +cc_library(paddle_capi_engine DEPS paddle_capi paddle_utils paddle_parameter paddle_math paddle_cuda paddle_proto) +cc_library(paddle_capi_layers DEPS paddle_function paddle_gserver) # Link the shared library for inference if(NOT IOS) @@ -57,7 +57,7 @@ endif() install(FILES ${CAPI_HEADERS} DESTINATION include/paddle) install(FILES ${CMAKE_CURRENT_BINARY_DIR}/config.h DESTINATION include/paddle) if(ANDROID) - install(TARGETS paddle_nn_engine paddle_layers paddle_capi_shared + install(TARGETS paddle_capi_whole paddle_capi_engine paddle_capi_layers paddle_capi_shared ARCHIVE DESTINATION lib/${ANDROID_ABI} LIBRARY DESTINATION lib/${ANDROID_ABI}) execute_process( @@ -82,7 +82,7 @@ if(ANDROID) )" ) else(ANDROID) - install(TARGETS paddle_nn_engine paddle_layers ARCHIVE DESTINATION lib) + install(TARGETS paddle_capi_whole paddle_capi_engine paddle_capi_layers ARCHIVE DESTINATION lib) if(NOT IOS) install(TARGETS paddle_capi_shared DESTINATION lib) endif() From 56d5db8bea96c52232bdc708d706b438d188a355 Mon Sep 17 00:00:00 2001 From: hedaoyuan Date: Thu, 19 Oct 2017 23:39:19 +0800 Subject: [PATCH 41/76] Bug fix of libpaddle_capi_whole.a in x86. --- paddle/capi/CMakeLists.txt | 32 +++++++++++++++++++++++--------- 1 file changed, 23 insertions(+), 9 deletions(-) diff --git a/paddle/capi/CMakeLists.txt b/paddle/capi/CMakeLists.txt index 2198f17378..e966d5d852 100644 --- a/paddle/capi/CMakeLists.txt +++ b/paddle/capi/CMakeLists.txt @@ -29,15 +29,29 @@ add_style_check_target(paddle_capi ${CAPI_SOURCES} ${CAPI_HEADER} add_dependencies(paddle_capi paddle_proto) # TODO: paddle_capi_whole will be removed. -set(PADDLE_CAPI_INFER_LIBS - paddle_utils - paddle_parameter - paddle_math - paddle_cuda - paddle_function - paddle_gserver - paddle_proto) -cc_library(paddle_capi_whole DEPS paddle_capi ${PADDLE_CAPI_INFER_LIBS}) +if(MOBILE_INFERENCE) + set(PADDLE_CAPI_INFER_LIBS + paddle_utils + paddle_parameter + paddle_math + paddle_cuda + paddle_function + paddle_gserver + paddle_proto) + cc_library(paddle_capi_whole DEPS paddle_capi ${PADDLE_CAPI_INFER_LIBS}) +else() + set(PADDLE_CAPI_INFER_LIBS + paddle_utils + paddle_parameter + paddle_math + paddle_cuda + paddle_function + paddle_gserver + paddle_proto + paddle_pserver + paddle_network) + cc_library(paddle_capi_whole DEPS paddle_capi ${PADDLE_CAPI_INFER_LIBS}) +endif() # Link the static library for inference cc_library(paddle_capi_engine DEPS paddle_capi paddle_utils paddle_parameter paddle_math paddle_cuda paddle_proto) From 3db52783012d20d5174e39ed4ae419179614a1d0 Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Thu, 19 Oct 2017 10:08:12 -0700 Subject: [PATCH 42/76] Feature/py executor test (#4922) * Implement FC layer with helper * Update LayerHelper * Add debug string for Python ProtoBuf and Rename `Sync` to `Flush` * Add check of ProtoBuf initialization * Layer wrapper for FC * Fix unittest * Fix CI * Add code generator * AttributeChecker Better error log and speicalize bool Since lots of types can be cast to bool * Complete mlp, fit_a_line * Expose get global scope * Make global scope not thread-safe 1. It is no need to make global scope thread-safe, since it will be invoked in Python main thread. 2. Do not free the global scope when C++ exit. Let the OS free memories, otherwise, we need to handle the destroy dependencies. See https://google.github.io/styleguide/cppguide.html#Static_and_Global_Variables * Fix * Implementation of simple conv_2d layer * Stash * Remove private data members in OpRegister * Fix bugs * Stash * Expose FeedFetchList as VarType * Change ProgramDesc not a global variable * Polish code style * Stash * Correct implement BlockDesc destructor * Correct implement BlockDesc destructor * Unify program as parameter name * Fix bugs * Add unittest * Fix unit test error * Remove unused functions * Add clone for Python Program * Working on executor * Stash * Add glog as dependencies of ops * Use VLOG to logging some information is helpful when we debug Paddle * Expose VarDesc::persistable to Python * Test executor * Complete unittest * Polish code * Fix merge error * Follow comment * Polish Python Code --- paddle/framework/CMakeLists.txt | 2 +- paddle/framework/executor.cc | 8 ++- paddle/framework/feed_fetch_method.h | 14 ++++- paddle/framework/framework.proto | 2 + paddle/framework/program_desc_test.cc | 2 +- paddle/framework/variable.h | 5 +- paddle/operators/feed_op.cc | 23 ++++++-- paddle/operators/fetch_op.cc | 20 +++++-- paddle/pybind/protobuf.cc | 4 +- paddle/pybind/pybind.cc | 20 ++++++- python/paddle/v2/framework/executor.py | 59 +++++++++++++++++++ python/paddle/v2/framework/framework.py | 10 +++- python/paddle/v2/framework/layers.py | 5 +- .../framework/tests/test_executor_and_mul.py | 36 +++++++++++ 14 files changed, 186 insertions(+), 24 deletions(-) create mode 100644 python/paddle/v2/framework/executor.py create mode 100644 python/paddle/v2/framework/tests/test_executor_and_mul.py diff --git a/paddle/framework/CMakeLists.txt b/paddle/framework/CMakeLists.txt index 6e32a1c99b..774c7b0217 100644 --- a/paddle/framework/CMakeLists.txt +++ b/paddle/framework/CMakeLists.txt @@ -43,7 +43,7 @@ add_custom_command(TARGET framework_py_proto POST_BUILD cc_library(backward SRCS backward.cc DEPS net_op) cc_test(backward_test SRCS backward_test.cc DEPS backward recurrent_op device_context) -cc_library(executor SRCS executor.cc DEPS op_registry device_context scope framework_proto backward) +cc_library(executor SRCS executor.cc DEPS op_registry device_context scope framework_proto backward glog) cc_library(prune SRCS prune.cc DEPS framework_proto) cc_test(prune_test SRCS prune_test.cc DEPS op_info prune recurrent_op device_context) diff --git a/paddle/framework/executor.cc b/paddle/framework/executor.cc index 00caa6e1d5..d50f0da032 100644 --- a/paddle/framework/executor.cc +++ b/paddle/framework/executor.cc @@ -68,9 +68,13 @@ void Executor::Run(const ProgramDesc& pdesc, Scope* scope, int block_id) { for (auto& var : block.vars()) { if (var.persistable()) { - scope->Var(var.name()); + auto* ptr = scope->Var(var.name()); + VLOG(3) << "Create Variable " << var.name() + << " global, which pointer is " << ptr; } else { - local_scope.Var(var.name()); + auto* ptr = local_scope.Var(var.name()); + VLOG(3) << "Create Variable " << var.name() + << " locally, which pointer is " << ptr; } } diff --git a/paddle/framework/feed_fetch_method.h b/paddle/framework/feed_fetch_method.h index 826d180bfc..9b23ad271c 100644 --- a/paddle/framework/feed_fetch_method.h +++ b/paddle/framework/feed_fetch_method.h @@ -13,6 +13,8 @@ See the License for the specific language governing permissions and limitations under the License. */ #pragma once +#include "glog/logging.h" +#include "paddle/framework/feed_fetch_type.h" #include "paddle/framework/scope.h" #include "paddle/framework/variable.h" @@ -24,6 +26,7 @@ void SetFeedVariable(const LoDTensor& input, const std::string& var_name, size_t index) { // If var_name Variable is not found in GlobalScope, a new variable will // be created. + VLOG(3) << "SetFeedVariable name=" << var_name << " index=" << index; Variable* g_feed_value = GetGlobalScope().Var(var_name); auto& feed_inputs = *(g_feed_value->GetMutable>()); @@ -40,10 +43,15 @@ LoDTensor& GetFetchVariable(const std::string& var_name, size_t index) { // Since we want to fetch LodTensor from a variable, the variable must // be created alreadly. Variable* g_fetch_value = GetGlobalScope().FindVar(var_name); - auto& fetch_outputs = - *(g_fetch_value->GetMutable>()); + PADDLE_ENFORCE(g_fetch_value->IsType(), + "Only %s can be invoked by GetFetchVariable", + typeid(FeedFetchList).name()); + auto& fetch_outputs = *g_fetch_value->GetMutable(); + auto& tensor = fetch_outputs[index]; + VLOG(3) << "Fetch " << var_name << " with index " << index + << " shape= " << tensor.dims(); PADDLE_ENFORCE_LT(index, fetch_outputs.size()); - return fetch_outputs[index]; + return tensor; } } // namespace framework diff --git a/paddle/framework/framework.proto b/paddle/framework/framework.proto index 008fb45fb7..2aa961f140 100644 --- a/paddle/framework/framework.proto +++ b/paddle/framework/framework.proto @@ -112,6 +112,8 @@ message VarDesc { enum VarType { LOD_TENSOR = 1; SELECTED_ROWS = 2; + FEED_MINIBATCH = 3; + FETCH_LIST = 4; } required string name = 1; required VarType type = 2; diff --git a/paddle/framework/program_desc_test.cc b/paddle/framework/program_desc_test.cc index 32ee275429..c9709a2d3f 100644 --- a/paddle/framework/program_desc_test.cc +++ b/paddle/framework/program_desc_test.cc @@ -80,4 +80,4 @@ TEST(ProgramDesc, copy_ctor) { // different and it is correct. } } // namespace framework -} // namespace paddle \ No newline at end of file +} // namespace paddle diff --git a/paddle/framework/variable.h b/paddle/framework/variable.h index 38fc2720a3..a80f0e66b5 100644 --- a/paddle/framework/variable.h +++ b/paddle/framework/variable.h @@ -25,7 +25,10 @@ class Variable { public: template const T& Get() const { - PADDLE_ENFORCE(IsType(), "Variable must be type %s", typeid(T).name()); + PADDLE_ENFORCE(holder_ != nullptr, "Variable must hold some thing"); + PADDLE_ENFORCE(IsType(), + "Variable must be type %s, the holding type is %s", + typeid(T).name(), holder_->Type().name()); return *static_cast(holder_->Ptr()); } diff --git a/paddle/operators/feed_op.cc b/paddle/operators/feed_op.cc index d742bbe51b..bf453c8596 100644 --- a/paddle/operators/feed_op.cc +++ b/paddle/operators/feed_op.cc @@ -26,8 +26,9 @@ class FeedOp : public framework::OperatorBase { : OperatorBase(type, inputs, outputs, attrs) {} void Run(const framework::Scope &scope, const platform::DeviceContext &dev_ctx) const override { - auto feed_var_name = Input("Input"); + auto feed_var_name = Input("X"); auto *feed_var = scope.FindVar(feed_var_name); + PADDLE_ENFORCE(feed_var != nullptr, "Cannot find feed_var in scope, feed_var_name is %s", feed_var_name); @@ -40,6 +41,9 @@ class FeedOp : public framework::OperatorBase { auto col = Attr("col"); + VLOG(3) << "Feed Var " << feed_var_name << "'s " << col << " column to var" + << out_name; + auto &feed_list = feed_var->Get(); auto &feed_item = feed_list.at(static_cast(col)); auto *out_item = out_var->GetMutable(); @@ -48,10 +52,21 @@ class FeedOp : public framework::OperatorBase { } }; +class FeedOpInfoMaker : public framework::OpProtoAndCheckerMaker { + public: + FeedOpInfoMaker(framework::OpProto *proto, + framework::OpAttrChecker *op_checker) + : OpProtoAndCheckerMaker(proto, op_checker) { + AddInput("X", "The input of feed op"); + AddOutput("Out", "The output of feed op"); + AddComment("feed op, it should not be configured by users directly"); + AddAttr("col", "column of feed"); + } +}; + } // namespace operators } // namespace paddle -// We do not need to register OpInfoMaker, -// since feed operator will not be used by end users directly REGISTER_OPERATOR(feed, paddle::operators::FeedOp, - paddle::framework::EmptyGradOpMaker); + paddle::framework::EmptyGradOpMaker, + paddle::operators::FeedOpInfoMaker); diff --git a/paddle/operators/fetch_op.cc b/paddle/operators/fetch_op.cc index 55d6ac0939..524e77d6ad 100644 --- a/paddle/operators/fetch_op.cc +++ b/paddle/operators/fetch_op.cc @@ -27,7 +27,7 @@ class FetchOp : public framework::OperatorBase { void Run(const framework::Scope &scope, const platform::DeviceContext &dev_ctx) const override { - auto fetch_var_name = Input("Input"); + auto fetch_var_name = Input("X"); auto *fetch_var = scope.FindVar(fetch_var_name); PADDLE_ENFORCE(fetch_var != nullptr, "Cannot find fetch variable in scope, fetch_var_name is %s", @@ -52,13 +52,25 @@ class FetchOp : public framework::OperatorBase { // FIXME(yuyang18): Should we assume the fetch operator always generate // CPU outputs? dst_item.CopyFromTensor(src_item, platform::CPUPlace(), dev_ctx); + + VLOG(3) << "Fetch variable " << fetch_var_name << " to " << out_name; } }; +class FetchOpInfoMaker : public framework::OpProtoAndCheckerMaker { + public: + FetchOpInfoMaker(framework::OpProto *proto, + framework::OpAttrChecker *op_checker) + : OpProtoAndCheckerMaker(proto, op_checker) { + AddInput("X", "The input of fetch op"); + AddOutput("Out", "The output of fetch op"); + AddComment("fetch op, it should not be configured by users directly"); + AddAttr("col", "column of fetch"); + } +}; } // namespace operators } // namespace paddle -// We do not need to register OpInfoMaker, -// since fetch operator will not be used by end users directly REGISTER_OPERATOR(fetch, paddle::operators::FetchOp, - paddle::framework::EmptyGradOpMaker); + paddle::framework::EmptyGradOpMaker, + paddle::operators::FetchOpInfoMaker); diff --git a/paddle/pybind/protobuf.cc b/paddle/pybind/protobuf.cc index 58739d888a..405ac544e1 100644 --- a/paddle/pybind/protobuf.cc +++ b/paddle/pybind/protobuf.cc @@ -222,7 +222,9 @@ void BindVarDsec(py::module &m) { py::enum_(var_desc, "VarType", "") .value("LOD_TENSOR", VarDesc::LOD_TENSOR) - .value("SELECTED_ROWS", VarDesc::SELECTED_ROWS); + .value("SELECTED_ROWS", VarDesc::SELECTED_ROWS) + .value("FEED_MINIBATCH", VarDesc::FEED_MINIBATCH) + .value("FETCH_LIST", VarDesc::FETCH_LIST); } void BindOpDesc(py::module &m) { diff --git a/paddle/pybind/pybind.cc b/paddle/pybind/pybind.cc index 16661b93e5..84ebe3c2b8 100644 --- a/paddle/pybind/pybind.cc +++ b/paddle/pybind/pybind.cc @@ -111,6 +111,7 @@ PYBIND11_PLUGIN(core) { new (&instance) LoDTensor(new_lod); #endif }) + .def("__init__", [](LoDTensor &instance) { new (&instance) LoDTensor(); }) .def("set_lod", [](LoDTensor &self, const std::vector> &lod) { #ifndef PADDLE_WITH_CUDA @@ -216,7 +217,8 @@ All parameter, weight, gradient are variables in Paddle. .def(py::init<>()) .def("new_scope", [](Scope &self) -> Scope * { return &self.NewScope(); }, py::return_value_policy::reference) - .def("drop_kids", &Scope::DropKids); + .def("drop_kids", &Scope::DropKids) + .def_static("global_scope", &GetGlobalScope); //! @note: Be careful! PyBind will return std::string as an unicode, not //! Python str. If you want a str object, you should cast them in Python. @@ -264,6 +266,17 @@ All parameter, weight, gradient are variables in Paddle. .def(py::init<>()) .def("__str__", string::to_string); + py::class_(m, "Place") + .def(py::init<>()) + .def("set_place", + [](platform::Place &self, const platform::CPUPlace &cpu_place) { + self = cpu_place; + }) + .def("set_place", + [](platform::Place &self, const platform::GPUPlace &gpu_place) { + self = gpu_place; + }); + py::class_(m, "Operator") .def_static("create", [](py::bytes protobin) { @@ -437,14 +450,15 @@ All parameter, weight, gradient are variables in Paddle. py::class_(m, "Executor") .def(py::init &>()) .def("run", - [](Executor &self, const ProgramDesc &program_desc, int block_id) { + [](Executor &self, ProgramDescBind *program_bind, int block_id) { framework::Scope &global_scope = GetGlobalScope(); - self.Run(program_desc, &global_scope, block_id); + self.Run(*program_bind->Proto(), &global_scope, block_id); }); m.def("unique_integer", UniqueIntegerGenerator); m.def("is_compile_gpu", IsCompileGPU); + //! FIXME: it is no need to `set_xxx_float/double/int` m.def("set_feed_variable_float", framework::SetFeedVariable); m.def("set_feed_variable_double", framework::SetFeedVariable); m.def("set_feed_variable_int", framework::SetFeedVariable); diff --git a/python/paddle/v2/framework/executor.py b/python/paddle/v2/framework/executor.py new file mode 100644 index 0000000000..8da5daad99 --- /dev/null +++ b/python/paddle/v2/framework/executor.py @@ -0,0 +1,59 @@ +import paddle.v2.framework.core as core +from paddle.v2.framework.framework import Block, Program + + +class Executor(object): + def __init__(self, places): + if not isinstance(places, list) and not isinstance(places, tuple): + places = [places] + + act_places = [] + for each in places: + p = core.Place() + p.set_place(each) + act_places.append(p) + + self.executor = core.Executor(act_places) + + def run(self, + program, + feed, + fetch_list, + feed_var_name='feed', + fetch_var_name='fetch'): + if not isinstance(program, Program): + raise TypeError() + + program = program.clone() + global_block = program.global_block() + feed_var = global_block.create_var( + name=feed_var_name, + type=core.VarDesc.VarType.FEED_MINIBATCH, + persistable=True) + + for i, name in enumerate(feed): + out = global_block.var(name) + global_block.prepend_op( + 'feed', + inputs={'X': [feed_var]}, + outputs={'Out': [out]}, + attrs={'col': i}) + # FIXME + core.set_feed_variable_float(feed[name], feed_var.name, i) + + fetch_var = global_block.create_var( + name=fetch_var_name, + type=core.VarDesc.VarType.FETCH_LIST, + persistable=True) + for i, var in enumerate(fetch_list): + global_block.append_op( + type='fetch', + inputs={'X': [var]}, + outputs={'Out': [fetch_var]}, + attrs={'col': i}) + + self.executor.run(program.desc, 0) + return [ + core.get_fetch_variable(fetch_var_name, i) + for i in xrange(len(fetch_list)) + ] diff --git a/python/paddle/v2/framework/framework.py b/python/paddle/v2/framework/framework.py index a24c78171e..a68f2afcfa 100644 --- a/python/paddle/v2/framework/framework.py +++ b/python/paddle/v2/framework/framework.py @@ -256,7 +256,8 @@ class Operator(object): self.desc.set_block_attr(attr_name, attrs[attr_name].desc) self.desc.check_attrs() - self.desc.infer_shape(self.block.desc) + if type not in {'feed', 'fetch'}: + self.desc.infer_shape(self.block.desc) def __str__(self): protostr = self.desc.serialize_to_string() @@ -323,9 +324,12 @@ class Block(object): return self.desc.id def var(self, name): - if name not in self.vars: + if not isinstance(name, basestring): + raise TypeError() + v = self.vars.get(name, None) + if v is None: raise ValueError("var %s not in this block" % name) - return self.vars[name] + return v def all_parameters(self): return {v for k, v in self.vars.iteritems() if isinstance(v, Parameter)} diff --git a/python/paddle/v2/framework/layers.py b/python/paddle/v2/framework/layers.py index c7397716c4..329a6830b6 100644 --- a/python/paddle/v2/framework/layers.py +++ b/python/paddle/v2/framework/layers.py @@ -55,9 +55,11 @@ def data(name, shape, data_type='float32', type=core.VarDesc.VarType.LOD_TENSOR, + append_batch_size=True, program=None): helper = LayerHelper('data', **locals()) - shape = [-1] + shape # append batch size as -1 + if append_batch_size: + shape = [-1] + shape # append batch size as -1 return helper.create_global_variable( name=name, shape=shape, dtype=data_type, type=type) @@ -112,6 +114,7 @@ def _create_op_func_(op_type): _create_op_func_('mean') +_create_op_func_('mul') _create_op_func_('pool2d') diff --git a/python/paddle/v2/framework/tests/test_executor_and_mul.py b/python/paddle/v2/framework/tests/test_executor_and_mul.py new file mode 100644 index 0000000000..35f7757111 --- /dev/null +++ b/python/paddle/v2/framework/tests/test_executor_and_mul.py @@ -0,0 +1,36 @@ +import unittest +from paddle.v2.framework.layers import mul, data +import paddle.v2.framework.core as core +from paddle.v2.framework.executor import Executor +from paddle.v2.framework.framework import g_program +import numpy + + +class TestExecutor(unittest.TestCase): + def test_mul(self): + a = data(name='a', shape=[784], data_type='float32') + b = data( + name='b', + shape=[784, 100], + data_type='float32', + append_batch_size=False) + out = mul(x=a, y=b) + place = core.CPUPlace() + a_np = numpy.random.random((100, 784)).astype('float32') + tensor_a = core.LoDTensor() + tensor_a.set(a_np, place) + b_np = numpy.random.random((784, 100)).astype('float32') + tensor_b = core.LoDTensor() + tensor_b.set(b_np, place) + exe = Executor(place) + outs = exe.run(g_program, + feed={'a': tensor_a, + 'b': tensor_b}, + fetch_list=[out]) + out = numpy.array(outs[0]) + self.assertEqual((100, 100), out.shape) + self.assertTrue(numpy.allclose(out, numpy.dot(a_np, b_np))) + + +if __name__ == '__main__': + unittest.main() From 11bebeb2dc15e30c12ea12d87b85c34611d483bd Mon Sep 17 00:00:00 2001 From: Abhinav Arora Date: Thu, 19 Oct 2017 10:37:51 -0700 Subject: [PATCH 43/76] Removing updates of Beta1 and Beta2 power accumulators outside the op (#4925) --- paddle/operators/adam_op.cc | 12 +------- paddle/operators/adam_op.h | 13 ++------ .../paddle/v2/framework/tests/test_adam_op.py | 30 ++++++++----------- 3 files changed, 15 insertions(+), 40 deletions(-) diff --git a/paddle/operators/adam_op.cc b/paddle/operators/adam_op.cc index e3db70ea12..3572de06bd 100644 --- a/paddle/operators/adam_op.cc +++ b/paddle/operators/adam_op.cc @@ -43,10 +43,6 @@ class AdamOp : public framework::OperatorWithKernel { "Output(Moment1Out) of AdamOp should not be null."); PADDLE_ENFORCE(ctx->HasOutput("Moment2Out"), "Output(Moment2Out) of AdamOp should not be null."); - PADDLE_ENFORCE(ctx->HasOutput("Beta1PowOut"), - "Output(Beta1PowOut) of AdamOp should not be null."); - PADDLE_ENFORCE(ctx->HasOutput("Beta2PowOut"), - "Output(Beta2PowOut) of AdamOp should not be null."); auto lr_dims = ctx->GetInputDim("LearningRate"); PADDLE_ENFORCE_EQ(framework::product(lr_dims), 1, @@ -72,8 +68,6 @@ class AdamOp : public framework::OperatorWithKernel { ctx->SetOutputDim("ParamOut", param_dims); ctx->SetOutputDim("Moment1Out", param_dims); ctx->SetOutputDim("Moment2Out", param_dims); - ctx->SetOutputDim("Beta1PowOut", beta1_pow_dims); - ctx->SetOutputDim("Beta2PowOut", beta2_pow_dims); } }; @@ -92,8 +86,6 @@ class AdamOpMaker : public framework::OpProtoAndCheckerMaker { AddOutput("ParamOut", "(Tensor) Output parameter"); AddOutput("Moment1Out", "(Tensor) Output first moment"); AddOutput("Moment2Out", "(Tensor) Output second moment"); - AddOutput("Beta1PowOut", "(Tensor) Output beta1 power accumulator"); - AddOutput("Beta2PowOut", "(Tensor) Output beta2 power accumulator"); AddAttr("beta1", "(float, default 0.9) " @@ -121,10 +113,8 @@ Adam updates: moment1_out = beta1 * moment1 + (1 − beta1) * grad moment2_out = beta2 * moment2 + (1 − beta2) * grad * grad -beta1_pow_out = beta1_pow * beta1 -beta2_pow_out = beta2_pow * beta2 learning_rate_t = learning_rate_t * - sqrt(1 - beta2_pow_out) / (1 - beta1_pow_out) + sqrt(1 - beta2_pow) / (1 - beta1_pow) param_out = param - learning_rate_t * moment1/ (sqrt(moment2) + epsilon) References: diff --git a/paddle/operators/adam_op.h b/paddle/operators/adam_op.h index 789c2f14b3..45938006db 100644 --- a/paddle/operators/adam_op.h +++ b/paddle/operators/adam_op.h @@ -26,14 +26,10 @@ class AdamOpKernel : public framework::OpKernel { auto param_out_tensor = ctx.Output("ParamOut"); auto moment1_out_tensor = ctx.Output("Moment1Out"); auto moment2_out_tensor = ctx.Output("Moment2Out"); - auto beta1_pow_out_tensor = ctx.Output("Beta1PowOut"); - auto beta2_pow_out_tensor = ctx.Output("Beta2PowOut"); param_out_tensor->mutable_data(ctx.GetPlace()); moment1_out_tensor->mutable_data(ctx.GetPlace()); moment2_out_tensor->mutable_data(ctx.GetPlace()); - beta1_pow_out_tensor->mutable_data(ctx.GetPlace()); - beta2_pow_out_tensor->mutable_data(ctx.GetPlace()); float beta1 = ctx.Attr("beta1"); float beta2 = ctx.Attr("beta2"); @@ -56,18 +52,13 @@ class AdamOpKernel : public framework::OpKernel { auto param_out = framework::EigenVector::Flatten(*param_out_tensor); auto moment1_out = framework::EigenVector::Flatten(*moment1_out_tensor); auto moment2_out = framework::EigenVector::Flatten(*moment2_out_tensor); - auto beta1_pow_out = - framework::EigenVector::Flatten(*beta1_pow_out_tensor); - auto beta2_pow_out = - framework::EigenVector::Flatten(*beta2_pow_out_tensor); auto place = ctx.GetEigenDevice(); moment1_out.device(place) = beta1 * moment1 + (1 - beta1) * grad; moment2_out.device(place) = beta2 * moment2 + (1 - beta2) * grad.square(); - beta1_pow_out.device(place) = beta1_pow * beta1; - beta2_pow_out.device(place) = beta2_pow * beta2; + // All of these are tensors of 1 element - auto lr_t = lr * (1 - beta2_pow_out).sqrt() / (1 - beta1_pow_out); + auto lr_t = lr * (1 - beta2_pow).sqrt() / (1 - beta1_pow); // Eigen does not support automatic broadcast // Get dimensions of moment vector to broadcast lr_t Eigen::DSizes m_dsize(moment1_out_tensor->numel()); diff --git a/python/paddle/v2/framework/tests/test_adam_op.py b/python/paddle/v2/framework/tests/test_adam_op.py index ff6faafa6e..a0d6655d4c 100644 --- a/python/paddle/v2/framework/tests/test_adam_op.py +++ b/python/paddle/v2/framework/tests/test_adam_op.py @@ -33,14 +33,12 @@ class TestAdamOp1(OpTest): self.attrs = {'epsilon': epsilon, 'beta1': beta1, 'beta2': beta2} - param_out, moment1_out, moment2_out, beta1_pow_out, \ - beta2_pow_out = adam_step(self.inputs, self.attrs) + param_out, moment1_out, \ + moment2_out = adam_step(self.inputs, self.attrs) self.outputs = { 'Moment1Out': moment1_out, 'Moment2Out': moment2_out, - 'Beta1PowOut': beta1_pow_out, - 'Beta2PowOut': beta2_pow_out, 'ParamOut': param_out } @@ -78,14 +76,12 @@ class TestAdamOp2(OpTest): attributes = {'epsilon': epsilon, 'beta1': beta1, 'beta2': beta2} - param_out, moment1_out, moment2_out, beta1_pow_out, \ - beta2_pow_out = adam_step(self.inputs, attributes) + param_out, moment1_out, \ + moment2_out = adam_step(self.inputs, attributes) self.outputs = { 'Moment1Out': moment1_out, 'Moment2Out': moment2_out, - 'Beta1PowOut': beta1_pow_out, - 'Beta2PowOut': beta2_pow_out, 'ParamOut': param_out } @@ -127,14 +123,12 @@ class TestAdamOpMultipleSteps(OpTest): def test_check_output(self): for _ in range(self.num_steps): - param_out, moment1_out, moment2_out, beta1_pow_out, \ - beta2_pow_out = adam_step(self.inputs, self.attrs) + param_out, moment1_out, \ + moment2_out = adam_step(self.inputs, self.attrs) self.outputs = { 'Moment1Out': moment1_out, 'Moment2Out': moment2_out, - 'Beta1PowOut': beta1_pow_out, - 'Beta2PowOut': beta2_pow_out, 'ParamOut': param_out } @@ -145,8 +139,10 @@ class TestAdamOpMultipleSteps(OpTest): self.inputs['Param'] = param_out self.inputs['Moment1'] = moment1_out self.inputs['Moment2'] = moment2_out - self.inputs['Beta1Pow'] = beta1_pow_out - self.inputs['Beta2Pow'] = beta2_pow_out + + # Update powers of Beta1 and Beta2 for next time step + self.inputs['Beta1Pow'] *= self.attrs['beta1'] + self.inputs['Beta2Pow'] *= self.attrs['beta1'] # Randomize gradient for next step self.inputs['Grad'] = np.random.uniform( @@ -175,11 +171,9 @@ def adam_step(inputs, attributes): moment1_out = beta1 * moment1 + (1 - beta1) * grad moment2_out = beta2 * moment2 + (1 - beta2) * np.square(grad) - beta1_pow_out = beta1_pow * beta1 - beta2_pow_out = beta2_pow * beta2 - lr_t = lr * np.sqrt(1 - beta2_pow_out) / (1 - beta1_pow_out) + lr_t = lr * np.sqrt(1 - beta2_pow) / (1 - beta1_pow) param_out = param - lr_t * (moment1_out / (np.sqrt(moment2_out) + epsilon)) - return param_out, moment1_out, moment2_out, beta1_pow_out, beta2_pow_out + return param_out, moment1_out, moment2_out if __name__ == "__main__": From 77cac5cdb882bc390fa854b22b1365e941b99731 Mon Sep 17 00:00:00 2001 From: Abhinav Arora Date: Thu, 19 Oct 2017 10:53:14 -0700 Subject: [PATCH 44/76] Removing updates of Beta1 power accumulators outside the op (#4931) --- paddle/operators/adamax_op.cc | 7 +--- paddle/operators/adamax_op.h | 7 +--- .../v2/framework/tests/test_adamax_op.py | 32 ++++++++----------- 3 files changed, 15 insertions(+), 31 deletions(-) diff --git a/paddle/operators/adamax_op.cc b/paddle/operators/adamax_op.cc index e848333ef8..ff25657741 100644 --- a/paddle/operators/adamax_op.cc +++ b/paddle/operators/adamax_op.cc @@ -41,8 +41,6 @@ class AdamaxOp : public framework::OperatorWithKernel { "Output(MomentOut) of AdamaxOp should not be null."); PADDLE_ENFORCE(ctx->HasOutput("InfNormOut"), "Output(InfNormOut) of AdamaxOp should not be null."); - PADDLE_ENFORCE(ctx->HasOutput("Beta1PowOut"), - "Output(Beta1PowOut) of AdamaxOp should not be null."); auto lr_dims = ctx->GetInputDim("LearningRate"); PADDLE_ENFORCE_EQ(framework::product(lr_dims), 1, @@ -64,7 +62,6 @@ class AdamaxOp : public framework::OperatorWithKernel { ctx->SetOutputDim("ParamOut", param_dims); ctx->SetOutputDim("MomentOut", param_dims); ctx->SetOutputDim("InfNormOut", param_dims); - ctx->SetOutputDim("Beta1PowOut", beta1_pow_dims); } }; @@ -86,7 +83,6 @@ class AdamaxOpMaker : public framework::OpProtoAndCheckerMaker { AddOutput("InfNormOut", "(Tensor) " "Output exponentially weighted infinity norm"); - AddOutput("Beta1PowOut", "(Tensor) Output beta1 power accumulator"); AddAttr("beta1", "(float, default 0.9) " @@ -113,8 +109,7 @@ Adamax updates: moment_out = beta1 * moment + (1 - beta1) * grad inf_norm_out = max(beta2 * inf_norm + epsilon, abs(grad)) -beta1_pow_out = beta1_pow * beta1 -learning_rate_t = learning_rate/(1 - beta1_pow_out) +learning_rate_t = learning_rate/(1 - beta1_pow) param_out = param - learning_rate_t * moment_out/inf_norm_out The original paper does not have an epsilon attribute. diff --git a/paddle/operators/adamax_op.h b/paddle/operators/adamax_op.h index 9677b1bb78..2c99832ec0 100644 --- a/paddle/operators/adamax_op.h +++ b/paddle/operators/adamax_op.h @@ -26,12 +26,10 @@ class AdamaxOpKernel : public framework::OpKernel { auto param_out_tensor = ctx.Output("ParamOut"); auto moment_out_tensor = ctx.Output("MomentOut"); auto inf_norm_out_tensor = ctx.Output("InfNormOut"); - auto beta1_pow_out_tensor = ctx.Output("Beta1PowOut"); param_out_tensor->mutable_data(ctx.GetPlace()); moment_out_tensor->mutable_data(ctx.GetPlace()); inf_norm_out_tensor->mutable_data(ctx.GetPlace()); - beta1_pow_out_tensor->mutable_data(ctx.GetPlace()); float beta1 = ctx.Attr("beta1"); float beta2 = ctx.Attr("beta2"); @@ -53,15 +51,12 @@ class AdamaxOpKernel : public framework::OpKernel { auto moment_out = framework::EigenVector::Flatten(*moment_out_tensor); auto inf_norm_out = framework::EigenVector::Flatten(*inf_norm_out_tensor); - auto beta1_pow_out = - framework::EigenVector::Flatten(*beta1_pow_out_tensor); auto place = ctx.GetEigenDevice(); moment_out.device(place) = beta1 * moment + (1 - beta1) * grad; inf_norm_out.device(place) = grad.abs().cwiseMax((beta2 * inf_norm) + epsilon); - beta1_pow_out.device(place) = beta1_pow * beta1; - auto lr_t = lr / (1 - beta1_pow_out); + auto lr_t = lr / (1 - beta1_pow); Eigen::DSizes m_dsize(moment_out_tensor->numel()); param_out.device(place) = param - lr_t.broadcast(m_dsize) * (moment_out / inf_norm_out); diff --git a/python/paddle/v2/framework/tests/test_adamax_op.py b/python/paddle/v2/framework/tests/test_adamax_op.py index af81075d6a..8e5a15aa3d 100644 --- a/python/paddle/v2/framework/tests/test_adamax_op.py +++ b/python/paddle/v2/framework/tests/test_adamax_op.py @@ -31,14 +31,13 @@ class TestAdamaxOp1(OpTest): self.attrs = {'beta1': beta1, 'beta2': beta2, 'epsilon': epsilon} - param_out, moment_out, inf_norm_out, beta1_pow_out = adamax_step( - self.inputs, self.attrs) + param_out, moment_out, inf_norm_out = adamax_step(self.inputs, + self.attrs) self.outputs = { 'ParamOut': param_out, 'MomentOut': moment_out, - 'InfNormOut': inf_norm_out, - 'Beta1PowOut': beta1_pow_out + 'InfNormOut': inf_norm_out } def test_check_output(self): @@ -73,14 +72,12 @@ class TestAdamaxOp2(OpTest): } attrs = {'beta1': beta1, 'beta2': beta2, 'epsilon': epsilon} - param_out, moment_out, inf_norm_out, beta1_pow_out = adamax_step( - self.inputs, attrs) + param_out, moment_out, inf_norm_out = adamax_step(self.inputs, attrs) self.outputs = { 'ParamOut': param_out, 'MomentOut': moment_out, - 'InfNormOut': inf_norm_out, - 'Beta1PowOut': beta1_pow_out + 'InfNormOut': inf_norm_out } def test_check_output(self): @@ -117,19 +114,15 @@ class TestAdamaxOpMultipleSteps(OpTest): self.attrs = {'beta1': beta1, 'beta2': beta2, 'epsilon': epsilon} - param_out, moment_out, inf_norm_out, beta1_pow_out = adamax_step( - self.inputs, self.attrs) - def test_check_output(self): for _ in range(self.num_steps): - param_out, moment_out, inf_norm_out, beta1_pow_out = adamax_step( - self.inputs, self.attrs) + param_out, moment_out, inf_norm_out = adamax_step(self.inputs, + self.attrs) self.outputs = { 'ParamOut': param_out, 'MomentOut': moment_out, - 'InfNormOut': inf_norm_out, - 'Beta1PowOut': beta1_pow_out + 'InfNormOut': inf_norm_out } # Verify output for this step @@ -139,7 +132,9 @@ class TestAdamaxOpMultipleSteps(OpTest): self.inputs['Param'] = param_out self.inputs['Moment'] = moment_out self.inputs['InfNorm'] = inf_norm_out - self.inputs['Beta1Pow'] = beta1_pow_out + + # Update Beta1 Power accumulator for next step + self.inputs['Beta1Pow'] *= self.attrs['beta1'] # Randomize gradient for next step self.inputs['Grad'] = np.random.uniform( @@ -167,11 +162,10 @@ def adamax_step(inputs, attributes): moment_out = beta1 * moment + (1 - beta1) * grad inf_norm_out = np.maximum(beta2 * inf_norm + epsilon, np.abs(grad)) - beta1_pow_out = beta1_pow * beta1 - lr_t = (lr / (1 - beta1_pow_out)) + lr_t = (lr / (1 - beta1_pow)) param_out = param - lr_t * np.divide(moment_out, inf_norm_out) - return param_out, moment_out, inf_norm_out, beta1_pow_out + return param_out, moment_out, inf_norm_out if __name__ == "__main__": From 1f1be6c97a4d9f93a39bf126ed1c12d9cac15517 Mon Sep 17 00:00:00 2001 From: fengjiayi Date: Thu, 19 Oct 2017 12:02:02 -0700 Subject: [PATCH 45/76] Test recognize_digits_conv (#4926) * Init * unify layer names * Update * Add pool2d layer * Test recognize_digits_conv * Clean up --- python/paddle/v2/framework/framework.py | 4 +- python/paddle/v2/framework/layers.py | 52 +++++++++++++++++-- python/paddle/v2/framework/nets.py | 24 +++++++++ .../paddle/v2/framework/tests/test_layers.py | 51 ++++++++++++++---- 4 files changed, 116 insertions(+), 15 deletions(-) create mode 100644 python/paddle/v2/framework/nets.py diff --git a/python/paddle/v2/framework/framework.py b/python/paddle/v2/framework/framework.py index a68f2afcfa..622e09fdde 100644 --- a/python/paddle/v2/framework/framework.py +++ b/python/paddle/v2/framework/framework.py @@ -432,11 +432,13 @@ class Program(object): def current_block(self): return self.blocks[self.current_block_idx] - def append_backward(self, target, no_grad_set): + def append_backward(self, target, no_grad_set=None): """ return map(param_name -> (grad_name, block_index, op_index)) """ assert isinstance(target, Variable) + if no_grad_set is None: + no_grad_set = set() param_to_grad_info = self.desc.append_backward(target.desc, no_grad_set) self.sync_with_cpp() return param_to_grad_info diff --git a/python/paddle/v2/framework/layers.py b/python/paddle/v2/framework/layers.py index 329a6830b6..236427efce 100644 --- a/python/paddle/v2/framework/layers.py +++ b/python/paddle/v2/framework/layers.py @@ -3,7 +3,7 @@ import paddle.v2.framework.core as core from paddle.v2.framework.framework import OpProtoHolder, Variable import re -__all__ = ['fc', 'data', 'cross_entropy', 'conv2d'] +__all__ = ['fc', 'data', 'cross_entropy', 'conv2d', 'pool2d'] def fc(input, @@ -35,7 +35,10 @@ def fc(input, "Y": w, }, outputs={"Out": tmp}, - attrs={'x_num_col_dims': num_flatten_dims}) + attrs={ + 'x_num_col_dims': num_flatten_dims, + 'y_num_col_dims': len(input_shape) - num_flatten_dims + }) mul_results.append(tmp) # sum @@ -115,7 +118,6 @@ def _create_op_func_(op_type): _create_op_func_('mean') _create_op_func_('mul') -_create_op_func_('pool2d') def cross_entropy(input, label, **kwargs): @@ -170,6 +172,13 @@ def conv2d(input, raise ValueError("num_channels must be divisible by groups.") num_filter_channels = num_channels / groups + if isinstance(filter_size, int): + filter_size = [filter_size, filter_size] + if isinstance(stride, int): + stride = [stride, stride] + if isinstance(padding, int): + padding = [padding, padding] + input_shape = input.shape filter_shape = [num_filters, num_filter_channels] + filter_size filter = helper.create_parameter( @@ -190,3 +199,40 @@ def conv2d(input, pre_act = helper.append_bias_op(pre_bias) return helper.append_activation(pre_act) + + +def pool2d(input, + pool_size, + pool_type, + pool_stride=[1, 1], + pool_padding=[0, 0], + global_pooling=False, + program=None): + if pool_type not in ["max", "avg"]: + raise ValueError( + "Unknown pool_type: '%s'. It can only be 'max' or 'avg'.", + str(pool_type)) + if isinstance(pool_size, int): + pool_size = [pool_size, pool_size] + if isinstance(pool_stride, int): + pool_stride = [pool_stride, pool_stride] + if isinstance(pool_padding, int): + pool_padding = [pool_padding, pool_padding] + + helper = LayerHelper('conv2d', **locals()) + dtype = helper.input_dtype() + pool_out = helper.create_tmp_variable(dtype) + + helper.append_op( + type="pool2d", + inputs={"X": input}, + outputs={"Out": pool_out}, + attrs={ + "pooling_type": pool_type, + "ksize": pool_size, + "global_pooling": global_pooling, + "strides": pool_stride, + "paddings": pool_padding + }) + + return pool_out diff --git a/python/paddle/v2/framework/nets.py b/python/paddle/v2/framework/nets.py new file mode 100644 index 0000000000..381da55da3 --- /dev/null +++ b/python/paddle/v2/framework/nets.py @@ -0,0 +1,24 @@ +import paddle.v2.framework.layers as layers + + +def simple_img_conv_pool(input, + filter_size, + num_filters, + pool_size, + pool_stride, + act, + program=None): + conv_out = layers.conv2d( + input=input, + num_filters=num_filters, + filter_size=filter_size, + act=act, + program=program) + + pool_out = layers.pool2d( + input=conv_out, + pool_size=pool_size, + pool_type='max', + pool_stride=pool_stride, + program=program) + return pool_out diff --git a/python/paddle/v2/framework/tests/test_layers.py b/python/paddle/v2/framework/tests/test_layers.py index dbbb653538..4ecc02b12d 100644 --- a/python/paddle/v2/framework/tests/test_layers.py +++ b/python/paddle/v2/framework/tests/test_layers.py @@ -1,4 +1,5 @@ import paddle.v2.framework.layers as layers +import paddle.v2.framework.nets as nets from paddle.v2.framework.framework import Program, g_program import paddle.v2.framework.core as core import unittest @@ -18,7 +19,7 @@ class TestBook(unittest.TestCase): avg_cost = layers.mean(x=cost, program=program) self.assertIsNotNone(avg_cost) - program.append_backward(avg_cost, set()) + program.append_backward(avg_cost) print str(program) def test_recognize_digits_mlp(self): @@ -38,24 +39,52 @@ class TestBook(unittest.TestCase): cost = layers.cross_entropy(input=predict, label=label, program=program) avg_cost = layers.mean(x=cost, program=program) self.assertIsNotNone(avg_cost) - # print str(program) + print str(program) def test_simple_conv2d(self): - pd = core.ProgramDesc.__create_program_desc__() - program = Program(desc=pd) - images = data_layer( + program = Program() + images = layers.data( name='pixel', shape=[3, 48, 48], data_type='int32', program=program) - conv2d_layer( + layers.conv2d( input=images, num_filters=3, filter_size=[4, 4], program=program) - # print str(program) + print str(program) - def test_simple_conv2d(self): + def test_recognize_digits_conv(self): program = Program() + images = layers.data( - name='pixel', shape=[3, 48, 48], data_type='int32', program=program) - layers.conv2d( - input=images, num_filters=3, filter_size=[4, 4], program=program) + name='pixel', + shape=[1, 28, 28], + data_type='float32', + program=program) + label = layers.data( + name='label', shape=[1], data_type='int32', program=program) + conv_pool_1 = nets.simple_img_conv_pool( + input=images, + filter_size=5, + num_filters=2, + pool_size=2, + pool_stride=2, + act="relu", + program=program) + conv_pool_2 = nets.simple_img_conv_pool( + input=conv_pool_1, + filter_size=5, + num_filters=4, + pool_size=2, + pool_stride=2, + act="relu", + program=program) + + predict = layers.fc(input=conv_pool_2, + size=10, + act="softmax", + program=program) + cost = layers.cross_entropy(input=predict, label=label, program=program) + avg_cost = layers.mean(x=cost, program=program) + + program.append_backward(avg_cost) print str(program) From 43702a89d5b5311281ef92be40d1e1ce9a88abab Mon Sep 17 00:00:00 2001 From: Abhinav Arora Date: Thu, 19 Oct 2017 14:18:40 -0700 Subject: [PATCH 46/76] Correcting some grammatical mistakes in register_grad_op.md (#4938) --- doc/design/register_grad_op.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/doc/design/register_grad_op.md b/doc/design/register_grad_op.md index 9f1ce4bae7..8d973eb531 100644 --- a/doc/design/register_grad_op.md +++ b/doc/design/register_grad_op.md @@ -3,17 +3,17 @@ ## The Problem Posed -Currently, for each C++ operator class definition, there registers a *gradient operator creator* function, which takes a C++ operator instance and returns the corresponding gradient operator instance. +Currently, for each C++ operator class definition, a *gradient operator creator* function is registered, which takes as input a C++ operator instance and returns the corresponding gradient operator instance. -However, we noticed two problems with the current deisgn: +However, we noticed two problems with the current design: -1. As we decided to separate the *compilation* and *execution* phases, we need to change the creator to take an `OpDesc` protobuf message in a `ProgramDesc` and inserts corresponding `OpDesc` messages into the `ProgramDesc` message. +1. As we decided to separate the *compilation* and the *execution* phases, we need to change the creator to take an `OpDesc` protobuf message in a `ProgramDesc` and inserts corresponding `OpDesc` messages into the `ProgramDesc` message. -1. Some operator's gradient computation requires more than one gradient operators. For example, the gradient of *minus* consists of two operators -- an identity operaotr and a scale operator. So we need to make the registration mechanism to support the mapping from an operator to a set of operators for gradient computation. +1. For some operators, the gradient computation can be written in terms of existing operators. For example, the gradient of *minus* operator consists of two operators -- an *identity* operator followed by a *scale* operator. Hence the registration mechanism needs to support mapping from an operator to a set of operators for the gradient computation. ## The Current Implementation -The C++ class `OpInfos` store in a association map which key is the operator type. The `grad_op_type` indicate associated gradient operator type. Operator can create gradient operator by `OpInfo::creator_` of gradient. The pseudo code is +Instances of the C++ class `OpInfo` are stored an associative map whose key is the operator type. The `grad_op_type` indicates the associated gradient operator type. An operator can create the gradient operator by invoking `OpInfo::creator_` of the gradient operator. The pseudo code is as follows ```cpp struct OpInfo { @@ -31,16 +31,16 @@ OperatorBase* CreateGradientOperator(const OperatorBase& op) { ## Proposed Solution -The mapping relationship between an operator and its gradient operators is a function. The interface of that function is: +The mapping relationship between an operator and its gradient operators is a function. The interface of this function is: ```cpp // (OpDesc) --> vector std::function(const OpDescBind&)>; ``` -The function takes an `OpDescBind` of the forward operator and returns one or many gradient operator descriptions. `OpDescBind` is a C++ wrapper for protobuf message `OpDesc` to manipulate `OpDesc` fast. +The function takes an `OpDescBind` of the forward operator and returns one or many gradient operator descriptions. `OpDescBind` is a C++ wrapper for the protobuf message `OpDesc` for rapid manipulation of `OpDesc`. -The `GradOpDescMaker` will be registered in `OpInfo`, to replace `grad_op_type_` field. The `OpInfo` should be +The `GradOpDescMaker` will be registered in `OpInfo` and will replace the `grad_op_type_` field. The `OpInfo` should look like ```cpp struct OpInfo { @@ -49,7 +49,7 @@ struct OpInfo { }; ``` -The `grad_op_maker_ ` is `nullptr` if the operator does not have associated gradient operators. +The `grad_op_maker_ ` is a `nullptr` if the operator does not have any associated gradient operators. We propose a base class called `GradOpDescMakerBase` to let operator developers generate `Gradient Operators` easily. The public interface of that class is @@ -74,7 +74,7 @@ func = [] (const OpDescBind& fwd_op) { We can write many helper functions since the `GradOpDescMakerBase` is a class now. The basic helper functions get the variables of `Input`, `Output`, `InputGradient` and `OutputGradient` in the forwarding operator. -We should chagne register macros at the same time. In the current solution, there is no difference between forwarding operators and backward operators. So `REGISTER_OP` just register one operator. If the `REGISTER_OPERATOR ` contains `OpProtoAndCheckerMaker` and `GradOpDescMaker`, we just list them in the same macro. It can be done by a macro contains `__VA_ARGS__`. +We should change register macros at the same time. In the current solution, there is no difference between forwarding operators and backward operators. So `REGISTER_OP` just register one operator. If the `REGISTER_OPERATOR ` contains `OpProtoAndCheckerMaker` and `GradOpDescMaker`, we just list them in the same macro. It can be done by a macro contains `__VA_ARGS__`. The user interface should be From 9e6404441c58be4c8b457f2152d7e5ee039e9cec Mon Sep 17 00:00:00 2001 From: qijun Date: Thu, 19 Oct 2017 14:49:21 -0700 Subject: [PATCH 47/76] fix elementwise add bug --- paddle/operators/elementwise_op_function.h | 8 +----- .../tests/test_elementwise_add_op.py | 28 +++++++++++++++++++ 2 files changed, 29 insertions(+), 7 deletions(-) diff --git a/paddle/operators/elementwise_op_function.h b/paddle/operators/elementwise_op_function.h index 3eb97f60b5..488a35aafc 100644 --- a/paddle/operators/elementwise_op_function.h +++ b/paddle/operators/elementwise_op_function.h @@ -108,7 +108,7 @@ void ElementwiseCompute(const framework::ExecutionContext& ctx) { PADDLE_ENFORCE_GE(x_dims.size(), y_dims.size(), "Rank of first input must >= rank of second input.") - if (x_dims == y_dims || product(y_dims) == 1) { + if (x_dims == y_dims) { functor f; f.template Run(x, y, z, ctx); return; @@ -174,12 +174,6 @@ void ElementwiseGradCompute(const framework::ExecutionContext& ctx) { return; } - if (product(y_dims) == 1) { - functor1 f; - f(place, x, y, out, dx, dy, dout); - return; - } - int axis = ctx.Attr("axis"); axis = (axis == -1 ? x_dims.size() - y_dims.size() : axis); diff --git a/python/paddle/v2/framework/tests/test_elementwise_add_op.py b/python/paddle/v2/framework/tests/test_elementwise_add_op.py index f3101a709b..57daddd569 100644 --- a/python/paddle/v2/framework/tests/test_elementwise_add_op.py +++ b/python/paddle/v2/framework/tests/test_elementwise_add_op.py @@ -92,5 +92,33 @@ class TestElementwiseAddOp_broadcast_3(TestElementwiseOp): } +class TestElementwiseAddOp_rowwise_add_0(TestElementwiseOp): + def setUp(self): + self.op_type = "elementwise_add" + self.inputs = { + 'X': np.random.rand(2, 3, 4).astype(np.float32), + 'Y': np.random.rand(3, 4).astype(np.float32) + } + + self.attrs = {'axis': 1} + self.outputs = { + 'Out': self.inputs['X'] + self.inputs['Y'].reshape(1, 3, 4) + } + + +class TestElementwiseAddOp_rowwise_add_1(TestElementwiseOp): + def setUp(self): + self.op_type = "elementwise_add" + self.inputs = { + 'X': np.random.rand(2, 1).astype(np.float32), + 'Y': np.random.rand(1).astype(np.float32) + } + + self.attrs = {'axis': 1} + self.outputs = { + 'Out': self.inputs['X'] + self.inputs['Y'].reshape(1, 1) + } + + if __name__ == '__main__': unittest.main() From c532b967411b9e4aa89ebb5878a0a44e7f117431 Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Thu, 19 Oct 2017 16:03:53 -0700 Subject: [PATCH 48/76] Remove template parameter for Tensor methods (#4937) * Remove template parameter for Tensor methods * Also check the type is correct when data() * Simplize holder_ * Fix accuracy_op * Register Code --- paddle/framework/data_type.h | 2 + paddle/framework/feed_fetch_method.h | 2 +- paddle/framework/tensor.h | 46 ++++---- paddle/framework/tensor_array.cc | 34 +++--- paddle/framework/tensor_array_test.cc | 2 +- paddle/framework/tensor_impl.h | 100 ++++++++++++++---- paddle/framework/tensor_test.cc | 32 +++--- paddle/operators/accuracy_op.cc | 7 +- paddle/operators/accuracy_op.cu | 18 ++-- paddle/operators/conv2d_op.h | 32 +++--- paddle/operators/dynamic_recurrent_op.cc | 15 ++- paddle/operators/feed_op.cc | 2 +- paddle/operators/fetch_op.cc | 2 +- paddle/operators/math/im2col_test.cc | 8 +- paddle/operators/math/math_function_test.cu | 28 ++--- .../math/selected_rows_functor_test.cu | 4 +- paddle/operators/math/vol2col_test.cc | 8 +- paddle/operators/matmul_op.h | 6 +- paddle/operators/mul_op.h | 28 ++--- paddle/operators/multiplex_op.cu | 6 +- paddle/operators/recurrent_op.cc | 4 +- paddle/operators/reshape_op.h | 4 +- paddle/operators/rnn/recurrent_op_utils.cc | 8 +- paddle/operators/scatter_op.cu | 4 +- paddle/operators/scatter_op.h | 4 +- paddle/operators/sequence_concat_op.h | 16 +-- paddle/operators/sequence_pool_op.h | 12 +-- paddle/operators/sequence_softmax_op.h | 10 +- .../softmax_with_cross_entropy_op.cu | 2 +- .../operators/softmax_with_cross_entropy_op.h | 2 +- paddle/pybind/pybind.cc | 2 + 31 files changed, 248 insertions(+), 202 deletions(-) diff --git a/paddle/framework/data_type.h b/paddle/framework/data_type.h index 649899d425..c25a62c2b1 100644 --- a/paddle/framework/data_type.h +++ b/paddle/framework/data_type.h @@ -26,6 +26,8 @@ inline DataType ToDataType(std::type_index type) { return DataType::FP64; } else if (typeid(int).hash_code() == type.hash_code()) { return DataType::INT32; + } else if (typeid(int64_t).hash_code() == type.hash_code()) { + return DataType::INT64; } else { PADDLE_THROW("Not supported"); } diff --git a/paddle/framework/feed_fetch_method.h b/paddle/framework/feed_fetch_method.h index 9b23ad271c..d58736dcb1 100644 --- a/paddle/framework/feed_fetch_method.h +++ b/paddle/framework/feed_fetch_method.h @@ -34,7 +34,7 @@ void SetFeedVariable(const LoDTensor& input, const std::string& var_name, feed_inputs.resize(index + 1); } // shared data with input tensor - feed_inputs[index].ShareDataWith(input); + feed_inputs[index].ShareDataWith(input); // set lod feed_inputs[index].set_lod(input.lod()); } diff --git a/paddle/framework/tensor.h b/paddle/framework/tensor.h index bc430852de..3a2bdaf086 100644 --- a/paddle/framework/tensor.h +++ b/paddle/framework/tensor.h @@ -60,6 +60,10 @@ class Tensor { template inline T* mutable_data(platform::Place place); + inline void* mutable_data(platform::Place place, std::type_index type); + + inline void* mutable_data(platform::Place place); + /** * @brief Return a pointer to mutable memory block. * @@ -81,7 +85,6 @@ class Tensor { inline Tensor& Resize(const DDim& dims); /*! The internal of two tensors share the same memory block. */ - template inline Tensor& ShareDataWith(const Tensor& src); /** @@ -96,26 +99,9 @@ class Tensor { // TODO(qijun): https://github.com/PaddlePaddle/Paddle/issues/4647 // Remove `CopyFrom` and `CopyFromVector` from Tensor interface // and make them global functions - template inline void CopyFrom(const Tensor& src, const platform::Place& dst_place, const platform::DeviceContext& ctx); - // FIXME(yuyang18): CopyFrom should without template T, use the replace - // `CopyFrom` with `CopyFromTensor` - inline void CopyFromTensor(const Tensor& src, - const platform::Place& dst_place, - const platform::DeviceContext& ctx) { - // NOLINTNEXTLINES_8 cpplint.py will recognize below lines as functions. - // That is a bug of cpplint.py. Just ignore lint these lines. - if (src.type() == std::type_index(typeid(double))) { - CopyFrom(src, dst_place, ctx); - } else if (src.type() == std::type_index(typeid(float))) { - CopyFrom(src, dst_place, ctx); - } else if (src.type() == std::type_index(typeid(int))) { - CopyFrom(src, dst_place, ctx); - } - } - /** * @brief Copy the content of an external vector to a tensor. * @@ -135,7 +121,6 @@ class Tensor { * @param[in] begin_idx The begin index of the slice. * @param[in] end_idx The end index of the slice. */ - template inline Tensor Slice(const int& begin_idx, const int& end_idx) const; platform::Place place() const { @@ -146,7 +131,6 @@ class Tensor { std::type_index type() const { return holder_->type(); } private: - template inline void check_memory_size() const; private: @@ -155,20 +139,22 @@ class Tensor { * parameter of Variable. */ struct Placeholder { - virtual ~Placeholder() {} + virtual ~Placeholder() = default; virtual void* ptr() const = 0; virtual size_t size() const = 0; virtual std::type_index type() const = 0; virtual platform::Place place() const = 0; + virtual void set_type(std::type_index type) = 0; }; - template + template struct PlaceholderImpl : public Placeholder { - PlaceholderImpl(Place place, size_t size) - : ptr_(static_cast(memory::Alloc(place, size)), - memory::PODDeleter(place)), + PlaceholderImpl(Place place, size_t size, std::type_index type) + : ptr_(static_cast(memory::Alloc(place, size)), + memory::PODDeleter(place)), place_(place), - size_(size) { + size_(size), + type_(type) { PADDLE_ENFORCE_NOT_NULL(ptr_, "Insufficient %s memory to allocation.", (is_cpu_place(place_) ? "CPU" : "GPU")); } @@ -176,16 +162,20 @@ class Tensor { virtual size_t size() const { return size_; } virtual platform::Place place() const { return place_; } virtual void* ptr() const { return static_cast(ptr_.get()); } - virtual std::type_index type() const { return std::type_index(typeid(T)); } + virtual std::type_index type() const { return type_; } + virtual void set_type(std::type_index type) { type_ = type; } /*! the pointer of memory block. */ - std::unique_ptr> ptr_; + std::unique_ptr> ptr_; /*! the place of memory block. */ platform::Place place_; /*! the size of memory block. */ size_t size_; + + /* the current type of memory */ + std::type_index type_; }; /*! holds the memory block if allocated. */ diff --git a/paddle/framework/tensor_array.cc b/paddle/framework/tensor_array.cc index 06459cbfd7..4c82c36383 100644 --- a/paddle/framework/tensor_array.cc +++ b/paddle/framework/tensor_array.cc @@ -106,8 +106,8 @@ void TensorArray::Write(size_t index, const LoDTensor& value) { values_[index].Resize(value.dims()); values_[index].mutable_data(platform::CPUPlace()); - values_[index].CopyFrom(value, platform::CPUPlace(), - platform::CPUDeviceContext()); + values_[index].CopyFrom(value, platform::CPUPlace(), + platform::CPUDeviceContext()); } void TensorArray::WriteShared(size_t index, const LoDTensor& value) { @@ -116,7 +116,7 @@ void TensorArray::WriteShared(size_t index, const LoDTensor& value) { values_.resize(index + 1); } - values_[index].ShareDataWith(value); + values_[index].ShareDataWith(value); } LoDTensor TensorArray::Pack(size_t level, const std::vector& meta, @@ -163,9 +163,9 @@ LoDTensor TensorArray::Stack() const { result.mutable_data(platform::CPUPlace()); for (size_t idx = 0; idx < size(); idx++) { - result.Slice(idx, idx + 1) - .CopyFrom(Read(idx), platform::CPUPlace(), - platform::CPUDeviceContext()); + result.Slice(idx, idx + 1) + .CopyFrom(Read(idx), platform::CPUPlace(), + platform::CPUDeviceContext()); } return result; } @@ -191,13 +191,12 @@ void TensorArray::Unstack(const LoDTensor& source, bool data_shared) const { auto& value = values_[elem]; if (data_shared) { // share memory - value.ShareDataWith(source.Slice(elem, elem + 1)); + value.ShareDataWith(source.Slice(elem, elem + 1)); } else { // copy value.Resize(value_dims); - value.CopyFrom(source.Slice(elem, elem + 1), - platform::CPUPlace(), - platform::CPUDeviceContext()); + value.CopyFrom(source.Slice(elem, elem + 1), platform::CPUPlace(), + platform::CPUDeviceContext()); } } } @@ -242,11 +241,10 @@ LoDTensor DynamicBatchUnpacker::GetBatch(size_t index) { for (size_t i = 0; i < indice.size(); i++) { auto index = indice[i]; - auto target = result.Slice(i, i + 1); - auto slice = source->Slice(index, index + 1); + auto target = result.Slice(i, i + 1); + auto slice = source->Slice(index, index + 1); - target.CopyFrom(slice, platform::CPUPlace(), - platform::CPUDeviceContext()); + target.CopyFrom(slice, platform::CPUPlace(), platform::CPUDeviceContext()); } return result; @@ -277,10 +275,10 @@ LoDTensor PackDynamicBatch(const std::vector& source, // target is result[index] auto index = seq_meta.begin + batch_id; if (index >= seq_meta.end) break; - auto source_ = source[batch_id].Slice(seq_id, seq_id + 1); - auto target = result.Slice(index, index + 1); - target.CopyFrom(source_, platform::CPUPlace(), - platform::CPUDeviceContext()); + auto source_ = source[batch_id].Slice(seq_id, seq_id + 1); + auto target = result.Slice(index, index + 1); + target.CopyFrom(source_, platform::CPUPlace(), + platform::CPUDeviceContext()); } } diff --git a/paddle/framework/tensor_array_test.cc b/paddle/framework/tensor_array_test.cc index d9f52509cd..9470ac5e6e 100644 --- a/paddle/framework/tensor_array_test.cc +++ b/paddle/framework/tensor_array_test.cc @@ -91,7 +91,7 @@ class TensorArrayPackTester : public ::testing::Test { size_t begin = level[i]; size_t end = level[i + 1]; for (size_t j = begin; j < end; j++) { - auto record = source.Slice(j, j + 1); + auto record = source.Slice(j, j + 1); for (int dim = 0; dim < 128; dim++) { record.mutable_data(platform::CPUPlace())[dim] = j - begin; } diff --git a/paddle/framework/tensor_impl.h b/paddle/framework/tensor_impl.h index ce73e0a9ed..f6e801bbb4 100644 --- a/paddle/framework/tensor_impl.h +++ b/paddle/framework/tensor_impl.h @@ -19,12 +19,50 @@ limitations under the License. */ namespace paddle { namespace framework { +template +struct SizeOfTypeFunctor; + template +struct SizeOfTypeFunctor { + size_t operator()(std::type_index type) const { + if (typeid(T).hash_code() == type.hash_code()) { + return sizeof(T); + } else { + return 0UL; + } + } +}; + +template <> +struct SizeOfTypeFunctor<> { + size_t operator()(std::type_index type) const { return 0UL; } +}; + +template +struct SizeOfTypeFunctor { + size_t operator()(std::type_index type) const { + SizeOfTypeFunctor head; + size_t head_size = head(type); + if (head_size != 0) { + return head_size; + } + SizeOfTypeFunctor tail; + return tail(type); + } +}; + +static inline size_t SizeOfType(std::type_index type) { + SizeOfTypeFunctor functor; + size_t size = functor(type); + PADDLE_ENFORCE(size != 0UL, "Cannot get size of type %s", type.name()); + return size; +} + inline void Tensor::check_memory_size() const { PADDLE_ENFORCE_NOT_NULL( holder_, "Tensor holds no memory. Call Tensor::mutable_data first."); PADDLE_ENFORCE_GE( - holder_->size(), numel() * sizeof(T) + offset_, + holder_->size(), numel() * SizeOfType(type()) + offset_, "Tensor's dims_ is out of bound. Call Tensor::mutable_data " "first to re-allocate memory.\n" "or maybe the required data-type mismatches the data already stored."); @@ -32,14 +70,23 @@ inline void Tensor::check_memory_size() const { template inline const T* Tensor::data() const { - check_memory_size(); + check_memory_size(); + PADDLE_ENFORCE(std::is_same::value || + holder_->type().hash_code() == typeid(T).hash_code(), + "Tensor holds the wrong type, it holds %s", + this->holder_->type().name()); + return reinterpret_cast( reinterpret_cast(holder_->ptr()) + offset_); } template inline T* Tensor::data() { - check_memory_size(); + check_memory_size(); + PADDLE_ENFORCE(std::is_same::value || + holder_->type().hash_code() == typeid(T).hash_code(), + "Tensor holds the wrong type, it holds %s", + this->holder_->type().name()); return reinterpret_cast(reinterpret_cast(holder_->ptr()) + offset_); } @@ -54,51 +101,62 @@ inline T* Tensor::mutable_data(DDim dims, platform::Place place) { template inline T* Tensor::mutable_data(platform::Place place) { static_assert(std::is_pod::value, "T must be POD"); + return reinterpret_cast(mutable_data(place, typeid(T))); +} + +inline void* Tensor::mutable_data(platform::Place place, std::type_index type) { + if (holder_ != nullptr) { + holder_->set_type(type); + } PADDLE_ENFORCE_GT(numel(), 0, "Tensor's numel must be larger than zero to call " "Tensor::mutable_data. Call Tensor::set_dim first."); + int64_t size = numel() * SizeOfType(type); /* some versions of boost::variant don't have operator!= */ - int64_t size = numel() * sizeof(T); if (holder_ == nullptr || !(holder_->place() == place) || holder_->size() < size + offset_) { if (platform::is_cpu_place(place)) { - holder_.reset(new PlaceholderImpl( - boost::get(place), size)); + holder_.reset(new PlaceholderImpl( + boost::get(place), size, type)); } else if (platform::is_gpu_place(place)) { #ifndef PADDLE_WITH_CUDA PADDLE_THROW("'GPUPlace' is not supported in CPU only device."); } #else - holder_.reset(new PlaceholderImpl( - boost::get(place), size)); + holder_.reset(new PlaceholderImpl( + boost::get(place), size, type)); } #endif offset_ = 0; } - return reinterpret_cast(reinterpret_cast(holder_->ptr()) + - offset_); + return reinterpret_cast(reinterpret_cast(holder_->ptr()) + + offset_); +} + +inline void* Tensor::mutable_data(platform::Place place) { + PADDLE_ENFORCE(this->holder_ != nullptr, + "Cannot invoke mutable data if current hold nothing"); + return mutable_data(place, holder_->type()); } -template inline Tensor& Tensor::ShareDataWith(const Tensor& src) { - src.check_memory_size(); + src.check_memory_size(); *this = src; return *this; } -template inline void Tensor::CopyFrom(const Tensor& src, const platform::Place& dst_place, const platform::DeviceContext& ctx) { - src.check_memory_size(); + src.check_memory_size(); Resize(src.dims()); auto src_place = src.holder_->place(); - auto src_ptr = static_cast(src.data()); + auto src_ptr = src.data(); - auto dst_ptr = static_cast(mutable_data(dst_place)); + auto dst_ptr = mutable_data(dst_place, src.type()); - auto size = src.numel() * sizeof(T); + auto size = src.numel() * SizeOfType(src.type()); if (platform::is_cpu_place(src_place) && platform::is_cpu_place(dst_place)) { memory::Copy(boost::get(dst_place), dst_ptr, @@ -165,9 +223,8 @@ inline void Tensor::CopyFromVector(const std::vector& src, #endif } -template inline Tensor Tensor::Slice(const int& begin_idx, const int& end_idx) const { - check_memory_size(); + check_memory_size(); PADDLE_ENFORCE_GE(begin_idx, 0, "Slice begin index is less than zero."); PADDLE_ENFORCE_LE(end_idx, dims_[0], "Slice end index is out of bound."); PADDLE_ENFORCE_LT(begin_idx, end_idx, @@ -182,7 +239,7 @@ inline Tensor Tensor::Slice(const int& begin_idx, const int& end_idx) const { DDim dst_dims = dims_; dst_dims[0] = end_idx - begin_idx; dst.Resize(dst_dims); - dst.offset_ = offset_ + begin_idx * base * sizeof(T); + dst.offset_ = offset_ + begin_idx * base * SizeOfType(type()); return dst; } } @@ -196,10 +253,9 @@ inline const DDim& Tensor::dims() const { return dims_; } inline int64_t Tensor::numel() const { return product(dims_); } -template inline Tensor ReshapeToMatrix(const Tensor& src, int num_col_dims) { Tensor res; - res.ShareDataWith(src); + res.ShareDataWith(src); res.Resize(flatten_to_2d(src.dims(), num_col_dims)); return res; } diff --git a/paddle/framework/tensor_test.cc b/paddle/framework/tensor_test.cc index 0b62fe08ce..1bb0fb71b0 100644 --- a/paddle/framework/tensor_test.cc +++ b/paddle/framework/tensor_test.cc @@ -108,7 +108,7 @@ TEST(Tensor, ShareDataWith) { // Try to share data form uninitialized tensor bool caught = false; try { - dst_tensor.ShareDataWith(src_tensor); + dst_tensor.ShareDataWith(src_tensor); } catch (paddle::platform::EnforceNotMet err) { caught = true; std::string msg = @@ -122,7 +122,7 @@ TEST(Tensor, ShareDataWith) { ASSERT_TRUE(caught); src_tensor.mutable_data(make_ddim({2, 3, 4}), CPUPlace()); - dst_tensor.ShareDataWith(src_tensor); + dst_tensor.ShareDataWith(src_tensor); ASSERT_EQ(src_tensor.data(), dst_tensor.data()); } @@ -131,7 +131,7 @@ TEST(Tensor, ShareDataWith) { Tensor src_tensor; Tensor dst_tensor; src_tensor.mutable_data(make_ddim({2, 3, 4}), GPUPlace()); - dst_tensor.ShareDataWith(src_tensor); + dst_tensor.ShareDataWith(src_tensor); ASSERT_EQ(src_tensor.data(), dst_tensor.data()); } #endif @@ -143,7 +143,7 @@ TEST(Tensor, Slice) { { Tensor src_tensor; src_tensor.mutable_data(make_ddim({5, 3, 4}), CPUPlace()); - Tensor slice_tensor = src_tensor.Slice(1, 3); + Tensor slice_tensor = src_tensor.Slice(1, 3); DDim slice_dims = slice_tensor.dims(); ASSERT_EQ(arity(slice_dims), 3); EXPECT_EQ(slice_dims[0], 2); @@ -167,7 +167,7 @@ TEST(Tensor, Slice) { { Tensor src_tensor; src_tensor.mutable_data(make_ddim({6, 9}), GPUPlace()); - Tensor slice_tensor = src_tensor.Slice(2, 6); + Tensor slice_tensor = src_tensor.Slice(2, 6); DDim slice_dims = slice_tensor.dims(); ASSERT_EQ(arity(slice_dims), 2); EXPECT_EQ(slice_dims[0], 4); @@ -202,7 +202,7 @@ TEST(Tensor, CopyFrom) { memcpy(src_ptr, arr, 9 * sizeof(int)); auto cpu_place = new paddle::platform::CPUPlace(); - dst_tensor.CopyFrom(src_tensor, *cpu_place, cpu_ctx); + dst_tensor.CopyFrom(src_tensor, *cpu_place, cpu_ctx); const int* dst_ptr = dst_tensor.data(); ASSERT_NE(src_ptr, dst_ptr); @@ -210,8 +210,8 @@ TEST(Tensor, CopyFrom) { EXPECT_EQ(src_ptr[i], dst_ptr[i]); } - Tensor slice_tensor = src_tensor.Slice(1, 2); - dst_tensor.CopyFrom(slice_tensor, *cpu_place, cpu_ctx); + Tensor slice_tensor = src_tensor.Slice(1, 2); + dst_tensor.CopyFrom(slice_tensor, *cpu_place, cpu_ctx); const int* slice_ptr = slice_tensor.data(); dst_ptr = dst_tensor.data(); ASSERT_NE(dst_ptr, slice_ptr); @@ -233,11 +233,11 @@ TEST(Tensor, CopyFrom) { // CPU Tensor to GPU Tensor auto gpu_place = new paddle::platform::GPUPlace(0); CUDADeviceContext gpu_ctx(*gpu_place); - gpu_tensor.CopyFrom(src_tensor, *gpu_place, gpu_ctx); + gpu_tensor.CopyFrom(src_tensor, *gpu_place, gpu_ctx); // GPU Tensor to CPU Tensor auto cpu_place = new paddle::platform::CPUPlace(); - dst_tensor.CopyFrom(gpu_tensor, *cpu_place, gpu_ctx); + dst_tensor.CopyFrom(gpu_tensor, *cpu_place, gpu_ctx); // Sync before Compare Tensors gpu_ctx.Wait(); @@ -247,13 +247,13 @@ TEST(Tensor, CopyFrom) { EXPECT_EQ(src_ptr[i], dst_ptr[i]); } - Tensor slice_tensor = src_tensor.Slice(1, 2); + Tensor slice_tensor = src_tensor.Slice(1, 2); // CPU Slice Tensor to GPU Tensor - gpu_tensor.CopyFrom(slice_tensor, *gpu_place, gpu_ctx); + gpu_tensor.CopyFrom(slice_tensor, *gpu_place, gpu_ctx); // GPU Tensor to CPU Tensor - dst_tensor.CopyFrom(gpu_tensor, *cpu_place, gpu_ctx); + dst_tensor.CopyFrom(gpu_tensor, *cpu_place, gpu_ctx); // Sync before Compare Slice Tensors gpu_ctx.Wait(); @@ -320,7 +320,7 @@ TEST(Tensor, CopyFromVector) { CUDADeviceContext gpu_ctx(*gpu_place); gpu_tensor.CopyFromVector(src_vec, gpu_ctx); // Copy from GPU to CPU tensor for comparison - dst_tensor.CopyFrom(gpu_tensor, *cpu_place, gpu_ctx); + dst_tensor.CopyFrom(gpu_tensor, *cpu_place, gpu_ctx); // Sync before Compare Tensors gpu_ctx.Wait(); @@ -340,7 +340,7 @@ TEST(Tensor, CopyFromVector) { cpu_tensor.CopyFromVector(src_vec, cpu_ctx); gpu_tensor.Resize(make_ddim({2, 2})); gpu_tensor.CopyFromVector(src_vec, gpu_ctx); - dst_tensor.CopyFrom(gpu_tensor, *cpu_place, gpu_ctx); + dst_tensor.CopyFrom(gpu_tensor, *cpu_place, gpu_ctx); // Sync before Compare Tensors gpu_ctx.Wait(); @@ -368,7 +368,7 @@ TEST(Tensor, ReshapeToMatrix) { for (int i = 0; i < 2 * 3 * 4 * 9; ++i) { src_ptr[i] = i; } - Tensor res = ReshapeToMatrix(src, 2); + Tensor res = ReshapeToMatrix(src, 2); ASSERT_EQ(res.dims()[0], 2 * 3); ASSERT_EQ(res.dims()[1], 4 * 9); } diff --git a/paddle/operators/accuracy_op.cc b/paddle/operators/accuracy_op.cc index 037bb49abc..e0a00ecaf0 100644 --- a/paddle/operators/accuracy_op.cc +++ b/paddle/operators/accuracy_op.cc @@ -69,5 +69,8 @@ information, or not. But the output only shares the LoD with input `Inference`. namespace ops = paddle::operators; REGISTER_OP_WITHOUT_GRADIENT(accuracy, ops::AccuracyOp, ops::AccuracyOpMaker); -REGISTER_OP_CPU_KERNEL(accuracy, - ops::AccuracyKernel); +REGISTER_OP_CPU_KERNEL( + accuracy, ops::AccuracyKernel, + ops::AccuracyKernel, + ops::AccuracyKernel, + ops::AccuracyKernel); diff --git a/paddle/operators/accuracy_op.cu b/paddle/operators/accuracy_op.cu index 0ca9ef941d..54e6ab99dc 100644 --- a/paddle/operators/accuracy_op.cu +++ b/paddle/operators/accuracy_op.cu @@ -21,9 +21,9 @@ namespace paddle { namespace operators { using platform::PADDLE_CUDA_NUM_THREADS; -template -__global__ void AccuracyCudaKernel(const int N, const int D, const int* Xdata, - const int* labeldata, float* accuracy) { +template +__global__ void AccuracyCudaKernel(const int N, const int D, const T* Xdata, + const T* labeldata, float* accuracy) { int count = 0; __shared__ int total[BlockSize]; @@ -57,8 +57,8 @@ class AccuracyOpCUDAKernel : public framework::OpKernel { auto* accuracy = ctx.Output("Accuracy"); // FIXME(typhoonzero): only support indices currently // if add support for output values, how to detect the data type? - const int* inference_data = inference->data(); - const int* label_data = label->data(); + const T* inference_data = inference->data(); + const T* label_data = label->data(); float* accuracy_data = accuracy->mutable_data(ctx.GetPlace()); size_t num_samples = inference->dims()[0]; @@ -69,7 +69,7 @@ class AccuracyOpCUDAKernel : public framework::OpKernel { return; } - AccuracyCudaKernel<<< + AccuracyCudaKernel<<< 1, PADDLE_CUDA_NUM_THREADS, 0, reinterpret_cast( ctx.device_context()) @@ -81,5 +81,7 @@ class AccuracyOpCUDAKernel : public framework::OpKernel { } // namespace operators } // namespace paddle -REGISTER_OP_GPU_KERNEL(accuracy, - paddle::operators::AccuracyOpCUDAKernel); +REGISTER_OP_GPU_KERNEL(accuracy, paddle::operators::AccuracyOpCUDAKernel, + paddle::operators::AccuracyOpCUDAKernel, + paddle::operators::AccuracyOpCUDAKernel, + paddle::operators::AccuracyOpCUDAKernel); diff --git a/paddle/operators/conv2d_op.h b/paddle/operators/conv2d_op.h index bd1734879e..f629728f68 100644 --- a/paddle/operators/conv2d_op.h +++ b/paddle/operators/conv2d_op.h @@ -108,17 +108,17 @@ class GemmConv2DKernel : public framework::OpKernel { int in_step = input_channels / groups; int out_step = output_channels / groups; for (int i = 0; i < batch_size; i++) { - Tensor in_batch = input->Slice(i, i + 1).Resize(input_shape); - Tensor out_batch = output->Slice(i, i + 1).Resize(output_matrix_shape); + Tensor in_batch = input->Slice(i, i + 1).Resize(input_shape); + Tensor out_batch = output->Slice(i, i + 1).Resize(output_matrix_shape); for (int g = 0; g < groups; g++) { // im2col - Tensor in_slice = in_batch.Slice(g * in_step, (g + 1) * in_step); + Tensor in_slice = in_batch.Slice(g * in_step, (g + 1) * in_step); im2col(context.device_context(), in_slice, col, strides[0], strides[1], paddings[0], paddings[1]); // gemm - Tensor out_slice = out_batch.Slice(g * out_step, (g + 1) * out_step); - Tensor filter_slice = filter.Slice(g * out_step, (g + 1) * out_step); + Tensor out_slice = out_batch.Slice(g * out_step, (g + 1) * out_step); + Tensor filter_slice = filter.Slice(g * out_step, (g + 1) * out_step); math::matmul(context.device_context(), filter_slice, false, col_matrix, false, T(1.0), &out_slice, T(0.0)); } @@ -198,22 +198,20 @@ class GemmConvGrad2DKernel : public framework::OpKernel { for (int i = 0; i < batch_size; i++) { Tensor out_grad_batch = - output_grad->Slice(i, i + 1).Resize(output_matrix_shape); - Tensor in_grad_batch = - input_grad->Slice(i, i + 1).Resize(input_shape); + output_grad->Slice(i, i + 1).Resize(output_matrix_shape); + Tensor in_grad_batch = input_grad->Slice(i, i + 1).Resize(input_shape); for (int g = 0; g < groups; g++) { // gemm Tensor out_grad_slice = - out_grad_batch.Slice(g * out_step, (g + 1) * out_step); - Tensor filter_slice = - filter.Slice(g * out_step, (g + 1) * out_step); + out_grad_batch.Slice(g * out_step, (g + 1) * out_step); + Tensor filter_slice = filter.Slice(g * out_step, (g + 1) * out_step); math::matmul(context.device_context(), filter_slice, true, out_grad_slice, false, T(1.0), &col_matrix, T(0.0)); // col2im Tensor in_grad_slice = - in_grad_batch.Slice(g * in_step, (g + 1) * in_step); + in_grad_batch.Slice(g * in_step, (g + 1) * in_step); col2im(context.device_context(), in_grad_slice, col, strides[0], strides[1], paddings[0], paddings[1]); } @@ -229,19 +227,19 @@ class GemmConvGrad2DKernel : public framework::OpKernel { for (int i = 0; i < batch_size; i++) { Tensor out_grad_batch = - output_grad->Slice(i, i + 1).Resize(output_matrix_shape); - Tensor in_batch = input->Slice(i, i + 1).Resize(input_shape); + output_grad->Slice(i, i + 1).Resize(output_matrix_shape); + Tensor in_batch = input->Slice(i, i + 1).Resize(input_shape); for (int g = 0; g < groups; g++) { // im2col Tensor out_grad_slice = - out_grad_batch.Slice(g * out_step, (g + 1) * out_step); - Tensor in_slice = in_batch.Slice(g * in_step, (g + 1) * in_step); + out_grad_batch.Slice(g * out_step, (g + 1) * out_step); + Tensor in_slice = in_batch.Slice(g * in_step, (g + 1) * in_step); im2col(context.device_context(), in_slice, col, strides[0], strides[1], paddings[0], paddings[1]); // gemm Tensor filter_grad_slice = - filter_grad_.Slice(g * out_step, (g + 1) * out_step); + filter_grad_.Slice(g * out_step, (g + 1) * out_step); math::matmul(context.device_context(), out_grad_slice, false, col_matrix, true, T(1.0), &filter_grad_slice, T(1.0)); diff --git a/paddle/operators/dynamic_recurrent_op.cc b/paddle/operators/dynamic_recurrent_op.cc index 03f33e28d4..62962be205 100644 --- a/paddle/operators/dynamic_recurrent_op.cc +++ b/paddle/operators/dynamic_recurrent_op.cc @@ -48,12 +48,11 @@ inline void ReorderBootState(const DySeqMetaBatch& metas, const LoDTensor& boot_state, LoDTensor* tensor, const platform::Place& dst_place) { for (size_t seq_id = 0; seq_id < metas.size(); seq_id++) { - auto slice = tensor->Slice(seq_id, seq_id + 1); + auto slice = tensor->Slice(seq_id, seq_id + 1); auto boot_slice = - boot_state.Slice(metas[seq_id].ori_idx, metas[seq_id].ori_idx + 1); + boot_state.Slice(metas[seq_id].ori_idx, metas[seq_id].ori_idx + 1); // TODO(superjom) pass in device context as an argument - slice.template CopyFrom(boot_slice, dst_place, - platform::CPUDeviceContext()); + slice.CopyFrom(boot_slice, dst_place, platform::CPUDeviceContext()); } } @@ -138,7 +137,7 @@ void DynamicRecurrentOp::WriteStepInputs() const { if (var == nullptr) { var = step_scope.Var(item.first); } - var->GetMutable()->ShareDataWith(tensor); + var->GetMutable()->ShareDataWith(tensor); } } } @@ -206,7 +205,7 @@ void DynamicRecurrentOp::ConcatOutputs() const { for (auto& item : step_outputs_) { auto tensor = item.second.Pack(level, some_meta, some_lod); auto* output = cache_.outlinks[item.first]->GetMutable(); - const_cast(output)->ShareDataWith(tensor); + const_cast(output)->ShareDataWith(tensor); } } @@ -260,8 +259,8 @@ void DynamicRecurrentOp::LinkState(const rnn::MemoryAttr& memory, } // shink and share from previous state - auto shrinked_pre_state = pre_state->Slice(0, num_instances); - state_pre.ShareDataWith(shrinked_pre_state); + auto shrinked_pre_state = pre_state->Slice(0, num_instances); + state_pre.ShareDataWith(shrinked_pre_state); } void DynamicRecurrentOp::ArgCache::Init( diff --git a/paddle/operators/feed_op.cc b/paddle/operators/feed_op.cc index bf453c8596..0f1722a538 100644 --- a/paddle/operators/feed_op.cc +++ b/paddle/operators/feed_op.cc @@ -47,7 +47,7 @@ class FeedOp : public framework::OperatorBase { auto &feed_list = feed_var->Get(); auto &feed_item = feed_list.at(static_cast(col)); auto *out_item = out_var->GetMutable(); - out_item->CopyFromTensor(feed_item, dev_ctx.GetPlace(), dev_ctx); + out_item->CopyFrom(feed_item, dev_ctx.GetPlace(), dev_ctx); out_item->set_lod(feed_item.lod()); } }; diff --git a/paddle/operators/fetch_op.cc b/paddle/operators/fetch_op.cc index 524e77d6ad..c1b3d66bac 100644 --- a/paddle/operators/fetch_op.cc +++ b/paddle/operators/fetch_op.cc @@ -51,7 +51,7 @@ class FetchOp : public framework::OperatorBase { // FIXME(yuyang18): Should we assume the fetch operator always generate // CPU outputs? - dst_item.CopyFromTensor(src_item, platform::CPUPlace(), dev_ctx); + dst_item.CopyFrom(src_item, platform::CPUPlace(), dev_ctx); VLOG(3) << "Fetch variable " << fetch_var_name << " to " << out_name; } diff --git a/paddle/operators/math/im2col_test.cc b/paddle/operators/math/im2col_test.cc index 9c506ae89b..443c94b83f 100644 --- a/paddle/operators/math/im2col_test.cc +++ b/paddle/operators/math/im2col_test.cc @@ -64,7 +64,7 @@ void testIm2col() { if (paddle::platform::is_cpu_place(*place)) { input = input_tmp; } else { - input.CopyFrom(input_tmp, *place, *context); + input.CopyFrom(input_tmp, *place, *context); } output_cfo.mutable_data( {1, filter_size, filter_size, output_height, output_width}, *place); @@ -85,8 +85,7 @@ void testIm2col() { if (paddle::platform::is_cpu_place(*place)) { out_cfo_ptr = output_cfo.data(); } else { - output_tmp.CopyFrom(output_cfo, paddle::platform::CPUPlace(), - *context); + output_tmp.CopyFrom(output_cfo, paddle::platform::CPUPlace(), *context); out_cfo_ptr = output_tmp.data(); } EXPECT_EQ(out_cfo_ptr[0], 0); @@ -102,8 +101,7 @@ void testIm2col() { if (paddle::platform::is_cpu_place(*place)) { out_ocf_ptr = output_ocf.data(); } else { - output_tmp.CopyFrom(output_ocf, paddle::platform::CPUPlace(), - *context); + output_tmp.CopyFrom(output_ocf, paddle::platform::CPUPlace(), *context); out_ocf_ptr = output_tmp.data(); } EXPECT_EQ(out_ocf_ptr[0], 0); diff --git a/paddle/operators/math/math_function_test.cu b/paddle/operators/math/math_function_test.cu index 14359d835b..8b22c71552 100644 --- a/paddle/operators/math/math_function_test.cu +++ b/paddle/operators/math/math_function_test.cu @@ -16,15 +16,15 @@ TEST(math_function, notrans_mul_trans) { auto* gpu_place = new paddle::platform::GPUPlace(0); paddle::platform::CUDADeviceContext context(*gpu_place); - input1_gpu.CopyFrom(input1, *gpu_place, context); - input2_gpu.CopyFrom(input1, *gpu_place, context); + input1_gpu.CopyFrom(input1, *gpu_place, context); + input2_gpu.CopyFrom(input1, *gpu_place, context); out_gpu.mutable_data({2, 2}, *gpu_place); paddle::operators::math::matmul( context, input1_gpu, false, input2_gpu, true, 1, &out_gpu, 0); - out.CopyFrom(out_gpu, *cpu_place, context); + out.CopyFrom(out_gpu, *cpu_place, context); float* out_ptr = out.data(); context.Wait(); @@ -50,15 +50,15 @@ TEST(math_function, trans_mul_notrans) { auto* gpu_place = new paddle::platform::GPUPlace(0); paddle::platform::CUDADeviceContext context(*gpu_place); - input1_gpu.CopyFrom(input1, *gpu_place, context); - input2_gpu.CopyFrom(input1, *gpu_place, context); + input1_gpu.CopyFrom(input1, *gpu_place, context); + input2_gpu.CopyFrom(input1, *gpu_place, context); out_gpu.mutable_data({3, 3}, *gpu_place); paddle::operators::math::matmul( context, input1_gpu, true, input2_gpu, false, 1, &out_gpu, 0); - out.CopyFrom(out_gpu, *cpu_place, context); + out.CopyFrom(out_gpu, *cpu_place, context); float* out_ptr = out.data(); context.Wait(); @@ -99,9 +99,9 @@ TEST(math_function, gemm_notrans_cublas) { auto* gpu_place = new paddle::platform::GPUPlace(0); paddle::platform::CUDADeviceContext context(*gpu_place); - input1_gpu.CopyFrom(input1, *gpu_place, context); - input2_gpu.CopyFrom(input2, *gpu_place, context); - input3_gpu.CopyFrom(input3, *gpu_place, context); + input1_gpu.CopyFrom(input1, *gpu_place, context); + input2_gpu.CopyFrom(input2, *gpu_place, context); + input3_gpu.CopyFrom(input3, *gpu_place, context); float* a = input1_gpu.data(); float* b = input2_gpu.data(); float* c = input3_gpu.mutable_data(*gpu_place); @@ -109,7 +109,7 @@ TEST(math_function, gemm_notrans_cublas) { paddle::operators::math::gemm( context, false, false, m, n, k, 1, a, 3, b + 1, 4, 1, c + 1, 4); - input3.CopyFrom(input3_gpu, *cpu_place, context); + input3.CopyFrom(input3_gpu, *cpu_place, context); // numpy code: // a = np.arange(6).reshape(2, 3) @@ -154,9 +154,9 @@ TEST(math_function, gemm_trans_cublas) { auto* gpu_place = new paddle::platform::GPUPlace(0); paddle::platform::CUDADeviceContext context(*gpu_place); - input1_gpu.CopyFrom(input1, *gpu_place, context); - input2_gpu.CopyFrom(input2, *gpu_place, context); - input3_gpu.CopyFrom(input3, *gpu_place, context); + input1_gpu.CopyFrom(input1, *gpu_place, context); + input2_gpu.CopyFrom(input2, *gpu_place, context); + input3_gpu.CopyFrom(input3, *gpu_place, context); float* a = input1_gpu.data(); float* b = input2_gpu.data(); float* c = input3_gpu.mutable_data(*gpu_place); @@ -164,7 +164,7 @@ TEST(math_function, gemm_trans_cublas) { paddle::operators::math::gemm( context, false, true, m, n, k, 1, a, 3, b + 3, 3, 1, c + 1, 4); - input3.CopyFrom(input3_gpu, *cpu_place, context); + input3.CopyFrom(input3_gpu, *cpu_place, context); context.Wait(); EXPECT_EQ(input3_ptr[0], 0); diff --git a/paddle/operators/math/selected_rows_functor_test.cu b/paddle/operators/math/selected_rows_functor_test.cu index 8a9f25b982..69607c5afc 100644 --- a/paddle/operators/math/selected_rows_functor_test.cu +++ b/paddle/operators/math/selected_rows_functor_test.cu @@ -67,7 +67,7 @@ TEST(selected_rows_functor, gpu_add) { EXPECT_EQ(out_rows[6], 9); Tensor out_cpu; - out_cpu.CopyFrom(*out_value, cpu_place, ctx); + out_cpu.CopyFrom(*out_value, cpu_place, ctx); ctx.Wait(); auto* out_cpu_data = out_cpu.data(); @@ -94,7 +94,7 @@ TEST(selected_rows_functor, gpu_add) { add_tensor_functor(ctx, *output, *tensor1, tensor2.get()); Tensor tensor2_cpu; - tensor2_cpu.CopyFrom(*tensor2, cpu_place, ctx); + tensor2_cpu.CopyFrom(*tensor2, cpu_place, ctx); ctx.Wait(); auto* tensor2_cpu_data = tensor2_cpu.data(); diff --git a/paddle/operators/math/vol2col_test.cc b/paddle/operators/math/vol2col_test.cc index 2d69218843..74590d17cd 100644 --- a/paddle/operators/math/vol2col_test.cc +++ b/paddle/operators/math/vol2col_test.cc @@ -78,7 +78,7 @@ void testVol2col() { if (paddle::platform::is_cpu_place(*place)) { input = input_tmp; } else { - input.CopyFrom(input_tmp, *place, *context); + input.CopyFrom(input_tmp, *place, *context); } output.mutable_data({1, filter_size, filter_size, filter_size, output_depth, output_height, output_width}, @@ -93,7 +93,7 @@ void testVol2col() { if (paddle::platform::is_cpu_place(*place)) { out_cfo_ptr = output.data(); } else { - output_tmp.CopyFrom(output, paddle::platform::CPUPlace(), *context); + output_tmp.CopyFrom(output, paddle::platform::CPUPlace(), *context); out_cfo_ptr = output_tmp.data(); } @@ -107,7 +107,7 @@ void testVol2col() { if (paddle::platform::is_cpu_place(*place)) { input = input_tmp; } else { - input.CopyFrom(input_tmp, *place, *context); + input.CopyFrom(input_tmp, *place, *context); } paddle::operators::math::Col2VolFunctor col2vol; @@ -118,7 +118,7 @@ void testVol2col() { if (paddle::platform::is_cpu_place(*place)) { in_ptr = input.data(); } else { - input_tmp.CopyFrom(input, paddle::platform::CPUPlace(), *context); + input_tmp.CopyFrom(input, paddle::platform::CPUPlace(), *context); in_ptr = input_tmp.data(); } diff --git a/paddle/operators/matmul_op.h b/paddle/operators/matmul_op.h index 8ae54e1eec..5ce30740c9 100644 --- a/paddle/operators/matmul_op.h +++ b/paddle/operators/matmul_op.h @@ -46,7 +46,7 @@ class MatMulKernel : public framework::OpKernel { template inline Tensor Reshape(const Tensor& input, const DDim& dims) { Tensor output; - output.ShareDataWith(input); + output.ShareDataWith(input); output.Resize(dims); return output; } @@ -56,7 +56,7 @@ inline Tensor Reshape(const Tensor& input, const DDim& dims) { template Tensor CombineBatchAndM(const Tensor& input) { Tensor output; - output.ShareDataWith(input); + output.ShareDataWith(input); auto in_dims = input.dims(); if (in_dims.size() == 3) { std::vector out_dims = {in_dims[0] * in_dims[1], in_dims[2]}; @@ -80,7 +80,7 @@ Tensor CombineBatchAndN(const framework::ExecutionContext& context, std::vector out_dims = {in_dims[1], in_dims[0] * in_dims[2]}; output.Resize(make_ddim(out_dims)); } else { - output.ShareDataWith(input); + output.ShareDataWith(input); } return output; } diff --git a/paddle/operators/mul_op.h b/paddle/operators/mul_op.h index 684b1ea0c0..3f3e77595b 100644 --- a/paddle/operators/mul_op.h +++ b/paddle/operators/mul_op.h @@ -36,12 +36,12 @@ class MulKernel : public framework::OpKernel { Tensor* z = context.Output("Out"); const Tensor x_matrix = x->dims().size() > 2 - ? framework::ReshapeToMatrix( + ? framework::ReshapeToMatrix( *x, context.template Attr("x_num_col_dims")) : *x; const Tensor y_matrix = y->dims().size() > 2 - ? framework::ReshapeToMatrix( + ? framework::ReshapeToMatrix( *y, context.template Attr("y_num_col_dims")) : *y; @@ -59,30 +59,30 @@ class MulGradKernel : public framework::OpKernel { int y_num_col_dims = ctx.template Attr("y_num_col_dims"); const Tensor* x = ctx.Input("X"); const Tensor* y = ctx.Input("Y"); - const Tensor x_matrix = - x->dims().size() > 2 ? framework::ReshapeToMatrix(*x, x_num_col_dims) - : *x; - const Tensor y_matrix = - y->dims().size() > 2 ? framework::ReshapeToMatrix(*y, y_num_col_dims) - : *y; + const Tensor x_matrix = x->dims().size() > 2 + ? framework::ReshapeToMatrix(*x, x_num_col_dims) + : *x; + const Tensor y_matrix = y->dims().size() > 2 + ? framework::ReshapeToMatrix(*y, y_num_col_dims) + : *y; const Tensor* dout = ctx.Input(framework::GradVarName("Out")); Tensor* dx = ctx.Output(framework::GradVarName("X")); Tensor* dy = ctx.Output(framework::GradVarName("Y")); if (dx) { dx->mutable_data(ctx.GetPlace()); - Tensor dx_matrix = dx->dims().size() > 2 ? framework::ReshapeToMatrix( - *dx, x_num_col_dims) - : *dx; + Tensor dx_matrix = dx->dims().size() > 2 + ? framework::ReshapeToMatrix(*dx, x_num_col_dims) + : *dx; // dx = dout * y'. dx: M x K, dout : M x N, y : K x N math::matmul(ctx.device_context(), *dout, false, y_matrix, true, 1, &dx_matrix, 0); } if (dy) { dy->mutable_data(ctx.GetPlace()); - Tensor dy_matrix = dy->dims().size() > 2 ? framework::ReshapeToMatrix( - *dy, y_num_col_dims) - : *dy; + Tensor dy_matrix = dy->dims().size() > 2 + ? framework::ReshapeToMatrix(*dy, y_num_col_dims) + : *dy; // dy = x' * dout. dy K x N, dout : M x N, x : M x K math::matmul(ctx.device_context(), x_matrix, true, *dout, false, 1, &dy_matrix, 0); diff --git a/paddle/operators/multiplex_op.cu b/paddle/operators/multiplex_op.cu index 10cb0e005f..143a14fef5 100644 --- a/paddle/operators/multiplex_op.cu +++ b/paddle/operators/multiplex_op.cu @@ -33,8 +33,7 @@ class MultiplexGPUKernel : public framework::OpKernel { auto cols = ins[0]->numel() / rows; // copy index to cpu Tensor index_t_cpu; - index_t_cpu.CopyFrom(*ids, platform::CPUPlace(), - ctx.device_context()); + index_t_cpu.CopyFrom(*ids, platform::CPUPlace(), ctx.device_context()); auto* index = index_t_cpu.data(); auto stream = reinterpret_cast( ctx.device_context()) @@ -71,8 +70,7 @@ class MultiplexGradGPUKernel : public framework::OpKernel { auto cols = ins[0]->numel() / rows; // copy index to cpu Tensor index_t_cpu; - index_t_cpu.CopyFrom(*ids, platform::CPUPlace(), - ctx.device_context()); + index_t_cpu.CopyFrom(*ids, platform::CPUPlace(), ctx.device_context()); auto* index = index_t_cpu.data(); auto stream = reinterpret_cast( diff --git a/paddle/operators/recurrent_op.cc b/paddle/operators/recurrent_op.cc index e3d08378c2..dcc90e5d87 100644 --- a/paddle/operators/recurrent_op.cc +++ b/paddle/operators/recurrent_op.cc @@ -95,7 +95,7 @@ void RecurrentAlgorithm::InitMemories(Scope* step_scope) const { step_scope->FindVar(attr.boot_var)->GetMutable(); pre_mem->Resize(boot_mem->dims()); PADDLE_ENFORCE_EQ(pre_mem->dims().size(), 2); - pre_mem->ShareDataWith(*boot_mem); + pre_mem->ShareDataWith(*boot_mem); } } @@ -171,7 +171,7 @@ void RecurrentGradientAlgorithm::LinkBootMemoryGradients( auto* boot_mem_grad = step_scope->Var(attr.boot_var)->GetMutable(); boot_mem_grad->Resize(mem_grad->dims()); - boot_mem_grad->ShareDataWith(*mem_grad); + boot_mem_grad->ShareDataWith(*mem_grad); } } diff --git a/paddle/operators/reshape_op.h b/paddle/operators/reshape_op.h index 3ba4611458..c89cdf8cab 100644 --- a/paddle/operators/reshape_op.h +++ b/paddle/operators/reshape_op.h @@ -33,7 +33,7 @@ class ReshapeKernel : public framework::OpKernel { std::transform(shape.begin(), shape.end(), shape_int64.begin(), [](int a) { return static_cast(a); }); auto out_dims = framework::make_ddim(shape_int64); - out->CopyFrom(*in, ctx.GetPlace(), ctx.device_context()); + out->CopyFrom(*in, ctx.GetPlace(), ctx.device_context()); out->Resize(out_dims); } }; @@ -47,7 +47,7 @@ class ReshapeGradKernel : public framework::OpKernel { d_x->mutable_data(ctx.GetPlace()); auto in_dims = d_x->dims(); - d_x->CopyFrom(*d_out, ctx.GetPlace(), ctx.device_context()); + d_x->CopyFrom(*d_out, ctx.GetPlace(), ctx.device_context()); d_x->Resize(in_dims); } }; diff --git a/paddle/operators/rnn/recurrent_op_utils.cc b/paddle/operators/rnn/recurrent_op_utils.cc index 30b8ddeb5b..d0725f5023 100644 --- a/paddle/operators/rnn/recurrent_op_utils.cc +++ b/paddle/operators/rnn/recurrent_op_utils.cc @@ -43,7 +43,7 @@ void SegmentInputs(const std::vector& step_scopes, step_scopes[j]->Var(inlinks[i])->GetMutable(); // The input of operators of each step is Tensor here. // Maybe need to modify Slice function. - *step_input = input->Slice(j, j + 1); + *step_input = input->Slice(j, j + 1); step_input->Resize(step_dims); } } @@ -71,8 +71,8 @@ void ConcatOutputs(const std::vector& step_scopes, step_scopes[j]->FindVar(outlinks[i])->GetMutable(); // TODO(luotao02) data type and platform::DeviceContext() should set // correctly - (output->Slice(j, j + 1)) - .CopyFrom(*step_output, platform::CPUPlace(), ctx); + (output->Slice(j, j + 1)) + .CopyFrom(*step_output, platform::CPUPlace(), ctx); } } } @@ -95,7 +95,7 @@ void LinkMemories(const std::vector& scopes, auto* mem = scope->FindVar(attr.pre_var)->GetMutable(); auto* linked_mem = linked_scope->FindVar(attr.var)->GetMutable(); mem->Resize(linked_mem->dims()); - mem->ShareDataWith(*linked_mem); + mem->ShareDataWith(*linked_mem); } } diff --git a/paddle/operators/scatter_op.cu b/paddle/operators/scatter_op.cu index 06f4d75944..3b32ae2fb7 100644 --- a/paddle/operators/scatter_op.cu +++ b/paddle/operators/scatter_op.cu @@ -30,7 +30,7 @@ class ScatterOpCUDAKernel : public framework::OpKernel { auto *Updates = ctx.Input("Updates"); auto *Out = ctx.Output("Out"); - Out->ShareDataWith(*Ref); + Out->ShareDataWith(*Ref); GPUScatterAssign(ctx.device_context(), *Updates, *Index, Out); } @@ -48,7 +48,7 @@ class ScatterGradOpCUDAKernel : public framework::OpKernel { auto *dOut = ctx.Input(framework::GradVarName("Out")); // In place gradient: dRef = dO - dRef->ShareDataWith(*dOut); + dRef->ShareDataWith(*dOut); dUpdates->mutable_data(ctx.GetPlace()); // Gradient by Gather: dUpdates = dO[Index] GPUGather(ctx.device_context(), *dOut, *Index, dUpdates); diff --git a/paddle/operators/scatter_op.h b/paddle/operators/scatter_op.h index 6101219006..1a4f6f99bf 100644 --- a/paddle/operators/scatter_op.h +++ b/paddle/operators/scatter_op.h @@ -35,7 +35,7 @@ class ScatterOpKernel : public framework::OpKernel { auto *Out = ctx.Output("Out"); // In place output: Out = Ref, Out[Index] += Updates - Out->ShareDataWith(*Ref); + Out->ShareDataWith(*Ref); // Apply ScatterUpdate: Out[index] += Updates[:] ScatterAssign(ctx.device_context(), *Updates, *Index, Out); } @@ -53,7 +53,7 @@ class ScatterGradientOpKernel : public framework::OpKernel { auto *dOut = ctx.Input(framework::GradVarName("Out")); // In place gradient: dRef = dO - dRef->ShareDataWith(*dOut); + dRef->ShareDataWith(*dOut); dUpdates->mutable_data(ctx.GetPlace()); // Gradient by Gather: dUpdates += dO[Index] CPUGather(ctx.device_context(), *dOut, *Index, dUpdates); diff --git a/paddle/operators/sequence_concat_op.h b/paddle/operators/sequence_concat_op.h index a197a05bbb..6adf96120c 100644 --- a/paddle/operators/sequence_concat_op.h +++ b/paddle/operators/sequence_concat_op.h @@ -87,16 +87,16 @@ class SequenceConcatOpKernel : public framework::OpKernel { auto out_lod_level = out_lod[level]; for (size_t i = 0; i < out_lod_level.size() - 1; ++i) { - Tensor out_t = out->Slice(static_cast(out_lod_level[i]), - static_cast(out_lod_level[i + 1])); + Tensor out_t = out->Slice(static_cast(out_lod_level[i]), + static_cast(out_lod_level[i + 1])); auto out_stride = framework::stride(out_t.dims()); size_t offset = 0; for (size_t j = 0; j < n; ++j) { auto in_lod_level = ins[j]->lod()[level]; auto in_stride = framework::stride(ins[j]->dims()); - Tensor in_t = ins[j]->Slice(static_cast(in_lod_level[i]), - static_cast(in_lod_level[i + 1])); + Tensor in_t = ins[j]->Slice(static_cast(in_lod_level[i]), + static_cast(in_lod_level[i + 1])); size_t axis_dim = in_t.dims()[axis]; StridedMemcpy(ctx.device_context(), in_t.data(), in_stride, in_t.dims(), out_stride, out_t.data() + offset); @@ -130,8 +130,8 @@ class SequenceConcatGradOpKernel : public framework::OpKernel { for (size_t i = 0; i < out_lod_level.size() - 1; ++i) { Tensor out_grad_t = - out_grad->Slice(static_cast(out_lod_level[i]), - static_cast(out_lod_level[i + 1])); + out_grad->Slice(static_cast(out_lod_level[i]), + static_cast(out_lod_level[i + 1])); auto out_grad_stride = framework::stride(out_grad_t.dims()); size_t offset = 0; @@ -139,8 +139,8 @@ class SequenceConcatGradOpKernel : public framework::OpKernel { auto x_grad_lod_level = x_grads[j]->lod()[level]; auto x_grad_stride = framework::stride(x_grads[j]->dims()); Tensor x_grad_t = - x_grads[j]->Slice(static_cast(x_grad_lod_level[i]), - static_cast(x_grad_lod_level[i + 1])); + x_grads[j]->Slice(static_cast(x_grad_lod_level[i]), + static_cast(x_grad_lod_level[i + 1])); size_t axis_dim = x_grad_t.dims()[axis]; StridedMemcpy(ctx.device_context(), out_grad_t.data() + offset, out_grad_stride, out_grad_t.dims(), x_grad_stride, diff --git a/paddle/operators/sequence_pool_op.h b/paddle/operators/sequence_pool_op.h index a5569d1aac..0de6cafe9c 100644 --- a/paddle/operators/sequence_pool_op.h +++ b/paddle/operators/sequence_pool_op.h @@ -64,9 +64,9 @@ class SequencePoolKernel : public framework::OpKernel { out->mutable_data(context.GetPlace()); auto place = context.GetEigenDevice(); for (int i = 0; i < static_cast(lod_level_0.size()) - 1; ++i) { - Tensor in_t = in->Slice(static_cast(lod_level_0[i]), - static_cast(lod_level_0[i + 1])); - Tensor out_t = out->Slice(i, i + 1); + Tensor in_t = in->Slice(static_cast(lod_level_0[i]), + static_cast(lod_level_0[i + 1])); + Tensor out_t = out->Slice(i, i + 1); int64_t h = static_cast(lod_level_0[i + 1] - lod_level_0[i]); auto in_e = EigenMatrix::From(in_t, framework::make_ddim({h, w})); auto out_e = EigenVector::Flatten(out_t); @@ -116,9 +116,9 @@ class SequencePoolGradKernel : public framework::OpKernel { } auto place = context.GetEigenDevice(); for (int i = 0; i < static_cast(lod.size()) - 1; ++i) { - auto in_g_t = in_g->Slice(static_cast(lod[i]), - static_cast(lod[i + 1])); - auto out_g_t = out_g->Slice(i, i + 1); + auto in_g_t = + in_g->Slice(static_cast(lod[i]), static_cast(lod[i + 1])); + auto out_g_t = out_g->Slice(i, i + 1); int64_t h = static_cast(lod[i + 1] - lod[i]); auto in_g_e = EigenMatrix::From(in_g_t, {h, w}); auto out_g_e = EigenMatrix::From(out_g_t, {1, w}); diff --git a/paddle/operators/sequence_softmax_op.h b/paddle/operators/sequence_softmax_op.h index 96d87c404d..3eb1e2844d 100644 --- a/paddle/operators/sequence_softmax_op.h +++ b/paddle/operators/sequence_softmax_op.h @@ -46,8 +46,8 @@ class SequenceSoftmaxKernel : public framework::OpKernel { for (int i = 0; i < static_cast(lod[level].size()) - 1; ++i) { int start_pos = static_cast(lod[level][i]); int end_pos = static_cast(lod[level][i + 1]); - Tensor x_i = x->Slice(start_pos, end_pos); - Tensor out_i = out->Slice(start_pos, end_pos); + Tensor x_i = x->Slice(start_pos, end_pos); + Tensor out_i = out->Slice(start_pos, end_pos); // Reshape from (end_pos - start_pos) x 1UL to 1UL x (end_pos - start_pos) framework::DDim dims_i = framework::make_ddim({1UL, end_pos - start_pos}); @@ -75,9 +75,9 @@ class SequenceSoftmaxGradKernel : public framework::OpKernel { int start_pos = static_cast(lod[level][i]); int end_pos = static_cast(lod[level][i + 1]); - Tensor out_i = out->Slice(start_pos, end_pos); - Tensor out_grad_i = out_grad->Slice(start_pos, end_pos); - Tensor x_grad_i = x_grad->Slice(start_pos, end_pos); + Tensor out_i = out->Slice(start_pos, end_pos); + Tensor out_grad_i = out_grad->Slice(start_pos, end_pos); + Tensor x_grad_i = x_grad->Slice(start_pos, end_pos); // Reshape from (end_pos - start_pos) x 1UL to 1UL x (end_pos - start_pos) framework::DDim dims_i = framework::make_ddim({1UL, end_pos - start_pos}); diff --git a/paddle/operators/softmax_with_cross_entropy_op.cu b/paddle/operators/softmax_with_cross_entropy_op.cu index d03a1a7658..68ac2b0ea3 100644 --- a/paddle/operators/softmax_with_cross_entropy_op.cu +++ b/paddle/operators/softmax_with_cross_entropy_op.cu @@ -85,7 +85,7 @@ class SoftmaxWithCrossEntropyGradCUDAKernel : public framework::OpKernel { context.Input(framework::GradVarName("Loss"))->data(); Tensor* logit_grad = context.Output(framework::GradVarName("Logits")); - logit_grad->ShareDataWith(*context.Input("Softmax")); + logit_grad->ShareDataWith(*context.Input("Softmax")); T* logit_grad_data = logit_grad->data(); const int batch_size = logit_grad->dims()[0]; diff --git a/paddle/operators/softmax_with_cross_entropy_op.h b/paddle/operators/softmax_with_cross_entropy_op.h index 66d7bc1569..01027cf63f 100644 --- a/paddle/operators/softmax_with_cross_entropy_op.h +++ b/paddle/operators/softmax_with_cross_entropy_op.h @@ -57,7 +57,7 @@ class SoftmaxWithCrossEntropyGradKernel : public framework::OpKernel { const Tensor* labels = context.Input("Label"); Tensor* logit_grad = context.Output(framework::GradVarName("Logits")); - logit_grad->ShareDataWith(*context.Input("Softmax")); + logit_grad->ShareDataWith(*context.Input("Softmax")); const int class_num = logit_grad->dims()[1]; if (context.Attr("soft_label")) { diff --git a/paddle/pybind/pybind.cc b/paddle/pybind/pybind.cc index 84ebe3c2b8..3455c82e67 100644 --- a/paddle/pybind/pybind.cc +++ b/paddle/pybind/pybind.cc @@ -84,10 +84,12 @@ PYBIND11_PLUGIN(core) { .def("set", PyCPUTensorSetFromArray) .def("set", PyCPUTensorSetFromArray) .def("set", PyCPUTensorSetFromArray) + .def("set", PyCPUTensorSetFromArray) #ifdef PADDLE_WITH_CUDA .def("set", PyCUDATensorSetFromArray) .def("set", PyCUDATensorSetFromArray) .def("set", PyCUDATensorSetFromArray) + .def("set", PyCUDATensorSetFromArray) #endif .def("shape", [](Tensor &self) { return vectorize(self.dims()); }) .def("set_float_element", TensorSetElement) From 42f2dd4041c4bc584194cb55470190f8233be70f Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Thu, 19 Oct 2017 17:03:49 -0700 Subject: [PATCH 49/76] Unify `set_feed_variable` to one method (#4949) --- paddle/framework/feed_fetch_method.h | 1 - paddle/pybind/pybind.cc | 5 +---- python/paddle/v2/framework/executor.py | 3 +-- python/paddle/v2/framework/tests/test_feed_fetch_method.py | 2 +- 4 files changed, 3 insertions(+), 8 deletions(-) diff --git a/paddle/framework/feed_fetch_method.h b/paddle/framework/feed_fetch_method.h index d58736dcb1..3ef70043d6 100644 --- a/paddle/framework/feed_fetch_method.h +++ b/paddle/framework/feed_fetch_method.h @@ -21,7 +21,6 @@ limitations under the License. */ namespace paddle { namespace framework { -template void SetFeedVariable(const LoDTensor& input, const std::string& var_name, size_t index) { // If var_name Variable is not found in GlobalScope, a new variable will diff --git a/paddle/pybind/pybind.cc b/paddle/pybind/pybind.cc index 3455c82e67..94c9706f79 100644 --- a/paddle/pybind/pybind.cc +++ b/paddle/pybind/pybind.cc @@ -460,10 +460,7 @@ All parameter, weight, gradient are variables in Paddle. m.def("unique_integer", UniqueIntegerGenerator); m.def("is_compile_gpu", IsCompileGPU); - //! FIXME: it is no need to `set_xxx_float/double/int` - m.def("set_feed_variable_float", framework::SetFeedVariable); - m.def("set_feed_variable_double", framework::SetFeedVariable); - m.def("set_feed_variable_int", framework::SetFeedVariable); + m.def("set_feed_variable", framework::SetFeedVariable); m.def("get_fetch_variable", framework::GetFetchVariable); BindProgramDesc(m); diff --git a/python/paddle/v2/framework/executor.py b/python/paddle/v2/framework/executor.py index 8da5daad99..1adc10c233 100644 --- a/python/paddle/v2/framework/executor.py +++ b/python/paddle/v2/framework/executor.py @@ -38,8 +38,7 @@ class Executor(object): inputs={'X': [feed_var]}, outputs={'Out': [out]}, attrs={'col': i}) - # FIXME - core.set_feed_variable_float(feed[name], feed_var.name, i) + core.set_feed_variable(feed[name], feed_var.name, i) fetch_var = global_block.create_var( name=fetch_var_name, diff --git a/python/paddle/v2/framework/tests/test_feed_fetch_method.py b/python/paddle/v2/framework/tests/test_feed_fetch_method.py index 47eedddcb6..8b9b44440d 100644 --- a/python/paddle/v2/framework/tests/test_feed_fetch_method.py +++ b/python/paddle/v2/framework/tests/test_feed_fetch_method.py @@ -12,7 +12,7 @@ class TestFeedFetch(unittest.TestCase): input_tensor = core.LoDTensor([[0, 2, 4]]) input_tensor.set(input_array, place) - core.set_feed_variable_float(input_tensor, "feed", 0) + core.set_feed_variable(input_tensor, "feed", 0) output_tensor = core.get_fetch_variable("feed", 0) From d75b00c2210c247bcf626bf3239fd6f7dc115e49 Mon Sep 17 00:00:00 2001 From: tensor-tang Date: Thu, 19 Oct 2017 17:18:22 +0800 Subject: [PATCH 50/76] refine the gtest log info and vlog order, and change the size of test to make unit test faster refine comment and log of mkldnnlayer --- paddle/gserver/layers/MKLDNNBase.h | 4 +- paddle/gserver/layers/MKLDNNLayer.cpp | 7 ++- paddle/gserver/layers/MKLDNNLayer.h | 8 ++-- paddle/gserver/tests/MKLDNNTester.cpp | 44 ++++++++++--------- paddle/gserver/tests/MKLDNNTester.h | 8 +--- .../sample_trainer_config_branch_net.conf | 26 +++++------ .../sample_trainer_config_simple_net.conf | 2 +- 7 files changed, 52 insertions(+), 47 deletions(-) diff --git a/paddle/gserver/layers/MKLDNNBase.h b/paddle/gserver/layers/MKLDNNBase.h index 4c0234e7b3..af02a37cad 100644 --- a/paddle/gserver/layers/MKLDNNBase.h +++ b/paddle/gserver/layers/MKLDNNBase.h @@ -21,8 +21,8 @@ namespace paddle { typedef enum { MKLDNN_BASE = 1, // basical info of MKLDNN MKLDNN_TESTS = 1, // gtest info of MKLDNN - MKLDNN_SIZES = 2, // size info of MKLDNN - MKLDNN_FMTS = 3, // format info of MKLDNN + MKLDNN_FMTS = 2, // format info of MKLDNN + MKLDNN_SIZES = 3, // size info of MKLDNN MKLDNN_ALL = 4, // show all info of MKLDNN } MKLDNN_LOG_LEVEL; diff --git a/paddle/gserver/layers/MKLDNNLayer.cpp b/paddle/gserver/layers/MKLDNNLayer.cpp index 91f0ff5bd3..f4968c4af3 100644 --- a/paddle/gserver/layers/MKLDNNLayer.cpp +++ b/paddle/gserver/layers/MKLDNNLayer.cpp @@ -105,6 +105,10 @@ void MKLDNNLayer::backward(const UpdateCallback& callback) { // external output grad is not necessary // since output may be mkldnn internal buffer or merge them directly. CHECK(outGrad_) << "internal output grad is necessary"; + if (extOutGrad_) { + CHECK_EQ(extOutGrad_->getData(), output_.grad->getData()) + << "the external buffer should share the same data with output_.grad"; + } if (cvtOutGrad_) { pipelineBwd_.insert(pipelineBwd_.begin(), *cvtOutGrad_); } @@ -293,7 +297,6 @@ void MKLDNNLayer::resetMergeGrad(MKLDNNMatrixPtr& out) { for (auto it = outputMap_.begin(); it != outputMap_.end(); ++it) { MKLDNNMatrixPtr src = std::dynamic_pointer_cast(it->second->grad); - VLOG(MKLDNN_BASE) << getName() << " has output grad " << it->first; CHECK(src) << "should be MKLDNNMatrix"; auto srcDims = src->getDims(); auto dstDims = out->getDims(); @@ -301,6 +304,8 @@ void MKLDNNLayer::resetMergeGrad(MKLDNNMatrixPtr& out) { for (size_t i = 0; i < srcDims.size(); ++i) { CHECK_EQ(srcDims[i], dstDims[i]); } + VLOG(MKLDNN_BASE) << getName() << " has output grad " << it->first + << ", format " << src->getFormat(); srcPDs.push_back(src->getPrimitiveDesc()); srcs.push_back(*src); } diff --git a/paddle/gserver/layers/MKLDNNLayer.h b/paddle/gserver/layers/MKLDNNLayer.h index faad434526..656b5ee2d7 100644 --- a/paddle/gserver/layers/MKLDNNLayer.h +++ b/paddle/gserver/layers/MKLDNNLayer.h @@ -58,13 +58,13 @@ protected: std::vector pipelineFwd_; std::vector pipelineBwd_; - /// value and grad are seperate as internal and external buffers. + /// value and grad are seperated as internal and external buffers. /// each MKLDNNLayer must init or reset internal buffer at least, /// and the external buffer format is always nchw of nc(when h==w==1), /// which is the same format as paddle. - /// When mixed with cpu device, the output_.value and output_.grad - /// always save the external data. - /// When all layers are all mkldnn layers, they could be internal data. + /// The output_.value and output_.grad always save the external data, + /// when mixed with cpu device. + /// When all layers are mkldnn layers, they could save internal data. /// below MKLDNNMatrix buffers are all internal buffers MKLDNNMatrixPtr inVal_; MKLDNNMatrixPtr inGrad_; diff --git a/paddle/gserver/tests/MKLDNNTester.cpp b/paddle/gserver/tests/MKLDNNTester.cpp index 3bf6a9e176..0a19fe2333 100644 --- a/paddle/gserver/tests/MKLDNNTester.cpp +++ b/paddle/gserver/tests/MKLDNNTester.cpp @@ -97,7 +97,7 @@ void MKLDNNTester::randomWgtDatas() { parameters_[REF][i]->randomize(); dnnValue->copyFrom(*refValue); - VLOG(lvl_) << "Random weight data " << parameters_[DNN][i]->getName(); + VLOG(MKLDNN_TESTS) << "Random weight " << parameters_[DNN][i]->getName(); printVector(dnnValue); } } @@ -109,7 +109,7 @@ void MKLDNNTester::randomBotDatas() { dataLayers_[REF][i]->getOutputValue()->randomizeUniform(); dataLayers_[DNN][i]->getOutputValue()->copyFrom( *(dataLayers_[REF][i]->getOutputValue())); - VLOG(lvl_) << "Input " << i << " data:"; + VLOG(MKLDNN_TESTS) << "Random Foward, InputValue " << i; printMatrix(dataLayers_[REF][i]->getOutputValue()); } } @@ -118,12 +118,12 @@ void MKLDNNTester::randomTopDiffs() { refLayer_->getOutputGrad()->randomizeUniform(); dnnLayer_->getOutput(CPU_DEVICE) .grad->copyFrom(*(refLayer_->getOutputGrad())); - VLOG(lvl_) << "Random Backward Input, TopDiff: "; + VLOG(MKLDNN_TESTS) << "Random Backward, OutputGrad"; printMatrix(refLayer_->getOutputGrad()); } void MKLDNNTester::checkForward() { - VLOG(MKLDNN_ALL) << "Check Forward"; + VLOG(MKLDNN_TESTS) << "Check Forward"; printTopDatas(); double delta = compareMatrix(dnnLayer_->getOutputValue(), refLayer_->getOutputValue()); @@ -131,15 +131,15 @@ void MKLDNNTester::checkForward() { } void MKLDNNTester::checkBackwardData() { - VLOG(MKLDNN_ALL) << "Check Backward Data"; + VLOG(MKLDNN_TESTS) << "Check Backward Data"; // TODO(TJ): uncomment me when batch norm ready // const bool isBN = dnnLayer_->getType() == "mkldnn_batch_norm"; for (size_t i = 0; i < dataLayers_[DNN].size(); ++i) { const MatrixPtr& dnnDiff = dataLayers_[DNN][i]->getOutputGrad(); const MatrixPtr& refDiff = dataLayers_[REF][i]->getOutputGrad(); - VLOG(lvl_) << "Mkldnn Backward Output BotDiff " << i; + VLOG(MKLDNN_ALL) << "MKLDNN Backward Result: InputGrad " << i; printMatrix(dnnDiff); - VLOG(lvl_) << "Reference Backward Output BotDiff " << i; + VLOG(MKLDNN_ALL) << "Reference Backward Result: InputGrad " << i; printMatrix(refDiff); double delta = compareMatrix(dnnDiff, refDiff); @@ -153,7 +153,7 @@ void MKLDNNTester::checkBackwardData() { } void MKLDNNTester::checkBackwardWgts() { - VLOG(MKLDNN_ALL) << "Check Backward Weight"; + VLOG(MKLDNN_TESTS) << "Check Backward Weight"; CHECK_EQ(parameters_[DNN].size(), parameters_[REF].size()); vector dnnWgts; // used to temply save mkldnn weights saveWgt(parameters_[DNN], dnnWgts); @@ -165,9 +165,11 @@ void MKLDNNTester::checkBackwardWgts() { for (size_t i = 0; i < parameters_[DNN].size(); ++i) { const VectorPtr& dnn = parameters_[DNN][i]->getBuf(PARAMETER_VALUE); const VectorPtr& ref = parameters_[REF][i]->getBuf(PARAMETER_VALUE); - VLOG(lvl_) << "Mkldnn Output weight " << parameters_[DNN][i]->getName(); + VLOG(MKLDNN_ALL) << "MKLDNN Result: weight value" + << parameters_[DNN][i]->getName(); printVector(dnn); - VLOG(lvl_) << "Reference Output weight " << parameters_[REF][i]->getName(); + VLOG(MKLDNN_ALL) << "Reference Result: weight value " + << parameters_[REF][i]->getName(); printVector(ref); double delta = compareVector(dnn, ref); @@ -240,7 +242,8 @@ void MKLDNNTester::printTopDatas() { } for (int n = 0; n < NUM; ++n) { - VLOG(lvl_) << testLayers_[n]->getType() << " forward output TopData: "; + VLOG(MKLDNN_ALL) << testLayers_[n]->getType() + << " Forward Result: OutputValue"; printMatrix(testLayers_[n]->getOutputValue()); } } @@ -252,7 +255,7 @@ void MKLDNNTester::printMatrix(const MatrixPtr& m) { std::ostringstream ostr; m->print(ostr); - VLOG(lvl_) << std::endl << ostr.str(); + VLOG(MKLDNN_ALL) << std::endl << ostr.str(); } void MKLDNNTester::printVector(const VectorPtr& v) { @@ -262,7 +265,7 @@ void MKLDNNTester::printVector(const VectorPtr& v) { std::ostringstream ostr; v->print(ostr, v->getSize()); - VLOG(lvl_) << std::endl << ostr.str(); + VLOG(MKLDNN_ALL) << std::endl << ostr.str(); } double MKLDNNTester::getDelta(const real* d1, @@ -314,7 +317,7 @@ void MKLDNNTester::runOnce() { UpdateCallback updateCallback = [](Parameter* para) { auto& grad = para->getBuf(PARAMETER_GRADIENT); auto& value = para->getBuf(PARAMETER_VALUE); - real lr = 1e-3; + real lr = 1e-2; value->add(*grad, lr); grad->zeroMem(); }; @@ -340,10 +343,9 @@ void MKLDNNTester::run(const TestConfig& dnn, size_t batchSize, size_t inputImgH, size_t inputImgW, + bool printDetails, size_t iter, - float epsilon, - bool log, - int level) { + float epsilon) { CHECK(dnn.layerConfig.type().compare(0, 7, "mkldnn_") == 0 || dnn.layerConfig.active_type().compare(0, 7, "mkldnn_") == 0) << "should be MKLDNN layer or MKLDNN activation"; @@ -359,10 +361,9 @@ void MKLDNNTester::run(const TestConfig& dnn, ih_ = inputImgH; iw_ = inputImgW; + log_ = printDetails; iter_ = iter; eps_ = epsilon; - log_ = log; - lvl_ = level; // Firstly test mkldnn init from PARAM_FORMAT_ORIGINAL weight reset(dnn, ref, batchSize); @@ -531,9 +532,11 @@ void MKLDNNTester::getOutResult(const std::string& configPath, void MKLDNNTester::compareResult(DataOut& ref, DataOut& dnn, float eps) { CHECK_EQ(ref.outValues.size(), dnn.outValues.size()); CHECK_EQ(ref.paraValues.size(), dnn.paraValues.size()); + VLOG(MKLDNN_TESTS) << "compare value size: " << ref.outValues.size(); for (size_t i = 0; i < ref.outValues.size(); i++) { EXPECT_LE(fabs(compareMatrix(ref.outValues[i], dnn.outValues[i])), eps); } + VLOG(MKLDNN_TESTS) << "compare param size: " << ref.outValues.size(); for (size_t i = 0; i < ref.paraValues.size(); i++) { EXPECT_LE(fabs(compareVector(ref.paraValues[i], dnn.paraValues[i])), eps); } @@ -544,9 +547,10 @@ void MKLDNNTester::runBranchesTest(const std::string& configPath, float eps) { DataIn in; initArgument(in, configPath, iter); - DataOut outCpu, outDnn; + VLOG(MKLDNN_TESTS) << "runing cpu network"; getOutResult(configPath, in, outCpu, false, iter); + VLOG(MKLDNN_TESTS) << "runing mkldnn network"; getOutResult(configPath, in, outDnn, true, iter); compareResult(outCpu, outDnn, eps); diff --git a/paddle/gserver/tests/MKLDNNTester.h b/paddle/gserver/tests/MKLDNNTester.h index 51abfcb67e..c385d1c727 100644 --- a/paddle/gserver/tests/MKLDNNTester.h +++ b/paddle/gserver/tests/MKLDNNTester.h @@ -58,8 +58,6 @@ protected: size_t iter_; /// whether to print out the details bool log_; - /// vlog level to print the matrix details datas - int lvl_; /// epsilon float eps_; /// input image size, default 1 @@ -70,7 +68,6 @@ public: iter_ = iter; eps_ = epsilon; log_ = false; - lvl_ = MKLDNN_ALL; } ~MKLDNNTester() {} @@ -81,10 +78,9 @@ public: size_t batchSize, size_t inputImgH = 1, size_t inputImgW = 1, + bool printDetails = false, size_t iter = 3, - float epsilon = 1e-4, - bool log = false, - int level = MKLDNN_ALL); + float epsilon = 1e-4); static void runBranchesTest(const std::string& configPath, size_t iter = 3, float eps = 1e-4); diff --git a/paddle/trainer/tests/sample_trainer_config_branch_net.conf b/paddle/trainer/tests/sample_trainer_config_branch_net.conf index c2594bc13c..a073708a18 100644 --- a/paddle/trainer/tests/sample_trainer_config_branch_net.conf +++ b/paddle/trainer/tests/sample_trainer_config_branch_net.conf @@ -17,7 +17,7 @@ from paddle.trainer_config_helpers import * ################################### Data Configuration ################################### TrainData(ProtoData(files = "trainer/tests/mnist.list")) ################################### Algorithm Configuration ################################### -settings(batch_size = 256, +settings(batch_size = 128, learning_method = MomentumOptimizer(momentum=0.5, sparse=False)) ################################### Network Configuration ################################### data = data_layer(name ="input", size=784) @@ -44,10 +44,11 @@ a2 = img_conv_layer(input=tmp, shared_biases=True, act=ReluActivation()) -tmp = concat_layer(input=[a1, a2]) +tmp = addto_layer(input=[a1, a2], + act=ReluActivation(), + bias_attr=False) tmp = img_pool_layer(input=tmp, - num_channels=64, pool_size=3, stride=2, padding=1, @@ -55,35 +56,34 @@ tmp = img_pool_layer(input=tmp, b1 = img_conv_layer(input=tmp, filter_size=3, - num_filters=64, + num_filters=32, padding=1, shared_biases=True, act=ReluActivation()) b1 = img_pool_layer(input=b1, pool_size=3, - stride=1, - padding=1, + stride=2, + padding=0, pool_type=MaxPooling()) b2 = img_conv_layer(input=tmp, - filter_size=5, + filter_size=3, num_filters=64, - padding=2, + padding=1, shared_biases=True, act=ReluActivation()) b2 = img_pool_layer(input=b2, pool_size=5, - stride=1, - padding=2, + stride=2, + padding=1, pool_type=MaxPooling()) -tmp = addto_layer(input=[b1, b2], - act=ReluActivation(), - bias_attr=False) +tmp = concat_layer(input=[b1, b2]) tmp = img_pool_layer(input=tmp, + num_channels=96, pool_size=3, stride=2, padding=1, diff --git a/paddle/trainer/tests/sample_trainer_config_simple_net.conf b/paddle/trainer/tests/sample_trainer_config_simple_net.conf index 77f7816153..2ba71884d0 100644 --- a/paddle/trainer/tests/sample_trainer_config_simple_net.conf +++ b/paddle/trainer/tests/sample_trainer_config_simple_net.conf @@ -17,7 +17,7 @@ from paddle.trainer_config_helpers import * ################################### Data Configuration ################################### TrainData(ProtoData(files = "trainer/tests/mnist.list")) ################################### Algorithm Configuration ################################### -settings(batch_size = 1000, +settings(batch_size = 128, learning_method = MomentumOptimizer(momentum=0.5, sparse=False)) ################################### Network Configuration ################################### data = data_layer(name ="input", size=784) From af4dac4ac30cbf84bebadf09c823f0432300fa4d Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Thu, 19 Oct 2017 18:57:03 -0700 Subject: [PATCH 51/76] Feature/free kid scope (#4951) * Delete kid * Delete local scope --- paddle/framework/executor.cc | 3 +-- paddle/framework/scope.cc | 7 +++++++ paddle/framework/scope.h | 2 ++ 3 files changed, 10 insertions(+), 2 deletions(-) diff --git a/paddle/framework/executor.cc b/paddle/framework/executor.cc index d50f0da032..1f1e4edda8 100644 --- a/paddle/framework/executor.cc +++ b/paddle/framework/executor.cc @@ -84,8 +84,7 @@ void Executor::Run(const ProgramDesc& pdesc, Scope* scope, int block_id) { op->Run(local_scope, *device); } - // TODO(tonyyang-svail): - // - Destroy local_scope + scope->DeleteScope(&local_scope); } } // namespace framework diff --git a/paddle/framework/scope.cc b/paddle/framework/scope.cc index 5bf5e91f25..b8e116c430 100644 --- a/paddle/framework/scope.cc +++ b/paddle/framework/scope.cc @@ -65,6 +65,13 @@ void Scope::DropKids() { kids_.clear(); } +void Scope::DeleteScope(Scope* scope) { + auto it = std::find(this->kids_.begin(), this->kids_.end(), scope); + PADDLE_ENFORCE(it != this->kids_.end(), "Cannot find %p as kid scope", scope); + this->kids_.erase(it); + delete scope; +} + framework::Scope& GetGlobalScope() { static framework::Scope* g_scope = nullptr; if (g_scope == nullptr) { diff --git a/paddle/framework/scope.h b/paddle/framework/scope.h index a7fce3514b..78ff136ee1 100644 --- a/paddle/framework/scope.h +++ b/paddle/framework/scope.h @@ -59,6 +59,8 @@ class Scope { /// Find the scope or an ancestor scope that contains the given variable. const Scope* FindScope(const Variable* var) const; + void DeleteScope(Scope* scope); + /// Drop all kids scopes belonged to this scope. void DropKids(); From 9903e49f94e08fdfe64ca43d40ca1470cb00fbb3 Mon Sep 17 00:00:00 2001 From: QI JUN Date: Thu, 19 Oct 2017 19:34:41 -0700 Subject: [PATCH 52/76] add test_fit_a_line (#4936) * add test_fit_a_line * Update * fix persistable bug * fix elementwise add bug * set correct attr for bias op in fc layer * set correct attr for bias op in fc layer * Update 1. Add init_program to hold initializers 2. bug fix * add test_fit_a_line * fix persistable bug * fix elementwise add bug * fix type * add gitignore * Complete fit_a_line test * revert code * Clean up * Revert "revert code" This reverts commit eb1aa015cda4fc12b6dc778ada6c3507b98134f5. * Refine * Fix unit test --- paddle/operators/uniform_random_op.cc | 8 +- python/paddle/v2/framework/framework.py | 34 ++++----- python/paddle/v2/framework/layer_helper.py | 31 +++++--- python/paddle/v2/framework/layers.py | 12 ++- python/paddle/v2/framework/tests/.gitignore | 1 + .../v2/framework/tests/test_fit_a_line.py | 73 +++++++++++++++++++ .../framework/tests/test_uniform_random_op.py | 2 +- 7 files changed, 123 insertions(+), 38 deletions(-) create mode 100644 python/paddle/v2/framework/tests/.gitignore create mode 100644 python/paddle/v2/framework/tests/test_fit_a_line.py diff --git a/paddle/operators/uniform_random_op.cc b/paddle/operators/uniform_random_op.cc index 612bdd70db..f244ddc51f 100644 --- a/paddle/operators/uniform_random_op.cc +++ b/paddle/operators/uniform_random_op.cc @@ -53,10 +53,10 @@ class UniformRandomOp : public framework::OperatorWithKernel { PADDLE_ENFORCE( ctx->Attrs().Get("min") < ctx->Attrs().Get("max"), "uniform_random's min must less then max"); - auto& dims = ctx->Attrs().Get>("dims"); + auto& shape = ctx->Attrs().Get>("shape"); std::vector temp; - temp.reserve(dims.size()); - for (auto dim : dims) { + temp.reserve(shape.size()); + for (auto dim : shape) { temp.push_back(static_cast(dim)); } ctx->SetOutputDim("Out", framework::make_ddim(temp)); @@ -78,7 +78,7 @@ class UniformRandomOpMaker : public framework::OpProtoAndCheckerMaker { AddComment(R"DOC(Uniform random operator. Used to initialize tensor with uniform random generator. )DOC"); - AddAttr>("dims", "the dimension of random tensor"); + AddAttr>("shape", "the dimension of random tensor"); AddAttr("min", "Minimum value of uniform random").SetDefault(-1.0f); AddAttr("max", "Maximun value of uniform random").SetDefault(1.0f); AddAttr("seed", diff --git a/python/paddle/v2/framework/framework.py b/python/paddle/v2/framework/framework.py index 622e09fdde..03a3dacf25 100644 --- a/python/paddle/v2/framework/framework.py +++ b/python/paddle/v2/framework/framework.py @@ -15,7 +15,7 @@ class Variable(object): shape=None, dtype=None, lod_level=None, - persistable=False, + persistable=None, **kwargs): self.block = block @@ -343,6 +343,8 @@ class Block(object): def create_parameter(self, *args, **kwargs): global_block = self.program.global_block() param = Parameter(global_block, *args, **kwargs) + if 'init_attr' in kwargs: + self._prepend_initialize_ops_(param, kwargs['init_attr']) return param def append_op(self, *args, **kwargs): @@ -401,6 +403,17 @@ class Block(object): for index in range(len(self.ops)): assert self.ops[index].desc == ops_in_cpp[index] + def _prepend_initialize_ops_(self, param, init_attr): + op_type = init_attr['type'] + init_attr['shape'] = param.shape + init_attr['data_type'] = int(param.data_type) + op = self.prepend_op( + type=op_type, + inputs=None, + outputs={'Out': [param]}, + attrs=init_attr) + param.op = op + class Program(object): def __init__(self): @@ -475,27 +488,10 @@ class Parameter(Variable): Variable.__init__( self, block, persistable=True, shape=shape, dtype=dtype, **kwargs) self.trainable = kwargs.get('trainable', True) - self.init_attr = kwargs.get('initialize_attr', { - 'type': 'uniform_random', - 'min': -1.0, - 'max': 1.0 - }) self.optimize_attr = kwargs.get('optimize_attr', {'learning_rate': 1.0}) - self._append_initialize_ops_() - - def _append_initialize_ops_(self): - attr = self.init_attr - op_type = attr.pop('type', None) - block = self.block - assert isinstance(block, Block) - shape = self.shape - attr['dims'] = shape - attr['data_type'] = int(self.data_type) - op = block.prepend_op( - type=op_type, inputs=None, outputs={'Out': [self]}, attrs=attr) - self.op = op # program is a global instance. g_program = Program() +g_init_program = Program() diff --git a/python/paddle/v2/framework/layer_helper.py b/python/paddle/v2/framework/layer_helper.py index 6615bdcd3b..849a6f4306 100644 --- a/python/paddle/v2/framework/layer_helper.py +++ b/python/paddle/v2/framework/layer_helper.py @@ -1,4 +1,4 @@ -from paddle.v2.framework.framework import Variable, OpProtoHolder, g_program +from paddle.v2.framework.framework import Variable, OpProtoHolder, g_program, g_init_program import paddle.v2.framework.core as core import copy import itertools @@ -29,6 +29,14 @@ class LayerHelper(object): else: return prog + @property + def init_program(self): + prog = self.kwargs.get('init_program', None) + if prog is None: + return g_init_program + else: + return prog + def append_op(self, *args, **kwargs): return self.program.current_block().append_op(*args, **kwargs) @@ -66,16 +74,14 @@ class LayerHelper(object): actual = self.kwargs.get('param_attr', None) return actual if actual is not None else default - def bias_attr(self, shape, dtype): + def bias_attr(self): bias_attr = self.kwargs.get('bias_attr', None) if bias_attr is True: bias_attr = { 'name': None, 'init_attr': { 'type': 'fill_constant', - 'value': 0.0, - 'shape': shape, - 'dataType': dtype + 'value': 0.0 } } return bias_attr @@ -113,22 +119,27 @@ class LayerHelper(object): def create_parameter(self, attr, shape, dtype, suffix='w'): if attr['name'] is None: attr['name'] = unique_name(".".join([self.name, suffix])) - return self.program.global_block().create_parameter( + self.init_program.global_block().create_parameter( name=attr['name'], dtype=dtype, shape=shape, - initialize_attr=attr['init_attr']) + init_attr=attr['init_attr']) + return self.program.global_block().create_parameter( + name=attr['name'], dtype=dtype, shape=shape) def create_tmp_variable(self, dtype): return self.program.current_block().create_var( - name=unique_name(".".join([self.name, 'tmp'])), dtype=dtype) + name=unique_name(".".join([self.name, 'tmp'])), + dtype=dtype, + persistable=False) def create_global_variable(self, *args, **kwargs): - return self.program.global_block().create_var(*args, **kwargs) + return self.program.global_block().create_var( + *args, persistable=False, **kwargs) def append_bias_op(self, input_var): size = list(input_var.shape[1:]) - bias_attr = self.bias_attr(size, dtype=input_var.data_type) + bias_attr = self.bias_attr() if not bias_attr: return input_var diff --git a/python/paddle/v2/framework/layers.py b/python/paddle/v2/framework/layers.py index 236427efce..ac77aefa15 100644 --- a/python/paddle/v2/framework/layers.py +++ b/python/paddle/v2/framework/layers.py @@ -13,7 +13,8 @@ def fc(input, name=None, act=None, num_flatten_dims=1, - program=None): + program=None, + init_program=None): # create helper helper = LayerHelper('fc', **locals()) @@ -59,7 +60,8 @@ def data(name, data_type='float32', type=core.VarDesc.VarType.LOD_TENSOR, append_batch_size=True, - program=None): + program=None, + init_program=None): helper = LayerHelper('data', **locals()) if append_batch_size: shape = [-1] + shape # append batch size as -1 @@ -160,7 +162,8 @@ def conv2d(input, padding=None, bias_attr=None, param_attr=None, - program=None): + program=None, + init_program=None): helper = LayerHelper('conv2d', **locals()) dtype = helper.input_dtype() @@ -207,7 +210,8 @@ def pool2d(input, pool_stride=[1, 1], pool_padding=[0, 0], global_pooling=False, - program=None): + program=None, + init_program=None): if pool_type not in ["max", "avg"]: raise ValueError( "Unknown pool_type: '%s'. It can only be 'max' or 'avg'.", diff --git a/python/paddle/v2/framework/tests/.gitignore b/python/paddle/v2/framework/tests/.gitignore new file mode 100644 index 0000000000..28433306d4 --- /dev/null +++ b/python/paddle/v2/framework/tests/.gitignore @@ -0,0 +1 @@ +image/ diff --git a/python/paddle/v2/framework/tests/test_fit_a_line.py b/python/paddle/v2/framework/tests/test_fit_a_line.py new file mode 100644 index 0000000000..b20e335789 --- /dev/null +++ b/python/paddle/v2/framework/tests/test_fit_a_line.py @@ -0,0 +1,73 @@ +import paddle.v2 as paddle +import paddle.v2.framework.layers as layers +import paddle.v2.framework.core as core +import paddle.v2.framework.optimizer as optimizer + +from paddle.v2.framework.framework import Program, g_program +from paddle.v2.framework.executor import Executor + +import numpy as np + +init_program = Program() +program = Program() +x = layers.data( + name='x', + shape=[13], + data_type='float32', + program=program, + init_program=init_program) + +y_predict = layers.fc(input=x, + size=1, + act=None, + program=program, + init_program=init_program) + +y = layers.data( + name='y', + shape=[1], + data_type='float32', + program=program, + init_program=init_program) + +cost = layers.square_error_cost( + input=y_predict, label=y, program=program, init_program=init_program) +avg_cost = layers.mean(x=cost, program=program, init_program=init_program) + +sgd_optimizer = optimizer.SGDOptimizer(learning_rate=0.001) +opts = sgd_optimizer.minimize(avg_cost) + +BATCH_SIZE = 20 + +train_reader = paddle.batch( + paddle.reader.shuffle( + paddle.dataset.uci_housing.train(), buf_size=500), + batch_size=BATCH_SIZE) + +place = core.CPUPlace() +exe = Executor(place) + +exe.run(init_program, feed={}, fetch_list=[]) + +PASS_NUM = 100 +for pass_id in range(PASS_NUM): + for data in train_reader(): + x_data = np.array(map(lambda x: x[0], data)).astype("float32") + y_data = np.array(map(lambda x: x[1], data)).astype("float32") + + tensor_x = core.LoDTensor() + tensor_x.set(x_data, place) + # print tensor_x.get_dims() + + tensor_y = core.LoDTensor() + tensor_y.set(y_data, place) + # print tensor_y.get_dims() + outs = exe.run(program, + feed={'x': tensor_x, + 'y': tensor_y}, + fetch_list=[avg_cost]) + out = np.array(outs[0]) + + if out[0] < 10.0: + exit(0) # if avg cost less than 10.0, we think our code is good. +exit(1) diff --git a/python/paddle/v2/framework/tests/test_uniform_random_op.py b/python/paddle/v2/framework/tests/test_uniform_random_op.py index a2d28a65a6..ded777105e 100644 --- a/python/paddle/v2/framework/tests/test_uniform_random_op.py +++ b/python/paddle/v2/framework/tests/test_uniform_random_op.py @@ -19,7 +19,7 @@ class TestUniformRandomOp(unittest.TestCase): op = Operator( "uniform_random", Out='X', - dims=[1000, 784], + shape=[1000, 784], min=-5.0, max=10.0, seed=10) From 102a5f349926539c256afca54108241cc5e313c6 Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Thu, 19 Oct 2017 19:43:31 -0700 Subject: [PATCH 53/76] Feature/remove global scope (#4950) * Unify `set_feed_variable` to one method * Move global scope to python, not in C++ --- paddle/framework/feed_fetch_method.h | 11 ++++++----- paddle/framework/scope.cc | 8 -------- paddle/framework/scope.h | 3 --- paddle/pybind/pybind.cc | 12 +++++------- python/paddle/v2/framework/executor.py | 14 ++++++++++---- .../v2/framework/tests/test_feed_fetch_method.py | 5 +++-- 6 files changed, 24 insertions(+), 29 deletions(-) diff --git a/paddle/framework/feed_fetch_method.h b/paddle/framework/feed_fetch_method.h index 3ef70043d6..7feacb1e24 100644 --- a/paddle/framework/feed_fetch_method.h +++ b/paddle/framework/feed_fetch_method.h @@ -21,12 +21,12 @@ limitations under the License. */ namespace paddle { namespace framework { -void SetFeedVariable(const LoDTensor& input, const std::string& var_name, - size_t index) { +void SetFeedVariable(Scope* scope, const LoDTensor& input, + const std::string& var_name, size_t index) { // If var_name Variable is not found in GlobalScope, a new variable will // be created. VLOG(3) << "SetFeedVariable name=" << var_name << " index=" << index; - Variable* g_feed_value = GetGlobalScope().Var(var_name); + Variable* g_feed_value = scope->Var(var_name); auto& feed_inputs = *(g_feed_value->GetMutable>()); if (index >= feed_inputs.size()) { @@ -38,10 +38,11 @@ void SetFeedVariable(const LoDTensor& input, const std::string& var_name, feed_inputs[index].set_lod(input.lod()); } -LoDTensor& GetFetchVariable(const std::string& var_name, size_t index) { +LoDTensor& GetFetchVariable(const Scope& scope, const std::string& var_name, + size_t index) { // Since we want to fetch LodTensor from a variable, the variable must // be created alreadly. - Variable* g_fetch_value = GetGlobalScope().FindVar(var_name); + Variable* g_fetch_value = scope.FindVar(var_name); PADDLE_ENFORCE(g_fetch_value->IsType(), "Only %s can be invoked by GetFetchVariable", typeid(FeedFetchList).name()); diff --git a/paddle/framework/scope.cc b/paddle/framework/scope.cc index b8e116c430..ac3ac649f9 100644 --- a/paddle/framework/scope.cc +++ b/paddle/framework/scope.cc @@ -72,13 +72,5 @@ void Scope::DeleteScope(Scope* scope) { delete scope; } -framework::Scope& GetGlobalScope() { - static framework::Scope* g_scope = nullptr; - if (g_scope == nullptr) { - g_scope = new framework::Scope(); - } - return *g_scope; -} - } // namespace framework } // namespace paddle diff --git a/paddle/framework/scope.h b/paddle/framework/scope.h index 78ff136ee1..7206b53068 100644 --- a/paddle/framework/scope.h +++ b/paddle/framework/scope.h @@ -74,8 +74,5 @@ class Scope { DISABLE_COPY_AND_ASSIGN(Scope); }; - -framework::Scope& GetGlobalScope(); - } // namespace framework } // namespace paddle diff --git a/paddle/pybind/pybind.cc b/paddle/pybind/pybind.cc index 94c9706f79..9ef47b88fd 100644 --- a/paddle/pybind/pybind.cc +++ b/paddle/pybind/pybind.cc @@ -219,8 +219,7 @@ All parameter, weight, gradient are variables in Paddle. .def(py::init<>()) .def("new_scope", [](Scope &self) -> Scope * { return &self.NewScope(); }, py::return_value_policy::reference) - .def("drop_kids", &Scope::DropKids) - .def_static("global_scope", &GetGlobalScope); + .def("drop_kids", &Scope::DropKids); //! @note: Be careful! PyBind will return std::string as an unicode, not //! Python str. If you want a str object, you should cast them in Python. @@ -451,11 +450,10 @@ All parameter, weight, gradient are variables in Paddle. py::class_(m, "Executor") .def(py::init &>()) - .def("run", - [](Executor &self, ProgramDescBind *program_bind, int block_id) { - framework::Scope &global_scope = GetGlobalScope(); - self.Run(*program_bind->Proto(), &global_scope, block_id); - }); + .def("run", [](Executor &self, ProgramDescBind *program_bind, + Scope *scope, int block_id) { + self.Run(*program_bind->Proto(), scope, block_id); + }); m.def("unique_integer", UniqueIntegerGenerator); diff --git a/python/paddle/v2/framework/executor.py b/python/paddle/v2/framework/executor.py index 1adc10c233..82b83d4bb6 100644 --- a/python/paddle/v2/framework/executor.py +++ b/python/paddle/v2/framework/executor.py @@ -1,6 +1,8 @@ import paddle.v2.framework.core as core from paddle.v2.framework.framework import Block, Program +g_scope = core.Scope() + class Executor(object): def __init__(self, places): @@ -20,10 +22,14 @@ class Executor(object): feed, fetch_list, feed_var_name='feed', - fetch_var_name='fetch'): + fetch_var_name='fetch', + scope=None): if not isinstance(program, Program): raise TypeError() + if scope is None: + scope = g_scope + program = program.clone() global_block = program.global_block() feed_var = global_block.create_var( @@ -38,7 +44,7 @@ class Executor(object): inputs={'X': [feed_var]}, outputs={'Out': [out]}, attrs={'col': i}) - core.set_feed_variable(feed[name], feed_var.name, i) + core.set_feed_variable(scope, feed[name], feed_var.name, i) fetch_var = global_block.create_var( name=fetch_var_name, @@ -51,8 +57,8 @@ class Executor(object): outputs={'Out': [fetch_var]}, attrs={'col': i}) - self.executor.run(program.desc, 0) + self.executor.run(program.desc, scope, 0) return [ - core.get_fetch_variable(fetch_var_name, i) + core.get_fetch_variable(scope, fetch_var_name, i) for i in xrange(len(fetch_list)) ] diff --git a/python/paddle/v2/framework/tests/test_feed_fetch_method.py b/python/paddle/v2/framework/tests/test_feed_fetch_method.py index 8b9b44440d..fbd659ece0 100644 --- a/python/paddle/v2/framework/tests/test_feed_fetch_method.py +++ b/python/paddle/v2/framework/tests/test_feed_fetch_method.py @@ -5,6 +5,7 @@ import numpy as np class TestFeedFetch(unittest.TestCase): def test_feed_fetch(self): + scope = core.Scope() place = core.CPUPlace() input_array = np.ones((4, 4, 6)).astype("float32") input_array[0, 0, 0] = 3 @@ -12,9 +13,9 @@ class TestFeedFetch(unittest.TestCase): input_tensor = core.LoDTensor([[0, 2, 4]]) input_tensor.set(input_array, place) - core.set_feed_variable(input_tensor, "feed", 0) + core.set_feed_variable(scope, input_tensor, "feed", 0) - output_tensor = core.get_fetch_variable("feed", 0) + output_tensor = core.get_fetch_variable(scope, "feed", 0) output_lod = output_tensor.lod() self.assertEqual(0, output_lod[0][0]) From 5d2fe7cd917432f3873cc1c7de6648d50d2d9a9f Mon Sep 17 00:00:00 2001 From: hedaoyuan Date: Fri, 20 Oct 2017 11:06:59 +0800 Subject: [PATCH 54/76] Fix cc_library paddle_capi_whole. --- paddle/capi/CMakeLists.txt | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/paddle/capi/CMakeLists.txt b/paddle/capi/CMakeLists.txt index e966d5d852..4ff82bafad 100644 --- a/paddle/capi/CMakeLists.txt +++ b/paddle/capi/CMakeLists.txt @@ -38,7 +38,6 @@ if(MOBILE_INFERENCE) paddle_function paddle_gserver paddle_proto) - cc_library(paddle_capi_whole DEPS paddle_capi ${PADDLE_CAPI_INFER_LIBS}) else() set(PADDLE_CAPI_INFER_LIBS paddle_utils @@ -50,8 +49,8 @@ else() paddle_proto paddle_pserver paddle_network) - cc_library(paddle_capi_whole DEPS paddle_capi ${PADDLE_CAPI_INFER_LIBS}) endif() +cc_library(paddle_capi_whole DEPS paddle_capi ${PADDLE_CAPI_INFER_LIBS}) # Link the static library for inference cc_library(paddle_capi_engine DEPS paddle_capi paddle_utils paddle_parameter paddle_math paddle_cuda paddle_proto) From 8278d97e3aeea3952a53705591a5d2ebf8245dc8 Mon Sep 17 00:00:00 2001 From: qijun Date: Thu, 19 Oct 2017 20:58:42 -0700 Subject: [PATCH 55/76] add book02.recognize_digits mlp train test --- .../framework/tests/test_cross_entropy_op.py | 2 +- .../tests/test_recognize_digits_mlp.py | 83 +++++++++++++++++++ 2 files changed, 84 insertions(+), 1 deletion(-) create mode 100644 python/paddle/v2/framework/tests/test_recognize_digits_mlp.py diff --git a/python/paddle/v2/framework/tests/test_cross_entropy_op.py b/python/paddle/v2/framework/tests/test_cross_entropy_op.py index 919b6c3f67..e1c45c2674 100644 --- a/python/paddle/v2/framework/tests/test_cross_entropy_op.py +++ b/python/paddle/v2/framework/tests/test_cross_entropy_op.py @@ -21,7 +21,7 @@ class TestCrossEntropyOp1(OpTest): self.inputs = {"X": X, "Label": label} self.outputs = {"Y": cross_entropy} - self.attrs = {"softLabel": False} + self.attrs = {"soft_label": False} def test_check_output(self): self.check_output() diff --git a/python/paddle/v2/framework/tests/test_recognize_digits_mlp.py b/python/paddle/v2/framework/tests/test_recognize_digits_mlp.py new file mode 100644 index 0000000000..a985d1f3d3 --- /dev/null +++ b/python/paddle/v2/framework/tests/test_recognize_digits_mlp.py @@ -0,0 +1,83 @@ +import paddle.v2 as paddle +import paddle.v2.framework.layers as layers +import paddle.v2.framework.core as core +import paddle.v2.framework.optimizer as optimizer + +from paddle.v2.framework.framework import Program, g_program +from paddle.v2.framework.executor import Executor + +import numpy as np + +init_program = Program() +program = Program() +image = layers.data( + name='x', + shape=[784], + data_type='float32', + program=program, + init_program=init_program) + +hidden1 = layers.fc(input=image, + size=128, + act='relu', + program=program, + init_program=init_program) +hidden2 = layers.fc(input=hidden1, + size=64, + act='relu', + program=program, + init_program=init_program) + +predict = layers.fc(input=hidden2, + size=10, + act='softmax', + program=program, + init_program=init_program) + +label = layers.data( + name='y', + shape=[1], + data_type='int32', + program=program, + init_program=init_program) + +cost = layers.cross_entropy( + input=predict, label=label, program=program, init_program=init_program) +avg_cost = layers.mean(x=cost, program=program, init_program=init_program) + +sgd_optimizer = optimizer.SGDOptimizer(learning_rate=0.001) +opts = sgd_optimizer.minimize(avg_cost) + +BATCH_SIZE = 128 + +train_reader = paddle.batch( + paddle.reader.shuffle( + paddle.dataset.mnist.train(), buf_size=8192), + batch_size=BATCH_SIZE) + +place = core.CPUPlace() +exe = Executor(place) + +exe.run(init_program, feed={}, fetch_list=[]) + +PASS_NUM = 100 +for pass_id in range(PASS_NUM): + for data in train_reader(): + x_data = np.array(map(lambda x: x[0], data)).astype("float32") + y_data = np.array(map(lambda x: x[1], data)).astype("int32") + y_data = np.expand_dims(y_data, axis=1) + + tensor_x = core.LoDTensor() + tensor_x.set(x_data, place) + + tensor_y = core.LoDTensor() + tensor_y.set(y_data, place) + + outs = exe.run(program, + feed={'x': tensor_x, + 'y': tensor_y}, + fetch_list=[avg_cost]) + out = np.array(outs[0]) + if out[0] < 5.0: + exit(0) # if avg cost less than 5.0, we think our code is good. +exit(1) From 0e31d7d71b330ef5335b17605ce6845d349fb5c9 Mon Sep 17 00:00:00 2001 From: Abhinav Arora Date: Thu, 19 Oct 2017 21:03:33 -0700 Subject: [PATCH 56/76] Adding the interface for the momentum optimizer (#4919) * Adding the interface for the momentum optimizer * Adding a comment about accumulators --- python/paddle/v2/framework/optimizer.py | 188 ++++++++++++++++-- .../v2/framework/tests/test_optimizer.py | 46 ++++- 2 files changed, 213 insertions(+), 21 deletions(-) diff --git a/python/paddle/v2/framework/optimizer.py b/python/paddle/v2/framework/optimizer.py index e356a7aadb..f992a42c40 100644 --- a/python/paddle/v2/framework/optimizer.py +++ b/python/paddle/v2/framework/optimizer.py @@ -1,32 +1,104 @@ import paddle.v2.framework.framework as framework +from collections import defaultdict -__all__ = ['SGDOptimizer'] +__all__ = ['SGDOptimizer', 'MomentumOptimizer'] class Optimizer(object): """Optimizer Base class. Define the common interface of an optimizer. - User should not use this class directly, but need to use one of it's implementation. + User should not use this class directly, + but need to use one of it's implementation. """ def __init__(self): - pass + # Dictionary of accumulators. Some optimizer subclasses need to + # allocate and manage extra variables associated with the parameters + # to train. These variables are called accumulators. + # {accum_name : { paramter_name : accumulator_for_parameter, ...}, ...} + self._accumulators = defaultdict(lambda: dict()) def _append_optimize_op(self, block, param_and_grad): """ append optimize operator to block and return all the added optimize_op """ raise NotImplementedError() - def create_backward_pass(self, loss, parameter_list=None, no_grad_set=None): + def _initialize_tensors(self, block): + """Create all necessary tensors, that will be shared for all parameter updates. + + Tensors like learning rate should be initialized here. + + Args: + block: the block in which the loss variable is present + """ + pass + + def _create_accumulators(self, block, parameters): + """Create all accumulators needed by the parameters + + Args: + block: the block in which the loss variable is present + parameters: list of parameter variables for the optimizer """ - create and add gradient Operators in BlockDesc to Compute gradients of `loss` - for parameters in parameter_list + pass + + def _add_accumulator(self, block, name, param, dtype=None, fill_value=0.0): + """Utility function to add an accumulator for a parameter + + Args: + block: the block in which the loss variable is present + name: name of the accumulator + param: parameter variable for which accumulator is to be added + dtype: data type of the accumulator variable + fill_value: value to initialize the accumulator variable + """ + if (name in self._accumulators and + param.name in self._accumulators[name]): + raise Exception("Accumulator {} already exists for parmeter {}". + format(name, param.name)) + global_block = block.program.global_block() + param_shape = list(param.shape) + param_acc = global_block.create_var( + dtype=dtype, shape=param_shape, lod_level=0) + + # Initialize the accumulator with fill_value + # FIXME: Fix when Initialization design has been implemented + # https://github.com/PaddlePaddle/Paddle/pull/4852 + global_block.append_op( + type="fill_constant", + outputs={"Out": param_acc}, + attrs={"shape": param_shape, + "value": fill_value}) + + # Add to accumulators dict + self._accumulators[name][param.name] = param_acc + + def _get_accumulator(self, name, param): + """Utility function to fetch an accumulator for a parameter + + Args: + name: name of the accumulator + param: parameter variable for which accumulator is to be fetched + + Returns: + accumulator variable for the parameter + """ + if (name not in self._accumulators or + param.name not in self._accumulators[name]): + raise Exception("Accumulator {} does not exist for parameter {}". + format(name, param.name)) + return self._accumulators[name][param.name] + + def create_backward_pass(self, loss, parameter_list=None, no_grad_set=None): + """Create and add gradient Operators in BlockDesc to compute + gradients of `loss` for parameters in parameter_list Args: loss: an variable generated by cost function. no_grad_set: variable that should not create gradient - parameter_list: parameters that need to compute gradient and update to optimize the lost. + parameter_list: parameters that need to compute gradient and + update to optimize the lost. Returns: list of (parameters, gradients) pair. @@ -48,7 +120,8 @@ class Optimizer(object): if not grad_block.has_var(grad_info[0]): raise Exception("grad block[%d] did not have grad var %s" % grad_info[1], grad_info[0]) - param_var = loss.block.var(param) + # Get the param var from the global block + param_var = loss.block.program.global_block().var(param) grad_var = grad_block.var(grad_info[0]) if loss.block.has_var(grad_info[0]): params_and_grads.append((param_var, grad_var)) @@ -64,14 +137,29 @@ class Optimizer(object): parameters_and_grads: a list of (variable, gradient) pair to update. Returns: - optmization_op_list: a list of optimization operator that will update parameter using gradient. + optmization_op_list: a list of optimization operator that will update + parameter using gradient. """ + # This is a default implementation of create_optimization_pass that + # can be shared by most optimizers. This implementation assumes that + # the subclass will implement the _append_optimize_op method and the + # _initialize_tensors method. The subclass can extend the + # _create_accumulators method if it needs to create accumulators + # for parameters. + + # Create any accumulators + self._create_accumulators(loss.block, + [p[0] for p in parameters_and_grads]) + # Create any necessary tensors + self._initialize_tensors(loss.block) + optimize_ops = [] for param_and_grad in parameters_and_grads: if param_and_grad[1] is not None: optimize_op = self._append_optimize_op(loss.block, param_and_grad) optimize_ops.append(optimize_op) + return optimize_ops def minimize(self, loss, parameter_list=None, no_grad_set=None): @@ -92,33 +180,95 @@ class SGDOptimizer(Optimizer): def __init__(self, learning_rate): assert learning_rate is not None - super(Optimizer, self).__init__() + super(SGDOptimizer, self).__init__() self.type = "sgd" self._learning_rate = learning_rate - def _append_optimize_op(self, block, param_and_grad): + def _initialize_tensors(self, block): assert isinstance(block, framework.Block) lr_shape = [1] - # create a var for learning_rate - lr = block.create_var(dtype="float32", shape=lr_shape, lod_level=0) + # create a variable for learning_rate + self._lr = block.create_var( + dtype="float32", shape=lr_shape, lod_level=0) # create an op to init the learning_rate - init_op = block.append_op( + # FIXME: Fix when Initialization design has been implemented + # https://github.com/PaddlePaddle/Paddle/pull/4852 + block.append_op( type="fill_constant", - outputs={"Out": lr}, + outputs={"Out": self._lr}, attrs={"shape": lr_shape, "value": self._learning_rate}) + def _append_optimize_op(self, block, param_and_grad): + assert isinstance(block, framework.Block) + # create the optimize op sgd_op = block.append_op( type=self.type, inputs={ "Param": param_and_grad[0], "Grad": param_and_grad[1], - "LearningRate": lr + "LearningRate": self._lr }, - outputs={"ParamOut": param_and_grad[0]}, - attrs={"shape": [1], - "value": self._learning_rate}) + outputs={"ParamOut": param_and_grad[0]}) return sgd_op + + +class MomentumOptimizer(Optimizer): + """Simple Momentum optimizer with velocity state + """ + _velocity_acc_str = "velocity" + + def __init__(self, learning_rate, momentum): + assert learning_rate is not None + assert momentum is not None + super(MomentumOptimizer, self).__init__() + self.type = "momentum" + self._learning_rate = learning_rate + self._momentum = momentum + + def _initialize_tensors(self, block): + assert isinstance(block, framework.Block) + lr_shape = [1] + # create a variable for learning_rate + self._lr = block.create_var( + dtype="float32", shape=lr_shape, lod_level=0) + + # create an op to init the learning_rate + # FIXME: Fix when Initialization design has been implemented + # https://github.com/PaddlePaddle/Paddle/pull/4852 + block.append_op( + type="fill_constant", + outputs={"Out": self._lr}, + attrs={"shape": lr_shape, + "value": self._learning_rate}) + + def _create_accumulators(self, block, parameters): + assert isinstance(block, framework.Block) + + for p in parameters: + self._add_accumulator(block, self._velocity_acc_str, p, 'float32') + + def _append_optimize_op(self, block, param_and_grad): + assert isinstance(block, framework.Block) + + velocity_acc = self._get_accumulator(self._velocity_acc_str, + param_and_grad[0]) + # create the momentum optimize op + momentum_op = block.append_op( + type=self.type, + inputs={ + "Param": param_and_grad[0], + "Grad": param_and_grad[1], + "Velocity": velocity_acc, + "LearningRate": self._lr + }, + outputs={ + "ParamOut": param_and_grad[0], + "VelocityOut": velocity_acc + }, + attrs={"mu": self._momentum}) + + return momentum_op diff --git a/python/paddle/v2/framework/tests/test_optimizer.py b/python/paddle/v2/framework/tests/test_optimizer.py index 3d6fa70737..e6a142ac36 100644 --- a/python/paddle/v2/framework/tests/test_optimizer.py +++ b/python/paddle/v2/framework/tests/test_optimizer.py @@ -6,7 +6,7 @@ import paddle.v2.framework.optimizer as optimizer class TestOptimizer(unittest.TestCase): def test_sgd_optimizer(self): - program = framework.g_program + program = framework.Program() block = program.global_block() mul_x = block.create_parameter( dtype="float32", shape=[5, 10], lod_level=0, name="mul.x") @@ -14,7 +14,7 @@ class TestOptimizer(unittest.TestCase): dtype="float32", shape=[10, 8], lod_level=0, name="mul.y") mul_out = block.create_var( dtype="float32", shape=[5, 8], lod_level=0, name="mul.out") - mul_op = block.append_op( + block.append_op( type="mul", inputs={"X": mul_x, "Y": mul_y}, @@ -27,5 +27,47 @@ class TestOptimizer(unittest.TestCase): self.assertEqual(sgd_op.type, "sgd") +class TestMomentumOptimizer(unittest.TestCase): + class MockMomentum(optimizer.MomentumOptimizer): + def get_accumulators(self): + return self._accumulators + + def get_velocity_str(self): + return self._velocity_acc_str + + def test_momentum_optimizer(self): + program = framework.Program() + block = program.global_block() + mul_x = block.create_parameter( + dtype="float32", shape=[5, 10], lod_level=0, name="mul.x") + mul_y = block.create_var( + dtype="float32", shape=[10, 8], lod_level=0, name="mul.y") + mul_out = block.create_var( + dtype="float32", shape=[5, 8], lod_level=0, name="mul.out") + block.append_op( + type="mul", + inputs={"X": mul_x, + "Y": mul_y}, + outputs={"Out": mul_out}, + attrs={"x_num_col_dims": 1}) + momentum_optimizer = self.MockMomentum(learning_rate=0.01, momentum=0.2) + params_grads = momentum_optimizer.create_backward_pass(mul_out) + self.assertEqual(len(params_grads), 1) + self.assertEqual(len(momentum_optimizer.get_accumulators()), 0) + opts = momentum_optimizer.create_optimization_pass(params_grads, + mul_out) + self.assertEqual(len(opts), 1) + sgd_op = opts[0] + self.assertEqual(sgd_op.type, "momentum") + + # Check accumulators + accumulators = momentum_optimizer.get_accumulators() + self.assertEqual(len(accumulators), 1) + self.assertTrue(momentum_optimizer.get_velocity_str() in accumulators) + velocity_acc = accumulators[momentum_optimizer.get_velocity_str()] + self.assertEqual(len(velocity_acc), 1) + self.assertTrue(mul_x.name in velocity_acc) + + if __name__ == '__main__': unittest.main() From 09c0c82ec9e5c2bff8da4a8598e80ea962fce390 Mon Sep 17 00:00:00 2001 From: Abhinav Arora Date: Thu, 19 Oct 2017 22:51:13 -0700 Subject: [PATCH 57/76] Adding increment op (#4940) * Adding incremnt op * Fixing comment about step attribute --- paddle/operators/increment_op.cc | 75 +++++++++++++++++++ paddle/operators/increment_op.cu | 19 +++++ paddle/operators/increment_op.h | 40 ++++++++++ .../v2/framework/tests/test_increment_op.py | 41 ++++++++++ 4 files changed, 175 insertions(+) create mode 100644 paddle/operators/increment_op.cc create mode 100644 paddle/operators/increment_op.cu create mode 100644 paddle/operators/increment_op.h create mode 100644 python/paddle/v2/framework/tests/test_increment_op.py diff --git a/paddle/operators/increment_op.cc b/paddle/operators/increment_op.cc new file mode 100644 index 0000000000..139392c691 --- /dev/null +++ b/paddle/operators/increment_op.cc @@ -0,0 +1,75 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include "paddle/operators/increment_op.h" + +namespace paddle { +namespace operators { + +class IncrementOp : public framework::OperatorWithKernel { + public: + using framework::OperatorWithKernel::OperatorWithKernel; + + void InferShape(framework::InferShapeContext *ctx) const override { + PADDLE_ENFORCE(ctx->HasInput("X"), + "Input(X) of IncrementOp should not be null."); + PADDLE_ENFORCE(ctx->HasOutput("Out"), + "Output(Out) of IncrementOp should not be null."); + ctx->SetOutputDim("Out", ctx->GetInputDim("X")); + ctx->ShareLoD("X", /*->*/ "Out"); + } +}; + +template +class IncrementOpMaker : public framework::OpProtoAndCheckerMaker { + public: + IncrementOpMaker(framework::OpProto *proto, + framework::OpAttrChecker *op_checker) + : OpProtoAndCheckerMaker(proto, op_checker) { + AddInput("X", "(Tensor) The input tensor of increment operator"); + AddOutput("Out", "(Tensor) The output tensor of increment operator."); + AddComment(R"DOC(Increment operator + +The equation is: Out = X + step +)DOC"); + AddAttr("step", + "The step size by which the " + "input tensor will be incremented.") + .SetDefault(1.0); + } +}; + +class IncrementGradOpMaker : public framework::SingleGradOpDescMaker { + public: + using framework::SingleGradOpDescMaker::SingleGradOpDescMaker; + + std::unique_ptr Apply() const override { + auto *grad_op = new framework::OpDescBind(); + grad_op->SetType("scale"); + grad_op->SetInput("X", OutputGrad("Out")); + grad_op->SetOutput("Out", InputGrad("X")); + grad_op->SetAttr("scale", 1.0f); + return std::unique_ptr(grad_op); + } +}; + +} // namespace operators +} // namespace paddle + +namespace ops = paddle::operators; + +REGISTER_OPERATOR(increment, ops::IncrementOp, ops::IncrementOpMaker, + ops::IncrementGradOpMaker); +REGISTER_OP_CPU_KERNEL(increment, + ops::IncrementKernel); diff --git a/paddle/operators/increment_op.cu b/paddle/operators/increment_op.cu new file mode 100644 index 0000000000..659c380d14 --- /dev/null +++ b/paddle/operators/increment_op.cu @@ -0,0 +1,19 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include "paddle/operators/increment_op.h" + +REGISTER_OP_GPU_KERNEL( + increment, + paddle::operators::IncrementKernel); diff --git a/paddle/operators/increment_op.h b/paddle/operators/increment_op.h new file mode 100644 index 0000000000..342e254fc4 --- /dev/null +++ b/paddle/operators/increment_op.h @@ -0,0 +1,40 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#pragma once + +#include "paddle/framework/eigen.h" +#include "paddle/framework/op_registry.h" + +namespace paddle { +namespace operators { +template +class IncrementKernel : public framework::OpKernel { + public: + virtual void Compute(const framework::ExecutionContext& context) const { + auto* tensor = context.Output("Out"); + auto* in = context.Input("X"); + tensor->mutable_data(in->place()); + + auto step = static_cast(context.Attr("step")); + + auto eigen_out = framework::EigenVector::Flatten(*tensor); + auto eigen_in = framework::EigenVector::Flatten(*in); + auto& place = context.GetEigenDevice(); + eigen_out.device(place) = eigen_in + step; + } +}; + +} // namespace operators +} // namespace paddle diff --git a/python/paddle/v2/framework/tests/test_increment_op.py b/python/paddle/v2/framework/tests/test_increment_op.py new file mode 100644 index 0000000000..e174272b05 --- /dev/null +++ b/python/paddle/v2/framework/tests/test_increment_op.py @@ -0,0 +1,41 @@ +import unittest +import numpy as np +from op_test import OpTest + + +class TestIncrementOpPositiveStep(OpTest): + """Test increment op with positive step + """ + + def setUp(self): + self.op_type = "increment" + self.inputs = {'X': np.random.random((10, 10)).astype("float32")} + self.attrs = {'step': 14.8} + self.outputs = {'Out': self.inputs['X'] + self.attrs['step']} + + def test_check_output(self): + self.check_output() + + def test_check_grad(self): + self.check_grad(['X'], 'Out') + + +class TestIncrementOpNegativeStep(OpTest): + """Test increment op with negative step + """ + + def setUp(self): + self.op_type = "increment" + self.inputs = {'X': np.random.random((10, 10)).astype("float32")} + self.attrs = {'step': -3.8} + self.outputs = {'Out': self.inputs['X'] + self.attrs['step']} + + def test_check_output(self): + self.check_output() + + def test_check_grad(self): + self.check_grad(['X'], 'Out') + + +if __name__ == "__main__": + unittest.main() From 2bb2c318e9ddf0eec8313d43be97dfe20b16e127 Mon Sep 17 00:00:00 2001 From: hedaoyuan Date: Fri, 20 Oct 2017 15:09:13 +0800 Subject: [PATCH 58/76] Change the name of the export.map to paddle_capi.map which need to be released in each version. --- paddle/capi/CMakeLists.txt | 4 ++-- paddle/capi/export.sym | 0 paddle/capi/{export.map => paddle_capi.map} | 0 3 files changed, 2 insertions(+), 2 deletions(-) delete mode 100644 paddle/capi/export.sym rename paddle/capi/{export.map => paddle_capi.map} (100%) diff --git a/paddle/capi/CMakeLists.txt b/paddle/capi/CMakeLists.txt index 4ff82bafad..e767856d50 100644 --- a/paddle/capi/CMakeLists.txt +++ b/paddle/capi/CMakeLists.txt @@ -58,8 +58,7 @@ cc_library(paddle_capi_layers DEPS paddle_function paddle_gserver) # Link the shared library for inference if(NOT IOS) - set(LINK_FLAGS " -Wl,--retain-symbols-file ${CMAKE_CURRENT_SOURCE_DIR}/export.sym -Wl,--version-script ${CMAKE_CURRENT_SOURCE_DIR}/export.map") - # TODO: merge mkl into paddle_capi_shared + set(LINK_FLAGS "-Wl,--version-script ${CMAKE_CURRENT_SOURCE_DIR}/paddle_capi.map") add_library(paddle_capi_shared SHARED ${CAPI_SOURCES}) set_target_properties(paddle_capi_shared PROPERTIES LINK_FLAGS "${LINK_FLAGS}") target_include_directories(paddle_capi_shared PUBLIC ${CMAKE_CURRENT_BINARY_DIR}) @@ -68,6 +67,7 @@ endif() # install library & headers. install(FILES ${CAPI_HEADERS} DESTINATION include/paddle) +install(FILES paddle_capi.map DESTINATION include/paddle) install(FILES ${CMAKE_CURRENT_BINARY_DIR}/config.h DESTINATION include/paddle) if(ANDROID) install(TARGETS paddle_capi_whole paddle_capi_engine paddle_capi_layers paddle_capi_shared diff --git a/paddle/capi/export.sym b/paddle/capi/export.sym deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/paddle/capi/export.map b/paddle/capi/paddle_capi.map similarity index 100% rename from paddle/capi/export.map rename to paddle/capi/paddle_capi.map From fdaf0772c4ac2ee2e766ddfa804cf49c65f0904d Mon Sep 17 00:00:00 2001 From: Kexin Zhao Date: Fri, 20 Oct 2017 00:28:00 -0700 Subject: [PATCH 59/76] add adagrad optimizer python implementation --- python/paddle/v2/framework/optimizer.py | 59 ++++++++++++++++++- .../v2/framework/tests/test_optimizer.py | 41 +++++++++++++ 2 files changed, 99 insertions(+), 1 deletion(-) diff --git a/python/paddle/v2/framework/optimizer.py b/python/paddle/v2/framework/optimizer.py index f992a42c40..51d435668c 100644 --- a/python/paddle/v2/framework/optimizer.py +++ b/python/paddle/v2/framework/optimizer.py @@ -1,7 +1,7 @@ import paddle.v2.framework.framework as framework from collections import defaultdict -__all__ = ['SGDOptimizer', 'MomentumOptimizer'] +__all__ = ['SGDOptimizer', 'MomentumOptimizer', 'AdagradOptimizer'] class Optimizer(object): @@ -272,3 +272,60 @@ class MomentumOptimizer(Optimizer): attrs={"mu": self._momentum}) return momentum_op + + +class AdagradOptimizer(Optimizer): + """Simple Adagrad optimizer with moment state + """ + _moment_acc_str = "moment" + + def __init__(self, learning_rate, epsilon): + assert learning_rate is not None + assert epsilon is not None + super(AdagradOptimizer, self).__init__() + self.type = "adagrad" + self._learning_rate = learning_rate + self._epsilon = epsilon + + def _initialize_tensors(self, block): + assert isinstance(block, framework.Block) + lr_shape = [1] + # create a variable for learning_rate + self._lr = block.create_var( + dtype="float32", shape=lr_shape, lod_level=0) + + # create an op to init the learning_rate + # FIXME: Fix when Initialization design has been implemented + # https://github.com/PaddlePaddle/Paddle/pull/4852 + block.append_op( + type="fill_constant", + outputs={"Out": self._lr}, + attrs={"shape": lr_shape, + "value": self._learning_rate}) + + def _create_accumulators(self, block, parameters): + assert isinstance(block, framework.Block) + + for p in parameters: + self._add_accumulator(block, self._moment_acc_str, p, 'float32') + + def _append_optimize_op(self, block, param_and_grad): + assert isinstance(block, framework.Block) + + moment_acc = self._get_accumulator(self._moment_acc_str, + param_and_grad[0]) + + # create the adagrad optimizer op + adagrad_op = block.append_op( + type=self.type, + inputs={ + "Param": param_and_grad[0], + "Grad": param_and_grad[1], + "Moment": moment_acc, + "LearningRate": self._lr + }, + outputs={"ParamOut": param_and_grad[0], + "MomentOut": moment_acc}, + attrs={"epsilon": self._epsilon}) + + return adagrad_op diff --git a/python/paddle/v2/framework/tests/test_optimizer.py b/python/paddle/v2/framework/tests/test_optimizer.py index e6a142ac36..3d1715bf62 100644 --- a/python/paddle/v2/framework/tests/test_optimizer.py +++ b/python/paddle/v2/framework/tests/test_optimizer.py @@ -69,5 +69,46 @@ class TestMomentumOptimizer(unittest.TestCase): self.assertTrue(mul_x.name in velocity_acc) +class TestAdagradOptimizer(unittest.TestCase): + class MockAdagrad(optimizer.AdagradOptimizer): + def get_accumulators(self): + return self._accumulators + + def get_moment_str(self): + return self._moment_acc_str + + def test_adagrad_optimizer(self): + program = framework.Program() + block = program.global_block() + mul_x = block.create_parameter( + dtype="float32", shape=[5, 10], lod_level=0, name="mul.x") + mul_y = block.create_var( + dtype="float32", shape=[10, 8], lod_level=0, name="mul.y") + mul_out = block.create_var( + dtype="float32", shape=[5, 8], lod_level=0, name="mul.out") + block.append_op( + type="mul", + inputs={"X": mul_x, + "Y": mul_y}, + outputs={"Out": mul_out}, + attrs={"x_num_col_dims": 1}) + adagrad_optimizer = self.MockAdagrad(learning_rate=0.01, epsilon=1.0e-6) + params_grads = adagrad_optimizer.create_backward_pass(mul_out) + self.assertEqual(len(params_grads), 1) + self.assertEqual(len(adagrad_optimizer.get_accumulators()), 0) + opts = adagrad_optimizer.create_optimization_pass(params_grads, mul_out) + self.assertEqual(len(opts), 1) + adagrad_op = opts[0] + self.assertEqual(adagrad_op.type, "adagrad") + + # check accumulators + accumulators = adagrad_optimizer.get_accumulators() + self.assertEqual(len(accumulators), 1) + self.assertTrue(adagrad_optimizer.get_moment_str() in accumulators) + moment_acc = accumulators[adagrad_optimizer.get_moment_str()] + self.assertEqual(len(moment_acc), 1) + self.assertTrue(mul_x.name in moment_acc) + + if __name__ == '__main__': unittest.main() From 5b5cb0781aeec6967da205040395a17d5bec2380 Mon Sep 17 00:00:00 2001 From: tensor-tang Date: Fri, 20 Oct 2017 16:41:32 +0800 Subject: [PATCH 60/76] add branch tests for pool and fc --- paddle/gserver/tests/mkldnn_branches_fc.conf | 58 ++++++++++++++++++ .../gserver/tests/mkldnn_branches_pool.conf | 60 +++++++++++++++++++ paddle/gserver/tests/test_MKLDNN.cpp | 2 +- 3 files changed, 119 insertions(+), 1 deletion(-) create mode 100644 paddle/gserver/tests/mkldnn_branches_fc.conf create mode 100644 paddle/gserver/tests/mkldnn_branches_pool.conf diff --git a/paddle/gserver/tests/mkldnn_branches_fc.conf b/paddle/gserver/tests/mkldnn_branches_fc.conf new file mode 100644 index 0000000000..fb85425c2b --- /dev/null +++ b/paddle/gserver/tests/mkldnn_branches_fc.conf @@ -0,0 +1,58 @@ +# Copyright (c) 2017 PaddlePaddle Authors. All Rights Reserved +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from paddle.trainer_config_helpers import * + +settings(batch_size=16) +channels = get_config_arg("channels", int, 2) + +def two_fc(input, group_name): + out1 = fc_layer(input=input, + name=group_name+'_fc1', + size=channels, + bias_attr=False, + act=LinearActivation()) + + out2 = fc_layer(input=input, + name=group_name+'_fc2', + size=channels, + bias_attr=False, + act=LinearActivation()) + return out1, out2 + +data = data_layer(name ="input", size=channels*16*16) + +conv = img_conv_layer(input=data, + num_channels=channels, + filter_size=3, + num_filters=channels, + padding=1, + shared_biases=True, + act=LinearActivation()) + +pool = img_pool_layer(input=conv, + pool_size=3, + stride=2, + padding=1, + pool_type=AvgPooling()) + +a1, a2 = two_fc(input=pool, group_name='a') + +concat = concat_layer(input=[a1, a2]) + +b1, b2 = two_fc(input=pool, group_name='b') + +addto = addto_layer(input=[b1, b2]) + +outputs([concat, addto]) diff --git a/paddle/gserver/tests/mkldnn_branches_pool.conf b/paddle/gserver/tests/mkldnn_branches_pool.conf new file mode 100644 index 0000000000..ca17c74752 --- /dev/null +++ b/paddle/gserver/tests/mkldnn_branches_pool.conf @@ -0,0 +1,60 @@ +# Copyright (c) 2017 PaddlePaddle Authors. All Rights Reserved +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from paddle.trainer_config_helpers import * + +settings(batch_size=16) +channels = get_config_arg("channels", int, 2) + +def two_pool(input, group_name): + out1 = img_pool_layer(input=input, + name=group_name+'_pool1', + pool_size=3, + stride=2, + padding=0, + pool_type=MaxPooling()) + + out2 = img_pool_layer(input=input, + name=group_name+'_pool2', + pool_size=5, + stride=2, + padding=1, + pool_type=MaxPooling()) + return out1, out2 + +data = data_layer(name ="input", size=channels*16*16) + +conv = img_conv_layer(input=data, + num_channels=channels, + filter_size=3, + num_filters=channels, + padding=1, + shared_biases=True, + act=LinearActivation()) + +pool = img_pool_layer(input=conv, + pool_size=3, + stride=1, + padding=1, + pool_type=AvgPooling()) + +a1, a2 = two_pool(input=pool, group_name='a') + +concat = concat_layer(input=[a1, a2]) + +b1, b2 = two_pool(input=pool, group_name='b') + +addto = addto_layer(input=[b1, b2]) + +outputs([concat, addto]) diff --git a/paddle/gserver/tests/test_MKLDNN.cpp b/paddle/gserver/tests/test_MKLDNN.cpp index 3571fbb9e3..6cb4ca5e08 100644 --- a/paddle/gserver/tests/test_MKLDNN.cpp +++ b/paddle/gserver/tests/test_MKLDNN.cpp @@ -250,7 +250,7 @@ TEST(MKLDNNActivation, Activations) { DECLARE_string(config_args); TEST(MKLDNNLayer, branches) { - std::vector cases = {"conv"}; + std::vector cases = {"conv", "pool", "fc"}; for (auto name : cases) { std::string config = "./gserver/tests/mkldnn_branches_" + name + ".conf"; for (auto channels : {2, 32}) { From 36ebf00f98e3a75d5337ce1f308f61cde0f8fb55 Mon Sep 17 00:00:00 2001 From: Luo Tao Date: Fri, 20 Oct 2017 17:40:55 +0800 Subject: [PATCH 61/76] rename sparse_vector to sparse_float_vector, and fix typo --- doc/faq/local/index_cn.rst | 2 +- python/paddle/trainer/PyDataProvider2.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/faq/local/index_cn.rst b/doc/faq/local/index_cn.rst index 75c4ba028e..0e939a2671 100644 --- a/doc/faq/local/index_cn.rst +++ b/doc/faq/local/index_cn.rst @@ -174,7 +174,7 @@ decoder_inputs = paddle.layer.fc( 1. 两者都是对梯度的截断，但截断时机不同，前者在 :code:`optimzier` 更新网络参数时应用；后者在激活函数反向计算时被调用； 2. 截断对象不同：前者截断可学习参数的梯度，后者截断回传给前层的梯度; -除此之外，还可以通过减小学习律或者对数据进行归一化处理来解决这类问题。 +除此之外，还可以通过减小学习率或者对数据进行归一化处理来解决这类问题。 5. 如何调用 infer 接口输出多个layer的预测结果 ----------------------------------------------- diff --git a/python/paddle/trainer/PyDataProvider2.py b/python/paddle/trainer/PyDataProvider2.py index 248da4ae8d..045e3c0279 100644 --- a/python/paddle/trainer/PyDataProvider2.py +++ b/python/paddle/trainer/PyDataProvider2.py @@ -175,7 +175,7 @@ def index_slot(value_range, seq_type=SequenceType.NO_SEQUENCE): dense_vector = dense_slot sparse_binary_vector = sparse_non_value_slot -sparse_vector = sparse_value_slot +sparse_float_vector = sparse_value_slot integer_value = index_slot # dense_array can be used for variable-length input feature. From 5c892db64ca22200e9d245da3ff72d1dfca3738d Mon Sep 17 00:00:00 2001 From: tensor-tang Date: Fri, 20 Oct 2017 21:43:54 +0800 Subject: [PATCH 62/76] remove unused code refine comments and bias fix typo and todo --- paddle/gserver/layers/MKLDNNConvLayer.cpp | 8 ++--- paddle/gserver/layers/MKLDNNFcLayer.cpp | 21 ++++++------ paddle/gserver/layers/MKLDNNLayer.cpp | 7 ++-- paddle/gserver/layers/MKLDNNLayer.h | 39 ++++++++++++----------- 4 files changed, 37 insertions(+), 38 deletions(-) diff --git a/paddle/gserver/layers/MKLDNNConvLayer.cpp b/paddle/gserver/layers/MKLDNNConvLayer.cpp index 3fbfb1ab1f..83f4e4e615 100644 --- a/paddle/gserver/layers/MKLDNNConvLayer.cpp +++ b/paddle/gserver/layers/MKLDNNConvLayer.cpp @@ -210,11 +210,11 @@ void MKLDNNConvLayer::resetFwdBuffers( resetWithMatrix(wgt, weight_->getW(), pd->weights_primitive_desc()); - bias = nullptr; - if (biases_ == nullptr || biases_->getW() == nullptr) { - return; + if (biases_ && biases_->getW()) { + resetWithMatrix(bias, biases_->getW(), pd->bias_primitive_desc()); + } else { + bias = nullptr; } - resetWithMatrix(bias, biases_->getW(), pd->bias_primitive_desc()); } void MKLDNNConvLayer::resetFwdPipeline( diff --git a/paddle/gserver/layers/MKLDNNFcLayer.cpp b/paddle/gserver/layers/MKLDNNFcLayer.cpp index 9f82a3b747..d82063a713 100644 --- a/paddle/gserver/layers/MKLDNNFcLayer.cpp +++ b/paddle/gserver/layers/MKLDNNFcLayer.cpp @@ -134,10 +134,6 @@ void MKLDNNFcLayer::resetFwdBuffers(MKLDNNMatrixPtr& in, CHECK(in); in->downSpatial(); - // if (extInVal_) { - // extInVal_->downSpatial(); - // } - auto outPD = MKLDNNMatrix::createPrimitiveDesc({bs_, oc_}, format::nc, engine_); resetOutValue(out, outPD); @@ -153,11 +149,12 @@ void MKLDNNFcLayer::resetFwdBuffers(MKLDNNMatrixPtr& in, resetWithMatrix(wgt, weight_->getW(), wgtPD); wgt->downSpatial(); - if (biases_ == nullptr || biases_->getW() == nullptr) { - return; + if (biases_ && biases_->getW()) { + auto biasPD = MKLDNNMatrix::createPrimitiveDesc({oc_}, format::x, engine_); + resetWithMatrix(bias, biases_->getW(), biasPD); + } else { + bias = nullptr; } - auto biasPD = MKLDNNMatrix::createPrimitiveDesc({oc_}, format::x, engine_); - resetWithMatrix(bias, biases_->getW(), biasPD); } void MKLDNNFcLayer::resetFwdPD(std::shared_ptr& pd, @@ -207,11 +204,11 @@ void MKLDNNFcLayer::resetBwdBuffers(MKLDNNMatrixPtr& in, CHECK(wgtVal_); resetWithMatrix(wgt, weight_->getWGrad(), wgtVal_->getPrimitiveDesc()); - bias = nullptr; - if (biasVal_ == nullptr) { - return; + if (biasVal_) { + resetWithMatrix(bias, biases_->getWGrad(), biasVal_->getPrimitiveDesc()); + } else { + bias = nullptr; } - resetWithMatrix(bias, biases_->getWGrad(), biasVal_->getPrimitiveDesc()); } void MKLDNNFcLayer::resetBwdWgtPD( diff --git a/paddle/gserver/layers/MKLDNNLayer.cpp b/paddle/gserver/layers/MKLDNNLayer.cpp index f4968c4af3..6bb19976b5 100644 --- a/paddle/gserver/layers/MKLDNNLayer.cpp +++ b/paddle/gserver/layers/MKLDNNLayer.cpp @@ -60,7 +60,7 @@ void MKLDNNLayer::forward(PassType passType) { resetFwd(pipelineFwd_, inVal_, wgtVal_, biasVal_, outVal_); // MKLDNNLayer output value should be MKLDNNMatrix // so external output value is necessary. - // then external input value is not necessary, + // Then external input value is not necessary, // since input may be mkldnn internal buffer. CHECK(extOutVal_) << "external output value is necessary"; output_.value = std::dynamic_pointer_cast(extOutVal_); @@ -235,8 +235,8 @@ void MKLDNNLayer::resetInGrad(MKLDNNMatrixPtr& in, in = MKLDNNMatrix::create(intPD, inMat); Argument& arg = input->getOutput(this->getName()); arg.grad = std::dynamic_pointer_cast(in); - CHECK(inVal_ != nullptr && inVal_->getPrimitiveDesc() == intPD) - << "should have internal input value and primitive desc must equal"; + CHECK(inVal_); + CHECK(inVal_->getPrimitiveDesc() == intPD) << "the primitive desc must equal"; if (inputIsOnlyMKLDNN()) { return; } @@ -246,6 +246,7 @@ void MKLDNNLayer::resetInGrad(MKLDNNMatrixPtr& in, return; } // need create reorder + // TODO(TJ): add macro definition to simplify it CHECK(extInVal_ != nullptr && isPaddleFormat(extInVal_->getFormat())) << "should have external input value and the format must be nchw(nc)"; extInGrad_ = MKLDNNMatrix::create(extInVal_->getPrimitiveDesc(), inMat); diff --git a/paddle/gserver/layers/MKLDNNLayer.h b/paddle/gserver/layers/MKLDNNLayer.h index 656b5ee2d7..9b54c95b55 100644 --- a/paddle/gserver/layers/MKLDNNLayer.h +++ b/paddle/gserver/layers/MKLDNNLayer.h @@ -58,14 +58,15 @@ protected: std::vector pipelineFwd_; std::vector pipelineBwd_; - /// value and grad are seperated as internal and external buffers. - /// each MKLDNNLayer must init or reset internal buffer at least, - /// and the external buffer format is always nchw of nc(when h==w==1), - /// which is the same format as paddle. - /// The output_.value and output_.grad always save the external data, - /// when mixed with cpu device. - /// When all layers are mkldnn layers, they could save internal data. - /// below MKLDNNMatrix buffers are all internal buffers + /* Value and grad are seperated as internal and external buffers. + * Each MKLDNNLayer must init or reset internal buffer at least, + * and the external buffer format is always nchw of nc(when h==w==1), + * which is the same format as paddle. + * The output_.value and output_.grad always save the external data, + * when mixed with cpu device. + * When all layers are mkldnn layers, they could save internal data. + */ + // below MKLDNNMatrix buffers are all internal buffers MKLDNNMatrixPtr inVal_; MKLDNNMatrixPtr inGrad_; MKLDNNMatrixPtr outVal_; @@ -120,8 +121,8 @@ public: ~MKLDNNLayer() {} virtual bool init(const LayerMap& layerMap, const ParameterMap& parameterMap); - void forward(PassType passType) override; - void backward(const UpdateCallback& callback) override; + virtual void forward(PassType passType); + virtual void backward(const UpdateCallback& callback); /** * reshape the input image sizes @@ -217,7 +218,7 @@ protected: * reset output grad from internal primitive desc. * merge grad if necessary. * reset both internal and external buffer and create reorder if necessary. - * note: about merge grad, when this layer has serval outputs, + * note: about merge grad, when this layer has several outputs, * it could not be mixed with cpu device, * since it can not get memory desc from cpu device. */ @@ -225,7 +226,7 @@ protected: /** * reset the merge grad primitive if necessary. - * note: do not support the grads are mixed with cpu device, + * note: do not support the grads mixed with cpu device, * since it can not get memory desc from cpu device. */ void resetMergeGrad(MKLDNNMatrixPtr& out); @@ -313,17 +314,17 @@ protected: * print the mkldnn memory format of grad */ virtual void printGradFormat() { - if (extInGrad_) { - VLOG(MKLDNN_FMTS) << extInGrad_->getFormat() << " <<< "; - } - if (inGrad_) { - VLOG(MKLDNN_FMTS) << inGrad_->getFormat() << " <<<"; + if (extOutGrad_) { + VLOG(MKLDNN_FMTS) << extOutGrad_->getFormat(); } if (outGrad_) { VLOG(MKLDNN_FMTS) << outGrad_->getFormat() << " <<< "; } - if (extOutGrad_) { - VLOG(MKLDNN_FMTS) << extOutGrad_->getFormat(); + if (inGrad_) { + VLOG(MKLDNN_FMTS) << inGrad_->getFormat() << " <<<"; + } + if (extInGrad_) { + VLOG(MKLDNN_FMTS) << extInGrad_->getFormat() << " <<< "; } if (wgtGrad_) { VLOG(MKLDNN_FMTS) << "Weight grad format: " << wgtGrad_->getFormat(); From 5380a5471be25668d1137b9aec6439c9fbe28460 Mon Sep 17 00:00:00 2001 From: kavyasrinet Date: Fri, 20 Oct 2017 08:24:52 -0700 Subject: [PATCH 63/76] Adding Nesterov Momentum (#4948) --- paddle/operators/momentum_op.cc | 9 +++- paddle/operators/momentum_op.h | 9 +++- .../v2/framework/tests/test_momentum_op.py | 45 ++++++++++++++++++- .../v2/framework/tests/test_rmsprop_op.py | 2 +- 4 files changed, 59 insertions(+), 6 deletions(-) diff --git a/paddle/operators/momentum_op.cc b/paddle/operators/momentum_op.cc index 9be4d15a43..2d4d6f1372 100644 --- a/paddle/operators/momentum_op.cc +++ b/paddle/operators/momentum_op.cc @@ -75,12 +75,17 @@ class MomentumOpMaker : public framework::OpProtoAndCheckerMaker { AddOutput("VelocityOut", "(Tensor) Output updated velocity"); AddAttr("mu", "(float) Momentum coefficient"); + AddAttr("useNesterov", "(bool) Use Nesterov Momentum") + .SetDefault(false); AddComment(R"DOC( -Momentum Algorithm (momentum). +Momentum Algorithm with a flag for Nestrov Moemntum (momentum). velocity = mu * velocity + gradient -param = param - learning_rate * velocity +if (use_nesterov): + param = param - gradient * learning_rate + mu * velocity * learning_rate +else: + param = param - learning_rate * velocity )DOC"); } diff --git a/paddle/operators/momentum_op.h b/paddle/operators/momentum_op.h index f7a724f048..e6d6d1da3d 100644 --- a/paddle/operators/momentum_op.h +++ b/paddle/operators/momentum_op.h @@ -34,6 +34,7 @@ class MomentumOpKernel : public framework::OpKernel { velocity_out->mutable_data(ctx.GetPlace()); float mu = ctx.Attr("mu"); + bool use_nesterov = ctx.Attr("useNesterov"); auto p_out = framework::EigenVector::Flatten(*param_out); auto v_out = framework::EigenVector::Flatten(*velocity_out); @@ -46,8 +47,14 @@ class MomentumOpKernel : public framework::OpKernel { auto place = ctx.GetEigenDevice(); Eigen::DSizes grad_dsize(grad->numel()); + v_out.device(place) = v * mu + g; - p_out.device(place) = p - lr.broadcast(grad_dsize) * v_out; + if (use_nesterov) { + p_out.device(place) = p - g * lr.broadcast(grad_dsize) + + v_out * mu * lr.broadcast(grad_dsize); + } else { + p_out.device(place) = p - lr.broadcast(grad_dsize) * v_out; + } } }; diff --git a/python/paddle/v2/framework/tests/test_momentum_op.py b/python/paddle/v2/framework/tests/test_momentum_op.py index d3353ff6e4..654d31975a 100644 --- a/python/paddle/v2/framework/tests/test_momentum_op.py +++ b/python/paddle/v2/framework/tests/test_momentum_op.py @@ -3,7 +3,7 @@ import numpy as np from op_test import OpTest -class TestMomentumOp(OpTest): +class TestMomentumOp1(OpTest): def setUp(self): self.op_type = "momentum" @@ -12,6 +12,7 @@ class TestMomentumOp(OpTest): velocity = np.zeros((123, 321)).astype("float32") learning_rate = np.array([0.001]).astype("float32") mu = 0.0001 + use_nesterov = False self.inputs = { 'Param': param, @@ -23,7 +24,47 @@ class TestMomentumOp(OpTest): self.attrs = {'mu': mu} velocity_out = mu * velocity + grad - param_out = param - learning_rate * velocity_out + if use_nesterov: + param_out = param - grad * learning_rate + \ + velocity_out * mu * learning_rate + else: + param_out = param - learning_rate * velocity_out + + self.outputs = {'ParamOut': param_out, 'VelocityOut': velocity_out} + + def test_check_output(self): + self.check_output() + + +class TestMomentumOp2(OpTest): + '''Test Momentum with defaukt values for attributes + ''' + + def setUp(self): + self.op_type = "momentum" + + param = np.random.random((123, 321)).astype("float32") + grad = np.random.random((123, 321)).astype("float32") + velocity = np.zeros((123, 321)).astype("float32") + learning_rate = np.array([0.001]).astype("float32") + mu = 0.0001 + use_nesterov = True + + self.inputs = { + 'Param': param, + 'Grad': grad, + 'Velocity': velocity, + 'LearningRate': learning_rate + } + + self.attrs = {'mu': mu, 'useNesterov': use_nesterov} + + velocity_out = mu * velocity + grad + if use_nesterov: + param_out = param - grad * learning_rate + \ + velocity_out * mu * learning_rate + else: + param_out = param - learning_rate * velocity_out self.outputs = {'ParamOut': param_out, 'VelocityOut': velocity_out} diff --git a/python/paddle/v2/framework/tests/test_rmsprop_op.py b/python/paddle/v2/framework/tests/test_rmsprop_op.py index 3e5ff733e9..237bcfccce 100644 --- a/python/paddle/v2/framework/tests/test_rmsprop_op.py +++ b/python/paddle/v2/framework/tests/test_rmsprop_op.py @@ -46,7 +46,7 @@ class TestRmspropOp1(OpTest): class TestRmspropOp2(OpTest): - '''Test RMSProp with defaukt values for attributes + '''Test RMSProp with default values for attributes ''' def setUp(self): From 07ea9adec0531402bc31906d6ae85edaf96f413b Mon Sep 17 00:00:00 2001 From: Yan Chunwei Date: Fri, 20 Oct 2017 13:09:58 -0400 Subject: [PATCH 64/76] feature/dynamic recurrent op forward and backward (#4799) --- doc/design/block.md | 2 +- paddle/framework/backward.cc | 16 +- paddle/operators/dynamic_recurrent_op.cc | 309 +++++++++++------- paddle/operators/dynamic_recurrent_op.h | 165 ++++++---- paddle/operators/dynamic_recurrent_op_test.cc | 48 ++- paddle/operators/recurrent_op.cc | 26 +- paddle/operators/rnn/recurrent_op_utils.cc | 22 +- paddle/operators/rnn/recurrent_op_utils.h | 12 +- paddle/pybind/pybind.cc | 10 +- .../tests/test_dynamic_recurrent_op.py | 131 +++++--- .../v2/framework/tests/test_recurrent_op.py | 20 +- 11 files changed, 478 insertions(+), 283 deletions(-) diff --git a/doc/design/block.md b/doc/design/block.md index 7cbf0d55b1..4066122c0e 100644 --- a/doc/design/block.md +++ b/doc/design/block.md @@ -189,7 +189,7 @@ OpDesc { inputs = {0} // the index of x in vars of BlockDesc above outputs = {5, 3} // indices of act and hidden_out in vars of BlockDesc above attrs { - "memories" : {1} // the index of h + "states" : {1} // the index of h "step_net" : } }; diff --git a/paddle/framework/backward.cc b/paddle/framework/backward.cc index fb552fe344..1ae7fb60f0 100644 --- a/paddle/framework/backward.cc +++ b/paddle/framework/backward.cc @@ -21,6 +21,7 @@ #include "paddle/framework/block_desc.h" #include "paddle/framework/op_registry.h" +#include "paddle/operators/dynamic_recurrent_op.h" #include "paddle/operators/net_op.h" #include "paddle/operators/recurrent_op.h" @@ -220,8 +221,7 @@ static std::unique_ptr BackwardRecursive( // process recurrent gradient op as a special operator. if (forwardOp.Type() == "recurrent") { // NOTE clean up cycle call somewhere (RNN's stepnet constains itself), - // or - // this will result in infinite loop. + // or this will result in infinite loop. const auto& rnnop = *static_cast(&forwardOp); auto rnn_grad_op = @@ -231,6 +231,18 @@ static std::unique_ptr BackwardRecursive( // create stepnet's gradient op rnn_grad_op->set_stepnet( BackwardRecursive(stepnet_op, no_grad_names, grad_to_var, uniq_id)); + } else if (forwardOp.Type() == "dynamic_recurrent") { + // NOTE clean up cycle call somewhere (RNN's stepnet constains itself), + // or this will result in infinite loop. + const auto& rnnop = + *static_cast(&forwardOp); + auto rnn_grad_op = + static_cast(grad_op.get()); + const auto& stepnet_op = + *static_cast(&rnnop.rnn.GetStepUnit()); + // create stepnet's gradient op + rnn_grad_op->rnn.SetStepUnit( + BackwardRecursive(stepnet_op, no_grad_names, grad_to_var, uniq_id)); } if (net->ops_.empty()) { // Current no aux op is added to network diff --git a/paddle/operators/dynamic_recurrent_op.cc b/paddle/operators/dynamic_recurrent_op.cc index 62962be205..dce8c8d835 100644 --- a/paddle/operators/dynamic_recurrent_op.cc +++ b/paddle/operators/dynamic_recurrent_op.cc @@ -23,6 +23,7 @@ using framework::Scope; using framework::TensorArray; using framework::LoDTensor; using framework::Variable; +using framework::OperatorBase; using framework::DySeqMetaBatch; namespace detail { @@ -43,10 +44,9 @@ inline void CreateVariables(Scope& scope, * be reordered, but the RNN op should not change the `boot_state` as an input * variable's content. */ -template -inline void ReorderBootState(const DySeqMetaBatch& metas, - const LoDTensor& boot_state, LoDTensor* tensor, - const platform::Place& dst_place) { +inline void ReorderInitialState(const DySeqMetaBatch& metas, + const LoDTensor& boot_state, LoDTensor* tensor, + const platform::Place& dst_place) { for (size_t seq_id = 0; seq_id < metas.size(); seq_id++) { auto slice = tensor->Slice(seq_id, seq_id + 1); auto boot_slice = @@ -56,58 +56,60 @@ inline void ReorderBootState(const DySeqMetaBatch& metas, } } -} // namespace detail - -class DynamicRecurrentOpProtoAndCheckerMaker - : public framework::OpProtoAndCheckerMaker { - public: - DynamicRecurrentOpProtoAndCheckerMaker(framework::OpProto* proto, - framework::OpAttrChecker* op_checker) - : OpProtoAndCheckerMaker(proto, op_checker) { - const auto& name = DynamicRecurrentOp::kArgName; - // inputs and outputs stored in proto - AddInput(name.inlinks, - "the inputs that need to be segmented for each step.") - .AsDuplicable(); - AddInput(name.boot_memories, "variables to initialize memories.") - .AsDuplicable(); - - AddOutput(name.outlinks, "the outputs that need to concated for all steps.") - .AsDuplicable(); - AddOutput(name.step_scopes, "step scopes"); - - // Attributes stored in AttributeMap - AddAttr>(name.pre_memories, - "names of pre-memories"); - AddAttr>(name.memories, "names of memories"); - - AddComment("This is a RNN operator for varience-length sequences."); +inline void RestoreInitialState(const DySeqMetaBatch& metas, + const LoDTensor& tensor, LoDTensor* boot_state, + const platform::Place& dst_place) { + for (size_t seq_id = 0; seq_id < metas.size(); seq_id++) { + auto slice = tensor.Slice(seq_id, seq_id + 1); + auto boot_slice = + boot_state->Slice(metas[seq_id].ori_idx, metas[seq_id].ori_idx + 1); + boot_slice.CopyFrom(slice, dst_place, platform::CPUDeviceContext()); } -}; +} -void DynamicRecurrentOp::Run(const Scope& scope, - const platform::DeviceContext& dev_ctx) const { - cache_.Init(kArgName, *this, scope, &arg_); +} // namespace detail + +// Implementation for forward propagation. +template <> +void RNNAlgorithm::Run( + const framework::Scope& scope, const framework::OperatorBase& op, + const platform::DeviceContext& dev_ctx) { + SetComputeMode(ComputeMode::kForward); + cache_.Init(kArgNames[mode_], op, scope, &dev_ctx, &arg_); SplitInputs(); CreateScopes(); WriteStepInputs(); InitStates(); WriteStepOutputs(); + RunSteps(); + ConcatOutputs(); +} - // call stepnet in all the time steps - for (size_t step = 0; step < cache_.num_steps; step++) { - auto& step_scope = cache_.GetScope(step); - stepnet_->Run(step_scope, dev_ctx); +// Implementation for backward propagation. +template <> +void RNNAlgorithm::Run( + const framework::Scope& scope, const framework::OperatorBase& op, + const platform::DeviceContext& dev_ctx) { + SetComputeMode(ComputeMode::kBackward); + cache_.Init(kArgNames[mode_], op, scope, &dev_ctx, &arg_); + SplitInputs(); + WriteStepInputs(); + InitStates(); + WriteStepOutputs(); + RunSteps(); + // copy boot-states' gradients back. + for (const auto& state : arg_.states) { + ExportInitialStateGradient(state); } ConcatOutputs(); } -void DynamicRecurrentOp::SplitInputs() const { +void RNNAlgorithm::SplitInputs() { // TODO(superjom) make level a config // TODO(superjom) check all the inputs has the same LoD int level = 0; - for (const auto& item : cache_.inlinks) { + for (const auto& item : cache_.inputs) { const auto& var = item.second; const auto& tensor = var->Get(); TensorArray& ta = step_inputs_[item.first]; @@ -124,8 +126,8 @@ void DynamicRecurrentOp::SplitInputs() const { } } -void DynamicRecurrentOp::WriteStepInputs() const { - for (const auto& item : cache_.inlinks) { +void RNNAlgorithm::WriteStepInputs() { + for (const auto& item : cache_.inputs) { auto ta_it = step_inputs_.find(item.first); PADDLE_ENFORCE(ta_it != step_inputs_.end(), "step_inputs_ not compatible with memory set"); @@ -142,15 +144,15 @@ void DynamicRecurrentOp::WriteStepInputs() const { } } -void DynamicRecurrentOp::WriteStepOutputs() const { +void RNNAlgorithm::WriteStepOutputs() { // initialize step outputs - for (const auto& item : cache_.outlinks) { + for (const auto& item : cache_.outputs) { step_outputs_.emplace(item.first, TensorArray()); } PADDLE_ENFORCE_GT(step_outputs_.size(), 0UL); } -void DynamicRecurrentOp::CreateScopes() const { +void RNNAlgorithm::CreateScopes() { PADDLE_ENFORCE_GT(cache_.num_steps, 0); // resize scopes size_t num_scopes_need_create = cache_.num_steps - cache_.scopes->size(); @@ -159,19 +161,19 @@ void DynamicRecurrentOp::CreateScopes() const { } // init temporary inputs - PADDLE_ENFORCE_NOT_NULL(stepnet_, "stepnet should be set first"); - std::vector memories; - std::vector pre_memories; - std::vector stepnet_outputs; - std::transform(arg_.memories.begin(), arg_.memories.end(), - std::back_inserter(memories), - [](const rnn::MemoryAttr& m) { return m.var; }); - std::transform(arg_.memories.begin(), arg_.memories.end(), - std::back_inserter(pre_memories), - [](const rnn::MemoryAttr& m) { return m.pre_var; }); - for (const auto& item : stepnet_->Outputs()) { + PADDLE_ENFORCE_NOT_NULL(step_unit_, "stepnet should be set first"); + std::vector states; + std::vector ex_states; + std::vector step_unit_outputs; + std::transform(arg_.states.begin(), arg_.states.end(), + std::back_inserter(states), + [](const rnn::StateAttr& m) { return m.var; }); + std::transform(arg_.states.begin(), arg_.states.end(), + std::back_inserter(ex_states), + [](const rnn::StateAttr& m) { return m.pre_var; }); + for (const auto& item : step_unit_->Outputs()) { for (const auto& var : item.second) { - stepnet_outputs.push_back(var); + step_unit_outputs.push_back(var); } } @@ -179,13 +181,13 @@ void DynamicRecurrentOp::CreateScopes() const { auto& scope = cache_.GetScope(step); detail::CreateVariables(scope, arg_.inlinks); detail::CreateVariables(scope, arg_.outlinks); - detail::CreateVariables(scope, memories); - detail::CreateVariables(scope, pre_memories); - detail::CreateVariables(scope, stepnet_outputs); + detail::CreateVariables(scope, states); + detail::CreateVariables(scope, ex_states); + detail::CreateVariables(scope, step_unit_outputs); } } -void DynamicRecurrentOp::ConcatOutputs() const { +void RNNAlgorithm::ConcatOutputs() { // TODO(superjom) transform this to a config int level = 0; for (size_t step = 0; step < cache_.num_steps; step++) { @@ -198,31 +200,45 @@ void DynamicRecurrentOp::ConcatOutputs() const { item.second.WriteShared(step, *tensor); } } - // the inlinks' lods should be the same, so randomly get one lod. + // the inputs' lods should be the same, so randomly get one lod. const auto& some_lod = cache_.scope->FindVar(arg_.inlinks.front())->Get().lod(); const auto& some_meta = dy_seq_metas_[arg_.inlinks.front()]; for (auto& item : step_outputs_) { auto tensor = item.second.Pack(level, some_meta, some_lod); - auto* output = cache_.outlinks[item.first]->GetMutable(); + auto* output = cache_.outputs[item.first]->GetMutable(); const_cast(output)->ShareDataWith(tensor); } } -void DynamicRecurrentOp::InitStates() const { +void RNNAlgorithm::RunSteps() { + if (IsBackward()) { + // call stepnet in all the time steps reversely + for (int step = cache_.num_steps - 1; step >= 0; step--) { + auto& step_scope = cache_.GetScope(step); + step_unit_->Run(step_scope, *cache_.dev_ctx); + } + } else { + for (size_t step = 0; step < cache_.num_steps; step++) { + auto& step_scope = cache_.GetScope(step); + step_unit_->Run(step_scope, *cache_.dev_ctx); + } + } +} + +void RNNAlgorithm::InitStates() { for (size_t step = 0; step < cache_.num_steps; step++) { - for (const auto& memory : arg_.memories) { - CreateState(memory, step); - LinkState(memory, step); + for (const auto& state : arg_.states) { + CreateState(state, step); + LinkState(state, step); } } } -void DynamicRecurrentOp::CreateState(const rnn::MemoryAttr& memory, - size_t step) const { +void RNNAlgorithm::CreateState(const rnn::StateAttr& state_attr, size_t step) { auto& scope = cache_.GetScope(step); - auto& state = *cache_.GetTensor(scope, memory.var); - auto& boot_state = *cache_.GetTensor(*cache_.scope, memory.boot_var); + auto& state = *cache_.GetTensor(scope, state_attr.var); + auto& boot_state = *cache_.GetTensor(*cache_.scope, state_attr.boot_var); size_t num_instances = step_inputs_[arg_.inlinks.front()].Read(step).dims()[0]; @@ -231,56 +247,79 @@ void DynamicRecurrentOp::CreateState(const rnn::MemoryAttr& memory, state.Resize(dims); state.mutable_data(platform::CPUPlace()); - states_[memory.var].WriteShared(step, state); + states_[state_attr.var].WriteShared(step, state); } -void DynamicRecurrentOp::LinkState(const rnn::MemoryAttr& memory, - size_t step) const { +void RNNAlgorithm::LinkState(const rnn::StateAttr& state, size_t step) { auto& scope = cache_.GetScope(step); - auto& state_pre = *cache_.GetTensor(scope, memory.pre_var); + auto& state_pre = *cache_.GetTensor(scope, state.pre_var); + + // process the first state's boot-state(the 0-step in forward mode or the + // last step in backward mode) + // Only forward mode need to link the boot-state to the `pre-state` in first + // time step. In backward mode, need to copy the gradient of `pre-state` in + // first time step to the gradient of `boot-state`. + if (step == 0 && IsForward()) { + LinkInitialState(state); + } else { + size_t num_instances = + step_inputs_[arg_.inlinks.front()].Read(step).dims()[0]; + auto* pre_state = cache_.GetTensor(cache_.GetScope(step - 1), state.var); + // shink and share from previous state + auto shrinked_pre_state = pre_state->Slice(0, num_instances); + state_pre.ShareDataWith(shrinked_pre_state); + } +} +void RNNAlgorithm::LinkInitialState(const rnn::StateAttr& state) { // all the step_inputs' metas should be the same, just randomly select one // and get the dyseq meta. const auto& some_meta = dy_seq_metas_[arg_.inlinks.front()]; - size_t num_instances = - step_inputs_[arg_.inlinks.front()].Read(step).dims()[0]; + auto& scope = cache_.GetScope(0); + auto& state_pre = *cache_.GetTensor(scope, state.pre_var); + auto* pre_state = cache_.GetTensor(*cache_.scope, state.boot_var); + pre_state->mutable_data(platform::CPUPlace()); + // allocate state + state_pre.Resize(pre_state->dims()); + state_pre.mutable_data(platform::CPUPlace()); + detail::ReorderInitialState(some_meta, *pre_state, &state_pre, + pre_state->place()); +} - LoDTensor* pre_state{nullptr}; - if (step == 0) { - pre_state = cache_.GetTensor(*cache_.scope, memory.boot_var); - pre_state->mutable_data(platform::CPUPlace()); - // allocate memory - state_pre.Resize(pre_state->dims()); - state_pre.mutable_data(platform::CPUPlace()); - detail::ReorderBootState(some_meta, *pre_state, &state_pre, - pre_state->place()); - } else { - pre_state = cache_.GetTensor(cache_.GetScope(step - 1), memory.var); - } +void RNNAlgorithm::ExportInitialStateGradient(const rnn::StateAttr& state) { + // all the step_inputs' metas should be the same, just randomly select one + // and get the dyseq meta. + const auto& some_meta = dy_seq_metas_[arg_.inlinks.front()]; + auto& scope = cache_.GetScope(0); - // shink and share from previous state - auto shrinked_pre_state = pre_state->Slice(0, num_instances); - state_pre.ShareDataWith(shrinked_pre_state); + auto& state_pre = *cache_.GetTensor(scope, state.pre_var); + auto& pre_state = *cache_.GetTensor(*cache_.scope, state.boot_var); + pre_state.Resize(state_pre.dims()); + detail::RestoreInitialState(some_meta, state_pre, &pre_state, + pre_state.place()); } -void DynamicRecurrentOp::ArgCache::Init( - const rnn::ArgumentName& name, const paddle::framework::OperatorBase& op, - const paddle::framework::Scope& scope, rnn::Argument* arg) { +void RNNAlgorithm::ArgCache::Init(const rnn::ArgumentName& name, + const paddle::framework::OperatorBase& op, + const paddle::framework::Scope& scope, + platform::DeviceContext const* dev_ctx, + rnn::Argument* arg) { this->scope = &scope; InitArgument(name, op, arg); CacheScopes(scope, *arg); CacheInlinks(scope, arg->inlinks); CacheOutlinks(scope, arg->outlinks); + this->dev_ctx = dev_ctx; } -void DynamicRecurrentOp::ArgCache::InitArgument(const rnn::ArgumentName& name, - const OperatorBase& op, - rnn::Argument* arg) { +void RNNAlgorithm::ArgCache::InitArgument(const rnn::ArgumentName& name, + const OperatorBase& op, + rnn::Argument* arg) { rnn::InitArgument(name, arg, op, false /*is_grad*/); } -void DynamicRecurrentOp::ArgCache::CacheScopes(const Scope& scope, - const rnn::Argument& arg) { +void RNNAlgorithm::ArgCache::CacheScopes(const Scope& scope, + const rnn::Argument& arg) { auto scopes_var = scope.FindVar(arg.step_scopes); PADDLE_ENFORCE(scopes_var != nullptr, "the step_scopes output argument [%s] should be created first " @@ -289,45 +328,85 @@ void DynamicRecurrentOp::ArgCache::CacheScopes(const Scope& scope, this->scopes = scopes_var->GetMutable>(); } -void DynamicRecurrentOp::ArgCache::CacheInlinks( +void RNNAlgorithm::ArgCache::CacheInlinks( const Scope& scope, const std::vector& names) { for (auto name : names) { auto* var = GetVariable(scope, name); - inlinks[name] = var; + inputs[name] = var; } } -void DynamicRecurrentOp::ArgCache::CacheOutlinks( +void RNNAlgorithm::ArgCache::CacheOutlinks( const Scope& scope, const std::vector& names) { for (auto name : names) { auto* var = GetVariable(scope, name); - outlinks[name] = var; + outputs[name] = var; } } -Variable* DynamicRecurrentOp::ArgCache::GetVariable(const Scope& scope, - const std::string& name) { +Variable* RNNAlgorithm::ArgCache::GetVariable(const Scope& scope, + const std::string& name) { auto* var = scope.FindVar(name); PADDLE_ENFORCE_NOT_NULL(var, "variable [%s] not exist in scope", name); return var; } -LoDTensor* DynamicRecurrentOp::ArgCache::GetTensor( - const framework::Scope& scope, const std::string& name) { +LoDTensor* RNNAlgorithm::ArgCache::GetTensor(const framework::Scope& scope, + const std::string& name) { auto* var = GetVariable(scope, name); return var->GetMutable(); } -const rnn::ArgumentName DynamicRecurrentOp::kArgName{ - "step_net", "step_scopes", "inlinks", "outlinks", - "memories", "pre_memories", "boot_memories"}; +const std::array RNNAlgorithm::kArgNames{ + rnn::ArgumentName{"step_unit", "step_scopes", "inputs", "outputs", "states", + "ex_states", "initial_states"}, + rnn::ArgumentName{"step_unit", "step_scopes@GRAD", "outputs@GRAD", + "inputs@GRAD", "states", "ex_states", + "initial_states@GRAD"}}; + +void DynamicRecurrentOp::Run(const framework::Scope& scope, + const platform::DeviceContext& dev_ctx) const { + rnn.Run( + scope, *dynamic_cast(this), dev_ctx); +} void DynamicRecurrentGradientOp::Run( - const Scope& scope, const platform::DeviceContext& dev_ctx) const {} + const Scope& scope, const platform::DeviceContext& dev_ctx) const { + rnn.Run( + scope, *dynamic_cast(this), dev_ctx); +} + +class DynamicRecurrentOpProtoAndCheckerMaker + : public framework::OpProtoAndCheckerMaker { + public: + DynamicRecurrentOpProtoAndCheckerMaker(framework::OpProto* proto, + framework::OpAttrChecker* op_checker) + : OpProtoAndCheckerMaker(proto, op_checker) { + const auto& name = + RNNAlgorithm::kArgNames[RNNAlgorithm::ComputeMode::kForward]; + // inputs and outputs stored in proto + AddInput(name.inlinks, + "the inputs that need to be segmented for each step.") + .AsDuplicable(); + AddInput(name.initial_states, "variables to initialize states.") + .AsDuplicable(); + + AddOutput(name.outlinks, "the outputs that need to concated for all steps.") + .AsDuplicable(); + AddOutput(name.step_scopes, "step scopes"); + + // Attributes stored in AttributeMap + AddAttr>(name.ex_states, "names of ex_states"); + AddAttr>(name.states, "names of states"); + + AddComment("This is a RNN operator for varience-length sequences."); + } +}; } // namespace operators } // namespace paddle -REGISTER_OP_WITHOUT_GRADIENT( - dynamic_recurrent, paddle::operators::DynamicRecurrentOp, - paddle::operators::DynamicRecurrentOpProtoAndCheckerMaker); +REGISTER_OP(dynamic_recurrent, paddle::operators::DynamicRecurrentOp, + paddle::operators::DynamicRecurrentOpProtoAndCheckerMaker, + dynamic_recurrent_grad, + paddle::operators::DynamicRecurrentGradientOp); diff --git a/paddle/operators/dynamic_recurrent_op.h b/paddle/operators/dynamic_recurrent_op.h index ec80a1c90e..5b0548c3a4 100644 --- a/paddle/operators/dynamic_recurrent_op.h +++ b/paddle/operators/dynamic_recurrent_op.h @@ -27,47 +27,39 @@ namespace paddle { namespace operators { -class DynamicRecurrentOp : public framework::OperatorBase { +class RNNAlgorithm { public: - static const rnn::ArgumentName kArgName; + enum ComputeMode { kForward = 0, kBackward = 1 }; + static const std::array kArgNames; using value_type = float; - DynamicRecurrentOp(const std::string& type, - const framework::VariableNameMap& inputs, - const framework::VariableNameMap& outputs, - const framework::AttributeMap& attrs) - : OperatorBase(type, inputs, outputs, attrs) {} - - DynamicRecurrentOp(const DynamicRecurrentOp& o) - : framework::OperatorBase( - static_cast(o)) { - // TODO(yuyang18): Implement copy ctor well. - PADDLE_THROW("Not implemented"); - } - - void Run(const framework::Scope& scope, - const platform::DeviceContext& dev_ctx) const override; - + /* + * Different `Run` method for forward and backward, `_` is just for template + * specifialization. + */ + template + void Run(const framework::Scope& scope, const framework::OperatorBase& op, + const platform::DeviceContext& dev_ctx); /* * Split the inputs(LoDTensors) to segments for each time step. */ - void SplitInputs() const; + void SplitInputs(); /* * Create step-scopes to store temporary outputs in each time steps. */ - void CreateScopes() const; + void CreateScopes(); /* * Link TensorArray steps to the corresponding variables located in * step-scopes. */ - void WriteStepInputs() const; + void WriteStepInputs(); /* * Write output of each step to the corresponding TensorArray. */ - void WriteStepOutputs() const; + void WriteStepOutputs(); /* * Initialize the states, each state will have a corresponding pre-state, @@ -75,54 +67,83 @@ class DynamicRecurrentOp : public framework::OperatorBase { * pre-state in the first time step will be initialized with an zero tensor or * a tensor in parent scope if is provided. */ - void InitStates() const; + void InitStates(); /* * Create state variables for each time step. */ - void CreateState(const rnn::MemoryAttr& memory, size_t step) const; + void CreateState(const rnn::StateAttr& state, size_t step); /* * Link pre-state variable in current scope to the state variable in the - * previous time step (scope). + * previous time step (scope) by reference. + */ + void LinkState(const rnn::StateAttr& state, size_t step); + + /* + * Link the pre-state of the first time step to the `boot-state` in parent's + * scope. + */ + void LinkInitialState(const rnn::StateAttr& state); + + /* + * Copy the gradient from `pre-state` in the first step-scope to the + * `boot-state` in parent's scope. + */ + void ExportInitialStateGradient(const rnn::StateAttr& state); + + /* + * Calculate time steps. */ - void LinkState(const rnn::MemoryAttr& memory, size_t step) const; + void RunSteps(); /* * Concatenate outputs in each time step and generate a LoDTensor. */ - void ConcatOutputs() const; + void ConcatOutputs(); + + void SetComputeMode(ComputeMode mode) { mode_ = mode; } + bool IsForward() const { return mode_ == ComputeMode::kForward; } + bool IsBackward() const { return mode_ == ComputeMode::kBackward; } /* - * set a stepnet that is created according to a RecurrentOp's stepnet. + * set a step unit that is created according to a RecurrentOp's step unit. */ - void SetStepNet(std::unique_ptr net) { - PADDLE_ENFORCE_NOT_NULL(net); - stepnet_ = std::move(net); + void SetStepUnit(std::unique_ptr step_unit) { + PADDLE_ENFORCE_NOT_NULL(step_unit); + step_unit_ = std::move(step_unit); } - const OperatorBase& GetStepNet() const { return *stepnet_; } + const framework::OperatorBase& GetStepUnit() const { return *step_unit_; } const framework::TensorArray& state(const std::string& name) const { - return states_[name]; + auto it = states_.find(name); + PADDLE_ENFORCE(it != states_.end()); + return it->second; } const framework::TensorArray& step_input(const std::string& name) const { - return step_inputs_[name]; + auto it = step_inputs_.find(name); + PADDLE_ENFORCE(it != step_inputs_.end()); + return it->second; } const framework::TensorArray& step_output(const std::string& name) const { - return step_outputs_[name]; + auto it = step_outputs_.find(name); + PADDLE_ENFORCE(it != step_outputs_.end()); + return it->second; } protected: struct ArgCache { framework::Scope const* scope; std::vector* scopes; - std::map inlinks; - std::map outlinks; + std::map inputs; + std::map outputs; + platform::DeviceContext const* dev_ctx; size_t num_steps{0}; - void Init(const rnn::ArgumentName& name, const OperatorBase& op, - const framework::Scope& scope, rnn::Argument* arg); + void Init(const rnn::ArgumentName& name, const framework::OperatorBase& op, + const framework::Scope& scope, + platform::DeviceContext const* dev_ctx, rnn::Argument* arg); framework::Scope& GetScope(size_t index) { PADDLE_ENFORCE_LT(index, num_steps); @@ -133,8 +154,8 @@ class DynamicRecurrentOp : public framework::OperatorBase { const std::string& name); private: - void InitArgument(const rnn::ArgumentName& name, const OperatorBase& op, - rnn::Argument* arg); + void InitArgument(const rnn::ArgumentName& name, + const framework::OperatorBase& op, rnn::Argument* arg); void CacheScopes(const framework::Scope& scope, const rnn::Argument& arg); void CacheInlinks(const framework::Scope& scope, const std::vector& names); @@ -145,27 +166,49 @@ class DynamicRecurrentOp : public framework::OperatorBase { }; private: - std::unique_ptr stepnet_; - mutable std::map states_; - mutable std::map step_inputs_; - mutable std::map step_outputs_; - mutable std::map> - dy_seq_metas_; - mutable rnn::Argument arg_; - mutable ArgCache cache_; + std::unique_ptr step_unit_; + std::map states_; + std::map step_inputs_; + std::map step_outputs_; + std::map> dy_seq_metas_; + rnn::Argument arg_; + ArgCache cache_; + ComputeMode mode_{ComputeMode::kForward}; #ifdef PADDLE_WITH_TESTING - friend class DynamicRecurrentOpTestHelper; - FRIEND_TEST(DynamicRecurrentOpTestHelper, SplitInputs); - FRIEND_TEST(DynamicRecurrentOpTestHelper, CreateCache); - FRIEND_TEST(DynamicRecurrentOpTestHelper, CreateScopes); - FRIEND_TEST(DynamicRecurrentOpTestHelper, WriteStepInputs); - FRIEND_TEST(DynamicRecurrentOpTestHelper, WriteStepOutputs); - FRIEND_TEST(DynamicRecurrentOpTestHelper, InitStates); - FRIEND_TEST(DynamicRecurrentOpTestHelper, ConcatOutputs); + // test forward + friend class RNNAlgorithmTestHelper; + FRIEND_TEST(RNNAlgorithmTestHelper, SplitInputs); + FRIEND_TEST(RNNAlgorithmTestHelper, CreateCache); + FRIEND_TEST(RNNAlgorithmTestHelper, CreateScopes); + FRIEND_TEST(RNNAlgorithmTestHelper, WriteStepInputs); + FRIEND_TEST(RNNAlgorithmTestHelper, WriteStepOutputs); + FRIEND_TEST(RNNAlgorithmTestHelper, InitStates); + FRIEND_TEST(RNNAlgorithmTestHelper, ConcatOutputs); +// TODO(superjom) test backward #endif }; +class DynamicRecurrentOp : public framework::OperatorBase { + public: + DynamicRecurrentOp(const std::string& type, + const framework::VariableNameMap& inputs, + const framework::VariableNameMap& outputs, + const framework::AttributeMap& attrs) + : OperatorBase(type, inputs, outputs, attrs) {} + + DynamicRecurrentOp(const DynamicRecurrentOp& o) + : framework::OperatorBase( + static_cast(o)) { + PADDLE_THROW("Not implemented"); + } + + void Run(const framework::Scope& scope, + const platform::DeviceContext& dev_ctx) const override; + + mutable RNNAlgorithm rnn; +}; + class DynamicRecurrentGradientOp : public framework::OperatorBase { public: DynamicRecurrentGradientOp(const std::string& type, @@ -174,8 +217,16 @@ class DynamicRecurrentGradientOp : public framework::OperatorBase { const framework::AttributeMap& attrs) : OperatorBase(type, inputs, outputs, attrs) {} + DynamicRecurrentGradientOp(const DynamicRecurrentGradientOp& o) + : framework::OperatorBase( + static_cast(o)) { + PADDLE_THROW("Not implemented"); + } + void Run(const framework::Scope& scope, const platform::DeviceContext& dev_ctx) const override; + + mutable RNNAlgorithm rnn; }; } // namespace operators diff --git a/paddle/operators/dynamic_recurrent_op_test.cc b/paddle/operators/dynamic_recurrent_op_test.cc index 36f405568d..fff63efb24 100644 --- a/paddle/operators/dynamic_recurrent_op_test.cc +++ b/paddle/operators/dynamic_recurrent_op_test.cc @@ -43,16 +43,16 @@ LoDTensor* CreateVar(Scope& scope, std::string name, framework::DDim dims, return tensor; } -class DynamicRecurrentOpTestHelper : public ::testing::Test { +class RNNAlgorithmTestHelper : public ::testing::Test { protected: - const rnn::ArgumentName argname = DynamicRecurrentOp::kArgName; + const rnn::ArgumentName argname = RNNAlgorithm::kArgNames[0]; virtual void SetUp() override { CreateGlobalVariables(); auto op_desc = CreateOpDesc(); op = paddle::framework::OpRegistry::CreateOp(op_desc, nullptr); - dop = dynamic_cast(op.get()); + dop = &(dynamic_cast(op.get())->rnn); InitCacheManually(); InitStepNet(); } @@ -63,20 +63,20 @@ class DynamicRecurrentOpTestHelper : public ::testing::Test { op_desc.set_type("dynamic_recurrent"); OpDescNewVar(argname.inlinks, {"in0"}, op_desc.add_inputs()); - OpDescNewVar(argname.boot_memories, {"boot_mem"}, op_desc.add_inputs()); + OpDescNewVar(argname.initial_states, {"boot_mem"}, op_desc.add_inputs()); OpDescNewVar(argname.step_scopes, {"step_scopes"}, op_desc.add_outputs()); OpDescNewVar(argname.outlinks, {"out0"}, op_desc.add_outputs()); - // set pre-memories + // set pre-states auto pre_memories = op_desc.mutable_attrs()->Add(); - pre_memories->set_name(argname.pre_memories); + pre_memories->set_name(argname.ex_states); pre_memories->set_type(paddle::framework::AttrType::STRINGS); auto pre_memories_item = pre_memories->add_strings(); *pre_memories_item = "mem@pre"; - // set memories + // set states auto memories = op_desc.mutable_attrs()->Add(); - memories->set_name(argname.memories); + memories->set_name(argname.states); memories->set_type(paddle::framework::AttrType::STRINGS); auto memories_item = memories->add_strings(); *memories_item = "mem"; @@ -113,32 +113,33 @@ class DynamicRecurrentOpTestHelper : public ::testing::Test { } void InitCacheManually() { - dop->cache_.Init(DynamicRecurrentOp::kArgName, *dop, scope, &dop->arg_); + dop->cache_.Init(RNNAlgorithm::kArgNames[0], *op, scope, &device_context, + &dop->arg_); } void InitStepNet() { std::unique_ptr stepnet{new NetOp}; dynamic_cast(stepnet.get()) ->AppendOp(std::unique_ptr(new TestOp( - "test", {{"inlinks", {"in0"}}, {"boot_memories", {"boot_mem"}}}, - {{"outlinks", {"out0"}}, {"step_scopes", {"step_scopes"}}}, {}))); - dop->SetStepNet(std::move(stepnet)); + "test", {{"inputs", {"in0"}}, {"initial_states", {"boot_mem"}}}, + {{"outputs", {"out0"}}, {"step_scopes", {"step_scopes"}}}, {}))); + dop->SetStepUnit(std::move(stepnet)); } protected: - DynamicRecurrentOp* dop; + RNNAlgorithm* dop; std::unique_ptr op; paddle::platform::CPUDeviceContext device_context; paddle::framework::Scope scope; }; -TEST_F(DynamicRecurrentOpTestHelper, CreateCache) { +TEST_F(RNNAlgorithmTestHelper, CreateCache) { const rnn::Argument& arg = dop->arg_; ASSERT_EQ(arg.inlinks.size(), 1UL); ASSERT_EQ(arg.outlinks.size(), 1UL); } -TEST_F(DynamicRecurrentOpTestHelper, SplitInputs) { +TEST_F(RNNAlgorithmTestHelper, SplitInputs) { dop->SplitInputs(); auto& in0_ta = dop->step_inputs_["in0"]; ASSERT_EQ(in0_ta.size(), 4UL); @@ -153,14 +154,14 @@ TEST_F(DynamicRecurrentOpTestHelper, SplitInputs) { EXPECT_EQ(batch3.dims()[0], 1); } -TEST_F(DynamicRecurrentOpTestHelper, CreateScopes) { +TEST_F(RNNAlgorithmTestHelper, CreateScopes) { dop->SplitInputs(); dop->CreateScopes(); ASSERT_EQ(dop->cache_.num_steps, 4UL); ASSERT_EQ(dop->cache_.scopes->size(), 4UL); } -TEST_F(DynamicRecurrentOpTestHelper, WriteStepInputs) { +TEST_F(RNNAlgorithmTestHelper, WriteStepInputs) { dop->SplitInputs(); dop->CreateScopes(); dop->WriteStepInputs(); @@ -173,7 +174,7 @@ TEST_F(DynamicRecurrentOpTestHelper, WriteStepInputs) { } } -TEST_F(DynamicRecurrentOpTestHelper, WriteStepOutputs) { +TEST_F(RNNAlgorithmTestHelper, WriteStepOutputs) { dop->SplitInputs(); dop->CreateScopes(); dop->WriteStepInputs(); @@ -187,11 +188,12 @@ TEST_F(DynamicRecurrentOpTestHelper, WriteStepOutputs) { } } -TEST_F(DynamicRecurrentOpTestHelper, ConcatOutputs) { +TEST_F(RNNAlgorithmTestHelper, ConcatOutputs) { // Let's leave this test to python unittest. } -TEST_F(DynamicRecurrentOpTestHelper, InitStates) { +TEST_F(RNNAlgorithmTestHelper, InitStates) { + dop->SetComputeMode(RNNAlgorithm::ComputeMode::kForward); dop->SplitInputs(); dop->CreateScopes(); dop->WriteStepInputs(); @@ -208,12 +210,6 @@ TEST_F(DynamicRecurrentOpTestHelper, InitStates) { auto* boot_state = scope.FindVar("boot_mem"); ASSERT_TRUE(boot_state != nullptr); - - if (step == 0) { - // check pre_state is a reference of boot_state - ASSERT_EQ(boot_state->Get().data(), - pre_state->Get().data()); - } } } diff --git a/paddle/operators/recurrent_op.cc b/paddle/operators/recurrent_op.cc index dcc90e5d87..40303e3adf 100644 --- a/paddle/operators/recurrent_op.cc +++ b/paddle/operators/recurrent_op.cc @@ -42,7 +42,7 @@ void RecurrentAlgorithm::Run(const Scope& scope, for (size_t step_id = 0; step_id < seq_len; step_id++) { if (step_id > 0) { - rnn::LinkMemories(step_scopes, arg_->memories, step_id, -1); + rnn::LinkMemories(step_scopes, arg_->states, step_id, -1); } (*stepnet_)->Run(*step_scopes[step_id], dev_ctx); } @@ -59,7 +59,8 @@ void RecurrentAlgorithm::CreateScopes(const Scope& scope, // Now all variables in scope must be created outside of op. PADDLE_ENFORCE_NOT_NULL(stepnet_); - PADDLE_ENFORCE(!(*stepnet_)->Outputs().empty(), "stepnet_ op has no outputs"); + PADDLE_ENFORCE(!(*stepnet_)->Outputs().empty(), + "step_unit_ op has no outputs"); if (seq_len > step_scopes->size()) { for (size_t i = step_scopes->size(); i < seq_len; ++i) { @@ -86,7 +87,7 @@ void RecurrentAlgorithm::CreateScopes(const Scope& scope, } void RecurrentAlgorithm::InitMemories(Scope* step_scope) const { - for (auto& attr : arg_->memories) { + for (auto& attr : arg_->states) { auto* pre_mem = step_scope->Var(attr.pre_var)->GetMutable(); PADDLE_ENFORCE(step_scope->FindVar(attr.boot_var) != nullptr, "memory [%s]'s boot variable [%s] not exists", attr.var, @@ -100,12 +101,12 @@ void RecurrentAlgorithm::InitMemories(Scope* step_scope) const { } const rnn::ArgumentName RecurrentOp::kArgName{ - "step_net", "step_scopes", "inlinks", "outlinks", - "memories", "pre_memories", "boot_memories"}; + "step_net", "step_scopes", "inputs", "outputs", + "states", "ex_states", "initial_states"}; const rnn::ArgumentName RecurrentGradientOp::kArgName{ - "step_net", "step_scopes@GRAD", "outlinks@GRAD", "inlinks@GRAD", - "memories", "pre_memories", "boot_memories@GRAD"}; + "step_net", "step_scopes@GRAD", "outputs@GRAD", "inputs@GRAD", + "states", "ex_states", "initial_states@GRAD"}; RecurrentOp::RecurrentOp(const std::string& type, const framework::VariableNameMap& inputs, @@ -127,7 +128,7 @@ class RecurrentAlgorithmProtoAndCheckerMaker AddInput(name.inlinks, "the inputs that need to be segmented for each step.") .AsDuplicable(); - AddInput(name.boot_memories, "variables to initialize memories.") + AddInput(name.initial_states, "variables to initialize states.") .AsDuplicable(); AddOutput(name.outlinks, "the outputs that need to concated for all steps.") @@ -135,9 +136,8 @@ class RecurrentAlgorithmProtoAndCheckerMaker AddOutput(name.step_scopes, "step scopes"); // Attributes stored in AttributeMap - AddAttr>(name.pre_memories, - "names of pre-memories"); - AddAttr>(name.memories, "names of memories"); + AddAttr>(name.ex_states, "names of pre-states"); + AddAttr>(name.states, "names of states"); AddComment("This is a recurrent group operator."); } @@ -152,7 +152,7 @@ void RecurrentGradientAlgorithm::Run( rnn::SegmentInputs(step_scopes, arg_->inlinks, seq_len); for (int step_id = seq_len - 1; step_id >= 0; --step_id) { if (static_cast(step_id) != seq_len - 1) { - rnn::LinkMemories(step_scopes, arg_->memories, step_id, 1); + rnn::LinkMemories(step_scopes, arg_->states, step_id, 1); } (*stepnet_)->Run(*step_scopes[step_id], dev_ctx); } @@ -162,7 +162,7 @@ void RecurrentGradientAlgorithm::Run( void RecurrentGradientAlgorithm::LinkBootMemoryGradients( Scope* step_scope) const { - for (auto& attr : arg_->memories) { + for (auto& attr : arg_->states) { PADDLE_ENFORCE(step_scope->FindVar(attr.var) != nullptr, "memory variable [%s] does not exists", attr.var); PADDLE_ENFORCE(step_scope->FindVar(attr.boot_var) != nullptr, diff --git a/paddle/operators/rnn/recurrent_op_utils.cc b/paddle/operators/rnn/recurrent_op_utils.cc index d0725f5023..ee61ea300c 100644 --- a/paddle/operators/rnn/recurrent_op_utils.cc +++ b/paddle/operators/rnn/recurrent_op_utils.cc @@ -36,7 +36,7 @@ void SegmentInputs(const std::vector& step_scopes, LoDTensor* input = input_var->GetMutable(); f::DDim dims = input->dims(); PADDLE_ENFORCE_EQ(static_cast(dims[0]), seq_len, - "all the inlinks be the same length"); + "all the inputs be the same length"); f::DDim step_dims = slice_ddim(dims, 1, dims.size()); for (size_t j = 0; j < seq_len; j++) { Tensor* step_input = @@ -78,7 +78,7 @@ void ConcatOutputs(const std::vector& step_scopes, } void LinkMemories(const std::vector& scopes, - const std::vector& memories, + const std::vector& memories, const size_t step_id, const int offset) { PADDLE_ENFORCE_LT(step_id, scopes.size(), "step [%d] is out of range of step scopes' size [%d]", @@ -106,26 +106,26 @@ void InitArgument(const ArgumentName& name, Argument* arg, arg->inlinks = op.Inputs(name.inlinks); arg->outlinks = op.Outputs(name.outlinks); - auto& boot_memories = - is_grad ? op.Outputs(name.boot_memories) : op.Inputs(name.boot_memories); + auto& boot_memories = is_grad ? op.Outputs(name.initial_states) + : op.Inputs(name.initial_states); // attributes - auto& memories = op.Attr>(name.memories); - auto& pre_memories = op.Attr>(name.pre_memories); + auto& memories = op.Attr>(name.states); + auto& pre_memories = op.Attr>(name.ex_states); PADDLE_ENFORCE(memories.size() == boot_memories.size(), - "the size of memories, boot_memories don't match:%d,%d", + "the size of states, initial_states don't match:%d,%d", memories.size(), boot_memories.size()); PADDLE_ENFORCE(pre_memories.size() == boot_memories.size(), - "the size of pre_memories, boot_memories don't match:%d,%d", + "the size of ex_states, initial_states don't match:%d,%d", pre_memories.size(), boot_memories.size()); - PADDLE_ENFORCE(memories.size() > 0, "more than 1 memories should be set"); + PADDLE_ENFORCE(memories.size() > 0, "more than 1 states should be set"); for (size_t i = 0; i < memories.size(); ++i) { - rnn::MemoryAttr mem_attr; + rnn::StateAttr mem_attr; mem_attr.var = memories[i]; mem_attr.pre_var = pre_memories[i]; mem_attr.boot_var = boot_memories[i]; - (arg->memories).push_back(mem_attr); + (arg->states).push_back(mem_attr); } } diff --git a/paddle/operators/rnn/recurrent_op_utils.h b/paddle/operators/rnn/recurrent_op_utils.h index fe173edb24..fb0e158e07 100644 --- a/paddle/operators/rnn/recurrent_op_utils.h +++ b/paddle/operators/rnn/recurrent_op_utils.h @@ -31,7 +31,7 @@ using Scope = framework::Scope; * boot memories in father scope. Other attributes are copied from Op's proto * attributes. */ -struct MemoryAttr { +struct StateAttr { // name of current state variable std::string var; // name of previous step's state variable @@ -46,7 +46,7 @@ struct Argument { std::string step_scopes; std::vector inlinks; std::vector outlinks; - std::vector memories; + std::vector states; }; struct ArgumentName { @@ -54,9 +54,9 @@ struct ArgumentName { std::string step_scopes; std::string inlinks; std::string outlinks; - std::string memories; // the memory name - std::string pre_memories; // the previous memory name - std::string boot_memories; // the boot memory name + std::string states; // the memory name + std::string ex_states; // the previous memory name + std::string initial_states; // the boot memory name }; /** @@ -74,7 +74,7 @@ void ConcatOutputs(const std::vector& step_scopes, const size_t seq_len, const platform::DeviceContext& ctx); void LinkMemories(const std::vector& step_scopes, - const std::vector& memories, const size_t step_id, + const std::vector& memories, const size_t step_id, const int offset); void InitArgument(const ArgumentName& name, Argument* arg, diff --git a/paddle/pybind/pybind.cc b/paddle/pybind/pybind.cc index 9ef47b88fd..e5ddc14587 100644 --- a/paddle/pybind/pybind.cc +++ b/paddle/pybind/pybind.cc @@ -413,18 +413,18 @@ All parameter, weight, gradient are variables in Paddle. return static_cast( rnn_op.release()); }) - .def("set_stepnet", + .def("set_step_unit", [](operators::DynamicRecurrentOp &self, const operators::NetOp &net) - -> void { self.SetStepNet(net.Clone()); }) + -> void { self.rnn.SetStepUnit(net.Clone()); }) .def("get_state", [](operators::DynamicRecurrentOp &self, const std::string &name) - -> const TensorArray & { return self.state(name); }) + -> const TensorArray & { return self.rnn.state(name); }) .def("get_step_input", [](operators::DynamicRecurrentOp &self, const std::string &name) - -> const TensorArray & { return self.step_input(name); }) + -> const TensorArray & { return self.rnn.step_input(name); }) .def("get_step_output", [](operators::DynamicRecurrentOp &self, const std::string &name) - -> const TensorArray & { return self.step_output(name); }); + -> const TensorArray & { return self.rnn.step_output(name); }); // cond_op py::class_(m, "CondOp") diff --git a/python/paddle/v2/framework/tests/test_dynamic_recurrent_op.py b/python/paddle/v2/framework/tests/test_dynamic_recurrent_op.py index 2b01e43454..fa2ccd0c3b 100644 --- a/python/paddle/v2/framework/tests/test_dynamic_recurrent_op.py +++ b/python/paddle/v2/framework/tests/test_dynamic_recurrent_op.py @@ -4,6 +4,12 @@ import unittest from paddle.v2.framework.op import Operator, DynamicRecurrentOp import numpy as np +# for siplicity, just one level LoD +lod_py = [[0, 4, 7, 9, 10]] +input_dim = 30 +num_sents = len(lod_py[0]) - 1 +weight_dim = 15 + def create_tensor(scope, name, shape, np_data): tensor = scope.var(name).get_tensor() @@ -12,6 +18,17 @@ def create_tensor(scope, name, shape, np_data): return tensor +class PyRNNStep(object): + def __init__(self): + + self.x = np.random.normal(size=(lod_py[0][-1], + input_dim)).astype("float32") + self.W = np.random.normal(size=(input_dim, input_dim)).astype("float32") + self.U = np.random.normal(size=(input_dim, input_dim)).astype("float32") + self.h_boot = np.random.normal(size=(num_sents, + input_dim)).astype("float32") + + class DynamicRecurrentOpTest(unittest.TestCase): ''' Test RNNOp @@ -23,17 +40,13 @@ class DynamicRecurrentOpTest(unittest.TestCase): - U vars: - x - memories: + states: - h outputs: - h ''' - # for siplicity, just one level LoD - lod_py = [[0, 4, 7, 9, 10]] - input_dim = 30 - num_sents = len(lod_py[0]) - 1 - weight_dim = 15 + py = PyRNNStep() def forward(self): self.scope = core.Scope() @@ -42,64 +55,55 @@ class DynamicRecurrentOpTest(unittest.TestCase): self.create_step_net() ctx = core.DeviceContext.create(core.CPUPlace()) self.rnnop.run(self.scope, ctx) - state = self.rnnop.get_state("h@mem") + state = self.rnnop.get_state("h@state") print 'state size: ', state.size() step_inputs = self.rnnop.get_step_input("x") print "x size ", step_inputs.size() for i in range(step_inputs.size()): print "x %d" % i, np.array(step_inputs.read(i).get_dims()) - step_outputs = self.rnnop.get_step_output('h@mem') + step_outputs = self.rnnop.get_step_output('h@state') print 'step_outputs.size ', step_outputs.size() - output = self.scope.find_var("h@mem").get_tensor() - + output = self.scope.find_var("h@state").get_tensor() print 'output', np.array(output).shape def create_global_variables(self): - x = np.random.normal(size=(self.lod_py[0][-1], - self.input_dim)).astype("float32") - W = np.random.normal(size=(self.input_dim, - self.input_dim)).astype("float32") - U = np.random.normal(size=(self.input_dim, - self.input_dim)).astype("float32") - h_boot = np.random.normal(size=(self.num_sents, - self.input_dim)).astype("float32") # create inlink - x_tensor = create_tensor(self.scope, "x", - [self.num_sents, self.input_dim], x) - x_tensor.set_lod(self.lod_py) - create_tensor(self.scope, "W", [self.input_dim, self.input_dim], W) - create_tensor(self.scope, "U", [self.input_dim, self.input_dim], U) - create_tensor(self.scope, "h_boot", [self.num_sents, self.input_dim], - h_boot) + x_tensor = create_tensor(self.scope, "x", [num_sents, input_dim], + self.py.x) + x_tensor.set_lod(lod_py) + create_tensor(self.scope, "W", [input_dim, input_dim], self.py.W) + create_tensor(self.scope, "U", [input_dim, input_dim], self.py.U) + create_tensor(self.scope, "h_boot", [num_sents, input_dim], + self.py.h_boot) self.scope.var("step_scopes") - self.scope.var("h@mem") + self.scope.var("h@state") def create_rnn_op(self): # create RNNOp self.rnnop = DynamicRecurrentOp( # inputs - inlinks=["x"], - boot_memories=["h_boot"], - step_net="stepnet", + inputs=["x"], + initial_states=["h_boot"], + step_net="step_unit", # outputs - outlinks=["h@mem"], + outputs=["h@state"], step_scopes="step_scopes", # attributes - pre_memories=["h@pre"], - memories=["h@mem"]) + ex_states=["h@pre"], + states=["h@state"]) def create_step_net(self): - stepnet = core.Net.create() + step_unit = core.Net.create() x_fc_op = Operator("mul", X="x", Y="W", Out="Wx") h_fc_op = Operator("mul", X="h@pre", Y="U", Out="Uh") sum_op = Operator("sum", X=["Wx", "Uh"], Out="sum") - sig_op = Operator("sigmoid", X="sum", Y="h@mem") + sig_op = Operator("sigmoid", X="sum", Y="h@state") for op in [x_fc_op, h_fc_op, sum_op, sig_op]: - stepnet.append_op(op) - stepnet.complete_add_op(True) - self.rnnop.set_stepnet(stepnet) + step_unit.append_op(op) + step_unit.complete_add_op(True) + self.rnnop.set_step_unit(step_unit) def test_forward(self): print 'test recurrent op forward' @@ -107,5 +111,58 @@ class DynamicRecurrentOpTest(unittest.TestCase): print 'pd_output', pd_output +class RecurrentGradientOpTest(unittest.TestCase): + py = PyRNNStep() + + def create_forward_op(self): + # create RNNOp + self.forward_op = DynamicRecurrentOp( + # inputs + inputs=["x"], + initial_states=["h_boot"], + step_net="step_unit", + # outputs + outputs=["h@state"], + step_scopes="step_scopes", + # attributes + ex_states=["h@pre"], + states=["h@state"]) + + def create_gradient_op(self): + a = set() + backward_op = core.DynamicRecurrentOp.backward(self.forward_op, a) + + def create_step_net(self): + step_unit = core.Net.create() + x_fc_op = Operator("mul", X="x", Y="W", Out="Wx") + h_fc_op = Operator("mul", X="h@pre", Y="U", Out="Uh") + sum_op = Operator("sum", X=["Wx", "Uh"], Out="sum") + sig_op = Operator("sigmoid", X="sum", Y="h@state") + + for op in [x_fc_op, h_fc_op, sum_op, sig_op]: + step_unit.append_op(op) + step_unit.complete_add_op(True) + self.forward_op.set_step_unit(step_unit) + + def create_global_variables(self): + # create inlink + x_tensor = create_tensor(self.scope, "x", [num_sents, input_dim], + self.py.x) + x_tensor.set_lod(lod_py) + create_tensor(self.scope, "W", [input_dim, input_dim], self.py.W) + create_tensor(self.scope, "U", [input_dim, input_dim], self.py.U) + create_tensor(self.scope, "h_boot", [num_sents, input_dim], + self.py.h_boot) + self.scope.var("step_scopes") + self.scope.var("h@state") + + def test_grad(self): + self.scope = core.Scope() + self.create_forward_op() + self.create_global_variables() + self.create_step_net() + self.create_gradient_op() + + if __name__ == '__main__': unittest.main() diff --git a/python/paddle/v2/framework/tests/test_recurrent_op.py b/python/paddle/v2/framework/tests/test_recurrent_op.py index 191ce0b0c8..cc4008c0d8 100644 --- a/python/paddle/v2/framework/tests/test_recurrent_op.py +++ b/python/paddle/v2/framework/tests/test_recurrent_op.py @@ -132,15 +132,15 @@ class RecurrentOpTest(unittest.TestCase): # create RNNOp self.rnnop = RecurrentOp( # inputs - inlinks=["x"], - boot_memories=["h_boot"], + inputs=["x"], + initial_states=["h_boot"], step_net="stepnet", # outputs - outlinks=["h@mem"], + outputs=["h@mem"], step_scopes="step_scopes", # attributes - pre_memories=["h@pre"], - memories=["h@mem"]) + ex_states=["h@pre"], + states=["h@mem"]) def create_step_net(self): stepnet = core.Net.create() @@ -169,15 +169,15 @@ class RecurrentGradientOpTest(unittest.TestCase): def create_forward_op(self): self.forward_op = RecurrentOp( # inputs - inlinks=["x"], - boot_memories=["h_boot"], + inputs=["x"], + initial_states=["h_boot"], step_net="stepnet", # outputs - outlinks=["h"], + outputs=["h"], step_scopes="step_scopes", # attributes - pre_memories=["h@pre"], - memories=["h@alias"]) + ex_states=["h@pre"], + states=["h@alias"]) # create a stepnet for RNN stepnet = core.Net.create() From 05ece8481e8ed3c254cc7a66ca7e4f3583a36d61 Mon Sep 17 00:00:00 2001 From: fengjiayi Date: Fri, 20 Oct 2017 10:15:33 -0700 Subject: [PATCH 65/76] Trainable conv net of MNIST (#4960) * Init file * Update * Update * Complete conv net of MNIST --- python/paddle/v2/framework/nets.py | 9 +- .../tests/test_recognize_digits_conv.py | 92 +++++++++++++++++++ 2 files changed, 98 insertions(+), 3 deletions(-) create mode 100644 python/paddle/v2/framework/tests/test_recognize_digits_conv.py diff --git a/python/paddle/v2/framework/nets.py b/python/paddle/v2/framework/nets.py index 381da55da3..8a83ebfb96 100644 --- a/python/paddle/v2/framework/nets.py +++ b/python/paddle/v2/framework/nets.py @@ -7,18 +7,21 @@ def simple_img_conv_pool(input, pool_size, pool_stride, act, - program=None): + program=None, + init_program=None): conv_out = layers.conv2d( input=input, num_filters=num_filters, filter_size=filter_size, act=act, - program=program) + program=program, + init_program=init_program) pool_out = layers.pool2d( input=conv_out, pool_size=pool_size, pool_type='max', pool_stride=pool_stride, - program=program) + program=program, + init_program=init_program) return pool_out diff --git a/python/paddle/v2/framework/tests/test_recognize_digits_conv.py b/python/paddle/v2/framework/tests/test_recognize_digits_conv.py new file mode 100644 index 0000000000..2b305213df --- /dev/null +++ b/python/paddle/v2/framework/tests/test_recognize_digits_conv.py @@ -0,0 +1,92 @@ +import paddle.v2 as paddle +import paddle.v2.framework.layers as layers +import paddle.v2.framework.nets as nets +import paddle.v2.framework.core as core +import paddle.v2.framework.optimizer as optimizer + +from paddle.v2.framework.framework import Program, g_program +from paddle.v2.framework.executor import Executor + +import numpy as np + +init_program = Program() +program = Program() + +images = layers.data( + name='pixel', + shape=[1, 28, 28], + data_type='float32', + program=program, + init_program=init_program) +label = layers.data( + name='label', + shape=[1], + data_type='int32', + program=program, + init_program=init_program) +conv_pool_1 = nets.simple_img_conv_pool( + input=images, + filter_size=5, + num_filters=20, + pool_size=2, + pool_stride=2, + act="relu", + program=program, + init_program=init_program) +conv_pool_2 = nets.simple_img_conv_pool( + input=conv_pool_1, + filter_size=5, + num_filters=50, + pool_size=2, + pool_stride=2, + act="relu", + program=program, + init_program=init_program) + +predict = layers.fc(input=conv_pool_2, + size=10, + act="softmax", + program=program, + init_program=init_program) +cost = layers.cross_entropy( + input=predict, label=label, program=program, init_program=init_program) +avg_cost = layers.mean(x=cost, program=program) + +sgd_optimizer = optimizer.SGDOptimizer(learning_rate=0.001) +opts = sgd_optimizer.minimize(avg_cost) + +BATCH_SIZE = 50 +PASS_NUM = 1 +train_reader = paddle.batch( + paddle.reader.shuffle( + paddle.dataset.mnist.train(), buf_size=500), + batch_size=BATCH_SIZE) + +place = core.CPUPlace() +exe = Executor(place) + +exe.run(init_program, feed={}, fetch_list=[]) + +for pass_id in range(PASS_NUM): + count = 0 + for data in train_reader(): + img_data = np.array(map(lambda x: x[0].reshape([1, 28, 28]), + data)).astype("float32") + y_data = np.array(map(lambda x: x[1], data)).astype("int32") + y_data = y_data.reshape([BATCH_SIZE, 1]) + + tensor_img = core.LoDTensor() + tensor_y = core.LoDTensor() + tensor_img.set(img_data, place) + tensor_y.set(y_data, place) + + outs = exe.run(program, + feed={"pixel": tensor_img, + "label": tensor_y}, + fetch_list=[avg_cost]) + + loss = np.array(outs[0]) + + if loss < 10.0: + exit(0) # if avg cost less than 10.0, we think our code is good. +exit(1) From 7edc1d96c6df4a4bf6004823c3dca1197a7686ef Mon Sep 17 00:00:00 2001 From: qijun Date: Fri, 20 Oct 2017 10:48:30 -0700 Subject: [PATCH 66/76] fix clang build error --- paddle/operators/dynamic_recurrent_op.cc | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/paddle/operators/dynamic_recurrent_op.cc b/paddle/operators/dynamic_recurrent_op.cc index dce8c8d835..a0b06ac1dc 100644 --- a/paddle/operators/dynamic_recurrent_op.cc +++ b/paddle/operators/dynamic_recurrent_op.cc @@ -358,11 +358,11 @@ LoDTensor* RNNAlgorithm::ArgCache::GetTensor(const framework::Scope& scope, } const std::array RNNAlgorithm::kArgNames{ - rnn::ArgumentName{"step_unit", "step_scopes", "inputs", "outputs", "states", - "ex_states", "initial_states"}, - rnn::ArgumentName{"step_unit", "step_scopes@GRAD", "outputs@GRAD", - "inputs@GRAD", "states", "ex_states", - "initial_states@GRAD"}}; + {rnn::ArgumentName{"step_unit", "step_scopes", "inputs", "outputs", + "states", "ex_states", "initial_states"}, + rnn::ArgumentName{"step_unit", "step_scopes@GRAD", "outputs@GRAD", + "inputs@GRAD", "states", "ex_states", + "initial_states@GRAD"}}}; void DynamicRecurrentOp::Run(const framework::Scope& scope, const platform::DeviceContext& dev_ctx) const { From db7b11719b89002e3ceaf8fa3b5d66bf87559fed Mon Sep 17 00:00:00 2001 From: Yan Chunwei Date: Fri, 20 Oct 2017 14:08:48 -0400 Subject: [PATCH 67/76] change lod tensor to absolute offsets (#4952) --- paddle/framework/lod_tensor.cc | 69 ++++++++++++++++------------- paddle/framework/lod_tensor.h | 25 ++++++++--- paddle/framework/lod_tensor_test.cc | 31 +++++++------ 3 files changed, 73 insertions(+), 52 deletions(-) diff --git a/paddle/framework/lod_tensor.cc b/paddle/framework/lod_tensor.cc index 5b7badf89c..7c0ea0df78 100644 --- a/paddle/framework/lod_tensor.cc +++ b/paddle/framework/lod_tensor.cc @@ -25,31 +25,50 @@ LoD SliceLevels(const LoD& in, size_t level_begin, size_t level_end) { for (size_t i = level_begin; i < level_end; i++) { new_lod.emplace_back(in.at(i)); } + // transform the lowest level to absolute offset. + LoD abs_offset_lod = ToAbsOffset(in); + new_lod.back() = abs_offset_lod[level_end - 1]; return new_lod; } LoD SliceInLevel(const LoD& in, size_t level, size_t elem_begin, size_t elem_end) { - // slice the lod. - LoD new_lod; - new_lod.reserve(in.size() - level); - auto start = in.at(level)[elem_begin]; - auto end = in.at(level)[elem_end]; - - for (auto it = in.begin() + level; it != in.end(); it++) { - auto it_begin = std::find(it->begin(), it->end(), start); - auto it_end = std::find(it_begin, it->end(), end); - PADDLE_ENFORCE(it_begin != it->end(), "error in parsing lod info"); - PADDLE_ENFORCE(it_end != it->end(), "error in parsing lod info"); - new_lod.emplace_back(it_begin, it_end + 1); - // reset offset if tensor is copyed and sliced. - std::transform(new_lod.back().begin(), new_lod.back().end(), - new_lod.back().begin(), - [start](int v) { return v - start; }); - PADDLE_ENFORCE_EQ(new_lod.back().front(), 0, "error in slice LoD"); + PADDLE_ENFORCE_LT(level, in.size()); + PADDLE_ENFORCE_LT(elem_end, in[level].size()); + + LoD res; + res.resize(in.size() - level); + // copy the first level + res[0].assign(in[level].begin() + elem_begin, + in[level].begin() + elem_end + 1); + for (size_t lvl = 1; lvl < res.size(); lvl++) { + const auto& in_level = in[level + lvl]; + const auto& above_level = res[lvl - 1]; + auto& out_level = res[lvl]; + out_level.assign(in_level.begin() + above_level.front(), + in_level.begin() + above_level.back() + 1); } - PADDLE_ENFORCE_LE(new_lod.size(), in.size()); - return new_lod; + for (size_t lvl = 0; lvl < res.size(); lvl++) { + // to make the first offset equals 0, all the elements minus the first + // element + size_t front = res[lvl].front(); + for (auto& ele : res[lvl]) { + ele -= front; + } + } + return res; +} + +LoD ToAbsOffset(const LoD& in) { + // the lowest level stores relative offsets + if (in.empty() || in.size() == 1) return in; + LoD result = in; + for (int level = result.size() - 2; level >= 0; level--) { + for (auto& ele : result[level]) { + ele = result[level + 1][ele]; + } + } + return result; } bool operator==(const LoD& a, const LoD& b) { @@ -75,17 +94,7 @@ bool operator==(const LoD& a, const LoD& b) { size_t LoDTensor::NumElements(size_t level, size_t idx) const { PADDLE_ENFORCE_LT(level, NumLevels()); PADDLE_ENFORCE_LT(idx, NumElements(level)); - // the last level of LoD, just return number of records in Tensor - if (level == NumLevels() - 1) { - return lod_[level][idx + 1] - lod_[level][idx]; - } - // high level of LoD, and there is another lower level, return number of - // lower-level elements - auto tmp = SliceInLevel(lod_, level, idx, idx + 1); - PADDLE_ENFORCE_GE(tmp.size(), 2); - // there is a 0 as a placeholder stored in LoD, so the number of elements - // equals lod.size() - 1 - return tmp[1].size() - 1; + return lod_[level][idx + 1] - lod_[level][idx]; } void LoDTensor::ShrinkLevels(size_t level_begin, size_t level_end) { diff --git a/paddle/framework/lod_tensor.h b/paddle/framework/lod_tensor.h index 3d893baa35..dec59a5750 100644 --- a/paddle/framework/lod_tensor.h +++ b/paddle/framework/lod_tensor.h @@ -39,23 +39,36 @@ using Vector = thrust::host_vector< #endif /* - * 3-level LoD stores + * LoD is short for Level of Details. * - * 0 10 20 - * 0 5 10 15 20 - * 0 2 5 7 10 12 15 20 - * - * - in a level, each element indicates offset in the underlying Tensor + * - in a level, each element indicates relative offset of the lower level * - the first element should be 0 and that indicates that this sequence start * from 0 * - each sequence's begin and end(no-inclusive) is level[id, id+1] + * + * For example: + * 3-level LoD stores + * + * 0 2 3 + * 0 2 4 7 + * 0 2 5 7 10 12 15 20 */ using LoD = std::vector>; +/* + * Slice levels from a LoD. + * NOTE the lowest level should always be the absolute offsets of the underlying + * tensor instances. So if higher layers are sliced without the lowest level, + * the lower level of the sliced LoD will be transformed to the absolute offset. + */ LoD SliceLevels(const LoD& in, size_t level_begin, size_t level_end); LoD SliceInLevel(const LoD& in, size_t level, size_t elem_begin, size_t elem_end); +/* + * Transform an LoD from relative offsets to absolute offsets. + */ +LoD ToAbsOffset(const LoD& in); bool operator==(const LoD& a, const LoD& b); diff --git a/paddle/framework/lod_tensor_test.cc b/paddle/framework/lod_tensor_test.cc index 44f09f584f..e1e15abecf 100644 --- a/paddle/framework/lod_tensor_test.cc +++ b/paddle/framework/lod_tensor_test.cc @@ -30,8 +30,8 @@ class LoDTensorTester : public ::testing::Test { // 0 5 10 15 20 // 0 2 5 7 10 12 15 20 LoD lod; - lod.push_back(std::vector{0, 10, 20}); - lod.push_back(std::vector{0, 5, 10, 15, 20}); + lod.push_back(std::vector{0, 2, 3}); + lod.push_back(std::vector{0, 2, 5, 8}); lod.push_back(std::vector{0, 2, 5, 7, 10, 12, 15, 17, 20}); ASSERT_EQ(lod.size(), 3UL); @@ -52,14 +52,14 @@ TEST_F(LoDTensorTester, NumLevels) { ASSERT_EQ(lod_tensor_.NumLevels(), 3UL); } TEST_F(LoDTensorTester, NumElements) { ASSERT_EQ(lod_tensor_.NumElements(0), 2UL); - ASSERT_EQ(lod_tensor_.NumElements(1), 4UL); + ASSERT_EQ(lod_tensor_.NumElements(1), 3UL); ASSERT_EQ(lod_tensor_.NumElements(2), 8UL); } TEST_F(LoDTensorTester, NumElements2) { ASSERT_EQ(lod_tensor_.NumElements(0, 0), 2UL); - ASSERT_EQ(lod_tensor_.NumElements(0, 1), 2UL); - ASSERT_EQ(lod_tensor_.NumElements(1, 1), 2UL); + ASSERT_EQ(lod_tensor_.NumElements(0, 1), 1UL); + ASSERT_EQ(lod_tensor_.NumElements(1, 1), 3UL); } TEST_F(LoDTensorTester, ShrinkLevels) { @@ -68,17 +68,16 @@ TEST_F(LoDTensorTester, ShrinkLevels) { LoDTensor new_lod_tensor = lod_tensor_; new_lod_tensor.ShrinkLevels(level, level + 1); ASSERT_EQ(new_lod_tensor.NumLevels(), 1UL); - ASSERT_EQ(new_lod_tensor.NumElements(0), lod_tensor_.NumElements(level)); ASSERT_EQ(new_lod_tensor.data(), lod_tensor_.data()); } // shrink 2 level for (size_t level = 0; level < 2UL; ++level) { LoDTensor new_lod_tensor = lod_tensor_; new_lod_tensor.ShrinkLevels(level, level + 2); + // the lowest level's last element should be the tensor's batch_size. + ASSERT_EQ(new_lod_tensor.lod().back().back(), + lod_tensor_.lod().back().back()); ASSERT_EQ(new_lod_tensor.NumLevels(), 2UL); - ASSERT_EQ(new_lod_tensor.NumElements(0), lod_tensor_.NumElements(level)); - ASSERT_EQ(new_lod_tensor.NumElements(1), - lod_tensor_.NumElements(level + 1)); ASSERT_EQ(new_lod_tensor.data(), lod_tensor_.data()); } } @@ -86,19 +85,19 @@ TEST_F(LoDTensorTester, ShrinkLevels) { TEST_F(LoDTensorTester, ShrinkInLevel) { size_t level = 0; LoDTensor new_lod_tensor = lod_tensor_; - new_lod_tensor.ShrinkInLevel(level, 0, 2); + new_lod_tensor.ShrinkInLevel(level, 0, 1); EXPECT_EQ(new_lod_tensor.NumLevels(), 3UL); - EXPECT_EQ(new_lod_tensor.NumElements(0), 2UL); - EXPECT_EQ(new_lod_tensor.NumElements(1), 4UL); - EXPECT_EQ(new_lod_tensor.NumElements(2), 8UL); + EXPECT_EQ(new_lod_tensor.NumElements(0), 1UL); + EXPECT_EQ(new_lod_tensor.NumElements(1), 2UL); + EXPECT_EQ(new_lod_tensor.NumElements(2), 5UL); ASSERT_EQ(new_lod_tensor.data(), lod_tensor_.data()); level = 1; new_lod_tensor = lod_tensor_; - new_lod_tensor.ShrinkInLevel(level, 0, 2); + new_lod_tensor.ShrinkInLevel(level, 1, 2); ASSERT_EQ(new_lod_tensor.NumLevels(), 2UL); - ASSERT_EQ(new_lod_tensor.NumElements(0), 2UL); - ASSERT_EQ(new_lod_tensor.NumElements(1), 4UL); + ASSERT_EQ(new_lod_tensor.NumElements(0), 1UL); + ASSERT_EQ(new_lod_tensor.NumElements(1), 3UL); ASSERT_EQ(new_lod_tensor.data(), lod_tensor_.data()); } From 7256cbe1de900906f7c3be5d34a75a9ae5176978 Mon Sep 17 00:00:00 2001 From: Kexin Zhao Date: Fri, 20 Oct 2017 14:11:50 -0700 Subject: [PATCH 68/76] add default value to epsilon --- python/paddle/v2/framework/optimizer.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/python/paddle/v2/framework/optimizer.py b/python/paddle/v2/framework/optimizer.py index 51d435668c..ba2713e34d 100644 --- a/python/paddle/v2/framework/optimizer.py +++ b/python/paddle/v2/framework/optimizer.py @@ -279,7 +279,7 @@ class AdagradOptimizer(Optimizer): """ _moment_acc_str = "moment" - def __init__(self, learning_rate, epsilon): + def __init__(self, learning_rate, epsilon=1.0e-6): assert learning_rate is not None assert epsilon is not None super(AdagradOptimizer, self).__init__() From 86437a8dda6e7fc4e7a133136011f1f78908e898 Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Fri, 20 Oct 2017 14:18:14 -0700 Subject: [PATCH 69/76] Global function, op_support_gpu (#4980) --- paddle/framework/operator.cc | 15 +++++++++++++++ paddle/framework/operator.h | 2 ++ paddle/pybind/pybind.cc | 2 ++ .../v2/framework/tests/test_op_support_gpu.py | 11 +++++++++++ 4 files changed, 30 insertions(+) create mode 100644 python/paddle/v2/framework/tests/test_op_support_gpu.py diff --git a/paddle/framework/operator.cc b/paddle/framework/operator.cc index 2fca816f35..a67625fa88 100644 --- a/paddle/framework/operator.cc +++ b/paddle/framework/operator.cc @@ -252,5 +252,20 @@ std::ostream& operator<<(std::ostream& os, return os; } +bool OpSupportGPU(const std::string& op_type) { + auto& all_kernels = OperatorWithKernel::AllOpKernels(); + auto it = all_kernels.find(op_type); + if (it == all_kernels.end()) { + // All control operator must support GPU + return true; + } + for (auto& kern_pair : it->second) { + if (platform::is_gpu_place(kern_pair.first.place_)) { + return true; + } + } + return false; +} + } // namespace framework } // namespace paddle diff --git a/paddle/framework/operator.h b/paddle/framework/operator.h index 12cd307297..9d7fe1f5ba 100644 --- a/paddle/framework/operator.h +++ b/paddle/framework/operator.h @@ -649,5 +649,7 @@ class OperatorWithKernel : public OperatorBase { std::ostream& operator<<(std::ostream& os, const OperatorWithKernel::OpKernelKey& kernel_key); +extern bool OpSupportGPU(const std::string& op_type); + } // namespace framework } // namespace paddle diff --git a/paddle/pybind/pybind.cc b/paddle/pybind/pybind.cc index e5ddc14587..26b793a4bb 100644 --- a/paddle/pybind/pybind.cc +++ b/paddle/pybind/pybind.cc @@ -466,6 +466,8 @@ All parameter, weight, gradient are variables in Paddle. BindVarDsec(m); BindOpDesc(m); + m.def("op_support_gpu", OpSupportGPU); + return m.ptr(); } } // namespace pybind diff --git a/python/paddle/v2/framework/tests/test_op_support_gpu.py b/python/paddle/v2/framework/tests/test_op_support_gpu.py new file mode 100644 index 0000000000..dd36c666c4 --- /dev/null +++ b/python/paddle/v2/framework/tests/test_op_support_gpu.py @@ -0,0 +1,11 @@ +import unittest +import paddle.v2.framework.core as core + + +class TestOpSupportGPU(unittest.TestCase): + def test_case(self): + self.assertEqual(core.is_compile_gpu(), core.op_support_gpu("sum")) + + +if __name__ == '__main__': + unittest.main() From e9e0d7d774d2fa73a7621ee0bfc5f87718115cc0 Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Fri, 20 Oct 2017 14:18:28 -0700 Subject: [PATCH 70/76] Correct the dependencies (#4978) --- paddle/framework/CMakeLists.txt | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/paddle/framework/CMakeLists.txt b/paddle/framework/CMakeLists.txt index 774c7b0217..dbe76a8eaf 100644 --- a/paddle/framework/CMakeLists.txt +++ b/paddle/framework/CMakeLists.txt @@ -19,15 +19,15 @@ cc_test(scope_test SRCS scope_test.cc DEPS scope) proto_library(framework_proto SRCS framework.proto) cc_library(attribute SRCS attribute.cc DEPS framework_proto) -cc_library(proto_desc SRCS var_desc.cc op_desc.cc block_desc.cc program_desc.cc DEPS attribute ddim op_info) cc_test(program_desc_test SRCS program_desc_test.cc DEPS proto_desc) cc_library(op_proto_maker SRCS op_proto_maker.cc DEPS framework_proto attribute) cc_test(op_proto_maker_test SRCS op_proto_maker_test.cc DEPS op_proto_maker) cc_library(op_info SRCS op_info.cc DEPS attribute framework_proto) -cc_library(operator SRCS operator.cc DEPS op_info device_context tensor scope proto_desc glog) +cc_library(operator SRCS operator.cc DEPS op_info device_context tensor scope glog) cc_test(operator_test SRCS operator_test.cc DEPS operator op_registry) +cc_library(proto_desc SRCS var_desc.cc op_desc.cc block_desc.cc program_desc.cc DEPS attribute ddim op_info operator) -cc_library(op_registry SRCS op_registry.cc DEPS op_proto_maker op_info operator glog) +cc_library(op_registry SRCS op_registry.cc DEPS op_proto_maker op_info operator glog proto_desc) cc_test(op_registry_test SRCS op_registry_test.cc DEPS op_registry) py_proto_compile(framework_py_proto SRCS framework.proto) From 784fc32bfa27526eb83b5561225933949abebac2 Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Sat, 21 Oct 2017 06:00:34 +0800 Subject: [PATCH 71/76] Add nccl to docker image --- Dockerfile | 2 +- paddle/scripts/docker/build.sh | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/Dockerfile b/Dockerfile index 136db772cc..150344a811 100644 --- a/Dockerfile +++ b/Dockerfile @@ -22,7 +22,7 @@ COPY ./paddle/scripts/docker/root/ /root/ RUN apt-get update && \ apt-get install -y \ - git python-pip python-dev openssh-server bison \ + git python-pip python-dev openssh-server bison libnccl-dev \ wget unzip unrar tar xz-utils bzip2 gzip coreutils ntp \ curl sed grep graphviz libjpeg-dev zlib1g-dev \ python-matplotlib gcc-4.8 g++-4.8 \ diff --git a/paddle/scripts/docker/build.sh b/paddle/scripts/docker/build.sh index 2ac455d771..a08716c5a5 100644 --- a/paddle/scripts/docker/build.sh +++ b/paddle/scripts/docker/build.sh @@ -141,10 +141,17 @@ RUN sed -i '${APT_MIRROR}' /etc/apt/sources.list EOF fi +if [[ ${WITH_GPU} == "ON" ]]; then + NCCL_DEPS="apt-get install -y libnccl-dev &&" +else + NCCL_DEPS="" +fi + cat >> /paddle/build/Dockerfile < Date: Fri, 20 Oct 2017 22:52:51 +0000 Subject: [PATCH 72/76] add optional in op proto --- paddle/framework/framework.proto | 1 + paddle/framework/op_proto_maker.h | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/paddle/framework/framework.proto b/paddle/framework/framework.proto index 2aa961f140..3d023535ef 100644 --- a/paddle/framework/framework.proto +++ b/paddle/framework/framework.proto @@ -68,6 +68,7 @@ message OpProto { optional bool duplicable = 3 [ default = false ]; optional bool intermediate = 4 [ default = false ]; + optional bool dispensable = 5 [ default = false ]; } // AttrProto describes the C++ type Attribute. diff --git a/paddle/framework/op_proto_maker.h b/paddle/framework/op_proto_maker.h index a134befd90..44e8ab1689 100644 --- a/paddle/framework/op_proto_maker.h +++ b/paddle/framework/op_proto_maker.h @@ -44,6 +44,11 @@ class OpProtoAndCheckerMaker { var_->set_intermediate(true); return *this; } + + VariableBuilder& AsDispensable() { + var_->set_dispensable(true); + return *this; + } }; VariableBuilder AddInput(const std::string& name, const std::string& comment); From e7f627036a7f8fc5b105e30c50e1e99a254cb718 Mon Sep 17 00:00:00 2001 From: Qiao Longfei Date: Sat, 21 Oct 2017 14:53:23 -0700 Subject: [PATCH 73/76] fix InferShapeContext Has interface (#4994) --- paddle/framework/operator.h | 32 ++++++++++++++++++++++++++++---- 1 file changed, 28 insertions(+), 4 deletions(-) diff --git a/paddle/framework/operator.h b/paddle/framework/operator.h index 9d7fe1f5ba..79a452b616 100644 --- a/paddle/framework/operator.h +++ b/paddle/framework/operator.h @@ -327,6 +327,9 @@ class CompileTimeInferShapeContext : public InferShapeContext { bool HasInput(const std::string& name) const override { const std::vector& input_names = op_.Input(name); auto length = input_names.size(); + if (length == 0) { + return false; + } PADDLE_ENFORCE_EQ(length, 1UL, "Input(%s) should have only one value, " "but it have %d now", @@ -337,6 +340,9 @@ class CompileTimeInferShapeContext : public InferShapeContext { bool HasOutput(const std::string& name) const override { const std::vector& output_names = op_.Output(name); auto length = output_names.size(); + if (length == 0) { + return false; + } PADDLE_ENFORCE_EQ(length, 1UL, "Output(%s) should have only one value, " "but it have %d now", @@ -346,7 +352,9 @@ class CompileTimeInferShapeContext : public InferShapeContext { bool HasInputs(const std::string& name) const override { const std::vector& input_names = op_.Input(name); - PADDLE_ENFORCE(!input_names.empty(), "Inputs(%s) length is 0", name); + if (input_names.empty()) { + return false; + } for (auto& input : input_names) { if (!block_.HasVar(input)) return false; } @@ -355,7 +363,9 @@ class CompileTimeInferShapeContext : public InferShapeContext { bool HasOutputs(const std::string& name) const override { const std::vector& output_names = op_.Output(name); - PADDLE_ENFORCE(!output_names.empty(), "Inputs(%s) length is 0", name); + if (output_names.empty()) { + return false; + } for (auto& output : output_names) { if (!block_.HasVar(output)) return false; } @@ -421,13 +431,27 @@ class RuntimeInferShapeContext : public InferShapeContext { : op_(op), scope_(scope) {} bool HasInput(const std::string& name) const override { - auto ipt = op_.Input(name); + auto& ins = Inputs(name); + size_t length = ins.size(); + if (length == 0) { + return false; + } + PADDLE_ENFORCE_EQ(length, 1UL, "Input %s should have more than one inputs", + name); + auto ipt = ins[0]; auto* var = ipt == kEmptyVarName ? nullptr : scope_.FindVar(ipt); return var != nullptr; } bool HasOutput(const std::string& name) const override { - auto ipt = op_.Output(name); + auto& outs = Outputs(name); + size_t length = outs.size(); + if (length == 0) { + return false; + } + PADDLE_ENFORCE_EQ(length, 1UL, "Output %s should have more than one inputs", + name); + auto ipt = outs[0]; auto* var = ipt == kEmptyVarName ? nullptr : scope_.FindVar(ipt); return var != nullptr; } From 54ffafa123d4da3d217c2e80b1db644d74a89206 Mon Sep 17 00:00:00 2001 From: Qiao Longfei Date: Sat, 21 Oct 2017 16:01:49 -0700 Subject: [PATCH 74/76] use context to get attribute (#4997) --- paddle/operators/clip_op.cc | 4 ++-- paddle/operators/gaussian_random_op.cc | 2 +- paddle/operators/uniform_random_op.cc | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/paddle/operators/clip_op.cc b/paddle/operators/clip_op.cc index 2d029394dd..f80204c683 100644 --- a/paddle/operators/clip_op.cc +++ b/paddle/operators/clip_op.cc @@ -27,8 +27,8 @@ class ClipOp : public framework::OperatorWithKernel { PADDLE_ENFORCE(ctx->HasOutput("Out"), "Output(Out) of ClipOp should not be null."); auto x_dims = ctx->GetInputDim("X"); - auto max = Attr("max"); - auto min = Attr("min"); + auto max = ctx->Attrs().Get("max"); + auto min = ctx->Attrs().Get("min"); PADDLE_ENFORCE_LT(min, max, "max should be greater than min."); ctx->SetOutputDim("Out", x_dims); ctx->ShareLoD("X", /*->*/ "Out"); diff --git a/paddle/operators/gaussian_random_op.cc b/paddle/operators/gaussian_random_op.cc index f59f497d9f..04dfdf7c48 100644 --- a/paddle/operators/gaussian_random_op.cc +++ b/paddle/operators/gaussian_random_op.cc @@ -59,7 +59,7 @@ class GaussianRandomOp : public framework::OperatorWithKernel { protected: framework::DataType IndicateDataType( const framework::ExecutionContext& ctx) const override { - return static_cast(Attr("data_type")); + return static_cast(ctx.Attr("data_type")); } }; diff --git a/paddle/operators/uniform_random_op.cc b/paddle/operators/uniform_random_op.cc index f244ddc51f..39b53948e3 100644 --- a/paddle/operators/uniform_random_op.cc +++ b/paddle/operators/uniform_random_op.cc @@ -65,7 +65,7 @@ class UniformRandomOp : public framework::OperatorWithKernel { protected: framework::DataType IndicateDataType( const framework::ExecutionContext& ctx) const override { - return static_cast(Attr("data_type")); + return static_cast(ctx.Attr("data_type")); } }; From c91de280d783d531792e8a458cc50342eb405f59 Mon Sep 17 00:00:00 2001 From: Qiao Longfei Date: Sun, 22 Oct 2017 10:54:42 -0700 Subject: [PATCH 75/76] CompileTime InferShape should find var recursively in stack of blocks (#4998) * recursive find var in BlockDesc * add HasVarRecursive and FindVarRecursive to BlockDesc * fix FindVarRecursive --- paddle/framework/block_desc.cc | 15 ++++++++++++++- paddle/framework/block_desc.h | 5 +++++ paddle/framework/operator.h | 12 ++++++------ paddle/framework/program_desc.cc | 4 ++-- paddle/framework/program_desc.h | 1 + paddle/framework/proto_desc.h | 26 ++++++++++++++++++++++++++ 6 files changed, 54 insertions(+), 9 deletions(-) create mode 100644 paddle/framework/proto_desc.h diff --git a/paddle/framework/block_desc.cc b/paddle/framework/block_desc.cc index 21d4fdaf06..251e340e6d 100644 --- a/paddle/framework/block_desc.cc +++ b/paddle/framework/block_desc.cc @@ -41,6 +41,19 @@ bool BlockDescBind::HasVar(const std::string &name) const { return vars_.find(name) != vars_.end(); } +VarDescBind *BlockDescBind::FindVarRecursive(const std::string &name) const { + auto it = vars_.find(name); + if (it == vars_.end()) { + return Parent() == kNoneBlockIndex ? nullptr + : ParentBlock()->FindVarRecursive(name); + } + return it->second.get(); +} + +bool BlockDescBind::HasVarRecursive(const std::string &name) const { + return FindVarRecursive(name) != nullptr; +} + std::vector BlockDescBind::AllVars() const { std::vector res; for (const auto &p : vars_) { @@ -97,7 +110,7 @@ void BlockDescBind::Flush() { } BlockDescBind *BlockDescBind::ParentBlock() const { - if (this->desc_->parent_idx() == -1) { + if (this->desc_->parent_idx() == kNoneBlockIndex) { return nullptr; } return prog_->Block(static_cast(this->desc_->parent_idx())); diff --git a/paddle/framework/block_desc.h b/paddle/framework/block_desc.h index 7d1d33f686..c685050850 100644 --- a/paddle/framework/block_desc.h +++ b/paddle/framework/block_desc.h @@ -21,6 +21,7 @@ limitations under the License. */ #include #include "paddle/framework/op_desc.h" +#include "paddle/framework/proto_desc.h" #include "paddle/framework/var_desc.h" #include "paddle/platform/macros.h" @@ -56,6 +57,10 @@ class BlockDescBind { bool HasVar(const std::string &var_name) const; + VarDescBind *FindVarRecursive(const std::string &name_bytes) const; + + bool HasVarRecursive(const std::string &var_name) const; + std::set LocalVarNames() const { std::set var_names; for (auto &var : vars_) { diff --git a/paddle/framework/operator.h b/paddle/framework/operator.h index 79a452b616..0d0304ac9e 100644 --- a/paddle/framework/operator.h +++ b/paddle/framework/operator.h @@ -334,7 +334,7 @@ class CompileTimeInferShapeContext : public InferShapeContext { "Input(%s) should have only one value, " "but it have %d now", name, length); - return block_.HasVar(input_names[0]); + return block_.HasVarRecursive(input_names[0]); } bool HasOutput(const std::string& name) const override { @@ -347,7 +347,7 @@ class CompileTimeInferShapeContext : public InferShapeContext { "Output(%s) should have only one value, " "but it have %d now", name, length); - return block_.HasVar(output_names[0]); + return block_.HasVarRecursive(output_names[0]); } bool HasInputs(const std::string& name) const override { @@ -356,7 +356,7 @@ class CompileTimeInferShapeContext : public InferShapeContext { return false; } for (auto& input : input_names) { - if (!block_.HasVar(input)) return false; + if (!block_.HasVarRecursive(input)) return false; } return true; } @@ -367,7 +367,7 @@ class CompileTimeInferShapeContext : public InferShapeContext { return false; } for (auto& output : output_names) { - if (!block_.HasVar(output)) return false; + if (!block_.HasVarRecursive(output)) return false; } return true; } @@ -414,11 +414,11 @@ class CompileTimeInferShapeContext : public InferShapeContext { private: DDim GetDim(const std::string& name) const override { - return framework::make_ddim(block_.FindVar(name)->Shape()); + return framework::make_ddim(block_.FindVarRecursive(name)->Shape()); } void SetDim(const std::string& name, const DDim& dim) override { - block_.FindVar(name)->SetShape(framework::vectorize(dim)); + block_.FindVarRecursive(name)->SetShape(framework::vectorize(dim)); } const OpDescBind& op_; diff --git a/paddle/framework/program_desc.cc b/paddle/framework/program_desc.cc index e2349cefe0..8e99bba811 100644 --- a/paddle/framework/program_desc.cc +++ b/paddle/framework/program_desc.cc @@ -35,8 +35,8 @@ ProgramDesc *ProgramDescBind::Proto() { ProgramDescBind::ProgramDescBind() { auto *block = prog_.mutable_blocks()->Add(); - block->set_idx(0); - block->set_parent_idx(-1); + block->set_idx(kRootBlockIndex); + block->set_parent_idx(kNoneBlockIndex); blocks_.emplace_back(new BlockDescBind(this, block)); } diff --git a/paddle/framework/program_desc.h b/paddle/framework/program_desc.h index 20cc1a2325..dc4cd7cc73 100644 --- a/paddle/framework/program_desc.h +++ b/paddle/framework/program_desc.h @@ -17,6 +17,7 @@ limitations under the License. */ #include #include #include "paddle/framework/framework.pb.h" +#include "paddle/framework/proto_desc.h" #include "paddle/platform/macros.h" namespace paddle { diff --git a/paddle/framework/proto_desc.h b/paddle/framework/proto_desc.h new file mode 100644 index 0000000000..fa01224fef --- /dev/null +++ b/paddle/framework/proto_desc.h @@ -0,0 +1,26 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. */ + +#pragma once + +namespace paddle { +namespace framework { + +// The Index of first Block in Program. also called root block. +constexpr int kRootBlockIndex = 0; +// The Parent Index of root Block, this block does not exist. +constexpr int kNoneBlockIndex = -1; + +} // namespace framework +} // namespace paddle From 70d9d953e60992fc0cf7c1a58936452fb3e76b06 Mon Sep 17 00:00:00 2001 From: Luo Tao Date: Mon, 23 Oct 2017 11:36:01 +0800 Subject: [PATCH 76/76] rename sparse_vector to sparse_float_vector in tests --- paddle/gserver/tests/test_PyDataProvider2.py | 5 ++++- python/paddle/trainer/PyDataProvider2.py | 8 ++++---- python/paddle/v2/tests/test_data_feeder.py | 2 +- 3 files changed, 9 insertions(+), 6 deletions(-) diff --git a/paddle/gserver/tests/test_PyDataProvider2.py b/paddle/gserver/tests/test_PyDataProvider2.py index 2e6225519f..0d0fe476ff 100644 --- a/paddle/gserver/tests/test_PyDataProvider2.py +++ b/paddle/gserver/tests/test_PyDataProvider2.py @@ -51,7 +51,10 @@ def test_sparse_non_value_no_seq(setting, filename): yield [(i + 1) * (j + 1) for j in xrange(10)] -@provider(input_types=[sparse_vector(30000, seq_type=SequenceType.NO_SEQUENCE)]) +@provider(input_types=[ + sparse_float_vector( + 30000, seq_type=SequenceType.NO_SEQUENCE) +]) def test_sparse_value_no_seq(setting, filename): for i in xrange(200): yield [((i + 1) * (j + 1), float(j) / float(i + 1)) for j in xrange(10)] diff --git a/python/paddle/trainer/PyDataProvider2.py b/python/paddle/trainer/PyDataProvider2.py index 045e3c0279..05635833bf 100644 --- a/python/paddle/trainer/PyDataProvider2.py +++ b/python/paddle/trainer/PyDataProvider2.py @@ -216,7 +216,7 @@ def sparse_binary_vector_sub_sequence(dim): return sparse_binary_vector(dim, seq_type=SequenceType.SUB_SEQUENCE) -def sparse_vector_sequence(dim): +def sparse_float_vector_sequence(dim): """ Data type of a sequence of sparse vector, which most elements are zero, others could be any float value. @@ -226,11 +226,11 @@ def sparse_vector_sequence(dim): :return: An input type object :rtype: InputType """ - return sparse_vector(dim, seq_type=SequenceType.SEQUENCE) + return sparse_float_vector(dim, seq_type=SequenceType.SEQUENCE) -def sparse_vector_sub_sequence(dim): - return sparse_vector(dim, seq_type=SequenceType.SUB_SEQUENCE) +def sparse_float_vector_sub_sequence(dim): + return sparse_float_vector(dim, seq_type=SequenceType.SUB_SEQUENCE) def integer_value_sequence(value_range): diff --git a/python/paddle/v2/tests/test_data_feeder.py b/python/paddle/v2/tests/test_data_feeder.py index 83da678da3..63905c04cf 100644 --- a/python/paddle/v2/tests/test_data_feeder.py +++ b/python/paddle/v2/tests/test_data_feeder.py @@ -97,7 +97,7 @@ class DataFeederTest(unittest.TestCase): each_sample.append(zip(a, b)) data.append(each_sample) - feeder = DataFeeder([('input', data_type.sparse_vector(dim))], + feeder = DataFeeder([('input', data_type.sparse_float_vector(dim))], {'input': 0}) arg = feeder(data) output = arg.getSlotValue(0)