From daea7b2956d181d9ae53922183a6c3f4aa3771a8 Mon Sep 17 00:00:00 2001 From: dangqingqing Date: Fri, 9 Dec 2016 15:47:58 +0800 Subject: [PATCH 01/88] fix dead link for quick start --- doc/demo/quick_start/index_en.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/demo/quick_start/index_en.md b/doc/demo/quick_start/index_en.md index 659485d9be..6e2056b69f 100644 --- a/doc/demo/quick_start/index_en.md +++ b/doc/demo/quick_start/index_en.md @@ -164,7 +164,7 @@ You will describe four kinds of network architectures in this section.
![](./PipelineNetwork_en.jpg)
First, you will build a logistic regression model. Later, you will also get chance to build other more powerful network architectures. -For more detailed documentation, you could refer to: Layer documentation。All configuration files are in `demo/quick_start` directory. +For more detailed documentation, you could refer to: Layer documentation。All configuration files are in `demo/quick_start` directory. ### Logistic Regression The architecture is illustrated in the following picture: From af7f407ecf468c0b28d5969e2a26d01511f09af2 Mon Sep 17 00:00:00 2001 From: dangqingqing Date: Mon, 12 Dec 2016 11:58:56 +0800 Subject: [PATCH 02/88] update --- doc/demo/quick_start/index_en.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/demo/quick_start/index_en.md b/doc/demo/quick_start/index_en.md index 6e2056b69f..c2d6f3a6b6 100644 --- a/doc/demo/quick_start/index_en.md +++ b/doc/demo/quick_start/index_en.md @@ -164,7 +164,7 @@ You will describe four kinds of network architectures in this section.
![](./PipelineNetwork_en.jpg)
First, you will build a logistic regression model. Later, you will also get chance to build other more powerful network architectures. -For more detailed documentation, you could refer to: Layer documentation。All configuration files are in `demo/quick_start` directory. +For more detailed documentation, you could refer to: layer documentation. All configuration files are in `demo/quick_start` directory. ### Logistic Regression The architecture is illustrated in the following picture: From 88b7ed8f16f0f76a5d1b5256b9af9de7bbd16bc1 Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Wed, 14 Dec 2016 16:21:56 +0800 Subject: [PATCH 03/88] Fix Travis-CI build for release --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index ffe3bc193b..a30857b600 100644 --- a/.travis.yml +++ b/.travis.yml @@ -50,7 +50,7 @@ before_install: fi - if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then sudo paddle/scripts/travis/before_install.linux.sh; fi - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then paddle/scripts/travis/before_install.osx.sh; fi - - pip install wheel protobuf sphinx breathe recommonmark virtualenv numpy + - pip install wheel protobuf 'sphinx==1.4.9' breathe recommonmark virtualenv numpy script: - paddle/scripts/travis/main.sh notifications: From 576176d4b232cb4cf17b0ecddefc21cafbbeb203 Mon Sep 17 00:00:00 2001 From: Yu Yang Date: Wed, 14 Dec 2016 16:24:45 +0800 Subject: [PATCH 04/88] Remove typo in documentation. --- doc/ui/data_provider/pydataprovider2.rst | 8 ++++---- doc/ui/index.md | 2 +- doc_cn/ui/data_provider/pydataprovider2.rst | 8 ++++---- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/doc/ui/data_provider/pydataprovider2.rst b/doc/ui/data_provider/pydataprovider2.rst index e105d3be30..fcf230522f 100644 --- a/doc/ui/data_provider/pydataprovider2.rst +++ b/doc/ui/data_provider/pydataprovider2.rst @@ -174,12 +174,12 @@ input_types +++++++++++ PaddlePaddle has four data types, and three sequence types. -The four data types are: +The four data types are: * :code:`dense_vector`: dense float vector. * :code:`sparse_binary_vector`: sparse binary vector, most of the value is 0, and the non zero elements are fixed to 1. -* :code:`sparse_float_vector`: sparse float vector, most of the value is 0, and some +* :code:`sparse_vector`: sparse float vector, most of the value is 0, and some non zero elements can be any float value. They are given by the user. * :code:`integer`: an integer scalar, that is especially used for label or word index. @@ -200,7 +200,7 @@ in the above table. +----------------------+---------------------+-----------------------------------+------------------------------------------------+ | sparse_binary_vector | [i, i, ...] | [[i, ...], [i, ...], ...] | [[[i, ...], ...], [[i, ...], ...],...] | +----------------------+---------------------+-----------------------------------+------------------------------------------------+ -| sparse_float_vector | [(i,f), (i,f), ...] | [[(i,f), ...], [(i,f), ...], ...] | [[[(i,f), ...], ...], [[(i,f), ...], ...],...] | +| sparse_vector | [(i,f), (i,f), ...] | [[(i,f), ...], [(i,f), ...], ...] | [[[(i,f), ...], ...], [[(i,f), ...], ...],...] | +----------------------+---------------------+-----------------------------------+------------------------------------------------+ | integer_value | i | [i, i, ...] | [[i, ...], [i, ...], ...] | +----------------------+---------------------+-----------------------------------+------------------------------------------------+ @@ -227,7 +227,7 @@ Its parameters lists as follows: * :code:`is_train` is a bool parameter that indicates the DataProvider is used in training or testing. * :code:`file_list` is the list of all files. - + * User-defined parameters args can be set in training configuration. Note, PaddlePaddle reserves the right to add pre-defined parameter, so please diff --git a/doc/ui/index.md b/doc/ui/index.md index 9c1ba27bdc..5aa051b4e9 100644 --- a/doc/ui/index.md +++ b/doc/ui/index.md @@ -7,7 +7,7 @@ ## API Reference -* [Model Config Interface](api/trainer_config_helpers/index.md) +* [Model Config Interface](api/trainer_config_helpers/index.rst) ## Command Line Argument diff --git a/doc_cn/ui/data_provider/pydataprovider2.rst b/doc_cn/ui/data_provider/pydataprovider2.rst index 80b40084d8..34db4a97d9 100644 --- a/doc_cn/ui/data_provider/pydataprovider2.rst +++ b/doc_cn/ui/data_provider/pydataprovider2.rst @@ -53,7 +53,7 @@ process函数调用多次 :code:`yield` 即可。 :code:`yield` 是Python的一 .. literalinclude:: mnist_config.py -这里说明了训练数据是 'train.list'，而没有测试数据。引用的DataProvider是 'mnist_provider' +这里说明了训练数据是 'train.list'，而没有测试数据。引用的DataProvider是 'mnist_provider' 这个模块中的 'process' 函数。 同时，根据模型配置文件中 :code:`data_layer` 的名字，用户也可以显式指定返回的数据对应关系。例如: @@ -152,7 +152,7 @@ PaddlePaddle的数据包括四种主要类型，和三种序列模式。其中 * dense_vector 表示稠密的浮点数向量。 * sparse_binary_vector 表示稀疏的零一向量，即大部分值为0，有值的位置只能取1 -* sparse_float_vector 表示稀疏的向量，即大部分值为0，有值的部分可以是任何浮点数 +* sparse_vector 表示稀疏的向量，即大部分值为0，有值的部分可以是任何浮点数 * integer 表示整数标签。 而三种序列模式为 @@ -170,7 +170,7 @@ PaddlePaddle的数据包括四种主要类型，和三种序列模式。其中 +----------------------+---------------------+-----------------------------------+------------------------------------------------+ | sparse_binary_vector | [i, i, ...] | [[i, ...], [i, ...], ...] | [[[i, ...], ...], [[i, ...], ...],...] | +----------------------+---------------------+-----------------------------------+------------------------------------------------+ -| sparse_float_vector | [(i,f), (i,f), ...] | [[(i,f), ...], [(i,f), ...], ...] | [[[(i,f), ...], ...], [[(i,f), ...], ...],...] | +| sparse_vector | [(i,f), (i,f), ...] | [[(i,f), ...], [(i,f), ...], ...] | [[[(i,f), ...], ...], [[(i,f), ...], ...],...] | +----------------------+---------------------+-----------------------------------+------------------------------------------------+ | integer_value | i | [i, i, ...] | [[i, ...], [i, ...], ...] | +----------------------+---------------------+-----------------------------------+------------------------------------------------+ @@ -202,7 +202,7 @@ DataProvider提供了两种简单的Cache策略。他们是 * CacheType.NO_CACHE 不缓存任何数据，每次都会从python端读取数据 * CacheType.CACHE_PASS_IN_MEM 第一个pass会从python端读取数据，剩下的pass会直接从内存里 - 读取数据。 + 读取数据。 注意事项 From d1d50412209836624d6220ef9825f417af75cac4 Mon Sep 17 00:00:00 2001 From: Luo Tao Date: Mon, 19 Dec 2016 16:32:45 +0800 Subject: [PATCH 05/88] fix typo --- doc_cn/build_and_install/install/docker_install.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc_cn/build_and_install/install/docker_install.rst b/doc_cn/build_and_install/install/docker_install.rst index a5f5fb117e..8298b79bf7 100644 --- a/doc_cn/build_and_install/install/docker_install.rst +++ b/doc_cn/build_and_install/install/docker_install.rst @@ -67,7 +67,7 @@ mac osx或者是windows机器，请参考 .. code-block:: bash - $ docker run -it paddledev/paddlepaddle:cpu-latest + $ docker run -it paddledev/paddle:cpu-latest 即可启动和进入PaddlePaddle的container。如果运行GPU版本的PaddlePaddle，则需要先将 cuda相关的Driver和设备映射进container中，脚本类似于 From fd5c77002bf8c7ad3011271a65c2f9609454e549 Mon Sep 17 00:00:00 2001 From: zhuoyuan Date: Tue, 7 Feb 2017 15:55:46 -0800 Subject: [PATCH 06/88] light version of cnn for MNIST --- demo/mnist/light_mnist.py | 71 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100644 demo/mnist/light_mnist.py diff --git a/demo/mnist/light_mnist.py b/demo/mnist/light_mnist.py new file mode 100644 index 0000000000..271f73cd4d --- /dev/null +++ b/demo/mnist/light_mnist.py @@ -0,0 +1,71 @@ +from paddle.trainer_config_helpers import * + +is_predict = get_config_arg("is_predict", bool, False) + +####################Data Configuration ################## + +if not is_predict: + data_dir = './data/' + define_py_data_sources2( + train_list=data_dir + 'train.list', + test_list=data_dir + 'test.list', + module='mnist_provider', + obj='process') + +######################Algorithm Configuration ############# +# settings( +# batch_size=128, +# learning_rate=0.1 / 128.0, +# learning_method=MomentumOptimizer(0.9), +# regularization=L2Regularization(0.0005 * 128)) +settings( + batch_size=50, + learning_rate=0.001, + learning_method=AdamOptimizer()) + +#######################Network Configuration ############# + +data_size = 1 * 28 * 28 +label_size = 10 +img = data_layer(name='pixel', size=data_size) + +# small_vgg is predined in trainer_config_helpers.network +# predict = small_vgg(input_image=img, num_channels=1, num_classes=label_size) + +# light cnn +def light_cnn(input_image, num_channels, num_classes): + def __light__(ipt, num_filter=128, times=1, conv_filter_size=3, dropouts=0, num_channels_=None): + return img_conv_group( + input=ipt, + num_channels=num_channels_, + pool_size=2, + pool_stride=2, + conv_padding=0, + conv_num_filter=[num_filter] * times, + conv_filter_size=conv_filter_size, + conv_act=ReluActivation(), + conv_with_batchnorm=True, + conv_batchnorm_drop_rate=dropouts, + pool_type=MaxPooling()) + + tmp = __light__(input_image, num_filter=128, num_channels_=num_channels) + tmp = __light__(tmp, num_filter=128) + tmp = __light__(tmp, num_filter=128) + tmp = __light__(tmp, num_filter=128, conv_filter_size=1) + + #tmp = img_pool_layer(input=tmp, stride=2, pool_size=2, pool_type=MaxPooling()) + #tmp = dropout_layer(input=tmp, dropout_rate=0.5) + tmp = fc_layer(input=tmp, size = num_classes, act=SoftmaxActivation()) + # tmp = fc_layer(input=tmp, size=512, layer_attr=ExtraAttr(drop_rate=0.5), act=LinearActivation()) + # tmp = batch_norm_layer(input=tmp, act=ReluActivation()) + # return fc_layer(input=tmp, size=num_classes, act=SoftmaxActivation()) + return tmp + +predict = light_cnn(input_image=img, num_channels=1, num_classes=label_size) + +if not is_predict: + lbl = data_layer(name="label", size=label_size) + inputs(img, lbl) + outputs(classification_cost(input=predict, label=lbl)) +else: + outputs(predict) From bc8ba1de750637773052892c5c008f30cd998844 Mon Sep 17 00:00:00 2001 From: zhuoyuan Date: Tue, 7 Feb 2017 16:02:35 -0800 Subject: [PATCH 07/88] modify light mnist file --- demo/mnist/light_mnist.py | 5 ----- 1 file changed, 5 deletions(-) diff --git a/demo/mnist/light_mnist.py b/demo/mnist/light_mnist.py index 271f73cd4d..466cc2e2f5 100644 --- a/demo/mnist/light_mnist.py +++ b/demo/mnist/light_mnist.py @@ -53,12 +53,7 @@ def light_cnn(input_image, num_channels, num_classes): tmp = __light__(tmp, num_filter=128) tmp = __light__(tmp, num_filter=128, conv_filter_size=1) - #tmp = img_pool_layer(input=tmp, stride=2, pool_size=2, pool_type=MaxPooling()) - #tmp = dropout_layer(input=tmp, dropout_rate=0.5) tmp = fc_layer(input=tmp, size = num_classes, act=SoftmaxActivation()) - # tmp = fc_layer(input=tmp, size=512, layer_attr=ExtraAttr(drop_rate=0.5), act=LinearActivation()) - # tmp = batch_norm_layer(input=tmp, act=ReluActivation()) - # return fc_layer(input=tmp, size=num_classes, act=SoftmaxActivation()) return tmp predict = light_cnn(input_image=img, num_channels=1, num_classes=label_size) From cf437a89920e66d9193b5386c8223d8089d4f6c2 Mon Sep 17 00:00:00 2001 From: zhuoyuan Date: Tue, 7 Feb 2017 16:11:05 -0800 Subject: [PATCH 08/88] modified cnn of mnist light --- demo/mnist/light_mnist.py | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/demo/mnist/light_mnist.py b/demo/mnist/light_mnist.py index 466cc2e2f5..4e70159981 100644 --- a/demo/mnist/light_mnist.py +++ b/demo/mnist/light_mnist.py @@ -18,10 +18,7 @@ if not is_predict: # learning_rate=0.1 / 128.0, # learning_method=MomentumOptimizer(0.9), # regularization=L2Regularization(0.0005 * 128)) -settings( - batch_size=50, - learning_rate=0.001, - learning_method=AdamOptimizer()) +settings(batch_size=50, learning_rate=0.001, learning_method=AdamOptimizer()) #######################Network Configuration ############# @@ -32,9 +29,15 @@ img = data_layer(name='pixel', size=data_size) # small_vgg is predined in trainer_config_helpers.network # predict = small_vgg(input_image=img, num_channels=1, num_classes=label_size) + # light cnn def light_cnn(input_image, num_channels, num_classes): - def __light__(ipt, num_filter=128, times=1, conv_filter_size=3, dropouts=0, num_channels_=None): + def __light__(ipt, + num_filter=128, + times=1, + conv_filter_size=3, + dropouts=0, + num_channels_=None): return img_conv_group( input=ipt, num_channels=num_channels_, @@ -53,9 +56,10 @@ def light_cnn(input_image, num_channels, num_classes): tmp = __light__(tmp, num_filter=128) tmp = __light__(tmp, num_filter=128, conv_filter_size=1) - tmp = fc_layer(input=tmp, size = num_classes, act=SoftmaxActivation()) + tmp = fc_layer(input=tmp, size=num_classes, act=SoftmaxActivation()) return tmp + predict = light_cnn(input_image=img, num_channels=1, num_classes=label_size) if not is_predict: From e42c3854fa6c6683d52c19d65ca2f5e3833996c4 Mon Sep 17 00:00:00 2001 From: zhuoyuan Date: Wed, 8 Feb 2017 15:49:19 -0800 Subject: [PATCH 09/88] follow helin's comments --- demo/mnist/light_mnist.py | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/demo/mnist/light_mnist.py b/demo/mnist/light_mnist.py index 4e70159981..d796a3cc06 100644 --- a/demo/mnist/light_mnist.py +++ b/demo/mnist/light_mnist.py @@ -1,3 +1,17 @@ +# Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserved +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + from paddle.trainer_config_helpers import * is_predict = get_config_arg("is_predict", bool, False) @@ -13,11 +27,6 @@ if not is_predict: obj='process') ######################Algorithm Configuration ############# -# settings( -# batch_size=128, -# learning_rate=0.1 / 128.0, -# learning_method=MomentumOptimizer(0.9), -# regularization=L2Regularization(0.0005 * 128)) settings(batch_size=50, learning_rate=0.001, learning_method=AdamOptimizer()) #######################Network Configuration ############# @@ -26,11 +35,10 @@ data_size = 1 * 28 * 28 label_size = 10 img = data_layer(name='pixel', size=data_size) -# small_vgg is predined in trainer_config_helpers.network -# predict = small_vgg(input_image=img, num_channels=1, num_classes=label_size) - - # light cnn +# A shallower cnn model: [CNN, BN, ReLU, Max-Pooling] x4 + FC x1 +# Easier to train for mnist dataset and quite efficient +# Final performance is close to deeper ones on tasks such as digital and character classification def light_cnn(input_image, num_channels, num_classes): def __light__(ipt, num_filter=128, From e43f66cd1c2187ca397a903b0dbfe06729f920c7 Mon Sep 17 00:00:00 2001 From: zhuoyuan Date: Wed, 8 Feb 2017 15:54:39 -0800 Subject: [PATCH 10/88] after pre-commit --- demo/mnist/light_mnist.py | 1 + 1 file changed, 1 insertion(+) diff --git a/demo/mnist/light_mnist.py b/demo/mnist/light_mnist.py index d796a3cc06..3340905435 100644 --- a/demo/mnist/light_mnist.py +++ b/demo/mnist/light_mnist.py @@ -35,6 +35,7 @@ data_size = 1 * 28 * 28 label_size = 10 img = data_layer(name='pixel', size=data_size) + # light cnn # A shallower cnn model: [CNN, BN, ReLU, Max-Pooling] x4 + FC x1 # Easier to train for mnist dataset and quite efficient From 9572e11d73ede8040d6495f8e588b61fb9ea20ed Mon Sep 17 00:00:00 2001 From: Helin Wang Date: Mon, 1 May 2017 18:27:28 -0700 Subject: [PATCH 11/88] Design doc: master process --- doc/design/cluster_train/data_dispatch.md | 6 +- doc/design/cluster_train/master_process.md | 89 +++++++++++++++++++ doc/design/cluster_train/src/dataset.graffle | Bin 0 -> 2770 bytes doc/design/cluster_train/src/dataset.png | Bin 0 -> 10845 bytes 4 files changed, 92 insertions(+), 3 deletions(-) create mode 100644 doc/design/cluster_train/master_process.md create mode 100644 doc/design/cluster_train/src/dataset.graffle create mode 100644 doc/design/cluster_train/src/dataset.png diff --git a/doc/design/cluster_train/data_dispatch.md b/doc/design/cluster_train/data_dispatch.md index a3eb4e28db..f60c3b843d 100644 --- a/doc/design/cluster_train/data_dispatch.md +++ b/doc/design/cluster_train/data_dispatch.md @@ -21,7 +21,7 @@ ### 文件预处理 -在数据集可以被训练之前，文件需要预先被转换成PaddlePaddle集群内部的存储格式（SSTable）。我们提供两个转换方式： +在数据集可以被训练之前，文件需要预先被转换成PaddlePaddle集群内部的存储格式（RecordIO）。我们提供两个转换方式： - 提供给用户本地转换的库，用户可以编写程序完成转换。 - 用户可以上传自己的数据集，在集群运行MapReduce job完成转换。 @@ -92,11 +92,11 @@ random_images-00099-of-00099 #### 进行训练 -PaddlePaddle提供专用的[data reader creator](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/reader/README.md#python-data-reader-design-doc)，生成给定SSTable文件对应的data reader。**无论在本地还是在云端，reader的使用方式都是一致的**： +PaddlePaddle提供专用的[data reader creator](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/reader/README.md#python-data-reader-design-doc)，生成给定RecordIO文件对应的data reader。**无论在本地还是在云端，reader的使用方式都是一致的**： ```python # ... -reader = paddle.reader.creator.SSTable("/home/random_images-*-of-*") +reader = paddle.reader.creator.RecordIO("/home/random_images-*-of-*") batch_reader = paddle.batch(paddle.dataset.mnist.train(), 128) trainer.train(batch_reader, ...) ``` diff --git a/doc/design/cluster_train/master_process.md b/doc/design/cluster_train/master_process.md new file mode 100644 index 0000000000..949811b4f7 --- /dev/null +++ b/doc/design/cluster_train/master_process.md @@ -0,0 +1,89 @@ +# Design Doc: Master Process + +For an overview of master process' role, please refer to [distributed training design doc](./README.md). In this design doc we will discuss the master process in more details. The master will be implemented in [golang](https://golang.org/). + +## Dataset + + + +A dataset is represented by a list of files in *RecordIO* format on the distributed filesystem, each RecordIO file consists of multiple *blocks*, and each block has multiple data instances. + +## Task Queue + +As mentioned in [distributed training design doc](./README.md), a *task* is a data shard that the master process assigns to the trainer process to train on. A task consists of one or multiple *blocks* from one or multiple files. The master process maintains *task queues* to track the training progress. + +### Task Queue Creation + +1. Each trainer will make an RPC call (using [golang rpc](https://golang.org/pkg/net/rpc/)) to the master process, telling it the RecordIO files representing the dataset specified by the user. Since every trainer will tell the master process the same dataset, only the first RPC call will be honored. + + The RPC interface is: + ```go + func (m *RPCServer) ReportDataset(Paths []string, dummy *int) error { + } + ``` +1. The master process will scan through each RecordIO file to generate the *block index* and know how many blocks does each file have. A block can be referenced by the file path and the index of the block within the file. The block index is in memory data structure that enables fast access to each block, and the index of the block with the file is an integer start from 0, representing the n-th block within the file. + + The definition of the block is: + ```go + type Block struct { + Idx int // index of the block within the file + Path string + Index recordio.Index // block index + } + ``` +1. Blocks are grouped into tasks, and tasks are filled into the todo queue. The pending queue and the done queue are initialized with no element. + + The definition of the task is: + ```go + type Task struct { + Index int + Blocks []Block + } + ``` + + The elements in the tasks queues is of type `TaskEntry`, containing a timeout counter (described in [task retry logic](#task-retry-logic)), and a task: + ```go + type TaskEntry struct { + NumTimeout int + Task Task + } + ``` + + The definition of task queues is: + ```go + type TaskQueues struct { + Todo []TaskEntry + Pending map[int]TaskEntry // map from task index to task entry + Done []TaskEntry + } + ``` + +### Task Queue Persistence + +The task queues need to be persisted on [etcd](https://github.com/coreos/etcd) for fault recovery. Since the task queues only change once a task is completed or timed out, which is not very frequent, we can afford to synchronize with etcd every time the task queues change. + +We will serialize the task queues data structure with [gob encoding](https://golang.org/pkg/encoding/gob/), compress with gzip, and save into etcd synchronously under key `/task_queues`. + +### Task Dispatch + +The trainer will make an RPC call to master to get a new task when: + +- the trainer first started, or +- the trainer finishes a task. + +The RPC interface is: +```go +func (m *RPCServer) GetTask(finished *Task, result *Task) error { +} +``` +Argument `finished` will be `nil` when the trainer is just started. + +During the RPC call the master will do the following: + +- Make a copy of the task queues, and update the copy reflecting the finished tasks and the new pending tasks. +- Synchronize the copy of task queues with etcd using a transaction conditioned on holding the master lock. +- Replace the task queues with the copy and report to the trainer with the new tasks if succeeded, or discard the copy and report the error to the trainer if failed. + +### Task Retry Logic + +When a task is dispatched to the trainer, the master will schedule a function for execution after the timeout duration (based on the moving average of task completion time). If the task entry in still in the pending queue, its timeout counter will increase by one, and the task will be moved to todo queue. If the timeout counter is above the threshold, the master will log the error and discard the task. diff --git a/doc/design/cluster_train/src/dataset.graffle b/doc/design/cluster_train/src/dataset.graffle new file mode 100644 index 0000000000000000000000000000000000000000..c10a423ed16a23229a9ee33d11bfc82bb59646c8 GIT binary patch literal 2770 zcmV;@3N7^?iwFP!000030PS3BbK1xf{T%-at-r1fk^lo)$6F)bj*ZRB1d6I#gESx+ zJ%}WXNy`8I_UHk6fyvG0+NqVRj2?4(rl&Q1x;4gq`S-J?GEc;`6hr@X%4OJ7hUlUp zDf;}=>7ZXfFP;8GmOJdra8|dsD1TvB>ox(6~?;&;IuB3?e z)yap;1=(C7(N>)JZR4sbLDbAfVj;1J-ZsL zYy(M;E6a|l%N1h!3Rqmw@pTEtUdoEN74i(>!6?G=+qOD=|9;M~Jb%WVm-yn#8S_2I zUR<0pe7?lJyo4q}1uH7H;(hdFyd=BPBQmY(PQX{456;4c_3VvGc1T}$w-LHD7Q5C! zv)f)Z@p`A%PP6u1P#v)I)+GMf?!>jFMaG7q9kEO*SbxISovF-~-2(BA%APT7(0T!h zp7M>v@a`j^Paq~_s4GoJsxsH(iy^L*gYGS{2wiOPCc)}e?&31*ocK_W%QSu0<{Sf!t4?))rS#66B zyKiJhE@Y;tY7h~IY!~ouIVGW7*fb3BmN=U!l*2!z?V1lt&wonV{3&S*e3L|b@QiPn z^J(Ns1UW0=%IQ(YlCgfxuu+EGPmw==w-o0?T+TVZ zz)wU&G%#(-vw8QzT20|PUt~|dPiBU0+f#KiV<$6Rfuks6(;_y@0Mk4X8^U#)tjJ$4 zfEDQq0`OlGlSq7_FqxZ#==5Z&t_U0fHCbX)n%K{U3l=^(j&}|hC3@ibymKhA^pMLr zkX+e;2%Y_CkLPQHrthONwHo9Htv0#mXlC(bOa9M4R| z8OsQ&AueK`8!&n3{C{ByzF2qzmdK|pk+MX}62A{i@VPf&i9*T}DNCd*@%~wYEuJx4 znR6(jP|6hO75o2UiHmp45+yeLlm8)>I8NI9Dd~R)ON4ev!s~~PE}xWFcD(W`snBhg zL!7cR=Di9Fjc&LkdU5nA1p3%*eiqS zdiSZ6&k}F4EyI;lmPlD5Wr^R1CEjFPhD})_Wr>s}-akv^a_&~l8*R*R>BdaDF_Ui0 z{L43HxZ*qCnBgzVZ})wf9G50jX)={2Q|~^R@>$|dHfD0^D_6=ADNFoDEb%5AGwJqi z$`Y@!M7LkBMl`}U_Adlvf0LTm>S)NZkLn)K*9Mynd)m8|6ow$2Fv>*B%6Q zhCLa^t=0`R6B~F-Odw5nJeY__8(4 zwGbU_w7!Eu#u`G6h6h;~y`d@#cLPq4MXj#G?;6+JEEztl{yPyOuqO*pR_gsx^ig$f#TGutFcrD|OVu zJ%6BUJQNl%VXeH9TJB8zYu|Lj#X#%9?FZ{yTWd?g93dpE;iH=MAlbznjXQ#cnj?B- zQDcEdYe7b}y#I{gHixLHsIJwnEu^g*Qv^rZz>VEn3oMQIA7f|(>)vLao9ufEJNz#V zJM0_o^Oe$lzI31OAK&MDmw~F=zz`H0Uq1kiBU^B{ioqcad{?P{lfi%<)K1*q&nAmlbc<;IjRu-;6z*B+#~?2WHm zo%--q#yU$KRrkf2`Xx=ku-;6vx9I)ctnjF>ch2fy5;~;Rnc(N(bknI z8q;7^X~@RtQyQrL>5XIl9etzk$c^J@U<`!Cd=3I(fzLQwHU;`J2Dc(%@zCb;{*bs| z?XcE~u`v4o$uYu?1^*aX@1yFFmBV%lyF^rnC_(RkjtS%(pyorw9A#ujAV|xH_?TiG zfgKx5wGrgvBLBepj4TwwRO|=@`S=hKQwDZN*h9lf{Obt#Bz%Z{ecE*=oxr{ONcY5l z0XamuL9TTKqJ;Y@)4s~HL6%M8*py+9WY{X05oY$Xr})4@g5Bemfr160S`iIXky`r9 zNG|$7TO-u~yyTVJrK%3{7C#y$g@@De2ovKtgR&Ve&YRN&TpTq&3S}OJiX_?l^6Nm1 z3C%iYB;WL*|6>goi)gGAiC8!A%15MVQGyU_2hb0SrA*_c4@R|K6yLnZb7IAc7Vxo} z1uYGL6$p_wF^?z_@tKfi-v}3Am|@i-Ss&q4OR-)pe}KS2!QDfFdE1ePp~gAT+{RkM zGjqmg5?0kcTn0VGHdn#k9u%@pYTf5LFpjaZ_d5^L6)jL4V^2hOprT@m#;u}P19c;O zM=M1<&B68vkcd-%BLZhLYy_`5YS#3*qQ`ZS3_GTbPHS@90Q!dLb6buLz&M7?! z@LU(x5m@TD`yH*_6Aarnv|DTjis=Nu@8NenQw?l~iNQP1vtgRwv6eeR(=e59hK^Ox zThSv6H9nANNR-qQ20EDjA{jEn0yiCAtB_v8HDrq1Se`Z?R0Qo6NG<#P_17^ZuBbJv zq25=()Sp$z!Var{LuU5Rn1`~^q5kjTHOLH>CQ9nwa@rGm|9-*oXH2oozq~RET?bu4 z3wDu&#rZoth%Qb1e&v(s8_QqOeVBTk2XUtp>co4pfG)w(ovvCYY=F?cJ41%)J13LR0)&^H7n3+0X938xm&ZiTtOJ zr$<@9vwoM%L;SV11bfDu=iI;6Y)iYv4G}Cgx97c#LLM-@-&%=$5d2Eyeh>MO(0kIY YAREP)V|R4QE?%Ad7qtPw^$k)00GL^19RL6T literal 0 HcmV?d00001 diff --git a/doc/design/cluster_train/src/dataset.png b/doc/design/cluster_train/src/dataset.png new file mode 100644 index 0000000000000000000000000000000000000000..2fb7f1cce3b6dd21489392557826e95a9f207c34 GIT binary patch literal 10845 zcmeI21yt03_vZ%$1q6hlq`QS7MUa&4Zb@mRQ=~&e5u`(g2FU>dK@boTk?xR?knWad zKZC!0p8xLIXV3FL=f7w7?D=_)Iy2l)-S7R}`?~KTQcXn`2a5~~0)gPj%SmZKAgGk! z^#KMd__t#8#t`@q#Z5!@5v07AViml>be7Y1gFtQ*ApfF3(%#+$1|zmwdhUA4Nd~NwZeD=?E z{@AP8IJr9kFSy!1k#}^rbOnRm&5*kj<@sy$zqa_F=TddGwFIZW9?kvN=>PcHU-rT< zd!#X$XXcR$fX%%Nu1g<5>mnkCS#c(la5?VEpxQ z#_|kq&#(xQ+*npiQaUq**tEM6q;V1a7Q-Bpok@7>G)Zbnl~QjoE4SnH9&McUZa;MM zA9Ct@ZnD()d(+=z=JHd(Cy&#E)uCly_%hrnbN{XXb}d{i03I8=!y$?n2tlKP1f^ig zsTpfB#u59Omq%mKKu|EmX;#qk*jIyl11}{ZsCa>axoG`FvMw5=Nb}DG;<~8Ys1W4f zSr!!cHubk!H<4p@(E1ZUH$WiB86GG=j9dtr6D7cK3}F;^#vM)w1VbEx`WhNh%I?&B z@!^OjUG+7EVAJo$?Zd6g%D$%PNa8U3s`xz==@b-< zPtHxy7cvGscEj(gtozCzf7B>+oN2lk%Dk+$8OR~!wetNHE%5C%Wl8t1_L#AE>c3`N zLSB-bW?4}O)}qL!vm`>? zR|i?YXIFpW2sj`JoieS5Mpkh`Szhz_m&HRVvY zmNl4x>1rTX7Fwn%cA*tMUGJeev9NB)pQoC6Ip6kDLG|qeb%FOPZ?0n&1%cmBbM@s| zK?)}!XQ4O4(B4GA=_en0wfqJm;9@1Ey}eba`>s8E}4FiOO4L)x%@!r&#NM zbv9$&lU8HY;0eqwH>&!&7mGJ##ZCv7GXorPn=dw783%nj`GLdf=clL)hV4Qr={di)<-Sa@tLCc_%M;AV1u2EBjws?H zzCN{cdUIx?ZiG?8hc6rxzw=QGRWf+m2s@mn>kq{)cVnV)Mx!jii~9D^@rf!@s$7|Fnah$H__gK^N5P#To zP{wuE`205^|H<5)0Po*hMm@rC;b%F%$`5d3g%bU0 zbKSOc`36Wy5L7C+7UM$u)_$Ll={t+ zI~DI}&E5$(4#B75aF)N{nZ#jVtYLn3v>Da9u0~5>8;K94v};c`)*EPGE$|-!&f8;D zkD8Y=Qo39nD(p&N%_SW?+irGpJ$`Y0IaKlc`!lj`%F`Z{g%J8N+(>I#$(N(&N|Ti1 zRhlr4u^XLCNV8iJkzKSOP};v6KbmbNn09ydf@n4Vf-Rq@WX#yYK@+!Y84sji2%4~> zI9sfb(e6`(&D}rQi48R_k>s1Mv37bL?D~`AG)Vot{r-SA^4M_tz;f?i20jdA@3jsyw z>P&Ht*Nu{xqG2MVtKZF6)5T#@X|%B^DJeNkvo~44Earak4vu-XXsoHZcC&!7G~}Z- zz8&&O;7OsbDBzW;WYU5Y*U5si6vM%ftZEo^pe#k`6!swj!mg7|> z(L_&Iavy(m{c)Be^4xQg&vEL3V$!`M-$C0H$c;y2_qVzlmh#Hwq}PUv;?Mo5{g2J| zlw1e7Piy+^F>%Z2?l## z_6#N$&GB^b%=o4EO6Tscc7$;w*=Rz(V)E;|+&Q2!Q2XvkY)yR($0q#(3l61KcoP15 zYx<$`Pie!UxKHzF5yvPUMW`MGw&t|pjD)CBsnO`_&IP^v$gH#Q)Umw z)6s>Vblor_b4*SADwyD@56h|->Ru>7m9R~MDn#~4V@S0a8Hla{wLSIJemZYqa-#wz z(r8L##@zQkJUQR*t6^m^=VF56{4#c7%F;&gTKBHxCm|?k`?^zkL9wJ*wsLOJRSZO# z8`<+Z-$Kaf@=1&q49=V{2^bjbDoM`}FsqbmSSWzlX4^d@f5-_b?_&NwD>aHq3ZEV4- z1?@r%-uEJf)#k}*L@zFb?-Nl=KA%rWS(0|1_dgF=0RY)bL z%YsKdV3jqIx<_GHZP_JLbY!R#Ym|+=4;F9QrTlb!cDysUj%cP*re`I%g^Kt}0Om0L zK{!n(?PzK@TCvl)#9n}o^^S&(mB-D1qcrlRsBMkc0lNb5oqF(|ccm_Y$?gNfv|HEJ zqKGN5Jdcc$NJqk-XgRt51$t9coSQ6#zDe{PY-C(|8u zHKM2F(@lQQ0TmtXrSe#^J+5<3<}^}BWXnGX_b!@R%ztCN@?>wZisYrJ$kXu(Cgn5$ z-E^mlETjRb=>ks1bR zzM^6>r-$Nq7L&j$_ zTAHdeQd-4S`J^3}HR)|0NO)$24-UvsZlV?5gHF|5oa`@;l^G_3D;~$F_+sFb_ihex zd%1CA@m$uk{iXI8YTc>DpN-Gi3QG9eJ_cO*1BfRVaP$S25$u)RtD5bj>tbh~d^}a;$yMg8#-RA?*6GZv@7jn z&}%^xLG&E0kHMbxP+35`nW9N~zrBl~A|W&%ShHE~&1hBIm~PNTIR`DH_B%C+Lc7sY z)-F3@LvT>wa5?F;`wn*HaZIeT`*EPr(NA0nXy{Gll?@)QFq34D4=u#tqN%fZZMqso zTyPJiKk&WNOj9HQvjT+ShG$b+n9V(|oSbvvXM5l=OQ#MJ`3cw!e}`gkqW04}e0xRO zvp8`(D_fR--Q>A}9S9-41-7Q@`Yim*w_mjcv#^Ixrv_!V-Teu~fP=*HPd;0IC)=ScMSq60~OeIS0 zVyc3R>D)ol5{u_+#? zF%p?nj>!Y#yLfNkt7Lgd-(H6jIV@EU4Ik503OdgQCKj1?zoC~+NaZvVa!?xHi||lL zW`C4RY0gv#5)2fMODXhSg5;i_DG~jg3+j6W@z)LByZm~@y7zRe2ugPMuvnv*xT8u~ zhqf@gl3|%#X2Fp^RuZT}YkCW2qpB4qzm<~7d8a~d1Z$3BCY7&5?)B?ZJ3vN`sFeAu z8K5$nXLZ$UHQu}Co17!nPmVTz7-%^Cc%L6-3oE)wvWY%;R!{b-`nFsNg--Xk;arj} z!rqarTq7AXezz5TR3F=p2f;^p_O5~fm%cqJK&>!yf`7mT9WFmB>*~gjDxVf<*qXgl zN|FG7#>lU*oZnGd-)}8o?&W7Z^a59GY>-GZ=8IEQchPv6VeNHjfh3TOjbD1)X>H<< ztq_gtv!?ZK^iz=2atUoNxWI!%n<$!1!L|T9xFag zb@5(d!?ZY+^!uA2jl>bO?bMXD`hS$S?*uXo%d){B-MKJ&SjCO;M(b7jQTR>d*T>6 z*P2kdBvg{nWLRO?G=F>9UDl40!Y@3H>}zJBKzd`W(7KJR-dUh}2a>@Zk$sJGWFULo8RB;O31*4SSax$)9~b7%Y1%GBzi!>tpeC3w8VMH zm*O;@fQdG+i%`I>A!v_)8QyJ;KQliB2;uo+7-M`Qq~dMIfJ27pEJ zP?iIz?{RZ;Yl*O`LL>TA@6Tbv`7X!Eh;FHV#%Yq7iof-`A!xsz&QV@o9-b$g+eCxc zN1fb8@M*)%Wgl&d#>VlpP)ag~$|Q0_$1$Ke7E2#6N3LWHD9`(cP&E$H?^j^gmsO-3 z*DkdLzD_Qa{69XV{Cq3#)C^rBJ4^~&Q_;X&jn~-fD?&f#-BXaN1+TQB%mvt0ZJ?I zn(J~eQbPs#$X2^Y$R7I8b0gNgL%uT^M(r6Ti9i;mPP&%p$xN{Te}Ha)gkS(J@AiP^ z(9UH8^wMOwX#!6E=f^H>MxaO*n%666z6I>!1(+3ZWcVFwxTs&fi)=|8AXB9@KKWg8 z97!D}OebipD}{)UJMc!0{TQw$I!6+ad8Iqmfr~&pso=q+3jYL9YUmA+8T--&5PFs7 zVEy><74;{Z6E$XjQ#JO>eYf9g{Gi z|1qhv8dTc=0E-*jkYH+gI+()!M z%7EdAr>j338xo6dsbWfwJ3gUY8yI=p2_B2K9%hx`$imZ!>LL)a$uJY03y3S#RQ&5~YQiTAOgfB6YiBgIB+KA=L>aNCisNXeFv;YAz zrjV9%@5nr_QAxZ?Pgz#w?E^EXynFB|94&EjkyoMPA`*QRduZ%~1zBC6gKlVR>=HfC z0LI|vv96;T=vG&7st;IEJyec{LDO&C!qrQ~`UZeYDi!3Qp>I+gF?xmWAL;leXsuj< z63RZe+MgXl;ocIAuEPzbVx~E>ljs%=Q4g_nX>;X&SV;y$#OByQbL-N0HsKYZnE=2il4^(U;-JuEAR|4n%m54wXSw5dIB-+VVS7MJQjyM#uO>#Z#2DPL z(Jc+-hgN;?GR8xY*NRh);_RFUjA zzT$8ilZlUR*5Hb97;zu6oELn58wK8UU56HuOz}WP_dQx{&l+CIDpU!x>mfh(Y$D?U z>5Ri@fQoC8-MeIjqfx4;+Vj#TLch_chEO`3Q6XVvrny;U+ z-M2~Pktx`bj-myP2b(v2L;CrPQ zZ+z7gPR)Iy4pk{LwFRA*M)<1FKBZ5T9MMe{E%KNg^zJ|fxU}{YeXD)$tO+Dt!f#qL&W6Ayi5jY!WCCwl0rxJXF`fs6@ z^X2arK(&%1@p}wG$LAIV@LBm^0cGmfnvnfhk!OD-q4}hQgavsYzO~z^yXP+Y!Yy5* zBm#hvf|qWVhCKT*|9_Em{v}V%c5bWqRJ4MsOvPgn(P;L!4D`1Q^tTN3@5(@8pZ=DC z{+5CMmVy43f&Rajf&L|GiGKmOPXKK?#$A+tGsvi(kP`vJl-kCw3xeTLc;xPezEjU% z3?66*%FZ&QO>BxrVYu?ucg!AZp$F%a}9TWuQmJzwtY3;<^-V@|H*zMO6hL z_vtnSiF^O0zZUn;C1|p!U;Waxb-8gZi#dk|qAgxQk`l7cuvL8ok-zgPFRda~{>U`= FzX23JaCHCx literal 0 HcmV?d00001 From 6e9f9ae75c09d8a78470c5bfb4f0bb88701a7c3a Mon Sep 17 00:00:00 2001 From: qijun Date: Wed, 3 May 2017 17:46:58 +0800 Subject: [PATCH 12/88] refactor use concepts doc --- doc/getstarted/concepts/src/train.py | 52 ++++++ doc/getstarted/concepts/use_concepts_cn.rst | 149 ++++++++++++++++++ doc/getstarted/index_cn.rst | 1 + doc/howto/index_cn.rst | 1 - .../usage/concepts/src/pserver_topology.dot | 68 -------- .../usage/concepts/src/trainer_config.py | 29 ---- doc/howto/usage/concepts/use_concepts_cn.rst | 139 ---------------- 7 files changed, 202 insertions(+), 237 deletions(-) create mode 100644 doc/getstarted/concepts/src/train.py create mode 100644 doc/getstarted/concepts/use_concepts_cn.rst delete mode 100644 doc/howto/usage/concepts/src/pserver_topology.dot delete mode 100644 doc/howto/usage/concepts/src/trainer_config.py delete mode 100644 doc/howto/usage/concepts/use_concepts_cn.rst diff --git a/doc/getstarted/concepts/src/train.py b/doc/getstarted/concepts/src/train.py new file mode 100644 index 0000000000..679d0a931a --- /dev/null +++ b/doc/getstarted/concepts/src/train.py @@ -0,0 +1,52 @@ +import paddle.v2 as paddle +import numpy as np + +# init paddle +paddle.init(use_gpu=False) + +# network config +x = paddle.layer.data(name='x', type=paddle.data_type.dense_vector(2)) +y_predict = paddle.layer.fc(input=x, size=1, act=paddle.activation.Linear()) +y = paddle.layer.data(name='y', type=paddle.data_type.dense_vector(1)) +cost = paddle.layer.mse_cost(input=y_predict, label=y) + +# create parameters +parameters = paddle.parameters.create(cost) +# create optimizer +optimizer = paddle.optimizer.Momentum(momentum=0) +# create trainer +trainer = paddle.trainer.SGD(cost=cost, + parameters=parameters, + update_equation=optimizer) + + +# event_handler to print training info +def event_handler(event): + if isinstance(event, paddle.event.EndIteration): + if event.batch_id % 1 == 0: + print "Pass %d, Batch %d, Cost %f" % (event.pass_id, event.batch_id, + event.cost) + + +# define training dataset reader +def train_reader(): + train_x = np.array([[1, 1], [1, 2], [3, 4], [5, 2]]) + train_y = np.array([-2, -3, -7, -7]) + + def reader(): + for i in xrange(train_y.shape[0]): + yield train_x[i], train_y[i] + + return reader + + +# define feeding map +feeding = {'x': 0, 'y': 1} + +# training +trainer.train( + reader=paddle.batch( + train_reader(), batch_size=1), + feeding=feeding, + event_handler=event_handler, + num_passes=100) diff --git a/doc/getstarted/concepts/use_concepts_cn.rst b/doc/getstarted/concepts/use_concepts_cn.rst new file mode 100644 index 0000000000..c4b32cc1a8 --- /dev/null +++ b/doc/getstarted/concepts/use_concepts_cn.rst @@ -0,0 +1,149 @@ +############ +新手入门 +############ + +PaddlePaddle是源于百度的一个深度学习平台。PaddlePaddle为深度学习研究人员提供了丰富的API，可以轻松的完成神经网络配置，模型训练等任务。 +这里将介绍PaddlePaddle的基本使用概念，并且展示了如何利用PaddlePaddle来解决一个经典的线性回归问题。 +在使用该文档之前，请参考 `安装文档 `_ 完成PaddlePaddle的安装。 + + +配置网络 +============ + +加载PaddlePaddle +---------------------- + +在进行网络配置之前，首先需要加载相应的Python库，并进行初始化操作。 + +.. code-block:: bash + + import paddle.v2 as paddle + import numpy as np + paddle.init(use_gpu=False) + + +搭建神经网络 +----------------------- + +搭建神经网络就像使用积木搭建宝塔一样。在PaddlePaddle中，layer是我们的积木，而神经网络是我们要搭建的宝塔。我们使用不同的layer进行组合，来搭建神经网络。 +宝塔的底端需要坚实的基座来支撑，同样，神经网络也需要一些特定的layer作为输入接口，来完成网络的训练。 + +例如，我们可以定义如下layer来描述神经网络的输入： + +.. code-block:: bash + + x = paddle.layer.data(name='x', type=paddle.data_type.dense_vector(2)) + y = paddle.layer.data(name='y', type=paddle.data_type.dense_vector(1)) + +其中x表示输入数据是一个维度为2的稠密向量，y表示输入数据是一个维度为1的稠密向量。 + +PaddlePaddle支持不同类型的输入数据，主要包括四种类型，和三种序列模式。 + +四种数据类型： + +* dense_vector：稠密的浮点数向量。 +* sparse_binary_vector：稀疏的01向量，即大部分值为0，但有值的地方必须为1。 +* sparse_float_vector：稀疏的向量，即大部分值为0，但有值的部分可以是任何浮点数。 +* integer：整数标签。 + +三种序列模式： + +* SequenceType.NO_SEQUENCE：不是一条序列 +* SequenceType.SEQUENCE：是一条时间序列 +* SequenceType.SUB_SEQUENCE： 是一条时间序列，且序列的每一个元素还是一个时间序列。 + +不同的数据类型和序列模式返回的格式不同，列表如下： + ++----------------------+---------------------+-----------------------------------+------------------------------------------------+ +| | NO_SEQUENCE | SEQUENCE | SUB_SEQUENCE | ++======================+=====================+===================================+================================================+ +| dense_vector | [f, f, ...] | [[f, ...], [f, ...], ...] | [[[f, ...], ...], [[f, ...], ...],...] | ++----------------------+---------------------+-----------------------------------+------------------------------------------------+ +| sparse_binary_vector | [i, i, ...] | [[i, ...], [i, ...], ...] | [[[i, ...], ...], [[i, ...], ...],...] | ++----------------------+---------------------+-----------------------------------+------------------------------------------------+ +| sparse_float_vector | [(i,f), (i,f), ...] | [[(i,f), ...], [(i,f), ...], ...] | [[[(i,f), ...], ...], [[(i,f), ...], ...],...] | ++----------------------+---------------------+-----------------------------------+------------------------------------------------+ +| integer_value | i | [i, i, ...] | [[i, ...], [i, ...], ...] | ++----------------------+---------------------+-----------------------------------+------------------------------------------------+ + +其中，f代表一个浮点数，i代表一个整数。 + +注意：对sparse_binary_vector和sparse_float_vector，PaddlePaddle存的是有值位置的索引。例如， + +- 对一个5维非序列的稀疏01向量 ``[0, 1, 1, 0, 0]`` ，类型是sparse_binary_vector，返回的是 ``[1, 2]`` 。 +- 对一个5维非序列的稀疏浮点向量 ``[0, 0.5, 0.7, 0, 0]`` ，类型是sparse_float_vector，返回的是 ``[(1, 0.5), (2, 0.7)]`` 。 + + +在定义输入layer之后，我们可以使用其他layer进行组合。在组合时，需要指定layer的输入来源。 + +例如，我们可以定义如下的layer组合： + +.. code-block:: bash + + y_predict = paddle.layer.fc(input=x, size=1, act=paddle.activation.Linear()) + cost = paddle.layer.mse_cost(input=y_predict, label=y) + +其中，x与y为之前描述的输入层；而y_predict是接收x作为输入，接上一个全连接层；cost接收y_predict与y作为输入，接上均方误差层。 + +最后一层cost中记录了神经网络的所有拓扑结构，通过组合不同的layer，我们即可完成神经网络的搭建。 + + +训练模型 +============ + +在完成神经网络的搭建之后，我们首先需要根据神经网络结构来创建所需要优化的parameters，并创建optimizer。 +之后，我们可以创建trainer来对网络进行训练。 + +.. code-block:: bash + + parameters = paddle.parameters.create(cost) + optimizer = paddle.optimizer.Momentum(momentum=0) + trainer = paddle.trainer.SGD(cost=cost, + parameters=parameters, + update_equation=optimizer) + +其中，trianer接收三个参数，包括神经网络拓扑结构，神经网络参数以及迭代方程。 + +在搭建神经网络的过程中，我们仅仅对神经网络的输入进行了描述。而trainer需要读取训练数据进行训练，PaddlePaddle中通过reader来加载数据。 + +.. code-block:: bash + + # define training dataset reader + def train_reader(): + train_x = np.array([[1, 1], [1, 2], [3, 4], [5, 2]]) + train_y = np.array([-2, -3, -7, -7]) + def reader(): + for i in xrange(train_y.shape[0]): + yield train_x[i], train_y[i] + return reader + +最终我们可以调用trainer的train方法启动训练： + +.. code-block:: bash + + # define feeding map + feeding = {'x': 0, 'y': 1} + + # event_handler to print training info + def event_handler(event): + if isinstance(event, paddle.event.EndIteration): + if event.batch_id % 1 == 0: + print "Pass %d, Batch %d, Cost %f" % ( + event.pass_id, event.batch_id, event.cost) + # training + trainer.train( + reader=paddle.batch(train_reader(), batch_size=1), + feeding=feeding, + event_handler=event_handler, + num_passes=100) + + +线性回归完整示例 +============== + +下面给出在三维空间中使用线性回归拟合一条直线的例子： + +.. literalinclude:: src/train.py + :linenos: + +有关线性回归的实际应用，可以参考Paddle book的 `第一章节 `_ \ No newline at end of file diff --git a/doc/getstarted/index_cn.rst b/doc/getstarted/index_cn.rst index cadf092f8f..0cb27f802c 100644 --- a/doc/getstarted/index_cn.rst +++ b/doc/getstarted/index_cn.rst @@ -5,5 +5,6 @@ :maxdepth: 1 build_and_install/index_cn.rst + concepts/use_concepts_cn.rst - `深度学习入门课程 `_ diff --git a/doc/howto/index_cn.rst b/doc/howto/index_cn.rst index 5b84eea491..26449a6365 100644 --- a/doc/howto/index_cn.rst +++ b/doc/howto/index_cn.rst @@ -8,7 +8,6 @@ :maxdepth: 1 usage/cmd_parameter/index_cn.rst - usage/concepts/use_concepts_cn.rst usage/cluster/cluster_train_cn.md usage/k8s/k8s_basis_cn.md usage/k8s/k8s_cn.md diff --git a/doc/howto/usage/concepts/src/pserver_topology.dot b/doc/howto/usage/concepts/src/pserver_topology.dot deleted file mode 100644 index 9ff658b849..0000000000 --- a/doc/howto/usage/concepts/src/pserver_topology.dot +++ /dev/null @@ -1,68 +0,0 @@ -graph pp_topology { - rankdir=BT; - subgraph cluster_node0 { - style=filled; - color=lightgrey; - node [style=filled, color=white, shape=box]; - label = "机器0" - - pserver0 [label="Parameter \n Server 0"] - trainer0 [label="Trainer 0"] - } - subgraph cluster_node1 { - style=filled; - color=lightgrey; - node [style=filled, color=white, shape=box]; - label = "机器1" - - pserver1 [label="Parameter \n Server 1"] - trainer1 [label="Trainer 1"] - } - - subgraph cluster_node2 { - style=filled; - color=lightgrey; - node [style=filled, color=white, shape=box]; - label = "机器2" - - pserver2 [label="Parameter \n Server 2"] - trainer2 [label="Trainer 2"] - } - - subgraph cluster_node3 { - style=filled; - color=lightgrey; - node [style=filled, color=white, shape=box]; - label = "机器3" - - pserver3 [label="Parameter \n Server 3"] - trainer3 [label="Trainer 3"] - } - - data [label="数据", shape=hexagon] - - trainer0 -- pserver0 - trainer0 -- pserver1 - trainer0 -- pserver2 - trainer0 -- pserver3 - - trainer1 -- pserver0 - trainer1 -- pserver1 - trainer1 -- pserver2 - trainer1 -- pserver3 - - trainer2 -- pserver0 - trainer2 -- pserver1 - trainer2 -- pserver2 - trainer2 -- pserver3 - - trainer3 -- pserver0 - trainer3 -- pserver1 - trainer3 -- pserver2 - trainer3 -- pserver3 - - data -- trainer0 - data -- trainer1 - data -- trainer2 - data -- trainer3 -} diff --git a/doc/howto/usage/concepts/src/trainer_config.py b/doc/howto/usage/concepts/src/trainer_config.py deleted file mode 100644 index 3eccbd7bc1..0000000000 --- a/doc/howto/usage/concepts/src/trainer_config.py +++ /dev/null @@ -1,29 +0,0 @@ -from paddle.trainer_config_helpers import * - -define_py_data_sources2( - train_list='train.list', - test_list='test.list', - module='provider', - obj='process') -settings( - batch_size=128, - learning_rate=1e-3, - learning_method=AdamOptimizer(), - regularization=L2Regularization(0.5)) - -img = data_layer(name='pixel', size=28 * 28) - -hidden1 = simple_img_conv_pool( - input=img, filter_size=3, num_filters=32, pool_size=3, num_channel=1) - -hidden2 = fc_layer( - input=hidden1, - size=200, - act=TanhActivation(), - layer_attr=ExtraAttr(drop_rate=0.5)) -predict = fc_layer(input=hidden2, size=10, act=SoftmaxActivation()) - -outputs( - classification_cost( - input=predict, label=data_layer( - name='label', size=10))) diff --git a/doc/howto/usage/concepts/use_concepts_cn.rst b/doc/howto/usage/concepts/use_concepts_cn.rst deleted file mode 100644 index fa334bcbb9..0000000000 --- a/doc/howto/usage/concepts/use_concepts_cn.rst +++ /dev/null @@ -1,139 +0,0 @@ -############ -基本使用概念 -############ - -PaddlePaddle是一个深度学习框架，支持单机模式和多机模式。 - -单机模式用命令 ``paddle train`` 可以启动一个trainer进程，单机训练通常只包括一个trainer进程。如果数据规模比较大，希望加速训练，可以启动分布式作业。一个分布式作业里包括若干trainer进程和若干Parameter Server（或称pserver）进程。用命令 ``paddle pserver`` 可以启动 pserver 进程，pserver进程用于协调多个trainer进程之间的通信。 - -本文首先介绍trainer进程中的一些使用概念，然后介绍pserver进程中概念。 - -.. contents:: - -系统框图 -======== - -下图描述了用户使用框图，PaddlePaddle的trainer进程里内嵌了Python解释器，trainer进程可以利用这个解释器执行Python脚本，Python脚本里定义了模型配置、训练算法、以及数据读取函数。其中，数据读取程序往往定义在一个单独Python脚本文件里，被称为数据提供器（DataProvider），通常是一个Python函数。模型配置、训练算法通常定义在另一单独Python文件中, 称为训练配置文件。下面将分别介绍这两部分。 - -.. graphviz:: - - digraph pp_process { - rankdir=LR; - config_file [label="用户神经网络配置"]; - subgraph cluster_pp { - style=filled; - color=lightgrey; - node [style=filled, color=white, shape=box]; - label = "PaddlePaddle C++"; - py [label="Python解释器"]; - } - data_provider [label="用户数据解析"]; - config_file -> py; - py -> data_provider [dir="back"]; - } - -数据提供器 -========== - -DataProvider是PaddlePaddle系统的数据提供器，将用户的原始数据转换成系统可以识别的数据类型。每当系统需要新的数据训练时, trainer进程会调用DataProvider函数返回数据。当所有数据读取完一轮后，DataProvider返回空数据，通知系统一轮数据读取结束，并且系统每一轮训练开始时会重置DataProvider。需要注意的是，DataProvider是被系统调用，而不是新数据驱动系统，一些随机化噪声添加都应该在DataProvider中完成。 - -在不同的应用里，训练数据的格式往往各不相同。因此，为了用户能够灵活的处理数据，我们提供了Python处理数据的接口，称为 ``PyDataProvider`` 。在 ``PyDataProvider`` 中，系统C++模块接管了shuffle、处理batch、GPU和CPU通信、双缓冲、异步读取等问题，一些情况下(如：``min_pool_size=0``)需要Python接口里处理shuffle，可以参考 :ref:`api_pydataprovider2` 继续深入了解。 - - -训练配置文件 -============ - -训练配置文件主要包括数据源、优化算法、网络结构配置三部分。 其中数据源配置与DataProvider的关系是：DataProvider里定义数据读取函数，训练配置文件的数据源配置中指定DataProvider文件名字、生成数据函数接口，请不要混淆。 - -一个简单的训练配置文件为： - -.. literalinclude:: src/trainer_config.py - :linenos: - -文件开头 ``from paddle.trainer_config_helpers import *`` ，是因为PaddlePaddle配置文件与C++模块通信的最基础协议是protobuf，为了避免用户直接写复杂的protobuf string，我们为用户定以Python接口来配置网络，该Python代码可以生成protobuf包，这就是 :ref:`api_trainer_config` 的作用。因此，在文件的开始，需要import这些函数。 这个包里面包含了模型配置需要的各个模块。 - -下面分别介绍数据源配置、优化算法配置、网络结构配置这三部分该概念。 - -数据源配置 ----------- - -使用 ``PyDataProvider2`` 的函数 ``define_py_data_sources2`` 配置数据源。``define_py_data_sources2`` 里通过train_list和test_list指定是训练文件列表和测试文件列表。 如果传入字符串的话，是指一个数据列表文件。这个数据列表文件中包含的是每一个训练或者测试文件的路径。如果传入一个list的话，则会默认生成一个list文件，再传入给train.list或者test.list。 - -``module`` 和 ``obj`` 指定了DataProvider的文件名和返回数据的函数名。更详细的使用，请参考 :ref:`api_pydataprovider2` 。 - -优化算法配置 ------------- - -通过 :ref:`api_trainer_config_helpers_optimizers_settings` 接口设置神经网络所使用的训练参数和 :ref:`api_trainer_config_helpers_optimizers` ，包括学习率、batch_size、优化算法、正则方法等，具体的使用方法请参考 :ref:`api_trainer_config_helpers_optimizers_settings` 文档。 - -网络结构配置 ------------- - -神经网络配置主要包括网络连接、激活函数、损失函数、评估器。 - -- 网络连接： 主要由Layer组成，每个Layer返回的都是一个 ``LayerOutput`` 对象，Layer里面可以定义参数属性、激活类型等。 - - 为了更灵活的配置，PaddlePaddle提供了基于 Projection 或者 Operator 的配置，这两个需要与 ``mixed_layer`` 配合使用。这里简单介绍Layer、Projection、Operator的概念: - - - Layer: 神经网络的某一层，可以有可学习的参数，一般是封装了许多复杂操作的集合。 - - Projection：需要与 ``mixed_layer`` 配合使用，含可学习参数。 - - Operator： 需要与 ``mixed_layer`` 配合使用，不含可学习参数，输入全是其他Layer的输出。 - - - 这个配置文件网络由 ``data_layer`` 、 ``simple_img_conv_pool`` 、 ``fc_layer`` 组成。 - - - :ref:`api_trainer_config_helpers_layers_data_layer` ： 通常每个配置文件都会包括 ``data_layer`` ，定义输入数据大小。 - - :ref:`api_trainer_config_helpers_network_simple_img_conv_pool` ：是一个组合层，包括了图像的卷积 (convolution)和池化(pooling)。 - - :ref:`api_trainer_config_helpers_layers_fc_layer` ：全连接层，激活函数为Softmax，这里也可叫分类层。 - -- 损失函数和评估器：损失函数即为网络的优化目标，评估器可以评价模型结果。 - - PaddlePaddle包括很多损失函数和评估起，详细可以参考 :ref:`api_trainer_config_helpers_layers_cost_layers` 和 :ref:`api_trainer_config_helpers_evaluators` 。这里 ``classification_cost`` 默认使用多类交叉熵损失函数和分类错误率统计评估器。 - -- ``outputs``: 标记网络输出的函数为 ``outputs`` 。 - - 训练阶段，网络的输出为神经网络的优化目标；预测阶段，网络的输出也可通过 ``outputs`` 标记。 - - -这里对 ``mixed_layer`` 稍做详细说明， 该Layer将多个输入(Projection 或 Operator)累加求和，具体计算是通过内部的 Projection 和 Operator 完成，然后加 Bias 和 activation 操作， - -例如，和 ``fc_layer`` 同样功能的 ``mixed_layer`` 是: - -.. code-block:: python - - data = data_layer(name='data', size=200) - with mixed_layer(size=200) as out: - out += full_matrix_projection(input=data) - -PaddlePaddle 可以使用 ``mixed layer`` 配置出非常复杂的网络，甚至可以直接配置一个完整的LSTM。用户可以参考 :ref:`api_trainer_config_helpers_layers_mixed_layer` 的相关文档进行配置。 - - -分布式训练 -========== - -PaddlePaddle多机采用经典的 Parameter Server 架构对多个节点的 trainer 进行同步。多机训练的经典拓扑结构如下\: - -.. graphviz:: src/pserver_topology.dot - -图中每个灰色方块是一台机器，在每个机器中，先使用命令 ``paddle pserver`` 启动一个pserver进程，并指定端口号，可能的参数是\: - -.. code-block:: bash - - paddle pserver --port=5000 --num_gradient_servers=4 --tcp_rdma='tcp' --nics='eth0' - -* ``--port=5000`` : 指定 pserver 进程端口是 5000 。 -* ``--gradient_servers=4`` : 有四个训练进程(PaddlePaddle 将 trainer 也称作 GradientServer ，因为其为负责提供Gradient) 。 -* ``--tcp_rdma='tcp' --nics=`eth0```: 指定以太网类型为TCP网络，指定网络接口名字为eth0。 - -启动之后 pserver 进程之后，需要启动 trainer 训练进程，在各个机器上运行如下命令\: - -.. code-block:: bash - - paddle train --port=5000 --pservers=192.168.100.101,192.168.100.102,192.168.100.103,192.168.100.104 --config=... - -对于简单的多机协同训练使用上述方式即可。另外，pserver/train 通常在高级情况下，还需要设置下面两个参数\： - -* --ports_num\: 一个 pserver 进程共绑定多少个端口用来做稠密更新，默认是1。 -* --ports_num_for_sparse\: 一个pserver进程共绑定多少端口用来做稀疏更新，默认是0。 - -使用手工指定端口数量，是因为Paddle的网络通信中，使用了 int32 作为消息长度，比较容易在大模型下溢出。所以，在 pserver 进程中可以启动多个子线程去接受 trainer 的数据，这样单个子线程的长度就不会溢出了。但是这个值不可以调的过大，因为增加这个值，对性能尤其是内存占用有一定的开销，另外稀疏更新的端口如果太大的话，很容易导致某一个参数服务器没有分配到任何参数。 From c3c6acd3465cf2364959c7f96b4e0998f6ecf8bb Mon Sep 17 00:00:00 2001 From: qijun Date: Wed, 3 May 2017 19:05:36 +0800 Subject: [PATCH 13/88] update some links --- doc/getstarted/concepts/use_concepts_cn.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/getstarted/concepts/use_concepts_cn.rst b/doc/getstarted/concepts/use_concepts_cn.rst index c4b32cc1a8..ea0b78f3c3 100644 --- a/doc/getstarted/concepts/use_concepts_cn.rst +++ b/doc/getstarted/concepts/use_concepts_cn.rst @@ -1,5 +1,5 @@ ############ -新手入门 +基本使用概念 ############ PaddlePaddle是源于百度的一个深度学习平台。PaddlePaddle为深度学习研究人员提供了丰富的API，可以轻松的完成神经网络配置，模型训练等任务。 @@ -146,4 +146,4 @@ PaddlePaddle支持不同类型的输入数据，主要包括四种类型，和 .. literalinclude:: src/train.py :linenos: -有关线性回归的实际应用，可以参考Paddle book的 `第一章节 `_ \ No newline at end of file +有关线性回归的实际应用，可以参考Paddle book的 `第一章节 `_ \ No newline at end of file From 74ad98e993d05042f59307bb41d420cd1b1cfe4e Mon Sep 17 00:00:00 2001 From: chrisxu2016 <823254351@qq.com> Date: Fri, 5 May 2017 11:13:22 +0800 Subject: [PATCH 14/88] add release.cn.md --- RELEASE.cn.md | 83 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100755 RELEASE.cn.md diff --git a/RELEASE.cn.md b/RELEASE.cn.md new file mode 100755 index 0000000000..e28586cf54 --- /dev/null +++ b/RELEASE.cn.md @@ -0,0 +1,83 @@ +# v0.10.0版本 + +我们非常高兴发布了PaddlePaddleV0.10.0版。在这一版，开发了新的[Python API](http://research.baidu.com/paddlepaddles-new-api-simplifies-deep-learning-programs/)。 + +- 旧的Python API由于难以学习和使用已经过时了。使用旧版本的API至少需要两份python文件，即定义数据生成器和网络拓扑结构的文件。用户通过运行paddle_trainer的C++程序来启动PaddlePaddle任务，该程序调用Python解释器来运行定义网络拓扑结构的配置脚本，然后通过迭代加载数据生成器提供的小批量数据启动训练循环。这与Python的现代编辑方式不符，比如Jupyter Notebook。 + +- 新版的API被称为 *V2 API*，允许我们在单个.py文件中编辑更短的Python程序来定义网络结构和数据。此外，该Python程序也可以在Jupyter Notebook中运行，因为PaddlePaddle可以作为共享库来被Python程序加载和使用，这也是入门级的Python程序使用方式。 + +基于新的API，我们提供了一个在线的学习文档 [Deep Learning 101](http://book.paddlepaddle.org/index.en.html) 及其[中文版本](http://book.paddlepaddle.org/)。 + +我们还致力于迭代更新新版API的在线文档。我们将在下一个版本中发布更多的改进文档。 + +我们还致力于将新版API引入分布式模型训练中（通过MPI和Kubernetes）。这项工作正在进行中。我们将在下一个版本中发布更多内容。 + +## 新特点 + +* 发布新版[Python API](http://research.baidu.com/paddlepaddles-new-api-simplifies-deep-learning-programs/)。 +* 学习文档 [Deep Learning 101](http://book.paddlepaddle.org/index.en.html) 及其[中文版本](http://book.paddlepaddle.org/)。 +* 支持矩形输入的CNN。 +* 为seqlastin和seqfirstin提供stride pooling。 +* 在`trainer_config_helpers`中暴露`seq_concat_layer/seq_reshape_layer`。 +* 添加公共数据集包：CIFAR，MNIST，IMDB，WMT14，CONLL05，movielens，imikolov。 +* 针对Single Shot Multibox Detection增加 Prior box layer。 +* 增加光滑的L1损失。 +* 在V2 API中增加 data reader 创建器和修饰器。 +* 增加cmrnorm投影的CPU实现。 + + +## 改进 + +* 提供`paddle_trainer`的Python virtualenv支持。 +* 增加用于自动格式化代码pre-commit hooks。 +* 升级protobuf到3.x版本。 +* 在Python数据生成器中提供一个检测数据类型的选项。 +* 加速GPU中average层的后向反馈。 +* 细化文档。 +* 使用Travis-CI检查文档中的死链接。 +* 增加解释`sparse_vector的示例。 +* 在layer_math.py中添加ReLU。 +* 简化Quick Start示例中的数据处理流程。 +* 支持CUDNN Deconv。 +* 在v2 API中增加数据feeder。 +* 在情感分析示例的演示中增加对标准输入流中样本的预测。 +* 提供图像预处理的多进程接口。 +* 增加V1 API的基准文档。 +* 在`layer_math.py`中增加ReLU。 +* 提供公共数据集的自动下载包。 +* 将`Argument::sumCost`重新命名为`Argument::sum`。 +* 将Argument::sum暴露给python。 +* 为矩阵相关的表达式评估增加一个新的`TensorExpression`实现。 +* 增加延迟分配来优化批处理多表达式计算。 +* 增加抽象的类函数及其实现。 + * `PadFunc` 和 `PadGradFunc`。 + * `ContextProjectionForwardFunc` 和 `ContextProjectionBackwardFunc`。 + * `CosSimBackward` 和 `CosSimBackwardFunc`。 + * `CrossMapNormalFunc` 和 `CrossMapNormalGradFunc`。 + * `MulFunc`。 +* 增加`AutoCompare`和`FunctionCompare`类，使得编写比较gpu和cpu版本函数的单元测试更容易。 +* 生成`libpaddle_test_main.a`并删除测试文件内的主函数。 +* 支持PyDataProvider2中numpy中的稠密向量。 +* 清理代码库，删除一些复制粘贴的代码片段。 + * 增加`SparseRowMatrix`的抽样类`RowBuffer`。 + * 清理`GradientMachine`的接口。 + * 在layer中增加`override`关键字。 + * 简化`Evaluator::create`，使用`ClassRegister`来创建`Evaluator`。 +* 下载演示的数据集时检查MD5校验。 +* 添加`paddle::Error`，用于替代Paddle中的`LOG(FATAL)`。 + + +## 错误修复 + +* 检查`recurrent_group`的layer输入类型。 +* 不要用.cu源文件运行`clang-format`。 +* 修复`LogActivation`的使用错误。 +* 修复运行`test_layerHelpers`多次的错误。 +* 修复seq2seq示例超出原型消息大小限制的错误。 +* 修复在GPU模式下dataprovider转换的错误。 +* 修复`GatedRecurrentLayer`中的错误。 +* 修复在测试多个模型时`BatchNorm`的错误。 +* 修复paramRelu在单元测试时崩溃的错误。 +* 修复`CpuSparseMatrix`编译时相关的警告。 +* 修复`MultiGradientMachine`在`trainer_count > batch_size`时的错误。 +* 修复`PyDataProvider2`阻止异步加载数据的错误。 From 7302d40a376673d935f7bd682ee6748ec1e15ad9 Mon Sep 17 00:00:00 2001 From: chrisxu2016 <823254351@qq.com> Date: Fri, 5 May 2017 11:16:42 +0800 Subject: [PATCH 15/88] add RELEASE.cn.md --- RELEASE.cn.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RELEASE.cn.md b/RELEASE.cn.md index e28586cf54..e27b1f480f 100755 --- a/RELEASE.cn.md +++ b/RELEASE.cn.md @@ -1,6 +1,6 @@ # v0.10.0版本 -我们非常高兴发布了PaddlePaddleV0.10.0版。在这一版，开发了新的[Python API](http://research.baidu.com/paddlepaddles-new-api-simplifies-deep-learning-programs/)。 +我们非常高兴发布了PaddlePaddle V0.10.0版。并开发了新的[Python API](http://research.baidu.com/paddlepaddles-new-api-simplifies-deep-learning-programs/)。 - 旧的Python API由于难以学习和使用已经过时了。使用旧版本的API至少需要两份python文件，即定义数据生成器和网络拓扑结构的文件。用户通过运行paddle_trainer的C++程序来启动PaddlePaddle任务，该程序调用Python解释器来运行定义网络拓扑结构的配置脚本，然后通过迭代加载数据生成器提供的小批量数据启动训练循环。这与Python的现代编辑方式不符，比如Jupyter Notebook。 From 4b6d891903494fe888abebfebe603a58a4cb381a Mon Sep 17 00:00:00 2001 From: xushaoyong <823254351@qq.com> Date: Fri, 5 May 2017 12:02:44 +0800 Subject: [PATCH 16/88] Update RELEASE.cn.md --- RELEASE.cn.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/RELEASE.cn.md b/RELEASE.cn.md index e27b1f480f..d9eb8af98b 100755 --- a/RELEASE.cn.md +++ b/RELEASE.cn.md @@ -1,8 +1,8 @@ # v0.10.0版本 -我们非常高兴发布了PaddlePaddle V0.10.0版。并开发了新的[Python API](http://research.baidu.com/paddlepaddles-new-api-simplifies-deep-learning-programs/)。 +我们非常高兴发布了PaddlePaddle V0.10.0版,并开发了新的[Python API](http://research.baidu.com/paddlepaddles-new-api-simplifies-deep-learning-programs/)。 -- 旧的Python API由于难以学习和使用已经过时了。使用旧版本的API至少需要两份python文件，即定义数据生成器和网络拓扑结构的文件。用户通过运行paddle_trainer的C++程序来启动PaddlePaddle任务，该程序调用Python解释器来运行定义网络拓扑结构的配置脚本，然后通过迭代加载数据生成器提供的小批量数据启动训练循环。这与Python的现代编辑方式不符，比如Jupyter Notebook。 +- 旧的Python API由于难以学习和使用已经过时了。使用旧版本的API至少需要两份python文件，分别是定义数据生成器和定义网络拓扑结构的文件。用户通过运行`paddle_trainer`的C++程序来启动PaddlePaddle任务，该程序调用Python解释器来运行定义网络拓扑结构的文件，然后通过迭代加载数据生成器提供的小批量数据启动训练循环。这与Python的现代编辑方式不符，比如Jupyter Notebook。 - 新版的API被称为 *V2 API*，允许我们在单个.py文件中编辑更短的Python程序来定义网络结构和数据。此外，该Python程序也可以在Jupyter Notebook中运行，因为PaddlePaddle可以作为共享库来被Python程序加载和使用，这也是入门级的Python程序使用方式。 From 3aa9ebd5b434749ad819d0095ba7ce233ddd2ca4 Mon Sep 17 00:00:00 2001 From: xushaoyong <823254351@qq.com> Date: Fri, 5 May 2017 12:09:33 +0800 Subject: [PATCH 17/88] Update RELEASE.cn.md --- RELEASE.cn.md | 25 +++++++++++-------------- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/RELEASE.cn.md b/RELEASE.cn.md index d9eb8af98b..c67e9756a8 100755 --- a/RELEASE.cn.md +++ b/RELEASE.cn.md @@ -4,18 +4,16 @@ - 旧的Python API由于难以学习和使用已经过时了。使用旧版本的API至少需要两份python文件，分别是定义数据生成器和定义网络拓扑结构的文件。用户通过运行`paddle_trainer`的C++程序来启动PaddlePaddle任务，该程序调用Python解释器来运行定义网络拓扑结构的文件，然后通过迭代加载数据生成器提供的小批量数据启动训练循环。这与Python的现代编辑方式不符，比如Jupyter Notebook。 -- 新版的API被称为 *V2 API*，允许我们在单个.py文件中编辑更短的Python程序来定义网络结构和数据。此外，该Python程序也可以在Jupyter Notebook中运行，因为PaddlePaddle可以作为共享库来被Python程序加载和使用，这也是入门级的Python程序使用方式。 +- 新版的API被称为 *V2 API*，允许我们在单个.py文件中，通过编辑更短的Python程序来定义网络结构和数据。此外，该Python程序也可以在Jupyter Notebook中运行，因为PaddlePaddle可以作为共享库来被Python程序加载和使用。 基于新的API，我们提供了一个在线的学习文档 [Deep Learning 101](http://book.paddlepaddle.org/index.en.html) 及其[中文版本](http://book.paddlepaddle.org/)。 -我们还致力于迭代更新新版API的在线文档。我们将在下一个版本中发布更多的改进文档。 - -我们还致力于将新版API引入分布式模型训练中（通过MPI和Kubernetes）。这项工作正在进行中。我们将在下一个版本中发布更多内容。 +我们还致力于迭代更新新版API的在线文档，并将新版API引入分布式集群（包括MPI和Kubernetes）训练中。我们将在下一个版本中发布更多的内容。 ## 新特点 * 发布新版[Python API](http://research.baidu.com/paddlepaddles-new-api-simplifies-deep-learning-programs/)。 -* 学习文档 [Deep Learning 101](http://book.paddlepaddle.org/index.en.html) 及其[中文版本](http://book.paddlepaddle.org/)。 +* 学习深度学习系列课程 [Deep Learning 101](http://book.paddlepaddle.org/index.en.html) 及其[中文版本](http://book.paddlepaddle.org/)。 * 支持矩形输入的CNN。 * 为seqlastin和seqfirstin提供stride pooling。 * 在`trainer_config_helpers`中暴露`seq_concat_layer/seq_reshape_layer`。 @@ -29,13 +27,13 @@ ## 改进 * 提供`paddle_trainer`的Python virtualenv支持。 -* 增加用于自动格式化代码pre-commit hooks。 +* 增加代码自动格式化的pre-commit hooks。 * 升级protobuf到3.x版本。 * 在Python数据生成器中提供一个检测数据类型的选项。 -* 加速GPU中average层的后向反馈。 +* 加速GPU中average层的后向反馈计算。 * 细化文档。 * 使用Travis-CI检查文档中的死链接。 -* 增加解释`sparse_vector的示例。 +* 增加解释`sparse_vector`的示例。 * 在layer_math.py中添加ReLU。 * 简化Quick Start示例中的数据处理流程。 * 支持CUDNN Deconv。 @@ -45,11 +43,10 @@ * 增加V1 API的基准文档。 * 在`layer_math.py`中增加ReLU。 * 提供公共数据集的自动下载包。 -* 将`Argument::sumCost`重新命名为`Argument::sum`。 -* 将Argument::sum暴露给python。 +* 将`Argument::sumCost`重新命名为`Argument::sum`，并暴露给python。 * 为矩阵相关的表达式评估增加一个新的`TensorExpression`实现。 * 增加延迟分配来优化批处理多表达式计算。 -* 增加抽象的类函数及其实现。 +* 增加抽象的类函数及其实现： * `PadFunc` 和 `PadGradFunc`。 * `ContextProjectionForwardFunc` 和 `ContextProjectionBackwardFunc`。 * `CosSimBackward` 和 `CosSimBackwardFunc`。 @@ -57,8 +54,8 @@ * `MulFunc`。 * 增加`AutoCompare`和`FunctionCompare`类，使得编写比较gpu和cpu版本函数的单元测试更容易。 * 生成`libpaddle_test_main.a`并删除测试文件内的主函数。 -* 支持PyDataProvider2中numpy中的稠密向量。 -* 清理代码库，删除一些复制粘贴的代码片段。 +* 支持PyDataProvider2中numpy的稠密向量。 +* 清理代码库，删除一些复制粘贴的代码片段： * 增加`SparseRowMatrix`的抽样类`RowBuffer`。 * 清理`GradientMachine`的接口。 * 在layer中增加`override`关键字。 @@ -73,7 +70,7 @@ * 不要用.cu源文件运行`clang-format`。 * 修复`LogActivation`的使用错误。 * 修复运行`test_layerHelpers`多次的错误。 -* 修复seq2seq示例超出原型消息大小限制的错误。 +* 修复seq2seq示例超出消息大小限制的错误。 * 修复在GPU模式下dataprovider转换的错误。 * 修复`GatedRecurrentLayer`中的错误。 * 修复在测试多个模型时`BatchNorm`的错误。 From c003efda6963ecd924e4990c4877807ee88c47ab Mon Sep 17 00:00:00 2001 From: xushaoyong <823254351@qq.com> Date: Fri, 5 May 2017 12:12:00 +0800 Subject: [PATCH 18/88] Update RELEASE.cn.md --- RELEASE.cn.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RELEASE.cn.md b/RELEASE.cn.md index c67e9756a8..5c467a853e 100755 --- a/RELEASE.cn.md +++ b/RELEASE.cn.md @@ -1,6 +1,6 @@ # v0.10.0版本 -我们非常高兴发布了PaddlePaddle V0.10.0版,并开发了新的[Python API](http://research.baidu.com/paddlepaddles-new-api-simplifies-deep-learning-programs/)。 +我们非常高兴发布了PaddlePaddle V0.10.0版，并开发了新的[Python API](http://research.baidu.com/paddlepaddles-new-api-simplifies-deep-learning-programs/)。 - 旧的Python API由于难以学习和使用已经过时了。使用旧版本的API至少需要两份python文件，分别是定义数据生成器和定义网络拓扑结构的文件。用户通过运行`paddle_trainer`的C++程序来启动PaddlePaddle任务，该程序调用Python解释器来运行定义网络拓扑结构的文件，然后通过迭代加载数据生成器提供的小批量数据启动训练循环。这与Python的现代编辑方式不符，比如Jupyter Notebook。 From a687a0b4efb1c03ef2cc19c12250630dc0717603 Mon Sep 17 00:00:00 2001 From: xushaoyong <823254351@qq.com> Date: Fri, 5 May 2017 12:14:32 +0800 Subject: [PATCH 19/88] Update RELEASE.cn.md --- RELEASE.cn.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RELEASE.cn.md b/RELEASE.cn.md index 5c467a853e..5deaf230a8 100755 --- a/RELEASE.cn.md +++ b/RELEASE.cn.md @@ -13,7 +13,7 @@ ## 新特点 * 发布新版[Python API](http://research.baidu.com/paddlepaddles-new-api-simplifies-deep-learning-programs/)。 -* 学习深度学习系列课程 [Deep Learning 101](http://book.paddlepaddle.org/index.en.html) 及其[中文版本](http://book.paddlepaddle.org/)。 +* 发布深度学习系列课程 [Deep Learning 101](http://book.paddlepaddle.org/index.en.html) 及其[中文版本](http://book.paddlepaddle.org/)。 * 支持矩形输入的CNN。 * 为seqlastin和seqfirstin提供stride pooling。 * 在`trainer_config_helpers`中暴露`seq_concat_layer/seq_reshape_layer`。 From 70d15e84bebfc9bbb7d1a1aa37e32ed9806033ac Mon Sep 17 00:00:00 2001 From: yangyaming Date: Fri, 5 May 2017 12:29:10 +0800 Subject: [PATCH 20/88] Add dataset ptb --- python/paddle/v2/dataset/imikolov.py | 4 +- python/paddle/v2/dataset/ptb.py | 169 +++++++++++++++++++++ python/paddle/v2/dataset/tests/ptb_test.py | 53 +++++++ 3 files changed, 224 insertions(+), 2 deletions(-) create mode 100644 python/paddle/v2/dataset/ptb.py create mode 100644 python/paddle/v2/dataset/tests/ptb_test.py diff --git a/python/paddle/v2/dataset/imikolov.py b/python/paddle/v2/dataset/imikolov.py index bf88fe1557..97b9f7d915 100644 --- a/python/paddle/v2/dataset/imikolov.py +++ b/python/paddle/v2/dataset/imikolov.py @@ -41,7 +41,7 @@ def word_count(f, word_freq=None): return word_freq -def build_dict(typo_freq=50): +def build_dict(min_word_freq=50): """ Build a word dictionary from the corpus, Keys of the dictionary are words, and values are zero-based IDs of these words. @@ -59,7 +59,7 @@ def build_dict(typo_freq=50): # remove for now, since we will set it as last index del word_freq[''] - word_freq = filter(lambda x: x[1] > typo_freq, word_freq.items()) + word_freq = filter(lambda x: x[1] > min_word_freq, word_freq.items()) word_freq_sorted = sorted(word_freq, key=lambda x: (-x[1], x[0])) words, _ = list(zip(*word_freq_sorted)) diff --git a/python/paddle/v2/dataset/ptb.py b/python/paddle/v2/dataset/ptb.py new file mode 100644 index 0000000000..ea68daf9c6 --- /dev/null +++ b/python/paddle/v2/dataset/ptb.py @@ -0,0 +1,169 @@ +# Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserved +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +langauge model's simple dataset. + +This module will download dataset from +http://www.fit.vutbr.cz/~imikolov/rnnlm/ and parse training set and test set +into paddle reader creators. +""" +import paddle.v2.dataset.common +import collections +import tarfile + +__all__ = ['train', 'test', 'build_dict'] + +URL = 'http://www.fit.vutbr.cz/~imikolov/rnnlm/simple-examples.tgz' +MD5 = '30177ea32e27c525793142b6bf2c8e2d' + + +def word_count(f, word_freq=None): + if word_freq is None: + word_freq = collections.defaultdict(int) + + for l in f: + for w in l.strip().split(): + word_freq[w] += 1 + word_freq[''] += 1 + word_freq[''] += 1 + + return word_freq + + +def build_dict(min_word_freq=50): + """ + Build a word dictionary from the corpus, Keys of the dictionary are words, + and values are zero-based IDs of these words. + """ + train_filename = './simple-examples/data/ptb.train.txt' + test_filename = './simple-examples/data/ptb.valid.txt' + with tarfile.open( + paddle.v2.dataset.common.download( + paddle.v2.dataset.imikolov.URL, 'imikolov', + paddle.v2.dataset.imikolov.MD5)) as tf: + trainf = tf.extractfile(train_filename) + testf = tf.extractfile(test_filename) + word_freq = word_count(testf, word_count(trainf)) + if '' in word_freq: + # remove for now, since we will set it as last index + del word_freq[''] + + word_freq = filter(lambda x: x[1] > min_word_freq, word_freq.items()) + + word_freq_sorted = sorted(word_freq, key=lambda x: (-x[1], x[0])) + words, _ = list(zip(*word_freq_sorted)) + word_idx = dict(zip(words, xrange(len(words)))) + word_idx[''] = len(words) + + return word_idx + + +def reader_creator(filename, reader_type, word_idx, n=-1): + def reader(): + with tarfile.open( + paddle.v2.dataset.common.download( + paddle.v2.dataset.imikolov.URL, 'imikolov', + paddle.v2.dataset.imikolov.MD5)) as tf: + f = tf.extractfile(filename) + + UNK = word_idx[''] + + for l in f: + if 'ngram' == reader_type: + assert n > -1, 'Invalid gram length' + l = [''] + l.strip().split() + [''] + if len(l) < n: continue + l = [word_idx.get(w, UNK) for w in l] + for i in range(n, len(l) + 1): + yield tuple(l[i - n:i]) + elif 'seq' == reader_type: + l = l.strip().split() + l = [word_idx.get(w, UNK) for w in l] + src_seq = [word_idx['']] + l + trg_seq = l + [word_idx['']] + yield src_seq, trg_seq + + return reader + + +def ngram_train(word_idx, n): + """ + ptb ngram type training set creator. + + It returns a reader creator, each sample in the reader is a word ID + tuple. + + :param word_idx: word dictionary + :type word_idx: dict + :param n: sliding window size + :type n: int + :return: Training reader creator + :rtype: callable + """ + return reader_creator('./simple-examples/data/ptb.train.txt', 'ngram', + word_idx, n) + + +def ngram_test(word_idx, n): + """ + ptb ngram test set creator. + + It returns a reader creator, each sample in the reader is a word ID + tuple. + + :param word_idx: word dictionary + :type word_idx: dict + :param n: sliding window size + :type n: int + :return: Test reader creator + :rtype: callable + """ + return reader_creator('./simple-examples/data/ptb.valid.txt', 'ngram', + word_idx, n) + + +def seq_train(word_idx): + """ + ptb sequence type training set creator. + + It returns a reader creator, each sample in the reader is a word ID + pair. + + :param word_idx: word dictionary + :type word_idx: dict + :return: Test reader creator + :rtype: callable + """ + return reader_creator('./simple-examples/data/ptb.train.txt', 'seq', + word_idx) + + +def seq_test(word_idx): + """ + ptb sequence type test set creator. + + It returns a reader creator, each sample in the reader is a word ID + pair. + + :param word_idx: word dictionary + :type word_idx: dict + :return: Test reader creator + :rtype: callable + """ + return reader_creator('./simple-examples/data/ptb.valid.txt', 'seq', + word_idx) + + +def fetch(): + paddle.v2.dataset.common.download(URL, "imikolov", MD5) diff --git a/python/paddle/v2/dataset/tests/ptb_test.py b/python/paddle/v2/dataset/tests/ptb_test.py new file mode 100644 index 0000000000..5e584a734d --- /dev/null +++ b/python/paddle/v2/dataset/tests/ptb_test.py @@ -0,0 +1,53 @@ +import paddle.v2.dataset.ptb +import unittest + +WORD_DICT = paddle.v2.dataset.ptb.build_dict() + + +class TestMikolov(unittest.TestCase): + def check_reader(self, reader, n): + for l in reader(): + self.assertEqual(len(l), n) + + def test_ngram_train(self): + n = 5 + self.check_reader(paddle.v2.dataset.ptb.ngram_train(WORD_DICT, n), n) + + def test_ngram_test(self): + n = 5 + self.check_reader(paddle.v2.dataset.ptb.ngram_test(WORD_DICT, n), n) + + def test_seq_train(self): + first_line = 'aer banknote berlitz calloway centrust cluett fromstein '\ + 'gitano guterman hydro-quebec ipo kia memotec mlx nahb punts '\ + 'rake regatta rubens sim snack-food ssangyong swapo wachter' + first_line = [ + WORD_DICT.get(ch, WORD_DICT['']) + for ch in first_line.split(' ') + ] + for l in paddle.v2.dataset.ptb.seq_train(WORD_DICT)(): + read_line = l[0][1:] + break + + self.assertEqual(first_line, read_line) + + def test_seq_test(self): + first_line = 'consumers may want to move their telephones a little '\ + 'closer to the tv set' + first_line = [ + WORD_DICT.get(ch, WORD_DICT['']) + for ch in first_line.split(' ') + ] + for l in paddle.v2.dataset.ptb.seq_test(WORD_DICT)(): + read_line = l[0][1:] + break + + self.assertEqual(first_line, read_line) + + def test_total(self): + _, idx = zip(*WORD_DICT.items()) + self.assertEqual(sorted(idx)[-1], len(WORD_DICT) - 1) + + +if __name__ == '__main__': + unittest.main() From 286696aa2b300b63d460f117c834500e87ca407a Mon Sep 17 00:00:00 2001 From: yangyaming Date: Sat, 6 May 2017 17:33:12 +0800 Subject: [PATCH 21/88] extend imikolov instead of adding ptb --- python/paddle/v2/dataset/imikolov.py | 44 +++-- python/paddle/v2/dataset/ptb.py | 169 ------------------ .../paddle/v2/dataset/tests/imikolov_test.py | 27 +++ python/paddle/v2/dataset/tests/ptb_test.py | 53 ------ 4 files changed, 60 insertions(+), 233 deletions(-) delete mode 100644 python/paddle/v2/dataset/ptb.py delete mode 100644 python/paddle/v2/dataset/tests/ptb_test.py diff --git a/python/paddle/v2/dataset/imikolov.py b/python/paddle/v2/dataset/imikolov.py index 97b9f7d915..dd3a4552d2 100644 --- a/python/paddle/v2/dataset/imikolov.py +++ b/python/paddle/v2/dataset/imikolov.py @@ -28,6 +28,11 @@ URL = 'http://www.fit.vutbr.cz/~imikolov/rnnlm/simple-examples.tgz' MD5 = '30177ea32e27c525793142b6bf2c8e2d' +class DataType(object): + NGRAM = 1 + SEQ = 2 + + def word_count(f, word_freq=None): if word_freq is None: word_freq = collections.defaultdict(int) @@ -69,7 +74,7 @@ def build_dict(min_word_freq=50): return word_idx -def reader_creator(filename, word_idx, n): +def reader_creator(filename, word_idx, n, data_type): def reader(): with tarfile.open( paddle.v2.dataset.common.download( @@ -79,16 +84,27 @@ def reader_creator(filename, word_idx, n): UNK = word_idx[''] for l in f: - l = [''] + l.strip().split() + [''] - if len(l) >= n: + if DataType.NGRAM == data_type: + assert n > -1, 'Invalid gram length' + l = [''] + l.strip().split() + [''] + if len(l) >= n: + l = [word_idx.get(w, UNK) for w in l] + for i in range(n, len(l) + 1): + yield tuple(l[i - n:i]) + elif DataType.SEQ == data_type: + l = l.strip().split() l = [word_idx.get(w, UNK) for w in l] - for i in range(n, len(l) + 1): - yield tuple(l[i - n:i]) + src_seq = [word_idx['']] + l + trg_seq = l + [word_idx['']] + if n > 0 and len(src_seq) > n: continue + yield src_seq, trg_seq + else: + assert False, 'Unknow data type' return reader -def train(word_idx, n): +def train(word_idx, n, data_type=DataType.NGRAM): """ imikolov training set creator. @@ -97,15 +113,18 @@ def train(word_idx, n): :param word_idx: word dictionary :type word_idx: dict - :param n: sliding window size + :param n: sliding window size if type is ngram, otherwise max length of sequence :type n: int + :param data_type: data type (ngram or sequence) + :type data_type: member variable of DataType (NGRAM or SEQ) :return: Training reader creator :rtype: callable """ - return reader_creator('./simple-examples/data/ptb.train.txt', word_idx, n) + return reader_creator('./simple-examples/data/ptb.train.txt', word_idx, n, + data_type) -def test(word_idx, n): +def test(word_idx, n, data_type=DataType.NGRAM): """ imikolov test set creator. @@ -114,12 +133,15 @@ def test(word_idx, n): :param word_idx: word dictionary :type word_idx: dict - :param n: sliding window size + :param n: sliding window size if type is ngram, otherwise max length of sequence :type n: int + :param data_type: data type (ngram or sequence) + :type data_type: member variable of DataType (NGRAM or SEQ) :return: Test reader creator :rtype: callable """ - return reader_creator('./simple-examples/data/ptb.valid.txt', word_idx, n) + return reader_creator('./simple-examples/data/ptb.valid.txt', word_idx, n, + data_type) def fetch(): diff --git a/python/paddle/v2/dataset/ptb.py b/python/paddle/v2/dataset/ptb.py deleted file mode 100644 index ea68daf9c6..0000000000 --- a/python/paddle/v2/dataset/ptb.py +++ /dev/null @@ -1,169 +0,0 @@ -# Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserved -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -""" -langauge model's simple dataset. - -This module will download dataset from -http://www.fit.vutbr.cz/~imikolov/rnnlm/ and parse training set and test set -into paddle reader creators. -""" -import paddle.v2.dataset.common -import collections -import tarfile - -__all__ = ['train', 'test', 'build_dict'] - -URL = 'http://www.fit.vutbr.cz/~imikolov/rnnlm/simple-examples.tgz' -MD5 = '30177ea32e27c525793142b6bf2c8e2d' - - -def word_count(f, word_freq=None): - if word_freq is None: - word_freq = collections.defaultdict(int) - - for l in f: - for w in l.strip().split(): - word_freq[w] += 1 - word_freq[''] += 1 - word_freq[''] += 1 - - return word_freq - - -def build_dict(min_word_freq=50): - """ - Build a word dictionary from the corpus, Keys of the dictionary are words, - and values are zero-based IDs of these words. - """ - train_filename = './simple-examples/data/ptb.train.txt' - test_filename = './simple-examples/data/ptb.valid.txt' - with tarfile.open( - paddle.v2.dataset.common.download( - paddle.v2.dataset.imikolov.URL, 'imikolov', - paddle.v2.dataset.imikolov.MD5)) as tf: - trainf = tf.extractfile(train_filename) - testf = tf.extractfile(test_filename) - word_freq = word_count(testf, word_count(trainf)) - if '' in word_freq: - # remove for now, since we will set it as last index - del word_freq[''] - - word_freq = filter(lambda x: x[1] > min_word_freq, word_freq.items()) - - word_freq_sorted = sorted(word_freq, key=lambda x: (-x[1], x[0])) - words, _ = list(zip(*word_freq_sorted)) - word_idx = dict(zip(words, xrange(len(words)))) - word_idx[''] = len(words) - - return word_idx - - -def reader_creator(filename, reader_type, word_idx, n=-1): - def reader(): - with tarfile.open( - paddle.v2.dataset.common.download( - paddle.v2.dataset.imikolov.URL, 'imikolov', - paddle.v2.dataset.imikolov.MD5)) as tf: - f = tf.extractfile(filename) - - UNK = word_idx[''] - - for l in f: - if 'ngram' == reader_type: - assert n > -1, 'Invalid gram length' - l = [''] + l.strip().split() + [''] - if len(l) < n: continue - l = [word_idx.get(w, UNK) for w in l] - for i in range(n, len(l) + 1): - yield tuple(l[i - n:i]) - elif 'seq' == reader_type: - l = l.strip().split() - l = [word_idx.get(w, UNK) for w in l] - src_seq = [word_idx['']] + l - trg_seq = l + [word_idx['']] - yield src_seq, trg_seq - - return reader - - -def ngram_train(word_idx, n): - """ - ptb ngram type training set creator. - - It returns a reader creator, each sample in the reader is a word ID - tuple. - - :param word_idx: word dictionary - :type word_idx: dict - :param n: sliding window size - :type n: int - :return: Training reader creator - :rtype: callable - """ - return reader_creator('./simple-examples/data/ptb.train.txt', 'ngram', - word_idx, n) - - -def ngram_test(word_idx, n): - """ - ptb ngram test set creator. - - It returns a reader creator, each sample in the reader is a word ID - tuple. - - :param word_idx: word dictionary - :type word_idx: dict - :param n: sliding window size - :type n: int - :return: Test reader creator - :rtype: callable - """ - return reader_creator('./simple-examples/data/ptb.valid.txt', 'ngram', - word_idx, n) - - -def seq_train(word_idx): - """ - ptb sequence type training set creator. - - It returns a reader creator, each sample in the reader is a word ID - pair. - - :param word_idx: word dictionary - :type word_idx: dict - :return: Test reader creator - :rtype: callable - """ - return reader_creator('./simple-examples/data/ptb.train.txt', 'seq', - word_idx) - - -def seq_test(word_idx): - """ - ptb sequence type test set creator. - - It returns a reader creator, each sample in the reader is a word ID - pair. - - :param word_idx: word dictionary - :type word_idx: dict - :return: Test reader creator - :rtype: callable - """ - return reader_creator('./simple-examples/data/ptb.valid.txt', 'seq', - word_idx) - - -def fetch(): - paddle.v2.dataset.common.download(URL, "imikolov", MD5) diff --git a/python/paddle/v2/dataset/tests/imikolov_test.py b/python/paddle/v2/dataset/tests/imikolov_test.py index 009e55243a..4e52810e6b 100644 --- a/python/paddle/v2/dataset/tests/imikolov_test.py +++ b/python/paddle/v2/dataset/tests/imikolov_test.py @@ -13,10 +13,37 @@ class TestMikolov(unittest.TestCase): n = 5 self.check_reader(paddle.v2.dataset.imikolov.train(WORD_DICT, n), n) + first_line = 'aer banknote berlitz calloway centrust cluett fromstein '\ + 'gitano guterman hydro-quebec ipo kia memotec mlx nahb punts '\ + 'rake regatta rubens sim snack-food ssangyong swapo wachter' + first_line = [ + WORD_DICT.get(ch, WORD_DICT['']) + for ch in first_line.split(' ') + ] + for l in paddle.v2.dataset.imikolov.train( + WORD_DICT, n=-1, + data_type=paddle.v2.dataset.imikolov.DataType.SEQ)(): + read_line = l[0][1:] + break + self.assertEqual(first_line, read_line) + def test_test(self): n = 5 self.check_reader(paddle.v2.dataset.imikolov.test(WORD_DICT, n), n) + first_line = 'consumers may want to move their telephones a little '\ + 'closer to the tv set' + first_line = [ + WORD_DICT.get(ch, WORD_DICT['']) + for ch in first_line.split(' ') + ] + for l in paddle.v2.dataset.imikolov.test( + WORD_DICT, n=-1, + data_type=paddle.v2.dataset.imikolov.DataType.SEQ)(): + read_line = l[0][1:] + break + self.assertEqual(first_line, read_line) + def test_total(self): _, idx = zip(*WORD_DICT.items()) self.assertEqual(sorted(idx)[-1], len(WORD_DICT) - 1) diff --git a/python/paddle/v2/dataset/tests/ptb_test.py b/python/paddle/v2/dataset/tests/ptb_test.py deleted file mode 100644 index 5e584a734d..0000000000 --- a/python/paddle/v2/dataset/tests/ptb_test.py +++ /dev/null @@ -1,53 +0,0 @@ -import paddle.v2.dataset.ptb -import unittest - -WORD_DICT = paddle.v2.dataset.ptb.build_dict() - - -class TestMikolov(unittest.TestCase): - def check_reader(self, reader, n): - for l in reader(): - self.assertEqual(len(l), n) - - def test_ngram_train(self): - n = 5 - self.check_reader(paddle.v2.dataset.ptb.ngram_train(WORD_DICT, n), n) - - def test_ngram_test(self): - n = 5 - self.check_reader(paddle.v2.dataset.ptb.ngram_test(WORD_DICT, n), n) - - def test_seq_train(self): - first_line = 'aer banknote berlitz calloway centrust cluett fromstein '\ - 'gitano guterman hydro-quebec ipo kia memotec mlx nahb punts '\ - 'rake regatta rubens sim snack-food ssangyong swapo wachter' - first_line = [ - WORD_DICT.get(ch, WORD_DICT['']) - for ch in first_line.split(' ') - ] - for l in paddle.v2.dataset.ptb.seq_train(WORD_DICT)(): - read_line = l[0][1:] - break - - self.assertEqual(first_line, read_line) - - def test_seq_test(self): - first_line = 'consumers may want to move their telephones a little '\ - 'closer to the tv set' - first_line = [ - WORD_DICT.get(ch, WORD_DICT['']) - for ch in first_line.split(' ') - ] - for l in paddle.v2.dataset.ptb.seq_test(WORD_DICT)(): - read_line = l[0][1:] - break - - self.assertEqual(first_line, read_line) - - def test_total(self): - _, idx = zip(*WORD_DICT.items()) - self.assertEqual(sorted(idx)[-1], len(WORD_DICT) - 1) - - -if __name__ == '__main__': - unittest.main() From fa288b70afe934cafe368429b85227c67dfc5171 Mon Sep 17 00:00:00 2001 From: Yi Wang Date: Thu, 4 May 2017 10:45:33 -0700 Subject: [PATCH 22/88] Ongong work --- Dockerfile | 25 +++-- doc/howto/dev/write_docs_cn.rst | 9 +- doc/howto/dev/write_docs_en.rst | 77 +++++++++++++ paddle/scripts/docker/build.sh | 106 +++++++++++++----- paddle/scripts/tools/build_docs/build_docs.sh | 42 ++----- 5 files changed, 180 insertions(+), 79 deletions(-) create mode 100644 doc/howto/dev/write_docs_en.rst diff --git a/Dockerfile b/Dockerfile index c3ad0c9c2f..d24042d63e 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,6 +1,6 @@ # A image for building paddle binaries # Use cuda devel base image for both cpu and gpu environment -FROM nvidia/cuda:8.0-cudnn5-devel-ubuntu14.04 +FROM nvidia/cuda:8.0-cudnn5-devel-ubuntu16.04 MAINTAINER PaddlePaddle Authors ARG UBUNTU_MIRROR @@ -23,11 +23,13 @@ ENV HOME /root COPY ./paddle/scripts/docker/root/ /root/ RUN apt-get update && \ - apt-get install -y git python-pip python-dev openssh-server bison && \ - apt-get install -y wget unzip tar xz-utils bzip2 gzip coreutils && \ - apt-get install -y curl sed grep graphviz libjpeg-dev zlib1g-dev && \ - apt-get install -y python-numpy python-matplotlib gcc g++ gfortran && \ - apt-get install -y automake locales clang-format-3.8 swig doxygen && \ + apt-get install -y \ + git python-pip python-dev openssh-server bison \ + wget unzip tar xz-utils bzip2 gzip coreutils \ + curl sed grep graphviz libjpeg-dev zlib1g-dev \ + python-numpy python-matplotlib gcc g++ gfortran \ + automake locales clang-format-3.8 swig doxygen cmake \ + clang-3.8 llvm-3.8 libclang-3.8-dev && \ apt-get clean -y # git credential to skip password typing @@ -51,11 +53,12 @@ RUN pip install --upgrade pip && \ RUN apt-get install -y libssl-dev libffi-dev RUN pip install certifi urllib3[secure] -RUN curl -sSL https://cmake.org/files/v3.4/cmake-3.4.1.tar.gz | tar -xz && \ - cd cmake-3.4.1 && ./bootstrap && make -j `nproc` && make install && \ - cd .. && rm -rf cmake-3.4.1 - -VOLUME ["/woboq_out"] +# Install woboq_codebrowser to /woboq +RUN git clone https://github.com/woboq/woboq_codebrowser /woboq && \ + (cd /woboq \ + cmake -DLLVM_CONFIG_EXECUTABLE=/usr/bin/llvm-config-3.8 \ + -DCMAKE_BUILD_TYPE=Release . \ + make) # Configure OpenSSH server. c.f. https://docs.docker.com/engine/examples/running_ssh_service RUN mkdir /var/run/sshd diff --git a/doc/howto/dev/write_docs_cn.rst b/doc/howto/dev/write_docs_cn.rst index d536f53abc..5d1297d079 100644 --- a/doc/howto/dev/write_docs_cn.rst +++ b/doc/howto/dev/write_docs_cn.rst @@ -2,14 +2,13 @@ 如何贡献/修改文档 ################## -PaddlePaddle的文档包括英文文档 ``doc`` 和中文文档 ``doc_cn`` 两个部分。文档都是通过 `cmake`_ 驱动 `sphinx`_ 编译生成，生成后的文档分别存储在编译目录的 ``doc`` 和 ``doc_cn`` 两个子目录下。 +PaddlePaddle的文档文件都在 :code:`doc` 这个子目录里。源文件是 `RST `_ 格式的。 在编译PaddlePaddle源码的时候，可以选择让 `cmake`_ 调用 `sphinx `_ 从 RST 文件生成 HTML 格式的文档。 如何构建PaddlePaddle的文档 ========================== -PaddlePaddle的文档构建有直接构建和基于Docker构建两种方式，我们提供了一个构建脚本build_docs.sh来进行构建。 -PaddlePaddle文档需要准备的环境相对较复杂，所以我们推荐使用基于Docker来构建PaddlePaddle的文档。 +为了简化大家安装文档构建工具的过程，我们提供一个Docker image。 使用Docker构建PaddlePaddle的文档 @@ -74,5 +73,5 @@ PaddlePaddle文档使用 `sphinx`_ 自动生成，用户可以参考sphinx教程 -.. _cmake: https://cmake.org/ -.. _sphinx: http://www.sphinx-doc.org/en/1.4.8/ +.. _cmake: +.. _sphinx: diff --git a/doc/howto/dev/write_docs_en.rst b/doc/howto/dev/write_docs_en.rst new file mode 100644 index 0000000000..65e7edca94 --- /dev/null +++ b/doc/howto/dev/write_docs_en.rst @@ -0,0 +1,77 @@ +############### +Build Documents +############### + +Document files of PaddlePaddle are in sub-directory :code:`doc`. Source files are in `RST `_ format. We can build the document by letting `cmake`_ invoke `sphinx `_ to convert RST files into HTML files. + + +How to Build Documents +====================== + +To save the time of installing building tools, we provide a Docker image. + + +使用Docker构建PaddlePaddle的文档 +-------------------------------- + +使用Docker构建PaddlePaddle的文档，需要在系统里先安装好Docker工具包。Docker安装请参考 `Docker的官网 `_ 。安装好Docker之后可以使用源码目录下的脚本构建文档，即 + +.. code-block:: bash + + cd TO_YOUR_PADDLE_CLONE_PATH + cd paddle/scripts/tools/build_docs + bash build_docs.sh with_docker + +编译完成后，会在当前目录生成两个子目录\: + +* doc 英文文档目录 +* doc_cn 中文文档目录 + +打开浏览器访问对应目录下的index.html即可访问本地文档。 + + + +直接构建PaddlePaddle的文档 +-------------------------- + +因为PaddlePaddle的v2 api文档生成过程依赖于py_paddle Python包，用户需要首先确认py_paddle包已经安装。 + +.. code-block:: bash + + python -c "import py_paddle" + +如果提示错误，那么用户需要在本地编译安装PaddlePaddle，请参考 `源码编译文档 `_ 。 +注意，用户在首次编译安装PaddlePaddle时，请将WITH_DOC选项关闭。在编译安装正确之后，请再次确认py_paddle包已经安装，即可进行下一步操作。 + +如果提示正确，可以执行以下命令编译生成文档，即 + +.. code-block:: bash + + cd TO_YOUR_PADDLE_CLONE_PATH + cd paddle/scripts/tools/build_docs + bash build_docs.sh local + +编译完成之后，会在当前目录生成两个子目录\: + +* doc 英文文档目录 +* doc_cn 中文文档目录 + +打开浏览器访问对应目录下的index.html即可访问本地文档。 + + +如何书写PaddlePaddle的文档 +========================== + +PaddlePaddle文档使用 `sphinx`_ 自动生成，用户可以参考sphinx教程进行书写。 + +如何更新www.paddlepaddle.org文档 +================================ + +开发者给PaddlePaddle代码增加的注释以PR的形式提交到github中，提交方式可参见 `贡献文档 `_ 。 +目前PaddlePaddle的develop分支的文档是自动触发更新的，用户可以分别查看最新的 `中文文档 `_ 和 +`英文文档 `_ 。 + + + +.. _cmake: https://cmake.org/ +.. _sphinx: http://www.sphinx-doc.org/en/1.4.8/ diff --git a/paddle/scripts/docker/build.sh b/paddle/scripts/docker/build.sh index bc8eef4ea8..3b458b8864 100644 --- a/paddle/scripts/docker/build.sh +++ b/paddle/scripts/docker/build.sh @@ -4,9 +4,9 @@ set -xe # Set BASE_IMAGE according to env variables if [ ${WITH_GPU} == "ON" ]; then - BASE_IMAGE="nvidia/cuda:8.0-cudnn5-runtime-ubuntu14.04" + BASE_IMAGE="nvidia/cuda:8.0-cudnn5-runtime-ubuntu16.04" else - BASE_IMAGE="ubuntu:14.04" + BASE_IMAGE="ubuntu:16.04" fi DOCKERFILE_GPU_ENV="" @@ -22,6 +22,20 @@ cd /paddle/build # build script will not fail if *.deb does not exist rm *.deb 2>/dev/null || true +cat < /paddle/build/Dockerfile <]" - echo "This script generates doc and doc_cn in the script's directory." - echo "These are common commands used in various situations:" - echo " with_docker build doc and doc_cn with docker" - echo " local build doc and doc_cn locally" -} - - -case "$1" in - "with_docker") - docker run --rm -v $PWD/../../../../:/paddle \ - -e "WITH_GPU=OFF" -e "WITH_AVX=ON" -e "WITH_DOC=ON" paddledev/paddle:dev - ;; - "local") - mkdir -p doc - mkdir -p doc_cn - PADDLE_SOURCE_DIR=$PWD/../../../../ - mkdir -p $PADDLE_SOURCE_DIR/build_doc - pushd $PADDLE_SOURCE_DIR/build_doc - cmake .. -DWITH_DOC=ON - make paddle_docs paddle_docs_cn - popd - cp -r $PADDLE_SOURCE_DIR/build_doc/doc/en/html/* doc - cp -r $PADDLE_SOURCE_DIR/build_doc/doc/cn/html/* doc_cn - rm -rf $PADDLE_SOURCE_DIR/build_doc - ;; - "--help") - usage - ;; - *) - usage - ;; -esac +docker run --rm \ + -v $(git rev-parse --show-toplevel):/paddle \ + -e "WITH_GPU=OFF" \ + -e "WITH_AVX=ON" \ + -e "WITH_DOC=ON" \ + -e "WOBOQ=ON" \ + ${1:-"paddledev/paddle:dev"} From 6066b36e1d3a1ff7afb3bfe0295ec6c3b6dd1af7 Mon Sep 17 00:00:00 2001 From: Yi Wang Date: Sun, 7 May 2017 05:42:47 -0700 Subject: [PATCH 23/88] Update base image --- paddle/api/CMakeLists.txt | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/paddle/api/CMakeLists.txt b/paddle/api/CMakeLists.txt index 4d0dacae90..f2c56a4a36 100644 --- a/paddle/api/CMakeLists.txt +++ b/paddle/api/CMakeLists.txt @@ -92,7 +92,9 @@ add_custom_command(OUTPUT ${PROJ_ROOT}/paddle/py_paddle/_swig_paddle.so # TODO(yuyang18) : make wheel name calculated by cmake add_custom_target(python_api_wheel ALL DEPENDS ${PROJ_ROOT}/paddle/py_paddle/_swig_paddle.so) -install(DIRECTORY ${PROJ_ROOT}/paddle/dist/ DESTINATION opt/paddle/share/wheels) +install(DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/dist/ + DESTINATION opt/paddle/share/wheels +) if(WITH_TESTING) IF(NOT PY_PIP_FOUND) From e84e69e1416c33db368fcace76d7db1791d6264b Mon Sep 17 00:00:00 2001 From: Yi Wang Date: Sun, 7 May 2017 07:27:32 -0700 Subject: [PATCH 24/88] Update Dockerfile and build.sh --- paddle/api/CMakeLists.txt | 2 +- paddle/scripts/docker/build.sh | 11 ----------- 2 files changed, 1 insertion(+), 12 deletions(-) diff --git a/paddle/api/CMakeLists.txt b/paddle/api/CMakeLists.txt index f2c56a4a36..1cec77c0ca 100644 --- a/paddle/api/CMakeLists.txt +++ b/paddle/api/CMakeLists.txt @@ -92,7 +92,7 @@ add_custom_command(OUTPUT ${PROJ_ROOT}/paddle/py_paddle/_swig_paddle.so # TODO(yuyang18) : make wheel name calculated by cmake add_custom_target(python_api_wheel ALL DEPENDS ${PROJ_ROOT}/paddle/py_paddle/_swig_paddle.so) -install(DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/dist/ +install(DIRECTORY ${CMAKE_SOURCE_DIR}/paddle/dist/ DESTINATION opt/paddle/share/wheels ) diff --git a/paddle/scripts/docker/build.sh b/paddle/scripts/docker/build.sh index 3b458b8864..9739ec9555 100644 --- a/paddle/scripts/docker/build.sh +++ b/paddle/scripts/docker/build.sh @@ -128,17 +128,6 @@ Generate /paddle/build/Dockerfile ... ======================================== EOF -if [ ${WITH_GPU} == "ON" ]; then - BASE_IMAGE="nvidia/cuda:8.0-cudnn5-runtime-ubuntu14.04" - # additional packages to install when building gpu images - GPU_DOCKER_PKG="python-pip python-dev" -else - BASE_IMAGE="python:2.7.13-slim" - # FIXME: Python base image uses different python version than WITH_GPU - # need to change PYTHONHOME to /usr/local when using python base image - CPU_DOCKER_PYTHON_HOME_ENV="ENV PYTHONHOME /usr/local" -fi - cat > /paddle/build/Dockerfile < From e7ae0ecd69831f1cd7560fa16e057eca96551f8e Mon Sep 17 00:00:00 2001 From: qijun Date: Mon, 8 May 2017 15:34:22 +0800 Subject: [PATCH 25/88] update some links --- doc/getstarted/concepts/use_concepts_cn.rst | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/doc/getstarted/concepts/use_concepts_cn.rst b/doc/getstarted/concepts/use_concepts_cn.rst index ea0b78f3c3..e63ca11102 100644 --- a/doc/getstarted/concepts/use_concepts_cn.rst +++ b/doc/getstarted/concepts/use_concepts_cn.rst @@ -2,9 +2,9 @@ 基本使用概念 ############ -PaddlePaddle是源于百度的一个深度学习平台。PaddlePaddle为深度学习研究人员提供了丰富的API，可以轻松的完成神经网络配置，模型训练等任务。 +PaddlePaddle是源于百度的一个深度学习平台。PaddlePaddle为深度学习研究人员提供了丰富的API，可以轻松地完成神经网络配置，模型训练等任务。 这里将介绍PaddlePaddle的基本使用概念，并且展示了如何利用PaddlePaddle来解决一个经典的线性回归问题。 -在使用该文档之前，请参考 `安装文档 `_ 完成PaddlePaddle的安装。 +在使用该文档之前，请参考 `安装文档 <../build_and_install/index_cn.html>`_ 完成PaddlePaddle的安装。 配置网络 @@ -102,7 +102,7 @@ PaddlePaddle支持不同类型的输入数据，主要包括四种类型，和 parameters=parameters, update_equation=optimizer) -其中，trianer接收三个参数，包括神经网络拓扑结构，神经网络参数以及迭代方程。 +其中，trainer接收三个参数，包括神经网络拓扑结构、神经网络参数以及迭代方程。 在搭建神经网络的过程中，我们仅仅对神经网络的输入进行了描述。而trainer需要读取训练数据进行训练，PaddlePaddle中通过reader来加载数据。 @@ -137,6 +137,7 @@ PaddlePaddle支持不同类型的输入数据，主要包括四种类型，和 event_handler=event_handler, num_passes=100) +关于PaddlePaddle的更多使用方法请参考 `进阶指南 <../../howto/index_cn.html>`_。 线性回归完整示例 ============== @@ -146,4 +147,4 @@ PaddlePaddle支持不同类型的输入数据，主要包括四种类型，和 .. literalinclude:: src/train.py :linenos: -有关线性回归的实际应用，可以参考Paddle book的 `第一章节 `_ \ No newline at end of file +有关线性回归的实际应用，可以参考PaddlePaddle book的 `第一章节 `_。 \ No newline at end of file From 191a3268da8c22bab5bbdeced5e684fefcf7c8d3 Mon Sep 17 00:00:00 2001 From: Helin Wang Date: Mon, 8 May 2017 17:42:10 -0700 Subject: [PATCH 26/88] fix according to comments --- doc/design/cluster_train/data_dispatch.md | 4 ++-- doc/design/cluster_train/master_process.md | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/design/cluster_train/data_dispatch.md b/doc/design/cluster_train/data_dispatch.md index f60c3b843d..241902cca4 100644 --- a/doc/design/cluster_train/data_dispatch.md +++ b/doc/design/cluster_train/data_dispatch.md @@ -23,8 +23,8 @@ 在数据集可以被训练之前，文件需要预先被转换成PaddlePaddle集群内部的存储格式（RecordIO）。我们提供两个转换方式： -- 提供给用户本地转换的库，用户可以编写程序完成转换。 -- 用户可以上传自己的数据集，在集群运行MapReduce job完成转换。 +1. 用户在本地转换好再上传 +1. 用户上传数据后，在机群上运行转换程序 转换生成的文件名会是以下格式： diff --git a/doc/design/cluster_train/master_process.md b/doc/design/cluster_train/master_process.md index 949811b4f7..e0be8df634 100644 --- a/doc/design/cluster_train/master_process.md +++ b/doc/design/cluster_train/master_process.md @@ -1,12 +1,12 @@ # Design Doc: Master Process -For an overview of master process' role, please refer to [distributed training design doc](./README.md). In this design doc we will discuss the master process in more details. The master will be implemented in [golang](https://golang.org/). +For an overview of master process' role, please refer to [distributed training design doc](./README.md). In this design doc we will discuss the master process in more details. The master will be implemented in [Go](https://golang.org/). ## Dataset -A dataset is represented by a list of files in *RecordIO* format on the distributed filesystem, each RecordIO file consists of multiple *blocks*, and each block has multiple data instances. +A dataset is a list of files in *RecordIO* format. A RecordIO file consists of chunks, whereas each chunk consists some records. ## Task Queue @@ -14,7 +14,7 @@ As mentioned in [distributed training design doc](./README.md), a *task* is a da ### Task Queue Creation -1. Each trainer will make an RPC call (using [golang rpc](https://golang.org/pkg/net/rpc/)) to the master process, telling it the RecordIO files representing the dataset specified by the user. Since every trainer will tell the master process the same dataset, only the first RPC call will be honored. +1. Each trainer will make an RPC call (using Go's [rpc](https://golang.org/pkg/net/rpc/) package) to the master process, telling it the RecordIO files representing the dataset specified by the user. Since every trainer will tell the master process the same dataset, only the first RPC call will be honored. The RPC interface is: ```go From adb6d43e0bbcc75c1a051bd4a64fc66779d79454 Mon Sep 17 00:00:00 2001 From: Helin Wang Date: Mon, 8 May 2017 18:34:50 -0700 Subject: [PATCH 27/88] fix according to comments --- doc/design/cluster_train/master_process.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/design/cluster_train/master_process.md b/doc/design/cluster_train/master_process.md index e0be8df634..d9c6655954 100644 --- a/doc/design/cluster_train/master_process.md +++ b/doc/design/cluster_train/master_process.md @@ -87,3 +87,5 @@ During the RPC call the master will do the following: ### Task Retry Logic When a task is dispatched to the trainer, the master will schedule a function for execution after the timeout duration (based on the moving average of task completion time). If the task entry in still in the pending queue, its timeout counter will increase by one, and the task will be moved to todo queue. If the timeout counter is above the threshold, the master will log the error and discard the task. + +Please note that since a timed out task could be completed after it has been dispatched for retry, so it is possible for a task to be processed multiple times. We do not try to prevent it from happening since it's fine to train on the same task multiple times due to the stochastic nature of the stochastic gradient decent algorithm. From a6f248f5476f0a5ce38bbdd2e85276d929d3f55c Mon Sep 17 00:00:00 2001 From: Helin Wang Date: Tue, 9 May 2017 13:54:57 -0700 Subject: [PATCH 28/88] change master process to master server process --- doc/design/cluster_train/README.md | 26 +++++++++++----------- doc/design/cluster_train/master_process.md | 10 ++++----- 2 files changed, 18 insertions(+), 18 deletions(-) diff --git a/doc/design/cluster_train/README.md b/doc/design/cluster_train/README.md index b88a8f382b..74961f8005 100644 --- a/doc/design/cluster_train/README.md +++ b/doc/design/cluster_train/README.md @@ -15,7 +15,7 @@ This poses technical challenges to PaddlePaddle: A training job will be created once user asks Paddle cloud to train a model. The training job is made up of different processes that collaboratively consume data and produce a trained model. There are three kinds of processes: -1. the *master process*, which dispatches tasks to +1. the *master server process*, which dispatches tasks to 1. one or more *trainer processes*, which run distributed training and synchronize gradients/models via 1. one or more *parameter server processes*, where each holds a shard of the global model, and receive the uploaded gradients from every *trainer process*, so they can run the optimize functions to update their parameters. @@ -27,9 +27,9 @@ By coordinating these processes, PaddlePaddle supports use both Synchronize Stoc When training with sync SGD, parameter servers wait for all trainers to finish gradients update and then send the updated parameters to trainers, training can not proceed until the trainer received the updated parameters. This creates a synchronization point between trainers. When training with async SGD, each trainer upload gradient and download new parameters individually, without the synchronization with other trainers. Using asyc SGD will be faster in terms of time per pass, but have more noise in gradient since trainers are likely to have a stale model. -### Master Process +### Master Server Process -The master process will: +The master server process will: - Partition a dataset into [tasks](#task) and dispatch tasks to trainers. - Keep track of training progress on the dataset with [task queue](#task-queue). A training job will iterate on the dataset for a full pass until it goes into next pass. @@ -41,11 +41,11 @@ A task is a data shard to be trained. The total number of tasks will be much big #### Task Queue -The master process has three task queues to track training progress. As illustrated in the graph below, Job A and Job B both have one master process. Each master process has three task queues. +The master server has three task queues to track training progress. As illustrated in the graph below, Job A and Job B both have one master server. Each master server process has three task queues. -- The todo queue holds tasks to be dispatched. When a job starts, the master process fills in the todo queue with all tasks. +- The todo queue holds tasks to be dispatched. When a job starts, the master server fills in the todo queue with all tasks. - The pending queue holds tasks that are currently training by trainers. - the done queue holds tasks that are already trained. @@ -54,10 +54,10 @@ The life cycle of a single task is illustrated below: 1. When a new pass of training starts, all tasks will be placed in the todo queue. -1. The master process will dispatch few tasks to each trainer at a time, puts them in the pending queue and waits for completion. -1. The trainer will work on its tasks and tell the master process once a task is completed. The master process will dispatch a new task to that trainer. -1. If a task timeout. the master process will move it back to the todo queue. The timeout count will increase by one. If the timeout count is above a threshold, the task is likely to cause a trainer to crash, so it will be discarded. -1. The master process will move completed task to the done queue. When the todo queue is empty, the master process will start a new pass by moving all tasks in the done queue to todo queue and reset the timeout counter of all tasks to zero. +1. The master server will dispatch few tasks to each trainer at a time, puts them in the pending queue and waits for completion. +1. The trainer will work on its tasks and tell the master server once a task is completed. The master server will dispatch a new task to that trainer. +1. If a task timeout. the master server will move it back to the todo queue. The timeout count will increase by one. If the timeout count is above a threshold, the task is likely to cause a trainer to crash, so it will be discarded. +1. The master server will move completed task to the done queue. When the todo queue is empty, the master server will start a new pass by moving all tasks in the done queue to todo queue and reset the timeout counter of all tasks to zero. ### Trainer Process @@ -93,7 +93,7 @@ The communication pattern between the trainers and the parameter servers depends ## Fault Tolerant -The training job will pause if the master processes is dead, or any of the parameter server process is dead. They will be started by [Kubernetes](https://kubernetes.io/) and recover in few minutes. Please refer to [fault recovery](#fault-recovery). +The training job will pause if the master server processes is dead, or any of the parameter server process is dead. They will be started by [Kubernetes](https://kubernetes.io/) and recover in few minutes. Please refer to [fault recovery](#fault-recovery). The training job will continue to make progress if there is at least one training process running. The strategy depends on the type of optimization algorithm: @@ -113,7 +113,7 @@ Now we will introduce how each process recovers from a failure, the graph below -### Master Process +### Master Server Process When the master is started by the Kubernetes, it executes the following steps at startup: @@ -122,7 +122,7 @@ When the master is started by the Kubernetes, it executes the following steps at 1. Watches the trainer prefix keys `/trainer/` on etcd to find the live trainers. 1. Starts dispatching the tasks to the trainers, and updates task queue using an etcd transaction to ensure lock is held during the update. -When the master process is dead for any reason, Kubernetes will restart it. It will be online again with all states recovered from etcd in few minutes. +When the master server process is dead for any reason, Kubernetes will restart it. It will be online again with all states recovered from etcd in few minutes. ### Trainer Process @@ -132,7 +132,7 @@ When the trainer is started by the Kubernetes, it executes the following steps a 1. Generates a unique ID, and sets key `/trainer/` with its contact address as value. The key will be deleted when the lease expires, so the master will be aware of the trainer being online and offline. 1. Waits for tasks from the master to start training. -If trainer's etcd lease expires, it will try set key `/trainer/` again so that the master process can discover the trainer again. +If trainer's etcd lease expires, it will try set key `/trainer/` again so that the master server can discover the trainer again. When a trainer fails, Kuberentes would try to restart it. The recovered trainer would fetch tasks from the TODO queue and go on training. diff --git a/doc/design/cluster_train/master_process.md b/doc/design/cluster_train/master_process.md index d9c6655954..2334f51390 100644 --- a/doc/design/cluster_train/master_process.md +++ b/doc/design/cluster_train/master_process.md @@ -1,6 +1,6 @@ -# Design Doc: Master Process +# Design Doc: Master Server -For an overview of master process' role, please refer to [distributed training design doc](./README.md). In this design doc we will discuss the master process in more details. The master will be implemented in [Go](https://golang.org/). +For an overview of master server' role, please refer to [distributed training design doc](./README.md). In this design doc we will discuss the master server in more details. The master will be implemented in [Go](https://golang.org/). ## Dataset @@ -10,18 +10,18 @@ A dataset is a list of files in *RecordIO* format. A RecordIO file consists of c ## Task Queue -As mentioned in [distributed training design doc](./README.md), a *task* is a data shard that the master process assigns to the trainer process to train on. A task consists of one or multiple *blocks* from one or multiple files. The master process maintains *task queues* to track the training progress. +As mentioned in [distributed training design doc](./README.md), a *task* is a data shard that the master server assigns to the trainer process to train on. A task consists of one or multiple *blocks* from one or multiple files. The master server maintains *task queues* to track the training progress. ### Task Queue Creation -1. Each trainer will make an RPC call (using Go's [rpc](https://golang.org/pkg/net/rpc/) package) to the master process, telling it the RecordIO files representing the dataset specified by the user. Since every trainer will tell the master process the same dataset, only the first RPC call will be honored. +1. Each trainer will make an RPC call (using Go's [rpc](https://golang.org/pkg/net/rpc/) package) to the master server, telling it the RecordIO files representing the dataset specified by the user. Since every trainer will tell the master server the same dataset, only the first RPC call will be honored. The RPC interface is: ```go func (m *RPCServer) ReportDataset(Paths []string, dummy *int) error { } ``` -1. The master process will scan through each RecordIO file to generate the *block index* and know how many blocks does each file have. A block can be referenced by the file path and the index of the block within the file. The block index is in memory data structure that enables fast access to each block, and the index of the block with the file is an integer start from 0, representing the n-th block within the file. +1. The master server will scan through each RecordIO file to generate the *block index* and know how many blocks does each file have. A block can be referenced by the file path and the index of the block within the file. The block index is in memory data structure that enables fast access to each block, and the index of the block with the file is an integer start from 0, representing the n-th block within the file. The definition of the block is: ```go From 7a78e02d5299a6d7ecdac43348d03aa786e80e55 Mon Sep 17 00:00:00 2001 From: Helin Wang Date: Tue, 9 May 2017 14:04:17 -0700 Subject: [PATCH 29/88] file rename --- .../cluster_train/{master_process.md => master_server.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename doc/design/cluster_train/{master_process.md => master_server.md} (94%) diff --git a/doc/design/cluster_train/master_process.md b/doc/design/cluster_train/master_server.md similarity index 94% rename from doc/design/cluster_train/master_process.md rename to doc/design/cluster_train/master_server.md index 2334f51390..bb83076525 100644 --- a/doc/design/cluster_train/master_process.md +++ b/doc/design/cluster_train/master_server.md @@ -1,6 +1,6 @@ # Design Doc: Master Server -For an overview of master server' role, please refer to [distributed training design doc](./README.md). In this design doc we will discuss the master server in more details. The master will be implemented in [Go](https://golang.org/). +For an overview of master server's role, please refer to [distributed training design doc](./README.md). In this design doc we will discuss the master server in more details. The master will be implemented in [Go](https://golang.org/). ## Dataset From 50602e6afc70c00bba16bbbca7a2142da6b1e4a7 Mon Sep 17 00:00:00 2001 From: Helin Wang Date: Tue, 9 May 2017 17:37:46 -0700 Subject: [PATCH 30/88] Trainer Communication Library design doc, the Go interface part --- doc/design/cluster_train/trainer.md | 76 +++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 doc/design/cluster_train/trainer.md diff --git a/doc/design/cluster_train/trainer.md b/doc/design/cluster_train/trainer.md new file mode 100644 index 0000000000..638cfb179b --- /dev/null +++ b/doc/design/cluster_train/trainer.md @@ -0,0 +1,76 @@ +# Design Doc: Trainer Communication Library + +For an overview of trainer's role, please refer to [distributed training design doc](README.md). In this design doc, we will discuss the trainer's communication library, which will manage communication with parameter servers and the [master server](master_server.md). The library will be implemented in [Go](https://golang.org/) and made available as a static or dynamic library with a C header file. + +## Go Interface + +The Go interface is the basic abstraction of communications with the master server and parameter servers. We will add another layer on top (add retry logic, polish interface with C idiom) before exposing the library with a [C interface](#c-interface). + +```go +// MasterClient is the client to the master server. +type MasterClient struct {} + +// GetTask gets a new task by telling the master server the finished task. +// Use nil as the finished task when getting the task for the first time. +func (*MasterClient) GetTask(finished master.Task) (master.Task, error) + +// ElementType is the type of elements of a Parameter. +type ElementType int + +// Different element types. +const ( + Int32 ElementType = iota + UInt32 + Int64 + UInt64 + Float32 + Float64 +) + +// Parameter is a piece data to sync with the parameter server. +type Parameter struct { + Name string + ElementType ElementType + Buffer []byte +} + +// Gradient is the gradient of the parameter. +type Gradient Parameter + +// PServerClient is the client to parameter servers. +type PServerClient struct {} + +// UpdateRule specifies the rule for updating parameters with gradients. +type UpdateRule struct { + UpdateMethod pserver.UpdateMethod + LearningRate float32 +} + +// ParamInitChans returns a send channel for parameter initialization. +// +// ParamInitChans will be called from multiple trainers, only one trainer should +// initialize the parameters on parameter servers, other trainers will instead +// get the initialized parameters from parameter servers using GetParam. +// +// If send channel is not nil, the trainer is selected to do the initialization, +// the trainer needs to signal for finishing initializing the parameters by +// closing the send channel. +func (*PServerClient) ParamInitChan() (send chan<- Parameter, err error) + +// SendGrad sends gradients to parameter servers. +func (*PServerClient) SendGrad(method UpdateMethod, grads []Gradient) error + +// GetParam gets parameters from parameter servers. +func (*PServerClient) GetParam(names []string) ([]Parameter, error) + +// Save indicates parameters to save the parameter to the given path. +// +// Path needs to be the path to a distributed file system which is visible +// to all parameter servers. +func (*PServerClient) Save(path string) error +``` +Please see [master server design doc](master_server.md) for the definition of `master.Task`. + +## C Interface + +TODO From 0ed68da9b36bbe388d678b9a56fbcca3090381b4 Mon Sep 17 00:00:00 2001 From: Helin Wang Date: Tue, 9 May 2017 17:56:09 -0700 Subject: [PATCH 31/88] update grammar --- doc/design/cluster_train/trainer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design/cluster_train/trainer.md b/doc/design/cluster_train/trainer.md index 638cfb179b..c9c090dc98 100644 --- a/doc/design/cluster_train/trainer.md +++ b/doc/design/cluster_train/trainer.md @@ -27,7 +27,7 @@ const ( Float64 ) -// Parameter is a piece data to sync with the parameter server. +// Parameter is a piece of data to sync with the parameter server. type Parameter struct { Name string ElementType ElementType From cd978b70f4162e454a1c1bbfa67a52250469a4d4 Mon Sep 17 00:00:00 2001 From: Helin Wang Date: Tue, 9 May 2017 18:02:51 -0700 Subject: [PATCH 32/88] add SetParam --- doc/design/cluster_train/trainer.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/doc/design/cluster_train/trainer.md b/doc/design/cluster_train/trainer.md index c9c090dc98..bcb4a9c09d 100644 --- a/doc/design/cluster_train/trainer.md +++ b/doc/design/cluster_train/trainer.md @@ -57,9 +57,15 @@ type UpdateRule struct { // closing the send channel. func (*PServerClient) ParamInitChan() (send chan<- Parameter, err error) -// SendGrad sends gradients to parameter servers. +// SendGrad sends gradients to parameter servers for updating parameters. func (*PServerClient) SendGrad(method UpdateMethod, grads []Gradient) error +// SetParam sets parameters. +// +// SetParam can be used for the parameters that are not suitable for updating +// using gradients. +func (*PServerClient) SetParam(params []Paramter) error + // GetParam gets parameters from parameter servers. func (*PServerClient) GetParam(names []string) ([]Parameter, error) From 70c30efcaabb030c9c07045ad3f9cb0a133183d4 Mon Sep 17 00:00:00 2001 From: dangqingqing Date: Wed, 10 May 2017 10:03:49 +0800 Subject: [PATCH 33/88] image preprcoess module --- python/paddle/v2/__init__.py | 3 +- python/paddle/v2/image.py | 223 ++++++++++++++++++++++++++ python/paddle/v2/tests/CMakeLists.txt | 3 +- python/paddle/v2/tests/cat.jpg | Bin 0 -> 57218 bytes python/paddle/v2/tests/test_image.py | 42 +++++ python/setup.py.in | 1 + 6 files changed, 270 insertions(+), 2 deletions(-) create mode 100644 python/paddle/v2/image.py create mode 100644 python/paddle/v2/tests/cat.jpg create mode 100644 python/paddle/v2/tests/test_image.py diff --git a/python/paddle/v2/__init__.py b/python/paddle/v2/__init__.py index 7c8f6ea62f..35131594f0 100644 --- a/python/paddle/v2/__init__.py +++ b/python/paddle/v2/__init__.py @@ -32,11 +32,12 @@ import networks import py_paddle.swig_paddle as api import minibatch import plot +import image __all__ = [ 'optimizer', 'layer', 'activation', 'parameters', 'init', 'trainer', 'event', 'data_type', 'attr', 'pooling', 'data_feeder', 'dataset', 'reader', - 'topology', 'networks', 'infer', 'plot', 'evaluator' + 'topology', 'networks', 'infer', 'plot', 'evaluator', 'image' ] diff --git a/python/paddle/v2/image.py b/python/paddle/v2/image.py new file mode 100644 index 0000000000..20dcc96782 --- /dev/null +++ b/python/paddle/v2/image.py @@ -0,0 +1,223 @@ +import numpy as np +try: + import cv2 +except ImportError: + cv2 = None + +from cv2 import resize + +__all__ = [ + "load_image", "resize_short", "to_chw", "center_crop", "random_crop", + "left_right_flip", "simple_transform", "load_and_transform" +] +""" +This file contains some common interface for image preprocess. +Many users are confused about the image layout. We introduce +the image layout firstly. + +- CHW Layout + - The abbreviations: C=channel, H=Height, W=Width + - The default image layout is HWC opened by cv2 or PIL. + PaddlePaddle only support the image layout with CHW. + CHW is simply a transpose of HWC. It must transpose + the input image. + +- Color format: RGB or BGR + OpenCV use BGR color format. PIL use RGB color format. Both + formats can be used for training. But it must be noted that, + the format should be keep consistent between the training and + inference peroid. +""" + + +def load_image(file, is_color=True): + """ + Load an color or gray image from the file path. + + Example usage: + + .. code-block:: python + im = load_image('cat.jpg') + + :param file: the input image path. + :type file: string + :param is_color: If set is_color True, it will load and + return a color image. Otherwise, it will + load and return a gray image. + """ + flag = cv2.CV_LOAD_IMAGE_COLOR if is_color else \ + cv2.CV_LOAD_IMAGE_GRAYSCALE + im = cv2.imread(file, flag) + return im + + +def resize_short(im, size): + """ + Resize an image so that the length of shorter edge is size. + + Example usage: + + .. code-block:: python + im = load_image('cat.jpg') + im = resize_short(im, 256) + + :param im: the input image with HWC layout. + :type im: ndarray + :param size: the shorter edge size of image after resizing. + :type size: int + """ + assert im.shape[-1] == 1 or im.shape[-1] == 3 + h, w = im.shape[:2] + h_new, w_new = size, size + if h > w: + h_new = size * h / w + else: + w_new = size * w / h + im = resize(im, (h_new, w_new), interpolation=cv2.INTER_CUBIC) + return im + + +def to_chw(im, order=(2, 0, 1)): + """ + Transpose the input image order. The image layout is HWC format + opened by cv2 or PIL. Transposed the input image to CHW layouts + by order (2,0,1). + + Example usage: + + .. code-block:: python + im = load_image('cat.jpg') + im = resize_short(im, 256) + im = to_chw(im) + + :param im: the input image with HWC layout. + :type im: ndarray + :param order: the transposed order. + :type order: tuple|list + """ + assert len(im.shape) == len(order) + im = im.transpose(order) + return im + + +def center_crop(im, size, is_color=True): + """ + Crop the center of image with size. + + Example usage: + + .. code-block:: python + im = center_crop(im, 224) + + :param im: the input image with HWC layout. + :type im: ndarray + :param size: the cropping size + :type size: int + :param is_color: whether the image is color or not. + :type is_color: bool + """ + h, w = im.shape[:2] + h_start = (h - size) / 2 + w_start = (w - size) / 2 + h_end, w_end = h_start + size, w_start + size + if is_color: + im = im[h_start:h_end, w_start:w_end, :] + else: + im = im[h_start:h_end, w_start:w_end] + return im + + +def random_crop(im, size, is_color=True): + """ + Randomly crop input image with size. + + Example usage: + + .. code-block:: python + im = random_crop(im, 224) + + :param im: the input image with HWC layout. + :type im: ndarray + :param size: the cropping size + :type size: int + :param is_color: whether the image is color or not. + :type is_color: bool + """ + h, w = im.shape[:2] + h_start = np.random.randint(0, h - size + 1) + w_start = np.random.randint(0, w - size + 1) + h_end, w_end = h_start + size, w_start + size + if is_color: + im = im[h_start:h_end, w_start:w_end, :] + else: + im = im[h_start:h_end, w_start:w_end] + return im + + +def left_right_flip(im): + """ + Flip an image along the horizontal direction. + Return the flipped image. + + Example usage: + + .. code-block:: python + im = left_right_flip(im) + + :paam im: input image with HWC layout + :type im: ndarray + """ + if len(im.shape) == 3: + return im[:, ::-1, :] + else: + return im[:, ::-1, :] + + +def simple_transform(im, resize_size, crop_size, is_train, is_color=True): + """ + Simply data argumentation for traing. These operations includes + resizing, croping and flipping. + + :param im: The input image with HWC layout. + :type im: ndarray + :param resize_size: The shorter edge length of the resized image. + :type resize_size: int + :param crop_size: The cropping size. + :type crop_size: int + :param is_train: Whether it is training or not. + :type is_train: bool + """ + im = resize_short(im, resize_size) + if is_train: + im = random_crop(im, crop_size) + if np.random.randint(2) == 0: + im = left_right_flip(im) + else: + im = center_crop(im, crop_size) + im = to_chw(im) + + return im + + +def load_and_transform(filename, + resize_size, + crop_size, + is_train, + is_color=True): + """ + Load image from the input file `filename` and transform image for + data argumentation. Please refer the `simple_transform` interface + for the transform operation. + + :param filename: The file name of input image. + :type filename: string + :param resize_size: The shorter edge length of the resized image. + :type resize_size: int + :param crop_size: The cropping size. + :type crop_size: int + :param is_train: Whether it is training or not. + :type is_train: bool + """ + im = load_image(filename) + im = simple_transform(im, resize_size, crop_size, is_train, is_color) + return im diff --git a/python/paddle/v2/tests/CMakeLists.txt b/python/paddle/v2/tests/CMakeLists.txt index 5554a37df0..0b8c78b465 100644 --- a/python/paddle/v2/tests/CMakeLists.txt +++ b/python/paddle/v2/tests/CMakeLists.txt @@ -1 +1,2 @@ -add_python_test(test_v2_api test_data_feeder.py test_parameters.py test_layer.py test_rnn_layer.py test_topology.py) +add_python_test(test_v2_api test_data_feeder.py test_parameters.py +test_layer.py test_rnn_layer.py test_topology.py test_image) diff --git a/python/paddle/v2/tests/cat.jpg b/python/paddle/v2/tests/cat.jpg new file mode 100644 index 0000000000000000000000000000000000000000..bc1fbbd371216b9904b522ed302700c79d2e4876 GIT binary patch literal 57218 zcmbTcXH?T&^fefI??LIk1rR9#>Ai#!Nv*34S-X=ZL#@+H5Ob$9ML`|P{V`M30M4ZvzJCU& zUI7dS3rebKsVZv9DS;LL=SRrs=;#<37`d33xDwQv@{jUZ1KNlG}1tk?V4J{o#!}SAgtN?N{3JP*c3Mwi}%Il}2ufGRS zvQe=ME9g;k*tpS%1OgS~vnyyt^_%-SVY5HPl-z?7=;&|U;I?-54vtRF9-a@qynPVJ;E>R;C*cv1iAl*RscFwpIk|cH1%*Y$n93?F zuDS+aTlc1=wXMCQv#Wbx@cq#6$mrPk9Fa7?u(D?8Gzz{!@Bz*zXO)={|njw1?>OEwG3dSAiF+13O0Za z;ByTOwm4ChWV29^1G&sL$D?W+5rz7snkNFI#nfYUAyBkLzPqV#6w1I?s2?(ioJO`>i6jf|D{&5RM_ zJ#OwIiUAAW(MtKNQ2VeqUn3pSJNR$YWNun@s8Fk*2+ zmoS(skO#yh!~<33al=t#xTU~d_>&2Vg zGE*_qqSi=wkdqog%^pj~>Rv;GdWN(PZ@>=6ciTl!IZ9L@Mf&rdB1uumG5q++-D=Yt zpFlXIFHC+tRfwxi8il6MhI%cH0dN(Ou(?j^@aiWx2^1o3PPLl34a8>~zWu_3YXVXa zW7FlVj3j5h1?8$H&b8#FK&Uif^fC$sMz_GBRmviQ5U9AYvJnFmHx0}?6b>|rv@tbJ zbC#;aQi6$=XV?*7-?L=h^@b+AKwDZdlN8XslnU;eN*`E_3WW|!lTgyiwe~4}0l+m7 zm3l=Jj}aIZQJssCqhN71FbKsGY($~JPJ)C2T!?Xwh!`k$v#ci8E>VEJa&o~l2`^E) zpnN=M05Kj>Zn$0ud_MQmHsv~NkzSnKbz}-d;UUFN)U!EF$%Ft&!x6eY_@rHuitUIzmdT>SzHfN_rE zYTREK;Va2$tG8?1vaR1-+8!sSO`e#k!>)stt*F&25vm+WW?BVa*)FLJFoeeJLQ7U3COuuf$R{mDpfw4e&WUII2pL4> z0y)bKRmL1g8m4$MvMT$_ldW(XfbsK(wiyG;cvQe=;D z6{>+N8cC_uS#*xVK*@-8wB5}h_P29>L(ukIKMC}*h)B`A(k){n@h&k}Pf{I70VX}) zPKJWENqhn+zyrzYi;>El(kQNxb+k1IDx8ibfU2M1i;}^E3okK<+B7CGJrn&S^RxgU z7KU`&OJhq=Z~u z9WalXK7z3s6Dkj{BVz{g_#^b{zRDB0u~-5mHPPIH`XvV7GRH$DCq+iKT?l?DRZb;> zG1AWEwDI6>M+d^#AV!XC3e{dEq;YCBD`j;|ob3dxFZz~Bp1nJT1^5K2454-}p%SX0 zub>jH%mp(du)yl9&s7&;qen@241@22ohjQB2Nia9`hV9Tuu)skYfxp zfL+*Ky+&4As4-L(gT`3sZ=>gIV8VJejq>mXyk{H?Lq&t}5CJbo!I=Ol7?(hElSo}i zz7*b;RDN`K!~;?R!QX8(v*P3s@F^I`q7MMhfYzIvubI>hhH)#Wg;dAG*q}q?00e0U zL^=eIxaacCp)}@Qu)=({7Wp3dYBmc$)`MCgEPF_L&XalMI?OCdbJ74;K2klDytEe) z_eGBz2wP85RM!V>!gz|Q(srv$5c%Znd7;&FsQR=lo=|z70A*KE+fY$kuSyHo>vS%z zVb04XV=+UN6HV(n`!EJ`If*ceC}OfwuPRR~<;4FwofKgNPeB%GiEc;^(OawJ)Htl# zCUwm$0}T!W&^urbXsxRzPdNU>iaLoh5d#BR2U?0Wiln7sfj{xTNn?&&U(7%>W&zkD z28ao8&%jworW3#C%_#sv+rVC%q3N?=KJIfrv6U5-Mp_7v`YZ+ku)!d}Ue{DqjFpIo z=;;EV;5dEUXN@Qt0m9Y9CD%1lsK|{%@XC6|=?8PSyEw&?NVv44JPwPiV80~{Ze38W zWE}=C!xzvnAHD13p#lMO$ck%f@kPWuB6$EJ3JxUhqCH^lrA8`QOhdwwitaHCYW ziT;{ID2}skmPdHFh>T#moE3`wAgf?v<}!oeaHCtt_(&9E3dkTv!r(8av_`;n)$91M zFo?9lI3xD;c|1sS*^t(~`ochL;B{pn`ndZ9l+)`iluA>%b>fDkfUN`lW=vez%0bpS zfxIp(@)=;z!XLCJ#!3S4B z^)P~%xUpa{a8eDumk*P@$N(|^I>h>Bgjexk<~&=lA}qW*R}cK@EQ`5@-lxW^L3cP8 zoDL;GSg5i0+h~aD;6JhA|OF%#v7_z;xSk#Qby zCfX zhy`i=16-bUH2b;iWw4EL6^U|xx9o4stsYd&$)E^bjvcOtR*wPQjkc5)=c;Y7&n{cn ztsicPR#dZnYViA*wE|lE9YwOhIo7VB2vu}LHTZ$I3FG zjsq?S&C>>8kpa6)BftT*+9)`rvB8EJ#0AfErX|oZvGOQcJ;Z0|RXocD z)#lB6YLfQmMI?Yp>lmH{lB361Igb}&;7yal8nL#rpq5<}PKN^%&HAJ?Np(mz7zj(b zhcLHwol{t^+vvQpT~dR})Fj3a8TNC4YPKukRs^X-g%9hPVKk9z;ST@-<}lU8?@=tg zTrL8ASZ}qUlz3r=sD2ZbW+sJIHbpC0T{GYS-XfqfUJne&d$|i0mlvRkS>1<2-<&M9 z@Jasz+>FI}4V^ddJF~AUzshq04DwO?&*QjiZdtws+ZO$SJ0(og<}#~C#ygFHJER3} zHDF&R1%Ir;YuYOi#=KTGGps~fLmFpB-FLTxXf-`UUd@ZGX5O=DeM~C2W$ZIG;^1IV zA`ojp@cn=cTl7&o{?IAL_bU2NT_;MWvOh&^O646ROLc6;m2?lH`FZ+;_J>rvhoYi0 z8D}oNX-tN^+a&yj5DR02#cMl94N#&TCVXUZ)vx!1e72XX^vQQ}z(BN6^BL0Ov&rvwuY5oT|i!oD%}4UOelECB{R$*D+S$b+yZo>J3q4jERx~!bp{AgbT;kQLWu(&p@kI z-64WQOJ$LNxP-;tm;q4QiGkB-3G}V&oi$SH@k`k>oD!YmAtr+vG2uXLf11w}2nr_o8}RXzFqv2q~4G@lcl7hRKrrs zio8Q;Hb@C{(Zf8r59e2zVtZyjZn~LET4)*^+auWXKueC_J0_@2M3|mY-ck z*;1C))Hd?|B3B@{yl{^y({suQ*z{I?g=}t5)vVm8Vw{7g^>Ob*IWhJ^h;n!umi(D8 z^E8kHD)9o7hcak^Ae4-e-y|d>^I14KwVh_`2oIthqmO7Jv!bpd zXxjVI=~dKH2`fV=!E1=p(=L1tk73`D6?KJu&32ts#fR!#V+8t58%gb zkkFW&@#ch8YBI8n&ygs0hS256M7y_^_dbc(1X+8YQFlvyeWAtopR^sleGrx3WPsk) zzPl<(Na292N5|lxbY$Pr{tJVYaB<|qeEV9=z5c)irG>VP)GnBcGD$W&sgpj344tLl z(R_M2r|x1RSingNv-cbPq4Os)*)yQLR$=3vX6$Djl3m$>VY^WV>>y6fNO08S>gTJx z^>Z~bY`b(sPJ#5yRE~!bezhb9xD&C+BE})1lWuZxRX4Nnb~3qR{HKhhPFf!}`|Zjb zN`s6FvXM=zA$^cXEYy=_npk>%_fTtEWb|YT7E3BDZx|)a(w(xB~D@YUXXs( z_OMM)r0tV1wpBxigr(s}rM8XDV{pdOW;I!D+r<9r?ySIje9pRn*DgWQy`M7nruMo3 zp7xe6+NL6`bBtvn&Y^%Owc_#l@gu%G%gu9R0l!Y`7oD9)lS#2RcP+uIsPvnUO7iew zpzr88s*6~juaCiDY%i-M@mKo_4W1U{3Is{&VkIn5I-?aE414>Kc0C)eIvtF8!F8;$ z^z83GPa}H%gI;VNa~sb<*xC+vZF|+`6PX6F?5_wj&)n}0NsdDCeaeCbP8sSH@y=on z;;0G*b5z^r8nZg-p+ebsUZwdnW+kXdwcXf|a$fvdE#x^Hy#cJ@VY*x&nRebcK3o{V za1}WSnv$lW%pFl?qN}7BtL2RF`o^THZ=U?!;r@zH|5qLObzSIhb;y(?Z#-Bk{%-Dv z4022yQun1M>j`UQe{(x${97E+$~i)fFVHkIk9p_9c4=zR1iTkwQ|OTae4>?X)^zcr zN&fk#TH`Hzy8(~CKH-<}NqOjNGnY%-K2i`wCq z)cR8oAwKqaQ^0FJihEoWz?O2aJAP4cuJu{M`0W=HR8bUtb-8=iAGcK(ceFY@|3Kk3 z%KHUUljR{_X-JDk>Ii-_w#2xp&&e4=zkG=C`UeP! z{2%^?FGoEJOo=`m^vSrL7p=4?@L$`DC$YV8c2;?A`f{AwrGL-bNcK2se=9?W9@2NQ zThe~K6!anMlecoh8rTmIvsr~d$}r{p)qYfty4+96^45#es2#2ZNC2Ws^8W+9}^vRuquV z_9=mJKv(s^#RjI;+bQ4`Qd&0RLp6mN6WyNhDbW&`WS_>vU5ZdLL$nUL8>*_E*0zl% ze!M$c1t`b6i-yzm-hZ1pN%WA`1LbcXoxZ~GRA+pnY_#M1(_p@xTd?uE4Uux38!mX~ zNA8r?UI}tnXG(?9`|5H&MLfe^WT$2n@(-Ys|H$KWZkN~-@eD7fGx*|<+7?=>tHqee zI!wroJM5h-p5qER_fTsw`7IA0F;en|XKzm{O1RXXQXV<$GaY)U4FT7iS8kXZzKCKe z4n9-cI3_@fvwT}xVk4B=biP;8T|K(z_fi)jahw+vp#J-bO?Cc(aD)iA^j=Rq$E{Li0io?cVOn!Iu zUBxc_|?#6Si{oclK`lNAiZ$}&2e?Lm3ZrI)>i@1 z(v3q{e-f#r_Q5Lwp8({@oz3Z=ax903aWD;ZBb-_Sk*NrF}z4+2~{ z>>d)kwbf4XAxr3mL_@8sjt@Ip(XG_0eOsQ{bvLL^{#m`Vt#G7veiYOY9wL|_PbQy@!s@%~;k@_083`{<_q=~?}#fIWS$R`M+Z zp)DsUSh9Q*vj8Nj*nVMwwKUkz6y$nww1iN%;7I??cM0E;P%j!8*J-?|sJsDESu)ak z$Ss%!OsDiuP~&%}sH|Onmn}HDFq>(J*dDEF6@Is9z@}9yqV%h@)x~+rgS)_V@pp5l zV8zH;rMZ?RV|1~5j)d5JfXcT^kPSCPBcG*P3{+_HAeaYiFfLws@fJ|V zn%n$7AI*%ZXxZ6bpc+MOU#-{$1tGJR*@B>!<)pw@P-nxMaah`9pbbZYk&_InQIg5P zaEz^M$fT3_sklR;dH`|<9>4ygiI@s>DjY}hm<0moWH=a@z{AKu{IoMxxQ{rZ%&H9b z5VKWa7de4%kSDp&2Z!L?=S0B6;8%e*O4dt7*Z%3sEGAA2eyuj)R;v!IVg&Gp}f2JyllBETYRwX`R07u3lxsY z#C>KLVnmM^uFvBU_HP}CsOiGj-|3Rgc8Zk0((7-0#O+WJQy<@#kUG(;M+lL1TW1 zC;vUZwG_n9nOOEd^a{Bgo@wQl-;rLX6B*pE;}WZMKQAJ(^kY!Y+feYWwI>t4y1f(S zHe~^HW0wnA;TskJda?-xX&t#U)aAx_yp#6y$oS~!puop;y;d`eOnmB!V%K1dCZB6G zvMXmP!~X9rrbms2HB@%M4PlV2c!v_&qxz34rD#bMz42~hO7 zneYAo{w77*)#y&@+DY?lE3ZEL;Db6~3qV=wZh64p=^O}reyNZ*!kb`D zG3DZ>aC)dCJ3k)>xc5<%+sl>gv3}r8q!hkLwkAaH=)48Z9vu{0Ch?qO((My@9CsfD zpU{2C&-;l-)gu85{Wl`SlX+j}Y-InP`4+8kmmHWXIsOj-3C`nXkUtaEd9oMic&?7T z$*%PFZsuhtJgVc0;c|nsq>8klXk{GIPSlI`r-$$lYy^Bis|%75O&h)FHHjQ)yK^Jx zMb3MZubJ?iMKrp*j_bNn;|!V-jO5nJikQ3$zmol?N8YDYjNouf>X0rz1lO&07+$S@X4H z|FlLj^SzY9ppXE^%~($8_P1f_1`BGdr$o|WkST6*#8Yc~*cd5%XWTL}48N%(7i~IZ zU%DnnLt=iUGH$k2rC`LRSt>30GJfB-z#)sAY%08##OImu5)bozTFb1+R#H}!Aw7g; z67|fEi&gh)D=YfsTeHPiG8=*C5>*&_qpj*5A!x+6xj@?e?9g1Y<0slBe!BaJLn=#z z?Zkucx90+5A-P99QfyiuFukTkusl|J)%6DeLrCa&0x-ogMHfdtJ5 zA?%-3e@gOYj(|@X1IncDJ~dpmi~7s2BbU-@BRs~<5*&!sgH*_*_a~5ZMIK!Utj+z` z5r9svdRCF;Z)%(Ui|+|%EuZd<2 zCySS87}9>W@FvZXa6P`|hVgAS&D6A=qmc*WmEFwE_1VB#?XWSf;phmi{;5aGZ}*=Z z7ainViuw=r#Fhm*2WeP&a2T{XO(+=5oNfqO&$IbEE17WKXz+M2K=AiWr@-^Au}^$6 zId1qJBD-74dtukplQNi;-B0-xdt6h^vWB?4_hvd;V4hy-MCz;7ko#1rH&0|isNs_P z^KwIe=);oRVt+Yn>pCBe(cZuE#(Dl1>~a>P@^s52LgiYy8Yxqmy{s3Jbr!IH+@Z^ zt&b0{J9*hFnvNRTTP$}|<{|=EDTr*U#pDT9z@@?Cem6AW zPrJLvO6R%F_!Ud&(uHy0luk>wqH4tx7!gwURfElg6Rg>0I4>*7c|&5TDZC;RNFbpH zfsS+UVFE}IpX>|#d7$;EAjaj$X1h?+cFBN5Y_R}(V_GTtYrLv}UDd-nV4^WX36qzS zb!2&90ld!OK5)DAO=J|$Yti*up_`LKqNefI+|nvJZ78r&($hAV7hVk- zCkTLk*U^4NMIbGFeTyj*Z7`)qipH-rN(4P@R)kH}18mtY>@Bq{mPI6Lo`bG0TFZEM z-ja+OeEp0u7w!0=@t}lORQz`b+RwK1cbM!bXND9@JUz6}65{>EJ!J|tS(z8EG&t@( z_(1mN;R&6!K@fG|{vv0Y+{=d`-fb4iSUJshEqUmZp*xRCr*6DH{5ILv;zRpDMX%=& zo7F^fOx^8wTh9}1g*W9wyS?O8=$VBG1s>S9)`LU=21OMxI)q@b$I*3Sr#ymgk)@?b zy&do18L03m3y^2Td zD#t^|`kQvAUTx;n=kLa_A`c*Po7M(G!PXz96 z34A)M@P`9lTkSeMVfz-Fx2g7(MFf}fSdN`JT=ml7);Gd(H+Z|>g}DZWNQomKVu028f|u=QgPrEtQ<5mI&ip!r8)r-?_Us69G>7wwrvIyR{3dj^|;0 z4PWIba;H^Q8?a~14R-Sc@!T!#US!+Qgb$@vn;Fa*7m9oJ`O_l*VZ;`GW{w8j=7DG;V zHCgHK&L{M?0(wIZ%4uz5JDtlfOzzL?DovCGPXm4*S&I+4@K>q9%W(J7e3|-IZb~Z4 ze0q2#RUB4+3+=ZSEB)@VAKF|+nYD}6C_^Y4G)-i)E{w(sG-(m*v01=1IUmU0SV>&)JiO(cGw znSb3RCE(8Ip(-AEnV=YfqbAB*QB9m!pK^ zRO@d(Fk9W2;qMWRBp{~hYQ0h9aiF|Sael-`P-%u#XB+D85;j&0Wgz61$ouosd*KDs zN4XnDlB*GJyV?H$`}H!ik56RC;?I+n-BgHOV|xK@SNA;w%hZ6WmbRVzfnUdi{|4H= zJw0~UDAANvgiHm|?Hm!p&h29nXb8SO;B9uP;L+Ks^4MO12vvUH@gUPkz#3VW70Hui z#&6TBUi;M*IsU8;@-g$fs0o0@S_iKjHf+`Mq4jNQU|V#cU3KzzWlB+IT4{*TsXNWC zgQN?8+6xZnC=d!lc{y}NMhyZKzXq};Q*V-M|ow?a;ej7OQlV~`{iMqx-Em7_3p zL;C#$JpP#EqXbW0$2>!dF_qXl)*C4zs?}!=`5+2VSLAe|#qkO3OKlm*PHz?N$5nIT z2*x2d$nzlFNEJ)u6bKjeD7sjjv3rsAZABgkxzi{~6wp)Pp7C zZGP3} zsUoi>&8|)Gb{ge7-uw%7DbjtHkqg)xzXz0i*460)EyDCz^CRI(K?7@!nve7q1p|-bUsvr4^ zhfU6d_tgIZN|7Zo@8jjrJEb6D*FEi!exb-P?e*8olP}`g{27Bh`2Q?k>AVb4`YHpy z9$JW*v+$>;|NCG9Qy%iXsHkeGUz%egb?uJ{CC?mKoTkF1@n%ChZs1LQ?ewP%YXgc4 z*sVrf&o|V{gi1JD4n$ywf*kiIVebwtjC5Pq@(J7S6!d<$OT5%w8=a#8ZM#yi$kon>-9t5UVtctB~^_~wx$>b3YF zH+^N#Ws<(_wdZvBDC(Qo!hk;0AfUJMep$M0>7%eL!?Zb-*m=-~cFp2ZJ)6_yltp7R ziyckwulb@4)Q3R&2#6r%lNS4Yx#C@`VXMBH>hp*^aHu19#00#FOuR4 z){|O1)Ac51x*%#KRDPHfLfOMkE0}MhdJt@EDw-+nLDBK_ZwdYAJJJCyo7Vh5ZlAfQ zx8yMwr-@C5UO{KKuRtAYN#M#uGZI56*cr<+c+5EI=&ICCcgY+(j2oSk2j7oYoVWa_ zkwK$qHXe8iIN7xsjP~p-Rl`0Mr634O=^TWPtOGdn#T=N~sN~88%%!>Rdp=>$mkCpk%zb~ zz7{HTUO&A!^rI|~=Ye4J`(?`D2m8c@?tP7T*%W;;Uyl`U%EW!x2gUSfa*cTkWX5y{ z!&F{H6f!0yIeHsyzSAcOx@#G$^%9aN_U<>&Q&k1WyLR}YZnx+O3^Kl%+|z0`apvsm z@fqrNDQQxDzFSQ!F;pgUU;FF?!;>#d>{Y;>`cFGcgSwWI%Fp~0uYQv-ri>@rdK3Ub z=K}YFcGzeHTg|wZmEPt2Eujl*rkz{~)3d(mUv|p``t#>^K3RaYf18}uIXEB{%YVF$ z0T;4=nG@Gi)qxW{?z_1ARaO22sE3MsLRD7_N1vFs3`N zZ@2#kK%gWgx9sJjk7i=3F~Its?2{WhnHNLe;w{QnA?AKLOPc)eS+?xXj1-+#sI+W5 z{+wihQr1jD(wMa$EHq?cft!bnItrOT>eDg8s~Sa@PdClQp#5Be8B~b(Ize&F$*H?e z^riPJq`q%^|KRLFR#dKLFi`x_8E#2f7$TfWPdOT+6d`LwM)A*)86|%r=dno*=!W2l zK&YVx7R|~Wq{Cr1ZkDy=7nX2@?l|Fp7+KMPRE_Gjy3 z;&k1mpv}l}-CXOBu_=YpRqcvYwV$FVb#D~VSO?Z(4qlr?H5t5dmi!zT(J^*fnYfz^OrZ>TFP6ms&1){v!FlKlXJz~5 z^u71|I3_sxY3~t3{_~jsbs>XMCA&0oOJwY@jdqICBuQ@m~KUH|fQT9x25mYt?#x=^hDapm=SL z3>(}S^t3zQfwMc_o-L(V}3u(Sli;a~*eJw%#T24dp?vyIRZNSuUr%Tye|nz((OUnSh(S zHqyPqT zxu(&210x9+3@5Tb1H*xt1tzBjI{=@yIo+hR!|}m&&AD#%Jy6rYs|Nm!<>~d@reR|E zPLWf!<#r}W;+d40LXO{4kMEQ%+TCW%{&m)9I~SQB+)O;Y_0ZRpS9mz6D)CorvT4MZ z&fBJ#;SlyapNxOMK0DPu8y{@c)J^F~??3bU+ttVr%Tf7isP&=tGtkJFp%7^jwo!ys z2)A6Gwr?tYG`#e!@4H|ce@yQqhl4vG^{n-FPlR>wJEzL|e(PPL_gb7?IKMKCzndh@ zg%8gu-uVb&A?m!M^5fRKH4c@-^#{jZS)(KAXmnDj*iFDm0=9pU4KbjE-GLG2&laRY z=e*`&E`0wkK|8$nCBgM|Q@T6%JTCfum|NKnCY+90FL)olabX|)*=$3@kujE|Y|9D0 zSHV};--$k@5S+qIezksV5VT?KxVUMP35&h_P^b7=?RQ+W7A-vU zx`W!g%h_K;1T>W*wj!1MhO%0}hj%ufU+KbjHLo(GHWZu<(~jUW z8S4{mZn!uDS@iQWuZ5`um;BTWz3zBME-LdER6Gupb{PreX}&P`@X<rG!E{~5;rQVP9ekZ!8fx6L?j&0L_ULp2oISD;h!k%8z@laH|tkLD!Lt$|_ zF$<;_9W=)0LYtW<2acDo(xbftWoM0DRGz56o@CbJZ9e#Xbm@;!J{qubK1pI=Xw3KO`CL_S9$8Zu`7MbMST4=df;UOxh z8}$dt)jr`T@GK5(%{_3By_5-o2C_l;_)v+9iEJ(e1>sn`-To^WK44A{I(+;TfR zw9nM#L?s(|J_T} zcu{t8X}`7j9lYe8DPH4Fb8W|-u`D!~uB3YFL}Uu;Bayy#TFUBC?83w^To^fh~s`nLxXBa~$-fcijm0J_XLj~;D zPE18hIKHZfY%I6k+Hl{IjnPXHZ9Ww^a}K}S>F0+0fQX_?gQ9g_TEBvu?}`CR)=Hn3 zu;>+Bi&W(2pUItWZ-obf4vx{$*J_C9Y<%eH@pi+Vxi2z!q}nP*hJANr8LjZ2ordJ3 zrFwzaT8ypljV{tiFt9fNVm(IQs_5N@<*~K;TXs#cAkDW&zapa?BvOF9AM6+yHxo?x zJ2iv%GjUpNAYnI=RW*24MFfKtrhjbdK)9 zs42@s{AJi$WKQ@<@1w!`j~$~qG8?}#xh`YVUAzp-XYogxQj_sDGzsN#n^8Y?(qYsZ zsqc+89t!TQ%9}LP)Q|#@w)tJ*pJnzJPwKf%e~3{%XrV~_#nUDdrtoR$2U3bh11a`t8Ud^7KD1g$v?lM^Gn%fjRz%VoX()wfhHFW4i~I+7JG=*|p6S zn|;lfn^Xio+~__ZdEMFY^0C!}vmbwmx$$#Y@ih&8hLI9H4HZjW>B!akD?RxF@{A{j zD-m(g#61&_J!$7{doS7bv1$*@%)IRK+*bU(@?!!rj>K zh~QVywdAijhV1gZ+LO$U12pIHP~F{jGmM%0koD_00rB-kdD@P*qR<>a(brBCHE0?Vce?-uZQ+zaJLMOtE95}KOH!ZO* z6dRCEm4yJD9nS2zoDxz@-Fi?G=mJ)V;VXLT2cFBPh@yAf)q?&Wc*2 za0txAq(=8)){Rh_-&HmL0M2cpTqaBHl+(XOd#z_1)-~=uFAHMwTQyE#ugWrsdDPBe zP@U7#HRWIICiGtSpu6jj<=V%t2w@Mwx`ZI@l3uR zi{P&9?1!6;Q+yA&wq+Yotfltif)=^C_Eei4rYUW*Rw90=y_)^_KQcJX+?2t#Z(Du_ z+3g#1T8btQMo^WX1Um{~vqPE|Yn%)q&_4j#G{tU~wC@8=Gj^?2Vp94S^S22-DR*)Y z%6bnduL7brNS8#O z(P7;lrE61pq~M{K>CN_|*^%D{*`E^pQ%w-!TPR*X)4_TwQ;#eekKbr{J?*#+tyC_|p;eO#lLhG@?8?t1-{>;~!8`m=-jc`Uq_2O?8 zP7Kip#SU7-U&hPu)dtx-dp0z2W(*%J!(=~4mo5dz8p}OOq1GGB zs=sSmn00D!&gxH@3DUw7cqdL%;W$3eC#tIFUy<*9EFPNP>g(U+$n+n%0x4QgJSuB9 zV4S@(wIofedEF&HCFr^xwp%uNtyB09aQ}5GfKxMB?Q*+Cjgpb9gI^;~?{C|~ zhpVi&KG*WM5DXqVzL^W)&>BJ3YQ88$p2aAfZCrR&ouJ}7H#h$#>U=S$WpB{P6ny<| z7xR3jwZR1crZJmVcXZ0iZU3tNWgDD1Fz0ZSBBMs<@9VjaHqjma8Ug2*--6NcA5RFL zX6VNv&x}>7ce%o*#bVwy8W}#adc-qZ_l7ZuEsR>=PoNFd*YoMYuxTH^Qij4Qrw0_e z$M>f%h+2J<(eC-A0Y!&ssTA@^;-{DPYxN=e5w5YR*)y=($0|AIJ8@)JG^b;AN-_`3 zYE*esw3BwOkdH1LzSYtgv@(YAk6zW?=E^nDHhRNmf} zDR~pjD@CK}t@;n3-8QFh{^3i628|W*3O}I=zz|>xUM&3cX>h@=1hlm zCAoI@ZOqBj?f$#ZNq1j$Aj4|gNIWqj%5)=eb((nP6G(cx_>AL%tX$Th5PI z#%A?h)%C1geHhznN5`7AZA&XWxIm62u(<8oV?iPgPayVEX{#M-`*x@XJ!j+JU2LzR zBRxF6=ae0XSuMf?TTKiewLdBT0p{&I$1Uo^>>UTlw(pR=*xg-3VU5xxJrpN`HQ!(X zto5ycaTYP=i{0JKMvqm?ET=K8F1zPX6#25Ongqjt|8;D+|2w;lM-;(6fd<;Y_U}A0Z#U6i%OsWI zKphbY(=vZb=L>lg6p3k;U9n~el4k7uxbbfj@9Pb=$70S)T@xY z<}W&O!aYelx~%c7vaVqRAp(iu#+6xUh|*PkeR|Ue7gIiI#pc4*+`W-{!QfwH_Eqhj z&ul06g=w}wrW1Z8C|r`NWfHrDQy4Vd6}Ik5Y`Yz<#3YdnhVxT{ zEf=lBxun7P!J48Ee;(N=zq$EGTEgydja-KM(}$`DrIwNTLOgE^vY;m^o`Ii~(0@50 zDOJ&yWmcrMF7AXaahDD0;ERQVgZ+_777v&IvL8~ibBm_iO@H#-(6Mo5iv8&nG<{sU z)F>*^p|&UWS$~xzBo}tmX7RDzKLAg~&*Y8($DEqO(In4IgVJ~Dh`~^*HAZHxz|wps zk{XS$Crtti<@~CWMA|FSj`?p-TOYFHFU%QZ_!dE6#eRvg>}=A3s}U0H_Yl;Tuyzp} zQdi=!ryP$k z?KS6Xscd7IJfzxg;PIA0$US;?4@e%5-Ip#b7f$WK8LPp5r_Fex8_S1M z7`ivGW$M8nH78N7!@5zR~FT;UnVyH9$>4#sCRb4jD=^m!tIR*qn=eFj0!;0OHb)AthhcX)6 zjuA**L@_}v7$XZaO1rczS}$f@ulYD+AfDBPZx6=2v9A(&PHV_1RE@6%R153kgkW<# z01IM%oni!zIz_^&8%=U0#RWpixz5;}nU^#8KlhZ9B@NJ#~`Jo+AJFoC3{vnXB>WJ}nS-z)@u(Ei1s?U*~)7A6i^M%brCC+xQ3F zo02)|-Ya3?gQ=RWEI+N5kuQHaDN<#6udO9>Ny=yw8;cXqoyXHtw8~7a&XBivr(EB! zKjcsS#DASNPYr>Tg4wD5^Lo7XIkvT`(~bAi)`f*#*v{kl|CXNJ={#_|(|*#iSmQDB z4^a1v&Nh4T@L>0JNchh(x%S~9?WuO_`GiSZ%)SEO&ox_g{G!}MS-uHnM)Alm(k#T# z=4fnX<>%Nn?Ozh_3V5Qa?QB_wUz|H}XyscGYimE`opFaL$p#2s_!xirT%j`)6Y3nw z0{*76RpGC9r87N;RoecsqWGf@GW{xcpx#-hXhlGtX8#{xu{Q73Hy68n!bivV_JLAs ziMfAtaC-F#Kcj3t^Sv%l$s0V4IQz8>T%!Ua;g9vx91ljVfgbzLcg?vzf9je_o3t16 z-Wi?={62l6tdG##eVFBa9wB;k-9qe%+!86%1X9KEr|RlhE!3JY+-kO_^P+Wpq9*va zZs1-<(Yfd|dKD{KTD6gm>%XMC=l=iKmJ@Rw>%~evPQDg%tyI}X@{Bu{AA1s7? z+qW_NIQ==QyH+-725^~d`kYZ9iK`5lwa7hNulfG~dZ@`eJhI4sRNOY7Z%@*ti*sqg zM{~jS8%Amsk1R9Fr_2Yh`OQ_^Ng3MAo-zw#I3tzF>Hc$8%a1UCJK+BSpUSJeeqQFs zIl%P>pN3fa4u2u*`qd#aPm&Ze0fHABvCwrqdXrFxSjxWF+(r~&Mv*pVnL$FzX;}0Q!GAnf zWb>`y^2_;4*uXb&@3>KN(8-O`#Ko3&UhJ{_x%31Ml`V|YHL-1gD)E&Z{VA~9TBCs! z(+L-*@09-lc{%jT9mvp?Ew4stB%JlS9fy^%3^;q7UX{jt$Q6&ZsRe^yI6N4IR60aS3h+t zbPh*7xT=C_A(duEB0zJRSGLA?I{T|w?iHRRwsHwyn}^45f1&2N$=FC9Foqf9Zc(}) zr=?tjQCXlRl)C-rjZO&1b@v^rJxKkcE$3^>sg;#S`qcG5?FznYpxCJdY&+Nk-NInO=Mtxx9psgL<+2OsSZPpwR3QX3<4Rhtad*PE_sBYd?}MKoHE zfm0!>>e#74s35eamvA*y%d}#b%)-?Sw6EHoyCSK1+|-LEYdJd**)tl6n+G(k!x^g5 zNHA%}Wkj4wNMx#Beml|Ud)A6Zwk??ltvMvB23l2==9`#U)DnzTGRO$3hFnx;Vrw?S z!b)<~%+ne2)P<^BGm-{8)77}B*nHFjgI;<(;$YP%HC!5yHxz=v?bfSEYG&rEN^wF! zLg$)jYHPPO9{y?Aq>G4hMOX)lr4w;h194Rt*wQPgNNNL8zG?60Ynvi?b5LBRX zP|U?qPXdq%QPzMg$5W1#UOJdJy=FsT!lNRjV^I@KR}@r$)FG+rYCx28NWy?=tkp

{4ciM8UKAoy%x!4*r7zKKxkGg$EJ9~<_VU5>!1-lRie@?W`j7Ud` z9CY2*oyU8PhIt)E03$pr4CClWU*}fQno+lQ)*Xqe%VHUu%Kh7SQb_#A^%Q1U=UC@a zpW$^pgYW7+>4>%t8+$62Uf|WB0bQYp$jGX4A#`aMg1nRJr@!Y`ZCJgyW?!BxsUF|f zkOHf#ql|`Ajmmi+l}@OWFb{=&*xon?>N|Z!F;x}UkKH~-b5pbhW&wFt8B^;(3^t7z zbZqho!5s}tY~%$DzcP#!>$qo;{uMkdSxi#oE^w!zT1K~TGbDWKHjqA?F!k+8uBa2} z#`#-f+O4=_gPxspxs*^vL{aBPh`Dy}`iknoSKd z2T-udfOC_^Ju0lL%Q!qPPCrlatv@Uk+n&7R^Zu2cJYXv-G5LWR_dn!Tb4A6Ow=;Qd zh2v=J_!Cw%*ob)){lNMB2pInWKGjVS&od(rodFp3^!zJI1p?HL?o?!Y0LG@8ySa0A zRxj?%2PcP@anGj|>wpSKI5-*hI6Xh{+M)K4NFd}nLG}LtIjsl(097;OoUqTf1#F$n z8nMbq%{SXY8B)EEU-QK}Fv3OwiDg3n0FVGwHk;U7CT3jzB^FGpkY>f8f{{XLAy1;2!RY^GA{(m7+=&@|HS=*ku>g^35rf zo5CK56u3RocdU}^)?qF5HFz^9{3pjD$Gy?W&jQc#Sk)dtU;5g%f=*RjG#;GUu??lYfJFz2Z0020`gVWemSzWMIMgbd`^uQFZI}eiOix~0&?iZ)$RBo=t zqkN<^%rTzGJx};k(0dMZ1 z>V3^IB)YiDENdEeW;oB#cBlD)7#O44y_tqtsf^PfnbT2+FYN!?6`D^)R^9ncdByd7GD>$E{bH>8WZsi6s93Hc2@(Bob8?Vogm85!S8F<+f@oRH6P}wMkzh zpTCMpkjK2)f29U)D!J>LYqkYiVA*6#?^DGtd)8QXCapk7BC@E=*`Upps&X;rtY<4# zqGU;9OWV9vr#isOJbN?H2Hy! z-OVXD6rJf9@rqY5k!6KP-kwb}_^Xa50z<&!f_b1(ift6ojCiV&w>4%?l}ba_nnYNJ zLsi3Nsq(c(qZO)&q!r6Z_!T-nYLc}|uZpfyXB`VgsxmpI2hBWlQqhwfwK~)TgHtKu zrHaI&lnk0`oYaV<)RM^*U{mu*1_dWMtjGpxVd9M`wsx9cfYvwK-6MJ2Y4YPxGcSDC3f0%u)s&(R28ZYMJcC z#APN$+tfbw1%3MTq-$wJq1j1R zVZmkjz53I{E}>>vTmi$V=V|;pkN&k-R0;B98R!oq>FrVZIk@v9A!Gv*MtI2e6b7SA zOsqs)dvw9;`qc4&>SJZY2Fj7f4ND&7KvjuV{uksCM2iX9Lb8H!xej*c)Ou3TE;D8( zRmmH_fP;+vdsNX#>&U_9Di2Tp08!~omNL+lc@&U-`kn#pPsq_=x z-3G*s_onTmk`GnE{$iKQ*(vA)3>-mbz)pu-FAZ`dZeiS_-+AV6* zWD9%dNUEo}Y<&(ZpqgE}Sz(Nk&*lFB)~-WMQ!LMr!Icl?&Obq2P1)O{x}1Z62ewcD z0A7k)HY&+oBQhvdMDh+cw$u4%`I?3m@^ovt{ULpzae`Q{^{K7dTHVtiG>l0dh{s>^ z=}ftdu8k$XLo*`&Y(K3Q*5TxA>F7Sdha;-Drg-$MY1pmwu21@)q&Lu}8fP(P(E zodJd=W&|uFEuN*Zz#l?IJ57;%#0qhT^A0+ZkMYG^*lg_NE%$~oo_XZ{ILSX;R3&wO z#*NzOu^R*n3Xi%zZM`$ztSD2vc5+85AQeeAE$riLj-0?6M12njnBzaK*{Z&YO4*C&gC}~yOv?; z_5Qu9UG1Jp_V?=Lx{q)1sOQ+Bp^898aK!%r5znun^rVK_iF(S9h}Uxux=#v#clIcUIey$NVd!dqwiemn05wJJxaYT%VUEjsgx90QEi3wF6AXOGW^&Cx3UC z?mv*=RLg8v9%fzh(Nt&r1!(=5iw(*Uo`m)Kf!pe7t8EzJGTq5yNorwpIK!4X*n{1- zAJVln>vRMkynUAyEOy`!Ku|uFqi(1$Lvun`6K0*D+Moei4Fb&;4^vt2KTKAPQX1M8 zFrKG3s_3VdMF;6zoXA#2w37n|DBYF4K8C$r&^%;U3*tKsk`srIVZ{z z7!XGU9@NEPYUR{sL=tjXe6Pn}LsWJZAZEFe1+ge>(mHOW1DccqX^lv%(fd%VgT-0c zNv5)5nk_O&`cj&SSoEipni~c<42q4#NDmYh$e~Dr6kcl53{w?&sZ>&sNfRN+rpeNp z4h=i4W`!NdcGOP0TQy{WDhZDqRU@L~RM1XpBG?rhN>-hR7D&ex7;svnk;xRu$sEvK z%OuhRkxVa24-^>4vRaco(+g0i*12MECQ7v^=}{$2I#sgBWEE0n;MLX~)GnE(g<~6^fwYs&zT2wh-LZs;9MEdKyXSJrt8x6>untI#k;+SvFwE zg7Z>=PDts+aZzzxatgAGP(&)qQe?5)c&M0Ttyt|)$yt&_d8V3CN>??kQ9}SUy(!^w zNW;AnLK8gF5k@E{6p|_^a+;ZVs1l?N?@mK#lygYf6oIJk(>|3cU5K}57elvmo`cig zyDb(4C^H!czH6J%?H)^j>|76VUFEb$k%9oO`cbmEtRtx-C{@6!lH^sX8mY=#r%F<1 zR8@-3>N zs)a^;vIk?I{{USHgB*}VP%Li624$Nhlmlzr69^Xop4$uie!#zh@m4d$` zBmJCpswAqrSLHcn9XRD_?F!4W*Co3UdHg+UK@el|oa3=LBAl@W$N++Rqww^kG*tpbZZVF7>FJN@(xy+G zfT~gZW2fQ%6h1d#g;VtYpW#grVRrz#eF5YD0IgAXa_p=;t474~q;u_1%B-YIAAA+` zXQ7BLym-Zs3cH|LCKA~4l!C=LNBSAcHl-Oi{%7hkMrr%@~TlvvdmPTUyywd zU*}e?zD9UlfQ&eP-pBm=)@_(?JjBOuC3^B0bovlMtR<|GymX0sU}tOsRo9$$&-oRb zs-7?j+{jm_Q(E(H3UPvR7&k%d{xzGbL|Ig+<$){pP`pNu13JPQgAnQ_X8aMAJVC7 zVPb(3k{z}c9Y{bhzf#2oX#Nv8{6so?he>h6k@I_d56ITf+11uP-!M~_J(ub4$N9x` z8hnJ@Fj;qDh-Oj4Hhy9~he7nNwlTI9L?CgT_Fmu5rDIF6rCk}8vhBD9u&(EY1BDf* zYQ|>?9~mUE^d6?MH93|!#B2VroE2Vnf$RAGxE-rf%$Va_bRfuo)^or=-TXWI{U~1* zg)fHYz+iU4ZUY{yTR8fk=qka881QhzsUTx_{{UK`g~V8i6MJCio~EaeJALv#fY(ay zbFm-G35h;#+d}@Hn5wvUEXcv%yN-%IGu!Z}XSzi@R0cnw_Vp%|Nb(K`KU@LqDjb%@ za=phm3m(LjAnrovpRG1KR6&p!w`OCEew8{~p00r7pev9lk}?#MOJly@=qaLEPGYP! zq79whn*-_wDH;em6C#XpgOYzrkhu+&j41x|a(@r-6p}0K+!S(oXW)G)+zJ;E+qOV* zqqsjVe-WSNDx)lL@Gaag_lHCH)`XBdj1~ZQ!0S+$OdP?GM^-13`BHAFG@l3uZ+>R2clnPvukX!fsKO{_y_*JWzzmp_#wC_|}v%oD9}V=chGd7iIOT zOqPwLlOnOT)k7XisruH03{Em?rmtr6!gkiGO`{n^;AbGpz>pO1YJ)~qh&@5YYTj79 z1eP^MRn8kdct2Xl)Xl3RxI$Y3tSopGM7cFN;8Zgt^&{8-sNWqbLe&c&nyDKmFrXaM zbInLk%TN}hI%_7QzGl~p&t3f%6c9M-(7L4z*H2 zR^&MqRmU~FO`=Aq4M`jv=7`rOuEifTrpc2T99T6cms5!8+iM2Qn<3eOQI)BmiijMV z7OY5maZ^S)s7D5-PAb_=4<{8AoYJ|)M9;-%ZAq});i>&<27YPi-K#`^oOGlkB7si> zG?>U3VAN-mR%A?yghoeN5oi^0NuF^}>BTc{TPB7kJTb*AQkdopG5F9p9@T2sPd84x z4h=XbeF8e2W{rFbC}j;@DoPnR7_1E&NBce+Ndq5B=t6~D0bTdd>4If>AFWqn4stlD zrXA{|$U1ZQRt)GvWXL>Y=}`Ga+m!>{ig``^{KM-?p_jHt)~4m80u~&Azg*H4OksdN zqMl!9EqF_!6F!awft~7!;;zPj#t*GPg4t-RLhCU7pZ9neSg;vH^R{C+xR7D{( zw;&J+@6YQ(mdWP?bs+Ih$>c~6)|wRw7}|dkR7miw!*}XNX~75wE5YlGe=N|E?ci=+ z{2H7B4o(lgX{0NV1@5HeQZl*fFnvcuR!H5!VV^^h{V59ik_hY16gY=5QKMooFvHMx zs1+U~xdVmzXCB6{H<(W+Bh*v`6=8?idnw21{{ZW#aVebwf=iS(zWH!32&8 z91Q(&kHW5M=^Mnv@Z&jePyV%M=oc}`B*nB()%7sS^FdH0>D`{+F4~XUx zR!f%1A2V>?r2ha))s{jmnj^+T4^znh01xL`Vkc>%a!x~V3C;&>dsW%oODeLj%JHcD zNF($h*E}z&(>q-gDk{vFW0wFD2VuebedAEw%v)5k;PEVnk9Q;dDp30+A#uWw$PvwD zeW3pUY1xsS1Z;!b=LfHT->q!)iJH}$;zF0FbE_bH^am$D{bBjhs!Bhz(aSk)#~kD4 zIXV0RC+ZfcJ4)7zaJb&eFfoi0O*dMeQ5~!Z2}aAauVaosK2Ok6zT!&l5oxC0Nj9Qj zZrTS-V0Nyy*>Mncz){ft6~*YPk-DGzsuI`DXZ`6T7&tut057j! zO3JZyR@$Uzxg2NM4$J)N=B&{WcTDMRV^xflBc7~#(`FdQvLlEppQ^_FUnzbB201yxdrfF3qke2fcj@9GV+U>q^&L8M{{Ysgj0)U3@5ew+YSRoKnUF7h9MbMpIZ)63e+smS zxm<={yBI<0asGcQVFZk^+}*G`eJW>^x9|n`-ZNEM8-C+3!R^!eP{9i7fOC!B=Rc)W zkT@SJaC(wOT~&1>c_e!e@u=h^gSWS(A(`xj+YcE0Y2q}5PiZp409Pq28#g9%-n}CJ-^`4ITy_1Qmnk7>NXl{0#8I}9 z+XAMNHd1#CRcQzVn!@^=$qg(3$*S)t&sv7!Vmclwy2x18h}joB&w67V^HG9vPgA>! z3P@DRHF6`;t0xR9`{%`qZjxGIk%f zowp*Kqau`y3Rlz=tVYN&P%x`JnvK_*cQocR=3G^LRoT}fs%<`%ab;TQz6TVlywtmB z#szYtiLx>&F-pYdl}CECE8I{ikKUxHV7oaVAyCser(wMZl+#RM~8= zDdMWL8n-B_j;AyrNRPcrl#3o}RE&dG^bT=K9M$hE z3S+MnBCKZ>=cOjfu<3(LBADz&u(9H{w8%!`nF5N;jB|n3wzRtmCVcUV(vs9^S)FBs zXs9rA?^nne;;LEMn4FyMV^Jn)7?w`(+?M<*MP|lJ06ot&RzG>O>^Q5DgN?up z`*3PeM6q%P&|DtjxT{FA4ns%n$3LxCiC7%5V1L>*bS_sUllXsHxRD|jGW6WP_thXI z@Ji>~nwJVk2k{*$Bd`O4N%aScWRDpGzCRj~TqyaAkHDI&z$os_dm4?OV9bQ5?~_)9 z(V!Vv{pRPfrnpm(u6XyXnYD{sY=@I>J$U2ssx9K1GRMoAN$k>sqw}hAk5MQ~p}V{X z$C!BPGyWo!qk)AP9-XTh{6++U9t(+EyIZi31N7aKQ^Bji$8n#_W1b{Nk3Udb-`N1++P9{&JJxv1@)0t1F8rs7BPs`@;E z5<mNL(gN*Kb2`d-!|NUaIq7C{{Ve+`Tn%qm=)e2 zqiFlX^CbTODv_af0RI5fE>1Iy4nL)51#6p1SFsFi8tLDHk%9_?{5d%OMyTDu!u|jO z$C&DI=uX^zXEmiAs;rlClZAIx>CgF2xWa;Z%G5D#oDh&WxWTlwQ5C2|v!MM)8Rx2YMt+ z<)OeOh~zKRIXUT?=bo(E7Dm34=C0p4$vNGy@CTqjLDst4R%TGo_nQRcx1g?POoB93 z50o3XF&{C`I{yGKO6l#vW*%Nu7mdy7=s(7)lv$lwCdn@C8fMD24i8bBd*k}{tcW9q z8MdZdpIq)I)YjeI)Nw${#|4ft)Sqk`#)jf{n6WGde(J73`mySNPfFTp8A43lg-AeI zOS$zp1OEW6L6MG7I%hw1dB@>Y((7_O1F?j56vWTuW|+l^ZvBfsEt!C zqe+316nDwT{{UL2QyE}eV|)Jq`s)F;+ZeYoodW0Y$luHV0M%9_)*z7;i}1raJFsX; z4cyxTGCn|`&YJvn4Ufkap1OoB(cxk24OvT=vGQ$Sp=JJ6S`(CDeGs zK<({FqdHa#_esxegT+#~XWO{T_+qrCaS_KSzCr3MF6QzR{0_u_bW}*RK^*6KaHN=I7tNX+YTmxyw1rumo1^yYMS`vlfeT zAZ0a?djxws#PY0w@+_iDmu8wLXNO&Tg}S(i;5hyH8*xC z3sPk6;y$k$&I12+}5OO zxEbf@aYRr_mLYi+0b`$R)S;we3lOW=0h3U9>kz;g_a>9=MYvp(>(Eu1*t)@S&g1x0 zPO6K%ug%zh^{R{A>;$vt9kOv$C)6f}bg6X`DN}5{G|r#J$^Pf{tX*2@#;utf zrgPu$AO5Oi-OTPhypz*CaY3+ViOh#{9QDO0%b!BrRCQ;bYnsf~kUYx}`H#xd63zY< zhZd!JmdkfN#%?9%0IQ9I{(5@nu>AcirSV;iM^9;H2g!u)a5|Ho6n%PqGhASm?qo7a zjm^#gz#f2m4wcfO7{>M`&9|XtH4CrexVY<*M5G$i`$nf6YEto~1Z7c)Vt*0O>scUS zG7mKy-Oq0m#}uxp!vt=}y(seD%uUGIvGK&Vb00R{kx$-eGByHRzTJ2gr*vy5Mwy^^seRG_8`q!QKW5n+Bt)1%(kU~m&pJQEF z-30rYxxosE3rtw$uwG{{4Ldk5Q$*F9&bI+&E? z$XE=4+ZgBbHKA`hNopKFAd$yjrz81QGBz(NDV+mjDuiR->q?-4p}&ST>EE1V^%Th5 zJ4G^_=Q;Y~ts)k=X&eoO0*rS%s8I&NZG&3KiSutevFV<1RIcIQZU(}qYXOgKr~GTC zir5F=)HXjtYK`T(E-Rhm$Ga8D$1SM?^Ng2ZllW!&4@m! z$JZwnywL5Dvw^+F3I70iW~oa4TE>HsjifiOFu%_=k0GzLELr9+owk-F@t%Mjb^ic9 zm8JcpakX0q9P^xig>usB2Vv!i3xSTLo&`a7sXHqwouR%^0psiXev~aXA?kEDdZb_# z8xS`Hmcc*Q@~rgMp-_JA#Pu8xObX>DpD5QC3{o{8jz7=;0A9I0TV0y?`A`!Ly6^xz zx4AUbT$s_`-~2$fv|@?@W&BC{dJ5;SJZEu!a*(D!+gzM|Pxx08C;B=wj57hBQJT=P zz9^xwv@jU}4!wBvtyMK^h0=CL*wpXT86q&T&niPB{y^aVAC)#Q6nShK>raV*>d06V z_2e3%bkR#YG-Zk$U>aa&*73dmQ+HBxcCS-w+%zL5rALp9Nad3J)3(`vO zbI>BVE`Dd;eOLm2Ig&41vmmKQZbLTGEG5-bPO!m1ob~9W$U? zkN`Pj?~Kwtz-)|j^sMOT7#v`GQ;dL~G70U_)uT3vWt%I8WeR;y@)bhrV5bBr2NgCc zvqHhoaa;D%aacC-F~G^iYukdV25Q!XGg%uVm6Q=x<9?jglsT;vGia$Q3V;an&MNJ~ z{JE@Y+@l;CndoaxCSL^yKq_QKJmRsE=LeHklK27Evgb8)%@Yc@Y78g`097@;2OQMN zKI7J_oe<#1Gzv!oqHARXrCHt$E;CNzv7Xk7dsHuA2*xX4U>wsJQ?O`ZzM4HMvuR+S zoL5D#R8jy1CV@pW5BoVl=}ucUQ_{3pQyD<(P|XxkPstrIQlttg%zY^VIp(J2@}hM` z>?%PPYADW4T2`in)rJVb>qzOIf~Np=G#p}?Cz?G%%bKvlm=~JC9tt4<(uL-y37TIo zj+LQEgS`Z1rZHlTqNR+^YxuhhCA?brtgv~Kmik>nkTACyqKop<~QfL(H zQ$#YZ4MKWVw_U=kNSLOHv0cqZUWqa)A(BlnW4C0~sA-D4Q@T(nlWxT@y)$(tY1pyF z=vgvqCp1P5X;?+nb5d@|sE~0_98$0WCWQOcq*BCyxaOKeH+0sCO63DLrFC}93UHf6 zU}-QQP;r{*)PREn71BeAMYa8sZSk7{+y zrzBQn%QUO#Lo0!~u+f@4sjhr#aEC-;fkyuN(ZgIEQoSLyDk=ud>N$*2p zVnw;wK*`VhMy^A5BLR%#@fFTJ@M(bB>-GbNSqF&Fh9>Vx915HK3nz1Mn9qEw1gwMoaZH>O>E+fWt1*>z!*Q) zl-nqTq+z)ne7O~1tcQg2zJP820QJ&{Sd+Ye-)zLzv5H$5%1xeg;|oJJmvd^X^93i4 zK|ZzS)&Na!f-{bz(!H<579$0NDb7LL85#co>(`1aO{ME|LozgI>~WEding1JE;^UI zfD4Vvo#`K74$De>&)t?nGhAjF}Y2F6AXy z9FJpOf#LmE`%=L-ureUp!1V|G^IRq2z$Lgfr{S4e%K99!Rhxz${i$-^Hyhr^a$s+k zNdExV;DSN*&-AO1lOrT*avAaO*R43h&uqa)BqL$K>T%cdH7daH%+3J$co;e72m1d2 zFM5MEYFo1xK%+l4Fg-E<0M@N&NGbw=c7S;4(?8a;t>f7d$>kJh9CpQPTSu?}y~fk+ z$NvCaRisNcAOp%3LjEGfu({4gJ#o!wqHO?n0~HkMxer{9mAj4Aj8v@7UhJrpf;cL~ z{$L;HpTe*;Yp*5U2nAPw-FfJLophRvUo+$dWys^d9Y^7c6s+DI5Hl0=j)D#`qLst@=`}5=I!m%HELKHBs`J{+A-^2Dvx@`I?Bb^*6@v}ds${HoW2L=UV0jyidp=|#7-NESV)AOo=S)l+J!CuE5D?7NFSxD$E1Xf}ncU$S$08U`;-B ziS}n)lY@?NR^pR5!Oe3v)5zE)*43<;12ie!icqpNiQK%_KA|4pm@R9m$f}oU#Z7Ix zA}-A49!wKan}R#lo5TT+t!KB=xn`_(#oZSox;Yhk*7Q8GN6us0pTuUc)rhCVKbLy{ z0PV&PtdaF$*W5!Ju91r@DeiBqYdJL*0hYwr2}7Zqpebo<99i!#GF>D zMou;)ViZ+o1-_Lp-ZbsG6<4^nDKhQmp=PVHpv6^a@sfH}G-(!B$flE;MrsHFoGA*T_MOh$Z>^H3GRtMac}sLRa+whO?flTQ^J z^`J?MjwvxfJYtjzcP`=;ry#+lq{tLpo0F;HotlAOqz02HaeQ%4f^Cj7(x856V;MbZ zp`EUiDzXd;-BZU}4A?qN0rEQe^~?7n;Gdp&)69UKp|VtofvA81CSD=cQSe7TSJoze=RAgC-hK z2Yi|?Qb=n6x%solUA0s^JF;aZhgN@-Rm+=D>Ohf*_7$3_ut|l4wmnTDrfEcrAlriD zsl{kP8X)<$?mxmYRkY_bN1dBfIL9B(uDn}vcLm4rf-1RmGji6&_-05~a4WZ=1$q4V zt8v}L;9y{S@%}Z6hTteMgd@IlpVFd{h}>DQ-|UXG?$E5B=Tsz*IpqQVB1RAEN`@&p zGVlKCg~a@!i)t zEzyGdAbG;e&@lc$`&9Q+!E(Dm7$Z5Zt6uP(%&JUeOoWry<~i-3O5=xzFW|d)f;2Jh z$Z-5;>5Nd}8EY3}(`ol0hT7aozdWz47n5zvoMC#L^Hzt2^&PGLoUAdD6*>Ist)GQ2 zuWq)ryrM4VLz2hX4k;*Ed(3ZFqA&*Q`Mk}i%Lli5x{D-taJIs}TaGb`^qn)nQzCD0 z?m*~!W80vl)OVjZ z!Ny4(aon1U-U5uf7a(ISPY3*u>sm2F0rRoA94Z6ew1jSFCt_d;L+wnJOAnXUtg7}H z&$ThvBn8K){OhNoiO%Y_%_JwwVhQAW51m~I4%_fH15EN4uDa6uf@(RgQ|xdY?|i>smU0gOUQb+O(oc17#sh4Dfkg!=C(~rFKx;wb_LQxNg z*&f{u9;R~rNS@y2){{x!1Kg~r?R- zpDD?-vJRQgq0KS;$-pvYaay-=6Cuj&Ur;(zVzqL9f1FhE*uG;jC`Rxy3H2haqDvpk z#tmEk%SHL)-IM$|rx+PHagWC|IE_s*7()}uKSNq}-zyFQsNl6nAeSbzp|p$vf@?c5 zwp+GfK;sp$ZO~R^5QQKBO>00f2NiKfb7q81#X3M8X_1kNbMh;pY+=;H)Gfk-O6IU8 zkSQX%d&MUsHG_9(GDxhdF2`Lb+|Fx^bQKUK#@n!63=I@V(ZjGPRSO%k?q(xIyw=+V-_eoso$g6DQJYmv9OImK$* zTpW&QQfP@l>LZXfOdB15j*HhK!7Zy(Cd+@4KOdck5?!R3yBDs&(vs*Gfk zO=R8Y%45%Z$#x?7fvF>Nnq!lXN@_PWo`+rTb0l*{DpIt~rl~XLlNCDDpi>b@H9k11 zD20Y;SLLY!(iG=3(P@Iv)V&83#%u~15@IuAp)Ho7nuHlSsku>l5)PEB!KVeM1*y0; zJsO{%T8kJo@N-s}l!nPrGS!9|6%iB(vb%9n9%^?L7Ac7_gRMPi3(Y(&OJLa2YBtEJ zSW|Y;n3E$`Ak^Ebfhp>=+cdaltXr!Y1{RnUdRB#u5^n`insC*WjmtPyW4U$<@0XDRwZ$t;>SUqA}g1?N;E^?GTN}&5-v4 zfkR0&@AHv>dJdIhIT=RO{Do4CNtBsZ@l^p-U5Yxf>sgNg19t7J>PIK&D`HsZk+KF& zQ<}zh<(RkV1xtt~Lu%0(Lgxq3m=^y4>sG{3B#ZK*f!vk<02~^%I1);@=MTsp;-$FL_c3HKsq``xAC*^FriH#; zqz-eOkL6P0*=k7Hj!`NuBu&5G3<~}$#a)MhIp5zX*&jj<4l54a4x@56bA>;xZCa=? zgD;fn*LjZwenyD4f>vgNmh#8SnLWdDY6)!PowFny0gc$l<=(79k-ss_G2aq#{EbeL zV*!G>JuIS`^+#ZqLDIeCJ(m=pbu8y53O=q+)We`?Q{;vLh*nz>0M0Hhb*eH9CAP$oc{nX z=Um3VjDRx*MiJ4wa>$7KWFrkib=){08y#|{=I6Vxg4lb=nZz$gT2lf$!t|H zjiii#Mm|;P{uN$GnTX6l0pE5AAJ?^Oh{kh_nubXBp#T7Vx>n5@+{cm}kD0?292{~* zVBgwHzk0D?pEd_OhI#yPkF9mmh6(d-+wI3nrz~;1zXT4172T0>Qu1ExDx9V$~()0<&qXpf( z?Gh`aH_P)f;1a*gR}+^jcRS$=#^5w&=$08%654^RFci~89-jBCyAD52R(AxQt zMZB<<+}Xj?x!V2+BTH7&K&(MseIX|N0vnKArbe@@k8m5l9obWVVP zf2C8{p-W}=XNLf&AC_u2ymVYFtfX`Ij|bP9j&z60*=~E|Z^B;64nF9{e!n+0LTy&< zlF6u?; z)}nSdr*kG?xjAw46;Xng3NlGJ8LP9EBZ19Om0NIO)EtWEZ0K?3i%dz_#6s;Ez~t3e zl#|U_yKguNSAn^Q_5QUGe zfm*U6fyt^9l~rc!22DdqwJzpu%Wj>IWy1deQ&$moz^Xdqn#@egARpdr{(RN*BZFM@ z+H!I^@ZR*LF|tN_lTo69-iLBeCYWRk(wwY!#KlpQOShVGigi)*19tOLfyFaC(}PnT z#oJ9`QZHIY9MN*FLPlyC22EPqQMzK2V%)2?o{D`cK$$f#kBXau8jI^s#wp=2DH@%I z*up7>pPEk8$he^7Q(0*`Q-SPgY%V~{QI)F>DiS%Ra7=2gDHPx_N<~eXpn$NuG~6C(04YI z0ALDJlSrHnY5DZ7rxT$vyT`^&Rh9OPD65ilRV4vOtzkV4Vr5*Ha(5cfkvZozr#?wF zLS8<#&i6W_u@Y@&2(je*R-`s*vo3b~R#3Z&*RL(tinSi0B*q3&~Hx{=o&lF?tGJ=2E#b&cd9Dr<j)hbEu&5~-Ywz9ug0DUTFzbKW-;hBAkCfafg8fKyAM#TtAqJ-{HtA20bQ~#@Oc6?{zQXaF{Zkl@|(E$6gNwD z8S1H({(}^$19`+nZWQO1FO&UG@~bgMq=3;kKAFemnzmL9gtsy=Wf%i2$G@?;r>X2X$I#btq-pWmTgDJs&-JG% z-^X0{9+gz6sOrM4XFU--ah;%UBQqAx{^7=oM-f=mCjGSJ3RZ* zkVbpZ12$ym12szKA^{;XGti#3d>kAaWXrt>9B03!4GQ|6apLjcnbgEHksSw=~obK*>dRHA< zO}m|x>ZKbpyf8}`3=hr@9Y=HOJ!)Wqn`kI|l_Lxfw|dhSVPe4<0AL(;^dhQHY~U3H zNHf4;mS3l>OQ&Q?nrx!#th}qE^dLBI(}Psaq-I>ROSE(*7ay%^q;CHJD?2_qf(Qff zAk;S!TEvGUHa*E80r?7_W|}YBTSE%-WMHUeBeL`Qj8+}Z&&*4bn|KPqbNbfR_3e{x z21Ll`=kLOv3S#$s@ zC)aIHYpq8q#DD|rF;{d$i`>zOsN)1xo#>+y9zAnVE5|fs#OVBTGlN-^T{XLvP)0ou z_|w=*H!s|^yn_hjei_X^($NkNB-KlMnP*{q5r!EM^9#%O9tcC%xO>zjQ5&iBDtwAVp z&}3Di6mHNrLs*wsLPkw%-8lq;F<94akVqWXRSueE2^>{dk%cA9?1b=YnZHcn^{!^l zsG@bx*;xnhoc{nytm%`w z62OcKmX6DtwqvI?w{E};3g_(|xZu{bEax0m!Pw`hu4@TY4z(N3PYX1O=oHn6^KXO&OZv;byM3l&*~C+my7`7F|>V6Wm)pUvnuoDru@cQM$a)S zKIp4bLm=jxVB#nXW0pUlsaof8gv3Y9e{3FG>>5^o{c3E*bZCn!jla2V*d0{l{)hSsk|{3Xak@2Q{@hqUf#Wq+*3$7xma~RI z{pDs-e=qQ@Yc#_y=q_MAep!nt{KaVq>vTk{-j8MgUWKo`& zjK$I)E}i1JqlA^qWiKRzpF?&HZ|B)E}D>aj!e*E^}r z1c!%tTy;=b5%mCpT$K_!VA~@r&leC%dSs|y%ZyY^FJf$?06P8PL8=kYEMw%2K_{e{ zKdIub#XRHYHvRc3I(}8pYn$G~n(E~9Ap_sH@~oXg?b%P3L;8T3tMc6!f89t>{{VHc zr}L`LWRPLqhtTG!CPy+bHkNJ1IRVd5D%=8C$Y5Bn(zOMwilBj!*NmD$f%26fT9*?= zG4R|Cxfuj?+t#f4o^SyC$KI{ZbWnMV?RYo1mWoIk&|~L8a zJ*q{U1D+{a1~}rQJC}|Ksu4Ee=}AWZblU4;V>cGPfN8qkUrRpii>&Zi>G1r&4@r38TE7_7uZ zQPi5d0Ry0`S{X}ISsdb=Mpvy=S#Ubj{?MRuYh=tC9L=y2M*w8kEj*ju-}}s3x34cF z`#ghi+*I_gcwwbLdJ;4JYnu}8)KSNYyOM#k9k`C#A^br`ezhAajw!NSGv^2Js_nd< zwWaK=k4iDJu=x}aamlJ|LLZn?$R!6M_^70EMh{w+Xo|1FsE=B(SFJ$es>jTyjcq-~;l*_pvJIlSdlbnP(^;kzU~^rx zo3YbP7u4i&QUifaRUmUpYik>j$(H(6ck}|SFgWI`NZWq3gpp~L<-#vLYcAk!B%0D- zTO^(;q_O8Gpsbp?q+N_jlLVgiojbbloO{;n&~@UdOKYA;{3||IYNU>SM3Av17wx)n5Ey?t%uLB>Lt~~`DrE?hp92_s>nt6fSt~2UuL2a3jPpw#30*r0RrlL!z zVZD)d$wKGy6+Bvb7%zao-R)ZeG1Wj6;U?3F)1OmRp<-P^D@dh6Jh*`azRo{0TTuDf z`JGefNA#@afBER!kEL6NQ=G1G>~UJhLn%9#gGldzb0&QOAIh~XCcO*>-h@W4^t^@Y$UQV{{T{l)PtIxWgz_9RDYjSRAP-A7!99N z4O&Q}L6%oz?l}Exp&2sbKH-HU=nj9CSd2F8AJ5zOi` zHn1bt{{XL4C6JN#BxOFb59v}|OOSBOk^R^I02<4Zbn4m2?~s3$Ln#FC+DyMPu7BPD zkHWI9ba+@2kh>@SGmtp{0KQKn@T>Dn9l~LVf6$NXSyMc4xdmk>i59O(R&gWZhcY z+qohs&O44jPPIl!EyCn1tLRLdf2Cc#xO0#bgWNH!8SN#KLdCKP?_BLAvAk~gCz9z< z5)s&Y$_+vNr4hjLi1g(Clt*l;dh_qiOt#Um;YNKiQ0`UYh9;y{zE!~W=B=#Wcmk9S<3%GJkfJzF)k!q}(ao1oH^% z`P6S99MjvL)f1^8jC~CV2&8bsKdn`gP0Ce|IN;NsMumuF;P({-_;uWeBhrG>GUbR8 z@g7vyKYe7;Ny~0u;bGpjFXm{8bR-aZ8iL;T3-ZYL^FN7CM)$5di;?ND({x`l);&z9 z?YHrtrC5m`E(yr=G+kS@teF`xlfm?rT|9-FV|Y4nHcpVHp~HXBaQ@etkbm=1ZQd?2amb zysqT>5VDw$12r2G-npyJgbqzWRH(=wjcnuC^puK{BV&rEDP?X}uE@lm4M4*=$<0eN zMP1pbo2@iTDOH0S-Nwl9oCgJ`W4$3@yii*;nVNwn{0vdLq1}_kNUBJr;u47}H4c3# zm^q|wl@gAG@{BywPZVdGTvkz9fVeHPOmz za25b0O&QPZ+(3!0VF5iAOZOST;-}WN&p3z^ilp5(Az^KLx$@28J1*)?_UTL!It+QA zDD>izE46;iGt;$Br5}@2gfcnUj0IvpomdE13;?Io9Moj42@XPztPLt$CU${RX8ztlS9kJJ*wcJjz$^e&<*b2whETEd%)DOHdo%j`< zOQm#aO{?m8$NVMo>Junc*r<8!o;^Evt1GQG(%bft&PQtD{AHm{q}(*t*Do625w*6Q z)g24OlFGu;{{Rfq|o~1a!t622-JW40E%1ki8G;V_+k6+4&H95J< z<&JXM;l84{S?~6IirZJ(tXgV}gzj&UbCPPKTU#>T9VK*jGhIRk=4x)A;xTJ<_U|_E z!zelv+XLxdZGGZf zmybi*p`U7@DT2eMYBKIu1ExDs8h31C)OM;7og^~c0vbm87_WneTqqa_o=G4T9u>#@6q`6bN>L=tD2pZ z02YolUilnXH8suLE&Gd^P!C5x%cXP4wYj26l1btV5J*i5#Gm$?Kgd)++mOq_@b;K9 z-w*yYtEpZ$6Mdd>{{T&`(Z5_~t;H-ZM*S|t`>L{kCaAspeZ^YECAfk{{{WCKoVoN8 z5B-|0!XaWeX;y>KhLisQimeD@v6BI;UO#x|Kh#yo?JUpTl4L%TAS3ezj#uO^Vzdbc zL47t1dWj?-)~}>?rx>tB_YPX6iq9YF75aMAVY;g*Bh*$+wYj3O@7*x@b5TCn`P|t50CzMp z;N*0m9E={7De5w`{!5N!QJ+w2J>mczvqnMx0BaRu_Acsq^sHIr*iLirO}lh7itItC zS%JwNlk=wX&(62SMGi39~HH&to8*-eA!fi9AntK}dlN-xzlf=sS z!z&DCqH8}Oh6L>=gXvi|((RR97=|9Ex)?6EW1Xx==Ugw`dTesZPYEt%@V~&W5wX2` zcOI1s^x6 z@}z~{#M8?Vy(-jppssOGdu%Tvp+v^TD1K-5tE-H0P{#%&>BU4IEX9yJ@>{YRODf zHiBvL+?ygb>rdO7aF3dlM^vem$Wvsfj|Q*2dF@r+6O7W8Fm%ef=AXqhsRtCJrDEV_ z;5%;>6Py}(YK4wkmnbL_BUQ)=HAx3GZXrU3r6?O#Wj6tlT@|q#KnA(JJ`K(}=DMpp zZ~nw za?PLiSNv(4+oAx2{hq&{T6qNfRBpmC#YL<;jFEy3tJKmYVLoT&y*(+3o%@g8Jw2*M z8TmyoF_1_Mt)9Z6Xn^fnA~30eP`r8hK;+Qk7c*WWLB&GQ5D7J|&|rI14FgC%Rmk~F zPQTKfmql>(BmU7~JgZm)N_!_6`s0)MlUmwjZX;6I8~Q4659QXW<4x1f6^+FGK2RLz z+xSQRJZZ0}T=~0Lt*-ty0%V>20mnaw$||FExSqQmD2aMG!Sn-~VYXKLi};h$vQtnH zFo)$50rLiJ%s#l_A8u;eYA)Q6Saccx01DReBFvskR7K>HPi5ovsFfHlN~lhsH%e8% ziH|2O-_oA*#@{f;@9#P3{(Y*Ix(H+3zTl^5QR!1hDgrPtPXir)3aYHE3n?r+zCS@o zK|!2j>rUWmTIUJC%hAoaC=0DL5X3{Pm?5jHl(xIUKR2kB8y z8!f<64svU6MDYBV_X_d`STo7*_)!(9**b|tx#1rVu6))|a@(*+KZSb6lWMRgLXnOu z2TzG$i)e@jPfoSF9nf_cZNv-^c*ytv02=CqQq=Nl)N*Fh$jU}a^y4G1(9_I{#H4_A z$ie2an)mlx8Fw#noDuyqRNCQ*>db)hz;ZsKwz@_ZxzK&8D6@`n?Nx4XgJ3F;#2zaw zz>T&*O9RG39DntVU4=efx!e!%p1=Kn!j53E5BfvNl-R)b546{3+gWh-Nn1>IvZg07_P4F?9$ekpnF63wCgRT>JV~6tcWhn6(R4k-xk~$Na|! z@-^4Zd1ojwOo@Jo8RBi7nR#E#aaPk!gpJ7*!F^$MnSY`D>cz4xx&6(|$bB|r z^v!3=x|MRYF>zO7OmIb8MvhVQh@ATEDAgc>KFsKQ2N@rjt5FrmQ9xAT9oL$2f({SoPSz1^U9o~s<5GjWt`Ff=V>mS^$mAT;xC-&B4D27BR*_TgkmKBQ zR>S9~2hyW(G03TuQyOJFi1GWuk46TV@fZ1{+JD_OG>VwUYLuu`l{l;W$*s#Xk>!w1M`Kubm;~pLYU5w7 zK6_VCd!zpVjZ~E_E(Qg|p*^HdKQUQ4)ZUjW6`;2=%5eDS_LN}7GI z%0}Afnz7LNXB#te+VCU!Q}?S^*5KJQ$Ije1`gf{pYPWEJcyaez(xbN2&85K8uZ9)Q z(8soW{VE|FY0&cJ2SK<$f&K=Diy)DnGJ9lJ^h(Rn8s#*pzRjqrIq1a?@~+4*1LYN^ zHF4o*`@UYo(42EpF4NwkErCq?Kta;CH?_+_MPcjmfYHP|DPxa*D?{yy+z@LIQn-#; zoaZ=feE<~|TI6;)u`Q?;G@D}3%`dA2U%+$vnzYb5pS{g3jv%p#7~!$F{ZBOcBva7S zrxc)(wI$00i1R9rlq`hjate-Hhe7g;imunW2O|cl=xnaZriaZ0g#xHOqh#dqPHWd6 zbmpLv@vw4EO+&G2=6XU4XBB!Ya5=?A8lOtP82M|i$CW8+J_P{a)U!l3^r%Q+K^0+} zF@k#2anO~iSs@rS;TgwDa=8{tFI=Ssot!zb4U;^X$XoV5i?Q* zspCM^DT51|)YUSPk0zs$Fe>p(4ns?k4;1Xuj8#Vknyo)I3F4wah}Ch=dVGAgYBF=c>U}DBlq&rEdk#$`40ARq(Zzwe z&I#?_qw@(kQG$3F>T1r_iGO*^e{g?Vv>~N850yYs)!(3~-S+VjCBNsRb^HfvSCF{N zV`%6HN^-zS8?)2yj=%kH#(@-!Wt$2MG5krVeUC4-*zW$~4bAxWAB|a(SmPUaaor<5 zey#o$PI(!VV-XBpe6$>&!@tzhX>}U7tu?)&{n`hYpa^#y`?vH__|+?^69yn}(H%sPE)LVm$f4)agpy%mO<=CqU8c!^2VA*MI zygbB__-^V?u&S+aux8&&j!!Z8X~_NK`zZbq^yl8Jzngfgad{_~ZoEdSxX-9X>G<^e z(xBgN&Zq1)JsH=TCH4oYKadp~y}z$d^DES4g{s?@6Ed`lH#BN;5PG|I`gN^LpowLb zzWU`<3}t5{sN=&ebU9g-+b0qoYdf|_+y%S$5DL~uL@i6Fp zhqYDEtXlh7KonyGxE(%~AKTVe!NMTwPp7q1lI0?ZF#x*{k$t^SxAUy|WK!5>C!TRb zoqRXHwLFH`L;nEQs`K24*ACbKVB__wH(Hv-FP5is^2hoLl-EM6%$xfNUsp%K z`T<>30yq?7g#eOArEt%vIleOIAUIm-Z8a#^FC*^Y=lRpUnR2^I=SK^Llc+p!YDBro z+Sm<_IjkkP4yvf#K2y)-`qYN)mXkxyO{cwUd5J@u^#FDHj{gA8kl)E6%x8jlBiGQ1 zq=?QybDVTFY86G!;s`xy^Ea@pTZ>V+!C6-$x#&mZO`hf2CV0qX95)?4qM&=xYE&>1 z#heTr5%fKew@RDr=_)PVoXVu}mKa9=0A+{ppV0eLO^Br(!yl4W^J4zvbzh;aTS=vY z0$Hikq|>nu%#?bNkyz zFu!=;oP7^p=T2cfKnce1dKVjk`ZxaoUZRdjr6bH9TlSF$ABg_|3c9jQ5c%-v-^REh zzP*p*&04b&3IuLID1E+R{xxbfZM;#Au_9}frsd}6^4e#EJ=aK>U?@s_Qz~}kX5gx_Q4TJAPD=$4g4{Ccyk;hR{g2~gN zpkjG|@J0;=Onu{#>rlxc-G@wiGXDUDFtA~TaL3T_e=3j?;jvWA!#hqX(#x>yV}s~x zI_g#AWCPop&PK6Bva*Kc;Ea!Wg>XOvv3ckWna&|fo0m&o@LxNkSmg@+AdNp{QCZ zqZv76^;2BUyv+mQT&-!tsLvnXZNjm1?eOShJRFv=WyWdFe7o7=K83dr)}Rt<$o~NA z(slPm&+?~@O4-NH^1gbSUotl!qbc;-YF$dP-Q0y%y&Wd6^8HJe+JyfAc0~aH0JsH7 zsM*1*+Ie$utkx2&7bo|t-}}SV zS9Pb}toCtB+>U;txx3+Os@wC3=TY)W>Hh%Mr)avnLpRuO6d_IqKcBr+WgFcy8gqpy zHl5S_4zqsl)Yit{l?j*2#!f-#M|x{YPaxn{_eXMRtDzclpYqRv-mp_M+{YhX?~lW& zt$T)H1F?S9ou#9BvP+EZ8$tBXTEt&>Mp zh#oAid8(4ymd##iS3i4-X#n!CK+%lwv}|w^)x`R8rJI*f%{X>`4{RQEE(;gkB9lUg}g zZbgWQHGbcw?1fRmW7GctuUSaZ#Bk?(kKj|l`k&}OI<}I$gfkrAa0dVnKs!}NUCK`8 z%ri3`v4R)79;f^(UIz-G4ozWMu{?CGc!&T2S8-Uh(lb+A5w7Mmo3<`VtgS-$E=Fm& zB(BDy-iVTGwevdFyLn5D3g+NuxtwD-&2-k0VFuAt2{|#;ii#y}l?$jGxxn=ARvQb` zlj~I^%f1Fn6Wgh)mWD`ZW_H?xYW}D3t4k3nfq>!K7xg&;K%u@hY*507d=nY(vU04`w(aQ$FTaEdczzm=1$*4Q_1{lyw?pN+5koD zi_m|Gqix0LS~Ecd1&{Z#58~*f=}|P#BDPp64!wu}0A8XBv5dc8?*9OdK{Sl`T>f1J z4ptq92}Vgy-^6?rPnG3g{RKS-ss2K#s4pkkZCy;P2$thNDlc#kW%X|T2OV2> z{=cud5iCj(=5sM(KjZ4@PDv;JS;l*l(?8I272Vmja&W717id-cw#nHC`(CH|)b}&T zVGKzMkjg)Bj5%EOm529_Vc1q|5E6XWUop2~5dQ!h{{VpR_)=}HKd^0ncB*PQC z{G_g5)l~Ff_nY(|T8qxYeWFk>Psp8*{{UEj^}T5>SUkQ@m`qWU^2z=7{wDr{smDLs zZnsE2MH{yE3Of(w_||RPLXs@YEAPvaT#e38sj8-9yr1rhagfQEWAia2{c1BG?xgp? z{eSw^E$UK4Qe3x_D*|~qC)=<1R%2f|bqB6%LRiAvk?+Z>MgYx~91cZGk)lY+y}6c2 znRj(K{LeqqsXUT>qA5!sm#Fo?z%{oup7v(iJ&jVD*A3GY`HHeKUM=E8>6XZ9d~V2h zDLZfmK9zi1mWyuRn2hF^>{-Ca7|kwVu^qM4u|DF&@%h&3-DC#OC#_^9xl^@);#`>ygj$%~~^Ny502v^Nf#r zciIKaYysp9@%-xrZWt=CC*>i59jfv(M$2&#$2^bjpHHEyQX!)3R}J@E2UEkNe-P|P zzrAQ$yV)?77B6o65&XgT{&kx!(9z0^`>8sAu-(-AdR5ywwljRYk+^uzfAi~9o}{y& zj3mn&#Q5zS4oV;H`;YcdBer@~pDk{9nF7s>?p|^|bNoZ~6_*$`*#)>qeT3v&mzW*(hwC!OCG|h zF%|-;&JI0^;+j;W4CGT|aOxdT1Y{B1denkUt_}ztsEA0~tJO=zAK?lpa-ptIsOGD*d`vRrG*IyCWTltmITtPJ8*ICN|pYIxK^5{{RtH zKGkJ?`=&BlPC6X#L;Y$SjY2GeCJuNfy=Tg*?Y=TP(a_P5DrJ^ZtpWcM- z$Klqo?st61cv~^SKsgx>+UOJWif>)+mMWdT(#0Q@~cKx zITMgb{#AcX)THvv7YntA1x;xyCApeaFzepf(zuZQowH8U{!1~>EOKg{)9zJO9ZgFR zlmG^BIIPpu-iPlhxvs{LAf9SB7#SHf*umhMm}j0ityZU7(4iyX^dgZaVmLLR@Z@~z z6neK62l_N;fLQzRC~$j?NedIm+yyO8cSw$0?mmnDG#NK81_JsH1xCjM1PW=`R=E_a z!Lv?{s?^bfII60IHF{3RlG;p|HbJVFF&8yvlfaw(}8-JI2snL&!mak%tmeCdJ3 zSB=O#R*cc*Y6xKj^sKpT#bPyQYItV{nz*+9!jCKi$gPsl5*Uw#sgitgOk`fw6DbEH zr7^TgE9JV?8JSw_{xOJ>6L?q%}tM=ARs3RA{Wtgc+%U5#w*p-lw4Y4)ox! zsA$Q2ye{TohU|S(qyC3t-MYt*35G$Upe)0bR3X(=?4?lqP zrWFzJS9FJ>e~Q8e=n-`$zrBH?#G9y2I&sYBd z0qs>;K*j+lpaZ$3S1rPf=Yfv(D@-iTBKhu*%eZs1*?mvFE8KRl^(&_?Hi>WFWQ!Swe&y@x%;RGLScW|8rnZ2)i-dJkb!vb!7C zLKfX|cb0NV%R9H~O-}O&t^Ck7mr}5=bx!T|JpQ#_La0n`NI<@1wj9W9D*xXex^p)K7*6}DK7W}C$>-H zP+XFt7>Dr23HrAu`HITrq)+GYxg&v-_|t{RSXqh2SSb9BP@Y003O$GEPeLSxx8y1i z#Qt=yGTNUu5xb9X#*jyhckfje;hoPI2a2^6z^)1U(irl9;Yh|tK9x>b0SbLjO1NJK zIpowkx149AP&8ML05(2)91%`1g~obzse%o>9&?&=sPgDSY@pHYa%wfY2`E#+Clt3c z43MqPDulU_44oZ#`ubHNGgRFyp@8e05!_T~_iY@bqPF5coo6dH^#`cyT5+j(fIfu& zw9_SoX|m=>NcnjJuG=@3N=g2$xNmQIh(W8)rZWyb*RyI;Nz7TKH&7N2$V?}Q~t3BAMG#rkL6k=av1wNri$mu zDjGJ7Gb!)RD_$7P5(y(YL!GDk8pMrH+rS;5&;C7Ic8|3PZdCOY;`%a8OL8{lN6Xz$ zZ~nDe357d(Q`8!x8yH`%RP;4$N;d9Z-9>Dla#{caWas7Yp4EXp%jc*viB=dcbPI#K>2x*~i>U}f89;2=*-MkRl+n*r&AbjXWdDJGA}%4vredE&mf+KTcP$fT*+u?qS?7D+fG5+PwP-U z>`r!1>BA19vYPYl7VRyoGX?wcH!PpxJ?kO$WRr1aZOPXiYFzA6r*om6IienHG6^Ca z<$ABwRz>~B$c*XN_|bYdzCWc$ZK%kYlHOd7xXnuU7I*4*e&%EMiVTD9RDUt~8qKu! zG>lq$ob;}fatKm-2IxoEx8qHP_Ayhq9_ndz3mNRnIr-sp@=AwtpT?qD0a20&9>3>` zUm#UWv%VylL;*D zWQ3lfTZ8o-4I;_rsa8F@)h1?wU9I;?;MBL(7MDhJF!+Kw+ZXzD5}t-ge{}x7DwaaNbhnL%^~$A0y% zGysp1IDqVSXZh7eRtE)$`f*uJC9^T9QZ*xDrt*>n!vF?BZYxe7nh_wCI()z!pVG2# z?0(Y;8N&~{$F*kZTBNo&Ceq{;aC&w>N~pBa%kR$lo!;d(~-dh~!|3P2`Z+%}TTG`Fdy8 zuIgKrs@SR*X~Co_ry$jdQ9;2K7QhB^QsZKiC4q#|gG7-5&MP;=_vvt(3FKDQzyT}O zRPL^M3LVO*!*Bwx$AYzvB?$Pdu-x&AOG52p@dFi3bHNoapY2trW+x)8G$zW7gttn5 z&;>je^rUUoPQ=}fP!^OsX9Ad8Ja?%fQd>Bzw`RApB@wfVdO{Ruy<3R}-juRN3e#~X zn5q~Krl68k+;C`>2q&7)bZz;`snHdwb9G3hHBb@}#Y-Icp|m>^%r=V5k%4}4eT7-N zy2{{HTXgd!1lOHjZc|Z5HAi=3TUgo93}AW+O^m0dXhj%Q6v>wdfnD^~k)&f}Qpc@1 zW6dr(=~9njRLe+bmw{2^X{nCttuqy>T8Xm%3ZdKzikYX_nB?B5fNVK^SW%|uwc1zu0JQdv(3qbz$=i69>;9Ov&hudw#2=D4{_pKyG< zx$ZlCKgO#{;Q>|Ze)E4ydrBAOA9(fls_t<^VzCj)Y*cT+z&v+8wJt^~76ph331h$m zl1*mHAl>uhwM@u~iEfDtjHE!}pWONnf69^FEJ-9Rw_crAKk+}HsuHuv#pGhn+_5K< zOKrn)OPlwd$3Uy{qNu=8>?$U9cUY4wNx%SPlj%@n=g(nQ$uJShI^pXO^O{^VraKyrkSF~{+rPxnV}&Z)jt{eM^dgq6(;Mn;K}AN514k%J-i z_8x=r>sBqTqLJR>N=oOmBYYpdHEf9co*dmU$wzha%b&^5=rb zyB~8xdR>FP?1Z|wdzNYB7#$UVx+=0D3%&m8jmO{Gq?LopRetL6MZ+q_sDAD@ztDF7 z06w&vySoc$W!)100J|sVW9CYI`wze$)|nh<810PI=^H$I-Dsk0Jl1~72_KbI5VX(FK}n4n%>ii(@{rj#Xz(yKP*keD@ce~Wgpt? zZ^^aIP`}ZeOVCKQ4a?)Pn3tnM26!LTg`8Q8&_c!B#wX z{{R}RsOpDGw(|xaMmUX6M*jfoRamaG*QPtxQ(cWx=)&qmEO8Sac6y(o?@Dz!SD!*B zriRvyZ4>X+Audd$kRR_M@6UXmhtjsvZRC|0an*ml^#1?~$cE?aWg`G8fwOOJJ5Hf7xMjzf4%xd>7J@zcow&QUFo;`hswM|=C)|{;DX0_GDw7z6+%OK=)gHLk~v=xCC zx`-bujuKPoeGlkrIbCNsVS;^oA4<;+hUeD@+PNJ@wCHxIC$CZBdz)K_j#OypPO5v= zw4Eja+qW`_PGhWW6{F%`}^UqIfY`Bh1+X0n*9s4{s39IFpdXl?Gc4K{n4 zkX}fkmvB~Zyc}*xJ(zSqTCF{t7j8wiEpGZ7c^l+^_Za^GBl()IygG5g6y;@_KPq0_ zD<0n8O0_GiG8}EHDJ+sjBSA7UNQVrLFjk?wD(SU{2fw9Qp7cj5+iGa?#-j-`{{XFs zdmrLoWB6jBy0>(VU0r5@_>ST}RlSdJeUCIXZBWy`=Ra{9+eyL79%>huIQh9#S0{Wi zNd_xAXz&R`$E8CyiY!FoSFS}^NXQ}MJa?>^q$tDRt!P0pLJ0gtVI!g_*|cC-1{J?+ z3a}N4WD;BvTaY6r2&~rRR4tov-E&puu!=+XdHPkz4Q?@i&O44$512Wen-D*xZ7mpR#=5YmR74NBdDSj9 zb=kDL1zk_zN2so@FqccZG6VM>wNC!V4Nl+6kZhHCj#1T3WLR9>X*Y(}@L}_i-JE)Z zT3+htoBKDtL@PKa2Bc6h zN|E%Yv;zgUV-&&)jIME2siudTXxiDa)Iz&_q~g6AJM_4PST=Gi#xFrhOAdh6&W++X zg#=Q$!02n!r&CKDT&JUzQzmNMN&@D&IJK2^1WBL9t;ei!_o^i;u{fP8N~_kH_Jk*s zSifnKRNi=B=T&CAW#K{1E@LFMI%SqJDme~%S3hrgH*jkDU6a=$t`6pkD0}&WO!ljm z((}(1Rc2Q1D#9ouiq$|;Pob)lAYQb|Za`CmS&q)0wNhKA8Km8af}O>feS1 zEyR=DW6pn(e(Y*Cs{H_~D-b;fDbEnaODP$V$sv3W6jqg`;gV5r1{sGW_8z33!n3A= zX{GX^A2H`SzyrA_uoW$;BnV)DorLB|2*H&907V^#;%aO=of^!hEvVap`_KLF@D(gm z2++uVaHG`LI!?^%1bz6cZ7LT4ACWj;e|k^3N%S`;q> zGc*4Hs9a|qw`2Kx)n{+8hEUnNek6S7U_DfO`k%wrrn|XloXpr4EarKzR+G=FDL^{>R3{t^A(@%*ZQi!x;N6zGm$ zapxzkXj;OF9G8;-VTYoh`Do{&{X3ujy+iky_9QR%TX7%!wv|DD_7k%GPgD3+WS3^< z9eY)FndOn4iDqM#C%L3z#ACH5AV`|szG7JQIW)49BxLuIvG@=Ezw)GtJVwXx6>DHk zqx*~c(4K%pDOeD9W6)G%I1JrSYJ7-gpK<3L3Y0=eBvQB)1Vw|KaezNsks%1RKEfkD zl^rLVy3T26?WnAET)#r?=liZrDS9H0g1Pu-& zHBI8j&{o68i0Xu?oG+=O!%V92;Dg$!%7Y*eTGn*=YMiNtUt>ZdsHM;xRTqJX5g6Sn$UqhF)saE0zjJY*O2`y~hzsu2-lBkSf~1JitKW zor$>%0*Jp;{VGWFw1eE#(?$ps#auVg(qt2-DhI7Z(x@d#zz2i%qDI*0z5f8inxm*V zSfq43vWCad=B+ZJ8?wFNk{f`*hVQ%&uealhjdcMAs9@-r00ZgNe>$xl>AQ#)z+^c3 zj%v12OK6?j_p7-#W3D+Lp{!K(b~dLCWSO$pb2B_jPRAI}r!^!6W6LW5K*H4s7{_d8 zla|L{TFGmvkgs5N1DuWcrL^ce&GdLZ=oAIs@euCmaC zAw+=uj)$X;)`CNBl-ns)JC~g>RE&(b1(~dx`-CJwas+OjU(&~=| z*n4+%K-r00^N)Jfg61O;xwZ!f=9(eZPI`}Q{{TNjT)a*mLh>f~*kPBxJ-rQCQ7)Y_ zG;-!9`I205>5@M}Ju~f58k*hz0K*dqTUl;t-`K}6`)#2GPt7E2+tr7+sO?xXj7(T{ z?kiS1#ZkPSMB#l=(+S%~ zP>GbQDEz9`z2c;QWw}EovE%{wqdu$t6_{pBkKsMOl(b6k6p|6R-yp1 z#~^qg?n=E6vGhLGSg{<`mUmKHNb!&)g~!bsbZ`E>SeDv0h}_ze@P87Yyob>L0F7Ho z>WNO;GdAH??mr>^{{UFq)7rBv?rwDHoFP)J#E{4`k`HnHdWzGYEDgZbS)&(7W$7aD z`0wvZ?6i>Ut6P0W-R86mHVnxVa`Air0JOgU0G)GF#`fnp%AWPP3p^Jp;o3J-m61*e zHDd18EkJ(kM-AJ5%RZ6Jf7$GXVc3E@nym`uI_79(A1T^tqbff2P_nvN6bC%|invoehRCd>mCkv+Nl}Sx92&Q8 za`F79k?&c8Go8xZ`c=4#0hT<9o>Oi}$~KXrr6lD@2hi0^>sart%rh@eRlA|}6(lo8 zK~@LrPmo}6cQ5KGG}fr|>fxnL-WwQpH`f|n(6_li?!RUg*FkW%k%FuP{{T9|xw1>h zN=P>;=togm_V+g$J<7#!#iBoQPR5AN#t@TE=<4_FB^Z$1!Jv*#R+|*C>JctmY=CgW znTeosk{pWNO6Q|WqMGwOuTj&$fI5z%xl9z8fO;DA3*EnEz~|-SxOR^h1oq8#LF!=_ zdzyA$V8$5NNv6Wm&Y_9gIW>!?sK_F^T_|k@o2Oc-SSI;Z&3(PIg%s#*QbsuyS1%wq zJOP@0ozY2f=e;DfxS_G99fB`WQ#GV&xxl9_!W7`uE8E?Xk_|LALv)K6EgN9gYU`3) zOSQ&8tlPatc_b>J;;MOa%5re1vH>FsYM zjxCdpeQRjSGeWiYDwu>)b5cVX&V!3Q|PcpibSl2Qc zWhyg)TNX%@3^D6c-02F6TzzUR<*_X$W?RMb4At1})s1JHpCVv{a1Cl(OtOrMTWHGB zXkCKI2{ibm#Rx)opE9;3y=X0IHgk z{hAhr5td#~;%It|*6<+!{;xNVY=kLBTy#D})p!{jjj3Hyo8p0oFlH@MK zjPqO3tbz~#!Nn-H^<>3gQSmpC9FqS45&2j8{{TvTxM?3G@jf9L0lD3}B95f{3b}tj zn(k@7(=pD7J_!@s~e6$sXK<{5$)fYK3>R7jL}BHCrnScX8_&104r* zO~GUGjy-C0h>@2)YGEHhI#k$WVThfodz_k>p?OFjjXp3Gk6%ijCFF5V#N1?d;Xv+b z?GUR$Fe$7CD!ECyNzfv5>zbz~UBK)r+~l97S6I$@?^4P`qmPc9RC0`*Rjgqu76T-j zVkt}jUwVe#4E)BdI++-HW~WHPrU0Z-PGrFLthq)VkEcr7n~()%yhM@@QfOd^f^Nk_ zG+9!8dRC8;RwCWU0+}>ncARwLlH3eHq&9sHYi7lPw@^C@&yq3bx(6*{DP@GDbLbxZ;id@d;RT;#N+Dv?B08;{{V$lkIR}8l4mEO@0yPGe9s^(c0bF{BO}t8r^76_PSPm_I486J z0QKp`B^7bgwJKL-`xrcl+U%9xw5qn=fQ%o;oA#x+x;DV_Mh86)Z}aa?yE6TnQ3oul zTmJyBj`*%h-f5wqd1fpdC!Vy{x|vgpto@_;9lSUMDH!cnQI}zcU-fKq5BCqR_|`Oc z%QSFY=ZQY>{_BpvpI^qcB5>fTBC~D9gzwaBqH9Op8isUTu7{+t^~d-g+__8Lq22^-jm!5K{{SQHQ29!Q;&6P4dV~BY*YN&TtSZ=0pK_k3 zX0jt(+eT4A+cZ7lZ|bM9tk~J3BX?ehHNS4MthfXVsD|I0kt%<5`q6WyLo{A|3AGPY z?-}$y-iE46n^bkeX=ut2NT(U(q85(2Ati?yKvHK_jjJBryGIRIF&@ zb5?t3uHk8|hSCOJEcEvt)rtP5Z=0du`-;)DxoyFL*z3^L*S0Br8%YX?<`MSoHL*HP{(BI4R>8@9K&LXJNp=t1s1>h08PaTee(%Si>uo)xpc4Dhe| zxfEsLKHjV9zm+*PC9dVh@!Mf7&Hz1YR^5rgQb6xn%V#yL!5oHG8wG!_H2OL%`GRG?HakSOo14IC9k?TzvHFDL(ycZ*K^)-oi zq{q0E!1k?JU87bMGnaIBcN>dxKa8;jN^@ZYx%j{{U_ZrjcvaoLSZGgTjwuUhCl0`b|#yMjmyWpLT zX}u3Tw-O}F7|$ma*F-#vTy!K;ZM1MNH_&|3Q=~T;PFfLv5|(R;Kg*yw)Tn{c+M)uqX<|F4@z4(gGZbSTZj8Z1wRV7 zDf&I;5V@{`eIX>cAdYig2AieD6e9{La$Jedn-1`S$^Tax6N3|5H4rA)R84;?D;&aN@sW>#oHI$^8N(MjnMYx{RKYb_i>n^IbY&$?;r52sik!U z5JhFpt%3_t#n(G?ienB=1yar&xnf0HNkf2pQj*ZRnk$irH_8iitFgrMPaU)ix<@zz z-kTWbC$(uTm4@dm39TzQOc9&L19&@O2m9UI@cwx8sBJA8dxTD& zr1k@;tw}=2;GiNj8&Q1+`RDPgit1-B#&W~9H)G2=$3L0=l=O=ovTD;rb}l}(EKpz% zS}X)97jyYj%5zqhFKSkgXz4|XxmGyIV^<=NVD~h{z#LTR&ot5#V0hxAZgM*gwIg@M zCNP+xf+{k1rUMEtz=kkwVPqaaN#}2T36^-EzzsLlNkg7(=^XB9+FuJNIG_McU5L8P~kT!|u(scycyIxGjk9edSTVNo+HcjOV; zr8@*|EzJA9Ijv+4OEMGkvZ!6Y#PzGT(g@{3!v*;O_Bh2$9npt$7emc~_|JS*StCJr zZjeYGC2~O=@kG7r7g92mQkPDp1QM&3R7Tw2G8~m``qt_!cJOiXs2%ymQ?ZGbM`s`e z9KULUd!uu3fk{}<{F&~3O;gu%N|gPpQClK8<(*{nK|kv| zA75(LSlp-%fzeOQzr?>o=~nGvNafud@Ou%~pEpCHMMY?1Y4AUnECdp+yOiUd%l&bh z)3h!|(s>o4Z6jA(h(H7!vW}F=tkoZSa6PJJ8=_>Ci6WDJ6~6al>;C}Pt7Sy7^Y?{8 z6h2_d7(Ma)>dFQO=KJ5yx!P%*bEf056QN~SQ}f6Czxvf;8H%4d06nBb(EkASjSPSs zj`YB$H^>|wzV%V5VqsUK8+l>~`$euyXR7y8=xZ_J>_!i*ZbIfKByy}UN2sPGiF-l$FlPbr!}M1qj>*M|!mj94jjj!;nX6ds~Cor@c4IXt`6Bq?ys93GKz% zXN(1$j5-dEJxBOf45Y4Ah8P@Tw`I#HWjkDQ20xyNZfp7o>mmzgWw zentbGx!cgw5+ECJs|=N0xsFuxA%DKU{-fTSchQ-@Ab65h-3bK#wImX{0J3xa zDjSO<3oF9^04s8iIv@VOwIo*=ZL)Gf`@)k=J%+5&qK>3s{Mf1O{!&7o4{E5YjKS29 zd;8V6Q;x zV|0F%+e>V+AS82Cp2?FW46mWCo>QtbIL27rikz!et4Ur1jmK%=irKu# z$DX35hIL%SCLIq2TB%1H54qG-&9*~^;r%FUnHf%;WUV6Yyd_(uSZKlRPK+O#pq^r! z;=L$Z);#p|xZ@Ila5~cksK9Ns*zPx0qDEwkfyOFh9%(jZ8D3X1u?t!|gr|d1NoH7s*A-HB98AQuc{w#Q*_G+m zw7jsZjC86q+?b!I6)tYVvblEI+6YK)Y8bDCb27Q%jYOBK$&y7^HJm$5h?-|ZVc>D~~q1>F- zY70X=H{w1M~oHBB@^Xs;!*$oo%2PhCD+XA(eir>LzjDM-!^eJNsWjiDS-9R{7A zg_~2pKbwP6n2A4gBkNMyS~)v_I6aMOrK%1{=CpE8P;TaY8hSU)$Tgsmgct^%Kwbqj zw>*mHoXSxH$0z>)*HJM!=A6x*wHux@ST<;o(%9=!vCn!!bKaQEg(uRY<=A0yAS51@ zUiHi{Iv>8sNzY;F{xp^kv`H!K#3((_{{UK{zJ2?H9F|f(yiv_%a$CAFyy7V)MV2~; zc3aw~+45&Kj2jd+W*x*NHbLN1tcw?I!+}rKW{Eyo0blloL2L4iAue5j7_F1KG^rnT znz69KW*Jq%BiGurUP88!D;&u=GdcYi^dGHdGXRAYj4x6-X{!}F8*Q#A9OlULzSI(1Rhwgpkt^d^okv)X4EdARy0{{R#GszgkI zPo0CvcKy-vuh4X*jC_Rmtvrmp#Av}Ij+J6G80k^=%}*apQ(Kb9B6r6%9Fga(SqF-2 zfB~AT5>fJqH6Y+%Q;$6;E4GtBG>{HE(iWhpCyu6^Av_aOP~7Jjrn3Yd=B8x|ed;Zv zIX!3@6E1l@Jt_%CIVYO6BZXS3=OnQoN{L9)F)2Ji9G*!%O;?jF%onFX2YR+onMdi%cW&mv+MMrRakn%Twx=%s=H3X~!Jj5IxgQ4kCn=Z(!HPZd1 z)F{U8JMmdMg~OOh1mw8{dk$$WWRiQCi;O8?bM2fFO4jD&ODlcn$-wP`PxPR*WJNBj z(gJO%B>JRkTvJ05g1ZUGce?P*Zg3TehfT#@TEx)p2q*-H@NFcW5Oqr08 zjo;om^aB_k{Y=k2T9lmY&;~!7!Rug^zV1FLb6H$L zDkveJYbo!?Ut?QP**wk)BL?^Ae!i6ss|+^LG>sva;a76`W8^?R`}M1YTh#ATQjaN_ z46|PsK@-NYovgm6*NVQD0H7^~$6;A|JaRM=GReo8jk)0Q*0n-OY(VHwJanp+u3U6l z4vmaXayjU7dWx{E9}F@v)Ou4E7L;xs`xvU@E#cK*(TeUEan{Nz}JtvF;dX$@+R#<#6aSpdV_k ziGnuZA4--|(R|p+^rF`lW}Tw9Wa}7lfPS9FqhLn_Q5(lEsyC--Z7g zx+65sx^=o4w-)>oo%>c5^lW0>G94R&E2?xXK3d1WfCYb-2N}g~wmKs#nMo>YspZZ@ zcVJ*hcn|D-{p&JwJ*=%1szFnX8j|4|PU!meAW$!o@ddr(?P0`o5z6~FsHt|A`suYZhh1rCcRA}3=SjsT-A#c8! z$KpK=A%pBrtbj=i@~w`(-pAUi8B}1ddGAlOx-qjO<>6!VAwQ_}H2JNeE;lt`xwm`< z8%Kd<;7YP5(^D;ZsGa#f=ktHrbuvu-@oV!4qR5CIvcIp(Lf zi+9qZ7jHv~Dx-IcEvB?o;i5S9s}bs!(u@`=eZ^Li%rbnq$f|N$73eExIm2?@T9Mq@ z#7@~qQ}i`T_e~cADn4=0ip~*Ar{&1{RG(?NhxpIqN3~CMX&hH{Rhk*%;ei76mf(|eM_vB}D|-)gY-ir^3iM{!L^&>XI6 z!i)-!Xk%9uAF|6aTd)PNNc5_AknUlQ^if{SqLeOkrul4C7VORydz@EKYXYPoA4({w zq40_c6PQCG=~#D`8{GDyirPx$r>RaG285DG+f%9u4il{uQbLn=8uo6@hmC;s6n|g~ zfC8{=^v@JgSIC+b+GEqfwcC7uLqy ze+a1syi3tV6jO3xWIb@50HiWviYgqLN+@Y;ifK6*2Z|`EWmt`cLm}=@6j4yiMQI-& zcA+vBQlsx*$9Ta8XymBe?7c~Z+!rm60TO^YlOYi(g@Emv=x|`MbOY!7@xW^S@5tH?z zik0ymWi2X~p!qZOtqT|Vq?7xL!S_6WQ$-cB-0P7ja4<45dJ3?I6~Emilp|gmPI)Su;U-rszs-}x)mIz zcIW%ub43(fYhr7CA>o4I)xLRxf)5~M<+%Rgf! z%!P|O?c3A}D6Jswc2hZx4>tEwMI&P*sXq15C`9=VHucCkZ~nCuR*!Q;D%(QNUKt5- z*qXYNsX())03Cg(qLq2rg;=mSlixa6t!|>k?m8tl#F*K znA(lIzS$MWZNrg86uGRfZ9#L*x6ofR=E{*G_Bg>6rD*Eo89a`2MHO8Uw4$BH&mPvn z%ChmE#CE1HBOa7dSk#KS$v7(x#wqCBVDzGj=T6T=dsU9t1L+J38A~L?&rwCN?2kS)@%Tb(_p%t+z#s_MC0HTVBE3=KC|Jir} Bp^E?j literal 0 HcmV?d00001 diff --git a/python/paddle/v2/tests/test_image.py b/python/paddle/v2/tests/test_image.py new file mode 100644 index 0000000000..5aa612f91d --- /dev/null +++ b/python/paddle/v2/tests/test_image.py @@ -0,0 +1,42 @@ +# Copyright PaddlePaddle contributors. All Rights Reserved +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import unittest +import numpy as np + +import paddle.v2.image as image + + +class Image(unittest.TestCase): + def test_resize_flip_chw(self): + # resize + im = image.load_image('cat.jpg') + im = image.resize_short(im, 256) + self.assertEqual(256, min(im.shape[:2])) + self.assertEqual(3, im.shape[2]) + + # flip + im = left_right_flip(im) + im2 = np.flip(im, 1) + self.assertEqual(im.all(), im2.all()) + + # to_chw + h, w, c = im.shape + im = image.to_chw(im) + self.assertEqual(c, im.shape[0]) + self.assertEqual(h, im.shape[1]) + self.assertEqual(w, im.shape[2]) + + +if __name__ == '__main__': + unittest.main() diff --git a/python/setup.py.in b/python/setup.py.in index 5dfb46192a..7d9438e3f8 100644 --- a/python/setup.py.in +++ b/python/setup.py.in @@ -18,6 +18,7 @@ setup(name='paddle', "numpy", "protobuf==${PROTOBUF_VERSION}", "matplotlib", + "opencv-python", ], packages=packages, package_dir={ From 60ef01773f2df0266cfed3e60dac72ac4c823f60 Mon Sep 17 00:00:00 2001 From: gongweibao Date: Wed, 10 May 2017 16:14:51 +0800 Subject: [PATCH 34/88] modify by talks --- doc/design/cluster_train/data_dispatch.md | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/doc/design/cluster_train/data_dispatch.md b/doc/design/cluster_train/data_dispatch.md index a3eb4e28db..7690cb279d 100644 --- a/doc/design/cluster_train/data_dispatch.md +++ b/doc/design/cluster_train/data_dispatch.md @@ -12,16 +12,18 @@ 在上图中显示了在一个实际生产环境中的应用（人脸识别）的数据流图。生产环境的日志数据会通过实时流的方式（Kafka）和离线数据的方式（HDFS）存储，并在集群中运行多个分布式数据处理任务，比如流式数据处理（online data process），离线批处理（offline data process）完成数据的预处理，提供给paddle作为训练数据。用于也可以上传labeled data到分布式存储补充训练数据。在paddle之上运行的深度学习训练输出的模型会提供给在线人脸识别的应用使用。 ### 训练数据的存储 +We select CephFS to store our data. -选择CephFS作为训练数据的存储服务。 +From the perspective of user program running in a Pod, it is only I/O with the local filesystem, as -在Kubernetes上运行的不同的计算框架，可以通过Volume或PersistentVolume挂载存储空间到每个容器中。 +1. the home directory should have been mapped to the Pod-local directory `/home`, and +1. some shared directories, e.g., the pre-downloaded `paddle.v2.dataset` data, should have been mapped to the Pod-local directory `/common`. -在CephFS存储系统中的公开目录，需要保存一些预置的公开数据集（比如MNIST, BOW, ImageNet数据集等），并且可以被提交的job直接使用。 +and from the perspective of our client tool `paddle`, it has to refer to files in the distributed filesystem in a special format, just like `/pfs/$DATACENTER/home/$USER/cifa/...`. ### 文件预处理 -在数据集可以被训练之前，文件需要预先被转换成PaddlePaddle集群内部的存储格式（SSTable）。我们提供两个转换方式： +在数据集可以被训练之前，文件需要预先被转换成PaddlePaddle集群内部的存储格式[RecordIO](https://github.com/PaddlePaddle/Paddle/issues/1947)。我们提供两个转换方式： - 提供给用户本地转换的库，用户可以编写程序完成转换。 - 用户可以上传自己的数据集，在集群运行MapReduce job完成转换。 @@ -92,11 +94,11 @@ random_images-00099-of-00099 #### 进行训练 -PaddlePaddle提供专用的[data reader creator](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/reader/README.md#python-data-reader-design-doc)，生成给定SSTable文件对应的data reader。**无论在本地还是在云端，reader的使用方式都是一致的**： +PaddlePaddle提供专用的[data reader creator](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/reader/README.md#python-data-reader-design-doc)，生成给定`RecordIO`文件对应的data reader。**无论在本地还是在云端，reader的使用方式都是一致的**： ```python # ... -reader = paddle.reader.creator.SSTable("/home/random_images-*-of-*") +reader = paddle.reader.creator.RecordIO("/home/random_images-*-of-*") batch_reader = paddle.batch(paddle.dataset.mnist.train(), 128) trainer.train(batch_reader, ...) ``` @@ -107,13 +109,14 @@ trainer.train(batch_reader, ...) 使用下面命令，可以把本地的数据上传到存储集群中。 -```bash -paddle cp filenames pfs://home/folder/ +```bash +paddle pfs cp filenames /pfs/folder/ ``` 比如，把之前示例中转换完毕的random_images数据集上传到云端的`/home/`可以用以下指令： -```bash -paddle cp random_images-*-of-* pfs://home/ + +```bash +paddle pfs cp random_images-*-of-* /pfs/folder/ ``` ## TODO From e3fd0d56f46b26894c682213f17272c122d7c2a8 Mon Sep 17 00:00:00 2001 From: dangqingqing Date: Wed, 10 May 2017 16:34:24 +0800 Subject: [PATCH 35/88] Pass unit test --- paddle/scripts/run_python_tests.sh | 2 +- python/paddle/v2/image.py | 19 +++++++++++++++++-- python/paddle/v2/tests/CMakeLists.txt | 2 +- python/paddle/v2/tests/test_image.py | 2 +- 4 files changed, 20 insertions(+), 5 deletions(-) diff --git a/paddle/scripts/run_python_tests.sh b/paddle/scripts/run_python_tests.sh index 05c187522c..02d2cdb977 100755 --- a/paddle/scripts/run_python_tests.sh +++ b/paddle/scripts/run_python_tests.sh @@ -29,7 +29,7 @@ if [ $USE_VIRTUALENV_FOR_TEST -ne 0 ]; then fi export PYTHONPATH=$SCRIPTPATH/../../python/ -$PYTHON -m pip install $SCRIPTPATH/../dist/*.whl requests matplotlib ipython==5.3 +$PYTHON -m pip install $SCRIPTPATH/../dist/*.whl requests matplotlib opencv-python ipython==5.3 for fn in "$@" do diff --git a/python/paddle/v2/image.py b/python/paddle/v2/image.py index 20dcc96782..ca75efd90d 100644 --- a/python/paddle/v2/image.py +++ b/python/paddle/v2/image.py @@ -45,8 +45,13 @@ def load_image(file, is_color=True): return a color image. Otherwise, it will load and return a gray image. """ - flag = cv2.CV_LOAD_IMAGE_COLOR if is_color else \ - cv2.CV_LOAD_IMAGE_GRAYSCALE + # cv2.IMAGE_COLOR for OpenCV3 + # cv2.CV_LOAD_IMAGE_COLOR for older OpenCV Version + # cv2.IMAGE_GRAYSCALE for OpenCV3 + # cv2.CV_LOAD_IMAGE_GRAYSCALE for older OpenCV Version + # Here, use constant 1 and 0 + # 1: COLOR, 0: GRAYSCALE + flag = 1 if is_color else 0 im = cv2.imread(file, flag) return im @@ -178,6 +183,11 @@ def simple_transform(im, resize_size, crop_size, is_train, is_color=True): Simply data argumentation for traing. These operations includes resizing, croping and flipping. + Example usage: + + .. code-block:: python + im = simple_transform(im, 256, 224, True) + :param im: The input image with HWC layout. :type im: ndarray :param resize_size: The shorter edge length of the resized image. @@ -209,6 +219,11 @@ def load_and_transform(filename, data argumentation. Please refer the `simple_transform` interface for the transform operation. + Example usage: + + .. code-block:: python + im = load_and_transform('cat.jpg', 256, 224, True) + :param filename: The file name of input image. :type filename: string :param resize_size: The shorter edge length of the resized image. diff --git a/python/paddle/v2/tests/CMakeLists.txt b/python/paddle/v2/tests/CMakeLists.txt index 0b8c78b465..eb02e53706 100644 --- a/python/paddle/v2/tests/CMakeLists.txt +++ b/python/paddle/v2/tests/CMakeLists.txt @@ -1,2 +1,2 @@ add_python_test(test_v2_api test_data_feeder.py test_parameters.py -test_layer.py test_rnn_layer.py test_topology.py test_image) +test_layer.py test_rnn_layer.py test_topology.py test_image.py) diff --git a/python/paddle/v2/tests/test_image.py b/python/paddle/v2/tests/test_image.py index 5aa612f91d..b2d773510d 100644 --- a/python/paddle/v2/tests/test_image.py +++ b/python/paddle/v2/tests/test_image.py @@ -26,7 +26,7 @@ class Image(unittest.TestCase): self.assertEqual(3, im.shape[2]) # flip - im = left_right_flip(im) + im = image.left_right_flip(im) im2 = np.flip(im, 1) self.assertEqual(im.all(), im2.all()) From ad0608847e8eef659f57d580c3816af653275339 Mon Sep 17 00:00:00 2001 From: Yancey1989 Date: Wed, 10 May 2017 19:30:04 +0800 Subject: [PATCH 36/88] Fix paddle.init bug --- python/paddle/v2/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/python/paddle/v2/__init__.py b/python/paddle/v2/__init__.py index 70c8ec0baa..d13850597e 100644 --- a/python/paddle/v2/__init__.py +++ b/python/paddle/v2/__init__.py @@ -52,7 +52,7 @@ def init(**kwargs): args_dict.update(kwargs) # NOTE: overwrite arguments from ENV if it is in kwargs for key in args_dict.keys(): - args.append('--%s=%s' % (key, str(kwargs[key]))) + args.append('--%s=%s' % (key, str(args_dict[key]))) api.initPaddle(*args) From 3513935f7ee4a6a9d6eb06b37dd681c015a37694 Mon Sep 17 00:00:00 2001 From: dangqingqing Date: Wed, 10 May 2017 19:30:46 +0800 Subject: [PATCH 37/88] refine the annotation. --- python/paddle/v2/image.py | 30 ++++++++++++++---------------- 1 file changed, 14 insertions(+), 16 deletions(-) diff --git a/python/paddle/v2/image.py b/python/paddle/v2/image.py index ca75efd90d..13f53919aa 100644 --- a/python/paddle/v2/image.py +++ b/python/paddle/v2/image.py @@ -11,22 +11,20 @@ __all__ = [ "left_right_flip", "simple_transform", "load_and_transform" ] """ -This file contains some common interface for image preprocess. +This file contains some common interfaces for image preprocess. Many users are confused about the image layout. We introduce -the image layout firstly. +the image layout as follows. - CHW Layout - The abbreviations: C=channel, H=Height, W=Width - - The default image layout is HWC opened by cv2 or PIL. - PaddlePaddle only support the image layout with CHW. - CHW is simply a transpose of HWC. It must transpose - the input image. + - The default layout of image opened by cv2 or PIL is HWC. + PaddlePaddle only supports the CHW layout. And CHW is simply + a transpose of HWC. It must transpose the input image. - Color format: RGB or BGR OpenCV use BGR color format. PIL use RGB color format. Both - formats can be used for training. But it must be noted that, - the format should be keep consistent between the training and - inference peroid. + formats can be used for training. Noted that, the format should + be keep consistent between the training and inference peroid. """ @@ -85,8 +83,8 @@ def resize_short(im, size): def to_chw(im, order=(2, 0, 1)): """ Transpose the input image order. The image layout is HWC format - opened by cv2 or PIL. Transposed the input image to CHW layouts - by order (2,0,1). + opened by cv2 or PIL. Transpose the input image to CHW layout + according the order (2,0,1). Example usage: @@ -116,7 +114,7 @@ def center_crop(im, size, is_color=True): :param im: the input image with HWC layout. :type im: ndarray - :param size: the cropping size + :param size: the cropping size. :type size: int :param is_color: whether the image is color or not. :type is_color: bool @@ -143,7 +141,7 @@ def random_crop(im, size, is_color=True): :param im: the input image with HWC layout. :type im: ndarray - :param size: the cropping size + :param size: the cropping size. :type size: int :param is_color: whether the image is color or not. :type is_color: bool @@ -180,7 +178,7 @@ def left_right_flip(im): def simple_transform(im, resize_size, crop_size, is_train, is_color=True): """ - Simply data argumentation for traing. These operations includes + Simply data argumentation for training. These operations include resizing, croping and flipping. Example usage: @@ -216,8 +214,8 @@ def load_and_transform(filename, is_color=True): """ Load image from the input file `filename` and transform image for - data argumentation. Please refer the `simple_transform` interface - for the transform operation. + data argumentation. Please refer to the `simple_transform` interface + for the transform operations. Example usage: From e59c183afb5c86055f0b29b77e37b1d5f0673c9d Mon Sep 17 00:00:00 2001 From: Yi Wang Date: Wed, 10 May 2017 11:12:32 -0700 Subject: [PATCH 38/88] Remove -D CPACK_DEBIAN_PACKAGE_DEPENDS="" from the invocation of cpack --- paddle/scripts/docker/build.sh | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/paddle/scripts/docker/build.sh b/paddle/scripts/docker/build.sh index b1a8274b1d..101b44e6c6 100644 --- a/paddle/scripts/docker/build.sh +++ b/paddle/scripts/docker/build.sh @@ -113,14 +113,12 @@ fi # generate deb package for current build # FIXME(typhoonzero): should we remove paddle/scripts/deb ? -# FIXME: CPACK_DEBIAN_PACKAGE_DEPENDS removes all dev dependencies, must -# install them in docker cat < Date: Wed, 10 May 2017 11:15:49 -0700 Subject: [PATCH 39/88] Remove the incomplete file write_docs_en.rst. Will create it in another PR. --- doc/howto/dev/write_docs_en.rst | 77 --------------------------------- 1 file changed, 77 deletions(-) delete mode 100644 doc/howto/dev/write_docs_en.rst diff --git a/doc/howto/dev/write_docs_en.rst b/doc/howto/dev/write_docs_en.rst deleted file mode 100644 index 65e7edca94..0000000000 --- a/doc/howto/dev/write_docs_en.rst +++ /dev/null @@ -1,77 +0,0 @@ -############### -Build Documents -############### - -Document files of PaddlePaddle are in sub-directory :code:`doc`. Source files are in `RST `_ format. We can build the document by letting `cmake`_ invoke `sphinx `_ to convert RST files into HTML files. - - -How to Build Documents -====================== - -To save the time of installing building tools, we provide a Docker image. - - -使用Docker构建PaddlePaddle的文档 --------------------------------- - -使用Docker构建PaddlePaddle的文档，需要在系统里先安装好Docker工具包。Docker安装请参考 `Docker的官网 `_ 。安装好Docker之后可以使用源码目录下的脚本构建文档，即 - -.. code-block:: bash - - cd TO_YOUR_PADDLE_CLONE_PATH - cd paddle/scripts/tools/build_docs - bash build_docs.sh with_docker - -编译完成后，会在当前目录生成两个子目录\: - -* doc 英文文档目录 -* doc_cn 中文文档目录 - -打开浏览器访问对应目录下的index.html即可访问本地文档。 - - - -直接构建PaddlePaddle的文档 --------------------------- - -因为PaddlePaddle的v2 api文档生成过程依赖于py_paddle Python包，用户需要首先确认py_paddle包已经安装。 - -.. code-block:: bash - - python -c "import py_paddle" - -如果提示错误，那么用户需要在本地编译安装PaddlePaddle，请参考 `源码编译文档 `_ 。 -注意，用户在首次编译安装PaddlePaddle时，请将WITH_DOC选项关闭。在编译安装正确之后，请再次确认py_paddle包已经安装，即可进行下一步操作。 - -如果提示正确，可以执行以下命令编译生成文档，即 - -.. code-block:: bash - - cd TO_YOUR_PADDLE_CLONE_PATH - cd paddle/scripts/tools/build_docs - bash build_docs.sh local - -编译完成之后，会在当前目录生成两个子目录\: - -* doc 英文文档目录 -* doc_cn 中文文档目录 - -打开浏览器访问对应目录下的index.html即可访问本地文档。 - - -如何书写PaddlePaddle的文档 -========================== - -PaddlePaddle文档使用 `sphinx`_ 自动生成，用户可以参考sphinx教程进行书写。 - -如何更新www.paddlepaddle.org文档 -================================ - -开发者给PaddlePaddle代码增加的注释以PR的形式提交到github中，提交方式可参见 `贡献文档 `_ 。 -目前PaddlePaddle的develop分支的文档是自动触发更新的，用户可以分别查看最新的 `中文文档 `_ 和 -`英文文档 `_ 。 - - - -.. _cmake: https://cmake.org/ -.. _sphinx: http://www.sphinx-doc.org/en/1.4.8/ From 66a3cfe36f10c95dbccb1b3e2b08aafb02247437 Mon Sep 17 00:00:00 2001 From: Helin Wang Date: Wed, 10 May 2017 16:41:37 -0700 Subject: [PATCH 40/88] change to C API --- doc/design/cluster_train/pserver_client.md | 95 ++++++++++++++++++++++ doc/design/cluster_train/trainer.md | 82 ------------------- 2 files changed, 95 insertions(+), 82 deletions(-) create mode 100644 doc/design/cluster_train/pserver_client.md delete mode 100644 doc/design/cluster_train/trainer.md diff --git a/doc/design/cluster_train/pserver_client.md b/doc/design/cluster_train/pserver_client.md new file mode 100644 index 0000000000..62edd349f3 --- /dev/null +++ b/doc/design/cluster_train/pserver_client.md @@ -0,0 +1,95 @@ +# Design Doc: The Client Library of Parameter Server + +For an overview of trainer's role, please refer to [distributed training design doc](README.md). In this design doc, we will discuss the parameter server's client library, which will manage communication with parameter servers. The library will be implemented in [Go](https://golang.org/) and made available as a static or dynamic library with a C header file. + +## C Interface + +```c +#define PADDLE_ELEMENT_TYPE_INT32 0 +#define PADDLE_ELEMENT_TYPE_UINT32 1 +#define PADDLE_ELEMENT_TYPE_INT64 2 +#define PADDLE_ELEMENT_TYPE_UINT64 3 +#define PADDLE_ELEMENT_TYPE_FLOAT32 4 +#define PADDLE_ELEMENT_TYPE_FLOAT64 5 + +typedef struct paddle_pserver_client paddle_pserver_client; + +/** + * @brief paddle_new_pserver_client creates a new parameter server + * client. + */ +paddle_pserver_client* paddle_new_pserver_client(); + +/** + * @brief paddle_pserver_client_release releases the parameter server + * client. + */ +void paddle_pserver_client_release(paddle_pserver_client* client); + +/** + * @brief paddle_begin_init_param begins to initialize parameters + * on parameter servers. + * + * paddle_begin_init_param will be called from multiple trainers, only + * one trainer will be selected to initialize the parameters on + * parameter servers. Other trainers will be blocked until the + * initialization is done, and they need to get the initialized + * parameters from parameter servers using @paddle_get_param. + * + * @return 1 if trainer is selected to initialize parameter + * servers, otherwise 0. + */ +int paddle_begin_init_param(paddle_pserver_client* client); + +/** + * @brief paddle_init_param initializes the parameter on parameter + * servers. + * + * @return 0 if successful, otherwise -1. On failure the trainer need + * to restart the entire initialization process starting from + * paddle_begin_init_param. Or simply exit the program and wait for + * cluster management system to restart trainer. + */ +int paddle_init_param(paddle_pserver_client* client, const char* name, int element_type, const void* content); + +/** + * @brief paddle_finish_init_param tells parameter servers client has + * sent all parameters to parameter servers as initialization. + * + * @return 0 if successful, otherwise -1. On failure the trainer need + * to restart the entire initialization process starting from + * paddle_begin_init_param. Or simply exit the program and wait for + * cluster management system to restart trainer. + */ +int paddle_finish_init_param(paddle_pserver_client* client); + +/** + * @brief paddle_send_grad sends gradients to parameter servers for + * updating parameters. + * + * @return 0 if successful, otherwise -1. + */ +int paddle_send_grad(paddle_pserver_client* client, const char* name, int element_type, const void* content); + +/** + * @brief paddle_set_param sets a parameter on parameter servers. + * + * @return 0 if successful, otherwise -1. + */ +int paddle_set_param(paddle_pserver_client* client, const char* name, int element_type, const void* content); + +/** + * @brief paddle_get_param gets the parameter from parameter servers. + * + * @return 0 if successful, otherwise -1. + */ +int paddle_get_param(paddle_pserver_client* client, const char* name, void** dst, int* dstLen); + +/** + * @brief paddle_save_model indicates parameters to save the parameter + * to the given path + * + * @return 0 if successful, otherwise -1. + */ +int paddle_save_model(paddle_pserver_client* client, const char* path); +``` diff --git a/doc/design/cluster_train/trainer.md b/doc/design/cluster_train/trainer.md deleted file mode 100644 index bcb4a9c09d..0000000000 --- a/doc/design/cluster_train/trainer.md +++ /dev/null @@ -1,82 +0,0 @@ -# Design Doc: Trainer Communication Library - -For an overview of trainer's role, please refer to [distributed training design doc](README.md). In this design doc, we will discuss the trainer's communication library, which will manage communication with parameter servers and the [master server](master_server.md). The library will be implemented in [Go](https://golang.org/) and made available as a static or dynamic library with a C header file. - -## Go Interface - -The Go interface is the basic abstraction of communications with the master server and parameter servers. We will add another layer on top (add retry logic, polish interface with C idiom) before exposing the library with a [C interface](#c-interface). - -```go -// MasterClient is the client to the master server. -type MasterClient struct {} - -// GetTask gets a new task by telling the master server the finished task. -// Use nil as the finished task when getting the task for the first time. -func (*MasterClient) GetTask(finished master.Task) (master.Task, error) - -// ElementType is the type of elements of a Parameter. -type ElementType int - -// Different element types. -const ( - Int32 ElementType = iota - UInt32 - Int64 - UInt64 - Float32 - Float64 -) - -// Parameter is a piece of data to sync with the parameter server. -type Parameter struct { - Name string - ElementType ElementType - Buffer []byte -} - -// Gradient is the gradient of the parameter. -type Gradient Parameter - -// PServerClient is the client to parameter servers. -type PServerClient struct {} - -// UpdateRule specifies the rule for updating parameters with gradients. -type UpdateRule struct { - UpdateMethod pserver.UpdateMethod - LearningRate float32 -} - -// ParamInitChans returns a send channel for parameter initialization. -// -// ParamInitChans will be called from multiple trainers, only one trainer should -// initialize the parameters on parameter servers, other trainers will instead -// get the initialized parameters from parameter servers using GetParam. -// -// If send channel is not nil, the trainer is selected to do the initialization, -// the trainer needs to signal for finishing initializing the parameters by -// closing the send channel. -func (*PServerClient) ParamInitChan() (send chan<- Parameter, err error) - -// SendGrad sends gradients to parameter servers for updating parameters. -func (*PServerClient) SendGrad(method UpdateMethod, grads []Gradient) error - -// SetParam sets parameters. -// -// SetParam can be used for the parameters that are not suitable for updating -// using gradients. -func (*PServerClient) SetParam(params []Paramter) error - -// GetParam gets parameters from parameter servers. -func (*PServerClient) GetParam(names []string) ([]Parameter, error) - -// Save indicates parameters to save the parameter to the given path. -// -// Path needs to be the path to a distributed file system which is visible -// to all parameter servers. -func (*PServerClient) Save(path string) error -``` -Please see [master server design doc](master_server.md) for the definition of `master.Task`. - -## C Interface - -TODO From 0064fcb082930bbaf3d2f144697af677db0b0d6a Mon Sep 17 00:00:00 2001 From: Helin Wang Date: Wed, 10 May 2017 17:19:54 -0700 Subject: [PATCH 41/88] update C API --- doc/design/cluster_train/pserver_client.md | 49 +++++++++++++++------- 1 file changed, 34 insertions(+), 15 deletions(-) diff --git a/doc/design/cluster_train/pserver_client.md b/doc/design/cluster_train/pserver_client.md index 62edd349f3..7ecd4cff06 100644 --- a/doc/design/cluster_train/pserver_client.md +++ b/doc/design/cluster_train/pserver_client.md @@ -12,6 +12,13 @@ For an overview of trainer's role, please refer to [distributed training design #define PADDLE_ELEMENT_TYPE_FLOAT32 4 #define PADDLE_ELEMENT_TYPE_FLOAT64 5 +typedef struct { + char* name; + int element_type; + void* content; + int content_len; +} paddle_parameter, paddle_gradient; + typedef struct paddle_pserver_client paddle_pserver_client; /** @@ -27,33 +34,36 @@ paddle_pserver_client* paddle_new_pserver_client(); void paddle_pserver_client_release(paddle_pserver_client* client); /** - * @brief paddle_begin_init_param begins to initialize parameters + * @brief paddle_begin_init_params begins to initialize parameters * on parameter servers. * - * paddle_begin_init_param will be called from multiple trainers, only + * paddle_begin_init_params will be called from multiple trainers, only * one trainer will be selected to initialize the parameters on * parameter servers. Other trainers will be blocked until the * initialization is done, and they need to get the initialized - * parameters from parameter servers using @paddle_get_param. + * parameters from parameter servers using @paddle_get_params. * - * @return 1 if trainer is selected to initialize parameter - * servers, otherwise 0. + * @param config_proto serialized parameter server configuration + * protobuffer. + * @return 1 if trainer is selected to initialize parameter servers, + * otherwise 0. */ -int paddle_begin_init_param(paddle_pserver_client* client); +int paddle_begin_init_params(paddle_pserver_client* client, const char* config_proto); /** * @brief paddle_init_param initializes the parameter on parameter * servers. * + * @param param the parameter to initialize. * @return 0 if successful, otherwise -1. On failure the trainer need * to restart the entire initialization process starting from * paddle_begin_init_param. Or simply exit the program and wait for * cluster management system to restart trainer. */ -int paddle_init_param(paddle_pserver_client* client, const char* name, int element_type, const void* content); +int paddle_init_param(paddle_pserver_client* client, paddle_parameter params); /** - * @brief paddle_finish_init_param tells parameter servers client has + * @brief paddle_finish_init_params tells parameter servers client has * sent all parameters to parameter servers as initialization. * * @return 0 if successful, otherwise -1. On failure the trainer need @@ -61,34 +71,43 @@ int paddle_init_param(paddle_pserver_client* client, const char* name, int eleme * paddle_begin_init_param. Or simply exit the program and wait for * cluster management system to restart trainer. */ -int paddle_finish_init_param(paddle_pserver_client* client); +int paddle_finish_init_params(paddle_pserver_client* client); /** - * @brief paddle_send_grad sends gradients to parameter servers for + * @brief paddle_send_grads sends gradients to parameter servers for * updating parameters. * + * @param grads the array of gradients to send. + * @param total the total number of gradient inside the gradient array. + * @param learning_rate the learning rate for the gradients. * @return 0 if successful, otherwise -1. */ -int paddle_send_grad(paddle_pserver_client* client, const char* name, int element_type, const void* content); +int paddle_send_grads(paddle_pserver_client* client, const paddle_gradient* grads, int total, double learning_rate); /** - * @brief paddle_set_param sets a parameter on parameter servers. + * @brief paddle_set_params sets parameters to parameter servers. * + * @param params the array of parameters to set to parameter servers. + * @param total number of parameters inside the parameter array. * @return 0 if successful, otherwise -1. */ -int paddle_set_param(paddle_pserver_client* client, const char* name, int element_type, const void* content); +int paddle_set_params(paddle_pserver_client* client, const paddle_parameter* params, int total); /** - * @brief paddle_get_param gets the parameter from parameter servers. + * @brief paddle_get_params gets parameters from parameter servers. * + * @param names the array of names of the parameters to get. + * @param dst the destination array of parameters to save to. + * @param total the total number of parameters to get. * @return 0 if successful, otherwise -1. */ -int paddle_get_param(paddle_pserver_client* client, const char* name, void** dst, int* dstLen); +int paddle_get_params(paddle_pserver_client* client, const char** names, paddle_parameter* dst, int total); /** * @brief paddle_save_model indicates parameters to save the parameter * to the given path * + * @param path the path to save parameters. * @return 0 if successful, otherwise -1. */ int paddle_save_model(paddle_pserver_client* client, const char* path); From 1525216078c6fbaa2f3cf201f52943ed11f81c63 Mon Sep 17 00:00:00 2001 From: Helin Wang Date: Wed, 10 May 2017 17:30:27 -0700 Subject: [PATCH 42/88] update documentation --- doc/design/cluster_train/pserver_client.md | 41 +++++++++------------- 1 file changed, 16 insertions(+), 25 deletions(-) diff --git a/doc/design/cluster_train/pserver_client.md b/doc/design/cluster_train/pserver_client.md index 7ecd4cff06..8c4ed46199 100644 --- a/doc/design/cluster_train/pserver_client.md +++ b/doc/design/cluster_train/pserver_client.md @@ -21,32 +21,23 @@ typedef struct { typedef struct paddle_pserver_client paddle_pserver_client; -/** - * @brief paddle_new_pserver_client creates a new parameter server - * client. - */ paddle_pserver_client* paddle_new_pserver_client(); - -/** - * @brief paddle_pserver_client_release releases the parameter server - * client. - */ void paddle_pserver_client_release(paddle_pserver_client* client); /** - * @brief paddle_begin_init_params begins to initialize parameters - * on parameter servers. + * @brief paddle_begin_init_params begins to initialize parameters on + * parameter servers. * - * paddle_begin_init_params will be called from multiple trainers, only - * one trainer will be selected to initialize the parameters on + * paddle_begin_init_params will be called from multiple trainers, + * only one trainer will be selected to initialize the parameters on * parameter servers. Other trainers will be blocked until the * initialization is done, and they need to get the initialized * parameters from parameter servers using @paddle_get_params. * - * @param config_proto serialized parameter server configuration - * protobuffer. - * @return 1 if trainer is selected to initialize parameter servers, - * otherwise 0. + * @param config_proto serialized parameter server configuration in + * Protocol Buffers format. + * @return 1 if the trainer is selected to initialize parameter + * servers, otherwise 0. */ int paddle_begin_init_params(paddle_pserver_client* client, const char* config_proto); @@ -55,10 +46,10 @@ int paddle_begin_init_params(paddle_pserver_client* client, const char* config_p * servers. * * @param param the parameter to initialize. - * @return 0 if successful, otherwise -1. On failure the trainer need - * to restart the entire initialization process starting from - * paddle_begin_init_param. Or simply exit the program and wait for - * cluster management system to restart trainer. + * @return 0 if successful, otherwise -1. On failure, the trainer need + * to restart the entire initialization process (starting + * from @paddle_begin_init_param). Or simply exit the program and wait + * for cluster management system to restart trainer. */ int paddle_init_param(paddle_pserver_client* client, paddle_parameter params); @@ -66,10 +57,10 @@ int paddle_init_param(paddle_pserver_client* client, paddle_parameter params); * @brief paddle_finish_init_params tells parameter servers client has * sent all parameters to parameter servers as initialization. * - * @return 0 if successful, otherwise -1. On failure the trainer need - * to restart the entire initialization process starting from - * paddle_begin_init_param. Or simply exit the program and wait for - * cluster management system to restart trainer. + * @return 0 if successful, otherwise -1. On failure, the trainer need + * to restart the entire initialization process (starting + * from @paddle_begin_init_param). Or simply exit the program and wait + * for cluster management system to restart trainer. */ int paddle_finish_init_params(paddle_pserver_client* client); From 31654e267f1b2645de249b96ca5e8099aa030a32 Mon Sep 17 00:00:00 2001 From: Helin Wang Date: Wed, 10 May 2017 17:42:22 -0700 Subject: [PATCH 43/88] fix grammar --- doc/design/cluster_train/pserver_client.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/doc/design/cluster_train/pserver_client.md b/doc/design/cluster_train/pserver_client.md index 8c4ed46199..96cc649b2c 100644 --- a/doc/design/cluster_train/pserver_client.md +++ b/doc/design/cluster_train/pserver_client.md @@ -46,10 +46,10 @@ int paddle_begin_init_params(paddle_pserver_client* client, const char* config_p * servers. * * @param param the parameter to initialize. - * @return 0 if successful, otherwise -1. On failure, the trainer need - * to restart the entire initialization process (starting - * from @paddle_begin_init_param). Or simply exit the program and wait - * for cluster management system to restart trainer. + * @return 0 if successful, otherwise -1. On failure, the trainer + * needs to restart the entire initialization process (starting from + * @paddle_begin_init_param). Or simply exit the program and wait for + * the cluster management system to restart the trainer. */ int paddle_init_param(paddle_pserver_client* client, paddle_parameter params); @@ -57,10 +57,10 @@ int paddle_init_param(paddle_pserver_client* client, paddle_parameter params); * @brief paddle_finish_init_params tells parameter servers client has * sent all parameters to parameter servers as initialization. * - * @return 0 if successful, otherwise -1. On failure, the trainer need - * to restart the entire initialization process (starting - * from @paddle_begin_init_param). Or simply exit the program and wait - * for cluster management system to restart trainer. + * @return 0 if successful, otherwise -1. On failure, the trainer + * needs to restart the entire initialization process (starting from + * @paddle_begin_init_param). Or simply exit the program and wait for + * the cluster management system to restart the trainer. */ int paddle_finish_init_params(paddle_pserver_client* client); @@ -79,7 +79,8 @@ int paddle_send_grads(paddle_pserver_client* client, const paddle_gradient* grad * @brief paddle_set_params sets parameters to parameter servers. * * @param params the array of parameters to set to parameter servers. - * @param total number of parameters inside the parameter array. + * @param total the total number of parameters inside the parameter + * array. * @return 0 if successful, otherwise -1. */ int paddle_set_params(paddle_pserver_client* client, const paddle_parameter* params, int total); From 8673366329b11bb102a5639b516c84d938f25863 Mon Sep 17 00:00:00 2001 From: Yi Wang Date: Wed, 10 May 2017 18:17:11 -0700 Subject: [PATCH 44/88] Abandon my change to the Chinese document --- doc/howto/dev/write_docs_cn.rst | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/doc/howto/dev/write_docs_cn.rst b/doc/howto/dev/write_docs_cn.rst index 5d1297d079..d536f53abc 100644 --- a/doc/howto/dev/write_docs_cn.rst +++ b/doc/howto/dev/write_docs_cn.rst @@ -2,13 +2,14 @@ 如何贡献/修改文档 ################## -PaddlePaddle的文档文件都在 :code:`doc` 这个子目录里。源文件是 `RST `_ 格式的。 在编译PaddlePaddle源码的时候，可以选择让 `cmake`_ 调用 `sphinx `_ 从 RST 文件生成 HTML 格式的文档。 +PaddlePaddle的文档包括英文文档 ``doc`` 和中文文档 ``doc_cn`` 两个部分。文档都是通过 `cmake`_ 驱动 `sphinx`_ 编译生成，生成后的文档分别存储在编译目录的 ``doc`` 和 ``doc_cn`` 两个子目录下。 如何构建PaddlePaddle的文档 ========================== -为了简化大家安装文档构建工具的过程，我们提供一个Docker image。 +PaddlePaddle的文档构建有直接构建和基于Docker构建两种方式，我们提供了一个构建脚本build_docs.sh来进行构建。 +PaddlePaddle文档需要准备的环境相对较复杂，所以我们推荐使用基于Docker来构建PaddlePaddle的文档。 使用Docker构建PaddlePaddle的文档 @@ -73,5 +74,5 @@ PaddlePaddle文档使用 `sphinx`_ 自动生成，用户可以参考sphinx教程 -.. _cmake: -.. _sphinx: +.. _cmake: https://cmake.org/ +.. _sphinx: http://www.sphinx-doc.org/en/1.4.8/ From f1b89f93204abc0db45ed155af049374fbe4b75b Mon Sep 17 00:00:00 2001 From: Helin Wang Date: Wed, 10 May 2017 18:33:12 -0700 Subject: [PATCH 45/88] add illustration --- doc/design/cluster_train/pserver_client.md | 14 ++++++++++++++ doc/design/cluster_train/src/init_lock.graffle | Bin 0 -> 3090 bytes doc/design/cluster_train/src/init_lock.png | Bin 0 -> 26774 bytes .../cluster_train/src/pserver_init.graffle | Bin 0 -> 2622 bytes doc/design/cluster_train/src/pserver_init.png | Bin 0 -> 28853 bytes 5 files changed, 14 insertions(+) create mode 100644 doc/design/cluster_train/src/init_lock.graffle create mode 100644 doc/design/cluster_train/src/init_lock.png create mode 100644 doc/design/cluster_train/src/pserver_init.graffle create mode 100644 doc/design/cluster_train/src/pserver_init.png diff --git a/doc/design/cluster_train/pserver_client.md b/doc/design/cluster_train/pserver_client.md index 96cc649b2c..c1cb93434e 100644 --- a/doc/design/cluster_train/pserver_client.md +++ b/doc/design/cluster_train/pserver_client.md @@ -2,6 +2,20 @@ For an overview of trainer's role, please refer to [distributed training design doc](README.md). In this design doc, we will discuss the parameter server's client library, which will manage communication with parameter servers. The library will be implemented in [Go](https://golang.org/) and made available as a static or dynamic library with a C header file. +## Parameter Initialization + +The parameters on parameter servers need to be initialized. To provide maximum flexibility, we need to allow trainer initialized the parameters. Only one trainer will do the initialization, the other trainers will wait for the completion of initialization and get the parameters from the parameter servers. + +To select the trainer for initialization, every trainer will try to get a distributed lock, whoever owns the lock will do the initialization. As illustrated below: + + +The select process is encapsulated in the C API function: +```c +int paddle_begin_init_params(paddle_pserver_client* client, const char* config_proto); +``` +The selected trainer's call to `paddle_begin_init_params` will return with 1, and the other trainers' call to `paddle_begin_init_params` will block until initialization is done, and return 0. As illustrated below: + + ## C Interface ```c diff --git a/doc/design/cluster_train/src/init_lock.graffle b/doc/design/cluster_train/src/init_lock.graffle new file mode 100644 index 0000000000000000000000000000000000000000..fa9149f21b1311eed48ef72ec55e556559d0fc94 GIT binary patch literal 3090 zcmV+t4DItDiwFP!000030PS4sbK1HR|J?o+zWI6*kl%Jsnlpv~fhHupLSQm;N7xFu zF*dbLNZQH&epfON8=Dk*)8`>G!4K`OR;%5gwX&?^e;&sU_dxu>_T2Xed=4FO#5Fz3 zc8Bi|I_>h2cJR;1+vES7UKZQEs}eVH>>%W>I)(FUkvllb=Z(q4A^H4i`;@ynueRD8 z$jIkQ^#kr;6o!*``TT4)%VA2DGrch-4DwgLHz9ubbPmFffFWmv)&W!#Nl&N)cP-lt zPu{*cz9&y7h8fxquh2%(s zY3Lc&$7v9z)aB)rz6EUF)B0Tt>Yj{jbICL}#GH&$1dFnzPG7zp@ko;nxg$jvo)5V% zq96;0oPbp2`8m-D%2;%8ke1OJ;R#vQzQ$8MT`??F7vimm$Xel(Lspav-J?Pf>D{kN zW5rSVsgGwXy;hplFXQTfotGwAoUKkGTLz@>dE+hIq=?-I99;VL&|W#n!l>wwLvicH zI9m2+lA@2gfc8L&$#bS-*Kq8in|6k@QjEGa!6t}U()$EEC!%tkXD16%xS&3Mio=vX z0^-#pBp>lShxm6qdWS^rF9d%n?WhpXU*|s~j_qyoZ7*}!A~s{JE>YBe7Q-; zz314;<2q~Ec1Zhal48RUE$g(*7f(HNIwo$o9>q)Zi7ETdeK?>%lM(evr(BDuSV)PQ zm{WM2N2;ibk|gr{A;)VA@nC9!1 zS{c-iH%pT?Nx+xU~YOL^YOUb#H&PWs^^m(oP?xQGlCJe;^vDJWW&-XT#DD64S263 zd%b~cp0cJxvUKI%w|;oX-(okg*^fCH^6<>KH9gbAv`9iybP-Gka4o6m?U&m-&ke(Y zb9;ww?_3*><_P-%2@wZi@jya&;?~KO{QV5HrssG*5dVGa6ASwIcIXr0(vyKRC2$0) z+X?oq+wf6V5d8AIzz&K=4>ZW=K|}N)ii||m8Hpq@5*aZP;HC$KXC#UWBvh{LyEvTs z*tz{=(i@>Y9EHTa!zO?odi$2OA`k`c*1SVp)wM$osPLGC05#7W9yyUwyAZhPh)eca z>@l@1;_#B36BI=jBxt4vjZ-7UL9)gxvZkp>QzY>ZK^*P15J%k|adafBG6Hu4x7M{i z;_MOUk4BsgkU*ntGa!KBtL~k*xt}RTMn5n6WzC&fWWu7bU{{lO&5_aM6(-;_`ByC(re`!{EoSXph#d%0jazQcGeYf zpKI)Mjh~QfVDr<|_6g^B=KUczpa)|8lsOJT)N(qK6iL=(iHAl>^yw(&Bu&>8g-3u# zyr@RFBmeHWBfb{ypdE2X768S09*IcibuBT*+QZHsc76`*1XI%_!Oxjm)uf!N2{5El zbQzi^t8BO<=5$pQbqUcR(WF0RxFft0?C86Wb~I6zWh4Wh>oDY@uetZIvxl8Oezfy1 zn_Gcg7Qhvy+(|o z-)*+(jltM7M%6~O?bNS&&X*&I4 z;bI`v{gKxw;?8}cRQph33?ShGB$RNsj&N6>_HJsjQEQcq?)~Y#*XlY$3)HGV3@Z9l z#4usio&nS8KzMvGhsDmwD1U@@6#L_zVf1ioRKwj;Np3TSPNn?R>sn8}n}XL7oARL2 zt$<``FqO>nR;6qYD%TCDc@#BTtc}dd2pZ)C{YDW83zr?fFfzww7yf6sTt}q)=ukBy zvzF?ojq*j&sCEpho-^rHK)uR9kkMS-MYudpdIK{(`O${@gIo)iZ044I$KxuBb`*)JL%p@cX3ilXUTx5i$Bgu<-L>s$xhn#F=X*RXTGAF z3bOWtomAR8>A%5A2e10T|G_>g?o;eE0WZ5jQSbhFm~$yCLYoK>QFT#D;YlWz6$@JIcc^PC@dxKQs#S+_nMg zuxyzo%8X%|Ez{f^8d?U)?VD@V>(+w)bux2i3@m*HLYkzf^Yac>v(!L!qcf{U3Gk>< ztVGiReP~$!Vc@prIxnsp#HroikBsX9W4QR-|9I2r(WKg>Kl!{|H9+#fCvjOQfSg1dOw7M1)SbAN>mC^A1KDB=B)Ovwm`1$J}rziFWsQ9)OpF~f7 z;7?1}zWiQHiLYkF8RZ53==qeqNNu_4n|quFI+Wwn@HWVxOH&QEqVl(*3ZRuFO&R#Z z!G@RaiB_kKlsRrlZVXVF+7=1U*>ZsO56~0f8l%3p1ACDE%sshwCz=hv!o$@v_Uv5} zqDRRX2#fF(3yHtQb^sp;nY6v+$M(yq@@Oud?u-$9^;35PA`ap=@$8JvxksYFI4=~} z@i?Q;Sp8gvS6#tn{5iB**3XKsZ8FO$-4r@dk2eiB2an;}?sG)CJZ^i#dW!l%WuC$vWIFf~i=Fuvq!p^N2(DP>%`{{nsUc23-3I7(F@ zpy*G~H6(}tbZZqd{v8wyXWqo#iar@XpyK_efJn#B1s=L2+=V`XfM|^f3DF5`p zW#7%Wb=hLe3;CV6Vatn15ZPhRiwp08(0wS*uxmMF>qgbRkZf;)OW%en5}on_jzrs* zY%QF^Qvv$*KG=em2JYrStG5bX7<%Ii?8i1viN6e8h`w_?943KbelDRT%&$-@s^QG@ z?a!Wz9njl$*Bmunk?C0!)H5o2aZGEBU;rlbgKo_+hebAs?C|~3;Ov@>@QN_PYYVkc z{car@)o#gXeFB>Dqcba3URM8{+=c&-c_<3KB>xJs6+iLrJVShX(dc4?r|^&;smZNs zvTC7z&?IzoCX(Ttze7hf=`ikl_C(v8d=K$S=w%&o=@J^m8%6<5f~`x}dHQ;EH@(@m zdl3QYRvQs9*lycl8i=9|lMb5EFd8#@dIXHp z`*M~2_`ogp7LR>?qebi9dGV0b6ggwZu+}K4O?<4jC~r<3;%_8?9va}U6?bhbKKfZc z`A|lS<}Ckh@c;k- literal 0 HcmV?d00001 diff --git a/doc/design/cluster_train/src/init_lock.png b/doc/design/cluster_train/src/init_lock.png new file mode 100644 index 0000000000000000000000000000000000000000..92404ee6d6c0f9a7727952bae3c869ba338ecd7f GIT binary patch literal 26774 zcmdqJbzGH8{69!{oI{s%=b=NoyHmQmk?wAgKGK4KiU^`ecS{QhlF}&B(&!HN-u?c5 zyRX;&vHR!V*S%iOanAG1GiPQ#?;4V{HI;C&D6kL^5O7tL<#iDdkW3H|5VbInz&j`R z>e%3ah`zc?vIun(RQuosrkApbF9HHK;lqE32+xbiK!Z7FePcgk4RtYFPj_x>J5L*X z?m%}h&>8_jJWve$=x*<4O&jR$=HV+AC_(>E3o-EX!`nRcwEr~mbCsYs*3hPv^YpQ& z733D+hS5u6(bCe2``9^%>B=kq_i^w_g5Jr`&r6JlCm|ThnE}1^FP-H9~FOiS4_^+-OI<`*B4x0Qc(P#mj9Rc{?~o} zZLjU*>E{VX!N=KF#lz3u2YlGi`k{A{eE)s=|GCBg{aRW+&i3G`|M@iUf1m!}-}|q< zIM2g_|8E2F?EODOy`N|}*>NojT5D=shRODs!0}+1}2d!%=+zxUOnds`$ zzyE-dtIIB{oJjIQSC)zCrAB3(s!H%jWu0fAP>D@s+f-j@47;je6U#2X|G0Y@H2!YC zeexo#$+yIJ!dD_9w99vA%x6+?x5)J{gnylpW-YP_W+FvWp*!9!jR?Q}^-1c_ zm2Hd9HoL*=54VSHEV&vftcc1<2U|~+;d=~J zmjtsfJ46hmkV+jkZ!)VFSec&pV)IZ1@0HKJHh8>Wnayrg7doq>)B&z}QHK;1hp0U)fonE!(;(Pn0kFH8}?XoIqrp3Eln1QW)Dh-ked7HyfJ(`g)Qv1fWSk-rn^<-J8`{J z=!vwbk6Db1QrCW}de)svBh~4?)YRxW>9N{=Jt_59eNlMXt8e%jLuu&gh(@81OeVg> zIRt+$j?|(c&BY)dqD&E0d=rO(*t!xkFW@p;6Mla)kz6Jbf0u6&%=Szo_*~?;8@V_j zO?Oy$f}A|MUf8J(D+gWU>U#oB4bzKDn!t5Lddsli22TiQNK>S80&1R&LX)%2E%CZUr#*nDv2S$zPgZ<|`%kb;%ua*%X<9drLlrjd>U2K}? z-Wc)B7}aQ(gAocnRTqDD36@DTD)xlr?LnP+n@!RqA-4i9l!`@6mZIa;kRxceI_662 z)5aj;V@f2A=K?a7;0pZj#?-&>j29LYtokpxXGmO-t;rK!&ly=->#aomy?6auCt=l% zK$u~lyK8;8+_INbuz`+8HM~9ZQq5fyjF~t-^+;lp2~x02qr-ThVOzjay2P4;KCKj3 zL_3aMjxBSR5%<6IEyFPxbUl8~zm2%ruh!EPjU$&2oSvP1sg{q1OHlyBWY#RgI6E;f zl!}mCW5`a$Cy`G?bicp({q*Pjz<&H>5iwC4xn=8ShSaLQjuj^J%kjF%caX@0%xar~ zZo&^1rx8cf;^L>U}QF-g0C z)gYF5KHrpcg%pzQGdpf!kLBUOHCDu$7_uh~C2&){D)n-dQ0#I0LLqnV=5?grk$fw@ z*2}MyA6BwUvm~6N8=lEN-H9UScTfjgq*^Y!!An)*9qeaO&mcR9%GlPbIjlX}`*6n+ znNogS$3wVI4k}{PX8Dsg|F^v22aS`GDWp72Y(Wd^flE#9;to6==LGpDs*#aFtIvhp zlWkl+2MDeseyba$IqdDY@{6aGvHUnun2*a{Nq@T;6HOz&#=}As`g%3|);-mH;hnPf zdpnh7CgpDZDc&KfzQol(nrpu|Nz4-2t)!xJ>W?~aDO~z9<36!-7U*JX1}$U6L{bg} zn^d7|z|5RVNsvMgTUgGUoTlj=)p*fTDTVP!DRUHNx?M~^c`zZyuoKDLeh59ER%*8S z?qu6MB|4qB9hu5uRL4k~gSkRcmu7@yrW{PwS!B~pQ6*;d+2}i*wybjmMPhCH_l|uG zc7yfl318Qv$Nr3HNNTe^=rm>>!B4Y3I!zU?`nYGV1pRc!;v#|); zCKf;-UQaHGPhGw%TV4Z6ecq6em-k8ug73eJ%s7{w_Q77J8f0N zNDwQxiGZQIWMU2nJ^r;AQ(Da!p?Cyt{CqPX%rBzGbp^dLEXJ2zQf)U7g(*06DENa) zXE{&T9iPxl$HljlHn!Y#PL1zW^pQ{gUY_ZqF;wT;ef!{?>T@jD#A^(l!XpG!pGC3~ z@*0Dq;1!$IYa1a9&lx7QyyLiNzVV~FwnLv*aS`9|s;U%;nXj17oHspGnMBm|77{S5 z2&_6YlO(F$SZQip4`vW}s6!4Kut~Y^ZmmTldq}cVM9RdzEWu(}+)6m-)=&CaQm*vX z>J`{BJuwoo&)yE1dwO<--|vjU`(|!~&UU`%b|+Fd5)+Dt_SO^%CWe-X`R&!aE%J%e z7T{3{x)7Db@(z5n&5)Ro`Ww2u&6{W~Dd?=rPUp~&e2$LSpPy*X?IuD!SF>igS2;PrUBlBD2au6EN={F3s^5OZJb_yq1*XX|j2}sC} zN}0=kHK@^4>G)g0>MEhVjVyhjpbx1gC0=du@bl1Jx7t*lcNbW3-*IK9YU#PSN1;tnVhnt`Q#y?=nhMtx? z6*GlC?myrLpwfYyY4-yb=um)Dk{oTkcT#-9tuFf~>Frbb*;?idmu6Xg5g^0m$! z{QLLrY!dcK&m9{B=}VQy8-q9VKi_pxLxPwbCLX>4oEvS!iVt&EU$*8PoC9J$&?4;$`7`?1zk$RE_nI) zX4B(+ryopd-u|Ft&>41T|i!04*-h*Z|mVUaoX{vkfV+iA~u`Y zl_qz}Z~*{w?4|AU3rfJmIwv~RHk_~4DlL~w;5%%38-uNu!)nb#3_)sGABu0h_i2wO6WNNFM;TE3m(sMxK(8+U+#Z#B50X2@)|cJi zU{KR#nxd@Bs#h4Z1u~o1CuaM;(tYW}k>zs9f;5ADzsS>*{p0PY)gQ|(XWu@$^1}W1 zXP5$UGyARO%dGq2SxN#ou1+?)00;%x_%Em&)^0Dq<)ET`I8rB()`yCu8`ha!=Negv z{-~*ljfWhwAmL00N_w`RC`Kf+yz)tjJnvoJpvn5iE5VD-K#!vm5dVQ3psMR^ElKj3 z!B-}5FkoXuGTZ1;;8xLb7tyj9SL9JQzkl+>$NQT~<8s_uWTk~>%;??)gB5g45rs}2 zxDEh92*6_UuC6~fpBa~0pZ*m5YSt%12p(;hE`iEtQsT1U>Z?V_ zTa!kIuDesU%fl70>J{zUa};|4?5YFHH%H6dcXz_P$<5?!XIxY)7k?R>r`<~7=Es|N z6JkGW0W|3H|Md+S4QHXncNf4FnLL0=y(g#;DHJvUME_{?&LV+2nDvKjwPvwX1zXfG zAZQs)k@1U=qKjyi=@$ynKl7%ZAAh|UWQSm&EPTHQlnOJphJY^=eD}yx)nHe-x!TN_ z6h9kd$PoA0n<}5u&7hW$yg%2I-yQQXM*rrt-0Yo3A$al4l_v|D=tNAV$d`Cr7K2A2 z(bhm#0ddP9$yY`RCm@Y7MV$6)}7oN^(pGYROEhVaU4xg2`zF3cLs?yr5; z{;n3vkGVI(K(;9K>}+wFd%eIfqL-QQC2_Rza*TfxHc|02$-1nMfoLW9i{|6BML+67 zK%8>`dIggahtT~!AjZj!eE|ap$e_*9d>%{ijUpAzJHJj5HA@j?Pow^;BYSS_dvUOcOCex28qm!S~Cmnj;vb9aOc)I-=GdnUPMTPJr zU@(=OadXpV)4E^cV!?KQ=4Jb&Hgwj>Q)d`-WBh?oCD>|$0`tjpZNfe}M^P3pfj^Fn zl$v$a?8ULpWAruOplJotlcvJJKfV57TLrX(%X)%lSI*W$ht@q-WSr@SJDAZ@gOZ(Prd7k1|gpV5;l-6TA`BwJxq{P>E?Q8Gd)7 zN=^RA(w|5B23)-iYd(qrYDQ?pdQe$!RncZ>qc`U(G%R8yeU-emadX>qgwi zNDGjb00qNPJoQEW^A-U!Wis24cJz!kFYAG1;%R0LZfb=wI*0Q?<VYs+DQ3zF0+wiC_9*vx~jeJMmNSs=I~Nc{43cmyqp@-2^(c(t5w4dsZ#zI ze>O!w%sx{8>Z{f8_LqrfU$7@Jml$OTV?37XGjl*G_WF#)wzV1=Xb*x)3Zi=tr^rs!7jLc2at}Gn11S*z7QKEZlQ7c~!C4z-+4>Iz=n% z)~>EX&vg*qgG##ag2;DwNko@9>~D zkL@qGt3HfWr^1VsB{_zuLpVKNsH;4uITCN*I}d{ek=tzQ;(PxoK8|cVY)u&%*vo>W z*f=Bxifh8-tv3j{zK|`xAmm`C!hk*AbHm7Y2#kG2r6*1pBEBTTqSU-9lAxWWCx)Ft zD%2+^-diM<$^qRE!2O%N22kOIUF>)fYf`c9+*{Lpg8Jh6b^;gqd;56$7t-dffIdty zS@~*nR#>UawW|SvFWE#Q`M&`_m7N_b`p zE8}$zl)N}!o0J=KLHU|$T`xu^>Dl8Z=9%NU$iFamy2ulgb<^%%oMv(wtYyN?d6OK> zzIUw83(9d)_+1l=puuu}+yf{Sh`TlE-)BJu7-^?XiCAde10p4W&6s)!0>=P>r!S6e z=?#u~&}UQCF-iTY(Ag~xppH93*pwe-B6-3*OE_F+Y656Yfdb6ejA+$$tk~)}wIHq0 z=6hvA29v)|5$UmBq)?#F2y)eg2&A zW~7ah1ca*WCoFQM`nn#_-STKbfEIXmUdKO`G)979S}kmmcO%EOAe2z8BDo!i znDc)~W~KXg05RiTp%pVIn^KNZ;vE>N_sD8IeEh$M1Bpsa9Vz0><+~NSsEB`2qNJu! zl;WUGkG)a2H9fs)h>SA$p=j61Cqy-!xTU9Pm)hLVoy<%MB@auYSpmu!vk1pGf z(7s_#bZjDGXrwPa+w(aI*I*YO!FBuW3jWsbFIw(K6Wal?9yV7fr^`U*Hu8F3x z6(Lc29#};H_j8o?UwQf!UNBvuWcy1A=0rV2#YfV?DPvZ~e@XLo5@>n7c;xc&Uj}{`6WM%B!%6g?EpqoTb4%~*4FB~-x~0G#__s&4 z90kJ*kE;5VS-H&gANyn-FcBP&wIFuUkf6efUlMN(axoF{q3cD2c%mouvIg4>C|gQD zDkhHKvK~~7vyr%+4O{4S66;1|M&UrMNVPRft=>=w5slZg+T&9JYK8pOUKCPbo+YZG zEnk4VXYeusquzw|Hf5qjVyREeAlDL5-kX;!cbw=fq`h(f@PnvYTJ5gzFvV~aHq|&3 zY@iL)-*5=exV(o6H@`8QggPzMmp4!O&W7of?_h^b(gJ<@=k6LvF+|WEw6M7!Tu4PCJzwg+Y;`ECkbX@1 z_gWiYw4~nZRi~FS<+#bxreMGXCs2Y&u=RDKq=BT7=rQ^-bEDVn#k*DsC276*61l?^ zEJ4e?6p!OZNI@#VWO{A>zj9)if#fAp`1mEKDksXabmT{AAfp?dGBu_*fz_%`PQtFw zdTpjk?uW943q7ayGx`IRPfp9ucEh(%UKt`old<(sHTGAeY3H6c05iuDs79om#$~S% zgsEh#wbyTVAEy8y$dJ>(Ws}Egz3wh3Y?*3I7 zb_o#yrVJ3Fe_C9#FD5+jy#+Q&f_*?CH`~@3q8O7QUNngQHrWfqlhcj=O10a0xzdh9nRO8rP_;uh!4qx&mw1r8N@YD8LfQh zv6lRM=bO%Tp0A#zh!Geax$Gz&wWyQ}x;jf9P8c`R3ozb{r>+V$-8v$W3Gk&a42U;m zEuSWI#X?DM-bgW~fvHssxOtcTJ52na`UWizQ(o;n%;J+viR6d(RdnaG5+fT1tIp7k}XZiYEd-L`K$ z5Io(Ns*cfV79j=yv0`|A-$>B3#H3yo+4eLrSnPXE>ECt1;hks0!#t)H4aCh zVQ|9JdY30HKK20j#V|Ei5+j+&M^=mit)$6q@jYkWs~4dups0Q_&$jK6eMJhs;EbXJ zmk8jUX!phA9_KIIRgod7+BRQnLKw!;L00q{4WGKmVijRK3oamCoLgSA(TjtK z<9-488K*r9OuW$XMgLPEI)kpw52jJ>s|i_302v z@1sU?y4_J+=4;tHrra~bbi=Jnhv|6l;t{af@kF;ET`IZ6LB;}$?LoeJvjc$ejxgoV zzBRqfGxZ?nwdsc{QSn;$#;UuFw1Yp4RuAaA1$t^t8sD2%sh<6(=(Oa!fMEb0)kFu? zp<1Nw0{D>yaCYDxnKygUJ>LMFUh5<66d=^^?LWU@e*buT_LW{z=t0BKlTRKRWq^ae zee#ih3ix!jwFLrzSh~>{GkN8&0a5

?^QzZmlwkCBs^iLIKjO8EzT)3arMfVTO!Q zV1xGaJo(^scXfions$z%$kps)1wWX5l?|QDIRlWb*#Da|kp=K)fsWHHUv(1z$o21^ zUwI5(&o%(a2CfI(x-WT=2#!VGXrmsWp)^}nZf?aN80uQ^&$mKGPDnWDi%DDp<2rH# zw6xsn2$~jay$-`Sz2Z^pvwB+go>AMM^F0A42G{Yz$5pZn1@D(VR{3ejY6wH&@I;yaw){-T_^J!&S#I6A4vSn0XP^tB7wy!a!<6Mw+vVc#Wc+o2W}(*dl>df+2n(h4(pH^pW-5byEe_x5}_QJ9<};>qJ;qeLK3e7UDB{SrLQ#q$QPU%c`mz3bWaam zXb|XrY`F@QF$gjP@)f~h5+=eE1{aetQ-jy~ygwKMdI;ys z1>{C}zbGkno(xh#sL__R0`AxMH`5GGkzE$XfQb z3E>jR)_8XOCgH6QOu&0&W#fAZv?4m*)^{y-isWe!R8rr%+@3tbbOia~;XdaaJf*hT zWG9ryckLBN#3g!6b3CqQJ=jTfajS4j~hTuKJY7HuQaJ5J&G{ z46bB5HVqk>@F^A?yNG%QOGiJ|aNf@uY${{Q44AR*N2kA~uGV9ch88~CTA?%rzzaBa zD-&D#o3S{DBePrxV|i%#ov&^L98w+x|B*{&v|DwPZG5a6ldIvfRnfxK=fk~dd5ks} zcJCp&{qg(iR(+WD(VA|2NC=%&#U^Vgf`d=ly$r{{Ur9;Ix6Zd(ihgBv~aTnAWv@)!zT!kNJ&W1>ORj98k-eR zd(chh8U@pJ7GE`%U5L8(hMss677Ke)hej6SyvF|wlLU{W|Cl<1teRx_y`nw`k#>;= z>nO$A0_F;Q%mkZ6XP$54b8z>X6p8$XYF{li)|(R#!TpAFilr+4kPTRN{)cbPX9YQ9 zF})EYIWP7%SYzR{dEHz#+PWI*6^d!~ReY|>rmX=-9@%l60L7Uk47#Q_!>IX!9^sR| zV2L9}nqUnN`#)0XFpUtA*>%f|bZ{E)#^Am&j(L!JqFko+kM>!y}2{4-IX) z6iGVVCaxF^r?#dD%zW-ojlIWfP%*!&i3mo~IG?j1gbxU%9Hm;@Y?Z?cY&EWH`4Opp z>dMc-KD5~1DZeGVLZ?H+lN1`1io&58AhigLI=s;GWlU1TIM8gF_Dr-nrHUG&iJ&^8 zjrcMc_0C^}noFNPhpURfy7t3#H$G~#OrO_tYqqqP6g7SY>I@sUhVpbGNBy&3b zjVzZbbH>rYshNfOzEU5P-{tN_qqI0Q{rR-Bn|iyfa^`gseWo{(3Lno}6TKJW#j?yA zZqWjPhyjX#WdXZQ&@&6O0@9UQMdm03tbo{3cLI10V<144F<%|2l~aFlSG^sJd2J8} zIVJ{34M4Pz-Ur1+sGCWR2ghqy<-C@@n_d%V_{1>6;Uhb7o*F=6E%Ixho~rTu16Plr zQ#s|A1tFFd%~K&`$l4X>y$F6;^5`%VvkNb0wPo*{>>!Xpm)r_y-;R)%^;9PAw=zOLVZI=v8Fg+AGI86kiW+k@$ zmswW2Om-wMI{v*^JGwVk0I=d1Mg%dwZJuH%GthuQy47ton{lG+vv@T?Kb?_C58XC; zy)NH9$tM83LNA)5ENLJP@s~)z010Vp$~#qjBH(UHKQ(&Zn~ePhg?o9iKkoaXwud=-{EO%UU@!tOx$RHb8sXn`X^z#&6ZA5w9m0GYq`+ zlCfi!0=QKt!uzSp4{cD{pJ1S%BaSg1^b=79F*%E2?A`VHX=8^`3w=uzL6V$fZuG_N z()VTSfDDTv!KD(uWx;+HZ}cGz#7Q3HET`m%lKN-{9~M4i!UO{RIlz7Zl-o}fi5^Jx zewvG!=K@3o5a&8-rY>Mj0Pu=90|3`_XV@)2FbHA^HHV8t-^aFQBwMdx8a?h+(ipeZ zv11tDoXA-7D?$}f2-_S?JIRjyI}wXX_@v1V6AzRYAF!4k#tIOjeAzbu&o!?EGP^!l zpN)=MMjY%W@3;Ds2rt2I0(J}_Ux!r}`%~}&Oaf@p0oYi~--;Q>HBObV^;wN@I_oI_ zjLmLyoh9iz2b)snvS=qW*au5EhPcUn$)Tx&)1v(fI3^K$@6zsF|{4dV7CBAf-G9kyM#ZrB$7aJ)&Rmk?qL6U2~83=FofW)0^ zbC{D2a%NFLRe`zGjF>cgeFvDjH;#>zXccFp_xR$M6c-`spO`pVO_VG7I@3=Ri5WzDf#q#Y+sY-? zHrFwSyCb$StidCn8b8?p>mr&9pYKul)%S%Tr)A}Tzw|FQI?@0#yP)tr24GTee5EWc zse7XEn^M0z^Y-Ij?2v<#Nz)el&m`wMUtMmi*MUDHe$r1Umw*QxIo`gqPix_nM~e^0 zty?PTRFJU)MiU$!U?O{3Uiign6XfcEjQs&9`2weCyx8eE0Cx60+Ztghbpx5Yy62JP zB!6P?PG|=b;=1+{SyMz$Z2Nt8U!-vNMh6c)3LwLFsk0h!?=b$XlunhjtM4(O{i_ZL zC4R)K2&CyWwjkkn?OLpav|JD^?}&;@b3dBPAs73hwQP%f)sYFAnO*3&1 z_Q&e0NN|4=^83d?r#QslAM3X_+kv=totY>NHtY0zpq0TMi^RVtDjJ3>V{3ddhI@g8 zXTZ;S+@Gu|d-tKIk&~n!8U_(U*a)40LvO>%g+AeE5nN7~21yu*qdQzyFN{BpUF!!s zSH821V?tj&_hF=XwxA1G5CulWx!$L9C37mdJC)PZ}h@` zJ481nViQJIn6?4!TzKy_-QeJZJU@T{u6PNkW*cCeS6!C(lW}NrLfh67e{YH6jUEl1)=heCZY#->Dlf6pjFoLVt*MsB1>n%&lf>IEfoAALtFHePuH& zXXW|0;)#y!Jj9-(FwWr}=llc>*y2G%*7$=H)W79`xaf11Y5(UBGB|x^OR|Oj0xpyI#cZ%4gdKMCvoneCLXPJu&5VwPfUkr!`y?S zA6t^GY|Xzl-BjYYAI$-L{y|-u#tfzGJ4AHb3?HUTxH6Q*%u@i6#sr1`gTi$twy03z z+u4U$W?ty}I1S%kUpeVF%^qTSOoMT1M#nj*>K{)e@vSzu^?91;&RdEIcsRgxYecdg z#&!l<_-&BKO6wfUFFSi-Y?=qLMF`=I{;Mpl%F!2Obepo7-#M2-o&w#IqUz~U`d{#NR|l6*B-%_SoCMvx$E) zYYw~hPnTam?>X5V!lw{mWxX}5Ub(lP!^WjO*bP{6|I{Ec1p=-9XXCk(D(1os8`p z7kd1|9+_f7>a=?;3RiVuo}7}xH+By<&=du~c8qa?uwyE^AB$X398^Y|v!!>PH=~2Q z?A<082WNE*p}uhunm>q1cTg_M43k~1K)SbuU4mt>IQ(WGFv4w|?_l1vJjwUOo)qS; zWLVh6_k0fW^FDWcI|yb+r=om8dERN7s(C3IcJ@A})ThZKn z38}@H_>++Nmq4C51KQtkI-^@Ckmo+V*HD>%gP|RD4Z@@=Y*#pq5o{q*HX}y8gYGHGa5FvxO{QIfo{KoPRkr8qG1w=js)@Dwd)gN(aUx5Z@ zl{{1(zfA&85BQTG3~#`pj^79h-M4Vf6DXoqVgfT&YeM>u(l+_i__2}w4e1@Ze*(jK zrdo@JNCUWQI2t>F@wF(_jQ~qG%8e&N8EBO~pL_+|x+bEnExPAxJuXQZ>ELoLF?yIy z{l*`Fe1bQUSoreE^vD`xiZ=VGY3O^$PH-$S501OTL#QTFKXw-q3V#fF zlUhCTYjER(0Z_T|;682ixktiaO9}t^3}LvBUG9zo7%JH%wIJancqUzBJ9MxPgsV&o zy3t%CjN_tu;x%yG1Ej0WmTQ+?Z?{IWUlO&Vzbk!E*INSaIF=p|X^)EY@a0yv)8G%44Qi?`IS z0s5K{KT|KF73r*VoGh`H2!;ho=kOVzh?H)DzY7N>1IWb^cEg#qxA)_4{)Bxmv*&um z>icpJVM~u7?6{c-k)|=~vBH5yOUSGq2?gg} zAjP!53Un$Iy;JqSE#~r0VK`|hl(q5X_cl_u10q(dsyXKFrW^Bad<@PHx9k^tu9jpF zkO;BL2*i`IBi{fi)PnGdcKf1dNz>r~l?FBcb>G>GD~pu{Q5_|W=jF_S83?9*^>XMA zx+E0u@oK>VjOoIG_1vMI7gyh4xOyL5436UT)2tN`2qk`c0eODJU!hr`F*_vQFu*DI zW20Llg<=0}T|WSC>Zuur%(4i@84P<~j?yxA4OT!d6tV2363dt$u6uy*0-X`xCfK2g zt@-3guJg6udu*xVzn7b}_&Adqg2e@#csSdy4{F+j(hOzSz0v`e>2bSW=9-L_ouQ)G zjG(GTH9ls+vuS_Q9_S5VItF_gk*eD40Qy9v4&=OZXzr)QL%x^QyIj zXbUc-kex)h>V(;ZkdHBQ#tFG&(f3q+k#`T#T7v+`t_FX`^yn*astLBK3j%+NX%xIl zBF;0FU{960H3&AK=duwrJ=qxOc&+`rG6~^Ax6v7Ssk))z@ng}-CsW`IOZr#!56`6L zd~zh4A9jMJfX-~HN6!egyZ-(OxyX4)JcY7YCL?+0a7-jWq!5#@bHs%)yk<$!ksglE z>?ML&nYu+HkX^0Ej(~A}YLfF$3a{^Lb&K;fz`z%@cY`v3E30--91EPG@|Bt+`o8u7Tvr4 z0CL(tW&ZQ2Nf9Y0H3sKf#e$DT3leOX9YT;lg` z=i=f(Y(-Wv_WNGj+MQ)`rkYoKAZTf2uDWiC68Q5Rn7OpwyYB!A1=Q6>UsF_M^W@QT zle;OiJ@aq~V4)|@4l$~lpJ1VbC%u(Je1p~Sci%fMSFt~K#~nvOlkST#E4Uv3ck%*De55yLly6If3C zCL8x*K>1eo2iC1^*^Vd3kREmY4JTl6asU+ZP+iKwU|8WBbWjAS32 z%~PXTcvH;uqyF)URWt+fRR)bzNfgxiVGvPHmVGuaLEf1btvDLrXya4`0bBMuDSO{z z54ym0&ZUoRt`J!fa4rAhr0jAbSsNQ*H%FdEvdFxtYpchm%tGpYR15spM3T`v^I&E6 zkMzaA&|ZfHKEc%4r%lptft>cS3=>U!rSWLwQ!f&MqCFf!O5;WR9H%JMRLp;(&G4~; zfk0YQpF@bdKIkMl_HD}tvtfBoBzzjl<*FpoNL2vVY>r-#z8}EP$83fndOWxt?998L z3z5DXK?W-l%8_G|KotC2@`8SbTuBRxFMk#-&Uhrj5+`HgCnh_I$nESXY+L9>mxi`T zLR`ZEiwx4rC?N{gnb%Jilo@G+whe%pV|`*g`tw&zkQ}ks^?nShQwF1LS^uyJCh#}u z3YSPit`YkrKWlgbwd8R04iET6u>Ow3rV+;1u*4SB3MV$;FS?geTIj#>V;u~k*dl#qx@`m}%=9(`pGHgaytq=tNG6SvN9 z)>19;+vl>MoNS#~9QvS=UjS0{F+>sNC6Dibdh9Iiw#T7uw)@2wi^2J+OG$m5S@UoU zwQ-9vwd0HOCt!-37@D%hOo)wzWZ9Uo*H#V_c66y*fe3IluR)jO(Ir5+iOdM$>pV08 z;R{jIdhn!v+u1__eBx} z{;kO!TZ3=eU!@bI!pq+s$0@MIVoZw`yySn{@QCyV@;bFeUT1Sj*lCKkfZ!3TO+Ed| z({O+-$6i{~zM0W^t~2Ylv4?=_;|9U7LJ^W8Bem#XCof*;&!mxXd1CW^E$Ss$S~DrbFgMMbfz#XC|9a(X`|(XGy@5=wctJWTTp&U2g2pV{(Vdun%ZT~-`1ph z&~i#z^v*GHIH>_YzfzJi4F8-!mK`}zQFVa6lSY1M_Np$m^gB55>BfsT$>oKV>ZE&b<9KANdZJH zqjOExG}HNjLB>90jFs8IJ!$dKDHTtT#V$L>@rUr=w}b%#U#5KRr_m0~-MC!@>m5Mv zQjQ5MaPgMmD}f(lJ)Yw8Ozs`GG~*qQL2N@ty|T@ezwd=-i|Y5Mka{v*(sm;wkiM|y z_wt^c1hk+AOBj+#^NO-2{;hg*8lt2NiM+1W(y@8JR(ZGGK2x`BN>3)f!1ACKjgXCO z;_*U*EgOWmzs96rdSpyz@d<3ox`vS&uL4DKJ04r5+efZogeXyWz)m;HOqtv8R>o>8 zy0xIoZgqCv9yt#?DT?}-hW8sa1)aYG>u6={BNJ8|JL&e%JY7`dxu#GvER$7a1O#Ty zhu;O@H2Zij%~X*sgW9)@wNaA5UkMf-dP!bf@MJ?UyA4eDabn_S3fUk8vWSA2bpjz5 zUJt=cdItP>hUMr3ooJ0;A5i}&>S;kdRnOLWoNRx;BqdX@W{#WvnBI%LfUFUtBP_Mt1_);=xP)NSv$6g*ciCQ{KLsdfp{vo5W-t>|ks8 z!TxdeRLTM+5S5f8q)kg#%7}gHWs=-X%p*w;r_3+w5vb;Zx2UI5)nX$mBM#0u>Fxy=p;hOIGaKiQ6O8_$q%A;sGlf26M^VL;Y!^}zy_sL z;J1x^`7BHet4@(^MizncT_TKW%UFLjwEhbh3BKK65Lv7DTbr1U996xIV>;RnSZLUD z=al-&6`#*swJF^hG0wf`$$gcBqkcjO{S^`&1ZLv{s%Yoz%&lf`72HxsfHi)O$268Vc~5Yn{(c`3J8BBL*g@AfwAyx2Q&me(OX+O*9mmI7#AaP0LkX>J zmr5q{niTOdb(}azg~=!?yiY(!6o0yiA4NQhlk=Anz23@(r+OdqlK6AZNtV)gSGm^0dOSSmxB7Fk__e$m zta))5dQnDfb`0#wfc8hWclbH|cno{vQD2r|6Tvt=l%6Vu!#K6)oBzD6oswZLJqj4$ zcMgWeK3DoS@tgIE{yfnlo^yqi2HFeM?DC`lZF<7 zFMZ6U5$t|PC0?C;+k6>Ht4Ssw2v}eNn_b8uBV}JKeS(5qtUA^C&#(9{>UZuwW>)RlkJ&`anT$3{$d>ed~l zLvu?Gn^9rG%nby^I5k22^@&@|YF9*rfAPlznW`tzGrRbpD_~OjuFZfP+5mgd=$x$rm3W|+yxnQOGe&oJ3lP0dUVF(=`v-5@MLziJzfDhmJRS7) zm<2h^om9*K<6!If`_HGCR38Hejr9twbw*31Fopw|3&X|-JD)HWZ3ZVp+IFU7^gl0Vunx(_EVE0#-Vaepr2Y3 zm9A)L$KVwX|VKk$W+6E{wxdCZLDkU{SLTRPZo0jevrGkK>D5W$yU=lJ?K)O>wM~aI2 zz4rA!j_3Iop2y!jjt@FzyW)JGuNQCeb>IB};YC-sWCMf^)0Pe3u=-RTPmKON`JBtu zvL?O4!2O>T(5YVq8MG#xDQx*nWXq%rV4fa3s+B-MaE?L3WAv&HK4Y&=zIC1LdY`_^ zs@2iP-%4Ao5}N~Knye~0VkVDPV-X}-0CS4f8Rb{K<4?gB9f@|<;;a7N{YtICSUc>c za)xkQ9OT6>Kvg-<6`hy1(k>`*L-SA&9e7mlcoGh73!BD?0_}?`7XD3;Rd95WcpHYs zEnkINsp$+hcFpWwUxO5~;r^z0ckX?w2VxomWakAJza6E0w z+{S@w-y@rJ3$5M7&IsTaVSibEzQT<-2Zq2PlMUIM&FcyDG6&DXf;?caRzWPVPR8Rd z81}fNxx9x(#ZIX~Zx&38r+=j&j=|RmXWwuTprlhXdk89Fb@E%{JR0R$eJbi&Sbrx& zd35c+-xV`>l#-Qd#`@STgldT(AbQ|9@IH@=u1?Rqw;t?|Cv1q#bC1Ib+SOn$a*U%+kA_cGw)@R&Hw!B@?ag?7*%iJDGv^>oYUDRjDQ z{6z}@b*4*0lNa|L%s&JE{QA}>h__e$L#mXyh||nK^?m1>hURsCR#2fRS`ndB`EG=XdztX9J2f9>29*z0moXQq%70X zWcY&Cf=Au?D$cOcd;sbtA_V_+cs9sK98^}`JJ@@k8%e(@xr^QT-*2~j^IgTb#M3Cp z1Sfen-E&Q9tBE3Y+~p`lU|bs^Ae@D4N<}#VdJigU-r-LbKfMg!T6a27n=dXu9i)s_ znhb3S$h}BGEZg!_yw_n42CsaP7cPm(TGD{>(|J*na%9 zSX}J5TpoZ%oc9lSlx}Jl@bO;-mW>G{`v?<1Kx|ctJ=KgG=HI_WnP_Zt+*W6$@Sy&_ zO+zCsUVEgqE2qaF#32H^=-;;$AA%3&oI}F(FPN88{otd$w~r~d>!|QXyyuNJc(_B8 zdM-C=whu8e9SFJvH|!|n+e?+*8ltG?T~Y`AbIx(FZwwlled z0pA*ETIexWmo#iIwPi!MPY1pM1ZdSo-_BzzUUvB?*6m(izjekNo^zmH0R~om=t76E zRdTjyle!{j2jOu=1>!C67K2V`w!8YB_;4GFH41}p?}!Nt0~Ob6pK>KGufJH=gshK5!Z?|N9ioW383IEz z2vqZ>g*!PrNEt-x00z|kH#!o*>WQ`05XTPXUSww>NGo)WTvN^A$D%Q6G)133);5<8 zJE(==7+Qk&uKVtn-KQ&*go%9>nd14)HbUTYBa~3gyBZ~O3yHC_oRX6!`=~ZZ8282^pZ}SQLl;u#jrmTI6j(4COyC~a!kMIsok&r3O=3)IHG)h}!q{AXkH2MwXt)P^) z{zF7y+i8YfeTOy510HU`41>e5I;e1F`0ALl)gh z=BZWH@2{ptGLkL0#5U|S%9)}yws{V1@$r3Dd)kY-32K@D*!RCuwF(H&QSmED{$9p@ z+n@r`Hk%n`^LhhlO${< z_sr>o>2{(E$l+KYKpn@iALPUl@*>*(X3TxfZi$)px$cRX@LJI}d5Yq3!cxs3J2Q*K z!L^}p3T|pr2@e^}hoOF}Ix#MKmv|`?ZPhfPH&~z3`F24zpb+G9FsHsu6W;Q&&H%b* zwP>K6J!HE~c}PTfxp)Ys^vdK?sOtZgV+@e)C^(o+6iENEpc-Sa;F(n&>{f>3E%osWqKjL0mYk6Q9Jf zcH!(j_Ynm5-2yK3I)vVh^cjGQ`b|7#JwXuE$NJSa=mVf5Zff`WSKY@I-DwL~dQLM~ zW0bU-Cf(gO74r@ik?JB0;dx0O$xM@RsgP3(our*$*F5L)xDY}9F%w5z4MdKe-~^q) z#@6Kp;-*0Ih>nc$;>y83Df0S9v332Md0>nNEcH$RQ=KJy|4S4-S$+T9ENgGV6&AXG zxDKA>$~m+Fn1z6HejmjQ{ZV$&`M3Kje>dh6uTNUF_M)7<0VD0BNATB75t^0g!A9Q! z0i|?PM#bKbCEr-8obRVlJfzeOPJM=vanzb0fRAA?E58gfvkVKF`-HW_)j81-xG6UY zhI4W~rUJ$g@yttJLhcF{!JLU%_XiRHLii}ETY9^lRoA*0+F2Mc7)edU_f#dnzAwcR z@sX-Q+LS`WhBgM?&yp4yr(f2ApBk!@G98Ax;0G|dA!$i8x))D zM@!e9t;x0c)gTl9O@of_3~FKZw;psVyVeH%`RN+T{dK4NU4k8JEn5M;uQ+&RPoc7G z(wXQM$SN&OIUj1Nyk_j`11YcyWe1Bp8<|U1$T`;@XF3hCvOkEqzU?IPM_xN%3-g}F z!zWx4dD19#q4SoZl&#dDYjs|nn^`QjZ(AnGHMH_-kfyG@wnK`o402TNI@h7E_BsG9 zV!0)R$HS7mTh3#9Ro6tiXR`Y^c$rxU6{+Xzq;w_(ei*4QMOs{0Lh~dxbL9R3gu-;M zR$A&wi=vt}hU;}RM*Yn@)TeF#RI9?uD^e@pCCGUViLi^c=TXjln#Q_Tv{{6>y=%2)n?6e_$8d1$E@ z!|Q3ceTvj3?kBVDkR|Ru1x|_H%V2|Z$qaduZL73!=vFkP=(QmprX!bp$u@>BNgyKsX9+W1?1iqel=(KI=z_{H6YThQ)gJE4uIW@eR zUpA+e8OuhKmcJCp+*~J7)=J9SdqG_xHz`s<^hPrIYk>&Jfe@;RQ|1z+df-NDX5CDl zKxIDs7RuiYesFu|tSg=+i}8WVBWJh$Z_o)&R{r^MH|_R`bwS}Gox)Kl_sa*59c zL~m-TU!uMcIY6=Eta`Qqr*maIsT=0Ark2Lc57)CeS~3q5R?|+?$wS>GPp8&G4Su2O z0f72oMdi!t!x$sheH(dat_gXI@-&1oPPAl_!8QofQ9VHMeC4vE5SNAcRR~co9@pZp zV4^xkrrb)~!xsL`Ry8#)66NSoF3FeJHN-8(ZFKj<=HN`J$d#v}jm8UM=$N_h=HgBOQFD{Qt z?Bh*uj<4-Vc=(~xw^Eyz^l_COgQ$Ov88eeVgp>bSaqxC|#4X+ky4oEHXG#!``k(Xz zqvds-Pc9&*{*l&2hH=}MKj%(JHxcxk6;l;M5X;S_T^Gf&SuM~FcdACpQB~8SQ8w$~ z0xHD!`#9cRKkvJfw|pU2Ih%zJ*+&U9x@H)^LuP$?t&vvl?r)i=aEI8g58jS`a%uP6 zBN(}i8`r2FUY~zbKkOWe*SbRSJYohf9yR51ttau6LVj%S9C{5Mb<9@zfqM-iAx{}D zYVVmV>p?#ZUwkq=h+Xj>aAnA#)km%S^hVL(LA$pF#r6|a+;9T#a@Fr_+aE&>=?>Bu za(1-uh|f38s8a~zQw!1gWR+dF`%YtM^f%3tFUUb|G$*_;FPVsLr}}B{znE{%(LL4u zc1}d@j5q1epA6mp)$6J2yKOiHmVg2yZyK3q%uU%JRzfb?aq2tb0bm%f<-!p~cI!(0 z6V3ZIjrfT2sEO##G)-s^2J>#DVL7yV88xsjs4fyLc2Aps8bW-OlsHkzqJKV`So_-a zJZt4QSqf$L21GX~dL#=h+rbn#(BWm&nL136yEY+R_gME6N5!o)=-fzBu`6T>?}sn( zzf}%?y!?9P>7$MT!umP&qZ+cGBxS>lXF%DG)kR~s0D;faHX$$_mE4TBB_1(3f^9}! z3Zvem`nR&}>0)aV`3jnMtq>9U4Yxi5ubtPh<4NJ2!ekLSD$ORS_D_#@ZHF2O=q$^7 zuuK+^S7?RKw+>jqn69n5ndkDN-Yry%5-N)pe%Si-GC9I-J=tWKX%g6b=(9iLDeG2R zY;oHxG2pqgNO5kURKP3<1U@9l4W)W7;1?N{-v%5NOvzNyullw}!|Z(Om;TWUTfT`6 zR=GEb-3o`@?}3XE=w-3!vneHk!5(Bb2;J-bltm~{Di2#uP&Fp@+YV0}%LEr!_{h5! z&-kzd7tI$Z`Fo3OZqe~K>~R(c)hih&7h#+269*;LM!RE5BA+A5!>}{eC4IbaVZ%S& zc*p%wD{p)N%m@<0QsrXFh79#Ssz93}z1InyJWV)^Z_)?8nbuf`7gt8-&(^h=yt}U^ ze1O>)3(n`o_@eH~RIguCPXE10FQ^{b!fW{y4LXUV#}d)Cyu9Zgv##!Uomml%yI(S} zE}rv8@&Q`3ecZo|u~>tqvn)*6N!huZN~WAoa0n8f{iKt&v*2pp!|JPOA-?eUURUXd zZ=qk?RECfmtL0h0h@cINopC9!YYOdi690MrDSjc?RlW~oD2Qc^W{p2gilSkaT!BJ@ zoxN3Bz+ps_qDQ$9;LDJ)K*9oDGR&Ale}F>8{7X6JL3zA;M%c7Ba}hy2Ir33$+oubW zA1?b_ipB2}D3=8) zQWZZBpnCK#)!_@#P-qm?CSV(JNP${U3OPFQxpF2o{l)p9>yR10E0X$+D`&A}XsFut zH>=E1Hb#5{)e4%H8Bm0J&S_0vPaJM3E6V|uV z1T0cH_UQ6R(H>^b-1s3GFvph{_~5~PXf zNYi|4eR>gSp;`@c#Hk;kKR1c4NItxss4i zkW#w;$b4;E5}H|Q;UGmL&z3(`XiN#r6L0S9SXe(B&?)o2Jo2Xf?8bFP{FcuK#q6J{ zq$lSW!!kgU1gZ^#NW6U#>rr?mm-}oi7{056jHB?xrp3Q35q?#%vGif=zvdqbBc22P zdL-lv?n4W(9FdWEG!lg7^w(=TNL79>S)zwQRN`owFK#AzgA7b0KLcr(PxC@T=S14| zZYOu9y++Mi9#h&TQ#XrG8wE~G;dWZ`|M1Y?3)yx0w3Q#|BCAa&wW_}6p^}s91Ck#? zxB##{)nT%2b7k16h}Jlp_r28MdQGS(IU`7S!iQFdv&D?CXE;5YAQlOnqLKN@chP=I-a0ad=jBlf%1JEr(d%5uab|)xw3Ygemv@WyH4+0 zF4WfRKi>bYLe*b4$*g`VWJnY7>;|auMx>H829zA(Z{C1B5wI?G3zDf{0_1CU(yzkQ zNk0IexxZJ?FzNWMjr6I8p45M#_$D2EV$hx$IVibQnA^u}vp%z(;&pSg++w%}qOK8j zC91Y%H5pXHy^cMCBVr4mfx$Zy6do1(3VQ$`I{n~5Yp(&3IFC?J`AI3OxO?7F81 zzm;ZDkUl#H-R$+NS!99>|9K5MNW3)(Mfy+wM*-?%^tdtEzplW32JqXkv7wv)84yKD zPm}TefA_`n8{HOaTIS$hNg1lZL=6BjuA(N|*(77v_3Zv>v>d=2i5})YZK)z)l4uY&>7dUxK|cTd3s+L_+FHaL)N*-vHLig*zXN zftNWesROVwOsd;LM$y1G=Bd3JS^ywvRWJ@(Td?e4Awr``T04;XmXDLbcfe4i)9cLg>dIT&ig)V)m+=pOs=}y;4=73dcC$Sk=alP>f z{Z^_Lg4A91%Vw!4-F^Bv{n_=`pK_(Hj?2OLxH~n~_-t;BpJVn1VDd-?>Ofyz4B$cp z?TOt@9eM4`rAoJ~f>Vs`i)cM1IE5lZI-`%kzx>nE+dtstwQW6!NMP<+x1*ZiyDE;2->$y7T*nxY!+#W!ZrV#h@__JO! z;(~+`5D9`cvKiAwVfT*YYy^$#JrJKi2rA{picd9L+>6}T_=1V_mXzl-Tc^(uG_cK`_? zx0HZCjvUlUVFfRa08K^8%Ydt40n|{zE11OVS-X9h!~GvlUD12mAB^5FI+DXO$)dOQ z1}i;UKyCpNbU{HPSeAL$3}5M!Dq@39Rp>%ssMxmvFuk~{aj~XrJU9+j0jrao{u85)z;o z%g8qxfj?u=?*A4HD50_2TOCXQ&HuCW{)!L%B|h%MLPY^GYLd=Xw;?+|N?^eB&9;lM037eqYAvT~T!cUT^ve$HZ&U zDR&7ysdJnMngFYd6K}wqvv?5rJUJ_e@Q;xsb=burm8}w2$2eKLCb*=#w}+pDe;0Kc z@1;#-xo|O#>XBo`>Y=mT_n6GVH^!_7mn4}#T2rzpVOxYkc7*^dVW}mDCGHFQL|1~{V_n9aD0Mh_F9k`! zZs*tex=#Rc$I8(QZ6Ge@#7+TmGm~BlU1RK>fYcAk+M5mx$JNzqt4Jg12~IS#IO9`5 z>}|DQyg4Ds9$aN^LnoR+Rx6f&2GZVQnM^+B#(@;!ev$Vqq`Fgz9OLiCEYf@+fkh6; z*ZLZVxVqFQQADT?+|~c1ZPw5&N&v#E?7jLx5R>l)hk$!4@m#;bO(wJ6_;C~qP{U^c%SU{ z;>i?PTvEUDM^G|17Iro^mymB=9l{jq6}3!^p$1J9rguYpz{H7h*+|BDy@p z#-&SEIf$iG7L@(=!1-*rZc&E;_bn0}b~ejihaiM!>x`o917YgOCQHe89nV|Fmkn|L z%fRrc6kczfDP$J;F=LWvO0Y|`X^qZ9XE;zuD|L)?;25SR^x+bLVWLJM8hpeosdjr3 zOmw{na-%#cxb-cuuc&K$lwAX6A<5qX1VM!KHMXqM2 z4(rFqYoMrhA8#xwpeLL1=@9;|ZgPYjOQ^_5SXxQLIGkolRF=)8*Kj-R5kg< z&5IPhT#j56BuyG5-^lwz@99AzhB8#xLIXg`oMeQ{A}_eMDGt-JsJiC0njK_KSNANM zRZipuSYNNi*{r28*kt#^`EgGRO>s-i$K=}up-b=h4E1u8K9h?W@g~VL_k1$UyD$T# zj?+NWt*>)$(r@K;p>K-e$BLi|AsCTH0U|Y5!xt#OFX)x0dVF`p?BZW=ERjdDWPAGq zK(#lW->4TLulq1uHUIMo0KH}@t5LGQQ9At!r*T&8Yk6>;_u6Vln=}=h2qvLZ=IKJc zD{bSLje6(<#Af5LM1bz?f;=et0AU6I6gT>R11`NA6gL=Rldvec9#-}nK=f+~@A0X& z=u-3~*Z?6Et>FpZIb5H+ffP zocd}a`{G7i%0SI|g^4)rno%z&X<(DUEI3>w*irjkw0@N?yoW)&u%m-Q=F=k!N68EC zR%7%`k=%K@bjqy#lM1Pp%v@I^>k!Gy)V&bZtM0zcn;ml@z4QSDc)mqVQC$`yXZ0k( zSz_|VGmP1$P7S36A-N>$QAgm0;VX-^w^6W^uJ#WFU4_ch&Y`o)dTy19=h>LaA1ziN^l@!`A=mgRB?+KK=jFUoDB=X#e_S!y-BI z8E@TTb*sv)=MQemmwmii8uaRz%!7ZNiaO@x5Bc1~kPh2GDQPnB%Rt9O`=u5p=6?WY CHo3XsiwFP!000030PS3DbJ{u<{@nZuZ@%o^*#u-8^U|B-=mYI%h z3#`W2t}Q}nr~iE=+klM?C3l;UcAc4E*?M$zB%S9-Ncf!m^)xZKN94HJGCv&h8F0uU zQ?)c~jz1jUcg~NChkuEwZ1${k#@$`j z+Z~R|$Y!fIhuon~i2XjBoy}$$$WR&8nlPZ7y>l!ZIb?oCg&orZ8I5R%v>IP}L>>B6 z!zwvFcz5!E=BKhs@FTi{bL7-b4LzM?*;AT{O@hYAIRz)#MRHMu){Rw2sK~A_d07ZL z4rGaUCoXYllTWFJ8EazVv13i`%q>mT95@;osBn@EAR#ng$Y%tqy-*NKrCbPcl3hsl zHJB1h);>>N5>uC#V)~Y$`oQWpHCnf=V|7_5=!gKu5bh#xsnfS_$2`dMN8E9~Bz!;O zz6nLKaKs6qkpKQY(g;?tVn8=8qpd?5t!iH@Q$Jf3Sg0<9Tj9gnWNx4pa-qAQNkv)@ ztK3kwpMU1S*-EdKW;eHCb!a~?yJV4CCH%g0(ZI4MJD5oYnvc-Eb?_Lk3}n%%;D`g^ z-V2pzm?RYdc#co z43d~S0^keBKsx4m4)E`J>AhIu{tf6~OgkpTa@NaVAOo9R4>BF11(Z%oy5!T6Jz}sn6S(IOakenHW}r@ zQlA=g1yQRDiw+f73Xa%wB3s`@37eOWxSUwZNFHIuLWbUp0=rk3VFEhniEP&?Lat~-!FoHBWv<3%=#UXV zgr8Jk|*^ zM^L4{0UR874RZy78>%DvQDN=B!edqd0WO>6u!#wV2L9@KB#ut1boM7G6SL;f=tk1d zXOCDC_rx6`I&R2Ipi^e^ru`IO7= zZ8j29Mx~7Uxr|Z}UDbbL^HM;pBhEn-VMdqrKbb^4pW73Kaw&yU3cWCeUWhmaAtO+> z6r^DOkp%kP2ozqAJJRpOomXSR1u`BH!798#Q?w%!Ud`sesGH4R^voV`9S33)IUL+( z*Q-vwGoXstc0{5Pt^1{rjMd&?f*r1`Fe)Lm(e*|)EyTRI*F6iqlumK!6!(knS$O2x z+m?j@QtG7C`Hi+Ld^+uK$3jT=&r?GE;yV@|hxWE%A*8$LDT98=4GU3{GKGEaR|sOd zU-AFEU!lH<{R+PL2KFm9aw{%=V+GMF5*ZP92VsX$D($!y-K6wxE+Nq=ga$;6xrI%R zVZDtTokI+`(Lro;5o-n-zfE)li8{vT*{w6}M%n3X`#O6k>tnrk_qUh^z@B-QOL>;^ z?DbZ)BT6<`4=-!~@OD;Sfop;!?tNC1QnsaRdp)+TX!ghYR_1FE4ixrGw|q*slx};Y z8+={5(dO)Zek-JeO9}TDP8s;uAl#3S8(x@h#guL--S$YgzwzBIydS&$WIz-)@9IA9 zkfX%M&p1Rr_iASx;&1h1uU6;$Y)vCb;Nqq%ldsxE!|3;TfIgOuLH)v#bIo$I-mV4^ zz1+CH2j_so?7rCrgI=q{(#k}Y7m#|h-Z5_O`V-sei%nS{Tvmf;od6P)C>!-=`Eq95 ztCMqj-<{WMm$iiyxnZ}P)y9L>uIlm=t+ZwL#g?rIk{&=*^Fd}3VDq-EH(;*}q)t;P zxr(@y;+NKF>6=HTR+`I=c2(|OpHbwA6w0Zwy*}4~7 zl2W^8s#lxfM!uibmtEJa)TqimV?3ZL4;~e>UGtkTxD6ow<0dj14-dNBRj$7dK7VZX zW%{QE5SGqnrMgTjQRK3~u9{r7<=XY4qJFb{>pgCksjAKLK(8or!!1u{6=fl%4r}`O z!-tjWg)HUmADFQW5cbmGEuEnW%QH0PO`M^LeDWFES~P*0gSEOjvXa9Bu-vs5#)CJ; z`Q;1aKJprVwj2hpuyNePA@|BsACR{G-g;!B2%)>!#Y+6`_J67}^{Sh==lwBaacpB~x-;kFn+o|seuM$>*0u1Y%E!j74 zk}zgo{ak`o-O<|++fIlx4NL1{G)v0eL>*d>H6@$LBb#^$Mkx=A<01h%>4R3gAR!zECjO(IVB&LkX!Zut{^c6D4M2}S<0RK-Y z6@eg9C%aamSN<6al9^@Wo$yiqP!%6G5hBi?YdAI$c@EyCglOxC7K+;P($KFh4Ud9g zFh7ozA&BpVd~&1=Ul4jmsEljt(VIIiph>?9?A)jui=f@R;MT#kipZdB@gsh)lbwY# z_f)3i`rz(>WyIY~w3~L>BE*_pLnjn&CH&Kih3JuCK@xQgK{^MbZhqyp{2DGS2YaOO8axvR^`c)b)A;acd$*;ti{S7J}NXY0Q7j?v^#%HG~m` z?i9bsbK4zE;*Q7??adxGHEXun|5m1sLp#mu^!N%G#G`qg zJK=@fYG+8|b4MJXVy|-i~QEqK+Z-09U=w(D7%NX@bYK% z##2I*f+C+fP}!*qg6?mjam0azUtKP;Fk!SOmf^qR9TAR3w&9BzgLEPNw(KT4gzzsQ gH&f(Yg7l4S zJ7?zHs;Nu;sX)`+t7WZT&-eD*e36zEf``R`1p@bfgfNjg;i|8z~E4y{(*xfB)tVPj2p=-+bTDhk9bu=?Iw_$bU!2g|t75M%snHnGWcNSX{4t!+^X)Gg78(G>KS(xKK)%#i7!p@chAOGn>&tHGF)7D7; zpDUT${M{{}gVayIq5eokL;YMgP?Y^Cm6gxJ%+gxV#s;XLlY#wr&i|11&vpLFFKuXH zYXOXcwULg9xvicxP}uhKQ|mZs|1SMM&+$LjC1q`-2lV=P>5qSx{&(8n`Pr$TI{vSL z_-mAZrvk&w3Cm9XJY}4)rK_~_U|`%}BK$n^j^KOAP_FU{7hO`khXP1%-~^=6q?LG- ze^#M`)ZvOls8&3zlv@08z2RRgm6qZclNY`a;OS71;TMP^#+Tr6KJ-oC#@&DLAhDp0 zZ=j7A`8;#zx)IsnuvokAa+vI}F?XsT0epS^$cyLYi|5Ib8X1Y&P~X$jqlVs!d~BSA zyi^xh0j}pD>g!oW)w|-}qRRUPOb@S&7fh~sILBHYEhrGyO3-r`fAD40JQ*^?M4d3z zw{m4@m~gPaKR=1U6GW|83ir_dq;dBkrRB-fVyeJCr>LQ%<)Ov4_ey^IQ!MHgP>iVk zTlRmwBKhu8#e8`f&4Q&5eX-Qkgf<83wKkjhxBl?&d1Z1CVEDPbhB$Bg?F4Yk1cKk- zReQ8}f)}a&K#*Zkt3s7zUXE7MfMG&E7fFi)IX06Lw|pPaU^G^JLj-@j`1oMpR&ly9 zka&Bse>=>FsP2Bft~iUqX!=%^;JDcBbbv`IPnHNxZzr7l4ApS5cwR}!~f zhEs&Exz_8wc-h9|`FPcjb2dXo(&@Z`z48QJakwqF0^+aakTFE$@{kObG@P=t%dFQ$ zF_YXbrhzK#@SI>ZI9I>Y<%q}2(u5cgBDI5dCW>}7UvS-@p*mZSD=0-%YjPG+o$pQ4 zQMi4G1g^exkAKtk-H>warP{9=7)}Kp1xya-GYgN~S)<%!e_2^MxyR!$k0jTVo*ILe zWqR$O&;?t1eKA!=qxwMSn6Vg5r;e7}k9#0f`wEmw7Mk2{iNs>ARF9SeaH`E_tMvyG zY7pJ7*JBmyD)5_|>`EPvS4@g)bL{tLj_xxu4x4TVb6Vt3n9V`hu1Brk2!ym1AJ?fp zywKmAI}Z%~$`o=qT)=|mx;tw3L)WM=n`P8(r8l1_oSrJvxrgVp-+S9@I#anVLXr$} zGr?NUufX?svkNx31cmLGX0_URzB}2x9-}JVB;l1yXgLSXyx2)#*UapCW!db0Uz%nK zS+|uD-1PWxyZRC@hdd)s{^#;+y#w_D_~pC<&IES=0+zIBDyQ3m$K$=rL9M|^mdK=r z%l>dWAEgZmC8gPNo1bG>z~IgKE{RkM&_qOs-3j>WvB6}vV`6i&qsE7uUE!Yx3pG}& zM|t<1?=32c%PKNj-mHG$9L8a`T#7eHdB^3>?k!+mx1F=Mx5tfz9ob?y?hQrFLEY{k`>8nS?Xpi-RRg6M}7c^R)vH`(}rs>wa;#_;`0Z6fiTECnwYPT_&@TLK!Y@WQgmr!49TN8FwPv?e=0k zPY#NcBx&YDIoh4aax&|Bl-yD4yy$eL5$Z}t;rZ6&vR4Q!@#+)YHl|N z{nJ)}K01iOwdhmT$z~TZgnjGMq?aFvFIO-G<3Y>|>Pi{YF-V63?<(FBO!NExNGKLt zVN^?yDeF|NkDsqPB8a8Rbb7=-8hCP&V3rzd#N%lnMER0Z2kpF2mV(vmZ1eWnUfe~_ zDSm4f+-bj0ZD@}hg4;zO?cG9lCkr%) zf@o@E>MLJE$KCa*6tFn0t?9bbc1rRu3#=KB6(}J-mRY^DA)$e5$P9i%$s5$1(Ne(N z=>}c4Hf8s8vmz~ne73_p?lL6;fY!0YfCk642c-((_)})EanENzd%b313q<bBRBTUpw@W)qQ$L=9;mVTJeV@xrylXy)j?kPeMpG=8(ex8>k77)2U?!rNCf{)C0 zI^8IZEG6>~-yeo+E59tz9HefUW*L$fXzlv^?0b&glB>@<Rn1cEW=b$_kaAu^j%?4@sB^1j;IZPcUW3;5H*Y*w^mRF$t{@>NCy#NuE^8(^(uU`MsRKc8H!Pgp|eKzChqyPVhPeXwPR4Z)Et%U~)! z4iMVW%3k&|c2#SMI4U^y1mW}IcHh0q<_PTbcd1e;(SR27=4@iz1Kz1joJzhC=s53> zj>sVD(!Ww%aay^0n3BHBeVdi96()Z%T=eBI~g|VcFw)a>~bA`|0 zQMST7BgCi!>mtUs^EcCV@+l?%dN@EnMu#?Ky@$56YN5>5^_J7sw$sN^QH$c@VQJVJ zJ_5dw^jp|Re80hKX$#Rn~9s6#Il^_r++~lFa%49h9g(f{t!UB$@fb{ z+*x4Dj{g8%k*&rCdzU0W#BYv2D0}C#GFFnRPxm*XG z{F)&%25eGb3<3WA8y+Cz3#KJ}W~#yjkg(_P5AF=iuzb)3vi9xY3BZjf?3p7ATCOnu zGvHo^B;0ajGeUv;1b-$W5EsBrWZ$8Dk-w_(1Z5XI(iu1W@=qHlpx%v+cipt2VbSW^snqPRdYwlS{PN=aLZo!UMUkyquR#7<_Y8JCs zq`}gFlLWYXV?y8vJfxx^+wen?GHC^8*=oYESD{M&7nbRWZ+M4|qygnzHyhiC3?&K5FyTXrMj5p;cby1!WKJZ)m?~GCoPh3HlitcN>xP+`ft~x zKHEp1EFsj;%c`in!}Rae&+p)CMh<|N6`V$@6zRV6Tp!L|or^PR9^OV{SxIB5=4PIk zFrF7KsD{eBOCK#7?Iz*&NThNUqTkw8 zf?Muh+i>52$$@WB^=*(N`{GMV-O;qTD^I}zgYu_Jg{o=I?#E8C4$+Yt+S=;GC`uBc z5#2fI`E9n4r}?uSTgd4=p&e;RpMpc3nj33zCbpMBjpv_&V*7rFJ1^ z?+~+4a5*c1+xy2+N**$cU_t(l8sywMf`ArQ6n{yH;qd zvG~D~ln(_ZE1rr#eC~;Td1%O3^3f=uzqDAt4O(KlsrA*4Ho`s=8)JO^Orn#fQ3eqt zTVf;IspQvp3-g7_-*XqG&esQ;-&mftxxFmW+1B629DopEGsM0Mt|p-yZ8-KV39{g# z)d`5}e?{j}x4kg4w-~AUbJVSWhp1`vocwI(W}SF7?wxDFbUCr+z&R<+VWmU={bZug zt#~hj&0#b#2?)!Y2Mg@jnRiEekGgO+?_zvlZ77Jo& z@VA(CXPpIO!D!o1o!*CG!=9^T9#0YH9Xiz>YR-+{1R*t z>xw_Xq%97vQ9)OBYn<735hgvEO4e6cs23%@K_0V`)S>TW_%$I+jb*?(>E{=E2{(%t zv>~@iFKANqGffivJ6cq=Ql4Da!1~VZ_J|Z0*FCMAyCx}=Ed6~8_Vf?24@ycNC|TpF zg!`V@6p191Gv9@vN(2WOIuwwQO5J?KJ)|@(<%%>g`de)lI%JzN6h9t(6yd;ab}eyH zwcvm{m29L4xG1R5oyN4YZEdt&Ad)3E6xvPBcSjeFe1j3>Zf$<6{(f8BbXqZga%3$i znn*8T%xp;>hPvZrt?P^FDut$G{J?myp{Thu$AX_1lcMvHXQhhj_i!F4cgF7(w@Al& z%wFC}DcF=Laq)aJSIaZLx?0OiW1^-+MQInq?>O=xpY@9HD@VrUlsw z&AqU(DfT14)v4%5brPmXFT85Nr|^vMTu}whPFlUn%F--r4Gnn=bB)O$({SV#&TXRF z-H&;T(w$tCEZ_rd*Ohk=VAtzV*sn8&QxLJdf7ftEEv(d2mz|2tJCkS12f ztq&I8lv;O*zmPe|6nLJ%g@NuM2}>8n94M-$ip;&_;)0i=Nk%IrJf$YwL7w!pSzFan zXTE)XNw&-ZeyMr@?|2PiG%ndwsDXcUu=FOB`7Lt{Zpe^)?8_w9qvOIW6*=6}%(6v9 za#$JKBPS3nuL07Q4h+QQ5;(6GJSMiI-b**-e%cq+g$iMk1FRz#qGqqyRlA`NB?S)* z>TPBUM&UO4%4;#UP_6P@%Ck(V8(uSZLvtD17qd2M$x#jQ?NGc3n`vY8ON0l-+-)_a zqt(LaVxF8kqS1oShFhuoic7xEbXBJ}i+^mOY(qju4*XGw_(&wS z39MWN^^*~9_2WY!m-s=cV3U400>}s%uIgy%LLZZf*h@x+7JI)Uu4@Io!YF8Lx_dK! zdgC0S6$$O4jP#cLNVcV2KC^l(KbH(?irW#22cMn$+-AJc_D-OIP>mQT^jlNduKuvu zy`C=DIFByVMVmAHG5HK)X&>VnV^x7OXGW~_gTQ2ur{oCfq~h+F`h#oZL2G&|;ykIu z&_tNOB0md6KQcRl(ts+#cs_TQ%@&IGLak^)+GvL}8Y5|CNi*X94zXwLBbCvZBImMEMizY$)3U3e@omm- zxe#<-lHYf`lI_PjDpq@RSFY~sSi{tE+Og=+F0!Sl4x#da0#Na8U;S2gJkE9(q9`u# zQkFho0J1|svC4$1w-*jOsj{}6l~>SvzxTNJ9@hG4GP@?6FLp>GWoST!MkD2@bmxc^ok$1DRYo3u3U%-xomm(eemJ=k{x0&9+1{WVHZNDHA`hrBN! z(cZ3Ue0Sgbz?}Q>W+lOJH#0VrZr-lDhRT#Z*l7p$ogQ|o-EnWb{$z|6M@z#T^&Ee@C2I95ULFm*-KcK{Z-5z$ve1-|S@f*@E- z66lF)7{;q)Z{{p%kPO2VcDe2hn|G{)pyV+}XmHn(nfHA?t4FugY^@3>1_|_5tNDC( zZP@XqCY-ZxS}H>ytYM1*UDjL84*9ggIn@@@t+)|Yt-5yLt?rsEmC~03t_X{=R7cwFV?dY&j59OhN+KB3(u#}gG}|+} zY9j-(EQM>o_@7V(X%PV4j?1fZhv(>|1D{(iPx&e?{<&98PrQA2tatd_s~`b@x66(` z(ml(&uBZSMFRu12Jj=Ui!N7oNFYD+(U;T<0VDhG}*v4ljCw&7jc}H7U|MS(4C4ib; z!>C<^{xngV0wC}pzp(g!rXtaEb7;NO;Rnz&QVcIZfY1^DbwY~{!w&;~=gsP|#33l+)IZo9Z^i1A7WIDqp>vm63;_i4IDLz81zshOvaCLKaGPduDy;Z$OKYcx0aM z10c_^yy#CEgVg71^AZAm;Fp$v?t(Bsa2XVoIa{=eE2U%8&C7YJqFwdXm)&UFFTmLUc>24kX%p= zUU5%QjQ>nB?dcA`)QD%lXL&WVSL~FtHauK}%|%L?yC;M`l9HU!g>xI000%#+fr; zzl)Bh%j@PrImnMZx=aJ3cAX!w_Nsx5gL7l|2g>RL88~ zN>h?xkMK_4lM3d(c2+5V5*Q4t`A7%6MlSbq}C|p!6-^c2< zG9_f24*S*orvO6Tb4Y&$?vq_f>qtgX{=ft81Cz!F#fXX6o~rpQQx=v8;#L10$;Edb zQ5>_`k8rX@Ifd%Pk($x%dDfSfMvngab9IWuVbQ(gy8E(5R$gasDKwe|5Bu46<)+I8 z3#RJ%SnVVG%qJkqvYG3eF(qQf%)`YjM$oRfhPAT3?ojvioJ@Lj8&NXMlSy`O8__za zVR8~4WhS|No3l?PjQ!<$lvCLxpVLeoaM6>D*!@DO`wDrdtZy*pWk}X#rfg$1`?PAl z{3zB?r$Nug_=rKA{W~|0Q!+D4{T#JLu^IJ!Q$GV(e|>(<+J<0p7gebHyjGBD&`^X4A(W~7>?Z-_UMMKKK9h({ zHY08{C2r1taX4#+kh|0Bg*KY&Bf^o^bkJ*QNiWq*M-ZWiYcSdHaM+CPEfged&6Jrc3}dJ# zhuH>BFL#K(LbD=FS1&kh&!hMM{N?O=_?p( z7CV8KK5N5fROL<4?x5HxjvDOZF zKIR!yUF?vMj)agaz;ouli_DUJ&2l(2GVH=EKQ7U7y~CDet*}^im|GX_(6<@?(o;fp zTY(vi(n}L*kmL#?_v?A~5e`gv%P}6(S@Ovhy;oH@_g7R>B}q2vVsA{bNgoxt7gCk; z!$QUr(L7AMMHp<_H9aaTTrGmH%9Ujq5A*GUii+nDubusA%M5IXA}3zo?^YrEiz#|Z zd%}YVz?>3Ospqjz)~l+&m!d{R-7-Usd87HUE1a0$>Gc{Keg3j&$eq2sVOP}s@*VhI zq(($NcH|Ab-rZ;f>TpA(R39~1xTd?b^Q#ew7N=3YYbocR9R^%wkhpo)*z~?$ce5Se z(Kucm*HoE_J6<|wYbR^Eds7Da^<5vhkfG0w4L>a`=jq!iII($NdD5Lq$0+SdPRBpvcc7>(l~cst&x9ezLgTwipG+H2O~9)3ZB_+$wm3NB8Pj4v1RqX zGy{nUGF2!I*Y>3sXAExddu&|L#FGA3CH>)m4hlJ~zE9^{xRozHQyKmhorK!aappDx zAxINNVu2MMa%>%mSnoxf>Sv#?N16JY`J-@*>nH44gR)2CuSzAB^&!DQHy~tR%hrk7 z>iRj$ADt-I>tC=>+T0#0q!HD62H-9=$)1Ff(Mzr{BndkHl%P$P?J+oC(emU z;D$3fkT!eM*X^F~bM$%7?;K-AsS-yk)~wsiSn1rApE~_6xHu9#IZLLvXdp)usb>9BqfC_#0LVpOurMw^H7UOXy(B1>c zQj{XrUED_Yp$1iq-DV}>I9!prw}!e7a$#DQQRDxFA9bmAgyR*i{JI3Cb)7Kzqi)ho z7}XaaHk>9;iNeH7!#u81UmX(X(x37)tr53Q5^3?3^0c?X!Y{Kns30z8X*PjX z)OZMmKn45~!~;hmCH5j*6Y5s@DE8SNB@honL`1}I(CLYR5A9D${r z_^C3>LJ)CCI5aKI!=Sxo3N(YorUSVZ%D_A3=h%x|dWFGie20B~TD8RLZ38Fh>2Wyp z0ZHNU^zh{D_BuhvWexUSkzXc4xQ_(^FT9RSx;0+dJ6?E5->x&Q^{W?>WK&Uyp~u&7 z_uF{bH4zRQer`VTsLZUA$q^EOo)UGKU|7He?9IJNiQpDTE!|*m6#d8%XzE&t}RlJk^TB7|yTJhkE(jH@_ zi6Z$=3U1C5UvDq#tUgnwXe&Ry-?)3vImK)-h`I;i|#gmV+Z5E2m`2c#0PfEBV$ zJOy=C$T}_rqG0wY64`;FVC2-`KpY{Oh6Zds#w}hb^+arArhOU61%5Uf^%Rd2t zKhP!LcgFTn&s}nT>XJbx=5yo&)RQCI_KJZ`^iSl2#0enqGMx^>bGLo*fNp>GPr&+T z7E*zlkK#Wuf_VOl8#o^87IhCLCFbfR^*xG9 zbcFU<@lq@nN!gfQ7biLcfdHi%r7Xs~uC-K;*rcSpo7ll$tLmz{U4k5z(fzNkw@p|o zr6wq2qnm5;ePe6vita9_kIJPJLBhBXPlMI`Q7pHAg+SRXWOWi~B@l== z`t+DYI(h{TM?&1PZHC$^QpZ1GyJmZ0m_l6XWbijRD^WF)HB+XB#OiPM&S>hb%3D5f_fVy;<&o$bI+(QpN{+~z>1cpxUk<%B-kv|iQ0oDL2|{!byv!mq{u z5RsUj=JrA9WXUS9War8ER`@P>h`e#lROx-G9%y;-XGcSZ?)$};gXE5s`KPF#kkUL>;}()py>Q?Itu(?-7=T6N9?5TSi32mbE=yZc_0E z!9V_Oxj?G8AErQ-(oLz{`4Q}A_(2;B8Srglv09&lHeNwwUR z***|?f$%(uR1Kw`mfXgf1gS;(>1q9uKK+w|Pe)Dj8rZ;-H`HqrRgR&h zKAODV$g7vf3&ebi6UKMrKK{Kn zrv?@x{)a{Q_Te_%V8poCX)<5&P4!Rb-2BIv5QorhJ;<(d;w*>Sc5r|Wu7Q(NjCJL> zB;}0=gN_qyeW=)~JS+qqiblX(xsAighe})FBD%GS!bdd%*VD97A1~x;Z;#LPWVhO#h z*DLAg=B*|J`V&4lX6*ZO%2pJ=s#BqaG3|^dFNwu{fZW5phW9RaQli8%ja+x{o}`O{ zaalpx667Tfx<98_WwP5Iu#NJeG`_itTL|Aq0TbnKV7L}e1byaj=UsyGU0y)*s3s~R$uVMY zTE=mDi&%G5G*4aAjM?=P{3};Y?{GB{*UP>p`*?qRygg?tX$CZzA6}pAjCfMP`EAi4 z8kcE&B)%}`F40<5o*iz{>8dmQ!s1u^TzZ5)n13m_iHQ30xH6_VK28{dFq{;$r63yh zO`l6FtmR+&eYBc;aKm|$D~HHxLfPXBFUTAnrE{9kxp|5Q=4dFNA;$|dzv%Uf-m5Vv z3`CpM<@c?%e;m2AGMZ8|-=!wF%|pPiFAJM65l+!)eY9zgipPP&Pm`H1IbG{j%{~x$ z!C*b4<(uv#Bs=a5=+H^WLi6;#wp$daLOn^;Yi*L&>zeAcpG3AcGJkCQE0we+4F65W zlia4JKhl+{x5xNWZWCt|B&?Z0jbq`V7n*JX2NN%xqDuB(pR5|3*y4ojnSY8SgM7aw zoM3%{G8K6ALc=Eaq49e~e|!$x7wC+Dw0G(_C%s*^Kn&rF8}6%tzxDy|rdKk8o60LI zD~ort0Z`8y9oVjiFk6{e@z=U>2E7j@e^O$Sd3kqZ?=ej`L4yr#5>%6QtIvG^H<5=- z$7POcC|6b5Appi!J z|Anr?X;T3Gync@d_s9E!5~vBV&WeNSG~F}EXakffZ|Y0&$BH40>;446i)DI4&(IX~ zBn6itmxDc*x*`Jb$Gt0t<5_(rWqcBl1-l`hORdC$0et(EiTnk`6)UYU8CpNS$kG+^ z5+Rw2k9uf3p&Q zxOMtZ+OgyAr|ZKCf&M;I zVA`yn=zZ=L5gW>TkRnBtO6pW1>nddGazrHMyLTrG!PZx4>LfpHreNU=-W1pC6}jH% zgEJ%*Yii&Ud6QH$hj$u?STEe7il_EGj%t{=kQ_S!!b&t0Nz1Vab z9P(hwkJQ*qp|wIrXazRI?pb@ zGpKF)QF-{?>GJT~S-Ul_8%H&vUI0CZs=e7nTk9|i*t>DfM7K_yb$}_IR~H`*?A910bRCTOeDvw}wg6O`{+iGFn=y9~9?2Fdbz-l*yJr?FO`0FIL$N;& z-49U2WAXn`jt$M)-qJOAT@Y^Ax0lpum=0{zuBUHJl^;ccv=clTHhtXV{9pkwA&myY ztYqYS0%L5!VXhcgo~*iiJXte!6eL-KyH?oVEKzhp0T;~3j#Pv_H)6~{HgC-rO61tJf8}=OeD0;RQ{UWea&b&0Z~R-j>#&MKrm=dgXaugCDVz0V2)`H~$CZ z`?<84K{9+}Gp~&WYi-iWdeE;mD%iEiv{sEtp_(w3%=%qFKw(I;z{1^eaaDHyi#tt? z#q7nwLCr3ozQk+xU-o=Bc`8aT=L=&%jH{&_x?dh~$>|nxNfRB$M_L(G4gy>`_~J`R z4xC2nGCYbTD(_CygcMGy3wonu#0st;M!_Lak*L|N6shDio>b(BKJ%F0GV(P;lIy33 zkD*#ZjBKjrIJkMY>+wxZVj?{=SPc=8C#eo5!-QD@S9cp|;re=L-wQxeTKN+!WFA+d zrMhnffF_fJ-cDeV6Li=MQhP00??bZJFXSgPTA!nM?Pk040gf~*F<>>y?^+*rKbQJ# zaj~q!6@I6P)<9<20}^egqX%`NfSlV|Ta{qs^E0R&)kH-k*+nk)NE=Ldk?$qQv+dog zmXGG4O68Y6@Xf~Dy`{LF(ZdMxE#2+l8JFgte`!3vybIt4AD7U~ZM|z)yaF_V}L^s^8;YmB&w_(GQ*+Yf7i#7ppaAC+04%~E>;^=AZ{f5ZCVj)->_EocsDI&ht&Pm ztx|*UOiu> z<{pkh8o1Z$9=RV3cI3*bo#;K+QG`+|$gq@`knjRIYV+{5GJ9a5cZW|PQFB_w>ksHB zm^e{%p*!}vr0SkBS1a7_&Jq$NctFYs2MWP-HLv~#-r{xZ*P5(H)1d3-Ts5i=+Xx`c zWdo?t{NGe~`H=K_G#Q}6r7INfK&!N+mlhR~mgA36E@{LUrM$@er&l;BsTZM3dXeAc z#@0TNwYEP}v#F7}`CkV)D6WZHnTe#?t>-UE`SP$_**(T@f|sHhlc-dqH8=}|M<)DC zPnHKE^z%Z%NP22ygHpl1Ox;U!zHKM;x)%fK$R>3bhQh?}`0k_G+|7lsye141eI?J`tLKsoAc>+@5g< zIPazYkRCTg2ViH&n}SyP!*$`@cTcF-tkakBj|n0Xkms}sbitp=?pO=}g}YEn7r{R@ zNt*(A9264%;u)uYQUf@3%vQbk?wJtF0Ob0>$ouBmWRBDfx@sqW%A`Vu)H3;vM>JjQ zjR-_n7}(~~vCcrkiZ^iOnzw@XD~`zBt%%FAe-Sbut^zdof0F?F-=qgW*$tkw=U@q3 z7MAI^a(mj!U@-g;F{(r#KX>URH7SZs{v!^Bh8Ec&+LkmAqaCxY5yZtJJ$kia}jfnt(|I$cK$9bgk`q^~ulR$BC2fj&t$LD^hW^Xni)rNInC3Gs~gXzXD2{_xW)teku z8H-?ZkQ~kWK3$&;w16&k3{JHqBr*4kF^Tg2hTh>HLy?KS{;$goRHNwQ2Y?|WWd(l# zPBcz`B!P)y_L7rAZ{SRo{^=c--M7JTa#Zl5d^Pl;a}R<{gTzPkvQ9SYtl|@#hF?&H z({_N~`3mo;fb4RCgWm8|+k4Anc-nin?HjG-gfXDvKz(4;Gn{Ovs^8)s38NcLOTKx; zt&)s#C&;qrP`NE=;E*jZCu9>V-X+3fHKN{-ibd=c&d#?MmCKW#o-Dc4jCiqV7*o!J zzH+mEa&c;X7ML_H$a)zyLsF*`f0tjVM!0Qsvd1A??9F}^l<&qS@TNb!Jz#D7(ydni z$y~lAxwwEeQkJM@Uy)OyPTcMOwbEa5GI11Ijl z&?yn?qlBYi0drp28TL*+v%BwDA%U1XMfl`&!y_lbd&S3L5N{g(oBZdBM~08sMcu@P?=iMxjSk~(OPHW8cUhSe>x;Y=P8XM?2rXj>5hS&_O=7g^LRQ5MDM=e;;-xLl7{LHy!Os$ zQL>t}WI!NR2$s0{d@8PaI_s(cB|`tc{4br?6H|nNV>aPSrmZvpvznV=X6gq1bH?Qr z{T%3t>Z<)UU6tHQ5Tt*}zt+K2dx-h%-5FDcsDNu64)AJFBK}K5!8vQEkS7IJ06%0y zmuUZ@%33uyF7+Fc=$(^3aj(l4$iu>1r9v0`5xwK%sUV|s5)rmEsJ;!6U`D`h)P%tR zSmV9gAwwgdbY97C<6_xeS|Xy>iMt@iFiT0)Jgy-c z4p~cS5ih#nlYwk?lAC8lMD{025voibrBzVgCq67w4G?Wu-Y69skHB|UHdV14;5GG= z3V1=4vDuqCRU!e73svt~5?(*tiX7M!!}|2Fu|jem^9eR0##Jp1!sJ%nSoqm`jFU$2LO<736Rc^L*MSgC5l0O!#kRl`gC~mVkAEqyHFz| zB@XazoEk4?o&)you95V6XGo3XiXHBy86HFALQ-a$qwn%*xSyc^GEfY{xJH8=TspR9 zTe*VZ`}%?XkdZASN0Z@)RbmryhWG5oN=f(xu{MRk=sozhNL zsBVSXV6Y@h8r1v6d{3fMb$T!`Dzm4*1wdLkL~K}!-h=@*;Eyvn1dF-^=)HP$-+;}h z80g^wG_lN%YC#fk*Dt?*`?U}AzV}&t`;ih^LSn>j-6&VpGv&WKrNAN1U@TVT+$7oZ zk~fYH#V(UXX<)J5q^=iG!~aLMHwu|-&LmK?>q6M;(5ub?2?8HPvyev6Hrvr30#rFW9@ zN$HJ%2!qCE9kkk;RD!(OPxQlVo*@=Q5+m9(H=x+p|0C~5w zmUspO>tFINa6V8R5&yj=8bo@-{1b#y%Jol!ZF zR)gvV%s+e*L?23MNF}df;wD)_z*`x9y(IY;bh1Bf`SUY2kv^R~yd1I0Avk)6SoNA~ ztd5jJWQiU6Xr@dip;}Z6)Uj0uU^ornJs|pb%x(0c;|#a_SO5>HceXJ$RLNOprj?x# z2@j%pbP72A(jskZs^TaL@lm2B;nWp^>Phr{-l->52VRzn$=yWqLl|8_;vQ&LhLwwu7^NR56gxgxQ>C&EiLD1ee% zT%)40_e528d*>h?YhIB7A5J#xDur&v@v*@3xaEXIJ^e|8xKo#6})s{nxq54A;FW z_UE}1ehObiZVG{Y{J(U0+}Y|D3dp0QgcMf^+xTeI021L}#v^|1S;t=cMKRf7$yS4d#_8 zLFyVuJn+ml>D{i@Mhe9tydv5j=m$EQ$(38(R+NC7CgDXl~ARV1!-z=WF&Xbk%<*P0E zK$Wo2h6_V4H4T#LF9E8-MXcnRHY?MxKTEW9w%snv3RU#$@vA@V#WoIcxyemX|Il10 zR}9DYL-F;jsDi07O`hj6Cu`6=a+h(AEV{ac8%ph?Y!p-_N_DPVI=}WEDeh{#Yb~kI z4T!+dUU+{y19xke(!wo{G+iNau}{^}M0KOVSQ3Mt(>qtP#wFF;rztWWIH0$$ukYV~ zCGG!^%N_ffB4w7xI7u4!mwiMMg++gLc5UhmvBu43mBuN7QI=y<(Qy-CtG4G;Kr)Cy z^>y`E9!aLcaR1!d62OIV6vPX|+cG`l`>A^HKu84k5M72ADSPbeRP&iIjssGYB(J{84Yf9^u`Ewj<7%A!ReiTRd=ww?nsMO1UQp4qV89Z6?i|usq zM!vf6*N2OP@X-0S^293cD-^-MkBM4uOC$g`c<+}4dh#BvMaLzN;iI37>*SKcM@Mzf zxLI6<%W2o=g4e2V32VP@xNL(Px#fv_Xd>)Kvf+fo6^9=ajOoNCQ^I42913MSn>m&L zkk89uO9>g0ZO<5@*{8jE*pQo9>KCtVimj}lnT8!gdo>W)WbK6R1@*$`;Yc~pp*)#= z?^5P6FWL8{#hv=Yu&^`21O(3RhxSgf+LLnKYQSM&j+6G~ZTZP--EFn$CXE>lGKaa6 z#_uXZF?Xa_TVd`DTvFpnNol1 z$LCNc&_?1KL;eWo;_UIZ&{UP0geh8*qw-sE-<$TJOD4DXg$27Dh3j&qZOmLY+MD;l z!P==N&9_Xgtkp!9#wMI@qq+u(BGidJB|9B%^#jZD9L(X`H?}l+P-lQgv%)DvgiY7q zjsaEBu=fI9UmD%Uuz9+yXKs9<|5g;0^4-+*-h8}+cK^|Veg`kdy?Z{j#Y2^6Ci(%L zBbFGG*rwqd;TfLO7R^eI`1p+w5?2M++i&mKe(LJSU5`p!PUt_T+~KDj=qOSjCl_m` zD@{@}z3b{;)H!Hgz??IXwU2Wtp=V*aW&w5tY9Q8AfEZ`F1r)18 zvZf<6a8Q*0r5Ju!|INV3I)-zu(}PB%FEURT-t5FJRpYI9JoBfp!^XC(1E$QEaVm1J z^B9aZCC}6aI(=N+RCwU}d_+R~cDZ09kc4$mlIPn5+0F<67HE*IUYPQ}vu(?x^m_BrbL*MOnY8PE&wG+2wb zaj%CZuJ4L6^gYZU7^W2S`^ZkThwo9cZoN@h=q`QT4F;{&( zp;s6sWPV&^PO)4lE}_9Z%5HDqet~|-6ztK>RTUaTP4kwxxq+=${9VE5d3{)aAEL4K zmKdVRwPx5>J*@67>wU1p`-X9?`Q;G3?jrqBV~H*v`Ewh-Qhbqe14z3+udr%=n>}+I0AG{j-qVVUTz?r% z(;%w+%B(~-S2(e6{BkrqRX(pcb$c0)IXq(<>Y}?zUADIFCK9*kfzSx*Olz9m=$9Y| ztK(nW8;E{D*nW0-<`-sO5c|^RD9NGBy;$J&DmE&}=;5UCz^w5+$$h~u7EtEzD}R`V zveqqWupXk2p;l&|OO6k9{Wj-g;gIvi#+bc$a24+8jVR~aVT9%=sLz3?ZFo^%4LgSz zhzj-l)G)chtqI`ha1KPQLAr`u%4$`F7EG5kr(3ekO9*bffeEr*6;)_AICvL%HhSUY z5#ztmo}O*_GV<BlKuo?nSm<~ zzXV>PDl=eQb*{{0f#HE#xWym%gF;Ko_iawORLmx@EWu;<_}^bF8`{)bC&(idKiG1_ zF(@4I&%(Q9>8ro9tym~7CGb!Df0{eXsHon*-4lZnB141JkP-sYh#*5qhcrkjp>&9J zH#l@iH;B?9DJk7u($XCxAbs}e?>GPTtY@8b-kh_}S!-U-p4s=k_uTt?U*GFV0*l$m0nAE_94lw)nQ8TekVT#LVOrA!UZJ zSE1Q)FX|05xesC?vCiCmgDUF(UaP$yM} zE`&^{NU^UT*M94D@^X$!rkS~`Et5;3m>O-TEHEL129b(^LCim1h$g zLhSR%5K{bOt96?Z?cl!6a)`gLF_z!ZYnMJBwY#<=Y+jQ@nEg=U1pv#YIbMQe5(NxG zGFu5i`^%npGdAqtZEQT7ZAKqie59?GM#Ri;zUvK>dAxcq<#AYL)4B0gE1$GK!PLQ% z<~4fZvzsnQ<&SZDfm)VG_`$Y41b#>$j2B38XU+=ZbNL=DhDo|GT)J06Glnm}n()xA z8|UnNNL<8@F|yyzxwVUg`H+vU1joqn@u4by#L!Kw5DQn;;gzr}`k`im5l=q)cX8MVs@3k*8VM=uav zHzLaYmcmIc@iZ_UZtGL@s}O_g5FUEfrHn+LfhnCnEnFClJT(cyg7ymti95Kwm_?o^ z3Z+Qk7tw*$jR4Kg3JI@u9|;FnTu`S3Vcnxevhg6t*kBvX(E!N?leWRj=i0F?{6`JP zKOlE?j(r({HBN>;Zso8+PSm?F^1S6%nA!O)(0_YBv;(x?RI#OnK%LkPM2C+jKC@RH zpQ-_@f3g_ef)3DpJJ)qm)N)K}p;$b#zsB_Xr}OqKtg*~M;_AaGx*$Q&+$SeuBY9fp z+w7fRt-Dfn)i3k+0p4t4DDsZz`y8MRF9QEVi!U2@CiypB!*0oX zs>_Me2=1r*X=_LO7pN=I_Vstt2syh~eNBD3cUryU3vB^XkmxP(CfX;u1=SE_ zZ8%|wv)8_@MVvs-eA%f0DZAJ-pV2J-YtJ>q6FA-i`Qz5ZRxJHC1J=`>W z-pB#D_hi)lvih!jF?Vn29B(o&FXyQ>9dK!;qx?EwMev!kJmq&IG1XQ3m>!N&9p5Ef z#6}vKo15PgBHJ`-)kiYxG3`$=t@D6){kbc%Vb&$snxV=vt1g_peZZ~t6Wid zszK(f*!_xCba^0$FiPrkS9OcX4pEZz7Uq~lHv?Vm(G(}y%gqqlXlm4Wesh!M=d?T8r2@&j` z^S0|Fv(`E%N&Xqll@Fy3h1+`wjd1W&XnK#|v*X&5VlLpVdA9I9{Zz9j2I@EI_g(r~ z^-0em%VUXE9g0FF=4_;HH8#B+2-q0>4TJ{YYXMLAX%`asI5BtcJ^LQG*Xxyo)Xo=W z1Z{9u+DLY0Pz2$w|1^lEFZB;z|1|FA`o{bK(}17+$@4aTTFtPxOBx;xoxw`YODfxu zo#xEb z&^Xs)BUnB_%miQLNQI@;(YK_u^qp>kDR(}AQuuk#LkXLt65pc}b*;3%m z#|JZm9h9rxKUu#z0m3&4pgE}^5SZ#+;c#j+J+;B|zb?TBVAg&m0!zrhls;;!+`Tq+ zIjG(4yCOnN?h}9K0Y3T)m%O{2;aakgf`8bVN`&9gdd5`_ZA78P@qunA5X3<`kY+NA zR3K#j?{fEm;PJl&_QZ6Ycc1EgEYj+*FM{E0Rpfvz+l&zIK2tdl|`clmE}WkqwZ2L#X>{z zm#(whUsPmv<72*SKID!6)1;k?DqoMGZ8?O+EO)EQK$uY%IQ@PucT7XTt6r_RwbNwc z)iFrDiVh(v0s0maj|ss=XqLvQ({f0?Dg-gb_`5~_L2>vI%u2Yu5gSskq7w&HF3GY= zRpe3A5J2VP?+h&lXp_HMX9y;6VenfA<&Z}i`2#A~G}*)dazb(=>OUGa_^)gZ#{V00 zgXNh1kK>SKcq`Z@?xB0%c3xAv@m9q#sfaDPwMLPaBhgnA-vHI5HQ6r`e{3P12XMtGMGJXH$!Q43 z*Ayr4C-jWazq2d-+BB{<;JPfRzFage2YC9FgcO0DZ2jOL*%`dRStJXD>*ns0ri&)* z=%xL!CV~co{%2tUyo7rAG-1(l8f$T%R9S`y-2R4*u~2;de$uzLw{h~uTa#jIPR$}? z(<<=OD`T^{7p=HAb1;()CZIE-7*KZ&5H;m7Uz-$4_Lxljdb~5DaoJ2a-ngDpYq%^C zvlxouSykBQ0UcZ+{xWmNH>J;bmUB?4=x3m62HxV?L50IT$eJ_`y-2EED>8j@o zF5(fnfORn@Xn>pkTKbD8dd>1#M)%13s{6q=T~h(q?x4;&tP^W?R=Z>;XazU06*=E~ z2(Z`AJyotLnz9M*rfu)cy^F_1tDv;4%6f%Rl4*msZ(o)jC#>IDJG19>YkG-wvN)>O@{x;$b{6VZ!_q`nprm*3-J&OKJj`d-$O5@nN!p5Ytc&3-!WtB7Ecn;bV zm*)pQ1iD>Zs~av{S)xjp-sdt@=AO}6Zyrhtww=T=zdL~LJ#uRDCRrf8yVMBiUQ5{l zJzXHaTa%rCD|mp%bHEm^L8Um*JMJIIGhpnzjSBRtw`L|4DDxVh0dhndsJetl!e*j+ z-l2R3v4>Z{kyJS|_!1tlyO`t6e(u=hx?R!`>vP-q!yHSjy~P%FB`ra1WiZH|Vi;3owHgDKqljh?cG z7nnQLU3k_1ie;TxOGrAt6o2yH_V;#L(En;*MDxCPB{a4;M(V+jX|d0x9TT%#kNM8e zJMU8BL6!eUz-mVf?)CpwlddK@oaG4+_<+cl&ei?o$|OIdaB8UTh1az#ka)L<_osQw ztRZJZSszbTrRN!}Ub4-0uZfMqH`nBgru^(3LBP8BhvSK3VLMTdKB8Z3fK6Xcd;cu^tuwc&qm^f(G2JuseK{9e&>-8X5=0O9OWWK}I$Z zCmQgUHJ&Y>N?P}*g~()5y_L{f;Md4rmY9INQ?<=MaF2gczlVP&5d9^8_v4Gd_Qnvs zzinQ}AF`8t{u>?d6hE&DOJM&?*5eXf?v6449@-`yz0&t*p9TmvPam6dkud>73SD#G zOH(B@(^!m20+9EJS3}uLh{O$h-+F=USu5M|kaDg#qoUQYqyQKOF&JPIR}ri531TFZ zvqJkA{>=FMI=Sfw?D9YdE7%rc{RJi$&HEVH1DaEl9I>ZV6ZwFs zn(yZIAr=Y^<-%ujUc-OXk|X>Q#P-fCUiv2#8)y69K= zN@$2MD#J&qX8&!|7~%BmF=YYZN&y!N5FK$O56f8Cow(fP?Yk=^9bP~Z;WQULCN*`*_| z*QlWrmw}eM#PvYXR3f!eIDRV?nSB_uEfC@bTDHQBuI_1VY=WBW*6;B?lwJwJ=hgoV zb-0;Y-`ju_^YHYi?=h`td&x0NW9~><@d*)-*@cI7jY?Pqv7x?gd?C?1ES{oWzFPZ$ zIV9h!qDH5R;qibMr4js%BA@5&F1OU_PvJpSo(ZldetpSOdJeA{VcmB!`8=XriymY# zZ${z(3TLgBiKN(Loz6Gl-({y(>D~gT(J#cM#M(j4>h(oy4jEhYvBkE-{uw2X4?6qp z@4F=My^o1Lzq=IJn5g6>UkMr3&`kmc6nU>gB7A_bGe{lMy@V=$w z!NP>OB?TaX9JtYd=f!vas0PFK&1>|_=9srZA=y-I0%RAAr0rcantequ$4B}tB2TD79EaZT476fwY( z>I0;FMaGi_eNlH283$q_=)or2(mL`plmsEDXeBtzAvK+Ij53*z})B_ryi)voxkY&{)gwKJOR1SUmc*Caw>8d8!z!9ul(&_fs32mM zC^}3H9gwaCxQcZ`6-R_F55jClY$+d!VWK#*%?0_D`ogKq4-j|7W2#Pn$e z`PU-WEHEa_H!*F>9j?d?_pCJSyqDB-wU`v^df;VScW~D9CVR^g}7w?E6ZJ8WOa^YVAUCF!1#xDI*xa z@$keoWmu)wjTq;lph*+Vl4QmMV*j49O=rOA2FJbn_kL?kZ|OA&5yTNUg3?aY;ofq? zZgYbZRz&_&&J1DfXzAD3&=%dX7rupd%i~H zzs4mboh`A|hjHL7)l*oBOJ}(`Hq_12!u1v0+L(q0T9+I7Lp5BEwVnl}XGo$h!AU+(0D}4SYCxy5wzFHLa_Fh4`HW$oic6oznH|KI}~n z+-1!x?$Xz{ZyVTmB_dt+UnAogVm}(RHVD*DiHZ$hT`PgvDn4JKD7U3xI6nHtJa;Y< z57VuD_v{_VN(YOXW3b#urMA!^`*pU4n-~)50kJ)yD9B|0a(7=xo9Cg7Vf=Tb4>xHTpA6gZ=Uq zvK}P4dz^&xTB=fsPmC4Cnq+4)$%o>E$0-*!NIZ_}lCHwOX@AtM8ZOfR$-1_YR7sx9 z`G`06Zi@0V)1r@rKwlNYti6DPpgD>_rx|fQ&o=#vBG?s zha!}p>|yLZH=X~y9KHkRuMWsc)A$iv5FNH&oL9^yJ7;?RbWx!e~oH;|aWrR&JK0A0!+nnKUv;oh&L7#eu?2CU4?qj!>Hhzkd(? z4JTS%Gn)<#MTjxI%08#r!bN7By%JwWcLJ(WYiwA8!Ya2OJvCEI&Yno)fBgjKgu<-j z0iR>(hVG}*h*uqt8XjW_)O^?NsofsQvT;m;%=bksq?VKY`Y?;*aT#gmEN~~HWet`{ zvil+~=gd!dyeN-5DVg#OC&v?%xEI_y#FLo3hBH2~?4m=tjT25-?PCNQj2dkA_G_j6 zN(QEH!z|*%bmdQ(9p*h<2%?flmw9qVYQGeUQ(fAb4&(+?U)vqzaQO1kv*>k?d&&+` zO;<4WBc7vBa?~F%ZX#4}UED!2f><8FM^6a8sFvXHdaSvN_6N1&>uZj3+F zSu8@R2j5&a*olq9yD7w3B3ee9+1lT-f`B^qy|XjrLO!*sO0?5--M1kpjx1sYxy3-G z`MC*UtdKJ%o_Oqc)v^YG=lc@+inh-uZ|8~`7(PjV+P2^`H|^+LJzi}k8|%^!e}X7{ z4w<0tvebiXQWzxIrUGRWCaoi&OwDfNWCjw%(G@i5(n)z|P@R9E_G z7(A12{=i9_ZOSt)j+YkphTYMB%Xi^-8yI&Y%rLjBUUVUZ-7?DYr9etMi>nn10Zm7> zH^lXG`mW9F$Yh<%op;AVUqs#Rx3?*OcgIZnzCUoWo2O4>(RS1PJXliXK=@jK@zQk9 zx-pdL-R{dl)?Tcgu8SStebcd%fU5SFCKeS*)3%A!e(6upXo@v(gB2_J@LswczD<03 zKPRIlye89B_a1fOe)4Y-y67m?J}0v={QTy_*>DBPrVM6^w<{#IJA#g?i0G%>gwI>M)PD?B}J`AC3dR1YczC(DT1$B&p!s_NAV;EeK(-S2@~i zH^!lty|+91L_~(+DDiQOphpGE%%+>O+L5^=kW{SuGtIq;f+Syznw|-K<%wEWq;ZOI z@?q5+%{Nx~i&)Udd7&dYW3DGQ#+KO60?P%TZY!WoB_fs!2_<*@(fVP7fB?34@l-YC zd{K;#N(*x@4Li+!I$VM`uqAB@j^--YSTP)9eqyWm=!%vI+Ep;Tpapq&W#+yOPCd;Q zHH5?XGAOUG9Hgsg_Ns4NzqFaKaGx}e~8nQJC-p5i$ zK}EbTY=DrNxFtXzsb)t(?a$(lLsug@3*d9DuU-91Lxs zTmcXv(iI$;6Qt$~_#sOGKV*e1ay(=Yf$?;bMs5G6e2xK5O=i?|5qb7ph!KWyLMD)7 z20WAXfM>F=3HixJslZWWhR^~eat4+K=8jG_!4vt(?T8_AHTkG#AO{@-vN-nr72V#~ zloJj0$)A$WsUHsoWd{Kx7=MI8Tqgy3 Date: Wed, 10 May 2017 18:34:04 -0700 Subject: [PATCH 46/88] change img positioning --- doc/design/cluster_train/pserver_client.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/design/cluster_train/pserver_client.md b/doc/design/cluster_train/pserver_client.md index c1cb93434e..56469fc215 100644 --- a/doc/design/cluster_train/pserver_client.md +++ b/doc/design/cluster_train/pserver_client.md @@ -7,6 +7,7 @@ For an overview of trainer's role, please refer to [distributed training design The parameters on parameter servers need to be initialized. To provide maximum flexibility, we need to allow trainer initialized the parameters. Only one trainer will do the initialization, the other trainers will wait for the completion of initialization and get the parameters from the parameter servers. To select the trainer for initialization, every trainer will try to get a distributed lock, whoever owns the lock will do the initialization. As illustrated below: + The select process is encapsulated in the C API function: @@ -14,6 +15,7 @@ The select process is encapsulated in the C API function: int paddle_begin_init_params(paddle_pserver_client* client, const char* config_proto); ``` The selected trainer's call to `paddle_begin_init_params` will return with 1, and the other trainers' call to `paddle_begin_init_params` will block until initialization is done, and return 0. As illustrated below: + ## C Interface From c583447e900af61ac69fa6b0a41e9ab10cf3e90a Mon Sep 17 00:00:00 2001 From: Helin Wang Date: Wed, 10 May 2017 18:36:46 -0700 Subject: [PATCH 47/88] add subtitle --- doc/design/cluster_train/pserver_client.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/design/cluster_train/pserver_client.md b/doc/design/cluster_train/pserver_client.md index 56469fc215..ee40eb32c5 100644 --- a/doc/design/cluster_train/pserver_client.md +++ b/doc/design/cluster_train/pserver_client.md @@ -6,10 +6,14 @@ For an overview of trainer's role, please refer to [distributed training design The parameters on parameter servers need to be initialized. To provide maximum flexibility, we need to allow trainer initialized the parameters. Only one trainer will do the initialization, the other trainers will wait for the completion of initialization and get the parameters from the parameter servers. +### Trainer Selection + To select the trainer for initialization, every trainer will try to get a distributed lock, whoever owns the lock will do the initialization. As illustrated below: +### Selection Process + The select process is encapsulated in the C API function: ```c int paddle_begin_init_params(paddle_pserver_client* client, const char* config_proto); From 63c74257de0081ed158c3cb48c4eb55b95c3a78a Mon Sep 17 00:00:00 2001 From: gongweibao Date: Thu, 11 May 2017 09:50:43 +0800 Subject: [PATCH 48/88] modify by comments --- doc/design/cluster_train/data_dispatch.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/design/cluster_train/data_dispatch.md b/doc/design/cluster_train/data_dispatch.md index 7690cb279d..64f24bc05e 100644 --- a/doc/design/cluster_train/data_dispatch.md +++ b/doc/design/cluster_train/data_dispatch.md @@ -14,7 +14,7 @@ ### 训练数据的存储 We select CephFS to store our data. -From the perspective of user program running in a Pod, it is only I/O with the local filesystem, as +From the perspective of user program running in a Pod, it is mounted locally, as 1. the home directory should have been mapped to the Pod-local directory `/home`, and 1. some shared directories, e.g., the pre-downloaded `paddle.v2.dataset` data, should have been mapped to the Pod-local directory `/common`. @@ -98,7 +98,7 @@ PaddlePaddle提供专用的[data reader creator](https://github.com/PaddlePaddle ```python # ... -reader = paddle.reader.creator.RecordIO("/home/random_images-*-of-*") +reader = paddle.reader.creator.RecordIO("/home/user_name/random_images-*-of-*") batch_reader = paddle.batch(paddle.dataset.mnist.train(), 128) trainer.train(batch_reader, ...) ``` @@ -110,13 +110,13 @@ trainer.train(batch_reader, ...) 使用下面命令，可以把本地的数据上传到存储集群中。 ```bash -paddle pfs cp filenames /pfs/folder/ +paddle pfs cp filenames /pfs/$DATACENTER/home/$USER/folder/ ``` 比如，把之前示例中转换完毕的random_images数据集上传到云端的`/home/`可以用以下指令： ```bash -paddle pfs cp random_images-*-of-* /pfs/folder/ +paddle pfs cp random_images-*-of-* /pfs/$DATACENTER/home/$USER/folder/ ``` ## TODO From 45d800b34a880b34d0a69b9e0e698d0e7dea5dd6 Mon Sep 17 00:00:00 2001 From: liaogang Date: Thu, 11 May 2017 14:16:41 +0800 Subject: [PATCH 49/88] add tensor doc --- paddle/tensor/README.md | 125 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 125 insertions(+) create mode 100644 paddle/tensor/README.md diff --git a/paddle/tensor/README.md b/paddle/tensor/README.md new file mode 100644 index 0000000000..ea1fddb665 --- /dev/null +++ b/paddle/tensor/README.md @@ -0,0 +1,125 @@ +# Tensor: An Unified Data Type in PaddlePaddle + +## Pain Point + +In this week, we discussed several potential weaknesses of PaddlePaddle caused by rapid iteration and development to promote new business products on the line in recent four years. For instance, current Matrix/Vector implementation in PaddlePaddle are long and tedious to read, which interfered seriously with the contribution of both fresh and professional engineers. More seriously for this issue, it will also become too challenging to maintain over time. + + +## Learn from Majel + +Consequently, we decide to refactor PaddlePaddle step-by-step. First, refactor and replace Matrix/Vector to Tensor, a modern terminology in the deep learning system. Fortunately, we can learn from Majel how to define a Tensor. + +To simplify heterogeneous resource allocation in any dimensions (1-9) and types (double, float, float16), Majel consists of several primitives such as `Dim`, `Place` and `Array`, all of them are standard C++ classes. + +1. `Place`: memory location [i.e. CPU/GPU]. +2. `Allocation`: heterogeneous resource allocator [i.e. 20MB in GPU]. +3. `Dim`: size of each dimension. [i.e. Dim<4>({10, 2, 5, 1})] +4. `Array`: dynamic array consists of `Place`, `Dim`, and a pointer to memory. + +If you dig deeper into Majel source code, you will find Majel heavily use `boost.variant`. The variant class template is a safe, generic, stack-based discriminated union container, **offering a simple solution for manipulating an object from a heterogeneous set of types in a uniform manner**. Whereas standard containers such as std::vector may be thought of as "multi-value, single type," variant is "multi-type, single value." + +As a simple example, consider the following: + +```c++ +#include "boost/variant.hpp" +#include + +class my_visitor : public boost::static_visitor +{ +public: + int operator()(int i) const + { + return i; + } + + int operator()(const std::string & str) const + { + return str.length(); + } +}; + +int main() +{ + boost::variant< int, std::string > u("hello world"); + std::cout << u; // output: hello world + + int result = boost::apply_visitor( my_visitor(), u ); + std::cout << result; // output: 11 (i.e., length of "hello world") +} +``` + +In Majel, `DDimVar` is derived from `Dim`, `DArrayVar` is from `Array`. + +```c++ +template +struct Dim { +... +int head; +Dim tail; +} +``` + +```c++ +template +class Array : public Buffer { + ... +private: + Dim size_; + Dim stride_; + T* ptr_; +}; +``` + +```c++ +typedef boost::variant Place; +typedef boost::variant, Dim<2>, Dim<3>, Dim<4>, Dim<5>, + Dim<6>, Dim<7>, Dim<8>, Dim<9>> DDimVar; +typedef boost::variant< + Array, + Array, + Array, + Array, + + Array, + Array, + Array, + Array, + + Array, + Array, + Array, + Array > DArrayVar; +``` + +Because `variant` may be thought of as "multi-type, single value", we can utilize it to implement unified interfaces for PaddlePaddle. + +## implement Tensor in Paddle + +Before writing code, please make sure you already look through Majel Source Code and grabbed the design philosophy of `DArray` in Majel. + +To assign subtasks to our colleagues, we have to discuss how to divide it to independent subtasks. + +- [ ] 1. First, we need to consider the third-party dependencies in Majel. + +Majel heavily use `boost.variant`, but we don't want to integrate `boost` into PaddlePaddle. It's better to replace boost using the lightweight implementation. https://github.com/mapbox/variant Mapbox variant has the same speedy performance of `boost::variant `but is faster to compile, results in smaller binaries, and has no dependencies. + +> @gangliao + +- [ ] 2. Re-implement `Place` and `Allocation/Memory` + +I found @wangkuiyi submitted a pull request includes `Place`. @gangliao and @qijun could re-implement `Allocation', because we have the GPU development experience before joining Paddle team. + +> @wangkuiyi @gangliao @qijun + +- [ ] 3. Re-implement `Dim`. +`Dim` is an excellent implementation in Majel. + +> ??? + +- [ ] 4. Re-implement `Array/Tensor`. + +> Prerequisites: 1 - 3 + +- [ ] 5. Re-implement fundamental operators for `Array/Tensor`. + +> Prerequisites: 1 - 4 From 9fbeb3e8842a9f3520f72f47f5fec2a8a4811108 Mon Sep 17 00:00:00 2001 From: gongweibao Date: Thu, 11 May 2017 14:20:33 +0800 Subject: [PATCH 50/88] modify by comments --- doc/design/cluster_train/data_dispatch.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/doc/design/cluster_train/data_dispatch.md b/doc/design/cluster_train/data_dispatch.md index 64f24bc05e..beee05787f 100644 --- a/doc/design/cluster_train/data_dispatch.md +++ b/doc/design/cluster_train/data_dispatch.md @@ -118,6 +118,23 @@ paddle pfs cp filenames /pfs/$DATACENTER/home/$USER/folder/ ```bash paddle pfs cp random_images-*-of-* /pfs/$DATACENTER/home/$USER/folder/ ``` + +需要`$DATACENTER`的配置写到配置文件中，例如 + +``` +# config file +[datacenter_1] +username=wuyi +usercert=wuyi.pem +userkey=wuyi-key.pem +endpoint=datacenter1.paddlepaddle.org + +[datacenter_2] +username=wuyi +usercert=wuyi.pem +userkey=wuyi-key.pem +endpoint=datacenter2.paddlepaddle.org +``` ## TODO ### 支持用户自定义的数据预处理job From 85502783aaa85a43c82a851e9944f535f09c394f Mon Sep 17 00:00:00 2001 From: Yancey1989 Date: Thu, 11 May 2017 14:24:33 +0800 Subject: [PATCH 51/88] Bugfix: checkout input_type failed --- python/paddle/trainer/PyDataProvider2.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/python/paddle/trainer/PyDataProvider2.py b/python/paddle/trainer/PyDataProvider2.py index 0e752c117c..a36f0ebfdc 100644 --- a/python/paddle/trainer/PyDataProvider2.py +++ b/python/paddle/trainer/PyDataProvider2.py @@ -270,7 +270,7 @@ class CheckWrapper(object): assert isinstance(each, collections.Sequence) for d in each: assert isinstance(d, float) - assert len(each, input_type.dim) + assert len(each) == input_type.dim elif input_type.type == DataType.Index: assert isinstance(each, int) assert each < input_type.dim @@ -304,7 +304,7 @@ class CheckInputTypeWrapper(object): def __call__(self, obj, filename): for items in self.generator(obj, filename): try: - # dict type is required for input_types when item is dict type + # dict type is required for input_types when item is dict type assert (isinstance(items, dict) and \ not isinstance(self.input_types, dict))==False yield items From ea25aab57592bcc4f3645b417c2ea000147f800d Mon Sep 17 00:00:00 2001 From: liaogang Date: Thu, 11 May 2017 14:26:48 +0800 Subject: [PATCH 52/88] update doc --- paddle/tensor/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/paddle/tensor/README.md b/paddle/tensor/README.md index ea1fddb665..71074e9b6b 100644 --- a/paddle/tensor/README.md +++ b/paddle/tensor/README.md @@ -5,7 +5,7 @@ In this week, we discussed several potential weaknesses of PaddlePaddle caused by rapid iteration and development to promote new business products on the line in recent four years. For instance, current Matrix/Vector implementation in PaddlePaddle are long and tedious to read, which interfered seriously with the contribution of both fresh and professional engineers. More seriously for this issue, it will also become too challenging to maintain over time. -## Learn from Majel +## Learn from Majel Consequently, we decide to refactor PaddlePaddle step-by-step. First, refactor and replace Matrix/Vector to Tensor, a modern terminology in the deep learning system. Fortunately, we can learn from Majel how to define a Tensor. From 1ba60de83610c3e37062bb479c94f9b94f6aa883 Mon Sep 17 00:00:00 2001 From: liaogang Date: Thu, 11 May 2017 14:31:03 +0800 Subject: [PATCH 53/88] update docs --- paddle/tensor/README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/paddle/tensor/README.md b/paddle/tensor/README.md index 71074e9b6b..b37ed72216 100644 --- a/paddle/tensor/README.md +++ b/paddle/tensor/README.md @@ -101,18 +101,19 @@ To assign subtasks to our colleagues, we have to discuss how to divide it to ind - [ ] 1. First, we need to consider the third-party dependencies in Majel. -Majel heavily use `boost.variant`, but we don't want to integrate `boost` into PaddlePaddle. It's better to replace boost using the lightweight implementation. https://github.com/mapbox/variant Mapbox variant has the same speedy performance of `boost::variant `but is faster to compile, results in smaller binaries, and has no dependencies. + Majel heavily use `boost.variant`, but we don't want to integrate `boost` into PaddlePaddle. It's better to replace boost using the lightweight implementation. https://github.com/mapbox/variant Mapbox variant has the same speedy performance of `boost::variant `but is faster to compile, results in smaller binaries, and has no dependencies. > @gangliao - [ ] 2. Re-implement `Place` and `Allocation/Memory` -I found @wangkuiyi submitted a pull request includes `Place`. @gangliao and @qijun could re-implement `Allocation', because we have the GPU development experience before joining Paddle team. + I found @wangkuiyi submitted a pull request includes `Place`. @gangliao and @qijun could re-implement `Allocation`, because we have the GPU development experience before joining Paddle team. > @wangkuiyi @gangliao @qijun - [ ] 3. Re-implement `Dim`. -`Dim` is an excellent implementation in Majel. + + `Dim` is an excellent implementation in Majel. > ??? From 3c5e0f8a5db328205d29916a8fc9a6ced9a2e2ba Mon Sep 17 00:00:00 2001 From: dangqingqing Date: Thu, 11 May 2017 15:18:06 +0800 Subject: [PATCH 54/88] remove g_pass_height_width in config_parse.py --- python/paddle/trainer/config_parser.py | 17 +---------------- 1 file changed, 1 insertion(+), 16 deletions(-) diff --git a/python/paddle/trainer/config_parser.py b/python/paddle/trainer/config_parser.py index 32e31fe2c4..57d30b088b 100644 --- a/python/paddle/trainer/config_parser.py +++ b/python/paddle/trainer/config_parser.py @@ -138,14 +138,7 @@ def init_config_environment( g_root_submodel=None, g_submodel_map={}, g_submodel_stack=[], - g_add_submodel_suffix=False, - - # Whether current layer needs to pass the image height and width. - # Default value is true, but if it encounters recurrent_layer_group, - # it will be false. The reason is that image is converted to be sequence, - # image height will be sequence length, and image width will be feature - # length of each timestep. - g_pass_height_width=True, ): + g_add_submodel_suffix=False, ): for k, v in locals().iteritems(): globals()[k] = copy.deepcopy(v) @@ -1437,12 +1430,6 @@ class LayerBase(object): g_current_submodel.layer_names.append(self.config.name) - if self.config.type != 'data' and g_pass_height_width: - height = self.get_input_layer(0).height - width = self.get_input_layer(0).width - if height and width: - self.set_layer_height_width(height, width) - def get_input_layer(self, input_index): return g_layer_map[self.config.inputs[input_index].input_layer_name] @@ -3164,8 +3151,6 @@ class WarpCTCLayer(LayerBase): @config_layer('recurrent_layer_group') class RecurrentLayerGroup(LayerBase): def __init__(self, name, device=None): - global g_pass_height_width - g_pass_height_width = False super(RecurrentLayerGroup, self).__init__( name, 'recurrent_layer_group', 0, inputs=[], device=device) From 805088c615c32437cbdcd910582eb8e8bf174025 Mon Sep 17 00:00:00 2001 From: dangqingqing Date: Thu, 11 May 2017 16:54:07 +0800 Subject: [PATCH 55/88] pass unit test --- .../tests/configs/protostr/test_bilinear_interp.protostr | 2 -- .../tests/configs/protostr/test_maxout.protostr | 4 ---- 2 files changed, 6 deletions(-) diff --git a/python/paddle/trainer_config_helpers/tests/configs/protostr/test_bilinear_interp.protostr b/python/paddle/trainer_config_helpers/tests/configs/protostr/test_bilinear_interp.protostr index 9fae596f28..fd5224ca55 100644 --- a/python/paddle/trainer_config_helpers/tests/configs/protostr/test_bilinear_interp.protostr +++ b/python/paddle/trainer_config_helpers/tests/configs/protostr/test_bilinear_interp.protostr @@ -90,8 +90,6 @@ layers { input_layer_name: "__pool_0__" input_parameter_name: "___fc_layer_0__.w0" } - height: 32 - width: 32 } parameters { name: "___conv_0__.w0" diff --git a/python/paddle/trainer_config_helpers/tests/configs/protostr/test_maxout.protostr b/python/paddle/trainer_config_helpers/tests/configs/protostr/test_maxout.protostr index c763a95f9d..03f4f3a31d 100644 --- a/python/paddle/trainer_config_helpers/tests/configs/protostr/test_maxout.protostr +++ b/python/paddle/trainer_config_helpers/tests/configs/protostr/test_maxout.protostr @@ -153,8 +153,6 @@ layers { img_size_y: 0 } } - height: 24 - width: 24 } layers { name: "__fc_layer_0__" @@ -165,8 +163,6 @@ layers { input_layer_name: "__block_expand_layer_0__" input_parameter_name: "___fc_layer_0__.w0" } - height: 24 - width: 24 } parameters { name: "___conv_0__.w0" From 02990b824d64ede5d038019f78f1dc34503e1f45 Mon Sep 17 00:00:00 2001 From: Yi Wang Date: Wed, 10 May 2017 11:15:49 -0700 Subject: [PATCH 56/88] Initial trial to port Majel to Paddle --- Dockerfile | 11 +++++++- majel/Makefile | 10 ++++++++ majel/place.cu | 61 +++++++++++++++++++++++++++++++++++++++++++++ majel/place.h | 50 +++++++++++++++++++++++++++++++++++++ majel/place_test.cu | 40 +++++++++++++++++++++++++++++ 5 files changed, 171 insertions(+), 1 deletion(-) create mode 100644 majel/Makefile create mode 100644 majel/place.cu create mode 100644 majel/place.h create mode 100644 majel/place_test.cu diff --git a/Dockerfile b/Dockerfile index 0d2f569114..dc44388732 100644 --- a/Dockerfile +++ b/Dockerfile @@ -29,7 +29,7 @@ RUN apt-get update && \ curl sed grep graphviz libjpeg-dev zlib1g-dev \ python-numpy python-matplotlib gcc g++ \ automake locales clang-format-3.8 swig doxygen cmake \ - liblapack-dev liblapacke-dev \ + liblapack-dev liblapacke-dev libboost-dev \ clang-3.8 llvm-3.8 libclang-3.8-dev && \ apt-get clean -y @@ -61,6 +61,15 @@ RUN git clone https://github.com/woboq/woboq_codebrowser /woboq && \ -DCMAKE_BUILD_TYPE=Release . \ make) +# Install gtest. +# +# NOTE: This is added for quick hack of the development work of +# majel-in-paddle. +RUN git clone https://github.com/google/googletest /gtest && \ + cd /gtest && \ + git checkout -b release-1.8.0 && \ + cmake . && make install + # Configure OpenSSH server. c.f. https://docs.docker.com/engine/examples/running_ssh_service RUN mkdir /var/run/sshd RUN echo 'root:root' | chpasswd diff --git a/majel/Makefile b/majel/Makefile new file mode 100644 index 0000000000..403c1d3a6a --- /dev/null +++ b/majel/Makefile @@ -0,0 +1,10 @@ +CCFLAGS = -std=c++11 -I/work/paddle +CC = nvcc + +all : place_test + +place.o : place.h place.cu + $(CC) $(CCFLAGS) -c place.cu -o $@ + +place_test : place.o place_test.cu + $(CC) $(CCFLAGS) -lgtest -lgtest_main $^ -o $@ diff --git a/majel/place.cu b/majel/place.cu new file mode 100644 index 0000000000..ee84e96048 --- /dev/null +++ b/majel/place.cu @@ -0,0 +1,61 @@ +#include + +namespace majel { + +namespace detail { + +class PlacePrinter + : public boost::static_visitor<> { +private: + std::ostream& os_; +public: + PlacePrinter(std::ostream& os) : os_(os) {} + + void operator()(const CpuPlace&) { + os_ << "CpuPlace"; + } + + void operator()(const GpuPlace& p) { + os_ << "GpuPlace(" << p.device << ")"; + } +}; + +} // namespace majel + +static Place the_default_place; + +void set_place(const Place& place) { + the_default_place = place; +} + +const Place& get_place() { + return the_default_place; +} + +const GpuPlace default_gpu() { + return GpuPlace(0); +} + +const CpuPlace default_cpu() { + return CpuPlace(); +} + +bool is_gpu_place(const Place& p) { + return boost::apply_visitor(IsGpuPlace(), p); +} + +bool is_cpu_place(const Place& p) { + return !boost::apply_visitor(IsGpuPlace(), p); +} + +bool places_are_same_class(const Place& p1, const Place& p2) { + return is_gpu_place(p1) == is_gpu_place(p2); +} + +std::ostream& operator<<(std::ostream& os, const majel::Place& p) { + majel::detail::PlacePrinter printer(os); + boost::apply_visitor(printer, p); + return os; +} + +} // namespace majel diff --git a/majel/place.h b/majel/place.h new file mode 100644 index 0000000000..ad3dc3fe0b --- /dev/null +++ b/majel/place.h @@ -0,0 +1,50 @@ +#pragma once +#include +#include + +namespace majel { + +struct CpuPlace { + CpuPlace() {} // WORKAROUND: for some reason, omitting this constructor + // causes errors with boost 1.59 and OSX + // needed for variant equality comparison + inline bool operator==(const CpuPlace&) const { return true; } + + inline bool operator!=(const CpuPlace&) const { return false; } +}; + +struct GpuPlace { + GpuPlace(int d) : device(d) {} + + // needed for variant equality comparison + inline bool operator==(const GpuPlace& o) const { return device == o.device; } + + inline bool operator!=(const GpuPlace& o) const { return !(*this == o); } + + GpuPlace() : GpuPlace(0) {} + int device; +}; + +class IsGpuPlace : public boost::static_visitor { +public: + bool operator()(const CpuPlace&) const { return false; } + + bool operator()(const GpuPlace& gpu) const { return true; } +}; + +typedef boost::variant Place; + +void set_place(const Place&); + +const Place& get_place(); + +const GpuPlace default_gpu(); +const CpuPlace default_cpu(); + +bool is_gpu_place(const Place&); +bool is_cpu_place(const Place&); +bool places_are_same_class(const Place&, const Place&); + +std::ostream& operator<<(std::ostream&, const majel::Place&); + +} // namespace majel diff --git a/majel/place_test.cu b/majel/place_test.cu new file mode 100644 index 0000000000..cb82946130 --- /dev/null +++ b/majel/place_test.cu @@ -0,0 +1,40 @@ +#include "gtest/gtest.h" +#include "majel/place.h" +#include + +TEST(Place, Equality) { + majel::CpuPlace cpu; + majel::GpuPlace g0(0), g1(1), gg0(0); + + EXPECT_EQ(cpu, cpu); + EXPECT_EQ(g0, g0); + EXPECT_EQ(g1, g1); + EXPECT_EQ(g0, gg0); + + EXPECT_NE(g0, g1); + + EXPECT_TRUE(majel::places_are_same_class(g0, gg0)); + EXPECT_FALSE(majel::places_are_same_class(g0, cpu)); +} + +TEST(Place, Default) { + EXPECT_TRUE(majel::is_gpu_place( majel::get_place())); + EXPECT_TRUE(majel::is_gpu_place( majel::default_gpu())); + EXPECT_TRUE(majel::is_cpu_place( majel::default_cpu())); + + majel::set_place(majel::CpuPlace()); + EXPECT_TRUE(majel::is_cpu_place( majel::get_place())); +} + +TEST(Place, Print) { + { + std::stringstream ss; + ss << majel::GpuPlace(1); + EXPECT_EQ("GpuPlace(1)", ss.str()); + } + { + std::stringstream ss; + ss << majel::CpuPlace(); + EXPECT_EQ("CpuPlace", ss.str()); + } +} From 97955f063e39c3b580a70ed42c2baa77a0efbeeb Mon Sep 17 00:00:00 2001 From: liaogang Date: Fri, 12 May 2017 00:23:54 +0800 Subject: [PATCH 57/88] rename majel --- paddle/{tensor => majel}/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename paddle/{tensor => majel}/README.md (99%) diff --git a/paddle/tensor/README.md b/paddle/majel/README.md similarity index 99% rename from paddle/tensor/README.md rename to paddle/majel/README.md index b37ed72216..5539853056 100644 --- a/paddle/tensor/README.md +++ b/paddle/majel/README.md @@ -9,7 +9,7 @@ In this week, we discussed several potential weaknesses of PaddlePaddle caused b Consequently, we decide to refactor PaddlePaddle step-by-step. First, refactor and replace Matrix/Vector to Tensor, a modern terminology in the deep learning system. Fortunately, we can learn from Majel how to define a Tensor. -To simplify heterogeneous resource allocation in any dimensions (1-9) and types (double, float, float16), Majel consists of several primitives such as `Dim`, `Place` and `Array`, all of them are standard C++ classes. +To simplify heterogeneous resource allocation in any dimensions (1-9) and types (double, float, float16), Majel consists of several primitives such as `Dim`, `Place` and `Array`, all of them are standard C++ class templates. 1. `Place`: memory location [i.e. CPU/GPU]. 2. `Allocation`: heterogeneous resource allocator [i.e. 20MB in GPU]. From 00d4d89a28bece66daea64989bed2990d5838df5 Mon Sep 17 00:00:00 2001 From: Helin Wang Date: Thu, 11 May 2017 17:37:02 -0700 Subject: [PATCH 58/88] add more sections --- doc/design/cluster_train/pserver_client.md | 59 +++++++++++++++------- 1 file changed, 42 insertions(+), 17 deletions(-) diff --git a/doc/design/cluster_train/pserver_client.md b/doc/design/cluster_train/pserver_client.md index ee40eb32c5..e1fa452639 100644 --- a/doc/design/cluster_train/pserver_client.md +++ b/doc/design/cluster_train/pserver_client.md @@ -2,6 +2,39 @@ For an overview of trainer's role, please refer to [distributed training design doc](README.md). In this design doc, we will discuss the parameter server's client library, which will manage communication with parameter servers. The library will be implemented in [Go](https://golang.org/) and made available as a static or dynamic library with a C header file. +## Parameter Partition + +Each parameter will be partitioned into parameter chunks to make the parameters evenly distributed on parameter servers. The partition is done automatically by the client library. The *sparse parameter* require a little different treatment: + +### Sparse Parameter + +The sparse parameter is a parameter that is updated sparsely. The name is somewhat misleading, it does not have a sparse representation, it is conceptually a dense vector. + +Because a sparse parameter is updated sparsely, the trainer will have to partition the sparse parameter. Because the parameter server will merge all sparse parameter shard into the same file when saving the parameter. It needs special naming convention: + +If a sparse parameter is partitioned into n shards, they should be named as: + +```text +name:sparse-0 +name:sparse-1 +... +name:sparse-n-1 +``` + +## Gradient Optimization + +There are two ways to perform model optimization according to gradients: + +- On Client + The client does forward and backward update multiple steps. In each step, the gradients are calculated each step and a new model is generated. After some steps, the client will calculate the difference between the newest model and the old model at step 0. The difference will be updated to parameter servers. Parameter servers will just update parameters according to the difference without any optimization using gradients (such as Adam and L1 regularization). + +- On Parameter Server + The client will send gradients to parameter servers, the parameter server will do the optimization using gradients. + +## L1 and L2 Regularization + +PaddlePaddle allows L1 or L2 regularizations to be specified per parameter, so when the trainer initializes the parameter. When the parameter server is doing the optimization, the trainer needs to pass a parameter configuration to parameter servers to indicate the Regularization. + ## Parameter Initialization The parameters on parameter servers need to be initialized. To provide maximum flexibility, we need to allow trainer initialized the parameters. Only one trainer will do the initialization, the other trainers will wait for the completion of initialization and get the parameters from the parameter servers. @@ -54,24 +87,25 @@ void paddle_pserver_client_release(paddle_pserver_client* client); * initialization is done, and they need to get the initialized * parameters from parameter servers using @paddle_get_params. * - * @param config_proto serialized parameter server configuration in + * @param pserver_config_proto serialized parameter server configuration in * Protocol Buffers format. * @return 1 if the trainer is selected to initialize parameter * servers, otherwise 0. */ -int paddle_begin_init_params(paddle_pserver_client* client, const char* config_proto); +int paddle_begin_init_params(paddle_pserver_client* client, const char* pserver_config_proto); /** * @brief paddle_init_param initializes the parameter on parameter * servers. * * @param param the parameter to initialize. + * @param param_config_proto the configuration for the parameter. * @return 0 if successful, otherwise -1. On failure, the trainer * needs to restart the entire initialization process (starting from * @paddle_begin_init_param). Or simply exit the program and wait for * the cluster management system to restart the trainer. */ -int paddle_init_param(paddle_pserver_client* client, paddle_parameter params); +int paddle_init_param(paddle_pserver_client* client, paddle_parameter params, const char* param_config_proto); /** * @brief paddle_finish_init_params tells parameter servers client has @@ -89,31 +123,22 @@ int paddle_finish_init_params(paddle_pserver_client* client); * updating parameters. * * @param grads the array of gradients to send. - * @param total the total number of gradient inside the gradient array. + * @param len the length of the gradient array. * @param learning_rate the learning rate for the gradients. * @return 0 if successful, otherwise -1. */ -int paddle_send_grads(paddle_pserver_client* client, const paddle_gradient* grads, int total, double learning_rate); - -/** - * @brief paddle_set_params sets parameters to parameter servers. - * - * @param params the array of parameters to set to parameter servers. - * @param total the total number of parameters inside the parameter - * array. - * @return 0 if successful, otherwise -1. - */ -int paddle_set_params(paddle_pserver_client* client, const paddle_parameter* params, int total); +int paddle_send_grads(paddle_pserver_client* client, const paddle_gradient* grads, int len); /** * @brief paddle_get_params gets parameters from parameter servers. * * @param names the array of names of the parameters to get. * @param dst the destination array of parameters to save to. - * @param total the total number of parameters to get. + * @param len the length of the names array and the paddle_parameter + * array. * @return 0 if successful, otherwise -1. */ -int paddle_get_params(paddle_pserver_client* client, const char** names, paddle_parameter* dst, int total); +int paddle_get_params(paddle_pserver_client* client, const char** names, paddle_parameter* dst, int len); /** * @brief paddle_save_model indicates parameters to save the parameter From 3da6c853400ab19557108b865f493f8ec73fb538 Mon Sep 17 00:00:00 2001 From: Helin Wang Date: Thu, 11 May 2017 17:40:51 -0700 Subject: [PATCH 59/88] use enum instead of define --- doc/design/cluster_train/pserver_client.md | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/doc/design/cluster_train/pserver_client.md b/doc/design/cluster_train/pserver_client.md index e1fa452639..0a45a86117 100644 --- a/doc/design/cluster_train/pserver_client.md +++ b/doc/design/cluster_train/pserver_client.md @@ -58,18 +58,20 @@ The selected trainer's call to `paddle_begin_init_params` will return with 1, an ## C Interface ```c -#define PADDLE_ELEMENT_TYPE_INT32 0 -#define PADDLE_ELEMENT_TYPE_UINT32 1 -#define PADDLE_ELEMENT_TYPE_INT64 2 -#define PADDLE_ELEMENT_TYPE_UINT64 3 -#define PADDLE_ELEMENT_TYPE_FLOAT32 4 -#define PADDLE_ELEMENT_TYPE_FLOAT64 5 +typedef enum { + PADDLE_ELEMENT_TYPE_INT32 = 0, + PADDLE_ELEMENT_TYPE_UINT32 = 1, + PADDLE_ELEMENT_TYPE_INT64 = 2, + PADDLE_ELEMENT_TYPE_UINT64 = 3, + PADDLE_ELEMENT_TYPE_FLOAT32 = 4, + PADDLE_ELEMENT_TYPE_FLOAT64 = 5, +} paddle_element_type; typedef struct { - char* name; - int element_type; - void* content; - int content_len; + char* name; + paddle_element_type element_type; + void* content; + int content_len; } paddle_parameter, paddle_gradient; typedef struct paddle_pserver_client paddle_pserver_client; From 747994d00fbc9ab202bd362438b9d3f7c8976217 Mon Sep 17 00:00:00 2001 From: Helin Wang Date: Thu, 11 May 2017 17:54:41 -0700 Subject: [PATCH 60/88] polish wording --- doc/design/cluster_train/pserver_client.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/doc/design/cluster_train/pserver_client.md b/doc/design/cluster_train/pserver_client.md index 0a45a86117..0531630fb8 100644 --- a/doc/design/cluster_train/pserver_client.md +++ b/doc/design/cluster_train/pserver_client.md @@ -8,7 +8,7 @@ Each parameter will be partitioned into parameter chunks to make the parameters ### Sparse Parameter -The sparse parameter is a parameter that is updated sparsely. The name is somewhat misleading, it does not have a sparse representation, it is conceptually a dense vector. +The sparse parameter is a parameter that is updated sparsely. The name is somewhat misleading, it does not have a sparse representation, it has the same representation as a dense vector. Because a sparse parameter is updated sparsely, the trainer will have to partition the sparse parameter. Because the parameter server will merge all sparse parameter shard into the same file when saving the parameter. It needs special naming convention: @@ -21,14 +21,18 @@ name:sparse-1 name:sparse-n-1 ``` -## Gradient Optimization +The library is unaware of the partition, and treat each parameter independently. Only when saving parameters, the parameter servers will merge the sparse parameters according to the naming convention. -There are two ways to perform model optimization according to gradients: +## Model Optimization Using Gradient + +There are two ways to perform model optimization using gradients: - On Client - The client does forward and backward update multiple steps. In each step, the gradients are calculated each step and a new model is generated. After some steps, the client will calculate the difference between the newest model and the old model at step 0. The difference will be updated to parameter servers. Parameter servers will just update parameters according to the difference without any optimization using gradients (such as Adam and L1 regularization). + + The client does forward and backward update multiple steps. In each step, the gradients are calculated each step and a new model is generated. After some steps, the client will calculate the difference between the newest model and the old model at step 0. The difference will be updated to parameter servers. Parameter servers will just update parameters using the difference without any optimization using gradients (such as Adam and L1 regularization). - On Parameter Server + The client will send gradients to parameter servers, the parameter server will do the optimization using gradients. ## L1 and L2 Regularization From 7d7473842f2ad84bf00988b161ad25992b17b8c4 Mon Sep 17 00:00:00 2001 From: Helin Wang Date: Thu, 11 May 2017 18:00:55 -0700 Subject: [PATCH 61/88] polish wording --- doc/design/cluster_train/pserver_client.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/doc/design/cluster_train/pserver_client.md b/doc/design/cluster_train/pserver_client.md index 0531630fb8..500894fac7 100644 --- a/doc/design/cluster_train/pserver_client.md +++ b/doc/design/cluster_train/pserver_client.md @@ -23,25 +23,25 @@ name:sparse-n-1 The library is unaware of the partition, and treat each parameter independently. Only when saving parameters, the parameter servers will merge the sparse parameters according to the naming convention. -## Model Optimization Using Gradient +## Model Optimization Using Gradients There are two ways to perform model optimization using gradients: - On Client - The client does forward and backward update multiple steps. In each step, the gradients are calculated each step and a new model is generated. After some steps, the client will calculate the difference between the newest model and the old model at step 0. The difference will be updated to parameter servers. Parameter servers will just update parameters using the difference without any optimization using gradients (such as Adam and L1 regularization). + The client does multiple steps of forward and backward update. In each step, the gradients are calculated and a new model is generated. After some steps, the client will calculate the difference between the newest model and the old model at step 0. The difference will be updated to parameter servers. Parameter servers will just update parameters using the difference without any optimization using gradients (such as Adam and L1 regularization). - On Parameter Server - The client will send gradients to parameter servers, the parameter server will do the optimization using gradients. + The client will send accumulated gradients to parameter servers, the parameter server will do the optimization using gradients. ## L1 and L2 Regularization -PaddlePaddle allows L1 or L2 regularizations to be specified per parameter, so when the trainer initializes the parameter. When the parameter server is doing the optimization, the trainer needs to pass a parameter configuration to parameter servers to indicate the Regularization. +PaddlePaddle allows L1 or L2 regularizations to be specified per parameter, so when the trainer initializes the parameter it needs include a parameter configuration when L1 or L2 regularization is necessary. ## Parameter Initialization -The parameters on parameter servers need to be initialized. To provide maximum flexibility, we need to allow trainer initialized the parameters. Only one trainer will do the initialization, the other trainers will wait for the completion of initialization and get the parameters from the parameter servers. +The parameters on parameter servers need to be initialized. To provide maximum flexibility, the trainer will initialize the parameters. Only one trainer will do the initialization, the other trainers will wait for the completion of initialization and get the parameters from the parameter servers. ### Trainer Selection @@ -49,9 +49,9 @@ To select the trainer for initialization, every trainer will try to get a distri -### Selection Process +### Trainer Selection Process -The select process is encapsulated in the C API function: +The trainer select process is encapsulated in the C API function: ```c int paddle_begin_init_params(paddle_pserver_client* client, const char* config_proto); ``` From b7e4b2ddc5db6c3fcea711748a62dd32cc7a17c7 Mon Sep 17 00:00:00 2001 From: Yancey1989 Date: Fri, 12 May 2017 13:42:39 +0800 Subject: [PATCH 62/88] submit job design doc --- .../cluster_train/src/submit-job.graffle | Bin 0 -> 3931 bytes doc/design/cluster_train/src/submit-job.png | Bin 0 -> 52772 bytes doc/design/cluster_train/submit-job.md | 127 ++++++++++++++++++ 3 files changed, 127 insertions(+) create mode 100644 doc/design/cluster_train/src/submit-job.graffle create mode 100644 doc/design/cluster_train/src/submit-job.png create mode 100644 doc/design/cluster_train/submit-job.md diff --git a/doc/design/cluster_train/src/submit-job.graffle b/doc/design/cluster_train/src/submit-job.graffle new file mode 100644 index 0000000000000000000000000000000000000000..677cdfb6d9a32168bf71729eb841fa1ca0dd31d6 GIT binary patch literal 3931 zcmV-h52WxPiwFP!000030PS6AbK5o({#^eGzWMUDZ*2+y@6=7lXX4bcUE4{TjAl9z z3CoNrQXwHbPA32TT~HTz=x}VwN8!mt2X_Gw*nRd{0EE{6_|l2AOO|F~694`T>*$%r zVm}GOxc&RHx2L;n?z2B$tgQcM`*7>@!_kh`jlzs;M{hUx_qMcWYmLT6w;QoWWBYVl zJKEnnIn|(yMq}sTnfC0Q^X~ITV_0(Df~}P0)o)YIMs*K_3Bt}AGS`Tu zVdpu)&#|ffir`N!Zd6E;mi0f-c=hkz&wJbZzf?4OY06@La?ThZsf96TZI-@(@`rB+ z;=5tQPOrO_7SbM1%0)x3bkDaFzt@2Z%Dp-)7aNs2UiQL(W&4SL!Gdx<2Zxnfa_M(r z7Pg{Fvp(#cl;!t2w9U4X9@LpuTT^Mwei&a&D!tm6H?$oyeiQi&nvr*YRy4TANCceb zY+#y`ZNq|p8nT`vFb}8RHu?pzn;6HN5w+Il&? z`TOjQ-==%8JH7lP3V?k;J8W9<{g>MEW4abGp1bnQYEHth)2qp^*HrS&#zYq`MQjHV z*F~OVhc&@t&Jc2mi3kEzWXDgyvT`?!;;6vkPRP^?? zWu@Vrkm5}ZdYNC3k-m5tIAc&9dz}#U$!=;*BV5$93je z*T7ukq)YuAo<}+moue0qz23;*jKsIYtQ*nm6Q4#kZ39xSb-DJtmcRK@ZQsE3GC}So z^)Xyj?WkRuj1xsrmCR7J^9Cb(X~`wSQB7}!7Xe_|LObUt{WfpiNK8=KfA zj*Dy}JQEwAN7ze*G)#*Q6YHx4p+(hKXNNUlP}9b$)EL>CDL=za8fRht zpWkg`Vv(kw_z4x4EM)dSWW5%Ab_rVf);vpM&RbFQ3^mW<5MKRr3jCNO?G=kI8HY^w zfc4mK-$B+Fwhh9+HB%N4%WAe$#$xfN74;ar=2HJ_vM*goP+2gNhwqzRng&h&#j+9n zk1@$#Y*)MxV&*Ska`0jpxrpKABBq&(K-*DYTfEpf7qJ}>5#@#{WOBWfM$ON@_<{?X z%;GcZ^CU$pEBW1J2GN@S8PYC#E#~t`^Tp&0(>^7w8i21Eq4xmi_1Ie-@}hoCjG7WW z1olo39Z4~Zo@_2ca)P;xhPTFAb6hMi76}?KDIq~!U%di~f1CJ$kv$)LPY!L~gwS@B zz}MxjFHj1}r-MA^YloQ}I_0%j?UQ2cDc z{48K+oVI{b4XB#^UW$1=9scCuThk{6Xj6OGHE)Y&F+o9eTo<J~!8#K+9RPj>0%7ZbrK?9xTs8)w1AyE(>n4?)mP? ztSPEndYy?)o!E{^AapsFX^D+swo}OqgcA}^hp|A8qV|WV^#>IQ?k*4n;z^fOYxTk?Xs!%Cb$i+QqGJOw#B{dd z9hV=5_z>{^gw(=koMa4iSpRn5-xZt*bg(@wj++oj!#5Gg{m2txEaX=x`cy}Xwc zTG6E20`d&h@z5RdqzY`$v;B7trlm;JBP2~vkomX^WVgEr8#QzE)zU|gZ&%QGt#*T4=Jpyx+Xa&Ryh#xT!n~x5N^WSM$ z)dU_ZBwpeiv2!P9Jc>imZbpgK9I-;;e*+R*j}D35{Am=0!^@o`cJC91)f}NUV_fUya1mH(R^nIUr)2Wj|WgzIgza=CaQ~ zsb9BJ$5g7FqNSapAV52evsBCGdRX!<`~8BqMqT(YOb;B#G?wlkwmREmPM5sxDbM2fL2O_U`6&@D zf>d<*Xy~ZXUeah=o=&XNIamg%nWV#>HQKwpY;$l`a8$u@o|(EJh?xERgQpLbe|{6Tsz4^m#k%QOs%JW^`LNaCbR zeW*!l$$$sSG%%NbLq8>{P${45C*NHzNee?gJlIyprQ52W{5R)(jO4&`#ACplm)?uV(Lyb0rN$Q7&J;#S-xmp@@*KS=3}&*8C>sG<|)&EARI z#WP2T?FtfJ)3BIPo(P({&*C;euc?R(&aEV#i--YfcBSIHj2kwVUnSjeZk@8}AEqIT zc}a7s{paR#7&GBBQykzpx!Q=rc3r8src@BG_;0`chJ4=Y;3vNs)q&x7 z*$V@fZTLLA9Nl95LA+yS;SV(S*$s8yW^5f_9;A27d?L~kPt`lbEb`33cos{S;AM3ExED!VvC9H*%Pl(V^7}B5+;VK=rLDKJ z%0H*HRn@4G9~i(fYs-!#EzStSZF~Lsc~AW8w`JQkUGXtVI`3%IlPph)9EBh<&ERXv zL**+8t|fCES!j-)NvE5#EISlwGZ@$m3q_TXT4aDKzW#>dCU!phFV&dvnR zU~=)YcQy88vUj2WPb2@dBVq1h>TKocYUN;00cqFR#KFx~kctX&qksSXr=PA?7XNoA zdzZh%0s~}${KCS<%*yg_+u%|G$WeZA2Rlb+a~BuTz7UtdpELjG*#F(}@qaJZbk5D55l*2>jP$L91%q1r0N8!)%_b)kWw}(HbE_2l0dW~l6?(UXo zI9wh2dVkj~ec1DR*!Fu+vJDWSz(jwFRr+0N__6=HEpWXE1)}{D{J*XLX^}Ko1am2-C1WD>fA3AvrBgftPj_Eqq|?=M0zu7U{Hb?|wJFcay2ns~}b0qGGo6$Ssr ztg!8>@9*;b)%DBRqeee`Zo_0OQd5XXr?#~R?bEerNA zMVN&jjDP=3P!hU7pE_qexw%?=ll|^;Gug1vFud*E*$B^MlWlIeuQd5P*Z1rk-x5E> z`uj?A`^~8mioDTGds?EO`Z#3lcdZnY7nS&}mj|yW6^D_k=@fRn1W<+&^OltwD ziLlh+2ec7V(NmkCEa-uvbn|M#RTF~gFE z-#*d8MTdJJfSx`WIcn^{lq6s!BDev)lZfddYt#3vkHGL5{Nr~m!{@iU&u$SrhGT_pzwwAH z9Q(=)O1t(GJG3FSuhOJh<|5H%IIf4&W|_sy(meX6n?%r$-=jk@wETUXtp12T+ruiE z$>UI(L>f(a|mmnr@Q*A%Rv3*VyjuT2a7 z2-oBbR?5M%aNmRa*)Z340m|_&nt#R*v2+MLx5r%C(hvL0_veAJLxwo4MQcm!19umU zD;p1`&GO|OyRYcrV%AVeEpymKthRSpE@60>TRvfUa_P}zrxaH{Kt4sr_4v0Y5emhN zuYn56^|)ePa6;+L=3Xvz3ha3imrOq*siaj>-tHDp(S}MmX&Uy`Gb(uWeE91x3L$s> zg((ed7cH{!O!mB#gyxk`Y47mxedE2uTItcF_{JTJwrvcqCKlzZb|~ZAuZ4z4x+AoqrM;2=fpo~S1pMO{{E*OI%N5U4uo}^hw6EPN|^btn9TKuL^ z5L^5*8aIahuiMLf-|4~f@;-c4gjRogX9$XF{8{{a0xEazx-o~in@!g0?tWh-^v7fh znlx=aRp`FD^e~GDjat_yf=|MR&)*#K*daatb(96O>fn4Adm*Q;Y?h@0r&G9E7xH>* zic2e}v-ae4cmm6{Rg!__X_bBy{-@*_`Ehno;QIe{^3=fdA0k3U^!)GP1BmaxY%wsl`^Rhkxj`o)CSu@A82@$0|2Zt; zO$na#mkKRy;{S;&tT4dS#z5uplKju%aRbDtC4;RHNLP$*R+9~Vy(VR8Gi_@ICUuyC zg2R*MLNLhfPgaVHvKFjcg3P7r{%uyB0+VbBS;T|~la2+3z|iMp{F6AuYu#*T`$fYeuiD+@@2^5lBTX5SJ7e35OYzs?=5g*Uo-EzQCLXcolndfcptF%oDs~5Qi_o23$(`Kh`|+BjrTm7~z~+@jm+U`0)G93*o!t9>O;n zP9?>KZ_g*BT`VfG1)6Lsnzz7R_%G*7wKBrQUB*o+1nTiZ_Z0ITEyZ1pd2GstToqgCSdR-SjTT(XY$eUO~dS% zW)@bqoeq*eUiX7fIR^1BtX*(8&ysh!7j5&Xc~*T~_}|Aee>tCY8RaX|f5X=+n1N2& z%fPo^)wAFJ`>+)qjgZ|ezPsUOD}6gYqnA4NPXFo0>aRp7jY3yTE+Yc+(nH5XRqQEP z+<$>l9T{|`KOzqN`#G2ZZV`%Km~Yw7!Sv_$->>~OP>sYI9u3`>lh^bL;@DV_v z-m^iQbh@%v9N#A?NA($k0W=UkxGi|2Z5AhaT0!GH0J^pfP*QLDQQMhf8UphV5UwvT z7ajUNTyA9k#bZH%VC{nAq z!Rz5B-t%12s9pE~Ewf z1LKTgAEdjnti;wXBeV!SMHZ3bJpq6XARw2HyG<6p5YzHd14_c+zXMHsFJkGK4ri-Q zjEKq#L!0S32{4oMH_A}u*20p$`sZRYI32GJ;#X{IM=Y|uETqdfhnj8pR@I$91>?up zt_7kzY>+>$ZEj+$ICR|KnvzGnV163Qo|e8_ZBc1X-OIkGRPSW}kGE%~K&nj+hLt)tE*_iOQYNlg1(%JJL=kG4(-yF67why&*9nr#(p(H<0 z5;H;&b@vKpVq)7&RJ$xHffjDen6>Y0CB*w1!qlnP@PL1})~2Gxq~n6oD{9Dk^>eWP zyMWHN^2jOma9w9FD#tKuw~5L>9m2F^gMr=nW>q~gq4)2xzi|`?hEin*DfJr$TzjPN zb6$~FSkaFn^2*#uqO~dqQ8u%-iMbDd);vEpSZNeZq+aHEYOp~fAMyjboLH^HZFru3 zmQMd;PKhw!j1P2)>vHe0{;{wBxC>ZFpbQXW-~HcwhVBUnxAIPe-C_TDmH+fEKn#Ri z5lX^~e^=Yz_Ce4tXhJoX_@6;^iU1q-$oUZUmLDx3l-yT4xy)|SigYjV0hL@xkEKs~ zFR)#T;IAQAQ(!WteusKq0|SjX%(tDProffd*D~|^O{V$B*A@-)7U|%T_?;$X!|`rP zbmn#`3E!WC_3yrD*C=?qlLv_sy*NsGifTP2fBS4fjp^i=9_={2x`NpDaJ%QR>^8gX zu^KIaM$BysGAm+!*PXg4#XXOj0h)B5(}AKC6J!bN+;AG_&tF;>&}SgdAB5hORo}Ul zw|nIv&Zl-<0xv-fLYYs%FtBKOfI@R4nfbx}8^uGrzy7^(fV2q75=9d%hRyU`7h>D} zx&EX>G5pL@8PU=Oz8LB*;vQ*dStBd!*v2VcPhjqgbLfDe*r*1PFEJA`3 zn1?mnyO{9iA9mb6qhI^Nu5Ts1?C1k-*BzUjSN8e&2d{{~NUY$B#7RH+y0FrkEl3er z<=^dLPIdgYdGsB!dh&3835aw zx4V1YTXYY%=Eg3pqMzV<)hk6Q(see2rv`^nI}{<<5r{9ZJadOYq#$jz<+viHY&7YTKpZED+bJ}mvu`~%3p|vMO?mwc_$mGO(|-ef&M@M8ggv%q{#5PI66IheSp_mT&7R&bMpcggI9L(sZl}l7`cEtA3|4Tm_;qkN2bRCF|N? zKDM9MGGqVsCa{qJhaupPeJ=08Vh!F~k7UWRDokZvFwyF(y#??^F4ZUr`bCu|9SENa za6SyZccE$DBrN_I<-J5FYy&RQQtoU+wtmDEYyb@}zxkG_i4ldc!v03777NV&&m58M`$UyHd8#|s?m%q-D zX9pqc&8HE?mj`vszZ}k52*G#ScM{;n8b~iS$ z{JV50mdgN#>W7PdQ8e`Y5$tz6l}+~UT!mw7H!0HeA0OOlM4^lyNSqbb)pR>y5sO&X z@o;P6t3qvdWfv>C@n5bt5jCI&b1A+Hd&`v?#-Z&bgrc%|8zkd1W(f*2(R32oe{B&a z;8Gf@D;n$TWk=LRkS~SNeIQ6AtcwdiLU6VVSC>5pZk#0<43=P_*PPVIpME5o#J?71 z=>Zhc`EF_DDDOrT@H|q&rGrJfDZ9*5B=<^s`KM8Q$T9T=4 zt#c-k#k&AG4)9^a=hZudEM{vM;sgN5YFc<9%k?Wi!Fem(H*H1S?`Fu*m$l^dad%uL zmtLdn$%phET@!$5gaMwtbwBja2Rq(2|$S+>LTVaL3Jv!^3sI zFjYM-{yHR|aeCu?e+ZCQv46KbbMsG-RigBaK+-`a-_wNMiV^mq7l6mgDM;Z4Fnu@g z&Pzw-t97aDfzffAmS+f|MP~$LO5A4k2?i8tOQcPL@if=^ODg0>D}66Y({AGC)!~jE z+CCO?*RPipz+s{Alm|EYykG)UoI?GlQijHLSfh7G3cNqw>wiCa@MtK)`28?T!u1!? z_=G~O{#>*F&Q0QXcf$S)AtK~8MWPXuS5h;`pu<#d0kw*cB18NFKb<2SnE{=tWha+J z5#(U$IL&xd6A*%vs09S*gVQ@%Vzx@&i9?X1+i*6dWSil=Vz9uUk|uF_-{?E|gNF0? z%U(sR;{Cn*&p2uRli7|3uG<)Kc}FJ3n{jar6)pjg4KkM3VRulM-2YyE%=*Zwk$Xhv zE2rx!VpkW-UY|=qZ|D_eC_SAAf6qnYci6nm?h7b^bF^cSvFFG8UZl0Y&2aiEI#tIV+ zwZppx)9&x;!{55G5+MI;t&YB$cU!d~*JyeUTbvVrL2;$2i`u6il zd3JK|eYO{lfL36^{~-0+0^}jq^XAM_4*ofiP|8P3J8f0Wu+Oyp6OLQFO}^!Wi&To| z{l)B=rI2?6@LLdG+%cy2r27v2WhQ74cJ}`kvr7f3q$8Na)?VlN)Jt z-MV_OxOXvTLW~`2Vy#G}Q82VqUwEQ|cdIFTBS@Hm_Cebu{JGvcRNdgarY0eQa>l(6 zN{Gp^zgHX4PLZEqb%qezf79Vyu!Qo-SeICOn`TuQ*J%=&^m2k}+j}ho_^9)7@h)_Y zCner0ILObPEXVcVU3_nc^E|R|zp?1Og#d|i&dIJ^!G9PINUVE{c;4%Wz}8WSkSzFe zzbdHsDD0z>iX~@kk*9^Qilfrnf%owEn zhS8#)FK%{Atk!N5yQ%#E^@0pJlNOnDPVV98Xo*6f4uSQswD*IV-S-*j^uUWzTX6TnV-#7=-(AsC#;?8`CzbQLMOYrd8i&JTJLmdrP3oJ@SnL&V>IkY!s zjoJIqtjzrO6ld{?=LvRGvip)#zj_d!rPFO9^Fv{2LRY@2$|O%Xd!NB{0M)POHKuco zaFin;H7yMJ`cj*d{{P4U3gu}3Mn8IirhQAt9h-H>~N^7@G*%lZK9GcePeFeMYQss?Ir8N;Bxw;4-23 z==)2eXEiayC0-?c;{Vodo-wD)8*PeX1QG<^BsdkhFq%(K zx6Kz0ei5+UgEA3aK3=D#!lbiYCCN(j7HT5Ny-86oBHCz?yd^=~@VCQVSM{svxx_-w z<6!?8*n~&|0$M9NP1Q_ztT(j|@ML<~nC%F|_GK{=nh41Jm}(dalj8StOjKD!yBY|k zc8uaNW3*&f?Anve-0IS?Rs=&mqkllmV5~?nZK_63X#KOe0)He#IX7s&(=$(x5g7K^ zmxsIoUt;)oKN~)rCO4yc!IBd*>nfOi4^JU#9u4+R?~FdBN96Z=Cu?$aX3G=1EfWT8fnBB`{pCvkA|3556}HeC`oZ%JQdsV>xorvvt<(Fxn{ zQvp|vdPrRHH3MCetnsTq5SKJa1egNaTXxi6MC3e%$hUWi&$8gd7%Si;^G4>JBz^lT zepped9N;(S)qbemM`p^R+QN09iqRdmRZaVuqs>94NeM6MCAoY5+s|HBR9B!{Q3ow1 zP3T&!qO8kHgY6q|_)~dtM&3O)bjpCRSQiy%fbYn_6oUw)y(5RP7TY{IhS!j*F3u z(t@wt{L^Rn`rJNEYC@@OF*WRssNai~zy_m8qpjyr)x$g|x1!gt4Z3fpa^s(D6Ib5M zrD?P@z)^9r!8~?!Od#GlR_3~&cqc!uQSKa>@jwlweGEf54MmH=F`vh!#ixS=$4?BI zf-DMroxQKb4t4R}fLb;V6vz!nHhnfot;${1Sdrn84;H=hz9cE#=TccUxEE*3CId)~ zrRl8SvtQ1YGRE66%ED}1@dWoL_h!YXPppN$+w=a>I&?Mz+qGJA5)H7ys)zi2sQ6y= zehNM`iUCCs(VB!%zEEJD>nnsj>VQros_{pMao?GoGd!+*I%FhVXq+(W37F}U+?0aR zFef#(81bx8T;fl;+6yBpxDnF9gUV z!F^vd$VKg!Gt^2I_{qd7xd+c%F;KtDGJ2L4LqXN@*m%1ZZiPxfp%umy3kStLJ+=_8 zB-D`uKBb)*9TY)uyVd|UQ_m@?B{}6Y#x+o4vu!nEfq9NdBzN4ss{RbL?6V61!ACaLx(f!djTcr)?3?EzZr!X`9Mp&o=%ncT1Sz$q9~#t{Vj8oU{#H&lfndo_lX#1E3E7+ zD83(I$-Vy^J}@q`$u*>$X^n1J=2h32$_!^QjFFbYT=aq=WAaORNp72!?#u_AE-0-2 zDuh9!813c$Z)awe5$>fPC$n{BQaY2lEh2-(NKg8zd}TpYD>qD&G(DpIo4h5@Bla#X zqLu5{nL60thtiDHx*W%)5^2q)f_{eF_O)?tbnGWqi6Mltb$F+a90u8eq=n5UHC02;yDiVUBfidlCL|9qJtV-Qvg!r z2-%P+iW%y`sJ%YQI@}QyALwgyy)alVi9o!{PZ^92+VdK7UHposAbWAAb!d(99< z@jX4I)w9tlTOB;|d-R@sA@+FNw{AIv$r`sH5D;M`%48!*gbcSp6j0;oIJ7nJv@FsZ zKBYdeHFa5x>@3fh`&||Fd^x(RTi(%9Bk8cc+mwXhk;N**T)CAW@4eCeYspqLDo2o~ zrVT9`{h*tHfzv}*8_^k8=EOpDeFjqiSY~cLMQMXmRkuOcQh@xj6IgW3S2P%DFE`z( z+58a^x9OWgo4T#gtkw4D@ZqZ-K9>+8hVjkKlk1N_l^46Uka#hr6msJxA;6G^i-4yq zO+}nZLkbQ@TA&nKG=WH^c%5}TbV39>=Tz1zr?yyL=Ylxr2u|UOzd3x|s*PgzO z^R-ur+MJP1GOGH$iV_EA*tR2w^1+)-CzV?yql}e$#6-y;U2Y|**%i!@!=|L3NfZ2H zN)#oyRy*#kAk_}c4p7qCf>bRvv5jF1ZLB&)*=Rrw>Q71x%7b!0x(>vaT1$rRyk=-$ z0>*fRADdq;C9GoCm&ao!B9F1GDzM2RFIK4K;zjTMfd!}B%9SJj$77g2N%|j>NC^ut z(0(r;Th*;}Q(h`dMZCC9?zrpQSHFFSOpAhysy?DJhKN=5>I88J)d-rBzKld_kjAkK zBw-DpMuKj%sq!5WE5-Y;xm>Yrx1>}DYDS=zqDiv%wmHzWy5ygNkUGVC7%J<}Ra2BU$W*tsm#{2@1@WrNv1@{5vv1Je=`luK)y>;r*@_mFjEJPHnGH{h^&v zM67F&_sG&vrriU^tiP7G`xMkiEE!+sZwSm+c{gA4nBt)aeNahy0fI*OmE)BY;FUCWo7Yg`XA4BMj#;;{{lX6R6L4Ekwk5F>^Y9!HQn;9cKbD=FjG&F8a zz;gh}(h9tMp$U`cVST;e7xs2Fl;i`NU+NiohMA#YgcV^y@~KrI%U%L??=Jh7M>L0a zu}etpF4w14;@;l&i4HH}@O5q3sy_}{xl6>-Z@VFtLYy!M<_>b8yhkhOsVR4+(?~7> zU_?De4_l88D|X%pI=#KYx?Vo=ZJbt>PnsHqP>-9OFUpW1hLr=0N%y47u7q~uQz#j` zk(|dOv=(Cvt1{v~U>(K;&Z0btU^R?eXU{-77uWr`|3;WYy91QG-T@}8j|@`I19kBm zZ21?)Bh5fVz{sLfI1C}vKo#%S0uz0h+#l1qEx5dGqADp$5S5(2x&? z3qRhB-LB^)vpfQY#NbWbUlw%-6;4jnF@1Lc6U!D5Z{e@aGo3on)mfG>x-FY{;7ht2 zD*De`%NrJu&6kXV(1yDuR3Fcagg43bT_+^(av;0`Ncb5?(>D3gLCM|qydc>?%t!d~ z&PBi(B>atHl`NqVP2N980Vkg$V~p~m%D=X^hIP#0;bwbVw6QI93uOCPH*{R|aqeu( z&0A@VjafL5nmM4N-^(6>N(!XRitK&0>`{~lQFQf_`(?=2kRK?vva}s{Bar)iz$kdn zOybmw1=4F;24PSVhVXk~Zjz~UfBk@^4IIC4N+&2H?Y*0sA)t?PfQLam1alSS&6Rh0 zSu%y}@_83LtW5lmI2=z`6?_$be$O6Dv#AK**qsJ6AzjUiaVV+9*39{olEZxr5>3il zeu5$q&%zeni41(e0sq05XjYF@q&ySXn|mY^<1g&ieZv><`W;bsd8P_>a8$U7E- zVQmkuD`-gtfqX>unJErjv$w|cbM5L^)K=vw)?aakQ8!1#(k*ktWB9gGq(({S*vrGo zII6g%?f!7abto_e$7|epWe{FHr_h<_`{?JkKEx)*_uuTC@-}T1F+)Cyhi;iCl*hzL z!T>#8kJBvBaM<4H*p&A@W=@m(s;uU#b`rifyOYrZ@W$TnUZ6!oZf^78N;fP!L*qV1hlZ2QBlO{woswR5^6 z-BimG&e9~(@c+o{=qUem^LxSxSWR!aK4RXoS~+5^h1bO_uPrFY&pbX{b%^I#zIsTI zU4`<}N>z~}3?7>D*Z`$IE3}n$O6h)J+Ikp`bkLm;fy{zUWq~q0STTip=cvuE}<(QVl(c>5=)0>0lqMAO;Rnsvc7B{p!d!Ka%OKbxCsvE_4N+zvcy0J}!q^ zT2KQ&$9Dd=c13c)>#NeFN=ULe|B(4hch=FvWU{DOCZmYYhB=#?OeD{xo%u+>JBKJ1g^838iZv13yobAlr$Nb!@Iqgtx= zM<*A=Os-1C%@mCg3G6i?K;csne|xrAzuMACk>6bPeu=F z6(A+MFP3fBPX#roI$ESl*heZx=@s~=+p(YM;Ce_d_>^jkOY23>8v1?77EwX-LcOew z4-i~VlwRiSs#dHA9xBP9K>%5$$czNy7&WPYb|T5Lv5ra4JujLY`x+^mHI8JStLatq zxBeN`m$}uG)N2(oucp;GR$d$9l-XpKaVI*Arg~X4E;*yHF_*|GzhBl75rtN(cU%RF zc^iaf>?ixcSg9p3Gjrxl&18JooS*Rg02OCj)mb2_k*-TQoT;>g2dXRtbQOVuqPm?v z**?a*CvwJxIuS8}JdsZ>!J19@nRX?$yr_nDdLbiG3remla3o{IGh087Dvx=q`<|r0OdK%VwHbMlMLzV{f^NFqGNED`+}8;MZp?;{tGHp?M3eI zB3Awn&OAi7(sX?>Ty8H^RI`=7l6=EZ)(w7>>8APS`Q&TcOq^MUrlvAHkMD(16ow#h zloPAYv7EPPHx<>LAmq!<>8xntOr#9jq##E+GrbM6Y+CABXF!ah9JW)s_(o8KTD%N| zA#*t;z*};70G`6>a~Ma3X#6hRDGy(;KP_aT_nVZ6!o}Z zR{0boj2BDg>%3!loOE5s=Q0SGJ{iYT5^YPOG&RwY@^yZIwsm^{nf_P=MNq2cwQFCUd_!+5K z;{4h}*=Otbw3q5epu~0XOPTO=`B`_h*Ed9sAC$^=svCaj?FpZ>a4SBKu#*UBF}EFy zQU2V&Jco0UspB2>?D{nfwRJ?G`8xPi78lceOwfvweR6>cZpmFR*a!CnC$fJ(JXxrE zt_slqk*Zg~FcG;M0lAGQ{LwR6u#ornyGP5&UP-}ZJ?HROSjmz;9W2?F3#3!y!IcyLV6cJGgW9}%z-5sdc)Y7XS8Rag~#MseYD)pTu{Kiy zdV|Cnt@xZ>8-%7c-$fvxuufqUyJZ=*C;BKxy6+eFJtaUND${-yvxSGkS=BjPLD~9? z%^9fRZ|Am9<^WFIER~Dm8{=u4`rfY)sre(1%tp-Js6|#KDBK=|2p!tylj5%@8C%ah zmj)zs=S@F%DMR%Qfw)+}T|=Uf?Q_2bviE)%nXGp2gF90Ygy)Pu8^(L$e^-37f= zq2->wagr?Ze>ztZS3&Zw*^>Ls|rleNMy%g+#>t&pZt+`svy@(O0uvmG`_r`YV|#!9_Q5AlrJ$wb<1a83Iy zVo~=AM1rgEMail_1MUAx&Dz!r)X6(7dD#osn6uz2DXmoF|rsQ5~59sP;UwNM<4lRK)GkMnV7U274)WeoN=KTg5`@U`3@9%}u zaEhPZW2~|U*<|6R0t;fTLQD)?qcVV9EN!TWE(+K%Ua%@+y6m=RyA^d}9j&4!{iKZ4 zcgiOS)&)ctxN&g>8lMc zGgG;gW8_!56Ak32?Y}#iP9vqH-i~9@ZQq*?7hQL4{U!{O=$p&^z5+kMZXY=^Q8?HFiJhK+7AYvZl&2UOJRFpP0!fYUEiJi82H^^oVYzdv9jNQ z|K7-D6Eh4f*^(sS1083~cc^5}1#>f|{{4Jfc{JrLFB(k=jXGVQEt5{FXJ$dXlnnU^ zWO;3fTj9iE!+iqD1K&RAAE9QXSj6}+ktco2=Z?PL6n~FN#aP`%M|mZ6T6SI%qNni; zlJrxK-(q;z*7#I9`Naby*2 z6w?WoiJRhdS~(66=*`B^`O|Uw84N%TB6)(69(W(704f zkzM1dq^VifR@;k|&v9i-PhwRz-b&Q*6?rBm&Xk!E)%&Q!XcRE?7;-o$N(=U5nLAJf z!ZBIL*fQJww9M-0lXFgcr9ikpQ)M(s%R604IgmB`pw%E$C)6pf{#&S2D z#A?a&!8&bW5RuWL8frF7pugxwTc|{#6g$rdYMx0dEJ!S=Gv@&(D)6IBC+*wrP}!dd z>Qz=e#Q~CXmr-(O(9>HJ(wuiaLDp1oxbp`G%fVjW3{*|cWR1`)WknU+c5H!_6yx73 zUMs?ho}dQOJ48H1;Gi!zuEsJjf3m|SLoE>PbTv-P83_12ICwY+D?j7S4iDGrt7#$? z=+&CFpLLP-&Z@-W^b>MmP(vF1ejYgzzG8NQxLN8F6={km4}FW!05pFyVN2|+B?tF4(B?}Z zP`fjS!hek@j=;6RBC2X7?dK0_3jdM*27V`r7MLNg{I|OTT@8SiVy8W;u!Mb&E#4&P z_ll2>s1tTKJl|Qg|7W@qjcCCdWk@DTaMY&QbwL z6ra}q2}KFl?-88Is}A}GD!Xv>Ey}ONyRWs>o+_(|UsNExC{9W>M&#^(LXt=;w~6Tj zo6Fw3k4Pp6E)YDHDZ|HJYfoWICtWZ7`YSabZ_F6u7#2g|Lwjz^L}HJVyzH=s%6-n^ zuSB4;g$eMbIwoZ-Ns*lzBN}pVL6A#*DZ}B${= z^j72d%&61ZGl>o0Mkspqph%&@U7sU8Ghp{nZ`Yz{dE)Dg(He{ye2z@3B0_OYhO?3x z!~Zm|N>gT??HUa!Z$PRL;l`0lN&Ty(_#~I}@uoK&l+t37Ai9FtH_wljW&adMY4m=d)NJG&xF(>Kh8*0Bs2WSWx{ zs;^#-T)TWegA>Q+dnfB~_JvdI9M#M?;YKXB%4Y-y7kyVYepTI{XThWOWwy=C*-R%9 zVGmGa(NCJVp?Q{^TJWCBt#utSM&+fKQ>B=?-Q)3%G|}NH^x9L5#{t|xAB@mgyhe1+ z!qIiHaxqhsTz-?-ZwN43n3oc-@Gs%fDnR-G`Z4#G$hNET+v;C0VtM=_p z13V zG_g4S^3W@428uI=vA>t-dGzpUn_Nz%!`_5SvlG$NO^w~6Ke(Bd?#*705Ho0F)PHTLc`%D6<3A?Cn6N}Xr)(1zP z%yq8X&i2ySTd8eCGUVvJz$T6L_g-Az)@JGN2zcVFR}yZLlJ-Ecc3F6MaiM)@xvv!b zM_~^c2qLqR)LF=G+mFCCZciRp4K_Vw$-NThtLfLs2xks8LD|v?q2JHlH~BoAme#e) zd7aeZ@blLDoj*U_BTK#qJ2hMwS9W|_5);Fh2YD0_lr|fO4plSa6Er9$UGWvn6x{dd zj`U3$Cqr1Uw?`9Brhegd2z0q7==Gf+{!HR^s`kGgw@MaHk(C;bl%`%5L~N=@rM@P? zw#m?Wl@a=tw2oS0AUcA-6>r0eL#!bPU7WFD)db75mfN7V(xIExLD$o z<+nbMY08eI;)R=tqd!lG_riOAR6NB(IFz;N30zmvTb2t@?Ed7`1unD%Ur1HMd&6!*J5T^hC6rCC*EZLHuJLx z-5Cj;Y0g>AxV2lRt!F_`5|}LtT_+-6wCSh8H};^vbR11Evw* z1YSL99KRJW*9M5APUJ)V(b5C5@LQ+{%EJ&$+>_TXBf`U^*XRlZK0L8xqpdca_#Mpt znYV!Uv%-ulg8}2N6Kwi(DNhk+fAYwi;O0~E)|%ISr8;q=b=^gg3+ugRczCLzSql+& zf2I&7pg}6O{IN{Ikpw=_xq0T_{fpVV89>ckb}FRbPyTKYNlSXPhhK}nPmTZ zvTPUs+~{Xvzxfwb<&1yc1%Ti(qN$UL^u@w&v9admgytU+3y@f!5xo882(}(r%X}ncj8z2bON`elq%v2?XmYEJ5 zxFXGA&q~fR?eR$u9OmcjBs;)9O$ba8F*3rU1JSFGym$v4En6cd?6S;LJ?r&%SZ}L5 zZbS;z?sE#B+`jPC5BO8aW((*6`(1V~hxVQ>t#2EU%1?f(ndcA-YW*almuTNKIMHuG zvF1=jjAYM|6UC2!3~fjXlbj}Y4mPNzv?SV31e6HuTB*T(!w$oA=B(yEkPMzTpIU=Y zE@=`mpNo!E4JpOsuA>H10A2#H5xkc-ViL%I2Gq-k8#m>GXj3CSgVWhGh!tS3Cu~!< zvW+)R`4bp(X^JLwq)KJIL~MgZ)?y~i<3=@PG<&s;wT8YV%YZl%BQ1TmB*%PgCd-OP z1vB?jhPF!}^=Iut>Sy|DcHqaTBL2q&Kqdj+mOutmK+%N7SoDNvhHF;Olqx1w~m8H#Me$dOOE0Jif7I1B|0`Dj%tyno5M12Fre^YliSaU9{ek% zosyV^{CRq`KiAEBFMD@J6+epcV#tdbrWt>e-($b%KGmm}|NNr$(W%MK1D$_|V#pp4 zkS?GIxC+FW^9}ciJ*`*)HL|r~fq`ouccpOGS@BmGzvCg9E}%eCgc>jQg*|L;lLLtQ zJ+$Q+Ab>NZMj8AFos^!`3m()>4a?xba}f{M>Z-Hd5IO5g^MHpNiy1Se%10FkAh#Eq$n1dyxBxW(kltlGX&Q#oG=_dd<~ai|fq<0SzV1EW zcPDNQK0#(II|lm^A=?r-S0Qv@f=cWz*l8@@Y1j`6WzjmDe8mT|p_h%<4U6_0@s3D= z%MFixQDmqzkkSu24b<}VC2&R&3&_0V3YitlvtUD}3D8Nu>fa%RsGvD)!G_zig#v0G z%pDpyUi5cRM;(t35FNL8?BXD!l81b%lvvZseNLroM4% z_OugQ#n6UTQK2}_MoeUi==RgOS|*#>j9mwnDZCJ+<9)yTJ;U$jR7Kf{h?nbp)QHuh z`Tviow~nf+`@V0b%=iIZ`o@=f-=bqI!8Q1N4p-)l<+%5>n;+87XZvMR1&1-FRDv{k_h{oKK zrL=2ToS7vE3VZrUCwD2BDnIV|hL9Y%9Qmzf`3nnhWokdyh(8C*Uy~JiM8<7fdOd4- z>%(y^K9du0Zn&s1pxe zl*>~We%7_&JrL<-p#(z~6PIQ}LJywUt9)`IRA>P{$7LSS{`gTqbOa4nBb!gclYXC_ zgm(PSDvI>(X1=TCYrK6`9}e71Uxe#G^thT2tYS)^1*=_tu79I+R9rBl!_trsH(-m! z5`}@-id|sXx(4?ZJ<5WA&?~2~sP&(Fi5j^7bATh}Zp;FNWL)uzRYRj)T3AVjtv)~V z*YPLRTN{Gd%Ww6Km3~eFJBCvqmXN8(Fhlmd<_r`gz#+8(unsW5Rn@fTb5sRQU#I~i zHix}X7Wic-qBH>CI4g+KysYTRcN2gM9aroC9IM)%id(c>=3PJRom4(qXf&lxf|9?- zgP}1n-WL1bRy05iY1#GkjpX{9&NGP9pgR%_9O-kPnG#W9e`P}c>;vP40Nwa3|>uc?S4l*0youA_x z2AvJ0oB1q4_FsZuK;P5VTBdLf_5~7lWXKx=>cIv0>1Wr)kUcP%Jq~}W`Y-+<7_ba@ zk*BslH%BfvQ-LE0Z4z~dAe0zRQXP`z2=#UQz`6&;wn(uiDK8O(`h9YZcUcX(~)B1d>0Fzx^okt<|(`rfu} zeLdaM7BqV{FLTRN8LZ4rci4QhJpCY^<0#%#pGMD1YM_VWH>L^&nos zh{GBLBM{EhCIv=OG>Q|K3#7?Xtjuk!wCc*A}14r)tlSHxa2PS&%5JGx@ z0qu*M!z8IrR0$7hVPNnGnIL?i>iS5OKU4Zh2-M0)ZrzAr9!@UXSNI>HJ_gL^cO*4O zy>Hp}p@LR7*(L-W^MI-*TM_K7s!>Z2Ze!>Q@?8&Zm9l98nuePd&k2t9W-D{*Nd|Zh znCi)!fyaq0ju@(uW?zpF?5HQ_;FH8a7;?-;GU&?)gLaY^N!B0dFwB9K{yjxK-=CG> zkFmmSE?(n#ApHbvZx^8Evc(=Q58ZJK;2a1-VFt^DNp~$&FyI0Y5x(H>fK(g5mY+Yy z{k&9%ZRO5^EdJ7FsDp@`Vf+t3AGBHCN_A=(gm7I@9U(_sr3yajyKsc3iu+!r_<_8@ zvHwuHbwtArI98=cebKpo1K++$tsw1-2P46F38*kNIm_bL@_3CNAWEZ|`gJA^CTE+a zPPUWL?o*n!-ItTCCvW(h3&3-u>laul4tLcUN~M1ccfE|He$()^2S|80&Z}Gi?fvWS zBEOcCR0KS|;4Y9ToTL1KVu${qQ{PJfNy;N&Ew?jl@h&6z;jb6VjPWherelH1-EA3? z7p;a!ITU?11$40rdqY2vYTkld{V3p|06ft5z-dQx(n9`y_`rskc0*&Cs_n7o%agku zuuE#mT!5Xa6Dgmh9q9(g3|h!i`QLDupHY}{3DUhKS~dX^hoy&KxrM>wdlTLbW;C74 zt{s`v;$T}RIX44Fhd(;Y`FSQ7*scx|q~S08dqe(p4aQLjPs4ndff|eaq~&Rb!jn*` zMVwQd)0jV17JrpZJKrzPu?5nq8rE$%{^-R;^VO`iuy;d4xf&#Y#E#{_)PoXMM? z18w-PqiWGeATD#7hC?wLx~15RvlX2^PY4D1{q)02vlzXn7c3bEf(BZnStg2n& zS*!cfCNG(&Wf(*deme|&fWSg!%+m0C0C=pph#eLIa^wYZB^!=Vo(6~n@a{%Lqo=5* zxdZoP%{UGS?pg!b^2=jqik z$`f-3s<4;9%gJp+RG6z;vLyi`I3R>cXC$#0(q_}@xB$YbaCJBlt@LL#UBU+@r!4eY zLbh`Ocs_7AvOG)Ijl7H>p;cWUQZUpSHB(=N4EtGX-1M{ z|7Y5MC{Im~@hHfht%C^h8yha^Oz_Ec*($z zPdDu%snId)7#CcIo~;AFNG#(_AdKX#m?c`O42L)2SG9urg%tqq)-8w|+hJA}b_b^M zi~%AOcC2PhPo!m@L6SZ#&NTplj^C(oE`l_i$||x{iYc3| zZQ^Y(VM+vlBqM!xxTh?ip@}q@vryA(Pb<(1kmeip9Z|Kx`G!#5O|W`wQ1y`Z&-4t; zc1;hDO50yO1L9vQ(K8Cx>TtOv=K&VyC$I54yx|okdVT=FH%?^l#;l4S1zLidF)J&_ z5Z*@qwOShaaTXNzlc?~Y)7T_PAN!7Ayi>TO4(Qs;{rXf{hA> zhH-{#``Hxv)7Of+yi{*Ucx~Y_nX8$HIG+^nN8TB+FY1ta3yvfO5MjE#c02=}V1ue! z@e^-nOA;0ot$wgJ9|N3Km_hE+q}9-yI@md@^zjs!m#h8w1$4)T9hyG^@u$B5OztY1 zruR#n!Z*9{xa>YW;CD4Ak)Y}qN#YlNLp~MUHIEZ^n6_4e_=>&CYCFX)W zmccV75b4LCn-~tM5l@sj7YeNgkAZ~$DD}yU*gKDCb&v%@(OviPicG^zls+r;4XV7K zA%Gm}O(fVp(cG4hZV+JbUjWWqH%pW#kU7wQh4tMityX1ZqoFZ43y9VD=|3Vrz=vbc z0Xc@QXsK&+&*$wlYRtEPF4i~+vAIqK zyous(uKiFcJ&YdHo8L_d;0(wWpq`xUwokXjU44LToD@=ScyI{m#MA&5Bq-yrQ^`k2 z)aF5By-hEB=R>Ca8Q5l#&s;wV3`yZQota!nlRJPoE=P6c`9#p`5b+TL_!)h|SdDBr8m262CXX2as`` z^vPnNz!iVb0S0Oo7GQ9x-F&fSwxjnGiNZhzk#CmWQU&!C6tAtUBV1zDPtrk%L~dfH zleGED>Bk=egTv!0w@|yK(;lRzimJ#;JtKoE#%;E;2w(* zJMBh#OR$EySW_>3!q@X9QQ~R=H2h?ZLV>EC_15@=4mjvUh{#QPL)@R1fqGb(9XM*l zUIJKyU$9SYz@gy@Rp)$h`0tS#(#v*nqK9n^xn_hDt%%QQSx-EWKLv`ljIQg&*Z5wK z8^8JfOTTb(`45rrjuU-KjmX6qTj*mvj@n#KI|9jfhFAm>oM^1yv*GUEp}2de;f_2? zkB$gx9m1L78Cc!siW5t!>z|f5d%FZkvu86-=ur*S5A(%(PR+;%FQ?QC`^TY(Rn@ZY z{=5pi($(nL?+{wrHJkjdX|TK=x#4S-UR4)(?YZ=ew?Uj9<7rnP@jkw+ z$EB`j@)>=n`cbpMMAJ&+(LA!F$*a{X+b5XkF~ug6%D~Y9Az5D_ru4@^$@v^YOg$iZhY)OlI(beFs4R=Iw!J z$*=us?5Rwa-d+`c3pCj1fT%yvCFHu&#i`>e?^#|-5b{af0@Zre;ei18aOxxc{XkJ6 z5V0aCvjLbDR6C%01iRG&%dPjmvD7f)3;7n%QF2#fo9q zw?Mt^)Mk9qc`m9qP#~3m3oqUsA9?=FkmTPLQFFJ!Q)y#+H)JO7=b$vSvzQ_D&R{hI zrpJob3cVqKq%yBA1XxC1RvUi@6xqAMY)GE3!~xdXcR++0u`@N~owx4|I5~m%Av4QR z6m1+$o;T-HIg_<*D(dMr1MJGCS|x5<5M1)|xELn}NejAdi4|Fb;kir;E7x*FMiI7! z##4%WPP}3|S}ugfEk~f=m>B`!X`~kf9WfnqaX4Bw8y1ZL0}$U8fNXf{tTk^L5YjTA z$S7y-b%FQ@UIAZvyiloq8XT&tcfM*x3gFggQbI-(4s43x`W?``5MC>e`SU+~Nw_n_ zTp6HxJcuHq?=Fd`re)5PG{*D1GX@gk$q8oxmL%f%^X=DmyZ-HHhv8{Rf{r$yX0u9s z+yxZcs-JKlTa;G(0u+y!3`^7uW#EYw3!+wuY_D&5Vg=(*_(d`CLc5}djMz8u6%u~T zr;%@VKLfZ^+XWKT-X#>rDl+5+Z2=6Xq))SWz@E*)PaQ^x1n#ED>lzBe>ewif5Nk!N zzTQ8N0SG&+{*8q!iWdsHz1QgiqEN6TgEUlZt}bmYUmR4&aP?kx6h@a?s>imW3mB>ZNjdvWwW?_ftfR8d8;ae*+)g$DRsU4qv~O9t*o4+Ln!V&!7aR9 zmwYQ@WCbV-857VGblsH)YKMyEDr9UuB^jj5D|KyAb*4z|W`=sBVtBQ|9FXS=t^!g>{*ycz>pZg# zjJ9-5O!@kNl~pfws$2rMMiGv>l@dimT$SesdI2Rn5q(ddEnQtgCweC8{pMH)$r+tO znY?@wE{6v0b4P=pce$#&Z3Bd{A9c)sytjGl)B8X$cLTs|289cT!jGUVC=%Re1*Vq} zIOLMvWVThBfCmO2BT2)9<(a`^|J|0oYEn z5Zm48VWBth=y_?p3!r45zWOaY7Rj-bJ^lYpWAPw7w>+N`dVi)Fiznr91AyHWA$=DM z<1^<+HhnDDUhe%xiJ~JMZ-djkUAaWfU@L%}t|@#lQuP+J6D}afrCeC|Bin5a>GN5e zW?B*Y9yT$5Rkb7S(!>gx+=60#_Gz)2IO4%)jJ*a810Z)g`c;0?SZRNWk{_Vx{o}m# z7wQ0fBZL7zz${2RNGz(_1mq?`&L*(oJW&tv_kyr6tq?i7Hc}X) z=BdmA(5UX=TvsGp(d9g2^dc_f`tclv9iA70`m>rYo}>FIL#ea-kx8X;5a%1C&F7$X zHgd-6Pz(Kdl;NmOa?W}ax)$?+dMngBs>;9d(E9e-!%ZAve%5OA2g}C&H?Ew<7?o1) z<@CEzZTN=HH}y)1=DnyNFkNu%G)XSBeFr_N35j`pz|2V+8+m@;G&X%c04cz+A^7o0 z;E~o2;Qaq>3ZE{LtvJG+a1_kw1W8&bO~sCz5`ANhrS8+i(djLpp4&5Cv#dF&9E$20 zQH4gNKW|gdym=Lk-=^rkXx$KwE$R3BbhGs3)J$xSkQ( zYF#Nu+#6YQ0R_xZBK!^7zds4B*I}{zMyHmrn6V7+l^7Nu*>^W+zcf|J$W3qmtb7oA zTo<4=EOlyQL`1nRX|Q0$B>0=+d%byjv6vbB1+h;%JuvKz*{_fGpWo+g{j8ioGHF8+ zCsoXjMwIixr2W0+m^o!vYCs?)oj4z|o6*UL)6slL$u&m#BX*(ip?qcyhGUda}?S1;8TyS>=Dsrh)dR2UgOIH~(*lnAXg_%m&G zK&G*de_juhL!<1&0?5EO#ij#7WSyqzQQa$_1b9svtTET=FrPpZ3NXXSk zRdy~wj|XpW_{=6yzQaQPfQU$w17d_Vr}-NR)>ahGG-MA_4p5N`lfH#@gT!^k;{oIE z3r!d|t{vYPWfFxCoT?k5R9{tN(YIi$B^vd9cWHG$dh`pLt+KiNrB=BztoY0l8>3vi zq&ry0BjjxyGo3;1kVzQ*wcse~oCQ=DT`$_D)a8i)#-gj>cb`u z*Q*VAlj^Wlnmxr9o2c|L$oOH&4}KnsfkAiISa;6z8vr_kz%0c0SN?fcaB|&%37QW}xRI2>rQP(a#})KVXM}_$}f? z*ikYOm+D)SmL-BbNg)ENtKcY3byc{ZI+Q5&wJ>{sb^~hGKydsf7`yED?7kxu_^P7d z<%6jjEA5I-If|`^z%1I4o*IhTm2MtT5%PF=Di93oj1Cl*4tq8{yr26tFQ5qP0fS94*spS5LT8?auau@4|JG^`#r_PeB-xQCHcu5q z51n|)YgoIJT4fh5d7&yKc-NG`TRZh|e`?fW+c+*rzHn32B9)H~*CfN4-NDxY?v~EE zjqvD`SxnnK>$Ksq!<=q=7E;!^FJo_Y`BNuiWvMm69tS`Lm&PCJx}A>HHubt0$l*0ab_Hf?6wsIg4~UJNpxMooqt3gf-VCU-*VmFgsBZM| z)JIzLnpR5(A;9amdqPot+AqAHoO{j}$6by-hKwvW+A%cTS+S<={ayEepDW;63#U9L zBV(gX@S@g4MXs)N@mg4a_qfR!8+{wh+;C{T`KVvpGr+zNuTc1?)NV+-Jj_%$0Hkpy zw!oVVFz&1wuz5YYWryDF<=Rjn?-<7$DisHjxGwH_f2Jws6k}y)BetzGp~zr#-LEIp z8#aZ5Y=>y7$`T-ykN2BXv783h?QAh(-at^3e{Gc~6iNn0{p;`HlJC`mn`h%EV}eC7 zdme$&?tm->uWj_DpHA5KSjuz+OhmP~oJ-pp2eeO1DLs(y7FU5nx;`?vlIUJgjph-a z!XY?YDa*=b5ZzTF_SM_9 zm-l~7bNw;`e*3zsO%HId%2(>`0r3Q;LIrZ@V^U7-Hb9&%l;rbv>~ruR)Uz| zC$TR5+i~WN%CzsI18$-<@Zm!wWKH@);K&Pmf&2oBK2U^{`58G9AK2yn1X)6Dw9k(L zZJsOtJ{GH_)N}r;>rG^m7N5)}eR=-4R1wDxl%JS+RP|u6`)_iZ!R{6VZU@LX$737_ zpnU{S#X@}9CHV5Gs|T{s>O}Dc)r;RWayBh<$%=tk=1)iIv?!75H3l>82Bx?{9<(mW z;P8>w53Hv=@R%l#;)Y1T$u}3;j+=c!sZHh=8`u73#~p}7hF5JQ%!?HIo<=gOofWZ35J)4=5z1ofJd-lM(9w|a24Vf>|E7%S z|CC{7;-o|wgs{nV|1+5YK3bG6q27qY9S)T6KCu=Kn*h>5sYiUfZs z7n`l2^m28S9uOskeZM5jWNRzQI?KV#N+dgKQ#yWe+k&{!12 zk?-JCx;ZEgWTf|1G{W09s6Vwy8+lnH__sKAFs=izRM_mDCloStyAc-57kiUFsr=?-gmn@5&=sU~!byhEFDa_c&XYnay z|5sPDf(Ik~^s@e9fOM}JtpANp^6f9VH7ch|1Btb1*oZ9>{Y+li@4YBneJ_sm&wFwB zhdUSkf5uZqBk?D8zY$f9Lva90D2Uh*C|&DsdJcaq)p4sf{mG?VTwUgrzz`cv3hDU5 z#R2zQFc{U#M@4kH5nk5b3JhbPkiQ4fBB52C*vi-2#Zw zp6SA0>2o!OS}swrwxL;vE@ZY6+$vy4cO(kf)E4M@%<-H#Vh-$o68kxxj24U{h>oK? zEG-D1s#NK;nRu*+%lus8f(e*Hi)KDpfShTsfdcD>RKbV{Q+-)B`qrm0hbZh7!X=OA zDq|xM?pwA*YBhY14eC)Y$SWo01Uk~<8?5R1`ZZ4drKVI?y$1*l43Bd>cnB@q^96Ip z`yVihW)wAo2y8tsdLJanhf&W}P^=+qLiFx9RYM;;#pj3F0u@9D0Pu85RqR+FZ(mf= zFVjKJ#?++X(j%7_5l)+r$<@2B%ok4%i!f5cw@?)RuW9qdU=5Y4ddsas% zgVDsjJU}S2t7aozRSri^sycCK`I(dFG)ZUTg<;(Dq#xP}$+kO1EK@r3U$vOgJE;#I zERCp)`#L!qXO*O5{7%EMG4VK|4yP?y*PzUjFsdyG|?joiwq|fY& z!EJ`9izB*IGx1W4Q2)R9sAtUBpQ&5Sr}!sm8B!`N2ZC}Htxb^P;lx3QDKhH*npLX+Thh_I4 z$pb-)qIjtr2y63(Pb$2Vze0fgYDQvG+~# z+H`PHEe<>8njR}dNUv3mLU*TR+$X0xt-VMz_>bWMKZRh-DSm-2UZl3!fDcbpZqHb| zPf{cx*?ij9m0h2OPkhY!m>*26=L=@VY6|01O_h$1K>%GtS?TN7i^8z~y0o)D1>y|L zfRXMbNEsvROp|6OL#`~d7T<-%Ms{^F1py~8SeJi6bOZn;c0Qwa(&&*>40i?Q3!qw5 zB3MEZwd#D313 z;-EGU!c9}+e8I3diMk{F#c(eI98PpNcX_xQMc;X*$LJ^iiZ;uHq6l6MG;JW{7t59E zBvI!k1Od14m+3$NPH;JsI&=*!QkH=D1ARohUNdxD0%Y*7_Dqv#wn*?9OTF314dM>xOTS+Q z-aG^PzUc9M!XE&Ye78m1Ma@Zd>OVYCigp~EUbg!0Epp9uO?uA(tf}OP%lWsuK(d{_ za$~iy%{L3O&ktA_Lp2QP#S zjbI)_r0We5|8akO{&2++A`0|%^3hUpd4c?GAWr?FZT&gObQJW-7L-A;wFYVVjtvrY z{bf}cURE2dzf8QnSdlP+1Y#Q~5*c!22Cz@`MOydv z4Y}`2j3|*?orz0gu7FOAF^pUf4fats|1H^yysFc>kP+QSHkV8=PKW}*lou$yilJ*k z9;tInyd+lu?u8r!S;k+Wd3X;XgWGoqHxNd}4AukXLwJ*L0&lv*vB|2l_@_#SbW`n> z^wg2nQ7yriJ{40LUprnTMHKq3FLl3yDeBfIuSe@5wXzGAQI3GF7;9fRpHS)=Gh-%o7OU zFrvv6EJ5qr;y4U;-tQM7bU`4roKJ1MlRq>Mq~^$cIt2|51@Q`qSJ8Hh8gxp0^DY#6 zJp6c#J^8fOYg?8aYaG}>q~bqfx?EmLHg2=;RZ3o=#WnLV%P*I;1lc;A3~(k10K=2e zfl5H~(lCim_ybHAa~-+!UjjK92he{&!(ck^%4Z-C3oi`H7#CPoOXl5FIFHHc2Pw;M z7~7Tw0>CV%Q~!E-}-LD2D2`!MolZL**>7;Mk6 zgH6k{fdi;1|1s?xQr$y(n8psB=ye|lF%^dIsJmlSGL!vLjbLljc^KO(v@kG5Dso6A zHAkP4K}bw0wf1(s1P8mxx7nZ1g+COHj_`e}r;pn>rcIz;jFp_^ZapU6e?W&CuOJ#R z?+Oln0=rssRz6|laq3R7fipNsL+1ocihT|nG(?~%B#KKeFr*nCf~4`!lBk)W;PQ72 zgH&_swI6|GyZ$9;mBqkRMn|cvJX8-T38J!|Wj-Z3_O|&L3$Zx)mXy8kFbwYJq@Qfw zC{#JIX*?Y5S{|{{H2m+B#&l}UmS@iwPYyShrIP2t5fkhKK`0%Xbq9)Qzgh>(0By$+ zU`)ORv}V?-?@S)`cm#i;)`)8a$8*4Chy&SLDs{0J&YIPfQ!D19smUe62crs#ofiS>;2$VMnfQmX(W9S|BQodkbHzG%OkbS5a-R9 zaR@lKLEl)zFggm%2ogJVNe^|k{}$u2m~|B!c43sg`Q%xe0j%wN*<1!P4b*?Z0ffP) zJ!o6`(2*vGIK%w_7@UDVxoq{;splI=5!JFX8OZ@Pw%(k5{X5{c#|pABeN74pkmohc zhTKft_T?y6h(ajT{@dp95l!c^zXQ=^!B3u+m zpK)c`ZFckX@HADqkom7mUkC7z2;#d%jh;CmSAExp7H1QNnGJX#-TF3d!+M z--*Efo)1IO>lLH^Y1%^6lbpf)BNqIZV8ZZTMI@y0dwt(S(;ubp+QaN?38OrfYPWJ~ zEqIBmQXmfEQOV2h;<$l~&}|~mR8ycs<^nXPwO#3wTvaT}2YNFg=QIoR_Tu{|hKZMY zVDgk%R5G+8BgOWFylDKSWjk=S^ii?fM&0 zeXQ-o_PpW5Fv3+~4E9jexRM$mpPVZky%_WOLB^G-QK1h1Q*sCb zr&gQz+y&iSr&kGzwChD1>a8e+M$78$^z@`)lC1$h);=~&eF7qS=RPPu zYg2Cv)qDfnc6%_P|C*K;DVVw(_NOGXsZ8Z8{b$a>wQ>%xbu5jW#Pw^7L+-yyy@-Ra zP-}iuC7j5+c^tGOySj0{3ZRA2CMzI^UG2b)U&;it;T&}Z|GA3b9r*H!664a)qwYeo z^#fsByU>v#-FZO|%bkMFENMhg041Etl$}(%yFfBJ1P=X^LPW1WKzoHe*U~traMTjQ z?fn5xuSU%bBd!_$toNb#KBX+k{r9)4xxbcoLjyGF!Wj(*>!!5b7&n6n6PD>3t!<9-HMiY|MCumH5Xqz44>irAj_w?!GqdEL0ehF+~Sju{i0b@r!ny;e@ zv_GSVLh9vGTk=4lyw$eYl%OB_VaVOMWJjAI36ddn~d=u zklI%Jd_IgeXvwjJZ0}zk1Fqg)gJ!DnnMOy0(+pYd$17uf9!`r-oi58Q9|JBuCiFz# zfGeZN-~Hbg8A25bDZriPYo|-qY}R$VJVeuqkQYXa0Hi4Xk|==k7F$WP9Tyn#&O zAQH{&gO|fn6H5m$wHtQ$d|OZ>ez4s7yv*?a?yuGd2twBwH#SHzGr)ViURWBZpxycCM*6- z?c{j5ogk!Nsq1igH0!TIJ%gk%TyqDu4b6!+3@Q&|+dcQK->-fQ{8ek#!F`ZOCNM&K zG*dARz+2xua2{p(dq!}P-=4pH=N(7Pl_)otnj`M-(O^9Qo$~Z&CS)^>EQ7!PEwT8f zM&G@CkBNo_)=yI+Q|g`3SC8c+uX&9Vbf70lTg?)zL122w7J5#(**)1-QEPq~8PyH4 z>x`T1r(9P*SoJ(DputNsfkl4sI@D%we#>U?;{F#lL&Ju3{!{0Jb`#h3fs^OHU}v;h zd5YH^?>`t%dN2sA zjaOAM5p(eOCi0U0cqdUxM2DFc$tQrcPjs7aOU+IWbx%jra`juC7p*S0W?i!i99HC?x{Z4}3S zOIt*6t=t>W_8clhoj7cvWGKW-j+4=4Z_iYsSP^tt+Syy+XNA^=A~~WyY||y^}`9|4UuD_*Rv*M+GrK zg+VTsprALx5sv>p{uj8A?V-kb>Rc&f-%N>HM z`$*S3PgMxNh6XEq^fl{lVKN=Db|1I|=qf}I!zSvJ;rm01bt9>*B^Xa+5yEi;=dm7% z&dAmd&}@PeY4{~klfdaCA&5v&HkHg8I6r9$xGknYwiK?Dy}d;fFD{{Bsof~OZn%UB zIyT}r+dag>I#B9plar9^xf}Y$s1&}(lcp>KyG^BDl^tUo&=D(Dx2RHWyBwx!^R6Ik z`&HRAuQoa0zZ6z@mhY@Fu$q1v}I;8{X&fF9&mBgn0d`^5cO29{xr@atSIU)+G z)^~=}vdxz>HB9yn0i2=aO>u7q?^Zwn9^CX!&4#>Ev;-MmobmNW*C-a9TP<^otkJ;C zFVD8ZT0Y){F%F!AYSNpE#}QhO?mXy})>KxBK8ueG`!Ks&wE-4ixrxT)*Po_wV_`m8 zGgQzG#tCJ|A>p$d%asVU&sZ%syk5(fMygn5H$@d-fw9n1hl0^}rTl*dz6qIo{rv#J z=;k;OK@L`eei8j!$c#JgkT0wqUv3jLsWTWcey-1Yv-)+|^G3OwL%9?r@h2DLsMq08 zHH$W$0B4(;wz+Q)hE?OA>y|@b0x4ax7>wAPEi}8~4`VC2%*Ngof`BOaXz2*K=*WS* zd&@vpkfDHXqS~-jt(3y(%2V=0IwaYo%AoEJj8%N^&z`7;2V5R`|5%eCDAHuYRHybu zcI`1LNB!^e#OY5#7qjIHVSoI?ikJ6yppK>ea`Af}K38MX==&XviF$DS3}_aJdOBa+ z95U_g5^)&IGBQ3e;yJk-w<5r6yJBUdf&K4!In@BD6d&?@=+_u;eoIS&d)SXbK3Y&H z0f#B(tNMz<1763&PC>6jY0sHSp}w!4!M!H4Q<&W6>&W+vTWT(mNY&m9c| z3bp@z9_~rb0sGRH+Uo;k{}b=sU7Pq) z(?^l{iBm=m3~WXvV35UX^@H~!1@Bz}kI82iz;>wJmi~34jL-^!fm2P*5N_CFFUVy#kWU7l0>Fg-Q#5Kk6lD+RQObboFRxsfiN7LWZTHSDF&_mykVgth9t@~a2O{Xj<|JI|DBP2@`aI~AB5^-U-e+M6~2 zC(s^{SRN;3QIj!9r)Qi!U+cUEXZt#?&~t-*?dvY+unS)G7u#y+ehCI$!Uyl;sKXMA zt{1ZNt<5|aiVs%-cQP%4>5KVL?%R`Ut%XX(dLhCNy~P~pIRs++XI|oPsDz2RVRS#+ z8F~%34oGYdSZ;B-VrKyR?0dNo6x&9-@hyc4@C>ulZO?~eSA7;gn)A{)5;RF?q5h`4 zfx078PEMQ5jcB(H$gM=sQLn3ZKm_R=SK0^7lmitL7VB2UO zpiv+5GfBb&E82m!n>V~%-~x5jj^Zv#tMn2?^r4>jZ+9)rR#s1>)@6E8mj5otq4ApsRQ2uk@aE2-A#6o81dVIoaI z*XLjo3uiHHv{O6J7H}GrNdN$h*2id=$Kfk|b+-4>%$Y;LN}Ic);}Mr}Q+WmHZfuGv zz+u<1<&Pe}5bsflne2Z+x74arroOYy{i&#B-*@J@|H*LeY8CB`za<@d`22JDHFje#8uLbyKw7O~YE z!3A3_A)-0Ef9B5fgcwF0rXC!-iQEJs*z+NFyQgy{@(E-Wj_Q{?f?%0={<(4VYF7#| z5=kX3w(y_jGj^_ zw#DaRWH^cPtJ)-)iml)TEm==m+)J(2Vs{y8`ajV5V1dMva)>rP1J<$6dVvS;iO{KI zx1V_Mjn_zupg`OGAX%izpqfpp-1Y}|p3U&`l8I6h38AC3x=w76LO{yDIhPNW4o7tE zP@5N4eq|+gcZ!P~*FZ4ct38^{ME(uHSLGwY<}uF>!TntX6%*4al z0fy&@I-tk7N#ii7PufBadB#c=nA#75g~zpi5%8k7ZKs`S=n9C;I8bYTv6m)lA;s+!@v+ zUaRM+rtKaj*+tNYu7sZ~HllGc+Lfs80JJ&3Wj$JlbBwWII)A5MTq!b}!h}rkcCjO~ z-^-02Jd)!Lu<|z51FpX$N0qu2c=s#?%rO;6_;h%%-mR;EV3vn`M~ zNh>u(`)swBfEwQbBZn4_B>RAxeh<{~qck3eJS|xwK@n7xtu=eslwpIop;gqFqM+tS ziF#${a#F}|sOy9O_%IgzJ{ANE)lIv%J zkJV;98LQt|CT=#IQO&_UlZ+1-8b4X>5UPq?$@x9qvOjkJ-fbU-4&dM7I4#T(Pa6#B2PO0_JFYa*LtbDQW$m_H7cWHq=f#7lykSfL%9@ zH`I?Czj4P>m{jk}2ATWETgBde#4KUH28c%!=@%2<8)3YBSCP?$0Ex1T*=Gc=Bh?=M z5YK70?17nt$iLo39uW}XVjdsB(0m8!)^VJ)J6DAyHHn4;FG>2)Uu(LPUvP=w=x-Ky*pq z^~pXhHo^@Hvz>)=k^KI1{vu2Wg)b@B8ni`ESILQq1>(&zU!q$oMWa$aU+{9sx>hDC zeEnR%H#41C1FF!|X@6tUVCvn~d~~YFf-eSpVYkK7&@hdP{|rk(Sb5wjA}c_- zjPVvi|6vGLs=urDz2A}I{n^CM>k?*fQ_?Nhk?jUePnaBTDzcLz-yC7?fiqG{!(O2v zfm76~4oP$9L4{dI3)9Oki-u?$nCvG%yY63sg=GkHvS~+j$hd|Lb9;g6!`%k)ZR|qw zxNAVx(f^)7717*@M7H7kwja0ux$QDq*aq)v(v_Sr?afr%pp=Y~j8{VdQW47+BrGz1 zY69YCR{R6zi7ig+pCl|YJDB8?j5d27An3%}+dE%*?u|95`Qdeb$S-UEq-gTcs?4g) z^p+?dyVisx3G?CE)qCF`=07>zJ!76rAhk)V_wl0 zcc|QI81W#AU;G&>+)?iD#4qy>zjg>75abz(?}N$7=vW#Be+{Mz216B7o_q&REpXKw zVca?DlMYbK2KyfT3HZ;I{l+Rd_QE>DD^{?$@jCyvl`zz8*HMB zqF~n(->I&f0HcY}0=Nzlb|8C21jc(uR2@>CCia`bq1d(FlZf`Xr2}+5-0+x@*C_1i zJ;^bPF-k-ac~CUOgCgJ#mAfp7x?9a}eOA#bA3YS{3m?n-I4oThPbopxNHIYlv4kbG zVwv01#%yU@D42Vr8)p@-kp4O4>l|s-emPooBm0&W7S_~d1%(a~(({!z9-R8tDJ9f_ zSc)iB?jZWqnrXHC2n0D7X1kkVkHA!-X zBv#_7*vEm)Uyyw?jb-25r0@60Q;AD{vXfm&85ASp8TlMABCk*4L{Li4oAsE}SMHJO z0p%~p87YwPtXkhV6s4*jO*?!cG=z=2z1x~7g8!oQw}kup&C0UmoL|u%Q?T{SlFaMO zbDFZw-VK|vJ{q+9LfI?s&h(hv{daLN4JTThX6Hpbhd8%>nQ+DCuV|i(VM*mb?8#J3 zRS(IWhzuSTL~$e=nGta7a~2n0k`~G^#x;1~z(`%C zE7Z|RXK318u=bauKZ$1Iaes#+fh^ICm`nR7o(X|b;+!Wu6FG~VA@nRMi&UU-`2;>d zOZgxHVdu*>PFR%AWSrgMhs>c;nnGC~H~EB`FU;Z@8xP2mn4Z|K+sEN&#w3dGq#%I= z0skn>gd)~p`K^?2VP@^(=m{gE$NiSL(OR<{!A}-X?FjW&JqTVCSGW%E&lRdWst0)G ze6sLX+4Yn4o8ic`lhvOxE+MbtYmk@r9~)jt^RsB%?~Q2fC_hTGRy3jVKAyvm<>zjA zeG%2&!Q5q6zcrEkNT^WW9CILgMhR*nqvfbI%n0B{NdvgdLkgvmS|x^tS`KiX84~F| zEUr24tf$PXC%l;PdT*4>AL6E9cT~@QQr)&b2?8eJ1W74-Z9RH-#;{D=XV3LKqe)F< z>Fo@hZC*zXZ~DgzK9$7v9yJ)tY3h~bvm-g8L7R<677D|gW<*!*wbP!}KUnuPiASS^ zJ>)BUy-_Y2U$Il4M4%!2lXGc_C{fii?kS9TneUF!FER@TG>;UM@l(IR{#++W5q4iQ))| zQr$ra?QGo0a*A?nBF0YP(5Dz9+2Qte5bXGD9zC9feV!P^n_1%h(T3;U5*dz0xrWo< z-k&7iZ%HsR27YAFkE6w?cCc^0zx2c7j~&KmD;+Oe7RLO_meMCmQ{;NBtVc0Gc7*-DG%$-4dkQ!tbIHf&R{FO`MNvxfTzFse1-7 zn>^c~Eo|P-acImX`Hj9`iSIs@q47wb_@^(oKc9~6GU_ep{Uef3(xq}SOntNPuEVh2 z4lH%|GLDS>zxLiTD$4eY9)@vfLAtw3I+c`?7Dc*2x=R`c=@t;_ZfOJt1O%j{1tg@A zkdl&8;(v|${GR7s>;K{X_I`O?vvfIY81A_4bM14^-uqBdaIPFio|QZmVwS)|)x;un zk{dW6y3av#hMVSiwL!ScVFbpAmhi5vIl<2md(~-qsx4xo`E;r~h`Ox9Nvx zG~BcSFLjvve_&M)Z`lca5Im;OGHC6(ea8QEm`U6yV3Q!qIISG#4K-drDNO9D*=luB zE+%iX^y!pcy%gpI)u|rD^NXOq7&+U1*b}*L%OlFnVe;-Zz7Z{6JS4Nf8pi9iD5jnk(w;t3 zv_Uv|cZ5hMBGM-!mdq+Fq_8ND+8tP7i;b>lPvvHnA{Y=^6o{nYMUZ405m~dQ@X4+TrFa&ve0!caV zguZ4v(JTaR=1E=jOoFqP-W@4;H06+yo8Og@ic|5;Cgtuo2s@IGRG6P$;ajD#W|A88w7&>5C>~eis?Uu?=Z7 zshg4iTywJN5G+Z?3}-OjOhoNcQpmIr$#O>XGSBci_vJkI3*N(b6jRANzg(D26ity@ zc0H>rCZ;K;q(~1-f+DAQrkeOEX1;kMOG^kn(p<(TdgZKXIV+KL)Qv`41tFG5$vN)m z<%>$`$_mQGgwxOlx}bKz2U|F_@(x85tnrHmM^fvhu;*A5YLD@LW|uj{OHa>ZR5Xnn3$x*}9}t9j4Sf_H9u>V1PgSvmsz?KqNZhxVWo?9(H>Jgv=ojj*f3U zhl3G%@uM6&o!&7T8Hc2&`^^qJjUBJEIPx>*Y?y3H-iOOExs9(@as@9hP(DUdax*YN7>Ip{x6(uD#b~{p zpnNM&NcjdmRiiH-q?;sPS`$!3XiKQ&k#f+KPMxx|i^s_eS*U9~fI=S7s-tj8B3nye zr}7Tg=F=1~+rGLJV1sU;;(nQpUr9XNS8e*5%Mh-2iF0%|+AXdZo1^ypE7FvlOLFLw z%b-=Ss7IDu_m3dQgUY<~*)J+ zrga8_4n|g}w720in`mq@z7RUxYhtTg)Y?hRs=-8j0wczLYHNJ6mCbP0+qiU&+bo^h z$MbxmVkY*cYZ8mh@)dK}h+1ZX9Hf((5Gmlr)iK~V9o;SjS7Mq`vL5FFzyTq+*m$)mr# zAJVc;LY;#hkiej=%7}?Z_mJbEVnFP00BbfmPl1X5abT6EGmGL6qlG?AaQ*#%z(KT{1zE8dCw@BVATGe%FE*I{LhKXZS#3Ua8;L zEYzoZ5I16kjj!}6vWK<3s zGuf%-P<2AiGMzHj(4e|4uk?#B4s1r^iS#P~Ri}_Fix9i^+s?8L<~h*F0pZ zyDUc9UU+6DceYK3DKl>I8k+2M*QRRoF>`!eiH&PwGurd|D6L4#pu$UtN@L{YWp6&R zc{lou0LS!Q4?8NyuB}dxeU?zCBUL{R`x>VakC%RZIC@&2tf-KvdDVxaKFWNX#lVLI zDIwuaAJ>Y|iMo^-?uDJPf{BJVZ^ z#V&+N$YIp@y*|S>-6pTdIfl8FD`YO$czZ@ibM%9%ev*}q zX-r&;3zr;x-QmACQ1e;0(3kgOM#H|NuCy}g<7m|m%}}*ij!r1U@!_`)e&leQn5mZv z{;ubh3f~R{PS}!s!*w2xM9D)raQM+xhqO~S(?0gJb{`Mf2Mm_3e4H4xUy;7VZGDvb zVO^{9H7Wu5T$Z!l%a6D4*@kpmi=5GI245ukCeGe6syYCs-qSq~6}iYZ$I+t?qxJV|PD zwFvOxa4y>A`P`n_?Lg+a9)nna^v)=%WKZu1df`J&X(^u2s9OSV=qO-}5e{hMgWaa&w ziQ}C;wWYC5qKobh^^&@L+=P#>U}8`;*#1#*oeMT8+%8boJs2`I!^wg8MDBz z7?XrBjQk}c_(M<1*cwPWZ?CQ11t>}zTG`3t=^d903t$MHS#n5o3ACh?-Lv=Pm<<%C zww_PO91+8y7?92WUUoOH`sE?nr-sYO!`D0fMgb-a`iCW(HmpbP>k(3(7;9M_&stJ2 z$m0!#-c5Hn_^)(@4wYrUxM7)9-#EJ|uX|(d42Zh4nW>?YjO+sGRQZQ3vuOKQcx*pPZx8tVuxG=JS-s?D_XL^HTL^e~z>M-nZB zyU~i6L)dYC=1EPc}&Mjb^J7vMHM#i+2XJ%B2h=vl!szIpxuXZ#ebwlyE!Jo zMrj@wb>^~c?Jsk_acO>ka?Rwn#fqvOW?sU!H zIpdVy_Bo=azaz4~%+gTA>*s;B62Qqe-fYOy9z&hq7HL?niBGi8 zB=q+w!_cVlkt094OFG6`xMJ>fh~1|kvS3CQ+?o|k7Yv9%)h=t+4$pzSx|D2rw#XAM+J)Jb?R>PE0bgO^F*(j*%^<*VGCZN4^1~XYJRO2 zy{NNrpx@`8T4G*jSm%z2uyhxxvvdp)x>BobDIyy665k`{kuVS2^1heTUR|p1Zvfen z$y!K%);Y`-*uVuTW9jhYveYv}GR^AId~p>;bIq5~~kMv)G3E`O|q+ zFUr+#Lr-GYpQR4dU!jZp3%XTU%c}ISs-3awC*b}JTDPts-h54H+R5i>W;@PAh_Q(G zBLQbDAyRA@G^}Ip%9!}m;nCFy&_1f^;gi{WLiGJTr@X}gsgv)$FbPY43cE$|i9XUA zU2_HAd4bs3IjZ3CEry`!3Vl&ecL^c1*#2;!rZsh2Ux#Lhr|7v|c-=LPWVCS<`wqhK za`dX2)-s{p8~V;1q<}^EJ?Hq9a9bRgV`u%Ao~{hfz8!DP_3&o};{B9|(vE`Cql6NrvWYjG+h!ZaA=OXy_lBt7Gto?ZiCoMJP$=maV zmN{25(hrUUM^xEvVR)mGQw6!wX$=ZK_@+p9NiKdrVEvigJ8x5uV}G1Ib0k!zFfBY3 zl-9d*g zEmGU;2J74G<%ZY|As(}V1WBcak1c!AO$ygydrI}md#-IK(6y#C;oCx5)m{$u;GeY2 zp>U#W455dR>X$-PH(tjnMwbzqd_m9v{nlM?Un~>uuUq(mz0JG{UZ`*QY?yE}eKDqk zgJY6s{oXP-Rv&5yFR?$lVKd0=`$EnsZ~crz>Dz79F4`gFVRbza~kYrXZzRRnhgj!u@MdL|GivcfQ`pxAQvnB?|X<|LPrB@GJ*^9=LK8F zrzJ(_9T^S|Sw7_>6(7vvF+JBmz(9R^LO&06X9)fEv+V3Q9i#x7(AeEVmmo@q+cynTGs;5+i3>k%;hB!t0z3kDn+*)8CB;Lm7aU zu$G7>4vlgI8;$D#$8B`h z9LH3;`p;MyIQRi`{Ahz4RQ5kq$#TdOCDvpz_DqUZ(jLI1U!YO}%GRjFuU%jstUgIs zm+0?2GoIl&bC^;^6bZ~P>ER2M2@)~4U&XzxJqhS7Hl$W`v!i~9q% z$%^XW#!HqQKR(M{YH=T2lFU7iM=iVcAI4_9!~Jcmnm5QXhtdV3jcWH9l5ywlclzz5 zhSLSN7@1}VEdjT0S6Ak(laE{)fHk$Ij2gXwmJ-7(FO2Y0oTPk}dIx*p7fr05sTAMV zg_HP-g6d!Plpk}#k9yiIf)C2~*JaeaLp!bP!Y_MBj7|Jy6JDB2JS2#T0}M>%307++ ztzpY}YsO->4kJ-})JRxbb*kYOqdrZB3=otPpHvt$Csva>Zt=HFP#&>;r2cA85&80S zvMlx!4E@&`)uhjPcM>g~Y=!R72-+rp++0%2y7!3yAIQ^}3H_Xh>AHp6g7Yp0ibd^1 zUzGM#DD#Gh{L0Lbsl6`JsgPOwMROv`yAuxboAYHpQvd%w5D#RR0lP!L)(oXs)xK!g z=iz8t1-DVF7oY{Xm~^;*DOSw@q`2!GX|AhV_kmyTENuR*8?ZuMtqpqL-RLj}n$+RL zul!CGE`in0Z2o+qs-k_9XP@g$B4F{9)QG+T><4@C;fD@PUYrR=E$(cMrb3KOzE{h9 zuvkyPS1m5QdTlrcn&e8R63q4BluC_My8QOWiw1#1h&9%x{sV~cQWd_<4{{$&x$6+8Xg^zra{jvU>9v#~g;`{z-cNa;j7 zk)eL}ZFqMtQ>33xzDT8(47dcx8s&D^#k)2vUqrXQ!S~Ej%(zzg34RW&pL~^@%)9Di zCof5#aks$W!oj=sB|x_#SFm`?`3&%RN54IkLQ3quUF;;{N_*y#AMwK>;`5{Ln^yj{ z8TSAe6hQ|ygotg|&B2fw)h;!rS_%9y6jU^E>QL`|LOOk7NYB+-sF)N13*zQ9s;tW4 z&`uZ(r)^C3bM&wk0emb#=?cm(BDKYT$M!STjTcqB>~Rn41AGL{T`#jhR)aRbvT~_} zQG?K#p^*uLKt6m=yIY`5(0KMDIMmFNmbAuYj%Q)nIvj@z9a(;-n337)!D9MzMy(pv zOp%v)&iW&nrS<~>0J5CKmoIkQ@E$Xvb^}V<+$lj^Ldu}%d6t6 zn}e(P?@dRX#OJDq6MhvOSv}chHh?%l>DlHt6EhS9%hE}L}H{JFBDp=8sqD;(AQOp@F}=~8a*}s+|F;NofEWOY#8D+bdEC=Dl0StSZ!4b2Q z&P}aN_I7tB`0>rX*U;vI?To)_0*g>2iPu1&C>Ex#*Z1le z&;%vG+z3F-&we#h`soUv*0M6b8rp~*_@yaITsub!tQ=R*h`_Hm7f`k;);}a}q zLi6c&3Aa;3DgcqM=tTda({Z^WAX>Ws4%)-av=Ib8fn_q_E;jcM8DItr&{-X-^9*i3 z-}};I#=Key_Ro62s2ftDnN(6j|EBev_vin#8n01=tx4pr0L=@02|6V*qyTWY=TT$n zRhK^l1DZTQtSwN8$1!;RnGUbt7gD#padnp#3GqcGFgQc;i>>k>y6`l*W`&N+r+hgtwoG8m3|wh_ zs<73+;=#rclyz(Ftb1I)&mR{F5@OH>Bt~E^1KH|VBS0YZ+Cj~Ivi;x-M$K#=ung(p z!q@o@Zp(WR$mbYMh0z-{@?|kReY5$k;ywlfz1~nUotM+G(GSU+V@VPtuRQTkS-`sd zC?Kj~XR@@6#~{CaF}%_7f0blNnL){g9WMdnufnK}{Mm>1ky#bvrVIywKL|tzs}T%w z9fHq*s~OlEfbrG~GN>=gb3i1tnJBg|?OkjJ$iWv(0v34-$WBow`%5%MgQ)Zkwx8)K z`v7Pe(!Kxud!~p>zHSZan=o*5avkv|-WoMI8oK3QpD=U~d;SD#645{9N2bQV5W6Id zy8?thI>H@5l76gHQm#Y?OP=2b&OoRqoVs=0?lG!C-nFxBzQQYle}ovGq$CcfXd63V zvO%mlYu6bV)UwPWEvi!<@kfTi$9r$)X?F?g?WRfRo2#?A0jz8Uz5MNRWl1%v4;K~} z?vFo<0v9%k`)%RfSuhF~wbwEX>`eiH3{+O8*#M3Ce@HMETD|%4Pr+aexjZhz7D8nQ zpjBA9xsp-jgp*p6A^Ut!(RVc6!5z0J4|aJDh`5#(fKMy+zAh&s5F32-11cpCDe|7$ zO3zypk{U>i7}?x=iP`iM?_{U`Y= zfc!N%2MGWE!Zi$&Me{lSkP!gYE&+WK`(2j1;*r2DZ_BHnC%nVoXKKHK1<>sf)b^dH z8lNX_1n^K3i7Y|lqt_{*tOmAd2Z`^PB3owfDgpDZ9c?&6cr!sAJA1-GTuGJ*aFI)O zlG*;^1Qezq|F~CHDn{nrhTzUiznQpz(eVH&hQXt6IKi);8v8M&_5xR;@ZoG>GN@zP zl4OKQK8>t@8(osp!DFefj}>3i%a!uvkwZQ(L{Po#z+Qc-k@E+_xxPR`71bIpFyH?C zrmwxdnU^&dV!oHm_5`4|u)e#|y|`t+PWSGE!O85Jr9ogEPrJXd@1EeL-U$T z4W}Wxv|hG9$sZ42Lmzv*AwaSJHkK#5n=eXJUC*- zO%d|(7Z6j{QC4<%ya~R-9jp8fIr^WJqw0BIt|8%x!k^gKhrv@Yyr_+!uV>x!Fajb~ zVE90WrV8cym9u`{FDQoBPY9^C{BnAGhyWoKgb^~WNnld?YoGj=G%o9OMR$pSuK}Lv zz>ND@V4V`9oK#aw^l(ntjYd;;ZqZgVCtUg+>tD5-R68!MC+UW?F8t`VOxB+O z)6W0_ZWd&xa_%#rLgo~v%F5e_Y7)N`ESoyO5R50&^Y|Umtd+JXL}+~icUZ#zXTk*_ zHN(S`N1j5CI)^`Vym3J!)lv{>%{Q+HV=T^fQN%$VK*TY-A5~cs5g*O=dQ@HVN)xEB zrZS0gr!K`k0OuXJP$9%EoDfbp$ZAns!G7dWjDLuWzCYUK#B3{HO4H!QRX_0iD&r6! z(8_DB0mQ%pu#4BZ>~w`MOozD z5}lIu3M6K9cH7mObpE|$6(KABiA zeH-d7?(+oQ6k(TD*@{NZ&rMECK+}hqaCO?U5(zHX910dOA-iV3CHqTOSg>)itiH-q zZ%~HGU{5Xoy>=g{(6*}A;mX{!?RcZVr6=dV5pm29hRd}t=3kx<+NVq`a3kP0Pg^q6LY7ak-S#+vm$Ug;o z+rLO&34@g7iLu{@Qvd&)#LBj1&T{f316k@7ebdwKsrxeT4mONQ?peR zs5WXtPs|l3 zuX9!R<$9v^Gr^z$5ne>y@BIcqu2G2aBI-h1T~35tY*)E(@%K|Aj~;TSn1Uy^<-mAnC4=s3)L7j! zLQpIko&lkvY;@|KlFRJN;h)9v=!78vd!yu) zxFPO7tpYkNrx^2FN-`bNu$SQn$IktxaDq1f+xyS*H@|%A^c6h;X!oCGB1=07n9wr* z7n9z0Wm3|tdCpif2Unex$R?(PEh zkm3){S0pf8yo3(yPd$&d|1QSm_n?4>a03BlqobvY^T)fDxvE{OGDVL~nR0ndP1{_z zoBl$4Mg3AO0N9(?s!^53ta2E&qoNG4|=V9YLhWyGL8DqgehqxC`X zOer&q0<2;OB2U@lq8TE9Gkx4PM+7&UY(Lwsc1IS@vYo~7kIVplVacNkpi|Yf8#vep zF{`xxagh>Sz`}-P@Y)@4VIyM1tvIHc&AnQNrCQ!#EOO^rS;P^{`hasAMos=0^l|Z)Z2=-6F zjZugte+oi+yZ^6t;ACG+gZ>R%A3RZCAn+-8g>Y3geCO%XKas#z@w@K|s+gsNc_V+e zj0hO90{T*8wLHF!PS>1yvJ}NRavfj?U!PL=5daM z|M|91x3<45tu?qLFFfV`N*V!iJW;CIb^YhrzLT1KBEBW;2iO-v)To3VGZhuIR27rF z5sLD3Rh=+xV}D?GL6#aXs#l_x1wcI+Y)6PtH_ug;b_aZCeE`PfKP&AaHnJEtCK0>E zjt?j-jWwL#O=i`|2Z38?DT&pvv=q!d`mk zpU4D(Ak@4KxsKIf^y*6ng^_IWU%)d+c4Ml_aE^Mx2~qO_{@YUk#LXvaiNC{cMoA`y z1>ug-*9RG(d&vBKLLMucjAV)eF1pDl((M$V!*Dz4FRV0!Ty5hjkyk zDy>1chz8jAAIUsO|5$O6Cs{TLw(Y>){*uVl!Vp|4geo0WQz&h#L7j8}90F6F#bhSf zelt6;K&&aof526fj_nkVC^_G)P{Bhd2hUutA>Lm+BL!Wh2L_2irZo>dExH?G#Z3DX z=z32Y!S-~{j*A3yZVdTJZ~wqzr9TrWfq=J<{zLtFhsX}88E9j8>7%>{7K)(vjD$fc zl@kCQw2)3!4FBW?{iMkD9zMhtF^wg8KRP$l9p~YPV={LH2m8^>fp|nx*6{JhR^qQ$ zI_8VKDl6QXBE4^eZ4X9HCprJy$3Ubq18>Gg%A@@ow4|V2 z52YZ&G1{vB#Q*vE2u1KFGdUIFtNwYRGk{CTVrhf#Aax3nn7}UEqikB&Mwf z5qJj&xtUdjC;4yM_50M(CBZ2k7|X=|_Z&L__}uAu^!UG*;u3M%f!KnOf3F!Xa(EE5 z>XyuJT6VuSBoveA=gGA3vBwW`M4*0+B#C~>>E-s}*bpKtHmxN72mRo&J}Djon)wN> zHg<4V(8*wT^&2UHQjc!u?SvtQ0a{oh;@}giq1Tt2lFG{)J3Z5IIa*AZYK9Py&2bID z5-b##!`2>@W&Hk}8l+OcDVy(^(mnl03*a47dLE$S1Bf8BtS@l)4fH#W_qX4rp*1EL zDC^QXpY6@PeqAt~3)arHR8S5C$|Z>v*pL001ngUeTpn78vmTep4Y>LV0y(*;tA$Uw zcIEFc?ogKoO$v1|@QcUeXT1NF1HW};<`n~hKLNOwFh9l`uw8%HP);7;qq8uU&lD zpKm?}vBCgYR3M#jmQ_7mW)4+l`3*zj@RvIqK^FkQ78#=w^Ev=J=?G#*^+w9JFXUS7 z1k36fk_SD}B*%RtIbwn5$D5;@a>8ivel%e;uKcjyXb$w7`JFa~((0{77l9TQU{pkD zezGZ#99+*jrJ%NfnYB=!87L;BB_vw4{!3P}VBDQ5sN;ZjIsJc@4)A~hF|@)M;_=^w zN81Bkdy!$5TdwxsrS}dI`v;9H|90nomuL!8K^TpKl_Rw2|GZ$kAOig9iX&FWpI3qz zcp}(P%eVi1tif?Ag9z~M@^i~yJn=i@I-TKAQM$dQbb+JcLb!h^jgR_mmn=2 zO&1(|9}|b@ZAh7b3x#?*UTRLCKnFn-5&?_`B;&4+Nh~B>lw-P7g?orX4j6St*>c@W5Ge(cU3BSx?;ynCLmSNH`8oep0~27GTjGdlWI2VBo>0Jg4O zc3f(ws`VX{MstCItnzjHh?i})sGB9Q0TE04`Mc|5N{2fZuB&rUmP3#039&8%Ymq)X z8z6BY6AbkiQ9KJA%aztq0myS$=8kx>lG1A~e}s8HD1!loFVHVDq)Zmtor>04d`v{- z{uVH{C&gnhpux?Mp}OzMso%)iy8!Uc9a0|Jx>nae3rGyW%@d?g+IS#?2E8OP;LRt# zeF~~8*NyFs6KCB+tw7=Q>yr3{0_GTxk(3;UteAlC6FFx-e%5d)r@EuMQV-p7`1 z5ksID3L3q8!7P~b|Kj(Uk22??uPQC3>>;&cg=bDcny z?34q;eGlQd4XQ!~j(c=U2s?6MJ(=~<8$82^xQ5T9l&j>j&6B2yLL(o}@c=+%wR>BP zNDn@0Jy-=@KqRb@JGV0vzvq>10qA*k{xv?gMu7rUY=NWLRN0fh4R|B~FPU)~fgqW> zzCL@ePuLhUDi|BJlON6H1A3Lf(n&u|$aU@E6LrmH(p&&X4SeQG)N=&dP-is$7%3bQ zfPFmYl~R}!d_WT7D($#tm12{x`N7cYz4qOu+(TATejyDTXE)wFy zzl~(wAEp!d26EGgOgTpoZe!B%Y$+d*=53Fq!^tMWD#`@KjzCasqX8{$V0x0!1(%-( z!CB&!CTM%=)>{%v(WXPIii+9&f?W*Z!08}ifiO1`$jX5o5FRjqAeHbFS&$0NNZ)%8 z0-(`5&&n$$0HKI3DYiJHANMJD_74E%dIewuf!zxH4h4i4is#-JvJ#y_QA&{}kmSx7 zz!1XDuZt!-R=b+Pp{bW({~D)>zieP~2v&OZ87Qoniw!;`$h4x;&O4B0bi7%K;;1SG zeIqKn@xqVGoiFxVcDQ^e8O;nQbl|}GfO4M2c@QKF*cj}QD3o_FY0nUaZ{=`t;E;fb zZx%^N)xQvx?KKP-5Y!*Ky{PEM_XhW(E`1xAa5mldY?b^JA+!lbN*0KDD!JA?LU^`3 zyyr&vm6aU*q<0zsaVn6W^wYYbf2C=B%31B#<2-0v0<0>RC|8$2#YMn=W-Op5Bt~!j zjXD#k1>{o9_z&y4<2P8@N$C`PdKnKg{;<8f3ZTavZ8PCHldRB2SGL#8`152(!D)d~ zo&1#z@yzTc=s0WKF;v$7`#~?r>F1lTJG{ z)CPT>?dp*`JI1uY`r{%p%hXYw!wHufQ-B`DA!+IFIslUxzTPjO_APOJSdf#7DZCzO z7>a!kOvu(bZt`E%ZklGTzo_auh%^yos&zD5bJLfe{8F8~)_tXAHkg5EwF^5Oau;g} zFzw{)u!e1XpQ@xFHt=H{%iVAPRkc^6{~}`}eC7nr`A*u8@gn6dQ&5I?h!zMznbxbW z&HGZXlObmnBkO{EZ3KSQAN}( zBpBY1#xHJe>K`j0?lg(srksFZZRbSLkW34+J8y?mbKz3&mq~DtD^*v(7s;TE=A7U2MTwt^0wLt!o_eJAoTE@1Si15TYrSQ@cqwsPztNs|eZ@q*J8?LjXXn>W z++;HE3#QmIqAOeVAB^yO=f`M7Opmm}v**TH&I-~8sfuleFx_?n;U*F9dB;1ZS;*3uNX$AS(sOa2B% zqErjWtS;?BL{2SL-upGs49%U_n*|q{ThK=zqSYuUF4vn1RJnYt39l)4?GI38g>jX9 zfQx)BwpX{Hqnbu9ik@=E9+w}?A8}x|36bbp$fihhuC~!Mu=Y9Qq*;b~=2~PQzK9%(IKhy!U(5PYbui}*lsK) zkj+p`IM)m?mr zEXpKLq`$igjoBlxiD;C^$>-x8sI$j9wzAN7#1Ut~x%G*9tnQqNKtIU86@8G$a)`A2 z!Q`s4GZ^>=46{hsB0W+9d8_Nx;s{j5IUx)Yq}|Ie7?5A0;ZwbvqcPBPX<}N0w7-uD z=3SNN^W4>m?@50W!7V-i77TvId2Ku|1h;LiGTCXM;{V*+QY-=YI_|W(MddWlZ-)H* zT_v!b09}x1BplN}u^O2`4#ryY-D?(|L|C^XyOfOZ6_J%l3E!vDJ1MdGK`HcFuP-oU zPE8?iPiO@&GNRtDe&`n7nHi$XXEot!=y|wI1-onz9cumh!M44(Jfr|!f;gczdnj3U zh)#S9OvpOQR>ZO{!ys7gg^I34-Fy$DNBC&2#- zfBE^A@bNA2W#ivOiVpk^waD$BE-SZrsE}TOa6Yvhx@$g$%X)U=qkGp%Jah%JvESdS znl$tn!l?7ZVvp*>Pcx*j>Jp-_2^b(n-`V09i{zkH!z^RS(sli^%lJ3Zsm8rC_E?Nr zxi|1QTpH}Ok&Ljzw791fsP;x`q}=;RwFBKP#TYA5m||L>H-k8+3f%MIu?HXU;ktR% z@*vG|4GJQHYpe(0+_{VJO?1RKAOqt^(pCNYZNI_$fVYhKX2KN6Sax|w0e4Tt4>IA^fU3e4ifmID5oY{^}r + +- `paddle.job.dist_train()` will call the Job Server API `/v1/packages` to upload the trainer package and save them on CephFS, and then call `/v1/trainer/job` to submit the PaddlePaddle distributed job. +- `/v1/trainer/job` will start a building job for preparing the runtime Docker image. When the building job is finished, Job Server will submit the PaddlePaddle distributed job to Kubernetes. +- *NOTE*: For the first version, we will not prepare the runtime Docker image, instead, the package is uploaded to Paddle Cloud, and Paddle Cloud will mount the package in a temporary folder into the base Docker image. We will not support custom Python dependencies in the first version as well. + +You can call `paddle.job.dist_train` and provide distributed training configuration as the parameters: +```python +paddle.job.dist_train( + trainer=dist_trainer(), + paddle_job=PaddleJob( + job_name = "paddle-cloud", + entry_point = "python %s"%__file__, + trainer_package = "/example/word2vec", + image = "yancey1989/paddle-job", + trainers = 10, + pservers = 3, + trainer_cpu = 1, + trainer_gpu = 1, + trainer_mem = "10G", + pserver_cpu = 1, + pserver_mem = "2G" + )) +``` + +The parameter `trainer` of `paddle.job.dist_train` is a function and you can implement it as follows: +```python +def dist_trainer(): + def trainer_creator(): + trainer = paddle.v2.trainer.SGD(...) + trainer.train(...) + return trainer_creator +``` + +The pseudo code of `paddle.job.dist_train` is as follows: +```python +def dist_train(trainer, paddle_job): + # if the code is running on cloud, set PADDLE_ON_CLOUD=YES + if os.getenv("RUNNING_ON_CLOUD", "NO") == "NO": + #submit the paddle job + paddle_job.submit() + else: + #start the training + trainer() +``` +### PaddleJob Parameters +parameter | type | explanation + --- | --- | --- +job_name | str | the unique name for the training job +entry_point | str | entry point for startup trainer process +trainer_package | str | trainer package file path which user have the access right +image|str|the [base image](#base-docker-image) for building the [runtime image](#runtime-docker-image) +pservers|int| Parameter Server process count +trainers|int| Trainer process count +pserver_cpu|int| CPU count for each Parameter Server process +pserver_mem|str| memory allocated for each Parameter Server process, a plain integer using one of these suffixes: E, P, T, G, M, K +trainer_cpu|int| CPU count for each Trainer process +trainer_mem|str| memory allocated for each Trainer process, a plain integer using one of these suffixes: E, P, T, G, M, K +trainer_gpu|int| GPU count for each Trainer process, if you only want CPU, do not set this parameter + +### Deploy Parameter Server, Trainer and Master Process + - Deploy PaddlePaddle Parameter Server processes, it's a Kubernetes ReplicaSet. + - Deploy PaddlePaddle Trainer processes, it's a Kubernetes Job. + - Deploy PaddlePaddle Master processes, it's a Kubernetes ReplicaSet. + +## Job Server + +- RESTful API + + Job server provides RESTful HTTP API for receiving the trainer package and displaying + PaddlePaddle job related informations. + - `POST /v1/package` receive the trainer package and save them on CephFS + - `POST /v1/trainer/job` submit a trainer job + - `GET /v1/jobs/` list all jobs + - `GET /v1/jobs/` the status of a job + - `DELETE /v1/jobs/` delete a job + - `GET /v1/version` job server version + +- Build Runtime Docker Image on Kubernetes + + `paddle.job.dist_train` will upload the trainer package to Job Server, save them on the distributed filesystem, and then start up a job for building the runtime Docker image that gets scheduled by Kubernetes to run during training. + + There are some benefits for building runtime Docker image on JobServer: + - On Paddle Cloud, users will run the trainer code in a Jupyter Notebook which is a Kubernetes Pod, if we want to execute `docker build` in the Pod, we should mount the host's `docker.sock` to the Pod, user's code will connect the host's Docker Engine directly, it's not safe. + - Users only need to upload the training package files, does not need to install docker engine, docker registry as dependencies. + - If we want to change another image type, such as RKT, users do not need to care about it. + +- Deploy Parameter Server, Trainer and Master Processes + + `POST /v1/trainer/job` receives the distributed training parameters, and deploy the job as follows: + - Deploy PaddlePaddle Parameter Server processes, it's a Kubernetes ReplicaSet. + - Deploy PaddlePaddle Trainer processes, it's a Kubernetes Job. + - Deploy PaddlePaddle Master processes, it's a Kubernetes ReplicaSet. From 3026a6b74a9d71bf5c617195a6a28765ff641189 Mon Sep 17 00:00:00 2001 From: gongweibao Date: Fri, 12 May 2017 14:52:54 +0800 Subject: [PATCH 63/88] fix by comments --- doc/design/cluster_train/data_dispatch.md | 39 ++++++++++++++++------- 1 file changed, 28 insertions(+), 11 deletions(-) diff --git a/doc/design/cluster_train/data_dispatch.md b/doc/design/cluster_train/data_dispatch.md index beee05787f..39fbe55c26 100644 --- a/doc/design/cluster_train/data_dispatch.md +++ b/doc/design/cluster_train/data_dispatch.md @@ -1,29 +1,32 @@ ## 训练数据的存储和分发 +### 概念解释 + ### 流程介绍 生产环境中的训练数据集通常体积很大，并被存储在诸如Hadoop HDFS，Ceph，AWS S3之类的分布式存储之上。这些分布式存储服务通常会把数据切割成多个分片分布式的存储在多个节点之上。这样就可以在云端执行多种数据类计算任务，包括： * 数据预处理任务 * Paddle训练任务 * 在线模型预测服务 +

+ +
- - -在上图中显示了在一个实际生产环境中的应用（人脸识别）的数据流图。生产环境的日志数据会通过实时流的方式（Kafka）和离线数据的方式（HDFS）存储，并在集群中运行多个分布式数据处理任务，比如流式数据处理（online data process），离线批处理（offline data process）完成数据的预处理，提供给paddle作为训练数据。用于也可以上传labeled data到分布式存储补充训练数据。在paddle之上运行的深度学习训练输出的模型会提供给在线人脸识别的应用使用。 - -### 训练数据的存储 -We select CephFS to store our data. +在上图中显示了在一个实际生产环境中的应用（人脸识别）的数据流图。生产环境的日志数据会通过实时流的方式（Kafka）和离线数据的方式（HDFS）存储，并在集群中运行多个分布式数据处理任务，比如流式数据处理（online data process），离线批处理（offline data process）完成数据的预处理，提供给paddle作为训练数据。用户也可以上传labeled data到分布式存储补充训练数据。在paddle之上运行的深度学习训练输出的模型会提供给在线人脸识别的应用使用。 -From the perspective of user program running in a Pod, it is mounted locally, as +### 训练数据存储 +我们选择[CephFS](http://docs.ceph.com/docs/master/cephfs/)作为存储系统。 -1. the home directory should have been mapped to the Pod-local directory `/home`, and -1. some shared directories, e.g., the pre-downloaded `paddle.v2.dataset` data, should have been mapped to the Pod-local directory `/common`. +- 无论是从[PFSClient](../file_manager/README.md)的角度，还是从[Pod](https://kubernetes.io/docs/concepts/workloads/pods/pod/)中运行任务的角度，统一用`/pfs/$DATACENTER/home/$USER`来访问用户自己的数据。 +- `/pfs/$DATACENTER/Common`下存放公共数据集合 -and from the perspective of our client tool `paddle`, it has to refer to files in the distributed filesystem in a special format, just like `/pfs/$DATACENTER/home/$USER/cifa/...`. +
+ +
### 文件预处理 -在数据集可以被训练之前，文件需要预先被转换成PaddlePaddle集群内部的存储格式[RecordIO](https://github.com/PaddlePaddle/Paddle/issues/1947)。我们提供两个转换方式： +在开始训练之前, 数据集需要预先被转换成PaddlePaddle分布式训练使用的存储格[RecordIO](https://github.com/PaddlePaddle/Paddle/issues/1947)。我们提供两个转换方式： - 提供给用户本地转换的库，用户可以编写程序完成转换。 - 用户可以上传自己的数据集，在集群运行MapReduce job完成转换。 @@ -136,5 +139,19 @@ userkey=wuyi-key.pem endpoint=datacenter2.paddlepaddle.org ``` ## TODO +### 文件访问的权限 +控制用户权限 + +- `Common`数据集合只读不能写 + - 现在mount到本地以后读写权限 +- 用户可以把自己的数据分享给别人 + +### 文件访问方式 +不用mount的方式来访问数据，而是直接用API的接口远程访问 + +``` +f = open('/pfs/datacenter/home/user/test1.dat') +``` + ### 支持用户自定义的数据预处理job From a35981756e18ce7f4306b2dc5ffb52908eb78eb9 Mon Sep 17 00:00:00 2001 From: gongweibao Date: Fri, 12 May 2017 14:57:34 +0800 Subject: [PATCH 64/88] add pics --- .../cluster_train/src/file_storage.graffle | Bin 0 -> 3031 bytes doc/design/cluster_train/src/file_storage.png | Bin 0 -> 42615 bytes 2 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 doc/design/cluster_train/src/file_storage.graffle create mode 100644 doc/design/cluster_train/src/file_storage.png diff --git a/doc/design/cluster_train/src/file_storage.graffle b/doc/design/cluster_train/src/file_storage.graffle new file mode 100644 index 0000000000000000000000000000000000000000..95ad2758ccae252dd322c497e7b167135cf478a8 GIT binary patch literal 3031 zcmV;|3n=s-iwFP!000030PS4ca^tuWeO`YBqpzusEU2 z_GyKZ8JN4YnTUHr9D3KZjNowRyMr6NIMj{6y2U4GfxVV%;=6+~yGsi#H^5`;9fE`M z@^o2+wrv?Gpe!#U&Z|nu^AJ1v?!XT`+T}wkVTm}dHTKACS~{g?O%IJmHdYSG4a zsK_OrO3y2Trin3%gYuGdB*82oy7|k@4>ICHDI;$M88@tb*Q9MvCzi3Q^q`0bGDhJq z%T_Xd{=5%mDC}|jnylRKai2w1(Dpc91MvPn(FsJr&U{5Sc=Za zdl8X!gN2RPluOy8LdufdtV?6XQF+Zn^R--S$=auJaj2eGDp{VbPa<9V_=*r`12d^2 z_ZIo5o;9}C2C`Ht*yBK4d$Af_>l4n@kxVBc~i`tL=l!0O%X*+6Cj{|F79yx2we*Tdd|y9zshs0r#)+_mY|34f^5A( zy_-XAlX;f3gZCm+&FdOpqB@(~I&>MX{IRomO(nB2qFWOV@UL?FA!vja=Ufe~jr^6Xa`y z7zD8r5s1-+KO53(K?FOWuSe8Gf}wpq0@owgqDNzbJRb*uJHqxY4(Ke}#xwlmJ*72> zO*~5cy*^8QL>UkCf`##G?QW6&~zdPmyPaKg(O!9}RXLSmQ~6-4QZq z+yOf~VdFk8bJxZQa8*2=G`idc3xoK+AofR!xTH>%^@gOLWAnwP?$2vtNl--^0H7?$ zs-Q@FoTx}8dS6jgSbrBpAf`#3HaIKtbfrDAxvZ z$KC3F2ajhq@-`1ZVblrkBpJ|O2>I|JtL(t@Vt7yiI|xEZB4{CrD25~w2uWydc93C6 zBD2`V_*$Nef|-ZxjUrijI`Yfk*YuuVt+#viPI*EcT>ie_t#|$>agFwAkCRlP6vi%D z-2$0R=z`i7$ZUbk%K(|PMz?BPG=X_VXD@J7HrWXXjID00SSrO4Fq7yBRoi2og7 zWIh`ZD34Fi`97`Y`1Bc}x8j;xWr;C6lvuzNWO$$I@UmPGrIMhi8ju+d6a~2ictxNX zL@KHw3qlE~q6|d}E+NH{JdMbjlD3ewFke6FVB@HJUP6Ese~@JV00r?F8H)t@2-!X^ zCy1yCr$Ac4s>wQ>Aj{rVlE-2eW%FFaREzBLH%!x#CQzeLL^esQpaf-&reaVt^_^ZL5!Or;py;%s2cOKNV|#E30sc_2uUbb$7tbr_t!{l%IX>-!2B78BfqMbM z3Rjn%9=kDg41Ll%Z}sf<*}$3F1L0hsT)nG5+zV;K3Y4&7x6Ui?=Kd{eYQa@ky}7t( zH~IwCb;mx|&u85UZLL<9dSQi5)BeS5+P6cevB2lm{zPv~b?IH*o(_&E=h~3JyQQ8O zR*L!nq~1BN`9qQ@2D2LJuRQ#*0ubE> z;Z5*0MMR$a5&&Nu6*ZRX@=AyWgfIhs!>efW_GP^-komHg^_B}!Q)2#)mZ8APA9(@p zaS&>+{Cbb{O}yPBh7Wn8^*bmr#qrWBD=zm@7~_*=UtQRiYho|U`D@X1YQJjJv&=Z4 zO>R78rKuf>OGLW{v?1kt^b7GAd6l_s(OyuL)fgzjtg?nVbXltLMpWKLR6gxx&UpIX zcv#@+iC$-vlxsW54Xe$WWn%w?7&q8VZ;niiX^i^9@~vTZ3ubbnjn-Ap&=~uf&6C-^ zlP!l5R@r(y_BPn2`VQl1k0dvOt`=RQ1sAqs+kbK!rcQ%9ohlMu9$USMSW)FXQrt#P z&Y1oBw}1Wm&p&fHW_3m{<1=00&Eo7XEIiLEE%F`OQ8EjPy+?N~C$>YkX!2mChUFRL zz2(+o8(5c~rlwpir0rrlwV}kbzU7lXkfOIkG3)L){+Mwoi%EivFfQ8|%$ChgusaSW znFe?e{qdR#c!9#>RD)stX9yU~$<*42KIuSI#G4|I$U4w_Gj^nn^6LtObs5}e}_Z(;a5ECG7W z@$Zr8Yp_oS`*0vmUCcs90r2QxWS+YA;^~;uA@fZ?%Rxyu`X)y>QS_`J8~*NbmbTuDgNdJ*M`H ze59G3Bp#-R3u2k?@FJ;BnAGMa&7#P>IORjfX1lhs80Gdl(OAS1az_6+;EK_Tb)c#fNVm%}~p ZRy-tS!pHg;esbz?=fBA`h-V2)000rd;<*3- literal 0 HcmV?d00001 diff --git a/doc/design/cluster_train/src/file_storage.png b/doc/design/cluster_train/src/file_storage.png new file mode 100644 index 0000000000000000000000000000000000000000..b744bed48551b4361171bf4502e24cdff69a68fb GIT binary patch literal 42615 zcmeFYV~{Azw(r}TJ!jjtZQHhO+jjSC+qP}nwry*+-FvgFS006wkOh78GNC8v5|H2TkV;Jbx^_`!!a4>xXre;YJ%O->z*YTwk=>Sp~{#`Wg*i~aVO*L9c2 z^b0`Fh6w}+Sq&mU>ARy|O)~}qJ$YIv4iGFa09*`U{G=E{3+Rs*V8y$A9j4XjDPgB& z)1mY2udnL3qoG1O0AGLEsKX*Vc{%RzA)+YW6lef{&>7Wq>Xcpigj4=C1_oBJ5}3fKIpzCaJR(~224yQT=kSkxRsAT5JIK^y*DMr^ECvJtf+My7lrF=p`8a`hQJfD91sx}F=)*yAPgJcunh%C zIw&w;02oj0Y>;^S_&i$jLH6-W`@XK{z9E-tjtr6m&HTQC__4>>$_| z(TZt&c4pP-y@I{s(_d*Fr1!83()|LNrimjX1e zXQYb3>}RE?vH}AgsL})W2b~pRD$u2;(DsKXkoE8Cp5|R{S6J^KGPnSE1dxJIbo`cZ zE`+e5!h3Ldbzq|+;xXU_*k*j}G3_I`x9Besy?Umo#6!RKeQ$by>#?RVrAVdlOQIE_ z&xe=|Zwx}}4bg4aDptTR0a!#E=@Ha|tu(H1T7or!dq8_Ytp{WEO7GdSf=d513ApPO z+KsbSYvJGE(*&+US`NS%lG?d({m}uq1&0%2vOj<0_WbdT@I3yk{r>9{!B0|_CAXyJjv0mcE|tw|W9Q1+g*nlzl8vuu=tIvEVKY_6L7?0K z>*R@vXRJb!Lt=RRI<*STGR2decWg(vXNG5&XMkt+JIaqD|6=~SKI%Z?K$0+2A+N#0 zFbgCYGHnIV&XBGVJvbA&&%F6?Z1P_5SMgqPHSt98OmT!U^0BQkpu>+bl|-F#T18F8 zIK{^jO?fnh!cz9qTg9gmZDp^~1fw>3g_Obx=cAUixU}dr(aRF``3NFJ1M0}fJvw{r!<(x5cMYYQ1!P4_6BbAb5jvhn|bRw?8Vq+y;H-p zg;UO@!ZY^ss@ck!&f?(t`bCK)j*{g{m~)L&lC$LVZ8~453Mg05pHS|QwL#87p8eqc zfCOQ&<#IuCnX@r;{VUt7!!4uFk)cIm4WrcY(DBzr%0*;F!;(0uqbVu4r{NR#9}=IKi5i!2rx7P(HH4=IsFQW2AY>3|w- zRxwt&mrxuPndzIx?GVn{mz9_Emon>Lr@99*7qaKHZK@`A8+!)f!B~q}t5G6YKQmLL z>7+$6Krw-cGEqyJkBVfp7jijy#>* zCfq>YQQmo;6<#IoC>=9hSsi8`Zk`1^+}y1mGW5tB+o>{D*y&(wM|NF$BX+}f(|4PE zzJBt5B>}MZu;{w<>=UU=JWHej@!?X~y6nW`X67*QbyE*gL8+^E-MiJO{R!HMf>j?W zi&={qm2IBMn-S4#)61)AvK{?%xXZMQL%@k2MgU46h0h|q6(t|BC&VPOpWmNfo=1}8 zM!nu{Y1q1W+j00o;iz1pVdqe|HB?ZKzU1qOVr_Q-7H4 zD#I&7MHA%;`${##K+|p`+;#YQ?|Bwv`1dB5LFiN{QE0W>RR{gnI$?G}zZgzDaC|h0 zOtD}AV*$Lpf&BEH(evp0<05(aa^)2q>aSan0E>&|3oQofdj#b|J~N8BwE0uh`gyWB zqz_#5G21cH>E{M!&&yP&x%209_T}^H?XFLuB9)_QwG>Ke<#o_TC=F&vE5}V&M}!74 zXN7g+$jR1pk-ETt%*M9=O+3bG78;L(3;hJiRI3RfngWc$O8(l#vv@ zIHq{GRk1WTwG9*$6v`Df6n`#h)o! zwXPx6TxrANtnQTcHhI?UptYnG>0)`s=(6tAd{24Rx^TYXaac9HdH$sOWc@^SdVUIX z-py0vsr!ff>GLEY;inCbF=vK*!UyUb>sRPPd|a`(42AddTc_TP`=7m-VcFyQ7?bEA zw*Kl*(YO6`(JP0uyEv3G8gt=E5t<0MNT+ViNBe`soyCRoj&8B9%N~qP7LzZ>mwE?I zd(VT)KbQCIeNW)m(LOcrGt6*|H;(j9_Gbp4B!M!E*;YI*o}tgT`sSA^@5^&G?%V2J zZ(cr}ne1Ftx)7`NmETzxCJsywM}tg`Fd*+as4pSiGQl)>+WEN&*gHwqdA^Ae6t58l zK<+BQbN;@!c{<7vf%yNfJ$`fAaR5kR-fC2cVgH)c-&6Ai0R;slM+DB#A|?6{1KdjhDU!Y#HOi9y zhyJ0Wf|STYM!4rC{D)l5(0uFKN%A6p?qBQu?^Pq1at0yQO%LQv7zDJ$2o#$tk_Jg@ z)ffEJb;jn`LO7ul%4;Mm;sFC}4*w0>4juWwO~e-m1PJIOWPtP|@q6~4>I@0t|2@7h z$m?%jhA88z{ye8t)*2q9aD3`sB6_aVxDX*xE?b~UlIrUcN4Su=V;&ETW!equZD0vkPuney>}%XN|g zVW?sFkCm{20yI|>ANmV&JiDa^{f-9VIrHJ0=2hqf@7E9l#E%AjQsjTudk^IYpt+0A zUQ>jacUmZExtSKauM}7l%DZ|a#;Oj1x%u#0nYe=N#gx4RMWfj$ZxrFEomG>HU1XIe zb4*YXoIEnQ<{`Aw_0$!rPM^w4f$?aK;M-vLzm$o^|8WN3OkTvUyAw^2mT)>_>Lnx6 z#J;HfV(EPun$<&gWEn2(OE3gny`~j}12@1e?`q`huC!tUl8X( z!5$LcK>eL%Jc)nXPd8<7&B?4Gtx_)<)ao-}fVA>alW27PMfPpowHyR~Q$(;_CD7q* zVK_NWI=MIGR~;&BLCJnQ8mQ@@oaH}`b|MG#qk8{3lFFgP?YiN(_eUjE1_DKBQdMr- z*nQ9G+w2SIJVR?m_0l6M5>rL^Li0IO8r>YLzvm>+!S00`(-<_;_Y(9SA9{py zHJ137jb?a$*_w@g%E2V$@xYY~UkUZso;d>1CjsAJWaPt(^IPyEJAb+o4YhNNEoX81 zUeh${C(AQxP9-IT-;*GRpEiz-w7TnD@cpZ=wE}c$$+5-`2_`mzVG9Fnc8LW_+X<@ zYxWdpd#@dBEG!gcf1T@m@rw)>CToX9MfLRXmh6X7RP`Jt>r}x!Imybx`UeEq9VV(Y zu?9h5fHs2QVEuD{N0RS3ZHmR4L74&_Ph|=?IWcj+p5!ARr=_7UEG_NrNAm=M8*uyF zj*_T{%4R+eh?u!;A|(UZ3dN1*0L#}bIRPo1$dG9nrbMf5JJ=v=Zbe3Z!LH&g((`IB*id4d9o zx2oX4oP0klZVtc)JBbgRd<;&ImfFpoQi-GjV4$z}W1gGl?hmNL!^7wARfmm@iD$4m z6b;_={>0&Ry4}n?T{2k&WO3=5d@^U$&GCFuAs`bcv&S($EUBu3HkrPmCCRi^MjBu1un z-Tl>dbjt1Xyc?`8r!bbLUn$I+pr4BvQL0e97ldK`2h(6U>XOPX6$VTb5m1MnN-K~p zt?d_wu;tKJ%OB}G>k8Je`^)LQ$mpV-Qj;YD%98n=5}sC!W~xg5t(yl^){Ib+<_ke* zKj5YTO3u_4cm;mnp|F0@?K_SU*%Cr$K4I$Wo`H~byQfVDx~|N78SIgPtoWKIG&K<{Q{G{_fDM&6cl%A9ga1> zOz6;O!*TlhZ*&BnQHj3y-^=aLfNQ3vF#$s?JBlt87MSahug|BUeUEdL{tWkr+K&-e zFT^D(=z%G$YCr-F2{NM3XCUL%q`iRTY!Rh%OvAO)gm#?Xx)9FxPG9Vsy^IwNG{ z?d}{X5)-dafO2Zgq2i~iUVpMnUrT{p6CBsI6@qvcrDJ6;K9xsMM=qJ1vNCZBWOG7MG8ge$2OM+qr3f?zSZwU5 zkdR-hl*nG#+@7w}5JQ-vIS4>qj_T}C^R7VQ{!}4$3wYF;?$20l!WSBd5r@TAqH3KW z<$K8blEHe3A`H-jKk$#%_!C49^y-+noT$i)aGOzJfVQ13JqKj)CE`d>8wy~ zKi}I&rl@0Rkqwc$!x9;!T1S3~Xzn;(z;A>9Qejtjhq6=!hRH9<_`pDPF-F^y|W>BP`LM-J&d`c5E3v_iKL^*R0wf}Jvv+fO| z&iS}z_OV<8g~l1?i0#72b&i8}=4X~9fEScWSl@UTJ}wp`JN!vA$bfV2$oMz|v(=j6 zhSRcEkR%hmARp~j`!)AI&*wf*1d6t6Zwz!fUgn~=1}OLuA#M}t(0O&Y_4~5VM=~4+ zhQ&lG!!WFXoq#ZXgwc2s+Ue=((6!H(hk$T;NCb#`qVN0HC4Ki)pW+cwh@DWJ+@cXV zl&~^lL`g&(wZIvGlTeV5;W$+_6Fntuj4Ej~R?uXckz&%3XcldqI7S4!NRXxo8uXn- zP0>mSl8+olg#7$n**hT?^AfRl5@Y@lrC5A_jX0)7Q2+Y^?O9mnw1E}7j7K3EU;c)u z;)`mW`hmd$;wfv$BvV=hJApuQyHZ=0=2-+~d7KiKoNyOd0Re&d+*Ax=M^$M-<}EWK zh8#Rk|LhKwY1dg#xBQ)NtnvlXZsjyPfmBOz>)gfo7Xx-`fmDfo5!+eB9Ar8-#~fuJ z)oe+^EjxXMy%kf~WY0lsdt}M6)1X8z z!`ruB6E#_``V&bc>8>@HWmGNrl+_4`CG&KAHgaMQuZA!M6Okd zduc^RNeqv=$0~Nw4hw-8C;`gC@?gPX!0hZZh}62t{U*5xn|zbAB3E3kcInX@?cs{2 z4w%3P^3TkA(7#5cKLkubMjpfL!G~$u&&gjpl#32HO)t290p@%# zgx*mN@2C^DDF`C=(sJo_bM{a~yhyBGX6AgjM}A47A-hu9&dsd4G4%61O{$(#1KXl` zAER(&&Ki^QmpWh$Ka5y>ybfeXLlqO{rv#_{n%$HgVYjEIo6YN}>Y=|WQgGoD7N4|! zmuen!aj(ByDn?&!%+U%ur2-T&5MWgv70D8qJcB)-~qBB805AM4gKL71RMd~E7rpunIghMo=t z*Y&D|h{E$*H(9kPBvU61Fi5qAm&>rQV#(4_G^0qobK&6n}}&QBiNC`>l}5WYqh-0C0@Kssm^ZO~yk7*oJb9nlw+ZKEQrKr`~`#Oh8t;n>gH@~#Yy?z&?{_K3JV z&0`gJiLeN5MpUA}R^bZIo2G#xG`Bz>T5n>VOFb#NFbRpsJ!~2Q6nvet!!3O!oeKmgk;JF-x|ef@aQ~LBF^{e;m7`Ne`Ggw!x|p0f3Hf|%YLeZ> zUqXuNwQr#r7WXRdga@Z4Lh8`hCBXy|k?JETcGav!Emso9C^n+4cXxLWCH3fTQ{(j* zfzV8tyRZ0JO8N^eU0Eb+<4y;U!lY|d)e?^@4Vt3@g_oQcZh^vy=(2V%B!|i9-+7?0 zlv1Lc(iJVe{^GJ%c@K6KJ=%KR3=z9P1M;}KFp zjMOwu0)BQH)LJ$n;YhQdkz_b_&=4x9?GCa+=5^svDJ74a6y!E<{S3o zR72C=j|>9JNVoCA@p5t5281gWK@*cKc@@kAM8!3Swq4m|ED6ULVxee36O^Xs=wp!; z34{bhM1_N~2`PpLt@sNG2N+~vwZaJ!Re~?UDqGU(P=|qWGHH6~ZZ3z*x(6nNXku!C zw=xw3Bzkl5oNmsD=D6G_`|bo}WTVf0C|$bm-Cs{%j3#QC=D5g#^iK-^+Rwoi67aF+ zRyb3a_AdpEQ;yOZS?e50C`7tpHWacj$r~Kd;~i=5!V2TpXLJ)d!Xo4BO-Sl;1qbWV zx6nAJTr~40OI1nTC~MW+)0En=EC<_q*#bu^Wm!4Wncap@u$yAD3)ttGZLlv%5v2;n z_55QMPcCei5IGWy67Y`^bM?6fhst$JEZ5mxjy0TWG_ac>3p-j6!ISX~WMuU+#Kl9! zyB8CRoyRHa+Brt4?qY7q3$R|t5vw!gqeqTjEvb^h72^sI#2BVpiZX~&k4%^n*PUJv zZHgn138uNcByL@#mFznQwdThaRh!{u6;CfRmKPIH#l&wM$vmix-!`ImIjr&$WQe!RBnd&NY5PK+KOPBahc~dU_2RL~{kr<}bE<4z>wn zu6eQ)P3uJUhYYllFze1nxLc}n!{q19+a2l0TOf6vUaYSqbRgg~Vv@N8nX`v;VKD3U z=OVZ(ku}yOuuFG3fM<`)xL?*vMx=4$U_b+hFIn}>X?$04F5xA!GS~FY1H>tDYm{%* z!Y4fMWSdUOe~+YQo+qg_2mnTl-ZWS5TMW}McCFM4C{nvmNbAm4 z=$<523lPi5B8T^{K1RYDwa283RiM*M%FgO=rWwP?xKrL5(IhfP+vaQ0)+|ALc$oYZ zt{qgVCKG<@0CaoDu(Q$#9&Ma;qWbwNZe_Y&vaN;WO0Au`X-r_Rm?K#UHuNrKekRYH z<&nSNe30I~QaZoQXycz1#Xv5PrUQ(TH5r^n$)tbJbo<1*=JBA2DL#mwdaHydB(IQ` zHekS=Dx(QN~)T47V;$k$1k>KB}*?z610c zFkc-5KYB=2%V+Au{RQlaG5?6TBq2i@%yc*dEdM@XdN;pFKq)f+8Ec^_sk9X?5^%p( z_t4$admwC}L;s*qHMYZ$9Xivl3CpGIsNM};{v(w>qXgiHI;y{^GdNHMxJ>*uR=3u` z?EuDhF8Fm$VL|>@ZW))-;c&s0ly87OA@;hmP z-&Z49wnDf(9(SgFJ{bmjG_J93pSr)ie3QfWo{;>YFqn7Bbw0|bK9ACEf+FJ1I?X*3 zi`#6Y9f#v>eZmtnFIeTSE@hc~p0l)Hj4gAPmwHpZxwaB*e!_6=pUos9;~Dii4^VD~ zue}i|2!E&(3Zno43-vPkkJ*!?@8pNff49)JtRAr{+CVpI&BUWisUu}rkUsmJAVv3TG6oXV56itq9^Q|GXdgTf=~ z)|yMOLsloa;P`P1e_<>L8t^Ttz@d=yfz_)EW3#!zLzE-P&nn;7ivn?W`#woZ)jO=r zoPC>zmdEf}dn=JdG0H6p3X zxJeTEStin3KUZJl1zgXcZl~`}xq98H(*QTTAsmcFdl#}Hf?^I6+t`LyAX!sb9MoxwUQ z5=Xo4C&}AL&;a9oK=OA16-RU%s?c1ZzT6)g>-ygeKFY^Z8imC zE#%!~Ic(7ZU3mhk|5+@a)@Y27E!bf7Ox5kUg5M8a*z~Xr!Xa~foZFs+I5L?vzUai^ zKER@2|AKd^7vWdRz0nE+<%qUXZ`*_bDM$X87f`C5i@uegW4yujZA1lf2KCUa!LY5O zp(D1)LD$5)`@JZw@zs7@)$vj>1QU>L+cOg;$C0$olt|Q$YO4EuyH>e9XqoDdXOGEY ztfgX(q8Cg=XBv4y4^@m-^q&QgCSy^$`*Sb+gFz{1Te0AGJxyby`;yr6*WRkwHhpK4 zKV{`K!o%|4Xx*Za&`5a92^2=0SRmdkIosS=wB5#|p1$ls-r}(1pOps$f8*Z=r3(gv z=<6@-#y|=zEGh3F>~7W!hMamX;A_%M+79($g4JThqgV~?BH(*Gpk;8@!9)*@e5I8X zw)8G4(iC+O^2s@q>@bhM@07yaQ++|w0daBUhn2?r#`78&CPA?FG*uxv31l2b&lYH#-O>g2{VZ_PBD?FZWRb?2#8;Im@+M^4pP+>pSYE~L)ThN z*+~fSHs|K*DI&A?k68?cNQpCCya)-Gj;7Q_=2Q^@8aQY8Lf(1wH#}1(F$Scwp3|3WGG=v_YX0qmQr?Tjx!V;B4G@^NJEovGJ3=g zhb+YHWh`H0kgO-EVYI&e;j^}`vU|I1e#!-zaL0v}9s0cPe9F36(y%cL*u1!~G`QIt z6qc2ZF~M$o#cGNEE}}~6jAya5xt_JIvPHV+A-bzzJfQBa=`YS@UNuvK|`MUBkZuc{Vv@s=0L2|2%bE{KC3cWQ$#Dls>6g_6M3GRWN| z@mw;7u`Cx?(bOu~cBy32i8~Zz$WgnMg+&S)sa{X1EM}KiZ8Fnh8#t+aN^78tm6s|e zPg2Rad8c_Vx6juRAEBEx^!ki57ZsFHj?Fn8L6=+N*f-j>D-59rYhiqOd=zr6$U_E~ zRw^SKqvhqh8zoyH6$z>|Wj`ERqsZ$NRb=otzLp!_4ZfgHB{!fJnSebj=W|g2E8qZu zEQDd44dOa%wYN4@`3Pnu%B{(4S*fO^9INjNC>N2&F?A%y$=)Z;kJJzdh#&LNpgI>z zN5kdKi!QW2wkRa&)6pw|3z1AcQ%h5TyKLA6n%B+z)@ga>U}v!tb`JJGE>TBe~J8l@*` zS-FU)T>r%DRx2EE{LJkalRdhZP))9_;w+0t3h8j6B4aV_-xkc@NdB!MeEGsbR&Qxd zCL$Gl`_uvJeKaHO^_nJs{(8rxy>44)5Nr+sRW)|0SVT-96R~Wtt^ltER_l7pyX*e|kNj`5Bs#41uQ3Y^VygCRw#tzF zumTcrxgkY_3MT#BuC4&HpJaNVj_J%5Xfm>q)Xl0IlBas`?9aNL>xZQZ`IQ2!b+i=F z(Pi(=YxSOud7A=1b|0|$A5A~eml-JH{pp*!y@DU3Y{jze56Uh;T3Ly)daf&~Dy+q^ zCnvaCq92sBqT@^Qh+WTMFFQ)eteeOfDv!$sOAq}Yqt>1P!cR#E- zuWFyVE(jQ&6c!$GjX1}?VUHmYwZxe?6ZzTu>^g(+3!R~K@++E8#}v6{8~)ED>uY5Q zq9rYytVDmAQdXAR5t`@AxXx8l-lS#|T9Ye8MLY@tg%873ry1s%^arxcl-y8^az%?$ z*UoRqc~HmkYng!o6qdcA9hg56RPY$!bAMEV&%A-hV(c|fkV?reEnFSeUiYcxZvsi}F9PR@+0JM}G|sV62TQbFI1lt%9jK2Bk6H(^ za2;xG3zx%B*K$_Boh(c>-r{&FKSDk7eC5(zt|IUDr}wAbAkvg{r_PUGU?R#&aMrGj z$mK`lCZ5gp#CfDv@fJ|Gbgu7<_#NGj&__~P%@W^bs@uA#BtNL0 zi^1p-+X3~|xqBw6fG`y7W(2oeIws>6%;|>8feihM?8+jMh4HbIt3z-BTOWJSz2gA` z^QN;^K|HciP&G9b4ja~Euk-G$=yEIuhcebmgU5@`H3$C6vxKxXaHhFJWw$hr=EE^< zE{a|N$e->5i{(7WC@82rGHD(zC^xB0bwcWDe^>hX zl>J`~8yXphkQLU?}3m`T4vb5*B7YD?MRV-hPYye7Zou;Lf_2AEwJWdlM437Tn8K zmve!-VCttKW}7`Z7SM(aeC!uJtO(SEg8-==$IFA?vu(^%7C|kT%Rl=A*WCrjFG?HI zJgbrNqm(0(h=?eLh}Uje%jL8>Guw!}zB+ChM`HNT$QPg7rrm_K&d(brvY>NNpkkTq zpo)#(+wVs#?uh)CB~=Mh4@9l`)mvDNmDaE(03@>!o3nl^^$ULeb-H{a(wB&pj|Q@1 z0d;C8rGW_SW8FNQCU2)Ojl5{0IaGk3%-ap*@|b@`+_bf5<~#jCY8@Whr{v?HRyUR3^j3{}7-3F1TD@ zX2TRIw%9F?E1ktzx*lkbgGGkWIb94FEWfOG_esFoVUur`KOo>9ykYRhat;N%$kQiu z?wl{FJVD{QhtY-`1{K{ zt=--LUa4dQQ6I_Q&KI=xT`gweH(@>EP^+Xo72NbPeK5I)t zCVQc{3ZclOiuiODDvxAjXc9L!=Nz#aRLHl~$tg*JcEvhC7ZgFz8cbS;0DAf{;L$Vxanu=fX%g=G}j(}U9XEp}>^N_m1#c4Jj#H3J?PfNL~v zRf0aoh>DDKH(Hl#ofVok2K#1XLVJ=~_TN^2)T|5T*Qn~XS3x7-i1sfTVMQyR3~q{haOqhfqh|pWy!QEHfTHEpa%&HdTJ>SR~xenrW3J)ZDD87 z2F`bIg3)QgK+(556E_Fma`Z`-!52|c6N3hsWBgj6XVAlL$-vf#f%vTO;exrU`*Ngt zF&eKFl+IKEo<}prk|C>v@{vKG$v)DE`^gS}2M~+ZOu29j;<4F$ zk(q%<;Rcye^mUb>-WLPgsBYmzlzpX^XMX-K1C_}M3l$~{pm)|p191(ya zR@J=EZ*@}&q8B-|&_8W8gWCdeMjXsWIKA`X-DeLcXW~hKUBiWvB{};9Hi2f53~qZH z4X`K~7>Veh<$@ailpyE!>%7;zUofI^z`yLl|K~n(Zy?JgJ12d*!z6GP1-nH z-5yY(+}wIQM&T5!T4~Q9?2M{p@|4?ERW4vPvi!=<%IeB1*(^UFg^rGXFRX2j1IScn zz+HY`Z!&HEL-JDiVk=Qq>+2i#=~<~Y-~vz4z-xb4(rD~$kjGExTf)kRsyw&Z*;qCs zGVnjGM5ndW4FZi{H0y_-66a0}5paDGsC1;oB(4SQngrk^Q0B3fej7gO_B%V`En z`FwbAUYdvoaS!%AJ1DC@JUf|KTv*_B%F=KtTI@`#n{T_TWIr%B6m)6|@b_=v&r5)R z%1yv*)Q?&-5E#|vF8^arsvJ=r_+ADSmK)|yf8bvvhu5uB|tu9;`r#uI9l^~ zI(zw+3@wp9*HFPeHIbk>uD}^eIe2eDy%e3vT+!s@4F(Z0r$GKu#N~42 z3UV<5A}<^OWi9l;ec=yG#?{L5et!JT#?9Yom+$wazu~H_-3d)l0Ki2v<>0g|2fC?` zLmj8J$>6h|@~E!X`s=HA7A7V>r}xbk zbmqP#^TAaI_d(L9gVMTpl-)mc*-O3b^LFaNhg|=~RXu-WVSzjgsKFqTAc`bfPD_iq zMdGtim_XvW3eGG(uk+dWxsV@U&%-?Ykr$*KnNf&g(nIjpDuV@;mC*#>xcF-{)%@673^I2hX2GV41@^GVXl>TM=(k#6*bIt%~y4keg$WX}n{VwF=D~}x(9{wQu zTixzVLgzIO+e;SwH+tYn_+oL|dAXbtjWCj5FDeJ|0>j5BWNz0hq+-mD+&&z-;p488 z3xesUcN>RS(RVtugY&&Bf9x-z)P8fVKOXSXNOT|t8umO|R>qKpR^ttgjnA-}7OB_G z(dYjvhNoM%fiaj&EtLC|#{M+Y;THj_1pBkOkoN0ZEXU_t-|KxA0%+uI$Kq)cP=mfh zY8zH|{HrW;J{p@VZ2KEzT07zcE6$osV}huyt#v=pGo{g6wkQwj=!K?Xp`LB zku3MQ%(#lq(B{Bgg}mP^NwHRdSyN@z$XquAP*($vPM>3vNb?I*wxIbf8RzFfpR<)s z8x;<*AP|>m&%eMncUeW^O9H@e;fF&Ehy?(1(dYB|Y3t(y;D* zyj*VtWV*y|%jQtH@CFi5K+zixnOwm@*)rRUL=uVZ`cUh*nMLG$m~d}c*{{h=c_4Aw zZz@z8;poDK3TC)DNXi_cw%rjFK+clX3?-}EQ#BaW`vw`X21G^WWj(ev{X*eb_OIHH z^*uR2KS*E<^{4H2a(}b_{*i=tH~c0`WT&Bls@3v`ZfMFK#15h;r}l!@`<&W4!C7PwT+7>xQob~94Wf8s2J>H9r8+6*b#svAjXd2xlD>81^7dOSmDL!SF zbo(a)Rpi;YV31CK+5e0x2W5Pk2CDN{JC1Q- zQ1c>vW%tLW&CQxcb#6@!Y=1D!@J6e{gWbASGae=71kLDb(|g~LDD}6&vZebOvwHG$ zR}BW_f?I4Gi)=-z;NFmb7`&Ka$Sc$ImbC<)cpj%2V}?NpPBaFRMvwFxTF_g9F&o-U zI~(hY2~6HF{bEgu+Ksh+!Atbx=13GfHwL6j({ayB2}uDc$g7Q6jpGj zh1NRk5eZSq*J1ZGqKw7BcZ)r$)b?n5BSMfc;r*57LlADRm`SfL>a1QLz6(+~TX+6^ zYv6%RSMZ-O+Is4#z5gXxPxVZx-EBGN)_2PYw6H#y# zwj8%5gIXXWVRw6)z}?kQ!pFreHe{}-GG*^`;LO3R&0+UNAI$O9gP!Qff>i-g`wRhyH8bC8 zQybJ_jO0l2VN zWIEG$3FV}muz;|#S%BQ}NX$8D|E!^S)Bgi$g*a$xkE$GMFln2TjV`QJd*@amb!=A6 zNMxFT5lW6DAs_A;k%o`;v$pjcU@$Q}v2PY%j+e-cHBzc8BN!WD@pCj36k9`!^Iv94 zY*5$=(kBq#Ghs#V0Gw38dR8&VrkxAxB=oG7aRBY;LIukF$eK8PMxVkAUiK2$kuRnE z7m$5|3{+TH_FMREk6-?9Fa}$Vnng@;At9 z(*KokGM4<`gp+KS`~NH9B>4~FS%XN$z&`f1RvkxtO-C&i&&MVlgI^JjO>u|_8rc4E^_ck{wCo81sR)I zo0qsfZ?sa|)}i%y~OEUX@mAc77lH=7((jsaj@Y)6F!bC zh;>mJ5)(;>PCu^9QC=XVsK%K;i!_Wd2QNAvN{5}$$7?(`@nx{6)SEG|4I`0On(YFI z{|1}Vo%BhbD*MA0yhNiNmaoAt;!87Av))V%v8!56YRq3(LO)7Fl9~$FmWklvu4Wmk zfEEmNHoz^nIx#3XKhJFo*2v;p6FP|ftV#YwE@6d%6x=`$>cgDT``pip3^`15nQ*@U z=^q*hMR=&SKNL5Z&f*&SYtzxD;8<6#&c#kZdztfQ2cK8WD(@~s86~-I(|6n_$v6D(HnzH$Fi6?0ru4Yz$@^B&DWe3cH(g-1vu4QZhdZ zB)cutzM@io804O{RdnkfXcr+3o6NpZqFh#Sn_hkzWu2sQQ(0nR%_zfg~_5SWjKtnU_v=Yf5 zj2>khscV>s$}}hq9r|dG)!GRc<~)8L$Hh#ga%)?hZL)F)9!buWvv(^qM#FS=W*(7N zMxI}`p=UUt-tdkd3c|2lOzshc%~p%v#vOhcB=w9-EG^NS>8pHqJi9 zdMh?c#VC&XBAi#op%K;4^?9g>=4sazbVo!LbgBl_wP-p$*0T`&3$P0n)LZB~0Y%dX z*0bDbnT>(kkANc(MukA3X7qln&qkwvoiX`gxPHcPAAoP5vnC^=7bagIfSyh=?_R)B zT#P;1fT1wm)f-#L;mbzPB<9{&MJ5A{w8j=W8mD`JBtg=g+e_+#h(UBHU77RE0XaCc zX2zsP#Jyt@%pNIHCU7RLP;yO{Z9+GNE$n3U9VnA#V5y|bY36&Z2z$q!Nv=B!>icLzi zQEdGO3u;BQAvs5!PN=)xP`#0WWtuug*PL%kz~A>!B_W~np1qt*l`nJ{`(Kp3Wmr~Q z7d9#-jWhy+BHhy6($d}C4bt7x(%s$N-3`*+-Q9H->VEe==R1GCA3PTq&tk0^V~)JX zy_B?5B1TxMhq4f3bV=Qcb86~!cLyU2u>ZeUAD|OuH;!9RkMK3Sy6FD$^q#|szU3nM zWHFo#`BlFL*+2R~R)#5rmzZqQ?!vubb z{JRGhhO=DZX-(wZTvXrH14n+LiVuR{Hf7=7A@pokJ4-FECZ(JUY?ILp-Xe5NeHNx2 z&TeVJ)EXNsV>mFPaTVuwb9?)tQGRV~2!)LMp`=ARhpgX}oMSK-bhNv$(e!I5Ow@|y zmu(Aanb?Z6MeJizI7NkYnZQlhGqM!En&NF7l=Yy zHOK9V@QP)M^ivCZf5R*c&nzhbH_GsxjXmLx?^rJ2OdKJllK;x@pim71t>VEzo{2Pm znSaZ_wydP?ed8guY)NbNZQC;Bv?|Di^97>56(N%##`-v3!SjBkw``apj|4H`v-xQj zY;Y_OReY80De7cb4XlADM6K9FegT&lOjD7|R`UCuvvmmQ@8qvNCV`Ow;ccsUrvv1r zyBwb=(1X;Yh3ELAqNCRUTx~}TrAnH-o^+z?Sa=&ZA&}qw$bDqHURp@Ch`)43u(+bF zN6aQ3<$Q#kz~v$#&j#A_w&r+zYGa93so^TVa~#B+Th0hcn#fdeXuLVEn9GlH!81v( zGAVv5Cl;~LcWYjL%k5au#7Z00sm2;x zA#w^HJx3N1(ZQ`lk8j;P*7xK`3lu_-Y|eZj84>4_>ZDc!qE{?jVnhN*0MMjsRcE}& znJ^6W@Rr}CI|u7}622_86>Fx7szI~^L4s%PHoxLUs78C_+;p`8fv;v`43rnveX1G( ziv@-!BgXp6NG;T&csWY6# z(O}4+QL;D#_(xTLEigZwBwbA|J^42#XA2*MnVl+(wkUI@_xa@EctMv0R<+%?mtts8 zK6_4s*P)o1e16(dm)G-Rz3(9HP>U;!jjiT-#ai6hVGwcDy^X292)8{2&roo~yatCYDx|2p z=f%Z+xDkT0xPL=!UaeK^A|^rB1Y07Vw)&kW6gTp?;^BgQliYpP;p@l;lYuEmmRG#R z@mody^7AS9SdZKwjZPZR)!}uS{&j(&q*d{3IRiGyZnCx&6V4bXL#0%_HP|Yo`}6F8 zxZ8CBpEZISW%+jV;r7waV=t`tuP}_rpwzBdfu#0O!JVLUlEM3tpCS&TTlPFXF|76? zj+O=7Ek5x4lg8MHTpzJ5C^~fPRYSORxu7u#_<+(33(Odc@%=TuXG*eE_oQ%}DEEBD zO-^!Q$>TymhajJa2lY`$>=(W_Xh9C);L}A)IzKwdY!*~T8sEXe(U&yrlRZ72u643b zO`8oFHNj^7YMjsO_3J=CpzpKIF`+vbNwz~O-l>~mRuxz{exzjVR6kF_TiFg#jFqZr z6cS@V!0)fDDv(+BDat^v!QQr3EGid>dH=~|ry@kC%-IZLaSw~Z*Qs8vS+ z>T)tMlR-V5WJXkND0B286l`cwk?@*y#N?vx^DQ2uC?OH&6RlHK_nTNtx$ zqA1D6NT%hN*OlZ<2^xaas#^=V6x2)_p;StG-#3il?TIQaeP$b6ta0sr5D^MEFk1kJLY`O)V=u^MqsJKW>oyWnUCAmC}w+6OJD8I z{(U&Ecfy~w_Qpbn-NEn#;C-KHXpmA}&V_?+W7X_1sahYcA9n@0IvQ*bsx%TArx$ zN2vUhlzC4N&iQ!CN25Q|TUgpsFj`@ZNu_F7Z_^G`ZvwC_nvl3&?nY@G&o@H{aS;gy ze4J^+$Q2_+4(Lb7!SomeTchm-1M+{Ds58EO-}E9{qO5tx)JYGPpPAPD!cGFNv-$pK zN%K8DT9z$Nd{dQ}-`@cyxteKk5P_i(fuMjW%O+@^M*PlKYMvtA0&P?I7^n`9XT3;3 zS5#Tqm&29Tj~e&mv1#Zy%r@I_e(w})cDE9pxZgz4mYXdwqN1VMmVEo}LE^Azs;kTH zX@#=@VUKvBjSH#um=%f^8YCQ`^_tO5YlN~Kkg*TfWmQAoHQuctf1#q&^xzS0i?v%5 zjpKp#_ci9@7Smw;zWeh%<;S)`>egUVnVeN}jXRj*h6Adv0s=Kk@rNxB$DQrokTmYk zE{98v7^x47JM&rjxRzr-?8;B^pl|uPn=4(FOubL+T2HTq3CK`gy<83T=cj_w$zSk< z#bV69iE;Y&rYg>Eu^+Y%^!|Q!`+)xzD3BdK9T%nEi|$i*K4NIjDmSfzDx3-g`W5X1 zcOUOAoOCnj9!kVf(ukpda)%+UHa7SEvUBdJO5e51`evDi__yV+`98m!Rf)M`m?sxKv9V?HI0LL3AIJt*r9(FXK0l;@9lbCdG|Q5H2oS6u5Wl zABbFLr!~ZwQr!-CDFP9&D_}wsQAb^YQsk2CHR9adoN|2KR*XuSb~nr`+V3Y?f(yF& zY?&6PzaIDRPFT-t=wE&ru2e3pyU_F-_6zdu<+u8(*Vp0%rLHRSr@JOQ(fNCA0^gE9 zzWdt8yR%*qsi3uQw&z4VIZ^yK^^oS&BD~6*i%;;$`8Y zgY?>k0QzvV+k($yRX_Jg9M75l}^K&i?qh|l(ULD)j z4^ji5=y-yZS5PosNKNX!7lO^Cy*(7~aI<6|9vkZq^k%6=(T+Jw(-FT;;Qb;vGf=`l zh|NCu!R<=JcnE5(R~^;Hm#IfLw+1Dzm6$k2(G%}_-g(-G8p;Q|0}|jn+bG94$;$DO zl&k&&N)DBOGX;9M0&|Ga!mR7QnwlURldv&W=jhBpiG0AFgON#9WCtyF&O6wyh+FTx z(8PdGIGiv>O7RsS3M!IUNB6l}{S9+v=6!tK_w{j;j996Fm>|C91Iui7#LZ69?93Ek zx-eQ!eR2DP0{%2St9}3pl~nye?T1SMKH%02h9LU(cnr2)6tY?KUBonH2cf4YR1*Pc zzW;zKp;^ysFpCaG%Mg1J@dItv{xW#56!&5MFTnFouq%6`FZ^lwfgjtdXuev(_yY5A zfyA+{qeO@!P4L5_P&2LC+aIADla{&`OAV|(-yGj$cX$85ni3yfN%pu~bP-1{MYX?1 znGi1qiTfDq?UO~i{WIfhb(5Lv%%t`%5fsx`)m&r&Z)Nk8P;jR>kA&$gyDIg1>ukmX zPS<;SyQ6u2y$t_)I>*<|sCy5}c`Z&wB^$??rk4FtLf;KUw!G`b|wwo~V zO#N7yBH#|;*wg_UU;P=ELbRqcCL(A(F2KP=kRoO@-a`V%RuPjb`RQF{xse>>_<3OY z@p)~TNDk?6WOx|8Gt%=$LhIUtO^oIgWssH26aAq&J4oSSc^L+arP&d-BAQM&4hD?X zV?cdGt5Oc)kRl?4X`KI$brR;05uSPy!Zq4zzIv2<3k-KJiXReHZ7HVwn&I%mg*ui8 z7zXP(eR{Lo-`-xa27sbAxj;?PZavD_M<|H*B-?bV+5?DBIffht$i7uEI^*YSBDX3L z0ZvnvxWxn5(ng3)M-oIyd_-Yk;k@GF?ssmdU=$iHs^lAd>4y~fuT={;W>AhsBqkYR zpi}D3%E4m3et)1fnZ%95n4~}AsG-;|vmJ9OTr2v4AafvDSuK505R>25RzLt1j$k%~ zs5Vu~h(LtpXB@Fa(sNCcbSwajBU}TT@=|_vlcLvCGbV`G5Q@DIaLg7^Qwx#87nJf7 zOPQ1b3koPEZ;Y^=(CgZ;Yye1; zFI~3{y%0JNdsbaT9^j7vY9L2r_MQOwcf$y_GWctE#C@kd;_+RFI;3KclY#?G6EZF5 zCmBy;NvTf-f?e=LiGgQjIYKv&tE3!idmT!j*iDOI(+(TR6*9e>rr|w#jE^HIOdLm8 z#K>6m3rw>ZU@|x3P*N#5J3l2}Wn_OF&MUPLP`soy9d^Q!C+IU~#I!tA_7$XK!%aQi zCd-&fZ4Zbv_@Z~sTeJeucL3e~3DTPcO6BBkxOO-;p%OpVeb(RdUClQ4idvO~V-q@} zo(~@Ul2q_!jdzmU>zgu_t_&rU0Gdr1K^Yhr9Dqy_-hj_C)B5e%hGY%0ap0#vE&tpv zL;#P0`^>xS19Q-;-ZDM5pOvQaGo+Akz06}chq;T4C}bj(ZiyRjGKdpdmvFH0>M*on z;vF(asB^2@C#)>;C4p01IOsT2uD&1!9K~T9qCIGoTIVhyG%{DU`%&3sb02YF-Jl#7{f`^#^d! zEa1V2+U%Erc?AiI2B5jiveqyt87(Cv)C_Zja02tD7(cmjGA=9n^d8q?x%astV}DH* znzOAmn(Xk82!Zy;npJdj|4h_Uhs5;O-NV2%d{*+O-^MjpZ4{~D+ZL_pvTzeE(k8XT zREdF-)_g_6Ou~SJUOWK#TqTW&QiqVkSjX_Z)w_P#+oZxRRZvrk z8bS9x`rzQcpC+RXg5Eqm_Wbt3+|-Bg@Kfg3l#A!93`TJ`H#b+W7c5RLDk`5Y9nmD; zARETMaZfzb-iv*gFL^m=jG4r1d#x>*9Q?|OA>*rKDZN)98^SCj6?PVxw1J@JiyQo$ zT#aq-=nB1oFle)aBVajNDXME&%a%r z`PYu8qx+NT@l-Z@1dHY?bhnp>73Y!Efe0R(?V&LkpMgVwkfaBmX|atEuQ8e(1>f^o zbQPEfX^PvuQFGt~Q1e`Lhsf6bbE@4?R-=ep$Ka2Ch*e-Ol&@_uCEKwQUbDak;}WrR)~jl4*1g!7sE_en|(;A=;+1}>ldjnO@daE zB0lqR`{SZ)%jD&S7<>xPd<~#}mTRs?7%e`S9Z=5AyT&sng;NC~B~v2e3hJcT$(K64 z*Arpn{u%H_ppRw*29bI(+1Ze%=XWtA{(K13O!h;Bx1VV{EPsUfy)!J@S|I-u0`$Yi!c z4_0>R(jIu+iph_2c)8z?OZf9RLBeLsH81&3ZUA4{2Eeq!Z)8L|>0*%0od@6z=b&JW z-BZCdAU1;96{O+p%fV-BsA3dHeH5HUU3S=3c$}OX)NyMJtoNQ}=iC}nNo5E;k3YR) zd?XK(>0hhNJ8to!!otu|py>zEmPK-z>z$3#_R2 zc6YASIT1UAA~}W3_eYv>*0ui-!6P$t{#?4c`L5wsxN$?=fB<5#2$u>gov0H z?JxMz74>(ZsonKsGzk*r)tycNf4&=a�@bTUii=8=lw7{~g2`KwJ7`7~EYu?gPrQ zY~kp(_2q%f>>CaaPDtZ%gD5%jM2^^j+}&+_Rz#LcoZHIP0jSVrrK!q$Kzdf#*? z4{EBi6ll{&*mF&{S&#&Q`{{K~pU?=krPJRt+fLheHzPK$q z?j+Z(2a>afEU2Wo&|ISLrN_QonEq9GaZEw8AYnhQt4#3A6}ZnKXADO&GFo5?(W3X> zEB#L2@NoCb^U2G;o|;0Dq9=b4`kq{T!$MjE8-RS`vzOGxIzrN1Fb;!By%X^`>IL={ zC~=Cup7X6ICF@MyGo{@t9@T0IJ{}$ryOZT&GyIHQy2_G5Bf;+O8)eAWwwST3>Mwt&?A8tO3%t1gTaiDS3Inq9O_oEUT+21)Z&d zXvG#6mdPt_D;FBLs01tr>k6|g)8r`yg)~k>&>nxd5nqrwH>aAXFSbBiE1=t+BMzP% zVz1p|hR(F1@@$u;=yO!e>mlIb;cur(-F&&8&%zWN?T$R8iKudh-~g7ZUtv)6xZbvh zx{sr5S5HDDfxLfkuBlLPBcozZ0c9U6T<-UB9iH!YUXuXDUkL~Zb|0R#RgVP2-}?fL zdTGc|P+gYcu7_o-p30yDVt@X<6A04AIz6a~?g04Jx**p*5|KnoaI^Ex{(=3nBcw6p ze0th9Rj0jd$<9RfzP81;vW@SPO8yJJ%v~xK;_wk5JuQDqL|VJ&pa9B(di&}9av$dW z(Z-HzkUeepASI8^6pVVcHjZ^ti0?DX;quAXEz`b*Fbc&DTL~?>qYA#u!k>>;xC{(o zmQJTF>;(y@K32&u8&qow4lD^$@dqPjkxo}SfA1Q+E+xGGLFx3j^n;bnn%66ECYq_}l>ZS2+`-aGeX)S;lxsTI?Pe44> zrygNpMwx~YpfvlADE#n?ion7dr+V9qDY-NKP>tEKL@9-F&3j#KwoDgcTX9BPT>0&I zgegg%cu!NmiB$&8dxbT2&c`z8Tun?j+}st85Q(9^vxMJP-UHkRXyLxnqGPr{{va(a z{SyxdngPaYf07jN|8m|DYEJ{Up;@Bs58CnRawA4d`pL3u@F%ypFiwoGZ(nD0+*7^b z&|Deh$>gBz1yYfE3(gM_o~^a3_od+KHE{MT4G0slrfu%WFJ!bBVclrPDaa*;cM{KxI*{vPxr29r6&_^3UP_!A-SCezg^j(QE5+yXH(jdkXQ`-|xV`w;pPye9N zG=)%Bq4+<_y6foRznU?Q&r))75dNK>o1401D6WziLlNNj5gyCdv}A8JE?oqn+9_$| zkx0?$a>?{M_Fln>Et2=#PkUS$+#{s@VXV>628ceC&$tm3Z6$~JDdBLj3Ol(#vz#C~ z%82GGE|Flv7G5k*#$F75*`VnkE;1A#WRKIlywtO?p#zk=_s8c`|2PH{bb#B!tE{Xn zWh7<~ASk>91*sm(y62WuQ#)b-IKJ?o4ts$^#xL|~$zF<WXth0WZ!WS#?s)BjML%_yiaRj$^?N9)5bnS>1DUbBN$K^*?x~OB|4i)}j^^ZIaER z7Vbs|~g?G49%_ME_ZaB}%h zo5J>Y06)2R0Po}b$PV2@fp{Ol*P+&QzhBrlG}HxfxonLl5xrWE-2@LT8p&jLTaxTH z*CWoxR8kcphJsc353)_aen|#nF^G^u9al=DPue;-)NzvQwY#Xg-EFDCPQl}O@gr>g zg9T(n9!3)^liB9w&t7`VV3$4>Sh;fZ+h4*qXYNbF~WL9 zWodE%kjV9gm1jm=07VfM6}4JWShC-~825b?C9iG!hi}~ZC#w!X8 zz|dGg$R`h={zed5_!Woa`%6;TQC*E&a9A2)Pm{oUM1w;mPDbXJr>Xz?H}wVDn*e+sCT83d`Xm8^43Dj+iLZdU1EY0sY8t zd*l9yO&FmG1`2`nl^YR6tn3p6bsDsBx-u`4D4}Ku3{RD66L&hHnRj42S*$Wv8H`BV zXCZib9zq@-%DnVB`qr>c5{%Yghw-<%+8m*G*cX}LSMzt zMHsA>zv$Zuo0~Cd(cIj6Igo@AExhrBY*6!wnw$@^atO*RK6J7{?m~LBz9<;1LKsw~ zO3r3|kq&7X{o%Ch{CPokpU3p;8&747vKKQh+kM&MSU!*`jMbX*DJF^`t+5P#ELzMhi|9om^GC#}8(ClC=chumhrxnN?DjmFI znjM z>H1x(MN6u(nuG48h3neWFw=l~rlX$&_WEY>(AH=~>v6*sG{j3e-)y1ASViEAmI~_R zz#)?n6w$!3hI6|%7|il}O9Y2$7Va-n*Y36K=3#g4xaw1=%#P<-Cij0#_Ys{nw%_rAy?*b7NdHd9%?a zn^D6hb>(;?H{GPYs=dZA>?ZMM9p`c!_{mvlN?lw{Y>A-wH3}sP-h0-Pn3vMWhd!R-^I=RD|A4>U1U8-CALMfSWAI!L!M4QcsiSO?<(DXWOVgdZ>6!SsKD#kNL$Jiks00A zP~-BuANq07`zja+awk^FRX{gfKT;rab>Dc$my52A6zG9ux-3ny0T4R5f@FO$l|mi+ zoI;~M&|;1{G4DgZVe1+)in3b29NitRklCm2>znw2m+sl)M!k$Gg>6(wfx_&0m|Axd zqJyFu_%ESr2?pmdPt4MI8%_slCVolHN{O2@mjE`Fx`lR!UvE5W7q{h;#~VHSTojB3 zSz5DWVVkdc=0`HOyLt@9Qe`4D{ewks5R8gDw4X+q@*VO14CxK10W~+bRgB$rP4sok z1)JUBuZvMG_n3OixE8_S5P)Un2MH+=iFqs}$C~wv^1|=GKPEH2ML{cJ6j!$&n@&wkL zLWBI-R>V9S5YWKNw5oS44SP9>^p5AcbE8dIW!&5D-vygc*_srz@Ud-g>ye3kNjPc0 zMq-r6%zMJ=Wm(L0-sC3er28vfL{E!G)M#G2D3C8OIvl<$i zHn7%%t)1@A7zREaY9ePiazfa@j^yWfeg-u6nha0V;|MKe?cu*o*g**%U&TER4@4Qz z$T*@3ZUy%1_olVY!PZ$gI9{{d&V~ayh+k&4OF2NkeK+19<*a=ALET0M<=Idlk#ai3 zEU30Q+;7NH0UF~;d~M^c$uaXyi)1(9vApcdF?!pnLPG|K9woh3^ex_)k~gVCjxy z2pGnTq?!d{IG-xCQTCDCHI08(v*k@tMWLbLN5vb!mav%+6*%OL5|()1KjbJCPj9Pi1j7fMM?~&6fAIa&^PUH}jg1YUTf<^6QiAjBx#yRc2sAy6 zu{jZS^aTu*H2DS9jo$T4FN3Q`{Y9L;Kk`;V1={IpPtwD0%X@M&5=aENP(a>BE4#+ zKk7*un2L1+2NRP*GMoLhNojMAdUGJIsr(4uA(>!WhDP`(%g%>-)e>uK%~)!f*^6zU z(`k#bIm4!K(R(i7x17BJok9jo&DLqLTBCe2tL^7FlHNQbd%5ppijjKZ7_a){j}jS% z0+h&y-dGJcB_LumU?H9L{d)u85)TVl;M?Hm+Mx1fr_{A7b(AYiT_Rvu?!R(RWAuLk;y{7)$M6u zc2NJXto@$*-)|Yy>HvDz!TGJi@ zW_mWD0t4eq^vHmVt6X$h?}FCT)7t~K%OD=iA5XHOZGWn;zT0Mx-t_MRii1Ou3}~89 z6+|C)n5r}2e{nd~^!4%C1(5Ai_A9Ombny{Eqp)4Qy$3*rX0mdW^WXyg&7ZxL)OcN8 z;AKH$e5pVR{P9Eb`}e9t$`o&J z@9@7e?dgOJj2EoM`Q~`O-p1h1J+8&ssatL{00z#im8WC-3Z6uG+QaW8>QJ%*ZEx=~ z1;eJ%6jayPiS>I!ag0}5T=ftL;r|NQWW*Z~e$29KFJGiUTFZj^IZjWPg&MPafPJ_i znp~kkERpg4HE&&QGE>o$kx&DFc3M?MKb*uOqoD4XZQgt(Z?M0(-0XbSg(5-n=~EgW zV?5&@TeB($%I=>Le9UUS`GeJVKPxyG?)a08pOl!{E7wv|+j{EDV=f)p_fB}GUEmKf zYW7+W9>?KjlUfz!40&WYvpoGLkqO zwi8S)fuMn$f}$u%_OL)01iP)Cn&3#|VSlDH)l_4))%E@oudPt{)M;dJ%X$y6b~FAT z-SUPLnb=MMTBzNIgAyAr5p2czDw>{G5_M96*5YGbyN>Z=u(+H3vH}wC(*`K)4WJ_WVUE*$EVS^&c!OV*Sn0;EPuzr z%=S8t#TY8Jf>+fBP>$+Dkj@?DgWAbM-eBJJM zJbG&=ks8WS1WOSZN_@rG6QO{YRIJ0?Y69%<1-6XH%|=5v&g6!qBqxYnfnYS)Pw! zch@l{?40%W^#g(|Z+z=;Fw5=nDr%{#L4?n4!rU(L4GL5@dP0(YH@jcvFYVIHlLPfu z?7nwB%{@H<#2KkJ*lKL7lJjiD#Jx%zI$%T(HKP}0?lv|y$*gluXzmM|9codX%B}U#QFs~NZre1ne~eQ4xNoLX1M z9D+&L6OXOY-zntxuPx*DM%VPo2wY+VYN2Y@{e5u{Q#_1uvfKfQjB%iLf`*Ih2KdPD|8vnYC9o?Y&kum^ zxdOC->POeha6rw;h9ohg{%7q@P&MKC8SaZEQaJ(k`st(Em6Awe33+^Cgg^G=kCVdo z>TT$^&RwB0rWsD;grHL>N)9hFfB5f>kb%WXc-q?8$-TXGFq^C32YUF5^vyqn{=0%6 z_R#d_CHqeLW{LDh8B8lKGJr!M2Pl08AO$sAL{iQBj|YRuo#qDtMPdwf*#S#E2|)Ph zHQU|*I$m5$ON$|jZtk!1@p|jkdeEZuytS6ODx*k%VLu0;%wv|AELl73NBGb1!jOS^ z!rKeJJU=nmAFJ-zRRC(QaBcDbbezAl$u%s2%W?w$I|)ymB|83|KqFVaRGv>T=kL3JPC-=U-kgb2U%_H z465eKab=2-qW@m${CXv$`>H2oQh4A&vOLV&0&jya?-NVC-;%Xv0*B@v$jX1eJ=IzQSZ4=@hKg39?EZ7xRTkj3xW6#O66nVO@5vS@pcGoH zw9Gd<*FsJPlKppy$^qx2Tfi0|7qb}|lab)qD$1?{u!l)fL9C>4yuW?zKOcF1z?BTA z#v8wY2}@A7K315wKG{7zow%2VeD3=v!~m}K#6khWg%GJ}QvtjA2>|*ndbKwp2oP6G za&vPJqjQ`8D`rrW9)M0_@KD57`o1^LuTo<=27nLsa=BfB5m!sWIez|kizF!l2{JyL zI&QZszXmFI5+PU&1DN?j-@X+sf%fD4XOGBWfQ1Uwyo!DVM0)~AQC{}phcTbr7eH?P z_kOB?6xI3pyVtyH$7VE*JFFyE2QqN*8~9NpLD|Md$Z?tGJuAz*MXBP8^L z4rd@Jk#FkXqtztofdoT+NM8{AudI0FLVe*~Bjw=X_|FEO^1eRz|F{rQ(m_N*^4k=R|{C9y>ySyVm6lMxn8nR;X?tP-R|LuM2TmkNhD zm?m?7*}Jm+cucj^BalNyo!{9UIZ`&?VqC-cxN9d?CxteBqj1DIQCVtcv!i}HgPI9K zbkfqx6!(;`--s=4yjWCe4)fycWJ}U5#_t*tStHW=JaMDkaA_VLQ&CpN`e=FY=%{&s z#nE^QF&ro%`D4#NKJw$_Pj$0oEIo6z>>cU(gY?FSg02^txr^2OtA@R?N4Md@Z&T!> zl#}EK5mh^uqF30QyizxzveeG%a+6`+^F|X9cTTmZT|>QwOLweK+KO@WDOSsqcgO0E(4t@fB{0sV6gX)+4cCQ|i-qe(}lCNV@tUrQfh zpR?kbjJdn~joss+iFi9F&x3?~tK7KrN2o{ruf``0UBj0Y84(R#c+zUPKBoV)6Jv0^ zi_gSdK-o52FH(X@ZI);0E?AlcM;SRIicw^;Ey!_m_&RLrc1I&n9)%g`{zbY{KV;(K z1ec^{$z@1*>N16r{)=^VNiwU9jdm~j?JX<(;j*%nuWf&+MMYyvP4N2GChqjS(iqmF zUd*`YW=b|ZUGDx}R3=bk-)}VNROJ`6_#(y2^9m6;e07d*!3}z_~}&Rv1w?LRpZFX|G;wR3(Ea~4+wd9N2jdM}4$)*EPw%Kt@-5P8t+NwDcFb4^(7NsT$bBtvA33C|h z$|@Dg5dJ$2>%5x@vC_Y4*5eaK=h!e3b@BZ9CFM87Ehn^>0tS>#Etc9b4AdiTAOK zsRttOO$Jh86yq8n_|7<4VrWahoZvs)G}xUixhr!5FJGQ{Zf+FgI@aayzqed>LSQ2HAz5|t<-_;xI5(@2;lTT`%bBL}kxZ}Wkm?<(|2e#A zp%DA7KA&|qL<48o;i=+wA}dyR&JS3WwLFWbvPCg$O=oyHa!ak;?v@%>(IIMZ)8MxU zn4P#y?(TezMCD_qhd&gwp&y-3)S4Sl3sXo1Ouc7OVQ?;?x2eMnDJvE5Y?yt#JKfO5 zVb*m|uI2h@KRxdKaHTo=hy+*3(>pAivR-P=hdV`Uqlf#@6WTGXq=GWkCs^mkWtM>M zpJRBvPbc_g=mW60>_$-&_Cvj*KlDdmMWLoyJRWf3krV{NPHspSZ^PYbhq9n{^MvIcVPOW#cHxVdbD!$Yl5a4hY zedsXTy2=!bigZ>E^oVIFJ2m9|pLb+SE}1VZP_2;tq4{iz)-N;Ysu({Q)RUNyo&BdLDdwQ1q|qrNtVodPadU?+4O$Msw37X*#Dol|o=&3AZVu z4P89EApOY*pBlZ29p}p|Gloh}^c&=VCWyyp^|e!za2SWOMlqKpQEzw5zlFbuc*v9b z*@~X%p|>5V3!geZqZ>^yjnmsai=fk(vA5%XoNSBRA>KAUbS^D-6 zku9L*ELJfzw20QJui)l~KbMkihAJtWj+8bI-$lpPtA*7-jrnl0Re#rIiVSm2c2xIU zWD(6Ix|WIMo>0<<=w3J5+h3~n7EzJj(3KkC;%PpftS~Ai&9FD&k8?|AJmG|Iz0$ve zIWB!>bRB6qe;7-Wt;u$7STqL=UISbWit6rE@qBinjXL&Woi@$0yu=Qn5B{=5EXpf4|IAyG?$1KJK$;HKIO* zRk!QPURW9>Wh79?LpfS18NzyZGa~ZBk#m?SXj_#Qj6&Qp5q=ViCOcN6RiZ>UCXO>q z!YfF}`eijVUtcA-#}^IUM4To2CLsPgyT7$xWAekVHRY!Pa0TJ~l{nZV4rMLBZ26vf z%?^FUv=)0VeQU9?`IiP~)2L%Cti#P^tv1bxcjLE6VBbWX5tcH}wh34g5_8ooT6Jr> z@-!6OafBq8@KMYzgJvbWdt211p7Az@E-Iu7Ta@=%bD}~9hZH3OKR?(;wXPTvW;b?{ zyI!YE@Vn9Ssix<7i;;g*HaM{HWu<^=F+ zI*q{N54yYv7i4vBOeV!d!wx4`4R#E-J(e3f51}#HBnn?>d9Jr59QOhe;*d+COoB9i zZIhTis8=H#=cM|p<5iY$lMCn@``n>94>3Gl&X*>GiaZhHyL$7MJO$aLCp0iF*RO_s z!LTi7v=s`x;8^qh$}ws05TLOsIy|I0Qj%!0tsSq>m}0R>ENsOKRlIl`tgq@^Uf;Ex z=6#(|9Vg}1L1D9#tFZO)-sFl2AF=mPNMH!=imX^rRP~}}JgR!TMEU5g$@LFZHB-ZP zI|hmY?jbIHQ2e7mtmobDK^Cf44pVdS7l=?pnf2ns{l@gO)&dypf!`hhQWehfCpOvS zB#-uOQvvS|Ua>E0rA-#C^e}4)GZ_v6AjTGUqj#I4V%Vmk@Oznd$2Ezc2Aou#(w+It z*XA~YJxG<8RK8;9p(AIUYJ*4^?da*>+qoXN$9p}zo486}3v@UZ&gNQF1M^oc)mg|G z`sIn4OnRi3LtO7lCsVsCV=MP!o{Yx0*aEzf0Ta)0`x@~IdrYQx8IH8|qgbDfc33i~ z{a2a$20{Gus0RI*27j)M;&5SbwCpR8g}<>fnVm$9L)q8J>36A|m5< z_I0a=Y;Jw3c`;`y4bAX731Ynr);+vtHuXcLt^ND8h8cd8%ljV^o3wM{BQ7ob_c&PC z=qa)q(Q^}&Se_Z@?u@4#e-MSxP6#95+J;uX+_F8uv?i?ThP)k#_*_zcvGmSRuj?~P zaFML_`De;`)8(LuYO5`Jx^)_vmwpP?k|GI5qX1g)K$S+N{vdRtN>^fw>2+cKkUkv? z67n>lK|y00a#WZVKPc&Fa5y>jbM!*iDNNIs9#;IbJ?wde@#E-7dRxsOpVza+^AFjsNOb^mSmmVxquwE8i*SRYEB|7X7;u`IyZhp`Aqk-$8 zd~T%=T0Fu2I43WjDEJ%8T2dhnU&qi+G3)N8)TPAsOS75RP)c_Zm1@sJ5l8^Vr zJ#^I6e2sCACGq=}FPBq2cDQc5KKjK6*+$+C@zr&M!;4q&2~~Dh&taFxh2QMA=QK#z z7ayIPtx9_|CPo4f6Y6fZviFtf+}+QPIgcst+`*g-3!rg7yD@1fgwk@|>D$XCkQ(^a z*CD7Z^kkJ|wEACdh0<5kn;;)%AB~7n8(C%r5cN<^JUyw9nmq7cu-X4W{`jH9i*0P) zM|y=ZN3QbnJ8!}L$`Yl!3g*kRK|Qn`@5W^!kE`h8`by&*Ex%pBFF&w)tY&6$6z6xc zaWQ-K{jHLXZ|KZ+Ho(O`FpnrTBI$qRUkm0&ovQ7+GR(}`UoA=}Q3@F8)EROLsizIw zCgHr5p&i3q`L>evf!>5*e2SvffwbOua&k8I%y_E5h;2^BGG9$THOi`}l4X^4$>71H zj{>69UABF=vidmwY()EZvkiD%&ti zSb5sf(bfkV6_P7Yt`h_MSZBW0ws6_G>)Q@i^$_$|P$!>+d`Xlk!h%>k&TDd8IO8mf z+UD#&5PgEKqVn>IaBDw`$Q%K~$H&n7iR7oL`7SR=d$TdqME&u3FP=Q>dm8r^h5om5 z7DT7?-VCfxO%AC^@xhe;WE!}Op(fANvS9-0YLRjD%oVTN7Sg@ z5j0TN8xkniCz1N|i1W=GY-8cCJhCR{okk@ZqT5F12Uv}?^<}sL6NhGB70}(NQ*K7O zGlwW8chJaZ7E`8{kLl`X#rR?8X(-D##lBu#u~+Grgk}{{G#wQ#=t`?erWJAax0)R`A=eIx#bfNl1W?vZSzMdXu-lBFMt?jOnT=mbqYJe)&&1j% zGqz}PH*IIBbT%_BcL}!n#?zXJ-4xOfbZ+K2sop~M=9luxAxvcmpZ<5}Z@Eqiywaf`h;#R^~ZGxpE>(7{yHiYDaU114zCvTQ z6#2@M7HbE>hpZIl0XFKOf~eeJuz2I;8o7Y#&-@Q}|I6s!#&9=-qi!L6oUIZ`O>+14 zyJV%CB{^Zf{vS>Vkag}QSO-;ZDWEz3$G_P<06|198CDCte}&BE!+$brQ2!UXH*W6# zoAltVCWjkP2j&+PECdpj_dv0w2zbTBN7=Ogf6k+aJuHw&w1085JsRbDap~^r+HHM# zDg|B_lmGha33B-&Nmur!r9!-X_J2k!8TMM8#yul#$ZC772Mb3Mv4C`IJfCAMS$4t` zqMXA2vm4xP8$elry+kzcCr}p7MEvk2gFtST18Idsd@;P^zsmJ5{4}AMAkGW^=`?{~|oO7Mm^PDR| z&MB$%$p~}v%a?_xqxdtZ+chRnJk9yE;}w^2)uw_PchCss0{MRED4++Y0k=bG%+y<< z3TmQw<^rTcqo(*imi9o&#P~V?zDkF9f^Gj@BWO^f4$a0JSk)(~X;yYFNOhL8Whw~M zC&@bGiKgT9#l^+@s+`jaF00P|tI4yovsxl!J1GwrpjywwfhMvj1{)a;k3<~!5G{Om z9%{QL0TVJ_oF1%%Jb$6Jx;HiL8py{3S##E)Xp2FnV#`KM)bZJ{BpJJXKBw^jw>-Ya zoJWz5A;Z2ElCi=^mqxVd7QDEB{P?kgWZ|KiLSG_ZSEYlovppS_s%_O_0OZ4b3~uPB zfmC!Pzmlo@rv>OoKYB<1j(`e=k)40uVw)*7@k~Snf!#sp&0APxt_@s>QeMGxT|hA~ zC`?N7`>$R9IrE%d)^htXs^4Xb#fvmhO#q}KAI?4Y zbS2<7q*PF|+7WSLP|0B05vAb%4G%MFBGqF^PhUT}#%&^NJk95>SxXwL$Sdlb<%Xd& zpm}laJ2w*>n~|E@O^h$u5D|xJ)@I+Ps*VCHGSn|-Pkjc@1R4*oS>$b1iq1e`qaDzf zh$?N}no1;O3gMp+y7?a1twDmD`J(e1frb9%Gz9;EHFIdN{&4T zeHVIdutto`)JMH#Rno?DNR5I4DTln z!HRN~$7AZf?O;vy5o~l@xYcy1CW99-8P~`4$TeKHAkJ}J=;4frkghocxJWxFH${lX zG%ayLlX?LoGhASmj}~s@M&=q6prkbSP4!oG>-w*pKc%6e;2?dx>H~2L<+1TfAQ|-$ zgvS~7Tji_u+8FmV_XFpp4}=F6YLZ-y@eBjwG88kT#S0U8Ft~)tKq~_XF3^hI^OUjE z1HILfvA(5%b>Q?0Mm4GS+p`vgj;h5yk$$AO-To;fEFBA`UkTaH|QB_KVT08k-8;I}Y>HZCQI;kWS&%!59 z`6172M5J>Xtm%q3v8Aj)nLRKp48%m4W>ef7K&Gel5yd0=Zjn)mKFHJv5(P*H&}PeOb9RCo_wJbp?=>E~ zy^Fj%5TQ;j*H_}BqoXT)SKYDSdfL7C7C{HcF$TtMn81Kn$TY_B>YcFU{o0-#Is$Yi zz)E6*g6as|pw6z817yw1ZaZ5Q$}a@KFEF~%8LDO+rgJBqH#bjFUj24td3kx_wq)sh zrNx2gomG-Sr};L91}}?f=h?!SQTC^Yvm&HB0GKEN1k3_B8NP#70dpC3fO2NIGYc70 z+)2hR+UZ#XHnv)h8HT`=lwDHLXj4~-96NWf4vu(NpbgpGJp5NKcnl`ri6usb4l86O z5M2eX>6xNhuURXI>!t{{vb!tb(nh$~#0zEt_psZ}HHa5lsRUWKrIo-TD(k@e#kj5i zTr7g->8uTdG8`Lgkc0`4-G+7631=eKsM^}%Ka^S&A*YoCeOCaC;_z{y@4(qrV&v#I ztbKsl;(SL%Ge;Pa9^~rCF=N@ z3{X^)j@aC7SereI1vz=aZlH0g`M^AwHy^ZA0W)RzvuPZ}>K=CEkI;2(EB6TT{?W|K z@_oxyUKe%%QpQuBJ8$XA`RwxN2XH!IX~%0)7zEKmR^PXffiXPB@`Qc2LJpiX@Kz_> zz9UCdZ{J@R@floIY|wLQ6i#y()L%FLghiO35)u+xNb}vR0$Hyp)Dwr8Zoo3@=ZSK5 zT7j)k7Wy2>q@pqsFum=dGM7QEz*qUkAO5P zj(`+8=%dC%lf9Abt|%7LrET(?81?AUSN)j&pmG^`vg?YGo5 z%SxMfd|5-KQwwE(68SXJs~MaT8A`434&9}EN36H7zbWFhIQ$2lY*IvYV8TbdEFD?G z1g^qeK)X%7ig8in$Y^J ztxY#ZM7smp$q}DiA7$t^;-eU4VfcP6ZzOOm6fgJt>D`j5XpBGw@Ubi1CKWI@ld5kV zrObmuBwVbt!Sm8#B&?hdt-?7=W(z005)%_iL-kPEJ1LDA&a-AGD>r;x#&MWzvu-_L zcWtCxdauS$^xC9j(`-0*?J5SNKmz=Q2)G3qBH!=DbDvc==IG3R7$z?xfAmfiUvU8I zJ~(;7|5#@x^s>%p6u_au6Qw7~I}cg)pL_mimCex8YKY-7p?ncu4O!bv1Xa6Ff>FU#f zph^kLCR1!c3~GDkOwLe-G}96ui9hBBn0<U)beZNSyW70ZU?4gN3VG^yFmiUs2Z*T?*gkXF&_w=*f@kEA5i^U@vqa&} zHR*(Ue|}(`GJ)Q)W5=Ux9(RRp9-*@jxdOlZw!ognfROO_%Nl^&FAhj$`BZwdw%6X)GL}2y%E!c!ewEd^d1pH&wq6)svA1SkMdtov)MX2EDKg{MUbh }c zlNdXxPZ=d~WRzlsmSIPh%F+}1Sy)KusP5Qf2Wh0_y7ztraK8rC(&H(vYz{Jo#YS!c z>FMb@sS1V_fx5q=7iD-ipN?2)t8?1+(53q>Bm!PGy#MR3-ZsBHMt;wx9rbv}qAB`A zZB6*%4~Y{qik-KRR7XG=n3E+f-A)}8?&L7Am96S?aL`NRvB?AAxPd53_5Zc}nc3xp zxwLMu;*X+xVJNstH3&#=+j5JuFrD^@i)+f2m-{=RF9t5XgEVWKug@%_ntIWbTDl`mq0c|2J zUzed{^1Ki8*)iZ^w`q^!DPWz95YVdsami9}z4z{LOY+)LB%A@_MMJEvdJ)+J7-(kb zWosZbF$CsH!XhFhaPlPZn*g>M<>JmzD3(7+BT~15Ll49D5RJptfoXYVkyhBgeg|!^ zDM}q#l?Nbbo0(q_~+L447VTz&6lCK_J$vFk}Wv#8FQ)Uut$7cV|#MyEY4 zHO-K3cqbmGrf_{kv2xnAEQ*t-AC3-7ywr$}-eoFCi zS!QOA^-b3whwMTk%cVG2SPfktD|l#8_F8)u%P~oOB?Xg@C8eZVyhO{A3c~{Tt}K7k z&Wm;Qt+UF?$}g21aPIW|(ae_XpDy`1*Ur21Zq4P9o%RCm@p(G@InLucT}{(c6%taS zjd3RZvd79(8KNSLVor?8UM6{B`()30%unL+!u_c-p(+-cUtdeTIsEaiZ;dKJ#P5<; z=}SpT6W`%gHKP;dyUz88D#9^2~IwYiNS0+|N17U!ec2dxl@=lB3?nA#jBtD zi;ABAe>(aHJP57$pNTsQzj;{mIR5r~Gy2H%uNcM4!GZ~C3Lh95rKPPIdBbu3;ZKEY z40|QW5`H*zn8>^2ydTC@hwh&!#2kEj@^!_FOHXgL98!IrCuHmBDS8!Wb}(8GjV8_h zS0n5lnNs4#Sb^{&;-8f0$gt z`sD7f`4bKu%s0NU|M7p&+hg3L_>t>XDO(PoqB3cXlm|IfI3F)ych+`?g64@iF7R{``u#LJ)V_Uc)J=&T!5P2)~xc)x?=A{ z-t}c!d}*tHzS`p~jG;{pwf*Fvp5nC3l|Jh+Ztc_=BCjC68ovLhtEb{AkvoRZzHGBd zSqRG-*RKb($uSz*9aM_ey#QFAg$6mDTO@o$eMyx!1q) zWzUMUyC?lYZ3-||Q*$KQChnIvW3A+nQSgazLI>YM3qeM+3T3p=Bz z82Dd^bTeiLRPw#sXR$a&XMu4ftX^(y-_-ibB&hcF?!2+@J12>AFD}}>C&yRENOl_$ z2O{1q3*4$(3psk^p^oI^8JZ2`0euc)_2z?1NUG7`R1+q|vVBzoHo)40HJ=;jg2G3t zV;d4KoyPG7yLqWi1Z!?07MdmGz0x^s=01Thiev~r(;ih{eCL_WvVi`_3pv%BSGyW4 zCXClq?EQ0NTzaP+Z8EDZ9MkGmOeDFc+ph+Pj_GzONKHN~`}HeWVEIoNqgoB-^z&>v z!js}vGk|EOFKF!vEY-y*B#BrBmyJu*&OZ_!@@0ck0$nr8YK z-S}a&VobO}L0xKR#jmN7H?PH2zKUJnW%)bU!5KmDtlUqQ0S>%S5}hji_Qaru{rbxo zK9xQCo4B%Ainbn>FjC|_`JY#X#kzQ3a?iQ@n_d-It`VQfWMAT%f2XdVBChqTRV{MO z;#9iXL?$_zOb+wod*6%_Ti`ydxpOf}cKM3}d!MAAKVIa7^~(9TYgd@<%Q8P%JDu@# zE;%x}KU!q0j3CzYR^X2>Y2_F`w$0{VIi#v_dKtgcvaPOlCnXl7)M+p?G3%pM?HcEv zbSBuGL&+M?>37BsIY3#V$gLUUdtS3xrv_46nVi|Fq2CVi0c@i2cedIgApi@65cSe8 z%e?(|MZ(+nBp_R)KMtFkg(L5Wy+_+_)gS-3Q}`flu>2xFmP%y8x1~W8PO9D{#t-k1 zlKpT7cRKU`+!sTT>O8IV+Zk^>w5k#~a-u6EnwqqRSvmyc@TsN#^x!{^9B%_*rS~+= z?SIChDH+-A{GUO5k57A>k3^Eq8E|83$oq*`$w7WaBmEq4`lu+D1n_7ynG*5 zt&JKys|G|ePu+xB|0(j!m_}auP|DVS2F3Ou w+ze};q}{)j9at!S)?57Yub!bA#JRK{S(CQq$VI|;8u+JqO6O#$n$^w!0WwA*=Kufz literal 0 HcmV?d00001 From 17511370841a0b2cdb0c04a22394086c9e842263 Mon Sep 17 00:00:00 2001 From: gongweibao Date: Fri, 12 May 2017 15:05:42 +0800 Subject: [PATCH 65/88] fix bugs --- doc/design/cluster_train/data_dispatch.md | 4 ++-- .../cluster_train/src/file_storage.graffle | Bin 3031 -> 3031 bytes doc/design/cluster_train/src/file_storage.png | Bin 42615 -> 42121 bytes 3 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/design/cluster_train/data_dispatch.md b/doc/design/cluster_train/data_dispatch.md index 39fbe55c26..c82e7b558e 100644 --- a/doc/design/cluster_train/data_dispatch.md +++ b/doc/design/cluster_train/data_dispatch.md @@ -101,7 +101,7 @@ PaddlePaddle提供专用的[data reader creator](https://github.com/PaddlePaddle ```python # ... -reader = paddle.reader.creator.RecordIO("/home/user_name/random_images-*-of-*") +reader = paddle.reader.creator.RecordIO("/pfs/datacenter_name/home/user_name/random_images-*-of-*") batch_reader = paddle.batch(paddle.dataset.mnist.train(), 128) trainer.train(batch_reader, ...) ``` @@ -150,7 +150,7 @@ endpoint=datacenter2.paddlepaddle.org 不用mount的方式来访问数据，而是直接用API的接口远程访问 ``` -f = open('/pfs/datacenter/home/user/test1.dat') +f = open('/pfs/datacenter_name/home/user_name/test1.dat') ``` diff --git a/doc/design/cluster_train/src/file_storage.graffle b/doc/design/cluster_train/src/file_storage.graffle index 95ad2758ccae252dd322c497e7b167135cf478a8..5331f407f7ee77fca263be24eb83ed1d44ca2bd4 100644 GIT binary patch delta 2724 zcmV;V3S0Hp7uOeEABzY8000000|4z@+fw7m8NObgLZS99V0Y`RU6w8Rh#AI!4=|>v zO0}i7twmCMBpKtG;wD$gGo*5pH^>X*ZMIT*i*!ra))CAOY=(&`$~yGlfB*gWfB3qk z*0Ej_0e=vP()}LySyTmWj}xFE-QOpA!D?1*J1+4*=kBC>9< zu<@F5DST8&S(2M|X{1O11P`y-%#8R-}k;-%}4i94PT)5JB)pW0~A zHISWYkvcCo4Y;4U@Z>ICwPD$~x0q(CA?hn>+RzYd#F#nQ4bncmQk{^|-Gc%9EX?Hi ze1Fm+Q~nflC{ctZMN>pk(*y{pr;F5`fWmx>4)7kw%Sp$|^Q@;`YpIr?hwp-Hy+OU3 zLv53Jmb8QY&?Ga|#0^^Rh!lHW+?3h9(7KpnAIvQ7aG6*T3kC@@&J72drbm`-$D3Xk z=LdwMh!B_XA=FHe8CTPb@=k@Zm*JgOa+B-=LVsu;ZQ~jK@t)Ee#3mjk{_)zwCND|X zV-I7Ooeb?6rbkM3ZQ@aYh6)dMuBXT|!=L4?>yHMz4y^Gc!0recH12?%ov?AAm$_?W z1h^`mP8wbAf`vhRUl9BwMO;#+%6dam&#{EDsr&PqSQ1o`1^_4vvMMOj9w#bNiQZQf zReu&VNdR(~X?dO&@!B>TB%=Yyi3TT@`$#Z|^NB@xHOVNVJtMvq#1g_vQc#jaK@oXH z)1bslGSuXl=uzbk3Br^a&qVPi)1s&P`$SCJMxHN^JWvq1Gs?Ar+;O+M-@)UVjl9hR zP#ATBJ4pug7eYQf$SOPVycix-zz%{Cl79$VNFs_Mi3CCtTALkY7?Q{=b}_z|=b~Wd zA$y}pR-TUh^7l2pr&sIkUcFPE5C@mP?|18+|4Ce3=!j zr_~&vJ|py2Tyv{gV$2RD7BB@F-lsaeEEhznBq*u|WQGGpK`sGa5hw|@NO2@jBeJHXEo3ds*N-~bIO?945Mae0B-uYeK|DsrB0)YvwvWpRB5J}ZkXEp2 zvJNN6vNx6Fv6w~KJl8PQBD?$zX@6SM1ZotD$da@QN>J8#I`dF@0Z1B8i^La9+W8Kn zJU*WwZl%#s3_L-Jm4Ac6a!DzP zq9j0u#soUgRq0tFu^bW=VmpOr&(Ng8##DjT7v+-l&4RZEo!H`K@&5&k(_Rq9MFu2j z;W{y-*aJ#0h%kP)1rhPNfNZRC0IwVo=-f>`2Zs6dsycp~s>NmL(urV!0U zp|qWeZXYP*zs@KKY^fuB>QAA=9>qGt+Hz*Jkg^y!dP~vVVF}d)rp8zD=v| z^csnyygyX&*EIEuRkD3lLVgy6p7W zjiF=clh%2wXSdG=&eR?V=lbO8UH#!+NE23|gcZAWUU@h7Z&6bVuDa^Y#YMZ(C#bGF z_OX6G>rQBEwYt;`D}QvF_Ah4Bz8yM^1wOC#CwgP5OYiFTbZ|sD*M|JvE%n5(Qq%_^ z_0D8%Cx1qBLOW{vSD~1l+Bv91 zebC3%hTftw9;3_3jnwNn+RSW4GOw;Ha^LAL`*7iZYCCPSGS+pyGN(Vaa`|SgU!7Nc z{b<0BKyTjYgL!2VwNMw?O{g=m^oCQr@^xq49O}ouJ|U>NG?>*$f92ti6@chA2ycR~ zDI)URmjL+UsDG%jsV=XCSRmXb1m5r}n!J5kZwq9;>}9>>Le!L)|D$Cnu<}PxEYIA0p*gql04K~x8BNJm9qrR|wYna`FnM}0Ny2=?EV?VQbGP`%O zk!uD+YPj181X>g~hBH`t+)tiVFRn8;DZRF&P z*{^^5*MG17{4Rn zmRpN$U|o8ensT*}wu@DelLQ%IT(&WoEt{QScN|PI z4e%iP<24oV0)@w^2E+Q#5HOgNskIS((t)UmH-AMQk#(T=Xzb$PIrKjDi0%aAAZ5%e zOMg#HYxMBNfv5Lk^L*i6?1&NGa5MIMa*Npp=_BNtHr}{HZ4%(kyMOAjEeweP$^Qnx zlA4;=#>`o4s?agL@i(AlPFw6`?QVqx0dd|VFSeaOz5>5wJ+cW3l0o6&Tm;D&zYbYZ z3x7w%vpx|Q*;H=JO&MvnA%mDKqUSX9=pu+dIqd@p4lj>wV_c=z=4l^ZO@>Dowoc2d zb*b0mk3U{rwt?Qdtn1y6l%{cK&#TSH#Xq5l_)FBoBG6;}&nOh~60takeRRm>@1lJh z2SE}w6*QArUdwAHGmBimMaY-Js3j|aSASKY%6pum@%K+O%B-)0*_TRij!V3S;qR~s z&})u=k4#^KeKOdG199qN7CH)mM+YPG)U_8+$CM74Z~9pdO1jZEIl@HIvx02+`}be} z@}Ix|BY{Vv8V;SDhGMW_9^t#_t2Z0KyRb4t3&=I_Gwb$U>*EZsr#g@HzVG9@8-G~d zV`{(1N1E9r@sJ)ah-JRRi=;YXQk$1_7DeX8QNKF#JgT%%=~>VXvfrp_c!~J5nodGd z1nd(!LNuY&*t8r%5#)S9r#neX6vv}0g3ipwvFXnmLq^XAOguzS6<=Cr9CWSpsYp4W z-ifOGsfUIeso`M=I(VE2Vk>npT{%eM-k8{TZL+F}e`u#*LdZxikUhhGZdAy5K%V0# e-{o-6yA_W~nfS3jhF_gJ-1#rhappY?O8@{i`9<&m delta 2726 zcmV;X3R(5n7uOeFABzY8000000|4z@+j8T$5q(~N1>@Sc*b=}ycfH=Cu9n9m+q&2z zm#b0`2}xX&godOnYo`2?r{o(_dC4E-2l6*tsr*F%(vo1 zHMZ6UvQ#SA<3L<{u^L_L6VA{_L%>ELQ;FC!$JKvrYwTu?AuCi!-JK$XvRKi37} z@kxJ!OnFnxokS6q6ipFDO%ouXelG5D0tj6T0(#ENNx#Z-tfxI|sg|IJ?}BW-LA{$p zZIgMHw1fT7A~V#)4O;Gq6gyqqlG(k`x|m`g%q;G2nNSdm1qn0GjRu*fN0x2Jn_d^^ z$AhAP5SQ>F(oB#USJR8~PDQbo;hk1;$|95R0ziN0EZW91{Np{PHHb|-O8n!shfQ9R zuE!q6E;||8Gfa<^>e|Gk01Xu$>|9TgXNEt^Th|{Ab{$ybNr2rEGHBcZJ3C?HJ}-0E z#t3j#Je@SU+yx7R_`V?aM~b+lPL=hBq@H8*#is7hYhp=IMH&F0EXb;$NPC>9NF{n- zQB;3f&?EuK5{m{rFR^)c8x4}t0OUl26U%)h7{vL+BD|Vp6w#g$-wI+A!b(z5l0-of zc}3Hp#7i>NM=pzC7|kLFCRT*9LOO-Rgb^k7qXW zHV;5y)Cuk+8PHz{`S2jC?7;J4cu)a52tt2IB4{CrD25~w2uWydc93C6BD2`V_*$Ne zf|-ZxjUrijI`Yfk*YuuVt+#viPI*EcT>ie_t#|$>agFwAkCRlP6vi%D-2$0R=z`i7 z$ZUbk%K(|PMz?BPG=X_VXD@J7HrWXXjID00SSrO4Fq7yBRoi2og7WIh`ZD35hQ8$5T%l!s2Y$N4ip8s1b9WD7(^R{ujdtO3-6@QRq{{RK?7#WKM`3TuQE+>en38z3> z!K%qRoFL2IRFcPH7G?8X!&Hmx@;84>(~>4oqfkUPNvog)WsRqqhsp~;(s)`VzTl*t z?=Z^qC&AaT!7tJy1#cKw}jzir0r=HF~C-G5B zf~G3GsL7HfY6=7FkS5F&ps;^SiS`JH-v(e8UJ9_oX99LngR%%@MdMW!3RIJF1=dEsiQaL{Dg9f1G(}8;d!U|WH zogTX}bPRpcI&byt_SwLh+5_QSpIp7GKimsx!U~kIVzLR-dbtabHaB5e+?#!D*{n*zh1T~iivl{8IJp8c&5Zwmh zP4G2EM4tN+0AGI`6*ZRX@=AyWgfIhs!>efW_GP^-komHg^_B}!Q)2#)mZ8APA9(@p zaS&>+{Cbb{O}yPBh7Wn8^*bmr#qrWBD=zm@7~_*=UtQRiYho|U`D@X1YQJjJv&=Z4 zO>R78rKuf>OGLW{v?1kt^b7GAd6l_s(OyuL)fgzjtg?TGIdoa7@kUhMMpQoSWzKl| z-gsEx>4{!vl$2{b$qlQ`nPp=Cgcvv2OmB`%jA@Me!t$+Qb_-^5qK(#7&d?b9naz{g zy^}465?0xIJoYx&ruq)!X^$i~g02=_q6HVWW7~gn8>UW!JDn;LULIS$iC9tPJW||7 zPR^M9`nP|7{rb;8b2(;pMla(tUEs~)>@F-k&nqqR9okVc3yQr*cP%HjL$_%1V5NrT z8RWg?)?yo2m!77kTrH&SVmh^<#IwHTlRl85w?i@O?l}IKaVd*Qf{ZXO+ZfE2%}%g8 z4knofco6;ZnhJPj^nn^6L+4|lof4em5^rJn zJ1hZu&GGM%>1(i02K#UzPF>7GM*;BYU}T=U_TuT7(joIrKg&T$H~J<=I8pShARGSv z{nx+z=kNbW;E||?LzB}`3>M5Id>4K5W&?N^R%U1cxdwh_-M(vmoZ)q<^GNUeKCXYe zf#p4>_KSR^nVlpariTk+neXr-sZN;G<|WOd$hm!mP06loDb-9CrOFocyvY3nb|m&{;V-%^lZSyL-bVfrDeuJ*Giv? zl=J1CsLG#uXtTn?oI)PyF=k_g=+zY{J6WjySux)+mAa`;qLD4?$XRP*PdhTKBs&1 zO^s@nx#!Gb*akq!xIg1jC) zI0kJ9d{lQx9T0R4cqLILMy0Q_{bi)>Pk!_PV)}p!b$j% zE62$7eUi0YR_lzy$m=w&+jZa8_+Y{=Z;-+wi_`Rc>Sol?{c)_Gl5^BCYpFNT-Swzq z(iO7`5%c9nG3hbsUI;2dxVvJP@EcQXqHXt? zDFn6&35$&!h-|{EXAq=KnAa%+v;a_h9hlKca}bat+Y_UE78`9kRfb(M(jsQfDMhpa zgDcMdAZdF=W?T@X@y%5VFK_RA3qkl^Az7d2mE0H1QjNhs)L@eV=Wr0)Lb5j;AaSuE zEJpS}dLvrC8SVQ)kP^Zi3V@b?AmqSBOb%E0ez6gxxrgv2ME~KN=<@|tkQ&*qLFdaN z*tG!YBFw$tcbl(Der7hP8IUrYV9j7v{wq1Sx4vRJ$Pqp&I^U%IsG$8WkumZFVPmKZ zAg~4YW9W9lU4`D|8POsB2*4l{pbFyVRLX-Y!6yoe<>DM-9r8Fpw8Gp8AmtF5z`H?t z`l6(O&FC7c;Z&XvAqJ{+!(8HUpp6GQcNf|~cz{_2RChOR@wp&-LC6t;5Rt(D zKwuKGKyfBT4i()-A*zEJ5|fO9ExFrkFvpDhf=LXcYAj=Ma+LRBxT+=6?buWiLQ;q;6<{xWz}R+emcoVDFUdF zC@)Ht%Gl-tab#qQ&vWMGs}-^p_$4MuBgj-yHA2Tw_fR8IJ5a4jL*oE(RdFf8GR|#9gyTeMu z8j09NL@Wqt+>yd7lH8(gqPV#`hg=SryrEhG{1O*=(IdlyfZ?E#s(8BO4odDpfv?N( zm?VKee{+l-8hgYlCfO&3$FI<7s_Uf>BoCwr z`zqr3r!dSM1BpsY(W4`zb5Iw>_~(1xOgKJukL0stkEEJpqGYBd+A#I-`Y_nu+ptQa zb~&SxhEkl;eTjwwwqju^SLwCVLy4BM=TL%S>vzSJ!cnLF=Cru9=rr-O5k z1$p$r_JNXw#g_jJ`k6Tu!_>F9!7DjJX`NF7RfO500m zNamwP`8d+qjX9M+VQf`3wq4!U4-dwh!&{0H z!-L99k!6w<%Ye+B%UBx67@yE!(P*asMbDsdrxC9q*W78#?1$&A#=HA+||J~Ki(jIfSf|0E%YlW>g?nY$inHPE& za@Y4R+p3#a(`Ykvxwpl-ML@zs97Y03B16nBx*nwvu`R+Xwv*qN zU!F&iAtmxILVZh@~P?rB}^`<{udvQX2U*jA)=zL!qZu z3#B)}d7j~!p`w9xj(@J2p|4@P8tyXixcxWP3#XdQ^tWirg@UFu(`hqD zCiOE^(-?1rIKwu>loO9YHjlGZ$LW*D6Rw4m>W$8Kks_7-3AGejS>+YT1_X6BcuR*h z7Y8&Tm6PI%QRG-lx>#LDeR)$%BdPTh-2h85$3fqGNF`m%aLZClAHU0od@@BW#VGFL ze$ClsbP9nhzmsLtVyE}ZgNN07CD@E#oL_ZdMz1=5>YLK@%x%;oo~FpWNU+nk)3x*c zh28ouOi=O2J1)p?b3 z<#ii@qwf1J1W0yB_xN_4O5UnZHa%MmVlHCm1b2MPs=1m!6++8LOFX=2UsVSt8u=HL zsFX33Jh{jD`Bd>V*0l5$6&1^s)RmwXLXZ1pY5Au-r*{nOJAS@RL5d3ZP}xX29PcqO zb=_v)bFVq|-%G!#Rn|j|PmS-f3q_vb4y2=|L#F4axsShOS)1FMa6Xh@RSz(fXi|1j zy4N0}9x0p`9*v|XQve_;tE)*hu&vbR?hnm&=iJYvR@$nbG}~;~PM+rw7otYkzgcTE zn|j~8(yge=G*w!&I{_TCUdE1_>^0{#Bb_bIS)5lKn{H{(TV_vI-S?^n)=nN&AFLke zj!uq{PP+JOJajJk9^McA6QHaKjCeBK65hVPaD0T$#>W*)%F%c&ymaVJx?OI^49FkU z#~4QkarRZei@)rgh@aaZ-^5{+F_?*ti7`aDMmlzB+}rIYZqCh~w0B8-oOR=_u^WFl zJk{It*m>+$UY^~y^*+E}M0?k~PO_n}TseGqv^&;+rwEjr%eLfq_6U8v)-yX(d0m*c zcH01SzIb}`WODJ+>A)@3SAIra-w2Spxq)&yqM!mNkUpeL3nUY;X(uPfFs~F@C;7&P z2m*#QAh}B*kNG=Z4zr@d(-1y{EJB(f7P>@H-jsCfy&mJ?s{xJVWHU1tQ|rjpP~A{{6dqN1OXL)C(uq7)e-{$1QZN&GM&jAs-aL*)PceLqr?IR9@| zcqJw51!ixOrvGQPKFiSs?Eh2RUp4hXzKr*Qp7HXFxA^b1`m3T}{(I8@RP=wo=>Lq- z|C;0fGnxKR$`heD@_%U~G&)$amKzm|3|zvwIDddZH*)iCoYduo_XoGmv;Jua^o%Hu z9o^rG`6UemZkkvc>5wR)ABqwbv*-(JNUB&)RK=~-e@b=Ez^Blz?SvG9{jHIIOS`WC z8DX!(|LM%&jK2Lls)Akx`t?yzHWVutY0xug^|4?7v!WO9Zz2S!kUFR+MPF>> z3#I-;%MV$>1baG^e|cAk2OPa~P*0Qp$HNYAzDD0hB!5`wVItx$1}zRsb*sB;qhg0! zLdoSVsRr+$-;)lI#O)k~L}>X|^c6=?9kwC{GsmHv(NcV8!-fi%En^W@FZf1Fz%WG# zpENGXANa2ohp+~w288fq9?U(W&{>mAKDgo#MOE`~)Hqal8h%s$*lO`8iM1a5W!o4m zk^+Umtk+7`sF9e;4qCyvL@tZPW>er*p7v)Gow#JGuQO6XZ$HG(nu6rb!UiQ}T%wq+ z^t@Cn{qDV_|5)$9`~{G!Y|0|&OhruGsNmF_jJAF_qEyWGWXw@G%mVodg79 z!CMNhT&bVjmA(1$t(3^M?UlWi&{9@7@gB9&YBelbc9xiX9Q|v_>uKQsvQmEK03UTZ z-ExJtV|AJ3AvLPwuH3O``K zkk1~7`W6TiYw%#P5s(Q*gJ4QsdqaVH%hD&rbhZE0vBS!}GkMaJ#XTIg2mu>v&JfP< zJ?FU1LHR#smVb~ync?^GLc^W(qH_6!7Vy&3KoWYS2We=*Nu(K_WfXs-J|S(o?tA2(Qx28Lq;&0kp9ASbBJA~Q12IOqHc0*nVz$B z$6HbMFHdb$y$;r+gr&lNcZyq}0O3d9IfjoyDDolEXG_-5nWOR3SQIM$c?;oZIJ=34 ztjl#dpz22XpMmG3F!+y@bE2C6Uo!vpc1H3|)Po`im6@5Gob>U2zsc4b2!zA3x!M-- zcz-&m9G8)iIezc@c(VpNYq%X1np{95B0>_VXxKu~cfHy&ZGAmdTt(Hh~W>cua}$;fyGmg zn&0PF*UkW;aeDk%)IqE>9nrym16_RZeml{A-1JA1|M}DRv%ME3Wo5lf^~N1h(R|O) zw41HqLrKP%e35>|SIp+VP+|SatcR}xn_Lk{kau@?9M0z%FM@sq)EY>bh{3`zv)a$o z$&5~?s?Fmbyk4&^(it2mXynrJz0V*S#M8Tivfas%L>>||G+)!@bk&bdd0K5ze$>y= z{C=?{+k^Ukxq!Q%GgaWrj^^>k_moz(fv5n5fTr8}g#&et9bA=JU0FG(>+Q_?n8Wj# zWq4%drcG(Lau+dRB>(Ka4O%uyhR}z`4AOU`(4*LX@9X%_4HdK_RBU2 zS4CUx_1EuCJpSx_sR0yOTG6INc;vx5v-qoZK%6Iv`TQ+$O#F>HIL<&P)%dzAmuX39 z*e^IV(S-UJ1cKGwB{ku<{l~Bo(d{tK>zqjXE0iCzJ+!C1pgb|i(WX6ngKl>GWmF6Y zY(e2z{C^JBkchmtNLi8ydg8mDNs2Z;pfSY)aN|qOq3&q7Pq5n^O1>U2qE3Yqbo-Zg zMTY!b^)dR!UjrAsZf)@ss=;wlhc+S{}psFWZMnkni3a^kz+;C7{;KVzwaDcG9EYx8+#te3dV(l5zn52 zBoKshzJPVU_MJ1z>`43$N<-GK7V`j?XgUiLJ&i{AbIq(=P^sqk>nfbFYz_ErDx&YI zC#0nj8YA{8#Y7^IwI?hCkJF7_k;_c3LS*s^9zGHPRnLm37zBbv-yWQiAtb3UqU#!pmhkTK`=4~9(!s}6tgLfAPKG&m-GkYkanNzVOBbsUb8oZG{D~Mz$ zRyD>9igu~Q+mtM#qk=OjaLQg;F%+}Fa3k29A&JEDjvVz?y5`rnzC0M2!7%)=xj-eW zRrHRt*JSAl`iLf-h55D`?LS{^TM7pOY-5@GE~2D)^SlA|h5EJcVr=j&M$q&g8WG=c zLu0Gny*xmZB5fD%8lEUw)KLbIGX(F5f}SrU;X?3n_V6x_8RF#Uff!a0)$TkK+j*w6 z<5&xNJE!9iKU6Y#?bv`+7AvZ>6HGl8c~{Nj5kl%j6-$MSO?Eg@#y`*)a+=?vjc_4{ za`y+T58q)%5$VM2E)5E%A4v--wu5r#Ty*Ck7fZHW4G~m%h8C{AM;$<*UySE6L47}K zxaB=eQ3>9l_#&V^gIx!oA%$u}3hk>!B+xhv@bcC_vg zM+OHc-*1cgO>dKWJ|sztkuUv0-X6|aRK!fnuF(clxJ1g&feDawEAV3(YRjgQi8-CB z%s*GIsgHruXe^`{QhN{#kqseBh+RIJv$e!Og%FAYy4JqZzC*zv&6~k0`0hYDH{>?m< z66Fiue|4!RoKzKRjWpV*KQWp^K*rolGbd(Bkhuscn&Q4$QcjH~@XxGoN|>~pQFKaP z`6Ov=6K!EnxY1=c7juuCj(jkmvrT1edKImlG}K9#bFIwC{HBvA&wF2HrgN~Il8op+ z>@<(Me0z>__`}sS+da(LUj)>XT|07V;LKv#S2jymOVkb6cy7(Vh-Yf3K8kRK9Bydt z&!0-1WCmZ<*lAeM61KDZo|l)GFx1^>xiYKQ-eA5!UCCn3>wS~Hp)-<1r&?zKQzV^A zOpe3L{A~?D&0v+m?My#kq2^yvLEHYcpKg}BJHpTvv0~kY_Zqa-w((X~H>TBWL*#W| zpWS2CG?G_sg$F@Z@dCqfnU27%Go~4ISsz?W{2epBiLBfaz2aAR)I>KG)SP76pT|M@ z3Tkjz6*)13Ye5gU`>$*RqOCJkiyk1XL6AuwRurd2(yOdn-(&W`k;}xmOgx1b3tr@v z6~Jzwqg?@t{zUeV(mX#CMA@A-V>-sMx4Ls9WJsOgzRK}r!7VNjb+u2$m*z@nE@;nA zRD}%pdKNSQw?m5?V-S-idAgs)cv-TjNf9VaN$Syf(ln!Bvm+_46?6)naMKG9%-$aK zc&UD)DIf<6(Ym-GeN}VsS(0|STS!fbK;VMMIzmWgis-PVKQfN&Va}?U=<-|sP~Nh_ zX_K189hV|TvjrBZO-!nWz75f=?qK0{AyAy$Huo#Opx|YOR7Cu6_Snwq>c0 zK0Gh>A2mVEXRs}b6rGQ?6V8vYoZhHoV`KGqpI$_d2w|F?-q!CF3kMm4`NP?gvV8&R;gU*`X2r9 zh;%l<|6c2Kd-yXF6A3XwR-C3S5`->bR?Fi`C5X zZa^yVu^&OO(r(2dqbP!J`ZNltoqnwCS6Zmr$6RWd-saKdHkTx+bo5sSBozONEt)}3 zuG+)usC@GSHq;cDk{KETBs#0;SlSwT=K!#Yi7^w;qB1m+BnWh#Y-*~C=><#Wya18> z`Kj_NS<4Fb6F_41u=q}YoG7R7h?K5n6SKVAymb1El!o)6Mmexvd{E^f>=!rBvXwiV zdAS7Iv1t@h^@+lQgWt2JynN0&+coaMt4F%BfiWvk5iL!#Y>um;NTLxov4w8TA^pv< zk(Zt~`gpVMt}jWu;8#8I8g{mVe9_#(f?31d9KbP9b0k8)vPvY9K*i*noK>=vepZ}H zLEYS&k^=5aH&%;0qe7uhcMxHL{)2_#gJssXUn{h1rZEKf>O z-+hCfnTW%sh6N@RNJp+0H6kKl0Gt3}Nl8ijjs>KJCQsw=4TTT@fKY~!QzK2N*D1|1 zWqEF1$g#u%yftp-9jggMDiuJ|3FXbF35!z3Ev;?*UIF3v3ou4WNePF-@V$H=ex+#LM(_xPFA*hDRnCE_F6X zo~q1hHqNj!vPA1i-KGwp9SI}Wy8BZ`-lb>spi^y##|$Kv>{+7#yOC8HwkYEfC)>Pf z3oE%+r+HJOc)c14)3Tb>JHkwVIt=GV%!GM2)~s%SnnyC z)qMQ@a+Z@-cJ%GuYB2yISdRLF=u|Kudgdal-}siR_S^+iR9f2yLCk0IH-f6$4RS%1 zx?gC0lWWKLUaz0uNlBLDAT?v>dVWZ2Q;c}HtE@Jka^?Iyjnz{V%tghDPZQtOUZB}X z``Zh%yUtRxZT|N!x$Iw$N5f^7&_h&qq>#bhQFXjeZ!v#woC5hS(VeVnNN!?VAlV%; zS9$vCuW1azBvMgccX+(n(+3wIA@1CG;5UXCM8l52_43H|DO(mZnym<^ac-UOtPLuO znNJ6hb2G$Q`nF8^fro~RX>aIsH&OzEg5;-dDqC9>@n=mNKMOGvS<5!6BzfA>l2i5w z&3-VZq6O&nkh3ie_S9PgKWLuztDQ@)*~bWqrIAH{TnP#5^f5!|e2LX1sm|@9n4FiB z+uO1XA!AOKJVs5oO__15r$aSUjZ+xtd`#8IP*ME>hgUg!U6SC>L9nlTlvomp^vfaI zB6Q90#PPIG+8D)xD^6M77xH_2X<>e6EBXV=)1;+$lO6fun)GGZk|dS%&vy?=hUh;+wrpT8we_R!CcSKifewy3!XT(ANW;vWx>*JlR-ByB&oB$?P9Uv z9N@y5(+UKEA;(P;d;41K@s+_aDPGy<8B$M8iuT0(r*sitO?7{@9JX|DL}ZLP!>2*z zz7>O{N!0B}Zu)gD8n3NxrNy2?d}T3_7*J@*bN5Y;4Wlb}8P_^xjOlfaM zYP#3)aqee6hFm8wGeItAEzLDeGIg6*pH@CC@W)WlYhp8K!ui z(3Mu=zAl=Gt`y5c?9B(O<}Jw6wKlB3w#|a!*ooGc-mg0ZU)oisUpt9wbo(GZPSiIP z<~AH=WOADHhp{@U&q}G5rSeQ*&+o1`b^C^3qZy}OuU>}xdJc7>E$g*9zg$e&-+Ynd z4ot1nD~Me2O(}elKj4U> zl8?HKQcfh53#;G;Y0ju!kiQbmXia7auJ&euF7O#=u`xC{1WsiqElJZ^WpZ3Z9NcUC ze!UcPR4uEIze^KZ3%UKe_7dm22e^>908%lD#VpJ3O1a`TitY(6%V-KL-Pfsui7NS! zg$?G{s2N4L6>WFXjYL^~_WMKWw|!&yZ>AqfEiPUvhhBx~Z|w zk1lPQYi-W-%LHDa5H()rOj;XrmnUhXDk_POQo;CDI?|c3i-y#5H@4b<1HC{K(&AXg zhzE4I&K0$<*x$eD68-+3p&AsZ-^iMU@8>gv(U57F=t68$D*ver$d4o1?|B|b)*GF8 zZBA#`ig0Un`$p}x&fIGeeG~JeH(}q}E{P1{aj86{%ODWPj7@9BE2gV1dkM8(e->Lg z*O-VK?((zoyxbW4Q0Uy;ltNDT5ZH4RajqaBqguUG=P`T#UN9q+j!lv|-BONh$7bji z<27p0cA2F9?woUp{akJKMsF0zzO(CV8B)Y%^TH6S;rlkQJWa`2!_SBLg0SqX_=1Ll z-Ie{WYqtt(u^u0YF>XgJVefEz2s&}FXz9cD z_1b`-;YS(XAh-+u)p*diu<-DnUe>mS3=Ui3`D*QI%%SVN!^SUk`zvv}#d?j#$c(nKuVI+kMytWDbNTlxLF15id22aDlt`^!Fhg5(qM_4E%E5&f^i%%G7zb^N(L7&6x^lKD8pO1lEkFhQSgK6DC zAbZ|ssw1$M0gq;1rf(4jjnQ-(oJ&k+DUIWKOf67InV<=pRm8Owt|QoLAfPtExR=y1 zs?(beCeq1(fGacyMmw0UCm`b&IjIqz=#$#{gM`7KjVQ-JQsVx#Ew6lKmm=_iD1red zvwW{_g%f-I*<#LcqGRp}!;ap!M}Dkb#*$inS36bLM;^D7QOP$99W=m0W5xsO-~49$ zmd-qyoRJ4Hrp_dhqz8^a&23_yZ5m%G6f?*+?x7w`>d%R=9>x1pKJeNYIfYn$I3Oe|!1zdu=7Ralu1 zgzLkbGcK3(8P;PJFNK`}7bz55GW61cZp8?MZ($~tSO~7CgiDPvH<&Ba37Khi*bd8I z-5A_#>B^>nWyHL6o`t~X1cxzY`HDnrOvDDWl9{ug8d)XQ|w zu7J>v#TJ_&4gFFP>Q=a77DJn}cBf?)JG^J&EQdO2QY~35rK2Y@fQ0Y3?}$tjMjMuN z>a+v5=o)<$YrXQ>c^|H;tLt)Q)B18ST}&Ai7^v3b^ZUz%xpl`~5+EZfK0cpLt6eoe zQ*OkzRcZYOS3d{vP6cxovO)1;9=$R4OLHhf;x%=56IE)>j1;C=%@5Y(BXTKJ~ z=^@ZdU;KS)ZL7s=BH_)_KWXIrW)LB}{O+8`LF`YeX4RdUsjp^6?!+ozebP}`_H>iA z{)nN=igepaZlE3Mu_A7}=FZW57u{;t&lL&B={$sI}f0? zoB_p1tR=FOu5!1iI*sGvrKPN6RJu^C*w*8{EBmO1k^Qv}#u>FPD{eVM*2L1zmSx4C zF;!kCnucm|*T8x++wzBRJIYEJ#^p@*g@)<}JPk35*7}ueMVPH_+opV~Gc0YoRmvEK z=C>aErretLX$q{1UQx;BT9Yo=Wz)EG+YC)($(!a}+QY}zrXD1d8vweqBSo#3OcK7_HN>aJW^oCG9U zG=?P811Jscg50LU+_E1dGICS2K?Ir;(aIzDY$mSb-a75lg}wb?EDS<+yey0GNC>}L zy%!2RB$t91EmemO)ljOF^&J_Sr?7q;b&A|ICQSNKV-|7i?W>{B)xC9mEU`AZbs3e` zRzqKcDQTrhF3X%Xb-h)J7V+bBTq$%O?Ck6$#(1n)HkkqiWBS*1Og_(r%F3&z#dj)V zM`X^b+Rha^$A%#CTdmQ#4lARtz2E1my?)g%Spe+q>yM4fy^m!Oc6++jM;%~Xhc}o# zjn#ZE9*MLm#M#;t=35vet>)s3VJ0Z7hY`$T)#H@6q>RZW{`cb`H!t=Z^+S*e3!Z4>p|^i^%D+G0plL?tFG>Qx0NrqL=TD(B@Riqm$dl z=Od3}xhp&6&StmGTb=42PF6G{(0G`u9J>nVHYOq+Fcn3NwQlR?N37zk*tl)V zfMCKD*GmkuZUHz&g;tjuH>lmYOF_JpaO5|F9f-H*bdlHfAo>zLA^#n zQ3^MPD@{a_W~Ft&kQ~?;)CH{{C#v zu>r~~x{1&>z zjYLkMa-0;rsdeD!(@(ip=6znf@G6$(99ObT zv}|7a4XJHx<9lj76q0Z%fnbZLQHM~?WV%Q>k=K~CBt*U{(KyboYvKJWCw*U^fA}>f zxAN+?mV|C#V!*Dv31uqDRU|l9LPgMZ=E>yX^%joz%q?uP;afz41V@7E{ z3`Ah(x8v1puHgByqi(zY-s0YPin>O&Uvpe|(sVdXSsEPu7S6Ob^QrqNLXG31Q*D_B zcHEmV0TF4F#~)(d&RjlRhfhd&0-14Dw}_xl-UI7ASzgbk9-m?JMUtd3{R;ev85%7b z7D<~9^LeK&{IT2zX!OKZAU|eBij>^BuJEj0?A`;p?rZo^Gtn&xk7Zlr+p{!xSUgw!m_gA6UKFOIRSijs|*0kFxYpv0S_a=q_{wXn?#;b93!?yX%y@P$6o@Jyrne z3s{&`5pm7=tnxO(QLR_x(%gku?182=^;^;37eahgQ+i1gw&`W0dr3NP7G7khgcHT| z$X$$Cvy?y}d%OuLt}CxYq5y zBH@@@D;w`@Cc2YY)AFob$D1akXpoJpBpb<=fuhSSMWH+nL7WIUcog-Fvk6GzXy^sar{!+xIzMU;jw9{5>8s2G7Kx>5a-dDhnhW#>8G%Mn{b8aGPi^zd9uiw_yF z7d0o-ge9xghTrs{OO9-?;~p~MPmp~0!bk@1Uv&lxkq^bd!Tx2l^QW17DWy8|q*}C! zTTQ3(Ed>J6clY6)S8ao*?L1LlM6{PUqKtJ+W~=2jyus>MlDa(UB&Qv}_ZAVIODK-iktKPkz6u zFV)~gJ9QE*X{|Ze-yvbWepP-y3J+qtzsw{0Siu_Pbme&(`#s@x&&)c`vu@Gavoq0q zQ>s`z+rIfUl|{Jr;$iagY#F-fu}K+qD#vC4=Ae4!;q=;V)X;+TffgdO>Th=)p$DG` z!kvh+{>VDCY!fcG#A5Fkd32`+4V>jww zBkTYs#HjTE8_EFTokw??9L~(Rj(x*JF>}N7LGv%aJ2p0!F}4_fA(ho!qch-L!e`8V zJc}2uZLJFNzOzS6L@nclfKyJc=qaus{fgE(oF!UTnCCBN8^ z{!elBg^(FyC8T3e!L3CE=MC!oSCS@@LMY?A^Mk=nJ%~d7VEz3TO z5}sOgt-(u`H1l|2{{3%2v*nDb`_W<>K}aq-E=j+P__L;3!1nrmg){On0!?SL?f!Js=XzE^Zin9=hPd(YCd1fdVjJQk$7Tr9A)q|()na5F}`iEo(UnN|1``0Ek2OzDG(#KpQy z?XjagLfou3htwwm(i?3z6v!}=*#QC*$k4kN`&epRbIicPSwUeR2nNjFN5-HDz87g2 zOZu%yODnZMi@qaEe{cZdH7<~(Zp69T)t-tVf-eAA^k3>Y9)v|I~OA(zs7D3>|ICaU}!$~-QOnCYs8 zg@vI4LSecvOiWDV)0Jwnb>{bRCpkSs!_nJC}4uH*~ zji?jNfEbI3K_-i+7dYoq?KYl(J{3-1(c!Uo^YZ+&PvKFcyy|%NJi4sImT7WfVqpRP zqO~6kvD4Dhvb0tOhecmfOZS|aIGTyYmGqlD~#qm=*PseZmcd?1uC zm7a8NkYT95=FRheJWc3C@~!MOQAj)I2wOGX2%_o5POSVc?$@z3 zp=V=A4=;K^)c-YVhMGV}sl}wY*TI_97^&R=DitrbQwp`eIkVlNS~J{YUR-}$HEXS% z>{n5K(HCIaq`M`Z-{90HlL-AcO?9aIc|iQYs6(X%4efhQ(QW;?z+YeO3?yE9T<=O4 zLjAR=!|>p_9wgbctaa@%d+~LiC|Fcx!!VeupvhJ@a8%=l(%qV0h~f93(_^Czwv1;^ zO>JzLko{v(Om)X%j`Mv3sPA}8{iDzh2W|SM&emXv+BoR`bBp{L^jp6=lbd#^=-=7l ze$oxZ(%dk^5`T5$o` zZz_c0Lell=TtE|kHm`dlSG~8b)5C*^^8)-gv+4LM)XTl5(1Xm;+N!w3TcI3Fu-D8T zh|Nq0?auVf#3}6^k(Bdll9`DWj6v-%uPCo>mT2}_PPbdib(b9k$IE>#i-pnjzM4xO zekcT;4dH)aIOSRqYSO@by3j|+0WB|BnXTaM+FqhD@7v}f>2z&uo|Df5(ch%Kr z^+*MW*+#7{|6uR}L8^LZBNny&l4kqbP5;fUt5yVWc3#*1s_*g1C3zdEogb55qf7Nk z-8r{}(+ecEI$B_RTjzB735T)WkfwAXTkALNiGs)N^-YPD+85WWREGm$Rm`OTUTpV{ z?WMkn8j1Rv$MwKUjwi0HDyLyJXoo}}lcF|B>1mqJ;S)e5yvhmcRX9*}_&<@YPLjG+572ami=iYE{d?204{?(R`R2&$(cgqs9Fna&?>5PzwVrz} z%8R~m?P~;vsdDWRY4R zg()|z#*M#|SdF7oALvIk@%UDT7jKE%kzy81&kO!5jzaC%n*TP^d&pT{`U7sd?xL>& zYb3VO4ed#ey+_#=KAfC|=&$e`OQFR!DAQyp`z7!S^K{t_>%QEDkiH3BZG@-m%K$Sm zO2au(1|44b9eUrxW)^!R(_*S$e$RSiKf2OgF&qI!FgS5xhmD%`fVzgV zI*l?kfRmcPvh_rPdA*VKcXW02ygxsALPr6qTV9TWVUoDUg&c~l%I1T4tV^tLd$k73 zkjJ?sL-5cnx!|8DV3rf zW*0~-radW1K92TL=pCsiXLSa@@uA6~7fwauaTrwFi*ndd`a8i%J~L&x@L<(c3wycd zQz(9U;wzT-{>k4)Obz$o+xXUlCTB^6N1y28!!kQQ=~9MV>X4b5xjA)er}1dP6PpZ1 zV`*Suk;JTCJ|nUWK9msS$APZb)92njxC}RL*^hl`W&fTsbgvIXeVn<(`COowbvuCVO&#B|AE|1V?8X}nsBxO7$DI1|rC$NTluhN_36S-lG70OU6fElRbVLxDqv%`%*w zYJ-~`-&aFQn0GSz%JDt@aKB$NZh#{2kk8lrGFNN2Cuw=H-m&}_`NN>mSq(9`dYzb1 zU>@$na8umDB7@;7nGTRHn|Vf^PdQ$mTP1E&24obQ8_Sp$G+0eahF=3YrWw<>ryE}L zni{lY8ZnbtDhn#4X@=xfT*s^`Z5on<49Rv?O3ASbV}p|T(rGJ7Wu{N(@DVDcm^Jm3 zHtgt&+2$)9D~DAK3vBfz@-w>k3Tqlyuqk=Xi^h(~s<<7h=BCwXvSO;wG0L4>fNS>_ z^UCT#R5^QiSIy*BBb?a_KR+-!e*O*DMIbSEd2VLf$*DqRB4>GJC8IkWQehGW>aBy3}aGZk^lR9l{wi&RN{tEpL+ zq(v5*D__oZ!0B{R`oG~HC+Q!-6Z*W)B~Cm|=wPBwLwo zm4XUBEn@pa98O`q>4p|6*;(Slj~4J1XfuTEXQs zb9S#af?4AsoQoOH(6=pNT62cL+^CW_dPnoe-W66Hb>|fuwV>{jV;gbhz^trN+j{!1 zENt$1*OpXdm9*2U3yrZ%=46@E(fws46$54N!d*`)W>$?@FkEfXlLev~-AruK%4(1= z+yR+r*lp!cMafEq=}V-0_9qVZIDLKcrffIny5l<*!rECb79KF2MK^^N?@6MgJSGlg z(7ruFIb~(Aa{O;B=N%7P5#Kmbqj~RB+$vMEWW%$~%ysLRE(b}XeVUGw2*DvSKmnso z_WpRO(W9)*rS+Tp*jan7M2gbG5t{@DON+Uo9+vX*M(DbavR+II(iF&Otk|D>E+*SG zJ|&jJ>fMb&fQZpzG?8DX?qs=Va6a}-Ub3$Fy&`aw0TcovwdO)AFt}dU&acrwKWvnF zu=j@=o6`2u+$$B-PZGZ)?r&ib>5Cd9P!pY7-_cAoN7K933$3iu_+>pW#Qqmjz=85#Z#MBwe&5XUL5%{{O|)RR&buJYA(j zLJ^P#L6Gk5?(PQZmhLWTk?!v9ZUO1;7P)kHzrU-Gf8P(>56WFOW_M@ioH-k>6S8afT=QYDkrd&aveAy>A+ej=I z1xmHCA}aI8$VbClOk6RXVIod)i4Cx-HL*bHm~$Stloz)g;h9>vTC=2GdofXFDHihT zV$`9jdo-BuBxzZH$Zd_ARnm|3+IMzq<`*j5%A)y`p^)Jl5IVy#8Rd+6Zy}9>&Iyh#&5i5(I(g#q0TmrjD$9Q`(Sh z^2WI#|6n9CN-n1#BLp3?I@CTBUDUl;u&b10*AB7-L>m<^XN-Eqa2g=#C9ktaZE-c7 zUNj9E?@C+L_AoHLU5Zp}(h7IQ&d%B7ueh)UJwThB2ftTK7*5T=vHg4xBNLvIACz;H zD80Z^fEkxvoDE@Qgzix9KaBD<`BY+wWspP*Wce|G{R08Ia%@4iR2t&sz&pchfWZN? z3TALPzWuaap^0$GTxqrRVkxl1R(hzMQ*-`^oCJc`#Qj(a5kxiivfP5z2b(1ueO{`(-o7Vm*TRRkPLPLJ#80MD4_*yM%xJK zn@)+k*p{I0i-jdCIKZa1@ogl%!R3_6&X%M`oIHYw5}qEO2-3J5l0V1Kvt#WPKtXR0 z%8EUa8n9rKLFH2RkWpUtN!rxPt3_q5_^=tP9&k#wRE=U(L2ZIutOUsvdMH`ejj7IV5CE;_MYK4 zSe3waf!Ur5uf{h?-vM8$-J+5c%v^1BSzm*Mj67=^=H~ePBbvDQA>Lj}w-Bc6$EN&< z6{A(e6lTWmmPXwJ!wPb9p0q{TN)z9Na>qdNg2dXK7+Fa#JLY;5$DESIoWk?#{+Ibo10%`}p5d)k93e^&71)8l`PA-$?##V(;>v6jUPyD70`?}S% z;G;senjwq^m72UN>B{IF^&y8D*!r8b$YRElYTu+{wW>R zTG*I1th_VFJWoAV*Z(7CF4P8!s0z%%R|Z9s+P7eFhS{*@2thHu)CC37nn7THa~P$C z;B-C`Qc!rqbAP~C@c@t{u)w8h4?snO@TX7Np%%>j$vErw~i?1B+>>#TA`iqpOnT) ziP1PH=M$?~%W?1uZ7ZV%QIq5X8Wx^bO@nkDYw6Sb>|DaJO~bp>45k@I*}VE1@mok| zxjOs=dr&9WK)xtx)|Z~2S z1~`$d+71tsZU!kUz9X?N`vB+(G8&nTWrs395s6sKt9Pjb$&ICqjh$hq^P~;J4(@?f zU<~*j{N!QaS9s@dW#`g#YTC64wb#eN0s3$8AQ@_z(}N;waftE@6>zJLlT3m`@;zFT}My`zX zT?wD-!{`p@rlY0OJjpr22x6fIm}lyzr(7I1vtSNvEHA!u_FyPPN`eIDw_5W1X*&W z*S?AQfm?Mn&Ck_zEyz_%j!!eDPT~ExPHfHQ`>FqPn5XM4n>_R@JZHR5@w{JxF+@Ig zwN{o1QD4dI_cWCET=|}^Fid?e4Z%TdHy0I zw$ZnrwD)MP9}Te9L}HDYN3}c|+wWJ@gf`$*`V%YJ_8SLh9FFiT=WF4EW#dam)LaDP z&(5Y?8gj8)k%Tl)3lLfL2K;s#?U&>QN9@R<+@km#9qZ1jyTTyNG~LJRuUwAXuD@nv zL>UdmJBoijYEzMv{%xx^5ZOsrAfgPI_b)}^Sq@;%OqD%OHll0A=j8(Ic#~5mq!6R; z48OH3z4h%013`)l&o{)egt67UFyn2(-6W+AF5^O$S3Mzxq!E4IQ+UpC+Q%-kc`>dP zxjj1(i6jo~VrEf`-!UAzE9h5ccu6!pj?qp;(T26VpkCV)kcGRQmoswXKqa1vG!*2a zRGtwGUsf}KM>I|h1y11|EY?Xbu%26|QgL~2QR?nSH|NUzo^)9>)DZ_+wD>dkRKD9| zr*7-P@7ja%l<)NOLD$~&J7%FDFy z<_S6=Qmzg40~vW3cpyci@J~DTW#MSAD>NcqGlX2Av%GEuOx>(R%!&|+d7|<;UvcSHuna&N!_m{hM2%Ew}HR8 zS!=byA+JO9y@H?Cd;W$?@~eRV`pclV6j4GiBt-bIGqRx0-6g`B>-Tn`zdRecsyVFTI#d z;8j*y-;vGAcBS`jRt{poYF6RR&9?a}(_<_hrR~ zqLX|T9pAS^CYxE4c?w9Z)w>QpKhLEnk4hn@ojDnaq&Wrd7$*mTBcFskA zGHXSXt^}sEIHJ=>f1efWIqQx`rwA6AiQg567}!@XJ%&#gY@dqw;BJJ$p!g>ce-5So z`WqKw3-qsN@7^yb&$x9mEp84F-$;DbNgO6H2cniw&cBrWkuKHLCdHVz9W;_mwYXq~ zq;L)c_*;r5W`3ZKMIv<(Syu3uw~X%R6BW_Nn8kc~{f@ssb%y;dBUCLsPV3V2tA8j8VYLosL^4yH>!3x3U-@d4?w1}1JV$be z*}*WeOs3{3&UblJW3;`T%vY)3!~H_tn5O5l(}$&t4_re>>JJP?AMD;htO#b-*JA)c zCY{(<6|XGWRJND(5ymwi;lYQIFCQ^jy(D5sZ!c<-t^VMJUBbv->m0uzeb6uCC9{MG zCX=Eb0=q-L3nBl`h4zW6p$(Dw8yqFiCL<*4uDYyZDA#&edT8p!Y13dM(~tFdWI2W` z#SF574h~#m`w@)WwCjr%B2&^h#oA|;6j+62#Y(DLCicBHddWP^61}BB`{cq*42=Un2fxrOrq1TeBi&hX$nx;RoIiX(9b<)AeM)ak4bd7aq(`in z<|V$iFd|;gG~Qzrbkhe}gV(O~wyD23oiv!(`%x=&jVB?7jP}0G*TM;kh$z%MvS~Dl z#7(`|;6Zqq0r&89-JzONTB_5AP1}C+-fti#^0!h$ryddEQV2?X^byF*sh--@ z-U?xN_KG(k|86>pWhV8%f{pt5=5FUad~>^NW&o{S$y!LZE|=2 znA_e?^7-mOz4h^G3BU(?ehtKQUH>V*FN%c4ci~H0&*fQ-aP3JQhp^MsBEqlWQe z2qv56&%1et$GcMt3E4F_JUjJD3n*|m477j*21+x7X>w-^mIf6@UFTPBCoR_x>cCZKa{(}x1Nu3N_>5~YUqdn3g2ISIqWogayZ8Q zItg}y|5)4#v*evTJ0`7;Quq0>*EMJB-CoH@N=nEj*6Net{)2^@*Ka242@G^wUjNWV z(8UsKPA4L?FEMF!s#Chcdz?3jA|fUhZ0g)Me`=Qia{LuPB-;cs4SwU!`puhF1awt- zd3kypNhq@dHNi+un!T_~Ts3RkN3VPz@^)lo6jbKme=P5D)EtezCLoC7eb_)TXBA|* z#TMfSjqzbqo9*kp2T#Z;h@Q`++TUq)Xl%?GaU6)lBtCoHYy_V!+mGqbSL#F7noL>( zCU%z$Ca}=P#>QT_-|w`8NG^Y~UJ?OB6Ly@dFzYbiiUsHC`1whe8aF&LvJZ%|vI2I< z74t95Wg%|iYgG9qjLP7CESm=kSv&ap3}?`Ee#;fe#QbmFbyCHilr8}hoi_X!aHeE0 z8hzgA4!i-ec7(JXx8ta4JI3(}J4vPkX?F)iLh|o65DX0uB4k)0KEnh_Fanw0Z~n~s zM}~-&7UlWA(4zBwk;Z?8yv1qCaMK|2e*z00i(Pft40r5J(v@_SoiJ>*Bs%_FrFL z=do1ZRtERwd{r_u9gsUJA=`!VT43;o@D9%lhK%7O^kxMBeSda!9i7O71IxU?cEEhV zC=iuyJFW+}oB)`+F!&D2oYTnq^56%DImHGXR&bTR@9wZTNOo+LfLt5yMiT5g9UfyI z(ozr1(Y(A`8U&~fYaWc)*w`lkV-o{RB&xDrX@2=ZtYL{i1kCpIMlTTA_p>_fSpe~y_s7MY(3vC7Y|Oy0%E21qia8vr+goKzOSPxtu8~C&TMhT_AkHF6!w=u@uQYZ$V9Igx&I(LZyg0 z>JK%Y;|dY_)W<*FxY=~zLUPQ0)Sg3wJyc~S%!M3Z6Q49`8QEVC+5h=K4<m{eywb-dgW43Qetd+Qn7UOZj;GfFh=DoMu9q&vE!Iwl{rR@@DR0QUANjvXFGo4 zxrrWQ!YtxT-QNZb22EOsQ+d7d&CJYx3=bTH`-+#x9O5?N>Y6t~P44 zw|<{byWsQ8b3$!MjUPcd&jP91xevRvdqNhFv)mQlTD|t^l%yRes<$Tplh!cl+xB~? z+43;~hja1oFe#{Qw`QdgC!@PG0yMupBY{jwD2U56iv&WC=4ep8$O@oH zd6g`yrmlD_xLV(=ufq~TFXa5Ir3 z+Mq5Q%sS?Xy7b*P@MD?z_X9JLnlNK6C5m5RhbGoHWTUkN`;qd!jl2WDd%y?hzzRS) zEqt-hFQ3<8c{@ug+Cu?8)bHW~0c&vzU{`cNsVNTo(GpOhwv=m|%yH|qbS@euj1S*- z_w>*M>R2CeTSloqo;wN&1#r}4T5^F?r`{=xc^+BQ$jD3A?J)bZfz~NIpuwDZ;VS~r zg=Qy*#Rr89&I&I7##bVPh&lnplzzR;I&($h$l$|+Zwb&WvrdsLX#winZUC<{ z3QW}p?!Csn;u4L#XdK9>&VNj!SFL?0;~1Ejn443wyvh*t8JhUZ)Lu(Hes-RoZIkbU z0|Gu~&3|}zzTmR_xd+KQD=84@KbPe^-Bx}%tDGa4ba$5tU-d(lfrykDCQQ)2%5_1D zsm=La$tRVVOiTXL&?JiEvMLq585k;Q{D=blM1R?^>rY)28jYrKJTq&#N*gXuNv{Hu zYy^h0T_x05-x4Q)V>y7`CJa#SoFT%QwW)*%IGv;p)!6LwpT^2!EZ=aW!ZD=_?ZM!D zI=F`suz<{uXtjOa6sgorn{lhRaSe!Ysm{^1o+AJFrkm#z4SL zhb59qeQS7J=TR=DmJRKtzGC#(Pk}~LFmI3Oa=;=(ElMj#LUzDO0ghd?i9W;M!yHHi zb;ZUro0f(|hJ?uDa|XS_Vn}RW)7N!Hi;T>dF*(J3 z=p#g59^%f9!)IW@?@OQ3OVxmi3PI=VqZPGU)7M_NL%N^l%C%AXoAKlzuj(N?7AhfU zmyMbUrCSCL0jkEa)^?qww!w!hvWn0B!u>Wq0~{;;-9?WfYE@Re3_pp}P64I{SBqg2 zuPE`pSYGzDi#cz04q9|v^bQTB?brqeCkW4>MZi9mKqT8;wN>(KqT_e0?arqg^!Zj3 z6csk-<`xo$*P<32XZLh^u;$}TTpump_C?}_RUN`jzIt!@wpS7Rb?#CI#=}^0pc#9B z@38L^Gc1JiSCtMCrUjCBa9}+=FkgVhpzl=#iw^zwA9>L)r13ZVRp!!q8wSURSd%Hv zb|S*nDOGSfXE>ZIa_}}n4>Zg|q=^xgk&IXKM(b))#fOGzXC*?FOUH0TK2j_N9jPl z!?62kT}Rq5ATcsQL!sY9#%W=#G8$pqUf>Ffs#!A_MsP5WXuh}|r8lc4`@p>t zld9G#pq=}ezOhj0&}{D^lT1F{{x&L!U_O?pE9P4~NWG?VN6qjj46)Iml1JM2EgUYa zU2=TJgINPFESF&;2)B}9Px&OKm)hb8!7m`-2-himLy6z3%DZrZ2E4~QeIdC4(xyKd zE9;z5U4@3DEXGOdfy+!e=y^Kj&4|L-=B$ep*TKM&KBR>!$dW?4^!9Tf_E>5ldAcWO zJV*Z6^d>LAy-Gh5KS;aK_4X;&;39~y+kFuz@~F>D(ZSK%T4 zv8w3$6@*D{;}DcF+!`(i`LEUqzBC+$&(gZ4{>SfR7dNuE@^V1Aak#^3rhaz@2`Mz} z*lYw8_PR=>((BV-z7hHWqiEFj(Z>gQHyx_zrQ1b?hE6tZr|%ErjacFd(_EgdbxU!x zQ){0eH+N$tXnZ!UC^8Puc8jG+eZA`Vm2~w@*@@rrNYzK}xfTZ{M}JV>ZYtU6`;_=_ z8}61NVbDTlq`A&3RW4UQiMc`3EEOID?X>>!){+Tyj0&0URErG5{aky`Nu69o%P(>=eKMzZwP9Z9R}oies`15hff+FRSHL)lQ|(x)d@gp+(1bt=7Y%| zkV1OTLA??RenG--WewI#1rkcFTFxXT5?idIgZj8X;219nGp{~Po5*iJK$wJpCZ|6X zu=`polRk4>p9TLcMM!lWZ6EKNEUnw_DdSl?5bYBqfb3R<0$vt+31zO`dD&H~J-q?c z(M`mXD;o0F%sBgMwVka?^-{G}8Nx)JeBn>FF6K7>?qD^{FeGYXrQ~5)`mF;&HKlxx zeBzZ(`m}|gip{{Jj{U*YYe=b9E)9Rlc%O2Y0TW6MrWTv%vSX71lV(%rB%zgx6uqm_ z*-G-T*rYbqC54R1N}Q3_Zexum9VsIa`X=5>aG!iRFySr+G%eol0Fi0))7}0|i?a>) z)6FKc8#M#UE8p&exwkDkizf=I84z`<59M$1-ZAh!a$?d3yk^!0YRonCjbHRnJDS}Y z3(@voNKkN)+0}k?9##y%@~avd{fV<6Vl*4W*yX%XR)(Eoi=QQf;(~UrbG7r#M9tG8 zujmSE9pH#vu8jkttoJT$b`4{)&T``B_#XRF6-gSSoqIE)XC!HB)I{q_!$T05K=3R+ zBQ(0Do3*?f!qoNH2#>y^wRjvmG_-%2ao~~$j}Ef%mhsi7r|NKL+ki)0V;F%w2$--k8r;*fW4VT>qypK8 z54!+~bqdT`tByjUK{Z7+hS=SNG_o1#8D?-I z_(t=@{%B&+=_Y#KBJ?s!Q4|#-yd*)R^J?< zvDoAe2n`+cyNJJ*gS2~ge^0{ik4PU2xBNr9jWiC&$ztV_=^7K+H5LfNDr~L$!rLzM^WeBn$!atp=qgqccIt66haw73K--yjkA~{w26vc;j z8U5Z0VB5n+|2#q$g{kAh&${BE81d;F`~Sw{(*mVMKitFPGT-cIak$#1DV@UNsERlz zXLYOSfzb?0aP3O54rwXIkBB%22-w9Da3)4N`ZfBGLK^^}%?%UUY><+vLiBMXv zrH+>xWXm+_5b~X}O8?CkKjC`l0v9%mWuZ;SqYB``@{f~r$qN~4!9_YD^eNccKpT|E z8v~7Wwn3EdB+*b&#cD@KA^orI7cadL8sB$YE;m*kXiZr@y%v*}R_^e6e2BqnB3vgv zdAwfR4`&=bS?Ce{(^m*hA3cuBB{D)uXFyennssakG`J(Ow9j}Qeu zl4U#5yBq+aJtG1E!}F-l^)7#OEUY3xQcnxOJ|nytK48uvk^rWjpJF`;DA4Y@A-?!C zv!dWH0#3>$5H!NbmO$}~V**4#8tvvn^%`S0VCMWXzrUrIg9b8{CGt5U za~)60LkYC1`?F=2Kn_3!h;H?z97n#wI+m-et0&VN1c41h+j+%t^mCSWi__&)jQv+`s4ps*oy<+wWz7oLi1a_`3}*(;u;MN6iF<)PN1j5kP}G90Vx! z4gs+?3ONTyGf%w%-|WDX%PEZf)rvJ>VvdX5?Z70b5U{>nuq#mc7}cK>#rvcNG+NGL zwfL*m>1er#vb?eTmfy%*Y2kF-=?f~(AN#%(?qu$|cm{G*9Gk8LuD_!1CY|_Iqk!78 zVfe}?poWg~*eTpH7#VLmM-*jy>~ZxPXb|-?w9Ect$JP8*jiAQ?B{#SBQoWTT@N-#{ z@i=@tVdzb_z+$vD@xn$b^tWFkNBwdRYAPuyDHOQQk5*b{GWq?@vuvxqp$;JMuDaxG z=#1B_e2fd0T28tV(-P9?zqcxC5?#XSueg_UA(#cHh>h-vye@;#v_(syi`Z8nHL$GWolf%xq+sv2CGKMv;dn@r1 zWB*2Ak$zm6o0ynzZmu4Nqp&pA8&n=bM%9xowU4rI<8H6OEEi;Wxp^9#YI+`g;&ifX zN&9vbM%(^*^>DkQ{ce!H{$;Ianky)2R2TU`7li{TH#>vue;PNNDUMqk6cqWhrJ-tk zO8j;|NXG8~W!U3D^oTud1=_1#TD8PcT1q&iuj9|W-a>kH2*J&Q`T*#n5o{Y_9-eyD zAA=7n9g{T2S=$U8X7jgi1*Hydezmr8;qiE=0ZU=$cJ8@$_yo?@{xWh;YFj(30@$^& zw=aJ_^4kIQjd-Z>2pm?Y%Qlg%KX*xCIJhIkqLFOAex7W559ebQ;V5Q)N}Z9rmoQQ7 zzPT$IF)@myFD_xS>P44FS@a7`Yq}bb_knmngpsRIzb9x^FsA*OrOmT&6q&b*c8T)n z@!;{P2ZfhS5qeQ)<4ouF_I5PU>)xQN{lH`8sx33KlN@;P?05)fa`kM10CtrCru}#% zg_ZX9^~Sw_H_Rj#)Vo|(mR=F?dcgtv8+uBGm!4i8DEIO3 zJ_FOZN;r4D0@LdBs)|*Nhvjp<>mAHeJHT$YBp9!uwej)l!!n8WsP}bq?P@z@CMD#l zzr2YA`@=qbhlq_diS@qGhjB8}(vB)RC7yQWUw;1=_(M(?N~;PY+s0slH$oQ3_d6AH zvHb58e+gAgyxyE{7H0HpXj5=`cfa;r{t^;KsXxm8zbhK(y%wNdoo==?>(mznq{=OT zf*%>Z!=G0Ld;d=L8Pv~4W3_V@zkWn>9Jii-FaDzQdi7rm#qZbm^cF`&Jx+TuEr3&Q z)NWSUml(YD!3PWYqZ}8sVdc~kx&&Zpg@C~9!tT)g-wP0!zrs+T68C)SSfuRWS6Dx( z;vRrBxpHr0?Y{IY;URgFaLFef|8F;o9Yq}$l1s2 zxCvZ!KjwA>MpNyT``2lLLj7FjQ_?SWU+)f(?$HSdDsZeD`PaOk+m4BNexHJWdMsD| zf_sXG%ITpNDPOa}x(q1Xxy9We{(l4NM;vE7GPt&yXOTS%-`jeZ}%}Zb-?d+9Qs7OeJj`BV~ zNTiE1{B7)o?;Qm&z`*!609;AK-;+t>FxwtY8=ZDlW=qTc*RxA0AHC*KBo2VjwNP{p zzOAfQnw429jsM^36};%WAg{f*@jAx=rc%tQK59cWt z(n?KcwE>&F=}fV5^TXMYEIpabzxLo%USe1k84tCc_Td1KSV}EWCWC~})tKC5%kjU( z=A(nJG?ir5++Vb)m;sK!VomENVM>+KLSyUHKU)-RIv-zdy__AgVJ7 z6nH9c2K%X-s84XAU?*TgA z6TXnJ;HNjXe*L+$Eu zwda}jxF6WSKK;+Fr-5>LQH3#mC>N#sxTm_e($WCnYz38-l^Tr{34eYE+`B@vmu~qM z8uP!_0B0P$M?@oWqw%gz0jF}VTe%79U-zUDf$r6Qwtc$l`VLkJ#0ILMrv%f!eN&KB z1u^_H5#|q_bWzFQ$_Up0o0TLM2_J8}Ufpk{5E~@=+qoCMFTjJy>8nC)eTBm!6Ci(m9`$tag6Z!Zwu0Rok?uG?t^_}p%62E`>w9}51R2GiBR zVNmx|w6bPtNs;|R;{l71|K*Qy? zy=nb7PC_|hU_Vp%MdSYd&YzD8z+hGVoc}jQ;6?$TR>E}O{@)kZ(F=X`I7Dj%1eh!& zBd@WU&(fPr3k80vBn-6Bos;*Dtf&|h=$gxBwz3qpm1G=9Y^@~RYA}(?Xd=iu(e(d4 z34PC?v12>#r>`-L%O;p2R74DGB#geL)X5`ecq{7S4=X;mHD#3j~O zvROs4@I2RIJa{I5n-!G^&fh6N%|V}-RHEh83EP7{~>@1kht)A{8_8&&`Sziqur>ArW=Stf|7wDJL+NacEO$wg_&_MrA)UdXHuhvq zgkp9oHRXka8|R9rBthJ*H>x`USF_CAvg&ksAbXPA>2ATr&qbv}u5kg+;3&CeE8>}{ zRaG*JetxSu+X<~UbvpwL{n|Z;KrePG4Xel&8}rv;lK)*Pe0k6htBjAw+oIy|o6~ht zZ|$ubwFAXIZ4PZ~ye4Lj~v#q;}a#KB|q4RZ6s@3p# zQ}RQ_nPO_!mMgP&>gT1nj^F}e<>OmZ z#ycV3;X~P~RUn(|6tu+ELkCkp)D>+($BqcdrX#~HNJm_X3r2QhGj~{17gBdbCdUJV zE%uL=z5NxeukCD~dXFQpnUzaOBa|FGlFvLMP0FzIbnudQitL{D)|W`3gLxF~CiS5e zf-8yVE_Rqmnvzs}b4BNQ&_*vV8KV=OhAW9Rs}SR(Z(j`iZwj zcdH&(la~MOSJXrZd~a)LEPS+7m-wR}_IA`CKe}E2Iy?v5o1Pkrv98Gy^tXvCI9&3i zp4~y7dG{j)t^V$zlgFG|#vRu~$pdFXqvDiz8kIeaqrajPRn|`r5|2Z~rC9HsT|Qa1 zrwm`7O%5|>bM1mEkK?0L*XgMh6lSvxTDYc0+9%+rMykZdlB;_uc-lG{c@@&xcY7Vr zN2WbShty0*BhE;qqRu{=xA!-|kWU#!N?+yPA2DqosPT**M4cMWQ_0G$ej7^MthcT| zkcY>SaB?4x1BoVi>Lh$4-b!HgY$w(#-J6T^cu76D(A3~8Ho zf~nYah~vkf?%uA5Ag$s9&mg1-&EkmNC#_9eaL~#G-Kw`)9V1 z>51Acd@{^q-g_?ieC<^CbHhW^MGG0%YkAvy(na@#zZnV$oz4pGse4u`m2V)^&!_W2 z7p@(0Vo~&=*3%McoX#r2^&>NzPct%6Yw@m-Glk0DWvuf09?vQW@FBCHWv#L_dZmcu z25O15;t(v%HZda+-EEtR(uK+JjeN{SYb5Mlt@Ep+d$Yy+y2H4-m7ug*!!`VsnViDy z#e?zJj2>Oi`y9V`sF~%>eo`(;I&E8vvv9A{-}PMVYP^dpSEr0RZerQwNcBazh3+?0 zOC4$nGsv^`p1#=ic5hQjyfS6Vss|vKKlh4)e6hZ)Q8#P#zIRnuQ6;ubSE}0JSaJ!oELVDPu)(Q` zdQ0r* zU{EwgH0)S0*-88TguPa-rrv419b8`1*4D&`@TO)|$&nK7eoN0LvAeML_LzG5u7Gsl zqB)s}V#A!lvHBA%Z^=O;*>wA`E4l|ge;!iecJ8H$|A`zmQ`cR%kr_5mT~vrfR3iPP zeCRd&iKLzRxJb{YHTj)6L<=d1tWS~AWH&pXbpGtAY3-GE`voW@)aiX0xA}raE(1n; zp|vL&1%*w>`(m!X+4>P|ZjS51k>BX%()UPm*~ueH)*?M|0(9eoJytvz)2kBVi|?q6Qon+9lJVGeTUh#wJk{D+ zXp?~)BW%fiS6MU3bb5={drn7PV1C_ena{H$u}^xEi`G$XT{g07=Akj)EE)M;eaY`d#f zbrsx6qZ+&=Yu%RC;P8wSxirs{&(|&(1KxV7Xi?!NzAMU*&gf|Q>BPhuSGGGk#xt{u zL7hq<8yE;%dzu7AJUxJFu6=H|8qp4!hN3I@~=CJ<}c-&FSmKJwkLW3eE`)| zc!vXwRWktQtb#){E>+a=he6ec^+Ee57!O%eZdP<>F{DBgQQiRd17#)#lZ-bVVBm zF_7v#WLTEhaOwQ5t_h?vCCaTiB(r#~nQU>jFDb>xtKH+icoswsN{7mmb)Up)(RMkZ z%v;LscMytC*S?yHZq4Cv_`&l_?cxr!&U&{BQ`ehvXm;#2G$j!(rSOY{vm^v->X2J) z_qu10(?Kjaw8Hjchn3@IWaH=iMUG(>gWRQJ|6T4QJ-5L7HMWrh$?=<=p9B=6j)OL6 zjMKu3UYZv#bk**ME-lV`-ZP&|q8T1dGn{XK*$QF#T8|Y_G)9e*tTj~!;$6kSDN6|! zVQXqQ=AEzgXVMq)j+tB)gkp@B8 z%>mAoj>?O~?2&uP=WaQ9P8KHC=K~z~Bz(7h3%3)_YlQ=;bG~1xZ0rr=;u4+u|2eaT z?k{bwt?MMKt+uG9D7ln9Enlg$E{xMwHy|ZQHuP;+jU-HBy_agVs{gs3!>>xso|x4o z%aeum=)NmrPhz)>TlO#{v>?udP@~91&MmT=a1UdDJGfXcS@ynp4^tjin?ZVrY6%&`F^m#i* z#;oPV1>=X1g~cnEypQ6|Xgn6TLH(DVTieMec5PP+E+hSgAN9+VHa0&5iad%uN{yZC zFZrjP__)Q-aTxt#jT%mmAGl| z@Ol6Hw+^OX$@n1p{NUS-8P*hhP&R=yph#o2mbJG7HN;|auIjV%5lt+(_@rbkTp%38 zblvtTaWSb!*5f28ABAOG;v~^v(B%SjxH9dy9_LCXIoarVnK7PWcD>Y(}8<) zburn&ORA~(%-ow%Yp6x-#BMt$CDG^&Iq=s}N3}g($mZq*Hna{|460*F4T$Hmm!#_j>ARNPoL+69Wk=zpy@ihXf&%OFpuy29V%-$NLnzNer{by^oG8;@ zkg{ea{RVE2lhe*!BuJW#z3i-33$GYoZKhWXVxKiws2Q?S4>J+J7O<*joT;8dmBzPv z3TQw6YM=f{HqEYbHUkAJVb;rM+GF5scdUqy($IrA-I9>h%!E!Ew%|`@p~|p!Yc|PV#Vq{6WhH98aA25l4@RCgVXXES)cWd(Kcq9`UOrdTS9h&K`_Qr8}t<| zfjpuSK6b1tN0%#}l*DvJ&M52Sg%jm-`|?e{n4|ldr+uq0S|aQIw6ZPZux#N)&S={r z5yPU#YPP2ge%|x1hik`k$dZtQvRZpb-OJxnU~d)5(FjWq&K2&xVH%!u3YP@q z+|~32zxIf|$yiEN@jkQbNO$(Ax=p}Vo82l|Xb62YW@T$kaggf_{i!K%A zr=@RCp2dgsVl&CRn+!WF2smJ;>-Cw8#V5HfHB5C_W^cA9)r+s0^a+`im}MO+BTQ$8 zRPlDrR_9=uwAZi=bIGTqu4|b%&qFt_)X#lQ3RQE?c%QswcDRX_3d=i1L;s?Mf}`o6 z>Il*%fi&sf!jX1`H8+PSm#mqV7SG6w1gTw;GKiLGIk?V!2&2uh z0;`+Btq3h3*fENshnKX-UQ5=9>&C(n&L*>>zCC~bDIaz@yr@Bt?kt!pE)@u5E$VX?|ITyg=fQCGUp|b_2TsYm z{WYWZqABBtB@fOnwFPQVnQ$_@GhA7}r0Nl+IoDp2{8*WocvU8&g;b_vCOznMX+*Mw zT*uK6iO>1pmKun<4rDJ}6z_D3^mLQSi1IJ6uUaQU287Kq{OOc_U(a5MIk$#)K7jm>z_3ee{a+$0u4CPU8_w#Aut4MIJU^J!_5 z7Dp`@T)iaBp0?r}8XxL$dlkzUb{7UMVOiO4BXttxjT;z^R;#v{$tSl{-0-#)9MXP# zmj<~E+bZ5KgkR^*JH6W;pj+u8GmptGIt9fhVbFQX#F8;oxp4Hf;-#~L(#hMr!pYck z3I+Pp_NCARv1c`|jF313tHdqhF8!1KGAik4p|-+dvaN?x zfon5d(+taMb%>isoU26vlwyWClpm7JwYE9@%C*{wDt&NVGip*~=$xv7Woj<==qK7= zFQ=663w09wI~mWVjN-2lKgL zkjL??{XP7llG z=4Z+q;-wcauspGm%nS0B$V+SxZUaQg@Lth(-I|VK@^D?PCUbk5vt(@Ux6^37_Fu5iy$psFM4A}1J z6J#UNvPuu2|Ig9Cl*(M}a?n50Ih@BVE>DXL$wiuuS0~zYbco?XxhNvl%(^l*Dv`gJ zF<0>#R*dZ+mvr*7ud`7v8!t^Zava)?vy9>RS-8_qMXiu^bvz^!Sgdn8GG{lhOZ_2oGs{osEiLE9o)1j9p0-KvHx)M7ru(hHg}VD0M!To_p$O@}9yhA^=q=Vs>$;=!ZiX|a z5)lAz!xo+UWFO{vt^2~prle@P2d+!}uWcl7%KHU{dOYoTt(hV>AbThwD?0}uw`sv| zpMe68A|PM=fHoiYcg>(0UvxS6u*!K-tpaSV54W0#tglt--jyO)pRgXU}1fgZ9ca8)N`+)(UJ^Ybwt3{;dRE{8zpC;eR{g1DD?f4 zn+vYjXug<~Oxqv?7$5q4u@M6Qg9rXg4A1M9KoAiTVKW+s9|(qmpLCewIfKHeKi|sr6 zKDPh0>1=5dzWV_^z_v957+-tSMKQqBC^-Nj`hCFv<{})Ji%uqJXmZ(1;3ZIvW``nh zt#V#|elbAzce~pyNRp}GqPqHpD)|e5A}AX@@t92)4Lya=eh%eMba(EXt8H7qDJdyA6o=YEKt5^);}3s_LP_n7OKQxWcI>)Fv!~XX z629@f)^F*-_1@LtUV-6TL8+^(yqvX&oXpNt(MS_;1)~|UsI0;|TUXN1k>q)LUY1%p z;rm24&bc9;+DbZXs$}?&D@ub&tX*dJah}ao@t#6cALcFI@@qARLTLb89YHUDqv5xU zBY;uw25QM2Q9jlL+$DQ5U@!)HdOjjXo?R3YavkMfW#!Ht&cYv*?tN#<+{_;IuX0aI z0198XdG)>@u7^Y5aNtC~M%dyg9Ej*Nmk{nH$F=4^vHkGT947W=iE7)=&?*W(4Gxk9 zL;$}!cY5{y4F2M`M#+HV^0EljgP2YP6c>f_WwfiujL7;)60Pl@p@;-(HySNX@fj`v z`mDHI-gkY`$7uvLjyg|41Az^wfq{X==b}IRU(fjhYEuApR}A3xoQ14pUn$huhqH*o zhl}wXl|xzAwc^yn7X2mjr`qlj`7M!qtI*Vl9+%`z40%V`LGg#8u(Uf*zs+sc^1GM`uBIHT|iJxZ(=<(srYllwm(|*%-8HEKM_aQBXGylLL2uH<66?F{2SD282j<+sB28MZp=Nx$kgZf`n>7b^Jk5 z3V?z7+eV-%mN}Lo&M!Yk&Y8WBf=+{pSW)lCU$5YcZSFw-yKD8g7cc$*WNUGJ>*sZ& z2tdk9#QEbibrke#AjjvLg4=oD@O3M>9SitG*4SPK+~1L0OY8 z{FpK@FlYx(kqjq2jza;n1Ds+B5KQtu+{TSehthu;1TUFAumPnc`SyXcE>eI+>=>$f zE@{$FXFKg}tsV@&YTU6ltr##;3LjoxhkOn-a?39sah<15ouUJQ5|p~HDy9x;8BA#o zw_MlsJk;x9^StHbJ0r?az?4A!?}kX&cXL;k-)cjvc_?1fO6y)3jkt^c%8h%puz@V; zDbm(Whi8Z+yf+-ks0I=4?HcX{1j&$1GcYAEh|2wS>2y2xk#n4uR$}}186bU5y{R<=uqxOeQw`DwK+RJo>F+F2{LN#-$Cw!|Iga~*m0GGFj0+aGXxLl(f!tCvje4U?uHkDYn zp~q0>|0*$YRBV*(BOEI4Dwhr}8t3cli(HZ-S&QAYF6w#f@TaUxT}dB!p#>jd{-8B2 zQDeTJHEe0J3eu8V!ctG4JFs&s<{DfPYpNB7%=`fveRW1J1cFVhRzJk6Tk}oyz)Jw=X#7cL( zWFlT?(VV&C^`s(kvk?^hC)nw6gx@SL`~)Tcf>as%&fsb(rV;(U8w%ei++~`g!$sL9 znxa9c%alFeli^o7P~nQ?ayBs^9C`)w?aw7{%6)MYX3n+j3hMidL$fK5wVn-p9y=^tX4 zCL53{T@~}UQTo9Ms(|6mc}9L_W{T*IzP1Gq8f8v~RG}4h5U@md;P%REJ^`p*L@lyb zK=0-iij`;NXO^}AlUPX%qG9}2EQisbUkW-V3!};|t`hR9Xn>qJz#n(buQ z8j5}R_^y6&c&iIgLeM3b#Bb@{TSL9c=7Qi0=* z202VG@Np$iA##V0!kZ)I&&v2V;a~Z+Uz#T+Uq!=svDd&uZ)6h%=J`6C1rB%WlKawG zR(BzrAbNYw2XX#Qsm&I2(!5nh^ksN=`1uVtGLbcymYABF9mB&cf6-R*qIKLXr%lebuG_Zy-6tN z!`3+fUR2AlrrBeFp0Z=)PG0^PGA@&oojnR&GKdem!Xt$D9@EodOb^O2{5HMLdzS&& zc*?f?h;1x1;_p~$_9(&^-yl($nl2-MToY)MCBCG80ZkwF0FpwP%YFFk zIRZrPxm^;r|Dx9nAr3iA8hfoTBw4@{z;n7r|3aR8(y**SR%tW@;Uz8a z${vmckrXutIh*HqcWukn^0JjPEe%bbPSR2f`7TsB7MA&mJs|O1dF9CgoPkMi_IHv+ z-{?^cp9;RsAv5K3!SWNkr4I^rHJ(xJTl9pN{>*Eus&_k|HD{7B(UDg%KVzu4lBz*- z*3g_CV2jTB&5Lre(EiRhKW?hn5u@dXpO!p_wm`a&Zv!%q4pW8P^&H`7(8Xu4w7E}dXg^~YtwksGBWgBZ7t$lv_ptlItBqHn2a5v%GO zbEWntV=VIwt9>%>YD>}&1Q(3aOFDgXGzq#XEAek4)e!^ofHhp8F%E~g+M9K4s6Imf zK?i@|>_ev#;?rA447~mtA?mt+x<$|Mar598a2>Pd*V8pI{kk#RE8YpCtYkZQQq6tU z`lb5&mx4{9glP8q($w-QpVHe;FXnq*_i%JRU4oBKlI=ndIx9N2iO22L*xv3Otvq$W za_XVZ7$2r!c6c>JiK}g;s65(lm{Ieu%R*7^B)uaW)lXy z1Tv49G-kF3oNcX*7T$TUDB%Wqk{2QQrK;@afUm-V3bc8d{a1-1J-uf4?!Nrg(~jCg zF|z5|EZm{|v3QyjzY+0AM`C0crpi4BFL3}j5vLHt|MH9*?H#65wfXxN!(E}KB^|Ca zSeH%@iR-Dkr8f)nr&^53s~$4xs}C3B9m!rlT9SfIEX@7d6$8$jN;YKXy*M+>?q}NwVe}0V7W7z7c-S%T56^fqXr(z^(Zm2n zw;o%sXOkBt`~tq;=do)^r(ff1plSwrrv?rwnLi!yZ5_Y1XMvWrKoJp|3)jl*lryHF z?9CKLuFi=5cy>2h^`2>bwP)kTPzKnTpSr<0x9DX}vhJCJ&B1maX4%_T`?+b3iMgZN!_wS}hl3Mn zl`@*N75&;6(q+S-&y9m$RZAbMyV>0iEs~Wwlyl~Mlbmf@C=ZX!!Yi0G#JNH?va}<4 z0%Fotw&(^U>q_=s;(n#P*LdO`h2$C;!wWyAt%mDIlmfqaz9Xq?30|o(pGdlP`t9uQ zN{oVGX8D}^Q~mO<3rkcP+_)B!X|2ng}k1lw1glnH#a*>63OSVJ#g@*?uBjH>W@kg3PQfSuJ5CDoD6MUFff_ z%Re7ruJ3eww(XFb`Hq=SpZE8)dlv@gC|XT~e;vzIEZN;M`ocw>@PpUY=2*VM!P-$X z`bcWcgn|Vy z9sO>~-P?VASq0DOA~f&GL;>cy3>;!DHQw2ElggT57N! zUH>1#G<@Oz6%;u_bVeK*xR@APOH0ee(KAmXytkm*cRTcF;ly4zD+l38cFgTAM4V>o zkVsoH5}-qai-K?nM9!^EQ!=LTbcV;AUU50 zFvJ+4tioTzMpO(K-?NALwb^T>t<8 literal 42615 zcmeFYV~{Azw(r}TJ!jjtZQHhO+jjSC+qP}nwry*+-FvgFS006wkOh78GNC8v5|H2TkV;Jbx^_`!!a4>xXre;YJ%O->z*YTwk=>Sp~{#`Wg*i~aVO*L9c2 z^b0`Fh6w}+Sq&mU>ARy|O)~}qJ$YIv4iGFa09*`U{G=E{3+Rs*V8y$A9j4XjDPgB& z)1mY2udnL3qoG1O0AGLEsKX*Vc{%RzA)+YW6lef{&>7Wq>Xcpigj4=C1_oBJ5}3fKIpzCaJR(~224yQT=kSkxRsAT5JIK^y*DMr^ECvJtf+My7lrF=p`8a`hQJfD91sx}F=)*yAPgJcunh%C zIw&w;02oj0Y>;^S_&i$jLH6-W`@XK{z9E-tjtr6m&HTQC__4>>$_| z(TZt&c4pP-y@I{s(_d*Fr1!83()|LNrimjX1e zXQYb3>}RE?vH}AgsL})W2b~pRD$u2;(DsKXkoE8Cp5|R{S6J^KGPnSE1dxJIbo`cZ zE`+e5!h3Ldbzq|+;xXU_*k*j}G3_I`x9Besy?Umo#6!RKeQ$by>#?RVrAVdlOQIE_ z&xe=|Zwx}}4bg4aDptTR0a!#E=@Ha|tu(H1T7or!dq8_Ytp{WEO7GdSf=d513ApPO z+KsbSYvJGE(*&+US`NS%lG?d({m}uq1&0%2vOj<0_WbdT@I3yk{r>9{!B0|_CAXyJjv0mcE|tw|W9Q1+g*nlzl8vuu=tIvEVKY_6L7?0K z>*R@vXRJb!Lt=RRI<*STGR2decWg(vXNG5&XMkt+JIaqD|6=~SKI%Z?K$0+2A+N#0 zFbgCYGHnIV&XBGVJvbA&&%F6?Z1P_5SMgqPHSt98OmT!U^0BQkpu>+bl|-F#T18F8 zIK{^jO?fnh!cz9qTg9gmZDp^~1fw>3g_Obx=cAUixU}dr(aRF``3NFJ1M0}fJvw{r!<(x5cMYYQ1!P4_6BbAb5jvhn|bRw?8Vq+y;H-p zg;UO@!ZY^ss@ck!&f?(t`bCK)j*{g{m~)L&lC$LVZ8~453Mg05pHS|QwL#87p8eqc zfCOQ&<#IuCnX@r;{VUt7!!4uFk)cIm4WrcY(DBzr%0*;F!;(0uqbVu4r{NR#9}=IKi5i!2rx7P(HH4=IsFQW2AY>3|w- zRxwt&mrxuPndzIx?GVn{mz9_Emon>Lr@99*7qaKHZK@`A8+!)f!B~q}t5G6YKQmLL z>7+$6Krw-cGEqyJkBVfp7jijy#>* zCfq>YQQmo;6<#IoC>=9hSsi8`Zk`1^+}y1mGW5tB+o>{D*y&(wM|NF$BX+}f(|4PE zzJBt5B>}MZu;{w<>=UU=JWHej@!?X~y6nW`X67*QbyE*gL8+^E-MiJO{R!HMf>j?W zi&={qm2IBMn-S4#)61)AvK{?%xXZMQL%@k2MgU46h0h|q6(t|BC&VPOpWmNfo=1}8 zM!nu{Y1q1W+j00o;iz1pVdqe|HB?ZKzU1qOVr_Q-7H4 zD#I&7MHA%;`${##K+|p`+;#YQ?|Bwv`1dB5LFiN{QE0W>RR{gnI$?G}zZgzDaC|h0 zOtD}AV*$Lpf&BEH(evp0<05(aa^)2q>aSan0E>&|3oQofdj#b|J~N8BwE0uh`gyWB zqz_#5G21cH>E{M!&&yP&x%209_T}^H?XFLuB9)_QwG>Ke<#o_TC=F&vE5}V&M}!74 zXN7g+$jR1pk-ETt%*M9=O+3bG78;L(3;hJiRI3RfngWc$O8(l#vv@ zIHq{GRk1WTwG9*$6v`Df6n`#h)o! zwXPx6TxrANtnQTcHhI?UptYnG>0)`s=(6tAd{24Rx^TYXaac9HdH$sOWc@^SdVUIX z-py0vsr!ff>GLEY;inCbF=vK*!UyUb>sRPPd|a`(42AddTc_TP`=7m-VcFyQ7?bEA zw*Kl*(YO6`(JP0uyEv3G8gt=E5t<0MNT+ViNBe`soyCRoj&8B9%N~qP7LzZ>mwE?I zd(VT)KbQCIeNW)m(LOcrGt6*|H;(j9_Gbp4B!M!E*;YI*o}tgT`sSA^@5^&G?%V2J zZ(cr}ne1Ftx)7`NmETzxCJsywM}tg`Fd*+as4pSiGQl)>+WEN&*gHwqdA^Ae6t58l zK<+BQbN;@!c{<7vf%yNfJ$`fAaR5kR-fC2cVgH)c-&6Ai0R;slM+DB#A|?6{1KdjhDU!Y#HOi9y zhyJ0Wf|STYM!4rC{D)l5(0uFKN%A6p?qBQu?^Pq1at0yQO%LQv7zDJ$2o#$tk_Jg@ z)ffEJb;jn`LO7ul%4;Mm;sFC}4*w0>4juWwO~e-m1PJIOWPtP|@q6~4>I@0t|2@7h z$m?%jhA88z{ye8t)*2q9aD3`sB6_aVxDX*xE?b~UlIrUcN4Su=V;&ETW!equZD0vkPuney>}%XN|g zVW?sFkCm{20yI|>ANmV&JiDa^{f-9VIrHJ0=2hqf@7E9l#E%AjQsjTudk^IYpt+0A zUQ>jacUmZExtSKauM}7l%DZ|a#;Oj1x%u#0nYe=N#gx4RMWfj$ZxrFEomG>HU1XIe zb4*YXoIEnQ<{`Aw_0$!rPM^w4f$?aK;M-vLzm$o^|8WN3OkTvUyAw^2mT)>_>Lnx6 z#J;HfV(EPun$<&gWEn2(OE3gny`~j}12@1e?`q`huC!tUl8X( z!5$LcK>eL%Jc)nXPd8<7&B?4Gtx_)<)ao-}fVA>alW27PMfPpowHyR~Q$(;_CD7q* zVK_NWI=MIGR~;&BLCJnQ8mQ@@oaH}`b|MG#qk8{3lFFgP?YiN(_eUjE1_DKBQdMr- z*nQ9G+w2SIJVR?m_0l6M5>rL^Li0IO8r>YLzvm>+!S00`(-<_;_Y(9SA9{py zHJ137jb?a$*_w@g%E2V$@xYY~UkUZso;d>1CjsAJWaPt(^IPyEJAb+o4YhNNEoX81 zUeh${C(AQxP9-IT-;*GRpEiz-w7TnD@cpZ=wE}c$$+5-`2_`mzVG9Fnc8LW_+X<@ zYxWdpd#@dBEG!gcf1T@m@rw)>CToX9MfLRXmh6X7RP`Jt>r}x!Imybx`UeEq9VV(Y zu?9h5fHs2QVEuD{N0RS3ZHmR4L74&_Ph|=?IWcj+p5!ARr=_7UEG_NrNAm=M8*uyF zj*_T{%4R+eh?u!;A|(UZ3dN1*0L#}bIRPo1$dG9nrbMf5JJ=v=Zbe3Z!LH&g((`IB*id4d9o zx2oX4oP0klZVtc)JBbgRd<;&ImfFpoQi-GjV4$z}W1gGl?hmNL!^7wARfmm@iD$4m z6b;_={>0&Ry4}n?T{2k&WO3=5d@^U$&GCFuAs`bcv&S($EUBu3HkrPmCCRi^MjBu1un z-Tl>dbjt1Xyc?`8r!bbLUn$I+pr4BvQL0e97ldK`2h(6U>XOPX6$VTb5m1MnN-K~p zt?d_wu;tKJ%OB}G>k8Je`^)LQ$mpV-Qj;YD%98n=5}sC!W~xg5t(yl^){Ib+<_ke* zKj5YTO3u_4cm;mnp|F0@?K_SU*%Cr$K4I$Wo`H~byQfVDx~|N78SIgPtoWKIG&K<{Q{G{_fDM&6cl%A9ga1> zOz6;O!*TlhZ*&BnQHj3y-^=aLfNQ3vF#$s?JBlt87MSahug|BUeUEdL{tWkr+K&-e zFT^D(=z%G$YCr-F2{NM3XCUL%q`iRTY!Rh%OvAO)gm#?Xx)9FxPG9Vsy^IwNG{ z?d}{X5)-dafO2Zgq2i~iUVpMnUrT{p6CBsI6@qvcrDJ6;K9xsMM=qJ1vNCZBWOG7MG8ge$2OM+qr3f?zSZwU5 zkdR-hl*nG#+@7w}5JQ-vIS4>qj_T}C^R7VQ{!}4$3wYF;?$20l!WSBd5r@TAqH3KW z<$K8blEHe3A`H-jKk$#%_!C49^y-+noT$i)aGOzJfVQ13JqKj)CE`d>8wy~ zKi}I&rl@0Rkqwc$!x9;!T1S3~Xzn;(z;A>9Qejtjhq6=!hRH9<_`pDPF-F^y|W>BP`LM-J&d`c5E3v_iKL^*R0wf}Jvv+fO| z&iS}z_OV<8g~l1?i0#72b&i8}=4X~9fEScWSl@UTJ}wp`JN!vA$bfV2$oMz|v(=j6 zhSRcEkR%hmARp~j`!)AI&*wf*1d6t6Zwz!fUgn~=1}OLuA#M}t(0O&Y_4~5VM=~4+ zhQ&lG!!WFXoq#ZXgwc2s+Ue=((6!H(hk$T;NCb#`qVN0HC4Ki)pW+cwh@DWJ+@cXV zl&~^lL`g&(wZIvGlTeV5;W$+_6Fntuj4Ej~R?uXckz&%3XcldqI7S4!NRXxo8uXn- zP0>mSl8+olg#7$n**hT?^AfRl5@Y@lrC5A_jX0)7Q2+Y^?O9mnw1E}7j7K3EU;c)u z;)`mW`hmd$;wfv$BvV=hJApuQyHZ=0=2-+~d7KiKoNyOd0Re&d+*Ax=M^$M-<}EWK zh8#Rk|LhKwY1dg#xBQ)NtnvlXZsjyPfmBOz>)gfo7Xx-`fmDfo5!+eB9Ar8-#~fuJ z)oe+^EjxXMy%kf~WY0lsdt}M6)1X8z z!`ruB6E#_``V&bc>8>@HWmGNrl+_4`CG&KAHgaMQuZA!M6Okd zduc^RNeqv=$0~Nw4hw-8C;`gC@?gPX!0hZZh}62t{U*5xn|zbAB3E3kcInX@?cs{2 z4w%3P^3TkA(7#5cKLkubMjpfL!G~$u&&gjpl#32HO)t290p@%# zgx*mN@2C^DDF`C=(sJo_bM{a~yhyBGX6AgjM}A47A-hu9&dsd4G4%61O{$(#1KXl` zAER(&&Ki^QmpWh$Ka5y>ybfeXLlqO{rv#_{n%$HgVYjEIo6YN}>Y=|WQgGoD7N4|! zmuen!aj(ByDn?&!%+U%ur2-T&5MWgv70D8qJcB)-~qBB805AM4gKL71RMd~E7rpunIghMo=t z*Y&D|h{E$*H(9kPBvU61Fi5qAm&>rQV#(4_G^0qobK&6n}}&QBiNC`>l}5WYqh-0C0@Kssm^ZO~yk7*oJb9nlw+ZKEQrKr`~`#Oh8t;n>gH@~#Yy?z&?{_K3JV z&0`gJiLeN5MpUA}R^bZIo2G#xG`Bz>T5n>VOFb#NFbRpsJ!~2Q6nvet!!3O!oeKmgk;JF-x|ef@aQ~LBF^{e;m7`Ne`Ggw!x|p0f3Hf|%YLeZ> zUqXuNwQr#r7WXRdga@Z4Lh8`hCBXy|k?JETcGav!Emso9C^n+4cXxLWCH3fTQ{(j* zfzV8tyRZ0JO8N^eU0Eb+<4y;U!lY|d)e?^@4Vt3@g_oQcZh^vy=(2V%B!|i9-+7?0 zlv1Lc(iJVe{^GJ%c@K6KJ=%KR3=z9P1M;}KFp zjMOwu0)BQH)LJ$n;YhQdkz_b_&=4x9?GCa+=5^svDJ74a6y!E<{S3o zR72C=j|>9JNVoCA@p5t5281gWK@*cKc@@kAM8!3Swq4m|ED6ULVxee36O^Xs=wp!; z34{bhM1_N~2`PpLt@sNG2N+~vwZaJ!Re~?UDqGU(P=|qWGHH6~ZZ3z*x(6nNXku!C zw=xw3Bzkl5oNmsD=D6G_`|bo}WTVf0C|$bm-Cs{%j3#QC=D5g#^iK-^+Rwoi67aF+ zRyb3a_AdpEQ;yOZS?e50C`7tpHWacj$r~Kd;~i=5!V2TpXLJ)d!Xo4BO-Sl;1qbWV zx6nAJTr~40OI1nTC~MW+)0En=EC<_q*#bu^Wm!4Wncap@u$yAD3)ttGZLlv%5v2;n z_55QMPcCei5IGWy67Y`^bM?6fhst$JEZ5mxjy0TWG_ac>3p-j6!ISX~WMuU+#Kl9! zyB8CRoyRHa+Brt4?qY7q3$R|t5vw!gqeqTjEvb^h72^sI#2BVpiZX~&k4%^n*PUJv zZHgn138uNcByL@#mFznQwdThaRh!{u6;CfRmKPIH#l&wM$vmix-!`ImIjr&$WQe!RBnd&NY5PK+KOPBahc~dU_2RL~{kr<}bE<4z>wn zu6eQ)P3uJUhYYllFze1nxLc}n!{q19+a2l0TOf6vUaYSqbRgg~Vv@N8nX`v;VKD3U z=OVZ(ku}yOuuFG3fM<`)xL?*vMx=4$U_b+hFIn}>X?$04F5xA!GS~FY1H>tDYm{%* z!Y4fMWSdUOe~+YQo+qg_2mnTl-ZWS5TMW}McCFM4C{nvmNbAm4 z=$<523lPi5B8T^{K1RYDwa283RiM*M%FgO=rWwP?xKrL5(IhfP+vaQ0)+|ALc$oYZ zt{qgVCKG<@0CaoDu(Q$#9&Ma;qWbwNZe_Y&vaN;WO0Au`X-r_Rm?K#UHuNrKekRYH z<&nSNe30I~QaZoQXycz1#Xv5PrUQ(TH5r^n$)tbJbo<1*=JBA2DL#mwdaHydB(IQ` zHekS=Dx(QN~)T47V;$k$1k>KB}*?z610c zFkc-5KYB=2%V+Au{RQlaG5?6TBq2i@%yc*dEdM@XdN;pFKq)f+8Ec^_sk9X?5^%p( z_t4$admwC}L;s*qHMYZ$9Xivl3CpGIsNM};{v(w>qXgiHI;y{^GdNHMxJ>*uR=3u` z?EuDhF8Fm$VL|>@ZW))-;c&s0ly87OA@;hmP z-&Z49wnDf(9(SgFJ{bmjG_J93pSr)ie3QfWo{;>YFqn7Bbw0|bK9ACEf+FJ1I?X*3 zi`#6Y9f#v>eZmtnFIeTSE@hc~p0l)Hj4gAPmwHpZxwaB*e!_6=pUos9;~Dii4^VD~ zue}i|2!E&(3Zno43-vPkkJ*!?@8pNff49)JtRAr{+CVpI&BUWisUu}rkUsmJAVv3TG6oXV56itq9^Q|GXdgTf=~ z)|yMOLsloa;P`P1e_<>L8t^Ttz@d=yfz_)EW3#!zLzE-P&nn;7ivn?W`#woZ)jO=r zoPC>zmdEf}dn=JdG0H6p3X zxJeTEStin3KUZJl1zgXcZl~`}xq98H(*QTTAsmcFdl#}Hf?^I6+t`LyAX!sb9MoxwUQ z5=Xo4C&}AL&;a9oK=OA16-RU%s?c1ZzT6)g>-ygeKFY^Z8imC zE#%!~Ic(7ZU3mhk|5+@a)@Y27E!bf7Ox5kUg5M8a*z~Xr!Xa~foZFs+I5L?vzUai^ zKER@2|AKd^7vWdRz0nE+<%qUXZ`*_bDM$X87f`C5i@uegW4yujZA1lf2KCUa!LY5O zp(D1)LD$5)`@JZw@zs7@)$vj>1QU>L+cOg;$C0$olt|Q$YO4EuyH>e9XqoDdXOGEY ztfgX(q8Cg=XBv4y4^@m-^q&QgCSy^$`*Sb+gFz{1Te0AGJxyby`;yr6*WRkwHhpK4 zKV{`K!o%|4Xx*Za&`5a92^2=0SRmdkIosS=wB5#|p1$ls-r}(1pOps$f8*Z=r3(gv z=<6@-#y|=zEGh3F>~7W!hMamX;A_%M+79($g4JThqgV~?BH(*Gpk;8@!9)*@e5I8X zw)8G4(iC+O^2s@q>@bhM@07yaQ++|w0daBUhn2?r#`78&CPA?FG*uxv31l2b&lYH#-O>g2{VZ_PBD?FZWRb?2#8;Im@+M^4pP+>pSYE~L)ThN z*+~fSHs|K*DI&A?k68?cNQpCCya)-Gj;7Q_=2Q^@8aQY8Lf(1wH#}1(F$Scwp3|3WGG=v_YX0qmQr?Tjx!V;B4G@^NJEovGJ3=g zhb+YHWh`H0kgO-EVYI&e;j^}`vU|I1e#!-zaL0v}9s0cPe9F36(y%cL*u1!~G`QIt z6qc2ZF~M$o#cGNEE}}~6jAya5xt_JIvPHV+A-bzzJfQBa=`YS@UNuvK|`MUBkZuc{Vv@s=0L2|2%bE{KC3cWQ$#Dls>6g_6M3GRWN| z@mw;7u`Cx?(bOu~cBy32i8~Zz$WgnMg+&S)sa{X1EM}KiZ8Fnh8#t+aN^78tm6s|e zPg2Rad8c_Vx6juRAEBEx^!ki57ZsFHj?Fn8L6=+N*f-j>D-59rYhiqOd=zr6$U_E~ zRw^SKqvhqh8zoyH6$z>|Wj`ERqsZ$NRb=otzLp!_4ZfgHB{!fJnSebj=W|g2E8qZu zEQDd44dOa%wYN4@`3Pnu%B{(4S*fO^9INjNC>N2&F?A%y$=)Z;kJJzdh#&LNpgI>z zN5kdKi!QW2wkRa&)6pw|3z1AcQ%h5TyKLA6n%B+z)@ga>U}v!tb`JJGE>TBe~J8l@*` zS-FU)T>r%DRx2EE{LJkalRdhZP))9_;w+0t3h8j6B4aV_-xkc@NdB!MeEGsbR&Qxd zCL$Gl`_uvJeKaHO^_nJs{(8rxy>44)5Nr+sRW)|0SVT-96R~Wtt^ltER_l7pyX*e|kNj`5Bs#41uQ3Y^VygCRw#tzF zumTcrxgkY_3MT#BuC4&HpJaNVj_J%5Xfm>q)Xl0IlBas`?9aNL>xZQZ`IQ2!b+i=F z(Pi(=YxSOud7A=1b|0|$A5A~eml-JH{pp*!y@DU3Y{jze56Uh;T3Ly)daf&~Dy+q^ zCnvaCq92sBqT@^Qh+WTMFFQ)eteeOfDv!$sOAq}Yqt>1P!cR#E- zuWFyVE(jQ&6c!$GjX1}?VUHmYwZxe?6ZzTu>^g(+3!R~K@++E8#}v6{8~)ED>uY5Q zq9rYytVDmAQdXAR5t`@AxXx8l-lS#|T9Ye8MLY@tg%873ry1s%^arxcl-y8^az%?$ z*UoRqc~HmkYng!o6qdcA9hg56RPY$!bAMEV&%A-hV(c|fkV?reEnFSeUiYcxZvsi}F9PR@+0JM}G|sV62TQbFI1lt%9jK2Bk6H(^ za2;xG3zx%B*K$_Boh(c>-r{&FKSDk7eC5(zt|IUDr}wAbAkvg{r_PUGU?R#&aMrGj z$mK`lCZ5gp#CfDv@fJ|Gbgu7<_#NGj&__~P%@W^bs@uA#BtNL0 zi^1p-+X3~|xqBw6fG`y7W(2oeIws>6%;|>8feihM?8+jMh4HbIt3z-BTOWJSz2gA` z^QN;^K|HciP&G9b4ja~Euk-G$=yEIuhcebmgU5@`H3$C6vxKxXaHhFJWw$hr=EE^< zE{a|N$e->5i{(7WC@82rGHD(zC^xB0bwcWDe^>hX zl>J`~8yXphkQLU?}3m`T4vb5*B7YD?MRV-hPYye7Zou;Lf_2AEwJWdlM437Tn8K zmve!-VCttKW}7`Z7SM(aeC!uJtO(SEg8-==$IFA?vu(^%7C|kT%Rl=A*WCrjFG?HI zJgbrNqm(0(h=?eLh}Uje%jL8>Guw!}zB+ChM`HNT$QPg7rrm_K&d(brvY>NNpkkTq zpo)#(+wVs#?uh)CB~=Mh4@9l`)mvDNmDaE(03@>!o3nl^^$ULeb-H{a(wB&pj|Q@1 z0d;C8rGW_SW8FNQCU2)Ojl5{0IaGk3%-ap*@|b@`+_bf5<~#jCY8@Whr{v?HRyUR3^j3{}7-3F1TD@ zX2TRIw%9F?E1ktzx*lkbgGGkWIb94FEWfOG_esFoVUur`KOo>9ykYRhat;N%$kQiu z?wl{FJVD{QhtY-`1{K{ zt=--LUa4dQQ6I_Q&KI=xT`gweH(@>EP^+Xo72NbPeK5I)t zCVQc{3ZclOiuiODDvxAjXc9L!=Nz#aRLHl~$tg*JcEvhC7ZgFz8cbS;0DAf{;L$Vxanu=fX%g=G}j(}U9XEp}>^N_m1#c4Jj#H3J?PfNL~v zRf0aoh>DDKH(Hl#ofVok2K#1XLVJ=~_TN^2)T|5T*Qn~XS3x7-i1sfTVMQyR3~q{haOqhfqh|pWy!QEHfTHEpa%&HdTJ>SR~xenrW3J)ZDD87 z2F`bIg3)QgK+(556E_Fma`Z`-!52|c6N3hsWBgj6XVAlL$-vf#f%vTO;exrU`*Ngt zF&eKFl+IKEo<}prk|C>v@{vKG$v)DE`^gS}2M~+ZOu29j;<4F$ zk(q%<;Rcye^mUb>-WLPgsBYmzlzpX^XMX-K1C_}M3l$~{pm)|p191(ya zR@J=EZ*@}&q8B-|&_8W8gWCdeMjXsWIKA`X-DeLcXW~hKUBiWvB{};9Hi2f53~qZH z4X`K~7>Veh<$@ailpyE!>%7;zUofI^z`yLl|K~n(Zy?JgJ12d*!z6GP1-nH z-5yY(+}wIQM&T5!T4~Q9?2M{p@|4?ERW4vPvi!=<%IeB1*(^UFg^rGXFRX2j1IScn zz+HY`Z!&HEL-JDiVk=Qq>+2i#=~<~Y-~vz4z-xb4(rD~$kjGExTf)kRsyw&Z*;qCs zGVnjGM5ndW4FZi{H0y_-66a0}5paDGsC1;oB(4SQngrk^Q0B3fej7gO_B%V`En z`FwbAUYdvoaS!%AJ1DC@JUf|KTv*_B%F=KtTI@`#n{T_TWIr%B6m)6|@b_=v&r5)R z%1yv*)Q?&-5E#|vF8^arsvJ=r_+ADSmK)|yf8bvvhu5uB|tu9;`r#uI9l^~ zI(zw+3@wp9*HFPeHIbk>uD}^eIe2eDy%e3vT+!s@4F(Z0r$GKu#N~42 z3UV<5A}<^OWi9l;ec=yG#?{L5et!JT#?9Yom+$wazu~H_-3d)l0Ki2v<>0g|2fC?` zLmj8J$>6h|@~E!X`s=HA7A7V>r}xbk zbmqP#^TAaI_d(L9gVMTpl-)mc*-O3b^LFaNhg|=~RXu-WVSzjgsKFqTAc`bfPD_iq zMdGtim_XvW3eGG(uk+dWxsV@U&%-?Ykr$*KnNf&g(nIjpDuV@;mC*#>xcF-{)%@673^I2hX2GV41@^GVXl>TM=(k#6*bIt%~y4keg$WX}n{VwF=D~}x(9{wQu zTixzVLgzIO+e;SwH+tYn_+oL|dAXbtjWCj5FDeJ|0>j5BWNz0hq+-mD+&&z-;p488 z3xesUcN>RS(RVtugY&&Bf9x-z)P8fVKOXSXNOT|t8umO|R>qKpR^ttgjnA-}7OB_G z(dYjvhNoM%fiaj&EtLC|#{M+Y;THj_1pBkOkoN0ZEXU_t-|KxA0%+uI$Kq)cP=mfh zY8zH|{HrW;J{p@VZ2KEzT07zcE6$osV}huyt#v=pGo{g6wkQwj=!K?Xp`LB zku3MQ%(#lq(B{Bgg}mP^NwHRdSyN@z$XquAP*($vPM>3vNb?I*wxIbf8RzFfpR<)s z8x;<*AP|>m&%eMncUeW^O9H@e;fF&Ehy?(1(dYB|Y3t(y;D* zyj*VtWV*y|%jQtH@CFi5K+zixnOwm@*)rRUL=uVZ`cUh*nMLG$m~d}c*{{h=c_4Aw zZz@z8;poDK3TC)DNXi_cw%rjFK+clX3?-}EQ#BaW`vw`X21G^WWj(ev{X*eb_OIHH z^*uR2KS*E<^{4H2a(}b_{*i=tH~c0`WT&Bls@3v`ZfMFK#15h;r}l!@`<&W4!C7PwT+7>xQob~94Wf8s2J>H9r8+6*b#svAjXd2xlD>81^7dOSmDL!SF zbo(a)Rpi;YV31CK+5e0x2W5Pk2CDN{JC1Q- zQ1c>vW%tLW&CQxcb#6@!Y=1D!@J6e{gWbASGae=71kLDb(|g~LDD}6&vZebOvwHG$ zR}BW_f?I4Gi)=-z;NFmb7`&Ka$Sc$ImbC<)cpj%2V}?NpPBaFRMvwFxTF_g9F&o-U zI~(hY2~6HF{bEgu+Ksh+!Atbx=13GfHwL6j({ayB2}uDc$g7Q6jpGj zh1NRk5eZSq*J1ZGqKw7BcZ)r$)b?n5BSMfc;r*57LlADRm`SfL>a1QLz6(+~TX+6^ zYv6%RSMZ-O+Is4#z5gXxPxVZx-EBGN)_2PYw6H#y# zwj8%5gIXXWVRw6)z}?kQ!pFreHe{}-GG*^`;LO3R&0+UNAI$O9gP!Qff>i-g`wRhyH8bC8 zQybJ_jO0l2VN zWIEG$3FV}muz;|#S%BQ}NX$8D|E!^S)Bgi$g*a$xkE$GMFln2TjV`QJd*@amb!=A6 zNMxFT5lW6DAs_A;k%o`;v$pjcU@$Q}v2PY%j+e-cHBzc8BN!WD@pCj36k9`!^Iv94 zY*5$=(kBq#Ghs#V0Gw38dR8&VrkxAxB=oG7aRBY;LIukF$eK8PMxVkAUiK2$kuRnE z7m$5|3{+TH_FMREk6-?9Fa}$Vnng@;At9 z(*KokGM4<`gp+KS`~NH9B>4~FS%XN$z&`f1RvkxtO-C&i&&MVlgI^JjO>u|_8rc4E^_ck{wCo81sR)I zo0qsfZ?sa|)}i%y~OEUX@mAc77lH=7((jsaj@Y)6F!bC zh;>mJ5)(;>PCu^9QC=XVsK%K;i!_Wd2QNAvN{5}$$7?(`@nx{6)SEG|4I`0On(YFI z{|1}Vo%BhbD*MA0yhNiNmaoAt;!87Av))V%v8!56YRq3(LO)7Fl9~$FmWklvu4Wmk zfEEmNHoz^nIx#3XKhJFo*2v;p6FP|ftV#YwE@6d%6x=`$>cgDT``pip3^`15nQ*@U z=^q*hMR=&SKNL5Z&f*&SYtzxD;8<6#&c#kZdztfQ2cK8WD(@~s86~-I(|6n_$v6D(HnzH$Fi6?0ru4Yz$@^B&DWe3cH(g-1vu4QZhdZ zB)cutzM@io804O{RdnkfXcr+3o6NpZqFh#Sn_hkzWu2sQQ(0nR%_zfg~_5SWjKtnU_v=Yf5 zj2>khscV>s$}}hq9r|dG)!GRc<~)8L$Hh#ga%)?hZL)F)9!buWvv(^qM#FS=W*(7N zMxI}`p=UUt-tdkd3c|2lOzshc%~p%v#vOhcB=w9-EG^NS>8pHqJi9 zdMh?c#VC&XBAi#op%K;4^?9g>=4sazbVo!LbgBl_wP-p$*0T`&3$P0n)LZB~0Y%dX z*0bDbnT>(kkANc(MukA3X7qln&qkwvoiX`gxPHcPAAoP5vnC^=7bagIfSyh=?_R)B zT#P;1fT1wm)f-#L;mbzPB<9{&MJ5A{w8j=W8mD`JBtg=g+e_+#h(UBHU77RE0XaCc zX2zsP#Jyt@%pNIHCU7RLP;yO{Z9+GNE$n3U9VnA#V5y|bY36&Z2z$q!Nv=B!>icLzi zQEdGO3u;BQAvs5!PN=)xP`#0WWtuug*PL%kz~A>!B_W~np1qt*l`nJ{`(Kp3Wmr~Q z7d9#-jWhy+BHhy6($d}C4bt7x(%s$N-3`*+-Q9H->VEe==R1GCA3PTq&tk0^V~)JX zy_B?5B1TxMhq4f3bV=Qcb86~!cLyU2u>ZeUAD|OuH;!9RkMK3Sy6FD$^q#|szU3nM zWHFo#`BlFL*+2R~R)#5rmzZqQ?!vubb z{JRGhhO=DZX-(wZTvXrH14n+LiVuR{Hf7=7A@pokJ4-FECZ(JUY?ILp-Xe5NeHNx2 z&TeVJ)EXNsV>mFPaTVuwb9?)tQGRV~2!)LMp`=ARhpgX}oMSK-bhNv$(e!I5Ow@|y zmu(Aanb?Z6MeJizI7NkYnZQlhGqM!En&NF7l=Yy zHOK9V@QP)M^ivCZf5R*c&nzhbH_GsxjXmLx?^rJ2OdKJllK;x@pim71t>VEzo{2Pm znSaZ_wydP?ed8guY)NbNZQC;Bv?|Di^97>56(N%##`-v3!SjBkw``apj|4H`v-xQj zY;Y_OReY80De7cb4XlADM6K9FegT&lOjD7|R`UCuvvmmQ@8qvNCV`Ow;ccsUrvv1r zyBwb=(1X;Yh3ELAqNCRUTx~}TrAnH-o^+z?Sa=&ZA&}qw$bDqHURp@Ch`)43u(+bF zN6aQ3<$Q#kz~v$#&j#A_w&r+zYGa93so^TVa~#B+Th0hcn#fdeXuLVEn9GlH!81v( zGAVv5Cl;~LcWYjL%k5au#7Z00sm2;x zA#w^HJx3N1(ZQ`lk8j;P*7xK`3lu_-Y|eZj84>4_>ZDc!qE{?jVnhN*0MMjsRcE}& znJ^6W@Rr}CI|u7}622_86>Fx7szI~^L4s%PHoxLUs78C_+;p`8fv;v`43rnveX1G( ziv@-!BgXp6NG;T&csWY6# z(O}4+QL;D#_(xTLEigZwBwbA|J^42#XA2*MnVl+(wkUI@_xa@EctMv0R<+%?mtts8 zK6_4s*P)o1e16(dm)G-Rz3(9HP>U;!jjiT-#ai6hVGwcDy^X292)8{2&roo~yatCYDx|2p z=f%Z+xDkT0xPL=!UaeK^A|^rB1Y07Vw)&kW6gTp?;^BgQliYpP;p@l;lYuEmmRG#R z@mody^7AS9SdZKwjZPZR)!}uS{&j(&q*d{3IRiGyZnCx&6V4bXL#0%_HP|Yo`}6F8 zxZ8CBpEZISW%+jV;r7waV=t`tuP}_rpwzBdfu#0O!JVLUlEM3tpCS&TTlPFXF|76? zj+O=7Ek5x4lg8MHTpzJ5C^~fPRYSORxu7u#_<+(33(Odc@%=TuXG*eE_oQ%}DEEBD zO-^!Q$>TymhajJa2lY`$>=(W_Xh9C);L}A)IzKwdY!*~T8sEXe(U&yrlRZ72u643b zO`8oFHNj^7YMjsO_3J=CpzpKIF`+vbNwz~O-l>~mRuxz{exzjVR6kF_TiFg#jFqZr z6cS@V!0)fDDv(+BDat^v!QQr3EGid>dH=~|ry@kC%-IZLaSw~Z*Qs8vS+ z>T)tMlR-V5WJXkND0B286l`cwk?@*y#N?vx^DQ2uC?OH&6RlHK_nTNtx$ zqA1D6NT%hN*OlZ<2^xaas#^=V6x2)_p;StG-#3il?TIQaeP$b6ta0sr5D^MEFk1kJLY`O)V=u^MqsJKW>oyWnUCAmC}w+6OJD8I z{(U&Ecfy~w_Qpbn-NEn#;C-KHXpmA}&V_?+W7X_1sahYcA9n@0IvQ*bsx%TArx$ zN2vUhlzC4N&iQ!CN25Q|TUgpsFj`@ZNu_F7Z_^G`ZvwC_nvl3&?nY@G&o@H{aS;gy ze4J^+$Q2_+4(Lb7!SomeTchm-1M+{Ds58EO-}E9{qO5tx)JYGPpPAPD!cGFNv-$pK zN%K8DT9z$Nd{dQ}-`@cyxteKk5P_i(fuMjW%O+@^M*PlKYMvtA0&P?I7^n`9XT3;3 zS5#Tqm&29Tj~e&mv1#Zy%r@I_e(w})cDE9pxZgz4mYXdwqN1VMmVEo}LE^Azs;kTH zX@#=@VUKvBjSH#um=%f^8YCQ`^_tO5YlN~Kkg*TfWmQAoHQuctf1#q&^xzS0i?v%5 zjpKp#_ci9@7Smw;zWeh%<;S)`>egUVnVeN}jXRj*h6Adv0s=Kk@rNxB$DQrokTmYk zE{98v7^x47JM&rjxRzr-?8;B^pl|uPn=4(FOubL+T2HTq3CK`gy<83T=cj_w$zSk< z#bV69iE;Y&rYg>Eu^+Y%^!|Q!`+)xzD3BdK9T%nEi|$i*K4NIjDmSfzDx3-g`W5X1 zcOUOAoOCnj9!kVf(ukpda)%+UHa7SEvUBdJO5e51`evDi__yV+`98m!Rf)M`m?sxKv9V?HI0LL3AIJt*r9(FXK0l;@9lbCdG|Q5H2oS6u5Wl zABbFLr!~ZwQr!-CDFP9&D_}wsQAb^YQsk2CHR9adoN|2KR*XuSb~nr`+V3Y?f(yF& zY?&6PzaIDRPFT-t=wE&ru2e3pyU_F-_6zdu<+u8(*Vp0%rLHRSr@JOQ(fNCA0^gE9 zzWdt8yR%*qsi3uQw&z4VIZ^yK^^oS&BD~6*i%;;$`8Y zgY?>k0QzvV+k($yRX_Jg9M75l}^K&i?qh|l(ULD)j z4^ji5=y-yZS5PosNKNX!7lO^Cy*(7~aI<6|9vkZq^k%6=(T+Jw(-FT;;Qb;vGf=`l zh|NCu!R<=JcnE5(R~^;Hm#IfLw+1Dzm6$k2(G%}_-g(-G8p;Q|0}|jn+bG94$;$DO zl&k&&N)DBOGX;9M0&|Ga!mR7QnwlURldv&W=jhBpiG0AFgON#9WCtyF&O6wyh+FTx z(8PdGIGiv>O7RsS3M!IUNB6l}{S9+v=6!tK_w{j;j996Fm>|C91Iui7#LZ69?93Ek zx-eQ!eR2DP0{%2St9}3pl~nye?T1SMKH%02h9LU(cnr2)6tY?KUBonH2cf4YR1*Pc zzW;zKp;^ysFpCaG%Mg1J@dItv{xW#56!&5MFTnFouq%6`FZ^lwfgjtdXuev(_yY5A zfyA+{qeO@!P4L5_P&2LC+aIADla{&`OAV|(-yGj$cX$85ni3yfN%pu~bP-1{MYX?1 znGi1qiTfDq?UO~i{WIfhb(5Lv%%t`%5fsx`)m&r&Z)Nk8P;jR>kA&$gyDIg1>ukmX zPS<;SyQ6u2y$t_)I>*<|sCy5}c`Z&wB^$??rk4FtLf;KUw!G`b|wwo~V zO#N7yBH#|;*wg_UU;P=ELbRqcCL(A(F2KP=kRoO@-a`V%RuPjb`RQF{xse>>_<3OY z@p)~TNDk?6WOx|8Gt%=$LhIUtO^oIgWssH26aAq&J4oSSc^L+arP&d-BAQM&4hD?X zV?cdGt5Oc)kRl?4X`KI$brR;05uSPy!Zq4zzIv2<3k-KJiXReHZ7HVwn&I%mg*ui8 z7zXP(eR{Lo-`-xa27sbAxj;?PZavD_M<|H*B-?bV+5?DBIffht$i7uEI^*YSBDX3L z0ZvnvxWxn5(ng3)M-oIyd_-Yk;k@GF?ssmdU=$iHs^lAd>4y~fuT={;W>AhsBqkYR zpi}D3%E4m3et)1fnZ%95n4~}AsG-;|vmJ9OTr2v4AafvDSuK505R>25RzLt1j$k%~ zs5Vu~h(LtpXB@Fa(sNCcbSwajBU}TT@=|_vlcLvCGbV`G5Q@DIaLg7^Qwx#87nJf7 zOPQ1b3koPEZ;Y^=(CgZ;Yye1; zFI~3{y%0JNdsbaT9^j7vY9L2r_MQOwcf$y_GWctE#C@kd;_+RFI;3KclY#?G6EZF5 zCmBy;NvTf-f?e=LiGgQjIYKv&tE3!idmT!j*iDOI(+(TR6*9e>rr|w#jE^HIOdLm8 z#K>6m3rw>ZU@|x3P*N#5J3l2}Wn_OF&MUPLP`soy9d^Q!C+IU~#I!tA_7$XK!%aQi zCd-&fZ4Zbv_@Z~sTeJeucL3e~3DTPcO6BBkxOO-;p%OpVeb(RdUClQ4idvO~V-q@} zo(~@Ul2q_!jdzmU>zgu_t_&rU0Gdr1K^Yhr9Dqy_-hj_C)B5e%hGY%0ap0#vE&tpv zL;#P0`^>xS19Q-;-ZDM5pOvQaGo+Akz06}chq;T4C}bj(ZiyRjGKdpdmvFH0>M*on z;vF(asB^2@C#)>;C4p01IOsT2uD&1!9K~T9qCIGoTIVhyG%{DU`%&3sb02YF-Jl#7{f`^#^d! zEa1V2+U%Erc?AiI2B5jiveqyt87(Cv)C_Zja02tD7(cmjGA=9n^d8q?x%astV}DH* znzOAmn(Xk82!Zy;npJdj|4h_Uhs5;O-NV2%d{*+O-^MjpZ4{~D+ZL_pvTzeE(k8XT zREdF-)_g_6Ou~SJUOWK#TqTW&QiqVkSjX_Z)w_P#+oZxRRZvrk z8bS9x`rzQcpC+RXg5Eqm_Wbt3+|-Bg@Kfg3l#A!93`TJ`H#b+W7c5RLDk`5Y9nmD; zARETMaZfzb-iv*gFL^m=jG4r1d#x>*9Q?|OA>*rKDZN)98^SCj6?PVxw1J@JiyQo$ zT#aq-=nB1oFle)aBVajNDXME&%a%r z`PYu8qx+NT@l-Z@1dHY?bhnp>73Y!Efe0R(?V&LkpMgVwkfaBmX|atEuQ8e(1>f^o zbQPEfX^PvuQFGt~Q1e`Lhsf6bbE@4?R-=ep$Ka2Ch*e-Ol&@_uCEKwQUbDak;}WrR)~jl4*1g!7sE_en|(;A=;+1}>ldjnO@daE zB0lqR`{SZ)%jD&S7<>xPd<~#}mTRs?7%e`S9Z=5AyT&sng;NC~B~v2e3hJcT$(K64 z*Arpn{u%H_ppRw*29bI(+1Ze%=XWtA{(K13O!h;Bx1VV{EPsUfy)!J@S|I-u0`$Yi!c z4_0>R(jIu+iph_2c)8z?OZf9RLBeLsH81&3ZUA4{2Eeq!Z)8L|>0*%0od@6z=b&JW z-BZCdAU1;96{O+p%fV-BsA3dHeH5HUU3S=3c$}OX)NyMJtoNQ}=iC}nNo5E;k3YR) zd?XK(>0hhNJ8to!!otu|py>zEmPK-z>z$3#_R2 zc6YASIT1UAA~}W3_eYv>*0ui-!6P$t{#?4c`L5wsxN$?=fB<5#2$u>gov0H z?JxMz74>(ZsonKsGzk*r)tycNf4&=a�@bTUii=8=lw7{~g2`KwJ7`7~EYu?gPrQ zY~kp(_2q%f>>CaaPDtZ%gD5%jM2^^j+}&+_Rz#LcoZHIP0jSVrrK!q$Kzdf#*? z4{EBi6ll{&*mF&{S&#&Q`{{K~pU?=krPJRt+fLheHzPK$q z?j+Z(2a>afEU2Wo&|ISLrN_QonEq9GaZEw8AYnhQt4#3A6}ZnKXADO&GFo5?(W3X> zEB#L2@NoCb^U2G;o|;0Dq9=b4`kq{T!$MjE8-RS`vzOGxIzrN1Fb;!By%X^`>IL={ zC~=Cup7X6ICF@MyGo{@t9@T0IJ{}$ryOZT&GyIHQy2_G5Bf;+O8)eAWwwST3>Mwt&?A8tO3%t1gTaiDS3Inq9O_oEUT+21)Z&d zXvG#6mdPt_D;FBLs01tr>k6|g)8r`yg)~k>&>nxd5nqrwH>aAXFSbBiE1=t+BMzP% zVz1p|hR(F1@@$u;=yO!e>mlIb;cur(-F&&8&%zWN?T$R8iKudh-~g7ZUtv)6xZbvh zx{sr5S5HDDfxLfkuBlLPBcozZ0c9U6T<-UB9iH!YUXuXDUkL~Zb|0R#RgVP2-}?fL zdTGc|P+gYcu7_o-p30yDVt@X<6A04AIz6a~?g04Jx**p*5|KnoaI^Ex{(=3nBcw6p ze0th9Rj0jd$<9RfzP81;vW@SPO8yJJ%v~xK;_wk5JuQDqL|VJ&pa9B(di&}9av$dW z(Z-HzkUeepASI8^6pVVcHjZ^ti0?DX;quAXEz`b*Fbc&DTL~?>qYA#u!k>>;xC{(o zmQJTF>;(y@K32&u8&qow4lD^$@dqPjkxo}SfA1Q+E+xGGLFx3j^n;bnn%66ECYq_}l>ZS2+`-aGeX)S;lxsTI?Pe44> zrygNpMwx~YpfvlADE#n?ion7dr+V9qDY-NKP>tEKL@9-F&3j#KwoDgcTX9BPT>0&I zgegg%cu!NmiB$&8dxbT2&c`z8Tun?j+}st85Q(9^vxMJP-UHkRXyLxnqGPr{{va(a z{SyxdngPaYf07jN|8m|DYEJ{Up;@Bs58CnRawA4d`pL3u@F%ypFiwoGZ(nD0+*7^b z&|Deh$>gBz1yYfE3(gM_o~^a3_od+KHE{MT4G0slrfu%WFJ!bBVclrPDaa*;cM{KxI*{vPxr29r6&_^3UP_!A-SCezg^j(QE5+yXH(jdkXQ`-|xV`w;pPye9N zG=)%Bq4+<_y6foRznU?Q&r))75dNK>o1401D6WziLlNNj5gyCdv}A8JE?oqn+9_$| zkx0?$a>?{M_Fln>Et2=#PkUS$+#{s@VXV>628ceC&$tm3Z6$~JDdBLj3Ol(#vz#C~ z%82GGE|Flv7G5k*#$F75*`VnkE;1A#WRKIlywtO?p#zk=_s8c`|2PH{bb#B!tE{Xn zWh7<~ASk>91*sm(y62WuQ#)b-IKJ?o4ts$^#xL|~$zF<WXth0WZ!WS#?s)BjML%_yiaRj$^?N9)5bnS>1DUbBN$K^*?x~OB|4i)}j^^ZIaER z7Vbs|~g?G49%_ME_ZaB}%h zo5J>Y06)2R0Po}b$PV2@fp{Ol*P+&QzhBrlG}HxfxonLl5xrWE-2@LT8p&jLTaxTH z*CWoxR8kcphJsc353)_aen|#nF^G^u9al=DPue;-)NzvQwY#Xg-EFDCPQl}O@gr>g zg9T(n9!3)^liB9w&t7`VV3$4>Sh;fZ+h4*qXYNbF~WL9 zWodE%kjV9gm1jm=07VfM6}4JWShC-~825b?C9iG!hi}~ZC#w!X8 zz|dGg$R`h={zed5_!Woa`%6;TQC*E&a9A2)Pm{oUM1w;mPDbXJr>Xz?H}wVDn*e+sCT83d`Xm8^43Dj+iLZdU1EY0sY8t zd*l9yO&FmG1`2`nl^YR6tn3p6bsDsBx-u`4D4}Ku3{RD66L&hHnRj42S*$Wv8H`BV zXCZib9zq@-%DnVB`qr>c5{%Yghw-<%+8m*G*cX}LSMzt zMHsA>zv$Zuo0~Cd(cIj6Igo@AExhrBY*6!wnw$@^atO*RK6J7{?m~LBz9<;1LKsw~ zO3r3|kq&7X{o%Ch{CPokpU3p;8&747vKKQh+kM&MSU!*`jMbX*DJF^`t+5P#ELzMhi|9om^GC#}8(ClC=chumhrxnN?DjmFI znjM z>H1x(MN6u(nuG48h3neWFw=l~rlX$&_WEY>(AH=~>v6*sG{j3e-)y1ASViEAmI~_R zz#)?n6w$!3hI6|%7|il}O9Y2$7Va-n*Y36K=3#g4xaw1=%#P<-Cij0#_Ys{nw%_rAy?*b7NdHd9%?a zn^D6hb>(;?H{GPYs=dZA>?ZMM9p`c!_{mvlN?lw{Y>A-wH3}sP-h0-Pn3vMWhd!R-^I=RD|A4>U1U8-CALMfSWAI!L!M4QcsiSO?<(DXWOVgdZ>6!SsKD#kNL$Jiks00A zP~-BuANq07`zja+awk^FRX{gfKT;rab>Dc$my52A6zG9ux-3ny0T4R5f@FO$l|mi+ zoI;~M&|;1{G4DgZVe1+)in3b29NitRklCm2>znw2m+sl)M!k$Gg>6(wfx_&0m|Axd zqJyFu_%ESr2?pmdPt4MI8%_slCVolHN{O2@mjE`Fx`lR!UvE5W7q{h;#~VHSTojB3 zSz5DWVVkdc=0`HOyLt@9Qe`4D{ewks5R8gDw4X+q@*VO14CxK10W~+bRgB$rP4sok z1)JUBuZvMG_n3OixE8_S5P)Un2MH+=iFqs}$C~wv^1|=GKPEH2ML{cJ6j!$&n@&wkL zLWBI-R>V9S5YWKNw5oS44SP9>^p5AcbE8dIW!&5D-vygc*_srz@Ud-g>ye3kNjPc0 zMq-r6%zMJ=Wm(L0-sC3er28vfL{E!G)M#G2D3C8OIvl<$i zHn7%%t)1@A7zREaY9ePiazfa@j^yWfeg-u6nha0V;|MKe?cu*o*g**%U&TER4@4Qz z$T*@3ZUy%1_olVY!PZ$gI9{{d&V~ayh+k&4OF2NkeK+19<*a=ALET0M<=Idlk#ai3 zEU30Q+;7NH0UF~;d~M^c$uaXyi)1(9vApcdF?!pnLPG|K9woh3^ex_)k~gVCjxy z2pGnTq?!d{IG-xCQTCDCHI08(v*k@tMWLbLN5vb!mav%+6*%OL5|()1KjbJCPj9Pi1j7fMM?~&6fAIa&^PUH}jg1YUTf<^6QiAjBx#yRc2sAy6 zu{jZS^aTu*H2DS9jo$T4FN3Q`{Y9L;Kk`;V1={IpPtwD0%X@M&5=aENP(a>BE4#+ zKk7*un2L1+2NRP*GMoLhNojMAdUGJIsr(4uA(>!WhDP`(%g%>-)e>uK%~)!f*^6zU z(`k#bIm4!K(R(i7x17BJok9jo&DLqLTBCe2tL^7FlHNQbd%5ppijjKZ7_a){j}jS% z0+h&y-dGJcB_LumU?H9L{d)u85)TVl;M?Hm+Mx1fr_{A7b(AYiT_Rvu?!R(RWAuLk;y{7)$M6u zc2NJXto@$*-)|Yy>HvDz!TGJi@ zW_mWD0t4eq^vHmVt6X$h?}FCT)7t~K%OD=iA5XHOZGWn;zT0Mx-t_MRii1Ou3}~89 z6+|C)n5r}2e{nd~^!4%C1(5Ai_A9Ombny{Eqp)4Qy$3*rX0mdW^WXyg&7ZxL)OcN8 z;AKH$e5pVR{P9Eb`}e9t$`o&J z@9@7e?dgOJj2EoM`Q~`O-p1h1J+8&ssatL{00z#im8WC-3Z6uG+QaW8>QJ%*ZEx=~ z1;eJ%6jayPiS>I!ag0}5T=ftL;r|NQWW*Z~e$29KFJGiUTFZj^IZjWPg&MPafPJ_i znp~kkERpg4HE&&QGE>o$kx&DFc3M?MKb*uOqoD4XZQgt(Z?M0(-0XbSg(5-n=~EgW zV?5&@TeB($%I=>Le9UUS`GeJVKPxyG?)a08pOl!{E7wv|+j{EDV=f)p_fB}GUEmKf zYW7+W9>?KjlUfz!40&WYvpoGLkqO zwi8S)fuMn$f}$u%_OL)01iP)Cn&3#|VSlDH)l_4))%E@oudPt{)M;dJ%X$y6b~FAT z-SUPLnb=MMTBzNIgAyAr5p2czDw>{G5_M96*5YGbyN>Z=u(+H3vH}wC(*`K)4WJ_WVUE*$EVS^&c!OV*Sn0;EPuzr z%=S8t#TY8Jf>+fBP>$+Dkj@?DgWAbM-eBJJM zJbG&=ks8WS1WOSZN_@rG6QO{YRIJ0?Y69%<1-6XH%|=5v&g6!qBqxYnfnYS)Pw! zch@l{?40%W^#g(|Z+z=;Fw5=nDr%{#L4?n4!rU(L4GL5@dP0(YH@jcvFYVIHlLPfu z?7nwB%{@H<#2KkJ*lKL7lJjiD#Jx%zI$%T(HKP}0?lv|y$*gluXzmM|9codX%B}U#QFs~NZre1ne~eQ4xNoLX1M z9D+&L6OXOY-zntxuPx*DM%VPo2wY+VYN2Y@{e5u{Q#_1uvfKfQjB%iLf`*Ih2KdPD|8vnYC9o?Y&kum^ zxdOC->POeha6rw;h9ohg{%7q@P&MKC8SaZEQaJ(k`st(Em6Awe33+^Cgg^G=kCVdo z>TT$^&RwB0rWsD;grHL>N)9hFfB5f>kb%WXc-q?8$-TXGFq^C32YUF5^vyqn{=0%6 z_R#d_CHqeLW{LDh8B8lKGJr!M2Pl08AO$sAL{iQBj|YRuo#qDtMPdwf*#S#E2|)Ph zHQU|*I$m5$ON$|jZtk!1@p|jkdeEZuytS6ODx*k%VLu0;%wv|AELl73NBGb1!jOS^ z!rKeJJU=nmAFJ-zRRC(QaBcDbbezAl$u%s2%W?w$I|)ymB|83|KqFVaRGv>T=kL3JPC-=U-kgb2U%_H z465eKab=2-qW@m${CXv$`>H2oQh4A&vOLV&0&jya?-NVC-;%Xv0*B@v$jX1eJ=IzQSZ4=@hKg39?EZ7xRTkj3xW6#O66nVO@5vS@pcGoH zw9Gd<*FsJPlKppy$^qx2Tfi0|7qb}|lab)qD$1?{u!l)fL9C>4yuW?zKOcF1z?BTA z#v8wY2}@A7K315wKG{7zow%2VeD3=v!~m}K#6khWg%GJ}QvtjA2>|*ndbKwp2oP6G za&vPJqjQ`8D`rrW9)M0_@KD57`o1^LuTo<=27nLsa=BfB5m!sWIez|kizF!l2{JyL zI&QZszXmFI5+PU&1DN?j-@X+sf%fD4XOGBWfQ1Uwyo!DVM0)~AQC{}phcTbr7eH?P z_kOB?6xI3pyVtyH$7VE*JFFyE2QqN*8~9NpLD|Md$Z?tGJuAz*MXBP8^L z4rd@Jk#FkXqtztofdoT+NM8{AudI0FLVe*~Bjw=X_|FEO^1eRz|F{rQ(m_N*^4k=R|{C9y>ySyVm6lMxn8nR;X?tP-R|LuM2TmkNhD zm?m?7*}Jm+cucj^BalNyo!{9UIZ`&?VqC-cxN9d?CxteBqj1DIQCVtcv!i}HgPI9K zbkfqx6!(;`--s=4yjWCe4)fycWJ}U5#_t*tStHW=JaMDkaA_VLQ&CpN`e=FY=%{&s z#nE^QF&ro%`D4#NKJw$_Pj$0oEIo6z>>cU(gY?FSg02^txr^2OtA@R?N4Md@Z&T!> zl#}EK5mh^uqF30QyizxzveeG%a+6`+^F|X9cTTmZT|>QwOLweK+KO@WDOSsqcgO0E(4t@fB{0sV6gX)+4cCQ|i-qe(}lCNV@tUrQfh zpR?kbjJdn~joss+iFi9F&x3?~tK7KrN2o{ruf``0UBj0Y84(R#c+zUPKBoV)6Jv0^ zi_gSdK-o52FH(X@ZI);0E?AlcM;SRIicw^;Ey!_m_&RLrc1I&n9)%g`{zbY{KV;(K z1ec^{$z@1*>N16r{)=^VNiwU9jdm~j?JX<(;j*%nuWf&+MMYyvP4N2GChqjS(iqmF zUd*`YW=b|ZUGDx}R3=bk-)}VNROJ`6_#(y2^9m6;e07d*!3}z_~}&Rv1w?LRpZFX|G;wR3(Ea~4+wd9N2jdM}4$)*EPw%Kt@-5P8t+NwDcFb4^(7NsT$bBtvA33C|h z$|@Dg5dJ$2>%5x@vC_Y4*5eaK=h!e3b@BZ9CFM87Ehn^>0tS>#Etc9b4AdiTAOK zsRttOO$Jh86yq8n_|7<4VrWahoZvs)G}xUixhr!5FJGQ{Zf+FgI@aayzqed>LSQ2HAz5|t<-_;xI5(@2;lTT`%bBL}kxZ}Wkm?<(|2e#A zp%DA7KA&|qL<48o;i=+wA}dyR&JS3WwLFWbvPCg$O=oyHa!ak;?v@%>(IIMZ)8MxU zn4P#y?(TezMCD_qhd&gwp&y-3)S4Sl3sXo1Ouc7OVQ?;?x2eMnDJvE5Y?yt#JKfO5 zVb*m|uI2h@KRxdKaHTo=hy+*3(>pAivR-P=hdV`Uqlf#@6WTGXq=GWkCs^mkWtM>M zpJRBvPbc_g=mW60>_$-&_Cvj*KlDdmMWLoyJRWf3krV{NPHspSZ^PYbhq9n{^MvIcVPOW#cHxVdbD!$Yl5a4hY zedsXTy2=!bigZ>E^oVIFJ2m9|pLb+SE}1VZP_2;tq4{iz)-N;Ysu({Q)RUNyo&BdLDdwQ1q|qrNtVodPadU?+4O$Msw37X*#Dol|o=&3AZVu z4P89EApOY*pBlZ29p}p|Gloh}^c&=VCWyyp^|e!za2SWOMlqKpQEzw5zlFbuc*v9b z*@~X%p|>5V3!geZqZ>^yjnmsai=fk(vA5%XoNSBRA>KAUbS^D-6 zku9L*ELJfzw20QJui)l~KbMkihAJtWj+8bI-$lpPtA*7-jrnl0Re#rIiVSm2c2xIU zWD(6Ix|WIMo>0<<=w3J5+h3~n7EzJj(3KkC;%PpftS~Ai&9FD&k8?|AJmG|Iz0$ve zIWB!>bRB6qe;7-Wt;u$7STqL=UISbWit6rE@qBinjXL&Woi@$0yu=Qn5B{=5EXpf4|IAyG?$1KJK$;HKIO* zRk!QPURW9>Wh79?LpfS18NzyZGa~ZBk#m?SXj_#Qj6&Qp5q=ViCOcN6RiZ>UCXO>q z!YfF}`eijVUtcA-#}^IUM4To2CLsPgyT7$xWAekVHRY!Pa0TJ~l{nZV4rMLBZ26vf z%?^FUv=)0VeQU9?`IiP~)2L%Cti#P^tv1bxcjLE6VBbWX5tcH}wh34g5_8ooT6Jr> z@-!6OafBq8@KMYzgJvbWdt211p7Az@E-Iu7Ta@=%bD}~9hZH3OKR?(;wXPTvW;b?{ zyI!YE@Vn9Ssix<7i;;g*HaM{HWu<^=F+ zI*q{N54yYv7i4vBOeV!d!wx4`4R#E-J(e3f51}#HBnn?>d9Jr59QOhe;*d+COoB9i zZIhTis8=H#=cM|p<5iY$lMCn@``n>94>3Gl&X*>GiaZhHyL$7MJO$aLCp0iF*RO_s z!LTi7v=s`x;8^qh$}ws05TLOsIy|I0Qj%!0tsSq>m}0R>ENsOKRlIl`tgq@^Uf;Ex z=6#(|9Vg}1L1D9#tFZO)-sFl2AF=mPNMH!=imX^rRP~}}JgR!TMEU5g$@LFZHB-ZP zI|hmY?jbIHQ2e7mtmobDK^Cf44pVdS7l=?pnf2ns{l@gO)&dypf!`hhQWehfCpOvS zB#-uOQvvS|Ua>E0rA-#C^e}4)GZ_v6AjTGUqj#I4V%Vmk@Oznd$2Ezc2Aou#(w+It z*XA~YJxG<8RK8;9p(AIUYJ*4^?da*>+qoXN$9p}zo486}3v@UZ&gNQF1M^oc)mg|G z`sIn4OnRi3LtO7lCsVsCV=MP!o{Yx0*aEzf0Ta)0`x@~IdrYQx8IH8|qgbDfc33i~ z{a2a$20{Gus0RI*27j)M;&5SbwCpR8g}<>fnVm$9L)q8J>36A|m5< z_I0a=Y;Jw3c`;`y4bAX731Ynr);+vtHuXcLt^ND8h8cd8%ljV^o3wM{BQ7ob_c&PC z=qa)q(Q^}&Se_Z@?u@4#e-MSxP6#95+J;uX+_F8uv?i?ThP)k#_*_zcvGmSRuj?~P zaFML_`De;`)8(LuYO5`Jx^)_vmwpP?k|GI5qX1g)K$S+N{vdRtN>^fw>2+cKkUkv? z67n>lK|y00a#WZVKPc&Fa5y>jbM!*iDNNIs9#;IbJ?wde@#E-7dRxsOpVza+^AFjsNOb^mSmmVxquwE8i*SRYEB|7X7;u`IyZhp`Aqk-$8 zd~T%=T0Fu2I43WjDEJ%8T2dhnU&qi+G3)N8)TPAsOS75RP)c_Zm1@sJ5l8^Vr zJ#^I6e2sCACGq=}FPBq2cDQc5KKjK6*+$+C@zr&M!;4q&2~~Dh&taFxh2QMA=QK#z z7ayIPtx9_|CPo4f6Y6fZviFtf+}+QPIgcst+`*g-3!rg7yD@1fgwk@|>D$XCkQ(^a z*CD7Z^kkJ|wEACdh0<5kn;;)%AB~7n8(C%r5cN<^JUyw9nmq7cu-X4W{`jH9i*0P) zM|y=ZN3QbnJ8!}L$`Yl!3g*kRK|Qn`@5W^!kE`h8`by&*Ex%pBFF&w)tY&6$6z6xc zaWQ-K{jHLXZ|KZ+Ho(O`FpnrTBI$qRUkm0&ovQ7+GR(}`UoA=}Q3@F8)EROLsizIw zCgHr5p&i3q`L>evf!>5*e2SvffwbOua&k8I%y_E5h;2^BGG9$THOi`}l4X^4$>71H zj{>69UABF=vidmwY()EZvkiD%&ti zSb5sf(bfkV6_P7Yt`h_MSZBW0ws6_G>)Q@i^$_$|P$!>+d`Xlk!h%>k&TDd8IO8mf z+UD#&5PgEKqVn>IaBDw`$Q%K~$H&n7iR7oL`7SR=d$TdqME&u3FP=Q>dm8r^h5om5 z7DT7?-VCfxO%AC^@xhe;WE!}Op(fANvS9-0YLRjD%oVTN7Sg@ z5j0TN8xkniCz1N|i1W=GY-8cCJhCR{okk@ZqT5F12Uv}?^<}sL6NhGB70}(NQ*K7O zGlwW8chJaZ7E`8{kLl`X#rR?8X(-D##lBu#u~+Grgk}{{G#wQ#=t`?erWJAax0)R`A=eIx#bfNl1W?vZSzMdXu-lBFMt?jOnT=mbqYJe)&&1j% zGqz}PH*IIBbT%_BcL}!n#?zXJ-4xOfbZ+K2sop~M=9luxAxvcmpZ<5}Z@Eqiywaf`h;#R^~ZGxpE>(7{yHiYDaU114zCvTQ z6#2@M7HbE>hpZIl0XFKOf~eeJuz2I;8o7Y#&-@Q}|I6s!#&9=-qi!L6oUIZ`O>+14 zyJV%CB{^Zf{vS>Vkag}QSO-;ZDWEz3$G_P<06|198CDCte}&BE!+$brQ2!UXH*W6# zoAltVCWjkP2j&+PECdpj_dv0w2zbTBN7=Ogf6k+aJuHw&w1085JsRbDap~^r+HHM# zDg|B_lmGha33B-&Nmur!r9!-X_J2k!8TMM8#yul#$ZC772Mb3Mv4C`IJfCAMS$4t` zqMXA2vm4xP8$elry+kzcCr}p7MEvk2gFtST18Idsd@;P^zsmJ5{4}AMAkGW^=`?{~|oO7Mm^PDR| z&MB$%$p~}v%a?_xqxdtZ+chRnJk9yE;}w^2)uw_PchCss0{MRED4++Y0k=bG%+y<< z3TmQw<^rTcqo(*imi9o&#P~V?zDkF9f^Gj@BWO^f4$a0JSk)(~X;yYFNOhL8Whw~M zC&@bGiKgT9#l^+@s+`jaF00P|tI4yovsxl!J1GwrpjywwfhMvj1{)a;k3<~!5G{Om z9%{QL0TVJ_oF1%%Jb$6Jx;HiL8py{3S##E)Xp2FnV#`KM)bZJ{BpJJXKBw^jw>-Ya zoJWz5A;Z2ElCi=^mqxVd7QDEB{P?kgWZ|KiLSG_ZSEYlovppS_s%_O_0OZ4b3~uPB zfmC!Pzmlo@rv>OoKYB<1j(`e=k)40uVw)*7@k~Snf!#sp&0APxt_@s>QeMGxT|hA~ zC`?N7`>$R9IrE%d)^htXs^4Xb#fvmhO#q}KAI?4Y zbS2<7q*PF|+7WSLP|0B05vAb%4G%MFBGqF^PhUT}#%&^NJk95>SxXwL$Sdlb<%Xd& zpm}laJ2w*>n~|E@O^h$u5D|xJ)@I+Ps*VCHGSn|-Pkjc@1R4*oS>$b1iq1e`qaDzf zh$?N}no1;O3gMp+y7?a1twDmD`J(e1frb9%Gz9;EHFIdN{&4T zeHVIdutto`)JMH#Rno?DNR5I4DTln z!HRN~$7AZf?O;vy5o~l@xYcy1CW99-8P~`4$TeKHAkJ}J=;4frkghocxJWxFH${lX zG%ayLlX?LoGhASmj}~s@M&=q6prkbSP4!oG>-w*pKc%6e;2?dx>H~2L<+1TfAQ|-$ zgvS~7Tji_u+8FmV_XFpp4}=F6YLZ-y@eBjwG88kT#S0U8Ft~)tKq~_XF3^hI^OUjE z1HILfvA(5%b>Q?0Mm4GS+p`vgj;h5yk$$AO-To;fEFBA`UkTaH|QB_KVT08k-8;I}Y>HZCQI;kWS&%!59 z`6172M5J>Xtm%q3v8Aj)nLRKp48%m4W>ef7K&Gel5yd0=Zjn)mKFHJv5(P*H&}PeOb9RCo_wJbp?=>E~ zy^Fj%5TQ;j*H_}BqoXT)SKYDSdfL7C7C{HcF$TtMn81Kn$TY_B>YcFU{o0-#Is$Yi zz)E6*g6as|pw6z817yw1ZaZ5Q$}a@KFEF~%8LDO+rgJBqH#bjFUj24td3kx_wq)sh zrNx2gomG-Sr};L91}}?f=h?!SQTC^Yvm&HB0GKEN1k3_B8NP#70dpC3fO2NIGYc70 z+)2hR+UZ#XHnv)h8HT`=lwDHLXj4~-96NWf4vu(NpbgpGJp5NKcnl`ri6usb4l86O z5M2eX>6xNhuURXI>!t{{vb!tb(nh$~#0zEt_psZ}HHa5lsRUWKrIo-TD(k@e#kj5i zTr7g->8uTdG8`Lgkc0`4-G+7631=eKsM^}%Ka^S&A*YoCeOCaC;_z{y@4(qrV&v#I ztbKsl;(SL%Ge;Pa9^~rCF=N@ z3{X^)j@aC7SereI1vz=aZlH0g`M^AwHy^ZA0W)RzvuPZ}>K=CEkI;2(EB6TT{?W|K z@_oxyUKe%%QpQuBJ8$XA`RwxN2XH!IX~%0)7zEKmR^PXffiXPB@`Qc2LJpiX@Kz_> zz9UCdZ{J@R@floIY|wLQ6i#y()L%FLghiO35)u+xNb}vR0$Hyp)Dwr8Zoo3@=ZSK5 zT7j)k7Wy2>q@pqsFum=dGM7QEz*qUkAO5P zj(`+8=%dC%lf9Abt|%7LrET(?81?AUSN)j&pmG^`vg?YGo5 z%SxMfd|5-KQwwE(68SXJs~MaT8A`434&9}EN36H7zbWFhIQ$2lY*IvYV8TbdEFD?G z1g^qeK)X%7ig8in$Y^J ztxY#ZM7smp$q}DiA7$t^;-eU4VfcP6ZzOOm6fgJt>D`j5XpBGw@Ubi1CKWI@ld5kV zrObmuBwVbt!Sm8#B&?hdt-?7=W(z005)%_iL-kPEJ1LDA&a-AGD>r;x#&MWzvu-_L zcWtCxdauS$^xC9j(`-0*?J5SNKmz=Q2)G3qBH!=DbDvc==IG3R7$z?xfAmfiUvU8I zJ~(;7|5#@x^s>%p6u_au6Qw7~I}cg)pL_mimCex8YKY-7p?ncu4O!bv1Xa6Ff>FU#f zph^kLCR1!c3~GDkOwLe-G}96ui9hBBn0<U)beZNSyW70ZU?4gN3VG^yFmiUs2Z*T?*gkXF&_w=*f@kEA5i^U@vqa&} zHR*(Ue|}(`GJ)Q)W5=Ux9(RRp9-*@jxdOlZw!ognfROO_%Nl^&FAhj$`BZwdw%6X)GL}2y%E!c!ewEd^d1pH&wq6)svA1SkMdtov)MX2EDKg{MUbh }c zlNdXxPZ=d~WRzlsmSIPh%F+}1Sy)KusP5Qf2Wh0_y7ztraK8rC(&H(vYz{Jo#YS!c z>FMb@sS1V_fx5q=7iD-ipN?2)t8?1+(53q>Bm!PGy#MR3-ZsBHMt;wx9rbv}qAB`A zZB6*%4~Y{qik-KRR7XG=n3E+f-A)}8?&L7Am96S?aL`NRvB?AAxPd53_5Zc}nc3xp zxwLMu;*X+xVJNstH3&#=+j5JuFrD^@i)+f2m-{=RF9t5XgEVWKug@%_ntIWbTDl`mq0c|2J zUzed{^1Ki8*)iZ^w`q^!DPWz95YVdsami9}z4z{LOY+)LB%A@_MMJEvdJ)+J7-(kb zWosZbF$CsH!XhFhaPlPZn*g>M<>JmzD3(7+BT~15Ll49D5RJptfoXYVkyhBgeg|!^ zDM}q#l?Nbbo0(q_~+L447VTz&6lCK_J$vFk}Wv#8FQ)Uut$7cV|#MyEY4 zHO-K3cqbmGrf_{kv2xnAEQ*t-AC3-7ywr$}-eoFCi zS!QOA^-b3whwMTk%cVG2SPfktD|l#8_F8)u%P~oOB?Xg@C8eZVyhO{A3c~{Tt}K7k z&Wm;Qt+UF?$}g21aPIW|(ae_XpDy`1*Ur21Zq4P9o%RCm@p(G@InLucT}{(c6%taS zjd3RZvd79(8KNSLVor?8UM6{B`()30%unL+!u_c-p(+-cUtdeTIsEaiZ;dKJ#P5<; z=}SpT6W`%gHKP;dyUz88D#9^2~IwYiNS0+|N17U!ec2dxl@=lB3?nA#jBtD zi;ABAe>(aHJP57$pNTsQzj;{mIR5r~Gy2H%uNcM4!GZ~C3Lh95rKPPIdBbu3;ZKEY z40|QW5`H*zn8>^2ydTC@hwh&!#2kEj@^!_FOHXgL98!IrCuHmBDS8!Wb}(8GjV8_h zS0n5lnNs4#Sb^{&;-8f0$gt z`sD7f`4bKu%s0NU|M7p&+hg3L_>t>XDO(PoqB3cXlm|IfI3F)ych+`?g64@iF7R{``u#LJ)V_Uc)J=&T!5P2)~xc)x?=A{ z-t}c!d}*tHzS`p~jG;{pwf*Fvp5nC3l|Jh+Ztc_=BCjC68ovLhtEb{AkvoRZzHGBd zSqRG-*RKb($uSz*9aM_ey#QFAg$6mDTO@o$eMyx!1q) zWzUMUyC?lYZ3-||Q*$KQChnIvW3A+nQSgazLI>YM3qeM+3T3p=Bz z82Dd^bTeiLRPw#sXR$a&XMu4ftX^(y-_-ibB&hcF?!2+@J12>AFD}}>C&yRENOl_$ z2O{1q3*4$(3psk^p^oI^8JZ2`0euc)_2z?1NUG7`R1+q|vVBzoHo)40HJ=;jg2G3t zV;d4KoyPG7yLqWi1Z!?07MdmGz0x^s=01Thiev~r(;ih{eCL_WvVi`_3pv%BSGyW4 zCXClq?EQ0NTzaP+Z8EDZ9MkGmOeDFc+ph+Pj_GzONKHN~`}HeWVEIoNqgoB-^z&>v z!js}vGk|EOFKF!vEY-y*B#BrBmyJu*&OZ_!@@0ck0$nr8YK z-S}a&VobO}L0xKR#jmN7H?PH2zKUJnW%)bU!5KmDtlUqQ0S>%S5}hji_Qaru{rbxo zK9xQCo4B%Ainbn>FjC|_`JY#X#kzQ3a?iQ@n_d-It`VQfWMAT%f2XdVBChqTRV{MO z;#9iXL?$_zOb+wod*6%_Ti`ydxpOf}cKM3}d!MAAKVIa7^~(9TYgd@<%Q8P%JDu@# zE;%x}KU!q0j3CzYR^X2>Y2_F`w$0{VIi#v_dKtgcvaPOlCnXl7)M+p?G3%pM?HcEv zbSBuGL&+M?>37BsIY3#V$gLUUdtS3xrv_46nVi|Fq2CVi0c@i2cedIgApi@65cSe8 z%e?(|MZ(+nBp_R)KMtFkg(L5Wy+_+_)gS-3Q}`flu>2xFmP%y8x1~W8PO9D{#t-k1 zlKpT7cRKU`+!sTT>O8IV+Zk^>w5k#~a-u6EnwqqRSvmyc@TsN#^x!{^9B%_*rS~+= z?SIChDH+-A{GUO5k57A>k3^Eq8E|83$oq*`$w7WaBmEq4`lu+D1n_7ynG*5 zt&JKys|G|ePu+xB|0(j!m_}auP|DVS2F3Ou w+ze};q}{)j9at!S)?57Yub!bA#JRK{S(CQq$VI|;8u+JqO6O#$n$^w!0WwA*=Kufz From 785ac9e0afceb4b60e49e82c024bb8ebc7376b39 Mon Sep 17 00:00:00 2001 From: gongweibao Date: Fri, 12 May 2017 15:09:41 +0800 Subject: [PATCH 66/88] fix bugs --- doc/design/cluster_train/data_dispatch.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/doc/design/cluster_train/data_dispatch.md b/doc/design/cluster_train/data_dispatch.md index c82e7b558e..5b70e802ba 100644 --- a/doc/design/cluster_train/data_dispatch.md +++ b/doc/design/cluster_train/data_dispatch.md @@ -18,7 +18,7 @@ 我们选择[CephFS](http://docs.ceph.com/docs/master/cephfs/)作为存储系统。 - 无论是从[PFSClient](../file_manager/README.md)的角度，还是从[Pod](https://kubernetes.io/docs/concepts/workloads/pods/pod/)中运行任务的角度，统一用`/pfs/$DATACENTER/home/$USER`来访问用户自己的数据。 -- `/pfs/$DATACENTER/Common`下存放公共数据集合 +- `/pfs/$DATACENTER/common`下存放公共数据集合
@@ -142,13 +142,15 @@ endpoint=datacenter2.paddlepaddle.org ### 文件访问的权限 控制用户权限 -- `Common`数据集合只读不能写 - - 现在mount到本地以后读写权限 +- `/pfs/$DATACENTER/common`数据集合只读不能写 + - 现在mount到本地以后有读写权限 - 用户可以把自己的数据分享给别人 ### 文件访问方式 不用mount的方式来访问数据，而是直接用API的接口远程访问 +例如： + ``` f = open('/pfs/datacenter_name/home/user_name/test1.dat') ``` From 6801c1b5945d35d1a2ddbb1e04408441d542f563 Mon Sep 17 00:00:00 2001 From: gongweibao Date: Fri, 12 May 2017 15:30:40 +0800 Subject: [PATCH 67/88] fix username --- doc/design/cluster_train/data_dispatch.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/design/cluster_train/data_dispatch.md b/doc/design/cluster_train/data_dispatch.md index 5b70e802ba..e7798b24f6 100644 --- a/doc/design/cluster_train/data_dispatch.md +++ b/doc/design/cluster_train/data_dispatch.md @@ -127,15 +127,15 @@ paddle pfs cp random_images-*-of-* /pfs/$DATACENTER/home/$USER/folder/ ``` # config file [datacenter_1] -username=wuyi -usercert=wuyi.pem -userkey=wuyi-key.pem +username=user +usercert=user.pem +userkey=user-key.pem endpoint=datacenter1.paddlepaddle.org [datacenter_2] -username=wuyi -usercert=wuyi.pem -userkey=wuyi-key.pem +username=user +usercert=user.pem +userkey=user-key.pem endpoint=datacenter2.paddlepaddle.org ``` ## TODO From 3f347ef539e5e6a81e5ba2747d2fc306b15f759d Mon Sep 17 00:00:00 2001 From: luotao02 Date: Fri, 12 May 2017 16:50:53 +0800 Subject: [PATCH 68/88] fix error cmake infomation --- paddle/utils/DynamicLoader.cpp | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) diff --git a/paddle/utils/DynamicLoader.cpp b/paddle/utils/DynamicLoader.cpp index 76cf3c3006..5aa1fe12d8 100644 --- a/paddle/utils/DynamicLoader.cpp +++ b/paddle/utils/DynamicLoader.cpp @@ -104,20 +104,9 @@ static inline void GetDsoHandleFromSearchPath(const std::string& search_root, CHECK(nullptr != *dso_handle) << "Failed to find dynamic library: " << dlPath << " (" << dlerror() << ") \n" << "Please specify its path correctly using " - "one of the following ways: \n" - - << "Method 1. set cuda and cudnn lib path at " - "runtime. " - << "http://www.paddlepaddle.org/doc/ui/" - "cmd_argument/" - "argument_outline.html \n" - << "For instance, issue command: paddle train " - "--use_gpu=1 " - << "--cuda_dir=/usr/local/cuda/lib64 " - "--cudnn_dir=/usr/local/cudnn/lib " - "...\n" - - << "Method 2. set environment variable " + "following ways: \n" + + << "Method. set environment variable " "LD_LIBRARY_PATH on Linux or " << "DYLD_LIBRARY_PATH on Mac OS. \n" << "For instance, issue command: export " From 20aef7febf0d33128da6d2c485d98eb1f6e83e2e Mon Sep 17 00:00:00 2001 From: luotao02 Date: Fri, 12 May 2017 17:49:41 +0800 Subject: [PATCH 69/88] faq --- doc/faq/index_cn.rst | 614 ++++++++++++++++++++++--------------------- 1 file changed, 313 insertions(+), 301 deletions(-) diff --git a/doc/faq/index_cn.rst b/doc/faq/index_cn.rst index df5e172252..081e3a60a2 100644 --- a/doc/faq/index_cn.rst +++ b/doc/faq/index_cn.rst @@ -1,301 +1,313 @@ -#################### -FAQ -#################### - -.. contents:: - -1. 如何减少内存占用 ---------------------------------- - -神经网络的训练本身是一个非常消耗内存和显存的工作，经常会消耗数10GB的内存和数GB的显存。 -PaddlePaddle的内存占用主要分为如下几个方面\: - -* DataProvider缓冲池内存（只针对内存） -* 神经元激活内存（针对内存和显存） -* 参数内存 （针对内存和显存） -* 其他内存杂项 - -其中，其他内存杂项是指PaddlePaddle本身所用的一些内存，包括字符串分配，临时变量等等，暂不考虑在内。 - -减少DataProvider缓冲池内存 -++++++++++++++++++++++++++ - -PyDataProvider使用的是异步加载，同时在内存里直接随即选取数据来做Shuffle。即 - -.. graphviz:: - - digraph { - rankdir=LR; - 数据文件 -> 内存池 -> PaddlePaddle训练 - } - -所以，减小这个内存池即可减小内存占用，同时也可以加速开始训练前数据载入的过程。但是，这 -个内存池实际上决定了shuffle的粒度。所以，如果将这个内存池减小，又要保证数据是随机的， -那么最好将数据文件在每次读取之前做一次shuffle。可能的代码为 - -.. literalinclude:: src/reduce_min_pool_size.py - -这样做可以极大的减少内存占用，并且可能会加速训练过程，详细文档参考 :ref:`api_pydataprovider2` 。 - -神经元激活内存 -++++++++++++++ - -神经网络在训练的时候，会对每一个激活暂存一些数据，如神经元激活值等。 -在反向传递的时候，这些数据会被用来更新参数。这些数据使用的内存主要和两个参数有关系， -一是batch size，另一个是每条序列(Sequence)长度。所以，其实也是和每个mini-batch中包含 -的时间步信息成正比。 - -所以做法可以有两种： - -* 减小batch size。 即在网络配置中 :code:`settings(batch_size=1000)` 设置成一个小一些的值。但是batch size本身是神经网络的超参数，减小batch size可能会对训练结果产生影响。 -* 减小序列的长度，或者直接扔掉非常长的序列。比如，一个数据集大部分序列长度是100-200, - 但是突然有一个10000长的序列，就很容易导致内存超限，特别是在LSTM等RNN中。 - -参数内存 -++++++++ - -PaddlePaddle支持非常多的优化算法(Optimizer)，不同的优化算法需要使用不同大小的内存。 -例如使用 :code:`adadelta` 算法，则需要使用等于权重参数规模大约5倍的内存。举例，如果参数保存下来的模型目录 -文件为 :code:`100M`， 那么该优化算法至少需要 :code:`500M` 的内存。 - -可以考虑使用一些优化算法，例如 :code:`momentum`。 - -2. 如何加速PaddlePaddle的训练速度 ---------------------------------- - -加速PaddlePaddle训练可以考虑从以下几个方面\： - -* 减少数据载入的耗时 -* 加速训练速度 -* 利用分布式训练驾驭更多的计算资源 - -减少数据载入的耗时 -++++++++++++++++++ - -使用\ :code:`pydataprovider`\ 时，可以减少缓存池的大小，同时设置内存缓存功能，即可以极大的加速数据载入流程。 -:code:`DataProvider` 缓存池的减小，和之前减小通过减小缓存池来减小内存占用的原理一致。 - -.. literalinclude:: src/reduce_min_pool_size.py - -同时 :code:`@provider` 接口有一个 :code:`cache` 参数来控制缓存方法，将其设置成 :code:`CacheType.CACHE_PASS_IN_MEM` 的话，会将第一个 :code:`pass` (过完所有训练数据即为一个pass)生成的数据缓存在内存里，在之后的 :code:`pass` 中，不会再从 :code:`python` 端读取数据，而是直接从内存的缓存里读取数据。这也会极大减少数据读入的耗时。 - - -加速训练速度 -++++++++++++ - -PaddlePaddle支持Sparse的训练，sparse训练需要训练特征是 :code:`sparse_binary_vector` 、 :code:`sparse_vector` 、或者 :code:`integer_value` 的任一一种。同时，与这个训练数据交互的Layer，需要将其Parameter设置成 sparse 更新模式，即设置 :code:`sparse_update=True` - -这里使用简单的 :code:`word2vec` 训练语言模型距离，具体使用方法为\: - -使用一个词前两个词和后两个词，来预测这个中间的词。这个任务的DataProvider为\: - -.. literalinclude:: src/word2vec_dataprovider.py - -这个任务的配置为\: - -.. literalinclude:: src/word2vec_config.py - - -利用更多的计算资源 -++++++++++++++++++ - -利用更多的计算资源可以分为一下几个方式来进行\: - -* 单机CPU训练 - - * 使用多线程训练。设置命令行参数 :code:`trainer_count`。 - -* 单机GPU训练 - - * 使用显卡训练。设置命令行参数 :code:`use_gpu`。 - * 使用多块显卡训练。设置命令行参数 :code:`use_gpu` 和 :code:`trainer_count` 。 - -* 多机训练 - - * 请参考 :ref:`cluster_train` 。 - - -3. 遇到“非法指令”或者是“illegal instruction” --------------------------------------------- - -PaddlePaddle使用avx SIMD指令提高cpu执行效率，因此错误的使用二进制发行版可能会导致这种错误，请选择正确的版本。 - -4. 如何选择SGD算法的学习率 --------------------------- - -在采用sgd/async_sgd进行训练时，一个重要的问题是选择正确的learning_rate。如果learning_rate太大，那么训练有可能不收敛，如果learning_rate太小，那么收敛可能很慢，导致训练时间过长。 - -通常做法是从一个比较大的learning_rate开始试，如果不收敛，那减少学习率10倍继续试验，直到训练收敛为止。那么如何判断训练不收敛呢？可以估计出如果模型采用不变的输出最小的cost0是多少。 - -如果训练过程的的cost明显高于这个常数输出的cost，那么我们可以判断为训练不收敛。举一个例子，假如我们是三分类问题，采用multi-class-cross-entropy作为cost，数据中0,1,2三类的比例为 :code:`0.2, 0.5, 0.3` , 那么常数输出所能达到的最小cost是 :code:`-(0.2*log(0.2)+0.5*log(0.5)+0.3*log(0.3))=1.03` 。如果训练一个pass（或者更早）后，cost还大于这个数，那么可以认为训练不收敛，应该降低学习率。 - - -5. 如何初始化参数 ------------------ - -默认情况下，PaddlePaddle使用均值0，标准差为 :math:`\frac{1}{\sqrt{d}}` 来初始化参数。其中 :math:`d` 为参数矩阵的宽度。这种初始化方式在一般情况下不会产生很差的结果。如果用户想要自定义初始化方式，PaddlePaddle目前提供两种参数初始化的方式\: - -* 高斯分布。将 :code:`param_attr` 设置成 :code:`param_attr=ParamAttr(initial_mean=0.0, initial_std=1.0)` -* 均匀分布。将 :code:`param_attr` 设置成 :code:`param_attr=ParamAttr(initial_max=1.0, initial_min=-1.0)` - -比如设置一个全连接层的参数初始化方式和bias初始化方式，可以使用如下代码。 - -.. code-block:: python - - hidden = fc_layer(input=ipt, param_attr=ParamAttr(initial_max=1.0, initial_min=-1.0), - bias_attr=ParamAttr(initial_mean=1.0, initial_std=0.0)) - -上述代码将bias全部初始化为1.0, 同时将参数初始化为 :code:`[1.0, -1.0]` 的均匀分布。 - -6. 如何共享参数 ---------------- - -PaddlePaddle的参数使用名字 :code:`name` 作为参数的ID，相同名字的参数，会共享参数。设置参数的名字，可以使用 :code:`ParamAttr(name="YOUR_PARAM_NAME")` 来设置。更方便的设置方式，是使得要共享的参数使用同样的 :code:`ParamAttr` 对象。 - -简单的全连接网络，参数共享的配置示例为\: - -.. literalinclude:: ../../python/paddle/trainer_config_helpers/tests/configs/shared_fc.py - -这里 :code:`hidden_a` 和 :code:`hidden_b` 使用了同样的parameter和bias。并且softmax层的两个输入也使用了同样的参数 :code:`softmax_param`。 - -7. \*-cp27mu-linux_x86_64.whl is not a supported wheel on this platform. ------------------------------------------------------------------------- - -出现这个问题的主要原因是，系统编译wheel包的时候，使用的 :code:`wheel` 包是最新的， -而系统中的 :code:`pip` 包比较老。具体的解决方法是，更新 :code:`pip` 包并重新编译PaddlePaddle。 -更新 :code:`pip` 包的方法是\: - -.. code-block:: bash - - pip install --upgrade pip - -8. python相关的单元测试都过不了 --------------------------------- - -如果出现以下python相关的单元测试都过不了的情况： - -.. code-block:: bash - - 24 - test_PyDataProvider (Failed) - 26 - test_RecurrentGradientMachine (Failed) - 27 - test_NetworkCompare (Failed) - 28 - test_PyDataProvider2 (Failed) - 32 - test_Prediction (Failed) - 33 - test_Compare (Failed) - 34 - test_Trainer (Failed) - 35 - test_TrainerOnePass (Failed) - 36 - test_CompareTwoNets (Failed) - 37 - test_CompareTwoOpts (Failed) - 38 - test_CompareSparse (Failed) - 39 - test_recurrent_machine_generation (Failed) - 40 - test_PyDataProviderWrapper (Failed) - 41 - test_config_parser (Failed) - 42 - test_swig_api (Failed) - 43 - layers_test (Failed) - -并且查询PaddlePaddle单元测试的日志，提示： - -.. code-block:: bash - - paddle package is already in your PYTHONPATH. But unittest need a clean environment. - Please uninstall paddle package before start unittest. Try to 'pip uninstall paddle'. - -解决办法是： - -* 卸载PaddlePaddle包 :code:`pip uninstall paddle`, 清理掉老旧的PaddlePaddle安装包，使得单元测试有一个干净的环境。如果PaddlePaddle包已经在python的site-packages里面，单元测试会引用site-packages里面的python包，而不是源码目录里 :code:`/python` 目录下的python包。同时，即便设置 :code:`PYTHONPATH` 到 :code:`/python` 也没用，因为python的搜索路径是优先已经安装的python包。 - - -9. 运行Docker GPU镜像出现 "CUDA driver version is insufficient" ----------------------------------------------------------------- - -用户在使用PaddlePaddle GPU的Docker镜像的时候，常常出现 `Cuda Error: CUDA driver version is insufficient for CUDA runtime version`, 原因在于没有把机器上CUDA相关的驱动和库映射到容器内部。 -具体的解决方法是： - -.. code-block:: bash - - $ export CUDA_SO="$(\ls usr/lib64/libcuda* | xargs -I{} echo '-v {}:{}') $(\ls /usr/lib64/libnvidia* | xargs -I{} echo '-v {}:{}')" - $ export DEVICES=$(\ls /dev/nvidia* | xargs -I{} echo '--device {}:{}') - $ docker run ${CUDA_SO} ${DEVICES} -it paddledev/paddlepaddle:latest-gpu - -更多关于Docker的安装与使用, 请参考 `PaddlePaddle Docker 文档 `_ 。 - - -10. CMake源码编译, 找到的PythonLibs和PythonInterp版本不一致 ----------------------------------------------------------------- - -这是目前CMake寻找Python的逻辑存在缺陷，如果系统安装了多个Python版本，CMake找到的Python库和Python解释器版本可能有不一致现象，导致编译PaddlePaddle失败。正确的解决方法是， -用户强制指定特定的Python版本，具体操作如下： - - .. code-block:: bash - - cmake .. -DPYTHON_EXECUTABLE= -DPYTHON_LIBRARY= -DPYTHON_INCLUDE_DIR= - -用户需要指定本机上Python的路径：````, ````, ```` - -10. A protocol message was rejected because it was too big ----------------------------------------------------------- - -如果在训练NLP相关模型时，出现以下错误： - -.. code-block:: bash - - [libprotobuf ERROR google/protobuf/io/coded_stream.cc:171] A protocol message was rejected because it was too big (more than 67108864 bytes). To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h. - F1205 14:59:50.295174 14703 TrainerConfigHelper.cpp:59] Check failed: m->conf.ParseFromString(configProtoStr) - -可能的原因是：传给dataprovider的某一个args过大，一般是由于直接传递大字典导致的。错误的define_py_data_sources2类似： - -.. code-block:: python - - src_dict = dict() - for line_count, line in enumerate(open(src_dict_path, "r")): - src_dict[line.strip()] = line_count - - define_py_data_sources2( - train_list, - test_list, - module="dataprovider", - obj="process", - args={"src_dict": src_dict}) - -解决方案是：将字典的地址作为args传给dataprovider，然后在dataprovider里面根据该地址加载字典。即define_py_data_sources2应改为： - -.. code-block:: python - - define_py_data_sources2( - train_list, - test_list, - module="dataprovider", - obj="process", - args={"src_dict_path": src_dict_path}) - -完整源码可参考 `seqToseq `_ 示例。 - -11. 如何指定GPU设备 -------------------- - -例如机器上有4块GPU，编号从0开始，指定使用2、3号GPU： - -* 方式1：通过 `CUDA_VISIBLE_DEVICES `_ 环境变量来指定特定的GPU。 - -.. code-block:: bash - - env CUDA_VISIBLE_DEVICES=2,3 paddle train --use_gpu=true --trainer_count=2 - -* 方式2：通过命令行参数 ``--gpu_id`` 指定。 - -.. code-block:: bash - - paddle train --use_gpu=true --trainer_count=2 --gpu_id=2 - - -12. 训练过程中出现 :code:`Floating point exception`, 训练因此退出怎么办? ------------------------------------------------------------------------- - -Paddle二进制在运行时捕获了浮点数异常，只要出现浮点数异常(即训练过程中出现NaN或者Inf)，立刻退出。浮点异常通常的原因是浮点数溢出、除零等问题。 -主要原因包括两个方面: - -* 训练过程中参数或者训练过程中的梯度尺度过大，导致参数累加，乘除等时候，导致了浮点数溢出。 -* 模型一直不收敛，发散到了一个数值特别大的地方。 -* 训练数据有问题，导致参数收敛到了一些奇异的情况。或者输入数据尺度过大，有些特征的取值达到数百万，这时进行矩阵乘法运算就可能导致浮点数溢出。 - -主要的解决办法是减小学习律或者对数据进行归一化处理。 +#################### +FAQ +#################### + +.. contents:: + +1. 如何减少内存占用 +--------------------------------- + +神经网络的训练本身是一个非常消耗内存和显存的工作，经常会消耗数10GB的内存和数GB的显存。 +PaddlePaddle的内存占用主要分为如下几个方面\: + +* DataProvider缓冲池内存（只针对内存） +* 神经元激活内存（针对内存和显存） +* 参数内存 （针对内存和显存） +* 其他内存杂项 + +其中，其他内存杂项是指PaddlePaddle本身所用的一些内存，包括字符串分配，临时变量等等，暂不考虑在内。 + +减少DataProvider缓冲池内存 +++++++++++++++++++++++++++ + +PyDataProvider使用的是异步加载，同时在内存里直接随即选取数据来做Shuffle。即 + +.. graphviz:: + + digraph { + rankdir=LR; + 数据文件 -> 内存池 -> PaddlePaddle训练 + } + +所以，减小这个内存池即可减小内存占用，同时也可以加速开始训练前数据载入的过程。但是，这 +个内存池实际上决定了shuffle的粒度。所以，如果将这个内存池减小，又要保证数据是随机的， +那么最好将数据文件在每次读取之前做一次shuffle。可能的代码为 + +.. literalinclude:: src/reduce_min_pool_size.py + +这样做可以极大的减少内存占用，并且可能会加速训练过程，详细文档参考 :ref:`api_pydataprovider2` 。 + +神经元激活内存 +++++++++++++++ + +神经网络在训练的时候，会对每一个激活暂存一些数据，如神经元激活值等。 +在反向传递的时候，这些数据会被用来更新参数。这些数据使用的内存主要和两个参数有关系， +一是batch size，另一个是每条序列(Sequence)长度。所以，其实也是和每个mini-batch中包含 +的时间步信息成正比。 + +所以做法可以有两种： + +* 减小batch size。 即在网络配置中 :code:`settings(batch_size=1000)` 设置成一个小一些的值。但是batch size本身是神经网络的超参数，减小batch size可能会对训练结果产生影响。 +* 减小序列的长度，或者直接扔掉非常长的序列。比如，一个数据集大部分序列长度是100-200, + 但是突然有一个10000长的序列，就很容易导致内存超限，特别是在LSTM等RNN中。 + +参数内存 +++++++++ + +PaddlePaddle支持非常多的优化算法(Optimizer)，不同的优化算法需要使用不同大小的内存。 +例如使用 :code:`adadelta` 算法，则需要使用等于权重参数规模大约5倍的内存。举例，如果参数保存下来的模型目录 +文件为 :code:`100M`， 那么该优化算法至少需要 :code:`500M` 的内存。 + +可以考虑使用一些优化算法，例如 :code:`momentum`。 + +2. 如何加速PaddlePaddle的训练速度 +--------------------------------- + +加速PaddlePaddle训练可以考虑从以下几个方面\： + +* 减少数据载入的耗时 +* 加速训练速度 +* 利用分布式训练驾驭更多的计算资源 + +减少数据载入的耗时 +++++++++++++++++++ + +使用\ :code:`pydataprovider`\ 时，可以减少缓存池的大小，同时设置内存缓存功能，即可以极大的加速数据载入流程。 +:code:`DataProvider` 缓存池的减小，和之前减小通过减小缓存池来减小内存占用的原理一致。 + +.. literalinclude:: src/reduce_min_pool_size.py + +同时 :code:`@provider` 接口有一个 :code:`cache` 参数来控制缓存方法，将其设置成 :code:`CacheType.CACHE_PASS_IN_MEM` 的话，会将第一个 :code:`pass` (过完所有训练数据即为一个pass)生成的数据缓存在内存里，在之后的 :code:`pass` 中，不会再从 :code:`python` 端读取数据，而是直接从内存的缓存里读取数据。这也会极大减少数据读入的耗时。 + + +加速训练速度 +++++++++++++ + +PaddlePaddle支持Sparse的训练，sparse训练需要训练特征是 :code:`sparse_binary_vector` 、 :code:`sparse_vector` 、或者 :code:`integer_value` 的任一一种。同时，与这个训练数据交互的Layer，需要将其Parameter设置成 sparse 更新模式，即设置 :code:`sparse_update=True` + +这里使用简单的 :code:`word2vec` 训练语言模型距离，具体使用方法为\: + +使用一个词前两个词和后两个词，来预测这个中间的词。这个任务的DataProvider为\: + +.. literalinclude:: src/word2vec_dataprovider.py + +这个任务的配置为\: + +.. literalinclude:: src/word2vec_config.py + + +利用更多的计算资源 +++++++++++++++++++ + +利用更多的计算资源可以分为一下几个方式来进行\: + +* 单机CPU训练 + + * 使用多线程训练。设置命令行参数 :code:`trainer_count`。 + +* 单机GPU训练 + + * 使用显卡训练。设置命令行参数 :code:`use_gpu`。 + * 使用多块显卡训练。设置命令行参数 :code:`use_gpu` 和 :code:`trainer_count` 。 + +* 多机训练 + + * 请参考 :ref:`cluster_train` 。 + + +3. 遇到“非法指令”或者是“illegal instruction” +-------------------------------------------- + +PaddlePaddle使用avx SIMD指令提高cpu执行效率，因此错误的使用二进制发行版可能会导致这种错误，请选择正确的版本。 + +4. 如何选择SGD算法的学习率 +-------------------------- + +在采用sgd/async_sgd进行训练时，一个重要的问题是选择正确的learning_rate。如果learning_rate太大，那么训练有可能不收敛，如果learning_rate太小，那么收敛可能很慢，导致训练时间过长。 + +通常做法是从一个比较大的learning_rate开始试，如果不收敛，那减少学习率10倍继续试验，直到训练收敛为止。那么如何判断训练不收敛呢？可以估计出如果模型采用不变的输出最小的cost0是多少。 + +如果训练过程的的cost明显高于这个常数输出的cost，那么我们可以判断为训练不收敛。举一个例子，假如我们是三分类问题，采用multi-class-cross-entropy作为cost，数据中0,1,2三类的比例为 :code:`0.2, 0.5, 0.3` , 那么常数输出所能达到的最小cost是 :code:`-(0.2*log(0.2)+0.5*log(0.5)+0.3*log(0.3))=1.03` 。如果训练一个pass（或者更早）后，cost还大于这个数，那么可以认为训练不收敛，应该降低学习率。 + + +5. 如何初始化参数 +----------------- + +默认情况下，PaddlePaddle使用均值0，标准差为 :math:`\frac{1}{\sqrt{d}}` 来初始化参数。其中 :math:`d` 为参数矩阵的宽度。这种初始化方式在一般情况下不会产生很差的结果。如果用户想要自定义初始化方式，PaddlePaddle目前提供两种参数初始化的方式\: + +* 高斯分布。将 :code:`param_attr` 设置成 :code:`param_attr=ParamAttr(initial_mean=0.0, initial_std=1.0)` +* 均匀分布。将 :code:`param_attr` 设置成 :code:`param_attr=ParamAttr(initial_max=1.0, initial_min=-1.0)` + +比如设置一个全连接层的参数初始化方式和bias初始化方式，可以使用如下代码。 + +.. code-block:: python + + hidden = fc_layer(input=ipt, param_attr=ParamAttr(initial_max=1.0, initial_min=-1.0), + bias_attr=ParamAttr(initial_mean=1.0, initial_std=0.0)) + +上述代码将bias全部初始化为1.0, 同时将参数初始化为 :code:`[1.0, -1.0]` 的均匀分布。 + +6. 如何共享参数 +--------------- + +PaddlePaddle的参数使用名字 :code:`name` 作为参数的ID，相同名字的参数，会共享参数。设置参数的名字，可以使用 :code:`ParamAttr(name="YOUR_PARAM_NAME")` 来设置。更方便的设置方式，是使得要共享的参数使用同样的 :code:`ParamAttr` 对象。 + +简单的全连接网络，参数共享的配置示例为\: + +.. literalinclude:: ../../python/paddle/trainer_config_helpers/tests/configs/shared_fc.py + +这里 :code:`hidden_a` 和 :code:`hidden_b` 使用了同样的parameter和bias。并且softmax层的两个输入也使用了同样的参数 :code:`softmax_param`。 + +7. \*-cp27mu-linux_x86_64.whl is not a supported wheel on this platform. +------------------------------------------------------------------------ + +出现这个问题的主要原因是，系统编译wheel包的时候，使用的 :code:`wheel` 包是最新的， +而系统中的 :code:`pip` 包比较老。具体的解决方法是，更新 :code:`pip` 包并重新编译PaddlePaddle。 +更新 :code:`pip` 包的方法是\: + +.. code-block:: bash + + pip install --upgrade pip + +8. python相关的单元测试都过不了 +-------------------------------- + +如果出现以下python相关的单元测试都过不了的情况： + +.. code-block:: bash + + 24 - test_PyDataProvider (Failed) + 26 - test_RecurrentGradientMachine (Failed) + 27 - test_NetworkCompare (Failed) + 28 - test_PyDataProvider2 (Failed) + 32 - test_Prediction (Failed) + 33 - test_Compare (Failed) + 34 - test_Trainer (Failed) + 35 - test_TrainerOnePass (Failed) + 36 - test_CompareTwoNets (Failed) + 37 - test_CompareTwoOpts (Failed) + 38 - test_CompareSparse (Failed) + 39 - test_recurrent_machine_generation (Failed) + 40 - test_PyDataProviderWrapper (Failed) + 41 - test_config_parser (Failed) + 42 - test_swig_api (Failed) + 43 - layers_test (Failed) + +并且查询PaddlePaddle单元测试的日志，提示： + +.. code-block:: bash + + paddle package is already in your PYTHONPATH. But unittest need a clean environment. + Please uninstall paddle package before start unittest. Try to 'pip uninstall paddle'. + +解决办法是： + +* 卸载PaddlePaddle包 :code:`pip uninstall paddle`, 清理掉老旧的PaddlePaddle安装包，使得单元测试有一个干净的环境。如果PaddlePaddle包已经在python的site-packages里面，单元测试会引用site-packages里面的python包，而不是源码目录里 :code:`/python` 目录下的python包。同时，即便设置 :code:`PYTHONPATH` 到 :code:`/python` 也没用，因为python的搜索路径是优先已经安装的python包。 + + +9. 运行Docker GPU镜像出现 "CUDA driver version is insufficient" +---------------------------------------------------------------- + +用户在使用PaddlePaddle GPU的Docker镜像的时候，常常出现 `Cuda Error: CUDA driver version is insufficient for CUDA runtime version`, 原因在于没有把机器上CUDA相关的驱动和库映射到容器内部。 +具体的解决方法是： + +.. code-block:: bash + + $ export CUDA_SO="$(\ls usr/lib64/libcuda* | xargs -I{} echo '-v {}:{}') $(\ls /usr/lib64/libnvidia* | xargs -I{} echo '-v {}:{}')" + $ export DEVICES=$(\ls /dev/nvidia* | xargs -I{} echo '--device {}:{}') + $ docker run ${CUDA_SO} ${DEVICES} -it paddledev/paddlepaddle:latest-gpu + +更多关于Docker的安装与使用, 请参考 `PaddlePaddle Docker 文档 `_ 。 + + +10. CMake源码编译, 找到的PythonLibs和PythonInterp版本不一致 +---------------------------------------------------------------- + +这是目前CMake寻找Python的逻辑存在缺陷，如果系统安装了多个Python版本，CMake找到的Python库和Python解释器版本可能有不一致现象，导致编译PaddlePaddle失败。正确的解决方法是， +用户强制指定特定的Python版本，具体操作如下： + + .. code-block:: bash + + cmake .. -DPYTHON_EXECUTABLE= -DPYTHON_LIBRARY= -DPYTHON_INCLUDE_DIR= + +用户需要指定本机上Python的路径：````, ````, ```` + +11. CMake源码编译，Paddle版本号为0.0.0 +-------------------------------------- + +如果运行 :code:`paddle version`, 出现 :code:`PaddlePaddle 0.0.0`；或者运行 :code:`cmake ..`，出现 + + .. code-block:: bash + + CMake Warning at cmake/version.cmake:20 (message): + Cannot add paddle version from git tag + +那么用户需要拉取所有的远程分支到本机，命令为 :code:`git fetch upstream`，然后重新cmake即可。 + +12. A protocol message was rejected because it was too big +---------------------------------------------------------- + +如果在训练NLP相关模型时，出现以下错误： + +.. code-block:: bash + + [libprotobuf ERROR google/protobuf/io/coded_stream.cc:171] A protocol message was rejected because it was too big (more than 67108864 bytes). To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h. + F1205 14:59:50.295174 14703 TrainerConfigHelper.cpp:59] Check failed: m->conf.ParseFromString(configProtoStr) + +可能的原因是：传给dataprovider的某一个args过大，一般是由于直接传递大字典导致的。错误的define_py_data_sources2类似： + +.. code-block:: python + + src_dict = dict() + for line_count, line in enumerate(open(src_dict_path, "r")): + src_dict[line.strip()] = line_count + + define_py_data_sources2( + train_list, + test_list, + module="dataprovider", + obj="process", + args={"src_dict": src_dict}) + +解决方案是：将字典的地址作为args传给dataprovider，然后在dataprovider里面根据该地址加载字典。即define_py_data_sources2应改为： + +.. code-block:: python + + define_py_data_sources2( + train_list, + test_list, + module="dataprovider", + obj="process", + args={"src_dict_path": src_dict_path}) + +完整源码可参考 `seqToseq `_ 示例。 + +13. 如何指定GPU设备 +------------------- + +例如机器上有4块GPU，编号从0开始，指定使用2、3号GPU： + +* 方式1：通过 `CUDA_VISIBLE_DEVICES `_ 环境变量来指定特定的GPU。 + +.. code-block:: bash + + env CUDA_VISIBLE_DEVICES=2,3 paddle train --use_gpu=true --trainer_count=2 + +* 方式2：通过命令行参数 ``--gpu_id`` 指定。 + +.. code-block:: bash + + paddle train --use_gpu=true --trainer_count=2 --gpu_id=2 + + +14. 训练过程中出现 :code:`Floating point exception`, 训练因此退出怎么办? +------------------------------------------------------------------------ + +Paddle二进制在运行时捕获了浮点数异常，只要出现浮点数异常(即训练过程中出现NaN或者Inf)，立刻退出。浮点异常通常的原因是浮点数溢出、除零等问题。 +主要原因包括两个方面: + +* 训练过程中参数或者训练过程中的梯度尺度过大，导致参数累加，乘除等时候，导致了浮点数溢出。 +* 模型一直不收敛，发散到了一个数值特别大的地方。 +* 训练数据有问题，导致参数收敛到了一些奇异的情况。或者输入数据尺度过大，有些特征的取值达到数百万，这时进行矩阵乘法运算就可能导致浮点数溢出。 + +主要的解决办法是减小学习律或者对数据进行归一化处理。 From df00e2928d1f69f55ef6aba1a91ea820a2c2d612 Mon Sep 17 00:00:00 2001 From: luotao02 Date: Fri, 12 May 2017 17:51:05 +0800 Subject: [PATCH 70/88] follow comments --- paddle/utils/DynamicLoader.cpp | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/paddle/utils/DynamicLoader.cpp b/paddle/utils/DynamicLoader.cpp index 5aa1fe12d8..5604a90038 100644 --- a/paddle/utils/DynamicLoader.cpp +++ b/paddle/utils/DynamicLoader.cpp @@ -115,9 +115,7 @@ static inline void GetDsoHandleFromSearchPath(const std::string& search_root, << "Note: After Mac OS 10.11, using the " "DYLD_LIBRARY_PATH is impossible " << "unless System Integrity Protection (SIP) " - "is disabled. However, " - "method 1 " - << "always work well."; + "is disabled."; } void GetCublasDsoHandle(void** dso_handle) { From 494aa8f63fce3def92d46fbb89871709a13ec43a Mon Sep 17 00:00:00 2001 From: Luo Tao Date: Fri, 12 May 2017 18:25:25 +0800 Subject: [PATCH 71/88] add explanation of paddle version 0.0.0 in faq --- doc/faq/index_cn.rst | 626 +++++++++++++++++++++---------------------- 1 file changed, 313 insertions(+), 313 deletions(-) diff --git a/doc/faq/index_cn.rst b/doc/faq/index_cn.rst index 081e3a60a2..c14160d55e 100644 --- a/doc/faq/index_cn.rst +++ b/doc/faq/index_cn.rst @@ -1,313 +1,313 @@ -#################### -FAQ -#################### - -.. contents:: - -1. 如何减少内存占用 ---------------------------------- - -神经网络的训练本身是一个非常消耗内存和显存的工作，经常会消耗数10GB的内存和数GB的显存。 -PaddlePaddle的内存占用主要分为如下几个方面\: - -* DataProvider缓冲池内存（只针对内存） -* 神经元激活内存（针对内存和显存） -* 参数内存 （针对内存和显存） -* 其他内存杂项 - -其中，其他内存杂项是指PaddlePaddle本身所用的一些内存，包括字符串分配，临时变量等等，暂不考虑在内。 - -减少DataProvider缓冲池内存 -++++++++++++++++++++++++++ - -PyDataProvider使用的是异步加载，同时在内存里直接随即选取数据来做Shuffle。即 - -.. graphviz:: - - digraph { - rankdir=LR; - 数据文件 -> 内存池 -> PaddlePaddle训练 - } - -所以，减小这个内存池即可减小内存占用，同时也可以加速开始训练前数据载入的过程。但是，这 -个内存池实际上决定了shuffle的粒度。所以，如果将这个内存池减小，又要保证数据是随机的， -那么最好将数据文件在每次读取之前做一次shuffle。可能的代码为 - -.. literalinclude:: src/reduce_min_pool_size.py - -这样做可以极大的减少内存占用，并且可能会加速训练过程，详细文档参考 :ref:`api_pydataprovider2` 。 - -神经元激活内存 -++++++++++++++ - -神经网络在训练的时候，会对每一个激活暂存一些数据，如神经元激活值等。 -在反向传递的时候，这些数据会被用来更新参数。这些数据使用的内存主要和两个参数有关系， -一是batch size，另一个是每条序列(Sequence)长度。所以，其实也是和每个mini-batch中包含 -的时间步信息成正比。 - -所以做法可以有两种： - -* 减小batch size。 即在网络配置中 :code:`settings(batch_size=1000)` 设置成一个小一些的值。但是batch size本身是神经网络的超参数，减小batch size可能会对训练结果产生影响。 -* 减小序列的长度，或者直接扔掉非常长的序列。比如，一个数据集大部分序列长度是100-200, - 但是突然有一个10000长的序列，就很容易导致内存超限，特别是在LSTM等RNN中。 - -参数内存 -++++++++ - -PaddlePaddle支持非常多的优化算法(Optimizer)，不同的优化算法需要使用不同大小的内存。 -例如使用 :code:`adadelta` 算法，则需要使用等于权重参数规模大约5倍的内存。举例，如果参数保存下来的模型目录 -文件为 :code:`100M`， 那么该优化算法至少需要 :code:`500M` 的内存。 - -可以考虑使用一些优化算法，例如 :code:`momentum`。 - -2. 如何加速PaddlePaddle的训练速度 ---------------------------------- - -加速PaddlePaddle训练可以考虑从以下几个方面\： - -* 减少数据载入的耗时 -* 加速训练速度 -* 利用分布式训练驾驭更多的计算资源 - -减少数据载入的耗时 -++++++++++++++++++ - -使用\ :code:`pydataprovider`\ 时，可以减少缓存池的大小，同时设置内存缓存功能，即可以极大的加速数据载入流程。 -:code:`DataProvider` 缓存池的减小，和之前减小通过减小缓存池来减小内存占用的原理一致。 - -.. literalinclude:: src/reduce_min_pool_size.py - -同时 :code:`@provider` 接口有一个 :code:`cache` 参数来控制缓存方法，将其设置成 :code:`CacheType.CACHE_PASS_IN_MEM` 的话，会将第一个 :code:`pass` (过完所有训练数据即为一个pass)生成的数据缓存在内存里，在之后的 :code:`pass` 中，不会再从 :code:`python` 端读取数据，而是直接从内存的缓存里读取数据。这也会极大减少数据读入的耗时。 - - -加速训练速度 -++++++++++++ - -PaddlePaddle支持Sparse的训练，sparse训练需要训练特征是 :code:`sparse_binary_vector` 、 :code:`sparse_vector` 、或者 :code:`integer_value` 的任一一种。同时，与这个训练数据交互的Layer，需要将其Parameter设置成 sparse 更新模式，即设置 :code:`sparse_update=True` - -这里使用简单的 :code:`word2vec` 训练语言模型距离，具体使用方法为\: - -使用一个词前两个词和后两个词，来预测这个中间的词。这个任务的DataProvider为\: - -.. literalinclude:: src/word2vec_dataprovider.py - -这个任务的配置为\: - -.. literalinclude:: src/word2vec_config.py - - -利用更多的计算资源 -++++++++++++++++++ - -利用更多的计算资源可以分为一下几个方式来进行\: - -* 单机CPU训练 - - * 使用多线程训练。设置命令行参数 :code:`trainer_count`。 - -* 单机GPU训练 - - * 使用显卡训练。设置命令行参数 :code:`use_gpu`。 - * 使用多块显卡训练。设置命令行参数 :code:`use_gpu` 和 :code:`trainer_count` 。 - -* 多机训练 - - * 请参考 :ref:`cluster_train` 。 - - -3. 遇到“非法指令”或者是“illegal instruction” --------------------------------------------- - -PaddlePaddle使用avx SIMD指令提高cpu执行效率，因此错误的使用二进制发行版可能会导致这种错误，请选择正确的版本。 - -4. 如何选择SGD算法的学习率 --------------------------- - -在采用sgd/async_sgd进行训练时，一个重要的问题是选择正确的learning_rate。如果learning_rate太大，那么训练有可能不收敛，如果learning_rate太小，那么收敛可能很慢，导致训练时间过长。 - -通常做法是从一个比较大的learning_rate开始试，如果不收敛，那减少学习率10倍继续试验，直到训练收敛为止。那么如何判断训练不收敛呢？可以估计出如果模型采用不变的输出最小的cost0是多少。 - -如果训练过程的的cost明显高于这个常数输出的cost，那么我们可以判断为训练不收敛。举一个例子，假如我们是三分类问题，采用multi-class-cross-entropy作为cost，数据中0,1,2三类的比例为 :code:`0.2, 0.5, 0.3` , 那么常数输出所能达到的最小cost是 :code:`-(0.2*log(0.2)+0.5*log(0.5)+0.3*log(0.3))=1.03` 。如果训练一个pass（或者更早）后，cost还大于这个数，那么可以认为训练不收敛，应该降低学习率。 - - -5. 如何初始化参数 ------------------ - -默认情况下，PaddlePaddle使用均值0，标准差为 :math:`\frac{1}{\sqrt{d}}` 来初始化参数。其中 :math:`d` 为参数矩阵的宽度。这种初始化方式在一般情况下不会产生很差的结果。如果用户想要自定义初始化方式，PaddlePaddle目前提供两种参数初始化的方式\: - -* 高斯分布。将 :code:`param_attr` 设置成 :code:`param_attr=ParamAttr(initial_mean=0.0, initial_std=1.0)` -* 均匀分布。将 :code:`param_attr` 设置成 :code:`param_attr=ParamAttr(initial_max=1.0, initial_min=-1.0)` - -比如设置一个全连接层的参数初始化方式和bias初始化方式，可以使用如下代码。 - -.. code-block:: python - - hidden = fc_layer(input=ipt, param_attr=ParamAttr(initial_max=1.0, initial_min=-1.0), - bias_attr=ParamAttr(initial_mean=1.0, initial_std=0.0)) - -上述代码将bias全部初始化为1.0, 同时将参数初始化为 :code:`[1.0, -1.0]` 的均匀分布。 - -6. 如何共享参数 ---------------- - -PaddlePaddle的参数使用名字 :code:`name` 作为参数的ID，相同名字的参数，会共享参数。设置参数的名字，可以使用 :code:`ParamAttr(name="YOUR_PARAM_NAME")` 来设置。更方便的设置方式，是使得要共享的参数使用同样的 :code:`ParamAttr` 对象。 - -简单的全连接网络，参数共享的配置示例为\: - -.. literalinclude:: ../../python/paddle/trainer_config_helpers/tests/configs/shared_fc.py - -这里 :code:`hidden_a` 和 :code:`hidden_b` 使用了同样的parameter和bias。并且softmax层的两个输入也使用了同样的参数 :code:`softmax_param`。 - -7. \*-cp27mu-linux_x86_64.whl is not a supported wheel on this platform. ------------------------------------------------------------------------- - -出现这个问题的主要原因是，系统编译wheel包的时候，使用的 :code:`wheel` 包是最新的， -而系统中的 :code:`pip` 包比较老。具体的解决方法是，更新 :code:`pip` 包并重新编译PaddlePaddle。 -更新 :code:`pip` 包的方法是\: - -.. code-block:: bash - - pip install --upgrade pip - -8. python相关的单元测试都过不了 --------------------------------- - -如果出现以下python相关的单元测试都过不了的情况： - -.. code-block:: bash - - 24 - test_PyDataProvider (Failed) - 26 - test_RecurrentGradientMachine (Failed) - 27 - test_NetworkCompare (Failed) - 28 - test_PyDataProvider2 (Failed) - 32 - test_Prediction (Failed) - 33 - test_Compare (Failed) - 34 - test_Trainer (Failed) - 35 - test_TrainerOnePass (Failed) - 36 - test_CompareTwoNets (Failed) - 37 - test_CompareTwoOpts (Failed) - 38 - test_CompareSparse (Failed) - 39 - test_recurrent_machine_generation (Failed) - 40 - test_PyDataProviderWrapper (Failed) - 41 - test_config_parser (Failed) - 42 - test_swig_api (Failed) - 43 - layers_test (Failed) - -并且查询PaddlePaddle单元测试的日志，提示： - -.. code-block:: bash - - paddle package is already in your PYTHONPATH. But unittest need a clean environment. - Please uninstall paddle package before start unittest. Try to 'pip uninstall paddle'. - -解决办法是： - -* 卸载PaddlePaddle包 :code:`pip uninstall paddle`, 清理掉老旧的PaddlePaddle安装包，使得单元测试有一个干净的环境。如果PaddlePaddle包已经在python的site-packages里面，单元测试会引用site-packages里面的python包，而不是源码目录里 :code:`/python` 目录下的python包。同时，即便设置 :code:`PYTHONPATH` 到 :code:`/python` 也没用，因为python的搜索路径是优先已经安装的python包。 - - -9. 运行Docker GPU镜像出现 "CUDA driver version is insufficient" ----------------------------------------------------------------- - -用户在使用PaddlePaddle GPU的Docker镜像的时候，常常出现 `Cuda Error: CUDA driver version is insufficient for CUDA runtime version`, 原因在于没有把机器上CUDA相关的驱动和库映射到容器内部。 -具体的解决方法是： - -.. code-block:: bash - - $ export CUDA_SO="$(\ls usr/lib64/libcuda* | xargs -I{} echo '-v {}:{}') $(\ls /usr/lib64/libnvidia* | xargs -I{} echo '-v {}:{}')" - $ export DEVICES=$(\ls /dev/nvidia* | xargs -I{} echo '--device {}:{}') - $ docker run ${CUDA_SO} ${DEVICES} -it paddledev/paddlepaddle:latest-gpu - -更多关于Docker的安装与使用, 请参考 `PaddlePaddle Docker 文档 `_ 。 - - -10. CMake源码编译, 找到的PythonLibs和PythonInterp版本不一致 ----------------------------------------------------------------- - -这是目前CMake寻找Python的逻辑存在缺陷，如果系统安装了多个Python版本，CMake找到的Python库和Python解释器版本可能有不一致现象，导致编译PaddlePaddle失败。正确的解决方法是， -用户强制指定特定的Python版本，具体操作如下： - - .. code-block:: bash - - cmake .. -DPYTHON_EXECUTABLE= -DPYTHON_LIBRARY= -DPYTHON_INCLUDE_DIR= - -用户需要指定本机上Python的路径：````, ````, ```` - -11. CMake源码编译，Paddle版本号为0.0.0 --------------------------------------- - -如果运行 :code:`paddle version`, 出现 :code:`PaddlePaddle 0.0.0`；或者运行 :code:`cmake ..`，出现 - - .. code-block:: bash - - CMake Warning at cmake/version.cmake:20 (message): - Cannot add paddle version from git tag - -那么用户需要拉取所有的远程分支到本机，命令为 :code:`git fetch upstream`，然后重新cmake即可。 - -12. A protocol message was rejected because it was too big ----------------------------------------------------------- - -如果在训练NLP相关模型时，出现以下错误： - -.. code-block:: bash - - [libprotobuf ERROR google/protobuf/io/coded_stream.cc:171] A protocol message was rejected because it was too big (more than 67108864 bytes). To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h. - F1205 14:59:50.295174 14703 TrainerConfigHelper.cpp:59] Check failed: m->conf.ParseFromString(configProtoStr) - -可能的原因是：传给dataprovider的某一个args过大，一般是由于直接传递大字典导致的。错误的define_py_data_sources2类似： - -.. code-block:: python - - src_dict = dict() - for line_count, line in enumerate(open(src_dict_path, "r")): - src_dict[line.strip()] = line_count - - define_py_data_sources2( - train_list, - test_list, - module="dataprovider", - obj="process", - args={"src_dict": src_dict}) - -解决方案是：将字典的地址作为args传给dataprovider，然后在dataprovider里面根据该地址加载字典。即define_py_data_sources2应改为： - -.. code-block:: python - - define_py_data_sources2( - train_list, - test_list, - module="dataprovider", - obj="process", - args={"src_dict_path": src_dict_path}) - -完整源码可参考 `seqToseq `_ 示例。 - -13. 如何指定GPU设备 -------------------- - -例如机器上有4块GPU，编号从0开始，指定使用2、3号GPU： - -* 方式1：通过 `CUDA_VISIBLE_DEVICES `_ 环境变量来指定特定的GPU。 - -.. code-block:: bash - - env CUDA_VISIBLE_DEVICES=2,3 paddle train --use_gpu=true --trainer_count=2 - -* 方式2：通过命令行参数 ``--gpu_id`` 指定。 - -.. code-block:: bash - - paddle train --use_gpu=true --trainer_count=2 --gpu_id=2 - - -14. 训练过程中出现 :code:`Floating point exception`, 训练因此退出怎么办? ------------------------------------------------------------------------- - -Paddle二进制在运行时捕获了浮点数异常，只要出现浮点数异常(即训练过程中出现NaN或者Inf)，立刻退出。浮点异常通常的原因是浮点数溢出、除零等问题。 -主要原因包括两个方面: - -* 训练过程中参数或者训练过程中的梯度尺度过大，导致参数累加，乘除等时候，导致了浮点数溢出。 -* 模型一直不收敛，发散到了一个数值特别大的地方。 -* 训练数据有问题，导致参数收敛到了一些奇异的情况。或者输入数据尺度过大，有些特征的取值达到数百万，这时进行矩阵乘法运算就可能导致浮点数溢出。 - -主要的解决办法是减小学习律或者对数据进行归一化处理。 +#################### +FAQ +#################### + +.. contents:: + +1. 如何减少内存占用 +--------------------------------- + +神经网络的训练本身是一个非常消耗内存和显存的工作，经常会消耗数10GB的内存和数GB的显存。 +PaddlePaddle的内存占用主要分为如下几个方面\: + +* DataProvider缓冲池内存（只针对内存） +* 神经元激活内存（针对内存和显存） +* 参数内存 （针对内存和显存） +* 其他内存杂项 + +其中，其他内存杂项是指PaddlePaddle本身所用的一些内存，包括字符串分配，临时变量等等，暂不考虑在内。 + +减少DataProvider缓冲池内存 +++++++++++++++++++++++++++ + +PyDataProvider使用的是异步加载，同时在内存里直接随即选取数据来做Shuffle。即 + +.. graphviz:: + + digraph { + rankdir=LR; + 数据文件 -> 内存池 -> PaddlePaddle训练 + } + +所以，减小这个内存池即可减小内存占用，同时也可以加速开始训练前数据载入的过程。但是，这 +个内存池实际上决定了shuffle的粒度。所以，如果将这个内存池减小，又要保证数据是随机的， +那么最好将数据文件在每次读取之前做一次shuffle。可能的代码为 + +.. literalinclude:: src/reduce_min_pool_size.py + +这样做可以极大的减少内存占用，并且可能会加速训练过程，详细文档参考 :ref:`api_pydataprovider2` 。 + +神经元激活内存 +++++++++++++++ + +神经网络在训练的时候，会对每一个激活暂存一些数据，如神经元激活值等。 +在反向传递的时候，这些数据会被用来更新参数。这些数据使用的内存主要和两个参数有关系， +一是batch size，另一个是每条序列(Sequence)长度。所以，其实也是和每个mini-batch中包含 +的时间步信息成正比。 + +所以做法可以有两种： + +* 减小batch size。 即在网络配置中 :code:`settings(batch_size=1000)` 设置成一个小一些的值。但是batch size本身是神经网络的超参数，减小batch size可能会对训练结果产生影响。 +* 减小序列的长度，或者直接扔掉非常长的序列。比如，一个数据集大部分序列长度是100-200, + 但是突然有一个10000长的序列，就很容易导致内存超限，特别是在LSTM等RNN中。 + +参数内存 +++++++++ + +PaddlePaddle支持非常多的优化算法(Optimizer)，不同的优化算法需要使用不同大小的内存。 +例如使用 :code:`adadelta` 算法，则需要使用等于权重参数规模大约5倍的内存。举例，如果参数保存下来的模型目录 +文件为 :code:`100M`， 那么该优化算法至少需要 :code:`500M` 的内存。 + +可以考虑使用一些优化算法，例如 :code:`momentum`。 + +2. 如何加速PaddlePaddle的训练速度 +--------------------------------- + +加速PaddlePaddle训练可以考虑从以下几个方面\： + +* 减少数据载入的耗时 +* 加速训练速度 +* 利用分布式训练驾驭更多的计算资源 + +减少数据载入的耗时 +++++++++++++++++++ + +使用\ :code:`pydataprovider`\ 时，可以减少缓存池的大小，同时设置内存缓存功能，即可以极大的加速数据载入流程。 +:code:`DataProvider` 缓存池的减小，和之前减小通过减小缓存池来减小内存占用的原理一致。 + +.. literalinclude:: src/reduce_min_pool_size.py + +同时 :code:`@provider` 接口有一个 :code:`cache` 参数来控制缓存方法，将其设置成 :code:`CacheType.CACHE_PASS_IN_MEM` 的话，会将第一个 :code:`pass` (过完所有训练数据即为一个pass)生成的数据缓存在内存里，在之后的 :code:`pass` 中，不会再从 :code:`python` 端读取数据，而是直接从内存的缓存里读取数据。这也会极大减少数据读入的耗时。 + + +加速训练速度 +++++++++++++ + +PaddlePaddle支持Sparse的训练，sparse训练需要训练特征是 :code:`sparse_binary_vector` 、 :code:`sparse_vector` 、或者 :code:`integer_value` 的任一一种。同时，与这个训练数据交互的Layer，需要将其Parameter设置成 sparse 更新模式，即设置 :code:`sparse_update=True` + +这里使用简单的 :code:`word2vec` 训练语言模型距离，具体使用方法为\: + +使用一个词前两个词和后两个词，来预测这个中间的词。这个任务的DataProvider为\: + +.. literalinclude:: src/word2vec_dataprovider.py + +这个任务的配置为\: + +.. literalinclude:: src/word2vec_config.py + + +利用更多的计算资源 +++++++++++++++++++ + +利用更多的计算资源可以分为一下几个方式来进行\: + +* 单机CPU训练 + + * 使用多线程训练。设置命令行参数 :code:`trainer_count`。 + +* 单机GPU训练 + + * 使用显卡训练。设置命令行参数 :code:`use_gpu`。 + * 使用多块显卡训练。设置命令行参数 :code:`use_gpu` 和 :code:`trainer_count` 。 + +* 多机训练 + + * 请参考 :ref:`cluster_train` 。 + + +3. 遇到“非法指令”或者是“illegal instruction” +-------------------------------------------- + +PaddlePaddle使用avx SIMD指令提高cpu执行效率，因此错误的使用二进制发行版可能会导致这种错误，请选择正确的版本。 + +4. 如何选择SGD算法的学习率 +-------------------------- + +在采用sgd/async_sgd进行训练时，一个重要的问题是选择正确的learning_rate。如果learning_rate太大，那么训练有可能不收敛，如果learning_rate太小，那么收敛可能很慢，导致训练时间过长。 + +通常做法是从一个比较大的learning_rate开始试，如果不收敛，那减少学习率10倍继续试验，直到训练收敛为止。那么如何判断训练不收敛呢？可以估计出如果模型采用不变的输出最小的cost0是多少。 + +如果训练过程的的cost明显高于这个常数输出的cost，那么我们可以判断为训练不收敛。举一个例子，假如我们是三分类问题，采用multi-class-cross-entropy作为cost，数据中0,1,2三类的比例为 :code:`0.2, 0.5, 0.3` , 那么常数输出所能达到的最小cost是 :code:`-(0.2*log(0.2)+0.5*log(0.5)+0.3*log(0.3))=1.03` 。如果训练一个pass（或者更早）后，cost还大于这个数，那么可以认为训练不收敛，应该降低学习率。 + + +5. 如何初始化参数 +----------------- + +默认情况下，PaddlePaddle使用均值0，标准差为 :math:`\frac{1}{\sqrt{d}}` 来初始化参数。其中 :math:`d` 为参数矩阵的宽度。这种初始化方式在一般情况下不会产生很差的结果。如果用户想要自定义初始化方式，PaddlePaddle目前提供两种参数初始化的方式\: + +* 高斯分布。将 :code:`param_attr` 设置成 :code:`param_attr=ParamAttr(initial_mean=0.0, initial_std=1.0)` +* 均匀分布。将 :code:`param_attr` 设置成 :code:`param_attr=ParamAttr(initial_max=1.0, initial_min=-1.0)` + +比如设置一个全连接层的参数初始化方式和bias初始化方式，可以使用如下代码。 + +.. code-block:: python + + hidden = fc_layer(input=ipt, param_attr=ParamAttr(initial_max=1.0, initial_min=-1.0), + bias_attr=ParamAttr(initial_mean=1.0, initial_std=0.0)) + +上述代码将bias全部初始化为1.0, 同时将参数初始化为 :code:`[1.0, -1.0]` 的均匀分布。 + +6. 如何共享参数 +--------------- + +PaddlePaddle的参数使用名字 :code:`name` 作为参数的ID，相同名字的参数，会共享参数。设置参数的名字，可以使用 :code:`ParamAttr(name="YOUR_PARAM_NAME")` 来设置。更方便的设置方式，是使得要共享的参数使用同样的 :code:`ParamAttr` 对象。 + +简单的全连接网络，参数共享的配置示例为\: + +.. literalinclude:: ../../python/paddle/trainer_config_helpers/tests/configs/shared_fc.py + +这里 :code:`hidden_a` 和 :code:`hidden_b` 使用了同样的parameter和bias。并且softmax层的两个输入也使用了同样的参数 :code:`softmax_param`。 + +7. \*-cp27mu-linux_x86_64.whl is not a supported wheel on this platform. +------------------------------------------------------------------------ + +出现这个问题的主要原因是，系统编译wheel包的时候，使用的 :code:`wheel` 包是最新的， +而系统中的 :code:`pip` 包比较老。具体的解决方法是，更新 :code:`pip` 包并重新编译PaddlePaddle。 +更新 :code:`pip` 包的方法是\: + +.. code-block:: bash + + pip install --upgrade pip + +8. python相关的单元测试都过不了 +-------------------------------- + +如果出现以下python相关的单元测试都过不了的情况： + +.. code-block:: bash + + 24 - test_PyDataProvider (Failed) + 26 - test_RecurrentGradientMachine (Failed) + 27 - test_NetworkCompare (Failed) + 28 - test_PyDataProvider2 (Failed) + 32 - test_Prediction (Failed) + 33 - test_Compare (Failed) + 34 - test_Trainer (Failed) + 35 - test_TrainerOnePass (Failed) + 36 - test_CompareTwoNets (Failed) + 37 - test_CompareTwoOpts (Failed) + 38 - test_CompareSparse (Failed) + 39 - test_recurrent_machine_generation (Failed) + 40 - test_PyDataProviderWrapper (Failed) + 41 - test_config_parser (Failed) + 42 - test_swig_api (Failed) + 43 - layers_test (Failed) + +并且查询PaddlePaddle单元测试的日志，提示： + +.. code-block:: bash + + paddle package is already in your PYTHONPATH. But unittest need a clean environment. + Please uninstall paddle package before start unittest. Try to 'pip uninstall paddle'. + +解决办法是： + +* 卸载PaddlePaddle包 :code:`pip uninstall paddle`, 清理掉老旧的PaddlePaddle安装包，使得单元测试有一个干净的环境。如果PaddlePaddle包已经在python的site-packages里面，单元测试会引用site-packages里面的python包，而不是源码目录里 :code:`/python` 目录下的python包。同时，即便设置 :code:`PYTHONPATH` 到 :code:`/python` 也没用，因为python的搜索路径是优先已经安装的python包。 + + +9. 运行Docker GPU镜像出现 "CUDA driver version is insufficient" +---------------------------------------------------------------- + +用户在使用PaddlePaddle GPU的Docker镜像的时候，常常出现 `Cuda Error: CUDA driver version is insufficient for CUDA runtime version`, 原因在于没有把机器上CUDA相关的驱动和库映射到容器内部。 +具体的解决方法是： + +.. code-block:: bash + + $ export CUDA_SO="$(\ls usr/lib64/libcuda* | xargs -I{} echo '-v {}:{}') $(\ls /usr/lib64/libnvidia* | xargs -I{} echo '-v {}:{}')" + $ export DEVICES=$(\ls /dev/nvidia* | xargs -I{} echo '--device {}:{}') + $ docker run ${CUDA_SO} ${DEVICES} -it paddledev/paddlepaddle:latest-gpu + +更多关于Docker的安装与使用, 请参考 `PaddlePaddle Docker 文档 `_ 。 + + +10. CMake源码编译, 找到的PythonLibs和PythonInterp版本不一致 +---------------------------------------------------------------- + +这是目前CMake寻找Python的逻辑存在缺陷，如果系统安装了多个Python版本，CMake找到的Python库和Python解释器版本可能有不一致现象，导致编译PaddlePaddle失败。正确的解决方法是， +用户强制指定特定的Python版本，具体操作如下： + + .. code-block:: bash + + cmake .. -DPYTHON_EXECUTABLE= -DPYTHON_LIBRARY= -DPYTHON_INCLUDE_DIR= + +用户需要指定本机上Python的路径：````, ````, ```` + +11. CMake源码编译，Paddle版本号为0.0.0 +-------------------------------------- + +如果运行 :code:`paddle version`, 出现 :code:`PaddlePaddle 0.0.0`；或者运行 :code:`cmake ..`，出现 + +.. code-block:: bash + + CMake Warning at cmake/version.cmake:20 (message): + Cannot add paddle version from git tag + +那么用户需要拉取所有的远程分支到本机，命令为 :code:`git fetch upstream`，然后重新cmake即可。 + +12. A protocol message was rejected because it was too big +---------------------------------------------------------- + +如果在训练NLP相关模型时，出现以下错误： + +.. code-block:: bash + + [libprotobuf ERROR google/protobuf/io/coded_stream.cc:171] A protocol message was rejected because it was too big (more than 67108864 bytes). To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h. + F1205 14:59:50.295174 14703 TrainerConfigHelper.cpp:59] Check failed: m->conf.ParseFromString(configProtoStr) + +可能的原因是：传给dataprovider的某一个args过大，一般是由于直接传递大字典导致的。错误的define_py_data_sources2类似： + +.. code-block:: python + + src_dict = dict() + for line_count, line in enumerate(open(src_dict_path, "r")): + src_dict[line.strip()] = line_count + + define_py_data_sources2( + train_list, + test_list, + module="dataprovider", + obj="process", + args={"src_dict": src_dict}) + +解决方案是：将字典的地址作为args传给dataprovider，然后在dataprovider里面根据该地址加载字典。即define_py_data_sources2应改为： + +.. code-block:: python + + define_py_data_sources2( + train_list, + test_list, + module="dataprovider", + obj="process", + args={"src_dict_path": src_dict_path}) + +完整源码可参考 `seqToseq `_ 示例。 + +13. 如何指定GPU设备 +------------------- + +例如机器上有4块GPU，编号从0开始，指定使用2、3号GPU： + +* 方式1：通过 `CUDA_VISIBLE_DEVICES `_ 环境变量来指定特定的GPU。 + +.. code-block:: bash + + env CUDA_VISIBLE_DEVICES=2,3 paddle train --use_gpu=true --trainer_count=2 + +* 方式2：通过命令行参数 ``--gpu_id`` 指定。 + +.. code-block:: bash + + paddle train --use_gpu=true --trainer_count=2 --gpu_id=2 + + +14. 训练过程中出现 :code:`Floating point exception`, 训练因此退出怎么办? +------------------------------------------------------------------------ + +Paddle二进制在运行时捕获了浮点数异常，只要出现浮点数异常(即训练过程中出现NaN或者Inf)，立刻退出。浮点异常通常的原因是浮点数溢出、除零等问题。 +主要原因包括两个方面: + +* 训练过程中参数或者训练过程中的梯度尺度过大，导致参数累加，乘除等时候，导致了浮点数溢出。 +* 模型一直不收敛，发散到了一个数值特别大的地方。 +* 训练数据有问题，导致参数收敛到了一些奇异的情况。或者输入数据尺度过大，有些特征的取值达到数百万，这时进行矩阵乘法运算就可能导致浮点数溢出。 + +主要的解决办法是减小学习律或者对数据进行归一化处理。 From 33625aeaebf895d866b79af7ba1c99f92d8a86b9 Mon Sep 17 00:00:00 2001 From: gongweibao Date: Fri, 12 May 2017 19:28:24 +0800 Subject: [PATCH 72/88] fix common to read-only --- doc/design/cluster_train/data_dispatch.md | 4 +--- .../cluster_train/src/file_storage.graffle | Bin 3031 -> 3059 bytes doc/design/cluster_train/src/file_storage.png | Bin 42121 -> 43413 bytes 3 files changed, 1 insertion(+), 3 deletions(-) diff --git a/doc/design/cluster_train/data_dispatch.md b/doc/design/cluster_train/data_dispatch.md index e7798b24f6..9f7167498c 100644 --- a/doc/design/cluster_train/data_dispatch.md +++ b/doc/design/cluster_train/data_dispatch.md @@ -19,6 +19,7 @@ - 无论是从[PFSClient](../file_manager/README.md)的角度，还是从[Pod](https://kubernetes.io/docs/concepts/workloads/pods/pod/)中运行任务的角度，统一用`/pfs/$DATACENTER/home/$USER`来访问用户自己的数据。 - `/pfs/$DATACENTER/common`下存放公共数据集合 + - 做只读挂载
@@ -141,9 +142,6 @@ endpoint=datacenter2.paddlepaddle.org ## TODO ### 文件访问的权限 控制用户权限 - -- `/pfs/$DATACENTER/common`数据集合只读不能写 - - 现在mount到本地以后有读写权限 - 用户可以把自己的数据分享给别人 ### 文件访问方式 diff --git a/doc/design/cluster_train/src/file_storage.graffle b/doc/design/cluster_train/src/file_storage.graffle index 5331f407f7ee77fca263be24eb83ed1d44ca2bd4..50a17e70fa255495337c529a3bf12a5c0024a5be 100644 GIT binary patch literal 3059 zcmVuhiwFP!000030PS2|a^tuWeqNt~aqU}d3gACyz22gJEssaG^<$4*u1Y~9 zByvp>8j`ZCnet1Xk~5_8k{jd#xy@E8w+KL55-C!)ykqZ<6S-XC4;qa|qx(ZQNbumN zJJ02AaS%Gh|8W=cV3)(bNi4@7{kYrf*7vpDpAL5p{!l%wbo*yDZtOZ?#GUoZC(R1C zyI(39jW z+ou(HE3$TJGYR*MIP|XNn9r#s1pfu;4NHB?rVf{P_qnx-<&dFOw<_&A#w`kjA+c6iF9u)CF#wfyh z*+Ql-U-qE_#XW9cSJeAG?u(>}`W`3f0N&qcI>9PdTomScbZj)n%hp%wG^@)BbJ2x# zFA}m&G;{HiaxQyPNLiAbWofE7DX#`-x|C}vS?e?{4%PEQCG)f8NutXTUlHPMU?vsh z-=gp|a7ND3K;}vXdmKn>FIA)Ce8xHYWC+*@hcQ(#dL)24NX=6LA5_95VKU($Rh3dqd?jGzn zWFS-E@mYgh@urkJNfP8$U6mwV7a^d2F70t5h+PW;dM+qgzbbO9r#)+~mZ*#GqI|u6 zty4hll4+i_jl{lcj2RcAL9XecmfCfXy9HC!S&F0=+U%M5aI}MN7%i^5uHU_c!Gbpr?e(< zNkEBzxDK!-$ny0lz}ROe19yVyky2e-co?C9DuA8qF$%2sXMXGYlgX|lXJkj%A0m_D z4%pd=8xIAAyEcb_tKhL+?{F6k2J!tG>`xSNNu4V14Ou;>=8H|;U(h99)FcW3P!SbP zROLNRQe~drS5-|Bby)<8%+Np(WH!%k(IATkpa2a{9RCq8Nb^Y{yq0B@(4LXdiBc27 zLQ=rXlBi08s_Re|WCiLNO)@BC<#v}^E zF2>ghd=yOrS5k}Ut z0|NE&>ABdawE~|$BlK2UbBio7Wrs2YOi_XND;=Jfi;~QXs-^>l#etHj@IX*S8iUB| zYAB+}15Hw(M1xC6<4A!bvaYUL$WoYZKk8uPXaqq?xvDFhFO%&a}85lXP3WXnwE8u8igvcNm>IuRCIx69-1HmSr=%L z^n#OivBRj(p9EjW2ER^^n##)pgfv%!K!<{&FlSbIRTDHxgd)@wN!Q{<9g(`PD87wY zsCdYfmrA+hXQo_|tk6tLQ-K0h87hjlO}Vxy*KeM3i9!-{QPZ>biSN!hkK)2M@_M<* zOZqjDm;7~+*J@ytilZPyY#ppEmsMVpWD&B!Or$BVM$gs~D>2d9Fi#b9S}w0`Op{oB zN#W)H**LCFb6h+x{=bOm^cRfiRQ5Pl6!mz?7}Ag*$}bpe{AOz-(zh2DZZd;r9=tvh zc)Wo%iQi_Teg49xNj#7E_fI{jf6h{+#6v&>OA)YCnYj-XcmYTP1R8~6fnq!JCRu&! zfxGxpfjfL|;I2bO0*a>Snx@g*Qr!mb+ra&I3*28jTz_hE2HG2+oQ3V=yq%nX_%G*42`(&Z`y+%=F3mtN@=FMJY zl-}S}r@zgaZaa5v_R`FY=aPxlOZwZkdhKmmeY;!Fz?y|3l$@50=(BZn?Dj7qz#q%* zRr81#(s}v3*{Rv(H0>i|2-Y^}hCS zFQ$nrP{Oj?JTJeWhPSApM^_!~=HjAN?-5ipJonf*pLA^6TD2y3;|lGD`?J+>ZwFp| zhR-WK+o+EX`F+hD_m3#&>Oi==rJfj9O8Nlg?zy0c1F>namip6E%K6GOj7H;zbS_=H zeA+wOaJZm&_F^)Ibi|6`fN6?$S@9yEHa z!{?Ony#Mjoz7mgZvtiSYy5UtUX1jV0%1IxLQKfD)sf@?yvV0?V+nzqLnu*Md>$1}G zI`ckUgr8ep%PNly!zfSbPqkFK85vjS<aYBskNiLJnvWDe;!W1?pv)8}$gixp!UtiD zPo8~c<~qKGgFNSNMbj%AR~vz2r32dR4pdgUvNdrY(JlkpnDPVqnFNfy!rXS^DD#^rn;s>su=RKxY_MPTJ;viM$?gbUEV?8ME^O;| z_~d>}gMvGqDl%T4TD?hFQRO^R+(KT#nEmp%fBo{$KXV0Ubw)4KGhGx+((EoAJS{3+ z=R35cWD>3S9>aIM)K=YflLsp`oxmg?9KV{{$vXEmZN=4M+76~u8_GQEn?LCTEA(zG zX3Za^A2ZHnu}G8?#%DW(*|OOQ_D7MOYd`?WpP*|%5NUY4(qLTw83IOAGIlni&pHql z@n)SzVS4t+>HqGMwmtr+vd(mxQ(89sb?;E7Gy5WW+XcY;{FDLY|=H{6x| zf!t!YQ~C(`mWwy;P>V!(^X{JpY%fD*K#IQwFsG*GwJ~#sO=UW!H{k}f%xTvtFu!_kU#Kk)(!4lharX2Ifh8mwfzYgYYh{ z%%TP4oA{Y^d%p8&f|sezBfam1xaLPr@R-^!iX+YZB=ImkToA{4k7rqR!mKtgX%ruZl2?DCLSn2to8|1%LQx7uuv`8nRB!N9{ABssrsSALm;u0wMl1|6YN|cUA7X(GKNf^4On=9o+`d{taQ+|(5EUFe1a#b@~18uY@~*VA!y@K2E-QXh`)lv6tVca zOW}~c literal 3031 zcmV;|3n=s-iwFP!000030PS4cQsc-OzFwX}q4q9dck8TOmM!^+8ODGQFs7(VwWYSL zMN)et8RMDaCRfQbq;iut$P46cwo-YEbW7OQ5zG#3hKVW4I`rRv|NZxW_`0Rm!OwS& z&D~jOA9SGz+>zkf`jt% zbXkSAZ5b$_EH5I?t4hf85Igzqzz;my`cvq-{?pma(ezpoj-DM&U2Z zRx*A5ybon4>~Z@Lh|>KY_gPd0ZI2V6Al=_5dckT|ZRBT}bSG$v*UhKuv})@LOWFB& zHzKlbu(0u(aw&XNNLiAbb!n_PDzABHzLsk(S^G3D4psC@C(E<-Nu*34UlHPLU?)}N z-Xj0hv&PoiLY7(udmM;sFIJ;#eZm>~Xbi$3$kZbC%yD(w8oOC@$VwGbcc;jpELQX$ zM)si~AC$wBB`LjNR$fCvMj#$QWgkfUkmrE>1O11P`y-%#8R-}k;-%}4i94PT)5JB) zpW0~AHISWYkvcCo4Y;4U@Z>ICwPD$~x0q(CA?hn>+RzYd#F#nQ4bncmQk{^|-Gc%9 zEX?Hie9|IQ{uFa4QG_K$Q$$hI1PG|7i`1Qf!hDMk@E*s@Nyp0btfyUTsg|IJ?}BW- zLA{$pZIgMHw1fT7Bs0{+4O;Gq6nkCVl-a$|x|m`g%q;G3nOG1D1_?9H4F{Q~N0x2J zn_d^^2ZW-C5SQ>F)J%{WSJR8~PKB|T;hk1;$|6!SBum$Jd+h}yc#T}&3V)30SQF%H zgBS#{5)p{eg+Ck8Ye57%pRY&MM1rAxJp$Jw*P=&bf;=AwfIGtWEe>cNZQ~jK@t)Ee z#3mjk{_)zwCND|XV-I7Ooeb?6rbkM3ZQ@aYh6)dMuBXT|!=L4?>yHMz4y^Gc!0rec zH12?%ov?AAm$_?W1h^`mP8wbAf`vhRUl9BwMO;#+%6dam&#{EDsr&PqSQ1o`1^_4v zvMMOj9w#bNiQZQfRTeZ!0CJdVd7c*W+BO;_qXEc?1}B#LNHB==iA8ud$ta>dBfb^H z62eMSP?AJJ5qU+^pu|fu)a01xQRNN^!ju`$MDZrmqNn=%L`>U8o-dC)P!PE@%C&*q zakskP!Q+{Yyv+kp7Ne1*6LOwjmDm(DJ7#>u>4uTMp2wF%YiXn*vLK0e=9b_1i z$Sig-zLw{rVCErvqexbsj{NfXHNB@->+N2>Q=Sk9m%s0K>z)5eT%<0Mrmg|SOk zw?HNnx}dfNGFu??GC=06(XHARoxr@JvlqB3OSZ+1`lK^4ikxv-DKa(u#s0@X;(tdN zna>6U%H!!d->20apFSh>R$OzdSz^o%B^EFR8Q!Nlyet<)sU#?>24scJ8#I`dF@0Z1B8 zi^La9+W8KnJU*WwZl%#s3_L-J zm4m`^NhyhZd>+@GL4dOV&y?^TQ z{BtHgYDv&kg%>qhl0;46!?zepfC3bDDbXGQ@!J6G!b<^m_)NeqYETw|tZ2NdLV>Du z3)r`S{hI~sPo-y|zVYi*GR1X>T*rZFARV@5;RRY%;QXQG44~ zuf9#I@AMjpux6qFMW+=#_+%a(+k;C8@P~?h)jA@&cwRYgb?cML@o67404<*m+zSv^ zxVr50*o~oM=#$oYt7o^*2F}zT2RtWeUPu#GpoA5>bzXTl_is^C3$D8A&BaB# z(I=>`JNB`DKI=|sYqh%63oCS*_Ah4Bz8yM^1wOC#CwgP5OYiFTbZ|sD*M|JvE%n5( zQq%_^_0D8%Cq{EZJ8Jt^p_rZ8 zIjBT^(8tw=-l8%dqsz*T)ay9f%xpz6udXX{-{~&c_r5A*i`DnAJ#s<>8MNfao>| zZ-TEWBJ$jq0QlmlsIjRouY_12+$99w@G6?TeOYe{WWMZWz2!pGl$igcWhk)nM_zz? z9E92{zuqH#6L0s3;X~eN{SHb@alG`(ipzZz#`t8}R~NSBn%K*7{#rDh+OgX7EHe&h zlUq+&X=+E}649;!ZAkeZ{X#rOUS)1uv>6m-H3mvBtE^!TU6yLR5tX+Ql}~$_GoHRT z9u|0dqSqNE<=Rei$7*wCnb1Rc{00q zvgJ_1DqD}o-Ui!L-(fuMk>p0u)uKzZ;KKH7`%iAe)M;?1sUqR!vDKT16;;k7#ckx| zjM=Y$``54k{4Rn zmRpN$U|o8ensT*}wu@DelLQ%IT(&WoEt{QScN|PI z4e%iP<24oV0)@w^2E+Q#5HOgNskIS((t)UmH$@(ib)ffX?Bd`#^gi{7?gZl?Wy~u} ze@{$n^zg-jr}tv>eBoZ~h!Ng!GxmFOi`fS0BjlPk-nc_;65!3df9kO<42c2B{|3O4 znwr5NEvIz>3LE+(C1j!h`4p~tP zN5r!}5f|B1Zp%#>X|^GQm@J~_H1y~qh(0;(0|^c%eH#Zs z5;YYxlUQEMYbG;`T)#!gm%^weD}Yy3pvrrkqVe}nG|H^6gV~o#aE?p7h2ih83D9ef ze~(OGgMBjChXZlyVir0IfJX--^VGE$PsfxFnQ!`84obSwH#x#Y(X)bV`1|)?|MH)| z|097%q8bjJoQ7hsU>@PS=&Lsyz`L+ALkq|?@H6Z7UF+ivuctbX^uF)ox*J&DV`{(1 zN1E9r@sJ)ah-JRRi=;YXQk$1_7DeX8QNKF#JgT%%=~>VXvfrp_c!~J5nodGd1nd(! zLNuY&*t8r%5#)S9r#neX6vv}0g3ipwvFXnmLq^XAOguzS6<=Cr9CWSpsYp4W-ifOG zsfUIeso`M=I(VE2Vk>npT}a{HnAmr1vZ{!GXs2L8$Ve`bJ;Q!(RLFTip5rIq<#5lt Z6^}`o__02QU!6MK`7h6L<~i8?O- z8amzn{H%^U94fR0^6`_8Iw-PLRQM4-L>9%L0uSW(ZALwvK4nKS;aISNKS6H7rVqDy z%H|ArJ3SKx`z-2)5(o@BolY9a=M8w0nVAEs1R*f0jw*Y6FaON94uv9iQiK!3xPM_` zk=X;ZCb)& zI|n?Q@cIP^;WyOVECEUYu>DV{i5YVspkv!plLrV_ zdnINZAd{)z8x&sN-VYW6uziBEJ}>LJujr-PBZJf+GXWPc;Jbpdw;Vumu|O;)_QHJ; zt+*zKe&D2pP=EM=OMu{VU?OJ5Dttk01ZW<>eF;&8eG`2^kp-v`{ThFQu7KR|1Ft|m z_%YdFD*Kt)AZL7&`3=$nQsuv%gLCIA_7gF}N9`xBv>z3u-xVTSo&a zKA5ZEhx~U`@IihkL;_?1oSaH|U}e}u0kK@{KNx>_9KhS4?)edNh{26`C96$tsuY$s`Wr!VRN8N1v>W>+JJk2SOrw~H1GUyLG%KbBLpHMffa^h6tqBc zCPfSt-9;j*gBlf+jDaq|Hx=NDX&)iH!G4bD)&GS_J_NbvbKM)D&ym8KB9kI0ja7s_ zA7VPZF$k+a#JF9nT!Fjgv z6KA8*BDf)-4O)e^?2j`f^ZVKbyaRX(i68{9H-GK=^!|kMH2$Rf2Kj;FD=kl!g35C$ufy(_CBi=^NrAEl&8g+O^(s$9ml z7>F$+Q+$!LBwwwVt;i=aLmEM*j;tLziF|+@iQI{7O&S`f8CMlYL&i)dMLI3M73)DK zk}#KKmuw(q8AqMW#lT9)N+c7{77w2wn@FEfmDokaOYNULn{=J9GqyjbJf@w9RYb%B zm&P3_v@Xdl+AfNdyZ4970i8Egm!D7KGB0|3Y(#S`XuK+(F1eGEdxRfz4HlgwP+@>$ z@(;ixRw>CoF+6^qUX5Xy=25{bwj{l^R<1dR;i>Qi*r3q3;Q=ZcfQW{e_nU~p* zQj*iR>vJ?+G%T8aHMy9Z{=@>fj@)#iO;Y!mrK_YbvYX?W=Q?)Yr$iRXL`(vw18cQe z##rW_!*N$+rf-_G!#HK1SDw$G%dLMN|J;u`lRy35rVg;(*fk6f##_W&jS|Cy$V`!C zloiYPmbsX*I)yeht<9p{La##4pnb0$uPxWoWy|b{(w1re`p4?BYqoO{^!n;y)X`><{m{8TXoh7TRoiZ$gXoQ)NaIXrf$H; z%ZK1+5)elZ`%mYdJu*$HC#f_L0YVxZ=il*!nK`Ti-SmTWaGIK3cdj)WS3$p{5H&~2 zV%B0t<(p^nX2kT{^z&+(Y(}pRc35`^NO*|DNWPKC5VMPJMJYz?im;09<@e{8=TRiN z(yzB$7_~0mbR4|XIH*==+1eNGjwK{ws7X{ARCA&DM}~%!Mm~?h8>!V$7--Z&7z}e> zWO!z%X=7aAU#MpoYTIsvy9__=KFxv+2W&zahE9c&g;u*>bTDnLlV%t6OAy3^#z#}g z6$=-z6d)@aDo)=TKaIXUEK-**S6(1tLf(M+o1ZP8=`hpZp{N!Ln9|Ip%^&}&pQoBb zdnd#mvl*kDerjOzI8Sw)JAFFkT0X7b?)nfZQaha1NTHQgUH{eyr^N9FbIfYLza zq_l1lIoX;nR@Yfy-W=0JYW++%%u>v8)V~x`N!L2oy4u>$=kh6^Oc6^lfwOX0bAA<_ zLg32hWZAsZ<^B5TVf9f7GVd4XR~?wqr^T20uKY587xjdvBeEnC?6m82y4M-p=!;&exPpu>c`xV*{)v2&$+|*v{&bgPt7FtCAACHEgSyh zo(E6@1UrNWd^=8MZ}n%J-W>)p7qJV1`yXrSxjG7pq2&`L9$vI>>ci7be9OvIs%Xld z+*5o%)bX@8bq$r2l**O0lp&TwPljY^`DQ)m_KfU172apRi3;^n*+@E^9566;-(^2= zZ#oSB}Pga@06GuC;cHsrT(0 z-MW@cbEP%Alcr*PtZz0Q(Oq_f2Zi}Si;^BwI)>%!@V`$5(4=INvQqtzqb@#!(b zX*XYu$Iq)Dj~{>h6CkV!On5Tf65cUiIX*)d;^T@X=t4*L{GEt@o{O< z@v*hauUZ7)Uzr)FlK|4dhflG(iH`F(ZnaTkfA86CU`IS~!GmXecpxBtAPHdsWf$PH zP8gSj#m3?HW%4w66j;c$90*i-{#0dRaOD~lO|nL&1%!gYUc&}*PJDy_-5&V^7e8>6 zbxDo{ir(rV1Qbo2H6~?jGq~Dbsgj_iP*zkF2~GdF1YssgztQ8@%Fd&Q!yo(?x7*XV zhn;8ZYu%lIn<@?WYR_0AU|3Rqkbg8pQZRb>IH$VQfAoI+DF;d(YL}l782RrFk{U$W z&#$a~RsA2`guv7SxS;>_1OX8rNqKokb{wgH&d5&|V(0@V{`W>$0x1s#U3Ot&DDn4h zAFvWITi6&NsJ}Nbw6Dno7B?+u|Ir--lt>u$&zB|i%^~I|qbzAlk0kr&NDlG<-Yav?qBNszsmix)&GB3E+kahQr&;n!}kCvZIA20@34lD%|3bYJy)bQ zvmzKfdY>=LR~8%JEzuHl^w(-F(Eqh6d#*@wPzk98`m6O;^K#|*sRUq!!hwqW;VM7! z8vbj6VTk)k&VYj$IsMb6_~w53kC~Yh2Fd^OA2z;UD^-V(--P0yiyBk&tN(v(0!$!~ z$>3VSha-&yPZuC+K%!9QmlMd{@=wVAA1C0K;3KuR>B1(CGi{=W_~oZC0-{s@?$W*m z@QeTPQ685JhWHxo&nH-g{vW9^g~2p!DcxDoE2oCGHnG%E?UCr?O3g6_X$6P2m>SZ^ z9PtLD93?^&uI&Z}D)}Ei@()q%Hhtc;mWZa5WGC_*SG2T?=7pQ2MzxS31A0V)0{D@c z;Qr$!dwP)N8A9Hgd22j-)FIQ+p)NYllpyqcU|wIz#fMI$mM}kpIyf zq37GG(?Hp1QL-fjGfF!oU1Jw za4zSyPf|iny;42}m!F&8w(%Ime`|kjdR%G-(sF;>vJf$WPZc*%7uk;YQ<*mr1bLT; z$0IS1=HoUL)N~2@$3ks(9AjKk+UBhr7g|KSD_+R>HnYx#$rvgU1_{ESAmggb0{{dYOB!a zr<7;(X6Qyi_&I(KTyT_W!43%vhwBRMW7`Vd9 zqVYxV%jhoK^M>_+kcl_d)#IB))S%&A?s$KEse)!~(`}q74)7yGAvlgNR#h ztK_3Dq$7h^&vFHt&C1@ziMkjxl(Jl@N+6M|DLyb9M+qN%_oW!o71Y9a6-FiM-i=R5 z^NW*7*B`ar4TLCiSAri&&7K7A^%6)_glZ%7n2`SWH+&cKVbFXZPGi-f1dc{tNbxTM zMIfwL4Gbz$4G0h;B(As(6`!p7CXOtDLT1WU%@@Oxd33g3{ATe$x7|hAd8@F3!CEW* zH+A_ElGvb+WeTXPgXOi!TXgXE5$DYaA%n#YY2i`K|fpHuZHH+6}d=TaoRW$}U@xE8`8U3l|dRjeWnCnzS zNX*$X9F6<;6LaLCu0+b^uIp@&Ijlk6EV9_!fX7FUZ9s#v(Bl6YWx#@cnGmQzrsID_ zAFzL!*8lGRFZ5V{{?TBBdV<0X@+~SN_joxjwv1v1O#MDvt$#YJpXSQY($u`W^?pCq z`*YIrv>QIVh=h#|?WpZ`T-dT7Cv~^e2@eA!GqzUyx7>-SJ%S?LWPXke9!wRl~0s|`JHpNtH#Nu zA|g~&y5NKEs>ghft<#RB`z0+)OH1@m(XlmL!+zam|3i%Bc{#Emoh)}T1ASb~H_x;eIUu!)OMV{}i z^fMW@KN^T#3eU@P4E-M1_w%*;v!Czd8bMh_W#`iSliSS9%w~5Wlx*01A`Ot3$Q!B?m@>6bcpw#Mfd3Q#{Sfch(IRD?Mt5TaFgY+x8q|*S{%l zY*XuMIh~-lFI32y6zs(nnq@QRynqljLQ)>li7S1B6WAE4w4|hG`{RM#6PMcwHZd`= zq^fFZY)VBPJ=Ai&;PbY$wChlvrSIovVzfJzef-+=|k zn@t_9FNjSRUij8;_QY9UR&YfDj_J5n`P??IbDF5Q8edCz)#awta)i#VFDuQTLW!Vf z05Yfb1*qxF1J7VBsc$Nb>_t%lEyaJy1p4OaXe+cx}i*nf0; z;Eqw!M-_Lg6Yh1(+Da{WgzJXQgU=LyEAeUbQf%+l9KB!LZF6UH#LW^_l>vURsj?a* zxj9);^X*z=qJd#hjG;4R?SXY<;V9JkM}RRS-Ab)7Al-$P&TW4nRo&y#4?bh#{h0%K z3cm;7Ts`0yn(1@KpKDF0+>G!l5tXwnbi{%?2ac7nJXQJu!uFJ@Z1^ zfYh1`tCi&n>g=A~URKLZffEHvW`hr5GECPqR}hTEE;5Ty(Y?RMR@6NGce z=y;<1PF$rsl2|BMnXUiF9i*|bQ4>oMoxL!I+IYde#MEuRNT&bpcpeD_1vGpxZ#p(M z){_C#oX}T*|F5rYHP4c)uh0U1JtayigHc&9J`cZpbfV zPVK4(R*B9)LJ~SN>>N3Lx1_J$nN7dHKYC{+goaV@#!~H>R+V%ri}oO6fVHW(*7h0| zR~^IuIX%ON4kFSW-={UvF}RM=$*C^kmyp|(XUmHP*)Ku#7&z_2qLQ7$Y@}4GnOIAY#a99Y-4LJ_d3r-sHrvC`T+9R_eW3} zjlz2l61-U{i?`vO%~L-((~ZJLLWXX%VJgs%qs=QR?NEEOl%g>3Q~dh1b^|cv=Ujm5 zr8D>E4EF&yZ;&Q|Y6d!gNG;+%Bqp^Z(}~Tm#{(K|LViD^$*A)%&U`RE9$_U|Oeg+; zsLQy&;rx@O)|^d&(!)QWT6bla12qCe1PTRf4pN6cOi4-keW_Z{+T*@lkJSh}jYu-` z`o7wG^QjlX>*w1|ayQF9*?J7dnf|0c2aD{IDpqypL)!Ix_Q!&_v^0y!be3gE{Wl?6 z(V;ry@d!GtCQR>_`E1zQ6S&~RI=HVu!KZ`S!DZ<~z}eIcTG6OhK22f$hf?q>Ims|| zf11h`QPDg^Bw3J%(Rj7l_d43|F`Bfa89|##h6~F35}DuasM8`u0ZH0gaK+E3O}P(x9hZaX<|p>rpKw^spijs&7~SA*3}n)dHn2usw_oIu~K**qmb@9YFU)(r)bz~Z|-WzI)w`_YP-44<&N#!M@$xtQ+Av?E-r43 zWbuA++2iXX&;qxOBq@x4?|>{ZwXE)Xp9_WN z)#ABdP*Sfmf+Cm8nkcmp`iV(d4AZ>ngoE$8k7NV@>`k(7FTmsKx*d~w9MI_b*8TB( zUEf|Zk4C3e_aXRv8n)re)F%A__L4;W00E5Q)$YW6<~ok(XuzjBK-(7-;poPj_kzYv zZweFDCJg^f#z#et3njv@aHBj>u^t{RhuaqYia!TX3ynil&7Z^ z8a#3GLt2LJrR}=n^(?Z4-c6ta%!%hczj(lenCwEe`>|cToUk50(hFUKP`C7wrygIqDBhFOV!8&f9b(hQ; z(c$k)Ib*(d<$dq+U+s6BB5sF$B8=s>HC2n5xvkjkOO;pmdfJNL#;`YRC&iqJ=&e}m znkya#tG8+iAe^3uhr%K{yPVpL+ICJ_!g);hVmx!^NE0;3AJ0M$lT=HRSegS2UP!8Q z8t%LT+y|J9&pzg0zxVkirk1+wCzZEN8%9~O%OawMKj-CU`e}*UE9ycZ`T8#Qs&hl# z!2{oqe|CAk!nSW#(ediN%q2$*6L@cMxLoBB$-=o6BPTH#L5BE$+gtAbXuKx)ye3$a zB$dU4Rq%;i*7MHX(;vAb20xGR>Bw?;Xql|9nrl!#7_UB2%@_XctD;4RYzuy^xJKY{6ss2v^W+u6$w<({Qcp$byDBAfNiR%D(U;; zx&1wCevp{>5k0*wq5ykJ^RxQtQ15vi^1RmTsREZ3ye-0ktU}#JQQ6gWtZg4EPFKUP zBDHXf!met039$h64}eS)zT0s$HNQg0!1XHJaO||xK?|MmoSv3;p&oKU+RqOpSm8|B zgdqBWsCc8~QJqQ=Pld}Kkd!04ej3Zm$gFoFZ`_U`CQz$T4y%^+)X)c=Ut7CXQeDYv zR-PfMnpc;%;_z7tJ^5a`f1_D4cM~3I2gr>rqqTeBIhL``7#yLdWRxtlPDB&dFiJR> zq|0xUV}#C>?T@`^$_p)`csvlb)h!#Kf=Rd&v8-*nP*=2zn6r~eKbB1=TgrlGYILEs zZllKxbtMz+9Q}?FLME>9uu&(ex2_bK@dkqBe_>lhb9W~u_a4{)lCN8vunN}L$T~-E zY@EP&dd%&tk@=}|3Yn%K zqc_~OsMFK|>;tSypoXZXY^e0&PYwc*-QRuUQc}!T9oM3J+aKrK5qEbELLFH`I^e)=I5o0Oq&)cFl;JJ|2?A3)P?mo}>X>UhrzALVhGm zJ2~rHYJm$3qjrU|?X>iWK9)kDm3K#nJg`RPmgfSrcD=Y$sm(B5w@~*N_0lY_1GO;a zMm&EteR)ZN_5Q3vRQhP-eT6PuD(XnBT57d}=5+XFQ+E-k%W%(iu7XZz)8uQWcZTXd zin><5_3=XF#3M>qrbv{m;}FxLR(CoT_{|@K>vZ{plvlFn@h_Y6D?Dp?fRkLwQx$DV zvkhnwL(kUOq6Fh;zcp3z#U$)}LW<3r((ApQ)HJ=S_1|>kEMWJy@DrM5k-KlSB!etj zsJ9)8&59=~9~}qo(_0@W)Lou>$79^j`C~AoU}^L*bty&OkTzTy5t>1{V*IUIAJd<2 za?+_xBJF1MiKXZ_G<^7;cM1{sm4^nf)ix)Y7_~okVywCPuFQ0{n?+sk+;>AM?i@I4 zzeou?zuY2e>S`%i((fx!JkR^_ZYNc7^Jb026cl3Z`3UM(woEmSl_(m~TPAjDmi?9U zZ!~lLBrT`8B)iTXtCojiL1JwW7VHX`JF>v&b+NctoGZX$(P`0vyg4$!B-4&oiSb zt@{sH+PGt%#W_;e&B?UmEK!l{9q##%pkD84r#Q^diXq&SCS>Nqn#4T zv{7Y5wS_Yt!_S~%NY+96&KhVLrpy}MsG&7Gqp-t$^)9qTXm^Tt*9X*^-w&f*YnsYJ z1E4zM;{dy+-HTMOVicnMHEvMM$lA8gjn&3-tV`5}CG?WG4FRY>J!iK0cMW?Z3QD3~ zU!B1~1gN2yjl;wm*#2}q+vC9I_dY%jKMwq5ft+;TS67Qq#zclZW%PDZVj5Pc0jUcC z{&hYTn4Rn|j~%mYe9No;bfrvQ(rUH}d~4(Dz+SCl%^jOB6ZuGObe~$cr?8oqVb?#J z%6vN2`-HZw%XxHq`R$J|)k%0r=<7wA^MIy=Ls*`(#8Y-3IVSQMfyEq?np)baRrWs7 z2JJY=EfOf+-)YIy$0@IR;0BGI)G2{%H_%}7c83-15?g07NNdXc@S|OoY$)S2 zf5v7cclp8yJvqOqOtOi-B(5ICRB@ywGv2$HLkc!FdK&1cy8kX z&P53Qdt|n2^MXP^Ww%*%#A0tO5&rO$wlB>(`zEakU|E0u8l^GIG#;)#`3Pj$DXF9! zU`f~@QIHT-GcHCXjap90NjxMiu$w0B>INn$cHY6Kf}}%*A>wk*Td3*k_bc7ZGq(G& z6X~K}@7VMeH@S>yj8$?ZIm%sPLeP+!nqBTPc5x)(@Y%s>`_+ZcEXYbhM2}tIr z&CUtK&}buQBYCZ|k!R#DUgU%yZ&nwi>g1LIuc2<#?$* zA>Z|}$Y@FOzK>^-j?vgulywzXL=X)9m3!Ws!i$o59oPHFdyH~@SOwHtt~U`G-Wig*;&cmaG!6||%ygf%Qa`4nr|SkRu0n7<11|fj3Pe_nU+z!-R4s-`tQF3e zouXm2*N~02`?!KT;qio6Y~2u$LDu10wW=NjmwYbDs#yes zy3bh(vgj90+6SWt{Ili3HtK6GHlJ1%C~6yQ#tKN|vl}HQs6VD1_w3Pl>fD`fb+wcqD4EP|z)Ve3@c9FTkSkHv)tAL~fht2k*0T-RR%Rz{C80z&A@B zyheEzSU*#UL)}eRa4!jhV94gQwyvhITW4MGhBN58=z}T>RuXk%gq^Mk%AL~%ktkf8 zKZ`0^<~lLZ@EKwU78WuEuXQ3W#$D(_p6S_~I{BYu?o~nC4I7V1J64Y?Xz-^cS#yks z!#y7;Pxm(+zba~|mtSn!KbGT;snAC>bO`oSh5wpwSYOUM9CsHcFrjE%+KuAAz%;2e zuz!j7vG`W8oH~kc6huibs-O_-Gpgz5 zj#Db5)doi=zM`jCSJBwXzg}=oi$?zuR<%;LlS@;ntJ2ZW38I8)Y%P}!ew6b zEGYlY^F1?Z#md%Is&oF&C*Z zPD;J@!G{8=g;S}t2%ZB`tqoiE!pDPH3a+UW$66f*m`@ZLOVv8{NLuUnxUN%H(z1vq zb3c#!5fCqhS>d22`nt_?%cE~?>8yNf=k>*wTlWfGTt=G->I@MFe0JPL-ye%F7&M&Y zt?aUOVkyoW-1bZ9SQU9c5gfeBKW^_Don?}z^#h8%9@oB?b`}aYPtwS#=oh_yCIV+W9MgX& zeN`3!*#K|X;lR7(QtD>;^AGCPzeS}Rv?`)>{WF9$-k)C2gOs|}JAdVBgMwzx3o14; z10JwTCUV-VuPd_P zYXoJah_f6m7*4`IGPoG8Zncmq8#Bb&S&4Mkie4J&(%s9bGl8fK)nhW$u@Fp(J}qDv zqi{id7{tQ?4yl^;aMxCK~>a?umY-(P`t`{tWV{9p^BLp)PajoT|ZM+)N zfZDTK2QjauV>4R)zC)}eSKpgv%xe-ZZ50+`!$4>&@hUVNajhZW!Bk>juio8Hk&q+F z_K3T!)O#55Zo7FVm?3bH4!i`QQx4DUwBz{`U-aI6Wxa9P1@cvvR zcv=Z#5j?Fb{LF&~2<+SX@*EK1&qSd>pGR<3Qlmfc>!G&R|# z^lZ?8k35r~+hOH0+wbv!>0|)+eaD|FKveT3FV!&9+#|950UY}n@At9ziS~2hU-oeD z7^T7EjBL%8Sh09Zpbe1(R?astU0Jd$8*76PKxNj0WLC+N(g5fvMXQBq+Fe^kxn>pU z@aLFFS4Q(CTDXrj6(gNWmC5o0q2 zD}<=M2s`AQrHqDfRS7PbuTYfJG=P#{C$Q$)c}z2lGXy_EZ?*1+Q3S3KlfzOrFk6A= zCwNL_yip#{nEF+ql`(X0^0EpMo8UX@va9^wh_WU{XhBy=)|^9JWVCn=4v7DCc`1%9 z7TN?@0-8AZJx$3Axa#{F*S?>{OcJp@xdp=;G|6;Du(an)`UBpQRmugFq4vW} zl1w(aXYpTF*WM0Vvd(GJd<#mgnVAE@k^Qi&SnHawd!eltV3m#>HAgZLt<)Xyu>H5Q z$MePa!RRYH!QDHWrCJ{k`_3gokh23fm)$Y$}Z?o=dTUgo~2 zX&6pOMm%a#eP7mfw+!aE_xP@vwzs=mW4=sJtJMU{?R3UL;9z+ZpvMZMp)uP+eWjV7Z%OBmg&DCAVdHJX;a z*{*?>YpV_BbK3E7S9*0Ci3UG*0Yt%@WBqtn$GhXE4aHY0d5rdS-*ts)J`}B9bT3n{ zYVd7e8+D=KXB+4B_s);aORwPe%PyR+qDgeE+r8Pjol6Q^V|Mz1;KT9|@v0^<^h1Tu ziItgIUoe4J$b)30qSG3m(o#}+RhuM%nyzc%C6E1~bcwqoMqw;%{HX1w9nlo5o2Um?*637vuEJd|JMW9%>}0M7_;{>SK@Z19ba! z>#@dU61ybXUy4D6Ev}b<*n~Fboo%3ds!=25G?kKo&6cq&tUl&D1F(|Fqe5?5ihmgZ z{iM_~H`&eeUikNxjD~ntW`soR$@ySd0}J=kX{jZ3azVXtek7gPi5bq2e!$xcH!l@L z)Ed~zE*+wtG%~F?LrIUH0H<@ICCt>L>^W3HN3XGw(Xa5l8+Zzsh3RIF!_x%*mNRK$ z68p0dS;I~Q>+W}_Y`=NpTb4m0AJv};etRlb1GYYwYp>E7pf2IRTVX)VunE6PR9}Tc zegv?W;ZG2ABHt|{!v0mYn6ps*j+-%>Aa#G(RBXUVp1zRRCi}u|E{-N_>LUYpU9-b> zKoWW8RDHJdqp5p{rD@zt!PFLId2rb`(zeD%m1r2&9}64`dm@qP4|@YVP2YrP@M$cW zpy!k#8r}@@zfYg@%8y#eG0b?XrDY%9t{6#;GE%QO{!+2c(tmUfuEV5Ch3{_{=qPL3 zN(GRTn_#P024$5o&x%jdB%<17CjBZnO2#CMlFlBcDCb?Itg=&6-4Rb_Lf*7RPmhg@ z`(=dguzMgSi2}axw7&geGLpcm!Ykwh_U)4cu7*fj6r?`_h%a5JTId*0_ODHhC}(2s zhVhiJCXFrsVg6~5!3(%?IH-j?s3m!EYk#4`JLoqtp;efIyu1(J+YW*o17V}jTl)4J zc@nKgUa_2`XnuL^qQ;u9EBX8cdZ-v*aM@E zlK&7d!k@t*v(J*!@xg&#wIt@1F6thq$rmh34hy*I{=cZMuaY=MmgsjmuYgRRlWxQ_ zZ55A7ziJ&4K0+sQGxwmSsTwUOcjoGiKh*3f%%+2K87zAYTSaU*23SmMd%Y}4s#+OP zaFW6M!K4-$JPE#}K6n3C;S2VS1lWrW!m!$b0VOf#;`zH%3=-tNlxS;(3Pbmn5Uc*f z*=hTOBqGF!0nq)N~X9-807JZwOi;L@OaG>9BVa6<=4k(Mm7DTpKiC`=}ovW;@ z?E6VAwMW<$%L4<)9e^6v!3P&V2#xBLan)xJ zME?Z{2FR>Wq#L+O{F_kl_x++pdS_q~a^;{(Fbo`B5{ZpD<&_Q>U>e+VlQuUR4GP$J zIzC=b?``|x*!l7HhNG8*&qlV2-F@z;e2A1=G-oPN@d{Vo-Y>m}XY<8u&sG>m){r_L zx7^*7Mos67Ob;A;{lL}Rohns*{MPFz+=oL9Sf=hQ@tF(?x|f9-(?K>E>Bc^#^+bMS zvnn;z0<8VcnVLEyTW~XM*3h9Y|6lOn8z1m|_hc^vEg7lr7GaKVN+Oy#-eHbYs2aoi zR&w)N{xZME;Z;Upjn3k$#0 zMkHsi@y&RX72OuDqrKmR+Syc-xCUKs_D9uQY;sk_6aCgq?7Myjtf-{RIie9wHwqL% z9Owo{yHKOBl^bRv$klK%smtbCC?*k`=L&o7ZZ#>qNC zUnLUHhH7Pv=me$$c;j4ohO588gZnm;#_rIz>dMMMyY2D^3Weabq@zw%Kx2Lf*8+L< zd3Aq(fAD}Z4=b6>`Eedwu5xf)R_gT0%R;G=rl~THYT3QURwNDI`<+q*9`{l)i(BHz zx>dUjH=TS~bX^B{x{4T8&=^xGt5x@kte|qO}3=CtFtLJ{p}Y1c5SuC z%0-A;nZr>HX6t2~yP%1P_~LS#h9Wd(a<>oGh0Z{XL=v#_F1h@B8|aRTRo`@H3MW<*76Jc@W@&FR>ua(Wy z7IG_e%+mDKh!D%AUpt8QFmeeO{hnm5)p4_e;l{0m#@%0I5wl?_EGRV?g#b_lyaL(D zd~crGO~(S-udCZJjMkyLas^G=a_H04f} z#-i48R37GiHMa{;M|uQHq#+U#@D8&=gLc^-8fZkbP7(dUar8ub4``6Ooj7q%d7~?- zD5q8fG_nc>zZB4ynhZUDBU3Z!bh%W=6u73&F^Eb;|N!NP-#>HDV5Jrwi3Lj}X*^=Vmm7NQXz z49d2ObRQ&XHRQm-!L3Pc|Bgvh{6FOR9U8D~@F`6lZL=KY%*F7b)31m=A$c%B#7 z_R%>NmJ(xrqPj)A%y0;&y9n(+sBkFz3cq(gP)sxr=wY5{ss zvZLf$|KE~1A%;o+Aaf2HjQzE6c+RXR%@=nkwnl;@*YbjF0L7JVOd(al8qYnR35H?rsMaV#90}yukV^}Br06kE;b>ZV>SDF0xP3+D994$vX34Qf%Gtu#C;W2 zFRfq80I~gqD1$P3C~X)Q)Aj%|gHyv#J9vJ3D<1IEofv7|Dm9q694{2wkKzy8fE3OJ zr+>@H7eTZ2K#Yb)hr>AgisIDseq0%NWKTA8el)7Ao^vd%;leosW<(A3#iL+>+_+*f z)hW*0k`Q*gV0*vq5kyjCd(8bSzxh1vy@f~EO+Z>53(% zsEkOKW{%OmG7DDZ6m4r0_7u8O?R}wW-avSnp>j&rGJ4T*L#R_|y%F|(5v}O|K4SuM zDQ6Gf@dE>7(=%)004!}PBX}89%I8+ZXo2a=VZq`8KSK}f^ap=YSHA5RZ7^Sq@X%w| zBX5bRHQyJmV{@}F(YXZpLbSND(qrMZ^=c1&Nv<~Rrv=XC3jH)eoxOSkx2rZ`-M5&8 zGgqR~7{hCb4Dl8iR!%~s*8KA6sWd8JBp7$ZDM7;Giiu4&bv-cqv+H|N zcKW*LPJI`KukYereeBVosng$6(=_iK_cR_?aTIj!KyYjJjtO|l*ifi!o7Dq&a-V-8 zv5h*a%7$&n>R(btcY0Et7lo@$pUZ&ffD0_gi-ZB`nhvr9MO0Tnk=g+u2-v= zT=xbeyE%ooJ(nO6-}InsF8+Dc$in*Ul1#M^Ad zreSi@mx^olR{NPO%{uk4IZX7*f6rO0q69PU1R$JZ?xGKiwL{3fV(AIgk)Wy3ABL~E z8mu1bC&;PHuX9e+uV1}8DrswLlcCay zJV+|@u!&F>)HwuGxK2u~~-rDT|5M;^>Gl$cD4H_A*U&eMWTB!d%n=<|$e>3@YHeOgFeY8JS$d(v*OihlUDeLBbFCow zK_e#POHZN$MT4G!s-l}#NAK!apq4~vZc_RfnsFK_Z5${`H8U;Qfa`S{$1J81Y#~~z zhpVJ}S-6f2X6mf3pZj%){>_OiTC0@m$&-|qVVB>vFiEtiP{{0T0(H{RiNA`%d|L=_ z{R|$IpR-b;5=0~Cp{XCJ-gO%JvZk5eG-hfv94&YiFTlAuNvv%iMx=?;$8N{TWDYX{n zVeA;7FE`CDnMsEKJV^XU*^`>d^+aNtfo&`^P_*oX+dV6ys~1ff=Wm(s|L_<&VjypE z$4Ij|(uAsbdp*9OLFz6Me??s_yJ0jD)?T;iL$}j>)*e^A5dMNDQI!I-aHx4OAO=vZ zf`Wo~Qia&%TS77zop*CmjvHUMXVfy=%WJjc=ow~1!I=)EbOr7kSW@GR=S2U)F*g1O z$4Fe`9h{6aKus76cU6j4IG|-6E8j?y{(>=%yd;)D#NiGCT}8__D`BZzX+t^PpmmES zea9B|v$hUZ{5BZYo$?!W&A~s9yt2AmmNPAftui$UE`>P4Y8`{kgP+o^c9AKPR$sTn!b5S^dKJWv^h+UhrpU;x?^Jlfa@H8e< z%{ZLjUVHt9e6pOMbN|;(#xHW-%-sBXPKsg2x%(AQ*X!9>I+F`c?{#gNFf7_hV^izC z1{WDq7Hq+b)PQR6^|HG<`s?(3TEDjc7$-$9U~ayCEGQ!*v-|5@@jdmW52I^_75d+h z#nig=MVV!bW7@VQ|B6v~meqZ)DJvDNut&4R%={$r)g?GtiB5Xh(~86eR|OU2N)!GV zT@TJGRNX`g=6s?(YbwZrU88~I29rewJNV^cmrr7#(syq!5d18pa11L)z~$Al3d#J6 zkJdeGjoBNI#wKuWzTN%m`*3J>cWl(L?Q9X?W1QW)QD(a$n;o7hb_!$Zhh9ixJ-Xrk zADV+zq&KCsVxGlRp)bs$;zb6N$vX|`U!28&Z{qx<1kbW1MK>KP1nk2k`asdS$f(RY z^f_tsqe7&%QJ8&LVkT#CjD~kv(p?{QMFyv&+$@Y~mB5s9k-_3kWl6`ShP3edhXq$< zQR?c?$m`eg@RT|rnnM4yLG$beNayK0?o3Fm138G2XHOE(z1=Y!D`hY0?~-<lo&j4c?Gax^IUu^QH(5)D7={>*}og=FLY4zAd;fLKu{0~ z=?3XWx=RV^?(QyWkQR{c?(UFoM7q1X;nLmk9gN@W#~&;f>z*?+duDb%`?;m&%jXUh zoC|&u-BJ@Sxe_b`nyIkr^X~*%5$5=eqa+Jn=){G6P!@p9-1vsU2o+$)u(|t|9Jl(T zhaaWmg2mmqLd}by$cj}>U%9W-ONKe=OOXWoYz7)% zenOq{I>Rxfk$AHxSdPQm~*`!K#4_M^*H1%7l^6?Mt4=K_~bpR-V*Q$`>}F?^)iDq z5BHSc1@v`07bq+)S5+=Q5l+#G@X1u;!p6vSqksa>s;MlxKWepI8$T5dUR#xXHJJRe zu}}4J;f0nDQ743&H#rq#L)ii5qwqaU`@?Qjoxui-xtmi9 zdDGg@FNt%nUQY?{^PPSvZY+QdbH91rlyXPhfe~xwAG08qXai zj+WD#nhhs&Nr$)$YH1Z;)K7^;@&f3z3JqObkOP)w6FbeJNKFai+lbWSRDx?;$|c_t z2)W`4f7I|7BqSjN)&4LY5xS?lLF&I;0BsR589VVG&--D%XV!qVE;J)MXI51o44)`# zDSVTI-PT(omMWN%kLGbqBX@YGd?rf4qlpEpR;*kkFa4nOT|!ESTCQ3pvl;$@>*O%% z>4+_O1WS6wB#+GEW19(q`;EE9teip4R*dc-HG)w+t6HXzfVhJz+PP9671E9z!~%6S z&_{N>M4GN?*eLheCGg0?S$S!M0I{bIns{);m?mc*0irMRY)Z|_$M|F9-=2Sdf)8a6kO6i*FX$+FodWl_KYlF+XH) z$@3n>yjyRW_2r0jfkOEIb@9AUy;kzMxBQMiX+WEWBV=5z=| z_=>m$7^VX=Efi>ANp$n2Q(1wV^>9h9}Mve=`5gQY%0z zIowV6xR0vL_!Sh${cxck2xIY(kdUfky7%+buSj<0k=n}=eT$2Y)h93J?lln!#8C*2 zD3dhFLj`|T@#v7!N%GNkD)L*%M#kxAxI3?!zZUDfM}pQ@hm16QUlYP`0T0gH2}ID* zVvhFmii)P&FSKs@l+xihoA^Q^yx_fphvlNuizlQu{U&-<`TBHl}qg)0DI1Sc`*NK zvzYI|zl0DLLAEQgA|_yOCihAV1NHO8h&-Gu(SDm zoefcDR@R*s!PKW;JNTXe(xuyA(DU1nsEBqtOx4Qpk~b&qe8a)*|*=wc;9jk^Zapi4Xg;7~7In z1EB>KDjks!E7N)y!NKaM1Hv%J-4BHr~sNEiF zUVPlqO>EKD=F|9a1%N88Q;g`2rjTc)Et_K?-|^Nns?RfIJ^F&Gcv1C zreso#sg@uHwx`C~N)M&nbmv64AkU&s!y`$Puo8WQxz32r?8TyPXx~hCt5YFJX7o16 zrLx16iL5|CS5;N5JYy?50+1sT5(1*5QJ!N)X*2>#K6J2Hm0z2K@fP~n!Wg4EUgDjX z<~wPB4YAJ=vf=PoXxN~#7kA-4W2bOVydJ2r9UkafCUZU!K;1W%Bnb#@g-!06oy^P# zV~6qlhm%9X;RaSY5`r#Z>U*fkJ3(SgAH2nNdC9ykdFUnNsx{Hp#Hux z7d%X;0GnuXUBfCIamW1eW@RVaX@`2jGPN9JE=@Uv+4j5KiPzf&v-wDZrnkYE4Hu~r z^NnP04f#J8LWN4Oe2txnxctOrdAIgv0jKfJa6?c;^P~d#hf%RR8Ar_HgxZc_sUJ#P zk|zrQvT}|?+!D^~)}ZAP-)hYtMpvg|TL+?f+a=&+0Twd)w91?dBQKf2PC>UA=c1A=FfFve|;stdeE)vPC(=RSBdPPA;2XOPNPS5be_JlC(_Z?4s4vF{@IZ z=7?hyK%olh={3vH35&#Pq|R3392hWUcm3*mu$;V+*A&wYjUA;a8zi?lSIDBnN>Nd@ zJUa_S@{dSOH16pm)M5`%>GU&g;B0=XFZ_sKLv5nMdJXDp? zIQi{Tk8~S`BS)c?yKcGLeckbN{lQiDWHg9DvI7^n%pXR}bAm*_AH}EkIb5Q=ck5FQ zgQh{;Ho?H%T5fyKBtC^LV$5-n`k|J>9f^=gSSK2%GRl_35OU*{byZBkKMewHekilo z^sqw*FivcZW}M-zifhJIs3t%ziY9 z%BXtt^qs6%@10zvnvC+~x7;olroO5sETa~~s0iCng1}0k-b#-zwA>6@5wf#m&d$zW zx8py0vfFN{ODFv4;D3MrT*`fP8$i2c0iV|4s>bY5W2xMZ{77gMv%?vxx5UgxDfZ$q zej9R)G#MJr zkeIh}!ZXrc{XuYZ=ZBYS8D5syOORRNMNDC3HK?{+9M>P$;L(>;M+W)(egK?mfC)

BgY9$FB^DIlR72W{Rsr50-(@B1EMO{BZt$%CRG^NfFJSX%Y=C>1&JDP z*37kp#MBTnsT}jf7#pm=ov%|v*(L&lA2wK!DFdYNBERammRW!5xQB;gTR^4}o#&bD zpokcXtjRDr-{GB0(|ikZ@S6r~R!uyw?33Kh1jv_xVBNcX5k}3#BisPMrg57QL$3f5 zu1hmRTm$LYz4yy$FBuTZe>=Jcx*WVKDj4#sAr@5v8>r*owkLWzF4#@?@QvBUrP&aV z;6t)W-uW=>5qQ~>0p4CcBlZpsIva&idZ*Dt8{Z0G^qheDSOy9%YBU-7h57CI?zez| z(e7uU?Qq{6#&3k$-}0@UFu*0|XooR<1Sy%u6H>e60{h%P{AGK%V{!)Ilc*lcf&^$I z;;bJorq|bkc2}F1IlrO)M|$z&J;9G%>-KbHe^X>tegENhe*`mMVG3?genG+Lo%@$v zF|D6K(KcfU2u+WfzmwV8N}aTLJH3_v9UQ<=8k_H@6aTQ`AaC^j5CL{Qi{s++%ucjd z`Ht!&fzjxAE6HkYrOk(&mUc>X7>;#eMb07mC!@t&l_d=qw_EQy3a|dE%rPwC@4yNQ zrfqz{gffo}&c6>Zv zv|EEo{s6uNE+PW=DttFoMM{goG!O@u$WHh@+xkSE%+iZSSniOL!&8P($lD;WZWE6P zLvCVg%V+b!s_FBN`4`NnUBIu>XxKDI_+af32s_dsq_btY;HPzDQ=Zy69A9>ebzka=rPQ5wX z*4yZbaGTS2?>(njN~ir&7gqA8%WYdr`d}do?PSpt9;MmC+kV*2@w$85N&z z0qdjaG`NTBRcZdDMMPejM8Go2GWKh|cL*Dor4bebK_JkIHQO`W6(_uGD&~ zz;!vJ7tiH=S6i3tu#* zeV7WbXB^T9FXYR$C!MFN;A(Y5uhyrBBh}p-pl3dWNH|m*0AxvHN45aGCw2Q3{j~Y! zhNJ)m90No~+xU{S#Zx^R(NCYeL34^{q~;qzhy{z!@Wz3LhoA^4BS{&fIQ;arnj zG7s5os<^E;itHpv>mEWZjy4BkLU>=r{1`ERA;~M-zR|kMoHue!7m+A+FVZ2#1Po62 zO79{Qqj~vjqUv|ucFSaABd!9jIwyVncI^umAvtz^Y?bNvu=^9uOLOFZv z;K7-+h_CjI@Oum{=0;~S`o)qz5ga*4nqORgB`(&ieL*(&=_mFt@Q)Y&N4^CZMo4;a zm32L?r~9)mT*n0+06N~MTx}X7BQLLCRZ{~mN_pCb8U0x%KSL_LwJVmf-!i{p!GE$` zg+@r!yaX|vRkfdRu&gc~yQR`2QjC$^=4kjWps=+w2 z=4{33sU5pon#!uB{EMo0CFNbjB3Ti7a^m|f4)KH+IrVt1s<~-l!aQc84I(^6GW(@- z(AjV~35o^trqNYikS z?#o*1=L;ly@JP^>zfnBf*qeZUf{j(a6r+C)3mbw;rHD}>6?UIhgUN6(_{_jVTesgr z>#4=yCc);^e^mB;(>0;S2$LCav+KQSFw+HtmBbYVS0Nb|_iKs3EYZb!t-c7;d`>H4 z>cI&>S-h9WDq~nxfRJyvl@0T@mlT!GiZD{5J9qaLj+vB|+fn>nY!Tzrkt~aP0DnQ^ zCo!idK{tA3NYZ0m@LpD4&4S(Cx44FnN@?kQNq4}Au2x>y_sAxONTGIdt3X(giQV?R zk_nDrgk^>}3~;cs%AJ%|7X!0u9}njWKy<@0qNRh@x3`Mi2hAf|>s438h9bm>IDn$V zad{ZnwPrB&AptTSYXAdKck9D^>uTfjLuu-1ZtB5o<*VnK0wNZc0P>Yj&iL#QzZk($ zuRr2}O3bXw&wxe30>J_3h*FMZHk%Kj995yIq4M<2=1W&-i3TMiW8g1@#IOjZmZRj@)oyMAqO-ugbgMv4orp(>n%Vy+dTy!|W&oXKF~RW@#J&el}N4W1V$mr|4`2 zbE}NLJ7s!!l!foGnfIZf<{2Y|_R2Gfko!Z49i^_)sy~Dz`Uz6@KDMSGbT9?LCVYEU zNj!E!t$8(Y!P!8EV2U(LB&MfuNb6nL{d#^OpyisXtXWg5yIbmv8#W0Cws8UV;Rb(u z{&?NZyEbc-)(L=^U>X-gpS?C0 z-n5po1ZF_z*b%2|Mq0!(3_V@UpO4a)Kx(^j36&hhzsQox5af=n<|#T1>2}DWw?pG< z)e{KC8Kq@`&{8K*l}^3;ZrgwGO=Di+T_kA7{}c z`w10-rTCV8%gRG7js@*)Z7dJ=a|!+~>^*=^jRKGTI(bD2o-mioT~D!@2l%2C6co3Q zK!NaJ(nT2qU}kbgl2#27wt>dA@ulrZRq&>{(jZGp?7M3WjB~tV%#+Mu{+vAHRruX9 zlAJFVh=QlIErYpWBOpw#i0ssx)Syc!3dELu@9OiRHmy*=v*JK^CNOlFPD3W@p99Dm z26q|1YQ94QItg!gcOufPkc%t}mpBMKi0o-_CxhU7Ie9S747lEHC6e^IN;~WckT=sY z-=u0TD&@&cg4z(RvFDh(N_6c}(xiT4ZvA+kDXv-(=Brkso8B837_2@g%`F>tNe>Pr z>T4BKqrKlf(=wW-sJ&;3vEX_IVS3-MT4&oc+A7jcIyfR%_c<(8Kyrfm$uJl06(XxW zkA9pC&L^-X1>MC50<@u1DHzUMkk$Q9pn1!%kM%aoW>OT0D~pQYb!e=yMXw31H!Ri;s`b^g2AV5QP9we`FS8=Rrz? zE*66Us%VlUX^>^1o&(!ItS>RmXYKWE|CtQ&o`o!0Q0|IGrP3+vK3!2qKp(^YTseeX zoYk~ne@grM`;WhKiPz90NKDta8_C2RZnsQh zPC13_2Cq2OI~GtfpYb(G2#XqC3p4X~oUA0AI3 zsArdNuyCy`4u)ka6*CL3Wr+vGSSC&QUD|?_D}R{P?>2EaP|FYuWE|)(oLmQDXtVT= zWedj+GLh}AHRy3?5DD*9MG%QU^$Dmn$);*za?5r@jroP2z^)F6`5N4i&KNe%&W+oK ztHgRLFl!p(?p?u{nr>g$Q-$j;1`ie?mCyddv^~UfYrWKbPG|%E4wF~V2D;o3F4m4F zTi9($9nH3)!q5r#80lSnyk zRo`XYP<^|HF{QihUR8f}YU z?W<{3bddwUeb>{ck#!7s5yI*9MrFFnp^E(Vt6P2<=ybReTAd@=eIi5}^H+M;PxFoF z*ZF~WN9)Vv5s+}uC<5US-q6M$U~^WWi1}%Z#mIFdl2B2-jq8IxX|!YxHJqXhqW)0@*x7odN8qC}0A`=kRIG?MYL zWHK)omi7og;WTvG&W&Q26_WC^hCFeIRTb%b(}-rX?;uB9z1&IM31l}lD4{IVbX4^` zW>4|A(DMAi01A9Xn{tJlSI*T7e(4lTwDuNxP0VIf;%3vO3Rj1VKDh@>DBvDQ*O&8F z_G=w5xw)y8ixK6*ZSS=NI z2BGiO%v%<<3_MWhow|?9F4^#!(N30-dIhSZ=j7Y~ILbgga#NqgNCXV~qspr-py`Rt zE9~LUbl7aBJfpHKF%i*vR^KSmzio>=0k2y(|Yyd)<8<-WPZ01 z$4+9F3nzM!NtibU|B0AK29;zUh5LYAPCQ8jGQy$!lOp7Y$w{rthH~(soH$JoB+aE! zFVL&HNP}9P?%ZuwFz4I3Wjg6;{ZOJ67`X;AGyP;X7?_yx&^tcF zWkq6}_mT;e(J^WkUF$S*Q?jGb7;;fCM)m91NihfZLFRxVe1-3`%o}wl(0;;S>qpqU zeDM>jj1WPr#OAaDR=q+$;P&bWz&&4-sn%J)1DBpvf3#;>$Ax1=NVs48@o6*Ha*|r^ zPEd?28OGDx49j6U%FNKX&iOt-V_}A8?(^{6Y#{wp-JFN^M~<+O!W7|TqWXd&6l6x3 zplvB?Hp5UU^%@b8qFgjLJbG83Gl9guY?>Y}jiTI8@;)+B%bH7lG%iC%`$>V2_-4@{ zG6!0&C6!9n6X{RgA-ssk6`l`~2KdN7U|*DiFE_k|_H4QjOJl95CE_=)raJ~IhIP4; zDIm~9K~z82an&W&Mi%pXk5_%gD$d>gT(kYJ;0{28Y8Y{r2r!)f#imApGdjcbgI_ zlt)gODy}!u4Wxcmidnd1OeL-#4^+AMG`g!sADh@ag}nnO3Dtu1KxX3G0-J-oT5QaR z-x7qTKUC(L%TI;dO@^uFhX^|sl{I&BImYS!Y&7K)4eb(4ybCzJGUhXm){T@h*53(2 zmE>$_6Y{Oa&c##h|{LCIttFs*<{m@7X@Sq6epx zG(I=C>$K_Mr17E!1VJOS@t1@f+uBgn0cYFG`lnCJOlmLIwy3zTzK+!<(#H-Ra38{% zmd;$0@5Y3k=h%MGW4=)4&1hMSQ;IinPpw!;P$acjVkNb&vOIDyyeQ=ec@5GCPw8bY zEC>vQYo5tA{PNE?s)&gn=d?HmiVoHI-} zSj20hqkCY#A_ug%P&vT?;@y@Y(R{@24BgGE*IVe$tTyg<@4TRn4@AQ>I=o@0xWJxj zSCJUXO{`#0iE3b_ry7`KP6JUfO}JUM4IeQ>(>Wn8@hPF=y@o4sQcd7=Bejs6it}lt ztSJgu+DBtn54c0`TQaTc_EVJ^R8SUevw0Nw-$s7&^nGiZQMvz7n)hy>aT_mPm3*lz zi27*j0t~gWgA9{zO)+djm)SQAsfgw!nToQ$k-`lxn60a*M3K%1?|=0ZCf~=I zbs{Y*y{ey`?UTz_smj`cPY3li2%Q!5*vmvPF{_%q{wtd@n2D8Xin8jqG6u;{pGM5e znscv~J(kNr5(Y)I(qx$#2{VdaXlNOvTMAo-z#MRp3>B%*kB;-i72pXy!62XsJo&<3 zzekey!A(m{EYxgv*V9Ko_y>O!H_X8>{|TKPERLlEYwQ0 zS~WEqlAV>eZnm(Cx8s0y!P3_8(Z*3JI~!lIL+;z3EjliztQ)r-o|cv{NNo}P9tZk5 z224!fwZ4(Xav`6^Vh$0^d3H9%%~{KCarZjle*+>mL5TaGeO_)o{t1i4p#LpS?49dx zZTedh`T1butCW-qzIG1$UxA$JF5r$my8x)4SE3^!#8oIMer|)1R^|$6MNavDov0sI zP#7?$y7=x4+!4-@no;ARzf;P>Td^rFb9YOH`{1J1~g_|DNUN=dEB7A)RHe zfft+3RTZF%5t#kI1r>u8wJRtVBkk@@l>psvx1t#t88K`@*#-byMQIGp(4r6p&h z>*~Lsx)<||AfUTca(8c`tpn4gLyBVlBjD%Vj6qw$cFF0bbpO!oa-pAw+M&n|0W=W5AzRNjuda2oU65vSw9BApU?GYOXM{@w;>glihmkCr@Vlgg^ZBUOMpd2 zxx^HP_vzl24CA(&_i;72WfAC6XaMrg`bR_+Za>h}AsoO%29sF#LRgoqHMKk*st=pa zB;oM5W>mnua`t@oT+*w5pWtW|0%%GK*}=4D*60VI*;ig%T%37Z++iE3oImfbzwE)J z7LVJd=hF9K7SQ+JW45a|tL}0AIY7$X!(}_gu?jf!+x7r-Wzo7FPLjafN5{1_EJd`} z@9&jVewL2bt1#}%c-z*#C@G%Z&B&fdbqIjpD&Va18sN7II~!kmG1g#jW`DRan}D|Y zINRiG+d`fOJ1TrRKdLbA;Xs+~mXn=rJe-`Q3ghDR}u; zMfQPYWT$Wm~*M+v}@1RRnqCC7h7>%1;Re=wZ9ceYp|=XC*Uynsi`_ikGSKT3hbt&5CFOXy zS3&h{Hoa#_Paj>koJjc69!Q#S@>-}3X z7dbLtSt0j#BH;Y?`$MH?=#CEK?qHkU-j_Z=JT)J=4Li^JZ#FE?r@>({S!>1tbh9c) zKwtHPE@f)-6DdG#sG_360WgI#x@_HyAM!fvE06Yvg6eqZ^+K>&Y0=mkUi4Cx)@70( z7bukhT=(HXbocv?c@ctprxU(DYG$_wMV~)3v#^;kB4w$&_Dzc;xuMUTo z-S0{&_3LnbDGp+ilJ*uV4Sl%oDu&f6cmWHO%rWcY++q&4QJMw^S^!Vk9tJviP@z0h zz0%=4m6Ow_=wvAW+B_jZZTr#sTwgmqT(v&MX#7lo^NNHxe2X^h+1^si?++)w&=#;Q zg`Gd+s=v@C(d*g0wTnsY{PNRfvj*FbmIbd#L39oI^QD){IZy@;(SCfn`bQ^!UEF(# z=eN5IxMx&n@c|liC<r2LT)5(Ikm^?)!P%nXG905FNvU`T@d*KU|fc}ApT zRn^KRaR3^x_2Iw@G@d5~7b`CFkM$u~LDMGgp^fr0bw8g;U|Ihx^V;P8fFBXs#ZPst0Ucwey+bY~Et! zyT3962Id=td5>GVRU9N*A=T`0f6|SLj!xDXOppDK8G)Wi22u;mMjFM$*!Sm0%UVEG zKRqtRtFe!P|L5ZTlEDvG!1ygy(y(q142VChyy6|Xzd1w2##Xa*LHc)IKW+dUn9{oH zym<`JKrU-BYl9e=*ga4|U=SMbSt)--Igft~a0d*yj#-ri5VB_Bc_!?!g zf?NN}jes6$Qtb zpsEO5-lT~lCAww8M$k9LyfQTZP45XAu#2lajZcqvF==UKJ3Gc?h^CW;F$mMKh5rtU4{RFGk`&Mz zK3co%UIbV~Gr$Tr0Q59&Oy96`rL~YhM7}c<)PMWqW%`x?x)4ohxo_j~?r0j8t|^`Kut2s_38(}w{fmyI~Dq^{hpurK?e!A)%G(TpuRc09*zMm>h5#kJldFSskq5d7M0 zLSQV7n(2Iv#n41uem=QTp?VDCy1>6@oeyy8d3T5lzx{Wt!5}8dfEit=L?ZulIH0e+ z02PmKhotboco|U0ULfG=7xIXG{_7(DdN(g3a3_5vGXwvb6M?iFaA+ruL?r+H=gKSK zpY9_f|L;me7D@p&@Rqt4=6`Dkw*H z66RV!Dl|9q`XuHdk&==VANBmGDgtx{C8jHG`t5h@bjFf#CM_IFZsBed2$SMu6}M;y zJxohf5uD{8h+yZ#lX5#5A4{`VS0zb-+s4c*wO2JbzA;@j?H0H zn1q*Cqa)Y|yR1)XG7B-7R;l=ELRp61jhVqRTbF~0f=#oc<#2p)^So3KRH0sSm}DJ0 znOa?0eG$pYR9f3tazv;4B_C?fvC)?9)+A7dF3RyrPHpf+zH%9kY|>&xk+u6#TthWx zEtkwV)Io>K|-i^eaVg627_aeaEG#p*~EkUC7SI^GW|v3U?{j*UHBpL^5d^! zK=*!6b+J;ZrD2~zeJ|ENbQTM^{sgKiV+K>IRT+yS;s?o-sALreMUVW!i};Ni9{ z0Uj(8sl~lV$;F<v zyWO$seJ)%Xc;M(U!^U5l@7xQ8e|Fg@l*F|bb*5hW5TyS7gu?L2U8#7`(Vv2jY&QDs zfhfH{cdc6hqxu#HWuaNycb`%U!{uX4xqOh91RcX@jRY7c`VzIJFj)_F<{tON?v# zqpQg0_H7$ca4DKwD1yWvGSTWqt$h18aab?t+_{Eh*yf_S4a8&)4_N`p$1s$UA0hiA zG+g3ysT1P6DyHrow>{*hAv8#$P<5!IOKNS_!5=Me&pD1P!Z?>^u{7VPXHsc6AMSOY z-F{XYpeVeyPvD~K%UKPO3=|w~*SHLvH7uoQ9Ns#pp%|4AbgOK~=@DhIU~+Sw>Fs>z zmYIGtP6sd#YVsyNCMwO@%g?l)T+SK%O%_5zs1pzfc{9||D0y!cXszOuNBh=NWS>EI zeq~i?bU2u35G1lsZD_lJyDRmrrED^xvtVA{%DXZx3=hXR7#Nz$O?ZK_lrPACn|m?n z#@jEyf0((Tk;tu-$leotJ5wTASNfiJ$(c??ICNG9@y_hFM%CuL91o=aURK{>Cpuwk zv)+E^DPJ+snD8;CS+KFvktFeLuJVIJ9|}!~1RDH>aHlC2G=Pa-e!v#92(^s|($-vH-InEjm(_-o3HV2XY<%vCkzMHIfN3ajA@< zpku~bw6sS><*s^n3tP70tkM@Md&z9cBH?N1kF*4S&<+G4#pr-E4x} z#nSpYNAJG7Rmw;W@|L{J4QK3RnjEpIp}lk}#mg`j~SY)53EZA)&6G|oIY=Uj4*QngwWR&ga*OHyjX=GAR z-*QMlt>nIi$2~_by54Xw9+-hE&f~y&*AR@wAlBY}CEaK}uCf)Hp3^u!S?x*{;}8@d zB}m~RV27C*BR6HHJe`YWcx&I+bjQ4NsWw4Uk>@gZUhm&`C~PY;8;f_np@(a-FazUq zb{1xKw`^n_AVgIkb!R0*;ads&*UJI&flv4_eNJ*jwL*JKX-;3?Y!RCcv1HkU*Eg{f zz8Vq9MarWY{kpse5MkfdxaCnCb?}o6F(R+W2FrBku7b>k!nr(zt@`Qg=kJ{P!NPCw z6N;-(TQnsrwpf?Vy1s6i+Se1&7=4!TaPx)f$TgNmS6hDH`RL*&OG;;bDAC7z;wx)D zhSYk^YwK|0nD<$7Z$qT<`a|1ZKe84G8#@@q(m}tdoqL#)hmv1r{eaSWE0yG`?e)`u z>`-qMhl+C7?zk4HBTmIvp~@oO>=4{EW6H29w;I!T*M3{S(#o%^1LbABBzw+PKmt;J zKr6MfGC}d=#jP(4bd)SFYg-$52}`?&*_s8IJ|nc;l`9@=eXjAk$09e2x+B6jyU$jW zMQOXQO!ikaNpqV0EGLz1Ba#RDr@PDOSX~YCh3}+sMWk4X_y*4_uZ>AMTHUwYQ-dqZ z`Z**T%zUZDDCQLt>8nvf_D5AS-VsGdAHqYWWcJ@=1^DqqWG83cRT~UyZxScm<;*3# z9UMZgW?Ucmrm6CvJgt6jd$eyV)KOf9NpA*FLKr3ZVr_Ny)_?6Q0Qdi2VCaEvjbL_d z!FgGnWFe$OTG04#s9rZ)wlA}29&c+{gg*N73tQz{JWSMp;&sk<$yX(8-=!YV21W-o zHSF`)k(galryfj__cLv7cGZ}8NE9iiUEN`Z73NZXOtEJ9YX!B4Y}0}j5e6r;8CBnd z8t_y-;^g=WJoK}z6ecb_KDVgnS&CbGN9$|_88=Db#;kpj{9+k|cgT}J(<5AvkWjXg zw3X=WVrVvc6XQ)xAvu%9JfAvRwO96Dr%e&JdbyH2_%)JY66(69tns-9$VAAcV*lG{ zun93)kFHGw?XT}mHnLut-Bz*XJ&;JvWVLTybEV91 zmm2HS%?IxUzbE35GOq*vLV;`qcKl!&+;kn1oyCX|gobPZkuJUIaS|Z&ZL&!ymBgiL&=LJH|o7F!yK_ zJwu72Vw-Of1Gh+Ln5W=e%TYe+m+odU>G~|$(?(V4$8|ma8)_l7S7JEsrul}3cw8^vzC*yD=T^yEBlHuSY@3r0yF(G z>yC>%NF14tiXO~JjX$T6!-<8@$$w_+d0Du<1VV#-BvkD*PBxK}E#yQA7d?pj`DRfk z=wXL8Z^%F?F`QJ!wS|pQRrjd>re_AO;jqt}5mvYRr}}I6=qCHF!@)|TkWn!MG{*3H zqY>wnPv55LGRv6ObseZ48WtWK$?esTQ`t4UVe+wR4$|uqNY5qM57diFvUCUTG&n#T zHd|o>Pp*rQT^9y)IWMqX@|C(#jf9fTE#ZhRe&8F%&wjz9T6bnS6ITQ{>Xg7cOZz!M z)_VtCMny}UPM33bB$C@ie`DR1D~Qx(^oLS<2Y#=5L#&zANXPs6J5eNmlc`+V#IahR zdVKeodUU-aQge0H_ZVxBhWnN_L&AX}%vq50@TU9rrMP9qU|@LPL_4{__CBhUa1n%%|_obrM`{D`oAo&p0r4k1}zO#nZ~9{iYpAea-i&U zcBeh57(i6*2;LejA9e~UHn1ocL>%~;>_u>eC^vjsZ@iQVRs_uk@X7_fXUy4a76>_F z?5ann=;ER5!Q5q+Th?99*qczI+D=Vhi!tUj_N#v~E*;2<;(9e3NXD@jwVM>3wUwG? z$J}#8O(JCv!oIULmzayc%ZEqtd$kRUvQ6IGhi??UntufGibW*hE9kf5ShyG#R}u)G zBE0$0Rv}62UOZlMCAL0zQJqnQcd#tOZjxj#EYFJ^kZYVT;!?VID%>t^J%9AB1h;C! z+GuyARKsQOE#suswJ4d>>pc{U+};qS#yZ-$t82Omj?B=OnTJH7O9mpztyFqW1)AC? zQIpg6At}qMt@vYd5)$_==j$nn7USYG%4{=ALcC}LP}EBVcfFAPhSK6>QqGsXv%NDu ze}hiQubxk(?m&wvo@K0|ran3rZA;ETkiDQp{C1*FtR+jzMw9M(ZL$h2d!F|0nse{r zrstVDXRIW-=w321ce{%0?M@A*p+{4H%2;J~Q&w*&=69pM@@=>Q2W2N5OxE?oprT>t=?%nAprT+;*+GG< zyLzy0lG()+s%vu=3_7E+j5l$r%Rg`DT~KJ7&ErgHYf@%bH#ooSw&*sdDuuEE4#(!JBEGE2jBXIDx?p^o4#n>k&fILp+KIq~gJ+}?K_ zWSkyIIMve#xO*pGX-<#kIbYr{|*h@GL5q zpEv9jTWUWV%F$F9+#umhX$iXEdii&8x|^%6#P#nse0J)uh&tK3qECQ zn+5fZ$p~ga$H@X;Fw-wsd zy?;Ly!mZD?x{@|Xl4h|v*X8Ju*RydCyge9isHqvSg;%ckaNKxA<-Gdr>@1G7l|dB( z+ASL(;@txqZ1vWm5aJu;cGF-2q#2SpmU9`&fE*$f|8@hhWGF@O#6Wc3b8qxkvxx&JzMYouo*&{AV1xFrGZ`W@Psl?=C*Uuj07MWK^VrT6h+0=6aKFG#~o&k zhsVU^l;4=n0yHQZ<{m-WsF|j%%j)5sNrn`)(iFGzR2d#fEk#$qX_iCfy{4w7KBPx} zfy@eWp0#{q@FcP074X1x;GybZmVE*O;gOLU0A-&Lub4?fCSV_wnS*@%e2I}EBMFbW z!hGuXrtgdCli?h$!@0$h&|yAS!A&1&;`;%J8=8Iit7}9Ie37itJqaF_9Ik-xO+vwv z&Q4PY@(79*nTs@Ez-!1&KR_4_0KP3ooZC!AQBz#1F3}x@HqDfj6YK|fAtaijUhM;` zXR`|d+~P!KH{9_781k6IB;c~;BR5a&^`3X^uZDKe9?=YD(@ZSWCV?5V2))0}*&iG9 ze1}ufRF+y@4V4ch%jSQ9?30c^eL8ZF*UaCksi~O&QVcaF-`?X&WA+5$ z#l3Y6ru$hw;z_U_MQGh8WO0YLf)O$zZ(2hVYXf9{=p}3D z9JxUFN33%GDkzEebm#T>V=zE|$&Ld@prcCjL3 zdppQ1cQHXqSqNxj>i}WyjM~ux#5OCO?C*}m;A3+_7@YM{x)f-hpzLEj1}GHgc7PNs z0QK!V9O(eZXcz(7;v~YNqe+PqCsy%V;$Lur0%Sv2$=vMz{{A14X~73~mGwnmLf~RH zUO7XwD(({t2mJ|VF+7%o#;Y|kZ(McE%DOJ>kj@bb08|He0sjqCeSgX25My`t!jbqH z5|qG=i8)DwabusL!CR~aoV=pzV_jumgX9Z=TjCV^(rfkb9|g3kuIdXGRk7dVX5xtl z2L_>-8ZpR}N^kGU(RO2ir6y8s^>@-Asp@A@F}0kSCFv`pu`(z5#Km>%Bg9*P_>);L z_Pn7q7(rI8QqOOoZ;=n)5x%%*kB3Ut7#fIU^0`nBfOHOa22zYIAcF6&m2MKVNN5?8 zd%(KlxbT4>G%7~IkB=yU0@_XGuYcKf2LKa&pTr7MHvkCr9f!^#hK%D88J{5NX`_V& zkcx0Q0o5sTHV7*Nf$IEj-!?w5sfmF|F^g1rMEQ)8xOX(5Tv$p$XH;QdsBnlVY1%xm zUcVNRmzR$`V=Yhr1RA{Fk)ID+g#w${43s8aK|w(qgs#+CRsEW4qP)_|OQsek*B59#KY^t3xgMV-l0e`{+``8?lre%*eVLa~tooNsU0b1Q4k^ z_?OA*4SO>X($-Dm;N7y@X`<R%DgZs#!M~Idug$ZhVm-bzi6nl5~Gp15#)TEC95)@ z1ItLj)!~->>pS#hTt*@f=)?%go8`P4BdzR9 zN@Ny>j4MMukBBgsHV_yznEXhEN_)&qY1L$lc|PLzV9H8G#oUmH^AP34GVu_26iW77 z)D;s+#)Ac8P3cMz#mxBQ9|3IPz39O!<50En))kmg)CmO7_o3}#5%mPy6F54{?KDf| zmYq3@y3e;|D3f2jP=ElgElT+M+%KE{io|UK8bXH;=d8a3g;*LwmP?teflG67n0XJ6?o^V0JcrhD8)I2pZ4 zp6c4$Fx+I|QeJ}oZwLhlCL(<8kOmO(X@ZRs8=a{;hHz}2SM&lDZG?SXZ?goGVnaP= zY%ML*keCkGN+uAyWT{+XaJDfVpq;|ss}Dj%$WKP5Vr|-S*db0Ds4SVx>(*z$=_3_i ziG^7|CVR6E8PSGmOZ<+#udu&(_p#Upb0a-Rm(<&U^7wv)@e>%uY77jImua!4Og{ z{o=&I2W};UcOF3Ku@J*6wW_m^Y;Ol{MfO?GNuyFzvEZ;5^j$y)Yyxgr$%c5~cIdMF z{Q0vG27@g?-}g^X&cFVZZN8uh?ZWaAJx>=+YFk+#O%xXRLhxh-kkFG+z+Qguv4l?ctqbB{bA`*{s9SggKZD;|cN3o*S-zm`gs_!7sn~s+ou3 zquoX!j$ntF>A>Nv@dxqqYDhme2~ESsw9n!{6p}NNNLJ}JW0%>dmX*^LE(mYr^LhiA zP+o`chlhVt@Sha-p1nJ=peD5oxc~sswCAYFtB9GSZgq}_q`97OJ1ldFSJoOPX`e%e+uvfAggn(SIc$xys>0MtGm}MdCMw##( zIG`b3{wws@3~e3G`eT(_-~G;Fmr>G_`zjXGin+_Xi7|zffGW1P;X+|tm8;etlM*VJ z-qOn1rxCm-7YrcIzRUF9g_ylEUj6;A|6ENucS4z7r|+8U&G{`@J0CwDr4CQPjVD@oKSZ>a)i6uz?1!GZMV_;|ClpKQ3 z)pQI&u7)zjOy)fw>WGI?e;pUge;$@bW9@cWf#@ z_!!=R-Zf-|6)18(i5qo^@0oiRUeH|RGI&n#z$mefIg_Ae&KAUz8o7n-?oY$- z+w9yI$4I5_x%xpT?c6)^zD(EJ5T{S+ME^>8MOqx(D)}w$yWHh2@kN@j_THW6>+**l z=68&EQ14*ammXwh@&CzJxEcU=$$ltQ8rk%R_%27$cm=Zu(xZALQuKu8S@|5vS5x!^?7Z#tEIcYNd{zrbs#fHvkg(vwHk@9Ce<)oV0 z3Ojs5pH5q!bKNw_BVjf<7ep<5d!E|&^-S4Q!lhTD?O#)57prZx_m_8ENIunBi* z{&!mM#OHzs)N-%ug6*4I|43n6nT(BS^>}>N5Fs*+%acJDjfUbW9cBR=$Fiyq$p+lL z7W}-|wubUOAeJ*VEXL2h|JfW#QAYiQV~Ca8g|X=TIkVT7zE$jI8zs7WO^LB(bQ^8x zxo$n3^1V$~UPJMKJr!ySJK4GPA_;k?Uin@69D3TI)_rhxm%8ZgoSaU_f1sh3iwVBVQN(-L3}_PZ9BG0HVt*Y#8@ zRyDbhI^Tj*P&1HD;p}Tz(F_j)&BLpuUp_Yhx%v9Uvr>vD-%F^9TRJT^=rs2gZ*NHF zDr?$YKM#Y*om)-`3SJz5c<0I6Ux8wuoTaA?A7o#lI83;mHWXN}o8jo>Z9l1}bWgvb z%P~XH_KXeUp;&LY@x{2)7#3tI;F(7#;x^v(PX$(o!dT>rCk&i0d|6j zBOxH~jR!&C_p7YwLAy4^(G7EbU(wp;p$Rj%_QJ6t1<}1-gz`P#(_*absWG@8(XRcP zoyC@EUY~1i{(e$0`$wPgc5sKKx@;vLLt)TGKCxa>6$FDo@x&tnKXh<^{T^!Db?&X} zI=o? zbs7(JHj0n-O7UEAyUf*&bEItV9Pd>z8-3EgAT`~)t$#GfYqWPuE^APr%aGe#31zxA zyW?%Rm5A-9$4-msq2?d4Ru9kKId@lw#`edE6$w=-Di`_dHX>(&jW=&_Ze5XGyE3mF z#|u^zpYEe3`z>vcKI-U=#u451nvUQGvB{Y~*c+h#UK3TzGdX22xH92hk|c}0zWJG? z(mS`Q1?q!(b%LV#&fD{V-gJ1%VyDL&*_a&Mo#~;hBm>s8AK`Dx2F_bsH=3-_IcNq* z5JBZN1oEn{e!_TMLfx}+gQf7kuu@w^i?qY*+fuKXr|K>AMSn@Iw%yxkDzNk4 zM~?z(vj_cK+oD|5xDLFVGBTh#Uhw7ejZVl@XV$*c^hiy(3r(z9uhTeHxcTCq*5uim z=R@y5$FfX|g&#jVk}Q!@Wc0|qJM~^i!MfCQ{XhN?&dJ;m*O{^@8*E0*V06{&VIFMb z-x*u%e`}n-5oj3y@1S}sfi1YD!q4&Qo`^Q5&@ESN5<)y{GYBtxlO$`tvQIYDEW{38 zN_-zkV7dMNg?-X(>9LPiH*aJZ%d;D{irV@<S~5PiAt$fMtxmv6CzJZyO-}m7(94X zrV+zvFi%ASRS`22POr!2=Vg}j&IvuhWH)pa(Pvy2;b7pI;9kvwKiGs`>tr)Z!1KMz zJx&kQm)$WlB(FG_Vvn=0^_=ihv@&P$KEbt)6YT?!GhN%K{h#uP08nb9j)1V`m#zH3 zGQs1_AMZ%6sR@996Y?$ppBGBm=p*$J+0d$kXh&$s;8rV0&}MG{9%ij7_5NF?m;}4d`91O_@Vn7XZsQy7g6Z?KvLTg<$djx~22pbJxSz UAIlX0Ce6fjLS08KPsPIbf0w76rT_o{ literal 42121 zcmeFYRa7R+x~>Tn?oI)PyF=k_g=+zY{J6WjySux)+mAa`;qLD4?$XRP*PdhTKBs&1 zO^s@nx#!Gb*akq!xIg1jC) zI0kJ9d{lQx9T0R4cqLILMy0Q_{bi)>Pk!_PV)}p!b$j% zE62$7eUi0YR_lzy$m=w&+jZa8_+Y{=Z;-+wi_`Rc>Sol?{c)_Gl5^BCYpFNT-Swzq z(iO7`5%c9nG3hbsUI;2dxVvJP@EcQXqHXt? zDFn6&35$&!h-|{EXAq=KnAa%+v;a_h9hlKca}bat+Y_UE78`9kRfb(M(jsQfDMhpa zgDcMdAZdF=W?T@X@y%5VFK_RA3qkl^Az7d2mE0H1QjNhs)L@eV=Wr0)Lb5j;AaSuE zEJpS}dLvrC8SVQ)kP^Zi3V@b?AmqSBOb%E0ez6gxxrgv2ME~KN=<@|tkQ&*qLFdaN z*tG!YBFw$tcbl(Der7hP8IUrYV9j7v{wq1Sx4vRJ$Pqp&I^U%IsG$8WkumZFVPmKZ zAg~4YW9W9lU4`D|8POsB2*4l{pbFyVRLX-Y!6yoe<>DM-9r8Fpw8Gp8AmtF5z`H?t z`l6(O&FC7c;Z&XvAqJ{+!(8HUpp6GQcNf|~cz{_2RChOR@wp&-LC6t;5Rt(D zKwuKGKyfBT4i()-A*zEJ5|fO9ExFrkFvpDhf=LXcYAj=Ma+LRBxT+=6?buWiLQ;q;6<{xWz}R+emcoVDFUdF zC@)Ht%Gl-tab#qQ&vWMGs}-^p_$4MuBgj-yHA2Tw_fR8IJ5a4jL*oE(RdFf8GR|#9gyTeMu z8j09NL@Wqt+>yd7lH8(gqPV#`hg=SryrEhG{1O*=(IdlyfZ?E#s(8BO4odDpfv?N( zm?VKee{+l-8hgYlCfO&3$FI<7s_Uf>BoCwr z`zqr3r!dSM1BpsY(W4`zb5Iw>_~(1xOgKJukL0stkEEJpqGYBd+A#I-`Y_nu+ptQa zb~&SxhEkl;eTjwwwqju^SLwCVLy4BM=TL%S>vzSJ!cnLF=Cru9=rr-O5k z1$p$r_JNXw#g_jJ`k6Tu!_>F9!7DjJX`NF7RfO500m zNamwP`8d+qjX9M+VQf`3wq4!U4-dwh!&{0H z!-L99k!6w<%Ye+B%UBx67@yE!(P*asMbDsdrxC9q*W78#?1$&A#=HA+||J~Ki(jIfSf|0E%YlW>g?nY$inHPE& za@Y4R+p3#a(`Ykvxwpl-ML@zs97Y03B16nBx*nwvu`R+Xwv*qN zU!F&iAtmxILVZh@~P?rB}^`<{udvQX2U*jA)=zL!qZu z3#B)}d7j~!p`w9xj(@J2p|4@P8tyXixcxWP3#XdQ^tWirg@UFu(`hqD zCiOE^(-?1rIKwu>loO9YHjlGZ$LW*D6Rw4m>W$8Kks_7-3AGejS>+YT1_X6BcuR*h z7Y8&Tm6PI%QRG-lx>#LDeR)$%BdPTh-2h85$3fqGNF`m%aLZClAHU0od@@BW#VGFL ze$ClsbP9nhzmsLtVyE}ZgNN07CD@E#oL_ZdMz1=5>YLK@%x%;oo~FpWNU+nk)3x*c zh28ouOi=O2J1)p?b3 z<#ii@qwf1J1W0yB_xN_4O5UnZHa%MmVlHCm1b2MPs=1m!6++8LOFX=2UsVSt8u=HL zsFX33Jh{jD`Bd>V*0l5$6&1^s)RmwXLXZ1pY5Au-r*{nOJAS@RL5d3ZP}xX29PcqO zb=_v)bFVq|-%G!#Rn|j|PmS-f3q_vb4y2=|L#F4axsShOS)1FMa6Xh@RSz(fXi|1j zy4N0}9x0p`9*v|XQve_;tE)*hu&vbR?hnm&=iJYvR@$nbG}~;~PM+rw7otYkzgcTE zn|j~8(yge=G*w!&I{_TCUdE1_>^0{#Bb_bIS)5lKn{H{(TV_vI-S?^n)=nN&AFLke zj!uq{PP+JOJajJk9^McA6QHaKjCeBK65hVPaD0T$#>W*)%F%c&ymaVJx?OI^49FkU z#~4QkarRZei@)rgh@aaZ-^5{+F_?*ti7`aDMmlzB+}rIYZqCh~w0B8-oOR=_u^WFl zJk{It*m>+$UY^~y^*+E}M0?k~PO_n}TseGqv^&;+rwEjr%eLfq_6U8v)-yX(d0m*c zcH01SzIb}`WODJ+>A)@3SAIra-w2Spxq)&yqM!mNkUpeL3nUY;X(uPfFs~F@C;7&P z2m*#QAh}B*kNG=Z4zr@d(-1y{EJB(f7P>@H-jsCfy&mJ?s{xJVWHU1tQ|rjpP~A{{6dqN1OXL)C(uq7)e-{$1QZN&GM&jAs-aL*)PceLqr?IR9@| zcqJw51!ixOrvGQPKFiSs?Eh2RUp4hXzKr*Qp7HXFxA^b1`m3T}{(I8@RP=wo=>Lq- z|C;0fGnxKR$`heD@_%U~G&)$amKzm|3|zvwIDddZH*)iCoYduo_XoGmv;Jua^o%Hu z9o^rG`6UemZkkvc>5wR)ABqwbv*-(JNUB&)RK=~-e@b=Ez^Blz?SvG9{jHIIOS`WC z8DX!(|LM%&jK2Lls)Akx`t?yzHWVutY0xug^|4?7v!WO9Zz2S!kUFR+MPF>> z3#I-;%MV$>1baG^e|cAk2OPa~P*0Qp$HNYAzDD0hB!5`wVItx$1}zRsb*sB;qhg0! zLdoSVsRr+$-;)lI#O)k~L}>X|^c6=?9kwC{GsmHv(NcV8!-fi%En^W@FZf1Fz%WG# zpENGXANa2ohp+~w288fq9?U(W&{>mAKDgo#MOE`~)Hqal8h%s$*lO`8iM1a5W!o4m zk^+Umtk+7`sF9e;4qCyvL@tZPW>er*p7v)Gow#JGuQO6XZ$HG(nu6rb!UiQ}T%wq+ z^t@Cn{qDV_|5)$9`~{G!Y|0|&OhruGsNmF_jJAF_qEyWGWXw@G%mVodg79 z!CMNhT&bVjmA(1$t(3^M?UlWi&{9@7@gB9&YBelbc9xiX9Q|v_>uKQsvQmEK03UTZ z-ExJtV|AJ3AvLPwuH3O``K zkk1~7`W6TiYw%#P5s(Q*gJ4QsdqaVH%hD&rbhZE0vBS!}GkMaJ#XTIg2mu>v&JfP< zJ?FU1LHR#smVb~ync?^GLc^W(qH_6!7Vy&3KoWYS2We=*Nu(K_WfXs-J|S(o?tA2(Qx28Lq;&0kp9ASbBJA~Q12IOqHc0*nVz$B z$6HbMFHdb$y$;r+gr&lNcZyq}0O3d9IfjoyDDolEXG_-5nWOR3SQIM$c?;oZIJ=34 ztjl#dpz22XpMmG3F!+y@bE2C6Uo!vpc1H3|)Po`im6@5Gob>U2zsc4b2!zA3x!M-- zcz-&m9G8)iIezc@c(VpNYq%X1np{95B0>_VXxKu~cfHy&ZGAmdTt(Hh~W>cua}$;fyGmg zn&0PF*UkW;aeDk%)IqE>9nrym16_RZeml{A-1JA1|M}DRv%ME3Wo5lf^~N1h(R|O) zw41HqLrKP%e35>|SIp+VP+|SatcR}xn_Lk{kau@?9M0z%FM@sq)EY>bh{3`zv)a$o z$&5~?s?Fmbyk4&^(it2mXynrJz0V*S#M8Tivfas%L>>||G+)!@bk&bdd0K5ze$>y= z{C=?{+k^Ukxq!Q%GgaWrj^^>k_moz(fv5n5fTr8}g#&et9bA=JU0FG(>+Q_?n8Wj# zWq4%drcG(Lau+dRB>(Ka4O%uyhR}z`4AOU`(4*LX@9X%_4HdK_RBU2 zS4CUx_1EuCJpSx_sR0yOTG6INc;vx5v-qoZK%6Iv`TQ+$O#F>HIL<&P)%dzAmuX39 z*e^IV(S-UJ1cKGwB{ku<{l~Bo(d{tK>zqjXE0iCzJ+!C1pgb|i(WX6ngKl>GWmF6Y zY(e2z{C^JBkchmtNLi8ydg8mDNs2Z;pfSY)aN|qOq3&q7Pq5n^O1>U2qE3Yqbo-Zg zMTY!b^)dR!UjrAsZf)@ss=;wlhc+S{}psFWZMnkni3a^kz+;C7{;KVzwaDcG9EYx8+#te3dV(l5zn52 zBoKshzJPVU_MJ1z>`43$N<-GK7V`j?XgUiLJ&i{AbIq(=P^sqk>nfbFYz_ErDx&YI zC#0nj8YA{8#Y7^IwI?hCkJF7_k;_c3LS*s^9zGHPRnLm37zBbv-yWQiAtb3UqU#!pmhkTK`=4~9(!s}6tgLfAPKG&m-GkYkanNzVOBbsUb8oZG{D~Mz$ zRyD>9igu~Q+mtM#qk=OjaLQg;F%+}Fa3k29A&JEDjvVz?y5`rnzC0M2!7%)=xj-eW zRrHRt*JSAl`iLf-h55D`?LS{^TM7pOY-5@GE~2D)^SlA|h5EJcVr=j&M$q&g8WG=c zLu0Gny*xmZB5fD%8lEUw)KLbIGX(F5f}SrU;X?3n_V6x_8RF#Uff!a0)$TkK+j*w6 z<5&xNJE!9iKU6Y#?bv`+7AvZ>6HGl8c~{Nj5kl%j6-$MSO?Eg@#y`*)a+=?vjc_4{ za`y+T58q)%5$VM2E)5E%A4v--wu5r#Ty*Ck7fZHW4G~m%h8C{AM;$<*UySE6L47}K zxaB=eQ3>9l_#&V^gIx!oA%$u}3hk>!B+xhv@bcC_vg zM+OHc-*1cgO>dKWJ|sztkuUv0-X6|aRK!fnuF(clxJ1g&feDawEAV3(YRjgQi8-CB z%s*GIsgHruXe^`{QhN{#kqseBh+RIJv$e!Og%FAYy4JqZzC*zv&6~k0`0hYDH{>?m< z66Fiue|4!RoKzKRjWpV*KQWp^K*rolGbd(Bkhuscn&Q4$QcjH~@XxGoN|>~pQFKaP z`6Ov=6K!EnxY1=c7juuCj(jkmvrT1edKImlG}K9#bFIwC{HBvA&wF2HrgN~Il8op+ z>@<(Me0z>__`}sS+da(LUj)>XT|07V;LKv#S2jymOVkb6cy7(Vh-Yf3K8kRK9Bydt z&!0-1WCmZ<*lAeM61KDZo|l)GFx1^>xiYKQ-eA5!UCCn3>wS~Hp)-<1r&?zKQzV^A zOpe3L{A~?D&0v+m?My#kq2^yvLEHYcpKg}BJHpTvv0~kY_Zqa-w((X~H>TBWL*#W| zpWS2CG?G_sg$F@Z@dCqfnU27%Go~4ISsz?W{2epBiLBfaz2aAR)I>KG)SP76pT|M@ z3Tkjz6*)13Ye5gU`>$*RqOCJkiyk1XL6AuwRurd2(yOdn-(&W`k;}xmOgx1b3tr@v z6~Jzwqg?@t{zUeV(mX#CMA@A-V>-sMx4Ls9WJsOgzRK}r!7VNjb+u2$m*z@nE@;nA zRD}%pdKNSQw?m5?V-S-idAgs)cv-TjNf9VaN$Syf(ln!Bvm+_46?6)naMKG9%-$aK zc&UD)DIf<6(Ym-GeN}VsS(0|STS!fbK;VMMIzmWgis-PVKQfN&Va}?U=<-|sP~Nh_ zX_K189hV|TvjrBZO-!nWz75f=?qK0{AyAy$Huo#Opx|YOR7Cu6_Snwq>c0 zK0Gh>A2mVEXRs}b6rGQ?6V8vYoZhHoV`KGqpI$_d2w|F?-q!CF3kMm4`NP?gvV8&R;gU*`X2r9 zh;%l<|6c2Kd-yXF6A3XwR-C3S5`->bR?Fi`C5X zZa^yVu^&OO(r(2dqbP!J`ZNltoqnwCS6Zmr$6RWd-saKdHkTx+bo5sSBozONEt)}3 zuG+)usC@GSHq;cDk{KETBs#0;SlSwT=K!#Yi7^w;qB1m+BnWh#Y-*~C=><#Wya18> z`Kj_NS<4Fb6F_41u=q}YoG7R7h?K5n6SKVAymb1El!o)6Mmexvd{E^f>=!rBvXwiV zdAS7Iv1t@h^@+lQgWt2JynN0&+coaMt4F%BfiWvk5iL!#Y>um;NTLxov4w8TA^pv< zk(Zt~`gpVMt}jWu;8#8I8g{mVe9_#(f?31d9KbP9b0k8)vPvY9K*i*noK>=vepZ}H zLEYS&k^=5aH&%;0qe7uhcMxHL{)2_#gJssXUn{h1rZEKf>O z-+hCfnTW%sh6N@RNJp+0H6kKl0Gt3}Nl8ijjs>KJCQsw=4TTT@fKY~!QzK2N*D1|1 zWqEF1$g#u%yftp-9jggMDiuJ|3FXbF35!z3Ev;?*UIF3v3ou4WNePF-@V$H=ex+#LM(_xPFA*hDRnCE_F6X zo~q1hHqNj!vPA1i-KGwp9SI}Wy8BZ`-lb>spi^y##|$Kv>{+7#yOC8HwkYEfC)>Pf z3oE%+r+HJOc)c14)3Tb>JHkwVIt=GV%!GM2)~s%SnnyC z)qMQ@a+Z@-cJ%GuYB2yISdRLF=u|Kudgdal-}siR_S^+iR9f2yLCk0IH-f6$4RS%1 zx?gC0lWWKLUaz0uNlBLDAT?v>dVWZ2Q;c}HtE@Jka^?Iyjnz{V%tghDPZQtOUZB}X z``Zh%yUtRxZT|N!x$Iw$N5f^7&_h&qq>#bhQFXjeZ!v#woC5hS(VeVnNN!?VAlV%; zS9$vCuW1azBvMgccX+(n(+3wIA@1CG;5UXCM8l52_43H|DO(mZnym<^ac-UOtPLuO znNJ6hb2G$Q`nF8^fro~RX>aIsH&OzEg5;-dDqC9>@n=mNKMOGvS<5!6BzfA>l2i5w z&3-VZq6O&nkh3ie_S9PgKWLuztDQ@)*~bWqrIAH{TnP#5^f5!|e2LX1sm|@9n4FiB z+uO1XA!AOKJVs5oO__15r$aSUjZ+xtd`#8IP*ME>hgUg!U6SC>L9nlTlvomp^vfaI zB6Q90#PPIG+8D)xD^6M77xH_2X<>e6EBXV=)1;+$lO6fun)GGZk|dS%&vy?=hUh;+wrpT8we_R!CcSKifewy3!XT(ANW;vWx>*JlR-ByB&oB$?P9Uv z9N@y5(+UKEA;(P;d;41K@s+_aDPGy<8B$M8iuT0(r*sitO?7{@9JX|DL}ZLP!>2*z zz7>O{N!0B}Zu)gD8n3NxrNy2?d}T3_7*J@*bN5Y;4Wlb}8P_^xjOlfaM zYP#3)aqee6hFm8wGeItAEzLDeGIg6*pH@CC@W)WlYhp8K!ui z(3Mu=zAl=Gt`y5c?9B(O<}Jw6wKlB3w#|a!*ooGc-mg0ZU)oisUpt9wbo(GZPSiIP z<~AH=WOADHhp{@U&q}G5rSeQ*&+o1`b^C^3qZy}OuU>}xdJc7>E$g*9zg$e&-+Ynd z4ot1nD~Me2O(}elKj4U> zl8?HKQcfh53#;G;Y0ju!kiQbmXia7auJ&euF7O#=u`xC{1WsiqElJZ^WpZ3Z9NcUC ze!UcPR4uEIze^KZ3%UKe_7dm22e^>908%lD#VpJ3O1a`TitY(6%V-KL-Pfsui7NS! zg$?G{s2N4L6>WFXjYL^~_WMKWw|!&yZ>AqfEiPUvhhBx~Z|w zk1lPQYi-W-%LHDa5H()rOj;XrmnUhXDk_POQo;CDI?|c3i-y#5H@4b<1HC{K(&AXg zhzE4I&K0$<*x$eD68-+3p&AsZ-^iMU@8>gv(U57F=t68$D*ver$d4o1?|B|b)*GF8 zZBA#`ig0Un`$p}x&fIGeeG~JeH(}q}E{P1{aj86{%ODWPj7@9BE2gV1dkM8(e->Lg z*O-VK?((zoyxbW4Q0Uy;ltNDT5ZH4RajqaBqguUG=P`T#UN9q+j!lv|-BONh$7bji z<27p0cA2F9?woUp{akJKMsF0zzO(CV8B)Y%^TH6S;rlkQJWa`2!_SBLg0SqX_=1Ll z-Ie{WYqtt(u^u0YF>XgJVefEz2s&}FXz9cD z_1b`-;YS(XAh-+u)p*diu<-DnUe>mS3=Ui3`D*QI%%SVN!^SUk`zvv}#d?j#$c(nKuVI+kMytWDbNTlxLF15id22aDlt`^!Fhg5(qM_4E%E5&f^i%%G7zb^N(L7&6x^lKD8pO1lEkFhQSgK6DC zAbZ|ssw1$M0gq;1rf(4jjnQ-(oJ&k+DUIWKOf67InV<=pRm8Owt|QoLAfPtExR=y1 zs?(beCeq1(fGacyMmw0UCm`b&IjIqz=#$#{gM`7KjVQ-JQsVx#Ew6lKmm=_iD1red zvwW{_g%f-I*<#LcqGRp}!;ap!M}Dkb#*$inS36bLM;^D7QOP$99W=m0W5xsO-~49$ zmd-qyoRJ4Hrp_dhqz8^a&23_yZ5m%G6f?*+?x7w`>d%R=9>x1pKJeNYIfYn$I3Oe|!1zdu=7Ralu1 zgzLkbGcK3(8P;PJFNK`}7bz55GW61cZp8?MZ($~tSO~7CgiDPvH<&Ba37Khi*bd8I z-5A_#>B^>nWyHL6o`t~X1cxzY`HDnrOvDDWl9{ug8d)XQ|w zu7J>v#TJ_&4gFFP>Q=a77DJn}cBf?)JG^J&EQdO2QY~35rK2Y@fQ0Y3?}$tjMjMuN z>a+v5=o)<$YrXQ>c^|H;tLt)Q)B18ST}&Ai7^v3b^ZUz%xpl`~5+EZfK0cpLt6eoe zQ*OkzRcZYOS3d{vP6cxovO)1;9=$R4OLHhf;x%=56IE)>j1;C=%@5Y(BXTKJ~ z=^@ZdU;KS)ZL7s=BH_)_KWXIrW)LB}{O+8`LF`YeX4RdUsjp^6?!+ozebP}`_H>iA z{)nN=igepaZlE3Mu_A7}=FZW57u{;t&lL&B={$sI}f0? zoB_p1tR=FOu5!1iI*sGvrKPN6RJu^C*w*8{EBmO1k^Qv}#u>FPD{eVM*2L1zmSx4C zF;!kCnucm|*T8x++wzBRJIYEJ#^p@*g@)<}JPk35*7}ueMVPH_+opV~Gc0YoRmvEK z=C>aErretLX$q{1UQx;BT9Yo=Wz)EG+YC)($(!a}+QY}zrXD1d8vweqBSo#3OcK7_HN>aJW^oCG9U zG=?P811Jscg50LU+_E1dGICS2K?Ir;(aIzDY$mSb-a75lg}wb?EDS<+yey0GNC>}L zy%!2RB$t91EmemO)ljOF^&J_Sr?7q;b&A|ICQSNKV-|7i?W>{B)xC9mEU`AZbs3e` zRzqKcDQTrhF3X%Xb-h)J7V+bBTq$%O?Ck6$#(1n)HkkqiWBS*1Og_(r%F3&z#dj)V zM`X^b+Rha^$A%#CTdmQ#4lARtz2E1my?)g%Spe+q>yM4fy^m!Oc6++jM;%~Xhc}o# zjn#ZE9*MLm#M#;t=35vet>)s3VJ0Z7hY`$T)#H@6q>RZW{`cb`H!t=Z^+S*e3!Z4>p|^i^%D+G0plL?tFG>Qx0NrqL=TD(B@Riqm$dl z=Od3}xhp&6&StmGTb=42PF6G{(0G`u9J>nVHYOq+Fcn3NwQlR?N37zk*tl)V zfMCKD*GmkuZUHz&g;tjuH>lmYOF_JpaO5|F9f-H*bdlHfAo>zLA^#n zQ3^MPD@{a_W~Ft&kQ~?;)CH{{C#v zu>r~~x{1&>z zjYLkMa-0;rsdeD!(@(ip=6znf@G6$(99ObT zv}|7a4XJHx<9lj76q0Z%fnbZLQHM~?WV%Q>k=K~CBt*U{(KyboYvKJWCw*U^fA}>f zxAN+?mV|C#V!*Dv31uqDRU|l9LPgMZ=E>yX^%joz%q?uP;afz41V@7E{ z3`Ah(x8v1puHgByqi(zY-s0YPin>O&Uvpe|(sVdXSsEPu7S6Ob^QrqNLXG31Q*D_B zcHEmV0TF4F#~)(d&RjlRhfhd&0-14Dw}_xl-UI7ASzgbk9-m?JMUtd3{R;ev85%7b z7D<~9^LeK&{IT2zX!OKZAU|eBij>^BuJEj0?A`;p?rZo^Gtn&xk7Zlr+p{!xSUgw!m_gA6UKFOIRSijs|*0kFxYpv0S_a=q_{wXn?#;b93!?yX%y@P$6o@Jyrne z3s{&`5pm7=tnxO(QLR_x(%gku?182=^;^;37eahgQ+i1gw&`W0dr3NP7G7khgcHT| z$X$$Cvy?y}d%OuLt}CxYq5y zBH@@@D;w`@Cc2YY)AFob$D1akXpoJpBpb<=fuhSSMWH+nL7WIUcog-Fvk6GzXy^sar{!+xIzMU;jw9{5>8s2G7Kx>5a-dDhnhW#>8G%Mn{b8aGPi^zd9uiw_yF z7d0o-ge9xghTrs{OO9-?;~p~MPmp~0!bk@1Uv&lxkq^bd!Tx2l^QW17DWy8|q*}C! zTTQ3(Ed>J6clY6)S8ao*?L1LlM6{PUqKtJ+W~=2jyus>MlDa(UB&Qv}_ZAVIODK-iktKPkz6u zFV)~gJ9QE*X{|Ze-yvbWepP-y3J+qtzsw{0Siu_Pbme&(`#s@x&&)c`vu@Gavoq0q zQ>s`z+rIfUl|{Jr;$iagY#F-fu}K+qD#vC4=Ae4!;q=;V)X;+TffgdO>Th=)p$DG` z!kvh+{>VDCY!fcG#A5Fkd32`+4V>jww zBkTYs#HjTE8_EFTokw??9L~(Rj(x*JF>}N7LGv%aJ2p0!F}4_fA(ho!qch-L!e`8V zJc}2uZLJFNzOzS6L@nclfKyJc=qaus{fgE(oF!UTnCCBN8^ z{!elBg^(FyC8T3e!L3CE=MC!oSCS@@LMY?A^Mk=nJ%~d7VEz3TO z5}sOgt-(u`H1l|2{{3%2v*nDb`_W<>K}aq-E=j+P__L;3!1nrmg){On0!?SL?f!Js=XzE^Zin9=hPd(YCd1fdVjJQk$7Tr9A)q|()na5F}`iEo(UnN|1``0Ek2OzDG(#KpQy z?XjagLfou3htwwm(i?3z6v!}=*#QC*$k4kN`&epRbIicPSwUeR2nNjFN5-HDz87g2 zOZu%yODnZMi@qaEe{cZdH7<~(Zp69T)t-tVf-eAA^k3>Y9)v|I~OA(zs7D3>|ICaU}!$~-QOnCYs8 zg@vI4LSecvOiWDV)0Jwnb>{bRCpkSs!_nJC}4uH*~ zji?jNfEbI3K_-i+7dYoq?KYl(J{3-1(c!Uo^YZ+&PvKFcyy|%NJi4sImT7WfVqpRP zqO~6kvD4Dhvb0tOhecmfOZS|aIGTyYmGqlD~#qm=*PseZmcd?1uC zm7a8NkYT95=FRheJWc3C@~!MOQAj)I2wOGX2%_o5POSVc?$@z3 zp=V=A4=;K^)c-YVhMGV}sl}wY*TI_97^&R=DitrbQwp`eIkVlNS~J{YUR-}$HEXS% z>{n5K(HCIaq`M`Z-{90HlL-AcO?9aIc|iQYs6(X%4efhQ(QW;?z+YeO3?yE9T<=O4 zLjAR=!|>p_9wgbctaa@%d+~LiC|Fcx!!VeupvhJ@a8%=l(%qV0h~f93(_^Czwv1;^ zO>JzLko{v(Om)X%j`Mv3sPA}8{iDzh2W|SM&emXv+BoR`bBp{L^jp6=lbd#^=-=7l ze$oxZ(%dk^5`T5$o` zZz_c0Lell=TtE|kHm`dlSG~8b)5C*^^8)-gv+4LM)XTl5(1Xm;+N!w3TcI3Fu-D8T zh|Nq0?auVf#3}6^k(Bdll9`DWj6v-%uPCo>mT2}_PPbdib(b9k$IE>#i-pnjzM4xO zekcT;4dH)aIOSRqYSO@by3j|+0WB|BnXTaM+FqhD@7v}f>2z&uo|Df5(ch%Kr z^+*MW*+#7{|6uR}L8^LZBNny&l4kqbP5;fUt5yVWc3#*1s_*g1C3zdEogb55qf7Nk z-8r{}(+ecEI$B_RTjzB735T)WkfwAXTkALNiGs)N^-YPD+85WWREGm$Rm`OTUTpV{ z?WMkn8j1Rv$MwKUjwi0HDyLyJXoo}}lcF|B>1mqJ;S)e5yvhmcRX9*}_&<@YPLjG+572ami=iYE{d?204{?(R`R2&$(cgqs9Fna&?>5PzwVrz} z%8R~m?P~;vsdDWRY4R zg()|z#*M#|SdF7oALvIk@%UDT7jKE%kzy81&kO!5jzaC%n*TP^d&pT{`U7sd?xL>& zYb3VO4ed#ey+_#=KAfC|=&$e`OQFR!DAQyp`z7!S^K{t_>%QEDkiH3BZG@-m%K$Sm zO2au(1|44b9eUrxW)^!R(_*S$e$RSiKf2OgF&qI!FgS5xhmD%`fVzgV zI*l?kfRmcPvh_rPdA*VKcXW02ygxsALPr6qTV9TWVUoDUg&c~l%I1T4tV^tLd$k73 zkjJ?sL-5cnx!|8DV3rf zW*0~-radW1K92TL=pCsiXLSa@@uA6~7fwauaTrwFi*ndd`a8i%J~L&x@L<(c3wycd zQz(9U;wzT-{>k4)Obz$o+xXUlCTB^6N1y28!!kQQ=~9MV>X4b5xjA)er}1dP6PpZ1 zV`*Suk;JTCJ|nUWK9msS$APZb)92njxC}RL*^hl`W&fTsbgvIXeVn<(`COowbvuCVO&#B|AE|1V?8X}nsBxO7$DI1|rC$NTluhN_36S-lG70OU6fElRbVLxDqv%`%*w zYJ-~`-&aFQn0GSz%JDt@aKB$NZh#{2kk8lrGFNN2Cuw=H-m&}_`NN>mSq(9`dYzb1 zU>@$na8umDB7@;7nGTRHn|Vf^PdQ$mTP1E&24obQ8_Sp$G+0eahF=3YrWw<>ryE}L zni{lY8ZnbtDhn#4X@=xfT*s^`Z5on<49Rv?O3ASbV}p|T(rGJ7Wu{N(@DVDcm^Jm3 zHtgt&+2$)9D~DAK3vBfz@-w>k3Tqlyuqk=Xi^h(~s<<7h=BCwXvSO;wG0L4>fNS>_ z^UCT#R5^QiSIy*BBb?a_KR+-!e*O*DMIbSEd2VLf$*DqRB4>GJC8IkWQehGW>aBy3}aGZk^lR9l{wi&RN{tEpL+ zq(v5*D__oZ!0B{R`oG~HC+Q!-6Z*W)B~Cm|=wPBwLwo zm4XUBEn@pa98O`q>4p|6*;(Slj~4J1XfuTEXQs zb9S#af?4AsoQoOH(6=pNT62cL+^CW_dPnoe-W66Hb>|fuwV>{jV;gbhz^trN+j{!1 zENt$1*OpXdm9*2U3yrZ%=46@E(fws46$54N!d*`)W>$?@FkEfXlLev~-AruK%4(1= z+yR+r*lp!cMafEq=}V-0_9qVZIDLKcrffIny5l<*!rECb79KF2MK^^N?@6MgJSGlg z(7ruFIb~(Aa{O;B=N%7P5#Kmbqj~RB+$vMEWW%$~%ysLRE(b}XeVUGw2*DvSKmnso z_WpRO(W9)*rS+Tp*jan7M2gbG5t{@DON+Uo9+vX*M(DbavR+II(iF&Otk|D>E+*SG zJ|&jJ>fMb&fQZpzG?8DX?qs=Va6a}-Ub3$Fy&`aw0TcovwdO)AFt}dU&acrwKWvnF zu=j@=o6`2u+$$B-PZGZ)?r&ib>5Cd9P!pY7-_cAoN7K933$3iu_+>pW#Qqmjz=85#Z#MBwe&5XUL5%{{O|)RR&buJYA(j zLJ^P#L6Gk5?(PQZmhLWTk?!v9ZUO1;7P)kHzrU-Gf8P(>56WFOW_M@ioH-k>6S8afT=QYDkrd&aveAy>A+ej=I z1xmHCA}aI8$VbClOk6RXVIod)i4Cx-HL*bHm~$Stloz)g;h9>vTC=2GdofXFDHihT zV$`9jdo-BuBxzZH$Zd_ARnm|3+IMzq<`*j5%A)y`p^)Jl5IVy#8Rd+6Zy}9>&Iyh#&5i5(I(g#q0TmrjD$9Q`(Sh z^2WI#|6n9CN-n1#BLp3?I@CTBUDUl;u&b10*AB7-L>m<^XN-Eqa2g=#C9ktaZE-c7 zUNj9E?@C+L_AoHLU5Zp}(h7IQ&d%B7ueh)UJwThB2ftTK7*5T=vHg4xBNLvIACz;H zD80Z^fEkxvoDE@Qgzix9KaBD<`BY+wWspP*Wce|G{R08Ia%@4iR2t&sz&pchfWZN? z3TALPzWuaap^0$GTxqrRVkxl1R(hzMQ*-`^oCJc`#Qj(a5kxiivfP5z2b(1ueO{`(-o7Vm*TRRkPLPLJ#80MD4_*yM%xJK zn@)+k*p{I0i-jdCIKZa1@ogl%!R3_6&X%M`oIHYw5}qEO2-3J5l0V1Kvt#WPKtXR0 z%8EUa8n9rKLFH2RkWpUtN!rxPt3_q5_^=tP9&k#wRE=U(L2ZIutOUsvdMH`ejj7IV5CE;_MYK4 zSe3waf!Ur5uf{h?-vM8$-J+5c%v^1BSzm*Mj67=^=H~ePBbvDQA>Lj}w-Bc6$EN&< z6{A(e6lTWmmPXwJ!wPb9p0q{TN)z9Na>qdNg2dXK7+Fa#JLY;5$DESIoWk?#{+Ibo10%`}p5d)k93e^&71)8l`PA-$?##V(;>v6jUPyD70`?}S% z;G;senjwq^m72UN>B{IF^&y8D*!r8b$YRElYTu+{wW>R zTG*I1th_VFJWoAV*Z(7CF4P8!s0z%%R|Z9s+P7eFhS{*@2thHu)CC37nn7THa~P$C z;B-C`Qc!rqbAP~C@c@t{u)w8h4?snO@TX7Np%%>j$vErw~i?1B+>>#TA`iqpOnT) ziP1PH=M$?~%W?1uZ7ZV%QIq5X8Wx^bO@nkDYw6Sb>|DaJO~bp>45k@I*}VE1@mok| zxjOs=dr&9WK)xtx)|Z~2S z1~`$d+71tsZU!kUz9X?N`vB+(G8&nTWrs395s6sKt9Pjb$&ICqjh$hq^P~;J4(@?f zU<~*j{N!QaS9s@dW#`g#YTC64wb#eN0s3$8AQ@_z(}N;waftE@6>zJLlT3m`@;zFT}My`zX zT?wD-!{`p@rlY0OJjpr22x6fIm}lyzr(7I1vtSNvEHA!u_FyPPN`eIDw_5W1X*&W z*S?AQfm?Mn&Ck_zEyz_%j!!eDPT~ExPHfHQ`>FqPn5XM4n>_R@JZHR5@w{JxF+@Ig zwN{o1QD4dI_cWCET=|}^Fid?e4Z%TdHy0I zw$ZnrwD)MP9}Te9L}HDYN3}c|+wWJ@gf`$*`V%YJ_8SLh9FFiT=WF4EW#dam)LaDP z&(5Y?8gj8)k%Tl)3lLfL2K;s#?U&>QN9@R<+@km#9qZ1jyTTyNG~LJRuUwAXuD@nv zL>UdmJBoijYEzMv{%xx^5ZOsrAfgPI_b)}^Sq@;%OqD%OHll0A=j8(Ic#~5mq!6R; z48OH3z4h%013`)l&o{)egt67UFyn2(-6W+AF5^O$S3Mzxq!E4IQ+UpC+Q%-kc`>dP zxjj1(i6jo~VrEf`-!UAzE9h5ccu6!pj?qp;(T26VpkCV)kcGRQmoswXKqa1vG!*2a zRGtwGUsf}KM>I|h1y11|EY?Xbu%26|QgL~2QR?nSH|NUzo^)9>)DZ_+wD>dkRKD9| zr*7-P@7ja%l<)NOLD$~&J7%FDFy z<_S6=Qmzg40~vW3cpyci@J~DTW#MSAD>NcqGlX2Av%GEuOx>(R%!&|+d7|<;UvcSHuna&N!_m{hM2%Ew}HR8 zS!=byA+JO9y@H?Cd;W$?@~eRV`pclV6j4GiBt-bIGqRx0-6g`B>-Tn`zdRecsyVFTI#d z;8j*y-;vGAcBS`jRt{poYF6RR&9?a}(_<_hrR~ zqLX|T9pAS^CYxE4c?w9Z)w>QpKhLEnk4hn@ojDnaq&Wrd7$*mTBcFskA zGHXSXt^}sEIHJ=>f1efWIqQx`rwA6AiQg567}!@XJ%&#gY@dqw;BJJ$p!g>ce-5So z`WqKw3-qsN@7^yb&$x9mEp84F-$;DbNgO6H2cniw&cBrWkuKHLCdHVz9W;_mwYXq~ zq;L)c_*;r5W`3ZKMIv<(Syu3uw~X%R6BW_Nn8kc~{f@ssb%y;dBUCLsPV3V2tA8j8VYLosL^4yH>!3x3U-@d4?w1}1JV$be z*}*WeOs3{3&UblJW3;`T%vY)3!~H_tn5O5l(}$&t4_re>>JJP?AMD;htO#b-*JA)c zCY{(<6|XGWRJND(5ymwi;lYQIFCQ^jy(D5sZ!c<-t^VMJUBbv->m0uzeb6uCC9{MG zCX=Eb0=q-L3nBl`h4zW6p$(Dw8yqFiCL<*4uDYyZDA#&edT8p!Y13dM(~tFdWI2W` z#SF574h~#m`w@)WwCjr%B2&^h#oA|;6j+62#Y(DLCicBHddWP^61}BB`{cq*42=Un2fxrOrq1TeBi&hX$nx;RoIiX(9b<)AeM)ak4bd7aq(`in z<|V$iFd|;gG~Qzrbkhe}gV(O~wyD23oiv!(`%x=&jVB?7jP}0G*TM;kh$z%MvS~Dl z#7(`|;6Zqq0r&89-JzONTB_5AP1}C+-fti#^0!h$ryddEQV2?X^byF*sh--@ z-U?xN_KG(k|86>pWhV8%f{pt5=5FUad~>^NW&o{S$y!LZE|=2 znA_e?^7-mOz4h^G3BU(?ehtKQUH>V*FN%c4ci~H0&*fQ-aP3JQhp^MsBEqlWQe z2qv56&%1et$GcMt3E4F_JUjJD3n*|m477j*21+x7X>w-^mIf6@UFTPBCoR_x>cCZKa{(}x1Nu3N_>5~YUqdn3g2ISIqWogayZ8Q zItg}y|5)4#v*evTJ0`7;Quq0>*EMJB-CoH@N=nEj*6Net{)2^@*Ka242@G^wUjNWV z(8UsKPA4L?FEMF!s#Chcdz?3jA|fUhZ0g)Me`=Qia{LuPB-;cs4SwU!`puhF1awt- zd3kypNhq@dHNi+un!T_~Ts3RkN3VPz@^)lo6jbKme=P5D)EtezCLoC7eb_)TXBA|* z#TMfSjqzbqo9*kp2T#Z;h@Q`++TUq)Xl%?GaU6)lBtCoHYy_V!+mGqbSL#F7noL>( zCU%z$Ca}=P#>QT_-|w`8NG^Y~UJ?OB6Ly@dFzYbiiUsHC`1whe8aF&LvJZ%|vI2I< z74t95Wg%|iYgG9qjLP7CESm=kSv&ap3}?`Ee#;fe#QbmFbyCHilr8}hoi_X!aHeE0 z8hzgA4!i-ec7(JXx8ta4JI3(}J4vPkX?F)iLh|o65DX0uB4k)0KEnh_Fanw0Z~n~s zM}~-&7UlWA(4zBwk;Z?8yv1qCaMK|2e*z00i(Pft40r5J(v@_SoiJ>*Bs%_FrFL z=do1ZRtERwd{r_u9gsUJA=`!VT43;o@D9%lhK%7O^kxMBeSda!9i7O71IxU?cEEhV zC=iuyJFW+}oB)`+F!&D2oYTnq^56%DImHGXR&bTR@9wZTNOo+LfLt5yMiT5g9UfyI z(ozr1(Y(A`8U&~fYaWc)*w`lkV-o{RB&xDrX@2=ZtYL{i1kCpIMlTTA_p>_fSpe~y_s7MY(3vC7Y|Oy0%E21qia8vr+goKzOSPxtu8~C&TMhT_AkHF6!w=u@uQYZ$V9Igx&I(LZyg0 z>JK%Y;|dY_)W<*FxY=~zLUPQ0)Sg3wJyc~S%!M3Z6Q49`8QEVC+5h=K4<m{eywb-dgW43Qetd+Qn7UOZj;GfFh=DoMu9q&vE!Iwl{rR@@DR0QUANjvXFGo4 zxrrWQ!YtxT-QNZb22EOsQ+d7d&CJYx3=bTH`-+#x9O5?N>Y6t~P44 zw|<{byWsQ8b3$!MjUPcd&jP91xevRvdqNhFv)mQlTD|t^l%yRes<$Tplh!cl+xB~? z+43;~hja1oFe#{Qw`QdgC!@PG0yMupBY{jwD2U56iv&WC=4ep8$O@oH zd6g`yrmlD_xLV(=ufq~TFXa5Ir3 z+Mq5Q%sS?Xy7b*P@MD?z_X9JLnlNK6C5m5RhbGoHWTUkN`;qd!jl2WDd%y?hzzRS) zEqt-hFQ3<8c{@ug+Cu?8)bHW~0c&vzU{`cNsVNTo(GpOhwv=m|%yH|qbS@euj1S*- z_w>*M>R2CeTSloqo;wN&1#r}4T5^F?r`{=xc^+BQ$jD3A?J)bZfz~NIpuwDZ;VS~r zg=Qy*#Rr89&I&I7##bVPh&lnplzzR;I&($h$l$|+Zwb&WvrdsLX#winZUC<{ z3QW}p?!Csn;u4L#XdK9>&VNj!SFL?0;~1Ejn443wyvh*t8JhUZ)Lu(Hes-RoZIkbU z0|Gu~&3|}zzTmR_xd+KQD=84@KbPe^-Bx}%tDGa4ba$5tU-d(lfrykDCQQ)2%5_1D zsm=La$tRVVOiTXL&?JiEvMLq585k;Q{D=blM1R?^>rY)28jYrKJTq&#N*gXuNv{Hu zYy^h0T_x05-x4Q)V>y7`CJa#SoFT%QwW)*%IGv;p)!6LwpT^2!EZ=aW!ZD=_?ZM!D zI=F`suz<{uXtjOa6sgorn{lhRaSe!Ysm{^1o+AJFrkm#z4SL zhb59qeQS7J=TR=DmJRKtzGC#(Pk}~LFmI3Oa=;=(ElMj#LUzDO0ghd?i9W;M!yHHi zb;ZUro0f(|hJ?uDa|XS_Vn}RW)7N!Hi;T>dF*(J3 z=p#g59^%f9!)IW@?@OQ3OVxmi3PI=VqZPGU)7M_NL%N^l%C%AXoAKlzuj(N?7AhfU zmyMbUrCSCL0jkEa)^?qww!w!hvWn0B!u>Wq0~{;;-9?WfYE@Re3_pp}P64I{SBqg2 zuPE`pSYGzDi#cz04q9|v^bQTB?brqeCkW4>MZi9mKqT8;wN>(KqT_e0?arqg^!Zj3 z6csk-<`xo$*P<32XZLh^u;$}TTpump_C?}_RUN`jzIt!@wpS7Rb?#CI#=}^0pc#9B z@38L^Gc1JiSCtMCrUjCBa9}+=FkgVhpzl=#iw^zwA9>L)r13ZVRp!!q8wSURSd%Hv zb|S*nDOGSfXE>ZIa_}}n4>Zg|q=^xgk&IXKM(b))#fOGzXC*?FOUH0TK2j_N9jPl z!?62kT}Rq5ATcsQL!sY9#%W=#G8$pqUf>Ffs#!A_MsP5WXuh}|r8lc4`@p>t zld9G#pq=}ezOhj0&}{D^lT1F{{x&L!U_O?pE9P4~NWG?VN6qjj46)Iml1JM2EgUYa zU2=TJgINPFESF&;2)B}9Px&OKm)hb8!7m`-2-himLy6z3%DZrZ2E4~QeIdC4(xyKd zE9;z5U4@3DEXGOdfy+!e=y^Kj&4|L-=B$ep*TKM&KBR>!$dW?4^!9Tf_E>5ldAcWO zJV*Z6^d>LAy-Gh5KS;aK_4X;&;39~y+kFuz@~F>D(ZSK%T4 zv8w3$6@*D{;}DcF+!`(i`LEUqzBC+$&(gZ4{>SfR7dNuE@^V1Aak#^3rhaz@2`Mz} z*lYw8_PR=>((BV-z7hHWqiEFj(Z>gQHyx_zrQ1b?hE6tZr|%ErjacFd(_EgdbxU!x zQ){0eH+N$tXnZ!UC^8Puc8jG+eZA`Vm2~w@*@@rrNYzK}xfTZ{M}JV>ZYtU6`;_=_ z8}61NVbDTlq`A&3RW4UQiMc`3EEOID?X>>!){+Tyj0&0URErG5{aky`Nu69o%P(>=eKMzZwP9Z9R}oies`15hff+FRSHL)lQ|(x)d@gp+(1bt=7Y%| zkV1OTLA??RenG--WewI#1rkcFTFxXT5?idIgZj8X;219nGp{~Po5*iJK$wJpCZ|6X zu=`polRk4>p9TLcMM!lWZ6EKNEUnw_DdSl?5bYBqfb3R<0$vt+31zO`dD&H~J-q?c z(M`mXD;o0F%sBgMwVka?^-{G}8Nx)JeBn>FF6K7>?qD^{FeGYXrQ~5)`mF;&HKlxx zeBzZ(`m}|gip{{Jj{U*YYe=b9E)9Rlc%O2Y0TW6MrWTv%vSX71lV(%rB%zgx6uqm_ z*-G-T*rYbqC54R1N}Q3_Zexum9VsIa`X=5>aG!iRFySr+G%eol0Fi0))7}0|i?a>) z)6FKc8#M#UE8p&exwkDkizf=I84z`<59M$1-ZAh!a$?d3yk^!0YRonCjbHRnJDS}Y z3(@voNKkN)+0}k?9##y%@~avd{fV<6Vl*4W*yX%XR)(Eoi=QQf;(~UrbG7r#M9tG8 zujmSE9pH#vu8jkttoJT$b`4{)&T``B_#XRF6-gSSoqIE)XC!HB)I{q_!$T05K=3R+ zBQ(0Do3*?f!qoNH2#>y^wRjvmG_-%2ao~~$j}Ef%mhsi7r|NKL+ki)0V;F%w2$--k8r;*fW4VT>qypK8 z54!+~bqdT`tByjUK{Z7+hS=SNG_o1#8D?-I z_(t=@{%B&+=_Y#KBJ?s!Q4|#-yd*)R^J?< zvDoAe2n`+cyNJJ*gS2~ge^0{ik4PU2xBNr9jWiC&$ztV_=^7K+H5LfNDr~L$!rLzM^WeBn$!atp=qgqccIt66haw73K--yjkA~{w26vc;j z8U5Z0VB5n+|2#q$g{kAh&${BE81d;F`~Sw{(*mVMKitFPGT-cIak$#1DV@UNsERlz zXLYOSfzb?0aP3O54rwXIkBB%22-w9Da3)4N`ZfBGLK^^}%?%UUY><+vLiBMXv zrH+>xWXm+_5b~X}O8?CkKjC`l0v9%mWuZ;SqYB``@{f~r$qN~4!9_YD^eNccKpT|E z8v~7Wwn3EdB+*b&#cD@KA^orI7cadL8sB$YE;m*kXiZr@y%v*}R_^e6e2BqnB3vgv zdAwfR4`&=bS?Ce{(^m*hA3cuBB{D)uXFyennssakG`J(Ow9j}Qeu zl4U#5yBq+aJtG1E!}F-l^)7#OEUY3xQcnxOJ|nytK48uvk^rWjpJF`;DA4Y@A-?!C zv!dWH0#3>$5H!NbmO$}~V**4#8tvvn^%`S0VCMWXzrUrIg9b8{CGt5U za~)60LkYC1`?F=2Kn_3!h;H?z97n#wI+m-et0&VN1c41h+j+%t^mCSWi__&)jQv+`s4ps*oy<+wWz7oLi1a_`3}*(;u;MN6iF<)PN1j5kP}G90Vx! z4gs+?3ONTyGf%w%-|WDX%PEZf)rvJ>VvdX5?Z70b5U{>nuq#mc7}cK>#rvcNG+NGL zwfL*m>1er#vb?eTmfy%*Y2kF-=?f~(AN#%(?qu$|cm{G*9Gk8LuD_!1CY|_Iqk!78 zVfe}?poWg~*eTpH7#VLmM-*jy>~ZxPXb|-?w9Ect$JP8*jiAQ?B{#SBQoWTT@N-#{ z@i=@tVdzb_z+$vD@xn$b^tWFkNBwdRYAPuyDHOQQk5*b{GWq?@vuvxqp$;JMuDaxG z=#1B_e2fd0T28tV(-P9?zqcxC5?#XSueg_UA(#cHh>h-vye@;#v_(syi`Z8nHL$GWolf%xq+sv2CGKMv;dn@r1 zWB*2Ak$zm6o0ynzZmu4Nqp&pA8&n=bM%9xowU4rI<8H6OEEi;Wxp^9#YI+`g;&ifX zN&9vbM%(^*^>DkQ{ce!H{$;Ianky)2R2TU`7li{TH#>vue;PNNDUMqk6cqWhrJ-tk zO8j;|NXG8~W!U3D^oTud1=_1#TD8PcT1q&iuj9|W-a>kH2*J&Q`T*#n5o{Y_9-eyD zAA=7n9g{T2S=$U8X7jgi1*Hydezmr8;qiE=0ZU=$cJ8@$_yo?@{xWh;YFj(30@$^& zw=aJ_^4kIQjd-Z>2pm?Y%Qlg%KX*xCIJhIkqLFOAex7W559ebQ;V5Q)N}Z9rmoQQ7 zzPT$IF)@myFD_xS>P44FS@a7`Yq}bb_knmngpsRIzb9x^FsA*OrOmT&6q&b*c8T)n z@!;{P2ZfhS5qeQ)<4ouF_I5PU>)xQN{lH`8sx33KlN@;P?05)fa`kM10CtrCru}#% zg_ZX9^~Sw_H_Rj#)Vo|(mR=F?dcgtv8+uBGm!4i8DEIO3 zJ_FOZN;r4D0@LdBs)|*Nhvjp<>mAHeJHT$YBp9!uwej)l!!n8WsP}bq?P@z@CMD#l zzr2YA`@=qbhlq_diS@qGhjB8}(vB)RC7yQWUw;1=_(M(?N~;PY+s0slH$oQ3_d6AH zvHb58e+gAgyxyE{7H0HpXj5=`cfa;r{t^;KsXxm8zbhK(y%wNdoo==?>(mznq{=OT zf*%>Z!=G0Ld;d=L8Pv~4W3_V@zkWn>9Jii-FaDzQdi7rm#qZbm^cF`&Jx+TuEr3&Q z)NWSUml(YD!3PWYqZ}8sVdc~kx&&Zpg@C~9!tT)g-wP0!zrs+T68C)SSfuRWS6Dx( z;vRrBxpHr0?Y{IY;URgFaLFef|8F;o9Yq}$l1s2 zxCvZ!KjwA>MpNyT``2lLLj7FjQ_?SWU+)f(?$HSdDsZeD`PaOk+m4BNexHJWdMsD| zf_sXG%ITpNDPOa}x(q1Xxy9We{(l4NM;vE7GPt&yXOTS%-`jeZ}%}Zb-?d+9Qs7OeJj`BV~ zNTiE1{B7)o?;Qm&z`*!609;AK-;+t>FxwtY8=ZDlW=qTc*RxA0AHC*KBo2VjwNP{p zzOAfQnw429jsM^36};%WAg{f*@jAx=rc%tQK59cWt z(n?KcwE>&F=}fV5^TXMYEIpabzxLo%USe1k84tCc_Td1KSV}EWCWC~})tKC5%kjU( z=A(nJG?ir5++Vb)m;sK!VomENVM>+KLSyUHKU)-RIv-zdy__AgVJ7 z6nH9c2K%X-s84XAU?*TgA z6TXnJ;HNjXe*L+$Eu zwda}jxF6WSKK;+Fr-5>LQH3#mC>N#sxTm_e($WCnYz38-l^Tr{34eYE+`B@vmu~qM z8uP!_0B0P$M?@oWqw%gz0jF}VTe%79U-zUDf$r6Qwtc$l`VLkJ#0ILMrv%f!eN&KB z1u^_H5#|q_bWzFQ$_Up0o0TLM2_J8}Ufpk{5E~@=+qoCMFTjJy>8nC)eTBm!6Ci(m9`$tag6Z!Zwu0Rok?uG?t^_}p%62E`>w9}51R2GiBR zVNmx|w6bPtNs;|R;{l71|K*Qy? zy=nb7PC_|hU_Vp%MdSYd&YzD8z+hGVoc}jQ;6?$TR>E}O{@)kZ(F=X`I7Dj%1eh!& zBd@WU&(fPr3k80vBn-6Bos;*Dtf&|h=$gxBwz3qpm1G=9Y^@~RYA}(?Xd=iu(e(d4 z34PC?v12>#r>`-L%O;p2R74DGB#geL)X5`ecq{7S4=X;mHD#3j~O zvROs4@I2RIJa{I5n-!G^&fh6N%|V}-RHEh83EP7{~>@1kht)A{8_8&&`Sziqur>ArW=Stf|7wDJL+NacEO$wg_&_MrA)UdXHuhvq zgkp9oHRXka8|R9rBthJ*H>x`USF_CAvg&ksAbXPA>2ATr&qbv}u5kg+;3&CeE8>}{ zRaG*JetxSu+X<~UbvpwL{n|Z;KrePG4Xel&8}rv;lK)*Pe0k6htBjAw+oIy|o6~ht zZ|$ubwFAXIZ4PZ~ye4Lj~v#q;}a#KB|q4RZ6s@3p# zQ}RQ_nPO_!mMgP&>gT1nj^F}e<>OmZ z#ycV3;X~P~RUn(|6tu+ELkCkp)D>+($BqcdrX#~HNJm_X3r2QhGj~{17gBdbCdUJV zE%uL=z5NxeukCD~dXFQpnUzaOBa|FGlFvLMP0FzIbnudQitL{D)|W`3gLxF~CiS5e zf-8yVE_Rqmnvzs}b4BNQ&_*vV8KV=OhAW9Rs}SR(Z(j`iZwj zcdH&(la~MOSJXrZd~a)LEPS+7m-wR}_IA`CKe}E2Iy?v5o1Pkrv98Gy^tXvCI9&3i zp4~y7dG{j)t^V$zlgFG|#vRu~$pdFXqvDiz8kIeaqrajPRn|`r5|2Z~rC9HsT|Qa1 zrwm`7O%5|>bM1mEkK?0L*XgMh6lSvxTDYc0+9%+rMykZdlB;_uc-lG{c@@&xcY7Vr zN2WbShty0*BhE;qqRu{=xA!-|kWU#!N?+yPA2DqosPT**M4cMWQ_0G$ej7^MthcT| zkcY>SaB?4x1BoVi>Lh$4-b!HgY$w(#-J6T^cu76D(A3~8Ho zf~nYah~vkf?%uA5Ag$s9&mg1-&EkmNC#_9eaL~#G-Kw`)9V1 z>51Acd@{^q-g_?ieC<^CbHhW^MGG0%YkAvy(na@#zZnV$oz4pGse4u`m2V)^&!_W2 z7p@(0Vo~&=*3%McoX#r2^&>NzPct%6Yw@m-Glk0DWvuf09?vQW@FBCHWv#L_dZmcu z25O15;t(v%HZda+-EEtR(uK+JjeN{SYb5Mlt@Ep+d$Yy+y2H4-m7ug*!!`VsnViDy z#e?zJj2>Oi`y9V`sF~%>eo`(;I&E8vvv9A{-}PMVYP^dpSEr0RZerQwNcBazh3+?0 zOC4$nGsv^`p1#=ic5hQjyfS6Vss|vKKlh4)e6hZ)Q8#P#zIRnuQ6;ubSE}0JSaJ!oELVDPu)(Q` zdQ0r* zU{EwgH0)S0*-88TguPa-rrv419b8`1*4D&`@TO)|$&nK7eoN0LvAeML_LzG5u7Gsl zqB)s}V#A!lvHBA%Z^=O;*>wA`E4l|ge;!iecJ8H$|A`zmQ`cR%kr_5mT~vrfR3iPP zeCRd&iKLzRxJb{YHTj)6L<=d1tWS~AWH&pXbpGtAY3-GE`voW@)aiX0xA}raE(1n; zp|vL&1%*w>`(m!X+4>P|ZjS51k>BX%()UPm*~ueH)*?M|0(9eoJytvz)2kBVi|?q6Qon+9lJVGeTUh#wJk{D+ zXp?~)BW%fiS6MU3bb5={drn7PV1C_ena{H$u}^xEi`G$XT{g07=Akj)EE)M;eaY`d#f zbrsx6qZ+&=Yu%RC;P8wSxirs{&(|&(1KxV7Xi?!NzAMU*&gf|Q>BPhuSGGGk#xt{u zL7hq<8yE;%dzu7AJUxJFu6=H|8qp4!hN3I@~=CJ<}c-&FSmKJwkLW3eE`)| zc!vXwRWktQtb#){E>+a=he6ec^+Ee57!O%eZdP<>F{DBgQQiRd17#)#lZ-bVVBm zF_7v#WLTEhaOwQ5t_h?vCCaTiB(r#~nQU>jFDb>xtKH+icoswsN{7mmb)Up)(RMkZ z%v;LscMytC*S?yHZq4Cv_`&l_?cxr!&U&{BQ`ehvXm;#2G$j!(rSOY{vm^v->X2J) z_qu10(?Kjaw8Hjchn3@IWaH=iMUG(>gWRQJ|6T4QJ-5L7HMWrh$?=<=p9B=6j)OL6 zjMKu3UYZv#bk**ME-lV`-ZP&|q8T1dGn{XK*$QF#T8|Y_G)9e*tTj~!;$6kSDN6|! zVQXqQ=AEzgXVMq)j+tB)gkp@B8 z%>mAoj>?O~?2&uP=WaQ9P8KHC=K~z~Bz(7h3%3)_YlQ=;bG~1xZ0rr=;u4+u|2eaT z?k{bwt?MMKt+uG9D7ln9Enlg$E{xMwHy|ZQHuP;+jU-HBy_agVs{gs3!>>xso|x4o z%aeum=)NmrPhz)>TlO#{v>?udP@~91&MmT=a1UdDJGfXcS@ynp4^tjin?ZVrY6%&`F^m#i* z#;oPV1>=X1g~cnEypQ6|Xgn6TLH(DVTieMec5PP+E+hSgAN9+VHa0&5iad%uN{yZC zFZrjP__)Q-aTxt#jT%mmAGl| z@Ol6Hw+^OX$@n1p{NUS-8P*hhP&R=yph#o2mbJG7HN;|auIjV%5lt+(_@rbkTp%38 zblvtTaWSb!*5f28ABAOG;v~^v(B%SjxH9dy9_LCXIoarVnK7PWcD>Y(}8<) zburn&ORA~(%-ow%Yp6x-#BMt$CDG^&Iq=s}N3}g($mZq*Hna{|460*F4T$Hmm!#_j>ARNPoL+69Wk=zpy@ihXf&%OFpuy29V%-$NLnzNer{by^oG8;@ zkg{ea{RVE2lhe*!BuJW#z3i-33$GYoZKhWXVxKiws2Q?S4>J+J7O<*joT;8dmBzPv z3TQw6YM=f{HqEYbHUkAJVb;rM+GF5scdUqy($IrA-I9>h%!E!Ew%|`@p~|p!Yc|PV#Vq{6WhH98aA25l4@RCgVXXES)cWd(Kcq9`UOrdTS9h&K`_Qr8}t<| zfjpuSK6b1tN0%#}l*DvJ&M52Sg%jm-`|?e{n4|ldr+uq0S|aQIw6ZPZux#N)&S={r z5yPU#YPP2ge%|x1hik`k$dZtQvRZpb-OJxnU~d)5(FjWq&K2&xVH%!u3YP@q z+|~32zxIf|$yiEN@jkQbNO$(Ax=p}Vo82l|Xb62YW@T$kaggf_{i!K%A zr=@RCp2dgsVl&CRn+!WF2smJ;>-Cw8#V5HfHB5C_W^cA9)r+s0^a+`im}MO+BTQ$8 zRPlDrR_9=uwAZi=bIGTqu4|b%&qFt_)X#lQ3RQE?c%QswcDRX_3d=i1L;s?Mf}`o6 z>Il*%fi&sf!jX1`H8+PSm#mqV7SG6w1gTw;GKiLGIk?V!2&2uh z0;`+Btq3h3*fENshnKX-UQ5=9>&C(n&L*>>zCC~bDIaz@yr@Bt?kt!pE)@u5E$VX?|ITyg=fQCGUp|b_2TsYm z{WYWZqABBtB@fOnwFPQVnQ$_@GhA7}r0Nl+IoDp2{8*WocvU8&g;b_vCOznMX+*Mw zT*uK6iO>1pmKun<4rDJ}6z_D3^mLQSi1IJ6uUaQU287Kq{OOc_U(a5MIk$#)K7jm>z_3ee{a+$0u4CPU8_w#Aut4MIJU^J!_5 z7Dp`@T)iaBp0?r}8XxL$dlkzUb{7UMVOiO4BXttxjT;z^R;#v{$tSl{-0-#)9MXP# zmj<~E+bZ5KgkR^*JH6W;pj+u8GmptGIt9fhVbFQX#F8;oxp4Hf;-#~L(#hMr!pYck z3I+Pp_NCARv1c`|jF313tHdqhF8!1KGAik4p|-+dvaN?x zfon5d(+taMb%>isoU26vlwyWClpm7JwYE9@%C*{wDt&NVGip*~=$xv7Woj<==qK7= zFQ=663w09wI~mWVjN-2lKgL zkjL??{XP7llG z=4Z+q;-wcauspGm%nS0B$V+SxZUaQg@Lth(-I|VK@^D?PCUbk5vt(@Ux6^37_Fu5iy$psFM4A}1J z6J#UNvPuu2|Ig9Cl*(M}a?n50Ih@BVE>DXL$wiuuS0~zYbco?XxhNvl%(^l*Dv`gJ zF<0>#R*dZ+mvr*7ud`7v8!t^Zava)?vy9>RS-8_qMXiu^bvz^!Sgdn8GG{lhOZ_2oGs{osEiLE9o)1j9p0-KvHx)M7ru(hHg}VD0M!To_p$O@}9yhA^=q=Vs>$;=!ZiX|a z5)lAz!xo+UWFO{vt^2~prle@P2d+!}uWcl7%KHU{dOYoTt(hV>AbThwD?0}uw`sv| zpMe68A|PM=fHoiYcg>(0UvxS6u*!K-tpaSV54W0#tglt--jyO)pRgXU}1fgZ9ca8)N`+)(UJ^Ybwt3{;dRE{8zpC;eR{g1DD?f4 zn+vYjXug<~Oxqv?7$5q4u@M6Qg9rXg4A1M9KoAiTVKW+s9|(qmpLCewIfKHeKi|sr6 zKDPh0>1=5dzWV_^z_v957+-tSMKQqBC^-Nj`hCFv<{})Ji%uqJXmZ(1;3ZIvW``nh zt#V#|elbAzce~pyNRp}GqPqHpD)|e5A}AX@@t92)4Lya=eh%eMba(EXt8H7qDJdyA6o=YEKt5^);}3s_LP_n7OKQxWcI>)Fv!~XX z629@f)^F*-_1@LtUV-6TL8+^(yqvX&oXpNt(MS_;1)~|UsI0;|TUXN1k>q)LUY1%p z;rm24&bc9;+DbZXs$}?&D@ub&tX*dJah}ao@t#6cALcFI@@qARLTLb89YHUDqv5xU zBY;uw25QM2Q9jlL+$DQ5U@!)HdOjjXo?R3YavkMfW#!Ht&cYv*?tN#<+{_;IuX0aI z0198XdG)>@u7^Y5aNtC~M%dyg9Ej*Nmk{nH$F=4^vHkGT947W=iE7)=&?*W(4Gxk9 zL;$}!cY5{y4F2M`M#+HV^0EljgP2YP6c>f_WwfiujL7;)60Pl@p@;-(HySNX@fj`v z`mDHI-gkY`$7uvLjyg|41Az^wfq{X==b}IRU(fjhYEuApR}A3xoQ14pUn$huhqH*o zhl}wXl|xzAwc^yn7X2mjr`qlj`7M!qtI*Vl9+%`z40%V`LGg#8u(Uf*zs+sc^1GM`uBIHT|iJxZ(=<(srYllwm(|*%-8HEKM_aQBXGylLL2uH<66?F{2SD282j<+sB28MZp=Nx$kgZf`n>7b^Jk5 z3V?z7+eV-%mN}Lo&M!Yk&Y8WBf=+{pSW)lCU$5YcZSFw-yKD8g7cc$*WNUGJ>*sZ& z2tdk9#QEbibrke#AjjvLg4=oD@O3M>9SitG*4SPK+~1L0OY8 z{FpK@FlYx(kqjq2jza;n1Ds+B5KQtu+{TSehthu;1TUFAumPnc`SyXcE>eI+>=>$f zE@{$FXFKg}tsV@&YTU6ltr##;3LjoxhkOn-a?39sah<15ouUJQ5|p~HDy9x;8BA#o zw_MlsJk;x9^StHbJ0r?az?4A!?}kX&cXL;k-)cjvc_?1fO6y)3jkt^c%8h%puz@V; zDbm(Whi8Z+yf+-ks0I=4?HcX{1j&$1GcYAEh|2wS>2y2xk#n4uR$}}186bU5y{R<=uqxOeQw`DwK+RJo>F+F2{LN#-$Cw!|Iga~*m0GGFj0+aGXxLl(f!tCvje4U?uHkDYn zp~q0>|0*$YRBV*(BOEI4Dwhr}8t3cli(HZ-S&QAYF6w#f@TaUxT}dB!p#>jd{-8B2 zQDeTJHEe0J3eu8V!ctG4JFs&s<{DfPYpNB7%=`fveRW1J1cFVhRzJk6Tk}oyz)Jw=X#7cL( zWFlT?(VV&C^`s(kvk?^hC)nw6gx@SL`~)Tcf>as%&fsb(rV;(U8w%ei++~`g!$sL9 znxa9c%alFeli^o7P~nQ?ayBs^9C`)w?aw7{%6)MYX3n+j3hMidL$fK5wVn-p9y=^tX4 zCL53{T@~}UQTo9Ms(|6mc}9L_W{T*IzP1Gq8f8v~RG}4h5U@md;P%REJ^`p*L@lyb zK=0-iij`;NXO^}AlUPX%qG9}2EQisbUkW-V3!};|t`hR9Xn>qJz#n(buQ z8j5}R_^y6&c&iIgLeM3b#Bb@{TSL9c=7Qi0=* z202VG@Np$iA##V0!kZ)I&&v2V;a~Z+Uz#T+Uq!=svDd&uZ)6h%=J`6C1rB%WlKawG zR(BzrAbNYw2XX#Qsm&I2(!5nh^ksN=`1uVtGLbcymYABF9mB&cf6-R*qIKLXr%lebuG_Zy-6tN z!`3+fUR2AlrrBeFp0Z=)PG0^PGA@&oojnR&GKdem!Xt$D9@EodOb^O2{5HMLdzS&& zc*?f?h;1x1;_p~$_9(&^-yl($nl2-MToY)MCBCG80ZkwF0FpwP%YFFk zIRZrPxm^;r|Dx9nAr3iA8hfoTBw4@{z;n7r|3aR8(y**SR%tW@;Uz8a z${vmckrXutIh*HqcWukn^0JjPEe%bbPSR2f`7TsB7MA&mJs|O1dF9CgoPkMi_IHv+ z-{?^cp9;RsAv5K3!SWNkr4I^rHJ(xJTl9pN{>*Eus&_k|HD{7B(UDg%KVzu4lBz*- z*3g_CV2jTB&5Lre(EiRhKW?hn5u@dXpO!p_wm`a&Zv!%q4pW8P^&H`7(8Xu4w7E}dXg^~YtwksGBWgBZ7t$lv_ptlItBqHn2a5v%GO zbEWntV=VIwt9>%>YD>}&1Q(3aOFDgXGzq#XEAek4)e!^ofHhp8F%E~g+M9K4s6Imf zK?i@|>_ev#;?rA447~mtA?mt+x<$|Mar598a2>Pd*V8pI{kk#RE8YpCtYkZQQq6tU z`lb5&mx4{9glP8q($w-QpVHe;FXnq*_i%JRU4oBKlI=ndIx9N2iO22L*xv3Otvq$W za_XVZ7$2r!c6c>JiK}g;s65(lm{Ieu%R*7^B)uaW)lXy z1Tv49G-kF3oNcX*7T$TUDB%Wqk{2QQrK;@afUm-V3bc8d{a1-1J-uf4?!Nrg(~jCg zF|z5|EZm{|v3QyjzY+0AM`C0crpi4BFL3}j5vLHt|MH9*?H#65wfXxN!(E}KB^|Ca zSeH%@iR-Dkr8f)nr&^53s~$4xs}C3B9m!rlT9SfIEX@7d6$8$jN;YKXy*M+>?q}NwVe}0V7W7z7c-S%T56^fqXr(z^(Zm2n zw;o%sXOkBt`~tq;=do)^r(ff1plSwrrv?rwnLi!yZ5_Y1XMvWrKoJp|3)jl*lryHF z?9CKLuFi=5cy>2h^`2>bwP)kTPzKnTpSr<0x9DX}vhJCJ&B1maX4%_T`?+b3iMgZN!_wS}hl3Mn zl`@*N75&;6(q+S-&y9m$RZAbMyV>0iEs~Wwlyl~Mlbmf@C=ZX!!Yi0G#JNH?va}<4 z0%Fotw&(^U>q_=s;(n#P*LdO`h2$C;!wWyAt%mDIlmfqaz9Xq?30|o(pGdlP`t9uQ zN{oVGX8D}^Q~mO<3rkcP+_)B!X|2ng}k1lw1glnH#a*>63OSVJ#g@*?uBjH>W@kg3PQfSuJ5CDoD6MUFff_ z%Re7ruJ3eww(XFb`Hq=SpZE8)dlv@gC|XT~e;vzIEZN;M`ocw>@PpUY=2*VM!P-$X z`bcWcgn|Vy z9sO>~-P?VASq0DOA~f&GL;>cy3>;!DHQw2ElggT57N! zUH>1#G<@Oz6%;u_bVeK*xR@APOH0ee(KAmXytkm*cRTcF;ly4zD+l38cFgTAM4V>o zkVsoH5}-qai-K?nM9!^EQ!=LTbcV;AUU50 zFvJ+4tioTzMpO(K-?NALwb^T>t<8 From 9b5e94034233a4285430f3f7e6fc61d37a5fed98 Mon Sep 17 00:00:00 2001 From: gongweibao Date: Fri, 12 May 2017 19:30:49 +0800 Subject: [PATCH 73/88] fix filenames --- doc/design/cluster_train/data_dispatch.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/design/cluster_train/data_dispatch.md b/doc/design/cluster_train/data_dispatch.md index 9f7167498c..a84fcd851d 100644 --- a/doc/design/cluster_train/data_dispatch.md +++ b/doc/design/cluster_train/data_dispatch.md @@ -114,7 +114,7 @@ trainer.train(batch_reader, ...) 使用下面命令，可以把本地的数据上传到存储集群中。 ```bash -paddle pfs cp filenames /pfs/$DATACENTER/home/$USER/folder/ +paddle pfs cp filename /pfs/$DATACENTER/home/$USER/folder/ ``` 比如，把之前示例中转换完毕的random_images数据集上传到云端的`/home/`可以用以下指令： @@ -141,7 +141,8 @@ endpoint=datacenter2.paddlepaddle.org ``` ## TODO ### 文件访问的权限 -控制用户权限 +控制用户权限 + - 用户可以把自己的数据分享给别人 ### 文件访问方式 From b0f070e013fea9e93728e3cfe1bfc5045132c51b Mon Sep 17 00:00:00 2001 From: liaogang Date: Sat, 13 May 2017 00:11:59 +0800 Subject: [PATCH 74/88] add cmake for majel --- paddle/CMakeLists.txt | 8 ++++++++ paddle/majel/CMakeLists.txt | 15 +++++++++++++++ paddle/majel/test/CMakeLists.txt | 10 ++++++++++ 3 files changed, 33 insertions(+) create mode 100644 paddle/majel/CMakeLists.txt create mode 100644 paddle/majel/test/CMakeLists.txt diff --git a/paddle/CMakeLists.txt b/paddle/CMakeLists.txt index eff296bcb0..c6fd9cc54a 100644 --- a/paddle/CMakeLists.txt +++ b/paddle/CMakeLists.txt @@ -9,6 +9,14 @@ add_subdirectory(pserver) add_subdirectory(trainer) add_subdirectory(scripts) +find_package(boost QUIET) + +if(Boost_FOUND) + include_directories(${Boost_INCLUDE_DIRS}) + include_directories(${CMAKE_CURRENT_SOURCE_DIR}) + add_subdirectory(majel) +endif() + if(WITH_C_API) add_subdirectory(capi) endif() diff --git a/paddle/majel/CMakeLists.txt b/paddle/majel/CMakeLists.txt new file mode 100644 index 0000000000..f2f57900c8 --- /dev/null +++ b/paddle/majel/CMakeLists.txt @@ -0,0 +1,15 @@ +set(MAJEL_CXX_FILES place.cpp) +set(MAJEL_CUDA_FILES "") + +if(CUDA_FOUND) + cuda_add_library(majel ${MAJEL_CUDA_FILES} ${MAJEL_CXX_FILES}) +else() + add_library(majel ${MAJEL_CXX_FILES}) +endif() + +file(GLOB_RECURSE CHECK_FILES RELATIVE "${CMAKE_CURRENT_SOURCE_DIR}" "*.cpp" "*.h") +add_style_check_target(majel ${CHECK_FILES}) + +if(WITH_TESTING) + add_subdirectory(test) +endif() \ No newline at end of file diff --git a/paddle/majel/test/CMakeLists.txt b/paddle/majel/test/CMakeLists.txt new file mode 100644 index 0000000000..a07b1bf5d8 --- /dev/null +++ b/paddle/majel/test/CMakeLists.txt @@ -0,0 +1,10 @@ +file(GLOB_RECURSE ALL_TEST_FILES RELATIVE "${CMAKE_CURRENT_SOURCE_DIR}" "*.cpp" "*.cc") + +add_executable(majel_tests ${ALL_TEST_FILES}) +add_dependencies(majel_tests majel ${external_project_dependencies}) +target_link_libraries(majel_tests + ${Boost_LIBRARIES} + ${GTEST_LIBRARIES} + majel + ) +add_test(majel_tests majel_tests) From 3a734557d80129cd0f557ee590b2f4db91b63276 Mon Sep 17 00:00:00 2001 From: liaogang Date: Sat, 13 May 2017 00:12:28 +0800 Subject: [PATCH 75/88] move majel into paddle/majel --- majel/Makefile | 10 ---------- majel/place.cu => paddle/majel/place.cpp | 2 +- {majel => paddle/majel}/place.h | 0 .../place_test.cu => paddle/majel/test/place_test.cpp | 0 paddle/majel/test/test_framework.cpp | 6 ++++++ 5 files changed, 7 insertions(+), 11 deletions(-) delete mode 100644 majel/Makefile rename majel/place.cu => paddle/majel/place.cpp (97%) rename {majel => paddle/majel}/place.h (100%) rename majel/place_test.cu => paddle/majel/test/place_test.cpp (100%) create mode 100644 paddle/majel/test/test_framework.cpp diff --git a/majel/Makefile b/majel/Makefile deleted file mode 100644 index 403c1d3a6a..0000000000 --- a/majel/Makefile +++ /dev/null @@ -1,10 +0,0 @@ -CCFLAGS = -std=c++11 -I/work/paddle -CC = nvcc - -all : place_test - -place.o : place.h place.cu - $(CC) $(CCFLAGS) -c place.cu -o $@ - -place_test : place.o place_test.cu - $(CC) $(CCFLAGS) -lgtest -lgtest_main $^ -o $@ diff --git a/majel/place.cu b/paddle/majel/place.cpp similarity index 97% rename from majel/place.cu rename to paddle/majel/place.cpp index ee84e96048..396534b84b 100644 --- a/majel/place.cu +++ b/paddle/majel/place.cpp @@ -1,4 +1,4 @@ -#include +#include "majel/place.h" namespace majel { diff --git a/majel/place.h b/paddle/majel/place.h similarity index 100% rename from majel/place.h rename to paddle/majel/place.h diff --git a/majel/place_test.cu b/paddle/majel/test/place_test.cpp similarity index 100% rename from majel/place_test.cu rename to paddle/majel/test/place_test.cpp diff --git a/paddle/majel/test/test_framework.cpp b/paddle/majel/test/test_framework.cpp new file mode 100644 index 0000000000..a62216ba85 --- /dev/null +++ b/paddle/majel/test/test_framework.cpp @@ -0,0 +1,6 @@ +#include "gtest/gtest.h" + +int main(int argc, char** argv) { + ::testing::InitGoogleTest(&argc, argv); + return RUN_ALL_TESTS(); +} From 9c501e90cf35ce159c5ddd26faea1fe955e4abdd Mon Sep 17 00:00:00 2001 From: liaogang Date: Sat, 13 May 2017 00:12:40 +0800 Subject: [PATCH 76/88] remove gtest in dockerfile --- Dockerfile | 9 --------- 1 file changed, 9 deletions(-) diff --git a/Dockerfile b/Dockerfile index dc44388732..ad0d086d3c 100644 --- a/Dockerfile +++ b/Dockerfile @@ -61,15 +61,6 @@ RUN git clone https://github.com/woboq/woboq_codebrowser /woboq && \ -DCMAKE_BUILD_TYPE=Release . \ make) -# Install gtest. -# -# NOTE: This is added for quick hack of the development work of -# majel-in-paddle. -RUN git clone https://github.com/google/googletest /gtest && \ - cd /gtest && \ - git checkout -b release-1.8.0 && \ - cmake . && make install - # Configure OpenSSH server. c.f. https://docs.docker.com/engine/examples/running_ssh_service RUN mkdir /var/run/sshd RUN echo 'root:root' | chpasswd From 92e0c893e4101eeb8303a07777cb7d097f705a9d Mon Sep 17 00:00:00 2001 From: liaogang Date: Sat, 13 May 2017 00:15:07 +0800 Subject: [PATCH 77/88] precommit --- paddle/majel/CMakeLists.txt | 2 +- paddle/majel/place.cpp | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/paddle/majel/CMakeLists.txt b/paddle/majel/CMakeLists.txt index f2f57900c8..f1cd4ee9a1 100644 --- a/paddle/majel/CMakeLists.txt +++ b/paddle/majel/CMakeLists.txt @@ -12,4 +12,4 @@ add_style_check_target(majel ${CHECK_FILES}) if(WITH_TESTING) add_subdirectory(test) -endif() \ No newline at end of file +endif() diff --git a/paddle/majel/place.cpp b/paddle/majel/place.cpp index 396534b84b..ee84e96048 100644 --- a/paddle/majel/place.cpp +++ b/paddle/majel/place.cpp @@ -1,4 +1,4 @@ -#include "majel/place.h" +#include namespace majel { From d8c8f419f974a9d02bddd937330e8201a3a2a699 Mon Sep 17 00:00:00 2001 From: liaogang Date: Sat, 13 May 2017 01:09:24 +0800 Subject: [PATCH 78/88] support build majel in subdir --- paddle/majel/.gitignore | 2 ++ paddle/majel/CMakeLists.txt | 31 +++++++++++++++++++++++++------ paddle/majel/test/CMakeLists.txt | 2 +- 3 files changed, 28 insertions(+), 7 deletions(-) create mode 100644 paddle/majel/.gitignore diff --git a/paddle/majel/.gitignore b/paddle/majel/.gitignore new file mode 100644 index 0000000000..1f5acdebb5 --- /dev/null +++ b/paddle/majel/.gitignore @@ -0,0 +1,2 @@ +build +third-party \ No newline at end of file diff --git a/paddle/majel/CMakeLists.txt b/paddle/majel/CMakeLists.txt index f1cd4ee9a1..f3bc984a26 100644 --- a/paddle/majel/CMakeLists.txt +++ b/paddle/majel/CMakeLists.txt @@ -1,3 +1,26 @@ +cmake_minimum_required(VERSION 3.0) + +if(GTEST_INCLUDE_DIR AND GTEST_LIBRARIES) + message("-- Found gtest (include: ${GTEST_INCLUDE_DIR}, library: ${GTEST_LIBRARIES})") +else() + # find #include + get_filename_component(PARENT_DIR ${CMAKE_CURRENT_SOURCE_DIR} DIRECTORY) + include_directories(${PARENT_DIR}) + + # find cmake directory modules + get_filename_component(PARENT_DIR ${PARENT_DIR} DIRECTORY) + set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${PARENT_DIR}/cmake") + + # enable c++11 + set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11") + + # enable gtest + set(THIRD_PARTY_PATH ${CMAKE_CURRENT_SOURCE_DIR}/third_party) + set(WITH_TESTING ON) + include(external/gtest) +endif() + +########################### Build Majel ############################# set(MAJEL_CXX_FILES place.cpp) set(MAJEL_CUDA_FILES "") @@ -6,10 +29,6 @@ if(CUDA_FOUND) else() add_library(majel ${MAJEL_CXX_FILES}) endif() +########################### Build Majel ############################# -file(GLOB_RECURSE CHECK_FILES RELATIVE "${CMAKE_CURRENT_SOURCE_DIR}" "*.cpp" "*.h") -add_style_check_target(majel ${CHECK_FILES}) - -if(WITH_TESTING) - add_subdirectory(test) -endif() +add_subdirectory(test) diff --git a/paddle/majel/test/CMakeLists.txt b/paddle/majel/test/CMakeLists.txt index a07b1bf5d8..0cc7103b03 100644 --- a/paddle/majel/test/CMakeLists.txt +++ b/paddle/majel/test/CMakeLists.txt @@ -1,7 +1,7 @@ file(GLOB_RECURSE ALL_TEST_FILES RELATIVE "${CMAKE_CURRENT_SOURCE_DIR}" "*.cpp" "*.cc") add_executable(majel_tests ${ALL_TEST_FILES}) -add_dependencies(majel_tests majel ${external_project_dependencies}) +add_dependencies(majel_tests majel) target_link_libraries(majel_tests ${Boost_LIBRARIES} ${GTEST_LIBRARIES} From 05b3196e0a9e92e7123fb51fe1a7f46992843607 Mon Sep 17 00:00:00 2001 From: liaogang Date: Sat, 13 May 2017 01:10:48 +0800 Subject: [PATCH 79/88] clang-format --- paddle/majel/place.cpp | 44 ++++++++++------------------ paddle/majel/test/place_test.cpp | 16 +++++----- paddle/majel/test/test_framework.cpp | 4 +-- 3 files changed, 26 insertions(+), 38 deletions(-) diff --git a/paddle/majel/place.cpp b/paddle/majel/place.cpp index ee84e96048..eecd8e5b73 100644 --- a/paddle/majel/place.cpp +++ b/paddle/majel/place.cpp @@ -4,58 +4,46 @@ namespace majel { namespace detail { -class PlacePrinter - : public boost::static_visitor<> { +class PlacePrinter : public boost::static_visitor<> { private: - std::ostream& os_; + std::ostream& os_; + public: - PlacePrinter(std::ostream& os) : os_(os) {} + PlacePrinter(std::ostream& os) : os_(os) {} - void operator()(const CpuPlace&) { - os_ << "CpuPlace"; - } + void operator()(const CpuPlace&) { os_ << "CpuPlace"; } - void operator()(const GpuPlace& p) { - os_ << "GpuPlace(" << p.device << ")"; - } + void operator()(const GpuPlace& p) { os_ << "GpuPlace(" << p.device << ")"; } }; } // namespace majel static Place the_default_place; -void set_place(const Place& place) { - the_default_place = place; -} +void set_place(const Place& place) { the_default_place = place; } -const Place& get_place() { - return the_default_place; -} +const Place& get_place() { return the_default_place; } -const GpuPlace default_gpu() { - return GpuPlace(0); -} +const GpuPlace default_gpu() { return GpuPlace(0); } -const CpuPlace default_cpu() { - return CpuPlace(); -} +const CpuPlace default_cpu() { return CpuPlace(); } bool is_gpu_place(const Place& p) { - return boost::apply_visitor(IsGpuPlace(), p); + return boost::apply_visitor(IsGpuPlace(), p); } bool is_cpu_place(const Place& p) { - return !boost::apply_visitor(IsGpuPlace(), p); + return !boost::apply_visitor(IsGpuPlace(), p); } bool places_are_same_class(const Place& p1, const Place& p2) { - return is_gpu_place(p1) == is_gpu_place(p2); + return is_gpu_place(p1) == is_gpu_place(p2); } std::ostream& operator<<(std::ostream& os, const majel::Place& p) { - majel::detail::PlacePrinter printer(os); - boost::apply_visitor(printer, p); - return os; + majel::detail::PlacePrinter printer(os); + boost::apply_visitor(printer, p); + return os; } } // namespace majel diff --git a/paddle/majel/test/place_test.cpp b/paddle/majel/test/place_test.cpp index cb82946130..c9a53802b2 100644 --- a/paddle/majel/test/place_test.cpp +++ b/paddle/majel/test/place_test.cpp @@ -1,6 +1,6 @@ -#include "gtest/gtest.h" #include "majel/place.h" #include +#include "gtest/gtest.h" TEST(Place, Equality) { majel::CpuPlace cpu; @@ -18,12 +18,12 @@ TEST(Place, Equality) { } TEST(Place, Default) { - EXPECT_TRUE(majel::is_gpu_place( majel::get_place())); - EXPECT_TRUE(majel::is_gpu_place( majel::default_gpu())); - EXPECT_TRUE(majel::is_cpu_place( majel::default_cpu())); + EXPECT_TRUE(majel::is_gpu_place(majel::get_place())); + EXPECT_TRUE(majel::is_gpu_place(majel::default_gpu())); + EXPECT_TRUE(majel::is_cpu_place(majel::default_cpu())); majel::set_place(majel::CpuPlace()); - EXPECT_TRUE(majel::is_cpu_place( majel::get_place())); + EXPECT_TRUE(majel::is_cpu_place(majel::get_place())); } TEST(Place, Print) { @@ -33,8 +33,8 @@ TEST(Place, Print) { EXPECT_EQ("GpuPlace(1)", ss.str()); } { - std::stringstream ss; - ss << majel::CpuPlace(); - EXPECT_EQ("CpuPlace", ss.str()); + std::stringstream ss; + ss << majel::CpuPlace(); + EXPECT_EQ("CpuPlace", ss.str()); } } diff --git a/paddle/majel/test/test_framework.cpp b/paddle/majel/test/test_framework.cpp index a62216ba85..443e2dbb3f 100644 --- a/paddle/majel/test/test_framework.cpp +++ b/paddle/majel/test/test_framework.cpp @@ -1,6 +1,6 @@ #include "gtest/gtest.h" int main(int argc, char** argv) { - ::testing::InitGoogleTest(&argc, argv); - return RUN_ALL_TESTS(); + ::testing::InitGoogleTest(&argc, argv); + return RUN_ALL_TESTS(); } From d444251c4a43facb2d15f0952b6312a616210c22 Mon Sep 17 00:00:00 2001 From: liaogang Date: Sat, 13 May 2017 01:15:04 +0800 Subject: [PATCH 80/88] refine format --- paddle/majel/CMakeLists.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/paddle/majel/CMakeLists.txt b/paddle/majel/CMakeLists.txt index f3bc984a26..0c91fa72da 100644 --- a/paddle/majel/CMakeLists.txt +++ b/paddle/majel/CMakeLists.txt @@ -29,6 +29,6 @@ if(CUDA_FOUND) else() add_library(majel ${MAJEL_CXX_FILES}) endif() -########################### Build Majel ############################# +##################################################################### add_subdirectory(test) From 5081c691b1f706080dc7e87c6ca459162f4a8ca8 Mon Sep 17 00:00:00 2001 From: Helin Wang Date: Fri, 12 May 2017 11:14:11 -0700 Subject: [PATCH 81/88] rename parameter chunks to parameter blocks --- doc/design/cluster_train/pserver_client.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design/cluster_train/pserver_client.md b/doc/design/cluster_train/pserver_client.md index 500894fac7..392bab25e9 100644 --- a/doc/design/cluster_train/pserver_client.md +++ b/doc/design/cluster_train/pserver_client.md @@ -4,7 +4,7 @@ For an overview of trainer's role, please refer to [distributed training design ## Parameter Partition -Each parameter will be partitioned into parameter chunks to make the parameters evenly distributed on parameter servers. The partition is done automatically by the client library. The *sparse parameter* require a little different treatment: +Each parameter will be partitioned into parameter blocks to make the parameters evenly distributed on parameter servers. The partition is done automatically by the client library. The *sparse parameter* require a little different treatment: ### Sparse Parameter From 4bf8e372b950699285ec2f59c437483240cce01c Mon Sep 17 00:00:00 2001 From: liaogang Date: Sat, 13 May 2017 10:59:56 +0800 Subject: [PATCH 82/88] Add Google style conventions to Majel --- .travis.yml | 1 + paddle/majel/CMakeLists.txt | 2 +- paddle/majel/{place.cpp => place.cc} | 0 paddle/majel/test/CMakeLists.txt | 3 ++- paddle/majel/test/{place_test.cpp => place_test.cc} | 0 paddle/majel/test/test_framework.cpp | 6 ------ 6 files changed, 4 insertions(+), 8 deletions(-) rename paddle/majel/{place.cpp => place.cc} (100%) rename paddle/majel/test/{place_test.cpp => place_test.cc} (100%) delete mode 100644 paddle/majel/test/test_framework.cpp diff --git a/.travis.yml b/.travis.yml index 865e21f046..d73fd39aa7 100644 --- a/.travis.yml +++ b/.travis.yml @@ -25,6 +25,7 @@ addons: - python2.7-dev - python-numpy - python-wheel + - libboost-dev - curl - swig - graphviz diff --git a/paddle/majel/CMakeLists.txt b/paddle/majel/CMakeLists.txt index 0c91fa72da..baa3bb9e91 100644 --- a/paddle/majel/CMakeLists.txt +++ b/paddle/majel/CMakeLists.txt @@ -21,7 +21,7 @@ else() endif() ########################### Build Majel ############################# -set(MAJEL_CXX_FILES place.cpp) +set(MAJEL_CXX_FILES place.cc) set(MAJEL_CUDA_FILES "") if(CUDA_FOUND) diff --git a/paddle/majel/place.cpp b/paddle/majel/place.cc similarity index 100% rename from paddle/majel/place.cpp rename to paddle/majel/place.cc diff --git a/paddle/majel/test/CMakeLists.txt b/paddle/majel/test/CMakeLists.txt index 0cc7103b03..46da6ff89b 100644 --- a/paddle/majel/test/CMakeLists.txt +++ b/paddle/majel/test/CMakeLists.txt @@ -1,10 +1,11 @@ -file(GLOB_RECURSE ALL_TEST_FILES RELATIVE "${CMAKE_CURRENT_SOURCE_DIR}" "*.cpp" "*.cc") +file(GLOB_RECURSE ALL_TEST_FILES RELATIVE "${CMAKE_CURRENT_SOURCE_DIR}" "*.cc") add_executable(majel_tests ${ALL_TEST_FILES}) add_dependencies(majel_tests majel) target_link_libraries(majel_tests ${Boost_LIBRARIES} ${GTEST_LIBRARIES} + ${GTEST_MAIN_LIBRARIES} majel ) add_test(majel_tests majel_tests) diff --git a/paddle/majel/test/place_test.cpp b/paddle/majel/test/place_test.cc similarity index 100% rename from paddle/majel/test/place_test.cpp rename to paddle/majel/test/place_test.cc diff --git a/paddle/majel/test/test_framework.cpp b/paddle/majel/test/test_framework.cpp deleted file mode 100644 index 443e2dbb3f..0000000000 --- a/paddle/majel/test/test_framework.cpp +++ /dev/null @@ -1,6 +0,0 @@ -#include "gtest/gtest.h" - -int main(int argc, char** argv) { - ::testing::InitGoogleTest(&argc, argv); - return RUN_ALL_TESTS(); -} From 1171014d3ffc31aae5c4572c495c110975220444 Mon Sep 17 00:00:00 2001 From: caoying03 Date: Mon, 15 May 2017 11:32:06 +0800 Subject: [PATCH 83/88] add param_attr to nce_layer and enable multiple inputs. --- python/paddle/trainer_config_helpers/layers.py | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/python/paddle/trainer_config_helpers/layers.py b/python/paddle/trainer_config_helpers/layers.py index 31652613fb..52c7a57a1f 100755 --- a/python/paddle/trainer_config_helpers/layers.py +++ b/python/paddle/trainer_config_helpers/layers.py @@ -4921,12 +4921,14 @@ def crf_decoding_layer(input, @wrap_act_default(act=SigmoidActivation()) @wrap_bias_attr_default(has_bias=True) +@wrap_param_attr_default() @wrap_name_default() @layer_support() def nce_layer(input, label, num_classes, act=None, + param_attr=None, weight=None, num_neg_samples=10, neg_distribution=None, @@ -4957,6 +4959,8 @@ def nce_layer(input, :type num_classes: int :param act: Activation, default is Sigmoid. :type act: BaseActivation + :param param_attr: The Parameter Attribute|list. + :type param_attr: ParameterAttribute :param num_neg_samples: number of negative samples. Default is 10. :type num_neg_samples: int :param neg_distribution: The distribution for generating the random negative labels. @@ -4972,7 +4976,16 @@ def nce_layer(input, """ if isinstance(input, LayerOutput): input = [input] + assert not isinstance(param_attr, collections.Sequence) + param_attr = [param_attr] + else: + if isinstance(param_attr, collections.Sequence): + assert len(input) == len(param_attr) + else: + param_attr = [copy.deepcopy(param_attr) for _ in range(len(input))] + assert isinstance(input, collections.Sequence) + assert isinstance(label, LayerOutput) assert label.layer_type == LayerType.DATA if neg_distribution is not None: @@ -4984,9 +4997,9 @@ def nce_layer(input, ipts_for_layer = [] parents = [] - for each_input in input: + for each_input, attr in zip(input, param_attr): assert isinstance(each_input, LayerOutput) - ipts_for_layer.append(each_input.name) + ipts_for_layer.append(Input(each_input.name, **attr.attr)) parents.append(each_input) ipts_for_layer.append(label.name) parents.append(label) From 3e36e564fba2d8e22754a9c2aea458450ba492ca Mon Sep 17 00:00:00 2001 From: Luo Tao Date: Mon, 15 May 2017 13:24:45 +0800 Subject: [PATCH 84/88] add multiple outputs in annotation of paddle.infer --- python/paddle/v2/inference.py | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/python/paddle/v2/inference.py b/python/paddle/v2/inference.py index b4bb384969..2c1ae08f46 100644 --- a/python/paddle/v2/inference.py +++ b/python/paddle/v2/inference.py @@ -97,8 +97,22 @@ def infer(output_layer, parameters, input, feeding=None, field='value'): input=SomeData) print result + If there are multiple outout_layers and fields, the examples usages: + + .. code-block:: python + + result = paddle.infer(output_layer=[prediction1, prediction2], + parameters=parameters, + input=SomeData, + field=[id, value]]) + print result + + The result order is prediction1.id, prediction2.id, prediction1.value, + prediction2.value. + :param output_layer: output of the neural network that would be inferred - :type output_layer: paddle.v2.config_base.Layer + :type output_layer: paddle.v2.config_base.Layer or a list of + paddle.v2.config_base.Layer :param parameters: parameters of the neural network. :type parameters: paddle.v2.parameters.Parameters :param input: input data batch. Should be a python iterable object, and each From 7556ceffa6d0174ce7d2135b62b5c4c888400dcb Mon Sep 17 00:00:00 2001 From: caoying03 Date: Mon, 15 May 2017 13:17:36 +0800 Subject: [PATCH 85/88] add unitest for nce layer. --- .../paddle/trainer_config_helpers/layers.py | 7 ++- .../protostr/test_cost_layers.protostr | 39 +++++++++++++++ .../test_cost_layers_with_weight.protostr | 49 +++++++++++++++++++ .../tests/configs/test_cost_layers.py | 4 +- .../configs/test_cost_layers_with_weight.py | 7 ++- 5 files changed, 102 insertions(+), 4 deletions(-) diff --git a/python/paddle/trainer_config_helpers/layers.py b/python/paddle/trainer_config_helpers/layers.py index 52c7a57a1f..2af7c9c9c4 100755 --- a/python/paddle/trainer_config_helpers/layers.py +++ b/python/paddle/trainer_config_helpers/layers.py @@ -4926,7 +4926,7 @@ def crf_decoding_layer(input, @layer_support() def nce_layer(input, label, - num_classes, + num_classes=None, act=None, param_attr=None, weight=None, @@ -4944,7 +4944,8 @@ def nce_layer(input, .. code-block:: python - cost = nce_layer(input=layer1, label=layer2, weight=layer3, + cost = nce_layer(input=[layer1, layer2], label=layer2, + param_attr=[attr1, attr2], weight=layer3, num_classes=3, neg_distribution=[0.1,0.3,0.6]) :param name: layer name @@ -4988,6 +4989,8 @@ def nce_layer(input, assert isinstance(label, LayerOutput) assert label.layer_type == LayerType.DATA + if num_classes is None: + num_classes = label.size if neg_distribution is not None: assert isinstance(neg_distribution, collections.Sequence) assert len(neg_distribution) == num_classes diff --git a/python/paddle/trainer_config_helpers/tests/configs/protostr/test_cost_layers.protostr b/python/paddle/trainer_config_helpers/tests/configs/protostr/test_cost_layers.protostr index 05fd1c99d2..05847344be 100644 --- a/python/paddle/trainer_config_helpers/tests/configs/protostr/test_cost_layers.protostr +++ b/python/paddle/trainer_config_helpers/tests/configs/protostr/test_cost_layers.protostr @@ -215,6 +215,22 @@ layers { } coeff: 1.0 } +layers { + name: "__nce_layer_0__" + type: "nce" + size: 1 + active_type: "sigmoid" + inputs { + input_layer_name: "__fc_layer_0__" + input_parameter_name: "___nce_layer_0__.w0" + } + inputs { + input_layer_name: "labels" + } + bias_parameter_name: "___nce_layer_0__.wbias" + num_classes: 5000 + num_neg_samples: 10 +} parameters { name: "___fc_layer_0__.w0" size: 800 @@ -245,6 +261,26 @@ parameters { initial_strategy: 0 initial_smart: true } +parameters { + name: "___nce_layer_0__.w0" + size: 20000 + initial_mean: 0.0 + initial_std: 0.0141421356237 + dims: 5000 + dims: 4 + initial_strategy: 0 + initial_smart: true +} +parameters { + name: "___nce_layer_0__.wbias" + size: 5000 + initial_mean: 0.0 + initial_std: 0.0 + dims: 1 + dims: 5000 + initial_strategy: 0 + initial_smart: false +} input_layer_names: "input" input_layer_names: "labels" input_layer_names: "crf_label" @@ -267,6 +303,7 @@ output_layer_names: "__cross_entropy_with_selfnorm_0__" output_layer_names: "__huber_cost_0__" output_layer_names: "__multi_binary_label_cross_entropy_0__" output_layer_names: "__sum_cost_0__" +output_layer_names: "__nce_layer_0__" sub_models { name: "root" layer_names: "input" @@ -292,6 +329,7 @@ sub_models { layer_names: "__huber_cost_0__" layer_names: "__multi_binary_label_cross_entropy_0__" layer_names: "__sum_cost_0__" + layer_names: "__nce_layer_0__" input_layer_names: "input" input_layer_names: "labels" input_layer_names: "crf_label" @@ -314,6 +352,7 @@ sub_models { output_layer_names: "__huber_cost_0__" output_layer_names: "__multi_binary_label_cross_entropy_0__" output_layer_names: "__sum_cost_0__" + output_layer_names: "__nce_layer_0__" is_recurrent_layer_group: false } diff --git a/python/paddle/trainer_config_helpers/tests/configs/protostr/test_cost_layers_with_weight.protostr b/python/paddle/trainer_config_helpers/tests/configs/protostr/test_cost_layers_with_weight.protostr index 3244181a63..b7d74f85ab 100644 --- a/python/paddle/trainer_config_helpers/tests/configs/protostr/test_cost_layers_with_weight.protostr +++ b/python/paddle/trainer_config_helpers/tests/configs/protostr/test_cost_layers_with_weight.protostr @@ -60,6 +60,31 @@ layers { } coeff: 1.0 } +layers { + name: "multi_class_label" + type: "data" + size: 500 + active_type: "" +} +layers { + name: "__nce_layer_0__" + type: "nce" + size: 1 + active_type: "sigmoid" + inputs { + input_layer_name: "__fc_layer_0__" + input_parameter_name: "___nce_layer_0__.w0" + } + inputs { + input_layer_name: "multi_class_label" + } + inputs { + input_layer_name: "weight" + } + bias_parameter_name: "___nce_layer_0__.wbias" + num_classes: 500 + num_neg_samples: 10 +} parameters { name: "___fc_layer_0__.w0" size: 3000 @@ -80,9 +105,30 @@ parameters { initial_strategy: 0 initial_smart: false } +parameters { + name: "___nce_layer_0__.w0" + size: 5000 + initial_mean: 0.0 + initial_std: 0.04472135955 + dims: 500 + dims: 10 + initial_strategy: 0 + initial_smart: true +} +parameters { + name: "___nce_layer_0__.wbias" + size: 500 + initial_mean: 0.0 + initial_std: 0.0 + dims: 1 + dims: 500 + initial_strategy: 0 + initial_smart: false +} input_layer_names: "input" input_layer_names: "label" input_layer_names: "weight" +input_layer_names: "multi_class_label" output_layer_names: "__cost_0__" output_layer_names: "__mse_cost_0__" evaluators { @@ -100,9 +146,12 @@ sub_models { layer_names: "__fc_layer_0__" layer_names: "__cost_0__" layer_names: "__mse_cost_0__" + layer_names: "multi_class_label" + layer_names: "__nce_layer_0__" input_layer_names: "input" input_layer_names: "label" input_layer_names: "weight" + input_layer_names: "multi_class_label" output_layer_names: "__cost_0__" output_layer_names: "__mse_cost_0__" evaluator_names: "classification_error_evaluator" diff --git a/python/paddle/trainer_config_helpers/tests/configs/test_cost_layers.py b/python/paddle/trainer_config_helpers/tests/configs/test_cost_layers.py index 18ff6b48c4..d2a3b702a1 100644 --- a/python/paddle/trainer_config_helpers/tests/configs/test_cost_layers.py +++ b/python/paddle/trainer_config_helpers/tests/configs/test_cost_layers.py @@ -40,4 +40,6 @@ outputs( name='huber_label', size=1)), multi_binary_label_cross_entropy( input=probs, label=xe_label), - sum_cost(input=hidden)) + sum_cost(input=hidden), + nce_layer( + input=hidden, label=labels)) diff --git a/python/paddle/trainer_config_helpers/tests/configs/test_cost_layers_with_weight.py b/python/paddle/trainer_config_helpers/tests/configs/test_cost_layers_with_weight.py index 1c0aa7f9b9..c369062930 100644 --- a/python/paddle/trainer_config_helpers/tests/configs/test_cost_layers_with_weight.py +++ b/python/paddle/trainer_config_helpers/tests/configs/test_cost_layers_with_weight.py @@ -11,4 +11,9 @@ outputs( classification_cost( input=fc, label=lbl, weight=wt), mse_cost( - input=fc, label=lbl, weight=wt)) + input=fc, label=lbl, weight=wt), + nce_layer( + input=fc, + label=data_layer( + name='multi_class_label', size=500), + weight=wt)) From bc02d1aa96acce39522dc45b25bab84d420ed27a Mon Sep 17 00:00:00 2001 From: Luo Tao Date: Mon, 15 May 2017 14:11:48 +0800 Subject: [PATCH 86/88] follow comments --- python/paddle/v2/inference.py | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/python/paddle/v2/inference.py b/python/paddle/v2/inference.py index 2c1ae08f46..139339902e 100644 --- a/python/paddle/v2/inference.py +++ b/python/paddle/v2/inference.py @@ -88,7 +88,7 @@ def infer(output_layer, parameters, input, feeding=None, field='value'): Infer a neural network by given neural network output and parameters. The user should pass either a batch of input data or reader method. - Example usages: + Example usage for sinlge output_layer: .. code-block:: python @@ -97,7 +97,7 @@ def infer(output_layer, parameters, input, feeding=None, field='value'): input=SomeData) print result - If there are multiple outout_layers and fields, the examples usages: + Example usage for multiple outout_layers and fields: .. code-block:: python @@ -107,9 +107,6 @@ def infer(output_layer, parameters, input, feeding=None, field='value'): field=[id, value]]) print result - The result order is prediction1.id, prediction2.id, prediction1.value, - prediction2.value. - :param output_layer: output of the neural network that would be inferred :type output_layer: paddle.v2.config_base.Layer or a list of paddle.v2.config_base.Layer @@ -126,7 +123,9 @@ def infer(output_layer, parameters, input, feeding=None, field='value'): Note that `prob` only used when output_layer is beam_search or max_id. :type field: str - :return: a numpy array + :return: The prediction result. If there are multiple outout_layers and fields, + the return order is outout_layer1.field1, outout_layer2.field1, ..., + outout_layer1.field2, outout_layer2.field2 ... :rtype: numpy.ndarray """ From 87cb840ce981145ad5fbdbb34569546bbbf725d9 Mon Sep 17 00:00:00 2001 From: Peng Li Date: Mon, 15 May 2017 16:50:53 +0800 Subject: [PATCH 87/88] add coeff parameter to the helpers of mse_cost, crf_layer and smooth_l1_cost --- python/paddle/trainer_config_helpers/layers.py | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/python/paddle/trainer_config_helpers/layers.py b/python/paddle/trainer_config_helpers/layers.py index 2af7c9c9c4..3b6f0270de 100755 --- a/python/paddle/trainer_config_helpers/layers.py +++ b/python/paddle/trainer_config_helpers/layers.py @@ -3765,7 +3765,7 @@ def __cost_input__(input, label, weight=None): @wrap_name_default() @layer_support() -def mse_cost(input, label, weight=None, name=None, layer_attr=None): +def mse_cost(input, label, weight=None, name=None, coeff=1.0, layer_attr=None): """ mean squared error cost: @@ -3782,6 +3782,8 @@ def mse_cost(input, label, weight=None, name=None, layer_attr=None): :param weight: The weight affects the cost, namely the scale of cost. It is an optional argument. :type weight: LayerOutput + :param coeff: The coefficient affects the gradient in the backward. + :type coeff: float :param layer_attr: layer's extra attribute. :type layer_attr: ExtraLayerAttribute :return: LayerOutput object. @@ -3793,6 +3795,7 @@ def mse_cost(input, label, weight=None, name=None, layer_attr=None): inputs=ipts, type="square_error", name=name, + coeff=coeff, **ExtraLayerAttribute.to_kwargs(layer_attr)) return LayerOutput(name, LayerType.COST, parents=parents, size=1) @@ -4798,6 +4801,7 @@ def crf_layer(input, weight=None, param_attr=None, name=None, + coeff=1.0, layer_attr=None): """ A layer for calculating the cost of sequential conditional random @@ -4824,6 +4828,8 @@ def crf_layer(input, :type param_attr: ParameterAttribute :param name: The name of this layers. It is not necessary. :type name: None|basestring + :param coeff: The coefficient affects the gradient in the backward. + :type coeff: float :param layer_attr: Extra Layer config. :type layer_attr: ExtraLayerAttribute|None :return: LayerOutput object. @@ -4848,6 +4854,7 @@ def crf_layer(input, type=LayerType.CRF_LAYER, size=size, inputs=ipts, + coeff=coeff, **ExtraLayerAttribute.to_kwargs(layer_attr)) parents = [input, label] if weight is not None: @@ -5379,7 +5386,7 @@ def multi_binary_label_cross_entropy(input, @wrap_name_default() @layer_support() -def smooth_l1_cost(input, label, name=None, layer_attr=None): +def smooth_l1_cost(input, label, name=None, coeff=1.0, layer_attr=None): """ This is a L1 loss but more smooth. It requires that the size of input and label are equal. The formula is as follows, @@ -5408,6 +5415,8 @@ def smooth_l1_cost(input, label, name=None, layer_attr=None): :type input: LayerOutput :param name: The name of this layers. It is not necessary. :type name: None|basestring + :param coeff: The coefficient affects the gradient in the backward. + :type coeff: float :param layer_attr: Extra Layer Attribute. :type layer_attr: ExtraLayerAttribute :return: LayerOutput object. @@ -5421,6 +5430,7 @@ def smooth_l1_cost(input, label, name=None, layer_attr=None): name=name, type=LayerType.SMOOTH_L1, inputs=[input.name, label.name], + coeff=coeff, **ExtraLayerAttribute.to_kwargs(layer_attr)) return LayerOutput( name, LayerType.SMOOTH_L1, parents=[input, label], size=1) From bade3e976b5f60982fc3aa08ee7812a53000d857 Mon Sep 17 00:00:00 2001 From: Luo Tao Date: Mon, 15 May 2017 17:08:51 +0800 Subject: [PATCH 88/88] add appointments for contributing codes in document --- doc/howto/dev/contribute_to_paddle_cn.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/doc/howto/dev/contribute_to_paddle_cn.md b/doc/howto/dev/contribute_to_paddle_cn.md index 775938612e..a48b143c76 100644 --- a/doc/howto/dev/contribute_to_paddle_cn.md +++ b/doc/howto/dev/contribute_to_paddle_cn.md @@ -7,6 +7,7 @@ - 确保编译器选项 `WITH_STYLE_CHECK` 已打开，并且编译能通过代码样式检查。 - 所有代码必须具有单元测试。 - 通过所有单元测试。 +- 请遵守[提交代码的一些约定](#提交代码的一些约定)。 以下教程将指导您提交代码。 ## [Fork](https://help.github.com/articles/fork-a-repo/) @@ -217,3 +218,22 @@ upstream ``` 至此，我们就完成了一次代码贡献的过程。 + +## 提交代码的一些约定 + +为了使评审人在评审代码时更好地专注于代码本身，请您每次提交代码时，遵守以下约定： +1. 请保证Travis-CI 中单元测试能顺利通过。如果没过，说明提交的代码存在问题，评审人一般不做评审。 +2. 提交PUll Request前： + - 请注意commit的数量： + - 原因：如果仅仅修改一个文件但提交了十几个commit，每个commit只做了少量的修改，这会给评审人带来很大困扰。评审人需要逐一查看每个commit才能知道做了哪些修改，且不排除commit之间的修改存在相互覆盖的情况。 + - 建议：每次提交时，保持尽量少的commit，可以通过`git commit --amend`补充上次的commit。对已经Push到远程仓库的多个commit，可以参考[squash commits after push](http://stackoverflow.com/questions/5667884/how-to-squash-commits-in-git-after-they-have-been-pushed)。 + - 请注意每个commit的名称：应能反映当前commit的内容，不能太随意。 +3. 如果解决了某个Issue的问题，请在该PUll Request的**第一个**评论框中加上：`fix #issue_number`，这样当该PUll Request被合并后，会自动关闭对应的Issue。关键词包括：close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved，请选择合适的词汇。详细可参考[Closing issues via commit messages](https://help.github.com/articles/closing-issues-via-commit-messages)。 + +此外，在回复评审人意见时，请您遵守以下约定： +1. 评审人的每个意见都必须回复（这是开源社区的基本礼貌，别人帮了忙，应该说谢谢）： + - 对评审意见同意且按其修改完的，给个简单的`Done`即可； + - 对评审意见不同意的，请给出您自己的反驳理由。 +2. 如果评审意见比较多： + - 请给出总体的修改情况。 + - 请采用[start a review](https://help.github.com/articles/reviewing-proposed-changes-in-a-pull-request/)进行回复，而非直接回复的方式。原因是每个回复都会发送一封邮件，会造成邮件灾难。