From b63e1c6d8a3e44b68263399f9720165703deccfd Mon Sep 17 00:00:00 2001 From: dongzhihong Date: Mon, 14 Aug 2017 11:49:21 +0800 Subject: [PATCH 01/18] "op name" --- paddle/operators/name_convention.md | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 paddle/operators/name_convention.md diff --git a/paddle/operators/name_convention.md b/paddle/operators/name_convention.md new file mode 100644 index 0000000000..da5bcb7485 --- /dev/null +++ b/paddle/operators/name_convention.md @@ -0,0 +1,11 @@ +## Operator Name Convention + +To make the operator document itself more clear. we recommend operator names observe the listing conventions. + +### Input/Output names + +Variable name is uppercase. e.g. `X`, `Y` + +Tensor name is lowercase. e.g. `tensor` + +if only have one output, use `Out` From e9eee6f78559d6318e554b7b5ab021b271d8ddb6 Mon Sep 17 00:00:00 2001 From: dongzhihong Date: Tue, 15 Aug 2017 09:57:40 +0800 Subject: [PATCH 02/18] "polish words" --- paddle/operators/name_convention.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/paddle/operators/name_convention.md b/paddle/operators/name_convention.md index da5bcb7485..2260bf5660 100644 --- a/paddle/operators/name_convention.md +++ b/paddle/operators/name_convention.md @@ -4,8 +4,12 @@ To make the operator document itself more clear. we recommend operator names obs ### Input/Output names -Variable name is uppercase. e.g. `X`, `Y` +* Variable name is prefer uppercase. e.g. `X`, `Y`. But when the variable is tensor, its name should lowercase. e.g. `matrix`, to discriminate with otherone. -Tensor name is lowercase. e.g. `tensor` +* element wise operator, math operator or similar op, please obey common name convention. if the operator only have one output, use `Out`. -if only have one output, use `Out` +* we prefer more meaningful input/output name. + +### Best Practice +e.g. `rowwise_add`, inputs : `X`, `Y`, outputs : `Out` +e.g. `cosine` , inputs : `X`, `axis`, outputs : `Out` From 21d49744051a6ba0d2f6901cd8db8a242cfcc05a Mon Sep 17 00:00:00 2001 From: dongzhihong Date: Mon, 11 Sep 2017 11:01:29 -0700 Subject: [PATCH 03/18] "fix name" --- paddle/operators/name_convention.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/paddle/operators/name_convention.md b/paddle/operators/name_convention.md index 2260bf5660..280ab8d317 100644 --- a/paddle/operators/name_convention.md +++ b/paddle/operators/name_convention.md @@ -4,7 +4,7 @@ To make the operator document itself more clear. we recommend operator names obs ### Input/Output names -* Variable name is prefer uppercase. e.g. `X`, `Y`. But when the variable is tensor, its name should lowercase. e.g. `matrix`, to discriminate with otherone. +* Variable name is prefer uppercase. e.g. `X`, `Y`. But when the variable is tensor, its name should lowercase. e.g. `matrix`, to discriminate with other one. * element wise operator, math operator or similar op, please obey common name convention. if the operator only have one output, use `Out`. From 2b1450f1512753fa53717334a07b024efa8ffefa Mon Sep 17 00:00:00 2001 From: dongzhihong Date: Mon, 11 Sep 2017 19:28:39 -0700 Subject: [PATCH 04/18] rewrite the document --- paddle/framework/backward.md | 60 ++++++++++++------ paddle/framework/images/duplicate_op2.graffle | Bin 2434 -> 2611 bytes paddle/framework/images/duplicate_op2.png | Bin 24393 -> 24748 bytes 3 files changed, 39 insertions(+), 21 deletions(-) diff --git a/paddle/framework/backward.md b/paddle/framework/backward.md index c762811dfc..0859bf1d9b 100644 --- a/paddle/framework/backward.md +++ b/paddle/framework/backward.md @@ -2,9 +2,20 @@ ## Motivation -In Neural Network, the backpropagation algorithm follows the chain rule, so we need to compound the gradient operators/expressions together with the chain rule. Every forward network needs a backward network to construct the full computation graph, the operator/expression's backward pass will be generated respect to forward pass. +In Neural Network, many model is solved by the the backpropagation algorithm(known as BP) at present. Technically it caculates the gradient of the loss function, then distributed back through the networks. Follows the chain rule, so we need to compound the gradient operators/expressions together with the chain rule. Every forward network needs a backward network to construct the full computation graph, the operator/expression's backward pass will be generated respect to forward pass. -## Backward Operator Registry +## Implementation + +In this design doc, we exported only one API for generating the backward pass. + +```c++ +std::unique_ptr Backward(const OperatorBase& forwardOp, + const std::unordered_set& no_grad_vars); +``` + +The implementation behind it can be divided into two parts. Namely, ** Backward Operator Creating** and **Backward Operator Building**. + +###Backward Operator Registry A backward network is built up with several backward operators. Backward operators take forward operators' inputs outputs, and output gradients and then calculate its input gradients. @@ -25,7 +36,7 @@ REGISTER_OP(mul, MulOp, MulOpMaker, mul_grad, MulOpGrad); `mul_grad` is the type of backward operator, and `MulOpGrad` is its class name. -## Backward Opeartor Creating +###Backward Opeartor Creating Given a certain forward operator, we can get its corresponding backward operator by calling: @@ -43,40 +54,47 @@ The function `BuildGradOp` will sequentially execute following processes: 4. Building backward operator with `inputs`, `outputs` and forward operator's attributes. -## Backward Network Building - -A backward network is a series of backward operators. The main idea of building a backward network is creating backward operators in the inverted sequence and put them together. +###Backward Network Building -In our design, the network itself is also a kind of operator. So the operators contained by a big network may be some small network. - -given a forward network, it generates the backward network. We only care about the Gradients—`OutputGradients`, `InputGradients`. +A backward network is a series of backward operators. The main idea of building a backward network is creating backward operators in the inverted sequence and append them together one by one. There is some corner case need to process specially. 1. Op - when the input forward network is an Op, return its gradient Operator Immediately. + when the input forward network is an Op, return its gradient Operator Immediately. If all of its outputs are in no gradient set, then return a special `NoGradient` operator 2. NetOp - when the input forward network is a NetOp, it needs to call the sub NetOp/Operators backward function recursively. During the process, we need to collect the `OutputGradients` name according to the forward NetOp. + In our design, the network itself is also a kind of operator(**NetOp**). So the operators contained by a big network may be some small network. When the input forward network is a NetOp, it needs to call the sub NetOp/Operators backward function recursively. During the process, we need to collect the `OutputGradients` name according to the forward NetOp. + +3. RnnOp + + RnnOp is a nested stepnet operator. Backward module need to recusively call `Backward` for every stepnet. + +4. Shared Variable **shared variable**. As illustrated in the pictures, two operator's `Output` `Gradient` will overwrite their shared input variable. -

-
+

+
+ +​ pic 1. Shared variable in operators. + +

- 1. Shared variable in operators. +​ Share variable between operators or same input variable used in multiple operators leads to a duplicate gradient variable. As demo show above, we need to rename gradient name recursively and add a generic add operator replace the overwrite links. -

+

+
- Share variable between operators or same input variable used in multiple operators leads to a duplicate gradient variable. As demo show above, we need to rename gradient name recursively and add a generic add operator replace the overwrite links. +​ pic 2. Replace shared variable's gradient with `Add` operator. -

-
+

- 2. Replace shared variable's gradient with `Add` operator. +​ Because our framework find variable accord to its name, we need rename the output links. We add a suffix of number represent its position in clockwise. -

+5. Part of Gradient is Zero. + In the whole graph, there is some case of that one operator's gradient is not needed, but its input's gradient is a dependency link of other operator, we need to fill a same shape gradient matrix in the position. In our implement, we insert a special `fillZeroLike` operator. -​ Then collect the sub graph `OutputGradients`/`InputGradients` as the NetOp's and return it. +Follow these rules above, then collect the sub graph `OutputGradients`/`InputGradients` as the NetOp's and return it. diff --git a/paddle/framework/images/duplicate_op2.graffle b/paddle/framework/images/duplicate_op2.graffle index ede3bca30ae17d5af52505fd94dc2f79b23b57e0..5cec3bc64dbd44dc99e348485969f29bd128ceb1 100644 GIT binary patch literal 2611 zcmV-33e5E%iwFP!000030PS3BQ`<@s{e1ZqTE1-6Zjk3CX_QO2*I*LJ65wJ3o0O}z z26=325t1Xx0g~dsZ;ynnHyA?5V@;(Z;W<6i8g-xP?qR(C$JK?WeUf412L7v6tfN&; z`c5!#{n4w{f-O^UqTCMhE;>lL4)9Yx5 zd%NA927RNt)kFI%n4%S&A-lfYc-`3CN^+wx?-A{5)T+49RAIPiqmiiPLT|bs18E&BU*6emB0%N<5h@s1q zwCA~y2t%Q6ep`>?5FU90G}MC&-yMa)bfO=?-J4Jhhn_UoTepxL8k@FGD5y^?(=SMCHfBj=x0B&#b4v2lo5H&BwI?&=Oyjf>x(v>< z)?9Pi-U-EJrdFod{y||O3>YjaG;AhK9s`SpkR8*%?t{^C|mA`xS|e0 z*jhu@8pRqmUNZP+Y5zpPT-F|jNNgdlqgDc+-Ty7>#ifb%CJ~-eeIQ| zE@iRTt|xoflZuD{k0@gkJM09`^g{Y^*;VIrM=FzdYrm0VlN#498f3|ba!zEynv4ON(Aq=SGa)a}Fq)fH-n}ZGpUu3%3%+0H{FR}HuIJ^4&YI_TpIKoUkwVV$&9}-UouT3n$vw?X za`b4)utpLWaJd;P-Y29t7K8jNtUh;{_L_LFrtcD=K26_c11MQ*Nrz%<=0g229^$_6 zBRBcanT#+oNZ$#ZK&U1Iv6V0VAsB3Y=JkiDKlEL=I%5$=GDg~t^ghWL*6MvZmA~Hr z)=~BW;NSb991z3kk3uPZb#vxTC0rBKKl<`&A|Mxg(G#LG=We%fiRXpKzEc%tBZ*fh-m>t_3MVdh^L|O`X3$jLF|sk zvGj++0hUE8E6Kw~1ZjO|h_p{4e5v*SZqw60-yF3&e`2_2Hw_D3I_0N8Emck^zR}zw zxhUrmg7qcs-A#M;lJ=Igx1_zNr#2)CA%ylyqUX&zXceC4`>&s?t zyO)0yqQ+e2Z@Xlb&6Gv%psXD!2F)tu2`$HHR*BITp+OVZ(BNSJALVCm9(Cv3gBX@h z5wQ(0H$pifM6EJRXSR(kLbzd2Vw0QFbY#+|ZWGH!2DeRSTO5Qf4jDJY1dN$t3lpshggK`zc1@OdL=;%uN>&BSBlJ3+hJIk ztNBzwp6JP0bAC=z**!TD&UHa>RzS%x;I8>pvtaWj>zy&WaJ@#AP@N*D?jRl)&}ovM zfA+f;7#Q%O?n-%+yDpjUS(pz!miaK|z;dvVa>kH>O>C#k2N8i`FlG@TB+8q}3uCMU zSy9X_f(_e9n2+fuA*SL!$}JPKMq$&|k%fTw2q%V(=q>XxMr_7xP7T}U_LBLQ%=aTR zpY>qoYfg2sNt6pvmFP0ad71HqiS8qlQTE_u)T&EHm2y>`x7^V2YBJ~V;?>Jc>sc}_ zo@QF@!Qke;{gVfH4uW+HQ>q)vcPs>Mgno`9QLPU#;}Z_~VjxN_ZWt=aX1@kYu^N_Af!;%dC{_*t6YRlo2*z8DNO&Qf zOEaCwjdMG(Ir6gLO(2dSi(!;9jLeGgQ4lAa)urgPes{iVMoU&+vhtFZe+*V$)|48O zRQnM%B}N}zQ!<%ua%v-!a)ZGV2(dY3ROgr)+~C+ks?^<77E%Q$gESY7sLlvgq)rGd zr^tGtnWl~nkRf27hQ*m-E{jIXqR|grG%~DZx%{c9jX#Fk9=%bBIfV08^G+Z$bw)X? zoeX0ONXmfV-^My694?gF*d+G<=VqRX2%))nl`YIxi%5o#kbHhVdq#3v63LSl^6w%N z^o@uFQy>l7LW)T^BobIv5p1Lr&JaX7rft~b%kW4H+-X#`3;y^8=8n=RRpAh~zs2@u{Y&IO{Z{0o_QKFJ zPqX#wOmpc}+?&uH4%vsGIN`~#qi3mkuBc-Ndl;FKY&24{_-|d;{G6>>|GQK*@ zr*elq>5p=e$ZP;I+s3dtB9stT8ZeQ6fP(R5FmV^c7o81MJa6)zsG<>C)>5W3Qj3)5SF z26u)AV_CR!#Y&s-hR&k}u*#8}U1q<#8N_jL@lJ%f5W1fOUwELm>z7QfI*=0#6qnZwwAqe74|d*q?@uw3|K9XI?Cu0@_SfcLyWQ<^ zdwg(=419L_)^kQ%@cZE47=IX_?VNaRXy4j4dOh;?9I)ZRVaZg)Qd=Jl0#x%K95 z_4nj3{+j7gd~#R+hY$;dg*?8d`|8f~ETl0p(GzkoEkrm+>Tl$&(%<9P464lHNUEdy zvc^__hWB!$_SL@Si|GZEZ_#~`Q&LyLdVq; zKJK}3m6y^}^@ZRemADXLoT#DBfu|u?`n3-WL|lomT9k$F$OpqGqA%&}Y3Foc;%X0rN?45<2z4BNLm*#AkVI2T)G>)~z1FwBQ_ zxip7Bnqwq??}w4vVzdaU`BDiLqZdk&Sv6(eel+#6b+Q707j!1>YcAGTvUL#EJUcjD z*Ar(uQh9eNGNH5-$lR`&M{-o)u7`ZDM*d4Dsylmg$cgZ#mG{tC!K`tO`hCOdM|m2m VP)2r5zG-D+<$oBADfa+I007jCD%=17 literal 2434 zcmV-|34Qh-iwFP!000030PS0CQ`234qrC$1GRZk|M9#3Ffj2#sw*g@!Nvb?7E-y&w)e z^|>dv*IsO0vAgY-C;2CKNbgv55c=$PtErwsArB=RuxJ}?HK(`JD$qCgsDviZ)0StI zXcSR(^I|KO5xnv?NZ1h9As<9yJZc=n-J6K^`vJ4IniEJ4jV-r9bP%7|mSvF~WUD!C zIu&6oMa%zl9LrhPG8(hyw0wtpSE|dC7#ObNO(q8>Cc!$*t;+d7zDax;a(KgyQdR93_x2-};;-qf*% z&6fuJ+S)%6{8XeGfeW(J_|lE3A4o!8?)c;0qcoCL;(J+7crlh zW^WKN7OI=eV9el}p#ITgw<8J@lAqzdcmO8Y^RRM21hC$SMt)D;nihh;7?b2;IqE`) zkz5?4F1nsTbT@$*MgjrZQCeGFESx|r8z8D(9)(nnBO3HRdg_7XgP~+$pL*cuXl*Td z*_b$5&+8+t<@@&!5$y7wL+lOBfvM0$Ah=~=AVrht%8#%r($Qb&NL-dIV}f|U*Qr4V zc|)B-Pg$HeTNz#NP3L&Lh>Fcm}D=w~M=^A{i}yrQEO9epo4`hKAk zavF|{UDre=b{$v8ikOTvkg^@qG_i$^UxTHL8kRcb;R2G%x07oK7aK~k$NfGSZ*{@P z$@4{mJP|RMxs|XSdD$=$$U&xK8M z%B=uM?T}uwpdE>PGkqGw$yU;0f>qr073lHgUBHZwDG%m~gz5yj;JzFd(CNNmAsDEe9#&NIO{j)EnQCme zfpwU59m7C2umxVoJT?I|%w-00Ovkkh#Y4I|4~6L`nP4V^e}(RCMy z%f!`Vutnen{ zemsJepx-7 zUg{>9s!r&D#Y-H^Kez_qE?Avp(Gu`~GdAjq;t}~-h?!1wDsd%#@Q}KC@rO1FQsfc6 zuCchJ8x2?CC-CpGo2Z?Ddn{f~jC0Oz9&C1=!GW}5Ea#hCE9BR?WQTbhC#uu~R}=4e z*v_Q_)NHM0*bW=fh!*x&YgTs>e2|E_7Y_29z*AX#AuGa!D#t>7I~=lbkc&j79gykn zrt2W06JnG)OzJ;EKzSoZd?|d<*+9grdEQfXws&+8GWi&M4BpcjGA2ugKl6%r!sq=- ziq`zJ7Izcb8o@7-{?ah&(^}Go*kkfqj)Y{(zszyOSttwYtxt`bqQO{}Zd|d_4vf%M zyaZM`@_d&$>g)(9#q~QH<+7aepJmgg`+=ac=+KjU0~eh)BdfHBeG&1`BBTMx&BKz( zRXg&8ueiLXqn(!g?C-w~-d|wE{<9N&IM^30Yjr*X=JjoGv-@VT_=l`7zeYWZPZqU*0Qk>zIq^K8q!#pG!k;KY(yLfITn07yTy;^nogBjoHCU`$!cu%XBd}LwcY-l1Jf0w z=V-pjsmVYbhfMkOoLJx#KA4Zg;O=2(KNUC2CK3X4sVu^TkGot}c_~eHECdf}jcXdo zL=2Ulmxf$vHtrG;S5jF`$~5%YBkP`p{NtEqQTT$lKa1I3D0#FgDw9}Rh)~hviWebVi@Jay4+_DfzO;YLlGn0RjX-cO|Gws z(F;qG*=owH{B#^-^JE1819V9rDlV?CWb+`Zd3SKSPCzgBney&ZWTNY(KxQ&yp0YuK zyB?G7oX-WW?!wI==QJ2s#$oOXW{qpq@9|Y1SfZtyGA9Pozp_e}Z0H0>8 A_y7O^ diff --git a/paddle/framework/images/duplicate_op2.png b/paddle/framework/images/duplicate_op2.png index 4e872dc2caf3b0cbd0d5176f11a14801b538dc86..21cdd5cabf1b5203e1435a75b57770d2f702fa92 100644 GIT binary patch literal 24748 zcmZ_01ymeimnet_cXw-oySoz_C%6Q6cXx*X!6A5XcMVR^;1GhlySr><{@HnN_MGL= zO?TD(?kCr!h*DORK|v%$gn)oRk^Lm83IPEb0lfO)p@1tfhlzKfgNQm!Q_yE8kD>W@wEd_ag69+qHBU1-sGiFaaM*tcELeP^R_-JS5 zYDDg7XKU}m?rxgSlIaZ_*hukS=iZ`00<@*FMC%bPbPa8 zs{bJQfAL6~xtKUxIl5Xo*pvUmYh>);<|;%<`ER2C`T5T}U9HUj&rJ3%|Gh0>gDn5< zu&^<+viuKjV5s1~tNaoUc8<-?EgLb{~zN2i%ZGb$_&``e~o7Q@6rGFwf}|}Wcj!8|LY+BbISj96*y*L zL_wDSA(=2@?Kb-=1cWGrtfZKlC*-LC@ek$9lCtp1&$9RLuh z9kmX;V<_W6je(EqKiKpG0At`6)PIw=qZ?|drU_{g{hM85%j4e~=E5Kv65CIDmjD2W z2Fz|~UwdZVT_~GWu3j2puW zW$o$dxiDAT;;;^2bsJ5l`#17W#owMFXg^0!SD53)>bxQ`gKc!`@&hSj3f9 zCupi>kX-6qwf?dO7>biNIG7fd0v$F8y+U`m)o4O0@EJTYuw2w66t#m41A|knnnjAg| z9cdE;P8O9q3#MzCWrKG#7-4HImp>X!VPFg3^tn4}cRS`+Ylxnw`*jQW1Y*8!tzyPr1UKSmoRFHxUr=8#flSyiDRO zWg1mFKiyBAQ9=3kYN1E$7875~w@drT@F9=U4cG0D9cRp$uSv7OA3lu3#+ zdcrUK%hk#fIV$$rT>}AJu7;Ac3O&(8?M&%gPB(gqheuJYLvvNSEha;8m*rV}k-V$a z0Kd@~O-mRTNXORFa(mxc;&Ja))O7I!yE*z(po%9uT2gvxOAhv=5}i6-b9U5O@+ZrN&*P{Sz?UZl;bB8lwj~|${|K~m0p|k{n=^*6>pS8kf>mBrFH`;uVWyU7C*Ly zAN+n-qVNfA6I75rhPtXH0u71GiVpg)xq$Hj2Klyio#X0AO@S)V2CGs}f(f)usn2EP@%A$@-RdeylBTZIo2(Qc}{HHHMaFVAf1O)}v%hg7Qoxfre z5B{#)!MAh+rD)5F?8zsFYO39uD0tW-G$i_0*1YAw^MjRDMIv3~i2q49K!h+g-%W!q z-^1wdRaIC=Yz*2+sYLrGzosF@kFP+|PFN_ZTaAE|dzan}G zjQj3!|8Xd+W8)M2M{hhvEyvBiuoQZAXD^}$Nr$vdQ5Qr<13*6Hw2a8e$Xu14ZjbRu zNU|O6W%dH0{GRUyoYs^_5^ig4_5axo2A9!TD?mX4!U2auNL|vp1BWOY%Wb#RTIgP_ zRseqiuUHT+r)8H#gNcnDI#2_s6q>AMNIq`cg+MBocruY3VL$H=kroN<|MXfr7gH<5 zXE{e~2o~P%gmgvmKgxl0Baa(4oziCvR;$4X2i<^jX4Y@txEZCCil+Xi-ZoUCl*_Es zco2=l@SW4@N0wUZ4)Z?-T%U2gtp*hFTBi?#M&&{1{0|#kOw4-A=^x*@Z20r}tHlHU zEiQnY%N}C^ay1oAh?jgN6Gev}tGDmTNB(^7&3&~)C@*ln?Z^}fEp zKVNCPZgbh+k^JdE4)X)u2-El!w-1&8QsUo4C%KRob9e+|p9As9G(qih=lC_9T&;A$4#p=JnwXbsF}cVFXksU`1Q(R|#J*iHWn=Oa|{K4;Si8 zWRqx^zI3!^IlsTXk_fnyuYhRc(T5R8We;dCn{^4ux>R5Q9VTU{7V$0n`0%fo>t5Zw?HeB7>-<iE~^&_Ns0%JHi+kGvsa31v!uuqgbNTvJ_JDZz9 z(trU*Es+Zj5momUk7B4g;dcAzl9_tcn9svJ+j?%fYD~$o3HiOTAtx~q!}Eh?zIfP_ z8zx0Rd9~}|O!zi$w)n`nDiYNETWGnI!Mud@#x(Skfk?S#Ih>rW4%R&32mdklm$4{F+^;o z>~o3y?k=aas28xGEMcxVALzcknUCIvx+MebAkyQxnGBU>D#5!k8pGMf2X0vLBTK6 z_zOC1g+4mJgkmTo1JoGEbB)b=kvd3nP|ze5ldp^vmB%$WJ`gI&8KGL z9?j8Tu)OK-}T; zU;dkipgcVtk?&nm^$$q*`dx(xa-pZK&1mbjnLG1Uy20p_$m_p|9$B;yRzl+!ggiF; zVEM8g@F52P`=m;&I)vm)ES+sLduk)D@I{!@i)oh$U)Sc;yzvAbhaMx-cy%>}LGza) z`qumN<0AnMIF&Jj_>l+$>KB1if#6F5hVp-e!S%fQhi}6wG2i+g_nrnQ6kOU znp!8YcrIH6iUy|3z&UNyVY(GKg&yb3pxfVXPyQ=`Lo1VNGjVUpo5 zvh;4oiJm&tZ4CoA0|*A47+WD`QWxkLNysqqn;6W|1z3@m5US?*PM?-#0=&{ z6!42tr}R`zEm?H9G|O9u{?6Iu1%ro4K(1lFzP^ldgL8^ks8S60rX|S!;u|aYI@W?5_SzJ=QXjFzm0B6pd z?Ec8Y3ZMGQrn(xtJpD~1`42SFdTlXWMpsu?2D@3kG_UjMv1_`1U-%aruoJ0t4;bK! zN|;fpMMt#}vi_oW8=jH`4m(6ftJs;VK_1uSN|8>?UV1>>P#-rpH=Vyi+LUh#eiaP{ zP9FrHaLhjTDiJxz^hN!6b^#U}>yxfn7%L1`0gcfr6A-G*%MJP`2g8E?jsrHt1mDt6 zdtwA$3y6(I*oCWo+FkikJ5tJ7$mF} ze*=*L5-^`O#E@IO##sJ&-31FwXdx!f!KC^Q-HF)eyXeTsZ~NuDUX;fbqW6fiYsUVy zn1$K~sV?kLxPa?a4-pqbTBT3hN+UHSPk~3L@cD02d<~UaFD{nT55l+vtJ&hF0>qoz zUp;FJqC-KQrx4#_(j|daQZrV|tvsNi*eabH$1O0UQ^P=iGGk+J>HHoXzZ2`A!4LDR zX4DZ4aRmu*W^hH&N?4(qDFNy+7{`ruyW#jPhOZ5d5%05rIa(Glw&uaFtnL!>n5NRc zn8XkkW@QhH^`;}i5+As*Q9=T@)A$1uDXG&*D2s zB!Q|OkaUe|{7LEco4NTx*b^P=YQ$b!?;D0Fts0yd4*bcYq~CYhI(wJh676N$kSR!~ zsj&RiPh|MPpw>$!F%s5~ARin@!*iI;Eo1~>oyp{qag%}9G(OztW2U3>-UQnmy5JoP-l^z ziT?{P)>Hy86i1&74C!0vs$ll z#zAx}Rf;2gk)QZhT5v_nK}6JnQ@xA%i)~gPJar^QgK983A7g*e>5UatG&XKu=(U5G z>{}a&hIXZ)SB`F3Tp}&u@)jaqD&;kB{u(QnM7j*?Y)~3NEF(5>j@Lk&-9|ou;v(m`{a0Kx}OSm!6NGEkd zqBCLchcZh)r%)ql1k#01!93i>?c0K%&q7Yrq8hpfSm{Pb%w_F^00$at92wlFmuSR7 zm!^(kg`n|oN%oPg&ZFj3&Doch8F~8D zdY2!vNnt)aP!FtL75TP2NdH0bLRW7V*?oBr@ysPiN5j zlgbqK$YE{ss$2nn<*zn(8MR;RZsGqTCFFK9(5)TwfZP=tI_=2ivb{nx3@R9P>A}37 zzd&-#s0f^b7`1@wu{TCgPy$#p)+k_+WpG(?5!**Vj~?@dNSuX^IT!W_?Vm4c*KRI$;soJ^2oK#dcK6{(sl)sr zb6D{!>g|4WrM0s1w{%D&`u@H?MvJ`?AfuG%eBLwKRu@6D%OvmO&dWb#7hL`u6;ivw z9LMA>DafaG^K1 zw>a^(a>=c~aKcfq&@mh2j1Pi;MeWQ0(T+CVbg}*g-s==}Qd@AII4BGYOzahxd3!W^ zy`s4AU>>UxhKM7Mt-mA5D*w3f#Rk!t)B|8AIw#y*j)d^LWZE)Ig6~a#oJv-g3TffM z#^E!!QqppK4C80n388GIG1vCrg7>^0@>w*$J*-en+lHGUL zDD)Cj6|Woti;1kQX(}+dMN{ML<#AwppOTVtB3Hzr2A{da1(2x|u8ZVp&MUbx$1*Ud zA660Aq=qr<_oe{LB<)Y%oaI`(gI1#@U3o|bF{Sjr>!BKM8dT$Rsy?lNzCpiwlN8&Q$p7vnC)* z452_uhf}JP`=%pB*J(?4{ zD>9J+0u$;1al%=$-*jOd!Td`wuyB%FX~fW=57`gqNy2;VCe3ofZE4UxnGz#*vl%Is ztU?pYL`yMy^55#`?G>?7DcK`&K-t++XS6_mL0>P#qZv-)@5gg7F;SVs&npR8}F5bf5F{E@7z z1N(ozLnH6}-|jPcxnghzVM~F*zDm6*?(fzU3R+p$^ub;kUiuj`Z!LYcySzx!oJhHk z_g9t7;{&9~Q=yF@G$xbSIgjP3ink6ZXCK%7ZTR(nmXV6^fjtICRfdlw ztHA+x`tfI(Je(UQI^9|`g;E|*>+8wXQO7tYDEN?_REB$L2Iu37d44qpNGkSbVC&@= zIy%T-4{ZO!iWSZ12s5>0N8Fxf^_dXV+-Z;Qf#1InfGqF^Ex>kLltehPote|sZmQO4 zkeml;t$YtyK<=3r7vK?SUn;87zY&cV1<7;UL`Ubf1O_42kEuXb85!UP@A$!rw7$dT z=ax^oKChy=C;{(Z-z|V#_CDZgVu#%R#=DV~c=CvV@$KH6d-hX<*k25`F$V5K0;qrB zO_uri@ptHYQ+2&$2tR&bM+K7BjyO>E=L8pmS?zyfiwbM}r(r-<+Afy`slP8~#xeuS zZCaXIt(JVP-k<^qx};w}*eWER?873NP*rq#-||b1s`Q7SueDwxl5WB&IdNRxON7Hd z=3qE7L7k0*Cv#QOmRwD1OhX(7RsKR&H$|j0p}jkuCwCuHT$fIN@XysE;!&^I>2Q6# z?2lkELs<6EyZ`jv^!e`Cc{JdC)$esr|C8fcqm|G#96THS>x|FtXok-EM7}ig+f|Y1 z__|*=FdYw{%lp|O7h?R*udtYjz?#V%zhPu%ldVP~9`hm#rP9w#{QjbSPtOQ*hb|?X zIm6NRtKQt^C*9|3QtRzFZ_5ww^X#9BkdJS_Blz@_&Dhc5e8+6mboHK;w(GX>P3lrIj?b|Q_;%(LFab2bWABKy3r<9fA@f0H!zO?IxUCOCDW^Vxjy zisM3#;rA~Gho+4-&)WoczrjmmvCkd8(k0hKF+o~d=~KOnwVTdW{796PYd0rN6b}9r zlD&V|Z6m&YKr{X+Qs0Qj@`c5~iL~94iX6SVwWQ?FF_qiAYnH_>P09O%`-76(|} zjiu&S@BCo@Jo+2v0qhr8jXxC#lII%#*^>PTScz1n)d@LMG8Jb2*Mo6Nx}jFj1pAOTrxF+bbUI zlE#<;(qBpza7Q%Yw(VbQId^xw)oYzhpt@aKCCO7VG<@>&;V%U5j$XJ)gr7atE~;9B z3|8A~q$q{MBV_JQTSi3oCBL>g?>?IRQku~cNI;I8K8s1KQOas%nU55+|K!O0lJuNX zil-D30$aHf-D`%`dN=qwzd;&-knDPQs7YW-Syj^c5X(_XyZx(-nBhe(!bM0XsVa* zpvT0l6JK^&rx3qCJF&Dw7}D0xSiFwXD10x}X(E1@_giT`$lySmu0yP5et9cVWSUlf zx~Y!8AW{?LfYZif4uyq#5^MRWL}@<#e4|v5vN;UT&J| zeS0;E+~xI~tC5h8iKDgT3iV%S+ifF*e5C?WC=q zAOVMUDI#{yrA9g+mPCZRzg0JB6loM!O3S=WAr&f&+#KYhy^7f zo6y4cORhNE#ZR}_TmENRgYP6KUo%UTsG4ot+RcueN@>e@LM3`X1Q=)+Qkz~}UrU$3 zVTbtGFv~MV=?6aF8)ON;)=Gb}Pew$TU~-3KrM|qC{nBj9Z6nFYfo9md;4&~8V_bAV zA1q18IgF&`D3TZONY+)eQmGvz^6}4j^7QlYw2v`&Mz6)VZ74SDLu^IoIGbmi^CCJs zJ~HzR`lmL@sIe_#V(G=e3h&#j11sZsvRT=0aFs6&C8N0_LT3pC$)QB7yq_^09sA5q ziOylviNC5`?4Z*2SsR(6RF8_l!?FPA`!pdDgM zGD-`G3%(v08(PbbuXElou$}v#uensh8D4!EJX#MpHyow2d#Zy)Qp(|aHvP^io-5Ub z*}rk18C*Fn2Za*UDwSbKiOwJ1qfEoAg@1IliwUz5lG1nf`Ps1D6CB7}_TiO1;(MbD z)Qa6QcCXID(`_1tC7%ArxkwPVG8_$9T--UPv+#FXE^R47m+b+Rp-P!_>`J|s?;~AU~2^5u#cj3m*{?7QvL|6nwZE zy+yKM*4J*6Arz-lq&kY;HZ+p<7{%Rdyc+eGloel+^;SrRphW)AsUmzli-bhNCbu{# zHsjI6f%^s)ZXO%L$T(#Bfv*VCh`KHM{f+SLa3+JKdQX#W7L$~0BZLM;K?ZgVD%Rg+ zCj%cc2Cyrcjk;d?24t?zMP?+#ovYRzK?>G@6UULsaIoY$mfd8l0w2PQj#b;Yfbw** z=7M#?YHvFp4ExGzhc@I&UYwkh-oT*4xxG^+lPeqxBN1gR@<2{B#(9(m-R4p>lC6vX z{_jL@Ccwrlvc=R{uv4|Mvw?$XKF1ni)3Ut=b*hE(_y#t|^T+-f{@0U_&t0!Asm%87 z`xDo9B$`8ikB*u?4k@kh91@_5G^7V-U`%3@h9Nx$q3jB7P5S>S=c~-+P5aG%TC8x~ zr`z)7*<#U)fZ>5N?=Gc+xo4NqWd{GTPY#Kg>B*OX&KxgfA3{4Ms`h{ee0?LYQD=PT zcFaQL+^!<^GK~O#u*4x+SoqrlhIm?zkR$h~#TBYJT>XWeALbqTGrtF-^=v!5AW?&b z5(stcZ1pgW24!MhG5pr2KZ~5PKYo5i!tmGcny=2rOmt7 z{BvbkQ^o3=NZ@j1oF^dL77M0 z*}OWRhx8FPQmVTUu4)dB=lEYQ0s>_oo(ASND2fOyDqCP7Fv~(GmdpqJ4^w;~vliV; z8T!H9iAw%rU-NRt4&%AqanN2C;EfuBpqB16#ZOKrnht-7<@e@M!+f+_njRgr0WBBF zrNxor@bb7F%>lpD(Mxet*SE)ikLRmm`pS+af%^Jzu1GhPwNXa3l<2k}i1_5SV z;0o7mZyYhv%*s%!FyV2_cMXK#njzJnYtcRV+!PO^t8FeabI}nF^=>b(ud^k1eSf=v z_JX;x^jhsJj;?88C_>XB9*2YL)JB_57-7_nvzM?t6r*-nAV1o>NbQ+Q_nAiCEpuYW zzic5}x5Lw^ZuM`CLBN&d$gtPhhms?mN-hm?wV@jRsreMlZ+_vHTM+ioV{vg1TC$9p zEC#~Qp$AQBgI@_6%zo1}pZ$tT?-}y8D}Qd1`&=3NrGG=o@)T-Be)WUOdcHS%#&q-y z&>{d-Y^Oz8SXds3N=xo&>Y38@F(tl-hKH~H{Fv>R_$fnA4+Y*bI>rAjF;zZA`MDJh z1-}dq24~JlX)KH-o!2^Pzm2V+usu>7k$d@z3jWOA2$iJ7P}o@_{P%~1cWXk()5^n_ zJLvHzg}!#Hk55Ly#ZEkBFPS_Jq{6=R9Xb^(;%P@Ds)u zLnlyEJ)A9R#Sr=^s>EZz0+9%?l)Hsv`4*ex|$ z?N4NHB@$=RRAVft0FMDzVg-sTuOI{om0@1zX)umt6KI zK(xPB*?St)zT`=geYO`KbYl6^@lE6^HXL1! z-zBOQ3%KDTcTR<_F z^G7K>WK-6obgOG9A*92eX;FgJpxrK(;( z6Z>1-H*sGnY|m^AJjgf4Z z-fZNc51bl(t*|d|PX&}*x2vqU^0 z&QtkW5tZ_TG7P9LN7dkq>;9e&=3!WU~IpP)62a0MQC<=^e5JRAISfN z=7ZTQLQ>3QOER{pCHZ}&^}Ey%_$SnWNQ4*E%N0d8f_j0Ghl%78zIo}{w~UIf|3wnE zveAf+LNdzpkG(oug(;~B{h()34EYlu!uooSZqZvg_*{NhJxA&K49p5hGPKx zng)K2^5fynMqnl7bKU`3gZ8vWzy%8>5NTj)C}yFet5Gj&{e&E0%q4W?Z@4E#jfYr^ zq$$`#T~p(kJSHNiKobG$EF=^ZN|Y(}4Axt4Igm0V1ezN1#6onMtT&&bHLh-M)MT+- z^Wd_&>*T6|ddn7LD~a@`BW<-8O6}_?*WZGOX|5~j4~Z8w#;FIHn)(IdGfi7Et@=OR z)0axM8}?eAw$W)u#8Et>1&59$NKHAy_crEimDjB-=M9&A&f-ByGl^l zVk;1>9{LsC%E9kLMQX;J%@uXSr7w3x!)v2W3We6Mw>XgPHH=#{ZX)c65}Jy$|L9s$ zPHY$imOA|uv!SQKf5`$QWd~9#l|#Z6XE=THkQu9k1X3Bbf1DVj6pGPMo4iIwjTEPa z#u{Vk9gUqxP6kd1%yl3OwK70JiUdkMbCC?ATv}JCSA?4++?}l+^UfL*N9>*gT?>Vu z68fB)4y4(Mq{xb0aT%nfRDN2!z61v9=!obze{=+9R&5mKxf$51CNl)7LQ zq$BazI$2)}L`d#vIyKIWJ?wy@$ACfR_VE{mBwhkq6g+%h5M9%jIr+i~699WMWQv=2 zV96Z~>&SUOd`ZIlD40N|paUa)oza4FOZ$*;vI+AnC z%YQFCy$#}3P`!aBl2YN-XB_LT3?(=-FDfEmkpj;K3+aeVp%4lGPNsv_l(jwcC_W?= z@(w*%1q!PZNkdFO(w9S-hz4j{=W-WK4yN+sW-vKcR*$Fuq;G$jH7nHCrOW=@JrZZB z+3o*&1eoobFG4c#E#z389G-A**OMaS-?X%2>0|ogZsu2jJv|;KzCg%Xx9J-|903;vK$p^%0f~5!QRXDAAk*o#6Qzmi?$wZQ@?Ph}bZyNG~ z!bvH#F#9Oll#L{f#m8vy==1u@7isA^!IbLGJ@99L40ULtE!U8&(B|oId;y@ zhZa9>*F#C6?U7cAqDWSkP|;}j3iE%EoT-@}nyXBzZ*SN5?ui~~<0g8FUkkJ-HA4|e zg^MfkjMmK@|Gkm2@xGK7{b(Uf_*9Z(D6ZQBYY3CZgIJC5K>xq6@@WuWqh!A*DKmQ^ zKG|VB7OQHgq{%_!l?J8cK{q76u-B@X(%!fhpfj8yAI zeXztd5W1H)Dma)q1J}-6I|dW+NFhPM^8AQ97fQK-O$XDsP6-%lB(5`ZM=h%*%kRY&AUebn?`M2H3k;2h;|qs)t-m#sF}6YWLh zsk`Olre|+eom@K!Q+M)12#K-v9OCg&7HH|&LhG%m<5D&TU-fycPBgA1tfzFechx{l z9%n1OfkbP6nCB3iBqD&`1-zrYa)7?&>=CM?z=%PAxfbJ>ub)e1a~VW^BZ;`I8@+Fj z9u5`doE~#nN}A@Lq8DiAO^Q;oq6~}reXJ%V%d*#FOTIoC;5$F=Y1xj4q3U?GWEFfQ z8sOsM1mvW?5^xMgcZ^yP2)Nqm;XjK$_bZAyO#L zs)KNVPGkmJ+Jm{W&x*^qA0|fteFk*yuB@*cfXGV~P`IO)7!&dRY;!fA9%JrN!3wM2 z5(BwPjfTHbxE;?A2-+CuwbV0Oi|Lmp(2YPD+hQYaSe_-s`V}?B9Q?-Dl!bxQ@(2gi zrd~*jYI=O5CZa6tZmkM7%xu^XTO*Js^c>SCk7}eN_vdK1IOqH|<{Ex_`f|Wz0v*>> z(qA33*l`#%LNETeX8u^*xQTpqXi@+z`=CkT;`+crgm}bR(aMDS>+)<>U?3AdW8MFm z=?i<7M{%no36g2SK~Jm3Bm>1>dMhl&Ct@pRhy;{4wtH(H{m+Lo_08WS@E4a+pUz zBgfT$gW&sS{@H25789@>0DFD2#EIf&VQ@xVn=1?N zsmb}$xXoYW@hTg8^wCFN(_yh5Hb`d4E?b>TdyWllz4tMA6Dba9R=O447LQ{Qelz?+ z6orTz2x^L3T3Vi#i>y%03dld<#*s^ZnzRK{u1Esf8;hM@kMvc(`~85b^Q$rhWnDGf zC*++C`B`2(2ZHHU40f$o+QfR>^0tZ2)odarFh=0>i+dFwVh*a|i%PF>57dQ-VdGpp z42Yk3Oe`w+%Ft^$dh_C8&ER&1_|Lst^wYXT)*HUpEX(la8kJHU z+*T46Ei+eJR@Pc-mQG!ijd#hqT>9g&3YN`2zn!BJp<>uTi?+k>?y^!~1m-2*GSndK zoJ6onLiy}?EDks_MPodv$7*FFR;9k;@O*80Z&!8Xs@FYgLmX9_eAfr2Typ}0QKBmKm0iMnvyWY{1jA4Uwtll^ErcBWVY>i>juj+d)50eN=Qe^$=FN1c* z-(X(-8wv`yn<+C@7UcC{4k?}#PZ}}(9d;(Em_)&RQ&OFr{C#Dus~FKw3xV>w5@>Jx z_?|>wt`|{Pj{+2_Y3apjRVx{BSS)n4Cr;&IuCTvZ|Ln1GocOh>s=udwc*16N%j9D} znl+w?>~9_l0q?g)rmFL^OB(I5pdEc!Q)_0QvLR82(Ih7&?ljGnQC*g4)LJTD-KFRO zk)U4rHC1PNN#T|epPK9CMV3cO)Nh=75-f8syTs10Xv)nCBToQIIe6e3A!Xx6*c?-o z$WxXrD|Z=!Z{C)Jh1W6!oiQw$<=)B_J|LFOU=C^4B8iva_MJ!$R*J{6^nv>Mn!KyW zV4%#k4o9fkeI$m@b|r$?m>+3IomB;4eiR(m+#7WW+)%C0T{=4kKcUm=NY5Pp>OT5B zLBhq}4~{4OWN11%*u~A=fv2l66PA96;9zick%0osSAm29-}e~K_V+S`c`mpj6@wK3 zCj;}%n0;%wQs{$!c}+?#v10*e3U9t-Nua>P_>o)nRu;m!q~B_KW73ojRho8Y-`S-I zcA!U=%LN?&c2dNnQtqg1Jl}aih>k>v-!u+e>GVEhpu2~Ye^*buRl&0&3vsU3PK3tj zxiJtG6MI|caeaz}W-y+fr}&1>;It!q>{wWVXL&D8!Q$@Yz89C6Qd&lnc@n^KZlN4& zZC@Hmppod2)D^p!eH`Aiuw>esA)rHxz4Fa@r>==}--8Y6ihZR_f_Xl)f=8ex6dr90 z2X~Yf^c8}JDs?=$zGD$5P)t6Q{Jwvn8TVXA zi6sX?%K3xp!1v~q-4$z5h?tf3;;CMir~fMD7t6b~*jny(u3ru%~B5FvV-`)hW(@wmVx9wuvxxG;H*lq za8!mCwP4f1B9n^bJ$tOeB0n!7e8jWO?@IxFIG?CP6IV|jP5rF8y~d<|ussM)Hv*xd z(d%mmo1N3}3=TyXlcltwSg0^SIV*$BL7JJv$P8jtqfMYyNgiTNhVr$ifmnwr zrM1?)FZ$XX*YJkFOFW}2MAIjSm55^ly)??z#I|e42SLU3j)c`yOIE-17bN;vyoNPj}hEB2H#a ze3dyV(P0!~RchYEycz4dCO9#4=M2+$`}#l6VSPqtodlP5DJaQ)AhyHx9-@141V}@* zaRz9FqM4LW$V;#MiRNG3JW5+%|6Rw1VXVjGJP`h*YBIfkb5r+OZ3_%MZq-wT@hLK; z7FyV0EJ&LrcXrIl9tDRd2dIE&1&adOk?;J>K=%8FXuiwGtRFaAEC#p%WBR=@ zK;$R3R0uyNYWQvpFaRGig2E7%fdXn}k^{c3)M=BtTO?dSo}d!}HU4YIE2DXg-b(5! zN&xB8i_}WvFaP%Ym&a|*8cc?;-&F1ft~a+Im8jM_*r2G6PL~RZXieMhr%pWfv~{Fnvbo-Giqxl_ufw*ya7c}NJr;Z&q(!Y4cJ zoOM=L%$r}{4#>rOT;`^ZvwQmd&2G6^N-F81Q|<7&XHJU0Us&RbF&~W_d$wOIXE&Kx zV=&!oYPurVAbOdWGq&veOaa8vSpsa79*s_j>qvfel>R zc+RPpn%dL!_I#t2ihu)z<#52tdD@fzQHtQ>PQT21r(cQIo_tGASif|EHJlI7cKQ=} zq0w#VmgsS|M8NI67mqCYSfza@NVmmiQXl`|W0q+MvH}tuYMF)<3^tsCxa@oYV_4HA zAzS;WuVM+O+s0B)nzXr`vWc2C254(_M8#v-+~Xrt&#b%qWXTdv>CW?)NyCdjZOh`s z2Jo_lIW{-)3B8;nx;!sE`n~OqSR3kSw_8XlaOH*q;~+Q<8jEZ3bI%_ZZ!$6Ou6C;~ zF1GXU3+UR#<4$DGSFr-N+}B$lDd~@_LTeR_hMJC>psqGjj+*SZmAJ~)M>lgP7*Jz-)JDox#Rt3nO3vG@m!-hz{O@Om} z9~Sb!I2W35O&*Ig!|sJLcpdezc+95nFF5KfwsK6szEJ7J+L`QLh9 zj%9G`cSS#Au@ypqb)5HX7b-MG!qwz*IIaFBiotR`psaNv5zr97pcOhc%A|pqzLOj* zVU&LHdx1G5d$0)MyFiOVQpy%XLy9_3>h!~VFSNudh&TyL-QhF`b%J)w_0j91T$RSl+DeyKv=F$NBxC2uxQm?KeL%+l?k~CaBN}3PziahPFKR#-x=A z2@%2VExX_Hw{A!tgj~HXrsfK~t87Ajjw9~BhZOoDjg)r7r0Ko;81s7~yN2M~Pqlf* zA7f`l;J^^m$1R05*TX!_vrB2xb{`L|TY7#3q77M5lQU5fWecimFnQH(gZTgO*kg&HrKXN zI>yNOhWOk{K@)CdO^3A~e;w9?v87MebnW(56w+#7Ub${>Uhju0F?a9f0&YiUPs&uo zgafpxl$c+lMJI|GG-dw4uX(+~on5xL%PYB`Ma>@kX@0YSWe;dFBZ7aOG3h1Ro0O25 z=1>SG#lx7=eVLB%ByeOgpi2TtylZmKHWmy`Q&{UZUd&Mq?ILf*Dn(<9gsElvTwxEM zvwZ26UUwg!Smm27tQK&-1t@0dYsf@0|jhBe8fp zzWK#=Z%l4GZK@>J7`2p7{#E!#x5nqIoh9)Su{U4Od^Vcz0au(|%lo9^P4ub}JtMCh z;db#>2FI44kT(xD<`mlc{N{E9i@{l^6H)<(BW{-*sNPgsyCYfTbTNX+v^t3ef(QT1 z?BXuk_xWz7U`uR(}tS1lmc_-SypI_n7dxM^@i;r5Zx=v37B?+LzDoMu9Gm(kHw#u z-R<_W1)Hkv7+$E=a=GxNf2H|8wjnL^H%cL+IB`w?sf41O}!eG?e`8cXR)%%v&l7QNNsl-G)A$+|MRd6rvG)v1$ zW{aq>^wI#8B+io~RFODKwkq9AC{Gs;8B5-`glkM4T|aBqUtGtZ4u|3jc!G*I@i4mGT6@pco7L)AS}=II zu671%&IaW)k9U!J0?#&11@OC!5}_6^_BM-##-OYpjh?IFg1q*#1z%M*gE!!vu85Kp zR>5$aWa)C)%Qbo#PHxtUrMeY*&6V%JTo=pTE)bxonsuq(2rvNO5Kd_tKRH3_K#)nN&g_J^J>iD$Q5Qc&Kv{S4KJT zF2TPDC1t#|!759!n?cmJ%@&GHwz!M-0K&vd@PS!0A+p8ctlKk4@ZlBWSui#dhKiiT zZ8FAnGc{)?&d=#EMsqmOa4g;%4!OU_2dNX>~rvP_B0jSWcH#KKnK%q(I1DPt19w z!%U6xaa!j!_w)%w{PkFvSWUkXQ-X1UQFq>bs4(hO6dM=__w=T|jy%lW24Pvm$aAAZ zv)z<40}m%3?0?8;n<>uldNDms?^6HW2Yb1;jWXx~_ZjV>bNt{mc6~Ed3XzG=1@#)$ zbNce9_fxNgtDWSg|JoOPsErS863T3>gfwET@s6!Bc=94f^$6=^h4|%=aDvaQ$iz2~ zAI$F8+heQWYO{SV+_Jr1Bu6SU2I|6%{7F_?7`|Zp#s-{50+1c+<~ zTUpg*Y|wdeGx_&lruiu>LO_t3{`+45F2PoN-j8AH@$HXLgCq|%rnm@fuCW}?Ph-69 z#@o9LJ-EnP^c7PrQX)6cb^=aG914WUu`T*PnWG0JaVo)R+iO?n5IHeN8IE{#5o) zw!@^6<>!lRpEOdTM6U1tCot`oo+CTh{Y0!%`CkcGhZt$*$B6>Z5!SeU<#!LG+#26c zmy)R5tjEkYKGs=RXMOr5(ot5GT0Ok89kRYAPVh2nHu?XwcUDnRMe!bn0cDWxh9LzR zS{mt=0SO0?knWU{t^uSQN$HZ3?(R^A5a~u5Nol-?d)K{d-S_)=-}b|-nKOI;&yF+S z?;9IdqRX9!fCFy&XUtm)J1h|9_c!PMcOK-4nrw-lSDv0#_EUrhq4e~ArYCYeRU(G2 zQMmKOBqClntw=XHvW<(@M)I@{-%HMR?GcN}e@fDb2ZGSeb%xKgdp(IPjt(pG)X>2} z)beTnDR@-t3D|%sYPI7^^4*((lx6+^DYvK(w6f-qB&VK=iZq7IM2<<8Fs;D}*rj`|i=tq)#tG(<<69+D&Nji57Gus%sSO z1W4X-z?)^4<`!Pdj)PxL*MwS0^OoS=LTTfDpMnA-`RaQ3^+>E%hIo zGoz~G0hsW#<)`Vc#D5z5%C2V-EeoUbJb))+ohcOEKRj zeHd+V|7fA;2Q#H7BO7M2k%+>{Yv*xV&FrG{4a;*sH>G}i%b$-gi3YQFD_dz-;8{O7 zoZ`i$3t58dze);nmR0wcKLW%hRHz|3n9+Uotn4vwvM%>`EMt&ig;s|QZi3nMrod_F zC%sXXWk5*iSca^GzY_R@!!TOB$Z?P$2JF$ee{5*;-e5K>7%MVahhlYf3v86ur0z^O zo?EUTR#ZvmbA7Er0UEhT-VF6!j3oV2oOxY<_<2@hNZ-T8cI3VW4Xqkfj5ZWYOHpwV zMA7D;=M}YnD4vo0gzunNJ=<-K89-*TlpjAq^8zwD%*FfA5hO!@tvV(g1M_ z%1edQ+^BE3C)63Y@AmmeC|O9kL%4Sp=Ih0a^a%rjNBeX1!I46!s2EYJ*!9b;X;1xM z$chtG5wl^ozHPxq>KCB)3e36FokYBY=bZ6Kanx+)n1@}B8c_u(Z!OnF-Iz>j&IWK~ zF7g=_XawoTtb>J8lXJK_Qi#Hy4GxF%K^2GJCv{Lzd2?5-+iPSZAs zqC=y4_9A=w*|mcuFmta9X)BBVcRj8NG@@dzG4T=U+aKYCRFS{dZ;xE}-q=l*&E!}4 z<=)AeXuA3|dNW>FNrE@R%jAh}d?}hwVw@}&u8moJwPKEz@%O$bXQxmXyzF7I>ClP{ z7z891S-JWH&~tTd<*b;PgTh}tYogqPb;ph^{t7$dpXfQWa>F9`hLU4>&5CGj5qni% z$bQ_G!^h>mHtE)@$4f6xM;0S2Zi{l~7Ff$VGo*6be~UyW2Vi?)FpKB>HekJD%M#^8 zMD})0g<%$-GBmhO@QP&HNux0`qH7Get-oKR(x?ggyG&+p9d-Hj-yq@`(e#aNFLp~S zudU(T>X&&DTXajr(RJcWublIiPqCAhgfYE#Q<~#0oJMP03&mI_dhaD$R zqnE*9KGe%$zwhal9xSNcZWih5zr86Wa3Ww2|K4RcT7s(a=bg&QD6~Ab?m9`si)~56 zZKyXOPZK86rht5^Z~$9TH2+#6!?$F8vJJjD^e%l%BSEI`c|8dRahfw5o&V+F$WTgr zQ1GZ6WO#}v7@r|l??I_x$3C|7%9F{Cu+3fSV~q1+Tb35Lz(GN?bfyCp=vepO@Du+F zeVln@s^1{RAk<*W{T5aHFc!Ph%`EuP8qT6q{A zA@aH2O-6XeBAHx;__Hk$H!l<6Vrv-Oo)tfep~*ck{twgL$_&(CzZ%zcCjqXYpwZ82 zHpY^_*7!y;VX9NTI2JSWZxb_9@6M2Sue@w?%A#DIFx<)%)O*srvI!#$Ga z^{XntCE>}kOWiP`Vw6Q!?__sPFmBt-$Z5E*wCtjTLu%vRRbP z-s?NRw81kvQHbQtX>$&Vy9wvw=$G?7-Z;1UlGtR~_HJ0&@5#FNyCb^6p-j1ZI3){` zFZ6yw)JGdde`!{+od2#zK5fDsoB!Io_I2vO<#!3bnD|}z8{?t_>8>hj&rGBk@>>`GEKTcFn4EQF z+7GgHmJH5&&h_GUxkQfAq&GOBFNA=LF1;miL<=72P#)F}wu(#cD>8|kim88T$#A&9 z5bfLN+Ln&9@Md4F)XLhM$JwAcym#j^SL1?TqF{xTk4Kj`l%BcpyohbTM4$il^>vy& zJ`&14d}S!EaOPJ`QoAeHl*zK#rTv696=&h=dFF{w3#AYF83*h^_eQ@}I9g9ps3^7S z3T(7GBd)W@9GLL%4n`uqzP(P&!I0w%Csg-;h=Tth>A}mCAQ;3JFGV-$DG-V=ni#nl zg&5DNx9v6L@SLTkM!~`;25`Hj$*J{on9A|2yx zLYrJ)CNA&H07tDI#zxgRx#g;#Cod)*;i{~NRiPlksT30@x2#aH#FE^ zhI65!X4NFv=~It{-m~wBP>2w%vVm^>X9LsSG%gxuvpaHhJC$H-v_t^`mjVJwt3%ZK zL3J$d)g zzRiZR4`F}b#NJHuvbQxBJ5CmCD`a}aNTkYjeb-UyLtV>U-LLa0I@T7)Q}#i`8zlbi zSyg@8S|b0KEOWlU49?eW&{aVVy|gd<5?#piu_Y?#QoxZle)*~I!*)tJQwHyWct!ei zWMaGoDlah?munVf9ekASRcP4Chspd%e5EG;wbzO#Sgz&X#TR^-Ly+*%_&N zUHJ14PHjSILU2# z0jH%pSVTF{;T{cUc9T#h?rUwK8Y1lFpk2a;|r;I>ODheh1_GHAfn~JDf6I&*n z5lhuY2Ev5YtSI7~wY*%w0|>%|yySgk0f|@SAXA^)we-BJL5|S(*|9)}jgHSE|N6?! zLIMd0&$`?LzX;qj_9IfTy4%UiE>g+3-Y=C}-l|Y5cj6`1b@9;pS5A*(dV%M}|LT=m z3mwvQE4{2%QuJfKVwrf%{DIa+$?80txobqfaj0sk40Y+3?`Xt5N0Nqz58V#_v+6M_ z6%%prubh;zT5?{Qde2xOpIiyGW1U!O)(ijEoIz|3uNg(=Ha^i-Isd|$&BF;K6*o2F zL@G`g`z1Ee+nraGG)6J7^tRjENUHigK{)qv?9;>|0qG7gX8}`surcNE+MUu~>0j1| zAhwuxC{#}Q>qKh0d)`R>aIcIi{K4XsmGZ2fsX^Z@FtyD$sj7d1kVk%C@$dSu#;c%} zUP9j=>!OrO21Ax`jh-kcwt!l?9qa9Prk_DV55{I;749)-_r1bcoj%O``6_SeneqvE zXe1P+wvw$Z^UNVZR+9w-8cttt1+zk!pPcl3kwsT@YW7I2PPv=&1iFVU;$nk4cLm)` z#BE52GqXR#gEX^WvyjIx>1upx5LtqnK+b3)jc*(kCGei%-(q;x^J>D?;s{v7N;@6` z#TjyrN;ehCoUqA4v$MS$b7;WQfTO>>2L6amBQm{~+@$s-6@$$%=G_H;5I!X6s-mpg z);%>M$c1I5ttvBrdY9GPf%(^gGMvdSf2CGk^5%!z+Lai>tS7KnXB?zw?2`m-=1y_F z4iTS7*+M_6guh2Jzeo_brH4X3X_u*4^=T1hvF-Bw+va=}g~g`~%%ZEDjAPF?jC?f1 zur%)*=93;aI5)|_ejt^& zG^-=WaruvA3h#-(Lj(}k)x^fLJmEhJK~b}e9Vo>Mo6G$b&_VJrknI$n__GC%&p$tV z&EIP8X2r~AZfP3Lvhc!=;ajbb`3TFCWqIXT$C7{EVQ#``D^2V`X!1wBSRPwY27El< zGIm2N3j87`Dz@bX(v3IMl(!sBb+Rmmibd#xc5q>a!|Gs@*#_w-bn(z#I^tx$_vmUM z`P=O?GT@Iuwqtno36K3?q%c;e#*jD0WmO{de(8M{+OY>2YszT*ec&_~ZY|?I5hMX) z4MD3(c_>qnm_WFyPClB*IQgl5==KR{sfdNIotPWkoq@Me^7oTivNBVSafXi0;B{_= zbD6W$e9onF#WCIisCM<#1G!UMd#l1cA6SSIqcBzgQXUcW?EXK(3jXK;Jhogg45OQQ^UCDK?{nwChmP2gO)4l!8;2Q1s zGJlXgomLZO&#?MCVU;jEw#0`Mx2BWDwPS*RcJU5BO@+%lZ`k~Mv18nxA6b<2%x`&~dmDgP0i9Et zxbF>c@gZShRN~6VwAbn{0B{rFOq@^Oi_y@e%6_fRQ8>5c$XBC{NDP$V2L0D66>&55 zgEsQ$pR;SOne$GmU;K_+Wd2G!U)jTb>x9H*Ebb{mi)yY%pB;d|V9?=^bFZ%ThFM^- zoI{{c>_-5``hgw_Iccw2|>=mqB9ysBLdVf+LL`k3sDIk4#+X*pxCR??P_6A zqIH?s8jOQga_@tNtyu51YnG3+u8m_V1H@4 z6k=+rHOFM}{C0-sOi{r0QQ%u9;S0^YeB~|H4vw|Qc5e^Na6*ryaPT6hoNuz80?2Qm zr$;AU-O6Rlj<|J~t4#+1+Q%+SVu4)~zH#LOA6`{CiUI1=a2e%bud74_fn2V~m}T^3lW z&o5YvTvVielVw@fn8Jp<_ZX$+X4{M*w&h> zJVYFryiRx~v`2C>HiepYzs@56$X&zeVF+4&kCM-#o)N$6(kWKW|c?)xSjFCNm_=EVUL;#qM-h(Y_u zYL(^>WSj=4XMA)Yx=?Q})oebRQe!^af=EEZr}ZlJh0ztAy;_^4VUM4wR6F)%BQ)497)nbQ}mDM#iknr0ywtDtsZ{{%3bUdT>bz`I;}!w5}{{8jqfU9 ziNFe}lxR__v}^mTlK!h%P!d9sWeDElYK_%X9ga@B2Kcs4&WO1TeU28JPp0~br*YT7 zxR#t~#e5)MpzC>wTiP*kaO83p&@;mLdB+|yCH!frY>CFGC4$MhpZm3Bf@tvHX+I1J zMg;)z|Ku_H{A^e1Cjn2I&=-PnY-cGafq4s%j8R%F*V}3X8i{W{Pk#h0#NUDAmjTo+ znO^4UmHodH1!+t6{*PLxU{(28M<^t2tcTC901v(E3FeIpfD)r@{%mqK{Wcax=O+YA ztvul25G*n$L&+)tQ8Xj;EdflUBdOD6+T*>wXstdxIkz|aRcrA~61<*h7P2HPbj}Qn zMu5;Jfu8cF4TFqnNO^~D?G1`7jZ_>FdW%7gnG&EfL>cT0Cj#!0*kClL*p5jcwggxN z5#D22!3CA?J8>15e6X>x5s@qoP<>>e?l4IFPEkHU!RfJsq@MSmA~6kPB1t)V`jCKj z6@O6x5Lui;j(`vqKoj!51SW3MKkg^q)EU&gOd;tr&NiaCqj~HvskFz z%<6G85-2vye;oonY zv)sYZMgAzB^H3{Pa{WXDm z!EQDki^es8Nw6F9aVv=lY3+E1{uiSH@ck|@cdH0>BNU|?7; z+n5MwR`6{UB!O>N-Fo=zy}w)0$T6bliokH=M|slzwl>43>4kFgt#YAtb7(gD66bk`e(r=Y7CK&@C?1FsS4!|lrSz+C592Fq{*eqGov6q)@1Wtsz zGj)sH@{;*L(|X#eT!1m>(KhC4`_6%!F-IL}0`UfT1_6=-xW@m#Twv7xzjyyniedGi YyDqUP34`MC<6j6U$*Ia#N*e|J2fdZVBme*a literal 24393 zcmZ_018}85(?6PIW82Qgw#|(FJ)Q zr{^~_J>4f#QC<=O78e!-1O(xSl$a6-2xuPg;|L7~tRb6bVFvyKbykuT0jZhBKLHkC z9HcazK|tWp{(V3}GPALP1WT4G8ZH`gvOLE2whV?Q_C}@*9<~lZY7h`U4<6v7t*MJ4 zv4^dVoimRIKk0ucc!1CUsu@X%|3l(p%}=T!r${Vn?_^5M&cMdNL@EGFOiaw@WMamn zBqs6S=D-p^sfCM+0}mskySqDsJ1c{|lQ|WXmiBhU|M(gj*}J;%lal@u^uPc9r=KpCX8%W$o%4TR3;2SJ|Gr^l zW?*9c-@JiM`To`Nh}zpaIGH*-1NjTE^ZkeN|5f*Y}?^-h50zpelODgM8_6r3zgfp7hv*3AEH{eRc}m!6OD-;4iWiufNb|ECtHW&v0} z#{WHJ0Y9_Bo^i{PVxtpR&9G2FkbG&VLikxH*!XC!W7VzSfio5f~{ijJ<*>EX!EH8?ax zAmH;n7>RbaTz&WOK)~m5JzF43fRDeZQXm-;Py*DX8TBFknw^o+{*2&Pr^8#c8xQmSrJ1ZpGbs!y|OL`!?CvOjdQAv1v2G9QRRrF#1KKmwCy#hqRp`&Cw3fcpNx8)y*jsh z1L1I(bQX){i74KFr;EQG4~9LxycqPmH@CL7B9bG~N%A?tiaSmaa#B?JJa0{egoILP zRHt)z&2{D=iG`3&gge!$wJb)*^GV#uqkx=I30AYW*1LUqmsiwkba3$Tv+p7kuO6>< z1RZvTszzK-=7&a$Bk@V{$$&QC0zL^F9Uc;Lf3g78EdSTe%;4+xXadmH?>jg(H3dKQ z6D=fw2He7|0v0FoP}JIbtNo$Qcw~}nJpdP1#F>zg&_aupSc$U=4RpQIAm^ZzU_}FL zAOzjNQnj*F7vUgT#91ROSS9pH2dIu%(d2SuFdkANR%HUvhTMHXf=FyXYWnY$brs+Szc;AJ z{%X6+ey3YbyXjsiz5+W4sIahn%K7$H(I{IbBV=0;Xqx~Wz;`7;1pNuAPbSetuUqwGodn3uvh&S z+?N2(w$>3z_e;PQtbkqn3ac#wAVkQi)Hb*r4U8KlKf_XPO1<6X1OQ-)`fII}9ob3>9DM3R!v{-nWXWXRx=4;**wkL^)dsJK-`|#lfs^wL zhnAje2vVVH$NJ|im^ktpelsE*o2=YA4Hz41%HpW;4B8`PGM%PUNLPObJXZ|l+kn_8 zUfuT};K0651NH#|-rnAhQ5^X(1P@#eaI(@+j;E2W{{Z0fW-gRBL}kFxw|$YxT@wIB zF||e0WM26X2c|~{3}~EA!ZN6*#%7ttKafH<;wPz2}BrG_I(ccwJ3Y1&7a&gNdZ zm|Oi1V|d7jh={KC_eqaGcOATc*%=WMN8U&P>%V0(5^469ux2iz4FJWOafKNb5c^F* zN8zQT!|4r%FaGOG4ocXrl(aOevwtt)4^C#Jgh3B7U@hpw!vjWJkGO{i*9B|JG#uq? zZh{Ni2eRc<{&UyZ;8kK7`>M(;?7t}v5nN0jt*57FX=yo=%fC?HnGtPBsb*zGlg{Sc zv6wyeua8Qi@j0iGarh|aX^2Dubn674`(iPzEXY7$gao6iIgJ>nUFX85l+OnCsAfg{ zbTc75xmY5-P+i)J>o;o{3D95+pChX{CQFYd(vVU9E0ldA-|bbP)BN$nkHi)}QAi{gv>3p2l}WCtQ{?IvG`mvx<(y;IIO1 z`-TbHtQVEQj3yUocJjcX+ZCdupa9Nr&#iXXSUk>Vk0<#Yo}~_Vd;Hb&!$T8Cvk1^E zkpLG?7(~uP=*5zogAsExGp%NC1B1<0N5QXm)<}*C#1PwN`G4@g-Q(}CpfJPO^FCj{ z_3fKMk$#U~s%fxRiyUzB)2um;oG62Wf-y>Kl7AI?e>ai++ zWw#F>4kYp}jgr&s&H??1PFPh+rZG*`={G2BWJi8m0suH8|Ndyb~%rO@ZmpA$d$ z(Coi^9feY-Ahhn3b$jJ|5s@Jv$~U6+NHVNI&o|@9I$T!K#(nxh@G58 zk`)gQ3RFco#GlF$9D{_YZ#m&+QQeApiIYn&I3+nk16wBs+6`W^m?i~}QcaO? z`O#A3YpmGhZizM{-~j4NXB+eGS&;twd)Yv|Ssxs2O&LG(bDp5ZPhPFCRx6^`KkV}(NMh#v4~9ngHfC_J@kfM7l4_}KmH z2US)1>iNs{-oVSBp(qSHJG;|#Sq1Z8?fSbB)zsfx$<^GYx77T^ z;EfW*Jx9tRRv!U`tBsFI_bT*^l1QsYW?+3{|bD{WB@C=EQ)T1W4t9tq%8sDPegC zNXS&GpCVKqC-WumFLx_dn#!Y|ulJS~7F7LTMve%9hXz8}JTB$A8wc*yrfANvVs9dJ zWJz6eXqm>JpyvZ)bgfvP;U>l+)Ybnq&aPra^}NgQ<^_QX*Oh*kr}JGx)=%b8}@=XMT}d^0L~IN;GRnkO@!1lH?#lqHSq1I ztc>PQxdeanXYwA8yp0WAG4>$=8w+oSpsj=iq`YPrUGOODZKT>Y|0jTqw>j+$WwEjP z&(XV}?_lcZWLx}qN;CK7a^B#q2`FRDNTvVCQ27V!>>T@^_5Q3`AD+!{&~)(R(hX%u z4LF0i%gxb~GlIM$fW;(Ah5T_Cw)c+@j6Y=}-6*K`dUbjankrn{zR63rslmIIST;*w z&0e714}82m`TS?Ac?d7bs?)iQ=7J@_O-=xaCI)&hvf=HpI7dDN-VUpFqp;+L3#N8e zy3Cx7ALXZk*c@cN`{m{s#WZ#!ce#BOw}cE9^*E-Td6(o7rmaPOIUXKoN~nOr&c`$p z_fhRVTeBl_luGs5T;|a;@o!4!yK2og zh#ej@k{>KbYXR|^Glr9GQgJ@*B;dP(rZ*trY_M%8QW54de*1o7)S9QLF$Eg-uO2kP zr9~dT0s#cueQtWDdPnFD2(tQJUKk8y#QS@DKTD)Buzz?4tyXC@TF67f+x&<>P5UyQ z@Hc+rtaG<-u;y(8*oks6|Iw!AO;X~B2f`$C7ufYdc-!^U$@$QNO>+=W!w*5~JdfcX zGeb#@vzoO~1o^%&fb~kvqRO;e;0OGwUjV(%>UMxOG_QsXf?XdrIN=CjsmEEP%dbF) zfyf$j5;ErCGt;0lLqs&W5fQkz;HbX+k?5yaX>3$Dohi0RU{`vE#`An{O zG;phavC%RTx5#O~jnCx}uyV79re2Y3A)5+~*W!y|?m2n{@>e->;UIsCyCWTq$4bTq zIzd$!oXqK!=}a@;cb-?f(XeP

j3V*$mUR7c94nkJMrLh;svrT1m{ow!`TtDk7By z`MQ3Fs`-pY8?!%u&nTGQzYpxK=cXzH>&=fZ8a~Z@-oH6qXc^AXn;%O&9>o&X`yjm0 zM4j6;vO{NU6-%(;+g{ZK$elv1UbjzV&f_kaHad7@n6_B=q3@1?VXOa_z(`qnjRgH# z;l=qb0f&oGGc`AJ?i~e8&7?&0AC4NWR}g}qpk`!~vy+P=-v}`Jj(tZSZ4BtnR%xEA z5O8NCP3QfQ_oLPT5HGffgu))tleK66_0D#6T@Zv1ObLO@OKRqC?WzQQ2<4NOLITxt zXPro=pmgjI(|&b~vk9`WUj1duH-Kq_8f+0<4bmSy0`>@Al5aphJrvp?Qd(Aa*Pv1| zG?=0}9Pi5>P7rOlLvL_IGwwOzG4g0<(1iyOh+Pe&Z`>2s5jEqplWNC7k|{~v?l!f? ztW!JUwny6IOFMO4tq->Ggs#+7w99!CeglirrAMb^NNl6 zq_G`SE11=iXe%&*9Hk_i2B=${nNf%x0-5dt?%<#fW*?p{tI~N2BC%SZA>9rIXn1P? zm!g_|RT=C)@MW=P^y76q=72O<-bePiqCcdPKna$R&@$^>-FFS$YYaAo+vjgXHczgY zn+t4mrwSNa*g-+p4@ec$vNxp{fhfJcxl#Qb6d^`TQd#^c7{Ql`+L&i~GN*%r}t^oGWk zp|BW@frK}C7c`D$NkEAWV}|Cd+8HD^bfNshg6QZ^W+df|h0mvw;Z`VM zr9!|lr&eqN%S=zApmgS?g~MX7i5S2IOT1%nTFo-un(k6#vKTD112UQzrX^#&J8o#* zaGcqx;3vP6vpQW{4Nqj$|IL^xr=CpRhXSr$iyXOvOc+}Iup0GszQ!slrD?o1*w_dc zPbe6rj-?DzJHAAzV9ysVBPG)?ETBa_15uT1A9xY}bY81EZanzb1ZlgqHr68&9)%G# zoy+?B=SPMxeF%%G|J%n8k^KoqLFeQ)3ljgWCcKYkn;vE_lN8j=-0Se0n}bI}uCKq2 z55LGAyf01~%uFYy?JzXfhctYd-T*YFYGqD(jXz{8C(w&x7h>s{@x@hA&1T!td@l>D5)RM!hVce}(k* zAbgsIk zpU9tMdq8u*jCaVbn#Xo)o__x&+PhZoW4qf{LzZrPQBSokpw5*~CNC;A3(V2`X=vgv z=Z&Jgy!c4t?cI^&nTZShAQaEU5YIv#h!_wL}E-^%v`2~Yd;a{4+4M6jgW1VXr?W|lYV6nwMMXt*so};rfDXOY zh|C8aov&~6hF432S45=eXA}t0m<*d$I%~EqWc$B&a7zF)-<`_>78=ECphD|$n zOwG>YTHl8H1{C+>;6L*nKwoMVuw4|xB6R#-#%BpSaxN~s)l$6CN&SyfDlxXAwmffG z*iMD>?Ga}ND~7iIG+KtT{5Uz`Jjku>_Nu-m^0z-uRRep@P-Wp>@x z-)=HasyEfyozU9!)pIzTYud^3eE%K+o&3#rys3IO+`kWFT)}$ozD!Mc=GQNbEP)$h zirb~i6kT2s`HLPhBy4IGD9!HI-c0uVwGU1ssmZM8TBOih2sQV!^SwipG{LX&njC4K zv7r)0^$GAtUBf2~eGPkWk(y=)K7$hJY{g;n&o0fMwR*Olj&QKNdReAvH3q*6Jh?AT zxpE$U`Xpw<_pt%Y9%kR1@~_daVnhSM9=iAw!xn=h21k%~xW7Z%tlcb(0fO9KI@BNi zbF=w4&*8^+U*C5PeNy9)aAupFN>JtFO0sjpd$0g)?oWW#a9+C!K_)5+c%CrAhJ>z< zH=*DShlA1>$R{YqyHnZ~!5Zz3f)*^3yo4FbC~oxQoeVqG_nQKKUL$BT_@bxt1N{m6 zOpY40Lay!9wdx?B==OGQ^l4Sy{uYWfJ7v*$SKXc;D8l9Aqhj(tRy4=K+s;_4U-oWo zJD&mRyU`aN&MglWP1N=~aob(B@9*NH+4xSym;0uQ2N);BQ`N~s{5|)FFsNy{eB!tn z=_(YZW%{1C$&4{r%%f#G3HIDR_s{r^(PhY@tsh7C{-7n&ZdEFdODUB;MOQ4W9rmj% zUQeGm^qG2I@82cNpa$QfDDtoG&_jCi*vj98W38U_D;qJEQ+-$Apu^{woA+;1N=+vg zpB}ALhOLJ%80K3t;aHDIK-n?%Ta#%v;6W_k3@@G(KZEHvB)j!xEmOW z@gdFETsai7L;?j0quy|Lh#lPWvps`&N7A75WxCcfFRA z-mTGET+WccVi-(ls5lRv1Fu6S?VYn;aCuJY9;MyNZ5`isr4a08gXlUO<6~-U5)kxJh8Xu%sUijv!BEOFY=QCdHSfeh z&%lB|(vX`TrC76@bTmnxZ#F!xBfRUZJp5hnHhQs-$KSwn#L_r8bHBMkEC2a+xoNGm zjK}S+*Uk6z*O3QI;!o$PSBd~1;A=Mkm)_Smp*Ia}1~2VlI9tR+xqx$z+5`uk=hHe2 zhD;2baqD{gu|f-H=d{ink?wr<@k;*U>vR=y>4w>zN&`?TId+Fc6nvZ!j+B*38%!-R z{isRq?ymBiM!R$3A);rC+Iv7Tt!HQ*LGMFgsTIfNszUV|)SsGFho*YMe19ZB1OG|S z>8&Kce(yuVNODKdcKq%4?~>z00|lnk{o$5S+cMQkwW9^#!R^rfxYAH`icGfGeuzK3 z6M5W#A%2N;a-VanmEGr0?yFwzM7v8i%If*oJpY_T3uM@cEvTAa1DiCC#bLKT>?mN z{tA`h$$P2A-~&-eKcZD1=i1Bi?b1$0ii&5dpgh((q&b~@#vpQNN*D~3+{7o2F1PgK+EB8`PEI8;SEEH`L1Wf`yd zLstI zdk{^?f`FcF28L(8AY8pceXW+k__8YN#DXR;k5`LETTlJso46ocRAdI*`sH&6@0+=U zN`<_e&>O5u19rZ{Id(rTM_5{lQ86g{!J$3Mx*AC=Lw2Z%_F3AO2lV$37yr@rGfv4B zPkrJP&Y$ZU4K@Z~ajN}J;<_(jBBb`758M$-k-k?{%K1W|IUK1KloNtOVNg|nh1~~k zw2&Qw=vA}Yqf7(*vA(2ddzdOk1V#zsVqVoB(ZtPi=1WptE_>6$(Ya6v$QD2D-o>ka z-V)*YySG504U0IzD1?UiJG#FQ(A$gc-;v053%a2?3rX?#pinN>X63z&rUYJo3CEsl zP8AcnfX$OItaWh^B4+VT)Ou-9@ROQidc!V!#&!d?B^RjJZP8T(#8E@% zT^U&a8=T%W>??Su&ZySwmu>MVXb79{hQN!7w0vz-fA|@!SUeDA3tA7y0!)$*uUj#~ zBdA(^(UOr&hVNi`Nn@+YW^&+Yl=oJbri-suYR&MqsR!&w-M#g{LDgg(CS*82$>6<0 z&tV;e8yhGpMg!N8=dX+RYZHzvLwy#&@g=2w|iPIM=pTz?zC7?NO98P-I3pv z3V3`#cE4{ktl491%S;k%0_y1Sp^1O*s{()f5aGn3Sa+la$J4Kc7>q&VevbVbKIjeI3 z!7#hZ>8NaX-H#S_&2!yHPaPm$1PB~yagCT`s=1XcHjWKu@jUf3%&rqp1O5)H2Z{|` zffJcyYy4@ko-!Bl(Dl1zd%4gqSMiyiDeDK^YV}PaSlgyUUk^)Dfjy(c$<4r9-SqSx zzuVU39mBpzWn=RwUXump+tlsv_-Qw9G78WtWt;1rc>Xs-r5Y$jtaq_3^dV3nF%FdLsO z4~N6*i*5HusB&0njP}iRbZRsEe1$NQTF{&t;bx7hg4pL92ZEIu8Od<*>}P>1lfmvD z1yTj7LlxtddUrPVXTjJ@`n(ehU;mI`9xeJJ5QMoP)E|HO$_ZY9X1SRb+6o>Y_SYeK zW9iPC(|qzBbIX)f7iVX2;CTYg)4xO9(BYYV^HE z6RkkH23(JiO?R06ts%2AAa*lTw0(~?f)i2;@80u$^aX6wJu~`)_rlj0`*NR(40>gy75m7_DpfRb>rGgw zu=_BjUR5Xk^E6a=8lMBVq_~+&^`SJR>5FL&F38_YD~#3;jfdnO7YtT)L6U+vRPU*R zVg7mHY5Kn*hN*nG0l><0Ro`2-+YyG`+y`@xba`&G@7FpnJ}Wt8c$(~0hVgDMvjz&% z&*5Z^+9``&lO4OkCsI48tMQc{4=?vMoa@VOlJ zWpVDJp(3+DAjT&EE@;(+m^)dYJC4-2Fgrc0-g$%X5i(w)XC~j=jRs!7lDf(h?4E$D z{3kKwBf?NTf0ZmPqrNiP7Lz}eX1eq(YRhkYI)Tjhei6v!erE2_fHS=|RC`yRPhwmp z$VwPRZQ^ynCgRbwMwJ0YblUk992SX}M+Z0+=k?FtmAo;2& z^Axqx=;wJv!BI3uF!SD9ui3T5fo9MV9L%%m@xH_B99o-`(^fwY1b1f>mJ&Q%Tz2Qg zYj_@m&Nbk4WX8p5Y3Snenzy5$Ou&9s$ni6!C7pv?u{4A!$x94;t`w#l{v5!S7pQ<& z&spgoYX#8lpG{62BbF3Gra@KfqgbpBh9RpR-||hYRBnZnpE0_CXsvklUCzHi{Vdg=`2S7rJ> zed@NsL%7<I$S*2xKw8=`zac$Ax3R(- z?4}kHgNaC2kU5$T^Dah=IFU^^PNyv;9ZBaEgFfArzMs(>6$$7k=(CZrFOND7F&Y{X zlEbslq|zRExIfdEZ>zO^3qn>wam??^>GOo>{0eQ44I_roG7UJi;`oYw$YbM&*cD}i zL67(6I={tT&Tmk4FmFiv(GfGK5xs<=zyB7DQeo#@)4xC5`zz)H{!VB0J2!!m1i3>p zEVSJ&ZT6=@H`7g4V(cIn-;lOp(twYVFNUgx3J1bLe-aYj3&&Ta?5Ti5Y56p8ZnlzZ zv>U*F^N#)04Ao`u@ZQrv{DObRQ>9XqB`#qLY(h!L%C|6vTp#KowJ$@J?gtO&M{=Ow zch;jQyS{=K2w}1+Sg}A?1~2HGqL^f6g=lqX?AwHm6Yqxt*n0N(Emin6dhdjHJoPoQ zHF=_ytH4d?VyVy%kp#Zcs*T)wg;1rArG!Cz#5-d#;i*oJTojDHfg!z|wyS z!F;(M9$zLT!L8n_fe!3BHT=PYmI}9SSzV;xn>tA61l$R2wJ+>{G}l z-scy|VaR$LHtiZG2>A|(X~bjxgxC?``rf~iX^+mW!IT4F9Q^pRv)dG$_fe|11w3W( zIc&&tVH_F3U)p$>1N{rmN0n`%TQFdaK6Ql&6(Y=^wt?ap%E%;WFy|vdHBjMlKDo?4 zBYYlk&nnVG7%mrc^K$ax-F9_f9iB&Zy7zL0U za}@@kzw5la9Uyx^#3E67^Sc<^jA1C0?qtk%Ro4|?NEA@rMC&$4trAvZaF zqzqq=#yArbL*nuJK|jM_7Y45XFkaX1e@vV1D;6WUQ_7mvsg3eRQwh}jXbrtcFzR>1 z!@z)gGg-Txuc9~($W74%Xe4YHvoLXuoGBncjMB;ku#}wq5Q3lLK z0ZhlGHm}~DOxIX_SnIVKKlMf?Hz;CqslGj-7iomw0>fY=hj@*Vk(o&UxxS%GrGE6OjL#rPW6*{r9Zn6CcZLE3kcK(k&z9-8OAVWs zf0O)k?ho#Ou5QUT)~Vp_8fZH>?zw>(Zcq!#ER|xWkbuVo0dLZZda3kCSF;0R^LCLy z`6h|n5FkknTc|^9f^KwhhoZS0DRBRD8%8BN5Ru<19LuuY-YuWq--2jvR{+~672>z> zn*oCY99|eGjpT1+7lR9VU&eCr8`|PG6lCI^s1QKcOoNf(?^+QB7|;O}rof)8Jz1RS z{sl?8Ru&nbfw?yu%(rV=W0Yo0o-_vC{n(TenKZk3v7=IJRu`y1QI8J#`_|rGA=XB! zZHoRlK%X$ax9Ya>#-aAd;6ydMGlk`r?HYcYMWtda!rNwM_z$9OLrQUMz3x(O0^#J_ zs9>z;q?Y~teTUWjd6wp1nqAE{m2ib(y2h59+g$CkAptHW+9G@#0F5|Y^p>gxfNV$+ z^cbla6cO>t+FFL7(_)Zp5R{=+%o3-7N+iTfoyo$qc)C4#Twu$|d8$usIr&SX-Nt;1 z9bOs$oPmS-u5vQy5apIbCOBSWgwiofOx{UWQv3k@U-=jq>Z4|<}eYpTwKeiGa2QG~>n0Au-%;wxC} z-^R6p5s0MzJ~WT3z@Q3kPZI4>*U2d1Uz2U5^QNTW-)XkuE>aGbiV+$(J^A3#Vq>>W zNO(h1yrVbJ(7i#$QI8=6pym;LXat%H8i}{0+*yyxWpvw{5{Pn5q~XOS@hY|Izu{qT zGTy_z+qN_85i_W(+v!0jT|S(x$gL!lQ;&H5l^rnK>j= zx4$-5XlvrbOesD{;(}TEyuH0yAZ^qkokBC@T`q)-=X4#k`dac$aE5RUF3Dv9lszX} z`e*x)#F*jr;}QphR(n?eLpZem_p*qzpNJpZvdL?Em9h)KrNhmVOHkhT8ia7VO`q+a9fKEM8<`%>DKBw<}EP zEOYdgO#DP`Ru7To@n&rvVs6+Tq&Lu;i$(KYcLYejfPKRbQLX%LgEwA4yk(W>!ip~Q z@IdcFk?M7P??WTS=?rP$@zj*NiCe1$p&#add$u)KmE8TcF*V;(TI!{#fiXGL?t%`h zZ&H7f>0mDhb5SQPZDJlj*74|PY<*gUaZ4k;z!Ip#{`qrDhcF z6SDUCS@A$evXJDHgGF#tod+ON00V@qUa^u2T_*DKqECkPTc~;RKN+Ex*jT`zI9UC{ zseOc>K~g2`0!EL7{@E5m)lc>9i6kgzmoz(QP&iTm4ehZ3mUhgzsoyR+%~r@Pv7}oj zk+M}HvSLtg8`^o;c?>pP);)erF^DM+9%!U=OBj8IuOVXPbtRqJ<*t>@-v~mId^j-L zKVvfPguOb^CrU}2OTeA~iVmKD5hv=FmrcdruQDTj3O_$8{XnASnhpjN z3d*t#tXzKI;hT@Qt34D{kEp?*of#NYDnnH8)OWfa?RDY^)he@J6cHAL;Kj6CJom;v zKIZ9ib%!L?*27=&44;(*uWWB+fE}zx;X=&?U;s&XOmvcG)Hsj#_xo<$PanxR6NSl{ zT;JqR4l=6QGK)EpqmQV<?+TiEJ8;~S!(=4R zTd#NtZ=CkSID0Z6L$(7uPUB?V$eH;ezflruT%6qua|99wp00>Whvz2mjg7Y7dH%>s zNSMjWzT4kT7d{+MjqhKbEY$H)GU|QT?|wJg4LB{3B!ceD;lnskY!&)=&3!oYj(uy5 z3lCS$;_wTGS`reWBI^>XVGJxX?&EQl&Kn1V{QzFlo4UxpZ?xc)$(x4LaC@?pCWkAh z&_Hk5U8foBJ>5D~G-UxwZA-!d7T(`=PasF%;!_V01&dlKkMV;KMvBgs^}`r@W~p2P zj{_@j&_t#MCm2?sHO_bMm6@No(+vdtr>S+`9=B|_+-#-CH|C5VQ)*kGxs!T$z({9u zO)HJTng$2&t2gXK+q-;H>fLvA523)5>GqBMd4T@ZI6_~=!jOXzV{6GxPaHhF8TAks z>&?}T?d{9g2XlYM?$zo_e+`ZEW!t?z+Yj*Zc^RrV)Z{?iw@U&}%vMCAHvN)%UNOQ( zT>&WnK-K8uO{jrc#=Z|;n+YfGHJF_VwEH|w%qPV1CpfI*$q?I15<>eAwJ4(PZV1@8Y*Z(F(%~ zTmbv8E{fQ;h`n^nT;4A(e%-O-TU}l+M(y>vX@;G>;Ez93wz2(gZZBDE)MQee?~1`- zzT4ilDs_dSqVZ0GT4E7=t~49~2rSlm*$h(3cqA~v6q>di9_G=ZpcW|=U|iM6EC04&dWc#V*>TTCil^j=4*94$CU>T`KbW-pDq85@LWs1YJ>YPqz(}CBFoTiWogkT6QwH$w^jjc>fXTX7_eVC`^sbY>q5CqKKhhh` z;dgT66r}_L_)9IEEufegu2yMU^{t{i0;lb=}IJUEl2_V>TPywqlVa zyISnzRvs_L{dtd8x4W_nt#;EuVCmGpWAuR22m25|70enI3klMl?{&n;lKH^Y=v)P0r>hX_GN}UsFp%%E^#b9Nl>i8-Q z?IV}r!ab4!+0&~`h3b)rT>93#R4A@#(G93EXhX08p|xjBi^AASSpJep$QU#8nPf4& z+sU7fF;dNZ84|H@Z)mZ2gbcMm4jVeYKeydl$HM$!LH_qTE(b;N2o~$O@We3XB1r-1 zRaN7!n$Kl={L@_q7!|u!GV**o1rvdr3st~_Qv_=^?&;Ztra1FnyZ4o<%RSz^UBQi^ zRTZtKavBtv1}OUdCIK&36+S-fzW4R((cqZA6A2*^pGTL;3Asi{>;ri~?!=j$qEp+8 zH#3xvih=!J+BWf)$Iw)Mx4rflL%1V4ET)nE6mtZ+>M|VYT-TY`gNQA;#_wmQKrTp! zZS*Gb*cX6-a7Qlx`%BZ;miOiTg>q=wcG*rd@R{MI!`JA&_*$FDHG$eiWZ*di#!?r} zqRvp1vHbhH2&#UURuihCC*a;JTfeG?&r+Sxi{TtEo`|yd6pSn~U@3zYsqVY_qOJT< zg@3Jv=hBjD^Vj$2QWAZzzir5`v+E{s$(ygZ3{q@9*ao&Btbu`>=z3qYdkAl)^_{PP z$SS(B<^@wTa2ug^&^d*M>dhJ(elbfQUQw+aJiCIbTLJ7G5Z4HltJwo0`km$&3Wl&F`i%q-M z(S-`z&B~!aYbKtE;P2&SK$*xym=8=Q+YKa1x~7OI(Y3Ni0s`zkxBDeZ@@B}Oi)h`e zFOMh)gAOnrsHnCLT-Qvpm!nv_>sXdho;FDiZ)$toMx%5Dm$gr+B%9L53*g-%tyncn ztd`95jmL!x781Oirf2%ae&JZ#e!KWKdaaEs^L}Vg>xxnJ6zC8sc7U@cK1ba{VPqn~ zU@_EN?YPRdP87N7qMUE2tZQEFP{)v!C)o|aI4(B$e2%n`%4l=eQmr{hyo2V@0%0yt zhNJ+LYIb==_N3|feDGlWiYr^r52OsUe4H^9IMVU;Y4z;NRexru2>JGMYko{+>Mp`h z!Xm@Ikzf~ad)9Va4TrLl4#-;@L>8S3FhrL|`@EGSG5_pb4oBQ_XMD+WU)xd0iCb$d zcpC_Z`36NBX$rjCJH1UQljavuVU9~F5_xn}t!x#ofHn1CFV<{eS;wOaX*ZL(Khu7g zEhwMTwu95?OM|Gk-$99^2iJHTi6PYUOL*v53Y~!Nx2Y8eW~21}DfpbQ)i2P4eE@_}jXhWj!yXkXB(F(C__M+t1MHb|akmz=JyG zp*Aqstsf4z)C9}??YDB~X3v*bAJLzcs!MpzW6)qXv+fUR3mo~mgqS)cfya9n6qY4Q z4x3JxNbZLYiiJ`R4C(3Dy+zI)0^O#xONWv8tFtMi!6B01PlO8|}am?RH`kqMIwO8us$6b(3H-!9lfu*?|D;SUDWB z@kcYzG-Q*12$2^yNA_llpKmhPoVh`)B7HgNE4R}{1;qZ~H28RlpHy!~|Ajq6fn?~g zoF9$Mgb#Y2`t<_Go$)gX=aFl{)DqlAp&i4`K(P9-yQX!T&~)ODmKSun@0lcsOgoyO z!C0h@0*#8Xad*=v!;b=}X38xlEB3QGq=rdb=Watz#vg{?e$PhYAq_~ zs||!X+zFg%`axfJ5#OxR=Mu?-I6{QN=*HnlTM<4c!@THUYu11DXm9tewnWPPC7=0x z<>c@X*(Nm_ZFBxJ4GbIX`;ka0zTRw03z(=zWYp&?H`G&QbgfQxa+{Y0%@fGRJceQ;a!m&@Izr8Ff zhfb#fuu^?#ZpP!Z6I#6))i`D}!oMj*n3{Qx7Rxe~5O%F>rr!#fNb3RMK5+Q}m>opy zA4%W7gD>T0Jd7=rDeQFar9!}=Q|%rc7%w-AuVIUICZV-365wu5WpUCEA8cjWtYdqg zZH2S=P@kaUfN?NxonNeVB2-13+gwW`tv=^Y)}IRHK7CtI5JkWTwIf0P~bO+8ho0SXo$VbUP>xvls)@!28S`Fr0R~F_X!94051G?Ew(%=TghxqCb#^z~^J_ zyMghrmDNR{PZ6u0s9()9+WR@j<~|0UU>gS_n+wS|I0 zDfTOW*J98_mrm!%m;l2@LrcocjG_Q$B8KSKTXF+vPBnZF4nl!heup>4n(6k+cSyBY z@f{85Ytlri8>RFtuEAnE$Up}ab-sEz)0;6ZE%gGl9=~}JPd%qu&TV#r?sLIy!_U6l z01VEXu#ca;Gm}4LmwbJFUnB~;GsqCn(nME3pligzg+6>)oyQRaY>vB?o&P1U&{Iki z=4J;pq~&zETv6by3}jO*w%G3oc-{YfzBvE}8h5yy%-{W)>BE-|>9^;itaTXy=ERtp zj}Y`nYj7Kk%Mk0hH6^tXK0}(WY-RVwCfJ}?<+3=Y*pjO=XsQgL?;Ku{SHiW^xXn?r zD@ifsUARs@*VJ9v)o%(Bx!vK=1}Im=w=k zf{0)&tR>OAWy!AJ?*$9D>Bvi#-Z~oS8pM!Vvh2?ttjEJ@9WfWt#1Uj2^b7dI-x%Ks zPfkfW0?c!jkT9B7?ew_mgs|a;Vs*z(E2swr_xt)CAkpS@4=+YJp9&g?M4m<4Z{yr@^unUTzg}JM=FXS?)loo=v+x9>ev|c$d#`Sd;0D>^@h8 zM0?lMwzr1J{9w5)e|Up7FpD=DlfI&K1M)|Uw)*NHB7kfLhi)yzn^jz9*xy91e|a`T zt-D+|J?;(taXNG9-ZmcEkFO<|OD?p9((Qd@dCi8@p{UwiVBfyUq$sr^mwvTx+?Cn5 zn5TZvTJ?IXd&&CyEKnj!d@Bpg14B?0@?}5=_y2{toX#vBN&f!LY8QkEb0Z=mZ@9+ui(`!~2b&=lj%gN`AbT z*Rim-ioKo|%3-Vj-fsfh^^=%aesAoXjgf!pB)=S1)S9XDiJgYUfRpy4vGi~8d|-Ao z5K{NuO{p;#HQV=GDlce{(2-l(JbA^flA9TT5OKn#_ieZyctSqIU!Cm6>0@Da{a(=7 zTIeDwhK&6q_Q1ThlO5)`zjU)uJX)t?*(J%&o0BEmenP)XqMjsA3YkLt6bPkTl6aJ{ za8Mn#y14{8`tH(E?BFMmWgM-|i5AM>k`ruhrhN$&*vHCH1N|jTk3+=6COZ|7!ZxQw zS{%|tV*jR1R~V!-rq9NU^;RA*T>ZZAR}pH@&yYqp0>S_+v^S;AlTfR|flkD4-y2_zqFfwbt_uQl0{Mr8e` zwFYx&+mK${*4@5I%ImE9gL6pO z^M&}dh>SL>YuBx!+u5}t_(=i9)z0PBQM_p~KEO6_*@k$ViP~p9?PBLUW$*)+td^-d zFgLs}7z&6-UDhiMRp7!2!r3<>u~^Q}aEaj-m;}7RD-~2AAOxWQeha|uU(P2m+5Cn# z#D%rL#{L^GdBN%GWPolt@XqC+8bKOrXrbt^SUdbZSFCX7H=X{MOT6-!EH?V5g)oe; zk+E*yqeT0?Br!ex-GF>XO8WcGPD24YJT?=EjNt#%&RPCN8FYOdkXS@IWs#1hdud^p z6iGp(M3k16PM1bv5d*!SnKdz1Lnlb9Uy;nRCwX zTVh<3CgUZ5b6CKF&WD90z~YPxKIaL^eZPg>snl+xxPZ}MOcVD%6Y(lhq!fesF7plX zl%I}T(cu(wrz*}_ULmKw%t*jl-}9nYOG%fph78B@Xmj)wv9169_Gbi^InE(FDKJ8QyJcOVS zb+E-YNDdHNM?-{KAwt-8K$W{ZCNIcm%@ABZ6AKUdIxDz%beIeZwOlm@3q>8O>2XIXO8euQ)Ki#AL{Lk#Omo zcj*z*0|6R&+peSZUx@Cd`2&Sk-U(XbsFx0%7oqMFA>Et;0wj#Nl9fOAx5|W^@WwN`Z*Ups2OT)wzG=tZ!8MkAwRMv9%SEcLX(X@`&;x#ku+95Auov9v3 zT&WW05OnMGVt~T-Rkye*SFJax;(WmMc_4#HNxmwqAczyzv8alpl~3L$2lL?tLOPXN z8^W>%F2=?Aw^B_%`I96f@HqrIS%c=jCKEp(;<(KxfOh-8sp*x#cV= zo`)}eQ3fM4#^d@UlUeqRi4IfndSn_`~C&l)<}Qr zmb@7pr!7JHzR!U%wIi%&ZG|c@y+4?NtTZ{AKlz0$pVrv^cMj&lX{p=AB60U*XoSmN zpRLgjij%j{5>0QQzda1RpB{UObmO<;;M7bDja35D3H_6|hbiHRnIqB*>v{&x`r1K} zA+aoc`&f5~aJ)@?@-Br7@ z=j-{Tt_Q_C;!`D$yD|K$?oA*n1|H$*RAYi*nUhZGhaV(ut-6HuKp2t zMt)av2mFLv<&t5zuJ_BA6a2K}ROq?C?S35Eb&qUbPWe3@dY7JejR%$F5SpdDli_^t zsS%_Anz-p#?4`&_}~BLCn6K8HO+=c&BUpKT!?78Vu=q_oObktpn5{l}!V z>LiKUeZP~PC(>ekyiuM2DG<@B-e{2M>5~+4$ciQ=O^${$s5W_*jVeP?pAh7K=9Ir{ zMvNlL;Y!!vvx8rZBEx{}t=NaaXK_isl8KAw{uF1dnoH!}e9Z@Q+#mgQ65dHp)n1S@ zkU;nvZ3#NEtMeLcbu>ag3m2rh#ShQUwO8v0G{Xn)X=n~IEO5;A3SIpeh`BRZQ>l*e z*njJx`p*plyPH_&Tov-`Y-3oqd?HtE6q7<_rWgq-$a`F@JUq8`J&EOL7=`869W5RG z`b@6=dci%aPx4=8d+cLTX4h&w{E~5qNQH;ro8kU_?nMNojhtaY_+7dC^vI0d^x921 zpF>=~^&0+?JO(sdYE`nwCYNC4wcyFIOLBa6Uid28KI-jSlvHhofe5T+*c^JDY$3AE zd(uWHx~v%T0{`u?Z7_HZ=78XZauLp(b1g z#rc#*L)fOx)XUJiU0eD|-r(XoUE;`s!?x-xnE8hf4V@0+38YStw9qkaI$Xuk&(*GK5I$^!TFXSvv(7mP#sj!0LiUi%yQ^0@qOMr~&L zm!{-6dNX*0(OK1%T(v11M%BGj@mz*61Il^jZ22{3y`)^CTnu8cbtRLFTAJTS`s>&_QfaZz0|%rLr-FJZixVw+?Vq#JzksMBh${=%(|e7Vw-ra z5sV?OXKZYO`xqp?85I|&t*(9>%`j0L%($i!@Qo}lqowq+2fxlzFpD<g8w3RAnrC6y!hD%?OruB)2%L zmzEGHM+EJj>1BUCluK&2NLOdR({_^NAE!NG*Y9hEmoFv*5rbJrn1Y}d%!jHk)vm?8 zC9ZzC;>=Q!GBAnQcF097bGMzwcOP@B=Mn$;Z)CU-ZY6I~U^=(9C7PLLn~yju3^g^+ z*X1=96R(3)(IO`&XJKJ+Oc>0rGb&QyMtt#t42UMF1xaF@N)-hS!Xm|!g&>?oAI$&QT!_Ee9D~q zvulf`10vuG=Z!oYV}p0n)>wXA;1T?VHNCH78)9Xue z6-E=LMA`C4_7Hc81$&qPQ$3yFD!mznL~re42?;EcU0?^vFHEoUxe9&%X4&?L}I9EynwfUvEGTCW6}1 z;EQju0jwKFOiD@$3WWkn=+wc_Cubit!{oQt*sDH@)Fl87X-D%!p1>)*qw&=r)+swx zo7!{O=aa9jSnEtw-unhXb_K{di;Ig<89K&5XgH|9QPDal>#;q)ylhT)*S)mZ53icy zXyUJ5W1NkafT?3f5RODVv~sE8*LZOyJ~BV%H#eL$+G*Z%j z{0IaVLQ}80$}(*_W_#aHWJbBjbT8Q#;N7(s5OKcULH^=WJP;pxk(oARXst6Yq!h;& zjbe!qnFg)6bKJN(;R0MOwuJPyfmg><-V!cmHfSCRL90nY0eP>yWViJh3#kw4yVh5$ zyOf?^?`CUm@QK1^T~w?F&{!o+TmM|S&6*{ZNIP{v6dl`yPw*1sbGrZh(!h1uhlGZg zp!{+69Ti2bLymItJ;UjE20V5*H+R8wxbb4T^_~@UT`;@Rem23lfdUg_>$%gZLYR=^N|oaM#t&`g+MPevyqBENM($s+y&@wVv}?XNm91 zz(+svCl$#Q6@Nvc?~}dwN4WmXSX|N7wPH4VU=t$50Mc`J!$c(T(!2gXNC$NI1e_mj zO$QcGK26LLw!q&G#@R%rt#3odPJinfH1a(W;SY=5>O=n;HGO!=vHGkkes4z9Xc;Rd zoc8yG%w#DZzVhY%ELB7vd0Op5YE^Dh2FY@@cvhgx&JeXhpI7Z1P+6IKcgec9qOc1- zi@J=)r;a%B2>*IkgG^QR+|oekQ-A+1LV#!G(R2S#i_S+En_mdw+haW zb`y!44Xg+vQ`r{FCN0Nl{HX>B>o=QBSs?XG@N$Ev;|64Fipo>9!`2m~Eh6{cuhUVD zxoRiY+{ocQCEv7+|De-qm3xb)BB=A0(xs#+Dr?7qq^{KbT|}?5U)+nOe@R!7k3vyq zKtDT11a~#?rRXQD2XTrQnTy+01$?3Fo1IhXHIY@vT&4TWgNeaemY2|~s{W_pzX-AK z-CVcX%vlAmf3iRsI$53F9E_v!y?(=KXEz5QqHX?R)t-c=4jZq}etH-Me&F4F!73R( zq~5Qt9*t;i>m8}A@|HPcMISpcKNIHL`Ub!5QP)*qIUsrO;i_`&G%0Ro>|#PeE@+lLlKimsQSO_E|n*l1Gm3a?`0)X>gp@$PKhSd1Pp+ zx{t{UKmLBwF&*}99%~t^{Mppc*jdi5itPKTq`=7Q9ka#!KHim&RqH#dT@bCYdqe0w zQQlkh(2E~_wf3>i40?Z7Lj7TQmL{dBF<4sBE+@=^VJKFN<6I3yu)v(!9JA_@-0eB(wcnpEjzS^ z|MK`a-dKEm&Rc6)c5)!kHy|bM{1|+ts%YhV+|2YvhL{h-;6(Sz9ZB~*Yq=}f-E10- zM!D_E{0au{l>6rF{Ex;VmTz!b-Vy+8K>w{bMbxQ$Q!O4{?-camwM@cOx&E4dCdV~i zH_1Cs$S zdv|Y)G{aKo5b|WGlBL9O6^bTumUo(?%=j_jk}2DJ4?F$*JysPxI>oU4^l3R_4r`od4*2WjYJ>9feS*y(nESuNakB_S=-NZA-zXJBBUl&$}4+@4X-T5IO% zl0~ug0)zHeIGNC0kjc})FF7SFyAr)<9U-}@S6ezeEL@?FZvwcwNSEy<&%M=-LdE|i zHTom;RQZlc?nzfWEzH`DQNVX7r3RVaS2=}DjN}7mo-5cm)~2#0v9md;yLHZM8l-25 z0phB51gvaq`5LK}z<2l_ZzCKK-j55I=h}*jj{ql#`I;GbI5f<|(^`~)!e5v$Z_l8w zJV$nn)qKA}RH?O(JmtPYAU?I$gz=$U>5!7DY8;~+y=w6)UB~I^sj2qFQqJWOw_cQ# z6uNXB8t}LDy#1yFR2s+fvCv*m$L)fS<6=i`sVw<`Hc!ijTw(a@>q`_0i;F9Nf8S-( zX=rFj+Jj&%Un47U15vH#@)yPsUyaYgkQj(IJ+L13QaAsKm3~$dR!JnJ< z$q;vLuFWfaUe_~^=;RW()4VMJ-drjWi!nil4flNQ|klzc29JZt11h zZgS7_Nv}QYe{$dFyn4$J3eo`JNu;8pUDQkIw{JvlAdzJVIL=#{gR*DN)79Kr#yxer zw|zgE(}Nr0|D^|56_Oy>C1jY?LIa^0B+IlIxGcB$uc^O_f-@7I8;*v23ob6G-8VBT zkIQW`BJ(E~zeh=VYaDw-kGaCt$&M-5$4*4kJCJmfYLiWe%htN>z&hPHMOwt^zmK2pDo@8 zI5Lew_fC$F#qk1i?0($r%h(ny)4vs@&)Y)@0U*Yf7TSFO>?dWlmW2W&1qaweBdADx zR6|`InL#Q!v)wq_2g}q@Q`bbRUOf}RzxB}bC5mxDE{QA zz$%)#JRy4la~`U|2sZRuP-nLYSqX$ZmT%fFR3BN-P9&nB`H&hc5p@sCnmt-m1vHa< zDW_R_ZRinygtrSQKZJ!ue#I>St#J%sB#A#>ur&ZMPGse0-RiEcu53eqg`8*&9(?RF zrm3kpXp(0=mAKSQgpHjZZ&c;fy!d{vg}-075TQ^>Tuz7`*`2 zUV9i3J<$e#t18Dpy5LU~K`0-s6E^}vhX=l*5;SW3W*=Nxe=?$$xR4$!g6`g!-V`=glpX+Ci)nUWNjx_0sD*nBV|Hn ztkfLWoyUfTbfM-OLzyfk05W#IQ`T}gIB@inG4~> zP$At+S)W%*N}>B|b~pmuw#Y5EK6c!r8wBFw$b)QF%P5sC`~XJ~u0$G)H~YAqF0rc2 zVGtoBXLv+3<7LlY|C(=5Q}pr!$+7p(0^f)^P?355{k;^y26f_nT%-@cp6uhM%R#y6 z^!}Uifsl~_effFzM?~uH&be8d9jD5k#f6FP;wz8r17_##cVoc3_;`4P3e|RF;o59V zOG{a_j=kK1O`$K;DVMP`U;#?0H`QH0gOYH6Q~>#OLj-*3i!+)m|2nX^Yy~MCDW!DW znn6W>*E+M@?waNk}=Kzpc^;V?~z?G4~nB{J8vwWo)Rg9EP#FVT}C3c8CA4u&Z0m2GUn2H=O` z_LFxeA2GOv(XE2PhO7k%;$PhQ@IlfQ=d2tF_^JAZkCfw)rDDbeqz%U39)~*-V#|?X zG7WuEoP|7qj2C=#b~L7ayhClU>Ya|2yzxr6m4rZbCW%*mYR|dl3ZKLEV705^a4Q)& z)}VRk+@PNYjT0>9%bcFOUk{It26$&tw=RD$&x_#%^B*lEEQEO!Ae%Sj(neO!n#0H| z;sYuu)IOeBjcQ5r|J*-t ze3kf@DYKX=TzgFx+7axqs|Y9CDNI%>9_=p@fdwL4m&R01+flXT*(_>(EAA$3IKAd`3?0 z72}Y{%E%C3fnTMdrUt>GVUgrOtE23wgZf=XZVB*xUTHsol+!U&S%K45+H)78TcC-{ z9)FxL9Zv!Vmn)jQzWYF5pNjPe`Jb32T4z1-uf3L50Y6fK8a$%UqId?N#O4b($A?E8 zldCqM9v9u@5pLh@;{vebog44;-w*z4lK(D-|F> Date: Tue, 12 Sep 2017 17:03:17 +0800 Subject: [PATCH 05/18] fix bug in prelu parsing. --- python/paddle/trainer/config_parser.py | 1 + .../protostr/test_prelu_layer.protostr | 45 ++++++++++++++++++- .../tests/configs/test_prelu_layer.py | 2 + 3 files changed, 46 insertions(+), 2 deletions(-) diff --git a/python/paddle/trainer/config_parser.py b/python/paddle/trainer/config_parser.py index 356e1d8b6f..4f68a89534 100644 --- a/python/paddle/trainer/config_parser.py +++ b/python/paddle/trainer/config_parser.py @@ -2034,6 +2034,7 @@ class ParameterReluLayer(LayerBase): config_assert(input_layer.size % partial_sum == 0, "a wrong setting for partial_sum") self.set_layer_size(input_layer.size) + self.config.partial_sum = partial_sum self.create_input_parameter(0, input_layer.size / partial_sum) diff --git a/python/paddle/trainer_config_helpers/tests/configs/protostr/test_prelu_layer.protostr b/python/paddle/trainer_config_helpers/tests/configs/protostr/test_prelu_layer.protostr index 64d227565f..94ad56cab0 100644 --- a/python/paddle/trainer_config_helpers/tests/configs/protostr/test_prelu_layer.protostr +++ b/python/paddle/trainer_config_helpers/tests/configs/protostr/test_prelu_layer.protostr @@ -14,6 +14,29 @@ layers { input_layer_name: "input" input_parameter_name: "___prelu_layer_0__.w0" } + partial_sum: 1 +} +layers { + name: "__prelu_layer_1__" + type: "prelu" + size: 300 + active_type: "" + inputs { + input_layer_name: "input" + input_parameter_name: "___prelu_layer_1__.w0" + } + partial_sum: 1 +} +layers { + name: "__prelu_layer_2__" + type: "prelu" + size: 300 + active_type: "" + inputs { + input_layer_name: "input" + input_parameter_name: "___prelu_layer_2__.w0" + } + partial_sum: 5 } parameters { name: "___prelu_layer_0__.w0" @@ -23,14 +46,32 @@ parameters { initial_strategy: 0 initial_smart: true } +parameters { + name: "___prelu_layer_1__.w0" + size: 300 + initial_mean: 0.0 + initial_std: 0.057735026919 + initial_strategy: 0 + initial_smart: true +} +parameters { + name: "___prelu_layer_2__.w0" + size: 60 + initial_mean: 0.0 + initial_std: 0.129099444874 + initial_strategy: 0 + initial_smart: true +} input_layer_names: "input" -output_layer_names: "__prelu_layer_0__" +output_layer_names: "__prelu_layer_2__" sub_models { name: "root" layer_names: "input" layer_names: "__prelu_layer_0__" + layer_names: "__prelu_layer_1__" + layer_names: "__prelu_layer_2__" input_layer_names: "input" - output_layer_names: "__prelu_layer_0__" + output_layer_names: "__prelu_layer_2__" is_recurrent_layer_group: false } diff --git a/python/paddle/trainer_config_helpers/tests/configs/test_prelu_layer.py b/python/paddle/trainer_config_helpers/tests/configs/test_prelu_layer.py index 2e3057f323..aae90fab32 100644 --- a/python/paddle/trainer_config_helpers/tests/configs/test_prelu_layer.py +++ b/python/paddle/trainer_config_helpers/tests/configs/test_prelu_layer.py @@ -2,5 +2,7 @@ from paddle.trainer_config_helpers import * data = data_layer(name='input', size=300) prelu = prelu_layer(input=data) +prelu = prelu_layer(input=data, partial_sum=1) +prelu = prelu_layer(input=data, partial_sum=5) outputs(prelu) From 8a2ff350ed041e1805f860ec48d2a2c5fa4f9550 Mon Sep 17 00:00:00 2001 From: Luo Tao Date: Tue, 12 Sep 2017 18:28:17 +0800 Subject: [PATCH 06/18] simplify the python/paddle/v2/framework/tests/CMakeLists.txt --- doc/howto/dev/new_op_cn.md | 6 +-- .../paddle/v2/framework/tests/CMakeLists.txt | 43 +++---------------- .../tests/{mnist.py => test_mnist.py} | 0 3 files changed, 6 insertions(+), 43 deletions(-) rename python/paddle/v2/framework/tests/{mnist.py => test_mnist.py} (100%) diff --git a/doc/howto/dev/new_op_cn.md b/doc/howto/dev/new_op_cn.md index 58665e9f2b..07dce05df4 100644 --- a/doc/howto/dev/new_op_cn.md +++ b/doc/howto/dev/new_op_cn.md @@ -354,11 +354,7 @@ class TestMulGradOp(GradientChecker): ### 编译和执行单元测试 -单元测试编写完成之后，在[`python/paddle/v2/framework/tests/CMakeLists.txt`](https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/v2/framework/tests/CMakeLists.txt)中添加以下内容，将单元测试加入工程： - -``` -py_test(test_mul_op SRCS test_mul_op.py) -``` +无需修改 [`python/paddle/v2/framework/tests/CMakeLists.txt`](https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/v2/framework/tests/CMakeLists.txt) 文件，新增的 `test_*.py` 单元测试会被自动加入工程。 请注意，**不同于Op的编译测试，运行单元测试测时需要编译整个工程**，并且编译时需要打开`WITH_TESTING`, 即`cmake paddle_dir -DWITH_TESTING=ON`。编译成功后，执行下面的命令来运行单元测试： diff --git a/python/paddle/v2/framework/tests/CMakeLists.txt b/python/paddle/v2/framework/tests/CMakeLists.txt index 6b22c00082..4d7664469e 100644 --- a/python/paddle/v2/framework/tests/CMakeLists.txt +++ b/python/paddle/v2/framework/tests/CMakeLists.txt @@ -1,38 +1,5 @@ -py_test(test_net SRCS test_net.py) - -py_test(test_scope SRCS test_scope.py) - -py_test(test_tensor SRCS test_tensor.py) -py_test(test_mul_op SRCS test_mul_op.py) -py_test(test_cos_sim_op SRCS test_cos_sim_op.py) - -py_test(test_mean_op SRCS test_mean_op.py) - -py_test(test_protobuf SRCS test_protobuf.py) - -py_test(test_add_two_op SRCS test_add_two_op.py) -py_test(test_sigmoid_op SRCS test_sigmoid_op.py) -py_test(test_softmax_op SRCS test_softmax_op.py) -py_test(test_cross_entropy_op SRCS test_cross_entropy_op.py) -py_test(test_gather_op SRCS test_gather_op.py) -py_test(test_scatter_op SRCS test_scatter_op.py) -py_test(test_fill_zeros_like_op SRCS test_fill_zeros_like_op.py) -py_test(test_top_k_op SRCS test_top_k_op.py) - -py_test(test_rowwise_add_op SRCS test_rowwise_add_op.py) - -py_test(test_default_scope_funcs SRCS test_default_scope_funcs.py) - -py_test(test_operator SRCS test_operator.py) -py_test(test_gaussian_random_op SRCS test_gaussian_random_op.py) -py_test(test_uniform_random_op SRCS test_uniform_random_op.py) -py_test(test_recurrent_op SRCS test_recurrent_op.py) -py_test(test_sgd_op SRCS test_sgd_op.py) -py_test(test_gradient_checker SRCS test_gradient_checker.py) -py_test(test_lookup_table SRCS test_lookup_table.py) -py_test(test_scale_and_identity_op SRCS test_scale_and_identity_op.py) -py_test(test_sum_op SRCS test_sum_op.py) -py_test(mnist SRCS mnist.py) -py_test(test_concat_op SRCS test_concat_op.py) -py_test(test_squared_l2_distance_op SRCS test_squared_l2_distance_op.py) -py_test(test_reshape_op SRCS test_reshape_op.py) +file(GLOB TEST_OPS RELATIVE "${CMAKE_CURRENT_SOURCE_DIR}" "test_*.py") +string(REPLACE ".py" "" TEST_OPS "${TEST_OPS}") +foreach(src ${TEST_OPS}) + py_test(${src} SRCS ${src}.py) +endforeach() diff --git a/python/paddle/v2/framework/tests/mnist.py b/python/paddle/v2/framework/tests/test_mnist.py similarity index 100% rename from python/paddle/v2/framework/tests/mnist.py rename to python/paddle/v2/framework/tests/test_mnist.py From e76eb8534d51f3722d8cced2ff5a69beaa0e6515 Mon Sep 17 00:00:00 2001 From: chengduoZH Date: Tue, 12 Sep 2017 20:11:34 +0800 Subject: [PATCH 07/18] fix SwitchOrderLayer forward --- paddle/gserver/layers/SwitchOrderLayer.cpp | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/paddle/gserver/layers/SwitchOrderLayer.cpp b/paddle/gserver/layers/SwitchOrderLayer.cpp index d7eee6eaf0..e97809141a 100644 --- a/paddle/gserver/layers/SwitchOrderLayer.cpp +++ b/paddle/gserver/layers/SwitchOrderLayer.cpp @@ -83,8 +83,7 @@ void SwitchOrderLayer::forward(PassType passType) { setOutDims(); resetOutput(outDims_[0], outDims_[1] * outDims_[2] * outDims_[3]); if (heightAxis_.size() > 0) { - getOutputValue()->reshape(reshapeHeight_, reshapeWidth_); - getOutputGrad()->reshape(reshapeHeight_, reshapeWidth_); + resetOutput(reshapeHeight_, reshapeWidth_); } // switch NCHW to NHWC From 25be0ede764583f851fc1863ad9d2d65cab893c1 Mon Sep 17 00:00:00 2001 From: Helin Wang Date: Tue, 12 Sep 2017 11:29:49 -0700 Subject: [PATCH 08/18] fix cpplint error --- paddle/operators/reshape_op.h | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/paddle/operators/reshape_op.h b/paddle/operators/reshape_op.h index 26708e72dc..873acf3078 100644 --- a/paddle/operators/reshape_op.h +++ b/paddle/operators/reshape_op.h @@ -1,4 +1,3 @@ - /* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. Licensed under the Apache License, Version 2.0 (the "License"); @@ -52,5 +51,5 @@ class ReshapeGradKernel : public framework::OpKernel { d_x->Resize(in_dims); } }; -} -} +} // namespace operators +} // namespace paddle From 5c4dfdebcb12d17b8fe3090b874a496ea38dfcf4 Mon Sep 17 00:00:00 2001 From: dongzhihong Date: Tue, 12 Sep 2017 12:12:48 -0700 Subject: [PATCH 09/18] add more rules --- paddle/operators/name_convention.md | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/paddle/operators/name_convention.md b/paddle/operators/name_convention.md index 280ab8d317..182c74e78b 100644 --- a/paddle/operators/name_convention.md +++ b/paddle/operators/name_convention.md @@ -1,15 +1,27 @@ ## Operator Name Convention -To make the operator document itself more clear. we recommend operator names observe the listing conventions. +To make the operator document itself more clear, we recommend operator names obey the listing conventions. -### Input/Output names +### OpMaker names -* Variable name is prefer uppercase. e.g. `X`, `Y`. But when the variable is tensor, its name should lowercase. e.g. `matrix`, to discriminate with other one. +When defining an operator in Paddle, a corresponding `OpMaker` need to be defined. All the `Input`/`Output` and `attrs` will write into the `OpProto` , and will be used in client language to create operator. -* element wise operator, math operator or similar op, please obey common name convention. if the operator only have one output, use `Out`. +- Input/Output. + - names follow the `CamelCase` but the first character is uppercase. e.g. `X`, `Y`, `Matrix`, `LastAxisInMatrix`. Input/Output much more like Variables, we prefer to meaningful English words. + - If an operator's Input/Output are not meaningful words, input name starts from `X`. e.g. `X`, `Y`, and output name starts from `Out`. e.g. `Out`. -* we prefer more meaningful input/output name. +* Attribute. + * Attribute name follows the normal `CamelCase`. e.g. `x`, `y`, `axis`, `rowwiseMatrix`. Also, attribute name prefers to meaningful English words. +* Comments. + * Input/Output/Attr comment follow the format of `type:meaning`. e.g. `AddOutput("Out", "EigenTensor,Tensor: Output of XX")`. we prefer to more meaningful comment. Some comments like `The first input of Operator` contains no information, we forbid it. + * Operator comment format of` R"DOC(your comment here)DOC"`. if there is math calculation in this operator, you should write the equation in the comment. e.g. `Out = X + Y`. ### Best Practice -e.g. `rowwise_add`, inputs : `X`, `Y`, outputs : `Out` -e.g. `cosine` , inputs : `X`, `axis`, outputs : `Out` + +- The operator has one input, one output. e.g.`relu`, inputs: `X`, outputs: `Out`. + +- The operator has two input, one output. e.g. `rowwise_add`, inputs : `X`, `Y`, outputs : `Out`. + +- The operator contains attribute. e.g. `cosine`, inputs : `X`, `axis`, outputs : `Out`. + + ​ From d00e8a5f8350c38a9455b5fd604cac32e8b2cc62 Mon Sep 17 00:00:00 2001 From: dongzhihong Date: Tue, 12 Sep 2017 15:47:12 -0700 Subject: [PATCH 10/18] "add Op name example and fix format error" --- paddle/operators/name_convention.md | 54 +++++++++++++++++++++++------ 1 file changed, 43 insertions(+), 11 deletions(-) diff --git a/paddle/operators/name_convention.md b/paddle/operators/name_convention.md index 182c74e78b..8000dc8f08 100644 --- a/paddle/operators/name_convention.md +++ b/paddle/operators/name_convention.md @@ -1,27 +1,59 @@ -## Operator Name Convention +## Operator's Parameter Name Convention To make the operator document itself more clear, we recommend operator names obey the listing conventions. -### OpMaker names +### OpProtoMaker names -When defining an operator in Paddle, a corresponding `OpMaker` need to be defined. All the `Input`/`Output` and `attrs` will write into the `OpProto` , and will be used in client language to create operator. +When defining an operator in Paddle, a corresponding [OpProtoMaker](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/operator.h#L170) (TODO: OpProtoMaker Doc)need to be defined. All the Input/Output and Attributes will write into the [OpProto](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/framework.proto#L61) , and will be used in client language to create operator. - Input/Output. - - names follow the `CamelCase` but the first character is uppercase. e.g. `X`, `Y`, `Matrix`, `LastAxisInMatrix`. Input/Output much more like Variables, we prefer to meaningful English words. - - If an operator's Input/Output are not meaningful words, input name starts from `X`. e.g. `X`, `Y`, and output name starts from `Out`. e.g. `Out`. + - Input/Output names follow the **CamelCase**. e.g. `X`, `Y`, `Matrix`, `LastAxisInMatrix`. Input/Output much more like Variables, we prefer to meaningful English words. + - If an operator's Input/Output are tensors in math, not match to any meaningful words, input name should starts from `X`. e.g. `X`, `Y`, and output name should starts from `Out`. e.g. `Out`. This rule make operators which have few inputs/outputs unified. -* Attribute. - * Attribute name follows the normal `CamelCase`. e.g. `x`, `y`, `axis`, `rowwiseMatrix`. Also, attribute name prefers to meaningful English words. -* Comments. - * Input/Output/Attr comment follow the format of `type:meaning`. e.g. `AddOutput("Out", "EigenTensor,Tensor: Output of XX")`. we prefer to more meaningful comment. Some comments like `The first input of Operator` contains no information, we forbid it. - * Operator comment format of` R"DOC(your comment here)DOC"`. if there is math calculation in this operator, you should write the equation in the comment. e.g. `Out = X + Y`. +- Attribute. + - Attribute name follows the **camelCase**. e.g. `x`, `y`, `axis`, `rowwiseMatrix`. Also, attribute name prefers to meaningful English words. + +- Comments. + - Input/Output/Attr comment follow the format of **(type,default value) usage**, corresponding to which type it can be and how it will be used in the operator. e.g. Attribute in Accumulator`"gamma" `,`(float, default 1.0) Accumulation multiplier` + - Operator comment format of` R"DOC(your comment here)DOC"`. You should explain the input/output of the operator first. If there is math calculation in this operator, you should write the equation in the comment. e.g. `Out = X + Y`. + +- Order. + - Follow the order of Input/Output, then Attribute, then Comments. See the example in best practice. ### Best Practice +Here we give some examples to show how these rules will be used. + - The operator has one input, one output. e.g.`relu`, inputs: `X`, outputs: `Out`. - The operator has two input, one output. e.g. `rowwise_add`, inputs : `X`, `Y`, outputs : `Out`. - The operator contains attribute. e.g. `cosine`, inputs : `X`, `axis`, outputs : `Out`. - ​ + We give a full example of Accumulator Operator. Its OpProtoMaker should look like below. + +```c++ +class AccumulateOpMaker : public framework::OpProtoAndCheckerMaker { +public: + AccumulateOpMaker(framework::OpProto *proto, + framework::OpAttrChecker *op_checker) + : OpProtoAndCheckerMaker(proto, op_checker) { + AddInput("X", "(Tensor) The input tensor that has to be accumulated to the output tensor. If the output size is not the same as input size, the output tensor is first reshaped and initialized to zero, and only then, accumulation is done."); + AddOutput("Out", "(Tensor) Accumulated output tensor"); + AddAttr("gamma", "(float, default 1.0) Accumulation multiplier"); + AddComment(R"DOC( +Accumulate operator accumulates the input tensor to the output tensor. If the +output tensor already has the right size, we add to it; otherwise, we first +initialize the output tensor to all zeros, and then do accumulation. Any +further calls to the operator, given that no one else fiddles with the output +in the interim, will do simple accumulations. +Accumulation is done as shown: + +Out = 1*X + gamma*Out + +where X is the input tensor, Y is the output tensor and gamma is the multiplier +argument. +)DOC"); + } +}; +``` From 594dece99625caa2b5a0de9998755f587348cbe5 Mon Sep 17 00:00:00 2001 From: dongzhihong Date: Tue, 12 Sep 2017 15:54:36 -0700 Subject: [PATCH 11/18] "fix typo" --- paddle/operators/name_convention.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/paddle/operators/name_convention.md b/paddle/operators/name_convention.md index 8000dc8f08..59d4019a3b 100644 --- a/paddle/operators/name_convention.md +++ b/paddle/operators/name_convention.md @@ -8,13 +8,13 @@ When defining an operator in Paddle, a corresponding [OpProtoMaker](https://gith - Input/Output. - Input/Output names follow the **CamelCase**. e.g. `X`, `Y`, `Matrix`, `LastAxisInMatrix`. Input/Output much more like Variables, we prefer to meaningful English words. - - If an operator's Input/Output are tensors in math, not match to any meaningful words, input name should starts from `X`. e.g. `X`, `Y`, and output name should starts from `Out`. e.g. `Out`. This rule make operators which have few inputs/outputs unified. + - If an operator's Input/Output are tensors in math, not match to any meaningful words, input name should starts from `X`. e.g. `X`, `Y`, and output name should starts from `Out`. e.g. `Out`. This rule intends making operators which have few inputs/outputs unified. - Attribute. - Attribute name follows the **camelCase**. e.g. `x`, `y`, `axis`, `rowwiseMatrix`. Also, attribute name prefers to meaningful English words. - Comments. - - Input/Output/Attr comment follow the format of **(type,default value) usage**, corresponding to which type it can be and how it will be used in the operator. e.g. Attribute in Accumulator`"gamma" `,`(float, default 1.0) Accumulation multiplier` + - Input/Output/Attr comment follow the format of **(type,default value) usage**, corresponding to which type it can be and how it will be used in the operator. e.g. Attribute in Accumulator`"gamma" `,`(float, default 1.0) Accumulation multiplier`. - Operator comment format of` R"DOC(your comment here)DOC"`. You should explain the input/output of the operator first. If there is math calculation in this operator, you should write the equation in the comment. e.g. `Out = X + Y`. - Order. From 15fccfefb5afe9cf145dc045c7e4ecb6613d8b71 Mon Sep 17 00:00:00 2001 From: dongzhihong Date: Tue, 12 Sep 2017 15:59:58 -0700 Subject: [PATCH 12/18] "remove used words" --- paddle/operators/name_convention.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/paddle/operators/name_convention.md b/paddle/operators/name_convention.md index 59d4019a3b..a090e0b545 100644 --- a/paddle/operators/name_convention.md +++ b/paddle/operators/name_convention.md @@ -30,7 +30,7 @@ Here we give some examples to show how these rules will be used. - The operator contains attribute. e.g. `cosine`, inputs : `X`, `axis`, outputs : `Out`. - We give a full example of Accumulator Operator. Its OpProtoMaker should look like below. + We give a full example of Accumulator Operator. ```c++ class AccumulateOpMaker : public framework::OpProtoAndCheckerMaker { From a7e3325aade2b36816026cf311f70b393dbeae8b Mon Sep 17 00:00:00 2001 From: dongzhihong Date: Tue, 12 Sep 2017 17:09:35 -0700 Subject: [PATCH 13/18] "fix typos" --- paddle/framework/backward.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/paddle/framework/backward.md b/paddle/framework/backward.md index 0859bf1d9b..d0494f50d7 100644 --- a/paddle/framework/backward.md +++ b/paddle/framework/backward.md @@ -2,7 +2,7 @@ ## Motivation -In Neural Network, many model is solved by the the backpropagation algorithm(known as BP) at present. Technically it caculates the gradient of the loss function, then distributed back through the networks. Follows the chain rule, so we need to compound the gradient operators/expressions together with the chain rule. Every forward network needs a backward network to construct the full computation graph, the operator/expression's backward pass will be generated respect to forward pass. +In Neural Network, many model is solved by the the backpropagation algorithm(known as BP) at present. Technically it caculates the gradient of the loss function, then distributed back through the networks. Follows the chain rule, so we need a module chains the gradient operators/expressions together with to construct the backward pass. Every forward network needs a backward network to construct the full computation graph, the operator/expression's backward pass will be generated respect to forward pass. ## Implementation @@ -13,7 +13,7 @@ std::unique_ptr Backward(const OperatorBase& forwardOp, const std::unordered_set& no_grad_vars); ``` -The implementation behind it can be divided into two parts. Namely, ** Backward Operator Creating** and **Backward Operator Building**. +The implementation behind it can be divided into two parts, ** Backward Operator Creating** and **Backward Operator Building**. ###Backward Operator Registry @@ -60,7 +60,7 @@ A backward network is a series of backward operators. The main idea of building 1. Op - when the input forward network is an Op, return its gradient Operator Immediately. If all of its outputs are in no gradient set, then return a special `NoGradient` operator + when the input forward network is an Op, return its gradient Operator Immediately. If all of its outputs are in no gradient set, then return a special `NOP`. 2. NetOp @@ -70,27 +70,27 @@ A backward network is a series of backward operators. The main idea of building RnnOp is a nested stepnet operator. Backward module need to recusively call `Backward` for every stepnet. -4. Shared Variable +4. Sharing Variables - **shared variable**. As illustrated in the pictures, two operator's `Output` `Gradient` will overwrite their shared input variable. + **sharing variables**. As illustrated in the pictures, two operator's `Output` `Gradient` will overwrite their sharing input variable.


-​ pic 1. Shared variable in operators. +​ pic 1. Sharing variables in operators.

-​ Share variable between operators or same input variable used in multiple operators leads to a duplicate gradient variable. As demo show above, we need to rename gradient name recursively and add a generic add operator replace the overwrite links. +​ Sharing variable between operators or same input variable used in multiple operators leads to a duplicate gradient variable. As demo show above, we need to rename gradient name recursively and add a generic add operator to replace the overwrite links.


-​ pic 2. Replace shared variable's gradient with `Add` operator. +​ pic 2. Replace sharing variable's gradient with `Add` operator.

-​ Because our framework find variable accord to its name, we need rename the output links. We add a suffix of number represent its position in clockwise. +​ Because our framework finds variables accord to their names, we need to rename the output links. We add a suffix of number to represent its position in clockwise. 5. Part of Gradient is Zero. From 6d03ca33475b75b22bd306d57ac1d0aaf681dd46 Mon Sep 17 00:00:00 2001 From: Luo Tao Date: Wed, 13 Sep 2017 10:10:20 +0800 Subject: [PATCH 14/18] refine new_op_cn.md --- doc/howto/dev/new_op_cn.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/howto/dev/new_op_cn.md b/doc/howto/dev/new_op_cn.md index 07dce05df4..e3892849ab 100644 --- a/doc/howto/dev/new_op_cn.md +++ b/doc/howto/dev/new_op_cn.md @@ -262,7 +262,7 @@ MulOp(const std::string &type, const framework::VariableNameMap &inputs, - 生成库 - 无需修改 [`paddle/pybind/CMakeLists.txt`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/pybind/CMakeLists.txt)文件，`paddle/operators` 目录下新增的 `*_op.cc` 文件会被自动添加链接到生成的lib库中。 + `paddle/operators` 目录下新增的 `*_op.cc` 文件会被自动添加链接到生成的lib库中。 ## 实现单元测试 @@ -354,7 +354,7 @@ class TestMulGradOp(GradientChecker): ### 编译和执行单元测试 -无需修改 [`python/paddle/v2/framework/tests/CMakeLists.txt`](https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/v2/framework/tests/CMakeLists.txt) 文件，新增的 `test_*.py` 单元测试会被自动加入工程。 +`python/paddle/v2/framework/tests` 目录下新增的 `test_*.py` 单元测试会被自动加入工程进行编译。 请注意，**不同于Op的编译测试，运行单元测试测时需要编译整个工程**，并且编译时需要打开`WITH_TESTING`, 即`cmake paddle_dir -DWITH_TESTING=ON`。编译成功后，执行下面的命令来运行单元测试： From bc9e20d9ed399d6b21c31afa4c294b7bb7371e43 Mon Sep 17 00:00:00 2001 From: dongzhihong Date: Tue, 12 Sep 2017 20:01:50 -0700 Subject: [PATCH 15/18] "update img alt" --- paddle/framework/backward.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/paddle/framework/backward.md b/paddle/framework/backward.md index d0494f50d7..61c80635b8 100644 --- a/paddle/framework/backward.md +++ b/paddle/framework/backward.md @@ -13,9 +13,9 @@ std::unique_ptr Backward(const OperatorBase& forwardOp, const std::unordered_set& no_grad_vars); ``` -The implementation behind it can be divided into two parts, ** Backward Operator Creating** and **Backward Operator Building**. +The implementation behind it can be divided into two parts, **Backward Operator Creating** and **Backward Operator Building**. -###Backward Operator Registry +### Backward Operator Registry A backward network is built up with several backward operators. Backward operators take forward operators' inputs outputs, and output gradients and then calculate its input gradients. @@ -36,7 +36,7 @@ REGISTER_OP(mul, MulOp, MulOpMaker, mul_grad, MulOpGrad); `mul_grad` is the type of backward operator, and `MulOpGrad` is its class name. -###Backward Opeartor Creating +### Backward Opeartor Creating Given a certain forward operator, we can get its corresponding backward operator by calling: @@ -54,13 +54,13 @@ The function `BuildGradOp` will sequentially execute following processes: 4. Building backward operator with `inputs`, `outputs` and forward operator's attributes. -###Backward Network Building +### Backward Network Building A backward network is a series of backward operators. The main idea of building a backward network is creating backward operators in the inverted sequence and append them together one by one. There is some corner case need to process specially. 1. Op - when the input forward network is an Op, return its gradient Operator Immediately. If all of its outputs are in no gradient set, then return a special `NOP`. + When the input forward network is an Op, return its gradient Operator Immediately. If all of its outputs are in no gradient set, then return a special `NOP`. 2. NetOp @@ -72,12 +72,12 @@ A backward network is a series of backward operators. The main idea of building 4. Sharing Variables - **sharing variables**. As illustrated in the pictures, two operator's `Output` `Gradient` will overwrite their sharing input variable. + **sharing variables**. As illustrated in the pictures, two operator's share the same variable name of W@GRAD, which will overwrite their sharing input variable.

-
+Sharing variables in operators.
-​ pic 1. Sharing variables in operators. +​ pic 1.

From 885fa893324b3c51f676c706e09d5472822fffe2 Mon Sep 17 00:00:00 2001 From: dongzhihong Date: Tue, 12 Sep 2017 20:05:13 -0700 Subject: [PATCH 16/18] "remove the alt" --- paddle/framework/backward.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/paddle/framework/backward.md b/paddle/framework/backward.md index 61c80635b8..19e1850e46 100644 --- a/paddle/framework/backward.md +++ b/paddle/framework/backward.md @@ -75,9 +75,9 @@ A backward network is a series of backward operators. The main idea of building **sharing variables**. As illustrated in the pictures, two operator's share the same variable name of W@GRAD, which will overwrite their sharing input variable.

-Sharing variables in operators.
+
-​ pic 1. +​ pic 1. Sharing variables in operators.

From a90274eb5ce32025ed9492d969502cc3157cee52 Mon Sep 17 00:00:00 2001 From: dongzhihong Date: Tue, 12 Sep 2017 20:07:38 -0700 Subject: [PATCH 17/18] "update words" --- paddle/framework/backward.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/paddle/framework/backward.md b/paddle/framework/backward.md index 19e1850e46..0a6d762bc8 100644 --- a/paddle/framework/backward.md +++ b/paddle/framework/backward.md @@ -17,7 +17,7 @@ The implementation behind it can be divided into two parts, **Backward Operator ### Backward Operator Registry -A backward network is built up with several backward operators. Backward operators take forward operators' inputs outputs, and output gradients and then calculate its input gradients. +A backward network is built up with several backward operators. Backward operators take forward operators' inputs, outputs, and output gradients and then calculate its input gradients. | | forward operator | backward operator | ---------------------- | ---------------- |------------------------- | From 8778957cfc26a76c1495c406ffdfb66755503565 Mon Sep 17 00:00:00 2001 From: gongweibao Date: Wed, 13 Sep 2017 14:18:30 +0800 Subject: [PATCH 18/18] Add element-wise multiplication operator. (#3787) Add element-wise multiplication operator --- paddle/operators/elementwise_mul_op.cc | 109 +++++++++++ paddle/operators/elementwise_mul_op.cu | 25 +++ paddle/operators/elementwise_mul_op.h | 185 ++++++++++++++++++ paddle/pybind/pybind.cc | 1 + .../tests/test_elementwise_mul_op.py | 157 +++++++++++++++ 5 files changed, 477 insertions(+) create mode 100644 paddle/operators/elementwise_mul_op.cc create mode 100644 paddle/operators/elementwise_mul_op.cu create mode 100644 paddle/operators/elementwise_mul_op.h create mode 100644 python/paddle/v2/framework/tests/test_elementwise_mul_op.py diff --git a/paddle/operators/elementwise_mul_op.cc b/paddle/operators/elementwise_mul_op.cc new file mode 100644 index 0000000000..1742925545 --- /dev/null +++ b/paddle/operators/elementwise_mul_op.cc @@ -0,0 +1,109 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#include "paddle/operators/elementwise_mul_op.h" + +namespace paddle { +namespace operators { + +using Tensor = framework::Tensor; + +class ElementWiseMulOp : public framework::OperatorWithKernel { + public: + using framework::OperatorWithKernel::OperatorWithKernel; + + protected: + void InferShape(const framework::InferShapeContext &ctx) const override { + PADDLE_ENFORCE_NOT_NULL(ctx.InputVar("X"), "Input(X) should not be null"); + PADDLE_ENFORCE_NOT_NULL(ctx.InputVar("Y"), "Input(Y) should not be null"); + auto x_dim = ctx.Input("X")->dims(); + auto y_dim = ctx.Input("Y")->dims(); + PADDLE_ENFORCE_GE(x_dim.size(), y_dim.size(), + "Rank of first input must >= rank of second input.") + ctx.Output("Out")->Resize(x_dim); + } +}; + +class ElementWiseMulOpMaker : public framework::OpProtoAndCheckerMaker { + public: + ElementWiseMulOpMaker(framework::OpProto *proto, + framework::OpAttrChecker *op_checker) + : OpProtoAndCheckerMaker(proto, op_checker) { + AddInput("X", "The first input of elementwise mul op"); + AddInput("Y", "The second input of elementwise mul op"); + AddAttr("axis", + R"DOC( +When shape(Y) does not equal shape(X),Y will be broadcasted +to match the shape of X and axis should be dimension index Y in X + )DOC") + .SetDefault(-1) + .EqualGreaterThan(-1); + + AddOutput("Out", "The output of elementwise mul op"); + AddComment(R"DOC( +Limited elementwise multiple operator.The equation is: Out = X ⊙ Y. +1. The shape of Y should be same with X or +2. Y's shape is a subset of X. + Y will be broadcasted to match the shape of X and axis should be dimension index Y in X. + example: + shape(X) = (2, 3, 4, 5), shape(Y) = (,) + shape(X) = (2, 3, 4, 5), shape(Y) = (5,) + shape(X) = (2, 3, 4, 5), shape(Y) = (4, 5) + shape(X) = (2, 3, 4, 5), shape(Y) = (3, 4), with axis=1 + shape(X) = (2, 3, 4, 5), shape(Y) = (2), with axis=0 +)DOC"); + } +}; + +class ElementWiseMulOpGrad : public framework::OperatorWithKernel { + public: + using framework::OperatorWithKernel::OperatorWithKernel; + + protected: + void InferShape(const framework::InferShapeContext &ctx) const override { + PADDLE_ENFORCE_NOT_NULL(ctx.InputVar("X"), "Input(X) should not be null"); + PADDLE_ENFORCE_NOT_NULL(ctx.InputVar("Y"), "Input(Y) should not be null"); + PADDLE_ENFORCE_NOT_NULL(ctx.InputVar(framework::GradVarName("Out")), + "Input(Out@GRAD) should not be null"); + + auto x_dims = ctx.Input("X")->dims(); + auto y_dims = ctx.Input("Y")->dims(); + auto out_dims = ctx.Input(framework::GradVarName("Out"))->dims(); + auto *x_grad = ctx.Output(framework::GradVarName("X")); + auto *y_grad = ctx.Output(framework::GradVarName("Y")); + + PADDLE_ENFORCE_GE(x_dims.size(), y_dims.size(), + "Rank of first input must >= rank of second input.") + + if (x_grad) { + x_grad->Resize(x_dims); + } + + if (y_grad) { + y_grad->Resize(y_dims); + } + } +}; +} // namespace operators +} // namespace paddle + +namespace ops = paddle::operators; +REGISTER_OP(elementwise_mul, ops::ElementWiseMulOp, ops::ElementWiseMulOpMaker, + elementwise_mul_grad, ops::ElementWiseMulOpGrad); +REGISTER_OP_CPU_KERNEL( + elementwise_mul, + ops::ElementWiseMulKernel); +REGISTER_OP_CPU_KERNEL( + elementwise_mul_grad, + ops::ElementWiseMulGradKernel); diff --git a/paddle/operators/elementwise_mul_op.cu b/paddle/operators/elementwise_mul_op.cu new file mode 100644 index 0000000000..56f2087c22 --- /dev/null +++ b/paddle/operators/elementwise_mul_op.cu @@ -0,0 +1,25 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#define EIGEN_USE_GPU +#include "paddle/operators/elementwise_mul_op.h" + +namespace ops = paddle::operators; + +REGISTER_OP_GPU_KERNEL( + elementwise_mul, + ops::ElementWiseMulKernel); +REGISTER_OP_GPU_KERNEL( + elementwise_mul_grad, + ops::ElementWiseMulGradKernel); diff --git a/paddle/operators/elementwise_mul_op.h b/paddle/operators/elementwise_mul_op.h new file mode 100644 index 0000000000..e9ed679179 --- /dev/null +++ b/paddle/operators/elementwise_mul_op.h @@ -0,0 +1,185 @@ +/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. */ + +#pragma once +#include +#include "paddle/framework/eigen.h" +#include "paddle/framework/op_registry.h" +#include "paddle/operators/math/math_function.h" + +namespace paddle { +namespace operators { +/* + * Out = X ⊙ Y + * 1. shape(X) = (2, 3, 4, 5), shape(Y) = (3, 4), with axis=1 + * pre=2, n=3*4, post=5 + * 2. shape(X) = (2, 3, 4, 5), shape(Y) = (4,5) + * pre=2*3, n=4*5, post=1 + */ + +inline void get_mid_dims(const framework::DDim& x_dims, + const framework::DDim& y_dims, const int axis, + int& pre, int& n, int& post) { + pre = 1; + n = 1; + post = 1; + for (int i = 0; i < axis; ++i) { + pre *= x_dims[i]; + } + + for (int i = 0; i < y_dims.size(); ++i) { + PADDLE_ENFORCE_EQ(x_dims[i + axis], y_dims[i], + "Broadcast dimension mismatch."); + n *= y_dims[i]; + } + + for (int i = axis + y_dims.size(); i < x_dims.size(); ++i) { + post *= x_dims[i]; + } +} + +template +class ElementWiseMulKernel : public framework::OpKernel { + public: + void Compute(const framework::ExecutionContext& ctx) const override { + using Tensor = framework::Tensor; + + auto* x = ctx.Input("X"); + auto* y = ctx.Input("Y"); + auto* z = ctx.Output("Out"); + z->mutable_data(ctx.GetPlace()); + + auto x_e = framework::EigenVector::Flatten(*x); + auto y_e = framework::EigenVector::Flatten(*y); + auto z_e = framework::EigenVector::Flatten(*z); + + auto x_dims = x->dims(); + auto y_dims = y->dims(); + PADDLE_ENFORCE_GE(x_dims.size(), y_dims.size(), + "Rank of first input must >= rank of second input.") + + if (x_dims == y_dims || product(y_dims) == 1) { + z_e.device(ctx.GetEigenDevice()) = x_e * y_e; + return; + } + + int axis = ctx.Attr("axis"); + axis = (axis == -1 ? x_dims.size() - y_dims.size() : axis); + PADDLE_ENFORCE(axis >= 0 && axis < x_dims.size(), + "Axis should be in range [0, x_dims)"); + + int pre, n, post; + get_mid_dims(x_dims, y_dims, axis, pre, n, post); + if (post == 1) { + auto y_bcast = y_e.reshape(Eigen::DSizes(1, n)) + .broadcast(Eigen::DSizes(pre, 1)) + .reshape(Eigen::DSizes(x_e.size())); + z_e.device(ctx.GetEigenDevice()) = x_e * y_bcast; + return; + } else { + auto y_bcast = y_e.reshape(Eigen::DSizes(1, n, 1)) + .broadcast(Eigen::DSizes(pre, 1, post)) + .reshape(Eigen::DSizes(x_e.size())); + z_e.device(ctx.GetEigenDevice()) = x_e * y_bcast; + return; + } + } +}; + +template +class ElementWiseMulGradKernel : public framework::OpKernel { + public: + void Compute(const framework::ExecutionContext& ctx) const override { + using Tensor = framework::Tensor; + + auto* x = ctx.Input("X"); + auto* y = ctx.Input("Y"); + auto* dout = ctx.Input(framework::GradVarName("Out")); + + auto x_e = framework::EigenVector::Flatten(*x); + auto y_e = framework::EigenVector::Flatten(*y); + auto dout_e = framework::EigenVector::Flatten(*dout); + + auto x_dims = x->dims(); + auto y_dims = y->dims(); + + auto* dx = ctx.Output(framework::GradVarName("X")); + auto* dy = ctx.Output(framework::GradVarName("Y")); + if (dx) { + dx->mutable_data(ctx.GetPlace()); + } + if (dy) { + dy->mutable_data(ctx.GetPlace()); + } + + if (x_dims == y_dims || product(y_dims) == 1) { + if (dx) { + auto dx_e = framework::EigenVector::Flatten(*dx); + dx_e.device(ctx.GetEigenDevice()) = dout_e * y_e; + } + + if (dy) { + auto dy_e = framework::EigenVector::Flatten(*dy); + dy_e.device(ctx.GetEigenDevice()) = x_e * dout_e; + } + return; + } + + int axis = ctx.Attr("axis"); + axis = (axis == -1 ? x_dims.size() - y_dims.size() : axis); + + int pre, n, post; + get_mid_dims(x_dims, y_dims, axis, pre, n, post); + + // TODO(gongweibao): wrap reshape to a function. + if (post == 1) { + auto y_e_bcast = y_e.reshape(Eigen::DSizes(1, n)) + .broadcast(Eigen::DSizes(pre, 1)) + .reshape(Eigen::DSizes(x_e.size())); + if (dx) { + auto dx_e = framework::EigenVector::Flatten(*dx); + dx_e.device(ctx.GetEigenDevice()) = dout_e * y_e_bcast; + } + + if (dy) { + auto dy_e = framework::EigenVector::Flatten(*dy); + dy_e.device(ctx.GetEigenDevice()) = + (x_e * dout_e) + .reshape(Eigen::DSizes(pre, n)) + .sum(Eigen::array{{0}}); + } + return; + } else { + auto y_e_bcast = y_e.reshape(Eigen::DSizes(1, n, 1)) + .broadcast(Eigen::DSizes(pre, 1, post)) + .reshape(Eigen::DSizes(x_e.size())); + if (dx) { + auto dx_e = framework::EigenVector::Flatten(*dx); + dx_e.device(ctx.GetEigenDevice()) = dout_e * y_e_bcast; + } + + if (dy) { + auto dy_e = framework::EigenVector::Flatten(*dy); + dy_e.device(ctx.GetEigenDevice()) = + (x_e * dout_e) + .reshape(Eigen::DSizes(pre, n, post)) + .sum(Eigen::array{{0, 2}}); + } + return; + } + } +}; + +} // namespace operators +} // namespace paddle diff --git a/paddle/pybind/pybind.cc b/paddle/pybind/pybind.cc index 16a2368aae..ef62d6e997 100644 --- a/paddle/pybind/pybind.cc +++ b/paddle/pybind/pybind.cc @@ -35,6 +35,7 @@ USE_OP(add); USE_OP(onehot_cross_entropy); USE_OP(sgd); USE_OP(mul); +USE_OP(elementwise_mul); USE_OP(mean); USE_OP(sigmoid); USE_OP(softmax); diff --git a/python/paddle/v2/framework/tests/test_elementwise_mul_op.py b/python/paddle/v2/framework/tests/test_elementwise_mul_op.py new file mode 100644 index 0000000000..e268cfddb2 --- /dev/null +++ b/python/paddle/v2/framework/tests/test_elementwise_mul_op.py @@ -0,0 +1,157 @@ +import unittest +import numpy as np +from op_test import OpTest + + +class TestElementwiseMulOp_Matrix(OpTest): + def setUp(self): + self.op_type = "elementwise_mul" + """ Warning + CPU gradient check error! + 'X': np.random.random((32,84)).astype("float32"), + 'Y': np.random.random((32,84)).astype("float32") + """ + self.inputs = { + 'X': np.random.uniform(0.1, 1, [13, 17]).astype("float32"), + 'Y': np.random.uniform(0.1, 1, [13, 17]).astype("float32") + } + self.outputs = {'Out': np.multiply(self.inputs['X'], self.inputs['Y'])} + + def test_check_output(self): + self.check_output() + + def test_check_grad_normal(self): + self.check_grad(['X', 'Y'], 'Out', max_relative_error=0.1) + + def test_check_grad_ingore_x(self): + self.check_grad( + ['Y'], 'Out', max_relative_error=0.1, no_grad_set=set("X")) + + def test_check_grad_ingore_y(self): + self.check_grad( + ['X'], 'Out', max_relative_error=0.1, no_grad_set=set('Y')) + + +class TestElementwiseMulOp_Vector(OpTest): + def setUp(self): + self.op_type = "elementwise_mul" + self.inputs = { + 'X': np.random.random((32, )).astype("float32"), + 'Y': np.random.random((32, )).astype("float32") + } + self.outputs = {'Out': np.multiply(self.inputs['X'], self.inputs['Y'])} + + def test_check_output(self): + self.check_output() + + def test_check_grad_normal(self): + self.check_grad(['X', 'Y'], 'Out', max_relative_error=0.1) + + def test_check_grad_ingore_x(self): + self.check_grad( + ['Y'], 'Out', max_relative_error=0.1, no_grad_set=set("X")) + + def test_check_grad_ingore_y(self): + self.check_grad( + ['X'], 'Out', max_relative_error=0.1, no_grad_set=set('Y')) + + +class TestElementwiseMulOp_broadcast_0(OpTest): + def setUp(self): + self.op_type = "elementwise_mul" + self.inputs = { + 'X': np.random.rand(2, 3, 4).astype(np.float32), + 'Y': np.random.rand(2).astype(np.float32) + } + + self.attrs = {'axis': 0} + self.outputs = { + 'Out': self.inputs['X'] * self.inputs['Y'].reshape(2, 1, 1) + } + + def test_check_output(self): + self.check_output() + + def test_check_grad_normal(self): + self.check_grad(['X', 'Y'], 'Out', max_relative_error=0.1) + + def test_check_grad_ingore_x(self): + self.check_grad( + ['Y'], 'Out', max_relative_error=0.1, no_grad_set=set("X")) + + def test_check_grad_ingore_y(self): + self.check_grad( + ['X'], 'Out', max_relative_error=0.1, no_grad_set=set('Y')) + + +class TestElementwiseMulOp_broadcast_1(OpTest): + def setUp(self): + self.op_type = "elementwise_mul" + self.inputs = { + 'X': np.random.rand(2, 3, 4).astype(np.float32), + 'Y': np.random.rand(3).astype(np.float32) + } + + self.attrs = {'axis': 1} + self.outputs = { + 'Out': self.inputs['X'] * self.inputs['Y'].reshape(1, 3, 1) + } + + def test_check_output(self): + self.check_output() + + def test_check_grad_normal(self): + self.check_grad(['X', 'Y'], 'Out', max_relative_error=0.1) + + def test_check_grad_ingore_x(self): + self.check_grad( + ['Y'], 'Out', max_relative_error=0.1, no_grad_set=set("X")) + + def test_check_grad_ingore_y(self): + self.check_grad( + ['X'], 'Out', max_relative_error=0.1, no_grad_set=set('Y')) + + +class TestElementwiseMulOp_broadcast_2(OpTest): + def setUp(self): + self.op_type = "elementwise_mul" + self.inputs = { + 'X': np.random.rand(2, 3, 4).astype(np.float32), + 'Y': np.random.rand(4).astype(np.float32) + } + + self.outputs = { + 'Out': self.inputs['X'] * self.inputs['Y'].reshape(1, 1, 4) + } + + def test_check_output(self): + self.check_output() + + def test_check_grad_normal(self): + self.check_grad(['X', 'Y'], 'Out', max_relative_error=0.1) + + def test_check_grad_ingore_x(self): + self.check_grad( + ['Y'], 'Out', max_relative_error=0.1, no_grad_set=set("X")) + + def test_check_grad_ingore_y(self): + self.check_grad( + ['X'], 'Out', max_relative_error=0.1, no_grad_set=set('Y')) + + +class TestElementwiseMulOp_broadcast_3(OpTest): + def setUp(self): + self.op_type = "elementwise_mul" + self.inputs = { + 'X': np.random.rand(2, 3, 4, 5).astype(np.float32), + 'Y': np.random.rand(3, 4).astype(np.float32) + } + + self.attrs = {'axis': 1} + self.outputs = { + 'Out': self.inputs['X'] * self.inputs['Y'].reshape(1, 3, 4, 1) + } + + +if __name__ == '__main__': + unittest.main()