9 Commits (3d015f1cf529915ab52cb8aef7c475f67fb128b5)

Author	SHA1	Message	Date
Zhen Wang	7f7dfccf20	Support pure fp16 training for AMP API. (#29544) ... * add cast ops before and after unsupported fp16 ops. * Keep partial net in FP32 pattern. * Support check_finite_and_unscale and update_loss_scaling for FP16 calculation mode. * Add fp16 support for adam op. * add multi precision attr for adam. * Fix the bug of test_multi_precision_fp16_train UT. * Code format for CI. * Fix the redefine error about MPTypeTrait on windows. * fix bugs of the _create_accumulators func in Momentum. * fix bug when inserting post cast op. * Add the update_loss_scaling op in allow_set of UnusedVarCheck. * Update for ci coverage. * Add some doc for OptimizerWithMixedPrecision. * Fix the code style. * Imporve the doc of `amp_init`. * Change for fp16 testing if users have the infer program defined in separate way.	4 years ago
Zhen Wang	5ac71b36fb	Remove tensor copy in the update_loss_scaling op. (#29426) ... * remove tensor copy in the update_loss_scaling op * not use thrust. * fix some cuda memory access error.	4 years ago
LoveAn	671555ed32	Compiling operator libraries with Unity build (#29130) ... * Compiling operator libraries with Unity Build on Windows CPU. * Compiling operator libraries with Unity Build on Windows GPU, no_test, test=windows_ci * Add option in windows ci script, no_test, test=windows_ci * Optimize parallel compiling, test=develop * remove limit of parallel compile and skip some ops in UB, test=develop * remove changes of header file, test=develop * remove changes of header file, test=develop * fix test_eye_op unittest failed, test=develop * Compiling operator libraries with Unity Build on Linux, test=develop * set default WITH_UNITY_BUILD=OFF, test=develop * Move unity build rules into a single file and add comment, test=develop * optimize parallel compilation, test=develop * fix undefined reference error on coverage ci, test=develop	4 years ago
WangXi	0c2a51d240	optimizer amp, all use fp16 communication, overlap last comm and compute (#28957)	4 years ago
Zhen Wang	365c2c9c89	fix error message showing in UpdateLossScalingOp (#27596)	4 years ago
Zhen Wang	d708b21074	Update amp_check_finite_and_scale_op and add an updating_loss_scaling op for static graph amp training. (#26240) ... * update amp_check_finite_and_scale_op for static_amp. * use amp_check_finite_and_scale in static graph amp. * update grads to zero when grads own infinite values(as for amp_checkout_finite_and_scale op). * add update_loss_scaling op in cpp. * add update_loss_scaling_op unit test. * update the doc of the check_finite_and_unscale op * Update the process of gradients updating skipping if the gradients have infinite values. * update the way to zero grads. * update test_update_loss_scaling_op.py * add log info when find infinite grads. * add the unit test for UpdateLossScaling Layer.	4 years ago
Leo Chen	2d95280e1f	Feature/Enable Auto-Mixed-Precision in dynamic graph (#24903) ... * add auto_cast, test=develop * add loss scaler, test=develop * add comments, test=develop * refine code, test=develop * refine code, test=develop * do not set flags automatically, test=develop * fix custom op bug, test=develop * add more test, test=develop * refine enable logic, test=develop * enable amp test with GPU, test=develop * add unittest * add test for found_inf * follow comments * follow comments * remove global variable, use singleton * add some notes * update comments * update comments * update comments * add use_dynamic_loss_scaling argument * refine found_inf * refine found_inf	5 years ago
Leo Chen	a7cb97a1a5	Fix/isfinite on windows (#24927) ... * refine isfinite, test=develop * use namespace std of isfinite, test=develop, test=win_gpu	5 years ago
Leo Chen	1e818158f5	Feature/add amp_checkout_finite_and_scale op (#24875) ... * add amp_check_finite_and_scale op, test=develop * add cpu kernel, test=develop * use bool, test=develop * follow comments, test=develop	5 years ago