32 Commits (17030ff28b9a54bb57779e9b8448a6d222110ec5)

Author	SHA1	Message	Date
lilong12	c3974d0e2a	[3D-parallel] Reformat pipeline parallel (#31786) ... * update, test=develop	4 years ago
arlesniak	7ccf6b6030	[oneDNN] Initial bf16 amp integration (#31093)	4 years ago
huangxu96	138620084c	Add fleet amp_init() (#30572) ... * add fleet amp.init() * add unittest for fleet_amp_init	4 years ago
huangxu96	342d62de60	add amp example document (#30314)	4 years ago
Zhen Wang	7f7dfccf20	Support pure fp16 training for AMP API. (#29544) ... * add cast ops before and after unsupported fp16 ops. * Keep partial net in FP32 pattern. * Support check_finite_and_unscale and update_loss_scaling for FP16 calculation mode. * Add fp16 support for adam op. * add multi precision attr for adam. * Fix the bug of test_multi_precision_fp16_train UT. * Code format for CI. * Fix the redefine error about MPTypeTrait on windows. * fix bugs of the _create_accumulators func in Momentum. * fix bug when inserting post cast op. * Add the update_loss_scaling op in allow_set of UnusedVarCheck. * Update for ci coverage. * Add some doc for OptimizerWithMixedPrecision. * Fix the code style. * Imporve the doc of `amp_init`. * Change for fp16 testing if users have the infer program defined in separate way.	4 years ago
WangXi	ab04997846	[fleet] combine amp and gradient merge, test=develop (#30086)	4 years ago
huangxu96	c05170d3d8	add alias for fluid.contrib.mixed_precision (#29562) ... * add alias for fluid.contrib.mixed_precision	4 years ago
Aurelius84	5d530c9319	fix amp support fleet (#29491)	4 years ago
Zhen Wang	be3777a50a	Add pure fp16 training with master weights. (#27712) ... * add the weight decay func for the momentum op * Add the multi_precision function in Momentum Optimizer. * Make sure that the initial value of master weights are same with the fp16 weights. * add static loss scaling. * add the rescale_grad function in the pure fp16 training. * use the original momentum updating method. * Polish some codes, such as variable names. * add docstring for apis. * update the var creation details of _create_master_weight. * not modify codes about imperative momentum updating. * Fix the error of test_dist_sparse_tensor_load_momentum UT. * add unit test for multi precision fp16 training. * add more unit tests for CI. * Use lower threshold values for allclose comparing in test_multi_precision_fp16_train UT. * For CI Coverage Checking.	4 years ago
furnace	7584bb5096	Layer norm fp16 (#29169) ... * add fp16 for layer_norm op * revert layernorm api * fix forward * fix forward * fix backward for layernorm with fp16 * fix unit test for layernorm with fp16 * fix with_mkldnn compile error for layernorm with fp16 * 1. revert to PADDLE_ENFORCE_NOT_NULL, 2. change static_cast<float> to static_cast<U> * fix with_mkldnn compile error for layernorm with fp16 * fix with_mkldnn compile error for layernorm with fp16 Co-authored-by: zhiqiu <chenqiuliang@baidu.com>	4 years ago
WangXi	0c2a51d240	optimizer amp, all use fp16 communication, overlap last comm and compute (#28957)	4 years ago
Leo Chen	11e32baf1e	Add matmtl_v2 to amp list (#28693) ... * add matmtl_v2 to amp list * support dygraph	4 years ago
Leo Chen	71d6220772	Skip reader op in mixed_precision decorator (#28353) ... * skip reader op in mixed_precision decorator * add ut	4 years ago
WangXi	0a1862d1d2	fleet combine amp dgc recompute meta optimizer (#27643)	4 years ago
Zhang Ting	906e7f921e	add fuse_bn_act op (#27230) ... * add fused_bn_add_relu op	4 years ago
Zhen Wang	d708b21074	Update amp_check_finite_and_scale_op and add an updating_loss_scaling op for static graph amp training. (#26240) ... * update amp_check_finite_and_scale_op for static_amp. * use amp_check_finite_and_scale in static graph amp. * update grads to zero when grads own infinite values(as for amp_checkout_finite_and_scale op). * add update_loss_scaling op in cpp. * add update_loss_scaling_op unit test. * update the doc of the check_finite_and_unscale op * Update the process of gradients updating skipping if the gradients have infinite values. * update the way to zero grads. * update test_update_loss_scaling_op.py * add log info when find infinite grads. * add the unit test for UpdateLossScaling Layer.	4 years ago
Zhen Wang	bcdbac1753	fix some cast error. (#26884)	5 years ago
mapingshuo	f0e743f136	fix AMP and recompute (#23551) ... * allow amp and recompute working together	5 years ago
gongweibao	5e07db15e6	fix init scaling value test=develop (#22145)	5 years ago
Zhen Wang	be2e3e67d9	Fix some typos in AMP. (#21354) ... * fix some typos in AMP. test=develop * delete useless codes. test=develop	5 years ago
gongweibao	3255fe69bb	Add custom black variable name set in amp interface. (#20875) ... * add custom black varname test=develop * fix dtype test=develop * fix num test=develop * fix ut test=develop * fix coverage test=develop * fix blackvar names test=develop	5 years ago
gongweibao	1d82025e89	Add interface so user can get scaled loss when they use customized loss. (#20571)	5 years ago
gongweibao	7b9e33972a	delete backward return list test=develop (#20294)	5 years ago
Jie Fang	d9db94d752	Optimize amp for multi-gpu to enable FP16 gradients transfer across gpus. (#19714) ... Optimize amp for multi-gpu to enable FP16 gradients transfer across gpus	5 years ago
gongweibao	6c2bc29cc0	Fix float16 optimizer. (#19682) ... Fix float16 optimizer	6 years ago
Jie Fang	c6a598a276	init new amp, optimize inserting cast op for batchnorm (#18596) ... init new amp, optimize inserting cast op for batchnorm	6 years ago
gongweibao	abaf87be2b	Change backward_guard to optimize_guard to maximize the allreduce overlap. (#19506) ... Change backward_guard to optimize_guard to maximize the allreduce overlap	6 years ago
Zeng Jinle	5dce1da680	remove reset recordio usage (#19519)	6 years ago
Jie Fang	2b4ef509ea	init custom black white list (#18377) ... test=develop	6 years ago
Jie Fang	172c2facef	init black/white lists (#17847) ... test=develop	6 years ago
Jie Fang	30e178fa2c	init auto loss scaling (#17194) ... * init auto loss scaling test=develop * change API.spec * change ifelse to switch and use reduce_sum to optimize checking isfinite test=develop * Remove redundant code test=develop	6 years ago
Yibing Liu	beda78258f	Init mixed precision training interface (#16856) ... * Init mixed precision training interface * Add fp16 test script test=develop * All initializers support float16 test=develop * Code cleanup & add more code annotations test=develop * Update API spec test=develop * Add usage example in doc test=develop	6 years ago