Paddle

Commit Graph

Author	SHA1	Message	Date
Zhen Wang	be3777a50a	Add pure fp16 training with master weights. (#27712 ) * add the weight decay func for the momentum op * Add the multi_precision function in Momentum Optimizer. * Make sure that the initial value of master weights are same with the fp16 weights. * add static loss scaling. * add the rescale_grad function in the pure fp16 training. * use the original momentum updating method. * Polish some codes, such as variable names. * add docstring for apis. * update the var creation details of _create_master_weight. * not modify codes about imperative momentum updating. * Fix the error of test_dist_sparse_tensor_load_momentum UT. * add unit test for multi precision fp16 training. * add more unit tests for CI. * Use lower threshold values for allclose comparing in test_multi_precision_fp16_train UT. * For CI Coverage Checking.	5 years ago
Leo Chen	4556ad76b4	Upgrade string literals to raw string [part 2](#29217 )	5 years ago
furnace	8ff3550658	refactor momentum op to combine weight (#27414 ) * refactor momentum op to combine weight_decay (scale op and sum op)	5 years ago

Author

SHA1

Message

Date

Zhen Wang

be3777a50a

Add pure fp16 training with master weights. (#27712 )

* add the weight decay func for the momentum op

* Add the multi_precision function in Momentum Optimizer.

* Make sure that the initial value of master weights are same with the fp16 weights.

* add static loss scaling.

* add the rescale_grad function in the pure fp16 training.

* use the original momentum updating method.

* Polish some codes, such as variable names.

* add docstring for apis.

* update the var creation details of _create_master_weight.

* not modify codes about imperative momentum updating.

* Fix the error of test_dist_sparse_tensor_load_momentum UT.

* add unit test for multi precision fp16 training.

* add more unit tests for CI.

* Use lower threshold values for allclose comparing in test_multi_precision_fp16_train UT.

* For CI Coverage Checking.

Leo Chen

4556ad76b4

Upgrade string literals to raw string [part 2](#29217 )

furnace

8ff3550658

refactor momentum op to combine weight (#27414 )

* refactor momentum op to combine weight_decay (scale op and sum op)

3 Commits (adaec0073d02c0ea55bcabc4671ebfc8dbd3182c)