56 Commits (8c19d7aa2f89a38b3a68e53c73d88af16a3de8ce)

Author	SHA1	Message	Date
Thunderbrook	3789a69923	solve bug in heter mode (#31531) ... * heter bug * format * format	4 years ago
Thunderbrook	c4f279fe8d	support multi node in heterps (#31102) ... * push multi node * multi node * MultiThread * remove log * solve bug in 30829	4 years ago
Qi Li	a60d93fb77	[ROCM] update fluid framework for rocm (part2), test=develop (#31010)	4 years ago
Thunderbrook	565354f676	support save multi sparse table in one path (#31108) ... * save multi table one path * format	4 years ago
wanghuancoder	35c5b23f68	use iwyu clean include second time, test=develop (#30829) ... * use iwyu clean include second time, test=develop	4 years ago
Thunderbrook	0b8e1fadc5	add topo-aware in heter-ps (#30087) ... * add topo aware * resource.h * topo aware * format	4 years ago
Thunderbrook	09b6e71928	heter box (#29734) ... * 　add heter box * add trainer, worker, wrapper... * format * for ci * format * remove boost get * boost & copyright * rename * 　rename * format * format * format Co-authored-by: yaoxuefeng6 <yaoxuefeng@baidu.com>	4 years ago
Thunderbrook	0073f9bdb0	support ps-gpu (#28752) ... * ps gpu transpile * ps gpu * remove op * gps trainer * local ps * add macro * HeterBox * def cuda * tab * code style * style Co-authored-by: Thunderbrook <a754913769#163.com>	4 years ago
wanghuancoder	41aad9bfcd	revert 4 files, from clear include by iwyu, test=develop (#27895)	4 years ago
Thunderbrook	6f69a4cb05	add xpu in heter mode (#27000) ... * add xpu in heter mode test=develop * BOOST_CONST_GET; PADDLE_THROW test=develop * code style test=develop * code style test=develop * code style test=develop * refine test=develop * refine test=develop * refine test=develop * refine code test=develop	4 years ago
wanghuancoder	df43905f12	use iwyu clean include (#27267) ... * use iwyu clean include, test=develop, test=win * compilation error, test=develop * fix compilation error2, test=develop * fix compilation error3, test=develop * fix compilation error4, test=develop * fix compilation error5, test=develop * fix compilation error6, test=develop * fix compilation error7, test=develop * fix compilation error8, test=develop * fix compilation error8, test=develop * fix compilation error10, test=develop * fix compilation error11, test=develop	4 years ago
Thunderbrook	5205748481	fix eigen in push sparse; fix hadoop command (#26872) ... * fix eigen in push sparse; fix hadoop command test=develop * add log in load_combine_op test=develop	5 years ago
yaoxuefeng	a47d92d868	fleet add save with whitelist test=develop (#23376)	5 years ago
Thunderbrook	0cb60c700d	add heter ps mode (#25682) ... * add heter ps mode * code style test=develop * add with_pslib test=develop * unitest test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * test monitor test=develop * prepare trainer test=develop * code style test=develop	5 years ago
Chen Weihang	d1062d5278	Replace all errors thrown by LOG(FATAL) with PADDLE_THROW (#24759) ... * remove REPLACE_ENFORCE_GLOG compile option & add ci rule prohibit LOG(FATAL) using, test=develop * remove ci test case, test=develop * replace all LOG(FATAL) & polish message, test=develop * fix typo, test=develop * polish error info detail, test=develop	5 years ago
xujiaqi01	1034ca316f	add timeout and http store in communication (#23436) ... * add timeout and http store in communication, add revert and confirm in fleet * test=develop	5 years ago
xujiaqi01	d98084e7ec	add save with prefix (#23449) ... * add save with prefix * test=develop	5 years ago
xujiaqi01	3a45767d49	add fleet pslib pull and push sparse op and push dense op (#23139) ... * add fleet pslib pull and push sparse op and push dense op * test=develop	5 years ago
xujiaqi01	68ea1ad55b	add clear one table (#23089) ... * add clear_one_table * test=develop	5 years ago
yaoxuefeng	2235ee1a5e	multi-loss optimization by adding a DownpourOpt worker (#22025) ... * update * update test=develop * update compile set test=develop * update compile set test=develop * update test=develop * update test=develop * update test=develop * update compile setting test=develop * update compile setting test=develop * update run demo test=develop * update test=develop * update test=develop * fix test=develop * update test=develop * update test=develop * update test=develop * update test=develop * update test=develop * update test=develop * update test=develop * update test=develop * update test=develop * update format test=develop * update format test=develop * update style test=develop * update style test=develop * change style test=develop * change style test=develop * change style test=develop * add dataset unittest test=develop * update test=develop * update for record test=develop * udpate style for record test=develop * update for record test=develop * update for record test=develop * update for record test=develop * fix format test=develop * update test=develop * update test=develop * update test=develop * update test=develop * update test=develop	5 years ago
xujiaqi01	e3a457d34b	add collective communication library in fleet (#22211) ... * add collective communication library in fleet to replace mpi * test=develop	5 years ago
Thunderbrook	c3cf42d0f7	add table id in cache shuffle (#21585) ... * general table * add sparse table test=develop * no cvm test=develop * add no_cvm test=develop * add note test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * add key of optimizer test=develop * solve pslib stop core test=develop * barrier test=develop * add notes test=develop * add table id in cache shuffle test=develop * table id test=develop * code style test=develop	5 years ago
xujiaqi01	c05706fe73	fix code style of fleet_wrapper (#21639) ... * fix code style of fleet_wrapper * test=develop	5 years ago
Thunderbrook	9a7832f8be	print table stat info for pslib (#21296) ... * print table stat test=develop * notes test=develop * notes test=develop	5 years ago
Thunderbrook	0d17c1b816	solve pslib core in stop worker (#21263) ... * general table * add sparse table test=develop * no cvm test=develop * add no_cvm test=develop * add note test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * add key of optimizer test=develop * solve pslib stop core test=develop * barrier test=develop * add notes test=develop	5 years ago
Thunderbrook	349e82d669	support general embedding params (#21217) ... * general table * add sparse table test=develop * no cvm test=develop * add no_cvm test=develop * add note test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * add key of optimizer test=develop	5 years ago
xujiaqi01	23876de55b	fix cache table bug, add save_paddle_inference_model, fix hdfs util bug (#21052) ... * fix cache table bug * add save_paddle_inference_model * fix hdfs util bug * test=develop	5 years ago
xujiaqi01	9e045170c0	add copy table (#21086) ... * copy some feasigns and corresponding embeddings from one sparse table to another * copy all feasigns and corresponding embeddings from one sparse table to another * copy all dense params from one table to another * copy some local vars to other local vars	5 years ago
xujiaqi01	48669aa8f0	fix several sparse table issuses (#20686) ... * no longer need to define all embedding layers (no one less) of all slots in each program. make trainer_param repeated in ps.proto. * add find_distributed_lookup_table_grads instead of hard code GRAD * support embedding stop gradient. push sparse has error before fix this.* * fix fill sparse, skip slots which do not have embedding. each slot's embedding in a sparse table should be used in all training programs before fix this. * fix pull sparse, skip slots which do not have embedding. * fix collect feasign label info, skip slots which do not have embedding. * support when there are multi sparse tables in one or multi training programs, each program can pull/push its own related sparse tables instead of all sparse tables. * test=develop	5 years ago
xujiaqi01	cedc04775c	support change shuffle and train thread num (#19841) ... * support change shuffle thread num * support change train thread num * fix receive shuffle data of each channel * data norm stop gradient * add check thread_tensor type and root_tensor type when merge metric * remove sleep in shuffle, add config * add config of pslib client to client communication * fix xbox str * add data norm op testcase * add flush in trainer finalize	5 years ago
xujiaqi01	6bf298bf09	support preload thread, optimize hdfs log, fix master+patch bug (#19695) ... * support preload thread * sleep before fleet wrapper exit for pslib core dump * optimize hdfs log * fix master+patch bug	5 years ago
jiaqi	b104ea0684	add get_last_save_xbox_base/get_last_save_xbox (#19122) ... * add get_last_save_xbox_base/get_last_save_xbox * fix fleet_util bug of load paddle model * add doc string in fleet api	6 years ago
yaoxuefeng	9150cf50fc	add save cache model api in fleet& add slots shuffle in dataset module & add metric op to calculate ctr related metrics (#18871) ... * add ctr related metric layer test=develop * add save cache and slots shuffle test=develop * add save cache and slots shuffle test=develop * fix error * fix error * fix style for ci * fix for comments * change SlotsShuffle input to std::strinf for generality * fix style * fix style * fix style * fix style * fix style * fix style * fix stylr * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * change non-const reference to pointer * fix style * fix style * fix style test=develop * fix style test=develop * add return ins num in ctr metric op * change dtype to float in metric_op.py * fix error test=develop * fix style test=develop * fix API spec * fix API spec * fix API spec test=develop * add UT test=develop	6 years ago
Thunderbrook	52c1431eee	add clear_model interface in fleetwrapper (#18815) ... * dump slot * test * proto * dump slot * test * proto * code style * code style * code style * style * add delete after unseen days * add unseen days * code style * conflict solve test=develop * add clear model * code style test=develop * code style test=develop	6 years ago
fuyinno4	c167a4b4dd	Fix shrink-dense and add scale-datanorm (#18746) ... Fix FleetWrapper: 1. fix shrink dense: just scale show 2. add datanorm scale: divide datanorm's gradient by batch_size	6 years ago
Thunderbrook	d8396281ef	add slot to sparse table (#18686) ... The change includes 2 things: 1. save delta model and shrink table are control by the same parameter before, now add delete_after_unseen_days to control shrink table. 2. value in sparse table has no slot before, now add slot in sparse table, and add DownpureCtrAccessor to support the new meta. test=develop	6 years ago
jiaqi	d18aabb472	support patch data, add load_one_table, fix bug (#18509) ... （1）support patch data （merge slots of instances of same line id, modify dense layer which changes its size） （2）add fleet load_one_table interface, support load from paddle model and load from pslib model （3）fix push sparse bug which cause push sparse cost more time（about 10% in my testcase） （4）when some slots are not in one of your network (join/update, etc.)，data feed、collect label info、push/pull sparse will skip these slots， instead of throw error. （5）add more debug info in TrainFilesWithProfiler	6 years ago
jiaqi	66d51206b1	add save/load model, shrink table, cvm, config file & fix pull dense bug (#17118) ... * add save/load model, shrink table, cvm, config file & fix pull dense bug test=develop * fix global shuffle bug, fix pull dense bug, fix release memeory bug, fix shrink error add client flush, add get data size test=develop * fix global shuffle bug test=develop * fix global shuffle bug test=develop * fix code style test=develop * fix code style & modify pslib cmake test=develop * fix error of _role_maker test=develop * fix code style test=develop * fix code style test=develop * fix code style test=develop * fix code style test=develop * fix code style test=develop * fix windows compile error of fleet test=develop * fix global shuffle bug * add comment test=develop * update pslib.cmake test=develop * fix fill sparse bug test=develop * fix push sparse bug test=develop	6 years ago
xjqbest	5e5139283b	fix runtime error ... test=develop	6 years ago
dongdaxiang	718ea6dbd5	fix fleet code style ... test=develop	6 years ago
xjqbest	a99c8d0c29	fix client to client communication bug ... test=develop	6 years ago
dongdaxiang	98dda08a85	fix pull sparse slow problem ... test=develop	6 years ago
dongdaxiang	d4514949bf	remove local random engine in fleet with rand_r() ... test=develop	6 years ago
dongdaxiang	365be5d559	support win32 flag in io.cc shell.cc, fix code style problem in fleet_wrapper, fix lodtensor_printer_test problem ... test=develop	6 years ago
xjqbest	b7940c2918	fix bug of gen_worker_desc and set_filelist, add some doc	6 years ago
xujiaqi01	f5c6a14b54	fix runtime error	6 years ago
xujiaqi01	a5b1a0e12b	support multi dataset && add init model && fix bug	6 years ago
dongdaxiang	b7a202aa38	add distributed optimizer factory	6 years ago
xujiaqi01	d25389fefd	add some log && fix error	6 years ago
dongdaxiang	317eb0aad3	add incubate for unified API	6 years ago