63 Commits (develop)

Author	SHA1	Message	Date
Thunderbrook	565354f676	support save multi sparse table in one path (#31108) ... * save multi table one path * format	5 years ago
Thunderbrook	0b8e1fadc5	add topo-aware in heter-ps (#30087) ... * add topo aware * resource.h * topo aware * format	5 years ago
Thunderbrook	09b6e71928	heter box (#29734) ... * 　add heter box * add trainer, worker, wrapper... * format * for ci * format * remove boost get * boost & copyright * rename * 　rename * format * format * format Co-authored-by: yaoxuefeng6 <yaoxuefeng@baidu.com>	5 years ago
yaoxuefeng	545df287fc	add user_define_dump (#28596)	6 years ago
Leo Chen	3815d7aa40	Upgrade string literals to raw string (#28989) ... * upgrade comment string to raw string * fix string in * fix string with ' ' * revert update on comments * upgrade only necessary * fix sample code checker * fix comments with '''	6 years ago
Thunderbrook	0073f9bdb0	support ps-gpu (#28752) ... * ps gpu transpile * ps gpu * remove op * gps trainer * local ps * add macro * HeterBox * def cuda * tab * code style * style Co-authored-by: Thunderbrook <a754913769#163.com>	6 years ago
zhang wenhui	5a83496c8d	Multi task (#26002) ... * add multitask * add multitask, test=develop * fix code style, test=develop * add partail push dense, test=develop * fix has_kay in py3, test=develop * fix, test=develop * fix, test=develop * fix, test=develop	6 years ago
yaoxuefeng	a47d92d868	fleet add save with whitelist test=develop (#23376)	6 years ago
Thunderbrook	0cb60c700d	add heter ps mode (#25682) ... * add heter ps mode * code style test=develop * add with_pslib test=develop * unitest test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * test monitor test=develop * prepare trainer test=develop * code style test=develop	6 years ago
tangwei12	caa90a6510	Integrated Trainer of Parameter Server (API add `fluid.contrib.layers.sparse_embedding` only) (#22957) ... * Integrated Trainer of Parameter Server	6 years ago
xujiaqi01	1034ca316f	add timeout and http store in communication (#23436) ... * add timeout and http store in communication, add revert and confirm in fleet * test=develop	6 years ago
xujiaqi01	15ce8e21d8	add unit accessor (#23703) ... * add unit accessor in fleet, support DownpourUnitAccessor * test=develop	6 years ago
xujiaqi01	70f7c23fd2	fix add dense table (#23681) ... * fix add dense table test=develop * fix test=develop * fix test=develop	6 years ago
xujiaqi01	d98084e7ec	add save with prefix (#23449) ... * add save with prefix * test=develop	6 years ago
xujiaqi01	21f59779ea	fix dump slot in strategy (#23398)	6 years ago
xujiaqi01	3a45767d49	add fleet pslib pull and push sparse op and push dense op (#23139) ... * add fleet pslib pull and push sparse op and push dense op * test=develop	6 years ago
xujiaqi01	c8f9e66b71	fix no_cvm in config_fleet (#22818) ... * fix no cvm, when accessor is DownpourSparseValueAccessor, set no_cvm=True * test=develop	6 years ago
xujiaqi01	68ea1ad55b	add clear one table (#23089) ... * add clear_one_table * test=develop	6 years ago
yaoxuefeng	995a6376f7	add pslib SparseDoubleTable test=develop (#23053)	6 years ago
tangwei12	c4a6a0e2e4	Revert "Integrated API of Parameter Server (#22710)" test=develop (#23071) ... This reverts commit 66fce9e824.	6 years ago
tangwei12	66fce9e824	Integrated API of Parameter Server (#22710) ... Fleet Parameter Server API Integrated	6 years ago
yaoxuefeng	c5cbe7f07b	fix add grad bug test=develop (#22924) ... * fix add grad bug test=develop * update style test=develop	6 years ago
tianshuo78520a	433cef03e5	fix typo word (#22784)	6 years ago
tianshuo78520a	d2ba91aad1	fix typo words (#22653)	6 years ago
yaoxuefeng	2235ee1a5e	multi-loss optimization by adding a DownpourOpt worker (#22025) ... * update * update test=develop * update compile set test=develop * update compile set test=develop * update test=develop * update test=develop * update test=develop * update compile setting test=develop * update compile setting test=develop * update run demo test=develop * update test=develop * update test=develop * fix test=develop * update test=develop * update test=develop * update test=develop * update test=develop * update test=develop * update test=develop * update test=develop * update test=develop * update test=develop * update format test=develop * update format test=develop * update style test=develop * update style test=develop * change style test=develop * change style test=develop * change style test=develop * add dataset unittest test=develop * update test=develop * update for record test=develop * udpate style for record test=develop * update for record test=develop * update for record test=develop * update for record test=develop * fix format test=develop * update test=develop * update test=develop * update test=develop * update test=develop * update test=develop	6 years ago
xujiaqi01	371f377bea	add GeneralRoleMaker (#22295) ... * add GeneralRoleMaker which is for general usage * test=develop	6 years ago
xujiaqi01	f1178e9d79	fix fleet save bug (#21362) ... * fix fleet save bug of save_infernece_model * test=develop	7 years ago
Thunderbrook	9a7832f8be	print table stat info for pslib (#21296) ... * print table stat test=develop * notes test=develop * notes test=develop	7 years ago
xujiaqi01	319d2ba925	fix fs_client_param bug (#21212) ... * fix fs_client_param bug， user can set this config through fleet_desc_file or fleet config * test=develop	7 years ago
Thunderbrook	0d17c1b816	solve pslib core in stop worker (#21263) ... * general table * add sparse table test=develop * no cvm test=develop * add no_cvm test=develop * add note test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * add key of optimizer test=develop * solve pslib stop core test=develop * barrier test=develop * add notes test=develop	7 years ago
Thunderbrook	349e82d669	support general embedding params (#21217) ... * general table * add sparse table test=develop * no cvm test=develop * add no_cvm test=develop * add note test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * add key of optimizer test=develop	7 years ago
xujiaqi01	23876de55b	fix cache table bug, add save_paddle_inference_model, fix hdfs util bug (#21052) ... * fix cache table bug * add save_paddle_inference_model * fix hdfs util bug * test=develop	7 years ago
xujiaqi01	9e045170c0	add copy table (#21086) ... * copy some feasigns and corresponding embeddings from one sparse table to another * copy all feasigns and corresponding embeddings from one sparse table to another * copy all dense params from one table to another * copy some local vars to other local vars	7 years ago
Thunderbrook	5970e8ac5e	find lookup table in order (#20932) ... test=develop	7 years ago
Thunderbrook	59bcdc8a19	support dump param of model into afs (#20302) ... * support dump param to afs test=develop * code style test=develop * code style test=develop * dump param test=develop * dump param test=develop * dump param test=develop * dump param test=develop	7 years ago
xujiaqi01	48669aa8f0	fix several sparse table issuses (#20686) ... * no longer need to define all embedding layers (no one less) of all slots in each program. make trainer_param repeated in ps.proto. * add find_distributed_lookup_table_grads instead of hard code GRAD * support embedding stop gradient. push sparse has error before fix this.* * fix fill sparse, skip slots which do not have embedding. each slot's embedding in a sparse table should be used in all training programs before fix this. * fix pull sparse, skip slots which do not have embedding. * fix collect feasign label info, skip slots which do not have embedding. * support when there are multi sparse tables in one or multi training programs, each program can pull/push its own related sparse tables instead of all sparse tables. * test=develop	7 years ago
xujiaqi01	5223b0dd9d	add check nan / inf in downpour worker (#20694) ... * add check nan / inf in downpour worker during training * test=develop	7 years ago
Thunderbrook	f76a32df4a	dump fix dov vec file num (#20539) ... * support dump multi file test=develop * dump fix num file test=develop	7 years ago
zhang wenhui	b521992041	fix converter , test=develop (#20522)	7 years ago
zhang wenhui	b82e6520e1	fix pslib datanorm double bug (#20297) ... * fix fc sort . test=develop	7 years ago
zhang wenhui	b28d4a824f	fix fleet_desc delete_after_unseen_day bug in node.py (#20091)	7 years ago
xujiaqi01	cedc04775c	support change shuffle and train thread num (#19841) ... * support change shuffle thread num * support change train thread num * fix receive shuffle data of each channel * data norm stop gradient * add check thread_tensor type and root_tensor type when merge metric * remove sleep in shuffle, add config * add config of pslib client to client communication * fix xbox str * add data norm op testcase * add flush in trainer finalize	7 years ago
yaoxuefeng	10ca3f9609	add thread scope stat accurate metrics test=develop (#19480) ... * add thread scope stat accurate metrics test=develop * fix style * fix style * fix style * fix style test=develop * fix style test=develop * fix style test=develop * fix style test=develop * fix style test=develop * fix style test=develop * fix style test=develop * fix conflict * fix style * fix style test=develop * fix error test=develop * fix error test=develop	7 years ago
Thunderbrook	1fe468d319	support debug each output of each ins (#19004) ... * dump slot * test * proto * dump slot * test * proto * code style * code style * code style * style * add delete after unseen days * add unseen days * code style * conflict solve test=develop * add clear model * code style test=develop * code style test=develop * support debug tensor of each ins test=develop * support debug tensor of each ins test=develop * learning rate * code style * code style * code style * code style * code style * code style * code style * code style * code style * code style * code style * code style * code style test=develop * code style test=develop * unitest * style * style * multi phase * add channel * code style * style * style * unitest * style * define * define test=develop * style test=develop * rm define test=develop * linux * linux test=develop * style test=develop * output format test=develop * windows ci test=develop	7 years ago
zhang wenhui	bd35a7f0a6	support fc sort by number, test=develop (#19466) ... fleet_desc sort fc name by dictionary sort, but we want to sort by number.	7 years ago
zhang wenhui	0d7949831b	fix fleet_desc bug && support format for abacus hotstart (#19430) ... fix fleet_desc dense_table unsort bug ，not support format for abacus hotstart yet.	7 years ago
zhang wenhui	4a3c4b8fa4	add fleet_desc config feature & multi_sparse table, test=develop (#18827) ... add fleet_desc config feature & multi_sparse table,	7 years ago
jiaqi	b86be13c15	fix default value (#19193) ... * fix default value in ps_pb2.py: delta_keep_days 30 -> 16 * test=develop	7 years ago
jiaqi	b104ea0684	add get_last_save_xbox_base/get_last_save_xbox (#19122) ... * add get_last_save_xbox_base/get_last_save_xbox * fix fleet_util bug of load paddle model * add doc string in fleet api	7 years ago
jiaqi	bfd514c730	fix default value of fleet desc (#19176) ... * fix default value of fleet desc, default values are same with jingpai * print log when save model	7 years ago