Paddle

Commit Graph

Author	SHA1	Message	Date
Thunderbrook	349e82d669	support general embedding params (#21217 ) * general table * add sparse table test=develop * no cvm test=develop * add no_cvm test=develop * add note test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * code style test=develop * add key of optimizer test=develop	7 years ago
xujiaqi01	23876de55b	fix cache table bug, add save_paddle_inference_model, fix hdfs util bug (#21052 ) * fix cache table bug * add save_paddle_inference_model * fix hdfs util bug * test=develop	7 years ago
xujiaqi01	9e045170c0	add copy table (#21086 ) * copy some feasigns and corresponding embeddings from one sparse table to another * copy all feasigns and corresponding embeddings from one sparse table to another * copy all dense params from one table to another * copy some local vars to other local vars	7 years ago
Thunderbrook	5970e8ac5e	find lookup table in order (#20932 ) test=develop	7 years ago
Thunderbrook	59bcdc8a19	support dump param of model into afs (#20302 ) * support dump param to afs test=develop * code style test=develop * code style test=develop * dump param test=develop * dump param test=develop * dump param test=develop * dump param test=develop	7 years ago
xujiaqi01	48669aa8f0	fix several sparse table issuses (#20686 ) * no longer need to define all embedding layers (no one less) of all slots in each program. make trainer_param repeated in ps.proto. * add find_distributed_lookup_table_grads instead of hard code GRAD * support embedding stop gradient. push sparse has error before fix this.* * fix fill sparse, skip slots which do not have embedding. each slot's embedding in a sparse table should be used in all training programs before fix this. * fix pull sparse, skip slots which do not have embedding. * fix collect feasign label info, skip slots which do not have embedding. * support when there are multi sparse tables in one or multi training programs, each program can pull/push its own related sparse tables instead of all sparse tables. * test=develop	7 years ago
xujiaqi01	5223b0dd9d	add check nan / inf in downpour worker (#20694 ) * add check nan / inf in downpour worker during training * test=develop	7 years ago
Thunderbrook	f76a32df4a	dump fix dov vec file num (#20539 ) * support dump multi file test=develop * dump fix num file test=develop	7 years ago
zhang wenhui	b521992041	fix converter , test=develop (#20522 )	7 years ago
zhang wenhui	b82e6520e1	fix pslib datanorm double bug (#20297 ) * fix fc sort . test=develop	7 years ago
zhang wenhui	b28d4a824f	fix fleet_desc delete_after_unseen_day bug in node.py (#20091 )	7 years ago
xujiaqi01	cedc04775c	support change shuffle and train thread num (#19841 ) * support change shuffle thread num * support change train thread num * fix receive shuffle data of each channel * data norm stop gradient * add check thread_tensor type and root_tensor type when merge metric * remove sleep in shuffle, add config * add config of pslib client to client communication * fix xbox str * add data norm op testcase * add flush in trainer finalize	7 years ago
yaoxuefeng	10ca3f9609	add thread scope stat accurate metrics test=develop (#19480 ) * add thread scope stat accurate metrics test=develop * fix style * fix style * fix style * fix style test=develop * fix style test=develop * fix style test=develop * fix style test=develop * fix style test=develop * fix style test=develop * fix style test=develop * fix conflict * fix style * fix style test=develop * fix error test=develop * fix error test=develop	7 years ago
Thunderbrook	1fe468d319	support debug each output of each ins (#19004 ) * dump slot * test * proto * dump slot * test * proto * code style * code style * code style * style * add delete after unseen days * add unseen days * code style * conflict solve test=develop * add clear model * code style test=develop * code style test=develop * support debug tensor of each ins test=develop * support debug tensor of each ins test=develop * learning rate * code style * code style * code style * code style * code style * code style * code style * code style * code style * code style * code style * code style * code style test=develop * code style test=develop * unitest * style * style * multi phase * add channel * code style * style * style * unitest * style * define * define test=develop * style test=develop * rm define test=develop * linux * linux test=develop * style test=develop * output format test=develop * windows ci test=develop	7 years ago
zhang wenhui	bd35a7f0a6	support fc sort by number, test=develop (#19466 ) fleet_desc sort fc name by dictionary sort, but we want to sort by number.	7 years ago
zhang wenhui	0d7949831b	fix fleet_desc bug && support format for abacus hotstart (#19430 ) fix fleet_desc dense_table unsort bug ，not support format for abacus hotstart yet.	7 years ago
zhang wenhui	4a3c4b8fa4	add fleet_desc config feature & multi_sparse table, test=develop (#18827 ) add fleet_desc config feature & multi_sparse table,	7 years ago
jiaqi	b86be13c15	fix default value (#19193 ) * fix default value in ps_pb2.py: delta_keep_days 30 -> 16 * test=develop	7 years ago
jiaqi	b104ea0684	add get_last_save_xbox_base/get_last_save_xbox (#19122 ) * add get_last_save_xbox_base/get_last_save_xbox * fix fleet_util bug of load paddle model * add doc string in fleet api	7 years ago
jiaqi	bfd514c730	fix default value of fleet desc (#19176 ) * fix default value of fleet desc, default values are same with jingpai * print log when save model	7 years ago
yaoxuefeng	9150cf50fc	add save cache model api in fleet& add slots shuffle in dataset module & add metric op to calculate ctr related metrics (#18871 ) * add ctr related metric layer test=develop * add save cache and slots shuffle test=develop * add save cache and slots shuffle test=develop * fix error * fix error * fix style for ci * fix for comments * change SlotsShuffle input to std::strinf for generality * fix style * fix style * fix style * fix style * fix style * fix style * fix stylr * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * change non-const reference to pointer * fix style * fix style * fix style test=develop * fix style test=develop * add return ins num in ctr metric op * change dtype to float in metric_op.py * fix error test=develop * fix style test=develop * fix API spec * fix API spec * fix API spec test=develop * add UT test=develop	7 years ago
jiaqi	768059b3a0	adjust ins weight according to nid slot (#18784 ) adjust ins weight according to nid slot , user can specify adjust_ins_weight in strategy	7 years ago
jiaqi	233746d89d	set fleet_send_batch_num a default value according to trainer num (1) set fleet_send_batch_num a default value according to trainer num， the previous 80000 is fixed，if trainer num is much less or larger than 100，global shuffle may have timeout error. (2) fix load one table bug, add barrier	7 years ago
Thunderbrook	52c1431eee	add clear_model interface in fleetwrapper (#18815 ) * dump slot * test * proto * dump slot * test * proto * code style * code style * code style * style * add delete after unseen days * add unseen days * code style * conflict solve test=develop * add clear model * code style test=develop * code style test=develop	7 years ago
fuyinno4	c167a4b4dd	Fix shrink-dense and add scale-datanorm (#18746 ) Fix FleetWrapper: 1. fix shrink dense: just scale show 2. add datanorm scale: divide datanorm's gradient by batch_size	7 years ago
Thunderbrook	d8396281ef	add slot to sparse table (#18686 ) The change includes 2 things: 1. save delta model and shrink table are control by the same parameter before, now add delete_after_unseen_days to control shrink table. 2. value in sparse table has no slot before, now add slot in sparse table, and add DownpureCtrAccessor to support the new meta. test=develop	7 years ago
jiaqi	d18aabb472	support patch data, add load_one_table, fix bug (#18509 ) （1）support patch data （merge slots of instances of same line id, modify dense layer which changes its size）（2）add fleet load_one_table interface, support load from paddle model and load from pslib model （3）fix push sparse bug which cause push sparse cost more time（about 10% in my testcase）（4）when some slots are not in one of your network (join/update, etc.)，data feed、collect label info、push/pull sparse will skip these slots， instead of throw error. （5）add more debug info in TrainFilesWithProfiler	7 years ago
tangwei12	101f74cb19	fix save/load in fleet (#17675 ) * fix save/load in Fleet * add UT framework of Fleet	7 years ago
Qiao Longfei	58f7695ab2	Async exe support communicator (#17386 ) Async exe support communicator	7 years ago
jiaqi	05df39ac06	support sparse table get shard_num from TableParameter (#17443 ) test=develop	7 years ago
jiaqi	34369944f5	support config file, cvm, load, save, shrink (#17319 ) * support config file, cvm, load, save, shrink test=develop * fix error of worker_num & add table.compress_in_save test=develop * fix code style test=develop * fix save model bug test=develop	7 years ago
tangwei12	565d309501	Reformat fleet API (#17135 ) * fix some logic in distributed transpiler, test=develop * reformat fleet API, test=develop	7 years ago
tangwei12	1a4a51db2b	Fleet unify distributed training (#16791 ) * implement distributed transpiler with fleet	7 years ago

33 Commits (6fc3e8ec84af9e9fa2f191cbcc402de080962a67)