Paddle

Commit Graph

Author	SHA1	Message	Date
wanghuancoder	35c5b23f68	use iwyu clean include second time, test=develop (#30829 ) * use iwyu clean include second time, test=develop	4 years ago
Thunderbrook	5205748481	fix eigen in push sparse; fix hadoop command (#26872 ) * fix eigen in push sparse; fix hadoop command test=develop * add log in load_combine_op test=develop	5 years ago
hutuxian	5822862d8a	Monitor Framework (#24079 ) * Add a StatValue class in the backend to represent a stat. * Add a singleton StatRegistry to maintain the collection of stats. * For the sake of code neatness, we only support type of int and float, which can cover most of the scenarios.	5 years ago
hutuxian	1c224e26af	support CMatchAuc (#24990 ) Support CMatchAucCalculator based on CMatchRankAucCalculator with a new parameter ignore_rank	5 years ago
hutuxian	e6b87b3193	Support AucRunner in PaddleBox (#22884 ) * Support AucRunner in PaddleBox * update some code style	5 years ago
ShenLiang	94dfb7d770	opt the postprocess, test=develop (#24155 )	5 years ago
ShenLiang	7f0b2c7407	fix memory leaking problem of dataset, test=develop (#23955 )	5 years ago
ShenLiang	5223e2bbc4	Add a new DataFeed named PaddleBoxDataFeed (#23321 ) * add paddleboxdatafeed * add ifdef linux and boxps * add untest for datafeed * fix untest of test_paddlebox_datafeed * fix untest * rename function	5 years ago
yaoxuefeng	660ff18488	fix datsset test=develop (#23043 )	5 years ago
hutuxian	53a2b68f4e	support customized download command in dataset (#22782 ) * user can call dataset.set_download_cmd to set its customized download cmd * add UT to cover this scenario	5 years ago
hutuxian	175954d894	PaddleBox Framework Part2 (#22466 ) * Add two types of Metric Calculator: MultiTaskCalculator & CmatchRankCalculator. * Add a config for DynamicAdjustChannelNum function to denote whether we will discard the remaining instances when they are not be distributed evenly. * Remove CPU code in Pull/PushSparse and we will add it back when testing it fully. * Fix some known issues: such as copying persistable vars after one epoch running.	5 years ago
yaoxuefeng	2235ee1a5e	multi-loss optimization by adding a DownpourOpt worker (#22025 ) * update * update test=develop * update compile set test=develop * update compile set test=develop * update test=develop * update test=develop * update test=develop * update compile setting test=develop * update compile setting test=develop * update run demo test=develop * update test=develop * update test=develop * fix test=develop * update test=develop * update test=develop * update test=develop * update test=develop * update test=develop * update test=develop * update test=develop * update test=develop * update test=develop * update format test=develop * update format test=develop * update style test=develop * update style test=develop * change style test=develop * change style test=develop * change style test=develop * add dataset unittest test=develop * update test=develop * update for record test=develop * udpate style for record test=develop * update for record test=develop * update for record test=develop * update for record test=develop * fix format test=develop * update test=develop * update test=develop * update test=develop * update test=develop * update test=develop	5 years ago
xujiaqi01	371f377bea	add GeneralRoleMaker (#22295 ) * add GeneralRoleMaker which is for general usage * test=develop	5 years ago
xujiaqi01	f404157205	fix master patch when slot is dense (#21580 ) * fix master patch when slot is dense * test=develop	5 years ago
hutuxian	c5aec2fe68	Paddlebox Related to Framework (#21586 ) * Add a single_process_multi_thread transpiler. * Add some UTs. * Fix some API description.	5 years ago
Tao Luo	01fa4ead61	fix -Wno-error=sign-compare warning in gcc8 (#21434 ) * fix -Wno-error=sign-compare warning in gcc8 test=develop * fix warning in distributed codes test=develop	5 years ago
Tao Luo	c0656dcb1a	remove -Wno-error=sign-compare, make warning as error (#21358 ) * remove -Wno-error=sign-compare, make warning as error test=develop test=document_fix * fix exist compile warning test=develop	5 years ago
xujiaqi01	1d1a07937a	simplify master+patch，remove ins when size != merge_size or has conflict slot (#20913 ) * remove duplicate code and duplicate config of master+patch * drop all ins which has conflict slot or size < merge_size * user only need to set merge size，if ins num of same id is not equal to merge size, just drop these ins * user must make sure master data and patch data has no same slot whose feasigns are both non-zero, otherwise these ins will be dropped. (slot list should still be the same of both master and patch) * test=develop	5 years ago
xujiaqi01	22b80e1246	fix parse content in CreatePreLoadReaders (#20258 ) Fix parse content in CreatePreLoadReaders. Before this fix, if you use dataset.set_parse_content and dataset.preload, parse content didn't work.	5 years ago
xujiaqi01	cedc04775c	support change shuffle and train thread num (#19841 ) * support change shuffle thread num * support change train thread num * fix receive shuffle data of each channel * data norm stop gradient * add check thread_tensor type and root_tensor type when merge metric * remove sleep in shuffle, add config * add config of pslib client to client communication * fix xbox str * add data norm op testcase * add flush in trainer finalize	5 years ago
xujiaqi01	6bf298bf09	support preload thread, optimize hdfs log, fix master+patch bug (#19695 ) * support preload thread * sleep before fleet wrapper exit for pslib core dump * optimize hdfs log * fix master+patch bug	5 years ago
Thunderbrook	1fe468d319	support debug each output of each ins (#19004 ) * dump slot * test * proto * dump slot * test * proto * code style * code style * code style * style * add delete after unseen days * add unseen days * code style * conflict solve test=develop * add clear model * code style test=develop * code style test=develop * support debug tensor of each ins test=develop * support debug tensor of each ins test=develop * learning rate * code style * code style * code style * code style * code style * code style * code style * code style * code style * code style * code style * code style * code style test=develop * code style test=develop * unitest * style * style * multi phase * add channel * code style * style * style * unitest * style * define * define test=develop * style test=develop * rm define test=develop * linux * linux test=develop * style test=develop * output format test=develop * windows ci test=develop	6 years ago
yaoxuefeng	9150cf50fc	add save cache model api in fleet& add slots shuffle in dataset module & add metric op to calculate ctr related metrics (#18871 ) * add ctr related metric layer test=develop * add save cache and slots shuffle test=develop * add save cache and slots shuffle test=develop * fix error * fix error * fix style for ci * fix for comments * change SlotsShuffle input to std::strinf for generality * fix style * fix style * fix style * fix style * fix style * fix style * fix stylr * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * change non-const reference to pointer * fix style * fix style * fix style test=develop * fix style test=develop * add return ins num in ctr metric op * change dtype to float in metric_op.py * fix error test=develop * fix style test=develop * fix API spec * fix API spec * fix API spec test=develop * add UT test=develop	6 years ago
jiaqi	02c370c3dc	support filelist size < trainer num && fix pull dense (#18956 ) * support filelist size < trainer num * pull dense when stop, to make sure local dense params are same as pserver, so save paddle model will save dense model same as pserver * enable QueueDataset train same filelist for serveral times	6 years ago
jiaqi	d18aabb472	support patch data, add load_one_table, fix bug (#18509 ) （1）support patch data （merge slots of instances of same line id, modify dense layer which changes its size）（2）add fleet load_one_table interface, support load from paddle model and load from pslib model （3）fix push sparse bug which cause push sparse cost more time（about 10% in my testcase）（4）when some slots are not in one of your network (join/update, etc.)，data feed、collect label info、push/pull sparse will skip these slots， instead of throw error. （5）add more debug info in TrainFilesWithProfiler	6 years ago
guru4elephant	d714bf037c	remove async executor and add data_feed.proto to the deps of train demo (#18659 ) * remove async executor and add data_feed.proto to the deps of train demo	6 years ago
jiaqi	3f8031e256	dataset (#17973 ) (1) use channel instead of vector/BlockingQueue in Dataset，to keep same with existing implementation, and make code more readable and flexible (dataset single output channel or multi output channel). one previous memory out of limit problem is cause by not release memory after training. (2) add Record because MultiSlotType costs too much memory (80B)，fix memory out of limit problem. (3) add Channel, Archive in paddle/fluid/framework (4) change dataset from shared_ptr to unique_ptr in pybind (5) move create/destroy readers from trainer to dataset (6) move shuffle from datafeed to dataset. dataset holds memory, datafeed is only for load data and feed data to network. (7) fix thread num bug of Dataset when filelist size < thread num (8) support set_queue_num in InMemoryDataset	6 years ago
jiaqi	66d51206b1	add save/load model, shrink table, cvm, config file & fix pull dense bug (#17118 ) * add save/load model, shrink table, cvm, config file & fix pull dense bug test=develop * fix global shuffle bug, fix pull dense bug, fix release memeory bug, fix shrink error add client flush, add get data size test=develop * fix global shuffle bug test=develop * fix global shuffle bug test=develop * fix code style test=develop * fix code style & modify pslib cmake test=develop * fix error of _role_maker test=develop * fix code style test=develop * fix code style test=develop * fix code style test=develop * fix code style test=develop * fix code style test=develop * fix windows compile error of fleet test=develop * fix global shuffle bug * add comment test=develop * update pslib.cmake test=develop * fix fill sparse bug test=develop * fix push sparse bug test=develop	6 years ago
xjqbest	6a57e8075a	remove trainer_id in datafeed and dataset test=develop	6 years ago
xjqbest	271b7147cc	fix dataset bug test=develop	6 years ago
dongdaxiang	ed31874397	undefine rand_r() test=develop	6 years ago
dongdaxiang	d4514949bf	remove local random engine in fleet with rand_r() test=develop	6 years ago
dongdaxiang	45eb6f0765	run pre-commit check files and fix code style problem test=develop	6 years ago
xjqbest	e95cafd9a7	fix code style & add dataset testcase test=develop	6 years ago
xjqbest	be74de2c61	fix code style & fix register bug & add release_memory test=develop	6 years ago
xjqbest	589467f24c	fix bug	6 years ago
xjqbest	b7940c2918	fix bug of gen_worker_desc and set_filelist, add some doc	6 years ago
xjqbest	a34fe6248f	add some doc	6 years ago
xujiaqi01	a5b1a0e12b	support multi dataset && add init model && fix bug	6 years ago
dongdaxiang	3c65cc1bbd	add document for role_maker and fleet parameter, data_generator	6 years ago
dongdaxiang	f6c9232a3d	fix dataset float32 type problem	6 years ago
xujiaqi01	d25389fefd	add some log && fix error	6 years ago
xujiaqi01	39449ba0b9	fix bug && add DestroyReaders in trainer	6 years ago
xujiaqi01	ecfc7df913	add dataset factory && fix style	6 years ago
xujiaqi01	3cea00bd52	store memory data in Dataset && fix bug	6 years ago
dongdaxiang	b66f0074b6	fix data reading bugs in api, add VLOG(3) log for setup	6 years ago
dongdaxiang	b415ec27e8	make Dataset* as an argument	6 years ago
xjqbest	dd67ad08a2	modify c++ and python dataset related code & fix bug	6 years ago
dongdaxiang	e36bbcc871	fix some typo and CMakefile.txt	6 years ago
xjqbest	824b84d185	add DataSet and InMemoryDataFeed, support load data into memory and shuffle data	6 years ago

50 Commits (e50bc2c2a6fcfec748d5bb991588a2fdc2ab4caf)