18 Commits (7ac748adb4b9bfbf5fc0fc8b7722d65618d85ead)

Author	SHA1	Message	Date
jiaqi	d18aabb472	support patch data, add load_one_table, fix bug (#18509) ... （1）support patch data （merge slots of instances of same line id, modify dense layer which changes its size） （2）add fleet load_one_table interface, support load from paddle model and load from pslib model （3）fix push sparse bug which cause push sparse cost more time（about 10% in my testcase） （4）when some slots are not in one of your network (join/update, etc.)，data feed、collect label info、push/pull sparse will skip these slots， instead of throw error. （5）add more debug info in TrainFilesWithProfiler	6 years ago
jiaqi	3f8031e256	dataset (#17973) ... (1) use channel instead of vector/BlockingQueue in Dataset，to keep same with existing implementation, and make code more readable and flexible (dataset single output channel or multi output channel). one previous memory out of limit problem is cause by not release memory after training. (2) add Record because MultiSlotType costs too much memory (80B)，fix memory out of limit problem. (3) add Channel, Archive in paddle/fluid/framework (4) change dataset from shared_ptr to unique_ptr in pybind (5) move create/destroy readers from trainer to dataset (6) move shuffle from datafeed to dataset. dataset holds memory, datafeed is only for load data and feed data to network. (7) fix thread num bug of Dataset when filelist size < thread num (8) support set_queue_num in InMemoryDataset	6 years ago
jiaqi	66d51206b1	add save/load model, shrink table, cvm, config file & fix pull dense bug (#17118) ... * add save/load model, shrink table, cvm, config file & fix pull dense bug test=develop * fix global shuffle bug, fix pull dense bug, fix release memeory bug, fix shrink error add client flush, add get data size test=develop * fix global shuffle bug test=develop * fix global shuffle bug test=develop * fix code style test=develop * fix code style & modify pslib cmake test=develop * fix error of _role_maker test=develop * fix code style test=develop * fix code style test=develop * fix code style test=develop * fix code style test=develop * fix code style test=develop * fix windows compile error of fleet test=develop * fix global shuffle bug * add comment test=develop * update pslib.cmake test=develop * fix fill sparse bug test=develop * fix push sparse bug test=develop	7 years ago
xjqbest	6a57e8075a	remove trainer_id in datafeed and dataset ... test=develop	7 years ago
xjqbest	271b7147cc	fix dataset bug ... test=develop	7 years ago
dongdaxiang	d4514949bf	remove local random engine in fleet with rand_r() ... test=develop	7 years ago
dongdaxiang	45eb6f0765	run pre-commit check files and fix code style problem ... test=develop	7 years ago
xjqbest	e95cafd9a7	fix code style & add dataset testcase ... test=develop	7 years ago
xjqbest	be74de2c61	fix code style & fix register bug & add release_memory ... test=develop	7 years ago
xjqbest	a34fe6248f	add some doc	7 years ago
xujiaqi01	a5b1a0e12b	support multi dataset && add init model && fix bug	7 years ago
xujiaqi01	d25389fefd	add some log && fix error	7 years ago
dongdaxiang	e657c127a8	hide opt_info in distirbuted optimizer	7 years ago
xujiaqi01	3cea00bd52	store memory data in Dataset && fix bug	7 years ago
dongdaxiang	b415ec27e8	make Dataset* as an argument	7 years ago
xjqbest	dd67ad08a2	modify c++ and python dataset related code & fix bug	7 years ago
dongdaxiang	e36bbcc871	fix some typo and CMakefile.txt	7 years ago
xjqbest	824b84d185	add DataSet and InMemoryDataFeed, support load data into memory and shuffle data	7 years ago