18 Commits (08fa98f7cc75187d7a2030e07ae1e349979f27cc)

Author	SHA1	Message	Date
fuyinno4	c167a4b4dd	Fix shrink-dense and add scale-datanorm (#18746) ... Fix FleetWrapper: 1. fix shrink dense: just scale show 2. add datanorm scale: divide datanorm's gradient by batch_size	6 years ago
Thunderbrook	d8396281ef	add slot to sparse table (#18686) ... The change includes 2 things: 1. save delta model and shrink table are control by the same parameter before, now add delete_after_unseen_days to control shrink table. 2. value in sparse table has no slot before, now add slot in sparse table, and add DownpureCtrAccessor to support the new meta. test=develop	6 years ago
jiaqi	93a2b317f7	fix data feed ptr error (#18419) ... fix data feed ptr runtime error, pipeline trainer will core in some cases, so set it nullptr as default value.	6 years ago
jiaqi	3f8031e256	dataset (#17973) ... (1) use channel instead of vector/BlockingQueue in Dataset，to keep same with existing implementation, and make code more readable and flexible (dataset single output channel or multi output channel). one previous memory out of limit problem is cause by not release memory after training. (2) add Record because MultiSlotType costs too much memory (80B)，fix memory out of limit problem. (3) add Channel, Archive in paddle/fluid/framework (4) change dataset from shared_ptr to unique_ptr in pybind (5) move create/destroy readers from trainer to dataset (6) move shuffle from datafeed to dataset. dataset holds memory, datafeed is only for load data and feed data to network. (7) fix thread num bug of Dataset when filelist size < thread num (8) support set_queue_num in InMemoryDataset	6 years ago
hutuxian	969e6378b9	Pipeline Concurrency (#17402) ... Add Pipeline Concurrency Train Mode: - Cpp: pipeline_trainer & section_worker - Python: PipelineOptimizer - Add a new data_feed type: PrivateInstantDataFeed - Add a test demo of pipeline trainer and the test model is gnn - Do not support win32 now	6 years ago
jiaqi	66d51206b1	add save/load model, shrink table, cvm, config file & fix pull dense bug (#17118) ... * add save/load model, shrink table, cvm, config file & fix pull dense bug test=develop * fix global shuffle bug, fix pull dense bug, fix release memeory bug, fix shrink error add client flush, add get data size test=develop * fix global shuffle bug test=develop * fix global shuffle bug test=develop * fix code style test=develop * fix code style & modify pslib cmake test=develop * fix error of _role_maker test=develop * fix code style test=develop * fix code style test=develop * fix code style test=develop * fix code style test=develop * fix code style test=develop * fix windows compile error of fleet test=develop * fix global shuffle bug * add comment test=develop * update pslib.cmake test=develop * fix fill sparse bug test=develop * fix push sparse bug test=develop	6 years ago
dongdaxiang	60b7bf6fa6	add infer_from_dataset for inference	6 years ago
dongdaxiang	9e51ad4a65	fix io and fs compile on mac ... test=develop	6 years ago
dongdaxiang	4ce35815fb	fix windows GLOG problem ... test=develop	6 years ago
dongdaxiang	b95b80bc76	add doc string for executor and update API.spec ... test=develop	6 years ago
dongdaxiang	68d7bf3de5	add fetch var function ... test=develop	6 years ago
dongdaxiang	2644b88685	add comment for MPI Symetric role maker ... test=develop	6 years ago
heqiaozhi	9bca1926c1	refactor & fix bug	6 years ago
dongdaxiang	cf1360643f	add printer for fetch variable	6 years ago
dongdaxiang	97d5cd30f0	make pull dense worker work	6 years ago
dongdaxiang	378037c535	make s_instance_ private to ensure singleton	6 years ago
dongdaxiang	c165012031	refine device_worker and trainer code ... test=develop	6 years ago
dongdaxiang	855bf579d2	add dist_multi_trainer for distributed training, add trainer_factory and device_worker_factory so that we can easily extend new training mode, add pull dense worker which is a singleton for parameter fetching	6 years ago