16 Commits (26ae6d49e481cfc68250aba453fb0a598460bb9a)

Author	SHA1	Message	Date
jiaqi	3f8031e256	dataset (#17973) ... (1) use channel instead of vector/BlockingQueue in Dataset，to keep same with existing implementation, and make code more readable and flexible (dataset single output channel or multi output channel). one previous memory out of limit problem is cause by not release memory after training. (2) add Record because MultiSlotType costs too much memory (80B)，fix memory out of limit problem. (3) add Channel, Archive in paddle/fluid/framework (4) change dataset from shared_ptr to unique_ptr in pybind (5) move create/destroy readers from trainer to dataset (6) move shuffle from datafeed to dataset. dataset holds memory, datafeed is only for load data and feed data to network. (7) fix thread num bug of Dataset when filelist size < thread num (8) support set_queue_num in InMemoryDataset	6 years ago
dongdaxiang	ba15d6b164	move root_scope->DropKids() into Finalize() so that we do not have to drop all the kids ... test=develop	6 years ago
xujiaqi01	a5b1a0e12b	support multi dataset && add init model && fix bug	6 years ago
dongdaxiang	6af697adb0	add trainfileswithprofiler for downpour worker	6 years ago
dongdaxiang	2644b88685	add comment for MPI Symetric role maker ... test=develop	6 years ago
dongdaxiang	cf45c54340	add distributed optimizer factory	6 years ago
xujiaqi01	39449ba0b9	fix bug && add DestroyReaders in trainer	6 years ago
dongdaxiang	328f11b8b6	refactor downpour optimization ... test=develop	6 years ago
dongdaxiang	b66f0074b6	fix data reading bugs in api, add VLOG(3) log for setup	6 years ago
dongdaxiang	b415ec27e8	make Dataset* as an argument	6 years ago
xjqbest	dd67ad08a2	modify c++ and python dataset related code & fix bug	6 years ago
dongdaxiang	2486389793	add RunFromDataset in executor	6 years ago
xjqbest	824b84d185	add DataSet and InMemoryDataFeed, support load data into memory and shuffle data	6 years ago
dongdaxiang	39014b9f9f	fix class register problem	6 years ago
dongdaxiang	c165012031	refine device_worker and trainer code ... test=develop	6 years ago
dongdaxiang	855bf579d2	add dist_multi_trainer for distributed training, add trainer_factory and device_worker_factory so that we can easily extend new training mode, add pull dense worker which is a singleton for parameter fetching	6 years ago