91 Commits (88e6dc4ac5a5f0a4ed0c54365e4210528da6f3ab)

Author	SHA1	Message	Date
xujiaqi01	6bf298bf09	support preload thread, optimize hdfs log, fix master+patch bug (#19695) ... * support preload thread * sleep before fleet wrapper exit for pslib core dump * optimize hdfs log * fix master+patch bug	5 years ago
hutuxian	c756b5d231	Paddlebox Framework (#18982) ... * Support looking up embeddings from BoxPS. * Add a _pull_box_sparse op, for now this op is not exposed to users. * Add a BoxHelper class, providing 'BeginPass', 'EndPass', 'FeedPass' functions and so on. * Add 'BoxPSDataset' in python code. * Add a compile options WITH_BOX_PS and a MACRO PADDLE_WITH_BOX_PS. * Add UT. * More concrete information pls refer to: https://github.com/PaddlePaddle/Paddle/pull/18982	6 years ago
jiaqi	b104ea0684	add get_last_save_xbox_base/get_last_save_xbox (#19122) ... * add get_last_save_xbox_base/get_last_save_xbox * fix fleet_util bug of load paddle model * add doc string in fleet api	6 years ago
yaoxuefeng	9150cf50fc	add save cache model api in fleet& add slots shuffle in dataset module & add metric op to calculate ctr related metrics (#18871) ... * add ctr related metric layer test=develop * add save cache and slots shuffle test=develop * add save cache and slots shuffle test=develop * fix error * fix error * fix style for ci * fix for comments * change SlotsShuffle input to std::strinf for generality * fix style * fix style * fix style * fix style * fix style * fix style * fix stylr * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * change non-const reference to pointer * fix style * fix style * fix style test=develop * fix style test=develop * add return ins num in ctr metric op * change dtype to float in metric_op.py * fix error test=develop * fix style test=develop * fix API spec * fix API spec * fix API spec test=develop * add UT test=develop	6 years ago
Thunderbrook	52c1431eee	add clear_model interface in fleetwrapper (#18815) ... * dump slot * test * proto * dump slot * test * proto * code style * code style * code style * style * add delete after unseen days * add unseen days * code style * conflict solve test=develop * add clear model * code style test=develop * code style test=develop	6 years ago
fuyinno4	c167a4b4dd	Fix shrink-dense and add scale-datanorm (#18746) ... Fix FleetWrapper: 1. fix shrink dense: just scale show 2. add datanorm scale: divide datanorm's gradient by batch_size	6 years ago
Thunderbrook	d8396281ef	add slot to sparse table (#18686) ... The change includes 2 things: 1. save delta model and shrink table are control by the same parameter before, now add delete_after_unseen_days to control shrink table. 2. value in sparse table has no slot before, now add slot in sparse table, and add DownpureCtrAccessor to support the new meta. test=develop	6 years ago
jiaqi	d18aabb472	support patch data, add load_one_table, fix bug (#18509) ... （1）support patch data （merge slots of instances of same line id, modify dense layer which changes its size） （2）add fleet load_one_table interface, support load from paddle model and load from pslib model （3）fix push sparse bug which cause push sparse cost more time（about 10% in my testcase） （4）when some slots are not in one of your network (join/update, etc.)，data feed、collect label info、push/pull sparse will skip these slots， instead of throw error. （5）add more debug info in TrainFilesWithProfiler	6 years ago
jiaqi	66d51206b1	add save/load model, shrink table, cvm, config file & fix pull dense bug (#17118) ... * add save/load model, shrink table, cvm, config file & fix pull dense bug test=develop * fix global shuffle bug, fix pull dense bug, fix release memeory bug, fix shrink error add client flush, add get data size test=develop * fix global shuffle bug test=develop * fix global shuffle bug test=develop * fix code style test=develop * fix code style & modify pslib cmake test=develop * fix error of _role_maker test=develop * fix code style test=develop * fix code style test=develop * fix code style test=develop * fix code style test=develop * fix code style test=develop * fix windows compile error of fleet test=develop * fix global shuffle bug * add comment test=develop * update pslib.cmake test=develop * fix fill sparse bug test=develop * fix push sparse bug test=develop	6 years ago
wopeizl	51a0243a56	fix nccl wrapper on windows ... test=develop	6 years ago
dongdaxiang	2ab2869c2d	fix GPU compile error problem	6 years ago
dongdaxiang	466d177d09	add pybind dependency ... test=develop	6 years ago
dongdaxiang	b091139049	add nccl wrapper for python API	6 years ago
dongdaxiang	fff795e5c8	add nccl_wrapper	6 years ago
xjqbest	5e5139283b	fix runtime error ... test=develop	6 years ago
dongdaxiang	718ea6dbd5	fix fleet code style ... test=develop	6 years ago
xjqbest	782ab2e2bd	add some doc ... test=develop	6 years ago
xjqbest	a99c8d0c29	fix client to client communication bug ... test=develop	6 years ago
dongdaxiang	98dda08a85	fix pull sparse slow problem ... test=develop	6 years ago
dongdaxiang	2708108a08	fix fleet_wrapper compile on windows ... test=develop	6 years ago
dongdaxiang	d4514949bf	remove local random engine in fleet with rand_r() ... test=develop	6 years ago
dongdaxiang	a0b59773af	fix code style	6 years ago
dongdaxiang	365be5d559	support win32 flag in io.cc shell.cc, fix code style problem in fleet_wrapper, fix lodtensor_printer_test problem ... test=develop	6 years ago
xjqbest	b7940c2918	fix bug of gen_worker_desc and set_filelist, add some doc	6 years ago
xjqbest	a34fe6248f	add some doc	6 years ago
xujiaqi01	f5c6a14b54	fix runtime error	6 years ago
xujiaqi01	a5b1a0e12b	support multi dataset && add init model && fix bug	6 years ago
dongdaxiang	b7a202aa38	add distributed optimizer factory	6 years ago
xujiaqi01	d25389fefd	add some log && fix error	6 years ago
dongdaxiang	317eb0aad3	add incubate for unified API	6 years ago
xujiaqi01	39449ba0b9	fix bug && add DestroyReaders in trainer	6 years ago
xujiaqi01	ecfc7df913	add dataset factory && fix style	6 years ago
xujiaqi01	3cea00bd52	store memory data in Dataset && fix bug	6 years ago
dongdaxiang	be757096da	add pybind for fleet	6 years ago
dongdaxiang	d65cb13ad5	add pslib flag on fleet_wrapper CMakefile	6 years ago
dongdaxiang	6de9ebc65c	refine VLOG in fleet_wrapper.h ... test=develop	6 years ago
dongdaxiang	f2bde9c241	fix destructor problem	6 years ago
dongdaxiang	378037c535	make s_instance_ private to ensure singleton	6 years ago
dongdaxiang	c165012031	refine device_worker and trainer code ... test=develop	6 years ago
dongdaxiang	24a8001142	make -DWITH_PSLIB=ON compilable	6 years ago
dongdaxiang	855bf579d2	add dist_multi_trainer for distributed training, add trainer_factory and device_worker_factory so that we can easily extend new training mode, add pull dense worker which is a singleton for parameter fetching	6 years ago