66 Commits (c4eb5d0378cadd0fe8ed0f079746de448aaae3c0)

Author	SHA1	Message	Date
yaoxuefeng	545df287fc	add user_define_dump (#28596)	4 years ago
yaoxuefeng	d726fd5e86	enhance dataset err msg (#27363)	4 years ago
Chen Weihang	dfb3ae1b9b	Polish some error message in framework holder - Part 1 (#25509) ... * polish some error message in framework, test=develop * fix unittest error, test=develop * replace PADDLE_ENFORCE, test=develop * polish details based review comment, test=develop	5 years ago
hutuxian	5822862d8a	Monitor Framework (#24079) ... * Add a StatValue class in the backend to represent a stat. * Add a singleton StatRegistry to maintain the collection of stats. * For the sake of code neatness, we only support type of int and float, which can cover most of the scenarios.	5 years ago
hutuxian	e6b87b3193	Support AucRunner in PaddleBox (#22884) ... * Support AucRunner in PaddleBox * update some code style	5 years ago
hutuxian	0ec3a42e97	Random Dump (#24477) ... * Refactor code for dump_field & dump_param: abstracting the common function in base class. * Support dump randomly & random with lineid * Support specify the random interval, which avoids printing too much logs.	5 years ago
wawltor	d1e1d85881	add the graph batch reader for pslib mode (#24178) ... Add the pslib graph batch reader mode, add the test case for this change	5 years ago
hutuxian	9ff558a46f	Optimize DataFeed (#23957) ... * Make batch_float_feasigns & batch_uint64_feasigns as member variable	5 years ago
hutuxian	94a3789fd0	Add AfsAPI in PaddleBox (#23419) ... * Involves AfsAPI to resolve slow downloading. * Mainly used in PaddleBox	5 years ago
ShenLiang	5223e2bbc4	Add a new DataFeed named PaddleBoxDataFeed (#23321) ... * add paddleboxdatafeed * add ifdef linux and boxps * add untest for datafeed * fix untest of test_paddlebox_datafeed * fix untest * rename function	5 years ago
Tao Luo	01fa4ead61	fix -Wno-error=sign-compare warning in gcc8 (#21434) ... * fix -Wno-error=sign-compare warning in gcc8 test=develop * fix warning in distributed codes test=develop	5 years ago
Tao Luo	c0656dcb1a	remove -Wno-error=sign-compare, make warning as error (#21358) ... * remove -Wno-error=sign-compare, make warning as error test=develop test=document_fix * fix exist compile warning test=develop	5 years ago
Chen Weihang	4bd9463630	fix detail error message error, test=develop (#21170)	5 years ago
Chen Weihang	8414575b78	Add examples for error message writing specification - PreconditionNotMet, Unimplemented, Unavailable (#21137) ... * add examples for error spec, test=develop * change ENFORCE to ENFORCE_**, test=develop	5 years ago
Chen Weihang	7e5f74b825	Add examples for error message writing specification - InvalidArgument (#21132) ... * add examples for error msg spec, test=develop * change ENFORCE to ENFORCE_**, test=develop * fix error, test=develop	5 years ago
Thunderbrook	1fe468d319	support debug each output of each ins (#19004) ... * dump slot * test * proto * dump slot * test * proto * code style * code style * code style * style * add delete after unseen days * add unseen days * code style * conflict solve test=develop * add clear model * code style test=develop * code style test=develop * support debug tensor of each ins test=develop * support debug tensor of each ins test=develop * learning rate * code style * code style * code style * code style * code style * code style * code style * code style * code style * code style * code style * code style * code style test=develop * code style test=develop * unitest * style * style * multi phase * add channel * code style * style * style * unitest * style * define * define test=develop * style test=develop * rm define test=develop * linux * linux test=develop * style test=develop * output format test=develop * windows ci test=develop	6 years ago
yaoxuefeng	9150cf50fc	add save cache model api in fleet& add slots shuffle in dataset module & add metric op to calculate ctr related metrics (#18871) ... * add ctr related metric layer test=develop * add save cache and slots shuffle test=develop * add save cache and slots shuffle test=develop * fix error * fix error * fix style for ci * fix for comments * change SlotsShuffle input to std::strinf for generality * fix style * fix style * fix style * fix style * fix style * fix style * fix stylr * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * fix style * change non-const reference to pointer * fix style * fix style * fix style test=develop * fix style test=develop * add return ins num in ctr metric op * change dtype to float in metric_op.py * fix error test=develop * fix style test=develop * fix API spec * fix API spec * fix API spec test=develop * add UT test=develop	6 years ago
hutuxian	5a80cc8431	Datafeed support reading to cuda place directly. (#19071) ... * add a place field in DataFeed to denote which place it will feed data to. * abstract the copy process in CopyToFeedTensor function * add UT for float32 type and for CUDAPlace	6 years ago
jiaqi	fc038da749	fix QueueDataset queue size (#19016) ... * fix QueueDataset queue size，set queue size = batch size * 100, to avoid too many instances in channel when training is much slower than reading data.	6 years ago
jiaqi	d18aabb472	support patch data, add load_one_table, fix bug (#18509) ... （1）support patch data （merge slots of instances of same line id, modify dense layer which changes its size） （2）add fleet load_one_table interface, support load from paddle model and load from pslib model （3）fix push sparse bug which cause push sparse cost more time（about 10% in my testcase） （4）when some slots are not in one of your network (join/update, etc.)，data feed、collect label info、push/pull sparse will skip these slots， instead of throw error. （5）add more debug info in TrainFilesWithProfiler	6 years ago
guru4elephant	d714bf037c	remove async executor and add data_feed.proto to the deps of train demo (#18659) ... * remove async executor and add data_feed.proto to the deps of train demo	6 years ago
jiaqi	3f8031e256	dataset (#17973) ... (1) use channel instead of vector/BlockingQueue in Dataset，to keep same with existing implementation, and make code more readable and flexible (dataset single output channel or multi output channel). one previous memory out of limit problem is cause by not release memory after training. (2) add Record because MultiSlotType costs too much memory (80B)，fix memory out of limit problem. (3) add Channel, Archive in paddle/fluid/framework (4) change dataset from shared_ptr to unique_ptr in pybind (5) move create/destroy readers from trainer to dataset (6) move shuffle from datafeed to dataset. dataset holds memory, datafeed is only for load data and feed data to network. (7) fix thread num bug of Dataset when filelist size < thread num (8) support set_queue_num in InMemoryDataset	6 years ago
hutuxian	969e6378b9	Pipeline Concurrency (#17402) ... Add Pipeline Concurrency Train Mode: - Cpp: pipeline_trainer & section_worker - Python: PipelineOptimizer - Add a new data_feed type: PrivateInstantDataFeed - Add a test demo of pipeline trainer and the test model is gnn - Do not support win32 now	6 years ago
guru4elephant	43c9561e9a	add inductive shape index (#17435) ... add inductive shape index	6 years ago
jiaqi	66d51206b1	add save/load model, shrink table, cvm, config file & fix pull dense bug (#17118) ... * add save/load model, shrink table, cvm, config file & fix pull dense bug test=develop * fix global shuffle bug, fix pull dense bug, fix release memeory bug, fix shrink error add client flush, add get data size test=develop * fix global shuffle bug test=develop * fix global shuffle bug test=develop * fix code style test=develop * fix code style & modify pslib cmake test=develop * fix error of _role_maker test=develop * fix code style test=develop * fix code style test=develop * fix code style test=develop * fix code style test=develop * fix code style test=develop * fix windows compile error of fleet test=develop * fix global shuffle bug * add comment test=develop * update pslib.cmake test=develop * fix fill sparse bug test=develop * fix push sparse bug test=develop	6 years ago
guru4elephant	5d6a1fcf16	fix infer_from_dataset and train_from_dataset (#17243) ... * fix train_from_dataset and infer_from_dataset example * add inductive dim for data_reader, example: shape=[-1, 1], then -1 will be inducted through run-time reading of number of elements	6 years ago
xjqbest	10991e00a9	fix bug of num > INT_MAX	6 years ago
xjqbest	241120d94d	fix bug of num > INT_MAX	6 years ago
xjqbest	dac70ad4c5	fix bug of num > INT_MAX	6 years ago
xjqbest	74471397cf	fix bug of num > INT_MAX	6 years ago
dongdaxiang	ea07eb8cd2	remove comment in data_feed.cc ... develop=test	6 years ago
dongdaxiang	05464e7c5c	add gpu training for Executor.train_from_dataset ... test=develop	6 years ago
xjqbest	6a57e8075a	remove trainer_id in datafeed and dataset ... test=develop	6 years ago
xjqbest	271b7147cc	fix dataset bug ... test=develop	6 years ago
xjqbest	a99c8d0c29	fix client to client communication bug ... test=develop	6 years ago
dongdaxiang	f7e4813804	add WIN32 for rand_r and usleep ... test=develop	6 years ago
dongdaxiang	cedbc161da	add more _LINUX maroc on data_feed.cc for mac and window compile ... test=develop	6 years ago
dongdaxiang	c5980c3566	add _LINUX macro ... test=develop	6 years ago
dongdaxiang	e3107a6ae0	fix windows compile problem ... test=develop	6 years ago
dongdaxiang	d4514949bf	remove local random engine in fleet with rand_r() ... test=develop	6 years ago
dongdaxiang	45eb6f0765	run pre-commit check files and fix code style problem ... test=develop	6 years ago
xjqbest	be74de2c61	fix code style & fix register bug & add release_memory ... test=develop	6 years ago
xujiaqi01	a5b1a0e12b	support multi dataset && add init model && fix bug	6 years ago
xujiaqi01	d25389fefd	add some log && fix error	6 years ago
xujiaqi01	39449ba0b9	fix bug && add DestroyReaders in trainer	6 years ago
xujiaqi01	ecfc7df913	add dataset factory && fix style	6 years ago
xujiaqi01	3cea00bd52	store memory data in Dataset && fix bug	6 years ago
dongdaxiang	b66f0074b6	fix data reading bugs in api, add VLOG(3) log for setup	6 years ago
xjqbest	dd67ad08a2	modify c++ and python dataset related code & fix bug	6 years ago
dongdaxiang	cc4def6ba5	fix some conflict for compilation	6 years ago