Run Python OP tests in a single Python process to improve test time. (#8362)
Currently, our tests run with 2 GPUs, the init time is absurdly long:
about 4s for each process. Currently, we run each OP test on
different processes. This PR:
1. create cmake function py_test_modules which will generate the
Makefile that runs a list of Python unittest module in a single Python
process.
2. move all "python unittest compatible" (e.g., used the unittest
package, not just a regular python file). from fluid/tests to
fluid/tests/unittests.
3. cmake now will run all OP tests in fluid/tests/unittests in a
single process, except the time-consuming tests, they are separated
into different processes to utilize parallelism. Please make sure to
use the unittest package if you put the python test file in
fluid/tests/unittests
4. remove all exit(0) from fluid/tests/unittests/*.py, exit(0) is used
to disable unittest, we can not do it when running all tests in a
single process since it will terminate the process without running the
other tests. Instead, the test is disabled in
fluid/tests/unittests/CMakeLists.txt. FIXME is added for each disabled
item. Please disable the unittest from
fluid/tests/unittests/CMakeLists.txt, instead of adding exit(0) to the
Python file, for all Python file in fluid/tests/unittests/.
5. add an option WITH_FAST_BUNDLE_TEST. When OFF, will run the unit
tests in separate process so that they can be tested individually.
7 years ago
file ( GLOB TEST_OPS RELATIVE "${CMAKE_CURRENT_SOURCE_DIR}" "test_*.py" )
string ( REPLACE ".py" "" TEST_OPS "${TEST_OPS}" )
set ( GC_ENVS FLAGS_eager_delete_tensor_gb=0.0 FLAGS_fast_eager_deletion_mode=1 FLAGS_memory_fraction_of_eager_deletion=1.0 )
Run Python OP tests in a single Python process to improve test time. (#8362)
Currently, our tests run with 2 GPUs, the init time is absurdly long:
about 4s for each process. Currently, we run each OP test on
different processes. This PR:
1. create cmake function py_test_modules which will generate the
Makefile that runs a list of Python unittest module in a single Python
process.
2. move all "python unittest compatible" (e.g., used the unittest
package, not just a regular python file). from fluid/tests to
fluid/tests/unittests.
3. cmake now will run all OP tests in fluid/tests/unittests in a
single process, except the time-consuming tests, they are separated
into different processes to utilize parallelism. Please make sure to
use the unittest package if you put the python test file in
fluid/tests/unittests
4. remove all exit(0) from fluid/tests/unittests/*.py, exit(0) is used
to disable unittest, we can not do it when running all tests in a
single process since it will terminate the process without running the
other tests. Instead, the test is disabled in
fluid/tests/unittests/CMakeLists.txt. FIXME is added for each disabled
item. Please disable the unittest from
fluid/tests/unittests/CMakeLists.txt, instead of adding exit(0) to the
Python file, for all Python file in fluid/tests/unittests/.
5. add an option WITH_FAST_BUNDLE_TEST. When OFF, will run the unit
tests in separate process so that they can be tested individually.
7 years ago
if ( NOT WITH_DISTRIBUTE )
list ( REMOVE_ITEM TEST_OPS test_recv_op )
list ( REMOVE_ITEM TEST_OPS test_dist_transpiler )
list ( REMOVE_ITEM TEST_OPS test_simple_dist_transpiler )
list ( REMOVE_ITEM TEST_OPS test_listen_and_serv_op )
LIST ( REMOVE_ITEM TEST_OPS test_dist_mnist )
LIST ( REMOVE_ITEM TEST_OPS test_dist_mnist_fleetapi )
LIST ( REMOVE_ITEM TEST_OPS test_dist_mnist_dgc_nccl )
LIST ( REMOVE_ITEM TEST_OPS test_dist_mnist_hallreduce )
LIST ( REMOVE_ITEM TEST_OPS test_dist_mnist_multi_comm )
LIST ( REMOVE_ITEM TEST_OPS test_dist_mnist_ring_allreduce )
LIST ( REMOVE_ITEM TEST_OPS test_dist_mnist_backward_deps )
LIST ( REMOVE_ITEM TEST_OPS test_dist_mnist_lars )
LIST ( REMOVE_ITEM TEST_OPS test_dist_word2vec )
LIST ( REMOVE_ITEM TEST_OPS test_dist_ctr )
LIST ( REMOVE_ITEM TEST_OPS test_dist_simnet_bow )
LIST ( REMOVE_ITEM TEST_OPS test_dist_mnist_batch_merge )
LIST ( REMOVE_ITEM TEST_OPS test_dist_text_classification )
LIST ( REMOVE_ITEM TEST_OPS test_nce_remote_table_op )
LIST ( REMOVE_ITEM TEST_OPS test_hsigmoid_remote_table_op )
LIST ( REMOVE_ITEM TEST_OPS test_dist_fleet_ctr )
Run Python OP tests in a single Python process to improve test time. (#8362)
Currently, our tests run with 2 GPUs, the init time is absurdly long:
about 4s for each process. Currently, we run each OP test on
different processes. This PR:
1. create cmake function py_test_modules which will generate the
Makefile that runs a list of Python unittest module in a single Python
process.
2. move all "python unittest compatible" (e.g., used the unittest
package, not just a regular python file). from fluid/tests to
fluid/tests/unittests.
3. cmake now will run all OP tests in fluid/tests/unittests in a
single process, except the time-consuming tests, they are separated
into different processes to utilize parallelism. Please make sure to
use the unittest package if you put the python test file in
fluid/tests/unittests
4. remove all exit(0) from fluid/tests/unittests/*.py, exit(0) is used
to disable unittest, we can not do it when running all tests in a
single process since it will terminate the process without running the
other tests. Instead, the test is disabled in
fluid/tests/unittests/CMakeLists.txt. FIXME is added for each disabled
item. Please disable the unittest from
fluid/tests/unittests/CMakeLists.txt, instead of adding exit(0) to the
Python file, for all Python file in fluid/tests/unittests/.
5. add an option WITH_FAST_BUNDLE_TEST. When OFF, will run the unit
tests in separate process so that they can be tested individually.
7 years ago
endif ( NOT WITH_DISTRIBUTE )
if ( NOT WITH_GPU OR WIN32 )
LIST ( REMOVE_ITEM TEST_OPS test_allgather )
LIST ( REMOVE_ITEM TEST_OPS test_allreduce )
LIST ( REMOVE_ITEM TEST_OPS test_broadcast )
LIST ( REMOVE_ITEM TEST_OPS test_reducescatter )
endif ( )
LIST ( REMOVE_ITEM TEST_OPS test_launch )
if ( NOT ${ WITH_GPU } )
LIST ( REMOVE_ITEM TEST_OPS test_conv2d_fusion_op )
LIST ( REMOVE_ITEM TEST_OPS test_parallel_dygraph_mnist ) # TODO(Yancey1989): parallel dygraph support CPU device in future
elseif ( ${ CUDNN_VERSION } VERSION_LESS 7100 )
LIST ( REMOVE_ITEM TEST_OPS test_conv2d_fusion_op )
endif ( )
if ( NOT WITH_GPU OR WIN32 )
LIST ( REMOVE_ITEM TEST_OPS test_pipeline )
endif ( )
Run Python OP tests in a single Python process to improve test time. (#8362)
Currently, our tests run with 2 GPUs, the init time is absurdly long:
about 4s for each process. Currently, we run each OP test on
different processes. This PR:
1. create cmake function py_test_modules which will generate the
Makefile that runs a list of Python unittest module in a single Python
process.
2. move all "python unittest compatible" (e.g., used the unittest
package, not just a regular python file). from fluid/tests to
fluid/tests/unittests.
3. cmake now will run all OP tests in fluid/tests/unittests in a
single process, except the time-consuming tests, they are separated
into different processes to utilize parallelism. Please make sure to
use the unittest package if you put the python test file in
fluid/tests/unittests
4. remove all exit(0) from fluid/tests/unittests/*.py, exit(0) is used
to disable unittest, we can not do it when running all tests in a
single process since it will terminate the process without running the
other tests. Instead, the test is disabled in
fluid/tests/unittests/CMakeLists.txt. FIXME is added for each disabled
item. Please disable the unittest from
fluid/tests/unittests/CMakeLists.txt, instead of adding exit(0) to the
Python file, for all Python file in fluid/tests/unittests/.
5. add an option WITH_FAST_BUNDLE_TEST. When OFF, will run the unit
tests in separate process so that they can be tested individually.
7 years ago
list ( REMOVE_ITEM TEST_OPS test_seq_concat_op ) # FIXME(helin): https://github.com/PaddlePaddle/Paddle/issues/8290
list ( REMOVE_ITEM TEST_OPS test_modified_huber_loss_op ) # FIXME(qijun) https://github.com/PaddlePaddle/Paddle/issues/5184
Run Python OP tests in a single Python process to improve test time. (#8362)
Currently, our tests run with 2 GPUs, the init time is absurdly long:
about 4s for each process. Currently, we run each OP test on
different processes. This PR:
1. create cmake function py_test_modules which will generate the
Makefile that runs a list of Python unittest module in a single Python
process.
2. move all "python unittest compatible" (e.g., used the unittest
package, not just a regular python file). from fluid/tests to
fluid/tests/unittests.
3. cmake now will run all OP tests in fluid/tests/unittests in a
single process, except the time-consuming tests, they are separated
into different processes to utilize parallelism. Please make sure to
use the unittest package if you put the python test file in
fluid/tests/unittests
4. remove all exit(0) from fluid/tests/unittests/*.py, exit(0) is used
to disable unittest, we can not do it when running all tests in a
single process since it will terminate the process without running the
other tests. Instead, the test is disabled in
fluid/tests/unittests/CMakeLists.txt. FIXME is added for each disabled
item. Please disable the unittest from
fluid/tests/unittests/CMakeLists.txt, instead of adding exit(0) to the
Python file, for all Python file in fluid/tests/unittests/.
5. add an option WITH_FAST_BUNDLE_TEST. When OFF, will run the unit
tests in separate process so that they can be tested individually.
7 years ago
list ( REMOVE_ITEM TEST_OPS test_lstm_unit_op ) # # FIXME(qijun) https://github.com/PaddlePaddle/Paddle/issues/5185
list ( REMOVE_ITEM TEST_OPS test_cond_op ) # FIXME(qijun): https://github.com/PaddlePaddle/Paddle/issues/5101#issuecomment-339814957
Run Python OP tests in a single Python process to improve test time. (#8362)
Currently, our tests run with 2 GPUs, the init time is absurdly long:
about 4s for each process. Currently, we run each OP test on
different processes. This PR:
1. create cmake function py_test_modules which will generate the
Makefile that runs a list of Python unittest module in a single Python
process.
2. move all "python unittest compatible" (e.g., used the unittest
package, not just a regular python file). from fluid/tests to
fluid/tests/unittests.
3. cmake now will run all OP tests in fluid/tests/unittests in a
single process, except the time-consuming tests, they are separated
into different processes to utilize parallelism. Please make sure to
use the unittest package if you put the python test file in
fluid/tests/unittests
4. remove all exit(0) from fluid/tests/unittests/*.py, exit(0) is used
to disable unittest, we can not do it when running all tests in a
single process since it will terminate the process without running the
other tests. Instead, the test is disabled in
fluid/tests/unittests/CMakeLists.txt. FIXME is added for each disabled
item. Please disable the unittest from
fluid/tests/unittests/CMakeLists.txt, instead of adding exit(0) to the
Python file, for all Python file in fluid/tests/unittests/.
5. add an option WITH_FAST_BUNDLE_TEST. When OFF, will run the unit
tests in separate process so that they can be tested individually.
7 years ago
list ( REMOVE_ITEM TEST_OPS op_test ) # op_test is a helper python file, not a test
list ( REMOVE_ITEM TEST_OPS decorator_helper ) # decorator_helper is a helper python file, not a test
if ( APPLE )
if ( NOT WITH_DISTRIBUTE )
list ( REMOVE_ITEM TEST_OPS test_desc_clone )
list ( REMOVE_ITEM TEST_OPS test_program_code )
endif ( NOT WITH_DISTRIBUTE )
message ( WARNING "These tests has been disabled in OSX before being fixed:\n test_fuse_elewise_add_act_pass \n test_detection_map_op \n test_dist_se_resnext_*" )
# this op is not support on mac
list ( REMOVE_ITEM TEST_OPS test_fusion_seqexpand_concat_fc_op )
# TODO: add the unitest back when it fixed
list ( REMOVE_ITEM TEST_OPS test_detection_map_op )
list ( REMOVE_ITEM TEST_OPS test_dist_se_resnext_dgc )
list ( REMOVE_ITEM TEST_OPS test_dist_se_resnext_sync )
list ( REMOVE_ITEM TEST_OPS test_dist_se_resnext_async )
list ( REMOVE_ITEM TEST_OPS test_dist_se_resnext_sync_with_memopt )
# TODO(tangwei12): add the unitest back when it fixed
list ( REMOVE_ITEM TEST_OPS test_dist_word2vec )
list ( REMOVE_ITEM TEST_OPS test_fuse_elewise_add_act_pass )
endif ( )
if ( NOT WITH_MKLML )
# this op is not support on openblas
list ( REMOVE_ITEM TEST_OPS test_fusion_seqexpand_concat_fc_op )
endif ( )
Run Python OP tests in a single Python process to improve test time. (#8362)
Currently, our tests run with 2 GPUs, the init time is absurdly long:
about 4s for each process. Currently, we run each OP test on
different processes. This PR:
1. create cmake function py_test_modules which will generate the
Makefile that runs a list of Python unittest module in a single Python
process.
2. move all "python unittest compatible" (e.g., used the unittest
package, not just a regular python file). from fluid/tests to
fluid/tests/unittests.
3. cmake now will run all OP tests in fluid/tests/unittests in a
single process, except the time-consuming tests, they are separated
into different processes to utilize parallelism. Please make sure to
use the unittest package if you put the python test file in
fluid/tests/unittests
4. remove all exit(0) from fluid/tests/unittests/*.py, exit(0) is used
to disable unittest, we can not do it when running all tests in a
single process since it will terminate the process without running the
other tests. Instead, the test is disabled in
fluid/tests/unittests/CMakeLists.txt. FIXME is added for each disabled
item. Please disable the unittest from
fluid/tests/unittests/CMakeLists.txt, instead of adding exit(0) to the
Python file, for all Python file in fluid/tests/unittests/.
5. add an option WITH_FAST_BUNDLE_TEST. When OFF, will run the unit
tests in separate process so that they can be tested individually.
7 years ago
if ( NOT WITH_MKL )
list ( REMOVE_ITEM TEST_OPS test_var_conv_2d )
endif ( NOT WITH_MKL )
Extend Matmul to support matrix multiplication with multiple heads (#18570)
* extend matmul op to support multiple head multiplication
With the support of multiple head, the multiplication of two big matrixes is
split into multiplication of several (head_number) small matrixes. e.g. if
Mat A is [3, 24] and Mat B is [24, 4], when multiple A and B with head_number
as 4, Mat A will be split as 4 matrix of [3, 6] and Mat B will be 4 matrix of
[6, 4]. The result of final matrix will be 4 matrix of [3, 4], i.e. [3, 16].
6 years ago
if ( WITH_GPU OR NOT WITH_MKLML )
# matmul with multiple heads need MKL support
LIST ( REMOVE_ITEM TEST_OPS test_matmul_op_with_head )
endif ( )
Run Python OP tests in a single Python process to improve test time. (#8362)
Currently, our tests run with 2 GPUs, the init time is absurdly long:
about 4s for each process. Currently, we run each OP test on
different processes. This PR:
1. create cmake function py_test_modules which will generate the
Makefile that runs a list of Python unittest module in a single Python
process.
2. move all "python unittest compatible" (e.g., used the unittest
package, not just a regular python file). from fluid/tests to
fluid/tests/unittests.
3. cmake now will run all OP tests in fluid/tests/unittests in a
single process, except the time-consuming tests, they are separated
into different processes to utilize parallelism. Please make sure to
use the unittest package if you put the python test file in
fluid/tests/unittests
4. remove all exit(0) from fluid/tests/unittests/*.py, exit(0) is used
to disable unittest, we can not do it when running all tests in a
single process since it will terminate the process without running the
other tests. Instead, the test is disabled in
fluid/tests/unittests/CMakeLists.txt. FIXME is added for each disabled
item. Please disable the unittest from
fluid/tests/unittests/CMakeLists.txt, instead of adding exit(0) to the
Python file, for all Python file in fluid/tests/unittests/.
5. add an option WITH_FAST_BUNDLE_TEST. When OFF, will run the unit
tests in separate process so that they can be tested individually.
7 years ago
function ( py_test_modules TARGET_NAME )
if ( WITH_TESTING )
set ( options SERIAL )
Run Python OP tests in a single Python process to improve test time. (#8362)
Currently, our tests run with 2 GPUs, the init time is absurdly long:
about 4s for each process. Currently, we run each OP test on
different processes. This PR:
1. create cmake function py_test_modules which will generate the
Makefile that runs a list of Python unittest module in a single Python
process.
2. move all "python unittest compatible" (e.g., used the unittest
package, not just a regular python file). from fluid/tests to
fluid/tests/unittests.
3. cmake now will run all OP tests in fluid/tests/unittests in a
single process, except the time-consuming tests, they are separated
into different processes to utilize parallelism. Please make sure to
use the unittest package if you put the python test file in
fluid/tests/unittests
4. remove all exit(0) from fluid/tests/unittests/*.py, exit(0) is used
to disable unittest, we can not do it when running all tests in a
single process since it will terminate the process without running the
other tests. Instead, the test is disabled in
fluid/tests/unittests/CMakeLists.txt. FIXME is added for each disabled
item. Please disable the unittest from
fluid/tests/unittests/CMakeLists.txt, instead of adding exit(0) to the
Python file, for all Python file in fluid/tests/unittests/.
5. add an option WITH_FAST_BUNDLE_TEST. When OFF, will run the unit
tests in separate process so that they can be tested individually.
7 years ago
set ( oneValueArgs "" )
set ( multiValueArgs MODULES DEPS ENVS )
Run Python OP tests in a single Python process to improve test time. (#8362)
Currently, our tests run with 2 GPUs, the init time is absurdly long:
about 4s for each process. Currently, we run each OP test on
different processes. This PR:
1. create cmake function py_test_modules which will generate the
Makefile that runs a list of Python unittest module in a single Python
process.
2. move all "python unittest compatible" (e.g., used the unittest
package, not just a regular python file). from fluid/tests to
fluid/tests/unittests.
3. cmake now will run all OP tests in fluid/tests/unittests in a
single process, except the time-consuming tests, they are separated
into different processes to utilize parallelism. Please make sure to
use the unittest package if you put the python test file in
fluid/tests/unittests
4. remove all exit(0) from fluid/tests/unittests/*.py, exit(0) is used
to disable unittest, we can not do it when running all tests in a
single process since it will terminate the process without running the
other tests. Instead, the test is disabled in
fluid/tests/unittests/CMakeLists.txt. FIXME is added for each disabled
item. Please disable the unittest from
fluid/tests/unittests/CMakeLists.txt, instead of adding exit(0) to the
Python file, for all Python file in fluid/tests/unittests/.
5. add an option WITH_FAST_BUNDLE_TEST. When OFF, will run the unit
tests in separate process so that they can be tested individually.
7 years ago
cmake_parse_arguments ( py_test_modules "${options}" "${oneValueArgs}" "${multiValueArgs}" ${ ARGN } )
if ( WITH_COVERAGE )
add_test ( NAME ${ TARGET_NAME }
C O M M A N D $ { C M A K E _ C O M M A N D } - E e n v P Y T H O N P A T H = $ { P A D D L E _ B I N A R Y _ D I R } / p y t h o n $ { p y _ t e s t _ m o d u l e s _ E N V S }
C O V E R A G E _ F I L E = $ { P A D D L E _ B I N A R Y _ D I R } / p y t h o n - c o v e r a g e . d a t a
$ { P Y T H O N _ E X E C U T A B L E } - m c o v e r a g e r u n - - b r a n c h - p $ { P A D D L E _ S O U R C E _ D I R } / t o o l s / t e s t _ r u n n e r . p y $ { p y _ t e s t _ m o d u l e s _ M O D U L E S }
W O R K I N G _ D I R E C T O R Y $ { C M A K E _ C U R R E N T _ B I N A R Y _ D I R } )
else ( )
add_test ( NAME ${ TARGET_NAME }
C O M M A N D $ { C M A K E _ C O M M A N D } - E e n v P Y T H O N P A T H = $ { P A D D L E _ B I N A R Y _ D I R } / p y t h o n $ { p y _ t e s t _ m o d u l e s _ E N V S }
$ { P Y T H O N _ E X E C U T A B L E } $ { P A D D L E _ S O U R C E _ D I R } / t o o l s / t e s t _ r u n n e r . p y $ { p y _ t e s t _ m o d u l e s _ M O D U L E S }
W O R K I N G _ D I R E C T O R Y $ { C M A K E _ C U R R E N T _ B I N A R Y _ D I R } )
endif ( )
if ( py_test_modules_SERIAL )
set_property ( TEST ${ TARGET_NAME } PROPERTY RUN_SERIAL 1 )
endif ( )
set_tests_properties ( ${ TARGET_NAME } PROPERTIES TIMEOUT 350 )
Run Python OP tests in a single Python process to improve test time. (#8362)
Currently, our tests run with 2 GPUs, the init time is absurdly long:
about 4s for each process. Currently, we run each OP test on
different processes. This PR:
1. create cmake function py_test_modules which will generate the
Makefile that runs a list of Python unittest module in a single Python
process.
2. move all "python unittest compatible" (e.g., used the unittest
package, not just a regular python file). from fluid/tests to
fluid/tests/unittests.
3. cmake now will run all OP tests in fluid/tests/unittests in a
single process, except the time-consuming tests, they are separated
into different processes to utilize parallelism. Please make sure to
use the unittest package if you put the python test file in
fluid/tests/unittests
4. remove all exit(0) from fluid/tests/unittests/*.py, exit(0) is used
to disable unittest, we can not do it when running all tests in a
single process since it will terminate the process without running the
other tests. Instead, the test is disabled in
fluid/tests/unittests/CMakeLists.txt. FIXME is added for each disabled
item. Please disable the unittest from
fluid/tests/unittests/CMakeLists.txt, instead of adding exit(0) to the
Python file, for all Python file in fluid/tests/unittests/.
5. add an option WITH_FAST_BUNDLE_TEST. When OFF, will run the unit
tests in separate process so that they can be tested individually.
7 years ago
endif ( )
endfunction ( )
function ( bash_test_modules TARGET_NAME )
if ( NOT WITH_TESTING )
return ( )
endif ( )
set ( options SERIAL )
set ( oneValueArgs "" )
set ( multiValueArgs MODULES DEPS ENVS )
cmake_parse_arguments ( bash_test_modules "${options}" "${oneValueArgs}" "${multiValueArgs}" ${ ARGN } )
message ( STATUS "CMAKE_CURRENT_BINARY_DIR:" ${ CMAKE_CURRENT_BINARY_DIR } )
add_test ( NAME ${ TARGET_NAME }
C O M M A N D $ { C M A K E _ C O M M A N D } - E e n v P Y T H O N P A T H = $ { P A D D L E _ B I N A R Y _ D I R } / p y t h o n $ { b a s h _ t e s t _ m o d u l e s _ E N V S }
b a s h $ { C M A K E _ C U R R E N T _ B I N A R Y _ D I R } / $ { b a s h _ t e s t _ m o d u l e s _ M O D U L E S }
W O R K I N G _ D I R E C T O R Y $ { C M A K E _ C U R R E N T _ B I N A R Y _ D I R } )
if ( bash_test_modules_SERIAL )
set_property ( TEST ${ TARGET_NAME } PROPERTY RUN_SERIAL 1 )
endif ( )
set_tests_properties ( ${ TARGET_NAME } PROPERTIES TIMEOUT 600 )
endfunction ( )
Run Python OP tests in a single Python process to improve test time. (#8362)
Currently, our tests run with 2 GPUs, the init time is absurdly long:
about 4s for each process. Currently, we run each OP test on
different processes. This PR:
1. create cmake function py_test_modules which will generate the
Makefile that runs a list of Python unittest module in a single Python
process.
2. move all "python unittest compatible" (e.g., used the unittest
package, not just a regular python file). from fluid/tests to
fluid/tests/unittests.
3. cmake now will run all OP tests in fluid/tests/unittests in a
single process, except the time-consuming tests, they are separated
into different processes to utilize parallelism. Please make sure to
use the unittest package if you put the python test file in
fluid/tests/unittests
4. remove all exit(0) from fluid/tests/unittests/*.py, exit(0) is used
to disable unittest, we can not do it when running all tests in a
single process since it will terminate the process without running the
other tests. Instead, the test is disabled in
fluid/tests/unittests/CMakeLists.txt. FIXME is added for each disabled
item. Please disable the unittest from
fluid/tests/unittests/CMakeLists.txt, instead of adding exit(0) to the
Python file, for all Python file in fluid/tests/unittests/.
5. add an option WITH_FAST_BUNDLE_TEST. When OFF, will run the unit
tests in separate process so that they can be tested individually.
7 years ago
list ( REMOVE_ITEM TEST_OPS test_warpctc_op )
list ( REMOVE_ITEM TEST_OPS test_dist_train )
list ( REMOVE_ITEM TEST_OPS test_dist_transpiler )
list ( REMOVE_ITEM TEST_OPS test_parallel_executor_crf )
list ( REMOVE_ITEM TEST_OPS test_parallel_executor_crf_auto_growth )
list ( REMOVE_ITEM TEST_OPS test_parallel_executor_fetch_feed )
list ( REMOVE_ITEM TEST_OPS test_dist_se_resnext_dgc )
list ( REMOVE_ITEM TEST_OPS test_dist_se_resnext_sync )
list ( REMOVE_ITEM TEST_OPS test_dist_se_resnext_async )
list ( REMOVE_ITEM TEST_OPS test_dist_se_resnext_sync_with_memopt )
list ( REMOVE_ITEM TEST_OPS test_dgc_op )
list ( REMOVE_ITEM TEST_OPS test_dist_se_resnext_nccl )
list ( REMOVE_ITEM TEST_OPS test_dist_transformer )
list ( REMOVE_ITEM TEST_OPS test_parallel_executor_transformer )
list ( REMOVE_ITEM TEST_OPS test_parallel_executor_transformer_auto_growth )
list ( REMOVE_ITEM TEST_OPS test_bilinear_interp_op )
list ( REMOVE_ITEM TEST_OPS test_nearest_interp_op )
list ( REMOVE_ITEM TEST_OPS test_imperative_resnet )
list ( REMOVE_ITEM TEST_OPS test_imperative_resnet_sorted_gradient )
list ( REMOVE_ITEM TEST_OPS test_imperative_mnist_sorted_gradient )
list ( REMOVE_ITEM TEST_OPS test_imperative_se_resnext )
list ( REMOVE_ITEM TEST_OPS test_imperative_mnist )
list ( REMOVE_ITEM TEST_OPS test_ir_memory_optimize_transformer )
list ( REMOVE_ITEM TEST_OPS test_layers )
list ( REMOVE_ITEM TEST_OPS test_imperative_ocr_attention_model )
list ( REMOVE_ITEM TEST_OPS test_async_ssa_graph_executor_mnist )
list ( REMOVE_ITEM TEST_OPS test_install_check )
list ( REMOVE_ITEM TEST_OPS test_basic_gru_api )
list ( REMOVE_ITEM TEST_OPS test_basic_gru_unit_op )
list ( REMOVE_ITEM TEST_OPS test_basic_lstm_api )
list ( REMOVE_ITEM TEST_OPS test_basic_lstm_unit_op )
# Some ops need to check results when gc is enabled
# Currently, only ops that register NoNeedBufferVarsInference need to do this test
set ( TEST_OPS_WITH_GC
t e s t _ a f f i n e _ c h a n n e l _ o p
t e s t _ c o n c a t _ o p
t e s t _ e l e m e n t w i s e _ a d d _ o p
t e s t _ e l e m e n t w i s e _ s u b _ o p
t e s t _ f i l l _ c o n s t a n t _ b a t c h _ s i z e _ l i k e _ o p
t e s t _ f i l l _ z e r o s _ l i k e 2 _ o p
t e s t _ g a t h e r _ o p
t e s t _ g a u s s i a n _ r a n d o m _ b a t c h _ s i z e _ l i k e _ o p
t e s t _ l i n e a r _ c h a i n _ c r f _ o p
t e s t _ l o d _ r e s e t _ o p
t e s t _ l o o k u p _ t a b l e _ o p
t e s t _ m e a n _ o p
t e s t _ p a d 2 d _ o p
t e s t _ s c a t t e r _ o p
t e s t _ s e q u e n c e _ c o n c a t
t e s t _ s e q _ c o n v
t e s t _ s e q _ p o o l
t e s t _ s e q u e n c e _ e x p a n d _ a s
t e s t _ s e q u e n c e _ e x p a n d
t e s t _ s e q u e n c e _ p a d _ o p
t e s t _ s e q u e n c e _ u n p a d _ o p
t e s t _ s e q u e n c e _ s c a t t e r _ o p
t e s t _ s e q u e n c e _ s l i c e _ o p
t e s t _ s l i c e _ o p
t e s t _ s p a c e _ t o _ d e p t h _ o p
t e s t _ s q u a r e d _ l 2 _ d i s t a n c e _ o p
t e s t _ u n i f o r m _ r a n d o m _ b a t c h _ s i z e _ l i k e _ o p )
foreach ( TEST_OP ${ TEST_OPS_WITH_GC } )
list ( REMOVE_ITEM TEST_OPS ${ TEST_OP } )
py_test_modules ( ${ TEST_OP } MODULES ${ TEST_OP } ENVS ${ GC_ENVS } )
endforeach ( )
foreach ( TEST_OP ${ TEST_OPS } )
py_test_modules ( ${ TEST_OP } MODULES ${ TEST_OP } )
endforeach ( TEST_OP )
py_test_modules ( test_adam_op_multi_thread MODULES test_adam_op ENVS FLAGS_inner_op_parallelism=4 )
py_test_modules ( test_warpctc_op MODULES test_warpctc_op )
py_test_modules ( test_bilinear_interp_op MODULES test_bilinear_interp_op ENVS ${ GC_ENVS } )
py_test_modules ( test_nearest_interp_op MODULES test_nearest_interp_op ENVS ${ GC_ENVS } )
py_test_modules ( test_imperative_resnet MODULES test_imperative_resnet ENVS
F L A G S _ c u d n n _ d e t e r m i n i s t i c = 1 S E R I A L )
set_tests_properties ( test_imperative_resnet PROPERTIES LABELS "RUN_TYPE=EXCLUSIVE" )
py_test_modules ( test_imperative_resnet_sorted_gradient MODULES test_imperative_resnet_sorted_gradient ENVS
F L A G S _ c u d n n _ d e t e r m i n i s t i c = 1 S E R I A L )
set_tests_properties ( test_imperative_resnet_sorted_gradient PROPERTIES LABELS "RUN_TYPE=EXCLUSIVE" )
py_test_modules ( test_imperative_mnist MODULES test_imperative_mnist ENVS
F L A G S _ c u d n n _ d e t e r m i n i s t i c = 1 )
py_test_modules ( test_imperative_mnist_sorted_gradient MODULES test_imperative_mnist_sorted_gradient ENVS
F L A G S _ c u d n n _ d e t e r m i n i s t i c = 1 )
py_test_modules ( test_imperative_se_resnext MODULES test_imperative_se_resnext ENVS
F L A G S _ c u d n n _ d e t e r m i n i s t i c = 1 S E R I A L )
set_tests_properties ( test_imperative_se_resnext PROPERTIES LABELS "RUN_TYPE=EXCLUSIVE" )
py_test_modules ( test_imperative_ocr_attention_model MODULES test_imperative_ocr_attention_model ENVS
F L A G S _ c u d n n _ d e t e r m i n i s t i c = 1 S E R I A L )
py_test_modules ( test_install_check MODULES test_install_check ENVS
F L A G S _ c u d n n _ d e t e r m i n i s t i c = 1 S E R I A L )
set_tests_properties ( test_install_check PROPERTIES LABELS "RUN_TYPE=DIST" )
if ( WITH_DISTRIBUTE )
py_test_modules ( test_dist_train MODULES test_dist_train )
set_tests_properties ( test_listen_and_serv_op PROPERTIES TIMEOUT 20 LABELS "RUN_TYPE=EXCLUSIVE" )
if ( WITH_DGC )
py_test_modules ( test_dgc_op MODULES test_dgc_op )
endif ( )
if ( NOT APPLE )
set_tests_properties ( test_dist_mnist PROPERTIES TIMEOUT 350 LABELS "RUN_TYPE=EXCLUSIVE" )
set_tests_properties ( test_dist_mnist_dgc_nccl PROPERTIES TIMEOUT 350 LABELS "RUN_TYPE=EXCLUSIVE" )
set_tests_properties ( test_dist_mnist_hallreduce PROPERTIES TIMEOUT 350 LABELS "RUN_TYPE=EXCLUSIVE" )
set_tests_properties ( test_dist_mnist_multi_comm PROPERTIES TIMEOUT 350 LABELS "RUN_TYPE=EXCLUSIVE" )
set_tests_properties ( test_dist_mnist_ring_allreduce PROPERTIES TIMEOUT 350 LABELS "RUN_TYPE=EXCLUSIVE" )
set_tests_properties ( test_dist_mnist_backward_deps PROPERTIES TIMEOUT 350 LABELS "RUN_TYPE=EXCLUSIVE" )
set_tests_properties ( test_dist_mnist_fleetapi PROPERTIES TIMEOUT 350 LABELS "RUN_TYPE=EXCLUSIVE" )
set_tests_properties ( test_dist_mnist_lars PROPERTIES TIMEOUT 350 LABELS "RUN_TYPE=EXCLUSIVE" )
set_tests_properties ( test_dist_word2vec PROPERTIES TIMEOUT 350 LABELS "RUN_TYPE=EXCLUSIVE" )
set_tests_properties ( test_dist_simnet_bow PROPERTIES TIMEOUT 350 LABELS "RUN_TYPE=EXCLUSIVE" )
set_tests_properties ( test_dist_text_classification PROPERTIES TIMEOUT 350 LABELS "RUN_TYPE=EXCLUSIVE" )
list ( REMOVE_ITEM TEST_OPS test_dist_se_resnext_dgc )
list ( REMOVE_ITEM TEST_OPS test_dist_se_resnext_sync )
list ( REMOVE_ITEM TEST_OPS test_dist_se_resnext_async )
list ( REMOVE_ITEM TEST_OPS test_dist_se_resnext_sync_with_memopt )
py_test_modules ( test_dist_se_resnext_dgc MODULES test_dist_se_resnext_dgc )
py_test_modules ( test_dist_se_resnext_sync MODULES test_dist_se_resnext_sync )
py_test_modules ( test_dist_se_resnext_nccl MODULES test_dist_se_resnext_nccl )
bash_test_modules ( test_launch MODULES test_launch.sh )
# FIXME(typhoonzero): add these tests back
# py_test_modules(test_dist_transformer MODULES test_dist_transformer)
# set_tests_properties(test_dist_transformer PROPERTIES TIMEOUT 1000)
set_tests_properties ( test_dist_se_resnext_dgc PROPERTIES LABELS "RUN_TYPE=EXCLUSIVE" )
set_tests_properties ( test_dist_se_resnext_sync PROPERTIES LABELS "RUN_TYPE=EXCLUSIVE" )
set_tests_properties ( test_dist_se_resnext_nccl PROPERTIES LABELS "RUN_TYPE=EXCLUSIVE" )
endif ( NOT APPLE )
# py_test_modules(test_dist_transpiler MODULES test_dist_transpiler)
endif ( )
py_test_modules ( test_parallel_executor_crf MODULES test_parallel_executor_crf )
py_test_modules ( test_parallel_executor_crf_auto_growth MODULES test_parallel_executor_crf_auto_growth ENVS FLAGS_allocator_strategy=auto_growth )
py_test_modules ( test_parallel_executor_fetch_feed MODULES test_parallel_executor_fetch_feed )
set_tests_properties ( test_parallel_executor_fetch_feed PROPERTIES TIMEOUT 450 )
set_tests_properties ( test_parallel_executor_seresnext PROPERTIES TIMEOUT 740 )
py_test_modules ( test_parallel_executor_transformer MODULES test_parallel_executor_transformer )
py_test_modules ( test_parallel_executor_transformer_auto_growth MODULES test_parallel_executor_transformer_auto_growth ENVS FLAGS_allocator_strategy=auto_growth )
py_test_modules ( test_layers MODULES test_layers ENVS FLAGS_cudnn_deterministic=1 )
if ( NOT WIN32 )
py_test_modules ( test_ir_memory_optimize_transformer MODULES test_ir_memory_optimize_transformer )
endif ( )
if ( CMAKE_BUILD_TYPE STREQUAL "Debug" )
# change the timeout from 600 to 2200, because in debug mode, this test need more time.
set_tests_properties ( test_parallel_executor_seresnext PROPERTIES TIMEOUT 2200 )
endif ( )
if ( WITH_NGRAPH )
add_subdirectory ( ngraph )
endif ( )
if ( WITH_MKLDNN )
add_subdirectory ( mkldnn )
endif ( )
if ( WITH_DISTRIBUTE )
set_tests_properties ( test_listen_and_serv_op test_nce_remote_table_op test_hsigmoid_remote_table_op
P R O P E R T I E S L A B E L S " R U N _ T Y P E = D I S T " )
endif ( )
set_tests_properties ( test_recordio_reader test_parallel_executor_test_while_train test_parallel_executor_mnist
t e s t _ p a r a l l e l _ e x e c u t o r _ s e r e s n e x t t e s t _ p a r a l l e l _ e x e c u t o r _ c r f t e s t _ s y n c _ b a t c h _ n o r m _ o p
t e s t _ p a r a l l e l _ e x e c u t o r _ c r f _ a u t o _ g r o w t h t e s t _ b u f f e r _ s h a r e d _ m e m o r y _ r e u s e _ p a s s _ a n d _ f u s e _ o p t i m i z a t i o n _ o p _ p a s s
Feature/mem opt pass refactor (#18735)
* first version memory optimize pass, test=develop
* remove move_tensor_sharing_pass, test=develop
* refine code comments, add unittests, test=develop
* turn off memory_optimize by default, test=develop
* follow huihuang's comments, test=develop
* follow chengduoZH's comments, test=develop
* fix grammar error, add const qualifier, fix pass_test exception message, test=develop
* follow chengduoZH's comments 2nd, test=develop
6 years ago
t e s t _ b u f f e r _ s h a r e d _ m e m o r y _ r e u s e _ p a s s P R O P E R T I E S L A B E L S " R U N _ T Y P E = D I S T " )
