Support Various-Length Return Grammar in Dy2stat. This PR is a follow-up of https://github.com/PaddlePaddle/Paddle/pull/25176 .
The basic idea is putting no-value placeholder variables at `return` statement to make all `return` statement have same length, after that the static graph can have fixed fetch output (code at return_transformer.py). Then remove those no-value placeholder when we finally return dygraph result (code at partial_program.py).
However, various length return in Bert model is still not supported. The dy2stat can change the code as I wish but some ops which check shape at compile time (e.g. Reshape, MatMul) will throw error because of the no-value-placeholder may not have the required shape. Is this a matter? To me, those no-value placeholder will be replaced as really values meeting shape requirements at run time, so I think the solution should be some way to do the compile-time checking. By the way, every time when we have dynamic shape, it often causes problem in dy2stat. We should find a way to handle it in the future.
Fixing various return in Bert is my TODO thing and I will also find some other existing models for verification.
This PR added basic support for 'return' grammar in dy2stat. It supports the control flow of 'return'.
The basics idea is using a return value variable to store the early return statements and boolean state variables with if-else to skip the statements after the return statements.
**This PR is very basic support. There are some corner cases I didn't develop/test**. For example, 'return None', 'return different length of variables', 'return non-tensor and tensor together', 'no return statement'. **These corner cases will be done in my next PRs**. Target date is this week.
**Note**:
1. for the unit test, I changed test_program_translator.py because the StaticCode of `dyfunc_with_if_else` will change. To guarantee the correctness of `dyfunc_with_if_else`, I also run it in `TestRecursiveReturn` in test_return.py.
2. I commented the early return code in bert_dygraph_model.py because 'return different length of variables' is unsupported now. I also know that there are some other models used early return and we didn't enable it in the unit test. I will add support for it in next PRs and then re-enable those tests.
* The arg of append() can be not Tensor temporarily.
* Add Seq2Seq as ProgramTranslator Unit Test.
* set dtype of vocab_size_tensor to int64 to pass Windows-CI.
* Support int and long: int or long -> six.integer_types.
* Modify test_tensor_shape: fix bug and modify comment.
* Support convert_var_shape to convert var.shape stmt
* Modify code in ifelse_simple_func.py because don't support return non-Tensor in Tensor-dependent 'if' stament currently.
* Convert the return variables of Tensor-dependent 'if' staments to Tensor if it not. test=develop
* Move function 'convert_len' to file convert_operators.py
* Support that for statements are transformed to while statements.
* Fix bug: raise None -> return None.
* Support variable loaded and created in loop.
* Use int64 in Py2 and Py3 in function to_static_variable.
* cast var in convert_logical_XX.
* Add convert_ifelse function in convert_operators.py
* Add logical_transformer. Remove LogicalTransformer from loop_transformer.py
* Revert modified tests in PR24799(convert_while_stmt).
* Comment and modify code that doesn't support `return` statement.
* Remove unnecessary class: MergeAssignTransformer, NodeTestTransformer and IfConditionVisitor in ifelse_transformer.
* Support return variable in only one of if body or else.
* remove after_visit in IfElseTransformer.
* Modify the result of get_name_ids in test_ifelse_basic.py
* Add unittest to test the new case.
* Modify code according to reviews.
* Support convert_while_loop.
* Comment code that not supported 'if' in test_break_continue.
* Convert int into tensor to support 'if' stmt in for/while loop.
* Add unittest to test all cases of convert_logical_XX.
* Add unittest to test all cases of convert_while_loop.
* Fix bug in LogicalOpTransformer. test=develop
* Support to create LoDTensorArray in control flow (cond and while_loop)
* Fix bug: return LoDTensorArray in while_loop
* Change code in list_transformer.py to accommodate the new features.
* fix numpy ndarray mul var base error; test=develop
* add comment for __array_ufunc__ ; test=develop
* move unitest from imperative math op path to test_math_op_patch_var_base;
test=develop
* support to train in static
* support to independent decorator
* remove in_dygraph_mode condition in ProgramTranslator
* fix import param_guard and add train/eval test=develop
* Modify into ShareVarsFromScope and rm __all__ in partial_program test=develop
* Replace dygraph_to_static_func with @declarative or program_translator.get_func in test_list.py
* Add comments in ConditionalBlock.
* Support list pop last item.
* Support pop the i-th item.
* Support an empty tensor array as Input in assign op and set the kernel type is float.
* Simplify code for gast.If in is_control_flow_to_transform.
* Move IsControlFlowVisitor to file utils.
* Don't use convert_call for build-in func in CallTransformer.
* Optimize api is_control_flow_to_transform.
* Polish the document of IsControlFlowVisitor.
To prepare for publishing APIs, I added tests for that we can access dy2stat through:
@fluid.dygraph.declarative
@fluid.dygraph.jit.declarative
fluid.dygraph.ProgramTranslator()
fluid.dygraph.dygraph_to_static.ProgramTranslator()
fluid.dygraph.dygraph_to_static.program_translator.ProgramTranslator()
It surprised me that we had bugs on those different usages. I have fixed them.
I also added example codes for these new APIs
This PR also pulls my current PR https://github.com/PaddlePaddle/Paddle/pull/23880, so the PR history is long. For reviewer information, you could review this PR after https://github.com/PaddlePaddle/Paddle/pull/23880 is merged
1. Rename Dygraph To Static Decorators to declarative
2. dygraph_to_static_func is still using in some training tests, I cannot delete it now.
3. Add some API docs
* add Conv/ConvTranspose layers in paddle.nn.layer, test=develop
* add example code in docstring, test=develop
* update unittests to work with test_runner, test=develop
* static model runner basic implement, test=develop
* add run program op to execute loaded program, test=develop
* refactor static model runner & run program op, test=develop
* reset engine.cc to resolve conflict
* adapt the change of dygraph double grad, test=develop
* refactor impl to solve control flow error, test=develop
* clear debug code, test=develop
* fix ci str compatible error & checkout dygraph grad maker & add example, test=develop
* hide api & add op test, test=develop
* fix run program op test places error, test=develop
* fix program by review comment, test=develop
* delete change var desc name, test=develop
* fix other program by review comment, test=develop
* remove _static_graph_guard, test=develop
* add selectedrows test, test=develop
* remove desc parser, test=develop
* fix detail program, test=develop
* change socpe create & add test, test=develop
This CR makes two changes:
1. In old loop_transformer, if a class variable, such as "self.a, foo.bar" is a loop var, the Dy2stat will fail because `def func(self.foo)` is not legal syntax. We support class variable by renaming.
2. After https://github.com/PaddlePaddle/Paddle/pull/22892 is merged, we can support `while x < 10` in dygraph. I enable those tests in corresponding Dy2stat
1. Add data_layer_not_check because it is needed in dygraph_to_static where input can be variable size
2. Remove warnings in static analysis because python cannot do exact static analysis
* Remove Read-Only Basic Type Variable in loop_vars
test=develop
* Put class support in loop for future thing
test=develop
* Modify based on reviewer's comment
test=develop
1. Add Decorator 'dygraph_to_static_program'
2. Add corresponding ProgramTranslator.get_program
3. Add ProgramTranslator.save_inference_model
4. Modified some warning information of dy2stat
5. Change program cache to contain startup_program because for users who gets program to run, they may like to initialize startup program
This PR does exact the thing as the title. The reason is that we plan to develop 4 decorators
"dygraph_to_static_code"
"dygraph_to_static_program"
"dygraph_to_static_func"
"dygraph_to_static_output"
The 4 decorators will emphasize different part when translating dygraph to static graph. Decorator name "dygraph_to_static_graph" is too big for the function it implements.
As the title, we add decorator "dygraph_to_static_code", and add related "get_code", "get_func", "get_output" for ProgramTranslator. Next step will be adding "dygraph_to_static_program"
1. Rename AutoTracer to ProgramTranslator
2. Rename cached_program to program_cache
3. Remove some functor style __call__
4. Dict key should be string but not hash code of string
1. Add support for Break and Continue in Dygraph to Static
2. Also add support for gast.Not in NodeTestTransformer
3. Also add support for logical op transformation in LoopTransformer
* add dygraph parallel env doc, test=develop
* polish details, test=develop, test=document_fix
* fix examples error in other apis, test=develop
* fix more example error in other api, test=develop
* add white list for gpu examples, test=develop, test=document_fix
1. Add basic support for `for in range` loop
2. Move `test_dygraph_to_static_*` to `dygraph_to_static` dir and rename them
3. Add test case for dict in while_loop
1. copy.deepcopy in NameVisitor should be changed to copy.copy to make hash or set work
2. read_context should be type of gast.Load()/gast.AugLoad(), not gast.Load/gast.AugLoad
Add basic support for while in translating dygraph to static
1. Analysis the variable liveness in class NameVisitor
2. Replace while key word using while_loop API
* Support fetch and run program in the process of dygraph_to_static_output. test=develop
* fix to_source(gast) and remove dygraph API such as Conv2D, Linear. test=develop
1. Considering functions, I have to change the node type from single value to a set. Because python function is allowed to return different types. The set represent all possible types
2. I added scope_name and scope_type for AstVarScope, because in python functions, variable may have different scope. For example:
```
a = 3
def foo(b):
a = 9
return a + b
```
the `a` in `foo` is different to the `a` out of `foo`. Similar to class field. The scope_name will help me to know the function name when static analysis finds a `return` sentence.
* support dygraph to static graph for simple case.
* add test for dygraph API recognition. test=develop
* support basic to_variable api. test=develop
* update dict: dygraph_class_to_static_api
* add all tests of dygraph api. test=develop
* use gast/astor instead of ast/codegen for the compatibility of PY2 and PY3. test=develop
* add arg 'num_flatten_dims' for fc ast node. test=develop
* Modify names of class by Camel-Case.
* support nested if/else
* support to derivate returns the parameter list automatically
* polish tranform function of slice
* fix modify x.numpy()[i] slice function
* support to transform ast.node into callable function
* fix get_name_ids bug and add more unittest test=develop
* fix requirements.txt test=develop
* remove useless import statement test=develop
* Fixed version compatibility issues in param of function test=develop
* use decorater to test ast_to_func test=develop
* add textwrap.dedent for source_code test=develop
* polish code comment
* fix compatibility with python2 and python3 test=develop
* fix gast version error test=develop
* fix gast repo test=develop
* polish transfer_from_node_type code test=develop
* add nested_if_else unittest test=develop
* split IfElseTransformer test=develop
* specify gast version test=develop
* fix ast_to_func root type test=develop
This PR provides very basic and simple framework for transforming Dygraph to Static Graph.
API names, final outputs are not determined yet. Feel free to modify or add class/function/type when you think the framework is not extendable for you.
* modify fc to linear in sample code, test=develop
* remove FC, test=develop
* remove warnings, test=develop
* drop fluid/imperative/README.md , test=develop
* change fc to linear, test=develop
* polish code style, test=develop
* dygraph mode support linear lr warm up; test=develop
* add unitest for linear warmup; test=develop
* add input type check; test=develop
* fix type check assert error; test=develop
* change type error; test=develop
* test=develop, fix docker with paddle nccl problem
* don't expose numerous Tensor.set(), test=develop
* fix condition, test=develop
* fix float16 bug, test=develop
* feed should be Tensor or np.array, not Variable or number, test=develop
* use forcecast to copy numpy slice to new array, test=develop
* remove float16-uint16 hacking, test=develop
* add variable method to varbase and refactor to_variable to support return varbase
* support kwargs in varbase constructor
* add VarBase constructor to support default python args
* refine varbase initial method
* reset branch
* fix ut for change VarBase error info to PaddleEnforce
* cherry is parameter change before
* overload isinstance to replace too many change of is_variable
* rm useless files
* rm useless code merged by git
* test=develop, fix some ut failed error
* test=develop, fix test_graph_wrapper
* add some tests, test=develop
* refine __getitem__, test=develop
* add tests, test=develop
* fix err_msg, test=develop
* add param & grad shape check for sgd op
* add _reshape_inplece interface for dygraph parallel
* refine unittest based paddle/models scripts, test=develop
* add unittest for parallel grad fuse, test=develop
Huihuang Zheng
hong
Aurelius84
Zhou Wei
Aurelius84
liym27
Leo Chen
Chen Weihang
Cindy Cai
Xing Wu
zhongpu
qingqing01
Yibing Liu
wawltor
songyouwei
silingtong123
Feiyu Chan
Zeng Jinle
tianshuo78520a
Wojciech Uss