**Problem**
In our old shape transformer logic, if user write:
```
s = tensor.shape
...
y = paddle.some_api(s)
```
Dy2stat will change it to
```
...
y = paddle.some_api(convert_var_shape(tensor))
```
However it will cause fatal bug if user changes the shape of `x` after assign. For example:
```
s = tensor.shape
...
tensor = paddle.some_change_shape_api(tensor)
...
y = paddle.some_api(s)
```
Then the Dy2stat will get wrong result because the code is translated into:
```
tensor = paddle.some_change_shape_api(tensor)
...
y = paddle.some_api(convert_var_shape(tensor)) # tensor shape has been changed, not origin `s` value
```
**Solution Logic**
It can not be solved in the old logic, so I refactoring tensor_shape_transformer logic. Now we will use `s` to store shape attribute and generate a var `s__STATIC_CONVERT_VAR_SHAPE_SUFFIX` to store static shape API `shape(tensor)`
```
s = tensor.shape
...
y = paddle.some_api(s)
```
Dy2stat will change it to
```
s = tensor.shape
s__STATIC_CONVERT_VAR_SHAPE_SUFFIX = shape(tensor)
...
y = paddle.some_api(choose_shape_attr_or_api(s, s__STATIC_CONVERT_VAR_SHAPE_SUFFIX ))
```
In this case, the code is consistent with origin dygraph meaning and it fixed the change after assign bug.
**Code Key Note**
To help reviewers, the key change of this PR is changing `self.name_to_var_shape` from "mapping name to shape node" to "mapping name to its STATIC_CONVERT_VAR_SHAPE_SUFFIX name", then if a variable name has the SUFFIX, we can choose to use attribute shape or shape api. Other changes go with the key change.
**Consideration**
The issue of this PR is that we store extra static `shape` API result, will it harms the speed of Dy2stat? In some cases it will, but we argue that the benefit would be greater than the cost.
1. The extra calling to static `shape` API will happen when coder assign among shape variables. Take the following dygraph code as an instance:
```
s1 = tensor.shape
s2 = s1
s3 = s2
...
```
Then we called extra static `shape` APIs again and again, however users seldom write code like this.
2. If the shape variable is used a lot, for example:
```
s = tensor.shape
y1 = paddle.some_api1(s)
y2 = paddle.some_api2(s)
y3 = paddle.some_api3(s)
```
Our old logic will create 3 shape APIs but now just 1. This is more common user code pattern. In fact, if reviewers take a look at the current unit test in this PR, you could see the op numbers decrease after this PR. So we argue that this PR can also improve speed in this code pattern.
* add more dispatch marco
* add more dispatch marco
* add more tests
* revert unneeded change
* add timeout for test dispatch
* add float and complex test
* remove and marco
* [static setitem] support the index step > 1. tensor_a[::3] = value
* [static setitem] support the index step < 0. Eg: tensor_a[::-3] = value
* [static setitem] support the index is Tensor. eg: tensor_a[tensor_3:0:-1] = value
* Add op version.
As the title, when slice_node like 1:3 being passed to idx of convert_var_shape, it will cause syntax error because a function cannot take this as argument. This PR fixed it.
* add more unitest for ABI compatibility
* add more unittest
* refine warning style
* support compile multi custom ops in same time
* fix not import paddle in unittest
* fix typo
* add more unittest
* add comment for details
* Add conv transpose BF16
* Share function GetWeightsTz
* Adjust to review and fix op compatibility
* Add bias to unique handler name
* Remove errors related to paddle enforce
* Add conv2d_transpose to bf16 list and kernel refator
Dy2stat didn't support tuple as iteration variable in the past. This PR added there main cases:
1). Non-enumerate case: for var1, var2 in var|var.numpy() will be re-written as:
for FOR_ITER_TUPLE_PREFIX_x in var | var.numpy():
var1 = FOR_ITER_TUPLE_PREFIX_x[0]
var2 = FOR_ITER_TUPLE_PREFIX_x[1]
2). Enumerate out tuple case: for t in enumerate(var|var.numpy) will be rewritten as:
for FOR_ITER_TUPLE_INDEX_PREFIX_x, FOR_ITER_TUPLE_PREFIX_x in enumerate(var|var.numpy):
t = (FOR_ITER_TUPLE_INDEX_PREFIX_x, FOR_ITER_TUPLE_PREFIX_x)
3). Enumerate inner tuple case: for i, (var1, (var2, va3)) in enumerate(var|var.numpy()) will
be re-written as:
for i, FOR_ITER_TUPLE_PREFIX_x in var | var.numpy():
var1 = FOR_ITER_TUPLE_PREFIX_x[0]
var2 = FOR_ITER_TUPLE_PREFIX_x[1][0]
var3 = FOR_ITER_TUPLE_PREFIX_x[1][1]
* support setup.py to compile custom op
* move file into paddle.utils.cpp_extension
* support python setup.py install
* refine code style
* Enrich code and add unittest
* initial commit: simple demo
* polish copyright format
* add grap op simple demo
* adapt uncertain number of argument
* change trait marco name
* add place & dtype support for add kernel
* add dispath and infershape func
* poish code & add notes
* add dynamic_loader dep for paddle_framework
* add new custom op test dir
* polish impl details
* add unittest for new custom op
* fix failed unittest
* Costum op (#1)
* fix compile error
* wrap framework tensor with LoDTensor
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* add CustomTensor default constructor
* add size() for CustomTensor
* make size const for CustomTensor
* refactor place related api to circle the concept
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* make place const
* make Tensor copy
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* remove additional head of framework
* use back to shared ptr for custom tensor
* use back to shared ptr for custom tensor
* use back to shared ptr for custom tensor
* use back to shared ptr for custom tensor
* use back to shared ptr for custom tensor
* use back to shared ptr for custom tensor
* add gpu test
* merge latest cwh code in
* adjust ut code of custom op
* adjust ut code of custom op
* adjust ut code of custom op
* Remove ShareData from user && Change CustomTensor to Tensor && Support more data type (#2)
* fix compile error
* wrap framework tensor with LoDTensor
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* add CustomTensor default constructor
* add size() for CustomTensor
* make size const for CustomTensor
* refactor place related api to circle the concept
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* make place const
* make Tensor copy
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* remove additional head of framework
* use back to shared ptr for custom tensor
* use back to shared ptr for custom tensor
* use back to shared ptr for custom tensor
* use back to shared ptr for custom tensor
* use back to shared ptr for custom tensor
* use back to shared ptr for custom tensor
* add gpu test
* merge latest cwh code in
* adjust ut code of custom op
* adjust ut code of custom op
* adjust ut code of custom op
* adjust ut code of custom op
* adjust ut code of custom op
* hid share data from and to
* rename CustomTensor to Tensor
* refactor register design & add test
* change op_funtion to op_meta_info
* split op meta info into .h and .cc
* move get methods into friend class
* move OpMetaInfoHelper into framework space
* move CustomTensorUtils into framework space
* change pybind api name
* move PD C API into op meta info
* add register custom op api
* remove inference cmake change
* refactor copy to api && change Reshape to lowercase && support more dtype && add more test (#3)
* fix compile error
* wrap framework tensor with LoDTensor
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* add CustomTensor default constructor
* add size() for CustomTensor
* make size const for CustomTensor
* refactor place related api to circle the concept
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* make place const
* make Tensor copy
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* remove additional head of framework
* use back to shared ptr for custom tensor
* use back to shared ptr for custom tensor
* use back to shared ptr for custom tensor
* use back to shared ptr for custom tensor
* use back to shared ptr for custom tensor
* use back to shared ptr for custom tensor
* add gpu test
* merge latest cwh code in
* adjust ut code of custom op
* adjust ut code of custom op
* adjust ut code of custom op
* adjust ut code of custom op
* adjust ut code of custom op
* hid share data from and to
* rename CustomTensor to Tensor
* support multi dtype
* remove lod, make reshape lowercase, add copy test and refactor copy api
* remove lod, make reshape lowercase, add copy test and refactor copy api
* remove lod, make reshape lowercase, add copy test and refactor copy api
* remove lod, make reshape lowercase, add copy test and refactor copy api
* fix copy to error
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* polish detail & error message
* polish test details
* Add cast api && Change copy related api to copy_to && add more test (#4)
* fix compile error
* wrap framework tensor with LoDTensor
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* add CustomTensor default constructor
* add size() for CustomTensor
* make size const for CustomTensor
* refactor place related api to circle the concept
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* fix compile error
* make place const
* make Tensor copy
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* debug CustomTensor core
* remove additional head of framework
* use back to shared ptr for custom tensor
* use back to shared ptr for custom tensor
* use back to shared ptr for custom tensor
* use back to shared ptr for custom tensor
* use back to shared ptr for custom tensor
* use back to shared ptr for custom tensor
* add gpu test
* merge latest cwh code in
* adjust ut code of custom op
* adjust ut code of custom op
* adjust ut code of custom op
* adjust ut code of custom op
* adjust ut code of custom op
* hid share data from and to
* rename CustomTensor to Tensor
* support multi dtype
* remove lod, make reshape lowercase, add copy test and refactor copy api
* remove lod, make reshape lowercase, add copy test and refactor copy api
* remove lod, make reshape lowercase, add copy test and refactor copy api
* remove lod, make reshape lowercase, add copy test and refactor copy api
* fix copy to error
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* add more test
* add type cast
* add cast and make copy to api
* add cast and make copy to api
* add cast and make copy to api
* add cast and make copy to api
* merge cwh code
* merge cwh code
* merge cwh code
* merge cwh code
* merge cwh code
* add more error log
* add more error log
* polish code
* used for test
* remove test comment
* remove test comment
* fix uint8 type error
* fix lost uint8 type error
* add test for coverage
* polish details by reviewer comments
* add prefix for DISABLE_COPY_AND_ASSIGN
Co-authored-by: Jiabin Yang <360788950@qq.com>
* support xpu inference with analysis predictor, test=develop
* merge the cmake of the xpu toolchain, test=develop
* add c-apis, test=develop
* fix a bug in extern_xpu, test=develop
* support setup.py to compile custom op
* move file into paddle.utils.cpp_extension
* support python setup.py install
* refine code style
* Enrich code and add unittest
* Polish code and api doc
* fix cpp_extension not include in package
* fix relative import
* fix os.makedirs exist_ok param compatibility PY2
* add compile flags in test_jit_load
* rewrite abs op
* rewrite abs op and remove abs in activation
* remove abs register in old codes
* fix abs_grad type error
* fix abs double_grad output name error
* modify abs_grad, abs_grad_grad functor for windows building
* format code style
* fix the bug of result is nan when the divisor is zero
* add missing abs attr and add abs for float16
* Avoid bug on 'MAC python3.5/6'.
* Choose the saving method according to the OS.
* smaller length of '_unpack_saved_dict' for MAC OS.
* add version information of Python.
* Edit comment.
* add view strategy on squeeze,unsqueeze,reshape,flatten
* add squeeze unittest
* add unittests
* use View strategy as name rather than Reuse Allacation
* fix view api doc
* fix format
* use core.ops when input of reshape2 is Tensor
* fix test_cross_entropy_loss error because of reshape2
* fix test_cross_entropy_loss error because of reshape2
* add inplace strategy
* add elementwise_add sub
* let backward op not use inplace
* grad op do not use inplace
* fix memory increase error and add leaf error message
* delete selected_rows
* change op_function
* little change
* solve HandleViewBetweenInputAndOutput
* add unittest and leaf error message
* merge view error
* optimize op_function_generator format and support sum inplace op
* fix format of basic_engine
* fix format for framework
* little change of variable wrapper
* add reshape, squeeze, unsqueeze, scatter api
* add relu elu tanh softmax inplace api
* fix test_squeeze_op unittest
* fix test_relu_op unittest
* fix comment problems
* delete sample code of inplace api
* add reference of grad_pending_nodes in basic_engine
* fix unittest name
* add inplace apis into wlist
* fix error message
* add PADDLE_ENFORCE for set grad op twice
* fix head file error
* upgrade oneDNN version to 2.0 master branch
* - Added workarounds for new lib onednn change
* fix regex
Co-authored-by: Jacek Czaja <jacek.czaja@intel.com>
* fix bug of using ignore_index and reduction,test=develop
* fix bug of celoss when using ignore_index and reduction, test=develop
* improve performance when ignore_index=-100, test=develop
* add test in test_cross_entropy_loss.py for coverage rate, test=develop
* rm comment in test_cross_entropy_loss.py, test=develop
* del hard code of "float64" in python/paddle/nn/functional/loss.py, test=develop
* change mask to a more simplified implementation, test=develop
* del comment in python/paddle/nn/functional/loss.py, test=develop
* del hard code and change mask to a more simplified implementation, test=develop
* change mask to a more simplified implementation, test=develop
* change mask to a more simplified implementation, test=develop
* add view strategy on squeeze,unsqueeze,reshape,flatten
* add squeeze unittest
* add unittests
* use View strategy as name rather than Reuse Allacation
* fix view api doc
* fix format
* use core.ops when input of reshape2 is Tensor
* fix test_cross_entropy_loss error because of reshape2
* delete selected_rows
* change op_function
* little change
* solve HandleViewBetweenInputAndOutput
1. When x is Variable, call nn.shape(x) only in following cases:
1）The shape of x is used in control flow condition.
2）The dim to be used is negetive
2. When x is Variable, but x.shape or x.shape[idx] doesn't contain negetive value, don't convert to paddle.shape()
* change to tensor copy sync
* change to tensor copy sync
* make copy_to safe when use TensorCopy
* refine code
* add ut
* add cudapinned garbagecollector
* add testcase: cpu place -> cuda pinned place
1. when slice_item is a slice:
1) the start of __getitem__ should be std::max(start, 0) if slice
2) the start of __getitem__ should be std::min(end, dim)
2. when slice_item is an integer, it should be in [-dim_len, dim_len)
3. Fix error message to use accurate data
* Support storage of large parameters
* Reduce the complexity of the unittest
* Reduce the complexity of the unittest,commented out unittest for
* add unittest for static.save/load
* Increase the timeout threshold of 'test_static_save_load'
* Increase the timeout threshold of 'test_static_save_load'
* Increase the timeout threshold of 'test_static_save_load' and 'test_paddle_save_load'
* Increase the timeout threshold of 'test_static_save_load' and 'test_paddle_save_load'
* dot op support complex types
* matmul support complex types
* add test case
* matmul broadcast gradient support complex
* move conjFunctor to complex_functor.h
Huihuang Zheng
tangwei12
Zhou Wei
Chen Weihang
TTerror
123malin
Shibo Tao
liym27
Jacek Czaja
Aurelius84
ShenLiang
Kaipeng Deng
Guanghua Yu
Pei Yang
joanna.wozna.intel
Wojciech Uss
WeiXin
liuyuhui
wanghuancoder
cucuzg
wawltor
JamesLim
AshburnLee
石晓伟
joejiong
Adam Osewski
WangXi
Shang Zhizhou
xiemoyuan
liu zhengxi
jakpiase
Tao Luo
Leo Chen
chentianyu03
Qi Li
Zhen Wang
huangxu96
lilong12
ykkk2333
Zhang Ting
hutuxian
gongweibao
QingshuChen
taixiurong
JZ-LIANG
lijianshe02
guofei
pangyoki
lidanqing
YUNSHEN XIE
chajchaj
Double_V
furnace
Bai Yifan
XiaoguangHu
wangchaochaohu
Chengmo
ceci3
Jiaqi Liu
cc
littletomatodonkey