|
|
|
|
@ -476,42 +476,110 @@ def embedding(input,
|
|
|
|
|
param_attr=None,
|
|
|
|
|
dtype='float32'):
|
|
|
|
|
"""
|
|
|
|
|
**Embedding Layer**
|
|
|
|
|
|
|
|
|
|
This layer is used to lookup embeddings of IDs, provided by :attr:`input`, in
|
|
|
|
|
a lookup table. The result of this lookup is the embedding of each ID in the
|
|
|
|
|
:attr:`input`.
|
|
|
|
|
**WARING:** This OP will be deprecated in a future release. This OP requires the
|
|
|
|
|
last dimension of Tensor shape must be equal to 1. It is recommended to use
|
|
|
|
|
fluid. :ref:`api_fluid_embedding` .
|
|
|
|
|
|
|
|
|
|
All the input variables are passed in as local variables to the LayerHelper
|
|
|
|
|
constructor.
|
|
|
|
|
|
|
|
|
|
Args:
|
|
|
|
|
input(Variable): Input is a Tensor<int64> Variable, which contains the IDs information.
|
|
|
|
|
The value of the input IDs should satisfy :math:`0<= id < size[0]`.
|
|
|
|
|
size(tuple|list): The shape of the look up table parameter. It should
|
|
|
|
|
have two elements which indicate the size of the dictionary of
|
|
|
|
|
embeddings and the size of each embedding vector respectively.
|
|
|
|
|
is_sparse(bool): The flag indicating whether to use sparse update.
|
|
|
|
|
is_distributed(bool): Whether to run lookup table from remote parameter server.
|
|
|
|
|
padding_idx(int|long|None): It will output all-zero padding data whenever
|
|
|
|
|
lookup encounters :math:`padding\_idx` in Ids. If set :attr:`None`, it makes
|
|
|
|
|
no effect to output. If :math:`padding\_idx < 0`, the :math:`padding\_idx`
|
|
|
|
|
will automatically be converted to :math:`size[0] + padding\_idx` to use.
|
|
|
|
|
Default: None.
|
|
|
|
|
param_attr(ParamAttr): Parameters for this layer.
|
|
|
|
|
dtype(np.dtype|core.VarDesc.VarType|str): The dtype refers to the data type of output
|
|
|
|
|
tensor. It can be float32, float_16, int etc.
|
|
|
|
|
The operator is used to lookup embeddings vector of ids provided by :attr:`input` .
|
|
|
|
|
It automatically constructs a 2D embedding matrix based on the
|
|
|
|
|
input :attr:`size` (vocab_size, emb_size) and :attr:`dtype` .
|
|
|
|
|
|
|
|
|
|
Returns:
|
|
|
|
|
Variable: The tensor variable storing the embeddings of the \
|
|
|
|
|
supplied inputs.
|
|
|
|
|
This OP requires the last dimension of Tensor shape must be equal to 1. The shape
|
|
|
|
|
of output Tensor is generated by replacing the last dimension of the input Tensor shape
|
|
|
|
|
with emb_size.
|
|
|
|
|
|
|
|
|
|
**Note:** The id in :attr:`input` must satisfy :math:`0 =< id < size[0]` ,
|
|
|
|
|
otherwise the program will throw an exception and exit.
|
|
|
|
|
|
|
|
|
|
.. code-block:: text
|
|
|
|
|
|
|
|
|
|
Case 1:
|
|
|
|
|
|
|
|
|
|
input is a Tensor. padding_idx = -1
|
|
|
|
|
input.data = [[[1], [3]], [[2], [4]], [[4], [127]]]
|
|
|
|
|
input.shape = [3, 2, 1]
|
|
|
|
|
Given size = [128, 16]
|
|
|
|
|
output is a Tensor:
|
|
|
|
|
out.shape = [3, 2, 16]
|
|
|
|
|
out.data = [[[0.129435295, 0.244512452, ..., 0.436322452],
|
|
|
|
|
[0.345421456, 0.524563927, ..., 0.144534654]],
|
|
|
|
|
|
|
|
|
|
[[0.345249859, 0.124939536, ..., 0.194353745],
|
|
|
|
|
[0.945345345, 0.435394634, ..., 0.435345365]],
|
|
|
|
|
|
|
|
|
|
[[0.945345345, 0.435394634, ..., 0.435345365],
|
|
|
|
|
[0.0, 0.0, ..., 0.0 ]]] # padding data
|
|
|
|
|
The input padding_idx is less than 0, it is automatically converted to padding_idx = -1 + 128 = 127
|
|
|
|
|
It will pad all-zero data when ids is 127.
|
|
|
|
|
|
|
|
|
|
Case 2:
|
|
|
|
|
|
|
|
|
|
input is a LoDTensor with 1-level LoD. padding_idx = 0
|
|
|
|
|
input.lod = [[2, 3]]
|
|
|
|
|
input.data = [[1], [3], [2], [4], [0]]
|
|
|
|
|
input.shape = [5, 1]
|
|
|
|
|
Given size = [128, 16]
|
|
|
|
|
output is a LoDTensor:
|
|
|
|
|
out.lod = [[2, 3]]
|
|
|
|
|
out.shape = [5, 16]
|
|
|
|
|
out.data = [[0.129435295, 0.244512452, ..., 0.436322452],
|
|
|
|
|
[0.345421456, 0.524563927, ..., 0.144534654],
|
|
|
|
|
[0.345249859, 0.124939536, ..., 0.194353745],
|
|
|
|
|
[0.945345345, 0.435394634, ..., 0.435345365],
|
|
|
|
|
[0.0, 0.0, ..., 0.0 ]] # padding data
|
|
|
|
|
It will pad all-zero data when ids is 0.
|
|
|
|
|
|
|
|
|
|
Args:
|
|
|
|
|
input(Variable): A Tensor or LoDTensor with type int64, which contains the id information.
|
|
|
|
|
The last dimension of Tensor shape must be equal to 1. The value of the input id should
|
|
|
|
|
satisfy :math:`0<= id < size[0]` .
|
|
|
|
|
size(tuple|list): The shape of lookup table parameter. It should have two elements which
|
|
|
|
|
indicates the size of the dictionary of embeddings and the size of each embedding vector respectively.
|
|
|
|
|
is_sparse(bool): The flag indicating whether to use sparse update. This parameter only
|
|
|
|
|
affects the performance of the backwards gradient update. It is recommended to set
|
|
|
|
|
True because sparse update is faster. But some optimizer does not support sparse update,
|
|
|
|
|
such as :ref:`api_fluid_optimizer_AdadeltaOptimizer` , :ref:`api_fluid_optimizer_AdamaxOptimizer` ,
|
|
|
|
|
:ref:`api_fluid_optimizer_DecayedAdagradOptimizer` , :ref:`api_fluid_optimizer_FtrlOptimizer` ,
|
|
|
|
|
:ref:`api_fluid_optimizer_LambOptimizer` and :ref:`api_fluid_optimizer_LarsMomentumOptimizer` .
|
|
|
|
|
In these case, is_sparse must be False. Default: False.
|
|
|
|
|
is_distributed(bool): Whether to store the embedding matrix in a distributed manner. Only used
|
|
|
|
|
in multi-machine distributed CPU training. Default: False.
|
|
|
|
|
padding_idx(int|long|None): padding_idx needs to be in the interval [-vocab_size, vocab_size).
|
|
|
|
|
If :math:`padding\_idx < 0`, the :math:`padding\_idx` will automatically be converted
|
|
|
|
|
to :math:`vocab\_size + padding\_idx` . It will output all-zero padding data whenever lookup
|
|
|
|
|
encounters :math:`padding\_idx` in id. And the padding data will not be updated while training.
|
|
|
|
|
If set None, it makes no effect to output. Default: None.
|
|
|
|
|
param_attr(ParamAttr): To specify the weight parameter property. Default: None, which means the
|
|
|
|
|
default weight parameter property is used. See usage for details in :ref:`api_fluid_ParamAttr` . In addition,
|
|
|
|
|
user-defined or pre-trained word vectors can be loaded with the :attr:`param_attr` parameter.
|
|
|
|
|
The local word vector needs to be transformed into numpy format, and the shape of local word
|
|
|
|
|
vector shoud be consistent with :attr:`size` . Then :ref:`api_fluid_initializer_NumpyArrayInitializer`
|
|
|
|
|
is used to load custom or pre-trained word vectors. See code example 2 for details.
|
|
|
|
|
dtype(str|core.VarDesc.VarType): It refers to the data type of output Tensor.
|
|
|
|
|
It must be float32 or float64. Default: float32.
|
|
|
|
|
|
|
|
|
|
Returns:
|
|
|
|
|
Variable: Embedding Tensor or LoDTensor mapped by input. The data type is the same as :attr:`dtype` .
|
|
|
|
|
|
|
|
|
|
Examples:
|
|
|
|
|
.. code-block:: python
|
|
|
|
|
|
|
|
|
|
import paddle.fluid as fluid
|
|
|
|
|
data = fluid.layers.data(name='sequence', shape=[1], dtype='int64', lod_level=1)
|
|
|
|
|
emb = fluid.layers.embedding(input=data, size=[128, 64])
|
|
|
|
|
import numpy as np
|
|
|
|
|
data = fluid.data(name='x', shape=[None, 1], dtype='int64')
|
|
|
|
|
|
|
|
|
|
# exampel 1
|
|
|
|
|
emb_1 = fluid.embedding(input=data, size=[128, 64])
|
|
|
|
|
|
|
|
|
|
# example 2: load custom or pre-trained word vectors
|
|
|
|
|
weight_data = np.random.random(size=(128, 100)) # word vectors with numpy format
|
|
|
|
|
w_param_attrs = fluid.ParamAttr(
|
|
|
|
|
name="emb_weight",
|
|
|
|
|
learning_rate=0.5,
|
|
|
|
|
initializer=fluid.initializer.NumpyArrayInitializer(weight_data),
|
|
|
|
|
trainable=True)
|
|
|
|
|
emb_2 = fluid.layers.embedding(input=data, size=(128, 100), param_attr=w_param_attrs, dtype='float32')
|
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
helper = LayerHelper('embedding', **locals())
|
|
|
|
|
@ -7841,25 +7909,83 @@ def smooth_l1(x, y, inside_weight=None, outside_weight=None, sigma=None):
|
|
|
|
|
|
|
|
|
|
def one_hot(input, depth, allow_out_of_range=False):
|
|
|
|
|
"""
|
|
|
|
|
This layer creates the one-hot representations for input indices.
|
|
|
|
|
|
|
|
|
|
Args:
|
|
|
|
|
input(Variable): Input indices, last dimension must be 1.
|
|
|
|
|
depth(scalar): An interger defining the depth of the one-hot dimension.
|
|
|
|
|
**WARING:** This OP requires the last dimension of Tensor shape must be equal to 1.
|
|
|
|
|
This OP will be deprecated in a future release. It is recommended to use fluid. :ref:`api_fluid_one_hot` .
|
|
|
|
|
|
|
|
|
|
The operator converts each id in the input to an one-hot vector with a
|
|
|
|
|
:attr:`depth` length. The value in the vector dimension corresponding to the id
|
|
|
|
|
is 1, and the value in the remaining dimension is 0.
|
|
|
|
|
|
|
|
|
|
The shape of output Tensor or LoDTensor is generated by adding :attr:`depth` dimension
|
|
|
|
|
behind the last dimension of the input shape.
|
|
|
|
|
|
|
|
|
|
.. code-block:: text
|
|
|
|
|
|
|
|
|
|
Example 1 (allow_out_of_range=False):
|
|
|
|
|
|
|
|
|
|
input:
|
|
|
|
|
X.shape = [4, 1]
|
|
|
|
|
X.data = [[1], [1], [3], [0]]
|
|
|
|
|
depth = 4
|
|
|
|
|
|
|
|
|
|
output:
|
|
|
|
|
Out.shape = [4, 4]
|
|
|
|
|
Out.data = [[0., 1., 0., 0.],
|
|
|
|
|
[0., 1., 0., 0.],
|
|
|
|
|
[0., 0., 0., 1.],
|
|
|
|
|
[1., 0., 0., 0.]]
|
|
|
|
|
|
|
|
|
|
Example 2 (allow_out_of_range=True):
|
|
|
|
|
|
|
|
|
|
input:
|
|
|
|
|
X.shape = [4, 1]
|
|
|
|
|
X.data = [[1], [1], [5], [0]]
|
|
|
|
|
depth = 4
|
|
|
|
|
allow_out_of_range = True
|
|
|
|
|
|
|
|
|
|
output:
|
|
|
|
|
Out.shape = [4, 4]
|
|
|
|
|
Out.data = [[0., 1., 0., 0.],
|
|
|
|
|
[0., 1., 0., 0.],
|
|
|
|
|
[0., 0., 0., 0.], # This id is 5, which goes beyond depth, so set it all-zeros data.
|
|
|
|
|
[1., 0., 0., 0.]]
|
|
|
|
|
|
|
|
|
|
Example 3 (allow_out_of_range=False):
|
|
|
|
|
|
|
|
|
|
input:
|
|
|
|
|
X.shape = [4, 1]
|
|
|
|
|
X.data = [[1], [1], [5], [0]]
|
|
|
|
|
depth = 4
|
|
|
|
|
allow_out_of_range = False
|
|
|
|
|
|
|
|
|
|
output: Throw an exception for Illegal value
|
|
|
|
|
The second dimension in X is 5, which is greater than depth.
|
|
|
|
|
Allow_out_of_range =False means that does not allow the word id to exceed depth,
|
|
|
|
|
so it throws an exception.
|
|
|
|
|
|
|
|
|
|
Args:
|
|
|
|
|
input(Variable): Tensor or LoDTensor with shape :math:`[N_1, N_2, ..., N_k, 1]` ,
|
|
|
|
|
which contains at least one dimension and the last dimension must be 1.
|
|
|
|
|
The data type is int32 or int64.
|
|
|
|
|
depth(scalar): An integer defining the :attr:`depth` of the one hot dimension. If input
|
|
|
|
|
is word id, depth is generally the dictionary size.
|
|
|
|
|
allow_out_of_range(bool): A bool value indicating whether the input
|
|
|
|
|
indices could be out of range [0, depth). When input indices are
|
|
|
|
|
out of range, exceptions is raised if allow_out_of_range is False,
|
|
|
|
|
or zero-filling representations is created if it is set True
|
|
|
|
|
indices could be out of range :math:`[0, depth)` . When input indices are
|
|
|
|
|
out of range, exceptions :code:`Illegal value` is raised if :attr:`allow_out_of_range`
|
|
|
|
|
is False, or zero-filling representations is created if it is set True.
|
|
|
|
|
Default: False.
|
|
|
|
|
|
|
|
|
|
Returns:
|
|
|
|
|
Variable: The one-hot representations of input.
|
|
|
|
|
Variable: The one-hot representations of input. A Tensor or LoDTensor with type float32.
|
|
|
|
|
|
|
|
|
|
Examples:
|
|
|
|
|
.. code-block:: python
|
|
|
|
|
|
|
|
|
|
import paddle.fluid as fluid
|
|
|
|
|
label = fluid.layers.data(name="label", shape=[1], dtype="int64")
|
|
|
|
|
one_hot_label = fluid.layers.one_hot(input=label, depth=10)
|
|
|
|
|
# Correspond to the first example above, where label.shape is [4, 1] and one_hot_label.shape is [4, 4].
|
|
|
|
|
label = fluid.data(name="label", shape=[4, 1], dtype="int64")
|
|
|
|
|
one_hot_label = fluid.layers.one_hot(input=label, depth=4)
|
|
|
|
|
"""
|
|
|
|
|
helper = LayerHelper("one_hot", **locals())
|
|
|
|
|
|
|
|
|
|
|
Aurelius84
5 years ago
committed by
GitHub