383564c417
fix perf_data test
5 years ago
709dfd7e81
!2738 [Dataset C++ UT] Implement C++ UT common class for object detection tests.
...
Merge pull request !2738 from islam_amin/object_ops_ut_cc
5 years ago
593ee1eb34
C++ UT common files for object detection tests
5 years ago
88bb65768e
Clean up part 1
...
Removed extra fields in schema
Fixed test cases
Fixing CI
5 years ago
71cede8173
remove de_error.h
5 years ago
9b4399480a
!2321 Connector throughput performance metric
...
Merge pull request !2321 from Alexey_Shevlyakov/connector_throughput_dev
5 years ago
41d16a3c77
enhance: 1. fix api comment, 2. del dead code
5 years ago
698abf75fc
Connector throughput + PerfData + CyclicArray
5 years ago
81bf4bde1d
AutoIndexObj primary should start with 0
5 years ago
c22eac743d
subtree creation in python apis
...
updates
fix
fix cpplint
fix
5 years ago
d9b1825552
!2374 C++ UT for CutOut and Pad
...
Merge pull request !2374 from MahdiRahmaniHanzaki/I1L892-pad-cutout-ut
5 years ago
e829535a86
change GetTensor into GetRow to avoid NullPtr
5 years ago
0d52888fc5
fix misspell and check parameters
5 years ago
bca037e1be
added cpp test cases for pad and cut out
5 years ago
cf82aa9035
init remove storage op in c++
...
init remove storage op test case c++
remove source c++ files
5 years ago
11826fb256
DuplicateOp
5 years ago
5515016dba
remove graphengine changes
...
remove graphengine changes
concat op
Truncate Pair
concat_op
remove graph engine changes
5 years ago
ffc8a3c362
!2274 add PadEndOp
...
Merge pull request !2274 from xunxue/padend
5 years ago
e4322baece
!2316 [Dataset] Move python tree transformation into cpp pass
...
Merge pull request !2316 from JunhanHu/alter_node_cpp
5 years ago
87aa9c8f7a
PadEndOp
5 years ago
57f3732ac3
Move python tree transformation into cpp pass.
5 years ago
5850b99161
!2211 Add TruncatePair Op to dataset
...
Merge pull request !2211 from h.farahat/pair_truncate
5 years ago
b9495a9ccc
Truncate Pair
5 years ago
a4048e192c
!2256 Rename Sampler::Reset() to Sampler::ResetSampler()
...
Merge pull request !2256 from JesseKLee/sampler
5 years ago
ea829e89c3
!2198 Add MaskOp to dataset
...
Merge pull request !2198 from h.farahat/mask_op
5 years ago
dcfdff6021
Rename Sampler Reset
5 years ago
4f16f036be
Add WhitespaceTokenizer and UnicodeScriptTokenizer for nlp
...
add CaseFold, NormalizeUTF8
add RegexReplace
add RegexTokenizer
add BasicTokenizer
add WordpieceTokenizer
add BertTokenizer
5 years ago
f2462bb00d
Mask Op
5 years ago
625f2421b5
!1977 random walk in gnn node2vec
...
Merge pull request !1977 from JonathanY/randomwalk
5 years ago
a1f194c971
!2188 Remove B+ tree deadcode and add an additional output to Search function
...
Merge pull request !2188 from JesseKLee/deadcode
5 years ago
287ca71260
Remove dead code
5 years ago
e2012a1de9
Slice Op
5 years ago
dceb3c2be4
!2026 Change C++ behavior in RandomCropAndResize
...
Merge pull request !2026 from MahdiRahmaniHanzaki/I1K3FL-random-resized-crop
5 years ago
add29f95f5
resolved issues in the comments
5 years ago
2cab44a637
!2028 FillOp for BERT
...
Merge pull request !2028 from nhussain/fillOp_learning
5 years ago
dd9bf09f0a
added FillOp for #119 - special Ops
5 years ago
87d2c27c7f
random walk v1
5 years ago
a8cf83acf2
!1932 Add CLUE dataset
...
Merge pull request !1932 from jiangzhiwen/dataset/clue
5 years ago
e0e167a000
add CLUE dataset
5 years ago
c661087b99
!2029 Create TensorRow class that supports a row id
...
Merge pull request !2029 from lixiachen/lixia/tensorrow
5 years ago
6ae88c3987
Create TensorRow class that supports a row id.
5 years ago
0c5e47111b
- Move MutableBuffer to be protected
...
- Add new CreateTensors
5 years ago
beefb20c01
!1983 Remove inheritance of Sampler from DatasetOp
...
Merge pull request !1983 from JesseKLee/sampler
5 years ago
b3da41bd7a
!1913 CocoDataset implementation
...
Merge pull request !1913 from xiefangqi/xfq_support_coco
5 years ago
097cbcc0cb
CocoDataset implementation
5 years ago
255adf7c6e
Remove inheritance of Sampler from DatasetOp
5 years ago
3ece8dd090
1. support get_all_edges, get_nodes_from_edge, get_sampled_neighbors, get_neg_sampled_neighbors and graph_info API
...
2. mod cora and citeseer conversion script
5 years ago
5eb95599f6
!1874 Updates to string tensor
...
Merge pull request !1874 from h.farahat/string_tensor_updates
5 years ago
f837ddc956
- Bug when empty strings sent to Python
...
- Support accepting Numpy of str as input
- Support batching strings
- Core logic of batch&pad is static
- Make Pad a utility function
5 years ago
51bc0c0460
consistent design for num_samples
...
updates
more update
more work
more fixin
post rebase updates
clang formatting
code review recovery
ci fixes
updates
update
update
update
5 years ago
3b80c10b1e
Fix some timing hole in these two testcases
5 years ago
e311473401
!1453 Raise mse threshold in RandomCropDecodeResizeOp
...
Merge pull request !1453 from Alexey_Shevlyakov/fix_random_crop_decode_resize_ut
5 years ago
a8fa847556
Allow join_all to be blocking
5 years ago
7a807cbff9
fix ut
5 years ago
641112a40c
Initial Drop
5 years ago
964a757db2
!1378 Just want to debug this case in Jenkin environment
...
Merge pull request !1378 from JesseKLee/cond_var_debug
5 years ago
06ae724261
Again
5 years ago
458436186c
!1365 Clean up work for text python sub-package
...
Merge pull request !1365 from h.farahat/text_namespace
5 years ago
6f733ec113
!1308 Stage 2 of adding support for string Tensor
...
Merge pull request !1308 from h.farahat/string_tensor2
5 years ago
6c21e556c4
Clean up work for text python package
5 years ago
599a449e0b
Support processing GNN data
5 years ago
df361d1d26
Change mem layout of string tensor
...
add support for MindRecord and TFRecord
----
optimize tensorshape
optimize tensorshape and FlatIndex
TFRecord and MindRecord support for string tensor
Modify mem layout
Add new constructor
Add method Allocate
Change some GetMutableBuffer usages to AllocateBuffer
5 years ago
2e3d55ed87
!1281 Implementation of SplitOp
...
Merge pull request !1281 from Peilin/splitOp
5 years ago
71e8bb1960
general split case done, chaining sampler (basic case) is working
...
implementation 99% complete
everything and tested except for repeatable shuffling
tested most basic/typical split usecases
cleanup
some more cleanup
fix CI
more ci fix
more ci fixes
more ci fix
more ci fix
more ci fix
added more tests, fixed some bugs
some more clean up and test cases
added shard/shuffle before split warning/error
addressed code review comments and ci
fixed ci
5 years ago
451c20a6f5
Add UnicodeCharTokenizer for nlp
5 years ago
163b6b7ea7
add jieba c++ code
5 years ago
e8ca243364
-Add DE_STRING
...
-replace switch'case by indexing
- Add test case
- Add constructors
- Add getItem string
- Fix bugs
- Add more tests
- Tensor iterator
- asNumpy
- TextFileDataset
- Tensor(Numpy)
- Super > 2D
- Add more test cases for GeneratorDataset
- Change StartAddr to GetBuffer and GetMutableNuffer
- Raise an error if batch is used with strings
Clean-up work
5 years ago
c680cfbf27
!1157 dataset: add concat operation for dataset
...
Merge pull request !1157 from ms_yan/concat_dataset
5 years ago
ab031ee9ea
!1126 VOCDataset support object detection function
...
Merge pull request !1126 from xiefangqi/voc_support_detection
5 years ago
c937bad53f
minddata support voc
5 years ago
c0fa7b4b19
init commit of concat dataset
...
change to use __add__ operation instead ds.concat
5 years ago
9e116e15fb
!1139 Change Thread::Join to async call with time out
...
Merge pull request !1139 from JesseKLee/timing_hole
5 years ago
1a052f7155
!1131 [Data]fix cmake build format
...
Merge pull request !1131 from xulei/filter_op
5 years ago
f2be49d85e
Fix a racing condition
5 years ago
903b64daa2
fix cmake build format
5 years ago
aa3f89e74f
mindrecord support read file list
5 years ago
3c643e7298
Move from dataset_dev branch to here
5 years ago
c2d364a573
batch with padding implemented
...
support for 1 specific dimension to be None, added validator
fix various CI complains
another round of CI fixes
ci
refactor parts of the code
code refactor
ci fix
comments added, fix bugs
address review comments
address review comments
review cmts
added simple perf test script
update pad code
perf imprv
5 years ago
9a2daae84c
del unused change_mode_op
5 years ago
2303453753
!869 Random data op
...
Merge pull request !869 from JesseKLee/random_data_op
5 years ago
270bf831a9
Random Data Op
5 years ago
98939d839c
!856 Fix skip op bug
...
Merge pull request !856 from jiangzhiwen/dataset/skip_thread
5 years ago
34bfa2f7c9
fix skip
5 years ago
332398d20a
Check return code from WaitPost::Register
5 years ago
6369cf27bd
!406 added first row crc check for when reading tfrecord files
...
Merge pull request !406 from Peilin/first-row-crc-check
5 years ago
c705ea5e5b
add filterOp code
5 years ago
9bc2134cb7
added checking of first row crc to find invalid tfrecord files
...
addressed code review comments. added check in python layer to exclude directories and to raise an error if a pattern does not match any file
fixed clang format
fixed cppcheck
fixed cppcheck (used std::accumulate and std::copy_if). regenerated tfrecord file to contain correct header, it was a dummy header before
fixed cppcheck: added const reference for string parameter for lambdas, fixed clang format: whitespace adjustments
more clang whitespace fixes...
changed print to logger.info
5 years ago
d8176a77f4
!425 New api "TextFileDataset"
...
Merge pull request !425 from yanghaitao/yht_textfiledatasetv2
5 years ago
c5cfb09e66
Repair some MS_LOG problem
5 years ago
2795e492ff
TextFileDataset
5 years ago
f0c07c3fa6
Realize take op and add ut
5 years ago
3f7054dccb
add skip dataset op
5 years ago
cf026096a6
!183 Mindspore.dataset CPP sampler for GeneratorDataset
...
Merge pull request !183 from JunhanHu/cpp_sampler
5 years ago
9739d3b048
Add CPP sampler support for GeneratorDataset
5 years ago
0fc23eee0f
Support nested repeat
5 years ago
cc75cb357c
!168 remove mindspore::make_unique and make_unique.h
...
Merge pull request !168 from Alexey_Shevlyakov/remove_mindspore_make_unique
5 years ago
20d1b64443
fix: error info is not exactly when column list invalid
5 years ago
9d0fde29f4
remove ENABLE_MINDRECORD flag
5 years ago
84d780c1a4
remove make_unique.h
5 years ago
d37b8c4f2d
fix RandomCropDecodeResize test
5 years ago
080bcda021
Replace std::cout with MS_LOG in dataset unit test
...
Signed-off-by: Jonathan Yan <jonathan.yan1@huawei.com>
5 years ago
930a1fb0a8
initial version
...
Signed-off-by: leonwanghui <leon.wanghui@huawei.com>
5 years ago
ale
mindspore-ci-bot
islam_amin
Eric
Alexey Shevlyakov
jonyguo
Jesse Lee
Jamie Nisbet
ms_yan
heleiwang
Mahdi
hesham
nhussain
xunxue
Junhan Hu
qianlong
Jonathan Yan
jiangzhiwen
Lixia Chen
xiefangqi
mindspore-ci-bot
Peilin Wang
xulei2020
mindspore-ci-bot
liyong
Zirui Wu
yanghaitao
jzw
zhunaipan