0e27dccbcf
!3093 VOCDataset output change to multi-columns
...
Merge pull request !3093 from xiefangqi/md_voc_multi_columns
5 years ago
28ebd73096
fix bug when int or float is numpy type
5 years ago
572750cd40
Added cpp Equalize op
...
adding missed files
Fixed Errors
fixed bugs
fixed bugs
fixed bugs
fixed bugs
fixed bugs
Updated Python UT for Equalize op
Added cpp Equalize op
adding missed files
Fixed Errors
fixed bugs
fixed bugs
fixed bugs
fixed bugs
fixed bugs
Updated Python UT for Equalize op
Fixed comment style
update files
deleted files
added files
added image_utils.h
Fixed PyLint and CPPLint Errors
updated tensor_op.h
5 years ago
b064cbab5d
fix upper bound
5 years ago
b091f74c23
!3016 Add CSV dataset loader
...
Merge pull request !3016 from jiangzhiwen/dataset/csv
5 years ago
2f506b7985
CSV dataset loader
5 years ago
fa7fa8a162
!3234 [MD] fix bug when save tfrecord data
...
Merge pull request !3234 from liyong126/fix_save_tfrecod
5 years ago
f521532a06
fix field_name probelem from tfrecord to mindrecord
5 years ago
18b519ae0f
add sentence piece
5 years ago
fbd9e3d402
vocdataset output multi-columns
5 years ago
008b91b2a1
inject epoch ctrl op in the execution tree and send eos at the end of epoch
5 years ago
4945d34a41
!3133 [MD] fix num_sample in distributed sampler
...
Merge pull request !3133 from liyong126/md_num_of_samples
5 years ago
a84092e89d
!3102 [MD] Add additional parameter checks in RandomCropWithBBoxOp and RandomResizeWithBbox
...
Merge pull request !3102 from nhussain/random_crop_fixes
5 years ago
e2ea1fa0df
activate num_samples in distributed samplers
5 years ago
1bcbbb08d6
!2973 [MD] Add implementation of SlidingWindow TensorOp, c++ & python
...
Merge pull request !2973 from luoyang/update
5 years ago
1a518e1cff
!3144 [MD] Fix ResizeOp behaviour
...
Merge pull request !3144 from nhussain/dataset_check
5 years ago
c984b0fc88
fix size tuple
5 years ago
35c0f1d2e6
!3143 Add python UT to MnistDataset op
...
Merge pull request !3143 from TinaMengtingZhang/dataset-op-ut-mnist
5 years ago
ab37e87d7a
adding Mnist python ut coverage
5 years ago
5647889c0d
Added AutoContrast Op
5 years ago
37e8439c0a
!3077 supporting cpp invert operation
...
Merge pull request !3077 from ava/invert
5 years ago
c07a8d2033
!3137 Cleanup dataset UT: Remove deprecated save_and_check
...
Merge pull request !3137 from cathwong/ckw_dataset_ut_cleanup7
5 years ago
5871ba8a05
more validator fixes
5 years ago
bc676fe250
save op in minddataset
5 years ago
35c3a63701
support cpp invert operation
5 years ago
04eb4f89c7
Cleanup dataset UT: Replace save_and_check
5 years ago
d233c54139
implemented cpp random choice, apply and compos
...
python part of random ops
added random select sub policy
validators added
comments added, remaining issues addressed
add more python test cases
fix ci
fix CI
fix order of include files
addr review cmts
addr review cmts
reorg file
fix compile err
address review cmts
address review cmts
5 years ago
4136892a3e
add SlidingWindow Op
5 years ago
bc0a53cfb1
!3030 minddata support ps-lite feature
...
Merge pull request !3030 from xiefangqi/md_support-pslite
5 years ago
ba0143402c
!3067 Cleanup dataset UT: Remove unneeded tf data files and tests
...
Merge pull request !3067 from cathwong/ckw_dataset_ut_cleanup6
5 years ago
219a716eae
!3066 fix some batch's get_dataset_size and some text validator inconsistency
...
Merge pull request !3066 from ZiruiWu/fix_validator
5 years ago
bed93a9ead
!3039 [Dataset Python UT] adding coverage to Cifar Op and update test dataset
...
Merge pull request !3039 from TinaMengtingZhang/dataset-op-ut-cifar
5 years ago
63185cb20f
fix some validators errors
...
address review cmts
addr review cmts
5 years ago
ae1ed327ba
Cleanup dataset UT: Remove unneeded data files and tests
5 years ago
05b2a57d2a
fix validation errors, and fix try catch error tests
5 years ago
340d98a4d1
added test case to cifar_op
...
update cifar10 dataset
fixing missing error handling code in validator
5 years ago
add19a591c
md support ps-lite
5 years ago
48654be14d
!3040 BoundingBoxAugment op fix ratio
...
Merge pull request !3040 from islam_amin/bounding_box_augment_bugs
5 years ago
56da3b0ae1
Fixing ratio bug with BoundingBoxAugment
5 years ago
7c1bc5192b
Initial Drop of CacheOp Phase I
5 years ago
6284c42a76
!2941 MD tokenizer support output offsets
...
Merge pull request !2941 from xiefangqi/md_add_offsets_feature
5 years ago
47060631e5
add offsets feature to tokenizer
5 years ago
541456044d
!2934 GraphData support random_walk and get_edge_feature
...
Merge pull request !2934 from heleiwang/gnn_july
5 years ago
2f565f4c20
!2776 Update API of dataset.text.lookup
...
Merge pull request !2776 from ZiruiWu/master
5 years ago
7a046a1d70
support get_edge_feature
5 years ago
7b15e5a742
rework on lookup
...
add test caser
fix ci
address review cmts
ci
addr review cmt
fix typo
address review cmts
add 2 more test cases
cpplint fix
addr cpplint
addr ci
fix tst case err
fix doc str
5 years ago
2c7fd248f8
fixes for PR-2908: avoid empty strings for column names
5 years ago
6c37ea3be0
fix validators
...
fixed random_apply tests
fix validators
fixed random_apply tests
fix engine validation
5 years ago
915ddd25dd
!2892 gnn random walk pr 1977 comment follow up
...
Merge pull request !2892 from tony_liu2/master
5 years ago
421b80bf8f
updating python unit tests based on floating point bounding boxes + adding coco dataset in unit tests
5 years ago
14899a1410
fix gnn random walk pr 1977 comments
...
add fix to random resize decode crop test case
fix pylint issues
5 years ago
9ba937b1af
!2920 Updating 3 BBox AugOps for Float + Common Files for BBox Float Support - (VertFlip/RandomCrop/RandomResizeCrop)
...
Merge pull request !2920 from danishnxt/AugOps-M1-Python
5 years ago
c9929fd8a1
!2900 Update BoundingBoxAugment and RandomHorizontalFlipWithBBox to use float
...
Merge pull request !2900 from islam_amin/object_ops_cc_float
5 years ago
92d2011b8b
!2924 [Dataset Python UT] Disable md5 random perspective test
...
Merge pull request !2924 from TinaMengtingZhang/redo-skip-md5-random-perspective-test
5 years ago
78c370f72a
First Commit - New Infrastructure - Python UT tests + Common Aug Files
...
fix accidental change overwrite
fix 2
updated inits from 0 to 0.0 for float
improvd python ut comments
updated macro
5 years ago
8455f957ac
disable md5 random perspective test case
5 years ago
dfc3409f67
Update RandomHorizontalFlipWithBBox and BoundingBouxAugment C++ Ops to use floats
5 years ago
62d55fdbce
vocdataset support float bbox
5 years ago
b5066e81ee
!2308 Unskip md5 python UT case for RandomPerspective
...
Merge pull request !2308 from TinaMengtingZhang/unskip-case-random-perspective
5 years ago
dc03019200
unskip md5 testcase for RandomPerspective
5 years ago
5a9673ee6f
fix_annotate bug fix
5 years ago
29aa589972
added check for invalid type for boolean args
5 years ago
365c901ee0
get default value if num_parallel_worker is None
5 years ago
d08a89ab87
!2737 makes 0 an invaild bucket size
...
Merge pull request !2737 from Peilin/zero-input-bucket-check
5 years ago
1227719cdd
!2603 [Dataset Python UT] Cleanup mse and md5 validation
...
Merge pull request !2603 from TinaMengtingZhang/I1LTCA-mse-check-cleanup
5 years ago
9468b49e28
bucket_batch_sizes must be strictly positive, 0 is not a valid batch size
5 years ago
c70e6987ba
!2624 Fix config test
...
Merge pull request !2624 from EricZ/fix_config_test
5 years ago
9b4399480a
!2321 Connector throughput performance metric
...
Merge pull request !2321 from Alexey_Shevlyakov/connector_throughput_dev
5 years ago
5cd3136355
remove local defined mse and add missing mse/md5 validation
5 years ago
7fa0d9e7e4
add paramter check for numpyslices and num_shards
5 years ago
32a2d5d21c
!2629 get monitor sampling interval from json file
...
Merge pull request !2629 from yanghaitao/yht_sampling_interval
5 years ago
363489d00f
!2580 BasicTokenizer do not case fold on preserved words
...
Merge pull request !2580 from qianlong21st/fix_basic_tokenizer
5 years ago
cae77c0c22
BasicTokenizer not case fold on preserverd words
5 years ago
68e2097897
fix split erroer message
5 years ago
fd701b8504
set monitor sampling interval through json file
5 years ago
a4f5802924
Added Fix for test config
5 years ago
698abf75fc
Connector throughput + PerfData + CyclicArray
5 years ago
a1b17bd2e2
!2593 python unit tests for randomResizeWithBBox and ResizeWithBBox
...
Merge pull request !2593 from ava/python_ut_tests
5 years ago
1e869146e9
applying comments removing VOC
5 years ago
277aba5326
dataset: Fixup docs; remove pylint disabled messages in UT
5 years ago
e11c953225
!2587 Implementing Python UTs for BoundingBoxAugment and RandomHorizontalFlipWithBBox
...
Merge pull request !2587 from imaaamin/object_ops_ut
5 years ago
067592ccf9
!2590 Updated Python UTs for RandomVerticalFlipWithBBox, RandomCropWithBBox and RandomResizedCropWithBBox
...
Merge pull request !2590 from danishnxt/AugOps2
5 years ago
0f58f0338e
updating ut for RandomHorizontalFlipWithBBox and BBoxAugment
5 years ago
b0a10c26a4
!2532 adding random-resize-with-bbox-op and resize-with-bbox-op
...
Merge pull request !2532 from ava/NNNewMaster
5 years ago
3d1ecaaeb5
updated UT test for Python (3) AugOps with BBox - MD5 checks + imrpv comments
5 years ago
587e260236
addressing comments
5 years ago
c22eac743d
subtree creation in python apis
...
updates
fix
fix cpplint
fix
5 years ago
a8f392ca42
updated test util file + new BoundingBoxCheck + fixed VOCDataset annotations
...
Style Error fix
fixed PyLint problem
reverting testVOC2012 due to CI break for existing test
reverting testVOC2012 due to CI break for existing test-2
updated old error messages to confirm with global standard
addressing PR 2355 Comments - 1
addressing PR 2355 Comments - 2
addressing PR 2355 Comments - 3
5 years ago
4e3bfcf4c9
!2306 [Dataset] Code review & improve quality
5 years ago
beb436f457
!2398 Bug in Slice when multiple rows are used
...
Merge pull request !2398 from h.farahat/slice_bug
5 years ago
a9c309da4c
!2404 [Dataset]rename input variable name to children and output variable to parent
...
Merge pull request !2404 from xulei/input_output
5 years ago
1ea38eb60c
!2375 Add Python Tokenizer
...
Merge pull request !2375 from h.farahat/python_tokenizer
5 years ago
2412ee09ce
fix distributedSampler reshuffle and fix random_device failed
5 years ago
78a8bc302d
!2380 Fix CocoDataset issue
...
Merge pull request !2380 from xiefangqi/xfq_fix_coco_issue_01
5 years ago
3b71bd0d5d
rename input to children, output to parent
5 years ago
68030e6a4b
Slice Bug
5 years ago
980ddd32a2
change output of WordpieceTokenizer and BertTokenizer to 1-D string tensors
5 years ago
5703a10b8b
fix CocoDataset issue
5 years ago
bbf69912be
!2363 fix TextFildDataset and CLUEDataset does not support to_device
...
Merge pull request !2363 from yanghaitao/yht_fix_textfiledataset
5 years ago
e981c67acd
Python Tokenizer
...
!38 Synchronize with latest Ascend software suite 17 Jun 2020
Merge pull request !38 from yanghaoran/master
5 years ago
674415f7be
Cleanup work for Concate, Mask, Slice, PadEnd and TruncatePair
5 years ago
bc4b1c2460
!2373 Comments + improved casting/parameters + UT tests for 3 AugOps (Augmentation Ops RandomCropWithBBox, Random Vertical FlipWithWithBBox and Random CropAndResizeWithWithBBox)
...
Merge pull request !2373 from danishnxt/AugOpsD1
5 years ago
7bdcc319b2
!2325 Addressing comments from PR2314
...
Merge pull request !2325 from imaaamin/object_ops_pr2
5 years ago
6442a85b75
comments added, ptrs to consts, UT tests for BBOps
...
updated comments for 3 BB AugOps
addressing comments, void -> status, macros added, fixed test cmt
changed types -> value, fixed ut test funcs, grammar
bug type fix
fixed condition in tests for viz
5 years ago
edc42c5b85
addressed comments from reviews
5 years ago
d89101b95f
add missing test
5 years ago
9f336bb735
fix TextFileDataset and CLUEDataset failed with to_device
5 years ago
1e36b0649f
remove graphengine changes
...
remove graphengine changes
concat op
Truncate Pair
concat_op
remove graph engine changes
ToNumberOp implementation almost done
ToNumberOp complete
ci fix
ci fix
ci fix
ci fix
ci fix
ci fix
ci fix
ci fix
ci fix
ci fix
merge conflicts
5 years ago
11826fb256
DuplicateOp
5 years ago
90bb9320aa
!2314 Implemented RandomHorizontalFlipWithBBox and BoundingBoxAugment C++ Ops
...
Merge pull request !2314 from imaaamin/object_ops_pr
5 years ago
a83baafbf6
!2253 Add ConcatOp to Dataset
...
Merge pull request !2253 from nhussain/concat_op
5 years ago
3784220056
!2317 rework on dataset.text.vocab to support any user special_tokens
...
Merge pull request !2317 from ZiruiWu/vocab_rework
5 years ago
5515016dba
remove graphengine changes
...
remove graphengine changes
concat op
Truncate Pair
concat_op
remove graph engine changes
5 years ago
ffc8a3c362
!2274 add PadEndOp
...
Merge pull request !2274 from xunxue/padend
5 years ago
e4322baece
!2316 [Dataset] Move python tree transformation into cpp pass
...
Merge pull request !2316 from JunhanHu/alter_node_cpp
5 years ago
0af156d169
added two ops for object detection, random horizontal flip with bbox and bouding box augment. along with their unit tests
5 years ago
87aa9c8f7a
PadEndOp
5 years ago
57f3732ac3
Move python tree transformation into cpp pass.
5 years ago
b6e9504b31
phase I of Vocab rework
...
phase II vocab rework
added more test cases
fix api doc string
address review cmts and fix CI
address ci complains
fix review cmts
ci
5 years ago
8f2674850b
address API doc style and content
...
addr review cmts
revert
ci
private class
ci
fix ci
address ci
address review cmts
fix style err
5 years ago
5850b99161
!2211 Add TruncatePair Op to dataset
...
Merge pull request !2211 from h.farahat/pair_truncate
5 years ago
d0c5071cab
!1993 [Dataset] Fix codedex.
...
Merge pull request !1993 from luoyang/pylint
5 years ago
8d1dae46ac
Throw error when load config failed
5 years ago
b9495a9ccc
Truncate Pair
5 years ago
ea829e89c3
!2198 Add MaskOp to dataset
...
Merge pull request !2198 from h.farahat/mask_op
5 years ago
808380a9f9
!2142 Cleanup dataset UT: refactor visualize()
...
Merge pull request !2142 from Tinazhang/reorg-visualize
5 years ago
fdafc690f8
repair problem in NumpySlicesDataset
5 years ago
169eaf73c7
reorganize util function visualize()
5 years ago
dee8471d23
!1993 [Dataset] Fix codedex.
5 years ago
4f16f036be
Add WhitespaceTokenizer and UnicodeScriptTokenizer for nlp
...
add CaseFold, NormalizeUTF8
add RegexReplace
add RegexTokenizer
add BasicTokenizer
add WordpieceTokenizer
add BertTokenizer
5 years ago
f2462bb00d
Mask Op
5 years ago
ca94916ef9
!2184 CocoDataset add category_id check
...
Merge pull request !2184 from xiefangqi/xfq_fix_coco_issue
5 years ago
625f2421b5
!1977 random walk in gnn node2vec
...
Merge pull request !1977 from JonathanY/randomwalk
5 years ago
3adbde56b2
!2162 SliceOp
...
Merge pull request !2162 from h.farahat/slice_op
5 years ago
898b2fdecd
!2187 Fix comment display issues in BuildVocabDataset
...
Merge pull request !2187 from ZiruiWu/vocab_rework
5 years ago
2794883644
fix selected minor issues
...
fix review comments
5 years ago
aab190fcdb
CocoDataset add category_id validation
5 years ago
d4f8f57c7e
add split in minddataset
5 years ago
e2012a1de9
Slice Op
5 years ago
dceb3c2be4
!2026 Change C++ behavior in RandomCropAndResize
...
Merge pull request !2026 from MahdiRahmaniHanzaki/I1K3FL-random-resized-crop
5 years ago
add29f95f5
resolved issues in the comments
5 years ago
2cab44a637
!2028 FillOp for BERT
...
Merge pull request !2028 from nhussain/fillOp_learning
5 years ago
dd9bf09f0a
added FillOp for #119 - special Ops
5 years ago
848e07d022
initial commit, start of BucketBatchByLengthOp
...
c implementation done, just need to call batch/pad
added python api and validator
added pybind/de_pipeline stuff, fixed some compile errors, figure out how null py::function works
added tiny bit of doc
integrated with static batch methods
fixed some bugs
some more bug fixes and cleanup
ci fix
fix ci
ci fix
fix ci
added test_cases and debugged
addressed code review comments
addressed code review comments
ci fix
ci fix
addressed code review comments
addressed code review comments
5 years ago
87d2c27c7f
random walk v1
5 years ago
a8cf83acf2
!1932 Add CLUE dataset
...
Merge pull request !1932 from jiangzhiwen/dataset/clue
5 years ago
e0e167a000
add CLUE dataset
5 years ago
a22763b612
Bug in CIFAR after removing GetMutabble buffer
...
Fixes # I1KIPC
5 years ago
bb2082daec
fix: MindDataset padded log error
5 years ago
3085e51e45
!1922 check parameter num_samples of sampler
...
Merge pull request !1922 from yanghaitao/yht_check_num_samples_sampler
5 years ago
1b5d9a1194
!1981 Implemented from_dataset, a way to build a vocab from existing dataset
...
Merge pull request !1981 from ZiruiWu/from_dataset_dev
5 years ago
880ce5ea26
implemented from_dataset
...
fix complie error
more tests
address CI complains
fix ci
adress review comments
address review cmts
5 years ago
4e333a2f22
MindData profiler infrastructure
5 years ago
0a95223f25
!1910 RandomColorAdjust error for grayscale images
...
Merge pull request !1910 from MahdiRahmaniHanzaki/I1J9SQ-random-color-adjust-bug
5 years ago
8f16cff12e
add para check for sampler
5 years ago
1de7271afc
add floatxx test case
5 years ago
bc22c172b8
add TensorDataset and its ut
5 years ago
251a6667a5
!1989 fix MindDataset distribute shuffle error
...
Merge pull request !1989 from guozhijian/fix_distribute_shuffle
5 years ago
b3da41bd7a
!1913 CocoDataset implementation
...
Merge pull request !1913 from xiefangqi/xfq_support_coco
5 years ago
097cbcc0cb
CocoDataset implementation
5 years ago
96859f44b6
fix: MindDataset distribute shuffle bug
5 years ago
ba2e2dced4
fixed some warnings and typos
5 years ago
2812cd57db
fix pylint error
5 years ago
fe0c3e9a8f
!1908 Add UT to six data vision transforms ops
...
Merge pull request !1908 from Tinazhang/ut-normalize
5 years ago
3ece8dd090
1. support get_all_edges, get_nodes_from_edge, get_sampled_neighbors, get_neg_sampled_neighbors and graph_info API
...
2. mod cora and citeseer conversion script
5 years ago
5eb95599f6
!1874 Updates to string tensor
...
Merge pull request !1874 from h.farahat/string_tensor_updates
5 years ago
c3de5c7a54
adding new TCs to six ops and skip md5 case in RandomPerspective
5 years ago
f837ddc956
- Bug when empty strings sent to Python
...
- Support accepting Numpy of str as input
- Support batching strings
- Core logic of batch&pad is static
- Make Pad a utility function
5 years ago
feff8899ac
support padding samples
5 years ago
5c21616293
!1807 Implemented Ngram TensorOp for dataset
...
Merge pull request !1807 from ZiruiWu/ngram_dev
5 years ago
dbf9936ec4
Implemented n-gram for dataset TensorOp
5 years ago
51bc0c0460
consistent design for num_samples
...
updates
more update
more work
more fixin
post rebase updates
clang formatting
code review recovery
ci fixes
updates
update
update
update
5 years ago
33fe66fd1e
fix error handling for bigger crop size case
5 years ago
5522edc4b5
!1713 [Dataset] Clean pylint.
5 years ago
7b911886ec
Fixing some tiny faults about Pylint in my code(ops)
5 years ago
7878743400
!1638 fixed rounding edge case in split
...
Merge pull request !1638 from Peilin/splitOp-after-testing
5 years ago
0e3dd8149a
!1703 Adding sync_wait input check
...
Merge pull request !1703 from EricZ/master
5 years ago
c366b3fb18
!1704 fix num rows bug in sampler
...
Merge pull request !1704 from Peilin/fix-random-sampler-bug
5 years ago
8e1b2107ce
Added example for sync_wait bad batch_num
...
Added broken test case for cv
Fixed error handling in sync_wait
5 years ago
976b4793d1
!1544 Python Unit Test for TenCrop
...
Merge pull request !1544 from MahdiRahmaniHanzaki/I1IHNG-ten-crop
5 years ago
3ef3d1433d
fixed num_samples bug for RandomSampler
5 years ago
f4a61fdbb2
Adding new UT cases and missing TCs in PY Ops.
5 years ago
5469be2a97
fixed bug for split, RandomSampler and some other cleanup
...
add another test case
typo
merge conflict
another PR changed testing behavior, updated test cases in this commit
added input check for use_sampler
addressed code review comments
fixed pylint, not related to my changes
fixed edge case of rounding in getting split sizes
fix pylint
5 years ago
dfc097019b
Added TenCrop test
...
Added unit tests for both testing the functinality of the TenCrop and
its error messages. Due to the similarity of this method to FiveCrop the
test cases are similar to FiveCrop test cases.
Signed-off-by: Mahdi <mahdi.rahmani.hanzaki@huawei.com>
added error_msg function call in the main method
refactored the test and added visual representation of the results
Separated the two error cases into two different functions and used the
visualize function in util.py to plot the result of TenCrop.
Signed-off-by: Mahdi <mahdi.rahmani.hanzaki@huawei.com>
Added new test cases
Added new test cases including test case for checking the error message
when the size variable is not a positive integer, test case for
rectangle crop, test case for vertical flip setting, and testing for
similarity of the result of TenCrop for the same input data in different runs.
Signed-off-by: Mahdi <mahdi.rahmani.hanzaki@huawei.com>
changed visualize in test_five_crop
Changed the visualize function in test_five_crop to use the already
existing function in util.py
Signed-off-by: Mahdi <mahdi.rahmani.hanzaki@huawei.com>
made generate_golden variable global
5 years ago
21da86b393
!1591 add get_dataset_size for CelebADataset
...
Merge pull request !1591 from yanghaitao/yht_celeba_get_dataset_size
5 years ago
cc6c7a3f60
add get_dataset_size to celebadataset
5 years ago
0f4dc13e8e
!1478 [Dataset] Fix CI warnings, clean pylint.
...
Merge pull request !1478 from luoyang/pylint
5 years ago
9b2a778d94
!1478 [Dataset] clean pylint.
5 years ago
5e4728c50f
add voc support split
5 years ago
d4c93575e7
fixed bug for split, RandomSampler and some other cleanup
...
add another test case
typo
merge conflict
another PR changed testing behavior, updated test cases in this commit
added input check for use_sampler
addressed code review comments
fixed pylint, not related to my changes
5 years ago
abca62f407
10:00 26/5 clean pylint
5 years ago
0a8ef2fe72
!1455 Cleanup dataset UT: restore config support
...
Merge pull request !1455 from cathwong/ckw_dataset_config_util
5 years ago
f891e1755c
Cleanup dataset UT: restore config support
5 years ago
aeb4c52f2d
!1379 Creating new UTs to three vision transformation ops
...
Merge pull request !1379 from Tinazhang/linearTrans
5 years ago
718a96430f
!1389 Security issues in recent string tensor PR
...
Merge pull request !1389 from h.farahat/fix_security_issues
5 years ago
a6b8451a33
!1380 make ShuffleOp have deterministic behavior for subsequent epochs
...
Merge pull request !1380 from Peilin/shuffle-subsequent-epoch-deterministic
5 years ago
f28f883cac
fix comments error and modify parameter check
5 years ago
c0f0e154f5
Clean up work for text python package
5 years ago
41456ac824
!1369 dataset: delete StorageDataset
...
Merge pull request !1369 from ms_yan/del_storage
5 years ago
7322839b04
add UTs for LinearTransformation, ToPIL, ToType
5 years ago
0cbcc7200b
made shuffle determinisitc for subsequent epochs
5 years ago
61639be1e0
!1375 Cleanup dataset UT: remove num_parallel_workers=1 in test_exception
...
Merge pull request !1375 from cathwong/ckw_dataset_ut_cleanup5
5 years ago
a528797253
!1377 Code Fix for Uniform Augmentation
...
Merge pull request !1377 from Tinazhang/cc
5 years ago
458436186c
!1365 Clean up work for text python sub-package
...
Merge pull request !1365 from h.farahat/text_namespace
5 years ago
b390883c6a
Bug fix
5 years ago
702005d403
Cleanup dataset UT: remove num_parallel_workers=1 in test_exception
5 years ago
d5e896b51c
delete storageDataset Op API and its test case
5 years ago
6f733ec113
!1308 Stage 2 of adding support for string Tensor
...
Merge pull request !1308 from h.farahat/string_tensor2
5 years ago
34236ce1f1
fix pylint
5 years ago
3363d4e834
!1249 Add GNN dataset processing API
...
Merge pull request !1249 from heleiwang/hlw_gnn_data
5 years ago
6c21e556c4
Clean up work for text python package
5 years ago
599a449e0b
Support processing GNN data
5 years ago
fbdba6e4da
clean pylint
5 years ago
df361d1d26
Change mem layout of string tensor
...
add support for MindRecord and TFRecord
----
optimize tensorshape
optimize tensorshape and FlatIndex
TFRecord and MindRecord support for string tensor
Modify mem layout
Add new constructor
Add method Allocate
Change some GetMutableBuffer usages to AllocateBuffer
5 years ago
58e6d7d950
!1341 Added lookup and vocab to mindspore.dataset.text
...
Merge pull request !1341 from ZiruiWu/vocab_and_lookup
5 years ago
bb51bb88d7
add compress in mindrecord
5 years ago
2e3d55ed87
!1281 Implementation of SplitOp
...
Merge pull request !1281 from Peilin/splitOp
5 years ago
39b9aedf68
!1342 Bug fix on issue Core dump on GPU when train with lenet with AU
...
Merge pull request !1342 from Tinazhang/cc
5 years ago
71e8bb1960
general split case done, chaining sampler (basic case) is working
...
implementation 99% complete
everything and tested except for repeatable shuffling
tested most basic/typical split usecases
cleanup
some more cleanup
fix CI
more ci fix
more ci fixes
more ci fix
more ci fix
more ci fix
added more tests, fixed some bugs
some more clean up and test cases
added shard/shuffle before split warning/error
addressed code review comments and ci
fixed ci
5 years ago
e9e40b688b
Bug fix
5 years ago
25ab2ef303
Implemented lookup and vocab
5 years ago
46949fc327
!1307 Cleanup dataset UT: unskip and enhance TFRecord sharding tests
...
Merge pull request !1307 from cathwong/ckw_dataset_ut_unskip2
5 years ago
451c20a6f5
Add UnicodeCharTokenizer for nlp
5 years ago
93e7c97a96
!1272 [Dataset] MindData Tree Optimizer Infrastructure
...
Merge pull request !1272 from JunhanHu/minddata_opt
5 years ago
b78894e02b
Cleanup dataset UT: unskip and enhance TFRecord sharding tests
5 years ago
f44d213503
MindData optimizer infrastructure.
5 years ago
163b6b7ea7
add jieba c++ code
5 years ago
17cecf2cf5
Added TCs to RandomCrop and RandomCropAndResize and modified visalize() calling
5 years ago
5a914994ba
clean pylint
5 years ago
bcfaff97f9
clean pylint
5 years ago
e8ca243364
-Add DE_STRING
...
-replace switch'case by indexing
- Add test case
- Add constructors
- Add getItem string
- Fix bugs
- Add more tests
- Tensor iterator
- asNumpy
- TextFileDataset
- Tensor(Numpy)
- Super > 2D
- Add more test cases for GeneratorDataset
- Change StartAddr to GetBuffer and GetMutableNuffer
- Raise an error if batch is used with strings
Clean-up work
5 years ago
cb2814b498
flat_map first commit
5 years ago
c680cfbf27
!1157 dataset: add concat operation for dataset
...
Merge pull request !1157 from ms_yan/concat_dataset
5 years ago
ab031ee9ea
!1126 VOCDataset support object detection function
...
Merge pull request !1126 from xiefangqi/voc_support_detection
5 years ago
c937bad53f
minddata support voc
5 years ago
c0fa7b4b19
init commit of concat dataset
...
change to use __add__ operation instead ds.concat
5 years ago
be2e7531ca
fix: MindDataset parameter shard_id & num_shards check
5 years ago
913074e656
Cleanup dataset UT: resolve skipped test units
5 years ago
aa3f89e74f
mindrecord support read file list
5 years ago
49ef53f164
Cleanup dataset UT: util.py internals
5 years ago
2860fd9338
!984 Add unit test case for HWC2CHW.
...
Merge pull request !984 from Tinazhang/hwc2chw
5 years ago
c8b5586c7f
add unit test for HWC2CHWC
5 years ago
58226addd6
Cleanup dataset UT: use md5 npz in test_zip for images
5 years ago
47f5abceb4
!960 Adding example for grayscale
...
Merge pull request !960 from EricZ/grayscale_fix
5 years ago
078dd86cfe
!507 Implemented padded_batch
...
Merge pull request !507 from ZiruiWu/batch_with_padding
5 years ago
de7625777f
!951 fix: MindDataset with columns_name parameter cause errors in some scenes
...
Merge pull request !951 from guozhijian/fix_read_by_columns
5 years ago
0f0548f21b
Added test case for grayscale support
5 years ago
c2d364a573
batch with padding implemented
...
support for 1 specific dimension to be None, added validator
fix various CI complains
another round of CI fixes
ci
refactor parts of the code
code refactor
ci fix
comments added, fix bugs
address review comments
address review comments
review cmts
added simple perf test script
update pad code
perf imprv
5 years ago
d4d236bcce
fix: use MindDataset by column_names get data error in some situation
5 years ago
b520ca9087
fix pk sampler in mindrecord
5 years ago
mindspore-ci-bot
liyong
Amir Lashkari
nhussain
jiangzhiwen
xulei2020
xiefangqi
anzhengqi
tinazhang
islam_amin
avakh
Cathy Wong
Zirui Wu
YangLuo
Jesse Lee
heleiwang
tony_liu2
Danish Farid
peilin-wang
yanghaitao
ms_yan
qianlong
Eric
Alexey Shevlyakov
Jamie Nisbet
hesham
xunxue
Junhan Hu
Mahdi
Jonathan Yan
jonyguo
liuwenhao4
mindspore-ci-bot
jinyaohui
mindspore-ci-bot