bb2082daec
fix: MindDataset padded log error
5 years ago
3085e51e45
!1922 check parameter num_samples of sampler
...
Merge pull request !1922 from yanghaitao/yht_check_num_samples_sampler
5 years ago
1b5d9a1194
!1981 Implemented from_dataset, a way to build a vocab from existing dataset
...
Merge pull request !1981 from ZiruiWu/from_dataset_dev
5 years ago
880ce5ea26
implemented from_dataset
...
fix complie error
more tests
address CI complains
fix ci
adress review comments
address review cmts
5 years ago
4e333a2f22
MindData profiler infrastructure
5 years ago
0a95223f25
!1910 RandomColorAdjust error for grayscale images
...
Merge pull request !1910 from MahdiRahmaniHanzaki/I1J9SQ-random-color-adjust-bug
5 years ago
8f16cff12e
add para check for sampler
5 years ago
1de7271afc
add floatxx test case
5 years ago
bc22c172b8
add TensorDataset and its ut
5 years ago
251a6667a5
!1989 fix MindDataset distribute shuffle error
...
Merge pull request !1989 from guozhijian/fix_distribute_shuffle
5 years ago
b3da41bd7a
!1913 CocoDataset implementation
...
Merge pull request !1913 from xiefangqi/xfq_support_coco
5 years ago
097cbcc0cb
CocoDataset implementation
5 years ago
96859f44b6
fix: MindDataset distribute shuffle bug
5 years ago
ba2e2dced4
fixed some warnings and typos
5 years ago
2812cd57db
fix pylint error
5 years ago
fe0c3e9a8f
!1908 Add UT to six data vision transforms ops
...
Merge pull request !1908 from Tinazhang/ut-normalize
5 years ago
3ece8dd090
1. support get_all_edges, get_nodes_from_edge, get_sampled_neighbors, get_neg_sampled_neighbors and graph_info API
...
2. mod cora and citeseer conversion script
5 years ago
5eb95599f6
!1874 Updates to string tensor
...
Merge pull request !1874 from h.farahat/string_tensor_updates
5 years ago
c3de5c7a54
adding new TCs to six ops and skip md5 case in RandomPerspective
5 years ago
f837ddc956
- Bug when empty strings sent to Python
...
- Support accepting Numpy of str as input
- Support batching strings
- Core logic of batch&pad is static
- Make Pad a utility function
5 years ago
feff8899ac
support padding samples
5 years ago
5c21616293
!1807 Implemented Ngram TensorOp for dataset
...
Merge pull request !1807 from ZiruiWu/ngram_dev
5 years ago
dbf9936ec4
Implemented n-gram for dataset TensorOp
5 years ago
51bc0c0460
consistent design for num_samples
...
updates
more update
more work
more fixin
post rebase updates
clang formatting
code review recovery
ci fixes
updates
update
update
update
5 years ago
33fe66fd1e
fix error handling for bigger crop size case
5 years ago
5522edc4b5
!1713 [Dataset] Clean pylint.
5 years ago
7b911886ec
Fixing some tiny faults about Pylint in my code(ops)
5 years ago
7878743400
!1638 fixed rounding edge case in split
...
Merge pull request !1638 from Peilin/splitOp-after-testing
5 years ago
0e3dd8149a
!1703 Adding sync_wait input check
...
Merge pull request !1703 from EricZ/master
5 years ago
c366b3fb18
!1704 fix num rows bug in sampler
...
Merge pull request !1704 from Peilin/fix-random-sampler-bug
5 years ago
8e1b2107ce
Added example for sync_wait bad batch_num
...
Added broken test case for cv
Fixed error handling in sync_wait
5 years ago
976b4793d1
!1544 Python Unit Test for TenCrop
...
Merge pull request !1544 from MahdiRahmaniHanzaki/I1IHNG-ten-crop
5 years ago
3ef3d1433d
fixed num_samples bug for RandomSampler
5 years ago
f4a61fdbb2
Adding new UT cases and missing TCs in PY Ops.
5 years ago
5469be2a97
fixed bug for split, RandomSampler and some other cleanup
...
add another test case
typo
merge conflict
another PR changed testing behavior, updated test cases in this commit
added input check for use_sampler
addressed code review comments
fixed pylint, not related to my changes
fixed edge case of rounding in getting split sizes
fix pylint
5 years ago
dfc097019b
Added TenCrop test
...
Added unit tests for both testing the functinality of the TenCrop and
its error messages. Due to the similarity of this method to FiveCrop the
test cases are similar to FiveCrop test cases.
Signed-off-by: Mahdi <mahdi.rahmani.hanzaki@huawei.com>
added error_msg function call in the main method
refactored the test and added visual representation of the results
Separated the two error cases into two different functions and used the
visualize function in util.py to plot the result of TenCrop.
Signed-off-by: Mahdi <mahdi.rahmani.hanzaki@huawei.com>
Added new test cases
Added new test cases including test case for checking the error message
when the size variable is not a positive integer, test case for
rectangle crop, test case for vertical flip setting, and testing for
similarity of the result of TenCrop for the same input data in different runs.
Signed-off-by: Mahdi <mahdi.rahmani.hanzaki@huawei.com>
changed visualize in test_five_crop
Changed the visualize function in test_five_crop to use the already
existing function in util.py
Signed-off-by: Mahdi <mahdi.rahmani.hanzaki@huawei.com>
made generate_golden variable global
5 years ago
21da86b393
!1591 add get_dataset_size for CelebADataset
...
Merge pull request !1591 from yanghaitao/yht_celeba_get_dataset_size
5 years ago
cc6c7a3f60
add get_dataset_size to celebadataset
5 years ago
0f4dc13e8e
!1478 [Dataset] Fix CI warnings, clean pylint.
...
Merge pull request !1478 from luoyang/pylint
5 years ago
9b2a778d94
!1478 [Dataset] clean pylint.
5 years ago
5e4728c50f
add voc support split
5 years ago
d4c93575e7
fixed bug for split, RandomSampler and some other cleanup
...
add another test case
typo
merge conflict
another PR changed testing behavior, updated test cases in this commit
added input check for use_sampler
addressed code review comments
fixed pylint, not related to my changes
5 years ago
abca62f407
10:00 26/5 clean pylint
5 years ago
0a8ef2fe72
!1455 Cleanup dataset UT: restore config support
...
Merge pull request !1455 from cathwong/ckw_dataset_config_util
5 years ago
f891e1755c
Cleanup dataset UT: restore config support
5 years ago
aeb4c52f2d
!1379 Creating new UTs to three vision transformation ops
...
Merge pull request !1379 from Tinazhang/linearTrans
5 years ago
718a96430f
!1389 Security issues in recent string tensor PR
...
Merge pull request !1389 from h.farahat/fix_security_issues
5 years ago
a6b8451a33
!1380 make ShuffleOp have deterministic behavior for subsequent epochs
...
Merge pull request !1380 from Peilin/shuffle-subsequent-epoch-deterministic
5 years ago
f28f883cac
fix comments error and modify parameter check
5 years ago
c0f0e154f5
Clean up work for text python package
5 years ago
41456ac824
!1369 dataset: delete StorageDataset
...
Merge pull request !1369 from ms_yan/del_storage
5 years ago
7322839b04
add UTs for LinearTransformation, ToPIL, ToType
5 years ago
0cbcc7200b
made shuffle determinisitc for subsequent epochs
5 years ago
61639be1e0
!1375 Cleanup dataset UT: remove num_parallel_workers=1 in test_exception
...
Merge pull request !1375 from cathwong/ckw_dataset_ut_cleanup5
5 years ago
a528797253
!1377 Code Fix for Uniform Augmentation
...
Merge pull request !1377 from Tinazhang/cc
5 years ago
458436186c
!1365 Clean up work for text python sub-package
...
Merge pull request !1365 from h.farahat/text_namespace
5 years ago
b390883c6a
Bug fix
5 years ago
702005d403
Cleanup dataset UT: remove num_parallel_workers=1 in test_exception
5 years ago
d5e896b51c
delete storageDataset Op API and its test case
5 years ago
6f733ec113
!1308 Stage 2 of adding support for string Tensor
...
Merge pull request !1308 from h.farahat/string_tensor2
5 years ago
34236ce1f1
fix pylint
5 years ago
3363d4e834
!1249 Add GNN dataset processing API
...
Merge pull request !1249 from heleiwang/hlw_gnn_data
5 years ago
6c21e556c4
Clean up work for text python package
5 years ago
599a449e0b
Support processing GNN data
5 years ago
fbdba6e4da
clean pylint
5 years ago
df361d1d26
Change mem layout of string tensor
...
add support for MindRecord and TFRecord
----
optimize tensorshape
optimize tensorshape and FlatIndex
TFRecord and MindRecord support for string tensor
Modify mem layout
Add new constructor
Add method Allocate
Change some GetMutableBuffer usages to AllocateBuffer
5 years ago
58e6d7d950
!1341 Added lookup and vocab to mindspore.dataset.text
...
Merge pull request !1341 from ZiruiWu/vocab_and_lookup
5 years ago
bb51bb88d7
add compress in mindrecord
5 years ago
2e3d55ed87
!1281 Implementation of SplitOp
...
Merge pull request !1281 from Peilin/splitOp
5 years ago
39b9aedf68
!1342 Bug fix on issue Core dump on GPU when train with lenet with AU
...
Merge pull request !1342 from Tinazhang/cc
5 years ago
71e8bb1960
general split case done, chaining sampler (basic case) is working
...
implementation 99% complete
everything and tested except for repeatable shuffling
tested most basic/typical split usecases
cleanup
some more cleanup
fix CI
more ci fix
more ci fixes
more ci fix
more ci fix
more ci fix
added more tests, fixed some bugs
some more clean up and test cases
added shard/shuffle before split warning/error
addressed code review comments and ci
fixed ci
5 years ago
e9e40b688b
Bug fix
5 years ago
25ab2ef303
Implemented lookup and vocab
5 years ago
46949fc327
!1307 Cleanup dataset UT: unskip and enhance TFRecord sharding tests
...
Merge pull request !1307 from cathwong/ckw_dataset_ut_unskip2
5 years ago
451c20a6f5
Add UnicodeCharTokenizer for nlp
5 years ago
93e7c97a96
!1272 [Dataset] MindData Tree Optimizer Infrastructure
...
Merge pull request !1272 from JunhanHu/minddata_opt
5 years ago
b78894e02b
Cleanup dataset UT: unskip and enhance TFRecord sharding tests
5 years ago
f44d213503
MindData optimizer infrastructure.
5 years ago
163b6b7ea7
add jieba c++ code
5 years ago
17cecf2cf5
Added TCs to RandomCrop and RandomCropAndResize and modified visalize() calling
5 years ago
5a914994ba
clean pylint
5 years ago
bcfaff97f9
clean pylint
5 years ago
e8ca243364
-Add DE_STRING
...
-replace switch'case by indexing
- Add test case
- Add constructors
- Add getItem string
- Fix bugs
- Add more tests
- Tensor iterator
- asNumpy
- TextFileDataset
- Tensor(Numpy)
- Super > 2D
- Add more test cases for GeneratorDataset
- Change StartAddr to GetBuffer and GetMutableNuffer
- Raise an error if batch is used with strings
Clean-up work
5 years ago
cb2814b498
flat_map first commit
5 years ago
c680cfbf27
!1157 dataset: add concat operation for dataset
...
Merge pull request !1157 from ms_yan/concat_dataset
5 years ago
ab031ee9ea
!1126 VOCDataset support object detection function
...
Merge pull request !1126 from xiefangqi/voc_support_detection
5 years ago
c937bad53f
minddata support voc
5 years ago
c0fa7b4b19
init commit of concat dataset
...
change to use __add__ operation instead ds.concat
5 years ago
be2e7531ca
fix: MindDataset parameter shard_id & num_shards check
5 years ago
913074e656
Cleanup dataset UT: resolve skipped test units
5 years ago
aa3f89e74f
mindrecord support read file list
5 years ago
49ef53f164
Cleanup dataset UT: util.py internals
5 years ago
2860fd9338
!984 Add unit test case for HWC2CHW.
...
Merge pull request !984 from Tinazhang/hwc2chw
5 years ago
c8b5586c7f
add unit test for HWC2CHWC
5 years ago
58226addd6
Cleanup dataset UT: use md5 npz in test_zip for images
5 years ago
47f5abceb4
!960 Adding example for grayscale
...
Merge pull request !960 from EricZ/grayscale_fix
5 years ago
078dd86cfe
!507 Implemented padded_batch
...
Merge pull request !507 from ZiruiWu/batch_with_padding
5 years ago
de7625777f
!951 fix: MindDataset with columns_name parameter cause errors in some scenes
...
Merge pull request !951 from guozhijian/fix_read_by_columns
5 years ago
0f0548f21b
Added test case for grayscale support
5 years ago
c2d364a573
batch with padding implemented
...
support for 1 specific dimension to be None, added validator
fix various CI complains
another round of CI fixes
ci
refactor parts of the code
code refactor
ci fix
comments added, fix bugs
address review comments
address review comments
review cmts
added simple perf test script
update pad code
perf imprv
5 years ago
d4d236bcce
fix: use MindDataset by column_names get data error in some situation
5 years ago
b520ca9087
fix pk sampler in mindrecord
5 years ago
772e6c1461
Cleanup dataset UT: test_batch, save_and_check support
5 years ago
36fffb7706
Added example md5 generation
...
Comparison example
Added md5 and comparison example for py_transforms
Added md5 check for images
5 years ago
83c68ca2ef
Skip pyfunc test case
5 years ago
26cb3e8a5f
Added test function to show that seed doesn't work.
...
Added testcase to show that c image aug don't use seed properly
Added passing test cases
Added working testcases for using seed
Added additional test cases to show seed use
Added test case for seed
5 years ago
c56fe3aa2d
modify take op with an operator
5 years ago
8af10eb51e
!875 Reject python OP in operations argument for C++ uniform augmentation OP
...
Merge pull request !875 from AdelShafiei/ua_py
5 years ago
d15bd04bfe
added input validation to reject python op in C++ uniform augmentation operations list
5 years ago
a606c2e4da
!872 [Dataset] Add schema support for GeneratorDataset
...
Merge pull request !872 from JunhanHu/generator_schema
5 years ago
2303453753
!869 Random data op
...
Merge pull request !869 from JesseKLee/random_data_op
5 years ago
c5a8ffe4f4
Add schema support for GeneratorDataset
5 years ago
5236d0c3c0
Replace print with logger.info
5 years ago
8d3695f666
!672 Added UT for uniform augmentation C++ OP
...
Merge pull request !672 from AdelShafiei/ua_ut
5 years ago
270bf831a9
Random Data Op
5 years ago
34bfa2f7c9
fix skip
5 years ago
3322e65da9
added ut for uniform augment C++ op
5 years ago
b37db1edf5
!603 [MD] update pk sampler in minddataset
...
Merge pull request !603 from liyong126/update_pk_sampler
5 years ago
f82e63fecc
!671 Added testcase for sync_wait
...
Merge pull request !671 from EricZ/master
5 years ago
0e3054d527
!466 Deepcopy problem when pyfunc cannot be pickled
...
Merge pull request !466 from h.farahat/deepcopy
5 years ago
bfba630aa2
update pK_sampler
5 years ago
8c3931cf1d
fix first epoch always shuffle with default seed in random sampler
...
fix ut
5 years ago
2d115cd04e
Added example for multiple iterator
...
Added new testcase for multi iterator
Addressing review
Fixed typo
5 years ago
a9e9266149
Deepcopy problem when pyfunc cannot be pickled
5 years ago
aad5771a62
!524 Added support for UA augmentation ops with tests
...
Merge pull request !524 from alashkari/ua-ops
5 years ago
56e7a7deb5
Added UniformAugment + Python Augmentation Ops
5 years ago
dc0491caf9
!508 [Dataset] Adding sync_wait operator for dataset
...
Merge pull request !508 from EricZ/master
5 years ago
cd94518769
X# This is a combination of 2 commits.
...
Initial commit for dataset op python
Added signature to barrier
Adde compiling barrier code
Rebasing, fixed new compile errors
Final fix for make_unique
Added pybind API for barrier
Fixed pyfunc invocation
python interface - sync_wait
!1 sync_wait python interface
* python interface - sync_wait
fix test
update test
update test
Added new test case
add test case
test for shuffle + batch
Added two-sync test case
Restrited that no shuffle after sync
Added sync to pipeline info
block first databuffer as well
Intelligently get batch size
Fix default case
Lock Pair shares among all iterators
Added fix for empty character
Fixed up test case formatting
Fix end of epoch in sync_wait
Fixing CI
5 years ago
78001ac9e6
Add multiprocessing support for Mindspore.Dataset.GeneratorDataset
5 years ago
fb18671b28
!506 [Dataset] Multiprocessing support for Pyfunc
...
Merge pull request !506 from JunhanHu/multiprocess_pyfunc
5 years ago
b13e7bc31a
Add python multiprocessing support for Mindspore.dataset
5 years ago
db80f4ff92
The num_samples and numRows in schema for TFRecordDataset are conflict
5 years ago
d9e4dcc33b
!483 Optimize skip dataset op
...
Merge pull request !483 from jiangzhiwen/dataset/skip_opt
5 years ago
f1542a90a3
add pk sampler
5 years ago
e1b109e8b8
optimize skip dataset op
5 years ago
60df369100
Fixup py Normalize doc: takes input CHW
5 years ago
6369cf27bd
!406 added first row crc check for when reading tfrecord files
...
Merge pull request !406 from Peilin/first-row-crc-check
5 years ago
98fbd30a5b
!460 [Data]Add filter operation
...
Merge pull request !460 from xulei/filter_master
5 years ago
822a3160e4
!404 [Dataset] Add Python Sampler support for CPP dataset
...
Merge pull request !404 from JunhanHu/python_sampler_for_cpp_dataset
5 years ago
c705ea5e5b
add filterOp code
5 years ago
9bc2134cb7
added checking of first row crc to find invalid tfrecord files
...
addressed code review comments. added check in python layer to exclude directories and to raise an error if a pattern does not match any file
fixed clang format
fixed cppcheck
fixed cppcheck (used std::accumulate and std::copy_if). regenerated tfrecord file to contain correct header, it was a dummy header before
fixed cppcheck: added const reference for string parameter for lambdas, fixed clang format: whitespace adjustments
more clang whitespace fixes...
changed print to logger.info
5 years ago
2795e492ff
TextFileDataset
5 years ago
43a2e99833
Add python sampler support for CPP dataset
5 years ago
f0c07c3fa6
Realize take op and add ut
5 years ago
80333e9f55
!435 Fix dataset serialize and deserialize for MindDataset
...
Merge pull request !435 from anthonyaje/fix_serdes_dataset
5 years ago
40f0a4a4f4
!333 Add skip op to Dataset
...
Merge pull request !333 from jiangzhiwen/dataset/skip
5 years ago
9e1b5efd1d
!434 Bug in cleaning dataset iterators
...
Merge pull request !434 from h.farahat/multi_itr_bug
5 years ago
ea297c0889
Fix dataset serdes for MindDataset
5 years ago
3c02c82771
Bug in weak reference.
...
Add new test cases
5 years ago
3f7054dccb
add skip dataset op
5 years ago
jonyguo
mindspore-ci-bot
Zirui Wu
Alexey Shevlyakov
yanghaitao
ms_yan
xiefangqi
Mahdi
Tinazhang
heleiwang
hesham
liyong
Jamie Nisbet
Yang
liuwenhao4
mindspore-ci-bot
Eric
Peilin Wang
Cathy Wong
jinyaohui
jonwe
qianlong
Junhan Hu
xulei2020
jiangzhiwen
mindspore-ci-bot
Adel Shafiei
Jesse Lee
Amir Lashkari
anthonyaje
jzw