2a799fe90e
!6018 Set the number of epochs in mode.train() non-sink Mode
...
Merge pull request !6018 from h.farahat/ms_nonsink_epochs
4 years ago
82bc278e26
minddata use new ps api
4 years ago
19ca9762db
skip graphdata distributed test case when asan mode
4 years ago
477808e622
Enable cache for other leaf ops
4 years ago
0d8015b043
!5300 [MD] Add nD Slice Support
...
Merge pull request !5300 from nhussain/slice_op
4 years ago
b785b7d0ff
!6813 dataset UT: check PIL version, delete test_callback_2maps
...
Merge pull request !6813 from cathwong/ckw_dataset_cleanup8
4 years ago
9cee0d2143
Add num_epochs to non-sink training
4 years ago
f2b07d907a
dataset UT: check PIL version, delete test_callback_2maps
4 years ago
4c0b3c1bd3
!6017 per_batch_map needs to support x input columns and y output columns
...
Merge pull request !6017 from ZiruiWu/per_batch_map_multi_col
4 years ago
e88e114a50
!5930 Cache server phase 2 single node
...
Merge pull request !5930 from Jamie/CacheOp_dev
4 years ago
77d507279c
first working implementation
...
add string implementation, can handle partial string slices
finish core implementation, added Index object
fix up the logic
add in new SliceOption object, cleaning up tests
4 years ago
983827ec5c
Rebase up to 88ded11f59
4 years ago
0cab373223
per_batch_map supports x num_columns to y num_columns
...
decided to use project after batch as well
per_batch_map 1->x colunm works
add more test case
add test cases, project after map
col order hahaah
fix ci
add test cases
addr review cmts
ci
fix build err
fix test case failure
ci
4 years ago
2f3add4acd
!6301 [MD] Combine c++ and python ops in map
...
Merge pull request !6301 from nhussain/c_py_compose
4 years ago
1c6c54ae0f
fix: testcase test_cutmix_batch_op.py slowly
4 years ago
fda9462682
embed python compose op
...
Leaf ops do self-reset.
embed python compose op
add implicit compose op
more tests
4 years ago
f2e34b2eaf
Clean up the view code
4 years ago
53a82fa6e0
!6472 fix get_dataset_size error in CelebaDataset when usage is not 'all'
...
Merge pull request !6472 from yanghaitao/yht_celeba_get_dataset_size
4 years ago
92aecf128f
!6549 [MD] minddata add iterator parameter validation
...
Merge pull request !6549 from xiefangqi/add_output_numpy_valid
4 years ago
4ff4c17632
fix get_dataset_size in CelebADataset when usage is not all
4 years ago
a6360cb2e4
add output_numpy validation to iterator
4 years ago
dd0d9fe9ab
fix ci error
4 years ago
da15495d7b
fix bucket_batch ci error
4 years ago
bef2118253
!6300 [MD] fix get_dataset_size bug when set num_samples in DistributedSampler
...
Merge pull request !6300 from liyong126/fix_md_num_samples
4 years ago
0d486adef1
!3974 Eliminate the pause for each cycle of the Repeat
...
Merge pull request !3974 from lixiachen/repeat_task2
4 years ago
930e85ed5c
fix get_dataset_size in distributedSampler & num_samples
4 years ago
f7adf648e9
dataset API docstring: Update datasets, samplers, graphdata and text
4 years ago
b5b333726f
Leaf ops do self-reset.
4 years ago
653c79b4af
md fix per_batch_map issue
4 years ago
16147669a6
fix num_samples in concatDataset
4 years ago
eb049b0e93
!5842 [Dataset] dataset op log changes
...
Merge pull request !5842 from luoyang/pylint
4 years ago
28e2b0253e
!6086 Order dict by column_id in create_dict_iterator()
...
Merge pull request !6086 from ZiruiWu/order_dict_in_create_dict_iterator
4 years ago
d54ba374b9
order dict by column_id in create_dict_iterator()
...
fix CI
4 years ago
f5a85510f2
[MD] dataset op log changes
4 years ago
96de2ffdb6
fix errors from old compose call in Python
4 years ago
d8c09067ce
!5801 [MD][Feature] minddata iterator output ms_tensor
...
Merge pull request !5801 from xiefangqi/md_modify_output_tensor
4 years ago
5b4642d10a
!4950 Fixing Minddata profiling monitor
...
Merge pull request !4950 from anthonyaje/monitor_fix
4 years ago
9b3c33e157
minddata iterator output ms_tensor
4 years ago
cf7d6eddc4
!6019 [MD] Move Random(Choice/Apply/Order) from vision to transforms module
...
Merge pull request !6019 from nhussain/move_random_choice_apply
4 years ago
a778868a5a
!5929 BucketBatchByLength column issue
...
Merge pull request !5929 from MahdiRahmaniHanzaki/bucket_batch_by_length_fix
4 years ago
f42f9ef9d7
move random apply order and choice ops
4 years ago
92e99ff224
change map calls
4 years ago
2d08d83247
Fixed connector_size and monitor.
4 years ago
1bb93580dd
Add usage flag to MNIST/CIFAR
...
ci fix 1
fix ci 2
fix ci 3
fix ci 4
fix ci 5
fix ci 6
fix ci 7
change var name in CelebA from dataset_type to usage
address review cmts
fix cpp ut fail
change mode to usage in VOCDataset
fix cmts
fix ci test case fail
4 years ago
d0410d6191
md delete set_dataset_size interface
4 years ago
c45f79d36b
!5384 [MD]-Api changes
...
Merge pull request !5384 from nhussain/api_changes
4 years ago
a152a65227
Fixed the column problem in BucketBatchByLength
4 years ago
bde9f18f5a
update lookup api to take in a type
...
ci
add test case
address some of the review cmts
address review cmts
4 years ago
3bac9d3713
switch input columns and operation
...
change ImagefolderDV2 name
change ds.transforms.vision to ds.vision
change batch api to match map api more closely
compose op changes
test_pylint
remove compose op from vision, move to transform module, refactor map and batch to use column_order
4 years ago
d76ac7c6e8
!5648 GetColumnNames for Python
...
Merge pull request !5648 from MahdiRahmaniHanzaki/get-col-name
5 years ago
6e7aab8c4f
Added GetColumnNames for Datasets
5 years ago
4d4c11b133
dataset API docstring cleanup: Standard product terms NumPy, Python
5 years ago
bb84f50407
!5473 optim pylint
...
Merge pull request !5473 from jinyaohui/master
5 years ago
27d5281641
Change epoch count to 1 for python testcases
5 years ago
8f3ebfd469
!5624 Optimize the performance of GraphData.get_neg_sampled_neighbors
...
Merge pull request !5624 from heleiwang/gnn_perf
5 years ago
857cf2f77f
Optimize the negative sampling process
5 years ago
5453b40311
!4477 add dataset compare function to utils
...
Merge pull request !4477 from tony_liu2/staging
5 years ago
7f6782be2a
dataset fixes: Update OneHot API docs; fixup Python UTs
5 years ago
a9972a7def
optim pylint
5 years ago
70bfd506a1
add function
...
pr comment fix
revert graphengine
5 years ago
7aaf4118bf
!5297 CutMixBatch and MixUpBatch fix for 2D one-hot labels
...
Merge pull request !5297 from MahdiRahmaniHanzaki/cutmix-fix
5 years ago
7ad7024c45
Fixed 2D one-hot label problems in CutMix and MixUp
5 years ago
94a4a3f20b
changing affine and posterize apis
5 years ago
e4d2f2fd3b
!5030 Adding a BBox class and fixing Posterize C Op testing.
...
Merge pull request !5030 from islam_amin/bbox_class
5 years ago
20b3134785
!5184 fix: padded dataset when no div and with repeat op
...
Merge pull request !5184 from guozhijian/fix_padded_with_no_div_repeat
5 years ago
f3fd7a5578
!5073 Add checks and exception handling DS callback
...
Merge pull request !5073 from h.farahat/map_callback_end
5 years ago
6df1a43ec2
fix: padded dataset with non div & repeat
5 years ago
346b97acb8
!5156 fix softdvpp coredump
...
Merge pull request !5156 from qianlong21st/master_fix_softdvpp_core
5 years ago
12dfca71f1
fix softdvpp coredump
5 years ago
67f5c89cd6
Check input image type for random posterize
5 years ago
641e751d5e
adding bbox class
5 years ago
a2c38d89f9
Added float32 support for CutMixBatch
5 years ago
8eeceb267b
- release the wait if training is ended
5 years ago
ac81886328
!4916 fix generator_dataset hangs and test_graphdata_distributed.py failing randomly
...
Merge pull request !4916 from heleiwang/gnn_fix_bug
5 years ago
9b503e4f38
!4955 Fixes for Dynamic Augmentation Ops
...
Merge pull request !4955 from MahdiRahmaniHanzaki/dynamic-ops-fix
5 years ago
a5f9b8f92e
Added fix for MixUpBatch and CutMixBatch and for RandomAffine
...
updated c color op descriptions
5 years ago
9d7250c483
!4776 Introduce 2 extra ctrl flags to DataBuffer in dataset, address remaining cmts to PR4632
...
Merge pull request !4776 from ZiruiWu/map_callback_follow_up
5 years ago
74c1e6da60
introducing pause and quit flags to DataBuffer
...
fix review cmts
fix ci
fix Ci
fixci
address ci
ci
- add timeout
- add more test cases
fix CI
address review cmts
5 years ago
82c888f065
!4930 Fix CI cifar hang issue
...
Merge pull request !4930 from xiefangqi/fix_cifar_nofile_issue
5 years ago
e3e7820413
fix cifar stuck problem
5 years ago
4870abc848
1. fix generator_dataset hangs
...
2. fix test_graphdata_distributed.py failing randomly
5 years ago
5b4b539751
fix: concat with none sample dataset
5 years ago
ed954dc407
!4782 Python UT fails on RandomPosterize
...
Merge pull request !4782 from islam_amin/bbox_class
5 years ago
85357238fe
Fixing posterize python ut
5 years ago
8a08d0c37b
Phase 2 of CacheOp
5 years ago
78c1aa1d96
Implemented Callback for Dataset
...
implment pause in MapOp, added more to callback
add ds_callback
- Initial drop of Python DSCallback
- Pybind DSCallback
- Pybind DSCallback
added callback to mapOp
- de_pipeline DSCallback
- de_pipeline DSCallback
add test case, segfault for now
fix seg fault
- de_pipeline DSCallback
remove 1 line
update callback test case, now works
use builder class for mapOp callback
- de_pipeline DSCallback
- de_pipeline DSCallback
- de_pipeline DSCallback
better test case
minor fix
add comments and minor clean ups
get rid of nullptr in MapOp, use other flag instead
fix a bug ParseMapOp only takes 1 callback
- Added WaitedDSCalabck
refactor callback param
fix text case incorrect number
- added testing
fix cpp test case
- added testing
- revert back lenet changes
- cleanup test_callbacks.py
- cleanup test_callbacks.py
fix CI stage I
fix CI stage II
fix CI and update epoch counter
- add validation
- add more testing test_callbacks.py
use random data op to do tests
adjust when to call EpochBegin/End
- add repeat with callback
- addressing reviewers' comments
- docstring and CI fixes
- docstring and CI fixes
- docstring and CI fixes
- rebase with upstream/master
fix cpp test case
fix review comments
addr review cmts, add test case
5 years ago
256dccc651
!4498 Gnn data processing supports distributed scenarios
...
Merge pull request !4498 from heleiwang/gnn_distributed
5 years ago
8ee4d8e92d
Gnn data processing supports distributed scenarios
5 years ago
50b5c8035b
!4701 fix concat get_dataset_size error
...
Merge pull request !4701 from guozhijian/fix_concat_get_dataset_size_err
5 years ago
738ae2c78d
change api to use std_vector
...
try to display image
5 years ago
979111d0ef
Added Posterize Op
5 years ago
b0b370336f
fix: concat get_dataset_size error
5 years ago
5f294fb90d
fix: random affine parameter check
5 years ago
113ff6ca24
!4367 CutMixBatch Augmentation Op
...
Merge pull request !4367 from MahdiRahmaniHanzaki/cutmix
5 years ago
3ecc53fb4e
Added CutMix
5 years ago
8bb4449fa8
add testcase for padded dataset with decode op
5 years ago
c410c6a6e2
soft_dvpp_decode_resize_jpeg_op issue resolved, input size can be an int, break the constraints of input size being sequence (height, width)
5 years ago
8526d5414d
RandomColor
5 years ago
7a10e07fc6
!4383 Add distributedSampler to concatOP
...
Merge pull request !4383 from genglishuai/genglshOK
5 years ago
4d4df68d1e
add DistributedSampler for Concat op
5 years ago
ed1244558c
!4366 Fixing param type in c++ api and fixing python validator for Repeat Op
...
Merge pull request !4366 from TinaMengtingZhang/cpp-api-repeat-count
5 years ago
0868720eb7
fix parameter type for repeat op in c++ api and added c++/python ut.
5 years ago
3e31ac6d62
all Dataset support get_dataset_size
5 years ago
9276a9d2fb
!3771 C++ API Support for TextFile Dataset
...
Merge pull request !3771 from cathwong/ckw_c_api_textfile_nosampler
5 years ago
477528de7f
random sharpness cpp op support
5 years ago
7766efd58e
add files
5 years ago
7f39b5cfd7
C++ API Support for TextFile Dataset and Unit Tests
5 years ago
c2997845c2
!4289 Fix pad op validate issue
...
Merge pull request !4289 from xiefangqi/fix_pad_validate
5 years ago
7ee1e46455
fix pad op validate issue
5 years ago
e9c4378ed8
add support of soft dvpp op
5 years ago
266df56570
!4220 [MD] add num_samples support in minddataset pk sampler
...
Merge pull request !4220 from liyong126/minddataset_pk_num_samples
5 years ago
e52d85ba9d
Added RandomAffine Op
5 years ago
9ad82f79fd
!3979 Adding MixUp operation
...
Merge pull request !3979 from MahdiRahmaniHanzaki/mixup
5 years ago
7341421d3b
fix num samples in pk sampler
5 years ago
3d7826b6df
test five crop problem
5 years ago
1896950ae5
Added Mixup
5 years ago
e19d382473
!3346 Maintain epoch/repeat count for ops
...
Merge pull request !3346 from lixiachen/repeat_rework
5 years ago
ac85b77b76
Maintain epoch/repeat count for ops
5 years ago
7ec0b5857a
!3977 [MD] Refactor Concatenate Op
...
Merge pull request !3977 from nhussain/multi_dim_concat_2
5 years ago
61769b2dd9
fix concat
5 years ago
27e51c5a63
!3553 Unify np array comparisons
...
Merge pull request !3553 from tony_liu2/staging
5 years ago
388e43f7cb
delete parameter block_reader for MindDataset
5 years ago
6017b050ac
add generatordataset support split
5 years ago
269b477684
use np.testing.assert instead of asserting
...
fix line lengths
fix spacing
reverse a few changes
5 years ago
d06da1d270
!3603 Check that the number columns of names and default matches
...
Merge pull request !3603 from jiangzhiwen/fix_column_names_exceeded
5 years ago
2cc6b5cb52
fix number of columns not match
5 years ago
1906ed1be8
fix cache description again
5 years ago
b6529b9096
fix cache description
5 years ago
620d951e5e
Merge branch 'ql/fix_cache_core' of https://gitee.com/qianlong21st/mindspore into ql/fix_cache_core
5 years ago
9c06a564d1
!2983 Dataset Tensor class cleanup
...
Merge pull request !2983 from h.farahat/tensor_class
5 years ago
bf87b1d1fb
fix cache op core dump
5 years ago
0a2980ca74
!3440 [MD]Fix Segementation Falut when SentencepieceTokenizer Op before zipOp and ConcatOp
...
Merge pull request !3440 from xulei/sentence_piece0715
5 years ago
adfbc891d3
- Add checks and testing for empty tensors
...
- cleanup work on createTensor and Tensor's constructors
5 years ago
06ed9ffd6a
!3183 Remove files on test fail for test_minddataset*.py
...
Merge pull request !3183 from tony_liu2/master
5 years ago
0af6d75761
add code
5 years ago
d28c63b6c0
fix diff workers disallow cache
5 years ago
21a5f06e93
!3328 Add testcases
...
Merge pull request !3328 from shenwei41/sw_master
5 years ago
1f4251a440
add try to minddataset python tests to remove files on fail
...
delete unrelated file
change finally to except else
fix spacing alignment
fix indentation
fix indentation
fix indentation
add fix to new test case
add if main for pytest
fix spacing
5 years ago
859fe6bc41
!3342 C++ API Support for Skip Dataset Op plus UTs
...
Merge pull request !3342 from cathwong/ckw_c_api_skip
5 years ago
df1300d9cb
!3294 Refactor opt/pre passes
...
Merge pull request !3294 from nsyca/removal_pass
5 years ago
8921a6099c
C++ API Support for Skip Dataset Op and UTs
5 years ago
c24122bb55
!3303 fix validator that checks the vocab_size is positive in vocab.from_file
...
Merge pull request !3303 from ZiruiWu/fix_vocab_validator
5 years ago
9fb1904ed8
Refactoring opt/pre
5 years ago
4eaa396ca7
Add testcases
5 years ago
6f186aafa0
Update Python UA op, AutoContrast Op and UT
...
Updated Validators.py
Fixed UA UT
Fixed UA UT
Fixed PyLint Errors
Added aditional changes
remove old golden AutoContrast UT file
updated Autocontrast python UT
Added golden files
Fixed issue
fixed UT
5 years ago
40ea84ae81
fix from_file validator minor error, 1 line change
...
add test case
5 years ago
c451146b14
!3255 Added cpp Equalize op to vision api
...
Merge pull request !3255 from alashkari/cpp_ops
5 years ago
5aad4bdfbf
change size
5 years ago
64a6ed56de
fix annotation
5 years ago
ad651f38bf
!3124 [MD] error occur when using numpy types
...
Merge pull request !3124 from liyong126/fix_numpy_generic
5 years ago
bfc18704d5
!3254 [MD]Add more strict upper bounds to CenterCropOp and RandomCropWithBboxOp
...
Merge pull request !3254 from nhussain/padding_upperbound
5 years ago
0e27dccbcf
!3093 VOCDataset output change to multi-columns
...
Merge pull request !3093 from xiefangqi/md_voc_multi_columns
5 years ago
28ebd73096
fix bug when int or float is numpy type
5 years ago
572750cd40
Added cpp Equalize op
...
adding missed files
Fixed Errors
fixed bugs
fixed bugs
fixed bugs
fixed bugs
fixed bugs
Updated Python UT for Equalize op
Added cpp Equalize op
adding missed files
Fixed Errors
fixed bugs
fixed bugs
fixed bugs
fixed bugs
fixed bugs
Updated Python UT for Equalize op
Fixed comment style
update files
deleted files
added files
added image_utils.h
Fixed PyLint and CPPLint Errors
updated tensor_op.h
5 years ago
b064cbab5d
fix upper bound
5 years ago
b091f74c23
!3016 Add CSV dataset loader
...
Merge pull request !3016 from jiangzhiwen/dataset/csv
5 years ago
2f506b7985
CSV dataset loader
5 years ago
fa7fa8a162
!3234 [MD] fix bug when save tfrecord data
...
Merge pull request !3234 from liyong126/fix_save_tfrecod
5 years ago
f521532a06
fix field_name probelem from tfrecord to mindrecord
5 years ago
18b519ae0f
add sentence piece
5 years ago
fbd9e3d402
vocdataset output multi-columns
5 years ago
008b91b2a1
inject epoch ctrl op in the execution tree and send eos at the end of epoch
5 years ago
4945d34a41
!3133 [MD] fix num_sample in distributed sampler
...
Merge pull request !3133 from liyong126/md_num_of_samples
5 years ago
a84092e89d
!3102 [MD] Add additional parameter checks in RandomCropWithBBoxOp and RandomResizeWithBbox
...
Merge pull request !3102 from nhussain/random_crop_fixes
5 years ago
e2ea1fa0df
activate num_samples in distributed samplers
5 years ago
1bcbbb08d6
!2973 [MD] Add implementation of SlidingWindow TensorOp, c++ & python
...
Merge pull request !2973 from luoyang/update
5 years ago
1a518e1cff
!3144 [MD] Fix ResizeOp behaviour
...
Merge pull request !3144 from nhussain/dataset_check
5 years ago
c984b0fc88
fix size tuple
5 years ago
35c0f1d2e6
!3143 Add python UT to MnistDataset op
...
Merge pull request !3143 from TinaMengtingZhang/dataset-op-ut-mnist
5 years ago
ab37e87d7a
adding Mnist python ut coverage
5 years ago
5647889c0d
Added AutoContrast Op
5 years ago
37e8439c0a
!3077 supporting cpp invert operation
...
Merge pull request !3077 from ava/invert
5 years ago
c07a8d2033
!3137 Cleanup dataset UT: Remove deprecated save_and_check
...
Merge pull request !3137 from cathwong/ckw_dataset_ut_cleanup7
5 years ago
5871ba8a05
more validator fixes
5 years ago
bc676fe250
save op in minddataset
5 years ago
35c3a63701
support cpp invert operation
5 years ago
04eb4f89c7
Cleanup dataset UT: Replace save_and_check
5 years ago
d233c54139
implemented cpp random choice, apply and compos
...
python part of random ops
added random select sub policy
validators added
comments added, remaining issues addressed
add more python test cases
fix ci
fix CI
fix order of include files
addr review cmts
addr review cmts
reorg file
fix compile err
address review cmts
address review cmts
5 years ago
4136892a3e
add SlidingWindow Op
5 years ago
bc0a53cfb1
!3030 minddata support ps-lite feature
...
Merge pull request !3030 from xiefangqi/md_support-pslite
5 years ago
ba0143402c
!3067 Cleanup dataset UT: Remove unneeded tf data files and tests
...
Merge pull request !3067 from cathwong/ckw_dataset_ut_cleanup6
5 years ago
219a716eae
!3066 fix some batch's get_dataset_size and some text validator inconsistency
...
Merge pull request !3066 from ZiruiWu/fix_validator
5 years ago
bed93a9ead
!3039 [Dataset Python UT] adding coverage to Cifar Op and update test dataset
...
Merge pull request !3039 from TinaMengtingZhang/dataset-op-ut-cifar
5 years ago
63185cb20f
fix some validators errors
...
address review cmts
addr review cmts
5 years ago
ae1ed327ba
Cleanup dataset UT: Remove unneeded data files and tests
5 years ago
05b2a57d2a
fix validation errors, and fix try catch error tests
5 years ago
340d98a4d1
added test case to cifar_op
...
update cifar10 dataset
fixing missing error handling code in validator
5 years ago
add19a591c
md support ps-lite
5 years ago
48654be14d
!3040 BoundingBoxAugment op fix ratio
...
Merge pull request !3040 from islam_amin/bounding_box_augment_bugs
5 years ago
56da3b0ae1
Fixing ratio bug with BoundingBoxAugment
5 years ago
7c1bc5192b
Initial Drop of CacheOp Phase I
5 years ago
6284c42a76
!2941 MD tokenizer support output offsets
...
Merge pull request !2941 from xiefangqi/md_add_offsets_feature
5 years ago
47060631e5
add offsets feature to tokenizer
5 years ago
541456044d
!2934 GraphData support random_walk and get_edge_feature
...
Merge pull request !2934 from heleiwang/gnn_july
5 years ago
2f565f4c20
!2776 Update API of dataset.text.lookup
...
Merge pull request !2776 from ZiruiWu/master
5 years ago
7a046a1d70
support get_edge_feature
5 years ago
7b15e5a742
rework on lookup
...
add test caser
fix ci
address review cmts
ci
addr review cmt
fix typo
address review cmts
add 2 more test cases
cpplint fix
addr cpplint
addr ci
fix tst case err
fix doc str
5 years ago
2c7fd248f8
fixes for PR-2908: avoid empty strings for column names
5 years ago
6c37ea3be0
fix validators
...
fixed random_apply tests
fix validators
fixed random_apply tests
fix engine validation
5 years ago
915ddd25dd
!2892 gnn random walk pr 1977 comment follow up
...
Merge pull request !2892 from tony_liu2/master
5 years ago
mindspore-ci-bot
xiefangqi
jonyguo
Lixia Chen
hesham
Cathy Wong
nhussain
Zirui Wu
shenwei41
yanghaitao1
liyong
YangLuo
anthonyaje
mahdi
heleiwang
jinyaohui
tony_liu2
islam_amin
qianlong
Jesse Lee
mayang
Alexey Shevlyakov
genglishuai
tinazhang
anzhengqi
avakh
dengyutao
jiangzhiwen
guansongsong
xulei2020
Nat Sutyanyong
Amir Lashkari
panfengfeng