Commit Graph

9 Commits

Author SHA1 Message Date
xingjun.wxj
e02a260c93 Refactor the task_datasets module
Refactor the task_datasets module:

1. Add new module modelscope.msdatasets.dataset_cls.custom_datasets.
2. Add new function: modelscope.msdatasets.ms_dataset.MsDataset.to_custom_dataset().
2. Add calling to_custom_dataset() func in MsDataset.load() to adapt new custom_datasets module.
3. Refactor the pipeline for loading custom dataset: 
	1) Only use MsDataset.load() function to load the custom datasets.
	2) Combine MsDataset.load() with class EpochBasedTrainer.
4. Add new entry func for building datasets in EpochBasedTrainer: see modelscope.trainers.trainer.EpochBasedTrainer.build_dataset()
5. Add new func to build the custom dataset from model configuration, see: modelscope.trainers.trainer.EpochBasedTrainer.build_dataset_from_cfg()
6. Add new registry function for building custom datasets, see: modelscope.msdatasets.dataset_cls.custom_datasets.builder.build_custom_dataset()
7. Refine the class SiameseUIETrainer to adapt the new custom_datasets module.
8. Add class TorchCustomDataset as a superclass for custom datasets classes.
9. To move modules/classes/functions:
	1) Move module msdatasets.audio to custom_datasets
	2) Move module msdatasets.cv to custom_datasets
	3) Move module bad_image_detecting to custom_datasets
	4) Move module damoyolo to custom_datasets
	5) Move module face_2d_keypoints to custom_datasets
	6) Move module hand_2d_keypoints to custom_datasets
	7) Move module human_wholebody_keypoint to custom_datasets
	8) Move module image_classification to custom_datasets
	9) Move module image_inpainting to custom_datasets
	10) Move module image_portrait_enhancement to custom_datasets
	11) Move module image_quality_assessment_degradation to custom_datasets
	12) Move module image_quality_assmessment_mos to custom_datasets
	13) Move class LanguageGuidedVideoSummarizationDataset to custom_datasets
	14) Move class MGeoRankingDataset to custom_datasets
	15) Move module movie_scene_segmentation custom_datasets
	16) Move module object_detection to custom_datasets
	17) Move module referring_video_object_segmentation to custom_datasets
	18) Move module sidd_image_denoising to custom_datasets
	19) Move module video_frame_interpolation to custom_datasets
	20) Move module video_stabilization to custom_datasets
	21) Move module video_super_resolution to custom_datasets
	22) Move class GoproImageDeblurringDataset to custom_datasets
	23) Move class EasyCVBaseDataset to custom_datasets
	24) Move class ImageInstanceSegmentationCocoDataset to custom_datasets
	25) Move class RedsImageDeblurringDataset to custom_datasets
	26) Move class TextRankingDataset to custom_datasets
	27) Move class VecoDataset to custom_datasets
	28) Move class VideoSummarizationDataset to custom_datasets
10. To delete modules/functions/classes:
	1) Del module task_datasets
	2) Del to_task_dataset() in EpochBasedTrainer
	3) Del build_dataset() in EpochBasedTrainer and renew a same name function.
11. Rename class Datasets to CustomDatasets in metainfo.py

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11872747
2023-03-10 09:03:32 +08:00
feiwu.yfw
2b64cf2bb6 [to #42322933]支持从dataset json文件中获取参数
* dataset json file add args
2022-08-30 15:15:15 +08:00
feiwu.yfw
39485426e7 [to #42322933]:fix msdataset
* 修复了zip文件不同打包模式下返回路径错误问题。
* 修复了替换了数据集文件重新下载时校验失败问题。
* 修复dataset oss文件在 REUSE 模式下重复下载的问题。
* 修复了csv数据集的meta json文件中某个split的meta和file字段都为''时加载所有split失败的问题。
 * 修复了不同版本datasets路径不一致的问题。
2022-08-26 22:41:13 +08:00
feiwu.yfw
35548bd492 [to #43875101]
msdataset add coco dataset
unify taskdataset and ms dataset
fix hf datasets
2022-08-17 22:51:22 +08:00
zhangzhicheng.zzc
9d0b38b4e4 [to #42322933] lazy load on trainer 2022-08-04 14:07:14 +08:00
wenmeng.zwm
d55525bfb6 [to #43112771] requirements check and lazy import support 2022-07-27 17:29:16 +08:00
hejunjie.hjj
c112e599fe [to #42322933]enable image segmentation test
enable image segmentation test
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/9522337
2022-07-26 17:49:20 +08:00
Yingda Chen
2e43b376f5 [to #42322933]disable image segmentation test temporarily 2022-07-26 13:17:29 +08:00
hejunjie.hjj
321292ceda [to #42322933] add image instance segmentation pipeline and finetune to MaaS-lib 2022-07-26 10:19:07 +08:00