3 Commits

Author SHA1 Message Date
mulin.lyh
cba4e40bc1 fix numpy pandas compatible issue
明确受影响的模型(damo):  
ONE-PEACE-4B	ModuleNotFoundError: MyCustomPipeline: MyCustomModel: No module named 'one_peace',缺少依赖。
cv_resnet50_face-reconstruction	 不兼容tf2  
nlp_automatic_post_editing_for_translation_en2de	tf2.0兼容性问题,tf1.x需要  
cv_resnet18_ocr-detection-word-level_damo	tf2.x兼容性问题  
cv_resnet18_ocr-detection-line-level_damo	tf兼容性问题  
cv_resnet101_detection_fewshot-defrcn	模型限制必须detection0.3+torch1.11.0"  
speech_dfsmn_ans_psm_48k_causal	"librosa, numpy兼容性问题  
cv_mdm_motion-generation	"依赖numpy版本兼容性问题:   File ""/opt/conda/lib/python3.8/site-packages/smplx/body_models.py"",  
cv_resnet50_ocr-detection-vlpt	numpy兼容性问题  
cv_clip-it_video-summarization_language-guided_en	tf兼容性问题

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/13744636
* numpy and pandas no version

* modify compatible issue

* fix numpy compatible issue

* modify ci

* fix lint issue

* replace Image.ANTIALIAS to Image.Resampling.LANCZOS pillow compatible

* skip uncompatible cases

* fix numpy compatible issue, skip cases that can not compatbile numpy or tensorflow2.x

* skip compatible cases

* fix clip model issue

* fix body 3d keypoints compatible issue
2023-08-22 23:04:31 +08:00
xingjun.wxj
e02a260c93 Refactor the task_datasets module
Refactor the task_datasets module:

1. Add new module modelscope.msdatasets.dataset_cls.custom_datasets.
2. Add new function: modelscope.msdatasets.ms_dataset.MsDataset.to_custom_dataset().
2. Add calling to_custom_dataset() func in MsDataset.load() to adapt new custom_datasets module.
3. Refactor the pipeline for loading custom dataset: 
	1) Only use MsDataset.load() function to load the custom datasets.
	2) Combine MsDataset.load() with class EpochBasedTrainer.
4. Add new entry func for building datasets in EpochBasedTrainer: see modelscope.trainers.trainer.EpochBasedTrainer.build_dataset()
5. Add new func to build the custom dataset from model configuration, see: modelscope.trainers.trainer.EpochBasedTrainer.build_dataset_from_cfg()
6. Add new registry function for building custom datasets, see: modelscope.msdatasets.dataset_cls.custom_datasets.builder.build_custom_dataset()
7. Refine the class SiameseUIETrainer to adapt the new custom_datasets module.
8. Add class TorchCustomDataset as a superclass for custom datasets classes.
9. To move modules/classes/functions:
	1) Move module msdatasets.audio to custom_datasets
	2) Move module msdatasets.cv to custom_datasets
	3) Move module bad_image_detecting to custom_datasets
	4) Move module damoyolo to custom_datasets
	5) Move module face_2d_keypoints to custom_datasets
	6) Move module hand_2d_keypoints to custom_datasets
	7) Move module human_wholebody_keypoint to custom_datasets
	8) Move module image_classification to custom_datasets
	9) Move module image_inpainting to custom_datasets
	10) Move module image_portrait_enhancement to custom_datasets
	11) Move module image_quality_assessment_degradation to custom_datasets
	12) Move module image_quality_assmessment_mos to custom_datasets
	13) Move class LanguageGuidedVideoSummarizationDataset to custom_datasets
	14) Move class MGeoRankingDataset to custom_datasets
	15) Move module movie_scene_segmentation custom_datasets
	16) Move module object_detection to custom_datasets
	17) Move module referring_video_object_segmentation to custom_datasets
	18) Move module sidd_image_denoising to custom_datasets
	19) Move module video_frame_interpolation to custom_datasets
	20) Move module video_stabilization to custom_datasets
	21) Move module video_super_resolution to custom_datasets
	22) Move class GoproImageDeblurringDataset to custom_datasets
	23) Move class EasyCVBaseDataset to custom_datasets
	24) Move class ImageInstanceSegmentationCocoDataset to custom_datasets
	25) Move class RedsImageDeblurringDataset to custom_datasets
	26) Move class TextRankingDataset to custom_datasets
	27) Move class VecoDataset to custom_datasets
	28) Move class VideoSummarizationDataset to custom_datasets
10. To delete modules/functions/classes:
	1) Del module task_datasets
	2) Del to_task_dataset() in EpochBasedTrainer
	3) Del build_dataset() in EpochBasedTrainer and renew a same name function.
11. Rename class Datasets to CustomDatasets in metainfo.py

Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/11872747
2023-03-10 09:03:32 +08:00
james.wjg
9b3a92e65d cv/language_guided_video_summarization增加finetune
cv/language_guided_video_summarization增加finetune
        Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10790262
2022-12-01 19:16:56 +08:00