mirror of
https://github.com/modelscope/modelscope.git
synced 2025-12-24 20:19:22 +01:00
1. Change the logic of using fast tokenizer from mode to user arguments and tokenizer_config.json
This is to fix the problem of RANER must use fast tokenizer in some special models.
Link: https://code.alibaba-inc.com/Ali-MaaS/MaaS-lib/codereview/10488982