Download HuggingFace Model
listing down the steps involved..
As a first step, lets get started by creating a virtual environment
python3 -m venv myenv
source /bin/active/myenv
After activating the virtual environment myenv
we need to get started with the packages installation. Below are the packages to be installed.
pip3 install torch tensorflow flax sentencepiece
the above packages if skipped will not be possible to download the model from transformers library. Hence we need to install pytorch, tensforflow, flax and also sentencepiece..!
Open up a jupyter notebook
jupyter notebook
This will open up a new tab, create a new .ipynb notebook and then will be able to import some transformer packages, for eg.,
from transformers import AutoModelForSequenceClassification,AutoTokenizer
import warnings, logging
warnings.simplefilter('ignore')
logging.disable(logging.WARNING)
After successful import of the transformers, we can actually specify the model which we wanted to download.
model_nm = 'microsoft/deberta-v3-small'
model = AutoModelForSequenceClassification.from_pretrained(model_nm, return_dict=True)
tokenizer = AutoTokenizer.from_pretrained(model_nm)
We can provide the absolute path to save the model
save_path = '/Users/<name>/<model-downloads>/podeberta_v3_small_pretrained_model_pytorch'
!mkdir {save_path}
We can save both the model and tokenizer in the same path
model.save_pretrained(save_path)
tokenizer.save_pretrained(save_path)
If we list down the files it saved -
!ls {save_path}
added_tokens.json special_tokens_map.json tokenizer_config.json
config.json spm.model
model.safetensors tokenizer.json
Thats how simple to download a hugging face model and save local to disk.