Now in Arxiv preprint, this short paper from Saama’s AI Research Lab proposes a new benchmark for cost- and time-effective natural language processing (NLP) models, trained on a single GPU.
The Saama Team’s Small-Bench NLP benchmark was trained with 15M parameters over just a few days and achieved an average score of 81.53. This is comparable to that of BERT-Base’s 82.20 model, trained with 110M parameters over several weeks.
The new benchmark facilitates research across a variety of NLP tasks. Now researchers with resource constraints can experiment with innovative ideas related to tokenization, pre-training, architecture, fine-tuning methods, and more.
The Saama team intends to work with collaborators and open-source contributors on extending small models to several domains and datasets, such as finance and biomedical.