Close Icon

Small-Bench NLP: Benchmark for Single GPU Trained Models

Now in Arxiv preprint, this short paper from Saama’s AI Research Lab proposes a new benchmark for cost- and time-effective natural language processing (NLP) models, trained on a single GPU.

The Saama Team’s Small-Bench NLP benchmark was trained with 15M parameters over just a few days and achieved an average score of 81.53. This is comparable to that of BERT-Base’s 82.20 model, trained with 110M parameters over several weeks.

The new benchmark facilitates research across a variety of NLP tasks. Now researchers with resource constraints can experiment with innovative ideas related to tokenization, pre-training, architecture, fine-tuning methods, and more.

The Saama team intends to work with collaborators and open-source contributors on extending small models to several domains and datasets, such as finance and biomedical.

  • Download Paper

  • You may unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our Privacy Policy.