Small Prompts, Big Energy and CO2 Impact: Benchmarking Ollama LLMs on CPU and GPU

Abstract

Energy efficiency is a crucial challenge when deploying Large Language Models (LLMs). Electricity usage and related CO2 emissions can differ greatly depending on model architecture, parameter size, prompt length, and inference hardware. In this study, we evaluate 31 popular Ollama models across CPU and GPU inference, resulting in 60 testing scenarios. Energy and carbon metrics were gathered using the NVML and CodeCarbon libraries, providing insights into the environmental impact of LLM inference in data center settings.