Tokens per second is the standard throughput metric for large-language-model workloads, counting how many text tokens a system generates or processes each second. For inference it captures serving speed and cost efficiency, while for training it reflects how fast a cluster consumes data. Data-center operators use it alongside utilization and power draw to size GPU fleets and benchmark inference and training performance.
Category: IT & Compute Infrastructure · Subcategory: AI & HPC Infrastructure · Part of the AIDC Technical Terminology reference.