Chinchilla aftermath

The consequences of the Chinchilla paper (Hoffmann et al. 2022): it turned out that large models were undertrained relative to the available data. In 2023-24 the industry "overtrained" smaller models to make them cheaper at inference (optimizing for cost-per-token rather than cost-per-train). The result: SLMs became economically attractive.