In recent years, a new paradigm for machine learning has started to become dominant in many settings. Instead of training machine learning models from scratch on each new domain or task of interest, we instead pretrain large models on a wide and diverse set of data (often including natural language data and/or multimodal data), then specialize these models for downstream tasks (e.g., through finetuning or prompting). Such models are often referred to as foundation models owning to the fact that the pretrained model serves as a "foundation" for later task-specific models. This new paradigm has fueled a number of advances in AI, including recent work in large language models (LLMs), the most currently-visible class of Foundation Model, but also including models for vision, audio, chemistry, and many other domains. The trend is also related to, but generally a superset of, work on "Generative AI": many foundation models such as LLMs or diffusion models are generative in nature (meaning they generate text or images), but many others are not.
With this new paradigm, however, there emerge substantial new questions for the field. What sorts of algorithmic and architectural techniques should be used for such models, and how (if at all) do they differ from traditional models? How does one evaluate the quality and performance of foundation models, especially in settings where there does not exist a great deal of downstream tasks data? And how do we grapple with the larger implications of these models and their ongoing adoption in industry?
The CMU Foundation and Language Model (FLAME) Center provides a hub for our many faculty working on these topics and related areas. The faculty comprising the center covers a wide set of perspectives and approaches to foundation models, and there is no single position of viewpoint taken by the center as a whole. Rather, the goal is to foster fundamental research, especially on topics where industrial labs have traditionally been slower to respond, such as in the evaluation, quantification, and broader implications of such models.