arxiv Preprint - In-Context Pretraining: Language Modeling Beyond Document Boundaries

In this episode we discuss In-Context Pretraining: Language Modeling Beyond Document Boundaries by Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Scott Yih, Mike Lewis. This paper introduces a new approach called IN-CONTEXT PRETRAINING for training large language models. It addresses the limitation of current LM training pipelines that concatenate random sets of short documents without providing signal for predicting the next document. IN-CONTEXT PRETRAINING reorders the pretraining data by combining semantically related documents to create coherent input contexts, resulting in improved performance in tasks that require complex contextual reasoning.