arXiv:2402.08268cs.LG
World Model on Million-Length Video And Language With Blockwise RingAttention
Abstract
Enabling long-context understanding remains a key challenge in scaling existing sequence models -- a crucial component in developing generally intelligent models that can process and operate over long temporal horizons that potentially consist of millions of tokens. In this paper, we aim to address these challenges by providing a comprehensive exploration of the full development process for producing 1M context language models and video-language models, setting new benchmarks in language retrieval and new capabilities in long video understanding. We detail our long context data curation process, progressive context extension from 4K to 1M tokens, and present an efficient open-source implementation for scalable training on long sequences. Additionally, we open-source a family of 7B parameter models capable of processing long text documents and videos exceeding 1M tokens.
From the paper[1]: > We additionally scale our inference code to support million-length sequences by implementing RingAttention for decoding. Inference for such long sequences requires a minimum of v4-128 with a TPU mesh sharding of 32 tensor parallelism, and 4 sequence…
woadwarrior01View on HNParent context
From https://arxiv.org/pdf/2402.08268.pdf > We trained our models using TPUv4-1024, which is approximately equivalent to 450 A100s > Inference for such long sequences requires a minimum of v4-128 So you'll need ~60 A100 for inference.
5kgView on HNParent context