World Model on Million-Length Video And Language With Blockwise RingAttention · HackerPapers discussions

World Model on Million-Length Video And Language With Blockwise RingAttention — HN submission

Mon, 08 Apr 2024 20:19:02 GMT

World Model on Million-Length Video and Language with Ring Attention

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

World Model on Million-Length Video And Language With Blockwise RingAttention — mentioned by 5kg

Fri, 16 Feb 2024 18:32:48 GMT

From https://arxiv.org/pdf/2402.08268.pdf > We trained our models using TPUv4-1024, which is approximately equivalent to 450 A100s > Inference for such long sequences requires a minimum of v4-128 So you'll need ~60 A100 for inference.

Paper and discussion history. A mention is not an endorsement.

World Model on Million-Length Video And Language With Blockwise RingAttention — mentioned by woadwarrior01

Fri, 16 Feb 2024 17:24:20 GMT

From the paper[1]: > We additionally scale our inference code to support million-length sequences by implementing RingAttention for decoding. Inference for such long sequences requires a minimum of v4-128 with a TPU mesh sharding of 32 tensor parallelism, and 4 sequence parallelism (ring dimension). Each TPU-v4 has 32 GiB of HBM memory, so about 4 TiB (128 x 32GiB) of memory in fp16, without quantization. [1]: https://arxiv.org/abs/2402.08268

Paper and discussion history. A mention is not an endorsement.

World Model on Million-Length Video And Language With Blockwise RingAttention — mentioned by andrewljohnson

Thu, 15 Feb 2024 00:24:50 GMT

Also from the post I saw on this: Paper: https://arxiv.org/abs/2402.08268 Models: https://huggingface.co/LargeWorldModel Website: https://largeworldmodel.github.io

Paper and discussion history. A mention is not an endorsement.