Sequence Packing - Tag - Naifan Li's Bloghttps://blog.omagiclee.com/tags/sequence-packing/Sequence Packing - Tag - Naifan Li's BlogHugo -- gohugo.ioen-usThu, 29 Jul 2021 12:16:16 +0800Summary: Sequence Packinghttps://blog.omagiclee.com/posts/llms/sequence-packing/summary/Thu, 29 Jul 2021 12:16:16 +0800Naifan Lihttps://blog.omagiclee.com/posts/llms/sequence-packing/summary/. Instead of padding, we concatenate multiple shorter sequences into a single, longer sequence. This minimizes wasted compute (through padding tokens)

References

]]>