))
MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
저장한 시리즈 ("피드 비활성화" status)
When?
This feed was archived on August 11, 2025 06:07 (
Why? 피드 비활성화 status. 잠시 서버에 문제가 발생해 팟캐스트를 불러오지 못합니다.
What now? You might be able to find a more up-to-date version using the search function. This series will no longer be checked for updates. If you believe this to be in error, please check if the publisher's feed link below is valid and contact support to request the feed be restored or if you have any other concerns about this.
Manage episode 394754374 series 2954468
Rob에서 제공하는 콘텐츠입니다. 에피소드, 그래픽, 팟캐스트 설명을 포함한 모든 팟캐스트 콘텐츠는 Rob 또는 해당 팟캐스트 플랫폼 파트너가 직접 업로드하고 제공합니다. 누군가가 귀하의 허락 없이 귀하의 저작물을 사용하고 있다고 생각되는 경우 여기에 설명된 절차를 따르실 수 있습니다 https://ko.player.fm/legal.
State Space Models (SSMs) have become serious contenders in the field of sequential modeling, challenging the dominance of Transformers. At the same time, Mixture of Experts (MoE) has significantly improved Transformer-based LLMs, including recent state-of-the-art open-source models. We propose that to unlock the potential of SSMs for scaling, they should be combined with MoE. We showcase this on Mamba, a recent SSM-based model that achieves remarkable, Transformer-like performance. Our model, MoE-Mamba, outperforms both Mamba and Transformer-MoE. In particular, MoE-Mamba reaches the same performance as Mamba in 2.2x less training steps while preserving the inference performance gains of Mamba against the Transformer.
2024: Maciej Pi'oro, Kamil Ciebiera, Krystian Kr'ol, Jan Ludziejewski, Sebastian Jaszczur
https://arxiv.org/pdf/2401.04081.pdf
…
continue reading
2024: Maciej Pi'oro, Kamil Ciebiera, Krystian Kr'ol, Jan Ludziejewski, Sebastian Jaszczur
https://arxiv.org/pdf/2401.04081.pdf
298 에피소드
공유
