Mistral 7B

Mistral AI team is proud to release Mistral 7B, the most powerful language model for its size to date.

Approaches CodeLlama 7B performance on code, while remaining good at English tasks

Uses Sliding Window Attention (SWA) to handle longer sequences at smaller cost

Sliding window attention exploits the stacked layers of a transformer to attend in the past beyond the window size: A token i at layer k attends to tokens [i-sliding_window, i] at layer k-1 . These tokens attended to tokens [i-2*sliding_window, i] . Higher layers have access to information further in the past than what the attention patterns seems to entail.

We're releasing Mistral 7B under the Apache 2.0 license, it can be used without restrictions.

Download it and use it anywhere (including locally) with our reference implementation ,

Deploy it on any cloud (AWS/GCP/Azure), using vLLM inference server and skypilot ,

Mistral 7B is easy to fine-tune on any task. As a demonstration, we're providing a model fine-tuned for chat, which outperforms Llama 2 13B chat.