On March 17, 2024, xAI announced the release of Grok-1, a 314 billion parameter Mixture-of-Experts language model, under the Apache 2.0 license. This groundbreaking model, distinguished by its vast parameter count and innovative architecture, was methodically trained from the ground up on an extensive corpus of text data without specific task optimization. Grok-1 stands out with 25% of its parameters being active for any given token, reflecting a strategic approach to model scaling and efficiency. The model, built using JAX and Rust, marks a significant advancement in machine learning, offering researchers and developers unparalleled access to one of the most powerful language processing tools available to date. Instructions for accessing and utilizing Grok-1's base model weights and architecture are provided on GitHub.
