mlab-LLM

mlab-LLM is a small language model pre-trained from scratch on Japanese text, without using any existing model. All training was done on a single lab GPU (RTX 4090) and took about 7 days. The goal is to use it as a chatbot that answers questions on our lab website.

I first pre-trained a model with about 300M parameters on Japanese web text, then fine-tuned it on general dialogue data and on question-answer data about the lab. After that, I converted the model into a Mixture of Experts (MoE). An MoE model has several "expert" networks and switches between them depending on the input. The conversion increases the model's capacity, with the aim of having it remember more about the lab.

The trained models are available on Hugging Face (MoE 900M, 300M Base, ONNX for browsers).