Feasibility of training a Turkmen LLM from scratch on 100MB data

[tllmmaster]

Hello everyone,
I have a question about training an LLM from scratch with a small dataset.
I have about 100MB of clean text data in the Turkmen language. My goal is to:

1. Pre-train a model from scratch (Next Word Prediction) using only this 100MB dataset.
2. Afterwards, perform Instruction Fine-tuning on this resulting model using the same data (converted into instruction format).

To clarify my expectations: I only need the model to answer questions based on the information contained in this training data. I do not expect it to have general knowledge or capabilities beyond what is in the dataset.

Is this feasible? Can I effectively get a working model by following this pipeline with such a small amount of data?

[rasbt]

Hi there, I think this sounds somewhat feasible, but in the case of such a small dataset plus

To clarify my expectations: I only need the model to answer questions based on the information contained in this training data.

maybe try a RAG setup first. You could try one of these notebooks. Not totally from scratch but should be easy to use: https://github.com/rasbt/RAGs

[tllmmaster]

Dear Sebastian,
Thank you so much for the quick reply and the recommendation. I agree that RAG would be the most practical solution for this use case.
However, this is a strict university assignment where the specific requirement is to build a model from scratch and then fine-tune it. Unfortunately, I cannot use RAG due to these constraints.
Given that I must proceed with training from scratch on this small 100MB dataset, could you advise on a reasonable model size (number of parameters) or architecture (e.g., a tiny GPT-2 config) that would minimize overfitting and allow the model to learn at least the basic structure of the language?
Best regards,
Turkmen engineer