From Scratch vs Pre-trained: A Dataset Size Analysis for Small-Scale Language Model Training

This research presents an empirical comparison of from-scratch versus pre-trained language model training strategies across small dataset sizes (1MB-20MB). The study reveals a critical dataset size threshold effect at 5-10MB where optimal training strategies diverge. Key findings include: from-scratch models achieving superior generalization on datasets <5MB through memorization mechanisms (PPL=1.0), while pre-trained models excel on datasets >10MB through genuine pattern learning. The work provides adaptive hyperparameter strategies and practical guidelines for model selection in resource-constrained scenarios. Implications extend to specialized domains, low-resource languages, and situations where gigabyte-scale datasets are unavailable.