What are pre-training and fine-tuning?
Definition
Pre-training and fine-tuning is a two-stage way to train a machine learning model. The model is first pre-trained on a large, broad dataset to learn general patterns, then fine-tuned on a smaller dataset for a specific task. In engineering simulation, fine-tuning adapts a model pre-trained on many simulations to a team's own geometries, physics and conditions.
Example
In research on foundation models for partial differential equations, models pre-trained on broad sets of PDE solutions and then fine-tuned reached a target accuracy with far fewer downstream examples than models trained from scratch. The gain shrinks as the new problem moves further from the pre-training data, so a fine-tuned model is still checked on held-out simulations of the target problem.
Sources
- Subramanian et al., "Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior," NeurIPS 2023, arxiv.org/abs/2306.00258
- Yosinski, Clune, Bengio, Lipson, "How transferable are features in deep neural networks?," NIPS 2014, arxiv.org/abs/1411.1792