What is a transformer in engineering simulation?
Definition
A transformer is a neural network architecture built on attention, a mechanism that lets each element of the input weigh its relation to every other element. In engineering simulation, transformers treat mesh points, or groups of points, as tokens and learn to predict physical fields such as pressure, velocity or stress across varied geometries.
Example
Plain attention gets expensive on large meshes, because its cost grows with the square of the number of points. Transolver (ICML 2024) groups mesh points with similar physical states and computes attention between the groups, which lets it predict fields on unstructured meshes such as cars and airfoils.
Sources
- Vaswani et al., "Attention Is All You Need," 2017, arxiv.org/abs/1706.03762
- Wu et al., "Transolver: A Fast Transformer Solver for PDEs on General Geometries," ICML 2024, arxiv.org/abs/2402.02366