Low compute constraints
Most of the time things move fast, but these days, there is a slowness creeping into the training. It isn't just the processing time; it is the bottleneck of generating synthetic data. The challenge is accurately understanding the domain that the data should represent and actually modeling it so we can gain visibility into what is going on. I work on transferring simulation to reality (Sim2Real), and right now, getting a single turnover takes at least 5 or 6 days on a single GPU.
Working in a place with a low-compute environment forces you to see problems strictly through the domain of utility and practicality. Simply trying to fine-tune a state-of-the-art model is not going to work without deep pockets. The ideal solution that works at scale crumbles under these conditions, forcing one to instead leverage clever hacks like changing input resolutions or even redefining the problem entirely to reduce the domain adaptation gap.
The convention, or the wisdom of the industry, no longer matches the expectations of a constrained environment. The matter comes down to the scale of the player. To the big players, the solution is more compute; to the small player, the solution must be algorithmic budget.
I had to learn the hard way that a solution for you isn't a solution for me. One cannot rely on foundation models to solve that one niche problem that only intricate domain knowledge can address. It will likely take a year or two for foundation models to catch up to these long-tail problems. Until then, we have to do it the hard way.
If you enjoy the content, feel free to subscribe to my blog via email or RSS feed. Also my game is available at Geometry Ronin