This is an automated archive made by the Lemmit Bot.

The original was posted on /r/machinelearning by /u/Even_Information4853 on 2024-11-03 15:05:44+00:00.

You may already know the Recipe for Training Neural Networks bible from Karpathy 2019

While most of the advices are still valid, the landscape of Deep Learning model/method has changed a lot since. Karpathy’s advices work well in the supervised learning setting, he does mention it:

stick with supervised learning. Do not get over-excited about unsupervised pretraining. Unlike what that blog post from 2008 tells you, as far as I know, no version of it has reported strong results in modern computer vision (though NLP seems to be doing pretty well with BERT and friends these days, quite likely owing to the more deliberate nature of text, and a higher signal to noise ratio).

I’ve been training a few image diffusion models recently, and I find it harder to make data driven decisions in the unsupervised setting. Metrics are less reliable, sometimes I train models with better losses but when I look at the samples they look worse

Do you know more modern recipes to train neural network in 2024? (and not just LLMs)