Reply to thread

It should be noted this is for doing single inference which is heavily memory bound. If you do multiple simultaneous inferences, the more the extra compute power of the 4090 will come into play (as long as the model fits into memory!). And of course training is a different beast.