It turned out to be problems caused by Autoptimize after all! After deactivating it, manually removing the cache it had created and then activating it again, I managed to try the settings out some more. There are two separate settings that causes the image problem after selecting a variation swatch – Image lazy-loading AND the second checkbox in the Javascript section; “Aggregate JS-files?”.
I am glad it started working again but is there anything you can do or recommend to compensate for the missing page optimization this causes? Lazy-loading of images and javascript aggregation seem like a good idea!