That is not the best solution. If we just remove the variant images then when a customer selects a color it will not show the color. If you select a color from the drop down menu it usually will bring up that colored product as the main image. For instance if I select the blue shirt option it shows the blue product variant. However, with your solution that will not longer be the case. The user will select a color and then the main image will stay the same, the white default product. Which is not a great user experience. There has to be a way to not show all the variant images as thumbnails while still keeping the variant images and allowing the user to see the color they select.
