Throw the baby out with the bathwater

Discard something valuable along with other things that are undesirable.

P1: "Ugh this cereal box has a slight dent in it, I'm going to have to get a refund" P2: "Ok let's not throw the baby out with the bathwater"