Throw the baby out with the bathwater, To

1. To discard good elements of something with the bad elements that are intended to be disposed of.