Go against the grain, To

1. To take a difficult route or path. 2. To move in the opposing direction from most others.