Frowning. It was a tool used by white civil rights supporters in the 50s and 60s in difficult areas during that phase of the civil rights movement. You don't start anything, you just fail to give the expected affirmation and instead just frown. It's a little thing but somewhat effective from someone in-group and doesn't give an excuse for a confrontation.