My opinion is this. We should all strive to adhere to logic, since it's demonstrated pretty conclusively over the years its own success when properly used. Secondly, we shouldn't slack in explaining our positions as thoroughly as needed when it's possible to do so.
I find it interesting to note that you started this thread wondering why people consider it a racist symbol and trying to assert that it isn't, when up to that point nobody even called it 'racist'. You're the first person to refer to it in those words.
Racial stereotyping IS racism. That's exactly what racism is. And just because 'racism' is such a strong, loaded word in the American discourse doesn't mean that stereotyping shouldn't be.