"Go back to China"

I was walking back to work on my lunch hour and crossing a busy intersection when this incident occurred. I could see that the walk sign was illuminated and I and many other pedestrians made our way across the street along the crosswalk. Although the walk sign was lit, a middle aged, white man in his car was aggressively inching and trying to turn left through the pedestrians and the crosswalk. As his window was open, I said and motioned to him that this was the crosswalk, as if to say pedestrians have the right of way when crossing. He did not take kindly to this, and yelled that I should "Go back to China!" and drove off.