I think what happened was since he was not explaining WHY he didn't want to go there, it was pretty much shrugged off. I personally assumed he just didn't want to be around the area after working there a long shift. It's understandable now that he explained, why it wouldn't work out for him at that particular location. It wasn't about the food but rather policy on being around the premises off of duty.
We don't want you fired dude. I'm sure if you would have explained, it would have gone a different way rather than you blowing up on a forum lol