If you take a small snack into another restaurant to eat, is it okay?

I went into a Huddle House (a 24 hour pancake house) and got a coffee and eat a banana I had in my back pack. Is that okay? (I did order the coffee from them). Can a restaurant make me leave or refuse to let me bring in my own food?