Why do people ume I am not paying attention?

A lot of the time I listen with my ears and not with my eyes. I take in the world through my ears. I will give you examples of people thinking I am not paying attention, I was at the stall and looking at this man coming. I saw him and was about to go over and serve him and he tapped on the table rudely and said "You need to sell more stuff". He was doing this to all the stalls though. When my friend is talking I listen and she asks me "Am I listening". Sometimes in cl I look down at my notes and the teacher stops and looks at me and does not think I am listening. I take in everything that goes on but do I not look like I am paying attention?