Here's a simple formula for you to use to see if something is potentially offensive / racist.
Replace group identity being referred to with your own group identity. If you think it is offensive, it is.
Let's say one of your co-workers is Hispanic. And let us further speculate that you have asked that co-worker to speak to a patient / client because the patient / client speaks Spanish and you do not. You need to meet with the patient / client. Your Hispanic co-worker is going to be going with you since you both perform the same general task, but this time your specific skill set is needed. He / she will be able to help with the generalities, but you are needed for the specifics just as the co-worker is needed for translation / communication. As the co-worker is on the phone making arrangements for you two to meet up with the client / patient, you start making comments asking where the meeting is taking place because you don't want to get mugged by the co-worker's "people".
Let's stop there for a second. Now we will boil down the comment in question. As has been established, your co-worker is Hispanic. You are African-American. So taking a look at the comment
Where are you telling him to meet us? I don't want to get mugged by your people!
Breaking the sentence down into it's components, "I" is a black person, "your people" are Latinos. So what you are saying is "I don't want to get mugged by the Hispanics". Implying that the Hispanic people are known for and have a reputation of mugging people.
Now if we reverse that, and your Hispanic co-worker says to you "I don't want to get mugged by your people", that would translate to "I don't want to get mugged by black people, because black people are known for and have a reputation for mugging people".
I am not speaking on either generality, nor am I speaking on the veracity of said statements. I am merely commenting on the fact that "black people mug people, so I am afraid to go into their neighborhoods" could be construed as racist. Therefore "Hispanic people mug people, so I am afraid to go into their neighborhoods" could also be construed as racist.
Next time on "I'm Here To Help": When you ask someone to translate for you because you don't speak the language of the person you need to communicate with, it is advantageous to not accuse your translator of telling the person they are speaking to on your behalf to "stop telling them to rob me, I don't know what you saying over there [sic]".
Tune in, won't you?