Father has racist friends

All the people my ex hangs out with say things like, "N... word this and n... word that." We have joint custody and it makes me uneasy my children are half black they are around these people. Do I have a right to be concerned and do u think I have grounds for full custody?