Can my mother , brother and I get deported?

My father came to the states on a asylum status. he brought myself my brother and my mother to the states after few years. Now my father is a permanent resident and rest of us are U.S citizens. There is a chance my father will get deported because of domestic violence occurrences, he is threatening my mother and I that if he gets deported he will tell the courts that he initially applied asylum under different name, and than when he didn't get approved he applied under different alias. This would be considered fraud, I believe. His claim is that since he brought us on the basis of his asylum case, we would get deported as well. My question , to you is , Does his claims have any validity? should we be worried? We just want him out of our life.