As he has never left his family for you, he wants to keep relations with you without disturbing his family life. If you can digest this fact then you may go on continuing this relation with secondary status (primary is his family). Otherwise, accept the fact that to live with him is your wish and not his. Try to prepare yourself to leave him. You should try to find someone who can give you emotional security as well as socially acceptable committed relationship.