Society has definite place in our life but inner happiness is much important. If you really think that you get more internal happiness with him then you should not get scared of society. People may opine about what you did but at the same time they are not bothered and really concerned. More over, these things are short lived so always put yourself ahead of the society. If you are happy and satisfied with your life then society automatically starts accepting the way you are.