I think this is between two of you. If both of you are ready to accept the past of each other then rest of the people can be ignored. You should not think that your past is acceptable and her past is not, just because she is a girl. It sounds too narrow and you need to show some sensitivity towards the issue. You can always stop people talking by putting your foot down and your acceptance for the same.