If you have normal hair growth at other parts like under arms, around sex organ, limbs etc; you need not to worry. Many normal males may  have poor hair growth on face and chest. It may be hereditary in some cases.  Your sexual abilities have nothing to do with this. Many female likes hairy chest but not all, remember male models are exposes clean shaved chests.