Your libido (desire to have sex) is ok and still you feel irritation that means you have some underlying emotional issues. You have to look for anything which can be causing emotional distress. I cannot suggest any thing further until I know more details.