If it’s too small - Yes it will matter to them and some women believe the bigger the better. 

For most men, the size of their penis also matters. In fact, penis size can either boost or break a man’s self-esteem. 

But intimacy and technique would have to go first rather than the size. If you know how to make your partner feel very satisfied when you have sex - then the size is not an issue.