how can male infertility be solved?

hi i am m/29 my wife is 28 , we are married for last 4 yrs , we started planning for a child , since last 2 yrs, but we did not get any result.we were both checked by the doc, while my wife was said to be normal, doc told me that i had very less semen . the report showed i had only 2-3 lacs, sometimes even absent. i was given to try some vitamins for 3 months that i using right now. After using i find that my penis size has grown , my semen has become little thick.
I would like your suggestion regarding me , we are very desperate to start a family . i had also undergone a scrotum test and a testes test. but the repors were normal .