Smoking is a well-known risk factor of reproductive health. However, the effect of paternal smoking on fertility has been less extensively examined. We conducted a cross-sectional study in a mountainous area of South-West China to assess the effect of male smoking on couples' fertility. A total of 8200 couples aged 18-49 years in the study area were invited to participate in the study. An in-person interview was performed to collect demographic characteristics of the couples, and husbands' life style factors including smoking and drinking habits. Information on time to pregnancy (TTP) was collected retrospectively. Infertility was defined as failure to achieve clinical pregnancy after regular unprotected intercourse for ≥12 months. Logistic regression model was used to estimate the association between male smoking and infertility. A total of 7025 couples were included in the final analysis. After adjusting for potential confounders, the couples were more likely to suffer from infertility if the husbands smoked (adjusted odds ratio [aOR] =1.28, 95% CI: 1.08-1.52) before the first pregnancy. After the analyses were performed according to husbands' smoking duration, an increased risk started at a relatively longer smoking duration of 5-10 years (aOR = 1.58, 95% CI: 1.26-1.99) and a stronger association (aOR = 3.34, 95% CI: 2.45-4.56) was observed in the group of ≥10 years. Similar patterns were found for the number of cigarettes smoked per day and the total amount of cigarettes smoked. From our findings, we conclude that male smoking may have an adverse impact on couples' infertility.