Investigating the genetic factors influencing human birth weight may lead to biological insights into fetal growth and long-term health. Genome-wide association studies of birth weight have highlighted associated variants in more than 200 regions of the genome, but the causal genes are mostly unknown. Rare genetic variants with robust evidence of association are more likely to point to causal genes, but to date, only a few rare variants are known to influence birth weight. We aimed to identify genes that harbour rare variants that impact birth weight when carried by either the fetus or the mother, by analysing whole exome sequence data in UK Biobank participants. We annotated rare (minor allele frequency <0.1%) protein-truncating or high impact missense variants on whole exome sequence data in up to 234,675 participants with data on their own birth weight (fetal variants), and up to 181,883 mothers who reported the birth weight of their first child (maternal variants). Variants within each gene were collapsed to perform gene burden tests and for each associated gene, we compared the observed fetal and maternal effects. We identified 8 genes with evidence of rare fetal variant effects on birth weight, of which 2 also showed maternal effects. One additional gene showed evidence of maternal effects only. We observed 10/11 directionally concordant associations in an independent sample of up to 45,622 individuals (sign test P=0.01). Of the genes identified, IGF1R and PAPPA2 (fetal and maternal-acting) have known roles in insulin-like growth factor bioavailability and signalling. PPARG, INHBE and ACVR1C (all fetal-acting) have known roles in adipose tissue regulation and rare variants in the latter two also showed associations with favourable adiposity patterns in adults. We highlight the dual role of PPARG in both adipocyte differentiation and placental angiogenesis. NOS3, NRK, and ADAMTS8 (fetal and maternal-acting) have been implicated in both placental function and hypertension. Analysis of rare coding variants has identified regulators of fetal adipose tissue and fetoplacental angiogenesis as determinants of birth weight, as well as further evidence for the role of insulin-like growth factors.