Transforming growth factor-beta 1 (TGF-beta 1) induces angiogenesis in vivo and capillary morphogenesis in vitro. Two receptor serine/threonine kinases (types I and II) have been identified as signal transducing TGF-beta receptors. We explored the possibility of inhibiting TGF-beta-mediated events in glomerular capillary endothelial cells using a TGF-beta type II receptor (T beta R-II) transdominant negative mutant. A mutant TGF-beta type II receptor (T beta R-IIM), lacking the cytoplasmic serine/threonine kinase domain, was produced by polymerase chain reaction using rat T beta R-II cDNA as template. Since T beta R-II and TGF-beta type I receptor (T beta R-I) heterodimerize for signal transduction, the mutant receptor competes for binding to wild-type T beta R-I, hence acting in a dominant negative fashion. Glomerular capillary endothelial cells were stably transfected with T beta R-IIM, and four independent clones were expanded. That the T beta R-IIM mRNA was expressed was shown by reverse transcriptase-polymerase chain reaction, RNase protection assay, and Northern analysis. Presence of cell surface T beta R-IIM protein was shown by affinity cross-linking with 125I-TGF-beta 1. In wild-type endothelial cells, TGF-beta 1 (2 ng/ml) significantly inhibited [3H]thymidine incorporation to 63 +/- 10% of control (n = 4). In transfected endothelial cells carrying T beta R-IIM, TGF-beta 1 stimulated [3H]thymidine incorporation to 131 +/- 9% of control (n = 4, p < 0.005). Also, in wild-type endothelial cells, endogenous and exogenous TGF-beta 1 induced apoptosis and associated capillary formation. Both apoptosis and capillary formation were uniformly and entirely absent in transfected endothelial cells carrying T beta R-IIM. This represents the first demonstration that capillary morphogenesis in vitro is associated with apoptosis, and that interference with T beta R-II signaling inhibits this process in glomerular capillary endothelial cells.