Composite membranes featuring metal-organic framework (MOF)-dispersed polymers have attracted tremendous attention in recent years. However, evaluating commercial viability is oftentimes obscured by the irreproducibility in both MOF synthesis and film manufacturing protocols. Variability in MOF property sets are typically ascribed to crystal defects resulting from subtle variations in synthesis, but quantitative studies investigating the role of defects on transport properties are exceedingly rare. Likewise, controlled film formation protocols are rarely reported in the open literature, making it difficult to provide substantial and informative structure-property correlations. This study aims to address these uncertainties. To this end, two samples of a prototypical MOF, UiO-66-NH2, were synthesized to feature similar particle size, morphology, and colloidal stability. However, defect engineering protocols coupled with careful screening experiments were developed to synthesize the two MOFs with maximally different porosities. Composite membranes were prepared for each MOF and a high-performance polymer, 6FDA-Durene, and then tested for light gas permeation measurements, revealing a small and unexpected enhancement in CO2/CH4 performance for samples containing low-porosity UiO-66-NH2. Mechanistic studies on sorption revealed a surprising 50% decrease in sorption capacity for high-porosity UiO-66-NH2, completely offsetting enhancements from increased gas diffusion. By using multiple replicate experiments, the sample-to-sample variation was large enough to obscure any differences in permeability and selectivity between the two types of MOF composites at low volume fractions. Application of the Maxwell model to extrapolate pure-MOF performance led to significant variations in predicted values, demonstrating the importance of collecting and reporting replicate experiments for membrane preparation and testing.