Risk factors for marijuana use in older adolescents and young adults have focused primarily on family environment and peer affiliation. A growing body of work has examined the relationship between environmental context and young adult substance use. This study builds on previous research linking neighborhood environment to young adult marijuana use by exploring two distinct features of neighborhoods, namely the physical (e.g., broken windows) and social environment (e.g., adults watching youth). Data were obtained from a longitudinal sample of 398 predominately African American young adults living in an urban environment. The data also included observational measures of physical and social order and disorder collected on the young adult's residential block. Exploratory structural equation modeling (ESEM) was utilized to test hypothesized relationships between these two features of the neighborhood environment and past year young adult marijuana use. A two-factor model of neighborhood environment with good fit indices was selected (CFI = 0.97, RMSEA = 0.037). There was a positive and significant direct effect from neighborhood physical disorder to marijuana use (0.219, p < 0.05) controlling for gender, race, and free and reduced price meal (FARPM) status. The direct effect from neighborhood social environment to marijuana use was not significant. These results converge with previous research linking vacant housing with young adult marijuana use but do not provide empirical support for the neighborhood social environment as a determinant of drug taking. Better explication of the social environment is needed to understand its relationship to drug use.