The photovoltaic effect in ferroelectric (FE) semiconductors, i.e., the FPV effect, is attracting increasing attention. However, the microscopic mechanism of the FPV effect is still largely unknown. Here, we investigated the effects of crystal structure, dielectric, FE, and domain features on the FPV effect in the 0.7BiFe1-xCrxO3-0.3BaTi1-xMnxO3 (BFC-BTM) (0.01 ≤ x ≤ 0.07) system. We observed a coexisting rhombohedral and tetragonal to pseudocubic phase transition for x ≥ 0.05. This structural transition was accompanied by a significant reduction in the lattice distortion and disappearance of macro-sized FE domains. We found a substantial increase and decrease in the degree of the diffuseness of the relaxor FE phase transition and the open-circuit photovoltage VOC, respectively, for x ≥ 0.05, in contrast to the relatively weak variation in the remnant polarization. Our results demonstrate that the FPV effect depends weakly on FE polarization but strongly on the complex submicron domain-related diffused relaxor FE phase transition and crystal structure in BFC-BTM. These results were discussed based on the non-centrosymmetry-related shift current mechanism.