In the exploration of new infrared nonlinear optical (IR NLO) materials, element substitution has been developed as an effective way to adjust the structural features and material performances. A series of new IR NLO materials have been discovered in the I-Ba-MIV-Q system ( I = Li, Na, Cu, and Ag; MIV = Si, Ge, and Sn; Q = S and Se), and they undergo interesting structural transformation with different element substitution except Li analogues. Herein, we have successfully synthesized three selenides with different space groups (Ag2BaSiSe4: I4̅2 m; Ag2BaGeSe4 and Ag2BaSnSe4: I222) in the above system and studied their properties through experimental and theoretical methods. Remarkably, the detailed analysis on the structural changes and properties comparison was also systematically investigated in the I-Ba-MIV-Q system and the results indicate that the distortion degrees of different IQ4 tetrahedra play the critical role to cause the structural transformation with the M or Q elements substitution. More importantly, we have also found that the structural changes have the close relationship with the distance d( I- I) between adjacent I cations in the I2BaSnSe4 system, which makes the four-membered rings formed by edge-sharing BaSe8 units change from the square to rhombus with the increase of d( I- I). The properties comparisons (band gap and NLO effect) in this system have been also systematically studied.