A three-dimensional (3D) linear model for digital breast tomosynthesis (DBT) was developed to investigate the effects of different imaging system parameters on the reconstructed image quality. In the present work, experimental validation of the model was performed on a prototype DBT system equipped with an amorphous selenium (a-Se) digital mammography detector and filtered back-projection (FBP) reconstruction methods. The detector can be operated in either full resolution with 85 microm pixel size or 2 x 1 pixel binning mode to reduce acquisition time. Twenty-five projection images were acquired with a nominal angular range of +/- 20 degrees. The images were reconstructed using a slice thickness of 1 mm with 0.085 x 0.085 mm in-plane pixel dimension. The imaging performance was characterized by spatial frequency-dependent parameters including a 3D noise power spectrum (NPS) and in-plane modulation transfer function (MTF). Scatter-free uniform x-ray images were acquired at four different exposure levels for noise analysis. An aluminum (Al) edge phantom with 0.2 mm thickness was imaged to measure the in-plane presampling MTF. The measured in-plane MTF and 3D NPS were both in good agreement with the model. The dependence of DBT image quality on reconstruction filters was investigated. It was found that the slice thickness (ST) filter, a Hanning window to limit the high-frequency components in the slice thickness direction, reduces noise aliasing and improves 3D DQE. An ACR phantom was imaged to investigate the effects of angular range and detector operational modes on reconstructed image quality. It was found that increasing the angular range improves the MTF at low frequencies, resulting in better detection of large-area, low-contrast mass lesions in the phantom. There is a trade-off between noise and resolution for pixel binning and full resolution modes, and the choice of detector mode will depend on radiation dose and the targeted lesion.