The development of machine vision-based technologies to replace human labor for rapid and exact detection of agricultural product quality has received extensive attention. In this study, we describe a low-rank representation of jointly multi-modal bag-of-feature (JMBoF) classification framework for inspecting the appearance quality of postharvest dry soybean seeds. Two categories of speeded-up robust features and spatial layout of L*a*b* color features are extracted to characterize the dry soybean seed kernel. The bag-of-feature model is used to generate a visual dictionary descriptor from the above two features, respectively. In order to exactly represent the image characteristics, we introduce the low-rank representation (LRR) method to eliminate the redundant information from the long joint two kinds of modal dictionary descriptors. The multiclass support vector machine algorithm is used to classify the encoding LRR of the jointly multi-modal bag of features. We validate our JMBoF classification algorithm on the soybean seed image dataset. The proposed method significantly outperforms the state-of-the-art single-modal bag of features methods in the literature, which could contribute in the future as a significant and valuable technology in postharvest dry soybean seed classification procedure.