This paper studies some properties and implications of higher-order mutual information functions, which should serve for the analysis of general complex systems. We note that the higher-order mutual information can either be positive or negative depending on the correlation among ensembles. Two opposite types of correlations are discussed in connection with the concept of frustration. Simple examples are presented to demonstrate that our concepts are especially helpful in understanding the nature of correlations in frustrated systems. The higher-order mutual information provides an appropriate measure of the frustration effect.