The optical spectroscopy measurements of gadolinium gallium garnet (GGG) crystals doped with Yb show evidence of the presence of non-equivalent optical centers with very similar radiative decay rates. The energy level schemes of those centers have been determined on the basis of optical absorption, luminescence and Raman experiments. Crystal field fitting resulted in two sets of slightly different crystal field parameters for two non-equivalent Yb centers. Both sets of parameters describe perfectly the experimentally detected Yb(3+) energy levels. Correlation between systematic trends in the experimental energy level schemes and crystal field parameters is discussed.