Definite Alzheimer's disease (AD) diagnosis is commonly done on ex vivo brain tissue using immuno-histochemical staining to visualize amyloid-beta (Aβ) aggregates, also known as Aβ plaques. Raman spectroscopy has shown its potential for non-invasive and label-free determination of bio-molecular compositions, aiding the post-mortem diagnosis of pathological tissue. Here, we investigated whether conventional Raman spectroscopy could be used for the detection of amyloid beta deposits in fixed, ex vivo human brain tissue, taken from the frontal cortex region. We examined the spectra and spectral maps of three severe AD cases and two healthy control cases and compared their spectral outcome among each other as well as to recent results in the literature obtained with various spectroscopic techniques. After hyperspectral Raman mapping, Aβ plaques were visualized using Thioflavin-S staining on the exact same tissue sections. As a result, we show that tiny diffuse or tangled-like morphological structures, visible under microscopic conditions on unstained tissue and often but erroneously assumed to be deposits of Aβ, are instead usually an aggregation of highly auto-fluorescent lipofuscin granulates without any, or limited, plaque or plaque-like association. The occurrence of these auto-fluorescent particles is equally distributed in both AD and healthy control cases. Therefore, they cannot be used as possible criteria for Alzheimer's disease diagnosis. Furthermore, a unique plaque-specific/Aβ spectrum could not be determined even after possible spectral interferences were carefully removed.