A 74-year-old man who was referred to our late onset hypogonadism clinic presented with sweating and loss of appetite. His aging males' symptoms (AMS) and international index of erectile function (IIEF-5) scores were 59 and 2, respectively. His hormonal examination revealed extremely low free testosterone values. The patient was started on androgen replacement therapy, but his symptoms did not improve. Additional hormonal examinations revealed low values for other anterior pituitary hormones. Magnetic resonance imaging revealed an intrasellar cystic mass with suprasellar extension. We considered this mass caused hypothalamic hypopituitarism. A load test for anterior pituitary hormones revealed panhypogonadism. His symptoms improved after administration of adrenal and thyroid hormones and androgen. Five months after start of drug administration, his AMS score improved to 29, but IIEF-5 score showed little change. As a matter of course, not only androgen but all pituitary-related hormones are needed for hypopituitarism patients.