Primary hypophysitis is believed to be a chronic inflammation of the pituitary tissue caused by the autoimmune mechanism. The disease can be classified based on morphology and histology simultaneously, but the relationships between these subtypes remain unclarified. Moreover, hypophysitis may occur as a part of systemic immunoglobulin G4 (IgG4)-related plasmacytic disease. A 74-year-old woman was initially diagnosed with infundibulo-hypophysitis. After a long period of stability, she suffered rapid deterioration with evolving endocrinopathies and visual symptoms. Biopsy specimen established the diagnosis as granulomatous hypophysitis with positive reaction for IgG4 in infiltrating plasma cells. Postoperative glucocorticoid administration improved her condition dramatically. This case illustrates two interesting points: The rapid deterioration after a long stable clinical course, and the presence of IgG4-positive tissue in the pituitary gland, which can be considered as "primary" hypophysitis with no systemic IgG4-related disease in other organs.