In this paper, we propose a clustering method with temporal ordering information for endoscopic image sequences. It is difficult to collect a sufficient amount of endoscopic image datasets to train machine learning techniques by manual labeling. The clustering of endoscopic images leads to group-based labeling, which is useful for reducing the cost of dataset construction. Therefore, in this paper, we propose a clustering method where the property of endoscopic image sequences is fully utilized. For the proposed method, a deep neural network was used to extract features from endoscopic images, and clustering with temporal ordering information was solved by dynamic programming. In the experiments, we clustered the esophagogastroduodenoscopy images. From the results, we confirmed that the performance was improved by using the sequential property.