Automatic detection of lung tumors and abnormal lymph nodes are useful in assisting lung cancer staging. This paper presents a novel detection method, by first identifying all abnormalities, then differentiating between lung tumors and abnormal lymph nodes based on their degree of overlap with the lung field and mediastinum. Regression-based appearance model and graph-based structure labeling are designed to estimate the actual lung field and mediastinum from the pathology-affected thoracic images adaptively. The proposed method is simple, effective and generalizable, and can be potentially applicable to other medical imaging domains as well. Promising results are demonstrated based on our evaluations on clinical PET-CT data sets from lung cancer patients.