Optimal management of anemia in patients with chronic kidney disease remains a divisive issue within the nephrology community. Because the evidence provided by successive randomized controlled trials has often proven to be incongruent, it is natural to consider whether methodological issues may be responsible. Using four large trials [US Normal Hematocrit, Canadian European Normalization of Hemoglobin, Cardiovascular Risk Reduction by Early Anemia Treatment with Epoetin Beta (CREATE) and Correction of Hemoglobin and Outcomes in Renal Insufficiency (CHOIR)], this review article highlights several methodological issues that may be important when trial evidence is translated into clinical practice. Issues discussed include heterogeneity of enrollment criteria, failure to conceal treatment allocation, generalizability of study interventions, systematic use of imbalanced co-interventions [especially dose of erythropoietin stimulating agent (ESA), confusion regarding stopping rules and interim analyses and failure to account for imbalances in important patient characteristics generated at randomization.