In 2004,five veterinarians examined 283 dairy cows from four Danish dairy herds and independently assigned scores for the clinical signs lameness, hock lesions and other cutaneous lesions to each cow using a clinical protocol. We evaluated the inter-observer and the intra-observer agreement using prevalence-adjusted, bias-adjusted kappa (PABAK). We chose two different cut-offs between the ordinal scores for classifying cows as healthy or diseased for each of the clinical signs, and we compared the ability of the observers to discriminate between healthy and diseased cows for the two cut-offs. Without any formal training of the observers we found PABAK in the range 0.36-0.88 and we concluded that the choice of cut-off had an effect on the observed agreement among the observers.