We present 42 new Y-chromosomal sequences from diverse Indian tribal and non-tribal populations, including the Jarawa and Onge from the Andaman Islands, which are analysed within a calibrated Y-chromosomal phylogeny incorporating South Asian (in total 305 individuals) and worldwide (in total 1286 individuals) data from the 1000 Genomes Project. In contrast to the more ancient ancestry in the South than in the North that has been claimed, we detected very similar coalescence times within Northern and Southern non-tribal Indian populations. A closest neighbour analysis in the phylogeny showed that Indian populations have an affinity towards Southern European populations and that the time of divergence from these populations substantially predated the Indo-European migration into India, probably reflecting ancient shared ancestry rather than the Indo-European migration, which had little effect on Indian male lineages. Among the tribal populations, the Birhor (Austro-Asiatic-speaking) and Irula (Dravidian-speaking) are the nearest neighbours of South Asian non-tribal populations, with a common origin in the last few millennia. In contrast, the Riang (Tibeto-Burman-speaking) and Andamanese have their nearest neighbour lineages in East Asia. The Jarawa and Onge shared haplogroup D lineages with each other within the last ~7000 years, but had diverged from Japanese haplogroup D Y-chromosomes ~53000 years ago, most likely by a split from a shared ancestral population. This analysis suggests that Indian populations have complex ancestry which cannot be explained by a single expansion model.