We already know that genes regulates speech - for example, FOXP2 [0] - and have successfully sequenced the human genome, and have started similar initiatives on other archaic human and primate species.
Phylogenetic Analysis has been fairly successful already in analyzing our genetic history, so I'm not sure why you'd think it's impossible.
>As a quick point and this is quite a late response, having the same FOXP2 gene may not be enough. New evidence suggests
>Using statistical software that evaluates gene expression based on the type of gene, Vanderbilt graduate student Laura Colbran found that Neandertal versions of the gene would have pumped out much less FOXP2 protein than expressed in modern brains. In living people, a rare mutation that causes members of a family to produce half the usual amount of FOXP2 protein also triggers severe speech defects, notes Simon Fisher, director of the Max Planck Institute for Psycholinguistics in Nijmegen, the Netherlands, who discovered the gene.
I'm not an expert, but I guess they mean "complex enough language" to discard isolated words and consider only "languages" that have "sentences that may have 5 words" or some more accurate criteria.
Well, we know that they could communicate (or be attractive enough) to interbreed with Homo sapiens — presumably they weren’t the genetically stronger species because their DNA is generally 1-3% represented in ours.
We don’t know, as you note. But the genetic evidence combined with tool use and their large Supra-orbital indices (fairly big brains, if shaped differently than ours) all would make that the preferred prior I’d say.
It's not clear how much interbreeding there was. It doesn't take many introgression events over the millennia to show up in the genetic record. And, to put it delicately, it's not clear any communication was involved in these events.
