Many times the phrase "The Fact of Evolution" gets thrown around without any further explanation. Most people don't understand that there are several different things meant by "The Fact of Evolution", and the problem comes in when we just assume that they are correct, or unintentionally read more than one of these meanings together.
Meaning #1 - There are dead things in the ground at a lower strata than ourselves and they look different from us. In addition, there are clearly discernable strata which tend to produce similar fossils.
Meaning #2 - Everything now living (a) has parent(s), and (b) is slightly different from its parent(s)
Meaning #3 - The universe operates mechanically
The problem is that most people stating "The Fact of Evolution" never seem to be able to separate out Meaning #1 and Meaning #2, and therefore consider that Meaning #2 is not the causitive agent for Meaning #1. Even if you assume evolutionary timescales, there is nothing in the pattern of fossils in the record which would recommend it over, say, progressive Creationism, unless of course you also mixed in meaning #3. If we assume other possible causes for the layering and the ordering, then #1 and #2 could be completely different phenomena altogether!
The problem is that most evolutionists will not disambiguate what they mean by the "Fact of Evolution". The idea that you should read #1 and #2 together is so engrained in their thinking that the possibility of splitting them sounds insane to them.