Ahmad won the election in the Ethiopian capital by 34 votes to Hayatou’s 20, official results showed.
“When you try to do something, you mean that you can do it,” Ahmad told reporters after the vote. “If I can’t do it, I never stand.”
Ahmad, a 57-year-old father of two, had a discreet playing and coaching career before he took the reins of the Madagascar football federation in 2003.
His rare and determined bid for “change” at the head of the CAF this year took many by surprise, and the incumbent was seen as the favourite.
Hayatou headed CAF since 1988 and is a senior vice president of FIFA. He has been credited with increasing the number of African teams at the World Cup and bringing in extra finance for the continent’s competitions.