Ali won the non-title bout at Wembley and triumphed when the pair met again three years later.
But Sir Henry later became the British, Commonwealth and European heavyweight champion.
And although he never won a world title belt, he was hugely popular with the British public and in 2000 he became the first ever boxer to be knighted.
He was also the first sportsman to be crowned Sports Personality of the Year twice.
After spending 17 years in the ring, Sir Henry then commentated on the sport for over 20 years.
Sir Henry
Sir Henry has been described as a 'true gentleman'
Tributes have been pouring in for the Londoner, who passed away just two days before his 77th birthday.
Writing on Twitter, the current WBA World Champion David Haye called him a "true warrior and a great human being".
Lennox Lewis, former undisputed World Heavyweight Boxing Champion, posted: "R.I.P. Sir Henry Cooper. Former British, Commonwealth and European Champion. My deepest condolences to the Cooper family."
Colin Hart, sports journalist and good friend of Sir Henry, told Sky News: "I'm not shocked he died, sadly, because I saw him deteriorate over the years as he got quite ill. He wasn't the same after the death of his wife (Albina) who was not only his right hand but also his left hand.
Sir Henry
Sir Henry lost to Muhammad Ali twice in his career
"He died of a broken heart."
Robert Smith, the general secretary of the British Boxing Board of Control, described Sir Henry as "one of the sporting icons, not just for the boxing public but sport in general".
Mr Smith continued: "He fought Muhammad Ali twice, once when he was Cassius Clay and once when he was Muhammad Ali, and he put up wonderful performances.
"Ali is possibly the greatest athlete there's ever been and Henry put up a great performance and just wasn't quite good enough on both occasions - but he's not the only one who wasn't good enough to beat Ali."
World Boxing Association light-welterweight champion Amir Khan told Sky News boxing is a "very respectful sport".
"Even though you go into the ring and have two guys beating each other up, afterwards we are friends, and I think Henry Cooper made a great example of that, especially how close he was with Muhammad Ali in the years after that fight.
"I look up to people like Henry Cooper for what he did for the sport and the way he was and the way he was outside the sport," he added.
Sir Henry is survived by his two sons Henry Marco and John Pietro.