I feel the same way about Phelps not necessarily being the best Olympian ever. He's the most decorated Olympian, that's for sure, but the best? That's questionable and debatable just like Coe and Rogge said. He's definitely a contender... as I would consider Usain Bolt if he were to win everything he entered this year...among others.
Like others have said, swimming gives far more opportunities to win, you can compete iin as many races as you like, provided you qualify. Gymnastics is another. Other sports don't have that to the same extent. I think if a person dominates the sport they are in for multiple games, or if they have achieved feats in ways never done before (like Bolt's fast 100m) that puts them in the running too.
Sometimes the most titles, medals, championships don't equal greatest ever.