Who else would keep the judges in check, give the contestants big bear hugs and entertain us with his own groovy little dance routines?
Dermot X Factor
Video credit: TCILPart3/YouTube
A couple of weeks ago O'Leary first hinted that his X Factor days may be over during an interview with the Mirror's 3am. “We normally sit down February or March but Simon wants to change stuff [this time]," he said. "So I need to see what changes he is going to make before a) I know if he wants me and b) whether I'm suitable for those changes. If he turns round and says 'half your gig is a voiceover,' then I will have to think about it. If he lets me do my thing then of course I would do it."
And today he revealed on Twitter that he wouldn't be returning to the show:
Dermot X Factor Twitter
O'Leary also thanked "ITV, Simon, The X Factor family and particularly the viewers" and had a message for whoever is going to take over from him: "You're about to be the conductor on the most brilliant, runaway train in showbiz. Good luck you'll love it."
But who will that be? There are whispers about former X Factor contestant Olly Murs taking over and he's currently the bookies' favourite for the job. Also in the running are American Idol host Ryan Seacrest, current co-host of The Voice Marvin Humes (also a former contestant), Ant and Dec, former Xtra Factor host Caroline Flack, Holly Willoughby, Zayn Malik and David Walliams.
O'Leary himself has his own opinion about who would be the best candidate for the job. In January he told BBC Newsbeat: "Emma Willis is a good friend and if I was ever not going to do the job I couldn't think of anyone better to do it than her."
What else is Simon Cowell plotting? It looks like anything could happen. Executive producer Richard Holloway recently revealed that "there will be some surprises this year." So Dermot might not be the only one saying farewell to the X Factor.
More entertainment
5 Things we'll always love Downton Abbey for
Cara Delevingne won't be playing the sexy girlfriend on screen
Why did Jessie J go mad at her fans on Twitter?