Club president Fiorentino Perez confirmed the news at a press conference, adding that a successor would be named next week.
“I would like to inform you that the board has taken the decision to relieve Carlo Ancelotti of his post as Real Madrid coach,” Perez said.
“It was a very difficult decision to make; the demands at this club are the utmost because Madrid always wants to win silverware.
“The affection that the players and the fans have for Carlo, is the same as the affection I myself have for him.
“I hope that next week, we can announce the identity of our new coach.”
Ancelotti also won Copa del Rey in 2013/14, and embarked on a record 22-game winning run this season.
However, the wheels came off as they conceded top spot in La Liga to Barcelona, were knocked out the Copa del Rey and crashed out the Champions League in the semi-finals, beaten over two legs by Juventus.
Rafael Benitez, currently in charge of Napoli, is the favourite to take over.