Real, who lost to Moldovan minnows Sheriff Tiraspol on Tuesday, blew a chance to pull ahead of Atletico Madrid, who beat crisis-hit Barcelona 2-0 on Saturday to draw level with their city rivals on 17 points.
Ancelotti said Real had played “our worst match of the season”.
“I agree, we played badly,” the Italian coach added. “We now need to use the international break to think about why this team’s attitude has changed in the space of a week.”
Espanyol took the lead on 17 minutes at the RCDE Stadium in Catalonia when Raul de Tomas found the net.
Aleix Vidal scored a sparkling second goal after nutmegging Nacho and smashing the ball past Real ‘keeper Thibaut Courtois 15 minutes into the second half.
The defeat leaves Real without a win in three games after they were held to a goalless draw by Villarreal in their previous league match.
Espanyol had won just once in seven games this season but they hung on grimly in five minutes of added time to take all three points.
Elsewhere in La Liga, Sevilla went down 1-0 to Granada. Getafe earned their first point of the season with a 1-1 draw against Real Sociedad.
Elche beat Celta Vigo 1-0. Villarreal were 2-0 winners over Real Betis.
In the German Bundesliga, Bayern Munich were beaten 2-1 by Eintrecht Frankfurt.
Bayer Leverkusen beat Arminia Bielefeld 4-0 to go level on points with Bayern at the top of the table.
Napoli stay top in the Italian Serie A with a 2-1 win over Fiorentina. AC Milan are second after beating Atalanta 3-2.
In France, there was a surprise defeat for Paris Saint-Germain who lost 2-0 at Rennes.