David Moyes' side were outplayed in Florence and started the second leg 2-0 behind and with it all to do.
They hit back to win 2-0 on the night to force extra-time and then penalties, but Yakubu hit the post and Phil Jagielka saw his effort saved by Frey as Fiorentina won 4-2 in the shoot-out.
Lee Carsley and Leon Osman replaced Tony Hibbert and Tim Cahill in Everton's starting line-up, while the Italians, currently fourth in Serie A, were once again without injured striker Adrian Mutu, so Christian Vieri led their attack.
Fiorentina, beaten UEFA Cup finalists in 1990, rode their luck in the opening stages and it came as little surprise when they fell behind in the 16th minute.
Steven Pienaar's cross should have been easily gathered by Frey, but the Fiorentina keeper made a hash of taking the ball, and Andy Johnson was able to bundle it into the empty net from five yards out.
The goal was just what Everton needed although, as at Sunderland on Sunday, Johnson appeared to force the ball over the line with his shoulder.
Five minutes later Mikel Arteta bent his free-kick around the wall, but Frey got across his goal well to make a diving save down by his post.
Yakubu then fashioned a chance for himself with some tricky footwork but Frey did well to save the former Middlesbrough and Portsmouth man's shot.
As Everton continued to press, Arteta's corner found Johnson inside the six-yard area but the striker sent his header over the bar.
Johnson had another shot superbly saved by Frey and, although Arteta gathered the rebound and turned the ball to Yakubu who found the net, the Spaniard was clearly in an offside position.
Everton continued to dominate proceedings after the break and Arteta's low corner caused chaos in the visitors defence with the ball eventually hitting Martin Jorgensen on the line.
Everton finally scored the second goal their pressure deserved in the 66th minute. Arteta was fed the ball just inside his own half, advanced a few paces and unleashed a superb low right-footed piledriver that beat Frey as it found the corner of the net.
There were no more goals in the last 24 minutes of normal time or extra-time, but Fiorentina kept their cool in the shoot-out to seal their place in the last eight of the competition.