“He abandons,” an Etixx-Quick Step spokesman said of the German.
Briton Chris Froome, Team Sky’s 2013 Tour winner, will take over as race leader going into Friday’s seventh stage, a 190.5 km ride from Livarot to Fougeres.
He leads American Tejay van Garderen (BMC Racing) by 13 seconds.
After a few days of crashes and nervous racing the peloton took it relatively easy on the Normandy coastline until the last kilometre when Martin lost his balance and several leading contenders followed him down.
Among the casualties were Italian Nibali, Colombian Nairo Quintana and van Garderen as Martin’s team mate, Zdenek Stybar of the Czech Republic, sprinted to victory in the 191.5-km leg from Abbeville to Le Havre.
Nibali had bruises on his knee while Quintana had blood on his arm.
Quintana’s Movistar team mate Alejandro Valverde also crashed earlier in the stage, hurting his leg and buttocks. Both will continue the race, the team said.
Slovakian Peter Sagan (Tinkoff-Saxo) was second in the stage and France’s Bryan Coquard (Europcar) took third place.
Regulations state that on a flat stage a rider held up in an incident in the final three km is credited with the same time as the winner.
That meant Martin retained the overall lead, until he withdrew, after being pushed by two team mates to the finish line.
“I was unlucky. I don’t even remember how I went down. I touched the rear wheel of the rider in front of me (Coquard) but it’s the Tour, luck and bad luck are always close,” Martin told reporters.
Swiss Fabian Cancellara, who held the overall leader’s yellow jersey earlier in the event, had to pull out because of a back injury after crashing on Monday.
Froome narrowly escaped Thursday’s carnage, managing to keep his balance while Nibali leaned on him as he fell.
Spain’s Alberto Contador (Tinkoff Saxo) is seventh, 36 seconds behind, while Nibali (Astana) is 12th and 1:38 adrift. Quintana lies 17th, 1:56 off the pace.
“I don’t know why this happened like this 500 metres from the line. I’m ok, my shoulder and my leg touched the ground,” said Nibali.
Daniel Teklehaimanot became the first African to wear the polka dot jersey for the mountains classification after the MTN-Qhubeka rider grabbed precious points in the three minor climbs of the day.
(Editing by Tony Jimenez/Alan Baldwin)