After being sent in to bat, India stumbled to 87 for five before Rayudu (124 not out from 133 balls) and Stuart Binny (77 from 76 balls) put on an Indian record 160 for the sixth wicket to help the tourists reach 255 for six in their 50 overs.
Zimbabwe battled to keep up with the rate on a slow wicket but late fireworks from captain Elton Chigumbura (104 not out from 101 balls) saw them need 10 off the final over.
They reduced that to six from three balls but Bhuvneshwar Kumar bowled brilliantly to concede just a single from there and India could breath again having restricted the hosts to 251 for seven.
Chigumbura waited 174 matches for his maiden ODI ton and now has two in as many innings but will be desperately disappointed not to have got his side over the line.
The day belonged to Rayudu as he reached his second ODI century off 117 balls, with only 22 of those runs coming from boundaries, and has now scored 225 runs in four innings in Zimbabwe with just two dismissals.
He was superbly supported by Binny, whose run-a-ball knock defied the conditions with the pair doing most of the damage as India scored 90 runs in the final 10 overs.
Binny continued his fine match with two for 54 with the ball but left-arm slow bowler Axel Patal (two for 41) was the pick of the tourists’ attack.
Off-spinner Harbhajan Singh, playing his first ODI in over four years, returned figures of one for 46 in his 10 overs.
The second ODI of the three-match series will be on Sunday at the same venue, with the teams also playing two Twenty20 games starting next Friday.
(Reporting by Nick Said; editing by Ken Ferris)