Baroni was born Phillip George Baroni in Massapequa Park in Long Island, New York to American parents. His father was a prominent New York attorney and his mother was a homemaker. Phillip attended Massapequa High School where he was on the wrestling team.
Baroni grew up having multiple street fights, a precursor to his career. The Baronis were next door neighbors of Jerry Seinfeld’s parents. Baroni attended Nassau Community College and became a two time All-American wrestler there, placing fifth and second in the nation. However, during the later part of his senior year he was expelled. Baroni then attended Hofstra University before transferring to Central Michigan University on a scholarship, where he was a double major in biology and psychology. He was also an amateur bodybuilder from the age of 17 to 20, competing in six shows, in which he placed either first or second. He studied kickboxing under Keith Trimble, and had seven matches, winning all by knockout. He also competed in 10 amateur boxing matches, also winning all 10 by knockout.