He joined the Navy in 1935 after graduating from high school. He served for nearly ten years and was ranked gunner's mate first class. He received many awards including the Navy Good Conduct Medal, WWII Victory Medal, Lone Sailor Award, Veteran of the Year, and the California Commendation Medal.

After his time in the service he lacked a career, and briefly worked in a factory. Unsatisfied with the work, his mother encouraged him to pursue acting. He attended the Randall School of Drama in Hartford and later moved to Virginia to work within the Barter Theater. After two years, he went to New York in 1949 where he performed in the Broadway play Harvey.

He appeared in his first film, The Whistle at Eaton Falls, in 1951 and continued film work for the next thirty years including roles in Flight of the Pheonix, The Dirty Dozen, Emperor of the North, Escape from New York, and most famously, The Wild Bunch. He also appeared in numerous television programs including Little House on the Prairie, The Love Boat, Magnum P.I., Murder, She Wrote, Walker, Texas Ranger, Home Improvement, Touched By an Angel, ER, and Wagon Train. 

Ernest married five times and had three daughters and one son in total. He died of kidney failure on July 8, 2012 at age 95.