He was born one of ten children. Father Walter was a Pastor and Jimmy grew up a devout christian. As a teenager Ellis worked in a cement finishing factory. He also had an interest in music and became a good singer in the local Church choir, years later wife Mary would join him. He continued the Church involvement all his adult life. He also admirred Joe Louis.  Ellis got into boxing as a teenager after watching a friend box fellow Louisville youngster Muhammad Ali on a local amateur boxing television show called Tomorrow’s Champions. “I had a friend of mine named Donnie Hall, and he fought Ali,” Ellis said. “Donnie lost, and I thought I could maybe be a fighter then.” Ellis went with Hall to Louisville’s Columbia Gym, where the coach was a police officer named Joe Martin.  Ellis won 59 of 66 amateur bouts and was a Golden Gloves champion. He boxed Ali twice as an amateur, with Ali winning the first bout and Ellis winning the second.

Ellis turned professional as a middleweight in 1961. Early in his pro career, he was trained and managed by Bud Bruner. With Bruner, he compiled a record of 15–5 (6 KOs). His five losses were decisions to top Middleweight contenders Holly Mims (whom he defeated in a rematch), Henry Hank, Rubin “Hurricane” Carter, Don Fullmer, and George Benton. This start probably helped his speed of punch, movement and finesse.  At the end of 1964, after losing three out of four fights, Ellis decided to leave Bruner. He later recalled Bruner fondly. “I liked him, and I fought a lot of top-rated fighters when I was with him, but eventually I had to move on,” Ellis said. “He did me justice, and we always remained friends.”  Ellis wrote a letter to an at first skeptical Angelo Dundee, the trainer of Ali, and asked him to handle his career. Dundee agreed to be both manager and trainer. Ellis became a sparring partner for Ali and fought on several of his early pre-Liston undercards. Six of his first eight fights with Dundee were on an Ali undercards.  Buy the mid 1960’s Ellis was fighting heavyweights. Being a tall natural athletic build he’d had increasing trouble keeping down to middleweight. Ferdie Percheco MD, who worked with both Ali & Ellis throughout their careers, called Ellis’s development from middleweight to heavywweight one of the most dramatic he could recall.

By 1966, Ellis was fighting as a heavyweight. When Ali was stripped of the world title for refusing to enter the military, the World Boxing Association staged an eight-man tournament that featured most of the top heavyweight contenders. Ellis, who was ranked eighth in the world after eight consecutive wins, was invited to be in the tournament. Joe Frazier, ranked second by the WBA, chose not to participate in the tournament. Instead, Frazier fought for the vacant New York State Athletic Commission World Heavyweight Championship, which he won with an eleventh-round knockout of Buster Mathis.  In the opening round of the tournament, Ellis fought Leotis Martin on August 5, 1967 in Houston, Texas. Ellis, the betting underdog, battered Martin’s face into a bloody mask, and the referee stopped the fight in the ninth round.  Ellis met Oscar Bonavena in the second round of the tournament. The fight took place on December 2, 1967 in Louisville, Kentucky. Ellis, once again the underdog, dropped the iron-jawed Bonavena with a right once in the third round and once in the ninth from a truely wicked left hook. After twelve rounds, Ellis was awarded a clear unanimous decision. He controlled the match throughout with perhaps the best display in his career. He made Oscar look basic and was only really in trouble himself in the ninth. But ironically, he turned the ninth around with a split second counter catching Oscar wide open and decking him badly. Ellis advanced to the tournament final.

In the tournament final, Ellis faced Jerry Quarry, a slight betting favorite, on April 27, 1968 in Oakland, California. In a dull fight, Ellis fought what Sports Illustrated called “a tactical masterpiece”. But he was very cautious and won a fifteen-round split decision  to capture the vacant WBA Heavyweight Championship. Quarry said, “If they’d given me the decision, I’d have given it back. I didn’t deserve it.”

In his only successful title defense, Ellis defeated Floyd Patterson by a controversial fifteen-round decision on September 14, 1968 in Stockholm, Sweden. Ellis, who suffered a broken nose in the second round, was awarded the decision by the referee, the sole judge. Many in the crowd of 30,000 disagreed with the decision and started chanting, “Floyd champ!” The New York Times scored the fight seven rounds to six for Ellis, with two even.  Following the defeat of Patterson, Ellis was out of the ring for seventeen months. He was going to fight Henry Cooper in the United Kingdom, even though the British Boxing Board of Control refused to recognize the fight as a world title bout: the BBBofC was affiliated with the World Boxing Council, who stated that they would only recognize a fight between Joe Frazier and a suitable contender as being for the world title. The fight was postponed a couple of times and eventually cancelled because Cooper injured his knee. Ellis then planned to fight Bob Cleroux in Montreal, but Cleroux lost what was supposed to be a tune-up fight against the lightly regarded Billy Joiner. Finally, Ellis was going to fight Gregorio Peralta in Argentina, but promoters canceled the fight 24 hours before it was to take place because of poor ticket sales.

On February 16, 1970, Ellis fought Joe Frazier to unify the World Heavyweight Championship at Madison Square Garden in New York City. The undefeated Frazier, a heavy betting favorite, proved to be too strong and powerful. Ellis, who had never been floored as a heavyweight, was knocked down twice in the fourth round by a relentless Frazier, and Angelo Dundee stopped the fight before the start of the fifth round. It was the first knockout loss for Ellis.

After winning his next three fights, Ellis fought Muhammad Ali in the Houston Astrodome on July 26, 1971. Angelo Dundee chose to work with Ellis for the fight. He was Ali’s trainer, but he was both manager and trainer for Ellis. Working with Ellis meant that he would get a bigger share of the purse. Ali understood completely and got Harry Wiley, who had worked with Henry Armstrong and Sugar Ray Robinson, to be his trainer for the Ellis fight. It was one of the few fights in Ali’s career in which Dundee was not in his corner.  Ellis fought well over the first three rounds, but the fight turned after Ellis was hurt by a right hand in the fourth round. The right hand “hurt me so bad I couldn’t really fight my best after that,” Ellis said. “It ruined me.” The referee stopped the fight in the twelfth round as Ellis, who remained on his feet. No knockdowns throughout the fight.

After the loss to Ali, Ellis won his next eight fights by knockout. But on June 18, 1973, he fought Earnie Shavers, who was 44–2 (43 KOs), at Madison Square Garden. Ellis, ranked fourth by the WBA, stunned Shavers with a chopping right to the jaw and backed him into a corner. Shavers took numerous shots in the corner before clinching. After the referee separated the fighters, Shavers put Ellis down for the count with a wickedly powerful single right uppercut to the chin. The time was 2:39 in the first round. It was a stunning win for Shavers.  Ellis came back with a knockout win against club fighter Memphis Al Jones, but with his skills in decline, he went winless in his next five fights. He lost a split decision to Boone Kirkman, fought a draw with Larry Middleton, dropped decisions to Ron Lyle and Joe Bugner, and was stopped in nine rounds in a rematch with Joe Frazier.  The rematch with Joe Frazier took place in Melbourne, Australia on March 2, 1975. Ellis won the first three rounds, but Frazier then picked up the intensity and took control. With Ellis bloody and battered, Angelo Dundee signaled for referee Bob Foster to stop the fight in the ninth round.

On May 6, 1975, in what would be his last fight, Ellis knocked out club fighter Carl Baker in the first round. He retired aged 35 after suffering a training injury that left him partially blind in his left eye. Ellis finished with a record of 40–12–1 (24 KOs).  After retiring from boxing, Ellis trained boxers. Later he worked for the Louisville Parks Department on athletic and recreational projects between 1989 and 2003.

In 2004 Ellis told the Washington Times “…All I ever wanted to be was a good fighter and good man.’ He seemed to achieve it. Brother Jeff gave a tribute on his death saying ” He was someone you could model yourself on” Ellis was a reserved family man who shunned flash although had a determined competitve streak in boxing.  With wife Mary he had six children,2 sons and 4 daughters. His brother Charles boxed in the 1964 Olympics. Ellis was personally kind and gracious. He maintained a brotherly relationship with Ali over all the decades. Ali himself often recalled Ellis as a great freind. Ellis wasn’t always pleased by the sparring partner tag but felt he had proved himself above that.  He suffered from dementia pugilistica, for over decade before his death. It was reported that Ellis’ condition was so bad that he believed his deceased wife, Mary who died in 2006, was still alive.

Ellis died of complications from dementia on May 6, 2014, in Louisville, Baptist Hospital, Kentucky. A tribute came in from Muhammad Ali; ‘In the world of heavyweights I always thought him one of the best. Ellis’s family considered that the boxing exacerated the demenia but didn’t necessarily cause it. His younger brother Jeff, who’d trained with Ellis in years past, commented that he himself now avoided watching boxing as he’d seen too many damaged by it. Ellis was survived by three brothers and a sister.[33] Son Jeff payed professional football and confirmed the family were always immensely proud of Elli’s achievements and world title.