Abercrombie was born and baptized in Baltimore, Maryland, although some accounts suggest he was a native of Tennessee. The son of John Joseph Abercrombie, Sr. and Sarah DeNormandie, their family was living in Nashville, Tennessee when the younger John entered the United States Military Academy in 1818. Graduating 37th of 40 from the United States Military Academy in 1822, Abercrombie’s class included future Union Army generals Joseph K. Mansfield, David Hunter, and George A. McCall, as well as future Confederate Army general Isaac R. Trimble.  Abercrombie began his long military career with garrison duty at Baton Rouge, Louisiana, until 1825. He became the adjutant of the 1st U.S. Infantry and was assigned administrative duty at regimental headquarters. He was promoted to first lieutenant in September 1828 and served in the Black Hawk War against the Sauk Indians in 1832. He was then on garrison duty in Illinois and Wisconsin.

Seeing additional combat action as a captain during the Seminole Wars, Abercrombie was brevetted a major for gallantry at the Battle of Lake Okeechobee. Serving on frontier duty throughout the 1830s and 40s within the South and Northwest, Abercrombie fought again during the Mexican-American War, where he gained distinction at the Battle of Monterrey and was promoted to lieutenant colonel.  Following the war, he married Mary Engle Patterson, the daughter of General Robert Patterson, a wealthy Philadelphia merchant. They would have nine children.  He founded Fort Abercrombie in what is now North Dakota in 1857, selecting the site and supervising construction of the wooden fort. He then commanded the garrison until promoted and reassigned to duty in Minnesota.

By the start of the Civil War, at the age of 63, Abercrombie was one of the oldest field officers in both the Union and Confederacy. He served as the commanding officer of the U.S. 7th Infantry in Minnesota with the full rank of colonel in the Regular Army.  Promoted to brigadier general of Volunteers in the Union Army on August 31, 1861, Abercrombie commanded troops under his father-in-law Robert Patterson in the Shenandoah Valley and at the Battle of Falling Waters. He was reassigned as commander of the I Corps’ 2nd Brigade, attached to the Army of the Potomac. He led the brigade throughout the Peninsula Campaign in 1862, during which he was wounded at the Battle of Seven Pines. During the Battle of Malvern Hill, his command assisted in repulsing the Confederate attack against Union forces. His unit later participated in several skirmishes during the Federal retreat to Harrison’s Landing.

After the conclusion of the Peninsula Campaign, Abercrombie was replaced as commander in favor of younger officers and spent the next year involved in the defense of Washington, D.C., and, in 1864, headed various Union supply depots in Virginia during the Overland Campaign. Abercrombie commanded the troops that defended a depot in Fredericksburg, Virginia, in June 1864 against an attack by Hampton’s Legion.  On March 13, 1865, Abercrombie was brevetted a brigadier general in the Regular Army in recognition of his long service. Abercrombie retired from active duty on June 12 after commanding Fort Schuyler in New York.  In his retirement, Abercrombie stayed associated with the U.S. Army. He served on court-martial duty for the next 3 years.  Abercrombie died at the age of 78 in Roslyn, New York. He was buried in The Woodlands Cemetery in Philadelphia, Pennsylvania.