A Black Canadian, Bannister was born in New Brunswick and moved to New England in the 1840s. He took studio art courses at the Lowell Institute alongside artists such as William Rimmer, and soon began to sell his portraits. One of the greatest influences on Bannister's was Christiana Carteaux, a descendant of Narragansett Indians who owned several hair salons in Boston. She married Bannister in 1857 and encouraged his artistic endeavors. The couple were also active abolitionists, and worked with Dr. John deGrasse, Frederick Douglass and William Lloyd Garrison.
While Bannister was well known in the artistic community of his adopted home and admired within the wider East Coast art world (he won a bronze medal for his "Under the Oaks" at the 1876 Philadelphia Centennial), he was largely forgotten for almost a century for a complexity of reasons, principally connected with racial prejudice.