James Lee Sr. born in 1808 died in 1889 married Penniah Gibson in 1830 together they had 3 sons two who lived into adulthood. James Lee Jr born in 1832 and Samuel Stacker Lee in born 1844 or 1845. Stacker married and had one son Samuel Stacker Jr. James Lee Jr married Rowena Bayliss in 1858, together they had 10 children:
- Rosa Lee born in 1859 and died in 1936. Rosa never married. She founded the James Lee Memorial Art Academy which became the Memphis Art Academy.
- Sarah Lee (also known as Sally) born in 1861 and died in 1948. In 1884 she married Sam Phillips and together they had one daughter Lady Lee Phillips. Their marriage ended in divorce. Lady Lee Phillips married Walter Edge who became governor of New Jersey and served as ambassador to France. They married in 1905 and had one child Walter Edge Jr. Lady Lee died 4 days after giving birth to Walter Edge Jr.
- Robert E. Lee born in 1863 and died in 1942. He married Elizabeth Morrow in 1908 and together had one son Robert E. Lee Jr. Robert E. Lee Jr married Mary Johnson in 1930. She had one son from a previous marriage – R. Wilkerson. Robert divorced and remarried and had one son Robert E Lee III. Robert E Lee Sr. was head of the Lee Line from 1895 until 1913.
- Rowena Lee born in 1882 and died March 1968. Rowena married Walter Teagle in 1907. Together they had one son Walter Teagle Jr in 1913. Walter Teagle Sr. became head of Standard Oil of New Jersey (now Exxon Corp.) in his mid 30’s.
- James Lee III born in 1867 and died in 1919. James Lee III later in life called himself James Lee Jr. He married Bodien Warriner daughter of his father’s law partner. Together they had one daughter Sadie Ardinger Lee in 1893, she never married. James Lee III worked for the Lee Line or affiliated businesses for several years and then went into a number of business ventures apart from the family.
- Bayliss G. Lee born in 1870 and died in 1926. He married Minnie Wright in 1917 who had one son Eldridge Wright. Bayliss and Minnie had a double wedding with Shelby R. Lee and Ruth Cochran Moon. Bayliss and Minnie’s marriage ended in divorce. Bayliss worked for Lee Bros and then took over the Bohlen Huse Coal and Ice business.
- G. Peters Lee born in 1872 and died in Jan 5, 1933. He married Edna Hill in 1910. Edna had 2 children by previous marriage, Kenneth Hill and Allien Hill. Together they had one daughter Rowena who never married. Peters worked for the Lee Line and tried to continue in the steamboat excursion business following the Lee Line voluntary liquidation in 1926.
- Georgia Lee. Born in 1874 and died in April 29 1958. She married Robert Parker in 1899. Their marriage ended in divorce.
- Ora Belle Lee. Born in 1876 and died July 31 1927. She never married
- Shelby R. Lee. born in 1880 and died in 1946. In 1917 he married Ruth Cochran Moon and had 3 sons. Shelby R Lee Jr born in 1918 and died in 2005, James Lee IV born in 1921 and died August 1, 1944 Rennes France fighting Fascism, Bayliss Lee born in 1926. Shelby Lee Jr. married Margaret Gunther in 1950 and had 4 sons. Bayliss Lee married Ann Dow in 1953 and had 6 sons. Shelby Lee Sr. worked for the Lee Line until it liquidated in 1926.