That portion (south of the Potomac) was originally ceded to DC by Virginia; it was retroceded by Congress to Virginia.
Virginia also voted to grant the new federal government a one hundred square mile tract of land on the Potomac River for the Federal City in 1789. In 1791, President Washington issued a proclamation establishing the boundaries of the Territory of Columbia. Thirty-one square miles of Virginia were included within the capital territory. In 1801, the territory was redesignated as the District of Columbia. The Virginia portion was organized into a county named Alexandria. (The Maryland portion was organized as Washington County). In 1846, Abraham Lincoln was elected to Congress from Illinois on a campaign plank that called for the abolition of slavery in the District of Columbia. The slave holding residents of Alexandria feared that Lincoln's proposal would be approved. They organized a referendum calling for the retrocession of Alexandria County back to Virginia. The referendum passed, and Congress gave its consent. Alexandria County now consists of Arlington County and part of the City of Alexandria. Congress abolished the slave trade in the District of Columbia as part of the Compromise of 1850. President Lincoln abolished slavery entirely in the District during the Civil War.
http://www.virginiaplaces.org/boundaries/boundaryk.html
Congress passed legislation on July 9, 1846 to return all the District's territory south of the Potomac River back to the Commonwealth of Virginia, pursuant to a referendum that would be held later in the year, and President Polk signed this first piece of legislation the next day. A referendum on retrocession was then held on September 1-2, 1846 and the residents of the City of Alexandria voted in favor of the retrocession, 734 to 116, however, the residents of Alexandria County voted against retrocession 106 to 29. Despite the objections of those living in Alexandria County, President Polk certified the referendum and issued a proclamation of transfer on September 7, 1846. However, the Virginia legislature did not immediately accept the retrocession offer. Virginia legislators were concerned that the people of Alexandria County had not been properly included in the retrocession proceedings. After months of debate, the Virginia General Assembly voted to formally accept the retrocession legislation on March 13, 1847.
In Abraham Lincoln's first State of the Union, delivered on December 3, 1861, he suggested restoring the District of Columbia to George Washington's original boundaries: The present insurrection [Civil War] shows, I think, that the extension of this District across the Potomac at the time of establishing the capital here was eminently wise, and consequently that the relinquishment of that portion of it which lies within the state of Virginia was unwise and dangerous. I submit for your consideration the expediency of regarding that part of the District and the restoration of the original boundaries thereof through negotiations with the State of Virginia. .. both the county AND the town of Alexandria were to pass the referendum. The county of Alexandria never voted in favor of retrocession, only the town voted for it.
http://www.nikolasschiller.com/blog/index.php/archives/2009/02/19/2393/
