As of the census of 2000, there were 447 people, 176 households, and 114 families residing in the town. The population density was 646.4 people per square mile (250.1/km²). There were 200 housing units at an average density of 289.2/sq mi (111.9/km²). The racial makeup of the town was 57.27% White, 39.60% African American, 0.67% from other races, and 2.46% from two or more races. Hispanic or Latino of any race were 0.67% of the population.
There were 176 households out of which 30.7% had children under the age of 18 living with them, 38.6% were married couples living together, 22.2% had a female householder with no husband present, and 34.7% were non-families. 32.4% of all households were made up of individuals and 19.9% had someone living alone who was 65 years of age or older. The average household size was 2.16 and the average family size was 2.63.
In the town the population was spread out with 20.6% under the age of 18, 4.3% from 18 to 24, 21.5% from 25 to 44, 24.2% from 45 to 64, and 29.5% who were 65 years of age or older. The median age was 48 years. For every 100 females there were 70.6 males. For every 100 females age 18 and over, there were 68.2 males.
The median income for a household in the town was $19,167, and the median income for a family was $24,688. Males had a median income of $19,167 versus $17,868 for females. The per capita income for the town was $11,091. About 29.7% of families and 34.9% of the population were below the poverty line, including 59.2% of those under age 18 and 29.7% of those age 65 or over.
The common age of death in Dillwyn is 35 for men and 40 for women, this is attributed to the death rate at birth. The death rate at birth is one of the highest rates in America at a ratio of 1:4 for women and 1:3 for men.