According to the lyrics on Dylan's official website, it seems to be an indeterminate number of people led by someone referred to as The Phantom of the Opera. (This is just one of the many references to the people in the song by comparing them to literary figures. Dylan himself probably knew who he meant by each of them, the rest of us can only guess.)
"Across the street they've nailed the curtains
They're getting ready for the feast
The Phantom of the Opera
A perfect image of a priest
They're spoonfeeding Casanova
To get him to feel more assured
Then they'll kill him with self-confidence
After poisoning him with words
And the Phantom's shouting to skinny girls
'Get Outa Here If You Don't Know
Casanova is just being punished for going
To Desolation Row'"
http://www.bobdylan.com/us/songs/desolation-row
