An interpret can never be right.
But, I think's it's about the "madmen", as Bowie's brother Terry.
"They send my friends away, to mansions cold and grey." Mental institusions...
"'Cause I'd rather stay here
With all the madmen
Than perish with the sadmen roaming free
And I'd rather play here
With all the madmen
For I'm quite content they're all as sane
It's like he feels that these mad-men, his friends that "they" send away, is not mad, because he doesn't feel that he himslef is. So he'd rather be with them, than the sadmen, that's not been send away. You know, they're sad just because they're not "mad", and the mad-men is fun. That's why they're sad-men. If they were fun, they would have been mad-men.
I mean, it's just a song about what the defenition of "mad" is. How can you tell who's mad, and who's not? I get the feeling he doesn't understand why they send these people away.
The album is very much inspired by his "mad" brother, and the alternative cover, is a cartoon of the "cold mansion" his brother stayed at.
"Gimme some good 'ole lobotomy" -as they did on the mad-men.
But a writer can always have more than one inspiration to a song, I don't think it's so important to the writer himself.
So I mean, there's alots of options. You're interpret could work!
"After All" is more a song about the hippies, he sings about that they're "just taller children, that's all." They're like everyone else.
Although I hold the more common view (that All The Madmen is a 'Terry' song), I think your interpretation has merit.
Songs like 'Battle for Britain', 'Aladdin Sane', 'Watch that Man', and 'Soul Love' seem to be centred on personal themes but with a back drop of war perhaps for dramatic effect. Thus in 'All the Madmen', I cannot imagine that the narator would physically 'perish' from mere association with sane (sad) people roaming free, but it certainly sounds a lot more dramatic to put it that way.
'They send my friends away to mansions cold and gray' as mentioned by Lady Androgene would be the institution styled asylums that were popular in in the 19th and early 20th centuries (and particularly around the London outer suburbs).
Bowie's brother Terry was committed to of one of these places - Cane Hill Mental Hospital. Then whilst TMWSTW was being demoed and written in a basement studio at Haddon Hall, Terry was commuting between those two locations - staying with Bowie on weekends but returning to the mad house during the week.
On the original cover of the album, we can see a picture of the asylum:
Cane Hill today (it's now closed down and abandoned):
Bowie in Australia 2004 | Join the Community
You know, the funny thing is, I've actually always liked the Terry-interpretation better myself. But still... I don't know. Actually my boyfriend was the first person who said that it could be an anti-war thing. And the more I thought of it, the more it seemed to make sense. But I guess there can be more than one interpretation!
And yes, After All is deffinately a hippie-song. And I LOVE it! It's the cutest song I've ever heard and also strangely sad at the same time.
Day after day they take some brain away...