I think his family was annoyed that he didn't quit any of his vices.
In a way I suppose I can understand the families thoughts, but to deny Sinatra these pleasures I fear would have killed him much earlier simply through boredom, that was his lifestyle and his life.
How ever he ended up be it with teleprompters etc... he was still Mr Cool and bloody great.