I think the best popular-experimental albums generally shouldn't even sound experimental any more. After all, the most important form of experimentation I think is clearly that which expands the boundaries of that which can be considered 'popular music'.
The first two Roxy Music albums are thus strong contenders, with the marriage of Ferry's relatively straight pop/rock leanings and Eno's ambitions in crafting sound-scapes, ambience and left-field pop. Their music was wildly experimental yet also very popular, also paving the way for the future popularity of - arguably even finer acts - the Talking Heads and Joy Division.
Though if I were to name one album, based on Adam's criteria, it would be Brian Eno's full-length solo debut, Here Come The Warm Jets, full of new sounds, complex off-kilter structures and free-associative song-writing all the while boasting a perversely pop-heavy sensibility. Thirty years down the line, while still exciting in the creative process, one of it's biggest triumphs is that it would sound right at home alongside the work of the Super Furry Animals, or the mid-nineties experimental-side of Blur. Many albums pertain to 'push the envelope', but Here Come The Warm Jets is an album which, in retrospect, smugly speaks for itself in having succeeded.
The Beatles of course do deserve a big honourable mention, for getting some of the more experimental albums of the sixties to the top of the charts. With The White Album they invented the commonly accepted "rules" of the double-album, and while Sgt. Pepper's is discredited more and more with every passing year it nontheless remains the point in mainstream pop where it was announced okay to tackle any genre, and with as much whimsy as you'd like. Though Revolver is the album that really deserves The Beatles nod in my mind, intoxicatingly experimental in it's eclecticism, approaches and song-writing matter. To fail to get caught up in the creative excitement of "Tomorrow Never Knows" - which still sounded cutting edge when The Chemical Brothers re-interpreted it, some thirty years later - is to lack appreciation for popular-experimental work, period.
The Beach Boys also deserve a warm round of applause, Brian Wilson practically defines the whole nature of popular-experimental music, being an artist constantly driven to experiment and create but always with the intent of making music that would be beautiful. The Beach Boys were often complex, but never challenging, and more fool anyone who finds that a flaw. From the incredible harmonies (results of a series of experiments, for all forty years of pop culture can lead one to take them for granted) to the very birth of progressive pop (aka prog-pop) music and incorporations of classical elements: which would sure come in handy on the B-side of The Beatles' Abbey Road, released some three years after Pet Sounds.
Pet Sounds would, of course, be the obvious choice of nomination for The Beach Boys, but arguments about as strong could be made for either The Beach Boys Today! or Smiley Smile, based on preference.
In terms of even Adam's tailor-made-for-Bowie criteria, I'm not sure Bowie has much of a fighting chance against the above choices, or more. I shan't even start with Pink Floyd.
I could be a genius if I just put my mind to it.