I'll start off with this (knowing full well I'm going to be crucified, burned at the stake, etc.): I do not really like Rubber Soul. I enjoy the silliness that is Drive My Car and the cowbell in it. I enjoy Norwegian Wood (This Bird Has Flown) even though it is admittedly about John cheating on his first wife (if only he did the same to Yoko). Nowhere Man is alright and has some good harmonies and whatnot. Michelle is a weird song but I don't hate it as much as others here. The rest of the songs are folksy songs that don't really matter.
What does matter is George Harrison, the saviour of the album. Think for Yourself is a lovely song but his best work here is If I Needed Someone full of minor falls and major lifts (and baffled kings composing hallelujah) during the title lines in the song which a guitar to echo Harrison's feelings. This is a truly emotive song that is far too overlooked on the album. The other song, much like Someone is I'm Looking Through You which has a similar guitar style, albeit more acoustic (until the end of the chorus). This is one of the few Lennon songs I could listen to forever. And you can't talk about Rubber Soul if you don't mention the marvelousness that is In My Life: a tribute to getting old and reminiscing. Harmonies abound, it cannot save the album.
And then there's With the Beatles, one of their early albums rife with cover songs and catchy, marketable pop songs but it does have its great sides. Notably All My Loving, which is too adorable and catchy, being the big hit from the album. Early non-politically-driven Lennon is the stuff I prefer and with his songs like All I've Got to Do and It Won't Be Long: two marvelously fun songs.
But don't entirely discount their cover songs. Their version of Money is the most well-known and George does Chuck Berry a nice honor by not entirely destroying Roll Over Beethoven. Even The Beatles' tendency to cover songs by female groups (Devil in Her Heart, Please Mr. Postman, Chains, etc.) end up being successful especially Devil in Her Heart due to its harmonizing. And, as big a Smokey Robinson fan as I am, I really like their cover of You Really Got a Hold on Me.
While not as deep as Rubber Soul, With the Beatles is a fantastically fun and underrated album. I'm more than happy to be the only vote for it.
Winner: With the Beatles