Lem's criticism (and I am paraphrasing liberally) is that SF was trivial pulp literature, produced in volume for an unsophisticated and undiscriminating audience. His criticism of fandom was that they demanded so little of SF writers (other than continuous and copious output) that the writers had little incentive to produce works of lasting value. Lem went on to say (and he named names) that despite some writers' pretensions to being serious writers, their work was in fact poorly structured, unimaginative, and lacking internal logic.
To quote from the liner notes of Microworlds, a book of Lem's essays on SF, "Lem writes with polemical passion about what he regards as science fiction's squandered potential; he sees it as bogged down in a rehash of myth and fairy tales. Too often, says Lem, science fiction resorts to well-worn patterns of primitive adventure literature, plays empty games with the tired devices of time travel and robots, and turns its back on time-honored cultural and intellectual values."
Needless to say, this did not go down well with some of the more self-important members of the SFWA. Phillip Jose Farmer was one of the most vocal in demanding that Lem's honorary membership be revoked. Ursula K. LeGuinn, among others, protested and demanded that Lem be reinstated. When Lem was invited to return to the SFWA, he refused. Not SF's finest hour.