Banks can be a kind of hit or miss at times, but I would still recommend reading all of them simply because it's not his writing so much as the world building I found most interesting about The Culture series. He creates this fascinating post-scarcity, post-species (i think), post-gender kind of society and then uses different characters from different walks of life to sell that society to the reader in the form of The Culture's recruitment process. In a way, the books become a kind of allegory for a Culture recruitment process of earth, I guess.

But also I think there's only couple books that really callback to each other so if you're not feeling one book move on and then come back to it later if you're more interested in The Culture's background.

Excession is my personal favorite. Focuses heavily on an existential threat to the Culture and dives very very deep into the AI politics that keeps it chugging along. Consider Phlebas is more of a "outsider looking in" type book to the Culture, and doesn't really dive in to the Culture itself.

As others have said, consider phlebas is an acquired taste, particularly poor as a first book in the series before you really understand what the culture is.

IIRC the second book published is an anthology, and some of those stories are good (but IIRC only two are explicitly in the setting, while the others are sci-fi that may or may not be). The Player of Games is generally considered the first good book in the series, detailing a shitty gamer bro douchebag getting coerced into going off to take part in a tournament in a highly stratified, chauvinist empire as a propaganda display, and generally growing as a person over the course of it (the book also deals heavily with gender, contrasting the egalitarian and genderfluid-as-a-norm Culture with the deeply sexist and chauvinist empire most of the story takes place in).

Unfortunately the first three books are the only ones I've read. I started the fourth one but never finished it, mostly because I read less than I should and partly because I'm not sure where it is.