I think you're right. I really enjoyed Season 4 but I think more so now that I rewatch it. Seasons 4 and 5 were incredibly depressing- or maybe hopeless is the word I'm looking for- I honestly really loved Season 6 (at least the first half- of course I enjoyed the whole thing but the first half was my favorite) if for no other reason than that it was such a huge breath of fresh air.

It felt like this giant cloud had been lifted and not even soulless!Sam could bring me down :P

Season 5 was very iffy- the quality of the episodes was not always that great, tbh.

But I love watching Sam and Dean's story and I don't think there is anything that would make me quit the show/stop loving it... well, I could think of a few things but nothing that I believe the show would actually do :P