Livre d'Or

Sometimes if these people act as if they like me more than I like them, I get into a negative cycle of getting annoyed with them because they make me feel guilty for disliking them for no moral reason.
[...]
I know in theory that moral qualities are more important than intelligence, but I have to admit I'm unlikely to be able to sustain interest in someone who isn't pretty spectacularly bright.

There seems to be a common theme here that some part of you feels as if you "ought" to have a different basis for deciding which people you like (and perhaps even love) than the one you in fact do have. It sounds from these comments as if you feel that you ought, morally speaking, to be taking an attitude to potential friends which is reminiscent of equal-opportunities employment law – if not more so, so that qualities such as intelligence and interesting conversation which quite reasonably make a big difference to whether or not you actually enjoy someone's company are things that you feel guilty for discriminating on.

Have I misunderstood, or exaggerated to the point of actual inaccuracy?

If I haven't, I venture to suggest that enjoying one another's company is pretty much the purpose of friendships and relationships, and hence that it's perfectly OK for you to choose your friends based on whether you enjoy being friends with them.