What was Cain's fault? Why did God reject his sacrifice but accept that of Abel?
I think it was his refusal to bear the tension of holding himself at the only spot where the life of the soul does not get snuffed out - at the border of Chaos and Order. This spot is like the confluence of co-incidences that make life possible on planet Earth.
Can this be proven? Not with any degree of certainty; after all, Jewish and Christian exegetes have struggled with the interpretation of this abstruse passage.
However, there are some clues that encourage me to think that I'm on the right track.
1. God condescends to counsel Cain, and in this part of the story, it seems that God is compassionately urging Cain to get his mindset right. So it's not as though Cain has done wrong out of malice; rather, he has "missed the mark," so to speak, and God urges him to aim better.
2. Cain's response to God's counsel is to produce and nurture resentment in his heart. I notice the same behaviour in myself. This is what makes me think Cain is the fallen everyman. What brings about this resentment? The fact that I think I've done my best and have been treated unfairly by others (i.e., by reality). But I am deceiving myself. I have not really done my best. Part of me wants to do justice to my work, but another part of me shrinks from putting myself in the vulnerable position of no assured success ("I tell you naught for your comfort,/Yea, naught for your desire,/Save that the sky grows darker yet/And the sea rises higher." - The Ballad of the White Horse, by GKC, also cf. The Long Defeat by Tolkien). Thus, pulled between two desires, I am brought to mediocrity. In this situation, I am neither in a mindset of full dedication, nor in one of full dissipation/indulgence. Therefore I think, "Come on, God! I've made sacrifices! Where's my reward?"
As JBP says, it's at the border between order and chaos that we personify the Logos that can create meaning. (Consciousness emerges at this border. [*] )