Why is it so hard to let go?

It's because you will always wonder what could have been. Also, I'm sure that you want your feelings validated, to know that he cared for you too. Maybe you just need to tell him, and then you can move on from there (either something really great will happen and you will be happy and glad you took the chance, or nothing will happen, but you will find out how he feels/felt, and you can begin to move on.) Better to take the risk and be honest and not spend the rest of your life wondering!