I always figured it was because you experience time relative to how long you have already lived. One year seems like forever when you are 5, because it's 20% of your life. One year at 50 is only 2% of your life, so it seems shorter.