Effectively you can visualize a "calendar tree". So, at the end of 2024, the user can add a summary note to the "2024" node. They can also do this at month level ("2024 Septermber"). Or at decade level. Sometimes they may want to add custom parent nodes spanning certain date range (if year/month/decade granularity is rigid for the use case). In the end, you just have a "tree of memories".

In the end, I recommend keeping your app's UX as simple as possible.