The earliest memory I can recall, was when I was in preschool building a block tower with my cousin who was also in my preschool class. Another kid in the class came over and knocked down the block tower with a plasic dinosaur. Since I was in preschool I must've been 3 or 4 years old. I probably remember it, because it made me really upset at the time. I feel like I'm the same person, however I have obviously grown up since then.
I too have a similar memory with blocks (this is an engineering school). While I don't remember our tower falling, I remember how tall we made it and how we somehow kept it balanced on wheels. Perhaps we both still feel like the person we were just because both of us have used blocks plenty of times after this experience and our memories may be part fabrication, part real memory.
