My earliest memory is not very detailed. I remember being in my back yard, next to our storage shed while my dad was cutting up wood, so the circumstances weren't very extraordinary. I think the wood was from a tree we had taken down in our backyard, which I might have found exciting but I dont remember the tree coming down at all (just vaguely remember the wood coming from a tree in our yard). I was probably around 4 or 5. It is such a vague memory that it doesn't really feel connected to the person I am now. I remember lots of random things from that time period (since we were living in a different house then) but don't really remember specific events.
A difference in my earliest memory was that it was filled with lots of emotion. I think all of the emotions helped me remember the circumstances very vividly. Maybe if you tried to think of a time when you were really upset, or really happy, you would be able to remember more specific details.
I have a similar memory. We used to have a huge tree in the sideyard (I was 3-4), I can remember it because of how large it was. Then at one point (4-5) we had the tree removed, and I look at the side yard and still to this day remember the tree being there, but for some reason I cannot remember the day we cut it down.