I guess the point is you’re middle aged in regards to your contribution to society. First 15-20 years of your life you pretty much just “take”, while the following 50ish you are expected to chip in. In those terms, 50 sounds about right as being referred to as “middle aged”.
It could actually be way more than that when you consider retirement (in much of the western world at least). You also can’t really “have agency over yourself” in the sense you mean without making use of what society puts at everyone else’s disposal (roads/internet/currency/etc), and freeloading comes with all sorts of drawbacks because society is shaped in a way that doesn’t reward it for obvious reasons.