It sounds like you didn't have to pick up younger siblings at school when you were younger. That's the reality for a lot of poor kids. Raised by a single parent, the older siblings are in charge of looking after the younger ones. The parent often does not get home from work until late (and may have multiple jobs).
I'm not saying the poor aren't disadvantaged. It's just that the disadvantages are more along the lines of attitudes towards education in the first place (along with everything else). There are also fewer people nearby that serve as an "example" on how to learn or get ahead.