what are the Real benefits of a "prestigious" residency Program?

I am applying for a pretty non-competitive specialty, just got Step 2 back and did way better than expected with a 268. Only shared this with close friends and family, but the few doctors I know personally immediately started telling me I have to rank prestigious programs high, move away, go to a big program, go to a big city, and that it's worth the sacrifices to do so. Is it really?

I always intended to stay close to home (not a big city) due to personal/family-related reasons. To go to the more highly ranked/"renowned" residencies in this speciality, I would have to move away. Prior to hearing about this score no one pressured me about it, even though my resume was good. But the few people that know about the score are now really laying it on about how where I train will affect my salary, negotiation power with jobs, influence, opportunities, etc. One doctor was like "this is your one shot in life and you'll regret it" which freaked me out as if I'm making some major mistake lmao.

What would I really be missing out on? I personally haven't changed my mind about what I want, but I'm also not super informed about the differences and how it would affect me as an attending practicing back at home.

ETA idek why the capitalization of the words in the title are erratic plz ignore that