When I joined Carnity earlier this year, I had same question in my mind. I have 4 cars with 3 of family member including me driving those. One of the Car was 2023 Prado GXR 4.0. I started driving Prado last year and did a lot of fun drive with a group of friends - but nothing serious ones. Later when I decided to upgrade my driving skill, I did few modifications in Prado and it is working amazing - Both for Onroad & Off Road. Spend around 30 grands and you can play with the car to intermediate levels. Tune the car for Lift, Bigger rims & tyres, some lights, basics like skid plate, tow hooks, integrated compressor and some recovery equipments.
Now I am done with the thought of buying a separate dedicated car for desert. This car works well for both my kids/ grocery / guest duty - as well as desert drives. I also love driving Prado ditching my RRS for added maneuverability.