The FJ Cruiser is superb offroad however the 'basic'/'retro' interior and limited comforts (no climate control, limited to zero sound proofing, etc) make it a choice for those that enjoy the simple things in life (and have another car to retreat to when needed, such as your Lexus).
As an owner, my hesitation recommending it is 100% as Chaitanya mentioned - finding a clean one on the used market is incredibly hard and the prices are inflated. If you do find one, I am 100% a fan and would say the offraod experience makes up for the on road comfort issues many time over! Also ATrac can be added (after market) to any FJ for minimal cost so if that's missing its not a big deal. Ideally do go for the car with rear Diff lock and 100% avoid any 2 wheel drive versions floating around.
The Prado is likely even more expensive however the chances of finding a clean one (not been offroad) are very high and they typically also come better equipped for daily driving with regards comforts.