@Sreenath G, not owning a manual Xterra I can't provide feedback on that specific car but generally speaking driving a manual in the desert, for the right type of driver, is the best.
Having driven both a automatic and a manual in the desert the biggest difference from a drivers point of view is the level of concentration and anticipation you need to have.
When driving a automatic whilst you still need a high degree of concentration should you be in the wrong gear you can change down instantly whereas in a manual by the time you have realised you are in the wrong gear it is too late as a manual downshift will ruin your momentum.
In my experience I found that I was paying even more attention to the car in front as well as looking further down the track to anticipate what is ahead.
If your idea of a drive in the desert should be fast paced and intense then a manual will definately suit you because you have never done so many gear changes in one drive as you do on a technical sand drive.
Hope this helps.