It used to have a huge effect on query performance, as it searched secondary storage for the objects. In recent versions, performance is better, but there's still a non-negligible impact as it searches the cache for every document in the hitlist.
If you only have cache, then there's no point to that setting -- you know where everything is.
-JD.