I recommend https://www.mat.univie.ac.at/~neum/physfaq/topics/position.html which, while a bit hard to read, is more comprehensive and written by someone more authoritative than me. My research didn't start there, but I covered several of the same sources before finding it.
The tl;dr is the "naive" position operator is not the right one, but even the "right" position operator is not invariant with respect to Lorentz boosts. QFT is indeed the resolution because the Unruh effect implies that the mere existence of a particle is not Lorentz invariant.
While it sounds like it isn't often mentioned in texts, the appropriate position operator is the mean position operator, X, presented in On the Dirac Theory of Spin 1/2 Particles and Its Non-Relativistic Limit (1950) by Foldy and Wouthuysen. It has the property that it reduces to the position operator in the non-relativistic limit. This operator was also found a bit earlier by Newton and Wigner in Localized States for Elementary Systems (1949) where it was proven to be the unique operator satisfying certain consistency criteria. Very recently, Fundamental Operators in Dirac Quantum Mechanics (2020) by Silenko, Zhang, and Zou argues that the Foldy-Wouthuysen representation is required to have a probabilistic interpretation of Dirac theory. (See the linked FAQ for references to simpler, more modern derivations of these operators.)
The above suggests that there is only really one "right" choice for the position operator. Nevertheless, the mean position operator is still not Lorentz invariant. Quoting the FAQ linked above:
Already the notion of a particle depends on the observer, as shown by the Unruh effect. It is no surprise that the position of something observer-dependent is also observer-dependent. It explains naturally why position operators are necessarily noninvariant under Lorentz boosts.
My understanding of this is that an inertial observer in a vacuum would observe no particles and thus (abusing the single-particle theory) the position operator would be everywhere zero, but an accelerating observer would experience (more) particles and thus a non-zero position operator. This suggests that any attempt at a particle-oriented description is doomed to issues. The next paragraph in the FAQ is:
Quantum fields are covariant and exist everywhere, so they need neither observers nor a particular position operator. That this is not the case for particles is - in view of the fact that physical objects existed long before observers came into existence - sufficient reasons why particles cannot be fundamental.