There has been much recent interest in high precision angular rotation measurement using photon orbital angular momentum to realize super-resolving angular rotation measurement. It is well known that quantum detection strategies can obtain a quantum-enhanced performance. Here, we prove that binary-outcome homodyne detection method can obtain a narrower signal peak, showing better resolution compared with the existing data processing method. Since the photon loss is unavoidable in the actual non-ideal optical system, this paper further discusses the impact of photon loss on the resolution and sensitivity of angular rotation measurement with binary-outcome homodyne detection method.