We demonstrate a polarization insensitive time-division demultiplexer using semiconductor optical amplifier based nonlinear interferometer in a polarization-diversity loop. The demultiplexer had a very low switching energy (<100 femtojoules) and error-free operation in demultiplexing a 40 Gb/s input was achieved. The residual polarization dependence of the incident signal was found to be only 0.3 dB. Capability of demultiplexing at higher bit rates was proved by the ultrafast switching of 160 Gb/s pulses.