Recent experiments demonstrate that boron arsenide (BAs) is a showcase material to study the role of higher-order four-phonon interactions in affecting heat conduction in semiconductors. Here we use first-principles calculations to identify a phenomenon in BAs and a related material - boron antimonide, that has never been predicted or experimentally observed for any other material: competing responses of three-phonon and four-phonon interactions to pressure rise cause a non-monotonic pressure dependence of thermal conductivity, κ, which first increases similar to most materials and then decreases. The resulting peak in κ shows a strong temperature dependence from rapid strengthening of four-phonon interactions relative to three-phonon processes with temperature. Our results reveal pressure as a knob to tune the interplay between the competing phonon scattering mechanisms in BAs and similar compounds, and provide clear experimental guidelines for observation in a readily accessible measurement regime.