SH2 domains have been recognized as promising targets for various human diseases. However, targeting SH2 domains with phosphopeptides or small-molecule inhibitors derived from bioisosteres of the phosphate group is still challenging. Identifying novel bioisosteres of the phosphate group to achieve favorable in vivo potency is urgently needed. Here, we report the feasibility of targeting the STAT3-SH2 domain with a boronic acid group and the identification of a highly potent inhibitor compound 7 by replacing the carboxylic acid of compound 4 with a boronic acid. Compound 7 shows higher binding affinity, better cellular potency, more favorable PK profiles, and higher in vivo antitumor activity than 4. The stronger anticancer effect of 7 partially stems from its covalent binding mode with the SH2 domain, verified by the washout experiments. The relatively high level of sequence conservation among SH2 domains makes the results presented here of general significance.