Penumbral blur is one of the major limitations of the high spatial resolution micro-CT, due to a nonideal large focal spot. Penumbral blur hinders the ability to resolve small features that may only be a few pixels in size. Reducing the focal spot size by decreasing the x-ray tube power is a straightforward solution, but it leads to prolonged scan durations. In this paper, we propose to mitigate the penumbral blur by learning the inverse finite focal spot model. First, we derived the finite focal spot model that builds a relationship from the ideal point source projection to the finite focal spot projection. Based on the derived model, we numerically compute a paired projection dataset. Second, we utilized two neural networks-U-net, and convolution modulation-based U-net (CMU-net) -to learn the inverse finite focal spot model. The goal is to estimate the ideal point source projection from the actual finite focal spot projection. CMU-net, which introduces convolution modulation blocks into the contracting path of the U-net, is proposed to boost the robustness of the U-net. Finally, the standard filtered back-projection (FBP) is employed for reconstruction using the estimated ideal point projection. The experiments show that both U-net and CMU-net can effectively reduce the penumbral blur, whereas CMU-net demonstrates better performance on the real data. Experiments on real measured data demonstrate that CMU-net is more robust than U-net and can effectively resolve fine details. This method has great potential in improving the efficiency of micro-CT acquisition. It allows increasing the tube power since our method can computationally compensate for the blur caused by an increased focal spot size.