Pancreatic ductal adenocarcinoma (PDAC) and cholangiocarcinoma (CCA) are both deadly cancers and they share many biological features besides their close anatomical location. One of the main histological features is neurotropism, which results in frequent perineural invasion. The underlying mechanism of cancer cells favoring growth by and through the nerve fibers is not fully understood. In this review, we provide knowledge of these cancers with frequent perineural invasion. We discuss nerve fiber crosstalk with the main different components of the tumor microenvironment (TME), the immune cells, and the fibroblasts. Also, we discuss the crosstalk between the nerve fibers and the cancer. We highlight the shared signaling pathways of the mechanisms behind perineural invasion in PDAC and CCA. Hereby we have focussed on signaling neurotransmitters and neuropeptides which may be a target for future therapies. Furthermore, we have summarized retrospective results of the previous literature about nerve fibers in PDAC and CCA patients. We provide our point of view in the potential for nerve fibers to be used as powerful biomarker for prognosis, as a tool to stratify patients for therapy or as a target in a (combination) therapy. Taking the presence of nerves into account can potentially change the field of personalized care in these neurotropic cancers.