Predicting drug-target binding affinity (DTA) is a crucial task in drug discovery research. Recent studies have demonstrated that pocket features and interactions between targets and drugs significantly improve the understanding of DTA. However, challenges remain, particularly in the detailed consideration of both global and local information and the further modeling of pocket features. In this paper, we propose a novel multimodal deep learning model named MMPD-DTA for predicting drug-target binding affinity to address these challenges. The MMPD-DTA model integrates graph and sequence modalities of targets, pockets, and drugs to capture both global and local target and drug information. The model introduces a novel pocket-drug graph (PD graph) that simultaneously models atomic interactions within the target, within the drug, and between the target and drug. We employ GraphSAGE for graph representation learning from the PD graph, complemented by sequence representation learning via transformers for the target sequence and graph representation learning via a graph isomorphism network for the drug molecular graph. These multimodal representations are then concatenated, and a multilayer perceptron generates the final binding affinity predictions. Experimental results on three real-world test sets demonstrate that the MMPD-DTA model outperforms baseline methods. Ablation studies further confirm the effectiveness of each module within the MMPD-DTA model. Our code is available at https://github.com/zhc-moushang/MMPD-DTA.