Graph neural networks have emerged as a popular and powerful tool for learning hierarchical representation of graph data. In complement to graph convolution operators, graph pooling is crucial for extracting hierarchical representation of data in graph neural networks. However, most recent graph pooling methods still fail to efficiently exploit the geometry of graph data. In this paper, we propose a novel graph pooling strategy that leverages node affinity to improve the hierarchical representation learning of graph data. Node affinity is computed by harmonizing the kernel representation of topology information and node features. In particular, a structure-aware kernel representation is introduced to explicitly exploit advanced topological information for efficient graph pooling without eigendecomposition of the graph Laplacian. Similarities of node signals are evaluated using the Gaussian radial basis function (RBF) in an adaptive way. Experimental results demonstrate that the proposed graph pooling strategy is able to achieve state-of-the-art performance on a collection of public graph classification benchmark datasets.