一种面向网络长文本的话题检测方法

A topic detection method for network long text

  • 摘要: 提出了一种面向网络长文本的话题检测方法.针对文本表示的高维稀疏性和忽略潜在语义的问题,提出了Word2vec&LDA(latent dirichlet allocation)的文本表示方法.将LDA提取的文本特征词隐含主题和Word2vec映射的特征词向量进行加权融合既能够进行降维的作用又可以较为完整的表示出文本信息.针对传统话题发现方法对长文本输入顺序敏感问题,提出了基于文本聚类的Single-Pass&HAC(hierarchical agglomerative clustering)的话题发现方法,在引入时间窗口和凝聚式层次聚类的基础上对于文本的输入顺序具有了更强的鲁棒性,同时提高了聚类的精度和效率.为了评估所提出方法的有效性,本文从某大学社交平台收集了来自真实世界的多源数据集,并基于此进行了大量的实验.实验结果证明,本文提出的方法相对于现有的方法,如VSM(state vector space model)、Single-Pass等拥有更好的效果,话题检测的精度提高了10%~20%.

     

    Abstract: Internet public opinion is an important source of people's views on social hotspots and national current affairs. Topic detection in network long text contributes toward the analysis of network public opinion. According to the results of topic detection, the policymaker can timely and reliably make scientific decisions. In general, topic detection can be divided into two steps, i.e., representation learning and topic discovery. However, common representation learning methods, such as state vector space model (VSM) and term frequency-inverse document frequency, often lead to the problems of high dimensionality, sparsity, and latent semantic loss, whereas traditional topic discovery methods depend heavily on the text input orders. To overcome these, a novel topic detection method was presented herein. First, Word2vec & latent Dirichlet allocation (LDA)-based methods for representation learning were proposed to avoid the problem of high-dimensional sparsity and neglect of latent semantics. Weighted fusion of the text feature word implicit topic extracted by LDA and the feature word vector of Word2vec mapping could not only perform dimensionality reduction but also completely represent text information. Furthermore, Single-Pass and hierarchical agglomerative clustering for topic discovery could be more robust for input orders. To evaluate the effectiveness and efficiency of the proposed method, extensive experiments were conducted on a real-world multi-source dataset, which was collected from university social platforms. The experimental results show that the proposed method outperforms other methods, such as VSM and Single-Pass, by improving the clustering accuracy by 10%-20%.

     

/

返回文章
返回