转译器 | 对专业特征强的现代汉语文本的无监督分析

对专业特征强的现代汉语文本的无监督分析

词发现 文本切词 EM算法 中国历史 博客

随着公开或私下的数字化的文本数据越来越多，出现了对从文本中有效、自动提取信息的计算工具的巨大需求。因为中文语言完全不同于字母语言，没有特定的词分界，所以大部分已有的中文文本挖掘工具都需要一个预设的词汇表或者大量的相关训练语料，但这并不是所有文本都具备的。我们在此介绍的是一种无监督的方法，从上至下的词语发现和分词（TopWORDS），从大体积的、非结构的中文文本中同时进行词语发掘、切分语词，并能提供方法对已发现词语进行排序，并进行更高层次的文本分析。TopWORDS特别适用于挖掘那些潜在词汇表未知，或者目标文本与可用的训练语料差异很大的，在线的、主题明确的文本。从TopWORDS导出的数据可以导入像Topic modeling, word embedding,以及association pattern finding这样的文本分析工具，其结果同于甚至优于使用监督切词工具的结果。

On the unsupervised analysis of domain-specific Chinese texts

• Ke Denga, Peter K. Bolb, Kate J. Lic, and Jun S. Liu a,d,1

  
  

• A Center for Statistical Science & Department of Industry Engineering, Tsinghua University, Beijing 100084, China; Department of East Asian Languages & Civilizations, Harvard

  
  

• University, Cambridge,MA 02138; cSawyer Business School, Suffolk University, Boston,MA 02108; and dDepartment of Statistics, Harvard University, Cambridge,MA 02138

Word Discovery | Text Segmentations | EM Algorithm | Chinese History | Blogs

With the growing availability of digitized text data both publicly and privately, there is a great need for effective computational tools to automatically extract information from texts. Because the Chinese language differs most significantly from alphabet-based languages in not specifying word boundaries, most existing Chinese text-mining methods require a prespecified vocabulary and/or a large relevant training corpus, which may not be available in some applications. We introduce an unsupervised method, top-down word discovery and segmentation (TopWORDS), for simultaneously discovering and segmenting words and phrases from large volumes of unstructured Chinese texts, and propose ways to order discovered words and conduct higher-level context analyses. TopWORDS is particularly useful for mining online and domain-specific texts where the underlying vocabulary is unknown or the texts of interest differ significantly from available training corpora. When outputs from TopWORDS are fed into context analysis tools such as topic modeling, word embedding, and association pattern finding, the results are as good as or better than that from using outputs of a supervised segmentation method. 

（论文详见“阅读原文”）

转载声明

未经许可，禁止转载零壹Lab所发文章。如需转载，请于显著位置注明出处，并放置清晰零壹Lab二维码。如需长期授权或其他未尽事宜请联系零壹Lab邮箱。零壹Lab保留追究私自转载者法律责任之权利。

零壹Lab邮箱：dh01lab@hotmail.com