XJIPC OpenIR  > 多语种信息技术研究室
Tailoring an Interpretable Neural Language Model
Zhang, YK (Zhang, Yike)[ 1,2 ]; Zhang, PY (Zhang, Pengyuan)[ 1,2 ]; Yan, YH (Yan, Yonghong)[ 1,2,3 ]
2019
Source PublicationIEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING
ISSN2329-9290
Volume27Issue:7Pages:1164-1178
Abstract

Neural networks have shown great potential in language modeling. Currently, the dominant approach to language modeling is based on recurrent neural networks (RNNs) and convolutional neural networks (CNNs). Nonetheless, it is not clear why RNNs and CNNs are suitable for the language modeling task since these neural models are lack of interpretability. The goal of this paper is to tailor an interpretable neural model as an alternative to RNNs and CNNs for the language modeling task. This paper proposes a unified framework for language modeling, which can partly interpret the rationales behind existing language models (LMs). Based on the proposed framework, an interpretable neural language model (INLM) is proposed, including a tailored architectural structure and a tailored learning method for the language modeling task. The proposed INLM can be approximated as a parameterized auto-regressive moving average model and provides interpretability in two aspects: component interpretability and prediction interpretability. Experiments demonstrate that the proposed INLM outperforms some typical neural LMs on several language modeling datasets and on the switchboard speech recognition task. Further experiments also show that the proposed INLM is competitive with the state-of-the-art long short-term memory LMs on the Penn Treebank andWikiText-2 datasets.

KeywordNeural language models interpretability autoregressive moving average speech recognition
DOI10.1109/TASLP.2019.2913087
Indexed BySCI
WOS IDWOS:000467568600005
Citation statistics
Document Type期刊论文
Identifierhttp://ir.xjipc.cas.cn/handle/365002/5751
Collection多语种信息技术研究室
Affiliation1.Chinese Acad Sci, Inst Acoust, Beijing 100190, Peoples R China
2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China
3.Chinese Acad Sci, Xinjiang Tech Inst Phys & Chem, Urumqi 830011, Peoples R China
Recommended Citation
GB/T 7714
Zhang, YK ,Zhang, PY ,Yan, YH . Tailoring an Interpretable Neural Language Model[J]. IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING,2019,27(7):1164-1178.
APA Zhang, YK ,Zhang, PY ,&Yan, YH .(2019).Tailoring an Interpretable Neural Language Model.IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING,27(7),1164-1178.
MLA Zhang, YK ,et al."Tailoring an Interpretable Neural Language Model".IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING 27.7(2019):1164-1178.
Files in This Item:
File Name/Size DocType Version Access License
Tailoring an Interpr(3044KB)期刊论文作者接受稿开放获取CC BY-NC-SAView Application Full Text
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Zhang, YK (Zhang, Yike)[ 1,2 ]]'s Articles
[Zhang, PY (Zhang, Pengyuan)[ 1,2 ]]'s Articles
[Yan, YH (Yan, Yonghong)[ 1,2,3 ]]'s Articles
Baidu academic
Similar articles in Baidu academic
[Zhang, YK (Zhang, Yike)[ 1,2 ]]'s Articles
[Zhang, PY (Zhang, Pengyuan)[ 1,2 ]]'s Articles
[Yan, YH (Yan, Yonghong)[ 1,2,3 ]]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Zhang, YK (Zhang, Yike)[ 1,2 ]]'s Articles
[Zhang, PY (Zhang, Pengyuan)[ 1,2 ]]'s Articles
[Yan, YH (Yan, Yonghong)[ 1,2,3 ]]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: Tailoring an Interpretable Neural Language Model.pdf
Format: Adobe PDF
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.