Elasticsearch support for traditional chinese

Question

I am trying to index and search Chinese into Elasticsearch. By using Smart Chinese Analysis (elasticsearch-analysis-smartcn) plugin I have managed to search characters and words for both simplified and traditional chinese. I have tried to insert the same text in both simplified and traditional chinese, but the search returns only one result (depending on how the search is performed); since the text is the same I would expect both results to be returned. I have read here that in order to support traditional chinese I must also install the STConvert Analysis (elasticsearch-analysis-stconvert) plugin. Can anyone provide a working example that uses these two plugins? (or an alternative method that achieves the same result)

The test index is created as

{  
   "settings":{  
      "analysis":{  
         "analyzer":{
            "chinese":{  
               "type":"smartcn"
            }
         }
      }
   },
   "mappings":{  
      "testType":{  
         "properties":{  
            "message":{  
               "store":"yes",
               "type":"string",
               "index":"analyzed",
               "analyzer":"chinese"
            },
            "documentText": {
               "store":"compress",
               "type":"string",
               "index":"analyzed",
               "analyzer":"chinese",
               "termVector":"with_positions_offsets"
            }
         }
      }
   }
}

and the two requests with the same text in simplified-traditional are

{
   "message": "汉字",
   "documentText": "制造器官的噴墨打印機 這是一種制造人體器官的裝置。這種裝置是利用打印機噴射生物 細胞、 生長激素、凝膠體，形成三維的生物活體組織。凝膠體主要是為細胞提供生長的平台，之后逐步形成所想要的器官或組織。這項技術可以人工方式制造心臟、肝臟、腎臟。這項研究已經取得了一定進展，目前正在研究如何將供應營養的血管印出來。這個創意目前已經得到了佳能等大公司的贊助"
}

{
   "message": "汉字",
   "documentText": "制造器官的喷墨打印机 这是一种制造人体器官的装置。这种装置是利用打印机喷射生物 细胞、 生长激素、凝胶体，形成叁维的生物活体组织。凝胶体主要是为细胞提供生长的平台，之后逐步形成所想要的器官或组织。这项技术可以人工方式制造心脏、肝脏、肾脏。这项研究已经取得了一定进展，目前正在研究如何将供应营养的血管印出来。这个创意目前已经得到了佳能等大公司的赞助"
}

Finally, a sample search that I want to return two results is

{  
   "query":{  
      "query_string":{  
         "query":"documentText : 制造器官的喷墨打印机",
         "default_operator":"AND"
      }
   }
}

Have you tried using the `_analyze` endpoint to see how your text is being tokenized? — IanGabes, Jul 15 '15 at 16:37
Yes. If I speficy no analyzer in the url I get 10 tokens (1 per symbol), using my custom chinese analyzer I get 5 tokens and finally using the stconvert analyzer I get only 1 token. — dchar, Jul 16 '15 at 06:45

score 4 · Answer 1 · answered Jul 17 '15 at 12:32

After many attempts I found a configuration that works. I did not manage to make smartcn work with stconvert plugin, so I used the cjk analyzer of elasticsearch, with an addition of icu_tokenizer instead. By using t2s and s2t as filters, each character is stored in both forms, traditional and simplified.

{  
   "settings":{  
      "analysis":{  
        "filter": {
          "english_stop": {
            "type":       "stop",
            "stopwords":  "_english_" 
          },
          "t2s_convert": {
            "type": "stconvert",
            "delimiter": ",",
            "convert_type": "t2s"
          },
          "s2t_convert": {
            "type": "stconvert",
            "delimiter": ",",
            "convert_type": "s2t"
          }
        },
        "analyzer": {
          "my_cjk": {
            "tokenizer": "icu_tokenizer",
            "filter": [
              "cjk_width",
              "lowercase",
              "cjk_bigram",
              "english_stop",
              "t2s_convert",
              "s2t_convert"
            ]
          }
        }
      }
   },
   "mappings":{  
      "testType":{  
         "properties":{  
            "message":{  
               "store":"yes",
               "type":"string",
               "index":"analyzed",
               "analyzer":"my_cjk"
            },
            "documentText": {
               "store":"compress",
               "type":"string",
               "index":"analyzed",
               "analyzer":"my_cjk",
               "termVector":"with_positions_offsets"
            }
         }
      }
   }
}

Elasticsearch support for traditional chinese

1 Answers1