English to Chinese Tech/Engineering IT (Information Technology) BERT

A recent work titled “What Does BERT Look At?” has shed some light on what BERT learns during pre-training. From this analysis of BERT’s self-attention mechanism, it is evident that BERT learns a substantial amount of linguistic knowledge. For example, some of BERT’s attention-heads attend to the direct objects of verbs, determiners of nouns such as definite articles, and even {coreferent mentions} (see Figure 2).