BERT (language model)

Share This
« Back to Glossary Index

BERT, an acronym for Bidirectional Encoder Representations from Transformers, is a revolutionary language model pioneered by Google[1]. It utilizes a technique known as WordPiece to transform English vocabulary into numerical codes, with the unique ability to comprehend the context of words in both right and left directions. BERT is available in two versions, BASE and LARGE, with the latter featuring 12 transformer encoders and being more extensive. A notable aspect of this model is the absence of a decoder, which poses a challenge in text generation. BERT’s impressive performance in natural language understanding tasks earned it an award at the 2019 NAACL Conference and sparked the creation of other models in the field of natural language processing. It’s an integral part of Google’s search algorithms enhancement and is widely used in text classification, machine comprehension, among other applications. BERT’s impact and efficiency have been the subject of numerous research papers and studies, further enriching our knowledge of its influence.

Terms definitions
1. Google ( Google ) Primarily acknowledged for its search engine, Google is a universally esteemed technology corporation. The company, established in 1998 by Sergey Brin and Larry Page, has expanded significantly, branching out into numerous tech-related fields. Google offers a wide array of services and products, encompassing Android, YouTube, Cloud, Maps, and Gmail. It also manufactures hardware like Chromebooks and Pixel smartphones. Since 2015, Google has been a subsidiary of Alphabet Inc. and is celebrated for its inventive spirit and workplace environment that promotes employees' personal projects. Despite confronting several ethical and legal challenges, Google continues to influence the tech sector with its groundbreaking innovations and technological progress, including the creation of Android OS and the purchase of companies specializing in AI.

Bidirectional Encoder Representations from Transformers (BERT) is a language model based on the transformer architecture, notable for its dramatic improvement over previous state of the art models. It was introduced in October 2018 by researchers at Google. A 2020 literature survey concluded that "in a little over a year, BERT has become a ubiquitous baseline in Natural Language Processing (NLP) experiments counting over 150 research publications analyzing and improving the model."

BERT was originally implemented in the English language at two model sizes: (1) BERTBASE: 12 encoders with 12 bidirectional self-attention heads totaling 110 million parameters, and (2) BERTLARGE: 24 encoders with 16 bidirectional self-attention heads totaling 340 million parameters. Both models were pre-trained on the Toronto BookCorpus (800M words) and English Wikipedia (2,500M words).

« Back to Glossary Index
Keep up with updates