What is Natural Language Processing?

We’re excited to bring Transform 2022 back in person on July 19 and pretty much July 20-28. Join AI and data leaders for insightful conversations and exciting networking opportunities. Register today


Teaching computers to understand human language has long been a goal of computer scientists. The natural language people use when talking to each other is complex and highly context-dependent. While humans can instinctively understand that different words are spoken at home, at work, at school, in a store, or in a religious building, none of these differences are apparent to a computer algorithm.

Over the decades of research, artificial intelligence (AI) scientists created algorithms that are beginning to reach a certain level of understanding. While the machines may not master some of the common nuances and multiple layers of meaning, they can grasp enough of the salient points to be practically useful.

Algorithms that fall under the label “natural language processing (NLP)” are used for roles in industry and homes. They are now reliable enough to be a regular part of customer service, maintenance and household chores. Devices from companies such as Google or Amazon routinely listen in and answer questions when addressed with the correct trigger word.

How are the algorithms designed?

The mathematical approaches are a mixture of a rigid rule-based structure and flexible probability. The structural approaches build models of sentences and sentences that are similar to the diagrams sometimes used to teach grammar to school children. They follow many of the same rules as textbooks, and they can reliably analyze the structure of large blocks of text.

These structural approaches begin to fail when words have multiple meanings. The canonical example is the use of the word “fly” in the sentence: “Time flies like an arrow, but fruit flies like bananas.” AI scientists have found that statistical approaches can reliably distinguish between the different meanings. The word “fly” can form a compound noun 95% of the time, it follows the word “fruit”.

How do AI scientists build models?

Some AI scientists have analyzed some large blocks of text that are easily found on the web to create comprehensive statistical models that can understand how context changes meaning. For example, a book on agriculture would be much more likely to use ‘fly’ as a noun, while a text about airplanes would probably use it as a verb. However, a book on crop dusting would be a challenge.

Machine Learning Algorithms can build complex models and detect patterns that may escape human detection. For example, it is now common practice to use the complex word choice statistics captured in these models to identify the author.

Some natural language processing algorithms focus on understanding spoken words captured by a microphone. These speech recognition algorithms also rely on similar mixtures of statistics and grammar rules to understand the flow of phonemes.

[Related: How NLP is overcoming the document bottleneck in digital threads]

How does natural language processing evolve?

Now that algorithms can provide useful help and demonstrate basic competencies, AI scientists are focusing on improving comprehension and adding more ability to tackle sentences of greater complexity. Part of this insight comes from creating more complex sets of rules and subrules to better capture human grammar and diction. Lately, however, the focus has been on using machine learning algorithms on large data sets to capture more statistical details about how words can be used.

AI scientists hope that larger datasets from digitized books, articles and commentaries can provide more in-depth insights. For example, Microsoft and Nvidia recently announced that they are Megatron Turing NLG 530Ban immense natural language model with 530 billion parameters arranged in 105 layers.

The training set contains a mix of documents gathered from the open internet and real news compiled to rule out common misinformation and fake news. After deduplication and cleaning, they built a training set with 270 billion tokens consisting of words and phrases.

The goal now is to improve reading comprehension, disambiguation of word meanings and inferences. It’s starting to reflect what people call “common sense,” it gets better as the models capture more basic details about the world.

In many ways, models and human language are beginning to evolve and even converge. As people use more natural language products, they begin to intuitively predict what the AI ​​will or will not understand and choose the best words. The AIs can adapt and the language shifts.

What are the established players creating?

Google offers a comprehensive suite of APIs for decoding websites, spoken words and printed documents. Some tools are built to translate spoken or printed words into digital form, and others are aimed at gaining some understanding of the digitized text. For example, one cloud APIs will perform optical character recognition while another will convert speech to text† Some, like the basics natural language API, are general tools with a lot of room for experimentation, while others focus specifically on common tasks such as form processing or medical knowledge† For example, the Document AI tool is available in versions adapted for the banking or the purchasing team

Amazon also offers a wide variety of APIs as cloud services for finding salient information in text files, spoken word or scanned documents. The core is: To understand, a tool that identifies key phrases, people, and sentiments in text files. a version, Understanding medically, is focused on understanding medical information contained in physician notes, clinical trial reports, and other medical records. They also provide pre-trained machine learning models for: translation and transcription† For some common uses, such as running a customer service chatbot, AWS provides tools such as: Lex to simplify adding an AI-based chatbot to a company’s web presence.

Microsoft also offers a wide variety of tools as part of: Azure Cognitive Services for understanding all forms of language. Their Language Studio starts with basic models and lets you train new versions to be deployed with their bot framework† Some APIs like Azure Cognitive Search integrate these models with other features to simplify website management. Some tools are more applied, such as: content moderator for detecting inappropriate language or Personalizer for finding good recommendations.

What are the startups doing?

Many of the startups apply natural language processing to concrete problems with clear revenue streams. grammatical, for example, creates a tool that proofs text documents to spot grammatical problems caused by problems such as verb tenses. The free version detects basic errors, while the $12 premium plan provides access to more advanced error checking, such as identifying plagiarism or helping users adopt a more confident and polite tone. The company is over 11 years old and is integrated with most online environments where text can be edited.

SoundHound provides a “voice AI platform” that other manufacturers can add so that their product can respond to voice commands triggered by a “wake word”. It provides “speech-to-meaning” capabilities that parse the requests into data structures for integration with other software routines.

Shield wants to support managers who need to monitor the text in their office spaces. Their ‘communication compliance’ software uses models built with multiple languages ​​for ‘behavioural communication monitoring’ to detect violations such as insider trading or harassment.

Nori Health plans to help sick people cope with chronic conditions with chatbots trained to advise them on how best to behave to alleviate the illness. They are starting with “digital therapies” for inflammatory conditions like Crohn’s disease and colitis.

smartling adapts natural language algorithms to better automate translation so that companies can better deliver their software to people who speak different languages. They provide a managed pipeline to simplify the process of creating multilingual documentation and sales literature on a large, multinational scale.

Is there anything natural language processing can’t do?

The standard algorithms are often successful in answering basic questions, but they rely heavily on connecting keywords with standard answers. Users of tools such as Siri from Apple or Alexa from Amazon quickly learn which types of sentences are registered correctly. However, they often fail to grasp or detect nuances when a word is used with a secondary or tertiary meaning. Basic sentence structures can work, but not more elaborate or ornate structures with subordinate sentences.

The search engines have become adept at predicting or understanding whether the user wants a product, a definition, or a reference to a document. However, this classification is largely probabilistic and the algorithms fail the user if the request does not follow the standard statistical pattern.

Some algorithms tackle the opposite problem, which is turning automated information into human-readable language. Some common news tasks, such as reporting the stock market movement or describing the outcome of a game, can be largely automated. The algorithms can even add some nuance that can be helpful, especially in areas of great statistical depth like baseball. The algorithms can search a box score and find unusual patterns such as a no-hitter and add it to the article. However, the lyrics tend to have a mechanical tone and readers quickly begin to anticipate the word choices falling into predictable patterns and forming clichés.

[Read more:Data and AI are keys to digital transformation – how can you ensure their integrity? ]

Leave a Comment

Your email address will not be published.