Below we’ll cover some of the key concepts that are important for NLPCraft:
Data model is a central concept in NLPCraft. It defines natural language interface to your public or private data sources like on-premise database or a cloud SaaS application. NLPCraft employs model-as-a-code approach where entire data model is an implementation of NCModel interface which can be developed using any JVM programming language like Java, Scala, Kotlin, or Groovy.
A data model defines:
Note that model-as-a-code approach allows you to use any software lifecycle tools and frameworks like various build tools, CI/SCM tools, IDEs, etc. to develop and maintain your data model. You don't have to use additional web-based tools to manage some aspects of your data models - your entire model and all of its components are part of your project source code.
Read more about data models here.
Named entity, also known as a model element or a token, is main a component defined by the NLPCraft data model. A named entity is one or more individual words that have a consistent semantic meaning and typically denote a real-world object, such as persons, locations, number, date and time, organizations, products, etc. Such object can be abstract or have a physical existence.
For example, in the following sentence:
Meeting is set for 12pm today in San Francisco.
the following named entities can be detected:
|DATE_TIME||12:00 September 1, 2019 GMT|
|GEO_CITY||San Francisco, CA USA|
In most cases named entities will have associated normalized value. It is especially important for named entities that have many different notational forms such as time and date, currency, geographical locations, etc. For example,
New York City and
NYC all refer to the same "New York City, NY USA" location which is a valid normalized form.
The process of detecting named entities is called Named Entity Recognition (NER). There are many different ways of how a certain named entity can be detected: through list of synonyms, by name, rule-based or by using statistical techniques like neural networks with large corpus of predefined data. NLPCraft allows you define named entities through powerful DSL and also supports named entities that can be composed from other named entities including named entities from external projects such OpenNLP, spaCy or Stanford CoreNLP.
Named entities allow you to abstract from basic linguistic forms like nouns and verbs to deal with the higher level semantic abstractions like geographical location or time when you are trying to understand the meaning of the sentence. One of the main goals of named entities is to act as an input ingredients for intent matching.
Read more in-depth about named entities here.
You can think of intent matching as regular expression matching where instead of characters you deal with detected named entities. Intent defines a pattern in terms of detected named entities (or tokens) and a callback to call when submitted sentence matches that pattern.
Intents can also match on the dialog flow additionally to the matching on the current user sentence. Dialog flow matching means matching an intent based on what intents were matched previously for the same user and data model, i.e. the flow of the dialog. Note that you should not confuse dialog flow intent matching with conversational STM that is used to fill in missing tokens from memory.
You can think of NLPCraft data model as a mechanism to define named entities and intents that use these named entities to pattern match the user input.
Learn more details about intent matching here.
NLPCraft provides automatic conversation management right out of the box. Conversation management is based on the idea of short-term-memory (STM). STM is automatically maintained by NLPCraft per each user and data model. Essentially, NLPCraft "remembers" the context of the conversation and can supply the currently missing elements from its memory (i.e. from STM). STM implementation is also fully integrated with intent matching.
Maintaining conversation state is necessary for effective context resolution, so that users could ask, for example, the following sequence of questions using example weather model:
User gets the current London’s weather. STM is empty at this moment so NLPCraft expects to get all necessary information from the user sentence. Meaningful parts of the sentence get stored in STM.
User gets the current Berlin’s weather. The only useful data in the user sentence is name of the city
Berlin. But since NLPCraft now has data from the previous question in its STM it can safely deduce that we are asking about
London in STM.
User gets the next week forecast for Berlin. Again, the only useful data in the user sentence is
next week and
forecast. STM supplies
Next week override
weather in STM.
Note that STM is maintained per user and per data model. Conversation management implementation is also smart enough to clear STM after certain period of time, i.e. it “forgets” the conversational context after few minutes of inactivity. Note also that conversational context can also be cleared explicitly via REST API.