<jupyter_start><jupyter_text>Use RASA DSL to Configure DeepPavlov's GO-Bot At DeepPavlov, we support a variety of industry-wide and popular standards to support developing Conversational AI solutions.DSLs, known as Domain-Specific Languages, provide a rich mechanism to define the behavior, or "the what", whilethe underlying system uses the parser to transform these definitions into commands that implement this behavior, or "the how" using the system's components.Until very recently we supported two such DSLs, including industry-standard [AIML](http://docs.deeppavlov.ai/en/master/features/skills/aiml_skill.html), as well as [DSL](http://docs.deeppavlov.ai/en/master/features/skills/dsl_skill.html) designed by one of our partners, EORA.In this tutorial, you will learn how to use another industrial DSL, or, better said, set of DSLs, introduced by RASA.ai,to build simple goal-oriented chatbots using DeepPavlov's GO-bot.This is the very beginning of our work focused on supporting RASA DSLs as a way to configure DeepPavlov-based goal-oriented chatbots,and therefore not all elements of the RASA DSLs are supported. It is also worth mentioning that in 0.12.0 release we want to focus on supporting tools to define the domain logic behind the goal-oriented assistant, and files like `config.yml` and others are out of scope for this release.To configure a DeepPavlov-based goal-oriented chatbot using these DSLs, you need to have at least three basic config files:* `stories.md` (or `stories-{trn, tst, val}.md` but these are just subsamples)* `nlu.md`* `domain.yml`These files allow you to define 3 key elements of the chatbot, including product-level stories, NLU training data, and your chatbot's domain.  Concepts Behind Stories.md, NLU.md, and Domain.yml  `stories.md` `stories.md` is a mechanism used to teach your chatbot how to respond to user messages. It allows you to control your chatbot's dialog management.These "stories" model real conversations between a user and a chatbot. This Markdown-based file is used to define a list of*stories*, and each *story* can have a list of one or more *intents* with (optional) corresponding *slots*, where each *intent*has one or more corresponding *actions* taken by the chatbot.These actions, in general, can be anything, from simple message replies, to programmable actions that call APIs of other services.*Note:* In this version, supported actions are limited to simple message replies.In a way, it can be seen as a *dialogues dataset*.*Note: Stories do not provide an ultimative instruction of how the bot should behave: it is up to the training process to infer the implicit underlying patterns controlling the dialogue.*  If you are looking for a way to make the bot follow the story templates strictly as defined, there is a known hack: the more times the model sees the training data, the better the model models the data, so the the desired behavior is achieved when the accuracy on the training data is 1.Such a situation is illustrated in section Basic Chatbot.  format Stories file is a markdown file of the following format:```markdown story_title(not used by algorithm, but useful to work with for humans)* user_action_label{"1st_slot_present_in_action": "slot1_value", .., "Nth_slot_present_in_action": "slotN_value"} - system_respective_utterance* another_user_action_of_the_same_format  - another_system_response... another_story_title...```**See examples below in this tutorial**  `nlu.md` `nlu.md` represents an NLU model of your chatbot. It allows you to provide training examples that show how your chatbot shouldunderstand user messages, and then train a model through these examples.While DeepPavlov's GO-bot supports JSON-based DSTC-2 format for training data, this Markdown Format introduced by RASA is the easiest one for humans to read and write.  format NLU file is a markdown file of the following format:```markdown intent:possible_user_action_label_1- An example of user text that has the possible_user_action_label_1 action label- Another example of user text that has the possible_user_action_label_1 action label... intent:possible_user_action_label_N- An example of user text that has the (possible_user_action_label_N)[action_label] action label...```**See examples below in this tutorial**  `domain.yml` `domain.yml` helps you to define the universe your chatbot lives in: what user inputs it expects to get, what actions it should be able to predict,how to respond, and what information to store.This YML format is relatively simple, and it can be seen as a dictionary of all components of your chatbot, including but not limited to intents,actions, responses, and other things.  format Domain file is a YAML file of the following format:```yaml slots section lists the possible slot names (aka slot types)  that are used in the domain (i.e. relevant for bot's tasks) currently only type: text is supportedslots:  slot1_name:    type: text  ...  slotN_name:    type: text entities list now follows the slots list 2nd level keys  and is present to support upcoming features. Stay tuned for updates with this!entities:- slot1_name...- slotN_name intents section lists the intents that can appear in the stories being kept together they do describe the user-side part of go-bot's experienceintents:  - user_action_label  - another_user_action_of_the_same_format  ... responses section lists the system response templates. Despite system response' titles being usually informative themselves   (one could even find them more appropriate when no actual "Natural Language" is needed     (e.g. for buttons actions in bot apps)) It is though extremely useful to be able to serialize the response title to text.  That's what this section content is needed for.responses:  system_utterance_1:    - text: "The text that system responds with"  another_system_response:    - text: "Here some text again"```**See examples below in this tutorial**  Basic Chatbot Let's build the simplest chatbot possible.This chatbot will be capable of processing three intents: *greeting*, *goodbye*, and *thanks*.<jupyter_code>DP_MIN_DEMO_DIR = "dp_minimal_demo_dir"  # we will work in this folder
import os

%cd /content
os.makedirs(DP_MIN_DEMO_DIR, exist_ok=True)
%cd {DP_MIN_DEMO_DIR}<jupyter_output>/content
/content/dp_minimal_demo_dir<jupyter_text>Stories.md: Basic Stories Example `stories.md` is pretty straightforward in this case. In it you define 3 stories, each having its own intent and response (utterance).Take into account the fact that you can combine all of these intents under one story, or add two intents to one story, and third to another one.<jupyter_code>%%writefile stories.md

## greet
* greet
  - utter_greet

## thank
* thank
  - utter_noworries

## goodbye
* bye
  - utter_bye<jupyter_output>Writing stories.md<jupyter_text>nlu.md: Basic NLU Training Data Example `nlu.md` has an NLU training data that enables DeepPavlov to recognize user phrases as belonging to one of the intents defined in `domain.yml`.<jupyter_code>%%writefile nlu.md

## intent:greet
- Hi
- Hey
- Hi bot
- Hey bot
- Hello
- Good morning
- hi again
- hi folks

## intent:bye
- goodbye
- goodnight
- good bye
- good night
- see ya
- toodle-oo
- bye bye
- gotta go
- farewell

## intent:thank
- Thanks
- Thank you
- Thank you so much
- Thanks bot
- Thanks for that
- cheers<jupyter_output>Writing nlu.md<jupyter_text>domain.yml: Basic Domain Example In this demo, `domain.yml` contains the list of: * possible user action intents, and* possible system response actions*Note:* Entities and slots are omitted in this example. See the more sophisticated example below to see how they can be defined in the `domain.yml`.<jupyter_code>%%writefile domain.yml

intents:
  - greet
  - bye
  - thank

responses:
  utter_noworries:
  - text: No worries!
  utter_greet:
  - text: Hi
  utter_bye:
  - text: Bye!<jupyter_output>Writing domain.yml<jupyter_text>The next step is to install the `deeppavlov` library.<jupyter_code>!pip install git+https://github.com/deepmipt/DeepPavlov.git@feature/gobot-md-yaml-config
!python -m deeppavlov install gobot_simple_dstc2<jupyter_output>Collecting git+https://github.com/deepmipt/DeepPavlov.git@feature/gobot-md-yaml-config
  Cloning https://github.com/deepmipt/DeepPavlov.git (to revision feature/gobot-md-yaml-config) to /tmp/pip-req-build-dsjafm4a
  Running command git clone -q https://github.com/deepmipt/DeepPavlov.git /tmp/pip-req-build-dsjafm4a
  Running command git checkout -b feature/gobot-md-yaml-config --track origin/feature/gobot-md-yaml-config
  Switched to a new branch 'feature/gobot-md-yaml-config'
  Branch 'feature/gobot-md-yaml-config' set up to track remote branch 'feature/gobot-md-yaml-config' from 'origin'.
Collecting aio-pika==6.4.1
[?25l  Downloading https://files.pythonhosted.org/packages/c8/07/196a4115cbef31fa0c3dabdea146f02dffe5e49998341d20dbe2278953bc/aio_pika-6.4.1-py3-none-any.whl (40kB)
[K     |████████████████████████████████| 51kB 2.1MB/s 
[?25hCollecting Cython==0.29.14
[?25l  Downloading https://files.pythonhosted.org/packages/df/d1/4d3f8a7a920e805488a966cc6ab55c978a712240f584445d703c08[...]<jupyter_text>Define the path to our DSL-based configuration files above (the folder we are in right now) and the folder used to store the trained bot.<jupyter_code>from deeppavlov import configs
from deeppavlov.core.common.file import read_json


gobot_config = read_json(configs.go_bot.gobot_md_yaml_minimal)

gobot_config['metadata']['variables']['DATA_PATH'] = '.'
gobot_config['metadata']['variables']['MODEL_PATH'] = '.'<jupyter_output><empty_output><jupyter_text>Since our data is basically the mock tutorial data we will use the same subsamples for all of the train (training set), test (test set) and valid (validation set) subsamples. However, for a real DeepPavlov-based goal-oriented bot you should use different train, test, and valid sample stories.md files.<jupyter_code>!cp stories.md stories-trn.md
!cp stories.md stories-tst.md 
!cp stories.md stories-val.md<jupyter_output><empty_output><jupyter_text>The next step is to train the bot:<jupyter_code>from deeppavlov import train_model

train_model(gobot_config, download=True)<jupyter_output>2020-08-07 08:18:09.343 INFO in 'deeppavlov.core.data.utils'['utils'] at line 94: Downloading from http://files.deeppavlov.ai/embeddings/glove.6B.100d.txt to /root/.deeppavlov/downloads/embeddings/glove.6B.100d.txt
347MB [01:35, 3.65MB/s]
2020-08-07 08:19:47.73 INFO in 'deeppavlov.core.data.utils'['utils'] at line 94: Downloading from http://files.deeppavlov.ai/datasets/gobot_md_yaml_minimal.tar.gz to /content/gobot_md_yaml_minimal.tar.gz
100%|██████████| 528/528 [00:00<00:00, 1.53MB/s]
2020-08-07 08:19:47.749 INFO in 'deeppavlov.core.data.utils'['utils'] at line 269: Extracting /content/gobot_md_yaml_minimal.tar.gz archive into /content/dp_minimal_demo_dir
2020-08-07 08:19:47.769 INFO in 'deeppavlov.dataset_readers.dstc2_reader'['dstc2_reader'] at line 112: [loading dialogs from /tmp/tmpixkz464w]
2020-08-07 08:19:47.776 INFO in 'deeppavlov.dataset_readers.dstc2_reader'['dstc2_reader'] at line 112: [loading dialogs from /tmp/tmpfcnmbf3s]
2020-08-07 08:19:47.784 INFO in 'deeppavlov.data[...]<jupyter_text>Finally, it's time to build our bot and experiment with it:<jupyter_code>from deeppavlov import build_model
bot = build_model(gobot_config)
bot.reset()

bot(["start"])
bot(["Hi"])[0][0].actions_tuple<jupyter_output><empty_output><jupyter_text>Our bot answers with "greeting" to our "greeting". How will it respond to some grateful message?<jupyter_code>bot.reset()
bot(["start"])

bot(["Thanks!"])[0][0].actions_tuple<jupyter_output><empty_output><jupyter_text>Ok, "no worries" is an expected response. Let's check if the "goodbye" user message is processed with the corresponding reply:<jupyter_code>bot.reset()
bot(["start"])

bot_response_actions = bot(["bye"])[0][0].actions_tuple
import yaml

system_utter2text = yaml.load(open("domain.yml"))["responses"]
system_utter2text[bot_response_actions[0]][0]["text"]<jupyter_output><empty_output><jupyter_text>Advanced Chatbot: Building a Restaurants Bot inspired by the DSTC Schema-Guided Dialogue Dataset<jupyter_code>DP_BIG_DEMO_DIR = "dp_big_demo_dir"  # we'll work in this directory
import os

%cd /content
os.makedirs(DP_BIG_DEMO_DIR, exist_ok=True)
%cd {DP_BIG_DEMO_DIR}<jupyter_output>/content
/content/dp_big_demo_dir<jupyter_text>While the previous demo was focused on figuring out how to work with the very simple goal-oriented chatbot, the reality of chatbots is rarely that simple. Take, for example, the use case for restaurants. People can search for them, ask about the menus, or book tables. These activities require a substantially more advanced configuration.In the purpose of this more realistic demo, we decided to go through a rather unusual route. To simplify the process of defining the domain and behavior of this chatbot, we took a famous industrial research dataset provided by the Dialogue Systems Technology Challenge known as DSTC, also known as [Schema Dataset](https://github.com/google-research-datasets/dstc8-schema-guided-dialogue).This dataset contains a huge number of the annotated human-machine conversations crowdsourced in an [M2M manner](https://arxiv.org/pdf/1801.04871.pdf) for various real-life scenarios and domains.One of these domains is dedicated to *Restaurants*. In it, users are performing a variety of the goal-oriented tasks like searching for restaurants or booking tables via interaction with the bot.Given the power and elegance of the DSTC format, we took liberty to use our internal **automatic conversion tool** to directly **transform** its data into the set of _stories.md_, _nlu.md_, _domain.yml_.*Note: the dataset used is this demo is quite large. The dataset files listings are provided in form of file subset listings. Feel free to examine the files yourself.*  Download the data used in this tutorial section<jupyter_code># let's get the mentioned converted Schema-dataset subset
!wget http://files.deeppavlov.ai/datasets/schema_resto_md_yaml_v2.tar.gz
!tar -zxf schema_resto_md_yaml_v2.tar.gz<jupyter_output>--2020-08-07 08:22:47--  http://files.deeppavlov.ai/datasets/schema_resto_md_yaml_v2.tar.gz
Resolving files.deeppavlov.ai (files.deeppavlov.ai)... 172.16.17.32
Connecting to files.deeppavlov.ai (files.deeppavlov.ai)|172.16.17.32|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 100665 (98K) [application/octet-stream]
Saving to: ‘schema_resto_md_yaml_v2.tar.gz’

schema_resto_md_yam 100%[===================>]  98.31K   110KB/s    in 0.9s    

2020-08-07 08:22:49 (110 KB/s) - ‘schema_resto_md_yaml_v2.tar.gz’ saved [100665/100665]<jupyter_text>Technical Note: Automatic Conversion from DSTC Schema Format to RASA DSLs  Schema dataset is provided in DSTC (Dialogue State Tracking Challenge) [format](https://github.com/google-research-datasets/dstc8-schema-guided-dialogue).The DSTC format has its own advantages: it is very detailed and allows for various additional info to be incorporated into the dataset itself. In it, there are two major components - Schema Representation, and Dialog Representation. The first component is dedicated to describing sets of Intents, Slots, and Entities that are used by a given service through an API. The second component is focused on describing actual dialogs that happen between users and services. It also includes actual labels for the aforementioned Intents, Slots, and Entities defined in the Schema component. However, while DSTC format is quite rich for building state-of-the-art systems that participate in the annual DSTC competitions, it takes a serious effort for the real-world developers to collect and annotate data using this format. In contrast, RASA DSLs we're illustrating here are quite different from the DSTC: they are meant to be neat and minimalistic, and to allow developers to define their domains from a rather scarce input information. As mentioned in the beginning of this part of the tutorial, we've performed an automatical conversion of the Schema Restaurants dataset from the DSTC format to RASA DSLs.  Slot Filler Any typical goal-oriented chatbot system uses a standard approach to define the way it works in the form of a pipeline. DeepPavlov's Go-Bot is quite permissive in which componentscan be used in it's pipeline; however, Slot Filler is the required one.Slot Filler, also known as slotfiller, is necessary to recognize and normalize slot-filling information provided in the user's utterances.For example, when user says that she wants to "book a table in *London*", slotfiller's job is to recognize that *London* in this phrase represents the required slot `city`.For the purposes of this demo, we are providing the pretrained slotfiller for the dataset. The small notebook on how the slotfiller was trained will be provided in one of the upcoming releases.<jupyter_code>from deeppavlov import configs, train_model, build_model
from deeppavlov.core.common.file import read_json

!python -m deeppavlov download schema_resto_md_yaml/ner_config.json
!python -m deeppavlov download schema_resto_md_yaml/slotfiller_config.json

slotfill_config = read_json("schema_resto_md_yaml/slotfiller_config.json")
slotfiller = build_model(slotfill_config, download=True)

slotfiller(["i'm looking for a thai food somewhere in SFO"])<jupyter_output><empty_output><jupyter_text>Seems OK. Let's save our slotfiller config to train and evaluate the restaurants bot, finally.<jupyter_code>import json

json.dump(slotfill_config, open('slotfill_config.json', 'wt'))<jupyter_output><empty_output><jupyter_text>Known Limitations While slotfilling technology uses the power of the industry-standard Named Entity Recognition (NER) method to recognize key slots in the given phrases, the quality of slot recognition can be substantially increased by combining this process with the data already known to the bot's developer. For example, having a finite list of cities that are supported by a given end-user solution (e.g., several cities in the Greater Seattle Area for local restaurant chain) aids slotfiller in a significant way. Typically, this information is stored in the database, though it may also be provided in the loose files like CSV (comma-separated values).However, in order to focus on the support of the RASA DSLs, we made a conscious decision to omit the support of such data in this demo. An additional demo highligting usage of such data will be provided in one of the consequent releases.Nevertheless our demo goal-oriented bot should still be able to generalize and use the global patterns in the conversations.   Stories.md: Advanced Stories Example `stories.md`*Note: As said above, this file has been auto-generated from the DSTC schema.*Like in the Basic Demo, Stories here define a variety of interactions between user and our bot.<jupyter_code>STORIES_FPATH = "schema_resto_md_yaml/stories.md"
!echo "stories file size (lines): $(wc -l {STORIES_FPATH})"

!echo -e '\n\npart of stories file is listed below\n'
!head -500 {STORIES_FPATH} | tail -30<jupyter_output>stories file size (lines): 4768 schema_resto_md_yaml/stories.md


part of stories file is listed below

  - utter_INFORM_Cuisine+INFORM_StreetAddress+NOTIFY_SUCCESS
* THANK_YOU+GOODBYE{}
  - utter_GOODBYE

## 25
* hi{}
  - utter_hi
* INFORM_INTENT_Intent{"intent": "FindRestaurants"}
  - utter_REQUEST_City
* INFORM_City{"city": "Oakland"}
  - utter_REQUEST_Cuisine
* INFORM_Cuisine{"cuisine": "Fish"}
  - utter_OFFER_RestaurantName+OFFER_City+INFORM_COUNT_Count
* REQUEST_ALTS{}
  - utter_OFFER_RestaurantName+OFFER_City
* INFORM_PriceRange+REQUEST_ALTS{"price_range": "moderate"}
  - utter_OFFER_RestaurantName+OFFER_City+INFORM_COUNT_Count
* REQUEST_ServesAlcohol+REQUEST_HasLiveMusic{}
  - utter_INFORM_HasLiveMusic+INFORM_ServesAlcohol
* INFORM_INTENT_Intent+SELECT{"intent": "ReserveRestaurant"}
  - utter_REQUEST_Time
* INFORM_PartySize+INFORM_Time{"party_size": "2", "time": "10:30 in the morning"}
  - utter_CONFIRM_RestaurantName+CONFIRM_City+CONFIRM_Time+CONFIRM_PartySize+CONFIRM_Date
* R[...]<jupyter_text>nlu.md: Advanced NLU Training Data Example `nlu.md`*Note: As said above, this file has been auto-generated from the DSTC schema, and it's quite large. Below you can see only a part of this file. Feel free to examine the entire file.*Like in the Basic Demo, `nlu.md` shows the examples of the user utterances for the supported intent classes.The slotfilling and NER information is provided in the form of the inline mark-up.<jupyter_code>NLU_FPATH = "schema_resto_md_yaml/nlu.md"
!echo "nlu file size (lines): $(wc -l {NLU_FPATH})"

!echo -e '\n\npart of nlu file is listed below\n'
!head -50 {NLU_FPATH} | tail -20<jupyter_output>nlu file size (lines): 2147 schema_resto_md_yaml/nlu.md


part of nlu file is listed below

- [Evening 7:15](time) at Mcdonald's.
- [6:15 in the evening](time) at Bistro liaison.
- Let's set it up for [6:30 pm](time) at Sakoon.
- I would like a reservation in [Flames Eatery](restaurant_name) at [19:00](time)
- I'm thinking about Rice Bowl, at [morning 11:30](time)

## intent:REQUEST_Cuisine+AFFIRM
- Sounds great. Do you know if they cook [Taiwanese](cuisine) food?
- yeah. what do they serve?

## intent:AFFIRM
- Yes
- That's great.
- Sure that sounds good.
- That works!
- that would be great.
- That sounds good.
- Yes that works for me
- Sure, that sounds great!
- This is good for me<jupyter_text>Let's take a closer look to some specific intent examples.<jupyter_code>!grep --no-group-separator -m 10 -A 1 -P "(INFORM_Cuisine|INFORM_City)" {NLU_FPATH}<jupyter_output>## intent:INFORM_City+INFORM_INTENT_Intent
- Can you find me somewhere to eat in Dublin?
## intent:INFORM_Cuisine
- Anyone that is [burger](cuisine) like in nature
## intent:INFORM_City+INFORM_Cuisine
- I'm looking for [European](cuisine) food in San Francisco.
## intent:INFORM_Cuisine+INFORM_INTENT_Intent
- I am looking for a [Greek](cuisine) place to eat please.
## intent:INFORM_City
- It's in San Francisco.
## intent:INFORM_Cuisine+REQUEST_ALTS
- I'd like another kind of restaurant. I'd like Sushi.
## intent:INFORM_City+INFORM_Time
- I want a restaurant in Mountain View, and a reservation at 1 o"clock in the afternoon.
## intent:INFORM_PriceRange+INFORM_Cuisine
- I would prefer some [Filipino](cuisine) food which is affordable and not too expensive.
## intent:INFORM_City+INFORM_Time+NEGATE
- No. Please make it for [7:30 pm](time) at Santa Clara.
## intent:INFORM_RestaurantName+INFORM_City+INFORM_Time
- the restaurant is la traviata.i want it in san fran.make it at [6:30 pm](time)<jupyter_text>domain.yml: Advanced Domain Example `domain.yml`*Note: As said above, this file has been auto-generated from the DSTC schema, and it's quite large. Below you can see only a part of this file. Feel free to examine the entire file.*The domain file now provides the list of slots and entities as defined by the DSTC schema, as well as the supported intent classes and system response text templates.<jupyter_code>DOMAIN_FPATH = "schema_resto_md_yaml/domain.yml"
!echo "domain file size (lines): $(wc -l {DOMAIN_FPATH})"

!echo -e '\n\nmost of domain file is listed below, just some portion of intents and response templates is skipped \n'
!head -40 {DOMAIN_FPATH} && echo "..."
!grep -B 1 -A 10 responses {DOMAIN_FPATH} && echo "..."
!grep --no-group-separator -A 1 OFFER_City: {DOMAIN_FPATH} && echo "..."
!grep --no-group-separator -A 1 CONFIRM_Time: {DOMAIN_FPATH} && echo "..."<jupyter_output>domain file size (lines): 478 schema_resto_md_yaml/domain.yml


most of domain file is listed below, just some portion of intents and response templates is skipped 

slots:
  price_range:
    type: text
  cuisine:
    type: text
  date:
    type: text
  restaurant_name:
    type: text
  intent:
    type: text
  party_size:
    type: text
  time:
    type: text
  has_live_music:
    type: text
  city:
    type: text
  serves_alcohol:
    type: text

entities:
- price_range
- cuisine
- date
- restaurant_name
- intent
- party_size
- time
- has_live_music
- city
- serves_alcohol

intents:
  - INFORM_RestaurantName+INFORM_Time
  - THANK_YOU+GOODBYE
  - INFORM_Time+INFORM_RestaurantName
  - INFORM_PriceRange+REQUEST_ALTS
  - REQUEST_PhoneNumber+REQUEST_StreetAddress+AFFIRM
...

responses:
  utter_REQUEST_Cuisine:
    - text: "What are you in the mood for? {cuisine}? {cuisine}"
  utter_REQUEST_City:
    - text: "What city are you interested in?"
  utter_OFFER_RestaurantName+OFFER_City:
    - [...]<jupyter_text>Now that we have all three key files, like in the Basic Demo, we can now proceed with our bot's training.<jupyter_code>from deeppavlov import configs
from deeppavlov.core.common.file import read_json


gobot_config = read_json(configs.go_bot.gobot_md_yaml_minimal)
gobot_config['chainer']['pipe'][-1]['slot_filler'] = {"config_path": "slotfill_config.json"}

gobot_config['metadata']['variables']['DATA_PATH'] = 'schema_resto_md_yaml'
gobot_config['metadata']['variables']['MODEL_PATH'] = '.'<jupyter_output><empty_output><jupyter_text>Since our data is the tutorial data we will use the same subsamples for all of train (training set), test (test set), and valid (validation set) subsamples.However, for a real DeepPavlov-based goal-oriented bot you should use different train, test, and valid sample stories.md files.<jupyter_code>!cp schema_resto_md_yaml/stories.md schema_resto_md_yaml/stories-trn.md
!cp schema_resto_md_yaml/stories.md schema_resto_md_yaml/stories-tst.md
!cp schema_resto_md_yaml/stories.md schema_resto_md_yaml/stories-val.md
from deeppavlov import train_model

train_model(gobot_config, download=False);
bot = build_model(gobot_config)<jupyter_output>2020-08-07 08:35:16.789 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 115: [loading vocabulary from /content/dp_big_demo_dir/word.dict]
2020-08-07 08:35:17.434 INFO in 'deeppavlov.dataset_readers.dstc2_reader'['dstc2_reader'] at line 112: [loading dialogs from /tmp/tmpu_18leqx]
2020-08-07 08:35:18.107 INFO in 'deeppavlov.dataset_readers.dstc2_reader'['dstc2_reader'] at line 112: [loading dialogs from /tmp/tmpu2tzz5gn]
2020-08-07 08:35:18.689 INFO in 'deeppavlov.dataset_readers.dstc2_reader'['dstc2_reader'] at line 112: [loading dialogs from /tmp/tmp4ceyk7ty]
2020-08-07 08:35:19.236 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 115: [loading vocabulary from /root/.deeppavlov/models/schema_ner/word.dict]
2020-08-07 08:35:19.240 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 115: [loading vocabulary from /root/.deeppavlov/models/schema_ner/tag.dict]
2020-08-07 08:35:19.243 INFO in 'deeppavlov.core.data.simple_vocab'['simple_[...]<jupyter_text>Let's see whether the bot works at all:<jupyter_code>bot.reset()

bot(["Hey!"])[0][0].actions_tuple

bot.reset()<jupyter_output><empty_output><jupyter_text>Ok, let's have a conversation:<jupyter_code>bot.reset()

bot(["Hi!"])[0][0].actions_tuple<jupyter_output><empty_output><jupyter_text>Awesome. Seems like our bot performs well. Let's get some action!<jupyter_code>bot(["I'd like to find a restaurant for this evening"])[0][0].actions_tuple<jupyter_output><empty_output><jupyter_text>The bot replies with the request to provide the necessary information, and we give it back:<jupyter_code>bot(["Somewhere in Oakland, sushi and for two people please"])[0][0].actions_tuple<jupyter_output><empty_output><jupyter_text>And so on.<jupyter_code>bot(["Cool! That's what I was looking for, thanks!"])[0][0].actions_tuple<jupyter_output><empty_output><jupyter_text>Let's say goodbye to our bot<jupyter_code>bot(["Bye bot"])[0][0].actions_tuple<jupyter_output><empty_output><jupyter_text>While it'd be nice for it to reply "Good bye!", it didn't. Why?Given that the DSTC dataset doesn't support this utterance, our bot can't properly react to such user's response. So to make our bot a bit more polite we have to add the -bye -bye utterances to the training data.Notice that you will have to add it to all 3 files (note that in case of `domain.yml` you have to add one line to *intents* and another two to *responses* sections of the file): stories.md```... goodbye* bye  - utter_bye...``` nlu.md```... intent:bye- goodbye- goodnight...``` domain.yml```...intents:  - bye...responses:  utter_bye:  - text: Bye!...```You will also have to re-use our stories.md for train, test, and valid stories.md files. Again, for the purposes of this demo, we use the same files for stories.md.However, for a real DeepPavlov-based goal-oriented bot you should use different train, test, and valid sample stories.md files.<jupyter_code>!sed -i -e "s|^\s*$|\* bye\n  - utter_BYE\n|g" {STORIES_FPATH}  # add bye to each story
!echo -e "  utter_BYE:\n    - text: \"Bye!\"" >> {DOMAIN_FPATH}  # add bye to nlu example
!grep -m 1 -A 3 bye ../dp_minimal_demo_dir/nlu.md >> {NLU_FPATH} && echo "" >> {NLU_FPATH}  # add bye to response templates
!sed -ie "s|intents:|intents:\n  - bye|g" {DOMAIN_FPATH}  # add bye to domain intents
!cp schema_resto_md_yaml/stories.md schema_resto_md_yaml/stories-trn.md
!cp schema_resto_md_yaml/stories.md schema_resto_md_yaml/stories-tst.md
!cp schema_resto_md_yaml/stories.md schema_resto_md_yaml/stories-val.md<jupyter_output><empty_output><jupyter_text>Re-training our bot:<jupyter_code>!rm -rf model  # remove the previous trained model
from deeppavlov import train_model

train_model(gobot_config, download=False);
bot = build_model(gobot_config)<jupyter_output><empty_output><jupyter_text>Checking it:<jupyter_code>bot.reset()
bot(["Hi!"])[0][0].actions_tuple
bot(["I'd like to find a restaurant for this evening"])[0][0].actions_tuple
bot(["Somewhere in Oakland, sushi and for two people please"])[0][0].actions_tuple
bot(["Cool! That's what I was looking for, thanks!"])[0][0].actions_tuple
bot(["Bye bot"])[0][0].actions_tuple<jupyter_output><empty_output><jupyter_text>Comparing with RASANow that we've run through a couple of demos, let's make sure that our configs work the same way as they do using RASA framework.<jupyter_code>%cd /content
!mkdir rasa_demo
%cd rasa_demo<jupyter_output>/content
/content/rasa_demo<jupyter_text>Let's install the RASA library<jupyter_code>!pip install rasa
!python -m spacy download en_core_web_md
!python -m spacy link en_core_web_md en
!mkdir data
!cp ../dp_big_demo_dir/schema_resto_md_yaml/{stories,nlu}.md data
!cp ../dp_big_demo_dir/schema_resto_md_yaml/domain.yml .<jupyter_output><empty_output><jupyter_text>We'll also use some simple rasa environment config from their repo<jupyter_code>!wget https://raw.githubusercontent.com/RasaHQ/rasa/1.10.x/examples/moodbot/config.yml
!rasa train<jupyter_output>[93m/usr/local/lib/python3.6/dist-packages/rasa/core/domain.py:151: FutureWarning: No tracker session configuration was found in the loaded domain. Domains without a session config will automatically receive a session expiration time of 60 minutes in Rasa version 2.0 if not configured otherwise.
  session_config = cls._get_session_config(data.get(SESSION_CONFIG_KEY, {}))
[0m[94mTraining Core model...[0m
2020-08-07 09:34:58 [1;30mINFO    [0m [34mroot[0m  - Generating grammar tables from /usr/lib/python3.6/lib2to3/Grammar.txt
2020-08-07 09:34:58 [1;30mINFO    [0m [34mroot[0m  - Generating grammar tables from /usr/lib/python3.6/lib2to3/PatternGrammar.txt
[93m/usr/local/lib/python3.6/dist-packages/rasa/core/domain.py:151: FutureWarning: No tracker session configuration was found in the loaded domain. Domains without a session config will automatically receive a session expiration time of 60 minutes in Rasa version 2.0 if not configured otherwise.
  session_config = cls._get_se[...]<jupyter_text>And when the bot is trained you can interact with it in the interactive mode:<jupyter_code>!cat | rasa shell<jupyter_output>2020-08-07 12:54:29 [1;30mINFO    [0m [34mroot[0m  - Generating grammar tables from /usr/lib/python3.6/lib2to3/Grammar.txt
2020-08-07 12:54:29 [1;30mINFO    [0m [34mroot[0m  - Generating grammar tables from /usr/lib/python3.6/lib2to3/PatternGrammar.txt
2020-08-07 12:54:29 [1;30mINFO    [0m [34mroot[0m  - Starting Rasa server on http://localhost:5005
2020-08-07 12:54:45 [1;30mINFO    [0m [34mrasa.nlu.components[0m  - Added 'SpacyNLP' to component cache. Key 'SpacyNLP-en'.
2020-08-07 12:54:51 [1;30mINFO    [0m [34mroot[0m  - Rasa server is up and running.
[92mBot loaded. Type a message and press enter (use '/stop' to exit): [0m
Warning: Input is not to a terminal (fd=0).
[15C[?7h[0m[?12l[?25hHi!
[0m
[J[?7h[0m[?12l[?25h[?2004l[94mhi[0m
[15C[?7h[0m[?12l[?25hI'd like to find a restaurant for this evening
[0m
[J[?7h[0m[?12l[?25h[?2004l[94mDo you have a preferred city and place?[0m
[15C[?7h[0m[?12l[?25hSomewhere in Oakland, sushi and for [...]
StarcoderdataJupyterStructuredCleanDedup