Handle user input with ASR

A code snippet that shows how to handle a user input with Automatic Speech Recognition (ASR). The user says their input at the prompt and their input is acknowledged via a speech-to-text message.

Example

Prerequisites

A Nexmo application contains the required configuration for your project. You can create an application using the Nexmo CLI (see below) or via the dashboard. To learn more about applications see our Nexmo concepts guide.

Install the CLI

npm install -g nexmo-cli

Create an application

Once you have the CLI installed you can use it to create a Nexmo application. Run the following command and make a note of the application ID that it returns. This is the value to use in NEXMO_APPLICATION_ID in the example below. It will also create private.key in the current directory which you will need in the Initialize your dependencies step

Nexmo needs to connect to your local machine to access your answer_url. We recommend using ngrok to do this. Make sure to change demo.ngrok.io in the examples below to your own ngrok URL.

nexmo app:create "User Input ASR Example" http://demo.ngrok.io/webhooks/answer http://demo.ngrok.io/webhooks/events --keyfile private.key

npm install express body-parser

Create a file named asr.js and add the following code:

Copy to Clipboard
const app = require('express')()
const bodyParser = require('body-parser')

app.use(bodyParser.json())

View full source

Write the code

Add the following to asr.js:

Copy to Clipboard
const onInboundCall = (request, response) => {
  const ncco = [{
      action: 'talk',
      text: 'Please say something',
    },
    {
      action: 'input',
      eventUrl: [`${request.protocol}://${request.get('host')}/webhooks/asr`],
      speech: {
        endOnSilence: 1,
        language: "en-US",
        uuid: [request.query.uuid]
      }
    }
  ]
  response.json(ncco)
}

const onInput = (request, response) => {
  const speech = request.body.speech.results[0].text

  const ncco = [{
    action: 'talk',
    text: `You said ${speech}`
  }]

  response.json(ncco)
}

app
  .get('/webhooks/answer', onInboundCall)
  .post('/webhooks/asr', onInput)

app.listen(3000)

View full source

Run your code

Save this file to your machine and run it:

node asr.js

Prerequisites

A Nexmo application contains the required configuration for your project. You can create an application using the Nexmo CLI (see below) or via the dashboard. To learn more about applications see our Nexmo concepts guide.

Install the CLI

npm install -g nexmo-cli

Create an application

Once you have the CLI installed you can use it to create a Nexmo application. Run the following command and make a note of the application ID that it returns. This is the value to use in NEXMO_APPLICATION_ID in the example below. It will also create private.key in the current directory which you will need in the Initialize your dependencies step

Nexmo needs to connect to your local machine to access your answer_url. We recommend using ngrok to do this. Make sure to change demo.ngrok.io in the examples below to your own ngrok URL.

nexmo app:create "User Input ASR Example" http://demo.ngrok.io/webhooks/answer http://demo.ngrok.io/webhooks/events --keyfile private.key

Add the following to `build.gradle`:

compile 'com.vonage:client:5.5.0'
compile 'com.sparkjava:spark-core:2.7.2'

Write the code

Add the following to the main method of the AsrInput class:

Copy to Clipboard
/*
 * Route to answer incoming calls.
 */
Route answerCallRoute = (req, res) -> {
    TalkAction intro = TalkAction
            .builder("Please say something")
            .build();

    SpeechSettings speechSettings = new SpeechSettings();
    speechSettings.setLanguage(SpeechSettings.Language.ENGLISH_UNITED_STATES);

    InputAction input = InputAction.builder()
            .type(Collections.singletonList("speech"))
            .eventUrl(String.format("%s://%s/webhooks/asr", req.scheme(), req.host()))
            .speech(speechSettings)
            .build();


    res.type("application/json");

    return new Ncco(intro, input).toJson();
};

/*
 * Route which returns NCCO saying which word was recognized.
 */
Route speechInputRoute = (req, res) -> {
    InputEvent event = InputEvent.fromJson(req.body());

    TalkAction response = TalkAction.builder(String.format("You said %s, Goodbye.",
            event.getSpeech().getResults().iterator().next().getText()
    )).build();
    res.type("application/json");

    return new Ncco(response).toJson();
};

Spark.port(3000);
Spark.get("/webhooks/answer", answerCallRoute);
Spark.post("/webhooks/asr", speechInputRoute);

View full source

Run your code

We can use the application plugin for Gradle to simplify the running of our application. Update your build.gradle with the following:

Copy to Clipboard
  apply plugin: 'application'
  mainClassName = project.hasProperty('main') ? project.getProperty('main') : ''

Run the following gradle command to execute your application, replacing com.vonage.quickstart.voice with the package containing AsrInput:

gradle run -Pmain=com.vonage.quickstart.voice.AsrInput

Prerequisites

A Nexmo application contains the required configuration for your project. You can create an application using the Nexmo CLI (see below) or via the dashboard. To learn more about applications see our Nexmo concepts guide.

Install the CLI

npm install -g nexmo-cli

Create an application

Once you have the CLI installed you can use it to create a Nexmo application. Run the following command and make a note of the application ID that it returns. This is the value to use in NEXMO_APPLICATION_ID in the example below. It will also create private.key in the current directory which you will need in the Initialize your dependencies step

Nexmo needs to connect to your local machine to access your answer_url. We recommend using ngrok to do this. Make sure to change demo.ngrok.io in the examples below to your own ngrok URL.

nexmo app:create "User Input ASR Example" http://demo.ngrok.io/webhooks/answer http://demo.ngrok.io/webhooks/events --keyfile private.key

Install-Package Vonage

Create a file named AsrController.cs and add the following code:

Copy to Clipboard
using Vonage.Voice.AnswerWebhooks;
using Vonage.Voice.EventWebhooks;
using Vonage.Voice.Nccos;
using Vonage.Utility;

View full source

Write the code

Add the following to AsrController.cs:

Copy to Clipboard
[HttpGet("[controller]/webhooks/answer")]
public IActionResult Answer()
{
    var host = Request.Host.ToString();
    //Uncomment the next line if using ngrok with --host-header option
    //host = Request.Headers["X-Original-Host"];

    var request = WebhookParser.ParseQuery<Answer>(Request.Query);
    var eventUrl = $"{Request.Scheme}://{host}/webhooks/asr";
    var speechSettings = new SpeechSettings { Language = "en-US", EndOnSilence = 1, Uuid = new[] { request.Uuid } };
    var inputAction = new MultiInputAction { Speech = speechSettings, EventUrl = new[] { eventUrl } };

    var talkAction = new TalkAction { Text = "Please speak now" };

    var ncco = new Ncco(talkAction, inputAction);
    return Ok(ncco.ToString());
}

[HttpPost("/webhooks/asr")]
public async Task<IActionResult> OnInput()
{            
    var input = await WebhookParser.ParseWebhookAsync<MultiInput>(Request.Body, Request.ContentType);
    var talkAction = new TalkAction();
    talkAction.Text = input.Speech.SpeechResults[0].Text;
    var ncco = new Ncco(talkAction);
    return Ok(ncco.ToString());
}

View full source

Prerequisites

A Nexmo application contains the required configuration for your project. You can create an application using the Nexmo CLI (see below) or via the dashboard. To learn more about applications see our Nexmo concepts guide.

Install the CLI

npm install -g nexmo-cli

Create an application

Once you have the CLI installed you can use it to create a Nexmo application. Run the following command and make a note of the application ID that it returns. This is the value to use in NEXMO_APPLICATION_ID in the example below. It will also create private.key in the current directory which you will need in the Initialize your dependencies step

Nexmo needs to connect to your local machine to access your answer_url. We recommend using ngrok to do this. Make sure to change demo.ngrok.io in the examples below to your own ngrok URL.

nexmo app:create "User Input ASR Example" http://demo.ngrok.io/webhooks/answer http://demo.ngrok.io/webhooks/events --keyfile private.key

composer require slim/slim:^3.8

Write the code

Add the following to index.php:

Copy to Clipboard
use Vonage\Voice\NCCO\Action\Talk;
use Vonage\Voice\NCCO\Action\Input;
use \Psr\Http\Message\ResponseInterface as Response;
use \Psr\Http\Message\ServerRequestInterface as Request;

require 'vendor/autoload.php';

$app = new \Slim\App;

$app->get('/webhooks/answer', function (Request $request, Response $response) {
    $uri = $request->getUri();
    $url = $uri->getScheme().'://'.$uri->getHost().':'.$uri->getPort().'/webhooks/asr';

    $inputAction = new Input();
    $inputAction
        ->setSpeechEndOnSilence(true)
        ->setSpeechLanguage('en-US')
        ->setEventWebhook(new Webhook($url))
    ;
    $ncco = new NCCO();
    $ncco
        ->addAction(new Talk('Please say something'))
        ->addAction($inputAction)
    ;

    return $response->withJson($ncco->toArray());
});

$app->map(['GET', 'POST'], '/webhooks/asr', function (Request $request, Response $response) {
    /** @var InputWebhook $input */
    $input = Factory::createFromRequest($request);

    $ncco = new NCCO();
    $ncco->addAction(new Talk('You said ' . $input->getSpeech()['results'][0]['text']));

    return $response->withJson($ncco->toArray());
});

$app->run();

View full source

Run your code

Save this file to your machine and run it:

php -t . -S localhost:3000

Prerequisites

A Nexmo application contains the required configuration for your project. You can create an application using the Nexmo CLI (see below) or via the dashboard. To learn more about applications see our Nexmo concepts guide.

Install the CLI

npm install -g nexmo-cli

Create an application

Once you have the CLI installed you can use it to create a Nexmo application. Run the following command and make a note of the application ID that it returns. This is the value to use in NEXMO_APPLICATION_ID in the example below. It will also create private.key in the current directory which you will need in the Initialize your dependencies step

Nexmo needs to connect to your local machine to access your answer_url. We recommend using ngrok to do this. Make sure to change demo.ngrok.io in the examples below to your own ngrok URL.

nexmo app:create "User Input ASR Example" http://demo.ngrok.io/webhooks/answer http://demo.ngrok.io/webhooks/events --keyfile private.key

pip install Flask request jsonify

Create a file named handle-user-input-with-asr.py and add the following code:

Copy to Clipboard
#!/usr/bin/env python3
from flask import Flask, request, jsonify

app = Flask(__name__)

View full source

Write the code

Add the following to handle-user-input-with-asr.py:

Copy to Clipboard
@app.route("/webhooks/answer", methods=["POST", "GET"])
def answer_call():
    ncco = [
        {"action": "talk", "text": "Please, tell me something",},
        {
            "action": "input",
            "eventUrl": [
                "{host}{endpoint}".format(
                    host=request.host_url, endpoint="webhooks/asr"
                )
            ],
            "speech": {
                "endOnSilence": 1,
                "language": "en-US",
                "uuid": [request.args.get("uuid")],
            },
        },
    ]
    return jsonify(ncco)


@app.route("/webhooks/asr", methods=["POST", "GET"])
def answer_asr():
    body = request.get_json()
    if body is not None and "speech" in body:
        speech = body["speech"]["results"][0]["text"]
        ncco = [
            {"action": "talk", "text": "Hello ,you said {speech}".format(speech=speech)}
        ]
    else:
        ncco = [{"action": "talk", "text": "Sorry, i don't undertand. Bye"}]

    return jsonify(ncco)


if __name__ == "__main__":
    app.run(port=3000)

View full source

Run your code

Save this file to your machine and run it:

python handle-user-input-with-asr.py

Prerequisites

A Nexmo application contains the required configuration for your project. You can create an application using the Nexmo CLI (see below) or via the dashboard. To learn more about applications see our Nexmo concepts guide.

Install the CLI

npm install -g nexmo-cli

Create an application

Once you have the CLI installed you can use it to create a Nexmo application. Run the following command and make a note of the application ID that it returns. This is the value to use in NEXMO_APPLICATION_ID in the example below. It will also create private.key in the current directory which you will need in the Initialize your dependencies step

Nexmo needs to connect to your local machine to access your answer_url. We recommend using ngrok to do this. Make sure to change demo.ngrok.io in the examples below to your own ngrok URL.

nexmo app:create "User Input ASR Example" http://demo.ngrok.io/webhooks/answer http://demo.ngrok.io/webhooks/events --keyfile private.key

gem install sinatra sinatra-contrib rack-contrib

Create a file named answer-inbound-call-with-asr.rb and add the following code:

Copy to Clipboard
require 'sinatra'
require 'sinatra/multi_route'
require 'rack/contrib'

use Rack::JSONBodyParser

View full source

Write the code

Add the following to answer-inbound-call-with-asr.rb:

Copy to Clipboard
before do
  content_type :json
end

route :get, :post, '/webhooks/answer' do
  [
    {
      action: 'talk',
      text: 'Please say something'
    },
    {
      action: 'input',
      eventUrl: ["#{request.base_url}/webhooks/asr"],
      speech: {
        endOnSilence: 1,
        uuid: [params[:uuid]],
        language: 'en-US'
      }
    }
  ].to_json
end

route :post, '/webhooks/asr' do
  [{
    action: 'talk',
    text: "You said #{params["speech"]["results"][0]["text"]}"
  }].to_json
end

route :post, '/webhooks/event' do
  puts params
end

set :port, 3000

View full source

Run your code

Save this file to your machine and run it:

ruby answer-inbound-call-with-asr.rb

Try it out

Call your Vonage Number. When the call is answered you will be asked to say a message. When you are finished, you will then hear your message repeated back to you via speech-to-text.