Skip to content
This repository has been archived by the owner on Feb 8, 2019. It is now read-only.

Home

Jump to bottom
eouia edited this page Sep 25, 2017 · 24 revisions

MMM-Assistant

Voice commander and Google-Assistant integrated for MagicMirror

Installation & Dependencies

  1. You SHOULD have a mic and speaker which can be used for recording and playing.
  2. You SHOULD be able to execute below commands on terminal.
arecord -d 10 test.wav && aplay test.wav

If fails, You should find the way to use mic and speaker on your MagicMirror. But I cannot give you a help about that kind of problem.

  1. This module MIGHT NOT be executed on RPI Zero or RPI 1. Because scripts are written in ES6 styles and there are some dependencies of advanced node and Electron. I didn't test this on RPI Zero or 1.

Step 1. pre-reqs

all required node modules are described in package.json but you might need some pre-reqs.

sudo apt-get update 
sudo apt-get upgrade 
sudo apt-get install libmagic-dev libatlas-base-dev libasound2-dev sox libsox-fmt-all libttspico-utils

I think, for RPI and ATB, this is enough. But for other platform or in some cases, you may need additional reqs. In that case, See below links;

Step 2. Install module

cd <Your MagicMirror Direcotry>/modules
git clone https://github.com/eouia/MMM-Assistant
cd MMM-Assistant
npm install

Step 3. Requirements.

3-1) Auth for Google Assistant & Cloud Speech

You need to get a JSON file for OAuth2 permissions. https://developers.google.com/assistant/sdk/develop/grpc/config-dev-project-and-account Create a project for MagicMirror. Enable Google Assistant API and Google Cloud Speech API.

detailed steps for Google Assistant API
  • Login on https://console.cloud.google.com/cloud-resource-manager
  • Select CREATE PROJECTS.
  • Give a project name to this new project.(Google will suggest a unique name. Anyway, the name is not important. set this by anything.) And press CREATE Button.
  • Now you might be in Dashboard of the newly created project. (Or select created project from your project lists)
  • Go to API Overview (API & Services > Dashboard) from Menu or Cards.
  • Select ENABLE APIS AND SERVICES
  • Search Google Assistant API and dive into.
  • Press ENABLE.
  • You need to create credentials. Go to https://console.developers.google.com/apis/credentials/oauthclient
  • Set the Configure consent screen. You need to set only Product name shown to users.
  • Select Application Type as other and give the name. Now you can get oAuth.
  • Download your client-secret-blahblahXXX.json and rename it as secret.json. Put that file in MMM-Assistant directory.
  • Then execute this on your MagicMirror terminal.(not via SSH)
cd <Your MagicMirror Direcotry>/modules/MMM-Assistant
node google-auth.js
  • It will try to open the browser for getting some credential keystring. copy and paste it where the google-auth.js prompts.
  • Now, you can find token.js under resources directory. Keep it safely.
detailed steps for Google Cloud Speech API
  • Back to API & Services > Dashboard again.
  • Select ENABLE APIS AND SERVICES
  • Search Google Cloud Speech API and dive into.
  • Press ENABLE.
  • Select Create Credentials and Service Account Key. (It's obvious to be able to use secret.json again. For safe, I'll create another key file for Google Cloud Speech)
  • Set the details (anything for service account name, Project>Owner for Role, anything for service account id, JSON for key type. After set, you can download your key file.
  • It would be used as config.stt.auth.keyFileName

Important Step 3-1) SHOULD be done before Step 4.

3-2) .pmdl for snowboy hotword detection.

This module needs 2 hotword models at least, one for activating Google-Assistant mode and the other for activating voice-commander mode. By default, smart_mirror.umdl and snowboy.umdl universal models(.umdl)are provided. You can add or change them with your personal models(.pmdl) Go to https://snowboy.kitt.ai/ and get .pmdl for your voice.

Step 4. Post Installation

cd <Your MagicMirror Direcotry>/modules/MMM-Assistant
npm install --save-dev electron-rebuild && ./node_modules/.bin/electron-rebuild --pre-gyp-fix

It takes somewhat long time.

Configuration

Simple version

{
  module: 'MMM-Assistant',
  position: 'bottom_left',
  config: {
    stt: {
      auth: [
        {
          projectId:'<your projectId>', //REQUIRED
          keyFilename: '<your project key file location>' //REQUIRED
        },
      ]
    }
  }
},

Full version & default values

{
  module: 'MMM-Assistant',
  position: 'bottom_left',
  config: {
    assistant: {
      auth: {
        keyFilePath: "secret.json", //When you want to change the location of this file, set this.
        savedTokensPath: "resources/tokens.js" //When you want to change the location of this file, set this.
      },
      audio: {
        encodingIn: "LINEAR16", //Ignore this. I think you don't need to change this.
        sampleRateOut: 16000 //Ignore this. I think you don't need to change this.
      }
    },
    snowboy: {
      models: [
        {
          file: "resources/smart_mirror.umdl",
          sensitivity: 0.5,
          hotwords : "MIRROR" //One of models should be "MIRROR"
        },
        {
          file: "resources/snowboy.umdl",
          sensitivity: 0.5,
          hotwords : "ASSISTANT" //One of models should be "ASSISTANT"
        }
      ]
    },
    record: {
      threshold: 0, //Ignore this. I think you don't need to change this.
      verbose:false, //true for checking recording status.
      recordProgram: 'arecord', //You can use 'rec', 'sox'. But I recommend use 'arecord'.
      silence: 2.0 //Ignore this. I think you don't need to change this.
    },
    stt: {
      auth: [{
        projectId: '<your google Project ID>', //ProjectId from Google Console
        keyFilename: '<your google Project Key File>'
      }], //You can use multi accounts for saving money.
      request: {
        encoding: 'LINEAR16', //Ignore this. I think you don't need to change this.
        sampleRateHertz: 16000, //Ignore this. I think you don't need to change this.
        languageCode: 'en-US' //See https://cloud.google.com/speech/docs/languages
      },
    },
    speak: {
      useAlert: true, //If you want to show the text of speech, set this true. But It could be ignored by command of modules directly.
      language: 'en-US', //If you want to set the default language of speech of command result, set this.
    },
    alias: [
      {
        "help :command" : ["teach me :command", "what is :command"]
      }
    ] // You can use aliases for difficult pronunciation or easy using.
  }
},

available languageCodes for STT : See https://cloud.google.com/speech/docs/languages

available languages for speak :
en-US English, en-GB Great Britain, de-DE German, es-ES Spanish, fr-FR French, it-IT Italian Other language supports are not yet. (I cannot find any easy way for implements. I hope someone to help me.)

Clone this wiki locally