ChatGPT Smart Speaker (speech recognition and text-to-speech using OpenAI and Google Speech Recognition)

Video Demos

Video Demo using activation word "Jeffers"

Video Demo with Vision

Equipment List:

- Raspberry Pi 4b 4GB

- VMini External USB Stereo Speaker

- VReSpeaker 4-Mic Array

- ANSMANN 10,000mAh Type-C 20W PD Power Bank

Running on your PC/Mac (use the chat.py or test.py script)

The chat.py and test.py scripts run directly on your PC/Mac. They both allow you to use speech recognition to input a prompt, send the prompt to OpenAI to generate a response, and then use gTTS to convert the response to an audio file and play the audio file on your Mac/PC. Your PC/Mac must have a working default microphone and speakers for this script to work. Please note that these scripts were designed on a Mac, so additional dependencies may be required on Windows and Linux. The difference between them is that chat.py is faster and always on and test.py acts like a standard smart speaker - only working once it hears the activation command (currently set to 'Jeffers').

Running on Raspberry Pi (use the pi.py script)

The pi.py script is a new and more advanced custom version of the smart_speaker.py script and is the most advanced script similar to a real smart speaker. The purpose of this script is to offload the wake up word to a custom model build via PicoVoice (https://console.picovoice.ai/). This improves efficiency and long term usage reliability. This script will be the main script for development moving forward due to greater reliability and more advanced features to be added regularly.

Prerequisites - chat.py

• You need to have a valid OpenAI API key. You can sign up for a free API key at https://platform.openai.com.
• You'll need to be running Python version 3.7.3 or higher. I am using 3.11.4 on a Mac and 3.7.3 on Raspberry Pi.
• Run brew install portaudio after installing HomeBrew: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
• You need to install the following packages: openai, gTTS, pyaudio, SpeechRecognition, playsound, python-dotenv and pyobjc if you are on a Mac. You can install these packages using pip or use pipenv if you wish to contain a virtual environment.
• Firstly, update your tools: pip install --upgrade pip setuptools then pip install openai pyaudio SpeechRecognition gTTS playsound python-dotenv apa102-pi gpiozero pyobjc

Prerequisites - pi.py

To run pi.py you will need a Raspberry Pi 4b (I'm using the 4GB model but 2GB should be enough), ReSpeaker 4-Mic Array for Raspberry Pi and USB speakers.

You will also need a developer account and API key with OpenAI (https://platform.openai.com/overview), a Tavily Search agent API key (https://app.tavily.com/sign-in) and an Access Key and Custom Voice Model with PicoVoice (https://console.picovoice.ai/) and (https://console.picovoice.ai/ppn respectively. Please create your own voice model and download the correct version for use on a Raspberry Pi)

Now on to the Pi setup. Let's get started!

Run the following on your Raspberry Pi terminal:

1. sudo apt update

2. sudo apt install python3-gpiozero

3. git clone https://github.com/Olney1/ChatGPT-OpenAI-Smart-Speaker

4. Firstly, update your tools: pip install --upgrade pip setuptools then pip install openai pyaudio SpeechRecognition gTTS pydub python-dotenv apa102-pi gpiozero Next, install the dependencies, pip install -r requirements.txt. I am using Python 3.9 #!/usr/bin/env python3.9. You can install these packages using pip or use pipenv if you wish to contain a virtual environment.

5. PyAudio relies on PortAudio as a dependency. You can install it using the following command: sudo apt-get install portaudio19-dev

6. Pydub dependencies: You need to have ffmpeg installed on your system. On a Raspberry Pi you can install it using: sudo apt-get install ffmpeg. You may also need simpleaudio if you run into issues with the script hanging when finding the wake word, so it's best to install these packages just in case: sudo apt-get install python3-dev (for development headers to compile) and install simpleaudio (for a different backend to play mp3 files) and sudo apt-get install libasound2-dev (necessary dependencies).

7. If you are using the RESPEAKER, follow this guide to install the required dependencies: (https://wiki.seeedstudio.com/ReSpeaker_4_Mic_Array_for_Raspberry_Pi/#getting-started). Then install support for the lights on the RESPEAKER board. You'll need APA102 LED: sudo apt install -y python3-rpi.gpio and then sudo pip3 install apa102-pi.

8. Activate SPI: sudo raspi-config; Go to "Interface Options"; Go to "SPI"; Enable SPI; While you are at it: Do change the default password! Exit the tool and reboot.

9. Get the Seeed voice card source code, install and reboot: git clone https://github.com/HinTak/seeed-voicecard.git cd seeed-voicecard sudo ./install.sh sudo reboot now

10. Finally, load audio output on Raspberry Pi sudo raspi-config -Select 1 System options -Select S2 Audio -Select your preferred Audio output device -Select Finish

Usage - applies to chat.py:

1. You'll need to set up the environment variables for your Open API Key. To do this create a .env file in the same directory and add your API Key to the file like this: OPENAI_API_KEY="API KEY GOES HERE". This is safer than hard coding your API key into the program. You must not change the name of the variable OPENAI_API_KEY.
2. Run the script using python chat.py.
3. The script will prompt you to say something. Speak a sentence into your microphone. You may need to allow the program permission to access your microphone on a Mac, a prompt should appear when running the program.
4. The script will send the spoken sentence to OpenAI, generate a response using the text-to-speech model, and play the response as an audio file.

Usage - applies to pi.py

1. You'll need to set up the environment variables for your Open API Key, PicoVoice Access Key and Tavily API key for agent searches. To do this create a .env file in the same directory and add your API Keys to the file like this: OPENAI_API_KEY="API KEY GOES HERE" and ACCESS_KEY="PICOVOICE ACCESS KEY GOES HERE" and TAVILY_API_KEY="API KEY GOES HERE". This is safer than hard coding your API key into the program.
2. Ensure that you have the pi.py script along with apa102.py and alexa_led_pattern.py scripts in the same folder saved on your Pi if using ReSpeaker.
3. Run the script using python3 pi.py or python3 pi.py 2> /dev/null on the Raspberry Pi. The second option omits all developer warnings and errors to keep the console focused purely on the print statements.
4. The script will prompt you to say the wake word which is programmed into the wake word custom model by Picovoice as 'Jeffers'. You can change this to any name you want. Once the wake word has been detected the lights will light up blue. It will now be ready for you to ask your question. When you have asked your question, or when the microphone picks up and processes noise, the lights will rotate a blue colour meaning that your recording sample/question is being sent to OpenAI.
5. The script will then generate a response using the text-to-speech model, and play the response as an audio file.

Customisation

• You can change the OpenAI model engine by modifying the value of model_engine. For example, to use the "gpt-3.5-turbo" model for a cheaper and quicker response but with a knowledge cut-off to Sep 2021, set model_engine = "gpt-3.5-turbo".
• You can change the language of the generated audio file by modifying the value of language. For example, to generate audio in French, set language = 'fr'.
• You can adjust the temperature parameter in the following line to control the randomness of the generated response:

response = client.chat.completions.create(
        model=model_engine,
        messages=[{"role": "system", "content": "You are a helpful smart speaker called Jeffers!"}, # Play about with more context here.
                  {"role": "user", "content": prompt}],
        max_tokens=1024,
        n=1,
        temperature=0.7,
    )
    return response

Higher values of temperature will result in more diverse and random responses, while lower values will result in more deterministic responses.

Important notes for Raspberry Pi Installation

As of May 2024, Seeed Studio has listed the ReSpeaker series among its retired products. It may not be compatible with the Raspberry Pi 5 due to hardware changes.

It is highly recommended to install the legacy version of Raspberry Pi on a Rasberry Pi 4b model if you have an ReSPEAKER. You can also simply buy a micro USB microphone and configure the input source for this using alsamixer and currently still use the ReSPEAKER for the lighting pattern.

If you are using the same USB speaker in my video you will need to run sudo apt-get install pulseaudio to install support for this. This may also require you to set a command to start pulseaudio on every boot: pulseaudio --start.

Adding a Start Command on Boot

Open the terminal and type: sudo nano /etc/rc. local

After important network/start commands add this: su -l pi -c '/usr/bin/python3 /home/pi/ChatGPT-OpenAI-Smart-Speaker/ && pulseaudio --start && python3 pi.py 2> /dev/null’

Be sure to leave the line exit 0 at the end, then save the file and exit. In nano, to exit, type Ctrl-x, and then Y

ReSpeaker

If you want to use ReSpeaker for the lights, you can purchase this from most of the major online stores that stock Raspberry Pi. Here is the online guide: https://wiki.seeedstudio.com/ReSpeaker_4_Mic_Array_for_Raspberry_Pi/

To test your microphone and speakers install Audacity on your Raspberry Pi:

sudo apt update

sudo apt install audacity

audacity

Other Possible Issues

On the raspberry pi you may encounter an error regarding the installation of flac.

See here for the resolution: https://raspberrypi.stackexchange.com/questions/137630/im-unable-to-install-flac-on-my-raspberry-pi-3

The files you will need are going to be here: https://archive.raspbian.org/raspbian/pool/main/f/flac/
Please note the links below may have changed or be updated, so please refer back to this link above for the latest file names and then update your command below.

sudo apt-get install libogg0

$ wget https://archive.raspbian.org/raspbian/pool/main/f/flac/libflac8_1.3.2-3+deb10u3_armhf.deb

$ wget https://archive.raspbian.org/raspbian/pool/main/f/flac/flac_1.3.2-3+deb10u3_armhf.deb

$ sudo dpkg -i libflac8_1.3.2-3+deb10u3_armhf.deb

$ sudo dpkg -i flac_1.3.2-3+deb10u3_armhf.deb

$ which flac /usr/bin/flac

sudo reboot

$ flac --version flac 1.3.2

You may find you need to install GStreamer if you encounter errors regarding Gst.

Install GStreamer: Open a terminal and run the following command to install GStreamer and its base plugins:

sudo apt-get install gstreamer1.0-tools gstreamer1.0-plugins-base gstreamer1.0-plugins-good This installs the GStreamer core, along with a set of essential and good-quality plugins.

Next, you need to install the Python bindings for GStreamer. Use this command:

sudo apt-get install python3-gst-1.0 This command installs the GStreamer bindings for Python 3.

Install Additional GStreamer Plugins (if needed): Depending on the audio formats you need to work with, you might need additional GStreamer plugins. For example, to install plugins for MP3 playback, use:

sudo apt-get install gstreamer1.0-plugins-ugly

To quit a running script on Pi from boot: ALT + PrtScSysRq (or Print button) + K

Credit to:

https://github.com/tinue/apa102-pi & Seeed Technology Limited for supplementary code.

Read more about what is next for the project

https://medium.com/@ben_olney/openai-smart-speaker-with-raspberry-pi-5e284d21a53e