1. Open the Visual Studio Code from the Applications directory on the flash drive
2. Click on the small green square in the bottom left corner of the screen
   
   

   

3. Select the “Connect Current Window to Host…” option
   
   

   

4. Type “pi@pibot##.local” and press enter
   
   

   Replace the text “pibot##” with the label on the pibot

   The password to enter when prompted is “DogsAndCatsAreNice2.” without the quotes. The '.' at the end is important!

   It may take a couple of minutes while it installs some files on the PiBot

   

5. Click the “Open Folder” button
   
   

   

6. Select the “pi” folder and click the “OK” button
   
   

   

7. Open a terminal in Visual Studio Code from the Terminal→New Terminal menu
8. Type the following commands and press enter after each line to install the software required for this lesson
   
   

   sudo apt-get update
   sudo apt-get install python3-picamera python3-pil python3-opencv libjpeg-dev libfreetype6 libfreetype6-dev zlib1g-dev libwebp-dev libopenjp2-7-dev libatlas-base-dev fonts-freefont-ttf git

9. Type the following commands and press enter after each line to install the Python libraries required for this lesson
   
   

   sudo pip3 install --upgrade numpy
   sudo pip3 install imutils

10. Type the following command and press enter to install the Python Tensorflow Lite runtime for this lesson
    \\

    sudo pip3 install https://github.com/google-coral/pycoral/releases/download/v2.0.0/tflite_runtime-2.5.0.post1-cp37-cp37m-linux_armv7l.whl

11. Type the following command and press enter to download the Python code for this lesson
    
    

    git clone https://git.jonathan.struebel-family.net/robot-club/pibot/lesson5.git

12. Type the following command and press enter to change into the lesson 5 directory
    
    

    cd lesson5

13. Type the following commands and press enter after each line to download the object detection models
    
    

    curl -L 'https://tfhub.dev/tensorflow/lite-model/efficientdet/lite0/detection/metadata/1?lite-format=tflite' -o efficientdet_lite0.tflite
    curl -L 'https://storage.googleapis.com/download.tensorflow.org/models/tflite/edgetpu/efficientdet_lite0_edgetpu_metadata.tflite' -o efficientdet_lite0_edgetpu.tflite

14. Type the following command and press enter to start the object detection MJPEG Streamer server for this lesson
    
    

    python3 webstreaming.py

15. Open the Safari App, type http://pibot##.local:8080 in the address bar, and press enter to test the object detection MJPEG Streamer software
    
    

    Replace the text “pibot##” with the label on the pibot

    Verify that the PiBot detects objects in front of the camera by drawing a box around them

    

16. Select the “osoyoowebcar” directory on the left and then click on the templates directory and select the index.html file
    
    

    

17. Replace the 8899 on line 34 with 8080
18. Save the change by using the File→Save menu
19. Open a terminal in Visual Studio Code from the Terminal→New Terminal menu
20. Type the following command and press enter to run the Python code for this lesson
    
    

    cd osoyoowebcar
    sudo python3 webcar.py

21. Open the Safari App, type http://pibot##.local in the address bar, and press enter
    
    

    Replace the text “pibot##” with the label on the pibot

    

22. Drive the PiBot around and see what objects it detects

The webstreaming.py file is another Flask app that streams images from the CSI Camera connected to the Raspberry Pi to your web browser after searching for objects within the image.

1. In the webstreaming.py file, find the decorators. What do you think each of the functions is doing?
   
   

   If you need a refresher on what decorators are, review Lesson 4

2. Find the generate function. What do you think this function does? Where is the function called?
   
   

   outputFrame is a global buffer that stores exactly one image from the camera. The cv2.imencode function converts (encodes) an image frame into a standard format, in this case the JPEG format

   The yield function temporarily stops the loop that it's in and returns the data that is passed as a parameter to the yield function.

3. Find the location in the code where data is saved to the outputFrame variable. What function is it in? What do you think this function does?
   
   

   The while loop reads each image from the camera, performs object detection on it, draws boxes around the objects, and prints some statistics on the image before saving the image to the global outputFrame buffer

   The detector variable is an instance of the ObjectDetector class. It get initialized on line 127

   The detect_objects function is run as a separate thread. Look at line 221 to see how to do that. Threads are a powerful way to allow different functions in the code to run in parallel. Using threads can help your code run faster.

4. Open the object_detector.py file
   
   

   

5. Find the detect function in the ObjectDetector class. Explore the different functions to see what it is doing.
   
   

   When self is used in front of a function call it is referring to a function defined within that same class

   The detect function uses the Tensorflow Lite (or TFLite) library to run machine learning neural networks that process data and provide information about what is in that data. Tensorflow Lite was developed by Google. Learn more about machine learning and neural nets in Algorithm Explanation.

6. When you're finished driving the PiBot around, hold down the control + C keys and type the following command in the Visual Studio Code terminal to shutdown the PiBot
   
   

   sudo shutdown -h now

7. Click on the small green rectangle in the bottom left corner of the screen and select “Close Remote Connection”
8. Wait 3-5 minutes for the green light on the Raspberry Pi to stop flashing and turn off the switch on the PiBot battery

Return to Lesson 5