Advertisement:
Read Later
In this tutorial, I will discuss modifying the Edge Impulse ESP-EYE firmware in order to run it on DFRobot FireBeetle ESP32 and FireBeetle Camera&Audio Media Board.
Since FireBeetle development boards and add-ons are relatively more budget-friendly than their counterparts on the market, I decided to share instructions on collecting data and running neural network models via FireBeetle ESP32 flashed with the modified Edge Impulse ESP-EYE firmware.
While modifying the ESP-EYE firmware, I encountered various errors and bugs due to the fact that FireBeetle products were not designed to run neural network models or be compatible with Edge Impulse. However, I managed to run the ESP-EYE firmware successfully in the end.
After following the instructions below, you can display a real-time video stream, capture images, and collect data from analog sensors on Edge Impulse. Nevertheless, the built-in I2S microphone is not working since it is not compatible with the current ESP-IDF version (4.4). Subsequently, I will explain the ESP-IDF I2S microphone bug in detail.
#️⃣ First of all, I installed the Edge Impulse CLI on my computer, as explained in this tutorial.
#️⃣ Then, I downloaded the Edge Impulse ESP-EYE firmware and its requirements by following these instructions.
#️⃣ Also, I installed the ESP IDF v4.4 from here.
#️⃣ After installing all the necessary tools and files, I started to modify the firmware.
📷 Camera (OV7725)
#️⃣ First, I added the correct pin connections for the FireBeetle camera&audio board in the ei_camera.h file.
#define CAMERA_MODEL_FIREBEETLE_COVER
...
#elif defined(CAMERA_MODEL_FIREBEETLE_COVER)
// FireBeetle Covers-Camera & Audio Media Board
// https://wiki.dfrobot.com/FireBeetle_Covers-Camera%26Audio_Media_Board_SKU_DFR0498
//
#define PWDN_GPIO_NUM -1
#define RESET_GPIO_NUM 0
#define XCLK_GPIO_NUM 21
#define SIOD_GPIO_NUM 26
#define SIOC_GPIO_NUM 27
#define Y9_GPIO_NUM 35
#define Y8_GPIO_NUM 34
#define Y7_GPIO_NUM 39
#define Y6_GPIO_NUM 36
#define Y5_GPIO_NUM 19
#define Y4_GPIO_NUM 18
#define Y3_GPIO_NUM 5
#define Y2_GPIO_NUM 17
#define VSYNC_GPIO_NUM 25
#define HREF_GPIO_NUM 23
#define PCLK_GPIO_NUM 22
#️⃣ Although FireBeetle ESP32 does not have SPI memory (PSRAM), the camera&audio board shares pin connections with the default PSRAM connections in the ESP-IDF. Therefore, PSRAM allocation should be disabled to avoid errors.
#️⃣ To disable PSRAM, open the ESP-IDF PowerShell and enter this command:
idf.py menuconfig
#️⃣ When the configuration page pops up, go to Component config ➡ ESP32-specific and disable Support for external, SPI-connected RAM.
#️⃣ After disabling PSRAM on the ESP-IDF, I modified the ei_camera.cpp file to save the image buffer to DRAM instead of PSRAM.
#️⃣ Also, I changed the pixel format since the OV7725 camera is not compatible with the JPEG format.
Available formats: YUV422, GRAYSCALE, or RGB565.
//XCLK 20MHz or 10MHz for OV2640 double FPS (Experimental)
.xclk_freq_hz = 20000000,
.ledc_timer = LEDC_TIMER_0,
.ledc_channel = LEDC_CHANNEL_0,
.pixel_format = PIXFORMAT_YUV422, //YUV422,GRAYSCALE,RGB565,JPEG
.frame_size = FRAMESIZE_QVGA, //QQVGA-UXGA Do not use sizes above QVGA when not JPEG
.jpeg_quality = 20, //0-63 lower number means higher quality
.fb_count = 1, //if more than one, i2s runs in continuous mode. Use only with JPEG
// No PSRAM
.fb_location = CAMERA_FB_IN_DRAM,
#️⃣ After flashing FireBeetle ESP32 with the modified files, you can display a real-time video stream and capture images (data) via the Edge Impulse CLI.
🎤 I2S Microphone (NAU8822)
#️⃣ First, I adjusted the NAU8822 I2S microphone pin connections and settings in the ei_microphone.cpp file according to DFRobot's official library.
// Start listening for audio: MONO @ 8/16KHz
i2s_config_t i2s_config = {
.mode = (i2s_mode_t)(I2S_MODE_MASTER | I2S_MODE_RX | I2S_MODE_TX),
.sample_rate = sampling_rate,
.bits_per_sample = (i2s_bits_per_sample_t)32,
.channel_format = I2S_CHANNEL_FMT_RIGHT_LEFT,
.communication_format = I2S_COMM_FORMAT_I2S,
.intr_alloc_flags = ESP_INTR_FLAG_LEVEL1,
.dma_buf_count = 8,
.dma_buf_len = 128,
.use_apll = false,
.tx_desc_auto_clear = false,
.fixed_mclk = -1,
};
i2s_pin_config_t pin_config = {
.bck_io_num = 5,
.ws_io_num = 17,
.data_out_num = 0,
.data_in_num = 39
};
esp_err_t ret = 0;
#️⃣ Even though the Edge Impulse CLI did not show any errors, I always obtained 0 from the microphone.
#️⃣ I found out this is a well-known issue with the ESP-IDF versions later than 4.0 for some I2S microphones:
#️⃣ When I tested my code with the same pin connections on the Arduino IDE (ESP-IDF 3.x), it worked as expected without generating only 0 as the output.
#️⃣ However, unfortunately, the Edge Impulse ESP-EYE firmware is not compatible with previous ESP-IDF versions.
#️⃣ Hence, I highly recommend utilizing the Edge Impulse Data Forwarder while collecting data with the built-in microphone (NAU8822) on the FireBeetle camera&audio board.
📡Analog Sensor
#️⃣ Plausibly, it does not require making any adjustments to collect analog sensor data with the Edge Impulse ESP-EYE firmware.
Modified Code Files
Download