makro

Search

SearchSearch

Firmware

Oct 14, 2023, 5 min read

Tasks

Requirements

  • [ ]

Intro

Intro here

Progress

Create PlatformIo project

Now that I have assembled a breadboard representation of the hardware using these modules, I can start to develop on, my first step in the firmware development process is to initialize a project.

(WRITE ABOUT IDES AND PLATFORMIO, CIRCUITPYTHON??)

Research Libraries

Initialize TFT-eSPI Library

Quick example test to verify the screen was connected correctly.

Connect to Network

Connecting to wifi on the ESP32 is a pretty simple task, the built in Arduino WiFi and HTTPclient libraries work great. Passing the device wifi credentials however is a more difficult task. During development I will just hardcode the SSID and password of my network, however to make this into a product, a WiFi Initialization Control Flow will be required, allowing users to connect the device to wifi from their phone or computer.

Show Image From Web

There are two potential methods of showing a chosen image on the display over the internet. The first way is to preemptively convert images to the RGB565 format, in this raw uncompressed format it is really easy for a small edge device such as an ESP32 to show on a TFT, however as it is uncompressed it is far more data to be sent over the internet than a JPG or PNG. The second method is to send an image over the internet in its compressed form and have the ESP32 decode it locally, this is more economical as the more functions I can perform locally onboard, the less operation costs incurred from running the server. Unfortunately the ESP32 does not have enough processing power to resize images, it can scale in multiples of 2 but that leaves for very few original resolutions to look good on the display. This leaves me needing a server to process the images regardless. However approaches will be made to minimise the number of reqests to my server, such as converting gifs into 1 large vertical image rather than returning many individual frames.

The image shown above was the first image I had shown on the screen that was downloaded from the internet. The image was actually taken directly from Spotify’s image source and is 2x the resolution of the screen, that is why its cropped to just the corner of the album cover.

Show GIF From Web

This is the first GIF file I have got playing on the screen! Using my backend, I split the gif into frames and then stacked them vertically. I then download that image to the ESP32 and change the vertical offset each loop, effectively showing each frame. Unfortunately the JPEG decoder I am currently using is not optimized for the hardware, and takes about 300ms to show each frame on the display, only resulting in a frame rate of ~3fps. I think a good target would be 15 fps meaning we need to show a frame in ~66ms, this does not include download time.

Optimize GIF Memory Handling

After changing the JPEG decoder from one found in the TFT_eSPI expansion library - TFT_eFEX, I was able to reduce the draw time of the image to 80 ms, which give a frame rate of 9.

Unfortunately further developments need to be made on the backend side before continuing to work on gifs, as the current processing time to split a gif into individual frames and stack them is lengthy.

A further complication is that while the ESP32 module has 2mb of PSRAM, only ~160kb of RAM are available to me at runtime. This means that displaying GIFs using a large stacked image will limit it to a limited number of frames or at a greatly reduced quality as the size of the image cannot (should not?) be larger than the available RAM.

To display GIFs in the future, I think the best method would be to split the gif into individual frames, encode them to base64 and send them all in 1 transaction as a JSON array of frames. The ESP32 will download each frame, base64 decode them and save to PSRAM/FLASH. Then it will load each frame from FLASH into RAM for the screen to show with DMA. While the image is being shown using DMA, the next frame will be loaded into the buffer.

Unfortunately before this solution is feasible I need to make huge optimizations on the backend as even reading the identification data of a 2-3mb GIF will take 20 seconds. I think that processing the image in the app using the phones/computers hardware may be a far more effective method.

Downloading each image independently is ok as long as the user only downloads them once, i.e. image is not downloaded on each rendering. Use LittleFS to save images?

WiFi Initialization Control Flow

Graph View