Grenoble INP ENSIMAGcole Nationale Suprieure dInformatique et de Mathmatiques Appliques

Internship report(Second Year Embedded Software Engineer Internship)

At Oxbern SAS

Development and implementation of theCrystalCubes firmware, a 3D LED Display

El Khadiri Yassine2A Option SLE

June 22 2015 September 18 2015

2Pierre Schefler Olivier AlphandOxbern Ensimag3 parvis Louis Nel 681 Rue de la Passerelle38016 Grenoble Cedex 1 38400 Saint-Martin-dHres

CONTENTS 3

Contents1 Introduction 5

2 General Overview 52.1 Oxbern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2 Crystal Cube . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 Mission Objectives 73.1 Problems with the current prototype . . . . . . . . . . . . . . . . . . . . . 73.2 Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.3 Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

4 Firmware Architecture 94.1 High level overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94.2 The STM32F4 Microprocessor . . . . . . . . . . . . . . . . . . . . . . . . . 10

4.2.1 System Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.2.2 System Clock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

4.3 Peripherals and Application Modules . . . . . . . . . . . . . . . . . . . . . 124.3.1 LEDs/ADC/PWM . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.3.2 SDIO/FATFS/DataBase . . . . . . . . . . . . . . . . . . . . . . . . 134.3.3 LTDC/I2C/STemWin/GUI . . . . . . . . . . . . . . . . . . . . . . 144.3.4 USB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.3.5 Lua . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4.4 Crystallography Application . . . . . . . . . . . . . . . . . . . . . . . . . . 15

5 Results 16

6 Hurdles and personal experience 16

7 Conclusion 17

A The STMs Peripheral Pinout Configuration 18

B Clock Configuration 19

C Simple Database file example 20

D GUI Examples 22

4 CONTENTS

Abstract

The projects objective is to develop a firmware for a 3D LED display.The firmware has to be fully modular, drive the LEDs, allow for brightness control

through a potentiometer, control a touch screen, load files from an SD card andcommunicate with a computer through a USB connection and run Lua scripts.

We chose to target STs STM32F429 micro-controller as it provides all of thenecessary peripherals. We first tested and validated the usage of each one of them onSTs STM32F429 Discovery Board, then we proceeded onto building the full firmwarethat runs on our custom PCB.

We were able to lay down all of the necessary groundwork of the firmware but somemodules like the Lua interpreter and the USB connection are still work in progress.

51 Introduction

This internships objective is to develop and implement a firmware from scratch for theCrystalCube, a 9x9x9 LED display. A firmware is a piece of software that provides lowlevel control over hardware peripherals on a micro-controller. This particular relationshipwith the underlining hardware makes it so that, in general, firmwares are written from theground up. The embedded software engineer has to come up with a suitable architecturefor the system at hand and squeeze the most out of the given hardware by either usingexisting drivers, modifying and adapting them for usage on the current system, or evenwriting them from scratch if necessary.

2 General Overview

2.1 Oxbern

Oxbern is a startup created on June 2015 by Pierre Schefler as part of his entrepreneurialproject. The idea of making a 3D LED display was born during his first years groupproject (see figure 1) where they decided to make it as a multidisciplinary exercise. It wasntsupposed to go any further but then came a crystallographic researcher from the Louis Nelinstitute showing interest in using the cube as a way the easily represent crystallographicpatterns.

Figure 1: First year (and the first) prototype

6 2 GENERAL OVERVIEW

2.2 Crystal Cube

The Crystal Cube (see figure 2) is Oxberns main product. Its a 3D 9x9x9 LED displayfor visualizing crystallographic patterns. Each LED can be toggled separately. Users caninteract with the Cube through 10 buttons and a 20 by 2 Serial LCD screen. For advancedusage, a simplistic API can be used to control the Cube through a USB connection with aPersonal Computer. To add new patterns, users must reload the firmware running on oneof the two arduinos necessary for the Cubes operation.

Figure 2: Second year prototype running on 2 Arduinos

73 Mission Objectives

3.1 Problems with the current prototype

The prototypes first iterations were based on Arduino boards (see figures 1 and 2). TwoArduino boards were necessary : one to drive the LEDs and an other one to manage theHMI 1 and handle the internally stored patterns.

From a production stand point this wasnt a viable solution as it not only was too cum-bersome and clumsy but also lacked scalability because of the Arduinos lack of memory,processing power and advanced peripherals. Hence, it was decided that the final versionof the product includes the possibility of controlling the LEDs brightness through a knobat the back of the Cube, a touch Screen for a more user friendly interface instead of theSerial LCD screen and the buttons, the possibility of launching Lua scripts directly on thecube and a more complete USB API.

3.2 Solution

Thus came the prototypes next iteration running on one of STs prototyping boards : theSTM32F429 Discovery Board [6]. As mentioned above, this internships main objective isto develop a firmware to drive the Cubes LEDs, control the touch display, interact withthe SD card and handle a USB connection with the provided software.

The firmware has to run on STs[8] STM32F429 Microcontroller [11], as it provides allof the necessary hardware peripherals needed for this project. This means that a certainfamiliarity with this chip and its internals, without being a thorough knowledge of all itsspecifics, had to be acquired throughout the firmwares development.

More specifically, acquaintance with the following peripherals had to be gained :

SPI (Serial Peripheral Interface) for driving the LEDs

TIM (Timer) and ADC (Analog to Digital Converter) for the LEDs brightness levelhandling

SDIO (Secure Digital Input Output) for SD card management

LTDC (LCD-TFT Driver Controller) and I2C (Inter-Integrated Circuit) for drivingthe touch screen

SDRAM for managing system memory and screen buffers

USB device stack for handling the USB communication with a computer

For every peripheral, ST provides some general purpose drivers[5] and example code toquickly get started.

1Human-machine interface

8 3 MISSION OBJECTIVES

3.3 Guidelines

I was asked to follow two specific guidelines during this project :

Test and develop code for each peripheral in a separate project before integratingthem all into a single code base.

Make the firmware as modular as possible with the ability to easily replace and/orremove modules with as little impact as possible on the rest of them and so thathardware modifications wont have as much impact on the application layer.

94 Firmware Architecture

4.1 High level overview

The firmwares architecture can be represented by a top down diagram (see figure 3). Eachperipheral can work relatively independently from the others (ex. touch screen updatingand the LEDs brightness adjustment). Hence, it was decided to use FreeRTOS as aminimalistic scheduler to easily implement the different modules that have to seem likerunning in parallel, as required by the new specifications of the product.

Figure 3: A general overview of the firmwares architecture

This block diagram represents the natural cutout that stands out in the firmwaresarchitecture between the different peripherals and the overlaying middleware. All of thefirmwares modules are built on top of the HAL2 Libraries. Then come the middlewareson top of that :

FreeRTOS : Handles scheduling between tasks FATFS : Handles the file system on the sd card STemWin : Handles widget and graphics and touch events on the touch screen Lua Interpreter : Handles Lua Scripts and provides a Lua environment USB Core and CDC stack : Handles the USB connection with the provided computer

software2Hardware Abstraction Layer

10 4 FIRMWARE ARCHITECTURE

As stated before, each stacked blocks with the same color on the figure 3 has beenvalidated in its own separate project3 then put back to back with the others in the formof a FreeRTOS task.

Figure 4: Code base repository architecture

Indeed, this directory structure[12] follows the guidelines about having a flexible moduleimplementation as each directory can technically be its own project with minimal changesto the overall firmware. Also, the use of HAL drivers makes it easy to change the micro-controller if needed (within the same family).

4.2 The STM32F4 Microprocessor

The STM32F429 is an upper mid-tier ARM processor that can run at 180 MHz with 192KB of SRAM and 1024 KB of FLASH memory while providing various functionality thanksto its peripherals (see Appendix A). In comparison, the Arduino Mega runs at 16 MHz with8 KB of SRAM and 128 KB of FLASH memory. As we are targeting an ARM processor

3Exception made for the Lua interpreter and the USB updating system which are still WIP at the timeof writing this paper

4.2 The STM32F4 Microprocessor 11

we used the ARM GCC Cross Compilation Toolchain on GNU/Linux and GNU/Linuxcompatible flashing tools [14].

4.2.1 System Memory

The LCD-TFT screen has a resolution of 800 by 480 pixels. For an ARGB88884 representa-tion, a buffer of 1500 KB is needed. Hence, it was mandatory to add an external SDRAM.We chose an SDRAM of 8 MB to be able to also load and store heavier databases fromthe SD Card. And to simplify memory management we decided to move the heap onto theSDRAM, leaving the stack more space where to grow if needed as show in figure 5.

Figure 5: Chip memory layout

Heap allocation is handled by Newlib5[4]s malloc through our implementation of thesbrk function.6

4.2.2 System Clock

The system clock tree on the STM32F429 has to be fine-tuned according to the peripheralsthat are to be activated. STs STM32CubeMx[10] comes with a clock configuration tool

48 bits per color and 8 bit for the alpha channel5Newlib is a C library intended for use on embedded systems.6More info about using newlib at https://balau82.wordpress.com/2010/12/16/

using-newlib-in-arm-bare-metal-programs/

12 4 FIRMWARE ARCHITECTURE

(See Appendix B) that helps with that. The biggest constraint in our system is the USBFull Speed peripheral that has to run at 48MHz, bringing down the system clock from180MHz to 168MHz.

4.3 Peripherals and Application Modules

4.3.1 LEDs/ADC/PWM

The Crystal Cubes LEDs are driven using 10 daisy chained shift registers[7]. One to selecta plane and the 9 others control the 81 leds on that plane. For example, to draw a squarein the second plane, we have to populate the sink drivers with these values :

1 uint16 bu f f e r = {2 0b0000000000000000 , // 9 th column3 0b0000000000000000 , // 8 th column4 0b0000000000000000 , // 7 th column5 0b0000000001111100 , // 6 th column6 0b0000000001000100 , // 5 th column7 0b0000000001111100 , // 4 th column8 0b0000000000000000 , // 3rd column9 0b0000000000000000 , // 2nd column10 0b0000000000000000 , // 1 s t column11 0b0000000000000010 // p lane s e l e c t i o n12 // ^^^^^^^13 // unused b i t s14 } ;

This is done by sending them over the chips SPI7 peripheral, at a rate of 5.25 MBits/s.

The LEDs brightness is controlled with a potentiometer sampled by the ADC8 periph-eral that outputs a 12 bit digit which is then turned into a percentage and used as thePWMs duty cycle generated via the STMs Timer peripheral as shown in figure 6.

7Serial Peripheral Interface : A simple synchronous bus interface used for short distance communication.8Analog to Digital Converter

4.3 Peripherals and Application Modules 13

Figure 6: PWM output for LED brightness control

4.3.2 SDIO/FATFS/DataBase

Data storage is based on FAT formated SD cards as the chip embeds an SDIO9 interfaceand ST provides a modified FATFS[1] middleware that goes with it (see figure 7).

The database files that contain the crystallographic patterns are based on a JSONformat[3]. A lightweight C json parser[15] is used to parse through the database files andbuild the necessary structures for the Crystallographic application. (see Appendix C foran example of a simple database file)

9Secure Digital Input Output

14 4 FIRMWARE ARCHITECTURE

Figure 7: FatFs Layered Architecture

4.3.3 LTDC/I2C/STemWin/GUI

The touch screen is a Newhaven Display 7" capacitive panel[2] with a resolution of 800 by480 at a 24 bits color depth. It has a built-in driver for the touch panel and no controller.This is where the on board LTDC module comes in handy as it only needs to be configuredwith the correct parameters (buffer location, screen resolution, pixel format...) to startdisplaying the buffers content onto the screen.

The touch panel has a built-in micro-controller that uses an I2C interface to send touchrelated information (see page 6 of [2]).

For the graphical interface10 (see Appendix D for an example of the gui), ST providesa modified version of emWin[13] called STemWin to use with their micro-controllers. Thismiddleware is LCD controller independent as it only works on the video memory bufferthe user provides. Pointer input devices and touch events are handled through callbackroutines called by the user 11.

4.3.4 USB

ST provides a USB core and USB CDC stack middleware[9] for handling serial communi-cation with a PC. For correct operation, the USB peripheral clock needs to be at 48MHzand enough heap space (at least 1024 bytes) with a working allocator should be available.

10WIP at the time of writing this paper11https://www.segger.com/emwin-pointer-input-devices.html

4.4 Crystallography Application 15

Each packet received over the USB connection is sent to a state machine for decoding andexecuting the encapsulated command according to the API.12

4.3.5 Lua

Lua scripts are handled by the Lua interpreter with specific native binded functions tocontrol the Cubes peripherals.

4.4 Crystallography Application

The Crystallography application follows a state machine (see figure 8). When turning thecube on, if an SD card is present, the user is prompted to select a DataBase file or a Luascript to load. If the SD Card is not detected an Error Message is displayed until the userinserts and sd card in the slot.

If the user selects a Lua script, the Lua interpreter is launched with the given script.If the user selects a DataBase file, the file is parsed and if it is well formatted, the useris prompted to select a pattern to display. If there was an error parsing the database filethen an error message is displayed and the user is prompted to select another file.

User interface examples are provided at Appendix D.

Figure 8: Crystallography Applications state machine

12The API is still work in progress

16 6 HURDLES AND PERSONAL EXPERIENCE

5 ResultsThe firmware is still work in progress as we are still tweaking PCB Layouts and solvingissues with some peripherals. At the time of editing this paper basic firmware functionalityis in place. Some issues with STs SDIO drivers reentrancy for use in a FreeRTOS awareenvironment still need to be solved, the USB API still needs to be implemented on the PCsoftware side and the Lua interpreter isnt part of the firmware yet.

6 Hurdles and personal experienceThis internship was quite heavy with new experiences as we practically built a hole com-puter system from the grounds up. It feels fulfilling despite having spent countless oursdebugging and fighting electronic failures. As I spent most of my time in the gnu debuggerI discovered the use of python scripts to custom format typedefs which Im sure will comein handy in future projects. While were on the topic of python scripting I also had theoccasion to sharpen my python skills by writing scripts that helped my mentor handle hiselectronic components database and automate to a certain degree the process of orderingthe necessary components in a given design.

One of the biggest hurdles I had - and Im still dealing with - is STs HAL driversimplementation. One of the key elements thats supposed to make the firmware work theway it was designed is having a real time kernel handle the different processes, in our caseFreeRTOS. The problem is that STs HAL drivers arent RTOS friendly as they are notalways reentrant. For simple peripherals like the SPI or the I2C buses, it doesnt seem tobe a problem, however, it is with time constrained peripherals such as the SDIO and theUSB. After spending quite some time searching the internet and reading forums frequentedby professional embedded systems engineers, it seems that the HAL drivers cause a lot ofproblems to many people and the general wide spread solution is to extensively modify oreven rewrite them13.

1 #i f (USE_RTOS == 1)2 / Reserved f o r f u t u r e use /3 #er r o r USE_RTOS should be 0 in the cur rent HAL r e l e a s e4 #else

Figure 9: Code found in the file stm32f4xx_hal_def.h of the current HAL drivers release

13http://www.eevblog.com/forum/microcontrollers/sts-%28stm32cube%29-software-ecosystem-is-terrible-how-can-we-fix-it/

17

7 ConclusionEven though some modules are still not quite finished yet, we laid all the necessary ground-work for a fully modular and easily customizable firmware. This will insure the firmwarescompatibility with future updates of the underlining hardware to some extent and allowfor a more straightforward code maintenance.

18 A THE STMS PERIPHERAL PINOUT CONFIGURATION

A The STMs Peripheral Pinout Configuration

19

B Clock Configuration

20 C SIMPLE DATABASE FILE EXAMPLE

C Simple Database file example

1 {2 " ch i l d r en " : [3 {4 " d e s c r i p t i o n " : "" ,5 " image" : "" ,6 "name" : " cubique cent r " ,7 " opt ions " : [8 ] ,9 " po in t s " : [10 [ 8 , 8 , 0 ] ,11 [ 8 , 6 , 0 ] ,12 [ 6 , 6 , 0 ] ,13 [ 6 , 8 , 0 ] ,14 [ 7 , 7 , 1 ] ,15 [ 6 , 8 , 2 ] ,16 [ 6 , 6 , 2 ] ,17 [ 8 , 8 , 2 ] ,18 [ 8 , 6 , 2 ]19 ] ,20 " type" : " pattern "21 } ,22 {23 " ch i l d r en " : [24 {25 " d e s c r i p t i o n " : "" ,26 " image" : "" ,27 "name" : " cubique f a c e s c en t r e s " ,28 " opt ions " : [29 ] ,30 " po in t s " : [31 [ 8 , 8 , 0 ] ,32 [ 8 , 6 , 0 ] ,33 [ 6 , 6 , 0 ] ,34 [ 6 , 8 , 0 ] ,35 [ 8 , 7 , 1 ] ,36 [ 6 , 7 , 1 ] ,37 [ 7 , 6 , 1 ] ,38 [ 7 , 8 , 1 ] ,39 [ 7 , 7 , 0 ] ,40 [ 8 , 8 , 2 ] ,41 [ 8 , 6 , 2 ] ,42 [ 6 , 6 , 2 ] ,43 [ 6 , 8 , 2 ] ,44 [ 7 , 7 , 2 ]45 ] ,46 " type" : " pattern "47 }48 ] ,49 "name" : " gourpe1" ,

21

50 " type" : "group"51 }52 ] ,53 "name" : "Newdatabase " ,54 " type" : "group" ,55 " ve r s i on " : "1"56 }

22 D GUI EXAMPLES

D GUI Examples

SD card content Click an item to load it.

Some folder

Some database

Some other folder

Some script

Path to groupPattern name1/12

Lorem ipsum dolor sit amet, consectetur adipisc-ing elit. Vivamus et varius diam. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos. Aenean nulla nisi, auctor et pharetra non, suscipit at erat. Suspendisse poten-ti. Vestibulum id orci quis mi pretium interdum. Sed vehicula nec arcu non suscipit. Quisque at arcu ut nisl molestie efficitur. Morbi pulvinar urna in semper feu-giat. Etiam sit amet malesuada est. In hac habitasse platea dictumst. Donec tempus vehicula accumsan. In vitae leo tellus.

REFERENCES 23

References[1] ChanN. Fatfs. http://elm-chan.org/fsw/ff/00index_e.html.

[2] Newhaven Display. Nhd-7.0-800480ef-atxl-ctp. http://www.newhavendisplay.com/specs/NHD-7.0-800480EF-ATXL-CTP.pdf.

[3] ECMA. Json. http://www.json.org/.

[4] Jeff Johnston Maintainers : Corinna Vinschen. Newlib. https://sourceware.org/newlib/.

[5] ST Microelectronics. Description of stm32f4xx hal drivers. http://www.st.com/st-web-ui/static/active/jp/resource/technical/document/user_manual/DM00105879.pdf.

[6] ST Microelectronics. Discovery kit for stm32f429/439 lines. http://www.st.com/st-web-ui/static/active/en/resource/technical/document/user_manual/DM00093903.pdf.

[7] ST Microelectronics. Low voltage 16-bit constant current led sink driver withlow power saving. http://www.st.com/web/en/resource/technical/document/datasheet/CD00126635.pdf.

[8] ST Microelectronics. Semiconductor manufacturing company. http://www.st.com/web/en/home.html.

[9] ST Microelectronics. Stm32cube usb device library. http://www.st.com/st-web-ui/static/active/en/resource/technical/document/user_manual/DM00108129.pdf.

[10] ST Microelectronics. Stm32cubemx. http://www.st.com/web/catalog/tools/FM147/CL1794/SC961/SS1533/PF259242?sc=stm32cube.

[11] ST Microelectronics. Stm32f427xx stm32f429xx. http://www.st.com/web/en/resource/technical/document/datasheet/DM00071990.pdf.

[12] Oxbern. Ccube firmware. https://github.com/ppff/CCube_Firmware.

[13] SEGGER. emwin. https://www.segger.com/emwin.html.

[14] texane. stlink. https://github.com/texane/stlink.

[15] udp. json-parser. https://github.com/udp/json-parser.

IntroductionGeneral OverviewOxbernCrystal Cube

Mission ObjectivesProblems with the current prototypeSolutionGuidelines

Firmware ArchitectureHigh level overviewThe STM32F4 MicroprocessorSystem MemorySystem Clock

Peripherals and Application ModulesLEDs/ADC/PWMSDIO/FATFS/DataBaseLTDC/I2C/STemWin/GUIUSBLua

Crystallography Application

ResultsHurdles and personal experienceConclusionThe STM's Peripheral Pinout ConfigurationClock ConfigurationSimple Database file exampleGUI Examples