MicroPython in the microcontroller¶

MicroPython is designed to run on a microcontroller. Programmers familiar with conventional computers may not be familiar with these hardware limitations. Especially RAM and non-volatile “disk”（flash memory）storage capacity is limited. This tutorial provides a method to make full use of limited resources. Since MicroPython runs on controllers based on various architectures, the method provided is universal：In some cases, Need to get detailed information from platform specific documentation.

Flash memory¶

On Pyboard, the simple way to solve the limited capacity is to install a micro SD card. But sometimes because the device does not have an SD card slot or for cost or power consumption reasons, This method is not feasible；Therefore, on-chip flash memory must be used. Firmware containing MicroPython subsystem is stored in onboard flash. Available capacity. Due to reasons related to the physical structure of flash memory, part of this capacity may not be accessible as a file system. Under these circumstances， This space can be used by incorporating the user module into the firmware version that is subsequently flashed into the device.

There are two ways to achieve this：Freeze module and freeze bytecode. Freeze module stores Python source code with firmware. Freezing bytecode uses a cross-compiler to convert source code to bytecode that is then stored with the firmware. The import statement can be used to access the module in both cases:

import mymodule

The process of generating frozen modules and bytecode depends on the platform；For instructions on building firmware, please refer to the README file in the relevant section of the source code tree。

In general, the steps are as follows:

Clone the MicroPython repository.
Obtain (platform specific) toolchain to build firmware.
Build cross compiler.
Place the module to be frozen in the specified directory (depends on freezing the module as source/bytecode).
Build firmware. Need specific instructions to build any type of freeze code-see platform documentation.
Flash the firmware to the device.

RAM¶

There are two stages to consider when reducing RAM usage：Compile and Execute. In addition to memory consumption, there is also a problem called heap fragmentation. In general, it is best to minimize repeated creation and damage of objects. The reason is described in the section related to heap（ heap）.

Compile stage¶

When importing the module, MicroPython compiles the code into bytecode, and then the MicroPython virtual machine (VM) executes the bytecode. The bytecode is stored in RAM. The compiler itself requires RAM, but it is only available after compilation.

If multiple modules have been imported, this will happen when there is not enough RAM to run the compiler. In this case, the import statement will raise a memory exception.

If the module instantiates a global object during import, RAM will be occupied during import and the compiler cannot use the RAM in subsequent imports. Usually, It is best to avoid code that runs on import； A better way is to have the initialization code run by the application after all modules are imported. This method maximizes the RAM available to the compiler.

If RAM is still not enough to compile all modules, one solution is to pre-compile the modules. MicroPython has a cross compiler， Python modules can be compiled into bytecode (see README in the mpy-cross directory). The extension of the generated bytecode file is .mpy. This file may be copied to the file system and imported in the usual way. Alternatively, some or all modules can be implemented as frozen bytecode： On most platforms, this saves more RAM because the bytecode runs directly from the flash memory and is not stored in the RAM.

Execution phase¶

There are many coding techniques that can reduce the use of RAM.

Constant

MicroPython provides the const keyword that can be used as follows:

from micropython import const
ROWS = const(33)
_COLS = const(0x10)
a = ROWS
b = _COLS

In both cases where a constant is assigned to a variable, the compiler will avoid encoding the lookup as a constant name by replacing its constant value. This saves bytecode, Which also saves RAM. But the ROWS value will occupy at least two machine words, which correspond to the key value and value in the globals dictionary. Must appear in the dictionary because another module may import or use it. This RAM can be saved by putting an underscore in front of the name (such as _COLS ). This RAM can be saved by using underscore as the name of in_COLS：This symbol is not visible outside the module, so it will not occupy RAM.

The parameter of const() can be any value calculated as an integer at compile time, such as 0x100 or 1 << 8 . It can even include other defined constant symbols, such as 1 << BIT 。

Constant Data Structure

If there is a large amount of constant data, and the platform supports execution from Flash, the RAM may be saved as follows. Data should be in Python module and frozen as bytecode. Data must be defined as bytes object. The compiler “know” that the bytes object is immutable, and ensures that the object remains in flash memory, not being copied into RAM. ustruct module assists in the conversion between types and other Python built-in types.

When considering the meaning of frozen bytecode, please note：In Python, strings, floating-point numbers, bytes, integers, and complex numbers are immutable. So these will be frozen into Flash. Therefore, in the following line

mystring = "The quick brown fox"

The actual string “The quick brown fox” will stay in Flash. At runtime, the string reference is assigned to the variable mystring . The quote occupies a machine word. In principle, long integers can be used to store constant data:

bar = 0xDEADBEEF0000DEADBEEF

As shown in the string example, at runtime, a reference to an arbitrary large integer is assigned to the variable bar. The reference occupies one machine byte.

It is expected that integer tuples can be used to store constant data with minimal RAM space. In the case of using the current compiler, this is invalid (code works, but RAM is not saved).

foo = (1, 2, 3, 4, 5, 6, 100000)

The runtime tuple will be located in RAM. This may be improved in the future.

No need to create objects

In many cases, objects may be created and destroyed unintentionally. This may reduce RAM availability due to fragmentation. The following section discusses such examples.

String Connection

Consider the following code snippet, whose purpose is to generate a constant string:

var = "foo" + "bar"
var1 = "foo" "bar"
var2 = """\
foo\
bar"""

Each code segment produces the same result, but the first code creates two unnecessary string objects at runtime and allocates more RAM for the connection before generating the third object. Other compilers perform more efficient linking at compile time, thereby reducing fragmentation.

In the case where a character string must be dynamically created before the string input stream (such as a file), if it is completed in a piecemeal manner, RAM will be saved. Create a substring (instead of creating a large string object) and enter it into the stream before processing the next string.

The best way to create dynamic strings is through the string format method:

var = "Temperature {:5.2f} Pressure {:06d}\n".format(temp, press)

Buffer Zone

When accessing devices such as UART, I2C, and SPI interfaces, use pre-allocated buffers to avoid unwanted object creation. Consider these two loops:

while True:
    var = spi.read(100)
    # process data

buf = bytearray(100)
while True:
    spi.readinto(buf)
    # process data in buf

The first loop creates a buffer on each pass, and the second loop reuses a pre-allocated buffer；This is both fast and effective in terms of memory fragmentation.

Byte less than integer

On most platforms, an integer consumes four bytes. Consider the call of these two functions foo() :

def foo(bar):
    for x in bar:
        print(x)
foo((1, 2, 0xff))
foo(b'\1\2\xff')

In the first call, create an integer tuple in RAM. The second call effectively creates the bytes object that consumes the least RAM. If the module is frozen as bytecode, the bytes object will remain in Flash.

String vs Bytes

Python3 introduced Unicode support, which introduced the difference between strings and byte arrays. As long as all characters in the string are ASCII (ie value <126), MicroPython ensures that Unicode strings do not take up extra space. If you need a full 8-bit value, you can use bytes and bytearray objects to ensure that no extra space is required. Please note: most string methods (e.g. str.strip()）also applies to bytes instances, so eliminating Unicode is not difficult.

s = 'the quick brown fox'   # A string instance
b = b'the quick brown fox'  # A bytes instance

Where you need to convert between strings and bytes, you can use the str.encode() and bytes.decode() method. Please note: strings and bytes are immutable. Any operation that takes this object as input and produces another object means that to produce the result, there is at least one RAM allocation. In the second line below, a new byte object is allocated. This will also happen if foo is a string.

foo = b'   empty whitespace'
foo = foo.lstrip()

Compiler Execution at Runtime

Python functions eval and exec call the compiler at runtime, which requires a lot of RAM. Please note: from micropython-lib pickle library uses exec . Object serialization using the ujson library may make more efficient use of RAM.

Store the String in Flash

Python strings are immutable, so they may be stored in read-only memory. The compiler can put the string defined in the Python code in Flash. As with freezing the module, you must have a copy of the source code tree on the PC and then use the toolchain to build the firmware. Even if the module has not been fully debugged, the program will still work as long as it can be imported and run.

After importing the module, execute:

micropython.qstr_info(1)

Then copy and paste all Q(xxx) lines into a text editor. Check and delete obviously invalid lines. Open the equivalent directory of the architecture that will be in stmhal (or in use) file qstrdefsport.h。Copy and paste the corrected line to the end of the file. Save the file, rebuild and flash the firmware. You can check the result by importing the module and sending it again:

micropython.qstr_info(1)

Q(xxx) Line should disappear.

Heap¶

When a running program instantiates an object, the necessary RAM will be allocated from a fixed-size pool, which is called the heap. When the object is out of range (In other words: no longer available for code), redundant objects are “garbage”. The “garbage collection” (GC) process reclaims this memory and returns it to the free heap. This process is automatic, but can be called directly by issuing gc.collect() .

The discussion in this regard is involved. For “quick fix”, the following content is regularly published:

gc.collect()
gc.threshold(gc.mem_free() // 4 + gc.mem_alloc())

Fragmentization¶

The program creates the object foo and then creates the object bar . Then foo is out of range, but bar remains. The RAM occupied by foo will be recycled by GC. However, if bar is assigned to a higher address, the RAM recovered from foo can only be used for objects no larger than foo . In complex or long-running programs, the heap can be fragmented：Despite the large amount of available RAM, there is not enough contiguous space to allocate specific objects, and the program fails due to memory errors.

The above technique aims to minimize this situation. When large permanent buffers or other objects are needed, it is better to execute the program、 Instantiate these buffers as early as possible before fragmentation. It can be further improved by monitoring the status of the reactor and controlling the GC. Summarized as follows.

Report¶

Many library functions can be used to report memory allocation and control GC. These can be found in the gc and micropython modules. The following example may be pasted in REPL (ctrl e enters paste mode, ctrl d runs it). Many library functions can be used to report memory allocation and control GC. These also exist in the gc and micropython modules. The following example may be pasted into the REPL（ ctrl e enters paste mode, ctrl d runs it）.

import gc
import micropython
gc.collect()
micropython.mem_info()
print('-----------------------------')
print('Initial free: {} allocated: {}'.format(gc.mem_free(), gc.mem_alloc()))
def func():
    a = bytearray(10000)
gc.collect()
print('Func definition: {} allocated: {}'.format(gc.mem_free(), gc.mem_alloc()))
func()
print('Func run free: {} allocated: {}'.format(gc.mem_free(), gc.mem_alloc()))
gc.collect()
print('Garbage collect free: {} allocated: {}'.format(gc.mem_free(), gc.mem_alloc()))
print('-----------------------------')
micropython.mem_info(1)

The method used above:

gc.collect() Mandatory garbage collection. See footnote.
micropython.mem_info() Print RAM utilization summary.
gc.mem_free() Returns the free heap size (in bytes).
gc.mem_alloc() Returns the number of bytes currently allocated.
micropython.mem_info(1) Print a table of heap utilization (see below for details).

The number generated depends on the platform, but you can see that the defined function uses a small amount of RAM in the form of bytecode issued by the compiler (the RAM used by the compiler has been recycled). Running the function uses more than 10KiB, but when returning, a is garbage, because it is out of range and cannot be referenced. THe last gc.collect() will restore memory.

The final output generated by micropython.mem_info(1) will be different, but may be explained as follows:

Each letter represents a memory block, each block is 16 bytes. Therefore, a row of the heap dump represents 0x400 bytes or 1KiB of RAM.

Control garbage collection¶

You can request GC at any time by issuing gc.collect() . Regular execution first helps prevent fragmentation, and secondly helps improve performance. GC may take a few milliseconds, and it takes less time when the workload is small (Only about 1ms on Pyboard). Explicit calls minimize latency， At the same time make sure that it appears under the conditions acceptable in the procedure.

Under the following circumstances, automatic GC will be activated. When the allocation fails, execute GC and retry the allocation. An exception will only be thrown if this allocation fails. Secondly, if the amount of available RAM is below the threshold, it will trigger automatic GC. This threshold can be adjusted as the execution progresses:

gc.collect()
gc.threshold(gc.mem_free() // 4 + gc.mem_alloc())

When more than 25% of the current free heap is occupied, GC will be triggered.

Generally, modules should use constructors or other initialization functions to instantiate data objects at runtime. This is because if this happens during initialization, when importing subsequent modules, the compiler may lack available RAM. If the module instantiates the data during import, then`gc.collect()` issued after the import will improve this problem.

String manipulation¶

MicroPython processes strings in an efficient manner and understands how they are processed. This helps design applications that run on microcontrollers. When the module is compiled, strings that appear multiple times are stored only once, and this process is called string resident.在In MicroPython, the resident string is called qstr . In the normally imported module, a single instance will be located in RAM, but as mentioned above, in the module frozen to bytecode, it will be located in Flash.

String comparison is also effectively performed using hashes (instead of performing character by character). Therefore, in terms of performance and RAM usage, the penalty for using strings instead of integers may be small-this may surprise C programmers.

Postscript¶

MicroPython transfers, returns and (by default) copies objects by reference. One reference occupies one machine word, so these processes are more efficient in terms of RAM usage and speed.

In the case of required variables (whose size is neither a byte nor a machine word), there will be a standard library that can help effectively store the variables and convert them. See array 、 ustruct and uctypes modules.

Footnote：gc.collect() return value¶

On Unix and Windows platforms, the gc.collect() method returns an integer, which represents the different memory reclaimed during recycling The number of areas (more precisely, the number of head blocks that become free blocks). For efficiency reasons, the baremetal port does not return this value.