It has taken somewhat longer than we originally intended to release this commit, much of that time was spent in detailed analysis of a number of issues with TCP/IP and the filesystem. The results of this work have been extremely worthwhile with performance gains being achieved for both networking and filesystem and a range of issues, incompatibilities and missing features now resolved.
In spite of the time spent on these issues this commit also includes a substantial number of new features and additions.
List of changes:
- Add detection of Raspberry Pi Zero W (WiFi and Bluetooth not yet supported)
- Update detection of Raspberry Pi A+ to include the latest revision with 512MB
- Add new keyboard repeat rate and mouse sample rate options to support additional devices
- Change TConsoleShellThread to use ConsoleGetKey() instead of ConsoleReadChar()
- Fix Rewrite() function to truncate an existing file if found
- Add new PL031 RTC (Real Time Clock) driver for QEMU platform
- Add new PL050 Keyboard and Mouse driver for QEMU platform
- Allow Raspberry Pi to boot if less than 256MB available by adjusting memory allocations (eg if fixup.dat file is missing)
- Fix incorrect free space value on FAT12/16 volume immediately after formatting
- Fix incorrect free and available sectors value on a disk with a volume but no partitions
- Enable idle thread to use WaitForEvent on each iteration to allow power saving and reduce heat (see below)
- Change SCHEDULER_INTERRUPTS_PER_SECOND in QEMU to 1000 due to minimum timer resolution of 1ms on Windows
- Fix Int64 alignment issue with TBCM283XMailboxTagGetBoardSerial due to compiler changes in 1.3.077
- Change FILESYS_CACHE_PAGE to 32KB by default for improved performance
- Implement priority Timer and Worker threads and add new priority flags to timer and worker requests
- Fix GetTickCount/GetTickCount64 on QEMU platform incorrectly handling rollover and multiple threads
- Add new SMSC LAN91C111 Network driver for QEMU platform
- Implement receiver side of TCP Fast Retransmit algorithm
- Fix deadlock issue when mounting extended partitions
- Fix deadlocks in shell file system commands when adding or deleting partitions or mounting drives
- Implement guard page on all thread stacks to allow prevention of stack overflows (see below)
- Rework disk cache to use a non recursive model for reads and writes
- Add new ThreadGetStackFree function to obtain the currently available stack space
- Improve disk cache readahead algorithm for increased performance
- Rework the TCP socket timer to handle higher throughput on both send and receive operations (see below)
- Fine tune and enhance TCP handling of retransmit, window management, duplicate acknowledge and zero window
- Adjust filesystem and network startup sequence to provide more reliable device discovery
- Change PL011 UART (and BCM270X UART0) drivers to dramatically reduce the number of interrupts triggered during transmit and receive
- Fix BCM270X UART0 driver to support receiving a single byte from the FIFO when the hardware does not trigger an interrupt
New QEMU platform drivers
This release includes new drivers for the QEMU platform that provide support for the Real Time Clock (RTC), Keyboard, Mouse and Network devices included in the Versatile PB emulation.
All of these drivers are automatically included and started when you include the QEMUVersatilePB unit in your project. The RTC, Keyboard and Mouse are enabled by default in QEMU and will work without any additional configuration.
In order to enable network support you need to add some parameters to the QEMU command line, this can easily be done by creating a QEMULauncher.ini file in the tools folder of your Ultibo installation (normally C:\Ultibo\Core\tools).
Add the following lines to the QEMULauncher.ini file to enable QEMU networking support:
Code: Select all
ExtraParams=-net user -net nic
If you would like to obtain a packet capture in Wireshark PCAP format you can add an extra parameter like this:
Code: Select all
ExtraParams=-net user -net nic -net dump
Additional options are available using different networking options such as TUN or TAP drivers depending on your host configuration, please see the QEMU documentation for more info.
QEMU support is now only lacking a disk driver in order to be fully functional, it is hoped that this will be available shortly.
Power management and heat reductions
Since the release of the Raspberry Pi 3B in early 2016 a number of people have commented on the temperature of the CPU when running Ultibo programs. We have noted a number of times that Ultibo currently lacks a CPU governor to reduce the CPU speed when idle and therefore runs constantly at whatever clock rate is set at startup.
Up until now we have recommended setting a lower clock rate in the config.txt file or fitting a heatsink in order to reduce the temperature, this release includes a small but significant change which can dramatically reduce the CPU temperature without changing the clock speed.
The design of Ultibo has always included an idle thread whose only task is to run when there is no other work to be done, it was always intended that the idle thread should use the power management features to place the CPU in a low power state when idle. After further work and experimentation we have finally been able to make this function correctly with both the Raspberry Pi 2 and 3 with the result that in normal operation the temperature of the CPU can be as much as 20 degrees celsius lower than with previous releases.
Exact results will vary and are dependent on many factors including ambient air temperature, enclosure, accessories and of course the current workload of the CPU. As an example, in our work environment today a Raspberry Pi 3B running at 1200MHz is able to maintain a temperature of around 50 degrees celsius without using a heatsink.
Prevention of stack overflow
While investigating a number of filesystem issues it was found that certain recursive functions could easily overflow the stack of the executing thread. Because Ultibo uses a single address space for all threads this overflow would ultimately overwrite memory belonging to some other purpose and cause either a crash or other unexplained behavior.
In order to limit the potential problems caused by inadvertently overflowing the stack during recursion we have implemented a guard page mechanism on each and every thread stack similar to the guard page provided at address 0x00000000 to prevent nil pointer reads and writes.
The mechanism is not currently very sophisticated (unlike the dynamic stack implemented in Windows for example) and will not normally be able to trigger an exception when a stack overflow occurs, however it will prevent further execution and stop random overwriting of data belonging to other items. This is a first step and will likely be refined further in order to provide a greater level of protection.
TCP throughput and performance improvements
As part of developing the SMSC LAN91C111 network driver for QEMU some oddities in the behavior of TCP were observed when presented with less than ideal circumstances. This led to a lengthy process of testing and investigating the TCP behavior under a range of conditions and the implementation of a significant number of fixes to address outstanding issues.
It isn't possible to detail every single change made but in general the further reworking of the TCP socket timer and implementation of correct handling for a number of TCP behaviors such as Fast Retransmit and Zero Window probe have resulted in a huge improvement in the ability of Ultibo applications to service network requests both inbound and outbound. The exact performance will always be dependent on the application being correctly written but as an example we are now able to demonstrate a small Ultibo application streaming multiple high definition video streams across the network simultaneously via HTTP.
In a connected Internet of Things environment it is critical to the success of Ultibo that we have solid and reliable networking support, these changes take Ultibo to a new level of capability and prepare the way for a range of exciting future developments.
For details of how to apply the latest source to your Ultibo core installation and rebuild your run time library see the wiki page Building from Source or watch the Building the RTL video on YouTube.