Latest commits (Ultibo core 1.3.245)

Releases, updates and announcements from the Ultibo team.
User avatar
Ultibo
Site Admin
Posts: 1279
Joined: Sat Dec 19, 2015 3:49 am
Location: Australia

Latest commits (Ultibo core 1.3.245)

Postby Ultibo » Fri Apr 14, 2017 5:13 am

The latest commits for Ultibo core are now available on GitHub. Changes include support for Raspberry Pi Zero W, additional QEMU drivers, TCP/IP and file system enhancements, bug fixes and more.

It has taken somewhat longer than we originally intended to release this commit, much of that time was spent in detailed analysis of a number of issues with TCP/IP and the filesystem. The results of this work have been extremely worthwhile with performance gains being achieved for both networking and filesystem and a range of issues, incompatibilities and missing features now resolved.

In spite of the time spent on these issues this commit also includes a substantial number of new features and additions.

List of changes:

  • Add detection of Raspberry Pi Zero W (WiFi and Bluetooth not yet supported)
  • Update detection of Raspberry Pi A+ to include the latest revision with 512MB
  • Add new keyboard repeat rate and mouse sample rate options to support additional devices
  • Change TConsoleShellThread to use ConsoleGetKey() instead of ConsoleReadChar()
  • Fix Rewrite() function to truncate an existing file if found
  • Add new PL031 RTC (Real Time Clock) driver for QEMU platform
  • Add new PL050 Keyboard and Mouse driver for QEMU platform
  • Allow Raspberry Pi to boot if less than 256MB available by adjusting memory allocations (eg if fixup.dat file is missing)
  • Fix incorrect free space value on FAT12/16 volume immediately after formatting
  • Fix incorrect free and available sectors value on a disk with a volume but no partitions
  • Enable idle thread to use WaitForEvent on each iteration to allow power saving and reduce heat (see below)
  • Change SCHEDULER_INTERRUPTS_PER_SECOND in QEMU to 1000 due to minimum timer resolution of 1ms on Windows
  • Fix Int64 alignment issue with TBCM283XMailboxTagGetBoardSerial due to compiler changes in 1.3.077
  • Change FILESYS_CACHE_PAGE to 32KB by default for improved performance
  • Implement priority Timer and Worker threads and add new priority flags to timer and worker requests
  • Fix GetTickCount/GetTickCount64 on QEMU platform incorrectly handling rollover and multiple threads
  • Add new SMSC LAN91C111 Network driver for QEMU platform
  • Implement receiver side of TCP Fast Retransmit algorithm
  • Fix deadlock issue when mounting extended partitions
  • Fix deadlocks in shell file system commands when adding or deleting partitions or mounting drives
  • Implement guard page on all thread stacks to allow prevention of stack overflows (see below)
  • Rework disk cache to use a non recursive model for reads and writes
  • Add new ThreadGetStackFree function to obtain the currently available stack space
  • Improve disk cache readahead algorithm for increased performance
  • Rework the TCP socket timer to handle higher throughput on both send and receive operations (see below)
  • Fine tune and enhance TCP handling of retransmit, window management, duplicate acknowledge and zero window
  • Adjust filesystem and network startup sequence to provide more reliable device discovery
  • Change PL011 UART (and BCM270X UART0) drivers to dramatically reduce the number of interrupts triggered during transmit and receive
  • Fix BCM270X UART0 driver to support receiving a single byte from the FIFO when the hardware does not trigger an interrupt

New QEMU platform drivers

This release includes new drivers for the QEMU platform that provide support for the Real Time Clock (RTC), Keyboard, Mouse and Network devices included in the Versatile PB emulation.

All of these drivers are automatically included and started when you include the QEMUVersatilePB unit in your project. The RTC, Keyboard and Mouse are enabled by default in QEMU and will work without any additional configuration.

In order to enable network support you need to add some parameters to the QEMU command line, this can easily be done by creating a QEMULauncher.ini file in the tools folder of your Ultibo installation (normally C:\Ultibo\Core\tools).

Add the following lines to the QEMULauncher.ini file to enable QEMU networking support:

Code: Select all

[QEMULauncher]
ExtraParams=-net user -net nic

If you would like to obtain a packet capture in Wireshark PCAP format you can add an extra parameter like this:

Code: Select all

[QEMULauncher]
ExtraParams=-net user -net nic -net dump

Additional options are available using different networking options such as TUN or TAP drivers depending on your host configuration, please see the QEMU documentation for more info.

QEMU support is now only lacking a disk driver in order to be fully functional, it is hoped that this will be available shortly.


Power management and heat reductions

Since the release of the Raspberry Pi 3B in early 2016 a number of people have commented on the temperature of the CPU when running Ultibo programs. We have noted a number of times that Ultibo currently lacks a CPU governor to reduce the CPU speed when idle and therefore runs constantly at whatever clock rate is set at startup.

Up until now we have recommended setting a lower clock rate in the config.txt file or fitting a heatsink in order to reduce the temperature, this release includes a small but significant change which can dramatically reduce the CPU temperature without changing the clock speed.

The design of Ultibo has always included an idle thread whose only task is to run when there is no other work to be done, it was always intended that the idle thread should use the power management features to place the CPU in a low power state when idle. After further work and experimentation we have finally been able to make this function correctly with both the Raspberry Pi 2 and 3 with the result that in normal operation the temperature of the CPU can be as much as 20 degrees celsius lower than with previous releases.

Exact results will vary and are dependent on many factors including ambient air temperature, enclosure, accessories and of course the current workload of the CPU. As an example, in our work environment today a Raspberry Pi 3B running at 1200MHz is able to maintain a temperature of around 50 degrees celsius without using a heatsink.


Prevention of stack overflow

While investigating a number of filesystem issues it was found that certain recursive functions could easily overflow the stack of the executing thread. Because Ultibo uses a single address space for all threads this overflow would ultimately overwrite memory belonging to some other purpose and cause either a crash or other unexplained behavior.

In order to limit the potential problems caused by inadvertently overflowing the stack during recursion we have implemented a guard page mechanism on each and every thread stack similar to the guard page provided at address 0x00000000 to prevent nil pointer reads and writes.

The mechanism is not currently very sophisticated (unlike the dynamic stack implemented in Windows for example) and will not normally be able to trigger an exception when a stack overflow occurs, however it will prevent further execution and stop random overwriting of data belonging to other items. This is a first step and will likely be refined further in order to provide a greater level of protection.


TCP throughput and performance improvements

As part of developing the SMSC LAN91C111 network driver for QEMU some oddities in the behavior of TCP were observed when presented with less than ideal circumstances. This led to a lengthy process of testing and investigating the TCP behavior under a range of conditions and the implementation of a significant number of fixes to address outstanding issues.

It isn't possible to detail every single change made but in general the further reworking of the TCP socket timer and implementation of correct handling for a number of TCP behaviors such as Fast Retransmit and Zero Window probe have resulted in a huge improvement in the ability of Ultibo applications to service network requests both inbound and outbound. The exact performance will always be dependent on the application being correctly written but as an example we are now able to demonstrate a small Ultibo application streaming multiple high definition video streams across the network simultaneously via HTTP.

In a connected Internet of Things environment it is critical to the success of Ultibo that we have solid and reliable networking support, these changes take Ultibo to a new level of capability and prepare the way for a range of exciting future developments.


For details of how to apply the latest source to your Ultibo core installation and rebuild your run time library see the wiki page Building from Source or watch the Building the RTL video on YouTube.
Ultibo.org | Make the future
https://ultibo.org
Gavinmc42
Posts: 852
Joined: Sun Jun 05, 2016 12:38 pm
Location: Brisbane, Australia

Re: Latest commits (Ultibo core 1.3.245)

Postby Gavinmc42 » Fri Apr 14, 2017 9:44 am

Had some build trouble in Linux under Wine, missing gnu utils?

Installed the old Windows exec version on a Windows box, copied new core master, RTL etc to the Ultibo folder and rebuilt.
Entire C:\Ultibo folder copied to USB stick and then copied to Linux box, hidden wine folder .wine/drive_c/Ultibo....
All working :P

QEMU will be interesting when disc stuff sorted, can have an QEMU Ultibo webserver updating kernel.bin in Ultibo net boxes?
You tried the Webserver example under QEMU?

What is parallel0 in the QEMU view?
Emulating GPIO on the PC printer port
pik33
Posts: 503
Joined: Fri Sep 30, 2016 6:30 pm
Location: Poland
Contact:

Re: Latest commits (Ultibo core 1.3.245)

Postby pik33 » Fri Apr 14, 2017 9:58 am

Instaled. Recompiled the player. Much cooler. Seems to be 10..15C cooler. <60C instead of >70 C (on active cooled, overclocked RPi) Slightly slower (screen time ~8500 us instead ~7800 us @ 1325/530/530 MHz)
develone
Posts: 152
Joined: Wed Dec 28, 2016 7:40 pm
Location: El Paso Tx USA

Re: Latest commits (Ultibo core 1.3.245)

Postby develone » Fri Apr 14, 2017 4:16 pm

Hello All,

04/14/17

updated github repositories Core, Examples, and Demo
commit 9c1838005eb2fbbca40377ffa61c7abef30339f6
Author: Ultibo <admin@ultibo.org>
Date: Fri Apr 14 12:17:26 2017 +1000

Latest commits (Ultibo core 1.3.245)

commit 3044a4264d09d8314198fdee1b8ef84180b97d96
Author: Ultibo <admin@ultibo.org>
Date: Fri Apr 14 16:02:37 2017 +1000

Updated examples for Ultibo core 1.3.245 release

Add QEMU support to more examples (Where appropriate)
Change Keyboard Input to use ConsoleGetKey function

These repositories were forks of the following repositories:
https://github.com/ultibohub/Core
https://github.com/ultibohub/Examples
https://github.com/ultibohub/Demo

On raspberrypi3

Created squashfs ultibo.img & Development.img raspberrypi3

Removed folders Development & ultibo

From repository https://github.com/develone/jpeg-2000-test executed the following commands:

jpeg-2000-test/bare-metal/build_ultibo_RPi/build_fpc_lazarus.sh

jpeg-2000-test/bare-metal/build_ultibo_RPi/build_ultibo_fpc.sh

jpeg-2000-test/bare-metal/build_ultibo_RPi/build_ultibo_rtl_armv7.sh

jpeg-2000-test/bare-metal/build_ultibo_RPi/build_ultibo_rtl_armv6.sh

cp jpeg-2000-test/bare-metal/build_ultibo_RPi/rpi*.cfg ultibo/core/fpc/bin

checking the latest version is on the RPi.

This shows 14 April 2017 ULTIBO_RELEASE_VERSION 1.3.2458


On a Ubuntu x86_64 running wine

vidal@sim2:~/.wine/drive_c/Ultibo/Core$ cp -R ~/wkg/Core/source/rtl/ultibo/ fpc/3.1.1/source/rtl/ultibo

Following the Tools/Build Ultibo RTL...


vidal@sim2:~$ cp -R ~/wkg/Examples/* ~/.wine/drive_c/Ultibo/Core/examples/

Cheers
User avatar
Ultibo
Site Admin
Posts: 1279
Joined: Sat Dec 19, 2015 3:49 am
Location: Australia

Re: Latest commits (Ultibo core 1.3.245)

Postby Ultibo » Sat Apr 15, 2017 12:17 am

Gavinmc42 wrote:Had some build trouble in Linux under Wine, missing gnu utils?

Sounds like the same issue as here which has already been resolved.
Ultibo.org | Make the future
https://ultibo.org
Gavinmc42
Posts: 852
Joined: Sun Jun 05, 2016 12:38 pm
Location: Brisbane, Australia

Re: Latest commits (Ultibo core 1.3.245)

Postby Gavinmc42 » Sat Apr 15, 2017 3:06 am

Must remember to take pill or remember to google myself.
Mid last year?
Rebuilt PC with new SDD and new MInt install since then.

Hmm now have USB with latest Ultibo on it, will it run from stick?
Ultibo as a portable application?

Portable app that runs on Windows/Linux/Pi2-3/Android?
QEMU Aarch64 running on portable app from tiny USB stick.
World's smallest development method?

Er guys you won't believe this. :o :o :o :o :o
Stuck the usb stick in and double clicked Lazarus.
Came up with message, clicked ok, have new E:/Ultibo drive.
Ultibo running from stick, probably works because I have Wine already installed on Mint.
Ran a QEMU example :P

Er Wine on Pi?
Crap, portable apps n tiny VM's, more stuff to learn.

"Write once - Compile anywhere" could this be true?
This Laz/FPC stuff keeps impressing me with it's cleanness and usability.

Anyway, now have a way to carry latest Ultibo with me anywhere.
Need nice keyring USB stick in metal case? Gold plated and engraved:lol:

:idea: Garry, marketing idea - Orange usb sticks with Ultibo on them?
Wrist band USB sticks? Watch band with USB plus micro sdcard carriers?

Wonder if Ultibo will run from phone used as drive?
Er, not on my dumb freebie Telstra gave me, the mass storage drive keeps turning off.
But Lazarus did boot from this phone as j:/ drive.
Android phone as mass storage might be better?
pik33
Posts: 503
Joined: Fri Sep 30, 2016 6:30 pm
Location: Poland
Contact:

Re: Latest commits (Ultibo core 1.3.245)

Postby pik33 » Sat Apr 15, 2017 9:45 am

Some more tests:

Another (newer) RPi3. Extremally overclocked - 1375/550/550 MHz. It can be used as a retromachine player without fan, heatsink only - ~70C; 50C with fan. This RPi is about 20C cooler with new Ultibo. And this RPi is not slower (~7400 us screen time, which didn't change) - I don't know why there are such time differences. This is the same build of the player.

MicroDos - I complained about slow file copying: this is no more a problem, the copy speed is 5..10 times faster. Then the "fake raster interrupts" effect while copying the file doesn't work anymore, I have to check why. This effect is no more needed as the copying is now fast.

Good :)

I have to check if NTFS partitions work now without hanging the system.
Edit: checked - it works now :)
Edit2: there is a bug in ntfs - I will write about this in another topic.

Is the kbd/mouse bug fixed?
mark
Posts: 624
Joined: Mon Oct 03, 2016 2:12 am
Location: Indianapolis, US

Re: Latest commits (Ultibo core 1.3.245)

Postby mark » Mon Apr 17, 2017 7:40 am

Ultibo wrote:The latest commits for Ultibo core are now available on GitHub.
    ...
  • Add new SMSC LAN91C111 Network driver for QEMU platform
    ...
Garry, have you tested tftp with qemu networking? If not, I will do so using an ultibo ram disk. I then would like to have ultibo restart after overlaying itself with the new image. I intend to implement bspatch (https://github.com/mendsley/bsdiff) as an ultibo unit so that only kernel diffs need to be sent.

Regards, Mark
User avatar
Ultibo
Site Admin
Posts: 1279
Joined: Sat Dec 19, 2015 3:49 am
Location: Australia

Re: Latest commits (Ultibo core 1.3.245)

Postby Ultibo » Mon Apr 17, 2017 10:12 am

mark wrote:have you tested tftp with qemu networking?

Not yet, started to explore getting the other networking options like TUN/TAP etc working but got slightly sidetracked trying to work out the exact details of the mysterious Windows TAP driver from OpenVPN.org. So far it seems to work in certain modes but not others (not an Ultibo specific issue).

The ShellUpdate module works in QEMU and fetches a new image via HTTP, also done some initial testing with WebStatus but not much else yet. Will continue to do more over the next few weeks to try out what is possible.

mark wrote:If not, I will do so using an ultibo ram disk. I then would like to have ultibo restart after overlaying itself with the new image.

RAM disk works fine, the Shutdown and Restart functions are not yet connected but there is such functionality available in QEMU, just need to manipulate the relevant registers.

mark wrote:I intend to implement bspatch (https://github.com/mendsley/bsdiff) as an ultibo unit so that only kernel diffs need to be sent.

Looks like it could be very useful.
Ultibo.org | Make the future
https://ultibo.org
mark
Posts: 624
Joined: Mon Oct 03, 2016 2:12 am
Location: Indianapolis, US

Re: Latest commits (Ultibo core 1.3.245)

Postby mark » Wed Apr 26, 2017 4:25 am

Ultibo wrote:... the Shutdown and Restart functions are not yet connected but there is such functionality available in QEMU, just need to manipulate the relevant registers.
...
Garry, can you point me to these registers? Thanks, Mark.

Return to “Ultibo”

Who is online

Users browsing this forum: No registered users and 1 guest