Usable operatingsystem with ultibo, threading + pe format?

The place to share and discuss your Ultibo projects.
wotanica
Posts: 30
Joined: Wed Feb 03, 2016 11:02 pm

Usable operatingsystem with ultibo, threading + pe format?

Postby wotanica » Sat Apr 14, 2018 1:25 am

First, I love this project more and more - completely blown away by the sheer quality of Ultibo!
I was initially sceptical due to the lack of GPU support - but seeing how you have not only delivered this, but the way you did it - is just awesome!

My question has to do with making a suitable operating-system for emulation, but also with the ultimate goal of being an OS in itself.
Obviously writing an OS is not something you slap-dash together over the weekend, but with time and persistence I think we can get a lot of work done.

My first question has to do with the threading model.
I am porting UAE (the amiga emulator) to pascal and im doing a bit of re-decorating from the old way this was implemented.
The idea is to allocate a thread for each of the central components of the Amiga (cpu, graphics, sound and IO) and use a message system to delegate signals between them.
Is there something like this (message queues etc) that are thread safe already in ultibo? Just so i dont re-invent the wheel.

My second question is more elaborate. Emulation is one thing, but what about a native API for running arm executables?
I realize that this is not an easy task since the .exe file must be loaded, parsed before you can even initialize and run any code.
But is there anything like process execution from ultibo already in place? It would not suprise me since you guys are chewing through this territory every day.
Being able to communicate with a child process is ultimately the trick im after.
In older systems we would allocate a fixed segment of memory and set of pointers there to various function lists.
But since we are now in multi-threaded, multi-core land, I wanted to ask first.

Any hints and tips is very welcome

Thank you
User avatar
Ultibo
Site Admin
Posts: 2079
Joined: Sat Dec 19, 2015 3:49 am
Location: Australia

Re: Usable operatingsystem with ultibo, threading + pe format?

Postby Ultibo » Sat Apr 14, 2018 3:46 am

wotanica wrote:My first question has to do with the threading model.
I am porting UAE (the amiga emulator) to pascal and im doing a bit of re-decorating from the old way this was implemented.
The idea is to allocate a thread for each of the central components of the Amiga (cpu, graphics, sound and IO) and use a message system to delegate signals between them.
Is there something like this (message queues etc) that are thread safe already in ultibo? Just so i dont re-invent the wheel.

There is an abundance of choices available for this, which one you use depends on your requirements.

First there is ThreadSendMessage() and ThreadReceiveMessage() which allow sending a TMessage structure (which looks a bit like a Windows message structure) from one thread to another regardless of CPU etc. This is very good for coordinating activity between threads so that one will wait until it receives a message from another before proceeding with an operation. There are also addition functions in this set such as ThreadWaitMessage(), ThreadAbandonMessage() and ThreadReceiveMessageEx().

Next comes the messageslot which uses the same TMessage structure as above but allows multiple threads to wait on it simultaneously, you can use it as a way to schedule work among a pool of threads who each return to waiting on the messageslot after they have completed the requested task. The functions available include MessageslotSend(), MessageslotReceive(), MessageslotReceiveEx() and MessageslotCount();

There is also the mailslot which passes an integer instead of a message structure, of course that integer can actually be a pointer to almost anything you choose (a block of memory, an object, a string). The other difference with the mailslot is that the MailslotSend() function will wait until there is space available rather than returning with an error code the way MessageslotSend() does. The mailslot again offers a very good way to queue work to other threads, the avaialble functions include MailslotSend(), MailslotSendEx(), MailslotReceive(), MailslotReceiveEx() and MailslotCount().

All of these can be found in the Threads unit and at the top of each function (or in the Wiki page for the Threads unit) you will find a brief description of the usage and parameters, if you need more info on a specific function just ask.

wotanica wrote:My second question is more elaborate. Emulation is one thing, but what about a native API for running arm executables?
I realize that this is not an easy task since the .exe file must be loaded, parsed before you can even initialize and run any code.
But is there anything like process execution from ultibo already in place? It would not suprise me since you guys are chewing through this territory every day.
Being able to communicate with a child process is ultimately the trick im after.
In older systems we would allocate a fixed segment of memory and set of pointers there to various function lists.
But since we are now in multi-threaded, multi-core land, I wanted to ask first.

The kernel to user mode interface (which is what allows processes) is one of the most complex parts of a general purpose OS and is currently a bit outside of the scope of the project, however the concept of loading and running other code is well within the realm of possibility.

We have said previously that we do have a functional prototype for loading a DLL or SO into memory and executing code from it, this is not ready yet to be released but as part of the work to allow access to develop for Ultibo using C/C++ (and other languages) we have made a significant step forward in defining how exporting of the API will actually work and the same code will support both statically compiled and dynamically compiled external libraries.

Even without dynamic library support it is quite possible with the current functionality to allocate a block of memory, change its configuration to allow code execution and then load some code into that block and execute it. Right now there is nothing automated about it and you have to do all the grunt work but for an emulator it could be quite workable.

It is even possible to map blocks of non contiguous memory into a virtual memory block that can be used by a (well behaved) piece of external code, the memory allocation in the Syscalls unit which provides memory management for external libraries does exactly this right now.

I'd recommend beginning with some simple experiments and working out the details from there to slowly build up the required functionality.
Ultibo.org | Make something amazing
https://ultibo.org
Gavinmc42
Posts: 1420
Joined: Sun Jun 05, 2016 12:38 pm
Location: Brisbane, Australia

Re: Usable operatingsystem with ultibo, threading + pe format?

Postby Gavinmc42 » Sat Apr 14, 2018 7:21 am

Amiga (cpu, graphics, sound and IO)

I count 4 things, we have 4 cores in a 3B+.

Instead of threads, how would it be done by cores?

One of things I want to do with the multicore versions is CNC stuff.
One core dedicated to GPIO etc.

The Multicore example works, does not mean I really understand it :oops:
But my brain is telling me Multicore is easier to do than multithread/multitasking.
I grasp the concept of separate cpu's doing different things better than slicing time up into bits.
wotanica
Posts: 30
Joined: Wed Feb 03, 2016 11:02 pm

Re: Usable operatingsystem with ultibo, threading + pe format?

Postby wotanica » Sun Apr 15, 2018 2:17 am

Back in the day we had a dos extender that was pretty amazing (WDOSX).
It implemented parts of WinAPI, including dll mapping, which was pretty cool for some embedded projects i did.
Sadly that is ancient by todays standard, but inspiring non-the-less.

Yes I am aware of the technical challenge of this line of coding, its probably one of the most advanced tasks a developer can do.
Libraries tend to be compiled for a relative start-address to, so one of the registers needs attention while calling or it will end in a spectacular access violation.

This is also why I wanted to start with bytecode execution, basically adopting the 68k instruction-set.
So emulation is one thing, but being able to run 68k ad-hoc is a second goal. The final goal would be to allow read, native code to run.
This is not unlike early Android experiments, but as we all remember, bytecodes on mobile devices quickly drained the battery - so a nativeapi is probably wise.

But OK, Ill poke around the kernel code and get familiar.

Thank you so much for helping me out, means a lot!
Cheers guys!
Gavinmc42
Posts: 1420
Joined: Sun Jun 05, 2016 12:38 pm
Location: Brisbane, Australia

Re: Usable operatingsystem with ultibo, threading + pe format?

Postby Gavinmc42 » Sun Apr 15, 2018 5:02 am

6502 and Z80 have been done and Davids has been talking Amiga 68xxx.
Exagear has done some serous x86 emulation on Pi's.

For me, been there, done that, used those chips before ;)
Time to learn how multicore ARM's work and especially Aarch64/NEON.
I figure by the time I master them they will be ancient history too :lol:

4 Arm cores, 48 ALU's in the VC4's 12 QPU's?
4, 8, .....1024 cpu's, how to make OS's that use all that power?
Nvidia says 1000x computing power by 2025 :o

GTX1080 has 3564 Cuda cores, 10,000 core soon? 100,000 cores, 1M cores :o
Currently I only have a vague clue on how to use 4 cores :oops:
User avatar
Ultibo
Site Admin
Posts: 2079
Joined: Sat Dec 19, 2015 3:49 am
Location: Australia

Re: Usable operatingsystem with ultibo, threading + pe format?

Postby Ultibo » Sun Apr 15, 2018 6:21 am

wotanica wrote:Back in the day we had a dos extender that was pretty amazing (WDOSX).
It implemented parts of WinAPI, including dll mapping, which was pretty cool for some embedded projects i did.
Sadly that is ancient by todays standard, but inspiring non-the-less.

While I agree that WDOSX was a cool tool (we used it for a number of things) it was a long time ago and I'm not sure that the comparison stacks up anymore.

Yes it did support loading DLL's but the WinAPI support was extremely thin (even simple stuff wasn't really supported) and it didn't do anything like threading, networking, filesystems (except accessing files via DOS), USB, multi-core, memory management, devices, graphics and so on.

Some people did put a lot of effort into making it a useful environment but by today's standards it was pretty much a one trick pony, it might be best not to let a nostalgic view of the past prevent us from seeing the power and possibilities of what is in front of us right now ;)

Loading of dynamic libraries will appear in Ultibo in due course and will cover all the nitty gritty details of code relocation, fixups, dynamic imports etc, on the other hand if you really want support for process execution then you'll need to do a lot of heavy lifting at this point.
Ultibo.org | Make something amazing
https://ultibo.org
Gavinmc42
Posts: 1420
Joined: Sun Jun 05, 2016 12:38 pm
Location: Brisbane, Australia

Re: Usable operatingsystem with ultibo, threading + pe format?

Postby Gavinmc42 » Sun Apr 15, 2018 7:10 am

Seem to remember DOS extenders were needed to get around a memory address limitation.
Yes I'm old, my first PC was 12MHz turbo 286 :lol:
We are sort of at that point now with the VC4/1GB limits.

But with 4 x 32KB caches L1 and 512KB cache L2 you could nearly run DOS in these caches.
How to do that?
in 8bit world a long time ago, 32KB was lots :D
pik33
Posts: 788
Joined: Fri Sep 30, 2016 6:30 pm
Location: Poland
Contact:

Re: Usable operatingsystem with ultibo, threading + pe format?

Postby pik33 » Sun Apr 15, 2018 1:11 pm

There is a PascalScript ready and working with Ultibo. This means a Pascal interpreter machine (instead of Basic) can be implemented.

6502 and Z80 have been done


I started some time ago a "65032" cpu; 6502 compatible but with 32-bit A, X, Y and PC registers :) I didnt do anything more than early tests; as the 65032 extensions broke 6502 illegal instructions compatibility I removed this from the ultibo-retro-gui code. Illegal instructions are needed for SID files replay.

I also thougt about something like "super-8-bit-Atari" emulator - 8 bit Atari compatible but then with this 65032 32-bit extension :)

Yes I'm old, my first PC was 12MHz turbo 286


It was my first PC too. 286, 12 MHz, 1 MB RAM, 1.2 FDD, no HDD

We are sort of at that point now with the VC4/1GB limits.


Amiga type solution can be implemented here. Low 4 GB can be a "chip ram" accessible by cpu and VC4 while the rest can be "fast ram" accessible by CPU only. This needs adding proper MMU logic to the SoC.
Gavinmc42
Posts: 1420
Joined: Sun Jun 05, 2016 12:38 pm
Location: Brisbane, Australia

Re: Usable operatingsystem with ultibo, threading + pe format?

Postby Gavinmc42 » Sun Apr 15, 2018 2:01 pm

This needs adding proper MMU logic to the SoC.

That magic is supposed to be in VC5 :D
Pi4 will fix this?

Not sure what I can do with Ultibo, 4 cores and even 1GB at the moment.
The biggest slice of memory I have needed so far was 8MB for a BMP screenshot ;)

Return to “Projects”

Who is online

Users browsing this forum: No registered users and 1 guest