Datalogger Ultibo/ATmega328p

The place to share and discuss your Ultibo projects.
Brutus
Posts: 32
Joined: Sun Jan 20, 2019 1:24 pm

Datalogger Ultibo/ATmega328p

Postby Brutus » Mon Apr 29, 2019 4:45 pm

Hi guys,

Long time no posting because I've been very busy on my project.

I've been reading the forum every so often and it is already a great knowledge base you have there.

The datalogger project I'm working on originally targeted being able being able to datalog up to 5 analog sensors at millisecond sampling rate on my old car (a Mazda MX3 V6) for engine diagnostics.

This was working more a less right with raspbian/ATMega, but there were glitches in the datalogging so I wasn't happy at all.

I then started with Ultibo 1 year or so ago and to make a long story short, I can now easily datalog 48 analog sensors using 8 ATMegas (Arduino pro mini hardware) in parallel and hundreds of digital or slow analog sensors in less than a millisecond without any glitches during writing on USB stick, all this displaying fancy FullHD fancy graphics at 60FPS (still no glitches).

The communication metrics are showing less than 20us from ATMega to availability in the RPi memory which is well above what the ATMega ADC is capable running 10 bits conversion.

This requires a RPi3 A+ or B+.
An RPi3 without the "+" will just reach 85°C+ in no time! (more on that later)
The ATMega 328p runs in pure assembler without a single byte RAM used (all registers).

The communication Protocol is home made. I started with SPI but figured out that high speed communication was not possible over a meter or so, which I did require.

The protocol is now basically a 2 bits parallel conversational communication with one clock from master and one clock from slave. This is of course done through bitbanging but still allows 50000 communications per second (4 bits query, 10 bits reply).

So if considering actual data payload, my bitbanging wouldn't look that ridiculous compared to a Mhz SPI communication like the ATMega is capable of and I am not limited by a meter or so anymore.

Now my next steps are:

1/Use Garry's advices on using timers of each dedicated processor instead of looping stupidly and pushing the processor 100% waiting for an event.

2/ .png screenshots to post a few pictures to you guys (and maybe become a superstar in having my application figuring in Ultibo application showcase) :D

Inputs on the things I learnt on the way:

-OpenVG will only run if nested on CPU0.

-When I tried a

Code: Select all

GPIODeviceInputEvent(GPIODeviceGetDefault, PIN_MI, GPIO_TRIGGER_EDGE, GPIO_EVENT_FLAG_REPEAT or GPIO_EVENT_FLAG_INTERRUPT, INFINITE, @edge_callback, nil);
 
OpenVG was showing nasty glitches all around. Everything goes to normal if you remove "GPIO_EVENT_FLAG_INTERRUPT".

-From what I found in doing my first true multi-processor application (only Ultibo allows to do that easily) is that the shared memory access feels fantastic at the beginning because it's just like each core feels like a separate system that can communicate with others almost instantly.
The funny thing is that when you start pushing things on each core with a millisecond maximum cycle time, this becomes the main bottleneck.

I'm now dreaming of better registers usage to lessen memory access (A bit like I did on the ATMega :lol: ).
Assembly programming can certainly give incredible results on a multi processor RPi! :)
Gavinmc42
Posts: 1586
Joined: Sun Jun 05, 2016 12:38 pm
Location: Brisbane, Australia

Re: Datalogger Ultibo/ATmega328p

Postby Gavinmc42 » Tue Apr 30, 2019 12:18 am

Glitch free datalogging :D
Part of the first things I tried when I found Ultibo.
Had to wait for hardware i2c to get working and then I never looked back at Python on Raspbian, yuk.
In between was some shell script logging on PiCore, still had glitches, just not as many.

I do i2c upto 5M but only at 10khz.
Some sort of RS485 differential signaling would be nice.
SPI to LVDS transcievers? Hmm, more googling ;)
User avatar
Ultibo
Site Admin
Posts: 2207
Joined: Sat Dec 19, 2015 3:49 am
Location: Australia

Re: Datalogger Ultibo/ATmega328p

Postby Ultibo » Tue Apr 30, 2019 12:20 am

Hi Brutus,

This is excellent, sounds like you are getting some really good results and we'll look forward to seeing some screenshots or photos of your project in operation.

Brutus wrote:-OpenVG will only run if nested on CPU0.

From what we can figure it is the EGL layer that has some code that locks it to the thread it was initialized by, so in theory you could start OpenVG on any CPU as long as it remains on the same thread during use. It's not uncommon for graphics code to be only suitable for running on one CPU (or one thread), the entire Delphi VCL is only accessible from the main application thread.

Brutus wrote:-From what I found in doing my first true multi-processor application (only Ultibo allows to do that easily) is that the shared memory access feels fantastic at the beginning because it's just like each core feels like a separate system that can communicate with others almost instantly.
The funny thing is that when you start pushing things on each core with a millisecond maximum cycle time, this becomes the main bottleneck.

That's a very good point, we see a lot of people mistakenly think that multiple cores will make everything instantly faster but the reality of sharing memory and keeping everything in sync does come at a cost.

Brutus wrote:I'm now dreaming of better registers usage to lessen memory access (A bit like I did on the ATMega :lol: ).
Assembly programming can certainly give incredible results on a multi processor RPi! :)

The sky really is the limit, once you have a working implementation then you can simply improve each element in turn until you get to where you want to go.

Thanks and keep us up to date with your progress.
Ultibo.org | Make something amazing
https://ultibo.org
Brutus
Posts: 32
Joined: Sun Jan 20, 2019 1:24 pm

Re: Datalogger Ultibo/ATmega328p

Postby Brutus » Tue Apr 30, 2019 6:47 am

Ultibo wrote:From what we can figure it is the EGL layer that has some code that locks it to the thread it was initialized by, so in theory you could start OpenVG on any CPU as long as it remains on the same thread during use. It's not uncommon for graphics code to be only suitable for running on one CPU (or one thread), the entire Delphi VCL is only accessible from the main application thread.


You are right, I think I started my graphics thread that includes OPENVG calls from a different core than the one I intended it to run on, which crashed it immediately.

So my punch list now:
-Timer on different cores as per previous discussion and inputs from you.
-OpenVG on other core than CPU0.
-Screenshot capturing framebuffer and saving it to .png (anybody knows if this has been done before on Ultibo?).
Gavinmc42
Posts: 1586
Joined: Sun Jun 05, 2016 12:38 pm
Location: Brisbane, Australia

Re: Datalogger Ultibo/ATmega328p

Postby Gavinmc42 » Tue Apr 30, 2019 8:47 am

Yep screenshot has been done, Ron was the first to do it very early?
viewtopic.php?f=15&t=534&hilit=screenshot+fc+image

I used FPimage to take that screenshot saved to ram drive and converted it to a png before saving to SD.

Garry pointed out the trick to get the OpenVG captured which I still need to try.
viewtopic.php?f=9&t=1269&p=8929&hilit=openvg+screenshot#p8929
Gavinmc42
Posts: 1586
Joined: Sun Jun 05, 2016 12:38 pm
Location: Brisbane, Australia

Re: Datalogger Ultibo/ATmega328p

Postby Gavinmc42 » Tue Apr 30, 2019 8:51 am

uscreenshot.pas is in the src of the ultibo units
https://github.com/ultibohub/Core/tree/ ... ounits/src
Gavinmc42
Posts: 1586
Joined: Sun Jun 05, 2016 12:38 pm
Location: Brisbane, Australia

Re: Datalogger Ultibo/ATmega328p

Postby Gavinmc42 » Tue Apr 30, 2019 8:53 am

Yep , I remember now about the datetime issue when writing files on non connected Pi's.
pik33
Posts: 857
Joined: Fri Sep 30, 2016 6:30 pm
Location: Poland
Contact:

Re: Datalogger Ultibo/ATmega328p

Postby pik33 » Tue Apr 30, 2019 10:40 am

The funny thing is that when you start pushing things on each core with a millisecond maximum cycle time, this becomes the main bottleneck.


The memory subsystem is a main bottleneck for RPi. You can overclock it in config.txt up to 600 MHz giving it up to 1.2 G transfers/sec. The same memory is used by GPU and its additional layers (egl): use 4 fullscreen layers and the RPi wil be slooooooooooooow. You may optimize your code to minimize memory accesses when possible.

Assembly programming can certainly give incredible results on a multi processor RPi!


ASM can speed things up to 20x :) as I experienced writing my SID emulator. First, you can optimize memory accesses. Second, you can place your constants and variables in the code area (after modifying memory block MMU flags) which saves one ldr every access. Third, you decide which variables will be kept in registers - I even use R14 which is safe while not calling anything from the procedure. And then instead calling math lib while multiplying long variables you can use umull/smull/umlal/smlal .
Brutus
Posts: 32
Joined: Sun Jan 20, 2019 1:24 pm

Re: Datalogger Ultibo/ATmega328p

Postby Brutus » Fri May 03, 2019 6:12 am

A lot of very interesting ideas, thank you.

pik33 wrote:The memory subsystem is a main bottleneck for RPi. You can overclock it in config.txt up to 600 MHz giving it up to 1.2 G transfers/sec.


I'll give it a go just for fun and see where the processor temperature goes...

pik33 wrote:The same memory is used by GPU and its additional layers (egl): use 4 fullscreen layers and the RPi wil be slooooooooooooow. You may optimize your code to minimize memory accesses when possible.


I use two Full HD layers for testing purpose, but really updates only one very often.
The final display will be much smaller and will not require more than 800x600 so this might not be a concern for me, but for my own curiosity, does the GPU updates them from memory whatever there was a change on the layer or not?

pik33 wrote:ASM can speed things up to 20x :) as I experienced writing my SID emulator. First, you can optimize memory accesses. Second, you can place your constants and variables in the code area (after modifying memory block MMU flags) which saves one ldr every access. Third, you decide which variables will be kept in registers - I even use R14 which is safe while not calling anything from the procedure. And then instead calling math lib while multiplying long variables you can use umull/smull/umlal/smlal .

[/quote]

I catch it for memory access optimization and the use of registers as well as using assembler for multiplication, but I have no idea about moving the variables and constants to the code area (although I think understand why it would save an ldr command).
I think I eventually need to take the time to read BCM2835 documentation.
Brutus
Posts: 32
Joined: Sun Jan 20, 2019 1:24 pm

Re: Datalogger Ultibo/ATmega328p

Postby Brutus » Fri May 03, 2019 6:27 am

Gavinmc42 wrote:Yep screenshot has been done, Ron was the first to do it very early?
viewtopic.php?f=15&t=534&hilit=screenshot+fc+image

I used FPimage to take that screenshot saved to ram drive and converted it to a png before saving to SD.

Garry pointed out the trick to get the OpenVG captured which I still need to try.
viewtopic.php?f=9&t=1269&p=8929&hilit=openvg+screenshot#p8929


Thanks Gavin,

I knew about uScreenshot but I really wanted to save the image as .png and I was missing the dispmanx feature to capture the screen so your links are going to be very useful. :)

Return to “Projects”

Who is online

Users browsing this forum: Gavinmc42 and 0 guests