SD Card Access Issue

Discussion and questions about programming with Ultibo.
asjsmile
Posts: 8
Joined: Fri May 03, 2019 10:49 am

SD Card Access Issue

Postby asjsmile » Thu May 09, 2019 1:21 pm

Dear Ultibo team,

I told that we have a problem in hw design and it's not sd card access issue in 2 days ago.

but unfortunately it's not solved after fixing hw design.
and we have still have questions on this issue and need your help.

we set the input signal as 3.3V, 60Hz to GPIO Interrupt.

when accessing to SD CARD in thread of CPU2,
the callback function in CPU1 irregularly delayed.


the blew are the answer for previous your reply post.
plz check and help us.

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////


Do you happen to have any other Pi versions to compare your results with, such as a Pi3B or Pi3B+?
=> No, i am sorry i don't have any other PI

========================================================================

Could you provide a bit more information about the tasks running on each CPU?

=> CPU0 : there is only 1 thread. interface with 2 other MCU (RS232), and it 's not using timer so it's free run mode.

=> CPU1 : there are 1+1 thread ( thread1 + subthread ) + GPIO Trigger interrupt (2EA) + Timer interrupt for 1ms interval.
thread1 : it's running with 1ms interval and processing time is normally around 450us.

=> CPU2 : this cpu is just gathering Raw data from ADC.
it's used timer interrupt with 65us period.

=> CPU3 : there is 1 thread, it handle to save data in SD CARD.

=> GPIO Interrupt : this interrupt is to measuring the frequency of input signal.
this measuring should be accurate and working consistently.
Our issue is this GPIO interrupt would be unstable when accessing SD card.

Are you using the Dedicated CPU model as shown in our examples or just creating a thread on each CPU?
=> No. i am not using "the Dedicated CPU model", just creating a thread on each CPU.

What is the nature of the external interrupt for CPU1, is it a GPIO trigger, a timer or something else?
=> it's GPIO trigger.

========================================================================

What are the characteristics of the data you are saving to the SD card,
is it a large block in one write (how large) or is it many small writes at different times?

=> there are 3 types files which i need to creat, Header file, configure file and data file.
the size of header and configure file is not much ( < 1KB )
but the data file size could be large. ( < 15MB )

it runs in one write per each files.

but i would like to say that i did test just creating a file without writing a data.
the failure symptoms happened.

apart from failure symptoms in cpu1, the files creation, writing a data in SD card works well.

Failure Symptoms :
=> the input signal for GPIO Interrupt is steadily 60 Hz ( Checked by scope )
=> we put GPIO toggling to check GPIO Trigger callback function working status. ( Toggle cycle : 60 Hz )
=> when it access SD card, the toggle cycle of GPIO Trigger callback function delayed irregularly.



========================================================================
asjsmile
Posts: 8
Joined: Fri May 03, 2019 10:49 am

Re: SD Card Access Issue

Postby asjsmile » Thu May 09, 2019 2:10 pm

i 'd like to add more information regarding of this,

when it access to SD CARD, we have checked also Timer interrupt of CPU1 and CPU2.

when it access to SD CARD, the Timer interrupt for 1ms thread loop in CPU1 and the Timer interrupt for 65us in CPU2
works well. we have put GPIO toggle and then checked by Scope. there are no problem in both timers.

SJ
User avatar
Ultibo
Site Admin
Posts: 2206
Joined: Sat Dec 19, 2015 3:49 am
Location: Australia

Re: SD Card Access Issue

Postby Ultibo » Fri May 10, 2019 1:03 am

Hello SJ,

We have a another question for you and something we'd like you to test.

From your description it sounds like the issue might be caused by the GPIO interrupts being delayed while the SD card interrupts are processed.

On the Raspberry Pi all of the peripheral interrupts go through one CPU (CPU0 by default) so the interrupt from the GPIO trigger will be occurring on CPU0 and even though you are using CPU3 to write data to the SD card the interrupts from the SD host will also be occurring on CPU0.

Is it possible for you to try a test with the data being written to a USB drive instead of the SD card?

I don't know if you are using the USB for anything else but you should be able to connect a USB flash drive to the host port on the CM3 and it should be available in Ultibo, one thing to be aware of is the USB may not appear as drive D:\ since the order of drives being discovered can change based on what devices are connected. So even if your SD card is currently C:\ drive when you boot with a USB connected the USB may become C:\ and the SD card D:\ instead.

The reason for testing the USB is because it has DMA support where the current SD card driver does not (so all transfers are done by the interrupt handler) which should mean that the USB is less likely to delay the GPIO interrupts.

asjsmile wrote:when it access to SD CARD, the Timer interrupt for 1ms thread loop in CPU1 and the Timer interrupt for 65us in CPU2
works well. we have put GPIO toggle and then checked by Scope. there are no problem in both timers.

Our other question relates to this, can we assume from your description that you are using the per CPU generic timers and not the ARM timer or Local timer devices?

The generic timers are one of the small number of devices that route interrupts to individual CPUs so using them on CPU1 and CPU2 would explain why they are not affected by the SD card interrupts on CPU0.

If we are correct about the timers and if writing the data to USB gives better results than the SD then we can think about a couple of possible ways to deal with it.
Ultibo.org | Make something amazing
https://ultibo.org
User avatar
Ultibo
Site Admin
Posts: 2206
Joined: Sat Dec 19, 2015 3:49 am
Location: Australia

Re: SD Card Access Issue

Postby Ultibo » Tue May 14, 2019 11:32 pm

Hello SJ, have you had an opportunity to test this using a USB drive?

We are wondering if adding FIQ support to the GPIO driver might help to resolve your issue but first we need to know the result of this test and get confirmation on our question about the timers.

Let us know your results.
Ultibo.org | Make something amazing
https://ultibo.org
asjsmile
Posts: 8
Joined: Fri May 03, 2019 10:49 am

Re: SD Card Access Issue

Postby asjsmile » Thu May 16, 2019 1:00 pm

Hi Ultibo Team,

have you had an opportunity to test this using a USB drive?
=> Yes, i am testing USB drive with Raspberry PI 3 model B and CM3L with my target Board which we are developing.
( i have obtained a PI 3 model B board recently )

I have learned and tested USB drive with 2 project which are ultibo provided
( Example Project : 08-FileHandling, Discovery Project : 02-ExploringUSB )

With a example project of " 08-FileHandling " in RPI3-Model-B,
I have succeed to access USB dirve in D:\.
( there was small coding modify work such as adding "DWCOTG" units, and remove "DeleteFile(Filename);" for compile error )

But i have still failed to access in my target board with CM3L for several days.

if i mention about H/W related with this, the CM3L connected with LAN9514,
there is one upstream is conected with CM3L USB D+,D-, there are 4 downstearm ( D2P,D2N ~ D5P,D5N ), the D2P/N is now using RS232 for debug console,
the D3P/N is now using for USB drive and the others will be used for 2xRS485 ( but they are not used for usb test ) and 1xEthernet.

In H/W side, we have taken some waves by scope with differential probe and compared good and bad cases.
( -> Good case : RPI3 model B with example 08 project. -> Bad case : My target board with CM3L )
The wave seems almost same. i can't find any problem in H/W part.

but i can't get a D:\ drive, and the usb stick is flickering persistingly.
As i know, the usb drivers are running in RTL and i am not an expert in USB and Ultibo.
i have tried to study, RTL for USB, but it's too difficult for me to understand.
please advise me as many as possible !!!!

We are wondering if adding FIQ support to the GPIO driver might help to resolve your issue but first we need to know the result of this test and get confirmation on our question about the timers.
Let us know your results.
=> about the timer, it's right as you assumed, they are Generic timer in CPU1, CPU2.

=> i am sorry i don't know how to use FIQ, i need to study first.
would you give me some examples or link to study for FIQ application related with this issue.

SJ
User avatar
Ultibo
Site Admin
Posts: 2206
Joined: Sat Dec 19, 2015 3:49 am
Location: Australia

Re: SD Card Access Issue

Postby Ultibo » Fri May 17, 2019 12:08 am

asjsmile wrote: But i have still failed to access in my target board with CM3L for several days.

if i mention about H/W related with this, the CM3L connected with LAN9514,
there is one upstream is conected with CM3L USB D+,D-, there are 4 downstearm ( D2P,D2N ~ D5P,D5N ), the D2P/N is now using RS232 for debug console,
the D3P/N is now using for USB drive and the others will be used for 2xRS485 ( but they are not used for usb test ) and 1xEthernet.

There can be a lot of factors that affect USB timing and because you are designing your own board it is very difficult to know exactly where the problem might be.

I'm assuming that the LAN9514 is working and the RS232 (an FTDI device maybe?) is also working, all that should be needed to enable support for USB disks is to make sure your project includes the USB, DWCOTG, Storage, FileSystem and FATFS units, or just include the RaspberryPi3 unit which includes all of these anyway.

If your design doesn't require USB disk support then it probably isn't necessary to spend a lot of time trying to work out where the problem is, since you have confirmed the timer usage then we can take a guess that the problem with the SD card is likely to be due to the interaction between the SD card and GPIO interrupts on CPU0.

asjsmile wrote: => i am sorry i don't know how to use FIQ, i need to study first.
would you give me some examples or link to study for FIQ application related with this issue.

Currently the GPIO driver in Ultibo doesn't have FIQ support but it is something which we could add quite easily so you can enable it in your project. Since FIQ (Fast interrupt) takes priority over IRQ (Standard Interrupt) then if the GPIO is using FIQ it should not be delayed while the SD card is writing data.

The other option we can see is to enable DMA support in the SD card driver but that is more complex to do so let's start with the FIQ support in the GPIO driver and see if that makes a difference.

We'll make the necessary changes and post and update when they are ready to test.
Ultibo.org | Make something amazing
https://ultibo.org
asjsmile
Posts: 8
Joined: Fri May 03, 2019 10:49 am

Re: SD Card Access Issue

Postby asjsmile » Fri May 17, 2019 3:18 am

to make sure your project includes the USB, DWCOTG, Storage, FileSystem and FATFS units, or just include the RaspberryPi3 unit which includes all of these anyway.


=> yes, i have check again the units that USB, DWCOTG, Storage, FileSystem and FATFS units, or just include the RaspberryPi3 are included.

If your design doesn't require USB disk support then it probably isn't necessary to spend a lot of time trying to work out where the problem is,
since you have confirmed the timer usage then we can take a guess that the problem with the SD card is likely to be due to the interaction between the SD card and GPIO interrupts on CPU0.


=> there is another reason to test USB disk, we are thinking saving time in SD card is slow. if USB disk would be faster, we will consider use it for event data saving.

Currently the GPIO driver in Ultibo doesn't have FIQ support but it is something which we could add quite easily so you can enable it in your project.
Since FIQ (Fast interrupt) takes priority over IRQ (Standard Interrupt) then if the GPIO is using FIQ it should not be delayed while the SD card is writing data.
The other option we can see is to enable DMA support in the SD card driver but that is more complex to do so let's start with the FIQ support in the GPIO driver
and see if that makes a difference.


=> Would you let me know, when can i get the update, FIQ support ?
I am sorry to say like this we are now urgent. our plan to get a certification test also delayed.
Hope your understanding.
User avatar
Ultibo
Site Admin
Posts: 2206
Joined: Sat Dec 19, 2015 3:49 am
Location: Australia

Re: SD Card Access Issue

Postby Ultibo » Fri May 17, 2019 4:00 am

asjsmile wrote:=> Would you let me know, when can i get the update, FIQ support ?

Just before we go down that path, I think I overlooked before that you said the GPIO interrupt rate should be 60Hz which is actually not very high (about 16ms between interrupts).

I think I might have confused that value with the 65us and 1ms timers, can I ask you to confirm that you are using the GPIODeviceInputEvent() function and if so are you passing the flag GPIO_EVENT_FLAG_INTERRUPT to that function?

We've discussed the use GPIO_EVENT_FLAG_INTERRUPT in a number of other posts (you can find them in a search) but basically it allows you to receive the event trigger directly from the interrupt handler and not via a worker thread. If you are not using this flag then the FIQ probably won't help because the delay will be due to the worker threads getting queued behind the threads doing the SD card writing.

There are strict rules for what you are allowed to do in the callback if you are using GPIO_EVENT_FLAG_INTERRUPT but if you need precise timing then it is the proper way to do it.

Let me know.
Ultibo.org | Make something amazing
https://ultibo.org
asjsmile
Posts: 8
Joined: Fri May 03, 2019 10:49 am

Re: SD Card Access Issue

Postby asjsmile » Fri May 17, 2019 5:04 am

Hi Ultibo Team,

yes, 60hz is not high frequency, but the tolerance is within +/-0.005 Hz.
as you said, we need precise timing.

and we are using " GPIO_EVENT_FLAG_INTERRUPT "

for your information, please refer to the below code for usage about GPIO_EVENT_FLAG_INTERRUPT in my project.

if anything is wrong, let me know.

Code: Select all

procedure Freq_Input_Setup;
var

begin

    ...
    // FREQ0_SYNC = GPIO42;
    // FREQ1_SYNC = GPIO43;
    GPIODeviceInputEvent(GPIODeviceGetDefault, FREQ0_SYNC, GPIO_TRIGGER_RISING, GPIO_EVENT_FLAG_INTERRUPT or GPIO_EVENT_FLAG_REPEAT, INFINITE, @Freq0_edge_Callback, nil);
    GPIODeviceInputEvent(GPIODeviceGetDefault, FREQ1_SYNC, GPIO_TRIGGER_RISING, GPIO_EVENT_FLAG_INTERRUPT or GPIO_EVENT_FLAG_REPEAT, INFINITE, @Freq1_edge_Callback, nil);   

    ...

end;     

procedure Freq0_edge_Callback(Data:Pointer;Pin,Trigger:LongWord);
begin

    FreqGetTime0[FreqCount0] := ClockGetTotal;

    Inc(FreqCount0);
    if(FreqCount0 > 5) then FreqCount0 := 0;
end;
User avatar
Ultibo
Site Admin
Posts: 2206
Joined: Sat Dec 19, 2015 3:49 am
Location: Australia

Re: SD Card Access Issue

Postby Ultibo » Sat May 18, 2019 12:56 am

We've just pushed an update that adds FIQ support for the GPIO driver in the Pi (all models).

To make use of this you'll need to download the latest RTL source from GitHub and rebuild your RTL.

The simplest option to enable FIQ for the GPIO is to use an init unit which we've shown in other examples, basically you create a unit that contains just the following:

Code: Select all

unit InitUnit;

interface

uses GlobalConfig;

implementation


initialization

 // Enable GPIO FIQ
 BCM2710GPIO_FIQ_ENABLED := True;
 BCM2710GPIO_FIQ_BANK_NO := 1;
 
end.

and then include it in your program file as the very first item in the uses clause like this:

Code: Select all

program MyProgram;

uses
 InitUnit,
 ...


Note that there are separate interrupts for each GPIO bank in the Pi and because only one FIQ is allowed we've added an extra parameter to specify which bank number to enable it for. Based on your code snippet above you are using GPIO42 and 43 which will both be on bank 1. There is an alternate way to make all GPIO interrupts appear as one single FIQ, if this change works in your testing then we'll look at modifying the driver to support that option as well.

Let us know your results.

asjsmile wrote:
I am sorry to say like this we are now urgent. our plan to get a certification test also delayed.

Just a friendly reminder to everyone, Ultibo is an open source project so if you think you might need our help with an issue please allow us plenty of time.

We try to help with every issue reported but we may not always have time to look at the problem immediately, we also like to ensure everyone's issues get attention, not just those in a hurry.

Thanks.
Ultibo.org | Make something amazing
https://ultibo.org

Return to “General”

Who is online

Users browsing this forum: No registered users and 0 guests